[Enter `^Ec?' for help] [spying] [ 176.460494] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 176.701341] XFS (dm-5): Mounting V5 Filesystem [ 176.710281] XFS (dm-5): Ending clean mount [ 176.796685] XFS (dm-5): Unmounting Filesystem [ OK ] Stopped Nagios Remote Plugin Executor. [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Login Service. [ OK ] Stopped User Manager for UID 1000. Stopping User runtime directory /run/user/1000... [ OK ] Unmounted /run/user/1000. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped User runtime directory /run/user/1000. [ OK ] Removed slice User Slice of UID 1000. Stopping Permit User Sessions... [ OK ] Stopped Permit User Sessions. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target NFS client services. Stopping GSSAPI Proxy Daemon... [ OK ] Stopped target User and Group Name Lookups. Stopping System Security Services Daemon... [ OK ] Stopped GSSAPI Proxy Daemon. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped System Security Services Daemon. [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped VDO volume services. [ OK ] Stopped target Basic System. [ OK ] Stopped Forward Password Requests to Plymouth Directory Watch. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed D-Bus System Message Bus Socket. [ OK [[ 178.040087] audit: type=1305 audit(1699258525.504:449): op=set audit_pid=0 old=736 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 0m] Closed SSSD Kerberos Cache Manager responder socket. [ OK ] C[ 178.061226] audit: type=1131 audit(1699258525.525:450): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' losed Avahi mDNS[ 178.083094] audit: type=1131 audit(1699258525.547:451): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' /DNS-SD Stack Ac[ 178.106749] audit: type=1131 audit(1699258525.571:452): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' tivation Socket.[ 178.130389] audit: type=1131 audit(1699258525.594:453): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK [ 178.153727] audit: type=1131 audit(1699258525.618:454): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysusers comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ] Closed CU[ 178.176825] audit: type=1131 audit(1699258525.641:455): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' PS Scheduler. [ OK ] Stopped target Paths. [ OK ] Stopped CUPS Scheduler. [ OK ] Stopped target System Initializati[ 178.213367] audit: type=1131 audit(1699258525.677:456): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-monitor comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' on. [ [ 178.235645] audit: type=1130 audit(1699258525.700:457): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 178.258461] audit: type=1131 audit(1699258525.700:458): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' OK ] Stopped target Swap. Stopping Load/Save Random Seed... [ OK ] Stopped Update is Completed. [ OK ] Stopped Rebuild Dynamic Linker Cache. [ OK ] Stopped Rebuild Hardware Database. [ OK ] Stopped Rebuild Journal Catalog. [ OK ] Stopped Apply K[ 178.308549] watchdog: watchdog0: watchdog did not stop! ernel Variables.[ 178.315142] printk: systemd-shutdow: 33 output lines suppressed due to ratelimiting [ OK ] Stopped Load Kernel Modul[ 178.327515] systemd-shutdown[1]: Syncing filesystems and block devices. es. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Forward Password Requests to Wall Directory Watch. Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. Stopping Security Auditing Service... [ OK ] Stopped Security Auditing Service. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Import network configuration from initramfs. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors…ng dmeventd or progress polling... [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create System Users. [ OK ] Stopped Remount Root and Kernel File Systems. [ OK ] Stopped Monitoring of LVM2 mirrors,…sing dmeventd or progress polling. [ OK ] Reached target Shutdown. [ OK ] Reached target Final Step. [ OK ] Started Reboot. [ OK [ 178.438246] systemd-shutdown[1]: Sending SIGTERM to remaining processes... ] Reached target Reboot. [ 178.466094] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 178.475089] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0 [ 178.482039] systemd-shutdown[1]: Unmounting file systems. [ 178.488071] [16313]: Remounting '/' read-only in with options 'seclabel'. [ 178.590815] EXT4-fs (sda1): re-mounted. Opts: [ 178.596126] systemd-shutdown[1]: All filesystems unmounted. [ 178.601701] systemd-shutdown[1]: Deactivating swaps. [ 178.606722] systemd-shutdown[1]: All swaps deactivated. [ 178.611942] systemd-shutdown[1]: Detaching loop devices. [ 178.617331] systemd-shutdown[1]: All loop devices detached. [ 178.784838] kvm: exiting hardware virtualization [ 178.789802] sd 4:0:0:0: [sda] Synchronizing SCSI cache [ 181.117278] reboot: Restarting system [ 181.120943] reboot: machine restart € € € Initializing Intel(R) Boot Agent XE v2.1.60 PXE 2.1 Build 088 (WfM 2.0)  Press Ctrl+S to enter the Setup Menu.                      Press Ctrl+S to enter the Setup Menu..  € €  el(R) Boot Agent XE v2.1.60 PXE 2.1 Build 088 (WfM 2.0)  Press Ctrl+S to enter the Setup Menu.                      Initializing Intel(R) Boot Agent XE v2.1.60 Press Ctrl+S to enter the Setup Menu..  € €  el(R) Boot Agent GE v1.5.72 PXE 2.1 Build 092 (WfM 2.0) Press Ctrl+S to enter the Setup Menu.                       Initializing Intel(R) Boot Agent GE v1.5.72 Press Ctrl+S to enter the Setup Menu..  € € € €  Intel(R) Boot Agent XE v2.1.60 Copyright (C) 1997-2009, Intel Corporation  CLIENT MAC ADDR: 0C C4 7A 88 6D F5 GUID: 00000000 0000 0000 0000 0CC47AD9382C DHCP.|                    DHCP.- DHCP.\ DHCP.| DHCP./ DHCP.- DHCP.\ DHCP.| CLIENT IP: 172.21.15.186 MASK: 255.255.240.0 DHCP IP: 172.21.0.10 GATEWAY IP: 172.21.15.254 PXE->EB: !PXE at 95D3:0070, entry point at 95D3:0106  UNDI code segment 95D3:3E70, data segment 8E94:73F0 (570-615kB)  UNDI device is PCI 03:00.1, type DIX+802.3  570kB free base memory after PXE unload iPXE initialising devices... iPXE initialising devices...ok iPXE 1.20.1+ (g4bd0) -- Open Source Network Boot Firmware -- http://ipxe.org Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP VLAN AoE ELF MBOOT PXE bzImage Menu PXEXT Configuring (net0 0c:c4:7a:88:6d:f5)... Configuring (net0 0c:c4:7a:88:6d:f5).... Configuring (net0 0c:c4:7a:88:6d:f5)..... Configuring (net0 0c:c4:7a:88:6d:f5)...... ok error: ../../grub-core/fs/fshelp.c:258:file `/boot/grub2/i386-pc/increment.mod' not found. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   Red Hat Enterprise Linux (6.6.0-rc5-g5b73517e0e2f) 8.4 (Ootpa)  Red Hat Enterprise Linux (4.18.0-425.3.1.el8.x86_64) 8.7 (Ootpa)  Red Hat Enterprise Linux (4.18.0-305.25.1.el8_4.x86_64) 8.4 (Ootpa)  Red Hat Enterprise Linux (4.18.0-80.el8.x86_64) 8.0 (Ootpa)  Red Hat Enterprise Linux (0-rescue-298e228f3b5e401e8ac0f5ae159a2e06) 8.0>            The selected entry will be started automatically in 5s.   Red Hat Enterprise Linux (6.6.0-rc5-g5b73517e0e2f) 8.4 (Ootpa)   Red Hat Enterprise Linux (4.18.0-425.3.1.el8.x86_64) 8.7 (Ootpa)  Red Hat Enterprise Linux (4.18.0-305.25.1.el8_4.x86_64) 8.4 (Ootpa)  Red Hat Enterprise Linux (4.18.0-80.el8.x86_64) 8.0 (Ootpa)  Red Hat Enterprise Linux (0-rescue-298e228f3b5e401e8ac0f5ae159a2e06) 8.0*           Use the * and * keys to change the selection.  Press 'e' to edit the selected item, or 'c' for a command prompt.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h        [ 0.000000] Linux version 6.6.0-rc5-g5b73517e0e2f (jenkins-build@braggi08) (gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-21), GNU ld version 2.30-113.el8) #1 SMP PREEMPT_DYNAMIC Wed Oct 25 03:22:38 UTC 2023 [ 0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f root=UUID=5baff67d-813b-4ad8-bba1-8331ae7ea36f ro crashkernel=auto console=ttyS1,115200 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000008abff] usable [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000078ec5fff] usable [ 0.000000] BIOS-e820: [mem 0x0000000078f78000-0x00000000798a7fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000798a8000-0x0000000079d44fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x0000000079d45000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000087fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] APIC: Static calls initialized [ 0.000000] SMBIOS 3.0 present. [ 0.000000] DMI: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 3499.727 MHz processor [ 0.000313] last_pfn = 0x880000 max_arch_pfn = 0x400000000 [ 0.000318] MTRR map: 5 entries (3 fixed + 2 variable; max 23), built from 10 variable MTRRs [ 0.000320] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000683] last_pfn = 0x78ec6 max_arch_pfn = 0x400000000 [ 0.005881] found SMP MP-table at [mem 0x000fcd40-0x000fcd4f] [ 0.005890] Using GB pages for direct mapping [ 0.006113] RAMDISK: [mem 0x315ec000-0x34aedfff] [ 0.006115] ACPI: Early table checksum verification disabled [ 0.006118] ACPI: RSDP 0x00000000000F05B0 000024 (v02 SUPERM) [ 0.006122] ACPI: XSDT 0x00000000798A80A0 0000BC (v01 01072009 AMI 00010013) [ 0.006127] ACPI: FACP 0x00000000798DB838 00010C (v05 SUPERM SMCI--MB 01072009 AMI 00010013) [ 0.006132] ACPI: DSDT 0x00000000798A81F0 033641 (v02 SUPERM SMCI--MB 01072009 INTL 20091013) [ 0.006136] ACPI: FACS 0x0000000079D43F80 000040 [ 0.006139] ACPI: APIC 0x00000000798DB948 0000C8 (v03 SUPERM SMCI--MB 01072009 AMI 00010013) [ 0.006142] ACPI: FPDT 0x00000000798DBA10 000044 (v01 SUPERM SMCI--MB 01072009 AMI 00010013) [ 0.006145] ACPI: FIDT 0x00000000798DBA58 00009C (v01 SUPERM SMCI--MB 01072009 AMI 00010013) [ 0.006148] ACPI: SPMI 0x00000000798DBAF8 000040 (v05 SUPERM SMCI--MB 00000000 AMI. 00000000) [ 0.006151] ACPI: MCFG 0x00000000798DBB38 00003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097) [ 0.006155] ACPI: UEFI 0x00000000798DBB78 000042 (v01 SUPERM SMCI--MB 01072009 00000000) [ 0.006158] ACPI: HPET 0x00000000798DBBC0 000038 (v01 SUPERM SMCI--MB 00000001 INTL 20091013) [ 0.006161] ACPI: WDDT 0x00000000798DBBF8 000040 (v01 SUPERM SMCI--MB 00000000 INTL 20091013) [ 0.006164] ACPI: SSDT 0x00000000798DBC38 01717F (v02 SUPERM PmMgt 00000001 INTL 20120913) [ 0.006167] ACPI: NITR 0x00000000798F2DB8 000071 (v02 SUPERM SMCI--MB 00000001 INTL 20091013) [ 0.006170] ACPI: SSDT 0x00000000798F2E30 00264C (v02 SUPERM SpsNm 00000002 INTL 20120913) [ 0.006174] ACPI: SSDT 0x00000000798F5480 000064 (v02 SUPERM SpsNvs 00000002 INTL 20120913) [ 0.006177] ACPI: PRAD 0x00000000798F54E8 000102 (v02 SUPERM SMCI--MB 00000002 INTL 20120913) [ 0.006180] ACPI: DMAR 0x00000000798F55F0 0000BC (v01 SUPERM SMCI--MB 00000001 INTL 20091013) [ 0.006183] ACPI: HEST 0x00000000798F56B0 00027C (v01 SUPERM SMCI--MB 00000001 INTL 00000001) [ 0.006186] ACPI: BERT 0x00000000798F5930 000030 (v01 SUPERM SMCI--MB 00000001 INTL 00000001) [ 0.006189] ACPI: ERST 0x00000000798F5960 000230 (v01 SUPERM SMCI--MB 00000001 INTL 00000001) [ 0.006192] ACPI: EINJ 0x00000000798F5B90 000130 (v01 SUPERM SMCI--MB 00000001 INTL 00000001) [ 0.006195] ACPI: Reserving FACP table memory at [mem 0x798db838-0x798db943] [ 0.006196] ACPI: Reserving DSDT table memory at [mem 0x798a81f0-0x798db830] [ 0.006197] ACPI: Reserving FACS table memory at [mem 0x79d43f80-0x79d43fbf] [ 0.006198] ACPI: Reserving APIC table memory at [mem 0x798db948-0x798dba0f] [ 0.006199] ACPI: Reserving FPDT table memory at [mem 0x798dba10-0x798dba53] [ 0.006200] ACPI: Reserving FIDT table memory at [mem 0x798dba58-0x798dbaf3] [ 0.006201] ACPI: Reserving SPMI table memory at [mem 0x798dbaf8-0x798dbb37] [ 0.006202] ACPI: Reserving MCFG table memory at [mem 0x798dbb38-0x798dbb73] [ 0.006203] ACPI: Reserving UEFI table memory at [mem 0x798dbb78-0x798dbbb9] [ 0.006204] ACPI: Reserving HPET table memory at [mem 0x798dbbc0-0x798dbbf7] [ 0.006205] ACPI: Reserving WDDT table memory at [mem 0x798dbbf8-0x798dbc37] [ 0.006206] ACPI: Reserving SSDT table memory at [mem 0x798dbc38-0x798f2db6] [ 0.006206] ACPI: Reserving NITR table memory at [mem 0x798f2db8-0x798f2e28] [ 0.006207] ACPI: Reserving SSDT table memory at [mem 0x798f2e30-0x798f547b] [ 0.006208] ACPI: Reserving SSDT table memory at [mem 0x798f5480-0x798f54e3] [ 0.006209] ACPI: Reserving PRAD table memory at [mem 0x798f54e8-0x798f55e9] [ 0.006210] ACPI: Reserving DMAR table memory at [mem 0x798f55f0-0x798f56ab] [ 0.006211] ACPI: Reserving HEST table memory at [mem 0x798f56b0-0x798f592b] [ 0.006212] ACPI: Reserving BERT table memory at [mem 0x798f5930-0x798f595f] [ 0.006213] ACPI: Reserving ERST table memory at [mem 0x798f5960-0x798f5b8f] [ 0.006214] ACPI: Reserving EINJ table memory at [mem 0x798f5b90-0x798f5cbf] [ 0.006284] No NUMA configuration found [ 0.006285] Faking a node at [mem 0x0000000000000000-0x000000087fffffff] [ 0.006289] NODE_DATA(0) allocated [mem 0x87ffeb000-0x87fffdfff] [ 0.006302] crashkernel: memory value expected [ 0.062706] Zone ranges: [ 0.062709] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.062712] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.062713] Normal [mem 0x0000000100000000-0x000000087fffffff] [ 0.062715] Movable zone start for each node [ 0.062717] Early memory node ranges [ 0.062717] node 0: [mem 0x0000000000001000-0x0000000000089fff] [ 0.062719] node 0: [mem 0x0000000000100000-0x0000000078ec5fff] [ 0.062721] node 0: [mem 0x0000000100000000-0x000000087fffffff] [ 0.062725] Initmem setup node 0 [mem 0x0000000000001000-0x000000087fffffff] [ 0.062730] On node 0, zone DMA: 1 pages in unavailable ranges [ 0.062764] On node 0, zone DMA: 118 pages in unavailable ranges [ 0.119506] On node 0, zone Normal: 28986 pages in unavailable ranges [ 0.119618] ACPI: PM-Timer IO Port: 0x408 [ 0.119625] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) [ 0.119626] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) [ 0.119627] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1]) [ 0.119628] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1]) [ 0.119629] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) [ 0.119630] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) [ 0.119631] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1]) [ 0.119631] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1]) [ 0.119646] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 [ 0.119651] IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47 [ 0.119653] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.119655] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.119659] ACPI: Using ACPI (MADT) for SMP configuration information [ 0.119660] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.119664] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0xb000020 (or later) [ 0.119665] smpboot: Allowing 8 CPUs, 0 hotplug CPUs [ 0.119675] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.119677] PM: hibernation: Registered nosave memory: [mem 0x0008a000-0x000dffff] [ 0.119678] PM: hibernation: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.119679] PM: hibernation: Registered nosave memory: [mem 0x78ec6000-0x78f77fff] [ 0.119681] PM: hibernation: Registered nosave memory: [mem 0x78f78000-0x798a7fff] [ 0.119681] PM: hibernation: Registered nosave memory: [mem 0x798a8000-0x79d44fff] [ 0.119682] PM: hibernation: Registered nosave memory: [mem 0x79d45000-0x8fffffff] [ 0.119683] PM: hibernation: Registered nosave memory: [mem 0x90000000-0xfed1bfff] [ 0.119684] PM: hibernation: Registered nosave memory: [mem 0xfed1c000-0xfed44fff] [ 0.119685] PM: hibernation: Registered nosave memory: [mem 0xfed45000-0xfeffffff] [ 0.119686] PM: hibernation: Registered nosave memory: [mem 0xff000000-0xffffffff] [ 0.119689] [mem 0x90000000-0xfed1bfff] available for PCI devices [ 0.119690] Booting paravirtualized kernel on bare hardware [ 0.119693] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.123483] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1 [ 0.123855] percpu: Embedded 58 pages/cpu s199136 r8192 d30240 u262144 [ 0.123877] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f root=UUID=5baff67d-813b-4ad8-bba1-8331ae7ea36f ro crashkernel=auto console=ttyS1,115200 [ 0.123926] Unknown kernel command line parameters "BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f", will be passed to user space. [ 0.123959] random: crng init done [ 0.127517] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear) [ 0.129284] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear) [ 0.129386] Fallback order for Node 0: 0 [ 0.129389] Built 1 zonelists, mobility grouping on. Total pages: 8228746 [ 0.129391] Policy zone: Normal [ 0.129397] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.129407] software IO TLB: area num 8. [ 0.217420] Memory: 32698136K/33438012K available (14336K kernel code, 2776K rwdata, 4868K rodata, 2232K init, 13068K bss, 739620K reserved, 0K cma-reserved) [ 0.217499] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 [ 0.217528] Kernel/User page tables isolation: enabled [ 0.217583] ftrace: allocating 43423 entries in 170 pages [ 0.223764] ftrace: allocated 170 pages with 4 groups [ 0.224696] Dynamic Preempt: voluntary [ 0.224776] Running RCU self tests [ 0.224777] Running RCU synchronous self tests [ 0.224783] rcu: Preemptible hierarchical RCU implementation. [ 0.224784] rcu: RCU lockdep checking is enabled. [ 0.224784] Trampoline variant of Tasks RCU enabled. [ 0.224785] Rude variant of Tasks RCU enabled. [ 0.224786] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.224811] Running RCU synchronous self tests [ 0.227400] NR_IRQS: 4352, nr_irqs: 896, preallocated irqs: 16 [ 0.227603] rcu: srcu_init: Setting srcu_struct sizes based on contention. [ 0.232936] Console: colour VGA+ 80x25 [ 0.232966] printk: console [ttyS1] enabled [ 1.235601] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 1.243334] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 1.247424] ... MAX_LOCK_DEPTH: 48 [ 1.251601] ... MAX_LOCKDEP_KEYS: 8192 [ 1.255952] ... CLASSHASH_SIZE: 4096 [ 1.260303] ... MAX_LOCKDEP_ENTRIES: 32768 [ 1.264743] ... MAX_LOCKDEP_CHAINS: 65536 [ 1.269180] ... CHAINHASH_SIZE: 32768 [ 1.273617] memory used by lock dependency info: 6493 kB [ 1.279009] memory used for stack traces: 4224 kB [ 1.283793] per task-struct memory footprint: 1920 bytes [ 1.289226] ACPI: Core revision 20230628 [ 1.293398] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns [ 1.302540] APIC: Switch to symmetric I/O mode setup [ 1.307580] DMAR: Host address width 46 [ 1.311415] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1 [ 1.316736] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de [ 1.324738] DMAR: RMRR base: 0x0000007ba6e000 end: 0x0000007ba7dfff [ 1.331004] DMAR: ATSR flags: 0x0 [ 1.334322] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0 [ 1.340591] DMAR-IR: IOAPIC id 1 under DRHD base 0xfbffc000 IOMMU 0 [ 1.346942] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 0 [ 1.353296] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000 [ 1.358695] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit. [ 1.358696] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting. [ 1.373743] DMAR-IR: Enabled IRQ remapping in xapic mode [ 1.379048] x2apic: IRQ remapping doesn't support X2APIC mode [ 1.385380] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 1.395541] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x32724efed25, max_idle_ns: 440795209572 ns [ 1.406060] Calibrating delay loop (skipped), value calculated using timer frequency.. 6999.45 BogoMIPS (lpj=3499727) [ 1.407070] CPU0: Thermal monitoring enabled (TM1) [ 1.408115] process: using mwait in idle threads [ 1.409056] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8 [ 1.410051] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4 [ 1.411053] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 1.412052] Spectre V2 : Kernel not compiled with retpoline; no mitigation available! [ 1.412052] Spectre V2 : Vulnerable [ 1.414051] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 1.415051] Speculative Store Bypass: Vulnerable [ 1.416060] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 1.417051] TAA: Vulnerable: Clear CPU buffers attempted, no microcode [ 1.418051] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode [ 1.419058] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 1.420051] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 1.421051] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 1.422051] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 1.423051] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 1.427620] Freeing SMP alternatives memory: 36K [ 1.428052] pid_max: default: 32768 minimum: 301 [ 1.429079] LSM: initializing lsm=capability,selinux [ 1.430055] SELinux: Initializing. [ 1.431149] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) [ 1.432090] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) [ 1.434649] Running RCU synchronous self tests [ 1.435054] Running RCU synchronous self tests [ 1.538678] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz (family: 0x6, model: 0x4f, stepping: 0x1) [ 1.540385] RCU Tasks: Setting shift to 3 and lim to 1 rcu_task_cb_adjust=1. [ 1.541086] RCU Tasks Rude: Setting shift to 3 and lim to 1 rcu_task_cb_adjust=1. [ 1.542086] Running RCU-tasks wait API self tests [ 1.649266] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver. [ 1.650057] ... version: 3 [ 1.651051] ... bit width: 48 [ 1.652051] ... generic registers: 4 [ 1.653051] ... value mask: 0000ffffffffffff [ 1.654051] ... max period: 00007fffffffffff [ 1.655051] ... fixed-purpose events: 3 [ 1.656051] ... event mask: 000000070000000f [ 1.657220] signal: max sigframe size: 1776 [ 1.658074] Estimated ratio of average max frequency by base frequency (times 1024): 1053 [ 1.659116] rcu: Hierarchical SRCU implementation. [ 1.660052] rcu: Max phase no-delay instances is 400. [ 1.662105] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter. [ 1.663234] smp: Bringing up secondary CPUs ... [ 1.665093] smpboot: x86: Booting SMP configuration: [ 1.666055] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 [ 1.669335] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 1.671052] TAA CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html for more details. [ 1.672056] MMIO Stale Data CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/processor_mmio_stale_data.html for more details. [ 1.673140] smp: Brought up 1 node, 8 CPUs [ 1.674055] smpboot: Max logical packages: 1 [ 1.675055] smpboot: Total of 8 processors activated (55995.63 BogoMIPS) [ 1.680222] devtmpfs: initialized [ 1.681221] x86/mm: Memory block size: 128MB [ 1.691630] ACPI: PM: Registering ACPI NVS region [mem 0x798a8000-0x79d44fff] (4837376 bytes) [ 1.692480] Running RCU synchronous self tests [ 1.693060] Running RCU synchronous self tests [ 1.694087] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 1.695055] futex hash table entries: 2048 (order: 6, 262144 bytes, linear) [ 1.696244] pinctrl core: initialized pinctrl subsystem [ 1.697205] PM: RTC time: 08:16:49, date: 2023-11-06 [ 1.699440] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 1.700503] audit: initializing netlink subsys (disabled) [ 1.701077] audit: type=2000 audit(1699258607.316:1): state=initialized audit_enabled=0 res=1 [ 1.701293] thermal_sys: Registered thermal governor 'fair_share' [ 1.702054] thermal_sys: Registered thermal governor 'bang_bang' [ 1.703052] thermal_sys: Registered thermal governor 'step_wise' [ 1.704052] thermal_sys: Registered thermal governor 'user_space' [ 1.705082] cpuidle: using governor menu [ 1.708068] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.709052] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.710227] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.711053] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry [ 1.712061] PCI: Using configuration type 1 for base access [ 1.714128] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible. [ 1.716131] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages [ 1.717053] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page [ 1.718052] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages [ 1.719051] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page [ 1.721119] cryptd: max_cpu_qlen set to 1000 [ 1.722234] ACPI: Added _OSI(Module Device) [ 1.723053] ACPI: Added _OSI(Processor Device) [ 1.724052] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.725053] ACPI: Added _OSI(Processor Aggregator Device) [ 1.753291] Callback from call_rcu_tasks_rude() invoked. [ 1.805244] ACPI: 4 ACPI AML tables successfully acquired and loaded [ 1.816608] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 1.826286] ACPI: Dynamic OEM Table Load: [ 1.857076] Callback from call_rcu_tasks() invoked. [ 1.876715] ACPI: _OSC evaluated successfully for all CPUs [ 1.878315] ACPI: Interpreter enabled [ 1.879091] ACPI: PM: (supports S0 S4 S5) [ 1.880056] ACPI: Using IOAPIC for interrupt routing [ 1.881091] HEST: Enabling Firmware First mode for corrected errors. [ 1.882190] HEST: Table parsing has been initialized. [ 1.883330] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 1.884054] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.885051] PCI: Using E820 reservations for host bridge windows [ 1.888132] ACPI: Enabled 5 GPEs in block 00 to 3F [ 1.988153] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus ff]) [ 1.994055] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 2.009264] acpi PNP0A03:03: _OSC: platform does not support [AER LTR] [ 2.017206] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.025052] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.033204] PCI host bridge to bus 0000:ff [ 2.037056] pci_bus 0000:ff: root bus resource [bus ff] [ 2.042090] pci 0000:ff:0b.0: [8086:6f81] type 00 class 0x088000 [ 2.048193] pci 0000:ff:0b.1: [8086:6f36] type 00 class 0x110100 [ 2.055148] pci 0000:ff:0b.2: [8086:6f37] type 00 class 0x110100 [ 2.061143] pci 0000:ff:0b.3: [8086:6f76] type 00 class 0x088000 [ 2.067147] pci 0000:ff:0c.0: [8086:6fe0] type 00 class 0x088000 [ 2.073104] pci 0000:ff:0c.1: [8086:6fe1] type 00 class 0x088000 [ 2.079145] pci 0000:ff:0c.2: [8086:6fe2] type 00 class 0x088000 [ 2.085143] pci 0000:ff:0c.3: [8086:6fe3] type 00 class 0x088000 [ 2.091145] pci 0000:ff:0f.0: [8086:6ff8] type 00 class 0x088000 [ 2.097143] pci 0000:ff:0f.1: [8086:6ff9] type 00 class 0x088000 [ 2.103144] pci 0000:ff:0f.4: [8086:6ffc] type 00 class 0x088000 [ 2.109141] pci 0000:ff:0f.5: [8086:6ffd] type 00 class 0x088000 [ 2.115147] pci 0000:ff:0f.6: [8086:6ffe] type 00 class 0x088000 [ 2.122142] pci 0000:ff:10.0: [8086:6f1d] type 00 class 0x088000 [ 2.128142] pci 0000:ff:10.1: [8086:6f34] type 00 class 0x110100 [ 2.134146] pci 0000:ff:10.5: [8086:6f1e] type 00 class 0x088000 [ 2.140136] pci 0000:ff:10.6: [8086:6f7d] type 00 class 0x110100 [ 2.146142] pci 0000:ff:10.7: [8086:6f1f] type 00 class 0x088000 [ 2.152143] pci 0000:ff:12.0: [8086:6fa0] type 00 class 0x088000 [ 2.158130] pci 0000:ff:12.1: [8086:6f30] type 00 class 0x110100 [ 2.164152] pci 0000:ff:13.0: [8086:6fa8] type 00 class 0x088000 [ 2.170251] pci 0000:ff:13.1: [8086:6f71] type 00 class 0x088000 [ 2.177162] pci 0000:ff:13.2: [8086:6faa] type 00 class 0x088000 [ 2.183165] pci 0000:ff:13.3: [8086:6fab] type 00 class 0x088000 [ 2.189055] pci 0000:ff:13.4: [8086:6fac] type 00 class 0x088000 [ 2.195163] pci 0000:ff:13.5: [8086:6fad] type 00 class 0x088000 [ 2.201161] pci 0000:ff:13.6: [8086:6fae] type 00 class 0x088000 [ 2.207143] pci 0000:ff:13.7: [8086:6faf] type 00 class 0x088000 [ 2.213147] pci 0000:ff:14.0: [8086:6fb0] type 00 class 0x088000 [ 2.219173] pci 0000:ff:14.1: [8086:6fb1] type 00 class 0x088000 [ 2.225165] pci 0000:ff:14.2: [8086:6fb2] type 00 class 0x088000 [ 2.232164] pci 0000:ff:14.3: [8086:6fb3] type 00 class 0x088000 [ 2.238163] pci 0000:ff:14.4: [8086:6fbc] type 00 class 0x088000 [ 2.244145] pci 0000:ff:14.5: [8086:6fbd] type 00 class 0x088000 [ 2.250113] pci 0000:ff:14.6: [8086:6fbe] type 00 class 0x088000 [ 2.256144] pci 0000:ff:14.7: [8086:6fbf] type 00 class 0x088000 [ 2.262147] pci 0000:ff:15.0: [8086:6fb4] type 00 class 0x088000 [ 2.268164] pci 0000:ff:15.1: [8086:6fb5] type 00 class 0x088000 [ 2.274164] pci 0000:ff:15.2: [8086:6fb6] type 00 class 0x088000 [ 2.280164] pci 0000:ff:15.3: [8086:6fb7] type 00 class 0x088000 [ 2.286170] pci 0000:ff:16.0: [8086:6f68] type 00 class 0x088000 [ 2.293151] pci 0000:ff:16.6: [8086:6f6e] type 00 class 0x088000 [ 2.299145] pci 0000:ff:16.7: [8086:6f6f] type 00 class 0x088000 [ 2.305145] pci 0000:ff:17.0: [8086:6fd0] type 00 class 0x088000 [ 2.311139] pci 0000:ff:17.4: [8086:6fb8] type 00 class 0x088000 [ 2.317146] pci 0000:ff:17.5: [8086:6fb9] type 00 class 0x088000 [ 2.323143] pci 0000:ff:17.6: [8086:6fba] type 00 class 0x088000 [ 2.329144] pci 0000:ff:17.7: [8086:6fbb] type 00 class 0x088000 [ 2.335150] pci 0000:ff:1e.0: [8086:6f98] type 00 class 0x088000 [ 2.341145] pci 0000:ff:1e.1: [8086:6f99] type 00 class 0x088000 [ 2.347146] pci 0000:ff:1e.2: [8086:6f9a] type 00 class 0x088000 [ 2.354145] pci 0000:ff:1e.3: [8086:6fc0] type 00 class 0x088000 [ 2.360130] pci 0000:ff:1e.4: [8086:6f9c] type 00 class 0x088000 [ 2.366146] pci 0000:ff:1f.0: [8086:6f88] type 00 class 0x088000 [ 2.372072] pci 0000:ff:1f.2: [8086:6f8a] type 00 class 0x088000 [ 2.386955] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe]) [ 2.393056] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 2.402422] acpi PNP0A08:00: _OSC: platform does not support [AER LTR] [ 2.409766] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.417052] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.427263] PCI host bridge to bus 0000:00 [ 2.431052] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 2.438052] pci_bus 0000:00: root bus resource [io 0x1000-0xffff window] [ 2.445054] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.452052] pci_bus 0000:00: root bus resource [mem 0xfedb0000-0xfedb000f window] [ 2.460052] pci_bus 0000:00: root bus resource [mem 0xfedc0000-0xfedc000f window] [ 2.467052] pci_bus 0000:00: root bus resource [mem 0x90000000-0xfbffbfff window] [ 2.475052] pci_bus 0000:00: root bus resource [bus 00-fe] [ 2.480101] pci 0000:00:00.0: [8086:6f00] type 00 class 0x060000 [ 2.486300] pci 0000:00:01.0: [8086:6f02] type 01 class 0x060400 [ 2.492109] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 2.499052] pci 0000:00:02.0: [8086:6f04] type 01 class 0x060400 [ 2.505109] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 2.511337] pci 0000:00:03.0: [8086:6f08] type 01 class 0x060400 [ 2.517109] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 2.523347] pci 0000:00:04.0: [8086:6f20] type 00 class 0x088000 [ 2.530061] pci 0000:00:04.0: reg 0x10: [mem 0xfb42c000-0xfb42ffff 64bit] [ 2.537265] pci 0000:00:04.1: [8086:6f21] type 00 class 0x088000 [ 2.543060] pci 0000:00:04.1: reg 0x10: [mem 0xfb428000-0xfb42bfff 64bit] [ 2.550264] pci 0000:00:04.2: [8086:6f22] type 00 class 0x088000 [ 2.556060] pci 0000:00:04.2: reg 0x10: [mem 0xfb424000-0xfb427fff 64bit] [ 2.563262] pci 0000:00:04.3: [8086:6f23] type 00 class 0x088000 [ 2.569060] pci 0000:00:04.3: reg 0x10: [mem 0xfb420000-0xfb423fff 64bit] [ 2.576265] pci 0000:00:04.4: [8086:6f24] type 00 class 0x088000 [ 2.582060] pci 0000:00:04.4: reg 0x10: [mem 0xfb41c000-0xfb41ffff 64bit] [ 2.589264] pci 0000:00:04.5: [8086:6f25] type 00 class 0x088000 [ 2.595060] pci 0000:00:04.5: reg 0x10: [mem 0xfb418000-0xfb41bfff 64bit] [ 2.602261] pci 0000:00:04.6: [8086:6f26] type 00 class 0x088000 [ 2.608060] pci 0000:00:04.6: reg 0x10: [mem 0xfb414000-0xfb417fff 64bit] [ 2.615267] pci 0000:00:04.7: [8086:6f27] type 00 class 0x088000 [ 2.621060] pci 0000:00:04.7: reg 0x10: [mem 0xfb410000-0xfb413fff 64bit] [ 2.628256] pci 0000:00:05.0: [8086:6f28] type 00 class 0x088000 [ 2.634102] pci 0000:00:05.1: [8086:6f29] type 00 class 0x088000 [ 2.640270] pci 0000:00:05.2: [8086:6f2a] type 00 class 0x088000 [ 2.646186] pci 0000:00:05.4: [8086:6f2c] type 00 class 0x080020 [ 2.652059] pci 0000:00:05.4: reg 0x10: [mem 0xfb439000-0xfb439fff] [ 2.659215] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000 [ 2.665206] pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601 [ 2.671064] pci 0000:00:11.4: reg 0x10: [io 0xf110-0xf117] [ 2.677058] pci 0000:00:11.4: reg 0x14: [io 0xf100-0xf103] [ 2.682058] pci 0000:00:11.4: reg 0x18: [io 0xf0f0-0xf0f7] [ 2.688058] pci 0000:00:11.4: reg 0x1c: [io 0xf0e0-0xf0e3] [ 2.693058] pci 0000:00:11.4: reg 0x20: [io 0xf020-0xf03f] [ 2.699058] pci 0000:00:11.4: reg 0x24: [mem 0xfb438000-0xfb4387ff] [ 2.705095] pci 0000:00:11.4: PME# supported from D3hot [ 2.710259] pci 0000:00:14.0: [8086:8d31] type 00 class 0x0c0330 [ 2.717067] pci 0000:00:14.0: reg 0x10: [mem 0xfb400000-0xfb40ffff 64bit] [ 2.723111] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 2.729326] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000 [ 2.736067] pci 0000:00:16.0: reg 0x10: [mem 0xfb437000-0xfb43700f 64bit] [ 2.742112] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 2.749235] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000 [ 2.755067] pci 0000:00:16.1: reg 0x10: [mem 0xfb436000-0xfb43600f 64bit] [ 2.762112] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 2.768250] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320 [ 2.774051] pci 0000:00:1a.0: reg 0x10: [mem 0xfb434000-0xfb4343ff] [ 2.780134] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 2.786264] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400 [ 2.793131] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 2.799121] pci 0000:00:1c.2: [8086:8d14] type 01 class 0x060400 [ 2.805133] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold [ 2.811294] pci 0000:00:1c.4: [8086:8d18] type 01 class 0x060400 [ 2.817133] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold [ 2.824303] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320 [ 2.830066] pci 0000:00:1d.0: reg 0x10: [mem 0xfb433000-0xfb4333ff] [ 2.836136] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 2.842278] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100 [ 2.848360] pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601 [ 2.855062] pci 0000:00:1f.2: reg 0x10: [io 0xf070-0xf077] [ 2.860057] pci 0000:00:1f.2: reg 0x14: [io 0xf060-0xf063] [ 2.866057] pci 0000:00:1f.2: reg 0x18: [io 0xf050-0xf057] [ 2.872057] pci 0000:00:1f.2: reg 0x1c: [io 0xf040-0xf043] [ 2.877057] pci 0000:00:1f.2: reg 0x20: [io 0xf000-0xf01f] [ 2.883057] pci 0000:00:1f.2: reg 0x24: [mem 0xfb432000-0xfb4327ff] [ 2.889088] pci 0000:00:1f.2: PME# supported from D3hot [ 2.894238] pci 0000:00:1f.3: [8086:8d22] type 00 class 0x0c0500 [ 2.900068] pci 0000:00:1f.3: reg 0x10: [mem 0xfb431000-0xfb4310ff 64bit] [ 2.907070] pci 0000:00:1f.3: reg 0x20: [io 0x0580-0x059f] [ 2.913466] acpiphp: Slot [1] registered [ 2.917098] pci 0000:00:01.0: PCI bridge to [bus 01] [ 2.922418] pci 0000:02:00.0: [8086:0953] type 00 class 0x010802 [ 2.929062] pci 0000:02:00.0: reg 0x10: [mem 0xfb310000-0xfb313fff 64bit] [ 2.935070] pci 0000:02:00.0: reg 0x30: [mem 0xfb300000-0xfb30ffff pref] [ 2.942055] pci 0000:02:00.0: enabling Extended Tags [ 2.947311] pci 0000:00:02.0: PCI bridge to [bus 02] [ 2.952054] pci 0000:00:02.0: bridge window [mem 0xfb300000-0xfb3fffff] [ 2.959432] pci 0000:03:00.0: [8086:10fb] type 00 class 0x020000 [ 2.965065] pci 0000:03:00.0: reg 0x10: [mem 0xfbd80000-0xfbdfffff 64bit pref] [ 2.973056] pci 0000:03:00.0: reg 0x18: [io 0xe020-0xe03f] [ 2.978063] pci 0000:03:00.0: reg 0x20: [mem 0xfbe04000-0xfbe07fff 64bit pref] [ 2.985056] pci 0000:03:00.0: reg 0x30: [mem 0xfb280000-0xfb2fffff pref] [ 2.992101] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold [ 2.998071] pci 0000:03:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 3.006051] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs) [ 3.016060] pci 0000:03:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 3.023051] pci 0000:03:00.0: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs) [ 3.033364] pci 0000:03:00.1: [8086:10fb] type 00 class 0x020000 [ 3.040063] pci 0000:03:00.1: reg 0x10: [mem 0xfbd00000-0xfbd7ffff 64bit pref] [ 3.047056] pci 0000:03:00.1: reg 0x18: [io 0xe000-0xe01f] [ 3.053062] pci 0000:03:00.1: reg 0x20: [mem 0xfbe00000-0xfbe03fff 64bit pref] [ 3.060056] pci 0000:03:00.1: reg 0x30: [mem 0xfb200000-0xfb27ffff pref] [ 3.067104] pci 0000:03:00.1: PME# supported from D0 D3hot D3cold [ 3.073068] pci 0000:03:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 3.080051] pci 0000:03:00.1: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR0 for 64 VFs) [ 3.090060] pci 0000:03:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 3.098053] pci 0000:03:00.1: VF(n) BAR3 space: [mem 0x00000000-0x000fffff 64bit pref] (contains BAR3 for 64 VFs) [ 3.108251] pci 0000:00:03.0: PCI bridge to [bus 03] [ 3.113053] pci 0000:00:03.0: bridge window [io 0xe000-0xefff] [ 3.119052] pci 0000:00:03.0: bridge window [mem 0xfb200000-0xfb2fffff] [ 3.126053] pci 0000:00:03.0: bridge window [mem 0xfbd00000-0xfbefffff 64bit pref] [ 3.134117] pci 0000:00:1c.0: PCI bridge to [bus 04] [ 3.139161] pci 0000:05:00.0: [1a03:1150] type 01 class 0x060400 [ 3.145132] pci 0000:05:00.0: enabling Extended Tags [ 3.150098] pci 0000:05:00.0: supports D1 D2 [ 3.154052] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 3.161231] pci 0000:00:1c.2: PCI bridge to [bus 05-06] [ 3.166055] pci 0000:00:1c.2: bridge window [io 0xd000-0xdfff] [ 3.172053] pci 0000:00:1c.2: bridge window [mem 0xfa000000-0xfb0fffff] [ 3.179103] pci_bus 0000:06: extended config space not accessible [ 3.185124] pci 0000:06:00.0: [1a03:2000] type 00 class 0x030000 [ 3.191075] pci 0000:06:00.0: reg 0x10: [mem 0xfa000000-0xfaffffff] [ 3.198064] pci 0000:06:00.0: reg 0x14: [mem 0xfb000000-0xfb01ffff] [ 3.204064] pci 0000:06:00.0: reg 0x18: [io 0xd000-0xd07f] [ 3.210122] pci 0000:06:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [ 3.218097] pci 0000:06:00.0: supports D1 D2 [ 3.222051] pci 0000:06:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 3.229252] pci 0000:05:00.0: PCI bridge to [bus 06] [ 3.234058] pci 0000:05:00.0: bridge window [io 0xd000-0xdfff] [ 3.240055] pci 0000:05:00.0: bridge window [mem 0xfa000000-0xfb0fffff] [ 3.247208] pci 0000:07:00.0: [8086:1521] type 00 class 0x020000 [ 3.253069] pci 0000:07:00.0: reg 0x10: [mem 0xfb120000-0xfb13ffff] [ 3.259070] pci 0000:07:00.0: reg 0x18: [io 0xc020-0xc03f] [ 3.265061] pci 0000:07:00.0: reg 0x1c: [mem 0xfb144000-0xfb147fff] [ 3.271184] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold [ 3.277089] pci 0000:07:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 3.285051] pci 0000:07:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs) [ 3.295075] pci 0000:07:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 3.302051] pci 0000:07:00.0: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs) [ 3.312400] pci 0000:07:00.1: [8086:1521] type 00 class 0x020000 [ 3.319070] pci 0000:07:00.1: reg 0x10: [mem 0xfb100000-0xfb11ffff] [ 3.325070] pci 0000:07:00.1: reg 0x18: [io 0xc000-0xc01f] [ 3.331061] pci 0000:07:00.1: reg 0x1c: [mem 0xfb140000-0xfb143fff] [ 3.337179] pci 0000:07:00.1: PME# supported from D0 D3hot D3cold [ 3.343088] pci 0000:07:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 3.350051] pci 0000:07:00.1: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs) [ 3.361075] pci 0000:07:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 3.368051] pci 0000:07:00.1: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs) [ 3.378379] pci 0000:00:1c.4: PCI bridge to [bus 07] [ 3.383054] pci 0000:00:1c.4: bridge window [io 0xc000-0xcfff] [ 3.389053] pci 0000:00:1c.4: bridge window [mem 0xfb100000-0xfb1fffff] [ 3.396055] pci 0000:00:1c.4: bridge has subordinate 07 but max busn 08 [ 3.405174] ACPI: PCI: Interrupt link LNKA configured for IRQ 11 [ 3.411131] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 3.417128] ACPI: PCI: Interrupt link LNKC configured for IRQ 5 [ 3.423130] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 3.429129] ACPI: PCI: Interrupt link LNKE configured for IRQ 0 [ 3.435052] ACPI: PCI: Interrupt link LNKE disabled [ 3.440127] ACPI: PCI: Interrupt link LNKF configured for IRQ 0 [ 3.446051] ACPI: PCI: Interrupt link LNKF disabled [ 3.451127] ACPI: PCI: Interrupt link LNKG configured for IRQ 0 [ 3.457051] ACPI: PCI: Interrupt link LNKG disabled [ 3.462127] ACPI: PCI: Interrupt link LNKH configured for IRQ 0 [ 3.468052] ACPI: PCI: Interrupt link LNKH disabled [ 3.473339] iommu: Default domain type: Translated [ 3.478052] iommu: DMA domain TLB invalidation policy: lazy mode [ 3.484300] SCSI subsystem initialized [ 3.488144] ACPI: bus type USB registered [ 3.492084] usbcore: registered new interface driver usbfs [ 3.498071] usbcore: registered new interface driver hub [ 3.503071] usbcore: registered new device driver usb [ 3.508115] EDAC MC: Ver: 3.0.0 [ 3.512201] NetLabel: Initializing [ 3.515052] NetLabel: domain hash size = 128 [ 3.520051] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 3.525078] NetLabel: unlabeled traffic allowed by default [ 3.531052] PCI: Using ACPI for IRQ routing [ 3.541587] pci 0000:06:00.0: vgaarb: setting as boot VGA device [ 3.542050] pci 0000:06:00.0: vgaarb: bridge control possible [ 3.542050] pci 0000:06:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none [ 3.562054] vgaarb: loaded [ 3.565124] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 3.571053] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 3.579188] clocksource: Switched to clocksource tsc-early [ 3.585087] VFS: Disk quotas dquot_6.6.0 [ 3.589031] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.596069] pnp: PnP ACPI init [ 3.599897] system 00:01: [io 0x0500-0x057f] has been reserved [ 3.605826] system 00:01: [io 0x0400-0x047f] has been reserved [ 3.611750] system 00:01: [io 0x0580-0x059f] has been reserved [ 3.617668] system 00:01: [io 0x0600-0x061f] has been reserved [ 3.623588] system 00:01: [io 0x0880-0x0883] has been reserved [ 3.629511] system 00:01: [io 0x0800-0x081f] has been reserved [ 3.635428] system 00:01: [mem 0xfed1c000-0xfed3ffff] has been reserved [ 3.642042] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved [ 3.648653] system 00:01: [mem 0xff000000-0xffffffff] has been reserved [ 3.655268] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved [ 3.661882] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved [ 3.668495] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved [ 3.675107] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved [ 3.682106] system 00:02: [io 0x0a00-0x0a0f] has been reserved [ 3.688025] system 00:02: [io 0x0a10-0x0a1f] has been reserved [ 3.693945] system 00:02: [io 0x0a20-0x0a2f] has been reserved [ 3.699862] system 00:02: [io 0x0a30-0x0a3f] has been reserved [ 3.705781] system 00:02: [io 0x0a40-0x0a4f] has been reserved [ 3.713398] pnp: PnP ACPI: found 5 devices [ 3.725706] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 3.734657] NET: Registered PF_INET protocol family [ 3.739755] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 3.749906] tcp_listen_portaddr_hash hash table entries: 16384 (order: 8, 1179648 bytes, linear) [ 3.758939] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 3.766743] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 3.775084] TCP bind hash table entries: 65536 (order: 11, 9437184 bytes, vmalloc hugepage) [ 3.785379] TCP: Hash tables configured (established 262144 bind 65536) [ 3.792180] UDP hash table entries: 16384 (order: 9, 2621440 bytes, linear) [ 3.799782] UDP-Lite hash table entries: 16384 (order: 9, 2621440 bytes, linear) [ 3.807802] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 3.813509] pci 0000:00:1c.4: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 07] add_size 100000 add_align 100000 [ 3.824992] pci 0000:00:1c.4: BAR 15: assigned [mem 0x90000000-0x900fffff 64bit pref] [ 3.832822] pci 0000:00:01.0: PCI bridge to [bus 01] [ 3.837792] pci 0000:00:02.0: PCI bridge to [bus 02] [ 3.842756] pci 0000:00:02.0: bridge window [mem 0xfb300000-0xfb3fffff] [ 3.849552] pci 0000:03:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref] [ 3.857144] pci 0000:03:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref] [ 3.865059] pci 0000:03:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref] [ 3.872708] pci 0000:03:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref] [ 3.880710] pci 0000:03:00.1: BAR 7: no space for [mem size 0x00100000 64bit pref] [ 3.888275] pci 0000:03:00.1: BAR 7: failed to assign [mem size 0x00100000 64bit pref] [ 3.896191] pci 0000:03:00.1: BAR 10: no space for [mem size 0x00100000 64bit pref] [ 3.903842] pci 0000:03:00.1: BAR 10: failed to assign [mem size 0x00100000 64bit pref] [ 3.911846] pci 0000:03:00.0: BAR 7: no space for [mem size 0x00100000 64bit pref] [ 3.919412] pci 0000:03:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit pref] [ 3.927325] pci 0000:03:00.0: BAR 10: no space for [mem size 0x00100000 64bit pref] [ 3.934975] pci 0000:03:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit pref] [ 3.942978] pci 0000:03:00.1: BAR 7: no space for [mem size 0x00100000 64bit pref] [ 3.950542] pci 0000:03:00.1: BAR 7: failed to assign [mem size 0x00100000 64bit pref] [ 3.958458] pci 0000:03:00.1: BAR 10: no space for [mem size 0x00100000 64bit pref] [ 3.966111] pci 0000:03:00.1: BAR 10: failed to assign [mem size 0x00100000 64bit pref] [ 3.974109] pci 0000:00:03.0: PCI bridge to [bus 03] [ 3.979078] pci 0000:00:03.0: bridge window [io 0xe000-0xefff] [ 3.985172] pci 0000:00:03.0: bridge window [mem 0xfb200000-0xfb2fffff] [ 3.991956] pci 0000:00:03.0: bridge window [mem 0xfbd00000-0xfbefffff 64bit pref] [ 3.999699] pci 0000:00:1c.0: PCI bridge to [bus 04] [ 4.004670] pci 0000:05:00.0: PCI bridge to [bus 06] [ 4.009629] pci 0000:05:00.0: bridge window [io 0xd000-0xdfff] [ 4.015726] pci 0000:05:00.0: bridge window [mem 0xfa000000-0xfb0fffff] [ 4.022519] pci 0000:00:1c.2: PCI bridge to [bus 05-06] [ 4.027743] pci 0000:00:1c.2: bridge window [io 0xd000-0xdfff] [ 4.033840] pci 0000:00:1c.2: bridge window [mem 0xfa000000-0xfb0fffff] [ 4.040632] pci 0000:07:00.0: BAR 7: assigned [mem 0x90000000-0x9001ffff 64bit pref] [ 4.048369] pci 0000:07:00.0: BAR 10: assigned [mem 0x90020000-0x9003ffff 64bit pref] [ 4.056193] pci 0000:07:00.1: BAR 7: assigned [mem 0x90040000-0x9005ffff 64bit pref] [ 4.063935] pci 0000:07:00.1: BAR 10: assigned [mem 0x90060000-0x9007ffff 64bit pref] [ 4.071761] pci 0000:00:1c.4: PCI bridge to [bus 07] [ 4.076726] pci 0000:00:1c.4: bridge window [io 0xc000-0xcfff] [ 4.082819] pci 0000:00:1c.4: bridge window [mem 0xfb100000-0xfb1fffff] [ 4.089605] pci 0000:00:1c.4: bridge window [mem 0x90000000-0x900fffff 64bit pref] [ 4.097348] pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc [ 4.106413] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 4.112592] pci_bus 0000:00: resource 5 [io 0x1000-0xffff window] [ 4.118769] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 4.125643] pci_bus 0000:00: resource 7 [mem 0xfedb0000-0xfedb000f window] [ 4.132516] pci_bus 0000:00: resource 8 [mem 0xfedc0000-0xfedc000f window] [ 4.139390] pci_bus 0000:00: resource 9 [mem 0x90000000-0xfbffbfff window] [ 4.146264] pci_bus 0000:02: resource 1 [mem 0xfb300000-0xfb3fffff] [ 4.152529] pci_bus 0000:03: resource 0 [io 0xe000-0xefff] [ 4.158101] pci_bus 0000:03: resource 1 [mem 0xfb200000-0xfb2fffff] [ 4.164369] pci_bus 0000:03: resource 2 [mem 0xfbd00000-0xfbefffff 64bit pref] [ 4.171589] pci_bus 0000:05: resource 0 [io 0xd000-0xdfff] [ 4.177164] pci_bus 0000:05: resource 1 [mem 0xfa000000-0xfb0fffff] [ 4.183430] pci_bus 0000:06: resource 0 [io 0xd000-0xdfff] [ 4.189003] pci_bus 0000:06: resource 1 [mem 0xfa000000-0xfb0fffff] [ 4.195269] pci_bus 0000:07: resource 0 [io 0xc000-0xcfff] [ 4.200841] pci_bus 0000:07: resource 1 [mem 0xfb100000-0xfb1fffff] [ 4.207109] pci_bus 0000:07: resource 2 [mem 0x90000000-0x900fffff 64bit pref] [ 4.214736] pci 0000:00:05.0: disabled boot interrupts on device [8086:6f28] [ 4.238707] pci 0000:00:1a.0: quirk_usb_early_handoff+0x0/0x690 took 16011 usecs [ 4.262751] pci 0000:00:1d.0: quirk_usb_early_handoff+0x0/0x690 took 16239 usecs [ 4.270168] pci 0000:02:00.0: CLS mismatch (64 != 32), using 64 bytes [ 4.276744] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.276861] Trying to unpack rootfs image as initramfs... [ 4.283184] software IO TLB: mapped [mem 0x0000000074ec6000-0x0000000078ec6000] (64MB) [ 4.296593] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 655360 ms ovfl timer [ 4.304506] RAPL PMU: hw unit of domain package 2^-14 Joules [ 4.310165] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 4.319802] Initialise system trusted keyrings [ 4.324359] workingset: timestamp_bits=37 max_order=23 bucket_order=0 [ 4.330809] zbud: loaded [ 4.340144] NET: Registered PF_ALG protocol family [ 4.344964] Key type asymmetric registered [ 4.349074] Asymmetric key parser 'x509' registered [ 4.353977] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.361459] io scheduler mq-deadline registered [ 4.365994] io scheduler kyber registered [ 4.371299] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.379039] pcieport 0000:00:01.0: PME: Signaling with IRQ 25 [ 4.385347] pcieport 0000:00:02.0: PME: Signaling with IRQ 27 [ 4.391735] pcieport 0000:00:03.0: PME: Signaling with IRQ 29 [ 4.397935] pcieport 0000:00:1c.0: PME: Signaling with IRQ 30 [ 4.404105] pcieport 0000:00:1c.2: PME: Signaling with IRQ 31 [ 4.410254] pcieport 0000:00:1c.4: PME: Signaling with IRQ 32 [ 4.416409] ACPI: \_SB_.SCK0.CP00: Found 2 idle states [ 4.423696] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0 [ 4.432130] ACPI: button: Power Button [PWRB] [ 4.436571] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 4.444114] ACPI: button: Power Button [PWRF] [ 4.451117] ERST: Error Record Serialization Table (ERST) support is initialized. [ 4.458651] pstore: Using crash dump compression: deflate [ 4.464073] pstore: Registered erst as persistent store backend [ 4.470196] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.476825] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.484474] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.493201] Non-volatile memory driver v1.3 [ 4.499283] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 4.507373] ahci 0000:00:11.4: flags: 64bit ncq pm led clo pio slum part ems apst [ 4.524566] scsi host0: ahci [ 4.527783] scsi host1: ahci [ 4.530938] scsi host2: ahci [ 4.534096] scsi host3: ahci [ 4.537118] ata1: SATA max UDMA/133 abar m2048@0xfb438000 port 0xfb438100 irq 33 [ 4.544510] ata2: SATA max UDMA/133 abar m2048@0xfb438000 port 0xfb438180 irq 33 [ 4.551902] ata3: SATA max UDMA/133 abar m2048@0xfb438000 port 0xfb438200 irq 33 [ 4.559294] ata4: SATA max UDMA/133 abar m2048@0xfb438000 port 0xfb438280 irq 33 [ 4.567166] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 4.575340] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst [ 4.598897] scsi host4: ahci [ 4.602096] scsi host5: ahci [ 4.605273] scsi host6: ahci [ 4.608456] scsi host7: ahci [ 4.611633] scsi host8: ahci [ 4.614802] scsi host9: ahci [ 4.617814] ata5: SATA max UDMA/133 abar m2048@0xfb432000 port 0xfb432100 irq 34 [ 4.625205] ata6: SATA max UDMA/133 abar m2048@0xfb432000 port 0xfb432180 irq 34 [ 4.632596] ata7: SATA max UDMA/133 abar m2048@0xfb432000 port 0xfb432200 irq 34 [ 4.639991] ata8: SATA max UDMA/133 abar m2048@0xfb432000 port 0xfb432280 irq 34 [ 4.647382] ata9: SATA max UDMA/133 abar m2048@0xfb432000 port 0xfb432300 irq 34 [ 4.654777] ata10: SATA max UDMA/133 abar m2048@0xfb432000 port 0xfb432380 irq 34 [ 4.663874] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 4.669396] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 4.676823] ehci-pci 0000:00:1a.0: debug port 2 [ 4.685360] ehci-pci 0000:00:1a.0: irq 18, io mem 0xfb434000 [ 4.697467] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 4.703464] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.06 [ 4.711734] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.718950] usb usb1: Product: EHCI Host Controller [ 4.723832] usb usb1: Manufacturer: Linux 6.6.0-rc5-g5b73517e0e2f ehci_hcd [ 4.730704] usb usb1: SerialNumber: 0000:00:1a.0 [ 4.735975] hub 1-0:1.0: USB hub found [ 4.739772] hub 1-0:1.0: 2 ports detected [ 4.744493] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 4.749945] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ 4.758451] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000000009810 [ 4.768178] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 4.768207] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 4.773554] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 [ 4.778787] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 4 [ 4.786023] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed [ 4.793434] ehci-pci 0000:00:1d.0: debug port 2 [ 4.799805] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.06 [ 4.808131] ehci-pci 0000:00:1d.0: irq 18, io mem 0xfb433000 [ 4.812468] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.824462] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 4.825343] usb usb2: Product: xHCI Host Controller [ 4.835955] usb usb2: Manufacturer: Linux 6.6.0-rc5-g5b73517e0e2f xhci-hcd [ 4.842884] usb usb2: SerialNumber: 0000:00:14.0 [ 4.848071] hub 2-0:1.0: USB hub found [ 4.851881] hub 2-0:1.0: 15 ports detected [ 4.863019] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.06 [ 4.871284] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.877629] ata2: SATA link down (SStatus 0 SControl 300) [ 4.878505] usb usb3: Product: xHCI Host Controller [ 4.883949] ata4: SATA link down (SStatus 0 SControl 300) [ 4.888799] usb usb3: Manufacturer: Linux 6.6.0-rc5-g5b73517e0e2f xhci-hcd [ 4.894230] ata3: SATA link down (SStatus 0 SControl 300) [ 4.901071] usb usb3: SerialNumber: 0000:00:14.0 [ 4.906492] ata1: SATA link down (SStatus 0 SControl 300) [ 4.911484] hub 3-0:1.0: USB hub found [ 4.920255] hub 3-0:1.0: 6 ports detected [ 4.928038] usb usb4: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.06 [ 4.928055] usbcore: registered new interface driver usbserial_generic [ 4.936308] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.942897] usbserial: USB Serial support registered for generic [ 4.950043] usb usb4: Product: EHCI Host Controller [ 4.950046] usb usb4: Manufacturer: Linux 6.6.0-rc5-g5b73517e0e2f ehci_hcd [ 4.956111] i8042: PNP: No PS/2 controller found. [ 4.960931] usb usb4: SerialNumber: 0000:00:1d.0 [ 4.967995] mousedev: PS/2 mouse device common for all mice [ 4.972846] hub 4-0:1.0: USB hub found [ 4.973711] ata7: SATA link down (SStatus 0 SControl 300) [ 4.973946] ata10: SATA link down (SStatus 0 SControl 300) [ 4.975020] ata6: SATA link down (SStatus 0 SControl 300) [ 4.976951] ata8: SATA link down (SStatus 0 SControl 300) [ 4.977227] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 4.977478] rtc_cmos 00:00: RTC can wake from S4 [ 4.978108] rtc_cmos 00:00: registered as rtc0 [ 4.978173] rtc_cmos 00:00: setting system clock to 2023-11-06T08:16:52 UTC (1699258612) [ 4.978250] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram [ 4.982723] hub 4-0:1.0: 2 ports detected [ 4.986478] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 4.986481] ata9: SATA link down (SStatus 0 SControl 300) [ 5.039739] Freeing initrd memory: 54280K [ 5.058212] device-mapper: uevent: version 1.0.3 [ 5.063044] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com [ 5.071627] intel_pstate: Intel P-state driver initializing [ 5.082241] hid: raw HID events driver (C) Jiri Kosina [ 5.087629] usbcore: registered new interface driver usbhid [ 5.093196] usbhid: USB HID core driver [ 5.097323] drop_monitor: Initializing network drop monitor service [ 5.103703] Initializing XFRM netlink socket [ 5.108015] NET: Registered PF_INET6 protocol family [ 5.113597] Segment Routing with IPv6 [ 5.117326] In-situ OAM (IOAM) with IPv6 [ 5.117336] ata5.00: ATA-9: ST1000NM0033-9ZM173, SN06, max UDMA/133 [ 5.119974] usb 1-1: New USB device found, idVendor=8087, idProduct=800a, bcdDevice= 0.05 [ 5.119984] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.120882] hub 1-1:1.0: USB hub found [ 5.120995]hub 1-1:1.0: 6 ports detected [ 5.121273] mip6: Mobile IPv6 [ 5.130473] usb 2-12: new high-speed USB device number 2 using xhci_hcd [ 5.135737] NET: Registered PF_PACKET protocol family [ 5.142914] ata5.00: NCQ Send/Recv Log not supported [ 5.170268] ata5.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 32), AA [ 5.178006] microcode: Microcode Update Driver: v2.2. [ 5.178012] IPI shorthand broadcast: enabled [ 5.178507] ata5.00: NCQ Send/Recv Log not supported [ 5.183090] AVX2 version of gcm_enc/dec engaged. [ 5.187337] ata5.00: configured for UDMA/133 [ 5.201338] AES CTR mode by8 optimization enabled [ 5.202215] scsi 4:0:0:0: Direct-Access ATA ST1000NM0033-9ZM SN06 PQ: 0 ANSI: 5 [ 5.210785] sched_clock: Marking stable (4053000974, 1157416069)->(5826640585, -616223542) [ 5.216029] sd 4:0:0:0: Attached scsi generic sg0 type 0 [ 5.216068] sd 4:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB) [ 5.216104] sd 4:0:0:0: [sda] Write Protect is off [ 5.216153] sd 4:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 5.216208] sd 4:0:0:0: [sda] Preferred minimum I/O size 512 bytes [ 5.228040] registered taskstats version 1 [ 5.234454] usb 4-1: new high-speed USB device number 2 using ehci-pci [ 5.235574] sda: sda1 [ 5.240292] Loading compiled-in X.509 certificates [ 5.272696] usb 2-12: New USB device found, idVendor=0557, idProduct=7000, bcdDevice= 0.00 [ 5.273539] sd 4:0:0:0: [sda] Attached SCSI disk [ 5.280501] Loaded X.509 cert 'Build time autogenerated kernel key: 9c64294c59e505ad76846e5cff1106b988963201' [ 5.281570] usb 2-12: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.287848] debug_vm_pgtable: [debug_vm_pgtable ]: Validating architecture page table helpers [ 5.297544] hub 2-12:1.0: USB hub found [ 5.316543] hub 2-12:1.0: 4 ports detected [ 5.320779] tsc: Refined TSC clocksource calibration: 3499.996 MHz [ 5.326998] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x32734d20948, max_idle_ns: 440795306379 ns [ 5.422534] Key type .fscrypt registered [ 5.422547] clocksource: Switched to clocksource tsc [ 5.426463] Key type fscrypt-provisioning registered [ 5.439737] usb 4-1: New USB device found, idVendor=8087, idProduct=8002, bcdDevice= 0.05 [ 5.447925] usb 4-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.455614] hub 4-1:1.0: USB hub found [ 5.459482] hub 4-1:1.0: 8 ports detected [ 5.479804] PM: Magic number: 15:48:270 [ 5.483893] acpi device:1da: hash matches [ 5.488519] clk: Disabling unused clocks [ 5.495128] Freeing unused kernel image (initmem) memory: 2232K [ 5.511535] Write protecting the kernel read-only data: 20480k [ 5.518066] Freeing unused kernel image (rodata/data gap) memory: 1276K [ 5.524758] rodata_test: all tests were successful [ 5.529550] Run /init as init process [ 5.562245] systemd[1]: systemd 239 (239-68.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.585021] systemd[1]: Detected architecture x86-64. [ 5.590096] systemd[1]: Running in initial RAM disk. Welcome to Red Hat Enterprise Linux 8.4 (Ootpa) dracut-049-209.git20220815.el8 (Initramfs)! [ 5.625642] systemd[1]: Set hostname to . [ 5.728248] systemd[1]: Listening on udev Control Socket. [ 5.733454] usb 2-12.1: new low-speed USB device number 3 using xhci_hcd [ OK Control Socket. 5.747621] sysSocket. [ OK ] Listening on udev Kernel Socket. [ 5.759529] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.769580] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.782167] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.808874] systemd[1]: Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Startin[ 5.849788] usb 2-12.1: New USB device found, idVendor=0557, idProduct=2419, bcdDevice= 1.00 g Load Kernel Modules... [ 5.858487] usb 2-12.1: New USB device stringduct=0, SerialNumber=0 [ 5.869769] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb2/2-12/2-12.1/2-12.1:1.0/0003:0557:2419.0001/input/input2 Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ 5.910103] fuse: init (API version 7.39) Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ 5.944396] hid-generic 0003:0557:2419.0001: input,hidraw0: USB HID v1.00 Keyboard [HID 0557:2419] on usb-0000:00:14.0-12.1/input0 [ OK ] Reached targ[ 5.958897] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb2/2-12/2-12.1/2-12.1:1.1/0003:0557:2419.0002/input/input3 et Timers. [ 5.973983] hid-generic 0003:0557:2419.0002: input,hidraw1: USB HID v1.00 Mouse [HID 0557:2419] on usb-0000:00:14.0-12.1/input1 [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Load Kernel Modules. [ OK ] Started Create Volatile Files and Directories. Starting Apply Kernel Variables... Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. Starting Show Plymouth Boot Screen... Starting dracut initqueue hook... [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Started Forward Password Requests to Plymouth Directory Watch. [ OK ] Reached target Basic System. [ 7.129496] dca service started, version 1.12.1 [ 7.142828] pps_core: LinuxPPS API ver. 1 registered [ 7.147792] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 7.173657] PTP clock support registered [ 7.181512] nvme nvme0: pci function 0000:02:00.0 [ 7.235591] nvme nvme0: 8/0/0 default/read/poll queues [ 7.346195] igb: Intel(R) Gigabit Ethernet Network Driver [ 7.351598] igb: Copyright (c) 2007-2014 Intel Corporation. [ 7.422005] igb 0000:07:00.0: added PHC on eth0 [ 7.426588] igb 0000:07:00.0: Intel(R) Gigabit Ethernet Network Connection [ 7.433466] igb 0000:07:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 0c:c4:7a:d9:38:2c [ 7.440692] igb 0000:07:00.0: eth0: PBA No: 050B00-000 [ 7.445842] igb 0000:07:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ OK [[ 7.456023] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver 0m] Found device[ 7.463044] ixgbe: Copyright (c) 1999-2016 Intel Corporation. ST1000NM0033-9ZM173 1. [ OK ] Reached target Initrd Root Device. [ 7.511738] igb 0000:07:00.1: added PHC on eth1 [ 7.516292] igb 0000:07:00.1: Intel(R) Gigabit Ethernet Network Connection [ 7.523172] igb 0000:07:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 0c:c4:7a:d9:38:2d [ 7.530382] igb 0000:07:00.1: eth1: PBA No: 050B00-000 [ 7.535519] igb 0000:07:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 7.547301] igb 0000:07:00.0 eno1: renamed from eth0 [ 7.569947] igb 0000:07:00.1 eno2: renamed from eth1 [ 8.606872] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0 [ 8.617179] ixgbe 0000:03:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link) [ 8.625709] ixgbe 0000:03:00.0: MAC: 2, PHY: 1, PBA No: 0210FF-0FF [ 8.631898] ixgbe 0000:03:00.0: 0c:c4:7a:88:6d:f4 [ 8.643541] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection [ 8.802850] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0 [ 8.813158] ixgbe 0000:03:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link) [ 8.821693] ixgbe 0000:03:00.1: MAC: 2, PHY: 14, SFP+: 4, PBA No: 0210FF-0FF [ 8.828746] ixgbe 0000:03:00.1: 0c:c4:7a:88:6d:f5 [ 8.840484] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection [ 8.852230] ixgbe 0000:03:00.0 enp3s0f0: renamed from eth0 [ 8.863680] ixgbe 0000:03:00.1 enp3s0f1: renamed from eth1 [ OK ] Started dracut initqueue hook. Starting File System Check on /dev/…67d-813b-4ad8-bba1-8331ae7ea36f... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/d…ff67d-813b-4ad8-bba1-8331ae7ea36f. Mounting /sysroot... [ 9.114363] EXT4-fs (sda1): mounted filesystem 5baff67d-813b-4ad8-bba1-8331ae7ea36f ro with ordered data mode. Quota mode: none. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 9.151601] systemd[1]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). Starting Setup Virtual Console... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. Starting Plymouth switch root service... [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Load Kernel Modules. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleanup udevd DB. [ OK ] Started Setup Virtual Console. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 9.888524] printk: systemd: 20 output lines suppressed due to ratelimiting [ 10.616693] SELinux: Permission watch in class filesystem not defined in policy. [ 10.624211] SELinux: Permission watch in class file not defined in policy. [ 10.631168] SELinux: Permission watch_mount in class file not defined in policy. [ 10.638639] SELinux: Permission watch_sb in class file not defined in policy. [ 10.645852] SELinux: Permission watch_with_perm in class file not defined in policy. [ 10.653675] SELinux: Permission watch_reads in class file not defined in policy. [ 10.661147] SELinux: Permission watch in class dir not defined in policy. [ 10.668023] SELinux: Permission watch_mount in class dir not defined in policy. [ 10.675411] SELinux: Permission watch_sb in class dir not defined in policy. [ 10.682536] SELinux: Permission watch_with_perm in class dir not defined in policy. [ 10.690268] SELinux: Permission watch_reads in class dir not defined in policy. [ 10.697663] SELinux: Permission watch in class lnk_file not defined in policy. [ 10.704966] SELinux: Permission watch_mount in class lnk_file not defined in policy. [ 10.712782] SELinux: Permission watch_sb in class lnk_file not defined in policy. [ 10.720340] SELinux: Permission watch_with_perm in class lnk_file not defined in policy. [ 10.728504] SELinux: Permission watch_reads in class lnk_file not defined in policy. [ 10.736326] SELinux: Permission watch in class chr_file not defined in policy. [ 10.743630] SELinux: Permission watch_mount in class chr_file not defined in policy. [ 10.751454] SELinux: Permission watch_sb in class chr_file not defined in policy. [ 10.759011] SELinux: Permission watch_with_perm in class chr_file not defined in policy. [ 10.767175] SELinux: Permission watch_reads in class chr_file not defined in policy. [ 10.774996] SELinux: Permission watch in class blk_file not defined in policy. [ 10.782300] SELinux: Permission watch_mount in class blk_file not defined in policy. [ 10.790118] SELinux: Permission watch_sb in class blk_file not defined in policy. [ 10.797684] SELinux: Permission watch_with_perm in class blk_file not defined in policy. [ 10.805848] SELinux: Permission watch_reads in class blk_file not defined in policy. [ 10.813666] SELinux: Permission watch in class sock_file not defined in policy. [ 10.821049] SELinux: Permission watch_mount in class sock_file not defined in policy. [ 10.828953] SELinux: Permission watch_sb in class sock_file not defined in policy. [ 10.836597] SELinux: Permission watch_with_perm in class sock_file not defined in policy. [ 10.844849] SELinux: Permission watch_reads in class sock_file not defined in policy. [ 10.852762] SELinux: Permission watch in class fifo_file not defined in policy. [ 10.860153] SELinux: Permission watch_mount in class fifo_file not defined in policy. [ 10.868059] SELinux: Permission watch_sb in class fifo_file not defined in policy. [ 10.875711] SELinux: Permission watch_with_perm in class fifo_file not defined in policy. [ 10.883963] SELinux: Permission watch_reads in class fifo_file not defined in policy. [ 10.891901] SELinux: Permission perfmon in class capability2 not defined in policy. [ 10.899631] SELinux: Permission bpf in class capability2 not defined in policy. [ 10.907015] SELinux: Permission checkpoint_restore in class capability2 not defined in policy. [ 10.915705] SELinux: Permission perfmon in class cap2_userns not defined in policy. [ 10.923439] SELinux: Permission bpf in class cap2_userns not defined in policy. [ 10.930823] SELinux: Permission checkpoint_restore in class cap2_userns not defined in policy. [ 10.939526] SELinux: Class mctp_socket not defined in policy. [ 10.945359] SELinux: Class anon_inode not defined in policy. [ 10.951103] SELinux: Class io_uring not defined in policy. [ 10.956667] SELinux: Class user_namespace not defined in policy. [ 10.962751] SELinux: the above unknown classes and permissions will be allowed [ 10.984219] SELinux: policy capability network_peer_controls=1 [ 10.990144] SELinux: policy capability open_perms=1 [ 10.995107] SELinux: policy capability extended_socket_class=1 [ 11.001023] SELinux: policy capability always_check_network=0 [ 11.006851] SELinux: policy capability cgroup_seclabel=1 [ 11.012247] SELinux: policy capability nnp_nosuid_transition=1 [ 11.018158] SELinux: policy capability genfs_seclabel_symlinks=0 [ 11.024243] SELinux: policy capability ioctl_skip_cloexec=0 [ 11.066681] audit: type=1403 audit(1699258618.588:2): auid=4294967295 ses=4294967295 lsm=selinux res=1 [ 11.069495] systemd[1]: Successfully loaded SELinux policy in 735.625ms. [ 11.162920] systemd[1]: Relabelled /dev, /run and /sys/fs/cgroup in 28.437ms. [ 11.176860] systemd[1]: systemd 239 (239-68.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.199496] systemd[1]: Detected architecture x86-64. Welcome to Red Hat Enterprise Linux 8.4 (Ootpa)! [ 11.212290] systemd[1]: Set hostname to . [ 12.154517] systemd[1]: systemd-journald.service: Succeeded. [ 12.162216] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.169221] systemd[1]: Stopped Switch Root. [ OK [[ 12.174737] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. 0m] Stopped Swit[ 12.186210] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. ch Root. [ 12.196402] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.215013] systemd[1]: Starting Journal Service... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on LVM2 poll daemon socket. Starting Load Kernel Modules... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.305326] EXT4-fs (sda1): re-mounted 5baff67d-813b-4ad8-bba1-8331ae7ea36f r/w. Quota mode: none. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped Plymouth switch root service. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Monitoring of LVM2 mirrors…ng dmeventd or progress polling... [ OK ] Set up automount Arbitrary Executab…rmats File System Automount Point. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Load/Save Random Seed... Starting Flush Journal to Persistent Storage... [ OK ] Started Load/Save Random Seed. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Setup Virtual Console. [ OK ] Started Load Kernel Modules. Mounting FUSE Control File System... Starting Apply Kernel Variables... [ OK ] Mounted FUSE Control File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.095447] ioatdma: Intel(R) QuickData Technology Driver 5.00 [ 13.113081] IPMI message handler: version 39.2 [ 13.120485] igb 0000:07:00.0: DCA enabled [ 13.124605] igb 0000:07:00.1: DCA enabled [ 13.143310] ipmi device interface [ 13.177961] power_meter ACPI000D:00: Found ACPI power meter. [ 13.184319] power_meter ACPI000D:00: Ignoring unsafe software power cap! [ 13.191085] power_meter ACPI000D:00: hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info(). [ 13.205024] ipmi_si: IPMI System Interface driver [ 13.209828] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS [ 13.216197] ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 13.223569] ipmi_si: Adding SMBIOS-specified kcs state machine [ 13.231282] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI [ 13.237513] ipmi_si IPI0001:00: ipmi_platform: [io 0x0ca2] regsize 1 spacing 1 irq 0 [ 13.250736] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI [ 13.259600] ipmi_si: Adding ACPI-specified kcs state machine [ 13.268309] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM event activation on device 259:0... [ 13.355576] ipmi_si IPI0001:00: The BMC does not support clearing the recv irq bit, compensating, but the BMC needs to be fixed. [ 13.397982] ipmi_si IPI0001:00: IPMI message handler: Found new BMC (man_id: 0x002a7c, prod_id: 0x0833, dev_id: 0x20) [ 13.414031] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 13.457528] ipmi_si IPI0001:00: IPMI kcs interface initialized [ 13.471185] ipmi_ssif: IPMI SSIF Interface driver [ 13.575981] iTCO_vendor_support: vendor-support=0 [ OK ] Started LVM [ 13.587317] iTCO_wdt iTCO_wdt.1.auto: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460) event activation[ 13.598441] iTCO_wdt iTCO_wdt.1.auto: initialized. heartbeat=30 sec (nowayout=0) on device 259:0[ 13.607254] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt . [ 13.615660] i2c i2c-2: 2/8 memory slots populated (from DMI) [ 13.621351] i2c i2c-2: Systems with more than 4 memory slots not supported yet, not instantiating SPD Starting LVM event activation on device 253:1... Starting LVM event activation on device 253:3... Starting LVM event activation on device 253:2... Starting LVM event activation on device 253:0... [ OK ] Started LVM event activation on device 253:0. [ OK ] Started LVM event activation on device 253:1. [ OK ] Started LVM event activation on device 253:3. [ OK ] Started LVM event activation on device 253:2. [ OK ] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Import network configuration from initramfs... Starting Tell Plymouth To Write Out Runtime Data... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ 14.411139] EDAC MC0: Giving out device to module sb_edac controller Broadwell SrcID#0_Ha#0: DEV 0000:ff:12.0 (INTERRUPT) [ 14.422125] EDAC sbridge: Ver: 1.1.2 [ 14.462691] intel_rapl_common: Found RAPL domain package [ 14.468076] intel_rapl_common: Found RAPL domain dram [ OK ] Started Create Volatile Files and Directories. Mounting RPC Pipe File System... Starting Security Auditing Service... Starting RPC Bind... [ OK ] Started RPC Bind. [ 14.830711] RPC: Registered named UNIX socket transport module. [ 14.836661] RPC: Registered udp transport module. [ 14.841370] RPC: Registered tcp transport module. [ 14.846073] RPC: Registered tcp-with-tls transport module. [ 14.851557] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Mounted RPC Pipe File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Generate summary of yesterday's process accounting. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Started CUPS Scheduler. [ OK ] Reached target Paths. [ OK ] Listening on SSSD Kerberos Cache Manager responder socket. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Run system activity accounting tool every 10 minutes. [ OK ] Reached target Timers. [ OK ] Listening on CUPS Scheduler. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started libstoragemgmt plug-in server daemon. Starting Avahi mDNS/DNS-SD Stack... [ OK ] Started D-Bus System Message Bus. Starting NTP client/server... Starting VDO volume services... Starting Authorization Manager... Starting Kernel Samepage Merging... Starting Network Manager... Starting System Security Services Daemon... Starting Resets System Activity Logs... Starting Hardware RNG Entropy Gatherer Wake threshold service... Starting Self Monitoring and Reporting Technology (SMART) Daemon... [ OK ] Started Machine Check Exception Logging Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Kernel Samepage Merging. Starting Kernel Samepage Merging (KSM) Tuning Daemon... [ OK ] Started Resets System Activity Logs. [ OK ] Started Hardware RNG Entropy Gatherer Wake threshold service. [ OK ] Started Kernel Samepage Merging (KSM) Tuning Daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Avahi mDNS/DNS-SD Stack. [ OK ] Started NTP client/server. [ OK ] Started Self Monitoring and Reporting Technology (SMART) Daemon. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Enable periodic update of entitlement certificates.... Starting CUPS Scheduler... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Enable periodic update of entitlement certificates.. Starting Hostname Service... [ OK ] Started Hostname Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started OpenSSH server daemon. [ OK ] Started Authorization Manager. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started CUPS Scheduler. [ 18.629938] ixgbe 0000:03:00.0: registered PHC device on enp3s0f0 [ 18.792264] ixgbe 0000:03:00.1: registered PHC device on enp3s0f1 [ 19.017504] ixgbe 0000:03:00.1 enp3s0f1: detected SFP+: 4 [ 19.216537] ixgbe 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ OK ] Started VDO volume services. [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Job spooling tools. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ 21.722965] msr: Write to unrecognized MSR 0x1b0 by x86_energy_perf (pid: 1079). [ 21.730369] msr: See https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/about for details. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Created slice User Slice of UID 1000. Starting User runtime directory /run/user/1000... [ OK ] Started User runtime directory /run/user/1000. Starting User Manager for UID 1000... [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting /etc/rc.d/rc.local Compatibility... [ OK ] Started Nagios Remote Plugin Executor. [ OK ] Started User Manager for UID 1000. [ OK ] Started Session 1 of user ubuntu. + '[' '!' -f /.cephlab_net_configured ']' + set +e + attempts=0 + myips= + '[' '' '!=' '' ']' + '[' 0 -ge 10 ']' ++ ip -4 addr ++ grep -oP '(?<=inet\s)\d+(\.\d+){3}' ++ grep -v '127.0.0.1\|127.0.1.1' [ OK ] Started Notify NFS peers of a restart. [FAILED] Failed to start Crash recovery kernel arming. See 'systemctl status kdump.service' for details. + myips=172.21.15.186 + attempts=1 + sleep 1 [ OK ] Started System Logging Service. + '[' 172.21.15.186 '!=' '' ']' + set -e + '[' -n 172.21.15.186 ']' + for ip in $myips + timeout 1s ping -I 172.21.15.186 -nq -c1 172.21.0.1 ++ dig +short -x 172.21.15.186 @172.21.0.1 ++ sed 's/\.com.*/\.com/g' + newhostname=smithi186.front.sepia.ceph.com + '[' -n smithi186.front.sepia.ceph.com ']' + hostname smithi186.front.sepia.ceph.com ++ hostname -d + newdomain=front.sepia.ceph.com ++ hostname -s + shorthostname=smithi186 + echo smithi186 + grep -q front.sepia.ceph.com /etc/hosts + sed -i 's/.*front.sepia.ceph.com.*/172.21.15.186 smithi186.front.sepia.ceph.com smithi186/g' /etc/hosts + break + command -v zypper + command -v apt-get + '[' -e /.cephlab_rc_local ']' + exit 0 [ OK ] Started /etc/rc.d/rc.local Compatibility. Starting Terminate Plymouth Boot Screen... Starting Hold until boot process finishes up... [ 26.186487] KGDB: Registered I/O driver kgdboc Red Hat Enterprise Linux 8.4 (Ootpa) Kernel 6.6.0-rc5-g5b73517e0e2f on an x86_64 Activate the web console with: systemctl enable --now cockpit.socket smithi186 login: [ 561.794203] SELinux: Permission watch in class filesystem not defined in policy. [ 561.801694] SELinux: Permission watch in class file not defined in policy. [ 561.808653] SELinux: Permission watch_mount in class file not defined in policy. [ 561.816133] SELinux: Permission watch_sb in class file not defined in policy. [ 561.823377] SELinux: Permission watch_with_perm in class file not defined in policy. [ 561.831207] SELinux: Permission watch_reads in class file not defined in policy. [ 561.838691] SELinux: Permission watch in class dir not defined in policy. [ 561.845563] SELinux: Permission watch_mount in class dir not defined in policy. [ 561.852958] SELinux: Permission watch_sb in class dir not defined in policy. [ 561.860089] SELinux: Permission watch_with_perm in class dir not defined in policy. [ 561.867826] SELinux: Permission watch_reads in class dir not defined in policy. [ 561.875224] SELinux: Permission watch in class lnk_file not defined in policy. [ 561.882526] SELinux: Permission watch_mount in class lnk_file not defined in policy. [ 561.890358] SELinux: Permission watch_sb in class lnk_file not defined in policy. [ 561.897926] SELinux: Permission watch_with_perm in class lnk_file not defined in policy. [ 561.906102] SELinux: Permission watch_reads in class lnk_file not defined in policy. [ 561.913928] SELinux: Permission watch in class chr_file not defined in policy. [ 561.921236] SELinux: Permission watch_mount in class chr_file not defined in policy. [ 561.929061] SELinux: Permission watch_sb in class chr_file not defined in policy. [ 561.936626] SELinux: Permission watch_with_perm in class chr_file not defined in policy. [ 561.944796] SELinux: Permission watch_reads in class chr_file not defined in policy. [ 561.952626] SELinux: Permission watch in class blk_file not defined in policy. [ 561.959928] SELinux: Permission watch_mount in class blk_file not defined in policy. [ 561.967756] SELinux: Permission watch_sb in class blk_file not defined in policy. [ 561.975322] SELinux: Permission watch_with_perm in class blk_file not defined in policy. [ 561.983494] SELinux: Permission watch_reads in class blk_file not defined in policy. [ 561.991325] SELinux: Permission watch in class sock_file not defined in policy. [ 561.998716] SELinux: Permission watch_mount in class sock_file not defined in policy. [ 562.006626] SELinux: Permission watch_sb in class sock_file not defined in policy. [ 562.014284] SELinux: Permission watch_with_perm in class sock_file not defined in policy. [ 562.022548] SELinux: Permission watch_reads in class sock_file not defined in policy. [ 562.030465] SELinux: Permission watch in class fifo_file not defined in policy. [ 562.037862] SELinux: Permission watch_mount in class fifo_file not defined in policy. [ 562.045776] SELinux: Permission watch_sb in class fifo_file not defined in policy. [ 562.053428] SELinux: Permission watch_with_perm in class fifo_file not defined in policy. [ 562.061692] SELinux: Permission watch_reads in class fifo_file not defined in policy. [ 562.069651] SELinux: Permission perfmon in class capability2 not defined in policy. [ 562.077393] SELinux: Permission bpf in class capability2 not defined in policy. [ 562.084784] SELinux: Permission checkpoint_restore in class capability2 not defined in policy. [ 562.093485] SELinux: Permission perfmon in class cap2_userns not defined in policy. [ 562.101226] SELinux: Permission bpf in class cap2_userns not defined in policy. [ 562.108621] SELinux: Permission checkpoint_restore in class cap2_userns not defined in policy. [ 562.117338] SELinux: Class mctp_socket not defined in policy. [ 562.123172] SELinux: Class anon_inode not defined in policy. [ 562.128916] SELinux: Class io_uring not defined in policy. [ 562.134489] SELinux: Class user_namespace not defined in policy. [ 562.140580] SELinux: the above unknown classes and permissions will be allowed [ 562.147896] SELinux: Converting 456 SID table entries... [ 562.176455] SELinux: policy capability network_peer_controls=1 [ 562.182373] SELinux: policy capability open_perms=1 [ 562.187341] SELinux: policy capability extended_socket_class=1 [ 562.193261] SELinux: policy capability always_check_network=0 [ 562.199090] SELinux: policy capability cgroup_seclabel=1 [ 562.204491] SELinux: policy capability nnp_nosuid_transition=1 [ 562.210427] SELinux: policy capability genfs_seclabel_symlinks=0 [ 562.216521] SELinux: policy capability ioctl_skip_cloexec=0 [ 580.947019] SELinux: Permission watch in class filesystem not defined in policy. [ 580.954504] SELinux: Permission watch in class file not defined in policy. [ 580.961461] SELinux: Permission watch_mount in class file not defined in policy. [ 580.968940] SELinux: Permission watch_sb in class file not defined in policy. [ 580.976158] SELinux: Permission watch_with_perm in class file not defined in policy. [ 580.983984] SELinux: Permission watch_reads in class file not defined in policy. [ 580.991465] SELinux: Permission watch in class dir not defined in policy. [ 580.998338] SELinux: Permission watch_mount in class dir not defined in policy. [ 581.005730] SELinux: Permission watch_sb in class dir not defined in policy. [ 581.012864] SELinux: Permission watch_with_perm in class dir not defined in policy. [ 581.020602] SELinux: Permission watch_reads in class dir not defined in policy. [ 581.027996] SELinux: Permission watch in class lnk_file not defined in policy. [ 581.035300] SELinux: Permission watch_mount in class lnk_file not defined in policy. [ 581.043124] SELinux: Permission watch_sb in class lnk_file not defined in policy. [ 581.050682] SELinux: Permission watch_with_perm in class lnk_file not defined in policy. [ 581.058847] SELinux: Permission watch_reads in class lnk_file not defined in policy. [ 581.066664] SELinux: Permission watch in class chr_file not defined in policy. [ 581.073963] SELinux: Permission watch_mount in class chr_file not defined in policy. [ 581.081790] SELinux: Permission watch_sb in class chr_file not defined in policy. [ 581.089354] SELinux: Permission watch_with_perm in class chr_file not defined in policy. [ 581.097537] SELinux: Permission watch_reads in class chr_file not defined in policy. [ 581.105372] SELinux: Permission watch in class blk_file not defined in policy. [ 581.112686] SELinux: Permission watch_mount in class blk_file not defined in policy. [ 581.120512] SELinux: Permission watch_sb in class blk_file not defined in policy. [ 581.128087] SELinux: Permission watch_with_perm in class blk_file not defined in policy. [ 581.136252] SELinux: Permission watch_reads in class blk_file not defined in policy. [ 581.144078] SELinux: Permission watch in class sock_file not defined in policy. [ 581.151472] SELinux: Permission watch_mount in class sock_file not defined in policy. [ 581.159400] SELinux: Permission watch_sb in class sock_file not defined in policy. [ 581.167061] SELinux: Permission watch_with_perm in class sock_file not defined in policy. [ 581.175322] SELinux: Permission watch_reads in class sock_file not defined in policy. [ 581.183234] SELinux: Permission watch in class fifo_file not defined in policy. [ 581.190625] SELinux: Permission watch_mount in class fifo_file not defined in policy. [ 581.198530] SELinux: Permission watch_sb in class fifo_file not defined in policy. [ 581.206175] SELinux: Permission watch_with_perm in class fifo_file not defined in policy. [ 581.214428] SELinux: Permission watch_reads in class fifo_file not defined in policy. [ 581.222383] SELinux: Permission perfmon in class capability2 not defined in policy. [ 581.230121] SELinux: Permission bpf in class capability2 not defined in policy. [ 581.237515] SELinux: Permission checkpoint_restore in class capability2 not defined in policy. [ 581.246211] SELinux: Permission perfmon in class cap2_userns not defined in policy. [ 581.253946] SELinux: Permission bpf in class cap2_userns not defined in policy. [ 581.261350] SELinux: Permission checkpoint_restore in class cap2_userns not defined in policy. [ 581.270057] SELinux: Class mctp_socket not defined in policy. [ 581.275909] SELinux: Class anon_inode not defined in policy. [ 581.281657] SELinux: Class io_uring not defined in policy. [ 581.287229] SELinux: Class user_namespace not defined in policy. [ 581.293337] SELinux: the above unknown classes and permissions will be allowed [ 581.300619] SELinux: Converting 456 SID table entries... [ 581.329198] SELinux: policy capability network_peer_controls=1 [ 581.335175] SELinux: policy capability open_perms=1 [ 581.340148] SELinux: policy capability extended_socket_class=1 [ 581.346074] SELinux: policy capability always_check_network=0 [ 581.351905] SELinux: policy capability cgroup_seclabel=1 [ 581.357305] SELinux: policy capability nnp_nosuid_transition=1 [ 581.363224] SELinux: policy capability genfs_seclabel_symlinks=0 [ 581.369319] SELinux: policy capability ioctl_skip_cloexec=0 [ 679.076959] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 679.087582] XFS (dm-4): Mounting V5 Filesystem 47b47096-0ab8-4f22-95fd-cd0bd3a91f31 [ 679.101385] evict_inodes inode 000000001db04d3d, i_count = 1, was skipped! [ 679.108354] XFS (dm-4): Ending clean mount [ 679.113224] xfs filesystem being mounted at /var/lib/ceph supports timestamps until 2038-01-19 (0x7fffffff) [ 778.752923] SELinux: Permission watch in class filesystem not defined in policy. [ 778.760408] SELinux: Permission watch in class file not defined in policy. [ 778.767371] SELinux: Permission watch_mount in class file not defined in policy. [ 778.774844] SELinux: Permission watch_sb in class file not defined in policy. [ 778.782067] SELinux: Permission watch_with_perm in class file not defined in policy. [ 778.789889] SELinux: Permission watch_reads in class file not defined in policy. [ 778.797368] SELinux: Permission watch in class dir not defined in policy. [ 778.804239] SELinux: Permission watch_mount in class dir not defined in policy. [ 778.811624] SELinux: Permission watch_sb in class dir not defined in policy. [ 778.818750] SELinux: Permission watch_with_perm in class dir not defined in policy. [ 778.826506] SELinux: Permission watch_reads in class dir not defined in policy. [ 778.833902] SELinux: Permission watch in class lnk_file not defined in policy. [ 778.841223] SELinux: Permission watch_mount in class lnk_file not defined in policy. [ 778.849065] SELinux: Permission watch_sb in class lnk_file not defined in policy. [ 778.856638] SELinux: Permission watch_with_perm in class lnk_file not defined in policy. [ 778.864805] SELinux: Permission watch_reads in class lnk_file not defined in policy. [ 778.872649] SELinux: Permission watch in class chr_file not defined in policy. [ 778.879956] SELinux: Permission watch_mount in class chr_file not defined in policy. [ 778.887783] SELinux: Permission watch_sb in class chr_file not defined in policy. [ 778.895346] SELinux: Permission watch_with_perm in class chr_file not defined in policy. [ 778.903522] SELinux: Permission watch_reads in class chr_file not defined in policy. [ 778.911347] SELinux: Permission watch in class blk_file not defined in policy. [ 778.918668] SELinux: Permission watch_mount in class blk_file not defined in policy. [ 778.926485] SELinux: Permission watch_sb in class blk_file not defined in policy. [ 778.934060] SELinux: Permission watch_with_perm in class blk_file not defined in policy. [ 778.942225] SELinux: Permission watch_reads in class blk_file not defined in policy. [ 778.950051] SELinux: Permission watch in class sock_file not defined in policy. [ 778.957444] SELinux: Permission watch_mount in class sock_file not defined in policy. [ 778.965355] SELinux: Permission watch_sb in class sock_file not defined in policy. [ 778.973069] SELinux: Permission watch_with_perm in class sock_file not defined in policy. [ 778.981383] SELinux: Permission watch_reads in class sock_file not defined in policy. [ 778.989294] SELinux: Permission watch in class fifo_file not defined in policy. [ 778.996679] SELinux: Permission watch_mount in class fifo_file not defined in policy. [ 779.004591] SELinux: Permission watch_sb in class fifo_file not defined in policy. [ 779.012235] SELinux: Permission watch_with_perm in class fifo_file not defined in policy. [ 779.020509] SELinux: Permission watch_reads in class fifo_file not defined in policy. [ 779.028466] SELinux: Permission perfmon in class capability2 not defined in policy. [ 779.036201] SELinux: Permission bpf in class capability2 not defined in policy. [ 779.043592] SELinux: Permission checkpoint_restore in class capability2 not defined in policy. [ 779.052288] SELinux: Permission perfmon in class cap2_userns not defined in policy. [ 779.060025] SELinux: Permission bpf in class cap2_userns not defined in policy. [ 779.067418] SELinux: Permission checkpoint_restore in class cap2_userns not defined in policy. [ 779.076127] SELinux: Class mctp_socket not defined in policy. [ 779.081979] SELinux: Class anon_inode not defined in policy. [ 779.087722] SELinux: Class io_uring not defined in policy. [ 779.093288] SELinux: Class user_namespace not defined in policy. [ 779.099381] SELinux: the above unknown classes and permissions will be allowed [ 779.106953] SELinux: Converting 480 SID table entries... [ 779.135767] SELinux: policy capability network_peer_controls=1 [ 779.141683] SELinux: policy capability open_perms=1 [ 779.146784] SELinux: policy capability extended_socket_class=1 [ 779.152726] SELinux: policy capability always_check_network=0 [ 779.158582] SELinux: policy capability cgroup_seclabel=1 [ 779.163984] SELinux: policy capability nnp_nosuid_transition=1 [ 779.169912] SELinux: policy capability genfs_seclabel_symlinks=0 [ 779.176014] SELinux: policy capability ioctl_skip_cloexec=0 [ 1064.416880] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 1066.086635] ceph-brx: port 1(brx.0) entered blocking state [ 1066.092138] ceph-brx: port 1(brx.0) entered disabled state [ 1066.097665] brx.0: entered allmulticast mode [ 1066.102083] brx.0: entered promiscuous mode [ 1066.106835] ceph-brx: port 1(brx.0) entered blocking state [ 1066.112375] ceph-brx: port 1(brx.0) entered forwarding state [ 1066.422328] FS-Cache: Loaded [ 1066.436424] Key type dns_resolver registered [ 1066.572700] Key type ceph registered [ 1066.576464] libceph: loaded (mon/osd proto 15/24) [ 1066.746474] ceph: loaded (mds proto 32) [ 1066.759700] libceph: mon2 (1)172.21.15.130:6790 session established [ 1066.767088] libceph: client4445 fsid cde022fc-3e6c-431e-bc4a-204d5c5d32a6 [ 2012.314915] ------------[ cut here ]------------ [ 2012.319963] list_del corruption. prev->next should be ffff8881a1d36920, but was ffff8881c2641ae0. (prev=ffff8881c2641ae0) [ 2012.331165] WARNING: CPU: 5 PID: 78680 at lib/list_debug.c:64 __list_del_entry_valid_or_report+0xb2/0xd0 [ 2012.340651] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2012.396157] CPU: 5 PID: 78680 Comm: dbench Tainted: G S 6.6.0-rc5-g5b73517e0e2f #1 [ 2012.405116] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2012.412613] RIP: 0010:__list_del_entry_valid_or_report+0xb2/0xd0 [ 2012.418617] Code: c3 48 89 fe 48 c7 c7 20 9a 1d 82 e8 c8 0b b0 ff 0f 0b 31 c0 c3 48 89 d1 48 89 f2 48 89 fe 48 c7 c7 58 9a 1d 82 e8 ae 0b b0 ff <0f> 0b 31 c0 c3 48 c7 c7 a0 9a 1d 82 e8 9d 0b b0 ff 0f 0b 31 c0 c3 [ 2012.437363] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2012.442590] RAX: 0000000000000000 RBX: ffff8881a1d361d0 RCX: 0000000000000000 [ 2012.449723] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2012.456855] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2012.463997] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d36920 [ 2012.471145] R13: ffff88817f867488 R14: ffff888100ee2000 R15: 0000000000000000 [ 2012.478271] FS: 00007f48f1832740(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2012.486356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2012.492111] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2012.499244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2012.506377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2012.513509] Call Trace: [ 2012.515971] [ 2012.518072] ? __warn+0x85/0x180 [ 2012.521302] ? __list_del_entry_valid_or_report+0xb2/0xd0 [ 2012.526702] ? report_bug+0xfc/0x1d0 [ 2012.530280] ? prb_read_valid+0x1b/0x20 [ 2012.534120] ? handle_bug+0x44/0x70 [ 2012.537630] ? exc_invalid_op+0x17/0x60 [ 2012.541469] ? asm_exc_invalid_op+0x1a/0x20 [ 2012.545678] ? __list_del_entry_valid_or_report+0xb2/0xd0 [ 2012.551073] ? __list_del_entry_valid_or_report+0xb2/0xd0 [ 2012.556471] __cap_delay_requeue+0xd6/0x1d0 [ceph] [ 2012.561309] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2012.566460] ceph_write_iter+0x5aa/0x770 [ceph] [ 2012.571002] ? lock_is_held_type+0xaa/0x120 [ 2012.575186] vfs_write+0x324/0x4f0 [ 2012.578612] ksys_pwrite64+0x65/0xa0 [ 2012.582185] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2012.587326] do_syscall_64+0x38/0x80 [ 2012.590906] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2012.595977] RIP: 0033:0x7f48f0eeb93b [ 2012.599554] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2012.618302] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2012.625882] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2012.633008] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2012.640150] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2012.647281] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2012.654406] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2012.661554] [ 2012.663740] irq event stamp: 12755147 [ 2012.667396] hardirqs last enabled at (12755157): [] console_unlock+0x106/0x120 [ 2012.676281] hardirqs last disabled at (12755172): [] console_unlock+0xeb/0x120 [ 2012.685060] softirqs last enabled at (12755170): [] __do_softirq+0x2be/0x371 [ 2012.693755] softirqs last disabled at (12755165): [] irq_exit_rcu+0xb0/0xd0 [ 2012.702273] ---[ end trace 0000000000000000 ]--- [ 2012.706894] ------------[ cut here ]------------ [ 2012.711512] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2012.723166] WARNING: CPU: 5 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2012.732122] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2012.787591] CPU: 5 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2012.796551] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2012.804029] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2012.809515] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2012.828255] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2012.833480] RAX: 0000000000000000 RBX: ffff8881a1d361d0 RCX: 0000000000000000 [ 2012.840614] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2012.847745] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2012.854871] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d36920 [ 2012.862002] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2012.869127] FS: 00007f48f1832740(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2012.877212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2012.882950] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2012.890073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2012.897198] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2012.904324] Call Trace: [ 2012.906776] [ 2012.908881] ? __warn+0x85/0x180 [ 2012.912115] ? __list_add_valid_or_report+0x7d/0xa0 [ 2012.916993] ? report_bug+0xfc/0x1d0 [ 2012.920574] ? prb_read_valid+0x1b/0x20 [ 2012.924413] ? handle_bug+0x44/0x70 [ 2012.927906] ? exc_invalid_op+0x17/0x60 [ 2012.931744] ? asm_exc_invalid_op+0x1a/0x20 [ 2012.935935] ? __list_add_valid_or_report+0x7d/0xa0 [ 2012.940811] ? __list_add_valid_or_report+0x7d/0xa0 [ 2012.945691] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2012.950581] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2012.955730] ceph_write_iter+0x5aa/0x770 [ceph] [ 2012.960276] ? lock_is_held_type+0xaa/0x120 [ 2012.964465] vfs_write+0x324/0x4f0 [ 2012.967873] ksys_pwrite64+0x65/0xa0 [ 2012.971446] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2012.976587] do_syscall_64+0x38/0x80 [ 2012.980167] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2012.985222] RIP: 0033:0x7f48f0eeb93b [ 2012.988800] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2013.007556] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2013.015119] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2013.022252] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2013.029384] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2013.036518] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2013.043651] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2013.050791] [ 2013.052976] irq event stamp: 12756161 [ 2013.056633] hardirqs last enabled at (12756171): [] console_unlock+0x106/0x120 [ 2013.065500] hardirqs last disabled at (12756180): [] console_unlock+0xeb/0x120 [ 2013.074279] softirqs last enabled at (12756148): [] __do_softirq+0x2be/0x371 [ 2013.082973] softirqs last disabled at (12756199): [] irq_exit_rcu+0xb0/0xd0 [ 2013.091492] ---[ end trace 0000000000000000 ]--- [ 2013.096274] ------------[ cut here ]------------ [ 2013.100905] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2013.112575] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2013.121552] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2013.177031] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2013.185989] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2013.193468] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2013.198955] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2013.217705] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2013.222935] RAX: 0000000000000000 RBX: ffff8881d0aeeb98 RCX: 0000000000000000 [ 2013.230067] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2013.237201] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2013.244335] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d0aef2e8 [ 2013.251466] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2013.258598] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2013.266686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2013.272430] CR2: 0000563ac8c0cbf8 CR3: 0000000120c1e001 CR4: 00000000003706f0 [ 2013.279565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2013.286695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2013.293844] Call Trace: [ 2013.296299] [ 2013.298407] ? __warn+0x85/0x180 [ 2013.301641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2013.306522] ? report_bug+0xfc/0x1d0 [ 2013.310108] ? prb_read_valid+0x1b/0x20 [ 2013.313951] ? handle_bug+0x44/0x70 [ 2013.317447] ? exc_invalid_op+0x17/0x60 [ 2013.321288] ? asm_exc_invalid_op+0x1a/0x20 [ 2013.325479] ? __list_add_valid_or_report+0x7d/0xa0 [ 2013.330363] ? __list_add_valid_or_report+0x7d/0xa0 [ 2013.335240] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2013.340142] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2013.345297] ceph_write_iter+0x5aa/0x770 [ceph] [ 2013.349852] ? lock_is_held_type+0xaa/0x120 [ 2013.354043] vfs_write+0x324/0x4f0 [ 2013.357464] ksys_pwrite64+0x65/0xa0 [ 2013.361039] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2013.366183] do_syscall_64+0x38/0x80 [ 2013.369760] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2013.374817] RIP: 0033:0x7f48f0eeb93b [ 2013.378401] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2013.397147] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2013.404712] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2013.411849] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2013.418978] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2013.426113] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2013.433242] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2013.440390] [ 2013.442578] irq event stamp: 12751003 [ 2013.446246] hardirqs last enabled at (12751013): [] console_unlock+0x106/0x120 [ 2013.455109] hardirqs last disabled at (12751022): [] console_unlock+0xeb/0x120 [ 2013.463892] softirqs last enabled at (12750990): [] __do_softirq+0x2be/0x371 [ 2013.472595] softirqs last disabled at (12751041): [] irq_exit_rcu+0xb0/0xd0 [ 2013.481120] ---[ end trace 0000000000000000 ]--- [ 2013.485738] ------------[ cut here ]------------ [ 2013.490406] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2013.502054] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2013.511014] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2013.566534] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2013.575514] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2013.582992] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2013.588494] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2013.607277] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2013.612510] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2013.619661] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2013.626794] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2013.633925] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d372e8 [ 2013.641076] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2013.648226] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2013.656330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2013.662091] CR2: 00007f322205d934 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2013.669243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2013.676394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2013.683563] Call Trace: [ 2013.686015] [ 2013.688139] ? __warn+0x85/0x180 [ 2013.691388] ? __list_add_valid_or_report+0x7d/0xa0 [ 2013.696302] ? report_bug+0xfc/0x1d0 [ 2013.699881] ? prb_read_valid+0x1b/0x20 [ 2013.703741] ? handle_bug+0x44/0x70 [ 2013.707238] ? exc_invalid_op+0x17/0x60 [ 2013.711097] ? asm_exc_invalid_op+0x1a/0x20 [ 2013.715325] ? __list_add_valid_or_report+0x7d/0xa0 [ 2013.720224] ? __list_add_valid_or_report+0x7d/0xa0 [ 2013.725109] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2013.730046] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2013.735231] ceph_write_iter+0x5aa/0x770 [ceph] [ 2013.739793] ? lock_is_held_type+0xaa/0x120 [ 2013.744002] vfs_write+0x324/0x4f0 [ 2013.747422] ksys_pwrite64+0x65/0xa0 [ 2013.751006] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2013.756181] do_syscall_64+0x38/0x80 [ 2013.759762] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2013.764822] RIP: 0033:0x7f48f0eeb93b [ 2013.768399] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2013.787165] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2013.794749] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2013.801879] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2013.809031] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2013.816181] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2013.823332] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2013.830509] [ 2013.832699] irq event stamp: 12758261 [ 2013.836364] hardirqs last enabled at (12758271): [] console_unlock+0x106/0x120 [ 2013.845250] hardirqs last disabled at (12758280): [] console_unlock+0xeb/0x120 [ 2013.854046] softirqs last enabled at (12757846): [] __do_softirq+0x2be/0x371 [ 2013.862760] softirqs last disabled at (12758299): [] irq_exit_rcu+0xb0/0xd0 [ 2013.871283] ---[ end trace 0000000000000000 ]--- [ 2013.875921] ------------[ cut here ]------------ [ 2013.880630] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2013.892281] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2013.901234] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2013.956712] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2013.965671] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2013.973154] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2013.978638] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2013.997385] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2014.002610] RAX: 0000000000000000 RBX: ffff8881d0aeeb98 RCX: 0000000000000000 [ 2014.009743] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2014.016875] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2014.024010] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d0aef2e8 [ 2014.031140] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2014.038264] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2014.046353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2014.052097] CR2: 00007f9bd54be000 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2014.059230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2014.066365] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2014.073499] Call Trace: [ 2014.075949] [ 2014.078057] ? __warn+0x85/0x180 [ 2014.081288] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.086166] ? report_bug+0xfc/0x1d0 [ 2014.089747] ? prb_read_valid+0x1b/0x20 [ 2014.093587] ? handle_bug+0x44/0x70 [ 2014.097078] ? exc_invalid_op+0x17/0x60 [ 2014.100918] ? asm_exc_invalid_op+0x1a/0x20 [ 2014.105110] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.109985] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2014.114881] try_get_cap_refs+0x433/0x880 [ceph] [ 2014.119509] ? _raw_spin_unlock+0x23/0x40 [ 2014.123525] __ceph_get_caps+0xf8/0x760 [ceph] [ 2014.127976] ? __vfs_getxattr+0x8d/0xb0 [ 2014.131817] ? dentry_needs_remove_privs+0x31/0x50 [ 2014.136608] ? __file_remove_privs+0x89/0x110 [ 2014.140967] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2014.145682] ? lock_release+0xc5/0x260 [ 2014.149436] ? ceph_write_iter+0x260/0x770 [ceph] [ 2014.154151] ceph_write_iter+0x260/0x770 [ceph] [ 2014.158695] ? lock_is_held_type+0xaa/0x120 [ 2014.162880] vfs_write+0x324/0x4f0 [ 2014.166289] ksys_pwrite64+0x65/0xa0 [ 2014.169869] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2014.175014] do_syscall_64+0x38/0x80 [ 2014.178591] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2014.183642] RIP: 0033:0x7f48f0eeb93b [ 2014.187221] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2014.205972] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2014.213536] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2014.220668] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2014.227798] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2014.234923] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2014.242048] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2014.249187] [ 2014.251372] irq event stamp: 12753083 [ 2014.255032] hardirqs last enabled at (12753093): [] console_unlock+0x106/0x120 [ 2014.263896] hardirqs last disabled at (12753102): [] console_unlock+0xeb/0x120 [ 2014.272676] softirqs last enabled at (12753030): [] __do_softirq+0x2be/0x371 [ 2014.281369] softirqs last disabled at (12753121): [] irq_exit_rcu+0xb0/0xd0 [ 2014.289889] ---[ end trace 0000000000000000 ]--- [ 2014.294509] ------------[ cut here ]------------ [ 2014.299128] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2014.310777] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2014.319728] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2014.375199] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2014.384165] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2014.391647] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2014.397132] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2014.415880] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2014.421104] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2014.428236] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2014.435370] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2014.442505] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d372e8 [ 2014.449635] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2014.456760] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2014.464846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2014.470592] CR2: 00007f9bd54c3000 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2014.477724] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2014.484849] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2014.491981] Call Trace: [ 2014.494441] [ 2014.496542] ? __warn+0x85/0x180 [ 2014.499776] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.504656] ? report_bug+0xfc/0x1d0 [ 2014.508235] ? prb_read_valid+0x1b/0x20 [ 2014.512087] ? handle_bug+0x44/0x70 [ 2014.515591] ? exc_invalid_op+0x17/0x60 [ 2014.519432] ? asm_exc_invalid_op+0x1a/0x20 [ 2014.523621] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.528496] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2014.533386] try_get_cap_refs+0x433/0x880 [ceph] [ 2014.538012] ? _raw_spin_unlock+0x23/0x40 [ 2014.542030] __ceph_get_caps+0xf8/0x760 [ceph] [ 2014.546488] ? __vfs_getxattr+0x8d/0xb0 [ 2014.550330] ? dentry_needs_remove_privs+0x31/0x50 [ 2014.555121] ? __file_remove_privs+0x89/0x110 [ 2014.559480] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2014.564196] ? lock_release+0xc5/0x260 [ 2014.567952] ? ceph_write_iter+0x260/0x770 [ceph] [ 2014.572661] ceph_write_iter+0x260/0x770 [ceph] [ 2014.577209] ? lock_is_held_type+0xaa/0x120 [ 2014.581393] vfs_write+0x324/0x4f0 [ 2014.584799] ksys_pwrite64+0x65/0xa0 [ 2014.588373] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2014.593516] do_syscall_64+0x38/0x80 [ 2014.597094] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2014.602145] RIP: 0033:0x7f48f0eeb93b [ 2014.605724] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2014.624473] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2014.632040] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2014.639170] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2014.646303] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2014.653435] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2014.660569] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2014.667708] [ 2014.669893] irq event stamp: 12760445 [ 2014.673552] hardirqs last enabled at (12760455): [] console_unlock+0x106/0x120 [ 2014.682427] hardirqs last disabled at (12760466): [] console_unlock+0xeb/0x120 [ 2014.691206] softirqs last enabled at (12760382): [] __do_softirq+0x2be/0x371 [ 2014.699901] softirqs last disabled at (12760485): [] irq_exit_rcu+0xb0/0xd0 [ 2014.708419] ---[ end trace 0000000000000000 ]--- [ 2014.713040] ------------[ cut here ]------------ [ 2014.717731] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2014.729376] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2014.738070] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2014.793566] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2014.802694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2014.810179] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2014.815700] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2014.821194] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2014.839940] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2014.845165] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2014.852299] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2014.859435] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2014.866567] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881e4f8e920 [ 2014.873708] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2014.880840] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2014.888924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2014.894671] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2014.901804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2014.908937] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2014.916068] Call Trace: [ 2014.918523] [ 2014.920629] ? __warn+0x85/0x180 [ 2014.923865] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.928749] ? report_bug+0xfc/0x1d0 [ 2014.932331] ? prb_read_valid+0x1b/0x20 [ 2014.936179] ? handle_bug+0x44/0x70 [ 2014.939670] ? exc_invalid_op+0x17/0x60 [ 2014.943514] ? asm_exc_invalid_op+0x1a/0x20 [ 2014.947715] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.952602] ? __list_add_valid_or_report+0x7d/0xa0 [ 2014.957480] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2014.962381] ceph_add_cap+0x20b/0x620 [ceph] [ 2014.966687] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2014.971427] ceph_fill_trace+0x143/0xf30 [ceph] [ 2014.975976] ? __mutex_lock+0x97/0xce0 [ 2014.979725] ? __mutex_lock+0xaec/0xce0 [ 2014.983572] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2014.988121] ? lock_downgrade+0xfd/0x250 [ 2014.992062] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2014.996450] ceph_con_process_message+0x74/0x150 [libceph] [ 2015.001961] ? calc_signature+0xdb/0x100 [libceph] [ 2015.006772] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2015.012116] ? process_scheduled_works+0x22a/0x510 [ 2015.016913] ? process_scheduled_works+0x22a/0x510 [ 2015.021709] ceph_con_workfn+0x344/0x750 [libceph] [ 2015.026523] ? process_scheduled_works+0x22a/0x510 [ 2015.031323] process_scheduled_works+0x2a4/0x510 [ 2015.035960] worker_thread+0x188/0x340 [ 2015.039713] ? rescuer_thread+0x310/0x310 [ 2015.043731] kthread+0xf5/0x130 [ 2015.046879] ? kthread_complete_and_exit+0x20/0x20 [ 2015.051675] ret_from_fork+0x31/0x40 [ 2015.055258] ? kthread_complete_and_exit+0x20/0x20 [ 2015.060051] ret_from_fork_asm+0x11/0x20 [ 2015.063990] [ 2015.066177] irq event stamp: 8775681 [ 2015.069759] hardirqs last enabled at (8775691): [] console_unlock+0x106/0x120 [ 2015.078537] hardirqs last disabled at (8775700): [] console_unlock+0xeb/0x120 [ 2015.087231] softirqs last enabled at (8775638): [] __do_softirq+0x2be/0x371 [ 2015.095836] softirqs last disabled at (8775633): [] irq_exit_rcu+0xb0/0xd0 [ 2015.104271] ---[ end trace 0000000000000000 ]--- [ 2015.108889] ------------[ cut here ]------------ [ 2015.113505] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2015.125152] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2015.134108] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2015.189578] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2015.198528] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2015.206007] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2015.211494] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2015.230240] RSP: 0018:ffffc90001ff7d18 EFLAGS: 00010286 [ 2015.235465] RAX: 0000000000000000 RBX: ffff8881e4d8eb98 RCX: 0000000000000000 [ 2015.242598] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2015.249730] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2015.256855] R10: 0000000000000001 R11: ffffc90001ff7ba8 R12: ffff8881e4d8f2e8 [ 2015.263990] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2015.271120] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2015.279210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2015.284954] CR2: 000056400e01ab90 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2015.292087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2015.299219] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2015.306352] Call Trace: [ 2015.308804] [ 2015.310902] ? __warn+0x85/0x180 [ 2015.314136] ? __list_add_valid_or_report+0x7d/0xa0 [ 2015.319014] ? report_bug+0xfc/0x1d0 [ 2015.322593] ? prb_read_valid+0x1b/0x20 [ 2015.326435] ? handle_bug+0x44/0x70 [ 2015.329927] ? exc_invalid_op+0x17/0x60 [ 2015.333764] ? asm_exc_invalid_op+0x1a/0x20 [ 2015.337955] ? __list_add_valid_or_report+0x7d/0xa0 [ 2015.342830] ? __list_add_valid_or_report+0x7d/0xa0 [ 2015.347709] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2015.352600] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2015.357750] ceph_write_iter+0x5aa/0x770 [ceph] [ 2015.362300] ? lock_is_held_type+0xaa/0x120 [ 2015.366484] vfs_write+0x324/0x4f0 [ 2015.369893] ksys_pwrite64+0x65/0xa0 [ 2015.373467] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2015.378611] do_syscall_64+0x38/0x80 [ 2015.382186] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2015.387251] RIP: 0033:0x7f48f0eeb93b [ 2015.390833] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2015.409578] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2015.417139] RAX: ffffffffffffffda RBX: 0000000002075980 RCX: 00007f48f0eeb93b [ 2015.424272] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 000000000000001a [ 2015.431404] RBP: 00000000000029ea R08: 0000000000001000 R09: 0000000000000003 [ 2015.438528] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e140 [ 2015.445654] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207e200 [ 2015.452793] [ 2015.454978] irq event stamp: 12764115 [ 2015.458646] hardirqs last enabled at (12764125): [] console_unlock+0x106/0x120 [ 2015.467511] hardirqs last disabled at (12764134): [] console_unlock+0xeb/0x120 [ 2015.476291] softirqs last enabled at (12764148): [] __do_softirq+0x2be/0x371 [ 2015.484984] softirqs last disabled at (12764143): [] irq_exit_rcu+0xb0/0xd0 [ 2015.493503] ---[ end trace 0000000000000000 ]--- [ 2015.498122] ------------[ cut here ]------------ [ 2015.502756] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2015.514408] WARNING: CPU: 6 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2015.523369] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2015.578848] CPU: 6 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2015.587808] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2015.595311] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2015.600815] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2015.619585] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2015.624814] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2015.631956] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2015.639096] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2015.646241] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d372e8 [ 2015.653379] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2015.660513] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2015.668607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2015.674365] CR2: 00007fb38f6c9028 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2015.681506] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2015.688646] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2015.695787] Call Trace: [ 2015.698249] [ 2015.700365] ? __warn+0x85/0x180 [ 2015.703606] ? __list_add_valid_or_report+0x7d/0xa0 [ 2015.708493] ? report_bug+0xfc/0x1d0 [ 2015.712083] ? prb_read_valid+0x1b/0x20 [ 2015.715931] ? handle_bug+0x44/0x70 [ 2015.719431] ? exc_invalid_op+0x17/0x60 [ 2015.723277] ? asm_exc_invalid_op+0x1a/0x20 [ 2015.727476] ? __list_add_valid_or_report+0x7d/0xa0 [ 2015.732363] ? __list_add_valid_or_report+0x7d/0xa0 [ 2015.737248] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2015.742148] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2015.747307] ceph_write_iter+0x5aa/0x770 [ceph] [ 2015.751857] ? lock_is_held_type+0xaa/0x120 [ 2015.756050] vfs_write+0x324/0x4f0 [ 2015.759467] ksys_pwrite64+0x65/0xa0 [ 2015.763048] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2015.768199] do_syscall_64+0x38/0x80 [ 2015.771788] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2015.776850] RIP: 0033:0x7f48f0eeb93b [ 2015.780437] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2015.799190] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2015.806765] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2015.813905] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2015.821038] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2015.828173] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2015.835313] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2015.842454] [ 2015.844648] irq event stamp: 12765571 [ 2015.848323] hardirqs last enabled at (12765627): [] console_unlock+0x106/0x120 [ 2015.857199] hardirqs last disabled at (12765688): [] console_unlock+0xeb/0x120 [ 2015.865992] softirqs last enabled at (12765740): [] __do_softirq+0x2be/0x371 [ 2015.874697] softirqs last disabled at (12765751): [] irq_exit_rcu+0xb0/0xd0 [ 2015.883225] ---[ end trace 0000000000000000 ]--- [ 2015.887854] ------------[ cut here ]------------ [ 2015.892472] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2015.904124] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2015.912826] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2015.968316] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2015.977445] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2015.984921] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2015.990428] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2015.995921] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2016.014667] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2016.019894] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2016.027028] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2016.034164] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2016.041304] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881e4f8e920 [ 2016.048432] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2016.055566] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2016.063652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2016.069398] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2016.076532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2016.083665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2016.090797] Call Trace: [ 2016.093250] [ 2016.095357] ? __warn+0x85/0x180 [ 2016.098595] ? __list_add_valid_or_report+0x7d/0xa0 [ 2016.103477] ? report_bug+0xfc/0x1d0 [ 2016.107058] ? prb_read_valid+0x1b/0x20 [ 2016.110898] ? handle_bug+0x44/0x70 [ 2016.114389] ? exc_invalid_op+0x17/0x60 [ 2016.118228] ? asm_exc_invalid_op+0x1a/0x20 [ 2016.122425] ? __list_add_valid_or_report+0x7d/0xa0 [ 2016.127304] ? __list_add_valid_or_report+0x7d/0xa0 [ 2016.132184] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2016.137083] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2016.141731] ceph_fill_trace+0x143/0xf30 [ceph] [ 2016.146274] ? __mutex_lock+0x97/0xce0 [ 2016.150027] ? __mutex_lock+0xaec/0xce0 [ 2016.153866] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2016.158418] ? lock_downgrade+0xfd/0x250 [ 2016.162362] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2016.166759] ceph_con_process_message+0x74/0x150 [libceph] [ 2016.172257] ? calc_signature+0xdb/0x100 [libceph] [ 2016.177082] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2016.182413] ? process_scheduled_works+0x22a/0x510 [ 2016.187215] ? process_scheduled_works+0x22a/0x510 [ 2016.192007] ceph_con_workfn+0x344/0x750 [libceph] [ 2016.196824] ? process_scheduled_works+0x22a/0x510 [ 2016.201619] process_scheduled_works+0x2a4/0x510 [ 2016.206249] worker_thread+0x188/0x340 [ 2016.210003] ? rescuer_thread+0x310/0x310 [ 2016.214023] kthread+0xf5/0x130 [ 2016.217169] ? kthread_complete_and_exit+0x20/0x20 [ 2016.221965] ret_from_fork+0x31/0x40 [ 2016.225552] ? kthread_complete_and_exit+0x20/0x20 [ 2016.230353] ret_from_fork_asm+0x11/0x20 [ 2016.234299] [ 2016.236495] irq event stamp: 8778455 [ 2016.240074] hardirqs last enabled at (8778465): [] console_unlock+0x106/0x120 [ 2016.248855] hardirqs last disabled at (8778474): [] console_unlock+0xeb/0x120 [ 2016.257554] softirqs last enabled at (8778422): [] __do_softirq+0x2be/0x371 [ 2016.266162] softirqs last disabled at (8778417): [] irq_exit_rcu+0xb0/0xd0 [ 2016.274594] ---[ end trace 0000000000000000 ]--- [ 2016.279214] ------------[ cut here ]------------ [ 2016.283837] list_del corruption. next->prev should be ffff8881f789cbc8, but was ffff8881a1d36920. (next=ffff88817f867478) [ 2016.294797] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:67 __list_del_entry_valid_or_report+0xc3/0xd0 [ 2016.304270] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2016.359792] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2016.368744] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2016.376221] RIP: 0010:__list_del_entry_valid_or_report+0xc3/0xd0 [ 2016.382228] Code: 0b 31 c0 c3 48 89 d1 48 89 f2 48 89 fe 48 c7 c7 58 9a 1d 82 e8 ae 0b b0 ff 0f 0b 31 c0 c3 48 c7 c7 a0 9a 1d 82 e8 9d 0b b0 ff <0f> 0b 31 c0 c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 d2 74 27 [ 2016.400975] RSP: 0018:ffffc90001ff7d18 EFLAGS: 00010286 [ 2016.406200] RAX: 0000000000000000 RBX: ffff8881f789c478 RCX: 0000000000000000 [ 2016.413334] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2016.420465] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2016.427600] R10: 0000000000000001 R11: ffffc90001ff7ba8 R12: ffff8881f789cbc8 [ 2016.434731] R13: ffff88817f867488 R14: ffff888100ee2000 R15: 0000000000000000 [ 2016.441864] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2016.449950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2016.455697] CR2: 00007f7ca5851d68 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2016.462829] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2016.469963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2016.477097] Call Trace: [ 2016.479547] [ 2016.481655] ? __warn+0x85/0x180 [ 2016.484889] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2016.490289] ? report_bug+0xfc/0x1d0 [ 2016.493876] ? prb_read_valid+0x1b/0x20 [ 2016.497716] ? handle_bug+0x44/0x70 [ 2016.501208] ? exc_invalid_op+0x17/0x60 [ 2016.505046] ? asm_exc_invalid_op+0x1a/0x20 [ 2016.509244] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2016.514651] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2016.520053] __cap_delay_requeue+0xd6/0x1d0 [ceph] [ 2016.524872] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2016.530040] ceph_write_iter+0x5aa/0x770 [ceph] [ 2016.534604] ? lock_is_held_type+0xaa/0x120 [ 2016.538802] vfs_write+0x324/0x4f0 [ 2016.542220] ksys_pwrite64+0x65/0xa0 [ 2016.545798] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2016.550940] do_syscall_64+0x38/0x80 [ 2016.554526] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2016.559579] RIP: 0033:0x7f48f0eeb93b [ 2016.563158] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2016.581902] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2016.589470] RAX: ffffffffffffffda RBX: 0000000002075990 RCX: 00007f48f0eeb93b [ 2016.596602] RDX: 0000000000002000 RSI: 000000000207e200 RDI: 000000000000001b [ 2016.603735] RBP: 00000000000029ee R08: 0000000000002000 R09: 0000000000000003 [ 2016.610867] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e140 [ 2016.617999] R13: 0000000000002000 R14: 0000000000002000 R15: 000000000207e200 [ 2016.625149] [ 2016.627343] irq event stamp: 12767447 [ 2016.631009] hardirqs last enabled at (12767457): [] console_unlock+0x106/0x120 [ 2016.639877] hardirqs last disabled at (12767466): [] console_unlock+0xeb/0x120 [ 2016.648658] softirqs last enabled at (12767482): [] __do_softirq+0x2be/0x371 [ 2016.657356] softirqs last disabled at (12767475): [] irq_exit_rcu+0xb0/0xd0 [ 2016.665877] ---[ end trace 0000000000000000 ]--- [ 2016.670495] ------------[ cut here ]------------ [ 2016.675115] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2016.686760] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2016.695719] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2016.751244] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2016.760200] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2016.767676] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2016.773164] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2016.791913] RSP: 0018:ffffc90001ff7d18 EFLAGS: 00010286 [ 2016.797143] RAX: 0000000000000000 RBX: ffff8881f789c478 RCX: 0000000000000000 [ 2016.804276] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2016.811409] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2016.818542] R10: 0000000000000001 R11: ffffc90001ff7ba8 R12: ffff8881f789cbc8 [ 2016.825678] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2016.832809] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2016.840893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2016.846641] CR2: 00007f7ca5851d68 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2016.853774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2016.860907] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2016.868037] Call Trace: [ 2016.870492] [ 2016.872601] ? __warn+0x85/0x180 [ 2016.875839] ? __list_add_valid_or_report+0x7d/0xa0 [ 2016.880719] ? report_bug+0xfc/0x1d0 [ 2016.884299] ? prb_read_valid+0x1b/0x20 [ 2016.888141] ? handle_bug+0x44/0x70 [ 2016.891640] ? exc_invalid_op+0x17/0x60 [ 2016.895478] ? asm_exc_invalid_op+0x1a/0x20 [ 2016.899674] ? __list_add_valid_or_report+0x7d/0xa0 [ 2016.904555] ? __list_add_valid_or_report+0x7d/0xa0 [ 2016.909439] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2016.914341] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2016.919505] ceph_write_iter+0x5aa/0x770 [ceph] [ 2016.924060] ? lock_is_held_type+0xaa/0x120 [ 2016.928253] vfs_write+0x324/0x4f0 [ 2016.931672] ksys_pwrite64+0x65/0xa0 [ 2016.935249] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2016.940393] do_syscall_64+0x38/0x80 [ 2016.943980] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2016.949030] RIP: 0033:0x7f48f0eeb93b [ 2016.952610] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2016.971361] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2016.978931] RAX: ffffffffffffffda RBX: 0000000002075990 RCX: 00007f48f0eeb93b [ 2016.986064] RDX: 0000000000002000 RSI: 000000000207e200 RDI: 000000000000001b [ 2016.993196] RBP: 00000000000029ee R08: 0000000000002000 R09: 0000000000000003 [ 2017.000328] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e140 [ 2017.007461] R13: 0000000000002000 R14: 0000000000002000 R15: 000000000207e200 [ 2017.014622] [ 2017.016813] irq event stamp: 12768557 [ 2017.020479] hardirqs last enabled at (12768573): [] console_unlock+0x106/0x120 [ 2017.029347] hardirqs last disabled at (12768588): [] console_unlock+0xeb/0x120 [ 2017.038124] softirqs last enabled at (12768602): [] __do_softirq+0x2be/0x371 [ 2017.046821] softirqs last disabled at (12768613): [] irq_exit_rcu+0xb0/0xd0 [ 2017.055346] ---[ end trace 0000000000000000 ]--- [ 2017.059967] ------------[ cut here ]------------ [ 2017.064584] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2017.076231] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2017.084928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2017.140413] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2017.149537] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2017.157016] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2017.162529] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2017.168023] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2017.186769] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2017.191996] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2017.199130] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2017.206262] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2017.213396] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881e4d0b838 [ 2017.220526] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2017.227660] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2017.235744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2017.241491] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2017.248624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2017.255757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2017.262888] Call Trace: [ 2017.265343] [ 2017.267448] ? __warn+0x85/0x180 [ 2017.270687] ? __list_add_valid_or_report+0x7d/0xa0 [ 2017.275572] ? report_bug+0xfc/0x1d0 [ 2017.279150] ? prb_read_valid+0x1b/0x20 [ 2017.282992] ? handle_bug+0x44/0x70 [ 2017.286491] ? exc_invalid_op+0x17/0x60 [ 2017.290329] ? asm_exc_invalid_op+0x1a/0x20 [ 2017.294527] ? __list_add_valid_or_report+0x7d/0xa0 [ 2017.299405] ? __list_add_valid_or_report+0x7d/0xa0 [ 2017.304283] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2017.309186] ceph_add_cap+0x20b/0x620 [ceph] [ 2017.313489] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2017.318222] ceph_fill_trace+0x143/0xf30 [ceph] [ 2017.322773] ? __mutex_lock+0x97/0xce0 [ 2017.326530] ? __mutex_lock+0xaec/0xce0 [ 2017.330373] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2017.334921] ? lock_downgrade+0xfd/0x250 [ 2017.338857] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2017.343245] ceph_con_process_message+0x74/0x150 [libceph] [ 2017.348755] ? calc_signature+0xdb/0x100 [libceph] [ 2017.353566] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2017.358910] ? process_scheduled_works+0x22a/0x510 [ 2017.363708] ? process_scheduled_works+0x22a/0x510 [ 2017.368504] ceph_con_workfn+0x344/0x750 [libceph] [ 2017.373317] ? process_scheduled_works+0x22a/0x510 [ 2017.378118] process_scheduled_works+0x2a4/0x510 [ 2017.382754] worker_thread+0x188/0x340 [ 2017.386510] ? rescuer_thread+0x310/0x310 [ 2017.390526] kthread+0xf5/0x130 [ 2017.393675] ? kthread_complete_and_exit+0x20/0x20 [ 2017.398479] ret_from_fork+0x31/0x40 [ 2017.402063] ? kthread_complete_and_exit+0x20/0x20 [ 2017.406855] ret_from_fork_asm+0x11/0x20 [ 2017.410793] [ 2017.412980] irq event stamp: 8781239 [ 2017.416561] hardirqs last enabled at (8781249): [] console_unlock+0x106/0x120 [ 2017.425340] hardirqs last disabled at (8781258): [] console_unlock+0xeb/0x120 [ 2017.434033] softirqs last enabled at (8781196): [] __do_softirq+0x2be/0x371 [ 2017.442648] softirqs last disabled at (8781191): [] irq_exit_rcu+0xb0/0xd0 [ 2017.451097] ---[ end trace 0000000000000000 ]--- [ 2017.455725] ------------[ cut here ]------------ [ 2017.460553] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2017.472218] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2017.481359] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2017.536864] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2017.546451] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2017.554395] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2017.560038] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2017.578794] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2017.584254] RAX: 0000000000000000 RBX: ffff8881e4d8eb98 RCX: 0000000000000000 [ 2017.591403] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2017.598745] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2017.605886] R10: 0000000000000001 R11: ffffc90001ff7b18 R12: ffff8881e4d8f2e8 [ 2017.613201] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2017.620352] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2017.628499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2017.634261] CR2: 000056400e01ab90 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2017.641448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2017.648584] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2017.655750] Call Trace: [ 2017.658202] [ 2017.660309] ? __warn+0x85/0x180 [ 2017.663540] ? __list_add_valid_or_report+0x7d/0xa0 [ 2017.668422] ? report_bug+0xfc/0x1d0 [ 2017.672009] ? prb_read_valid+0x1b/0x20 [ 2017.675852] ? handle_bug+0x44/0x70 [ 2017.679349] ? exc_invalid_op+0x17/0x60 [ 2017.683188] ? asm_exc_invalid_op+0x1a/0x20 [ 2017.687382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2017.692265] ? __list_add_valid_or_report+0x7d/0xa0 [ 2017.697158] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2017.702085] ceph_open+0x16b/0x440 [ceph] [ 2017.706118] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2017.711285] do_dentry_open+0x172/0x530 [ 2017.715128] path_openat+0x743/0x9d0 [ 2017.718713] do_filp_open+0xb8/0x120 [ 2017.722299] ? lock_release+0xc5/0x260 [ 2017.726058] ? _raw_spin_unlock+0x23/0x40 [ 2017.730075] ? alloc_fd+0x13f/0x200 [ 2017.733574] do_sys_openat2+0x249/0x290 [ 2017.737420] do_sys_open+0x4f/0x80 [ 2017.740836] do_syscall_64+0x38/0x80 [ 2017.744420] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2017.749472] RIP: 0033:0x7f48f0eed2f2 [ 2017.753050] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2017.771797] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2017.779362] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2017.786497] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2017.793628] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2017.800764] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2017.807894] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029ef [ 2017.815041] [ 2017.817228] irq event stamp: 12781517 [ 2017.820896] hardirqs last enabled at (12781527): [] console_unlock+0x106/0x120 [ 2017.829761] hardirqs last disabled at (12781536): [] console_unlock+0xeb/0x120 [ 2017.838540] softirqs last enabled at (12781504): [] __do_softirq+0x2be/0x371 [ 2017.847233] softirqs last disabled at (12781499): [] irq_exit_rcu+0xb0/0xd0 [ 2017.855754] ---[ end trace 0000000000000000 ]--- [ 2017.860373] ------------[ cut here ]------------ [ 2017.864990] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2017.876640] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2017.885593] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2017.941070] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2017.950024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2017.957499] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2017.962986] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2017.981732] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2017.986958] RAX: 0000000000000000 RBX: ffff8881d0aeeb98 RCX: 0000000000000000 [ 2017.994091] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2018.001224] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2018.008359] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d0aef2e8 [ 2018.015489] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2018.022623] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2018.030709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2018.036455] CR2: 0000563ac8c0cbf8 CR3: 0000000120c1e001 CR4: 00000000003706f0 [ 2018.043588] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2018.050718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2018.057843] Call Trace: [ 2018.060298] [ 2018.062407] ? __warn+0x85/0x180 [ 2018.065645] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.070524] ? report_bug+0xfc/0x1d0 [ 2018.074104] ? prb_read_valid+0x1b/0x20 [ 2018.077947] ? handle_bug+0x44/0x70 [ 2018.081436] ? exc_invalid_op+0x17/0x60 [ 2018.085276] ? asm_exc_invalid_op+0x1a/0x20 [ 2018.089465] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.094341] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.099219] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2018.104111] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2018.109260] ceph_write_iter+0x5aa/0x770 [ceph] [ 2018.113803] ? lock_is_held_type+0xaa/0x120 [ 2018.117993] vfs_write+0x324/0x4f0 [ 2018.121403] ksys_pwrite64+0x65/0xa0 [ 2018.124979] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2018.130118] do_syscall_64+0x38/0x80 [ 2018.133697] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2018.138751] RIP: 0033:0x7f48f0eeb93b [ 2018.142329] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2018.161074] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2018.168639] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2018.175764] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2018.182898] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2018.190030] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2018.197163] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2018.204303] [ 2018.206489] irq event stamp: 12761705 [ 2018.210154] hardirqs last enabled at (12761715): [] console_unlock+0x106/0x120 [ 2018.219022] hardirqs last disabled at (12761724): [] console_unlock+0xeb/0x120 [ 2018.227800] softirqs last enabled at (12761692): [] __do_softirq+0x2be/0x371 [ 2018.236496] softirqs last disabled at (12761687): [] irq_exit_rcu+0xb0/0xd0 [ 2018.245012] ---[ end trace 0000000000000000 ]--- [ 2018.249634] ------------[ cut here ]------------ [ 2018.254261] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2018.265911] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2018.274613] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2018.330108] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2018.339233] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2018.346712] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2018.352226] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2018.357719] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2018.376464] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2018.381694] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2018.388824] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2018.395959] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2018.403090] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881e4d0b838 [ 2018.410222] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2018.417356] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2018.425441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2018.431188] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2018.438321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2018.445452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2018.452583] Call Trace: [ 2018.455038] [ 2018.457144] ? __warn+0x85/0x180 [ 2018.460382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.465267] ? report_bug+0xfc/0x1d0 [ 2018.468846] ? prb_read_valid+0x1b/0x20 [ 2018.472694] ? handle_bug+0x44/0x70 [ 2018.476186] ? exc_invalid_op+0x17/0x60 [ 2018.480026] ? asm_exc_invalid_op+0x1a/0x20 [ 2018.484222] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.489101] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.493980] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2018.498879] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2018.503526] ceph_fill_trace+0x143/0xf30 [ceph] [ 2018.508071] ? __mutex_lock+0x97/0xce0 [ 2018.511824] ? __mutex_lock+0xaec/0xce0 [ 2018.515664] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2018.520222] ? lock_downgrade+0xfd/0x250 [ 2018.524168] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2018.528565] ceph_con_process_message+0x74/0x150 [libceph] [ 2018.534063] ? calc_signature+0xdb/0x100 [libceph] [ 2018.538889] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2018.544228] ? process_scheduled_works+0x22a/0x510 [ 2018.549029] ? process_scheduled_works+0x22a/0x510 [ 2018.553823] ceph_con_workfn+0x344/0x750 [libceph] [ 2018.558641] ? process_scheduled_works+0x22a/0x510 [ 2018.563433] process_scheduled_works+0x2a4/0x510 [ 2018.568064] worker_thread+0x188/0x340 [ 2018.571817] ? rescuer_thread+0x310/0x310 [ 2018.575830] kthread+0xf5/0x130 [ 2018.578982] ? kthread_complete_and_exit+0x20/0x20 [ 2018.583780] ret_from_fork+0x31/0x40 [ 2018.587365] ? kthread_complete_and_exit+0x20/0x20 [ 2018.592159] ret_from_fork_asm+0x11/0x20 [ 2018.596105] [ 2018.598301] irq event stamp: 8784129 [ 2018.601880] hardirqs last enabled at (8784139): [] console_unlock+0x106/0x120 [ 2018.610661] hardirqs last disabled at (8784148): [] console_unlock+0xeb/0x120 [ 2018.619360] softirqs last enabled at (8784096): [] __do_softirq+0x2be/0x371 [ 2018.627969] softirqs last disabled at (8784091): [] irq_exit_rcu+0xb0/0xd0 [ 2018.636408] ---[ end trace 0000000000000000 ]--- [ 2018.641030] ------------[ cut here ]------------ [ 2018.645644] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2018.657302] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2018.666257] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2018.721741] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2018.730696] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2018.738173] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2018.743665] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2018.762406] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2018.767637] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2018.774763] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2018.781888] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2018.789021] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d0aefcb0 [ 2018.796156] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2018.803286] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2018.811373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2018.817119] CR2: 00007fb39000f068 CR3: 0000000120c1e006 CR4: 00000000003706e0 [ 2018.824254] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2018.831384] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2018.838517] Call Trace: [ 2018.840972] [ 2018.843077] ? __warn+0x85/0x180 [ 2018.846308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.851190] ? report_bug+0xfc/0x1d0 [ 2018.854766] ? prb_read_valid+0x1b/0x20 [ 2018.858608] ? handle_bug+0x44/0x70 [ 2018.862100] ? exc_invalid_op+0x17/0x60 [ 2018.865938] ? asm_exc_invalid_op+0x1a/0x20 [ 2018.870132] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.875013] ? __list_add_valid_or_report+0x7d/0xa0 [ 2018.879902] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2018.884806] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2018.889961] ceph_write_iter+0x5aa/0x770 [ceph] [ 2018.894520] ? lock_is_held_type+0xaa/0x120 [ 2018.898712] vfs_write+0x324/0x4f0 [ 2018.902119] ksys_pwrite64+0x65/0xa0 [ 2018.905695] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2018.910839] do_syscall_64+0x38/0x80 [ 2018.914423] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2018.919476] RIP: 0033:0x7f48f0eeb93b [ 2018.923054] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2018.941802] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2018.949372] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2018.956499] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2018.963642] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2018.970774] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2018.977907] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2018.985050] [ 2018.987239] irq event stamp: 12764579 [ 2018.990897] hardirqs last enabled at (12764589): [] console_unlock+0x106/0x120 [ 2018.999764] hardirqs last disabled at (12764598): [] console_unlock+0xeb/0x120 [ 2019.008546] softirqs last enabled at (12764612): [] __do_softirq+0x2be/0x371 [ 2019.017236] softirqs last disabled at (12764607): [] irq_exit_rcu+0xb0/0xd0 [ 2019.025755] ---[ end trace 0000000000000000 ]--- [ 2019.030376] ------------[ cut here ]------------ [ 2019.034996] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2019.046650] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2019.055606] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2019.111146] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2019.120105] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2019.127585] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2019.133070] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2019.151817] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2019.157042] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2019.164175] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2019.171309] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2019.178442] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d372e8 [ 2019.185575] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2019.192708] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2019.200793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2019.206540] CR2: 00007fb38f6dc000 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2019.213672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2019.220804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2019.227938] Call Trace: [ 2019.230390] [ 2019.232499] ? __warn+0x85/0x180 [ 2019.235740] ? __list_add_valid_or_report+0x7d/0xa0 [ 2019.240629] ? report_bug+0xfc/0x1d0 [ 2019.244216] ? prb_read_valid+0x1b/0x20 [ 2019.248066] ? handle_bug+0x44/0x70 [ 2019.251565] ? exc_invalid_op+0x17/0x60 [ 2019.255404] ? asm_exc_invalid_op+0x1a/0x20 [ 2019.259600] ? __list_add_valid_or_report+0x7d/0xa0 [ 2019.264490] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2019.269395] try_get_cap_refs+0x433/0x880 [ceph] [ 2019.274039] ? _raw_spin_unlock+0x23/0x40 [ 2019.278066] __ceph_get_caps+0xf8/0x760 [ceph] [ 2019.282531] ? __vfs_getxattr+0x8d/0xb0 [ 2019.286376] ? dentry_needs_remove_privs+0x31/0x50 [ 2019.291173] ? __file_remove_privs+0x89/0x110 [ 2019.295531] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2019.300257] ? lock_release+0xc5/0x260 [ 2019.304021] ? ceph_write_iter+0x260/0x770 [ceph] [ 2019.308741] ceph_write_iter+0x260/0x770 [ceph] [ 2019.313300] ? lock_is_held_type+0xaa/0x120 [ 2019.317498] vfs_write+0x324/0x4f0 [ 2019.320919] ksys_pwrite64+0x65/0xa0 [ 2019.324504] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2019.329647] do_syscall_64+0x38/0x80 [ 2019.333233] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2019.338293] RIP: 0033:0x7f48f0eeb93b [ 2019.341883] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2019.360628] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2019.368200] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2019.375336] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2019.382469] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2019.389621] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2019.396759] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2019.403905] [ 2019.406102] irq event stamp: 12794233 [ 2019.409768] hardirqs last enabled at (12794243): [] console_unlock+0x106/0x120 [ 2019.418633] hardirqs last disabled at (12794252): [] console_unlock+0xeb/0x120 [ 2019.427413] softirqs last enabled at (12794200): [] __do_softirq+0x2be/0x371 [ 2019.436105] softirqs last disabled at (12794179): [] irq_exit_rcu+0xb0/0xd0 [ 2019.444627] ---[ end trace 0000000000000000 ]--- [ 2019.449246] ------------[ cut here ]------------ [ 2019.454015] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2019.465672] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2019.474630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2019.530110] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2019.539056] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2019.546536] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2019.552031] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2019.570776] RSP: 0018:ffffc90001fffaa8 EFLAGS: 00010282 [ 2019.576004] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2019.583136] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2019.590268] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2019.597402] R10: 0000000000000001 R11: ffffc90001fff938 R12: ffff8881e4f8e920 [ 2019.604534] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2019.611666] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2019.619752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2019.625489] CR2: 00007fb38f6e0000 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2019.632616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2019.639748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2019.646880] Call Trace: [ 2019.649334] [ 2019.651442] ? __warn+0x85/0x180 [ 2019.654673] ? __list_add_valid_or_report+0x7d/0xa0 [ 2019.659553] ? report_bug+0xfc/0x1d0 [ 2019.663132] ? prb_read_valid+0x1b/0x20 [ 2019.666972] ? handle_bug+0x44/0x70 [ 2019.670463] ? exc_invalid_op+0x17/0x60 [ 2019.674303] ? asm_exc_invalid_op+0x1a/0x20 [ 2019.678493] ? __list_add_valid_or_report+0x7d/0xa0 [ 2019.683368] ? __list_add_valid_or_report+0x7d/0xa0 [ 2019.688247] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2019.693139] ceph_open+0x16b/0x440 [ceph] [ 2019.697161] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2019.702306] do_dentry_open+0x172/0x530 [ 2019.706146] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2019.710782] lookup_open+0x37b/0x630 [ 2019.714363] ? verify_cpu+0x20/0x100 [ 2019.717943] ? verify_cpu+0x20/0x100 [ 2019.721521] path_openat+0x25a/0x9d0 [ 2019.725103] do_filp_open+0xb8/0x120 [ 2019.728685] ? lock_release+0xc5/0x260 [ 2019.732432] ? _raw_spin_unlock+0x23/0x40 [ 2019.736442] ? alloc_fd+0x13f/0x200 [ 2019.739938] do_sys_openat2+0x249/0x290 [ 2019.743778] do_sys_open+0x4f/0x80 [ 2019.747185] do_syscall_64+0x38/0x80 [ 2019.750762] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2019.755814] RIP: 0033:0x7f48f0eed2f2 [ 2019.759394] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2019.778141] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2019.785726] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2019.792859] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2019.799989] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000003 [ 2019.807121] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2019.814254] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a08 [ 2019.821393] [ 2019.823580] irq event stamp: 12763101 [ 2019.827245] hardirqs last enabled at (12763111): [] console_unlock+0x106/0x120 [ 2019.836112] hardirqs last disabled at (12763120): [] console_unlock+0xeb/0x120 [ 2019.844890] softirqs last enabled at (12763048): [] __do_softirq+0x2be/0x371 [ 2019.853584] softirqs last disabled at (12763139): [] irq_exit_rcu+0xb0/0xd0 [ 2019.862104] ---[ end trace 0000000000000000 ]--- [ 2019.866724] ------------[ cut here ]------------ [ 2019.871346] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2019.882997] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2019.891957] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2019.947449] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2019.956405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2019.963890] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2019.969385] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2019.988131] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2019.993357] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2020.000487] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2020.007622] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2020.014758] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d0aefcb0 [ 2020.021888] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2020.029019] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2020.037107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2020.042851] CR2: 0000563ac8c0cbf8 CR3: 0000000120c1e004 CR4: 00000000003706f0 [ 2020.049986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2020.057118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2020.064254] Call Trace: [ 2020.066705] [ 2020.068812] ? __warn+0x85/0x180 [ 2020.072051] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.076933] ? report_bug+0xfc/0x1d0 [ 2020.080522] ? prb_read_valid+0x1b/0x20 [ 2020.084373] ? handle_bug+0x44/0x70 [ 2020.087869] ? exc_invalid_op+0x17/0x60 [ 2020.091709] ? asm_exc_invalid_op+0x1a/0x20 [ 2020.095910] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.100796] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2020.105701] try_get_cap_refs+0x433/0x880 [ceph] [ 2020.110344] ? _raw_spin_unlock+0x23/0x40 [ 2020.114368] __ceph_get_caps+0xf8/0x760 [ceph] [ 2020.118839] ? __vfs_getxattr+0x8d/0xb0 [ 2020.122686] ? dentry_needs_remove_privs+0x31/0x50 [ 2020.127490] ? __file_remove_privs+0x89/0x110 [ 2020.131853] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2020.136578] ? lock_release+0xc5/0x260 [ 2020.140342] ? ceph_write_iter+0x260/0x770 [ceph] [ 2020.145075] ceph_write_iter+0x260/0x770 [ceph] [ 2020.149630] ? lock_is_held_type+0xaa/0x120 [ 2020.153832] vfs_write+0x324/0x4f0 [ 2020.157247] ksys_pwrite64+0x65/0xa0 [ 2020.160828] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2020.165977] do_syscall_64+0x38/0x80 [ 2020.169566] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2020.174623] RIP: 0033:0x7f48f0eeb93b [ 2020.178205] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2020.196949] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2020.204521] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2020.211663] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2020.218799] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2020.225931] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2020.233066] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2020.240217] [ 2020.242408] irq event stamp: 12767495 [ 2020.246072] hardirqs last enabled at (12767505): [] console_unlock+0x106/0x120 [ 2020.254940] hardirqs last disabled at (12767514): [] console_unlock+0xeb/0x120 [ 2020.263725] softirqs last enabled at (12767442): [] __do_softirq+0x2be/0x371 [ 2020.272421] softirqs last disabled at (12767437): [] irq_exit_rcu+0xb0/0xd0 [ 2020.280956] ---[ end trace 0000000000000000 ]--- [ 2020.285581] ------------[ cut here ]------------ [ 2020.290201] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2020.301852] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2020.310802] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2020.366264] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2020.375214] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2020.382693] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2020.388180] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2020.406925] RSP: 0018:ffffc90001fffc88 EFLAGS: 00010286 [ 2020.412151] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2020.419276] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2020.426400] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2020.433525] R10: 0000000000000001 R11: ffffc90001fffb18 R12: ffff8881e4f8e920 [ 2020.440656] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2020.447788] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2020.455869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2020.461614] CR2: 00007fb38f6dc000 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2020.468745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2020.475869] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2020.482993] Call Trace: [ 2020.485440] [ 2020.487546] ? __warn+0x85/0x180 [ 2020.490779] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.495664] ? report_bug+0xfc/0x1d0 [ 2020.499236] ? prb_read_valid+0x1b/0x20 [ 2020.503080] ? handle_bug+0x44/0x70 [ 2020.506568] ? exc_invalid_op+0x17/0x60 [ 2020.510399] ? asm_exc_invalid_op+0x1a/0x20 [ 2020.514590] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.519467] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.524346] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2020.529235] ceph_open+0x16b/0x440 [ceph] [ 2020.533256] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2020.538413] do_dentry_open+0x172/0x530 [ 2020.542260] path_openat+0x743/0x9d0 [ 2020.545843] do_filp_open+0xb8/0x120 [ 2020.549421] ? lock_release+0xc5/0x260 [ 2020.553174] ? _raw_spin_unlock+0x23/0x40 [ 2020.557183] ? alloc_fd+0x13f/0x200 [ 2020.560680] do_sys_openat2+0x249/0x290 [ 2020.564517] do_sys_open+0x4f/0x80 [ 2020.567923] do_syscall_64+0x38/0x80 [ 2020.571502] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2020.576552] RIP: 0033:0x7f48f0eed2f2 [ 2020.580135] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2020.598879] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2020.606445] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2020.613576] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2020.620701] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2020.627828] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2020.634960] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a09 [ 2020.642098] [ 2020.644285] irq event stamp: 12765607 [ 2020.647951] hardirqs last enabled at (12765617): [] console_unlock+0x106/0x120 [ 2020.656817] hardirqs last disabled at (12765626): [] console_unlock+0xeb/0x120 [ 2020.665595] softirqs last enabled at (12765574): [] __do_softirq+0x2be/0x371 [ 2020.674290] softirqs last disabled at (12765569): [] irq_exit_rcu+0xb0/0xd0 [ 2020.682808] ---[ end trace 0000000000000000 ]--- [ 2020.687420] ------------[ cut here ]------------ [ 2020.692041] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2020.703691] WARNING: CPU: 3 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2020.712651] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2020.768179] CPU: 3 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2020.777134] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2020.784611] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2020.790098] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2020.808848] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2020.814078] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2020.821213] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2020.828345] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2020.835478] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d372e8 [ 2020.842609] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2020.849747] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2020.857837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2020.863585] CR2: 00007f9bd551c000 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2020.870717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2020.877849] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2020.884984] Call Trace: [ 2020.887437] [ 2020.889544] ? __warn+0x85/0x180 [ 2020.892774] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.897656] ? report_bug+0xfc/0x1d0 [ 2020.901234] ? prb_read_valid+0x1b/0x20 [ 2020.905076] ? handle_bug+0x44/0x70 [ 2020.908575] ? exc_invalid_op+0x17/0x60 [ 2020.912414] ? asm_exc_invalid_op+0x1a/0x20 [ 2020.916609] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.921491] ? __list_add_valid_or_report+0x7d/0xa0 [ 2020.926377] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2020.931276] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2020.936435] ceph_write_iter+0x5aa/0x770 [ceph] [ 2020.940996] ? lock_is_held_type+0xaa/0x120 [ 2020.945188] vfs_write+0x324/0x4f0 [ 2020.948608] ksys_pwrite64+0x65/0xa0 [ 2020.952195] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2020.957336] do_syscall_64+0x38/0x80 [ 2020.960921] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2020.965974] RIP: 0033:0x7f48f0eeb93b [ 2020.969555] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2020.988300] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2020.995865] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2021.002999] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2021.010132] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2021.017263] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2021.024397] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2021.031544] [ 2021.033741] irq event stamp: 12798207 [ 2021.037405] hardirqs last enabled at (12798217): [] console_unlock+0x106/0x120 [ 2021.046273] hardirqs last disabled at (12798226): [] console_unlock+0xeb/0x120 [ 2021.055052] softirqs last enabled at (12797858): [] __do_softirq+0x2be/0x371 [ 2021.063745] softirqs last disabled at (12797853): [] irq_exit_rcu+0xb0/0xd0 [ 2021.072263] ---[ end trace 0000000000000000 ]--- [ 2021.076884] ------------[ cut here ]------------ [ 2021.081502] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2021.093149] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2021.102104] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2021.157573] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2021.166527] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2021.174004] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2021.179491] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2021.198235] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2021.203462] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2021.210594] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2021.217738] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2021.224871] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d0aefcb0 [ 2021.232003] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2021.239134] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2021.247222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2021.252966] CR2: 00007f9bd5527000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2021.260101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2021.267231] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2021.274369] Call Trace: [ 2021.276817] [ 2021.278925] ? __warn+0x85/0x180 [ 2021.282158] ? __list_add_valid_or_report+0x7d/0xa0 [ 2021.287048] ? report_bug+0xfc/0x1d0 [ 2021.290628] ? prb_read_valid+0x1b/0x20 [ 2021.294476] ? handle_bug+0x44/0x70 [ 2021.297975] ? exc_invalid_op+0x17/0x60 [ 2021.301819] ? asm_exc_invalid_op+0x1a/0x20 [ 2021.306016] ? __list_add_valid_or_report+0x7d/0xa0 [ 2021.310901] ? __list_add_valid_or_report+0x7d/0xa0 [ 2021.315783] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2021.320688] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2021.325843] ceph_write_iter+0x5aa/0x770 [ceph] [ 2021.330405] ? lock_is_held_type+0xaa/0x120 [ 2021.334597] vfs_write+0x324/0x4f0 [ 2021.338016] ksys_pwrite64+0x65/0xa0 [ 2021.341593] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2021.346738] do_syscall_64+0x38/0x80 [ 2021.350321] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2021.355378] RIP: 0033:0x7f48f0eeb93b [ 2021.358961] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2021.377715] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2021.385285] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2021.392431] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2021.399569] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2021.406701] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2021.413830] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2021.420977] [ 2021.423173] irq event stamp: 12770661 [ 2021.426840] hardirqs last enabled at (12770671): [] console_unlock+0x106/0x120 [ 2021.435707] hardirqs last disabled at (12770686): [] console_unlock+0xeb/0x120 [ 2021.444488] softirqs last enabled at (12770684): [] __do_softirq+0x2be/0x371 [ 2021.453187] softirqs last disabled at (12770679): [] irq_exit_rcu+0xb0/0xd0 [ 2021.461707] ---[ end trace 0000000000000000 ]--- [ 2021.466326] ------------[ cut here ]------------ [ 2021.471011] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2021.482651] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2021.491607] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2021.547088] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2021.556034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2021.563515] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2021.569001] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2021.587746] RSP: 0018:ffffc90001fffbe0 EFLAGS: 00010286 [ 2021.592972] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2021.600107] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2021.607239] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2021.614373] R10: 0000000000000001 R11: ffffc90001fffa70 R12: ffff8881e4f8e920 [ 2021.621505] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2021.628637] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2021.636722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2021.642469] CR2: 00007fb38f6dc000 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2021.649602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2021.656734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2021.663858] Call Trace: [ 2021.666302] [ 2021.668401] ? __warn+0x85/0x180 [ 2021.671635] ? __list_add_valid_or_report+0x7d/0xa0 [ 2021.676513] ? report_bug+0xfc/0x1d0 [ 2021.680093] ? prb_read_valid+0x1b/0x20 [ 2021.683933] ? handle_bug+0x44/0x70 [ 2021.687429] ? exc_invalid_op+0x17/0x60 [ 2021.691275] ? asm_exc_invalid_op+0x1a/0x20 [ 2021.695465] ? __list_add_valid_or_report+0x7d/0xa0 [ 2021.700348] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2021.705237] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2021.709863] ? _raw_spin_unlock+0x23/0x40 [ 2021.713879] __ceph_get_caps+0xf8/0x760 [ceph] [ 2021.718332] ? __vfs_getxattr+0x8d/0xb0 [ 2021.722173] ? dentry_needs_remove_privs+0x31/0x50 [ 2021.726963] ? __file_remove_privs+0x89/0x110 [ 2021.731322] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2021.736037] ? lock_release+0xc5/0x260 [ 2021.739790] ? ceph_write_iter+0x260/0x770 [ceph] [ 2021.744506] ceph_write_iter+0x260/0x770 [ceph] [ 2021.749050] ? lock_is_held_type+0xaa/0x120 [ 2021.753234] vfs_write+0x324/0x4f0 [ 2021.756642] ksys_pwrite64+0x65/0xa0 [ 2021.760216] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2021.765358] do_syscall_64+0x38/0x80 [ 2021.768936] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2021.773989] RIP: 0033:0x7f48f0eeb93b [ 2021.777568] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2021.796316] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2021.803881] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2021.811019] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000014 [ 2021.818148] RBP: 0000000000002a09 R08: 0000000000000001 R09: 0000000002074010 [ 2021.825278] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183e368 [ 2021.832402] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2021.839542] [ 2021.841729] irq event stamp: 12768493 [ 2021.845395] hardirqs last enabled at (12768503): [] console_unlock+0x106/0x120 [ 2021.854260] hardirqs last disabled at (12768512): [] console_unlock+0xeb/0x120 [ 2021.863039] softirqs last enabled at (12768450): [] __do_softirq+0x2be/0x371 [ 2021.871733] softirqs last disabled at (12768445): [] irq_exit_rcu+0xb0/0xd0 [ 2021.880252] ---[ end trace 0000000000000000 ]--- [ 2021.884872] ------------[ cut here ]------------ [ 2021.889492] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2021.901151] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2021.910120] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2021.965596] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2021.974566] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2021.982048] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2021.987541] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2022.006296] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2022.011523] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2022.018657] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2022.025798] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2022.032929] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d0aefcb0 [ 2022.040066] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2022.047208] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2022.055300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2022.061044] CR2: 00007f9bd5527000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2022.068177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2022.075310] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2022.082448] Call Trace: [ 2022.084906] [ 2022.087013] ? __warn+0x85/0x180 [ 2022.090252] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.095135] ? report_bug+0xfc/0x1d0 [ 2022.098719] ? prb_read_valid+0x1b/0x20 [ 2022.102564] ? handle_bug+0x44/0x70 [ 2022.106063] ? exc_invalid_op+0x17/0x60 [ 2022.109901] ? asm_exc_invalid_op+0x1a/0x20 [ 2022.114094] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.118977] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2022.123874] try_get_cap_refs+0x433/0x880 [ceph] [ 2022.128522] ? _raw_spin_unlock+0x23/0x40 [ 2022.132545] __ceph_get_caps+0xf8/0x760 [ceph] [ 2022.137014] ? __vfs_getxattr+0x8d/0xb0 [ 2022.140861] ? dentry_needs_remove_privs+0x31/0x50 [ 2022.145660] ? __file_remove_privs+0x89/0x110 [ 2022.150017] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2022.154745] ? lock_release+0xc5/0x260 [ 2022.158511] ? ceph_write_iter+0x260/0x770 [ceph] [ 2022.163237] ceph_write_iter+0x260/0x770 [ceph] [ 2022.167794] ? lock_is_held_type+0xaa/0x120 [ 2022.171989] vfs_write+0x324/0x4f0 [ 2022.175407] ksys_pwrite64+0x65/0xa0 [ 2022.178992] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2022.184132] do_syscall_64+0x38/0x80 [ 2022.187713] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2022.192770] RIP: 0033:0x7f48f0eeb93b [ 2022.196352] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2022.215096] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2022.222668] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2022.229816] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2022.236949] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2022.244080] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2022.251213] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2022.258357] [ 2022.260546] irq event stamp: 12772817 [ 2022.264212] hardirqs last enabled at (12772827): [] console_unlock+0x106/0x120 [ 2022.273079] hardirqs last disabled at (12772836): [] console_unlock+0xeb/0x120 [ 2022.281857] softirqs last enabled at (12772774): [] __do_softirq+0x2be/0x371 [ 2022.290553] softirqs last disabled at (12772769): [] irq_exit_rcu+0xb0/0xd0 [ 2022.299077] ---[ end trace 0000000000000000 ]--- [ 2022.303704] ------------[ cut here ]------------ [ 2022.308327] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2022.319975] WARNING: CPU: 1 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2022.328938] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2022.384466] CPU: 1 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2022.393421] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2022.400900] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2022.406393] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2022.425140] RSP: 0018:ffffc90001f7faa8 EFLAGS: 00010282 [ 2022.430375] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2022.437506] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2022.444641] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2022.451774] R10: 0000000000000001 R11: ffffc90001f7f938 R12: ffff8881e4d0b838 [ 2022.458905] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2022.466041] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2022.474132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2022.479878] CR2: 00007f9bd550e000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2022.487012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2022.494145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2022.501279] Call Trace: [ 2022.503732] [ 2022.505839] ? __warn+0x85/0x180 [ 2022.509080] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.513967] ? report_bug+0xfc/0x1d0 [ 2022.517554] ? prb_read_valid+0x1b/0x20 [ 2022.521397] ? handle_bug+0x44/0x70 [ 2022.524894] ? exc_invalid_op+0x17/0x60 [ 2022.528736] ? asm_exc_invalid_op+0x1a/0x20 [ 2022.532939] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.537829] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.542715] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2022.547617] ceph_open+0x16b/0x440 [ceph] [ 2022.551656] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2022.556821] do_dentry_open+0x172/0x530 [ 2022.560667] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2022.565319] lookup_open+0x37b/0x630 [ 2022.568914] ? verify_cpu+0x20/0x100 [ 2022.572498] ? verify_cpu+0x20/0x100 [ 2022.576087] path_openat+0x25a/0x9d0 [ 2022.579679] do_filp_open+0xb8/0x120 [ 2022.583268] ? lock_release+0xc5/0x260 [ 2022.587034] ? _raw_spin_unlock+0x23/0x40 [ 2022.591050] ? alloc_fd+0x13f/0x200 [ 2022.594555] do_sys_openat2+0x249/0x290 [ 2022.598403] do_sys_open+0x4f/0x80 [ 2022.601817] do_syscall_64+0x38/0x80 [ 2022.605403] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2022.610463] RIP: 0033:0x7f48f0eed2f2 [ 2022.614051] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2022.632798] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2022.640371] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2022.647503] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2022.654641] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2022.661778] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2022.668912] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a15 [ 2022.676061] [ 2022.678255] irq event stamp: 12810513 [ 2022.681921] hardirqs last enabled at (12810523): [] console_unlock+0x106/0x120 [ 2022.690796] hardirqs last disabled at (12810532): [] console_unlock+0xeb/0x120 [ 2022.699574] softirqs last enabled at (12810470): [] __do_softirq+0x2be/0x371 [ 2022.708268] softirqs last disabled at (12810465): [] irq_exit_rcu+0xb0/0xd0 [ 2022.716787] ---[ end trace 0000000000000000 ]--- [ 2022.721410] ------------[ cut here ]------------ [ 2022.726037] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2022.737708] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2022.746408] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2022.801896] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2022.811017] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2022.818497] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2022.824009] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2022.829494] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2022.848242] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2022.853470] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2022.860601] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2022.867734] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2022.874865] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881d1008750 [ 2022.881988] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2022.889114] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2022.897198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2022.902936] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2022.910060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2022.917185] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2022.924312] Call Trace: [ 2022.926765] [ 2022.928869] ? __warn+0x85/0x180 [ 2022.932104] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.936980] ? report_bug+0xfc/0x1d0 [ 2022.940559] ? prb_read_valid+0x1b/0x20 [ 2022.944402] ? handle_bug+0x44/0x70 [ 2022.947891] ? exc_invalid_op+0x17/0x60 [ 2022.951722] ? asm_exc_invalid_op+0x1a/0x20 [ 2022.955913] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.960789] ? __list_add_valid_or_report+0x7d/0xa0 [ 2022.965668] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2022.970567] ceph_add_cap+0x20b/0x620 [ceph] [ 2022.974853] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2022.979580] ceph_fill_trace+0x143/0xf30 [ceph] [ 2022.984119] ? __mutex_lock+0x97/0xce0 [ 2022.987871] ? __mutex_lock+0xaec/0xce0 [ 2022.991708] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2022.996257] ? lock_downgrade+0xfd/0x250 [ 2023.000190] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2023.004574] ceph_con_process_message+0x74/0x150 [libceph] [ 2023.010070] ? calc_signature+0xdb/0x100 [libceph] [ 2023.014876] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2023.020210] ? process_scheduled_works+0x22a/0x510 [ 2023.025007] ? process_scheduled_works+0x22a/0x510 [ 2023.029793] ceph_con_workfn+0x344/0x750 [libceph] [ 2023.034600] ? process_scheduled_works+0x22a/0x510 [ 2023.039386] process_scheduled_works+0x2a4/0x510 [ 2023.044028] worker_thread+0x188/0x340 [ 2023.047793] ? rescuer_thread+0x310/0x310 [ 2023.051813] kthread+0xf5/0x130 [ 2023.054969] ? kthread_complete_and_exit+0x20/0x20 [ 2023.059775] ret_from_fork+0x31/0x40 [ 2023.063356] ? kthread_complete_and_exit+0x20/0x20 [ 2023.068158] ret_from_fork_asm+0x11/0x20 [ 2023.072114] [ 2023.074311] irq event stamp: 8795437 [ 2023.077891] hardirqs last enabled at (8795453): [] console_unlock+0x106/0x120 [ 2023.086678] hardirqs last disabled at (8795468): [] console_unlock+0xeb/0x120 [ 2023.095370] softirqs last enabled at (8795482): [] __do_softirq+0x2be/0x371 [ 2023.103975] softirqs last disabled at (8795493): [] irq_exit_rcu+0xb0/0xd0 [ 2023.112409] ---[ end trace 0000000000000000 ]--- [ 2023.117029] ------------[ cut here ]------------ [ 2023.121648] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2023.133303] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2023.142257] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2023.197788] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2023.206738] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2023.214220] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2023.219704] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2023.238456] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2023.243684] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2023.250819] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2023.257950] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2023.265090] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d0aefcb0 [ 2023.272226] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2023.279358] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2023.287444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2023.293190] CR2: 00007f9bd553f000 CR3: 0000000120c1e002 CR4: 00000000003706f0 [ 2023.300326] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2023.307456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2023.314588] Call Trace: [ 2023.317041] [ 2023.319148] ? __warn+0x85/0x180 [ 2023.322382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2023.327261] ? report_bug+0xfc/0x1d0 [ 2023.330838] ? prb_read_valid+0x1b/0x20 [ 2023.334681] ? handle_bug+0x44/0x70 [ 2023.338172] ? exc_invalid_op+0x17/0x60 [ 2023.342013] ? asm_exc_invalid_op+0x1a/0x20 [ 2023.346206] ? __list_add_valid_or_report+0x7d/0xa0 [ 2023.351088] ? __list_add_valid_or_report+0x7d/0xa0 [ 2023.355973] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2023.360875] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2023.366046] ceph_write_iter+0x5aa/0x770 [ceph] [ 2023.370603] ? lock_is_held_type+0xaa/0x120 [ 2023.374796] vfs_write+0x324/0x4f0 [ 2023.378214] ksys_pwrite64+0x65/0xa0 [ 2023.381800] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2023.386943] do_syscall_64+0x38/0x80 [ 2023.390529] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2023.395579] RIP: 0033:0x7f48f0eeb93b [ 2023.399165] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2023.417914] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2023.425481] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2023.432615] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2023.439747] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2023.446878] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2023.454017] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2023.461168] [ 2023.463365] irq event stamp: 12775625 [ 2023.467031] hardirqs last enabled at (12775635): [] console_unlock+0x106/0x120 [ 2023.475896] hardirqs last disabled at (12775644): [] console_unlock+0xeb/0x120 [ 2023.484675] softirqs last enabled at (12775602): [] __do_softirq+0x2be/0x371 [ 2023.493368] softirqs last disabled at (12775595): [] irq_exit_rcu+0xb0/0xd0 [ 2023.501889] ---[ end trace 0000000000000000 ]--- [ 2023.506509] ------------[ cut here ]------------ [ 2023.511136] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2023.522790] WARNING: CPU: 3 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2023.531750] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2023.587285] CPU: 3 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2023.596245] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2023.603725] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2023.609224] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2023.627973] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2023.633198] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2023.640331] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2023.647464] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2023.654596] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d372e8 [ 2023.661732] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2023.668864] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2023.676948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2023.682695] CR2: 00007f9bd551c000 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2023.689827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2023.696961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2023.704093] Call Trace: [ 2023.706547] [ 2023.708653] ? __warn+0x85/0x180 [ 2023.711886] ? __list_add_valid_or_report+0x7d/0xa0 [ 2023.716765] ? report_bug+0xfc/0x1d0 [ 2023.720344] ? prb_read_valid+0x1b/0x20 [ 2023.724187] ? handle_bug+0x44/0x70 [ 2023.727686] ? exc_invalid_op+0x17/0x60 [ 2023.731524] ? asm_exc_invalid_op+0x1a/0x20 [ 2023.735726] ? __list_add_valid_or_report+0x7d/0xa0 [ 2023.740607] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2023.745509] try_get_cap_refs+0x433/0x880 [ceph] [ 2023.750153] ? _raw_spin_unlock+0x23/0x40 [ 2023.754179] __ceph_get_caps+0xf8/0x760 [ceph] [ 2023.758646] ? __vfs_getxattr+0x8d/0xb0 [ 2023.762497] ? dentry_needs_remove_privs+0x31/0x50 [ 2023.767292] ? __file_remove_privs+0x89/0x110 [ 2023.771653] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2023.776378] ? lock_release+0xc5/0x260 [ 2023.780141] ? ceph_write_iter+0x260/0x770 [ceph] [ 2023.784860] ceph_write_iter+0x260/0x770 [ceph] [ 2023.789419] ? lock_is_held_type+0xaa/0x120 [ 2023.793611] vfs_write+0x324/0x4f0 [ 2023.797032] ksys_pwrite64+0x65/0xa0 [ 2023.800615] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2023.805760] do_syscall_64+0x38/0x80 [ 2023.809344] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2023.814397] RIP: 0033:0x7f48f0eeb93b [ 2023.817976] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2023.836724] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2023.844297] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2023.851430] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2023.858562] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2023.865694] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2023.872828] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2023.879974] [ 2023.882170] irq event stamp: 12804301 [ 2023.885836] hardirqs last enabled at (12804311): [] console_unlock+0x106/0x120 [ 2023.894701] hardirqs last disabled at (12804320): [] console_unlock+0xeb/0x120 [ 2023.903481] softirqs last enabled at (12803870): [] __do_softirq+0x2be/0x371 [ 2023.912175] softirqs last disabled at (12803865): [] irq_exit_rcu+0xb0/0xd0 [ 2023.920698] ---[ end trace 0000000000000000 ]--- [ 2023.925317] ------------[ cut here ]------------ [ 2023.929941] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2023.941592] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2023.950552] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2024.006089] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2024.015047] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2024.022532] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2024.028025] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2024.046774] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2024.052005] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2024.059139] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2024.066272] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2024.073404] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d0aefcb0 [ 2024.080539] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2024.087670] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2024.095759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.101502] CR2: 00007f9bd553f000 CR3: 0000000120c1e002 CR4: 00000000003706f0 [ 2024.108635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2024.115771] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2024.122900] Call Trace: [ 2024.125354] [ 2024.127459] ? __warn+0x85/0x180 [ 2024.130693] ? __list_add_valid_or_report+0x7d/0xa0 [ 2024.135572] ? report_bug+0xfc/0x1d0 [ 2024.139151] ? prb_read_valid+0x1b/0x20 [ 2024.142995] ? handle_bug+0x44/0x70 [ 2024.146491] ? exc_invalid_op+0x17/0x60 [ 2024.150334] ? asm_exc_invalid_op+0x1a/0x20 [ 2024.154537] ? __list_add_valid_or_report+0x7d/0xa0 [ 2024.159425] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2024.164323] try_get_cap_refs+0x433/0x880 [ceph] [ 2024.168959] ? _raw_spin_unlock+0x23/0x40 [ 2024.172986] __ceph_get_caps+0xf8/0x760 [ceph] [ 2024.177452] ? __vfs_getxattr+0x8d/0xb0 [ 2024.181298] ? dentry_needs_remove_privs+0x31/0x50 [ 2024.186091] ? __file_remove_privs+0x89/0x110 [ 2024.190449] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2024.195176] ? lock_release+0xc5/0x260 [ 2024.198932] ? ceph_write_iter+0x260/0x770 [ceph] [ 2024.203649] ceph_write_iter+0x260/0x770 [ceph] [ 2024.208208] ? lock_is_held_type+0xaa/0x120 [ 2024.212404] vfs_write+0x324/0x4f0 [ 2024.215821] ksys_pwrite64+0x65/0xa0 [ 2024.219406] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2024.224552] do_syscall_64+0x38/0x80 [ 2024.228136] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2024.233195] RIP: 0033:0x7f48f0eeb93b [ 2024.236775] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2024.255523] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2024.263096] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2024.270227] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2024.277362] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2024.284494] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2024.291629] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2024.298773] [ 2024.300970] irq event stamp: 12777723 [ 2024.304636] hardirqs last enabled at (12777733): [] console_unlock+0x106/0x120 [ 2024.313502] hardirqs last disabled at (12777742): [] console_unlock+0xeb/0x120 [ 2024.322283] softirqs last enabled at (12777758): [] __do_softirq+0x2be/0x371 [ 2024.330981] softirqs last disabled at (12777751): [] irq_exit_rcu+0xb0/0xd0 [ 2024.339501] ---[ end trace 0000000000000000 ]--- [ 2024.344121] ------------[ cut here ]------------ [ 2024.348745] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2024.360397] WARNING: CPU: 4 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2024.369360] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2024.424904] CPU: 4 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2024.433874] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2024.441354] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2024.446841] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2024.465598] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [ 2024.470832] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2024.477962] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2024.485097] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2024.492234] R10: 0000000000000001 R11: ffffc90001f7fa70 R12: ffff8881e4d0b838 [ 2024.499375] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2024.506512] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2024.514599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.520350] CR2: 00007f9bd5544000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2024.527485] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2024.534619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2024.541750] Call Trace: [ 2024.544206] [ 2024.546310] ? __warn+0x85/0x180 [ 2024.549544] ? __list_add_valid_or_report+0x7d/0xa0 [ 2024.554425] ? report_bug+0xfc/0x1d0 [ 2024.558011] ? prb_read_valid+0x1b/0x20 [ 2024.561854] ? handle_bug+0x44/0x70 [ 2024.565353] ? exc_invalid_op+0x17/0x60 [ 2024.569191] ? asm_exc_invalid_op+0x1a/0x20 [ 2024.573388] ? __list_add_valid_or_report+0x7d/0xa0 [ 2024.578277] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2024.583185] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2024.587828] ? _raw_spin_unlock+0x23/0x40 [ 2024.591855] __ceph_get_caps+0xf8/0x760 [ceph] [ 2024.596320] ? __vfs_getxattr+0x8d/0xb0 [ 2024.600172] ? dentry_needs_remove_privs+0x31/0x50 [ 2024.604976] ? __file_remove_privs+0x89/0x110 [ 2024.609336] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2024.614061] ? lock_release+0xc5/0x260 [ 2024.617827] ? ceph_write_iter+0x260/0x770 [ceph] [ 2024.622554] ceph_write_iter+0x260/0x770 [ceph] [ 2024.627112] ? lock_is_held_type+0xaa/0x120 [ 2024.631303] vfs_write+0x324/0x4f0 [ 2024.634724] ksys_pwrite64+0x65/0xa0 [ 2024.638311] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2024.643460] do_syscall_64+0x38/0x80 [ 2024.647045] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2024.652097] RIP: 0033:0x7f48f0eeb93b [ 2024.655677] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2024.674422] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2024.681991] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2024.689123] RDX: 0000000000000001 RSI: 000000000207b860 RDI: 0000000000000005 [ 2024.696256] RBP: 0000000000002a15 R08: 0000000000000001 R09: 0000000002074010 [ 2024.703388] R10: 00000000000005ff R11: 0000000000000246 R12: 00007f48f183d000 [ 2024.710520] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b860 [ 2024.717669] [ 2024.719862] irq event stamp: 12815685 [ 2024.723528] hardirqs last enabled at (12815695): [] console_unlock+0x106/0x120 [ 2024.732403] hardirqs last disabled at (12815714): [] console_unlock+0xeb/0x120 [ 2024.741185] softirqs last enabled at (12815732): [] __do_softirq+0x2be/0x371 [ 2024.749886] softirqs last disabled at (12815723): [] irq_exit_rcu+0xb0/0xd0 [ 2024.758407] ---[ end trace 0000000000000000 ]--- [ 2024.763024] ------------[ cut here ]------------ [ 2024.767641] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2024.779288] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2024.788247] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2024.843722] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2024.852685] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2024.860168] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2024.865664] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2024.884410] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2024.889637] RAX: 0000000000000000 RBX: ffff8881d0aef560 RCX: 0000000000000000 [ 2024.896769] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2024.903903] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2024.911034] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d0aefcb0 [ 2024.918171] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2024.925305] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2024.933395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2024.939140] CR2: 00007f9bd553f000 CR3: 0000000120c1e002 CR4: 00000000003706f0 [ 2024.946276] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2024.953407] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2024.960539] Call Trace: [ 2024.962993] [ 2024.965102] ? __warn+0x85/0x180 [ 2024.968341] ? __list_add_valid_or_report+0x7d/0xa0 [ 2024.973222] ? report_bug+0xfc/0x1d0 [ 2024.976808] ? prb_read_valid+0x1b/0x20 [ 2024.980652] ? handle_bug+0x44/0x70 [ 2024.984150] ? exc_invalid_op+0x17/0x60 [ 2024.987989] ? asm_exc_invalid_op+0x1a/0x20 [ 2024.992182] ? __list_add_valid_or_report+0x7d/0xa0 [ 2024.997074] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.001961] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2025.006859] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2025.012029] ceph_write_iter+0x5aa/0x770 [ceph] [ 2025.016578] ? lock_is_held_type+0xaa/0x120 [ 2025.020772] vfs_write+0x324/0x4f0 [ 2025.024189] ksys_pwrite64+0x65/0xa0 [ 2025.027770] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2025.032909] do_syscall_64+0x38/0x80 [ 2025.036489] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2025.041547] RIP: 0033:0x7f48f0eeb93b [ 2025.045127] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2025.063875] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2025.071442] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2025.078573] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2025.085707] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2025.092837] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2025.099973] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2025.107116] [ 2025.109307] irq event stamp: 12779677 [ 2025.112970] hardirqs last enabled at (12779687): [] console_unlock+0x106/0x120 [ 2025.121839] hardirqs last disabled at (12779696): [] console_unlock+0xeb/0x120 [ 2025.130620] softirqs last enabled at (12779710): [] __do_softirq+0x2be/0x371 [ 2025.139321] softirqs last disabled at (12779705): [] irq_exit_rcu+0xb0/0xd0 [ 2025.147846] ---[ end trace 0000000000000000 ]--- [ 2025.152467] ------------[ cut here ]------------ [ 2025.157086] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2025.168737] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2025.177434] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2025.232986] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2025.242119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2025.249599] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2025.255108] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2025.260596] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2025.279342] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2025.284567] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2025.291700] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2025.298836] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2025.305966] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881d1008750 [ 2025.313099] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2025.320231] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2025.328318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2025.334063] CR2: 00007f0d182030c0 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2025.341198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2025.348330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2025.355462] Call Trace: [ 2025.357915] [ 2025.360024] ? __warn+0x85/0x180 [ 2025.363268] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.368154] ? report_bug+0xfc/0x1d0 [ 2025.371741] ? prb_read_valid+0x1b/0x20 [ 2025.375595] ? handle_bug+0x44/0x70 [ 2025.379089] ? exc_invalid_op+0x17/0x60 [ 2025.382931] ? asm_exc_invalid_op+0x1a/0x20 [ 2025.387146] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.392034] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.396916] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2025.401823] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2025.406499] ceph_fill_trace+0x143/0xf30 [ceph] [ 2025.411054] ? __mutex_lock+0x97/0xce0 [ 2025.414814] ? __mutex_lock+0xaec/0xce0 [ 2025.418660] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2025.423214] ? lock_downgrade+0xfd/0x250 [ 2025.427169] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2025.431581] ceph_con_process_message+0x74/0x150 [libceph] [ 2025.437087] ? calc_signature+0xdb/0x100 [libceph] [ 2025.441918] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2025.447269] ? process_scheduled_works+0x22a/0x510 [ 2025.452077] ? process_scheduled_works+0x22a/0x510 [ 2025.456873] ceph_con_workfn+0x344/0x750 [libceph] [ 2025.461692] ? process_scheduled_works+0x22a/0x510 [ 2025.466484] process_scheduled_works+0x2a4/0x510 [ 2025.471126] worker_thread+0x188/0x340 [ 2025.474887] ? rescuer_thread+0x310/0x310 [ 2025.478906] kthread+0xf5/0x130 [ 2025.482062] ? kthread_complete_and_exit+0x20/0x20 [ 2025.486866] ret_from_fork+0x31/0x40 [ 2025.490451] ? kthread_complete_and_exit+0x20/0x20 [ 2025.495252] ret_from_fork_asm+0x11/0x20 [ 2025.499215] [ 2025.501413] irq event stamp: 8814557 [ 2025.504992] hardirqs last enabled at (8814573): [] console_unlock+0x106/0x120 [ 2025.513779] hardirqs last disabled at (8814588): [] console_unlock+0xeb/0x120 [ 2025.522471] softirqs last enabled at (8814602): [] __do_softirq+0x2be/0x371 [ 2025.531078] softirqs last disabled at (8814613): [] irq_exit_rcu+0xb0/0xd0 [ 2025.539512] ---[ end trace 0000000000000000 ]--- [ 2025.544132] ------------[ cut here ]------------ [ 2025.548760] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2025.560410] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2025.569382] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2025.624915] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2025.633865] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2025.641344] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2025.646831] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2025.665578] RSP: 0018:ffffc90001f7fd18 EFLAGS: 00010286 [ 2025.670803] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2025.677935] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2025.685069] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2025.692202] R10: 0000000000000001 R11: ffffc90001f7fba8 R12: ffff8881e4d0b838 [ 2025.699335] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2025.706466] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2025.714551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2025.720297] CR2: 00007f9bd555d000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2025.727431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2025.734566] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2025.741697] Call Trace: [ 2025.744149] [ 2025.746258] ? __warn+0x85/0x180 [ 2025.749490] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.754370] ? report_bug+0xfc/0x1d0 [ 2025.757948] ? prb_read_valid+0x1b/0x20 [ 2025.761790] ? handle_bug+0x44/0x70 [ 2025.765279] ? exc_invalid_op+0x17/0x60 [ 2025.769122] ? asm_exc_invalid_op+0x1a/0x20 [ 2025.773316] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.778195] ? __list_add_valid_or_report+0x7d/0xa0 [ 2025.783081] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2025.787984] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2025.793142] ceph_write_iter+0x5aa/0x770 [ceph] [ 2025.797696] ? lock_is_held_type+0xaa/0x120 [ 2025.801886] vfs_write+0x324/0x4f0 [ 2025.805305] ksys_pwrite64+0x65/0xa0 [ 2025.808882] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2025.814025] do_syscall_64+0x38/0x80 [ 2025.817602] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2025.822654] RIP: 0033:0x7f48f0eeb93b [ 2025.826234] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2025.844979] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2025.852547] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2025.859678] RDX: 0000000000000001 RSI: 000000000207b860 RDI: 0000000000000005 [ 2025.866813] RBP: 0000000000002a15 R08: 0000000000000001 R09: 0000000002074010 [ 2025.873945] R10: 00000000000005ff R11: 0000000000000246 R12: 00007f48f183d000 [ 2025.881076] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b860 [ 2025.888223] [ 2025.890411] irq event stamp: 12818469 [ 2025.894077] hardirqs last enabled at (12818479): [] console_unlock+0x106/0x120 [ 2025.902944] hardirqs last disabled at (12818488): [] console_unlock+0xeb/0x120 [ 2025.911724] softirqs last enabled at (12818426): [] __do_softirq+0x2be/0x371 [ 2025.920415] softirqs last disabled at (12818421): [] irq_exit_rcu+0xb0/0xd0 [ 2025.928935] ---[ end trace 0000000000000000 ]--- [ 2025.933557] ------------[ cut here ]------------ [ 2025.938175] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2025.949827] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2025.958785] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2026.014321] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2026.023273] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2026.030754] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2026.036248] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2026.055002] RSP: 0018:ffffc90001957aa8 EFLAGS: 00010282 [ 2026.060229] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2026.067361] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2026.074494] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2026.081626] R10: 0000000000000001 R11: ffffc90001957938 R12: ffff8881d1008750 [ 2026.088759] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2026.095893] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2026.103979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2026.109727] CR2: 00007fb38f7680f0 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2026.116859] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2026.123993] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2026.131124] Call Trace: [ 2026.133577] [ 2026.135683] ? __warn+0x85/0x180 [ 2026.138918] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.143796] ? report_bug+0xfc/0x1d0 [ 2026.147374] ? prb_read_valid+0x1b/0x20 [ 2026.151218] ? handle_bug+0x44/0x70 [ 2026.154707] ? exc_invalid_op+0x17/0x60 [ 2026.158547] ? asm_exc_invalid_op+0x1a/0x20 [ 2026.162742] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.167623] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.172499] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2026.177400] ceph_open+0x16b/0x440 [ceph] [ 2026.181430] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2026.186584] do_dentry_open+0x172/0x530 [ 2026.190429] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2026.195079] lookup_open+0x37b/0x630 [ 2026.198665] ? verify_cpu+0x20/0x100 [ 2026.202250] ? verify_cpu+0x20/0x100 [ 2026.205837] path_openat+0x25a/0x9d0 [ 2026.209421] do_filp_open+0xb8/0x120 [ 2026.213008] ? lock_release+0xc5/0x260 [ 2026.216771] ? _raw_spin_unlock+0x23/0x40 [ 2026.220783] ? alloc_fd+0x13f/0x200 [ 2026.224280] do_sys_openat2+0x249/0x290 [ 2026.228129] do_sys_open+0x4f/0x80 [ 2026.231542] do_syscall_64+0x38/0x80 [ 2026.235126] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2026.240178] RIP: 0033:0x7f48f0eed2f2 [ 2026.243760] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2026.262516] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2026.270078] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2026.277211] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2026.284344] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000005 [ 2026.291477] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2026.298611] R13: 00007f48f183d228 R14: 00000000020767c0 R15: 000000000000298a [ 2026.305757] [ 2026.307945] irq event stamp: 12725869 [ 2026.311611] hardirqs last enabled at (12725879): [] console_unlock+0x106/0x120 [ 2026.320476] hardirqs last disabled at (12725888): [] console_unlock+0xeb/0x120 [ 2026.329259] softirqs last enabled at (12725902): [] __do_softirq+0x2be/0x371 [ 2026.337949] softirqs last disabled at (12725897): [] irq_exit_rcu+0xb0/0xd0 [ 2026.346469] ---[ end trace 0000000000000000 ]--- [ 2026.351089] ------------[ cut here ]------------ [ 2026.355710] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2026.367363] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2026.376324] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2026.431869] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2026.440826] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2026.448303] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2026.453790] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2026.472537] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2026.477762] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2026.484893] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2026.492029] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2026.499159] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2026.506305] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2026.513442] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2026.521533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2026.527282] CR2: 00007f9bd5561000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2026.534420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2026.541551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2026.548681] Call Trace: [ 2026.551137] [ 2026.553243] ? __warn+0x85/0x180 [ 2026.556475] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.561355] ? report_bug+0xfc/0x1d0 [ 2026.564934] ? prb_read_valid+0x1b/0x20 [ 2026.568775] ? handle_bug+0x44/0x70 [ 2026.572275] ? exc_invalid_op+0x17/0x60 [ 2026.576114] ? asm_exc_invalid_op+0x1a/0x20 [ 2026.580311] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.585191] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.590067] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2026.594968] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2026.600127] ceph_write_iter+0x5aa/0x770 [ceph] [ 2026.604681] ? lock_is_held_type+0xaa/0x120 [ 2026.608870] vfs_write+0x324/0x4f0 [ 2026.612291] ksys_pwrite64+0x65/0xa0 [ 2026.615867] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2026.621010] do_syscall_64+0x38/0x80 [ 2026.624589] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2026.629641] RIP: 0033:0x7f48f0eeb93b [ 2026.633219] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2026.651967] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2026.659531] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2026.666663] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2026.673805] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2026.680940] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2026.688072] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2026.695219] [ 2026.697406] irq event stamp: 12783885 [ 2026.701070] hardirqs last enabled at (12783895): [] console_unlock+0x106/0x120 [ 2026.709941] hardirqs last disabled at (12783912): [] console_unlock+0xeb/0x120 [ 2026.718718] softirqs last enabled at (12783910): [] __do_softirq+0x2be/0x371 [ 2026.727410] softirqs last disabled at (12783903): [] irq_exit_rcu+0xb0/0xd0 [ 2026.735938] ---[ end trace 0000000000000000 ]--- [ 2026.740559] ------------[ cut here ]------------ [ 2026.745186] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2026.756841] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2026.765794] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2026.821319] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2026.830268] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2026.837747] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2026.843235] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2026.861980] RSP: 0018:ffffc90001957c88 EFLAGS: 00010286 [ 2026.867206] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2026.874338] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2026.881471] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2026.888611] R10: 0000000000000001 R11: ffffc90001957b18 R12: ffff8881d1008750 [ 2026.895746] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2026.902879] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2026.910964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2026.916711] CR2: 00007fb38f7920b0 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2026.923846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2026.930976] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2026.938109] Call Trace: [ 2026.940563] [ 2026.942668] ? __warn+0x85/0x180 [ 2026.945902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.950783] ? report_bug+0xfc/0x1d0 [ 2026.954358] ? prb_read_valid+0x1b/0x20 [ 2026.958204] ? handle_bug+0x44/0x70 [ 2026.961692] ? exc_invalid_op+0x17/0x60 [ 2026.965531] ? asm_exc_invalid_op+0x1a/0x20 [ 2026.969727] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.974611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2026.979494] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2026.984395] ceph_open+0x16b/0x440 [ceph] [ 2026.988428] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2026.993579] do_dentry_open+0x172/0x530 [ 2026.997419] path_openat+0x743/0x9d0 [ 2027.001003] do_filp_open+0xb8/0x120 [ 2027.004591] ? lock_release+0xc5/0x260 [ 2027.008351] ? _raw_spin_unlock+0x23/0x40 [ 2027.012366] ? alloc_fd+0x13f/0x200 [ 2027.015866] do_sys_openat2+0x249/0x290 [ 2027.019719] do_sys_open+0x4f/0x80 [ 2027.023125] do_syscall_64+0x38/0x80 [ 2027.026704] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2027.031755] RIP: 0033:0x7f48f0eed2f2 [ 2027.035333] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2027.054083] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2027.061647] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2027.068780] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2027.075913] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2027.083044] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2027.090182] R13: 00007f48f183d228 R14: 00000000020767c0 R15: 000000000000298b [ 2027.097323] [ 2027.099510] irq event stamp: 12727883 [ 2027.103176] hardirqs last enabled at (12727893): [] console_unlock+0x106/0x120 [ 2027.112043] hardirqs last disabled at (12727902): [] console_unlock+0xeb/0x120 [ 2027.120826] softirqs last enabled at (12727916): [] __do_softirq+0x2be/0x371 [ 2027.129517] softirqs last disabled at (12727911): [] irq_exit_rcu+0xb0/0xd0 [ 2027.138036] ---[ end trace 0000000000000000 ]--- [ 2027.142656] ------------[ cut here ]------------ [ 2027.147274] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2027.158927] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2027.167883] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2027.223421] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2027.232374] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2027.239853] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2027.245341] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2027.264085] RSP: 0018:ffffc90001f7fc28 EFLAGS: 00010286 [ 2027.269312] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2027.276444] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2027.283577] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2027.290713] R10: 0000000000000001 R11: ffffc90001f7fab8 R12: ffff8881e4d0b838 [ 2027.297843] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2027.304977] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2027.313062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2027.318809] CR2: 00007f9bd555d000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2027.325942] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2027.333075] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2027.340205] Call Trace: [ 2027.342661] [ 2027.344766] ? __warn+0x85/0x180 [ 2027.347999] ? __list_add_valid_or_report+0x7d/0xa0 [ 2027.352882] ? report_bug+0xfc/0x1d0 [ 2027.356466] ? prb_read_valid+0x1b/0x20 [ 2027.360309] ? handle_bug+0x44/0x70 [ 2027.363800] ? exc_invalid_op+0x17/0x60 [ 2027.367639] ? asm_exc_invalid_op+0x1a/0x20 [ 2027.371834] ? __list_add_valid_or_report+0x7d/0xa0 [ 2027.376713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2027.381592] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2027.386497] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2027.391656] ? __ceph_caps_issued+0x78/0x110 [ceph] [ 2027.396554] __ceph_setattr+0x2c0/0xea0 [ceph] [ 2027.401028] ceph_setattr+0xaa/0x100 [ceph] [ 2027.405231] notify_change+0x237/0x4f0 [ 2027.408995] ? vfs_utimes+0x130/0x220 [ 2027.412661] vfs_utimes+0x130/0x220 [ 2027.416165] do_utimes+0xb7/0x120 [ 2027.419492] __x64_sys_utime+0xa1/0xd0 [ 2027.423249] do_syscall_64+0x38/0x80 [ 2027.426834] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2027.431885] RIP: 0033:0x7f48f0eec99b [ 2027.435465] Code: 64 48 33 0c 25 28 00 00 00 75 06 48 83 c4 10 5b c3 e8 79 09 02 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 84 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d bd 24 2d 00 f7 d8 64 89 01 48 [ 2027.454209] RSP: 002b:00007ffc53d5fff8 EFLAGS: 00000216 ORIG_RAX: 0000000000000084 [ 2027.461778] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eec99b [ 2027.468909] RDX: 00007ffc53d60010 RSI: 00007ffc53d60000 RDI: 0000000002098170 [ 2027.476041] RBP: 00007f48f183d000 R08: 0000000002076605 R09: 0000000000000000 [ 2027.483177] R10: 00007f48f0f711e0 R11: 0000000000000216 R12: 0000000000000004 [ 2027.490308] R13: 00000000000003ec R14: 00007f48f183d000 R15: 00000000020766e0 [ 2027.497453] [ 2027.499641] irq event stamp: 12822359 [ 2027.503308] hardirqs last enabled at (12822369): [] console_unlock+0x106/0x120 [ 2027.512173] hardirqs last disabled at (12822378): [] console_unlock+0xeb/0x120 [ 2027.520957] softirqs last enabled at (12822394): [] __do_softirq+0x2be/0x371 [ 2027.529646] softirqs last disabled at (12822387): [] irq_exit_rcu+0xb0/0xd0 [ 2027.538167] ---[ end trace 0000000000000000 ]--- [ 2027.542789] ------------[ cut here ]------------ [ 2027.547407] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2027.559053] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2027.568006] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2027.623526] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2027.632478] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2027.639957] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2027.645444] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2027.664189] RSP: 0018:ffffc90001fffd18 EFLAGS: 00010286 [ 2027.669417] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2027.676548] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2027.683683] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2027.690820] R10: 0000000000000001 R11: ffffc90001fffba8 R12: ffff8881e4f8e920 [ 2027.697948] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2027.705080] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2027.713165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2027.718912] CR2: 00007fb38f6e0000 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2027.726048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2027.733179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2027.740310] Call Trace: [ 2027.742767] [ 2027.744870] ? __warn+0x85/0x180 [ 2027.748103] ? __list_add_valid_or_report+0x7d/0xa0 [ 2027.752983] ? report_bug+0xfc/0x1d0 [ 2027.756561] ? prb_read_valid+0x1b/0x20 [ 2027.760403] ? handle_bug+0x44/0x70 [ 2027.763894] ? exc_invalid_op+0x17/0x60 [ 2027.767734] ? asm_exc_invalid_op+0x1a/0x20 [ 2027.771929] ? __list_add_valid_or_report+0x7d/0xa0 [ 2027.776809] ? __list_add_valid_or_report+0x7d/0xa0 [ 2027.781707] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2027.786608] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2027.791776] ceph_write_iter+0x5aa/0x770 [ceph] [ 2027.796332] ? lock_is_held_type+0xaa/0x120 [ 2027.800527] vfs_write+0x324/0x4f0 [ 2027.803946] ksys_pwrite64+0x65/0xa0 [ 2027.807521] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2027.812663] do_syscall_64+0x38/0x80 [ 2027.816242] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2027.821293] RIP: 0033:0x7f48f0eeb93b [ 2027.824880] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2027.843629] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2027.851194] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2027.858329] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000014 [ 2027.865459] RBP: 0000000000002a09 R08: 0000000000000001 R09: 0000000002074010 [ 2027.872593] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183e368 [ 2027.879728] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2027.886871] [ 2027.889058] irq event stamp: 12781895 [ 2027.892724] hardirqs last enabled at (12781905): [] console_unlock+0x106/0x120 [ 2027.901592] hardirqs last disabled at (12781914): [] console_unlock+0xeb/0x120 [ 2027.910372] softirqs last enabled at (12781872): [] __do_softirq+0x2be/0x371 [ 2027.919064] softirqs last disabled at (12781867): [] irq_exit_rcu+0xb0/0xd0 [ 2027.927589] ---[ end trace 0000000000000000 ]--- [ 2027.932205] ------------[ cut here ]------------ [ 2027.936849] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2027.948512] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2027.957481] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2028.013032] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2028.021982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2028.029461] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2028.034949] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2028.053717] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [ 2028.058947] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2028.066079] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2028.073212] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2028.080347] R10: 0000000000000001 R11: ffffc90001f7fa70 R12: ffff8881e4d0b838 [ 2028.087478] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2028.094611] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2028.102697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2028.108442] CR2: 00007f9bd555d000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2028.115576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2028.122709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2028.129841] Call Trace: [ 2028.132295] [ 2028.134401] ? __warn+0x85/0x180 [ 2028.137635] ? __list_add_valid_or_report+0x7d/0xa0 [ 2028.142515] ? report_bug+0xfc/0x1d0 [ 2028.146104] ? prb_read_valid+0x1b/0x20 [ 2028.149945] ? handle_bug+0x44/0x70 [ 2028.153443] ? exc_invalid_op+0x17/0x60 [ 2028.157282] ? asm_exc_invalid_op+0x1a/0x20 [ 2028.161478] ? __list_add_valid_or_report+0x7d/0xa0 [ 2028.166358] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2028.171257] try_get_cap_refs+0x433/0x880 [ceph] [ 2028.175891] ? _raw_spin_unlock+0x23/0x40 [ 2028.179918] __ceph_get_caps+0xf8/0x760 [ceph] [ 2028.184385] ? __vfs_getxattr+0x8d/0xb0 [ 2028.188226] ? dentry_needs_remove_privs+0x31/0x50 [ 2028.193024] ? __file_remove_privs+0x89/0x110 [ 2028.197382] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2028.202106] ? lock_release+0xc5/0x260 [ 2028.205864] ? ceph_write_iter+0x260/0x770 [ceph] [ 2028.210584] ceph_write_iter+0x260/0x770 [ceph] [ 2028.215141] ? lock_is_held_type+0xaa/0x120 [ 2028.219334] vfs_write+0x324/0x4f0 [ 2028.222752] ksys_pwrite64+0x65/0xa0 [ 2028.226330] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2028.231472] do_syscall_64+0x38/0x80 [ 2028.235049] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2028.240101] RIP: 0033:0x7f48f0eeb93b [ 2028.243683] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2028.262426] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2028.269993] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2028.277126] RDX: 000000000000df00 RSI: 0000000002088130 RDI: 0000000000000005 [ 2028.284258] RBP: 0000000000002a15 R08: 000000000000df00 R09: 0000000000000003 [ 2028.291390] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d000 [ 2028.298523] R13: 000000000000df00 R14: 000000000000df00 R15: 0000000002088130 [ 2028.305671] [ 2028.307871] irq event stamp: 12824519 [ 2028.311533] hardirqs last enabled at (12824529): [] console_unlock+0x106/0x120 [ 2028.320399] hardirqs last disabled at (12824538): [] console_unlock+0xeb/0x120 [ 2028.329178] softirqs last enabled at (12824516): [] __do_softirq+0x2be/0x371 [ 2028.337871] softirqs last disabled at (12824507): [] irq_exit_rcu+0xb0/0xd0 [ 2028.346392] ---[ end trace 0000000000000000 ]--- [ 2028.351010] ------------[ cut here ]------------ [ 2028.355631] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2028.367280] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2028.376239] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2028.431770] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2028.440725] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2028.448200] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2028.453687] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2028.472433] RSP: 0018:ffffc90001fffbe0 EFLAGS: 00010286 [ 2028.477657] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2028.484792] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2028.491925] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2028.499059] R10: 0000000000000001 R11: ffffc90001fffa70 R12: ffff8881e4f8e920 [ 2028.506189] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2028.513322] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2028.521407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2028.527156] CR2: 00007fb38f6e0000 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2028.534287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2028.541420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2028.548553] Call Trace: [ 2028.551007] [ 2028.553112] ? __warn+0x85/0x180 [ 2028.556345] ? __list_add_valid_or_report+0x7d/0xa0 [ 2028.561225] ? report_bug+0xfc/0x1d0 [ 2028.564809] ? prb_read_valid+0x1b/0x20 [ 2028.568656] ? handle_bug+0x44/0x70 [ 2028.572145] ? exc_invalid_op+0x17/0x60 [ 2028.575985] ? asm_exc_invalid_op+0x1a/0x20 [ 2028.580180] ? __list_add_valid_or_report+0x7d/0xa0 [ 2028.585061] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2028.589958] try_get_cap_refs+0x433/0x880 [ceph] [ 2028.594595] ? _raw_spin_unlock+0x23/0x40 [ 2028.598613] __ceph_get_caps+0xf8/0x760 [ceph] [ 2028.603078] ? __vfs_getxattr+0x8d/0xb0 [ 2028.606921] ? dentry_needs_remove_privs+0x31/0x50 [ 2028.611717] ? __file_remove_privs+0x89/0x110 [ 2028.616077] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2028.620801] ? lock_release+0xc5/0x260 [ 2028.624565] ? ceph_write_iter+0x260/0x770 [ceph] [ 2028.629287] ceph_write_iter+0x260/0x770 [ceph] [ 2028.633843] ? lock_is_held_type+0xaa/0x120 [ 2028.638035] vfs_write+0x324/0x4f0 [ 2028.641456] ksys_pwrite64+0x65/0xa0 [ 2028.645031] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2028.650174] do_syscall_64+0x38/0x80 [ 2028.653752] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2028.658805] RIP: 0033:0x7f48f0eeb93b [ 2028.662392] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2028.681138] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2028.688706] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2028.695846] RDX: 0000000000000800 RSI: 000000000207e200 RDI: 0000000000000014 [ 2028.702979] RBP: 0000000000002a09 R08: 0000000000000800 R09: 0000000000000003 [ 2028.710112] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2028.717244] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207e200 [ 2028.724390] [ 2028.726577] irq event stamp: 12784261 [ 2028.730247] hardirqs last enabled at (12784279): [] console_unlock+0x106/0x120 [ 2028.739110] hardirqs last disabled at (12784288): [] console_unlock+0xeb/0x120 [ 2028.747891] softirqs last enabled at (12784276): [] __do_softirq+0x2be/0x371 [ 2028.756583] softirqs last disabled at (12784269): [] irq_exit_rcu+0xb0/0xd0 [ 2028.765107] ---[ end trace 0000000000000000 ]--- [ 2028.769721] ------------[ cut here ]------------ [ 2028.774345] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2028.786005] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2028.794959] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2028.850490] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2028.859440] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2028.866919] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2028.872407] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2028.891151] RSP: 0018:ffffc90001f7fd18 EFLAGS: 00010286 [ 2028.896378] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2028.903512] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2028.910644] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2028.917784] R10: 0000000000000001 R11: ffffc90001f7fba8 R12: ffff8881e4d0b838 [ 2028.924918] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2028.932052] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2028.940137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2028.945884] CR2: 00007f9bd555d000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2028.953021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2028.960153] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2028.967283] Call Trace: [ 2028.969736] [ 2028.971841] ? __warn+0x85/0x180 [ 2028.975075] ? __list_add_valid_or_report+0x7d/0xa0 [ 2028.979954] ? report_bug+0xfc/0x1d0 [ 2028.983532] ? prb_read_valid+0x1b/0x20 [ 2028.987375] ? handle_bug+0x44/0x70 [ 2028.990865] ? exc_invalid_op+0x17/0x60 [ 2028.994706] ? asm_exc_invalid_op+0x1a/0x20 [ 2028.998900] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.003781] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.008659] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2029.013559] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2029.018721] ceph_write_iter+0x5aa/0x770 [ceph] [ 2029.023279] ? lock_is_held_type+0xaa/0x120 [ 2029.027471] vfs_write+0x324/0x4f0 [ 2029.030890] ksys_pwrite64+0x65/0xa0 [ 2029.034468] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2029.039609] do_syscall_64+0x38/0x80 [ 2029.043186] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2029.048238] RIP: 0033:0x7f48f0eeb93b [ 2029.051826] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2029.070572] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2029.078139] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2029.085274] RDX: 000000000000df00 RSI: 0000000002088130 RDI: 0000000000000005 [ 2029.092405] RBP: 0000000000002a15 R08: 000000000000df00 R09: 0000000000000003 [ 2029.099537] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d000 [ 2029.106677] R13: 000000000000df00 R14: 000000000000df00 R15: 0000000002088130 [ 2029.113829] [ 2029.116022] irq event stamp: 12826599 [ 2029.119688] hardirqs last enabled at (12826609): [] console_unlock+0x106/0x120 [ 2029.128555] hardirqs last disabled at (12826618): [] console_unlock+0xeb/0x120 [ 2029.137332] softirqs last enabled at (12826574): [] __do_softirq+0x2be/0x371 [ 2029.146026] softirqs last disabled at (12826567): [] irq_exit_rcu+0xb0/0xd0 [ 2029.154553] ---[ end trace 0000000000000000 ]--- [ 2029.159174] ------------[ cut here ]------------ [ 2029.163798] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2029.175458] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2029.184419] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2029.239895] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2029.248852] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2029.256328] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2029.261827] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2029.280578] RSP: 0018:ffffc90001fffd18 EFLAGS: 00010286 [ 2029.285810] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2029.292937] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2029.300070] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2029.307205] R10: 0000000000000001 R11: ffffc90001fffba8 R12: ffff8881e4f8e920 [ 2029.314336] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2029.321469] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2029.329556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2029.335302] CR2: 00007f9bd55b5028 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2029.342435] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2029.349568] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2029.356700] Call Trace: [ 2029.359151] [ 2029.361259] ? __warn+0x85/0x180 [ 2029.364492] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.369372] ? report_bug+0xfc/0x1d0 [ 2029.372961] ? prb_read_valid+0x1b/0x20 [ 2029.376801] ? handle_bug+0x44/0x70 [ 2029.380290] ? exc_invalid_op+0x17/0x60 [ 2029.384131] ? asm_exc_invalid_op+0x1a/0x20 [ 2029.388326] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.393208] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.398085] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2029.402986] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2029.408147] ceph_write_iter+0x5aa/0x770 [ceph] [ 2029.412705] ? lock_is_held_type+0xaa/0x120 [ 2029.416896] vfs_write+0x324/0x4f0 [ 2029.420316] ksys_pwrite64+0x65/0xa0 [ 2029.423894] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2029.429036] do_syscall_64+0x38/0x80 [ 2029.432615] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2029.437665] RIP: 0033:0x7f48f0eeb93b [ 2029.441244] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2029.459992] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2029.467557] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2029.474692] RDX: 0000000000000800 RSI: 000000000207e200 RDI: 0000000000000014 [ 2029.481825] RBP: 0000000000002a09 R08: 0000000000000800 R09: 0000000000000003 [ 2029.488955] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2029.496090] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207e200 [ 2029.503238] [ 2029.505431] irq event stamp: 12786145 [ 2029.509097] hardirqs last enabled at (12786155): [] console_unlock+0x106/0x120 [ 2029.517963] hardirqs last disabled at (12786164): [] console_unlock+0xeb/0x120 [ 2029.526744] softirqs last enabled at (12786122): [] __do_softirq+0x2be/0x371 [ 2029.535436] softirqs last disabled at (12786117): [] irq_exit_rcu+0xb0/0xd0 [ 2029.543958] ---[ end trace 0000000000000000 ]--- [ 2029.548575] ------------[ cut here ]------------ [ 2029.553196] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2029.564850] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2029.573810] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2029.629361] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2029.638318] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2029.645800] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2029.651285] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2029.670035] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [ 2029.675256] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2029.682390] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2029.689522] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2029.696657] R10: 0000000000000001 R11: ffffc90001f7fa70 R12: ffff8881e4d0b838 [ 2029.703792] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2029.710922] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2029.719008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2029.724755] CR2: 00007f9bd55bb000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2029.731900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2029.739027] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2029.746161] Call Trace: [ 2029.748613] [ 2029.750722] ? __warn+0x85/0x180 [ 2029.753953] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.758840] ? report_bug+0xfc/0x1d0 [ 2029.762420] ? prb_read_valid+0x1b/0x20 [ 2029.766266] ? handle_bug+0x44/0x70 [ 2029.769761] ? exc_invalid_op+0x17/0x60 [ 2029.773601] ? asm_exc_invalid_op+0x1a/0x20 [ 2029.777797] ? __list_add_valid_or_report+0x7d/0xa0 [ 2029.782678] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2029.787576] try_get_cap_refs+0x433/0x880 [ceph] [ 2029.792211] ? _raw_spin_unlock+0x23/0x40 [ 2029.796230] __ceph_get_caps+0xf8/0x760 [ceph] [ 2029.800695] ? __vfs_getxattr+0x8d/0xb0 [ 2029.804540] ? dentry_needs_remove_privs+0x31/0x50 [ 2029.809335] ? __file_remove_privs+0x89/0x110 [ 2029.813694] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2029.818418] ? lock_release+0xc5/0x260 [ 2029.822174] ? ceph_write_iter+0x260/0x770 [ceph] [ 2029.826893] ceph_write_iter+0x260/0x770 [ceph] [ 2029.831446] ? lock_is_held_type+0xaa/0x120 [ 2029.835635] vfs_write+0x324/0x4f0 [ 2029.839056] ksys_pwrite64+0x65/0xa0 [ 2029.842639] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2029.847783] do_syscall_64+0x38/0x80 [ 2029.851368] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2029.856420] RIP: 0033:0x7f48f0eeb93b [ 2029.860000] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2029.878746] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2029.886312] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2029.893445] RDX: 0000000000001000 RSI: 000000000207eb20 RDI: 0000000000000005 [ 2029.900579] RBP: 0000000000002a15 R08: 0000000000001000 R09: 0000000000000003 [ 2029.907710] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d000 [ 2029.914842] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207eb20 [ 2029.921993] [ 2029.924187] irq event stamp: 12828365 [ 2029.927851] hardirqs last enabled at (12828375): [] console_unlock+0x106/0x120 [ 2029.936720] hardirqs last disabled at (12828384): [] console_unlock+0xeb/0x120 [ 2029.945499] softirqs last enabled at (12828352): [] __do_softirq+0x2be/0x371 [ 2029.954192] softirqs last disabled at (12828347): [] irq_exit_rcu+0xb0/0xd0 [ 2029.962711] ---[ end trace 0000000000000000 ]--- [ 2029.967333] ------------[ cut here ]------------ [ 2029.971949] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2029.983598] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2029.992549] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2030.048063] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2030.057017] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2030.064492] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2030.069981] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2030.088725] RSP: 0018:ffffc90001fffc88 EFLAGS: 00010286 [ 2030.093952] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2030.101086] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2030.108218] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2030.115352] R10: 0000000000000001 R11: ffffc90001fffb18 R12: ffff8881e4f8e920 [ 2030.122484] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2030.129616] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2030.137701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2030.143448] CR2: 00007f9bd55bf000 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2030.150581] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2030.157713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2030.164847] Call Trace: [ 2030.167299] [ 2030.169405] ? __warn+0x85/0x180 [ 2030.172640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.177518] ? report_bug+0xfc/0x1d0 [ 2030.181110] ? prb_read_valid+0x1b/0x20 [ 2030.184956] ? handle_bug+0x44/0x70 [ 2030.188448] ? exc_invalid_op+0x17/0x60 [ 2030.192286] ? asm_exc_invalid_op+0x1a/0x20 [ 2030.196482] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.201362] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.206238] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2030.211140] ceph_open+0x16b/0x440 [ceph] [ 2030.215172] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2030.220324] do_dentry_open+0x172/0x530 [ 2030.224166] path_openat+0x743/0x9d0 [ 2030.227750] do_filp_open+0xb8/0x120 [ 2030.231338] ? lock_release+0xc5/0x260 [ 2030.235098] ? _raw_spin_unlock+0x23/0x40 [ 2030.239113] ? alloc_fd+0x13f/0x200 [ 2030.242612] do_sys_openat2+0x249/0x290 [ 2030.246459] do_sys_open+0x4f/0x80 [ 2030.249871] do_syscall_64+0x38/0x80 [ 2030.253448] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2030.258500] RIP: 0033:0x7f48f0eed2f2 [ 2030.262079] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2030.280828] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2030.288392] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2030.295524] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2030.302658] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2030.309792] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2030.316922] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a0a [ 2030.324070] [ 2030.326257] irq event stamp: 12788237 [ 2030.329924] hardirqs last enabled at (12788247): [] console_unlock+0x106/0x120 [ 2030.338788] hardirqs last disabled at (12788256): [] console_unlock+0xeb/0x120 [ 2030.347572] softirqs last enabled at (12788270): [] __do_softirq+0x2be/0x371 [ 2030.356262] softirqs last disabled at (12788265): [] irq_exit_rcu+0xb0/0xd0 [ 2030.364780] ---[ end trace 0000000000000000 ]--- [ 2030.369406] ------------[ cut here ]------------ [ 2030.374024] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2030.385666] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2030.394623] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2030.450148] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2030.459104] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2030.466584] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2030.472069] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2030.490822] RSP: 0018:ffffc90001f7fd18 EFLAGS: 00010286 [ 2030.496050] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2030.503182] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2030.510316] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2030.517448] R10: 0000000000000001 R11: ffffc90001f7fba8 R12: ffff8881e4d0b838 [ 2030.524583] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2030.531722] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2030.539807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2030.545553] CR2: 00007f9bd55bb000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2030.552687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2030.559818] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2030.566953] Call Trace: [ 2030.569407] [ 2030.571513] ? __warn+0x85/0x180 [ 2030.574744] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.579625] ? report_bug+0xfc/0x1d0 [ 2030.583204] ? prb_read_valid+0x1b/0x20 [ 2030.587053] ? handle_bug+0x44/0x70 [ 2030.590543] ? exc_invalid_op+0x17/0x60 [ 2030.594384] ? asm_exc_invalid_op+0x1a/0x20 [ 2030.598580] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.603460] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.608338] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2030.613238] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2030.618402] ceph_write_iter+0x5aa/0x770 [ceph] [ 2030.622957] ? lock_is_held_type+0xaa/0x120 [ 2030.627151] vfs_write+0x324/0x4f0 [ 2030.630570] ksys_pwrite64+0x65/0xa0 [ 2030.634146] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2030.639289] do_syscall_64+0x38/0x80 [ 2030.642867] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2030.647919] RIP: 0033:0x7f48f0eeb93b [ 2030.651503] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2030.670262] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2030.677828] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2030.684962] RDX: 0000000000001000 RSI: 000000000207eb20 RDI: 0000000000000005 [ 2030.692093] RBP: 0000000000002a15 R08: 0000000000001000 R09: 0000000000000003 [ 2030.699225] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d000 [ 2030.706359] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207eb20 [ 2030.713505] [ 2030.715692] irq event stamp: 12830239 [ 2030.719359] hardirqs last enabled at (12830249): [] console_unlock+0x106/0x120 [ 2030.728225] hardirqs last disabled at (12830258): [] console_unlock+0xeb/0x120 [ 2030.737006] softirqs last enabled at (12830186): [] __do_softirq+0x2be/0x371 [ 2030.745696] softirqs last disabled at (12830181): [] irq_exit_rcu+0xb0/0xd0 [ 2030.754223] ---[ end trace 0000000000000000 ]--- [ 2030.758849] ------------[ cut here ]------------ [ 2030.763550] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2030.775204] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2030.784171] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2030.839702] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2030.848667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2030.856149] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2030.861639] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2030.880393] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2030.885626] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2030.892757] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2030.899891] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2030.907021] R10: 0000000000000001 R11: ffffc90001fffa60 R12: ffff8881e4f8e920 [ 2030.914159] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2030.921287] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2030.929376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2030.935120] CR2: 00007fb38817dd28 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2030.942259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2030.949394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2030.956527] Call Trace: [ 2030.958979] [ 2030.961089] ? __warn+0x85/0x180 [ 2030.964327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.969210] ? report_bug+0xfc/0x1d0 [ 2030.972795] ? prb_read_valid+0x1b/0x20 [ 2030.976642] ? handle_bug+0x44/0x70 [ 2030.980140] ? exc_invalid_op+0x17/0x60 [ 2030.983985] ? asm_exc_invalid_op+0x1a/0x20 [ 2030.988178] ? __list_add_valid_or_report+0x7d/0xa0 [ 2030.993062] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2030.997959] try_get_cap_refs+0x433/0x880 [ceph] [ 2031.002613] __ceph_get_caps+0xf8/0x760 [ceph] [ 2031.007089] ? ceph_read_iter+0x134/0x860 [ceph] [ 2031.011727] ceph_read_iter+0x134/0x860 [ceph] [ 2031.016198] ? selinux_file_permission+0x119/0x150 [ 2031.021003] ? vfs_read+0x20f/0x300 [ 2031.024507] vfs_read+0x20f/0x300 [ 2031.027853] ksys_pread64+0x65/0xa0 [ 2031.031347] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2031.036490] do_syscall_64+0x38/0x80 [ 2031.040071] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2031.045132] RIP: 0033:0x7f48f0eeb88b [ 2031.048718] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2031.067462] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2031.075028] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb88b [ 2031.082161] RDX: 0000000000000800 RSI: 000000000207e200 RDI: 0000000000000014 [ 2031.089295] RBP: 0000000000002a0a R08: 000000000000005f R09: 0000000000000003 [ 2031.096429] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000800 [ 2031.103562] R13: 0000000000000800 R14: 000000000207e200 R15: 00007f48f183e368 [ 2031.110707] [ 2031.112904] irq event stamp: 12790195 [ 2031.116567] hardirqs last enabled at (12790205): [] console_unlock+0x106/0x120 [ 2031.125436] hardirqs last disabled at (12790214): [] console_unlock+0xeb/0x120 [ 2031.134215] softirqs last enabled at (12790182): [] __do_softirq+0x2be/0x371 [ 2031.142909] softirqs last disabled at (12790177): [] irq_exit_rcu+0xb0/0xd0 [ 2031.151435] ---[ end trace 0000000000000000 ]--- [ 2031.156057] ------------[ cut here ]------------ [ 2031.160675] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2031.172321] WARNING: CPU: 3 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2031.181277] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2031.236797] CPU: 3 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2031.245748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2031.253228] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2031.258716] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2031.277460] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2031.282688] RAX: 0000000000000000 RBX: ffff8881a1d36b98 RCX: 0000000000000000 [ 2031.289820] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2031.296953] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2031.304084] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d372e8 [ 2031.311223] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2031.318361] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2031.326445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2031.332192] CR2: 00007f9bd551c000 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2031.339324] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2031.346457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2031.353591] Call Trace: [ 2031.356043] [ 2031.358149] ? __warn+0x85/0x180 [ 2031.361384] ? __list_add_valid_or_report+0x7d/0xa0 [ 2031.366262] ? report_bug+0xfc/0x1d0 [ 2031.369841] ? prb_read_valid+0x1b/0x20 [ 2031.373683] ? handle_bug+0x44/0x70 [ 2031.377174] ? exc_invalid_op+0x17/0x60 [ 2031.381012] ? asm_exc_invalid_op+0x1a/0x20 [ 2031.385207] ? __list_add_valid_or_report+0x7d/0xa0 [ 2031.390089] ? __list_add_valid_or_report+0x7d/0xa0 [ 2031.394974] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2031.399875] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2031.405035] ceph_write_iter+0x5aa/0x770 [ceph] [ 2031.409594] ? lock_is_held_type+0xaa/0x120 [ 2031.413788] vfs_write+0x324/0x4f0 [ 2031.417208] ksys_pwrite64+0x65/0xa0 [ 2031.420792] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2031.425936] do_syscall_64+0x38/0x80 [ 2031.429522] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2031.434574] RIP: 0033:0x7f48f0eeb93b [ 2031.438153] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2031.456899] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2031.464465] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2031.471599] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2031.478730] RBP: 0000000000002a10 R08: 0000000000010000 R09: 0000000000000003 [ 2031.485864] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2031.492995] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2031.500143] [ 2031.502339] irq event stamp: 12821015 [ 2031.506005] hardirqs last enabled at (12821025): [] console_unlock+0x106/0x120 [ 2031.514872] hardirqs last disabled at (12821034): [] console_unlock+0xeb/0x120 [ 2031.523651] softirqs last enabled at (12820666): [] __do_softirq+0x2be/0x371 [ 2031.532345] softirqs last disabled at (12820661): [] irq_exit_rcu+0xb0/0xd0 [ 2031.540863] ---[ end trace 0000000000000000 ]--- [ 2031.545485] ------------[ cut here ]------------ [ 2031.550166] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2031.561996] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2031.570967] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2031.626549] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2031.635601] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2031.643205] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2031.648725] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2031.667596] RSP: 0018:ffffc90001fffbe0 EFLAGS: 00010286 [ 2031.672842] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2031.680099] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2031.687233] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2031.694501] R10: 0000000000000001 R11: ffffc90001fffa70 R12: ffff8881e4f8e920 [ 2031.701655] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2031.708876] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2031.716981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2031.722799] CR2: 00007f9bd55d3000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2031.729942] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2031.737134] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2031.744279] Call Trace: [ 2031.746810] [ 2031.748922] ? __warn+0x85/0x180 [ 2031.752163] ? __list_add_valid_or_report+0x7d/0xa0 [ 2031.757051] ? report_bug+0xfc/0x1d0 [ 2031.760644] ? prb_read_valid+0x1b/0x20 [ 2031.764494] ? handle_bug+0x44/0x70 [ 2031.767992] ? exc_invalid_op+0x17/0x60 [ 2031.771833] ? asm_exc_invalid_op+0x1a/0x20 [ 2031.776030] ? __list_add_valid_or_report+0x7d/0xa0 [ 2031.780909] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2031.785821] try_get_cap_refs+0x433/0x880 [ceph] [ 2031.790452] ? _raw_spin_unlock+0x23/0x40 [ 2031.794484] __ceph_get_caps+0xf8/0x760 [ceph] [ 2031.798946] ? __vfs_getxattr+0x8d/0xb0 [ 2031.802802] ? dentry_needs_remove_privs+0x31/0x50 [ 2031.807602] ? __file_remove_privs+0x89/0x110 [ 2031.811963] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2031.816687] ? lock_release+0xc5/0x260 [ 2031.820453] ? ceph_write_iter+0x260/0x770 [ceph] [ 2031.825172] ceph_write_iter+0x260/0x770 [ceph] [ 2031.829730] ? lock_is_held_type+0xaa/0x120 [ 2031.833924] vfs_write+0x324/0x4f0 [ 2031.837348] ksys_pwrite64+0x65/0xa0 [ 2031.840925] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2031.846069] do_syscall_64+0x38/0x80 [ 2031.849655] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2031.854708] RIP: 0033:0x7f48f0eeb93b [ 2031.858287] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2031.877031] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2031.884598] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2031.891730] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000014 [ 2031.898869] RBP: 0000000000002a0a R08: 0000000000001000 R09: 0000000000000003 [ 2031.906007] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2031.913138] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207e200 [ 2031.920282] [ 2031.922482] irq event stamp: 12794407 [ 2031.926148] hardirqs last enabled at (12794417): [] console_unlock+0x106/0x120 [ 2031.935020] hardirqs last disabled at (12794426): [] console_unlock+0xeb/0x120 [ 2031.943801] softirqs last enabled at (12794404): [] __do_softirq+0x2be/0x371 [ 2031.952495] softirqs last disabled at (12794395): [] irq_exit_rcu+0xb0/0xd0 [ 2031.961023] ---[ end trace 0000000000000000 ]--- [ 2031.965648] ------------[ cut here ]------------ [ 2031.970269] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2031.981925] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2031.990879] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2032.046421] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2032.055379] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2032.062859] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2032.068344] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2032.087090] RSP: 0018:ffffc90001957be0 EFLAGS: 00010286 [ 2032.092316] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2032.099449] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2032.106583] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2032.113717] R10: 0000000000000001 R11: ffffc90001957a70 R12: ffff8881d1008750 [ 2032.120847] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2032.127981] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2032.136067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2032.141814] CR2: 00007fb38808e998 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2032.148946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2032.156077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2032.163212] Call Trace: [ 2032.165664] [ 2032.167773] ? __warn+0x85/0x180 [ 2032.171012] ? __list_add_valid_or_report+0x7d/0xa0 [ 2032.175895] ? report_bug+0xfc/0x1d0 [ 2032.179478] ? prb_read_valid+0x1b/0x20 [ 2032.183323] ? handle_bug+0x44/0x70 [ 2032.186820] ? exc_invalid_op+0x17/0x60 [ 2032.190660] ? asm_exc_invalid_op+0x1a/0x20 [ 2032.194856] ? __list_add_valid_or_report+0x7d/0xa0 [ 2032.199745] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2032.204652] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2032.209295] ? _raw_spin_unlock+0x23/0x40 [ 2032.213323] __ceph_get_caps+0xf8/0x760 [ceph] [ 2032.217789] ? __vfs_getxattr+0x8d/0xb0 [ 2032.221640] ? dentry_needs_remove_privs+0x31/0x50 [ 2032.226438] ? __file_remove_privs+0x89/0x110 [ 2032.230803] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2032.235528] ? lock_release+0xc5/0x260 [ 2032.239287] ? ceph_write_iter+0x260/0x770 [ceph] [ 2032.244013] ceph_write_iter+0x260/0x770 [ceph] [ 2032.248572] ? lock_is_held_type+0xaa/0x120 [ 2032.252763] vfs_write+0x324/0x4f0 [ 2032.256183] ksys_pwrite64+0x65/0xa0 [ 2032.259769] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2032.264920] do_syscall_64+0x38/0x80 [ 2032.268506] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2032.273558] RIP: 0033:0x7f48f0eeb93b [ 2032.277136] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2032.295893] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2032.303474] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2032.310621] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000006 [ 2032.317757] RBP: 000000000000298b R08: 0000000000000001 R09: 0000000002074010 [ 2032.324893] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183d228 [ 2032.332023] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2032.339172] [ 2032.341367] irq event stamp: 12739811 [ 2032.345031] hardirqs last enabled at (12739821): [] console_unlock+0x106/0x120 [ 2032.353899] hardirqs last disabled at (12739830): [] console_unlock+0xeb/0x120 [ 2032.362677] softirqs last enabled at (12739758): [] __do_softirq+0x2be/0x371 [ 2032.371373] softirqs last disabled at (12739849): [] irq_exit_rcu+0xb0/0xd0 [ 2032.379899] ---[ end trace 0000000000000000 ]--- [ 2032.384521] ------------[ cut here ]------------ [ 2032.389151] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2032.400807] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2032.409765] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2032.465313] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2032.474275] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2032.481752] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2032.487236] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2032.505989] RSP: 0018:ffffc90001fffd18 EFLAGS: 00010286 [ 2032.511219] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2032.518351] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2032.525485] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2032.532616] R10: 0000000000000001 R11: ffffc90001fffba8 R12: ffff8881e4f8e920 [ 2032.539751] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2032.546882] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2032.554967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2032.560713] CR2: 00007f9bd55dc020 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2032.567848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2032.574981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2032.582114] Call Trace: [ 2032.584566] [ 2032.586672] ? __warn+0x85/0x180 [ 2032.589906] ? __list_add_valid_or_report+0x7d/0xa0 [ 2032.594786] ? report_bug+0xfc/0x1d0 [ 2032.598375] ? prb_read_valid+0x1b/0x20 [ 2032.602225] ? handle_bug+0x44/0x70 [ 2032.605722] ? exc_invalid_op+0x17/0x60 [ 2032.609562] ? asm_exc_invalid_op+0x1a/0x20 [ 2032.613758] ? __list_add_valid_or_report+0x7d/0xa0 [ 2032.618645] ? __list_add_valid_or_report+0x7d/0xa0 [ 2032.623524] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2032.628426] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2032.633592] ceph_write_iter+0x5aa/0x770 [ceph] [ 2032.638153] ? lock_is_held_type+0xaa/0x120 [ 2032.642344] vfs_write+0x324/0x4f0 [ 2032.645764] ksys_pwrite64+0x65/0xa0 [ 2032.649340] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2032.654484] do_syscall_64+0x38/0x80 [ 2032.658070] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2032.663122] RIP: 0033:0x7f48f0eeb93b [ 2032.666705] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2032.685458] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2032.693030] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2032.700163] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000014 [ 2032.707297] RBP: 0000000000002a0a R08: 0000000000001000 R09: 0000000000000003 [ 2032.714429] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2032.721562] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207e200 [ 2032.728709] [ 2032.730906] irq event stamp: 12796527 [ 2032.734571] hardirqs last enabled at (12796537): [] console_unlock+0x106/0x120 [ 2032.743438] hardirqs last disabled at (12796546): [] console_unlock+0xeb/0x120 [ 2032.752217] softirqs last enabled at (12796524): [] __do_softirq+0x2be/0x371 [ 2032.760910] softirqs last disabled at (12796519): [] irq_exit_rcu+0xb0/0xd0 [ 2032.769428] ---[ end trace 0000000000000000 ]--- [ 2032.774050] ------------[ cut here ]------------ [ 2032.778666] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2032.790315] WARNING: CPU: 2 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2032.799267] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2032.854736] CPU: 2 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2032.863686] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2032.871167] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2032.876653] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2032.895401] RSP: 0018:ffffc90001957d18 EFLAGS: 00010286 [ 2032.900624] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2032.907756] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2032.914880] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2032.922015] R10: 0000000000000001 R11: ffffc90001957ba8 R12: ffff8881d1008750 [ 2032.929149] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2032.936279] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2032.944364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2032.950104] CR2: 00007fb38f7e0000 CR3: 000000016696c006 CR4: 00000000003706e0 [ 2032.957236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2032.964360] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2032.971492] Call Trace: [ 2032.973939] [ 2032.976045] ? __warn+0x85/0x180 [ 2032.979276] ? __list_add_valid_or_report+0x7d/0xa0 [ 2032.984156] ? report_bug+0xfc/0x1d0 [ 2032.987737] ? prb_read_valid+0x1b/0x20 [ 2032.991582] ? handle_bug+0x44/0x70 [ 2032.995078] ? exc_invalid_op+0x17/0x60 [ 2032.998915] ? asm_exc_invalid_op+0x1a/0x20 [ 2033.003105] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.007980] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.012861] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2033.017750] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2033.022902] ceph_write_iter+0x5aa/0x770 [ceph] [ 2033.027441] ? lock_is_held_type+0xaa/0x120 [ 2033.031627] vfs_write+0x324/0x4f0 [ 2033.035035] ksys_pwrite64+0x65/0xa0 [ 2033.038608] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2033.043751] do_syscall_64+0x38/0x80 [ 2033.047328] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2033.052382] RIP: 0033:0x7f48f0eeb93b [ 2033.055961] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2033.074706] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2033.082273] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2033.089405] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000006 [ 2033.096537] RBP: 000000000000298b R08: 0000000000000001 R09: 0000000002074010 [ 2033.103661] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183d228 [ 2033.110785] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2033.117917] [ 2033.120104] irq event stamp: 12743095 [ 2033.123768] hardirqs last enabled at (12743105): [] console_unlock+0x106/0x120 [ 2033.132635] hardirqs last disabled at (12743114): [] console_unlock+0xeb/0x120 [ 2033.141414] softirqs last enabled at (12743082): [] __do_softirq+0x2be/0x371 [ 2033.150109] softirqs last disabled at (12743133): [] irq_exit_rcu+0xb0/0xd0 [ 2033.158627] ---[ end trace 0000000000000000 ]--- [ 2033.163249] ------------[ cut here ]------------ [ 2033.167868] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2033.179518] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2033.188481] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2033.244016] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2033.252978] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2033.260456] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2033.265944] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2033.284691] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2033.289929] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2033.297067] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2033.304197] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2033.311335] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d09f8750 [ 2033.318476] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2033.325604] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2033.333692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2033.339438] CR2: 00007f9bd559a000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2033.346571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2033.353709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2033.360846] Call Trace: [ 2033.363297] [ 2033.365403] ? __warn+0x85/0x180 [ 2033.368638] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.373518] ? report_bug+0xfc/0x1d0 [ 2033.377103] ? prb_read_valid+0x1b/0x20 [ 2033.380950] ? handle_bug+0x44/0x70 [ 2033.384449] ? exc_invalid_op+0x17/0x60 [ 2033.388303] ? asm_exc_invalid_op+0x1a/0x20 [ 2033.392498] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.397380] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2033.402287] try_get_cap_refs+0x433/0x880 [ceph] [ 2033.406937] ? _raw_spin_unlock+0x23/0x40 [ 2033.410958] __ceph_get_caps+0xf8/0x760 [ceph] [ 2033.415425] ? __vfs_getxattr+0x8d/0xb0 [ 2033.419280] ? dentry_needs_remove_privs+0x31/0x50 [ 2033.424079] ? __file_remove_privs+0x89/0x110 [ 2033.428437] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2033.433163] ? lock_release+0xc5/0x260 [ 2033.436918] ? ceph_write_iter+0x260/0x770 [ceph] [ 2033.441637] ceph_write_iter+0x260/0x770 [ceph] [ 2033.446201] ? lock_is_held_type+0xaa/0x120 [ 2033.450398] vfs_write+0x324/0x4f0 [ 2033.453817] ksys_pwrite64+0x65/0xa0 [ 2033.457403] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2033.462556] do_syscall_64+0x38/0x80 [ 2033.466140] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2033.471202] RIP: 0033:0x7f48f0eeb93b [ 2033.474782] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2033.493538] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2033.501116] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2033.508250] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2033.515383] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2033.522520] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2033.529666] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2033.536813] [ 2033.539009] irq event stamp: 12802909 [ 2033.542674] hardirqs last enabled at (12802919): [] console_unlock+0x106/0x120 [ 2033.551548] hardirqs last disabled at (12802940): [] console_unlock+0xeb/0x120 [ 2033.560332] softirqs last enabled at (12802956): [] __do_softirq+0x2be/0x371 [ 2033.569032] softirqs last disabled at (12802967): [] irq_exit_rcu+0xb0/0xd0 [ 2033.577558] ---[ end trace 0000000000000000 ]--- [ 2033.582179] ------------[ cut here ]------------ [ 2033.586794] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2033.598445] WARNING: CPU: 0 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2033.607402] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2033.662874] CPU: 0 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2033.671834] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2033.679315] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2033.684803] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2033.703547] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2033.708775] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2033.715907] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2033.723040] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2033.730174] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2033.737311] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2033.744446] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2033.752533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2033.758280] CR2: 00007fb38808e998 CR3: 0000000113500006 CR4: 00000000003706f0 [ 2033.765409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2033.772544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2033.779678] Call Trace: [ 2033.782132] [ 2033.784237] ? __warn+0x85/0x180 [ 2033.787471] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.792358] ? report_bug+0xfc/0x1d0 [ 2033.795938] ? prb_read_valid+0x1b/0x20 [ 2033.799777] ? handle_bug+0x44/0x70 [ 2033.803272] ? exc_invalid_op+0x17/0x60 [ 2033.807116] ? asm_exc_invalid_op+0x1a/0x20 [ 2033.811312] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.816190] ? __list_add_valid_or_report+0x7d/0xa0 [ 2033.821071] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2033.825972] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2033.831140] ceph_write_iter+0x5aa/0x770 [ceph] [ 2033.835689] ? lock_is_held_type+0xaa/0x120 [ 2033.839883] vfs_write+0x324/0x4f0 [ 2033.843300] ksys_pwrite64+0x65/0xa0 [ 2033.846882] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2033.852031] do_syscall_64+0x38/0x80 [ 2033.855609] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2033.860658] RIP: 0033:0x7f48f0eeb93b [ 2033.864239] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2033.882990] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2033.890565] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2033.897701] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2033.904835] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2033.911966] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2033.919101] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2033.926244] [ 2033.928436] irq event stamp: 12824937 [ 2033.932098] hardirqs last enabled at (12824947): [] console_unlock+0x106/0x120 [ 2033.940970] hardirqs last disabled at (12824956): [] console_unlock+0xeb/0x120 [ 2033.949766] softirqs last enabled at (12824972): [] __do_softirq+0x2be/0x371 [ 2033.958467] softirqs last disabled at (12824965): [] irq_exit_rcu+0xb0/0xd0 [ 2033.966992] ---[ end trace 0000000000000000 ]--- [ 2033.971613] ------------[ cut here ]------------ [ 2033.976253] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2033.987926] WARNING: CPU: 6 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2033.996905] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2034.052450] CPU: 6 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2034.061418] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2034.068918] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2034.074430] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2034.093191] RSP: 0018:ffffc90001957be0 EFLAGS: 00010286 [ 2034.098433] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2034.105583] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2034.112732] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2034.119883] R10: 0000000000000001 R11: ffffc90001957a70 R12: ffff8881d1008750 [ 2034.127034] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2034.134183] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2034.142287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2034.148048] CR2: 00007f9bd55dc020 CR3: 000000016696c003 CR4: 00000000003706e0 [ 2034.155186] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2034.162318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2034.169451] Call Trace: [ 2034.171908] [ 2034.174019] ? __warn+0x85/0x180 [ 2034.177254] ? __list_add_valid_or_report+0x7d/0xa0 [ 2034.182144] ? report_bug+0xfc/0x1d0 [ 2034.185728] ? prb_read_valid+0x1b/0x20 [ 2034.189571] ? handle_bug+0x44/0x70 [ 2034.193068] ? exc_invalid_op+0x17/0x60 [ 2034.196913] ? asm_exc_invalid_op+0x1a/0x20 [ 2034.201114] ? __list_add_valid_or_report+0x7d/0xa0 [ 2034.206002] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2034.210901] try_get_cap_refs+0x433/0x880 [ceph] [ 2034.215549] ? _raw_spin_unlock+0x23/0x40 [ 2034.219571] __ceph_get_caps+0xf8/0x760 [ceph] [ 2034.224038] ? __vfs_getxattr+0x8d/0xb0 [ 2034.227890] ? dentry_needs_remove_privs+0x31/0x50 [ 2034.232687] ? __file_remove_privs+0x89/0x110 [ 2034.237044] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2034.241769] ? lock_release+0xc5/0x260 [ 2034.245528] ? ceph_write_iter+0x260/0x770 [ceph] [ 2034.250253] ceph_write_iter+0x260/0x770 [ceph] [ 2034.254811] ? lock_is_held_type+0xaa/0x120 [ 2034.259002] vfs_write+0x324/0x4f0 [ 2034.262422] ksys_pwrite64+0x65/0xa0 [ 2034.265999] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2034.271141] do_syscall_64+0x38/0x80 [ 2034.274719] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2034.279770] RIP: 0033:0x7f48f0eeb93b [ 2034.283350] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2034.302097] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2034.309666] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2034.316795] RDX: 0000000000000800 RSI: 000000000207e080 RDI: 0000000000000006 [ 2034.323929] RBP: 000000000000298b R08: 0000000000000800 R09: 0000000000000003 [ 2034.331061] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d228 [ 2034.338193] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207e080 [ 2034.345340] [ 2034.347536] irq event stamp: 12746793 [ 2034.351203] hardirqs last enabled at (12746803): [] console_unlock+0x106/0x120 [ 2034.360068] hardirqs last disabled at (12746812): [] console_unlock+0xeb/0x120 [ 2034.368848] softirqs last enabled at (12746730): [] __do_softirq+0x2be/0x371 [ 2034.377543] softirqs last disabled at (12746831): [] irq_exit_rcu+0xb0/0xd0 [ 2034.386061] ---[ end trace 0000000000000000 ]--- [ 2034.390681] ------------[ cut here ]------------ [ 2034.395299] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2034.406949] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2034.415925] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2034.471401] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2034.480372] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2034.487853] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2034.493340] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2034.512097] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2034.517337] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2034.524468] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2034.531607] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2034.538747] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d37cb0 [ 2034.545889] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2034.553020] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2034.561104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2034.566849] CR2: 00007fb39af77000 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2034.573984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2034.581116] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2034.588249] Call Trace: [ 2034.590702] [ 2034.592810] ? __warn+0x85/0x180 [ 2034.596049] ? __list_add_valid_or_report+0x7d/0xa0 [ 2034.600936] ? report_bug+0xfc/0x1d0 [ 2034.604516] ? prb_read_valid+0x1b/0x20 [ 2034.608361] ? handle_bug+0x44/0x70 [ 2034.611856] ? exc_invalid_op+0x17/0x60 [ 2034.615702] ? asm_exc_invalid_op+0x1a/0x20 [ 2034.619900] ? __list_add_valid_or_report+0x7d/0xa0 [ 2034.624782] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2034.629680] try_get_cap_refs+0x433/0x880 [ceph] [ 2034.634326] ? _raw_spin_unlock+0x23/0x40 [ 2034.638350] __ceph_get_caps+0xf8/0x760 [ceph] [ 2034.642817] ? __vfs_getxattr+0x8d/0xb0 [ 2034.646659] ? dentry_needs_remove_privs+0x31/0x50 [ 2034.651458] ? __file_remove_privs+0x89/0x110 [ 2034.655816] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2034.660541] ? lock_release+0xc5/0x260 [ 2034.664298] ? ceph_write_iter+0x260/0x770 [ceph] [ 2034.669026] ceph_write_iter+0x260/0x770 [ceph] [ 2034.673581] ? lock_is_held_type+0xaa/0x120 [ 2034.677777] vfs_write+0x324/0x4f0 [ 2034.681192] ksys_pwrite64+0x65/0xa0 [ 2034.684771] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2034.689916] do_syscall_64+0x38/0x80 [ 2034.693498] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2034.698552] RIP: 0033:0x7f48f0eeb93b [ 2034.702132] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2034.720878] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2034.728446] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2034.735575] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2034.742711] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2034.749850] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2034.756987] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2034.764136] [ 2034.766328] irq event stamp: 12826959 [ 2034.769992] hardirqs last enabled at (12826969): [] console_unlock+0x106/0x120 [ 2034.778860] hardirqs last disabled at (12826978): [] console_unlock+0xeb/0x120 [ 2034.787639] softirqs last enabled at (12826926): [] __do_softirq+0x2be/0x371 [ 2034.796343] softirqs last disabled at (12826997): [] irq_exit_rcu+0xb0/0xd0 [ 2034.804867] ---[ end trace 0000000000000000 ]--- [ 2034.809489] ------------[ cut here ]------------ [ 2034.814104] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2034.825754] WARNING: CPU: 6 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2034.834707] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2034.890228] CPU: 6 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2034.899179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2034.906659] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2034.912150] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2034.930898] RSP: 0018:ffffc90001957d18 EFLAGS: 00010286 [ 2034.936125] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2034.943257] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2034.950390] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2034.957532] R10: 0000000000000001 R11: ffffc90001957ba8 R12: ffff8881d1008750 [ 2034.964665] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2034.971797] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2034.979884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2034.985630] CR2: 00007f9bd55dc020 CR3: 000000016696c003 CR4: 00000000003706e0 [ 2034.992765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2034.999896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2035.007029] Call Trace: [ 2035.009482] [ 2035.011587] ? __warn+0x85/0x180 [ 2035.014822] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.019700] ? report_bug+0xfc/0x1d0 [ 2035.023284] ? prb_read_valid+0x1b/0x20 [ 2035.027131] ? handle_bug+0x44/0x70 [ 2035.030620] ? exc_invalid_op+0x17/0x60 [ 2035.034460] ? asm_exc_invalid_op+0x1a/0x20 [ 2035.038657] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.043535] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.048412] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2035.053315] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2035.058478] ceph_write_iter+0x5aa/0x770 [ceph] [ 2035.063035] ? lock_is_held_type+0xaa/0x120 [ 2035.067227] vfs_write+0x324/0x4f0 [ 2035.070647] ksys_pwrite64+0x65/0xa0 [ 2035.074224] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2035.079364] do_syscall_64+0x38/0x80 [ 2035.082945] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2035.088003] RIP: 0033:0x7f48f0eeb93b [ 2035.091583] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2035.110329] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2035.117909] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2035.125048] RDX: 0000000000000800 RSI: 000000000207e080 RDI: 0000000000000006 [ 2035.132177] RBP: 000000000000298b R08: 0000000000000800 R09: 0000000000000003 [ 2035.139311] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d228 [ 2035.146443] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207e080 [ 2035.153589] [ 2035.155782] irq event stamp: 12748755 [ 2035.159444] hardirqs last enabled at (12748765): [] console_unlock+0x106/0x120 [ 2035.168309] hardirqs last disabled at (12748774): [] console_unlock+0xeb/0x120 [ 2035.177088] softirqs last enabled at (12748752): [] __do_softirq+0x2be/0x371 [ 2035.185783] softirqs last disabled at (12748745): [] irq_exit_rcu+0xb0/0xd0 [ 2035.194309] ---[ end trace 0000000000000000 ]--- [ 2035.198931] ------------[ cut here ]------------ [ 2035.203563] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2035.215226] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2035.224186] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2035.279714] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2035.288677] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2035.296154] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2035.301640] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2035.320385] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2035.325611] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2035.332745] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2035.339879] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2035.347013] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2035.354143] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2035.361276] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2035.369362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2035.375109] CR2: 00007fb39af77000 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2035.382243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2035.389374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2035.396507] Call Trace: [ 2035.398959] [ 2035.401066] ? __warn+0x85/0x180 [ 2035.404300] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.409179] ? report_bug+0xfc/0x1d0 [ 2035.412759] ? prb_read_valid+0x1b/0x20 [ 2035.416602] ? handle_bug+0x44/0x70 [ 2035.420091] ? exc_invalid_op+0x17/0x60 [ 2035.423938] ? asm_exc_invalid_op+0x1a/0x20 [ 2035.428134] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.433015] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.437898] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2035.442800] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2035.447961] ceph_write_iter+0x5aa/0x770 [ceph] [ 2035.452513] ? lock_is_held_type+0xaa/0x120 [ 2035.456705] vfs_write+0x324/0x4f0 [ 2035.460123] ksys_pwrite64+0x65/0xa0 [ 2035.463700] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2035.468842] do_syscall_64+0x38/0x80 [ 2035.472421] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2035.477476] RIP: 0033:0x7f48f0eeb93b [ 2035.481053] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2035.499799] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2035.507365] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2035.514497] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2035.521629] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2035.528762] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2035.535896] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2035.543045] [ 2035.545238] irq event stamp: 12829053 [ 2035.548904] hardirqs last enabled at (12829063): [] console_unlock+0x106/0x120 [ 2035.557771] hardirqs last disabled at (12829072): [] console_unlock+0xeb/0x120 [ 2035.566558] softirqs last enabled at (12829092): [] __do_softirq+0x2be/0x371 [ 2035.575252] softirqs last disabled at (12829083): [] irq_exit_rcu+0xb0/0xd0 [ 2035.583770] ---[ end trace 0000000000000000 ]--- [ 2035.588393] ------------[ cut here ]------------ [ 2035.593100] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2035.604745] WARNING: CPU: 1 PID: 78788 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2035.613699] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2035.669229] CPU: 1 PID: 78788 Comm: kworker/1:0 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2035.678614] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2035.686142] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2035.691659] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2035.697142] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2035.715891] RSP: 0018:ffffc90001a67958 EFLAGS: 00010286 [ 2035.721114] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2035.728250] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2035.735384] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2035.742514] R10: 0000000000000001 R11: ffffc90001a677e8 R12: ffff8881e4d0b838 [ 2035.749647] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2035.756780] FS: 0000000000000000(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2035.764867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2035.770610] CR2: 00007fb390027148 CR3: 000000017a308005 CR4: 00000000003706e0 [ 2035.777747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2035.784878] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2035.792011] Call Trace: [ 2035.794464] [ 2035.796571] ? __warn+0x85/0x180 [ 2035.799805] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.804694] ? report_bug+0xfc/0x1d0 [ 2035.808278] ? prb_read_valid+0x1b/0x20 [ 2035.812125] ? handle_bug+0x44/0x70 [ 2035.815621] ? exc_invalid_op+0x17/0x60 [ 2035.819461] ? asm_exc_invalid_op+0x1a/0x20 [ 2035.823672] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.828554] ? __list_add_valid_or_report+0x7d/0xa0 [ 2035.833437] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2035.838345] ceph_check_caps+0x999/0xae0 [ceph] [ 2035.842919] ? __lock_acquire+0xbfa/0x1e20 [ 2035.847025] ? __lock_acquire+0xb70/0x1e20 [ 2035.851141] ? lock_acquire+0xcb/0x2b0 [ 2035.854899] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2035.860486] ? find_held_lock+0x2d/0x90 [ 2035.864332] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2035.870007] ? lock_release+0xc5/0x260 [ 2035.873779] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2035.879286] writepages_finish+0x302/0x610 [ceph] [ 2035.884025] __complete_request+0x1c/0x80 [libceph] [ 2035.888926] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2035.893485] ? lock_release+0xc5/0x260 [ 2035.897252] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2035.901969] ceph_con_process_message+0x74/0x150 [libceph] [ 2035.907470] ? calc_signature+0xdb/0x100 [libceph] [ 2035.912290] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2035.917633] ? process_scheduled_works+0x22a/0x510 [ 2035.922440] ? process_scheduled_works+0x22a/0x510 [ 2035.927240] ceph_con_workfn+0x344/0x750 [libceph] [ 2035.932058] ? process_scheduled_works+0x22a/0x510 [ 2035.936850] process_scheduled_works+0x2a4/0x510 [ 2035.941490] worker_thread+0x188/0x340 [ 2035.945255] ? rescuer_thread+0x310/0x310 [ 2035.949270] kthread+0xf5/0x130 [ 2035.952418] ? kthread_complete_and_exit+0x20/0x20 [ 2035.957213] ret_from_fork+0x31/0x40 [ 2035.960797] ? kthread_complete_and_exit+0x20/0x20 [ 2035.965593] ret_from_fork_asm+0x11/0x20 [ 2035.969541] [ 2035.971734] irq event stamp: 627001 [ 2035.975226] hardirqs last enabled at (627017): [] console_unlock+0x106/0x120 [ 2035.983921] hardirqs last disabled at (627032): [] console_unlock+0xeb/0x120 [ 2035.992529] softirqs last enabled at (627050): [] __do_softirq+0x2be/0x371 [ 2036.001054] softirqs last disabled at (627061): [] irq_exit_rcu+0xb0/0xd0 [ 2036.009400] ---[ end trace 0000000000000000 ]--- [ 2036.014020] ------------[ cut here ]------------ [ 2036.018640] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2036.030284] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2036.038978] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2036.094455] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2036.103582] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2036.111062] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2036.116563] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2036.122049] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2036.140793] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2036.146020] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2036.153147] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2036.160278] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2036.167412] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881e4d31ae0 [ 2036.174544] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2036.181677] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2036.189761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.195500] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2036.202633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2036.209756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2036.216882] Call Trace: [ 2036.219335] [ 2036.221443] ? __warn+0x85/0x180 [ 2036.224674] ? __list_add_valid_or_report+0x7d/0xa0 [ 2036.229553] ? report_bug+0xfc/0x1d0 [ 2036.233133] ? prb_read_valid+0x1b/0x20 [ 2036.236972] ? handle_bug+0x44/0x70 [ 2036.240464] ? exc_invalid_op+0x17/0x60 [ 2036.244304] ? asm_exc_invalid_op+0x1a/0x20 [ 2036.248494] ? __list_add_valid_or_report+0x7d/0xa0 [ 2036.253371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2036.258248] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2036.263139] ceph_add_cap+0x20b/0x620 [ceph] [ 2036.267424] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2036.272145] ceph_fill_trace+0x143/0xf30 [ceph] [ 2036.276684] ? __mutex_lock+0x97/0xce0 [ 2036.280435] ? __mutex_lock+0xaec/0xce0 [ 2036.284275] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2036.288817] ? lock_downgrade+0xfd/0x250 [ 2036.292746] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2036.297122] ceph_con_process_message+0x74/0x150 [libceph] [ 2036.302614] ? calc_signature+0xdb/0x100 [libceph] [ 2036.307418] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2036.312741] ? process_scheduled_works+0x22a/0x510 [ 2036.317532] ? process_scheduled_works+0x22a/0x510 [ 2036.322326] ceph_con_workfn+0x344/0x750 [libceph] [ 2036.327134] ? process_scheduled_works+0x22a/0x510 [ 2036.331923] process_scheduled_works+0x2a4/0x510 [ 2036.336550] worker_thread+0x188/0x340 [ 2036.340299] ? rescuer_thread+0x310/0x310 [ 2036.344311] kthread+0xf5/0x130 [ 2036.347457] ? kthread_complete_and_exit+0x20/0x20 [ 2036.352251] ret_from_fork+0x31/0x40 [ 2036.355827] ? kthread_complete_and_exit+0x20/0x20 [ 2036.360624] ret_from_fork_asm+0x11/0x20 [ 2036.364553] [ 2036.366740] irq event stamp: 8906303 [ 2036.370317] hardirqs last enabled at (8906313): [] console_unlock+0x106/0x120 [ 2036.379096] hardirqs last disabled at (8906322): [] console_unlock+0xeb/0x120 [ 2036.387783] softirqs last enabled at (8906280): [] __do_softirq+0x2be/0x371 [ 2036.396388] softirqs last disabled at (8906275): [] irq_exit_rcu+0xb0/0xd0 [ 2036.404822] ---[ end trace 0000000000000000 ]--- [ 2036.409441] ------------[ cut here ]------------ [ 2036.414063] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2036.425709] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2036.434664] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2036.490184] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2036.499137] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2036.506614] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2036.512101] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2036.530848] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2036.536072] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2036.543207] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2036.550340] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2036.557471] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2036.564606] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2036.571737] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2036.579826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.585572] CR2: 00007f9bd55ef000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2036.592702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2036.599836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2036.606968] Call Trace: [ 2036.609421] [ 2036.611528] ? __warn+0x85/0x180 [ 2036.614761] ? __list_add_valid_or_report+0x7d/0xa0 [ 2036.619641] ? report_bug+0xfc/0x1d0 [ 2036.623219] ? prb_read_valid+0x1b/0x20 [ 2036.627061] ? handle_bug+0x44/0x70 [ 2036.630551] ? exc_invalid_op+0x17/0x60 [ 2036.634392] ? asm_exc_invalid_op+0x1a/0x20 [ 2036.638587] ? __list_add_valid_or_report+0x7d/0xa0 [ 2036.643468] ? __list_add_valid_or_report+0x7d/0xa0 [ 2036.648346] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2036.653244] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2036.658404] ceph_write_iter+0x5aa/0x770 [ceph] [ 2036.662960] ? lock_is_held_type+0xaa/0x120 [ 2036.667148] vfs_write+0x324/0x4f0 [ 2036.670567] ksys_pwrite64+0x65/0xa0 [ 2036.674145] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2036.679287] do_syscall_64+0x38/0x80 [ 2036.682865] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2036.687917] RIP: 0033:0x7f48f0eeb93b [ 2036.691498] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2036.710242] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2036.717808] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2036.724941] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2036.732073] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2036.739206] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2036.746340] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2036.753485] [ 2036.755673] irq event stamp: 12811477 [ 2036.759339] hardirqs last enabled at (12811487): [] console_unlock+0x106/0x120 [ 2036.768205] hardirqs last disabled at (12811496): [] console_unlock+0xeb/0x120 [ 2036.776984] softirqs last enabled at (12811288): [] __do_softirq+0x2be/0x371 [ 2036.785678] softirqs last disabled at (12811283): [] irq_exit_rcu+0xb0/0xd0 [ 2036.794197] ---[ end trace 0000000000000000 ]--- [ 2036.798819] ------------[ cut here ]------------ [ 2036.803439] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2036.815080] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2036.823776] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2036.879254] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2036.888377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2036.895850] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2036.901344] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2036.906830] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2036.925577] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2036.930801] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2036.937925] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2036.945051] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2036.952183] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881e4d31ae0 [ 2036.959317] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2036.966449] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2036.974534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.980271] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2036.987396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2036.994521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2037.001654] Call Trace: [ 2037.004105] [ 2037.006205] ? __warn+0x85/0x180 [ 2037.009439] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.014317] ? report_bug+0xfc/0x1d0 [ 2037.017896] ? prb_read_valid+0x1b/0x20 [ 2037.021737] ? handle_bug+0x44/0x70 [ 2037.025231] ? exc_invalid_op+0x17/0x60 [ 2037.029067] ? asm_exc_invalid_op+0x1a/0x20 [ 2037.033258] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.038133] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.043011] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2037.047903] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2037.052540] ceph_fill_trace+0x143/0xf30 [ceph] [ 2037.057085] ? __mutex_lock+0x97/0xce0 [ 2037.060840] ? __mutex_lock+0xaec/0xce0 [ 2037.064679] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2037.069220] ? lock_downgrade+0xfd/0x250 [ 2037.073154] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2037.077533] ceph_con_process_message+0x74/0x150 [libceph] [ 2037.083026] ? calc_signature+0xdb/0x100 [libceph] [ 2037.087830] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2037.093153] ? process_scheduled_works+0x22a/0x510 [ 2037.097944] ? process_scheduled_works+0x22a/0x510 [ 2037.102734] ceph_con_workfn+0x344/0x750 [libceph] [ 2037.107539] ? process_scheduled_works+0x22a/0x510 [ 2037.112329] process_scheduled_works+0x2a4/0x510 [ 2037.116954] worker_thread+0x188/0x340 [ 2037.120710] ? rescuer_thread+0x310/0x310 [ 2037.124722] kthread+0xf5/0x130 [ 2037.127868] ? kthread_complete_and_exit+0x20/0x20 [ 2037.132661] ret_from_fork+0x31/0x40 [ 2037.136242] ? kthread_complete_and_exit+0x20/0x20 [ 2037.141034] ret_from_fork_asm+0x11/0x20 [ 2037.144968] [ 2037.147161] irq event stamp: 8908185 [ 2037.150733] hardirqs last enabled at (8908195): [] console_unlock+0x106/0x120 [ 2037.159512] hardirqs last disabled at (8908204): [] console_unlock+0xeb/0x120 [ 2037.168203] softirqs last enabled at (8908152): [] __do_softirq+0x2be/0x371 [ 2037.176809] softirqs last disabled at (8908147): [] irq_exit_rcu+0xb0/0xd0 [ 2037.185242] ---[ end trace 0000000000000000 ]--- [ 2037.189853] ------------[ cut here ]------------ [ 2037.194485] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2037.206137] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2037.215095] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2037.270629] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2037.279594] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2037.287073] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2037.292559] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2037.311308] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2037.316540] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2037.323675] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2037.330807] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2037.337941] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d37cb0 [ 2037.345073] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2037.352203] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2037.360290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2037.366036] CR2: 00007fb39af77000 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2037.373174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2037.380310] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2037.387444] Call Trace: [ 2037.389896] [ 2037.392002] ? __warn+0x85/0x180 [ 2037.395236] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.400122] ? report_bug+0xfc/0x1d0 [ 2037.403703] ? prb_read_valid+0x1b/0x20 [ 2037.407555] ? handle_bug+0x44/0x70 [ 2037.411052] ? exc_invalid_op+0x17/0x60 [ 2037.414894] ? asm_exc_invalid_op+0x1a/0x20 [ 2037.419098] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.423986] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2037.428895] try_get_cap_refs+0x433/0x880 [ceph] [ 2037.433539] ? _raw_spin_unlock+0x23/0x40 [ 2037.437565] __ceph_get_caps+0xf8/0x760 [ceph] [ 2037.442036] ? __vfs_getxattr+0x8d/0xb0 [ 2037.445888] ? dentry_needs_remove_privs+0x31/0x50 [ 2037.450688] ? __file_remove_privs+0x89/0x110 [ 2037.455046] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2037.459770] ? lock_release+0xc5/0x260 [ 2037.463527] ? ceph_write_iter+0x260/0x770 [ceph] [ 2037.468246] ceph_write_iter+0x260/0x770 [ceph] [ 2037.472805] ? lock_is_held_type+0xaa/0x120 [ 2037.476996] vfs_write+0x324/0x4f0 [ 2037.480416] ksys_pwrite64+0x65/0xa0 [ 2037.484001] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2037.489144] do_syscall_64+0x38/0x80 [ 2037.492730] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2037.497783] RIP: 0033:0x7f48f0eeb93b [ 2037.501362] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2037.520114] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2037.527689] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2037.534824] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2037.541957] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2037.549088] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2037.556221] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2037.563369] [ 2037.565564] irq event stamp: 12838795 [ 2037.569229] hardirqs last enabled at (12838805): [] console_unlock+0x106/0x120 [ 2037.578099] hardirqs last disabled at (12838820): [] console_unlock+0xeb/0x120 [ 2037.586887] softirqs last enabled at (12838834): [] __do_softirq+0x2be/0x371 [ 2037.595586] softirqs last disabled at (12838829): [] irq_exit_rcu+0xb0/0xd0 [ 2037.604105] ---[ end trace 0000000000000000 ]--- [ 2037.608725] ------------[ cut here ]------------ [ 2037.613343] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2037.624995] WARNING: CPU: 6 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2037.633953] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2037.689477] CPU: 6 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2037.698428] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2037.705906] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2037.711392] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2037.730139] RSP: 0018:ffffc90001957c88 EFLAGS: 00010286 [ 2037.735364] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2037.742498] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2037.749631] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2037.756765] R10: 0000000000000001 R11: ffffc90001957b18 R12: ffff8881d1008750 [ 2037.763895] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2037.771029] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2037.779114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2037.784859] CR2: 00007f9bd4a12008 CR3: 000000016696c003 CR4: 00000000003706e0 [ 2037.791994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2037.799127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2037.806258] Call Trace: [ 2037.808711] [ 2037.810819] ? __warn+0x85/0x180 [ 2037.814053] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.818943] ? report_bug+0xfc/0x1d0 [ 2037.822532] ? prb_read_valid+0x1b/0x20 [ 2037.826380] ? handle_bug+0x44/0x70 [ 2037.829880] ? exc_invalid_op+0x17/0x60 [ 2037.833802] ? asm_exc_invalid_op+0x1a/0x20 [ 2037.837999] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.842880] ? __list_add_valid_or_report+0x7d/0xa0 [ 2037.847768] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2037.852678] ceph_open+0x16b/0x440 [ceph] [ 2037.856716] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2037.861879] do_dentry_open+0x172/0x530 [ 2037.865728] path_openat+0x743/0x9d0 [ 2037.869321] do_filp_open+0xb8/0x120 [ 2037.872909] ? lock_release+0xc5/0x260 [ 2037.876666] ? _raw_spin_unlock+0x23/0x40 [ 2037.880684] ? alloc_fd+0x13f/0x200 [ 2037.884192] do_sys_openat2+0x249/0x290 [ 2037.888039] do_sys_open+0x4f/0x80 [ 2037.891450] do_syscall_64+0x38/0x80 [ 2037.895038] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2037.900095] RIP: 0033:0x7f48f0eed2f2 [ 2037.903677] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2037.922425] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2037.929995] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2037.937129] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2037.944262] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2037.951396] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2037.958527] R13: 00007f48f183d228 R14: 00000000020767c0 R15: 000000000000298c [ 2037.965674] [ 2037.967870] irq event stamp: 12758039 [ 2037.971535] hardirqs last enabled at (12758049): [] console_unlock+0x106/0x120 [ 2037.980401] hardirqs last disabled at (12758058): [] console_unlock+0xeb/0x120 [ 2037.989188] softirqs last enabled at (12758078): [] __do_softirq+0x2be/0x371 [ 2037.997883] softirqs last disabled at (12758067): [] irq_exit_rcu+0xb0/0xd0 [ 2038.006411] ---[ end trace 0000000000000000 ]--- [ 2038.011030] ------------[ cut here ]------------ [ 2038.015647] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2038.027305] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2038.036259] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2038.091737] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2038.100690] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2038.108166] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2038.113653] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2038.132400] RSP: 0018:ffffc90001f7fc28 EFLAGS: 00010286 [ 2038.137632] RAX: 0000000000000000 RBX: ffff8881d0bea720 RCX: 0000000000000000 [ 2038.144768] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2038.151902] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2038.159035] R10: 0000000000000001 R11: ffffc90001f7fab8 R12: ffff8881d0beae70 [ 2038.166166] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2038.173297] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2038.181385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2038.187129] CR2: 00007fb38f7e0000 CR3: 0000000113532001 CR4: 00000000003706e0 [ 2038.194265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2038.201396] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2038.208529] Call Trace: [ 2038.210985] [ 2038.213090] ? __warn+0x85/0x180 [ 2038.216331] ? __list_add_valid_or_report+0x7d/0xa0 [ 2038.221213] ? report_bug+0xfc/0x1d0 [ 2038.224799] ? prb_read_valid+0x1b/0x20 [ 2038.228640] ? handle_bug+0x44/0x70 [ 2038.232138] ? exc_invalid_op+0x17/0x60 [ 2038.235983] ? asm_exc_invalid_op+0x1a/0x20 [ 2038.240180] ? __list_add_valid_or_report+0x7d/0xa0 [ 2038.245064] ? __list_add_valid_or_report+0x7d/0xa0 [ 2038.249950] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2038.254859] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2038.260014] ? __ceph_caps_issued+0x78/0x110 [ceph] [ 2038.264922] __ceph_setattr+0x2c0/0xea0 [ceph] [ 2038.269395] ceph_setattr+0xaa/0x100 [ceph] [ 2038.273606] notify_change+0x237/0x4f0 [ 2038.277369] ? vfs_utimes+0x130/0x220 [ 2038.281037] vfs_utimes+0x130/0x220 [ 2038.284538] do_utimes+0xb7/0x120 [ 2038.287867] __x64_sys_utime+0xa1/0xd0 [ 2038.291625] do_syscall_64+0x38/0x80 [ 2038.295209] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2038.300266] RIP: 0033:0x7f48f0eec99b [ 2038.303849] Code: 64 48 33 0c 25 28 00 00 00 75 06 48 83 c4 10 5b c3 e8 79 09 02 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 84 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d bd 24 2d 00 f7 d8 64 89 01 48 [ 2038.322592] RSP: 002b:00007ffc53d5fff8 EFLAGS: 00000206 ORIG_RAX: 0000000000000084 [ 2038.330159] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eec99b [ 2038.337292] RDX: 00007ffc53d60010 RSI: 00007ffc53d60000 RDI: 000000000207e960 [ 2038.344425] RBP: 00007f48f183d000 R08: 0000000002076605 R09: 0000000000000000 [ 2038.351560] R10: 00007f48f0f711e0 R11: 0000000000000206 R12: 0000000000000004 [ 2038.358692] R13: 00000000000003ec R14: 00007f48f183d000 R15: 00000000020766e0 [ 2038.365835] [ 2038.368024] irq event stamp: 12849085 [ 2038.371689] hardirqs last enabled at (12849095): [] console_unlock+0x106/0x120 [ 2038.380558] hardirqs last disabled at (12849104): [] console_unlock+0xeb/0x120 [ 2038.389336] softirqs last enabled at (12849052): [] __do_softirq+0x2be/0x371 [ 2038.398033] softirqs last disabled at (12849123): [] irq_exit_rcu+0xb0/0xd0 [ 2038.406557] ---[ end trace 0000000000000000 ]--- [ 2038.411180] ------------[ cut here ]------------ [ 2038.415804] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2038.427462] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2038.436416] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2038.491950] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2038.500904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2038.508385] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2038.513871] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2038.532616] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2038.537842] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2038.544975] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2038.552109] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2038.559242] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d09f8750 [ 2038.566375] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2038.573508] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2038.581592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2038.587339] CR2: 00007f9bd55ef000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2038.594473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2038.601604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2038.608737] Call Trace: [ 2038.611192] [ 2038.613299] ? __warn+0x85/0x180 [ 2038.616531] ? __list_add_valid_or_report+0x7d/0xa0 [ 2038.621417] ? report_bug+0xfc/0x1d0 [ 2038.624997] ? prb_read_valid+0x1b/0x20 [ 2038.628842] ? handle_bug+0x44/0x70 [ 2038.632339] ? exc_invalid_op+0x17/0x60 [ 2038.636177] ? asm_exc_invalid_op+0x1a/0x20 [ 2038.640374] ? __list_add_valid_or_report+0x7d/0xa0 [ 2038.645254] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2038.650154] try_get_cap_refs+0x433/0x880 [ceph] [ 2038.654797] ? _raw_spin_unlock+0x23/0x40 [ 2038.658825] __ceph_get_caps+0xf8/0x760 [ceph] [ 2038.663288] ? __vfs_getxattr+0x8d/0xb0 [ 2038.667133] ? dentry_needs_remove_privs+0x31/0x50 [ 2038.671929] ? __file_remove_privs+0x89/0x110 [ 2038.676288] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2038.681013] ? lock_release+0xc5/0x260 [ 2038.684774] ? ceph_write_iter+0x260/0x770 [ceph] [ 2038.689498] ceph_write_iter+0x260/0x770 [ceph] [ 2038.694055] ? lock_is_held_type+0xaa/0x120 [ 2038.698248] vfs_write+0x324/0x4f0 [ 2038.701666] ksys_pwrite64+0x65/0xa0 [ 2038.705254] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2038.710404] do_syscall_64+0x38/0x80 [ 2038.713993] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2038.719050] RIP: 0033:0x7f48f0eeb93b [ 2038.722629] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2038.741374] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2038.748946] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2038.756083] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2038.763215] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2038.770350] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2038.777481] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2038.784628] [ 2038.786823] irq event stamp: 12816015 [ 2038.790491] hardirqs last enabled at (12816025): [] console_unlock+0x106/0x120 [ 2038.799357] hardirqs last disabled at (12816034): [] console_unlock+0xeb/0x120 [ 2038.808136] softirqs last enabled at (12815952): [] __do_softirq+0x2be/0x371 [ 2038.816829] softirqs last disabled at (12815945): [] irq_exit_rcu+0xb0/0xd0 [ 2038.825349] ---[ end trace 0000000000000000 ]--- [ 2038.829970] ------------[ cut here ]------------ [ 2038.834597] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2038.846250] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2038.855204] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2038.910742] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2038.919693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2038.927174] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2038.932658] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2038.951414] RSP: 0018:ffffc90001f7fc28 EFLAGS: 00010286 [ 2038.956641] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2038.963774] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2038.970906] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2038.978037] R10: 0000000000000001 R11: ffffc90001f7fab8 R12: ffff8881e4d0b838 [ 2038.985173] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2038.992305] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2039.000389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2039.006136] CR2: 00007f9bd4aa9020 CR3: 0000000113532006 CR4: 00000000003706e0 [ 2039.013268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2039.020404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2039.027534] Call Trace: [ 2039.029988] [ 2039.032096] ? __warn+0x85/0x180 [ 2039.035327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.040208] ? report_bug+0xfc/0x1d0 [ 2039.043787] ? prb_read_valid+0x1b/0x20 [ 2039.047628] ? handle_bug+0x44/0x70 [ 2039.051117] ? exc_invalid_op+0x17/0x60 [ 2039.054958] ? asm_exc_invalid_op+0x1a/0x20 [ 2039.059154] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.064033] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.068911] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2039.073813] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2039.078979] __ceph_setattr+0x2c0/0xea0 [ceph] [ 2039.083461] ceph_setattr+0xaa/0x100 [ceph] [ 2039.087659] notify_change+0x237/0x4f0 [ 2039.091418] ? vfs_utimes+0x130/0x220 [ 2039.095083] vfs_utimes+0x130/0x220 [ 2039.098587] do_utimes+0xb7/0x120 [ 2039.101909] __x64_sys_utime+0xa1/0xd0 [ 2039.105673] do_syscall_64+0x38/0x80 [ 2039.109255] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2039.114308] RIP: 0033:0x7f48f0eec99b [ 2039.117887] Code: 64 48 33 0c 25 28 00 00 00 75 06 48 83 c4 10 5b c3 e8 79 09 02 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 84 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d bd 24 2d 00 f7 d8 64 89 01 48 [ 2039.136632] RSP: 002b:00007ffc53d5fff8 EFLAGS: 00000206 ORIG_RAX: 0000000000000084 [ 2039.144198] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eec99b [ 2039.151334] RDX: 00007ffc53d60010 RSI: 00007ffc53d60000 RDI: 0000000002098170 [ 2039.158465] RBP: 00007f48f183d000 R08: 0000000002076605 R09: 0000000000000000 [ 2039.165597] R10: 00007f48f0f711e0 R11: 0000000000000206 R12: 0000000000000004 [ 2039.172730] R13: 00000000000003ec R14: 00007f48f183d000 R15: 00000000020766e0 [ 2039.179876] [ 2039.182064] irq event stamp: 12851109 [ 2039.185730] hardirqs last enabled at (12851119): [] console_unlock+0x106/0x120 [ 2039.194597] hardirqs last disabled at (12851128): [] console_unlock+0xeb/0x120 [ 2039.203375] softirqs last enabled at (12851056): [] __do_softirq+0x2be/0x371 [ 2039.212068] softirqs last disabled at (12851051): [] irq_exit_rcu+0xb0/0xd0 [ 2039.220590] ---[ end trace 0000000000000000 ]--- [ 2039.225208] ------------[ cut here ]------------ [ 2039.229829] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2039.241482] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2039.250437] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2039.305965] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2039.314919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2039.322397] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2039.327883] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2039.346630] RSP: 0018:ffffc90001fdfaa8 EFLAGS: 00010282 [ 2039.351856] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2039.358988] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2039.366122] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2039.373254] R10: 0000000000000001 R11: ffffc90001fdf938 R12: ffff8881e4d31ae0 [ 2039.380387] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2039.387527] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2039.395616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2039.401362] CR2: 000056400e03adc0 CR3: 0000000179402004 CR4: 00000000003706e0 [ 2039.408494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2039.415627] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2039.422760] Call Trace: [ 2039.425213] [ 2039.427319] ? __warn+0x85/0x180 [ 2039.430552] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.435434] ? report_bug+0xfc/0x1d0 [ 2039.439010] ? prb_read_valid+0x1b/0x20 [ 2039.442852] ? handle_bug+0x44/0x70 [ 2039.446343] ? exc_invalid_op+0x17/0x60 [ 2039.450183] ? asm_exc_invalid_op+0x1a/0x20 [ 2039.454377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.459258] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.464134] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2039.469035] ceph_open+0x16b/0x440 [ceph] [ 2039.473068] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2039.478221] do_dentry_open+0x172/0x530 [ 2039.482062] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2039.486713] lookup_open+0x37b/0x630 [ 2039.490299] ? verify_cpu+0x20/0x100 [ 2039.493885] ? verify_cpu+0x20/0x100 [ 2039.497473] path_openat+0x25a/0x9d0 [ 2039.501065] do_filp_open+0xb8/0x120 [ 2039.504653] ? lock_release+0xc5/0x260 [ 2039.508409] ? _raw_spin_unlock+0x23/0x40 [ 2039.512426] ? alloc_fd+0x13f/0x200 [ 2039.515929] do_sys_openat2+0x249/0x290 [ 2039.519773] do_sys_open+0x4f/0x80 [ 2039.523184] do_syscall_64+0x38/0x80 [ 2039.526763] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2039.531814] RIP: 0033:0x7f48f0eed2f2 [ 2039.535394] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2039.554140] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2039.561707] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2039.568837] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2039.575970] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2039.583106] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2039.590236] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029af [ 2039.597385] [ 2039.599580] irq event stamp: 12717715 [ 2039.603246] hardirqs last enabled at (12717725): [] console_unlock+0x106/0x120 [ 2039.612112] hardirqs last disabled at (12717734): [] console_unlock+0xeb/0x120 [ 2039.620892] softirqs last enabled at (12717576): [] __do_softirq+0x2be/0x371 [ 2039.629584] softirqs last disabled at (12717569): [] irq_exit_rcu+0xb0/0xd0 [ 2039.638104] ---[ end trace 0000000000000000 ]--- [ 2039.642734] ------------[ cut here ]------------ [ 2039.647355] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2039.659017] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2039.667970] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2039.723501] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2039.732452] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2039.739931] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2039.745418] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2039.764171] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [ 2039.769399] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2039.776530] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2039.783663] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2039.790797] R10: 0000000000000001 R11: ffffc90001f7fa70 R12: ffff8881e4d0b838 [ 2039.797930] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2039.805061] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2039.813149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2039.818893] CR2: 00007f9bd464f988 CR3: 0000000113532006 CR4: 00000000003706e0 [ 2039.826026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2039.833159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2039.840292] Call Trace: [ 2039.842745] [ 2039.844853] ? __warn+0x85/0x180 [ 2039.848085] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.852966] ? report_bug+0xfc/0x1d0 [ 2039.856544] ? prb_read_valid+0x1b/0x20 [ 2039.860391] ? handle_bug+0x44/0x70 [ 2039.863885] ? exc_invalid_op+0x17/0x60 [ 2039.867725] ? asm_exc_invalid_op+0x1a/0x20 [ 2039.871920] ? __list_add_valid_or_report+0x7d/0xa0 [ 2039.876801] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2039.881699] try_get_cap_refs+0x433/0x880 [ceph] [ 2039.886334] ? _raw_spin_unlock+0x23/0x40 [ 2039.890354] __ceph_get_caps+0xf8/0x760 [ceph] [ 2039.894821] ? __vfs_getxattr+0x8d/0xb0 [ 2039.898670] ? dentry_needs_remove_privs+0x31/0x50 [ 2039.903471] ? __file_remove_privs+0x89/0x110 [ 2039.907833] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2039.912558] ? lock_release+0xc5/0x260 [ 2039.916314] ? ceph_write_iter+0x260/0x770 [ceph] [ 2039.921035] ceph_write_iter+0x260/0x770 [ceph] [ 2039.925601] ? lock_is_held_type+0xaa/0x120 [ 2039.929794] vfs_write+0x324/0x4f0 [ 2039.933213] ksys_pwrite64+0x65/0xa0 [ 2039.936790] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2039.941931] do_syscall_64+0x38/0x80 [ 2039.945509] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2039.950561] RIP: 0033:0x7f48f0eeb93b [ 2039.954139] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2039.972888] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2039.980452] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2039.987586] RDX: 0000000000001000 RSI: 000000000207eb20 RDI: 0000000000000005 [ 2039.994718] RBP: 0000000000002a15 R08: 0000000000001000 R09: 0000000000000003 [ 2040.001850] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d000 [ 2040.008985] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207eb20 [ 2040.016132] [ 2040.018325] irq event stamp: 12853129 [ 2040.021992] hardirqs last enabled at (12853139): [] console_unlock+0x106/0x120 [ 2040.030862] hardirqs last disabled at (12853160): [] console_unlock+0xeb/0x120 [ 2040.039641] softirqs last enabled at (12853176): [] __do_softirq+0x2be/0x371 [ 2040.048331] softirqs last disabled at (12853169): [] irq_exit_rcu+0xb0/0xd0 [ 2040.056850] ---[ end trace 0000000000000000 ]--- [ 2040.061470] ------------[ cut here ]------------ [ 2040.066091] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2040.077745] WARNING: CPU: 1 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2040.086699] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2040.142220] CPU: 1 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2040.151174] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2040.158652] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2040.164136] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2040.182887] RSP: 0018:ffffc90001957bd0 EFLAGS: 00010282 [ 2040.188110] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2040.195243] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2040.202376] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2040.209509] R10: 0000000000000001 R11: ffffc90001957a60 R12: ffff8881d1008750 [ 2040.216641] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2040.223776] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2040.231859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2040.237606] CR2: 00007fb38ea6dd40 CR3: 000000016696c003 CR4: 00000000003706e0 [ 2040.244739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2040.251871] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2040.259005] Call Trace: [ 2040.261459] [ 2040.263566] ? __warn+0x85/0x180 [ 2040.266799] ? __list_add_valid_or_report+0x7d/0xa0 [ 2040.271677] ? report_bug+0xfc/0x1d0 [ 2040.275258] ? prb_read_valid+0x1b/0x20 [ 2040.279100] ? handle_bug+0x44/0x70 [ 2040.282598] ? exc_invalid_op+0x17/0x60 [ 2040.286438] ? asm_exc_invalid_op+0x1a/0x20 [ 2040.290632] ? __list_add_valid_or_report+0x7d/0xa0 [ 2040.295513] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2040.300412] try_get_cap_refs+0x433/0x880 [ceph] [ 2040.305065] __ceph_get_caps+0xf8/0x760 [ceph] [ 2040.309544] ? ceph_read_iter+0x134/0x860 [ceph] [ 2040.314180] ceph_read_iter+0x134/0x860 [ceph] [ 2040.318652] ? selinux_file_permission+0x119/0x150 [ 2040.323449] ? vfs_read+0x20f/0x300 [ 2040.326946] vfs_read+0x20f/0x300 [ 2040.330275] ksys_pread64+0x65/0xa0 [ 2040.333765] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2040.338907] do_syscall_64+0x38/0x80 [ 2040.342487] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2040.347538] RIP: 0033:0x7f48f0eeb88b [ 2040.351117] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2040.369862] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2040.377427] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2040.384563] RDX: 0000000000000800 RSI: 000000000207e080 RDI: 0000000000000006 [ 2040.391695] RBP: 000000000000298c R08: 000000000000005f R09: 0000000000000003 [ 2040.398827] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000800 [ 2040.405960] R13: 0000000000000800 R14: 000000000207e080 R15: 00007f48f183d228 [ 2040.413107] [ 2040.415302] irq event stamp: 12769053 [ 2040.418970] hardirqs last enabled at (12769063): [] console_unlock+0x106/0x120 [ 2040.427836] hardirqs last disabled at (12769072): [] console_unlock+0xeb/0x120 [ 2040.436613] softirqs last enabled at (12768784): [] __do_softirq+0x2be/0x371 [ 2040.445310] softirqs last disabled at (12769091): [] irq_exit_rcu+0xb0/0xd0 [ 2040.453826] ---[ end trace 0000000000000000 ]--- [ 2040.458448] ------------[ cut here ]------------ [ 2040.463067] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2040.474708] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2040.483404] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2040.538883] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2040.548005] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2040.555481] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2040.560990] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2040.566476] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2040.585224] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2040.590449] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2040.597581] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2040.604714] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2040.611845] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881c2641118 [ 2040.618985] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2040.626121] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2040.634215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2040.639969] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2040.647102] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2040.654226] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2040.661351] Call Trace: [ 2040.663796] [ 2040.665903] ? __warn+0x85/0x180 [ 2040.669139] ? __list_add_valid_or_report+0x7d/0xa0 [ 2040.674018] ? report_bug+0xfc/0x1d0 [ 2040.677602] ? prb_read_valid+0x1b/0x20 [ 2040.681443] ? handle_bug+0x44/0x70 [ 2040.684934] ? exc_invalid_op+0x17/0x60 [ 2040.688773] ? asm_exc_invalid_op+0x1a/0x20 [ 2040.692964] ? __list_add_valid_or_report+0x7d/0xa0 [ 2040.697840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2040.702720] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2040.707608] ceph_add_cap+0x20b/0x620 [ceph] [ 2040.711895] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2040.716624] ceph_fill_trace+0x143/0xf30 [ceph] [ 2040.721163] ? __mutex_lock+0x97/0xce0 [ 2040.724912] ? __mutex_lock+0xaec/0xce0 [ 2040.728745] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2040.733288] ? lock_downgrade+0xfd/0x250 [ 2040.737218] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2040.741599] ceph_con_process_message+0x74/0x150 [libceph] [ 2040.747093] ? calc_signature+0xdb/0x100 [libceph] [ 2040.751896] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2040.757220] ? process_scheduled_works+0x22a/0x510 [ 2040.762011] ? process_scheduled_works+0x22a/0x510 [ 2040.766801] ceph_con_workfn+0x344/0x750 [libceph] [ 2040.771606] ? process_scheduled_works+0x22a/0x510 [ 2040.776395] process_scheduled_works+0x2a4/0x510 [ 2040.781027] worker_thread+0x188/0x340 [ 2040.784787] ? rescuer_thread+0x310/0x310 [ 2040.788796] kthread+0xf5/0x130 [ 2040.791945] ? kthread_complete_and_exit+0x20/0x20 [ 2040.796738] ret_from_fork+0x31/0x40 [ 2040.800317] ? kthread_complete_and_exit+0x20/0x20 [ 2040.805107] ret_from_fork_asm+0x11/0x20 [ 2040.809046] [ 2040.811235] irq event stamp: 8916559 [ 2040.814815] hardirqs last enabled at (8916569): [] console_unlock+0x106/0x120 [ 2040.823594] hardirqs last disabled at (8916578): [] console_unlock+0xeb/0x120 [ 2040.832287] softirqs last enabled at (8916516): [] __do_softirq+0x2be/0x371 [ 2040.840892] softirqs last disabled at (8916511): [] irq_exit_rcu+0xb0/0xd0 [ 2040.849327] ---[ end trace 0000000000000000 ]--- [ 2040.853948] ------------[ cut here ]------------ [ 2040.858569] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2040.870212] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2040.879168] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2040.934690] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2040.943649] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2040.951141] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2040.956633] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2040.975383] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2040.980615] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2040.987746] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2040.994886] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2041.002028] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2041.009163] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2041.016297] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2041.024383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2041.030135] CR2: 00007fb39af77000 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2041.037270] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2041.044406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2041.051538] Call Trace: [ 2041.053987] [ 2041.056101] ? __warn+0x85/0x180 [ 2041.059336] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.064219] ? report_bug+0xfc/0x1d0 [ 2041.067803] ? prb_read_valid+0x1b/0x20 [ 2041.071647] ? handle_bug+0x44/0x70 [ 2041.075143] ? exc_invalid_op+0x17/0x60 [ 2041.078986] ? asm_exc_invalid_op+0x1a/0x20 [ 2041.083187] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.088068] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.092946] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2041.097846] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2041.103006] ceph_write_iter+0x5aa/0x770 [ceph] [ 2041.107567] ? lock_is_held_type+0xaa/0x120 [ 2041.111759] vfs_write+0x324/0x4f0 [ 2041.115179] ksys_pwrite64+0x65/0xa0 [ 2041.118763] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2041.123907] do_syscall_64+0x38/0x80 [ 2041.127495] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2041.132552] RIP: 0033:0x7f48f0eeb93b [ 2041.136133] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2041.154878] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2041.162444] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2041.169576] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2041.176709] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2041.183845] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2041.190978] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2041.198123] [ 2041.200318] irq event stamp: 12848907 [ 2041.203985] hardirqs last enabled at (12848917): [] console_unlock+0x106/0x120 [ 2041.212852] hardirqs last disabled at (12848926): [] console_unlock+0xeb/0x120 [ 2041.221631] softirqs last enabled at (12848884): [] __do_softirq+0x2be/0x371 [ 2041.230323] softirqs last disabled at (12848879): [] irq_exit_rcu+0xb0/0xd0 [ 2041.238843] ---[ end trace 0000000000000000 ]--- [ 2041.243461] ------------[ cut here ]------------ [ 2041.248081] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2041.259729] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2041.268683] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2041.324213] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2041.333173] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2041.340652] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2041.346139] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2041.364884] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2041.370109] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2041.377243] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2041.384376] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2041.391510] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2041.398642] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2041.405774] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2041.413860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2041.419607] CR2: 00007f418ce07000 CR3: 0000000120c1e003 CR4: 00000000003706f0 [ 2041.426741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2041.433873] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2041.441004] Call Trace: [ 2041.443458] [ 2041.445565] ? __warn+0x85/0x180 [ 2041.448799] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.453678] ? report_bug+0xfc/0x1d0 [ 2041.457258] ? prb_read_valid+0x1b/0x20 [ 2041.461108] ? handle_bug+0x44/0x70 [ 2041.464604] ? exc_invalid_op+0x17/0x60 [ 2041.468446] ? asm_exc_invalid_op+0x1a/0x20 [ 2041.472639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.477522] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.482408] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2041.487307] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2041.492471] ceph_write_iter+0x5aa/0x770 [ceph] [ 2041.497044] ? lock_is_held_type+0xaa/0x120 [ 2041.501235] vfs_write+0x324/0x4f0 [ 2041.504690] ksys_pwrite64+0x65/0xa0 [ 2041.508270] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2041.513412] do_syscall_64+0x38/0x80 [ 2041.517009] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2041.522073] RIP: 0033:0x7f48f0eeb93b [ 2041.525662] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2041.544410] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2041.551974] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2041.559108] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2041.566241] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2041.573372] R10: 0000000000020000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2041.580505] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2041.587657] [ 2041.589848] irq event stamp: 12821949 [ 2041.593514] hardirqs last enabled at (12821959): [] console_unlock+0x106/0x120 [ 2041.602380] hardirqs last disabled at (12821968): [] console_unlock+0xeb/0x120 [ 2041.611161] softirqs last enabled at (12821936): [] __do_softirq+0x2be/0x371 [ 2041.619854] softirqs last disabled at (12821931): [] irq_exit_rcu+0xb0/0xd0 [ 2041.628372] ---[ end trace 0000000000000000 ]--- [ 2041.632994] ------------[ cut here ]------------ [ 2041.637613] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2041.649263] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2041.658221] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2041.713757] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2041.722722] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2041.730200] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2041.735687] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2041.754436] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2041.759677] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2041.766810] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2041.773941] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2041.781073] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d37cb0 [ 2041.788208] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2041.795340] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2041.803425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2041.809171] CR2: 00007fb39af77000 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2041.816305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2041.823442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2041.830578] Call Trace: [ 2041.833033] [ 2041.835138] ? __warn+0x85/0x180 [ 2041.838372] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.843253] ? report_bug+0xfc/0x1d0 [ 2041.846860] ? prb_read_valid+0x1b/0x20 [ 2041.850709] ? handle_bug+0x44/0x70 [ 2041.854208] ? exc_invalid_op+0x17/0x60 [ 2041.858047] ? asm_exc_invalid_op+0x1a/0x20 [ 2041.862249] ? __list_add_valid_or_report+0x7d/0xa0 [ 2041.867130] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2041.872029] try_get_cap_refs+0x433/0x880 [ceph] [ 2041.876669] ? _raw_spin_unlock+0x23/0x40 [ 2041.880690] __ceph_get_caps+0xf8/0x760 [ceph] [ 2041.885160] ? __vfs_getxattr+0x8d/0xb0 [ 2041.889007] ? dentry_needs_remove_privs+0x31/0x50 [ 2041.893804] ? __file_remove_privs+0x89/0x110 [ 2041.898162] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2041.902888] ? lock_release+0xc5/0x260 [ 2041.906645] ? ceph_write_iter+0x260/0x770 [ceph] [ 2041.911364] ceph_write_iter+0x260/0x770 [ceph] [ 2041.915921] ? lock_is_held_type+0xaa/0x120 [ 2041.920116] vfs_write+0x324/0x4f0 [ 2041.923535] ksys_pwrite64+0x65/0xa0 [ 2041.927121] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2041.932279] do_syscall_64+0x38/0x80 [ 2041.935865] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2041.940916] RIP: 0033:0x7f48f0eeb93b [ 2041.944495] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2041.963245] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2041.970819] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2041.977950] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2041.985089] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2041.992225] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2041.999357] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2042.006504] [ 2042.008699] irq event stamp: 12851165 [ 2042.012364] hardirqs last enabled at (12851175): [] console_unlock+0x106/0x120 [ 2042.021231] hardirqs last disabled at (12851184): [] console_unlock+0xeb/0x120 [ 2042.030019] softirqs last enabled at (12851206): [] __do_softirq+0x2be/0x371 [ 2042.038719] softirqs last disabled at (12851217): [] irq_exit_rcu+0xb0/0xd0 [ 2042.047241] ---[ end trace 0000000000000000 ]--- [ 2042.051862] ------------[ cut here ]------------ [ 2042.056476] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2042.068123] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2042.077082] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2042.132560] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2042.141519] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2042.148996] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2042.154483] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2042.173230] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2042.178458] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2042.185589] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2042.192722] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2042.199857] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d09f8750 [ 2042.206989] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2042.214120] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2042.222207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2042.227951] CR2: 00007f418ce07000 CR3: 0000000120c1e003 CR4: 00000000003706f0 [ 2042.235085] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2042.242216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2042.249352] Call Trace: [ 2042.251802] [ 2042.253911] ? __warn+0x85/0x180 [ 2042.257142] ? __list_add_valid_or_report+0x7d/0xa0 [ 2042.262026] ? report_bug+0xfc/0x1d0 [ 2042.265612] ? prb_read_valid+0x1b/0x20 [ 2042.269455] ? handle_bug+0x44/0x70 [ 2042.272951] ? exc_invalid_op+0x17/0x60 [ 2042.276792] ? asm_exc_invalid_op+0x1a/0x20 [ 2042.280994] ? __list_add_valid_or_report+0x7d/0xa0 [ 2042.285876] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2042.290803] try_get_cap_refs+0x433/0x880 [ceph] [ 2042.295448] ? _raw_spin_unlock+0x23/0x40 [ 2042.299468] __ceph_get_caps+0xf8/0x760 [ceph] [ 2042.303938] ? __vfs_getxattr+0x8d/0xb0 [ 2042.307787] ? dentry_needs_remove_privs+0x31/0x50 [ 2042.312585] ? __file_remove_privs+0x89/0x110 [ 2042.316945] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2042.321670] ? lock_release+0xc5/0x260 [ 2042.325432] ? ceph_write_iter+0x260/0x770 [ceph] [ 2042.330155] ceph_write_iter+0x260/0x770 [ceph] [ 2042.334709] ? lock_is_held_type+0xaa/0x120 [ 2042.338903] vfs_write+0x324/0x4f0 [ 2042.342319] ksys_pwrite64+0x65/0xa0 [ 2042.345901] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2042.351041] do_syscall_64+0x38/0x80 [ 2042.354621] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2042.359678] RIP: 0033:0x7f48f0eeb93b [ 2042.363259] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2042.382005] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2042.389573] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2042.396705] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2042.403838] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2042.410970] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2042.418101] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2042.425246] [ 2042.427438] irq event stamp: 12823979 [ 2042.431101] hardirqs last enabled at (12823989): [] console_unlock+0x106/0x120 [ 2042.439970] hardirqs last disabled at (12823998): [] console_unlock+0xeb/0x120 [ 2042.448747] softirqs last enabled at (12823926): [] __do_softirq+0x2be/0x371 [ 2042.457443] softirqs last disabled at (12823921): [] irq_exit_rcu+0xb0/0xd0 [ 2042.465972] ---[ end trace 0000000000000000 ]--- [ 2042.470592] ------------[ cut here ]------------ [ 2042.475215] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2042.486865] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2042.495820] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2042.551337] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2042.560290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2042.567770] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2042.573255] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2042.592003] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2042.597227] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2042.604363] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2042.611495] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2042.618628] R10: 0000000000000001 R11: ffffc90001fdfb18 R12: ffff8881e4d31ae0 [ 2042.625759] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2042.632893] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2042.640979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2042.646724] CR2: 000056400e03adc0 CR3: 0000000179402004 CR4: 00000000003706e0 [ 2042.653859] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2042.660991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2042.668124] Call Trace: [ 2042.670576] [ 2042.672683] ? __warn+0x85/0x180 [ 2042.675915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2042.680797] ? report_bug+0xfc/0x1d0 [ 2042.684383] ? prb_read_valid+0x1b/0x20 [ 2042.688224] ? handle_bug+0x44/0x70 [ 2042.691715] ? exc_invalid_op+0x17/0x60 [ 2042.695556] ? asm_exc_invalid_op+0x1a/0x20 [ 2042.699751] ? __list_add_valid_or_report+0x7d/0xa0 [ 2042.704630] ? __list_add_valid_or_report+0x7d/0xa0 [ 2042.709510] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2042.714409] ceph_open+0x16b/0x440 [ceph] [ 2042.718441] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2042.723592] do_dentry_open+0x172/0x530 [ 2042.727434] path_openat+0x743/0x9d0 [ 2042.731027] do_filp_open+0xb8/0x120 [ 2042.734616] ? lock_release+0xc5/0x260 [ 2042.738372] ? _raw_spin_unlock+0x23/0x40 [ 2042.742391] ? alloc_fd+0x13f/0x200 [ 2042.745889] do_sys_openat2+0x249/0x290 [ 2042.749735] do_sys_open+0x4f/0x80 [ 2042.753149] do_syscall_64+0x38/0x80 [ 2042.756734] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2042.761785] RIP: 0033:0x7f48f0eed2f2 [ 2042.765364] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2042.784111] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2042.791678] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2042.798809] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2042.805944] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2042.813076] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2042.820209] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b0 [ 2042.827356] [ 2042.829550] irq event stamp: 12727363 [ 2042.833217] hardirqs last enabled at (12727373): [] console_unlock+0x106/0x120 [ 2042.842084] hardirqs last disabled at (12727382): [] console_unlock+0xeb/0x120 [ 2042.850880] softirqs last enabled at (12727406): [] __do_softirq+0x2be/0x371 [ 2042.859574] softirqs last disabled at (12727393): [] irq_exit_rcu+0xb0/0xd0 [ 2042.868092] ---[ end trace 0000000000000000 ]--- [ 2042.872713] ------------[ cut here ]------------ [ 2042.877337] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2042.888992] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2042.897951] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2042.953493] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2042.962468] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2042.969949] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2042.975443] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2042.994190] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2042.999421] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2043.006561] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2043.013697] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2043.020829] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2043.027961] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2043.035102] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2043.043189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2043.048936] CR2: 00007f9bd4ad9020 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2043.056068] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2043.063204] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2043.070341] Call Trace: [ 2043.072795] [ 2043.074903] ? __warn+0x85/0x180 [ 2043.078138] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.083023] ? report_bug+0xfc/0x1d0 [ 2043.086603] ? prb_read_valid+0x1b/0x20 [ 2043.090444] ? handle_bug+0x44/0x70 [ 2043.093943] ? exc_invalid_op+0x17/0x60 [ 2043.097784] ? asm_exc_invalid_op+0x1a/0x20 [ 2043.101977] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.106857] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.111735] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2043.116637] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2043.121805] ceph_write_iter+0x5aa/0x770 [ceph] [ 2043.126367] ? lock_is_held_type+0xaa/0x120 [ 2043.130564] vfs_write+0x324/0x4f0 [ 2043.133985] ksys_pwrite64+0x65/0xa0 [ 2043.137562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2043.142705] do_syscall_64+0x38/0x80 [ 2043.146290] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2043.151342] RIP: 0033:0x7f48f0eeb93b [ 2043.154922] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2043.173669] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2043.181237] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2043.188366] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2043.195504] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2043.202646] R10: 0000000000030000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2043.209783] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2043.216928] [ 2043.219116] irq event stamp: 12826097 [ 2043.222783] hardirqs last enabled at (12826107): [] console_unlock+0x106/0x120 [ 2043.231649] hardirqs last disabled at (12826116): [] console_unlock+0xeb/0x120 [ 2043.240429] softirqs last enabled at (12826064): [] __do_softirq+0x2be/0x371 [ 2043.249122] softirqs last disabled at (12826055): [] irq_exit_rcu+0xb0/0xd0 [ 2043.257642] ---[ end trace 0000000000000000 ]--- [ 2043.262264] ------------[ cut here ]------------ [ 2043.266886] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2043.278533] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2043.287488] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2043.342966] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2043.351917] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2043.359396] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2043.364882] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2043.383629] RSP: 0018:ffffc90001fdfbe0 EFLAGS: 00010286 [ 2043.388854] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2043.395987] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2043.403120] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2043.410254] R10: 0000000000000001 R11: ffffc90001fdfa70 R12: ffff8881e4d31ae0 [ 2043.417386] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2043.424519] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2043.432605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2043.438351] CR2: 00007f418ce07000 CR3: 0000000179402005 CR4: 00000000003706f0 [ 2043.445498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2043.452625] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2043.459758] Call Trace: [ 2043.462209] [ 2043.464308] ? __warn+0x85/0x180 [ 2043.467540] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.472420] ? report_bug+0xfc/0x1d0 [ 2043.476000] ? prb_read_valid+0x1b/0x20 [ 2043.479840] ? handle_bug+0x44/0x70 [ 2043.483330] ? exc_invalid_op+0x17/0x60 [ 2043.487162] ? asm_exc_invalid_op+0x1a/0x20 [ 2043.491353] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.496229] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2043.501118] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2043.505746] ? _raw_spin_unlock+0x23/0x40 [ 2043.509762] __ceph_get_caps+0xf8/0x760 [ceph] [ 2043.514220] ? __vfs_getxattr+0x8d/0xb0 [ 2043.518063] ? dentry_needs_remove_privs+0x31/0x50 [ 2043.522853] ? __file_remove_privs+0x89/0x110 [ 2043.527210] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2043.531926] ? lock_release+0xc5/0x260 [ 2043.535682] ? ceph_write_iter+0x260/0x770 [ceph] [ 2043.540395] ceph_write_iter+0x260/0x770 [ceph] [ 2043.544939] ? lock_is_held_type+0xaa/0x120 [ 2043.549123] vfs_write+0x324/0x4f0 [ 2043.552534] ksys_pwrite64+0x65/0xa0 [ 2043.556116] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2043.561255] do_syscall_64+0x38/0x80 [ 2043.564835] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2043.569886] RIP: 0033:0x7f48f0eeb93b [ 2043.573465] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2043.592211] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2043.599778] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2043.606911] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000012 [ 2043.614043] RBP: 00000000000029b0 R08: 0000000000000001 R09: 0000000002074010 [ 2043.621178] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [ 2043.628310] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2043.635451] [ 2043.637646] irq event stamp: 12729417 [ 2043.641311] hardirqs last enabled at (12729427): [] console_unlock+0x106/0x120 [ 2043.650177] hardirqs last disabled at (12729436): [] console_unlock+0xeb/0x120 [ 2043.658956] softirqs last enabled at (12729364): [] __do_softirq+0x2be/0x371 [ 2043.667647] softirqs last disabled at (12729359): [] irq_exit_rcu+0xb0/0xd0 [ 2043.676170] ---[ end trace 0000000000000000 ]--- [ 2043.680788] ------------[ cut here ]------------ [ 2043.685407] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2043.697053] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2043.706008] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2043.761485] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2043.770437] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2043.777918] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2043.783402] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2043.802150] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2043.807375] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2043.814508] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2043.821640] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2043.828774] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d09f8750 [ 2043.835911] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2043.843040] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2043.851125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2043.856871] CR2: 00007fb38ecdd000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2043.864005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2043.871138] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2043.878275] Call Trace: [ 2043.880723] [ 2043.882830] ? __warn+0x85/0x180 [ 2043.886062] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.890941] ? report_bug+0xfc/0x1d0 [ 2043.894524] ? prb_read_valid+0x1b/0x20 [ 2043.898361] ? handle_bug+0x44/0x70 [ 2043.901853] ? exc_invalid_op+0x17/0x60 [ 2043.905693] ? asm_exc_invalid_op+0x1a/0x20 [ 2043.909882] ? __list_add_valid_or_report+0x7d/0xa0 [ 2043.914758] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2043.919648] try_get_cap_refs+0x433/0x880 [ceph] [ 2043.924276] ? _raw_spin_unlock+0x23/0x40 [ 2043.928290] __ceph_get_caps+0xf8/0x760 [ceph] [ 2043.932743] ? __vfs_getxattr+0x8d/0xb0 [ 2043.936591] ? dentry_needs_remove_privs+0x31/0x50 [ 2043.941382] ? __file_remove_privs+0x89/0x110 [ 2043.945742] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2043.950457] ? lock_release+0xc5/0x260 [ 2043.954212] ? ceph_write_iter+0x260/0x770 [ceph] [ 2043.958924] ceph_write_iter+0x260/0x770 [ceph] [ 2043.963469] ? lock_is_held_type+0xaa/0x120 [ 2043.967653] vfs_write+0x324/0x4f0 [ 2043.971062] ksys_pwrite64+0x65/0xa0 [ 2043.974636] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2043.979776] do_syscall_64+0x38/0x80 [ 2043.983356] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2043.988407] RIP: 0033:0x7f48f0eeb93b [ 2043.991987] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2044.010734] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2044.018298] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2044.025424] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2044.032557] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2044.039690] R10: 0000000000040000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2044.046823] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2044.053963] [ 2044.056148] irq event stamp: 12828011 [ 2044.059813] hardirqs last enabled at (12828021): [] console_unlock+0x106/0x120 [ 2044.068681] hardirqs last disabled at (12828030): [] console_unlock+0xeb/0x120 [ 2044.077461] softirqs last enabled at (12827792): [] __do_softirq+0x2be/0x371 [ 2044.086152] softirqs last disabled at (12827785): [] irq_exit_rcu+0xb0/0xd0 [ 2044.094671] ---[ end trace 0000000000000000 ]--- [ 2044.099284] ------------[ cut here ]------------ [ 2044.103904] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2044.115552] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2044.124507] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2044.179993] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2044.188946] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2044.196430] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2044.201919] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2044.220665] RSP: 0018:ffffc90001f7fd18 EFLAGS: 00010286 [ 2044.225899] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2044.233031] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2044.240165] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2044.247299] R10: 0000000000000001 R11: ffffc90001f7fba8 R12: ffff8881e4d0b838 [ 2044.254429] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2044.261563] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2044.269650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2044.275393] CR2: 00007fb38ebb0628 CR3: 0000000113532006 CR4: 00000000003706e0 [ 2044.282528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2044.289658] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2044.296795] Call Trace: [ 2044.299244] [ 2044.301355] ? __warn+0x85/0x180 [ 2044.304593] ? __list_add_valid_or_report+0x7d/0xa0 [ 2044.309476] ? report_bug+0xfc/0x1d0 [ 2044.313062] ? prb_read_valid+0x1b/0x20 [ 2044.316906] ? handle_bug+0x44/0x70 [ 2044.320401] ? exc_invalid_op+0x17/0x60 [ 2044.324243] ? asm_exc_invalid_op+0x1a/0x20 [ 2044.328434] ? __list_add_valid_or_report+0x7d/0xa0 [ 2044.333319] ? __list_add_valid_or_report+0x7d/0xa0 [ 2044.338202] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2044.343106] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2044.348262] ceph_write_iter+0x5aa/0x770 [ceph] [ 2044.352823] ? lock_is_held_type+0xaa/0x120 [ 2044.357014] vfs_write+0x324/0x4f0 [ 2044.360436] ksys_pwrite64+0x65/0xa0 [ 2044.364012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2044.369157] do_syscall_64+0x38/0x80 [ 2044.372741] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2044.377796] RIP: 0033:0x7f48f0eeb93b [ 2044.381381] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2044.400127] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2044.407692] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2044.414827] RDX: 0000000000001000 RSI: 000000000207eb20 RDI: 0000000000000005 [ 2044.421959] RBP: 0000000000002a15 R08: 0000000000001000 R09: 0000000000000003 [ 2044.429093] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d000 [ 2044.436226] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207eb20 [ 2044.443369] [ 2044.445559] irq event stamp: 12868103 [ 2044.449226] hardirqs last enabled at (12868113): [] console_unlock+0x106/0x120 [ 2044.458092] hardirqs last disabled at (12868122): [] console_unlock+0xeb/0x120 [ 2044.466872] softirqs last enabled at (12868090): [] __do_softirq+0x2be/0x371 [ 2044.475572] softirqs last disabled at (12868085): [] irq_exit_rcu+0xb0/0xd0 [ 2044.484094] ---[ end trace 0000000000000000 ]--- [ 2044.488711] ------------[ cut here ]------------ [ 2044.493471] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2044.505178] WARNING: CPU: 7 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2044.514185] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2044.569653] CPU: 7 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2044.578603] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2044.586081] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2044.591568] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2044.610314] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2044.615538] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2044.622663] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2044.629796] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2044.636928] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2044.644062] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2044.651195] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2044.659280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2044.665027] CR2: 00007fb38ece6040 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2044.672160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2044.679292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2044.686425] Call Trace: [ 2044.688870] [ 2044.690976] ? __warn+0x85/0x180 [ 2044.694208] ? __list_add_valid_or_report+0x7d/0xa0 [ 2044.699089] ? report_bug+0xfc/0x1d0 [ 2044.702666] ? prb_read_valid+0x1b/0x20 [ 2044.706509] ? handle_bug+0x44/0x70 [ 2044.709998] ? exc_invalid_op+0x17/0x60 [ 2044.713838] ? asm_exc_invalid_op+0x1a/0x20 [ 2044.718028] ? __list_add_valid_or_report+0x7d/0xa0 [ 2044.722905] ? __list_add_valid_or_report+0x7d/0xa0 [ 2044.727784] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2044.732682] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2044.737841] ceph_write_iter+0x5aa/0x770 [ceph] [ 2044.742383] ? lock_is_held_type+0xaa/0x120 [ 2044.746567] vfs_write+0x324/0x4f0 [ 2044.749974] ksys_pwrite64+0x65/0xa0 [ 2044.753550] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2044.758690] do_syscall_64+0x38/0x80 [ 2044.762270] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2044.767323] RIP: 0033:0x7f48f0eeb93b [ 2044.770904] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2044.789648] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2044.797212] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2044.804338] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2044.811469] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2044.818601] R10: 0000000000040000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2044.825727] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2044.832865] [ 2044.835054] irq event stamp: 12830415 [ 2044.838718] hardirqs last enabled at (12830425): [] console_unlock+0x106/0x120 [ 2044.847586] hardirqs last disabled at (12830434): [] console_unlock+0xeb/0x120 [ 2044.856365] softirqs last enabled at (12830412): [] __do_softirq+0x2be/0x371 [ 2044.865058] softirqs last disabled at (12830407): [] irq_exit_rcu+0xb0/0xd0 [ 2044.873575] ---[ end trace 0000000000000000 ]--- [ 2044.878192] ------------[ cut here ]------------ [ 2044.882904] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2044.894665] WARNING: CPU: 4 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2044.903706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2044.959224] CPU: 4 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2044.968228] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2044.975709] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2044.981196] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2044.999951] RSP: 0018:ffffc90001fdfd18 EFLAGS: 00010286 [ 2045.005179] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2045.012317] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2045.019452] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2045.026583] R10: 0000000000000001 R11: ffffc90001fdfba8 R12: ffff8881e4d31ae0 [ 2045.033720] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2045.040866] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2045.048954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2045.054698] CR2: 00007f9bd4ad9020 CR3: 0000000179402005 CR4: 00000000003706e0 [ 2045.061835] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2045.068980] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2045.076122] Call Trace: [ 2045.078576] [ 2045.080684] ? __warn+0x85/0x180 [ 2045.083923] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.088804] ? report_bug+0xfc/0x1d0 [ 2045.092382] ? prb_read_valid+0x1b/0x20 [ 2045.096227] ? handle_bug+0x44/0x70 [ 2045.099722] ? exc_invalid_op+0x17/0x60 [ 2045.103564] ? asm_exc_invalid_op+0x1a/0x20 [ 2045.107755] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.112639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.117523] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2045.122425] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2045.127584] ceph_write_iter+0x5aa/0x770 [ceph] [ 2045.132145] ? lock_is_held_type+0xaa/0x120 [ 2045.136335] vfs_write+0x324/0x4f0 [ 2045.139754] ksys_pwrite64+0x65/0xa0 [ 2045.143333] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2045.148479] do_syscall_64+0x38/0x80 [ 2045.152061] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2045.157114] RIP: 0033:0x7f48f0eeb93b [ 2045.160697] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2045.179451] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2045.187022] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2045.194168] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000012 [ 2045.201297] RBP: 00000000000029b0 R08: 0000000000000001 R09: 0000000002074010 [ 2045.208432] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [ 2045.215565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2045.222709] [ 2045.224898] irq event stamp: 12734149 [ 2045.228564] hardirqs last enabled at (12734159): [] console_unlock+0x106/0x120 [ 2045.237427] hardirqs last disabled at (12734168): [] console_unlock+0xeb/0x120 [ 2045.246209] softirqs last enabled at (12734146): [] __do_softirq+0x2be/0x371 [ 2045.254901] softirqs last disabled at (12734141): [] irq_exit_rcu+0xb0/0xd0 [ 2045.263423] ---[ end trace 0000000000000000 ]--- [ 2045.268050] ------------[ cut here ]------------ [ 2045.272666] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2045.284314] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2045.293267] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2045.348736] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2045.357689] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2045.365168] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2045.370654] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2045.389401] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2045.394626] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2045.401758] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2045.408882] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2045.416006] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d09f8750 [ 2045.423130] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2045.430256] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2045.438341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2045.444080] CR2: 00007fb38ecf0070 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2045.451212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2045.458344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2045.465470] Call Trace: [ 2045.467921] [ 2045.470027] ? __warn+0x85/0x180 [ 2045.473261] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.478143] ? report_bug+0xfc/0x1d0 [ 2045.481718] ? prb_read_valid+0x1b/0x20 [ 2045.485560] ? handle_bug+0x44/0x70 [ 2045.489050] ? exc_invalid_op+0x17/0x60 [ 2045.492882] ? asm_exc_invalid_op+0x1a/0x20 [ 2045.497074] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.501948] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2045.506839] try_get_cap_refs+0x433/0x880 [ceph] [ 2045.511464] ? _raw_spin_unlock+0x23/0x40 [ 2045.515480] __ceph_get_caps+0xf8/0x760 [ceph] [ 2045.519934] ? __vfs_getxattr+0x8d/0xb0 [ 2045.523773] ? dentry_needs_remove_privs+0x31/0x50 [ 2045.528563] ? __file_remove_privs+0x89/0x110 [ 2045.532922] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2045.537638] ? lock_release+0xc5/0x260 [ 2045.541394] ? ceph_write_iter+0x260/0x770 [ceph] [ 2045.546105] ceph_write_iter+0x260/0x770 [ceph] [ 2045.550649] ? lock_is_held_type+0xaa/0x120 [ 2045.554836] vfs_write+0x324/0x4f0 [ 2045.558243] ksys_pwrite64+0x65/0xa0 [ 2045.561817] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2045.566959] do_syscall_64+0x38/0x80 [ 2045.570538] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2045.575589] RIP: 0033:0x7f48f0eeb93b [ 2045.579170] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2045.597916] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2045.605480] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2045.612607] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2045.619739] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2045.626871] R10: 0000000000050000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2045.634003] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2045.641145] [ 2045.643329] irq event stamp: 12832279 [ 2045.646998] hardirqs last enabled at (12832295): [] console_unlock+0x106/0x120 [ 2045.655862] hardirqs last disabled at (12832304): [] console_unlock+0xeb/0x120 [ 2045.664640] softirqs last enabled at (12832292): [] __do_softirq+0x2be/0x371 [ 2045.673333] softirqs last disabled at (12832287): [] irq_exit_rcu+0xb0/0xd0 [ 2045.681854] ---[ end trace 0000000000000000 ]--- [ 2045.686473] ------------[ cut here ]------------ [ 2045.691093] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2045.702742] WARNING: CPU: 6 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2045.711698] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2045.767177] CPU: 6 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2045.776134] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2045.783640] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2045.789135] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2045.807880] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2045.813110] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2045.820245] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2045.827380] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2045.834516] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2045.841644] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2045.848779] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2045.856863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2045.862612] CR2: 00007fb38ebb0628 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2045.869743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2045.876876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2045.884012] Call Trace: [ 2045.886463] [ 2045.888567] ? __warn+0x85/0x180 [ 2045.891804] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.896692] ? report_bug+0xfc/0x1d0 [ 2045.900274] ? prb_read_valid+0x1b/0x20 [ 2045.904121] ? handle_bug+0x44/0x70 [ 2045.907619] ? exc_invalid_op+0x17/0x60 [ 2045.911458] ? asm_exc_invalid_op+0x1a/0x20 [ 2045.915653] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.920537] ? __list_add_valid_or_report+0x7d/0xa0 [ 2045.925421] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2045.930318] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2045.935481] ceph_write_iter+0x5aa/0x770 [ceph] [ 2045.940029] ? lock_is_held_type+0xaa/0x120 [ 2045.944225] vfs_write+0x324/0x4f0 [ 2045.947642] ksys_pwrite64+0x65/0xa0 [ 2045.951222] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2045.956362] do_syscall_64+0x38/0x80 [ 2045.959941] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2045.964995] RIP: 0033:0x7f48f0eeb93b [ 2045.968573] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2045.987316] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2045.994886] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2046.002016] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2046.009151] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2046.016281] R10: 0000000000040000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2046.023415] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2046.030563] [ 2046.032759] irq event stamp: 12854133 [ 2046.036424] hardirqs last enabled at (12854143): [] console_unlock+0x106/0x120 [ 2046.045290] hardirqs last disabled at (12854152): [] console_unlock+0xeb/0x120 [ 2046.054068] softirqs last enabled at (12854120): [] __do_softirq+0x2be/0x371 [ 2046.062763] softirqs last disabled at (12854115): [] irq_exit_rcu+0xb0/0xd0 [ 2046.071280] ---[ end trace 0000000000000000 ]--- [ 2046.075904] ------------[ cut here ]------------ [ 2046.080523] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2046.092166] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2046.100861] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2046.156337] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2046.165463] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2046.172944] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2046.178448] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2046.183932] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2046.202677] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2046.207905] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2046.215038] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2046.222172] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2046.229301] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881c2641118 [ 2046.236425] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2046.243551] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2046.251636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2046.257384] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2046.264516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2046.271648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2046.278780] Call Trace: [ 2046.281226] [ 2046.283333] ? __warn+0x85/0x180 [ 2046.286566] ? __list_add_valid_or_report+0x7d/0xa0 [ 2046.291444] ? report_bug+0xfc/0x1d0 [ 2046.295026] ? prb_read_valid+0x1b/0x20 [ 2046.298864] ? handle_bug+0x44/0x70 [ 2046.302356] ? exc_invalid_op+0x17/0x60 [ 2046.306196] ? asm_exc_invalid_op+0x1a/0x20 [ 2046.310387] ? __list_add_valid_or_report+0x7d/0xa0 [ 2046.315270] ? __list_add_valid_or_report+0x7d/0xa0 [ 2046.320149] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2046.325038] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2046.329676] ceph_fill_trace+0x143/0xf30 [ceph] [ 2046.334220] ? __mutex_lock+0x97/0xce0 [ 2046.337978] ? __mutex_lock+0xaec/0xce0 [ 2046.341819] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2046.346367] ? lock_downgrade+0xfd/0x250 [ 2046.350298] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2046.354678] ceph_con_process_message+0x74/0x150 [libceph] [ 2046.360172] ? calc_signature+0xdb/0x100 [libceph] [ 2046.364975] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2046.370300] ? process_scheduled_works+0x22a/0x510 [ 2046.375099] ? process_scheduled_works+0x22a/0x510 [ 2046.379888] ceph_con_workfn+0x344/0x750 [libceph] [ 2046.384693] ? process_scheduled_works+0x22a/0x510 [ 2046.389483] process_scheduled_works+0x2a4/0x510 [ 2046.394108] worker_thread+0x188/0x340 [ 2046.397856] ? rescuer_thread+0x310/0x310 [ 2046.401868] kthread+0xf5/0x130 [ 2046.405015] ? kthread_complete_and_exit+0x20/0x20 [ 2046.409808] ret_from_fork+0x31/0x40 [ 2046.413388] ? kthread_complete_and_exit+0x20/0x20 [ 2046.418179] ret_from_fork_asm+0x11/0x20 [ 2046.422112] [ 2046.424297] irq event stamp: 8928797 [ 2046.427877] hardirqs last enabled at (8928807): [] console_unlock+0x106/0x120 [ 2046.436656] hardirqs last disabled at (8928816): [] console_unlock+0xeb/0x120 [ 2046.445349] softirqs last enabled at (8928764): [] __do_softirq+0x2be/0x371 [ 2046.453954] softirqs last disabled at (8928759): [] irq_exit_rcu+0xb0/0xd0 [ 2046.462388] ---[ end trace 0000000000000000 ]--- [ 2046.467008] ------------[ cut here ]------------ [ 2046.471654] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2046.483323] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2046.492298] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2046.547848] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2046.556815] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2046.564315] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2046.569825] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2046.588590] RSP: 0018:ffffc90001fdfbe0 EFLAGS: 00010286 [ 2046.593838] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2046.600988] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2046.608142] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2046.615289] R10: 0000000000000001 R11: ffffc90001fdfa70 R12: ffff8881e4d31ae0 [ 2046.622443] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2046.629593] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2046.637706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2046.643468] CR2: 00007f9bd4afa000 CR3: 0000000179402005 CR4: 00000000003706f0 [ 2046.650616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2046.657769] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2046.664915] Call Trace: [ 2046.667389] [ 2046.669509] ? __warn+0x85/0x180 [ 2046.672760] ? __list_add_valid_or_report+0x7d/0xa0 [ 2046.677657] ? report_bug+0xfc/0x1d0 [ 2046.681251] ? prb_read_valid+0x1b/0x20 [ 2046.685113] ? handle_bug+0x44/0x70 [ 2046.688622] ? exc_invalid_op+0x17/0x60 [ 2046.692475] ? asm_exc_invalid_op+0x1a/0x20 [ 2046.696692] ? __list_add_valid_or_report+0x7d/0xa0 [ 2046.701597] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2046.706510] try_get_cap_refs+0x433/0x880 [ceph] [ 2046.711156] ? _raw_spin_unlock+0x23/0x40 [ 2046.715178] __ceph_get_caps+0xf8/0x760 [ceph] [ 2046.719644] ? __vfs_getxattr+0x8d/0xb0 [ 2046.723495] ? dentry_needs_remove_privs+0x31/0x50 [ 2046.728291] ? __file_remove_privs+0x89/0x110 [ 2046.732649] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2046.737375] ? lock_release+0xc5/0x260 [ 2046.741135] ? ceph_write_iter+0x260/0x770 [ceph] [ 2046.745856] ceph_write_iter+0x260/0x770 [ceph] [ 2046.750420] ? lock_is_held_type+0xaa/0x120 [ 2046.754618] vfs_write+0x324/0x4f0 [ 2046.758036] ksys_pwrite64+0x65/0xa0 [ 2046.761622] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2046.766765] do_syscall_64+0x38/0x80 [ 2046.770350] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2046.775402] RIP: 0033:0x7f48f0eeb93b [ 2046.778982] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2046.797732] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2046.805303] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2046.812435] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2046.819569] RBP: 00000000000029b0 R08: 0000000000000800 R09: 0000000000000003 [ 2046.826701] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dac8 [ 2046.833837] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207df80 [ 2046.840989] [ 2046.843187] irq event stamp: 12739967 [ 2046.846853] hardirqs last enabled at (12739977): [] console_unlock+0x106/0x120 [ 2046.855718] hardirqs last disabled at (12739986): [] console_unlock+0xeb/0x120 [ 2046.864500] softirqs last enabled at (12740002): [] __do_softirq+0x2be/0x371 [ 2046.873198] softirqs last disabled at (12739995): [] irq_exit_rcu+0xb0/0xd0 [ 2046.881719] ---[ end trace 0000000000000000 ]--- [ 2046.886340] ------------[ cut here ]------------ [ 2046.890964] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2046.902615] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2046.911317] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2046.966863] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2046.975988] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2046.983467] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2046.988980] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2046.994464] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2047.013211] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2047.018437] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2047.025569] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2047.032704] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2047.039836] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881e4da1118 [ 2047.046967] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2047.054101] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2047.062188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2047.067932] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2047.075067] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2047.082199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2047.089332] Call Trace: [ 2047.091785] [ 2047.093892] ? __warn+0x85/0x180 [ 2047.097127] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.102014] ? report_bug+0xfc/0x1d0 [ 2047.105603] ? prb_read_valid+0x1b/0x20 [ 2047.109453] ? handle_bug+0x44/0x70 [ 2047.112951] ? exc_invalid_op+0x17/0x60 [ 2047.116790] ? asm_exc_invalid_op+0x1a/0x20 [ 2047.120995] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.125884] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.130771] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2047.135669] ceph_add_cap+0x20b/0x620 [ceph] [ 2047.139976] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2047.144721] ceph_fill_trace+0x143/0xf30 [ceph] [ 2047.149275] ? __mutex_lock+0x97/0xce0 [ 2047.153034] ? __mutex_lock+0xaec/0xce0 [ 2047.156875] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2047.161428] ? lock_downgrade+0xfd/0x250 [ 2047.165368] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2047.169760] ceph_con_process_message+0x74/0x150 [libceph] [ 2047.175265] ? calc_signature+0xdb/0x100 [libceph] [ 2047.180088] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2047.185429] ? process_scheduled_works+0x22a/0x510 [ 2047.190228] ? process_scheduled_works+0x22a/0x510 [ 2047.195027] ceph_con_workfn+0x344/0x750 [libceph] [ 2047.199840] ? process_scheduled_works+0x22a/0x510 [ 2047.204636] process_scheduled_works+0x2a4/0x510 [ 2047.209267] worker_thread+0x188/0x340 [ 2047.213027] ? rescuer_thread+0x310/0x310 [ 2047.217048] kthread+0xf5/0x130 [ 2047.220193] ? kthread_complete_and_exit+0x20/0x20 [ 2047.224988] ret_from_fork+0x31/0x40 [ 2047.228577] ? kthread_complete_and_exit+0x20/0x20 [ 2047.233375] ret_from_fork_asm+0x11/0x20 [ 2047.237317] [ 2047.239510] irq event stamp: 8930905 [ 2047.243089] hardirqs last enabled at (8930915): [] console_unlock+0x106/0x120 [ 2047.251868] hardirqs last disabled at (8930924): [] console_unlock+0xeb/0x120 [ 2047.260566] softirqs last enabled at (8930940): [] __do_softirq+0x2be/0x371 [ 2047.269178] softirqs last disabled at (8930933): [] irq_exit_rcu+0xb0/0xd0 [ 2047.277611] ---[ end trace 0000000000000000 ]--- [ 2047.282231] ------------[ cut here ]------------ [ 2047.286848] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2047.298496] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2047.307452] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2047.362989] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2047.371942] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2047.379421] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2047.384914] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2047.403708] RSP: 0018:ffffc90001fefaa8 EFLAGS: 00010282 [ 2047.408987] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2047.416123] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2047.423256] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2047.430390] R10: 0000000000000001 R11: ffffc90001fef938 R12: ffff8881c2641118 [ 2047.437523] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2047.444655] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2047.452741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2047.458487] CR2: 00007fb38ed03038 CR3: 0000000108c72001 CR4: 00000000003706e0 [ 2047.465619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2047.472754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2047.479885] Call Trace: [ 2047.482343] [ 2047.484454] ? __warn+0x85/0x180 [ 2047.487687] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.492566] ? report_bug+0xfc/0x1d0 [ 2047.496151] ? prb_read_valid+0x1b/0x20 [ 2047.499997] ? handle_bug+0x44/0x70 [ 2047.503496] ? exc_invalid_op+0x17/0x60 [ 2047.507335] ? asm_exc_invalid_op+0x1a/0x20 [ 2047.511532] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.516421] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.521304] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2047.526206] ceph_open+0x16b/0x440 [ceph] [ 2047.530236] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2047.535390] do_dentry_open+0x172/0x530 [ 2047.539242] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2047.543891] lookup_open+0x37b/0x630 [ 2047.547487] ? verify_cpu+0x20/0x100 [ 2047.551072] ? verify_cpu+0x20/0x100 [ 2047.554661] path_openat+0x25a/0x9d0 [ 2047.558252] do_filp_open+0xb8/0x120 [ 2047.561840] ? lock_release+0xc5/0x260 [ 2047.565597] ? _raw_spin_unlock+0x23/0x40 [ 2047.569613] ? alloc_fd+0x13f/0x200 [ 2047.573112] do_sys_openat2+0x249/0x290 [ 2047.576959] do_sys_open+0x4f/0x80 [ 2047.580372] do_syscall_64+0x38/0x80 [ 2047.583957] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2047.589009] RIP: 0033:0x7f48f0eed2f2 [ 2047.592590] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2047.611337] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2047.618910] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2047.626043] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2047.633177] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2047.640309] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2047.647442] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029af [ 2047.654590] [ 2047.656786] irq event stamp: 12726243 [ 2047.660451] hardirqs last enabled at (12726253): [] console_unlock+0x106/0x120 [ 2047.669326] hardirqs last disabled at (12726262): [] console_unlock+0xeb/0x120 [ 2047.678106] softirqs last enabled at (12725874): [] __do_softirq+0x2be/0x371 [ 2047.686797] softirqs last disabled at (12725869): [] irq_exit_rcu+0xb0/0xd0 [ 2047.695317] ---[ end trace 0000000000000000 ]--- [ 2047.699937] ------------[ cut here ]------------ [ 2047.704554] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2047.716206] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2047.724906] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2047.780455] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2047.789584] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2047.797064] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2047.802568] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2047.808053] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2047.826803] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2047.832034] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2047.839167] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2047.846301] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2047.853434] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881e4da1118 [ 2047.860570] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2047.867699] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2047.875788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2047.881532] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2047.888665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2047.895799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2047.902931] Call Trace: [ 2047.905384] [ 2047.907489] ? __warn+0x85/0x180 [ 2047.910723] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.915613] ? report_bug+0xfc/0x1d0 [ 2047.919197] ? prb_read_valid+0x1b/0x20 [ 2047.923041] ? handle_bug+0x44/0x70 [ 2047.926543] ? exc_invalid_op+0x17/0x60 [ 2047.930387] ? asm_exc_invalid_op+0x1a/0x20 [ 2047.934585] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.939473] ? __list_add_valid_or_report+0x7d/0xa0 [ 2047.944359] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2047.949261] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2047.953912] ceph_fill_trace+0x143/0xf30 [ceph] [ 2047.958463] ? __mutex_lock+0x97/0xce0 [ 2047.962220] ? __mutex_lock+0xaec/0xce0 [ 2047.966061] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2047.970618] ? lock_downgrade+0xfd/0x250 [ 2047.974554] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2047.978947] ceph_con_process_message+0x74/0x150 [libceph] [ 2047.984451] ? calc_signature+0xdb/0x100 [libceph] [ 2047.989278] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2047.994617] ? process_scheduled_works+0x22a/0x510 [ 2047.999417] ? process_scheduled_works+0x22a/0x510 [ 2048.004210] ceph_con_workfn+0x344/0x750 [libceph] [ 2048.009026] ? process_scheduled_works+0x22a/0x510 [ 2048.013822] process_scheduled_works+0x2a4/0x510 [ 2048.018454] worker_thread+0x188/0x340 [ 2048.022218] ? rescuer_thread+0x310/0x310 [ 2048.026233] kthread+0xf5/0x130 [ 2048.029380] ? kthread_complete_and_exit+0x20/0x20 [ 2048.034176] ret_from_fork+0x31/0x40 [ 2048.037759] ? kthread_complete_and_exit+0x20/0x20 [ 2048.042555] ret_from_fork_asm+0x11/0x20 [ 2048.046504] [ 2048.048696] irq event stamp: 8932943 [ 2048.052278] hardirqs last enabled at (8932953): [] console_unlock+0x106/0x120 [ 2048.061059] hardirqs last disabled at (8932970): [] console_unlock+0xeb/0x120 [ 2048.069757] softirqs last enabled at (8932968): [] __do_softirq+0x2be/0x371 [ 2048.078364] softirqs last disabled at (8932961): [] irq_exit_rcu+0xb0/0xd0 [ 2048.086797] ---[ end trace 0000000000000000 ]--- [ 2048.091420] ------------[ cut here ]------------ [ 2048.096177] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2048.107838] WARNING: CPU: 7 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2048.116798] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2048.172327] CPU: 7 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2048.181298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2048.188780] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2048.194267] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2048.213013] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2048.218240] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2048.225371] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2048.232505] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2048.239636] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2048.246771] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2048.253903] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2048.261987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.267733] CR2: 00007fb38ece6040 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2048.274867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2048.282000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2048.289132] Call Trace: [ 2048.291587] [ 2048.293693] ? __warn+0x85/0x180 [ 2048.296927] ? __list_add_valid_or_report+0x7d/0xa0 [ 2048.301817] ? report_bug+0xfc/0x1d0 [ 2048.305403] ? prb_read_valid+0x1b/0x20 [ 2048.309249] ? handle_bug+0x44/0x70 [ 2048.312743] ? exc_invalid_op+0x17/0x60 [ 2048.316583] ? asm_exc_invalid_op+0x1a/0x20 [ 2048.320794] ? __list_add_valid_or_report+0x7d/0xa0 [ 2048.325677] ? __list_add_valid_or_report+0x7d/0xa0 [ 2048.330561] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2048.335465] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2048.340634] ceph_write_iter+0x5aa/0x770 [ceph] [ 2048.345195] ? lock_is_held_type+0xaa/0x120 [ 2048.349395] vfs_write+0x324/0x4f0 [ 2048.352819] ksys_pwrite64+0x65/0xa0 [ 2048.356406] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2048.361551] do_syscall_64+0x38/0x80 [ 2048.365135] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2048.370194] RIP: 0033:0x7f48f0eeb93b [ 2048.373776] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2048.392529] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2048.400094] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2048.407227] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2048.414360] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2048.421492] R10: 0000000000050000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2048.428625] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2048.435782] [ 2048.437980] irq event stamp: 12839711 [ 2048.441644] hardirqs last enabled at (12839727): [] console_unlock+0x106/0x120 [ 2048.450509] hardirqs last disabled at (12839742): [] console_unlock+0xeb/0x120 [ 2048.459288] softirqs last enabled at (12839756): [] __do_softirq+0x2be/0x371 [ 2048.467980] softirqs last disabled at (12839767): [] irq_exit_rcu+0xb0/0xd0 [ 2048.476501] ---[ end trace 0000000000000000 ]--- [ 2048.481122] ------------[ cut here ]------------ [ 2048.485737] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2048.497384] WARNING: CPU: 3 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2048.506338] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2048.561803] CPU: 3 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2048.570761] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2048.578239] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2048.583725] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2048.602477] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2048.607706] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2048.614841] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2048.621972] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2048.629106] R10: 0000000000000001 R11: ffffc90001fefb18 R12: ffff8881c2641118 [ 2048.636241] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2048.643372] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2048.651456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2048.657203] CR2: 00007fb38ed13088 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2048.664335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2048.671472] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2048.678610] Call Trace: [ 2048.681063] [ 2048.683172] ? __warn+0x85/0x180 [ 2048.686403] ? __list_add_valid_or_report+0x7d/0xa0 [ 2048.691282] ? report_bug+0xfc/0x1d0 [ 2048.694863] ? prb_read_valid+0x1b/0x20 [ 2048.698710] ? handle_bug+0x44/0x70 [ 2048.702206] ? exc_invalid_op+0x17/0x60 [ 2048.706050] ? asm_exc_invalid_op+0x1a/0x20 [ 2048.710244] ? __list_add_valid_or_report+0x7d/0xa0 [ 2048.715124] ? __list_add_valid_or_report+0x7d/0xa0 [ 2048.720003] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2048.724903] ceph_open+0x16b/0x440 [ceph] [ 2048.728935] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2048.734089] do_dentry_open+0x172/0x530 [ 2048.737934] path_openat+0x743/0x9d0 [ 2048.741520] do_filp_open+0xb8/0x120 [ 2048.745113] ? lock_release+0xc5/0x260 [ 2048.748867] ? _raw_spin_unlock+0x23/0x40 [ 2048.752888] ? alloc_fd+0x13f/0x200 [ 2048.756391] do_sys_openat2+0x249/0x290 [ 2048.760240] do_sys_open+0x4f/0x80 [ 2048.763650] do_syscall_64+0x38/0x80 [ 2048.767233] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2048.772290] RIP: 0033:0x7f48f0eed2f2 [ 2048.775870] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2048.794616] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2048.802182] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2048.809312] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2048.816447] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2048.823578] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2048.830717] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b0 [ 2048.837864] [ 2048.840056] irq event stamp: 12729391 [ 2048.843722] hardirqs last enabled at (12729401): [] console_unlock+0x106/0x120 [ 2048.852589] hardirqs last disabled at (12729410): [] console_unlock+0xeb/0x120 [ 2048.861366] softirqs last enabled at (12729368): [] __do_softirq+0x2be/0x371 [ 2048.870065] softirqs last disabled at (12729431): [] irq_exit_rcu+0xb0/0xd0 [ 2048.878587] ---[ end trace 0000000000000000 ]--- [ 2048.883208] ------------[ cut here ]------------ [ 2048.887844] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2048.899496] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2048.908457] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2048.963994] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2048.972965] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2048.980444] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2048.985937] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2049.004687] RSP: 0018:ffffc90001fcfaa8 EFLAGS: 00010282 [ 2049.009918] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2049.017052] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2049.024189] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2049.031325] R10: 0000000000000001 R11: ffffc90001fcf938 R12: ffff8881e4da1118 [ 2049.038461] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2049.045607] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2049.053694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2049.059440] CR2: 00007fb38ed17000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2049.066574] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2049.073711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2049.080848] Call Trace: [ 2049.083301] [ 2049.085408] ? __warn+0x85/0x180 [ 2049.088639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.093520] ? report_bug+0xfc/0x1d0 [ 2049.097097] ? prb_read_valid+0x1b/0x20 [ 2049.100943] ? handle_bug+0x44/0x70 [ 2049.104442] ? exc_invalid_op+0x17/0x60 [ 2049.108289] ? asm_exc_invalid_op+0x1a/0x20 [ 2049.112493] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.117382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.122265] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2049.127168] ceph_open+0x16b/0x440 [ceph] [ 2049.131203] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2049.136362] do_dentry_open+0x172/0x530 [ 2049.140209] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2049.144861] lookup_open+0x37b/0x630 [ 2049.148448] ? verify_cpu+0x20/0x100 [ 2049.152035] ? verify_cpu+0x20/0x100 [ 2049.155622] path_openat+0x25a/0x9d0 [ 2049.159213] do_filp_open+0xb8/0x120 [ 2049.162801] ? lock_release+0xc5/0x260 [ 2049.166559] ? _raw_spin_unlock+0x23/0x40 [ 2049.170577] ? alloc_fd+0x13f/0x200 [ 2049.174082] do_sys_openat2+0x249/0x290 [ 2049.177930] do_sys_open+0x4f/0x80 [ 2049.181347] do_syscall_64+0x38/0x80 [ 2049.184928] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2049.189979] RIP: 0033:0x7f48f0eed2f2 [ 2049.193560] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2049.212309] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2049.219881] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2049.227012] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2049.234145] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2049.241278] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2049.248414] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029af [ 2049.255560] [ 2049.257755] irq event stamp: 12722079 [ 2049.261421] hardirqs last enabled at (12722089): [] console_unlock+0x106/0x120 [ 2049.270288] hardirqs last disabled at (12722098): [] console_unlock+0xeb/0x120 [ 2049.279075] softirqs last enabled at (12722118): [] __do_softirq+0x2be/0x371 [ 2049.287775] softirqs last disabled at (12722129): [] irq_exit_rcu+0xb0/0xd0 [ 2049.296295] ---[ end trace 0000000000000000 ]--- [ 2049.300918] ------------[ cut here ]------------ [ 2049.305544] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2049.317190] WARNING: CPU: 7 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2049.326142] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2049.381699] CPU: 7 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2049.390668] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2049.398149] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2049.403632] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2049.422382] RSP: 0018:ffffc90001fefbe0 EFLAGS: 00010286 [ 2049.427606] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2049.434737] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2049.441872] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2049.449005] R10: 0000000000000001 R11: ffffc90001fefa70 R12: ffff8881c2641118 [ 2049.456138] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2049.463271] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2049.471359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2049.477102] CR2: 00007f9bd4b26000 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2049.484235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2049.491372] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2049.498502] Call Trace: [ 2049.500954] [ 2049.503064] ? __warn+0x85/0x180 [ 2049.506304] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.511192] ? report_bug+0xfc/0x1d0 [ 2049.514770] ? prb_read_valid+0x1b/0x20 [ 2049.518617] ? handle_bug+0x44/0x70 [ 2049.522111] ? exc_invalid_op+0x17/0x60 [ 2049.525950] ? asm_exc_invalid_op+0x1a/0x20 [ 2049.530148] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.535035] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2049.539943] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2049.544586] ? _raw_spin_unlock+0x23/0x40 [ 2049.548613] __ceph_get_caps+0xf8/0x760 [ceph] [ 2049.553078] ? __vfs_getxattr+0x8d/0xb0 [ 2049.556925] ? dentry_needs_remove_privs+0x31/0x50 [ 2049.561720] ? __file_remove_privs+0x89/0x110 [ 2049.566085] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2049.570813] ? lock_release+0xc5/0x260 [ 2049.574577] ? ceph_write_iter+0x260/0x770 [ceph] [ 2049.579304] ceph_write_iter+0x260/0x770 [ceph] [ 2049.583862] ? lock_is_held_type+0xaa/0x120 [ 2049.588059] vfs_write+0x324/0x4f0 [ 2049.591482] ksys_pwrite64+0x65/0xa0 [ 2049.595068] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2049.600211] do_syscall_64+0x38/0x80 [ 2049.603797] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2049.608849] RIP: 0033:0x7f48f0eeb93b [ 2049.612427] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2049.631175] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2049.638745] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2049.645880] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000012 [ 2049.653013] RBP: 00000000000029b0 R08: 0000000000000001 R09: 0000000002074010 [ 2049.660156] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183df18 [ 2049.667290] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2049.674437] [ 2049.676632] irq event stamp: 12731457 [ 2049.680297] hardirqs last enabled at (12731467): [] console_unlock+0x106/0x120 [ 2049.689164] hardirqs last disabled at (12731476): [] console_unlock+0xeb/0x120 [ 2049.697945] softirqs last enabled at (12731490): [] __do_softirq+0x2be/0x371 [ 2049.706644] softirqs last disabled at (12731485): [] irq_exit_rcu+0xb0/0xd0 [ 2049.715164] ---[ end trace 0000000000000000 ]--- [ 2049.719785] ------------[ cut here ]------------ [ 2049.724410] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2049.736058] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2049.745012] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2049.800535] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2049.809492] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2049.816973] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2049.822458] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2049.841204] RSP: 0018:ffffc90001fdfd18 EFLAGS: 00010286 [ 2049.846431] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2049.853563] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2049.860696] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2049.867832] R10: 0000000000000001 R11: ffffc90001fdfba8 R12: ffff8881e4d31ae0 [ 2049.874962] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2049.882095] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2049.890182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2049.895934] CR2: 00007f9bd4b0d008 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2049.903069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2049.910201] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2049.917334] Call Trace: [ 2049.919787] [ 2049.921895] ? __warn+0x85/0x180 [ 2049.925128] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.930006] ? report_bug+0xfc/0x1d0 [ 2049.933588] ? prb_read_valid+0x1b/0x20 [ 2049.937437] ? handle_bug+0x44/0x70 [ 2049.940935] ? exc_invalid_op+0x17/0x60 [ 2049.944773] ? asm_exc_invalid_op+0x1a/0x20 [ 2049.948970] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.953850] ? __list_add_valid_or_report+0x7d/0xa0 [ 2049.958728] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2049.963627] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2049.968787] ceph_write_iter+0x5aa/0x770 [ceph] [ 2049.973347] ? lock_is_held_type+0xaa/0x120 [ 2049.977541] vfs_write+0x324/0x4f0 [ 2049.980959] ksys_pwrite64+0x65/0xa0 [ 2049.984537] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2049.989680] do_syscall_64+0x38/0x80 [ 2049.993265] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2049.998319] RIP: 0033:0x7f48f0eeb93b [ 2050.001898] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2050.020643] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2050.028208] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2050.035343] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2050.042475] RBP: 00000000000029b0 R08: 0000000000000800 R09: 0000000000000003 [ 2050.049607] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dac8 [ 2050.056739] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207df80 [ 2050.063889] [ 2050.066083] irq event stamp: 12747167 [ 2050.069749] hardirqs last enabled at (12747177): [] console_unlock+0x106/0x120 [ 2050.078618] hardirqs last disabled at (12747192): [] console_unlock+0xeb/0x120 [ 2050.087394] softirqs last enabled at (12747190): [] __do_softirq+0x2be/0x371 [ 2050.096088] softirqs last disabled at (12747185): [] irq_exit_rcu+0xb0/0xd0 [ 2050.104608] ---[ end trace 0000000000000000 ]--- [ 2050.109228] ------------[ cut here ]------------ [ 2050.113847] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2050.125496] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2050.134455] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2050.189987] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2050.198945] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2050.206425] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2050.211911] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2050.230658] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2050.235884] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2050.243016] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2050.250150] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2050.257283] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d37cb0 [ 2050.264414] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2050.271548] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2050.279639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2050.285388] CR2: 00007f9bd4b03020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2050.292522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2050.299654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2050.306786] Call Trace: [ 2050.309240] [ 2050.311348] ? __warn+0x85/0x180 [ 2050.314581] ? __list_add_valid_or_report+0x7d/0xa0 [ 2050.319461] ? report_bug+0xfc/0x1d0 [ 2050.323047] ? prb_read_valid+0x1b/0x20 [ 2050.326890] ? handle_bug+0x44/0x70 [ 2050.330386] ? exc_invalid_op+0x17/0x60 [ 2050.334231] ? asm_exc_invalid_op+0x1a/0x20 [ 2050.338433] ? __list_add_valid_or_report+0x7d/0xa0 [ 2050.343321] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2050.348220] try_get_cap_refs+0x433/0x880 [ceph] [ 2050.352862] ? _raw_spin_unlock+0x23/0x40 [ 2050.356880] __ceph_get_caps+0xf8/0x760 [ceph] [ 2050.361346] ? __vfs_getxattr+0x8d/0xb0 [ 2050.365191] ? dentry_needs_remove_privs+0x31/0x50 [ 2050.369992] ? __file_remove_privs+0x89/0x110 [ 2050.374355] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2050.379079] ? lock_release+0xc5/0x260 [ 2050.382835] ? ceph_write_iter+0x260/0x770 [ceph] [ 2050.387558] ceph_write_iter+0x260/0x770 [ceph] [ 2050.392112] ? lock_is_held_type+0xaa/0x120 [ 2050.396304] vfs_write+0x324/0x4f0 [ 2050.399724] ksys_pwrite64+0x65/0xa0 [ 2050.403310] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2050.408458] do_syscall_64+0x38/0x80 [ 2050.412037] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2050.417090] RIP: 0033:0x7f48f0eeb93b [ 2050.420670] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2050.439415] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2050.446981] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2050.454115] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2050.461248] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2050.468381] R10: 0000000000050000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2050.475512] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2050.482660] [ 2050.484855] irq event stamp: 12864511 [ 2050.488522] hardirqs last enabled at (12864521): [] console_unlock+0x106/0x120 [ 2050.497388] hardirqs last disabled at (12864530): [] console_unlock+0xeb/0x120 [ 2050.506168] softirqs last enabled at (12864458): [] __do_softirq+0x2be/0x371 [ 2050.514861] softirqs last disabled at (12864453): [] irq_exit_rcu+0xb0/0xd0 [ 2050.523381] ---[ end trace 0000000000000000 ]--- [ 2050.528000] ------------[ cut here ]------------ [ 2050.532762] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2050.544426] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2050.553397] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2050.608932] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2050.617895] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2050.625386] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2050.630890] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2050.649653] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2050.654894] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2050.662047] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2050.669194] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2050.676346] R10: 0000000000000001 R11: ffffc90001fdfb18 R12: ffff8881e4d31ae0 [ 2050.683494] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2050.690638] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2050.698740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2050.704503] CR2: 00007f9bd4b0d008 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2050.711656] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2050.718809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2050.725955] Call Trace: [ 2050.728425] [ 2050.730548] ? __warn+0x85/0x180 [ 2050.733798] ? __list_add_valid_or_report+0x7d/0xa0 [ 2050.738701] ? report_bug+0xfc/0x1d0 [ 2050.742300] ? prb_read_valid+0x1b/0x20 [ 2050.746160] ? handle_bug+0x44/0x70 [ 2050.749666] ? exc_invalid_op+0x17/0x60 [ 2050.753522] ? asm_exc_invalid_op+0x1a/0x20 [ 2050.757737] ? __list_add_valid_or_report+0x7d/0xa0 [ 2050.762633] ? __list_add_valid_or_report+0x7d/0xa0 [ 2050.767529] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2050.772449] ceph_open+0x16b/0x440 [ceph] [ 2050.776499] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2050.781671] do_dentry_open+0x172/0x530 [ 2050.785559] path_openat+0x743/0x9d0 [ 2050.789162] do_filp_open+0xb8/0x120 [ 2050.792769] ? lock_release+0xc5/0x260 [ 2050.796541] ? _raw_spin_unlock+0x23/0x40 [ 2050.800567] ? alloc_fd+0x13f/0x200 [ 2050.804082] do_sys_openat2+0x249/0x290 [ 2050.807940] do_sys_open+0x4f/0x80 [ 2050.811361] do_syscall_64+0x38/0x80 [ 2050.814956] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2050.820025] RIP: 0033:0x7f48f0eed2f2 [ 2050.823621] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2050.842383] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2050.849970] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2050.857107] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2050.864238] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2050.871369] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2050.878504] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b1 [ 2050.885648] [ 2050.887836] irq event stamp: 12752417 [ 2050.891503] hardirqs last enabled at (12752427): [] console_unlock+0x106/0x120 [ 2050.900369] hardirqs last disabled at (12752436): [] console_unlock+0xeb/0x120 [ 2050.909149] softirqs last enabled at (12752374): [] __do_softirq+0x2be/0x371 [ 2050.917849] softirqs last disabled at (12752455): [] irq_exit_rcu+0xb0/0xd0 [ 2050.926372] ---[ end trace 0000000000000000 ]--- [ 2050.930988] ------------[ cut here ]------------ [ 2050.935609] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2050.947258] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2050.956222] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2051.011750] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2051.020709] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2051.028186] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2051.033673] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2051.052424] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2051.057655] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2051.064788] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2051.071919] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2051.079056] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2051.086185] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2051.093319] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2051.101406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2051.107150] CR2: 00007f9bd4b03020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2051.114284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2051.121417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2051.128551] Call Trace: [ 2051.131002] [ 2051.133108] ? __warn+0x85/0x180 [ 2051.136341] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.141226] ? report_bug+0xfc/0x1d0 [ 2051.144809] ? prb_read_valid+0x1b/0x20 [ 2051.148651] ? handle_bug+0x44/0x70 [ 2051.152141] ? exc_invalid_op+0x17/0x60 [ 2051.155981] ? asm_exc_invalid_op+0x1a/0x20 [ 2051.160176] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.165057] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.169943] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2051.174843] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2051.180016] ceph_write_iter+0x5aa/0x770 [ceph] [ 2051.184573] ? lock_is_held_type+0xaa/0x120 [ 2051.188764] vfs_write+0x324/0x4f0 [ 2051.192184] ksys_pwrite64+0x65/0xa0 [ 2051.195768] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2051.200913] do_syscall_64+0x38/0x80 [ 2051.204499] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2051.209551] RIP: 0033:0x7f48f0eeb93b [ 2051.213128] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2051.231875] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2051.239441] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2051.246573] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2051.253706] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2051.260840] R10: 0000000000050000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2051.267975] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2051.275120] [ 2051.277316] irq event stamp: 12868021 [ 2051.280980] hardirqs last enabled at (12868031): [] console_unlock+0x106/0x120 [ 2051.289847] hardirqs last disabled at (12868040): [] console_unlock+0xeb/0x120 [ 2051.298626] softirqs last enabled at (12867998): [] __do_softirq+0x2be/0x371 [ 2051.307324] softirqs last disabled at (12868059): [] irq_exit_rcu+0xb0/0xd0 [ 2051.315848] ---[ end trace 0000000000000000 ]--- [ 2051.320469] ------------[ cut here ]------------ [ 2051.325090] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2051.336733] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2051.345688] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2051.401214] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2051.410166] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2051.417648] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2051.423133] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2051.441881] RSP: 0018:ffffc90001fdfbd0 EFLAGS: 00010282 [ 2051.447107] RAX: 0000000000000000 RBX: ffff8881e4d31390 RCX: 0000000000000000 [ 2051.454241] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2051.461374] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2051.468507] R10: 0000000000000001 R11: ffffc90001fdfa60 R12: ffff8881e4d31ae0 [ 2051.475639] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2051.482772] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2051.490856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2051.496603] CR2: 00007f9bd4b0d008 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2051.503737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2051.510870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2051.518001] Call Trace: [ 2051.520455] [ 2051.522562] ? __warn+0x85/0x180 [ 2051.525794] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.530674] ? report_bug+0xfc/0x1d0 [ 2051.534254] ? prb_read_valid+0x1b/0x20 [ 2051.538097] ? handle_bug+0x44/0x70 [ 2051.541593] ? exc_invalid_op+0x17/0x60 [ 2051.545433] ? asm_exc_invalid_op+0x1a/0x20 [ 2051.549627] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.554508] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2051.559409] try_get_cap_refs+0x433/0x880 [ceph] [ 2051.564059] __ceph_get_caps+0xf8/0x760 [ceph] [ 2051.568541] ? ceph_read_iter+0x134/0x860 [ceph] [ 2051.573175] ceph_read_iter+0x134/0x860 [ceph] [ 2051.577647] ? selinux_file_permission+0x119/0x150 [ 2051.582445] ? vfs_read+0x20f/0x300 [ 2051.585942] vfs_read+0x20f/0x300 [ 2051.589273] ksys_pread64+0x65/0xa0 [ 2051.592771] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2051.597915] do_syscall_64+0x38/0x80 [ 2051.601500] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2051.606551] RIP: 0033:0x7f48f0eeb88b [ 2051.610131] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2051.628876] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2051.636441] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb88b [ 2051.643574] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2051.650708] RBP: 00000000000029b1 R08: 000000000000005f R09: 0000000000000003 [ 2051.657844] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000800 [ 2051.664977] R13: 0000000000000800 R14: 000000000207df80 R15: 00007f48f183dac8 [ 2051.672121] [ 2051.674316] irq event stamp: 12754345 [ 2051.677984] hardirqs last enabled at (12754355): [] console_unlock+0x106/0x120 [ 2051.686848] hardirqs last disabled at (12754364): [] console_unlock+0xeb/0x120 [ 2051.695629] softirqs last enabled at (12754322): [] __do_softirq+0x2be/0x371 [ 2051.704321] softirqs last disabled at (12754317): [] irq_exit_rcu+0xb0/0xd0 [ 2051.712840] ---[ end trace 0000000000000000 ]--- [ 2051.717460] ------------[ cut here ]------------ [ 2051.722078] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2051.733725] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2051.742682] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2051.798161] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2051.807119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2051.814598] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2051.820087] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2051.838838] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2051.844065] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2051.851196] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2051.858329] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2051.865464] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d37cb0 [ 2051.872596] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2051.879729] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2051.887816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2051.893564] CR2: 00007f9bd4b03020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2051.900695] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2051.907825] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2051.914960] Call Trace: [ 2051.917412] [ 2051.919520] ? __warn+0x85/0x180 [ 2051.922761] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.927642] ? report_bug+0xfc/0x1d0 [ 2051.931231] ? prb_read_valid+0x1b/0x20 [ 2051.935088] ? handle_bug+0x44/0x70 [ 2051.938586] ? exc_invalid_op+0x17/0x60 [ 2051.942426] ? asm_exc_invalid_op+0x1a/0x20 [ 2051.946620] ? __list_add_valid_or_report+0x7d/0xa0 [ 2051.951502] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2051.956398] try_get_cap_refs+0x433/0x880 [ceph] [ 2051.961035] ? _raw_spin_unlock+0x23/0x40 [ 2051.965061] __ceph_get_caps+0xf8/0x760 [ceph] [ 2051.969528] ? __vfs_getxattr+0x8d/0xb0 [ 2051.973377] ? dentry_needs_remove_privs+0x31/0x50 [ 2051.978178] ? __file_remove_privs+0x89/0x110 [ 2051.982543] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2051.987270] ? lock_release+0xc5/0x260 [ 2051.991031] ? ceph_write_iter+0x260/0x770 [ceph] [ 2051.995757] ceph_write_iter+0x260/0x770 [ceph] [ 2052.000309] ? lock_is_held_type+0xaa/0x120 [ 2052.004503] vfs_write+0x324/0x4f0 [ 2052.007921] ksys_pwrite64+0x65/0xa0 [ 2052.011499] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2052.016641] do_syscall_64+0x38/0x80 [ 2052.020221] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2052.025277] RIP: 0033:0x7f48f0eeb93b [ 2052.028857] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2052.047606] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2052.055170] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2052.062306] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2052.069437] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2052.076574] R10: 0000000000060000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2052.083710] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2052.090855] [ 2052.093045] irq event stamp: 12869977 [ 2052.096713] hardirqs last enabled at (12869987): [] console_unlock+0x106/0x120 [ 2052.105576] hardirqs last disabled at (12869996): [] console_unlock+0xeb/0x120 [ 2052.114358] softirqs last enabled at (12869944): [] __do_softirq+0x2be/0x371 [ 2052.123058] softirqs last disabled at (12869939): [] irq_exit_rcu+0xb0/0xd0 [ 2052.131581] ---[ end trace 0000000000000000 ]--- [ 2052.136197] ------------[ cut here ]------------ [ 2052.140815] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2052.152470] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2052.161430] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2052.216970] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2052.225926] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2052.233408] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2052.238900] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2052.257646] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2052.262873] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2052.270006] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2052.277138] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2052.284275] R10: 0000000000000001 R11: ffffc90001fd7a70 R12: ffff8881d09f8750 [ 2052.291413] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2052.298547] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2052.306633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2052.312378] CR2: 00007fb38ed20000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2052.319517] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2052.326654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2052.333785] Call Trace: [ 2052.336239] [ 2052.338346] ? __warn+0x85/0x180 [ 2052.341588] ? __list_add_valid_or_report+0x7d/0xa0 [ 2052.346481] ? report_bug+0xfc/0x1d0 [ 2052.350061] ? prb_read_valid+0x1b/0x20 [ 2052.353907] ? handle_bug+0x44/0x70 [ 2052.357403] ? exc_invalid_op+0x17/0x60 [ 2052.361242] ? asm_exc_invalid_op+0x1a/0x20 [ 2052.365440] ? __list_add_valid_or_report+0x7d/0xa0 [ 2052.370328] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2052.375240] try_get_cap_refs+0x433/0x880 [ceph] [ 2052.379879] ? _raw_spin_unlock+0x23/0x40 [ 2052.383907] __ceph_get_caps+0xf8/0x760 [ceph] [ 2052.388371] ? __vfs_getxattr+0x8d/0xb0 [ 2052.392225] ? dentry_needs_remove_privs+0x31/0x50 [ 2052.397026] ? __file_remove_privs+0x89/0x110 [ 2052.401387] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2052.406112] ? lock_release+0xc5/0x260 [ 2052.409878] ? ceph_write_iter+0x260/0x770 [ceph] [ 2052.414603] ceph_write_iter+0x260/0x770 [ceph] [ 2052.419165] ? lock_is_held_type+0xaa/0x120 [ 2052.423364] vfs_write+0x324/0x4f0 [ 2052.426784] ksys_pwrite64+0x65/0xa0 [ 2052.430369] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2052.435512] do_syscall_64+0x38/0x80 [ 2052.439098] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2052.444158] RIP: 0033:0x7f48f0eeb93b [ 2052.447737] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2052.466484] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2052.474062] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2052.481200] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2052.488335] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2052.495474] R10: 0000000000060000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2052.502607] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2052.509754] [ 2052.511948] irq event stamp: 12848743 [ 2052.515615] hardirqs last enabled at (12848753): [] console_unlock+0x106/0x120 [ 2052.524488] hardirqs last disabled at (12848770): [] console_unlock+0xeb/0x120 [ 2052.533269] softirqs last enabled at (12848768): [] __do_softirq+0x2be/0x371 [ 2052.541963] softirqs last disabled at (12848761): [] irq_exit_rcu+0xb0/0xd0 [ 2052.550482] ---[ end trace 0000000000000000 ]--- [ 2052.555103] ------------[ cut here ]------------ [ 2052.559720] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2052.571372] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2052.580071] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2052.635630] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2052.644761] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2052.652240] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2052.657758] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2052.663245] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2052.681989] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2052.687214] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2052.694347] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2052.701479] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2052.708605] R10: 0000000000000001 R11: ffffc9000030b828 R12: ffff8881d1009118 [ 2052.715738] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2052.722870] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2052.730956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2052.736701] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2052.743836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2052.750970] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2052.758099] Call Trace: [ 2052.760546] [ 2052.762652] ? __warn+0x85/0x180 [ 2052.765883] ? __list_add_valid_or_report+0x7d/0xa0 [ 2052.770764] ? report_bug+0xfc/0x1d0 [ 2052.774341] ? prb_read_valid+0x1b/0x20 [ 2052.778183] ? handle_bug+0x44/0x70 [ 2052.781675] ? exc_invalid_op+0x17/0x60 [ 2052.785515] ? asm_exc_invalid_op+0x1a/0x20 [ 2052.789706] ? __list_add_valid_or_report+0x7d/0xa0 [ 2052.794581] ? __list_add_valid_or_report+0x7d/0xa0 [ 2052.799460] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2052.804349] ceph_add_cap+0x20b/0x620 [ceph] [ 2052.808635] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2052.813365] ceph_fill_trace+0x143/0xf30 [ceph] [ 2052.817903] ? __mutex_lock+0x97/0xce0 [ 2052.821653] ? __mutex_lock+0xaec/0xce0 [ 2052.825494] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2052.830037] ? lock_downgrade+0xfd/0x250 [ 2052.833967] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2052.838346] ceph_con_process_message+0x74/0x150 [libceph] [ 2052.843842] ? calc_signature+0xdb/0x100 [libceph] [ 2052.848646] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2052.853970] ? process_scheduled_works+0x22a/0x510 [ 2052.858769] ? process_scheduled_works+0x22a/0x510 [ 2052.863558] ceph_con_workfn+0x344/0x750 [libceph] [ 2052.868363] ? process_scheduled_works+0x22a/0x510 [ 2052.873153] process_scheduled_works+0x2a4/0x510 [ 2052.877778] worker_thread+0x188/0x340 [ 2052.881526] ? rescuer_thread+0x310/0x310 [ 2052.885539] kthread+0xf5/0x130 [ 2052.888684] ? kthread_complete_and_exit+0x20/0x20 [ 2052.893477] ret_from_fork+0x31/0x40 [ 2052.897056] ? kthread_complete_and_exit+0x20/0x20 [ 2052.901849] ret_from_fork_asm+0x11/0x20 [ 2052.905782] [ 2052.907967] irq event stamp: 8943811 [ 2052.911547] hardirqs last enabled at (8943821): [] console_unlock+0x106/0x120 [ 2052.920326] hardirqs last disabled at (8943830): [] console_unlock+0xeb/0x120 [ 2052.929019] softirqs last enabled at (8943728): [] __do_softirq+0x2be/0x371 [ 2052.937624] softirqs last disabled at (8943723): [] irq_exit_rcu+0xb0/0xd0 [ 2052.946059] ---[ end trace 0000000000000000 ]--- [ 2052.950680] ------------[ cut here ]------------ [ 2052.955297] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2052.966966] WARNING: CPU: 7 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2052.975928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2053.031457] CPU: 7 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2053.040408] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2053.047887] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2053.053374] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2053.072121] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2053.077345] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2053.084480] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2053.091611] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2053.098747] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2053.105877] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2053.113010] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2053.121096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2053.126843] CR2: 00007fb38ed50000 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2053.133974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2053.141106] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2053.148239] Call Trace: [ 2053.150692] [ 2053.152799] ? __warn+0x85/0x180 [ 2053.156032] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.160914] ? report_bug+0xfc/0x1d0 [ 2053.164499] ? prb_read_valid+0x1b/0x20 [ 2053.168343] ? handle_bug+0x44/0x70 [ 2053.171841] ? exc_invalid_op+0x17/0x60 [ 2053.175680] ? asm_exc_invalid_op+0x1a/0x20 [ 2053.179876] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.184760] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.189642] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2053.194545] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2053.199716] ceph_write_iter+0x5aa/0x770 [ceph] [ 2053.204271] ? lock_is_held_type+0xaa/0x120 [ 2053.208465] vfs_write+0x324/0x4f0 [ 2053.211884] ksys_pwrite64+0x65/0xa0 [ 2053.215469] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2053.220612] do_syscall_64+0x38/0x80 [ 2053.224199] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2053.229250] RIP: 0033:0x7f48f0eeb93b [ 2053.232834] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2053.251583] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2053.259150] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2053.266291] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2053.273423] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2053.280561] R10: 0000000000060000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2053.287699] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2053.294845] [ 2053.297040] irq event stamp: 12850671 [ 2053.300710] hardirqs last enabled at (12850687): [] console_unlock+0x106/0x120 [ 2053.309582] hardirqs last disabled at (12850696): [] console_unlock+0xeb/0x120 [ 2053.318361] softirqs last enabled at (12850684): [] __do_softirq+0x2be/0x371 [ 2053.327055] softirqs last disabled at (12850679): [] irq_exit_rcu+0xb0/0xd0 [ 2053.335574] ---[ end trace 0000000000000000 ]--- [ 2053.340209] ------------[ cut here ]------------ [ 2053.344828] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2053.356479] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2053.365441] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2053.420970] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2053.429922] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2053.437400] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2053.442888] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2053.461633] RSP: 0018:ffffc90001fefd18 EFLAGS: 00010286 [ 2053.466859] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2053.473993] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2053.481124] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2053.488259] R10: 0000000000000001 R11: ffffc90001fefba8 R12: ffff8881c2641118 [ 2053.495393] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2053.502523] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2053.510610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2053.516355] CR2: 00007f9bd4b30010 CR3: 0000000108c72003 CR4: 00000000003706f0 [ 2053.523489] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2053.530621] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2053.537754] Call Trace: [ 2053.540208] [ 2053.542313] ? __warn+0x85/0x180 [ 2053.545547] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.550427] ? report_bug+0xfc/0x1d0 [ 2053.554007] ? prb_read_valid+0x1b/0x20 [ 2053.557846] ? handle_bug+0x44/0x70 [ 2053.561337] ? exc_invalid_op+0x17/0x60 [ 2053.565177] ? asm_exc_invalid_op+0x1a/0x20 [ 2053.569371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.574256] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.579136] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2053.584043] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2053.589212] ceph_write_iter+0x5aa/0x770 [ceph] [ 2053.593774] ? lock_is_held_type+0xaa/0x120 [ 2053.597974] vfs_write+0x324/0x4f0 [ 2053.601405] ksys_pwrite64+0x65/0xa0 [ 2053.604990] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2053.610136] do_syscall_64+0x38/0x80 [ 2053.613721] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2053.618774] RIP: 0033:0x7f48f0eeb93b [ 2053.622360] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2053.641107] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2053.648672] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2053.655805] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000012 [ 2053.662939] RBP: 00000000000029b0 R08: 0000000000000001 R09: 0000000002074010 [ 2053.670071] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183df18 [ 2053.677206] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2053.684361] [ 2053.686556] irq event stamp: 12743117 [ 2053.690222] hardirqs last enabled at (12743133): [] console_unlock+0x106/0x120 [ 2053.699087] hardirqs last disabled at (12743148): [] console_unlock+0xeb/0x120 [ 2053.707865] softirqs last enabled at (12743162): [] __do_softirq+0x2be/0x371 [ 2053.716560] softirqs last disabled at (12743173): [] irq_exit_rcu+0xb0/0xd0 [ 2053.725087] ---[ end trace 0000000000000000 ]--- [ 2053.729707] ------------[ cut here ]------------ [ 2053.734324] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2053.745973] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2053.754927] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2053.810399] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2053.819355] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2053.826835] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2053.832324] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2053.851077] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2053.856311] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2053.863449] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2053.870585] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2053.877725] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2053.884858] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2053.891991] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2053.900079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2053.905822] CR2: 00007f9bd4b03020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2053.912958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2053.920088] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2053.927224] Call Trace: [ 2053.929674] [ 2053.931781] ? __warn+0x85/0x180 [ 2053.935016] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.939895] ? report_bug+0xfc/0x1d0 [ 2053.943472] ? prb_read_valid+0x1b/0x20 [ 2053.947316] ? handle_bug+0x44/0x70 [ 2053.950804] ? exc_invalid_op+0x17/0x60 [ 2053.954645] ? asm_exc_invalid_op+0x1a/0x20 [ 2053.958841] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.963720] ? __list_add_valid_or_report+0x7d/0xa0 [ 2053.968599] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2053.973498] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2053.978661] ceph_write_iter+0x5aa/0x770 [ceph] [ 2053.983217] ? lock_is_held_type+0xaa/0x120 [ 2053.987410] vfs_write+0x324/0x4f0 [ 2053.990834] ksys_pwrite64+0x65/0xa0 [ 2053.994416] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2053.999559] do_syscall_64+0x38/0x80 [ 2054.003136] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2054.008186] RIP: 0033:0x7f48f0eeb93b [ 2054.011771] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2054.030512] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2054.038079] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2054.045220] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2054.052353] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2054.059487] R10: 0000000000060000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2054.066619] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2054.073766] [ 2054.075956] irq event stamp: 12874459 [ 2054.079618] hardirqs last enabled at (12874469): [] console_unlock+0x106/0x120 [ 2054.088484] hardirqs last disabled at (12874478): [] console_unlock+0xeb/0x120 [ 2054.097265] softirqs last enabled at (12874456): [] __do_softirq+0x2be/0x371 [ 2054.105957] softirqs last disabled at (12874451): [] irq_exit_rcu+0xb0/0xd0 [ 2054.114477] ---[ end trace 0000000000000000 ]--- [ 2054.119096] ------------[ cut here ]------------ [ 2054.123717] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2054.135365] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2054.144317] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2054.199849] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2054.208797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2054.216277] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2054.221763] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2054.240512] RSP: 0018:ffffc90001fefbe0 EFLAGS: 00010286 [ 2054.245737] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2054.252870] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2054.260003] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2054.267140] R10: 0000000000000001 R11: ffffc90001fefa70 R12: ffff8881c2641118 [ 2054.274269] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2054.281402] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2054.289485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2054.295233] CR2: 00007f9bd4b61000 CR3: 0000000108c72003 CR4: 00000000003706e0 [ 2054.302381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2054.309516] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2054.316648] Call Trace: [ 2054.319101] [ 2054.321209] ? __warn+0x85/0x180 [ 2054.324443] ? __list_add_valid_or_report+0x7d/0xa0 [ 2054.329321] ? report_bug+0xfc/0x1d0 [ 2054.332900] ? prb_read_valid+0x1b/0x20 [ 2054.336746] ? handle_bug+0x44/0x70 [ 2054.340240] ? exc_invalid_op+0x17/0x60 [ 2054.344079] ? asm_exc_invalid_op+0x1a/0x20 [ 2054.348281] ? __list_add_valid_or_report+0x7d/0xa0 [ 2054.353164] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2054.358067] try_get_cap_refs+0x433/0x880 [ceph] [ 2054.362708] ? _raw_spin_unlock+0x23/0x40 [ 2054.366727] __ceph_get_caps+0xf8/0x760 [ceph] [ 2054.371190] ? __vfs_getxattr+0x8d/0xb0 [ 2054.375036] ? dentry_needs_remove_privs+0x31/0x50 [ 2054.379831] ? __file_remove_privs+0x89/0x110 [ 2054.384188] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2054.388914] ? lock_release+0xc5/0x260 [ 2054.392671] ? ceph_write_iter+0x260/0x770 [ceph] [ 2054.397390] ceph_write_iter+0x260/0x770 [ceph] [ 2054.401957] ? lock_is_held_type+0xaa/0x120 [ 2054.406148] vfs_write+0x324/0x4f0 [ 2054.409569] ksys_pwrite64+0x65/0xa0 [ 2054.413154] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2054.418295] do_syscall_64+0x38/0x80 [ 2054.421876] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2054.426927] RIP: 0033:0x7f48f0eeb93b [ 2054.430510] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2054.449263] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2054.456834] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2054.463966] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2054.471099] RBP: 00000000000029b0 R08: 0000000000000800 R09: 0000000000000003 [ 2054.478233] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183df18 [ 2054.485371] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207df80 [ 2054.492514] [ 2054.494709] irq event stamp: 12744879 [ 2054.498377] hardirqs last enabled at (12744895): [] console_unlock+0x106/0x120 [ 2054.507248] hardirqs last disabled at (12744904): [] console_unlock+0xeb/0x120 [ 2054.516029] softirqs last enabled at (12744892): [] __do_softirq+0x2be/0x371 [ 2054.524723] softirqs last disabled at (12744923): [] irq_exit_rcu+0xb0/0xd0 [ 2054.533240] ---[ end trace 0000000000000000 ]--- [ 2054.537860] ------------[ cut here ]------------ [ 2054.542481] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2054.554129] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2054.563082] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2054.618613] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2054.627566] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2054.635042] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2054.640527] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2054.659273] RSP: 0018:ffffc90001fc7be0 EFLAGS: 00010286 [ 2054.664500] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2054.671634] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2054.678765] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2054.685900] R10: 0000000000000001 R11: ffffc90001fc7a70 R12: ffff8881a1d37cb0 [ 2054.693030] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2054.700166] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2054.708250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2054.713998] CR2: 00007f9bd4b03020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2054.721137] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2054.728274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2054.735403] Call Trace: [ 2054.737856] [ 2054.739963] ? __warn+0x85/0x180 [ 2054.743195] ? __list_add_valid_or_report+0x7d/0xa0 [ 2054.748076] ? report_bug+0xfc/0x1d0 [ 2054.751656] ? prb_read_valid+0x1b/0x20 [ 2054.755503] ? handle_bug+0x44/0x70 [ 2054.758994] ? exc_invalid_op+0x17/0x60 [ 2054.762836] ? asm_exc_invalid_op+0x1a/0x20 [ 2054.767030] ? __list_add_valid_or_report+0x7d/0xa0 [ 2054.771912] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2054.776809] try_get_cap_refs+0x433/0x880 [ceph] [ 2054.781444] ? _raw_spin_unlock+0x23/0x40 [ 2054.785463] __ceph_get_caps+0xf8/0x760 [ceph] [ 2054.789929] ? __vfs_getxattr+0x8d/0xb0 [ 2054.793771] ? dentry_needs_remove_privs+0x31/0x50 [ 2054.798569] ? __file_remove_privs+0x89/0x110 [ 2054.802928] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2054.807653] ? lock_release+0xc5/0x260 [ 2054.811408] ? ceph_write_iter+0x260/0x770 [ceph] [ 2054.816136] ceph_write_iter+0x260/0x770 [ceph] [ 2054.820695] ? lock_is_held_type+0xaa/0x120 [ 2054.824887] vfs_write+0x324/0x4f0 [ 2054.828305] ksys_pwrite64+0x65/0xa0 [ 2054.831883] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2054.837025] do_syscall_64+0x38/0x80 [ 2054.840602] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2054.845655] RIP: 0033:0x7f48f0eeb93b [ 2054.849236] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2054.867981] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2054.875546] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2054.882689] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2054.889821] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2054.896953] R10: 0000000000070000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2054.904087] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2054.911234] [ 2054.913432] irq event stamp: 12876479 [ 2054.917095] hardirqs last enabled at (12876489): [] console_unlock+0x106/0x120 [ 2054.925962] hardirqs last disabled at (12876498): [] console_unlock+0xeb/0x120 [ 2054.934741] softirqs last enabled at (12876476): [] __do_softirq+0x2be/0x371 [ 2054.943437] softirqs last disabled at (12876517): [] irq_exit_rcu+0xb0/0xd0 [ 2054.951962] ---[ end trace 0000000000000000 ]--- [ 2054.956582] ------------[ cut here ]------------ [ 2054.961201] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2054.972849] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2054.981802] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2055.037329] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2055.046290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2055.053773] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2055.059257] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2055.078004] RSP: 0018:ffffc90001fefd18 EFLAGS: 00010286 [ 2055.083230] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2055.090363] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2055.097494] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2055.104627] R10: 0000000000000001 R11: ffffc90001fefba8 R12: ffff8881c2641118 [ 2055.111760] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2055.118893] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2055.126978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2055.132726] CR2: 00007f9bd4b61000 CR3: 0000000108c72003 CR4: 00000000003706e0 [ 2055.139858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2055.146992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2055.154125] Call Trace: [ 2055.156576] [ 2055.158685] ? __warn+0x85/0x180 [ 2055.161917] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.166797] ? report_bug+0xfc/0x1d0 [ 2055.170374] ? prb_read_valid+0x1b/0x20 [ 2055.174217] ? handle_bug+0x44/0x70 [ 2055.177706] ? exc_invalid_op+0x17/0x60 [ 2055.181546] ? asm_exc_invalid_op+0x1a/0x20 [ 2055.185741] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.190621] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.195500] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2055.200399] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2055.205560] ceph_write_iter+0x5aa/0x770 [ceph] [ 2055.210113] ? lock_is_held_type+0xaa/0x120 [ 2055.214304] vfs_write+0x324/0x4f0 [ 2055.217723] ksys_pwrite64+0x65/0xa0 [ 2055.221302] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2055.226442] do_syscall_64+0x38/0x80 [ 2055.230021] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2055.235071] RIP: 0033:0x7f48f0eeb93b [ 2055.238652] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2055.257397] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2055.264964] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2055.272097] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2055.279235] RBP: 00000000000029b0 R08: 0000000000000800 R09: 0000000000000003 [ 2055.286371] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183df18 [ 2055.293505] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207df80 [ 2055.300651] [ 2055.302837] irq event stamp: 12746845 [ 2055.306504] hardirqs last enabled at (12746855): [] console_unlock+0x106/0x120 [ 2055.315373] hardirqs last disabled at (12746870): [] console_unlock+0xeb/0x120 [ 2055.324151] softirqs last enabled at (12746868): [] __do_softirq+0x2be/0x371 [ 2055.332842] softirqs last disabled at (12746863): [] irq_exit_rcu+0xb0/0xd0 [ 2055.341361] ---[ end trace 0000000000000000 ]--- [ 2055.345980] ------------[ cut here ]------------ [ 2055.350602] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2055.362251] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2055.371210] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2055.426743] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2055.435692] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2055.443170] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2055.448657] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2055.467403] RSP: 0018:ffffc90001fc7d18 EFLAGS: 00010286 [ 2055.472631] RAX: 0000000000000000 RBX: ffff8881a1d37560 RCX: 0000000000000000 [ 2055.479761] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2055.486894] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2055.494032] R10: 0000000000000001 R11: ffffc90001fc7ba8 R12: ffff8881a1d37cb0 [ 2055.501162] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2055.508294] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2055.516380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2055.522126] CR2: 00007f9bd4b03020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2055.529260] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2055.536392] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2055.543525] Call Trace: [ 2055.545977] [ 2055.548083] ? __warn+0x85/0x180 [ 2055.551316] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.556202] ? report_bug+0xfc/0x1d0 [ 2055.559790] ? prb_read_valid+0x1b/0x20 [ 2055.563634] ? handle_bug+0x44/0x70 [ 2055.567125] ? exc_invalid_op+0x17/0x60 [ 2055.570965] ? asm_exc_invalid_op+0x1a/0x20 [ 2055.575160] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.580039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.584916] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2055.589819] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2055.594980] ceph_write_iter+0x5aa/0x770 [ceph] [ 2055.599537] ? lock_is_held_type+0xaa/0x120 [ 2055.603732] vfs_write+0x324/0x4f0 [ 2055.607151] ksys_pwrite64+0x65/0xa0 [ 2055.610727] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2055.615871] do_syscall_64+0x38/0x80 [ 2055.619457] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2055.624507] RIP: 0033:0x7f48f0eeb93b [ 2055.628090] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2055.646832] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2055.654398] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2055.661538] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2055.668674] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2055.675806] R10: 0000000000070000 R11: 0000000000000246 R12: 00007f48f183d450 [ 2055.682942] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2055.690084] [ 2055.692273] irq event stamp: 12878437 [ 2055.695942] hardirqs last enabled at (12878453): [] console_unlock+0x106/0x120 [ 2055.704805] hardirqs last disabled at (12878462): [] console_unlock+0xeb/0x120 [ 2055.713585] softirqs last enabled at (12878450): [] __do_softirq+0x2be/0x371 [ 2055.722277] softirqs last disabled at (12878445): [] irq_exit_rcu+0xb0/0xd0 [ 2055.730798] ---[ end trace 0000000000000000 ]--- [ 2055.735417] ------------[ cut here ]------------ [ 2055.740038] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2055.751687] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2055.760650] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2055.816157] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2055.825130] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2055.832609] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2055.838096] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2055.856853] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2055.862084] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2055.869216] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2055.876349] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2055.883480] R10: 0000000000000001 R11: ffffc90001fefb18 R12: ffff8881c2641118 [ 2055.890617] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2055.897748] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2055.905838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2055.911586] CR2: 00007f9bd4b61000 CR3: 0000000108c72003 CR4: 00000000003706e0 [ 2055.918721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2055.925852] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2055.932986] Call Trace: [ 2055.935438] [ 2055.937547] ? __warn+0x85/0x180 [ 2055.940777] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.945661] ? report_bug+0xfc/0x1d0 [ 2055.949245] ? prb_read_valid+0x1b/0x20 [ 2055.953090] ? handle_bug+0x44/0x70 [ 2055.956588] ? exc_invalid_op+0x17/0x60 [ 2055.960430] ? asm_exc_invalid_op+0x1a/0x20 [ 2055.964628] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.969513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2055.974396] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2055.979303] ceph_open+0x16b/0x440 [ceph] [ 2055.983328] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2055.988481] do_dentry_open+0x172/0x530 [ 2055.992321] path_openat+0x743/0x9d0 [ 2055.995907] do_filp_open+0xb8/0x120 [ 2055.999493] ? lock_release+0xc5/0x260 [ 2056.003255] ? _raw_spin_unlock+0x23/0x40 [ 2056.007270] ? alloc_fd+0x13f/0x200 [ 2056.010770] do_sys_openat2+0x249/0x290 [ 2056.014614] do_sys_open+0x4f/0x80 [ 2056.018030] do_syscall_64+0x38/0x80 [ 2056.021617] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2056.026675] RIP: 0033:0x7f48f0eed2f2 [ 2056.030253] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2056.049001] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2056.056567] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2056.063701] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2056.070834] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2056.077965] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2056.085112] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b1 [ 2056.092264] [ 2056.094459] irq event stamp: 12748863 [ 2056.098126] hardirqs last enabled at (12748873): [] console_unlock+0x106/0x120 [ 2056.106999] hardirqs last disabled at (12748882): [] console_unlock+0xeb/0x120 [ 2056.115788] softirqs last enabled at (12748840): [] __do_softirq+0x2be/0x371 [ 2056.124492] softirqs last disabled at (12748835): [] irq_exit_rcu+0xb0/0xd0 [ 2056.133016] ---[ end trace 0000000000000000 ]--- [ 2056.137640] ------------[ cut here ]------------ [ 2056.142354] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2056.154014] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2056.163026] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2056.218508] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2056.227521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2056.235006] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2056.240560] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2056.259311] RSP: 0018:ffffc90001fcfc88 EFLAGS: 00010286 [ 2056.264537] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2056.271666] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2056.278806] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2056.285943] R10: 0000000000000001 R11: ffffc90001fcfb18 R12: ffff8881e4da1118 [ 2056.293075] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2056.300217] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2056.308307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2056.314047] CR2: 00007fb38ed17000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2056.321180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2056.328313] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2056.335445] Call Trace: [ 2056.337891] [ 2056.339999] ? __warn+0x85/0x180 [ 2056.343228] ? __list_add_valid_or_report+0x7d/0xa0 [ 2056.348110] ? report_bug+0xfc/0x1d0 [ 2056.351690] ? prb_read_valid+0x1b/0x20 [ 2056.355531] ? handle_bug+0x44/0x70 [ 2056.359022] ? exc_invalid_op+0x17/0x60 [ 2056.362860] ? asm_exc_invalid_op+0x1a/0x20 [ 2056.367055] ? __list_add_valid_or_report+0x7d/0xa0 [ 2056.371936] ? __list_add_valid_or_report+0x7d/0xa0 [ 2056.376813] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2056.381714] ceph_open+0x16b/0x440 [ceph] [ 2056.385735] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2056.390880] do_dentry_open+0x172/0x530 [ 2056.394722] path_openat+0x743/0x9d0 [ 2056.398310] do_filp_open+0xb8/0x120 [ 2056.401893] ? lock_release+0xc5/0x260 [ 2056.405656] ? _raw_spin_unlock+0x23/0x40 [ 2056.409669] ? alloc_fd+0x13f/0x200 [ 2056.413166] do_sys_openat2+0x249/0x290 [ 2056.417007] do_sys_open+0x4f/0x80 [ 2056.420411] do_syscall_64+0x38/0x80 [ 2056.423988] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2056.429040] RIP: 0033:0x7f48f0eed2f2 [ 2056.432621] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2056.451365] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2056.458930] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2056.466064] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2056.473204] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2056.480331] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2056.487461] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b0 [ 2056.494605] [ 2056.496796] irq event stamp: 12744541 [ 2056.500452] hardirqs last enabled at (12744557): [] console_unlock+0x106/0x120 [ 2056.509319] hardirqs last disabled at (12744572): [] console_unlock+0xeb/0x120 [ 2056.518100] softirqs last enabled at (12744590): [] __do_softirq+0x2be/0x371 [ 2056.526791] softirqs last disabled at (12744601): [] irq_exit_rcu+0xb0/0xd0 [ 2056.535312] ---[ end trace 0000000000000000 ]--- [ 2056.539933] ------------[ cut here ]------------ [ 2056.544723] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2056.556591] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2056.565700] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2056.621489] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2056.630444] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2056.637920] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2056.643407] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2056.662153] RSP: 0018:ffffc90001fefbd0 EFLAGS: 00010282 [ 2056.667379] RAX: 0000000000000000 RBX: ffff8881c26409c8 RCX: 0000000000000000 [ 2056.674513] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2056.681645] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2056.688786] R10: 0000000000000001 R11: ffffc90001fefa60 R12: ffff8881c2641118 [ 2056.695918] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2056.703051] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2056.711138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2056.716885] CR2: 00007f9bd4b7d000 CR3: 0000000108c72001 CR4: 00000000003706e0 [ 2056.724018] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2056.731150] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2056.738281] Call Trace: [ 2056.740735] [ 2056.742843] ? __warn+0x85/0x180 [ 2056.746076] ? __list_add_valid_or_report+0x7d/0xa0 [ 2056.750964] ? report_bug+0xfc/0x1d0 [ 2056.754555] ? prb_read_valid+0x1b/0x20 [ 2056.758404] ? handle_bug+0x44/0x70 [ 2056.761899] ? exc_invalid_op+0x17/0x60 [ 2056.765739] ? asm_exc_invalid_op+0x1a/0x20 [ 2056.769939] ? __list_add_valid_or_report+0x7d/0xa0 [ 2056.774825] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2056.779733] try_get_cap_refs+0x433/0x880 [ceph] [ 2056.784386] __ceph_get_caps+0xf8/0x760 [ceph] [ 2056.788876] ? ceph_read_iter+0x134/0x860 [ceph] [ 2056.793509] ceph_read_iter+0x134/0x860 [ceph] [ 2056.797981] ? selinux_file_permission+0x119/0x150 [ 2056.802778] ? vfs_read+0x20f/0x300 [ 2056.806273] vfs_read+0x20f/0x300 [ 2056.809606] ksys_pread64+0x65/0xa0 [ 2056.813103] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2056.818248] do_syscall_64+0x38/0x80 [ 2056.821833] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2056.826893] RIP: 0033:0x7f48f0eeb88b [ 2056.830472] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2056.849219] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2056.856783] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb88b [ 2056.863917] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2056.871048] RBP: 00000000000029b1 R08: 000000000000005f R09: 0000000000000003 [ 2056.878183] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000800 [ 2056.885317] R13: 0000000000000800 R14: 000000000207df80 R15: 00007f48f183df18 [ 2056.892462] [ 2056.894657] irq event stamp: 12752003 [ 2056.898324] hardirqs last enabled at (12752013): [] console_unlock+0x106/0x120 [ 2056.907191] hardirqs last disabled at (12752022): [] console_unlock+0xeb/0x120 [ 2056.915969] softirqs last enabled at (12751980): [] __do_softirq+0x2be/0x371 [ 2056.924662] softirqs last disabled at (12751975): [] irq_exit_rcu+0xb0/0xd0 [ 2056.933190] ---[ end trace 0000000000000000 ]--- [ 2056.937810] ------------[ cut here ]------------ [ 2056.942428] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2056.954072] WARNING: CPU: 6 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2056.963028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2057.018497] CPU: 6 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2057.027447] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2057.034919] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2057.040406] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2057.059157] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2057.064378] RAX: 0000000000000000 RBX: ffff8881d09f8000 RCX: 0000000000000000 [ 2057.071512] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2057.078643] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2057.085777] R10: 0000000000000001 R11: ffffc90001fd7ba8 R12: ffff8881d09f8750 [ 2057.092909] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2057.100040] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2057.108118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2057.113855] CR2: 00007f9bd4b81000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2057.120979] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2057.128104] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2057.135228] Call Trace: [ 2057.137672] [ 2057.139770] ? __warn+0x85/0x180 [ 2057.143004] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.147883] ? report_bug+0xfc/0x1d0 [ 2057.151463] ? prb_read_valid+0x1b/0x20 [ 2057.155302] ? handle_bug+0x44/0x70 [ 2057.158793] ? exc_invalid_op+0x17/0x60 [ 2057.162625] ? asm_exc_invalid_op+0x1a/0x20 [ 2057.166815] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.171692] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.176570] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2057.181461] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2057.186610] ceph_write_iter+0x5aa/0x770 [ceph] [ 2057.191151] ? lock_is_held_type+0xaa/0x120 [ 2057.195336] vfs_write+0x324/0x4f0 [ 2057.198746] ksys_pwrite64+0x65/0xa0 [ 2057.202321] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2057.207459] do_syscall_64+0x38/0x80 [ 2057.211038] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2057.216092] RIP: 0033:0x7f48f0eeb93b [ 2057.219669] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2057.238416] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2057.245982] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb93b [ 2057.253114] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000007 [ 2057.260238] RBP: 0000000000002a11 R08: 0000000000010000 R09: 0000000000000003 [ 2057.267362] R10: 0000000000070000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2057.274488] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2057.281628] [ 2057.283812] irq event stamp: 12858743 [ 2057.287471] hardirqs last enabled at (12858753): [] console_unlock+0x106/0x120 [ 2057.296337] hardirqs last disabled at (12858762): [] console_unlock+0xeb/0x120 [ 2057.305115] softirqs last enabled at (12858730): [] __do_softirq+0x2be/0x371 [ 2057.313809] softirqs last disabled at (12858725): [] irq_exit_rcu+0xb0/0xd0 [ 2057.322329] ---[ end trace 0000000000000000 ]--- [ 2057.326949] ------------[ cut here ]------------ [ 2057.331574] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2057.343223] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2057.351920] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2057.407419] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2057.416563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2057.424048] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2057.429556] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2057.435042] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2057.453788] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2057.459013] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2057.466147] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2057.473281] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2057.480412] R10: 0000000000000001 R11: ffffc9000030b8d8 R12: ffff8881d1009118 [ 2057.487544] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2057.494677] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2057.502764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2057.508511] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2057.515644] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2057.522774] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2057.529908] Call Trace: [ 2057.532360] [ 2057.534458] ? __warn+0x85/0x180 [ 2057.537692] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.542571] ? report_bug+0xfc/0x1d0 [ 2057.546149] ? prb_read_valid+0x1b/0x20 [ 2057.549992] ? handle_bug+0x44/0x70 [ 2057.553481] ? exc_invalid_op+0x17/0x60 [ 2057.557327] ? asm_exc_invalid_op+0x1a/0x20 [ 2057.561520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.566398] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.571275] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2057.576170] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2057.580823] ceph_fill_trace+0x143/0xf30 [ceph] [ 2057.585362] ? __mutex_lock+0x97/0xce0 [ 2057.589112] ? __mutex_lock+0xaec/0xce0 [ 2057.592953] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2057.597501] ? lock_downgrade+0xfd/0x250 [ 2057.601441] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2057.605831] ceph_con_process_message+0x74/0x150 [libceph] [ 2057.611331] ? calc_signature+0xdb/0x100 [libceph] [ 2057.616141] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2057.621474] ? process_scheduled_works+0x22a/0x510 [ 2057.626273] ? process_scheduled_works+0x22a/0x510 [ 2057.631069] ceph_con_workfn+0x344/0x750 [libceph] [ 2057.635873] ? process_scheduled_works+0x22a/0x510 [ 2057.640661] process_scheduled_works+0x2a4/0x510 [ 2057.645293] worker_thread+0x188/0x340 [ 2057.649045] ? rescuer_thread+0x310/0x310 [ 2057.653056] kthread+0xf5/0x130 [ 2057.656202] ? kthread_complete_and_exit+0x20/0x20 [ 2057.660997] ret_from_fork+0x31/0x40 [ 2057.664580] ? kthread_complete_and_exit+0x20/0x20 [ 2057.669377] ret_from_fork_asm+0x11/0x20 [ 2057.673318] [ 2057.675510] irq event stamp: 8955507 [ 2057.679090] hardirqs last enabled at (8955523): [] console_unlock+0x106/0x120 [ 2057.687869] hardirqs last disabled at (8955538): [] console_unlock+0xeb/0x120 [ 2057.696564] softirqs last enabled at (8955552): [] __do_softirq+0x2be/0x371 [ 2057.705170] softirqs last disabled at (8955563): [] irq_exit_rcu+0xb0/0xd0 [ 2057.713603] ---[ end trace 0000000000000000 ]--- [ 2057.718223] ------------[ cut here ]------------ [ 2057.722855] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2057.734507] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2057.743470] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2057.798958] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2057.807921] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2057.815426] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2057.820923] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2057.839677] RSP: 0018:ffffc90001fcfbe0 EFLAGS: 00010286 [ 2057.844912] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2057.852053] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2057.859193] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2057.866320] R10: 0000000000000001 R11: ffffc90001fcfa70 R12: ffff8881e4da1118 [ 2057.873453] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2057.880586] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2057.888671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2057.894416] CR2: 00007fb38ed7b000 CR3: 000000011aeae001 CR4: 00000000003706f0 [ 2057.901541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2057.908673] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2057.915797] Call Trace: [ 2057.918243] [ 2057.920350] ? __warn+0x85/0x180 [ 2057.923587] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.928473] ? report_bug+0xfc/0x1d0 [ 2057.932047] ? prb_read_valid+0x1b/0x20 [ 2057.935890] ? handle_bug+0x44/0x70 [ 2057.939381] ? exc_invalid_op+0x17/0x60 [ 2057.943221] ? asm_exc_invalid_op+0x1a/0x20 [ 2057.947412] ? __list_add_valid_or_report+0x7d/0xa0 [ 2057.952288] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2057.957177] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2057.961802] ? _raw_spin_unlock+0x23/0x40 [ 2057.965820] __ceph_get_caps+0xf8/0x760 [ceph] [ 2057.970280] ? __vfs_getxattr+0x8d/0xb0 [ 2057.974119] ? dentry_needs_remove_privs+0x31/0x50 [ 2057.978911] ? __file_remove_privs+0x89/0x110 [ 2057.983270] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2057.987985] ? lock_release+0xc5/0x260 [ 2057.991745] ? ceph_write_iter+0x260/0x770 [ceph] [ 2057.996462] ceph_write_iter+0x260/0x770 [ceph] [ 2058.001006] ? lock_is_held_type+0xaa/0x120 [ 2058.005190] vfs_write+0x324/0x4f0 [ 2058.008598] ksys_pwrite64+0x65/0xa0 [ 2058.012172] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2058.017314] do_syscall_64+0x38/0x80 [ 2058.020894] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2058.025945] RIP: 0033:0x7f48f0eeb93b [ 2058.029524] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2058.048270] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2058.055838] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2058.062971] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000012 [ 2058.070102] RBP: 00000000000029b0 R08: 0000000000000001 R09: 0000000002074010 [ 2058.077235] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183d678 [ 2058.084367] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2058.091507] [ 2058.093692] irq event stamp: 12748975 [ 2058.097349] hardirqs last enabled at (12748985): [] console_unlock+0x106/0x120 [ 2058.106216] hardirqs last disabled at (12748994): [] console_unlock+0xeb/0x120 [ 2058.114996] softirqs last enabled at (12748922): [] __do_softirq+0x2be/0x371 [ 2058.123691] softirqs last disabled at (12749013): [] irq_exit_rcu+0xb0/0xd0 [ 2058.132210] ---[ end trace 0000000000000000 ]--- [ 2058.136830] ------------[ cut here ]------------ [ 2058.141527] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2058.153572] WARNING: CPU: 1 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2058.162567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2058.218261] CPU: 1 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2058.227290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2058.234927] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2058.240447] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2058.259317] RSP: 0018:ffffc90001fe7aa8 EFLAGS: 00010282 [ 2058.264571] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2058.271760] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2058.278923] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2058.286098] R10: 0000000000000001 R11: ffffc90001fe7938 R12: ffff8881d1009118 [ 2058.293253] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2058.300401] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2058.308505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2058.314268] CR2: 00007f094f8418a0 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2058.321415] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2058.328573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2058.335739] Call Trace: [ 2058.338218] [ 2058.340342] ? __warn+0x85/0x180 [ 2058.343601] ? __list_add_valid_or_report+0x7d/0xa0 [ 2058.348497] ? report_bug+0xfc/0x1d0 [ 2058.352092] ? prb_read_valid+0x1b/0x20 [ 2058.355952] ? handle_bug+0x44/0x70 [ 2058.359459] ? exc_invalid_op+0x17/0x60 [ 2058.363311] ? asm_exc_invalid_op+0x1a/0x20 [ 2058.367520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2058.372418] ? __list_add_valid_or_report+0x7d/0xa0 [ 2058.377310] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2058.382221] ceph_open+0x16b/0x440 [ceph] [ 2058.386269] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2058.391438] do_dentry_open+0x172/0x530 [ 2058.395292] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2058.399958] lookup_open+0x37b/0x630 [ 2058.403562] ? verify_cpu+0x20/0x100 [ 2058.407158] ? verify_cpu+0x20/0x100 [ 2058.410749] path_openat+0x25a/0x9d0 [ 2058.414360] do_filp_open+0xb8/0x120 [ 2058.417959] ? lock_release+0xc5/0x260 [ 2058.421729] ? _raw_spin_unlock+0x23/0x40 [ 2058.425750] ? alloc_fd+0x13f/0x200 [ 2058.429261] do_sys_openat2+0x249/0x290 [ 2058.433129] do_sys_open+0x4f/0x80 [ 2058.436558] do_syscall_64+0x38/0x80 [ 2058.440153] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2058.445222] RIP: 0033:0x7f48f0eed2f2 [ 2058.448819] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2058.467589] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2058.475173] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2058.482327] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2058.489476] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000003 [ 2058.496623] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2058.503776] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a08 [ 2058.510939] [ 2058.513151] irq event stamp: 12759215 [ 2058.516835] hardirqs last enabled at (12759283): [] console_unlock+0x106/0x120 [ 2058.525717] hardirqs last disabled at (12759348): [] console_unlock+0xeb/0x120 [ 2058.534515] softirqs last enabled at (12759416): [] __do_softirq+0x2be/0x371 [ 2058.543227] softirqs last disabled at (12759427): [] irq_exit_rcu+0xb0/0xd0 [ 2058.551761] ---[ end trace 0000000000000000 ]--- [ 2058.556401] ------------[ cut here ]------------ [ 2058.561020] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2058.572665] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2058.581620] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2058.637093] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2058.646056] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2058.653537] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2058.659023] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2058.677767] RSP: 0018:ffffc90001fcfd18 EFLAGS: 00010286 [ 2058.682995] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2058.690127] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2058.697260] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2058.704395] R10: 0000000000000001 R11: ffffc90001fcfba8 R12: ffff8881e4da1118 [ 2058.711525] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2058.718649] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2058.726735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2058.732480] CR2: 00007fb38ed90000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2058.739620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2058.746749] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2058.753891] Call Trace: [ 2058.756351] [ 2058.758461] ? __warn+0x85/0x180 [ 2058.761690] ? __list_add_valid_or_report+0x7d/0xa0 [ 2058.766569] ? report_bug+0xfc/0x1d0 [ 2058.770150] ? prb_read_valid+0x1b/0x20 [ 2058.773990] ? handle_bug+0x44/0x70 [ 2058.777489] ? exc_invalid_op+0x17/0x60 [ 2058.781329] ? asm_exc_invalid_op+0x1a/0x20 [ 2058.785520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2058.790395] ? __list_add_valid_or_report+0x7d/0xa0 [ 2058.795278] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2058.800172] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2058.805322] ceph_write_iter+0x5aa/0x770 [ceph] [ 2058.809865] ? lock_is_held_type+0xaa/0x120 [ 2058.814049] vfs_write+0x324/0x4f0 [ 2058.817457] ksys_pwrite64+0x65/0xa0 [ 2058.821030] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2058.826173] do_syscall_64+0x38/0x80 [ 2058.829750] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2058.834805] RIP: 0033:0x7f48f0eeb93b [ 2058.838382] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2058.857146] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2058.864713] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2058.871845] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000012 [ 2058.878978] RBP: 00000000000029b0 R08: 0000000000000001 R09: 0000000002074010 [ 2058.886111] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183d678 [ 2058.893245] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2058.900383] [ 2058.902568] irq event stamp: 12751723 [ 2058.906235] hardirqs last enabled at (12751733): [] console_unlock+0x106/0x120 [ 2058.915104] hardirqs last disabled at (12751748): [] console_unlock+0xeb/0x120 [ 2058.923880] softirqs last enabled at (12751746): [] __do_softirq+0x2be/0x371 [ 2058.932573] softirqs last disabled at (12751741): [] irq_exit_rcu+0xb0/0xd0 [ 2058.941092] ---[ end trace 0000000000000000 ]--- [ 2058.945704] ------------[ cut here ]------------ [ 2058.950334] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2058.961989] WARNING: CPU: 3 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2058.970948] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2059.026472] CPU: 3 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2059.035426] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2059.042904] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2059.048391] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2059.067137] RSP: 0018:ffffc90001fe7c88 EFLAGS: 00010286 [ 2059.072364] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2059.079495] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2059.086629] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2059.093762] R10: 0000000000000001 R11: ffffc90001fe7b18 R12: ffff8881d1009118 [ 2059.100895] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2059.108028] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2059.116112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2059.121858] CR2: 00007f9bd4a1f000 CR3: 0000000108c08001 CR4: 00000000003706e0 [ 2059.128993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2059.136124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2059.143256] Call Trace: [ 2059.145712] [ 2059.147819] ? __warn+0x85/0x180 [ 2059.151050] ? __list_add_valid_or_report+0x7d/0xa0 [ 2059.155930] ? report_bug+0xfc/0x1d0 [ 2059.159509] ? prb_read_valid+0x1b/0x20 [ 2059.163356] ? handle_bug+0x44/0x70 [ 2059.166850] ? exc_invalid_op+0x17/0x60 [ 2059.170688] ? asm_exc_invalid_op+0x1a/0x20 [ 2059.174883] ? __list_add_valid_or_report+0x7d/0xa0 [ 2059.179764] ? __list_add_valid_or_report+0x7d/0xa0 [ 2059.184642] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2059.189544] ceph_open+0x16b/0x440 [ceph] [ 2059.193575] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2059.198738] do_dentry_open+0x172/0x530 [ 2059.202586] path_openat+0x743/0x9d0 [ 2059.206178] do_filp_open+0xb8/0x120 [ 2059.209766] ? lock_release+0xc5/0x260 [ 2059.213524] ? _raw_spin_unlock+0x23/0x40 [ 2059.217541] ? alloc_fd+0x13f/0x200 [ 2059.221039] do_sys_openat2+0x249/0x290 [ 2059.224887] do_sys_open+0x4f/0x80 [ 2059.228300] do_syscall_64+0x38/0x80 [ 2059.231887] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2059.236938] RIP: 0033:0x7f48f0eed2f2 [ 2059.240517] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2059.259264] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2059.266829] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2059.273962] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2059.281094] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2059.288228] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2059.295360] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a09 [ 2059.302507] [ 2059.304704] irq event stamp: 12761687 [ 2059.308368] hardirqs last enabled at (12761697): [] console_unlock+0x106/0x120 [ 2059.317235] hardirqs last disabled at (12761706): [] console_unlock+0xeb/0x120 [ 2059.326015] softirqs last enabled at (12761488): [] __do_softirq+0x2be/0x371 [ 2059.334707] softirqs last disabled at (12761483): [] irq_exit_rcu+0xb0/0xd0 [ 2059.343228] ---[ end trace 0000000000000000 ]--- [ 2059.347847] ------------[ cut here ]------------ [ 2059.352473] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2059.364130] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2059.373089] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2059.428566] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2059.437521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2059.445000] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2059.450485] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2059.469231] RSP: 0018:ffffc90001fcfbe0 EFLAGS: 00010286 [ 2059.474457] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2059.481589] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2059.488724] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2059.495858] R10: 0000000000000001 R11: ffffc90001fcfa70 R12: ffff8881e4da1118 [ 2059.502988] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2059.510114] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2059.518199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2059.523944] CR2: 00007fb39af7e000 CR3: 000000011aeae001 CR4: 00000000003706f0 [ 2059.531077] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2059.538203] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2059.545336] Call Trace: [ 2059.547794] [ 2059.549895] ? __warn+0x85/0x180 [ 2059.553128] ? __list_add_valid_or_report+0x7d/0xa0 [ 2059.558006] ? report_bug+0xfc/0x1d0 [ 2059.561588] ? prb_read_valid+0x1b/0x20 [ 2059.565426] ? handle_bug+0x44/0x70 [ 2059.568917] ? exc_invalid_op+0x17/0x60 [ 2059.572757] ? asm_exc_invalid_op+0x1a/0x20 [ 2059.576948] ? __list_add_valid_or_report+0x7d/0xa0 [ 2059.581824] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2059.586721] try_get_cap_refs+0x433/0x880 [ceph] [ 2059.591349] ? _raw_spin_unlock+0x23/0x40 [ 2059.595373] __ceph_get_caps+0xf8/0x760 [ceph] [ 2059.599834] ? __vfs_getxattr+0x8d/0xb0 [ 2059.603673] ? dentry_needs_remove_privs+0x31/0x50 [ 2059.608465] ? __file_remove_privs+0x89/0x110 [ 2059.612824] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2059.617540] ? lock_release+0xc5/0x260 [ 2059.621294] ? ceph_write_iter+0x260/0x770 [ceph] [ 2059.626008] ceph_write_iter+0x260/0x770 [ceph] [ 2059.630552] ? lock_is_held_type+0xaa/0x120 [ 2059.634737] vfs_write+0x324/0x4f0 [ 2059.638144] ksys_pwrite64+0x65/0xa0 [ 2059.641719] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2059.646860] do_syscall_64+0x38/0x80 [ 2059.650439] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2059.655491] RIP: 0033:0x7f48f0eeb93b [ 2059.659071] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2059.677817] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2059.685382] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2059.692517] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2059.699649] RBP: 00000000000029b0 R08: 0000000000000800 R09: 0000000000000003 [ 2059.706779] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d678 [ 2059.713904] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207df80 [ 2059.721044] [ 2059.723229] irq event stamp: 12755087 [ 2059.726886] hardirqs last enabled at (12755097): [] console_unlock+0x106/0x120 [ 2059.735752] hardirqs last disabled at (12755106): [] console_unlock+0xeb/0x120 [ 2059.744532] softirqs last enabled at (12755044): [] __do_softirq+0x2be/0x371 [ 2059.753237] softirqs last disabled at (12755127): [] irq_exit_rcu+0xb0/0xd0 [ 2059.761771] ---[ end trace 0000000000000000 ]--- [ 2059.766398] ------------[ cut here ]------------ [ 2059.771101] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2059.782766] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2059.791757] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2059.847247] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2059.856199] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2059.863677] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2059.869163] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2059.887907] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2059.893135] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2059.900267] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2059.907399] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2059.914534] R10: 0000000000000001 R11: ffffc90001fe7a70 R12: ffff8881d1009118 [ 2059.921667] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2059.928799] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2059.936891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2059.942631] CR2: 00007f9bd4ba4000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2059.949763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2059.956895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2059.964021] Call Trace: [ 2059.966474] [ 2059.968581] ? __warn+0x85/0x180 [ 2059.971813] ? __list_add_valid_or_report+0x7d/0xa0 [ 2059.976693] ? report_bug+0xfc/0x1d0 [ 2059.980272] ? prb_read_valid+0x1b/0x20 [ 2059.984112] ? handle_bug+0x44/0x70 [ 2059.987603] ? exc_invalid_op+0x17/0x60 [ 2059.991443] ? asm_exc_invalid_op+0x1a/0x20 [ 2059.995634] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.000510] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2060.005408] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2060.010035] ? _raw_spin_unlock+0x23/0x40 [ 2060.014050] __ceph_get_caps+0xf8/0x760 [ceph] [ 2060.018502] ? __vfs_getxattr+0x8d/0xb0 [ 2060.022343] ? dentry_needs_remove_privs+0x31/0x50 [ 2060.027133] ? __file_remove_privs+0x89/0x110 [ 2060.031492] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2060.036207] ? lock_release+0xc5/0x260 [ 2060.039962] ? ceph_write_iter+0x260/0x770 [ceph] [ 2060.044675] ceph_write_iter+0x260/0x770 [ceph] [ 2060.049221] ? lock_is_held_type+0xaa/0x120 [ 2060.053403] vfs_write+0x324/0x4f0 [ 2060.056813] ksys_pwrite64+0x65/0xa0 [ 2060.060387] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2060.065528] do_syscall_64+0x38/0x80 [ 2060.069106] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2060.074158] RIP: 0033:0x7f48f0eeb93b [ 2060.077737] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2060.096484] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2060.104050] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2060.111186] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000014 [ 2060.118317] RBP: 0000000000002a09 R08: 0000000000000001 R09: 0000000002074010 [ 2060.125450] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2060.132583] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2060.139723] [ 2060.141916] irq event stamp: 12764487 [ 2060.145584] hardirqs last enabled at (12764497): [] console_unlock+0x106/0x120 [ 2060.154447] hardirqs last disabled at (12764506): [] console_unlock+0xeb/0x120 [ 2060.163227] softirqs last enabled at (12764434): [] __do_softirq+0x2be/0x371 [ 2060.171921] softirqs last disabled at (12764429): [] irq_exit_rcu+0xb0/0xd0 [ 2060.180441] ---[ end trace 0000000000000000 ]--- [ 2060.185059] ------------[ cut here ]------------ [ 2060.189679] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2060.201332] WARNING: CPU: 3 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2060.210288] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2060.265755] CPU: 3 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2060.274710] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2060.282189] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2060.287674] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2060.306421] RSP: 0018:ffffc90001fcfd18 EFLAGS: 00010286 [ 2060.311646] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2060.318770] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2060.325896] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2060.333026] R10: 0000000000000001 R11: ffffc90001fcfba8 R12: ffff8881e4da1118 [ 2060.340152] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2060.347284] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2060.355370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2060.361114] CR2: 00007f9bd4bb0018 CR3: 000000011aeae002 CR4: 00000000003706e0 [ 2060.368241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2060.375379] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2060.382513] Call Trace: [ 2060.384960] [ 2060.387066] ? __warn+0x85/0x180 [ 2060.390300] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.395178] ? report_bug+0xfc/0x1d0 [ 2060.398758] ? prb_read_valid+0x1b/0x20 [ 2060.402598] ? handle_bug+0x44/0x70 [ 2060.406093] ? exc_invalid_op+0x17/0x60 [ 2060.409928] ? asm_exc_invalid_op+0x1a/0x20 [ 2060.414119] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.418994] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.423874] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2060.428765] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2060.433916] ceph_write_iter+0x5aa/0x770 [ceph] [ 2060.438465] ? lock_is_held_type+0xaa/0x120 [ 2060.442649] vfs_write+0x324/0x4f0 [ 2060.446058] ksys_pwrite64+0x65/0xa0 [ 2060.449630] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2060.454773] do_syscall_64+0x38/0x80 [ 2060.458351] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2060.463403] RIP: 0033:0x7f48f0eeb93b [ 2060.466983] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2060.485729] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2060.493295] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb93b [ 2060.500426] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2060.507551] RBP: 00000000000029b0 R08: 0000000000000800 R09: 0000000000000003 [ 2060.514677] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d678 [ 2060.521808] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207df80 [ 2060.528949] [ 2060.531133] irq event stamp: 12757159 [ 2060.534792] hardirqs last enabled at (12757169): [] console_unlock+0x106/0x120 [ 2060.543656] hardirqs last disabled at (12757178): [] console_unlock+0xeb/0x120 [ 2060.552436] softirqs last enabled at (12756930): [] __do_softirq+0x2be/0x371 [ 2060.561129] softirqs last disabled at (12756923): [] irq_exit_rcu+0xb0/0xd0 [ 2060.569648] ---[ end trace 0000000000000000 ]--- [ 2060.574260] ------------[ cut here ]------------ [ 2060.578905] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2060.590559] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2060.599523] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2060.655049] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2060.663999] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2060.671479] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2060.676963] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2060.695713] RSP: 0018:ffffc90001fe7d18 EFLAGS: 00010286 [ 2060.700938] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2060.708070] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2060.715203] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffdfff [ 2060.722336] R10: 0000000000000001 R11: ffffc90001fe7ba8 R12: ffff8881d1009118 [ 2060.729470] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2060.736602] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2060.744688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2060.750435] CR2: 00007f9bd4b7d000 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2060.757565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2060.764703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2060.771833] Call Trace: [ 2060.774285] [ 2060.776392] ? __warn+0x85/0x180 [ 2060.779625] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.784509] ? report_bug+0xfc/0x1d0 [ 2060.788094] ? prb_read_valid+0x1b/0x20 [ 2060.791944] ? handle_bug+0x44/0x70 [ 2060.795444] ? exc_invalid_op+0x17/0x60 [ 2060.799297] ? asm_exc_invalid_op+0x1a/0x20 [ 2060.803495] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.808386] ? __list_add_valid_or_report+0x7d/0xa0 [ 2060.813269] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2060.818170] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2060.823337] ceph_write_iter+0x5aa/0x770 [ceph] [ 2060.827902] ? lock_is_held_type+0xaa/0x120 [ 2060.832098] vfs_write+0x324/0x4f0 [ 2060.835517] ksys_pwrite64+0x65/0xa0 [ 2060.839095] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2060.844238] do_syscall_64+0x38/0x80 [ 2060.847824] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2060.852874] RIP: 0033:0x7f48f0eeb93b [ 2060.856455] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2060.875207] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2060.882775] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2060.889908] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000014 [ 2060.897041] RBP: 0000000000002a09 R08: 0000000000000001 R09: 0000000002074010 [ 2060.904174] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2060.911311] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2060.918454] [ 2060.920649] irq event stamp: 12766465 [ 2060.924317] hardirqs last enabled at (12766475): [] console_unlock+0x106/0x120 [ 2060.933183] hardirqs last disabled at (12766484): [] console_unlock+0xeb/0x120 [ 2060.941963] softirqs last enabled at (12766500): [] __do_softirq+0x2be/0x371 [ 2060.950653] softirqs last disabled at (12766493): [] irq_exit_rcu+0xb0/0xd0 [ 2060.959186] ---[ end trace 0000000000000000 ]--- [ 2060.963804] ------------[ cut here ]------------ [ 2060.968421] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2060.980068] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2060.989024] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2061.044493] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2061.053442] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2061.060921] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2061.066410] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2061.085163] RSP: 0018:ffffc90001fffc88 EFLAGS: 00010286 [ 2061.090390] RAX: 0000000000000000 RBX: ffff8881e4f8e1d0 RCX: 0000000000000000 [ 2061.097524] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2061.104655] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe01f [ 2061.111791] R10: fffffffffffc09a8 R11: ffffc90001fffb18 R12: ffff8881e4f8e920 [ 2061.118919] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2061.126055] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2061.134140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2061.139887] CR2: 00007f9bd4ba4000 CR3: 000000017eb7a001 CR4: 00000000003706e0 [ 2061.147017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2061.154157] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2061.161296] Call Trace: [ 2061.163749] [ 2061.165851] ? __warn+0x85/0x180 [ 2061.169087] ? __list_add_valid_or_report+0x7d/0xa0 [ 2061.173974] ? report_bug+0xfc/0x1d0 [ 2061.177557] ? prb_read_valid+0x1b/0x20 [ 2061.181401] ? handle_bug+0x44/0x70 [ 2061.184893] ? exc_invalid_op+0x17/0x60 [ 2061.188733] ? asm_exc_invalid_op+0x1a/0x20 [ 2061.192927] ? __list_add_valid_or_report+0x7d/0xa0 [ 2061.197807] ? __list_add_valid_or_report+0x7d/0xa0 [ 2061.202686] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2061.207586] ceph_open+0x16b/0x440 [ceph] [ 2061.211618] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2061.216769] do_dentry_open+0x172/0x530 [ 2061.220614] path_openat+0x743/0x9d0 [ 2061.224203] do_filp_open+0xb8/0x120 [ 2061.227794] ? lock_release+0xc5/0x260 [ 2061.231548] ? _raw_spin_unlock+0x23/0x40 [ 2061.235560] ? alloc_fd+0x13f/0x200 [ 2061.239064] do_sys_openat2+0x249/0x290 [ 2061.242916] do_sys_open+0x4f/0x80 [ 2061.246326] do_syscall_64+0x38/0x80 [ 2061.249914] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2061.254971] RIP: 0033:0x7f48f0eed2f2 [ 2061.258553] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2061.277297] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2061.284866] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2061.291997] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2061.299130] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2061.306263] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2061.313400] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a0d [ 2061.320541] [ 2061.322727] irq event stamp: 12801963 [ 2061.326386] hardirqs last enabled at (12801973): [] console_unlock+0x106/0x120 [ 2061.335257] hardirqs last disabled at (12801984): [] console_unlock+0xeb/0x120 [ 2061.344029] softirqs last enabled at (12801930): [] __do_softirq+0x2be/0x371 [ 2061.352722] softirqs last disabled at (12801925): [] irq_exit_rcu+0xb0/0xd0 [ 2061.361243] ---[ end trace 0000000000000000 ]--- [ 2061.365865] ------------[ cut here ]------------ [ 2061.370482] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2061.382131] WARNING: CPU: 0 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2061.391083] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2061.446553] CPU: 0 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2061.455507] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2061.462984] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2061.468470] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2061.487216] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2061.492444] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2061.499575] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2061.506706] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe059 [ 2061.513833] R10: fffffffffffc1b70 R11: ffffc90001fe7a70 R12: ffff8881d1009118 [ 2061.520963] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2061.528096] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2061.536182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2061.541920] CR2: 00007f9bd4be5000 CR3: 0000000108c08001 CR4: 00000000003706f0 [ 2061.549053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2061.556186] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2061.563319] Call Trace: [ 2061.565771] [ 2061.567878] ? __warn+0x85/0x180 [ 2061.571110] ? __list_add_valid_or_report+0x7d/0xa0 [ 2061.575991] ? report_bug+0xfc/0x1d0 [ 2061.579570] ? prb_read_valid+0x1b/0x20 [ 2061.583410] ? handle_bug+0x44/0x70 [ 2061.586901] ? exc_invalid_op+0x17/0x60 [ 2061.590741] ? asm_exc_invalid_op+0x1a/0x20 [ 2061.594932] ? __list_add_valid_or_report+0x7d/0xa0 [ 2061.599810] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2061.604706] try_get_cap_refs+0x433/0x880 [ceph] [ 2061.609334] ? _raw_spin_unlock+0x23/0x40 [ 2061.613348] __ceph_get_caps+0xf8/0x760 [ceph] [ 2061.617801] ? __vfs_getxattr+0x8d/0xb0 [ 2061.621640] ? dentry_needs_remove_privs+0x31/0x50 [ 2061.626430] ? __file_remove_privs+0x89/0x110 [ 2061.630791] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2061.635506] ? lock_release+0xc5/0x260 [ 2061.639259] ? ceph_write_iter+0x260/0x770 [ceph] [ 2061.643975] ceph_write_iter+0x260/0x770 [ceph] [ 2061.648519] ? lock_is_held_type+0xaa/0x120 [ 2061.652703] vfs_write+0x324/0x4f0 [ 2061.656111] ksys_pwrite64+0x65/0xa0 [ 2061.659686] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2061.664826] do_syscall_64+0x38/0x80 [ 2061.668406] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2061.673457] RIP: 0033:0x7f48f0eeb93b [ 2061.677035] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2061.695798] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2061.703368] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2061.710500] RDX: 0000000000000800 RSI: 000000000207e200 RDI: 0000000000000014 [ 2061.717630] RBP: 0000000000002a09 R08: 0000000000000800 R09: 0000000000000003 [ 2061.724759] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2061.731889] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207e200 [ 2061.739029] [ 2061.741214] irq event stamp: 12768487 [ 2061.744879] hardirqs last enabled at (12768497): [] console_unlock+0x106/0x120 [ 2061.753745] hardirqs last disabled at (12768506): [] console_unlock+0xeb/0x120 [ 2061.762527] softirqs last enabled at (12768434): [] __do_softirq+0x2be/0x371 [ 2061.771219] softirqs last disabled at (12768429): [] irq_exit_rcu+0xb0/0xd0 [ 2061.779741] ---[ end trace 0000000000000000 ]--- [ 2061.784359] ------------[ cut here ]------------ [ 2061.788978] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2061.800624] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2061.809320] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2061.864804] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2061.873929] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2061.881409] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2061.886913] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2061.892397] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2061.911142] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2061.916361] RAX: 0000000000000000 RBX: ffff8881d0bea720 RCX: 0000000000000000 [ 2061.923497] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2061.930628] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe096 [ 2061.937760] R10: fffffffffffc2d18 R11: ffffc9000030b828 R12: ffff8881d0beae70 [ 2061.944892] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2061.952025] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2061.960113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2061.965859] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2061.972991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2061.980123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2061.987255] Call Trace: [ 2061.989702] [ 2061.991811] ? __warn+0x85/0x180 [ 2061.995041] ? __list_add_valid_or_report+0x7d/0xa0 [ 2061.999921] ? report_bug+0xfc/0x1d0 [ 2062.003498] ? prb_read_valid+0x1b/0x20 [ 2062.007340] ? handle_bug+0x44/0x70 [ 2062.010832] ? exc_invalid_op+0x17/0x60 [ 2062.014669] ? asm_exc_invalid_op+0x1a/0x20 [ 2062.018860] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.023735] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.028614] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2062.033506] ceph_add_cap+0x20b/0x620 [ceph] [ 2062.037792] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2062.042516] ? __d_lookup+0xd6/0x200 [ 2062.046099] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2062.051427] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2062.055889] ceph_con_process_message+0x74/0x150 [libceph] [ 2062.061388] ? calc_signature+0xdb/0x100 [libceph] [ 2062.066205] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2062.071529] ? process_scheduled_works+0x22a/0x510 [ 2062.076328] ? process_scheduled_works+0x22a/0x510 [ 2062.081118] ceph_con_workfn+0x344/0x750 [libceph] [ 2062.085923] ? process_scheduled_works+0x22a/0x510 [ 2062.090713] process_scheduled_works+0x2a4/0x510 [ 2062.095337] worker_thread+0x188/0x340 [ 2062.099087] ? rescuer_thread+0x310/0x310 [ 2062.103097] kthread+0xf5/0x130 [ 2062.106243] ? kthread_complete_and_exit+0x20/0x20 [ 2062.111037] ret_from_fork+0x31/0x40 [ 2062.114617] ? kthread_complete_and_exit+0x20/0x20 [ 2062.119411] ret_from_fork_asm+0x11/0x20 [ 2062.123343] [ 2062.125537] irq event stamp: 9180903 [ 2062.129115] hardirqs last enabled at (9180913): [] console_unlock+0x106/0x120 [ 2062.137894] hardirqs last disabled at (9180922): [] console_unlock+0xeb/0x120 [ 2062.146588] softirqs last enabled at (9180900): [] __do_softirq+0x2be/0x371 [ 2062.155196] softirqs last disabled at (9180895): [] irq_exit_rcu+0xb0/0xd0 [ 2062.163626] ---[ end trace 0000000000000000 ]--- [ 2062.168246] ------------[ cut here ]------------ [ 2062.172866] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2062.184516] WARNING: CPU: 4 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2062.193476] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2062.249000] CPU: 4 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2062.257951] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2062.265429] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2062.270917] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2062.289666] RSP: 0018:ffffc90001fe7d18 EFLAGS: 00010286 [ 2062.294897] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2062.302036] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2062.309177] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe0d1 [ 2062.316313] R10: fffffffffffc3ca0 R11: ffffc90001fe7ba8 R12: ffff8881d1009118 [ 2062.323448] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2062.330588] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2062.338672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2062.344420] CR2: 00007fb38edeb000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2062.351553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2062.358684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2062.365819] Call Trace: [ 2062.368270] [ 2062.370379] ? __warn+0x85/0x180 [ 2062.373610] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.378491] ? report_bug+0xfc/0x1d0 [ 2062.382069] ? prb_read_valid+0x1b/0x20 [ 2062.385911] ? handle_bug+0x44/0x70 [ 2062.389414] ? exc_invalid_op+0x17/0x60 [ 2062.393258] ? asm_exc_invalid_op+0x1a/0x20 [ 2062.397454] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.402342] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.407221] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2062.412121] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2062.417291] ceph_write_iter+0x5aa/0x770 [ceph] [ 2062.421850] ? lock_is_held_type+0xaa/0x120 [ 2062.426041] vfs_write+0x324/0x4f0 [ 2062.429461] ksys_pwrite64+0x65/0xa0 [ 2062.433045] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2062.438188] do_syscall_64+0x38/0x80 [ 2062.441774] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2062.446826] RIP: 0033:0x7f48f0eeb93b [ 2062.450405] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2062.469162] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2062.476740] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2062.483877] RDX: 0000000000000800 RSI: 000000000207e200 RDI: 0000000000000014 [ 2062.491010] RBP: 0000000000002a09 R08: 0000000000000800 R09: 0000000000000003 [ 2062.498142] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2062.505276] R13: 0000000000000800 R14: 0000000000000800 R15: 000000000207e200 [ 2062.512429] [ 2062.514619] irq event stamp: 12770603 [ 2062.518284] hardirqs last enabled at (12770613): [] console_unlock+0x106/0x120 [ 2062.527151] hardirqs last disabled at (12770622): [] console_unlock+0xeb/0x120 [ 2062.535933] softirqs last enabled at (12770636): [] __do_softirq+0x2be/0x371 [ 2062.544630] softirqs last disabled at (12770631): [] irq_exit_rcu+0xb0/0xd0 [ 2062.553152] ---[ end trace 0000000000000000 ]--- [ 2062.557772] ------------[ cut here ]------------ [ 2062.562387] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2062.574035] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2062.582730] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2062.638220] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2062.647347] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2062.654830] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2062.660341] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2062.665825] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2062.684571] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2062.689798] RAX: 0000000000000000 RBX: ffff88811e404e40 RCX: 0000000000000000 [ 2062.696931] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2062.704065] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe107 [ 2062.711197] R10: fffffffffffc4ba0 R11: ffffc9000030b828 R12: ffff88811e405590 [ 2062.718329] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2062.725461] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2062.733548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2062.739294] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2062.746426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2062.753550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2062.760685] Call Trace: [ 2062.763137] [ 2062.765244] ? __warn+0x85/0x180 [ 2062.768478] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.773355] ? report_bug+0xfc/0x1d0 [ 2062.776934] ? prb_read_valid+0x1b/0x20 [ 2062.780777] ? handle_bug+0x44/0x70 [ 2062.784266] ? exc_invalid_op+0x17/0x60 [ 2062.788106] ? asm_exc_invalid_op+0x1a/0x20 [ 2062.792296] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.797172] ? __list_add_valid_or_report+0x7d/0xa0 [ 2062.802051] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2062.806952] ceph_add_cap+0x20b/0x620 [ceph] [ 2062.811244] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2062.815960] ? __d_lookup+0xd6/0x200 [ 2062.819543] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2062.824871] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2062.829335] ceph_con_process_message+0x74/0x150 [libceph] [ 2062.834834] ? calc_signature+0xdb/0x100 [libceph] [ 2062.839645] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2062.844973] ? process_scheduled_works+0x22a/0x510 [ 2062.849766] ? process_scheduled_works+0x22a/0x510 [ 2062.854554] ceph_con_workfn+0x344/0x750 [libceph] [ 2062.859360] ? process_scheduled_works+0x22a/0x510 [ 2062.864150] process_scheduled_works+0x2a4/0x510 [ 2062.868774] worker_thread+0x188/0x340 [ 2062.872523] ? rescuer_thread+0x310/0x310 [ 2062.876533] kthread+0xf5/0x130 [ 2062.879681] ? kthread_complete_and_exit+0x20/0x20 [ 2062.884473] ret_from_fork+0x31/0x40 [ 2062.888053] ? kthread_complete_and_exit+0x20/0x20 [ 2062.892846] ret_from_fork_asm+0x11/0x20 [ 2062.896778] [ 2062.898964] irq event stamp: 9182783 [ 2062.902542] hardirqs last enabled at (9182793): [] console_unlock+0x106/0x120 [ 2062.911321] hardirqs last disabled at (9182802): [] console_unlock+0xeb/0x120 [ 2062.920015] softirqs last enabled at (9182780): [] __do_softirq+0x2be/0x371 [ 2062.928621] softirqs last disabled at (9182775): [] irq_exit_rcu+0xb0/0xd0 [ 2062.937053] ---[ end trace 0000000000000000 ]--- [ 2062.941671] ------------[ cut here ]------------ [ 2062.946307] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2062.957972] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2062.966934] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2063.022463] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2063.031425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2063.038901] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2063.044387] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2063.063136] RSP: 0018:ffffc90001fe7c88 EFLAGS: 00010286 [ 2063.068367] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2063.075501] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2063.082633] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe142 [ 2063.089767] R10: fffffffffffc5a90 R11: ffffc90001fe7b18 R12: ffff8881d1009118 [ 2063.096901] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2063.104033] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2063.112118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2063.117865] CR2: 00007f9bd4be9000 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2063.124996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2063.132131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2063.139262] Call Trace: [ 2063.141716] [ 2063.143822] ? __warn+0x85/0x180 [ 2063.147055] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.151936] ? report_bug+0xfc/0x1d0 [ 2063.155523] ? prb_read_valid+0x1b/0x20 [ 2063.159364] ? handle_bug+0x44/0x70 [ 2063.162857] ? exc_invalid_op+0x17/0x60 [ 2063.166695] ? asm_exc_invalid_op+0x1a/0x20 [ 2063.170898] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.175777] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.180656] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2063.185556] ceph_open+0x16b/0x440 [ceph] [ 2063.189587] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2063.194743] do_dentry_open+0x172/0x530 [ 2063.198593] path_openat+0x743/0x9d0 [ 2063.202182] do_filp_open+0xb8/0x120 [ 2063.205772] ? lock_release+0xc5/0x260 [ 2063.209530] ? _raw_spin_unlock+0x23/0x40 [ 2063.213547] ? alloc_fd+0x13f/0x200 [ 2063.217043] do_sys_openat2+0x249/0x290 [ 2063.220890] do_sys_open+0x4f/0x80 [ 2063.224303] do_syscall_64+0x38/0x80 [ 2063.227894] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2063.232950] RIP: 0033:0x7f48f0eed2f2 [ 2063.236532] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2063.255277] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2063.262845] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2063.269975] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2063.277108] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2063.284241] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2063.291375] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0a [ 2063.298522] [ 2063.300716] irq event stamp: 12773637 [ 2063.304382] hardirqs last enabled at (12773647): [] console_unlock+0x106/0x120 [ 2063.313248] hardirqs last disabled at (12773656): [] console_unlock+0xeb/0x120 [ 2063.322028] softirqs last enabled at (12773624): [] __do_softirq+0x2be/0x371 [ 2063.330722] softirqs last disabled at (12773675): [] irq_exit_rcu+0xb0/0xd0 [ 2063.339240] ---[ end trace 0000000000000000 ]--- [ 2063.343861] ------------[ cut here ]------------ [ 2063.348478] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2063.360131] WARNING: CPU: 7 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2063.369090] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2063.424561] CPU: 7 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2063.433518] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2063.440997] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2063.446484] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2063.465229] RSP: 0018:ffffc90001fcfc88 EFLAGS: 00010286 [ 2063.470455] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2063.477589] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2063.484722] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe17c [ 2063.491855] R10: fffffffffffc69f0 R11: ffffc90001fcfb18 R12: ffff8881e4da1118 [ 2063.498987] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2063.506118] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2063.514207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2063.519951] CR2: 00007f9bd4b90000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2063.527085] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2063.534218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2063.541351] Call Trace: [ 2063.543804] [ 2063.545912] ? __warn+0x85/0x180 [ 2063.549150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.554033] ? report_bug+0xfc/0x1d0 [ 2063.557620] ? prb_read_valid+0x1b/0x20 [ 2063.561463] ? handle_bug+0x44/0x70 [ 2063.564958] ? exc_invalid_op+0x17/0x60 [ 2063.568801] ? asm_exc_invalid_op+0x1a/0x20 [ 2063.573002] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.577885] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.582769] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2063.587673] ceph_open+0x16b/0x440 [ceph] [ 2063.591700] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2063.596857] do_dentry_open+0x172/0x530 [ 2063.600704] path_openat+0x743/0x9d0 [ 2063.604299] do_filp_open+0xb8/0x120 [ 2063.607885] ? lock_release+0xc5/0x260 [ 2063.611645] ? _raw_spin_unlock+0x23/0x40 [ 2063.615660] ? alloc_fd+0x13f/0x200 [ 2063.619159] do_sys_openat2+0x249/0x290 [ 2063.623006] do_sys_open+0x4f/0x80 [ 2063.626421] do_syscall_64+0x38/0x80 [ 2063.630006] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2063.635056] RIP: 0033:0x7f48f0eed2f2 [ 2063.638635] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2063.657382] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2063.664946] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2063.672080] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2063.679213] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2063.686349] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2063.693479] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b1 [ 2063.700623] [ 2063.702813] irq event stamp: 12765255 [ 2063.706480] hardirqs last enabled at (12765265): [] console_unlock+0x106/0x120 [ 2063.715345] hardirqs last disabled at (12765274): [] console_unlock+0xeb/0x120 [ 2063.724124] softirqs last enabled at (12765222): [] __do_softirq+0x2be/0x371 [ 2063.732816] softirqs last disabled at (12765217): [] irq_exit_rcu+0xb0/0xd0 [ 2063.741338] ---[ end trace 0000000000000000 ]--- [ 2063.745956] ------------[ cut here ]------------ [ 2063.750578] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2063.762230] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2063.770928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2063.826480] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2063.835607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2063.843090] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2063.848599] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2063.854086] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2063.872835] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2063.878066] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2063.885199] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2063.892331] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe1b6 [ 2063.899464] R10: fffffffffffc78b8 R11: ffffc9000030b828 R12: ffff8881e4d0b838 [ 2063.906600] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2063.913729] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2063.921817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2063.927562] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2063.934696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2063.941830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2063.948960] Call Trace: [ 2063.951415] [ 2063.953522] ? __warn+0x85/0x180 [ 2063.956752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.961635] ? report_bug+0xfc/0x1d0 [ 2063.965220] ? prb_read_valid+0x1b/0x20 [ 2063.969071] ? handle_bug+0x44/0x70 [ 2063.972573] ? exc_invalid_op+0x17/0x60 [ 2063.976419] ? asm_exc_invalid_op+0x1a/0x20 [ 2063.980614] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.985503] ? __list_add_valid_or_report+0x7d/0xa0 [ 2063.990389] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2063.995291] ceph_add_cap+0x20b/0x620 [ceph] [ 2063.999596] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2064.004344] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2064.009692] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2064.014178] ceph_con_process_message+0x74/0x150 [libceph] [ 2064.019684] ? calc_signature+0xdb/0x100 [libceph] [ 2064.024506] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2064.029849] ? process_scheduled_works+0x22a/0x510 [ 2064.034653] ? process_scheduled_works+0x22a/0x510 [ 2064.039450] ceph_con_workfn+0x344/0x750 [libceph] [ 2064.044266] ? process_scheduled_works+0x22a/0x510 [ 2064.049063] process_scheduled_works+0x2a4/0x510 [ 2064.053693] worker_thread+0x188/0x340 [ 2064.057457] ? rescuer_thread+0x310/0x310 [ 2064.061474] kthread+0xf5/0x130 [ 2064.064622] ? kthread_complete_and_exit+0x20/0x20 [ 2064.069428] ret_from_fork+0x31/0x40 [ 2064.073010] ? kthread_complete_and_exit+0x20/0x20 [ 2064.077803] ret_from_fork_asm+0x11/0x20 [ 2064.081754] [ 2064.083947] irq event stamp: 9185607 [ 2064.087526] hardirqs last enabled at (9185617): [] console_unlock+0x106/0x120 [ 2064.096305] hardirqs last disabled at (9185626): [] console_unlock+0xeb/0x120 [ 2064.105003] softirqs last enabled at (9185642): [] __do_softirq+0x2be/0x371 [ 2064.113613] softirqs last disabled at (9185635): [] irq_exit_rcu+0xb0/0xd0 [ 2064.122045] ---[ end trace 0000000000000000 ]--- [ 2064.126666] ------------[ cut here ]------------ [ 2064.131286] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2064.142933] WARNING: CPU: 1 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2064.151886] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2064.207419] CPU: 1 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2064.216377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2064.223857] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2064.229353] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2064.248097] RSP: 0018:ffffc90001fe7bd0 EFLAGS: 00010282 [ 2064.253323] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2064.260457] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2064.267588] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe1f0 [ 2064.274722] R10: fffffffffffc89a8 R11: ffffc90001fe7a60 R12: ffff8881d1009118 [ 2064.281854] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2064.288989] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2064.297081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.302826] CR2: 00007f094f8418a0 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2064.309963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2064.317092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2064.324226] Call Trace: [ 2064.326679] [ 2064.328787] ? __warn+0x85/0x180 [ 2064.332019] ? __list_add_valid_or_report+0x7d/0xa0 [ 2064.336900] ? report_bug+0xfc/0x1d0 [ 2064.340486] ? prb_read_valid+0x1b/0x20 [ 2064.344329] ? handle_bug+0x44/0x70 [ 2064.347826] ? exc_invalid_op+0x17/0x60 [ 2064.351668] ? asm_exc_invalid_op+0x1a/0x20 [ 2064.355872] ? __list_add_valid_or_report+0x7d/0xa0 [ 2064.360765] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2064.365667] try_get_cap_refs+0x433/0x880 [ceph] [ 2064.370322] __ceph_get_caps+0xf8/0x760 [ceph] [ 2064.374809] ? ceph_read_iter+0x134/0x860 [ceph] [ 2064.379444] ceph_read_iter+0x134/0x860 [ceph] [ 2064.383923] ? selinux_file_permission+0x119/0x150 [ 2064.388723] ? vfs_read+0x20f/0x300 [ 2064.392217] vfs_read+0x20f/0x300 [ 2064.395551] ksys_pread64+0x65/0xa0 [ 2064.399049] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2064.404200] do_syscall_64+0x38/0x80 [ 2064.407785] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2064.412838] RIP: 0033:0x7f48f0eeb88b [ 2064.416425] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2064.435171] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2064.442745] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb88b [ 2064.449878] RDX: 0000000000000800 RSI: 000000000207e200 RDI: 0000000000000014 [ 2064.457010] RBP: 0000000000002a0a R08: 000000000000005f R09: 0000000000000003 [ 2064.464143] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000800 [ 2064.471277] R13: 0000000000000800 R14: 000000000207e200 R15: 00007f48f183dcf0 [ 2064.478423] [ 2064.480620] irq event stamp: 12778737 [ 2064.484284] hardirqs last enabled at (12778747): [] console_unlock+0x106/0x120 [ 2064.493151] hardirqs last disabled at (12778756): [] console_unlock+0xeb/0x120 [ 2064.501931] softirqs last enabled at (12778516): [] __do_softirq+0x2be/0x371 [ 2064.510624] softirqs last disabled at (12778511): [] irq_exit_rcu+0xb0/0xd0 [ 2064.519144] ---[ end trace 0000000000000000 ]--- [ 2064.523765] ------------[ cut here ]------------ [ 2064.528384] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2064.540040] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2064.548740] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2064.604225] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2064.613348] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2064.620829] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2064.626323] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2064.631808] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2064.650556] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2064.655781] RAX: 0000000000000000 RBX: ffff8881d0b5e1d0 RCX: 0000000000000000 [ 2064.662916] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2064.670049] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe228 [ 2064.677178] R10: fffffffffffc96f0 R11: ffffc9000030b828 R12: ffff8881d0b5e920 [ 2064.684313] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2064.691445] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2064.699528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.705268] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2064.712400] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2064.719524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2064.726649] Call Trace: [ 2064.729104] [ 2064.731209] ? __warn+0x85/0x180 [ 2064.734444] ? __list_add_valid_or_report+0x7d/0xa0 [ 2064.739322] ? report_bug+0xfc/0x1d0 [ 2064.742901] ? prb_read_valid+0x1b/0x20 [ 2064.746740] ? handle_bug+0x44/0x70 [ 2064.750231] ? exc_invalid_op+0x17/0x60 [ 2064.754071] ? asm_exc_invalid_op+0x1a/0x20 [ 2064.758262] ? __list_add_valid_or_report+0x7d/0xa0 [ 2064.763140] ? __list_add_valid_or_report+0x7d/0xa0 [ 2064.768017] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2064.772907] ceph_add_cap+0x20b/0x620 [ceph] [ 2064.777191] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2064.781907] ? __d_lookup+0xd6/0x200 [ 2064.785484] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2064.790812] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2064.795274] ceph_con_process_message+0x74/0x150 [libceph] [ 2064.800771] ? calc_signature+0xdb/0x100 [libceph] [ 2064.805580] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2064.810904] ? process_scheduled_works+0x22a/0x510 [ 2064.815695] ? process_scheduled_works+0x22a/0x510 [ 2064.820485] ceph_con_workfn+0x344/0x750 [libceph] [ 2064.825290] ? process_scheduled_works+0x22a/0x510 [ 2064.830079] process_scheduled_works+0x2a4/0x510 [ 2064.834706] worker_thread+0x188/0x340 [ 2064.838464] ? rescuer_thread+0x310/0x310 [ 2064.842473] kthread+0xf5/0x130 [ 2064.845620] ? kthread_complete_and_exit+0x20/0x20 [ 2064.850413] ret_from_fork+0x31/0x40 [ 2064.853992] ? kthread_complete_and_exit+0x20/0x20 [ 2064.858784] ret_from_fork_asm+0x11/0x20 [ 2064.862719] [ 2064.864913] irq event stamp: 9187709 [ 2064.868490] hardirqs last enabled at (9187719): [] console_unlock+0x106/0x120 [ 2064.877270] hardirqs last disabled at (9187728): [] console_unlock+0xeb/0x120 [ 2064.885966] softirqs last enabled at (9187744): [] __do_softirq+0x2be/0x371 [ 2064.894571] softirqs last disabled at (9187755): [] irq_exit_rcu+0xb0/0xd0 [ 2064.903003] ---[ end trace 0000000000000000 ]--- [ 2064.907624] ------------[ cut here ]------------ [ 2064.912240] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2064.923889] WARNING: CPU: 3 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2064.932849] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2064.988319] CPU: 3 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2064.997273] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2065.004752] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2065.010237] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2065.028987] RSP: 0018:ffffc90001fcfbd0 EFLAGS: 00010282 [ 2065.034210] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2065.041343] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2065.048475] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe263 [ 2065.055610] R10: fffffffffffca8a0 R11: ffffc90001fcfa60 R12: ffff8881e4da1118 [ 2065.062744] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2065.069875] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2065.077960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2065.083709] CR2: 00007f9bd4c07038 CR3: 000000011aeae002 CR4: 00000000003706e0 [ 2065.090841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2065.097974] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2065.105106] Call Trace: [ 2065.107559] [ 2065.109665] ? __warn+0x85/0x180 [ 2065.112898] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.117780] ? report_bug+0xfc/0x1d0 [ 2065.121355] ? prb_read_valid+0x1b/0x20 [ 2065.125199] ? handle_bug+0x44/0x70 [ 2065.128691] ? exc_invalid_op+0x17/0x60 [ 2065.132528] ? asm_exc_invalid_op+0x1a/0x20 [ 2065.136724] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.141605] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2065.146503] try_get_cap_refs+0x433/0x880 [ceph] [ 2065.151145] __ceph_get_caps+0xf8/0x760 [ceph] [ 2065.155628] ? ceph_read_iter+0x134/0x860 [ceph] [ 2065.160262] ceph_read_iter+0x134/0x860 [ceph] [ 2065.164734] ? selinux_file_permission+0x119/0x150 [ 2065.169530] ? vfs_read+0x20f/0x300 [ 2065.173028] vfs_read+0x20f/0x300 [ 2065.176359] ksys_pread64+0x65/0xa0 [ 2065.179850] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2065.184991] do_syscall_64+0x38/0x80 [ 2065.188578] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2065.193633] RIP: 0033:0x7f48f0eeb88b [ 2065.197217] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2065.215962] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2065.223530] RAX: ffffffffffffffda RBX: 0000000002075900 RCX: 00007f48f0eeb88b [ 2065.230662] RDX: 0000000000000800 RSI: 000000000207df80 RDI: 0000000000000012 [ 2065.237794] RBP: 00000000000029b1 R08: 000000000000005f R09: 0000000000000003 [ 2065.244928] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000800 [ 2065.252063] R13: 0000000000000800 R14: 000000000207df80 R15: 00007f48f183d678 [ 2065.259207] [ 2065.261394] irq event stamp: 12769613 [ 2065.265060] hardirqs last enabled at (12769623): [] console_unlock+0x106/0x120 [ 2065.273927] hardirqs last disabled at (12769632): [] console_unlock+0xeb/0x120 [ 2065.282711] softirqs last enabled at (12769648): [] __do_softirq+0x2be/0x371 [ 2065.291409] softirqs last disabled at (12769641): [] irq_exit_rcu+0xb0/0xd0 [ 2065.299940] ---[ end trace 0000000000000000 ]--- [ 2065.304557] ------------[ cut here ]------------ [ 2065.309173] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2065.320819] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2065.329514] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2065.384992] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2065.394117] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2065.401597] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2065.407092] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2065.412578] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2065.431323] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2065.436549] RAX: 0000000000000000 RBX: ffff8881e4d30000 RCX: 0000000000000000 [ 2065.443681] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2065.450807] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe29b [ 2065.457939] R10: fffffffffffcb748 R11: ffffc9000030b828 R12: ffff8881e4d30750 [ 2065.465071] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2065.472204] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2065.480292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2065.486037] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2065.493169] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2065.500302] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2065.507438] Call Trace: [ 2065.509889] [ 2065.511996] ? __warn+0x85/0x180 [ 2065.515228] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.520106] ? report_bug+0xfc/0x1d0 [ 2065.523686] ? prb_read_valid+0x1b/0x20 [ 2065.527528] ? handle_bug+0x44/0x70 [ 2065.531019] ? exc_invalid_op+0x17/0x60 [ 2065.534858] ? asm_exc_invalid_op+0x1a/0x20 [ 2065.539049] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.543925] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.548802] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2065.553694] ceph_add_cap+0x20b/0x620 [ceph] [ 2065.557987] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2065.562702] ? __d_lookup+0xd6/0x200 [ 2065.566287] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2065.571615] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2065.576077] ceph_con_process_message+0x74/0x150 [libceph] [ 2065.581574] ? calc_signature+0xdb/0x100 [libceph] [ 2065.586384] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2065.591707] ? process_scheduled_works+0x22a/0x510 [ 2065.596498] ? process_scheduled_works+0x22a/0x510 [ 2065.601287] ceph_con_workfn+0x344/0x750 [libceph] [ 2065.606092] ? process_scheduled_works+0x22a/0x510 [ 2065.610882] process_scheduled_works+0x2a4/0x510 [ 2065.615508] worker_thread+0x188/0x340 [ 2065.619255] ? rescuer_thread+0x310/0x310 [ 2065.623267] kthread+0xf5/0x130 [ 2065.626413] ? kthread_complete_and_exit+0x20/0x20 [ 2065.631206] ret_from_fork+0x31/0x40 [ 2065.634786] ? kthread_complete_and_exit+0x20/0x20 [ 2065.639579] ret_from_fork_asm+0x11/0x20 [ 2065.643511] [ 2065.645698] irq event stamp: 9189617 [ 2065.649278] hardirqs last enabled at (9189627): [] console_unlock+0x106/0x120 [ 2065.658055] hardirqs last disabled at (9189636): [] console_unlock+0xeb/0x120 [ 2065.666749] softirqs last enabled at (9189614): [] __do_softirq+0x2be/0x371 [ 2065.675354] softirqs last disabled at (9189609): [] irq_exit_rcu+0xb0/0xd0 [ 2065.683788] ---[ end trace 0000000000000000 ]--- [ 2065.688409] ------------[ cut here ]------------ [ 2065.693030] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2065.704679] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2065.713640] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2065.769120] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2065.778078] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2065.785556] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2065.791043] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2065.809786] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2065.815016] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2065.822147] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2065.829281] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe2d6 [ 2065.836415] R10: fffffffffffcc5e0 R11: ffffc90001fe7a70 R12: ffff8881d1009118 [ 2065.843546] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2065.850680] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2065.858766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2065.864510] CR2: 00007f9bd4c11008 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2065.871645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2065.878776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2065.885910] Call Trace: [ 2065.888360] [ 2065.890475] ? __warn+0x85/0x180 [ 2065.893708] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.898591] ? report_bug+0xfc/0x1d0 [ 2065.902168] ? prb_read_valid+0x1b/0x20 [ 2065.906011] ? handle_bug+0x44/0x70 [ 2065.909502] ? exc_invalid_op+0x17/0x60 [ 2065.913339] ? asm_exc_invalid_op+0x1a/0x20 [ 2065.917543] ? __list_add_valid_or_report+0x7d/0xa0 [ 2065.922426] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2065.927321] try_get_cap_refs+0x433/0x880 [ceph] [ 2065.931955] ? _raw_spin_unlock+0x23/0x40 [ 2065.935972] __ceph_get_caps+0xf8/0x760 [ceph] [ 2065.940432] ? __vfs_getxattr+0x8d/0xb0 [ 2065.944273] ? dentry_needs_remove_privs+0x31/0x50 [ 2065.949065] ? __file_remove_privs+0x89/0x110 [ 2065.953426] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2065.958146] ? lock_release+0xc5/0x260 [ 2065.961902] ? ceph_write_iter+0x260/0x770 [ceph] [ 2065.966613] ceph_write_iter+0x260/0x770 [ceph] [ 2065.971158] ? lock_is_held_type+0xaa/0x120 [ 2065.975342] vfs_write+0x324/0x4f0 [ 2065.978750] ksys_pwrite64+0x65/0xa0 [ 2065.982324] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2065.987467] do_syscall_64+0x38/0x80 [ 2065.991046] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2065.996098] RIP: 0033:0x7f48f0eeb93b [ 2065.999679] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2066.018423] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2066.025989] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2066.033122] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000014 [ 2066.040254] RBP: 0000000000002a0a R08: 0000000000001000 R09: 0000000000000003 [ 2066.047392] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2066.054519] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207e200 [ 2066.061659] [ 2066.063845] irq event stamp: 12783249 [ 2066.067501] hardirqs last enabled at (12783259): [] console_unlock+0x106/0x120 [ 2066.076369] hardirqs last disabled at (12783268): [] console_unlock+0xeb/0x120 [ 2066.085155] softirqs last enabled at (12783284): [] __do_softirq+0x2be/0x371 [ 2066.093855] softirqs last disabled at (12783295): [] irq_exit_rcu+0xb0/0xd0 [ 2066.102369] ---[ end trace 0000000000000000 ]--- [ 2066.106990] ------------[ cut here ]------------ [ 2066.111607] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2066.123262] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2066.131958] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2066.187437] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2066.196562] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2066.204043] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2066.209545] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2066.215030] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2066.233775] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2066.239001] RAX: 0000000000000000 RBX: ffff8881d0b5f560 RCX: 0000000000000000 [ 2066.246134] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2066.253269] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe313 [ 2066.260400] R10: fffffffffffcd458 R11: ffffc9000030b828 R12: ffff8881d0b5fcb0 [ 2066.267533] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2066.274665] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2066.282751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2066.288499] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2066.295632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2066.302764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2066.309897] Call Trace: [ 2066.312354] [ 2066.314458] ? __warn+0x85/0x180 [ 2066.317693] ? __list_add_valid_or_report+0x7d/0xa0 [ 2066.322569] ? report_bug+0xfc/0x1d0 [ 2066.326147] ? prb_read_valid+0x1b/0x20 [ 2066.329989] ? handle_bug+0x44/0x70 [ 2066.333484] ? exc_invalid_op+0x17/0x60 [ 2066.337322] ? asm_exc_invalid_op+0x1a/0x20 [ 2066.341511] ? __list_add_valid_or_report+0x7d/0xa0 [ 2066.346393] ? __list_add_valid_or_report+0x7d/0xa0 [ 2066.351274] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2066.356172] ceph_add_cap+0x20b/0x620 [ceph] [ 2066.360457] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2066.365173] ? __d_lookup+0xd6/0x200 [ 2066.368756] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2066.374084] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2066.378547] ceph_con_process_message+0x74/0x150 [libceph] [ 2066.384046] ? calc_signature+0xdb/0x100 [libceph] [ 2066.388856] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2066.394186] ? process_scheduled_works+0x22a/0x510 [ 2066.398986] ? process_scheduled_works+0x22a/0x510 [ 2066.403775] ceph_con_workfn+0x344/0x750 [libceph] [ 2066.408581] ? process_scheduled_works+0x22a/0x510 [ 2066.413369] process_scheduled_works+0x2a4/0x510 [ 2066.417997] worker_thread+0x188/0x340 [ 2066.421753] ? rescuer_thread+0x310/0x310 [ 2066.425763] kthread+0xf5/0x130 [ 2066.428910] ? kthread_complete_and_exit+0x20/0x20 [ 2066.433704] ret_from_fork+0x31/0x40 [ 2066.437282] ? kthread_complete_and_exit+0x20/0x20 [ 2066.442076] ret_from_fork_asm+0x11/0x20 [ 2066.446009] [ 2066.448204] irq event stamp: 9191567 [ 2066.451782] hardirqs last enabled at (9191577): [] console_unlock+0x106/0x120 [ 2066.460561] hardirqs last disabled at (9191592): [] console_unlock+0xeb/0x120 [ 2066.469254] softirqs last enabled at (9191590): [] __do_softirq+0x2be/0x371 [ 2066.477860] softirqs last disabled at (9191585): [] irq_exit_rcu+0xb0/0xd0 [ 2066.486293] ---[ end trace 0000000000000000 ]--- [ 2066.490914] ------------[ cut here ]------------ [ 2066.495533] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2066.507182] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2066.516136] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2066.571665] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2066.580616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2066.588101] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2066.593598] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2066.612344] RSP: 0018:ffffc90001fe7d18 EFLAGS: 00010286 [ 2066.617572] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2066.624705] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2066.631837] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe34e [ 2066.638970] R10: fffffffffffce400 R11: ffffc90001fe7ba8 R12: ffff8881d1009118 [ 2066.646104] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2066.653237] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2066.661328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2066.667076] CR2: 00007fb38ee20000 CR3: 0000000108c08005 CR4: 00000000003706e0 [ 2066.674210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2066.681342] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2066.688475] Call Trace: [ 2066.690927] [ 2066.693036] ? __warn+0x85/0x180 [ 2066.696267] ? __list_add_valid_or_report+0x7d/0xa0 [ 2066.701147] ? report_bug+0xfc/0x1d0 [ 2066.704726] ? prb_read_valid+0x1b/0x20 [ 2066.708568] ? handle_bug+0x44/0x70 [ 2066.712067] ? exc_invalid_op+0x17/0x60 [ 2066.715906] ? asm_exc_invalid_op+0x1a/0x20 [ 2066.720109] ? __list_add_valid_or_report+0x7d/0xa0 [ 2066.724990] ? __list_add_valid_or_report+0x7d/0xa0 [ 2066.729869] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2066.734770] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2066.739939] ceph_write_iter+0x5aa/0x770 [ceph] [ 2066.744503] ? lock_is_held_type+0xaa/0x120 [ 2066.748698] vfs_write+0x324/0x4f0 [ 2066.752118] ksys_pwrite64+0x65/0xa0 [ 2066.755704] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2066.760846] do_syscall_64+0x38/0x80 [ 2066.764431] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2066.769485] RIP: 0033:0x7f48f0eeb93b [ 2066.773072] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2066.791845] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2066.799421] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb93b [ 2066.806552] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000014 [ 2066.813684] RBP: 0000000000002a0a R08: 0000000000001000 R09: 0000000000000003 [ 2066.820818] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2066.827951] R13: 0000000000001000 R14: 0000000000001000 R15: 000000000207e200 [ 2066.835098] [ 2066.837294] irq event stamp: 12785099 [ 2066.840958] hardirqs last enabled at (12785109): [] console_unlock+0x106/0x120 [ 2066.849825] hardirqs last disabled at (12785118): [] console_unlock+0xeb/0x120 [ 2066.858604] softirqs last enabled at (12785020): [] __do_softirq+0x2be/0x371 [ 2066.867297] softirqs last disabled at (12785005): [] irq_exit_rcu+0xb0/0xd0 [ 2066.875817] ---[ end trace 0000000000000000 ]--- [ 2066.880439] ------------[ cut here ]------------ [ 2066.885090] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2066.896787] WARNING: CPU: 0 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2066.905760] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2066.961304] CPU: 0 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2066.970281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2066.977772] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2066.983274] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2067.002039] RSP: 0018:ffffc90001f7fc88 EFLAGS: 00010286 [ 2067.007279] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2067.014430] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2067.021581] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe384 [ 2067.028736] R10: fffffffffffcf440 R11: ffffc90001f7fb18 R12: ffff8881e4d0b838 [ 2067.035880] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2067.043031] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2067.051137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2067.056899] CR2: 00007f9bd4be5000 CR3: 0000000113532002 CR4: 00000000003706f0 [ 2067.064049] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2067.071200] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2067.078351] Call Trace: [ 2067.080822] [ 2067.082945] ? __warn+0x85/0x180 [ 2067.086195] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.091100] ? report_bug+0xfc/0x1d0 [ 2067.094702] ? prb_read_valid+0x1b/0x20 [ 2067.098550] ? handle_bug+0x44/0x70 [ 2067.102049] ? exc_invalid_op+0x17/0x60 [ 2067.105888] ? asm_exc_invalid_op+0x1a/0x20 [ 2067.110083] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.114962] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.119839] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2067.124741] ceph_open+0x16b/0x440 [ceph] [ 2067.128772] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2067.133924] do_dentry_open+0x172/0x530 [ 2067.137766] path_openat+0x743/0x9d0 [ 2067.141360] do_filp_open+0xb8/0x120 [ 2067.144947] ? lock_release+0xc5/0x260 [ 2067.148705] ? _raw_spin_unlock+0x23/0x40 [ 2067.152723] ? alloc_fd+0x13f/0x200 [ 2067.156230] do_sys_openat2+0x249/0x290 [ 2067.160078] do_sys_open+0x4f/0x80 [ 2067.163489] do_syscall_64+0x38/0x80 [ 2067.167073] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2067.172127] RIP: 0033:0x7f48f0eed2f2 [ 2067.175708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2067.194452] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2067.202018] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2067.209151] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2067.216286] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2067.223419] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2067.230550] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a17 [ 2067.237695] [ 2067.239884] irq event stamp: 12877225 [ 2067.243550] hardirqs last enabled at (12877235): [] console_unlock+0x106/0x120 [ 2067.252415] hardirqs last disabled at (12877244): [] console_unlock+0xeb/0x120 [ 2067.261196] softirqs last enabled at (12877192): [] __do_softirq+0x2be/0x371 [ 2067.269889] softirqs last disabled at (12877187): [] irq_exit_rcu+0xb0/0xd0 [ 2067.278408] ---[ end trace 0000000000000000 ]--- [ 2067.283028] ------------[ cut here ]------------ [ 2067.287643] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2067.299310] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2067.308006] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2067.363520] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2067.372640] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2067.380125] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2067.385623] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2067.391109] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2067.409874] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2067.415117] RAX: 0000000000000000 RBX: ffff8881a0d19390 RCX: 0000000000000000 [ 2067.422266] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2067.429410] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe3be [ 2067.436559] R10: fffffffffffd0610 R11: ffffc9000030b828 R12: ffff8881a0d19ae0 [ 2067.443690] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2067.450824] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2067.458925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2067.464683] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2067.471816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2067.478948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2067.486098] Call Trace: [ 2067.488569] [ 2067.490674] ? __warn+0x85/0x180 [ 2067.493909] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.498812] ? report_bug+0xfc/0x1d0 [ 2067.502409] ? prb_read_valid+0x1b/0x20 [ 2067.506268] ? handle_bug+0x44/0x70 [ 2067.509776] ? exc_invalid_op+0x17/0x60 [ 2067.513616] ? asm_exc_invalid_op+0x1a/0x20 [ 2067.517839] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.522715] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.527595] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2067.532484] ceph_add_cap+0x20b/0x620 [ceph] [ 2067.536812] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2067.541540] ? __d_lookup+0xd6/0x200 [ 2067.545138] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2067.550468] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2067.554945] ceph_con_process_message+0x74/0x150 [libceph] [ 2067.560445] ? calc_signature+0xdb/0x100 [libceph] [ 2067.565289] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2067.570613] ? process_scheduled_works+0x22a/0x510 [ 2067.575404] ? process_scheduled_works+0x22a/0x510 [ 2067.580212] ceph_con_workfn+0x344/0x750 [libceph] [ 2067.585033] ? process_scheduled_works+0x22a/0x510 [ 2067.589839] process_scheduled_works+0x2a4/0x510 [ 2067.594498] worker_thread+0x188/0x340 [ 2067.598254] ? rescuer_thread+0x310/0x310 [ 2067.602269] kthread+0xf5/0x130 [ 2067.605430] ? kthread_complete_and_exit+0x20/0x20 [ 2067.610242] ret_from_fork+0x31/0x40 [ 2067.613836] ? kthread_complete_and_exit+0x20/0x20 [ 2067.618649] ret_from_fork_asm+0x11/0x20 [ 2067.622585] [ 2067.624775] irq event stamp: 9194347 [ 2067.628354] hardirqs last enabled at (9194357): [] console_unlock+0x106/0x120 [ 2067.637151] hardirqs last disabled at (9194366): [] console_unlock+0xeb/0x120 [ 2067.645867] softirqs last enabled at (9194334): [] __do_softirq+0x2be/0x371 [ 2067.654477] softirqs last disabled at (9194329): [] irq_exit_rcu+0xb0/0xd0 [ 2067.662928] ---[ end trace 0000000000000000 ]--- [ 2067.667582] ------------[ cut here ]------------ [ 2067.672222] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2067.683864] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2067.692562] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2067.748036] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2067.757163] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2067.764643] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2067.770145] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2067.775630] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2067.794376] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2067.799602] RAX: 0000000000000000 RBX: ffff8881d13d1d58 RCX: 0000000000000000 [ 2067.806735] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2067.813868] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe3f9 [ 2067.821002] R10: fffffffffffd1568 R11: ffffc9000030b828 R12: ffff8881d13d24a8 [ 2067.828135] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2067.835267] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2067.843352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2067.849098] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2067.856233] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2067.863365] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2067.870497] Call Trace: [ 2067.872950] [ 2067.875056] ? __warn+0x85/0x180 [ 2067.878291] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.883169] ? report_bug+0xfc/0x1d0 [ 2067.886750] ? prb_read_valid+0x1b/0x20 [ 2067.890589] ? handle_bug+0x44/0x70 [ 2067.894081] ? exc_invalid_op+0x17/0x60 [ 2067.897918] ? asm_exc_invalid_op+0x1a/0x20 [ 2067.902110] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.906985] ? __list_add_valid_or_report+0x7d/0xa0 [ 2067.911868] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2067.916766] ceph_add_cap+0x20b/0x620 [ceph] [ 2067.921057] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2067.925773] ? __d_lookup+0xd6/0x200 [ 2067.929358] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2067.934692] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2067.939173] ceph_con_process_message+0x74/0x150 [libceph] [ 2067.944669] ? calc_signature+0xdb/0x100 [libceph] [ 2067.949499] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2067.954838] ? process_scheduled_works+0x22a/0x510 [ 2067.959631] ? process_scheduled_works+0x22a/0x510 [ 2067.964419] ceph_con_workfn+0x344/0x750 [libceph] [ 2067.969241] ? process_scheduled_works+0x22a/0x510 [ 2067.974049] process_scheduled_works+0x2a4/0x510 [ 2067.978691] worker_thread+0x188/0x340 [ 2067.982439] ? rescuer_thread+0x310/0x310 [ 2067.986469] kthread+0xf5/0x130 [ 2067.989633] ? kthread_complete_and_exit+0x20/0x20 [ 2067.994426] ret_from_fork+0x31/0x40 [ 2067.998022] ? kthread_complete_and_exit+0x20/0x20 [ 2068.002832] ret_from_fork_asm+0x11/0x20 [ 2068.006783] [ 2068.008977] irq event stamp: 9195411 [ 2068.012574] hardirqs last enabled at (9195427): [] console_unlock+0x106/0x120 [ 2068.021351] hardirqs last disabled at (9195436): [] console_unlock+0xeb/0x120 [ 2068.030062] softirqs last enabled at (9195424): [] __do_softirq+0x2be/0x371 [ 2068.038686] softirqs last disabled at (9195419): [] irq_exit_rcu+0xb0/0xd0 [ 2068.047119] ---[ end trace 0000000000000000 ]--- [ 2068.051897] ------------[ cut here ]------------ [ 2068.056536] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2068.068194] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2068.076893] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2068.132385] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2068.141544] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2068.149024] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2068.154536] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2068.160021] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2068.178786] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2068.184016] RAX: 0000000000000000 RBX: ffff8881d13d3ab0 RCX: 0000000000000000 [ 2068.191144] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2068.198277] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe434 [ 2068.205414] R10: fffffffffffd2830 R11: ffffc9000030b828 R12: ffff8881d13d4200 [ 2068.212561] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2068.219692] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2068.227779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2068.233541] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2068.240675] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2068.247808] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2068.254950] Call Trace: [ 2068.257403] [ 2068.259510] ? __warn+0x85/0x180 [ 2068.262766] ? __list_add_valid_or_report+0x7d/0xa0 [ 2068.267648] ? report_bug+0xfc/0x1d0 [ 2068.271226] ? prb_read_valid+0x1b/0x20 [ 2068.275068] ? handle_bug+0x44/0x70 [ 2068.278558] ? exc_invalid_op+0x17/0x60 [ 2068.282399] ? asm_exc_invalid_op+0x1a/0x20 [ 2068.286607] ? __list_add_valid_or_report+0x7d/0xa0 [ 2068.291489] ? __list_add_valid_or_report+0x7d/0xa0 [ 2068.296367] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2068.301278] ceph_add_cap+0x20b/0x620 [ceph] [ 2068.305578] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2068.310294] ? __d_lookup+0xd6/0x200 [ 2068.313879] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2068.319205] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2068.323669] ceph_con_process_message+0x74/0x150 [libceph] [ 2068.329164] ? calc_signature+0xdb/0x100 [libceph] [ 2068.334013] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2068.339361] ? process_scheduled_works+0x22a/0x510 [ 2068.344159] ? process_scheduled_works+0x22a/0x510 [ 2068.348950] ceph_con_workfn+0x344/0x750 [libceph] [ 2068.353755] ? process_scheduled_works+0x22a/0x510 [ 2068.358546] process_scheduled_works+0x2a4/0x510 [ 2068.363169] worker_thread+0x188/0x340 [ 2068.366925] ? rescuer_thread+0x310/0x310 [ 2068.370938] kthread+0xf5/0x130 [ 2068.374083] ? kthread_complete_and_exit+0x20/0x20 [ 2068.378877] ret_from_fork+0x31/0x40 [ 2068.382456] ? kthread_complete_and_exit+0x20/0x20 [ 2068.387266] ret_from_fork_asm+0x11/0x20 [ 2068.391198] [ 2068.393384] irq event stamp: 9196459 [ 2068.396966] hardirqs last enabled at (9196475): [] console_unlock+0x106/0x120 [ 2068.405742] hardirqs last disabled at (9196484): [] console_unlock+0xeb/0x120 [ 2068.414436] softirqs last enabled at (9196472): [] __do_softirq+0x2be/0x371 [ 2068.423041] softirqs last disabled at (9196467): [] irq_exit_rcu+0xb0/0xd0 [ 2068.431475] ---[ end trace 0000000000000000 ]--- [ 2068.436126] ------------[ cut here ]------------ [ 2068.440746] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2068.452413] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2068.461109] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2068.516604] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2068.525728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2068.533207] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2068.538718] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2068.544205] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2068.562967] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2068.568211] RAX: 0000000000000000 RBX: ffff8881d13d4478 RCX: 0000000000000000 [ 2068.575361] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2068.582511] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe46f [ 2068.589653] R10: fffffffffffd3ad0 R11: ffffc9000030b828 R12: ffff8881d13d4bc8 [ 2068.596786] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2068.603918] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2068.612013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2068.617778] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2068.624909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2068.632042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2068.639184] Call Trace: [ 2068.641655] [ 2068.643762] ? __warn+0x85/0x180 [ 2068.646995] ? __list_add_valid_or_report+0x7d/0xa0 [ 2068.651889] ? report_bug+0xfc/0x1d0 [ 2068.655488] ? prb_read_valid+0x1b/0x20 [ 2068.659345] ? handle_bug+0x44/0x70 [ 2068.662853] ? exc_invalid_op+0x17/0x60 [ 2068.666712] ? asm_exc_invalid_op+0x1a/0x20 [ 2068.670902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2068.675776] ? __list_add_valid_or_report+0x7d/0xa0 [ 2068.680656] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2068.685556] ceph_add_cap+0x20b/0x620 [ceph] [ 2068.689866] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2068.694589] ? __d_lookup+0xd6/0x200 [ 2068.698184] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2068.703535] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2068.707997] ceph_con_process_message+0x74/0x150 [libceph] [ 2068.713494] ? calc_signature+0xdb/0x100 [libceph] [ 2068.718323] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2068.723648] ? process_scheduled_works+0x22a/0x510 [ 2068.728465] ? process_scheduled_works+0x22a/0x510 [ 2068.733254] ceph_con_workfn+0x344/0x750 [libceph] [ 2068.738075] ? process_scheduled_works+0x22a/0x510 [ 2068.742883] process_scheduled_works+0x2a4/0x510 [ 2068.747548] worker_thread+0x188/0x340 [ 2068.751299] ? rescuer_thread+0x310/0x310 [ 2068.755311] kthread+0xf5/0x130 [ 2068.758475] ? kthread_complete_and_exit+0x20/0x20 [ 2068.763287] ret_from_fork+0x31/0x40 [ 2068.766882] ? kthread_complete_and_exit+0x20/0x20 [ 2068.771693] ret_from_fork_asm+0x11/0x20 [ 2068.775625] [ 2068.777819] irq event stamp: 9197531 [ 2068.781398] hardirqs last enabled at (9197547): [] console_unlock+0x106/0x120 [ 2068.790194] hardirqs last disabled at (9197556): [] console_unlock+0xeb/0x120 [ 2068.798906] softirqs last enabled at (9197544): [] __do_softirq+0x2be/0x371 [ 2068.807547] softirqs last disabled at (9197539): [] irq_exit_rcu+0xb0/0xd0 [ 2068.815979] ---[ end trace 0000000000000000 ]--- [ 2068.820627] ------------[ cut here ]------------ [ 2068.825268] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2068.836938] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2068.845632] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2068.901116] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2068.910257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2068.917754] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2068.923249] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2068.928752] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2068.947498] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2068.952742] RAX: 0000000000000000 RBX: ffff8881d1162720 RCX: 0000000000000000 [ 2068.959896] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2068.967023] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe4aa [ 2068.974175] R10: fffffffffffd4de8 R11: ffffc9000030b828 R12: ffff8881d1162e70 [ 2068.981323] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2068.988465] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2068.996569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2069.002315] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2069.009466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2069.016599] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2069.023732] Call Trace: [ 2069.026184] [ 2069.028310] ? __warn+0x85/0x180 [ 2069.031545] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.036421] ? report_bug+0xfc/0x1d0 [ 2069.040019] ? prb_read_valid+0x1b/0x20 [ 2069.043876] ? handle_bug+0x44/0x70 [ 2069.047384] ? exc_invalid_op+0x17/0x60 [ 2069.051240] ? asm_exc_invalid_op+0x1a/0x20 [ 2069.055466] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.060341] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.065236] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2069.070146] ceph_add_cap+0x20b/0x620 [ceph] [ 2069.074473] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2069.079216] ? __d_lookup+0xd6/0x200 [ 2069.082801] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2069.088145] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2069.092605] ceph_con_process_message+0x74/0x150 [libceph] [ 2069.098102] ? calc_signature+0xdb/0x100 [libceph] [ 2069.102951] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2069.108297] ? process_scheduled_works+0x22a/0x510 [ 2069.113088] ? process_scheduled_works+0x22a/0x510 [ 2069.117897] ceph_con_workfn+0x344/0x750 [libceph] [ 2069.122717] ? process_scheduled_works+0x22a/0x510 [ 2069.127508] process_scheduled_works+0x2a4/0x510 [ 2069.132170] worker_thread+0x188/0x340 [ 2069.135924] ? rescuer_thread+0x310/0x310 [ 2069.139954] kthread+0xf5/0x130 [ 2069.143117] ? kthread_complete_and_exit+0x20/0x20 [ 2069.147948] ret_from_fork+0x31/0x40 [ 2069.151524] ? kthread_complete_and_exit+0x20/0x20 [ 2069.156335] ret_from_fork_asm+0x11/0x20 [ 2069.160303] [ 2069.162494] irq event stamp: 9198599 [ 2069.166086] hardirqs last enabled at (9198615): [] console_unlock+0x106/0x120 [ 2069.174881] hardirqs last disabled at (9198624): [] console_unlock+0xeb/0x120 [ 2069.183590] softirqs last enabled at (9198612): [] __do_softirq+0x2be/0x371 [ 2069.192196] softirqs last disabled at (9198607): [] irq_exit_rcu+0xb0/0xd0 [ 2069.200646] ---[ end trace 0000000000000000 ]--- [ 2069.205288] ------------[ cut here ]------------ [ 2069.209926] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2069.221585] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2069.230303] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2069.285878] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2069.295029] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2069.302511] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2069.308041] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2069.313562] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2069.332307] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2069.337566] RAX: 0000000000000000 RBX: ffff8881c3fc3ab0 RCX: 0000000000000000 [ 2069.344705] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2069.351877] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe4e5 [ 2069.359010] R10: fffffffffffd5f78 R11: ffffc9000030b828 R12: ffff8881c3fc4200 [ 2069.366160] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2069.373311] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2069.381413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2069.387176] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2069.394327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2069.401478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2069.408611] Call Trace: [ 2069.411062] [ 2069.413187] ? __warn+0x85/0x180 [ 2069.416438] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.421352] ? report_bug+0xfc/0x1d0 [ 2069.424952] ? prb_read_valid+0x1b/0x20 [ 2069.428799] ? handle_bug+0x44/0x70 [ 2069.432289] ? exc_invalid_op+0x17/0x60 [ 2069.436147] ? asm_exc_invalid_op+0x1a/0x20 [ 2069.440377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.445283] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.450169] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2069.455103] ceph_add_cap+0x20b/0x620 [ceph] [ 2069.459420] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2069.464171] ? __d_lookup+0xd6/0x200 [ 2069.467761] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2069.473127] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2069.477608] ceph_con_process_message+0x74/0x150 [libceph] [ 2069.483110] ? calc_signature+0xdb/0x100 [libceph] [ 2069.487963] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2069.493319] ? process_scheduled_works+0x22a/0x510 [ 2069.498137] ? process_scheduled_works+0x22a/0x510 [ 2069.502952] ceph_con_workfn+0x344/0x750 [libceph] [ 2069.507763] ? process_scheduled_works+0x22a/0x510 [ 2069.512569] process_scheduled_works+0x2a4/0x510 [ 2069.517218] worker_thread+0x188/0x340 [ 2069.520995] ? rescuer_thread+0x310/0x310 [ 2069.525006] kthread+0xf5/0x130 [ 2069.528169] ? kthread_complete_and_exit+0x20/0x20 [ 2069.532998] ret_from_fork+0x31/0x40 [ 2069.536576] ? kthread_complete_and_exit+0x20/0x20 [ 2069.541389] ret_from_fork_asm+0x11/0x20 [ 2069.545354] [ 2069.547566] irq event stamp: 9199667 [ 2069.551146] hardirqs last enabled at (9199677): [] console_unlock+0x106/0x120 [ 2069.559924] hardirqs last disabled at (9199686): [] console_unlock+0xeb/0x120 [ 2069.568635] softirqs last enabled at (9199644): [] __do_softirq+0x2be/0x371 [ 2069.577240] softirqs last disabled at (9199639): [] irq_exit_rcu+0xb0/0xd0 [ 2069.585691] ---[ end trace 0000000000000000 ]--- [ 2069.590376] ------------[ cut here ]------------ [ 2069.595005] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2069.606652] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2069.615346] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2069.670851] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2069.679992] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2069.687473] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2069.692977] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2069.698463] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2069.717213] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2069.722445] RAX: 0000000000000000 RBX: ffff8881e4d36b98 RCX: 0000000000000000 [ 2069.729581] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2069.736718] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe520 [ 2069.743861] R10: fffffffffffd71d0 R11: ffffc9000030b828 R12: ffff8881e4d372e8 [ 2069.751009] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2069.758158] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2069.766262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2069.772025] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2069.779176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2069.786325] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2069.793459] Call Trace: [ 2069.795948] [ 2069.798053] ? __warn+0x85/0x180 [ 2069.801302] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.806199] ? report_bug+0xfc/0x1d0 [ 2069.809794] ? prb_read_valid+0x1b/0x20 [ 2069.813635] ? handle_bug+0x44/0x70 [ 2069.817127] ? exc_invalid_op+0x17/0x60 [ 2069.820977] ? asm_exc_invalid_op+0x1a/0x20 [ 2069.825202] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.830084] ? __list_add_valid_or_report+0x7d/0xa0 [ 2069.834981] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2069.839915] ceph_add_cap+0x20b/0x620 [ceph] [ 2069.844217] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2069.848951] ? __d_lookup+0xd6/0x200 [ 2069.852556] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2069.857887] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2069.862368] ceph_con_process_message+0x74/0x150 [libceph] [ 2069.867873] ? calc_signature+0xdb/0x100 [libceph] [ 2069.872703] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2069.878049] ? process_scheduled_works+0x22a/0x510 [ 2069.882843] ? process_scheduled_works+0x22a/0x510 [ 2069.887633] ceph_con_workfn+0x344/0x750 [libceph] [ 2069.892436] ? process_scheduled_works+0x22a/0x510 [ 2069.897243] process_scheduled_works+0x2a4/0x510 [ 2069.901904] worker_thread+0x188/0x340 [ 2069.905659] ? rescuer_thread+0x310/0x310 [ 2069.909672] kthread+0xf5/0x130 [ 2069.912818] ? kthread_complete_and_exit+0x20/0x20 [ 2069.917617] ret_from_fork+0x31/0x40 [ 2069.921199] ? kthread_complete_and_exit+0x20/0x20 [ 2069.926009] ret_from_fork_asm+0x11/0x20 [ 2069.929976] [ 2069.932162] irq event stamp: 9200731 [ 2069.935760] hardirqs last enabled at (9200747): [] console_unlock+0x106/0x120 [ 2069.944539] hardirqs last disabled at (9200756): [] console_unlock+0xeb/0x120 [ 2069.953246] softirqs last enabled at (9200744): [] __do_softirq+0x2be/0x371 [ 2069.961869] softirqs last disabled at (9200739): [] irq_exit_rcu+0xb0/0xd0 [ 2069.970313] ---[ end trace 0000000000000000 ]--- [ 2069.975114] ------------[ cut here ]------------ [ 2069.979737] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2069.991389] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2070.000085] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2070.055581] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2070.064704] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2070.072184] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2070.077687] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2070.083172] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2070.101920] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2070.107145] RAX: 0000000000000000 RBX: ffff8881e4eb0000 RCX: 0000000000000000 [ 2070.114277] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2070.121411] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe55b [ 2070.128543] R10: fffffffffffd8470 R11: ffffc9000030b828 R12: ffff8881e4eb0750 [ 2070.135668] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2070.142802] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2070.150886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2070.156632] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2070.163766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2070.170899] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2070.178030] Call Trace: [ 2070.180474] [ 2070.182572] ? __warn+0x85/0x180 [ 2070.185807] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.190688] ? report_bug+0xfc/0x1d0 [ 2070.194263] ? prb_read_valid+0x1b/0x20 [ 2070.198104] ? handle_bug+0x44/0x70 [ 2070.201596] ? exc_invalid_op+0x17/0x60 [ 2070.205436] ? asm_exc_invalid_op+0x1a/0x20 [ 2070.209629] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.214510] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.219390] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2070.224281] ceph_add_cap+0x20b/0x620 [ceph] [ 2070.228585] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2070.233307] ? __d_lookup+0xd6/0x200 [ 2070.236890] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2070.242217] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2070.246681] ceph_con_process_message+0x74/0x150 [libceph] [ 2070.252176] ? calc_signature+0xdb/0x100 [libceph] [ 2070.256981] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2070.262303] ? process_scheduled_works+0x22a/0x510 [ 2070.267095] ? process_scheduled_works+0x22a/0x510 [ 2070.271883] ceph_con_workfn+0x344/0x750 [libceph] [ 2070.276688] ? process_scheduled_works+0x22a/0x510 [ 2070.281477] process_scheduled_works+0x2a4/0x510 [ 2070.286105] worker_thread+0x188/0x340 [ 2070.289873] ? rescuer_thread+0x310/0x310 [ 2070.293881] kthread+0xf5/0x130 [ 2070.297029] ? kthread_complete_and_exit+0x20/0x20 [ 2070.301821] ret_from_fork+0x31/0x40 [ 2070.305398] ? kthread_complete_and_exit+0x20/0x20 [ 2070.310191] ret_from_fork_asm+0x11/0x20 [ 2070.314128] [ 2070.316318] irq event stamp: 9201773 [ 2070.319889] hardirqs last enabled at (9201783): [] console_unlock+0x106/0x120 [ 2070.328669] hardirqs last disabled at (9201792): [] console_unlock+0xeb/0x120 [ 2070.337363] softirqs last enabled at (9201760): [] __do_softirq+0x2be/0x371 [ 2070.345969] softirqs last disabled at (9201755): [] irq_exit_rcu+0xb0/0xd0 [ 2070.354401] ---[ end trace 0000000000000000 ]--- [ 2070.359114] ------------[ cut here ]------------ [ 2070.363734] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2070.375388] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2070.384089] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2070.439573] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2070.448704] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2070.456186] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2070.461700] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2070.467194] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2070.485942] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2070.491174] RAX: 0000000000000000 RBX: ffff8881d12d2720 RCX: 0000000000000000 [ 2070.498308] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2070.505440] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe596 [ 2070.512572] R10: fffffffffffd99a8 R11: ffffc9000030b828 R12: ffff8881d12d2e70 [ 2070.519706] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2070.526840] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2070.534925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2070.540672] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2070.547803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2070.554937] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2070.562068] Call Trace: [ 2070.564522] [ 2070.566628] ? __warn+0x85/0x180 [ 2070.569865] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.574749] ? report_bug+0xfc/0x1d0 [ 2070.578330] ? prb_read_valid+0x1b/0x20 [ 2070.582170] ? handle_bug+0x44/0x70 [ 2070.585661] ? exc_invalid_op+0x17/0x60 [ 2070.589500] ? asm_exc_invalid_op+0x1a/0x20 [ 2070.593695] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.598574] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.603454] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2070.608355] ceph_add_cap+0x20b/0x620 [ceph] [ 2070.612659] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2070.617385] ? __d_lookup+0xd6/0x200 [ 2070.620978] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2070.626317] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2070.630793] ceph_con_process_message+0x74/0x150 [libceph] [ 2070.636292] ? calc_signature+0xdb/0x100 [libceph] [ 2070.641118] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2070.646457] ? process_scheduled_works+0x22a/0x510 [ 2070.651258] ? process_scheduled_works+0x22a/0x510 [ 2070.656051] ceph_con_workfn+0x344/0x750 [libceph] [ 2070.660867] ? process_scheduled_works+0x22a/0x510 [ 2070.665663] process_scheduled_works+0x2a4/0x510 [ 2070.670294] worker_thread+0x188/0x340 [ 2070.674046] ? rescuer_thread+0x310/0x310 [ 2070.678059] kthread+0xf5/0x130 [ 2070.681205] ? kthread_complete_and_exit+0x20/0x20 [ 2070.686004] ret_from_fork+0x31/0x40 [ 2070.689585] ? kthread_complete_and_exit+0x20/0x20 [ 2070.694378] ret_from_fork_asm+0x11/0x20 [ 2070.698324] [ 2070.700513] irq event stamp: 9202993 [ 2070.704095] hardirqs last enabled at (9203009): [] console_unlock+0x106/0x120 [ 2070.712882] hardirqs last disabled at (9203018): [] console_unlock+0xeb/0x120 [ 2070.721583] softirqs last enabled at (9203006): [] __do_softirq+0x2be/0x371 [ 2070.730190] softirqs last disabled at (9203001): [] irq_exit_rcu+0xb0/0xd0 [ 2070.738629] ---[ end trace 0000000000000000 ]--- [ 2070.743250] ------------[ cut here ]------------ [ 2070.747867] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2070.759513] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2070.768467] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2070.823940] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2070.832888] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2070.840366] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2070.845853] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2070.864598] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2070.869830] RAX: 0000000000000000 RBX: ffff8881d1162720 RCX: 0000000000000000 [ 2070.876959] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2070.884091] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe5d1 [ 2070.891226] R10: fffffffffffdaab8 R11: ffffc90001fdfb18 R12: ffff8881d1162e70 [ 2070.898357] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2070.905489] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2070.913575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2070.919322] CR2: 00007fb38eec5048 CR3: 0000000179402003 CR4: 00000000003706e0 [ 2070.926455] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2070.933587] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2070.940711] Call Trace: [ 2070.943160] [ 2070.945264] ? __warn+0x85/0x180 [ 2070.948495] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.953374] ? report_bug+0xfc/0x1d0 [ 2070.956955] ? prb_read_valid+0x1b/0x20 [ 2070.960793] ? handle_bug+0x44/0x70 [ 2070.964285] ? exc_invalid_op+0x17/0x60 [ 2070.968124] ? asm_exc_invalid_op+0x1a/0x20 [ 2070.972315] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.977190] ? __list_add_valid_or_report+0x7d/0xa0 [ 2070.982069] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2070.986961] ceph_open+0x16b/0x440 [ceph] [ 2070.990982] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2070.996128] do_dentry_open+0x172/0x530 [ 2070.999969] path_openat+0x743/0x9d0 [ 2071.003550] do_filp_open+0xb8/0x120 [ 2071.007130] ? lock_release+0xc5/0x260 [ 2071.010881] ? _raw_spin_unlock+0x23/0x40 [ 2071.014891] ? alloc_fd+0x13f/0x200 [ 2071.018386] do_sys_openat2+0x249/0x290 [ 2071.022227] do_sys_open+0x4f/0x80 [ 2071.025631] do_syscall_64+0x38/0x80 [ 2071.029208] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2071.034263] RIP: 0033:0x7f48f0eed2f2 [ 2071.037841] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2071.056587] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2071.064153] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2071.071285] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2071.078409] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2071.085534] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2071.092666] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b2 [ 2071.099805] [ 2071.101993] irq event stamp: 12758941 [ 2071.105657] hardirqs last enabled at (12758951): [] console_unlock+0x106/0x120 [ 2071.114524] hardirqs last disabled at (12758960): [] console_unlock+0xeb/0x120 [ 2071.123303] softirqs last enabled at (12758908): [] __do_softirq+0x2be/0x371 [ 2071.131997] softirqs last disabled at (12758903): [] irq_exit_rcu+0xb0/0xd0 [ 2071.140521] ---[ end trace 0000000000000000 ]--- [ 2071.145138] ------------[ cut here ]------------ [ 2071.149758] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2071.161404] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2071.170101] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2071.225594] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2071.234720] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2071.242197] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2071.247703] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2071.253195] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2071.271943] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2071.277169] RAX: 0000000000000000 RBX: ffff8881f79bce40 RCX: 0000000000000000 [ 2071.284303] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2071.291435] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe60b [ 2071.298570] R10: fffffffffffdb9a0 R11: ffffc9000030b828 R12: ffff8881f79bd590 [ 2071.305710] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2071.312841] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2071.320929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2071.326673] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2071.333807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2071.340942] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2071.348074] Call Trace: [ 2071.350523] [ 2071.352632] ? __warn+0x85/0x180 [ 2071.355865] ? __list_add_valid_or_report+0x7d/0xa0 [ 2071.360747] ? report_bug+0xfc/0x1d0 [ 2071.364330] ? prb_read_valid+0x1b/0x20 [ 2071.368177] ? handle_bug+0x44/0x70 [ 2071.371672] ? exc_invalid_op+0x17/0x60 [ 2071.375514] ? asm_exc_invalid_op+0x1a/0x20 [ 2071.379715] ? __list_add_valid_or_report+0x7d/0xa0 [ 2071.384598] ? __list_add_valid_or_report+0x7d/0xa0 [ 2071.389482] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2071.394389] ceph_add_cap+0x20b/0x620 [ceph] [ 2071.398692] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2071.403431] ? __d_lookup+0xd6/0x200 [ 2071.407022] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2071.412370] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2071.416839] ceph_con_process_message+0x74/0x150 [libceph] [ 2071.422349] ? calc_signature+0xdb/0x100 [libceph] [ 2071.427161] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2071.432505] ? process_scheduled_works+0x22a/0x510 [ 2071.437304] ? process_scheduled_works+0x22a/0x510 [ 2071.442101] ceph_con_workfn+0x344/0x750 [libceph] [ 2071.446914] ? process_scheduled_works+0x22a/0x510 [ 2071.451711] process_scheduled_works+0x2a4/0x510 [ 2071.456338] worker_thread+0x188/0x340 [ 2071.460095] ? rescuer_thread+0x310/0x310 [ 2071.464113] kthread+0xf5/0x130 [ 2071.467260] ? kthread_complete_and_exit+0x20/0x20 [ 2071.472053] ret_from_fork+0x31/0x40 [ 2071.475641] ? kthread_complete_and_exit+0x20/0x20 [ 2071.480441] ret_from_fork_asm+0x11/0x20 [ 2071.484380] [ 2071.486568] irq event stamp: 9204875 [ 2071.490145] hardirqs last enabled at (9204887): [] console_unlock+0x106/0x120 [ 2071.498927] hardirqs last disabled at (9204896): [] console_unlock+0xeb/0x120 [ 2071.507621] softirqs last enabled at (9204862): [] __do_softirq+0x2be/0x371 [ 2071.516233] softirqs last disabled at (9204857): [] irq_exit_rcu+0xb0/0xd0 [ 2071.524669] ---[ end trace 0000000000000000 ]--- [ 2071.529288] ------------[ cut here ]------------ [ 2071.533903] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2071.545549] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2071.554506] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2071.609973] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2071.618927] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2071.626404] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2071.631891] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2071.650637] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2071.655864] RAX: 0000000000000000 RBX: ffff8881d1162720 RCX: 0000000000000000 [ 2071.662995] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2071.670124] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe646 [ 2071.677252] R10: fffffffffffdc980 R11: ffffc90001fdfb18 R12: ffff8881d1162e70 [ 2071.684386] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2071.691518] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2071.699604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2071.705343] CR2: 00007fb38eec5048 CR3: 0000000179402003 CR4: 00000000003706e0 [ 2071.712475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2071.719598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2071.726722] Call Trace: [ 2071.729166] [ 2071.731265] ? __warn+0x85/0x180 [ 2071.734499] ? __list_add_valid_or_report+0x7d/0xa0 [ 2071.739378] ? report_bug+0xfc/0x1d0 [ 2071.742957] ? prb_read_valid+0x1b/0x20 [ 2071.746797] ? handle_bug+0x44/0x70 [ 2071.750288] ? exc_invalid_op+0x17/0x60 [ 2071.754128] ? asm_exc_invalid_op+0x1a/0x20 [ 2071.758318] ? __list_add_valid_or_report+0x7d/0xa0 [ 2071.763196] ? __list_add_valid_or_report+0x7d/0xa0 [ 2071.768072] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2071.772962] ceph_open+0x16b/0x440 [ceph] [ 2071.776986] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2071.782146] do_dentry_open+0x172/0x530 [ 2071.785990] path_openat+0x743/0x9d0 [ 2071.789571] do_filp_open+0xb8/0x120 [ 2071.793150] ? lock_release+0xc5/0x260 [ 2071.796901] ? _raw_spin_unlock+0x23/0x40 [ 2071.800910] ? alloc_fd+0x13f/0x200 [ 2071.804406] do_sys_openat2+0x249/0x290 [ 2071.808245] do_sys_open+0x4f/0x80 [ 2071.811652] do_syscall_64+0x38/0x80 [ 2071.815229] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2071.820281] RIP: 0033:0x7f48f0eed2f2 [ 2071.823862] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2071.842608] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2071.850174] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2071.857305] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2071.864429] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2071.871555] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2071.878688] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b3 [ 2071.885827] [ 2071.888013] irq event stamp: 12761291 [ 2071.891679] hardirqs last enabled at (12761301): [] console_unlock+0x106/0x120 [ 2071.900546] hardirqs last disabled at (12761310): [] console_unlock+0xeb/0x120 [ 2071.909330] softirqs last enabled at (12761330): [] __do_softirq+0x2be/0x371 [ 2071.918017] softirqs last disabled at (12761319): [] irq_exit_rcu+0xb0/0xd0 [ 2071.926535] ---[ end trace 0000000000000000 ]--- [ 2071.931150] ------------[ cut here ]------------ [ 2071.935768] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2071.947412] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2071.956109] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2072.011594] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2072.020719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2072.028199] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2072.033694] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2072.039181] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2072.057925] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2072.063153] RAX: 0000000000000000 RBX: ffff8881d12d3ab0 RCX: 0000000000000000 [ 2072.070285] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2072.077417] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe680 [ 2072.084552] R10: fffffffffffdde10 R11: ffffc9000030b828 R12: ffff8881d12d4200 [ 2072.091684] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2072.098816] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2072.106905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2072.112649] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2072.119783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2072.126915] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2072.134046] Call Trace: [ 2072.136501] [ 2072.138607] ? __warn+0x85/0x180 [ 2072.141838] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.146718] ? report_bug+0xfc/0x1d0 [ 2072.150297] ? prb_read_valid+0x1b/0x20 [ 2072.154139] ? handle_bug+0x44/0x70 [ 2072.157630] ? exc_invalid_op+0x17/0x60 [ 2072.161470] ? asm_exc_invalid_op+0x1a/0x20 [ 2072.165661] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.170536] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.175415] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2072.180306] ceph_add_cap+0x20b/0x620 [ceph] [ 2072.184593] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2072.189314] ? __d_lookup+0xd6/0x200 [ 2072.192899] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2072.198226] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2072.202688] ceph_con_process_message+0x74/0x150 [libceph] [ 2072.208184] ? calc_signature+0xdb/0x100 [libceph] [ 2072.212999] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2072.218327] ? process_scheduled_works+0x22a/0x510 [ 2072.223118] ? process_scheduled_works+0x22a/0x510 [ 2072.227909] ceph_con_workfn+0x344/0x750 [libceph] [ 2072.232713] ? process_scheduled_works+0x22a/0x510 [ 2072.237503] process_scheduled_works+0x2a4/0x510 [ 2072.242128] worker_thread+0x188/0x340 [ 2072.245878] ? rescuer_thread+0x310/0x310 [ 2072.249888] kthread+0xf5/0x130 [ 2072.253035] ? kthread_complete_and_exit+0x20/0x20 [ 2072.257829] ret_from_fork+0x31/0x40 [ 2072.261406] ? kthread_complete_and_exit+0x20/0x20 [ 2072.266199] ret_from_fork_asm+0x11/0x20 [ 2072.270133] [ 2072.272326] irq event stamp: 9206725 [ 2072.275907] hardirqs last enabled at (9206735): [] console_unlock+0x106/0x120 [ 2072.284684] hardirqs last disabled at (9206744): [] console_unlock+0xeb/0x120 [ 2072.293379] softirqs last enabled at (9206712): [] __do_softirq+0x2be/0x371 [ 2072.301983] softirqs last disabled at (9206707): [] irq_exit_rcu+0xb0/0xd0 [ 2072.310417] ---[ end trace 0000000000000000 ]--- [ 2072.315037] ------------[ cut here ]------------ [ 2072.319656] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2072.331303] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2072.340259] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2072.395726] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2072.404678] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2072.412157] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2072.417644] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2072.436389] RSP: 0018:ffffc90001fdfbd0 EFLAGS: 00010282 [ 2072.441618] RAX: 0000000000000000 RBX: ffff8881d1162720 RCX: 0000000000000000 [ 2072.448749] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2072.455882] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe6bb [ 2072.463015] R10: fffffffffffdf108 R11: ffffc90001fdfa60 R12: ffff8881d1162e70 [ 2072.470149] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2072.477279] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2072.485365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2072.491111] CR2: 00007fb38eed0010 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2072.498236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2072.505371] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2072.512501] Call Trace: [ 2072.514956] [ 2072.517060] ? __warn+0x85/0x180 [ 2072.520294] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.525175] ? report_bug+0xfc/0x1d0 [ 2072.528752] ? prb_read_valid+0x1b/0x20 [ 2072.532598] ? handle_bug+0x44/0x70 [ 2072.536085] ? exc_invalid_op+0x17/0x60 [ 2072.539925] ? asm_exc_invalid_op+0x1a/0x20 [ 2072.544114] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.548989] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2072.553881] try_get_cap_refs+0x433/0x880 [ceph] [ 2072.558511] __ceph_get_caps+0xf8/0x760 [ceph] [ 2072.562978] ? ceph_read_iter+0x134/0x860 [ceph] [ 2072.567607] ceph_read_iter+0x134/0x860 [ceph] [ 2072.572065] ? selinux_file_permission+0x119/0x150 [ 2072.576855] ? vfs_read+0x20f/0x300 [ 2072.580346] vfs_read+0x20f/0x300 [ 2072.583670] ksys_pread64+0x65/0xa0 [ 2072.587159] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2072.592300] do_syscall_64+0x38/0x80 [ 2072.595879] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2072.600931] RIP: 0033:0x7f48f0eeb88b [ 2072.604509] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2072.623257] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2072.630822] RAX: ffffffffffffffda RBX: 0000000002075910 RCX: 00007f48f0eeb88b [ 2072.637955] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000013 [ 2072.645088] RBP: 00000000000029b3 R08: 0000000000000063 R09: 0000000000000003 [ 2072.652221] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2072.659355] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183dac8 [ 2072.666492] [ 2072.668677] irq event stamp: 12763341 [ 2072.672345] hardirqs last enabled at (12763351): [] console_unlock+0x106/0x120 [ 2072.681210] hardirqs last disabled at (12763360): [] console_unlock+0xeb/0x120 [ 2072.689989] softirqs last enabled at (12763318): [] __do_softirq+0x2be/0x371 [ 2072.698685] softirqs last disabled at (12763379): [] irq_exit_rcu+0xb0/0xd0 [ 2072.707202] ---[ end trace 0000000000000000 ]--- [ 2072.711822] ------------[ cut here ]------------ [ 2072.716441] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2072.728088] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2072.736783] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2072.792264] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2072.801384] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2072.808865] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2072.814367] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2072.819852] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2072.838600] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2072.843825] RAX: 0000000000000000 RBX: ffff8881d8219390 RCX: 0000000000000000 [ 2072.850958] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2072.858092] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe6f3 [ 2072.865223] R10: fffffffffffdffc0 R11: ffffc9000030b828 R12: ffff8881d8219ae0 [ 2072.872356] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2072.879483] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2072.887567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2072.893312] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2072.900447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2072.907580] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2072.914712] Call Trace: [ 2072.917163] [ 2072.919263] ? __warn+0x85/0x180 [ 2072.922495] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.927380] ? report_bug+0xfc/0x1d0 [ 2072.930963] ? prb_read_valid+0x1b/0x20 [ 2072.934804] ? handle_bug+0x44/0x70 [ 2072.938294] ? exc_invalid_op+0x17/0x60 [ 2072.942134] ? asm_exc_invalid_op+0x1a/0x20 [ 2072.946326] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.951202] ? __list_add_valid_or_report+0x7d/0xa0 [ 2072.956078] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2072.960969] ceph_add_cap+0x20b/0x620 [ceph] [ 2072.965256] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2072.969980] ? __d_lookup+0xd6/0x200 [ 2072.973563] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2072.978891] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2072.983354] ceph_con_process_message+0x74/0x150 [libceph] [ 2072.988849] ? calc_signature+0xdb/0x100 [libceph] [ 2072.993662] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2072.998993] ? process_scheduled_works+0x22a/0x510 [ 2073.003793] ? process_scheduled_works+0x22a/0x510 [ 2073.008582] ceph_con_workfn+0x344/0x750 [libceph] [ 2073.013393] ? process_scheduled_works+0x22a/0x510 [ 2073.018185] process_scheduled_works+0x2a4/0x510 [ 2073.022810] worker_thread+0x188/0x340 [ 2073.026559] ? rescuer_thread+0x310/0x310 [ 2073.030571] kthread+0xf5/0x130 [ 2073.033716] ? kthread_complete_and_exit+0x20/0x20 [ 2073.038510] ret_from_fork+0x31/0x40 [ 2073.042087] ? kthread_complete_and_exit+0x20/0x20 [ 2073.046883] ret_from_fork_asm+0x11/0x20 [ 2073.050815] [ 2073.053007] irq event stamp: 9208577 [ 2073.056587] hardirqs last enabled at (9208587): [] console_unlock+0x106/0x120 [ 2073.065367] hardirqs last disabled at (9208596): [] console_unlock+0xeb/0x120 [ 2073.074061] softirqs last enabled at (9208554): [] __do_softirq+0x2be/0x371 [ 2073.082667] softirqs last disabled at (9208549): [] irq_exit_rcu+0xb0/0xd0 [ 2073.091099] ---[ end trace 0000000000000000 ]--- [ 2073.095720] ------------[ cut here ]------------ [ 2073.100336] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2073.111974] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2073.120931] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2073.176390] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2073.185341] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2073.192811] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2073.198297] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2073.217046] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2073.222270] RAX: 0000000000000000 RBX: ffff8881d0b5e1d0 RCX: 0000000000000000 [ 2073.229394] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2073.236517] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe72e [ 2073.243642] R10: fffffffffffe12b0 R11: ffffc90001fdfb18 R12: ffff8881d0b5e920 [ 2073.250769] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2073.257900] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2073.265985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2073.271724] CR2: 00007fb38eec5048 CR3: 0000000179402003 CR4: 00000000003706e0 [ 2073.278856] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2073.285979] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2073.293104] Call Trace: [ 2073.295548] [ 2073.297646] ? __warn+0x85/0x180 [ 2073.300879] ? __list_add_valid_or_report+0x7d/0xa0 [ 2073.305759] ? report_bug+0xfc/0x1d0 [ 2073.309337] ? prb_read_valid+0x1b/0x20 [ 2073.313178] ? handle_bug+0x44/0x70 [ 2073.316670] ? exc_invalid_op+0x17/0x60 [ 2073.320509] ? asm_exc_invalid_op+0x1a/0x20 [ 2073.324700] ? __list_add_valid_or_report+0x7d/0xa0 [ 2073.329576] ? __list_add_valid_or_report+0x7d/0xa0 [ 2073.334455] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2073.339345] ceph_open+0x16b/0x440 [ceph] [ 2073.343366] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2073.348513] do_dentry_open+0x172/0x530 [ 2073.352352] path_openat+0x743/0x9d0 [ 2073.355934] do_filp_open+0xb8/0x120 [ 2073.359513] ? lock_release+0xc5/0x260 [ 2073.363264] ? _raw_spin_unlock+0x23/0x40 [ 2073.367276] ? alloc_fd+0x13f/0x200 [ 2073.370770] do_sys_openat2+0x249/0x290 [ 2073.374612] do_sys_open+0x4f/0x80 [ 2073.378014] do_syscall_64+0x38/0x80 [ 2073.381594] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2073.386647] RIP: 0033:0x7f48f0eed2f2 [ 2073.390225] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2073.408972] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2073.416537] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2073.423661] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2073.430785] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2073.437914] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2073.445046] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b4 [ 2073.452183] [ 2073.454367] irq event stamp: 12765307 [ 2073.458026] hardirqs last enabled at (12765317): [] console_unlock+0x106/0x120 [ 2073.466892] hardirqs last disabled at (12765326): [] console_unlock+0xeb/0x120 [ 2073.475673] softirqs last enabled at (12765340): [] __do_softirq+0x2be/0x371 [ 2073.484363] softirqs last disabled at (12765335): [] irq_exit_rcu+0xb0/0xd0 [ 2073.492883] ---[ end trace 0000000000000000 ]--- [ 2073.497495] ------------[ cut here ]------------ [ 2073.502114] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2073.513764] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2073.522455] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2073.577935] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2073.587060] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2073.594538] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2073.600040] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2073.605527] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2073.624272] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2073.629500] RAX: 0000000000000000 RBX: ffff8881d0aec478 RCX: 0000000000000000 [ 2073.636632] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2073.643766] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe768 [ 2073.650897] R10: fffffffffffe24d8 R11: ffffc9000030b828 R12: ffff8881d0aecbc8 [ 2073.658031] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2073.665164] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2073.673250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2073.678995] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2073.686129] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2073.693261] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2073.700393] Call Trace: [ 2073.702839] [ 2073.704945] ? __warn+0x85/0x180 [ 2073.708176] ? __list_add_valid_or_report+0x7d/0xa0 [ 2073.713056] ? report_bug+0xfc/0x1d0 [ 2073.716636] ? prb_read_valid+0x1b/0x20 [ 2073.720477] ? handle_bug+0x44/0x70 [ 2073.723969] ? exc_invalid_op+0x17/0x60 [ 2073.727808] ? asm_exc_invalid_op+0x1a/0x20 [ 2073.731999] ? __list_add_valid_or_report+0x7d/0xa0 [ 2073.736874] ? __list_add_valid_or_report+0x7d/0xa0 [ 2073.741753] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2073.746642] ceph_add_cap+0x20b/0x620 [ceph] [ 2073.750929] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2073.755653] ? __d_lookup+0xd6/0x200 [ 2073.759238] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2073.764572] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2073.769040] ceph_con_process_message+0x74/0x150 [libceph] [ 2073.774540] ? calc_signature+0xdb/0x100 [libceph] [ 2073.779356] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2073.784685] ? process_scheduled_works+0x22a/0x510 [ 2073.789484] ? process_scheduled_works+0x22a/0x510 [ 2073.794273] ceph_con_workfn+0x344/0x750 [libceph] [ 2073.799077] ? process_scheduled_works+0x22a/0x510 [ 2073.803867] process_scheduled_works+0x2a4/0x510 [ 2073.808493] worker_thread+0x188/0x340 [ 2073.812240] ? rescuer_thread+0x310/0x310 [ 2073.816251] kthread+0xf5/0x130 [ 2073.819398] ? kthread_complete_and_exit+0x20/0x20 [ 2073.824191] ret_from_fork+0x31/0x40 [ 2073.827770] ? kthread_complete_and_exit+0x20/0x20 [ 2073.832564] ret_from_fork_asm+0x11/0x20 [ 2073.836498] [ 2073.838691] irq event stamp: 9210445 [ 2073.842271] hardirqs last enabled at (9210455): [] console_unlock+0x106/0x120 [ 2073.851049] hardirqs last disabled at (9210464): [] console_unlock+0xeb/0x120 [ 2073.859743] softirqs last enabled at (9210432): [] __do_softirq+0x2be/0x371 [ 2073.868348] softirqs last disabled at (9210427): [] irq_exit_rcu+0xb0/0xd0 [ 2073.876782] ---[ end trace 0000000000000000 ]--- [ 2073.881402] ------------[ cut here ]------------ [ 2073.886020] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2073.897665] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2073.906622] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2073.962096] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2073.971051] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2073.978530] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2073.984017] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2074.002763] RSP: 0018:ffffc90001fdfbd0 EFLAGS: 00010282 [ 2074.007992] RAX: 0000000000000000 RBX: ffff8881d0b5e1d0 RCX: 0000000000000000 [ 2074.015121] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2074.022254] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe7a3 [ 2074.029386] R10: fffffffffffe34e8 R11: ffffc90001fdfa60 R12: ffff8881d0b5e920 [ 2074.036520] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2074.043653] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2074.051739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2074.057484] CR2: 00007fb38eed0010 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2074.064621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2074.071750] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2074.078882] Call Trace: [ 2074.081327] [ 2074.083425] ? __warn+0x85/0x180 [ 2074.086658] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.091537] ? report_bug+0xfc/0x1d0 [ 2074.095116] ? prb_read_valid+0x1b/0x20 [ 2074.098960] ? handle_bug+0x44/0x70 [ 2074.102449] ? exc_invalid_op+0x17/0x60 [ 2074.106288] ? asm_exc_invalid_op+0x1a/0x20 [ 2074.110478] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.115356] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2074.120246] try_get_cap_refs+0x433/0x880 [ceph] [ 2074.124884] __ceph_get_caps+0xf8/0x760 [ceph] [ 2074.129351] ? ceph_read_iter+0x134/0x860 [ceph] [ 2074.133979] ceph_read_iter+0x134/0x860 [ceph] [ 2074.138438] ? selinux_file_permission+0x119/0x150 [ 2074.143228] ? vfs_read+0x20f/0x300 [ 2074.146718] vfs_read+0x20f/0x300 [ 2074.150044] ksys_pread64+0x65/0xa0 [ 2074.153531] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2074.158671] do_syscall_64+0x38/0x80 [ 2074.162252] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2074.167303] RIP: 0033:0x7f48f0eeb88b [ 2074.170883] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2074.189628] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2074.197195] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb88b [ 2074.204328] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000014 [ 2074.211460] RBP: 00000000000029b4 R08: 0000000000000063 R09: 0000000000000003 [ 2074.218592] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2074.225726] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183dac8 [ 2074.232866] [ 2074.235051] irq event stamp: 12767175 [ 2074.238717] hardirqs last enabled at (12767185): [] console_unlock+0x106/0x120 [ 2074.247584] hardirqs last disabled at (12767194): [] console_unlock+0xeb/0x120 [ 2074.256362] softirqs last enabled at (12767152): [] __do_softirq+0x2be/0x371 [ 2074.265056] softirqs last disabled at (12767213): [] irq_exit_rcu+0xb0/0xd0 [ 2074.273575] ---[ end trace 0000000000000000 ]--- [ 2074.278195] ------------[ cut here ]------------ [ 2074.282814] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2074.294460] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2074.303156] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2074.358638] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2074.367767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2074.375246] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2074.380741] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2074.386226] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2074.404971] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2074.410199] RAX: 0000000000000000 RBX: ffff8881d0aece40 RCX: 0000000000000000 [ 2074.417332] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2074.424463] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe7db [ 2074.431597] R10: fffffffffffe45d8 R11: ffffc9000030b828 R12: ffff8881d0aed590 [ 2074.438729] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2074.445863] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2074.453949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2074.459693] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2074.466827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2074.473959] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2074.481093] Call Trace: [ 2074.483545] [ 2074.485652] ? __warn+0x85/0x180 [ 2074.488884] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.493765] ? report_bug+0xfc/0x1d0 [ 2074.497342] ? prb_read_valid+0x1b/0x20 [ 2074.501185] ? handle_bug+0x44/0x70 [ 2074.504675] ? exc_invalid_op+0x17/0x60 [ 2074.508515] ? asm_exc_invalid_op+0x1a/0x20 [ 2074.512708] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.517591] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.522468] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2074.527361] ceph_add_cap+0x20b/0x620 [ceph] [ 2074.531654] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2074.536379] ? __d_lookup+0xd6/0x200 [ 2074.539963] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2074.545291] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2074.549751] ceph_con_process_message+0x74/0x150 [libceph] [ 2074.555248] ? calc_signature+0xdb/0x100 [libceph] [ 2074.560059] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2074.565392] ? process_scheduled_works+0x22a/0x510 [ 2074.570183] ? process_scheduled_works+0x22a/0x510 [ 2074.574971] ceph_con_workfn+0x344/0x750 [libceph] [ 2074.579775] ? process_scheduled_works+0x22a/0x510 [ 2074.584566] process_scheduled_works+0x2a4/0x510 [ 2074.589191] worker_thread+0x188/0x340 [ 2074.592938] ? rescuer_thread+0x310/0x310 [ 2074.596951] kthread+0xf5/0x130 [ 2074.600097] ? kthread_complete_and_exit+0x20/0x20 [ 2074.604891] ret_from_fork+0x31/0x40 [ 2074.608469] ? kthread_complete_and_exit+0x20/0x20 [ 2074.613262] ret_from_fork_asm+0x11/0x20 [ 2074.617197] [ 2074.619391] irq event stamp: 9212277 [ 2074.622969] hardirqs last enabled at (9212287): [] console_unlock+0x106/0x120 [ 2074.631749] hardirqs last disabled at (9212296): [] console_unlock+0xeb/0x120 [ 2074.640442] softirqs last enabled at (9212264): [] __do_softirq+0x2be/0x371 [ 2074.649047] softirqs last disabled at (9212259): [] irq_exit_rcu+0xb0/0xd0 [ 2074.657481] ---[ end trace 0000000000000000 ]--- [ 2074.662100] ------------[ cut here ]------------ [ 2074.666719] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2074.678366] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2074.687321] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2074.742782] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2074.751732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2074.759210] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2074.764704] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2074.783455] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2074.788677] RAX: 0000000000000000 RBX: ffff8881d0b5f560 RCX: 0000000000000000 [ 2074.795810] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2074.802949] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe816 [ 2074.810087] R10: fffffffffffe5838 R11: ffffc90001fdfb18 R12: ffff8881d0b5fcb0 [ 2074.817226] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2074.824377] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2074.832465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2074.838208] CR2: 00007f9bd4cfb000 CR3: 0000000179402002 CR4: 00000000003706f0 [ 2074.845333] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2074.852469] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2074.859607] Call Trace: [ 2074.862051] [ 2074.864158] ? __warn+0x85/0x180 [ 2074.867392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.872271] ? report_bug+0xfc/0x1d0 [ 2074.875852] ? prb_read_valid+0x1b/0x20 [ 2074.879696] ? handle_bug+0x44/0x70 [ 2074.883190] ? exc_invalid_op+0x17/0x60 [ 2074.887030] ? asm_exc_invalid_op+0x1a/0x20 [ 2074.891220] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.896095] ? __list_add_valid_or_report+0x7d/0xa0 [ 2074.900974] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2074.905865] ceph_open+0x16b/0x440 [ceph] [ 2074.909887] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2074.915036] do_dentry_open+0x172/0x530 [ 2074.918873] path_openat+0x743/0x9d0 [ 2074.922455] do_filp_open+0xb8/0x120 [ 2074.926035] ? lock_release+0xc5/0x260 [ 2074.929785] ? _raw_spin_unlock+0x23/0x40 [ 2074.933797] ? alloc_fd+0x13f/0x200 [ 2074.937291] do_sys_openat2+0x249/0x290 [ 2074.941131] do_sys_open+0x4f/0x80 [ 2074.944534] do_syscall_64+0x38/0x80 [ 2074.948113] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2074.953166] RIP: 0033:0x7f48f0eed2f2 [ 2074.956745] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2074.975498] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2074.983065] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2074.990190] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2074.997315] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2075.004448] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2075.011580] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b5 [ 2075.018719] [ 2075.020906] irq event stamp: 12769447 [ 2075.024571] hardirqs last enabled at (12769457): [] console_unlock+0x106/0x120 [ 2075.033438] hardirqs last disabled at (12769466): [] console_unlock+0xeb/0x120 [ 2075.042218] softirqs last enabled at (12769414): [] __do_softirq+0x2be/0x371 [ 2075.050910] softirqs last disabled at (12769409): [] irq_exit_rcu+0xb0/0xd0 [ 2075.059429] ---[ end trace 0000000000000000 ]--- [ 2075.064040] ------------[ cut here ]------------ [ 2075.068661] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2075.080307] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2075.089003] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2075.144479] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2075.153606] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2075.161084] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2075.166580] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2075.172064] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2075.190813] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2075.196037] RAX: 0000000000000000 RBX: ffff8881d132eb98 RCX: 0000000000000000 [ 2075.203171] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2075.210304] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe850 [ 2075.217435] R10: fffffffffffe6818 R11: ffffc9000030b828 R12: ffff8881d132f2e8 [ 2075.224569] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2075.231701] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2075.239787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2075.245533] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2075.252667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2075.259800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2075.266931] Call Trace: [ 2075.269375] [ 2075.271474] ? __warn+0x85/0x180 [ 2075.274707] ? __list_add_valid_or_report+0x7d/0xa0 [ 2075.279587] ? report_bug+0xfc/0x1d0 [ 2075.283165] ? prb_read_valid+0x1b/0x20 [ 2075.287005] ? handle_bug+0x44/0x70 [ 2075.290497] ? exc_invalid_op+0x17/0x60 [ 2075.294337] ? asm_exc_invalid_op+0x1a/0x20 [ 2075.298527] ? __list_add_valid_or_report+0x7d/0xa0 [ 2075.303404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2075.308282] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2075.313171] ceph_add_cap+0x20b/0x620 [ceph] [ 2075.317456] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2075.322173] ? __d_lookup+0xd6/0x200 [ 2075.325759] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2075.331085] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2075.335547] ceph_con_process_message+0x74/0x150 [libceph] [ 2075.341042] ? calc_signature+0xdb/0x100 [libceph] [ 2075.345846] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2075.351179] ? process_scheduled_works+0x22a/0x510 [ 2075.355980] ? process_scheduled_works+0x22a/0x510 [ 2075.360768] ceph_con_workfn+0x344/0x750 [libceph] [ 2075.365572] ? process_scheduled_works+0x22a/0x510 [ 2075.370363] process_scheduled_works+0x2a4/0x510 [ 2075.374987] worker_thread+0x188/0x340 [ 2075.378743] ? rescuer_thread+0x310/0x310 [ 2075.382756] kthread+0xf5/0x130 [ 2075.385900] ? kthread_complete_and_exit+0x20/0x20 [ 2075.390695] ret_from_fork+0x31/0x40 [ 2075.394274] ? kthread_complete_and_exit+0x20/0x20 [ 2075.399070] ret_from_fork_asm+0x11/0x20 [ 2075.403000] [ 2075.405193] irq event stamp: 9214123 [ 2075.408777] hardirqs last enabled at (9214133): [] console_unlock+0x106/0x120 [ 2075.417562] hardirqs last disabled at (9214142): [] console_unlock+0xeb/0x120 [ 2075.426255] softirqs last enabled at (9214100): [] __do_softirq+0x2be/0x371 [ 2075.434860] softirqs last disabled at (9214095): [] irq_exit_rcu+0xb0/0xd0 [ 2075.443294] ---[ end trace 0000000000000000 ]--- [ 2075.447914] ------------[ cut here ]------------ [ 2075.452533] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2075.464181] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2075.473134] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2075.528608] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2075.537570] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2075.545051] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2075.550540] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2075.569293] RSP: 0018:ffffc90001fdfbd0 EFLAGS: 00010282 [ 2075.574520] RAX: 0000000000000000 RBX: ffff8881d0b5f560 RCX: 0000000000000000 [ 2075.581652] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2075.588786] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe88b [ 2075.595918] R10: fffffffffffe7aa8 R11: ffffc90001fdfa60 R12: ffff8881d0b5fcb0 [ 2075.603050] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2075.610183] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2075.618270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2075.624014] CR2: 00007f9bd4ce3000 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2075.631149] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2075.638279] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2075.645413] Call Trace: [ 2075.647866] [ 2075.649974] ? __warn+0x85/0x180 [ 2075.653207] ? __list_add_valid_or_report+0x7d/0xa0 [ 2075.658088] ? report_bug+0xfc/0x1d0 [ 2075.661675] ? prb_read_valid+0x1b/0x20 [ 2075.665518] ? handle_bug+0x44/0x70 [ 2075.669012] ? exc_invalid_op+0x17/0x60 [ 2075.672856] ? asm_exc_invalid_op+0x1a/0x20 [ 2075.677057] ? __list_add_valid_or_report+0x7d/0xa0 [ 2075.681939] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2075.686836] try_get_cap_refs+0x433/0x880 [ceph] [ 2075.691489] __ceph_get_caps+0xf8/0x760 [ceph] [ 2075.695972] ? ceph_read_iter+0x134/0x860 [ceph] [ 2075.700614] ceph_read_iter+0x134/0x860 [ceph] [ 2075.705074] ? selinux_file_permission+0x119/0x150 [ 2075.709874] ? vfs_read+0x20f/0x300 [ 2075.713369] vfs_read+0x20f/0x300 [ 2075.716700] ksys_pread64+0x65/0xa0 [ 2075.720190] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2075.725338] do_syscall_64+0x38/0x80 [ 2075.728918] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2075.733973] RIP: 0033:0x7f48f0eeb88b [ 2075.737550] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2075.756297] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2075.763882] RAX: ffffffffffffffda RBX: 0000000002075930 RCX: 00007f48f0eeb88b [ 2075.771014] RDX: 0000000000001000 RSI: 000000000207e040 RDI: 0000000000000015 [ 2075.778145] RBP: 00000000000029b5 R08: 0000000000000063 R09: 0000000000000003 [ 2075.785279] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2075.792415] R13: 0000000000001000 R14: 000000000207e040 R15: 00007f48f183dac8 [ 2075.799556] [ 2075.801746] irq event stamp: 12771339 [ 2075.805412] hardirqs last enabled at (12771349): [] console_unlock+0x106/0x120 [ 2075.814282] hardirqs last disabled at (12771358): [] console_unlock+0xeb/0x120 [ 2075.823066] softirqs last enabled at (12771316): [] __do_softirq+0x2be/0x371 [ 2075.831758] softirqs last disabled at (12771311): [] irq_exit_rcu+0xb0/0xd0 [ 2075.840280] ---[ end trace 0000000000000000 ]--- [ 2075.844898] ------------[ cut here ]------------ [ 2075.849514] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2075.861161] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2075.869859] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2075.925335] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2075.934459] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2075.941942] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2075.947444] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2075.952927] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2075.971676] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2075.976902] RAX: 0000000000000000 RBX: ffff8881d0aeb0e8 RCX: 0000000000000000 [ 2075.984033] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2075.991165] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe8c3 [ 2075.998307] R10: fffffffffffe8b30 R11: ffffc9000030b828 R12: ffff8881d0aeb838 [ 2076.005443] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2076.012574] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2076.020660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.026405] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2076.033537] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2076.040669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2076.047796] Call Trace: [ 2076.050247] [ 2076.052354] ? __warn+0x85/0x180 [ 2076.055586] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.060466] ? report_bug+0xfc/0x1d0 [ 2076.064046] ? prb_read_valid+0x1b/0x20 [ 2076.067886] ? handle_bug+0x44/0x70 [ 2076.071377] ? exc_invalid_op+0x17/0x60 [ 2076.075217] ? asm_exc_invalid_op+0x1a/0x20 [ 2076.079408] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.084283] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.089163] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2076.094054] ceph_add_cap+0x20b/0x620 [ceph] [ 2076.098338] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2076.103062] ? __d_lookup+0xd6/0x200 [ 2076.106639] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2076.111966] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2076.116427] ceph_con_process_message+0x74/0x150 [libceph] [ 2076.121923] ? calc_signature+0xdb/0x100 [libceph] [ 2076.126726] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2076.132049] ? process_scheduled_works+0x22a/0x510 [ 2076.136840] ? process_scheduled_works+0x22a/0x510 [ 2076.141629] ceph_con_workfn+0x344/0x750 [libceph] [ 2076.146434] ? process_scheduled_works+0x22a/0x510 [ 2076.151224] process_scheduled_works+0x2a4/0x510 [ 2076.155850] worker_thread+0x188/0x340 [ 2076.159599] ? rescuer_thread+0x310/0x310 [ 2076.163609] kthread+0xf5/0x130 [ 2076.166756] ? kthread_complete_and_exit+0x20/0x20 [ 2076.171549] ret_from_fork+0x31/0x40 [ 2076.175128] ? kthread_complete_and_exit+0x20/0x20 [ 2076.179923] ret_from_fork_asm+0x11/0x20 [ 2076.183855] [ 2076.186049] irq event stamp: 9216033 [ 2076.189629] hardirqs last enabled at (9216043): [] console_unlock+0x106/0x120 [ 2076.198408] hardirqs last disabled at (9216052): [] console_unlock+0xeb/0x120 [ 2076.207102] softirqs last enabled at (9216010): [] __do_softirq+0x2be/0x371 [ 2076.215706] softirqs last disabled at (9216005): [] irq_exit_rcu+0xb0/0xd0 [ 2076.224139] ---[ end trace 0000000000000000 ]--- [ 2076.228762] ------------[ cut here ]------------ [ 2076.233382] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2076.245126] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2076.254086] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2076.309593] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2076.318552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2076.326028] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2076.331516] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2076.350270] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2076.355497] RAX: 0000000000000000 RBX: ffff8881c3fc3ab0 RCX: 0000000000000000 [ 2076.362628] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2076.369761] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe8fe [ 2076.376896] R10: fffffffffffe9bf0 R11: ffffc90001fdfb18 R12: ffff8881c3fc4200 [ 2076.384026] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2076.391158] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2076.399246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.404990] CR2: 00007f9bd4ce3000 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2076.412126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2076.419256] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2076.426392] Call Trace: [ 2076.428841] [ 2076.430951] ? __warn+0x85/0x180 [ 2076.434181] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.439064] ? report_bug+0xfc/0x1d0 [ 2076.442652] ? prb_read_valid+0x1b/0x20 [ 2076.446492] ? handle_bug+0x44/0x70 [ 2076.449981] ? exc_invalid_op+0x17/0x60 [ 2076.453823] ? asm_exc_invalid_op+0x1a/0x20 [ 2076.458015] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.462903] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.467782] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2076.472686] ceph_open+0x16b/0x440 [ceph] [ 2076.476712] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2076.481868] do_dentry_open+0x172/0x530 [ 2076.485708] path_openat+0x743/0x9d0 [ 2076.489293] do_filp_open+0xb8/0x120 [ 2076.492880] ? lock_release+0xc5/0x260 [ 2076.496641] ? _raw_spin_unlock+0x23/0x40 [ 2076.500656] ? alloc_fd+0x13f/0x200 [ 2076.504158] do_sys_openat2+0x249/0x290 [ 2076.508001] do_sys_open+0x4f/0x80 [ 2076.511416] do_syscall_64+0x38/0x80 [ 2076.514999] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2076.520054] RIP: 0033:0x7f48f0eed2f2 [ 2076.523630] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2076.542382] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2076.549953] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2076.557087] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2076.564218] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2076.571354] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2076.578484] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b6 [ 2076.585627] [ 2076.587816] irq event stamp: 12773617 [ 2076.591484] hardirqs last enabled at (12773627): [] console_unlock+0x106/0x120 [ 2076.600349] hardirqs last disabled at (12773636): [] console_unlock+0xeb/0x120 [ 2076.609130] softirqs last enabled at (12773584): [] __do_softirq+0x2be/0x371 [ 2076.617822] softirqs last disabled at (12773579): [] irq_exit_rcu+0xb0/0xd0 [ 2076.626344] ---[ end trace 0000000000000000 ]--- [ 2076.630962] ------------[ cut here ]------------ [ 2076.635581] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2076.647227] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2076.655921] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2076.711400] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2076.720524] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2076.728004] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2076.733500] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2076.738990] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2076.757739] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2076.762966] RAX: 0000000000000000 RBX: ffff8881d0aee1d0 RCX: 0000000000000000 [ 2076.770097] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2076.777230] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe938 [ 2076.784364] R10: fffffffffffead28 R11: ffffc9000030b828 R12: ffff8881d0aee920 [ 2076.791497] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2076.798627] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2076.806717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2076.812460] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2076.819594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2076.826727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2076.833860] Call Trace: [ 2076.836313] [ 2076.838418] ? __warn+0x85/0x180 [ 2076.841653] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.846534] ? report_bug+0xfc/0x1d0 [ 2076.850109] ? prb_read_valid+0x1b/0x20 [ 2076.853952] ? handle_bug+0x44/0x70 [ 2076.857442] ? exc_invalid_op+0x17/0x60 [ 2076.861282] ? asm_exc_invalid_op+0x1a/0x20 [ 2076.865472] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.870347] ? __list_add_valid_or_report+0x7d/0xa0 [ 2076.875227] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2076.880117] ceph_add_cap+0x20b/0x620 [ceph] [ 2076.884401] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2076.889118] ? __d_lookup+0xd6/0x200 [ 2076.892703] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2076.898028] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2076.902494] ceph_con_process_message+0x74/0x150 [libceph] [ 2076.907987] ? calc_signature+0xdb/0x100 [libceph] [ 2076.912801] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2076.918130] ? process_scheduled_works+0x22a/0x510 [ 2076.922922] ? process_scheduled_works+0x22a/0x510 [ 2076.927713] ceph_con_workfn+0x344/0x750 [libceph] [ 2076.932516] ? process_scheduled_works+0x22a/0x510 [ 2076.937306] process_scheduled_works+0x2a4/0x510 [ 2076.941930] worker_thread+0x188/0x340 [ 2076.945682] ? rescuer_thread+0x310/0x310 [ 2076.949693] kthread+0xf5/0x130 [ 2076.952838] ? kthread_complete_and_exit+0x20/0x20 [ 2076.957633] ret_from_fork+0x31/0x40 [ 2076.961212] ? kthread_complete_and_exit+0x20/0x20 [ 2076.966005] ret_from_fork_asm+0x11/0x20 [ 2076.969937] [ 2076.972131] irq event stamp: 9217885 [ 2076.975710] hardirqs last enabled at (9217895): [] console_unlock+0x106/0x120 [ 2076.984489] hardirqs last disabled at (9217904): [] console_unlock+0xeb/0x120 [ 2076.993182] softirqs last enabled at (9217862): [] __do_softirq+0x2be/0x371 [ 2077.001789] softirqs last disabled at (9217923): [] irq_exit_rcu+0xb0/0xd0 [ 2077.010221] ---[ end trace 0000000000000000 ]--- [ 2077.014843] ------------[ cut here ]------------ [ 2077.019461] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2077.031109] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2077.040062] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2077.095535] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2077.104494] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2077.111970] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2077.117457] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2077.136201] RSP: 0018:ffffc90001fdfbd0 EFLAGS: 00010282 [ 2077.141431] RAX: 0000000000000000 RBX: ffff8881c3fc3ab0 RCX: 0000000000000000 [ 2077.148565] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2077.155696] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe973 [ 2077.162830] R10: fffffffffffebf00 R11: ffffc90001fdfa60 R12: ffff8881c3fc4200 [ 2077.169961] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2077.177092] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2077.185178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2077.190925] CR2: 00007f9bd4ce3000 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2077.198059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2077.205189] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2077.212324] Call Trace: [ 2077.214776] [ 2077.216884] ? __warn+0x85/0x180 [ 2077.220116] ? __list_add_valid_or_report+0x7d/0xa0 [ 2077.224997] ? report_bug+0xfc/0x1d0 [ 2077.228576] ? prb_read_valid+0x1b/0x20 [ 2077.232420] ? handle_bug+0x44/0x70 [ 2077.235914] ? exc_invalid_op+0x17/0x60 [ 2077.239755] ? asm_exc_invalid_op+0x1a/0x20 [ 2077.243949] ? __list_add_valid_or_report+0x7d/0xa0 [ 2077.248833] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2077.253729] try_get_cap_refs+0x433/0x880 [ceph] [ 2077.258380] __ceph_get_caps+0xf8/0x760 [ceph] [ 2077.262851] ? ceph_read_iter+0x134/0x860 [ceph] [ 2077.267491] ceph_read_iter+0x134/0x860 [ceph] [ 2077.271959] ? selinux_file_permission+0x119/0x150 [ 2077.276759] ? vfs_read+0x20f/0x300 [ 2077.280254] vfs_read+0x20f/0x300 [ 2077.283585] ksys_pread64+0x65/0xa0 [ 2077.287074] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2077.292222] do_syscall_64+0x38/0x80 [ 2077.295803] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2077.300855] RIP: 0033:0x7f48f0eeb88b [ 2077.304434] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2077.323181] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2077.330747] RAX: ffffffffffffffda RBX: 0000000002075940 RCX: 00007f48f0eeb88b [ 2077.337881] RDX: 0000000000001000 RSI: 000000000207e080 RDI: 0000000000000016 [ 2077.345016] RBP: 00000000000029b6 R08: 0000000000000063 R09: 0000000000000003 [ 2077.352157] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2077.359298] R13: 0000000000001000 R14: 000000000207e080 R15: 00007f48f183dac8 [ 2077.366440] [ 2077.368630] irq event stamp: 12775585 [ 2077.372298] hardirqs last enabled at (12775595): [] console_unlock+0x106/0x120 [ 2077.381169] hardirqs last disabled at (12775604): [] console_unlock+0xeb/0x120 [ 2077.389951] softirqs last enabled at (12775562): [] __do_softirq+0x2be/0x371 [ 2077.398642] softirqs last disabled at (12775557): [] irq_exit_rcu+0xb0/0xd0 [ 2077.407167] ---[ end trace 0000000000000000 ]--- [ 2077.411790] ------------[ cut here ]------------ [ 2077.416408] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2077.428058] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2077.436753] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2077.492238] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2077.501363] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2077.508847] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2077.514349] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2077.519838] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2077.538589] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2077.543824] RAX: 0000000000000000 RBX: ffff8881d0ae8000 RCX: 0000000000000000 [ 2077.550954] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2077.558087] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe9ab [ 2077.565219] R10: fffffffffffed080 R11: ffffc9000030b828 R12: ffff8881d0ae8750 [ 2077.572349] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2077.579479] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2077.587564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2077.593310] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2077.600440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2077.607573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2077.614709] Call Trace: [ 2077.617159] [ 2077.619267] ? __warn+0x85/0x180 [ 2077.622500] ? __list_add_valid_or_report+0x7d/0xa0 [ 2077.627380] ? report_bug+0xfc/0x1d0 [ 2077.630959] ? prb_read_valid+0x1b/0x20 [ 2077.634801] ? handle_bug+0x44/0x70 [ 2077.638290] ? exc_invalid_op+0x17/0x60 [ 2077.642131] ? asm_exc_invalid_op+0x1a/0x20 [ 2077.646325] ? __list_add_valid_or_report+0x7d/0xa0 [ 2077.651206] ? __list_add_valid_or_report+0x7d/0xa0 [ 2077.656081] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2077.660976] ceph_add_cap+0x20b/0x620 [ceph] [ 2077.665271] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2077.669996] ? __d_lookup+0xd6/0x200 [ 2077.673575] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2077.678907] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2077.683378] ceph_con_process_message+0x74/0x150 [libceph] [ 2077.688879] ? calc_signature+0xdb/0x100 [libceph] [ 2077.693696] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2077.699026] ? process_scheduled_works+0x22a/0x510 [ 2077.703822] ? process_scheduled_works+0x22a/0x510 [ 2077.708612] ceph_con_workfn+0x344/0x750 [libceph] [ 2077.713426] ? process_scheduled_works+0x22a/0x510 [ 2077.718224] process_scheduled_works+0x2a4/0x510 [ 2077.722853] worker_thread+0x188/0x340 [ 2077.726608] ? rescuer_thread+0x310/0x310 [ 2077.730627] kthread+0xf5/0x130 [ 2077.733772] ? kthread_complete_and_exit+0x20/0x20 [ 2077.738567] ret_from_fork+0x31/0x40 [ 2077.742152] ? kthread_complete_and_exit+0x20/0x20 [ 2077.746949] ret_from_fork_asm+0x11/0x20 [ 2077.750890] [ 2077.753083] irq event stamp: 9219739 [ 2077.756658] hardirqs last enabled at (9219749): [] console_unlock+0x106/0x120 [ 2077.765442] hardirqs last disabled at (9219758): [] console_unlock+0xeb/0x120 [ 2077.774146] softirqs last enabled at (9219772): [] __do_softirq+0x2be/0x371 [ 2077.782757] softirqs last disabled at (9219767): [] irq_exit_rcu+0xb0/0xd0 [ 2077.791190] ---[ end trace 0000000000000000 ]--- [ 2077.795809] ------------[ cut here ]------------ [ 2077.800428] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2077.812074] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2077.821027] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2077.876489] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2077.885439] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2077.892918] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2077.898404] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2077.917150] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2077.922384] RAX: 0000000000000000 RBX: ffff8881e4eb0000 RCX: 0000000000000000 [ 2077.929514] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2077.936643] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffe9e6 [ 2077.943776] R10: fffffffffffee378 R11: ffffc90001fdfb18 R12: ffff8881e4eb0750 [ 2077.950907] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2077.958033] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2077.966120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2077.971865] CR2: 00007fb38eed0010 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2077.978997] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2077.986122] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2077.993254] Call Trace: [ 2077.995708] [ 2077.997814] ? __warn+0x85/0x180 [ 2078.001046] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.005927] ? report_bug+0xfc/0x1d0 [ 2078.009506] ? prb_read_valid+0x1b/0x20 [ 2078.013346] ? handle_bug+0x44/0x70 [ 2078.016836] ? exc_invalid_op+0x17/0x60 [ 2078.020669] ? asm_exc_invalid_op+0x1a/0x20 [ 2078.024858] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.029734] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.034611] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2078.039503] ceph_open+0x16b/0x440 [ceph] [ 2078.043524] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2078.048673] do_dentry_open+0x172/0x530 [ 2078.052512] path_openat+0x743/0x9d0 [ 2078.056092] do_filp_open+0xb8/0x120 [ 2078.059671] ? lock_release+0xc5/0x260 [ 2078.063422] ? _raw_spin_unlock+0x23/0x40 [ 2078.067433] ? alloc_fd+0x13f/0x200 [ 2078.070929] do_sys_openat2+0x249/0x290 [ 2078.074770] do_sys_open+0x4f/0x80 [ 2078.078172] do_syscall_64+0x38/0x80 [ 2078.081753] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2078.086804] RIP: 0033:0x7f48f0eed2f2 [ 2078.090383] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2078.109130] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2078.116696] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2078.123829] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2078.130961] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2078.138087] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2078.145219] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029b7 [ 2078.152358] [ 2078.154545] irq event stamp: 12777569 [ 2078.158210] hardirqs last enabled at (12777579): [] console_unlock+0x106/0x120 [ 2078.167076] hardirqs last disabled at (12777588): [] console_unlock+0xeb/0x120 [ 2078.175855] softirqs last enabled at (12777536): [] __do_softirq+0x2be/0x371 [ 2078.184550] softirqs last disabled at (12777531): [] irq_exit_rcu+0xb0/0xd0 [ 2078.193068] ---[ end trace 0000000000000000 ]--- [ 2078.197681] ------------[ cut here ]------------ [ 2078.202301] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2078.213950] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2078.222641] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2078.278138] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2078.287271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2078.294750] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2078.300253] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2078.305739] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2078.324484] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2078.329712] RAX: 0000000000000000 RBX: ffff8881d12d1390 RCX: 0000000000000000 [ 2078.336845] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2078.343976] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffea20 [ 2078.351108] R10: fffffffffffef4a8 R11: ffffc9000030b828 R12: ffff8881d12d1ae0 [ 2078.358243] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2078.365376] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2078.373461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2078.379207] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2078.386341] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2078.393473] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2078.400608] Call Trace: [ 2078.403060] [ 2078.405165] ? __warn+0x85/0x180 [ 2078.408398] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.413279] ? report_bug+0xfc/0x1d0 [ 2078.416856] ? prb_read_valid+0x1b/0x20 [ 2078.420704] ? handle_bug+0x44/0x70 [ 2078.424199] ? exc_invalid_op+0x17/0x60 [ 2078.428038] ? asm_exc_invalid_op+0x1a/0x20 [ 2078.432226] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.437102] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.441983] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2078.446881] ceph_add_cap+0x20b/0x620 [ceph] [ 2078.451179] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2078.455903] ? __d_lookup+0xd6/0x200 [ 2078.459489] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2078.464817] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2078.469282] ceph_con_process_message+0x74/0x150 [libceph] [ 2078.474777] ? calc_signature+0xdb/0x100 [libceph] [ 2078.479590] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2078.484920] ? process_scheduled_works+0x22a/0x510 [ 2078.489716] ? process_scheduled_works+0x22a/0x510 [ 2078.494510] ceph_con_workfn+0x344/0x750 [libceph] [ 2078.499317] ? process_scheduled_works+0x22a/0x510 [ 2078.504114] process_scheduled_works+0x2a4/0x510 [ 2078.508741] worker_thread+0x188/0x340 [ 2078.512497] ? rescuer_thread+0x310/0x310 [ 2078.516509] kthread+0xf5/0x130 [ 2078.519654] ? kthread_complete_and_exit+0x20/0x20 [ 2078.524450] ret_from_fork+0x31/0x40 [ 2078.528033] ? kthread_complete_and_exit+0x20/0x20 [ 2078.532832] ret_from_fork_asm+0x11/0x20 [ 2078.536769] [ 2078.538962] irq event stamp: 9221599 [ 2078.542546] hardirqs last enabled at (9221615): [] console_unlock+0x106/0x120 [ 2078.551331] hardirqs last disabled at (9221624): [] console_unlock+0xeb/0x120 [ 2078.560025] softirqs last enabled at (9221612): [] __do_softirq+0x2be/0x371 [ 2078.568630] softirqs last disabled at (9221607): [] irq_exit_rcu+0xb0/0xd0 [ 2078.577064] ---[ end trace 0000000000000000 ]--- [ 2078.581682] ------------[ cut here ]------------ [ 2078.586305] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2078.598062] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2078.607022] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2078.662524] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2078.671481] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2078.678957] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2078.684445] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2078.703190] RSP: 0018:ffffc90001fdfbd0 EFLAGS: 00010282 [ 2078.708418] RAX: 0000000000000000 RBX: ffff8881e4eb0000 RCX: 0000000000000000 [ 2078.715549] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2078.722684] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffea5b [ 2078.729816] R10: ffffffffffff05b8 R11: ffffc90001fdfa60 R12: ffff8881e4eb0750 [ 2078.736949] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2078.744081] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2078.752169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2078.757913] CR2: 00007f9bd4ce3000 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2078.765061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2078.772196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2078.779328] Call Trace: [ 2078.781782] [ 2078.783889] ? __warn+0x85/0x180 [ 2078.787121] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.792019] ? report_bug+0xfc/0x1d0 [ 2078.795597] ? prb_read_valid+0x1b/0x20 [ 2078.799441] ? handle_bug+0x44/0x70 [ 2078.802938] ? exc_invalid_op+0x17/0x60 [ 2078.806780] ? asm_exc_invalid_op+0x1a/0x20 [ 2078.810980] ? __list_add_valid_or_report+0x7d/0xa0 [ 2078.815865] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2078.820760] try_get_cap_refs+0x433/0x880 [ceph] [ 2078.825411] __ceph_get_caps+0xf8/0x760 [ceph] [ 2078.829883] ? ceph_read_iter+0x134/0x860 [ceph] [ 2078.834521] ceph_read_iter+0x134/0x860 [ceph] [ 2078.838982] ? selinux_file_permission+0x119/0x150 [ 2078.843785] ? vfs_read+0x20f/0x300 [ 2078.847276] vfs_read+0x20f/0x300 [ 2078.850606] ksys_pread64+0x65/0xa0 [ 2078.854096] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2078.859244] do_syscall_64+0x38/0x80 [ 2078.862826] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2078.867880] RIP: 0033:0x7f48f0eeb88b [ 2078.871456] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2078.890204] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2078.897770] RAX: ffffffffffffffda RBX: 0000000002075950 RCX: 00007f48f0eeb88b [ 2078.904902] RDX: 000000000000021f RSI: 000000000207d800 RDI: 0000000000000017 [ 2078.912034] RBP: 00000000000029b7 R08: 000000000000021f R09: 00000000020767c0 [ 2078.919168] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000021f [ 2078.926301] R13: 000000000000021f R14: 000000000207d800 R15: 00007f48f183dac8 [ 2078.933443] [ 2078.935635] irq event stamp: 12779565 [ 2078.939302] hardirqs last enabled at (12779575): [] console_unlock+0x106/0x120 [ 2078.948171] hardirqs last disabled at (12779590): [] console_unlock+0xeb/0x120 [ 2078.956957] softirqs last enabled at (12779588): [] __do_softirq+0x2be/0x371 [ 2078.965656] softirqs last disabled at (12779583): [] irq_exit_rcu+0xb0/0xd0 [ 2078.974177] ---[ end trace 0000000000000000 ]--- [ 2078.978797] ------------[ cut here ]------------ [ 2078.983416] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2078.995064] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2079.003784] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2079.059332] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2079.068464] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2079.075943] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2079.081447] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2079.086951] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2079.105700] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2079.110931] RAX: 0000000000000000 RBX: ffff8881d0bd9d58 RCX: 0000000000000000 [ 2079.118064] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2079.125197] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffea93 [ 2079.132329] R10: ffffffffffff1388 R11: ffffc9000030b828 R12: ffff8881d0bda4a8 [ 2079.139466] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2079.146603] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2079.154690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2079.160437] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2079.167570] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2079.174704] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2079.181841] Call Trace: [ 2079.184297] [ 2079.186402] ? __warn+0x85/0x180 [ 2079.189637] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.194516] ? report_bug+0xfc/0x1d0 [ 2079.198096] ? prb_read_valid+0x1b/0x20 [ 2079.201939] ? handle_bug+0x44/0x70 [ 2079.205438] ? exc_invalid_op+0x17/0x60 [ 2079.209284] ? asm_exc_invalid_op+0x1a/0x20 [ 2079.213498] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.218420] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.223307] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2079.228233] ceph_add_cap+0x20b/0x620 [ceph] [ 2079.232556] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2079.237321] ? __d_lookup+0xd6/0x200 [ 2079.240935] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2079.246305] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2079.250787] ceph_con_process_message+0x74/0x150 [libceph] [ 2079.256294] ? calc_signature+0xdb/0x100 [libceph] [ 2079.261133] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2079.266491] ? process_scheduled_works+0x22a/0x510 [ 2079.271320] ? process_scheduled_works+0x22a/0x510 [ 2079.276139] ceph_con_workfn+0x344/0x750 [libceph] [ 2079.280981] ? process_scheduled_works+0x22a/0x510 [ 2079.285777] process_scheduled_works+0x2a4/0x510 [ 2079.290406] worker_thread+0x188/0x340 [ 2079.294188] ? rescuer_thread+0x310/0x310 [ 2079.298205] kthread+0xf5/0x130 [ 2079.301350] ? kthread_complete_and_exit+0x20/0x20 [ 2079.306150] ret_from_fork+0x31/0x40 [ 2079.309745] ? kthread_complete_and_exit+0x20/0x20 [ 2079.314544] ret_from_fork_asm+0x11/0x20 [ 2079.318492] [ 2079.320705] irq event stamp: 9223535 [ 2079.324284] hardirqs last enabled at (9223545): [] console_unlock+0x106/0x120 [ 2079.333062] hardirqs last disabled at (9223554): [] console_unlock+0xeb/0x120 [ 2079.341760] softirqs last enabled at (9223570): [] __do_softirq+0x2be/0x371 [ 2079.350370] softirqs last disabled at (9223563): [] irq_exit_rcu+0xb0/0xd0 [ 2079.358804] ---[ end trace 0000000000000000 ]--- [ 2079.363430] ------------[ cut here ]------------ [ 2079.368055] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2079.379730] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2079.388422] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2079.443919] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2079.453044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2079.460541] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2079.466064] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2079.471567] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2079.490330] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2079.495572] RAX: 0000000000000000 RBX: ffff8881d0aea720 RCX: 0000000000000000 [ 2079.502741] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2079.509875] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeace [ 2079.517005] R10: ffffffffffff23b8 R11: ffffc9000030b828 R12: ffff8881d0aeae70 [ 2079.524154] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2079.531306] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2079.539410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2079.545173] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2079.552321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2079.559472] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2079.566624] Call Trace: [ 2079.569114] [ 2079.571219] ? __warn+0x85/0x180 [ 2079.574487] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.579365] ? report_bug+0xfc/0x1d0 [ 2079.582965] ? prb_read_valid+0x1b/0x20 [ 2079.586816] ? handle_bug+0x44/0x70 [ 2079.590311] ? exc_invalid_op+0x17/0x60 [ 2079.594168] ? asm_exc_invalid_op+0x1a/0x20 [ 2079.598398] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.603295] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.608171] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2079.613101] ceph_add_cap+0x20b/0x620 [ceph] [ 2079.617414] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2079.622153] ? __d_lookup+0xd6/0x200 [ 2079.625755] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2079.631110] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2079.635598] ceph_con_process_message+0x74/0x150 [libceph] [ 2079.641119] ? calc_signature+0xdb/0x100 [libceph] [ 2079.645931] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2079.651281] ? process_scheduled_works+0x22a/0x510 [ 2079.656095] ? process_scheduled_works+0x22a/0x510 [ 2079.660885] ceph_con_workfn+0x344/0x750 [libceph] [ 2079.665721] ? process_scheduled_works+0x22a/0x510 [ 2079.670513] process_scheduled_works+0x2a4/0x510 [ 2079.675159] worker_thread+0x188/0x340 [ 2079.678915] ? rescuer_thread+0x310/0x310 [ 2079.682934] kthread+0xf5/0x130 [ 2079.686078] ? kthread_complete_and_exit+0x20/0x20 [ 2079.690892] ret_from_fork+0x31/0x40 [ 2079.694477] ? kthread_complete_and_exit+0x20/0x20 [ 2079.699289] ret_from_fork_asm+0x11/0x20 [ 2079.703257] [ 2079.705449] irq event stamp: 9224609 [ 2079.709046] hardirqs last enabled at (9224619): [] console_unlock+0x106/0x120 [ 2079.717842] hardirqs last disabled at (9224628): [] console_unlock+0xeb/0x120 [ 2079.726534] softirqs last enabled at (9224606): [] __do_softirq+0x2be/0x371 [ 2079.735157] softirqs last disabled at (9224601): [] irq_exit_rcu+0xb0/0xd0 [ 2079.743608] ---[ end trace 0000000000000000 ]--- [ 2079.748269] ------------[ cut here ]------------ [ 2079.752886] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2079.764526] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2079.773225] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2079.828700] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2079.837828] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2079.845303] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2079.850800] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2079.856285] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2079.875035] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2079.880266] RAX: 0000000000000000 RBX: ffff8881d0ae89c8 RCX: 0000000000000000 [ 2079.887399] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2079.894532] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeb09 [ 2079.901667] R10: ffffffffffff3658 R11: ffffc9000030b828 R12: ffff8881d0ae9118 [ 2079.908799] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2079.915931] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2079.924031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2079.929779] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2079.936914] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2079.944046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2079.951181] Call Trace: [ 2079.953632] [ 2079.955754] ? __warn+0x85/0x180 [ 2079.958991] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.963871] ? report_bug+0xfc/0x1d0 [ 2079.967448] ? prb_read_valid+0x1b/0x20 [ 2079.971291] ? handle_bug+0x44/0x70 [ 2079.974791] ? exc_invalid_op+0x17/0x60 [ 2079.978627] ? asm_exc_invalid_op+0x1a/0x20 [ 2079.982823] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.987700] ? __list_add_valid_or_report+0x7d/0xa0 [ 2079.992581] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2079.997483] ceph_add_cap+0x20b/0x620 [ceph] [ 2080.001779] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2080.006498] ? __d_lookup+0xd6/0x200 [ 2080.010116] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2080.015448] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2080.019917] ceph_con_process_message+0x74/0x150 [libceph] [ 2080.025412] ? calc_signature+0xdb/0x100 [libceph] [ 2080.030230] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2080.035556] ? process_scheduled_works+0x22a/0x510 [ 2080.040358] ? process_scheduled_works+0x22a/0x510 [ 2080.045154] ceph_con_workfn+0x344/0x750 [libceph] [ 2080.049958] ? process_scheduled_works+0x22a/0x510 [ 2080.054758] process_scheduled_works+0x2a4/0x510 [ 2080.059380] worker_thread+0x188/0x340 [ 2080.063131] ? rescuer_thread+0x310/0x310 [ 2080.067143] kthread+0xf5/0x130 [ 2080.070289] ? kthread_complete_and_exit+0x20/0x20 [ 2080.075083] ret_from_fork+0x31/0x40 [ 2080.078661] ? kthread_complete_and_exit+0x20/0x20 [ 2080.083456] ret_from_fork_asm+0x11/0x20 [ 2080.087385] [ 2080.089579] irq event stamp: 9225669 [ 2080.093163] hardirqs last enabled at (9225685): [] console_unlock+0x106/0x120 [ 2080.101937] hardirqs last disabled at (9225694): [] console_unlock+0xeb/0x120 [ 2080.110630] softirqs last enabled at (9225682): [] __do_softirq+0x2be/0x371 [ 2080.119238] softirqs last disabled at (9225677): [] irq_exit_rcu+0xb0/0xd0 [ 2080.127668] ---[ end trace 0000000000000000 ]--- [ 2080.132300] ------------[ cut here ]------------ [ 2080.136926] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2080.148576] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2080.157317] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2080.212869] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2080.221996] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2080.229472] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2080.234995] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2080.240488] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2080.259252] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2080.264494] RAX: 0000000000000000 RBX: ffff8881d0aebab0 RCX: 0000000000000000 [ 2080.271646] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2080.278780] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeb44 [ 2080.285915] R10: ffffffffffff4790 R11: ffffc9000030b828 R12: ffff8881d0aec200 [ 2080.293052] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2080.300203] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2080.308291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2080.314051] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2080.321203] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2080.328352] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2080.335504] Call Trace: [ 2080.337975] [ 2080.340098] ? __warn+0x85/0x180 [ 2080.343340] ? __list_add_valid_or_report+0x7d/0xa0 [ 2080.348244] ? report_bug+0xfc/0x1d0 [ 2080.351836] ? prb_read_valid+0x1b/0x20 [ 2080.355699] ? handle_bug+0x44/0x70 [ 2080.359199] ? exc_invalid_op+0x17/0x60 [ 2080.363079] ? asm_exc_invalid_op+0x1a/0x20 [ 2080.367294] ? __list_add_valid_or_report+0x7d/0xa0 [ 2080.372189] ? __list_add_valid_or_report+0x7d/0xa0 [ 2080.377068] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2080.381987] ceph_add_cap+0x20b/0x620 [ceph] [ 2080.386311] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2080.391078] ? __d_lookup+0xd6/0x200 [ 2080.394688] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2080.400039] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2080.404527] ceph_con_process_message+0x74/0x150 [libceph] [ 2080.410029] ? calc_signature+0xdb/0x100 [libceph] [ 2080.414855] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2080.420209] ? process_scheduled_works+0x22a/0x510 [ 2080.425013] ? process_scheduled_works+0x22a/0x510 [ 2080.429806] ceph_con_workfn+0x344/0x750 [libceph] [ 2080.434637] ? process_scheduled_works+0x22a/0x510 [ 2080.439435] process_scheduled_works+0x2a4/0x510 [ 2080.444099] worker_thread+0x188/0x340 [ 2080.447856] ? rescuer_thread+0x310/0x310 [ 2080.451873] kthread+0xf5/0x130 [ 2080.455021] ? kthread_complete_and_exit+0x20/0x20 [ 2080.459825] ret_from_fork+0x31/0x40 [ 2080.463408] ? kthread_complete_and_exit+0x20/0x20 [ 2080.468236] ret_from_fork_asm+0x11/0x20 [ 2080.472192] [ 2080.474389] irq event stamp: 9226715 [ 2080.477989] hardirqs last enabled at (9226731): [] console_unlock+0x106/0x120 [ 2080.486774] hardirqs last disabled at (9226740): [] console_unlock+0xeb/0x120 [ 2080.495468] softirqs last enabled at (9226728): [] __do_softirq+0x2be/0x371 [ 2080.504101] softirqs last disabled at (9226723): [] irq_exit_rcu+0xb0/0xd0 [ 2080.512541] ---[ end trace 0000000000000000 ]--- [ 2080.517209] ------------[ cut here ]------------ [ 2080.521834] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2080.533503] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2080.542198] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2080.597691] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2080.606836] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2080.614310] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2080.619827] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2080.625319] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2080.644084] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2080.649327] RAX: 0000000000000000 RBX: ffff8881d0ae9390 RCX: 0000000000000000 [ 2080.656477] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2080.663625] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeb7f [ 2080.670781] R10: ffffffffffff5950 R11: ffffc9000030b828 R12: ffff8881d0ae9ae0 [ 2080.677908] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2080.685051] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2080.693154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2080.698918] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2080.706061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2080.713210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2080.720361] Call Trace: [ 2080.722830] [ 2080.724937] ? __warn+0x85/0x180 [ 2080.728170] ? __list_add_valid_or_report+0x7d/0xa0 [ 2080.733084] ? report_bug+0xfc/0x1d0 [ 2080.736680] ? prb_read_valid+0x1b/0x20 [ 2080.740538] ? handle_bug+0x44/0x70 [ 2080.744031] ? exc_invalid_op+0x17/0x60 [ 2080.747888] ? asm_exc_invalid_op+0x1a/0x20 [ 2080.752098] ? __list_add_valid_or_report+0x7d/0xa0 [ 2080.756980] ? __list_add_valid_or_report+0x7d/0xa0 [ 2080.761857] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2080.766770] ceph_add_cap+0x20b/0x620 [ceph] [ 2080.771085] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2080.775812] ? __d_lookup+0xd6/0x200 [ 2080.779414] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2080.784773] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2080.789254] ceph_con_process_message+0x74/0x150 [libceph] [ 2080.794773] ? calc_signature+0xdb/0x100 [libceph] [ 2080.799608] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2080.804940] ? process_scheduled_works+0x22a/0x510 [ 2080.809759] ? process_scheduled_works+0x22a/0x510 [ 2080.814549] ceph_con_workfn+0x344/0x750 [libceph] [ 2080.819374] ? process_scheduled_works+0x22a/0x510 [ 2080.824171] process_scheduled_works+0x2a4/0x510 [ 2080.828817] worker_thread+0x188/0x340 [ 2080.832588] ? rescuer_thread+0x310/0x310 [ 2080.836601] kthread+0xf5/0x130 [ 2080.839775] ? kthread_complete_and_exit+0x20/0x20 [ 2080.844591] ret_from_fork+0x31/0x40 [ 2080.848171] ? kthread_complete_and_exit+0x20/0x20 [ 2080.852984] ret_from_fork_asm+0x11/0x20 [ 2080.856926] [ 2080.859117] irq event stamp: 9227775 [ 2080.862711] hardirqs last enabled at (9227791): [] console_unlock+0x106/0x120 [ 2080.871508] hardirqs last disabled at (9227800): [] console_unlock+0xeb/0x120 [ 2080.880218] softirqs last enabled at (9227788): [] __do_softirq+0x2be/0x371 [ 2080.888845] softirqs last disabled at (9227783): [] irq_exit_rcu+0xb0/0xd0 [ 2080.897275] ---[ end trace 0000000000000000 ]--- [ 2080.901918] ------------[ cut here ]------------ [ 2080.906550] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2080.918213] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2080.926916] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2080.982394] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2080.991536] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2080.999033] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2081.004562] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2081.010074] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2081.028839] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2081.034077] RAX: 0000000000000000 RBX: ffff8881d0bdeb98 RCX: 0000000000000000 [ 2081.041222] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2081.048371] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffebba [ 2081.055525] R10: ffffffffffff6ab0 R11: ffffc9000030b828 R12: ffff8881d0bdf2e8 [ 2081.062672] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2081.069823] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2081.077909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2081.083656] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2081.090787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2081.097920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2081.105061] Call Trace: [ 2081.107532] [ 2081.109676] ? __warn+0x85/0x180 [ 2081.112907] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.117790] ? report_bug+0xfc/0x1d0 [ 2081.121373] ? prb_read_valid+0x1b/0x20 [ 2081.125251] ? handle_bug+0x44/0x70 [ 2081.128740] ? exc_invalid_op+0x17/0x60 [ 2081.132596] ? asm_exc_invalid_op+0x1a/0x20 [ 2081.136791] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.141690] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.146569] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2081.151494] ceph_add_cap+0x20b/0x620 [ceph] [ 2081.155790] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2081.160532] ? __d_lookup+0xd6/0x200 [ 2081.164133] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2081.169486] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2081.173958] ceph_con_process_message+0x74/0x150 [libceph] [ 2081.179459] ? calc_signature+0xdb/0x100 [libceph] [ 2081.184310] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2081.189657] ? process_scheduled_works+0x22a/0x510 [ 2081.194473] ? process_scheduled_works+0x22a/0x510 [ 2081.199262] ceph_con_workfn+0x344/0x750 [libceph] [ 2081.204102] ? process_scheduled_works+0x22a/0x510 [ 2081.208892] process_scheduled_works+0x2a4/0x510 [ 2081.213537] worker_thread+0x188/0x340 [ 2081.217307] ? rescuer_thread+0x310/0x310 [ 2081.221319] kthread+0xf5/0x130 [ 2081.224483] ? kthread_complete_and_exit+0x20/0x20 [ 2081.229312] ret_from_fork+0x31/0x40 [ 2081.232890] ? kthread_complete_and_exit+0x20/0x20 [ 2081.237682] ret_from_fork_asm+0x11/0x20 [ 2081.241649] [ 2081.243836] irq event stamp: 9228829 [ 2081.247432] hardirqs last enabled at (9228845): [] console_unlock+0x106/0x120 [ 2081.256211] hardirqs last disabled at (9228854): [] console_unlock+0xeb/0x120 [ 2081.264921] softirqs last enabled at (9228842): [] __do_softirq+0x2be/0x371 [ 2081.273528] softirqs last disabled at (9228837): [] irq_exit_rcu+0xb0/0xd0 [ 2081.281979] ---[ end trace 0000000000000000 ]--- [ 2081.286762] ------------[ cut here ]------------ [ 2081.291383] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2081.303030] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2081.311725] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2081.367228] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2081.376354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2081.383831] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2081.389335] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2081.394820] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2081.413567] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2081.418794] RAX: 0000000000000000 RBX: ffff8881e4d80000 RCX: 0000000000000000 [ 2081.425926] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2081.433059] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffebf5 [ 2081.440192] R10: ffffffffffff7d48 R11: ffffc9000030b828 R12: ffff8881e4d80750 [ 2081.447323] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2081.454457] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2081.462543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2081.468288] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2081.475422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2081.482555] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2081.489688] Call Trace: [ 2081.492140] [ 2081.494247] ? __warn+0x85/0x180 [ 2081.497479] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.502361] ? report_bug+0xfc/0x1d0 [ 2081.505945] ? prb_read_valid+0x1b/0x20 [ 2081.509789] ? handle_bug+0x44/0x70 [ 2081.513280] ? exc_invalid_op+0x17/0x60 [ 2081.517120] ? asm_exc_invalid_op+0x1a/0x20 [ 2081.521313] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.526193] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.531072] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2081.535973] ceph_add_cap+0x20b/0x620 [ceph] [ 2081.540268] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2081.544993] ? __d_lookup+0xd6/0x200 [ 2081.548576] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2081.553915] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2081.558384] ceph_con_process_message+0x74/0x150 [libceph] [ 2081.563886] ? calc_signature+0xdb/0x100 [libceph] [ 2081.568699] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2081.574033] ? process_scheduled_works+0x22a/0x510 [ 2081.578832] ? process_scheduled_works+0x22a/0x510 [ 2081.583626] ceph_con_workfn+0x344/0x750 [libceph] [ 2081.588434] ? process_scheduled_works+0x22a/0x510 [ 2081.593230] process_scheduled_works+0x2a4/0x510 [ 2081.597859] worker_thread+0x188/0x340 [ 2081.601613] ? rescuer_thread+0x310/0x310 [ 2081.605624] kthread+0xf5/0x130 [ 2081.608785] ? kthread_complete_and_exit+0x20/0x20 [ 2081.613582] ret_from_fork+0x31/0x40 [ 2081.617158] ? kthread_complete_and_exit+0x20/0x20 [ 2081.621954] ret_from_fork_asm+0x11/0x20 [ 2081.625890] [ 2081.628081] irq event stamp: 9229873 [ 2081.631661] hardirqs last enabled at (9229891): [] console_unlock+0x106/0x120 [ 2081.640441] hardirqs last disabled at (9229914): [] console_unlock+0xeb/0x120 [ 2081.649131] softirqs last enabled at (9229912): [] __do_softirq+0x2be/0x371 [ 2081.657738] softirqs last disabled at (9229899): [] irq_exit_rcu+0xb0/0xd0 [ 2081.666171] ---[ end trace 0000000000000000 ]--- [ 2081.670803] ------------[ cut here ]------------ [ 2081.675439] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2081.687083] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2081.695776] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2081.751265] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2081.760395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2081.767876] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2081.773379] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2081.778864] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2081.797613] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2081.802837] RAX: 0000000000000000 RBX: ffff8881d0bdce40 RCX: 0000000000000000 [ 2081.809969] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2081.817104] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffec30 [ 2081.824234] R10: ffffffffffff8d58 R11: ffffc9000030b828 R12: ffff8881d0bdd590 [ 2081.831369] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2081.838501] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2081.846587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2081.852333] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2081.859467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2081.866599] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2081.873732] Call Trace: [ 2081.876186] [ 2081.878295] ? __warn+0x85/0x180 [ 2081.881537] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.886424] ? report_bug+0xfc/0x1d0 [ 2081.890009] ? prb_read_valid+0x1b/0x20 [ 2081.893851] ? handle_bug+0x44/0x70 [ 2081.897340] ? exc_invalid_op+0x17/0x60 [ 2081.901181] ? asm_exc_invalid_op+0x1a/0x20 [ 2081.905377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.910257] ? __list_add_valid_or_report+0x7d/0xa0 [ 2081.915134] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2081.920036] ceph_add_cap+0x20b/0x620 [ceph] [ 2081.924332] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2081.929056] ? __d_lookup+0xd6/0x200 [ 2081.932640] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2081.937976] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2081.942448] ceph_con_process_message+0x74/0x150 [libceph] [ 2081.947947] ? calc_signature+0xdb/0x100 [libceph] [ 2081.952767] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2081.958103] ? process_scheduled_works+0x22a/0x510 [ 2081.962904] ? process_scheduled_works+0x22a/0x510 [ 2081.967697] ceph_con_workfn+0x344/0x750 [libceph] [ 2081.972504] ? process_scheduled_works+0x22a/0x510 [ 2081.977302] process_scheduled_works+0x2a4/0x510 [ 2081.981929] worker_thread+0x188/0x340 [ 2081.985683] ? rescuer_thread+0x310/0x310 [ 2081.989696] kthread+0xf5/0x130 [ 2081.992840] ? kthread_complete_and_exit+0x20/0x20 [ 2081.997635] ret_from_fork+0x31/0x40 [ 2082.001212] ? kthread_complete_and_exit+0x20/0x20 [ 2082.006005] ret_from_fork_asm+0x11/0x20 [ 2082.009944] [ 2082.012136] irq event stamp: 9230935 [ 2082.015712] hardirqs last enabled at (9230945): [] console_unlock+0x106/0x120 [ 2082.024492] hardirqs last disabled at (9230954): [] console_unlock+0xeb/0x120 [ 2082.033183] softirqs last enabled at (9230932): [] __do_softirq+0x2be/0x371 [ 2082.041790] softirqs last disabled at (9230927): [] irq_exit_rcu+0xb0/0xd0 [ 2082.050224] ---[ end trace 0000000000000000 ]--- [ 2082.054848] ------------[ cut here ]------------ [ 2082.059463] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2082.071108] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2082.079804] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2082.135282] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2082.144406] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2082.151885] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2082.157389] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2082.162875] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2082.181623] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2082.186847] RAX: 0000000000000000 RBX: ffff8881d0bdb0e8 RCX: 0000000000000000 [ 2082.193979] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2082.201112] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffec6b [ 2082.208246] R10: ffffffffffffa248 R11: ffffc9000030b828 R12: ffff8881d0bdb838 [ 2082.215378] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2082.222510] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2082.230597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2082.236344] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2082.243475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2082.250607] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2082.257741] Call Trace: [ 2082.260194] [ 2082.262303] ? __warn+0x85/0x180 [ 2082.265534] ? __list_add_valid_or_report+0x7d/0xa0 [ 2082.270415] ? report_bug+0xfc/0x1d0 [ 2082.273993] ? prb_read_valid+0x1b/0x20 [ 2082.277833] ? handle_bug+0x44/0x70 [ 2082.281326] ? exc_invalid_op+0x17/0x60 [ 2082.285167] ? asm_exc_invalid_op+0x1a/0x20 [ 2082.289354] ? __list_add_valid_or_report+0x7d/0xa0 [ 2082.294230] ? __list_add_valid_or_report+0x7d/0xa0 [ 2082.299111] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2082.304008] ceph_add_cap+0x20b/0x620 [ceph] [ 2082.308293] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2082.313018] ? __d_lookup+0xd6/0x200 [ 2082.316606] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2082.321937] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2082.326399] ceph_con_process_message+0x74/0x150 [libceph] [ 2082.331897] ? calc_signature+0xdb/0x100 [libceph] [ 2082.336708] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2082.342041] ? process_scheduled_works+0x22a/0x510 [ 2082.346844] ? process_scheduled_works+0x22a/0x510 [ 2082.351638] ceph_con_workfn+0x344/0x750 [libceph] [ 2082.356442] ? process_scheduled_works+0x22a/0x510 [ 2082.361232] process_scheduled_works+0x2a4/0x510 [ 2082.365856] worker_thread+0x188/0x340 [ 2082.369606] ? rescuer_thread+0x310/0x310 [ 2082.373618] kthread+0xf5/0x130 [ 2082.376764] ? kthread_complete_and_exit+0x20/0x20 [ 2082.381564] ret_from_fork+0x31/0x40 [ 2082.385143] ? kthread_complete_and_exit+0x20/0x20 [ 2082.389938] ret_from_fork_asm+0x11/0x20 [ 2082.393871] [ 2082.396064] irq event stamp: 9231983 [ 2082.399643] hardirqs last enabled at (9231993): [] console_unlock+0x106/0x120 [ 2082.408424] hardirqs last disabled at (9232008): [] console_unlock+0xeb/0x120 [ 2082.417116] softirqs last enabled at (9232006): [] __do_softirq+0x2be/0x371 [ 2082.425721] softirqs last disabled at (9232001): [] irq_exit_rcu+0xb0/0xd0 [ 2082.434156] ---[ end trace 0000000000000000 ]--- [ 2082.438782] ------------[ cut here ]------------ [ 2082.443404] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2082.455100] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2082.463799] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2082.519282] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2082.528425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2082.535919] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2082.541415] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2082.546920] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2082.565665] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2082.570908] RAX: 0000000000000000 RBX: ffff8881d0b8ce40 RCX: 0000000000000000 [ 2082.578042] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2082.585193] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeca6 [ 2082.592326] R10: ffffffffffffb120 R11: ffffc9000030b828 R12: ffff8881d0b8d590 [ 2082.599476] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2082.606623] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2082.614727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2082.620492] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2082.627641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2082.634802] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2082.641932] Call Trace: [ 2082.644387] [ 2082.646530] ? __warn+0x85/0x180 [ 2082.649763] ? __list_add_valid_or_report+0x7d/0xa0 [ 2082.654667] ? report_bug+0xfc/0x1d0 [ 2082.658244] ? prb_read_valid+0x1b/0x20 [ 2082.662122] ? handle_bug+0x44/0x70 [ 2082.665611] ? exc_invalid_op+0x17/0x60 [ 2082.669470] ? asm_exc_invalid_op+0x1a/0x20 [ 2082.673697] ? __list_add_valid_or_report+0x7d/0xa0 [ 2082.678597] ? __list_add_valid_or_report+0x7d/0xa0 [ 2082.683475] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2082.688410] ceph_add_cap+0x20b/0x620 [ceph] [ 2082.692722] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2082.697466] ? __d_lookup+0xd6/0x200 [ 2082.701050] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2082.706401] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2082.710875] ceph_con_process_message+0x74/0x150 [libceph] [ 2082.716391] ? calc_signature+0xdb/0x100 [libceph] [ 2082.721222] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2082.726572] ? process_scheduled_works+0x22a/0x510 [ 2082.731390] ? process_scheduled_works+0x22a/0x510 [ 2082.736185] ceph_con_workfn+0x344/0x750 [libceph] [ 2082.741007] ? process_scheduled_works+0x22a/0x510 [ 2082.745806] process_scheduled_works+0x2a4/0x510 [ 2082.750450] worker_thread+0x188/0x340 [ 2082.754224] ? rescuer_thread+0x310/0x310 [ 2082.758235] kthread+0xf5/0x130 [ 2082.761399] ? kthread_complete_and_exit+0x20/0x20 [ 2082.766231] ret_from_fork+0x31/0x40 [ 2082.769814] ? kthread_complete_and_exit+0x20/0x20 [ 2082.774607] ret_from_fork_asm+0x11/0x20 [ 2082.778577] [ 2082.780792] irq event stamp: 9233033 [ 2082.784372] hardirqs last enabled at (9233043): [] console_unlock+0x106/0x120 [ 2082.793170] hardirqs last disabled at (9233052): [] console_unlock+0xeb/0x120 [ 2082.801880] softirqs last enabled at (9233028): [] __do_softirq+0x2be/0x371 [ 2082.810487] softirqs last disabled at (9233023): [] irq_exit_rcu+0xb0/0xd0 [ 2082.818936] ---[ end trace 0000000000000000 ]--- [ 2082.823692] ------------[ cut here ]------------ [ 2082.828441] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2082.840240] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2082.849199] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2082.904840] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2082.913800] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2082.921276] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2082.926770] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2082.945524] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2082.950749] RAX: 0000000000000000 RBX: ffff8881d12d1390 RCX: 0000000000000000 [ 2082.957885] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2082.965014] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffece1 [ 2082.972155] R10: ffffffffffffc5e8 R11: ffffc90001fefb18 R12: ffff8881d12d1ae0 [ 2082.979295] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2082.986423] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2082.994509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2083.000256] CR2: 00007fb38ef87000 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2083.007386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2083.014520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2083.021653] Call Trace: [ 2083.024106] [ 2083.026212] ? __warn+0x85/0x180 [ 2083.029448] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.034335] ? report_bug+0xfc/0x1d0 [ 2083.037923] ? prb_read_valid+0x1b/0x20 [ 2083.041764] ? handle_bug+0x44/0x70 [ 2083.045255] ? exc_invalid_op+0x17/0x60 [ 2083.049093] ? asm_exc_invalid_op+0x1a/0x20 [ 2083.053289] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.058168] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.063048] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2083.067946] ceph_open+0x16b/0x440 [ceph] [ 2083.071979] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2083.077131] do_dentry_open+0x172/0x530 [ 2083.080974] path_openat+0x743/0x9d0 [ 2083.084564] do_filp_open+0xb8/0x120 [ 2083.088155] ? lock_release+0xc5/0x260 [ 2083.091911] ? _raw_spin_unlock+0x23/0x40 [ 2083.095929] ? alloc_fd+0x13f/0x200 [ 2083.099424] do_sys_openat2+0x249/0x290 [ 2083.103268] do_sys_open+0x4f/0x80 [ 2083.106679] do_syscall_64+0x38/0x80 [ 2083.110266] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2083.115315] RIP: 0033:0x7f48f0eed2f2 [ 2083.118895] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2083.137642] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2083.145208] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2083.152339] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2083.159475] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2083.166608] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2083.173741] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b2 [ 2083.180883] [ 2083.183074] irq event stamp: 12754185 [ 2083.186739] hardirqs last enabled at (12754195): [] console_unlock+0x106/0x120 [ 2083.195607] hardirqs last disabled at (12754204): [] console_unlock+0xeb/0x120 [ 2083.204389] softirqs last enabled at (12754218): [] __do_softirq+0x2be/0x371 [ 2083.213088] softirqs last disabled at (12754213): [] irq_exit_rcu+0xb0/0xd0 [ 2083.221614] ---[ end trace 0000000000000000 ]--- [ 2083.226238] ------------[ cut here ]------------ [ 2083.230863] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2083.242514] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2083.251217] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2083.306774] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2083.315904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2083.323384] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2083.328894] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2083.334381] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2083.353132] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2083.358362] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2083.365494] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2083.372629] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffed1b [ 2083.379770] R10: ffffffffffffd760 R11: ffffc9000030b828 R12: ffff8881c3031118 [ 2083.386906] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2083.394043] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2083.402130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2083.407876] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2083.415011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2083.422154] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2083.429290] Call Trace: [ 2083.431746] [ 2083.433853] ? __warn+0x85/0x180 [ 2083.437094] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.441982] ? report_bug+0xfc/0x1d0 [ 2083.445569] ? prb_read_valid+0x1b/0x20 [ 2083.449412] ? handle_bug+0x44/0x70 [ 2083.452912] ? exc_invalid_op+0x17/0x60 [ 2083.456759] ? asm_exc_invalid_op+0x1a/0x20 [ 2083.460965] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.465853] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.470738] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2083.475639] ceph_add_cap+0x20b/0x620 [ceph] [ 2083.479947] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2083.484692] ? __d_lookup+0xd6/0x200 [ 2083.488287] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2083.493636] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2083.498122] ceph_con_process_message+0x74/0x150 [libceph] [ 2083.503628] ? calc_signature+0xdb/0x100 [libceph] [ 2083.508453] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2083.513799] ? process_scheduled_works+0x22a/0x510 [ 2083.518605] ? process_scheduled_works+0x22a/0x510 [ 2083.523405] ceph_con_workfn+0x344/0x750 [libceph] [ 2083.528220] ? process_scheduled_works+0x22a/0x510 [ 2083.533016] process_scheduled_works+0x2a4/0x510 [ 2083.537649] worker_thread+0x188/0x340 [ 2083.541409] ? rescuer_thread+0x310/0x310 [ 2083.545428] kthread+0xf5/0x130 [ 2083.548584] ? kthread_complete_and_exit+0x20/0x20 [ 2083.553391] ret_from_fork+0x31/0x40 [ 2083.556973] ? kthread_complete_and_exit+0x20/0x20 [ 2083.561775] ret_from_fork_asm+0x11/0x20 [ 2083.565723] [ 2083.567919] irq event stamp: 9235145 [ 2083.571497] hardirqs last enabled at (9235155): [] console_unlock+0x106/0x120 [ 2083.580277] hardirqs last disabled at (9235164): [] console_unlock+0xeb/0x120 [ 2083.588973] softirqs last enabled at (9235180): [] __do_softirq+0x2be/0x371 [ 2083.597584] softirqs last disabled at (9235173): [] irq_exit_rcu+0xb0/0xd0 [ 2083.606027] ---[ end trace 0000000000000000 ]--- [ 2083.610647] ------------[ cut here ]------------ [ 2083.615270] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2083.626918] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2083.635873] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2083.691352] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2083.700312] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2083.707788] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2083.713283] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2083.732031] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2083.737258] RAX: 0000000000000000 RBX: ffff8881d12d1390 RCX: 0000000000000000 [ 2083.744391] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2083.751523] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffed56 [ 2083.758656] R10: ffffffffffffe9e0 R11: ffffc90001fefb18 R12: ffff8881d12d1ae0 [ 2083.765790] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2083.772928] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2083.781015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2083.786761] CR2: 00007f9bd4d91000 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2083.793892] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2083.801028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2083.808158] Call Trace: [ 2083.810613] [ 2083.812718] ? __warn+0x85/0x180 [ 2083.815954] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.820843] ? report_bug+0xfc/0x1d0 [ 2083.824429] ? prb_read_valid+0x1b/0x20 [ 2083.828270] ? handle_bug+0x44/0x70 [ 2083.831760] ? exc_invalid_op+0x17/0x60 [ 2083.835599] ? asm_exc_invalid_op+0x1a/0x20 [ 2083.839801] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.844680] ? __list_add_valid_or_report+0x7d/0xa0 [ 2083.849560] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2083.854464] ceph_open+0x16b/0x440 [ceph] [ 2083.858492] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2083.863647] do_dentry_open+0x172/0x530 [ 2083.867486] path_openat+0x743/0x9d0 [ 2083.871073] do_filp_open+0xb8/0x120 [ 2083.874657] ? lock_release+0xc5/0x260 [ 2083.878418] ? _raw_spin_unlock+0x23/0x40 [ 2083.882435] ? alloc_fd+0x13f/0x200 [ 2083.885936] do_sys_openat2+0x249/0x290 [ 2083.889778] do_sys_open+0x4f/0x80 [ 2083.893183] do_syscall_64+0x38/0x80 [ 2083.896761] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2083.901815] RIP: 0033:0x7f48f0eed2f2 [ 2083.905392] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2083.924138] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2083.931703] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2083.938845] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2083.945979] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2083.953115] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2083.960243] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b3 [ 2083.967389] [ 2083.969579] irq event stamp: 12756483 [ 2083.973247] hardirqs last enabled at (12756493): [] console_unlock+0x106/0x120 [ 2083.982109] hardirqs last disabled at (12756502): [] console_unlock+0xeb/0x120 [ 2083.990892] softirqs last enabled at (12756450): [] __do_softirq+0x2be/0x371 [ 2083.999585] softirqs last disabled at (12756521): [] irq_exit_rcu+0xb0/0xd0 [ 2084.008104] ---[ end trace 0000000000000000 ]--- [ 2084.012721] ------------[ cut here ]------------ [ 2084.017341] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2084.028988] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2084.037682] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2084.093161] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2084.102284] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2084.109765] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2084.115268] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2084.120754] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2084.139500] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2084.144725] RAX: 0000000000000000 RBX: ffff8881d0a3ce40 RCX: 0000000000000000 [ 2084.151859] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2084.158994] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffed90 [ 2084.166124] R10: fffffffffffffc18 R11: ffffc9000030b828 R12: ffff8881d0a3d590 [ 2084.173257] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2084.180391] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2084.188475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.194220] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2084.201345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2084.208479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2084.215611] Call Trace: [ 2084.218064] [ 2084.220171] ? __warn+0x85/0x180 [ 2084.223404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2084.228283] ? report_bug+0xfc/0x1d0 [ 2084.231863] ? prb_read_valid+0x1b/0x20 [ 2084.235702] ? handle_bug+0x44/0x70 [ 2084.239195] ? exc_invalid_op+0x17/0x60 [ 2084.243035] ? asm_exc_invalid_op+0x1a/0x20 [ 2084.247224] ? __list_add_valid_or_report+0x7d/0xa0 [ 2084.252100] ? __list_add_valid_or_report+0x7d/0xa0 [ 2084.256979] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2084.261871] ceph_add_cap+0x20b/0x620 [ceph] [ 2084.266155] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2084.270879] ? __d_lookup+0xd6/0x200 [ 2084.274464] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2084.279792] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2084.284262] ceph_con_process_message+0x74/0x150 [libceph] [ 2084.289761] ? calc_signature+0xdb/0x100 [libceph] [ 2084.294570] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2084.299900] ? process_scheduled_works+0x22a/0x510 [ 2084.304694] ? process_scheduled_works+0x22a/0x510 [ 2084.309482] ceph_con_workfn+0x344/0x750 [libceph] [ 2084.314285] ? process_scheduled_works+0x22a/0x510 [ 2084.319075] process_scheduled_works+0x2a4/0x510 [ 2084.323702] worker_thread+0x188/0x340 [ 2084.327458] ? rescuer_thread+0x310/0x310 [ 2084.331471] kthread+0xf5/0x130 [ 2084.334615] ? kthread_complete_and_exit+0x20/0x20 [ 2084.339409] ret_from_fork+0x31/0x40 [ 2084.342989] ? kthread_complete_and_exit+0x20/0x20 [ 2084.347781] ret_from_fork_asm+0x11/0x20 [ 2084.351714] [ 2084.353902] irq event stamp: 9237083 [ 2084.357480] hardirqs last enabled at (9237093): [] console_unlock+0x106/0x120 [ 2084.366260] hardirqs last disabled at (9237102): [] console_unlock+0xeb/0x120 [ 2084.374951] softirqs last enabled at (9237080): [] __do_softirq+0x2be/0x371 [ 2084.383558] softirqs last disabled at (9237075): [] irq_exit_rcu+0xb0/0xd0 [ 2084.391990] ---[ end trace 0000000000000000 ]--- [ 2084.396610] ------------[ cut here ]------------ [ 2084.401234] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2084.412876] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2084.421831] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2084.477299] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2084.486250] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2084.493728] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2084.499215] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2084.517962] RSP: 0018:ffffc90001fefbd0 EFLAGS: 00010282 [ 2084.523187] RAX: 0000000000000000 RBX: ffff8881d12d1390 RCX: 0000000000000000 [ 2084.530311] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2084.537435] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffedcb [ 2084.544570] R10: 0000000000000db0 R11: ffffc90001fefa60 R12: ffff8881d12d1ae0 [ 2084.551705] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2084.558835] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2084.566921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.572666] CR2: 00007f9bd4a29000 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2084.579791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2084.586924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2084.594056] Call Trace: [ 2084.596499] [ 2084.598598] ? __warn+0x85/0x180 [ 2084.601832] ? __list_add_valid_or_report+0x7d/0xa0 [ 2084.606712] ? report_bug+0xfc/0x1d0 [ 2084.610292] ? prb_read_valid+0x1b/0x20 [ 2084.614132] ? handle_bug+0x44/0x70 [ 2084.617622] ? exc_invalid_op+0x17/0x60 [ 2084.621462] ? asm_exc_invalid_op+0x1a/0x20 [ 2084.625652] ? __list_add_valid_or_report+0x7d/0xa0 [ 2084.630528] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2084.635416] try_get_cap_refs+0x433/0x880 [ceph] [ 2084.640048] __ceph_get_caps+0xf8/0x760 [ceph] [ 2084.644517] ? ceph_read_iter+0x134/0x860 [ceph] [ 2084.649147] ceph_read_iter+0x134/0x860 [ceph] [ 2084.653603] ? selinux_file_permission+0x119/0x150 [ 2084.658393] ? vfs_read+0x20f/0x300 [ 2084.661884] vfs_read+0x20f/0x300 [ 2084.665209] ksys_pread64+0x65/0xa0 [ 2084.668705] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2084.673846] do_syscall_64+0x38/0x80 [ 2084.677424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2084.682475] RIP: 0033:0x7f48f0eeb88b [ 2084.686056] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2084.704802] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2084.712365] RAX: ffffffffffffffda RBX: 0000000002075910 RCX: 00007f48f0eeb88b [ 2084.719492] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000013 [ 2084.726624] RBP: 00000000000029b3 R08: 0000000000000063 R09: 0000000000000003 [ 2084.733756] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2084.740890] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183df18 [ 2084.748029] [ 2084.750214] irq event stamp: 12758567 [ 2084.753873] hardirqs last enabled at (12758577): [] console_unlock+0x106/0x120 [ 2084.762739] hardirqs last disabled at (12758586): [] console_unlock+0xeb/0x120 [ 2084.771518] softirqs last enabled at (12758544): [] __do_softirq+0x2be/0x371 [ 2084.780211] softirqs last disabled at (12758539): [] irq_exit_rcu+0xb0/0xd0 [ 2084.788731] ---[ end trace 0000000000000000 ]--- [ 2084.793352] ------------[ cut here ]------------ [ 2084.797984] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2084.809631] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2084.818592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2084.874112] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2084.883062] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2084.890543] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2084.896028] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2084.914780] RSP: 0018:ffffc90001fd7c88 EFLAGS: 00010286 [ 2084.920009] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2084.927142] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2084.934277] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffee03 [ 2084.941408] R10: 0000000000001de8 R11: ffffc90001fd7b18 R12: ffff8881c3031118 [ 2084.948540] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2084.955675] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2084.963765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2084.969515] CR2: 00007f9bd4cdb000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2084.976647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2084.983780] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2084.990915] Call Trace: [ 2084.993366] [ 2084.995472] ? __warn+0x85/0x180 [ 2084.998707] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.003587] ? report_bug+0xfc/0x1d0 [ 2085.007164] ? prb_read_valid+0x1b/0x20 [ 2085.011007] ? handle_bug+0x44/0x70 [ 2085.014496] ? exc_invalid_op+0x17/0x60 [ 2085.018338] ? asm_exc_invalid_op+0x1a/0x20 [ 2085.022532] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.027412] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.032296] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2085.037197] ceph_open+0x16b/0x440 [ceph] [ 2085.041228] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2085.046382] do_dentry_open+0x172/0x530 [ 2085.050224] path_openat+0x743/0x9d0 [ 2085.053822] do_filp_open+0xb8/0x120 [ 2085.057412] ? lock_release+0xc5/0x260 [ 2085.061171] ? _raw_spin_unlock+0x23/0x40 [ 2085.065189] ? alloc_fd+0x13f/0x200 [ 2085.068686] do_sys_openat2+0x249/0x290 [ 2085.072533] do_sys_open+0x4f/0x80 [ 2085.075945] do_syscall_64+0x38/0x80 [ 2085.079525] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2085.084575] RIP: 0033:0x7f48f0eed2f2 [ 2085.088156] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2085.106901] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2085.114468] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2085.121600] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2085.128732] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2085.135865] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2085.143000] R13: 00007f48f183d8a0 R14: 00000000020767c0 R15: 0000000000002a12 [ 2085.150145] [ 2085.152333] irq event stamp: 12861669 [ 2085.155999] hardirqs last enabled at (12861679): [] console_unlock+0x106/0x120 [ 2085.164865] hardirqs last disabled at (12861688): [] console_unlock+0xeb/0x120 [ 2085.173648] softirqs last enabled at (12861702): [] __do_softirq+0x2be/0x371 [ 2085.182345] softirqs last disabled at (12861697): [] irq_exit_rcu+0xb0/0xd0 [ 2085.190865] ---[ end trace 0000000000000000 ]--- [ 2085.195486] ------------[ cut here ]------------ [ 2085.200107] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2085.211752] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2085.220456] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2085.276005] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2085.285136] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2085.292616] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2085.298127] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2085.303613] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2085.322361] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2085.327594] RAX: 0000000000000000 RBX: ffff8881d807e1d0 RCX: 0000000000000000 [ 2085.334728] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2085.341862] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffee3d [ 2085.348993] R10: 0000000000003078 R11: ffffc9000030b828 R12: ffff8881d807e920 [ 2085.356126] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2085.363260] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2085.371346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2085.377089] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2085.384226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2085.391365] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2085.398498] Call Trace: [ 2085.400952] [ 2085.403058] ? __warn+0x85/0x180 [ 2085.406293] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.411180] ? report_bug+0xfc/0x1d0 [ 2085.414767] ? prb_read_valid+0x1b/0x20 [ 2085.418618] ? handle_bug+0x44/0x70 [ 2085.422117] ? exc_invalid_op+0x17/0x60 [ 2085.425957] ? asm_exc_invalid_op+0x1a/0x20 [ 2085.430162] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.435051] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.439936] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2085.444836] ceph_add_cap+0x20b/0x620 [ceph] [ 2085.449144] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2085.453896] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2085.459250] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2085.463735] ceph_con_process_message+0x74/0x150 [libceph] [ 2085.469239] ? calc_signature+0xdb/0x100 [libceph] [ 2085.474060] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2085.479401] ? process_scheduled_works+0x22a/0x510 [ 2085.484203] ? process_scheduled_works+0x22a/0x510 [ 2085.488996] ceph_con_workfn+0x344/0x750 [libceph] [ 2085.493813] ? process_scheduled_works+0x22a/0x510 [ 2085.498608] process_scheduled_works+0x2a4/0x510 [ 2085.503242] worker_thread+0x188/0x340 [ 2085.507003] ? rescuer_thread+0x310/0x310 [ 2085.511022] kthread+0xf5/0x130 [ 2085.514177] ? kthread_complete_and_exit+0x20/0x20 [ 2085.518979] ret_from_fork+0x31/0x40 [ 2085.522564] ? kthread_complete_and_exit+0x20/0x20 [ 2085.527358] ret_from_fork_asm+0x11/0x20 [ 2085.531300] [ 2085.533495] irq event stamp: 9239905 [ 2085.537071] hardirqs last enabled at (9239915): [] console_unlock+0x106/0x120 [ 2085.545853] hardirqs last disabled at (9239924): [] console_unlock+0xeb/0x120 [ 2085.554552] softirqs last enabled at (9239902): [] __do_softirq+0x2be/0x371 [ 2085.563167] softirqs last disabled at (9239897): [] irq_exit_rcu+0xb0/0xd0 [ 2085.571600] ---[ end trace 0000000000000000 ]--- [ 2085.576220] ------------[ cut here ]------------ [ 2085.580837] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2085.592484] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2085.601439] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2085.656961] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2085.665912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2085.673390] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2085.678878] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2085.697629] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2085.702858] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2085.709990] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2085.717124] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffee77 [ 2085.724257] R10: 00000000000042b0 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2085.731392] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2085.738522] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2085.746608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2085.752355] CR2: 00007f9bd4cdb000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2085.759487] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2085.766635] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2085.773772] Call Trace: [ 2085.776223] [ 2085.778330] ? __warn+0x85/0x180 [ 2085.781564] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.786444] ? report_bug+0xfc/0x1d0 [ 2085.790029] ? prb_read_valid+0x1b/0x20 [ 2085.793874] ? handle_bug+0x44/0x70 [ 2085.797371] ? exc_invalid_op+0x17/0x60 [ 2085.801210] ? asm_exc_invalid_op+0x1a/0x20 [ 2085.805405] ? __list_add_valid_or_report+0x7d/0xa0 [ 2085.810288] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2085.815194] try_get_cap_refs+0x433/0x880 [ceph] [ 2085.819847] __ceph_get_caps+0xf8/0x760 [ceph] [ 2085.824326] ? ceph_read_iter+0x134/0x860 [ceph] [ 2085.828961] ceph_read_iter+0x134/0x860 [ceph] [ 2085.833434] ? selinux_file_permission+0x119/0x150 [ 2085.838231] ? vfs_read+0x20f/0x300 [ 2085.841728] vfs_read+0x20f/0x300 [ 2085.845058] ksys_pread64+0x65/0xa0 [ 2085.848558] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2085.853700] do_syscall_64+0x38/0x80 [ 2085.857285] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2085.862336] RIP: 0033:0x7f48f0eeb88b [ 2085.865915] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2085.884663] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2085.892228] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2085.899361] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2085.906494] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2085.913626] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2085.920760] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2085.927906] [ 2085.930103] irq event stamp: 12863485 [ 2085.933768] hardirqs last enabled at (12863495): [] console_unlock+0x106/0x120 [ 2085.942635] hardirqs last disabled at (12863504): [] console_unlock+0xeb/0x120 [ 2085.951413] softirqs last enabled at (12863166): [] __do_softirq+0x2be/0x371 [ 2085.960107] softirqs last disabled at (12863155): [] irq_exit_rcu+0xb0/0xd0 [ 2085.968625] ---[ end trace 0000000000000000 ]--- [ 2085.973246] ------------[ cut here ]------------ [ 2085.977868] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2085.989523] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2085.998226] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2086.053773] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2086.062907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2086.070385] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2086.075897] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2086.081384] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2086.100130] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2086.105357] RAX: 0000000000000000 RBX: ffff8881f789c478 RCX: 0000000000000000 [ 2086.112488] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2086.119624] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeeaf [ 2086.126764] R10: 0000000000005340 R11: ffffc9000030b828 R12: ffff8881f789cbc8 [ 2086.133896] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2086.141030] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2086.149116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2086.154860] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2086.161995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2086.169128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2086.176260] Call Trace: [ 2086.178712] [ 2086.180818] ? __warn+0x85/0x180 [ 2086.184055] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.188941] ? report_bug+0xfc/0x1d0 [ 2086.192527] ? prb_read_valid+0x1b/0x20 [ 2086.196371] ? handle_bug+0x44/0x70 [ 2086.199868] ? exc_invalid_op+0x17/0x60 [ 2086.203708] ? asm_exc_invalid_op+0x1a/0x20 [ 2086.207906] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.212794] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.217679] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2086.222580] ceph_add_cap+0x20b/0x620 [ceph] [ 2086.226888] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2086.231622] ? __d_lookup+0xd6/0x200 [ 2086.235211] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2086.240561] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2086.245048] ceph_con_process_message+0x74/0x150 [libceph] [ 2086.250554] ? calc_signature+0xdb/0x100 [libceph] [ 2086.255377] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2086.260717] ? process_scheduled_works+0x22a/0x510 [ 2086.265516] ? process_scheduled_works+0x22a/0x510 [ 2086.270312] ceph_con_workfn+0x344/0x750 [libceph] [ 2086.275126] ? process_scheduled_works+0x22a/0x510 [ 2086.279925] process_scheduled_works+0x2a4/0x510 [ 2086.284563] worker_thread+0x188/0x340 [ 2086.288325] ? rescuer_thread+0x310/0x310 [ 2086.292343] kthread+0xf5/0x130 [ 2086.295490] ? kthread_complete_and_exit+0x20/0x20 [ 2086.300292] ret_from_fork+0x31/0x40 [ 2086.303879] ? kthread_complete_and_exit+0x20/0x20 [ 2086.308683] ret_from_fork_asm+0x11/0x20 [ 2086.312629] [ 2086.314826] irq event stamp: 9241805 [ 2086.318404] hardirqs last enabled at (9241815): [] console_unlock+0x106/0x120 [ 2086.327185] hardirqs last disabled at (9241824): [] console_unlock+0xeb/0x120 [ 2086.335884] softirqs last enabled at (9241802): [] __do_softirq+0x2be/0x371 [ 2086.344492] softirqs last disabled at (9241797): [] irq_exit_rcu+0xb0/0xd0 [ 2086.352934] ---[ end trace 0000000000000000 ]--- [ 2086.357554] ------------[ cut here ]------------ [ 2086.362171] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2086.373817] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2086.382773] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2086.438301] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2086.447253] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2086.454731] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2086.460219] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2086.478969] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2086.484199] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2086.491332] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2086.498464] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffeeea [ 2086.505598] R10: 0000000000006508 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2086.512729] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2086.519864] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2086.527948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2086.533695] CR2: 00007f9bd4cdb000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2086.540828] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2086.547960] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2086.555095] Call Trace: [ 2086.557546] [ 2086.559654] ? __warn+0x85/0x180 [ 2086.562885] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.567765] ? report_bug+0xfc/0x1d0 [ 2086.571343] ? prb_read_valid+0x1b/0x20 [ 2086.575188] ? handle_bug+0x44/0x70 [ 2086.578684] ? exc_invalid_op+0x17/0x60 [ 2086.582525] ? asm_exc_invalid_op+0x1a/0x20 [ 2086.586721] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.591600] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2086.596500] try_get_cap_refs+0x433/0x880 [ceph] [ 2086.601153] __ceph_get_caps+0xf8/0x760 [ceph] [ 2086.605633] ? ceph_read_iter+0x134/0x860 [ceph] [ 2086.610271] ceph_read_iter+0x134/0x860 [ceph] [ 2086.614739] ? selinux_file_permission+0x119/0x150 [ 2086.619538] ? vfs_read+0x20f/0x300 [ 2086.623034] vfs_read+0x20f/0x300 [ 2086.626363] ksys_pread64+0x65/0xa0 [ 2086.629854] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2086.634997] do_syscall_64+0x38/0x80 [ 2086.638583] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2086.643634] RIP: 0033:0x7f48f0eeb88b [ 2086.647213] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2086.665960] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2086.673526] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2086.680658] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2086.687792] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2086.694925] R10: 0000000000001000 R11: 0000000000000246 R12: 0000000000001000 [ 2086.702056] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2086.709202] [ 2086.711391] irq event stamp: 12865447 [ 2086.715056] hardirqs last enabled at (12865457): [] console_unlock+0x106/0x120 [ 2086.723922] hardirqs last disabled at (12865466): [] console_unlock+0xeb/0x120 [ 2086.732704] softirqs last enabled at (12865268): [] __do_softirq+0x2be/0x371 [ 2086.741400] softirqs last disabled at (12865485): [] irq_exit_rcu+0xb0/0xd0 [ 2086.749923] ---[ end trace 0000000000000000 ]--- [ 2086.754545] ------------[ cut here ]------------ [ 2086.759163] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2086.770811] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2086.779515] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2086.835062] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2086.844196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2086.851674] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2086.857184] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2086.862670] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2086.881419] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2086.886654] RAX: 0000000000000000 RBX: ffff8881d807f560 RCX: 0000000000000000 [ 2086.893785] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2086.900921] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffef22 [ 2086.908061] R10: 00000000000076a0 R11: ffffc9000030b828 R12: ffff8881d807fcb0 [ 2086.915194] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2086.922326] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2086.930412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2086.936158] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2086.943302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2086.950441] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2086.957575] Call Trace: [ 2086.960027] [ 2086.962133] ? __warn+0x85/0x180 [ 2086.965368] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.970256] ? report_bug+0xfc/0x1d0 [ 2086.973842] ? prb_read_valid+0x1b/0x20 [ 2086.977693] ? handle_bug+0x44/0x70 [ 2086.981192] ? exc_invalid_op+0x17/0x60 [ 2086.985031] ? asm_exc_invalid_op+0x1a/0x20 [ 2086.989237] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.994128] ? __list_add_valid_or_report+0x7d/0xa0 [ 2086.999010] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2087.003913] ceph_add_cap+0x20b/0x620 [ceph] [ 2087.008219] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2087.012961] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2087.018313] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2087.022800] ceph_con_process_message+0x74/0x150 [libceph] [ 2087.028306] ? calc_signature+0xdb/0x100 [libceph] [ 2087.033127] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2087.038469] ? process_scheduled_works+0x22a/0x510 [ 2087.043267] ? process_scheduled_works+0x22a/0x510 [ 2087.048064] ceph_con_workfn+0x344/0x750 [libceph] [ 2087.052878] ? process_scheduled_works+0x22a/0x510 [ 2087.057676] process_scheduled_works+0x2a4/0x510 [ 2087.062317] worker_thread+0x188/0x340 [ 2087.066077] ? rescuer_thread+0x310/0x310 [ 2087.070096] kthread+0xf5/0x130 [ 2087.073242] ? kthread_complete_and_exit+0x20/0x20 [ 2087.078046] ret_from_fork+0x31/0x40 [ 2087.081631] ? kthread_complete_and_exit+0x20/0x20 [ 2087.086424] ret_from_fork_asm+0x11/0x20 [ 2087.090366] [ 2087.092561] irq event stamp: 9243679 [ 2087.096139] hardirqs last enabled at (9243689): [] console_unlock+0x106/0x120 [ 2087.104918] hardirqs last disabled at (9243698): [] console_unlock+0xeb/0x120 [ 2087.113612] softirqs last enabled at (9243676): [] __do_softirq+0x2be/0x371 [ 2087.122227] softirqs last disabled at (9243671): [] irq_exit_rcu+0xb0/0xd0 [ 2087.130668] ---[ end trace 0000000000000000 ]--- [ 2087.135288] ------------[ cut here ]------------ [ 2087.139914] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2087.151558] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2087.160512] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2087.215974] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2087.224924] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2087.232404] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2087.237889] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2087.256634] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2087.261867] RAX: 0000000000000000 RBX: ffff8881d12d2720 RCX: 0000000000000000 [ 2087.268993] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2087.276118] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffef5c [ 2087.283253] R10: 0000000000008880 R11: ffffc90001fefb18 R12: ffff8881d12d2e70 [ 2087.290383] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2087.297508] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2087.305594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2087.311339] CR2: 00007f9bd4a29000 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2087.318464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2087.325598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2087.332730] Call Trace: [ 2087.335174] [ 2087.337281] ? __warn+0x85/0x180 [ 2087.340512] ? __list_add_valid_or_report+0x7d/0xa0 [ 2087.345393] ? report_bug+0xfc/0x1d0 [ 2087.348974] ? prb_read_valid+0x1b/0x20 [ 2087.352813] ? handle_bug+0x44/0x70 [ 2087.356304] ? exc_invalid_op+0x17/0x60 [ 2087.360144] ? asm_exc_invalid_op+0x1a/0x20 [ 2087.364335] ? __list_add_valid_or_report+0x7d/0xa0 [ 2087.369210] ? __list_add_valid_or_report+0x7d/0xa0 [ 2087.374090] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2087.378978] ceph_open+0x16b/0x440 [ceph] [ 2087.383000] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2087.388147] do_dentry_open+0x172/0x530 [ 2087.391986] path_openat+0x743/0x9d0 [ 2087.395568] do_filp_open+0xb8/0x120 [ 2087.399148] ? lock_release+0xc5/0x260 [ 2087.402900] ? _raw_spin_unlock+0x23/0x40 [ 2087.406917] ? alloc_fd+0x13f/0x200 [ 2087.410414] do_sys_openat2+0x249/0x290 [ 2087.414255] do_sys_open+0x4f/0x80 [ 2087.417658] do_syscall_64+0x38/0x80 [ 2087.421236] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2087.426289] RIP: 0033:0x7f48f0eed2f2 [ 2087.429868] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2087.448614] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2087.456181] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2087.463315] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2087.470446] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2087.477571] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2087.484704] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b4 [ 2087.491842] [ 2087.494028] irq event stamp: 12765049 [ 2087.497686] hardirqs last enabled at (12765059): [] console_unlock+0x106/0x120 [ 2087.506553] hardirqs last disabled at (12765068): [] console_unlock+0xeb/0x120 [ 2087.515330] softirqs last enabled at (12765016): [] __do_softirq+0x2be/0x371 [ 2087.524023] softirqs last disabled at (12765011): [] irq_exit_rcu+0xb0/0xd0 [ 2087.532545] ---[ end trace 0000000000000000 ]--- [ 2087.537166] ------------[ cut here ]------------ [ 2087.541804] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2087.553461] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2087.562424] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2087.617950] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2087.626903] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2087.634382] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2087.639868] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2087.658615] RSP: 0018:ffffc90001fd7c28 EFLAGS: 00010286 [ 2087.663841] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2087.670973] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2087.678105] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffef96 [ 2087.685239] R10: 0000000000009ab0 R11: ffffc90001fd7ab8 R12: ffff8881c3031118 [ 2087.692371] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2087.699506] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2087.707590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2087.713338] CR2: 00007f9bd4cdb000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2087.720469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2087.727602] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2087.734737] Call Trace: [ 2087.737188] [ 2087.739295] ? __warn+0x85/0x180 [ 2087.742529] ? __list_add_valid_or_report+0x7d/0xa0 [ 2087.747407] ? report_bug+0xfc/0x1d0 [ 2087.750986] ? prb_read_valid+0x1b/0x20 [ 2087.754828] ? handle_bug+0x44/0x70 [ 2087.758321] ? exc_invalid_op+0x17/0x60 [ 2087.762161] ? asm_exc_invalid_op+0x1a/0x20 [ 2087.766367] ? __list_add_valid_or_report+0x7d/0xa0 [ 2087.771267] ? __list_add_valid_or_report+0x7d/0xa0 [ 2087.776149] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2087.781056] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2087.786220] ? __ceph_caps_issued+0x78/0x110 [ceph] [ 2087.791119] __ceph_setattr+0x2c0/0xea0 [ceph] [ 2087.795590] ceph_setattr+0xaa/0x100 [ceph] [ 2087.799793] notify_change+0x237/0x4f0 [ 2087.803559] ? vfs_utimes+0x130/0x220 [ 2087.807234] vfs_utimes+0x130/0x220 [ 2087.810738] do_utimes+0xb7/0x120 [ 2087.814068] __x64_sys_utime+0xa1/0xd0 [ 2087.817831] do_syscall_64+0x38/0x80 [ 2087.821415] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2087.826474] RIP: 0033:0x7f48f0eec99b [ 2087.830055] Code: 64 48 33 0c 25 28 00 00 00 75 06 48 83 c4 10 5b c3 e8 79 09 02 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 84 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d bd 24 2d 00 f7 d8 64 89 01 48 [ 2087.848800] RSP: 002b:00007ffc53d5fff8 EFLAGS: 00000206 ORIG_RAX: 0000000000000084 [ 2087.856374] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eec99b [ 2087.863507] RDX: 00007ffc53d60010 RSI: 00007ffc53d60000 RDI: 000000000207dfc0 [ 2087.870641] RBP: 00007f48f183d8a0 R08: 0000000002076605 R09: 0000000000000000 [ 2087.877772] R10: 00007f48f0f711e0 R11: 0000000000000206 R12: 0000000000000004 [ 2087.884905] R13: 00000000000003ec R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2087.892052] [ 2087.894248] irq event stamp: 12868243 [ 2087.897914] hardirqs last enabled at (12868253): [] console_unlock+0x106/0x120 [ 2087.906780] hardirqs last disabled at (12868262): [] console_unlock+0xeb/0x120 [ 2087.915561] softirqs last enabled at (12868024): [] __do_softirq+0x2be/0x371 [ 2087.924252] softirqs last disabled at (12868019): [] irq_exit_rcu+0xb0/0xd0 [ 2087.932772] ---[ end trace 0000000000000000 ]--- [ 2087.937393] ------------[ cut here ]------------ [ 2087.942009] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2087.953660] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2087.962366] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2088.017848] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2088.026976] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2088.034461] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2088.039972] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2088.045458] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2088.064206] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2088.069439] RAX: 0000000000000000 RBX: ffff8881c3fc5808 RCX: 0000000000000000 [ 2088.076574] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2088.083706] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffefcf [ 2088.090844] R10: 000000000000ab98 R11: ffffc9000030b828 R12: ffff8881c3fc5f58 [ 2088.097979] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2088.105113] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2088.113198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2088.118943] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2088.126078] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2088.133210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2088.140343] Call Trace: [ 2088.142797] [ 2088.144903] ? __warn+0x85/0x180 [ 2088.148136] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.153026] ? report_bug+0xfc/0x1d0 [ 2088.156613] ? prb_read_valid+0x1b/0x20 [ 2088.160462] ? handle_bug+0x44/0x70 [ 2088.163962] ? exc_invalid_op+0x17/0x60 [ 2088.167799] ? asm_exc_invalid_op+0x1a/0x20 [ 2088.171998] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.176885] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.181771] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2088.186671] ceph_add_cap+0x20b/0x620 [ceph] [ 2088.190979] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2088.195709] ? __d_lookup+0xd6/0x200 [ 2088.199304] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2088.204641] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2088.209120] ceph_con_process_message+0x74/0x150 [libceph] [ 2088.214619] ? calc_signature+0xdb/0x100 [libceph] [ 2088.219448] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2088.224782] ? process_scheduled_works+0x22a/0x510 [ 2088.229584] ? process_scheduled_works+0x22a/0x510 [ 2088.234379] ceph_con_workfn+0x344/0x750 [libceph] [ 2088.239194] ? process_scheduled_works+0x22a/0x510 [ 2088.243988] process_scheduled_works+0x2a4/0x510 [ 2088.248619] worker_thread+0x188/0x340 [ 2088.252373] ? rescuer_thread+0x310/0x310 [ 2088.256394] kthread+0xf5/0x130 [ 2088.259546] ? kthread_complete_and_exit+0x20/0x20 [ 2088.264344] ret_from_fork+0x31/0x40 [ 2088.267926] ? kthread_complete_and_exit+0x20/0x20 [ 2088.272725] ret_from_fork_asm+0x11/0x20 [ 2088.276667] [ 2088.278865] irq event stamp: 9246573 [ 2088.282446] hardirqs last enabled at (9246589): [] console_unlock+0x106/0x120 [ 2088.291234] hardirqs last disabled at (9246598): [] console_unlock+0xeb/0x120 [ 2088.299934] softirqs last enabled at (9246586): [] __do_softirq+0x2be/0x371 [ 2088.308549] softirqs last disabled at (9246581): [] irq_exit_rcu+0xb0/0xd0 [ 2088.316980] ---[ end trace 0000000000000000 ]--- [ 2088.321602] ------------[ cut here ]------------ [ 2088.326218] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2088.337867] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2088.346827] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2088.402360] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2088.411312] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2088.418790] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2088.424275] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2088.443027] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2088.448259] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2088.455390] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2088.462523] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff00a [ 2088.469656] R10: 000000000000bd18 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2088.476790] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2088.483921] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2088.492008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2088.497754] CR2: 00007f9bd4dcb018 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2088.504886] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2088.512020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2088.519153] Call Trace: [ 2088.521605] [ 2088.523712] ? __warn+0x85/0x180 [ 2088.526945] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.531825] ? report_bug+0xfc/0x1d0 [ 2088.535402] ? prb_read_valid+0x1b/0x20 [ 2088.539246] ? handle_bug+0x44/0x70 [ 2088.542745] ? exc_invalid_op+0x17/0x60 [ 2088.546583] ? asm_exc_invalid_op+0x1a/0x20 [ 2088.550780] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.555660] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2088.560559] try_get_cap_refs+0x433/0x880 [ceph] [ 2088.565212] __ceph_get_caps+0xf8/0x760 [ceph] [ 2088.569690] ? ceph_read_iter+0x134/0x860 [ceph] [ 2088.574326] ceph_read_iter+0x134/0x860 [ceph] [ 2088.578798] ? selinux_file_permission+0x119/0x150 [ 2088.583595] ? vfs_read+0x20f/0x300 [ 2088.587092] vfs_read+0x20f/0x300 [ 2088.590422] ksys_pread64+0x65/0xa0 [ 2088.593913] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2088.599056] do_syscall_64+0x38/0x80 [ 2088.602641] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2088.607693] RIP: 0033:0x7f48f0eeb88b [ 2088.611272] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2088.630017] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2088.637583] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2088.644718] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2088.651849] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2088.658982] R10: 0000000000002000 R11: 0000000000000246 R12: 0000000000001000 [ 2088.666116] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2088.673261] [ 2088.675450] irq event stamp: 12870435 [ 2088.679115] hardirqs last enabled at (12870445): [] console_unlock+0x106/0x120 [ 2088.687982] hardirqs last disabled at (12870454): [] console_unlock+0xeb/0x120 [ 2088.696762] softirqs last enabled at (12870000): [] __do_softirq+0x2be/0x371 [ 2088.705457] softirqs last disabled at (12870473): [] irq_exit_rcu+0xb0/0xd0 [ 2088.713982] ---[ end trace 0000000000000000 ]--- [ 2088.718603] ------------[ cut here ]------------ [ 2088.723219] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2088.734864] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2088.743821] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2088.799292] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2088.808256] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2088.815730] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2088.821215] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2088.839962] RSP: 0018:ffffc90001fefbd0 EFLAGS: 00010282 [ 2088.845187] RAX: 0000000000000000 RBX: ffff8881d12d2720 RCX: 0000000000000000 [ 2088.852320] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2088.859453] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff042 [ 2088.866586] R10: 000000000000ceb0 R11: ffffc90001fefa60 R12: ffff8881d12d2e70 [ 2088.873719] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2088.880852] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2088.888937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2088.894683] CR2: 00007f9bd4d91000 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2088.901816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2088.908949] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2088.916081] Call Trace: [ 2088.918536] [ 2088.920642] ? __warn+0x85/0x180 [ 2088.923874] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.928753] ? report_bug+0xfc/0x1d0 [ 2088.932334] ? prb_read_valid+0x1b/0x20 [ 2088.936176] ? handle_bug+0x44/0x70 [ 2088.939664] ? exc_invalid_op+0x17/0x60 [ 2088.943505] ? asm_exc_invalid_op+0x1a/0x20 [ 2088.947693] ? __list_add_valid_or_report+0x7d/0xa0 [ 2088.952570] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2088.957461] try_get_cap_refs+0x433/0x880 [ceph] [ 2088.962091] __ceph_get_caps+0xf8/0x760 [ceph] [ 2088.966557] ? ceph_read_iter+0x134/0x860 [ceph] [ 2088.971186] ceph_read_iter+0x134/0x860 [ceph] [ 2088.975646] ? selinux_file_permission+0x119/0x150 [ 2088.980435] ? vfs_read+0x20f/0x300 [ 2088.983926] vfs_read+0x20f/0x300 [ 2088.987250] ksys_pread64+0x65/0xa0 [ 2088.990736] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2088.995881] do_syscall_64+0x38/0x80 [ 2088.999464] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2089.004517] RIP: 0033:0x7f48f0eeb88b [ 2089.008096] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2089.026843] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2089.034409] RAX: ffffffffffffffda RBX: 0000000002075920 RCX: 00007f48f0eeb88b [ 2089.041542] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000014 [ 2089.048666] RBP: 00000000000029b4 R08: 0000000000000063 R09: 0000000000000003 [ 2089.055799] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2089.062933] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183df18 [ 2089.070072] [ 2089.072257] irq event stamp: 12769319 [ 2089.075914] hardirqs last enabled at (12769329): [] console_unlock+0x106/0x120 [ 2089.084781] hardirqs last disabled at (12769338): [] console_unlock+0xeb/0x120 [ 2089.093561] softirqs last enabled at (12769296): [] __do_softirq+0x2be/0x371 [ 2089.102254] softirqs last disabled at (12769291): [] irq_exit_rcu+0xb0/0xd0 [ 2089.110774] ---[ end trace 0000000000000000 ]--- [ 2089.115394] ------------[ cut here ]------------ [ 2089.120018] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2089.131663] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2089.140358] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2089.195843] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2089.204978] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2089.212456] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2089.217970] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2089.223462] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2089.242210] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2089.247443] RAX: 0000000000000000 RBX: ffff8881d12d89c8 RCX: 0000000000000000 [ 2089.254579] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2089.261721] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff07a [ 2089.268860] R10: 000000000000e040 R11: ffffc9000030b828 R12: ffff8881d12d9118 [ 2089.275989] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2089.283127] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2089.291219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2089.296969] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2089.304106] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2089.311241] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2089.318371] Call Trace: [ 2089.320828] [ 2089.322931] ? __warn+0x85/0x180 [ 2089.326169] ? __list_add_valid_or_report+0x7d/0xa0 [ 2089.331053] ? report_bug+0xfc/0x1d0 [ 2089.334633] ? prb_read_valid+0x1b/0x20 [ 2089.338482] ? handle_bug+0x44/0x70 [ 2089.341973] ? exc_invalid_op+0x17/0x60 [ 2089.345815] ? asm_exc_invalid_op+0x1a/0x20 [ 2089.350018] ? __list_add_valid_or_report+0x7d/0xa0 [ 2089.354906] ? __list_add_valid_or_report+0x7d/0xa0 [ 2089.359793] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2089.364693] ceph_add_cap+0x20b/0x620 [ceph] [ 2089.368999] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2089.373731] ? __d_lookup+0xd6/0x200 [ 2089.377323] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2089.382663] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2089.387151] ceph_con_process_message+0x74/0x150 [libceph] [ 2089.392652] ? calc_signature+0xdb/0x100 [libceph] [ 2089.397474] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2089.402812] ? process_scheduled_works+0x22a/0x510 [ 2089.407615] ? process_scheduled_works+0x22a/0x510 [ 2089.412407] ceph_con_workfn+0x344/0x750 [libceph] [ 2089.417224] ? process_scheduled_works+0x22a/0x510 [ 2089.422019] process_scheduled_works+0x2a4/0x510 [ 2089.426651] worker_thread+0x188/0x340 [ 2089.430410] ? rescuer_thread+0x310/0x310 [ 2089.434423] kthread+0xf5/0x130 [ 2089.437575] ? kthread_complete_and_exit+0x20/0x20 [ 2089.442373] ret_from_fork+0x31/0x40 [ 2089.445956] ? kthread_complete_and_exit+0x20/0x20 [ 2089.450752] ret_from_fork_asm+0x11/0x20 [ 2089.454697] [ 2089.456895] irq event stamp: 9249331 [ 2089.460472] hardirqs last enabled at (9249341): [] console_unlock+0x106/0x120 [ 2089.469255] hardirqs last disabled at (9249350): [] console_unlock+0xeb/0x120 [ 2089.477955] softirqs last enabled at (9249326): [] __do_softirq+0x2be/0x371 [ 2089.486561] softirqs last disabled at (9249321): [] irq_exit_rcu+0xb0/0xd0 [ 2089.494993] ---[ end trace 0000000000000000 ]--- [ 2089.499617] ------------[ cut here ]------------ [ 2089.504246] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2089.515903] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2089.524858] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2089.580329] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2089.589279] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2089.596757] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2089.602243] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2089.620990] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2089.626215] RAX: 0000000000000000 RBX: ffff8881d12d3ab0 RCX: 0000000000000000 [ 2089.633348] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2089.640481] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff0b5 [ 2089.647615] R10: 000000000000f1c8 R11: ffffc90001fefb18 R12: ffff8881d12d4200 [ 2089.654746] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2089.661878] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2089.669965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2089.675711] CR2: 00007f9bd4a29000 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2089.682844] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2089.689977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2089.697109] Call Trace: [ 2089.699553] [ 2089.701652] ? __warn+0x85/0x180 [ 2089.704883] ? __list_add_valid_or_report+0x7d/0xa0 [ 2089.709764] ? report_bug+0xfc/0x1d0 [ 2089.713345] ? prb_read_valid+0x1b/0x20 [ 2089.717185] ? handle_bug+0x44/0x70 [ 2089.720676] ? exc_invalid_op+0x17/0x60 [ 2089.724514] ? asm_exc_invalid_op+0x1a/0x20 [ 2089.728706] ? __list_add_valid_or_report+0x7d/0xa0 [ 2089.733581] ? __list_add_valid_or_report+0x7d/0xa0 [ 2089.738458] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2089.743350] ceph_open+0x16b/0x440 [ceph] [ 2089.747371] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2089.752522] do_dentry_open+0x172/0x530 [ 2089.756367] path_openat+0x743/0x9d0 [ 2089.759948] do_filp_open+0xb8/0x120 [ 2089.763535] ? lock_release+0xc5/0x260 [ 2089.767294] ? _raw_spin_unlock+0x23/0x40 [ 2089.771308] ? alloc_fd+0x13f/0x200 [ 2089.774809] do_sys_openat2+0x249/0x290 [ 2089.778656] do_sys_open+0x4f/0x80 [ 2089.782069] do_syscall_64+0x38/0x80 [ 2089.785653] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2089.790705] RIP: 0033:0x7f48f0eed2f2 [ 2089.794283] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2089.813030] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2089.820597] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2089.827732] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2089.834861] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2089.841999] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2089.849127] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b5 [ 2089.856272] [ 2089.858462] irq event stamp: 12771311 [ 2089.862128] hardirqs last enabled at (12771321): [] console_unlock+0x106/0x120 [ 2089.870994] hardirqs last disabled at (12771330): [] console_unlock+0xeb/0x120 [ 2089.879775] softirqs last enabled at (12771278): [] __do_softirq+0x2be/0x371 [ 2089.888465] softirqs last disabled at (12771273): [] irq_exit_rcu+0xb0/0xd0 [ 2089.896988] ---[ end trace 0000000000000000 ]--- [ 2089.901605] ------------[ cut here ]------------ [ 2089.906223] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2089.917870] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2089.926566] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2089.982043] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2089.991169] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2089.998648] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2090.004142] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2090.009626] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2090.028379] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2090.033610] RAX: 0000000000000000 RBX: ffff8881d81f5808 RCX: 0000000000000000 [ 2090.040741] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2090.047877] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff0ef [ 2090.055008] R10: 00000000000103f8 R11: ffffc9000030b828 R12: ffff8881d81f5f58 [ 2090.062142] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2090.069273] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2090.077359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2090.083104] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2090.090238] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2090.097371] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2090.104504] Call Trace: [ 2090.106957] [ 2090.109061] ? __warn+0x85/0x180 [ 2090.112297] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.117176] ? report_bug+0xfc/0x1d0 [ 2090.120753] ? prb_read_valid+0x1b/0x20 [ 2090.124595] ? handle_bug+0x44/0x70 [ 2090.128087] ? exc_invalid_op+0x17/0x60 [ 2090.131925] ? asm_exc_invalid_op+0x1a/0x20 [ 2090.136117] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.140993] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.145870] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2090.150761] ceph_add_cap+0x20b/0x620 [ceph] [ 2090.155046] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2090.159770] ? __d_lookup+0xd6/0x200 [ 2090.163355] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2090.168691] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2090.173153] ceph_con_process_message+0x74/0x150 [libceph] [ 2090.178651] ? calc_signature+0xdb/0x100 [libceph] [ 2090.183462] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2090.188792] ? process_scheduled_works+0x22a/0x510 [ 2090.193584] ? process_scheduled_works+0x22a/0x510 [ 2090.198376] ceph_con_workfn+0x344/0x750 [libceph] [ 2090.203177] ? process_scheduled_works+0x22a/0x510 [ 2090.207968] process_scheduled_works+0x2a4/0x510 [ 2090.212595] worker_thread+0x188/0x340 [ 2090.216349] ? rescuer_thread+0x310/0x310 [ 2090.220362] kthread+0xf5/0x130 [ 2090.223507] ? kthread_complete_and_exit+0x20/0x20 [ 2090.228302] ret_from_fork+0x31/0x40 [ 2090.231880] ? kthread_complete_and_exit+0x20/0x20 [ 2090.236673] ret_from_fork_asm+0x11/0x20 [ 2090.240606] [ 2090.242793] irq event stamp: 9251223 [ 2090.246372] hardirqs last enabled at (9251233): [] console_unlock+0x106/0x120 [ 2090.255150] hardirqs last disabled at (9251242): [] console_unlock+0xeb/0x120 [ 2090.263843] softirqs last enabled at (9251220): [] __do_softirq+0x2be/0x371 [ 2090.272450] softirqs last disabled at (9251215): [] irq_exit_rcu+0xb0/0xd0 [ 2090.280882] ---[ end trace 0000000000000000 ]--- [ 2090.285508] ------------[ cut here ]------------ [ 2090.290129] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2090.301777] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2090.310734] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2090.366261] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2090.375213] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2090.382692] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2090.388180] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2090.406925] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2090.412152] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2090.419289] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2090.426426] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff12a [ 2090.433560] R10: 0000000000011578 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2090.440690] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2090.447825] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2090.455914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2090.461666] CR2: 00007f9bd4dcb018 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2090.468798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2090.475932] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2090.483063] Call Trace: [ 2090.485516] [ 2090.487624] ? __warn+0x85/0x180 [ 2090.490857] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.495737] ? report_bug+0xfc/0x1d0 [ 2090.499317] ? prb_read_valid+0x1b/0x20 [ 2090.503166] ? handle_bug+0x44/0x70 [ 2090.506665] ? exc_invalid_op+0x17/0x60 [ 2090.510505] ? asm_exc_invalid_op+0x1a/0x20 [ 2090.514699] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.519579] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2090.524478] try_get_cap_refs+0x433/0x880 [ceph] [ 2090.529123] __ceph_get_caps+0xf8/0x760 [ceph] [ 2090.533602] ? ceph_read_iter+0x134/0x860 [ceph] [ 2090.538237] ceph_read_iter+0x134/0x860 [ceph] [ 2090.542711] ? selinux_file_permission+0x119/0x150 [ 2090.547506] ? vfs_read+0x20f/0x300 [ 2090.551003] vfs_read+0x20f/0x300 [ 2090.554335] ksys_pread64+0x65/0xa0 [ 2090.557833] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2090.562974] do_syscall_64+0x38/0x80 [ 2090.566553] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2090.571605] RIP: 0033:0x7f48f0eeb88b [ 2090.575183] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2090.593931] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2090.601497] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2090.608630] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2090.615762] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2090.622893] R10: 0000000000003000 R11: 0000000000000246 R12: 0000000000001000 [ 2090.630028] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2090.637174] [ 2090.639369] irq event stamp: 12874647 [ 2090.643035] hardirqs last enabled at (12874657): [] console_unlock+0x106/0x120 [ 2090.651901] hardirqs last disabled at (12874666): [] console_unlock+0xeb/0x120 [ 2090.660681] softirqs last enabled at (12874378): [] __do_softirq+0x2be/0x371 [ 2090.669375] softirqs last disabled at (12874373): [] irq_exit_rcu+0xb0/0xd0 [ 2090.677894] ---[ end trace 0000000000000000 ]--- [ 2090.682514] ------------[ cut here ]------------ [ 2090.687151] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2090.698814] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2090.707785] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2090.763320] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2090.772298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2090.779797] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2090.785297] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2090.804053] RSP: 0018:ffffc90001fefbd0 EFLAGS: 00010282 [ 2090.809284] RAX: 0000000000000000 RBX: ffff8881d12d3ab0 RCX: 0000000000000000 [ 2090.816416] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2090.823548] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff162 [ 2090.830681] R10: 0000000000012610 R11: ffffc90001fefa60 R12: ffff8881d12d4200 [ 2090.837817] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2090.844949] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2090.853034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2090.858780] CR2: 00007fb39af87000 CR3: 0000000108c72003 CR4: 00000000003706f0 [ 2090.865914] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2090.873051] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2090.880186] Call Trace: [ 2090.882640] [ 2090.884746] ? __warn+0x85/0x180 [ 2090.887980] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.892867] ? report_bug+0xfc/0x1d0 [ 2090.896444] ? prb_read_valid+0x1b/0x20 [ 2090.900292] ? handle_bug+0x44/0x70 [ 2090.903786] ? exc_invalid_op+0x17/0x60 [ 2090.907627] ? asm_exc_invalid_op+0x1a/0x20 [ 2090.911822] ? __list_add_valid_or_report+0x7d/0xa0 [ 2090.916703] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2090.921601] try_get_cap_refs+0x433/0x880 [ceph] [ 2090.926255] __ceph_get_caps+0xf8/0x760 [ceph] [ 2090.930734] ? ceph_read_iter+0x134/0x860 [ceph] [ 2090.935369] ceph_read_iter+0x134/0x860 [ceph] [ 2090.939840] ? selinux_file_permission+0x119/0x150 [ 2090.944638] ? vfs_read+0x20f/0x300 [ 2090.948155] vfs_read+0x20f/0x300 [ 2090.951493] ksys_pread64+0x65/0xa0 [ 2090.954991] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2090.960134] do_syscall_64+0x38/0x80 [ 2090.963720] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2090.968780] RIP: 0033:0x7f48f0eeb88b [ 2090.972357] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2090.991110] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2090.998680] RAX: ffffffffffffffda RBX: 0000000002075930 RCX: 00007f48f0eeb88b [ 2091.005811] RDX: 0000000000001000 RSI: 000000000207e040 RDI: 0000000000000015 [ 2091.012945] RBP: 00000000000029b5 R08: 0000000000000063 R09: 0000000000000003 [ 2091.020077] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2091.027214] R13: 0000000000001000 R14: 000000000207e040 R15: 00007f48f183df18 [ 2091.034366] [ 2091.036561] irq event stamp: 12774577 [ 2091.040227] hardirqs last enabled at (12774587): [] console_unlock+0x106/0x120 [ 2091.049094] hardirqs last disabled at (12774596): [] console_unlock+0xeb/0x120 [ 2091.057875] softirqs last enabled at (12774554): [] __do_softirq+0x2be/0x371 [ 2091.066570] softirqs last disabled at (12774615): [] irq_exit_rcu+0xb0/0xd0 [ 2091.075095] ---[ end trace 0000000000000000 ]--- [ 2091.079716] ------------[ cut here ]------------ [ 2091.084331] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2091.095977] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2091.104936] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2091.160416] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2091.169374] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2091.176854] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2091.182344] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2091.201092] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2091.206317] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2091.213450] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2091.220582] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff19a [ 2091.227715] R10: 00000000000137a8 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2091.234849] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2091.241981] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2091.250067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2091.255811] CR2: 00007fb38efd9040 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2091.262945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2091.270077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2091.277202] Call Trace: [ 2091.279656] [ 2091.281763] ? __warn+0x85/0x180 [ 2091.284994] ? __list_add_valid_or_report+0x7d/0xa0 [ 2091.289875] ? report_bug+0xfc/0x1d0 [ 2091.293452] ? prb_read_valid+0x1b/0x20 [ 2091.297295] ? handle_bug+0x44/0x70 [ 2091.300784] ? exc_invalid_op+0x17/0x60 [ 2091.304615] ? asm_exc_invalid_op+0x1a/0x20 [ 2091.308807] ? __list_add_valid_or_report+0x7d/0xa0 [ 2091.313682] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2091.318570] try_get_cap_refs+0x433/0x880 [ceph] [ 2091.323203] __ceph_get_caps+0xf8/0x760 [ceph] [ 2091.327669] ? ceph_read_iter+0x134/0x860 [ceph] [ 2091.332299] ceph_read_iter+0x134/0x860 [ceph] [ 2091.336762] ? selinux_file_permission+0x119/0x150 [ 2091.341557] ? vfs_read+0x20f/0x300 [ 2091.345047] vfs_read+0x20f/0x300 [ 2091.348370] ksys_pread64+0x65/0xa0 [ 2091.351858] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2091.356998] do_syscall_64+0x38/0x80 [ 2091.360582] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2091.365637] RIP: 0033:0x7f48f0eeb88b [ 2091.369208] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2091.387955] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2091.395524] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2091.402655] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2091.409788] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2091.416920] R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000001000 [ 2091.424054] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2091.431192] [ 2091.433379] irq event stamp: 12878149 [ 2091.437044] hardirqs last enabled at (12878159): [] console_unlock+0x106/0x120 [ 2091.445910] hardirqs last disabled at (12878168): [] console_unlock+0xeb/0x120 [ 2091.454690] softirqs last enabled at (12878126): [] __do_softirq+0x2be/0x371 [ 2091.463383] softirqs last disabled at (12878121): [] irq_exit_rcu+0xb0/0xd0 [ 2091.471902] ---[ end trace 0000000000000000 ]--- [ 2091.476522] ------------[ cut here ]------------ [ 2091.481143] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2091.492791] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2091.501744] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2091.557270] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2091.566226] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2091.573706] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2091.579191] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2091.597992] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2091.603215] RAX: 0000000000000000 RBX: ffff8881d0bd9d58 RCX: 0000000000000000 [ 2091.610349] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2091.617480] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff1d2 [ 2091.624615] R10: 0000000000014938 R11: ffffc90001fefb18 R12: ffff8881d0bda4a8 [ 2091.631748] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2091.638880] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2091.646966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2091.652713] CR2: 00007fb38efec000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2091.659845] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2091.666978] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2091.674110] Call Trace: [ 2091.676563] [ 2091.678670] ? __warn+0x85/0x180 [ 2091.681904] ? __list_add_valid_or_report+0x7d/0xa0 [ 2091.686784] ? report_bug+0xfc/0x1d0 [ 2091.690371] ? prb_read_valid+0x1b/0x20 [ 2091.694214] ? handle_bug+0x44/0x70 [ 2091.697710] ? exc_invalid_op+0x17/0x60 [ 2091.701552] ? asm_exc_invalid_op+0x1a/0x20 [ 2091.705746] ? __list_add_valid_or_report+0x7d/0xa0 [ 2091.710626] ? __list_add_valid_or_report+0x7d/0xa0 [ 2091.715504] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2091.720404] ceph_open+0x16b/0x440 [ceph] [ 2091.724436] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2091.729592] do_dentry_open+0x172/0x530 [ 2091.733439] path_openat+0x743/0x9d0 [ 2091.737033] do_filp_open+0xb8/0x120 [ 2091.740620] ? lock_release+0xc5/0x260 [ 2091.744376] ? _raw_spin_unlock+0x23/0x40 [ 2091.748394] ? alloc_fd+0x13f/0x200 [ 2091.751891] do_sys_openat2+0x249/0x290 [ 2091.755740] do_sys_open+0x4f/0x80 [ 2091.759152] do_syscall_64+0x38/0x80 [ 2091.762731] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2091.767783] RIP: 0033:0x7f48f0eed2f2 [ 2091.771360] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2091.790115] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2091.797701] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2091.804835] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2091.811965] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2091.819099] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2091.826231] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b6 [ 2091.833376] [ 2091.835565] irq event stamp: 12776541 [ 2091.839231] hardirqs last enabled at (12776551): [] console_unlock+0x106/0x120 [ 2091.848098] hardirqs last disabled at (12776560): [] console_unlock+0xeb/0x120 [ 2091.856882] softirqs last enabled at (12776576): [] __do_softirq+0x2be/0x371 [ 2091.865578] softirqs last disabled at (12776569): [] irq_exit_rcu+0xb0/0xd0 [ 2091.874098] ---[ end trace 0000000000000000 ]--- [ 2091.878717] ------------[ cut here ]------------ [ 2091.883339] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2091.894987] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2091.903689] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2091.959241] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2091.968369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2091.975849] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2091.981359] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2091.986854] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2092.005603] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2092.010835] RAX: 0000000000000000 RBX: ffff8881d0bd9390 RCX: 0000000000000000 [ 2092.017969] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2092.025104] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff20c [ 2092.032243] R10: 00000000000157f8 R11: ffffc9000030b828 R12: ffff8881d0bd9ae0 [ 2092.039376] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2092.046510] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2092.054603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.060348] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2092.067484] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2092.074616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2092.081748] Call Trace: [ 2092.084202] [ 2092.086309] ? __warn+0x85/0x180 [ 2092.089551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.094438] ? report_bug+0xfc/0x1d0 [ 2092.098026] ? prb_read_valid+0x1b/0x20 [ 2092.101876] ? handle_bug+0x44/0x70 [ 2092.105376] ? exc_invalid_op+0x17/0x60 [ 2092.109224] ? asm_exc_invalid_op+0x1a/0x20 [ 2092.113430] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.118316] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.123202] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2092.128105] ceph_add_cap+0x20b/0x620 [ceph] [ 2092.132411] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2092.137144] ? __d_lookup+0xd6/0x200 [ 2092.140735] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2092.146083] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2092.150571] ceph_con_process_message+0x74/0x150 [libceph] [ 2092.156077] ? calc_signature+0xdb/0x100 [libceph] [ 2092.160899] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2092.166240] ? process_scheduled_works+0x22a/0x510 [ 2092.171039] ? process_scheduled_works+0x22a/0x510 [ 2092.175836] ceph_con_workfn+0x344/0x750 [libceph] [ 2092.180651] ? process_scheduled_works+0x22a/0x510 [ 2092.185447] process_scheduled_works+0x2a4/0x510 [ 2092.190078] worker_thread+0x188/0x340 [ 2092.193839] ? rescuer_thread+0x310/0x310 [ 2092.197858] kthread+0xf5/0x130 [ 2092.201014] ? kthread_complete_and_exit+0x20/0x20 [ 2092.205817] ret_from_fork+0x31/0x40 [ 2092.209403] ? kthread_complete_and_exit+0x20/0x20 [ 2092.214206] ret_from_fork_asm+0x11/0x20 [ 2092.218153] [ 2092.220351] irq event stamp: 9255677 [ 2092.223936] hardirqs last enabled at (9255687): [] console_unlock+0x106/0x120 [ 2092.232722] hardirqs last disabled at (9255698): [] console_unlock+0xeb/0x120 [ 2092.241417] softirqs last enabled at (9255674): [] __do_softirq+0x2be/0x371 [ 2092.250032] softirqs last disabled at (9255669): [] irq_exit_rcu+0xb0/0xd0 [ 2092.258465] ---[ end trace 0000000000000000 ]--- [ 2092.263084] ------------[ cut here ]------------ [ 2092.267702] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2092.279348] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2092.288304] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2092.343832] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2092.352784] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2092.360263] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2092.365751] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2092.384496] RSP: 0018:ffffc90001fefbd0 EFLAGS: 00010282 [ 2092.389722] RAX: 0000000000000000 RBX: ffff8881d0bd9d58 RCX: 0000000000000000 [ 2092.396854] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2092.403987] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff247 [ 2092.411120] R10: 00000000000165a0 R11: ffffc90001fefa60 R12: ffff8881d0bda4a8 [ 2092.418253] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2092.425385] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2092.433474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.439220] CR2: 00007fb38efec000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2092.446351] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2092.453484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2092.460616] Call Trace: [ 2092.463069] [ 2092.465177] ? __warn+0x85/0x180 [ 2092.468409] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.473289] ? report_bug+0xfc/0x1d0 [ 2092.476866] ? prb_read_valid+0x1b/0x20 [ 2092.480711] ? handle_bug+0x44/0x70 [ 2092.484210] ? exc_invalid_op+0x17/0x60 [ 2092.488048] ? asm_exc_invalid_op+0x1a/0x20 [ 2092.492244] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.497128] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2092.502030] try_get_cap_refs+0x433/0x880 [ceph] [ 2092.506674] __ceph_get_caps+0xf8/0x760 [ceph] [ 2092.511157] ? ceph_read_iter+0x134/0x860 [ceph] [ 2092.515790] ceph_read_iter+0x134/0x860 [ceph] [ 2092.520263] ? selinux_file_permission+0x119/0x150 [ 2092.525061] ? vfs_read+0x20f/0x300 [ 2092.528556] vfs_read+0x20f/0x300 [ 2092.531887] ksys_pread64+0x65/0xa0 [ 2092.535387] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2092.540529] do_syscall_64+0x38/0x80 [ 2092.544115] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2092.549166] RIP: 0033:0x7f48f0eeb88b [ 2092.552746] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2092.571491] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2092.579056] RAX: ffffffffffffffda RBX: 0000000002075940 RCX: 00007f48f0eeb88b [ 2092.586189] RDX: 0000000000001000 RSI: 000000000207e080 RDI: 0000000000000016 [ 2092.593322] RBP: 00000000000029b6 R08: 0000000000000063 R09: 0000000000000003 [ 2092.600456] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2092.607590] R13: 0000000000001000 R14: 000000000207e080 R15: 00007f48f183df18 [ 2092.614736] [ 2092.616932] irq event stamp: 12778337 [ 2092.620597] hardirqs last enabled at (12778347): [] console_unlock+0x106/0x120 [ 2092.629468] hardirqs last disabled at (12778364): [] console_unlock+0xeb/0x120 [ 2092.638252] softirqs last enabled at (12778362): [] __do_softirq+0x2be/0x371 [ 2092.646944] softirqs last disabled at (12778355): [] irq_exit_rcu+0xb0/0xd0 [ 2092.655471] ---[ end trace 0000000000000000 ]--- [ 2092.660093] ------------[ cut here ]------------ [ 2092.664714] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2092.676361] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2092.685063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2092.740612] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2092.749743] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2092.757224] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2092.762735] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2092.768219] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2092.786968] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2092.792202] RAX: 0000000000000000 RBX: ffff8881d12db0e8 RCX: 0000000000000000 [ 2092.799336] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2092.806471] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff27f [ 2092.813610] R10: 0000000000017978 R11: ffffc9000030b828 R12: ffff8881d12db838 [ 2092.820741] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2092.827874] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2092.835961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.841707] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2092.848840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2092.855974] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2092.863105] Call Trace: [ 2092.865559] [ 2092.867665] ? __warn+0x85/0x180 [ 2092.870902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.875789] ? report_bug+0xfc/0x1d0 [ 2092.879374] ? prb_read_valid+0x1b/0x20 [ 2092.883218] ? handle_bug+0x44/0x70 [ 2092.886714] ? exc_invalid_op+0x17/0x60 [ 2092.890555] ? asm_exc_invalid_op+0x1a/0x20 [ 2092.894760] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.899649] ? __list_add_valid_or_report+0x7d/0xa0 [ 2092.904535] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2092.909444] ceph_add_cap+0x20b/0x620 [ceph] [ 2092.913750] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2092.918485] ? __d_lookup+0xd6/0x200 [ 2092.922074] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2092.927425] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2092.931910] ceph_con_process_message+0x74/0x150 [libceph] [ 2092.937419] ? calc_signature+0xdb/0x100 [libceph] [ 2092.942239] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2092.947580] ? process_scheduled_works+0x22a/0x510 [ 2092.952381] ? process_scheduled_works+0x22a/0x510 [ 2092.957173] ceph_con_workfn+0x344/0x750 [libceph] [ 2092.961994] ? process_scheduled_works+0x22a/0x510 [ 2092.966792] process_scheduled_works+0x2a4/0x510 [ 2092.971429] worker_thread+0x188/0x340 [ 2092.975188] ? rescuer_thread+0x310/0x310 [ 2092.979209] kthread+0xf5/0x130 [ 2092.982363] ? kthread_complete_and_exit+0x20/0x20 [ 2092.987167] ret_from_fork+0x31/0x40 [ 2092.990750] ? kthread_complete_and_exit+0x20/0x20 [ 2092.995545] ret_from_fork_asm+0x11/0x20 [ 2092.999494] [ 2093.001691] irq event stamp: 9257625 [ 2093.005276] hardirqs last enabled at (9257635): [] console_unlock+0x106/0x120 [ 2093.014056] hardirqs last disabled at (9257644): [] console_unlock+0xeb/0x120 [ 2093.022748] softirqs last enabled at (9257622): [] __do_softirq+0x2be/0x371 [ 2093.031355] softirqs last disabled at (9257617): [] irq_exit_rcu+0xb0/0xd0 [ 2093.039788] ---[ end trace 0000000000000000 ]--- [ 2093.044407] ------------[ cut here ]------------ [ 2093.049023] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2093.060672] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2093.069625] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2093.125145] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2093.134097] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2093.141582] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2093.147072] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2093.165819] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2093.171044] RAX: 0000000000000000 RBX: ffff8881d0b8ce40 RCX: 0000000000000000 [ 2093.178177] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2093.185310] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff2ba [ 2093.192443] R10: 0000000000018758 R11: ffffc90001fefb18 R12: ffff8881d0b8d590 [ 2093.199576] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2093.206708] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2093.214795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2093.220541] CR2: 00007fb38efec000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2093.227674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2093.234807] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2093.241938] Call Trace: [ 2093.244392] [ 2093.246499] ? __warn+0x85/0x180 [ 2093.249732] ? __list_add_valid_or_report+0x7d/0xa0 [ 2093.254613] ? report_bug+0xfc/0x1d0 [ 2093.258190] ? prb_read_valid+0x1b/0x20 [ 2093.262034] ? handle_bug+0x44/0x70 [ 2093.265531] ? exc_invalid_op+0x17/0x60 [ 2093.269376] ? asm_exc_invalid_op+0x1a/0x20 [ 2093.273575] ? __list_add_valid_or_report+0x7d/0xa0 [ 2093.278455] ? __list_add_valid_or_report+0x7d/0xa0 [ 2093.283333] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2093.288234] ceph_open+0x16b/0x440 [ceph] [ 2093.292263] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2093.297418] do_dentry_open+0x172/0x530 [ 2093.301258] path_openat+0x743/0x9d0 [ 2093.304842] do_filp_open+0xb8/0x120 [ 2093.308431] ? lock_release+0xc5/0x260 [ 2093.312188] ? _raw_spin_unlock+0x23/0x40 [ 2093.316207] ? alloc_fd+0x13f/0x200 [ 2093.319703] do_sys_openat2+0x249/0x290 [ 2093.323550] do_sys_open+0x4f/0x80 [ 2093.326968] do_syscall_64+0x38/0x80 [ 2093.330551] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2093.335603] RIP: 0033:0x7f48f0eed2f2 [ 2093.339181] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2093.357927] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2093.365494] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2093.372626] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2093.379760] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2093.386893] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2093.394024] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029b7 [ 2093.401175] [ 2093.403369] irq event stamp: 12780241 [ 2093.407033] hardirqs last enabled at (12780251): [] console_unlock+0x106/0x120 [ 2093.415899] hardirqs last disabled at (12780260): [] console_unlock+0xeb/0x120 [ 2093.424680] softirqs last enabled at (12780228): [] __do_softirq+0x2be/0x371 [ 2093.433372] softirqs last disabled at (12780221): [] irq_exit_rcu+0xb0/0xd0 [ 2093.441891] ---[ end trace 0000000000000000 ]--- [ 2093.446513] ------------[ cut here ]------------ [ 2093.451130] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2093.462780] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2093.471482] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2093.527032] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2093.536164] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2093.543643] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2093.549152] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2093.554640] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2093.573387] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2093.578621] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2093.585755] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2093.592890] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff2f4 [ 2093.600029] R10: 0000000000019898 R11: ffffc9000030b828 R12: ffff8881d12dcbc8 [ 2093.607160] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2093.614293] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2093.622380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2093.628125] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2093.635262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2093.642400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2093.649534] Call Trace: [ 2093.651986] [ 2093.654093] ? __warn+0x85/0x180 [ 2093.657327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2093.662216] ? report_bug+0xfc/0x1d0 [ 2093.665802] ? prb_read_valid+0x1b/0x20 [ 2093.669655] ? handle_bug+0x44/0x70 [ 2093.673152] ? exc_invalid_op+0x17/0x60 [ 2093.676993] ? asm_exc_invalid_op+0x1a/0x20 [ 2093.681196] ? __list_add_valid_or_report+0x7d/0xa0 [ 2093.686083] ? __list_add_valid_or_report+0x7d/0xa0 [ 2093.690973] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2093.695880] ceph_add_cap+0x20b/0x620 [ceph] [ 2093.700188] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2093.704930] ? __d_lookup+0xd6/0x200 [ 2093.708521] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2093.713870] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2093.718357] ceph_con_process_message+0x74/0x150 [libceph] [ 2093.723864] ? calc_signature+0xdb/0x100 [libceph] [ 2093.728684] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2093.734026] ? process_scheduled_works+0x22a/0x510 [ 2093.738826] ? process_scheduled_works+0x22a/0x510 [ 2093.743622] ceph_con_workfn+0x344/0x750 [libceph] [ 2093.748436] ? process_scheduled_works+0x22a/0x510 [ 2093.753232] process_scheduled_works+0x2a4/0x510 [ 2093.757862] worker_thread+0x188/0x340 [ 2093.761623] ? rescuer_thread+0x310/0x310 [ 2093.765644] kthread+0xf5/0x130 [ 2093.768799] ? kthread_complete_and_exit+0x20/0x20 [ 2093.773601] ret_from_fork+0x31/0x40 [ 2093.777188] ? kthread_complete_and_exit+0x20/0x20 [ 2093.781991] ret_from_fork_asm+0x11/0x20 [ 2093.785938] [ 2093.788136] irq event stamp: 9259505 [ 2093.791713] hardirqs last enabled at (9259515): [] console_unlock+0x106/0x120 [ 2093.800502] hardirqs last disabled at (9259524): [] console_unlock+0xeb/0x120 [ 2093.809201] softirqs last enabled at (9259502): [] __do_softirq+0x2be/0x371 [ 2093.817808] softirqs last disabled at (9259497): [] irq_exit_rcu+0xb0/0xd0 [ 2093.826243] ---[ end trace 0000000000000000 ]--- [ 2093.830863] ------------[ cut here ]------------ [ 2093.835488] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2093.847246] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2093.856204] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2093.911731] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2093.920691] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2093.928171] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2093.933657] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2093.952402] RSP: 0018:ffffc90001fefbd0 EFLAGS: 00010282 [ 2093.957628] RAX: 0000000000000000 RBX: ffff8881d0b8ce40 RCX: 0000000000000000 [ 2093.964763] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2093.971894] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff32f [ 2093.979029] R10: 000000000001aa40 R11: ffffc90001fefa60 R12: ffff8881d0b8d590 [ 2093.986161] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2093.993293] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2094.001379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2094.007126] CR2: 00007fb38efec000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2094.014258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2094.021390] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2094.028524] Call Trace: [ 2094.030977] [ 2094.033082] ? __warn+0x85/0x180 [ 2094.036315] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.041196] ? report_bug+0xfc/0x1d0 [ 2094.044775] ? prb_read_valid+0x1b/0x20 [ 2094.048618] ? handle_bug+0x44/0x70 [ 2094.052114] ? exc_invalid_op+0x17/0x60 [ 2094.055956] ? asm_exc_invalid_op+0x1a/0x20 [ 2094.060150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.065032] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2094.069939] try_get_cap_refs+0x433/0x880 [ceph] [ 2094.074591] __ceph_get_caps+0xf8/0x760 [ceph] [ 2094.079071] ? ceph_read_iter+0x134/0x860 [ceph] [ 2094.083707] ceph_read_iter+0x134/0x860 [ceph] [ 2094.088179] ? selinux_file_permission+0x119/0x150 [ 2094.092981] ? vfs_read+0x20f/0x300 [ 2094.096469] vfs_read+0x20f/0x300 [ 2094.099802] ksys_pread64+0x65/0xa0 [ 2094.103301] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2094.108444] do_syscall_64+0x38/0x80 [ 2094.112030] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2094.117082] RIP: 0033:0x7f48f0eeb88b [ 2094.120661] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2094.139407] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2094.146974] RAX: ffffffffffffffda RBX: 0000000002075950 RCX: 00007f48f0eeb88b [ 2094.154112] RDX: 000000000000021f RSI: 000000000207d800 RDI: 0000000000000017 [ 2094.161249] RBP: 00000000000029b7 R08: 000000000000021f R09: 00000000020767c0 [ 2094.168381] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000021f [ 2094.175517] R13: 000000000000021f R14: 000000000207d800 R15: 00007f48f183df18 [ 2094.182668] [ 2094.184864] irq event stamp: 12782219 [ 2094.188532] hardirqs last enabled at (12782229): [] console_unlock+0x106/0x120 [ 2094.197397] hardirqs last disabled at (12782238): [] console_unlock+0xeb/0x120 [ 2094.206177] softirqs last enabled at (12782196): [] __do_softirq+0x2be/0x371 [ 2094.214869] softirqs last disabled at (12782191): [] irq_exit_rcu+0xb0/0xd0 [ 2094.223388] ---[ end trace 0000000000000000 ]--- [ 2094.228008] ------------[ cut here ]------------ [ 2094.232627] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2094.244273] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2094.252968] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2094.308454] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2094.317578] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2094.325059] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2094.330562] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2094.336047] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2094.354793] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2094.360018] RAX: 0000000000000000 RBX: ffff8881d132d808 RCX: 0000000000000000 [ 2094.367152] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2094.374285] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff367 [ 2094.381419] R10: 000000000001bb80 R11: ffffc9000030b828 R12: ffff8881d132df58 [ 2094.388551] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2094.395684] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2094.403772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2094.409514] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2094.416648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2094.423782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2094.430912] Call Trace: [ 2094.433358] [ 2094.435465] ? __warn+0x85/0x180 [ 2094.438699] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.443579] ? report_bug+0xfc/0x1d0 [ 2094.447154] ? prb_read_valid+0x1b/0x20 [ 2094.450996] ? handle_bug+0x44/0x70 [ 2094.454488] ? exc_invalid_op+0x17/0x60 [ 2094.458329] ? asm_exc_invalid_op+0x1a/0x20 [ 2094.462519] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.467404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.472282] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2094.477172] ceph_add_cap+0x20b/0x620 [ceph] [ 2094.481458] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2094.486183] ? __d_lookup+0xd6/0x200 [ 2094.489766] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2094.495093] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2094.499556] ceph_con_process_message+0x74/0x150 [libceph] [ 2094.505051] ? calc_signature+0xdb/0x100 [libceph] [ 2094.509863] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2094.515195] ? process_scheduled_works+0x22a/0x510 [ 2094.519995] ? process_scheduled_works+0x22a/0x510 [ 2094.524785] ceph_con_workfn+0x344/0x750 [libceph] [ 2094.529588] ? process_scheduled_works+0x22a/0x510 [ 2094.534380] process_scheduled_works+0x2a4/0x510 [ 2094.539005] worker_thread+0x188/0x340 [ 2094.542761] ? rescuer_thread+0x310/0x310 [ 2094.546772] kthread+0xf5/0x130 [ 2094.549917] ? kthread_complete_and_exit+0x20/0x20 [ 2094.554712] ret_from_fork+0x31/0x40 [ 2094.558291] ? kthread_complete_and_exit+0x20/0x20 [ 2094.563085] ret_from_fork_asm+0x11/0x20 [ 2094.567018] [ 2094.569210] irq event stamp: 9261447 [ 2094.572791] hardirqs last enabled at (9261457): [] console_unlock+0x106/0x120 [ 2094.581569] hardirqs last disabled at (9261466): [] console_unlock+0xeb/0x120 [ 2094.590262] softirqs last enabled at (9261444): [] __do_softirq+0x2be/0x371 [ 2094.598869] softirqs last disabled at (9261439): [] irq_exit_rcu+0xb0/0xd0 [ 2094.607301] ---[ end trace 0000000000000000 ]--- [ 2094.611922] ------------[ cut here ]------------ [ 2094.616538] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2094.628186] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2094.637140] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2094.692611] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2094.701561] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2094.709039] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2094.714526] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2094.733274] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2094.738497] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2094.745622] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2094.752756] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff3a2 [ 2094.759890] R10: 000000000001cce0 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2094.767020] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2094.774145] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2094.782233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2094.787997] CR2: 00007fb38eff0038 CR3: 0000000120c1e006 CR4: 00000000003706f0 [ 2094.795133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2094.802259] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2094.809392] Call Trace: [ 2094.811836] [ 2094.813935] ? __warn+0x85/0x180 [ 2094.817167] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.822048] ? report_bug+0xfc/0x1d0 [ 2094.825625] ? prb_read_valid+0x1b/0x20 [ 2094.829469] ? handle_bug+0x44/0x70 [ 2094.832957] ? exc_invalid_op+0x17/0x60 [ 2094.836790] ? asm_exc_invalid_op+0x1a/0x20 [ 2094.840981] ? __list_add_valid_or_report+0x7d/0xa0 [ 2094.845856] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2094.850745] try_get_cap_refs+0x433/0x880 [ceph] [ 2094.855377] __ceph_get_caps+0xf8/0x760 [ceph] [ 2094.859850] ? ceph_read_iter+0x134/0x860 [ceph] [ 2094.864481] ceph_read_iter+0x134/0x860 [ceph] [ 2094.868940] ? selinux_file_permission+0x119/0x150 [ 2094.873730] ? vfs_read+0x20f/0x300 [ 2094.877220] vfs_read+0x20f/0x300 [ 2094.880546] ksys_pread64+0x65/0xa0 [ 2094.884041] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2094.889184] do_syscall_64+0x38/0x80 [ 2094.892759] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2094.897813] RIP: 0033:0x7f48f0eeb88b [ 2094.901392] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2094.920138] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2094.927703] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2094.934827] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2094.941960] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2094.949094] R10: 0000000000005000 R11: 0000000000000246 R12: 0000000000001000 [ 2094.956228] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2094.963365] [ 2094.965553] irq event stamp: 12886945 [ 2094.969217] hardirqs last enabled at (12886955): [] console_unlock+0x106/0x120 [ 2094.978084] hardirqs last disabled at (12886964): [] console_unlock+0xeb/0x120 [ 2094.986864] softirqs last enabled at (12886922): [] __do_softirq+0x2be/0x371 [ 2094.995555] softirqs last disabled at (12886917): [] irq_exit_rcu+0xb0/0xd0 [ 2095.004077] ---[ end trace 0000000000000000 ]--- [ 2095.008697] ------------[ cut here ]------------ [ 2095.013317] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2095.024961] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2095.033658] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2095.089136] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2095.098260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2095.105739] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2095.111235] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2095.116719] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2095.135466] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2095.140691] RAX: 0000000000000000 RBX: ffff8881d12d9d58 RCX: 0000000000000000 [ 2095.147825] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2095.154960] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff3da [ 2095.162092] R10: 000000000001de18 R11: ffffc9000030b828 R12: ffff8881d12da4a8 [ 2095.169222] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2095.176357] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2095.184444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2095.190188] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2095.197322] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2095.204455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2095.211586] Call Trace: [ 2095.214037] [ 2095.216137] ? __warn+0x85/0x180 [ 2095.219373] ? __list_add_valid_or_report+0x7d/0xa0 [ 2095.224250] ? report_bug+0xfc/0x1d0 [ 2095.227830] ? prb_read_valid+0x1b/0x20 [ 2095.231671] ? handle_bug+0x44/0x70 [ 2095.235160] ? exc_invalid_op+0x17/0x60 [ 2095.239002] ? asm_exc_invalid_op+0x1a/0x20 [ 2095.243192] ? __list_add_valid_or_report+0x7d/0xa0 [ 2095.248077] ? __list_add_valid_or_report+0x7d/0xa0 [ 2095.252954] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2095.257844] ceph_add_cap+0x20b/0x620 [ceph] [ 2095.262130] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2095.266855] ? __d_lookup+0xd6/0x200 [ 2095.270439] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2095.275766] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2095.280229] ceph_con_process_message+0x74/0x150 [libceph] [ 2095.285724] ? calc_signature+0xdb/0x100 [libceph] [ 2095.290537] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2095.295869] ? process_scheduled_works+0x22a/0x510 [ 2095.300668] ? process_scheduled_works+0x22a/0x510 [ 2095.305457] ceph_con_workfn+0x344/0x750 [libceph] [ 2095.310260] ? process_scheduled_works+0x22a/0x510 [ 2095.315051] process_scheduled_works+0x2a4/0x510 [ 2095.319676] worker_thread+0x188/0x340 [ 2095.323423] ? rescuer_thread+0x310/0x310 [ 2095.327435] kthread+0xf5/0x130 [ 2095.330582] ? kthread_complete_and_exit+0x20/0x20 [ 2095.335377] ret_from_fork+0x31/0x40 [ 2095.338954] ? kthread_complete_and_exit+0x20/0x20 [ 2095.343747] ret_from_fork_asm+0x11/0x20 [ 2095.347680] [ 2095.349867] irq event stamp: 9263337 [ 2095.353446] hardirqs last enabled at (9263347): [] console_unlock+0x106/0x120 [ 2095.362226] hardirqs last disabled at (9263356): [] console_unlock+0xeb/0x120 [ 2095.370919] softirqs last enabled at (9263334): [] __do_softirq+0x2be/0x371 [ 2095.379523] softirqs last disabled at (9263329): [] irq_exit_rcu+0xb0/0xd0 [ 2095.387958] ---[ end trace 0000000000000000 ]--- [ 2095.392577] ------------[ cut here ]------------ [ 2095.397195] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2095.408842] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2095.417798] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2095.473281] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2095.482234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2095.489712] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2095.495199] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2095.513947] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2095.519172] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2095.526308] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2095.533436] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff415 [ 2095.540562] R10: 000000000001ef50 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2095.547699] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2095.554827] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2095.562914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2095.568659] CR2: 00007fb38e420000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2095.575793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2095.582925] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2095.590057] Call Trace: [ 2095.592501] [ 2095.594600] ? __warn+0x85/0x180 [ 2095.597831] ? __list_add_valid_or_report+0x7d/0xa0 [ 2095.602711] ? report_bug+0xfc/0x1d0 [ 2095.606292] ? prb_read_valid+0x1b/0x20 [ 2095.610131] ? handle_bug+0x44/0x70 [ 2095.613622] ? exc_invalid_op+0x17/0x60 [ 2095.617463] ? asm_exc_invalid_op+0x1a/0x20 [ 2095.621653] ? __list_add_valid_or_report+0x7d/0xa0 [ 2095.626528] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2095.631419] try_get_cap_refs+0x433/0x880 [ceph] [ 2095.636054] __ceph_get_caps+0xf8/0x760 [ceph] [ 2095.640515] ? ceph_read_iter+0x134/0x860 [ceph] [ 2095.645143] ceph_read_iter+0x134/0x860 [ceph] [ 2095.649608] ? selinux_file_permission+0x119/0x150 [ 2095.654402] ? vfs_read+0x20f/0x300 [ 2095.657893] vfs_read+0x20f/0x300 [ 2095.661219] ksys_pread64+0x65/0xa0 [ 2095.664713] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2095.669858] do_syscall_64+0x38/0x80 [ 2095.673432] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2095.678487] RIP: 0033:0x7f48f0eeb88b [ 2095.682063] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2095.700818] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2095.708387] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2095.715518] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2095.722650] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2095.729774] R10: 0000000000006000 R11: 0000000000000246 R12: 0000000000001000 [ 2095.736901] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2095.744038] [ 2095.746225] irq event stamp: 12889199 [ 2095.749892] hardirqs last enabled at (12889209): [] console_unlock+0x106/0x120 [ 2095.758756] hardirqs last disabled at (12889218): [] console_unlock+0xeb/0x120 [ 2095.767535] softirqs last enabled at (12889176): [] __do_softirq+0x2be/0x371 [ 2095.776229] softirqs last disabled at (12889237): [] irq_exit_rcu+0xb0/0xd0 [ 2095.784752] ---[ end trace 0000000000000000 ]--- [ 2095.789371] ------------[ cut here ]------------ [ 2095.793992] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2095.805635] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2095.814330] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2095.869810] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2095.878934] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2095.886413] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2095.891908] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2095.897394] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2095.916139] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2095.921365] RAX: 0000000000000000 RBX: ffff8881d12dd808 RCX: 0000000000000000 [ 2095.928498] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2095.935632] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff44d [ 2095.942764] R10: 0000000000020018 R11: ffffc9000030b828 R12: ffff8881d12ddf58 [ 2095.949896] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2095.957029] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2095.965115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2095.970863] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2095.977993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2095.985126] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2095.992260] Call Trace: [ 2095.994713] [ 2095.996818] ? __warn+0x85/0x180 [ 2096.000055] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.004932] ? report_bug+0xfc/0x1d0 [ 2096.008510] ? prb_read_valid+0x1b/0x20 [ 2096.012350] ? handle_bug+0x44/0x70 [ 2096.015842] ? exc_invalid_op+0x17/0x60 [ 2096.019683] ? asm_exc_invalid_op+0x1a/0x20 [ 2096.023872] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.028749] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.033627] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2096.038518] ceph_add_cap+0x20b/0x620 [ceph] [ 2096.042802] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2096.047528] ? __d_lookup+0xd6/0x200 [ 2096.051111] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2096.056441] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2096.060911] ceph_con_process_message+0x74/0x150 [libceph] [ 2096.066406] ? calc_signature+0xdb/0x100 [libceph] [ 2096.071218] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2096.076550] ? process_scheduled_works+0x22a/0x510 [ 2096.081350] ? process_scheduled_works+0x22a/0x510 [ 2096.086137] ceph_con_workfn+0x344/0x750 [libceph] [ 2096.090943] ? process_scheduled_works+0x22a/0x510 [ 2096.095735] process_scheduled_works+0x2a4/0x510 [ 2096.100358] worker_thread+0x188/0x340 [ 2096.104115] ? rescuer_thread+0x310/0x310 [ 2096.108129] kthread+0xf5/0x130 [ 2096.111274] ? kthread_complete_and_exit+0x20/0x20 [ 2096.116066] ret_from_fork+0x31/0x40 [ 2096.119645] ? kthread_complete_and_exit+0x20/0x20 [ 2096.124439] ret_from_fork_asm+0x11/0x20 [ 2096.128373] [ 2096.130564] irq event stamp: 9265231 [ 2096.134142] hardirqs last enabled at (9265241): [] console_unlock+0x106/0x120 [ 2096.142922] hardirqs last disabled at (9265250): [] console_unlock+0xeb/0x120 [ 2096.151617] softirqs last enabled at (9265228): [] __do_softirq+0x2be/0x371 [ 2096.160224] softirqs last disabled at (9265223): [] irq_exit_rcu+0xb0/0xd0 [ 2096.168657] ---[ end trace 0000000000000000 ]--- [ 2096.173277] ------------[ cut here ]------------ [ 2096.177902] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2096.189552] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2096.198512] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2096.253983] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2096.262932] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2096.270413] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2096.275897] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2096.294645] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2096.299871] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2096.307005] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2096.314135] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff488 [ 2096.321270] R10: 0000000000021248 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2096.328406] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2096.335538] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2096.343621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2096.349366] CR2: 00007fb38e420000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2096.356499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2096.363632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2096.370771] Call Trace: [ 2096.373219] [ 2096.375326] ? __warn+0x85/0x180 [ 2096.378557] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.383437] ? report_bug+0xfc/0x1d0 [ 2096.387023] ? prb_read_valid+0x1b/0x20 [ 2096.390865] ? handle_bug+0x44/0x70 [ 2096.394356] ? exc_invalid_op+0x17/0x60 [ 2096.398196] ? asm_exc_invalid_op+0x1a/0x20 [ 2096.402387] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.407263] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2096.412152] try_get_cap_refs+0x433/0x880 [ceph] [ 2096.416786] __ceph_get_caps+0xf8/0x760 [ceph] [ 2096.421250] ? ceph_read_iter+0x134/0x860 [ceph] [ 2096.425877] ceph_read_iter+0x134/0x860 [ceph] [ 2096.430337] ? selinux_file_permission+0x119/0x150 [ 2096.435126] ? vfs_read+0x20f/0x300 [ 2096.438625] vfs_read+0x20f/0x300 [ 2096.441951] ksys_pread64+0x65/0xa0 [ 2096.445445] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2096.450592] do_syscall_64+0x38/0x80 [ 2096.454174] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2096.459228] RIP: 0033:0x7f48f0eeb88b [ 2096.462807] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2096.481552] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2096.489118] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2096.496243] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2096.503368] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2096.510500] R10: 0000000000007000 R11: 0000000000000246 R12: 0000000000001000 [ 2096.517634] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2096.524775] [ 2096.526968] irq event stamp: 12891941 [ 2096.530633] hardirqs last enabled at (12891951): [] console_unlock+0x106/0x120 [ 2096.539500] hardirqs last disabled at (12891960): [] console_unlock+0xeb/0x120 [ 2096.548285] softirqs last enabled at (12891980): [] __do_softirq+0x2be/0x371 [ 2096.556983] softirqs last disabled at (12891991): [] irq_exit_rcu+0xb0/0xd0 [ 2096.565500] ---[ end trace 0000000000000000 ]--- [ 2096.570121] ------------[ cut here ]------------ [ 2096.574741] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2096.586386] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2096.595083] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2096.650568] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2096.659694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2096.667172] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2096.672676] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2096.678160] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2096.696908] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2096.702136] RAX: 0000000000000000 RBX: ffff8881bf1a2720 RCX: 0000000000000000 [ 2096.709269] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2096.716403] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff4c0 [ 2096.723533] R10: 0000000000022268 R11: ffffc9000030b828 R12: ffff8881bf1a2e70 [ 2096.730666] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2096.737798] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2096.745884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2096.751630] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2096.758763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2096.765896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2096.773029] Call Trace: [ 2096.775481] [ 2096.777589] ? __warn+0x85/0x180 [ 2096.780824] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.785699] ? report_bug+0xfc/0x1d0 [ 2096.789281] ? prb_read_valid+0x1b/0x20 [ 2096.793123] ? handle_bug+0x44/0x70 [ 2096.796621] ? exc_invalid_op+0x17/0x60 [ 2096.800462] ? asm_exc_invalid_op+0x1a/0x20 [ 2096.804656] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.809537] ? __list_add_valid_or_report+0x7d/0xa0 [ 2096.814413] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2096.819315] ceph_add_cap+0x20b/0x620 [ceph] [ 2096.823611] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2096.828334] ? __d_lookup+0xd6/0x200 [ 2096.831920] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2096.837254] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2096.841726] ceph_con_process_message+0x74/0x150 [libceph] [ 2096.847228] ? calc_signature+0xdb/0x100 [libceph] [ 2096.852044] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2096.857375] ? process_scheduled_works+0x22a/0x510 [ 2096.862168] ? process_scheduled_works+0x22a/0x510 [ 2096.866962] ceph_con_workfn+0x344/0x750 [libceph] [ 2096.871776] ? process_scheduled_works+0x22a/0x510 [ 2096.876572] process_scheduled_works+0x2a4/0x510 [ 2096.881201] worker_thread+0x188/0x340 [ 2096.884953] ? rescuer_thread+0x310/0x310 [ 2096.888967] kthread+0xf5/0x130 [ 2096.892111] ? kthread_complete_and_exit+0x20/0x20 [ 2096.896906] ret_from_fork+0x31/0x40 [ 2096.900483] ? kthread_complete_and_exit+0x20/0x20 [ 2096.905277] ret_from_fork_asm+0x11/0x20 [ 2096.909213] [ 2096.911407] irq event stamp: 9267107 [ 2096.914985] hardirqs last enabled at (9267117): [] console_unlock+0x106/0x120 [ 2096.923762] hardirqs last disabled at (9267126): [] console_unlock+0xeb/0x120 [ 2096.932456] softirqs last enabled at (9267104): [] __do_softirq+0x2be/0x371 [ 2096.941062] softirqs last disabled at (9267099): [] irq_exit_rcu+0xb0/0xd0 [ 2096.949492] ---[ end trace 0000000000000000 ]--- [ 2096.954115] ------------[ cut here ]------------ [ 2096.958731] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2096.970377] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2096.979337] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2097.034812] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2097.043762] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2097.051241] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2097.056726] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2097.075474] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2097.080700] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2097.087833] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2097.094965] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff4fb [ 2097.102100] R10: 00000000000233e0 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2097.109246] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2097.116371] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2097.124457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2097.130195] CR2: 00007fb38e432038 CR3: 0000000120c1e006 CR4: 00000000003706f0 [ 2097.137328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2097.144461] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2097.151594] Call Trace: [ 2097.154046] [ 2097.156144] ? __warn+0x85/0x180 [ 2097.159378] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.164258] ? report_bug+0xfc/0x1d0 [ 2097.167837] ? prb_read_valid+0x1b/0x20 [ 2097.171677] ? handle_bug+0x44/0x70 [ 2097.175169] ? exc_invalid_op+0x17/0x60 [ 2097.179007] ? asm_exc_invalid_op+0x1a/0x20 [ 2097.183197] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.188074] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2097.192962] try_get_cap_refs+0x433/0x880 [ceph] [ 2097.197594] __ceph_get_caps+0xf8/0x760 [ceph] [ 2097.202052] ? ceph_read_iter+0x134/0x860 [ceph] [ 2097.206681] ceph_read_iter+0x134/0x860 [ceph] [ 2097.211141] ? selinux_file_permission+0x119/0x150 [ 2097.215931] ? vfs_read+0x20f/0x300 [ 2097.219421] vfs_read+0x20f/0x300 [ 2097.222746] ksys_pread64+0x65/0xa0 [ 2097.226233] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2097.231374] do_syscall_64+0x38/0x80 [ 2097.234951] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2097.240006] RIP: 0033:0x7f48f0eeb88b [ 2097.243583] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2097.262330] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2097.269895] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2097.277029] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2097.284162] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2097.291294] R10: 0000000000008000 R11: 0000000000000246 R12: 0000000000001000 [ 2097.298434] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2097.305577] [ 2097.307771] irq event stamp: 12893961 [ 2097.311437] hardirqs last enabled at (12893971): [] console_unlock+0x106/0x120 [ 2097.320302] hardirqs last disabled at (12893980): [] console_unlock+0xeb/0x120 [ 2097.329083] softirqs last enabled at (12893938): [] __do_softirq+0x2be/0x371 [ 2097.337776] softirqs last disabled at (12893999): [] irq_exit_rcu+0xb0/0xd0 [ 2097.346294] ---[ end trace 0000000000000000 ]--- [ 2097.350915] ------------[ cut here ]------------ [ 2097.355540] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2097.367196] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2097.375896] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2097.431454] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2097.440585] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2097.448065] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2097.453577] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2097.459070] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2097.477820] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2097.483051] RAX: 0000000000000000 RBX: ffff8881d807ce40 RCX: 0000000000000000 [ 2097.490186] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2097.497318] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff533 [ 2097.504454] R10: 0000000000024578 R11: ffffc9000030b828 R12: ffff8881d807d590 [ 2097.511597] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2097.518733] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2097.526821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2097.532573] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2097.539707] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2097.546843] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2097.553981] Call Trace: [ 2097.556434] [ 2097.558542] ? __warn+0x85/0x180 [ 2097.561785] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.566672] ? report_bug+0xfc/0x1d0 [ 2097.570258] ? prb_read_valid+0x1b/0x20 [ 2097.574103] ? handle_bug+0x44/0x70 [ 2097.577604] ? exc_invalid_op+0x17/0x60 [ 2097.581447] ? asm_exc_invalid_op+0x1a/0x20 [ 2097.585646] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.590534] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.595419] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2097.600321] ceph_add_cap+0x20b/0x620 [ceph] [ 2097.604627] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2097.609374] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2097.614721] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2097.619208] ceph_con_process_message+0x74/0x150 [libceph] [ 2097.624712] ? calc_signature+0xdb/0x100 [libceph] [ 2097.629538] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2097.634878] ? process_scheduled_works+0x22a/0x510 [ 2097.639686] ? process_scheduled_works+0x22a/0x510 [ 2097.644484] ceph_con_workfn+0x344/0x750 [libceph] [ 2097.649304] ? process_scheduled_works+0x22a/0x510 [ 2097.654102] process_scheduled_works+0x2a4/0x510 [ 2097.658740] worker_thread+0x188/0x340 [ 2097.662502] ? rescuer_thread+0x310/0x310 [ 2097.666520] kthread+0xf5/0x130 [ 2097.669667] ? kthread_complete_and_exit+0x20/0x20 [ 2097.674472] ret_from_fork+0x31/0x40 [ 2097.678059] ? kthread_complete_and_exit+0x20/0x20 [ 2097.682860] ret_from_fork_asm+0x11/0x20 [ 2097.686809] [ 2097.689002] irq event stamp: 9269007 [ 2097.692582] hardirqs last enabled at (9269017): [] console_unlock+0x106/0x120 [ 2097.701362] hardirqs last disabled at (9269026): [] console_unlock+0xeb/0x120 [ 2097.710065] softirqs last enabled at (9269042): [] __do_softirq+0x2be/0x371 [ 2097.718678] softirqs last disabled at (9269035): [] irq_exit_rcu+0xb0/0xd0 [ 2097.727119] ---[ end trace 0000000000000000 ]--- [ 2097.731739] ------------[ cut here ]------------ [ 2097.736352] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2097.747999] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2097.756959] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2097.812439] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2097.821405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2097.828882] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2097.834368] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2097.853113] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2097.858340] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2097.865473] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2097.872605] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff56d [ 2097.879740] R10: 00000000000256a8 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2097.886876] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2097.894006] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2097.902093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2097.907835] CR2: 00007fb38e43c008 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2097.914968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2097.922102] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2097.929233] Call Trace: [ 2097.931677] [ 2097.933784] ? __warn+0x85/0x180 [ 2097.937018] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.941897] ? report_bug+0xfc/0x1d0 [ 2097.945479] ? prb_read_valid+0x1b/0x20 [ 2097.949316] ? handle_bug+0x44/0x70 [ 2097.952807] ? exc_invalid_op+0x17/0x60 [ 2097.956648] ? asm_exc_invalid_op+0x1a/0x20 [ 2097.960839] ? __list_add_valid_or_report+0x7d/0xa0 [ 2097.965715] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2097.970602] try_get_cap_refs+0x433/0x880 [ceph] [ 2097.975235] __ceph_get_caps+0xf8/0x760 [ceph] [ 2097.979701] ? ceph_read_iter+0x134/0x860 [ceph] [ 2097.984330] ceph_read_iter+0x134/0x860 [ceph] [ 2097.988788] ? selinux_file_permission+0x119/0x150 [ 2097.993578] ? vfs_read+0x20f/0x300 [ 2097.997068] vfs_read+0x20f/0x300 [ 2098.000394] ksys_pread64+0x65/0xa0 [ 2098.003881] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2098.009023] do_syscall_64+0x38/0x80 [ 2098.012601] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2098.017655] RIP: 0033:0x7f48f0eeb88b [ 2098.021231] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2098.039980] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2098.047551] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2098.054677] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2098.061810] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2098.068944] R10: 0000000000009000 R11: 0000000000000246 R12: 0000000000001000 [ 2098.076078] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2098.083216] [ 2098.085400] irq event stamp: 12896031 [ 2098.089064] hardirqs last enabled at (12896041): [] console_unlock+0x106/0x120 [ 2098.097934] hardirqs last disabled at (12896050): [] console_unlock+0xeb/0x120 [ 2098.106714] softirqs last enabled at (12896008): [] __do_softirq+0x2be/0x371 [ 2098.115409] softirqs last disabled at (12896069): [] irq_exit_rcu+0xb0/0xd0 [ 2098.123932] ---[ end trace 0000000000000000 ]--- [ 2098.128547] ------------[ cut here ]------------ [ 2098.133168] list_del corruption. next->prev should be ffff8881d12d24a8, but was ffff8881f789cbc8. (next=ffff8881f789cbc8) [ 2098.144116] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:67 __list_del_entry_valid_or_report+0xc3/0xd0 [ 2098.153334] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2098.208820] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2098.217944] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2098.225425] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2098.230928] RIP: 0010:__list_del_entry_valid_or_report+0xc3/0xd0 [ 2098.236933] Code: 0b 31 c0 c3 48 89 d1 48 89 f2 48 89 fe 48 c7 c7 58 9a 1d 82 e8 ae 0b b0 ff 0f 0b 31 c0 c3 48 c7 c7 a0 9a 1d 82 e8 9d 0b b0 ff <0f> 0b 31 c0 c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 d2 74 27 [ 2098.255680] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2098.260905] RAX: 0000000000000000 RBX: ffff8881d12d1d58 RCX: 0000000000000000 [ 2098.268041] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2098.275172] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff5a5 [ 2098.282304] R10: 00000000000266d0 R11: ffffc9000030b828 R12: ffff8881d12d24a8 [ 2098.289437] R13: ffff88817f867488 R14: 00000000000005c0 R15: ffff888107955e10 [ 2098.296570] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2098.304658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2098.310401] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2098.317533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2098.324669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2098.331801] Call Trace: [ 2098.334253] [ 2098.336361] ? __warn+0x85/0x180 [ 2098.339596] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2098.344993] ? report_bug+0xfc/0x1d0 [ 2098.348571] ? prb_read_valid+0x1b/0x20 [ 2098.352415] ? handle_bug+0x44/0x70 [ 2098.355912] ? exc_invalid_op+0x17/0x60 [ 2098.359753] ? asm_exc_invalid_op+0x1a/0x20 [ 2098.363946] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2098.369345] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2098.374743] __cap_delay_requeue+0xd6/0x1d0 [ceph] [ 2098.379547] ceph_add_cap+0x20b/0x620 [ceph] [ 2098.383839] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2098.388562] ? __d_lookup+0xd6/0x200 [ 2098.392147] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2098.397484] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2098.401953] ceph_con_process_message+0x74/0x150 [libceph] [ 2098.407455] ? calc_signature+0xdb/0x100 [libceph] [ 2098.412266] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2098.417601] ? process_scheduled_works+0x22a/0x510 [ 2098.422397] ? process_scheduled_works+0x22a/0x510 [ 2098.427186] ceph_con_workfn+0x344/0x750 [libceph] [ 2098.431995] ? process_scheduled_works+0x22a/0x510 [ 2098.436791] process_scheduled_works+0x2a4/0x510 [ 2098.441419] worker_thread+0x188/0x340 [ 2098.445173] ? rescuer_thread+0x310/0x310 [ 2098.449186] kthread+0xf5/0x130 [ 2098.452330] ? kthread_complete_and_exit+0x20/0x20 [ 2098.457125] ret_from_fork+0x31/0x40 [ 2098.460704] ? kthread_complete_and_exit+0x20/0x20 [ 2098.465497] ret_from_fork_asm+0x11/0x20 [ 2098.469433] [ 2098.471624] irq event stamp: 9270931 [ 2098.475202] hardirqs last enabled at (9270941): [] console_unlock+0x106/0x120 [ 2098.483980] hardirqs last disabled at (9270950): [] console_unlock+0xeb/0x120 [ 2098.492674] softirqs last enabled at (9270928): [] __do_softirq+0x2be/0x371 [ 2098.501280] softirqs last disabled at (9270923): [] irq_exit_rcu+0xb0/0xd0 [ 2098.509713] ---[ end trace 0000000000000000 ]--- [ 2098.514334] ------------[ cut here ]------------ [ 2098.518950] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2098.530600] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2098.539296] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2098.594778] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2098.603908] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2098.611385] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2098.616888] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2098.622373] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2098.641118] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2098.646345] RAX: 0000000000000000 RBX: ffff8881d12d1d58 RCX: 0000000000000000 [ 2098.653479] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2098.660611] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff5e0 [ 2098.667746] R10: 0000000000027870 R11: ffffc9000030b828 R12: ffff8881d12d24a8 [ 2098.674877] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2098.682010] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2098.690097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2098.695842] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2098.702974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2098.710108] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2098.717240] Call Trace: [ 2098.719695] [ 2098.721801] ? __warn+0x85/0x180 [ 2098.725033] ? __list_add_valid_or_report+0x7d/0xa0 [ 2098.729914] ? report_bug+0xfc/0x1d0 [ 2098.733493] ? prb_read_valid+0x1b/0x20 [ 2098.737335] ? handle_bug+0x44/0x70 [ 2098.740831] ? exc_invalid_op+0x17/0x60 [ 2098.744672] ? asm_exc_invalid_op+0x1a/0x20 [ 2098.748867] ? __list_add_valid_or_report+0x7d/0xa0 [ 2098.753749] ? __list_add_valid_or_report+0x7d/0xa0 [ 2098.758633] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2098.763535] ceph_add_cap+0x20b/0x620 [ceph] [ 2098.767831] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2098.772554] ? __d_lookup+0xd6/0x200 [ 2098.776141] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2098.781477] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2098.785947] ceph_con_process_message+0x74/0x150 [libceph] [ 2098.791445] ? calc_signature+0xdb/0x100 [libceph] [ 2098.796252] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2098.801585] ? process_scheduled_works+0x22a/0x510 [ 2098.806380] ? process_scheduled_works+0x22a/0x510 [ 2098.811169] ceph_con_workfn+0x344/0x750 [libceph] [ 2098.815977] ? process_scheduled_works+0x22a/0x510 [ 2098.820774] process_scheduled_works+0x2a4/0x510 [ 2098.825403] worker_thread+0x188/0x340 [ 2098.829158] ? rescuer_thread+0x310/0x310 [ 2098.833168] kthread+0xf5/0x130 [ 2098.836315] ? kthread_complete_and_exit+0x20/0x20 [ 2098.841110] ret_from_fork+0x31/0x40 [ 2098.844686] ? kthread_complete_and_exit+0x20/0x20 [ 2098.849480] ret_from_fork_asm+0x11/0x20 [ 2098.853415] [ 2098.855607] irq event stamp: 9271953 [ 2098.859189] hardirqs last enabled at (9271969): [] console_unlock+0x106/0x120 [ 2098.867973] hardirqs last disabled at (9271978): [] console_unlock+0xeb/0x120 [ 2098.876667] softirqs last enabled at (9271966): [] __do_softirq+0x2be/0x371 [ 2098.885272] softirqs last disabled at (9271961): [] irq_exit_rcu+0xb0/0xd0 [ 2098.893705] ---[ end trace 0000000000000000 ]--- [ 2098.898328] ------------[ cut here ]------------ [ 2098.902947] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2098.914592] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2098.923546] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2098.979021] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2098.987973] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2098.995453] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2099.000938] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2099.019685] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2099.024911] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2099.032043] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2099.039192] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff61b [ 2099.046322] R10: 0000000000028b88 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2099.053450] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2099.060584] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2099.068671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2099.074415] CR2: 00007fb38e445070 CR3: 0000000120c1e006 CR4: 00000000003706f0 [ 2099.081540] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2099.088673] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2099.095806] Call Trace: [ 2099.098258] [ 2099.100365] ? __warn+0x85/0x180 [ 2099.103597] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.108478] ? report_bug+0xfc/0x1d0 [ 2099.112056] ? prb_read_valid+0x1b/0x20 [ 2099.115897] ? handle_bug+0x44/0x70 [ 2099.119390] ? exc_invalid_op+0x17/0x60 [ 2099.123229] ? asm_exc_invalid_op+0x1a/0x20 [ 2099.127418] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.132294] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2099.137184] try_get_cap_refs+0x433/0x880 [ceph] [ 2099.141816] __ceph_get_caps+0xf8/0x760 [ceph] [ 2099.146284] ? ceph_read_iter+0x134/0x860 [ceph] [ 2099.150916] ceph_read_iter+0x134/0x860 [ceph] [ 2099.155381] ? selinux_file_permission+0x119/0x150 [ 2099.160182] ? vfs_read+0x20f/0x300 [ 2099.163676] vfs_read+0x20f/0x300 [ 2099.167001] ksys_pread64+0x65/0xa0 [ 2099.170489] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2099.175631] do_syscall_64+0x38/0x80 [ 2099.179207] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2099.184261] RIP: 0033:0x7f48f0eeb88b [ 2099.187840] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2099.206585] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2099.214152] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2099.221285] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2099.228416] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2099.235541] R10: 000000000000a000 R11: 0000000000000246 R12: 0000000000001000 [ 2099.242676] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2099.249814] [ 2099.251999] irq event stamp: 12899115 [ 2099.255664] hardirqs last enabled at (12899125): [] console_unlock+0x106/0x120 [ 2099.264532] hardirqs last disabled at (12899134): [] console_unlock+0xeb/0x120 [ 2099.273310] softirqs last enabled at (12899092): [] __do_softirq+0x2be/0x371 [ 2099.282003] softirqs last disabled at (12899087): [] irq_exit_rcu+0xb0/0xd0 [ 2099.290523] ---[ end trace 0000000000000000 ]--- [ 2099.295144] ------------[ cut here ]------------ [ 2099.299767] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2099.311413] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2099.320104] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2099.375584] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2099.384706] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2099.392186] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2099.397681] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2099.403167] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2099.421914] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2099.427140] RAX: 0000000000000000 RBX: ffff8881d12d9390 RCX: 0000000000000000 [ 2099.434271] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2099.441407] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff653 [ 2099.448537] R10: 0000000000029b88 R11: ffffc9000030b828 R12: ffff8881d12d9ae0 [ 2099.455671] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2099.462804] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2099.470891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2099.476635] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2099.483769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2099.490903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2099.498034] Call Trace: [ 2099.500486] [ 2099.502593] ? __warn+0x85/0x180 [ 2099.505828] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.510705] ? report_bug+0xfc/0x1d0 [ 2099.514285] ? prb_read_valid+0x1b/0x20 [ 2099.518125] ? handle_bug+0x44/0x70 [ 2099.521616] ? exc_invalid_op+0x17/0x60 [ 2099.525457] ? asm_exc_invalid_op+0x1a/0x20 [ 2099.529646] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.534522] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.539401] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2099.544295] ceph_add_cap+0x20b/0x620 [ceph] [ 2099.548585] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2099.553309] ? __d_lookup+0xd6/0x200 [ 2099.556896] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2099.562220] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2099.566683] ceph_con_process_message+0x74/0x150 [libceph] [ 2099.572180] ? calc_signature+0xdb/0x100 [libceph] [ 2099.576993] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2099.582324] ? process_scheduled_works+0x22a/0x510 [ 2099.587114] ? process_scheduled_works+0x22a/0x510 [ 2099.591905] ceph_con_workfn+0x344/0x750 [libceph] [ 2099.596708] ? process_scheduled_works+0x22a/0x510 [ 2099.601498] process_scheduled_works+0x2a4/0x510 [ 2099.606124] worker_thread+0x188/0x340 [ 2099.609879] ? rescuer_thread+0x310/0x310 [ 2099.613892] kthread+0xf5/0x130 [ 2099.617039] ? kthread_complete_and_exit+0x20/0x20 [ 2099.621831] ret_from_fork+0x31/0x40 [ 2099.625411] ? kthread_complete_and_exit+0x20/0x20 [ 2099.630204] ret_from_fork_asm+0x11/0x20 [ 2099.634136] [ 2099.636323] irq event stamp: 9273851 [ 2099.639901] hardirqs last enabled at (9273861): [] console_unlock+0x106/0x120 [ 2099.648682] hardirqs last disabled at (9273870): [] console_unlock+0xeb/0x120 [ 2099.657372] softirqs last enabled at (9273848): [] __do_softirq+0x2be/0x371 [ 2099.665981] softirqs last disabled at (9273843): [] irq_exit_rcu+0xb0/0xd0 [ 2099.674413] ---[ end trace 0000000000000000 ]--- [ 2099.679033] ------------[ cut here ]------------ [ 2099.683656] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2099.695310] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2099.704262] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2099.759793] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2099.768744] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2099.776223] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2099.781710] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2099.800461] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2099.805691] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2099.812823] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2099.819957] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff68e [ 2099.827088] R10: 000000000002ad48 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2099.834222] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2099.841361] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2099.849451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2099.855196] CR2: 00007f9bd3845000 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2099.862330] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2099.869460] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2099.876594] Call Trace: [ 2099.879046] [ 2099.881154] ? __warn+0x85/0x180 [ 2099.884386] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.889266] ? report_bug+0xfc/0x1d0 [ 2099.892844] ? prb_read_valid+0x1b/0x20 [ 2099.896689] ? handle_bug+0x44/0x70 [ 2099.900185] ? exc_invalid_op+0x17/0x60 [ 2099.904025] ? asm_exc_invalid_op+0x1a/0x20 [ 2099.908221] ? __list_add_valid_or_report+0x7d/0xa0 [ 2099.913101] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2099.917999] try_get_cap_refs+0x433/0x880 [ceph] [ 2099.922644] __ceph_get_caps+0xf8/0x760 [ceph] [ 2099.927125] ? ceph_read_iter+0x134/0x860 [ceph] [ 2099.931759] ceph_read_iter+0x134/0x860 [ceph] [ 2099.936240] ? selinux_file_permission+0x119/0x150 [ 2099.941040] ? vfs_read+0x20f/0x300 [ 2099.944534] vfs_read+0x20f/0x300 [ 2099.947865] ksys_pread64+0x65/0xa0 [ 2099.951364] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2099.956505] do_syscall_64+0x38/0x80 [ 2099.960091] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2099.965144] RIP: 0033:0x7f48f0eeb88b [ 2099.968723] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2099.987469] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2099.995035] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2100.002168] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2100.009301] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2100.016434] R10: 000000000000b000 R11: 0000000000000246 R12: 0000000000001000 [ 2100.023569] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2100.030715] [ 2100.032911] irq event stamp: 12900927 [ 2100.036575] hardirqs last enabled at (12900937): [] console_unlock+0x106/0x120 [ 2100.045443] hardirqs last disabled at (12900946): [] console_unlock+0xeb/0x120 [ 2100.054222] softirqs last enabled at (12900768): [] __do_softirq+0x2be/0x371 [ 2100.062926] softirqs last disabled at (12900763): [] irq_exit_rcu+0xb0/0xd0 [ 2100.071455] ---[ end trace 0000000000000000 ]--- [ 2100.076089] ------------[ cut here ]------------ [ 2100.080711] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2100.092369] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2100.101064] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2100.156559] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2100.165684] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2100.173163] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2100.178676] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2100.184159] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2100.202907] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2100.208132] RAX: 0000000000000000 RBX: ffff8881d12d61d0 RCX: 0000000000000000 [ 2100.215266] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2100.222401] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff6c6 [ 2100.229532] R10: 000000000002bea0 R11: ffffc9000030b828 R12: ffff8881d12d6920 [ 2100.236666] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2100.243797] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2100.251884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2100.257628] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2100.264764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2100.271895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2100.279028] Call Trace: [ 2100.281482] [ 2100.283587] ? __warn+0x85/0x180 [ 2100.286821] ? __list_add_valid_or_report+0x7d/0xa0 [ 2100.291700] ? report_bug+0xfc/0x1d0 [ 2100.295277] ? prb_read_valid+0x1b/0x20 [ 2100.299118] ? handle_bug+0x44/0x70 [ 2100.302610] ? exc_invalid_op+0x17/0x60 [ 2100.306451] ? asm_exc_invalid_op+0x1a/0x20 [ 2100.310641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2100.315517] ? __list_add_valid_or_report+0x7d/0xa0 [ 2100.320395] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2100.325297] ceph_add_cap+0x20b/0x620 [ceph] [ 2100.329588] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2100.334313] ? __d_lookup+0xd6/0x200 [ 2100.337897] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2100.343223] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2100.347686] ceph_con_process_message+0x74/0x150 [libceph] [ 2100.353184] ? calc_signature+0xdb/0x100 [libceph] [ 2100.357995] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2100.363325] ? process_scheduled_works+0x22a/0x510 [ 2100.368118] ? process_scheduled_works+0x22a/0x510 [ 2100.372908] ceph_con_workfn+0x344/0x750 [libceph] [ 2100.377712] ? process_scheduled_works+0x22a/0x510 [ 2100.382500] process_scheduled_works+0x2a4/0x510 [ 2100.387125] worker_thread+0x188/0x340 [ 2100.390874] ? rescuer_thread+0x310/0x310 [ 2100.394885] kthread+0xf5/0x130 [ 2100.398033] ? kthread_complete_and_exit+0x20/0x20 [ 2100.402827] ret_from_fork+0x31/0x40 [ 2100.406404] ? kthread_complete_and_exit+0x20/0x20 [ 2100.411198] ret_from_fork_asm+0x11/0x20 [ 2100.415132] [ 2100.417325] irq event stamp: 9275755 [ 2100.420903] hardirqs last enabled at (9275765): [] console_unlock+0x106/0x120 [ 2100.429683] hardirqs last disabled at (9275774): [] console_unlock+0xeb/0x120 [ 2100.438376] softirqs last enabled at (9275752): [] __do_softirq+0x2be/0x371 [ 2100.446982] softirqs last disabled at (9275747): [] irq_exit_rcu+0xb0/0xd0 [ 2100.455416] ---[ end trace 0000000000000000 ]--- [ 2100.460034] ------------[ cut here ]------------ [ 2100.464655] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2100.476303] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2100.485260] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2100.540735] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2100.549684] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2100.557163] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2100.562649] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2100.581395] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2100.586633] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2100.593769] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2100.600895] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff701 [ 2100.608029] R10: 000000000002d1b8 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2100.615162] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2100.622294] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2100.630380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2100.636125] CR2: 0000561d8ba02068 CR3: 0000000120c1e002 CR4: 00000000003706f0 [ 2100.643250] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2100.650382] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2100.657506] Call Trace: [ 2100.659950] [ 2100.662050] ? __warn+0x85/0x180 [ 2100.665281] ? __list_add_valid_or_report+0x7d/0xa0 [ 2100.670163] ? report_bug+0xfc/0x1d0 [ 2100.673739] ? prb_read_valid+0x1b/0x20 [ 2100.677582] ? handle_bug+0x44/0x70 [ 2100.681071] ? exc_invalid_op+0x17/0x60 [ 2100.684903] ? asm_exc_invalid_op+0x1a/0x20 [ 2100.689101] ? __list_add_valid_or_report+0x7d/0xa0 [ 2100.693978] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2100.698868] try_get_cap_refs+0x433/0x880 [ceph] [ 2100.703501] __ceph_get_caps+0xf8/0x760 [ceph] [ 2100.707966] ? ceph_read_iter+0x134/0x860 [ceph] [ 2100.712594] ceph_read_iter+0x134/0x860 [ceph] [ 2100.717054] ? selinux_file_permission+0x119/0x150 [ 2100.721847] ? vfs_read+0x20f/0x300 [ 2100.725344] vfs_read+0x20f/0x300 [ 2100.728668] ksys_pread64+0x65/0xa0 [ 2100.732156] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2100.737296] do_syscall_64+0x38/0x80 [ 2100.740875] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2100.745927] RIP: 0033:0x7f48f0eeb88b [ 2100.749505] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2100.768251] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2100.775819] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2100.782951] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2100.790083] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2100.797207] R10: 000000000000c000 R11: 0000000000000246 R12: 0000000000001000 [ 2100.804348] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2100.811491] [ 2100.813684] irq event stamp: 12902867 [ 2100.817349] hardirqs last enabled at (12902877): [] console_unlock+0x106/0x120 [ 2100.826215] hardirqs last disabled at (12902886): [] console_unlock+0xeb/0x120 [ 2100.834996] softirqs last enabled at (12902844): [] __do_softirq+0x2be/0x371 [ 2100.843689] softirqs last disabled at (12902839): [] irq_exit_rcu+0xb0/0xd0 [ 2100.852209] ---[ end trace 0000000000000000 ]--- [ 2100.856827] ------------[ cut here ]------------ [ 2100.861450] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2100.873099] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2100.881788] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2100.937272] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2100.946400] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2100.953881] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2100.959389] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2100.964877] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2100.983623] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2100.988849] RAX: 0000000000000000 RBX: ffff8881d12da720 RCX: 0000000000000000 [ 2100.995983] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2101.003118] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff739 [ 2101.010247] R10: 000000000002e350 R11: ffffc9000030b828 R12: ffff8881d12dae70 [ 2101.017381] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2101.024514] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2101.032601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2101.038346] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2101.045479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2101.052610] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2101.059743] Call Trace: [ 2101.062196] [ 2101.064303] ? __warn+0x85/0x180 [ 2101.067536] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.072416] ? report_bug+0xfc/0x1d0 [ 2101.075994] ? prb_read_valid+0x1b/0x20 [ 2101.079837] ? handle_bug+0x44/0x70 [ 2101.083327] ? exc_invalid_op+0x17/0x60 [ 2101.087167] ? asm_exc_invalid_op+0x1a/0x20 [ 2101.091357] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.096234] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.101112] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2101.106012] ceph_add_cap+0x20b/0x620 [ceph] [ 2101.110307] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2101.115029] ? __d_lookup+0xd6/0x200 [ 2101.118613] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2101.123940] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2101.128405] ceph_con_process_message+0x74/0x150 [libceph] [ 2101.133901] ? calc_signature+0xdb/0x100 [libceph] [ 2101.138712] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2101.144043] ? process_scheduled_works+0x22a/0x510 [ 2101.148843] ? process_scheduled_works+0x22a/0x510 [ 2101.153631] ceph_con_workfn+0x344/0x750 [libceph] [ 2101.158436] ? process_scheduled_works+0x22a/0x510 [ 2101.163227] process_scheduled_works+0x2a4/0x510 [ 2101.167853] worker_thread+0x188/0x340 [ 2101.171608] ? rescuer_thread+0x310/0x310 [ 2101.175619] kthread+0xf5/0x130 [ 2101.178767] ? kthread_complete_and_exit+0x20/0x20 [ 2101.183559] ret_from_fork+0x31/0x40 [ 2101.187138] ? kthread_complete_and_exit+0x20/0x20 [ 2101.191932] ret_from_fork_asm+0x11/0x20 [ 2101.195864] [ 2101.198060] irq event stamp: 9277643 [ 2101.201638] hardirqs last enabled at (9277653): [] console_unlock+0x106/0x120 [ 2101.210417] hardirqs last disabled at (9277662): [] console_unlock+0xeb/0x120 [ 2101.219111] softirqs last enabled at (9277640): [] __do_softirq+0x2be/0x371 [ 2101.227715] softirqs last disabled at (9277635): [] irq_exit_rcu+0xb0/0xd0 [ 2101.236148] ---[ end trace 0000000000000000 ]--- [ 2101.240770] ------------[ cut here ]------------ [ 2101.245399] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2101.257085] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2101.266057] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2101.321555] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2101.330526] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2101.338013] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2101.343499] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2101.362244] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2101.367469] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2101.374599] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2101.381730] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff774 [ 2101.388859] R10: 000000000002f4d0 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2101.395989] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2101.403114] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2101.411199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2101.416939] CR2: 00007fb38e467060 CR3: 0000000120c1e006 CR4: 00000000003706e0 [ 2101.424073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2101.431203] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2101.438335] Call Trace: [ 2101.440781] [ 2101.442888] ? __warn+0x85/0x180 [ 2101.446119] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.450999] ? report_bug+0xfc/0x1d0 [ 2101.454579] ? prb_read_valid+0x1b/0x20 [ 2101.458418] ? handle_bug+0x44/0x70 [ 2101.461910] ? exc_invalid_op+0x17/0x60 [ 2101.465749] ? asm_exc_invalid_op+0x1a/0x20 [ 2101.469940] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.474820] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2101.479714] try_get_cap_refs+0x433/0x880 [ceph] [ 2101.484347] __ceph_get_caps+0xf8/0x760 [ceph] [ 2101.488812] ? ceph_read_iter+0x134/0x860 [ceph] [ 2101.493447] ceph_read_iter+0x134/0x860 [ceph] [ 2101.497909] ? selinux_file_permission+0x119/0x150 [ 2101.502703] ? vfs_read+0x20f/0x300 [ 2101.506198] vfs_read+0x20f/0x300 [ 2101.509522] ksys_pread64+0x65/0xa0 [ 2101.513010] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2101.518153] do_syscall_64+0x38/0x80 [ 2101.521730] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2101.526783] RIP: 0033:0x7f48f0eeb88b [ 2101.530360] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2101.549109] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2101.556680] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2101.563809] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2101.570939] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2101.578071] R10: 000000000000d000 R11: 0000000000000246 R12: 0000000000001000 [ 2101.585206] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2101.592343] [ 2101.594530] irq event stamp: 12905839 [ 2101.598201] hardirqs last enabled at (12905849): [] console_unlock+0x106/0x120 [ 2101.607071] hardirqs last disabled at (12905858): [] console_unlock+0xeb/0x120 [ 2101.615851] softirqs last enabled at (12905816): [] __do_softirq+0x2be/0x371 [ 2101.624543] softirqs last disabled at (12905877): [] irq_exit_rcu+0xb0/0xd0 [ 2101.633064] ---[ end trace 0000000000000000 ]--- [ 2101.637683] ------------[ cut here ]------------ [ 2101.642305] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2101.653948] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2101.662643] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2101.718122] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2101.727246] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2101.734725] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2101.740219] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2101.745706] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2101.764454] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2101.769677] RAX: 0000000000000000 RBX: ffff8881d12d7560 RCX: 0000000000000000 [ 2101.776811] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2101.783949] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff7ac [ 2101.791077] R10: 0000000000030668 R11: ffffc9000030b828 R12: ffff8881d12d7cb0 [ 2101.798211] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2101.805345] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2101.813429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2101.819174] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2101.826307] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2101.833441] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2101.840576] Call Trace: [ 2101.843027] [ 2101.845132] ? __warn+0x85/0x180 [ 2101.848368] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.853245] ? report_bug+0xfc/0x1d0 [ 2101.856825] ? prb_read_valid+0x1b/0x20 [ 2101.860665] ? handle_bug+0x44/0x70 [ 2101.864155] ? exc_invalid_op+0x17/0x60 [ 2101.868000] ? asm_exc_invalid_op+0x1a/0x20 [ 2101.872195] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.877071] ? __list_add_valid_or_report+0x7d/0xa0 [ 2101.881948] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2101.886840] ceph_add_cap+0x20b/0x620 [ceph] [ 2101.891134] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2101.895858] ? __d_lookup+0xd6/0x200 [ 2101.899441] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2101.904770] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2101.909231] ceph_con_process_message+0x74/0x150 [libceph] [ 2101.914730] ? calc_signature+0xdb/0x100 [libceph] [ 2101.919546] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2101.924872] ? process_scheduled_works+0x22a/0x510 [ 2101.929672] ? process_scheduled_works+0x22a/0x510 [ 2101.934461] ceph_con_workfn+0x344/0x750 [libceph] [ 2101.939264] ? process_scheduled_works+0x22a/0x510 [ 2101.944055] process_scheduled_works+0x2a4/0x510 [ 2101.948682] worker_thread+0x188/0x340 [ 2101.952438] ? rescuer_thread+0x310/0x310 [ 2101.956449] kthread+0xf5/0x130 [ 2101.959594] ? kthread_complete_and_exit+0x20/0x20 [ 2101.964388] ret_from_fork+0x31/0x40 [ 2101.967969] ? kthread_complete_and_exit+0x20/0x20 [ 2101.972760] ret_from_fork_asm+0x11/0x20 [ 2101.976694] [ 2101.978889] irq event stamp: 9279539 [ 2101.982467] hardirqs last enabled at (9279549): [] console_unlock+0x106/0x120 [ 2101.991246] hardirqs last disabled at (9279558): [] console_unlock+0xeb/0x120 [ 2101.999938] softirqs last enabled at (9279536): [] __do_softirq+0x2be/0x371 [ 2102.008545] softirqs last disabled at (9279531): [] irq_exit_rcu+0xb0/0xd0 [ 2102.016977] ---[ end trace 0000000000000000 ]--- [ 2102.021600] ------------[ cut here ]------------ [ 2102.026219] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2102.037865] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2102.046821] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2102.102297] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2102.111248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2102.118727] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2102.124214] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2102.142980] RSP: 0018:ffffc90001fd7bd0 EFLAGS: 00010282 [ 2102.148213] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2102.155343] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2102.162478] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff7e7 [ 2102.169613] R10: 00000000000316d0 R11: ffffc90001fd7a60 R12: ffff8881c3031118 [ 2102.176745] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2102.183876] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2102.191963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2102.197706] CR2: 00007fb38e470018 CR3: 0000000120c1e002 CR4: 00000000003706f0 [ 2102.204841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2102.211974] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2102.219108] Call Trace: [ 2102.221558] [ 2102.223666] ? __warn+0x85/0x180 [ 2102.226899] ? __list_add_valid_or_report+0x7d/0xa0 [ 2102.231783] ? report_bug+0xfc/0x1d0 [ 2102.235368] ? prb_read_valid+0x1b/0x20 [ 2102.239210] ? handle_bug+0x44/0x70 [ 2102.242707] ? exc_invalid_op+0x17/0x60 [ 2102.246547] ? asm_exc_invalid_op+0x1a/0x20 [ 2102.250740] ? __list_add_valid_or_report+0x7d/0xa0 [ 2102.255624] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2102.260521] try_get_cap_refs+0x433/0x880 [ceph] [ 2102.265172] __ceph_get_caps+0xf8/0x760 [ceph] [ 2102.269641] ? ceph_read_iter+0x134/0x860 [ceph] [ 2102.274281] ceph_read_iter+0x134/0x860 [ceph] [ 2102.278752] ? selinux_file_permission+0x119/0x150 [ 2102.283554] ? vfs_read+0x20f/0x300 [ 2102.287046] vfs_read+0x20f/0x300 [ 2102.290375] ksys_pread64+0x65/0xa0 [ 2102.293867] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2102.299013] do_syscall_64+0x38/0x80 [ 2102.302596] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2102.307648] RIP: 0033:0x7f48f0eeb88b [ 2102.311225] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2102.329973] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2102.337537] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2102.344671] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2102.351805] RBP: 0000000000002a12 R08: 0000000000000063 R09: 0000000000000003 [ 2102.358937] R10: 000000000000e000 R11: 0000000000000246 R12: 0000000000001000 [ 2102.366072] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183d8a0 [ 2102.373214] [ 2102.375403] irq event stamp: 12907783 [ 2102.379071] hardirqs last enabled at (12907793): [] console_unlock+0x106/0x120 [ 2102.387944] hardirqs last disabled at (12907802): [] console_unlock+0xeb/0x120 [ 2102.396726] softirqs last enabled at (12907760): [] __do_softirq+0x2be/0x371 [ 2102.405424] softirqs last disabled at (12907755): [] irq_exit_rcu+0xb0/0xd0 [ 2102.413945] ---[ end trace 0000000000000000 ]--- [ 2102.418564] ------------[ cut here ]------------ [ 2102.423185] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2102.434836] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2102.443538] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2102.499088] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2102.508218] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2102.515698] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2102.521207] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2102.526694] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2102.545445] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2102.550677] RAX: 0000000000000000 RBX: ffff8881c3fc61d0 RCX: 0000000000000000 [ 2102.557808] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2102.564941] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff81f [ 2102.572073] R10: 0000000000032688 R11: ffffc9000030b828 R12: ffff8881c3fc6920 [ 2102.579210] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2102.586348] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2102.594435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2102.600178] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2102.607316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2102.614458] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2102.621597] Call Trace: [ 2102.624049] [ 2102.626157] ? __warn+0x85/0x180 [ 2102.629398] ? __list_add_valid_or_report+0x7d/0xa0 [ 2102.634286] ? report_bug+0xfc/0x1d0 [ 2102.637874] ? prb_read_valid+0x1b/0x20 [ 2102.641725] ? handle_bug+0x44/0x70 [ 2102.645231] ? exc_invalid_op+0x17/0x60 [ 2102.649071] ? asm_exc_invalid_op+0x1a/0x20 [ 2102.653278] ? __list_add_valid_or_report+0x7d/0xa0 [ 2102.658165] ? __list_add_valid_or_report+0x7d/0xa0 [ 2102.663051] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2102.667951] ceph_add_cap+0x20b/0x620 [ceph] [ 2102.672257] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2102.676997] ? __d_lookup+0xd6/0x200 [ 2102.680591] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2102.685942] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2102.690429] ceph_con_process_message+0x74/0x150 [libceph] [ 2102.695933] ? calc_signature+0xdb/0x100 [libceph] [ 2102.700757] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2102.706098] ? process_scheduled_works+0x22a/0x510 [ 2102.710911] ? process_scheduled_works+0x22a/0x510 [ 2102.715710] ceph_con_workfn+0x344/0x750 [libceph] [ 2102.720525] ? process_scheduled_works+0x22a/0x510 [ 2102.725320] process_scheduled_works+0x2a4/0x510 [ 2102.729951] worker_thread+0x188/0x340 [ 2102.733714] ? rescuer_thread+0x310/0x310 [ 2102.737732] kthread+0xf5/0x130 [ 2102.740888] ? kthread_complete_and_exit+0x20/0x20 [ 2102.745694] ret_from_fork+0x31/0x40 [ 2102.749276] ? kthread_complete_and_exit+0x20/0x20 [ 2102.754070] ret_from_fork_asm+0x11/0x20 [ 2102.758019] [ 2102.760215] irq event stamp: 9281541 [ 2102.763793] hardirqs last enabled at (9281551): [] console_unlock+0x106/0x120 [ 2102.772581] hardirqs last disabled at (9281560): [] console_unlock+0xeb/0x120 [ 2102.781276] softirqs last enabled at (9281576): [] __do_softirq+0x2be/0x371 [ 2102.789890] softirqs last disabled at (9281569): [] irq_exit_rcu+0xb0/0xd0 [ 2102.798329] ---[ end trace 0000000000000000 ]--- [ 2102.802951] ------------[ cut here ]------------ [ 2102.807575] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2102.819224] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2102.828178] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2102.883715] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2102.892682] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2102.900169] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2102.905661] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2102.924418] RSP: 0018:ffffc90001fd7be0 EFLAGS: 00010286 [ 2102.929648] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2102.936781] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2102.943920] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff85a [ 2102.951057] R10: 0000000000033888 R11: ffffc90001fd7a70 R12: ffff8881c3031118 [ 2102.958191] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2102.965331] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2102.973416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2102.979161] CR2: 00007f9bd386b020 CR3: 0000000120c1e006 CR4: 00000000003706e0 [ 2102.986306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2102.993446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2103.000577] Call Trace: [ 2103.003032] [ 2103.005141] ? __warn+0x85/0x180 [ 2103.008382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.013267] ? report_bug+0xfc/0x1d0 [ 2103.016845] ? prb_read_valid+0x1b/0x20 [ 2103.020690] ? handle_bug+0x44/0x70 [ 2103.024189] ? exc_invalid_op+0x17/0x60 [ 2103.028028] ? asm_exc_invalid_op+0x1a/0x20 [ 2103.032230] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.037112] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2103.042017] try_get_cap_refs+0x433/0x880 [ceph] [ 2103.046653] ? _raw_spin_unlock+0x23/0x40 [ 2103.050681] __ceph_get_caps+0xf8/0x760 [ceph] [ 2103.055154] ? __vfs_getxattr+0x8d/0xb0 [ 2103.059000] ? dentry_needs_remove_privs+0x31/0x50 [ 2103.063801] ? __file_remove_privs+0x89/0x110 [ 2103.068163] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2103.072886] ? lock_release+0xc5/0x260 [ 2103.076644] ? ceph_write_iter+0x260/0x770 [ceph] [ 2103.081362] ceph_write_iter+0x260/0x770 [ceph] [ 2103.085920] ? lock_is_held_type+0xaa/0x120 [ 2103.090114] vfs_write+0x324/0x4f0 [ 2103.093532] ksys_pwrite64+0x65/0xa0 [ 2103.097117] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2103.102260] do_syscall_64+0x38/0x80 [ 2103.105846] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2103.110902] RIP: 0033:0x7f48f0eeb93b [ 2103.114487] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2103.133232] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2103.140798] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb93b [ 2103.147931] RDX: 0000000000002000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2103.155069] RBP: 0000000000002a12 R08: 0000000000002000 R09: 0000000000000003 [ 2103.162214] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2103.169347] R13: 0000000000002000 R14: 0000000000002000 R15: 000000000207e000 [ 2103.176495] [ 2103.178690] irq event stamp: 12910079 [ 2103.182358] hardirqs last enabled at (12910095): [] console_unlock+0x106/0x120 [ 2103.191230] hardirqs last disabled at (12910104): [] console_unlock+0xeb/0x120 [ 2103.200009] softirqs last enabled at (12910092): [] __do_softirq+0x2be/0x371 [ 2103.208702] softirqs last disabled at (12910087): [] irq_exit_rcu+0xb0/0xd0 [ 2103.217226] ---[ end trace 0000000000000000 ]--- [ 2103.221844] ------------[ cut here ]------------ [ 2103.226462] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2103.238111] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2103.246813] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2103.302359] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2103.311492] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2103.318973] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2103.324484] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2103.329970] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2103.348716] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2103.353943] RAX: 0000000000000000 RBX: ffff8881d11b4478 RCX: 0000000000000000 [ 2103.361076] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2103.368212] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff897 [ 2103.375352] R10: 00000000000348f0 R11: ffffc9000030b828 R12: ffff8881d11b4bc8 [ 2103.382481] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2103.389617] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2103.397702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2103.403448] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2103.410580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2103.417713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2103.424847] Call Trace: [ 2103.427300] [ 2103.429406] ? __warn+0x85/0x180 [ 2103.432641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.437528] ? report_bug+0xfc/0x1d0 [ 2103.441115] ? prb_read_valid+0x1b/0x20 [ 2103.444959] ? handle_bug+0x44/0x70 [ 2103.448454] ? exc_invalid_op+0x17/0x60 [ 2103.452296] ? asm_exc_invalid_op+0x1a/0x20 [ 2103.456500] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.461388] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.466275] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2103.471176] ceph_add_cap+0x20b/0x620 [ceph] [ 2103.475483] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2103.480216] ? __d_lookup+0xd6/0x200 [ 2103.483808] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2103.489165] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2103.493650] ceph_con_process_message+0x74/0x150 [libceph] [ 2103.499157] ? calc_signature+0xdb/0x100 [libceph] [ 2103.503982] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2103.509320] ? process_scheduled_works+0x22a/0x510 [ 2103.514121] ? process_scheduled_works+0x22a/0x510 [ 2103.518916] ceph_con_workfn+0x344/0x750 [libceph] [ 2103.523732] ? process_scheduled_works+0x22a/0x510 [ 2103.528528] process_scheduled_works+0x2a4/0x510 [ 2103.533162] worker_thread+0x188/0x340 [ 2103.536920] ? rescuer_thread+0x310/0x310 [ 2103.540939] kthread+0xf5/0x130 [ 2103.544085] ? kthread_complete_and_exit+0x20/0x20 [ 2103.548889] ret_from_fork+0x31/0x40 [ 2103.552475] ? kthread_complete_and_exit+0x20/0x20 [ 2103.557277] ret_from_fork_asm+0x11/0x20 [ 2103.561224] [ 2103.563422] irq event stamp: 9283535 [ 2103.566998] hardirqs last enabled at (9283545): [] console_unlock+0x106/0x120 [ 2103.575780] hardirqs last disabled at (9283554): [] console_unlock+0xeb/0x120 [ 2103.584481] softirqs last enabled at (9283532): [] __do_softirq+0x2be/0x371 [ 2103.593095] softirqs last disabled at (9283527): [] irq_exit_rcu+0xb0/0xd0 [ 2103.601529] ---[ end trace 0000000000000000 ]--- [ 2103.606151] ------------[ cut here ]------------ [ 2103.610773] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2103.622421] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2103.631376] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2103.686896] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2103.695847] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2103.703329] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2103.708812] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2103.727558] RSP: 0018:ffffc90001fd7d18 EFLAGS: 00010286 [ 2103.732784] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2103.739919] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2103.747050] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff8d2 [ 2103.754185] R10: 0000000000035ad0 R11: ffffc90001fd7ba8 R12: ffff8881c3031118 [ 2103.761317] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2103.768450] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2103.776535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2103.782282] CR2: 00007f06eda07000 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2103.789414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2103.796551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2103.803690] Call Trace: [ 2103.806143] [ 2103.808249] ? __warn+0x85/0x180 [ 2103.811481] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.816363] ? report_bug+0xfc/0x1d0 [ 2103.819947] ? prb_read_valid+0x1b/0x20 [ 2103.823791] ? handle_bug+0x44/0x70 [ 2103.827288] ? exc_invalid_op+0x17/0x60 [ 2103.831129] ? asm_exc_invalid_op+0x1a/0x20 [ 2103.835325] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.840205] ? __list_add_valid_or_report+0x7d/0xa0 [ 2103.845092] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2103.849991] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2103.855154] ceph_write_iter+0x5aa/0x770 [ceph] [ 2103.859709] ? lock_is_held_type+0xaa/0x120 [ 2103.863908] vfs_write+0x324/0x4f0 [ 2103.867322] ksys_pwrite64+0x65/0xa0 [ 2103.870901] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2103.876044] do_syscall_64+0x38/0x80 [ 2103.879628] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2103.884680] RIP: 0033:0x7f48f0eeb93b [ 2103.888261] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2103.907007] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2103.914572] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb93b [ 2103.921705] RDX: 0000000000002000 RSI: 000000000207e000 RDI: 0000000000000008 [ 2103.928839] RBP: 0000000000002a12 R08: 0000000000002000 R09: 0000000000000003 [ 2103.935972] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183d8a0 [ 2103.943104] R13: 0000000000002000 R14: 0000000000002000 R15: 000000000207e000 [ 2103.950250] [ 2103.952438] irq event stamp: 12911863 [ 2103.956105] hardirqs last enabled at (12911873): [] console_unlock+0x106/0x120 [ 2103.964970] hardirqs last disabled at (12911882): [] console_unlock+0xeb/0x120 [ 2103.973753] softirqs last enabled at (12911896): [] __do_softirq+0x2be/0x371 [ 2103.982450] softirqs last disabled at (12911891): [] irq_exit_rcu+0xb0/0xd0 [ 2103.990970] ---[ end trace 0000000000000000 ]--- [ 2103.995591] ------------[ cut here ]------------ [ 2104.000208] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2104.011856] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2104.020549] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2104.076038] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2104.085174] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2104.092652] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2104.098160] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2104.103645] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2104.122391] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2104.127619] RAX: 0000000000000000 RBX: ffff8881d0bda720 RCX: 0000000000000000 [ 2104.134750] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2104.141885] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff908 [ 2104.149016] R10: 0000000000036a48 R11: ffffc9000030b828 R12: ffff8881d0bdae70 [ 2104.156150] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2104.163283] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2104.171369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2104.177115] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2104.184249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2104.191380] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2104.198513] Call Trace: [ 2104.200965] [ 2104.203073] ? __warn+0x85/0x180 [ 2104.206308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2104.211186] ? report_bug+0xfc/0x1d0 [ 2104.214763] ? prb_read_valid+0x1b/0x20 [ 2104.218605] ? handle_bug+0x44/0x70 [ 2104.222094] ? exc_invalid_op+0x17/0x60 [ 2104.225935] ? asm_exc_invalid_op+0x1a/0x20 [ 2104.230127] ? __list_add_valid_or_report+0x7d/0xa0 [ 2104.235011] ? __list_add_valid_or_report+0x7d/0xa0 [ 2104.239888] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2104.244780] ceph_add_cap+0x20b/0x620 [ceph] [ 2104.249064] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2104.253789] ? __d_lookup+0xd6/0x200 [ 2104.257374] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2104.262701] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2104.267167] ceph_con_process_message+0x74/0x150 [libceph] [ 2104.272670] ? calc_signature+0xdb/0x100 [libceph] [ 2104.277479] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2104.282802] ? process_scheduled_works+0x22a/0x510 [ 2104.287593] ? process_scheduled_works+0x22a/0x510 [ 2104.292383] ceph_con_workfn+0x344/0x750 [libceph] [ 2104.297187] ? process_scheduled_works+0x22a/0x510 [ 2104.301978] process_scheduled_works+0x2a4/0x510 [ 2104.306602] worker_thread+0x188/0x340 [ 2104.310350] ? rescuer_thread+0x310/0x310 [ 2104.314363] kthread+0xf5/0x130 [ 2104.317508] ? kthread_complete_and_exit+0x20/0x20 [ 2104.322302] ret_from_fork+0x31/0x40 [ 2104.325882] ? kthread_complete_and_exit+0x20/0x20 [ 2104.330675] ret_from_fork_asm+0x11/0x20 [ 2104.334607] [ 2104.336802] irq event stamp: 9285455 [ 2104.340380] hardirqs last enabled at (9285465): [] console_unlock+0x106/0x120 [ 2104.349163] hardirqs last disabled at (9285474): [] console_unlock+0xeb/0x120 [ 2104.357852] softirqs last enabled at (9285452): [] __do_softirq+0x2be/0x371 [ 2104.366459] softirqs last disabled at (9285447): [] irq_exit_rcu+0xb0/0xd0 [ 2104.374892] ---[ end trace 0000000000000000 ]--- [ 2104.379656] ------------[ cut here ]------------ [ 2104.384269] list_del corruption. next->prev should be ffff8881d12d3838, but was ffff8881d12d24a8. (next=ffff8881d12d24a8) [ 2104.395221] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:67 __list_del_entry_valid_or_report+0xc3/0xd0 [ 2104.404440] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2104.459921] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2104.469051] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2104.476527] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2104.482023] RIP: 0010:__list_del_entry_valid_or_report+0xc3/0xd0 [ 2104.488028] Code: 0b 31 c0 c3 48 89 d1 48 89 f2 48 89 fe 48 c7 c7 58 9a 1d 82 e8 ae 0b b0 ff 0f 0b 31 c0 c3 48 c7 c7 a0 9a 1d 82 e8 9d 0b b0 ff <0f> 0b 31 c0 c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 d2 74 27 [ 2104.506792] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2104.512035] RAX: 0000000000000000 RBX: ffff8881d12d30e8 RCX: 0000000000000000 [ 2104.519191] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2104.526320] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff943 [ 2104.533453] R10: 0000000000037bf8 R11: ffffc9000030b828 R12: ffff8881d12d3838 [ 2104.540585] R13: ffff88817f867488 R14: 00000000000005bf R15: ffff888105d30f00 [ 2104.547734] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2104.555838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2104.561599] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2104.568743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2104.575892] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2104.583043] Call Trace: [ 2104.585513] [ 2104.587637] ? __warn+0x85/0x180 [ 2104.590886] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2104.596305] ? report_bug+0xfc/0x1d0 [ 2104.599883] ? prb_read_valid+0x1b/0x20 [ 2104.603740] ? handle_bug+0x44/0x70 [ 2104.607250] ? exc_invalid_op+0x17/0x60 [ 2104.611089] ? asm_exc_invalid_op+0x1a/0x20 [ 2104.615296] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2104.620691] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2104.626107] __cap_delay_requeue+0xd6/0x1d0 [ceph] [ 2104.630928] ceph_add_cap+0x20b/0x620 [ceph] [ 2104.635232] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2104.639964] ? __d_lookup+0xd6/0x200 [ 2104.643558] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2104.648902] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2104.653363] ceph_con_process_message+0x74/0x150 [libceph] [ 2104.658860] ? calc_signature+0xdb/0x100 [libceph] [ 2104.663685] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2104.669032] ? process_scheduled_works+0x22a/0x510 [ 2104.673831] ? process_scheduled_works+0x22a/0x510 [ 2104.678636] ceph_con_workfn+0x344/0x750 [libceph] [ 2104.683457] ? process_scheduled_works+0x22a/0x510 [ 2104.688250] process_scheduled_works+0x2a4/0x510 [ 2104.692891] worker_thread+0x188/0x340 [ 2104.696639] ? rescuer_thread+0x310/0x310 [ 2104.700667] kthread+0xf5/0x130 [ 2104.703832] ? kthread_complete_and_exit+0x20/0x20 [ 2104.708643] ret_from_fork+0x31/0x40 [ 2104.712238] ? kthread_complete_and_exit+0x20/0x20 [ 2104.717031] ret_from_fork_asm+0x11/0x20 [ 2104.721000] [ 2104.723193] irq event stamp: 9286479 [ 2104.726775] hardirqs last enabled at (9286495): [] console_unlock+0x106/0x120 [ 2104.735569] hardirqs last disabled at (9286504): [] console_unlock+0xeb/0x120 [ 2104.744281] softirqs last enabled at (9286492): [] __do_softirq+0x2be/0x371 [ 2104.752885] softirqs last disabled at (9286487): [] irq_exit_rcu+0xb0/0xd0 [ 2104.761337] ---[ end trace 0000000000000000 ]--- [ 2104.765957] ------------[ cut here ]------------ [ 2104.770592] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2104.782256] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2104.790952] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2104.846444] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2104.855573] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2104.863068] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2104.868579] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2104.874081] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2104.892847] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2104.898089] RAX: 0000000000000000 RBX: ffff8881d12d30e8 RCX: 0000000000000000 [ 2104.905241] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2104.912374] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff97e [ 2104.919508] R10: 0000000000038da0 R11: ffffc9000030b828 R12: ffff8881d12d3838 [ 2104.926657] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2104.933807] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2104.941910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2104.947673] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2104.954823] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2104.961973] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2104.969123] Call Trace: [ 2104.971584] [ 2104.973701] ? __warn+0x85/0x180 [ 2104.976951] ? __list_add_valid_or_report+0x7d/0xa0 [ 2104.981847] ? report_bug+0xfc/0x1d0 [ 2104.985443] ? prb_read_valid+0x1b/0x20 [ 2104.989284] ? handle_bug+0x44/0x70 [ 2104.992776] ? exc_invalid_op+0x17/0x60 [ 2104.996632] ? asm_exc_invalid_op+0x1a/0x20 [ 2105.000858] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.005733] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.010629] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2105.015537] ceph_add_cap+0x20b/0x620 [ceph] [ 2105.019857] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2105.024597] ? __d_lookup+0xd6/0x200 [ 2105.028195] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2105.033544] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2105.038025] ceph_con_process_message+0x74/0x150 [libceph] [ 2105.043519] ? calc_signature+0xdb/0x100 [libceph] [ 2105.048342] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2105.053691] ? process_scheduled_works+0x22a/0x510 [ 2105.058495] ? process_scheduled_works+0x22a/0x510 [ 2105.063306] ceph_con_workfn+0x344/0x750 [libceph] [ 2105.068110] ? process_scheduled_works+0x22a/0x510 [ 2105.072916] process_scheduled_works+0x2a4/0x510 [ 2105.077577] worker_thread+0x188/0x340 [ 2105.081325] ? rescuer_thread+0x310/0x310 [ 2105.085337] kthread+0xf5/0x130 [ 2105.088483] ? kthread_complete_and_exit+0x20/0x20 [ 2105.093293] ret_from_fork+0x31/0x40 [ 2105.096872] ? kthread_complete_and_exit+0x20/0x20 [ 2105.101682] ret_from_fork_asm+0x11/0x20 [ 2105.105652] [ 2105.107845] irq event stamp: 9287495 [ 2105.111425] hardirqs last enabled at (9287511): [] console_unlock+0x106/0x120 [ 2105.120205] hardirqs last disabled at (9287520): [] console_unlock+0xeb/0x120 [ 2105.128897] softirqs last enabled at (9287508): [] __do_softirq+0x2be/0x371 [ 2105.137521] softirqs last disabled at (9287503): [] irq_exit_rcu+0xb0/0xd0 [ 2105.145969] ---[ end trace 0000000000000000 ]--- [ 2105.150630] ------------[ cut here ]------------ [ 2105.155252] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2105.166918] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2105.175633] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2105.231115] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2105.240258] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2105.247736] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2105.253258] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2105.258743] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2105.277506] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2105.282750] RAX: 0000000000000000 RBX: ffff8881e4d2bab0 RCX: 0000000000000000 [ 2105.289902] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2105.297050] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff9b9 [ 2105.304204] R10: 0000000000039f48 R11: ffffc9000030b828 R12: ffff8881e4d2c200 [ 2105.311334] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2105.318466] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2105.326552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2105.332316] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2105.339447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2105.346580] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2105.353732] Call Trace: [ 2105.356211] [ 2105.358318] ? __warn+0x85/0x180 [ 2105.361550] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.366448] ? report_bug+0xfc/0x1d0 [ 2105.370044] ? prb_read_valid+0x1b/0x20 [ 2105.373903] ? handle_bug+0x44/0x70 [ 2105.377392] ? exc_invalid_op+0x17/0x60 [ 2105.381234] ? asm_exc_invalid_op+0x1a/0x20 [ 2105.385429] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.390308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.395204] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2105.400121] ceph_add_cap+0x20b/0x620 [ceph] [ 2105.404419] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2105.409158] ? __d_lookup+0xd6/0x200 [ 2105.412760] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2105.418114] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2105.422601] ceph_con_process_message+0x74/0x150 [libceph] [ 2105.428094] ? calc_signature+0xdb/0x100 [libceph] [ 2105.432947] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2105.438276] ? process_scheduled_works+0x22a/0x510 [ 2105.443094] ? process_scheduled_works+0x22a/0x510 [ 2105.447889] ceph_con_workfn+0x344/0x750 [libceph] [ 2105.452728] ? process_scheduled_works+0x22a/0x510 [ 2105.457527] process_scheduled_works+0x2a4/0x510 [ 2105.462195] worker_thread+0x188/0x340 [ 2105.465949] ? rescuer_thread+0x310/0x310 [ 2105.469961] kthread+0xf5/0x130 [ 2105.473127] ? kthread_complete_and_exit+0x20/0x20 [ 2105.477956] ret_from_fork+0x31/0x40 [ 2105.481533] ? kthread_complete_and_exit+0x20/0x20 [ 2105.486343] ret_from_fork_asm+0x11/0x20 [ 2105.490282] [ 2105.492487] irq event stamp: 9288549 [ 2105.496086] hardirqs last enabled at (9288565): [] console_unlock+0x106/0x120 [ 2105.504886] hardirqs last disabled at (9288574): [] console_unlock+0xeb/0x120 [ 2105.513584] softirqs last enabled at (9288562): [] __do_softirq+0x2be/0x371 [ 2105.522212] softirqs last disabled at (9288557): [] irq_exit_rcu+0xb0/0xd0 [ 2105.530648] ---[ end trace 0000000000000000 ]--- [ 2105.535403] ------------[ cut here ]------------ [ 2105.540029] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2105.551674] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2105.560625] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2105.616094] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2105.625046] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2105.632524] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2105.638011] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2105.656758] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2105.661988] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2105.669116] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2105.676248] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffff9f4 [ 2105.683372] R10: 000000000003b118 R11: ffffc90001ff7b18 R12: ffff8881d12dcbc8 [ 2105.690498] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2105.697630] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2105.705717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2105.711461] CR2: 00007f9bd3603a80 CR3: 0000000120cc8006 CR4: 00000000003706e0 [ 2105.718594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2105.725718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2105.732844] Call Trace: [ 2105.735296] [ 2105.737394] ? __warn+0x85/0x180 [ 2105.740627] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.745506] ? report_bug+0xfc/0x1d0 [ 2105.749086] ? prb_read_valid+0x1b/0x20 [ 2105.752925] ? handle_bug+0x44/0x70 [ 2105.756418] ? exc_invalid_op+0x17/0x60 [ 2105.760257] ? asm_exc_invalid_op+0x1a/0x20 [ 2105.764448] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.769323] ? __list_add_valid_or_report+0x7d/0xa0 [ 2105.774202] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2105.779093] ceph_open+0x16b/0x440 [ceph] [ 2105.783113] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2105.788260] do_dentry_open+0x172/0x530 [ 2105.792100] path_openat+0x743/0x9d0 [ 2105.795683] do_filp_open+0xb8/0x120 [ 2105.799260] ? lock_release+0xc5/0x260 [ 2105.803017] ? _raw_spin_unlock+0x23/0x40 [ 2105.807034] ? alloc_fd+0x13f/0x200 [ 2105.810526] do_sys_openat2+0x249/0x290 [ 2105.814365] do_sys_open+0x4f/0x80 [ 2105.817771] do_syscall_64+0x38/0x80 [ 2105.821352] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2105.826402] RIP: 0033:0x7f48f0eed2f2 [ 2105.829982] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2105.848732] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2105.856293] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2105.863417] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2105.870541] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2105.877667] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2105.884798] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f1 [ 2105.891928] [ 2105.894115] irq event stamp: 12783453 [ 2105.897774] hardirqs last enabled at (12783463): [] console_unlock+0x106/0x120 [ 2105.906639] hardirqs last disabled at (12783472): [] console_unlock+0xeb/0x120 [ 2105.915418] softirqs last enabled at (12783014): [] __do_softirq+0x2be/0x371 [ 2105.924112] softirqs last disabled at (12783491): [] irq_exit_rcu+0xb0/0xd0 [ 2105.932631] ---[ end trace 0000000000000000 ]--- [ 2105.937250] ------------[ cut here ]------------ [ 2105.941872] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2105.953526] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2105.962225] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2106.017778] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2106.026904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2106.034383] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2106.039896] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2106.045380] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2106.064128] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2106.069353] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2106.076488] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2106.083624] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffa2e [ 2106.090762] R10: 000000000003c2b8 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2106.097894] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2106.105028] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2106.113113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2106.118857] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2106.125990] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2106.133124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2106.140257] Call Trace: [ 2106.142711] [ 2106.144817] ? __warn+0x85/0x180 [ 2106.148053] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.152939] ? report_bug+0xfc/0x1d0 [ 2106.156526] ? prb_read_valid+0x1b/0x20 [ 2106.160370] ? handle_bug+0x44/0x70 [ 2106.163868] ? exc_invalid_op+0x17/0x60 [ 2106.167707] ? asm_exc_invalid_op+0x1a/0x20 [ 2106.171914] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.176801] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.181687] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2106.186587] ceph_add_cap+0x20b/0x620 [ceph] [ 2106.190895] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2106.195625] ? lock_acquire+0xcb/0x2b0 [ 2106.199386] ? lock_is_held_type+0xaa/0x120 [ 2106.203584] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2106.208051] ? __mutex_lock+0x97/0xce0 [ 2106.211809] ? __mutex_lock+0xaec/0xce0 [ 2106.215656] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2106.220211] ? lock_downgrade+0xfd/0x250 [ 2106.224156] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2106.228557] ceph_con_process_message+0x74/0x150 [libceph] [ 2106.234069] ? calc_signature+0xdb/0x100 [libceph] [ 2106.238892] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2106.244232] ? process_scheduled_works+0x22a/0x510 [ 2106.249035] ? process_scheduled_works+0x22a/0x510 [ 2106.253828] ceph_con_workfn+0x344/0x750 [libceph] [ 2106.258644] ? process_scheduled_works+0x22a/0x510 [ 2106.263439] process_scheduled_works+0x2a4/0x510 [ 2106.268071] worker_thread+0x188/0x340 [ 2106.271833] ? rescuer_thread+0x310/0x310 [ 2106.275855] kthread+0xf5/0x130 [ 2106.279005] ? kthread_complete_and_exit+0x20/0x20 [ 2106.283799] ret_from_fork+0x31/0x40 [ 2106.287386] ? kthread_complete_and_exit+0x20/0x20 [ 2106.292181] ret_from_fork_asm+0x11/0x20 [ 2106.296130] [ 2106.298325] irq event stamp: 9409055 [ 2106.301903] hardirqs last enabled at (9409065): [] console_unlock+0x106/0x120 [ 2106.310682] hardirqs last disabled at (9409074): [] console_unlock+0xeb/0x120 [ 2106.319376] softirqs last enabled at (9408992): [] __do_softirq+0x2be/0x371 [ 2106.327990] softirqs last disabled at (9408987): [] irq_exit_rcu+0xb0/0xd0 [ 2106.336425] ---[ end trace 0000000000000000 ]--- [ 2106.341046] ------------[ cut here ]------------ [ 2106.345683] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2106.357351] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2106.366316] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2106.421805] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2106.430774] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2106.438283] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2106.443778] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2106.462532] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2106.467764] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2106.474906] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2106.482047] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffa6e [ 2106.489190] R10: 000000000003d538 R11: ffffc90001ff7a60 R12: ffff8881d12dcbc8 [ 2106.496331] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2106.503473] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2106.511565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2106.517321] CR2: 00007f9bd3603a80 CR3: 0000000120cc8006 CR4: 00000000003706e0 [ 2106.524464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2106.531605] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2106.538745] Call Trace: [ 2106.541206] [ 2106.543313] ? __warn+0x85/0x180 [ 2106.546555] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.551445] ? report_bug+0xfc/0x1d0 [ 2106.555030] ? prb_read_valid+0x1b/0x20 [ 2106.558873] ? handle_bug+0x44/0x70 [ 2106.562363] ? exc_invalid_op+0x17/0x60 [ 2106.566205] ? asm_exc_invalid_op+0x1a/0x20 [ 2106.570394] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.575270] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2106.580169] try_get_cap_refs+0x433/0x880 [ceph] [ 2106.584808] __ceph_get_caps+0xf8/0x760 [ceph] [ 2106.589276] ? ceph_read_iter+0x134/0x860 [ceph] [ 2106.593902] ceph_read_iter+0x134/0x860 [ceph] [ 2106.598361] ? selinux_file_permission+0x119/0x150 [ 2106.603163] ? vfs_read+0x20f/0x300 [ 2106.606661] vfs_read+0x20f/0x300 [ 2106.609995] ksys_pread64+0x65/0xa0 [ 2106.613491] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2106.618636] do_syscall_64+0x38/0x80 [ 2106.622218] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2106.627269] RIP: 0033:0x7f48f0eeb88b [ 2106.630850] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2106.649595] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2106.657160] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2106.664293] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000005 [ 2106.671426] RBP: 00000000000029f1 R08: 0000000000000063 R09: 0000000000000003 [ 2106.678558] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2106.685691] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2106.692831] [ 2106.695019] irq event stamp: 12787087 [ 2106.698683] hardirqs last enabled at (12787097): [] console_unlock+0x106/0x120 [ 2106.707549] hardirqs last disabled at (12787106): [] console_unlock+0xeb/0x120 [ 2106.716328] softirqs last enabled at (12787084): [] __do_softirq+0x2be/0x371 [ 2106.725022] softirqs last disabled at (12787079): [] irq_exit_rcu+0xb0/0xd0 [ 2106.733541] ---[ end trace 0000000000000000 ]--- [ 2106.738154] ------------[ cut here ]------------ [ 2106.742773] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2106.754430] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2106.763383] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2106.818923] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2106.827874] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2106.835353] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2106.840841] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2106.859594] RSP: 0018:ffffc90001fe7c88 EFLAGS: 00010286 [ 2106.864820] RAX: 0000000000000000 RBX: ffff8881d10089c8 RCX: 0000000000000000 [ 2106.871953] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2106.879087] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffaa6 [ 2106.886219] R10: 000000000003e690 R11: ffffc90001fe7b18 R12: ffff8881d1009118 [ 2106.893354] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2106.900484] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2106.908570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2106.914323] CR2: 00007fb38e4a9000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2106.921459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2106.928594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2106.935723] Call Trace: [ 2106.938177] [ 2106.940284] ? __warn+0x85/0x180 [ 2106.943517] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.948395] ? report_bug+0xfc/0x1d0 [ 2106.951976] ? prb_read_valid+0x1b/0x20 [ 2106.955818] ? handle_bug+0x44/0x70 [ 2106.959317] ? exc_invalid_op+0x17/0x60 [ 2106.963156] ? asm_exc_invalid_op+0x1a/0x20 [ 2106.967351] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.972231] ? __list_add_valid_or_report+0x7d/0xa0 [ 2106.977108] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2106.982008] ceph_open+0x16b/0x440 [ceph] [ 2106.986041] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2106.991196] do_dentry_open+0x172/0x530 [ 2106.995044] path_openat+0x743/0x9d0 [ 2106.998638] do_filp_open+0xb8/0x120 [ 2107.002225] ? lock_release+0xc5/0x260 [ 2107.005981] ? _raw_spin_unlock+0x23/0x40 [ 2107.009998] ? alloc_fd+0x13f/0x200 [ 2107.013497] do_sys_openat2+0x249/0x290 [ 2107.017344] do_sys_open+0x4f/0x80 [ 2107.020756] do_syscall_64+0x38/0x80 [ 2107.024337] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2107.029395] RIP: 0033:0x7f48f0eed2f2 [ 2107.032977] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2107.051729] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2107.059296] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2107.066429] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2107.073560] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2107.080695] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2107.087827] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0d [ 2107.094972] [ 2107.097161] irq event stamp: 12788895 [ 2107.100827] hardirqs last enabled at (12788905): [] console_unlock+0x106/0x120 [ 2107.109694] hardirqs last disabled at (12788914): [] console_unlock+0xeb/0x120 [ 2107.118472] softirqs last enabled at (12788872): [] __do_softirq+0x2be/0x371 [ 2107.127165] softirqs last disabled at (12788867): [] irq_exit_rcu+0xb0/0xd0 [ 2107.135684] ---[ end trace 0000000000000000 ]--- [ 2107.140306] ------------[ cut here ]------------ [ 2107.144924] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2107.156568] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2107.165521] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2107.220984] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2107.229935] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2107.237412] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2107.242900] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2107.261646] RSP: 0018:ffffc90001fffc88 EFLAGS: 00010286 [ 2107.266873] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2107.274005] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2107.281137] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffae0 [ 2107.288270] R10: 000000000003f900 R11: ffffc90001fffb18 R12: ffff8881a1d35f58 [ 2107.295402] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2107.302526] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2107.310605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2107.316353] CR2: 00007f9bd3858008 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2107.323484] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2107.330615] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2107.337739] Call Trace: [ 2107.340185] [ 2107.342291] ? __warn+0x85/0x180 [ 2107.345525] ? __list_add_valid_or_report+0x7d/0xa0 [ 2107.350404] ? report_bug+0xfc/0x1d0 [ 2107.353984] ? prb_read_valid+0x1b/0x20 [ 2107.357823] ? handle_bug+0x44/0x70 [ 2107.361314] ? exc_invalid_op+0x17/0x60 [ 2107.365153] ? asm_exc_invalid_op+0x1a/0x20 [ 2107.369343] ? __list_add_valid_or_report+0x7d/0xa0 [ 2107.374220] ? __list_add_valid_or_report+0x7d/0xa0 [ 2107.379099] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2107.383989] ceph_open+0x16b/0x440 [ceph] [ 2107.388010] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2107.393157] do_dentry_open+0x172/0x530 [ 2107.396996] path_openat+0x743/0x9d0 [ 2107.400578] do_filp_open+0xb8/0x120 [ 2107.404157] ? lock_release+0xc5/0x260 [ 2107.407910] ? _raw_spin_unlock+0x23/0x40 [ 2107.411919] ? alloc_fd+0x13f/0x200 [ 2107.415416] do_sys_openat2+0x249/0x290 [ 2107.419257] do_sys_open+0x4f/0x80 [ 2107.422659] do_syscall_64+0x38/0x80 [ 2107.426238] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2107.431291] RIP: 0033:0x7f48f0eed2f2 [ 2107.434870] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2107.453615] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2107.461183] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2107.468315] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2107.475446] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2107.482574] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2107.489705] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a0f [ 2107.496843] [ 2107.499030] irq event stamp: 12809363 [ 2107.502697] hardirqs last enabled at (12809373): [] console_unlock+0x106/0x120 [ 2107.511561] hardirqs last disabled at (12809382): [] console_unlock+0xeb/0x120 [ 2107.520343] softirqs last enabled at (12809396): [] __do_softirq+0x2be/0x371 [ 2107.529035] softirqs last disabled at (12809391): [] irq_exit_rcu+0xb0/0xd0 [ 2107.537553] ---[ end trace 0000000000000000 ]--- [ 2107.542166] ------------[ cut here ]------------ [ 2107.546947] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2107.558806] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2107.567776] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2107.623345] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2107.632316] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2107.639883] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2107.645377] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2107.664255] RSP: 0018:ffffc90001fc7d50 EFLAGS: 00010286 [ 2107.669494] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2107.676775] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2107.683919] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffb1a [ 2107.691128] R10: 0000000000040c30 R11: ffffc90001fc7be0 R12: ffff8881d800c200 [ 2107.698269] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2107.705507] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2107.713608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2107.719509] CR2: 00007f9bd386b020 CR3: 0000000113500001 CR4: 00000000003706e0 [ 2107.726645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2107.733771] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2107.740906] Call Trace: [ 2107.743361] [ 2107.745464] ? __warn+0x85/0x180 [ 2107.748697] ? __list_add_valid_or_report+0x7d/0xa0 [ 2107.753578] ? report_bug+0xfc/0x1d0 [ 2107.757154] ? prb_read_valid+0x1b/0x20 [ 2107.760997] ? handle_bug+0x44/0x70 [ 2107.764486] ? exc_invalid_op+0x17/0x60 [ 2107.768327] ? asm_exc_invalid_op+0x1a/0x20 [ 2107.772523] ? __list_add_valid_or_report+0x7d/0xa0 [ 2107.777402] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2107.782306] ceph_readdir+0xc0/0x1670 [ceph] [ 2107.786602] ? wrap_directory_iterator+0x38/0x70 [ 2107.791228] ? wrap_directory_iterator+0x30/0x70 [ 2107.795843] ? lock_is_held_type+0xaa/0x120 [ 2107.800036] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2107.804926] wrap_directory_iterator+0x47/0x70 [ 2107.809377] iterate_dir+0x8d/0x170 [ 2107.812871] __x64_sys_getdents64+0x87/0x140 [ 2107.817150] ? __x64_sys_getdents64+0x140/0x140 [ 2107.821688] do_syscall_64+0x38/0x80 [ 2107.825271] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2107.830321] RIP: 0033:0x7f48f0ec480b [ 2107.833899] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2107.852647] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2107.860235] RAX: ffffffffffffffda RBX: 000000000207e000 RCX: 00007f48f0ec480b [ 2107.867362] RDX: 0000000000010000 RSI: 000000000207e030 RDI: 0000000000000008 [ 2107.874494] RBP: 000000000207e030 R08: 00000000000000db R09: 0000000000000003 [ 2107.881628] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2107.888760] R13: 0000000000000002 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2107.895910] [ 2107.898111] irq event stamp: 12897623 [ 2107.901780] hardirqs last enabled at (12897641): [] console_unlock+0x106/0x120 [ 2107.910645] hardirqs last disabled at (12897658): [] console_unlock+0xeb/0x120 [ 2107.919423] softirqs last enabled at (12897672): [] __do_softirq+0x2be/0x371 [ 2107.928117] softirqs last disabled at (12897683): [] irq_exit_rcu+0xb0/0xd0 [ 2107.936637] ---[ end trace 0000000000000000 ]--- [ 2107.941257] ------------[ cut here ]------------ [ 2107.945890] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2107.957569] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2107.966548] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2108.022046] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2108.031018] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2108.038511] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2108.044016] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2108.062790] RSP: 0018:ffffc90001fffaa0 EFLAGS: 00010286 [ 2108.068050] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2108.075199] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2108.082348] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffb52 [ 2108.089494] R10: 0000000000041d70 R11: ffffc90001fff930 R12: ffff8881a1d35f58 [ 2108.096639] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2108.103768] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2108.111858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2108.117599] CR2: 00007fb38e4a9000 CR3: 000000017eb7a003 CR4: 00000000003706e0 [ 2108.124735] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2108.131865] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2108.139003] Call Trace: [ 2108.141454] [ 2108.143559] ? __warn+0x85/0x180 [ 2108.146790] ? __list_add_valid_or_report+0x7d/0xa0 [ 2108.151673] ? report_bug+0xfc/0x1d0 [ 2108.155258] ? prb_read_valid+0x1b/0x20 [ 2108.159101] ? handle_bug+0x44/0x70 [ 2108.162591] ? exc_invalid_op+0x17/0x60 [ 2108.166433] ? asm_exc_invalid_op+0x1a/0x20 [ 2108.170632] ? __list_add_valid_or_report+0x7d/0xa0 [ 2108.175516] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2108.180413] ceph_check_caps+0x999/0xae0 [ceph] [ 2108.184979] ? lock_release+0xc5/0x260 [ 2108.188735] ceph_open+0x3cc/0x440 [ceph] [ 2108.192766] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2108.197917] do_dentry_open+0x172/0x530 [ 2108.201761] path_openat+0x743/0x9d0 [ 2108.205350] do_filp_open+0xb8/0x120 [ 2108.208940] ? lock_release+0xc5/0x260 [ 2108.212700] ? _raw_spin_unlock+0x23/0x40 [ 2108.216717] ? alloc_fd+0x13f/0x200 [ 2108.220223] do_sys_openat2+0x249/0x290 [ 2108.224066] do_sys_open+0x4f/0x80 [ 2108.227474] do_syscall_64+0x38/0x80 [ 2108.231061] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2108.236121] RIP: 0033:0x7f48f0eed2f2 [ 2108.239700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2108.258452] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2108.266023] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2108.273153] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2108.280287] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2108.287420] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2108.294550] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a0f [ 2108.301694] [ 2108.303888] irq event stamp: 12812045 [ 2108.307550] hardirqs last enabled at (12812055): [] console_unlock+0x106/0x120 [ 2108.316420] hardirqs last disabled at (12812064): [] console_unlock+0xeb/0x120 [ 2108.325197] softirqs last enabled at (12812012): [] __do_softirq+0x2be/0x371 [ 2108.333889] softirqs last disabled at (12812007): [] irq_exit_rcu+0xb0/0xd0 [ 2108.342408] ---[ end trace 0000000000000000 ]--- [ 2108.347032] ------------[ cut here ]------------ [ 2108.351653] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2108.363305] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2108.372258] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2108.427727] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2108.436678] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2108.444156] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2108.449642] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2108.468387] RSP: 0018:ffffc90001ff7b28 EFLAGS: 00010286 [ 2108.473614] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2108.480746] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2108.487878] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffb8d [ 2108.495005] R10: 0000000000042f48 R11: ffffc90001ff79b8 R12: ffff8881d12dcbc8 [ 2108.502137] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2108.509269] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2108.517348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2108.523094] CR2: 00007f9bd4d8c000 CR3: 0000000120cc8005 CR4: 00000000003706e0 [ 2108.530226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2108.537349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2108.544473] Call Trace: [ 2108.546920] [ 2108.549026] ? __warn+0x85/0x180 [ 2108.552257] ? __list_add_valid_or_report+0x7d/0xa0 [ 2108.557137] ? report_bug+0xfc/0x1d0 [ 2108.560717] ? prb_read_valid+0x1b/0x20 [ 2108.564562] ? handle_bug+0x44/0x70 [ 2108.568059] ? exc_invalid_op+0x17/0x60 [ 2108.571896] ? asm_exc_invalid_op+0x1a/0x20 [ 2108.576087] ? __list_add_valid_or_report+0x7d/0xa0 [ 2108.580964] ? __list_add_valid_or_report+0x7d/0xa0 [ 2108.585841] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2108.590733] ceph_check_caps+0x999/0xae0 [ceph] [ 2108.595272] ? lock_release+0xc5/0x260 [ 2108.599029] ? __lock_acquire+0xbfa/0x1e20 [ 2108.603127] ? __lock_acquire+0xbfa/0x1e20 [ 2108.607224] ? lock_acquire+0xcb/0x2b0 [ 2108.610976] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2108.615950] ? find_held_lock+0x2d/0x90 [ 2108.619789] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2108.624855] ? lock_release+0xc5/0x260 [ 2108.628608] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2108.633496] ceph_read_iter+0x26e/0x860 [ceph] [ 2108.637951] ? selinux_file_permission+0x119/0x150 [ 2108.642741] ? vfs_read+0x20f/0x300 [ 2108.646233] vfs_read+0x20f/0x300 [ 2108.649556] ksys_pread64+0x65/0xa0 [ 2108.653044] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2108.658184] do_syscall_64+0x38/0x80 [ 2108.661764] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2108.666816] RIP: 0033:0x7f48f0eeb88b [ 2108.670394] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2108.689143] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2108.696707] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2108.703839] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000005 [ 2108.710963] RBP: 00000000000029f1 R08: 0000000000000063 R09: 0000000000000003 [ 2108.718089] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2108.725221] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2108.732360] [ 2108.734547] irq event stamp: 12815361 [ 2108.738211] hardirqs last enabled at (12815371): [] console_unlock+0x106/0x120 [ 2108.747078] hardirqs last disabled at (12815380): [] console_unlock+0xeb/0x120 [ 2108.755860] softirqs last enabled at (12815394): [] __do_softirq+0x2be/0x371 [ 2108.764551] softirqs last disabled at (12815389): [] irq_exit_rcu+0xb0/0xd0 [ 2108.773070] ---[ end trace 0000000000000000 ]--- [ 2108.777692] ------------[ cut here ]------------ [ 2108.782314] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2108.794073] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2108.803172] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2108.858834] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2108.867906] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2108.875413] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2108.880931] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2108.899916] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2108.905170] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2108.912402] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2108.919539] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffbcd [ 2108.926685] R10: 0000000000044138 R11: ffffc90001fffa60 R12: ffff8881a1d35f58 [ 2108.933816] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2108.940951] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2108.949035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2108.954782] CR2: 00007f9bd3421000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2108.961919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2108.969061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2108.976196] Call Trace: [ 2108.978652] [ 2108.980758] ? __warn+0x85/0x180 [ 2108.983992] ? __list_add_valid_or_report+0x7d/0xa0 [ 2108.988882] ? report_bug+0xfc/0x1d0 [ 2108.992467] ? prb_read_valid+0x1b/0x20 [ 2108.996307] ? handle_bug+0x44/0x70 [ 2108.999799] ? exc_invalid_op+0x17/0x60 [ 2109.003637] ? asm_exc_invalid_op+0x1a/0x20 [ 2109.007831] ? __list_add_valid_or_report+0x7d/0xa0 [ 2109.012713] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2109.017614] try_get_cap_refs+0x159/0x880 [ceph] [ 2109.022258] __ceph_get_caps+0xf8/0x760 [ceph] [ 2109.026740] ? ceph_read_iter+0x134/0x860 [ceph] [ 2109.031371] ceph_read_iter+0x134/0x860 [ceph] [ 2109.035843] ? selinux_file_permission+0x119/0x150 [ 2109.040639] ? vfs_read+0x20f/0x300 [ 2109.044138] vfs_read+0x20f/0x300 [ 2109.047465] ksys_pread64+0x65/0xa0 [ 2109.050959] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2109.056110] do_syscall_64+0x38/0x80 [ 2109.059695] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2109.064745] RIP: 0033:0x7f48f0eeb88b [ 2109.068325] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2109.087071] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2109.094640] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2109.101769] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2109.108904] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2109.116035] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2109.123175] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2109.130322] [ 2109.132512] irq event stamp: 12815921 [ 2109.136178] hardirqs last enabled at (12815931): [] console_unlock+0x106/0x120 [ 2109.145045] hardirqs last disabled at (12815940): [] console_unlock+0xeb/0x120 [ 2109.153824] softirqs last enabled at (12815898): [] __do_softirq+0x2be/0x371 [ 2109.162516] softirqs last disabled at (12815893): [] irq_exit_rcu+0xb0/0xd0 [ 2109.171036] ---[ end trace 0000000000000000 ]--- [ 2109.175658] ------------[ cut here ]------------ [ 2109.180273] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2109.191921] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2109.200874] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2109.256344] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2109.265294] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2109.272774] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2109.278261] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2109.297009] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2109.302232] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2109.309356] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2109.316488] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffc05 [ 2109.323616] R10: 00000000000452d8 R11: ffffc90001ff7a60 R12: ffff8881d12dcbc8 [ 2109.330745] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2109.337870] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2109.345957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2109.351702] CR2: 00007fb38e4c7028 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2109.358826] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2109.365950] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2109.373074] Call Trace: [ 2109.375518] [ 2109.377617] ? __warn+0x85/0x180 [ 2109.380850] ? __list_add_valid_or_report+0x7d/0xa0 [ 2109.385729] ? report_bug+0xfc/0x1d0 [ 2109.389309] ? prb_read_valid+0x1b/0x20 [ 2109.393149] ? handle_bug+0x44/0x70 [ 2109.396640] ? exc_invalid_op+0x17/0x60 [ 2109.400479] ? asm_exc_invalid_op+0x1a/0x20 [ 2109.404669] ? __list_add_valid_or_report+0x7d/0xa0 [ 2109.409547] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2109.414435] try_get_cap_refs+0x433/0x880 [ceph] [ 2109.419066] __ceph_get_caps+0xf8/0x760 [ceph] [ 2109.423528] ? ceph_read_iter+0x134/0x860 [ceph] [ 2109.428154] ceph_read_iter+0x134/0x860 [ceph] [ 2109.432613] ? selinux_file_permission+0x119/0x150 [ 2109.437402] ? vfs_read+0x20f/0x300 [ 2109.440892] vfs_read+0x20f/0x300 [ 2109.444218] ksys_pread64+0x65/0xa0 [ 2109.447704] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2109.452848] do_syscall_64+0x38/0x80 [ 2109.456424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2109.461477] RIP: 0033:0x7f48f0eeb88b [ 2109.465055] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2109.483801] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2109.491367] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2109.498493] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000005 [ 2109.505626] RBP: 00000000000029f1 R08: 0000000000000063 R09: 0000000000000003 [ 2109.512759] R10: 0000000000001000 R11: 0000000000000246 R12: 0000000000001000 [ 2109.519892] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2109.527030] [ 2109.529218] irq event stamp: 12817313 [ 2109.532883] hardirqs last enabled at (12817323): [] console_unlock+0x106/0x120 [ 2109.541748] hardirqs last disabled at (12817332): [] console_unlock+0xeb/0x120 [ 2109.550532] softirqs last enabled at (12817348): [] __do_softirq+0x2be/0x371 [ 2109.559221] softirqs last disabled at (12817341): [] irq_exit_rcu+0xb0/0xd0 [ 2109.567741] ---[ end trace 0000000000000000 ]--- [ 2109.572361] ------------[ cut here ]------------ [ 2109.576980] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2109.588628] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2109.597584] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2109.653057] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2109.662014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2109.669492] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2109.674978] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2109.693723] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2109.698951] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2109.706082] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2109.713215] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffc3d [ 2109.720349] R10: 0000000000046470 R11: ffffc90001fffa60 R12: ffff8881a1d35f58 [ 2109.727481] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2109.734614] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2109.742699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2109.748446] CR2: 00007f9bd3421000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2109.755582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2109.762710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2109.769844] Call Trace: [ 2109.772296] [ 2109.774405] ? __warn+0x85/0x180 [ 2109.777637] ? __list_add_valid_or_report+0x7d/0xa0 [ 2109.782518] ? report_bug+0xfc/0x1d0 [ 2109.786105] ? prb_read_valid+0x1b/0x20 [ 2109.789949] ? handle_bug+0x44/0x70 [ 2109.793444] ? exc_invalid_op+0x17/0x60 [ 2109.797286] ? asm_exc_invalid_op+0x1a/0x20 [ 2109.801488] ? __list_add_valid_or_report+0x7d/0xa0 [ 2109.806373] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2109.811270] try_get_cap_refs+0x159/0x880 [ceph] [ 2109.815904] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2109.820454] ? wake_bit_function+0x70/0x70 [ 2109.824560] ? ceph_read_iter+0x134/0x860 [ceph] [ 2109.829186] ceph_read_iter+0x134/0x860 [ceph] [ 2109.833644] ? selinux_file_permission+0x119/0x150 [ 2109.838434] ? vfs_read+0x20f/0x300 [ 2109.841924] vfs_read+0x20f/0x300 [ 2109.845250] ksys_pread64+0x65/0xa0 [ 2109.848748] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2109.853886] do_syscall_64+0x38/0x80 [ 2109.857463] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2109.862517] RIP: 0033:0x7f48f0eeb88b [ 2109.866095] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2109.884843] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2109.892408] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2109.899541] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2109.906673] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2109.913799] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2109.920932] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2109.928071] [ 2109.930262] irq event stamp: 12817775 [ 2109.933924] hardirqs last enabled at (12817785): [] console_unlock+0x106/0x120 [ 2109.942796] hardirqs last disabled at (12817808): [] console_unlock+0xeb/0x120 [ 2109.951579] softirqs last enabled at (12817830): [] __do_softirq+0x2be/0x371 [ 2109.960271] softirqs last disabled at (12817817): [] irq_exit_rcu+0xb0/0xd0 [ 2109.968791] ---[ end trace 0000000000000000 ]--- [ 2109.973411] ------------[ cut here ]------------ [ 2109.978031] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2109.989685] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2109.998384] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2110.053932] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2110.063064] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2110.070545] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2110.076064] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2110.081557] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2110.100307] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2110.105539] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2110.112695] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2110.119853] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffc76 [ 2110.126989] R10: 0000000000047548 R11: ffffc9000030b828 R12: ffff8881e4d0b838 [ 2110.134127] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2110.141267] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2110.149358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2110.155104] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2110.162239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2110.169374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2110.176510] Call Trace: [ 2110.178965] [ 2110.181072] ? __warn+0x85/0x180 [ 2110.184313] ? __list_add_valid_or_report+0x7d/0xa0 [ 2110.189201] ? report_bug+0xfc/0x1d0 [ 2110.192789] ? prb_read_valid+0x1b/0x20 [ 2110.196641] ? handle_bug+0x44/0x70 [ 2110.200138] ? exc_invalid_op+0x17/0x60 [ 2110.203980] ? asm_exc_invalid_op+0x1a/0x20 [ 2110.208183] ? __list_add_valid_or_report+0x7d/0xa0 [ 2110.213070] ? __list_add_valid_or_report+0x7d/0xa0 [ 2110.217957] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2110.222860] ceph_add_cap+0x20b/0x620 [ceph] [ 2110.227165] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2110.231909] ceph_fill_trace+0x143/0xf30 [ceph] [ 2110.236464] ? __mutex_lock+0x97/0xce0 [ 2110.240221] ? __mutex_lock+0xaec/0xce0 [ 2110.244064] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2110.248621] ? lock_downgrade+0xfd/0x250 [ 2110.252565] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2110.256957] ceph_con_process_message+0x74/0x150 [libceph] [ 2110.262462] ? calc_signature+0xdb/0x100 [libceph] [ 2110.267287] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2110.272625] ? process_scheduled_works+0x22a/0x510 [ 2110.277435] ? process_scheduled_works+0x22a/0x510 [ 2110.282231] ceph_con_workfn+0x344/0x750 [libceph] [ 2110.287045] ? process_scheduled_works+0x22a/0x510 [ 2110.291842] process_scheduled_works+0x2a4/0x510 [ 2110.296483] worker_thread+0x188/0x340 [ 2110.300243] ? rescuer_thread+0x310/0x310 [ 2110.304261] kthread+0xf5/0x130 [ 2110.307416] ? kthread_complete_and_exit+0x20/0x20 [ 2110.312214] ret_from_fork+0x31/0x40 [ 2110.315799] ? kthread_complete_and_exit+0x20/0x20 [ 2110.320598] ret_from_fork_asm+0x11/0x20 [ 2110.324550] [ 2110.326744] irq event stamp: 9450883 [ 2110.330324] hardirqs last enabled at (9450899): [] console_unlock+0x106/0x120 [ 2110.339109] hardirqs last disabled at (9450908): [] console_unlock+0xeb/0x120 [ 2110.347804] softirqs last enabled at (9450896): [] __do_softirq+0x2be/0x371 [ 2110.356418] softirqs last disabled at (9450891): [] irq_exit_rcu+0xb0/0xd0 [ 2110.364858] ---[ end trace 0000000000000000 ]--- [ 2110.369480] ------------[ cut here ]------------ [ 2110.374108] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2110.385983] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2110.395057] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2110.450699] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2110.459757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2110.467368] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2110.472955] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2110.491804] RSP: 0018:ffffc90001ff7b28 EFLAGS: 00010286 [ 2110.497038] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2110.504167] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2110.511299] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffcb4 [ 2110.518438] R10: 00000000000487c0 R11: ffffc90001ff79b8 R12: ffff8881d12dcbc8 [ 2110.525573] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2110.532709] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2110.540792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2110.546539] CR2: 00007f9bd4d8c000 CR3: 0000000120cc8005 CR4: 00000000003706e0 [ 2110.553672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2110.560806] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2110.567938] Call Trace: [ 2110.570391] [ 2110.572496] ? __warn+0x85/0x180 [ 2110.575732] ? __list_add_valid_or_report+0x7d/0xa0 [ 2110.580619] ? report_bug+0xfc/0x1d0 [ 2110.584202] ? prb_read_valid+0x1b/0x20 [ 2110.588046] ? handle_bug+0x44/0x70 [ 2110.591538] ? exc_invalid_op+0x17/0x60 [ 2110.595377] ? asm_exc_invalid_op+0x1a/0x20 [ 2110.599574] ? __list_add_valid_or_report+0x7d/0xa0 [ 2110.604451] ? __list_add_valid_or_report+0x7d/0xa0 [ 2110.609333] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2110.614231] ceph_check_caps+0x999/0xae0 [ceph] [ 2110.618782] ? lock_release+0xc5/0x260 [ 2110.622546] ? __lock_acquire+0xbfa/0x1e20 [ 2110.626654] ? __lock_acquire+0xbfa/0x1e20 [ 2110.630758] ? lock_acquire+0xcb/0x2b0 [ 2110.634510] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2110.639492] ? find_held_lock+0x2d/0x90 [ 2110.643332] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2110.648404] ? lock_release+0xc5/0x260 [ 2110.652161] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2110.657055] ceph_read_iter+0x26e/0x860 [ceph] [ 2110.661532] ? selinux_file_permission+0x119/0x150 [ 2110.666326] ? vfs_read+0x20f/0x300 [ 2110.669827] vfs_read+0x20f/0x300 [ 2110.673155] ksys_pread64+0x65/0xa0 [ 2110.676646] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2110.681788] do_syscall_64+0x38/0x80 [ 2110.685368] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2110.690427] RIP: 0033:0x7f48f0eeb88b [ 2110.694008] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2110.712755] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2110.720327] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2110.727460] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000005 [ 2110.734592] RBP: 00000000000029f1 R08: 0000000000000063 R09: 0000000000000003 [ 2110.741725] R10: 0000000000001000 R11: 0000000000000246 R12: 0000000000001000 [ 2110.748859] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2110.756002] [ 2110.758193] irq event stamp: 12821621 [ 2110.761858] hardirqs last enabled at (12821631): [] console_unlock+0x106/0x120 [ 2110.770725] hardirqs last disabled at (12821640): [] console_unlock+0xeb/0x120 [ 2110.779506] softirqs last enabled at (12821654): [] __do_softirq+0x2be/0x371 [ 2110.788208] softirqs last disabled at (12821649): [] irq_exit_rcu+0xb0/0xd0 [ 2110.796733] ---[ end trace 0000000000000000 ]--- [ 2110.801355] ------------[ cut here ]------------ [ 2110.805985] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2110.817642] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2110.826600] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2110.882121] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2110.891071] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2110.898552] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2110.904037] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2110.922784] RSP: 0018:ffffc90001f7fc88 EFLAGS: 00010286 [ 2110.928010] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2110.935142] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2110.942287] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffcf4 [ 2110.949421] R10: 0000000000049ba8 R11: ffffc90001f7fb18 R12: ffff8881e4d0b838 [ 2110.956558] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2110.963692] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2110.971778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2110.977524] CR2: 00007f9bd38c5000 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2110.984660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2110.991788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2110.998922] Call Trace: [ 2111.001376] [ 2111.003482] ? __warn+0x85/0x180 [ 2111.006715] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.011594] ? report_bug+0xfc/0x1d0 [ 2111.015173] ? prb_read_valid+0x1b/0x20 [ 2111.019020] ? handle_bug+0x44/0x70 [ 2111.022513] ? exc_invalid_op+0x17/0x60 [ 2111.026353] ? asm_exc_invalid_op+0x1a/0x20 [ 2111.030550] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.035429] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.040307] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2111.045207] ceph_open+0x16b/0x440 [ceph] [ 2111.049238] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2111.054394] do_dentry_open+0x172/0x530 [ 2111.058241] path_openat+0x743/0x9d0 [ 2111.061825] do_filp_open+0xb8/0x120 [ 2111.065415] ? lock_release+0xc5/0x260 [ 2111.069170] ? _raw_spin_unlock+0x23/0x40 [ 2111.073190] ? alloc_fd+0x13f/0x200 [ 2111.076696] do_sys_openat2+0x249/0x290 [ 2111.080544] do_sys_open+0x4f/0x80 [ 2111.083957] do_syscall_64+0x38/0x80 [ 2111.087546] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2111.092602] RIP: 0033:0x7f48f0eed2f2 [ 2111.096184] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2111.114927] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2111.122497] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2111.129626] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2111.136766] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2111.143901] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2111.151034] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a19 [ 2111.158181] [ 2111.160376] irq event stamp: 12879547 [ 2111.164042] hardirqs last enabled at (12879557): [] console_unlock+0x106/0x120 [ 2111.172909] hardirqs last disabled at (12879566): [] console_unlock+0xeb/0x120 [ 2111.181687] softirqs last enabled at (12879534): [] __do_softirq+0x2be/0x371 [ 2111.190384] softirqs last disabled at (12879585): [] irq_exit_rcu+0xb0/0xd0 [ 2111.198909] ---[ end trace 0000000000000000 ]--- [ 2111.203530] ------------[ cut here ]------------ [ 2111.208154] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2111.219810] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2111.228767] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2111.284277] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2111.293239] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2111.300723] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2111.306211] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2111.324956] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2111.330184] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2111.337316] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2111.344450] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffd2e [ 2111.351585] R10: 000000000004ade0 R11: ffffc90001ff7a60 R12: ffff8881d12dcbc8 [ 2111.358713] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2111.365837] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2111.373923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2111.379661] CR2: 00007f9bd38d2000 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2111.386786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2111.393918] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2111.401042] Call Trace: [ 2111.403488] [ 2111.405594] ? __warn+0x85/0x180 [ 2111.408825] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.413705] ? report_bug+0xfc/0x1d0 [ 2111.417288] ? prb_read_valid+0x1b/0x20 [ 2111.421126] ? handle_bug+0x44/0x70 [ 2111.424617] ? exc_invalid_op+0x17/0x60 [ 2111.428456] ? asm_exc_invalid_op+0x1a/0x20 [ 2111.432647] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.437525] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2111.442413] try_get_cap_refs+0x433/0x880 [ceph] [ 2111.447044] __ceph_get_caps+0xf8/0x760 [ceph] [ 2111.451510] ? ceph_read_iter+0x134/0x860 [ceph] [ 2111.456137] ceph_read_iter+0x134/0x860 [ceph] [ 2111.460597] ? selinux_file_permission+0x119/0x150 [ 2111.465389] ? vfs_read+0x20f/0x300 [ 2111.468878] vfs_read+0x20f/0x300 [ 2111.472204] ksys_pread64+0x65/0xa0 [ 2111.475690] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2111.480834] do_syscall_64+0x38/0x80 [ 2111.484411] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2111.489461] RIP: 0033:0x7f48f0eeb88b [ 2111.493042] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2111.511787] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2111.519353] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2111.526485] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000005 [ 2111.533612] RBP: 00000000000029f1 R08: 0000000000000063 R09: 0000000000000003 [ 2111.540744] R10: 0000000000002000 R11: 0000000000000246 R12: 0000000000001000 [ 2111.547877] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2111.555016] [ 2111.557201] irq event stamp: 12823525 [ 2111.560863] hardirqs last enabled at (12823535): [] console_unlock+0x106/0x120 [ 2111.569727] hardirqs last disabled at (12823544): [] console_unlock+0xeb/0x120 [ 2111.578504] softirqs last enabled at (12823502): [] __do_softirq+0x2be/0x371 [ 2111.587196] softirqs last disabled at (12823497): [] irq_exit_rcu+0xb0/0xd0 [ 2111.595718] ---[ end trace 0000000000000000 ]--- [ 2111.600341] ------------[ cut here ]------------ [ 2111.604957] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2111.616606] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2111.625301] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2111.680785] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2111.689911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2111.697388] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2111.702892] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2111.708378] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2111.727125] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2111.732351] RAX: 0000000000000000 RBX: ffff8881d12961d0 RCX: 0000000000000000 [ 2111.739483] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2111.746619] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffd66 [ 2111.753750] R10: 000000000004bea8 R11: ffffc9000030b828 R12: ffff8881d1296920 [ 2111.760881] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2111.768016] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2111.776100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2111.781846] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2111.788980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2111.796113] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2111.803247] Call Trace: [ 2111.805701] [ 2111.807804] ? __warn+0x85/0x180 [ 2111.811039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.815918] ? report_bug+0xfc/0x1d0 [ 2111.819497] ? prb_read_valid+0x1b/0x20 [ 2111.823338] ? handle_bug+0x44/0x70 [ 2111.826827] ? exc_invalid_op+0x17/0x60 [ 2111.830668] ? asm_exc_invalid_op+0x1a/0x20 [ 2111.834858] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.839735] ? __list_add_valid_or_report+0x7d/0xa0 [ 2111.844614] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2111.849504] ceph_add_cap+0x20b/0x620 [ceph] [ 2111.853798] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2111.858526] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2111.863855] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2111.868324] ceph_con_process_message+0x74/0x150 [libceph] [ 2111.873821] ? calc_signature+0xdb/0x100 [libceph] [ 2111.878632] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2111.883965] ? process_scheduled_works+0x22a/0x510 [ 2111.888765] ? process_scheduled_works+0x22a/0x510 [ 2111.893554] ceph_con_workfn+0x344/0x750 [libceph] [ 2111.898358] ? process_scheduled_works+0x22a/0x510 [ 2111.903147] process_scheduled_works+0x2a4/0x510 [ 2111.907775] worker_thread+0x188/0x340 [ 2111.911530] ? rescuer_thread+0x310/0x310 [ 2111.915541] kthread+0xf5/0x130 [ 2111.918689] ? kthread_complete_and_exit+0x20/0x20 [ 2111.923481] ret_from_fork+0x31/0x40 [ 2111.927061] ? kthread_complete_and_exit+0x20/0x20 [ 2111.931852] ret_from_fork_asm+0x11/0x20 [ 2111.935786] [ 2111.937982] irq event stamp: 9457113 [ 2111.941560] hardirqs last enabled at (9457123): [] console_unlock+0x106/0x120 [ 2111.950339] hardirqs last disabled at (9457132): [] console_unlock+0xeb/0x120 [ 2111.959032] softirqs last enabled at (9457110): [] __do_softirq+0x2be/0x371 [ 2111.967644] softirqs last disabled at (9457151): [] irq_exit_rcu+0xb0/0xd0 [ 2111.976080] ---[ end trace 0000000000000000 ]--- [ 2111.980699] ------------[ cut here ]------------ [ 2111.985326] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2111.996975] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2112.005928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2112.061440] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2112.070392] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2112.077871] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2112.083359] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2112.102106] RSP: 0018:ffffc90001f7fbd0 EFLAGS: 00010282 [ 2112.107331] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2112.114463] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2112.121595] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffda0 [ 2112.128729] R10: 000000000004d020 R11: ffffc90001f7fa60 R12: ffff8881e4d0b838 [ 2112.135863] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2112.142995] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2112.151082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2112.156828] CR2: 00007f9bd38c5000 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2112.163967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2112.171101] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2112.178234] Call Trace: [ 2112.180688] [ 2112.182795] ? __warn+0x85/0x180 [ 2112.186028] ? __list_add_valid_or_report+0x7d/0xa0 [ 2112.190907] ? report_bug+0xfc/0x1d0 [ 2112.194484] ? prb_read_valid+0x1b/0x20 [ 2112.198329] ? handle_bug+0x44/0x70 [ 2112.201826] ? exc_invalid_op+0x17/0x60 [ 2112.205667] ? asm_exc_invalid_op+0x1a/0x20 [ 2112.209861] ? __list_add_valid_or_report+0x7d/0xa0 [ 2112.214742] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2112.219640] try_get_cap_refs+0x433/0x880 [ceph] [ 2112.224284] __ceph_get_caps+0xf8/0x760 [ceph] [ 2112.228768] ? ceph_read_iter+0x134/0x860 [ceph] [ 2112.233400] ceph_read_iter+0x134/0x860 [ceph] [ 2112.237870] ? selinux_file_permission+0x119/0x150 [ 2112.242669] ? vfs_read+0x20f/0x300 [ 2112.246165] vfs_read+0x20f/0x300 [ 2112.249496] ksys_pread64+0x65/0xa0 [ 2112.252986] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2112.258129] do_syscall_64+0x38/0x80 [ 2112.261715] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2112.266766] RIP: 0033:0x7f48f0eeb88b [ 2112.270345] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2112.289090] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2112.296659] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2112.303791] RDX: 0000000000001000 RSI: 000000000207eb20 RDI: 0000000000000005 [ 2112.310923] RBP: 0000000000002a19 R08: 000000000207eb10 R09: 0000000000000003 [ 2112.318057] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2112.325189] R13: 0000000000001000 R14: 000000000207eb20 R15: 00007f48f183d000 [ 2112.332333] [ 2112.334524] irq event stamp: 12882237 [ 2112.338189] hardirqs last enabled at (12882247): [] console_unlock+0x106/0x120 [ 2112.347056] hardirqs last disabled at (12882256): [] console_unlock+0xeb/0x120 [ 2112.355840] softirqs last enabled at (12882276): [] __do_softirq+0x2be/0x371 [ 2112.364541] softirqs last disabled at (12882289): [] irq_exit_rcu+0xb0/0xd0 [ 2112.373067] ---[ end trace 0000000000000000 ]--- [ 2112.377686] ------------[ cut here ]------------ [ 2112.382306] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2112.393951] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2112.402910] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2112.458381] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2112.467338] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2112.474820] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2112.480310] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2112.499055] RSP: 0018:ffffc90001ff7b28 EFLAGS: 00010286 [ 2112.504282] RAX: 0000000000000000 RBX: ffff8881d12dc478 RCX: 0000000000000000 [ 2112.511413] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2112.518546] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffdd8 [ 2112.525677] R10: 000000000004e160 R11: ffffc90001ff79b8 R12: ffff8881d12dcbc8 [ 2112.532813] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2112.539944] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2112.548031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2112.553774] CR2: 00007fb38e3d6be8 CR3: 0000000120cc8005 CR4: 00000000003706e0 [ 2112.560910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2112.568041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2112.575176] Call Trace: [ 2112.577626] [ 2112.579736] ? __warn+0x85/0x180 [ 2112.582965] ? __list_add_valid_or_report+0x7d/0xa0 [ 2112.587849] ? report_bug+0xfc/0x1d0 [ 2112.591439] ? prb_read_valid+0x1b/0x20 [ 2112.595287] ? handle_bug+0x44/0x70 [ 2112.598786] ? exc_invalid_op+0x17/0x60 [ 2112.602625] ? asm_exc_invalid_op+0x1a/0x20 [ 2112.606824] ? __list_add_valid_or_report+0x7d/0xa0 [ 2112.611709] ? __list_add_valid_or_report+0x7d/0xa0 [ 2112.616594] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2112.621497] ceph_check_caps+0x999/0xae0 [ceph] [ 2112.626043] ? lock_release+0xc5/0x260 [ 2112.629803] ? __lock_acquire+0xbfa/0x1e20 [ 2112.633906] ? __lock_acquire+0xbfa/0x1e20 [ 2112.638018] ? lock_acquire+0xcb/0x2b0 [ 2112.641770] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2112.646757] ? find_held_lock+0x2d/0x90 [ 2112.650604] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2112.655677] ? lock_release+0xc5/0x260 [ 2112.659439] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2112.664349] ceph_read_iter+0x26e/0x860 [ceph] [ 2112.668812] ? selinux_file_permission+0x119/0x150 [ 2112.673609] ? vfs_read+0x20f/0x300 [ 2112.677104] vfs_read+0x20f/0x300 [ 2112.680435] ksys_pread64+0x65/0xa0 [ 2112.683924] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2112.689070] do_syscall_64+0x38/0x80 [ 2112.692654] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2112.697707] RIP: 0033:0x7f48f0eeb88b [ 2112.701286] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2112.720033] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2112.727599] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2112.734733] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000005 [ 2112.741863] RBP: 00000000000029f1 R08: 0000000000000063 R09: 0000000000000003 [ 2112.748997] R10: 0000000000002000 R11: 0000000000000246 R12: 0000000000001000 [ 2112.756129] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2112.763273] [ 2112.765463] irq event stamp: 12826467 [ 2112.769131] hardirqs last enabled at (12826477): [] console_unlock+0x106/0x120 [ 2112.777996] hardirqs last disabled at (12826486): [] console_unlock+0xeb/0x120 [ 2112.786775] softirqs last enabled at (12826444): [] __do_softirq+0x2be/0x371 [ 2112.795469] softirqs last disabled at (12826505): [] irq_exit_rcu+0xb0/0xd0 [ 2112.803992] ---[ end trace 0000000000000000 ]--- [ 2112.808616] ------------[ cut here ]------------ [ 2112.813235] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2112.824884] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2112.833836] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2112.889347] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2112.898300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2112.905780] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2112.911264] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2112.930015] RSP: 0018:ffffc90001f7fbd0 EFLAGS: 00010282 [ 2112.935246] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2112.942379] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2112.949515] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffe18 [ 2112.956645] R10: 000000000004f828 R11: ffffc90001f7fa60 R12: ffff8881e4d0b838 [ 2112.963777] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2112.970912] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2112.978996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2112.984742] CR2: 00007f9bd38c5000 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2112.991877] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2112.999007] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2113.006143] Call Trace: [ 2113.008596] [ 2113.010700] ? __warn+0x85/0x180 [ 2113.013935] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.018813] ? report_bug+0xfc/0x1d0 [ 2113.022392] ? prb_read_valid+0x1b/0x20 [ 2113.026235] ? handle_bug+0x44/0x70 [ 2113.029724] ? exc_invalid_op+0x17/0x60 [ 2113.033565] ? asm_exc_invalid_op+0x1a/0x20 [ 2113.037759] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.042641] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2113.047537] try_get_cap_refs+0x433/0x880 [ceph] [ 2113.052180] __ceph_get_caps+0xf8/0x760 [ceph] [ 2113.056660] ? ceph_read_iter+0x134/0x860 [ceph] [ 2113.061298] ceph_read_iter+0x134/0x860 [ceph] [ 2113.065766] ? selinux_file_permission+0x119/0x150 [ 2113.070567] ? vfs_read+0x20f/0x300 [ 2113.074063] vfs_read+0x20f/0x300 [ 2113.077395] ksys_pread64+0x65/0xa0 [ 2113.080884] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2113.086027] do_syscall_64+0x38/0x80 [ 2113.089612] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2113.094663] RIP: 0033:0x7f48f0eeb88b [ 2113.098245] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2113.116989] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2113.124555] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2113.131688] RDX: 0000000000001000 RSI: 000000000207eb20 RDI: 0000000000000005 [ 2113.138822] RBP: 0000000000002a19 R08: 000000000207eb10 R09: 0000000000000003 [ 2113.145954] R10: 0000000000001000 R11: 0000000000000246 R12: 0000000000001000 [ 2113.153087] R13: 0000000000001000 R14: 000000000207eb20 R15: 00007f48f183d000 [ 2113.160232] [ 2113.162423] irq event stamp: 12884501 [ 2113.166088] hardirqs last enabled at (12884511): [] console_unlock+0x106/0x120 [ 2113.174955] hardirqs last disabled at (12884520): [] console_unlock+0xeb/0x120 [ 2113.183734] softirqs last enabled at (12884100): [] __do_softirq+0x2be/0x371 [ 2113.192427] softirqs last disabled at (12884539): [] irq_exit_rcu+0xb0/0xd0 [ 2113.200946] ---[ end trace 0000000000000000 ]--- [ 2113.205566] ------------[ cut here ]------------ [ 2113.210186] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2113.221834] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2113.230795] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2113.286265] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2113.295215] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2113.302694] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2113.308180] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2113.326925] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2113.332154] RAX: 0000000000000000 RBX: ffff8881d12d30e8 RCX: 0000000000000000 [ 2113.339284] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2113.346417] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffe50 [ 2113.353553] R10: 0000000000050710 R11: ffffc90001ff7b18 R12: ffff8881d12d3838 [ 2113.360683] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2113.367816] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2113.375903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2113.381649] CR2: 00007fb38e3d6be8 CR3: 0000000120cc8005 CR4: 00000000003706e0 [ 2113.388782] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2113.395913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2113.403047] Call Trace: [ 2113.405499] [ 2113.407609] ? __warn+0x85/0x180 [ 2113.410847] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.415729] ? report_bug+0xfc/0x1d0 [ 2113.419320] ? prb_read_valid+0x1b/0x20 [ 2113.423168] ? handle_bug+0x44/0x70 [ 2113.426663] ? exc_invalid_op+0x17/0x60 [ 2113.430509] ? asm_exc_invalid_op+0x1a/0x20 [ 2113.434707] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.439592] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.444475] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2113.449377] ceph_open+0x16b/0x440 [ceph] [ 2113.453405] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2113.458560] do_dentry_open+0x172/0x530 [ 2113.462400] path_openat+0x743/0x9d0 [ 2113.465985] do_filp_open+0xb8/0x120 [ 2113.469573] ? lock_release+0xc5/0x260 [ 2113.473332] ? _raw_spin_unlock+0x23/0x40 [ 2113.477348] ? alloc_fd+0x13f/0x200 [ 2113.480847] do_sys_openat2+0x249/0x290 [ 2113.484695] do_sys_open+0x4f/0x80 [ 2113.488107] do_syscall_64+0x38/0x80 [ 2113.491693] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2113.496745] RIP: 0033:0x7f48f0eed2f2 [ 2113.500325] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2113.519073] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2113.526644] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2113.533777] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2113.540912] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2113.548045] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2113.555175] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f2 [ 2113.562327] [ 2113.564515] irq event stamp: 12828451 [ 2113.568175] hardirqs last enabled at (12828461): [] console_unlock+0x106/0x120 [ 2113.577042] hardirqs last disabled at (12828470): [] console_unlock+0xeb/0x120 [ 2113.585821] softirqs last enabled at (12828416): [] __do_softirq+0x2be/0x371 [ 2113.594515] softirqs last disabled at (12828411): [] irq_exit_rcu+0xb0/0xd0 [ 2113.603032] ---[ end trace 0000000000000000 ]--- [ 2113.607655] ------------[ cut here ]------------ [ 2113.612272] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2113.623918] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2113.632616] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2113.688101] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2113.697225] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2113.704708] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2113.710208] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2113.715693] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2113.734441] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2113.739665] RAX: 0000000000000000 RBX: ffff8881d1008000 RCX: 0000000000000000 [ 2113.746798] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2113.753934] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffe8a [ 2113.761065] R10: 00000000000515e8 R11: ffffc9000030b828 R12: ffff8881d1008750 [ 2113.768197] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2113.775332] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2113.783416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2113.789162] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2113.796295] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2113.803427] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2113.810562] Call Trace: [ 2113.813013] [ 2113.815120] ? __warn+0x85/0x180 [ 2113.818355] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.823233] ? report_bug+0xfc/0x1d0 [ 2113.826812] ? prb_read_valid+0x1b/0x20 [ 2113.830653] ? handle_bug+0x44/0x70 [ 2113.834146] ? exc_invalid_op+0x17/0x60 [ 2113.837983] ? asm_exc_invalid_op+0x1a/0x20 [ 2113.842173] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.847050] ? __list_add_valid_or_report+0x7d/0xa0 [ 2113.851928] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2113.856819] ceph_add_cap+0x20b/0x620 [ceph] [ 2113.861113] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2113.865836] ? __d_lookup+0xd6/0x200 [ 2113.869420] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2113.874749] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2113.879218] ceph_con_process_message+0x74/0x150 [libceph] [ 2113.884717] ? calc_signature+0xdb/0x100 [libceph] [ 2113.889528] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2113.894860] ? process_scheduled_works+0x22a/0x510 [ 2113.899659] ? process_scheduled_works+0x22a/0x510 [ 2113.904447] ceph_con_workfn+0x344/0x750 [libceph] [ 2113.909251] ? process_scheduled_works+0x22a/0x510 [ 2113.914041] process_scheduled_works+0x2a4/0x510 [ 2113.918667] worker_thread+0x188/0x340 [ 2113.922415] ? rescuer_thread+0x310/0x310 [ 2113.926428] kthread+0xf5/0x130 [ 2113.929574] ? kthread_complete_and_exit+0x20/0x20 [ 2113.934366] ret_from_fork+0x31/0x40 [ 2113.937945] ? kthread_complete_and_exit+0x20/0x20 [ 2113.942739] ret_from_fork_asm+0x11/0x20 [ 2113.946671] [ 2113.948867] irq event stamp: 9461655 [ 2113.952444] hardirqs last enabled at (9461665): [] console_unlock+0x106/0x120 [ 2113.961225] hardirqs last disabled at (9461674): [] console_unlock+0xeb/0x120 [ 2113.969918] softirqs last enabled at (9461652): [] __do_softirq+0x2be/0x371 [ 2113.978523] softirqs last disabled at (9461647): [] irq_exit_rcu+0xb0/0xd0 [ 2113.986956] ---[ end trace 0000000000000000 ]--- [ 2113.991577] ------------[ cut here ]------------ [ 2113.996212] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2114.007881] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2114.016866] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2114.072413] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2114.081381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2114.088878] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2114.094383] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2114.113145] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2114.118390] RAX: 0000000000000000 RBX: ffff8881d12d30e8 RCX: 0000000000000000 [ 2114.125540] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2114.132690] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffec5 [ 2114.139839] R10: 0000000000052728 R11: ffffc90001ff7a60 R12: ffff8881d12d3838 [ 2114.146987] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2114.154138] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2114.162242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2114.168005] CR2: 00007f9bd38f6010 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2114.175155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2114.182306] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2114.189456] Call Trace: [ 2114.191924] [ 2114.194041] ? __warn+0x85/0x180 [ 2114.197292] ? __list_add_valid_or_report+0x7d/0xa0 [ 2114.202190] ? report_bug+0xfc/0x1d0 [ 2114.205782] ? prb_read_valid+0x1b/0x20 [ 2114.209631] ? handle_bug+0x44/0x70 [ 2114.213129] ? exc_invalid_op+0x17/0x60 [ 2114.216967] ? asm_exc_invalid_op+0x1a/0x20 [ 2114.221164] ? __list_add_valid_or_report+0x7d/0xa0 [ 2114.226047] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2114.230952] try_get_cap_refs+0x433/0x880 [ceph] [ 2114.235603] __ceph_get_caps+0xf8/0x760 [ceph] [ 2114.240085] ? ceph_read_iter+0x134/0x860 [ceph] [ 2114.244720] ceph_read_iter+0x134/0x860 [ceph] [ 2114.249192] ? selinux_file_permission+0x119/0x150 [ 2114.253989] ? vfs_read+0x20f/0x300 [ 2114.257486] vfs_read+0x20f/0x300 [ 2114.260815] ksys_pread64+0x65/0xa0 [ 2114.264314] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2114.269459] do_syscall_64+0x38/0x80 [ 2114.273044] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2114.278103] RIP: 0033:0x7f48f0eeb88b [ 2114.281682] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2114.300429] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2114.307994] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2114.315126] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000006 [ 2114.322259] RBP: 00000000000029f2 R08: 0000000000000063 R09: 0000000000000003 [ 2114.329393] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2114.336526] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2114.343672] [ 2114.345868] irq event stamp: 12831879 [ 2114.349535] hardirqs last enabled at (12831889): [] console_unlock+0x106/0x120 [ 2114.358401] hardirqs last disabled at (12831898): [] console_unlock+0xeb/0x120 [ 2114.367179] softirqs last enabled at (12831856): [] __do_softirq+0x2be/0x371 [ 2114.375877] softirqs last disabled at (12831917): [] irq_exit_rcu+0xb0/0xd0 [ 2114.384402] ---[ end trace 0000000000000000 ]--- [ 2114.389022] ------------[ cut here ]------------ [ 2114.393642] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2114.405290] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2114.413993] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2114.469478] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2114.478609] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2114.486089] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2114.491591] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2114.497079] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2114.515823] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2114.521050] RAX: 0000000000000000 RBX: ffff8881d1297560 RCX: 0000000000000000 [ 2114.528182] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2114.535319] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000fffffefd [ 2114.542447] R10: 0000000000053880 R11: ffffc9000030b828 R12: ffff8881d1297cb0 [ 2114.549579] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2114.556714] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2114.564800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2114.570546] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2114.577679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2114.584813] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2114.591944] Call Trace: [ 2114.594397] [ 2114.596504] ? __warn+0x85/0x180 [ 2114.599739] ? __list_add_valid_or_report+0x7d/0xa0 [ 2114.604618] ? report_bug+0xfc/0x1d0 [ 2114.608196] ? prb_read_valid+0x1b/0x20 [ 2114.612038] ? handle_bug+0x44/0x70 [ 2114.615526] ? exc_invalid_op+0x17/0x60 [ 2114.619366] ? asm_exc_invalid_op+0x1a/0x20 [ 2114.623557] ? __list_add_valid_or_report+0x7d/0xa0 [ 2114.628436] ? __list_add_valid_or_report+0x7d/0xa0 [ 2114.633321] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2114.638221] ceph_add_cap+0x20b/0x620 [ceph] [ 2114.642520] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2114.647241] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2114.652576] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2114.657046] ceph_con_process_message+0x74/0x150 [libceph] [ 2114.662545] ? calc_signature+0xdb/0x100 [libceph] [ 2114.667362] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2114.672695] ? process_scheduled_works+0x22a/0x510 [ 2114.677494] ? process_scheduled_works+0x22a/0x510 [ 2114.682288] ceph_con_workfn+0x344/0x750 [libceph] [ 2114.687102] ? process_scheduled_works+0x22a/0x510 [ 2114.691900] process_scheduled_works+0x2a4/0x510 [ 2114.696529] worker_thread+0x188/0x340 [ 2114.700285] ? rescuer_thread+0x310/0x310 [ 2114.704300] kthread+0xf5/0x130 [ 2114.707446] ? kthread_complete_and_exit+0x20/0x20 [ 2114.712240] ret_from_fork+0x31/0x40 [ 2114.715818] ? kthread_complete_and_exit+0x20/0x20 [ 2114.720613] ret_from_fork_asm+0x11/0x20 [ 2114.724549] [ 2114.726741] irq event stamp: 9463551 [ 2114.730319] hardirqs last enabled at (9463561): [] console_unlock+0x106/0x120 [ 2114.739097] hardirqs last disabled at (9463570): [] console_unlock+0xeb/0x120 [ 2114.747790] softirqs last enabled at (9463548): [] __do_softirq+0x2be/0x371 [ 2114.756399] softirqs last disabled at (9463543): [] irq_exit_rcu+0xb0/0xd0 [ 2114.764838] ---[ end trace 0000000000000000 ]--- [ 2114.769459] ------------[ cut here ]------------ [ 2114.774154] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2114.785827] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2114.794802] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2114.850338] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2114.859307] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2114.866804] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2114.872303] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2114.891063] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2114.896302] RAX: 0000000000000000 RBX: ffff8881d0bd9390 RCX: 0000000000000000 [ 2114.903446] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2114.910595] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffff37 [ 2114.917746] R10: 00000000000549f8 R11: ffffc90001ff7b18 R12: ffff8881d0bd9ae0 [ 2114.924896] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2114.932045] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2114.940149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2114.945910] CR2: 00007f9bd38f6010 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2114.953042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2114.960172] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2114.967305] Call Trace: [ 2114.969759] [ 2114.971864] ? __warn+0x85/0x180 [ 2114.975098] ? __list_add_valid_or_report+0x7d/0xa0 [ 2114.979979] ? report_bug+0xfc/0x1d0 [ 2114.983560] ? prb_read_valid+0x1b/0x20 [ 2114.987410] ? handle_bug+0x44/0x70 [ 2114.990906] ? exc_invalid_op+0x17/0x60 [ 2114.994747] ? asm_exc_invalid_op+0x1a/0x20 [ 2114.998940] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.003823] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.008708] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2115.013608] ceph_open+0x16b/0x440 [ceph] [ 2115.017639] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2115.022792] do_dentry_open+0x172/0x530 [ 2115.026634] path_openat+0x743/0x9d0 [ 2115.030226] do_filp_open+0xb8/0x120 [ 2115.033816] ? lock_release+0xc5/0x260 [ 2115.037572] ? _raw_spin_unlock+0x23/0x40 [ 2115.041590] ? alloc_fd+0x13f/0x200 [ 2115.045096] do_sys_openat2+0x249/0x290 [ 2115.048948] do_sys_open+0x4f/0x80 [ 2115.052357] do_syscall_64+0x38/0x80 [ 2115.055941] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2115.060995] RIP: 0033:0x7f48f0eed2f2 [ 2115.064575] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2115.083320] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2115.090887] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2115.098020] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2115.105152] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2115.112284] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2115.119416] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f3 [ 2115.126564] [ 2115.128759] irq event stamp: 12835275 [ 2115.132427] hardirqs last enabled at (12835285): [] console_unlock+0x106/0x120 [ 2115.141293] hardirqs last disabled at (12835294): [] console_unlock+0xeb/0x120 [ 2115.150074] softirqs last enabled at (12835308): [] __do_softirq+0x2be/0x371 [ 2115.158773] softirqs last disabled at (12835303): [] irq_exit_rcu+0xb0/0xd0 [ 2115.167293] ---[ end trace 0000000000000000 ]--- [ 2115.171913] ------------[ cut here ]------------ [ 2115.176530] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2115.188180] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2115.196873] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2115.252363] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2115.261493] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2115.268970] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2115.274465] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2115.279953] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2115.298698] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2115.303926] RAX: 0000000000000000 RBX: ffff8881f79de1d0 RCX: 0000000000000000 [ 2115.311056] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2115.318190] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffff71 [ 2115.325323] R10: 0000000000055b58 R11: ffffc9000030b828 R12: ffff8881f79de920 [ 2115.332457] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2115.339590] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2115.347675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2115.353420] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2115.360554] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2115.367688] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2115.374819] Call Trace: [ 2115.377272] [ 2115.379382] ? __warn+0x85/0x180 [ 2115.382622] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.387502] ? report_bug+0xfc/0x1d0 [ 2115.391087] ? prb_read_valid+0x1b/0x20 [ 2115.394930] ? handle_bug+0x44/0x70 [ 2115.398418] ? exc_invalid_op+0x17/0x60 [ 2115.402260] ? asm_exc_invalid_op+0x1a/0x20 [ 2115.406454] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.411332] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.416213] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2115.421113] ceph_add_cap+0x20b/0x620 [ceph] [ 2115.425411] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2115.430133] ? __d_lookup+0xd6/0x200 [ 2115.433715] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2115.439054] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2115.443526] ceph_con_process_message+0x74/0x150 [libceph] [ 2115.449029] ? calc_signature+0xdb/0x100 [libceph] [ 2115.453840] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2115.459173] ? process_scheduled_works+0x22a/0x510 [ 2115.463972] ? process_scheduled_works+0x22a/0x510 [ 2115.468767] ceph_con_workfn+0x344/0x750 [libceph] [ 2115.473583] ? process_scheduled_works+0x22a/0x510 [ 2115.478380] process_scheduled_works+0x2a4/0x510 [ 2115.483009] worker_thread+0x188/0x340 [ 2115.486762] ? rescuer_thread+0x310/0x310 [ 2115.490773] kthread+0xf5/0x130 [ 2115.493916] ? kthread_complete_and_exit+0x20/0x20 [ 2115.498716] ret_from_fork+0x31/0x40 [ 2115.502299] ? kthread_complete_and_exit+0x20/0x20 [ 2115.507094] ret_from_fork_asm+0x11/0x20 [ 2115.511030] [ 2115.513221] irq event stamp: 9465449 [ 2115.516799] hardirqs last enabled at (9465459): [] console_unlock+0x106/0x120 [ 2115.525577] hardirqs last disabled at (9465468): [] console_unlock+0xeb/0x120 [ 2115.534273] softirqs last enabled at (9465446): [] __do_softirq+0x2be/0x371 [ 2115.542875] softirqs last disabled at (9465441): [] irq_exit_rcu+0xb0/0xd0 [ 2115.551309] ---[ end trace 0000000000000000 ]--- [ 2115.555929] ------------[ cut here ]------------ [ 2115.560621] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2115.572273] WARNING: CPU: 4 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2115.581239] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2115.636711] CPU: 4 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2115.645673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2115.653155] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2115.658640] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2115.677383] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2115.682611] RAX: 0000000000000000 RBX: ffff8881d0bd9390 RCX: 0000000000000000 [ 2115.689744] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2115.696875] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffffac [ 2115.704001] R10: 0000000000056cb8 R11: ffffc90001ff7a60 R12: ffff8881d0bd9ae0 [ 2115.711133] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2115.718266] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2115.726353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2115.732098] CR2: 00007fb38e517000 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2115.739231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2115.746355] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2115.753479] Call Trace: [ 2115.755933] [ 2115.758040] ? __warn+0x85/0x180 [ 2115.761271] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.766151] ? report_bug+0xfc/0x1d0 [ 2115.769731] ? prb_read_valid+0x1b/0x20 [ 2115.773570] ? handle_bug+0x44/0x70 [ 2115.777061] ? exc_invalid_op+0x17/0x60 [ 2115.780892] ? asm_exc_invalid_op+0x1a/0x20 [ 2115.785083] ? __list_add_valid_or_report+0x7d/0xa0 [ 2115.789960] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2115.794849] try_get_cap_refs+0x433/0x880 [ceph] [ 2115.799482] __ceph_get_caps+0xf8/0x760 [ceph] [ 2115.803952] ? ceph_read_iter+0x134/0x860 [ceph] [ 2115.808578] ceph_read_iter+0x134/0x860 [ceph] [ 2115.813035] ? selinux_file_permission+0x119/0x150 [ 2115.817824] ? vfs_read+0x20f/0x300 [ 2115.821314] vfs_read+0x20f/0x300 [ 2115.824639] ksys_pread64+0x65/0xa0 [ 2115.828126] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2115.833269] do_syscall_64+0x38/0x80 [ 2115.836845] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2115.841898] RIP: 0033:0x7f48f0eeb88b [ 2115.845476] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2115.864226] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2115.871790] RAX: ffffffffffffffda RBX: 0000000002075850 RCX: 00007f48f0eeb88b [ 2115.878922] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000007 [ 2115.886047] RBP: 00000000000029f3 R08: 0000000000000063 R09: 0000000000000003 [ 2115.893179] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2115.900305] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2115.907442] [ 2115.909635] irq event stamp: 12837989 [ 2115.913294] hardirqs last enabled at (12837999): [] console_unlock+0x106/0x120 [ 2115.922162] hardirqs last disabled at (12838008): [] console_unlock+0xeb/0x120 [ 2115.930942] softirqs last enabled at (12837986): [] __do_softirq+0x2be/0x371 [ 2115.939639] softirqs last disabled at (12838027): [] irq_exit_rcu+0xb0/0xd0 [ 2115.948153] ---[ end trace 0000000000000000 ]--- [ 2115.952765] ------------[ cut here ]------------ [ 2115.957392] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2115.969056] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2115.977754] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2116.033244] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2116.042373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2116.049853] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2116.055368] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2116.060858] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2116.079604] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2116.084829] RAX: 0000000000000000 RBX: ffff8881f79d8000 RCX: 0000000000000000 [ 2116.091963] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2116.099097] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000000ffffffe4 [ 2116.106228] R10: 0000000000057e70 R11: ffffc9000030b828 R12: ffff8881f79d8750 [ 2116.113360] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2116.120494] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2116.128585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2116.134328] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2116.141459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2116.148594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2116.155725] Call Trace: [ 2116.158179] [ 2116.160284] ? __warn+0x85/0x180 [ 2116.163520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.168397] ? report_bug+0xfc/0x1d0 [ 2116.171975] ? prb_read_valid+0x1b/0x20 [ 2116.175817] ? handle_bug+0x44/0x70 [ 2116.179308] ? exc_invalid_op+0x17/0x60 [ 2116.183148] ? asm_exc_invalid_op+0x1a/0x20 [ 2116.187338] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.192214] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.197092] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2116.201995] ceph_add_cap+0x20b/0x620 [ceph] [ 2116.206284] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2116.211001] ? __d_lookup+0xd6/0x200 [ 2116.214588] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2116.219922] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2116.224385] ceph_con_process_message+0x74/0x150 [libceph] [ 2116.229890] ? calc_signature+0xdb/0x100 [libceph] [ 2116.234699] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2116.240023] ? process_scheduled_works+0x22a/0x510 [ 2116.244823] ? process_scheduled_works+0x22a/0x510 [ 2116.249612] ceph_con_workfn+0x344/0x750 [libceph] [ 2116.254415] ? process_scheduled_works+0x22a/0x510 [ 2116.259208] process_scheduled_works+0x2a4/0x510 [ 2116.263834] worker_thread+0x188/0x340 [ 2116.267590] ? rescuer_thread+0x310/0x310 [ 2116.271601] kthread+0xf5/0x130 [ 2116.274747] ? kthread_complete_and_exit+0x20/0x20 [ 2116.279541] ret_from_fork+0x31/0x40 [ 2116.283121] ? kthread_complete_and_exit+0x20/0x20 [ 2116.287912] ret_from_fork_asm+0x11/0x20 [ 2116.291845] [ 2116.294041] irq event stamp: 9467353 [ 2116.297619] hardirqs last enabled at (9467363): [] console_unlock+0x106/0x120 [ 2116.306398] hardirqs last disabled at (9467372): [] console_unlock+0xeb/0x120 [ 2116.315091] softirqs last enabled at (9467350): [] __do_softirq+0x2be/0x371 [ 2116.323697] softirqs last disabled at (9467345): [] irq_exit_rcu+0xb0/0xd0 [ 2116.332132] ---[ end trace 0000000000000000 ]--- [ 2116.336751] ------------[ cut here ]------------ [ 2116.341394] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2116.353060] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2116.362027] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2116.417515] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2116.426487] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2116.433968] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2116.439450] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2116.458198] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2116.463423] RAX: 0000000000000000 RBX: ffff8881d0bda720 RCX: 0000000000000000 [ 2116.470556] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2116.477689] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000001f [ 2116.484823] R10: 0000000000059110 R11: ffffc90001ff7b18 R12: ffff8881d0bdae70 [ 2116.491954] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2116.499087] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2116.507174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2116.512918] CR2: 00007fb38e5130c8 CR3: 0000000120cc8006 CR4: 00000000003706f0 [ 2116.520043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2116.527176] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2116.534309] Call Trace: [ 2116.536761] [ 2116.538860] ? __warn+0x85/0x180 [ 2116.542091] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.546972] ? report_bug+0xfc/0x1d0 [ 2116.550552] ? prb_read_valid+0x1b/0x20 [ 2116.554392] ? handle_bug+0x44/0x70 [ 2116.557882] ? exc_invalid_op+0x17/0x60 [ 2116.561722] ? asm_exc_invalid_op+0x1a/0x20 [ 2116.565913] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.570788] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.575665] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2116.580558] ceph_open+0x16b/0x440 [ceph] [ 2116.584579] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2116.589731] do_dentry_open+0x172/0x530 [ 2116.593575] path_openat+0x743/0x9d0 [ 2116.597156] do_filp_open+0xb8/0x120 [ 2116.600737] ? lock_release+0xc5/0x260 [ 2116.604487] ? _raw_spin_unlock+0x23/0x40 [ 2116.608496] ? alloc_fd+0x13f/0x200 [ 2116.611992] do_sys_openat2+0x249/0x290 [ 2116.615835] do_sys_open+0x4f/0x80 [ 2116.619236] do_syscall_64+0x38/0x80 [ 2116.622815] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2116.627867] RIP: 0033:0x7f48f0eed2f2 [ 2116.631447] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2116.650193] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2116.657758] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2116.664890] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2116.672015] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2116.679149] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2116.686281] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f4 [ 2116.693419] [ 2116.695607] irq event stamp: 12840411 [ 2116.699274] hardirqs last enabled at (12840421): [] console_unlock+0x106/0x120 [ 2116.708138] hardirqs last disabled at (12840430): [] console_unlock+0xeb/0x120 [ 2116.716918] softirqs last enabled at (12840378): [] __do_softirq+0x2be/0x371 [ 2116.725611] softirqs last disabled at (12840373): [] irq_exit_rcu+0xb0/0xd0 [ 2116.734130] ---[ end trace 0000000000000000 ]--- [ 2116.738750] ------------[ cut here ]------------ [ 2116.743369] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2116.755016] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2116.763710] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2116.819190] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2116.828312] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2116.835786] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2116.841280] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2116.846764] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2116.865515] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2116.870737] RAX: 0000000000000000 RBX: ffff8881d821bab0 RCX: 0000000000000000 [ 2116.877871] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2116.885005] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000059 [ 2116.892135] R10: 000000000005a288 R11: ffffc9000030b828 R12: ffff8881d821c200 [ 2116.899259] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2116.906386] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2116.914469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2116.920217] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2116.927349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2116.934482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2116.941614] Call Trace: [ 2116.944070] [ 2116.946175] ? __warn+0x85/0x180 [ 2116.949408] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.954287] ? report_bug+0xfc/0x1d0 [ 2116.957866] ? prb_read_valid+0x1b/0x20 [ 2116.961707] ? handle_bug+0x44/0x70 [ 2116.965199] ? exc_invalid_op+0x17/0x60 [ 2116.969040] ? asm_exc_invalid_op+0x1a/0x20 [ 2116.973230] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.978114] ? __list_add_valid_or_report+0x7d/0xa0 [ 2116.982991] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2116.987881] ceph_add_cap+0x20b/0x620 [ceph] [ 2116.992168] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2116.996892] ? __d_lookup+0xd6/0x200 [ 2117.000475] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2117.005803] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2117.010266] ceph_con_process_message+0x74/0x150 [libceph] [ 2117.015761] ? calc_signature+0xdb/0x100 [libceph] [ 2117.020573] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2117.025896] ? process_scheduled_works+0x22a/0x510 [ 2117.030687] ? process_scheduled_works+0x22a/0x510 [ 2117.035476] ceph_con_workfn+0x344/0x750 [libceph] [ 2117.040282] ? process_scheduled_works+0x22a/0x510 [ 2117.045070] process_scheduled_works+0x2a4/0x510 [ 2117.049698] worker_thread+0x188/0x340 [ 2117.053454] ? rescuer_thread+0x310/0x310 [ 2117.057464] kthread+0xf5/0x130 [ 2117.060612] ? kthread_complete_and_exit+0x20/0x20 [ 2117.065404] ret_from_fork+0x31/0x40 [ 2117.068983] ? kthread_complete_and_exit+0x20/0x20 [ 2117.073777] ret_from_fork_asm+0x11/0x20 [ 2117.077710] [ 2117.079905] irq event stamp: 9469291 [ 2117.083483] hardirqs last enabled at (9469301): [] console_unlock+0x106/0x120 [ 2117.092264] hardirqs last disabled at (9469310): [] console_unlock+0xeb/0x120 [ 2117.100956] softirqs last enabled at (9469288): [] __do_softirq+0x2be/0x371 [ 2117.109563] softirqs last disabled at (9469283): [] irq_exit_rcu+0xb0/0xd0 [ 2117.117995] ---[ end trace 0000000000000000 ]--- [ 2117.122613] ------------[ cut here ]------------ [ 2117.127233] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2117.138880] WARNING: CPU: 2 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2117.147834] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2117.203304] CPU: 2 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2117.212257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2117.219733] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2117.225220] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2117.243965] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2117.249192] RAX: 0000000000000000 RBX: ffff8881d0bda720 RCX: 0000000000000000 [ 2117.256324] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2117.263456] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000094 [ 2117.270582] R10: 000000000005b3c0 R11: ffffc90001ff7a60 R12: ffff8881d0bdae70 [ 2117.277715] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2117.284847] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2117.292932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2117.298669] CR2: 00007fac504418a0 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2117.305795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2117.312927] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2117.320059] Call Trace: [ 2117.322505] [ 2117.324611] ? __warn+0x85/0x180 [ 2117.327843] ? __list_add_valid_or_report+0x7d/0xa0 [ 2117.332722] ? report_bug+0xfc/0x1d0 [ 2117.336302] ? prb_read_valid+0x1b/0x20 [ 2117.340144] ? handle_bug+0x44/0x70 [ 2117.343633] ? exc_invalid_op+0x17/0x60 [ 2117.347464] ? asm_exc_invalid_op+0x1a/0x20 [ 2117.351655] ? __list_add_valid_or_report+0x7d/0xa0 [ 2117.356533] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2117.361422] try_get_cap_refs+0x433/0x880 [ceph] [ 2117.366051] __ceph_get_caps+0xf8/0x760 [ceph] [ 2117.370513] ? ceph_read_iter+0x134/0x860 [ceph] [ 2117.375138] ceph_read_iter+0x134/0x860 [ceph] [ 2117.379597] ? selinux_file_permission+0x119/0x150 [ 2117.384392] ? vfs_read+0x20f/0x300 [ 2117.387877] vfs_read+0x20f/0x300 [ 2117.391203] ksys_pread64+0x65/0xa0 [ 2117.394691] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2117.399832] do_syscall_64+0x38/0x80 [ 2117.403409] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2117.408462] RIP: 0033:0x7f48f0eeb88b [ 2117.412040] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2117.430787] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2117.438354] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2117.445486] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000008 [ 2117.452620] RBP: 00000000000029f4 R08: 0000000000000063 R09: 0000000000000003 [ 2117.459751] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2117.466877] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2117.474015] [ 2117.476202] irq event stamp: 12842257 [ 2117.479869] hardirqs last enabled at (12842267): [] console_unlock+0x106/0x120 [ 2117.488733] hardirqs last disabled at (12842276): [] console_unlock+0xeb/0x120 [ 2117.497515] softirqs last enabled at (12842234): [] __do_softirq+0x2be/0x371 [ 2117.506206] softirqs last disabled at (12842229): [] irq_exit_rcu+0xb0/0xd0 [ 2117.514727] ---[ end trace 0000000000000000 ]--- [ 2117.519346] ------------[ cut here ]------------ [ 2117.523965] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2117.535613] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2117.544308] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2117.599784] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2117.608910] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2117.616390] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2117.621890] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2117.627377] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2117.646123] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2117.651352] RAX: 0000000000000000 RBX: ffff8881d1166b98 RCX: 0000000000000000 [ 2117.658486] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2117.665618] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000000cc [ 2117.672750] R10: 000000000005c378 R11: ffffc9000030b828 R12: ffff8881d11672e8 [ 2117.679883] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2117.687014] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2117.695100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2117.700845] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2117.707979] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2117.715112] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2117.722246] Call Trace: [ 2117.724699] [ 2117.726806] ? __warn+0x85/0x180 [ 2117.730039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2117.734919] ? report_bug+0xfc/0x1d0 [ 2117.738497] ? prb_read_valid+0x1b/0x20 [ 2117.742337] ? handle_bug+0x44/0x70 [ 2117.745826] ? exc_invalid_op+0x17/0x60 [ 2117.749667] ? asm_exc_invalid_op+0x1a/0x20 [ 2117.753858] ? __list_add_valid_or_report+0x7d/0xa0 [ 2117.758734] ? __list_add_valid_or_report+0x7d/0xa0 [ 2117.763611] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2117.768503] ceph_add_cap+0x20b/0x620 [ceph] [ 2117.772787] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2117.777509] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2117.782837] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2117.787299] ceph_con_process_message+0x74/0x150 [libceph] [ 2117.792794] ? calc_signature+0xdb/0x100 [libceph] [ 2117.797606] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2117.802936] ? process_scheduled_works+0x22a/0x510 [ 2117.807731] ? process_scheduled_works+0x22a/0x510 [ 2117.812527] ceph_con_workfn+0x344/0x750 [libceph] [ 2117.817332] ? process_scheduled_works+0x22a/0x510 [ 2117.822122] process_scheduled_works+0x2a4/0x510 [ 2117.826747] worker_thread+0x188/0x340 [ 2117.830495] ? rescuer_thread+0x310/0x310 [ 2117.834508] kthread+0xf5/0x130 [ 2117.837652] ? kthread_complete_and_exit+0x20/0x20 [ 2117.842446] ret_from_fork+0x31/0x40 [ 2117.846024] ? kthread_complete_and_exit+0x20/0x20 [ 2117.850817] ret_from_fork_asm+0x11/0x20 [ 2117.854750] [ 2117.856938] irq event stamp: 9471163 [ 2117.860516] hardirqs last enabled at (9471173): [] console_unlock+0x106/0x120 [ 2117.869296] hardirqs last disabled at (9471182): [] console_unlock+0xeb/0x120 [ 2117.877988] softirqs last enabled at (9471160): [] __do_softirq+0x2be/0x371 [ 2117.886594] softirqs last disabled at (9471155): [] irq_exit_rcu+0xb0/0xd0 [ 2117.895026] ---[ end trace 0000000000000000 ]--- [ 2117.899647] ------------[ cut here ]------------ [ 2117.904264] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2117.915910] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2117.924874] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2117.980345] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2117.989296] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2117.996778] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2118.002262] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2118.021007] RSP: 0018:ffffc90001ff7b28 EFLAGS: 00010286 [ 2118.026235] RAX: 0000000000000000 RBX: ffff8881d0bda720 RCX: 0000000000000000 [ 2118.033365] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2118.040489] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000106 [ 2118.047616] R10: 000000000005d5b0 R11: ffffc90001ff79b8 R12: ffff8881d0bdae70 [ 2118.054747] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2118.061879] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2118.069967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2118.075711] CR2: 000055be3c315f44 CR3: 0000000120cc8003 CR4: 00000000003706e0 [ 2118.082844] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2118.089968] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2118.097092] Call Trace: [ 2118.099538] [ 2118.101645] ? __warn+0x85/0x180 [ 2118.104876] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.109755] ? report_bug+0xfc/0x1d0 [ 2118.113335] ? prb_read_valid+0x1b/0x20 [ 2118.117177] ? handle_bug+0x44/0x70 [ 2118.120669] ? exc_invalid_op+0x17/0x60 [ 2118.124506] ? asm_exc_invalid_op+0x1a/0x20 [ 2118.128696] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.133573] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.138456] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2118.143351] ceph_check_caps+0x999/0xae0 [ceph] [ 2118.147892] ? lock_release+0xc5/0x260 [ 2118.151647] ? __lock_acquire+0xbfa/0x1e20 [ 2118.155745] ? __lock_acquire+0xbfa/0x1e20 [ 2118.159842] ? lock_acquire+0xcb/0x2b0 [ 2118.163595] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2118.168569] ? find_held_lock+0x2d/0x90 [ 2118.172411] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2118.177472] ? lock_release+0xc5/0x260 [ 2118.181225] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2118.186114] ceph_read_iter+0x26e/0x860 [ceph] [ 2118.190569] ? selinux_file_permission+0x119/0x150 [ 2118.195361] ? vfs_read+0x20f/0x300 [ 2118.198851] vfs_read+0x20f/0x300 [ 2118.202176] ksys_pread64+0x65/0xa0 [ 2118.205663] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2118.210804] do_syscall_64+0x38/0x80 [ 2118.214381] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2118.219434] RIP: 0033:0x7f48f0eeb88b [ 2118.223012] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2118.241761] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2118.249325] RAX: ffffffffffffffda RBX: 0000000002075860 RCX: 00007f48f0eeb88b [ 2118.256457] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000008 [ 2118.263582] RBP: 00000000000029f4 R08: 0000000000000063 R09: 0000000000000003 [ 2118.270706] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2118.277830] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2118.284961] [ 2118.287149] irq event stamp: 12844225 [ 2118.290813] hardirqs last enabled at (12844235): [] console_unlock+0x106/0x120 [ 2118.299679] hardirqs last disabled at (12844244): [] console_unlock+0xeb/0x120 [ 2118.308460] softirqs last enabled at (12844258): [] __do_softirq+0x2be/0x371 [ 2118.317153] softirqs last disabled at (12844253): [] irq_exit_rcu+0xb0/0xd0 [ 2118.325671] ---[ end trace 0000000000000000 ]--- [ 2118.330282] ------------[ cut here ]------------ [ 2118.334902] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2118.346550] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2118.355244] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2118.410739] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2118.419865] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2118.427361] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2118.432873] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2118.438357] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2118.457107] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2118.462348] RAX: 0000000000000000 RBX: ffff8881d116e1d0 RCX: 0000000000000000 [ 2118.469500] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2118.476634] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000146 [ 2118.483764] R10: 000000000005e9f8 R11: ffffc9000030b828 R12: ffff8881d116e920 [ 2118.490915] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2118.498087] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2118.506172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2118.511932] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2118.519068] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2118.526216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2118.533366] Call Trace: [ 2118.535836] [ 2118.537959] ? __warn+0x85/0x180 [ 2118.541211] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.546105] ? report_bug+0xfc/0x1d0 [ 2118.549717] ? prb_read_valid+0x1b/0x20 [ 2118.553577] ? handle_bug+0x44/0x70 [ 2118.557069] ? exc_invalid_op+0x17/0x60 [ 2118.560926] ? asm_exc_invalid_op+0x1a/0x20 [ 2118.565150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.570027] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.574923] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2118.579830] ceph_add_cap+0x20b/0x620 [ceph] [ 2118.584149] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2118.588888] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2118.594231] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2118.598695] ceph_con_process_message+0x74/0x150 [libceph] [ 2118.604193] ? calc_signature+0xdb/0x100 [libceph] [ 2118.609021] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2118.614387] ? process_scheduled_works+0x22a/0x510 [ 2118.619178] ? process_scheduled_works+0x22a/0x510 [ 2118.623985] ceph_con_workfn+0x344/0x750 [libceph] [ 2118.628808] ? process_scheduled_works+0x22a/0x510 [ 2118.633615] process_scheduled_works+0x2a4/0x510 [ 2118.638237] worker_thread+0x188/0x340 [ 2118.641987] ? rescuer_thread+0x310/0x310 [ 2118.645998] kthread+0xf5/0x130 [ 2118.649145] ? kthread_complete_and_exit+0x20/0x20 [ 2118.653939] ret_from_fork+0x31/0x40 [ 2118.657516] ? kthread_complete_and_exit+0x20/0x20 [ 2118.662311] ret_from_fork_asm+0x11/0x20 [ 2118.666242] [ 2118.668446] irq event stamp: 9473115 [ 2118.672024] hardirqs last enabled at (9473125): [] console_unlock+0x106/0x120 [ 2118.680805] hardirqs last disabled at (9473134): [] console_unlock+0xeb/0x120 [ 2118.689497] softirqs last enabled at (9473112): [] __do_softirq+0x2be/0x371 [ 2118.698103] softirqs last disabled at (9473107): [] irq_exit_rcu+0xb0/0xd0 [ 2118.706538] ---[ end trace 0000000000000000 ]--- [ 2118.711353] ------------[ cut here ]------------ [ 2118.715969] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2118.727636] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2118.736337] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2118.791836] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2118.800984] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2118.808479] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2118.813979] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2118.819481] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2118.838229] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2118.843471] RAX: 0000000000000000 RBX: ffff8881f79deb98 RCX: 0000000000000000 [ 2118.850606] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2118.857737] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000180 [ 2118.864888] R10: 000000000005fb70 R11: ffffc9000030b828 R12: ffff8881f79df2e8 [ 2118.872038] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2118.879187] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2118.887292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2118.893054] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2118.900206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2118.907356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2118.914505] Call Trace: [ 2118.916958] [ 2118.919081] ? __warn+0x85/0x180 [ 2118.922332] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.927230] ? report_bug+0xfc/0x1d0 [ 2118.930828] ? prb_read_valid+0x1b/0x20 [ 2118.934684] ? handle_bug+0x44/0x70 [ 2118.938173] ? exc_invalid_op+0x17/0x60 [ 2118.942023] ? asm_exc_invalid_op+0x1a/0x20 [ 2118.946254] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.951134] ? __list_add_valid_or_report+0x7d/0xa0 [ 2118.956029] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2118.960936] ceph_add_cap+0x20b/0x620 [ceph] [ 2118.965255] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2118.969993] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2118.975338] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2118.979819] ceph_con_process_message+0x74/0x150 [libceph] [ 2118.985313] ? calc_signature+0xdb/0x100 [libceph] [ 2118.990152] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2118.995486] ? process_scheduled_works+0x22a/0x510 [ 2119.000283] ? process_scheduled_works+0x22a/0x510 [ 2119.005090] ceph_con_workfn+0x344/0x750 [libceph] [ 2119.009912] ? process_scheduled_works+0x22a/0x510 [ 2119.014731] process_scheduled_works+0x2a4/0x510 [ 2119.019389] worker_thread+0x188/0x340 [ 2119.023144] ? rescuer_thread+0x310/0x310 [ 2119.027173] kthread+0xf5/0x130 [ 2119.030337] ? kthread_complete_and_exit+0x20/0x20 [ 2119.035148] ret_from_fork+0x31/0x40 [ 2119.038743] ? kthread_complete_and_exit+0x20/0x20 [ 2119.043555] ret_from_fork_asm+0x11/0x20 [ 2119.047490] [ 2119.049682] irq event stamp: 9474253 [ 2119.053260] hardirqs last enabled at (9474263): [] console_unlock+0x106/0x120 [ 2119.062062] hardirqs last disabled at (9474278): [] console_unlock+0xeb/0x120 [ 2119.070769] softirqs last enabled at (9474276): [] __do_softirq+0x2be/0x371 [ 2119.079391] softirqs last disabled at (9474271): [] irq_exit_rcu+0xb0/0xd0 [ 2119.087842] ---[ end trace 0000000000000000 ]--- [ 2119.092684] ------------[ cut here ]------------ [ 2119.097302] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2119.108979] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2119.117673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2119.173153] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2119.182295] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2119.189790] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2119.195310] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2119.200814] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2119.219580] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2119.224803] RAX: 0000000000000000 RBX: ffff8881f79da720 RCX: 0000000000000000 [ 2119.231953] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2119.239106] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000001ba [ 2119.246254] R10: 0000000000060cf0 R11: ffffc9000030b828 R12: ffff8881f79dae70 [ 2119.253405] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2119.260554] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2119.268641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2119.274387] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2119.281539] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2119.288670] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2119.295802] Call Trace: [ 2119.298273] [ 2119.300397] ? __warn+0x85/0x180 [ 2119.303649] ? __list_add_valid_or_report+0x7d/0xa0 [ 2119.308527] ? report_bug+0xfc/0x1d0 [ 2119.312104] ? prb_read_valid+0x1b/0x20 [ 2119.315983] ? handle_bug+0x44/0x70 [ 2119.319473] ? exc_invalid_op+0x17/0x60 [ 2119.323310] ? asm_exc_invalid_op+0x1a/0x20 [ 2119.327519] ? __list_add_valid_or_report+0x7d/0xa0 [ 2119.332395] ? __list_add_valid_or_report+0x7d/0xa0 [ 2119.337291] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2119.342197] ceph_add_cap+0x20b/0x620 [ceph] [ 2119.346503] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2119.351248] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2119.356574] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2119.361054] ceph_con_process_message+0x74/0x150 [libceph] [ 2119.366551] ? calc_signature+0xdb/0x100 [libceph] [ 2119.371380] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2119.376711] ? process_scheduled_works+0x22a/0x510 [ 2119.381503] ? process_scheduled_works+0x22a/0x510 [ 2119.386293] ceph_con_workfn+0x344/0x750 [libceph] [ 2119.391113] ? process_scheduled_works+0x22a/0x510 [ 2119.395922] process_scheduled_works+0x2a4/0x510 [ 2119.400564] worker_thread+0x188/0x340 [ 2119.404321] ? rescuer_thread+0x310/0x310 [ 2119.408349] kthread+0xf5/0x130 [ 2119.411515] ? kthread_complete_and_exit+0x20/0x20 [ 2119.416306] ret_from_fork+0x31/0x40 [ 2119.419903] ? kthread_complete_and_exit+0x20/0x20 [ 2119.424730] ret_from_fork_asm+0x11/0x20 [ 2119.428681] [ 2119.430869] irq event stamp: 9475325 [ 2119.434468] hardirqs last enabled at (9475335): [] console_unlock+0x106/0x120 [ 2119.443250] hardirqs last disabled at (9475344): [] console_unlock+0xeb/0x120 [ 2119.451961] softirqs last enabled at (9475322): [] __do_softirq+0x2be/0x371 [ 2119.460584] softirqs last disabled at (9475317): [] irq_exit_rcu+0xb0/0xd0 [ 2119.469017] ---[ end trace 0000000000000000 ]--- [ 2119.473726] ------------[ cut here ]------------ [ 2119.478369] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2119.490034] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2119.498734] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2119.554214] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2119.563342] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2119.570818] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2119.576315] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2119.581801] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2119.600546] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2119.605772] RAX: 0000000000000000 RBX: ffff8881a1e309c8 RCX: 0000000000000000 [ 2119.612906] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2119.620036] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000001f4 [ 2119.627172] R10: 0000000000061d38 R11: ffffc9000030b828 R12: ffff8881a1e31118 [ 2119.634303] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2119.641437] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2119.649520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2119.655269] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2119.662402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2119.669533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2119.676666] Call Trace: [ 2119.679119] [ 2119.681226] ? __warn+0x85/0x180 [ 2119.684458] ? __list_add_valid_or_report+0x7d/0xa0 [ 2119.689340] ? report_bug+0xfc/0x1d0 [ 2119.692920] ? prb_read_valid+0x1b/0x20 [ 2119.696757] ? handle_bug+0x44/0x70 [ 2119.700249] ? exc_invalid_op+0x17/0x60 [ 2119.704089] ? asm_exc_invalid_op+0x1a/0x20 [ 2119.708279] ? __list_add_valid_or_report+0x7d/0xa0 [ 2119.713155] ? __list_add_valid_or_report+0x7d/0xa0 [ 2119.718033] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2119.722925] ceph_add_cap+0x20b/0x620 [ceph] [ 2119.727209] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2119.731941] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2119.737275] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2119.741738] ceph_con_process_message+0x74/0x150 [libceph] [ 2119.747232] ? calc_signature+0xdb/0x100 [libceph] [ 2119.752037] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2119.757370] ? process_scheduled_works+0x22a/0x510 [ 2119.762167] ? process_scheduled_works+0x22a/0x510 [ 2119.766957] ceph_con_workfn+0x344/0x750 [libceph] [ 2119.771761] ? process_scheduled_works+0x22a/0x510 [ 2119.776551] process_scheduled_works+0x2a4/0x510 [ 2119.781176] worker_thread+0x188/0x340 [ 2119.784924] ? rescuer_thread+0x310/0x310 [ 2119.788937] kthread+0xf5/0x130 [ 2119.792084] ? kthread_complete_and_exit+0x20/0x20 [ 2119.796875] ret_from_fork+0x31/0x40 [ 2119.800455] ? kthread_complete_and_exit+0x20/0x20 [ 2119.805250] ret_from_fork_asm+0x11/0x20 [ 2119.809182] [ 2119.811375] irq event stamp: 9476433 [ 2119.814954] hardirqs last enabled at (9476443): [] console_unlock+0x106/0x120 [ 2119.823736] hardirqs last disabled at (9476458): [] console_unlock+0xeb/0x120 [ 2119.832450] softirqs last enabled at (9476484): [] __do_softirq+0x2be/0x371 [ 2119.841063] softirqs last disabled at (9476495): [] irq_exit_rcu+0xb0/0xd0 [ 2119.849491] ---[ end trace 0000000000000000 ]--- [ 2119.854182] ------------[ cut here ]------------ [ 2119.858822] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2119.870482] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2119.879176] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2119.934671] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2119.943794] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2119.951283] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2119.956795] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2119.962298] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2119.981085] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2119.986315] RAX: 0000000000000000 RBX: ffff8881f79d89c8 RCX: 0000000000000000 [ 2119.993468] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2120.000599] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000022e [ 2120.007730] R10: 0000000000062f90 R11: ffffc9000030b828 R12: ffff8881f79d9118 [ 2120.014863] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2120.022011] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2120.030107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.035870] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2120.043020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2120.050171] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2120.057321] Call Trace: [ 2120.059792] [ 2120.061915] ? __warn+0x85/0x180 [ 2120.065168] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.070062] ? report_bug+0xfc/0x1d0 [ 2120.073641] ? prb_read_valid+0x1b/0x20 [ 2120.077483] ? handle_bug+0x44/0x70 [ 2120.080974] ? exc_invalid_op+0x17/0x60 [ 2120.084831] ? asm_exc_invalid_op+0x1a/0x20 [ 2120.089056] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.093932] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.098827] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2120.103752] ceph_add_cap+0x20b/0x620 [ceph] [ 2120.108072] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2120.112809] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2120.118155] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2120.122618] ceph_con_process_message+0x74/0x150 [libceph] [ 2120.128115] ? calc_signature+0xdb/0x100 [libceph] [ 2120.132961] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2120.138313] ? process_scheduled_works+0x22a/0x510 [ 2120.143127] ? process_scheduled_works+0x22a/0x510 [ 2120.147917] ceph_con_workfn+0x344/0x750 [libceph] [ 2120.152752] ? process_scheduled_works+0x22a/0x510 [ 2120.157562] process_scheduled_works+0x2a4/0x510 [ 2120.162207] worker_thread+0x188/0x340 [ 2120.165961] ? rescuer_thread+0x310/0x310 [ 2120.169990] kthread+0xf5/0x130 [ 2120.173153] ? kthread_complete_and_exit+0x20/0x20 [ 2120.177965] ret_from_fork+0x31/0x40 [ 2120.181561] ? kthread_complete_and_exit+0x20/0x20 [ 2120.186354] ret_from_fork_asm+0x11/0x20 [ 2120.190305] [ 2120.192500] irq event stamp: 9477565 [ 2120.196078] hardirqs last enabled at (9477575): [] console_unlock+0x106/0x120 [ 2120.204873] hardirqs last disabled at (9477584): [] console_unlock+0xeb/0x120 [ 2120.213585] softirqs last enabled at (9477562): [] __do_softirq+0x2be/0x371 [ 2120.222191] softirqs last disabled at (9477557): [] irq_exit_rcu+0xb0/0xd0 [ 2120.230639] ---[ end trace 0000000000000000 ]--- [ 2120.235341] ------------[ cut here ]------------ [ 2120.239966] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2120.251618] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2120.260311] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2120.315793] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2120.324937] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2120.332433] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2120.337954] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2120.343473] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2120.362222] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2120.367473] RAX: 0000000000000000 RBX: ffff8881d0bb5808 RCX: 0000000000000000 [ 2120.374605] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2120.381737] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000268 [ 2120.388863] R10: 0000000000063fc8 R11: ffffc9000030b828 R12: ffff8881d0bb5f58 [ 2120.396013] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2120.403163] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2120.411267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.417028] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2120.424172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2120.431321] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2120.438475] Call Trace: [ 2120.440926] [ 2120.443049] ? __warn+0x85/0x180 [ 2120.446297] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.451194] ? report_bug+0xfc/0x1d0 [ 2120.454794] ? prb_read_valid+0x1b/0x20 [ 2120.458651] ? handle_bug+0x44/0x70 [ 2120.462141] ? exc_invalid_op+0x17/0x60 [ 2120.465997] ? asm_exc_invalid_op+0x1a/0x20 [ 2120.470224] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.475108] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.480003] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2120.484910] ceph_add_cap+0x20b/0x620 [ceph] [ 2120.489231] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2120.493977] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2120.499323] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2120.503801] ceph_con_process_message+0x74/0x150 [libceph] [ 2120.509298] ? calc_signature+0xdb/0x100 [libceph] [ 2120.514143] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2120.519485] ? process_scheduled_works+0x22a/0x510 [ 2120.524293] ? process_scheduled_works+0x22a/0x510 [ 2120.529084] ceph_con_workfn+0x344/0x750 [libceph] [ 2120.533906] ? process_scheduled_works+0x22a/0x510 [ 2120.538713] process_scheduled_works+0x2a4/0x510 [ 2120.543355] worker_thread+0x188/0x340 [ 2120.547134] ? rescuer_thread+0x310/0x310 [ 2120.551150] kthread+0xf5/0x130 [ 2120.554311] ? kthread_complete_and_exit+0x20/0x20 [ 2120.559122] ret_from_fork+0x31/0x40 [ 2120.562720] ? kthread_complete_and_exit+0x20/0x20 [ 2120.567512] ret_from_fork_asm+0x11/0x20 [ 2120.571469] [ 2120.573656] irq event stamp: 9478681 [ 2120.577237] hardirqs last enabled at (9478691): [] console_unlock+0x106/0x120 [ 2120.586035] hardirqs last disabled at (9478706): [] console_unlock+0xeb/0x120 [ 2120.594743] softirqs last enabled at (9478720): [] __do_softirq+0x2be/0x371 [ 2120.603348] softirqs last disabled at (9478715): [] irq_exit_rcu+0xb0/0xd0 [ 2120.611799] ---[ end trace 0000000000000000 ]--- [ 2120.616475] ------------[ cut here ]------------ [ 2120.621094] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2120.632788] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2120.641483] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2120.696962] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2120.706106] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2120.713599] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2120.719095] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2120.724596] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2120.743343] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2120.748587] RAX: 0000000000000000 RBX: ffff8881f2076b98 RCX: 0000000000000000 [ 2120.755720] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2120.762856] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000002a2 [ 2120.770001] R10: 0000000000065180 R11: ffffc9000030b828 R12: ffff8881f20772e8 [ 2120.777144] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2120.784295] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2120.792397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.798162] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2120.805314] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2120.812481] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2120.819611] Call Trace: [ 2120.822056] [ 2120.824179] ? __warn+0x85/0x180 [ 2120.827431] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.832325] ? report_bug+0xfc/0x1d0 [ 2120.835924] ? prb_read_valid+0x1b/0x20 [ 2120.839782] ? handle_bug+0x44/0x70 [ 2120.843291] ? exc_invalid_op+0x17/0x60 [ 2120.847146] ? asm_exc_invalid_op+0x1a/0x20 [ 2120.851375] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.856257] ? __list_add_valid_or_report+0x7d/0xa0 [ 2120.861153] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2120.866078] ceph_add_cap+0x20b/0x620 [ceph] [ 2120.870381] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2120.875126] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2120.880475] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2120.884958] ceph_con_process_message+0x74/0x150 [libceph] [ 2120.890476] ? calc_signature+0xdb/0x100 [libceph] [ 2120.895303] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2120.900633] ? process_scheduled_works+0x22a/0x510 [ 2120.905443] ? process_scheduled_works+0x22a/0x510 [ 2120.910231] ceph_con_workfn+0x344/0x750 [libceph] [ 2120.915054] ? process_scheduled_works+0x22a/0x510 [ 2120.919861] process_scheduled_works+0x2a4/0x510 [ 2120.924503] worker_thread+0x188/0x340 [ 2120.928275] ? rescuer_thread+0x310/0x310 [ 2120.932291] kthread+0xf5/0x130 [ 2120.935456] ? kthread_complete_and_exit+0x20/0x20 [ 2120.940264] ret_from_fork+0x31/0x40 [ 2120.943860] ? kthread_complete_and_exit+0x20/0x20 [ 2120.948669] ret_from_fork_asm+0x11/0x20 [ 2120.952604] [ 2120.954799] irq event stamp: 9479739 [ 2120.958393] hardirqs last enabled at (9479749): [] console_unlock+0x106/0x120 [ 2120.967195] hardirqs last disabled at (9479758): [] console_unlock+0xeb/0x120 [ 2120.975899] softirqs last enabled at (9479736): [] __do_softirq+0x2be/0x371 [ 2120.984524] softirqs last disabled at (9479731): [] irq_exit_rcu+0xb0/0xd0 [ 2120.992957] ---[ end trace 0000000000000000 ]--- [ 2120.997737] ------------[ cut here ]------------ [ 2121.002376] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2121.014046] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2121.022739] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2121.078215] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2121.087359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2121.094854] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2121.100365] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2121.105869] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2121.124634] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2121.129859] RAX: 0000000000000000 RBX: ffff8881d1164478 RCX: 0000000000000000 [ 2121.137009] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2121.144157] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000002dc [ 2121.151301] R10: 00000000000661b8 R11: ffffc9000030b828 R12: ffff8881d1164bc8 [ 2121.158449] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2121.165592] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2121.173679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2121.179423] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2121.186566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2121.193699] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2121.200833] Call Trace: [ 2121.203301] [ 2121.205424] ? __warn+0x85/0x180 [ 2121.208677] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.213556] ? report_bug+0xfc/0x1d0 [ 2121.217137] ? prb_read_valid+0x1b/0x20 [ 2121.220992] ? handle_bug+0x44/0x70 [ 2121.224502] ? exc_invalid_op+0x17/0x60 [ 2121.228342] ? asm_exc_invalid_op+0x1a/0x20 [ 2121.232548] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.237422] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.242318] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2121.247227] ceph_add_cap+0x20b/0x620 [ceph] [ 2121.251531] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2121.256277] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2121.261604] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2121.266084] ceph_con_process_message+0x74/0x150 [libceph] [ 2121.271578] ? calc_signature+0xdb/0x100 [libceph] [ 2121.276383] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2121.281734] ? process_scheduled_works+0x22a/0x510 [ 2121.286533] ? process_scheduled_works+0x22a/0x510 [ 2121.291323] ceph_con_workfn+0x344/0x750 [libceph] [ 2121.296142] ? process_scheduled_works+0x22a/0x510 [ 2121.300951] process_scheduled_works+0x2a4/0x510 [ 2121.305592] worker_thread+0x188/0x340 [ 2121.309341] ? rescuer_thread+0x310/0x310 [ 2121.313372] kthread+0xf5/0x130 [ 2121.316533] ? kthread_complete_and_exit+0x20/0x20 [ 2121.321327] ret_from_fork+0x31/0x40 [ 2121.324923] ? kthread_complete_and_exit+0x20/0x20 [ 2121.329735] ret_from_fork_asm+0x11/0x20 [ 2121.333667] [ 2121.335859] irq event stamp: 9480773 [ 2121.339448] hardirqs last enabled at (9480783): [] console_unlock+0x106/0x120 [ 2121.348247] hardirqs last disabled at (9480800): [] console_unlock+0xeb/0x120 [ 2121.356957] softirqs last enabled at (9480798): [] __do_softirq+0x2be/0x371 [ 2121.365581] softirqs last disabled at (9480819): [] irq_exit_rcu+0xb0/0xd0 [ 2121.374011] ---[ end trace 0000000000000000 ]--- [ 2121.378661] ------------[ cut here ]------------ [ 2121.383303] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2121.394974] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2121.403674] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2121.459238] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2121.468363] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2121.475859] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2121.481390] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2121.486900] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2121.505670] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2121.510896] RAX: 0000000000000000 RBX: ffff8881d0b8a720 RCX: 0000000000000000 [ 2121.518048] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2121.525220] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000316 [ 2121.532358] R10: 0000000000067290 R11: ffffc9000030b828 R12: ffff8881d0b8ae70 [ 2121.539509] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2121.546640] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2121.554726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2121.560490] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2121.567625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2121.574754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2121.581889] Call Trace: [ 2121.584359] [ 2121.586499] ? __warn+0x85/0x180 [ 2121.589741] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.594623] ? report_bug+0xfc/0x1d0 [ 2121.598211] ? prb_read_valid+0x1b/0x20 [ 2121.602097] ? handle_bug+0x44/0x70 [ 2121.605594] ? exc_invalid_op+0x17/0x60 [ 2121.609460] ? asm_exc_invalid_op+0x1a/0x20 [ 2121.613664] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.618551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.623447] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2121.628371] ceph_add_cap+0x20b/0x620 [ceph] [ 2121.632675] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2121.637441] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2121.642825] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2121.647320] ceph_con_process_message+0x74/0x150 [libceph] [ 2121.652828] ? calc_signature+0xdb/0x100 [libceph] [ 2121.657671] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2121.663027] ? process_scheduled_works+0x22a/0x510 [ 2121.667844] ? process_scheduled_works+0x22a/0x510 [ 2121.672639] ceph_con_workfn+0x344/0x750 [libceph] [ 2121.677475] ? process_scheduled_works+0x22a/0x510 [ 2121.682265] process_scheduled_works+0x2a4/0x510 [ 2121.686933] worker_thread+0x188/0x340 [ 2121.690693] ? rescuer_thread+0x310/0x310 [ 2121.694706] kthread+0xf5/0x130 [ 2121.697860] ? kthread_complete_and_exit+0x20/0x20 [ 2121.702671] ret_from_fork+0x31/0x40 [ 2121.706259] ? kthread_complete_and_exit+0x20/0x20 [ 2121.711069] ret_from_fork_asm+0x11/0x20 [ 2121.715043] [ 2121.717262] irq event stamp: 9481831 [ 2121.720845] hardirqs last enabled at (9481841): [] console_unlock+0x106/0x120 [ 2121.729650] hardirqs last disabled at (9481850): [] console_unlock+0xeb/0x120 [ 2121.738342] softirqs last enabled at (9481828): [] __do_softirq+0x2be/0x371 [ 2121.746965] softirqs last disabled at (9481821): [] irq_exit_rcu+0xb0/0xd0 [ 2121.755416] ---[ end trace 0000000000000000 ]--- [ 2121.760156] ------------[ cut here ]------------ [ 2121.764809] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2121.776477] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2121.785166] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2121.840663] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2121.849789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2121.857282] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2121.862793] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2121.868298] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2121.887065] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2121.892305] RAX: 0000000000000000 RBX: ffff8881d80ac478 RCX: 0000000000000000 [ 2121.899455] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2121.906607] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000350 [ 2121.913738] R10: 00000000000682b8 R11: ffffc9000030b828 R12: ffff8881d80acbc8 [ 2121.920871] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2121.928021] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2121.936124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2121.941887] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2121.949038] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2121.956200] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2121.963329] Call Trace: [ 2121.965794] [ 2121.967916] ? __warn+0x85/0x180 [ 2121.971163] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.976083] ? report_bug+0xfc/0x1d0 [ 2121.979660] ? prb_read_valid+0x1b/0x20 [ 2121.983505] ? handle_bug+0x44/0x70 [ 2121.986997] ? exc_invalid_op+0x17/0x60 [ 2121.990846] ? asm_exc_invalid_op+0x1a/0x20 [ 2121.995072] ? __list_add_valid_or_report+0x7d/0xa0 [ 2121.999948] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.004844] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2122.009752] ceph_add_cap+0x20b/0x620 [ceph] [ 2122.014053] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2122.018809] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2122.024154] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2122.028617] ceph_con_process_message+0x74/0x150 [libceph] [ 2122.034110] ? calc_signature+0xdb/0x100 [libceph] [ 2122.038950] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2122.044290] ? process_scheduled_works+0x22a/0x510 [ 2122.049082] ? process_scheduled_works+0x22a/0x510 [ 2122.053889] ceph_con_workfn+0x344/0x750 [libceph] [ 2122.058710] ? process_scheduled_works+0x22a/0x510 [ 2122.063502] process_scheduled_works+0x2a4/0x510 [ 2122.068144] worker_thread+0x188/0x340 [ 2122.071900] ? rescuer_thread+0x310/0x310 [ 2122.075929] kthread+0xf5/0x130 [ 2122.079092] ? kthread_complete_and_exit+0x20/0x20 [ 2122.083902] ret_from_fork+0x31/0x40 [ 2122.087504] ? kthread_complete_and_exit+0x20/0x20 [ 2122.092291] ret_from_fork_asm+0x11/0x20 [ 2122.096261] [ 2122.098455] irq event stamp: 9482895 [ 2122.102050] hardirqs last enabled at (9482905): [] console_unlock+0x106/0x120 [ 2122.110847] hardirqs last disabled at (9482914): [] console_unlock+0xeb/0x120 [ 2122.119558] softirqs last enabled at (9482612): [] __do_softirq+0x2be/0x371 [ 2122.128163] softirqs last disabled at (9482607): [] irq_exit_rcu+0xb0/0xd0 [ 2122.136614] ---[ end trace 0000000000000000 ]--- [ 2122.141333] ------------[ cut here ]------------ [ 2122.145975] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2122.157632] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2122.166327] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2122.221848] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2122.230999] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2122.238496] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2122.243999] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2122.249508] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2122.268256] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2122.273500] RAX: 0000000000000000 RBX: ffff8881ec06b0e8 RCX: 0000000000000000 [ 2122.280632] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2122.287764] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000038a [ 2122.294915] R10: 0000000000069548 R11: ffffc9000030b828 R12: ffff8881ec06b838 [ 2122.302065] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2122.309216] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2122.317319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2122.323081] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2122.330231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2122.337381] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2122.344532] Call Trace: [ 2122.346987] [ 2122.349109] ? __warn+0x85/0x180 [ 2122.352360] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.357258] ? report_bug+0xfc/0x1d0 [ 2122.360855] ? prb_read_valid+0x1b/0x20 [ 2122.364716] ? handle_bug+0x44/0x70 [ 2122.368211] ? exc_invalid_op+0x17/0x60 [ 2122.372068] ? asm_exc_invalid_op+0x1a/0x20 [ 2122.376297] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.381178] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.386072] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2122.391001] ceph_add_cap+0x20b/0x620 [ceph] [ 2122.395312] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2122.400061] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2122.405416] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2122.409901] ceph_con_process_message+0x74/0x150 [libceph] [ 2122.415401] ? calc_signature+0xdb/0x100 [libceph] [ 2122.420250] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2122.425585] ? process_scheduled_works+0x22a/0x510 [ 2122.430402] ? process_scheduled_works+0x22a/0x510 [ 2122.435196] ceph_con_workfn+0x344/0x750 [libceph] [ 2122.440045] ? process_scheduled_works+0x22a/0x510 [ 2122.444832] process_scheduled_works+0x2a4/0x510 [ 2122.449504] worker_thread+0x188/0x340 [ 2122.453258] ? rescuer_thread+0x310/0x310 [ 2122.457271] kthread+0xf5/0x130 [ 2122.460434] ? kthread_complete_and_exit+0x20/0x20 [ 2122.465264] ret_from_fork+0x31/0x40 [ 2122.468841] ? kthread_complete_and_exit+0x20/0x20 [ 2122.473653] ret_from_fork_asm+0x11/0x20 [ 2122.477588] [ 2122.479794] irq event stamp: 9483957 [ 2122.483393] hardirqs last enabled at (9483967): [] console_unlock+0x106/0x120 [ 2122.492210] hardirqs last disabled at (9483982): [] console_unlock+0xeb/0x120 [ 2122.500899] softirqs last enabled at (9483980): [] __do_softirq+0x2be/0x371 [ 2122.509522] softirqs last disabled at (9483975): [] irq_exit_rcu+0xb0/0xd0 [ 2122.517955] ---[ end trace 0000000000000000 ]--- [ 2122.522672] ------------[ cut here ]------------ [ 2122.527315] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2122.538985] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2122.547685] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2122.603190] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2122.612340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2122.619818] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2122.625338] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2122.630839] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2122.649607] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2122.654832] RAX: 0000000000000000 RBX: ffff8881ec06ce40 RCX: 0000000000000000 [ 2122.661981] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2122.669132] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000003c4 [ 2122.676280] R10: 000000000006a6c8 R11: ffffc9000030b828 R12: ffff8881ec06d590 [ 2122.683429] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2122.690582] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2122.698666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2122.704414] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2122.711565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2122.718697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2122.725830] Call Trace: [ 2122.728299] [ 2122.730424] ? __warn+0x85/0x180 [ 2122.733674] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.738555] ? report_bug+0xfc/0x1d0 [ 2122.742131] ? prb_read_valid+0x1b/0x20 [ 2122.746010] ? handle_bug+0x44/0x70 [ 2122.749520] ? exc_invalid_op+0x17/0x60 [ 2122.753367] ? asm_exc_invalid_op+0x1a/0x20 [ 2122.757577] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.762476] ? __list_add_valid_or_report+0x7d/0xa0 [ 2122.767352] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2122.772287] ceph_add_cap+0x20b/0x620 [ceph] [ 2122.776584] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2122.781330] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2122.786662] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2122.791146] ceph_con_process_message+0x74/0x150 [libceph] [ 2122.796644] ? calc_signature+0xdb/0x100 [libceph] [ 2122.801470] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2122.806811] ? process_scheduled_works+0x22a/0x510 [ 2122.811612] ? process_scheduled_works+0x22a/0x510 [ 2122.816405] ceph_con_workfn+0x344/0x750 [libceph] [ 2122.821247] ? process_scheduled_works+0x22a/0x510 [ 2122.826044] process_scheduled_works+0x2a4/0x510 [ 2122.830673] worker_thread+0x188/0x340 [ 2122.834445] ? rescuer_thread+0x310/0x310 [ 2122.838455] kthread+0xf5/0x130 [ 2122.841620] ? kthread_complete_and_exit+0x20/0x20 [ 2122.846414] ret_from_fork+0x31/0x40 [ 2122.849990] ? kthread_complete_and_exit+0x20/0x20 [ 2122.854810] ret_from_fork_asm+0x11/0x20 [ 2122.858763] [ 2122.860954] irq event stamp: 9485051 [ 2122.864552] hardirqs last enabled at (9485061): [] console_unlock+0x106/0x120 [ 2122.873332] hardirqs last disabled at (9485070): [] console_unlock+0xeb/0x120 [ 2122.882040] softirqs last enabled at (9484796): [] __do_softirq+0x2be/0x371 [ 2122.890663] softirqs last disabled at (9484791): [] irq_exit_rcu+0xb0/0xd0 [ 2122.899097] ---[ end trace 0000000000000000 ]--- [ 2122.903844] ------------[ cut here ]------------ [ 2122.908501] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2122.920207] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2122.928905] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2122.984476] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2122.993604] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2123.001080] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2123.006614] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2123.012105] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2123.030852] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2123.036076] RAX: 0000000000000000 RBX: ffff8881d821c478 RCX: 0000000000000000 [ 2123.043211] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2123.050343] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000003fe [ 2123.057476] R10: 000000000006b668 R11: ffffc9000030b828 R12: ffff8881d821cbc8 [ 2123.064609] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2123.071743] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2123.079828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2123.085574] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2123.092706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2123.099840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2123.106971] Call Trace: [ 2123.109426] [ 2123.111534] ? __warn+0x85/0x180 [ 2123.114774] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.119657] ? report_bug+0xfc/0x1d0 [ 2123.123240] ? prb_read_valid+0x1b/0x20 [ 2123.127082] ? handle_bug+0x44/0x70 [ 2123.130573] ? exc_invalid_op+0x17/0x60 [ 2123.134412] ? asm_exc_invalid_op+0x1a/0x20 [ 2123.138611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.143500] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.148381] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2123.153327] ceph_add_cap+0x20b/0x620 [ceph] [ 2123.157634] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2123.162398] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2123.167748] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2123.172230] ceph_con_process_message+0x74/0x150 [libceph] [ 2123.177759] ? calc_signature+0xdb/0x100 [libceph] [ 2123.182579] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2123.187919] ? process_scheduled_works+0x22a/0x510 [ 2123.192720] ? process_scheduled_works+0x22a/0x510 [ 2123.197524] ceph_con_workfn+0x344/0x750 [libceph] [ 2123.202337] ? process_scheduled_works+0x22a/0x510 [ 2123.207135] process_scheduled_works+0x2a4/0x510 [ 2123.211767] worker_thread+0x188/0x340 [ 2123.215530] ? rescuer_thread+0x310/0x310 [ 2123.219547] kthread+0xf5/0x130 [ 2123.222692] ? kthread_complete_and_exit+0x20/0x20 [ 2123.227485] ret_from_fork+0x31/0x40 [ 2123.231065] ? kthread_complete_and_exit+0x20/0x20 [ 2123.235857] ret_from_fork_asm+0x11/0x20 [ 2123.239831] [ 2123.242027] irq event stamp: 9486165 [ 2123.245607] hardirqs last enabled at (9486175): [] console_unlock+0x106/0x120 [ 2123.254386] hardirqs last disabled at (9486184): [] console_unlock+0xeb/0x120 [ 2123.263080] softirqs last enabled at (9486198): [] __do_softirq+0x2be/0x371 [ 2123.271685] softirqs last disabled at (9486193): [] irq_exit_rcu+0xb0/0xd0 [ 2123.280117] ---[ end trace 0000000000000000 ]--- [ 2123.284997] ------------[ cut here ]------------ [ 2123.289640] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2123.301976] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2123.310798] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2123.366315] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2123.375445] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2123.382941] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2123.388461] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2123.393962] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2123.412726] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2123.417953] RAX: 0000000000000000 RBX: ffff8881d8323ab0 RCX: 0000000000000000 [ 2123.425104] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2123.432253] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000438 [ 2123.439404] R10: 000000000006c6a8 R11: ffffc9000030b828 R12: ffff8881d8324200 [ 2123.446552] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2123.453686] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2123.461773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2123.467526] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2123.474659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2123.481792] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2123.488926] Call Trace: [ 2123.491380] [ 2123.493521] ? __warn+0x85/0x180 [ 2123.496752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.501632] ? report_bug+0xfc/0x1d0 [ 2123.505210] ? prb_read_valid+0x1b/0x20 [ 2123.509070] ? handle_bug+0x44/0x70 [ 2123.512577] ? exc_invalid_op+0x17/0x60 [ 2123.516419] ? asm_exc_invalid_op+0x1a/0x20 [ 2123.520628] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.525528] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.530404] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2123.535312] ceph_add_cap+0x20b/0x620 [ceph] [ 2123.539616] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2123.544360] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2123.549689] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2123.554167] ceph_con_process_message+0x74/0x150 [libceph] [ 2123.559664] ? calc_signature+0xdb/0x100 [libceph] [ 2123.564468] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2123.569831] ? process_scheduled_works+0x22a/0x510 [ 2123.574643] ? process_scheduled_works+0x22a/0x510 [ 2123.579432] ceph_con_workfn+0x344/0x750 [libceph] [ 2123.584255] ? process_scheduled_works+0x22a/0x510 [ 2123.589063] process_scheduled_works+0x2a4/0x510 [ 2123.593704] worker_thread+0x188/0x340 [ 2123.597453] ? rescuer_thread+0x310/0x310 [ 2123.601480] kthread+0xf5/0x130 [ 2123.604644] ? kthread_complete_and_exit+0x20/0x20 [ 2123.609438] ret_from_fork+0x31/0x40 [ 2123.613035] ? kthread_complete_and_exit+0x20/0x20 [ 2123.617846] ret_from_fork_asm+0x11/0x20 [ 2123.621796] [ 2123.623989] irq event stamp: 9489159 [ 2123.627578] hardirqs last enabled at (9489169): [] console_unlock+0x106/0x120 [ 2123.636356] hardirqs last disabled at (9489178): [] console_unlock+0xeb/0x120 [ 2123.645067] softirqs last enabled at (9488924): [] __do_softirq+0x2be/0x371 [ 2123.653691] softirqs last disabled at (9488919): [] irq_exit_rcu+0xb0/0xd0 [ 2123.662122] ---[ end trace 0000000000000000 ]--- [ 2123.667626] ------------[ cut here ]------------ [ 2123.672301] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2123.684002] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2123.692789] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2123.748276] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2123.757398] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2123.764894] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2123.770408] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2123.775912] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2123.794674] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2123.799900] RAX: 0000000000000000 RBX: ffff8881d099a720 RCX: 0000000000000000 [ 2123.807033] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2123.814183] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000472 [ 2123.821333] R10: 000000000006d6e0 R11: ffffc9000030b828 R12: ffff8881d099ae70 [ 2123.828482] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2123.835624] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2123.843711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2123.849456] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2123.856598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2123.863732] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2123.870865] Call Trace: [ 2123.873333] [ 2123.875450] ? __warn+0x85/0x180 [ 2123.878701] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.883579] ? report_bug+0xfc/0x1d0 [ 2123.887156] ? prb_read_valid+0x1b/0x20 [ 2123.891017] ? handle_bug+0x44/0x70 [ 2123.894534] ? exc_invalid_op+0x17/0x60 [ 2123.898373] ? asm_exc_invalid_op+0x1a/0x20 [ 2123.902583] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.907465] ? __list_add_valid_or_report+0x7d/0xa0 [ 2123.912360] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2123.917287] ceph_add_cap+0x20b/0x620 [ceph] [ 2123.921572] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2123.926317] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2123.931644] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2123.936125] ceph_con_process_message+0x74/0x150 [libceph] [ 2123.941621] ? calc_signature+0xdb/0x100 [libceph] [ 2123.946435] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2123.951781] ? process_scheduled_works+0x22a/0x510 [ 2123.956574] ? process_scheduled_works+0x22a/0x510 [ 2123.961364] ceph_con_workfn+0x344/0x750 [libceph] [ 2123.966202] ? process_scheduled_works+0x22a/0x510 [ 2123.970993] process_scheduled_works+0x2a4/0x510 [ 2123.975635] worker_thread+0x188/0x340 [ 2123.979390] ? rescuer_thread+0x310/0x310 [ 2123.983421] kthread+0xf5/0x130 [ 2123.986583] ? kthread_complete_and_exit+0x20/0x20 [ 2123.991376] ret_from_fork+0x31/0x40 [ 2123.994973] ? kthread_complete_and_exit+0x20/0x20 [ 2123.999785] ret_from_fork_asm+0x11/0x20 [ 2124.003719] [ 2124.005911] irq event stamp: 9490495 [ 2124.009529] hardirqs last enabled at (9490505): [] console_unlock+0x106/0x120 [ 2124.018303] hardirqs last disabled at (9490514): [] console_unlock+0xeb/0x120 [ 2124.027013] softirqs last enabled at (9490482): [] __do_softirq+0x2be/0x371 [ 2124.035638] softirqs last disabled at (9490533): [] irq_exit_rcu+0xb0/0xd0 [ 2124.044070] ---[ end trace 0000000000000000 ]--- [ 2124.048713] ------------[ cut here ]------------ [ 2124.053352] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2124.065037] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2124.073730] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2124.129204] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2124.138332] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2124.145826] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2124.151340] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2124.156844] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2124.175607] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2124.180834] RAX: 0000000000000000 RBX: ffff8881d13ece40 RCX: 0000000000000000 [ 2124.187982] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2124.195124] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000004ac [ 2124.202275] R10: 000000000006e708 R11: ffffc9000030b828 R12: ffff8881d13ed590 [ 2124.209425] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2124.216575] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2124.224660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2124.230406] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2124.237549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2124.244680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2124.251828] Call Trace: [ 2124.254292] [ 2124.256417] ? __warn+0x85/0x180 [ 2124.259671] ? __list_add_valid_or_report+0x7d/0xa0 [ 2124.264548] ? report_bug+0xfc/0x1d0 [ 2124.268127] ? prb_read_valid+0x1b/0x20 [ 2124.271984] ? handle_bug+0x44/0x70 [ 2124.275492] ? exc_invalid_op+0x17/0x60 [ 2124.279349] ? asm_exc_invalid_op+0x1a/0x20 [ 2124.283558] ? __list_add_valid_or_report+0x7d/0xa0 [ 2124.288442] ? __list_add_valid_or_report+0x7d/0xa0 [ 2124.293339] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2124.298245] ceph_add_cap+0x20b/0x620 [ceph] [ 2124.302551] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2124.307296] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2124.312620] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2124.317101] ceph_con_process_message+0x74/0x150 [libceph] [ 2124.322597] ? calc_signature+0xdb/0x100 [libceph] [ 2124.327409] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2124.332750] ? process_scheduled_works+0x22a/0x510 [ 2124.337545] ? process_scheduled_works+0x22a/0x510 [ 2124.342341] ceph_con_workfn+0x344/0x750 [libceph] [ 2124.347160] ? process_scheduled_works+0x22a/0x510 [ 2124.351967] process_scheduled_works+0x2a4/0x510 [ 2124.356611] worker_thread+0x188/0x340 [ 2124.360367] ? rescuer_thread+0x310/0x310 [ 2124.364397] kthread+0xf5/0x130 [ 2124.367559] ? kthread_complete_and_exit+0x20/0x20 [ 2124.372355] ret_from_fork+0x31/0x40 [ 2124.375950] ? kthread_complete_and_exit+0x20/0x20 [ 2124.380761] ret_from_fork_asm+0x11/0x20 [ 2124.384693] [ 2124.386886] irq event stamp: 9491627 [ 2124.390476] hardirqs last enabled at (9491637): [] console_unlock+0x106/0x120 [ 2124.399273] hardirqs last disabled at (9491646): [] console_unlock+0xeb/0x120 [ 2124.407983] softirqs last enabled at (9491414): [] __do_softirq+0x2be/0x371 [ 2124.416606] softirqs last disabled at (9491409): [] irq_exit_rcu+0xb0/0xd0 [ 2124.425038] ---[ end trace 0000000000000000 ]--- [ 2124.429753] ------------[ cut here ]------------ [ 2124.434376] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2124.446054] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2124.454748] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2124.510226] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2124.519369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2124.526864] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2124.532377] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2124.537879] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2124.556646] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2124.561869] RAX: 0000000000000000 RBX: ffff8881d1165808 RCX: 0000000000000000 [ 2124.569012] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2124.576159] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000004e6 [ 2124.583303] R10: 000000000006f718 R11: ffffc9000030b828 R12: ffff8881d1165f58 [ 2124.590454] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2124.597605] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2124.605690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2124.611434] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2124.618585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2124.625717] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2124.632842] Call Trace: [ 2124.635312] [ 2124.637436] ? __warn+0x85/0x180 [ 2124.640687] ? __list_add_valid_or_report+0x7d/0xa0 [ 2124.645572] ? report_bug+0xfc/0x1d0 [ 2124.649155] ? prb_read_valid+0x1b/0x20 [ 2124.653012] ? handle_bug+0x44/0x70 [ 2124.656541] ? exc_invalid_op+0x17/0x60 [ 2124.660377] ? asm_exc_invalid_op+0x1a/0x20 [ 2124.664586] ? __list_add_valid_or_report+0x7d/0xa0 [ 2124.669460] ? __list_add_valid_or_report+0x7d/0xa0 [ 2124.674355] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2124.679264] ceph_add_cap+0x20b/0x620 [ceph] [ 2124.683566] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2124.688305] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2124.693633] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2124.698111] ceph_con_process_message+0x74/0x150 [libceph] [ 2124.703608] ? calc_signature+0xdb/0x100 [libceph] [ 2124.708419] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2124.713760] ? process_scheduled_works+0x22a/0x510 [ 2124.718552] ? process_scheduled_works+0x22a/0x510 [ 2124.723342] ceph_con_workfn+0x344/0x750 [libceph] [ 2124.728163] ? process_scheduled_works+0x22a/0x510 [ 2124.732969] process_scheduled_works+0x2a4/0x510 [ 2124.737611] worker_thread+0x188/0x340 [ 2124.741359] ? rescuer_thread+0x310/0x310 [ 2124.745390] kthread+0xf5/0x130 [ 2124.748553] ? kthread_complete_and_exit+0x20/0x20 [ 2124.753348] ret_from_fork+0x31/0x40 [ 2124.756944] ? kthread_complete_and_exit+0x20/0x20 [ 2124.761753] ret_from_fork_asm+0x11/0x20 [ 2124.765686] [ 2124.767872] irq event stamp: 9492691 [ 2124.771468] hardirqs last enabled at (9492701): [] console_unlock+0x106/0x120 [ 2124.780268] hardirqs last disabled at (9492716): [] console_unlock+0xeb/0x120 [ 2124.788976] softirqs last enabled at (9492714): [] __do_softirq+0x2be/0x371 [ 2124.797598] softirqs last disabled at (9492709): [] irq_exit_rcu+0xb0/0xd0 [ 2124.806029] ---[ end trace 0000000000000000 ]--- [ 2124.810670] ------------[ cut here ]------------ [ 2124.815298] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2124.827096] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2124.835928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2124.891807] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2124.901024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2124.908634] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2124.914137] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2124.919620] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2124.938373] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2124.943658] RAX: 0000000000000000 RBX: ffff8881d8219d58 RCX: 0000000000000000 [ 2124.950799] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2124.957929] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000520 [ 2124.965063] R10: 0000000000070510 R11: ffffc9000030b828 R12: ffff8881d821a4a8 [ 2124.972192] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2124.979325] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2124.987412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2124.993157] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2125.000290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2125.007423] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2125.014557] Call Trace: [ 2125.017010] [ 2125.019115] ? __warn+0x85/0x180 [ 2125.022347] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.027228] ? report_bug+0xfc/0x1d0 [ 2125.030806] ? prb_read_valid+0x1b/0x20 [ 2125.034649] ? handle_bug+0x44/0x70 [ 2125.038138] ? exc_invalid_op+0x17/0x60 [ 2125.041979] ? asm_exc_invalid_op+0x1a/0x20 [ 2125.046169] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.051045] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.055922] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2125.060813] ceph_add_cap+0x20b/0x620 [ceph] [ 2125.065098] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2125.069829] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2125.075155] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2125.079617] ceph_con_process_message+0x74/0x150 [libceph] [ 2125.085114] ? calc_signature+0xdb/0x100 [libceph] [ 2125.089927] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2125.095257] ? process_scheduled_works+0x22a/0x510 [ 2125.100049] ? process_scheduled_works+0x22a/0x510 [ 2125.104837] ceph_con_workfn+0x344/0x750 [libceph] [ 2125.109644] ? process_scheduled_works+0x22a/0x510 [ 2125.114431] process_scheduled_works+0x2a4/0x510 [ 2125.119058] worker_thread+0x188/0x340 [ 2125.122814] ? rescuer_thread+0x310/0x310 [ 2125.126825] kthread+0xf5/0x130 [ 2125.129973] ? kthread_complete_and_exit+0x20/0x20 [ 2125.134765] ret_from_fork+0x31/0x40 [ 2125.138344] ? kthread_complete_and_exit+0x20/0x20 [ 2125.143138] ret_from_fork_asm+0x11/0x20 [ 2125.147071] [ 2125.149264] irq event stamp: 9495831 [ 2125.152843] hardirqs last enabled at (9495841): [] console_unlock+0x106/0x120 [ 2125.161623] hardirqs last disabled at (9495850): [] console_unlock+0xeb/0x120 [ 2125.170317] softirqs last enabled at (9495698): [] __do_softirq+0x2be/0x371 [ 2125.178921] softirqs last disabled at (9495691): [] irq_exit_rcu+0xb0/0xd0 [ 2125.187356] ---[ end trace 0000000000000000 ]--- [ 2125.196535] ------------[ cut here ]------------ [ 2125.201240] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2125.212888] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2125.221618] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2125.277524] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2125.286659] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2125.294140] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2125.299641] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2125.305128] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2125.323874] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2125.329102] RAX: 0000000000000000 RBX: ffff8881e4d361d0 RCX: 0000000000000000 [ 2125.336232] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2125.343364] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000055a [ 2125.350489] R10: 0000000000071528 R11: ffffc9000030b828 R12: ffff8881e4d36920 [ 2125.357622] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2125.364755] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2125.372843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2125.378588] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2125.385721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2125.392853] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2125.399987] Call Trace: [ 2125.402439] [ 2125.404547] ? __warn+0x85/0x180 [ 2125.407779] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.412658] ? report_bug+0xfc/0x1d0 [ 2125.416238] ? prb_read_valid+0x1b/0x20 [ 2125.420079] ? handle_bug+0x44/0x70 [ 2125.423569] ? exc_invalid_op+0x17/0x60 [ 2125.427410] ? asm_exc_invalid_op+0x1a/0x20 [ 2125.431600] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.436485] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.441361] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2125.446253] ceph_add_cap+0x20b/0x620 [ceph] [ 2125.450545] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2125.455265] ceph_fill_trace+0x143/0xf30 [ceph] [ 2125.459806] ? __mutex_lock+0x97/0xce0 [ 2125.463558] ? __mutex_lock+0xaec/0xce0 [ 2125.467400] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2125.471949] ? lock_downgrade+0xfd/0x250 [ 2125.475879] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2125.480258] ceph_con_process_message+0x74/0x150 [libceph] [ 2125.485754] ? calc_signature+0xdb/0x100 [libceph] [ 2125.490566] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2125.495898] ? process_scheduled_works+0x22a/0x510 [ 2125.500690] ? process_scheduled_works+0x22a/0x510 [ 2125.505484] ceph_con_workfn+0x344/0x750 [libceph] [ 2125.510292] ? process_scheduled_works+0x22a/0x510 [ 2125.515083] process_scheduled_works+0x2a4/0x510 [ 2125.519706] worker_thread+0x188/0x340 [ 2125.523457] ? rescuer_thread+0x310/0x310 [ 2125.527467] kthread+0xf5/0x130 [ 2125.530612] ? kthread_complete_and_exit+0x20/0x20 [ 2125.535406] ret_from_fork+0x31/0x40 [ 2125.538987] ? kthread_complete_and_exit+0x20/0x20 [ 2125.543777] ret_from_fork_asm+0x11/0x20 [ 2125.547710] [ 2125.549898] irq event stamp: 9502799 [ 2125.553476] hardirqs last enabled at (9502809): [] console_unlock+0x106/0x120 [ 2125.562256] hardirqs last disabled at (9502818): [] console_unlock+0xeb/0x120 [ 2125.570950] softirqs last enabled at (9502796): [] __do_softirq+0x2be/0x371 [ 2125.579554] softirqs last disabled at (9502791): [] irq_exit_rcu+0xb0/0xd0 [ 2125.587989] ---[ end trace 0000000000000000 ]--- [ 2125.592609] ------------[ cut here ]------------ [ 2125.597225] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2125.608872] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2125.617567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2125.673044] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2125.682171] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2125.689650] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2125.695144] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2125.700629] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2125.719377] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2125.724602] RAX: 0000000000000000 RBX: ffff8881e4d361d0 RCX: 0000000000000000 [ 2125.731736] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2125.738869] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000598 [ 2125.746002] R10: 00000000000728e8 R11: ffffc9000030b8d8 R12: ffff8881e4d36920 [ 2125.753135] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2125.760268] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2125.768354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2125.774099] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2125.781232] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2125.788365] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2125.795498] Call Trace: [ 2125.797950] [ 2125.800057] ? __warn+0x85/0x180 [ 2125.803289] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.808171] ? report_bug+0xfc/0x1d0 [ 2125.811746] ? prb_read_valid+0x1b/0x20 [ 2125.815589] ? handle_bug+0x44/0x70 [ 2125.819081] ? exc_invalid_op+0x17/0x60 [ 2125.822919] ? asm_exc_invalid_op+0x1a/0x20 [ 2125.827111] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.831995] ? __list_add_valid_or_report+0x7d/0xa0 [ 2125.836872] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2125.841764] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2125.846402] ceph_fill_trace+0x143/0xf30 [ceph] [ 2125.850941] ? __mutex_lock+0x97/0xce0 [ 2125.854692] ? __mutex_lock+0xaec/0xce0 [ 2125.858533] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2125.863081] ? lock_downgrade+0xfd/0x250 [ 2125.867012] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2125.871393] ceph_con_process_message+0x74/0x150 [libceph] [ 2125.876886] ? calc_signature+0xdb/0x100 [libceph] [ 2125.881691] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2125.887014] ? process_scheduled_works+0x22a/0x510 [ 2125.891804] ? process_scheduled_works+0x22a/0x510 [ 2125.896594] ceph_con_workfn+0x344/0x750 [libceph] [ 2125.901400] ? process_scheduled_works+0x22a/0x510 [ 2125.906191] process_scheduled_works+0x2a4/0x510 [ 2125.910815] worker_thread+0x188/0x340 [ 2125.914563] ? rescuer_thread+0x310/0x310 [ 2125.918582] kthread+0xf5/0x130 [ 2125.921730] ? kthread_complete_and_exit+0x20/0x20 [ 2125.926525] ret_from_fork+0x31/0x40 [ 2125.930102] ? kthread_complete_and_exit+0x20/0x20 [ 2125.934896] ret_from_fork_asm+0x11/0x20 [ 2125.938829] [ 2125.941022] irq event stamp: 9503831 [ 2125.944600] hardirqs last enabled at (9503841): [] console_unlock+0x106/0x120 [ 2125.953381] hardirqs last disabled at (9503850): [] console_unlock+0xeb/0x120 [ 2125.962074] softirqs last enabled at (9503768): [] __do_softirq+0x2be/0x371 [ 2125.970680] softirqs last disabled at (9503759): [] irq_exit_rcu+0xb0/0xd0 [ 2125.979112] ---[ end trace 0000000000000000 ]--- [ 2125.983733] ------------[ cut here ]------------ [ 2125.988354] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2126.000003] WARNING: CPU: 0 PID: 78659 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2126.008965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2126.064453] CPU: 0 PID: 78659 Comm: kworker/0:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2126.073858] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2126.081339] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2126.086857] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2126.092346] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2126.111093] RSP: 0018:ffffc90001d1f958 EFLAGS: 00010286 [ 2126.116316] RAX: 0000000000000000 RBX: ffff8881d12d30e8 RCX: 0000000000000000 [ 2126.123453] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2126.130586] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000005d5 [ 2126.137718] R10: 00000000000739e8 R11: ffffc90001d1f7e8 R12: ffff8881d12d3838 [ 2126.144849] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2126.151982] FS: 0000000000000000(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2126.160068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2126.165815] CR2: 000055a0beaf5088 CR3: 000000000243e005 CR4: 00000000003706f0 [ 2126.172946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2126.180081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2126.187212] Call Trace: [ 2126.189668] [ 2126.191772] ? __warn+0x85/0x180 [ 2126.195009] ? __list_add_valid_or_report+0x7d/0xa0 [ 2126.199893] ? report_bug+0xfc/0x1d0 [ 2126.203474] ? prb_read_valid+0x1b/0x20 [ 2126.207321] ? handle_bug+0x44/0x70 [ 2126.210815] ? exc_invalid_op+0x17/0x60 [ 2126.214653] ? asm_exc_invalid_op+0x1a/0x20 [ 2126.218848] ? __list_add_valid_or_report+0x7d/0xa0 [ 2126.223729] ? __list_add_valid_or_report+0x7d/0xa0 [ 2126.228607] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2126.233508] ceph_check_caps+0x999/0xae0 [ceph] [ 2126.238072] ? __lock_acquire+0xbfa/0x1e20 [ 2126.242171] ? __lock_acquire+0xb70/0x1e20 [ 2126.246272] ? lock_downgrade+0x250/0x250 [ 2126.250293] ? lock_acquire+0xcb/0x2b0 [ 2126.254046] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2126.259636] ? find_held_lock+0x2d/0x90 [ 2126.263484] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2126.269159] ? lock_release+0xc5/0x260 [ 2126.272918] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2126.278418] writepages_finish+0x302/0x610 [ceph] [ 2126.283147] __complete_request+0x1c/0x80 [libceph] [ 2126.288045] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2126.292607] ? lock_release+0xc5/0x260 [ 2126.296367] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2126.301084] ceph_con_process_message+0x74/0x150 [libceph] [ 2126.306583] ? calc_signature+0xdb/0x100 [libceph] [ 2126.311406] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2126.316736] ? process_scheduled_works+0x22a/0x510 [ 2126.321543] ? process_scheduled_works+0x22a/0x510 [ 2126.326342] ceph_con_workfn+0x344/0x750 [libceph] [ 2126.331147] ? process_scheduled_works+0x22a/0x510 [ 2126.335944] process_scheduled_works+0x2a4/0x510 [ 2126.340573] worker_thread+0x188/0x340 [ 2126.344333] ? rescuer_thread+0x310/0x310 [ 2126.348346] kthread+0xf5/0x130 [ 2126.351493] ? kthread_complete_and_exit+0x20/0x20 [ 2126.356287] ret_from_fork+0x31/0x40 [ 2126.359866] ? kthread_complete_and_exit+0x20/0x20 [ 2126.364659] ret_from_fork_asm+0x11/0x20 [ 2126.368604] [ 2126.370794] irq event stamp: 5506137 [ 2126.374374] hardirqs last enabled at (5506147): [] console_unlock+0x106/0x120 [ 2126.383151] hardirqs last disabled at (5506156): [] console_unlock+0xeb/0x120 [ 2126.391848] softirqs last enabled at (5506170): [] __do_softirq+0x2be/0x371 [ 2126.400460] softirqs last disabled at (5506165): [] irq_exit_rcu+0xb0/0xd0 [ 2126.408892] ---[ end trace 0000000000000000 ]--- [ 2126.413513] ------------[ cut here ]------------ [ 2126.418133] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2126.429787] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2126.438744] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2126.494280] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2126.503234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2126.510711] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2126.516199] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2126.534945] RSP: 0018:ffffc90001fdfaa8 EFLAGS: 00010282 [ 2126.540171] RAX: 0000000000000000 RBX: ffff8881e4d361d0 RCX: 0000000000000000 [ 2126.547307] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2126.554445] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000061a [ 2126.561577] R10: 0000000000074f28 R11: ffffc90001fdf938 R12: ffff8881e4d36920 [ 2126.568711] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2126.575842] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2126.583929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2126.589674] CR2: 00007f9bd398e010 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2126.596808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2126.603942] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2126.611074] Call Trace: [ 2126.613528] [ 2126.615633] ? __warn+0x85/0x180 [ 2126.618866] ? __list_add_valid_or_report+0x7d/0xa0 [ 2126.623748] ? report_bug+0xfc/0x1d0 [ 2126.627334] ? prb_read_valid+0x1b/0x20 [ 2126.631176] ? handle_bug+0x44/0x70 [ 2126.634675] ? exc_invalid_op+0x17/0x60 [ 2126.638513] ? asm_exc_invalid_op+0x1a/0x20 [ 2126.642709] ? __list_add_valid_or_report+0x7d/0xa0 [ 2126.647590] ? __list_add_valid_or_report+0x7d/0xa0 [ 2126.652476] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2126.657378] ceph_open+0x16b/0x440 [ceph] [ 2126.661416] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2126.666568] do_dentry_open+0x172/0x530 [ 2126.670414] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2126.675072] lookup_open+0x37b/0x630 [ 2126.678666] ? verify_cpu+0x20/0x100 [ 2126.682252] ? verify_cpu+0x20/0x100 [ 2126.685840] path_openat+0x25a/0x9d0 [ 2126.689434] do_filp_open+0xb8/0x120 [ 2126.693029] ? lock_release+0xc5/0x260 [ 2126.696785] ? _raw_spin_unlock+0x23/0x40 [ 2126.700801] ? alloc_fd+0x13f/0x200 [ 2126.704301] do_sys_openat2+0x249/0x290 [ 2126.708146] do_sys_open+0x4f/0x80 [ 2126.711564] do_syscall_64+0x38/0x80 [ 2126.715147] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2126.720198] RIP: 0033:0x7f48f0eed2f2 [ 2126.723779] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2126.742523] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2126.750090] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2126.757223] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2126.764354] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2126.771489] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2126.778622] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bb [ 2126.785767] [ 2126.787963] irq event stamp: 12782631 [ 2126.791630] hardirqs last enabled at (12782641): [] console_unlock+0x106/0x120 [ 2126.800497] hardirqs last disabled at (12782650): [] console_unlock+0xeb/0x120 [ 2126.809274] softirqs last enabled at (12782462): [] __do_softirq+0x2be/0x371 [ 2126.817969] softirqs last disabled at (12782451): [] irq_exit_rcu+0xb0/0xd0 [ 2126.826488] ---[ end trace 0000000000000000 ]--- [ 2126.831108] ------------[ cut here ]------------ [ 2126.835726] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2126.847392] WARNING: CPU: 0 PID: 68426 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2126.856365] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2126.911847] CPU: 0 PID: 68426 Comm: kworker/0:3 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2126.921234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2126.928712] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2126.934224] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2126.939710] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2126.958462] RSP: 0018:ffffc90009257958 EFLAGS: 00010286 [ 2126.963691] RAX: 0000000000000000 RBX: ffff8881d12d3ab0 RCX: 0000000000000000 [ 2126.970824] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2126.977958] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000658 [ 2126.985090] R10: 0000000000076030 R11: ffffc900092577e8 R12: ffff8881d12d4200 [ 2126.992221] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2126.999356] FS: 0000000000000000(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2127.007441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2127.013188] CR2: 000055a0beaf5088 CR3: 000000000243e005 CR4: 00000000003706f0 [ 2127.020319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2127.027452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2127.034585] Call Trace: [ 2127.037040] [ 2127.039146] ? __warn+0x85/0x180 [ 2127.042382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.047267] ? report_bug+0xfc/0x1d0 [ 2127.050846] ? prb_read_valid+0x1b/0x20 [ 2127.054686] ? handle_bug+0x44/0x70 [ 2127.058178] ? exc_invalid_op+0x17/0x60 [ 2127.062017] ? asm_exc_invalid_op+0x1a/0x20 [ 2127.066212] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.071093] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.075972] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2127.080869] ceph_check_caps+0x999/0xae0 [ceph] [ 2127.085430] ? __lock_acquire+0xbfa/0x1e20 [ 2127.089535] ? __lock_acquire+0xb70/0x1e20 [ 2127.093635] ? lock_downgrade+0x250/0x250 [ 2127.097649] ? lock_acquire+0xcb/0x2b0 [ 2127.101401] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2127.106993] ? find_held_lock+0x2d/0x90 [ 2127.110843] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2127.116515] ? lock_release+0xc5/0x260 [ 2127.120275] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2127.125774] writepages_finish+0x302/0x610 [ceph] [ 2127.130511] __complete_request+0x1c/0x80 [libceph] [ 2127.135409] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2127.139970] ? lock_release+0xc5/0x260 [ 2127.143731] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2127.148448] ceph_con_process_message+0x74/0x150 [libceph] [ 2127.153944] ? calc_signature+0xdb/0x100 [libceph] [ 2127.158761] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2127.164091] ? process_scheduled_works+0x22a/0x510 [ 2127.168895] ? process_scheduled_works+0x22a/0x510 [ 2127.173689] ceph_con_workfn+0x344/0x750 [libceph] [ 2127.178503] ? process_scheduled_works+0x22a/0x510 [ 2127.183298] process_scheduled_works+0x2a4/0x510 [ 2127.187931] worker_thread+0x188/0x340 [ 2127.191682] ? rescuer_thread+0x310/0x310 [ 2127.195696] kthread+0xf5/0x130 [ 2127.198847] ? kthread_complete_and_exit+0x20/0x20 [ 2127.203645] ret_from_fork+0x31/0x40 [ 2127.207230] ? kthread_complete_and_exit+0x20/0x20 [ 2127.212023] ret_from_fork_asm+0x11/0x20 [ 2127.215960] [ 2127.218148] irq event stamp: 40290725 [ 2127.221813] hardirqs last enabled at (40290735): [] console_unlock+0x106/0x120 [ 2127.230683] hardirqs last disabled at (40290744): [] console_unlock+0xeb/0x120 [ 2127.239472] softirqs last enabled at (40290758): [] __do_softirq+0x2be/0x371 [ 2127.248172] softirqs last disabled at (40290753): [] irq_exit_rcu+0xb0/0xd0 [ 2127.256698] ---[ end trace 0000000000000000 ]--- [ 2127.261319] ------------[ cut here ]------------ [ 2127.266039] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2127.277694] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2127.286556] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2127.342174] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2127.351389] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2127.358876] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2127.364481] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2127.369994] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2127.388818] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2127.394082] RAX: 0000000000000000 RBX: ffff8881e4f8f560 RCX: 0000000000000000 [ 2127.401311] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2127.408529] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000069d [ 2127.415768] R10: 00000000000773b8 R11: ffffc9000030b828 R12: ffff8881e4f8fcb0 [ 2127.423019] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2127.430227] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2127.438387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2127.444217] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2127.451415] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2127.458551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2127.465713] Call Trace: [ 2127.468163] [ 2127.470268] ? __warn+0x85/0x180 [ 2127.473501] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.478382] ? report_bug+0xfc/0x1d0 [ 2127.481959] ? prb_read_valid+0x1b/0x20 [ 2127.485803] ? handle_bug+0x44/0x70 [ 2127.489291] ? exc_invalid_op+0x17/0x60 [ 2127.493130] ? asm_exc_invalid_op+0x1a/0x20 [ 2127.497327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.502208] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.507091] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2127.512002] ceph_add_cap+0x20b/0x620 [ceph] [ 2127.516303] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2127.521046] ceph_fill_trace+0x143/0xf30 [ceph] [ 2127.525588] ? __mutex_lock+0x97/0xce0 [ 2127.529341] ? __mutex_lock+0xaec/0xce0 [ 2127.533181] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2127.537730] ? lock_downgrade+0xfd/0x250 [ 2127.541667] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2127.546049] ceph_con_process_message+0x74/0x150 [libceph] [ 2127.551544] ? calc_signature+0xdb/0x100 [libceph] [ 2127.556362] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2127.561692] ? process_scheduled_works+0x22a/0x510 [ 2127.566493] ? process_scheduled_works+0x22a/0x510 [ 2127.571287] ceph_con_workfn+0x344/0x750 [libceph] [ 2127.576094] ? process_scheduled_works+0x22a/0x510 [ 2127.580891] process_scheduled_works+0x2a4/0x510 [ 2127.585520] worker_thread+0x188/0x340 [ 2127.589273] ? rescuer_thread+0x310/0x310 [ 2127.593285] kthread+0xf5/0x130 [ 2127.596431] ? kthread_complete_and_exit+0x20/0x20 [ 2127.601227] ret_from_fork+0x31/0x40 [ 2127.604810] ? kthread_complete_and_exit+0x20/0x20 [ 2127.609605] ret_from_fork_asm+0x11/0x20 [ 2127.613545] [ 2127.615739] irq event stamp: 9522857 [ 2127.619318] hardirqs last enabled at (9522873): [] console_unlock+0x106/0x120 [ 2127.628100] hardirqs last disabled at (9522888): [] console_unlock+0xeb/0x120 [ 2127.636792] softirqs last enabled at (9522902): [] __do_softirq+0x2be/0x371 [ 2127.645398] softirqs last disabled at (9522913): [] irq_exit_rcu+0xb0/0xd0 [ 2127.653832] ---[ end trace 0000000000000000 ]--- [ 2127.658450] ------------[ cut here ]------------ [ 2127.663067] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2127.674735] WARNING: CPU: 0 PID: 78659 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2127.683706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2127.739188] CPU: 0 PID: 78659 Comm: kworker/0:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2127.748580] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2127.756065] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2127.761580] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2127.767072] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2127.785819] RSP: 0018:ffffc90001d1f958 EFLAGS: 00010286 [ 2127.791051] RAX: 0000000000000000 RBX: ffff8881d0bd9390 RCX: 0000000000000000 [ 2127.798186] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2127.805323] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000006db [ 2127.812460] R10: 00000000000784a8 R11: ffffc90001d1f7e8 R12: ffff8881d0bd9ae0 [ 2127.819591] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2127.826724] FS: 0000000000000000(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2127.834811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2127.840558] CR2: 000055a0beaf5088 CR3: 000000000243e005 CR4: 00000000003706f0 [ 2127.847696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2127.854834] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2127.861965] Call Trace: [ 2127.864418] [ 2127.866522] ? __warn+0x85/0x180 [ 2127.869761] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.874646] ? report_bug+0xfc/0x1d0 [ 2127.878224] ? prb_read_valid+0x1b/0x20 [ 2127.882070] ? handle_bug+0x44/0x70 [ 2127.885565] ? exc_invalid_op+0x17/0x60 [ 2127.889405] ? asm_exc_invalid_op+0x1a/0x20 [ 2127.893599] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.898480] ? __list_add_valid_or_report+0x7d/0xa0 [ 2127.903359] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2127.908259] ceph_check_caps+0x999/0xae0 [ceph] [ 2127.912825] ? __lock_acquire+0xbfa/0x1e20 [ 2127.916923] ? __lock_acquire+0xb70/0x1e20 [ 2127.921023] ? lock_downgrade+0x250/0x250 [ 2127.925036] ? lock_acquire+0xcb/0x2b0 [ 2127.928786] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2127.934380] ? find_held_lock+0x2d/0x90 [ 2127.938226] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2127.943903] ? lock_release+0xc5/0x260 [ 2127.947660] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2127.953166] writepages_finish+0x302/0x610 [ceph] [ 2127.957899] __complete_request+0x1c/0x80 [libceph] [ 2127.962796] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2127.967358] ? lock_release+0xc5/0x260 [ 2127.971117] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2127.975827] ceph_con_process_message+0x74/0x150 [libceph] [ 2127.981323] ? calc_signature+0xdb/0x100 [libceph] [ 2127.986140] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2127.991470] ? process_scheduled_works+0x22a/0x510 [ 2127.996273] ? process_scheduled_works+0x22a/0x510 [ 2128.001070] ceph_con_workfn+0x344/0x750 [libceph] [ 2128.005892] ? process_scheduled_works+0x22a/0x510 [ 2128.010686] process_scheduled_works+0x2a4/0x510 [ 2128.015317] worker_thread+0x188/0x340 [ 2128.019070] ? rescuer_thread+0x310/0x310 [ 2128.023083] kthread+0xf5/0x130 [ 2128.026226] ? kthread_complete_and_exit+0x20/0x20 [ 2128.031032] ret_from_fork+0x31/0x40 [ 2128.034617] ? kthread_complete_and_exit+0x20/0x20 [ 2128.039412] ret_from_fork_asm+0x11/0x20 [ 2128.043357] [ 2128.045545] irq event stamp: 5517149 [ 2128.049123] hardirqs last enabled at (5517159): [] console_unlock+0x106/0x120 [ 2128.057909] hardirqs last disabled at (5517180): [] console_unlock+0xeb/0x120 [ 2128.066614] softirqs last enabled at (5517196): [] __do_softirq+0x2be/0x371 [ 2128.075237] softirqs last disabled at (5517207): [] irq_exit_rcu+0xb0/0xd0 [ 2128.083670] ---[ end trace 0000000000000000 ]--- [ 2128.088293] ------------[ cut here ]------------ [ 2128.092909] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2128.104558] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2128.113512] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2128.169032] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2128.177983] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2128.185463] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2128.190948] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2128.209696] RSP: 0018:ffffc90001fdf8c0 EFLAGS: 00010282 [ 2128.214921] RAX: 0000000000000000 RBX: ffff8881e4d361d0 RCX: 0000000000000000 [ 2128.222054] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2128.229185] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000720 [ 2128.236320] R10: 0000000000079860 R11: ffffc90001fdf750 R12: ffff8881e4d36920 [ 2128.243453] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2128.250588] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2128.258672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2128.264418] CR2: 00007f9bd398e010 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2128.271549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2128.278682] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2128.285815] Call Trace: [ 2128.288269] [ 2128.290375] ? __warn+0x85/0x180 [ 2128.293608] ? __list_add_valid_or_report+0x7d/0xa0 [ 2128.298490] ? report_bug+0xfc/0x1d0 [ 2128.302074] ? prb_read_valid+0x1b/0x20 [ 2128.305918] ? handle_bug+0x44/0x70 [ 2128.309408] ? exc_invalid_op+0x17/0x60 [ 2128.313247] ? asm_exc_invalid_op+0x1a/0x20 [ 2128.317441] ? __list_add_valid_or_report+0x7d/0xa0 [ 2128.322324] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2128.327230] ceph_check_caps+0x999/0xae0 [ceph] [ 2128.331792] ? handle_bug+0x44/0x70 [ 2128.335286] ? exc_invalid_op+0x17/0x60 [ 2128.339127] ? asm_exc_invalid_op+0x1a/0x20 [ 2128.343317] ? find_held_lock+0x2d/0x90 [ 2128.347160] ? ceph_open+0x175/0x440 [ceph] [ 2128.351364] ? lock_release+0xc5/0x260 [ 2128.355122] ceph_open+0x3cc/0x440 [ceph] [ 2128.359151] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2128.364304] do_dentry_open+0x172/0x530 [ 2128.368150] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2128.372808] lookup_open+0x37b/0x630 [ 2128.376401] ? verify_cpu+0x20/0x100 [ 2128.379987] ? verify_cpu+0x20/0x100 [ 2128.383573] path_openat+0x25a/0x9d0 [ 2128.387165] do_filp_open+0xb8/0x120 [ 2128.390755] ? lock_release+0xc5/0x260 [ 2128.394511] ? _raw_spin_unlock+0x23/0x40 [ 2128.398528] ? alloc_fd+0x13f/0x200 [ 2128.402026] do_sys_openat2+0x249/0x290 [ 2128.405873] do_sys_open+0x4f/0x80 [ 2128.409287] do_syscall_64+0x38/0x80 [ 2128.412873] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2128.417927] RIP: 0033:0x7f48f0eed2f2 [ 2128.421504] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2128.440250] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2128.447815] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2128.454950] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2128.462081] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2128.469215] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2128.476347] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bb [ 2128.483493] [ 2128.485683] irq event stamp: 12786295 [ 2128.489348] hardirqs last enabled at (12786305): [] console_unlock+0x106/0x120 [ 2128.498215] hardirqs last disabled at (12786314): [] console_unlock+0xeb/0x120 [ 2128.506993] softirqs last enabled at (12786076): [] __do_softirq+0x2be/0x371 [ 2128.515687] softirqs last disabled at (12786063): [] irq_exit_rcu+0xb0/0xd0 [ 2128.524207] ---[ end trace 0000000000000000 ]--- [ 2128.528826] ------------[ cut here ]------------ [ 2128.533447] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2128.545091] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2128.553786] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2128.609278] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2128.618406] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2128.625888] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2128.631397] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2128.636881] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2128.655629] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2128.660854] RAX: 0000000000000000 RBX: ffff8881e4f8f560 RCX: 0000000000000000 [ 2128.667990] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2128.675120] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000764 [ 2128.682254] R10: 000000000007ac48 R11: ffffc9000030b8d8 R12: ffff8881e4f8fcb0 [ 2128.689385] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2128.696519] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2128.704606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2128.710350] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2128.717485] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2128.724617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2128.731749] Call Trace: [ 2128.734203] [ 2128.736310] ? __warn+0x85/0x180 [ 2128.739544] ? __list_add_valid_or_report+0x7d/0xa0 [ 2128.744425] ? report_bug+0xfc/0x1d0 [ 2128.748008] ? prb_read_valid+0x1b/0x20 [ 2128.751854] ? handle_bug+0x44/0x70 [ 2128.755350] ? exc_invalid_op+0x17/0x60 [ 2128.759190] ? asm_exc_invalid_op+0x1a/0x20 [ 2128.763386] ? __list_add_valid_or_report+0x7d/0xa0 [ 2128.768265] ? __list_add_valid_or_report+0x7d/0xa0 [ 2128.773142] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2128.778045] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2128.782701] ceph_fill_trace+0x143/0xf30 [ceph] [ 2128.787245] ? __mutex_lock+0x97/0xce0 [ 2128.790995] ? __mutex_lock+0xaec/0xce0 [ 2128.794838] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2128.799386] ? lock_downgrade+0xfd/0x250 [ 2128.803330] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2128.807725] ceph_con_process_message+0x74/0x150 [libceph] [ 2128.813236] ? calc_signature+0xdb/0x100 [libceph] [ 2128.818051] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2128.823386] ? process_scheduled_works+0x22a/0x510 [ 2128.828185] ? process_scheduled_works+0x22a/0x510 [ 2128.833001] ceph_con_workfn+0x344/0x750 [libceph] [ 2128.837811] ? process_scheduled_works+0x22a/0x510 [ 2128.842607] process_scheduled_works+0x2a4/0x510 [ 2128.847241] worker_thread+0x188/0x340 [ 2128.850998] ? rescuer_thread+0x310/0x310 [ 2128.855011] kthread+0xf5/0x130 [ 2128.858156] ? kthread_complete_and_exit+0x20/0x20 [ 2128.862949] ret_from_fork+0x31/0x40 [ 2128.866528] ? kthread_complete_and_exit+0x20/0x20 [ 2128.871320] ret_from_fork_asm+0x11/0x20 [ 2128.875263] [ 2128.877455] irq event stamp: 9532449 [ 2128.881035] hardirqs last enabled at (9532469): [] console_unlock+0x106/0x120 [ 2128.889815] hardirqs last disabled at (9532484): [] console_unlock+0xeb/0x120 [ 2128.898509] softirqs last enabled at (9532498): [] __do_softirq+0x2be/0x371 [ 2128.907113] softirqs last disabled at (9532509): [] irq_exit_rcu+0xb0/0xd0 [ 2128.915547] ---[ end trace 0000000000000000 ]--- [ 2128.921172] ------------[ cut here ]------------ [ 2128.925794] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2128.937446] WARNING: CPU: 5 PID: 76079 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2128.946401] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2129.001876] CPU: 5 PID: 76079 Comm: kworker/5:0 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2129.011264] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2129.018742] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2129.024237] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2129.029722] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2129.048470] RSP: 0018:ffffc900081f7958 EFLAGS: 00010286 [ 2129.053693] RAX: 0000000000000000 RBX: ffff8881d0bd9d58 RCX: 0000000000000000 [ 2129.060835] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2129.067968] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000007a1 [ 2129.075098] R10: 000000000007bd20 R11: ffffc900081f77e8 R12: ffff8881d0bda4a8 [ 2129.082227] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2129.089360] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2129.097444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2129.103189] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2129.110322] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2129.117447] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2129.124579] Call Trace: [ 2129.127027] [ 2129.129132] ? __warn+0x85/0x180 [ 2129.132364] ? __list_add_valid_or_report+0x7d/0xa0 [ 2129.137246] ? report_bug+0xfc/0x1d0 [ 2129.140823] ? prb_read_valid+0x1b/0x20 [ 2129.144664] ? handle_bug+0x44/0x70 [ 2129.148154] ? exc_invalid_op+0x17/0x60 [ 2129.151994] ? asm_exc_invalid_op+0x1a/0x20 [ 2129.156185] ? __list_add_valid_or_report+0x7d/0xa0 [ 2129.161061] ? __list_add_valid_or_report+0x7d/0xa0 [ 2129.165938] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2129.170829] ceph_check_caps+0x999/0xae0 [ceph] [ 2129.175376] ? __lock_acquire+0xbfa/0x1e20 [ 2129.179478] ? __lock_acquire+0xb70/0x1e20 [ 2129.183580] ? lock_downgrade+0x250/0x250 [ 2129.187591] ? lock_acquire+0xcb/0x2b0 [ 2129.191342] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2129.196924] ? find_held_lock+0x2d/0x90 [ 2129.200764] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2129.206432] ? lock_release+0xc5/0x260 [ 2129.210186] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2129.215682] writepages_finish+0x302/0x610 [ceph] [ 2129.220409] __complete_request+0x1c/0x80 [libceph] [ 2129.225300] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2129.229849] ? lock_release+0xc5/0x260 [ 2129.233606] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2129.238309] ceph_con_process_message+0x74/0x150 [libceph] [ 2129.243802] ? calc_signature+0xdb/0x100 [libceph] [ 2129.248615] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2129.253944] ? process_scheduled_works+0x22a/0x510 [ 2129.258738] ? process_scheduled_works+0x22a/0x510 [ 2129.263528] ceph_con_workfn+0x344/0x750 [libceph] [ 2129.268331] ? process_scheduled_works+0x22a/0x510 [ 2129.273121] process_scheduled_works+0x2a4/0x510 [ 2129.277744] worker_thread+0x188/0x340 [ 2129.281494] ? rescuer_thread+0x310/0x310 [ 2129.285507] kthread+0xf5/0x130 [ 2129.288653] ? kthread_complete_and_exit+0x20/0x20 [ 2129.293447] ret_from_fork+0x31/0x40 [ 2129.297026] ? kthread_complete_and_exit+0x20/0x20 [ 2129.301817] ret_from_fork_asm+0x11/0x20 [ 2129.305750] [ 2129.307936] irq event stamp: 14119059 [ 2129.311606] hardirqs last enabled at (14119069): [] console_unlock+0x106/0x120 [ 2129.320469] hardirqs last disabled at (14119078): [] console_unlock+0xeb/0x120 [ 2129.329248] softirqs last enabled at (14118776): [] __do_softirq+0x2be/0x371 [ 2129.337942] softirqs last disabled at (14118771): [] irq_exit_rcu+0xb0/0xd0 [ 2129.346464] ---[ end trace 0000000000000000 ]--- [ 2129.354005] ------------[ cut here ]------------ [ 2129.358633] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2129.370980] WARNING: CPU: 5 PID: 76079 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2129.380183] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2129.435676] CPU: 5 PID: 76079 Comm: kworker/5:0 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2129.445667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2129.453250] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2129.458829] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2129.464400] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2129.483217] RSP: 0018:ffffc900081f7958 EFLAGS: 00010286 [ 2129.488454] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2129.495615] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2129.502749] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000007e6 [ 2129.509884] R10: 000000000007d540 R11: ffffc900081f77e8 R12: ffff8881c3031118 [ 2129.517014] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2129.524155] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2129.532280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2129.538026] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2129.545158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2129.552288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2129.559425] Call Trace: [ 2129.561878] [ 2129.563981] ? __warn+0x85/0x180 [ 2129.567213] ? __list_add_valid_or_report+0x7d/0xa0 [ 2129.572094] ? report_bug+0xfc/0x1d0 [ 2129.575674] ? prb_read_valid+0x1b/0x20 [ 2129.579513] ? handle_bug+0x44/0x70 [ 2129.583004] ? exc_invalid_op+0x17/0x60 [ 2129.586844] ? asm_exc_invalid_op+0x1a/0x20 [ 2129.591034] ? __list_add_valid_or_report+0x7d/0xa0 [ 2129.595909] ? __list_add_valid_or_report+0x7d/0xa0 [ 2129.600790] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2129.605689] ceph_check_caps+0x999/0xae0 [ceph] [ 2129.610239] ? __lock_acquire+0xbfa/0x1e20 [ 2129.614346] ? __lock_acquire+0xb70/0x1e20 [ 2129.618444] ? lock_downgrade+0x250/0x250 [ 2129.622459] ? lock_acquire+0xcb/0x2b0 [ 2129.626207] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2129.631791] ? find_held_lock+0x2d/0x90 [ 2129.635629] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2129.641299] ? lock_release+0xc5/0x260 [ 2129.645053] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2129.650548] writepages_finish+0x302/0x610 [ceph] [ 2129.655266] __complete_request+0x1c/0x80 [libceph] [ 2129.660158] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2129.664708] ? lock_release+0xc5/0x260 [ 2129.668462] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2129.673167] ceph_con_process_message+0x74/0x150 [libceph] [ 2129.678660] ? calc_signature+0xdb/0x100 [libceph] [ 2129.683464] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2129.688795] ? process_scheduled_works+0x22a/0x510 [ 2129.693589] ? process_scheduled_works+0x22a/0x510 [ 2129.698377] ceph_con_workfn+0x344/0x750 [libceph] [ 2129.703183] ? process_scheduled_works+0x22a/0x510 [ 2129.707971] process_scheduled_works+0x2a4/0x510 [ 2129.712596] worker_thread+0x188/0x340 [ 2129.716344] ? rescuer_thread+0x310/0x310 [ 2129.720357] kthread+0xf5/0x130 [ 2129.723503] ? kthread_complete_and_exit+0x20/0x20 [ 2129.728294] ret_from_fork+0x31/0x40 [ 2129.731874] ? kthread_complete_and_exit+0x20/0x20 [ 2129.736667] ret_from_fork_asm+0x11/0x20 [ 2129.740600] [ 2129.742786] irq event stamp: 14138183 [ 2129.746451] hardirqs last enabled at (14138193): [] console_unlock+0x106/0x120 [ 2129.755319] hardirqs last disabled at (14138202): [] console_unlock+0xeb/0x120 [ 2129.764099] softirqs last enabled at (14138080): [] __do_softirq+0x2be/0x371 [ 2129.772790] softirqs last disabled at (14138075): [] irq_exit_rcu+0xb0/0xd0 [ 2129.781309] ---[ end trace 0000000000000000 ]--- [ 2129.786605] ------------[ cut here ]------------ [ 2129.791466] list_del corruption. next->prev should be ffff8881e4ee5590, but was ffff8881d12d3838. (next=ffff8881d12d3838) [ 2129.802438] WARNING: CPU: 6 PID: 115 at lib/list_debug.c:67 __list_del_entry_valid_or_report+0xc3/0xd0 [ 2129.811771] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2129.867358] CPU: 6 PID: 115 Comm: kworker/u16:6 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2129.876777] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2129.884278] Workqueue: writeback wb_workfn (flush-ceph-1) [ 2129.889709] RIP: 0010:__list_del_entry_valid_or_report+0xc3/0xd0 [ 2129.895742] Code: 0b 31 c0 c3 48 89 d1 48 89 f2 48 89 fe 48 c7 c7 58 9a 1d 82 e8 ae 0b b0 ff 0f 0b 31 c0 c3 48 c7 c7 a0 9a 1d 82 e8 9d 0b b0 ff <0f> 0b 31 c0 c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 d2 74 27 [ 2129.914506] RSP: 0018:ffffc90000f5fb40 EFLAGS: 00010286 [ 2129.919754] RAX: 0000000000000000 RBX: ffff8881e4ee4e40 RCX: 0000000000000000 [ 2129.926906] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2129.934072] RBP: ffff8881e4ee52e8 R08: 0000000000000000 R09: c00000010000082b [ 2129.941225] R10: 000000000007e800 R11: ffffc90000f5f9d0 R12: ffff88817f867000 [ 2129.948387] R13: ffff8881e4ee5590 R14: 0000000000000001 R15: ffff88817f867488 [ 2129.955550] FS: 0000000000000000(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2129.963662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2129.969439] CR2: 00007fb38e5b7000 CR3: 000000000243e005 CR4: 00000000003706e0 [ 2129.976592] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2129.983740] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2129.990898] Call Trace: [ 2129.993376] [ 2129.995494] ? __warn+0x85/0x180 [ 2129.998747] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2130.004174] ? report_bug+0xfc/0x1d0 [ 2130.007774] ? prb_read_valid+0x1b/0x20 [ 2130.011654] ? handle_bug+0x44/0x70 [ 2130.015182] ? exc_invalid_op+0x17/0x60 [ 2130.019060] ? asm_exc_invalid_op+0x1a/0x20 [ 2130.023274] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2130.028699] ceph_write_inode+0x1c0/0x340 [ceph] [ 2130.033360] ? lock_release+0xc5/0x260 [ 2130.037135] __writeback_single_inode+0x346/0x430 [ 2130.041864] writeback_sb_inodes+0x1f4/0x4c0 [ 2130.046169] __writeback_inodes_wb+0x5f/0xc0 [ 2130.050480] wb_writeback+0x2c6/0x350 [ 2130.054185] wb_workfn+0x3a5/0x510 [ 2130.057632] ? process_scheduled_works+0x22a/0x510 [ 2130.062471] ? process_scheduled_works+0x22a/0x510 [ 2130.067284] process_scheduled_works+0x2a4/0x510 [ 2130.071934] worker_thread+0x188/0x340 [ 2130.075706] ? rescuer_thread+0x310/0x310 [ 2130.079735] kthread+0xf5/0x130 [ 2130.082898] ? kthread_complete_and_exit+0x20/0x20 [ 2130.087709] ret_from_fork+0x31/0x40 [ 2130.091305] ? kthread_complete_and_exit+0x20/0x20 [ 2130.096115] ret_from_fork_asm+0x11/0x20 [ 2130.100073] [ 2130.102295] irq event stamp: 10263313 [ 2130.105993] hardirqs last enabled at (10263371): [] console_unlock+0x106/0x120 [ 2130.114881] hardirqs last disabled at (10263430): [] console_unlock+0xeb/0x120 [ 2130.123675] softirqs last enabled at (10263482): [] __do_softirq+0x2be/0x371 [ 2130.132406] softirqs last disabled at (10263493): [] irq_exit_rcu+0xb0/0xd0 [ 2130.140950] ---[ end trace 0000000000000000 ]--- [ 2130.145586] ------------[ cut here ]------------ [ 2130.150206] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2130.161858] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2130.170815] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2130.226293] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2130.235242] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2130.242722] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2130.248208] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2130.266955] RSP: 0018:ffffc90001fefaa8 EFLAGS: 00010282 [ 2130.272180] RAX: 0000000000000000 RBX: ffff8881e4f8f560 RCX: 0000000000000000 [ 2130.279314] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2130.286447] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000861 [ 2130.293579] R10: 000000000007f908 R11: ffffc90001fef938 R12: ffff8881e4f8fcb0 [ 2130.300713] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2130.307848] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2130.315932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2130.321678] CR2: 00007f9bd346f000 CR3: 0000000108c72006 CR4: 00000000003706e0 [ 2130.328809] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2130.335933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2130.343066] Call Trace: [ 2130.345520] [ 2130.347627] ? __warn+0x85/0x180 [ 2130.350857] ? __list_add_valid_or_report+0x7d/0xa0 [ 2130.355738] ? report_bug+0xfc/0x1d0 [ 2130.359316] ? prb_read_valid+0x1b/0x20 [ 2130.363157] ? handle_bug+0x44/0x70 [ 2130.366650] ? exc_invalid_op+0x17/0x60 [ 2130.370489] ? asm_exc_invalid_op+0x1a/0x20 [ 2130.374679] ? __list_add_valid_or_report+0x7d/0xa0 [ 2130.379555] ? __list_add_valid_or_report+0x7d/0xa0 [ 2130.384433] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2130.389324] ceph_open+0x16b/0x440 [ceph] [ 2130.393345] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2130.398490] do_dentry_open+0x172/0x530 [ 2130.402333] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2130.406967] lookup_open+0x37b/0x630 [ 2130.410552] ? verify_cpu+0x20/0x100 [ 2130.414127] ? verify_cpu+0x20/0x100 [ 2130.417708] path_openat+0x25a/0x9d0 [ 2130.421290] do_filp_open+0xb8/0x120 [ 2130.424870] ? lock_release+0xc5/0x260 [ 2130.428623] ? _raw_spin_unlock+0x23/0x40 [ 2130.432636] ? alloc_fd+0x13f/0x200 [ 2130.436134] do_sys_openat2+0x249/0x290 [ 2130.439973] do_sys_open+0x4f/0x80 [ 2130.443378] do_syscall_64+0x38/0x80 [ 2130.446955] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2130.452009] RIP: 0033:0x7f48f0eed2f2 [ 2130.455590] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2130.474333] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2130.481901] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2130.489035] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2130.496166] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2130.503299] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2130.510432] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bb [ 2130.517573] [ 2130.519757] irq event stamp: 12785151 [ 2130.523423] hardirqs last enabled at (12785161): [] console_unlock+0x106/0x120 [ 2130.532290] hardirqs last disabled at (12785170): [] console_unlock+0xeb/0x120 [ 2130.541069] softirqs last enabled at (12785098): [] __do_softirq+0x2be/0x371 [ 2130.549762] softirqs last disabled at (12785093): [] irq_exit_rcu+0xb0/0xd0 [ 2130.558281] ---[ end trace 0000000000000000 ]--- [ 2130.562906] ------------[ cut here ]------------ [ 2130.562914] ceph: [cde022fc-3e6c-431e-bc4a-204d5c5d32a6 4445] send_renew_caps: mds0 caps stale [ 2130.567522] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2130.567546] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2130.596838] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2130.652313] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2130.661261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2130.668740] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2130.674226] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2130.692974] RSP: 0018:ffffc90001fef8c0 EFLAGS: 00010282 [ 2130.698199] RAX: 0000000000000000 RBX: ffff8881e4f8f560 RCX: 0000000000000027 [ 2130.705330] RDX: 0000000000000027 RSI: ffff88885fc9d880 RDI: ffff88885fc9d888 [ 2130.712455] RBP: ffff88817f867000 R08: 0000000000000001 R09: 0000000000000001 [ 2130.719590] R10: ffffc90001fef6b8 R11: ffffffff83115580 R12: ffff8881e4f8fcb0 [ 2130.726720] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2130.733854] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2130.741941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2130.747687] CR2: 00007f9bd346f000 CR3: 0000000108c72006 CR4: 00000000003706e0 [ 2130.754818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2130.761943] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2130.769075] Call Trace: [ 2130.771521] [ 2130.773633] ? __warn+0x85/0x180 [ 2130.776868] ? __list_add_valid_or_report+0x7d/0xa0 [ 2130.781749] ? report_bug+0xfc/0x1d0 [ 2130.785330] ? mark_held_locks+0x48/0x70 [ 2130.789255] ? handle_bug+0x44/0x70 [ 2130.792745] ? exc_invalid_op+0x17/0x60 [ 2130.796585] ? asm_exc_invalid_op+0x1a/0x20 [ 2130.800777] ? __list_add_valid_or_report+0x7d/0xa0 [ 2130.805656] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2130.810560] ceph_check_caps+0x999/0xae0 [ceph] [ 2130.815115] ? handle_bug+0x44/0x70 [ 2130.818613] ? exc_invalid_op+0x17/0x60 [ 2130.822446] ? asm_exc_invalid_op+0x1a/0x20 [ 2130.826640] ? find_held_lock+0x2d/0x90 [ 2130.830480] ? ceph_open+0x175/0x440 [ceph] [ 2130.834675] ? lock_release+0xc5/0x260 [ 2130.838432] ceph_open+0x3cc/0x440 [ceph] [ 2130.842452] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2130.847600] do_dentry_open+0x172/0x530 [ 2130.851439] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2130.856074] lookup_open+0x37b/0x630 [ 2130.859657] ? verify_cpu+0x20/0x100 [ 2130.863235] ? verify_cpu+0x20/0x100 [ 2130.866815] path_openat+0x25a/0x9d0 [ 2130.870396] do_filp_open+0xb8/0x120 [ 2130.873974] ? lock_release+0xc5/0x260 [ 2130.877725] ? _raw_spin_unlock+0x23/0x40 [ 2130.881737] ? alloc_fd+0x13f/0x200 [ 2130.885231] do_sys_openat2+0x249/0x290 [ 2130.889071] do_sys_open+0x4f/0x80 [ 2130.892476] do_syscall_64+0x38/0x80 [ 2130.896055] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2130.901107] RIP: 0033:0x7f48f0eed2f2 [ 2130.904688] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2130.923439] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2130.931017] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2130.938153] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2130.945286] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000000 [ 2130.952423] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2130.959560] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bb [ 2130.966701] [ 2130.968890] irq event stamp: 12786293 [ 2130.972555] hardirqs last enabled at (12786303): [] console_unlock+0x106/0x120 [ 2130.981431] hardirqs last disabled at (12786318): [] console_unlock+0xeb/0x120 [ 2130.990225] softirqs last enabled at (12786332): [] __do_softirq+0x2be/0x371 [ 2130.998927] softirqs last disabled at (12786327): [] irq_exit_rcu+0xb0/0xd0 [ 2131.007455] ---[ end trace 0000000000000000 ]--- [ 2131.012082] ------------[ cut here ]------------ [ 2131.016734] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2131.028682] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2131.037399] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2131.092980] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2131.102112] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2131.109625] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2131.115156] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2131.120669] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2131.139421] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2131.144669] RAX: 0000000000000000 RBX: ffff8881c2643ab0 RCX: 0000000000000000 [ 2131.151808] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2131.158954] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000008e4 [ 2131.166104] R10: 0000000000081dd8 R11: ffffc9000030b828 R12: ffff8881c2644200 [ 2131.173255] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2131.180408] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2131.188512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2131.194255] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2131.201406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2131.208559] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2131.215691] Call Trace: [ 2131.218162] [ 2131.220285] ? __warn+0x85/0x180 [ 2131.223543] ? __list_add_valid_or_report+0x7d/0xa0 [ 2131.228443] ? report_bug+0xfc/0x1d0 [ 2131.232025] ? prb_read_valid+0x1b/0x20 [ 2131.235886] ? handle_bug+0x44/0x70 [ 2131.239405] ? exc_invalid_op+0x17/0x60 [ 2131.243269] ? asm_exc_invalid_op+0x1a/0x20 [ 2131.247492] ? __list_add_valid_or_report+0x7d/0xa0 [ 2131.252396] ? __list_add_valid_or_report+0x7d/0xa0 [ 2131.257281] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2131.262227] ceph_add_cap+0x20b/0x620 [ceph] [ 2131.266551] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2131.271310] ceph_fill_trace+0x143/0xf30 [ceph] [ 2131.275857] ? __mutex_lock+0x97/0xce0 [ 2131.279615] ? __mutex_lock+0xaec/0xce0 [ 2131.283482] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2131.288048] ? lock_downgrade+0xfd/0x250 [ 2131.292003] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2131.296412] ceph_con_process_message+0x74/0x150 [libceph] [ 2131.301918] ? calc_signature+0xdb/0x100 [libceph] [ 2131.306741] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2131.312098] ? process_scheduled_works+0x22a/0x510 [ 2131.316900] ? process_scheduled_works+0x22a/0x510 [ 2131.321692] ceph_con_workfn+0x344/0x750 [libceph] [ 2131.326525] ? process_scheduled_works+0x22a/0x510 [ 2131.331322] process_scheduled_works+0x2a4/0x510 [ 2131.335995] worker_thread+0x188/0x340 [ 2131.339759] ? rescuer_thread+0x310/0x310 [ 2131.343776] kthread+0xf5/0x130 [ 2131.346949] ? kthread_complete_and_exit+0x20/0x20 [ 2131.351767] ret_from_fork+0x31/0x40 [ 2131.355357] ? kthread_complete_and_exit+0x20/0x20 [ 2131.360184] ret_from_fork_asm+0x11/0x20 [ 2131.364151] [ 2131.366345] irq event stamp: 9536885 [ 2131.369941] hardirqs last enabled at (9536895): [] console_unlock+0x106/0x120 [ 2131.378721] hardirqs last disabled at (9536904): [] console_unlock+0xeb/0x120 [ 2131.387413] softirqs last enabled at (9536842): [] __do_softirq+0x2be/0x371 [ 2131.396038] softirqs last disabled at (9536837): [] irq_exit_rcu+0xb0/0xd0 [ 2131.404478] ---[ end trace 0000000000000000 ]--- [ 2131.409137] ------------[ cut here ]------------ [ 2131.413832] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2131.425481] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2131.434257] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2131.489747] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2131.498875] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2131.506354] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2131.511850] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2131.517335] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2131.536082] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2131.541306] RAX: 0000000000000000 RBX: ffff8881c2643ab0 RCX: 0000000000000000 [ 2131.548431] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2131.555566] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000922 [ 2131.562697] R10: 0000000000082ea0 R11: ffffc9000030b8d8 R12: ffff8881c2644200 [ 2131.569830] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2131.576963] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2131.585049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2131.590794] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2131.597919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2131.605051] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2131.612185] Call Trace: [ 2131.614637] [ 2131.616744] ? __warn+0x85/0x180 [ 2131.619976] ? __list_add_valid_or_report+0x7d/0xa0 [ 2131.624858] ? report_bug+0xfc/0x1d0 [ 2131.628435] ? prb_read_valid+0x1b/0x20 [ 2131.632277] ? handle_bug+0x44/0x70 [ 2131.635768] ? exc_invalid_op+0x17/0x60 [ 2131.639608] ? asm_exc_invalid_op+0x1a/0x20 [ 2131.643798] ? __list_add_valid_or_report+0x7d/0xa0 [ 2131.648673] ? __list_add_valid_or_report+0x7d/0xa0 [ 2131.653552] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2131.658443] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2131.663077] ceph_fill_trace+0x143/0xf30 [ceph] [ 2131.667620] ? __mutex_lock+0x97/0xce0 [ 2131.671370] ? __mutex_lock+0xaec/0xce0 [ 2131.675211] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2131.679751] ? lock_downgrade+0xfd/0x250 [ 2131.683682] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2131.688063] ceph_con_process_message+0x74/0x150 [libceph] [ 2131.693558] ? calc_signature+0xdb/0x100 [libceph] [ 2131.698362] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2131.703685] ? process_scheduled_works+0x22a/0x510 [ 2131.708477] ? process_scheduled_works+0x22a/0x510 [ 2131.713267] ceph_con_workfn+0x344/0x750 [libceph] [ 2131.718070] ? process_scheduled_works+0x22a/0x510 [ 2131.722860] process_scheduled_works+0x2a4/0x510 [ 2131.727486] worker_thread+0x188/0x340 [ 2131.731241] ? rescuer_thread+0x310/0x310 [ 2131.735253] kthread+0xf5/0x130 [ 2131.738401] ? kthread_complete_and_exit+0x20/0x20 [ 2131.743195] ret_from_fork+0x31/0x40 [ 2131.746773] ? kthread_complete_and_exit+0x20/0x20 [ 2131.751565] ret_from_fork_asm+0x11/0x20 [ 2131.755497] [ 2131.757683] irq event stamp: 9538237 [ 2131.761263] hardirqs last enabled at (9538247): [] console_unlock+0x106/0x120 [ 2131.770043] hardirqs last disabled at (9538256): [] console_unlock+0xeb/0x120 [ 2131.778735] softirqs last enabled at (9538204): [] __do_softirq+0x2be/0x371 [ 2131.787345] softirqs last disabled at (9538199): [] irq_exit_rcu+0xb0/0xd0 [ 2131.795775] ---[ end trace 0000000000000000 ]--- [ 2131.800422] ------------[ cut here ]------------ [ 2131.805059] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2131.816714] WARNING: CPU: 7 PID: 89 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2131.825405] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2131.880875] CPU: 7 PID: 89 Comm: kworker/7:3 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2131.890000] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2131.897479] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2131.902973] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2131.908460] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2131.927205] RSP: 0018:ffffc90000e4f958 EFLAGS: 00010286 [ 2131.932433] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2131.939570] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2131.946699] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000095f [ 2131.953831] R10: 00000000000840f8 R11: ffffc90000e4f7e8 R12: ffff8881e4d0b838 [ 2131.960963] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2131.968096] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2131.976182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2131.981927] CR2: 00007f9bd3913000 CR3: 000000000243e006 CR4: 00000000003706e0 [ 2131.989052] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2131.996185] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2132.003309] Call Trace: [ 2132.005753] [ 2132.007863] ? __warn+0x85/0x180 [ 2132.011097] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.015979] ? report_bug+0xfc/0x1d0 [ 2132.019562] ? prb_read_valid+0x1b/0x20 [ 2132.023402] ? handle_bug+0x44/0x70 [ 2132.026891] ? exc_invalid_op+0x17/0x60 [ 2132.030733] ? asm_exc_invalid_op+0x1a/0x20 [ 2132.034921] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.039797] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.044676] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2132.049566] ceph_check_caps+0x999/0xae0 [ceph] [ 2132.054114] ? __lock_acquire+0xbfa/0x1e20 [ 2132.058215] ? __lock_acquire+0xb70/0x1e20 [ 2132.062314] ? lock_downgrade+0x250/0x250 [ 2132.066327] ? lock_acquire+0xcb/0x2b0 [ 2132.070080] ? ceph_put_wrbuffer_cap_refs+0x3e/0x3e0 [ceph] [ 2132.075662] ? find_held_lock+0x2d/0x90 [ 2132.079501] ? ceph_put_wrbuffer_cap_refs+0x1c9/0x3e0 [ceph] [ 2132.085169] ? lock_release+0xc5/0x260 [ 2132.088925] ceph_put_wrbuffer_cap_refs+0x32b/0x3e0 [ceph] [ 2132.094418] writepages_finish+0x302/0x610 [ceph] [ 2132.099137] __complete_request+0x1c/0x80 [libceph] [ 2132.104029] osd_dispatch+0x1bf/0xbd0 [libceph] [ 2132.108579] ? lock_release+0xc5/0x260 [ 2132.112332] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2132.117040] ceph_con_process_message+0x74/0x150 [libceph] [ 2132.122532] ? calc_signature+0xdb/0x100 [libceph] [ 2132.127343] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2132.132667] ? process_scheduled_works+0x22a/0x510 [ 2132.137460] ? process_scheduled_works+0x22a/0x510 [ 2132.142248] ceph_con_workfn+0x344/0x750 [libceph] [ 2132.147051] ? process_scheduled_works+0x22a/0x510 [ 2132.151841] process_scheduled_works+0x2a4/0x510 [ 2132.156467] worker_thread+0x188/0x340 [ 2132.160214] ? rescuer_thread+0x310/0x310 [ 2132.164226] kthread+0xf5/0x130 [ 2132.167373] ? kthread_complete_and_exit+0x20/0x20 [ 2132.172166] ret_from_fork+0x31/0x40 [ 2132.175746] ? kthread_complete_and_exit+0x20/0x20 [ 2132.180538] ret_from_fork_asm+0x11/0x20 [ 2132.184470] [ 2132.186657] irq event stamp: 68175265 [ 2132.190321] hardirqs last enabled at (68175275): [] console_unlock+0x106/0x120 [ 2132.199191] hardirqs last disabled at (68175284): [] console_unlock+0xeb/0x120 [ 2132.207970] softirqs last enabled at (68175298): [] __do_softirq+0x2be/0x371 [ 2132.216664] softirqs last disabled at (68175293): [] irq_exit_rcu+0xb0/0xd0 [ 2132.225181] ---[ end trace 0000000000000000 ]--- [ 2132.229802] ------------[ cut here ]------------ [ 2132.234420] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2132.246071] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2132.255031] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2132.310514] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2132.319473] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2132.326956] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2132.332444] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2132.351188] RSP: 0018:ffffc90001fd7aa8 EFLAGS: 00010282 [ 2132.356415] RAX: 0000000000000000 RBX: ffff8881c2643ab0 RCX: 0000000000000000 [ 2132.363550] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2132.370682] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000009a4 [ 2132.377815] R10: 0000000000085488 R11: ffffc90001fd7938 R12: ffff8881c2644200 [ 2132.384949] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2132.392080] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2132.400166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2132.405912] CR2: 00007f9bd346f000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2132.413045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2132.420177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2132.427311] Call Trace: [ 2132.429763] [ 2132.431871] ? __warn+0x85/0x180 [ 2132.435110] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.439994] ? report_bug+0xfc/0x1d0 [ 2132.443579] ? prb_read_valid+0x1b/0x20 [ 2132.447423] ? handle_bug+0x44/0x70 [ 2132.450920] ? exc_invalid_op+0x17/0x60 [ 2132.454761] ? asm_exc_invalid_op+0x1a/0x20 [ 2132.458963] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.463844] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.468730] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2132.473633] ceph_open+0x16b/0x440 [ceph] [ 2132.477666] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2132.482821] do_dentry_open+0x172/0x530 [ 2132.486669] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2132.491323] lookup_open+0x37b/0x630 [ 2132.494908] ? verify_cpu+0x20/0x100 [ 2132.498498] ? verify_cpu+0x20/0x100 [ 2132.502082] path_openat+0x25a/0x9d0 [ 2132.505680] do_filp_open+0xb8/0x120 [ 2132.509272] ? lock_release+0xc5/0x260 [ 2132.513032] ? _raw_spin_unlock+0x23/0x40 [ 2132.517046] ? alloc_fd+0x13f/0x200 [ 2132.520546] do_sys_openat2+0x249/0x290 [ 2132.524391] do_sys_open+0x4f/0x80 [ 2132.527807] do_syscall_64+0x38/0x80 [ 2132.531392] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2132.536445] RIP: 0033:0x7f48f0eed2f2 [ 2132.540024] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2132.558768] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2132.566335] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2132.573471] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2132.580609] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000004 [ 2132.587745] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2132.594882] R13: 00007f48f183d8a0 R14: 00000000020767c0 R15: 0000000000002a13 [ 2132.602029] [ 2132.604217] irq event stamp: 12914413 [ 2132.607885] hardirqs last enabled at (12914423): [] console_unlock+0x106/0x120 [ 2132.616759] hardirqs last disabled at (12914432): [] console_unlock+0xeb/0x120 [ 2132.625537] softirqs last enabled at (12914360): [] __do_softirq+0x2be/0x371 [ 2132.634230] softirqs last disabled at (12914355): [] irq_exit_rcu+0xb0/0xd0 [ 2132.642754] ---[ end trace 0000000000000000 ]--- [ 2132.647369] ------------[ cut here ]------------ [ 2132.652102] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2132.663747] WARNING: CPU: 7 PID: 78658 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2132.672701] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2132.728169] CPU: 7 PID: 78658 Comm: kworker/7:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2132.737553] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2132.745027] Workqueue: events delayed_work [ceph] [ 2132.749748] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2132.755235] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2132.773981] RSP: 0018:ffffc90001f77b90 EFLAGS: 00010282 [ 2132.779206] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2132.786331] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2132.793465] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000009e2 [ 2132.800598] R10: 0000000000086620 R11: ffffc90001f77a20 R12: ffff8881e4d0b838 [ 2132.807730] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2132.814863] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2132.822949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2132.828694] CR2: 00007f9bd3913000 CR3: 000000000243e006 CR4: 00000000003706e0 [ 2132.835829] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2132.842961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2132.850094] Call Trace: [ 2132.852546] [ 2132.854653] ? __warn+0x85/0x180 [ 2132.857886] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.862764] ? report_bug+0xfc/0x1d0 [ 2132.866343] ? prb_read_valid+0x1b/0x20 [ 2132.870185] ? handle_bug+0x44/0x70 [ 2132.873677] ? exc_invalid_op+0x17/0x60 [ 2132.877516] ? asm_exc_invalid_op+0x1a/0x20 [ 2132.881707] ? __list_add_valid_or_report+0x7d/0xa0 [ 2132.886582] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2132.891471] ceph_check_caps+0x999/0xae0 [ceph] [ 2132.896013] ? __lock_acquire+0xbfa/0x1e20 [ 2132.900123] ? lock_release+0xc5/0x260 [ 2132.903875] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2132.909117] delayed_work+0x1d8/0x280 [ceph] [ 2132.913399] ? process_scheduled_works+0x22a/0x510 [ 2132.918189] process_scheduled_works+0x2a4/0x510 [ 2132.922815] worker_thread+0x188/0x340 [ 2132.926569] ? rescuer_thread+0x310/0x310 [ 2132.930582] kthread+0xf5/0x130 [ 2132.933728] ? kthread_complete_and_exit+0x20/0x20 [ 2132.938522] ret_from_fork+0x31/0x40 [ 2132.942099] ? kthread_complete_and_exit+0x20/0x20 [ 2132.946894] ret_from_fork_asm+0x11/0x20 [ 2132.950825] [ 2132.953010] irq event stamp: 5363161 [ 2132.956582] hardirqs last enabled at (5363171): [] console_unlock+0x106/0x120 [ 2132.965365] hardirqs last disabled at (5363188): [] console_unlock+0xeb/0x120 [ 2132.974054] softirqs last enabled at (5363186): [] __do_softirq+0x2be/0x371 [ 2132.982660] softirqs last disabled at (5363181): [] irq_exit_rcu+0xb0/0xd0 [ 2132.991096] ---[ end trace 0000000000000000 ]--- [ 2132.995714] ------------[ cut here ]------------ [ 2133.000334] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2133.011981] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2133.020940] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2133.076418] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2133.085377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2133.092862] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2133.098350] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2133.117103] RSP: 0018:ffffc90001fd78c0 EFLAGS: 00010282 [ 2133.122332] RAX: 0000000000000000 RBX: ffff8881c2643ab0 RCX: 0000000000000000 [ 2133.129470] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2133.136605] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000a16 [ 2133.143735] R10: 00000000000875d8 R11: ffffc90001fd7750 R12: ffff8881c2644200 [ 2133.150876] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2133.158009] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2133.166097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2133.171841] CR2: 00007f9bd346f000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2133.178976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2133.186107] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2133.193243] Call Trace: [ 2133.195693] [ 2133.197801] ? __warn+0x85/0x180 [ 2133.201034] ? __list_add_valid_or_report+0x7d/0xa0 [ 2133.205915] ? report_bug+0xfc/0x1d0 [ 2133.209500] ? prb_read_valid+0x1b/0x20 [ 2133.213344] ? handle_bug+0x44/0x70 [ 2133.216845] ? exc_invalid_op+0x17/0x60 [ 2133.220683] ? asm_exc_invalid_op+0x1a/0x20 [ 2133.224884] ? __list_add_valid_or_report+0x7d/0xa0 [ 2133.229767] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2133.234664] ceph_check_caps+0x999/0xae0 [ceph] [ 2133.239227] ? handle_bug+0x44/0x70 [ 2133.242720] ? exc_invalid_op+0x17/0x60 [ 2133.246561] ? asm_exc_invalid_op+0x1a/0x20 [ 2133.250752] ? find_held_lock+0x2d/0x90 [ 2133.254597] ? ceph_open+0x175/0x440 [ceph] [ 2133.258799] ? lock_release+0xc5/0x260 [ 2133.262558] ceph_open+0x3cc/0x440 [ceph] [ 2133.266585] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2133.271738] do_dentry_open+0x172/0x530 [ 2133.275577] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2133.280227] lookup_open+0x37b/0x630 [ 2133.283815] ? verify_cpu+0x20/0x100 [ 2133.287405] ? verify_cpu+0x20/0x100 [ 2133.290989] path_openat+0x25a/0x9d0 [ 2133.294574] do_filp_open+0xb8/0x120 [ 2133.298160] ? lock_release+0xc5/0x260 [ 2133.301920] ? _raw_spin_unlock+0x23/0x40 [ 2133.305936] ? alloc_fd+0x13f/0x200 [ 2133.309435] do_sys_openat2+0x249/0x290 [ 2133.313279] do_sys_open+0x4f/0x80 [ 2133.316687] do_syscall_64+0x38/0x80 [ 2133.320270] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2133.325324] RIP: 0033:0x7f48f0eed2f2 [ 2133.328901] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2133.347652] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2133.355223] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2133.362357] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2133.369488] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000004 [ 2133.376624] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2133.383755] R13: 00007f48f183d8a0 R14: 00000000020767c0 R15: 0000000000002a13 [ 2133.390902] [ 2133.393098] irq event stamp: 12916271 [ 2133.396765] hardirqs last enabled at (12916281): [] console_unlock+0x106/0x120 [ 2133.405638] hardirqs last disabled at (12916290): [] console_unlock+0xeb/0x120 [ 2133.414422] softirqs last enabled at (12916304): [] __do_softirq+0x2be/0x371 [ 2133.423118] softirqs last disabled at (12916299): [] irq_exit_rcu+0xb0/0xd0 [ 2133.431640] ---[ end trace 0000000000000000 ]--- [ 2133.504500] ------------[ cut here ]------------ [ 2133.509160] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2133.520940] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2133.529635] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2133.585172] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2133.594326] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2133.601858] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2133.607369] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2133.612870] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2133.631650] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2133.636877] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2133.644039] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2133.651177] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000a5a [ 2133.658331] R10: 00000000000888f0 R11: ffffc9000030b828 R12: ffff8881d1295590 [ 2133.665519] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2133.672654] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2133.680758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2133.686537] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2133.693672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2133.700821] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2133.707958] Call Trace: [ 2133.710435] [ 2133.712569] ? __warn+0x85/0x180 [ 2133.715836] ? __list_add_valid_or_report+0x7d/0xa0 [ 2133.720723] ? report_bug+0xfc/0x1d0 [ 2133.724297] ? prb_read_valid+0x1b/0x20 [ 2133.728157] ? handle_bug+0x44/0x70 [ 2133.731666] ? exc_invalid_op+0x17/0x60 [ 2133.735523] ? asm_exc_invalid_op+0x1a/0x20 [ 2133.739733] ? __list_add_valid_or_report+0x7d/0xa0 [ 2133.744615] ? __list_add_valid_or_report+0x7d/0xa0 [ 2133.749512] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2133.754428] ceph_add_cap+0x20b/0x620 [ceph] [ 2133.758739] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2133.763539] ? lock_acquire+0xcb/0x2b0 [ 2133.767319] ceph_fill_trace+0x143/0xf30 [ceph] [ 2133.771863] ? __mutex_lock+0x97/0xce0 [ 2133.775615] ? __mutex_lock+0xaec/0xce0 [ 2133.779474] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2133.784035] ? lock_downgrade+0xfd/0x250 [ 2133.788010] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2133.792404] ceph_con_process_message+0x74/0x150 [libceph] [ 2133.797899] ? calc_signature+0xdb/0x100 [libceph] [ 2133.802714] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2133.808059] ? process_scheduled_works+0x22a/0x510 [ 2133.812850] ? process_scheduled_works+0x22a/0x510 [ 2133.817640] ceph_con_workfn+0x344/0x750 [libceph] [ 2133.822462] ? process_scheduled_works+0x22a/0x510 [ 2133.827269] process_scheduled_works+0x2a4/0x510 [ 2133.831912] worker_thread+0x188/0x340 [ 2133.835661] ? rescuer_thread+0x310/0x310 [ 2133.839672] kthread+0xf5/0x130 [ 2133.842844] ? kthread_complete_and_exit+0x20/0x20 [ 2133.847638] ret_from_fork+0x31/0x40 [ 2133.851235] ? kthread_complete_and_exit+0x20/0x20 [ 2133.856043] ret_from_fork_asm+0x11/0x20 [ 2133.860013] [ 2133.862205] irq event stamp: 9798903 [ 2133.865803] hardirqs last enabled at (9798913): [] console_unlock+0x106/0x120 [ 2133.874580] hardirqs last disabled at (9798922): [] console_unlock+0xeb/0x120 [ 2133.883293] softirqs last enabled at (9798844): [] __do_softirq+0x2be/0x371 [ 2133.891916] softirqs last disabled at (9798839): [] irq_exit_rcu+0xb0/0xd0 [ 2133.900348] ---[ end trace 0000000000000000 ]--- [ 2133.905046] ------------[ cut here ]------------ [ 2133.909690] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2133.921363] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2133.930060] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2133.985554] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2133.994752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2134.002231] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2134.007748] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2134.013232] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2134.031995] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2134.037241] RAX: 0000000000000000 RBX: ffff8881e4ee4e40 RCX: 0000000000000000 [ 2134.044390] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2134.051531] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000a99 [ 2134.058673] R10: 0000000000089f18 R11: ffffc9000030b828 R12: ffff8881e4ee5590 [ 2134.065823] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2134.072956] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2134.081043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2134.086805] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2134.093940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2134.101074] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2134.108220] Call Trace: [ 2134.110684] [ 2134.112806] ? __warn+0x85/0x180 [ 2134.116038] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.120938] ? report_bug+0xfc/0x1d0 [ 2134.124515] ? prb_read_valid+0x1b/0x20 [ 2134.128374] ? handle_bug+0x44/0x70 [ 2134.131884] ? exc_invalid_op+0x17/0x60 [ 2134.135723] ? asm_exc_invalid_op+0x1a/0x20 [ 2134.139915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.144797] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.149675] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2134.154583] ceph_add_cap+0x20b/0x620 [ceph] [ 2134.158887] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2134.163626] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2134.168270] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2134.173615] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2134.178077] ceph_con_process_message+0x74/0x150 [libceph] [ 2134.183572] ? calc_signature+0xdb/0x100 [libceph] [ 2134.188418] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2134.193743] ? process_scheduled_works+0x22a/0x510 [ 2134.198549] ? process_scheduled_works+0x22a/0x510 [ 2134.203340] ceph_con_workfn+0x344/0x750 [libceph] [ 2134.208160] ? process_scheduled_works+0x22a/0x510 [ 2134.212968] process_scheduled_works+0x2a4/0x510 [ 2134.217597] worker_thread+0x188/0x340 [ 2134.221350] ? rescuer_thread+0x310/0x310 [ 2134.225379] kthread+0xf5/0x130 [ 2134.228544] ? kthread_complete_and_exit+0x20/0x20 [ 2134.233354] ret_from_fork+0x31/0x40 [ 2134.236950] ? kthread_complete_and_exit+0x20/0x20 [ 2134.241744] ret_from_fork_asm+0x11/0x20 [ 2134.245697] [ 2134.247888] irq event stamp: 9800013 [ 2134.251465] hardirqs last enabled at (9800023): [] console_unlock+0x106/0x120 [ 2134.260264] hardirqs last disabled at (9800032): [] console_unlock+0xeb/0x120 [ 2134.268987] softirqs last enabled at (9800056): [] __do_softirq+0x2be/0x371 [ 2134.277589] softirqs last disabled at (9800043): [] irq_exit_rcu+0xb0/0xd0 [ 2134.286040] ---[ end trace 0000000000000000 ]--- [ 2134.290731] ------------[ cut here ]------------ [ 2134.295367] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2134.307036] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2134.315733] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2134.371210] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2134.380350] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2134.387839] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2134.393333] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2134.398837] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2134.417584] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2134.422828] RAX: 0000000000000000 RBX: ffff8881d132c478 RCX: 0000000000000000 [ 2134.429959] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2134.437094] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000ad4 [ 2134.444243] R10: 000000000008af90 R11: ffffc9000030b828 R12: ffff8881d132cbc8 [ 2134.451433] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2134.458563] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2134.466664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2134.472428] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2134.479577] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2134.486724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2134.493852] Call Trace: [ 2134.496305] [ 2134.498430] ? __warn+0x85/0x180 [ 2134.501677] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.506576] ? report_bug+0xfc/0x1d0 [ 2134.510171] ? prb_read_valid+0x1b/0x20 [ 2134.514056] ? handle_bug+0x44/0x70 [ 2134.517548] ? exc_invalid_op+0x17/0x60 [ 2134.521403] ? asm_exc_invalid_op+0x1a/0x20 [ 2134.525629] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.530504] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.535401] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2134.540308] ceph_add_cap+0x20b/0x620 [ceph] [ 2134.544628] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2134.549366] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2134.554714] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2134.559197] ceph_con_process_message+0x74/0x150 [libceph] [ 2134.564714] ? calc_signature+0xdb/0x100 [libceph] [ 2134.569542] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2134.574867] ? process_scheduled_works+0x22a/0x510 [ 2134.579666] ? process_scheduled_works+0x22a/0x510 [ 2134.584471] ceph_con_workfn+0x344/0x750 [libceph] [ 2134.589293] ? process_scheduled_works+0x22a/0x510 [ 2134.594101] process_scheduled_works+0x2a4/0x510 [ 2134.598743] worker_thread+0x188/0x340 [ 2134.602492] ? rescuer_thread+0x310/0x310 [ 2134.606503] kthread+0xf5/0x130 [ 2134.609668] ? kthread_complete_and_exit+0x20/0x20 [ 2134.614477] ret_from_fork+0x31/0x40 [ 2134.618075] ? kthread_complete_and_exit+0x20/0x20 [ 2134.622886] ret_from_fork_asm+0x11/0x20 [ 2134.626819] [ 2134.629024] irq event stamp: 9801123 [ 2134.632616] hardirqs last enabled at (9801133): [] console_unlock+0x106/0x120 [ 2134.641413] hardirqs last disabled at (9801142): [] console_unlock+0xeb/0x120 [ 2134.650124] softirqs last enabled at (9801110): [] __do_softirq+0x2be/0x371 [ 2134.658748] softirqs last disabled at (9801105): [] irq_exit_rcu+0xb0/0xd0 [ 2134.667179] ---[ end trace 0000000000000000 ]--- [ 2134.671815] ------------[ cut here ]------------ [ 2134.676429] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e46920. (prev=ffff8881a1d36920). [ 2134.688107] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2134.696796] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2134.752272] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2134.761414] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2134.768902] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2134.774397] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2134.779900] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2134.798651] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2134.803891] RAX: 0000000000000000 RBX: ffff8881d13cbab0 RCX: 0000000000000000 [ 2134.811025] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2134.818173] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000b0e [ 2134.825322] R10: 000000000008bf30 R11: ffffc9000030b828 R12: ffff8881d13cc200 [ 2134.832465] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2134.839615] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2134.847727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2134.853474] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2134.860622] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2134.867765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2134.874897] Call Trace: [ 2134.877341] [ 2134.879458] ? __warn+0x85/0x180 [ 2134.882732] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.887614] ? report_bug+0xfc/0x1d0 [ 2134.891210] ? prb_read_valid+0x1b/0x20 [ 2134.895067] ? handle_bug+0x44/0x70 [ 2134.898576] ? exc_invalid_op+0x17/0x60 [ 2134.902433] ? asm_exc_invalid_op+0x1a/0x20 [ 2134.906658] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.911532] ? __list_add_valid_or_report+0x7d/0xa0 [ 2134.916429] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2134.921336] ceph_add_cap+0x20b/0x620 [ceph] [ 2134.925658] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2134.930401] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2134.935739] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2134.940220] ceph_con_process_message+0x74/0x150 [libceph] [ 2134.945738] ? calc_signature+0xdb/0x100 [libceph] [ 2134.950562] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2134.955887] ? process_scheduled_works+0x22a/0x510 [ 2134.960705] ? process_scheduled_works+0x22a/0x510 [ 2134.965491] ceph_con_workfn+0x344/0x750 [libceph] [ 2134.970312] ? process_scheduled_works+0x22a/0x510 [ 2134.975121] process_scheduled_works+0x2a4/0x510 [ 2134.979764] worker_thread+0x188/0x340 [ 2134.983513] ? rescuer_thread+0x310/0x310 [ 2134.987541] kthread+0xf5/0x130 [ 2134.990687] ? kthread_complete_and_exit+0x20/0x20 [ 2134.995497] ret_from_fork+0x31/0x40 [ 2134.999093] ? kthread_complete_and_exit+0x20/0x20 [ 2135.003904] ret_from_fork_asm+0x11/0x20 [ 2135.007836] [ 2135.010023] irq event stamp: 9802159 [ 2135.013618] hardirqs last enabled at (9802169): [] console_unlock+0x106/0x120 [ 2135.022414] hardirqs last disabled at (9802178): [] console_unlock+0xeb/0x120 [ 2135.031125] softirqs last enabled at (9802022): [] __do_softirq+0x2be/0x371 [ 2135.039750] softirqs last disabled at (9802017): [] irq_exit_rcu+0xb0/0xd0 [ 2135.048181] ---[ end trace 0000000000000000 ]--- [ 2135.052957] ------------[ cut here ]------------ [ 2135.057570] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e47cb0. (prev=ffff8881a1d36920). [ 2135.069251] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2135.077945] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2135.133421] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2135.142566] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2135.150061] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2135.155573] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2135.161077] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2135.179842] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2135.185066] RAX: 0000000000000000 RBX: ffff8881d11b4e40 RCX: 0000000000000000 [ 2135.192198] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2135.199339] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000b48 [ 2135.206481] R10: 000000000008cf70 R11: ffffc9000030b828 R12: ffff8881d11b5590 [ 2135.213625] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2135.220776] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2135.228859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2135.234605] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2135.241755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2135.248889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2135.256034] Call Trace: [ 2135.258500] [ 2135.260624] ? __warn+0x85/0x180 [ 2135.263876] ? __list_add_valid_or_report+0x7d/0xa0 [ 2135.268753] ? report_bug+0xfc/0x1d0 [ 2135.272333] ? prb_read_valid+0x1b/0x20 [ 2135.276191] ? handle_bug+0x44/0x70 [ 2135.279699] ? exc_invalid_op+0x17/0x60 [ 2135.283539] ? asm_exc_invalid_op+0x1a/0x20 [ 2135.287748] ? __list_add_valid_or_report+0x7d/0xa0 [ 2135.292623] ? __list_add_valid_or_report+0x7d/0xa0 [ 2135.297517] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2135.302426] ceph_add_cap+0x20b/0x620 [ceph] [ 2135.306745] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2135.311478] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2135.316122] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2135.321464] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2135.325929] ceph_con_process_message+0x74/0x150 [libceph] [ 2135.331423] ? calc_signature+0xdb/0x100 [libceph] [ 2135.336258] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2135.341620] ? process_scheduled_works+0x22a/0x510 [ 2135.346412] ? process_scheduled_works+0x22a/0x510 [ 2135.351217] ceph_con_workfn+0x344/0x750 [libceph] [ 2135.356041] ? process_scheduled_works+0x22a/0x510 [ 2135.360847] process_scheduled_works+0x2a4/0x510 [ 2135.365489] worker_thread+0x188/0x340 [ 2135.369237] ? rescuer_thread+0x310/0x310 [ 2135.373249] kthread+0xf5/0x130 [ 2135.376394] ? kthread_complete_and_exit+0x20/0x20 [ 2135.381205] ret_from_fork+0x31/0x40 [ 2135.384803] ? kthread_complete_and_exit+0x20/0x20 [ 2135.389596] ret_from_fork_asm+0x11/0x20 [ 2135.393563] [ 2135.395750] irq event stamp: 9803179 [ 2135.399329] hardirqs last enabled at (9803195): [] console_unlock+0x106/0x120 [ 2135.408180] hardirqs last disabled at (9803274): [] console_unlock+0xeb/0x120 [ 2135.416870] softirqs last enabled at (9803286): [] __do_softirq+0x2be/0x371 [ 2135.425474] softirqs last disabled at (9803281): [] irq_exit_rcu+0xb0/0xd0 [ 2135.433926] ---[ end trace 0000000000000000 ]--- [ 2135.438580] ------------[ cut here ]------------ [ 2135.443201] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e47cb0. (prev=ffff8881a1d36920). [ 2135.454862] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2135.463556] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2135.519053] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2135.528194] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2135.535682] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2135.541204] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2135.546707] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2135.565471] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2135.570715] RAX: 0000000000000000 RBX: ffff8881d12c7560 RCX: 0000000000000000 [ 2135.577863] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2135.584988] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000b83 [ 2135.592119] R10: 000000000008dfe8 R11: ffffc9000030b828 R12: ffff8881d12c7cb0 [ 2135.599261] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2135.606413] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2135.614516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2135.620280] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2135.627429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2135.634581] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2135.641746] Call Trace: [ 2135.644191] [ 2135.646307] ? __warn+0x85/0x180 [ 2135.649557] ? __list_add_valid_or_report+0x7d/0xa0 [ 2135.654456] ? report_bug+0xfc/0x1d0 [ 2135.658051] ? prb_read_valid+0x1b/0x20 [ 2135.661909] ? handle_bug+0x44/0x70 [ 2135.665398] ? exc_invalid_op+0x17/0x60 [ 2135.669247] ? asm_exc_invalid_op+0x1a/0x20 [ 2135.673473] ? __list_add_valid_or_report+0x7d/0xa0 [ 2135.678348] ? __list_add_valid_or_report+0x7d/0xa0 [ 2135.683244] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2135.688152] ceph_add_cap+0x20b/0x620 [ceph] [ 2135.692472] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2135.697183] ? inode_insert5+0x10b/0x1c0 [ 2135.701125] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2135.705773] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2135.711117] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2135.715596] ceph_con_process_message+0x74/0x150 [libceph] [ 2135.721092] ? calc_signature+0xdb/0x100 [libceph] [ 2135.725913] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2135.731254] ? process_scheduled_works+0x22a/0x510 [ 2135.736063] ? process_scheduled_works+0x22a/0x510 [ 2135.740853] ceph_con_workfn+0x344/0x750 [libceph] [ 2135.745655] ? process_scheduled_works+0x22a/0x510 [ 2135.750464] process_scheduled_works+0x2a4/0x510 [ 2135.755106] worker_thread+0x188/0x340 [ 2135.758855] ? rescuer_thread+0x310/0x310 [ 2135.762866] kthread+0xf5/0x130 [ 2135.766013] ? kthread_complete_and_exit+0x20/0x20 [ 2135.770806] ret_from_fork+0x31/0x40 [ 2135.774386] ? kthread_complete_and_exit+0x20/0x20 [ 2135.779196] ret_from_fork_asm+0x11/0x20 [ 2135.783163] [ 2135.785348] irq event stamp: 9804307 [ 2135.788946] hardirqs last enabled at (9804317): [] console_unlock+0x106/0x120 [ 2135.797743] hardirqs last disabled at (9804326): [] console_unlock+0xeb/0x120 [ 2135.806433] softirqs last enabled at (9804274): [] __do_softirq+0x2be/0x371 [ 2135.815056] softirqs last disabled at (9804269): [] irq_exit_rcu+0xb0/0xd0 [ 2135.823508] ---[ end trace 0000000000000000 ]--- [ 2135.828202] ------------[ cut here ]------------ [ 2135.832824] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881c2e47cb0. (prev=ffff8881a1d36920). [ 2135.844489] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2135.853186] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2135.908678] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2135.917823] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2135.925300] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2135.930812] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2135.936298] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2135.955062] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2135.960306] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2135.967476] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2135.974606] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000bbf [ 2135.981757] R10: 000000000008f098 R11: ffffc9000030b828 R12: ffff8881d12c5590 [ 2135.988890] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2135.996020] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2136.004124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2136.009872] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2136.017005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2136.024154] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2136.031305] Call Trace: [ 2136.033773] [ 2136.035880] ? __warn+0x85/0x180 [ 2136.039113] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.044025] ? report_bug+0xfc/0x1d0 [ 2136.047624] ? prb_read_valid+0x1b/0x20 [ 2136.051481] ? handle_bug+0x44/0x70 [ 2136.054991] ? exc_invalid_op+0x17/0x60 [ 2136.058829] ? asm_exc_invalid_op+0x1a/0x20 [ 2136.063054] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.067931] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.072810] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2136.077700] ceph_add_cap+0x20b/0x620 [ceph] [ 2136.082009] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2136.086744] ? inode_insert5+0x10b/0x1c0 [ 2136.090673] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2136.095338] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2136.100681] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2136.105162] ceph_con_process_message+0x74/0x150 [libceph] [ 2136.110658] ? calc_signature+0xdb/0x100 [libceph] [ 2136.115488] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2136.120836] ? process_scheduled_works+0x22a/0x510 [ 2136.125644] ? process_scheduled_works+0x22a/0x510 [ 2136.130433] ceph_con_workfn+0x344/0x750 [libceph] [ 2136.135256] ? process_scheduled_works+0x22a/0x510 [ 2136.140063] process_scheduled_works+0x2a4/0x510 [ 2136.144689] worker_thread+0x188/0x340 [ 2136.148438] ? rescuer_thread+0x310/0x310 [ 2136.152448] kthread+0xf5/0x130 [ 2136.155612] ? kthread_complete_and_exit+0x20/0x20 [ 2136.160422] ret_from_fork+0x31/0x40 [ 2136.164018] ? kthread_complete_and_exit+0x20/0x20 [ 2136.168812] ret_from_fork_asm+0x11/0x20 [ 2136.172765] [ 2136.174956] irq event stamp: 9805399 [ 2136.178534] hardirqs last enabled at (9805409): [] console_unlock+0x106/0x120 [ 2136.187333] hardirqs last disabled at (9805418): [] console_unlock+0xeb/0x120 [ 2136.196057] softirqs last enabled at (9805386): [] __do_softirq+0x2be/0x371 [ 2136.204675] softirqs last disabled at (9805381): [] irq_exit_rcu+0xb0/0xd0 [ 2136.213124] ---[ end trace 0000000000000000 ]--- [ 2136.217774] ------------[ cut here ]------------ [ 2136.222410] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881f20b9ae0. (prev=ffff8881a1d36920). [ 2136.234074] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2136.242766] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2136.298243] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2136.307386] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2136.314883] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2136.320376] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2136.325879] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2136.344627] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2136.349871] RAX: 0000000000000000 RBX: ffff8881d13c8000 RCX: 0000000000000000 [ 2136.357002] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2136.364134] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000bfb [ 2136.371278] R10: 00000000000901e8 R11: ffffc9000030b828 R12: ffff8881d13c8750 [ 2136.378428] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2136.385579] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2136.393680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2136.399444] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2136.406595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2136.413760] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2136.420887] Call Trace: [ 2136.423340] [ 2136.425461] ? __warn+0x85/0x180 [ 2136.428713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.433609] ? report_bug+0xfc/0x1d0 [ 2136.437205] ? prb_read_valid+0x1b/0x20 [ 2136.441065] ? handle_bug+0x44/0x70 [ 2136.444573] ? exc_invalid_op+0x17/0x60 [ 2136.448430] ? asm_exc_invalid_op+0x1a/0x20 [ 2136.452653] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.457531] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.462425] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2136.467336] ceph_add_cap+0x20b/0x620 [ceph] [ 2136.471653] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2136.476392] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2136.481743] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2136.486224] ceph_con_process_message+0x74/0x150 [libceph] [ 2136.491723] ? calc_signature+0xdb/0x100 [libceph] [ 2136.496550] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2136.501891] ? process_scheduled_works+0x22a/0x510 [ 2136.506700] ? process_scheduled_works+0x22a/0x510 [ 2136.511490] ceph_con_workfn+0x344/0x750 [libceph] [ 2136.516312] ? process_scheduled_works+0x22a/0x510 [ 2136.521117] process_scheduled_works+0x2a4/0x510 [ 2136.525766] worker_thread+0x188/0x340 [ 2136.529517] ? rescuer_thread+0x310/0x310 [ 2136.533548] kthread+0xf5/0x130 [ 2136.536692] ? kthread_complete_and_exit+0x20/0x20 [ 2136.541504] ret_from_fork+0x31/0x40 [ 2136.545101] ? kthread_complete_and_exit+0x20/0x20 [ 2136.549911] ret_from_fork_asm+0x11/0x20 [ 2136.553845] [ 2136.556039] irq event stamp: 9806431 [ 2136.559635] hardirqs last enabled at (9806441): [] console_unlock+0x106/0x120 [ 2136.568429] hardirqs last disabled at (9806450): [] console_unlock+0xeb/0x120 [ 2136.577140] softirqs last enabled at (9806418): [] __do_softirq+0x2be/0x371 [ 2136.585765] softirqs last disabled at (9806413): [] irq_exit_rcu+0xb0/0xd0 [ 2136.594196] ---[ end trace 0000000000000000 ]--- [ 2136.598832] ------------[ cut here ]------------ [ 2136.603445] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881f20b9ae0. (prev=ffff8881a1d36920). [ 2136.615125] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2136.623820] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2136.679297] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2136.688441] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2136.695938] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2136.701432] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2136.706935] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2136.725682] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2136.730923] RAX: 0000000000000000 RBX: ffff8881d13ce1d0 RCX: 0000000000000000 [ 2136.738059] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2136.745208] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000c35 [ 2136.752357] R10: 0000000000091188 R11: ffffc9000030b828 R12: ffff8881d13ce920 [ 2136.759507] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2136.766648] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2136.774761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2136.780507] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2136.787651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2136.794798] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2136.801931] Call Trace: [ 2136.804375] [ 2136.806491] ? __warn+0x85/0x180 [ 2136.809762] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.814637] ? report_bug+0xfc/0x1d0 [ 2136.818235] ? prb_read_valid+0x1b/0x20 [ 2136.822092] ? handle_bug+0x44/0x70 [ 2136.825600] ? exc_invalid_op+0x17/0x60 [ 2136.829448] ? asm_exc_invalid_op+0x1a/0x20 [ 2136.833676] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.838549] ? __list_add_valid_or_report+0x7d/0xa0 [ 2136.843446] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2136.848353] ceph_add_cap+0x20b/0x620 [ceph] [ 2136.852640] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2136.857386] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2136.862729] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2136.867217] ceph_con_process_message+0x74/0x150 [libceph] [ 2136.872716] ? calc_signature+0xdb/0x100 [libceph] [ 2136.877545] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2136.882886] ? process_scheduled_works+0x22a/0x510 [ 2136.887685] ? process_scheduled_works+0x22a/0x510 [ 2136.892493] ceph_con_workfn+0x344/0x750 [libceph] [ 2136.897312] ? process_scheduled_works+0x22a/0x510 [ 2136.902120] process_scheduled_works+0x2a4/0x510 [ 2136.906771] worker_thread+0x188/0x340 [ 2136.910519] ? rescuer_thread+0x310/0x310 [ 2136.914549] kthread+0xf5/0x130 [ 2136.917712] ? kthread_complete_and_exit+0x20/0x20 [ 2136.922506] ret_from_fork+0x31/0x40 [ 2136.926102] ? kthread_complete_and_exit+0x20/0x20 [ 2136.930919] ret_from_fork_asm+0x11/0x20 [ 2136.934857] [ 2136.937047] irq event stamp: 9807471 [ 2136.940636] hardirqs last enabled at (9807481): [] console_unlock+0x106/0x120 [ 2136.949434] hardirqs last disabled at (9807496): [] console_unlock+0xeb/0x120 [ 2136.958142] softirqs last enabled at (9807494): [] __do_softirq+0x2be/0x371 [ 2136.966768] softirqs last disabled at (9807489): [] irq_exit_rcu+0xb0/0xd0 [ 2136.975198] ---[ end trace 0000000000000000 ]--- [ 2136.979973] ------------[ cut here ]------------ [ 2136.984606] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881f20b9ae0. (prev=ffff8881a1d36920). [ 2136.996266] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2137.004963] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2137.060437] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2137.069583] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2137.077078] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2137.082590] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2137.088092] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2137.106857] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2137.112084] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2137.119217] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2137.126364] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000c6f [ 2137.133509] R10: 00000000000921c8 R11: ffffc9000030b828 R12: ffff8881d12c5590 [ 2137.140657] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2137.147808] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2137.155895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2137.161640] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2137.168790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2137.175923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2137.183055] Call Trace: [ 2137.185517] [ 2137.187640] ? __warn+0x85/0x180 [ 2137.190890] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.195773] ? report_bug+0xfc/0x1d0 [ 2137.199351] ? prb_read_valid+0x1b/0x20 [ 2137.203208] ? handle_bug+0x44/0x70 [ 2137.206718] ? exc_invalid_op+0x17/0x60 [ 2137.210573] ? asm_exc_invalid_op+0x1a/0x20 [ 2137.214783] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.219668] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.224560] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2137.229471] ceph_add_cap+0x20b/0x620 [ceph] [ 2137.233780] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2137.238510] ? inode_insert5+0x10b/0x1c0 [ 2137.242435] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2137.247099] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2137.252442] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2137.256905] ceph_con_process_message+0x74/0x150 [libceph] [ 2137.262402] ? calc_signature+0xdb/0x100 [libceph] [ 2137.267279] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2137.272625] ? process_scheduled_works+0x22a/0x510 [ 2137.277422] ? process_scheduled_works+0x22a/0x510 [ 2137.282232] ceph_con_workfn+0x344/0x750 [libceph] [ 2137.287055] ? process_scheduled_works+0x22a/0x510 [ 2137.291868] process_scheduled_works+0x2a4/0x510 [ 2137.296510] worker_thread+0x188/0x340 [ 2137.300258] ? rescuer_thread+0x310/0x310 [ 2137.304270] kthread+0xf5/0x130 [ 2137.307433] ? kthread_complete_and_exit+0x20/0x20 [ 2137.312244] ret_from_fork+0x31/0x40 [ 2137.315840] ? kthread_complete_and_exit+0x20/0x20 [ 2137.320633] ret_from_fork_asm+0x11/0x20 [ 2137.324603] [ 2137.326797] irq event stamp: 9808573 [ 2137.330375] hardirqs last enabled at (9808583): [] console_unlock+0x106/0x120 [ 2137.339171] hardirqs last disabled at (9808592): [] console_unlock+0xeb/0x120 [ 2137.347880] softirqs last enabled at (9808560): [] __do_softirq+0x2be/0x371 [ 2137.356489] softirqs last disabled at (9808555): [] irq_exit_rcu+0xb0/0xd0 [ 2137.364939] ---[ end trace 0000000000000000 ]--- [ 2137.369571] ------------[ cut here ]------------ [ 2137.374187] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881f20bae70. (prev=ffff8881a1d36920). [ 2137.385849] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2137.394543] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2137.450054] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2137.459198] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2137.466698] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2137.472214] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2137.477719] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2137.496482] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2137.501727] RAX: 0000000000000000 RBX: ffff8881d0bde1d0 RCX: 0000000000000000 [ 2137.508879] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2137.516011] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000cab [ 2137.523141] R10: 0000000000093278 R11: ffffc9000030b828 R12: ffff8881d0bde920 [ 2137.530282] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2137.537423] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2137.545529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2137.551292] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2137.558442] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2137.565591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2137.572741] Call Trace: [ 2137.575202] [ 2137.577319] ? __warn+0x85/0x180 [ 2137.580569] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.585465] ? report_bug+0xfc/0x1d0 [ 2137.589062] ? prb_read_valid+0x1b/0x20 [ 2137.592920] ? handle_bug+0x44/0x70 [ 2137.596411] ? exc_invalid_op+0x17/0x60 [ 2137.600268] ? asm_exc_invalid_op+0x1a/0x20 [ 2137.604494] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.609369] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.614264] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2137.619174] ceph_add_cap+0x20b/0x620 [ceph] [ 2137.623492] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2137.628229] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2137.633557] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2137.638021] ceph_con_process_message+0x74/0x150 [libceph] [ 2137.643516] ? calc_signature+0xdb/0x100 [libceph] [ 2137.648336] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2137.653693] ? process_scheduled_works+0x22a/0x510 [ 2137.658503] ? process_scheduled_works+0x22a/0x510 [ 2137.663292] ceph_con_workfn+0x344/0x750 [libceph] [ 2137.668114] ? process_scheduled_works+0x22a/0x510 [ 2137.672922] process_scheduled_works+0x2a4/0x510 [ 2137.677565] worker_thread+0x188/0x340 [ 2137.681321] ? rescuer_thread+0x310/0x310 [ 2137.685350] kthread+0xf5/0x130 [ 2137.688497] ? kthread_complete_and_exit+0x20/0x20 [ 2137.693307] ret_from_fork+0x31/0x40 [ 2137.696903] ? kthread_complete_and_exit+0x20/0x20 [ 2137.701697] ret_from_fork_asm+0x11/0x20 [ 2137.705664] [ 2137.707857] irq event stamp: 9809589 [ 2137.711428] hardirqs last enabled at (9809599): [] console_unlock+0x106/0x120 [ 2137.720225] hardirqs last disabled at (9809608): [] console_unlock+0xeb/0x120 [ 2137.728937] softirqs last enabled at (9809622): [] __do_softirq+0x2be/0x371 [ 2137.737540] softirqs last disabled at (9809617): [] irq_exit_rcu+0xb0/0xd0 [ 2137.745993] ---[ end trace 0000000000000000 ]--- [ 2137.750660] ------------[ cut here ]------------ [ 2137.755295] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881ec2f72e8. (prev=ffff8881a1d36920). [ 2137.766941] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2137.775634] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2137.831128] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2137.840270] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2137.847787] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2137.853289] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2137.858793] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2137.877538] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2137.882785] RAX: 0000000000000000 RBX: ffff8881d13cce40 RCX: 0000000000000000 [ 2137.889914] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2137.897062] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000ce5 [ 2137.904188] R10: 00000000000942b0 R11: ffffc9000030b828 R12: ffff8881d13cd590 [ 2137.911337] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2137.918479] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2137.926584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2137.932346] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2137.939495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2137.946637] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2137.953789] Call Trace: [ 2137.956242] [ 2137.958364] ? __warn+0x85/0x180 [ 2137.961616] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.966512] ? report_bug+0xfc/0x1d0 [ 2137.970107] ? prb_read_valid+0x1b/0x20 [ 2137.973965] ? handle_bug+0x44/0x70 [ 2137.977457] ? exc_invalid_op+0x17/0x60 [ 2137.981315] ? asm_exc_invalid_op+0x1a/0x20 [ 2137.985542] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.990425] ? __list_add_valid_or_report+0x7d/0xa0 [ 2137.995321] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2138.000229] ceph_add_cap+0x20b/0x620 [ceph] [ 2138.004547] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2138.009283] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2138.014628] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2138.019107] ceph_con_process_message+0x74/0x150 [libceph] [ 2138.024605] ? calc_signature+0xdb/0x100 [libceph] [ 2138.029426] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2138.034784] ? process_scheduled_works+0x22a/0x510 [ 2138.039577] ? process_scheduled_works+0x22a/0x510 [ 2138.044381] ceph_con_workfn+0x344/0x750 [libceph] [ 2138.049203] ? process_scheduled_works+0x22a/0x510 [ 2138.054011] process_scheduled_works+0x2a4/0x510 [ 2138.058653] worker_thread+0x188/0x340 [ 2138.062401] ? rescuer_thread+0x310/0x310 [ 2138.066431] kthread+0xf5/0x130 [ 2138.069594] ? kthread_complete_and_exit+0x20/0x20 [ 2138.074388] ret_from_fork+0x31/0x40 [ 2138.077984] ? kthread_complete_and_exit+0x20/0x20 [ 2138.082788] ret_from_fork_asm+0x11/0x20 [ 2138.086734] [ 2138.088922] irq event stamp: 9810641 [ 2138.092501] hardirqs last enabled at (9810651): [] console_unlock+0x106/0x120 [ 2138.101299] hardirqs last disabled at (9810666): [] console_unlock+0xeb/0x120 [ 2138.110007] softirqs last enabled at (9810664): [] __do_softirq+0x2be/0x371 [ 2138.118613] softirqs last disabled at (9810659): [] irq_exit_rcu+0xb0/0xd0 [ 2138.127047] ---[ end trace 0000000000000000 ]--- [ 2138.131707] ------------[ cut here ]------------ [ 2138.136342] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881e4f9d590. (prev=ffff8881a1d36920). [ 2138.147987] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2138.156679] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2138.212171] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2138.221318] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2138.228812] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2138.234308] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2138.239813] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2138.258555] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2138.263792] RAX: 0000000000000000 RBX: ffff8881d0bdf560 RCX: 0000000000000000 [ 2138.270923] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2138.278048] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000d1f [ 2138.285180] R10: 0000000000095430 R11: ffffc9000030b828 R12: ffff8881d0bdfcb0 [ 2138.292323] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2138.299472] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2138.307576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2138.313339] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2138.320481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2138.327623] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2138.334790] Call Trace: [ 2138.337244] [ 2138.339366] ? __warn+0x85/0x180 [ 2138.342617] ? __list_add_valid_or_report+0x7d/0xa0 [ 2138.347513] ? report_bug+0xfc/0x1d0 [ 2138.351110] ? prb_read_valid+0x1b/0x20 [ 2138.354969] ? handle_bug+0x44/0x70 [ 2138.358462] ? exc_invalid_op+0x17/0x60 [ 2138.362316] ? asm_exc_invalid_op+0x1a/0x20 [ 2138.366543] ? __list_add_valid_or_report+0x7d/0xa0 [ 2138.371416] ? __list_add_valid_or_report+0x7d/0xa0 [ 2138.376313] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2138.381221] ceph_add_cap+0x20b/0x620 [ceph] [ 2138.385542] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2138.390287] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2138.395632] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2138.400111] ceph_con_process_message+0x74/0x150 [libceph] [ 2138.405608] ? calc_signature+0xdb/0x100 [libceph] [ 2138.410430] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2138.415794] ? process_scheduled_works+0x22a/0x510 [ 2138.420586] ? process_scheduled_works+0x22a/0x510 [ 2138.425395] ceph_con_workfn+0x344/0x750 [libceph] [ 2138.430215] ? process_scheduled_works+0x22a/0x510 [ 2138.435021] process_scheduled_works+0x2a4/0x510 [ 2138.439662] worker_thread+0x188/0x340 [ 2138.443413] ? rescuer_thread+0x310/0x310 [ 2138.447442] kthread+0xf5/0x130 [ 2138.450605] ? kthread_complete_and_exit+0x20/0x20 [ 2138.455401] ret_from_fork+0x31/0x40 [ 2138.458995] ? kthread_complete_and_exit+0x20/0x20 [ 2138.463797] ret_from_fork_asm+0x11/0x20 [ 2138.467729] [ 2138.469915] irq event stamp: 9811659 [ 2138.473495] hardirqs last enabled at (9811669): [] console_unlock+0x106/0x120 [ 2138.482291] hardirqs last disabled at (9811678): [] console_unlock+0xeb/0x120 [ 2138.491000] softirqs last enabled at (9811406): [] __do_softirq+0x2be/0x371 [ 2138.499608] softirqs last disabled at (9811697): [] irq_exit_rcu+0xb0/0xd0 [ 2138.508058] ---[ end trace 0000000000000000 ]--- [ 2138.512700] ------------[ cut here ]------------ [ 2138.517317] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881f218fcb0. (prev=ffff8881a1d36920). [ 2138.528976] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2138.537671] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2138.593165] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2138.602308] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2138.609801] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2138.615304] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2138.620803] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2138.639551] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2138.644799] RAX: 0000000000000000 RBX: ffff8881d13d09c8 RCX: 0000000000000000 [ 2138.651928] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2138.659074] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000d59 [ 2138.666217] R10: 0000000000096468 R11: ffffc9000030b828 R12: ffff8881d13d1118 [ 2138.673359] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2138.680501] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2138.688606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2138.694368] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2138.701510] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2138.708660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2138.715811] Call Trace: [ 2138.718262] [ 2138.720378] ? __warn+0x85/0x180 [ 2138.723629] ? __list_add_valid_or_report+0x7d/0xa0 [ 2138.728525] ? report_bug+0xfc/0x1d0 [ 2138.732121] ? prb_read_valid+0x1b/0x20 [ 2138.735981] ? handle_bug+0x44/0x70 [ 2138.739470] ? exc_invalid_op+0x17/0x60 [ 2138.743319] ? asm_exc_invalid_op+0x1a/0x20 [ 2138.747543] ? __list_add_valid_or_report+0x7d/0xa0 [ 2138.752418] ? __list_add_valid_or_report+0x7d/0xa0 [ 2138.757315] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2138.762222] ceph_add_cap+0x20b/0x620 [ceph] [ 2138.766540] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2138.771267] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2138.776615] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2138.781106] ceph_con_process_message+0x74/0x150 [libceph] [ 2138.786603] ? calc_signature+0xdb/0x100 [libceph] [ 2138.791432] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2138.796791] ? process_scheduled_works+0x22a/0x510 [ 2138.801586] ? process_scheduled_works+0x22a/0x510 [ 2138.806394] ceph_con_workfn+0x344/0x750 [libceph] [ 2138.811215] ? process_scheduled_works+0x22a/0x510 [ 2138.816025] process_scheduled_works+0x2a4/0x510 [ 2138.820665] worker_thread+0x188/0x340 [ 2138.824416] ? rescuer_thread+0x310/0x310 [ 2138.828443] kthread+0xf5/0x130 [ 2138.831609] ? kthread_complete_and_exit+0x20/0x20 [ 2138.836400] ret_from_fork+0x31/0x40 [ 2138.839997] ? kthread_complete_and_exit+0x20/0x20 [ 2138.844804] ret_from_fork_asm+0x11/0x20 [ 2138.848748] [ 2138.850934] irq event stamp: 9812673 [ 2138.854512] hardirqs last enabled at (9812683): [] console_unlock+0x106/0x120 [ 2138.863302] hardirqs last disabled at (9812692): [] console_unlock+0xeb/0x120 [ 2138.872015] softirqs last enabled at (9812706): [] __do_softirq+0x2be/0x371 [ 2138.880619] softirqs last disabled at (9812701): [] irq_exit_rcu+0xb0/0xd0 [ 2138.889082] ---[ end trace 0000000000000000 ]--- [ 2138.893737] ------------[ cut here ]------------ [ 2138.898351] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2138.909995] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2138.918688] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2138.974194] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2138.983322] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2138.990807] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2138.996305] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2139.001805] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2139.020553] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2139.025795] RAX: 0000000000000000 RBX: ffff8881e4da09c8 RCX: 0000000000000000 [ 2139.032929] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2139.040076] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000d93 [ 2139.047201] R10: 00000000000975e0 R11: ffffc9000030b828 R12: ffff8881e4da1118 [ 2139.054327] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2139.061459] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2139.069545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2139.075283] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2139.082409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2139.089541] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2139.096673] Call Trace: [ 2139.099125] [ 2139.101224] ? __warn+0x85/0x180 [ 2139.104475] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.109353] ? report_bug+0xfc/0x1d0 [ 2139.112932] ? prb_read_valid+0x1b/0x20 [ 2139.116772] ? handle_bug+0x44/0x70 [ 2139.120281] ? exc_invalid_op+0x17/0x60 [ 2139.124114] ? asm_exc_invalid_op+0x1a/0x20 [ 2139.128306] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.133189] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.138081] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2139.142993] ceph_add_cap+0x20b/0x620 [ceph] [ 2139.147303] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2139.152024] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2139.157349] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2139.161839] ceph_con_process_message+0x74/0x150 [libceph] [ 2139.167333] ? calc_signature+0xdb/0x100 [libceph] [ 2139.172164] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2139.177514] ? process_scheduled_works+0x22a/0x510 [ 2139.182304] ? process_scheduled_works+0x22a/0x510 [ 2139.187110] ceph_con_workfn+0x344/0x750 [libceph] [ 2139.191916] ? process_scheduled_works+0x22a/0x510 [ 2139.196706] process_scheduled_works+0x2a4/0x510 [ 2139.201331] worker_thread+0x188/0x340 [ 2139.205087] ? rescuer_thread+0x310/0x310 [ 2139.209100] kthread+0xf5/0x130 [ 2139.212246] ? kthread_complete_and_exit+0x20/0x20 [ 2139.217043] ret_from_fork+0x31/0x40 [ 2139.220616] ? kthread_complete_and_exit+0x20/0x20 [ 2139.225411] ret_from_fork_asm+0x11/0x20 [ 2139.229362] [ 2139.231555] irq event stamp: 9813719 [ 2139.235133] hardirqs last enabled at (9813729): [] console_unlock+0x106/0x120 [ 2139.243931] hardirqs last disabled at (9813738): [] console_unlock+0xeb/0x120 [ 2139.252624] softirqs last enabled at (9813486): [] __do_softirq+0x2be/0x371 [ 2139.261229] softirqs last disabled at (9813481): [] irq_exit_rcu+0xb0/0xd0 [ 2139.269662] ---[ end trace 0000000000000000 ]--- [ 2139.274432] ------------[ cut here ]------------ [ 2139.279054] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2139.290695] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2139.299391] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2139.354885] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2139.364010] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2139.371492] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2139.376993] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2139.382480] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2139.401245] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2139.406487] RAX: 0000000000000000 RBX: ffff8881d12d8000 RCX: 0000000000000000 [ 2139.413637] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2139.420802] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000dcd [ 2139.427928] R10: 0000000000098608 R11: ffffc9000030b828 R12: ffff8881d12d8750 [ 2139.435061] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2139.442192] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2139.450287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2139.456042] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2139.463175] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2139.470317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2139.477458] Call Trace: [ 2139.479921] [ 2139.482027] ? __warn+0x85/0x180 [ 2139.485260] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.490157] ? report_bug+0xfc/0x1d0 [ 2139.493753] ? prb_read_valid+0x1b/0x20 [ 2139.497612] ? handle_bug+0x44/0x70 [ 2139.501120] ? exc_invalid_op+0x17/0x60 [ 2139.504960] ? asm_exc_invalid_op+0x1a/0x20 [ 2139.509168] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.514044] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.518922] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2139.523818] ceph_add_cap+0x20b/0x620 [ceph] [ 2139.528122] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2139.532853] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2139.538199] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2139.542675] ceph_con_process_message+0x74/0x150 [libceph] [ 2139.548171] ? calc_signature+0xdb/0x100 [libceph] [ 2139.552994] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2139.558315] ? process_scheduled_works+0x22a/0x510 [ 2139.563125] ? process_scheduled_works+0x22a/0x510 [ 2139.567932] ceph_con_workfn+0x344/0x750 [libceph] [ 2139.572735] ? process_scheduled_works+0x22a/0x510 [ 2139.577543] process_scheduled_works+0x2a4/0x510 [ 2139.582202] worker_thread+0x188/0x340 [ 2139.585951] ? rescuer_thread+0x310/0x310 [ 2139.589963] kthread+0xf5/0x130 [ 2139.593121] ? kthread_complete_and_exit+0x20/0x20 [ 2139.597930] ret_from_fork+0x31/0x40 [ 2139.601508] ? kthread_complete_and_exit+0x20/0x20 [ 2139.606317] ret_from_fork_asm+0x11/0x20 [ 2139.610285] [ 2139.612471] irq event stamp: 9814727 [ 2139.616050] hardirqs last enabled at (9814737): [] console_unlock+0x106/0x120 [ 2139.624831] hardirqs last disabled at (9814746): [] console_unlock+0xeb/0x120 [ 2139.633522] softirqs last enabled at (9814554): [] __do_softirq+0x2be/0x371 [ 2139.642146] softirqs last disabled at (9814549): [] irq_exit_rcu+0xb0/0xd0 [ 2139.650595] ---[ end trace 0000000000000000 ]--- [ 2139.655249] ------------[ cut here ]------------ [ 2139.659866] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2139.671523] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2139.680212] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2139.735704] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2139.744848] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2139.752327] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2139.757840] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2139.763325] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2139.782094] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2139.787315] RAX: 0000000000000000 RBX: ffff8881d13c6b98 RCX: 0000000000000000 [ 2139.794449] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2139.801581] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000e07 [ 2139.808713] R10: 00000000000997d0 R11: ffffc9000030b828 R12: ffff8881d13c72e8 [ 2139.815865] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2139.822997] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2139.831084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2139.836830] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2139.843961] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2139.851096] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2139.858229] Call Trace: [ 2139.860681] [ 2139.862785] ? __warn+0x85/0x180 [ 2139.866019] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.870898] ? report_bug+0xfc/0x1d0 [ 2139.874479] ? prb_read_valid+0x1b/0x20 [ 2139.878320] ? handle_bug+0x44/0x70 [ 2139.881821] ? exc_invalid_op+0x17/0x60 [ 2139.885658] ? asm_exc_invalid_op+0x1a/0x20 [ 2139.889851] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.894733] ? __list_add_valid_or_report+0x7d/0xa0 [ 2139.899611] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2139.904503] ceph_add_cap+0x20b/0x620 [ceph] [ 2139.908822] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2139.913543] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2139.918897] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2139.923358] ceph_con_process_message+0x74/0x150 [libceph] [ 2139.928880] ? calc_signature+0xdb/0x100 [libceph] [ 2139.933693] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2139.939042] ? process_scheduled_works+0x22a/0x510 [ 2139.943842] ? process_scheduled_works+0x22a/0x510 [ 2139.948633] ceph_con_workfn+0x344/0x750 [libceph] [ 2139.953461] ? process_scheduled_works+0x22a/0x510 [ 2139.958269] process_scheduled_works+0x2a4/0x510 [ 2139.962892] worker_thread+0x188/0x340 [ 2139.966641] ? rescuer_thread+0x310/0x310 [ 2139.970653] kthread+0xf5/0x130 [ 2139.973818] ? kthread_complete_and_exit+0x20/0x20 [ 2139.978611] ret_from_fork+0x31/0x40 [ 2139.982190] ? kthread_complete_and_exit+0x20/0x20 [ 2139.986982] ret_from_fork_asm+0x11/0x20 [ 2139.990916] [ 2139.993101] irq event stamp: 9815737 [ 2139.996680] hardirqs last enabled at (9815747): [] console_unlock+0x106/0x120 [ 2140.005460] hardirqs last disabled at (9815756): [] console_unlock+0xeb/0x120 [ 2140.014153] softirqs last enabled at (9815564): [] __do_softirq+0x2be/0x371 [ 2140.022759] softirqs last disabled at (9815559): [] irq_exit_rcu+0xb0/0xd0 [ 2140.031191] ---[ end trace 0000000000000000 ]--- [ 2140.035832] ------------[ cut here ]------------ [ 2140.040449] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2140.052106] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2140.060823] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2140.116299] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2140.125427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2140.132923] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2140.138416] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2140.143904] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2140.162650] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2140.167875] RAX: 0000000000000000 RBX: ffff8881e4ee61d0 RCX: 0000000000000000 [ 2140.175010] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2140.182142] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000e41 [ 2140.189273] R10: 000000000009a950 R11: ffffc9000030b828 R12: ffff8881e4ee6920 [ 2140.196399] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2140.203531] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2140.211617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2140.217365] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2140.224497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2140.231628] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2140.238760] Call Trace: [ 2140.241223] [ 2140.243330] ? __warn+0x85/0x180 [ 2140.246562] ? __list_add_valid_or_report+0x7d/0xa0 [ 2140.251443] ? report_bug+0xfc/0x1d0 [ 2140.255020] ? prb_read_valid+0x1b/0x20 [ 2140.258864] ? handle_bug+0x44/0x70 [ 2140.262353] ? exc_invalid_op+0x17/0x60 [ 2140.266194] ? asm_exc_invalid_op+0x1a/0x20 [ 2140.270383] ? __list_add_valid_or_report+0x7d/0xa0 [ 2140.275261] ? __list_add_valid_or_report+0x7d/0xa0 [ 2140.280136] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2140.285029] ceph_add_cap+0x20b/0x620 [ceph] [ 2140.289312] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2140.294053] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2140.298680] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2140.304032] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2140.308496] ceph_con_process_message+0x74/0x150 [libceph] [ 2140.314016] ? calc_signature+0xdb/0x100 [libceph] [ 2140.318830] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2140.324160] ? process_scheduled_works+0x22a/0x510 [ 2140.328953] ? process_scheduled_works+0x22a/0x510 [ 2140.333740] ceph_con_workfn+0x344/0x750 [libceph] [ 2140.338545] ? process_scheduled_works+0x22a/0x510 [ 2140.343336] process_scheduled_works+0x2a4/0x510 [ 2140.347962] worker_thread+0x188/0x340 [ 2140.351717] ? rescuer_thread+0x310/0x310 [ 2140.355729] kthread+0xf5/0x130 [ 2140.358877] ? kthread_complete_and_exit+0x20/0x20 [ 2140.363669] ret_from_fork+0x31/0x40 [ 2140.367247] ? kthread_complete_and_exit+0x20/0x20 [ 2140.372041] ret_from_fork_asm+0x11/0x20 [ 2140.375976] [ 2140.378167] irq event stamp: 9816755 [ 2140.381747] hardirqs last enabled at (9816765): [] console_unlock+0x106/0x120 [ 2140.390528] hardirqs last disabled at (9816774): [] console_unlock+0xeb/0x120 [ 2140.399219] softirqs last enabled at (9816592): [] __do_softirq+0x2be/0x371 [ 2140.407829] softirqs last disabled at (9816587): [] irq_exit_rcu+0xb0/0xd0 [ 2140.416259] ---[ end trace 0000000000000000 ]--- [ 2140.420887] ------------[ cut here ]------------ [ 2140.425508] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2140.437152] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2140.445848] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2140.501323] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2140.510448] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2140.517920] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2140.523416] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2140.528900] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2140.547648] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2140.552875] RAX: 0000000000000000 RBX: ffff8881bf19d808 RCX: 0000000000000000 [ 2140.560006] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2140.567140] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000e7c [ 2140.574272] R10: 000000000009ba70 R11: ffffc9000030b828 R12: ffff8881bf19df58 [ 2140.581403] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2140.588528] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2140.596615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2140.602361] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2140.609495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2140.616626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2140.623760] Call Trace: [ 2140.626213] [ 2140.628317] ? __warn+0x85/0x180 [ 2140.631551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2140.636432] ? report_bug+0xfc/0x1d0 [ 2140.640010] ? prb_read_valid+0x1b/0x20 [ 2140.643852] ? handle_bug+0x44/0x70 [ 2140.647342] ? exc_invalid_op+0x17/0x60 [ 2140.651182] ? asm_exc_invalid_op+0x1a/0x20 [ 2140.655371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2140.660248] ? __list_add_valid_or_report+0x7d/0xa0 [ 2140.665126] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2140.670017] ceph_add_cap+0x20b/0x620 [ceph] [ 2140.674301] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2140.679049] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2140.684376] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2140.688864] ceph_con_process_message+0x74/0x150 [libceph] [ 2140.694362] ? calc_signature+0xdb/0x100 [libceph] [ 2140.699189] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2140.704537] ? process_scheduled_works+0x22a/0x510 [ 2140.709329] ? process_scheduled_works+0x22a/0x510 [ 2140.714119] ceph_con_workfn+0x344/0x750 [libceph] [ 2140.718923] ? process_scheduled_works+0x22a/0x510 [ 2140.723713] process_scheduled_works+0x2a4/0x510 [ 2140.728338] worker_thread+0x188/0x340 [ 2140.732086] ? rescuer_thread+0x310/0x310 [ 2140.736117] kthread+0xf5/0x130 [ 2140.739262] ? kthread_complete_and_exit+0x20/0x20 [ 2140.744056] ret_from_fork+0x31/0x40 [ 2140.747634] ? kthread_complete_and_exit+0x20/0x20 [ 2140.752427] ret_from_fork_asm+0x11/0x20 [ 2140.756359] [ 2140.758544] irq event stamp: 9817765 [ 2140.762134] hardirqs last enabled at (9817775): [] console_unlock+0x106/0x120 [ 2140.770912] hardirqs last disabled at (9817784): [] console_unlock+0xeb/0x120 [ 2140.779605] softirqs last enabled at (9817612): [] __do_softirq+0x2be/0x371 [ 2140.788215] softirqs last disabled at (9817607): [] irq_exit_rcu+0xb0/0xd0 [ 2140.796645] ---[ end trace 0000000000000000 ]--- [ 2140.801293] ------------[ cut here ]------------ [ 2140.805912] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2140.817557] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2140.826253] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2140.881756] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2140.890879] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2140.898359] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2140.903881] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2140.909364] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2140.928173] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2140.933403] RAX: 0000000000000000 RBX: ffff8881d0b89d58 RCX: 0000000000000000 [ 2140.940530] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2140.947657] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000eb6 [ 2140.954790] R10: 000000000009cc28 R11: ffffc9000030b828 R12: ffff8881d0b8a4a8 [ 2140.961939] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2140.969072] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2140.977157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2140.982903] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2140.990037] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2140.997170] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2141.004319] Call Trace: [ 2141.006763] [ 2141.008869] ? __warn+0x85/0x180 [ 2141.012118] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.017017] ? report_bug+0xfc/0x1d0 [ 2141.020595] ? prb_read_valid+0x1b/0x20 [ 2141.024439] ? handle_bug+0x44/0x70 [ 2141.027928] ? exc_invalid_op+0x17/0x60 [ 2141.031768] ? asm_exc_invalid_op+0x1a/0x20 [ 2141.035958] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.040842] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.045720] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2141.050611] ceph_add_cap+0x20b/0x620 [ceph] [ 2141.054923] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2141.059646] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2141.064298] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2141.069651] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2141.074138] ceph_con_process_message+0x74/0x150 [libceph] [ 2141.079653] ? calc_signature+0xdb/0x100 [libceph] [ 2141.084483] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2141.089848] ? process_scheduled_works+0x22a/0x510 [ 2141.094640] ? process_scheduled_works+0x22a/0x510 [ 2141.099429] ceph_con_workfn+0x344/0x750 [libceph] [ 2141.104232] ? process_scheduled_works+0x22a/0x510 [ 2141.109023] process_scheduled_works+0x2a4/0x510 [ 2141.113648] worker_thread+0x188/0x340 [ 2141.117396] ? rescuer_thread+0x310/0x310 [ 2141.121427] kthread+0xf5/0x130 [ 2141.124572] ? kthread_complete_and_exit+0x20/0x20 [ 2141.129365] ret_from_fork+0x31/0x40 [ 2141.132943] ? kthread_complete_and_exit+0x20/0x20 [ 2141.137737] ret_from_fork_asm+0x11/0x20 [ 2141.141670] [ 2141.143864] irq event stamp: 9818877 [ 2141.147443] hardirqs last enabled at (9818887): [] console_unlock+0x106/0x120 [ 2141.156229] hardirqs last disabled at (9818896): [] console_unlock+0xeb/0x120 [ 2141.164916] softirqs last enabled at (9818734): [] __do_softirq+0x2be/0x371 [ 2141.173520] softirqs last disabled at (9818729): [] irq_exit_rcu+0xb0/0xd0 [ 2141.181955] ---[ end trace 0000000000000000 ]--- [ 2141.186869] ------------[ cut here ]------------ [ 2141.191493] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2141.203171] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2141.212134] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2141.267619] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2141.276588] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2141.284084] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2141.289569] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2141.308334] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2141.313576] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2141.320717] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2141.327860] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000ef1 [ 2141.334995] R10: 000000000009dcb0 R11: ffffc90001fffa60 R12: ffff8881a1d35f58 [ 2141.342137] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2141.349285] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2141.357387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2141.363142] CR2: 00007f9bd346f000 CR3: 000000017eb7a003 CR4: 00000000003706e0 [ 2141.370284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2141.377427] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2141.384577] Call Trace: [ 2141.387047] [ 2141.389153] ? __warn+0x85/0x180 [ 2141.392407] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.397300] ? report_bug+0xfc/0x1d0 [ 2141.400897] ? prb_read_valid+0x1b/0x20 [ 2141.404736] ? handle_bug+0x44/0x70 [ 2141.408244] ? exc_invalid_op+0x17/0x60 [ 2141.412093] ? asm_exc_invalid_op+0x1a/0x20 [ 2141.416300] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.421177] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2141.426084] try_get_cap_refs+0x159/0x880 [ceph] [ 2141.430732] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2141.435292] ? wake_bit_function+0x70/0x70 [ 2141.439404] ? ceph_read_iter+0x134/0x860 [ceph] [ 2141.444031] ceph_read_iter+0x134/0x860 [ceph] [ 2141.448507] ? selinux_file_permission+0x119/0x150 [ 2141.453298] ? vfs_read+0x20f/0x300 [ 2141.456806] vfs_read+0x20f/0x300 [ 2141.460165] ksys_pread64+0x65/0xa0 [ 2141.463652] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2141.468834] do_syscall_64+0x38/0x80 [ 2141.472414] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2141.477486] RIP: 0033:0x7f48f0eeb88b [ 2141.481083] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2141.499829] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2141.507396] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2141.514542] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2141.521685] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2141.528844] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2141.535981] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2141.543117] [ 2141.545302] irq event stamp: 12818807 [ 2141.548978] hardirqs last enabled at (12818817): [] console_unlock+0x106/0x120 [ 2141.557851] hardirqs last disabled at (12818826): [] console_unlock+0xeb/0x120 [ 2141.566630] softirqs last enabled at (12818784): [] __do_softirq+0x2be/0x371 [ 2141.575340] softirqs last disabled at (12818779): [] irq_exit_rcu+0xb0/0xd0 [ 2141.583878] ---[ end trace 0000000000000000 ]--- [ 2141.588490] ------------[ cut here ]------------ [ 2141.593135] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2141.604799] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2141.613495] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2141.668986] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2141.678129] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2141.685600] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2141.691132] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2141.696615] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2141.715362] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2141.720608] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2141.727757] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2141.734889] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000f2a [ 2141.742014] R10: 000000000009ee30 R11: ffffc9000030b6a8 R12: ffff8881d800c200 [ 2141.749145] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2141.756287] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2141.764375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2141.770136] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2141.777262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2141.784395] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2141.791526] Call Trace: [ 2141.793972] [ 2141.796079] ? __warn+0x85/0x180 [ 2141.799311] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.804191] ? report_bug+0xfc/0x1d0 [ 2141.807768] ? prb_read_valid+0x1b/0x20 [ 2141.811610] ? handle_bug+0x44/0x70 [ 2141.815103] ? exc_invalid_op+0x17/0x60 [ 2141.818940] ? asm_exc_invalid_op+0x1a/0x20 [ 2141.823142] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.828027] ? __list_add_valid_or_report+0x7d/0xa0 [ 2141.832904] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2141.837793] ceph_check_caps+0x999/0xae0 [ceph] [ 2141.842361] ? lock_is_held_type+0xaa/0x120 [ 2141.846557] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2141.851289] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2141.856026] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2141.860860] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2141.865232] ? lock_release+0xc5/0x260 [ 2141.869001] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2141.873710] ceph_con_process_message+0x74/0x150 [libceph] [ 2141.879219] ? calc_signature+0xdb/0x100 [libceph] [ 2141.884048] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2141.889370] ? process_scheduled_works+0x22a/0x510 [ 2141.894163] ? process_scheduled_works+0x22a/0x510 [ 2141.898970] ceph_con_workfn+0x344/0x750 [libceph] [ 2141.903773] ? process_scheduled_works+0x22a/0x510 [ 2141.908564] process_scheduled_works+0x2a4/0x510 [ 2141.913188] worker_thread+0x188/0x340 [ 2141.916938] ? rescuer_thread+0x310/0x310 [ 2141.920953] kthread+0xf5/0x130 [ 2141.924095] ? kthread_complete_and_exit+0x20/0x20 [ 2141.928890] ret_from_fork+0x31/0x40 [ 2141.932466] ? kthread_complete_and_exit+0x20/0x20 [ 2141.937259] ret_from_fork_asm+0x11/0x20 [ 2141.941193] [ 2141.943379] irq event stamp: 9825091 [ 2141.946958] hardirqs last enabled at (9825101): [] console_unlock+0x106/0x120 [ 2141.955736] hardirqs last disabled at (9825110): [] console_unlock+0xeb/0x120 [ 2141.964423] softirqs last enabled at (9825028): [] __do_softirq+0x2be/0x371 [ 2141.973027] softirqs last disabled at (9825019): [] irq_exit_rcu+0xb0/0xd0 [ 2141.981462] ---[ end trace 0000000000000000 ]--- [ 2141.988363] ------------[ cut here ]------------ [ 2141.992984] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2142.004646] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2142.013596] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2142.069085] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2142.078034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2142.085512] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2142.091017] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2142.109764] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2142.115007] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2142.122153] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2142.129299] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000f68 [ 2142.136449] R10: 000000000009ffd8 R11: ffffc90001fffa60 R12: ffff8881a1d35f58 [ 2142.143580] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2142.150722] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2142.158827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2142.164588] CR2: 00007f9bd346f000 CR3: 000000017eb7a003 CR4: 00000000003706e0 [ 2142.171713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2142.178858] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2142.185989] Call Trace: [ 2142.188441] [ 2142.190558] ? __warn+0x85/0x180 [ 2142.193808] ? __list_add_valid_or_report+0x7d/0xa0 [ 2142.198705] ? report_bug+0xfc/0x1d0 [ 2142.202302] ? prb_read_valid+0x1b/0x20 [ 2142.206157] ? handle_bug+0x44/0x70 [ 2142.209667] ? exc_invalid_op+0x17/0x60 [ 2142.213524] ? asm_exc_invalid_op+0x1a/0x20 [ 2142.217748] ? __list_add_valid_or_report+0x7d/0xa0 [ 2142.222624] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2142.227532] try_get_cap_refs+0x159/0x880 [ceph] [ 2142.232197] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2142.236756] ? wake_bit_function+0x70/0x70 [ 2142.240865] ? ceph_read_iter+0x134/0x860 [ceph] [ 2142.245496] ceph_read_iter+0x134/0x860 [ceph] [ 2142.249971] ? selinux_file_permission+0x119/0x150 [ 2142.254762] ? vfs_read+0x20f/0x300 [ 2142.258269] vfs_read+0x20f/0x300 [ 2142.261620] ksys_pread64+0x65/0xa0 [ 2142.265110] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2142.270248] do_syscall_64+0x38/0x80 [ 2142.273862] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2142.278914] RIP: 0033:0x7f48f0eeb88b [ 2142.282493] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2142.301258] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2142.308861] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2142.315990] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2142.323116] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2142.330249] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2142.337397] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2142.344562] [ 2142.346748] irq event stamp: 12819817 [ 2142.350425] hardirqs last enabled at (12819827): [] console_unlock+0x106/0x120 [ 2142.359307] hardirqs last disabled at (12819836): [] console_unlock+0xeb/0x120 [ 2142.368097] softirqs last enabled at (12819784): [] __do_softirq+0x2be/0x371 [ 2142.376788] softirqs last disabled at (12819779): [] irq_exit_rcu+0xb0/0xd0 [ 2142.385325] ---[ end trace 0000000000000000 ]--- [ 2142.394646] ------------[ cut here ]------------ [ 2142.399274] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2142.410936] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2142.419630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2142.475134] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2142.484259] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2142.491740] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2142.497259] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2142.502746] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2142.521507] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2142.526733] RAX: 0000000000000000 RBX: ffff88811e404e40 RCX: 0000000000000000 [ 2142.533869] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2142.541000] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000fa1 [ 2142.548146] R10: 00000000000a10f8 R11: ffffc9000030b828 R12: ffff88811e405590 [ 2142.555292] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2142.562424] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2142.570510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2142.576256] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2142.583389] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2142.590522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2142.597656] Call Trace: [ 2142.600142] [ 2142.602248] ? __warn+0x85/0x180 [ 2142.605482] ? __list_add_valid_or_report+0x7d/0xa0 [ 2142.610418] ? report_bug+0xfc/0x1d0 [ 2142.614003] ? prb_read_valid+0x1b/0x20 [ 2142.617860] ? handle_bug+0x44/0x70 [ 2142.621350] ? exc_invalid_op+0x17/0x60 [ 2142.625190] ? asm_exc_invalid_op+0x1a/0x20 [ 2142.629383] ? __list_add_valid_or_report+0x7d/0xa0 [ 2142.634265] ? __list_add_valid_or_report+0x7d/0xa0 [ 2142.639146] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2142.644052] ceph_add_cap+0x20b/0x620 [ceph] [ 2142.648337] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2142.653076] ? ceph_ino_compare+0x30/0x30 [ceph] [ 2142.657705] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2142.663057] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2142.667518] ceph_con_process_message+0x74/0x150 [libceph] [ 2142.673040] ? calc_signature+0xdb/0x100 [libceph] [ 2142.677845] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2142.683192] ? process_scheduled_works+0x22a/0x510 [ 2142.687984] ? process_scheduled_works+0x22a/0x510 [ 2142.692772] ceph_con_workfn+0x344/0x750 [libceph] [ 2142.697579] ? process_scheduled_works+0x22a/0x510 [ 2142.702368] process_scheduled_works+0x2a4/0x510 [ 2142.706993] worker_thread+0x188/0x340 [ 2142.710741] ? rescuer_thread+0x310/0x310 [ 2142.714753] kthread+0xf5/0x130 [ 2142.717899] ? kthread_complete_and_exit+0x20/0x20 [ 2142.722692] ret_from_fork+0x31/0x40 [ 2142.726270] ? kthread_complete_and_exit+0x20/0x20 [ 2142.731064] ret_from_fork_asm+0x11/0x20 [ 2142.734997] [ 2142.737190] irq event stamp: 9852587 [ 2142.740788] hardirqs last enabled at (9852597): [] console_unlock+0x106/0x120 [ 2142.749568] hardirqs last disabled at (9852612): [] console_unlock+0xeb/0x120 [ 2142.758261] softirqs last enabled at (9852610): [] __do_softirq+0x2be/0x371 [ 2142.766871] softirqs last disabled at (9852605): [] irq_exit_rcu+0xb0/0xd0 [ 2142.775298] ---[ end trace 0000000000000000 ]--- [ 2142.780664] ------------[ cut here ]------------ [ 2142.785289] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2142.796945] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2142.805644] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2142.861163] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2142.870288] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2142.877766] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2142.883288] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2142.888774] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2142.907520] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2142.912746] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2142.919880] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2142.927013] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100000fdc [ 2142.934152] R10: 00000000000a2300 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2142.941278] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2142.948412] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2142.956497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2142.962241] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2142.969376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2142.976507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2142.983641] Call Trace: [ 2142.986095] [ 2142.988201] ? __warn+0x85/0x180 [ 2142.991433] ? __list_add_valid_or_report+0x7d/0xa0 [ 2142.996366] ? report_bug+0xfc/0x1d0 [ 2142.999947] ? prb_read_valid+0x1b/0x20 [ 2143.003785] ? handle_bug+0x44/0x70 [ 2143.007277] ? exc_invalid_op+0x17/0x60 [ 2143.011132] ? asm_exc_invalid_op+0x1a/0x20 [ 2143.015340] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.020216] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.025112] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2143.030002] ceph_add_cap+0x20b/0x620 [ceph] [ 2143.034286] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2143.039028] ? lock_acquire+0xcb/0x2b0 [ 2143.042781] ceph_fill_trace+0x143/0xf30 [ceph] [ 2143.047344] ? __mutex_lock+0x97/0xce0 [ 2143.051094] ? __mutex_lock+0xaec/0xce0 [ 2143.054935] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2143.059477] ? lock_downgrade+0xfd/0x250 [ 2143.063408] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2143.067802] ceph_con_process_message+0x74/0x150 [libceph] [ 2143.073319] ? calc_signature+0xdb/0x100 [libceph] [ 2143.078147] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2143.083471] ? process_scheduled_works+0x22a/0x510 [ 2143.088269] ? process_scheduled_works+0x22a/0x510 [ 2143.093061] ceph_con_workfn+0x344/0x750 [libceph] [ 2143.097881] ? process_scheduled_works+0x22a/0x510 [ 2143.102672] process_scheduled_works+0x2a4/0x510 [ 2143.107312] worker_thread+0x188/0x340 [ 2143.111063] ? rescuer_thread+0x310/0x310 [ 2143.115074] kthread+0xf5/0x130 [ 2143.118219] ? kthread_complete_and_exit+0x20/0x20 [ 2143.123013] ret_from_fork+0x31/0x40 [ 2143.126590] ? kthread_complete_and_exit+0x20/0x20 [ 2143.131383] ret_from_fork_asm+0x11/0x20 [ 2143.135316] [ 2143.137503] irq event stamp: 9856471 [ 2143.141083] hardirqs last enabled at (9856481): [] console_unlock+0x106/0x120 [ 2143.149881] hardirqs last disabled at (9856496): [] console_unlock+0xeb/0x120 [ 2143.158573] softirqs last enabled at (9856494): [] __do_softirq+0x2be/0x371 [ 2143.167179] softirqs last disabled at (9856489): [] irq_exit_rcu+0xb0/0xd0 [ 2143.175611] ---[ end trace 0000000000000000 ]--- [ 2143.183979] ------------[ cut here ]------------ [ 2143.188595] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2143.200240] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2143.208936] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2143.264434] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2143.273565] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2143.281045] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2143.286546] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2143.292031] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2143.310778] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2143.316004] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2143.323154] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2143.330286] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000101b [ 2143.337411] R10: 00000000000a3480 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2143.344546] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2143.351677] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2143.359763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2143.365511] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2143.372642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2143.379775] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2143.386925] Call Trace: [ 2143.389377] [ 2143.391484] ? __warn+0x85/0x180 [ 2143.394793] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.399677] ? report_bug+0xfc/0x1d0 [ 2143.403255] ? prb_read_valid+0x1b/0x20 [ 2143.407099] ? handle_bug+0x44/0x70 [ 2143.410587] ? exc_invalid_op+0x17/0x60 [ 2143.414425] ? asm_exc_invalid_op+0x1a/0x20 [ 2143.418617] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.423493] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.428369] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2143.433288] ceph_add_cap+0x20b/0x620 [ceph] [ 2143.437599] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2143.442339] ? lock_acquire+0xcb/0x2b0 [ 2143.446091] ceph_fill_trace+0x143/0xf30 [ceph] [ 2143.450628] ? __mutex_lock+0x97/0xce0 [ 2143.454378] ? __mutex_lock+0xaec/0xce0 [ 2143.458219] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2143.462778] ? lock_downgrade+0xfd/0x250 [ 2143.466708] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2143.471106] ceph_con_process_message+0x74/0x150 [libceph] [ 2143.476602] ? calc_signature+0xdb/0x100 [libceph] [ 2143.481404] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2143.486754] ? process_scheduled_works+0x22a/0x510 [ 2143.491546] ? process_scheduled_works+0x22a/0x510 [ 2143.496335] ceph_con_workfn+0x344/0x750 [libceph] [ 2143.501155] ? process_scheduled_works+0x22a/0x510 [ 2143.505965] process_scheduled_works+0x2a4/0x510 [ 2143.510588] worker_thread+0x188/0x340 [ 2143.514339] ? rescuer_thread+0x310/0x310 [ 2143.518348] kthread+0xf5/0x130 [ 2143.521497] ? kthread_complete_and_exit+0x20/0x20 [ 2143.526289] ret_from_fork+0x31/0x40 [ 2143.529886] ? kthread_complete_and_exit+0x20/0x20 [ 2143.534678] ret_from_fork_asm+0x11/0x20 [ 2143.538629] [ 2143.540813] irq event stamp: 9871553 [ 2143.544393] hardirqs last enabled at (9871563): [] console_unlock+0x106/0x120 [ 2143.553189] hardirqs last disabled at (9871572): [] console_unlock+0xeb/0x120 [ 2143.561904] softirqs last enabled at (9871500): [] __do_softirq+0x2be/0x371 [ 2143.570504] softirqs last disabled at (9871495): [] irq_exit_rcu+0xb0/0xd0 [ 2143.578940] ---[ end trace 0000000000000000 ]--- [ 2143.608875] ceph: [cde022fc-3e6c-431e-bc4a-204d5c5d32a6 4445] renewed_caps: mds0 caps renewed [ 2143.617779] ------------[ cut here ]------------ [ 2143.622420] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2143.634073] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2143.643036] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2143.698564] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2143.707524] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2143.715003] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2143.720488] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2143.739235] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2143.744461] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2143.751593] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2143.758726] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000105b [ 2143.765863] R10: 00000000000a4700 R11: ffffc90001fffa60 R12: ffff8881a1d35f58 [ 2143.773000] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2143.780134] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2143.788220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2143.793967] CR2: 00007f9bd346f000 CR3: 000000017eb7a003 CR4: 00000000003706e0 [ 2143.801097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2143.808235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2143.815374] Call Trace: [ 2143.817826] [ 2143.819932] ? __warn+0x85/0x180 [ 2143.823166] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.828045] ? report_bug+0xfc/0x1d0 [ 2143.831626] ? prb_read_valid+0x1b/0x20 [ 2143.835477] ? handle_bug+0x44/0x70 [ 2143.838973] ? exc_invalid_op+0x17/0x60 [ 2143.842814] ? asm_exc_invalid_op+0x1a/0x20 [ 2143.847017] ? __list_add_valid_or_report+0x7d/0xa0 [ 2143.851907] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2143.856815] try_get_cap_refs+0x433/0x880 [ceph] [ 2143.861466] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2143.866025] ? wake_bit_function+0x70/0x70 [ 2143.870137] ? ceph_read_iter+0x134/0x860 [ceph] [ 2143.874775] ceph_read_iter+0x134/0x860 [ceph] [ 2143.879249] ? selinux_file_permission+0x119/0x150 [ 2143.884045] ? vfs_read+0x20f/0x300 [ 2143.887541] vfs_read+0x20f/0x300 [ 2143.890876] ksys_pread64+0x65/0xa0 [ 2143.894374] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2143.899521] do_syscall_64+0x38/0x80 [ 2143.903107] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2143.908159] RIP: 0033:0x7f48f0eeb88b [ 2143.911740] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2143.930485] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2143.938050] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2143.945185] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2143.952315] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2143.959451] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2143.966583] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2143.973734] [ 2143.975925] irq event stamp: 12820841 [ 2143.979592] hardirqs last enabled at (12820851): [] console_unlock+0x106/0x120 [ 2143.988458] hardirqs last disabled at (12820860): [] console_unlock+0xeb/0x120 [ 2143.997236] softirqs last enabled at (12820828): [] __do_softirq+0x2be/0x371 [ 2144.005929] softirqs last disabled at (12820823): [] irq_exit_rcu+0xb0/0xd0 [ 2144.014448] ---[ end trace 0000000000000000 ]--- [ 2144.019069] ------------[ cut here ]------------ [ 2144.023686] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2144.035332] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2144.044293] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2144.099758] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2144.108714] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2144.116197] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2144.121684] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2144.140429] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2144.145655] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2144.152789] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2144.159922] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001094 [ 2144.167053] R10: 00000000000a5880 R11: ffffc90001fef9c8 R12: ffff888158a68750 [ 2144.174191] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2144.181318] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2144.189405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2144.195150] CR2: 00007f9bd4c63000 CR3: 0000000108c72003 CR4: 00000000003706e0 [ 2144.202286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2144.209417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2144.216549] Call Trace: [ 2144.219001] [ 2144.221111] ? __warn+0x85/0x180 [ 2144.224341] ? __list_add_valid_or_report+0x7d/0xa0 [ 2144.229224] ? report_bug+0xfc/0x1d0 [ 2144.232809] ? prb_read_valid+0x1b/0x20 [ 2144.236654] ? handle_bug+0x44/0x70 [ 2144.240153] ? exc_invalid_op+0x17/0x60 [ 2144.243992] ? asm_exc_invalid_op+0x1a/0x20 [ 2144.248183] ? __list_add_valid_or_report+0x7d/0xa0 [ 2144.253066] ? __list_add_valid_or_report+0x7d/0xa0 [ 2144.257942] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2144.262844] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2144.267571] lookup_fast+0x6d/0xf0 [ 2144.270982] walk_component+0x26/0x160 [ 2144.274742] link_path_walk+0x24d/0x380 [ 2144.278582] ? path_lookupat.isra.40+0x22/0x120 [ 2144.283115] path_lookupat.isra.40+0x6b/0x120 [ 2144.287476] filename_lookup+0xd4/0x190 [ 2144.291324] ? lock_acquire+0xcb/0x2b0 [ 2144.295081] ? __might_fault+0x45/0x80 [ 2144.298833] ? find_held_lock+0x2d/0x90 [ 2144.302673] ? __might_fault+0x45/0x80 [ 2144.306433] vfs_statx+0x89/0x150 [ 2144.309762] vfs_fstatat+0x5a/0xc0 [ 2144.313172] __do_sys_newstat+0x30/0x70 [ 2144.317009] ? lock_release+0xc5/0x260 [ 2144.320764] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2144.325647] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2144.330788] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2144.335930] do_syscall_64+0x38/0x80 [ 2144.339507] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2144.344561] RIP: 0033:0x7f48f0eeca79 [ 2144.348138] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2144.366887] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2144.374464] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2144.381604] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2144.388733] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2144.395866] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2144.403005] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2144.410143] [ 2144.412333] irq event stamp: 12788371 [ 2144.415998] hardirqs last enabled at (12788381): [] console_unlock+0x106/0x120 [ 2144.424866] hardirqs last disabled at (12788390): [] console_unlock+0xeb/0x120 [ 2144.433652] softirqs last enabled at (12788306): [] __do_softirq+0x2be/0x371 [ 2144.442348] softirqs last disabled at (12788411): [] irq_exit_rcu+0xb0/0xd0 [ 2144.450868] ---[ end trace 0000000000000000 ]--- [ 2144.455492] ------------[ cut here ]------------ [ 2144.460112] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2144.471758] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2144.480713] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2144.536182] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2144.545139] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2144.552614] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2144.558100] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2144.576846] RSP: 0018:ffffc90001fffb28 EFLAGS: 00010286 [ 2144.582071] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2144.589205] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2144.596339] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000010d6 [ 2144.603469] R10: 00000000000a6a38 R11: ffffc90001fff9b8 R12: ffff8881a1d35f58 [ 2144.610604] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2144.617735] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2144.625820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2144.631559] CR2: 00007f9bd3a7b000 CR3: 000000017eb7a001 CR4: 00000000003706e0 [ 2144.638691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2144.645824] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2144.652953] Call Trace: [ 2144.655403] [ 2144.657510] ? __warn+0x85/0x180 [ 2144.660742] ? __list_add_valid_or_report+0x7d/0xa0 [ 2144.665620] ? report_bug+0xfc/0x1d0 [ 2144.669200] ? prb_read_valid+0x1b/0x20 [ 2144.673040] ? handle_bug+0x44/0x70 [ 2144.676531] ? exc_invalid_op+0x17/0x60 [ 2144.680371] ? asm_exc_invalid_op+0x1a/0x20 [ 2144.684561] ? __list_add_valid_or_report+0x7d/0xa0 [ 2144.689437] ? __list_add_valid_or_report+0x7d/0xa0 [ 2144.694317] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2144.699206] ceph_check_caps+0x999/0xae0 [ceph] [ 2144.703748] ? lock_release+0xc5/0x260 [ 2144.707503] ? __lock_acquire+0xbfa/0x1e20 [ 2144.711599] ? __lock_acquire+0xbfa/0x1e20 [ 2144.715699] ? lock_acquire+0xcb/0x2b0 [ 2144.719449] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2144.724424] ? find_held_lock+0x2d/0x90 [ 2144.728264] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2144.733327] ? lock_release+0xc5/0x260 [ 2144.737081] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2144.741970] ceph_read_iter+0x26e/0x860 [ceph] [ 2144.746426] ? selinux_file_permission+0x119/0x150 [ 2144.751222] ? vfs_read+0x20f/0x300 [ 2144.754717] vfs_read+0x20f/0x300 [ 2144.758041] ksys_pread64+0x65/0xa0 [ 2144.761527] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2144.766667] do_syscall_64+0x38/0x80 [ 2144.770247] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2144.775298] RIP: 0033:0x7f48f0eeb88b [ 2144.778881] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2144.797625] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2144.805189] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2144.812325] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2144.819455] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2144.826589] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2144.833721] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2144.840860] [ 2144.843048] irq event stamp: 12822827 [ 2144.846712] hardirqs last enabled at (12822837): [] console_unlock+0x106/0x120 [ 2144.855579] hardirqs last disabled at (12822846): [] console_unlock+0xeb/0x120 [ 2144.864357] softirqs last enabled at (12822678): [] __do_softirq+0x2be/0x371 [ 2144.873052] softirqs last disabled at (12822667): [] irq_exit_rcu+0xb0/0xd0 [ 2144.881570] ---[ end trace 0000000000000000 ]--- [ 2144.886182] ------------[ cut here ]------------ [ 2144.890801] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2144.902451] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2144.911406] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2144.966888] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2144.975846] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2144.983322] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2144.988809] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2145.007552] RSP: 0018:ffffc90001fefac0 EFLAGS: 00010282 [ 2145.012780] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2145.019913] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2145.027045] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001116 [ 2145.034181] R10: 00000000000a7d08 R11: ffffc90001fef950 R12: ffff888158a68750 [ 2145.041312] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2145.048445] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2145.056531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2145.062275] CR2: 00007f9bd4c63000 CR3: 0000000108c72003 CR4: 00000000003706e0 [ 2145.069411] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2145.076542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2145.083676] Call Trace: [ 2145.086128] [ 2145.088235] ? __warn+0x85/0x180 [ 2145.091465] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.096348] ? report_bug+0xfc/0x1d0 [ 2145.099927] ? prb_read_valid+0x1b/0x20 [ 2145.103771] ? handle_bug+0x44/0x70 [ 2145.107266] ? exc_invalid_op+0x17/0x60 [ 2145.111108] ? asm_exc_invalid_op+0x1a/0x20 [ 2145.115300] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.120189] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2145.125092] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2145.130345] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2145.135152] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2145.139880] lookup_fast+0x6d/0xf0 [ 2145.143294] walk_component+0x26/0x160 [ 2145.147051] link_path_walk+0x24d/0x380 [ 2145.150897] ? path_lookupat.isra.40+0x22/0x120 [ 2145.155431] path_lookupat.isra.40+0x6b/0x120 [ 2145.159792] filename_lookup+0xd4/0x190 [ 2145.163645] ? lock_acquire+0xcb/0x2b0 [ 2145.167399] ? __might_fault+0x45/0x80 [ 2145.171158] ? find_held_lock+0x2d/0x90 [ 2145.175001] ? __might_fault+0x45/0x80 [ 2145.178764] vfs_statx+0x89/0x150 [ 2145.182095] vfs_fstatat+0x5a/0xc0 [ 2145.185504] __do_sys_newstat+0x30/0x70 [ 2145.189342] ? lock_release+0xc5/0x260 [ 2145.193097] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2145.197983] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2145.203121] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2145.208264] do_syscall_64+0x38/0x80 [ 2145.211850] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2145.216903] RIP: 0033:0x7f48f0eeca79 [ 2145.220480] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2145.239230] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2145.246800] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2145.253935] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2145.261066] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2145.268200] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2145.275333] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2145.282477] [ 2145.284667] irq event stamp: 12790481 [ 2145.288335] hardirqs last enabled at (12790491): [] console_unlock+0x106/0x120 [ 2145.297206] hardirqs last disabled at (12790500): [] console_unlock+0xeb/0x120 [ 2145.305991] softirqs last enabled at (12790514): [] __do_softirq+0x2be/0x371 [ 2145.314687] softirqs last disabled at (12790509): [] irq_exit_rcu+0xb0/0xd0 [ 2145.323210] ---[ end trace 0000000000000000 ]--- [ 2145.327827] ------------[ cut here ]------------ [ 2145.332464] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2145.344112] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2145.353063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2145.408533] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2145.417483] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2145.424958] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2145.430442] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2145.449188] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2145.454413] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2145.461536] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2145.468671] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001159 [ 2145.475803] R10: 00000000000a9010 R11: ffffc90001fffa60 R12: ffff8881a1d35f58 [ 2145.482927] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2145.490059] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2145.498149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2145.503896] CR2: 00007f9bd3a7b000 CR3: 000000017eb7a001 CR4: 00000000003706e0 [ 2145.511026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2145.518159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2145.525293] Call Trace: [ 2145.527743] [ 2145.529841] ? __warn+0x85/0x180 [ 2145.533075] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.537953] ? report_bug+0xfc/0x1d0 [ 2145.541531] ? prb_read_valid+0x1b/0x20 [ 2145.545373] ? handle_bug+0x44/0x70 [ 2145.548864] ? exc_invalid_op+0x17/0x60 [ 2145.552704] ? asm_exc_invalid_op+0x1a/0x20 [ 2145.556898] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.561781] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2145.566669] try_get_cap_refs+0x433/0x880 [ceph] [ 2145.571300] __ceph_get_caps+0xf8/0x760 [ceph] [ 2145.575768] ? ceph_read_iter+0x134/0x860 [ceph] [ 2145.580396] ceph_read_iter+0x134/0x860 [ceph] [ 2145.584855] ? selinux_file_permission+0x119/0x150 [ 2145.589644] ? vfs_read+0x20f/0x300 [ 2145.593136] vfs_read+0x20f/0x300 [ 2145.596459] ksys_pread64+0x65/0xa0 [ 2145.599946] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2145.605087] do_syscall_64+0x38/0x80 [ 2145.608667] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2145.613720] RIP: 0033:0x7f48f0eeb88b [ 2145.617298] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2145.636043] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2145.643609] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2145.650744] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2145.657876] RBP: 0000000000002a0f R08: 0000000000000063 R09: 0000000000000003 [ 2145.665007] R10: 000000000001f000 R11: 0000000000000246 R12: 0000000000001000 [ 2145.672131] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2145.679262] [ 2145.681452] irq event stamp: 12824779 [ 2145.685115] hardirqs last enabled at (12824789): [] console_unlock+0x106/0x120 [ 2145.693981] hardirqs last disabled at (12824798): [] console_unlock+0xeb/0x120 [ 2145.702761] softirqs last enabled at (12824776): [] __do_softirq+0x2be/0x371 [ 2145.711454] softirqs last disabled at (12824771): [] irq_exit_rcu+0xb0/0xd0 [ 2145.719972] ---[ end trace 0000000000000000 ]--- [ 2145.724584] ------------[ cut here ]------------ [ 2145.729204] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2145.740849] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2145.749808] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2145.805279] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2145.814240] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2145.821723] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2145.827209] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2145.845971] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2145.851198] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2145.858331] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2145.865464] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001191 [ 2145.872597] R10: 00000000000aa150 R11: ffffc90001fd79c8 R12: ffff888158a68750 [ 2145.879731] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2145.886863] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2145.894953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2145.900693] CR2: 00007fb39af97000 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2145.907819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2145.914952] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2145.922085] Call Trace: [ 2145.924536] [ 2145.926635] ? __warn+0x85/0x180 [ 2145.929867] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.934748] ? report_bug+0xfc/0x1d0 [ 2145.938325] ? prb_read_valid+0x1b/0x20 [ 2145.942168] ? handle_bug+0x44/0x70 [ 2145.945658] ? exc_invalid_op+0x17/0x60 [ 2145.949498] ? asm_exc_invalid_op+0x1a/0x20 [ 2145.953689] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.958565] ? __list_add_valid_or_report+0x7d/0xa0 [ 2145.963442] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2145.968334] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2145.973055] lookup_fast+0x6d/0xf0 [ 2145.976462] walk_component+0x26/0x160 [ 2145.980214] link_path_walk+0x24d/0x380 [ 2145.984054] ? path_lookupat.isra.40+0x22/0x120 [ 2145.988592] path_lookupat.isra.40+0x6b/0x120 [ 2145.992945] filename_lookup+0xd4/0x190 [ 2145.996786] ? lock_acquire+0xcb/0x2b0 [ 2146.000536] ? __might_fault+0x45/0x80 [ 2146.004289] ? find_held_lock+0x2d/0x90 [ 2146.008129] ? __might_fault+0x45/0x80 [ 2146.011883] vfs_statx+0x89/0x150 [ 2146.015204] vfs_fstatat+0x5a/0xc0 [ 2146.018608] __do_sys_newstat+0x30/0x70 [ 2146.022446] ? lock_release+0xc5/0x260 [ 2146.026202] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2146.031078] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2146.036217] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2146.041359] do_syscall_64+0x38/0x80 [ 2146.044936] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2146.049990] RIP: 0033:0x7f48f0eeca79 [ 2146.053568] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2146.072318] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2146.079892] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2146.087023] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2146.094155] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2146.101287] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2146.108419] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2146.115550] [ 2146.117738] irq event stamp: 12922595 [ 2146.121409] hardirqs last enabled at (12922615): [] console_unlock+0x106/0x120 [ 2146.130271] hardirqs last disabled at (12922634): [] console_unlock+0xeb/0x120 [ 2146.139051] softirqs last enabled at (12922648): [] __do_softirq+0x2be/0x371 [ 2146.147744] softirqs last disabled at (12922661): [] irq_exit_rcu+0xb0/0xd0 [ 2146.156265] ---[ end trace 0000000000000000 ]--- [ 2146.160883] ------------[ cut here ]------------ [ 2146.165501] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2146.177147] WARNING: CPU: 7 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2146.186102] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2146.241572] CPU: 7 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2146.250522] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2146.258000] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2146.263486] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2146.282234] RSP: 0018:ffffc90001fffb28 EFLAGS: 00010286 [ 2146.287460] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2146.294591] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2146.301715] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000011d3 [ 2146.308843] R10: 00000000000ab800 R11: ffffc90001fff9b8 R12: ffff8881a1d35f58 [ 2146.315975] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2146.323107] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2146.331192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2146.336931] CR2: 00007f9bd3913000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2146.344064] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2146.351196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2146.358328] Call Trace: [ 2146.360772] [ 2146.362869] ? __warn+0x85/0x180 [ 2146.366103] ? __list_add_valid_or_report+0x7d/0xa0 [ 2146.370984] ? report_bug+0xfc/0x1d0 [ 2146.374560] ? prb_read_valid+0x1b/0x20 [ 2146.378402] ? handle_bug+0x44/0x70 [ 2146.381895] ? exc_invalid_op+0x17/0x60 [ 2146.385735] ? asm_exc_invalid_op+0x1a/0x20 [ 2146.389925] ? __list_add_valid_or_report+0x7d/0xa0 [ 2146.394800] ? __list_add_valid_or_report+0x7d/0xa0 [ 2146.399679] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2146.404570] ceph_check_caps+0x999/0xae0 [ceph] [ 2146.409109] ? lock_release+0xc5/0x260 [ 2146.412864] ? __lock_acquire+0xbfa/0x1e20 [ 2146.416962] ? __lock_acquire+0xbfa/0x1e20 [ 2146.421060] ? lock_acquire+0xcb/0x2b0 [ 2146.424812] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2146.429786] ? find_held_lock+0x2d/0x90 [ 2146.433625] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2146.438689] ? lock_release+0xc5/0x260 [ 2146.442444] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2146.447333] ceph_read_iter+0x26e/0x860 [ceph] [ 2146.451788] ? selinux_file_permission+0x119/0x150 [ 2146.456578] ? vfs_read+0x20f/0x300 [ 2146.460068] vfs_read+0x20f/0x300 [ 2146.463393] ksys_pread64+0x65/0xa0 [ 2146.466880] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2146.472023] do_syscall_64+0x38/0x80 [ 2146.475600] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2146.480653] RIP: 0033:0x7f48f0eeb88b [ 2146.484231] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2146.502981] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2146.510544] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2146.517677] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2146.524809] RBP: 0000000000002a0f R08: 0000000000000063 R09: 0000000000000003 [ 2146.531934] R10: 000000000001f000 R11: 0000000000000246 R12: 0000000000001000 [ 2146.539066] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2146.546205] [ 2146.548392] irq event stamp: 12826851 [ 2146.552059] hardirqs last enabled at (12826861): [] console_unlock+0x106/0x120 [ 2146.560923] hardirqs last disabled at (12826870): [] console_unlock+0xeb/0x120 [ 2146.569706] softirqs last enabled at (12826884): [] __do_softirq+0x2be/0x371 [ 2146.578396] softirqs last disabled at (12826879): [] irq_exit_rcu+0xb0/0xd0 [ 2146.586916] ---[ end trace 0000000000000000 ]--- [ 2146.591540] ------------[ cut here ]------------ [ 2146.596154] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2146.607793] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2146.616747] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2146.672212] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2146.681158] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2146.688629] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2146.694116] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2146.712861] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2146.718089] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2146.725221] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2146.732345] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001213 [ 2146.739479] R10: 00000000000acbb0 R11: ffffc90001fd7950 R12: ffff888158a68750 [ 2146.746610] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2146.753734] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2146.761812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2146.767549] CR2: 00007fb38e6a4000 CR3: 0000000120c1e005 CR4: 00000000003706f0 [ 2146.774674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2146.781798] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2146.788931] Call Trace: [ 2146.791385] [ 2146.793492] ? __warn+0x85/0x180 [ 2146.796723] ? __list_add_valid_or_report+0x7d/0xa0 [ 2146.801603] ? report_bug+0xfc/0x1d0 [ 2146.805183] ? prb_read_valid+0x1b/0x20 [ 2146.809022] ? handle_bug+0x44/0x70 [ 2146.812513] ? exc_invalid_op+0x17/0x60 [ 2146.816353] ? asm_exc_invalid_op+0x1a/0x20 [ 2146.820544] ? __list_add_valid_or_report+0x7d/0xa0 [ 2146.825421] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2146.830309] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2146.835545] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2146.840344] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2146.845063] lookup_fast+0x6d/0xf0 [ 2146.848476] walk_component+0x26/0x160 [ 2146.852241] link_path_walk+0x24d/0x380 [ 2146.856083] ? path_lookupat.isra.40+0x22/0x120 [ 2146.860617] path_lookupat.isra.40+0x6b/0x120 [ 2146.864974] filename_lookup+0xd4/0x190 [ 2146.868820] ? lock_acquire+0xcb/0x2b0 [ 2146.872567] ? __might_fault+0x45/0x80 [ 2146.876319] ? find_held_lock+0x2d/0x90 [ 2146.880158] ? __might_fault+0x45/0x80 [ 2146.883919] vfs_statx+0x89/0x150 [ 2146.887241] vfs_fstatat+0x5a/0xc0 [ 2146.890647] __do_sys_newstat+0x30/0x70 [ 2146.894485] ? lock_release+0xc5/0x260 [ 2146.898239] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2146.903115] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2146.908257] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2146.913397] do_syscall_64+0x38/0x80 [ 2146.916975] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2146.922038] RIP: 0033:0x7f48f0eeca79 [ 2146.925623] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2146.944374] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2146.951936] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2146.959060] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2146.966185] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2146.973318] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2146.980455] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2146.987589] [ 2146.989777] irq event stamp: 12924765 [ 2146.993441] hardirqs last enabled at (12924775): [] console_unlock+0x106/0x120 [ 2147.002308] hardirqs last disabled at (12924784): [] console_unlock+0xeb/0x120 [ 2147.011088] softirqs last enabled at (12924798): [] __do_softirq+0x2be/0x371 [ 2147.019781] softirqs last disabled at (12924793): [] irq_exit_rcu+0xb0/0xd0 [ 2147.028300] ---[ end trace 0000000000000000 ]--- [ 2147.032923] ------------[ cut here ]------------ [ 2147.037541] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2147.049190] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2147.058144] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2147.113624] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2147.122583] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2147.130060] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2147.135548] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2147.154301] RSP: 0018:ffffc90001fffbe0 EFLAGS: 00010286 [ 2147.159529] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2147.166658] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2147.173799] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001256 [ 2147.180941] R10: 00000000000adeb0 R11: ffffc90001fffa70 R12: ffff8881a1d35f58 [ 2147.188079] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2147.195216] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2147.203304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2147.209049] CR2: 00007f9bd4c63000 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2147.216184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2147.223317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2147.230448] Call Trace: [ 2147.232901] [ 2147.235009] ? __warn+0x85/0x180 [ 2147.238239] ? __list_add_valid_or_report+0x7d/0xa0 [ 2147.243123] ? report_bug+0xfc/0x1d0 [ 2147.246708] ? prb_read_valid+0x1b/0x20 [ 2147.250552] ? handle_bug+0x44/0x70 [ 2147.254047] ? exc_invalid_op+0x17/0x60 [ 2147.257889] ? asm_exc_invalid_op+0x1a/0x20 [ 2147.262091] ? __list_add_valid_or_report+0x7d/0xa0 [ 2147.266973] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2147.271869] try_get_cap_refs+0x3c9/0x880 [ceph] [ 2147.276515] ? _raw_spin_unlock+0x23/0x40 [ 2147.280542] __ceph_get_caps+0xf8/0x760 [ceph] [ 2147.285009] ? __vfs_getxattr+0x8d/0xb0 [ 2147.288859] ? dentry_needs_remove_privs+0x31/0x50 [ 2147.293658] ? __file_remove_privs+0x89/0x110 [ 2147.298024] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2147.302751] ? lock_release+0xc5/0x260 [ 2147.306514] ? ceph_write_iter+0x260/0x770 [ceph] [ 2147.311246] ceph_write_iter+0x260/0x770 [ceph] [ 2147.315798] ? lock_is_held_type+0xaa/0x120 [ 2147.319994] vfs_write+0x324/0x4f0 [ 2147.323409] ksys_pwrite64+0x65/0xa0 [ 2147.326988] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2147.332129] do_syscall_64+0x38/0x80 [ 2147.335709] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2147.340768] RIP: 0033:0x7f48f0eeb93b [ 2147.344347] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2147.363092] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2147.370661] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2147.377795] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2147.384930] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2147.392066] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2147.399200] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2147.406344] [ 2147.408534] irq event stamp: 12828929 [ 2147.412200] hardirqs last enabled at (12828939): [] console_unlock+0x106/0x120 [ 2147.421067] hardirqs last disabled at (12828948): [] console_unlock+0xeb/0x120 [ 2147.429845] softirqs last enabled at (12828876): [] __do_softirq+0x2be/0x371 [ 2147.438541] softirqs last disabled at (12828871): [] irq_exit_rcu+0xb0/0xd0 [ 2147.447069] ---[ end trace 0000000000000000 ]--- [ 2147.451741] ------------[ cut here ]------------ [ 2147.456449] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2147.468119] WARNING: CPU: 2 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2147.477093] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2147.532596] CPU: 2 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2147.541563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2147.549054] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2147.554555] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2147.573309] RSP: 0018:ffffc90001fffd18 EFLAGS: 00010286 [ 2147.578545] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2147.585685] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2147.592828] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001293 [ 2147.599974] R10: 00000000000af0d0 R11: ffffc90001fffba8 R12: ffff8881a1d35f58 [ 2147.607119] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2147.614266] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2147.622364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2147.628118] CR2: 00007f9bd3aa7008 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2147.635264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2147.642411] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2147.649553] Call Trace: [ 2147.652013] [ 2147.654130] ? __warn+0x85/0x180 [ 2147.657379] ? __list_add_valid_or_report+0x7d/0xa0 [ 2147.662276] ? report_bug+0xfc/0x1d0 [ 2147.665875] ? prb_read_valid+0x1b/0x20 [ 2147.669730] ? handle_bug+0x44/0x70 [ 2147.673246] ? exc_invalid_op+0x17/0x60 [ 2147.677097] ? asm_exc_invalid_op+0x1a/0x20 [ 2147.681308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2147.686205] ? __list_add_valid_or_report+0x7d/0xa0 [ 2147.691100] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2147.696012] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2147.701168] ceph_write_iter+0x5aa/0x770 [ceph] [ 2147.705727] ? lock_is_held_type+0xaa/0x120 [ 2147.709919] vfs_write+0x324/0x4f0 [ 2147.713337] ksys_pwrite64+0x65/0xa0 [ 2147.716916] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2147.722060] do_syscall_64+0x38/0x80 [ 2147.725644] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2147.730699] RIP: 0033:0x7f48f0eeb93b [ 2147.734276] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2147.753024] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2147.760597] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2147.767731] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2147.774862] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2147.781995] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2147.789129] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2147.796272] [ 2147.798462] irq event stamp: 12832281 [ 2147.802128] hardirqs last enabled at (12832291): [] console_unlock+0x106/0x120 [ 2147.811000] hardirqs last disabled at (12832300): [] console_unlock+0xeb/0x120 [ 2147.819787] softirqs last enabled at (12832316): [] __do_softirq+0x2be/0x371 [ 2147.828486] softirqs last disabled at (12832327): [] irq_exit_rcu+0xb0/0xd0 [ 2147.837005] ---[ end trace 0000000000000000 ]--- [ 2147.841680] ------------[ cut here ]------------ [ 2147.846343] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2147.857982] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2147.866957] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2147.922425] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2147.931397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2147.938911] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2147.944394] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2147.963158] RSP: 0018:ffffc90001fffbe0 EFLAGS: 00010286 [ 2147.968381] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2147.975523] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2147.982665] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000012c9 [ 2147.989815] R10: 00000000000b0068 R11: ffffc90001fffa70 R12: ffff8881a1d35f58 [ 2147.996966] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2148.004098] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2148.012184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.017930] CR2: 00007fb38e6a4000 CR3: 000000017eb7a003 CR4: 00000000003706f0 [ 2148.025073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2148.032204] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2148.039329] Call Trace: [ 2148.041790] [ 2148.043906] ? __warn+0x85/0x180 [ 2148.047157] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.052036] ? report_bug+0xfc/0x1d0 [ 2148.055613] ? prb_read_valid+0x1b/0x20 [ 2148.059464] ? handle_bug+0x44/0x70 [ 2148.062971] ? exc_invalid_op+0x17/0x60 [ 2148.066813] ? asm_exc_invalid_op+0x1a/0x20 [ 2148.071019] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.075924] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2148.080824] try_get_cap_refs+0x433/0x880 [ceph] [ 2148.085474] ? _raw_spin_unlock+0x23/0x40 [ 2148.089523] __ceph_get_caps+0xf8/0x760 [ceph] [ 2148.093974] ? __vfs_getxattr+0x8d/0xb0 [ 2148.097815] ? dentry_needs_remove_privs+0x31/0x50 [ 2148.102624] ? __file_remove_privs+0x89/0x110 [ 2148.107000] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2148.111715] ? lock_release+0xc5/0x260 [ 2148.115505] ? ceph_write_iter+0x260/0x770 [ceph] [ 2148.120217] ceph_write_iter+0x260/0x770 [ceph] [ 2148.124778] ? lock_is_held_type+0xaa/0x120 [ 2148.128970] vfs_write+0x324/0x4f0 [ 2148.132399] ksys_pwrite64+0x65/0xa0 [ 2148.135972] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2148.141112] do_syscall_64+0x38/0x80 [ 2148.144693] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2148.149761] RIP: 0033:0x7f48f0eeb93b [ 2148.153358] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2148.172122] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2148.179688] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2148.186834] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2148.193981] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2148.201114] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2148.208247] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2148.215418] [ 2148.217605] irq event stamp: 12833661 [ 2148.221283] hardirqs last enabled at (12833677): [] console_unlock+0x106/0x120 [ 2148.230162] hardirqs last disabled at (12833686): [] console_unlock+0xeb/0x120 [ 2148.238960] softirqs last enabled at (12833674): [] __do_softirq+0x2be/0x371 [ 2148.247655] softirqs last disabled at (12833705): [] irq_exit_rcu+0xb0/0xd0 [ 2148.256191] ---[ end trace 0000000000000000 ]--- [ 2148.260875] ------------[ cut here ]------------ [ 2148.265520] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2148.277162] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2148.286118] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2148.341605] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2148.350579] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2148.358053] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2148.363536] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2148.382301] RSP: 0018:ffffc90001fffd18 EFLAGS: 00010286 [ 2148.387547] RAX: 0000000000000000 RBX: ffff8881a1d35808 RCX: 0000000000000000 [ 2148.394696] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2148.401844] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001306 [ 2148.408998] R10: 00000000000b1288 R11: ffffc90001fffba8 R12: ffff8881a1d35f58 [ 2148.416129] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2148.423261] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2148.431387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.437135] CR2: 00007fb38e6c0010 CR3: 000000017eb7a006 CR4: 00000000003706e0 [ 2148.444268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2148.451434] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2148.458562] Call Trace: [ 2148.461032] [ 2148.463138] ? __warn+0x85/0x180 [ 2148.466370] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.471266] ? report_bug+0xfc/0x1d0 [ 2148.474865] ? prb_read_valid+0x1b/0x20 [ 2148.478746] ? handle_bug+0x44/0x70 [ 2148.482270] ? exc_invalid_op+0x17/0x60 [ 2148.486104] ? asm_exc_invalid_op+0x1a/0x20 [ 2148.490310] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.495187] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.500067] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2148.504973] __ceph_mark_dirty_caps+0x192/0x330 [ceph] [ 2148.510123] ceph_write_iter+0x5aa/0x770 [ceph] [ 2148.514682] ? lock_is_held_type+0xaa/0x120 [ 2148.518867] vfs_write+0x324/0x4f0 [ 2148.522310] ksys_pwrite64+0x65/0xa0 [ 2148.525905] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2148.531044] do_syscall_64+0x38/0x80 [ 2148.534621] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2148.539691] RIP: 0033:0x7f48f0eeb93b [ 2148.543287] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2148.562053] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2148.569618] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2148.576790] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2148.583919] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2148.591067] R10: 0000000000010000 R11: 0000000000000246 R12: 00007f48f183e368 [ 2148.598200] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2148.605357] [ 2148.607543] irq event stamp: 12835015 [ 2148.611217] hardirqs last enabled at (12835025): [] console_unlock+0x106/0x120 [ 2148.620083] hardirqs last disabled at (12835034): [] console_unlock+0xeb/0x120 [ 2148.628862] softirqs last enabled at (12835002): [] __do_softirq+0x2be/0x371 [ 2148.637574] softirqs last disabled at (12834997): [] irq_exit_rcu+0xb0/0xd0 [ 2148.646109] ---[ end trace 0000000000000000 ]--- [ 2148.650781] ------------[ cut here ]------------ [ 2148.655405] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2148.667047] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2148.676002] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2148.731464] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2148.740436] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2148.747928] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2148.753430] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2148.772178] RSP: 0018:ffffc90001fffb38 EFLAGS: 00010282 [ 2148.777404] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2148.784553] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2148.791696] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000133c [ 2148.798847] R10: 00000000000b21b8 R11: ffffc90001fff9c8 R12: ffff888158a68750 [ 2148.805996] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2148.813133] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2148.821214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.826953] CR2: 00007fb38e6c4000 CR3: 000000017eb7a003 CR4: 00000000003706f0 [ 2148.834104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2148.841250] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2148.848425] Call Trace: [ 2148.850874] [ 2148.852997] ? __warn+0x85/0x180 [ 2148.856229] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.861127] ? report_bug+0xfc/0x1d0 [ 2148.864704] ? prb_read_valid+0x1b/0x20 [ 2148.868553] ? handle_bug+0x44/0x70 [ 2148.872062] ? exc_invalid_op+0x17/0x60 [ 2148.875893] ? asm_exc_invalid_op+0x1a/0x20 [ 2148.880092] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.884981] ? __list_add_valid_or_report+0x7d/0xa0 [ 2148.889855] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2148.894764] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2148.899514] lookup_fast+0x6d/0xf0 [ 2148.902918] walk_component+0x26/0x160 [ 2148.906708] link_path_walk+0x24d/0x380 [ 2148.910543] ? path_lookupat.isra.40+0x22/0x120 [ 2148.915089] path_lookupat.isra.40+0x6b/0x120 [ 2148.919446] filename_lookup+0xd4/0x190 [ 2148.923322] ? lock_acquire+0xcb/0x2b0 [ 2148.927072] ? __might_fault+0x45/0x80 [ 2148.930825] ? find_held_lock+0x2d/0x90 [ 2148.934681] ? __might_fault+0x45/0x80 [ 2148.938476] vfs_statx+0x89/0x150 [ 2148.941817] vfs_fstatat+0x5a/0xc0 [ 2148.945220] __do_sys_newstat+0x30/0x70 [ 2148.949059] ? lock_release+0xc5/0x260 [ 2148.952814] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2148.957709] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2148.962868] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2148.968022] do_syscall_64+0x38/0x80 [ 2148.971601] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2148.976671] RIP: 0033:0x7f48f0eeca79 [ 2148.980260] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2148.999026] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2149.006589] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2149.013745] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2149.020891] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2149.028039] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2149.035171] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2149.042319] [ 2149.044507] irq event stamp: 12836193 [ 2149.048181] hardirqs last enabled at (12836203): [] console_unlock+0x106/0x120 [ 2149.057047] hardirqs last disabled at (12836212): [] console_unlock+0xeb/0x120 [ 2149.065829] softirqs last enabled at (12836226): [] __do_softirq+0x2be/0x371 [ 2149.074538] softirqs last disabled at (12836221): [] irq_exit_rcu+0xb0/0xd0 [ 2149.083075] ---[ end trace 0000000000000000 ]--- [ 2149.087715] ------------[ cut here ]------------ [ 2149.092331] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2149.103993] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2149.112940] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2149.168428] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2149.177395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2149.184883] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2149.190387] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2149.209142] RSP: 0018:ffffc90001fffac0 EFLAGS: 00010282 [ 2149.214368] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2149.221509] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2149.228681] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000137e [ 2149.235810] R10: 00000000000b3500 R11: ffffc90001fff950 R12: ffff888158a68750 [ 2149.242979] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2149.250110] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2149.258196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2149.263942] CR2: 00007fb38e6c4000 CR3: 000000017eb7a003 CR4: 00000000003706f0 [ 2149.271088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2149.278216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2149.285350] Call Trace: [ 2149.287820] [ 2149.289935] ? __warn+0x85/0x180 [ 2149.293186] ? __list_add_valid_or_report+0x7d/0xa0 [ 2149.298067] ? report_bug+0xfc/0x1d0 [ 2149.301643] ? prb_read_valid+0x1b/0x20 [ 2149.305502] ? handle_bug+0x44/0x70 [ 2149.309011] ? exc_invalid_op+0x17/0x60 [ 2149.312849] ? asm_exc_invalid_op+0x1a/0x20 [ 2149.317058] ? __list_add_valid_or_report+0x7d/0xa0 [ 2149.321933] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2149.326847] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2149.332085] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2149.336883] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2149.341635] lookup_fast+0x6d/0xf0 [ 2149.345038] walk_component+0x26/0x160 [ 2149.348792] link_path_walk+0x24d/0x380 [ 2149.352647] ? path_lookupat.isra.40+0x22/0x120 [ 2149.357200] path_lookupat.isra.40+0x6b/0x120 [ 2149.361560] filename_lookup+0xd4/0x190 [ 2149.365433] ? lock_acquire+0xcb/0x2b0 [ 2149.369184] ? __might_fault+0x45/0x80 [ 2149.372937] ? find_held_lock+0x2d/0x90 [ 2149.376792] ? __might_fault+0x45/0x80 [ 2149.380541] vfs_statx+0x89/0x150 [ 2149.383894] vfs_fstatat+0x5a/0xc0 [ 2149.387300] __do_sys_newstat+0x30/0x70 [ 2149.391153] ? lock_release+0xc5/0x260 [ 2149.394909] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2149.399804] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2149.404982] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2149.410119] do_syscall_64+0x38/0x80 [ 2149.413696] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2149.418767] RIP: 0033:0x7f48f0eeca79 [ 2149.422364] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2149.441131] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2149.448693] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2149.455834] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2149.462991] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2149.470117] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2149.477265] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2149.484433] [ 2149.486619] irq event stamp: 12837343 [ 2149.490294] hardirqs last enabled at (12837353): [] console_unlock+0x106/0x120 [ 2149.499179] hardirqs last disabled at (12837362): [] console_unlock+0xeb/0x120 [ 2149.507957] softirqs last enabled at (12837376): [] __do_softirq+0x2be/0x371 [ 2149.516666] softirqs last disabled at (12837371): [] irq_exit_rcu+0xb0/0xd0 [ 2149.525186] ---[ end trace 0000000000000000 ]--- [ 2149.585725] ------------[ cut here ]------------ [ 2149.590349] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2149.602045] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2149.610742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2149.666246] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2149.675364] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2149.682838] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2149.688366] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2149.693852] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2149.712617] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2149.717842] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2149.724976] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2149.732108] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000013c1 [ 2149.739257] R10: 00000000000b46e8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2149.746409] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2149.753540] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2149.761626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2149.767371] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2149.774497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2149.781630] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2149.788767] Call Trace: [ 2149.791216] [ 2149.793340] ? __warn+0x85/0x180 [ 2149.796572] ? __list_add_valid_or_report+0x7d/0xa0 [ 2149.801451] ? report_bug+0xfc/0x1d0 [ 2149.805032] ? prb_read_valid+0x1b/0x20 [ 2149.808890] ? handle_bug+0x44/0x70 [ 2149.812379] ? exc_invalid_op+0x17/0x60 [ 2149.816228] ? asm_exc_invalid_op+0x1a/0x20 [ 2149.820436] ? __list_add_valid_or_report+0x7d/0xa0 [ 2149.825311] ? __list_add_valid_or_report+0x7d/0xa0 [ 2149.830191] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2149.835098] ceph_add_cap+0x20b/0x620 [ceph] [ 2149.839401] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2149.844159] ? lock_acquire+0xcb/0x2b0 [ 2149.847921] ceph_fill_trace+0x143/0xf30 [ceph] [ 2149.852483] ? __mutex_lock+0x97/0xce0 [ 2149.856233] ? __mutex_lock+0xaec/0xce0 [ 2149.860075] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2149.864615] ? lock_downgrade+0xfd/0x250 [ 2149.868547] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2149.872943] ceph_con_process_message+0x74/0x150 [libceph] [ 2149.878463] ? calc_signature+0xdb/0x100 [libceph] [ 2149.883283] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2149.888628] ? process_scheduled_works+0x22a/0x510 [ 2149.893427] ? process_scheduled_works+0x22a/0x510 [ 2149.898216] ceph_con_workfn+0x344/0x750 [libceph] [ 2149.903020] ? process_scheduled_works+0x22a/0x510 [ 2149.907810] process_scheduled_works+0x2a4/0x510 [ 2149.912435] worker_thread+0x188/0x340 [ 2149.916185] ? rescuer_thread+0x310/0x310 [ 2149.920198] kthread+0xf5/0x130 [ 2149.923343] ? kthread_complete_and_exit+0x20/0x20 [ 2149.928135] ret_from_fork+0x31/0x40 [ 2149.931713] ? kthread_complete_and_exit+0x20/0x20 [ 2149.936524] ret_from_fork_asm+0x11/0x20 [ 2149.940457] [ 2149.942643] irq event stamp: 10187427 [ 2149.946327] hardirqs last enabled at (10187437): [] console_unlock+0x106/0x120 [ 2149.955210] hardirqs last disabled at (10187446): [] console_unlock+0xeb/0x120 [ 2149.964001] softirqs last enabled at (10187374): [] __do_softirq+0x2be/0x371 [ 2149.972691] softirqs last disabled at (10187369): [] irq_exit_rcu+0xb0/0xd0 [ 2149.981209] ---[ end trace 0000000000000000 ]--- [ 2149.985910] ------------[ cut here ]------------ [ 2149.990552] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2150.002199] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2150.010894] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2150.066399] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2150.075521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2150.083020] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2150.088514] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2150.094017] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2150.112763] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2150.118006] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2150.125138] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2150.132288] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001400 [ 2150.139430] R10: 00000000000b59a0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2150.146571] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2150.153713] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2150.161808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2150.167573] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2150.174722] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2150.181873] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2150.189022] Call Trace: [ 2150.191476] [ 2150.193600] ? __warn+0x85/0x180 [ 2150.196850] ? __list_add_valid_or_report+0x7d/0xa0 [ 2150.201746] ? report_bug+0xfc/0x1d0 [ 2150.205341] ? prb_read_valid+0x1b/0x20 [ 2150.209202] ? handle_bug+0x44/0x70 [ 2150.212691] ? exc_invalid_op+0x17/0x60 [ 2150.216540] ? asm_exc_invalid_op+0x1a/0x20 [ 2150.220768] ? __list_add_valid_or_report+0x7d/0xa0 [ 2150.225650] ? __list_add_valid_or_report+0x7d/0xa0 [ 2150.230545] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2150.235454] ceph_add_cap+0x20b/0x620 [ceph] [ 2150.239774] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2150.244514] ? lock_acquire+0xcb/0x2b0 [ 2150.248265] ceph_fill_trace+0x143/0xf30 [ceph] [ 2150.252803] ? __mutex_lock+0x97/0xce0 [ 2150.256572] ? __mutex_lock+0xaec/0xce0 [ 2150.260428] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2150.265005] ? lock_downgrade+0xfd/0x250 [ 2150.268954] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2150.273351] ceph_con_process_message+0x74/0x150 [libceph] [ 2150.278845] ? calc_signature+0xdb/0x100 [libceph] [ 2150.283685] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2150.289018] ? process_scheduled_works+0x22a/0x510 [ 2150.293817] ? process_scheduled_works+0x22a/0x510 [ 2150.298624] ceph_con_workfn+0x344/0x750 [libceph] [ 2150.303443] ? process_scheduled_works+0x22a/0x510 [ 2150.308251] process_scheduled_works+0x2a4/0x510 [ 2150.312894] worker_thread+0x188/0x340 [ 2150.316643] ? rescuer_thread+0x310/0x310 [ 2150.320671] kthread+0xf5/0x130 [ 2150.323836] ? kthread_complete_and_exit+0x20/0x20 [ 2150.328645] ret_from_fork+0x31/0x40 [ 2150.332243] ? kthread_complete_and_exit+0x20/0x20 [ 2150.337035] ret_from_fork_asm+0x11/0x20 [ 2150.340989] [ 2150.343179] irq event stamp: 10188541 [ 2150.346844] hardirqs last enabled at (10188551): [] console_unlock+0x106/0x120 [ 2150.355730] hardirqs last disabled at (10188560): [] console_unlock+0xeb/0x120 [ 2150.364526] softirqs last enabled at (10188486): [] __do_softirq+0x2be/0x371 [ 2150.373235] softirqs last disabled at (10188481): [] irq_exit_rcu+0xb0/0xd0 [ 2150.381756] ---[ end trace 0000000000000000 ]--- [ 2150.386395] ------------[ cut here ]------------ [ 2150.391031] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2150.402675] WARNING: CPU: 3 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2150.411630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2150.467092] CPU: 3 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2150.476041] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2150.483520] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2150.489006] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2150.507755] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2150.512980] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2150.520111] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2150.527237] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000143f [ 2150.534368] R10: 00000000000b6b80 R11: ffffc90001fcf9c8 R12: ffff888158a68750 [ 2150.541494] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2150.548627] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2150.556711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2150.562450] CR2: 00007f9bd3a7b000 CR3: 000000011aeae003 CR4: 00000000003706e0 [ 2150.569584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2150.576715] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2150.583849] Call Trace: [ 2150.586301] [ 2150.588407] ? __warn+0x85/0x180 [ 2150.591640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2150.596519] ? report_bug+0xfc/0x1d0 [ 2150.600098] ? prb_read_valid+0x1b/0x20 [ 2150.603939] ? handle_bug+0x44/0x70 [ 2150.607430] ? exc_invalid_op+0x17/0x60 [ 2150.611261] ? asm_exc_invalid_op+0x1a/0x20 [ 2150.615453] ? __list_add_valid_or_report+0x7d/0xa0 [ 2150.620327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2150.625205] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2150.630097] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2150.634815] lookup_fast+0x6d/0xf0 [ 2150.638217] walk_component+0x26/0x160 [ 2150.641970] link_path_walk+0x24d/0x380 [ 2150.645809] ? path_lookupat.isra.40+0x22/0x120 [ 2150.650344] path_lookupat.isra.40+0x6b/0x120 [ 2150.654700] filename_lookup+0xd4/0x190 [ 2150.658543] ? lock_acquire+0xcb/0x2b0 [ 2150.662292] ? __might_fault+0x45/0x80 [ 2150.666045] ? find_held_lock+0x2d/0x90 [ 2150.669883] ? __might_fault+0x45/0x80 [ 2150.673639] vfs_statx+0x89/0x150 [ 2150.676960] vfs_fstatat+0x5a/0xc0 [ 2150.680363] __do_sys_newstat+0x30/0x70 [ 2150.684201] ? lock_release+0xc5/0x260 [ 2150.687958] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2150.692833] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2150.697975] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2150.703115] do_syscall_64+0x38/0x80 [ 2150.706692] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2150.711745] RIP: 0033:0x7f48f0eeca79 [ 2150.715324] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2150.734069] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2150.741637] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2150.748768] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2150.755894] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2150.763026] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2150.770160] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2150.777297] [ 2150.779485] irq event stamp: 12771851 [ 2150.783151] hardirqs last enabled at (12771861): [] console_unlock+0x106/0x120 [ 2150.792016] hardirqs last disabled at (12771870): [] console_unlock+0xeb/0x120 [ 2150.800797] softirqs last enabled at (12771382): [] __do_softirq+0x2be/0x371 [ 2150.809489] softirqs last disabled at (12771377): [] irq_exit_rcu+0xb0/0xd0 [ 2150.818011] ---[ end trace 0000000000000000 ]--- [ 2150.822631] ------------[ cut here ]------------ [ 2150.827248] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2150.838897] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2150.847592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2150.903077] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2150.912202] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2150.919680] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2150.925177] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2150.930662] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2150.949419] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2150.954643] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2150.961776] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2150.968910] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001481 [ 2150.976042] R10: 00000000000b7dd8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2150.983173] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2150.990307] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2150.998393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2151.004139] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2151.011269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2151.018405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2151.025538] Call Trace: [ 2151.027993] [ 2151.030097] ? __warn+0x85/0x180 [ 2151.033330] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.038212] ? report_bug+0xfc/0x1d0 [ 2151.041798] ? prb_read_valid+0x1b/0x20 [ 2151.045640] ? handle_bug+0x44/0x70 [ 2151.049138] ? exc_invalid_op+0x17/0x60 [ 2151.052979] ? asm_exc_invalid_op+0x1a/0x20 [ 2151.057173] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.062054] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.066939] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2151.071837] ceph_add_cap+0x20b/0x620 [ceph] [ 2151.076128] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2151.080852] ? lock_acquire+0xcb/0x2b0 [ 2151.084611] ceph_fill_trace+0x143/0xf30 [ceph] [ 2151.089154] ? __mutex_lock+0x97/0xce0 [ 2151.092906] ? __mutex_lock+0xaec/0xce0 [ 2151.096750] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2151.101296] ? lock_downgrade+0xfd/0x250 [ 2151.105228] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2151.109610] ceph_con_process_message+0x74/0x150 [libceph] [ 2151.115108] ? calc_signature+0xdb/0x100 [libceph] [ 2151.119912] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2151.125241] ? process_scheduled_works+0x22a/0x510 [ 2151.130042] ? process_scheduled_works+0x22a/0x510 [ 2151.134836] ceph_con_workfn+0x344/0x750 [libceph] [ 2151.139645] ? process_scheduled_works+0x22a/0x510 [ 2151.144438] process_scheduled_works+0x2a4/0x510 [ 2151.149068] worker_thread+0x188/0x340 [ 2151.152821] ? rescuer_thread+0x310/0x310 [ 2151.156832] kthread+0xf5/0x130 [ 2151.159977] ? kthread_complete_and_exit+0x20/0x20 [ 2151.164774] ret_from_fork+0x31/0x40 [ 2151.168358] ? kthread_complete_and_exit+0x20/0x20 [ 2151.173152] ret_from_fork_asm+0x11/0x20 [ 2151.177089] [ 2151.179276] irq event stamp: 10192613 [ 2151.182945] hardirqs last enabled at (10192623): [] console_unlock+0x106/0x120 [ 2151.191808] hardirqs last disabled at (10192632): [] console_unlock+0xeb/0x120 [ 2151.200590] softirqs last enabled at (10192420): [] __do_softirq+0x2be/0x371 [ 2151.209285] softirqs last disabled at (10192651): [] irq_exit_rcu+0xb0/0xd0 [ 2151.217812] ---[ end trace 0000000000000000 ]--- [ 2151.222433] ------------[ cut here ]------------ [ 2151.227056] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2151.238705] WARNING: CPU: 3 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2151.247661] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2151.303133] CPU: 3 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2151.312093] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2151.319569] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2151.325054] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2151.343802] RSP: 0018:ffffc90001fcfac0 EFLAGS: 00010282 [ 2151.349028] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2151.356159] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2151.363292] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000014c0 [ 2151.370425] R10: 00000000000b9178 R11: ffffc90001fcf950 R12: ffff888158a68750 [ 2151.377558] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2151.384691] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2151.392778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2151.398521] CR2: 00007f9bd3a7b000 CR3: 000000011aeae003 CR4: 00000000003706e0 [ 2151.405658] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2151.412788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2151.419922] Call Trace: [ 2151.422373] [ 2151.424482] ? __warn+0x85/0x180 [ 2151.427715] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.432597] ? report_bug+0xfc/0x1d0 [ 2151.436180] ? prb_read_valid+0x1b/0x20 [ 2151.440026] ? handle_bug+0x44/0x70 [ 2151.443521] ? exc_invalid_op+0x17/0x60 [ 2151.447362] ? asm_exc_invalid_op+0x1a/0x20 [ 2151.451555] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.456440] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2151.461336] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2151.466591] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2151.471398] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2151.476125] lookup_fast+0x6d/0xf0 [ 2151.479532] walk_component+0x26/0x160 [ 2151.483290] link_path_walk+0x24d/0x380 [ 2151.487133] ? path_lookupat.isra.40+0x22/0x120 [ 2151.491675] path_lookupat.isra.40+0x6b/0x120 [ 2151.496037] filename_lookup+0xd4/0x190 [ 2151.499882] ? lock_acquire+0xcb/0x2b0 [ 2151.503635] ? __might_fault+0x45/0x80 [ 2151.507390] ? find_held_lock+0x2d/0x90 [ 2151.511235] ? __might_fault+0x45/0x80 [ 2151.515003] vfs_statx+0x89/0x150 [ 2151.518325] vfs_fstatat+0x5a/0xc0 [ 2151.521733] __do_sys_newstat+0x30/0x70 [ 2151.525572] ? lock_release+0xc5/0x260 [ 2151.529329] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2151.534211] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2151.539350] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2151.544495] do_syscall_64+0x38/0x80 [ 2151.548079] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2151.553132] RIP: 0033:0x7f48f0eeca79 [ 2151.556709] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2151.575458] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2151.583032] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2151.590165] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2151.597296] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2151.604431] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2151.611563] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2151.618706] [ 2151.620895] irq event stamp: 12773823 [ 2151.624563] hardirqs last enabled at (12773833): [] console_unlock+0x106/0x120 [ 2151.633428] hardirqs last disabled at (12773842): [] console_unlock+0xeb/0x120 [ 2151.642210] softirqs last enabled at (12773572): [] __do_softirq+0x2be/0x371 [ 2151.650901] softirqs last disabled at (12773567): [] irq_exit_rcu+0xb0/0xd0 [ 2151.659420] ---[ end trace 0000000000000000 ]--- [ 2151.664040] ------------[ cut here ]------------ [ 2151.668662] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2151.680309] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2151.689275] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2151.744793] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2151.753752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2151.761229] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2151.766717] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2151.785467] RSP: 0018:ffffc90001fc7b38 EFLAGS: 00010282 [ 2151.790699] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2151.797830] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2151.804963] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001503 [ 2151.812095] R10: 00000000000ba710 R11: ffffc90001fc79c8 R12: ffff888158a68750 [ 2151.819236] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2151.826374] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2151.834465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2151.840212] CR2: 00007f9bd3aa7008 CR3: 0000000113500006 CR4: 00000000003706e0 [ 2151.847345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2151.854479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2151.861610] Call Trace: [ 2151.864062] [ 2151.866170] ? __warn+0x85/0x180 [ 2151.869404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.874292] ? report_bug+0xfc/0x1d0 [ 2151.877877] ? prb_read_valid+0x1b/0x20 [ 2151.881721] ? handle_bug+0x44/0x70 [ 2151.885221] ? exc_invalid_op+0x17/0x60 [ 2151.889059] ? asm_exc_invalid_op+0x1a/0x20 [ 2151.893254] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.898136] ? __list_add_valid_or_report+0x7d/0xa0 [ 2151.903019] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2151.907921] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2151.912647] lookup_fast+0x6d/0xf0 [ 2151.916060] walk_component+0x26/0x160 [ 2151.919820] link_path_walk+0x24d/0x380 [ 2151.923665] ? path_lookupat.isra.40+0x22/0x120 [ 2151.928203] path_lookupat.isra.40+0x6b/0x120 [ 2151.932568] filename_lookup+0xd4/0x190 [ 2151.936413] ? lock_acquire+0xcb/0x2b0 [ 2151.940167] ? __might_fault+0x45/0x80 [ 2151.943921] ? find_held_lock+0x2d/0x90 [ 2151.947771] ? __might_fault+0x45/0x80 [ 2151.951535] vfs_statx+0x89/0x150 [ 2151.954864] vfs_fstatat+0x5a/0xc0 [ 2151.958274] __do_sys_newstat+0x30/0x70 [ 2151.962121] ? lock_release+0xc5/0x260 [ 2151.965877] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2151.970759] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2151.975901] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2151.981045] do_syscall_64+0x38/0x80 [ 2151.984627] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2151.989679] RIP: 0033:0x7f48f0eeca79 [ 2151.993260] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2152.012011] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2152.019581] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2152.026712] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2152.033846] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2152.040983] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2152.048122] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2152.055266] [ 2152.057453] irq event stamp: 12901681 [ 2152.061120] hardirqs last enabled at (12901691): [] console_unlock+0x106/0x120 [ 2152.069986] hardirqs last disabled at (12901700): [] console_unlock+0xeb/0x120 [ 2152.078766] softirqs last enabled at (12901658): [] __do_softirq+0x2be/0x371 [ 2152.087461] softirqs last disabled at (12901719): [] irq_exit_rcu+0xb0/0xd0 [ 2152.095987] ---[ end trace 0000000000000000 ]--- [ 2152.100608] ------------[ cut here ]------------ [ 2152.105226] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2152.116878] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2152.125577] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2152.181122] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2152.190248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2152.197729] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2152.203240] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2152.208725] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2152.227474] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2152.232709] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2152.239842] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2152.246973] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001545 [ 2152.254106] R10: 00000000000bbb20 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2152.261238] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2152.268371] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2152.276458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2152.282202] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2152.289336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2152.296468] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2152.303602] Call Trace: [ 2152.306056] [ 2152.308162] ? __warn+0x85/0x180 [ 2152.311404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2152.316292] ? report_bug+0xfc/0x1d0 [ 2152.319878] ? prb_read_valid+0x1b/0x20 [ 2152.323721] ? handle_bug+0x44/0x70 [ 2152.327220] ? exc_invalid_op+0x17/0x60 [ 2152.331061] ? asm_exc_invalid_op+0x1a/0x20 [ 2152.335266] ? __list_add_valid_or_report+0x7d/0xa0 [ 2152.340154] ? __list_add_valid_or_report+0x7d/0xa0 [ 2152.345039] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2152.349941] ceph_add_cap+0x20b/0x620 [ceph] [ 2152.354247] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2152.358980] ? lock_acquire+0xcb/0x2b0 [ 2152.362746] ceph_fill_trace+0x143/0xf30 [ceph] [ 2152.367298] ? __mutex_lock+0x97/0xce0 [ 2152.371058] ? __mutex_lock+0xaec/0xce0 [ 2152.374908] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2152.379463] ? lock_downgrade+0xfd/0x250 [ 2152.383401] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2152.387799] ceph_con_process_message+0x74/0x150 [libceph] [ 2152.393305] ? calc_signature+0xdb/0x100 [libceph] [ 2152.398127] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2152.403469] ? process_scheduled_works+0x22a/0x510 [ 2152.408269] ? process_scheduled_works+0x22a/0x510 [ 2152.413064] ceph_con_workfn+0x344/0x750 [libceph] [ 2152.417880] ? process_scheduled_works+0x22a/0x510 [ 2152.422676] process_scheduled_works+0x2a4/0x510 [ 2152.427316] worker_thread+0x188/0x340 [ 2152.431076] ? rescuer_thread+0x310/0x310 [ 2152.435097] kthread+0xf5/0x130 [ 2152.438250] ? kthread_complete_and_exit+0x20/0x20 [ 2152.443046] ret_from_fork+0x31/0x40 [ 2152.446631] ? kthread_complete_and_exit+0x20/0x20 [ 2152.451425] ret_from_fork_asm+0x11/0x20 [ 2152.455374] [ 2152.457568] irq event stamp: 10195793 [ 2152.461233] hardirqs last enabled at (10195803): [] console_unlock+0x106/0x120 [ 2152.470101] hardirqs last disabled at (10195812): [] console_unlock+0xeb/0x120 [ 2152.478880] softirqs last enabled at (10195590): [] __do_softirq+0x2be/0x371 [ 2152.487583] softirqs last disabled at (10195831): [] irq_exit_rcu+0xb0/0xd0 [ 2152.496110] ---[ end trace 0000000000000000 ]--- [ 2152.500731] ------------[ cut here ]------------ [ 2152.505355] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2152.517002] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2152.525957] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2152.581486] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2152.590438] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2152.597918] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2152.603402] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2152.622150] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2152.627375] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2152.634508] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2152.641642] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001584 [ 2152.648775] R10: 00000000000bcc18 R11: ffffc90001fc7950 R12: ffff888158a68750 [ 2152.655907] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2152.663039] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2152.671125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2152.676872] CR2: 00007f3ec80030c0 CR3: 0000000113500001 CR4: 00000000003706e0 [ 2152.684004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2152.691136] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2152.698270] Call Trace: [ 2152.700723] [ 2152.702832] ? __warn+0x85/0x180 [ 2152.706063] ? __list_add_valid_or_report+0x7d/0xa0 [ 2152.710942] ? report_bug+0xfc/0x1d0 [ 2152.714520] ? prb_read_valid+0x1b/0x20 [ 2152.718365] ? handle_bug+0x44/0x70 [ 2152.721863] ? exc_invalid_op+0x17/0x60 [ 2152.725703] ? asm_exc_invalid_op+0x1a/0x20 [ 2152.729897] ? __list_add_valid_or_report+0x7d/0xa0 [ 2152.734780] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2152.739686] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2152.744937] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2152.749756] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2152.754482] lookup_fast+0x6d/0xf0 [ 2152.757892] walk_component+0x26/0x160 [ 2152.761647] link_path_walk+0x24d/0x380 [ 2152.765490] ? path_lookupat.isra.40+0x22/0x120 [ 2152.770029] path_lookupat.isra.40+0x6b/0x120 [ 2152.774396] filename_lookup+0xd4/0x190 [ 2152.778247] ? lock_acquire+0xcb/0x2b0 [ 2152.782000] ? __might_fault+0x45/0x80 [ 2152.785759] ? find_held_lock+0x2d/0x90 [ 2152.789603] ? __might_fault+0x45/0x80 [ 2152.793361] vfs_statx+0x89/0x150 [ 2152.796690] vfs_fstatat+0x5a/0xc0 [ 2152.800099] __do_sys_newstat+0x30/0x70 [ 2152.803937] ? lock_release+0xc5/0x260 [ 2152.807693] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2152.812577] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2152.817717] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2152.822860] do_syscall_64+0x38/0x80 [ 2152.826446] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2152.831497] RIP: 0033:0x7f48f0eeca79 [ 2152.835076] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2152.853822] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2152.861388] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2152.868522] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2152.875653] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2152.882786] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2152.889919] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2152.897113] [ 2152.899352] irq event stamp: 12903623 [ 2152.903015] hardirqs last enabled at (12903633): [] console_unlock+0x106/0x120 [ 2152.911883] hardirqs last disabled at (12903642): [] console_unlock+0xeb/0x120 [ 2152.920660] softirqs last enabled at (12903410): [] __do_softirq+0x2be/0x371 [ 2152.929355] softirqs last disabled at (12903405): [] irq_exit_rcu+0xb0/0xd0 [ 2152.937874] ---[ end trace 0000000000000000 ]--- [ 2152.942495] ------------[ cut here ]------------ [ 2152.947111] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2152.958758] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2152.967453] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2153.022940] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2153.032065] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2153.039543] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2153.045045] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2153.050531] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2153.069278] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2153.074505] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2153.081637] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2153.088770] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000015c7 [ 2153.095904] R10: 00000000000bdf88 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2153.103037] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2153.110170] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2153.118256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2153.124000] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2153.131135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2153.138270] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2153.145400] Call Trace: [ 2153.147854] [ 2153.149959] ? __warn+0x85/0x180 [ 2153.153193] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.158072] ? report_bug+0xfc/0x1d0 [ 2153.161652] ? prb_read_valid+0x1b/0x20 [ 2153.165494] ? handle_bug+0x44/0x70 [ 2153.168990] ? exc_invalid_op+0x17/0x60 [ 2153.172831] ? asm_exc_invalid_op+0x1a/0x20 [ 2153.177032] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.181915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.186792] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2153.191694] ceph_add_cap+0x20b/0x620 [ceph] [ 2153.195991] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2153.200712] ? lock_acquire+0xcb/0x2b0 [ 2153.204473] ceph_fill_trace+0x143/0xf30 [ceph] [ 2153.209017] ? __mutex_lock+0x97/0xce0 [ 2153.212767] ? __mutex_lock+0xaec/0xce0 [ 2153.216610] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2153.221157] ? lock_downgrade+0xfd/0x250 [ 2153.225094] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2153.229472] ceph_con_process_message+0x74/0x150 [libceph] [ 2153.234972] ? calc_signature+0xdb/0x100 [libceph] [ 2153.239788] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2153.245116] ? process_scheduled_works+0x22a/0x510 [ 2153.249921] ? process_scheduled_works+0x22a/0x510 [ 2153.254714] ceph_con_workfn+0x344/0x750 [libceph] [ 2153.259529] ? process_scheduled_works+0x22a/0x510 [ 2153.264326] process_scheduled_works+0x2a4/0x510 [ 2153.268954] worker_thread+0x188/0x340 [ 2153.272709] ? rescuer_thread+0x310/0x310 [ 2153.276719] kthread+0xf5/0x130 [ 2153.279869] ? kthread_complete_and_exit+0x20/0x20 [ 2153.284669] ret_from_fork+0x31/0x40 [ 2153.288246] ? kthread_complete_and_exit+0x20/0x20 [ 2153.293042] ret_from_fork_asm+0x11/0x20 [ 2153.296978] [ 2153.299165] irq event stamp: 10199519 [ 2153.302832] hardirqs last enabled at (10199529): [] console_unlock+0x106/0x120 [ 2153.311698] hardirqs last disabled at (10199538): [] console_unlock+0xeb/0x120 [ 2153.320479] softirqs last enabled at (10199554): [] __do_softirq+0x2be/0x371 [ 2153.329173] softirqs last disabled at (10199565): [] irq_exit_rcu+0xb0/0xd0 [ 2153.337692] ---[ end trace 0000000000000000 ]--- [ 2153.342314] ------------[ cut here ]------------ [ 2153.346943] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2153.358595] WARNING: CPU: 4 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2153.367552] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2153.423074] CPU: 4 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2153.432030] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2153.439507] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2153.444995] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2153.463742] RSP: 0018:ffffc90001fe7b38 EFLAGS: 00010282 [ 2153.468969] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2153.476101] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2153.483235] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001606 [ 2153.490372] R10: 00000000000bf0d0 R11: ffffc90001fe79c8 R12: ffff888158a68750 [ 2153.497514] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2153.504652] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2153.512735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2153.518482] CR2: 00007fb38e6f3028 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2153.525615] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2153.532750] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2153.539886] Call Trace: [ 2153.542343] [ 2153.544449] ? __warn+0x85/0x180 [ 2153.547682] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.552560] ? report_bug+0xfc/0x1d0 [ 2153.556143] ? prb_read_valid+0x1b/0x20 [ 2153.559991] ? handle_bug+0x44/0x70 [ 2153.563488] ? exc_invalid_op+0x17/0x60 [ 2153.567329] ? asm_exc_invalid_op+0x1a/0x20 [ 2153.571524] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.576404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.581282] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2153.586183] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2153.590917] lookup_fast+0x6d/0xf0 [ 2153.594327] walk_component+0x26/0x160 [ 2153.598083] link_path_walk+0x24d/0x380 [ 2153.601926] ? path_lookupat.isra.40+0x22/0x120 [ 2153.606465] path_lookupat.isra.40+0x6b/0x120 [ 2153.610831] filename_lookup+0xd4/0x190 [ 2153.614683] ? lock_acquire+0xcb/0x2b0 [ 2153.618435] ? __might_fault+0x45/0x80 [ 2153.622195] ? find_held_lock+0x2d/0x90 [ 2153.626042] ? __might_fault+0x45/0x80 [ 2153.629805] vfs_statx+0x89/0x150 [ 2153.633134] vfs_fstatat+0x5a/0xc0 [ 2153.636545] __do_sys_newstat+0x30/0x70 [ 2153.640393] ? lock_release+0xc5/0x260 [ 2153.644155] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2153.649041] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2153.654188] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2153.659337] do_syscall_64+0x38/0x80 [ 2153.662916] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2153.667968] RIP: 0033:0x7f48f0eeca79 [ 2153.671547] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2153.690296] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2153.697867] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2153.705001] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2153.712134] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2153.719265] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2153.726399] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2153.733545] [ 2153.735743] irq event stamp: 12797729 [ 2153.739408] hardirqs last enabled at (12797739): [] console_unlock+0x106/0x120 [ 2153.748275] hardirqs last disabled at (12797758): [] console_unlock+0xeb/0x120 [ 2153.757056] softirqs last enabled at (12797772): [] __do_softirq+0x2be/0x371 [ 2153.765754] softirqs last disabled at (12797767): [] irq_exit_rcu+0xb0/0xd0 [ 2153.774274] ---[ end trace 0000000000000000 ]--- [ 2153.778896] ------------[ cut here ]------------ [ 2153.783534] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2153.795215] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2153.804189] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2153.859672] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2153.868653] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2153.876150] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2153.881652] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2153.900416] RSP: 0018:ffffc90001ff7b38 EFLAGS: 00010282 [ 2153.905659] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2153.912807] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2153.919950] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001648 [ 2153.927093] R10: 00000000000c03d0 R11: ffffc90001ff79c8 R12: ffff888158a68750 [ 2153.934239] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2153.941370] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2153.949457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2153.955206] CR2: 00007f9bd3b07000 CR3: 0000000120cc8006 CR4: 00000000003706e0 [ 2153.962340] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2153.969476] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2153.976612] Call Trace: [ 2153.979063] [ 2153.981171] ? __warn+0x85/0x180 [ 2153.984410] ? __list_add_valid_or_report+0x7d/0xa0 [ 2153.989291] ? report_bug+0xfc/0x1d0 [ 2153.992872] ? prb_read_valid+0x1b/0x20 [ 2153.996720] ? handle_bug+0x44/0x70 [ 2154.000218] ? exc_invalid_op+0x17/0x60 [ 2154.004060] ? asm_exc_invalid_op+0x1a/0x20 [ 2154.008261] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.013143] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.018029] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2154.022934] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2154.027656] lookup_fast+0x6d/0xf0 [ 2154.031068] walk_component+0x26/0x160 [ 2154.034826] link_path_walk+0x24d/0x380 [ 2154.038665] ? path_lookupat.isra.40+0x22/0x120 [ 2154.043202] path_lookupat.isra.40+0x6b/0x120 [ 2154.047571] filename_lookup+0xd4/0x190 [ 2154.051418] ? lock_acquire+0xcb/0x2b0 [ 2154.055178] ? __might_fault+0x45/0x80 [ 2154.058937] ? find_held_lock+0x2d/0x90 [ 2154.062778] ? __might_fault+0x45/0x80 [ 2154.066541] vfs_statx+0x89/0x150 [ 2154.069873] vfs_fstatat+0x5a/0xc0 [ 2154.073280] __do_sys_newstat+0x30/0x70 [ 2154.077120] ? lock_release+0xc5/0x260 [ 2154.080876] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2154.085759] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2154.090900] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2154.096048] do_syscall_64+0x38/0x80 [ 2154.099626] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2154.104677] RIP: 0033:0x7f48f0eeca79 [ 2154.108258] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2154.127006] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2154.134573] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2154.141703] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2154.148838] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2154.155969] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2154.163102] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2154.170244] [ 2154.172438] irq event stamp: 12848929 [ 2154.176100] hardirqs last enabled at (12848939): [] console_unlock+0x106/0x120 [ 2154.184969] hardirqs last disabled at (12848948): [] console_unlock+0xeb/0x120 [ 2154.193750] softirqs last enabled at (12848962): [] __do_softirq+0x2be/0x371 [ 2154.202451] softirqs last disabled at (12848957): [] irq_exit_rcu+0xb0/0xd0 [ 2154.210976] ---[ end trace 0000000000000000 ]--- [ 2154.215602] ------------[ cut here ]------------ [ 2154.220248] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2154.231907] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2154.240861] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2154.296385] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2154.305343] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2154.312821] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2154.318308] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2154.337054] RSP: 0018:ffffc90001957b38 EFLAGS: 00010282 [ 2154.342282] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2154.349414] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2154.356546] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000168a [ 2154.363682] R10: 00000000000c1658 R11: ffffc900019579c8 R12: ffff888158a68750 [ 2154.370813] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2154.377946] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2154.386030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2154.391776] CR2: 00007fb38e713000 CR3: 000000016696c006 CR4: 00000000003706f0 [ 2154.398910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2154.406042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2154.413174] Call Trace: [ 2154.415628] [ 2154.417736] ? __warn+0x85/0x180 [ 2154.420969] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.425847] ? report_bug+0xfc/0x1d0 [ 2154.429429] ? prb_read_valid+0x1b/0x20 [ 2154.433277] ? handle_bug+0x44/0x70 [ 2154.436767] ? exc_invalid_op+0x17/0x60 [ 2154.440607] ? asm_exc_invalid_op+0x1a/0x20 [ 2154.444803] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.449683] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.454568] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2154.459468] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2154.464196] lookup_fast+0x6d/0xf0 [ 2154.467605] walk_component+0x26/0x160 [ 2154.471360] link_path_walk+0x24d/0x380 [ 2154.475204] ? path_lookupat.isra.40+0x22/0x120 [ 2154.479742] path_lookupat.isra.40+0x6b/0x120 [ 2154.484109] filename_lookup+0xd4/0x190 [ 2154.487960] ? lock_acquire+0xcb/0x2b0 [ 2154.491714] ? __might_fault+0x45/0x80 [ 2154.495471] ? find_held_lock+0x2d/0x90 [ 2154.499317] ? __might_fault+0x45/0x80 [ 2154.503074] vfs_statx+0x89/0x150 [ 2154.506404] vfs_fstatat+0x5a/0xc0 [ 2154.509813] __do_sys_newstat+0x30/0x70 [ 2154.513650] ? lock_release+0xc5/0x260 [ 2154.517408] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2154.522290] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2154.527432] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2154.532574] do_syscall_64+0x38/0x80 [ 2154.536159] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2154.541211] RIP: 0033:0x7f48f0eeca79 [ 2154.544791] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2154.563538] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2154.571102] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2154.578236] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2154.585366] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2154.592501] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2154.599632] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2154.606797] [ 2154.608997] irq event stamp: 12772743 [ 2154.612659] hardirqs last enabled at (12772753): [] console_unlock+0x106/0x120 [ 2154.621525] hardirqs last disabled at (12772762): [] console_unlock+0xeb/0x120 [ 2154.630306] softirqs last enabled at (12772776): [] __do_softirq+0x2be/0x371 [ 2154.638999] softirqs last disabled at (12772771): [] irq_exit_rcu+0xb0/0xd0 [ 2154.647518] ---[ end trace 0000000000000000 ]--- [ 2154.652137] ------------[ cut here ]------------ [ 2154.656754] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2154.668408] WARNING: CPU: 4 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2154.677376] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2154.732848] CPU: 4 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2154.741805] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2154.749281] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2154.754770] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2154.773515] RSP: 0018:ffffc90001fe7ac0 EFLAGS: 00010282 [ 2154.778741] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2154.785874] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2154.793007] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000016cc [ 2154.800140] R10: 00000000000c2d60 R11: ffffc90001fe7950 R12: ffff888158a68750 [ 2154.807274] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2154.814403] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2154.822491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2154.828242] CR2: 00007fb38e6f3028 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2154.835380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2154.842514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2154.849647] Call Trace: [ 2154.852097] [ 2154.854205] ? __warn+0x85/0x180 [ 2154.857435] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.862319] ? report_bug+0xfc/0x1d0 [ 2154.865907] ? prb_read_valid+0x1b/0x20 [ 2154.869756] ? handle_bug+0x44/0x70 [ 2154.873255] ? exc_invalid_op+0x17/0x60 [ 2154.877095] ? asm_exc_invalid_op+0x1a/0x20 [ 2154.881285] ? __list_add_valid_or_report+0x7d/0xa0 [ 2154.886170] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2154.891072] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2154.896324] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2154.901131] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2154.905857] lookup_fast+0x6d/0xf0 [ 2154.909263] walk_component+0x26/0x160 [ 2154.913025] link_path_walk+0x24d/0x380 [ 2154.916865] ? path_lookupat.isra.40+0x22/0x120 [ 2154.921404] path_lookupat.isra.40+0x6b/0x120 [ 2154.925769] filename_lookup+0xd4/0x190 [ 2154.929614] ? lock_acquire+0xcb/0x2b0 [ 2154.933365] ? __might_fault+0x45/0x80 [ 2154.937123] ? find_held_lock+0x2d/0x90 [ 2154.940968] ? __might_fault+0x45/0x80 [ 2154.944729] vfs_statx+0x89/0x150 [ 2154.948052] vfs_fstatat+0x5a/0xc0 [ 2154.951467] __do_sys_newstat+0x30/0x70 [ 2154.955312] ? lock_release+0xc5/0x260 [ 2154.959070] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2154.963950] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2154.969093] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2154.974232] do_syscall_64+0x38/0x80 [ 2154.977812] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2154.982861] RIP: 0033:0x7f48f0eeca79 [ 2154.986443] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2155.005193] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2155.012774] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2155.019906] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2155.027048] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2155.034180] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2155.041313] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2155.048458] [ 2155.050647] irq event stamp: 12801315 [ 2155.054312] hardirqs last enabled at (12801325): [] console_unlock+0x106/0x120 [ 2155.063179] hardirqs last disabled at (12801334): [] console_unlock+0xeb/0x120 [ 2155.071960] softirqs last enabled at (12801348): [] __do_softirq+0x2be/0x371 [ 2155.080668] softirqs last disabled at (12801361): [] irq_exit_rcu+0xb0/0xd0 [ 2155.089187] ---[ end trace 0000000000000000 ]--- [ 2155.093809] ------------[ cut here ]------------ [ 2155.098425] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2155.110074] WARNING: CPU: 6 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2155.119028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2155.174549] CPU: 6 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2155.183500] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2155.190980] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2155.196466] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2155.215211] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2155.220442] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2155.227572] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2155.234704] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000170f [ 2155.241838] R10: 00000000000c4048 R11: ffffc90001957950 R12: ffff888158a68750 [ 2155.248969] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2155.256101] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2155.264189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2155.269934] CR2: 00007f9bd3b1d000 CR3: 000000016696c004 CR4: 00000000003706e0 [ 2155.277069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2155.284200] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2155.291334] Call Trace: [ 2155.293785] [ 2155.295893] ? __warn+0x85/0x180 [ 2155.299124] ? __list_add_valid_or_report+0x7d/0xa0 [ 2155.304004] ? report_bug+0xfc/0x1d0 [ 2155.307585] ? prb_read_valid+0x1b/0x20 [ 2155.311425] ? handle_bug+0x44/0x70 [ 2155.314916] ? exc_invalid_op+0x17/0x60 [ 2155.318756] ? asm_exc_invalid_op+0x1a/0x20 [ 2155.322951] ? __list_add_valid_or_report+0x7d/0xa0 [ 2155.327830] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2155.332729] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2155.337982] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2155.342800] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2155.347527] lookup_fast+0x6d/0xf0 [ 2155.350936] walk_component+0x26/0x160 [ 2155.354690] link_path_walk+0x24d/0x380 [ 2155.358527] ? path_lookupat.isra.40+0x22/0x120 [ 2155.363075] path_lookupat.isra.40+0x6b/0x120 [ 2155.367442] filename_lookup+0xd4/0x190 [ 2155.371295] ? lock_acquire+0xcb/0x2b0 [ 2155.375045] ? __might_fault+0x45/0x80 [ 2155.378801] ? find_held_lock+0x2d/0x90 [ 2155.382647] ? __might_fault+0x45/0x80 [ 2155.386405] vfs_statx+0x89/0x150 [ 2155.389734] vfs_fstatat+0x5a/0xc0 [ 2155.393145] __do_sys_newstat+0x30/0x70 [ 2155.396991] ? lock_release+0xc5/0x260 [ 2155.400748] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2155.405630] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2155.410772] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2155.415913] do_syscall_64+0x38/0x80 [ 2155.419499] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2155.424551] RIP: 0033:0x7f48f0eeca79 [ 2155.428131] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2155.446879] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2155.454450] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2155.461583] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2155.468717] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2155.475849] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2155.482982] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2155.490128] [ 2155.492316] irq event stamp: 12774873 [ 2155.495981] hardirqs last enabled at (12774883): [] console_unlock+0x106/0x120 [ 2155.504849] hardirqs last disabled at (12774892): [] console_unlock+0xeb/0x120 [ 2155.513629] softirqs last enabled at (12774906): [] __do_softirq+0x2be/0x371 [ 2155.522320] softirqs last disabled at (12774901): [] irq_exit_rcu+0xb0/0xd0 [ 2155.530839] ---[ end trace 0000000000000000 ]--- [ 2155.535459] ------------[ cut here ]------------ [ 2155.540076] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2155.551722] WARNING: CPU: 2 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2155.560677] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2155.616138] CPU: 2 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2155.625090] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2155.632569] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2155.638054] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2155.656801] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2155.662027] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2155.669159] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2155.676283] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001752 [ 2155.683410] R10: 00000000000c5390 R11: ffffc90001ff7950 R12: ffff888158a68750 [ 2155.690541] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2155.697673] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2155.705752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2155.711497] CR2: 00007f9bd3b12028 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2155.718631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2155.725762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2155.732886] Call Trace: [ 2155.735332] [ 2155.737437] ? __warn+0x85/0x180 [ 2155.740672] ? __list_add_valid_or_report+0x7d/0xa0 [ 2155.745549] ? report_bug+0xfc/0x1d0 [ 2155.749130] ? prb_read_valid+0x1b/0x20 [ 2155.752970] ? handle_bug+0x44/0x70 [ 2155.756461] ? exc_invalid_op+0x17/0x60 [ 2155.760300] ? asm_exc_invalid_op+0x1a/0x20 [ 2155.764492] ? __list_add_valid_or_report+0x7d/0xa0 [ 2155.769366] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2155.774255] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2155.779491] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2155.784295] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2155.789009] lookup_fast+0x6d/0xf0 [ 2155.792411] walk_component+0x26/0x160 [ 2155.796164] link_path_walk+0x24d/0x380 [ 2155.800003] ? path_lookupat.isra.40+0x22/0x120 [ 2155.804537] path_lookupat.isra.40+0x6b/0x120 [ 2155.808897] filename_lookup+0xd4/0x190 [ 2155.812741] ? lock_acquire+0xcb/0x2b0 [ 2155.816488] ? __might_fault+0x45/0x80 [ 2155.820242] ? find_held_lock+0x2d/0x90 [ 2155.824081] ? __might_fault+0x45/0x80 [ 2155.827839] vfs_statx+0x89/0x150 [ 2155.831164] vfs_fstatat+0x5a/0xc0 [ 2155.834566] __do_sys_newstat+0x30/0x70 [ 2155.838406] ? lock_release+0xc5/0x260 [ 2155.842161] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2155.847039] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2155.852179] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2155.857317] do_syscall_64+0x38/0x80 [ 2155.860896] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2155.865949] RIP: 0033:0x7f48f0eeca79 [ 2155.869527] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2155.888269] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2155.895831] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2155.902955] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2155.910082] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2155.917214] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2155.924345] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2155.931484] [ 2155.933671] irq event stamp: 12852895 [ 2155.937327] hardirqs last enabled at (12852905): [] console_unlock+0x106/0x120 [ 2155.946196] hardirqs last disabled at (12852914): [] console_unlock+0xeb/0x120 [ 2155.954977] softirqs last enabled at (12852928): [] __do_softirq+0x2be/0x371 [ 2155.963668] softirqs last disabled at (12852923): [] irq_exit_rcu+0xb0/0xd0 [ 2155.972186] ---[ end trace 0000000000000000 ]--- [ 2155.976798] ------------[ cut here ]------------ [ 2155.981419] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2155.993087] WARNING: CPU: 1 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2156.002048] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2156.057593] CPU: 1 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2156.066562] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2156.074060] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2156.079569] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2156.098318] RSP: 0018:ffffc90001f7fb38 EFLAGS: 00010282 [ 2156.103542] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2156.110693] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2156.117865] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001795 [ 2156.124993] R10: 00000000000c6580 R11: ffffc90001f7f9c8 R12: ffff888158a68750 [ 2156.132144] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2156.139276] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2156.147362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2156.153112] CR2: 00007f3ec80030c0 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2156.160243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2156.167374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2156.174506] Call Trace: [ 2156.176979] [ 2156.179114] ? __warn+0x85/0x180 [ 2156.182352] ? __list_add_valid_or_report+0x7d/0xa0 [ 2156.187232] ? report_bug+0xfc/0x1d0 [ 2156.190810] ? prb_read_valid+0x1b/0x20 [ 2156.194688] ? handle_bug+0x44/0x70 [ 2156.198186] ? exc_invalid_op+0x17/0x60 [ 2156.202025] ? asm_exc_invalid_op+0x1a/0x20 [ 2156.206238] ? __list_add_valid_or_report+0x7d/0xa0 [ 2156.211120] ? __list_add_valid_or_report+0x7d/0xa0 [ 2156.216004] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2156.220939] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2156.225684] lookup_fast+0x6d/0xf0 [ 2156.229111] walk_component+0x26/0x160 [ 2156.232883] link_path_walk+0x24d/0x380 [ 2156.236727] ? path_lookupat.isra.40+0x22/0x120 [ 2156.241282] path_lookupat.isra.40+0x6b/0x120 [ 2156.245664] filename_lookup+0xd4/0x190 [ 2156.249527] ? lock_acquire+0xcb/0x2b0 [ 2156.253280] ? __might_fault+0x45/0x80 [ 2156.257053] ? find_held_lock+0x2d/0x90 [ 2156.260890] ? __might_fault+0x45/0x80 [ 2156.264684] vfs_statx+0x89/0x150 [ 2156.268030] vfs_fstatat+0x5a/0xc0 [ 2156.271456] __do_sys_newstat+0x30/0x70 [ 2156.275296] ? lock_release+0xc5/0x260 [ 2156.279069] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2156.283951] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2156.289116] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2156.294260] do_syscall_64+0x38/0x80 [ 2156.297846] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2156.302916] RIP: 0033:0x7f48f0eeca79 [ 2156.306512] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2156.325276] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2156.332841] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2156.339990] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2156.347142] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2156.354275] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2156.361408] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2156.368587] [ 2156.370776] irq event stamp: 12896969 [ 2156.374479] hardirqs last enabled at (12896985): [] console_unlock+0x106/0x120 [ 2156.383353] hardirqs last disabled at (12896994): [] console_unlock+0xeb/0x120 [ 2156.392131] softirqs last enabled at (12896982): [] __do_softirq+0x2be/0x371 [ 2156.400824] softirqs last disabled at (12896977): [] irq_exit_rcu+0xb0/0xd0 [ 2156.409361] ---[ end trace 0000000000000000 ]--- [ 2156.414000] ------------[ cut here ]------------ [ 2156.418617] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2156.430263] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2156.438957] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2156.494437] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2156.503561] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2156.511039] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2156.516543] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2156.522028] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2156.540776] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2156.545999] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2156.553135] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2156.560267] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000017d7 [ 2156.567400] R10: 00000000000c78c8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2156.574532] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2156.581665] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2156.589753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2156.595498] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2156.602631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2156.609762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2156.616895] Call Trace: [ 2156.619349] [ 2156.621456] ? __warn+0x85/0x180 [ 2156.624690] ? __list_add_valid_or_report+0x7d/0xa0 [ 2156.629569] ? report_bug+0xfc/0x1d0 [ 2156.633148] ? prb_read_valid+0x1b/0x20 [ 2156.636989] ? handle_bug+0x44/0x70 [ 2156.640480] ? exc_invalid_op+0x17/0x60 [ 2156.644318] ? asm_exc_invalid_op+0x1a/0x20 [ 2156.648513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2156.653395] ? __list_add_valid_or_report+0x7d/0xa0 [ 2156.658271] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2156.663173] ceph_add_cap+0x20b/0x620 [ceph] [ 2156.667468] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2156.672193] ? lock_acquire+0xcb/0x2b0 [ 2156.675953] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2156.680409] ? __mutex_lock+0x97/0xce0 [ 2156.684160] ? __mutex_lock+0xaec/0xce0 [ 2156.688002] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2156.692551] ? lock_downgrade+0xfd/0x250 [ 2156.696484] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2156.700864] ceph_con_process_message+0x74/0x150 [libceph] [ 2156.706364] ? calc_signature+0xdb/0x100 [libceph] [ 2156.711175] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2156.716497] ? process_scheduled_works+0x22a/0x510 [ 2156.721290] ? process_scheduled_works+0x22a/0x510 [ 2156.726080] ceph_con_workfn+0x344/0x750 [libceph] [ 2156.730895] ? process_scheduled_works+0x22a/0x510 [ 2156.735683] process_scheduled_works+0x2a4/0x510 [ 2156.740313] worker_thread+0x188/0x340 [ 2156.744067] ? rescuer_thread+0x310/0x310 [ 2156.748079] kthread+0xf5/0x130 [ 2156.751228] ? kthread_complete_and_exit+0x20/0x20 [ 2156.756027] ret_from_fork+0x31/0x40 [ 2156.759603] ? kthread_complete_and_exit+0x20/0x20 [ 2156.764399] ret_from_fork_asm+0x11/0x20 [ 2156.768335] [ 2156.770524] irq event stamp: 10220103 [ 2156.774191] hardirqs last enabled at (10220113): [] console_unlock+0x106/0x120 [ 2156.783055] hardirqs last disabled at (10220122): [] console_unlock+0xeb/0x120 [ 2156.791834] softirqs last enabled at (10220050): [] __do_softirq+0x2be/0x371 [ 2156.800530] softirqs last disabled at (10220045): [] irq_exit_rcu+0xb0/0xd0 [ 2156.809048] ---[ end trace 0000000000000000 ]--- [ 2156.813672] ------------[ cut here ]------------ [ 2156.818359] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2156.830014] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2156.838974] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2156.894445] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2156.903401] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2156.910880] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2156.916378] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2156.935141] RSP: 0018:ffffc90001f7fac0 EFLAGS: 00010282 [ 2156.940374] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2156.947500] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2156.954632] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001816 [ 2156.961763] R10: 00000000000c8b18 R11: ffffc90001f7f950 R12: ffff888158a68750 [ 2156.968890] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2156.976022] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2156.984107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2156.989846] CR2: 00007f9bd3b33018 CR3: 0000000113532001 CR4: 00000000003706e0 [ 2156.996978] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2157.004103] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2157.011234] Call Trace: [ 2157.013680] [ 2157.015786] ? __warn+0x85/0x180 [ 2157.019018] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.023902] ? report_bug+0xfc/0x1d0 [ 2157.027481] ? prb_read_valid+0x1b/0x20 [ 2157.031317] ? handle_bug+0x44/0x70 [ 2157.034810] ? exc_invalid_op+0x17/0x60 [ 2157.038649] ? asm_exc_invalid_op+0x1a/0x20 [ 2157.042840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.047714] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2157.052607] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2157.057849] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2157.062648] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2157.067365] lookup_fast+0x6d/0xf0 [ 2157.070767] walk_component+0x26/0x160 [ 2157.074521] link_path_walk+0x24d/0x380 [ 2157.078360] ? path_lookupat.isra.40+0x22/0x120 [ 2157.082895] path_lookupat.isra.40+0x6b/0x120 [ 2157.087254] filename_lookup+0xd4/0x190 [ 2157.091100] ? lock_acquire+0xcb/0x2b0 [ 2157.094851] ? __might_fault+0x45/0x80 [ 2157.098606] ? find_held_lock+0x2d/0x90 [ 2157.102444] ? __might_fault+0x45/0x80 [ 2157.106200] vfs_statx+0x89/0x150 [ 2157.109520] vfs_fstatat+0x5a/0xc0 [ 2157.112923] __do_sys_newstat+0x30/0x70 [ 2157.116763] ? lock_release+0xc5/0x260 [ 2157.120517] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2157.125395] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2157.130534] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2157.135674] do_syscall_64+0x38/0x80 [ 2157.139253] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2157.144306] RIP: 0033:0x7f48f0eeca79 [ 2157.147884] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2157.166633] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2157.174197] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2157.181331] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2157.188462] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2157.195594] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2157.202720] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2157.209858] [ 2157.212045] irq event stamp: 12899945 [ 2157.215702] hardirqs last enabled at (12899955): [] console_unlock+0x106/0x120 [ 2157.224570] hardirqs last disabled at (12899964): [] console_unlock+0xeb/0x120 [ 2157.233351] softirqs last enabled at (12899978): [] __do_softirq+0x2be/0x371 [ 2157.242041] softirqs last disabled at (12899973): [] irq_exit_rcu+0xb0/0xd0 [ 2157.250560] ---[ end trace 0000000000000000 ]--- [ 2157.255173] ------------[ cut here ]------------ [ 2157.259790] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2157.271455] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2157.280420] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2157.335896] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2157.344865] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2157.352352] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2157.357839] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2157.376601] RSP: 0018:ffffc90001fdfb38 EFLAGS: 00010282 [ 2157.381837] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2157.388977] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2157.396132] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001859 [ 2157.403261] R10: 00000000000c9e18 R11: ffffc90001fdf9c8 R12: ffff888158a68750 [ 2157.410389] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2157.417519] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2157.425613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2157.431367] CR2: 00007fb38e746000 CR3: 0000000179402006 CR4: 00000000003706e0 [ 2157.438491] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2157.445632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2157.452775] Call Trace: [ 2157.455237] [ 2157.457343] ? __warn+0x85/0x180 [ 2157.460576] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.465495] ? report_bug+0xfc/0x1d0 [ 2157.469079] ? prb_read_valid+0x1b/0x20 [ 2157.472936] ? handle_bug+0x44/0x70 [ 2157.476447] ? exc_invalid_op+0x17/0x60 [ 2157.480301] ? asm_exc_invalid_op+0x1a/0x20 [ 2157.484508] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.489385] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.494264] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2157.499156] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2157.503870] lookup_fast+0x6d/0xf0 [ 2157.507275] walk_component+0x26/0x160 [ 2157.511026] link_path_walk+0x24d/0x380 [ 2157.514881] ? path_lookupat.isra.40+0x22/0x120 [ 2157.519425] path_lookupat.isra.40+0x6b/0x120 [ 2157.523801] filename_lookup+0xd4/0x190 [ 2157.527676] ? lock_acquire+0xcb/0x2b0 [ 2157.531427] ? __might_fault+0x45/0x80 [ 2157.535199] ? find_held_lock+0x2d/0x90 [ 2157.539036] ? __might_fault+0x45/0x80 [ 2157.542829] vfs_statx+0x89/0x150 [ 2157.546145] vfs_fstatat+0x5a/0xc0 [ 2157.549549] __do_sys_newstat+0x30/0x70 [ 2157.553421] ? lock_release+0xc5/0x260 [ 2157.557186] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2157.562063] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2157.567220] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2157.572360] do_syscall_64+0x38/0x80 [ 2157.575939] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2157.581009] RIP: 0033:0x7f48f0eeca79 [ 2157.584599] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2157.603362] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2157.610927] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2157.618068] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2157.625210] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2157.632344] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2157.639475] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2157.646640] [ 2157.648826] irq event stamp: 12795995 [ 2157.652502] hardirqs last enabled at (12796005): [] console_unlock+0x106/0x120 [ 2157.661387] hardirqs last disabled at (12796014): [] console_unlock+0xeb/0x120 [ 2157.670165] softirqs last enabled at (12795932): [] __do_softirq+0x2be/0x371 [ 2157.678859] softirqs last disabled at (12796033): [] irq_exit_rcu+0xb0/0xd0 [ 2157.687410] ---[ end trace 0000000000000000 ]--- [ 2157.692067] ------------[ cut here ]------------ [ 2157.696688] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2157.708353] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2157.717047] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2157.772552] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2157.781673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2157.789155] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2157.794658] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2157.800142] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2157.818889] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2157.824117] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2157.831248] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2157.838381] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000189b [ 2157.845513] R10: 00000000000cb410 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2157.852646] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2157.859780] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2157.867865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2157.873614] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2157.880746] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2157.887876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2157.895011] Call Trace: [ 2157.897462] [ 2157.899572] ? __warn+0x85/0x180 [ 2157.902804] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.907682] ? report_bug+0xfc/0x1d0 [ 2157.911262] ? prb_read_valid+0x1b/0x20 [ 2157.915103] ? handle_bug+0x44/0x70 [ 2157.918594] ? exc_invalid_op+0x17/0x60 [ 2157.922439] ? asm_exc_invalid_op+0x1a/0x20 [ 2157.926635] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.931518] ? __list_add_valid_or_report+0x7d/0xa0 [ 2157.936393] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2157.941295] ceph_add_cap+0x20b/0x620 [ceph] [ 2157.945592] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2157.950319] ceph_fill_trace+0x143/0xf30 [ceph] [ 2157.954866] ? __mutex_lock+0x97/0xce0 [ 2157.958615] ? __mutex_lock+0xaec/0xce0 [ 2157.962457] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2157.967006] ? lock_downgrade+0xfd/0x250 [ 2157.970941] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2157.975320] ceph_con_process_message+0x74/0x150 [libceph] [ 2157.980820] ? calc_signature+0xdb/0x100 [libceph] [ 2157.985636] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2157.990969] ? process_scheduled_works+0x22a/0x510 [ 2157.995768] ? process_scheduled_works+0x22a/0x510 [ 2158.000563] ceph_con_workfn+0x344/0x750 [libceph] [ 2158.005377] ? process_scheduled_works+0x22a/0x510 [ 2158.010165] process_scheduled_works+0x2a4/0x510 [ 2158.014795] worker_thread+0x188/0x340 [ 2158.018549] ? rescuer_thread+0x310/0x310 [ 2158.022560] kthread+0xf5/0x130 [ 2158.025706] ? kthread_complete_and_exit+0x20/0x20 [ 2158.030503] ret_from_fork+0x31/0x40 [ 2158.034078] ? kthread_complete_and_exit+0x20/0x20 [ 2158.038871] ret_from_fork_asm+0x11/0x20 [ 2158.042807] [ 2158.044997] irq event stamp: 10223005 [ 2158.048662] hardirqs last enabled at (10223015): [] console_unlock+0x106/0x120 [ 2158.057530] hardirqs last disabled at (10223024): [] console_unlock+0xeb/0x120 [ 2158.066311] softirqs last enabled at (10222962): [] __do_softirq+0x2be/0x371 [ 2158.075001] softirqs last disabled at (10222957): [] irq_exit_rcu+0xb0/0xd0 [ 2158.083523] ---[ end trace 0000000000000000 ]--- [ 2158.088143] ------------[ cut here ]------------ [ 2158.092761] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2158.104412] WARNING: CPU: 3 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2158.113372] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2158.168911] CPU: 3 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2158.177861] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2158.185358] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2158.190844] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2158.209609] RSP: 0018:ffffc90001fdfac0 EFLAGS: 00010282 [ 2158.214852] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2158.222001] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2158.229153] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000018d9 [ 2158.236288] R10: 00000000000cc5e0 R11: ffffc90001fdf950 R12: ffff888158a68750 [ 2158.243433] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2158.250603] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2158.258687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2158.264452] CR2: 00007fb38e746000 CR3: 0000000179402006 CR4: 00000000003706e0 [ 2158.271602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2158.278752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2158.285902] Call Trace: [ 2158.288373] [ 2158.290478] ? __warn+0x85/0x180 [ 2158.293730] ? __list_add_valid_or_report+0x7d/0xa0 [ 2158.298644] ? report_bug+0xfc/0x1d0 [ 2158.302224] ? prb_read_valid+0x1b/0x20 [ 2158.306083] ? handle_bug+0x44/0x70 [ 2158.309580] ? exc_invalid_op+0x17/0x60 [ 2158.313444] ? asm_exc_invalid_op+0x1a/0x20 [ 2158.317675] ? __list_add_valid_or_report+0x7d/0xa0 [ 2158.322575] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2158.327490] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2158.332760] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2158.337592] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2158.342323] lookup_fast+0x6d/0xf0 [ 2158.345732] walk_component+0x26/0x160 [ 2158.349519] link_path_walk+0x24d/0x380 [ 2158.353357] ? path_lookupat.isra.40+0x22/0x120 [ 2158.357911] path_lookupat.isra.40+0x6b/0x120 [ 2158.362280] filename_lookup+0xd4/0x190 [ 2158.366150] ? lock_acquire+0xcb/0x2b0 [ 2158.369902] ? __might_fault+0x45/0x80 [ 2158.373673] ? find_held_lock+0x2d/0x90 [ 2158.377528] ? __might_fault+0x45/0x80 [ 2158.381306] vfs_statx+0x89/0x150 [ 2158.384651] vfs_fstatat+0x5a/0xc0 [ 2158.388077] __do_sys_newstat+0x30/0x70 [ 2158.391917] ? lock_release+0xc5/0x260 [ 2158.395690] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2158.400591] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2158.405749] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2158.410922] do_syscall_64+0x38/0x80 [ 2158.414502] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2158.419569] RIP: 0033:0x7f48f0eeca79 [ 2158.423169] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2158.441915] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2158.449479] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2158.456629] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2158.463779] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2158.470929] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2158.478080] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2158.485245] [ 2158.487440] irq event stamp: 12798019 [ 2158.491124] hardirqs last enabled at (12798029): [] console_unlock+0x106/0x120 [ 2158.500007] hardirqs last disabled at (12798038): [] console_unlock+0xeb/0x120 [ 2158.508823] softirqs last enabled at (12798052): [] __do_softirq+0x2be/0x371 [ 2158.517515] softirqs last disabled at (12798047): [] irq_exit_rcu+0xb0/0xd0 [ 2158.526051] ---[ end trace 0000000000000000 ]--- [ 2158.530706] ------------[ cut here ]------------ [ 2158.535324] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2158.546973] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2158.555665] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2158.611141] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2158.620269] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2158.627745] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2158.633243] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2158.638731] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2158.657476] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2158.662702] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2158.669832] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2158.676967] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000191c [ 2158.684098] R10: 00000000000cda30 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2158.691230] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2158.698365] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2158.706451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2158.712196] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2158.719329] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2158.726462] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2158.733595] Call Trace: [ 2158.736048] [ 2158.738156] ? __warn+0x85/0x180 [ 2158.741389] ? __list_add_valid_or_report+0x7d/0xa0 [ 2158.746267] ? report_bug+0xfc/0x1d0 [ 2158.749847] ? prb_read_valid+0x1b/0x20 [ 2158.753690] ? handle_bug+0x44/0x70 [ 2158.757178] ? exc_invalid_op+0x17/0x60 [ 2158.761020] ? asm_exc_invalid_op+0x1a/0x20 [ 2158.765211] ? __list_add_valid_or_report+0x7d/0xa0 [ 2158.770094] ? __list_add_valid_or_report+0x7d/0xa0 [ 2158.774973] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2158.779869] ceph_add_cap+0x20b/0x620 [ceph] [ 2158.784160] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2158.788882] ? lock_acquire+0xcb/0x2b0 [ 2158.792642] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2158.797100] ? __mutex_lock+0x97/0xce0 [ 2158.800851] ? __mutex_lock+0xaec/0xce0 [ 2158.804694] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2158.809239] ? lock_downgrade+0xfd/0x250 [ 2158.813175] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2158.817552] ceph_con_process_message+0x74/0x150 [libceph] [ 2158.823047] ? calc_signature+0xdb/0x100 [libceph] [ 2158.827859] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2158.833194] ? process_scheduled_works+0x22a/0x510 [ 2158.837994] ? process_scheduled_works+0x22a/0x510 [ 2158.842788] ceph_con_workfn+0x344/0x750 [libceph] [ 2158.847593] ? process_scheduled_works+0x22a/0x510 [ 2158.852384] process_scheduled_works+0x2a4/0x510 [ 2158.857011] worker_thread+0x188/0x340 [ 2158.860767] ? rescuer_thread+0x310/0x310 [ 2158.864779] kthread+0xf5/0x130 [ 2158.867922] ? kthread_complete_and_exit+0x20/0x20 [ 2158.872718] ret_from_fork+0x31/0x40 [ 2158.876294] ? kthread_complete_and_exit+0x20/0x20 [ 2158.881092] ret_from_fork_asm+0x11/0x20 [ 2158.885033] [ 2158.887221] irq event stamp: 10225071 [ 2158.890882] hardirqs last enabled at (10225081): [] console_unlock+0x106/0x120 [ 2158.899746] hardirqs last disabled at (10225096): [] console_unlock+0xeb/0x120 [ 2158.908527] softirqs last enabled at (10225094): [] __do_softirq+0x2be/0x371 [ 2158.917220] softirqs last disabled at (10225089): [] irq_exit_rcu+0xb0/0xd0 [ 2158.925738] ---[ end trace 0000000000000000 ]--- [ 2158.930361] ------------[ cut here ]------------ [ 2158.934975] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2158.946637] WARNING: CPU: 7 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2158.955600] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2159.011067] CPU: 7 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2159.020019] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2159.027495] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2159.032984] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2159.051728] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2159.056954] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2159.064086] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2159.071221] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000195b [ 2159.078354] R10: 00000000000ceb60 R11: ffffc90001fef9c8 R12: ffff8881c5350750 [ 2159.085487] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2159.092622] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2159.100709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2159.106462] CR2: 00007fb38e7570a0 CR3: 0000000108c72001 CR4: 00000000003706e0 [ 2159.113593] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2159.120724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2159.127859] Call Trace: [ 2159.130312] [ 2159.132419] ? __warn+0x85/0x180 [ 2159.135649] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.140533] ? report_bug+0xfc/0x1d0 [ 2159.144121] ? prb_read_valid+0x1b/0x20 [ 2159.147962] ? handle_bug+0x44/0x70 [ 2159.151456] ? exc_invalid_op+0x17/0x60 [ 2159.155300] ? asm_exc_invalid_op+0x1a/0x20 [ 2159.159501] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.164383] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.169269] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2159.174171] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2159.178895] lookup_fast+0x6d/0xf0 [ 2159.182306] walk_component+0x26/0x160 [ 2159.186059] link_path_walk+0x24d/0x380 [ 2159.189898] ? path_lookupat.isra.40+0x22/0x120 [ 2159.194431] path_lookupat.isra.40+0x6b/0x120 [ 2159.198792] filename_lookup+0xd4/0x190 [ 2159.202641] ? lock_acquire+0xcb/0x2b0 [ 2159.206400] ? __might_fault+0x45/0x80 [ 2159.210160] ? find_held_lock+0x2d/0x90 [ 2159.214000] ? __might_fault+0x45/0x80 [ 2159.217757] vfs_statx+0x89/0x150 [ 2159.221087] vfs_fstatat+0x5a/0xc0 [ 2159.224496] __do_sys_newstat+0x30/0x70 [ 2159.228336] ? lock_release+0xc5/0x260 [ 2159.232100] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2159.236982] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2159.242123] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2159.247265] do_syscall_64+0x38/0x80 [ 2159.250851] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2159.255902] RIP: 0033:0x7f48f0eeca79 [ 2159.259482] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2159.278234] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2159.285802] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2159.292935] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2159.300068] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2159.307202] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2159.314333] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2159.321478] [ 2159.323667] irq event stamp: 12794177 [ 2159.327335] hardirqs last enabled at (12794187): [] console_unlock+0x106/0x120 [ 2159.336200] hardirqs last disabled at (12794196): [] console_unlock+0xeb/0x120 [ 2159.344983] softirqs last enabled at (12794210): [] __do_softirq+0x2be/0x371 [ 2159.353681] softirqs last disabled at (12794205): [] irq_exit_rcu+0xb0/0xd0 [ 2159.362203] ---[ end trace 0000000000000000 ]--- [ 2159.366822] ------------[ cut here ]------------ [ 2159.371439] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2159.383086] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2159.391779] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2159.447260] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2159.456385] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2159.463864] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2159.469366] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2159.474851] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2159.493596] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2159.498823] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2159.505955] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2159.513088] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000199d [ 2159.520221] R10: 00000000000cfd30 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2159.527355] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2159.534488] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2159.542573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2159.548320] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2159.555469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2159.562601] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2159.569738] Call Trace: [ 2159.572188] [ 2159.574296] ? __warn+0x85/0x180 [ 2159.577527] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.582408] ? report_bug+0xfc/0x1d0 [ 2159.585985] ? prb_read_valid+0x1b/0x20 [ 2159.589829] ? handle_bug+0x44/0x70 [ 2159.593318] ? exc_invalid_op+0x17/0x60 [ 2159.597173] ? asm_exc_invalid_op+0x1a/0x20 [ 2159.601371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.606252] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.611127] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2159.616045] ceph_add_cap+0x20b/0x620 [ceph] [ 2159.620343] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2159.625071] ceph_fill_trace+0x143/0xf30 [ceph] [ 2159.629616] ? __mutex_lock+0x97/0xce0 [ 2159.633368] ? __mutex_lock+0xaec/0xce0 [ 2159.637209] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2159.641757] ? lock_downgrade+0xfd/0x250 [ 2159.645692] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2159.650071] ceph_con_process_message+0x74/0x150 [libceph] [ 2159.655564] ? calc_signature+0xdb/0x100 [libceph] [ 2159.660378] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2159.665710] ? process_scheduled_works+0x22a/0x510 [ 2159.670522] ? process_scheduled_works+0x22a/0x510 [ 2159.675315] ceph_con_workfn+0x344/0x750 [libceph] [ 2159.680120] ? process_scheduled_works+0x22a/0x510 [ 2159.684917] process_scheduled_works+0x2a4/0x510 [ 2159.689547] worker_thread+0x188/0x340 [ 2159.693300] ? rescuer_thread+0x310/0x310 [ 2159.697313] kthread+0xf5/0x130 [ 2159.700460] ? kthread_complete_and_exit+0x20/0x20 [ 2159.705255] ret_from_fork+0x31/0x40 [ 2159.708838] ? kthread_complete_and_exit+0x20/0x20 [ 2159.713633] ret_from_fork_asm+0x11/0x20 [ 2159.717568] [ 2159.719758] irq event stamp: 10227047 [ 2159.723423] hardirqs last enabled at (10227057): [] console_unlock+0x106/0x120 [ 2159.732290] hardirqs last disabled at (10227066): [] console_unlock+0xeb/0x120 [ 2159.741070] softirqs last enabled at (10227004): [] __do_softirq+0x2be/0x371 [ 2159.749763] softirqs last disabled at (10226999): [] irq_exit_rcu+0xb0/0xd0 [ 2159.758282] ---[ end trace 0000000000000000 ]--- [ 2159.762903] ------------[ cut here ]------------ [ 2159.767523] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2159.779170] WARNING: CPU: 7 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2159.788125] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2159.843603] CPU: 7 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2159.852552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2159.860030] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2159.865519] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2159.884264] RSP: 0018:ffffc90001fefac0 EFLAGS: 00010282 [ 2159.889491] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2159.896621] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2159.903757] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000019db [ 2159.910889] R10: 00000000000d0e38 R11: ffffc90001fef950 R12: ffff8881c5350750 [ 2159.918022] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2159.925155] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2159.933241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2159.938985] CR2: 00007fb38e7570a0 CR3: 0000000108c72001 CR4: 00000000003706e0 [ 2159.946121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2159.953252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2159.960386] Call Trace: [ 2159.962837] [ 2159.964946] ? __warn+0x85/0x180 [ 2159.968177] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.973057] ? report_bug+0xfc/0x1d0 [ 2159.976636] ? prb_read_valid+0x1b/0x20 [ 2159.980479] ? handle_bug+0x44/0x70 [ 2159.983976] ? exc_invalid_op+0x17/0x60 [ 2159.987818] ? asm_exc_invalid_op+0x1a/0x20 [ 2159.992008] ? __list_add_valid_or_report+0x7d/0xa0 [ 2159.996894] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2160.001790] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2160.007046] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2160.011850] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2160.016580] lookup_fast+0x6d/0xf0 [ 2160.019988] walk_component+0x26/0x160 [ 2160.023743] link_path_walk+0x24d/0x380 [ 2160.027578] ? path_lookupat.isra.40+0x22/0x120 [ 2160.032117] path_lookupat.isra.40+0x6b/0x120 [ 2160.036481] filename_lookup+0xd4/0x190 [ 2160.040329] ? lock_acquire+0xcb/0x2b0 [ 2160.044081] ? __might_fault+0x45/0x80 [ 2160.047836] ? find_held_lock+0x2d/0x90 [ 2160.051681] ? __might_fault+0x45/0x80 [ 2160.055443] vfs_statx+0x89/0x150 [ 2160.058768] vfs_fstatat+0x5a/0xc0 [ 2160.062180] __do_sys_newstat+0x30/0x70 [ 2160.066024] ? lock_release+0xc5/0x260 [ 2160.069782] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2160.074666] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2160.079808] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2160.084946] do_syscall_64+0x38/0x80 [ 2160.088525] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2160.093577] RIP: 0033:0x7f48f0eeca79 [ 2160.097160] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2160.115912] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2160.123478] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2160.130608] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2160.137744] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2160.144876] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2160.152009] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2160.159156] [ 2160.161349] irq event stamp: 12796455 [ 2160.165017] hardirqs last enabled at (12796465): [] console_unlock+0x106/0x120 [ 2160.173883] hardirqs last disabled at (12796474): [] console_unlock+0xeb/0x120 [ 2160.182665] softirqs last enabled at (12796488): [] __do_softirq+0x2be/0x371 [ 2160.191357] softirqs last disabled at (12796483): [] irq_exit_rcu+0xb0/0xd0 [ 2160.199882] ---[ end trace 0000000000000000 ]--- [ 2160.204505] ------------[ cut here ]------------ [ 2160.209123] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2160.220787] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2160.229481] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2160.284974] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2160.294117] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2160.301614] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2160.307126] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2160.312630] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2160.331393] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2160.336619] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2160.343768] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2160.350918] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001a1e [ 2160.358067] R10: 00000000000d2078 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2160.365209] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2160.372334] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2160.380421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2160.386184] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2160.393318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2160.400463] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2160.407609] Call Trace: [ 2160.410081] [ 2160.412201] ? __warn+0x85/0x180 [ 2160.415434] ? __list_add_valid_or_report+0x7d/0xa0 [ 2160.420315] ? report_bug+0xfc/0x1d0 [ 2160.423893] ? prb_read_valid+0x1b/0x20 [ 2160.427751] ? handle_bug+0x44/0x70 [ 2160.431260] ? exc_invalid_op+0x17/0x60 [ 2160.435099] ? asm_exc_invalid_op+0x1a/0x20 [ 2160.439308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2160.444188] ? __list_add_valid_or_report+0x7d/0xa0 [ 2160.449063] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2160.453986] ceph_add_cap+0x20b/0x620 [ceph] [ 2160.458272] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2160.463003] ? lock_acquire+0xcb/0x2b0 [ 2160.466772] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2160.471224] ? __mutex_lock+0x97/0xce0 [ 2160.474974] ? __mutex_lock+0xaec/0xce0 [ 2160.478828] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2160.483391] ? lock_downgrade+0xfd/0x250 [ 2160.487324] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2160.491719] ceph_con_process_message+0x74/0x150 [libceph] [ 2160.497214] ? calc_signature+0xdb/0x100 [libceph] [ 2160.502036] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2160.507361] ? process_scheduled_works+0x22a/0x510 [ 2160.512180] ? process_scheduled_works+0x22a/0x510 [ 2160.516974] ceph_con_workfn+0x344/0x750 [libceph] [ 2160.521780] ? process_scheduled_works+0x22a/0x510 [ 2160.526587] process_scheduled_works+0x2a4/0x510 [ 2160.531229] worker_thread+0x188/0x340 [ 2160.534976] ? rescuer_thread+0x310/0x310 [ 2160.539007] kthread+0xf5/0x130 [ 2160.542188] ? kthread_complete_and_exit+0x20/0x20 [ 2160.546980] ret_from_fork+0x31/0x40 [ 2160.550577] ? kthread_complete_and_exit+0x20/0x20 [ 2160.555387] ret_from_fork_asm+0x11/0x20 [ 2160.559319] [ 2160.561506] irq event stamp: 10229141 [ 2160.565189] hardirqs last enabled at (10229151): [] console_unlock+0x106/0x120 [ 2160.574053] hardirqs last disabled at (10229160): [] console_unlock+0xeb/0x120 [ 2160.582841] softirqs last enabled at (10228946): [] __do_softirq+0x2be/0x371 [ 2160.591555] softirqs last disabled at (10229179): [] irq_exit_rcu+0xb0/0xd0 [ 2160.600090] ---[ end trace 0000000000000000 ]--- [ 2160.604727] ------------[ cut here ]------------ [ 2160.609362] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2160.621027] WARNING: CPU: 4 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2160.629982] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2160.685478] CPU: 4 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2160.694454] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2160.701935] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2160.707437] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2160.726192] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2160.731418] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2160.738551] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2160.745700] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001a5d [ 2160.752844] R10: 00000000000d3288 R11: ffffc90001fd79c8 R12: ffff8881c5350750 [ 2160.759975] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2160.767125] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2160.775221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2160.780966] CR2: 00007fb38e6f3028 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2160.788116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2160.795268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2160.802398] Call Trace: [ 2160.804842] [ 2160.806959] ? __warn+0x85/0x180 [ 2160.810210] ? __list_add_valid_or_report+0x7d/0xa0 [ 2160.815087] ? report_bug+0xfc/0x1d0 [ 2160.818686] ? prb_read_valid+0x1b/0x20 [ 2160.822543] ? handle_bug+0x44/0x70 [ 2160.826051] ? exc_invalid_op+0x17/0x60 [ 2160.829909] ? asm_exc_invalid_op+0x1a/0x20 [ 2160.834133] ? __list_add_valid_or_report+0x7d/0xa0 [ 2160.839009] ? __list_add_valid_or_report+0x7d/0xa0 [ 2160.843904] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2160.848813] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2160.853563] lookup_fast+0x6d/0xf0 [ 2160.856989] walk_component+0x26/0x160 [ 2160.860756] link_path_walk+0x24d/0x380 [ 2160.864593] ? path_lookupat.isra.40+0x22/0x120 [ 2160.869143] path_lookupat.isra.40+0x6b/0x120 [ 2160.873525] filename_lookup+0xd4/0x190 [ 2160.877363] ? lock_acquire+0xcb/0x2b0 [ 2160.881111] ? __might_fault+0x45/0x80 [ 2160.884883] ? find_held_lock+0x2d/0x90 [ 2160.888738] ? __might_fault+0x45/0x80 [ 2160.892510] vfs_statx+0x89/0x150 [ 2160.895846] vfs_fstatat+0x5a/0xc0 [ 2160.899269] __do_sys_newstat+0x30/0x70 [ 2160.903109] ? lock_release+0xc5/0x260 [ 2160.906899] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2160.911775] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2160.916931] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2160.922090] do_syscall_64+0x38/0x80 [ 2160.925684] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2160.930767] RIP: 0033:0x7f48f0eeca79 [ 2160.934343] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2160.953092] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2160.960694] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2160.967826] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2160.974994] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2160.982124] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2160.989273] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2160.996412] [ 2160.998599] irq event stamp: 12928085 [ 2161.002278] hardirqs last enabled at (12928105): [] console_unlock+0x106/0x120 [ 2161.011144] hardirqs last disabled at (12928124): [] console_unlock+0xeb/0x120 [ 2161.019945] softirqs last enabled at (12928140): [] __do_softirq+0x2be/0x371 [ 2161.028662] softirqs last disabled at (12928153): [] irq_exit_rcu+0xb0/0xd0 [ 2161.037198] ---[ end trace 0000000000000000 ]--- [ 2161.041812] ------------[ cut here ]------------ [ 2161.046450] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2161.058101] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2161.066798] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2161.122289] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2161.131421] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2161.138901] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2161.144410] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2161.149896] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2161.168643] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2161.173869] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2161.181002] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2161.188134] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001a9f [ 2161.195267] R10: 00000000000d4588 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2161.202400] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2161.209533] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2161.217619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2161.223364] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2161.230499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2161.237632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2161.244767] Call Trace: [ 2161.247217] [ 2161.249323] ? __warn+0x85/0x180 [ 2161.252555] ? __list_add_valid_or_report+0x7d/0xa0 [ 2161.257436] ? report_bug+0xfc/0x1d0 [ 2161.261014] ? prb_read_valid+0x1b/0x20 [ 2161.264857] ? handle_bug+0x44/0x70 [ 2161.268347] ? exc_invalid_op+0x17/0x60 [ 2161.272185] ? asm_exc_invalid_op+0x1a/0x20 [ 2161.276378] ? __list_add_valid_or_report+0x7d/0xa0 [ 2161.281261] ? __list_add_valid_or_report+0x7d/0xa0 [ 2161.286138] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2161.291040] ceph_add_cap+0x20b/0x620 [ceph] [ 2161.295334] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2161.300064] ceph_fill_trace+0x143/0xf30 [ceph] [ 2161.304602] ? __mutex_lock+0x97/0xce0 [ 2161.308351] ? __mutex_lock+0xaec/0xce0 [ 2161.312193] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2161.316742] ? lock_downgrade+0xfd/0x250 [ 2161.320674] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2161.325053] ceph_con_process_message+0x74/0x150 [libceph] [ 2161.330550] ? calc_signature+0xdb/0x100 [libceph] [ 2161.335361] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2161.340683] ? process_scheduled_works+0x22a/0x510 [ 2161.345475] ? process_scheduled_works+0x22a/0x510 [ 2161.350264] ceph_con_workfn+0x344/0x750 [libceph] [ 2161.355070] ? process_scheduled_works+0x22a/0x510 [ 2161.359858] process_scheduled_works+0x2a4/0x510 [ 2161.364485] worker_thread+0x188/0x340 [ 2161.368240] ? rescuer_thread+0x310/0x310 [ 2161.372253] kthread+0xf5/0x130 [ 2161.375398] ? kthread_complete_and_exit+0x20/0x20 [ 2161.380193] ret_from_fork+0x31/0x40 [ 2161.383771] ? kthread_complete_and_exit+0x20/0x20 [ 2161.388564] ret_from_fork_asm+0x11/0x20 [ 2161.392497] [ 2161.394682] irq event stamp: 10231097 [ 2161.398347] hardirqs last enabled at (10231107): [] console_unlock+0x106/0x120 [ 2161.407216] hardirqs last disabled at (10231116): [] console_unlock+0xeb/0x120 [ 2161.415996] softirqs last enabled at (10231054): [] __do_softirq+0x2be/0x371 [ 2161.424686] softirqs last disabled at (10231049): [] irq_exit_rcu+0xb0/0xd0 [ 2161.433206] ---[ end trace 0000000000000000 ]--- [ 2161.437832] ------------[ cut here ]------------ [ 2161.442457] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2161.454112] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2161.463069] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2161.518606] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2161.527567] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2161.535044] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2161.540531] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2161.559279] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2161.564512] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2161.571645] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2161.578776] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001add [ 2161.585910] R10: 00000000000d5850 R11: ffffc90001fd7950 R12: ffff8881c5350750 [ 2161.593044] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2161.600179] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2161.608270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2161.614016] CR2: 00007f9bd3b4e000 CR3: 0000000120c1e006 CR4: 00000000003706f0 [ 2161.621148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2161.628283] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2161.635415] Call Trace: [ 2161.637867] [ 2161.639974] ? __warn+0x85/0x180 [ 2161.643208] ? __list_add_valid_or_report+0x7d/0xa0 [ 2161.648089] ? report_bug+0xfc/0x1d0 [ 2161.651674] ? prb_read_valid+0x1b/0x20 [ 2161.655519] ? handle_bug+0x44/0x70 [ 2161.659015] ? exc_invalid_op+0x17/0x60 [ 2161.662854] ? asm_exc_invalid_op+0x1a/0x20 [ 2161.667050] ? __list_add_valid_or_report+0x7d/0xa0 [ 2161.671930] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2161.676838] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2161.682091] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2161.686908] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2161.691635] lookup_fast+0x6d/0xf0 [ 2161.695046] walk_component+0x26/0x160 [ 2161.698810] link_path_walk+0x24d/0x380 [ 2161.702653] ? path_lookupat.isra.40+0x22/0x120 [ 2161.707191] path_lookupat.isra.40+0x6b/0x120 [ 2161.711556] filename_lookup+0xd4/0x190 [ 2161.715400] ? lock_acquire+0xcb/0x2b0 [ 2161.719156] ? __might_fault+0x45/0x80 [ 2161.722917] ? find_held_lock+0x2d/0x90 [ 2161.726765] ? __might_fault+0x45/0x80 [ 2161.730532] vfs_statx+0x89/0x150 [ 2161.733861] vfs_fstatat+0x5a/0xc0 [ 2161.737270] __do_sys_newstat+0x30/0x70 [ 2161.741116] ? lock_release+0xc5/0x260 [ 2161.744873] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2161.749756] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2161.754896] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2161.760037] do_syscall_64+0x38/0x80 [ 2161.763624] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2161.768676] RIP: 0033:0x7f48f0eeca79 [ 2161.772257] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2161.791004] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2161.798576] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2161.805708] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2161.812841] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2161.819976] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2161.827108] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2161.834253] [ 2161.836450] irq event stamp: 12930133 [ 2161.840116] hardirqs last enabled at (12930143): [] console_unlock+0x106/0x120 [ 2161.848981] hardirqs last disabled at (12930152): [] console_unlock+0xeb/0x120 [ 2161.857765] softirqs last enabled at (12930166): [] __do_softirq+0x2be/0x371 [ 2161.866463] softirqs last disabled at (12930161): [] irq_exit_rcu+0xb0/0xd0 [ 2161.874984] ---[ end trace 0000000000000000 ]--- [ 2161.879604] ------------[ cut here ]------------ [ 2161.884219] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2161.895884] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2161.904578] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2161.960104] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2161.969232] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2161.976710] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2161.982224] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2161.987710] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2162.006472] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2162.011699] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2162.018849] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2162.025998] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001b20 [ 2162.033141] R10: 00000000000d6f58 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2162.040290] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2162.047424] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2162.055510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2162.061274] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2162.068405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2162.075532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2162.082680] Call Trace: [ 2162.085150] [ 2162.087265] ? __warn+0x85/0x180 [ 2162.090506] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.095403] ? report_bug+0xfc/0x1d0 [ 2162.098981] ? prb_read_valid+0x1b/0x20 [ 2162.102842] ? handle_bug+0x44/0x70 [ 2162.106349] ? exc_invalid_op+0x17/0x60 [ 2162.110191] ? asm_exc_invalid_op+0x1a/0x20 [ 2162.114398] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.119274] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.124150] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2162.129060] ceph_add_cap+0x20b/0x620 [ceph] [ 2162.133362] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2162.138092] ? lock_acquire+0xcb/0x2b0 [ 2162.141862] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2162.146312] ? __mutex_lock+0x97/0xce0 [ 2162.150065] ? __mutex_lock+0xaec/0xce0 [ 2162.153922] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2162.158497] ? lock_downgrade+0xfd/0x250 [ 2162.162429] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2162.166825] ceph_con_process_message+0x74/0x150 [libceph] [ 2162.172321] ? calc_signature+0xdb/0x100 [libceph] [ 2162.177143] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2162.182466] ? process_scheduled_works+0x22a/0x510 [ 2162.187257] ? process_scheduled_works+0x22a/0x510 [ 2162.192046] ceph_con_workfn+0x344/0x750 [libceph] [ 2162.196869] ? process_scheduled_works+0x22a/0x510 [ 2162.201676] process_scheduled_works+0x2a4/0x510 [ 2162.206318] worker_thread+0x188/0x340 [ 2162.210067] ? rescuer_thread+0x310/0x310 [ 2162.214096] kthread+0xf5/0x130 [ 2162.217261] ? kthread_complete_and_exit+0x20/0x20 [ 2162.222051] ret_from_fork+0x31/0x40 [ 2162.225649] ? kthread_complete_and_exit+0x20/0x20 [ 2162.230459] ret_from_fork_asm+0x11/0x20 [ 2162.234392] [ 2162.236576] irq event stamp: 10234325 [ 2162.240252] hardirqs last enabled at (10234335): [] console_unlock+0x106/0x120 [ 2162.249119] hardirqs last disabled at (10234344): [] console_unlock+0xeb/0x120 [ 2162.257914] softirqs last enabled at (10234272): [] __do_softirq+0x2be/0x371 [ 2162.266624] softirqs last disabled at (10234267): [] irq_exit_rcu+0xb0/0xd0 [ 2162.275144] ---[ end trace 0000000000000000 ]--- [ 2162.279772] ------------[ cut here ]------------ [ 2162.284410] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2162.296078] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2162.305029] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2162.360539] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2162.369535] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2162.377018] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2162.382518] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2162.401282] RSP: 0018:ffffc90001fffb38 EFLAGS: 00010282 [ 2162.406508] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2162.413683] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2162.420809] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001b5f [ 2162.427960] R10: 00000000000d82d8 R11: ffffc90001fff9c8 R12: ffff8881c5350750 [ 2162.435108] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2162.442265] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2162.450344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2162.456091] CR2: 000055d21de75068 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2162.463242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2162.470374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2162.477505] Call Trace: [ 2162.479977] [ 2162.482101] ? __warn+0x85/0x180 [ 2162.485351] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.490234] ? report_bug+0xfc/0x1d0 [ 2162.493835] ? prb_read_valid+0x1b/0x20 [ 2162.497695] ? handle_bug+0x44/0x70 [ 2162.501186] ? exc_invalid_op+0x17/0x60 [ 2162.505055] ? asm_exc_invalid_op+0x1a/0x20 [ 2162.509245] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.514127] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.519020] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2162.523947] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2162.528684] lookup_fast+0x6d/0xf0 [ 2162.532092] walk_component+0x26/0x160 [ 2162.535883] link_path_walk+0x24d/0x380 [ 2162.539719] ? path_lookupat.isra.40+0x22/0x120 [ 2162.544272] path_lookupat.isra.40+0x6b/0x120 [ 2162.548648] filename_lookup+0xd4/0x190 [ 2162.552492] ? lock_acquire+0xcb/0x2b0 [ 2162.556249] ? __might_fault+0x45/0x80 [ 2162.560001] ? find_held_lock+0x2d/0x90 [ 2162.563856] ? __might_fault+0x45/0x80 [ 2162.567650] vfs_statx+0x89/0x150 [ 2162.570993] vfs_fstatat+0x5a/0xc0 [ 2162.574397] __do_sys_newstat+0x30/0x70 [ 2162.578235] ? lock_release+0xc5/0x260 [ 2162.582007] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2162.586883] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2162.592041] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2162.597217] do_syscall_64+0x38/0x80 [ 2162.600795] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2162.605861] RIP: 0033:0x7f48f0eeca79 [ 2162.609460] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2162.628227] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2162.635821] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2162.642948] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2162.650099] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2162.657230] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2162.664364] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2162.671537] [ 2162.673723] irq event stamp: 12840711 [ 2162.677416] hardirqs last enabled at (12840733): [] console_unlock+0x106/0x120 [ 2162.686284] hardirqs last disabled at (12840742): [] console_unlock+0xeb/0x120 [ 2162.695065] softirqs last enabled at (12840756): [] __do_softirq+0x2be/0x371 [ 2162.703773] softirqs last disabled at (12840767): [] irq_exit_rcu+0xb0/0xd0 [ 2162.712308] ---[ end trace 0000000000000000 ]--- [ 2162.716949] ------------[ cut here ]------------ [ 2162.721563] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2162.733230] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2162.741921] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2162.797415] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2162.806542] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2162.814030] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2162.819543] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2162.825045] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2162.843811] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2162.849052] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2162.856194] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2162.863336] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001ba1 [ 2162.870467] R10: 00000000000d95e8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2162.877592] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2162.884733] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2162.892828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2162.898583] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2162.905725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2162.912879] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2162.920026] Call Trace: [ 2162.922495] [ 2162.924592] ? __warn+0x85/0x180 [ 2162.927843] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.932740] ? report_bug+0xfc/0x1d0 [ 2162.936336] ? prb_read_valid+0x1b/0x20 [ 2162.940178] ? handle_bug+0x44/0x70 [ 2162.943686] ? exc_invalid_op+0x17/0x60 [ 2162.947542] ? asm_exc_invalid_op+0x1a/0x20 [ 2162.951769] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.956643] ? __list_add_valid_or_report+0x7d/0xa0 [ 2162.961529] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2162.966439] ceph_add_cap+0x20b/0x620 [ceph] [ 2162.970740] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2162.975475] ceph_fill_trace+0x143/0xf30 [ceph] [ 2162.980020] ? __mutex_lock+0x97/0xce0 [ 2162.983788] ? __mutex_lock+0xaec/0xce0 [ 2162.987645] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2162.992225] ? lock_downgrade+0xfd/0x250 [ 2162.996168] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2163.000556] ceph_con_process_message+0x74/0x150 [libceph] [ 2163.006052] ? calc_signature+0xdb/0x100 [libceph] [ 2163.010891] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2163.016237] ? process_scheduled_works+0x22a/0x510 [ 2163.021048] ? process_scheduled_works+0x22a/0x510 [ 2163.025838] ceph_con_workfn+0x344/0x750 [libceph] [ 2163.030658] ? process_scheduled_works+0x22a/0x510 [ 2163.035467] process_scheduled_works+0x2a4/0x510 [ 2163.040109] worker_thread+0x188/0x340 [ 2163.043857] ? rescuer_thread+0x310/0x310 [ 2163.047886] kthread+0xf5/0x130 [ 2163.051050] ? kthread_complete_and_exit+0x20/0x20 [ 2163.055862] ret_from_fork+0x31/0x40 [ 2163.059458] ? kthread_complete_and_exit+0x20/0x20 [ 2163.064252] ret_from_fork_asm+0x11/0x20 [ 2163.068207] [ 2163.070392] irq event stamp: 10236289 [ 2163.074051] hardirqs last enabled at (10236299): [] console_unlock+0x106/0x120 [ 2163.082937] hardirqs last disabled at (10236308): [] console_unlock+0xeb/0x120 [ 2163.091732] softirqs last enabled at (10236104): [] __do_softirq+0x2be/0x371 [ 2163.100445] softirqs last disabled at (10236327): [] irq_exit_rcu+0xb0/0xd0 [ 2163.108962] ---[ end trace 0000000000000000 ]--- [ 2163.113592] ------------[ cut here ]------------ [ 2163.118228] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2163.129872] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2163.138829] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2163.194289] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2163.203243] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2163.210719] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2163.216209] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2163.234952] RSP: 0018:ffffc90001fffac0 EFLAGS: 00010282 [ 2163.240177] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2163.247303] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2163.254436] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001bdf [ 2163.261570] R10: 00000000000da840 R11: ffffc90001fff950 R12: ffff8881c5350750 [ 2163.268700] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2163.275825] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2163.283911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2163.289657] CR2: 00007f9bd3b4e000 CR3: 000000017eb7a002 CR4: 00000000003706f0 [ 2163.296790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2163.303923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2163.311056] Call Trace: [ 2163.313507] [ 2163.315606] ? __warn+0x85/0x180 [ 2163.318840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2163.323717] ? report_bug+0xfc/0x1d0 [ 2163.327298] ? prb_read_valid+0x1b/0x20 [ 2163.331138] ? handle_bug+0x44/0x70 [ 2163.334629] ? exc_invalid_op+0x17/0x60 [ 2163.338469] ? asm_exc_invalid_op+0x1a/0x20 [ 2163.342660] ? __list_add_valid_or_report+0x7d/0xa0 [ 2163.347535] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2163.352426] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2163.357670] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2163.362482] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2163.367203] lookup_fast+0x6d/0xf0 [ 2163.370606] walk_component+0x26/0x160 [ 2163.374358] link_path_walk+0x24d/0x380 [ 2163.378198] ? path_lookupat.isra.40+0x22/0x120 [ 2163.382734] path_lookupat.isra.40+0x6b/0x120 [ 2163.387091] filename_lookup+0xd4/0x190 [ 2163.390933] ? lock_acquire+0xcb/0x2b0 [ 2163.394681] ? __might_fault+0x45/0x80 [ 2163.398436] ? find_held_lock+0x2d/0x90 [ 2163.402275] ? __might_fault+0x45/0x80 [ 2163.406029] vfs_statx+0x89/0x150 [ 2163.409350] vfs_fstatat+0x5a/0xc0 [ 2163.412753] __do_sys_newstat+0x30/0x70 [ 2163.416592] ? lock_release+0xc5/0x260 [ 2163.420357] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2163.425241] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2163.430381] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2163.435521] do_syscall_64+0x38/0x80 [ 2163.439101] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2163.444152] RIP: 0033:0x7f48f0eeca79 [ 2163.447730] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2163.466481] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2163.474043] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2163.481176] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2163.488309] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2163.495443] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2163.502576] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2163.509715] [ 2163.511901] irq event stamp: 12842781 [ 2163.515565] hardirqs last enabled at (12842791): [] console_unlock+0x106/0x120 [ 2163.524432] hardirqs last disabled at (12842800): [] console_unlock+0xeb/0x120 [ 2163.533219] softirqs last enabled at (12842814): [] __do_softirq+0x2be/0x371 [ 2163.541913] softirqs last disabled at (12842809): [] irq_exit_rcu+0xb0/0xd0 [ 2163.550433] ---[ end trace 0000000000000000 ]--- [ 2163.555045] ------------[ cut here ]------------ [ 2163.559664] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2163.571327] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2163.580022] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2163.635516] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2163.644659] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2163.652156] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2163.657667] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2163.663170] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2163.681935] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2163.687179] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2163.694328] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2163.701461] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001c22 [ 2163.708595] R10: 00000000000dbb78 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2163.715743] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2163.722886] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2163.730989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2163.736751] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2163.743895] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2163.751042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2163.758186] Call Trace: [ 2163.760657] [ 2163.762780] ? __warn+0x85/0x180 [ 2163.766029] ? __list_add_valid_or_report+0x7d/0xa0 [ 2163.770927] ? report_bug+0xfc/0x1d0 [ 2163.774535] ? prb_read_valid+0x1b/0x20 [ 2163.778390] ? handle_bug+0x44/0x70 [ 2163.781891] ? exc_invalid_op+0x17/0x60 [ 2163.785728] ? asm_exc_invalid_op+0x1a/0x20 [ 2163.789954] ? __list_add_valid_or_report+0x7d/0xa0 [ 2163.794830] ? __list_add_valid_or_report+0x7d/0xa0 [ 2163.799727] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2163.804652] ceph_add_cap+0x20b/0x620 [ceph] [ 2163.808952] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2163.813685] ? lock_acquire+0xcb/0x2b0 [ 2163.817437] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2163.821887] ? __mutex_lock+0x97/0xce0 [ 2163.825655] ? __mutex_lock+0xaec/0xce0 [ 2163.829505] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2163.834047] ? lock_downgrade+0xfd/0x250 [ 2163.838012] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2163.842384] ceph_con_process_message+0x74/0x150 [libceph] [ 2163.847880] ? calc_signature+0xdb/0x100 [libceph] [ 2163.852718] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2163.858057] ? process_scheduled_works+0x22a/0x510 [ 2163.862867] ? process_scheduled_works+0x22a/0x510 [ 2163.867656] ceph_con_workfn+0x344/0x750 [libceph] [ 2163.872477] ? process_scheduled_works+0x22a/0x510 [ 2163.877268] process_scheduled_works+0x2a4/0x510 [ 2163.881910] worker_thread+0x188/0x340 [ 2163.885658] ? rescuer_thread+0x310/0x310 [ 2163.889686] kthread+0xf5/0x130 [ 2163.892852] ? kthread_complete_and_exit+0x20/0x20 [ 2163.897663] ret_from_fork+0x31/0x40 [ 2163.901258] ? kthread_complete_and_exit+0x20/0x20 [ 2163.906051] ret_from_fork_asm+0x11/0x20 [ 2163.910018] [ 2163.912203] irq event stamp: 10238319 [ 2163.915877] hardirqs last enabled at (10238329): [] console_unlock+0x106/0x120 [ 2163.924762] hardirqs last disabled at (10238338): [] console_unlock+0xeb/0x120 [ 2163.933559] softirqs last enabled at (10238266): [] __do_softirq+0x2be/0x371 [ 2163.942268] softirqs last disabled at (10238261): [] irq_exit_rcu+0xb0/0xd0 [ 2163.950787] ---[ end trace 0000000000000000 ]--- [ 2163.955421] ------------[ cut here ]------------ [ 2163.960039] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2163.971717] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2163.980412] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2164.035888] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2164.045031] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2164.052544] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2164.058058] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2164.063561] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2164.082325] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2164.087551] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2164.094682] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2164.101825] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001c61 [ 2164.108976] R10: 00000000000dcde0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2164.116124] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2164.123267] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2164.131354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2164.137099] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2164.144252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2164.151384] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2164.158515] Call Trace: [ 2164.160967] [ 2164.163083] ? __warn+0x85/0x180 [ 2164.166333] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.171212] ? report_bug+0xfc/0x1d0 [ 2164.174808] ? prb_read_valid+0x1b/0x20 [ 2164.178650] ? handle_bug+0x44/0x70 [ 2164.182158] ? exc_invalid_op+0x17/0x60 [ 2164.186016] ? asm_exc_invalid_op+0x1a/0x20 [ 2164.190243] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.195125] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.200002] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2164.204911] ceph_add_cap+0x20b/0x620 [ceph] [ 2164.209235] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2164.213958] ceph_fill_trace+0x143/0xf30 [ceph] [ 2164.218500] ? __mutex_lock+0x97/0xce0 [ 2164.222256] ? __mutex_lock+0xaec/0xce0 [ 2164.226091] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2164.230648] ? lock_downgrade+0xfd/0x250 [ 2164.234614] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2164.239002] ceph_con_process_message+0x74/0x150 [libceph] [ 2164.244499] ? calc_signature+0xdb/0x100 [libceph] [ 2164.249304] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2164.254652] ? process_scheduled_works+0x22a/0x510 [ 2164.259443] ? process_scheduled_works+0x22a/0x510 [ 2164.264251] ceph_con_workfn+0x344/0x750 [libceph] [ 2164.269053] ? process_scheduled_works+0x22a/0x510 [ 2164.273862] process_scheduled_works+0x2a4/0x510 [ 2164.278504] worker_thread+0x188/0x340 [ 2164.282257] ? rescuer_thread+0x310/0x310 [ 2164.286263] kthread+0xf5/0x130 [ 2164.289409] ? kthread_complete_and_exit+0x20/0x20 [ 2164.294204] ret_from_fork+0x31/0x40 [ 2164.297797] ? kthread_complete_and_exit+0x20/0x20 [ 2164.302583] ret_from_fork_asm+0x11/0x20 [ 2164.306566] [ 2164.308755] irq event stamp: 10239377 [ 2164.312439] hardirqs last enabled at (10239387): [] console_unlock+0x106/0x120 [ 2164.321349] hardirqs last disabled at (10239450): [] console_unlock+0xeb/0x120 [ 2164.330128] softirqs last enabled at (10239460): [] __do_softirq+0x2be/0x371 [ 2164.338836] softirqs last disabled at (10239455): [] irq_exit_rcu+0xb0/0xd0 [ 2164.347374] ---[ end trace 0000000000000000 ]--- [ 2164.352036] ------------[ cut here ]------------ [ 2164.356665] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2164.368310] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2164.377007] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2164.432485] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2164.441607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2164.449089] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2164.454583] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2164.460069] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2164.478815] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2164.484040] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2164.491175] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2164.498307] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001c9f [ 2164.505439] R10: 00000000000de070 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2164.512572] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2164.519704] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2164.527792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2164.533537] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2164.540670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2164.547802] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2164.554928] Call Trace: [ 2164.557380] [ 2164.559486] ? __warn+0x85/0x180 [ 2164.562718] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.567598] ? report_bug+0xfc/0x1d0 [ 2164.571178] ? prb_read_valid+0x1b/0x20 [ 2164.575019] ? handle_bug+0x44/0x70 [ 2164.578509] ? exc_invalid_op+0x17/0x60 [ 2164.582351] ? asm_exc_invalid_op+0x1a/0x20 [ 2164.586540] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.591415] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.596294] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2164.601185] ceph_add_cap+0x20b/0x620 [ceph] [ 2164.605470] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2164.610192] ? lock_acquire+0xcb/0x2b0 [ 2164.613946] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2164.618395] ? __mutex_lock+0x97/0xce0 [ 2164.622146] ? __mutex_lock+0xaec/0xce0 [ 2164.625988] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2164.630529] ? lock_downgrade+0xfd/0x250 [ 2164.634459] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2164.638839] ceph_con_process_message+0x74/0x150 [libceph] [ 2164.644336] ? calc_signature+0xdb/0x100 [libceph] [ 2164.649148] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2164.654479] ? process_scheduled_works+0x22a/0x510 [ 2164.659270] ? process_scheduled_works+0x22a/0x510 [ 2164.664060] ceph_con_workfn+0x344/0x750 [libceph] [ 2164.668863] ? process_scheduled_works+0x22a/0x510 [ 2164.673653] process_scheduled_works+0x2a4/0x510 [ 2164.678279] worker_thread+0x188/0x340 [ 2164.682026] ? rescuer_thread+0x310/0x310 [ 2164.686038] kthread+0xf5/0x130 [ 2164.689186] ? kthread_complete_and_exit+0x20/0x20 [ 2164.693980] ret_from_fork+0x31/0x40 [ 2164.697557] ? kthread_complete_and_exit+0x20/0x20 [ 2164.702349] ret_from_fork_asm+0x11/0x20 [ 2164.706284] [ 2164.708477] irq event stamp: 10240545 [ 2164.712143] hardirqs last enabled at (10240555): [] console_unlock+0x106/0x120 [ 2164.721010] hardirqs last disabled at (10240564): [] console_unlock+0xeb/0x120 [ 2164.729789] softirqs last enabled at (10240492): [] __do_softirq+0x2be/0x371 [ 2164.738482] softirqs last disabled at (10240487): [] irq_exit_rcu+0xb0/0xd0 [ 2164.747001] ---[ end trace 0000000000000000 ]--- [ 2164.751620] ------------[ cut here ]------------ [ 2164.756241] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2164.767887] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2164.776841] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2164.832305] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2164.841255] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2164.848733] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2164.854219] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2164.872966] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2164.878191] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2164.885322] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2164.892447] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001cde [ 2164.899588] R10: 00000000000df158 R11: ffffc90001fcf9c8 R12: ffff8881c5350750 [ 2164.906723] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2164.913854] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2164.921940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2164.927687] CR2: 00007f9bd3b4e000 CR3: 000000011aeae003 CR4: 00000000003706f0 [ 2164.934821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2164.941954] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2164.949085] Call Trace: [ 2164.951530] [ 2164.953635] ? __warn+0x85/0x180 [ 2164.956869] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.961758] ? report_bug+0xfc/0x1d0 [ 2164.965349] ? prb_read_valid+0x1b/0x20 [ 2164.969186] ? handle_bug+0x44/0x70 [ 2164.972677] ? exc_invalid_op+0x17/0x60 [ 2164.976516] ? asm_exc_invalid_op+0x1a/0x20 [ 2164.980707] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.985583] ? __list_add_valid_or_report+0x7d/0xa0 [ 2164.990462] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2164.995351] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2165.000069] lookup_fast+0x6d/0xf0 [ 2165.003470] walk_component+0x26/0x160 [ 2165.007223] link_path_walk+0x24d/0x380 [ 2165.011063] ? path_lookupat.isra.40+0x22/0x120 [ 2165.015597] path_lookupat.isra.40+0x6b/0x120 [ 2165.019956] filename_lookup+0xd4/0x190 [ 2165.023798] ? lock_acquire+0xcb/0x2b0 [ 2165.027548] ? __might_fault+0x45/0x80 [ 2165.031301] ? find_held_lock+0x2d/0x90 [ 2165.035138] ? __might_fault+0x45/0x80 [ 2165.038894] vfs_statx+0x89/0x150 [ 2165.042213] vfs_fstatat+0x5a/0xc0 [ 2165.045617] __do_sys_newstat+0x30/0x70 [ 2165.049456] ? lock_release+0xc5/0x260 [ 2165.053210] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2165.058087] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2165.063227] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2165.068369] do_syscall_64+0x38/0x80 [ 2165.071949] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2165.076999] RIP: 0033:0x7f48f0eeca79 [ 2165.080581] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2165.099328] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2165.106890] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2165.114015] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2165.121149] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2165.128281] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2165.135413] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2165.142552] [ 2165.144739] irq event stamp: 12779471 [ 2165.148406] hardirqs last enabled at (12779481): [] console_unlock+0x106/0x120 [ 2165.157271] hardirqs last disabled at (12779490): [] console_unlock+0xeb/0x120 [ 2165.166052] softirqs last enabled at (12779504): [] __do_softirq+0x2be/0x371 [ 2165.174743] softirqs last disabled at (12779499): [] irq_exit_rcu+0xb0/0xd0 [ 2165.183265] ---[ end trace 0000000000000000 ]--- [ 2165.187884] ------------[ cut here ]------------ [ 2165.192501] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2165.204148] WARNING: CPU: 3 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2165.213104] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2165.268571] CPU: 3 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2165.277525] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2165.285002] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2165.290489] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2165.309234] RSP: 0018:ffffc90001fc7b38 EFLAGS: 00010282 [ 2165.314461] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2165.321584] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2165.328710] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001d20 [ 2165.335841] R10: 00000000000e0538 R11: ffffc90001fc79c8 R12: ffff8881c5350750 [ 2165.342965] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2165.350093] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2165.358182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2165.363924] CR2: 00007f9bd3115018 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2165.371056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2165.378188] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2165.385312] Call Trace: [ 2165.387756] [ 2165.389854] ? __warn+0x85/0x180 [ 2165.393087] ? __list_add_valid_or_report+0x7d/0xa0 [ 2165.397967] ? report_bug+0xfc/0x1d0 [ 2165.401546] ? prb_read_valid+0x1b/0x20 [ 2165.405387] ? handle_bug+0x44/0x70 [ 2165.408877] ? exc_invalid_op+0x17/0x60 [ 2165.412716] ? asm_exc_invalid_op+0x1a/0x20 [ 2165.416909] ? __list_add_valid_or_report+0x7d/0xa0 [ 2165.421784] ? __list_add_valid_or_report+0x7d/0xa0 [ 2165.426662] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2165.431552] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2165.436270] lookup_fast+0x6d/0xf0 [ 2165.439672] walk_component+0x26/0x160 [ 2165.443426] link_path_walk+0x24d/0x380 [ 2165.447263] ? path_lookupat.isra.40+0x22/0x120 [ 2165.451797] path_lookupat.isra.40+0x6b/0x120 [ 2165.456157] filename_lookup+0xd4/0x190 [ 2165.460000] ? lock_acquire+0xcb/0x2b0 [ 2165.463747] ? __might_fault+0x45/0x80 [ 2165.467501] ? find_held_lock+0x2d/0x90 [ 2165.471340] ? __might_fault+0x45/0x80 [ 2165.475096] vfs_statx+0x89/0x150 [ 2165.478416] vfs_fstatat+0x5a/0xc0 [ 2165.481819] __do_sys_newstat+0x30/0x70 [ 2165.485658] ? lock_release+0xc5/0x260 [ 2165.489413] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2165.494288] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2165.499430] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2165.504569] do_syscall_64+0x38/0x80 [ 2165.508149] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2165.513203] RIP: 0033:0x7f48f0eeca79 [ 2165.516781] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2165.535528] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2165.543091] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2165.550217] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2165.557348] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2165.564474] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2165.571605] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2165.578735] [ 2165.580922] irq event stamp: 12908057 [ 2165.584581] hardirqs last enabled at (12908067): [] console_unlock+0x106/0x120 [ 2165.593447] hardirqs last disabled at (12908076): [] console_unlock+0xeb/0x120 [ 2165.602228] softirqs last enabled at (12908092): [] __do_softirq+0x2be/0x371 [ 2165.610918] softirqs last disabled at (12908087): [] irq_exit_rcu+0xb0/0xd0 [ 2165.619443] ---[ end trace 0000000000000000 ]--- [ 2165.624061] ------------[ cut here ]------------ [ 2165.628679] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2165.640323] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2165.649018] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2165.704495] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2165.713622] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2165.721101] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2165.726597] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2165.732082] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2165.750830] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2165.756053] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2165.763186] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2165.770320] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001d62 [ 2165.777451] R10: 00000000000e1978 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2165.784585] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2165.791718] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2165.799804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2165.805551] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2165.812682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2165.819808] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2165.826938] Call Trace: [ 2165.829384] [ 2165.831493] ? __warn+0x85/0x180 [ 2165.834723] ? __list_add_valid_or_report+0x7d/0xa0 [ 2165.839602] ? report_bug+0xfc/0x1d0 [ 2165.843181] ? prb_read_valid+0x1b/0x20 [ 2165.847022] ? handle_bug+0x44/0x70 [ 2165.850515] ? exc_invalid_op+0x17/0x60 [ 2165.854354] ? asm_exc_invalid_op+0x1a/0x20 [ 2165.858544] ? __list_add_valid_or_report+0x7d/0xa0 [ 2165.863421] ? __list_add_valid_or_report+0x7d/0xa0 [ 2165.868298] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2165.873187] ceph_add_cap+0x20b/0x620 [ceph] [ 2165.877475] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2165.882202] ceph_fill_trace+0x143/0xf30 [ceph] [ 2165.886743] ? __mutex_lock+0x97/0xce0 [ 2165.890494] ? __mutex_lock+0xaec/0xce0 [ 2165.894334] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2165.898874] ? lock_downgrade+0xfd/0x250 [ 2165.902805] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2165.907176] ceph_con_process_message+0x74/0x150 [libceph] [ 2165.912672] ? calc_signature+0xdb/0x100 [libceph] [ 2165.917478] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2165.922808] ? process_scheduled_works+0x22a/0x510 [ 2165.927600] ? process_scheduled_works+0x22a/0x510 [ 2165.932389] ceph_con_workfn+0x344/0x750 [libceph] [ 2165.937192] ? process_scheduled_works+0x22a/0x510 [ 2165.941983] process_scheduled_works+0x2a4/0x510 [ 2165.946608] worker_thread+0x188/0x340 [ 2165.950356] ? rescuer_thread+0x310/0x310 [ 2165.954367] kthread+0xf5/0x130 [ 2165.957513] ? kthread_complete_and_exit+0x20/0x20 [ 2165.962308] ret_from_fork+0x31/0x40 [ 2165.965886] ? kthread_complete_and_exit+0x20/0x20 [ 2165.970681] ret_from_fork_asm+0x11/0x20 [ 2165.974613] [ 2165.976798] irq event stamp: 10243411 [ 2165.980463] hardirqs last enabled at (10243421): [] console_unlock+0x106/0x120 [ 2165.989330] hardirqs last disabled at (10243430): [] console_unlock+0xeb/0x120 [ 2165.998111] softirqs last enabled at (10243388): [] __do_softirq+0x2be/0x371 [ 2166.006802] softirqs last disabled at (10243383): [] irq_exit_rcu+0xb0/0xd0 [ 2166.015323] ---[ end trace 0000000000000000 ]--- [ 2166.019943] ------------[ cut here ]------------ [ 2166.024567] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2166.036208] WARNING: CPU: 7 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2166.045163] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2166.100629] CPU: 7 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2166.109583] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2166.117061] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2166.122547] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2166.141295] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2166.146524] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2166.153653] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2166.160784] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001da0 [ 2166.167910] R10: 00000000000e2b10 R11: ffffc90001fc7950 R12: ffff8881c5350750 [ 2166.175044] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2166.182175] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2166.190263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2166.196006] CR2: 00007fb38dad7c88 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2166.203132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2166.210264] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2166.217396] Call Trace: [ 2166.219841] [ 2166.221947] ? __warn+0x85/0x180 [ 2166.225180] ? __list_add_valid_or_report+0x7d/0xa0 [ 2166.230060] ? report_bug+0xfc/0x1d0 [ 2166.233638] ? prb_read_valid+0x1b/0x20 [ 2166.237481] ? handle_bug+0x44/0x70 [ 2166.240972] ? exc_invalid_op+0x17/0x60 [ 2166.244811] ? asm_exc_invalid_op+0x1a/0x20 [ 2166.249000] ? __list_add_valid_or_report+0x7d/0xa0 [ 2166.253878] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2166.258776] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2166.264019] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2166.268820] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2166.273535] lookup_fast+0x6d/0xf0 [ 2166.276939] walk_component+0x26/0x160 [ 2166.280693] link_path_walk+0x24d/0x380 [ 2166.284529] ? path_lookupat.isra.40+0x22/0x120 [ 2166.289056] path_lookupat.isra.40+0x6b/0x120 [ 2166.293416] filename_lookup+0xd4/0x190 [ 2166.297256] ? lock_acquire+0xcb/0x2b0 [ 2166.301007] ? __might_fault+0x45/0x80 [ 2166.304758] ? find_held_lock+0x2d/0x90 [ 2166.308599] ? __might_fault+0x45/0x80 [ 2166.312355] vfs_statx+0x89/0x150 [ 2166.315675] vfs_fstatat+0x5a/0xc0 [ 2166.319077] __do_sys_newstat+0x30/0x70 [ 2166.322917] ? lock_release+0xc5/0x260 [ 2166.326671] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2166.331547] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2166.336687] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2166.341828] do_syscall_64+0x38/0x80 [ 2166.345407] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2166.350460] RIP: 0033:0x7f48f0eeca79 [ 2166.354038] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2166.372786] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2166.380350] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2166.387482] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2166.394608] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2166.401741] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2166.408873] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2166.416018] [ 2166.418207] irq event stamp: 12910377 [ 2166.421866] hardirqs last enabled at (12910387): [] console_unlock+0x106/0x120 [ 2166.430731] hardirqs last disabled at (12910396): [] console_unlock+0xeb/0x120 [ 2166.439512] softirqs last enabled at (12910410): [] __do_softirq+0x2be/0x371 [ 2166.448203] softirqs last disabled at (12910405): [] irq_exit_rcu+0xb0/0xd0 [ 2166.456724] ---[ end trace 0000000000000000 ]--- [ 2166.461343] ------------[ cut here ]------------ [ 2166.465963] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2166.477628] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2166.486325] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2166.541807] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2166.550948] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2166.558447] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2166.563950] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2166.569455] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2166.588200] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2166.593444] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2166.600603] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2166.607733] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001de3 [ 2166.614885] R10: 00000000000e4258 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2166.622034] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2166.629184] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2166.637298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2166.643043] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2166.650194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2166.657345] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2166.664476] Call Trace: [ 2166.666929] [ 2166.669054] ? __warn+0x85/0x180 [ 2166.672304] ? __list_add_valid_or_report+0x7d/0xa0 [ 2166.677201] ? report_bug+0xfc/0x1d0 [ 2166.680780] ? prb_read_valid+0x1b/0x20 [ 2166.684656] ? handle_bug+0x44/0x70 [ 2166.688145] ? exc_invalid_op+0x17/0x60 [ 2166.692003] ? asm_exc_invalid_op+0x1a/0x20 [ 2166.696232] ? __list_add_valid_or_report+0x7d/0xa0 [ 2166.701131] ? __list_add_valid_or_report+0x7d/0xa0 [ 2166.706007] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2166.710934] ceph_add_cap+0x20b/0x620 [ceph] [ 2166.715247] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2166.719988] ? lock_acquire+0xcb/0x2b0 [ 2166.723764] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2166.728224] ? __mutex_lock+0x97/0xce0 [ 2166.731991] ? __mutex_lock+0xaec/0xce0 [ 2166.735868] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2166.740417] ? lock_downgrade+0xfd/0x250 [ 2166.744352] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2166.748746] ceph_con_process_message+0x74/0x150 [libceph] [ 2166.754239] ? calc_signature+0xdb/0x100 [libceph] [ 2166.759071] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2166.764404] ? process_scheduled_works+0x22a/0x510 [ 2166.769221] ? process_scheduled_works+0x22a/0x510 [ 2166.774016] ceph_con_workfn+0x344/0x750 [libceph] [ 2166.778857] ? process_scheduled_works+0x22a/0x510 [ 2166.783644] process_scheduled_works+0x2a4/0x510 [ 2166.788308] worker_thread+0x188/0x340 [ 2166.792081] ? rescuer_thread+0x310/0x310 [ 2166.796092] kthread+0xf5/0x130 [ 2166.799256] ? kthread_complete_and_exit+0x20/0x20 [ 2166.804076] ret_from_fork+0x31/0x40 [ 2166.807651] ? kthread_complete_and_exit+0x20/0x20 [ 2166.812463] ret_from_fork_asm+0x11/0x20 [ 2166.816400] [ 2166.818590] irq event stamp: 10245415 [ 2166.822291] hardirqs last enabled at (10245425): [] console_unlock+0x106/0x120 [ 2166.831157] hardirqs last disabled at (10245434): [] console_unlock+0xeb/0x120 [ 2166.839954] softirqs last enabled at (10245238): [] __do_softirq+0x2be/0x371 [ 2166.848664] softirqs last disabled at (10245233): [] irq_exit_rcu+0xb0/0xd0 [ 2166.857219] ---[ end trace 0000000000000000 ]--- [ 2166.861839] ------------[ cut here ]------------ [ 2166.866474] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2166.878143] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2166.887103] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2166.942659] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2166.951619] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2166.959099] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2166.964603] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2166.983367] RSP: 0018:ffffc90001fcfac0 EFLAGS: 00010282 [ 2166.988599] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2166.995749] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2167.002901] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001e22 [ 2167.010049] R10: 00000000000e54b8 R11: ffffc90001fcf950 R12: ffff8881c5350750 [ 2167.017222] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2167.024369] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2167.032461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2167.038232] CR2: 000055d21de75068 CR3: 000000011aeae006 CR4: 00000000003706e0 [ 2167.045369] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2167.052505] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2167.059641] Call Trace: [ 2167.062113] [ 2167.064235] ? __warn+0x85/0x180 [ 2167.067487] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.072369] ? report_bug+0xfc/0x1d0 [ 2167.075951] ? prb_read_valid+0x1b/0x20 [ 2167.079831] ? handle_bug+0x44/0x70 [ 2167.083322] ? exc_invalid_op+0x17/0x60 [ 2167.087161] ? asm_exc_invalid_op+0x1a/0x20 [ 2167.091372] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.096269] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2167.101174] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2167.106423] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2167.111255] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2167.116005] lookup_fast+0x6d/0xf0 [ 2167.119410] walk_component+0x26/0x160 [ 2167.123181] link_path_walk+0x24d/0x380 [ 2167.127018] ? path_lookupat.isra.40+0x22/0x120 [ 2167.131590] path_lookupat.isra.40+0x6b/0x120 [ 2167.135990] filename_lookup+0xd4/0x190 [ 2167.139851] ? lock_acquire+0xcb/0x2b0 [ 2167.143605] ? __might_fault+0x45/0x80 [ 2167.147381] ? find_held_lock+0x2d/0x90 [ 2167.151226] ? __might_fault+0x45/0x80 [ 2167.155019] vfs_statx+0x89/0x150 [ 2167.158346] vfs_fstatat+0x5a/0xc0 [ 2167.161773] __do_sys_newstat+0x30/0x70 [ 2167.165612] ? lock_release+0xc5/0x260 [ 2167.169387] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2167.174270] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2167.179439] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2167.184621] do_syscall_64+0x38/0x80 [ 2167.188208] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2167.193296] RIP: 0033:0x7f48f0eeca79 [ 2167.196871] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2167.215655] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2167.223227] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2167.230379] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2167.237512] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2167.244643] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2167.251812] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2167.258974] [ 2167.261163] irq event stamp: 12784823 [ 2167.264847] hardirqs last enabled at (12784833): [] console_unlock+0x106/0x120 [ 2167.273711] hardirqs last disabled at (12784842): [] console_unlock+0xeb/0x120 [ 2167.282513] softirqs last enabled at (12784858): [] __do_softirq+0x2be/0x371 [ 2167.291230] softirqs last disabled at (12784869): [] irq_exit_rcu+0xb0/0xd0 [ 2167.299780] ---[ end trace 0000000000000000 ]--- [ 2167.304405] ------------[ cut here ]------------ [ 2167.309024] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2167.320682] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2167.329379] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2167.384866] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2167.394007] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2167.401502] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2167.407001] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2167.412501] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2167.431250] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2167.436490] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2167.443624] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2167.450774] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001e65 [ 2167.457924] R10: 00000000000e6880 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2167.465065] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2167.472215] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2167.480321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2167.486063] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2167.493207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2167.500356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2167.507491] Call Trace: [ 2167.509943] [ 2167.512068] ? __warn+0x85/0x180 [ 2167.515318] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.520199] ? report_bug+0xfc/0x1d0 [ 2167.523794] ? prb_read_valid+0x1b/0x20 [ 2167.527671] ? handle_bug+0x44/0x70 [ 2167.531160] ? exc_invalid_op+0x17/0x60 [ 2167.535017] ? asm_exc_invalid_op+0x1a/0x20 [ 2167.539244] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.544145] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.549022] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2167.553928] ceph_add_cap+0x20b/0x620 [ceph] [ 2167.558247] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2167.562987] ceph_fill_trace+0x143/0xf30 [ceph] [ 2167.567526] ? __mutex_lock+0x97/0xce0 [ 2167.571280] ? __mutex_lock+0xaec/0xce0 [ 2167.575147] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2167.579695] ? lock_downgrade+0xfd/0x250 [ 2167.583661] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2167.588059] ceph_con_process_message+0x74/0x150 [libceph] [ 2167.593552] ? calc_signature+0xdb/0x100 [libceph] [ 2167.598367] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2167.603716] ? process_scheduled_works+0x22a/0x510 [ 2167.608517] ? process_scheduled_works+0x22a/0x510 [ 2167.613314] ceph_con_workfn+0x344/0x750 [libceph] [ 2167.618114] ? process_scheduled_works+0x22a/0x510 [ 2167.622924] process_scheduled_works+0x2a4/0x510 [ 2167.627568] worker_thread+0x188/0x340 [ 2167.631325] ? rescuer_thread+0x310/0x310 [ 2167.635336] kthread+0xf5/0x130 [ 2167.638482] ? kthread_complete_and_exit+0x20/0x20 [ 2167.643300] ret_from_fork+0x31/0x40 [ 2167.646877] ? kthread_complete_and_exit+0x20/0x20 [ 2167.651677] ret_from_fork_asm+0x11/0x20 [ 2167.655634] [ 2167.657823] irq event stamp: 10247391 [ 2167.661499] hardirqs last enabled at (10247401): [] console_unlock+0x106/0x120 [ 2167.670367] hardirqs last disabled at (10247410): [] console_unlock+0xeb/0x120 [ 2167.679145] softirqs last enabled at (10247378): [] __do_softirq+0x2be/0x371 [ 2167.687856] softirqs last disabled at (10247367): [] irq_exit_rcu+0xb0/0xd0 [ 2167.696392] ---[ end trace 0000000000000000 ]--- [ 2167.701015] ------------[ cut here ]------------ [ 2167.705650] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2167.717301] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2167.726261] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2167.781792] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2167.790750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2167.798228] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2167.803715] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2167.822461] RSP: 0018:ffffc90001fe7b38 EFLAGS: 00010282 [ 2167.827687] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2167.834826] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2167.841962] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001ea3 [ 2167.849097] R10: 00000000000e7ab0 R11: ffffc90001fe79c8 R12: ffff8881c5350750 [ 2167.856227] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2167.863361] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2167.871447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2167.877193] CR2: 00007f9bd3099000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2167.884325] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2167.891459] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2167.898589] Call Trace: [ 2167.901044] [ 2167.903150] ? __warn+0x85/0x180 [ 2167.906384] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.911264] ? report_bug+0xfc/0x1d0 [ 2167.914851] ? prb_read_valid+0x1b/0x20 [ 2167.918692] ? handle_bug+0x44/0x70 [ 2167.922182] ? exc_invalid_op+0x17/0x60 [ 2167.926023] ? asm_exc_invalid_op+0x1a/0x20 [ 2167.930219] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.935097] ? __list_add_valid_or_report+0x7d/0xa0 [ 2167.939976] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2167.944880] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2167.949612] lookup_fast+0x6d/0xf0 [ 2167.953022] walk_component+0x26/0x160 [ 2167.956774] link_path_walk+0x24d/0x380 [ 2167.960612] ? path_lookupat.isra.40+0x22/0x120 [ 2167.965149] path_lookupat.isra.40+0x6b/0x120 [ 2167.969522] filename_lookup+0xd4/0x190 [ 2167.973368] ? lock_acquire+0xcb/0x2b0 [ 2167.977121] ? __might_fault+0x45/0x80 [ 2167.980880] ? find_held_lock+0x2d/0x90 [ 2167.984724] ? __might_fault+0x45/0x80 [ 2167.988483] vfs_statx+0x89/0x150 [ 2167.991810] vfs_fstatat+0x5a/0xc0 [ 2167.995220] __do_sys_newstat+0x30/0x70 [ 2167.999058] ? lock_release+0xc5/0x260 [ 2168.002814] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2168.007698] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2168.012840] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2168.017988] do_syscall_64+0x38/0x80 [ 2168.021576] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2168.026626] RIP: 0033:0x7f48f0eeca79 [ 2168.030205] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2168.048953] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2168.056518] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2168.063651] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2168.070783] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2168.077917] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2168.085049] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2168.092195] [ 2168.094384] irq event stamp: 12806279 [ 2168.098049] hardirqs last enabled at (12806289): [] console_unlock+0x106/0x120 [ 2168.106915] hardirqs last disabled at (12806298): [] console_unlock+0xeb/0x120 [ 2168.115696] softirqs last enabled at (12806312): [] __do_softirq+0x2be/0x371 [ 2168.124395] softirqs last disabled at (12806307): [] irq_exit_rcu+0xb0/0xd0 [ 2168.132917] ---[ end trace 0000000000000000 ]--- [ 2168.137535] ------------[ cut here ]------------ [ 2168.142156] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2168.153805] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2168.162506] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2168.217988] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2168.227113] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2168.234595] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2168.240097] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2168.245584] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2168.264331] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2168.269556] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2168.276691] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2168.283820] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001ee5 [ 2168.290954] R10: 00000000000e8c78 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2168.298087] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2168.305222] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2168.313307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2168.319052] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2168.326184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2168.333319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2168.340452] Call Trace: [ 2168.342904] [ 2168.345011] ? __warn+0x85/0x180 [ 2168.348243] ? __list_add_valid_or_report+0x7d/0xa0 [ 2168.353124] ? report_bug+0xfc/0x1d0 [ 2168.356702] ? prb_read_valid+0x1b/0x20 [ 2168.360544] ? handle_bug+0x44/0x70 [ 2168.364035] ? exc_invalid_op+0x17/0x60 [ 2168.367873] ? asm_exc_invalid_op+0x1a/0x20 [ 2168.372068] ? __list_add_valid_or_report+0x7d/0xa0 [ 2168.376951] ? __list_add_valid_or_report+0x7d/0xa0 [ 2168.381835] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2168.386737] ceph_add_cap+0x20b/0x620 [ceph] [ 2168.391033] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2168.395755] ? lock_acquire+0xcb/0x2b0 [ 2168.399515] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2168.403972] ? __mutex_lock+0x97/0xce0 [ 2168.407723] ? __mutex_lock+0xaec/0xce0 [ 2168.411567] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2168.416113] ? lock_downgrade+0xfd/0x250 [ 2168.420048] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2168.424436] ceph_con_process_message+0x74/0x150 [libceph] [ 2168.429937] ? calc_signature+0xdb/0x100 [libceph] [ 2168.434753] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2168.440085] ? process_scheduled_works+0x22a/0x510 [ 2168.444883] ? process_scheduled_works+0x22a/0x510 [ 2168.449679] ceph_con_workfn+0x344/0x750 [libceph] [ 2168.454494] ? process_scheduled_works+0x22a/0x510 [ 2168.459288] process_scheduled_works+0x2a4/0x510 [ 2168.463919] worker_thread+0x188/0x340 [ 2168.467674] ? rescuer_thread+0x310/0x310 [ 2168.471683] kthread+0xf5/0x130 [ 2168.474831] ? kthread_complete_and_exit+0x20/0x20 [ 2168.479625] ret_from_fork+0x31/0x40 [ 2168.483202] ? kthread_complete_and_exit+0x20/0x20 [ 2168.487996] ret_from_fork_asm+0x11/0x20 [ 2168.491932] [ 2168.494124] irq event stamp: 10249399 [ 2168.497788] hardirqs last enabled at (10249409): [] console_unlock+0x106/0x120 [ 2168.506655] hardirqs last disabled at (10249418): [] console_unlock+0xeb/0x120 [ 2168.515436] softirqs last enabled at (10249206): [] __do_softirq+0x2be/0x371 [ 2168.524125] softirqs last disabled at (10249201): [] irq_exit_rcu+0xb0/0xd0 [ 2168.532649] ---[ end trace 0000000000000000 ]--- [ 2168.537266] ------------[ cut here ]------------ [ 2168.541883] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2168.553551] WARNING: CPU: 0 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2168.562505] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2168.617995] CPU: 0 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2168.626952] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2168.634447] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2168.639933] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2168.658678] RSP: 0018:ffffc90001fe7ac0 EFLAGS: 00010282 [ 2168.663905] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2168.671037] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2168.678171] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001f24 [ 2168.685326] R10: 00000000000e9f38 R11: ffffc90001fe7950 R12: ffff8881c5350750 [ 2168.692464] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2168.699596] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2168.707698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2168.713443] CR2: 00007f9bd3b4e000 CR3: 0000000108c08001 CR4: 00000000003706f0 [ 2168.720578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2168.727710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2168.734843] Call Trace: [ 2168.737296] [ 2168.739404] ? __warn+0x85/0x180 [ 2168.742636] ? __list_add_valid_or_report+0x7d/0xa0 [ 2168.747518] ? report_bug+0xfc/0x1d0 [ 2168.751103] ? prb_read_valid+0x1b/0x20 [ 2168.754946] ? handle_bug+0x44/0x70 [ 2168.758434] ? exc_invalid_op+0x17/0x60 [ 2168.762275] ? asm_exc_invalid_op+0x1a/0x20 [ 2168.766467] ? __list_add_valid_or_report+0x7d/0xa0 [ 2168.771352] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2168.776247] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2168.781498] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2168.786330] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2168.791055] lookup_fast+0x6d/0xf0 [ 2168.794464] walk_component+0x26/0x160 [ 2168.798220] link_path_walk+0x24d/0x380 [ 2168.802065] ? path_lookupat.isra.40+0x22/0x120 [ 2168.806618] path_lookupat.isra.40+0x6b/0x120 [ 2168.810987] filename_lookup+0xd4/0x190 [ 2168.814838] ? lock_acquire+0xcb/0x2b0 [ 2168.818592] ? __might_fault+0x45/0x80 [ 2168.822346] ? find_held_lock+0x2d/0x90 [ 2168.826185] ? __might_fault+0x45/0x80 [ 2168.829943] vfs_statx+0x89/0x150 [ 2168.833268] vfs_fstatat+0x5a/0xc0 [ 2168.836686] __do_sys_newstat+0x30/0x70 [ 2168.840545] ? lock_release+0xc5/0x260 [ 2168.844301] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2168.849201] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2168.854344] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2168.859486] do_syscall_64+0x38/0x80 [ 2168.863068] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2168.868123] RIP: 0033:0x7f48f0eeca79 [ 2168.871699] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2168.890453] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2168.898023] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2168.905155] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2168.912287] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2168.919420] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2168.926553] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2168.933698] [ 2168.935887] irq event stamp: 12808369 [ 2168.939555] hardirqs last enabled at (12808379): [] console_unlock+0x106/0x120 [ 2168.948422] hardirqs last disabled at (12808388): [] console_unlock+0xeb/0x120 [ 2168.957203] softirqs last enabled at (12808402): [] __do_softirq+0x2be/0x371 [ 2168.965900] softirqs last disabled at (12808397): [] irq_exit_rcu+0xb0/0xd0 [ 2168.974423] ---[ end trace 0000000000000000 ]--- [ 2168.979041] ------------[ cut here ]------------ [ 2168.983661] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2168.995313] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2169.004026] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2169.059567] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2169.068702] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2169.076179] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2169.081682] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2169.087167] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2169.105916] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2169.111142] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2169.118272] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2169.125407] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001f67 [ 2169.132540] R10: 00000000000eb230 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2169.139671] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2169.146803] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2169.154891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2169.160638] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2169.167770] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2169.174903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2169.182038] Call Trace: [ 2169.184489] [ 2169.186595] ? __warn+0x85/0x180 [ 2169.189844] ? __list_add_valid_or_report+0x7d/0xa0 [ 2169.194726] ? report_bug+0xfc/0x1d0 [ 2169.198304] ? prb_read_valid+0x1b/0x20 [ 2169.202147] ? handle_bug+0x44/0x70 [ 2169.205644] ? exc_invalid_op+0x17/0x60 [ 2169.209487] ? asm_exc_invalid_op+0x1a/0x20 [ 2169.213690] ? __list_add_valid_or_report+0x7d/0xa0 [ 2169.218580] ? __list_add_valid_or_report+0x7d/0xa0 [ 2169.223462] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2169.228364] ceph_add_cap+0x20b/0x620 [ceph] [ 2169.232672] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2169.237414] ceph_fill_trace+0x143/0xf30 [ceph] [ 2169.241961] ? __mutex_lock+0x97/0xce0 [ 2169.245721] ? __mutex_lock+0xaec/0xce0 [ 2169.249564] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2169.254120] ? lock_downgrade+0xfd/0x250 [ 2169.258055] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2169.262447] ceph_con_process_message+0x74/0x150 [libceph] [ 2169.267950] ? calc_signature+0xdb/0x100 [libceph] [ 2169.272767] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2169.278106] ? process_scheduled_works+0x22a/0x510 [ 2169.282907] ? process_scheduled_works+0x22a/0x510 [ 2169.287702] ceph_con_workfn+0x344/0x750 [libceph] [ 2169.292518] ? process_scheduled_works+0x22a/0x510 [ 2169.297317] process_scheduled_works+0x2a4/0x510 [ 2169.301953] worker_thread+0x188/0x340 [ 2169.305714] ? rescuer_thread+0x310/0x310 [ 2169.309734] kthread+0xf5/0x130 [ 2169.312879] ? kthread_complete_and_exit+0x20/0x20 [ 2169.317676] ret_from_fork+0x31/0x40 [ 2169.321261] ? kthread_complete_and_exit+0x20/0x20 [ 2169.326055] ret_from_fork_asm+0x11/0x20 [ 2169.330002] [ 2169.332198] irq event stamp: 10251353 [ 2169.335863] hardirqs last enabled at (10251363): [] console_unlock+0x106/0x120 [ 2169.344728] hardirqs last disabled at (10251372): [] console_unlock+0xeb/0x120 [ 2169.353510] softirqs last enabled at (10251310): [] __do_softirq+0x2be/0x371 [ 2169.362209] softirqs last disabled at (10251305): [] irq_exit_rcu+0xb0/0xd0 [ 2169.370729] ---[ end trace 0000000000000000 ]--- [ 2169.375369] ------------[ cut here ]------------ [ 2169.380001] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2169.391724] WARNING: CPU: 3 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2169.400681] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2169.456203] CPU: 3 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2169.465154] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2169.472634] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2169.478121] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2169.496871] RSP: 0018:ffffc90001957b38 EFLAGS: 00010282 [ 2169.502102] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2169.509235] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2169.516367] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001fa5 [ 2169.523500] R10: 00000000000ec348 R11: ffffc900019579c8 R12: ffff8881c5350750 [ 2169.530631] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2169.537764] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2169.545850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2169.551597] CR2: 00007fb38dd610d8 CR3: 000000016696c006 CR4: 00000000003706e0 [ 2169.558730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2169.565861] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2169.572996] Call Trace: [ 2169.575448] [ 2169.577554] ? __warn+0x85/0x180 [ 2169.580787] ? __list_add_valid_or_report+0x7d/0xa0 [ 2169.585669] ? report_bug+0xfc/0x1d0 [ 2169.589246] ? prb_read_valid+0x1b/0x20 [ 2169.593089] ? handle_bug+0x44/0x70 [ 2169.596587] ? exc_invalid_op+0x17/0x60 [ 2169.600428] ? asm_exc_invalid_op+0x1a/0x20 [ 2169.604621] ? __list_add_valid_or_report+0x7d/0xa0 [ 2169.609503] ? __list_add_valid_or_report+0x7d/0xa0 [ 2169.614380] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2169.619281] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2169.624011] lookup_fast+0x6d/0xf0 [ 2169.627417] walk_component+0x26/0x160 [ 2169.631172] link_path_walk+0x24d/0x380 [ 2169.635017] ? path_lookupat.isra.40+0x22/0x120 [ 2169.639553] path_lookupat.isra.40+0x6b/0x120 [ 2169.643919] filename_lookup+0xd4/0x190 [ 2169.647764] ? lock_acquire+0xcb/0x2b0 [ 2169.651519] ? __might_fault+0x45/0x80 [ 2169.655273] ? find_held_lock+0x2d/0x90 [ 2169.659119] ? __might_fault+0x45/0x80 [ 2169.662879] vfs_statx+0x89/0x150 [ 2169.666205] vfs_fstatat+0x5a/0xc0 [ 2169.669618] __do_sys_newstat+0x30/0x70 [ 2169.673462] ? lock_release+0xc5/0x260 [ 2169.677218] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2169.682101] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2169.687244] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2169.692384] do_syscall_64+0x38/0x80 [ 2169.695964] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2169.701025] RIP: 0033:0x7f48f0eeca79 [ 2169.704611] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2169.723356] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2169.730923] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2169.738055] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2169.745189] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2169.752333] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2169.759469] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2169.766618] [ 2169.768814] irq event stamp: 12779915 [ 2169.772480] hardirqs last enabled at (12779925): [] console_unlock+0x106/0x120 [ 2169.781346] hardirqs last disabled at (12779934): [] console_unlock+0xeb/0x120 [ 2169.790125] softirqs last enabled at (12779892): [] __do_softirq+0x2be/0x371 [ 2169.798818] softirqs last disabled at (12779885): [] irq_exit_rcu+0xb0/0xd0 [ 2169.807338] ---[ end trace 0000000000000000 ]--- [ 2169.811957] ------------[ cut here ]------------ [ 2169.816574] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2169.828224] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2169.836915] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2169.892394] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2169.901518] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2169.908997] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2169.914492] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2169.919979] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2169.938727] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2169.943952] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2169.951086] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2169.958218] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100001fe7 [ 2169.965351] R10: 00000000000eda20 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2169.972481] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2169.979613] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2169.987701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2169.993448] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2170.000579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2170.007711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2170.014844] Call Trace: [ 2170.017292] [ 2170.019397] ? __warn+0x85/0x180 [ 2170.022629] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.027509] ? report_bug+0xfc/0x1d0 [ 2170.031087] ? prb_read_valid+0x1b/0x20 [ 2170.034930] ? handle_bug+0x44/0x70 [ 2170.038420] ? exc_invalid_op+0x17/0x60 [ 2170.042261] ? asm_exc_invalid_op+0x1a/0x20 [ 2170.046450] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.051331] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.056212] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2170.061105] ceph_add_cap+0x20b/0x620 [ceph] [ 2170.065399] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2170.070120] ? lock_acquire+0xcb/0x2b0 [ 2170.073872] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2170.078324] ? __mutex_lock+0x97/0xce0 [ 2170.082083] ? __mutex_lock+0xaec/0xce0 [ 2170.085927] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2170.090472] ? lock_downgrade+0xfd/0x250 [ 2170.094404] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2170.098783] ceph_con_process_message+0x74/0x150 [libceph] [ 2170.104280] ? calc_signature+0xdb/0x100 [libceph] [ 2170.109093] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2170.114423] ? process_scheduled_works+0x22a/0x510 [ 2170.119215] ? process_scheduled_works+0x22a/0x510 [ 2170.124006] ceph_con_workfn+0x344/0x750 [libceph] [ 2170.128810] ? process_scheduled_works+0x22a/0x510 [ 2170.133599] process_scheduled_works+0x2a4/0x510 [ 2170.138224] worker_thread+0x188/0x340 [ 2170.141980] ? rescuer_thread+0x310/0x310 [ 2170.145994] kthread+0xf5/0x130 [ 2170.149140] ? kthread_complete_and_exit+0x20/0x20 [ 2170.153931] ret_from_fork+0x31/0x40 [ 2170.157512] ? kthread_complete_and_exit+0x20/0x20 [ 2170.162303] ret_from_fork_asm+0x11/0x20 [ 2170.166237] [ 2170.168431] irq event stamp: 10253381 [ 2170.172096] hardirqs last enabled at (10253391): [] console_unlock+0x106/0x120 [ 2170.180964] hardirqs last disabled at (10253400): [] console_unlock+0xeb/0x120 [ 2170.189743] softirqs last enabled at (10253188): [] __do_softirq+0x2be/0x371 [ 2170.198435] softirqs last disabled at (10253183): [] irq_exit_rcu+0xb0/0xd0 [ 2170.206957] ---[ end trace 0000000000000000 ]--- [ 2170.211576] ------------[ cut here ]------------ [ 2170.216194] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2170.227845] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2170.236806] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2170.292276] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2170.301236] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2170.308714] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2170.314201] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2170.332953] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2170.338190] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2170.345341] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2170.352473] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002026 [ 2170.359607] R10: 00000000000eec30 R11: ffffc90001957950 R12: ffff8881c5350750 [ 2170.366738] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2170.373869] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2170.381956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2170.387702] CR2: 00007f9bd3b4e000 CR3: 000000016696c001 CR4: 00000000003706f0 [ 2170.394835] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2170.401967] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2170.409103] Call Trace: [ 2170.411552] [ 2170.413661] ? __warn+0x85/0x180 [ 2170.416892] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.421775] ? report_bug+0xfc/0x1d0 [ 2170.425361] ? prb_read_valid+0x1b/0x20 [ 2170.429205] ? handle_bug+0x44/0x70 [ 2170.432701] ? exc_invalid_op+0x17/0x60 [ 2170.436543] ? asm_exc_invalid_op+0x1a/0x20 [ 2170.440733] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.445620] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2170.450515] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2170.455769] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2170.460577] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2170.465306] lookup_fast+0x6d/0xf0 [ 2170.468711] walk_component+0x26/0x160 [ 2170.472469] link_path_walk+0x24d/0x380 [ 2170.476312] ? path_lookupat.isra.40+0x22/0x120 [ 2170.480852] path_lookupat.isra.40+0x6b/0x120 [ 2170.485217] filename_lookup+0xd4/0x190 [ 2170.489061] ? lock_acquire+0xcb/0x2b0 [ 2170.492814] ? __might_fault+0x45/0x80 [ 2170.496570] ? find_held_lock+0x2d/0x90 [ 2170.500415] ? __might_fault+0x45/0x80 [ 2170.504175] vfs_statx+0x89/0x150 [ 2170.507500] vfs_fstatat+0x5a/0xc0 [ 2170.510914] __do_sys_newstat+0x30/0x70 [ 2170.514759] ? lock_release+0xc5/0x260 [ 2170.518516] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2170.523399] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2170.528541] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2170.533688] do_syscall_64+0x38/0x80 [ 2170.537268] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2170.542319] RIP: 0033:0x7f48f0eeca79 [ 2170.545898] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2170.564646] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2170.572211] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2170.579346] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2170.586477] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2170.593608] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2170.600743] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2170.607886] [ 2170.610077] irq event stamp: 12782017 [ 2170.613740] hardirqs last enabled at (12782027): [] console_unlock+0x106/0x120 [ 2170.622608] hardirqs last disabled at (12782036): [] console_unlock+0xeb/0x120 [ 2170.631389] softirqs last enabled at (12782050): [] __do_softirq+0x2be/0x371 [ 2170.640082] softirqs last disabled at (12782045): [] irq_exit_rcu+0xb0/0xd0 [ 2170.648609] ---[ end trace 0000000000000000 ]--- [ 2170.653231] ------------[ cut here ]------------ [ 2170.657845] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2170.669493] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2170.678189] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2170.733685] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2170.742809] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2170.750304] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2170.755814] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2170.761317] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2170.780082] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2170.785326] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2170.792477] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2170.799610] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002069 [ 2170.806742] R10: 00000000000f0060 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2170.813892] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2170.821033] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2170.829127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2170.834883] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2170.842024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2170.849176] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2170.856327] Call Trace: [ 2170.858796] [ 2170.860918] ? __warn+0x85/0x180 [ 2170.864169] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.869067] ? report_bug+0xfc/0x1d0 [ 2170.872663] ? prb_read_valid+0x1b/0x20 [ 2170.876520] ? handle_bug+0x44/0x70 [ 2170.880011] ? exc_invalid_op+0x17/0x60 [ 2170.883859] ? asm_exc_invalid_op+0x1a/0x20 [ 2170.888069] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.892944] ? __list_add_valid_or_report+0x7d/0xa0 [ 2170.897839] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2170.902765] ceph_add_cap+0x20b/0x620 [ceph] [ 2170.907066] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2170.911809] ceph_fill_trace+0x143/0xf30 [ceph] [ 2170.916370] ? __mutex_lock+0x97/0xce0 [ 2170.920121] ? __mutex_lock+0xaec/0xce0 [ 2170.923979] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2170.928538] ? lock_downgrade+0xfd/0x250 [ 2170.932467] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2170.936865] ceph_con_process_message+0x74/0x150 [libceph] [ 2170.942377] ? calc_signature+0xdb/0x100 [libceph] [ 2170.947199] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2170.952523] ? process_scheduled_works+0x22a/0x510 [ 2170.957331] ? process_scheduled_works+0x22a/0x510 [ 2170.962120] ceph_con_workfn+0x344/0x750 [libceph] [ 2170.966941] ? process_scheduled_works+0x22a/0x510 [ 2170.971748] process_scheduled_works+0x2a4/0x510 [ 2170.976393] worker_thread+0x188/0x340 [ 2170.980150] ? rescuer_thread+0x310/0x310 [ 2170.984178] kthread+0xf5/0x130 [ 2170.987343] ? kthread_complete_and_exit+0x20/0x20 [ 2170.992155] ret_from_fork+0x31/0x40 [ 2170.995731] ? kthread_complete_and_exit+0x20/0x20 [ 2171.000542] ret_from_fork_asm+0x11/0x20 [ 2171.004475] [ 2171.006668] irq event stamp: 10255353 [ 2171.010343] hardirqs last enabled at (10255363): [] console_unlock+0x106/0x120 [ 2171.019225] hardirqs last disabled at (10255374): [] console_unlock+0xeb/0x120 [ 2171.028007] softirqs last enabled at (10255310): [] __do_softirq+0x2be/0x371 [ 2171.036717] softirqs last disabled at (10255305): [] irq_exit_rcu+0xb0/0xd0 [ 2171.045253] ---[ end trace 0000000000000000 ]--- [ 2171.049935] ------------[ cut here ]------------ [ 2171.054578] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2171.066228] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2171.075180] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2171.130642] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2171.139591] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2171.147069] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2171.152556] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2171.171305] RSP: 0018:ffffc90001ff7b38 EFLAGS: 00010282 [ 2171.176528] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2171.183661] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2171.190785] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000020a7 [ 2171.197909] R10: 00000000000f1138 R11: ffffc90001ff79c8 R12: ffff8881c5350750 [ 2171.205033] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2171.212158] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2171.220243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2171.225981] CR2: 00007fb38dd7a000 CR3: 0000000120cc8004 CR4: 00000000003706e0 [ 2171.233105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2171.240238] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2171.247362] Call Trace: [ 2171.249807] [ 2171.251912] ? __warn+0x85/0x180 [ 2171.255145] ? __list_add_valid_or_report+0x7d/0xa0 [ 2171.260025] ? report_bug+0xfc/0x1d0 [ 2171.263603] ? prb_read_valid+0x1b/0x20 [ 2171.267446] ? handle_bug+0x44/0x70 [ 2171.270938] ? exc_invalid_op+0x17/0x60 [ 2171.274776] ? asm_exc_invalid_op+0x1a/0x20 [ 2171.278967] ? __list_add_valid_or_report+0x7d/0xa0 [ 2171.283842] ? __list_add_valid_or_report+0x7d/0xa0 [ 2171.288722] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2171.293611] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2171.298327] lookup_fast+0x6d/0xf0 [ 2171.301731] walk_component+0x26/0x160 [ 2171.305485] link_path_walk+0x24d/0x380 [ 2171.309321] ? path_lookupat.isra.40+0x22/0x120 [ 2171.313849] path_lookupat.isra.40+0x6b/0x120 [ 2171.318207] filename_lookup+0xd4/0x190 [ 2171.322048] ? lock_acquire+0xcb/0x2b0 [ 2171.325797] ? __might_fault+0x45/0x80 [ 2171.329552] ? find_held_lock+0x2d/0x90 [ 2171.333389] ? __might_fault+0x45/0x80 [ 2171.337144] vfs_statx+0x89/0x150 [ 2171.340466] vfs_fstatat+0x5a/0xc0 [ 2171.343870] __do_sys_newstat+0x30/0x70 [ 2171.347708] ? lock_release+0xc5/0x260 [ 2171.351461] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2171.356338] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2171.361480] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2171.366620] do_syscall_64+0x38/0x80 [ 2171.370198] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2171.375251] RIP: 0033:0x7f48f0eeca79 [ 2171.378837] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2171.397576] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2171.405143] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2171.412275] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2171.419408] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2171.426543] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2171.433673] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2171.440804] [ 2171.442991] irq event stamp: 12857775 [ 2171.446656] hardirqs last enabled at (12857785): [] console_unlock+0x106/0x120 [ 2171.455523] hardirqs last disabled at (12857794): [] console_unlock+0xeb/0x120 [ 2171.464302] softirqs last enabled at (12857742): [] __do_softirq+0x2be/0x371 [ 2171.472994] softirqs last disabled at (12857737): [] irq_exit_rcu+0xb0/0xd0 [ 2171.481514] ---[ end trace 0000000000000000 ]--- [ 2171.486135] ------------[ cut here ]------------ [ 2171.490752] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2171.502405] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2171.511099] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2171.566582] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2171.575705] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2171.583186] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2171.588690] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2171.594177] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2171.612922] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2171.618148] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2171.625285] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2171.632416] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000020e9 [ 2171.639548] R10: 00000000000f2468 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2171.646678] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2171.653812] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2171.661897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2171.667644] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2171.674778] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2171.681910] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2171.689041] Call Trace: [ 2171.691495] [ 2171.693602] ? __warn+0x85/0x180 [ 2171.696835] ? __list_add_valid_or_report+0x7d/0xa0 [ 2171.701717] ? report_bug+0xfc/0x1d0 [ 2171.705293] ? prb_read_valid+0x1b/0x20 [ 2171.709136] ? handle_bug+0x44/0x70 [ 2171.712626] ? exc_invalid_op+0x17/0x60 [ 2171.716467] ? asm_exc_invalid_op+0x1a/0x20 [ 2171.720659] ? __list_add_valid_or_report+0x7d/0xa0 [ 2171.725544] ? __list_add_valid_or_report+0x7d/0xa0 [ 2171.730418] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2171.735311] ceph_add_cap+0x20b/0x620 [ceph] [ 2171.739605] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2171.744328] ? lock_acquire+0xcb/0x2b0 [ 2171.748092] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2171.752550] ? __mutex_lock+0x97/0xce0 [ 2171.756299] ? __mutex_lock+0xaec/0xce0 [ 2171.760141] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2171.764690] ? lock_downgrade+0xfd/0x250 [ 2171.768619] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2171.772998] ceph_con_process_message+0x74/0x150 [libceph] [ 2171.778494] ? calc_signature+0xdb/0x100 [libceph] [ 2171.783310] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2171.788637] ? process_scheduled_works+0x22a/0x510 [ 2171.793428] ? process_scheduled_works+0x22a/0x510 [ 2171.798220] ceph_con_workfn+0x344/0x750 [libceph] [ 2171.803022] ? process_scheduled_works+0x22a/0x510 [ 2171.807812] process_scheduled_works+0x2a4/0x510 [ 2171.812436] worker_thread+0x188/0x340 [ 2171.816188] ? rescuer_thread+0x310/0x310 [ 2171.820200] kthread+0xf5/0x130 [ 2171.823346] ? kthread_complete_and_exit+0x20/0x20 [ 2171.828139] ret_from_fork+0x31/0x40 [ 2171.831717] ? kthread_complete_and_exit+0x20/0x20 [ 2171.836510] ret_from_fork_asm+0x11/0x20 [ 2171.840442] [ 2171.842635] irq event stamp: 10257373 [ 2171.846302] hardirqs last enabled at (10257383): [] console_unlock+0x106/0x120 [ 2171.855167] hardirqs last disabled at (10257392): [] console_unlock+0xeb/0x120 [ 2171.863949] softirqs last enabled at (10257180): [] __do_softirq+0x2be/0x371 [ 2171.872641] softirqs last disabled at (10257175): [] irq_exit_rcu+0xb0/0xd0 [ 2171.881162] ---[ end trace 0000000000000000 ]--- [ 2171.885781] ------------[ cut here ]------------ [ 2171.890399] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2171.902045] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2171.910999] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2171.966464] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2171.975410] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2171.982880] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2171.988368] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2172.007113] RSP: 0018:ffffc90001f7fb38 EFLAGS: 00010282 [ 2172.012339] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2172.019464] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2172.026596] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002128 [ 2172.033722] R10: 00000000000f3580 R11: ffffc90001f7f9c8 R12: ffff8881c5350750 [ 2172.040853] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2172.047978] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2172.056063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2172.061801] CR2: 00007ffd86bab221 CR3: 0000000113532005 CR4: 00000000003706e0 [ 2172.068926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2172.076058] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2172.083182] Call Trace: [ 2172.085628] [ 2172.087734] ? __warn+0x85/0x180 [ 2172.090966] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.095847] ? report_bug+0xfc/0x1d0 [ 2172.099425] ? prb_read_valid+0x1b/0x20 [ 2172.103266] ? handle_bug+0x44/0x70 [ 2172.106755] ? exc_invalid_op+0x17/0x60 [ 2172.110597] ? asm_exc_invalid_op+0x1a/0x20 [ 2172.114786] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.119662] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.124543] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2172.129432] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2172.134148] lookup_fast+0x6d/0xf0 [ 2172.137552] walk_component+0x26/0x160 [ 2172.141305] link_path_walk+0x24d/0x380 [ 2172.145143] ? path_lookupat.isra.40+0x22/0x120 [ 2172.149676] path_lookupat.isra.40+0x6b/0x120 [ 2172.154037] filename_lookup+0xd4/0x190 [ 2172.157876] ? lock_acquire+0xcb/0x2b0 [ 2172.161625] ? __might_fault+0x45/0x80 [ 2172.165381] ? find_held_lock+0x2d/0x90 [ 2172.169218] ? __might_fault+0x45/0x80 [ 2172.172975] vfs_statx+0x89/0x150 [ 2172.176292] vfs_fstatat+0x5a/0xc0 [ 2172.179697] __do_sys_newstat+0x30/0x70 [ 2172.183537] ? lock_release+0xc5/0x260 [ 2172.187290] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2172.192167] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2172.197309] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2172.202449] do_syscall_64+0x38/0x80 [ 2172.206027] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2172.211078] RIP: 0033:0x7f48f0eeca79 [ 2172.214659] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2172.233406] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2172.240970] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2172.248095] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2172.255227] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2172.262360] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2172.269485] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2172.276624] [ 2172.278810] irq event stamp: 12902889 [ 2172.282470] hardirqs last enabled at (12902899): [] console_unlock+0x106/0x120 [ 2172.291335] hardirqs last disabled at (12902908): [] console_unlock+0xeb/0x120 [ 2172.300116] softirqs last enabled at (12902922): [] __do_softirq+0x2be/0x371 [ 2172.308805] softirqs last disabled at (12902917): [] irq_exit_rcu+0xb0/0xd0 [ 2172.317325] ---[ end trace 0000000000000000 ]--- [ 2172.321939] ------------[ cut here ]------------ [ 2172.326556] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2172.338206] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2172.346901] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2172.402386] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2172.411508] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2172.418990] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2172.424491] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2172.429978] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2172.448724] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2172.453952] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2172.461084] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2172.468218] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000216a [ 2172.475349] R10: 00000000000f48d0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2172.482481] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2172.489614] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2172.497700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2172.503447] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2172.510580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2172.517713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2172.524849] Call Trace: [ 2172.527300] [ 2172.529406] ? __warn+0x85/0x180 [ 2172.532640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.537519] ? report_bug+0xfc/0x1d0 [ 2172.541098] ? prb_read_valid+0x1b/0x20 [ 2172.544941] ? handle_bug+0x44/0x70 [ 2172.548428] ? exc_invalid_op+0x17/0x60 [ 2172.552268] ? asm_exc_invalid_op+0x1a/0x20 [ 2172.556460] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.561344] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.566223] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2172.571122] ceph_add_cap+0x20b/0x620 [ceph] [ 2172.575417] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2172.580145] ceph_fill_trace+0x143/0xf30 [ceph] [ 2172.584683] ? __mutex_lock+0x97/0xce0 [ 2172.588433] ? __mutex_lock+0xaec/0xce0 [ 2172.592278] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2172.596825] ? lock_downgrade+0xfd/0x250 [ 2172.600758] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2172.605138] ceph_con_process_message+0x74/0x150 [libceph] [ 2172.610638] ? calc_signature+0xdb/0x100 [libceph] [ 2172.615441] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2172.620766] ? process_scheduled_works+0x22a/0x510 [ 2172.625564] ? process_scheduled_works+0x22a/0x510 [ 2172.630357] ceph_con_workfn+0x344/0x750 [libceph] [ 2172.635170] ? process_scheduled_works+0x22a/0x510 [ 2172.639960] process_scheduled_works+0x2a4/0x510 [ 2172.644587] worker_thread+0x188/0x340 [ 2172.648341] ? rescuer_thread+0x310/0x310 [ 2172.652352] kthread+0xf5/0x130 [ 2172.655501] ? kthread_complete_and_exit+0x20/0x20 [ 2172.660294] ret_from_fork+0x31/0x40 [ 2172.663872] ? kthread_complete_and_exit+0x20/0x20 [ 2172.668664] ret_from_fork_asm+0x11/0x20 [ 2172.672599] [ 2172.674790] irq event stamp: 10259327 [ 2172.678457] hardirqs last enabled at (10259337): [] console_unlock+0x106/0x120 [ 2172.687321] hardirqs last disabled at (10259346): [] console_unlock+0xeb/0x120 [ 2172.696101] softirqs last enabled at (10259284): [] __do_softirq+0x2be/0x371 [ 2172.704794] softirqs last disabled at (10259279): [] irq_exit_rcu+0xb0/0xd0 [ 2172.713314] ---[ end trace 0000000000000000 ]--- [ 2172.717937] ------------[ cut here ]------------ [ 2172.722552] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2172.734199] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2172.743155] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2172.798623] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2172.807575] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2172.815052] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2172.820538] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2172.839288] RSP: 0018:ffffc90001f7fac0 EFLAGS: 00010282 [ 2172.844511] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2172.851645] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2172.858775] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000021a8 [ 2172.865901] R10: 00000000000f59e0 R11: ffffc90001f7f950 R12: ffff8881c5350750 [ 2172.873025] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2172.880158] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2172.888243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2172.893982] CR2: 00007f9bd3099000 CR3: 0000000113532004 CR4: 00000000003706e0 [ 2172.901114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2172.908248] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2172.915383] Call Trace: [ 2172.917832] [ 2172.919929] ? __warn+0x85/0x180 [ 2172.923162] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.928044] ? report_bug+0xfc/0x1d0 [ 2172.931621] ? prb_read_valid+0x1b/0x20 [ 2172.935463] ? handle_bug+0x44/0x70 [ 2172.938954] ? exc_invalid_op+0x17/0x60 [ 2172.942792] ? asm_exc_invalid_op+0x1a/0x20 [ 2172.946984] ? __list_add_valid_or_report+0x7d/0xa0 [ 2172.951861] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2172.956748] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2172.961987] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2172.966793] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2172.971512] lookup_fast+0x6d/0xf0 [ 2172.974914] walk_component+0x26/0x160 [ 2172.978666] link_path_walk+0x24d/0x380 [ 2172.982506] ? path_lookupat.isra.40+0x22/0x120 [ 2172.987040] path_lookupat.isra.40+0x6b/0x120 [ 2172.991397] filename_lookup+0xd4/0x190 [ 2172.995240] ? lock_acquire+0xcb/0x2b0 [ 2172.998991] ? __might_fault+0x45/0x80 [ 2173.002743] ? find_held_lock+0x2d/0x90 [ 2173.006581] ? __might_fault+0x45/0x80 [ 2173.010337] vfs_statx+0x89/0x150 [ 2173.013657] vfs_fstatat+0x5a/0xc0 [ 2173.017060] __do_sys_newstat+0x30/0x70 [ 2173.020900] ? lock_release+0xc5/0x260 [ 2173.024654] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2173.029530] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2173.034672] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2173.039810] do_syscall_64+0x38/0x80 [ 2173.043390] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2173.048442] RIP: 0033:0x7f48f0eeca79 [ 2173.052022] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2173.070769] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2173.078335] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2173.085466] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2173.092598] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2173.099724] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2173.106855] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2173.113985] [ 2173.116172] irq event stamp: 12904867 [ 2173.119831] hardirqs last enabled at (12904877): [] console_unlock+0x106/0x120 [ 2173.128695] hardirqs last disabled at (12904886): [] console_unlock+0xeb/0x120 [ 2173.137478] softirqs last enabled at (12904900): [] __do_softirq+0x2be/0x371 [ 2173.146169] softirqs last disabled at (12904895): [] irq_exit_rcu+0xb0/0xd0 [ 2173.154688] ---[ end trace 0000000000000000 ]--- [ 2173.159299] ------------[ cut here ]------------ [ 2173.163918] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2173.175574] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2173.184528] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2173.239988] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2173.248957] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2173.256446] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2173.261933] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2173.280709] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2173.285946] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2173.293089] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2173.300230] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000021eb [ 2173.307381] R10: 00000000000f6b98 R11: ffffc90001ff7950 R12: ffff8881c5350750 [ 2173.314529] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2173.321654] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2173.329741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2173.335495] CR2: 00007fb38dad7c88 CR3: 0000000120cc8006 CR4: 00000000003706e0 [ 2173.342629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2173.349761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2173.356903] Call Trace: [ 2173.359364] [ 2173.361480] ? __warn+0x85/0x180 [ 2173.364713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2173.369609] ? report_bug+0xfc/0x1d0 [ 2173.373187] ? prb_read_valid+0x1b/0x20 [ 2173.377046] ? handle_bug+0x44/0x70 [ 2173.380554] ? exc_invalid_op+0x17/0x60 [ 2173.384411] ? asm_exc_invalid_op+0x1a/0x20 [ 2173.388602] ? __list_add_valid_or_report+0x7d/0xa0 [ 2173.393477] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2173.398366] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2173.403626] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2173.408430] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2173.413161] lookup_fast+0x6d/0xf0 [ 2173.416564] walk_component+0x26/0x160 [ 2173.420319] link_path_walk+0x24d/0x380 [ 2173.424195] ? path_lookupat.isra.40+0x22/0x120 [ 2173.428728] path_lookupat.isra.40+0x6b/0x120 [ 2173.433104] filename_lookup+0xd4/0x190 [ 2173.436979] ? lock_acquire+0xcb/0x2b0 [ 2173.440729] ? __might_fault+0x45/0x80 [ 2173.444499] ? find_held_lock+0x2d/0x90 [ 2173.448337] ? __might_fault+0x45/0x80 [ 2173.452119] vfs_statx+0x89/0x150 [ 2173.455438] vfs_fstatat+0x5a/0xc0 [ 2173.458844] __do_sys_newstat+0x30/0x70 [ 2173.462713] ? lock_release+0xc5/0x260 [ 2173.466479] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2173.471357] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2173.476514] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2173.481653] do_syscall_64+0x38/0x80 [ 2173.485233] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2173.490302] RIP: 0033:0x7f48f0eeca79 [ 2173.493889] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2173.512650] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2173.520212] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2173.527362] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2173.534504] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2173.541637] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2173.548767] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2173.555932] [ 2173.558120] irq event stamp: 12862545 [ 2173.561795] hardirqs last enabled at (12862555): [] console_unlock+0x106/0x120 [ 2173.570679] hardirqs last disabled at (12862564): [] console_unlock+0xeb/0x120 [ 2173.579459] softirqs last enabled at (12862578): [] __do_softirq+0x2be/0x371 [ 2173.588150] softirqs last disabled at (12862573): [] irq_exit_rcu+0xb0/0xd0 [ 2173.596686] ---[ end trace 0000000000000000 ]--- [ 2173.601300] ------------[ cut here ]------------ [ 2173.605939] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2173.617587] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2173.626280] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2173.681865] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2173.690994] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2173.698476] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2173.703985] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2173.709474] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2173.728228] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2173.733454] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2173.740587] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2173.747719] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000222e [ 2173.754851] R10: 00000000000f7ed0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2173.761984] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2173.769117] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2173.777202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2173.782950] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2173.790082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2173.797217] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2173.804352] Call Trace: [ 2173.806810] [ 2173.808917] ? __warn+0x85/0x180 [ 2173.812150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2173.817039] ? report_bug+0xfc/0x1d0 [ 2173.820626] ? prb_read_valid+0x1b/0x20 [ 2173.824468] ? handle_bug+0x44/0x70 [ 2173.827965] ? exc_invalid_op+0x17/0x60 [ 2173.831805] ? asm_exc_invalid_op+0x1a/0x20 [ 2173.836002] ? __list_add_valid_or_report+0x7d/0xa0 [ 2173.840906] ? __list_add_valid_or_report+0x7d/0xa0 [ 2173.845783] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2173.850723] ceph_add_cap+0x20b/0x620 [ceph] [ 2173.855044] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2173.859801] ? lock_acquire+0xcb/0x2b0 [ 2173.863590] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2173.868051] ? __mutex_lock+0x97/0xce0 [ 2173.871801] ? __mutex_lock+0xaec/0xce0 [ 2173.875646] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2173.880202] ? lock_downgrade+0xfd/0x250 [ 2173.884138] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2173.888555] ceph_con_process_message+0x74/0x150 [libceph] [ 2173.894059] ? calc_signature+0xdb/0x100 [libceph] [ 2173.898900] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2173.904257] ? process_scheduled_works+0x22a/0x510 [ 2173.909059] ? process_scheduled_works+0x22a/0x510 [ 2173.913854] ceph_con_workfn+0x344/0x750 [libceph] [ 2173.918703] ? process_scheduled_works+0x22a/0x510 [ 2173.923518] process_scheduled_works+0x2a4/0x510 [ 2173.928146] worker_thread+0x188/0x340 [ 2173.931925] ? rescuer_thread+0x310/0x310 [ 2173.935938] kthread+0xf5/0x130 [ 2173.939086] ? kthread_complete_and_exit+0x20/0x20 [ 2173.943886] ret_from_fork+0x31/0x40 [ 2173.947464] ? kthread_complete_and_exit+0x20/0x20 [ 2173.952256] ret_from_fork_asm+0x11/0x20 [ 2173.956197] [ 2173.958427] irq event stamp: 10262271 [ 2173.962092] hardirqs last enabled at (10262281): [] console_unlock+0x106/0x120 [ 2173.970959] hardirqs last disabled at (10262290): [] console_unlock+0xeb/0x120 [ 2173.979738] softirqs last enabled at (10262218): [] __do_softirq+0x2be/0x371 [ 2173.988450] softirqs last disabled at (10262213): [] irq_exit_rcu+0xb0/0xd0 [ 2173.996968] ---[ end trace 0000000000000000 ]--- [ 2174.001606] ------------[ cut here ]------------ [ 2174.006225] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2174.017885] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2174.026841] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2174.082318] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2174.091287] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2174.098774] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2174.104278] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2174.123043] RSP: 0018:ffffc90001fdfb38 EFLAGS: 00010282 [ 2174.128286] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2174.135435] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2174.142566] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000226d [ 2174.149708] R10: 00000000000f92d0 R11: ffffc90001fdf9c8 R12: ffff8881c5350750 [ 2174.156850] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2174.163999] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2174.172096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2174.177859] CR2: 00007f9bd3099000 CR3: 0000000179402005 CR4: 00000000003706e0 [ 2174.185000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2174.192142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2174.199282] Call Trace: [ 2174.201747] [ 2174.203869] ? __warn+0x85/0x180 [ 2174.207101] ? __list_add_valid_or_report+0x7d/0xa0 [ 2174.211981] ? report_bug+0xfc/0x1d0 [ 2174.215577] ? prb_read_valid+0x1b/0x20 [ 2174.219434] ? handle_bug+0x44/0x70 [ 2174.222926] ? exc_invalid_op+0x17/0x60 [ 2174.226773] ? asm_exc_invalid_op+0x1a/0x20 [ 2174.230990] ? __list_add_valid_or_report+0x7d/0xa0 [ 2174.235867] ? __list_add_valid_or_report+0x7d/0xa0 [ 2174.240762] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2174.245670] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2174.250405] lookup_fast+0x6d/0xf0 [ 2174.253807] walk_component+0x26/0x160 [ 2174.257560] link_path_walk+0x24d/0x380 [ 2174.261399] ? path_lookupat.isra.40+0x22/0x120 [ 2174.265950] path_lookupat.isra.40+0x6b/0x120 [ 2174.270310] filename_lookup+0xd4/0x190 [ 2174.274169] ? lock_acquire+0xcb/0x2b0 [ 2174.277934] ? __might_fault+0x45/0x80 [ 2174.281717] ? find_held_lock+0x2d/0x90 [ 2174.285571] ? __might_fault+0x45/0x80 [ 2174.289326] vfs_statx+0x89/0x150 [ 2174.292661] vfs_fstatat+0x5a/0xc0 [ 2174.296068] __do_sys_newstat+0x30/0x70 [ 2174.299922] ? lock_release+0xc5/0x260 [ 2174.303686] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2174.308563] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2174.313716] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2174.318868] do_syscall_64+0x38/0x80 [ 2174.322465] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2174.327519] RIP: 0033:0x7f48f0eeca79 [ 2174.331099] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2174.349860] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2174.357443] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2174.364568] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2174.371708] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2174.378850] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2174.385994] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2174.393164] [ 2174.395352] irq event stamp: 12806929 [ 2174.399026] hardirqs last enabled at (12806939): [] console_unlock+0x106/0x120 [ 2174.407902] hardirqs last disabled at (12806948): [] console_unlock+0xeb/0x120 [ 2174.416700] softirqs last enabled at (12806962): [] __do_softirq+0x2be/0x371 [ 2174.425393] softirqs last disabled at (12806957): [] irq_exit_rcu+0xb0/0xd0 [ 2174.433929] ---[ end trace 0000000000000000 ]--- [ 2174.438562] ------------[ cut here ]------------ [ 2174.443178] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2174.454856] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2174.463803] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2174.519282] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2174.528250] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2174.535719] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2174.541215] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2174.559979] RSP: 0018:ffffc90001fdfac0 EFLAGS: 00010282 [ 2174.565215] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2174.572354] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2174.579496] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000022af [ 2174.586621] R10: 00000000000fa618 R11: ffffc90001fdf950 R12: ffff8881c5350750 [ 2174.593753] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2174.600894] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2174.608989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2174.614745] CR2: 00007f9bd3099000 CR3: 0000000179402005 CR4: 00000000003706e0 [ 2174.621885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2174.629026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2174.636169] Call Trace: [ 2174.638631] [ 2174.640728] ? __warn+0x85/0x180 [ 2174.643978] ? __list_add_valid_or_report+0x7d/0xa0 [ 2174.648866] ? report_bug+0xfc/0x1d0 [ 2174.652463] ? prb_read_valid+0x1b/0x20 [ 2174.656304] ? handle_bug+0x44/0x70 [ 2174.659813] ? exc_invalid_op+0x17/0x60 [ 2174.663661] ? asm_exc_invalid_op+0x1a/0x20 [ 2174.667868] ? __list_add_valid_or_report+0x7d/0xa0 [ 2174.672744] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2174.677652] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2174.682884] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2174.687719] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2174.692454] lookup_fast+0x6d/0xf0 [ 2174.695857] walk_component+0x26/0x160 [ 2174.699620] link_path_walk+0x24d/0x380 [ 2174.703459] ? path_lookupat.isra.40+0x22/0x120 [ 2174.707992] path_lookupat.isra.40+0x6b/0x120 [ 2174.712369] filename_lookup+0xd4/0x190 [ 2174.716246] ? lock_acquire+0xcb/0x2b0 [ 2174.719994] ? __might_fault+0x45/0x80 [ 2174.723757] ? find_held_lock+0x2d/0x90 [ 2174.727614] ? __might_fault+0x45/0x80 [ 2174.731368] vfs_statx+0x89/0x150 [ 2174.734705] vfs_fstatat+0x5a/0xc0 [ 2174.738108] __do_sys_newstat+0x30/0x70 [ 2174.741956] ? lock_release+0xc5/0x260 [ 2174.745724] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2174.750616] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2174.755754] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2174.760911] do_syscall_64+0x38/0x80 [ 2174.764507] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2174.769559] RIP: 0033:0x7f48f0eeca79 [ 2174.773138] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2174.791896] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2174.799479] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2174.806610] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2174.813735] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2174.820877] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2174.828026] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2174.835191] [ 2174.837379] irq event stamp: 12808055 [ 2174.841044] hardirqs last enabled at (12808065): [] console_unlock+0x106/0x120 [ 2174.849928] hardirqs last disabled at (12808074): [] console_unlock+0xeb/0x120 [ 2174.858742] softirqs last enabled at (12808088): [] __do_softirq+0x2be/0x371 [ 2174.867452] softirqs last disabled at (12808083): [] irq_exit_rcu+0xb0/0xd0 [ 2174.875971] ---[ end trace 0000000000000000 ]--- [ 2174.880600] ------------[ cut here ]------------ [ 2174.885220] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2174.896866] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2174.905563] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2174.961045] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2174.970172] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2174.977653] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2174.983156] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2174.988659] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2175.007405] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2175.012633] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2175.019765] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2175.026898] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000022f2 [ 2175.034030] R10: 00000000000fb848 R11: ffffc9000030b828 R12: ffff88819f028750 [ 2175.041164] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2175.048297] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2175.056380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2175.062129] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2175.069261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2175.076393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2175.083528] Call Trace: [ 2175.085980] [ 2175.088086] ? __warn+0x85/0x180 [ 2175.091318] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.096200] ? report_bug+0xfc/0x1d0 [ 2175.099778] ? prb_read_valid+0x1b/0x20 [ 2175.103621] ? handle_bug+0x44/0x70 [ 2175.107110] ? exc_invalid_op+0x17/0x60 [ 2175.110948] ? asm_exc_invalid_op+0x1a/0x20 [ 2175.115140] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.120024] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.124902] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2175.129795] ceph_add_cap+0x20b/0x620 [ceph] [ 2175.134087] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2175.138836] ceph_fill_trace+0x143/0xf30 [ceph] [ 2175.143380] ? __mutex_lock+0x97/0xce0 [ 2175.147149] ? __mutex_lock+0xaec/0xce0 [ 2175.151007] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2175.155567] ? lock_downgrade+0xfd/0x250 [ 2175.159499] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2175.163894] ceph_con_process_message+0x74/0x150 [libceph] [ 2175.169390] ? calc_signature+0xdb/0x100 [libceph] [ 2175.174220] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2175.179556] ? process_scheduled_works+0x22a/0x510 [ 2175.184370] ? process_scheduled_works+0x22a/0x510 [ 2175.189156] ceph_con_workfn+0x344/0x750 [libceph] [ 2175.193998] ? process_scheduled_works+0x22a/0x510 [ 2175.198795] process_scheduled_works+0x2a4/0x510 [ 2175.203456] worker_thread+0x188/0x340 [ 2175.207209] ? rescuer_thread+0x310/0x310 [ 2175.211241] kthread+0xf5/0x130 [ 2175.214405] ? kthread_complete_and_exit+0x20/0x20 [ 2175.219224] ret_from_fork+0x31/0x40 [ 2175.222800] ? kthread_complete_and_exit+0x20/0x20 [ 2175.227616] ret_from_fork_asm+0x11/0x20 [ 2175.231546] [ 2175.233741] irq event stamp: 10265143 [ 2175.237443] hardirqs last enabled at (10265153): [] console_unlock+0x106/0x120 [ 2175.246307] hardirqs last disabled at (10265162): [] console_unlock+0xeb/0x120 [ 2175.255102] softirqs last enabled at (10265100): [] __do_softirq+0x2be/0x371 [ 2175.263812] softirqs last disabled at (10265095): [] irq_exit_rcu+0xb0/0xd0 [ 2175.272333] ---[ end trace 0000000000000000 ]--- [ 2175.277016] ------------[ cut here ]------------ [ 2175.281638] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2175.293297] WARNING: CPU: 3 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2175.302251] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2175.357745] CPU: 3 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2175.366732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2175.374228] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2175.379731] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2175.398495] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2175.403732] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2175.410880] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2175.418021] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002330 [ 2175.425162] R10: 00000000000fc940 R11: ffffc90001fef9c8 R12: ffff88819f028750 [ 2175.432304] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2175.439454] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2175.447540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2175.453286] CR2: 00007fb38dd7a000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2175.460428] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2175.467569] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2175.474707] Call Trace: [ 2175.477157] [ 2175.479278] ? __warn+0x85/0x180 [ 2175.482528] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.487408] ? report_bug+0xfc/0x1d0 [ 2175.491002] ? prb_read_valid+0x1b/0x20 [ 2175.494838] ? handle_bug+0x44/0x70 [ 2175.498345] ? exc_invalid_op+0x17/0x60 [ 2175.502201] ? asm_exc_invalid_op+0x1a/0x20 [ 2175.506417] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.511294] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.516191] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2175.521098] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2175.525849] lookup_fast+0x6d/0xf0 [ 2175.529253] walk_component+0x26/0x160 [ 2175.533023] link_path_walk+0x24d/0x380 [ 2175.536879] ? path_lookupat.isra.40+0x22/0x120 [ 2175.541449] path_lookupat.isra.40+0x6b/0x120 [ 2175.545807] filename_lookup+0xd4/0x190 [ 2175.549665] ? lock_acquire+0xcb/0x2b0 [ 2175.553414] ? __might_fault+0x45/0x80 [ 2175.557185] ? find_held_lock+0x2d/0x90 [ 2175.561023] ? __might_fault+0x45/0x80 [ 2175.564787] vfs_statx+0x89/0x150 [ 2175.568142] vfs_fstatat+0x5a/0xc0 [ 2175.571547] __do_sys_newstat+0x30/0x70 [ 2175.575385] ? lock_release+0xc5/0x260 [ 2175.579148] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2175.584041] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2175.589191] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2175.594349] do_syscall_64+0x38/0x80 [ 2175.597943] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2175.603014] RIP: 0033:0x7f48f0eeca79 [ 2175.606604] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2175.625349] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2175.632933] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2175.640074] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2175.647214] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2175.654357] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2175.661499] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2175.668638] [ 2175.670822] irq event stamp: 12797613 [ 2175.674498] hardirqs last enabled at (12797623): [] console_unlock+0x106/0x120 [ 2175.683364] hardirqs last disabled at (12797632): [] console_unlock+0xeb/0x120 [ 2175.692153] softirqs last enabled at (12797646): [] __do_softirq+0x2be/0x371 [ 2175.700863] softirqs last disabled at (12797641): [] irq_exit_rcu+0xb0/0xd0 [ 2175.709399] ---[ end trace 0000000000000000 ]--- [ 2175.714027] ------------[ cut here ]------------ [ 2175.718668] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2175.730332] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2175.739027] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2175.794531] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2175.803654] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2175.811134] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2175.816652] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2175.822140] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2175.840905] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2175.846144] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2175.853294] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2175.860463] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002372 [ 2175.867597] R10: 00000000000fdb38 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2175.874741] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2175.881888] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2175.889990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2175.895753] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2175.902905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2175.910054] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2175.917204] Call Trace: [ 2175.919668] [ 2175.921775] ? __warn+0x85/0x180 [ 2175.925023] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.929921] ? report_bug+0xfc/0x1d0 [ 2175.933517] ? prb_read_valid+0x1b/0x20 [ 2175.937376] ? handle_bug+0x44/0x70 [ 2175.940866] ? exc_invalid_op+0x17/0x60 [ 2175.944740] ? asm_exc_invalid_op+0x1a/0x20 [ 2175.948969] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.953868] ? __list_add_valid_or_report+0x7d/0xa0 [ 2175.958758] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2175.963675] ceph_add_cap+0x20b/0x620 [ceph] [ 2175.967982] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2175.972710] ? lock_acquire+0xcb/0x2b0 [ 2175.976474] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2175.980927] ? __mutex_lock+0x97/0xce0 [ 2175.984694] ? __mutex_lock+0xaec/0xce0 [ 2175.988536] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2175.993084] ? lock_downgrade+0xfd/0x250 [ 2175.997052] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2176.001460] ceph_con_process_message+0x74/0x150 [libceph] [ 2176.006959] ? calc_signature+0xdb/0x100 [libceph] [ 2176.011792] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2176.017142] ? process_scheduled_works+0x22a/0x510 [ 2176.021959] ? process_scheduled_works+0x22a/0x510 [ 2176.026757] ceph_con_workfn+0x344/0x750 [libceph] [ 2176.031586] ? process_scheduled_works+0x22a/0x510 [ 2176.036383] process_scheduled_works+0x2a4/0x510 [ 2176.041048] worker_thread+0x188/0x340 [ 2176.044818] ? rescuer_thread+0x310/0x310 [ 2176.048828] kthread+0xf5/0x130 [ 2176.052013] ? kthread_complete_and_exit+0x20/0x20 [ 2176.056820] ret_from_fork+0x31/0x40 [ 2176.060398] ? kthread_complete_and_exit+0x20/0x20 [ 2176.065211] ret_from_fork_asm+0x11/0x20 [ 2176.069182] [ 2176.071371] irq event stamp: 10268389 [ 2176.075052] hardirqs last enabled at (10268399): [] console_unlock+0x106/0x120 [ 2176.083937] hardirqs last disabled at (10268408): [] console_unlock+0xeb/0x120 [ 2176.092744] softirqs last enabled at (10268336): [] __do_softirq+0x2be/0x371 [ 2176.101464] softirqs last disabled at (10268331): [] irq_exit_rcu+0xb0/0xd0 [ 2176.109980] ---[ end trace 0000000000000000 ]--- [ 2176.114620] ------------[ cut here ]------------ [ 2176.119240] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2176.130900] WARNING: CPU: 7 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2176.139856] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2176.195337] CPU: 7 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2176.204284] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2176.211773] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2176.217278] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2176.236041] RSP: 0018:ffffc90001fefac0 EFLAGS: 00010282 [ 2176.241281] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2176.248423] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2176.255567] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000023b1 [ 2176.262701] R10: 00000000000fec80 R11: ffffc90001fef950 R12: ffff88819f028750 [ 2176.269830] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2176.276971] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2176.285068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.290823] CR2: 00007fb38dad7c88 CR3: 0000000108c72001 CR4: 00000000003706e0 [ 2176.297963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2176.305105] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2176.312246] Call Trace: [ 2176.314708] [ 2176.316817] ? __warn+0x85/0x180 [ 2176.320065] ? __list_add_valid_or_report+0x7d/0xa0 [ 2176.324961] ? report_bug+0xfc/0x1d0 [ 2176.328560] ? prb_read_valid+0x1b/0x20 [ 2176.332399] ? handle_bug+0x44/0x70 [ 2176.335907] ? exc_invalid_op+0x17/0x60 [ 2176.339760] ? asm_exc_invalid_op+0x1a/0x20 [ 2176.343979] ? __list_add_valid_or_report+0x7d/0xa0 [ 2176.348859] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2176.353772] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2176.359023] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2176.363843] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2176.368575] lookup_fast+0x6d/0xf0 [ 2176.371979] walk_component+0x26/0x160 [ 2176.375762] link_path_walk+0x24d/0x380 [ 2176.379615] ? path_lookupat.isra.40+0x22/0x120 [ 2176.384148] path_lookupat.isra.40+0x6b/0x120 [ 2176.388526] filename_lookup+0xd4/0x190 [ 2176.392383] ? lock_acquire+0xcb/0x2b0 [ 2176.396133] ? __might_fault+0x45/0x80 [ 2176.399903] ? find_held_lock+0x2d/0x90 [ 2176.403762] ? __might_fault+0x45/0x80 [ 2176.407532] vfs_statx+0x89/0x150 [ 2176.410871] vfs_fstatat+0x5a/0xc0 [ 2176.414275] __do_sys_newstat+0x30/0x70 [ 2176.418129] ? lock_release+0xc5/0x260 [ 2176.421893] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2176.426788] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2176.431944] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2176.437102] do_syscall_64+0x38/0x80 [ 2176.440700] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2176.445760] RIP: 0033:0x7f48f0eeca79 [ 2176.449356] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2176.468121] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2176.475705] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2176.482836] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2176.489986] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2176.497128] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2176.504278] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2176.511453] [ 2176.513637] irq event stamp: 12799645 [ 2176.517294] hardirqs last enabled at (12799655): [] console_unlock+0x106/0x120 [ 2176.526171] hardirqs last disabled at (12799664): [] console_unlock+0xeb/0x120 [ 2176.534951] softirqs last enabled at (12799678): [] __do_softirq+0x2be/0x371 [ 2176.543641] softirqs last disabled at (12799673): [] irq_exit_rcu+0xb0/0xd0 [ 2176.552161] ---[ end trace 0000000000000000 ]--- [ 2176.556774] ------------[ cut here ]------------ [ 2176.561396] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2176.573066] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2176.581762] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2176.637247] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2176.646372] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2176.653853] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2176.659363] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2176.664850] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2176.683597] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2176.688821] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2176.695954] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2176.703087] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000023f4 [ 2176.710219] R10: 00000000000ffe50 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2176.717354] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2176.724486] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2176.732572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.738317] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2176.745470] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2176.752602] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2176.759735] Call Trace: [ 2176.762205] [ 2176.764311] ? __warn+0x85/0x180 [ 2176.767544] ? __list_add_valid_or_report+0x7d/0xa0 [ 2176.772424] ? report_bug+0xfc/0x1d0 [ 2176.776002] ? prb_read_valid+0x1b/0x20 [ 2176.779844] ? handle_bug+0x44/0x70 [ 2176.783333] ? exc_invalid_op+0x17/0x60 [ 2176.787173] ? asm_exc_invalid_op+0x1a/0x20 [ 2176.791366] ? __list_add_valid_or_report+0x7d/0xa0 [ 2176.796251] ? __list_add_valid_or_report+0x7d/0xa0 [ 2176.801128] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2176.806020] ceph_add_cap+0x20b/0x620 [ceph] [ 2176.810313] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2176.815042] ceph_fill_trace+0x143/0xf30 [ceph] [ 2176.819581] ? __mutex_lock+0x97/0xce0 [ 2176.823332] ? __mutex_lock+0xaec/0xce0 [ 2176.827172] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2176.831715] ? lock_downgrade+0xfd/0x250 [ 2176.835645] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2176.840026] ceph_con_process_message+0x74/0x150 [libceph] [ 2176.845519] ? calc_signature+0xdb/0x100 [libceph] [ 2176.850335] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2176.855665] ? process_scheduled_works+0x22a/0x510 [ 2176.860483] ? process_scheduled_works+0x22a/0x510 [ 2176.865278] ceph_con_workfn+0x344/0x750 [libceph] [ 2176.870085] ? process_scheduled_works+0x22a/0x510 [ 2176.874874] process_scheduled_works+0x2a4/0x510 [ 2176.879500] worker_thread+0x188/0x340 [ 2176.883258] ? rescuer_thread+0x310/0x310 [ 2176.887266] kthread+0xf5/0x130 [ 2176.890414] ? kthread_complete_and_exit+0x20/0x20 [ 2176.895206] ret_from_fork+0x31/0x40 [ 2176.898785] ? kthread_complete_and_exit+0x20/0x20 [ 2176.903579] ret_from_fork_asm+0x11/0x20 [ 2176.907513] [ 2176.909704] irq event stamp: 10270323 [ 2176.913370] hardirqs last enabled at (10270333): [] console_unlock+0x106/0x120 [ 2176.922237] hardirqs last disabled at (10270342): [] console_unlock+0xeb/0x120 [ 2176.931023] softirqs last enabled at (10270356): [] __do_softirq+0x2be/0x371 [ 2176.939718] softirqs last disabled at (10270351): [] irq_exit_rcu+0xb0/0xd0 [ 2176.948237] ---[ end trace 0000000000000000 ]--- [ 2176.952916] ------------[ cut here ]------------ [ 2176.957540] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2176.969201] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2176.977896] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2177.033388] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2177.042531] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2177.050010] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2177.055525] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2177.061011] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2177.079763] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2177.085009] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2177.092162] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2177.099309] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002432 [ 2177.106476] R10: 0000000000100ff0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2177.113609] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2177.120757] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2177.128863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2177.134624] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2177.141774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2177.148924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2177.156076] Call Trace: [ 2177.158546] [ 2177.160653] ? __warn+0x85/0x180 [ 2177.163886] ? __list_add_valid_or_report+0x7d/0xa0 [ 2177.168784] ? report_bug+0xfc/0x1d0 [ 2177.172379] ? prb_read_valid+0x1b/0x20 [ 2177.176255] ? handle_bug+0x44/0x70 [ 2177.179761] ? exc_invalid_op+0x17/0x60 [ 2177.183624] ? asm_exc_invalid_op+0x1a/0x20 [ 2177.187846] ? __list_add_valid_or_report+0x7d/0xa0 [ 2177.192736] ? __list_add_valid_or_report+0x7d/0xa0 [ 2177.197620] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2177.202524] ceph_add_cap+0x20b/0x620 [ceph] [ 2177.206846] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2177.211586] ? lock_acquire+0xcb/0x2b0 [ 2177.215365] ? lock_is_held_type+0xaa/0x120 [ 2177.219561] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2177.224048] ? __mutex_lock+0x97/0xce0 [ 2177.227806] ? __mutex_lock+0xaec/0xce0 [ 2177.231666] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2177.236223] ? lock_downgrade+0xfd/0x250 [ 2177.240193] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2177.244595] ceph_con_process_message+0x74/0x150 [libceph] [ 2177.250107] ? calc_signature+0xdb/0x100 [libceph] [ 2177.254964] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2177.260322] ? process_scheduled_works+0x22a/0x510 [ 2177.265140] ? process_scheduled_works+0x22a/0x510 [ 2177.269934] ceph_con_workfn+0x344/0x750 [libceph] [ 2177.274782] ? process_scheduled_works+0x22a/0x510 [ 2177.279599] process_scheduled_works+0x2a4/0x510 [ 2177.284242] worker_thread+0x188/0x340 [ 2177.288014] ? rescuer_thread+0x310/0x310 [ 2177.292024] kthread+0xf5/0x130 [ 2177.295171] ? kthread_complete_and_exit+0x20/0x20 [ 2177.300000] ret_from_fork+0x31/0x40 [ 2177.303578] ? kthread_complete_and_exit+0x20/0x20 [ 2177.308370] ret_from_fork_asm+0x11/0x20 [ 2177.312341] [ 2177.314532] irq event stamp: 10271433 [ 2177.318197] hardirqs last enabled at (10271443): [] console_unlock+0x106/0x120 [ 2177.327081] hardirqs last disabled at (10271452): [] console_unlock+0xeb/0x120 [ 2177.335879] softirqs last enabled at (10270960): [] __do_softirq+0x2be/0x371 [ 2177.344589] softirqs last disabled at (10270955): [] irq_exit_rcu+0xb0/0xd0 [ 2177.353126] ---[ end trace 0000000000000000 ]--- [ 2177.357747] ------------[ cut here ]------------ [ 2177.362362] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2177.374020] WARNING: CPU: 7 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2177.382973] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2177.438478] CPU: 7 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2177.447426] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2177.454899] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2177.460403] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2177.479166] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2177.484407] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2177.491532] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2177.498658] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002472 [ 2177.505792] R10: 0000000000102240 R11: ffffc90001fd79c8 R12: ffff8881d10b72e8 [ 2177.512940] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2177.520103] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2177.528185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2177.533940] CR2: 00007fb38dad7c88 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2177.541082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2177.548222] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2177.555365] Call Trace: [ 2177.557826] [ 2177.559942] ? __warn+0x85/0x180 [ 2177.563191] ? __list_add_valid_or_report+0x7d/0xa0 [ 2177.568069] ? report_bug+0xfc/0x1d0 [ 2177.571667] ? prb_read_valid+0x1b/0x20 [ 2177.575509] ? handle_bug+0x44/0x70 [ 2177.578999] ? exc_invalid_op+0x17/0x60 [ 2177.582847] ? asm_exc_invalid_op+0x1a/0x20 [ 2177.587072] ? __list_add_valid_or_report+0x7d/0xa0 [ 2177.591947] ? __list_add_valid_or_report+0x7d/0xa0 [ 2177.596836] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2177.601742] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2177.606478] lookup_fast+0x6d/0xf0 [ 2177.609877] walk_component+0x26/0x160 [ 2177.613623] link_path_walk+0x24d/0x380 [ 2177.617465] ? path_lookupat.isra.40+0x22/0x120 [ 2177.622014] path_lookupat.isra.40+0x6b/0x120 [ 2177.626390] filename_lookup+0xd4/0x190 [ 2177.630265] ? lock_acquire+0xcb/0x2b0 [ 2177.634015] ? __might_fault+0x45/0x80 [ 2177.637791] ? find_held_lock+0x2d/0x90 [ 2177.641643] ? __might_fault+0x45/0x80 [ 2177.645399] vfs_statx+0x89/0x150 [ 2177.648735] vfs_fstatat+0x5a/0xc0 [ 2177.652140] __do_sys_newstat+0x30/0x70 [ 2177.655994] ? lock_release+0xc5/0x260 [ 2177.659774] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2177.664669] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2177.669809] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2177.674969] do_syscall_64+0x38/0x80 [ 2177.678564] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2177.683617] RIP: 0033:0x7f48f0eeca79 [ 2177.687195] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2177.705960] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2177.713545] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2177.720676] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2177.727808] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2177.734957] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2177.742099] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2177.749265] [ 2177.751450] irq event stamp: 12932149 [ 2177.755126] hardirqs last enabled at (12932159): [] console_unlock+0x106/0x120 [ 2177.764009] hardirqs last disabled at (12932168): [] console_unlock+0xeb/0x120 [ 2177.772806] softirqs last enabled at (12932182): [] __do_softirq+0x2be/0x371 [ 2177.781516] softirqs last disabled at (12932177): [] irq_exit_rcu+0xb0/0xd0 [ 2177.790035] ---[ end trace 0000000000000000 ]--- [ 2177.794665] ------------[ cut here ]------------ [ 2177.799285] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2177.810948] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2177.819644] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2177.875138] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2177.884263] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2177.891766] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2177.897284] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2177.902766] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2177.921532] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2177.926757] RAX: 0000000000000000 RBX: ffff8881d11409c8 RCX: 0000000000000000 [ 2177.933907] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2177.941056] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000024b4 [ 2177.948205] R10: 00000000001037a0 R11: ffffc9000030b828 R12: ffff8881d1141118 [ 2177.955379] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2177.962506] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2177.970593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2177.976337] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2177.983497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2177.990631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2177.997762] Call Trace: [ 2178.000218] [ 2178.002341] ? __warn+0x85/0x180 [ 2178.005591] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.010493] ? report_bug+0xfc/0x1d0 [ 2178.014077] ? prb_read_valid+0x1b/0x20 [ 2178.017953] ? handle_bug+0x44/0x70 [ 2178.021442] ? exc_invalid_op+0x17/0x60 [ 2178.025301] ? asm_exc_invalid_op+0x1a/0x20 [ 2178.029513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.034408] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.039286] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2178.044214] ceph_add_cap+0x20b/0x620 [ceph] [ 2178.048511] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2178.053255] ceph_fill_trace+0x143/0xf30 [ceph] [ 2178.057802] ? __mutex_lock+0x97/0xce0 [ 2178.061569] ? __mutex_lock+0xaec/0xce0 [ 2178.065427] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2178.069976] ? lock_downgrade+0xfd/0x250 [ 2178.073945] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2178.078342] ceph_con_process_message+0x74/0x150 [libceph] [ 2178.083844] ? calc_signature+0xdb/0x100 [libceph] [ 2178.088677] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2178.094025] ? process_scheduled_works+0x22a/0x510 [ 2178.098842] ? process_scheduled_works+0x22a/0x510 [ 2178.103639] ceph_con_workfn+0x344/0x750 [libceph] [ 2178.108474] ? process_scheduled_works+0x22a/0x510 [ 2178.113265] process_scheduled_works+0x2a4/0x510 [ 2178.117912] worker_thread+0x188/0x340 [ 2178.121669] ? rescuer_thread+0x310/0x310 [ 2178.125686] kthread+0xf5/0x130 [ 2178.128833] ? kthread_complete_and_exit+0x20/0x20 [ 2178.133648] ret_from_fork+0x31/0x40 [ 2178.137230] ? kthread_complete_and_exit+0x20/0x20 [ 2178.142023] ret_from_fork_asm+0x11/0x20 [ 2178.145992] [ 2178.148185] irq event stamp: 10273397 [ 2178.151868] hardirqs last enabled at (10273407): [] console_unlock+0x106/0x120 [ 2178.160754] hardirqs last disabled at (10273416): [] console_unlock+0xeb/0x120 [ 2178.169533] softirqs last enabled at (10273354): [] __do_softirq+0x2be/0x371 [ 2178.178224] softirqs last disabled at (10273349): [] irq_exit_rcu+0xb0/0xd0 [ 2178.186761] ---[ end trace 0000000000000000 ]--- [ 2178.191382] ------------[ cut here ]------------ [ 2178.196025] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2178.207676] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2178.216628] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2178.272095] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2178.281045] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2178.288518] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2178.294003] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2178.312751] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2178.317975] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2178.325100] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2178.332232] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000024f2 [ 2178.339357] R10: 0000000000104930 R11: ffffc90001fd7950 R12: ffff8881d10b72e8 [ 2178.346490] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2178.353622] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2178.361708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2178.367446] CR2: 00007fb38dd7a000 CR3: 0000000120c1e006 CR4: 00000000003706e0 [ 2178.374579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2178.381712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2178.388845] Call Trace: [ 2178.391298] [ 2178.393403] ? __warn+0x85/0x180 [ 2178.396635] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.401517] ? report_bug+0xfc/0x1d0 [ 2178.405095] ? prb_read_valid+0x1b/0x20 [ 2178.408935] ? handle_bug+0x44/0x70 [ 2178.412428] ? exc_invalid_op+0x17/0x60 [ 2178.416266] ? asm_exc_invalid_op+0x1a/0x20 [ 2178.420457] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.425333] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2178.430221] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2178.435457] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2178.440257] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2178.444974] lookup_fast+0x6d/0xf0 [ 2178.448378] walk_component+0x26/0x160 [ 2178.452131] link_path_walk+0x24d/0x380 [ 2178.455968] ? path_lookupat.isra.40+0x22/0x120 [ 2178.460505] path_lookupat.isra.40+0x6b/0x120 [ 2178.464864] filename_lookup+0xd4/0x190 [ 2178.468703] ? lock_acquire+0xcb/0x2b0 [ 2178.472452] ? __might_fault+0x45/0x80 [ 2178.476208] ? find_held_lock+0x2d/0x90 [ 2178.480045] ? __might_fault+0x45/0x80 [ 2178.483801] vfs_statx+0x89/0x150 [ 2178.487119] vfs_fstatat+0x5a/0xc0 [ 2178.490524] __do_sys_newstat+0x30/0x70 [ 2178.494363] ? lock_release+0xc5/0x260 [ 2178.498118] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2178.502995] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2178.508134] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2178.513274] do_syscall_64+0x38/0x80 [ 2178.516855] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2178.521906] RIP: 0033:0x7f48f0eeca79 [ 2178.525486] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2178.544232] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2178.551798] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2178.558931] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2178.566063] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2178.573189] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2178.580319] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2178.587450] [ 2178.589637] irq event stamp: 12934165 [ 2178.593294] hardirqs last enabled at (12934175): [] console_unlock+0x106/0x120 [ 2178.602162] hardirqs last disabled at (12934184): [] console_unlock+0xeb/0x120 [ 2178.610941] softirqs last enabled at (12934198): [] __do_softirq+0x2be/0x371 [ 2178.619634] softirqs last disabled at (12934193): [] irq_exit_rcu+0xb0/0xd0 [ 2178.628152] ---[ end trace 0000000000000000 ]--- [ 2178.632765] ------------[ cut here ]------------ [ 2178.637453] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2178.649125] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2178.658105] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2178.713574] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2178.722533] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2178.730005] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2178.735512] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2178.754256] RSP: 0018:ffffc90001fffb38 EFLAGS: 00010282 [ 2178.759510] RAX: 0000000000000000 RBX: ffff8881d11409c8 RCX: 0000000000000000 [ 2178.766639] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2178.773765] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002535 [ 2178.780908] R10: 0000000000105b90 R11: ffffc90001fff9c8 R12: ffff8881d1141118 [ 2178.788054] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2178.795196] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2178.803300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2178.809061] CR2: 00007f9bd31c8000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2178.816209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2178.823358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2178.830512] Call Trace: [ 2178.832960] [ 2178.835065] ? __warn+0x85/0x180 [ 2178.838299] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.843194] ? report_bug+0xfc/0x1d0 [ 2178.846794] ? prb_read_valid+0x1b/0x20 [ 2178.850650] ? handle_bug+0x44/0x70 [ 2178.854139] ? exc_invalid_op+0x17/0x60 [ 2178.857989] ? asm_exc_invalid_op+0x1a/0x20 [ 2178.862214] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.867089] ? __list_add_valid_or_report+0x7d/0xa0 [ 2178.871985] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2178.876892] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2178.881627] lookup_fast+0x6d/0xf0 [ 2178.885030] walk_component+0x26/0x160 [ 2178.888802] link_path_walk+0x24d/0x380 [ 2178.892655] ? path_lookupat.isra.40+0x22/0x120 [ 2178.897192] path_lookupat.isra.40+0x6b/0x120 [ 2178.901567] filename_lookup+0xd4/0x190 [ 2178.905432] ? lock_acquire+0xcb/0x2b0 [ 2178.909184] ? __might_fault+0x45/0x80 [ 2178.912977] ? find_held_lock+0x2d/0x90 [ 2178.916821] ? __might_fault+0x45/0x80 [ 2178.920595] vfs_statx+0x89/0x150 [ 2178.923928] vfs_fstatat+0x5a/0xc0 [ 2178.927333] __do_sys_newstat+0x30/0x70 [ 2178.931189] ? lock_release+0xc5/0x260 [ 2178.934960] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2178.939854] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2178.945014] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2178.950170] do_syscall_64+0x38/0x80 [ 2178.953767] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2178.958818] RIP: 0033:0x7f48f0eeca79 [ 2178.962416] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2178.981181] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2178.988767] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2178.995895] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2179.003046] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2179.010194] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2179.017335] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2179.024510] [ 2179.026695] irq event stamp: 12845745 [ 2179.030352] hardirqs last enabled at (12845755): [] console_unlock+0x106/0x120 [ 2179.039264] hardirqs last disabled at (12845776): [] console_unlock+0xeb/0x120 [ 2179.048064] softirqs last enabled at (12845800): [] __do_softirq+0x2be/0x371 [ 2179.056757] softirqs last disabled at (12845811): [] irq_exit_rcu+0xb0/0xd0 [ 2179.065294] ---[ end trace 0000000000000000 ]--- [ 2179.069914] ------------[ cut here ]------------ [ 2179.074547] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2179.086211] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2179.094907] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2179.150400] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2179.159544] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2179.167021] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2179.172534] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2179.178019] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2179.196800] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2179.202044] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2179.209196] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2179.216347] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002577 [ 2179.223512] R10: 0000000000106d58 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2179.230645] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2179.237794] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2179.245900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2179.251660] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2179.258804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2179.265954] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2179.273103] Call Trace: [ 2179.275567] [ 2179.277674] ? __warn+0x85/0x180 [ 2179.280905] ? __list_add_valid_or_report+0x7d/0xa0 [ 2179.285804] ? report_bug+0xfc/0x1d0 [ 2179.289398] ? prb_read_valid+0x1b/0x20 [ 2179.293275] ? handle_bug+0x44/0x70 [ 2179.296766] ? exc_invalid_op+0x17/0x60 [ 2179.300604] ? asm_exc_invalid_op+0x1a/0x20 [ 2179.304828] ? __list_add_valid_or_report+0x7d/0xa0 [ 2179.309709] ? __list_add_valid_or_report+0x7d/0xa0 [ 2179.314585] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2179.319499] ceph_add_cap+0x20b/0x620 [ceph] [ 2179.323811] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2179.328533] ? lock_acquire+0xcb/0x2b0 [ 2179.332303] ? lock_is_held_type+0xaa/0x120 [ 2179.336511] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2179.340966] ? __mutex_lock+0x97/0xce0 [ 2179.344720] ? __mutex_lock+0xaec/0xce0 [ 2179.348561] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2179.353110] ? lock_downgrade+0xfd/0x250 [ 2179.357077] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2179.361472] ceph_con_process_message+0x74/0x150 [libceph] [ 2179.366967] ? calc_signature+0xdb/0x100 [libceph] [ 2179.371815] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2179.377181] ? process_scheduled_works+0x22a/0x510 [ 2179.381998] ? process_scheduled_works+0x22a/0x510 [ 2179.386786] ceph_con_workfn+0x344/0x750 [libceph] [ 2179.391594] ? process_scheduled_works+0x22a/0x510 [ 2179.396389] process_scheduled_works+0x2a4/0x510 [ 2179.401052] worker_thread+0x188/0x340 [ 2179.404816] ? rescuer_thread+0x310/0x310 [ 2179.408845] kthread+0xf5/0x130 [ 2179.412010] ? kthread_complete_and_exit+0x20/0x20 [ 2179.416839] ret_from_fork+0x31/0x40 [ 2179.420414] ? kthread_complete_and_exit+0x20/0x20 [ 2179.425225] ret_from_fork_asm+0x11/0x20 [ 2179.429195] [ 2179.431388] irq event stamp: 10276291 [ 2179.435071] hardirqs last enabled at (10276301): [] console_unlock+0x106/0x120 [ 2179.443953] hardirqs last disabled at (10276310): [] console_unlock+0xeb/0x120 [ 2179.452750] softirqs last enabled at (10276248): [] __do_softirq+0x2be/0x371 [ 2179.461441] softirqs last disabled at (10276243): [] irq_exit_rcu+0xb0/0xd0 [ 2179.469979] ---[ end trace 0000000000000000 ]--- [ 2179.474617] ------------[ cut here ]------------ [ 2179.479298] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2179.490966] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2179.499922] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2179.555402] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2179.564369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2179.571856] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2179.577361] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2179.596125] RSP: 0018:ffffc90001fffac0 EFLAGS: 00010282 [ 2179.601369] RAX: 0000000000000000 RBX: ffff8881d11409c8 RCX: 0000000000000000 [ 2179.608514] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2179.615642] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000025b7 [ 2179.622775] R10: 0000000000107ea8 R11: ffffc90001fff950 R12: ffff8881d1141118 [ 2179.629908] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2179.637049] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2179.645152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2179.650907] CR2: 00007fb38dde2068 CR3: 000000017eb7a001 CR4: 00000000003706f0 [ 2179.658048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2179.665190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2179.672330] Call Trace: [ 2179.674810] [ 2179.676926] ? __warn+0x85/0x180 [ 2179.680176] ? __list_add_valid_or_report+0x7d/0xa0 [ 2179.685074] ? report_bug+0xfc/0x1d0 [ 2179.688669] ? prb_read_valid+0x1b/0x20 [ 2179.692526] ? handle_bug+0x44/0x70 [ 2179.696017] ? exc_invalid_op+0x17/0x60 [ 2179.699867] ? asm_exc_invalid_op+0x1a/0x20 [ 2179.704092] ? __list_add_valid_or_report+0x7d/0xa0 [ 2179.708968] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2179.713873] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2179.719142] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2179.723944] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2179.728679] lookup_fast+0x6d/0xf0 [ 2179.732081] walk_component+0x26/0x160 [ 2179.735851] link_path_walk+0x24d/0x380 [ 2179.739707] ? path_lookupat.isra.40+0x22/0x120 [ 2179.744241] path_lookupat.isra.40+0x6b/0x120 [ 2179.748620] filename_lookup+0xd4/0x190 [ 2179.752479] ? lock_acquire+0xcb/0x2b0 [ 2179.756225] ? __might_fault+0x45/0x80 [ 2179.759997] ? find_held_lock+0x2d/0x90 [ 2179.763854] ? __might_fault+0x45/0x80 [ 2179.767627] vfs_statx+0x89/0x150 [ 2179.770963] vfs_fstatat+0x5a/0xc0 [ 2179.774366] __do_sys_newstat+0x30/0x70 [ 2179.778223] ? lock_release+0xc5/0x260 [ 2179.781994] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2179.786890] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2179.792047] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2179.797203] do_syscall_64+0x38/0x80 [ 2179.800815] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2179.805878] RIP: 0033:0x7f48f0eeca79 [ 2179.809468] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2179.828231] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2179.835829] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2179.842973] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2179.850116] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2179.857265] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2179.864414] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2179.871564] [ 2179.873748] irq event stamp: 12848405 [ 2179.877406] hardirqs last enabled at (12848415): [] console_unlock+0x106/0x120 [ 2179.886288] hardirqs last disabled at (12848424): [] console_unlock+0xeb/0x120 [ 2179.895079] softirqs last enabled at (12848438): [] __do_softirq+0x2be/0x371 [ 2179.903788] softirqs last disabled at (12848433): [] irq_exit_rcu+0xb0/0xd0 [ 2179.912306] ---[ end trace 0000000000000000 ]--- [ 2179.916935] ------------[ cut here ]------------ [ 2179.921566] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2179.933213] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2179.941909] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2179.997385] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2180.006528] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2180.014006] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2180.019527] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2180.025010] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2180.043759] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2180.048985] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2180.056118] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2180.063250] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000025fa [ 2180.070384] R10: 0000000000109088 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2180.077534] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2180.084666] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2180.092754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.098498] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2180.105631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2180.112765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2180.119895] Call Trace: [ 2180.122350] [ 2180.124457] ? __warn+0x85/0x180 [ 2180.127690] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.132569] ? report_bug+0xfc/0x1d0 [ 2180.136149] ? prb_read_valid+0x1b/0x20 [ 2180.139990] ? handle_bug+0x44/0x70 [ 2180.143480] ? exc_invalid_op+0x17/0x60 [ 2180.147319] ? asm_exc_invalid_op+0x1a/0x20 [ 2180.151518] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.156420] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.161299] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2180.166192] ceph_add_cap+0x20b/0x620 [ceph] [ 2180.170485] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2180.175213] ceph_fill_trace+0x143/0xf30 [ceph] [ 2180.179753] ? __mutex_lock+0x97/0xce0 [ 2180.183502] ? __mutex_lock+0xaec/0xce0 [ 2180.187344] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2180.191897] ? lock_downgrade+0xfd/0x250 [ 2180.195834] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2180.200215] ceph_con_process_message+0x74/0x150 [libceph] [ 2180.205706] ? calc_signature+0xdb/0x100 [libceph] [ 2180.210541] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2180.215870] ? process_scheduled_works+0x22a/0x510 [ 2180.220672] ? process_scheduled_works+0x22a/0x510 [ 2180.225469] ceph_con_workfn+0x344/0x750 [libceph] [ 2180.230273] ? process_scheduled_works+0x22a/0x510 [ 2180.235062] process_scheduled_works+0x2a4/0x510 [ 2180.239689] worker_thread+0x188/0x340 [ 2180.243444] ? rescuer_thread+0x310/0x310 [ 2180.247455] kthread+0xf5/0x130 [ 2180.250603] ? kthread_complete_and_exit+0x20/0x20 [ 2180.255397] ret_from_fork+0x31/0x40 [ 2180.258975] ? kthread_complete_and_exit+0x20/0x20 [ 2180.263768] ret_from_fork_asm+0x11/0x20 [ 2180.267703] [ 2180.269892] irq event stamp: 10278247 [ 2180.273560] hardirqs last enabled at (10278257): [] console_unlock+0x106/0x120 [ 2180.282425] hardirqs last disabled at (10278266): [] console_unlock+0xeb/0x120 [ 2180.291205] softirqs last enabled at (10278064): [] __do_softirq+0x2be/0x371 [ 2180.299897] softirqs last disabled at (10278059): [] irq_exit_rcu+0xb0/0xd0 [ 2180.308417] ---[ end trace 0000000000000000 ]--- [ 2180.313083] ------------[ cut here ]------------ [ 2180.317705] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2180.329364] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2180.338057] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2180.393552] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2180.402678] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2180.410158] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2180.415671] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2180.421154] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2180.439920] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2180.445161] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2180.452313] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2180.459463] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002638 [ 2180.466614] R10: 000000000010a180 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2180.473745] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2180.480879] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2180.488982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.494745] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2180.501878] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2180.509026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2180.516177] Call Trace: [ 2180.518649] [ 2180.520754] ? __warn+0x85/0x180 [ 2180.523988] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.528885] ? report_bug+0xfc/0x1d0 [ 2180.532480] ? prb_read_valid+0x1b/0x20 [ 2180.536358] ? handle_bug+0x44/0x70 [ 2180.539847] ? exc_invalid_op+0x17/0x60 [ 2180.543705] ? asm_exc_invalid_op+0x1a/0x20 [ 2180.547916] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.552799] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.557675] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2180.562567] ceph_add_cap+0x20b/0x620 [ceph] [ 2180.566869] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2180.571593] ? lock_acquire+0xcb/0x2b0 [ 2180.575368] ? lock_is_held_type+0xaa/0x120 [ 2180.579555] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2180.584007] ? __mutex_lock+0x97/0xce0 [ 2180.587774] ? __mutex_lock+0xaec/0xce0 [ 2180.591616] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2180.596155] ? lock_downgrade+0xfd/0x250 [ 2180.600124] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2180.604529] ceph_con_process_message+0x74/0x150 [libceph] [ 2180.610023] ? calc_signature+0xdb/0x100 [libceph] [ 2180.614846] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2180.620195] ? process_scheduled_works+0x22a/0x510 [ 2180.625013] ? process_scheduled_works+0x22a/0x510 [ 2180.629800] ceph_con_workfn+0x344/0x750 [libceph] [ 2180.634605] ? process_scheduled_works+0x22a/0x510 [ 2180.639394] process_scheduled_works+0x2a4/0x510 [ 2180.644056] worker_thread+0x188/0x340 [ 2180.647826] ? rescuer_thread+0x310/0x310 [ 2180.651857] kthread+0xf5/0x130 [ 2180.655021] ? kthread_complete_and_exit+0x20/0x20 [ 2180.659851] ret_from_fork+0x31/0x40 [ 2180.663427] ? kthread_complete_and_exit+0x20/0x20 [ 2180.668239] ret_from_fork_asm+0x11/0x20 [ 2180.672208] [ 2180.674399] irq event stamp: 10279353 [ 2180.678082] hardirqs last enabled at (10279363): [] console_unlock+0x106/0x120 [ 2180.686966] hardirqs last disabled at (10279372): [] console_unlock+0xeb/0x120 [ 2180.695763] softirqs last enabled at (10278878): [] __do_softirq+0x2be/0x371 [ 2180.704457] softirqs last disabled at (10278873): [] irq_exit_rcu+0xb0/0xd0 [ 2180.713013] ---[ end trace 0000000000000000 ]--- [ 2180.717637] ------------[ cut here ]------------ [ 2180.722279] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2180.733939] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2180.742634] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2180.798112] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2180.807253] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2180.814751] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2180.820254] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2180.825758] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2180.844504] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2180.849757] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2180.856887] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2180.864038] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002678 [ 2180.871189] R10: 000000000010b408 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2180.878339] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2180.885490] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2180.893592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.899339] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2180.906488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2180.913640] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2180.920771] Call Trace: [ 2180.923226] [ 2180.925348] ? __warn+0x85/0x180 [ 2180.928601] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.933479] ? report_bug+0xfc/0x1d0 [ 2180.937075] ? prb_read_valid+0x1b/0x20 [ 2180.940951] ? handle_bug+0x44/0x70 [ 2180.944442] ? exc_invalid_op+0x17/0x60 [ 2180.948299] ? asm_exc_invalid_op+0x1a/0x20 [ 2180.952535] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.957418] ? __list_add_valid_or_report+0x7d/0xa0 [ 2180.962317] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2180.967231] ceph_add_cap+0x20b/0x620 [ceph] [ 2180.971555] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2180.976299] ceph_fill_trace+0x143/0xf30 [ceph] [ 2180.980843] ? __mutex_lock+0x97/0xce0 [ 2180.984612] ? __mutex_lock+0xaec/0xce0 [ 2180.988470] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2180.993019] ? lock_downgrade+0xfd/0x250 [ 2180.996989] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2181.001385] ceph_con_process_message+0x74/0x150 [libceph] [ 2181.006885] ? calc_signature+0xdb/0x100 [libceph] [ 2181.011718] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2181.017068] ? process_scheduled_works+0x22a/0x510 [ 2181.021885] ? process_scheduled_works+0x22a/0x510 [ 2181.026681] ceph_con_workfn+0x344/0x750 [libceph] [ 2181.031511] ? process_scheduled_works+0x22a/0x510 [ 2181.036300] process_scheduled_works+0x2a4/0x510 [ 2181.040963] worker_thread+0x188/0x340 [ 2181.044717] ? rescuer_thread+0x310/0x310 [ 2181.048729] kthread+0xf5/0x130 [ 2181.051876] ? kthread_complete_and_exit+0x20/0x20 [ 2181.056687] ret_from_fork+0x31/0x40 [ 2181.060264] ? kthread_complete_and_exit+0x20/0x20 [ 2181.065075] ret_from_fork_asm+0x11/0x20 [ 2181.069030] [ 2181.071220] irq event stamp: 10280413 [ 2181.074902] hardirqs last enabled at (10280423): [] console_unlock+0x106/0x120 [ 2181.083787] hardirqs last disabled at (10280432): [] console_unlock+0xeb/0x120 [ 2181.092565] softirqs last enabled at (10280178): [] __do_softirq+0x2be/0x371 [ 2181.101257] softirqs last disabled at (10280173): [] irq_exit_rcu+0xb0/0xd0 [ 2181.109794] ---[ end trace 0000000000000000 ]--- [ 2181.114416] ------------[ cut here ]------------ [ 2181.119055] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2181.130699] WARNING: CPU: 3 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2181.139655] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2181.195120] CPU: 3 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2181.204079] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2181.211561] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2181.217046] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2181.235791] RSP: 0018:ffffc90001fc7b38 EFLAGS: 00010282 [ 2181.241018] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2181.248150] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2181.255274] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000026b6 [ 2181.262400] R10: 000000000010c500 R11: ffffc90001fc79c8 R12: ffff8881d106df58 [ 2181.269536] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2181.276667] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2181.284754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2181.290499] CR2: 00007fb38ddf00a8 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2181.297634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2181.304765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2181.311899] Call Trace: [ 2181.314352] [ 2181.316457] ? __warn+0x85/0x180 [ 2181.319693] ? __list_add_valid_or_report+0x7d/0xa0 [ 2181.324580] ? report_bug+0xfc/0x1d0 [ 2181.328159] ? prb_read_valid+0x1b/0x20 [ 2181.331999] ? handle_bug+0x44/0x70 [ 2181.335491] ? exc_invalid_op+0x17/0x60 [ 2181.339328] ? asm_exc_invalid_op+0x1a/0x20 [ 2181.343523] ? __list_add_valid_or_report+0x7d/0xa0 [ 2181.348413] ? __list_add_valid_or_report+0x7d/0xa0 [ 2181.353291] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2181.358191] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2181.362918] lookup_fast+0x6d/0xf0 [ 2181.366327] walk_component+0x26/0x160 [ 2181.370080] link_path_walk+0x24d/0x380 [ 2181.373918] ? path_lookupat.isra.40+0x22/0x120 [ 2181.378455] path_lookupat.isra.40+0x6b/0x120 [ 2181.382814] filename_lookup+0xd4/0x190 [ 2181.386666] ? lock_acquire+0xcb/0x2b0 [ 2181.390420] ? __might_fault+0x45/0x80 [ 2181.394175] ? find_held_lock+0x2d/0x90 [ 2181.398016] ? __might_fault+0x45/0x80 [ 2181.401779] vfs_statx+0x89/0x150 [ 2181.405106] vfs_fstatat+0x5a/0xc0 [ 2181.408519] __do_sys_newstat+0x30/0x70 [ 2181.412358] ? lock_release+0xc5/0x260 [ 2181.416113] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2181.420995] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2181.426137] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2181.431276] do_syscall_64+0x38/0x80 [ 2181.434855] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2181.439906] RIP: 0033:0x7f48f0eeca79 [ 2181.443487] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2181.462235] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2181.469800] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2181.476931] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2181.484066] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2181.491197] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2181.498332] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2181.505468] [ 2181.507652] irq event stamp: 12913347 [ 2181.511309] hardirqs last enabled at (12913357): [] console_unlock+0x106/0x120 [ 2181.520177] hardirqs last disabled at (12913366): [] console_unlock+0xeb/0x120 [ 2181.528958] softirqs last enabled at (12913380): [] __do_softirq+0x2be/0x371 [ 2181.537650] softirqs last disabled at (12913375): [] irq_exit_rcu+0xb0/0xd0 [ 2181.546167] ---[ end trace 0000000000000000 ]--- [ 2181.550781] ------------[ cut here ]------------ [ 2181.555400] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2181.567046] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2181.575742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2181.631228] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2181.640352] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2181.647831] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2181.653336] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2181.658821] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2181.677566] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2181.682795] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2181.689925] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2181.697059] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000026f8 [ 2181.704191] R10: 000000000010d6d0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2181.711325] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2181.718458] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2181.726546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2181.732290] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2181.739428] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2181.746564] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2181.753696] Call Trace: [ 2181.756149] [ 2181.758256] ? __warn+0x85/0x180 [ 2181.761490] ? __list_add_valid_or_report+0x7d/0xa0 [ 2181.766369] ? report_bug+0xfc/0x1d0 [ 2181.769948] ? prb_read_valid+0x1b/0x20 [ 2181.773791] ? handle_bug+0x44/0x70 [ 2181.777279] ? exc_invalid_op+0x17/0x60 [ 2181.781117] ? asm_exc_invalid_op+0x1a/0x20 [ 2181.785315] ? __list_add_valid_or_report+0x7d/0xa0 [ 2181.790195] ? __list_add_valid_or_report+0x7d/0xa0 [ 2181.795074] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2181.799975] ceph_add_cap+0x20b/0x620 [ceph] [ 2181.804268] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2181.808992] ? lock_acquire+0xcb/0x2b0 [ 2181.812748] ? lock_is_held_type+0xaa/0x120 [ 2181.816935] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2181.821396] ? __mutex_lock+0x97/0xce0 [ 2181.825146] ? __mutex_lock+0xaec/0xce0 [ 2181.828988] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2181.833541] ? lock_downgrade+0xfd/0x250 [ 2181.837480] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2181.841860] ceph_con_process_message+0x74/0x150 [libceph] [ 2181.847363] ? calc_signature+0xdb/0x100 [libceph] [ 2181.852171] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2181.857500] ? process_scheduled_works+0x22a/0x510 [ 2181.862301] ? process_scheduled_works+0x22a/0x510 [ 2181.867095] ceph_con_workfn+0x344/0x750 [libceph] [ 2181.871908] ? process_scheduled_works+0x22a/0x510 [ 2181.876696] process_scheduled_works+0x2a4/0x510 [ 2181.881326] worker_thread+0x188/0x340 [ 2181.885081] ? rescuer_thread+0x310/0x310 [ 2181.889090] kthread+0xf5/0x130 [ 2181.892236] ? kthread_complete_and_exit+0x20/0x20 [ 2181.897032] ret_from_fork+0x31/0x40 [ 2181.900609] ? kthread_complete_and_exit+0x20/0x20 [ 2181.905403] ret_from_fork_asm+0x11/0x20 [ 2181.909339] [ 2181.911530] irq event stamp: 10282399 [ 2181.915196] hardirqs last enabled at (10282409): [] console_unlock+0x106/0x120 [ 2181.924060] hardirqs last disabled at (10282418): [] console_unlock+0xeb/0x120 [ 2181.932842] softirqs last enabled at (10281996): [] __do_softirq+0x2be/0x371 [ 2181.941534] softirqs last disabled at (10281991): [] irq_exit_rcu+0xb0/0xd0 [ 2181.950060] ---[ end trace 0000000000000000 ]--- [ 2181.954683] ------------[ cut here ]------------ [ 2181.959299] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2181.970946] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2181.979900] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2182.035367] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2182.044326] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2182.051801] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2182.057286] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2182.076032] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2182.081259] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2182.088391] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2182.095524] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002738 [ 2182.102659] R10: 000000000010e808 R11: ffffc90001fcf9c8 R12: ffff8881d106b838 [ 2182.109788] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2182.116914] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2182.125000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2182.130745] CR2: 00007fb38ddfb000 CR3: 000000011aeae003 CR4: 00000000003706f0 [ 2182.137869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2182.144994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2182.152119] Call Trace: [ 2182.154570] [ 2182.156669] ? __warn+0x85/0x180 [ 2182.159903] ? __list_add_valid_or_report+0x7d/0xa0 [ 2182.164780] ? report_bug+0xfc/0x1d0 [ 2182.168361] ? prb_read_valid+0x1b/0x20 [ 2182.172200] ? handle_bug+0x44/0x70 [ 2182.175693] ? exc_invalid_op+0x17/0x60 [ 2182.179532] ? asm_exc_invalid_op+0x1a/0x20 [ 2182.183723] ? __list_add_valid_or_report+0x7d/0xa0 [ 2182.188598] ? __list_add_valid_or_report+0x7d/0xa0 [ 2182.193478] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2182.198367] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2182.203085] lookup_fast+0x6d/0xf0 [ 2182.206486] walk_component+0x26/0x160 [ 2182.210241] link_path_walk+0x24d/0x380 [ 2182.214078] ? path_lookupat.isra.40+0x22/0x120 [ 2182.218615] path_lookupat.isra.40+0x6b/0x120 [ 2182.222972] filename_lookup+0xd4/0x190 [ 2182.226812] ? lock_acquire+0xcb/0x2b0 [ 2182.230564] ? __might_fault+0x45/0x80 [ 2182.234316] ? find_held_lock+0x2d/0x90 [ 2182.238154] ? __might_fault+0x45/0x80 [ 2182.241911] vfs_statx+0x89/0x150 [ 2182.245230] vfs_fstatat+0x5a/0xc0 [ 2182.248633] __do_sys_newstat+0x30/0x70 [ 2182.252471] ? lock_release+0xc5/0x260 [ 2182.256228] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2182.261104] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2182.266243] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2182.271385] do_syscall_64+0x38/0x80 [ 2182.274961] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2182.280017] RIP: 0033:0x7f48f0eeca79 [ 2182.283594] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2182.302350] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2182.309930] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2182.317057] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2182.324190] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2182.331323] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2182.338455] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2182.345595] [ 2182.347779] irq event stamp: 12787933 [ 2182.351438] hardirqs last enabled at (12787943): [] console_unlock+0x106/0x120 [ 2182.360303] hardirqs last disabled at (12787952): [] console_unlock+0xeb/0x120 [ 2182.369087] softirqs last enabled at (12787966): [] __do_softirq+0x2be/0x371 [ 2182.377777] softirqs last disabled at (12787961): [] irq_exit_rcu+0xb0/0xd0 [ 2182.386295] ---[ end trace 0000000000000000 ]--- [ 2182.390907] ------------[ cut here ]------------ [ 2182.395525] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2182.407175] WARNING: CPU: 7 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2182.416128] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2182.471590] CPU: 7 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2182.480539] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2182.488019] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2182.493505] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2182.512250] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2182.517476] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2182.524600] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2182.531726] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000277a [ 2182.538858] R10: 000000000010fae8 R11: ffffc90001fc7950 R12: ffff8881d106df58 [ 2182.545991] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2182.553125] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2182.561210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2182.566960] CR2: 00007fb38de00010 CR3: 0000000113500006 CR4: 00000000003706e0 [ 2182.574089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2182.581222] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2182.588354] Call Trace: [ 2182.590809] [ 2182.592912] ? __warn+0x85/0x180 [ 2182.596146] ? __list_add_valid_or_report+0x7d/0xa0 [ 2182.601027] ? report_bug+0xfc/0x1d0 [ 2182.604607] ? prb_read_valid+0x1b/0x20 [ 2182.608447] ? handle_bug+0x44/0x70 [ 2182.611936] ? exc_invalid_op+0x17/0x60 [ 2182.615768] ? asm_exc_invalid_op+0x1a/0x20 [ 2182.619959] ? __list_add_valid_or_report+0x7d/0xa0 [ 2182.624835] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2182.629724] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2182.634960] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2182.639760] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2182.644476] lookup_fast+0x6d/0xf0 [ 2182.647880] walk_component+0x26/0x160 [ 2182.651634] link_path_walk+0x24d/0x380 [ 2182.655471] ? path_lookupat.isra.40+0x22/0x120 [ 2182.660006] path_lookupat.isra.40+0x6b/0x120 [ 2182.664365] filename_lookup+0xd4/0x190 [ 2182.668208] ? lock_acquire+0xcb/0x2b0 [ 2182.671956] ? __might_fault+0x45/0x80 [ 2182.675708] ? find_held_lock+0x2d/0x90 [ 2182.679546] ? __might_fault+0x45/0x80 [ 2182.683303] vfs_statx+0x89/0x150 [ 2182.686622] vfs_fstatat+0x5a/0xc0 [ 2182.690026] __do_sys_newstat+0x30/0x70 [ 2182.693866] ? lock_release+0xc5/0x260 [ 2182.697620] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2182.702496] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2182.707638] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2182.712777] do_syscall_64+0x38/0x80 [ 2182.716354] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2182.721408] RIP: 0033:0x7f48f0eeca79 [ 2182.724987] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2182.743735] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2182.751299] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2182.758432] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2182.765564] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2182.772688] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2182.779813] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2182.786943] [ 2182.789131] irq event stamp: 12916337 [ 2182.792795] hardirqs last enabled at (12916347): [] console_unlock+0x106/0x120 [ 2182.801662] hardirqs last disabled at (12916356): [] console_unlock+0xeb/0x120 [ 2182.810442] softirqs last enabled at (12916370): [] __do_softirq+0x2be/0x371 [ 2182.819136] softirqs last disabled at (12916381): [] irq_exit_rcu+0xb0/0xd0 [ 2182.827674] ---[ end trace 0000000000000000 ]--- [ 2182.832294] ------------[ cut here ]------------ [ 2182.836910] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2182.848592] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2182.857546] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2182.913034] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2182.922003] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2182.929498] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2182.935001] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2182.953768] RSP: 0018:ffffc90001fcfac0 EFLAGS: 00010282 [ 2182.958991] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2182.966133] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2182.973283] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000027bd [ 2182.980433] R10: 0000000000110cc8 R11: ffffc90001fcf950 R12: ffff8881d106b838 [ 2182.987587] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2182.994715] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2183.002802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2183.008548] CR2: 00007fb38de0c030 CR3: 000000011aeae004 CR4: 00000000003706e0 [ 2183.015691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2183.022821] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2183.029947] Call Trace: [ 2183.032408] [ 2183.034522] ? __warn+0x85/0x180 [ 2183.037774] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.042653] ? report_bug+0xfc/0x1d0 [ 2183.046255] ? prb_read_valid+0x1b/0x20 [ 2183.050110] ? handle_bug+0x44/0x70 [ 2183.053604] ? exc_invalid_op+0x17/0x60 [ 2183.057459] ? asm_exc_invalid_op+0x1a/0x20 [ 2183.061647] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.066521] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2183.071427] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2183.076681] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2183.081481] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2183.086231] lookup_fast+0x6d/0xf0 [ 2183.089636] walk_component+0x26/0x160 [ 2183.093388] link_path_walk+0x24d/0x380 [ 2183.097245] ? path_lookupat.isra.40+0x22/0x120 [ 2183.101798] path_lookupat.isra.40+0x6b/0x120 [ 2183.106158] filename_lookup+0xd4/0x190 [ 2183.110032] ? lock_acquire+0xcb/0x2b0 [ 2183.113781] ? __might_fault+0x45/0x80 [ 2183.117534] ? find_held_lock+0x2d/0x90 [ 2183.121391] ? __might_fault+0x45/0x80 [ 2183.125163] vfs_statx+0x89/0x150 [ 2183.128500] vfs_fstatat+0x5a/0xc0 [ 2183.131921] __do_sys_newstat+0x30/0x70 [ 2183.135778] ? lock_release+0xc5/0x260 [ 2183.139532] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2183.144425] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2183.149591] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2183.154733] do_syscall_64+0x38/0x80 [ 2183.158310] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2183.163380] RIP: 0033:0x7f48f0eeca79 [ 2183.166970] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2183.185736] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2183.193320] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2183.200450] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2183.207599] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2183.214736] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2183.221879] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2183.229046] [ 2183.231234] irq event stamp: 12790221 [ 2183.234918] hardirqs last enabled at (12790231): [] console_unlock+0x106/0x120 [ 2183.243801] hardirqs last disabled at (12790250): [] console_unlock+0xeb/0x120 [ 2183.252594] softirqs last enabled at (12790264): [] __do_softirq+0x2be/0x371 [ 2183.261280] softirqs last disabled at (12790259): [] irq_exit_rcu+0xb0/0xd0 [ 2183.269819] ---[ end trace 0000000000000000 ]--- [ 2183.274439] ------------[ cut here ]------------ [ 2183.279074] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2183.290723] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2183.299417] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2183.354893] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2183.364020] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2183.371498] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2183.376993] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2183.382478] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2183.401226] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2183.406452] RAX: 0000000000000000 RBX: ffff8881c277f560 RCX: 0000000000000000 [ 2183.413598] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2183.420727] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002800 [ 2183.427875] R10: 0000000000111eb0 R11: ffffc9000030b828 R12: ffff8881c277fcb0 [ 2183.435008] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2183.442141] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2183.450229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2183.455976] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2183.463125] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2183.470257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2183.477389] Call Trace: [ 2183.479844] [ 2183.481950] ? __warn+0x85/0x180 [ 2183.485182] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.490064] ? report_bug+0xfc/0x1d0 [ 2183.493641] ? prb_read_valid+0x1b/0x20 [ 2183.497484] ? handle_bug+0x44/0x70 [ 2183.500972] ? exc_invalid_op+0x17/0x60 [ 2183.504814] ? asm_exc_invalid_op+0x1a/0x20 [ 2183.509006] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.513890] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.518765] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2183.523661] ceph_add_cap+0x20b/0x620 [ceph] [ 2183.527952] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2183.532682] ceph_fill_trace+0x143/0xf30 [ceph] [ 2183.537227] ? __mutex_lock+0x97/0xce0 [ 2183.540979] ? __mutex_lock+0xaec/0xce0 [ 2183.544822] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2183.549368] ? lock_downgrade+0xfd/0x250 [ 2183.553302] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2183.557681] ceph_con_process_message+0x74/0x150 [libceph] [ 2183.563174] ? calc_signature+0xdb/0x100 [libceph] [ 2183.567989] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2183.573321] ? process_scheduled_works+0x22a/0x510 [ 2183.578122] ? process_scheduled_works+0x22a/0x510 [ 2183.582909] ceph_con_workfn+0x344/0x750 [libceph] [ 2183.587716] ? process_scheduled_works+0x22a/0x510 [ 2183.592511] process_scheduled_works+0x2a4/0x510 [ 2183.597139] worker_thread+0x188/0x340 [ 2183.600894] ? rescuer_thread+0x310/0x310 [ 2183.604905] kthread+0xf5/0x130 [ 2183.608051] ? kthread_complete_and_exit+0x20/0x20 [ 2183.612847] ret_from_fork+0x31/0x40 [ 2183.616422] ? kthread_complete_and_exit+0x20/0x20 [ 2183.621217] ret_from_fork_asm+0x11/0x20 [ 2183.625152] [ 2183.627344] irq event stamp: 10286141 [ 2183.631008] hardirqs last enabled at (10286151): [] console_unlock+0x106/0x120 [ 2183.639889] hardirqs last disabled at (10286160): [] console_unlock+0xeb/0x120 [ 2183.648664] softirqs last enabled at (10285898): [] __do_softirq+0x2be/0x371 [ 2183.657357] softirqs last disabled at (10285893): [] irq_exit_rcu+0xb0/0xd0 [ 2183.665891] ---[ end trace 0000000000000000 ]--- [ 2183.670684] ------------[ cut here ]------------ [ 2183.675323] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2183.686987] WARNING: CPU: 4 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2183.695941] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2183.751446] CPU: 4 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2183.760398] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2183.767893] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2183.773395] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2183.792160] RSP: 0018:ffffc90001fe7b38 EFLAGS: 00010282 [ 2183.797401] RAX: 0000000000000000 RBX: ffff8881c277f560 RCX: 0000000000000000 [ 2183.804543] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2183.811684] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000283e [ 2183.818810] R10: 0000000000112fe0 R11: ffffc90001fe79c8 R12: ffff8881c277fcb0 [ 2183.825942] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2183.833082] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2183.841178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2183.846934] CR2: 00007f9bd320e000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2183.854083] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2183.861215] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2183.868358] Call Trace: [ 2183.870821] [ 2183.872928] ? __warn+0x85/0x180 [ 2183.876176] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.881072] ? report_bug+0xfc/0x1d0 [ 2183.884672] ? prb_read_valid+0x1b/0x20 [ 2183.888510] ? handle_bug+0x44/0x70 [ 2183.892017] ? exc_invalid_op+0x17/0x60 [ 2183.895883] ? asm_exc_invalid_op+0x1a/0x20 [ 2183.900108] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.904985] ? __list_add_valid_or_report+0x7d/0xa0 [ 2183.909893] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2183.914798] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2183.919530] lookup_fast+0x6d/0xf0 [ 2183.922933] walk_component+0x26/0x160 [ 2183.926704] link_path_walk+0x24d/0x380 [ 2183.930541] ? path_lookupat.isra.40+0x22/0x120 [ 2183.935086] path_lookupat.isra.40+0x6b/0x120 [ 2183.939462] filename_lookup+0xd4/0x190 [ 2183.943337] ? lock_acquire+0xcb/0x2b0 [ 2183.947088] ? __might_fault+0x45/0x80 [ 2183.950860] ? find_held_lock+0x2d/0x90 [ 2183.954697] ? __might_fault+0x45/0x80 [ 2183.958468] vfs_statx+0x89/0x150 [ 2183.961791] vfs_fstatat+0x5a/0xc0 [ 2183.965192] __do_sys_newstat+0x30/0x70 [ 2183.969050] ? lock_release+0xc5/0x260 [ 2183.972822] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2183.977700] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2183.982838] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2183.987979] do_syscall_64+0x38/0x80 [ 2183.991575] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2183.996645] RIP: 0033:0x7f48f0eeca79 [ 2184.000224] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2184.018987] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2184.026571] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2184.033723] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2184.040852] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2184.048000] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2184.055147] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2184.062300] [ 2184.064489] irq event stamp: 12809683 [ 2184.068162] hardirqs last enabled at (12809693): [] console_unlock+0x106/0x120 [ 2184.077045] hardirqs last disabled at (12809702): [] console_unlock+0xeb/0x120 [ 2184.085839] softirqs last enabled at (12809724): [] __do_softirq+0x2be/0x371 [ 2184.094548] softirqs last disabled at (12809735): [] irq_exit_rcu+0xb0/0xd0 [ 2184.103063] ---[ end trace 0000000000000000 ]--- [ 2184.107692] ------------[ cut here ]------------ [ 2184.112313] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2184.123977] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2184.132673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2184.188158] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2184.197300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2184.204796] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2184.210291] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2184.215796] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2184.234540] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2184.239784] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2184.246918] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2184.254067] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002880 [ 2184.261218] R10: 0000000000114140 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2184.268366] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2184.275518] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2184.283620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2184.289387] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2184.296518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2184.303667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2184.310801] Call Trace: [ 2184.313255] [ 2184.315378] ? __warn+0x85/0x180 [ 2184.318627] ? __list_add_valid_or_report+0x7d/0xa0 [ 2184.323507] ? report_bug+0xfc/0x1d0 [ 2184.327103] ? prb_read_valid+0x1b/0x20 [ 2184.330979] ? handle_bug+0x44/0x70 [ 2184.334471] ? exc_invalid_op+0x17/0x60 [ 2184.338329] ? asm_exc_invalid_op+0x1a/0x20 [ 2184.342554] ? __list_add_valid_or_report+0x7d/0xa0 [ 2184.347453] ? __list_add_valid_or_report+0x7d/0xa0 [ 2184.352333] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2184.357259] ceph_add_cap+0x20b/0x620 [ceph] [ 2184.361569] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2184.366291] ? lock_acquire+0xcb/0x2b0 [ 2184.370059] ? lock_is_held_type+0xaa/0x120 [ 2184.374264] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2184.378724] ? __mutex_lock+0x97/0xce0 [ 2184.382475] ? __mutex_lock+0xaec/0xce0 [ 2184.386352] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2184.390910] ? lock_downgrade+0xfd/0x250 [ 2184.394860] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2184.399256] ceph_con_process_message+0x74/0x150 [libceph] [ 2184.404759] ? calc_signature+0xdb/0x100 [libceph] [ 2184.409595] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2184.414928] ? process_scheduled_works+0x22a/0x510 [ 2184.419723] ? process_scheduled_works+0x22a/0x510 [ 2184.424519] ceph_con_workfn+0x344/0x750 [libceph] [ 2184.429357] ? process_scheduled_works+0x22a/0x510 [ 2184.434147] process_scheduled_works+0x2a4/0x510 [ 2184.438791] worker_thread+0x188/0x340 [ 2184.442564] ? rescuer_thread+0x310/0x310 [ 2184.446577] kthread+0xf5/0x130 [ 2184.449745] ? kthread_complete_and_exit+0x20/0x20 [ 2184.454558] ret_from_fork+0x31/0x40 [ 2184.458138] ? kthread_complete_and_exit+0x20/0x20 [ 2184.462947] ret_from_fork_asm+0x11/0x20 [ 2184.466933] [ 2184.469126] irq event stamp: 10288189 [ 2184.472810] hardirqs last enabled at (10288199): [] console_unlock+0x106/0x120 [ 2184.481675] hardirqs last disabled at (10288208): [] console_unlock+0xeb/0x120 [ 2184.490455] softirqs last enabled at (10287770): [] __do_softirq+0x2be/0x371 [ 2184.499164] softirqs last disabled at (10287765): [] irq_exit_rcu+0xb0/0xd0 [ 2184.507701] ---[ end trace 0000000000000000 ]--- [ 2184.512323] ------------[ cut here ]------------ [ 2184.516957] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2184.528622] WARNING: CPU: 0 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2184.537574] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2184.593063] CPU: 0 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2184.602031] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2184.609517] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2184.615043] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2184.633787] RSP: 0018:ffffc90001fe7ac0 EFLAGS: 00010282 [ 2184.639010] RAX: 0000000000000000 RBX: ffff8881c277f560 RCX: 0000000000000000 [ 2184.646153] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2184.653302] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000028c0 [ 2184.660446] R10: 00000000001153d8 R11: ffffc90001fe7950 R12: ffff8881c277fcb0 [ 2184.667613] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2184.674746] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2184.682830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2184.688567] CR2: 00007fb38ddfb000 CR3: 0000000108c08004 CR4: 00000000003706f0 [ 2184.695711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2184.702843] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2184.709977] Call Trace: [ 2184.712446] [ 2184.714562] ? __warn+0x85/0x180 [ 2184.717811] ? __list_add_valid_or_report+0x7d/0xa0 [ 2184.722692] ? report_bug+0xfc/0x1d0 [ 2184.726269] ? prb_read_valid+0x1b/0x20 [ 2184.730119] ? handle_bug+0x44/0x70 [ 2184.733626] ? exc_invalid_op+0x17/0x60 [ 2184.737459] ? asm_exc_invalid_op+0x1a/0x20 [ 2184.741666] ? __list_add_valid_or_report+0x7d/0xa0 [ 2184.746560] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2184.751448] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2184.756702] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2184.761502] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2184.766252] lookup_fast+0x6d/0xf0 [ 2184.769656] walk_component+0x26/0x160 [ 2184.773411] link_path_walk+0x24d/0x380 [ 2184.777265] ? path_lookupat.isra.40+0x22/0x120 [ 2184.781817] path_lookupat.isra.40+0x6b/0x120 [ 2184.786195] filename_lookup+0xd4/0x190 [ 2184.790053] ? lock_acquire+0xcb/0x2b0 [ 2184.793802] ? __might_fault+0x45/0x80 [ 2184.797555] ? find_held_lock+0x2d/0x90 [ 2184.801411] ? __might_fault+0x45/0x80 [ 2184.805184] vfs_statx+0x89/0x150 [ 2184.808538] vfs_fstatat+0x5a/0xc0 [ 2184.811941] __do_sys_newstat+0x30/0x70 [ 2184.815798] ? lock_release+0xc5/0x260 [ 2184.819552] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2184.824448] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2184.829621] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2184.834762] do_syscall_64+0x38/0x80 [ 2184.838340] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2184.843411] RIP: 0033:0x7f48f0eeca79 [ 2184.847008] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2184.865773] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2184.873336] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2184.880478] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2184.887621] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2184.894753] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2184.901900] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2184.909067] [ 2184.911254] irq event stamp: 12811741 [ 2184.914929] hardirqs last enabled at (12811751): [] console_unlock+0x106/0x120 [ 2184.923811] hardirqs last disabled at (12811760): [] console_unlock+0xeb/0x120 [ 2184.932608] softirqs last enabled at (12811776): [] __do_softirq+0x2be/0x371 [ 2184.941301] softirqs last disabled at (12811769): [] irq_exit_rcu+0xb0/0xd0 [ 2184.949839] ---[ end trace 0000000000000000 ]--- [ 2184.954461] ------------[ cut here ]------------ [ 2184.959095] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2184.970741] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2184.979436] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2185.034930] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2185.044057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2185.051535] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2185.057059] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2185.062543] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2185.081307] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2185.086534] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2185.093665] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2185.100799] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002903 [ 2185.107931] R10: 00000000001166f0 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2185.115063] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2185.122197] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2185.130282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2185.136029] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2185.143161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2185.150295] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2185.157428] Call Trace: [ 2185.159881] [ 2185.161987] ? __warn+0x85/0x180 [ 2185.165219] ? __list_add_valid_or_report+0x7d/0xa0 [ 2185.170100] ? report_bug+0xfc/0x1d0 [ 2185.173680] ? prb_read_valid+0x1b/0x20 [ 2185.177518] ? handle_bug+0x44/0x70 [ 2185.181011] ? exc_invalid_op+0x17/0x60 [ 2185.184851] ? asm_exc_invalid_op+0x1a/0x20 [ 2185.189040] ? __list_add_valid_or_report+0x7d/0xa0 [ 2185.193917] ? __list_add_valid_or_report+0x7d/0xa0 [ 2185.198795] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2185.203687] ceph_add_cap+0x20b/0x620 [ceph] [ 2185.207978] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2185.212726] ceph_fill_trace+0x143/0xf30 [ceph] [ 2185.217264] ? __mutex_lock+0x97/0xce0 [ 2185.221016] ? __mutex_lock+0xaec/0xce0 [ 2185.224857] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2185.229396] ? lock_downgrade+0xfd/0x250 [ 2185.233328] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2185.237726] ceph_con_process_message+0x74/0x150 [libceph] [ 2185.243220] ? calc_signature+0xdb/0x100 [libceph] [ 2185.248051] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2185.253398] ? process_scheduled_works+0x22a/0x510 [ 2185.258191] ? process_scheduled_works+0x22a/0x510 [ 2185.262980] ceph_con_workfn+0x344/0x750 [libceph] [ 2185.267787] ? process_scheduled_works+0x22a/0x510 [ 2185.272576] process_scheduled_works+0x2a4/0x510 [ 2185.277216] worker_thread+0x188/0x340 [ 2185.280966] ? rescuer_thread+0x310/0x310 [ 2185.284979] kthread+0xf5/0x130 [ 2185.288124] ? kthread_complete_and_exit+0x20/0x20 [ 2185.292918] ret_from_fork+0x31/0x40 [ 2185.296512] ? kthread_complete_and_exit+0x20/0x20 [ 2185.301307] ret_from_fork_asm+0x11/0x20 [ 2185.305256] [ 2185.307442] irq event stamp: 10290123 [ 2185.311106] hardirqs last enabled at (10290133): [] console_unlock+0x106/0x120 [ 2185.319975] hardirqs last disabled at (10290148): [] console_unlock+0xeb/0x120 [ 2185.328753] softirqs last enabled at (10290146): [] __do_softirq+0x2be/0x371 [ 2185.337445] softirqs last disabled at (10290141): [] irq_exit_rcu+0xb0/0xd0 [ 2185.345967] ---[ end trace 0000000000000000 ]--- [ 2185.350633] ------------[ cut here ]------------ [ 2185.355320] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2185.366988] WARNING: CPU: 4 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2185.375946] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2185.431414] CPU: 4 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2185.440384] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2185.447877] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2185.453364] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2185.472152] RSP: 0018:ffffc90001957b38 EFLAGS: 00010282 [ 2185.477381] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2185.484530] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2185.491670] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002941 [ 2185.498797] R10: 00000000001177e0 R11: ffffc900019579c8 R12: ffff8881d12224a8 [ 2185.505931] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2185.513101] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2185.521182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2185.526945] CR2: 00007f9bd320e000 CR3: 000000016696c001 CR4: 00000000003706e0 [ 2185.534096] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2185.541237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2185.548389] Call Trace: [ 2185.550861] [ 2185.552965] ? __warn+0x85/0x180 [ 2185.556213] ? __list_add_valid_or_report+0x7d/0xa0 [ 2185.561110] ? report_bug+0xfc/0x1d0 [ 2185.564710] ? prb_read_valid+0x1b/0x20 [ 2185.568548] ? handle_bug+0x44/0x70 [ 2185.572056] ? exc_invalid_op+0x17/0x60 [ 2185.575920] ? asm_exc_invalid_op+0x1a/0x20 [ 2185.580137] ? __list_add_valid_or_report+0x7d/0xa0 [ 2185.585015] ? __list_add_valid_or_report+0x7d/0xa0 [ 2185.589920] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2185.594829] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2185.599565] lookup_fast+0x6d/0xf0 [ 2185.602973] walk_component+0x26/0x160 [ 2185.606743] link_path_walk+0x24d/0x380 [ 2185.610582] ? path_lookupat.isra.40+0x22/0x120 [ 2185.615132] path_lookupat.isra.40+0x6b/0x120 [ 2185.619508] filename_lookup+0xd4/0x190 [ 2185.623367] ? lock_acquire+0xcb/0x2b0 [ 2185.627134] ? __might_fault+0x45/0x80 [ 2185.630923] ? find_held_lock+0x2d/0x90 [ 2185.634779] ? __might_fault+0x45/0x80 [ 2185.638533] vfs_statx+0x89/0x150 [ 2185.641870] vfs_fstatat+0x5a/0xc0 [ 2185.645274] __do_sys_newstat+0x30/0x70 [ 2185.649131] ? lock_release+0xc5/0x260 [ 2185.652919] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2185.657816] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2185.662954] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2185.668111] do_syscall_64+0x38/0x80 [ 2185.671708] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2185.676760] RIP: 0033:0x7f48f0eeca79 [ 2185.680340] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2185.699106] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2185.706686] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2185.713810] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2185.720943] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2185.728107] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2185.735234] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2185.742401] [ 2185.744587] irq event stamp: 12783719 [ 2185.748282] hardirqs last enabled at (12783739): [] console_unlock+0x106/0x120 [ 2185.757146] hardirqs last disabled at (12783748): [] console_unlock+0xeb/0x120 [ 2185.765963] softirqs last enabled at (12783768): [] __do_softirq+0x2be/0x371 [ 2185.774652] softirqs last disabled at (12783757): [] irq_exit_rcu+0xb0/0xd0 [ 2185.783171] ---[ end trace 0000000000000000 ]--- [ 2185.787799] ------------[ cut here ]------------ [ 2185.792420] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2185.804083] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2185.812779] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2185.868274] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2185.877398] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2185.884895] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2185.890426] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2185.895932] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2185.914692] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2185.919934] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2185.927085] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2185.934235] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002983 [ 2185.941384] R10: 0000000000118978 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2185.948535] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2185.955692] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2185.963782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2185.969527] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2185.976676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2185.983811] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2185.990942] Call Trace: [ 2185.993415] [ 2185.995537] ? __warn+0x85/0x180 [ 2185.998794] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.003679] ? report_bug+0xfc/0x1d0 [ 2186.007263] ? prb_read_valid+0x1b/0x20 [ 2186.011121] ? handle_bug+0x44/0x70 [ 2186.014643] ? exc_invalid_op+0x17/0x60 [ 2186.018477] ? asm_exc_invalid_op+0x1a/0x20 [ 2186.022686] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.027569] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.032465] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2186.037374] ceph_add_cap+0x20b/0x620 [ceph] [ 2186.041676] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2186.046415] ? lock_acquire+0xcb/0x2b0 [ 2186.050164] ? lock_is_held_type+0xaa/0x120 [ 2186.054369] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2186.058839] ? __mutex_lock+0x97/0xce0 [ 2186.062591] ? __mutex_lock+0xaec/0xce0 [ 2186.066448] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2186.071008] ? lock_downgrade+0xfd/0x250 [ 2186.074972] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2186.079370] ceph_con_process_message+0x74/0x150 [libceph] [ 2186.084866] ? calc_signature+0xdb/0x100 [libceph] [ 2186.089681] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2186.095026] ? process_scheduled_works+0x22a/0x510 [ 2186.099818] ? process_scheduled_works+0x22a/0x510 [ 2186.104608] ceph_con_workfn+0x344/0x750 [libceph] [ 2186.109436] ? process_scheduled_works+0x22a/0x510 [ 2186.114245] process_scheduled_works+0x2a4/0x510 [ 2186.118887] worker_thread+0x188/0x340 [ 2186.122650] ? rescuer_thread+0x310/0x310 [ 2186.126665] kthread+0xf5/0x130 [ 2186.129811] ? kthread_complete_and_exit+0x20/0x20 [ 2186.134604] ret_from_fork+0x31/0x40 [ 2186.138198] ? kthread_complete_and_exit+0x20/0x20 [ 2186.143001] ret_from_fork_asm+0x11/0x20 [ 2186.146968] [ 2186.149154] irq event stamp: 10292157 [ 2186.152831] hardirqs last enabled at (10292167): [] console_unlock+0x106/0x120 [ 2186.161697] hardirqs last disabled at (10292176): [] console_unlock+0xeb/0x120 [ 2186.170475] softirqs last enabled at (10292082): [] __do_softirq+0x2be/0x371 [ 2186.179185] softirqs last disabled at (10292077): [] irq_exit_rcu+0xb0/0xd0 [ 2186.187722] ---[ end trace 0000000000000000 ]--- [ 2186.192341] ------------[ cut here ]------------ [ 2186.196979] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2186.208642] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2186.217595] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2186.273073] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2186.282043] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2186.289540] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2186.295043] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2186.313808] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2186.319033] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2186.326183] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2186.333332] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000029c3 [ 2186.340474] R10: 0000000000119ad8 R11: ffffc90001957950 R12: ffff8881d12224a8 [ 2186.347641] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2186.354766] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2186.362851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2186.368588] CR2: 00007fb38ddfb000 CR3: 000000016696c005 CR4: 00000000003706f0 [ 2186.375732] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2186.382863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2186.389987] Call Trace: [ 2186.392449] [ 2186.394564] ? __warn+0x85/0x180 [ 2186.397814] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.402697] ? report_bug+0xfc/0x1d0 [ 2186.406273] ? prb_read_valid+0x1b/0x20 [ 2186.410132] ? handle_bug+0x44/0x70 [ 2186.413648] ? exc_invalid_op+0x17/0x60 [ 2186.417488] ? asm_exc_invalid_op+0x1a/0x20 [ 2186.421696] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.426571] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2186.431477] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2186.436731] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2186.441530] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2186.446282] lookup_fast+0x6d/0xf0 [ 2186.449686] walk_component+0x26/0x160 [ 2186.453439] link_path_walk+0x24d/0x380 [ 2186.457294] ? path_lookupat.isra.40+0x22/0x120 [ 2186.461837] path_lookupat.isra.40+0x6b/0x120 [ 2186.466198] filename_lookup+0xd4/0x190 [ 2186.470073] ? lock_acquire+0xcb/0x2b0 [ 2186.473823] ? __might_fault+0x45/0x80 [ 2186.477574] ? find_held_lock+0x2d/0x90 [ 2186.481431] ? __might_fault+0x45/0x80 [ 2186.485205] vfs_statx+0x89/0x150 [ 2186.488541] vfs_fstatat+0x5a/0xc0 [ 2186.491963] __do_sys_newstat+0x30/0x70 [ 2186.495821] ? lock_release+0xc5/0x260 [ 2186.499572] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2186.504468] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2186.509643] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2186.514784] do_syscall_64+0x38/0x80 [ 2186.518362] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2186.523430] RIP: 0033:0x7f48f0eeca79 [ 2186.527018] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2186.545790] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2186.553365] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2186.560490] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2186.567649] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2186.574773] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2186.581897] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2186.589044] [ 2186.591232] irq event stamp: 12785803 [ 2186.594907] hardirqs last enabled at (12785813): [] console_unlock+0x106/0x120 [ 2186.603773] hardirqs last disabled at (12785822): [] console_unlock+0xeb/0x120 [ 2186.612554] softirqs last enabled at (12785836): [] __do_softirq+0x2be/0x371 [ 2186.621261] softirqs last disabled at (12785831): [] irq_exit_rcu+0xb0/0xd0 [ 2186.629798] ---[ end trace 0000000000000000 ]--- [ 2186.634412] ------------[ cut here ]------------ [ 2186.639047] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2186.650692] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2186.659387] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2186.714882] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2186.724008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2186.731487] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2186.737008] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2186.742495] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2186.761241] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2186.766468] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2186.773600] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2186.780734] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002a06 [ 2186.787867] R10: 000000000011ade8 R11: ffffc9000030b828 R12: ffff8881c5351118 [ 2186.794998] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2186.802130] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2186.810217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2186.815980] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2186.823115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2186.830247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2186.837397] Call Trace: [ 2186.839851] [ 2186.841956] ? __warn+0x85/0x180 [ 2186.845206] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.850102] ? report_bug+0xfc/0x1d0 [ 2186.853683] ? prb_read_valid+0x1b/0x20 [ 2186.857524] ? handle_bug+0x44/0x70 [ 2186.861013] ? exc_invalid_op+0x17/0x60 [ 2186.864855] ? asm_exc_invalid_op+0x1a/0x20 [ 2186.869045] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.873938] ? __list_add_valid_or_report+0x7d/0xa0 [ 2186.878814] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2186.883708] ceph_add_cap+0x20b/0x620 [ceph] [ 2186.887994] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2186.892737] ceph_fill_trace+0x143/0xf30 [ceph] [ 2186.897276] ? __mutex_lock+0x97/0xce0 [ 2186.901027] ? __mutex_lock+0xaec/0xce0 [ 2186.904870] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2186.909409] ? lock_downgrade+0xfd/0x250 [ 2186.913341] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2186.917737] ceph_con_process_message+0x74/0x150 [libceph] [ 2186.923233] ? calc_signature+0xdb/0x100 [libceph] [ 2186.928064] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2186.933428] ? process_scheduled_works+0x22a/0x510 [ 2186.938221] ? process_scheduled_works+0x22a/0x510 [ 2186.943010] ceph_con_workfn+0x344/0x750 [libceph] [ 2186.947814] ? process_scheduled_works+0x22a/0x510 [ 2186.952604] process_scheduled_works+0x2a4/0x510 [ 2186.957246] worker_thread+0x188/0x340 [ 2186.960995] ? rescuer_thread+0x310/0x310 [ 2186.965007] kthread+0xf5/0x130 [ 2186.968153] ? kthread_complete_and_exit+0x20/0x20 [ 2186.972947] ret_from_fork+0x31/0x40 [ 2186.976524] ? kthread_complete_and_exit+0x20/0x20 [ 2186.981319] ret_from_fork_asm+0x11/0x20 [ 2186.985268] [ 2186.987453] irq event stamp: 10294091 [ 2186.991118] hardirqs last enabled at (10294101): [] console_unlock+0x106/0x120 [ 2186.999986] hardirqs last disabled at (10294110): [] console_unlock+0xeb/0x120 [ 2187.008766] softirqs last enabled at (10293858): [] __do_softirq+0x2be/0x371 [ 2187.017459] softirqs last disabled at (10293853): [] irq_exit_rcu+0xb0/0xd0 [ 2187.025978] ---[ end trace 0000000000000000 ]--- [ 2187.030650] ------------[ cut here ]------------ [ 2187.035269] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2187.046961] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2187.055661] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2187.111140] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2187.120282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2187.127778] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2187.133273] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2187.138777] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2187.157525] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2187.162765] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2187.169900] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2187.177033] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002a44 [ 2187.184183] R10: 000000000011be68 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2187.191331] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2187.198474] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2187.206577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2187.212339] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2187.219466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2187.226614] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2187.233757] Call Trace: [ 2187.236209] [ 2187.238324] ? __warn+0x85/0x180 [ 2187.241576] ? __list_add_valid_or_report+0x7d/0xa0 [ 2187.246471] ? report_bug+0xfc/0x1d0 [ 2187.250068] ? prb_read_valid+0x1b/0x20 [ 2187.253942] ? handle_bug+0x44/0x70 [ 2187.257453] ? exc_invalid_op+0x17/0x60 [ 2187.261308] ? asm_exc_invalid_op+0x1a/0x20 [ 2187.265535] ? __list_add_valid_or_report+0x7d/0xa0 [ 2187.270418] ? __list_add_valid_or_report+0x7d/0xa0 [ 2187.275313] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2187.280222] ceph_add_cap+0x20b/0x620 [ceph] [ 2187.284541] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2187.289282] ? lock_acquire+0xcb/0x2b0 [ 2187.293031] ? lock_is_held_type+0xaa/0x120 [ 2187.297236] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2187.301707] ? __mutex_lock+0x97/0xce0 [ 2187.305458] ? __mutex_lock+0xaec/0xce0 [ 2187.309315] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2187.313874] ? lock_downgrade+0xfd/0x250 [ 2187.317803] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2187.322199] ceph_con_process_message+0x74/0x150 [libceph] [ 2187.327697] ? calc_signature+0xdb/0x100 [libceph] [ 2187.332510] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2187.337859] ? process_scheduled_works+0x22a/0x510 [ 2187.342669] ? process_scheduled_works+0x22a/0x510 [ 2187.347456] ceph_con_workfn+0x344/0x750 [libceph] [ 2187.352277] ? process_scheduled_works+0x22a/0x510 [ 2187.357084] process_scheduled_works+0x2a4/0x510 [ 2187.361719] worker_thread+0x188/0x340 [ 2187.365476] ? rescuer_thread+0x310/0x310 [ 2187.369506] kthread+0xf5/0x130 [ 2187.372670] ? kthread_complete_and_exit+0x20/0x20 [ 2187.377463] ret_from_fork+0x31/0x40 [ 2187.381057] ? kthread_complete_and_exit+0x20/0x20 [ 2187.385868] ret_from_fork_asm+0x11/0x20 [ 2187.389801] [ 2187.391986] irq event stamp: 10295197 [ 2187.395668] hardirqs last enabled at (10295207): [] console_unlock+0x106/0x120 [ 2187.404536] hardirqs last disabled at (10295216): [] console_unlock+0xeb/0x120 [ 2187.413333] softirqs last enabled at (10295230): [] __do_softirq+0x2be/0x371 [ 2187.422043] softirqs last disabled at (10295225): [] irq_exit_rcu+0xb0/0xd0 [ 2187.430579] ---[ end trace 0000000000000000 ]--- [ 2187.435230] ------------[ cut here ]------------ [ 2187.439848] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2187.451507] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2187.460203] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2187.515697] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2187.524822] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2187.532293] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2187.537807] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2187.543291] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2187.562047] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2187.567291] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2187.574441] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2187.581590] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002a84 [ 2187.588733] R10: 000000000011d130 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2187.595864] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2187.602997] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2187.611092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2187.616856] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2187.623988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2187.631131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2187.638281] Call Trace: [ 2187.640750] [ 2187.642857] ? __warn+0x85/0x180 [ 2187.646089] ? __list_add_valid_or_report+0x7d/0xa0 [ 2187.650988] ? report_bug+0xfc/0x1d0 [ 2187.654583] ? prb_read_valid+0x1b/0x20 [ 2187.658440] ? handle_bug+0x44/0x70 [ 2187.661963] ? exc_invalid_op+0x17/0x60 [ 2187.665815] ? asm_exc_invalid_op+0x1a/0x20 [ 2187.670025] ? __list_add_valid_or_report+0x7d/0xa0 [ 2187.674907] ? __list_add_valid_or_report+0x7d/0xa0 [ 2187.679787] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2187.684678] ceph_add_cap+0x20b/0x620 [ceph] [ 2187.688991] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2187.693716] ceph_fill_trace+0x143/0xf30 [ceph] [ 2187.698256] ? __mutex_lock+0x97/0xce0 [ 2187.702008] ? __mutex_lock+0xaec/0xce0 [ 2187.705865] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2187.710407] ? lock_downgrade+0xfd/0x250 [ 2187.714371] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2187.718744] ceph_con_process_message+0x74/0x150 [libceph] [ 2187.724239] ? calc_signature+0xdb/0x100 [libceph] [ 2187.729060] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2187.734399] ? process_scheduled_works+0x22a/0x510 [ 2187.739208] ? process_scheduled_works+0x22a/0x510 [ 2187.744016] ceph_con_workfn+0x344/0x750 [libceph] [ 2187.748836] ? process_scheduled_works+0x22a/0x510 [ 2187.753626] process_scheduled_works+0x2a4/0x510 [ 2187.758278] worker_thread+0x188/0x340 [ 2187.762027] ? rescuer_thread+0x310/0x310 [ 2187.766055] kthread+0xf5/0x130 [ 2187.769219] ? kthread_complete_and_exit+0x20/0x20 [ 2187.774029] ret_from_fork+0x31/0x40 [ 2187.777625] ? kthread_complete_and_exit+0x20/0x20 [ 2187.782435] ret_from_fork_asm+0x11/0x20 [ 2187.786405] [ 2187.788597] irq event stamp: 10296241 [ 2187.792272] hardirqs last enabled at (10296251): [] console_unlock+0x106/0x120 [ 2187.801156] hardirqs last disabled at (10296260): [] console_unlock+0xeb/0x120 [ 2187.809962] softirqs last enabled at (10295968): [] __do_softirq+0x2be/0x371 [ 2187.818674] softirqs last disabled at (10295963): [] irq_exit_rcu+0xb0/0xd0 [ 2187.827191] ---[ end trace 0000000000000000 ]--- [ 2187.831828] ------------[ cut here ]------------ [ 2187.836447] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2187.848106] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2187.857062] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2187.912526] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2187.921483] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2187.928967] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2187.934453] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2187.953205] RSP: 0018:ffffc90001f7fb38 EFLAGS: 00010282 [ 2187.958435] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2187.965565] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2187.972700] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002ac2 [ 2187.979832] R10: 000000000011e240 R11: ffffc90001f7f9c8 R12: ffff8881c5351118 [ 2187.986970] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2187.994097] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2188.002186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2188.007930] CR2: 00007fe70ba1c0ec CR3: 0000000113532005 CR4: 00000000003706e0 [ 2188.015064] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2188.022194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2188.029330] Call Trace: [ 2188.031783] [ 2188.033889] ? __warn+0x85/0x180 [ 2188.037125] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.042012] ? report_bug+0xfc/0x1d0 [ 2188.045596] ? prb_read_valid+0x1b/0x20 [ 2188.049442] ? handle_bug+0x44/0x70 [ 2188.052939] ? exc_invalid_op+0x17/0x60 [ 2188.056780] ? asm_exc_invalid_op+0x1a/0x20 [ 2188.060981] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.065864] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.070747] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2188.075655] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2188.080379] lookup_fast+0x6d/0xf0 [ 2188.083782] walk_component+0x26/0x160 [ 2188.087540] link_path_walk+0x24d/0x380 [ 2188.091384] ? path_lookupat.isra.40+0x22/0x120 [ 2188.095922] path_lookupat.isra.40+0x6b/0x120 [ 2188.100286] filename_lookup+0xd4/0x190 [ 2188.104132] ? lock_acquire+0xcb/0x2b0 [ 2188.107885] ? __might_fault+0x45/0x80 [ 2188.111640] ? find_held_lock+0x2d/0x90 [ 2188.115477] ? __might_fault+0x45/0x80 [ 2188.119238] vfs_statx+0x89/0x150 [ 2188.122564] vfs_fstatat+0x5a/0xc0 [ 2188.125976] __do_sys_newstat+0x30/0x70 [ 2188.129816] ? lock_release+0xc5/0x260 [ 2188.133583] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2188.138461] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2188.143603] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2188.148747] do_syscall_64+0x38/0x80 [ 2188.152331] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2188.157380] RIP: 0033:0x7f48f0eeca79 [ 2188.160962] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2188.179707] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2188.187276] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2188.194406] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2188.201539] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2188.208670] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2188.215806] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2188.222948] [ 2188.225139] irq event stamp: 12907739 [ 2188.228804] hardirqs last enabled at (12907749): [] console_unlock+0x106/0x120 [ 2188.237672] hardirqs last disabled at (12907758): [] console_unlock+0xeb/0x120 [ 2188.246450] softirqs last enabled at (12907676): [] __do_softirq+0x2be/0x371 [ 2188.255146] softirqs last disabled at (12907777): [] irq_exit_rcu+0xb0/0xd0 [ 2188.263670] ---[ end trace 0000000000000000 ]--- [ 2188.268292] ------------[ cut here ]------------ [ 2188.272910] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2188.284555] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2188.293251] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2188.348727] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2188.357854] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2188.365332] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2188.370827] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2188.376312] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2188.395059] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2188.400284] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2188.407419] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2188.414549] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002b04 [ 2188.421683] R10: 000000000011f3e0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2188.428816] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2188.435950] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2188.444034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2188.449772] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2188.456906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2188.464037] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2188.471170] Call Trace: [ 2188.473617] [ 2188.475723] ? __warn+0x85/0x180 [ 2188.478954] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.483834] ? report_bug+0xfc/0x1d0 [ 2188.487413] ? prb_read_valid+0x1b/0x20 [ 2188.491255] ? handle_bug+0x44/0x70 [ 2188.494746] ? exc_invalid_op+0x17/0x60 [ 2188.498586] ? asm_exc_invalid_op+0x1a/0x20 [ 2188.502777] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.507661] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.512537] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2188.517428] ceph_add_cap+0x20b/0x620 [ceph] [ 2188.521714] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2188.526438] ? lock_acquire+0xcb/0x2b0 [ 2188.530196] ? lock_is_held_type+0xaa/0x120 [ 2188.534381] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2188.538833] ? __mutex_lock+0x97/0xce0 [ 2188.542585] ? __mutex_lock+0xaec/0xce0 [ 2188.546426] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2188.550968] ? lock_downgrade+0xfd/0x250 [ 2188.554897] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2188.559277] ceph_con_process_message+0x74/0x150 [libceph] [ 2188.564774] ? calc_signature+0xdb/0x100 [libceph] [ 2188.569588] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2188.574918] ? process_scheduled_works+0x22a/0x510 [ 2188.579707] ? process_scheduled_works+0x22a/0x510 [ 2188.584500] ceph_con_workfn+0x344/0x750 [libceph] [ 2188.589301] ? process_scheduled_works+0x22a/0x510 [ 2188.594093] process_scheduled_works+0x2a4/0x510 [ 2188.598719] worker_thread+0x188/0x340 [ 2188.602476] ? rescuer_thread+0x310/0x310 [ 2188.606485] kthread+0xf5/0x130 [ 2188.609632] ? kthread_complete_and_exit+0x20/0x20 [ 2188.614427] ret_from_fork+0x31/0x40 [ 2188.618006] ? kthread_complete_and_exit+0x20/0x20 [ 2188.622798] ret_from_fork_asm+0x11/0x20 [ 2188.626730] [ 2188.628916] irq event stamp: 10298251 [ 2188.632583] hardirqs last enabled at (10298261): [] console_unlock+0x106/0x120 [ 2188.641448] hardirqs last disabled at (10298270): [] console_unlock+0xeb/0x120 [ 2188.650228] softirqs last enabled at (10297818): [] __do_softirq+0x2be/0x371 [ 2188.658920] softirqs last disabled at (10297813): [] irq_exit_rcu+0xb0/0xd0 [ 2188.667441] ---[ end trace 0000000000000000 ]--- [ 2188.672060] ------------[ cut here ]------------ [ 2188.676679] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2188.688326] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2188.697285] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2188.752749] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2188.761706] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2188.769190] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2188.774678] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2188.793421] RSP: 0018:ffffc90001f7fac0 EFLAGS: 00010282 [ 2188.798650] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2188.805780] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2188.812915] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002b44 [ 2188.820046] R10: 00000000001205f0 R11: ffffc90001f7f950 R12: ffff8881c5351118 [ 2188.827181] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2188.834311] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2188.842399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2188.848145] CR2: 00007f9bd32b7028 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2188.855279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2188.862409] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2188.869544] Call Trace: [ 2188.871994] [ 2188.874105] ? __warn+0x85/0x180 [ 2188.877343] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.882226] ? report_bug+0xfc/0x1d0 [ 2188.885810] ? prb_read_valid+0x1b/0x20 [ 2188.889655] ? handle_bug+0x44/0x70 [ 2188.893155] ? exc_invalid_op+0x17/0x60 [ 2188.896994] ? asm_exc_invalid_op+0x1a/0x20 [ 2188.901194] ? __list_add_valid_or_report+0x7d/0xa0 [ 2188.906078] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2188.910974] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2188.916228] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2188.921035] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2188.925764] lookup_fast+0x6d/0xf0 [ 2188.929171] walk_component+0x26/0x160 [ 2188.932929] link_path_walk+0x24d/0x380 [ 2188.936772] ? path_lookupat.isra.40+0x22/0x120 [ 2188.941311] path_lookupat.isra.40+0x6b/0x120 [ 2188.945676] filename_lookup+0xd4/0x190 [ 2188.949521] ? lock_acquire+0xcb/0x2b0 [ 2188.953273] ? __might_fault+0x45/0x80 [ 2188.957032] ? find_held_lock+0x2d/0x90 [ 2188.960885] ? __might_fault+0x45/0x80 [ 2188.964653] vfs_statx+0x89/0x150 [ 2188.967977] vfs_fstatat+0x5a/0xc0 [ 2188.971391] __do_sys_newstat+0x30/0x70 [ 2188.975235] ? lock_release+0xc5/0x260 [ 2188.978993] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2188.983875] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2188.989018] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2188.994165] do_syscall_64+0x38/0x80 [ 2188.997744] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2189.002795] RIP: 0033:0x7f48f0eeca79 [ 2189.006376] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2189.025122] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2189.032690] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2189.039820] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2189.046952] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2189.054085] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2189.061218] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2189.068362] [ 2189.070553] irq event stamp: 12911233 [ 2189.074217] hardirqs last enabled at (12911243): [] console_unlock+0x106/0x120 [ 2189.083086] hardirqs last disabled at (12911252): [] console_unlock+0xeb/0x120 [ 2189.091875] softirqs last enabled at (12911266): [] __do_softirq+0x2be/0x371 [ 2189.100567] softirqs last disabled at (12911261): [] irq_exit_rcu+0xb0/0xd0 [ 2189.109092] ---[ end trace 0000000000000000 ]--- [ 2189.113716] ------------[ cut here ]------------ [ 2189.118330] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2189.130010] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2189.138698] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2189.194174] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2189.203318] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2189.210814] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2189.216309] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2189.221814] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2189.240560] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2189.245802] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2189.252936] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2189.260068] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002b87 [ 2189.267210] R10: 0000000000121b28 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2189.274360] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2189.281510] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2189.289614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2189.295375] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2189.302519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2189.309669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2189.316819] Call Trace: [ 2189.319271] [ 2189.321387] ? __warn+0x85/0x180 [ 2189.324638] ? __list_add_valid_or_report+0x7d/0xa0 [ 2189.329533] ? report_bug+0xfc/0x1d0 [ 2189.333129] ? prb_read_valid+0x1b/0x20 [ 2189.336995] ? handle_bug+0x44/0x70 [ 2189.340505] ? exc_invalid_op+0x17/0x60 [ 2189.344362] ? asm_exc_invalid_op+0x1a/0x20 [ 2189.348590] ? __list_add_valid_or_report+0x7d/0xa0 [ 2189.353472] ? __list_add_valid_or_report+0x7d/0xa0 [ 2189.358369] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2189.363292] ceph_add_cap+0x20b/0x620 [ceph] [ 2189.367614] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2189.372358] ceph_fill_trace+0x143/0xf30 [ceph] [ 2189.376898] ? __mutex_lock+0x97/0xce0 [ 2189.380651] ? __mutex_lock+0xaec/0xce0 [ 2189.384507] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2189.389065] ? lock_downgrade+0xfd/0x250 [ 2189.393033] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2189.397429] ceph_con_process_message+0x74/0x150 [libceph] [ 2189.402924] ? calc_signature+0xdb/0x100 [libceph] [ 2189.407729] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2189.413068] ? process_scheduled_works+0x22a/0x510 [ 2189.417859] ? process_scheduled_works+0x22a/0x510 [ 2189.422649] ceph_con_workfn+0x344/0x750 [libceph] [ 2189.427471] ? process_scheduled_works+0x22a/0x510 [ 2189.432279] process_scheduled_works+0x2a4/0x510 [ 2189.436922] worker_thread+0x188/0x340 [ 2189.440695] ? rescuer_thread+0x310/0x310 [ 2189.444706] kthread+0xf5/0x130 [ 2189.447854] ? kthread_complete_and_exit+0x20/0x20 [ 2189.452646] ret_from_fork+0x31/0x40 [ 2189.456241] ? kthread_complete_and_exit+0x20/0x20 [ 2189.461054] ret_from_fork_asm+0x11/0x20 [ 2189.465019] [ 2189.467205] irq event stamp: 10300195 [ 2189.470879] hardirqs last enabled at (10300205): [] console_unlock+0x106/0x120 [ 2189.479747] hardirqs last disabled at (10300214): [] console_unlock+0xeb/0x120 [ 2189.488526] softirqs last enabled at (10300172): [] __do_softirq+0x2be/0x371 [ 2189.497237] softirqs last disabled at (10300167): [] irq_exit_rcu+0xb0/0xd0 [ 2189.505775] ---[ end trace 0000000000000000 ]--- [ 2189.510392] ------------[ cut here ]------------ [ 2189.515038] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2189.526691] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2189.535652] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2189.591182] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2189.600140] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2189.607619] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2189.613108] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2189.631861] RSP: 0018:ffffc90001ff7b38 EFLAGS: 00010282 [ 2189.637085] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2189.644220] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2189.651351] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002bc5 [ 2189.658486] R10: 0000000000122ce0 R11: ffffc90001ff79c8 R12: ffff8881c25d24a8 [ 2189.665617] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2189.672750] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2189.680836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2189.686583] CR2: 00007fb38dec0000 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2189.693716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2189.700849] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2189.707981] Call Trace: [ 2189.710433] [ 2189.712540] ? __warn+0x85/0x180 [ 2189.715775] ? __list_add_valid_or_report+0x7d/0xa0 [ 2189.720653] ? report_bug+0xfc/0x1d0 [ 2189.724234] ? prb_read_valid+0x1b/0x20 [ 2189.728082] ? handle_bug+0x44/0x70 [ 2189.731573] ? exc_invalid_op+0x17/0x60 [ 2189.735411] ? asm_exc_invalid_op+0x1a/0x20 [ 2189.739608] ? __list_add_valid_or_report+0x7d/0xa0 [ 2189.744488] ? __list_add_valid_or_report+0x7d/0xa0 [ 2189.749366] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2189.754267] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2189.758994] lookup_fast+0x6d/0xf0 [ 2189.762403] walk_component+0x26/0x160 [ 2189.766157] link_path_walk+0x24d/0x380 [ 2189.770001] ? path_lookupat.isra.40+0x22/0x120 [ 2189.774538] path_lookupat.isra.40+0x6b/0x120 [ 2189.778906] filename_lookup+0xd4/0x190 [ 2189.782759] ? lock_acquire+0xcb/0x2b0 [ 2189.786511] ? __might_fault+0x45/0x80 [ 2189.790268] ? find_held_lock+0x2d/0x90 [ 2189.794112] ? __might_fault+0x45/0x80 [ 2189.797874] vfs_statx+0x89/0x150 [ 2189.801199] vfs_fstatat+0x5a/0xc0 [ 2189.804611] __do_sys_newstat+0x30/0x70 [ 2189.808454] ? lock_release+0xc5/0x260 [ 2189.812214] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2189.817097] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2189.822238] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2189.827379] do_syscall_64+0x38/0x80 [ 2189.830965] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2189.836016] RIP: 0033:0x7f48f0eeca79 [ 2189.839595] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2189.858345] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2189.865916] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2189.873052] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2189.880183] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2189.887316] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2189.894448] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2189.901594] [ 2189.903782] irq event stamp: 12867311 [ 2189.907448] hardirqs last enabled at (12867321): [] console_unlock+0x106/0x120 [ 2189.916315] hardirqs last disabled at (12867330): [] console_unlock+0xeb/0x120 [ 2189.925095] softirqs last enabled at (12867344): [] __do_softirq+0x2be/0x371 [ 2189.933787] softirqs last disabled at (12867339): [] irq_exit_rcu+0xb0/0xd0 [ 2189.942306] ---[ end trace 0000000000000000 ]--- [ 2189.946926] ------------[ cut here ]------------ [ 2189.951543] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2189.963189] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2189.971889] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2190.027371] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2190.036494] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2190.043974] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2190.049469] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2190.054956] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2190.073700] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2190.078927] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2190.086060] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2190.093192] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002c07 [ 2190.100326] R10: 0000000000123f50 R11: ffffc9000030b828 R12: ffff88819f028750 [ 2190.107458] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2190.114591] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2190.122678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2190.128423] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2190.135555] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2190.142681] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2190.149812] Call Trace: [ 2190.152258] [ 2190.154364] ? __warn+0x85/0x180 [ 2190.157598] ? __list_add_valid_or_report+0x7d/0xa0 [ 2190.162477] ? report_bug+0xfc/0x1d0 [ 2190.166054] ? prb_read_valid+0x1b/0x20 [ 2190.169896] ? handle_bug+0x44/0x70 [ 2190.173387] ? exc_invalid_op+0x17/0x60 [ 2190.177228] ? asm_exc_invalid_op+0x1a/0x20 [ 2190.181419] ? __list_add_valid_or_report+0x7d/0xa0 [ 2190.186302] ? __list_add_valid_or_report+0x7d/0xa0 [ 2190.191181] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2190.196072] ceph_add_cap+0x20b/0x620 [ceph] [ 2190.200356] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2190.205070] ? lock_acquire+0xcb/0x2b0 [ 2190.208820] ? lock_is_held_type+0xaa/0x120 [ 2190.213005] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2190.217460] ? __mutex_lock+0x97/0xce0 [ 2190.221210] ? __mutex_lock+0xaec/0xce0 [ 2190.225051] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2190.229593] ? lock_downgrade+0xfd/0x250 [ 2190.233522] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2190.237903] ceph_con_process_message+0x74/0x150 [libceph] [ 2190.243398] ? calc_signature+0xdb/0x100 [libceph] [ 2190.248202] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2190.253524] ? process_scheduled_works+0x22a/0x510 [ 2190.258318] ? process_scheduled_works+0x22a/0x510 [ 2190.263107] ceph_con_workfn+0x344/0x750 [libceph] [ 2190.267909] ? process_scheduled_works+0x22a/0x510 [ 2190.272701] process_scheduled_works+0x2a4/0x510 [ 2190.277324] worker_thread+0x188/0x340 [ 2190.281074] ? rescuer_thread+0x310/0x310 [ 2190.285090] kthread+0xf5/0x130 [ 2190.288240] ? kthread_complete_and_exit+0x20/0x20 [ 2190.293034] ret_from_fork+0x31/0x40 [ 2190.296613] ? kthread_complete_and_exit+0x20/0x20 [ 2190.301406] ret_from_fork_asm+0x11/0x20 [ 2190.305338] [ 2190.307523] irq event stamp: 10303383 [ 2190.311190] hardirqs last enabled at (10303393): [] console_unlock+0x106/0x120 [ 2190.320057] hardirqs last disabled at (10303402): [] console_unlock+0xeb/0x120 [ 2190.328835] softirqs last enabled at (10303200): [] __do_softirq+0x2be/0x371 [ 2190.337528] softirqs last disabled at (10303195): [] irq_exit_rcu+0xb0/0xd0 [ 2190.346047] ---[ end trace 0000000000000000 ]--- [ 2190.350669] ------------[ cut here ]------------ [ 2190.355287] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2190.366932] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2190.375889] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2190.431358] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2190.440307] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2190.447777] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2190.453265] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2190.472010] RSP: 0018:ffffc90001fdfb38 EFLAGS: 00010282 [ 2190.477238] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2190.484370] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2190.491502] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002c47 [ 2190.498636] R10: 0000000000125140 R11: ffffc90001fdf9c8 R12: ffff8881d1146920 [ 2190.505768] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2190.512900] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2190.520987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2190.526732] CR2: 00007f9bd32c7000 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2190.533856] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2190.540989] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2190.548113] Call Trace: [ 2190.550559] [ 2190.552665] ? __warn+0x85/0x180 [ 2190.555896] ? __list_add_valid_or_report+0x7d/0xa0 [ 2190.560778] ? report_bug+0xfc/0x1d0 [ 2190.564357] ? prb_read_valid+0x1b/0x20 [ 2190.568197] ? handle_bug+0x44/0x70 [ 2190.571688] ? exc_invalid_op+0x17/0x60 [ 2190.575527] ? asm_exc_invalid_op+0x1a/0x20 [ 2190.579719] ? __list_add_valid_or_report+0x7d/0xa0 [ 2190.584593] ? __list_add_valid_or_report+0x7d/0xa0 [ 2190.589473] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2190.594364] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2190.599080] lookup_fast+0x6d/0xf0 [ 2190.602482] walk_component+0x26/0x160 [ 2190.606236] link_path_walk+0x24d/0x380 [ 2190.610073] ? path_lookupat.isra.40+0x22/0x120 [ 2190.614608] path_lookupat.isra.40+0x6b/0x120 [ 2190.618968] filename_lookup+0xd4/0x190 [ 2190.622808] ? lock_acquire+0xcb/0x2b0 [ 2190.626560] ? __might_fault+0x45/0x80 [ 2190.630311] ? find_held_lock+0x2d/0x90 [ 2190.634150] ? __might_fault+0x45/0x80 [ 2190.637906] vfs_statx+0x89/0x150 [ 2190.641224] vfs_fstatat+0x5a/0xc0 [ 2190.644629] __do_sys_newstat+0x30/0x70 [ 2190.648467] ? lock_release+0xc5/0x260 [ 2190.652223] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2190.657098] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2190.662240] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2190.667379] do_syscall_64+0x38/0x80 [ 2190.670958] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2190.676012] RIP: 0033:0x7f48f0eeca79 [ 2190.679591] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2190.698339] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2190.705903] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2190.713036] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2190.720167] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2190.727291] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2190.734416] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2190.741546] [ 2190.743732] irq event stamp: 12810999 [ 2190.747391] hardirqs last enabled at (12811009): [] console_unlock+0x106/0x120 [ 2190.756255] hardirqs last disabled at (12811018): [] console_unlock+0xeb/0x120 [ 2190.765039] softirqs last enabled at (12811032): [] __do_softirq+0x2be/0x371 [ 2190.773729] softirqs last disabled at (12811027): [] irq_exit_rcu+0xb0/0xd0 [ 2190.782247] ---[ end trace 0000000000000000 ]--- [ 2190.786859] ------------[ cut here ]------------ [ 2190.791477] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2190.803115] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2190.811813] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2190.867293] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2190.876413] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2190.883893] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2190.889388] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2190.894874] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2190.913619] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2190.918847] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2190.925981] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2190.933113] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002c89 [ 2190.940246] R10: 00000000001262d0 R11: ffffc9000030b828 R12: ffff88819f02f2e8 [ 2190.947379] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2190.954512] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2190.962597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2190.968343] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2190.975476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2190.982608] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2190.989742] Call Trace: [ 2190.992194] [ 2190.994303] ? __warn+0x85/0x180 [ 2190.997534] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.002414] ? report_bug+0xfc/0x1d0 [ 2191.005991] ? prb_read_valid+0x1b/0x20 [ 2191.009832] ? handle_bug+0x44/0x70 [ 2191.013323] ? exc_invalid_op+0x17/0x60 [ 2191.017163] ? asm_exc_invalid_op+0x1a/0x20 [ 2191.021354] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.026230] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.031110] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2191.035998] ceph_add_cap+0x20b/0x620 [ceph] [ 2191.040285] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2191.045014] ceph_fill_trace+0x143/0xf30 [ceph] [ 2191.049551] ? __mutex_lock+0x97/0xce0 [ 2191.053304] ? __mutex_lock+0xaec/0xce0 [ 2191.057144] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2191.061684] ? lock_downgrade+0xfd/0x250 [ 2191.065617] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2191.069996] ceph_con_process_message+0x74/0x150 [libceph] [ 2191.075491] ? calc_signature+0xdb/0x100 [libceph] [ 2191.080296] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2191.085627] ? process_scheduled_works+0x22a/0x510 [ 2191.090419] ? process_scheduled_works+0x22a/0x510 [ 2191.095208] ceph_con_workfn+0x344/0x750 [libceph] [ 2191.100011] ? process_scheduled_works+0x22a/0x510 [ 2191.104801] process_scheduled_works+0x2a4/0x510 [ 2191.109428] worker_thread+0x188/0x340 [ 2191.113175] ? rescuer_thread+0x310/0x310 [ 2191.117188] kthread+0xf5/0x130 [ 2191.120332] ? kthread_complete_and_exit+0x20/0x20 [ 2191.125127] ret_from_fork+0x31/0x40 [ 2191.128708] ? kthread_complete_and_exit+0x20/0x20 [ 2191.133498] ret_from_fork_asm+0x11/0x20 [ 2191.137430] [ 2191.139615] irq event stamp: 10305325 [ 2191.143275] hardirqs last enabled at (10305335): [] console_unlock+0x106/0x120 [ 2191.152140] hardirqs last disabled at (10305344): [] console_unlock+0xeb/0x120 [ 2191.160920] softirqs last enabled at (10305282): [] __do_softirq+0x2be/0x371 [ 2191.169613] softirqs last disabled at (10305277): [] irq_exit_rcu+0xb0/0xd0 [ 2191.178131] ---[ end trace 0000000000000000 ]--- [ 2191.182743] ------------[ cut here ]------------ [ 2191.187362] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2191.199009] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2191.207965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2191.263424] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2191.272375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2191.279855] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2191.285341] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2191.304088] RSP: 0018:ffffc90001fdfac0 EFLAGS: 00010282 [ 2191.309312] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2191.316436] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2191.323560] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002cc7 [ 2191.330687] R10: 00000000001273e8 R11: ffffc90001fdf950 R12: ffff8881d1146920 [ 2191.337819] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2191.344951] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2191.353036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2191.358784] CR2: 00007f9bd32b7028 CR3: 0000000179402004 CR4: 00000000003706e0 [ 2191.365915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2191.373048] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2191.380172] Call Trace: [ 2191.382618] [ 2191.384724] ? __warn+0x85/0x180 [ 2191.387955] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.392838] ? report_bug+0xfc/0x1d0 [ 2191.396413] ? prb_read_valid+0x1b/0x20 [ 2191.400255] ? handle_bug+0x44/0x70 [ 2191.403746] ? exc_invalid_op+0x17/0x60 [ 2191.407577] ? asm_exc_invalid_op+0x1a/0x20 [ 2191.411767] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.416645] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2191.421535] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2191.426768] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2191.431569] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2191.436285] lookup_fast+0x6d/0xf0 [ 2191.439689] walk_component+0x26/0x160 [ 2191.443443] link_path_walk+0x24d/0x380 [ 2191.447280] ? path_lookupat.isra.40+0x22/0x120 [ 2191.451815] path_lookupat.isra.40+0x6b/0x120 [ 2191.456176] filename_lookup+0xd4/0x190 [ 2191.460014] ? lock_acquire+0xcb/0x2b0 [ 2191.463764] ? __might_fault+0x45/0x80 [ 2191.467518] ? find_held_lock+0x2d/0x90 [ 2191.471356] ? __might_fault+0x45/0x80 [ 2191.475113] vfs_statx+0x89/0x150 [ 2191.478433] vfs_fstatat+0x5a/0xc0 [ 2191.481836] __do_sys_newstat+0x30/0x70 [ 2191.485674] ? lock_release+0xc5/0x260 [ 2191.489430] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2191.494306] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2191.499447] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2191.504586] do_syscall_64+0x38/0x80 [ 2191.508165] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2191.513218] RIP: 0033:0x7f48f0eeca79 [ 2191.516796] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2191.535547] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2191.543108] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2191.550232] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2191.557358] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2191.564492] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2191.571623] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2191.578761] [ 2191.580948] irq event stamp: 12812987 [ 2191.584607] hardirqs last enabled at (12812997): [] console_unlock+0x106/0x120 [ 2191.593472] hardirqs last disabled at (12813006): [] console_unlock+0xeb/0x120 [ 2191.602252] softirqs last enabled at (12813020): [] __do_softirq+0x2be/0x371 [ 2191.610945] softirqs last disabled at (12813015): [] irq_exit_rcu+0xb0/0xd0 [ 2191.619463] ---[ end trace 0000000000000000 ]--- [ 2191.624075] ------------[ cut here ]------------ [ 2191.628693] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2191.640342] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2191.649040] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2191.704523] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2191.713646] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2191.721118] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2191.726621] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2191.732106] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2191.750854] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2191.756079] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2191.763213] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2191.770346] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002d0a [ 2191.777478] R10: 0000000000128608 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2191.784611] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2191.791743] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2191.799828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2191.805565] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2191.812690] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2191.819815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2191.826948] Call Trace: [ 2191.829402] [ 2191.831509] ? __warn+0x85/0x180 [ 2191.834740] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.839620] ? report_bug+0xfc/0x1d0 [ 2191.843199] ? prb_read_valid+0x1b/0x20 [ 2191.847039] ? handle_bug+0x44/0x70 [ 2191.850533] ? exc_invalid_op+0x17/0x60 [ 2191.854371] ? asm_exc_invalid_op+0x1a/0x20 [ 2191.858561] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.863436] ? __list_add_valid_or_report+0x7d/0xa0 [ 2191.868315] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2191.873207] ceph_add_cap+0x20b/0x620 [ceph] [ 2191.877490] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2191.882205] ? lock_acquire+0xcb/0x2b0 [ 2191.885955] ? lock_is_held_type+0xaa/0x120 [ 2191.890140] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2191.894593] ? __mutex_lock+0x97/0xce0 [ 2191.898348] ? __mutex_lock+0xaec/0xce0 [ 2191.902186] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2191.906727] ? lock_downgrade+0xfd/0x250 [ 2191.910659] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2191.915036] ceph_con_process_message+0x74/0x150 [libceph] [ 2191.920533] ? calc_signature+0xdb/0x100 [libceph] [ 2191.925336] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2191.930660] ? process_scheduled_works+0x22a/0x510 [ 2191.935451] ? process_scheduled_works+0x22a/0x510 [ 2191.940240] ceph_con_workfn+0x344/0x750 [libceph] [ 2191.945045] ? process_scheduled_works+0x22a/0x510 [ 2191.949835] process_scheduled_works+0x2a4/0x510 [ 2191.954465] worker_thread+0x188/0x340 [ 2191.958217] ? rescuer_thread+0x310/0x310 [ 2191.962229] kthread+0xf5/0x130 [ 2191.965380] ? kthread_complete_and_exit+0x20/0x20 [ 2191.970183] ret_from_fork+0x31/0x40 [ 2191.973764] ? kthread_complete_and_exit+0x20/0x20 [ 2191.978557] ret_from_fork_asm+0x11/0x20 [ 2191.982489] [ 2191.984675] irq event stamp: 10307333 [ 2191.988332] hardirqs last enabled at (10307343): [] console_unlock+0x106/0x120 [ 2191.997205] hardirqs last disabled at (10307352): [] console_unlock+0xeb/0x120 [ 2192.005996] softirqs last enabled at (10307290): [] __do_softirq+0x2be/0x371 [ 2192.014702] softirqs last disabled at (10307281): [] irq_exit_rcu+0xb0/0xd0 [ 2192.023227] ---[ end trace 0000000000000000 ]--- [ 2192.027846] ------------[ cut here ]------------ [ 2192.032466] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2192.044120] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2192.053078] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2192.108562] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2192.117515] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2192.124993] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2192.130480] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2192.149226] RSP: 0018:ffffc90001fefb98 EFLAGS: 00010282 [ 2192.154452] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2192.161583] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2192.168718] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002d4a [ 2192.175859] R10: 0000000000129768 R11: ffffc90001fefa28 R12: ffff88819f02f2e8 [ 2192.182992] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2192.190126] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2192.198211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2192.203957] CR2: 00007fb38deea030 CR3: 0000000108c72003 CR4: 00000000003706f0 [ 2192.211088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2192.218223] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2192.225355] Call Trace: [ 2192.227810] [ 2192.229915] ? __warn+0x85/0x180 [ 2192.233151] ? __list_add_valid_or_report+0x7d/0xa0 [ 2192.238036] ? report_bug+0xfc/0x1d0 [ 2192.241627] ? prb_read_valid+0x1b/0x20 [ 2192.245474] ? handle_bug+0x44/0x70 [ 2192.248964] ? exc_invalid_op+0x17/0x60 [ 2192.252804] ? asm_exc_invalid_op+0x1a/0x20 [ 2192.256999] ? __list_add_valid_or_report+0x7d/0xa0 [ 2192.261878] ? __list_add_valid_or_report+0x7d/0xa0 [ 2192.266761] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2192.271667] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2192.276394] lookup_fast+0x6d/0xf0 [ 2192.279803] walk_component+0x26/0x160 [ 2192.283558] path_lookupat.isra.40+0x58/0x120 [ 2192.287924] filename_lookup+0xd4/0x190 [ 2192.291769] ? lock_acquire+0xcb/0x2b0 [ 2192.295523] ? __might_fault+0x45/0x80 [ 2192.299277] ? find_held_lock+0x2d/0x90 [ 2192.303124] ? __might_fault+0x45/0x80 [ 2192.306892] vfs_statx+0x89/0x150 [ 2192.310240] vfs_fstatat+0x5a/0xc0 [ 2192.313650] __do_sys_newstat+0x30/0x70 [ 2192.317492] ? lock_release+0xc5/0x260 [ 2192.321252] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2192.326132] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2192.331274] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2192.336413] do_syscall_64+0x38/0x80 [ 2192.339995] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2192.345053] RIP: 0033:0x7f48f0eeca79 [ 2192.348636] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2192.367390] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2192.374965] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2192.382094] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2192.389229] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2192.396360] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2192.403492] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2192.410638] [ 2192.412827] irq event stamp: 12802691 [ 2192.416491] hardirqs last enabled at (12802701): [] console_unlock+0x106/0x120 [ 2192.425360] hardirqs last disabled at (12802710): [] console_unlock+0xeb/0x120 [ 2192.434139] softirqs last enabled at (12802628): [] __do_softirq+0x2be/0x371 [ 2192.442836] softirqs last disabled at (12802729): [] irq_exit_rcu+0xb0/0xd0 [ 2192.451358] ---[ end trace 0000000000000000 ]--- [ 2192.455981] ------------[ cut here ]------------ [ 2192.460598] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2192.472246] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2192.480940] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2192.536431] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2192.545559] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2192.553038] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2192.558551] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2192.564037] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2192.582783] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2192.588008] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2192.595141] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2192.602274] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002d8a [ 2192.609406] R10: 000000000012a8a0 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2192.616539] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2192.623672] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2192.631758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2192.637503] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2192.644628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2192.651760] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2192.658893] Call Trace: [ 2192.661340] [ 2192.663445] ? __warn+0x85/0x180 [ 2192.666677] ? __list_add_valid_or_report+0x7d/0xa0 [ 2192.671557] ? report_bug+0xfc/0x1d0 [ 2192.675135] ? prb_read_valid+0x1b/0x20 [ 2192.678978] ? handle_bug+0x44/0x70 [ 2192.682467] ? exc_invalid_op+0x17/0x60 [ 2192.686307] ? asm_exc_invalid_op+0x1a/0x20 [ 2192.690498] ? __list_add_valid_or_report+0x7d/0xa0 [ 2192.695373] ? __list_add_valid_or_report+0x7d/0xa0 [ 2192.700252] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2192.705143] ceph_add_cap+0x20b/0x620 [ceph] [ 2192.709427] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2192.714148] ceph_fill_trace+0x143/0xf30 [ceph] [ 2192.718688] ? __mutex_lock+0x97/0xce0 [ 2192.722440] ? __mutex_lock+0xaec/0xce0 [ 2192.726281] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2192.730820] ? lock_downgrade+0xfd/0x250 [ 2192.734752] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2192.739131] ceph_con_process_message+0x74/0x150 [libceph] [ 2192.744628] ? calc_signature+0xdb/0x100 [libceph] [ 2192.749441] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2192.754772] ? process_scheduled_works+0x22a/0x510 [ 2192.759572] ? process_scheduled_works+0x22a/0x510 [ 2192.764360] ceph_con_workfn+0x344/0x750 [libceph] [ 2192.769167] ? process_scheduled_works+0x22a/0x510 [ 2192.773954] process_scheduled_works+0x2a4/0x510 [ 2192.778582] worker_thread+0x188/0x340 [ 2192.782338] ? rescuer_thread+0x310/0x310 [ 2192.786348] kthread+0xf5/0x130 [ 2192.789496] ? kthread_complete_and_exit+0x20/0x20 [ 2192.794291] ret_from_fork+0x31/0x40 [ 2192.797876] ? kthread_complete_and_exit+0x20/0x20 [ 2192.802669] ret_from_fork_asm+0x11/0x20 [ 2192.806601] [ 2192.808786] irq event stamp: 10309267 [ 2192.812453] hardirqs last enabled at (10309277): [] console_unlock+0x106/0x120 [ 2192.821318] hardirqs last disabled at (10309286): [] console_unlock+0xeb/0x120 [ 2192.830098] softirqs last enabled at (10309224): [] __do_softirq+0x2be/0x371 [ 2192.838792] softirqs last disabled at (10309219): [] irq_exit_rcu+0xb0/0xd0 [ 2192.847310] ---[ end trace 0000000000000000 ]--- [ 2192.851930] ------------[ cut here ]------------ [ 2192.856552] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2192.868204] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2192.877166] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2192.932702] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2192.941667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2192.949151] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2192.954644] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2192.973391] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2192.978617] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2192.985748] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2192.992882] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002dc8 [ 2193.000015] R10: 000000000012ba38 R11: ffffc90001ff7950 R12: ffff8881c25d24a8 [ 2193.007149] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2193.014281] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2193.022366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2193.028112] CR2: 00007fb38de00010 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2193.035246] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2193.042379] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2193.049512] Call Trace: [ 2193.051963] [ 2193.054070] ? __warn+0x85/0x180 [ 2193.057305] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.062194] ? report_bug+0xfc/0x1d0 [ 2193.065779] ? prb_read_valid+0x1b/0x20 [ 2193.069624] ? handle_bug+0x44/0x70 [ 2193.073121] ? exc_invalid_op+0x17/0x60 [ 2193.076960] ? asm_exc_invalid_op+0x1a/0x20 [ 2193.081156] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.086044] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2193.090945] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2193.096198] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2193.101013] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2193.105748] lookup_fast+0x6d/0xf0 [ 2193.109160] walk_component+0x26/0x160 [ 2193.112926] link_path_walk+0x24d/0x380 [ 2193.116770] ? path_lookupat.isra.40+0x22/0x120 [ 2193.121318] path_lookupat.isra.40+0x6b/0x120 [ 2193.125681] filename_lookup+0xd4/0x190 [ 2193.129536] ? lock_acquire+0xcb/0x2b0 [ 2193.133296] ? __might_fault+0x45/0x80 [ 2193.137062] ? find_held_lock+0x2d/0x90 [ 2193.140906] ? __might_fault+0x45/0x80 [ 2193.144677] vfs_statx+0x89/0x150 [ 2193.148013] vfs_fstatat+0x5a/0xc0 [ 2193.151432] __do_sys_newstat+0x30/0x70 [ 2193.155275] ? lock_release+0xc5/0x260 [ 2193.159035] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2193.163915] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2193.169066] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2193.174214] do_syscall_64+0x38/0x80 [ 2193.177800] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2193.182859] RIP: 0033:0x7f48f0eeca79 [ 2193.186440] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2193.205187] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2193.212759] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2193.219893] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2193.227025] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2193.234159] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2193.241291] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2193.248445] [ 2193.250634] irq event stamp: 12875851 [ 2193.254300] hardirqs last enabled at (12875867): [] console_unlock+0x106/0x120 [ 2193.263167] hardirqs last disabled at (12875882): [] console_unlock+0xeb/0x120 [ 2193.271957] softirqs last enabled at (12875900): [] __do_softirq+0x2be/0x371 [ 2193.280655] softirqs last disabled at (12875911): [] irq_exit_rcu+0xb0/0xd0 [ 2193.289174] ---[ end trace 0000000000000000 ]--- [ 2193.293796] ------------[ cut here ]------------ [ 2193.298428] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2193.310084] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2193.319047] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2193.374519] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2193.383468] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2193.390946] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2193.396432] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2193.415179] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2193.420404] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2193.427539] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2193.434671] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002e0b [ 2193.441806] R10: 000000000012cfa0 R11: ffffc90001fd79c8 R12: ffff8881d10b0750 [ 2193.448935] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2193.456068] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2193.464154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2193.469900] CR2: 000056400e01ab90 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2193.477034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2193.484166] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2193.491299] Call Trace: [ 2193.493756] [ 2193.495859] ? __warn+0x85/0x180 [ 2193.499093] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.503971] ? report_bug+0xfc/0x1d0 [ 2193.507553] ? prb_read_valid+0x1b/0x20 [ 2193.511393] ? handle_bug+0x44/0x70 [ 2193.514881] ? exc_invalid_op+0x17/0x60 [ 2193.518724] ? asm_exc_invalid_op+0x1a/0x20 [ 2193.522915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.527797] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.532675] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2193.537568] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2193.542294] lookup_fast+0x6d/0xf0 [ 2193.545703] walk_component+0x26/0x160 [ 2193.549458] link_path_walk+0x24d/0x380 [ 2193.553295] ? path_lookupat.isra.40+0x22/0x120 [ 2193.557830] path_lookupat.isra.40+0x6b/0x120 [ 2193.562188] filename_lookup+0xd4/0x190 [ 2193.566032] ? lock_acquire+0xcb/0x2b0 [ 2193.569790] ? __might_fault+0x45/0x80 [ 2193.573542] ? find_held_lock+0x2d/0x90 [ 2193.577380] ? __might_fault+0x45/0x80 [ 2193.581137] vfs_statx+0x89/0x150 [ 2193.584456] vfs_fstatat+0x5a/0xc0 [ 2193.587859] __do_sys_newstat+0x30/0x70 [ 2193.591697] ? lock_release+0xc5/0x260 [ 2193.595453] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2193.600327] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2193.605469] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2193.610607] do_syscall_64+0x38/0x80 [ 2193.614188] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2193.619241] RIP: 0033:0x7f48f0eeca79 [ 2193.622821] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2193.641569] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2193.649132] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2193.656264] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2193.663397] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2193.670528] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2193.677662] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2193.684804] [ 2193.686997] irq event stamp: 12936481 [ 2193.690662] hardirqs last enabled at (12936491): [] console_unlock+0x106/0x120 [ 2193.699527] hardirqs last disabled at (12936500): [] console_unlock+0xeb/0x120 [ 2193.708308] softirqs last enabled at (12936514): [] __do_softirq+0x2be/0x371 [ 2193.717000] softirqs last disabled at (12936509): [] irq_exit_rcu+0xb0/0xd0 [ 2193.725521] ---[ end trace 0000000000000000 ]--- [ 2193.730140] ------------[ cut here ]------------ [ 2193.734779] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2193.746443] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2193.755138] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2193.810701] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2193.819848] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2193.827331] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2193.832853] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2193.838338] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2193.857102] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2193.862347] RAX: 0000000000000000 RBX: ffff8881d11409c8 RCX: 0000000000000000 [ 2193.869496] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2193.876647] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002e4d [ 2193.883797] R10: 000000000012e2c8 R11: ffffc9000030b828 R12: ffff8881d1141118 [ 2193.890930] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2193.898074] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2193.906174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2193.911937] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2193.919079] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2193.926228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2193.933380] Call Trace: [ 2193.935850] [ 2193.937958] ? __warn+0x85/0x180 [ 2193.941217] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.946106] ? report_bug+0xfc/0x1d0 [ 2193.949710] ? prb_read_valid+0x1b/0x20 [ 2193.953591] ? handle_bug+0x44/0x70 [ 2193.957088] ? exc_invalid_op+0x17/0x60 [ 2193.960950] ? asm_exc_invalid_op+0x1a/0x20 [ 2193.965164] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.970069] ? __list_add_valid_or_report+0x7d/0xa0 [ 2193.974965] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2193.979865] ceph_add_cap+0x20b/0x620 [ceph] [ 2193.984189] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2193.988921] ? lock_acquire+0xcb/0x2b0 [ 2193.992700] ? lock_is_held_type+0xaa/0x120 [ 2193.996896] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2194.001355] ? __mutex_lock+0x97/0xce0 [ 2194.005124] ? __mutex_lock+0xaec/0xce0 [ 2194.008983] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2194.013539] ? lock_downgrade+0xfd/0x250 [ 2194.017509] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2194.021902] ceph_con_process_message+0x74/0x150 [libceph] [ 2194.027408] ? calc_signature+0xdb/0x100 [libceph] [ 2194.032266] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2194.037622] ? process_scheduled_works+0x22a/0x510 [ 2194.042439] ? process_scheduled_works+0x22a/0x510 [ 2194.047236] ceph_con_workfn+0x344/0x750 [libceph] [ 2194.052099] ? process_scheduled_works+0x22a/0x510 [ 2194.056892] process_scheduled_works+0x2a4/0x510 [ 2194.061538] worker_thread+0x188/0x340 [ 2194.065301] ? rescuer_thread+0x310/0x310 [ 2194.069337] kthread+0xf5/0x130 [ 2194.072500] ? kthread_complete_and_exit+0x20/0x20 [ 2194.077332] ret_from_fork+0x31/0x40 [ 2194.080917] ? kthread_complete_and_exit+0x20/0x20 [ 2194.085709] ret_from_fork_asm+0x11/0x20 [ 2194.089684] [ 2194.091880] irq event stamp: 10312271 [ 2194.095544] hardirqs last enabled at (10312281): [] console_unlock+0x106/0x120 [ 2194.104427] hardirqs last disabled at (10312290): [] console_unlock+0xeb/0x120 [ 2194.113224] softirqs last enabled at (10312228): [] __do_softirq+0x2be/0x371 [ 2194.121934] softirqs last disabled at (10312223): [] irq_exit_rcu+0xb0/0xd0 [ 2194.130455] ---[ end trace 0000000000000000 ]--- [ 2194.135091] ------------[ cut here ]------------ [ 2194.139725] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2194.151409] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2194.160364] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2194.215900] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2194.224854] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2194.232333] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2194.237840] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2194.256591] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2194.261836] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2194.268967] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2194.276103] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002e8d [ 2194.283251] R10: 000000000012f4d0 R11: ffffc90001fd7950 R12: ffff8881d10b0750 [ 2194.290400] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2194.297550] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2194.305653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2194.311417] CR2: 00007fb38defc000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2194.318567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2194.325719] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2194.332867] Call Trace: [ 2194.335321] [ 2194.337426] ? __warn+0x85/0x180 [ 2194.340678] ? __list_add_valid_or_report+0x7d/0xa0 [ 2194.345557] ? report_bug+0xfc/0x1d0 [ 2194.349153] ? prb_read_valid+0x1b/0x20 [ 2194.353015] ? handle_bug+0x44/0x70 [ 2194.356512] ? exc_invalid_op+0x17/0x60 [ 2194.360370] ? asm_exc_invalid_op+0x1a/0x20 [ 2194.364596] ? __list_add_valid_or_report+0x7d/0xa0 [ 2194.369544] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2194.374510] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2194.379781] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2194.384610] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2194.389357] lookup_fast+0x6d/0xf0 [ 2194.392785] walk_component+0x26/0x160 [ 2194.396564] link_path_walk+0x24d/0x380 [ 2194.400400] ? path_lookupat.isra.40+0x22/0x120 [ 2194.404955] path_lookupat.isra.40+0x6b/0x120 [ 2194.409320] filename_lookup+0xd4/0x190 [ 2194.413200] ? lock_acquire+0xcb/0x2b0 [ 2194.416955] ? __might_fault+0x45/0x80 [ 2194.420707] ? find_held_lock+0x2d/0x90 [ 2194.424563] ? __might_fault+0x45/0x80 [ 2194.428357] vfs_statx+0x89/0x150 [ 2194.431702] vfs_fstatat+0x5a/0xc0 [ 2194.435113] __do_sys_newstat+0x30/0x70 [ 2194.438967] ? lock_release+0xc5/0x260 [ 2194.442741] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2194.447624] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2194.452798] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2194.457941] do_syscall_64+0x38/0x80 [ 2194.461519] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2194.466588] RIP: 0033:0x7f48f0eeca79 [ 2194.470167] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2194.488932] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2194.496499] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2194.503648] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2194.510798] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2194.517931] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2194.525080] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2194.532259] [ 2194.534449] irq event stamp: 12938447 [ 2194.538131] hardirqs last enabled at (12938457): [] console_unlock+0x106/0x120 [ 2194.547015] hardirqs last disabled at (12938466): [] console_unlock+0xeb/0x120 [ 2194.555798] softirqs last enabled at (12938384): [] __do_softirq+0x2be/0x371 [ 2194.564489] softirqs last disabled at (12938379): [] irq_exit_rcu+0xb0/0xd0 [ 2194.573027] ---[ end trace 0000000000000000 ]--- [ 2194.577646] ------------[ cut here ]------------ [ 2194.582262] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2194.593925] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2194.602621] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2194.658115] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2194.667257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2194.674753] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2194.680276] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2194.685795] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2194.704543] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2194.709796] RAX: 0000000000000000 RBX: ffff8881cccb8000 RCX: 0000000000000000 [ 2194.716927] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2194.724078] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002ed0 [ 2194.731208] R10: 0000000000130678 R11: ffffc9000030b828 R12: ffff8881cccb8750 [ 2194.738351] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2194.745492] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2194.753597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2194.759361] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2194.766509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2194.773652] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2194.780798] Call Trace: [ 2194.783247] [ 2194.785369] ? __warn+0x85/0x180 [ 2194.788622] ? __list_add_valid_or_report+0x7d/0xa0 [ 2194.793517] ? report_bug+0xfc/0x1d0 [ 2194.797113] ? prb_read_valid+0x1b/0x20 [ 2194.800971] ? handle_bug+0x44/0x70 [ 2194.804461] ? exc_invalid_op+0x17/0x60 [ 2194.808302] ? asm_exc_invalid_op+0x1a/0x20 [ 2194.812526] ? __list_add_valid_or_report+0x7d/0xa0 [ 2194.817402] ? __list_add_valid_or_report+0x7d/0xa0 [ 2194.822298] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2194.827206] ceph_add_cap+0x20b/0x620 [ceph] [ 2194.831525] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2194.836245] ceph_fill_trace+0x143/0xf30 [ceph] [ 2194.840800] ? __mutex_lock+0x97/0xce0 [ 2194.844546] ? __mutex_lock+0xaec/0xce0 [ 2194.848404] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2194.852964] ? lock_downgrade+0xfd/0x250 [ 2194.856894] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2194.861289] ceph_con_process_message+0x74/0x150 [libceph] [ 2194.866800] ? calc_signature+0xdb/0x100 [libceph] [ 2194.871607] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2194.876946] ? process_scheduled_works+0x22a/0x510 [ 2194.881754] ? process_scheduled_works+0x22a/0x510 [ 2194.886545] ceph_con_workfn+0x344/0x750 [libceph] [ 2194.891366] ? process_scheduled_works+0x22a/0x510 [ 2194.896175] process_scheduled_works+0x2a4/0x510 [ 2194.900817] worker_thread+0x188/0x340 [ 2194.904566] ? rescuer_thread+0x310/0x310 [ 2194.908593] kthread+0xf5/0x130 [ 2194.911756] ? kthread_complete_and_exit+0x20/0x20 [ 2194.916570] ret_from_fork+0x31/0x40 [ 2194.920165] ? kthread_complete_and_exit+0x20/0x20 [ 2194.924976] ret_from_fork_asm+0x11/0x20 [ 2194.928909] [ 2194.931101] irq event stamp: 10314319 [ 2194.934793] hardirqs last enabled at (10314329): [] console_unlock+0x106/0x120 [ 2194.943658] hardirqs last disabled at (10314338): [] console_unlock+0xeb/0x120 [ 2194.952447] softirqs last enabled at (10314276): [] __do_softirq+0x2be/0x371 [ 2194.961158] softirqs last disabled at (10314271): [] irq_exit_rcu+0xb0/0xd0 [ 2194.969695] ---[ end trace 0000000000000000 ]--- [ 2194.974333] ------------[ cut here ]------------ [ 2194.978967] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2194.990616] WARNING: CPU: 4 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2194.999569] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2195.055034] CPU: 4 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2195.063992] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2195.071474] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2195.076954] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2195.095702] RSP: 0018:ffffc90001fefb20 EFLAGS: 00010282 [ 2195.100927] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2195.108060] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2195.115184] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002f0e [ 2195.122318] R10: 0000000000131788 R11: ffffc90001fef9b0 R12: ffff88819f02f2e8 [ 2195.129451] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2195.136583] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2195.144668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2195.150416] CR2: 00007f9bd32ec000 CR3: 0000000108c72006 CR4: 00000000003706e0 [ 2195.157548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2195.164682] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2195.171817] Call Trace: [ 2195.174268] [ 2195.176374] ? __warn+0x85/0x180 [ 2195.179606] ? __list_add_valid_or_report+0x7d/0xa0 [ 2195.184486] ? report_bug+0xfc/0x1d0 [ 2195.188067] ? prb_read_valid+0x1b/0x20 [ 2195.191906] ? handle_bug+0x44/0x70 [ 2195.195396] ? exc_invalid_op+0x17/0x60 [ 2195.199242] ? asm_exc_invalid_op+0x1a/0x20 [ 2195.203435] ? __list_add_valid_or_report+0x7d/0xa0 [ 2195.208311] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2195.213199] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2195.218434] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2195.223236] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2195.227952] lookup_fast+0x6d/0xf0 [ 2195.231356] walk_component+0x26/0x160 [ 2195.235110] path_lookupat.isra.40+0x58/0x120 [ 2195.239477] filename_lookup+0xd4/0x190 [ 2195.243320] ? lock_acquire+0xcb/0x2b0 [ 2195.247068] ? __might_fault+0x45/0x80 [ 2195.250824] ? find_held_lock+0x2d/0x90 [ 2195.254660] ? __might_fault+0x45/0x80 [ 2195.258416] vfs_statx+0x89/0x150 [ 2195.261735] vfs_fstatat+0x5a/0xc0 [ 2195.265140] __do_sys_newstat+0x30/0x70 [ 2195.268985] ? lock_release+0xc5/0x260 [ 2195.272743] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2195.277619] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2195.282757] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2195.287897] do_syscall_64+0x38/0x80 [ 2195.291477] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2195.296530] RIP: 0033:0x7f48f0eeca79 [ 2195.300109] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2195.318855] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2195.326422] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2195.333554] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2195.340686] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2195.347818] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2195.354944] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2195.362082] [ 2195.364269] irq event stamp: 12809973 [ 2195.367936] hardirqs last enabled at (12809983): [] console_unlock+0x106/0x120 [ 2195.376802] hardirqs last disabled at (12810002): [] console_unlock+0xeb/0x120 [ 2195.385584] softirqs last enabled at (12810016): [] __do_softirq+0x2be/0x371 [ 2195.394273] softirqs last disabled at (12810011): [] irq_exit_rcu+0xb0/0xd0 [ 2195.402794] ---[ end trace 0000000000000000 ]--- [ 2195.407415] ------------[ cut here ]------------ [ 2195.412032] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2195.423695] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2195.432391] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2195.487886] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2195.497011] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2195.504492] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2195.510012] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2195.515497] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2195.534260] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2195.539504] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2195.546655] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2195.553814] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002f4f [ 2195.560947] R10: 0000000000132a80 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2195.568094] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2195.575246] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2195.583350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2195.589113] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2195.596262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2195.603415] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2195.610563] Call Trace: [ 2195.613033] [ 2195.615138] ? __warn+0x85/0x180 [ 2195.618392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2195.623288] ? report_bug+0xfc/0x1d0 [ 2195.626885] ? prb_read_valid+0x1b/0x20 [ 2195.630745] ? handle_bug+0x44/0x70 [ 2195.634234] ? exc_invalid_op+0x17/0x60 [ 2195.638095] ? asm_exc_invalid_op+0x1a/0x20 [ 2195.642329] ? __list_add_valid_or_report+0x7d/0xa0 [ 2195.647226] ? __list_add_valid_or_report+0x7d/0xa0 [ 2195.652105] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2195.657024] ceph_add_cap+0x20b/0x620 [ceph] [ 2195.661336] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2195.666060] ? lock_acquire+0xcb/0x2b0 [ 2195.669821] ? lock_is_held_type+0xaa/0x120 [ 2195.674011] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2195.678470] ? __mutex_lock+0x97/0xce0 [ 2195.682240] ? __mutex_lock+0xaec/0xce0 [ 2195.686114] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2195.690665] ? lock_downgrade+0xfd/0x250 [ 2195.694630] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2195.699008] ceph_con_process_message+0x74/0x150 [libceph] [ 2195.704504] ? calc_signature+0xdb/0x100 [libceph] [ 2195.709356] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2195.714704] ? process_scheduled_works+0x22a/0x510 [ 2195.719520] ? process_scheduled_works+0x22a/0x510 [ 2195.724317] ceph_con_workfn+0x344/0x750 [libceph] [ 2195.729157] ? process_scheduled_works+0x22a/0x510 [ 2195.733945] process_scheduled_works+0x2a4/0x510 [ 2195.738590] worker_thread+0x188/0x340 [ 2195.742363] ? rescuer_thread+0x310/0x310 [ 2195.746392] kthread+0xf5/0x130 [ 2195.749538] ? kthread_complete_and_exit+0x20/0x20 [ 2195.754367] ret_from_fork+0x31/0x40 [ 2195.757944] ? kthread_complete_and_exit+0x20/0x20 [ 2195.762735] ret_from_fork_asm+0x11/0x20 [ 2195.766708] [ 2195.768899] irq event stamp: 10316361 [ 2195.772564] hardirqs last enabled at (10316371): [] console_unlock+0x106/0x120 [ 2195.781428] hardirqs last disabled at (10316380): [] console_unlock+0xeb/0x120 [ 2195.790218] softirqs last enabled at (10316298): [] __do_softirq+0x2be/0x371 [ 2195.798929] softirqs last disabled at (10316293): [] irq_exit_rcu+0xb0/0xd0 [ 2195.807447] ---[ end trace 0000000000000000 ]--- [ 2195.812102] ------------[ cut here ]------------ [ 2195.816739] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2195.828402] WARNING: CPU: 7 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2195.837357] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2195.892840] CPU: 7 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2195.901826] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2195.909300] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2195.914820] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2195.933568] RSP: 0018:ffffc90001fffb38 EFLAGS: 00010282 [ 2195.938822] RAX: 0000000000000000 RBX: ffff8881cccb8000 RCX: 0000000000000000 [ 2195.945952] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2195.953087] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002f8f [ 2195.960218] R10: 0000000000133bb0 R11: ffffc90001fff9c8 R12: ffff8881cccb8750 [ 2195.967368] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2195.974508] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2195.982603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2195.988359] CR2: 00007fb38de00010 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2195.995510] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2196.002660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2196.009821] Call Trace: [ 2196.012269] [ 2196.014386] ? __warn+0x85/0x180 [ 2196.017636] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.022533] ? report_bug+0xfc/0x1d0 [ 2196.026130] ? prb_read_valid+0x1b/0x20 [ 2196.029987] ? handle_bug+0x44/0x70 [ 2196.033479] ? exc_invalid_op+0x17/0x60 [ 2196.037336] ? asm_exc_invalid_op+0x1a/0x20 [ 2196.041560] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.046437] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.051333] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2196.056241] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2196.060974] lookup_fast+0x6d/0xf0 [ 2196.064376] walk_component+0x26/0x160 [ 2196.068147] link_path_walk+0x24d/0x380 [ 2196.072003] ? path_lookupat.isra.40+0x22/0x120 [ 2196.076536] path_lookupat.isra.40+0x6b/0x120 [ 2196.080914] filename_lookup+0xd4/0x190 [ 2196.084770] ? lock_acquire+0xcb/0x2b0 [ 2196.088520] ? __might_fault+0x45/0x80 [ 2196.092292] ? find_held_lock+0x2d/0x90 [ 2196.096148] ? __might_fault+0x45/0x80 [ 2196.099920] vfs_statx+0x89/0x150 [ 2196.103241] vfs_fstatat+0x5a/0xc0 [ 2196.106661] __do_sys_newstat+0x30/0x70 [ 2196.110519] ? lock_release+0xc5/0x260 [ 2196.114290] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2196.119184] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2196.124340] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2196.129500] do_syscall_64+0x38/0x80 [ 2196.133110] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2196.138174] RIP: 0033:0x7f48f0eeca79 [ 2196.141770] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2196.160533] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2196.168116] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2196.175257] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2196.182401] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2196.189541] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2196.196683] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2196.203834] [ 2196.206027] irq event stamp: 12851371 [ 2196.209691] hardirqs last enabled at (12851381): [] console_unlock+0x106/0x120 [ 2196.218575] hardirqs last disabled at (12851390): [] console_unlock+0xeb/0x120 [ 2196.227363] softirqs last enabled at (12851404): [] __do_softirq+0x2be/0x371 [ 2196.236075] softirqs last disabled at (12851399): [] irq_exit_rcu+0xb0/0xd0 [ 2196.244592] ---[ end trace 0000000000000000 ]--- [ 2196.249221] ------------[ cut here ]------------ [ 2196.253859] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2196.265523] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2196.274217] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2196.329732] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2196.338858] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2196.346333] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2196.351845] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2196.357332] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2196.376095] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2196.381321] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2196.388470] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2196.395622] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100002fd1 [ 2196.402770] R10: 0000000000134ef0 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2196.409914] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2196.417045] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2196.425132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2196.430895] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2196.438029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2196.445160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2196.452302] Call Trace: [ 2196.454763] [ 2196.456881] ? __warn+0x85/0x180 [ 2196.460123] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.465017] ? report_bug+0xfc/0x1d0 [ 2196.468597] ? prb_read_valid+0x1b/0x20 [ 2196.472455] ? handle_bug+0x44/0x70 [ 2196.475963] ? exc_invalid_op+0x17/0x60 [ 2196.479822] ? asm_exc_invalid_op+0x1a/0x20 [ 2196.484012] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.488896] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.493772] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2196.498680] ceph_add_cap+0x20b/0x620 [ceph] [ 2196.502983] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2196.507721] ceph_fill_trace+0x143/0xf30 [ceph] [ 2196.512260] ? __mutex_lock+0x97/0xce0 [ 2196.516031] ? __mutex_lock+0xaec/0xce0 [ 2196.519871] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2196.524412] ? lock_downgrade+0xfd/0x250 [ 2196.528376] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2196.532764] ceph_con_process_message+0x74/0x150 [libceph] [ 2196.538261] ? calc_signature+0xdb/0x100 [libceph] [ 2196.543066] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2196.548412] ? process_scheduled_works+0x22a/0x510 [ 2196.553221] ? process_scheduled_works+0x22a/0x510 [ 2196.558013] ceph_con_workfn+0x344/0x750 [libceph] [ 2196.562833] ? process_scheduled_works+0x22a/0x510 [ 2196.567623] process_scheduled_works+0x2a4/0x510 [ 2196.572283] worker_thread+0x188/0x340 [ 2196.576031] ? rescuer_thread+0x310/0x310 [ 2196.580043] kthread+0xf5/0x130 [ 2196.583189] ? kthread_complete_and_exit+0x20/0x20 [ 2196.588001] ret_from_fork+0x31/0x40 [ 2196.591579] ? kthread_complete_and_exit+0x20/0x20 [ 2196.596388] ret_from_fork_asm+0x11/0x20 [ 2196.600357] [ 2196.602542] irq event stamp: 10318387 [ 2196.606225] hardirqs last enabled at (10318397): [] console_unlock+0x106/0x120 [ 2196.615121] hardirqs last disabled at (10318406): [] console_unlock+0xeb/0x120 [ 2196.623913] softirqs last enabled at (10318344): [] __do_softirq+0x2be/0x371 [ 2196.632608] softirqs last disabled at (10318339): [] irq_exit_rcu+0xb0/0xd0 [ 2196.641144] ---[ end trace 0000000000000000 ]--- [ 2196.645782] ------------[ cut here ]------------ [ 2196.650430] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2196.662078] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2196.671040] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2196.726559] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2196.735511] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2196.742990] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2196.748480] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2196.767233] RSP: 0018:ffffc90001fffac0 EFLAGS: 00010282 [ 2196.772457] RAX: 0000000000000000 RBX: ffff8881cccb8000 RCX: 0000000000000000 [ 2196.779590] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2196.786723] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000300f [ 2196.793858] R10: 0000000000135fd8 R11: ffffc90001fff950 R12: ffff8881cccb8750 [ 2196.800990] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2196.808122] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2196.816208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2196.821955] CR2: 000056400e01ab90 CR3: 000000017eb7a003 CR4: 00000000003706e0 [ 2196.829088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2196.836224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2196.843362] Call Trace: [ 2196.845813] [ 2196.847921] ? __warn+0x85/0x180 [ 2196.851155] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.856033] ? report_bug+0xfc/0x1d0 [ 2196.859614] ? prb_read_valid+0x1b/0x20 [ 2196.863456] ? handle_bug+0x44/0x70 [ 2196.866954] ? exc_invalid_op+0x17/0x60 [ 2196.870792] ? asm_exc_invalid_op+0x1a/0x20 [ 2196.874987] ? __list_add_valid_or_report+0x7d/0xa0 [ 2196.879869] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2196.884767] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2196.890020] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2196.894838] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2196.899565] lookup_fast+0x6d/0xf0 [ 2196.902973] walk_component+0x26/0x160 [ 2196.906728] link_path_walk+0x24d/0x380 [ 2196.910565] ? path_lookupat.isra.40+0x22/0x120 [ 2196.915103] path_lookupat.isra.40+0x6b/0x120 [ 2196.919467] filename_lookup+0xd4/0x190 [ 2196.923314] ? lock_acquire+0xcb/0x2b0 [ 2196.927065] ? __might_fault+0x45/0x80 [ 2196.930820] ? find_held_lock+0x2d/0x90 [ 2196.934657] ? __might_fault+0x45/0x80 [ 2196.938417] vfs_statx+0x89/0x150 [ 2196.941744] vfs_fstatat+0x5a/0xc0 [ 2196.945155] __do_sys_newstat+0x30/0x70 [ 2196.948994] ? lock_release+0xc5/0x260 [ 2196.952751] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2196.957632] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2196.962774] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2196.967916] do_syscall_64+0x38/0x80 [ 2196.971501] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2196.976552] RIP: 0033:0x7f48f0eeca79 [ 2196.980132] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2196.998881] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2197.006457] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2197.013585] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2197.020720] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2197.027852] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2197.034985] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2197.042130] [ 2197.044319] irq event stamp: 12856197 [ 2197.047985] hardirqs last enabled at (12856207): [] console_unlock+0x106/0x120 [ 2197.056850] hardirqs last disabled at (12856216): [] console_unlock+0xeb/0x120 [ 2197.065633] softirqs last enabled at (12856230): [] __do_softirq+0x2be/0x371 [ 2197.074332] softirqs last disabled at (12856225): [] irq_exit_rcu+0xb0/0xd0 [ 2197.082850] ---[ end trace 0000000000000000 ]--- [ 2197.087471] ------------[ cut here ]------------ [ 2197.092090] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2197.103762] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2197.112716] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2197.168216] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2197.177193] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2197.184666] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2197.190171] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2197.208933] RSP: 0018:ffffc90001fc7b38 EFLAGS: 00010282 [ 2197.214180] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2197.221309] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2197.228450] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003052 [ 2197.235595] R10: 0000000000137188 R11: ffffc90001fc79c8 R12: ffff8881d0bec200 [ 2197.242741] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2197.249888] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2197.257971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2197.263715] CR2: 00007f9bd32ec000 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2197.270858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2197.277992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2197.285127] Call Trace: [ 2197.287594] [ 2197.289708] ? __warn+0x85/0x180 [ 2197.292959] ? __list_add_valid_or_report+0x7d/0xa0 [ 2197.297846] ? report_bug+0xfc/0x1d0 [ 2197.301425] ? prb_read_valid+0x1b/0x20 [ 2197.305275] ? handle_bug+0x44/0x70 [ 2197.308768] ? exc_invalid_op+0x17/0x60 [ 2197.312606] ? asm_exc_invalid_op+0x1a/0x20 [ 2197.316817] ? __list_add_valid_or_report+0x7d/0xa0 [ 2197.321697] ? __list_add_valid_or_report+0x7d/0xa0 [ 2197.326586] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2197.331493] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2197.336245] lookup_fast+0x6d/0xf0 [ 2197.339647] walk_component+0x26/0x160 [ 2197.343418] link_path_walk+0x24d/0x380 [ 2197.347274] ? path_lookupat.isra.40+0x22/0x120 [ 2197.351850] path_lookupat.isra.40+0x6b/0x120 [ 2197.356210] filename_lookup+0xd4/0x190 [ 2197.360076] ? lock_acquire+0xcb/0x2b0 [ 2197.363848] ? __might_fault+0x45/0x80 [ 2197.367598] ? find_held_lock+0x2d/0x90 [ 2197.371454] ? __might_fault+0x45/0x80 [ 2197.375227] vfs_statx+0x89/0x150 [ 2197.378582] vfs_fstatat+0x5a/0xc0 [ 2197.381985] __do_sys_newstat+0x30/0x70 [ 2197.385844] ? lock_release+0xc5/0x260 [ 2197.389594] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2197.394490] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2197.399648] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2197.404805] do_syscall_64+0x38/0x80 [ 2197.408399] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2197.413462] RIP: 0033:0x7f48f0eeca79 [ 2197.417059] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2197.435807] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2197.443388] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2197.450528] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2197.457672] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2197.464812] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2197.471954] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2197.479092] [ 2197.481280] irq event stamp: 12918629 [ 2197.484954] hardirqs last enabled at (12918639): [] console_unlock+0x106/0x120 [ 2197.493841] hardirqs last disabled at (12918658): [] console_unlock+0xeb/0x120 [ 2197.502618] softirqs last enabled at (12918672): [] __do_softirq+0x2be/0x371 [ 2197.511328] softirqs last disabled at (12918667): [] irq_exit_rcu+0xb0/0xd0 [ 2197.519873] ---[ end trace 0000000000000000 ]--- [ 2197.524493] ------------[ cut here ]------------ [ 2197.529130] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2197.540793] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2197.549489] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2197.604984] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2197.614109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2197.621596] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2197.627091] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2197.632577] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2197.651324] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2197.656549] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2197.663681] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2197.670806] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003094 [ 2197.677939] R10: 0000000000138618 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2197.685072] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2197.692204] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2197.700292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2197.706038] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2197.713170] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2197.720302] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2197.727434] Call Trace: [ 2197.729880] [ 2197.731987] ? __warn+0x85/0x180 [ 2197.735220] ? __list_add_valid_or_report+0x7d/0xa0 [ 2197.740100] ? report_bug+0xfc/0x1d0 [ 2197.743676] ? prb_read_valid+0x1b/0x20 [ 2197.747518] ? handle_bug+0x44/0x70 [ 2197.751010] ? exc_invalid_op+0x17/0x60 [ 2197.754849] ? asm_exc_invalid_op+0x1a/0x20 [ 2197.759041] ? __list_add_valid_or_report+0x7d/0xa0 [ 2197.763916] ? __list_add_valid_or_report+0x7d/0xa0 [ 2197.768794] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2197.773685] ceph_add_cap+0x20b/0x620 [ceph] [ 2197.777969] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2197.782683] ? lock_acquire+0xcb/0x2b0 [ 2197.786433] ? lock_is_held_type+0xaa/0x120 [ 2197.790620] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2197.795074] ? __mutex_lock+0x97/0xce0 [ 2197.798827] ? __mutex_lock+0xaec/0xce0 [ 2197.802665] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2197.807206] ? lock_downgrade+0xfd/0x250 [ 2197.811136] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2197.815516] ceph_con_process_message+0x74/0x150 [libceph] [ 2197.821012] ? calc_signature+0xdb/0x100 [libceph] [ 2197.825815] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2197.831146] ? process_scheduled_works+0x22a/0x510 [ 2197.835940] ? process_scheduled_works+0x22a/0x510 [ 2197.840729] ceph_con_workfn+0x344/0x750 [libceph] [ 2197.845531] ? process_scheduled_works+0x22a/0x510 [ 2197.850321] process_scheduled_works+0x2a4/0x510 [ 2197.854947] worker_thread+0x188/0x340 [ 2197.858697] ? rescuer_thread+0x310/0x310 [ 2197.862708] kthread+0xf5/0x130 [ 2197.865856] ? kthread_complete_and_exit+0x20/0x20 [ 2197.870647] ret_from_fork+0x31/0x40 [ 2197.874227] ? kthread_complete_and_exit+0x20/0x20 [ 2197.879019] ret_from_fork_asm+0x11/0x20 [ 2197.882951] [ 2197.885136] irq event stamp: 10321389 [ 2197.888795] hardirqs last enabled at (10321399): [] console_unlock+0x106/0x120 [ 2197.897661] hardirqs last disabled at (10321408): [] console_unlock+0xeb/0x120 [ 2197.906440] softirqs last enabled at (10321326): [] __do_softirq+0x2be/0x371 [ 2197.915132] softirqs last disabled at (10321321): [] irq_exit_rcu+0xb0/0xd0 [ 2197.923657] ---[ end trace 0000000000000000 ]--- [ 2197.928273] ------------[ cut here ]------------ [ 2197.932905] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2197.944556] WARNING: CPU: 0 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2197.953519] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2198.008989] CPU: 0 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2198.017949] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2198.025428] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2198.030912] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2198.049663] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2198.054885] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2198.062019] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2198.069151] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000030d4 [ 2198.076285] R10: 0000000000139810 R11: ffffc90001fc7950 R12: ffff8881d0bec200 [ 2198.083425] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2198.090566] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2198.098653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2198.104398] CR2: 00007fb38d41d078 CR3: 0000000113500003 CR4: 00000000003706f0 [ 2198.111533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2198.118665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2198.125798] Call Trace: [ 2198.128250] [ 2198.130356] ? __warn+0x85/0x180 [ 2198.133590] ? __list_add_valid_or_report+0x7d/0xa0 [ 2198.138470] ? report_bug+0xfc/0x1d0 [ 2198.142051] ? prb_read_valid+0x1b/0x20 [ 2198.145888] ? handle_bug+0x44/0x70 [ 2198.149379] ? exc_invalid_op+0x17/0x60 [ 2198.153219] ? asm_exc_invalid_op+0x1a/0x20 [ 2198.157409] ? __list_add_valid_or_report+0x7d/0xa0 [ 2198.162287] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2198.167175] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2198.172410] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2198.177212] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2198.181928] lookup_fast+0x6d/0xf0 [ 2198.185331] walk_component+0x26/0x160 [ 2198.189085] link_path_walk+0x24d/0x380 [ 2198.192922] ? path_lookupat.isra.40+0x22/0x120 [ 2198.197457] path_lookupat.isra.40+0x6b/0x120 [ 2198.201817] filename_lookup+0xd4/0x190 [ 2198.205659] ? lock_acquire+0xcb/0x2b0 [ 2198.209406] ? __might_fault+0x45/0x80 [ 2198.213161] ? find_held_lock+0x2d/0x90 [ 2198.216999] ? __might_fault+0x45/0x80 [ 2198.220754] vfs_statx+0x89/0x150 [ 2198.224074] vfs_fstatat+0x5a/0xc0 [ 2198.227478] __do_sys_newstat+0x30/0x70 [ 2198.231316] ? lock_release+0xc5/0x260 [ 2198.235071] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2198.239949] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2198.245089] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2198.250228] do_syscall_64+0x38/0x80 [ 2198.253808] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2198.258860] RIP: 0033:0x7f48f0eeca79 [ 2198.262440] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2198.281187] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2198.288750] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2198.295882] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2198.303009] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2198.310143] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2198.317274] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2198.324413] [ 2198.326598] irq event stamp: 12921029 [ 2198.330257] hardirqs last enabled at (12921039): [] console_unlock+0x106/0x120 [ 2198.339123] hardirqs last disabled at (12921048): [] console_unlock+0xeb/0x120 [ 2198.347903] softirqs last enabled at (12921062): [] __do_softirq+0x2be/0x371 [ 2198.356594] softirqs last disabled at (12921057): [] irq_exit_rcu+0xb0/0xd0 [ 2198.365114] ---[ end trace 0000000000000000 ]--- [ 2198.369733] ------------[ cut here ]------------ [ 2198.374357] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2198.386003] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2198.394697] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2198.450174] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2198.459296] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2198.466777] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2198.472273] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2198.477758] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2198.496503] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2198.501730] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2198.508883] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2198.516016] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003117 [ 2198.523147] R10: 000000000013ae28 R11: ffffc9000030b828 R12: ffff8881d106cbc8 [ 2198.530279] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2198.537411] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2198.545496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2198.551243] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2198.558376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2198.565511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2198.572641] Call Trace: [ 2198.575095] [ 2198.577201] ? __warn+0x85/0x180 [ 2198.580436] ? __list_add_valid_or_report+0x7d/0xa0 [ 2198.585315] ? report_bug+0xfc/0x1d0 [ 2198.588893] ? prb_read_valid+0x1b/0x20 [ 2198.592736] ? handle_bug+0x44/0x70 [ 2198.596224] ? exc_invalid_op+0x17/0x60 [ 2198.600066] ? asm_exc_invalid_op+0x1a/0x20 [ 2198.604258] ? __list_add_valid_or_report+0x7d/0xa0 [ 2198.609156] ? __list_add_valid_or_report+0x7d/0xa0 [ 2198.614036] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2198.618927] ceph_add_cap+0x20b/0x620 [ceph] [ 2198.623221] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2198.627952] ceph_fill_trace+0x143/0xf30 [ceph] [ 2198.632488] ? __mutex_lock+0x97/0xce0 [ 2198.636239] ? __mutex_lock+0xaec/0xce0 [ 2198.640083] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2198.644632] ? lock_downgrade+0xfd/0x250 [ 2198.648563] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2198.652941] ceph_con_process_message+0x74/0x150 [libceph] [ 2198.658436] ? calc_signature+0xdb/0x100 [libceph] [ 2198.663251] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2198.668580] ? process_scheduled_works+0x22a/0x510 [ 2198.673373] ? process_scheduled_works+0x22a/0x510 [ 2198.678160] ceph_con_workfn+0x344/0x750 [libceph] [ 2198.682967] ? process_scheduled_works+0x22a/0x510 [ 2198.687754] process_scheduled_works+0x2a4/0x510 [ 2198.692382] worker_thread+0x188/0x340 [ 2198.696153] ? rescuer_thread+0x310/0x310 [ 2198.700167] kthread+0xf5/0x130 [ 2198.703313] ? kthread_complete_and_exit+0x20/0x20 [ 2198.708109] ret_from_fork+0x31/0x40 [ 2198.711683] ? kthread_complete_and_exit+0x20/0x20 [ 2198.716478] ret_from_fork_asm+0x11/0x20 [ 2198.720411] [ 2198.722603] irq event stamp: 10323425 [ 2198.726269] hardirqs last enabled at (10323435): [] console_unlock+0x106/0x120 [ 2198.735145] hardirqs last disabled at (10323444): [] console_unlock+0xeb/0x120 [ 2198.743925] softirqs last enabled at (10323382): [] __do_softirq+0x2be/0x371 [ 2198.752618] softirqs last disabled at (10323377): [] irq_exit_rcu+0xb0/0xd0 [ 2198.761153] ---[ end trace 0000000000000000 ]--- [ 2198.765822] ------------[ cut here ]------------ [ 2198.770457] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2198.782103] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2198.790795] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2198.846317] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2198.855442] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2198.862937] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2198.868433] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2198.873936] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2198.892682] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2198.897926] RAX: 0000000000000000 RBX: ffff8881c277f560 RCX: 0000000000000000 [ 2198.905057] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2198.912193] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003155 [ 2198.919341] R10: 000000000013bd70 R11: ffffc9000030b828 R12: ffff8881c277fcb0 [ 2198.926492] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2198.933641] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2198.941745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2198.947531] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2198.954658] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2198.961810] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2198.968960] Call Trace: [ 2198.971413] [ 2198.973537] ? __warn+0x85/0x180 [ 2198.976805] ? __list_add_valid_or_report+0x7d/0xa0 [ 2198.981684] ? report_bug+0xfc/0x1d0 [ 2198.985278] ? prb_read_valid+0x1b/0x20 [ 2198.989156] ? handle_bug+0x44/0x70 [ 2198.992664] ? exc_invalid_op+0x17/0x60 [ 2198.996521] ? asm_exc_invalid_op+0x1a/0x20 [ 2199.000746] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.005639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.010517] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2199.015453] ceph_add_cap+0x20b/0x620 [ceph] [ 2199.019765] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2199.024505] ? lock_acquire+0xcb/0x2b0 [ 2199.028279] ? lock_is_held_type+0xaa/0x120 [ 2199.032483] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2199.036935] ? __mutex_lock+0x97/0xce0 [ 2199.040704] ? __mutex_lock+0xaec/0xce0 [ 2199.044563] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2199.049111] ? lock_downgrade+0xfd/0x250 [ 2199.053046] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2199.057441] ceph_con_process_message+0x74/0x150 [libceph] [ 2199.062942] ? calc_signature+0xdb/0x100 [libceph] [ 2199.067776] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2199.073106] ? process_scheduled_works+0x22a/0x510 [ 2199.077908] ? process_scheduled_works+0x22a/0x510 [ 2199.082701] ceph_con_workfn+0x344/0x750 [libceph] [ 2199.087544] ? process_scheduled_works+0x22a/0x510 [ 2199.092339] process_scheduled_works+0x2a4/0x510 [ 2199.096986] worker_thread+0x188/0x340 [ 2199.100758] ? rescuer_thread+0x310/0x310 [ 2199.104788] kthread+0xf5/0x130 [ 2199.107933] ? kthread_complete_and_exit+0x20/0x20 [ 2199.112742] ret_from_fork+0x31/0x40 [ 2199.116320] ? kthread_complete_and_exit+0x20/0x20 [ 2199.121131] ret_from_fork_asm+0x11/0x20 [ 2199.125069] [ 2199.127257] irq event stamp: 10324569 [ 2199.130942] hardirqs last enabled at (10324579): [] console_unlock+0x106/0x120 [ 2199.139809] hardirqs last disabled at (10324588): [] console_unlock+0xeb/0x120 [ 2199.148603] softirqs last enabled at (10324506): [] __do_softirq+0x2be/0x371 [ 2199.157315] softirqs last disabled at (10324501): [] irq_exit_rcu+0xb0/0xd0 [ 2199.165872] ---[ end trace 0000000000000000 ]--- [ 2199.170492] ------------[ cut here ]------------ [ 2199.175132] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2199.186787] WARNING: CPU: 3 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2199.195742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2199.251220] CPU: 3 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2199.260190] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2199.267686] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2199.273190] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2199.291954] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2199.297177] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2199.304320] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2199.311462] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003195 [ 2199.318605] R10: 000000000013d360 R11: ffffc90001fcf9c8 R12: ffff8881d106cbc8 [ 2199.325753] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2199.332896] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2199.340980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2199.346717] CR2: 00007fb38d430000 CR3: 000000011aeae006 CR4: 00000000003706e0 [ 2199.353861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2199.360994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2199.368125] Call Trace: [ 2199.370569] [ 2199.372685] ? __warn+0x85/0x180 [ 2199.375935] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.380815] ? report_bug+0xfc/0x1d0 [ 2199.384411] ? prb_read_valid+0x1b/0x20 [ 2199.388270] ? handle_bug+0x44/0x70 [ 2199.391777] ? exc_invalid_op+0x17/0x60 [ 2199.395627] ? asm_exc_invalid_op+0x1a/0x20 [ 2199.399853] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.404726] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.409606] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2199.414513] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2199.419247] lookup_fast+0x6d/0xf0 [ 2199.422666] walk_component+0x26/0x160 [ 2199.426428] link_path_walk+0x24d/0x380 [ 2199.430283] ? path_lookupat.isra.40+0x22/0x120 [ 2199.434828] path_lookupat.isra.40+0x6b/0x120 [ 2199.439203] filename_lookup+0xd4/0x190 [ 2199.443064] ? lock_acquire+0xcb/0x2b0 [ 2199.446811] ? __might_fault+0x45/0x80 [ 2199.450574] ? find_held_lock+0x2d/0x90 [ 2199.454431] ? __might_fault+0x45/0x80 [ 2199.458202] vfs_statx+0x89/0x150 [ 2199.461539] vfs_fstatat+0x5a/0xc0 [ 2199.464963] __do_sys_newstat+0x30/0x70 [ 2199.468800] ? lock_release+0xc5/0x260 [ 2199.472562] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2199.477458] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2199.482615] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2199.487772] do_syscall_64+0x38/0x80 [ 2199.491368] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2199.496438] RIP: 0033:0x7f48f0eeca79 [ 2199.500037] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2199.518781] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2199.526365] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2199.533516] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2199.540665] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2199.547807] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2199.554957] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2199.562096] [ 2199.564283] irq event stamp: 12792249 [ 2199.567957] hardirqs last enabled at (12792259): [] console_unlock+0x106/0x120 [ 2199.576822] hardirqs last disabled at (12792268): [] console_unlock+0xeb/0x120 [ 2199.585612] softirqs last enabled at (12792282): [] __do_softirq+0x2be/0x371 [ 2199.594320] softirqs last disabled at (12792277): [] irq_exit_rcu+0xb0/0xd0 [ 2199.602849] ---[ end trace 0000000000000000 ]--- [ 2199.607478] ------------[ cut here ]------------ [ 2199.612116] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2199.623780] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2199.632474] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2199.687968] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2199.697092] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2199.704573] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2199.710086] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2199.715571] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2199.734334] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2199.739578] RAX: 0000000000000000 RBX: ffff8881c277a720 RCX: 0000000000000000 [ 2199.746728] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2199.753890] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000031d7 [ 2199.761020] R10: 000000000013e640 R11: ffffc9000030b828 R12: ffff8881c277ae70 [ 2199.768169] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2199.775303] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2199.783408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2199.789170] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2199.796320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2199.803470] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2199.810621] Call Trace: [ 2199.813090] [ 2199.815195] ? __warn+0x85/0x180 [ 2199.818467] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.823343] ? report_bug+0xfc/0x1d0 [ 2199.826940] ? prb_read_valid+0x1b/0x20 [ 2199.830801] ? handle_bug+0x44/0x70 [ 2199.834291] ? exc_invalid_op+0x17/0x60 [ 2199.838147] ? asm_exc_invalid_op+0x1a/0x20 [ 2199.842376] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.847275] ? __list_add_valid_or_report+0x7d/0xa0 [ 2199.852164] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2199.857072] ceph_add_cap+0x20b/0x620 [ceph] [ 2199.861382] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2199.866112] ceph_fill_trace+0x143/0xf30 [ceph] [ 2199.870658] ? __mutex_lock+0x97/0xce0 [ 2199.874426] ? __mutex_lock+0xaec/0xce0 [ 2199.878303] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2199.882852] ? lock_downgrade+0xfd/0x250 [ 2199.886800] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2199.891198] ceph_con_process_message+0x74/0x150 [libceph] [ 2199.896700] ? calc_signature+0xdb/0x100 [libceph] [ 2199.901541] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2199.906893] ? process_scheduled_works+0x22a/0x510 [ 2199.911708] ? process_scheduled_works+0x22a/0x510 [ 2199.916501] ceph_con_workfn+0x344/0x750 [libceph] [ 2199.921345] ? process_scheduled_works+0x22a/0x510 [ 2199.926140] process_scheduled_works+0x2a4/0x510 [ 2199.930785] worker_thread+0x188/0x340 [ 2199.934558] ? rescuer_thread+0x310/0x310 [ 2199.938569] kthread+0xf5/0x130 [ 2199.941734] ? kthread_complete_and_exit+0x20/0x20 [ 2199.946563] ret_from_fork+0x31/0x40 [ 2199.950139] ? kthread_complete_and_exit+0x20/0x20 [ 2199.954933] ret_from_fork_asm+0x11/0x20 [ 2199.958892] [ 2199.961085] irq event stamp: 10326561 [ 2199.964749] hardirqs last enabled at (10326571): [] console_unlock+0x106/0x120 [ 2199.973636] hardirqs last disabled at (10326580): [] console_unlock+0xeb/0x120 [ 2199.982432] softirqs last enabled at (10326516): [] __do_softirq+0x2be/0x371 [ 2199.991141] softirqs last disabled at (10326511): [] irq_exit_rcu+0xb0/0xd0 [ 2199.999661] ---[ end trace 0000000000000000 ]--- [ 2200.004299] ------------[ cut here ]------------ [ 2200.008937] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2200.020620] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2200.029601] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2200.085122] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2200.094081] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2200.101558] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2200.107062] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2200.125809] RSP: 0018:ffffc90001fcfac0 EFLAGS: 00010282 [ 2200.131052] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2200.138193] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2200.145342] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003215 [ 2200.152513] R10: 000000000013f868 R11: ffffc90001fcf950 R12: ffff8881d106cbc8 [ 2200.159643] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2200.166792] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2200.174898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2200.180644] CR2: 00007fb38d4370a0 CR3: 000000011aeae004 CR4: 00000000003706f0 [ 2200.187793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2200.194945] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2200.202077] Call Trace: [ 2200.204529] [ 2200.206654] ? __warn+0x85/0x180 [ 2200.209903] ? __list_add_valid_or_report+0x7d/0xa0 [ 2200.214800] ? report_bug+0xfc/0x1d0 [ 2200.218397] ? prb_read_valid+0x1b/0x20 [ 2200.222257] ? handle_bug+0x44/0x70 [ 2200.225756] ? exc_invalid_op+0x17/0x60 [ 2200.229611] ? asm_exc_invalid_op+0x1a/0x20 [ 2200.233842] ? __list_add_valid_or_report+0x7d/0xa0 [ 2200.238739] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2200.243659] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2200.248909] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2200.253742] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2200.258487] lookup_fast+0x6d/0xf0 [ 2200.261902] walk_component+0x26/0x160 [ 2200.265676] link_path_walk+0x24d/0x380 [ 2200.269515] ? path_lookupat.isra.40+0x22/0x120 [ 2200.274070] path_lookupat.isra.40+0x6b/0x120 [ 2200.278451] filename_lookup+0xd4/0x190 [ 2200.282316] ? lock_acquire+0xcb/0x2b0 [ 2200.286066] ? __might_fault+0x45/0x80 [ 2200.289839] ? find_held_lock+0x2d/0x90 [ 2200.293677] ? __might_fault+0x45/0x80 [ 2200.297470] vfs_statx+0x89/0x150 [ 2200.300816] vfs_fstatat+0x5a/0xc0 [ 2200.304243] __do_sys_newstat+0x30/0x70 [ 2200.308082] ? lock_release+0xc5/0x260 [ 2200.311854] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2200.316739] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2200.321902] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2200.327046] do_syscall_64+0x38/0x80 [ 2200.330623] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2200.335692] RIP: 0033:0x7f48f0eeca79 [ 2200.339290] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2200.358057] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2200.365629] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2200.372777] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2200.379928] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2200.387061] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2200.394202] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2200.401382] [ 2200.403599] irq event stamp: 12794489 [ 2200.407290] hardirqs last enabled at (12794515): [] console_unlock+0x106/0x120 [ 2200.416157] hardirqs last disabled at (12794524): [] console_unlock+0xeb/0x120 [ 2200.424938] softirqs last enabled at (12794538): [] __do_softirq+0x2be/0x371 [ 2200.433636] softirqs last disabled at (12794533): [] irq_exit_rcu+0xb0/0xd0 [ 2200.442175] ---[ end trace 0000000000000000 ]--- [ 2200.446810] ------------[ cut here ]------------ [ 2200.451448] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2200.463090] WARNING: CPU: 4 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2200.472046] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2200.527530] CPU: 4 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2200.536486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2200.543981] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2200.549466] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2200.568229] RSP: 0018:ffffc90001fe7b38 EFLAGS: 00010282 [ 2200.573455] RAX: 0000000000000000 RBX: ffff8881c277a720 RCX: 0000000000000000 [ 2200.580597] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2200.587760] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003258 [ 2200.594902] R10: 0000000000140b90 R11: ffffc90001fe79c8 R12: ffff8881c277ae70 [ 2200.602030] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2200.609162] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2200.617240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2200.622995] CR2: 00007f9bd32ec000 CR3: 0000000108c08004 CR4: 00000000003706e0 [ 2200.630128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2200.637260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2200.644404] Call Trace: [ 2200.646893] [ 2200.648997] ? __warn+0x85/0x180 [ 2200.652230] ? __list_add_valid_or_report+0x7d/0xa0 [ 2200.657125] ? report_bug+0xfc/0x1d0 [ 2200.660706] ? prb_read_valid+0x1b/0x20 [ 2200.664562] ? handle_bug+0x44/0x70 [ 2200.668071] ? exc_invalid_op+0x17/0x60 [ 2200.671906] ? asm_exc_invalid_op+0x1a/0x20 [ 2200.676092] ? __list_add_valid_or_report+0x7d/0xa0 [ 2200.680971] ? __list_add_valid_or_report+0x7d/0xa0 [ 2200.685848] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2200.690763] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2200.695497] lookup_fast+0x6d/0xf0 [ 2200.698904] walk_component+0x26/0x160 [ 2200.702653] link_path_walk+0x24d/0x380 [ 2200.706508] ? path_lookupat.isra.40+0x22/0x120 [ 2200.711054] path_lookupat.isra.40+0x6b/0x120 [ 2200.715411] filename_lookup+0xd4/0x190 [ 2200.719286] ? lock_acquire+0xcb/0x2b0 [ 2200.723038] ? __might_fault+0x45/0x80 [ 2200.726790] ? find_held_lock+0x2d/0x90 [ 2200.730645] ? __might_fault+0x45/0x80 [ 2200.734420] vfs_statx+0x89/0x150 [ 2200.737778] vfs_fstatat+0x5a/0xc0 [ 2200.741191] __do_sys_newstat+0x30/0x70 [ 2200.745043] ? lock_release+0xc5/0x260 [ 2200.748796] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2200.753711] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2200.758871] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2200.764013] do_syscall_64+0x38/0x80 [ 2200.767594] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2200.772662] RIP: 0033:0x7f48f0eeca79 [ 2200.776252] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2200.795017] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2200.802581] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2200.809729] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2200.816872] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2200.824022] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2200.831147] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2200.838302] [ 2200.840490] irq event stamp: 12814069 [ 2200.844173] hardirqs last enabled at (12814079): [] console_unlock+0x106/0x120 [ 2200.853067] hardirqs last disabled at (12814102): [] console_unlock+0xeb/0x120 [ 2200.861847] softirqs last enabled at (12814122): [] __do_softirq+0x2be/0x371 [ 2200.870556] softirqs last disabled at (12814111): [] irq_exit_rcu+0xb0/0xd0 [ 2200.879090] ---[ end trace 0000000000000000 ]--- [ 2200.883701] ------------[ cut here ]------------ [ 2200.888340] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2200.899988] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2200.908680] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2200.964192] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2200.973335] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2200.980833] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2200.986353] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2200.991856] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2201.010619] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2201.015845] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2201.022997] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2201.030132] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000329a [ 2201.037262] R10: 0000000000141f98 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2201.044412] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2201.051561] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2201.059666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2201.065428] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2201.072578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2201.079729] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2201.086878] Call Trace: [ 2201.089348] [ 2201.091458] ? __warn+0x85/0x180 [ 2201.094725] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.099605] ? report_bug+0xfc/0x1d0 [ 2201.103210] ? prb_read_valid+0x1b/0x20 [ 2201.107070] ? handle_bug+0x44/0x70 [ 2201.110557] ? exc_invalid_op+0x17/0x60 [ 2201.114414] ? asm_exc_invalid_op+0x1a/0x20 [ 2201.118641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.123543] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.128420] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2201.133346] ceph_add_cap+0x20b/0x620 [ceph] [ 2201.137658] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2201.142400] ? lock_acquire+0xcb/0x2b0 [ 2201.146156] ? lock_is_held_type+0xaa/0x120 [ 2201.150358] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2201.154813] ? __mutex_lock+0x97/0xce0 [ 2201.158600] ? __mutex_lock+0xaec/0xce0 [ 2201.162457] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2201.167006] ? lock_downgrade+0xfd/0x250 [ 2201.170938] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2201.175333] ceph_con_process_message+0x74/0x150 [libceph] [ 2201.180827] ? calc_signature+0xdb/0x100 [libceph] [ 2201.185666] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2201.190991] ? process_scheduled_works+0x22a/0x510 [ 2201.195809] ? process_scheduled_works+0x22a/0x510 [ 2201.200597] ceph_con_workfn+0x344/0x750 [libceph] [ 2201.205436] ? process_scheduled_works+0x22a/0x510 [ 2201.210226] process_scheduled_works+0x2a4/0x510 [ 2201.214894] worker_thread+0x188/0x340 [ 2201.218651] ? rescuer_thread+0x310/0x310 [ 2201.222665] kthread+0xf5/0x130 [ 2201.225828] ? kthread_complete_and_exit+0x20/0x20 [ 2201.230656] ret_from_fork+0x31/0x40 [ 2201.234232] ? kthread_complete_and_exit+0x20/0x20 [ 2201.239045] ret_from_fork_asm+0x11/0x20 [ 2201.242978] [ 2201.245169] irq event stamp: 10329571 [ 2201.248837] hardirqs last enabled at (10329581): [] console_unlock+0x106/0x120 [ 2201.257719] hardirqs last disabled at (10329590): [] console_unlock+0xeb/0x120 [ 2201.266516] softirqs last enabled at (10329508): [] __do_softirq+0x2be/0x371 [ 2201.275225] softirqs last disabled at (10329503): [] irq_exit_rcu+0xb0/0xd0 [ 2201.283762] ---[ end trace 0000000000000000 ]--- [ 2201.288420] ------------[ cut here ]------------ [ 2201.293077] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2201.304735] WARNING: CPU: 0 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2201.313690] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2201.369169] CPU: 0 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2201.378119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2201.385598] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2201.391100] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2201.409849] RSP: 0018:ffffc90001fe7ac0 EFLAGS: 00010282 [ 2201.415090] RAX: 0000000000000000 RBX: ffff8881c277a720 RCX: 0000000000000000 [ 2201.422216] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2201.429357] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000032da [ 2201.436508] R10: 0000000000143370 R11: ffffc90001fe7950 R12: ffff8881c277ae70 [ 2201.443658] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2201.450806] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2201.458916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2201.464657] CR2: 00007fb38d4370a0 CR3: 0000000108c08005 CR4: 00000000003706f0 [ 2201.471797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2201.478939] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2201.486063] Call Trace: [ 2201.488509] [ 2201.490624] ? __warn+0x85/0x180 [ 2201.493874] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.498770] ? report_bug+0xfc/0x1d0 [ 2201.502349] ? prb_read_valid+0x1b/0x20 [ 2201.506213] ? handle_bug+0x44/0x70 [ 2201.509715] ? exc_invalid_op+0x17/0x60 [ 2201.513563] ? asm_exc_invalid_op+0x1a/0x20 [ 2201.517781] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.522655] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2201.527562] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2201.532817] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2201.537634] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2201.542385] lookup_fast+0x6d/0xf0 [ 2201.545788] walk_component+0x26/0x160 [ 2201.549559] link_path_walk+0x24d/0x380 [ 2201.553415] ? path_lookupat.isra.40+0x22/0x120 [ 2201.557960] path_lookupat.isra.40+0x6b/0x120 [ 2201.562316] filename_lookup+0xd4/0x190 [ 2201.566183] ? lock_acquire+0xcb/0x2b0 [ 2201.569936] ? __might_fault+0x45/0x80 [ 2201.573686] ? find_held_lock+0x2d/0x90 [ 2201.577542] ? __might_fault+0x45/0x80 [ 2201.581316] vfs_statx+0x89/0x150 [ 2201.584668] vfs_fstatat+0x5a/0xc0 [ 2201.588073] __do_sys_newstat+0x30/0x70 [ 2201.591918] ? lock_release+0xc5/0x260 [ 2201.595666] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2201.600562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2201.605719] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2201.610875] do_syscall_64+0x38/0x80 [ 2201.614470] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2201.619515] RIP: 0033:0x7f48f0eeca79 [ 2201.623104] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2201.641851] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2201.649432] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2201.656575] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2201.663716] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2201.670856] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2201.678001] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2201.685138] [ 2201.687325] irq event stamp: 12816433 [ 2201.690999] hardirqs last enabled at (12816443): [] console_unlock+0x106/0x120 [ 2201.699864] hardirqs last disabled at (12816452): [] console_unlock+0xeb/0x120 [ 2201.708654] softirqs last enabled at (12816466): [] __do_softirq+0x2be/0x371 [ 2201.717365] softirqs last disabled at (12816461): [] irq_exit_rcu+0xb0/0xd0 [ 2201.725919] ---[ end trace 0000000000000000 ]--- [ 2201.730539] ------------[ cut here ]------------ [ 2201.735193] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2201.746836] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2201.755524] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2201.811023] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2201.820145] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2201.827624] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2201.833144] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2201.838629] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2201.857376] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2201.862604] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2201.869735] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2201.876870] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000331d [ 2201.884002] R10: 0000000000144678 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2201.891133] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2201.898259] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2201.906344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2201.912091] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2201.919223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2201.926358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2201.933488] Call Trace: [ 2201.935933] [ 2201.938041] ? __warn+0x85/0x180 [ 2201.941274] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.946151] ? report_bug+0xfc/0x1d0 [ 2201.949731] ? prb_read_valid+0x1b/0x20 [ 2201.953590] ? handle_bug+0x44/0x70 [ 2201.957079] ? exc_invalid_op+0x17/0x60 [ 2201.960929] ? asm_exc_invalid_op+0x1a/0x20 [ 2201.965120] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.969995] ? __list_add_valid_or_report+0x7d/0xa0 [ 2201.974874] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2201.979781] ceph_add_cap+0x20b/0x620 [ceph] [ 2201.984068] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2201.988794] ceph_fill_trace+0x143/0xf30 [ceph] [ 2201.993351] ? __mutex_lock+0x97/0xce0 [ 2201.997102] ? __mutex_lock+0xaec/0xce0 [ 2202.000935] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2202.005477] ? lock_downgrade+0xfd/0x250 [ 2202.009406] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2202.013802] ceph_con_process_message+0x74/0x150 [libceph] [ 2202.019317] ? calc_signature+0xdb/0x100 [libceph] [ 2202.024146] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2202.029468] ? process_scheduled_works+0x22a/0x510 [ 2202.034261] ? process_scheduled_works+0x22a/0x510 [ 2202.039050] ceph_con_workfn+0x344/0x750 [libceph] [ 2202.043854] ? process_scheduled_works+0x22a/0x510 [ 2202.048644] process_scheduled_works+0x2a4/0x510 [ 2202.053269] worker_thread+0x188/0x340 [ 2202.057019] ? rescuer_thread+0x310/0x310 [ 2202.061029] kthread+0xf5/0x130 [ 2202.064177] ? kthread_complete_and_exit+0x20/0x20 [ 2202.068972] ret_from_fork+0x31/0x40 [ 2202.072548] ? kthread_complete_and_exit+0x20/0x20 [ 2202.077359] ret_from_fork_asm+0x11/0x20 [ 2202.081290] [ 2202.083476] irq event stamp: 10331593 [ 2202.087133] hardirqs last enabled at (10331603): [] console_unlock+0x106/0x120 [ 2202.096002] hardirqs last disabled at (10331612): [] console_unlock+0xeb/0x120 [ 2202.104782] softirqs last enabled at (10331550): [] __do_softirq+0x2be/0x371 [ 2202.113473] softirqs last disabled at (10331545): [] irq_exit_rcu+0xb0/0xd0 [ 2202.121992] ---[ end trace 0000000000000000 ]--- [ 2202.126791] ------------[ cut here ]------------ [ 2202.131427] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2202.143074] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2202.151765] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2202.207278] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2202.216402] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2202.223897] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2202.229418] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2202.234933] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2202.253677] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2202.258930] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2202.266063] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2202.273213] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000335b [ 2202.280362] R10: 00000000001458e8 R11: ffffc9000030b828 R12: ffff8881c5351118 [ 2202.287511] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2202.294662] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2202.302764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2202.308521] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2202.315671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2202.322820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2202.329970] Call Trace: [ 2202.332424] [ 2202.334547] ? __warn+0x85/0x180 [ 2202.337817] ? __list_add_valid_or_report+0x7d/0xa0 [ 2202.342713] ? report_bug+0xfc/0x1d0 [ 2202.346290] ? prb_read_valid+0x1b/0x20 [ 2202.350153] ? handle_bug+0x44/0x70 [ 2202.353640] ? exc_invalid_op+0x17/0x60 [ 2202.357499] ? asm_exc_invalid_op+0x1a/0x20 [ 2202.361725] ? __list_add_valid_or_report+0x7d/0xa0 [ 2202.366625] ? __list_add_valid_or_report+0x7d/0xa0 [ 2202.371502] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2202.376428] ceph_add_cap+0x20b/0x620 [ceph] [ 2202.380740] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2202.385482] ? lock_acquire+0xcb/0x2b0 [ 2202.389257] ? lock_is_held_type+0xaa/0x120 [ 2202.393459] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2202.397931] ? __mutex_lock+0x97/0xce0 [ 2202.401682] ? __mutex_lock+0xaec/0xce0 [ 2202.405556] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2202.410106] ? lock_downgrade+0xfd/0x250 [ 2202.414040] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2202.418435] ceph_con_process_message+0x74/0x150 [libceph] [ 2202.423938] ? calc_signature+0xdb/0x100 [libceph] [ 2202.428769] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2202.434098] ? process_scheduled_works+0x22a/0x510 [ 2202.438912] ? process_scheduled_works+0x22a/0x510 [ 2202.443706] ceph_con_workfn+0x344/0x750 [libceph] [ 2202.448529] ? process_scheduled_works+0x22a/0x510 [ 2202.453317] process_scheduled_works+0x2a4/0x510 [ 2202.457965] worker_thread+0x188/0x340 [ 2202.461735] ? rescuer_thread+0x310/0x310 [ 2202.465765] kthread+0xf5/0x130 [ 2202.468933] ? kthread_complete_and_exit+0x20/0x20 [ 2202.473747] ret_from_fork+0x31/0x40 [ 2202.477323] ? kthread_complete_and_exit+0x20/0x20 [ 2202.482136] ret_from_fork_asm+0x11/0x20 [ 2202.486071] [ 2202.488261] irq event stamp: 10332743 [ 2202.491946] hardirqs last enabled at (10332753): [] console_unlock+0x106/0x120 [ 2202.500812] hardirqs last disabled at (10332762): [] console_unlock+0xeb/0x120 [ 2202.509607] softirqs last enabled at (10332690): [] __do_softirq+0x2be/0x371 [ 2202.518319] softirqs last disabled at (10332677): [] irq_exit_rcu+0xb0/0xd0 [ 2202.526855] ---[ end trace 0000000000000000 ]--- [ 2202.531510] ------------[ cut here ]------------ [ 2202.536129] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2202.547793] WARNING: CPU: 3 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2202.556746] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2202.612253] CPU: 3 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2202.621251] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2202.628725] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2202.634227] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2202.652991] RSP: 0018:ffffc90001957b38 EFLAGS: 00010282 [ 2202.658230] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2202.665375] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2202.672516] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000339b [ 2202.679660] R10: 0000000000146b80 R11: ffffc900019579c8 R12: ffff8881d1224bc8 [ 2202.686809] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2202.693950] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2202.702036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2202.707773] CR2: 00007fb38d456030 CR3: 000000016696c004 CR4: 00000000003706e0 [ 2202.714934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2202.722067] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2202.729199] Call Trace: [ 2202.731653] [ 2202.733757] ? __warn+0x85/0x180 [ 2202.736991] ? __list_add_valid_or_report+0x7d/0xa0 [ 2202.741870] ? report_bug+0xfc/0x1d0 [ 2202.745451] ? prb_read_valid+0x1b/0x20 [ 2202.749291] ? handle_bug+0x44/0x70 [ 2202.752781] ? exc_invalid_op+0x17/0x60 [ 2202.756620] ? asm_exc_invalid_op+0x1a/0x20 [ 2202.760810] ? __list_add_valid_or_report+0x7d/0xa0 [ 2202.765687] ? __list_add_valid_or_report+0x7d/0xa0 [ 2202.770565] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2202.775457] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2202.780172] lookup_fast+0x6d/0xf0 [ 2202.783576] walk_component+0x26/0x160 [ 2202.787345] link_path_walk+0x24d/0x380 [ 2202.791202] ? path_lookupat.isra.40+0x22/0x120 [ 2202.795754] path_lookupat.isra.40+0x6b/0x120 [ 2202.800130] filename_lookup+0xd4/0x190 [ 2202.803972] ? lock_acquire+0xcb/0x2b0 [ 2202.807721] ? __might_fault+0x45/0x80 [ 2202.811491] ? find_held_lock+0x2d/0x90 [ 2202.815347] ? __might_fault+0x45/0x80 [ 2202.819121] vfs_statx+0x89/0x150 [ 2202.822458] vfs_fstatat+0x5a/0xc0 [ 2202.825862] __do_sys_newstat+0x30/0x70 [ 2202.829718] ? lock_release+0xc5/0x260 [ 2202.833490] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2202.838383] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2202.843540] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2202.848698] do_syscall_64+0x38/0x80 [ 2202.852294] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2202.857363] RIP: 0033:0x7f48f0eeca79 [ 2202.860963] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2202.879707] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2202.887290] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2202.894440] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2202.901580] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2202.908723] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2202.915867] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2202.923020] [ 2202.925222] irq event stamp: 12788125 [ 2202.928899] hardirqs last enabled at (12788135): [] console_unlock+0x106/0x120 [ 2202.937782] hardirqs last disabled at (12788144): [] console_unlock+0xeb/0x120 [ 2202.946572] softirqs last enabled at (12788158): [] __do_softirq+0x2be/0x371 [ 2202.955280] softirqs last disabled at (12788153): [] irq_exit_rcu+0xb0/0xd0 [ 2202.963818] ---[ end trace 0000000000000000 ]--- [ 2202.968447] ------------[ cut here ]------------ [ 2202.973088] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2202.984734] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2202.993428] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2203.048968] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2203.058099] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2203.065578] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2203.071084] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2203.076577] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2203.095323] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2203.100548] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2203.107681] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2203.114817] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000033dd [ 2203.121951] R10: 0000000000147e68 R11: ffffc9000030b828 R12: ffff8881c5351ae0 [ 2203.129080] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2203.136229] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2203.144315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2203.150064] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2203.157196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2203.164328] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2203.171461] Call Trace: [ 2203.173914] [ 2203.176018] ? __warn+0x85/0x180 [ 2203.179255] ? __list_add_valid_or_report+0x7d/0xa0 [ 2203.184136] ? report_bug+0xfc/0x1d0 [ 2203.187720] ? prb_read_valid+0x1b/0x20 [ 2203.191564] ? handle_bug+0x44/0x70 [ 2203.195063] ? exc_invalid_op+0x17/0x60 [ 2203.198903] ? asm_exc_invalid_op+0x1a/0x20 [ 2203.203108] ? __list_add_valid_or_report+0x7d/0xa0 [ 2203.207996] ? __list_add_valid_or_report+0x7d/0xa0 [ 2203.212880] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2203.217781] ceph_add_cap+0x20b/0x620 [ceph] [ 2203.222080] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2203.226822] ceph_fill_trace+0x143/0xf30 [ceph] [ 2203.231369] ? __mutex_lock+0x97/0xce0 [ 2203.235120] ? __mutex_lock+0xaec/0xce0 [ 2203.238964] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2203.243519] ? lock_downgrade+0xfd/0x250 [ 2203.247453] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2203.251844] ceph_con_process_message+0x74/0x150 [libceph] [ 2203.257349] ? calc_signature+0xdb/0x100 [libceph] [ 2203.262166] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2203.267507] ? process_scheduled_works+0x22a/0x510 [ 2203.272307] ? process_scheduled_works+0x22a/0x510 [ 2203.277101] ceph_con_workfn+0x344/0x750 [libceph] [ 2203.281916] ? process_scheduled_works+0x22a/0x510 [ 2203.286736] process_scheduled_works+0x2a4/0x510 [ 2203.291369] worker_thread+0x188/0x340 [ 2203.295124] ? rescuer_thread+0x310/0x310 [ 2203.299141] kthread+0xf5/0x130 [ 2203.302287] ? kthread_complete_and_exit+0x20/0x20 [ 2203.307085] ret_from_fork+0x31/0x40 [ 2203.310667] ? kthread_complete_and_exit+0x20/0x20 [ 2203.315461] ret_from_fork_asm+0x11/0x20 [ 2203.319402] [ 2203.321596] irq event stamp: 10334741 [ 2203.325261] hardirqs last enabled at (10334751): [] console_unlock+0x106/0x120 [ 2203.334129] hardirqs last disabled at (10334760): [] console_unlock+0xeb/0x120 [ 2203.342909] softirqs last enabled at (10334698): [] __do_softirq+0x2be/0x371 [ 2203.351600] softirqs last disabled at (10334693): [] irq_exit_rcu+0xb0/0xd0 [ 2203.360120] ---[ end trace 0000000000000000 ]--- [ 2203.364740] ------------[ cut here ]------------ [ 2203.369361] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2203.381008] WARNING: CPU: 1 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2203.390009] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2203.445578] CPU: 1 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2203.454528] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2203.462006] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2203.467494] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2203.486240] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2203.491465] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2203.498598] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2203.505730] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000341b [ 2203.512865] R10: 0000000000148f48 R11: ffffc90001957950 R12: ffff8881d1224bc8 [ 2203.519996] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2203.527129] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2203.535216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2203.540961] CR2: 00007fb38defc000 CR3: 000000016696c004 CR4: 00000000003706e0 [ 2203.548094] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2203.555227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2203.562360] Call Trace: [ 2203.564812] [ 2203.566922] ? __warn+0x85/0x180 [ 2203.570160] ? __list_add_valid_or_report+0x7d/0xa0 [ 2203.575041] ? report_bug+0xfc/0x1d0 [ 2203.578618] ? prb_read_valid+0x1b/0x20 [ 2203.582464] ? handle_bug+0x44/0x70 [ 2203.585961] ? exc_invalid_op+0x17/0x60 [ 2203.589801] ? asm_exc_invalid_op+0x1a/0x20 [ 2203.594000] ? __list_add_valid_or_report+0x7d/0xa0 [ 2203.598885] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2203.603784] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2203.609038] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2203.613853] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2203.618582] lookup_fast+0x6d/0xf0 [ 2203.621991] walk_component+0x26/0x160 [ 2203.625745] link_path_walk+0x24d/0x380 [ 2203.629591] ? path_lookupat.isra.40+0x22/0x120 [ 2203.634127] path_lookupat.isra.40+0x6b/0x120 [ 2203.638493] filename_lookup+0xd4/0x190 [ 2203.642337] ? lock_acquire+0xcb/0x2b0 [ 2203.646095] ? __might_fault+0x45/0x80 [ 2203.649854] ? find_held_lock+0x2d/0x90 [ 2203.653692] ? __might_fault+0x45/0x80 [ 2203.657452] vfs_statx+0x89/0x150 [ 2203.660778] vfs_fstatat+0x5a/0xc0 [ 2203.664191] __do_sys_newstat+0x30/0x70 [ 2203.668036] ? lock_release+0xc5/0x260 [ 2203.671793] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2203.676674] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2203.681817] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2203.686957] do_syscall_64+0x38/0x80 [ 2203.690535] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2203.695586] RIP: 0033:0x7f48f0eeca79 [ 2203.699166] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2203.717912] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2203.725477] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2203.732611] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2203.739744] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2203.746878] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2203.754009] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2203.761156] [ 2203.763343] irq event stamp: 12790051 [ 2203.767010] hardirqs last enabled at (12790061): [] console_unlock+0x106/0x120 [ 2203.775877] hardirqs last disabled at (12790070): [] console_unlock+0xeb/0x120 [ 2203.784656] softirqs last enabled at (12789832): [] __do_softirq+0x2be/0x371 [ 2203.793348] softirqs last disabled at (12789825): [] irq_exit_rcu+0xb0/0xd0 [ 2203.801867] ---[ end trace 0000000000000000 ]--- [ 2203.806487] ------------[ cut here ]------------ [ 2203.811111] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2203.822759] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2203.831454] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2203.886938] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2203.896065] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2203.903538] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2203.909032] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2203.914516] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2203.933264] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2203.938492] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2203.945622] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2203.952757] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000345e [ 2203.959889] R10: 000000000014a108 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2203.967021] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2203.974154] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2203.982239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2203.987985] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2203.995118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2204.002251] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2204.009385] Call Trace: [ 2204.011836] [ 2204.013943] ? __warn+0x85/0x180 [ 2204.017178] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.022056] ? report_bug+0xfc/0x1d0 [ 2204.025635] ? prb_read_valid+0x1b/0x20 [ 2204.029476] ? handle_bug+0x44/0x70 [ 2204.032968] ? exc_invalid_op+0x17/0x60 [ 2204.036806] ? asm_exc_invalid_op+0x1a/0x20 [ 2204.040998] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.045873] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.050752] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2204.055643] ceph_add_cap+0x20b/0x620 [ceph] [ 2204.059927] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2204.064650] ? lock_acquire+0xcb/0x2b0 [ 2204.068400] ? lock_is_held_type+0xaa/0x120 [ 2204.072586] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2204.077040] ? __mutex_lock+0x97/0xce0 [ 2204.080792] ? __mutex_lock+0xaec/0xce0 [ 2204.084631] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2204.089171] ? lock_downgrade+0xfd/0x250 [ 2204.093101] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2204.097475] ceph_con_process_message+0x74/0x150 [libceph] [ 2204.102969] ? calc_signature+0xdb/0x100 [libceph] [ 2204.107774] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2204.113105] ? process_scheduled_works+0x22a/0x510 [ 2204.117896] ? process_scheduled_works+0x22a/0x510 [ 2204.122686] ceph_con_workfn+0x344/0x750 [libceph] [ 2204.127491] ? process_scheduled_works+0x22a/0x510 [ 2204.132279] process_scheduled_works+0x2a4/0x510 [ 2204.136906] worker_thread+0x188/0x340 [ 2204.140653] ? rescuer_thread+0x310/0x310 [ 2204.144667] kthread+0xf5/0x130 [ 2204.147811] ? kthread_complete_and_exit+0x20/0x20 [ 2204.152605] ret_from_fork+0x31/0x40 [ 2204.156182] ? kthread_complete_and_exit+0x20/0x20 [ 2204.160977] ret_from_fork_asm+0x11/0x20 [ 2204.164909] [ 2204.167094] irq event stamp: 10338763 [ 2204.170761] hardirqs last enabled at (10338773): [] console_unlock+0x106/0x120 [ 2204.179628] hardirqs last disabled at (10338782): [] console_unlock+0xeb/0x120 [ 2204.188406] softirqs last enabled at (10338700): [] __do_softirq+0x2be/0x371 [ 2204.197099] softirqs last disabled at (10338695): [] irq_exit_rcu+0xb0/0xd0 [ 2204.205618] ---[ end trace 0000000000000000 ]--- [ 2204.210231] ------------[ cut here ]------------ [ 2204.214850] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2204.226514] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2204.235468] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2204.290967] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2204.299913] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2204.307401] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2204.312906] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2204.331692] RSP: 0018:ffffc90001f7fb38 EFLAGS: 00010282 [ 2204.336923] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2204.344071] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2204.351203] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000349e [ 2204.358331] R10: 000000000014b3b0 R11: ffffc90001f7f9c8 R12: ffff8881c5351ae0 [ 2204.365477] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2204.372618] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2204.380713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.386468] CR2: 00007fb38d4d60c0 CR3: 0000000113532004 CR4: 00000000003706e0 [ 2204.393611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2204.400751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2204.407894] Call Trace: [ 2204.410357] [ 2204.412479] ? __warn+0x85/0x180 [ 2204.415729] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.420625] ? report_bug+0xfc/0x1d0 [ 2204.424222] ? prb_read_valid+0x1b/0x20 [ 2204.428062] ? handle_bug+0x44/0x70 [ 2204.431554] ? exc_invalid_op+0x17/0x60 [ 2204.435402] ? asm_exc_invalid_op+0x1a/0x20 [ 2204.439627] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.444503] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.449399] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2204.454305] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2204.459040] lookup_fast+0x6d/0xf0 [ 2204.462443] walk_component+0x26/0x160 [ 2204.466216] link_path_walk+0x24d/0x380 [ 2204.470053] ? path_lookupat.isra.40+0x22/0x120 [ 2204.474587] path_lookupat.isra.40+0x6b/0x120 [ 2204.478972] filename_lookup+0xd4/0x190 [ 2204.482813] ? lock_acquire+0xcb/0x2b0 [ 2204.486579] ? __might_fault+0x45/0x80 [ 2204.490353] ? find_held_lock+0x2d/0x90 [ 2204.494207] ? __might_fault+0x45/0x80 [ 2204.497974] vfs_statx+0x89/0x150 [ 2204.501307] vfs_fstatat+0x5a/0xc0 [ 2204.504711] __do_sys_newstat+0x30/0x70 [ 2204.508569] ? lock_release+0xc5/0x260 [ 2204.512361] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2204.517234] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2204.522373] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2204.527533] do_syscall_64+0x38/0x80 [ 2204.531128] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2204.536180] RIP: 0033:0x7f48f0eeca79 [ 2204.539760] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2204.558524] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2204.566108] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2204.573256] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2204.580398] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2204.587539] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2204.594682] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2204.601854] [ 2204.604041] irq event stamp: 12915613 [ 2204.607706] hardirqs last enabled at (12915623): [] console_unlock+0x106/0x120 [ 2204.616593] hardirqs last disabled at (12915638): [] console_unlock+0xeb/0x120 [ 2204.625388] softirqs last enabled at (12915652): [] __do_softirq+0x2be/0x371 [ 2204.634097] softirqs last disabled at (12915647): [] irq_exit_rcu+0xb0/0xd0 [ 2204.642620] ---[ end trace 0000000000000000 ]--- [ 2204.647256] ------------[ cut here ]------------ [ 2204.651895] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2204.663544] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2204.672260] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2204.727800] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2204.736923] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2204.744402] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2204.749908] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2204.755393] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2204.774139] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2204.779367] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2204.786498] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2204.793632] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000034e0 [ 2204.800763] R10: 000000000014c530 R11: ffffc9000030b828 R12: ffff8881d1144bc8 [ 2204.807896] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2204.815030] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2204.823115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.828861] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2204.835996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2204.843128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2204.850263] Call Trace: [ 2204.852714] [ 2204.854820] ? __warn+0x85/0x180 [ 2204.858056] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.862943] ? report_bug+0xfc/0x1d0 [ 2204.866527] ? prb_read_valid+0x1b/0x20 [ 2204.870370] ? handle_bug+0x44/0x70 [ 2204.873862] ? exc_invalid_op+0x17/0x60 [ 2204.877700] ? asm_exc_invalid_op+0x1a/0x20 [ 2204.881896] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.886776] ? __list_add_valid_or_report+0x7d/0xa0 [ 2204.891660] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2204.896562] ceph_add_cap+0x20b/0x620 [ceph] [ 2204.900861] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2204.905603] ceph_fill_trace+0x143/0xf30 [ceph] [ 2204.910151] ? __mutex_lock+0x97/0xce0 [ 2204.913909] ? __mutex_lock+0xaec/0xce0 [ 2204.917751] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2204.922301] ? lock_downgrade+0xfd/0x250 [ 2204.926236] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2204.930644] ceph_con_process_message+0x74/0x150 [libceph] [ 2204.936149] ? calc_signature+0xdb/0x100 [libceph] [ 2204.940990] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2204.946330] ? process_scheduled_works+0x22a/0x510 [ 2204.951131] ? process_scheduled_works+0x22a/0x510 [ 2204.955925] ceph_con_workfn+0x344/0x750 [libceph] [ 2204.960742] ? process_scheduled_works+0x22a/0x510 [ 2204.965536] process_scheduled_works+0x2a4/0x510 [ 2204.970169] worker_thread+0x188/0x340 [ 2204.973928] ? rescuer_thread+0x310/0x310 [ 2204.977940] kthread+0xf5/0x130 [ 2204.981086] ? kthread_complete_and_exit+0x20/0x20 [ 2204.985883] ret_from_fork+0x31/0x40 [ 2204.989468] ? kthread_complete_and_exit+0x20/0x20 [ 2204.994273] ret_from_fork_asm+0x11/0x20 [ 2204.998218] [ 2205.000413] irq event stamp: 10340935 [ 2205.004078] hardirqs last enabled at (10340945): [] console_unlock+0x106/0x120 [ 2205.012943] hardirqs last disabled at (10340954): [] console_unlock+0xeb/0x120 [ 2205.021725] softirqs last enabled at (10340892): [] __do_softirq+0x2be/0x371 [ 2205.030417] softirqs last disabled at (10340887): [] irq_exit_rcu+0xb0/0xd0 [ 2205.038937] ---[ end trace 0000000000000000 ]--- [ 2205.043556] ------------[ cut here ]------------ [ 2205.048175] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2205.059838] WARNING: CPU: 1 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2205.068793] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2205.124329] CPU: 1 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2205.133300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2205.140798] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2205.146300] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2205.165066] RSP: 0018:ffffc90001f7fac0 EFLAGS: 00010282 [ 2205.170290] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2205.177440] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2205.184589] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000351e [ 2205.191805] R10: 000000000014d760 R11: ffffc90001f7f950 R12: ffff8881c5351ae0 [ 2205.199001] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2205.206138] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2205.214224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2205.219986] CR2: 00007fb38defc000 CR3: 0000000113532005 CR4: 00000000003706e0 [ 2205.227119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2205.234252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2205.241384] Call Trace: [ 2205.243856] [ 2205.245989] ? __warn+0x85/0x180 [ 2205.249222] ? __list_add_valid_or_report+0x7d/0xa0 [ 2205.254101] ? report_bug+0xfc/0x1d0 [ 2205.257677] ? prb_read_valid+0x1b/0x20 [ 2205.261555] ? handle_bug+0x44/0x70 [ 2205.265047] ? exc_invalid_op+0x17/0x60 [ 2205.268886] ? asm_exc_invalid_op+0x1a/0x20 [ 2205.273098] ? __list_add_valid_or_report+0x7d/0xa0 [ 2205.277992] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2205.282915] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2205.288166] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2205.292992] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2205.297733] lookup_fast+0x6d/0xf0 [ 2205.301145] walk_component+0x26/0x160 [ 2205.304914] link_path_walk+0x24d/0x380 [ 2205.308752] ? path_lookupat.isra.40+0x22/0x120 [ 2205.313324] path_lookupat.isra.40+0x6b/0x120 [ 2205.317709] filename_lookup+0xd4/0x190 [ 2205.321577] ? lock_acquire+0xcb/0x2b0 [ 2205.325332] ? __might_fault+0x45/0x80 [ 2205.329103] ? find_held_lock+0x2d/0x90 [ 2205.332940] ? __might_fault+0x45/0x80 [ 2205.336716] vfs_statx+0x89/0x150 [ 2205.340047] vfs_fstatat+0x5a/0xc0 [ 2205.343473] __do_sys_newstat+0x30/0x70 [ 2205.347310] ? lock_release+0xc5/0x260 [ 2205.351085] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2205.355986] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2205.361129] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2205.366290] do_syscall_64+0x38/0x80 [ 2205.369871] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2205.374939] RIP: 0033:0x7f48f0eeca79 [ 2205.378536] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2205.397282] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2205.404866] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2205.412018] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2205.419150] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2205.426285] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2205.433433] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2205.440613] [ 2205.442801] irq event stamp: 12917535 [ 2205.446485] hardirqs last enabled at (12917545): [] console_unlock+0x106/0x120 [ 2205.455369] hardirqs last disabled at (12917554): [] console_unlock+0xeb/0x120 [ 2205.464165] softirqs last enabled at (12917212): [] __do_softirq+0x2be/0x371 [ 2205.472857] softirqs last disabled at (12917207): [] irq_exit_rcu+0xb0/0xd0 [ 2205.481395] ---[ end trace 0000000000000000 ]--- [ 2205.486031] ------------[ cut here ]------------ [ 2205.490652] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2205.502312] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2205.511010] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2205.566486] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2205.575627] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2205.583116] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2205.588610] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2205.594113] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2205.612861] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2205.618103] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2205.625236] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2205.632370] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003561 [ 2205.639520] R10: 000000000014ea80 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2205.646652] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2205.653800] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2205.661904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2205.667661] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2205.674809] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2205.681951] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2205.689093] Call Trace: [ 2205.691546] [ 2205.693661] ? __warn+0x85/0x180 [ 2205.696913] ? __list_add_valid_or_report+0x7d/0xa0 [ 2205.701808] ? report_bug+0xfc/0x1d0 [ 2205.705405] ? prb_read_valid+0x1b/0x20 [ 2205.709271] ? handle_bug+0x44/0x70 [ 2205.712780] ? exc_invalid_op+0x17/0x60 [ 2205.716635] ? asm_exc_invalid_op+0x1a/0x20 [ 2205.720861] ? __list_add_valid_or_report+0x7d/0xa0 [ 2205.725736] ? __list_add_valid_or_report+0x7d/0xa0 [ 2205.730632] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2205.735541] ceph_add_cap+0x20b/0x620 [ceph] [ 2205.739859] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2205.744591] ? lock_acquire+0xcb/0x2b0 [ 2205.748340] ? lock_is_held_type+0xaa/0x120 [ 2205.752546] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2205.757017] ? __mutex_lock+0x97/0xce0 [ 2205.760769] ? __mutex_lock+0xaec/0xce0 [ 2205.764627] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2205.769185] ? lock_downgrade+0xfd/0x250 [ 2205.773114] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2205.777510] ceph_con_process_message+0x74/0x150 [libceph] [ 2205.783008] ? calc_signature+0xdb/0x100 [libceph] [ 2205.787811] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2205.793151] ? process_scheduled_works+0x22a/0x510 [ 2205.797942] ? process_scheduled_works+0x22a/0x510 [ 2205.802751] ceph_con_workfn+0x344/0x750 [libceph] [ 2205.807571] ? process_scheduled_works+0x22a/0x510 [ 2205.812379] process_scheduled_works+0x2a4/0x510 [ 2205.817021] worker_thread+0x188/0x340 [ 2205.820769] ? rescuer_thread+0x310/0x310 [ 2205.824799] kthread+0xf5/0x130 [ 2205.827961] ? kthread_complete_and_exit+0x20/0x20 [ 2205.832773] ret_from_fork+0x31/0x40 [ 2205.836351] ? kthread_complete_and_exit+0x20/0x20 [ 2205.841161] ret_from_fork_asm+0x11/0x20 [ 2205.845095] [ 2205.847296] irq event stamp: 10343005 [ 2205.850993] hardirqs last enabled at (10343015): [] console_unlock+0x106/0x120 [ 2205.859856] hardirqs last disabled at (10343024): [] console_unlock+0xeb/0x120 [ 2205.868652] softirqs last enabled at (10342940): [] __do_softirq+0x2be/0x371 [ 2205.877363] softirqs last disabled at (10342935): [] irq_exit_rcu+0xb0/0xd0 [ 2205.885899] ---[ end trace 0000000000000000 ]--- [ 2205.890530] ------------[ cut here ]------------ [ 2205.895164] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2205.906830] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2205.915783] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2205.971267] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2205.980221] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2205.987698] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2205.993203] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2206.011951] RSP: 0018:ffffc90001fdfb98 EFLAGS: 00010282 [ 2206.017176] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2206.024309] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2206.031458] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000035a1 [ 2206.038600] R10: 000000000014fbc0 R11: ffffc90001fdfa28 R12: ffff8881d1144bc8 [ 2206.045742] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2206.052891] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2206.061000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2206.066741] CR2: 00007fb38d4d60c0 CR3: 0000000179402005 CR4: 00000000003706e0 [ 2206.073892] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2206.081026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2206.088160] Call Trace: [ 2206.090610] [ 2206.092727] ? __warn+0x85/0x180 [ 2206.095958] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.100854] ? report_bug+0xfc/0x1d0 [ 2206.104473] ? prb_read_valid+0x1b/0x20 [ 2206.108344] ? handle_bug+0x44/0x70 [ 2206.111836] ? exc_invalid_op+0x17/0x60 [ 2206.115692] ? asm_exc_invalid_op+0x1a/0x20 [ 2206.119917] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.124791] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.129689] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2206.134595] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2206.139346] lookup_fast+0x6d/0xf0 [ 2206.142751] walk_component+0x26/0x160 [ 2206.146520] path_lookupat.isra.40+0x58/0x120 [ 2206.150896] filename_lookup+0xd4/0x190 [ 2206.154756] ? lock_acquire+0xcb/0x2b0 [ 2206.158522] ? __might_fault+0x45/0x80 [ 2206.162308] ? find_held_lock+0x2d/0x90 [ 2206.166167] ? __might_fault+0x45/0x80 [ 2206.169942] vfs_statx+0x89/0x150 [ 2206.173259] vfs_fstatat+0x5a/0xc0 [ 2206.176664] __do_sys_newstat+0x30/0x70 [ 2206.180524] ? lock_release+0xc5/0x260 [ 2206.184309] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2206.189205] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2206.194344] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2206.199501] do_syscall_64+0x38/0x80 [ 2206.203097] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2206.208149] RIP: 0033:0x7f48f0eeca79 [ 2206.211731] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2206.230495] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2206.238061] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2206.245191] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2206.252325] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2206.259475] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2206.266606] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2206.273773] [ 2206.275959] irq event stamp: 12816135 [ 2206.279644] hardirqs last enabled at (12816145): [] console_unlock+0x106/0x120 [ 2206.288527] hardirqs last disabled at (12816154): [] console_unlock+0xeb/0x120 [ 2206.297322] softirqs last enabled at (12816072): [] __do_softirq+0x2be/0x371 [ 2206.306017] softirqs last disabled at (12816173): [] irq_exit_rcu+0xb0/0xd0 [ 2206.314536] ---[ end trace 0000000000000000 ]--- [ 2206.319155] ------------[ cut here ]------------ [ 2206.323773] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2206.335436] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2206.344132] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2206.399608] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2206.408752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2206.416249] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2206.421743] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2206.427246] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2206.446005] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2206.451227] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2206.458360] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2206.465510] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000035e1 [ 2206.472662] R10: 0000000000150e38 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2206.479811] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2206.486960] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2206.495064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2206.500811] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2206.507960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2206.515111] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2206.522243] Call Trace: [ 2206.524697] [ 2206.526820] ? __warn+0x85/0x180 [ 2206.530072] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.534950] ? report_bug+0xfc/0x1d0 [ 2206.538548] ? prb_read_valid+0x1b/0x20 [ 2206.542403] ? handle_bug+0x44/0x70 [ 2206.545912] ? exc_invalid_op+0x17/0x60 [ 2206.549761] ? asm_exc_invalid_op+0x1a/0x20 [ 2206.553991] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.558870] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.563752] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2206.568659] ceph_add_cap+0x20b/0x620 [ceph] [ 2206.572959] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2206.577696] ceph_fill_trace+0x143/0xf30 [ceph] [ 2206.582237] ? __mutex_lock+0x97/0xce0 [ 2206.586006] ? __mutex_lock+0xaec/0xce0 [ 2206.589845] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2206.594406] ? lock_downgrade+0xfd/0x250 [ 2206.598370] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2206.602768] ceph_con_process_message+0x74/0x150 [libceph] [ 2206.608262] ? calc_signature+0xdb/0x100 [libceph] [ 2206.613067] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2206.618416] ? process_scheduled_works+0x22a/0x510 [ 2206.623206] ? process_scheduled_works+0x22a/0x510 [ 2206.628010] ceph_con_workfn+0x344/0x750 [libceph] [ 2206.632817] ? process_scheduled_works+0x22a/0x510 [ 2206.637625] process_scheduled_works+0x2a4/0x510 [ 2206.642268] worker_thread+0x188/0x340 [ 2206.646016] ? rescuer_thread+0x310/0x310 [ 2206.650029] kthread+0xf5/0x130 [ 2206.653174] ? kthread_complete_and_exit+0x20/0x20 [ 2206.657968] ret_from_fork+0x31/0x40 [ 2206.661562] ? kthread_complete_and_exit+0x20/0x20 [ 2206.666373] ret_from_fork_asm+0x11/0x20 [ 2206.670341] [ 2206.672526] irq event stamp: 10345019 [ 2206.676201] hardirqs last enabled at (10345029): [] console_unlock+0x106/0x120 [ 2206.685067] hardirqs last disabled at (10345038): [] console_unlock+0xeb/0x120 [ 2206.693845] softirqs last enabled at (10344996): [] __do_softirq+0x2be/0x371 [ 2206.702556] softirqs last disabled at (10344991): [] irq_exit_rcu+0xb0/0xd0 [ 2206.711094] ---[ end trace 0000000000000000 ]--- [ 2206.715714] ------------[ cut here ]------------ [ 2206.720349] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2206.731999] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2206.740953] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2206.796421] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2206.805372] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2206.812851] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2206.818338] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2206.837084] RSP: 0018:ffffc90001fdfb20 EFLAGS: 00010282 [ 2206.842311] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2206.849442] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2206.856574] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000361f [ 2206.863700] R10: 0000000000151f10 R11: ffffc90001fdf9b0 R12: ffff8881d1144bc8 [ 2206.870833] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2206.877965] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2206.886057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2206.891797] CR2: 000055a0beba2290 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2206.898930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2206.906063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2206.913202] Call Trace: [ 2206.915659] [ 2206.917764] ? __warn+0x85/0x180 [ 2206.920996] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.925876] ? report_bug+0xfc/0x1d0 [ 2206.929457] ? prb_read_valid+0x1b/0x20 [ 2206.933297] ? handle_bug+0x44/0x70 [ 2206.936787] ? exc_invalid_op+0x17/0x60 [ 2206.940627] ? asm_exc_invalid_op+0x1a/0x20 [ 2206.944816] ? __list_add_valid_or_report+0x7d/0xa0 [ 2206.949694] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2206.954582] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2206.959816] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2206.964618] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2206.969334] lookup_fast+0x6d/0xf0 [ 2206.972739] walk_component+0x26/0x160 [ 2206.976490] path_lookupat.isra.40+0x58/0x120 [ 2206.980850] filename_lookup+0xd4/0x190 [ 2206.984693] ? lock_acquire+0xcb/0x2b0 [ 2206.988440] ? __might_fault+0x45/0x80 [ 2206.992197] ? find_held_lock+0x2d/0x90 [ 2206.996033] ? __might_fault+0x45/0x80 [ 2206.999790] vfs_statx+0x89/0x150 [ 2207.003109] vfs_fstatat+0x5a/0xc0 [ 2207.006513] __do_sys_newstat+0x30/0x70 [ 2207.010351] ? lock_release+0xc5/0x260 [ 2207.014106] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2207.018982] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2207.024132] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2207.029271] do_syscall_64+0x38/0x80 [ 2207.032851] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2207.037904] RIP: 0033:0x7f48f0eeca79 [ 2207.041480] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2207.060230] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2207.067795] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2207.074929] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2207.082061] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2207.089192] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2207.096325] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2207.103465] [ 2207.105653] irq event stamp: 12818141 [ 2207.109317] hardirqs last enabled at (12818151): [] console_unlock+0x106/0x120 [ 2207.118184] hardirqs last disabled at (12818160): [] console_unlock+0xeb/0x120 [ 2207.126964] softirqs last enabled at (12818174): [] __do_softirq+0x2be/0x371 [ 2207.135656] softirqs last disabled at (12818169): [] irq_exit_rcu+0xb0/0xd0 [ 2207.144175] ---[ end trace 0000000000000000 ]--- [ 2207.148795] ------------[ cut here ]------------ [ 2207.153417] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2207.165083] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2207.174046] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2207.229512] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2207.238483] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2207.245977] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2207.251487] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2207.270254] RSP: 0018:ffffc90001ff7b38 EFLAGS: 00010282 [ 2207.275479] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2207.282629] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2207.289781] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003660 [ 2207.296932] R10: 00000000001531c0 R11: ffffc90001ff79c8 R12: ffff8881c25d1118 [ 2207.304082] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2207.311211] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2207.319312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2207.325070] CR2: 00007fb38de00010 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2207.332205] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2207.339336] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2207.346487] Call Trace: [ 2207.348956] [ 2207.351082] ? __warn+0x85/0x180 [ 2207.354313] ? __list_add_valid_or_report+0x7d/0xa0 [ 2207.359215] ? report_bug+0xfc/0x1d0 [ 2207.362799] ? prb_read_valid+0x1b/0x20 [ 2207.366676] ? handle_bug+0x44/0x70 [ 2207.370164] ? exc_invalid_op+0x17/0x60 [ 2207.374022] ? asm_exc_invalid_op+0x1a/0x20 [ 2207.378217] ? __list_add_valid_or_report+0x7d/0xa0 [ 2207.383099] ? __list_add_valid_or_report+0x7d/0xa0 [ 2207.387976] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2207.392899] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2207.397653] lookup_fast+0x6d/0xf0 [ 2207.401064] walk_component+0x26/0x160 [ 2207.404817] link_path_walk+0x24d/0x380 [ 2207.408673] ? path_lookupat.isra.40+0x22/0x120 [ 2207.413226] path_lookupat.isra.40+0x6b/0x120 [ 2207.417610] filename_lookup+0xd4/0x190 [ 2207.421469] ? lock_acquire+0xcb/0x2b0 [ 2207.425231] ? __might_fault+0x45/0x80 [ 2207.428988] ? find_held_lock+0x2d/0x90 [ 2207.432850] ? __might_fault+0x45/0x80 [ 2207.436643] vfs_statx+0x89/0x150 [ 2207.439982] vfs_fstatat+0x5a/0xc0 [ 2207.443393] __do_sys_newstat+0x30/0x70 [ 2207.447249] ? lock_release+0xc5/0x260 [ 2207.451021] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2207.455897] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2207.461054] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2207.466196] do_syscall_64+0x38/0x80 [ 2207.469773] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2207.474843] RIP: 0033:0x7f48f0eeca79 [ 2207.478438] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2207.497207] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2207.504777] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2207.511929] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2207.519078] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2207.526211] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2207.533342] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2207.540520] [ 2207.542713] irq event stamp: 12877929 [ 2207.546395] hardirqs last enabled at (12877939): [] console_unlock+0x106/0x120 [ 2207.555296] hardirqs last disabled at (12877948): [] console_unlock+0xeb/0x120 [ 2207.564079] softirqs last enabled at (12877962): [] __do_softirq+0x2be/0x371 [ 2207.572777] softirqs last disabled at (12877957): [] irq_exit_rcu+0xb0/0xd0 [ 2207.581318] ---[ end trace 0000000000000000 ]--- [ 2207.585950] ------------[ cut here ]------------ [ 2207.590585] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2207.602241] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2207.610936] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2207.666438] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2207.675581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2207.683070] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2207.688564] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2207.694067] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2207.712815] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2207.718040] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2207.725172] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2207.732313] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000036a2 [ 2207.739455] R10: 00000000001544a0 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2207.746597] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2207.753747] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2207.761841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2207.767598] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2207.774748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2207.781888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2207.789038] Call Trace: [ 2207.791482] [ 2207.793599] ? __warn+0x85/0x180 [ 2207.796850] ? __list_add_valid_or_report+0x7d/0xa0 [ 2207.801728] ? report_bug+0xfc/0x1d0 [ 2207.805325] ? prb_read_valid+0x1b/0x20 [ 2207.809182] ? handle_bug+0x44/0x70 [ 2207.812675] ? exc_invalid_op+0x17/0x60 [ 2207.816531] ? asm_exc_invalid_op+0x1a/0x20 [ 2207.820739] ? __list_add_valid_or_report+0x7d/0xa0 [ 2207.825613] ? __list_add_valid_or_report+0x7d/0xa0 [ 2207.830511] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2207.835418] ceph_add_cap+0x20b/0x620 [ceph] [ 2207.839736] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2207.844468] ? lock_acquire+0xcb/0x2b0 [ 2207.848218] ? lock_is_held_type+0xaa/0x120 [ 2207.852405] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2207.856876] ? __mutex_lock+0x97/0xce0 [ 2207.860645] ? __mutex_lock+0xaec/0xce0 [ 2207.864503] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2207.869044] ? lock_downgrade+0xfd/0x250 [ 2207.872989] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2207.877386] ceph_con_process_message+0x74/0x150 [libceph] [ 2207.882884] ? calc_signature+0xdb/0x100 [libceph] [ 2207.887705] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2207.893045] ? process_scheduled_works+0x22a/0x510 [ 2207.897836] ? process_scheduled_works+0x22a/0x510 [ 2207.902643] ceph_con_workfn+0x344/0x750 [libceph] [ 2207.907465] ? process_scheduled_works+0x22a/0x510 [ 2207.912273] process_scheduled_works+0x2a4/0x510 [ 2207.916914] worker_thread+0x188/0x340 [ 2207.920662] ? rescuer_thread+0x310/0x310 [ 2207.924693] kthread+0xf5/0x130 [ 2207.927855] ? kthread_complete_and_exit+0x20/0x20 [ 2207.932666] ret_from_fork+0x31/0x40 [ 2207.936263] ? kthread_complete_and_exit+0x20/0x20 [ 2207.941055] ret_from_fork_asm+0x11/0x20 [ 2207.945010] [ 2207.947197] irq event stamp: 10348035 [ 2207.950858] hardirqs last enabled at (10348045): [] console_unlock+0x106/0x120 [ 2207.959724] hardirqs last disabled at (10348054): [] console_unlock+0xeb/0x120 [ 2207.968519] softirqs last enabled at (10347970): [] __do_softirq+0x2be/0x371 [ 2207.977229] softirqs last disabled at (10347965): [] irq_exit_rcu+0xb0/0xd0 [ 2207.985749] ---[ end trace 0000000000000000 ]--- [ 2207.990381] ------------[ cut here ]------------ [ 2207.995037] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2208.006700] WARNING: CPU: 7 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2208.015653] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2208.071139] CPU: 7 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2208.080093] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2208.087568] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2208.093055] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2208.111802] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2208.117035] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2208.124167] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2208.131320] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000036e2 [ 2208.138452] R10: 0000000000155730 R11: ffffc90001ff7950 R12: ffff8881c25d1118 [ 2208.145585] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2208.152717] FS: 00007f48f1832740(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2208.160804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.166549] CR2: 00007fb38de00010 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2208.173684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2208.180814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2208.187948] Call Trace: [ 2208.190400] [ 2208.192509] ? __warn+0x85/0x180 [ 2208.195740] ? __list_add_valid_or_report+0x7d/0xa0 [ 2208.200622] ? report_bug+0xfc/0x1d0 [ 2208.204200] ? prb_read_valid+0x1b/0x20 [ 2208.208043] ? handle_bug+0x44/0x70 [ 2208.211531] ? exc_invalid_op+0x17/0x60 [ 2208.215390] ? asm_exc_invalid_op+0x1a/0x20 [ 2208.219583] ? __list_add_valid_or_report+0x7d/0xa0 [ 2208.224464] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2208.229360] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2208.234608] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2208.239413] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2208.244147] lookup_fast+0x6d/0xf0 [ 2208.247550] walk_component+0x26/0x160 [ 2208.251324] link_path_walk+0x24d/0x380 [ 2208.255159] ? path_lookupat.isra.40+0x22/0x120 [ 2208.259700] path_lookupat.isra.40+0x6b/0x120 [ 2208.264062] filename_lookup+0xd4/0x190 [ 2208.267908] ? lock_acquire+0xcb/0x2b0 [ 2208.271662] ? __might_fault+0x45/0x80 [ 2208.275418] ? find_held_lock+0x2d/0x90 [ 2208.279281] ? __might_fault+0x45/0x80 [ 2208.283048] vfs_statx+0x89/0x150 [ 2208.286375] vfs_fstatat+0x5a/0xc0 [ 2208.289789] __do_sys_newstat+0x30/0x70 [ 2208.293633] ? lock_release+0xc5/0x260 [ 2208.297390] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2208.302272] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2208.307414] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2208.312569] do_syscall_64+0x38/0x80 [ 2208.316150] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2208.321201] RIP: 0033:0x7f48f0eeca79 [ 2208.324780] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2208.343528] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2208.351094] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2208.358224] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2208.365358] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2208.372490] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2208.379624] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2208.386770] [ 2208.388960] irq event stamp: 12879975 [ 2208.392623] hardirqs last enabled at (12879985): [] console_unlock+0x106/0x120 [ 2208.401490] hardirqs last disabled at (12879994): [] console_unlock+0xeb/0x120 [ 2208.410272] softirqs last enabled at (12880008): [] __do_softirq+0x2be/0x371 [ 2208.418964] softirqs last disabled at (12880003): [] irq_exit_rcu+0xb0/0xd0 [ 2208.427480] ---[ end trace 0000000000000000 ]--- [ 2208.432105] ------------[ cut here ]------------ [ 2208.436719] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2208.448364] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2208.457061] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2208.512538] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2208.521663] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2208.529143] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2208.534638] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2208.540125] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2208.558870] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2208.564096] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2208.571230] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2208.578361] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003725 [ 2208.585511] R10: 0000000000156a40 R11: ffffc9000030b828 R12: ffff8881c2cefcb0 [ 2208.592645] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2208.599777] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2208.607882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.613629] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2208.620761] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2208.627909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2208.635047] Call Trace: [ 2208.637494] [ 2208.639593] ? __warn+0x85/0x180 [ 2208.642829] ? __list_add_valid_or_report+0x7d/0xa0 [ 2208.647705] ? report_bug+0xfc/0x1d0 [ 2208.651285] ? prb_read_valid+0x1b/0x20 [ 2208.655125] ? handle_bug+0x44/0x70 [ 2208.658615] ? exc_invalid_op+0x17/0x60 [ 2208.662448] ? asm_exc_invalid_op+0x1a/0x20 [ 2208.666639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2208.671513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2208.676392] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2208.681302] ceph_add_cap+0x20b/0x620 [ceph] [ 2208.685584] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2208.690350] ceph_fill_trace+0x143/0xf30 [ceph] [ 2208.694888] ? __mutex_lock+0x97/0xce0 [ 2208.698639] ? __mutex_lock+0xaec/0xce0 [ 2208.702480] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2208.707048] ? lock_downgrade+0xfd/0x250 [ 2208.710977] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2208.715374] ceph_con_process_message+0x74/0x150 [libceph] [ 2208.720887] ? calc_signature+0xdb/0x100 [libceph] [ 2208.725716] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2208.731066] ? process_scheduled_works+0x22a/0x510 [ 2208.735858] ? process_scheduled_works+0x22a/0x510 [ 2208.740647] ceph_con_workfn+0x344/0x750 [libceph] [ 2208.745452] ? process_scheduled_works+0x22a/0x510 [ 2208.750241] process_scheduled_works+0x2a4/0x510 [ 2208.754866] worker_thread+0x188/0x340 [ 2208.758614] ? rescuer_thread+0x310/0x310 [ 2208.762644] kthread+0xf5/0x130 [ 2208.765791] ? kthread_complete_and_exit+0x20/0x20 [ 2208.770586] ret_from_fork+0x31/0x40 [ 2208.774163] ? kthread_complete_and_exit+0x20/0x20 [ 2208.778956] ret_from_fork_asm+0x11/0x20 [ 2208.782887] [ 2208.785074] irq event stamp: 10350069 [ 2208.788740] hardirqs last enabled at (10350079): [] console_unlock+0x106/0x120 [ 2208.797606] hardirqs last disabled at (10350088): [] console_unlock+0xeb/0x120 [ 2208.806386] softirqs last enabled at (10350026): [] __do_softirq+0x2be/0x371 [ 2208.815079] softirqs last disabled at (10350021): [] irq_exit_rcu+0xb0/0xd0 [ 2208.823597] ---[ end trace 0000000000000000 ]--- [ 2208.828228] ------------[ cut here ]------------ [ 2208.832848] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2208.844527] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2208.853221] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2208.908698] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2208.917841] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2208.925353] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2208.930867] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2208.936371] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2208.955136] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2208.960361] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2208.967512] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2208.974662] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003763 [ 2208.981810] R10: 0000000000157c90 R11: ffffc9000030b828 R12: ffff88819f02f2e8 [ 2208.988960] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2208.996113] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2209.004198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2209.009943] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2209.017094] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2209.024225] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2209.031359] Call Trace: [ 2209.033831] [ 2209.035953] ? __warn+0x85/0x180 [ 2209.039204] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.044084] ? report_bug+0xfc/0x1d0 [ 2209.047662] ? prb_read_valid+0x1b/0x20 [ 2209.051520] ? handle_bug+0x44/0x70 [ 2209.055045] ? exc_invalid_op+0x17/0x60 [ 2209.058885] ? asm_exc_invalid_op+0x1a/0x20 [ 2209.063094] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.067969] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.072864] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2209.077772] ceph_add_cap+0x20b/0x620 [ceph] [ 2209.082075] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2209.086807] ? lock_acquire+0xcb/0x2b0 [ 2209.090557] ? lock_is_held_type+0xaa/0x120 [ 2209.094759] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2209.099231] ? __mutex_lock+0x97/0xce0 [ 2209.102982] ? __mutex_lock+0xaec/0xce0 [ 2209.106840] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2209.111398] ? lock_downgrade+0xfd/0x250 [ 2209.115366] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2209.119760] ceph_con_process_message+0x74/0x150 [libceph] [ 2209.125256] ? calc_signature+0xdb/0x100 [libceph] [ 2209.130059] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2209.135401] ? process_scheduled_works+0x22a/0x510 [ 2209.140192] ? process_scheduled_works+0x22a/0x510 [ 2209.144985] ceph_con_workfn+0x344/0x750 [libceph] [ 2209.149813] ? process_scheduled_works+0x22a/0x510 [ 2209.154618] process_scheduled_works+0x2a4/0x510 [ 2209.159261] worker_thread+0x188/0x340 [ 2209.163011] ? rescuer_thread+0x310/0x310 [ 2209.167055] kthread+0xf5/0x130 [ 2209.170201] ? kthread_complete_and_exit+0x20/0x20 [ 2209.174996] ret_from_fork+0x31/0x40 [ 2209.178592] ? kthread_complete_and_exit+0x20/0x20 [ 2209.183400] ret_from_fork_asm+0x11/0x20 [ 2209.187370] [ 2209.189564] irq event stamp: 10351207 [ 2209.193246] hardirqs last enabled at (10351217): [] console_unlock+0x106/0x120 [ 2209.202113] hardirqs last disabled at (10351226): [] console_unlock+0xeb/0x120 [ 2209.210894] softirqs last enabled at (10351240): [] __do_softirq+0x2be/0x371 [ 2209.219635] softirqs last disabled at (10351253): [] irq_exit_rcu+0xb0/0xd0 [ 2209.228159] ---[ end trace 0000000000000000 ]--- [ 2209.232778] ------------[ cut here ]------------ [ 2209.237477] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2209.249130] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2209.258084] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2209.313556] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2209.322503] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2209.329973] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2209.335463] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2209.354206] RSP: 0018:ffffc90001fd7a78 EFLAGS: 00010286 [ 2209.359432] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2209.366556] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2209.373689] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000037a3 [ 2209.380824] R10: 0000000000158ee8 R11: ffffc90001fd7908 R12: ffff8881c2cefcb0 [ 2209.387954] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2209.395079] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2209.403165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2209.408911] CR2: 00007fb38d519068 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2209.416044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2209.423169] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2209.430301] Call Trace: [ 2209.432755] [ 2209.434860] ? __warn+0x85/0x180 [ 2209.438093] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.442974] ? report_bug+0xfc/0x1d0 [ 2209.446554] ? prb_read_valid+0x1b/0x20 [ 2209.450393] ? handle_bug+0x44/0x70 [ 2209.453885] ? exc_invalid_op+0x17/0x60 [ 2209.457723] ? asm_exc_invalid_op+0x1a/0x20 [ 2209.461915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.466790] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.471668] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2209.476558] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2209.481794] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2209.486593] ceph_lookup+0xf9/0x3f0 [ceph] [ 2209.490705] __lookup_slow+0xff/0x1a0 [ 2209.494375] walk_component+0x83/0x160 [ 2209.498130] path_lookupat.isra.40+0x58/0x120 [ 2209.502491] filename_lookup+0xd4/0x190 [ 2209.506332] ? lock_acquire+0xcb/0x2b0 [ 2209.510082] ? __might_fault+0x45/0x80 [ 2209.513833] ? find_held_lock+0x2d/0x90 [ 2209.517673] ? __might_fault+0x45/0x80 [ 2209.521428] vfs_statx+0x89/0x150 [ 2209.524747] vfs_fstatat+0x5a/0xc0 [ 2209.528152] __do_sys_newstat+0x30/0x70 [ 2209.531991] ? lock_release+0xc5/0x260 [ 2209.535744] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2209.540621] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2209.545761] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2209.550901] do_syscall_64+0x38/0x80 [ 2209.554479] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2209.559534] RIP: 0033:0x7f48f0eeca79 [ 2209.563112] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2209.581858] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2209.589425] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2209.596559] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2209.603689] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2209.610814] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2209.617946] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2209.625086] [ 2209.627272] irq event stamp: 12940733 [ 2209.630930] hardirqs last enabled at (12940743): [] console_unlock+0x106/0x120 [ 2209.639799] hardirqs last disabled at (12940752): [] console_unlock+0xeb/0x120 [ 2209.648576] softirqs last enabled at (12940766): [] __do_softirq+0x2be/0x371 [ 2209.657269] softirqs last disabled at (12940761): [] irq_exit_rcu+0xb0/0xd0 [ 2209.665788] ---[ end trace 0000000000000000 ]--- [ 2209.670408] ------------[ cut here ]------------ [ 2209.675029] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2209.686699] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2209.695394] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2209.750899] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2209.760059] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2209.767536] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2209.773067] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2209.778552] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2209.797316] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2209.802543] RAX: 0000000000000000 RBX: ffff8881cccb8000 RCX: 0000000000000000 [ 2209.809694] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2209.816863] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000037e5 [ 2209.823995] R10: 000000000015a1b8 R11: ffffc9000030b828 R12: ffff8881cccb8750 [ 2209.831144] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2209.838277] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2209.846362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2209.852127] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2209.859257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2209.866390] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2209.873540] Call Trace: [ 2209.876013] [ 2209.878138] ? __warn+0x85/0x180 [ 2209.881389] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.886266] ? report_bug+0xfc/0x1d0 [ 2209.889847] ? prb_read_valid+0x1b/0x20 [ 2209.893721] ? handle_bug+0x44/0x70 [ 2209.897211] ? exc_invalid_op+0x17/0x60 [ 2209.901069] ? asm_exc_invalid_op+0x1a/0x20 [ 2209.905262] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.910145] ? __list_add_valid_or_report+0x7d/0xa0 [ 2209.915021] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2209.919940] ceph_add_cap+0x20b/0x620 [ceph] [ 2209.924236] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2209.928977] ? lock_acquire+0xcb/0x2b0 [ 2209.932748] ? lock_is_held_type+0xaa/0x120 [ 2209.936951] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2209.941404] ? __mutex_lock+0x97/0xce0 [ 2209.945175] ? __mutex_lock+0xaec/0xce0 [ 2209.949033] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2209.953596] ? lock_downgrade+0xfd/0x250 [ 2209.957533] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2209.961914] ceph_con_process_message+0x74/0x150 [libceph] [ 2209.967412] ? calc_signature+0xdb/0x100 [libceph] [ 2209.972224] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2209.977546] ? process_scheduled_works+0x22a/0x510 [ 2209.982339] ? process_scheduled_works+0x22a/0x510 [ 2209.987127] ceph_con_workfn+0x344/0x750 [libceph] [ 2209.991936] ? process_scheduled_works+0x22a/0x510 [ 2209.996732] process_scheduled_works+0x2a4/0x510 [ 2210.001361] worker_thread+0x188/0x340 [ 2210.005118] ? rescuer_thread+0x310/0x310 [ 2210.009138] kthread+0xf5/0x130 [ 2210.012282] ? kthread_complete_and_exit+0x20/0x20 [ 2210.017075] ret_from_fork+0x31/0x40 [ 2210.020653] ? kthread_complete_and_exit+0x20/0x20 [ 2210.025446] ret_from_fork_asm+0x11/0x20 [ 2210.029382] [ 2210.031573] irq event stamp: 10353305 [ 2210.035239] hardirqs last enabled at (10353315): [] console_unlock+0x106/0x120 [ 2210.044105] hardirqs last disabled at (10353324): [] console_unlock+0xeb/0x120 [ 2210.052884] softirqs last enabled at (10353242): [] __do_softirq+0x2be/0x371 [ 2210.061577] softirqs last disabled at (10353237): [] irq_exit_rcu+0xb0/0xd0 [ 2210.070096] ---[ end trace 0000000000000000 ]--- [ 2210.074719] ------------[ cut here ]------------ [ 2210.079351] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2210.091016] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2210.099969] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2210.155458] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2210.164425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2210.171923] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2210.177426] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2210.196188] RSP: 0018:ffffc90001fefbf0 EFLAGS: 00010282 [ 2210.201413] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2210.208556] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2210.215705] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003825 [ 2210.222849] R10: 000000000015b330 R11: ffffc90001fefa80 R12: ffff888158a68750 [ 2210.229999] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2210.237147] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2210.245225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2210.250972] CR2: 00007f9bd26dd000 CR3: 0000000108c72004 CR4: 00000000003706f0 [ 2210.258112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2210.265237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2210.272374] Call Trace: [ 2210.274840] [ 2210.276954] ? __warn+0x85/0x180 [ 2210.280204] ? __list_add_valid_or_report+0x7d/0xa0 [ 2210.285087] ? report_bug+0xfc/0x1d0 [ 2210.288665] ? prb_read_valid+0x1b/0x20 [ 2210.292523] ? handle_bug+0x44/0x70 [ 2210.296031] ? exc_invalid_op+0x17/0x60 [ 2210.299887] ? asm_exc_invalid_op+0x1a/0x20 [ 2210.304096] ? __list_add_valid_or_report+0x7d/0xa0 [ 2210.308972] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2210.313877] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2210.318628] lookup_fast+0x6d/0xf0 [ 2210.322030] walk_component+0x26/0x160 [ 2210.325793] link_path_walk+0x24d/0x380 [ 2210.329650] ? path_openat+0x80/0x9d0 [ 2210.333334] path_openat+0x8e/0x9d0 [ 2210.336847] do_filp_open+0xb8/0x120 [ 2210.340462] ? lock_release+0xc5/0x260 [ 2210.344210] ? _raw_spin_unlock+0x23/0x40 [ 2210.348223] ? alloc_fd+0x13f/0x200 [ 2210.351718] do_sys_openat2+0x249/0x290 [ 2210.355575] do_sys_open+0x4f/0x80 [ 2210.358996] do_syscall_64+0x38/0x80 [ 2210.362591] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2210.367663] RIP: 0033:0x7f48f0ef2805 [ 2210.371261] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2210.390004] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2210.397589] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2210.404765] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2210.411924] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2210.419078] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2210.426205] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2210.433362] [ 2210.435566] irq event stamp: 12813027 [ 2210.439241] hardirqs last enabled at (12813037): [] console_unlock+0x106/0x120 [ 2210.448106] hardirqs last disabled at (12813046): [] console_unlock+0xeb/0x120 [ 2210.456887] softirqs last enabled at (12812994): [] __do_softirq+0x2be/0x371 [ 2210.465596] softirqs last disabled at (12812985): [] irq_exit_rcu+0xb0/0xd0 [ 2210.474135] ---[ end trace 0000000000000000 ]--- [ 2210.478753] ------------[ cut here ]------------ [ 2210.483393] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2210.495062] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2210.503757] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2210.559240] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2210.568371] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2210.575866] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2210.581387] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2210.586891] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2210.605654] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2210.610897] RAX: 0000000000000000 RBX: ffff8881a1d33ab0 RCX: 0000000000000000 [ 2210.618069] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2210.625209] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003860 [ 2210.632339] R10: 000000000015c3c8 R11: ffffc9000030b828 R12: ffff8881a1d34200 [ 2210.639471] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2210.646622] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2210.654725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2210.660509] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2210.667641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2210.674789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2210.681938] Call Trace: [ 2210.684400] [ 2210.686514] ? __warn+0x85/0x180 [ 2210.689766] ? __list_add_valid_or_report+0x7d/0xa0 [ 2210.694662] ? report_bug+0xfc/0x1d0 [ 2210.698259] ? prb_read_valid+0x1b/0x20 [ 2210.702099] ? handle_bug+0x44/0x70 [ 2210.705591] ? exc_invalid_op+0x17/0x60 [ 2210.709449] ? asm_exc_invalid_op+0x1a/0x20 [ 2210.713672] ? __list_add_valid_or_report+0x7d/0xa0 [ 2210.718548] ? __list_add_valid_or_report+0x7d/0xa0 [ 2210.723443] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2210.728352] ceph_add_cap+0x20b/0x620 [ceph] [ 2210.732655] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2210.737400] ceph_fill_trace+0x143/0xf30 [ceph] [ 2210.741941] ? __mutex_lock+0x97/0xce0 [ 2210.745707] ? __mutex_lock+0xaec/0xce0 [ 2210.749558] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2210.754118] ? lock_downgrade+0xfd/0x250 [ 2210.758071] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2210.762442] ceph_con_process_message+0x74/0x150 [libceph] [ 2210.767939] ? calc_signature+0xdb/0x100 [libceph] [ 2210.772779] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2210.778111] ? process_scheduled_works+0x22a/0x510 [ 2210.782911] ? process_scheduled_works+0x22a/0x510 [ 2210.787717] ceph_con_workfn+0x344/0x750 [libceph] [ 2210.792539] ? process_scheduled_works+0x22a/0x510 [ 2210.797346] process_scheduled_works+0x2a4/0x510 [ 2210.802006] worker_thread+0x188/0x340 [ 2210.805755] ? rescuer_thread+0x310/0x310 [ 2210.809782] kthread+0xf5/0x130 [ 2210.812946] ? kthread_complete_and_exit+0x20/0x20 [ 2210.817760] ret_from_fork+0x31/0x40 [ 2210.821368] ? kthread_complete_and_exit+0x20/0x20 [ 2210.826172] ret_from_fork_asm+0x11/0x20 [ 2210.830105] [ 2210.832290] irq event stamp: 10355247 [ 2210.835948] hardirqs last enabled at (10355257): [] console_unlock+0x106/0x120 [ 2210.844830] hardirqs last disabled at (10355266): [] console_unlock+0xeb/0x120 [ 2210.853619] softirqs last enabled at (10355204): [] __do_softirq+0x2be/0x371 [ 2210.862331] softirqs last disabled at (10355199): [] irq_exit_rcu+0xb0/0xd0 [ 2210.870850] ---[ end trace 0000000000000000 ]--- [ 2210.875487] ------------[ cut here ]------------ [ 2210.880124] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2210.891777] WARNING: CPU: 3 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2210.900737] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2210.956264] CPU: 3 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2210.965217] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2210.972696] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2210.978187] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2210.996937] RSP: 0018:ffffc90001fefbf0 EFLAGS: 00010282 [ 2211.002163] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2211.009296] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2211.016427] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000389e [ 2211.023562] R10: 000000000015d510 R11: ffffc90001fefa80 R12: ffff8881c5350750 [ 2211.030693] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2211.037827] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2211.045913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2211.051660] CR2: 00007f9bd2958000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2211.058793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2211.065925] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2211.073061] Call Trace: [ 2211.075511] [ 2211.077618] ? __warn+0x85/0x180 [ 2211.080851] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.085731] ? report_bug+0xfc/0x1d0 [ 2211.089318] ? prb_read_valid+0x1b/0x20 [ 2211.093159] ? handle_bug+0x44/0x70 [ 2211.096650] ? exc_invalid_op+0x17/0x60 [ 2211.100490] ? asm_exc_invalid_op+0x1a/0x20 [ 2211.104685] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.109569] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2211.114473] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2211.119199] lookup_fast+0x6d/0xf0 [ 2211.122608] walk_component+0x26/0x160 [ 2211.126364] link_path_walk+0x24d/0x380 [ 2211.130209] ? path_openat+0x80/0x9d0 [ 2211.133877] path_openat+0x8e/0x9d0 [ 2211.137382] do_filp_open+0xb8/0x120 [ 2211.140971] ? lock_release+0xc5/0x260 [ 2211.144729] ? _raw_spin_unlock+0x23/0x40 [ 2211.148746] ? alloc_fd+0x13f/0x200 [ 2211.152245] do_sys_openat2+0x249/0x290 [ 2211.156089] do_sys_open+0x4f/0x80 [ 2211.159503] do_syscall_64+0x38/0x80 [ 2211.163090] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2211.168143] RIP: 0033:0x7f48f0ef2805 [ 2211.171721] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2211.190467] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2211.198034] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2211.205167] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2211.212299] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2211.219433] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2211.226564] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2211.233710] [ 2211.235906] irq event stamp: 12816549 [ 2211.239574] hardirqs last enabled at (12816559): [] console_unlock+0x106/0x120 [ 2211.248440] hardirqs last disabled at (12816568): [] console_unlock+0xeb/0x120 [ 2211.257219] softirqs last enabled at (12816516): [] __do_softirq+0x2be/0x371 [ 2211.265911] softirqs last disabled at (12816511): [] irq_exit_rcu+0xb0/0xd0 [ 2211.274431] ---[ end trace 0000000000000000 ]--- [ 2211.279053] ------------[ cut here ]------------ [ 2211.283670] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2211.295318] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2211.304273] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2211.359754] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2211.368720] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2211.376197] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2211.381695] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2211.400439] RSP: 0018:ffffc90001fffa78 EFLAGS: 00010286 [ 2211.405666] RAX: 0000000000000000 RBX: ffff8881a1d33ab0 RCX: 0000000000000000 [ 2211.412796] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2211.419931] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000038d9 [ 2211.427072] R10: 000000000015e2d0 R11: ffffc90001fff908 R12: ffff8881a1d34200 [ 2211.434207] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2211.441336] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2211.449424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2211.455169] CR2: 00007fb38d534050 CR3: 000000017eb7a003 CR4: 00000000003706e0 [ 2211.462303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2211.469436] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2211.476577] Call Trace: [ 2211.479030] [ 2211.481138] ? __warn+0x85/0x180 [ 2211.484386] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.489271] ? report_bug+0xfc/0x1d0 [ 2211.492856] ? prb_read_valid+0x1b/0x20 [ 2211.496705] ? handle_bug+0x44/0x70 [ 2211.500205] ? exc_invalid_op+0x17/0x60 [ 2211.504053] ? asm_exc_invalid_op+0x1a/0x20 [ 2211.508252] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.513139] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.518022] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2211.522923] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2211.528166] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2211.532993] ceph_lookup+0xf9/0x3f0 [ceph] [ 2211.537111] __lookup_slow+0xff/0x1a0 [ 2211.540793] walk_component+0x83/0x160 [ 2211.544552] path_lookupat.isra.40+0x58/0x120 [ 2211.548915] filename_lookup+0xd4/0x190 [ 2211.552764] ? lock_acquire+0xcb/0x2b0 [ 2211.556523] ? __might_fault+0x45/0x80 [ 2211.560283] ? find_held_lock+0x2d/0x90 [ 2211.564122] ? __might_fault+0x45/0x80 [ 2211.567886] vfs_statx+0x89/0x150 [ 2211.571216] vfs_fstatat+0x5a/0xc0 [ 2211.574626] __do_sys_newstat+0x30/0x70 [ 2211.578469] ? lock_release+0xc5/0x260 [ 2211.582229] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2211.587113] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2211.592252] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2211.597396] do_syscall_64+0x38/0x80 [ 2211.600980] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2211.606042] RIP: 0033:0x7f48f0eeca79 [ 2211.609619] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2211.628370] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2211.635940] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2211.643077] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2211.650206] RBP: 00007ffc53d60520 R08: 0000000000002a10 R09: 00000000020767c0 [ 2211.657342] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2211.664471] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2211.671615] [ 2211.673806] irq event stamp: 12858477 [ 2211.677474] hardirqs last enabled at (12858487): [] console_unlock+0x106/0x120 [ 2211.686340] hardirqs last disabled at (12858506): [] console_unlock+0xeb/0x120 [ 2211.695121] softirqs last enabled at (12858520): [] __do_softirq+0x2be/0x371 [ 2211.703821] softirqs last disabled at (12858515): [] irq_exit_rcu+0xb0/0xd0 [ 2211.712339] ---[ end trace 0000000000000000 ]--- [ 2211.716969] ------------[ cut here ]------------ [ 2211.721586] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2211.733232] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2211.741927] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2211.797406] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2211.806529] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2211.814009] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2211.819504] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2211.824989] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2211.843739] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2211.848962] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2211.856095] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2211.863230] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000391b [ 2211.870361] R10: 000000000015f578 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2211.877492] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2211.884624] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2211.892713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2211.898458] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2211.905592] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2211.912730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2211.919864] Call Trace: [ 2211.922309] [ 2211.924417] ? __warn+0x85/0x180 [ 2211.927651] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.932529] ? report_bug+0xfc/0x1d0 [ 2211.936115] ? prb_read_valid+0x1b/0x20 [ 2211.939963] ? handle_bug+0x44/0x70 [ 2211.943462] ? exc_invalid_op+0x17/0x60 [ 2211.947308] ? asm_exc_invalid_op+0x1a/0x20 [ 2211.951505] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.956392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2211.961279] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2211.966178] ceph_add_cap+0x20b/0x620 [ceph] [ 2211.970486] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2211.975214] ? lock_acquire+0xcb/0x2b0 [ 2211.978973] ? lock_is_held_type+0xaa/0x120 [ 2211.983165] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2211.987628] ? __mutex_lock+0x97/0xce0 [ 2211.991389] ? __mutex_lock+0xaec/0xce0 [ 2211.995230] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2211.999776] ? lock_downgrade+0xfd/0x250 [ 2212.003714] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2212.008102] ceph_con_process_message+0x74/0x150 [libceph] [ 2212.013608] ? calc_signature+0xdb/0x100 [libceph] [ 2212.018423] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2212.023765] ? process_scheduled_works+0x22a/0x510 [ 2212.028564] ? process_scheduled_works+0x22a/0x510 [ 2212.033361] ceph_con_workfn+0x344/0x750 [libceph] [ 2212.038174] ? process_scheduled_works+0x22a/0x510 [ 2212.042972] process_scheduled_works+0x2a4/0x510 [ 2212.047608] worker_thread+0x188/0x340 [ 2212.051365] ? rescuer_thread+0x310/0x310 [ 2212.055385] kthread+0xf5/0x130 [ 2212.058540] ? kthread_complete_and_exit+0x20/0x20 [ 2212.063340] ret_from_fork+0x31/0x40 [ 2212.066919] ? kthread_complete_and_exit+0x20/0x20 [ 2212.071721] ret_from_fork_asm+0x11/0x20 [ 2212.075657] [ 2212.077846] irq event stamp: 10358189 [ 2212.081513] hardirqs last enabled at (10358199): [] console_unlock+0x106/0x120 [ 2212.090388] hardirqs last disabled at (10358208): [] console_unlock+0xeb/0x120 [ 2212.099171] softirqs last enabled at (10358222): [] __do_softirq+0x2be/0x371 [ 2212.107868] softirqs last disabled at (10358217): [] irq_exit_rcu+0xb0/0xd0 [ 2212.116388] ---[ end trace 0000000000000000 ]--- [ 2212.121010] ------------[ cut here ]------------ [ 2212.125632] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2212.137278] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2212.146233] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2212.201697] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2212.210644] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2212.218115] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2212.223602] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2212.242349] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2212.247574] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2212.254706] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2212.261831] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000395b [ 2212.268965] R10: 00000000001606f8 R11: ffffc90001fefb18 R12: ffff88819f028750 [ 2212.276097] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2212.283231] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2212.291315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2212.297062] CR2: 00007f9bd2962018 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2212.304193] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2212.311318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2212.318450] Call Trace: [ 2212.320897] [ 2212.323002] ? __warn+0x85/0x180 [ 2212.326236] ? __list_add_valid_or_report+0x7d/0xa0 [ 2212.331115] ? report_bug+0xfc/0x1d0 [ 2212.334694] ? prb_read_valid+0x1b/0x20 [ 2212.338534] ? handle_bug+0x44/0x70 [ 2212.342024] ? exc_invalid_op+0x17/0x60 [ 2212.345858] ? asm_exc_invalid_op+0x1a/0x20 [ 2212.350046] ? __list_add_valid_or_report+0x7d/0xa0 [ 2212.354921] ? __list_add_valid_or_report+0x7d/0xa0 [ 2212.359800] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2212.364691] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2212.369409] lookup_fast+0x6d/0xf0 [ 2212.372812] path_openat+0xe4/0x9d0 [ 2212.376306] do_filp_open+0xb8/0x120 [ 2212.379885] ? lock_release+0xc5/0x260 [ 2212.383637] ? _raw_spin_unlock+0x23/0x40 [ 2212.387649] ? alloc_fd+0x13f/0x200 [ 2212.391143] do_sys_openat2+0x249/0x290 [ 2212.394984] do_sys_open+0x4f/0x80 [ 2212.398387] do_syscall_64+0x38/0x80 [ 2212.401965] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2212.407019] RIP: 0033:0x7f48f0ef2805 [ 2212.410597] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2212.429344] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2212.436910] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2212.444042] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2212.451174] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2212.458300] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2212.465432] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2212.472570] [ 2212.474758] irq event stamp: 12819333 [ 2212.478424] hardirqs last enabled at (12819343): [] console_unlock+0x106/0x120 [ 2212.487289] hardirqs last disabled at (12819352): [] console_unlock+0xeb/0x120 [ 2212.496062] softirqs last enabled at (12819300): [] __do_softirq+0x2be/0x371 [ 2212.504754] softirqs last disabled at (12819295): [] irq_exit_rcu+0xb0/0xd0 [ 2212.513272] ---[ end trace 0000000000000000 ]--- [ 2212.517884] ------------[ cut here ]------------ [ 2212.522502] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2212.534150] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2212.542845] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2212.598324] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2212.607447] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2212.614929] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2212.620432] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2212.625916] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2212.644664] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2212.649890] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2212.657022] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2212.664157] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003994 [ 2212.671288] R10: 00000000001616e8 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2212.678421] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2212.685552] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2212.693638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2212.699385] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2212.706521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2212.713653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2212.720783] Call Trace: [ 2212.723228] [ 2212.725335] ? __warn+0x85/0x180 [ 2212.728568] ? __list_add_valid_or_report+0x7d/0xa0 [ 2212.733446] ? report_bug+0xfc/0x1d0 [ 2212.737026] ? prb_read_valid+0x1b/0x20 [ 2212.740866] ? handle_bug+0x44/0x70 [ 2212.744358] ? exc_invalid_op+0x17/0x60 [ 2212.748199] ? asm_exc_invalid_op+0x1a/0x20 [ 2212.752388] ? __list_add_valid_or_report+0x7d/0xa0 [ 2212.757264] ? __list_add_valid_or_report+0x7d/0xa0 [ 2212.762141] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2212.767033] ceph_add_cap+0x20b/0x620 [ceph] [ 2212.771318] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2212.776044] ? lock_is_held_type+0xaa/0x120 [ 2212.780233] ceph_fill_trace+0x143/0xf30 [ceph] [ 2212.784773] ? __mutex_lock+0x97/0xce0 [ 2212.788524] ? __mutex_lock+0xaec/0xce0 [ 2212.792367] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2212.796914] ? lock_downgrade+0xfd/0x250 [ 2212.800845] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2212.805224] ceph_con_process_message+0x74/0x150 [libceph] [ 2212.810720] ? calc_signature+0xdb/0x100 [libceph] [ 2212.815534] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2212.820866] ? process_scheduled_works+0x22a/0x510 [ 2212.825664] ? process_scheduled_works+0x22a/0x510 [ 2212.830454] ceph_con_workfn+0x344/0x750 [libceph] [ 2212.835257] ? process_scheduled_works+0x22a/0x510 [ 2212.840049] process_scheduled_works+0x2a4/0x510 [ 2212.844674] worker_thread+0x188/0x340 [ 2212.848429] ? rescuer_thread+0x310/0x310 [ 2212.852442] kthread+0xf5/0x130 [ 2212.855588] ? kthread_complete_and_exit+0x20/0x20 [ 2212.860382] ret_from_fork+0x31/0x40 [ 2212.863959] ? kthread_complete_and_exit+0x20/0x20 [ 2212.868752] ret_from_fork_asm+0x11/0x20 [ 2212.872686] [ 2212.874880] irq event stamp: 10360139 [ 2212.878545] hardirqs last enabled at (10360149): [] console_unlock+0x106/0x120 [ 2212.887412] hardirqs last disabled at (10360158): [] console_unlock+0xeb/0x120 [ 2212.896191] softirqs last enabled at (10360126): [] __do_softirq+0x2be/0x371 [ 2212.904883] softirqs last disabled at (10360121): [] irq_exit_rcu+0xb0/0xd0 [ 2212.913404] ---[ end trace 0000000000000000 ]--- [ 2212.918024] ------------[ cut here ]------------ [ 2212.922644] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2212.934292] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2212.943247] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2212.998786] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2213.007760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2213.015242] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2213.020728] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2213.039487] RSP: 0018:ffffc90001fefd50 EFLAGS: 00010286 [ 2213.044723] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2213.051860] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2213.058993] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000039d3 [ 2213.066125] R10: 0000000000162800 R11: ffffc90001fefbe0 R12: ffff88819f02f2e8 [ 2213.073260] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2213.080392] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2213.088477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2213.094224] CR2: 00007f9bd2971000 CR3: 0000000108c72004 CR4: 00000000003706f0 [ 2213.101356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2213.108489] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2213.115633] Call Trace: [ 2213.118099] [ 2213.120207] ? __warn+0x85/0x180 [ 2213.123441] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.128329] ? report_bug+0xfc/0x1d0 [ 2213.131915] ? prb_read_valid+0x1b/0x20 [ 2213.135759] ? handle_bug+0x44/0x70 [ 2213.139258] ? exc_invalid_op+0x17/0x60 [ 2213.143100] ? asm_exc_invalid_op+0x1a/0x20 [ 2213.147298] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.152180] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2213.157082] ceph_readdir+0xc0/0x1670 [ceph] [ 2213.161376] ? wrap_directory_iterator+0x38/0x70 [ 2213.166003] ? wrap_directory_iterator+0x30/0x70 [ 2213.170622] ? lock_is_held_type+0xaa/0x120 [ 2213.174813] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2213.179705] wrap_directory_iterator+0x47/0x70 [ 2213.184154] iterate_dir+0x8d/0x170 [ 2213.187654] __x64_sys_getdents64+0x87/0x140 [ 2213.191934] ? __x64_sys_getdents64+0x140/0x140 [ 2213.196472] do_syscall_64+0x38/0x80 [ 2213.200054] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2213.205109] RIP: 0033:0x7f48f0ec480b [ 2213.208694] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2213.227441] RSP: 002b:00007ffc53d5ff88 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2213.235006] RAX: ffffffffffffffda RBX: 000000000207e100 RCX: 00007f48f0ec480b [ 2213.242139] RDX: 0000000000010000 RSI: 000000000207e130 RDI: 0000000000000019 [ 2213.249272] RBP: 000000000207e130 R08: 00000000000000db R09: 0000000000000003 [ 2213.256406] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2213.263538] R13: 0000000000000002 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2213.270688] [ 2213.272880] irq event stamp: 12821333 [ 2213.276548] hardirqs last enabled at (12821343): [] console_unlock+0x106/0x120 [ 2213.285420] hardirqs last disabled at (12821352): [] console_unlock+0xeb/0x120 [ 2213.294201] softirqs last enabled at (12821320): [] __do_softirq+0x2be/0x371 [ 2213.302895] softirqs last disabled at (12821315): [] irq_exit_rcu+0xb0/0xd0 [ 2213.311421] ---[ end trace 0000000000000000 ]--- [ 2213.316041] ------------[ cut here ]------------ [ 2213.320661] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2213.332304] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2213.341258] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2213.396723] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2213.405683] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2213.413161] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2213.418644] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2213.437392] RSP: 0018:ffffc90001fc7b98 EFLAGS: 00010282 [ 2213.442616] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2213.449740] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2213.456865] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003a0b [ 2213.463989] R10: 00000000001637b0 R11: ffffc90001fc7a28 R12: ffff8881d800c200 [ 2213.471122] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2213.478255] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2213.486341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2213.492086] CR2: 00007fb38d534050 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2213.499213] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2213.506349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2213.513481] Call Trace: [ 2213.515930] [ 2213.518036] ? __warn+0x85/0x180 [ 2213.521270] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.526149] ? report_bug+0xfc/0x1d0 [ 2213.529728] ? prb_read_valid+0x1b/0x20 [ 2213.533569] ? handle_bug+0x44/0x70 [ 2213.537064] ? exc_invalid_op+0x17/0x60 [ 2213.540899] ? asm_exc_invalid_op+0x1a/0x20 [ 2213.545089] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.549964] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.554843] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2213.559733] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2213.564451] lookup_fast+0x6d/0xf0 [ 2213.567853] walk_component+0x26/0x160 [ 2213.571607] path_lookupat.isra.40+0x58/0x120 [ 2213.575967] filename_lookup+0xd4/0x190 [ 2213.579813] ? lock_acquire+0xcb/0x2b0 [ 2213.583566] ? __might_fault+0x45/0x80 [ 2213.587318] ? find_held_lock+0x2d/0x90 [ 2213.591159] ? __might_fault+0x45/0x80 [ 2213.594912] vfs_statx+0x89/0x150 [ 2213.598232] vfs_fstatat+0x5a/0xc0 [ 2213.601637] __do_sys_newstat+0x30/0x70 [ 2213.605475] ? lock_release+0xc5/0x260 [ 2213.609232] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2213.614112] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2213.619249] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2213.624388] do_syscall_64+0x38/0x80 [ 2213.627967] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2213.633017] RIP: 0033:0x7f48f0eeca79 [ 2213.636591] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2213.655338] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2213.662905] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2213.670034] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2213.677158] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2213.684290] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2213.691414] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2213.698545] [ 2213.700732] irq event stamp: 12923187 [ 2213.704396] hardirqs last enabled at (12923197): [] console_unlock+0x106/0x120 [ 2213.713265] hardirqs last disabled at (12923216): [] console_unlock+0xeb/0x120 [ 2213.722046] softirqs last enabled at (12923230): [] __do_softirq+0x2be/0x371 [ 2213.730736] softirqs last disabled at (12923225): [] irq_exit_rcu+0xb0/0xd0 [ 2213.739257] ---[ end trace 0000000000000000 ]--- [ 2213.743876] ------------[ cut here ]------------ [ 2213.748497] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2213.760141] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2213.768839] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2213.824317] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2213.833440] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2213.840919] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2213.846414] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2213.851899] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2213.870647] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2213.875873] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2213.883006] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2213.890140] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003a4b [ 2213.897271] R10: 0000000000164a98 R11: ffffc9000030b828 R12: ffff8881d106cbc8 [ 2213.904403] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2213.911537] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2213.919624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2213.925369] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2213.932502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2213.939634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2213.946766] Call Trace: [ 2213.949222] [ 2213.951326] ? __warn+0x85/0x180 [ 2213.954562] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.959439] ? report_bug+0xfc/0x1d0 [ 2213.963016] ? prb_read_valid+0x1b/0x20 [ 2213.966860] ? handle_bug+0x44/0x70 [ 2213.970349] ? exc_invalid_op+0x17/0x60 [ 2213.974190] ? asm_exc_invalid_op+0x1a/0x20 [ 2213.978379] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.983255] ? __list_add_valid_or_report+0x7d/0xa0 [ 2213.988135] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2213.993024] ceph_add_cap+0x20b/0x620 [ceph] [ 2213.997311] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2214.002032] ? lock_acquire+0xcb/0x2b0 [ 2214.005783] ? lock_is_held_type+0xaa/0x120 [ 2214.009969] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2214.014422] ? __mutex_lock+0x97/0xce0 [ 2214.018176] ? __mutex_lock+0xaec/0xce0 [ 2214.022015] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2214.026553] ? lock_downgrade+0xfd/0x250 [ 2214.030485] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2214.034867] ceph_con_process_message+0x74/0x150 [libceph] [ 2214.040361] ? calc_signature+0xdb/0x100 [libceph] [ 2214.045173] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2214.050504] ? process_scheduled_works+0x22a/0x510 [ 2214.055295] ? process_scheduled_works+0x22a/0x510 [ 2214.060086] ceph_con_workfn+0x344/0x750 [libceph] [ 2214.064890] ? process_scheduled_works+0x22a/0x510 [ 2214.069681] process_scheduled_works+0x2a4/0x510 [ 2214.074305] worker_thread+0x188/0x340 [ 2214.078063] ? rescuer_thread+0x310/0x310 [ 2214.082075] kthread+0xf5/0x130 [ 2214.085219] ? kthread_complete_and_exit+0x20/0x20 [ 2214.090013] ret_from_fork+0x31/0x40 [ 2214.093590] ? kthread_complete_and_exit+0x20/0x20 [ 2214.098384] ret_from_fork_asm+0x11/0x20 [ 2214.102317] [ 2214.104503] irq event stamp: 10363027 [ 2214.108169] hardirqs last enabled at (10363037): [] console_unlock+0x106/0x120 [ 2214.117036] hardirqs last disabled at (10363046): [] console_unlock+0xeb/0x120 [ 2214.125816] softirqs last enabled at (10362964): [] __do_softirq+0x2be/0x371 [ 2214.134507] softirqs last disabled at (10362959): [] irq_exit_rcu+0xb0/0xd0 [ 2214.143026] ---[ end trace 0000000000000000 ]--- [ 2214.147650] ------------[ cut here ]------------ [ 2214.152274] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2214.163919] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2214.172875] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2214.228339] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2214.237287] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2214.244765] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2214.250252] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2214.269000] RSP: 0018:ffffc90001fc7b20 EFLAGS: 00010282 [ 2214.274226] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2214.281360] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2214.288489] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003a8b [ 2214.295615] R10: 0000000000165b80 R11: ffffc90001fc79b0 R12: ffff8881d800c200 [ 2214.302747] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2214.309880] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2214.317967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2214.323711] CR2: 00007f9bd2981028 CR3: 0000000113500005 CR4: 00000000003706e0 [ 2214.330836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2214.337969] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2214.345102] Call Trace: [ 2214.347546] [ 2214.349653] ? __warn+0x85/0x180 [ 2214.352886] ? __list_add_valid_or_report+0x7d/0xa0 [ 2214.357766] ? report_bug+0xfc/0x1d0 [ 2214.361346] ? prb_read_valid+0x1b/0x20 [ 2214.365185] ? handle_bug+0x44/0x70 [ 2214.368676] ? exc_invalid_op+0x17/0x60 [ 2214.372516] ? asm_exc_invalid_op+0x1a/0x20 [ 2214.376706] ? __list_add_valid_or_report+0x7d/0xa0 [ 2214.381584] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2214.386471] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2214.391707] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2214.396507] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2214.401225] lookup_fast+0x6d/0xf0 [ 2214.404627] walk_component+0x26/0x160 [ 2214.408380] path_lookupat.isra.40+0x58/0x120 [ 2214.412740] filename_lookup+0xd4/0x190 [ 2214.416581] ? lock_acquire+0xcb/0x2b0 [ 2214.420331] ? __might_fault+0x45/0x80 [ 2214.424085] ? find_held_lock+0x2d/0x90 [ 2214.427924] ? __might_fault+0x45/0x80 [ 2214.431678] vfs_statx+0x89/0x150 [ 2214.434998] vfs_fstatat+0x5a/0xc0 [ 2214.438401] __do_sys_newstat+0x30/0x70 [ 2214.442239] ? lock_release+0xc5/0x260 [ 2214.445993] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2214.450872] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2214.456013] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2214.461153] do_syscall_64+0x38/0x80 [ 2214.464731] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2214.469783] RIP: 0033:0x7f48f0eeca79 [ 2214.473362] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2214.492110] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2214.499674] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2214.506798] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2214.513922] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2214.521046] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2214.528170] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2214.535301] [ 2214.537488] irq event stamp: 12925179 [ 2214.541146] hardirqs last enabled at (12925189): [] console_unlock+0x106/0x120 [ 2214.550012] hardirqs last disabled at (12925198): [] console_unlock+0xeb/0x120 [ 2214.558792] softirqs last enabled at (12925212): [] __do_softirq+0x2be/0x371 [ 2214.567484] softirqs last disabled at (12925207): [] irq_exit_rcu+0xb0/0xd0 [ 2214.576004] ---[ end trace 0000000000000000 ]--- [ 2214.580625] ------------[ cut here ]------------ [ 2214.585242] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2214.596908] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2214.605603] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2214.661096] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2214.670240] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2214.677719] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2214.683230] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2214.688715] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2214.707471] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2214.712715] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2214.719864] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2214.727007] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003acc [ 2214.734156] R10: 0000000000166e98 R11: ffffc9000030b828 R12: ffff8881d1295590 [ 2214.741289] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2214.748437] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2214.756536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2214.762299] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2214.769445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2214.776588] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2214.783731] Call Trace: [ 2214.786200] [ 2214.788306] ? __warn+0x85/0x180 [ 2214.791540] ? __list_add_valid_or_report+0x7d/0xa0 [ 2214.796444] ? report_bug+0xfc/0x1d0 [ 2214.800041] ? prb_read_valid+0x1b/0x20 [ 2214.803900] ? handle_bug+0x44/0x70 [ 2214.807408] ? exc_invalid_op+0x17/0x60 [ 2214.811256] ? asm_exc_invalid_op+0x1a/0x20 [ 2214.815468] ? __list_add_valid_or_report+0x7d/0xa0 [ 2214.820350] ? __list_add_valid_or_report+0x7d/0xa0 [ 2214.825228] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2214.830118] ceph_add_cap+0x20b/0x620 [ceph] [ 2214.834462] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2214.839184] ceph_fill_trace+0x143/0xf30 [ceph] [ 2214.843723] ? __mutex_lock+0x97/0xce0 [ 2214.847491] ? __mutex_lock+0xaec/0xce0 [ 2214.851344] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2214.855892] ? lock_downgrade+0xfd/0x250 [ 2214.859857] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2214.864238] ceph_con_process_message+0x74/0x150 [libceph] [ 2214.869731] ? calc_signature+0xdb/0x100 [libceph] [ 2214.874570] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2214.879910] ? process_scheduled_works+0x22a/0x510 [ 2214.884719] ? process_scheduled_works+0x22a/0x510 [ 2214.889509] ceph_con_workfn+0x344/0x750 [libceph] [ 2214.894331] ? process_scheduled_works+0x22a/0x510 [ 2214.899121] process_scheduled_works+0x2a4/0x510 [ 2214.903778] worker_thread+0x188/0x340 [ 2214.907530] ? rescuer_thread+0x310/0x310 [ 2214.911558] kthread+0xf5/0x130 [ 2214.914722] ? kthread_complete_and_exit+0x20/0x20 [ 2214.919552] ret_from_fork+0x31/0x40 [ 2214.923148] ? kthread_complete_and_exit+0x20/0x20 [ 2214.927941] ret_from_fork_asm+0x11/0x20 [ 2214.931907] [ 2214.934100] irq event stamp: 10365035 [ 2214.937782] hardirqs last enabled at (10365045): [] console_unlock+0x106/0x120 [ 2214.946667] hardirqs last disabled at (10365054): [] console_unlock+0xeb/0x120 [ 2214.955466] softirqs last enabled at (10365012): [] __do_softirq+0x2be/0x371 [ 2214.964173] softirqs last disabled at (10365007): [] irq_exit_rcu+0xb0/0xd0 [ 2214.972693] ---[ end trace 0000000000000000 ]--- [ 2214.977348] ------------[ cut here ]------------ [ 2214.981990] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2214.993662] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2215.002365] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2215.057927] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2215.067051] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2215.074549] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2215.080070] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2215.085572] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2215.104339] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2215.109571] RAX: 0000000000000000 RBX: ffff8881c277a720 RCX: 0000000000000000 [ 2215.116720] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2215.123891] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003b0a [ 2215.131021] R10: 0000000000167f10 R11: ffffc9000030b828 R12: ffff8881c277ae70 [ 2215.138172] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2215.145304] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2215.153392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2215.159156] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2215.166287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2215.173420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2215.180553] Call Trace: [ 2215.183043] [ 2215.185164] ? __warn+0x85/0x180 [ 2215.188401] ? __list_add_valid_or_report+0x7d/0xa0 [ 2215.193289] ? report_bug+0xfc/0x1d0 [ 2215.196875] ? prb_read_valid+0x1b/0x20 [ 2215.200752] ? handle_bug+0x44/0x70 [ 2215.204249] ? exc_invalid_op+0x17/0x60 [ 2215.208089] ? asm_exc_invalid_op+0x1a/0x20 [ 2215.212304] ? __list_add_valid_or_report+0x7d/0xa0 [ 2215.217191] ? __list_add_valid_or_report+0x7d/0xa0 [ 2215.222078] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2215.227012] ceph_add_cap+0x20b/0x620 [ceph] [ 2215.231319] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2215.236068] ? lock_acquire+0xcb/0x2b0 [ 2215.239847] ? lock_is_held_type+0xaa/0x120 [ 2215.244059] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2215.248536] ? __mutex_lock+0x97/0xce0 [ 2215.252293] ? __mutex_lock+0xaec/0xce0 [ 2215.256158] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2215.260713] ? lock_downgrade+0xfd/0x250 [ 2215.264685] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2215.269093] ceph_con_process_message+0x74/0x150 [libceph] [ 2215.274596] ? calc_signature+0xdb/0x100 [libceph] [ 2215.279431] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2215.284786] ? process_scheduled_works+0x22a/0x510 [ 2215.289604] ? process_scheduled_works+0x22a/0x510 [ 2215.294400] ceph_con_workfn+0x344/0x750 [libceph] [ 2215.299214] ? process_scheduled_works+0x22a/0x510 [ 2215.304013] process_scheduled_works+0x2a4/0x510 [ 2215.308677] worker_thread+0x188/0x340 [ 2215.312455] ? rescuer_thread+0x310/0x310 [ 2215.316505] kthread+0xf5/0x130 [ 2215.319657] ? kthread_complete_and_exit+0x20/0x20 [ 2215.324486] ret_from_fork+0x31/0x40 [ 2215.328073] ? kthread_complete_and_exit+0x20/0x20 [ 2215.332882] ret_from_fork_asm+0x11/0x20 [ 2215.336856] [ 2215.339054] irq event stamp: 10366163 [ 2215.342737] hardirqs last enabled at (10366173): [] console_unlock+0x106/0x120 [ 2215.351617] hardirqs last disabled at (10366182): [] console_unlock+0xeb/0x120 [ 2215.360415] softirqs last enabled at (10366100): [] __do_softirq+0x2be/0x371 [ 2215.369108] softirqs last disabled at (10366095): [] irq_exit_rcu+0xb0/0xd0 [ 2215.377645] ---[ end trace 0000000000000000 ]--- [ 2215.382285] ------------[ cut here ]------------ [ 2215.386908] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2215.398571] WARNING: CPU: 1 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2215.407528] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2215.463064] CPU: 1 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2215.472034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2215.479532] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2215.485034] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2215.503798] RSP: 0018:ffffc90001fcfa78 EFLAGS: 00010286 [ 2215.509041] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2215.516193] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2215.523326] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003b4a [ 2215.530459] R10: 0000000000169200 R11: ffffc90001fcf908 R12: ffff8881d1295590 [ 2215.537607] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2215.544777] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2215.552862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2215.558626] CR2: 00007f9bd296e000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2215.565774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2215.572926] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2215.580077] Call Trace: [ 2215.582545] [ 2215.584671] ? __warn+0x85/0x180 [ 2215.587919] ? __list_add_valid_or_report+0x7d/0xa0 [ 2215.592833] ? report_bug+0xfc/0x1d0 [ 2215.596414] ? prb_read_valid+0x1b/0x20 [ 2215.600257] ? handle_bug+0x44/0x70 [ 2215.603753] ? exc_invalid_op+0x17/0x60 [ 2215.607610] ? asm_exc_invalid_op+0x1a/0x20 [ 2215.611840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2215.616736] ? __list_add_valid_or_report+0x7d/0xa0 [ 2215.621615] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2215.626550] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2215.631813] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2215.636648] ceph_lookup+0xf9/0x3f0 [ceph] [ 2215.640783] __lookup_slow+0xff/0x1a0 [ 2215.644496] walk_component+0x83/0x160 [ 2215.648254] path_lookupat.isra.40+0x58/0x120 [ 2215.652638] filename_lookup+0xd4/0x190 [ 2215.656498] ? lock_acquire+0xcb/0x2b0 [ 2215.660253] ? __might_fault+0x45/0x80 [ 2215.664023] ? find_held_lock+0x2d/0x90 [ 2215.667863] ? __might_fault+0x45/0x80 [ 2215.671656] vfs_statx+0x89/0x150 [ 2215.675001] vfs_fstatat+0x5a/0xc0 [ 2215.678412] __do_sys_newstat+0x30/0x70 [ 2215.682249] ? lock_release+0xc5/0x260 [ 2215.686022] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2215.690906] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2215.696083] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2215.701223] do_syscall_64+0x38/0x80 [ 2215.704808] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2215.709878] RIP: 0033:0x7f48f0eeca79 [ 2215.713475] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2215.732239] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2215.739804] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2215.746954] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2215.754104] RBP: 00007ffc53d60520 R08: 00000000000029b2 R09: 00000000020767c0 [ 2215.761255] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2215.768389] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2215.775551] [ 2215.777740] irq event stamp: 12796461 [ 2215.781423] hardirqs last enabled at (12796471): [] console_unlock+0x106/0x120 [ 2215.790290] hardirqs last disabled at (12796480): [] console_unlock+0xeb/0x120 [ 2215.799069] softirqs last enabled at (12796042): [] __do_softirq+0x2be/0x371 [ 2215.807778] softirqs last disabled at (12796037): [] irq_exit_rcu+0xb0/0xd0 [ 2215.816316] ---[ end trace 0000000000000000 ]--- [ 2215.820935] ------------[ cut here ]------------ [ 2215.825570] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881d13cfcb0. (prev=ffff8881a1d36920). [ 2215.837217] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2215.845911] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2215.901407] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2215.910530] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2215.918019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2215.923529] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2215.929033] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2215.947790] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2215.953034] RAX: 0000000000000000 RBX: ffff8881d0bf4e40 RCX: 0000000000000000 [ 2215.960184] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2215.967318] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003b8c [ 2215.974484] R10: 000000000016a340 R11: ffffc9000030b828 R12: ffff8881d0bf5590 [ 2215.981616] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2215.988765] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2215.996861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2216.002619] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2216.009765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2216.016907] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2216.024058] Call Trace: [ 2216.026528] [ 2216.028652] ? __warn+0x85/0x180 [ 2216.031921] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.036799] ? report_bug+0xfc/0x1d0 [ 2216.040395] ? prb_read_valid+0x1b/0x20 [ 2216.044237] ? handle_bug+0x44/0x70 [ 2216.047726] ? exc_invalid_op+0x17/0x60 [ 2216.051583] ? asm_exc_invalid_op+0x1a/0x20 [ 2216.055809] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.060685] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.065579] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2216.070489] ceph_add_cap+0x20b/0x620 [ceph] [ 2216.074807] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2216.079545] ceph_fill_trace+0x143/0xf30 [ceph] [ 2216.084086] ? __mutex_lock+0x97/0xce0 [ 2216.087855] ? __mutex_lock+0xaec/0xce0 [ 2216.091712] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2216.096271] ? lock_downgrade+0xfd/0x250 [ 2216.100203] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2216.104597] ceph_con_process_message+0x74/0x150 [libceph] [ 2216.110093] ? calc_signature+0xdb/0x100 [libceph] [ 2216.114930] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2216.120255] ? process_scheduled_works+0x22a/0x510 [ 2216.125064] ? process_scheduled_works+0x22a/0x510 [ 2216.129852] ceph_con_workfn+0x344/0x750 [libceph] [ 2216.134675] ? process_scheduled_works+0x22a/0x510 [ 2216.139482] process_scheduled_works+0x2a4/0x510 [ 2216.144142] worker_thread+0x188/0x340 [ 2216.147899] ? rescuer_thread+0x310/0x310 [ 2216.151927] kthread+0xf5/0x130 [ 2216.155091] ? kthread_complete_and_exit+0x20/0x20 [ 2216.159905] ret_from_fork+0x31/0x40 [ 2216.163498] ? kthread_complete_and_exit+0x20/0x20 [ 2216.168309] ret_from_fork_asm+0x11/0x20 [ 2216.172242] [ 2216.174426] irq event stamp: 10368169 [ 2216.178093] hardirqs last enabled at (10368179): [] console_unlock+0x106/0x120 [ 2216.186978] hardirqs last disabled at (10368188): [] console_unlock+0xeb/0x120 [ 2216.195773] softirqs last enabled at (10368126): [] __do_softirq+0x2be/0x371 [ 2216.204485] softirqs last disabled at (10368207): [] irq_exit_rcu+0xb0/0xd0 [ 2216.213020] ---[ end trace 0000000000000000 ]--- [ 2216.217838] ------------[ cut here ]------------ [ 2216.222539] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2216.234191] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2216.243156] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2216.298636] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2216.307590] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2216.315072] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2216.320564] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2216.339314] RSP: 0018:ffffc90001fe7a78 EFLAGS: 00010286 [ 2216.344545] RAX: 0000000000000000 RBX: ffff8881d0bf4e40 RCX: 0000000000000000 [ 2216.351680] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2216.358821] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003bca [ 2216.365956] R10: 000000000016b5d0 R11: ffffc90001fe7908 R12: ffff8881d0bf5590 [ 2216.373095] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2216.380228] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2216.388314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2216.394054] CR2: 00007f9bd2981028 CR3: 0000000108c08001 CR4: 00000000003706e0 [ 2216.401184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2216.408318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2216.415451] Call Trace: [ 2216.417896] [ 2216.420001] ? __warn+0x85/0x180 [ 2216.423235] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.428114] ? report_bug+0xfc/0x1d0 [ 2216.431693] ? prb_read_valid+0x1b/0x20 [ 2216.435533] ? handle_bug+0x44/0x70 [ 2216.439024] ? exc_invalid_op+0x17/0x60 [ 2216.442857] ? asm_exc_invalid_op+0x1a/0x20 [ 2216.447046] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.451921] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.456801] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2216.461691] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2216.466927] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2216.471727] ceph_lookup+0xf9/0x3f0 [ceph] [ 2216.475836] __lookup_slow+0xff/0x1a0 [ 2216.479507] walk_component+0x83/0x160 [ 2216.483263] path_lookupat.isra.40+0x58/0x120 [ 2216.487622] filename_lookup+0xd4/0x190 [ 2216.491464] ? lock_acquire+0xcb/0x2b0 [ 2216.495212] ? __might_fault+0x45/0x80 [ 2216.498965] ? find_held_lock+0x2d/0x90 [ 2216.502803] ? __might_fault+0x45/0x80 [ 2216.506561] vfs_statx+0x89/0x150 [ 2216.509880] vfs_fstatat+0x5a/0xc0 [ 2216.513282] __do_sys_newstat+0x30/0x70 [ 2216.517121] ? lock_release+0xc5/0x260 [ 2216.520877] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2216.525753] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2216.530893] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2216.536033] do_syscall_64+0x38/0x80 [ 2216.539613] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2216.544665] RIP: 0033:0x7f48f0eeca79 [ 2216.548245] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2216.566991] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2216.574560] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2216.581688] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2216.588813] RBP: 00007ffc53d60520 R08: 0000000000002a0f R09: 00000000020767c0 [ 2216.595946] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2216.603079] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2216.610219] [ 2216.612403] irq event stamp: 12818957 [ 2216.616061] hardirqs last enabled at (12818967): [] console_unlock+0x106/0x120 [ 2216.624928] hardirqs last disabled at (12818976): [] console_unlock+0xeb/0x120 [ 2216.633709] softirqs last enabled at (12818990): [] __do_softirq+0x2be/0x371 [ 2216.642399] softirqs last disabled at (12818985): [] irq_exit_rcu+0xb0/0xd0 [ 2216.650920] ---[ end trace 0000000000000000 ]--- [ 2216.655539] ------------[ cut here ]------------ [ 2216.660161] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2216.671839] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2216.680536] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2216.736030] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2216.745189] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2216.752670] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2216.758190] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2216.763674] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2216.782430] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2216.787657] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2216.794805] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2216.801969] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003c0c [ 2216.809101] R10: 000000000016c708 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2216.816249] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2216.823382] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2216.831479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2216.837239] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2216.844374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2216.851534] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2216.858666] Call Trace: [ 2216.861135] [ 2216.863257] ? __warn+0x85/0x180 [ 2216.866492] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.871388] ? report_bug+0xfc/0x1d0 [ 2216.874966] ? prb_read_valid+0x1b/0x20 [ 2216.878825] ? handle_bug+0x44/0x70 [ 2216.882333] ? exc_invalid_op+0x17/0x60 [ 2216.886191] ? asm_exc_invalid_op+0x1a/0x20 [ 2216.890381] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.895256] ? __list_add_valid_or_report+0x7d/0xa0 [ 2216.900133] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2216.905042] ceph_add_cap+0x20b/0x620 [ceph] [ 2216.909328] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2216.914067] ? lock_acquire+0xcb/0x2b0 [ 2216.917816] ? lock_is_held_type+0xaa/0x120 [ 2216.922019] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2216.926493] ? __mutex_lock+0x97/0xce0 [ 2216.930245] ? __mutex_lock+0xaec/0xce0 [ 2216.934085] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2216.938642] ? lock_downgrade+0xfd/0x250 [ 2216.942607] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2216.947005] ceph_con_process_message+0x74/0x150 [libceph] [ 2216.952499] ? calc_signature+0xdb/0x100 [libceph] [ 2216.957306] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2216.962652] ? process_scheduled_works+0x22a/0x510 [ 2216.967444] ? process_scheduled_works+0x22a/0x510 [ 2216.972234] ceph_con_workfn+0x344/0x750 [libceph] [ 2216.977037] ? process_scheduled_works+0x22a/0x510 [ 2216.981844] process_scheduled_works+0x2a4/0x510 [ 2216.986494] worker_thread+0x188/0x340 [ 2216.990247] ? rescuer_thread+0x310/0x310 [ 2216.994257] kthread+0xf5/0x130 [ 2216.997404] ? kthread_complete_and_exit+0x20/0x20 [ 2217.002198] ret_from_fork+0x31/0x40 [ 2217.005775] ? kthread_complete_and_exit+0x20/0x20 [ 2217.010583] ret_from_fork_asm+0x11/0x20 [ 2217.014543] [ 2217.016730] irq event stamp: 10373925 [ 2217.020406] hardirqs last enabled at (10373935): [] console_unlock+0x106/0x120 [ 2217.029270] hardirqs last disabled at (10373944): [] console_unlock+0xeb/0x120 [ 2217.038050] softirqs last enabled at (10373862): [] __do_softirq+0x2be/0x371 [ 2217.046762] softirqs last disabled at (10373857): [] irq_exit_rcu+0xb0/0xd0 [ 2217.055279] ---[ end trace 0000000000000000 ]--- [ 2217.059927] ------------[ cut here ]------------ [ 2217.064546] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2217.076211] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2217.084905] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2217.140400] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2217.149523] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2217.157020] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2217.162531] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2217.168034] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2217.186789] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2217.192016] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2217.199156] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2217.206300] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003c4c [ 2217.213432] R10: 000000000016d9f8 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2217.220563] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2217.227706] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2217.235809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2217.241565] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2217.248717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2217.255848] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2217.262998] Call Trace: [ 2217.265460] [ 2217.267567] ? __warn+0x85/0x180 [ 2217.270818] ? __list_add_valid_or_report+0x7d/0xa0 [ 2217.275696] ? report_bug+0xfc/0x1d0 [ 2217.279292] ? prb_read_valid+0x1b/0x20 [ 2217.283133] ? handle_bug+0x44/0x70 [ 2217.286640] ? exc_invalid_op+0x17/0x60 [ 2217.290489] ? asm_exc_invalid_op+0x1a/0x20 [ 2217.294713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2217.299590] ? __list_add_valid_or_report+0x7d/0xa0 [ 2217.304485] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2217.309392] ceph_add_cap+0x20b/0x620 [ceph] [ 2217.313696] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2217.318417] ceph_fill_trace+0x143/0xf30 [ceph] [ 2217.322955] ? __mutex_lock+0x97/0xce0 [ 2217.326724] ? __mutex_lock+0xaec/0xce0 [ 2217.330572] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2217.335133] ? lock_downgrade+0xfd/0x250 [ 2217.339096] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2217.343500] ceph_con_process_message+0x74/0x150 [libceph] [ 2217.348999] ? calc_signature+0xdb/0x100 [libceph] [ 2217.353802] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2217.359160] ? process_scheduled_works+0x22a/0x510 [ 2217.363966] ? process_scheduled_works+0x22a/0x510 [ 2217.368759] ceph_con_workfn+0x344/0x750 [libceph] [ 2217.373581] ? process_scheduled_works+0x22a/0x510 [ 2217.378386] process_scheduled_works+0x2a4/0x510 [ 2217.383029] worker_thread+0x188/0x340 [ 2217.386778] ? rescuer_thread+0x310/0x310 [ 2217.390806] kthread+0xf5/0x130 [ 2217.393972] ? kthread_complete_and_exit+0x20/0x20 [ 2217.398801] ret_from_fork+0x31/0x40 [ 2217.402377] ? kthread_complete_and_exit+0x20/0x20 [ 2217.407191] ret_from_fork_asm+0x11/0x20 [ 2217.411138] [ 2217.413324] irq event stamp: 10374999 [ 2217.416989] hardirqs last enabled at (10375009): [] console_unlock+0x106/0x120 [ 2217.425873] hardirqs last disabled at (10375018): [] console_unlock+0xeb/0x120 [ 2217.434669] softirqs last enabled at (10374956): [] __do_softirq+0x2be/0x371 [ 2217.443382] softirqs last disabled at (10374951): [] irq_exit_rcu+0xb0/0xd0 [ 2217.451899] ---[ end trace 0000000000000000 ]--- [ 2217.456564] ------------[ cut here ]------------ [ 2217.461243] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2217.472909] WARNING: CPU: 1 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2217.481864] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2217.537402] CPU: 1 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2217.546354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2217.553831] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2217.559336] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2217.578101] RSP: 0018:ffffc90001957b98 EFLAGS: 00010282 [ 2217.583327] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2217.590459] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2217.597590] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003c8a [ 2217.604741] R10: 000000000016eae8 R11: ffffc90001957a28 R12: ffff8881d0bb4bc8 [ 2217.611891] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2217.619041] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2217.627144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2217.632908] CR2: 00007f9bd296e000 CR3: 000000016696c006 CR4: 00000000003706e0 [ 2217.640059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2217.647207] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2217.654341] Call Trace: [ 2217.656794] [ 2217.658917] ? __warn+0x85/0x180 [ 2217.662169] ? __list_add_valid_or_report+0x7d/0xa0 [ 2217.667071] ? report_bug+0xfc/0x1d0 [ 2217.670652] ? prb_read_valid+0x1b/0x20 [ 2217.674529] ? handle_bug+0x44/0x70 [ 2217.678020] ? exc_invalid_op+0x17/0x60 [ 2217.681875] ? asm_exc_invalid_op+0x1a/0x20 [ 2217.686106] ? __list_add_valid_or_report+0x7d/0xa0 [ 2217.691004] ? __list_add_valid_or_report+0x7d/0xa0 [ 2217.695882] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2217.700815] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2217.705561] lookup_fast+0x6d/0xf0 [ 2217.708987] walk_component+0x26/0x160 [ 2217.712745] path_lookupat.isra.40+0x58/0x120 [ 2217.717126] filename_lookup+0xd4/0x190 [ 2217.720986] ? lock_acquire+0xcb/0x2b0 [ 2217.724744] ? __might_fault+0x45/0x80 [ 2217.728515] ? find_held_lock+0x2d/0x90 [ 2217.732361] ? __might_fault+0x45/0x80 [ 2217.736137] vfs_statx+0x89/0x150 [ 2217.739465] vfs_fstatat+0x5a/0xc0 [ 2217.742893] __do_sys_newstat+0x30/0x70 [ 2217.746749] ? lock_release+0xc5/0x260 [ 2217.750522] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2217.755404] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2217.760562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2217.765720] do_syscall_64+0x38/0x80 [ 2217.769299] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2217.774352] RIP: 0033:0x7f48f0eeca79 [ 2217.777931] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2217.796694] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2217.804277] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2217.811458] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2217.818663] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2217.825796] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2217.832946] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2217.840128] [ 2217.842316] irq event stamp: 12791319 [ 2217.845981] hardirqs last enabled at (12791329): [] console_unlock+0x106/0x120 [ 2217.854867] hardirqs last disabled at (12791338): [] console_unlock+0xeb/0x120 [ 2217.863663] softirqs last enabled at (12791276): [] __do_softirq+0x2be/0x371 [ 2217.872372] softirqs last disabled at (12791261): [] irq_exit_rcu+0xb0/0xd0 [ 2217.880891] ---[ end trace 0000000000000000 ]--- [ 2217.885529] ------------[ cut here ]------------ [ 2217.890165] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2217.901812] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2217.910508] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2217.965993] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2217.975120] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2217.982597] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2217.988100] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2217.993586] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2218.012332] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2218.017560] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2218.024690] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2218.031823] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003cca [ 2218.038960] R10: 000000000016fc30 R11: ffffc9000030b828 R12: ffff8881c5351ae0 [ 2218.046089] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2218.053221] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2218.061309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2218.067054] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2218.074192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2218.081320] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2218.088452] Call Trace: [ 2218.090906] [ 2218.093013] ? __warn+0x85/0x180 [ 2218.096246] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.101124] ? report_bug+0xfc/0x1d0 [ 2218.104706] ? prb_read_valid+0x1b/0x20 [ 2218.108546] ? handle_bug+0x44/0x70 [ 2218.112034] ? exc_invalid_op+0x17/0x60 [ 2218.115877] ? asm_exc_invalid_op+0x1a/0x20 [ 2218.120069] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.124951] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.129827] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2218.134722] ceph_add_cap+0x20b/0x620 [ceph] [ 2218.139017] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2218.143739] ? lock_acquire+0xcb/0x2b0 [ 2218.147495] ? lock_is_held_type+0xaa/0x120 [ 2218.151681] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2218.156142] ? __mutex_lock+0x97/0xce0 [ 2218.159893] ? __mutex_lock+0xaec/0xce0 [ 2218.163735] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2218.168286] ? lock_downgrade+0xfd/0x250 [ 2218.172219] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2218.176599] ceph_con_process_message+0x74/0x150 [libceph] [ 2218.182098] ? calc_signature+0xdb/0x100 [libceph] [ 2218.186914] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2218.192246] ? process_scheduled_works+0x22a/0x510 [ 2218.197046] ? process_scheduled_works+0x22a/0x510 [ 2218.201841] ceph_con_workfn+0x344/0x750 [libceph] [ 2218.206656] ? process_scheduled_works+0x22a/0x510 [ 2218.211453] process_scheduled_works+0x2a4/0x510 [ 2218.216080] worker_thread+0x188/0x340 [ 2218.219837] ? rescuer_thread+0x310/0x310 [ 2218.223846] kthread+0xf5/0x130 [ 2218.226993] ? kthread_complete_and_exit+0x20/0x20 [ 2218.231791] ret_from_fork+0x31/0x40 [ 2218.235374] ? kthread_complete_and_exit+0x20/0x20 [ 2218.240167] ret_from_fork_asm+0x11/0x20 [ 2218.244102] [ 2218.246294] irq event stamp: 10377091 [ 2218.249958] hardirqs last enabled at (10377101): [] console_unlock+0x106/0x120 [ 2218.258826] hardirqs last disabled at (10377110): [] console_unlock+0xeb/0x120 [ 2218.267606] softirqs last enabled at (10377048): [] __do_softirq+0x2be/0x371 [ 2218.276300] softirqs last disabled at (10377129): [] irq_exit_rcu+0xb0/0xd0 [ 2218.284827] ---[ end trace 0000000000000000 ]--- [ 2218.289446] ------------[ cut here ]------------ [ 2218.294063] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2218.305711] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2218.314669] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2218.370138] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2218.379097] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2218.386573] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2218.392063] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2218.410807] RSP: 0018:ffffc90001957b20 EFLAGS: 00010282 [ 2218.416034] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2218.423165] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2218.430300] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003d0a [ 2218.437433] R10: 0000000000170d08 R11: ffffc900019579b0 R12: ffff8881d0bb4bc8 [ 2218.444564] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2218.451697] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2218.459785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2218.465528] CR2: 00007f9bd2971000 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2218.472662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2218.479795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2218.486929] Call Trace: [ 2218.489380] [ 2218.491501] ? __warn+0x85/0x180 [ 2218.494746] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.499636] ? report_bug+0xfc/0x1d0 [ 2218.503215] ? prb_read_valid+0x1b/0x20 [ 2218.507057] ? handle_bug+0x44/0x70 [ 2218.510555] ? exc_invalid_op+0x17/0x60 [ 2218.514397] ? asm_exc_invalid_op+0x1a/0x20 [ 2218.518596] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.523483] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2218.528376] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2218.533632] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2218.538436] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2218.543168] lookup_fast+0x6d/0xf0 [ 2218.546573] walk_component+0x26/0x160 [ 2218.550331] path_lookupat.isra.40+0x58/0x120 [ 2218.554696] filename_lookup+0xd4/0x190 [ 2218.558542] ? lock_acquire+0xcb/0x2b0 [ 2218.562295] ? __might_fault+0x45/0x80 [ 2218.566050] ? find_held_lock+0x2d/0x90 [ 2218.569898] ? __might_fault+0x45/0x80 [ 2218.573655] vfs_statx+0x89/0x150 [ 2218.576980] vfs_fstatat+0x5a/0xc0 [ 2218.580395] __do_sys_newstat+0x30/0x70 [ 2218.584239] ? lock_release+0xc5/0x260 [ 2218.587998] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2218.592881] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2218.598023] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2218.603169] do_syscall_64+0x38/0x80 [ 2218.606748] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2218.611800] RIP: 0033:0x7f48f0eeca79 [ 2218.615381] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2218.634127] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2218.641693] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2218.648825] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2218.655958] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2218.663094] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2218.670223] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2218.677368] [ 2218.679556] irq event stamp: 12793389 [ 2218.683223] hardirqs last enabled at (12793399): [] console_unlock+0x106/0x120 [ 2218.692090] hardirqs last disabled at (12793408): [] console_unlock+0xeb/0x120 [ 2218.700870] softirqs last enabled at (12793422): [] __do_softirq+0x2be/0x371 [ 2218.709563] softirqs last disabled at (12793417): [] irq_exit_rcu+0xb0/0xd0 [ 2218.718080] ---[ end trace 0000000000000000 ]--- [ 2218.722702] ------------[ cut here ]------------ [ 2218.727319] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2218.738981] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2218.747678] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2218.803171] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2218.812315] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2218.819794] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2218.825305] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2218.830792] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2218.849576] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2218.854827] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2218.861976] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2218.869109] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003d4b [ 2218.876258] R10: 0000000000171fb0 R11: ffffc9000030b828 R12: ffff8881c5357cb0 [ 2218.883391] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2218.890522] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2218.898617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2218.904372] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2218.911506] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2218.918667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2218.925796] Call Trace: [ 2218.928260] [ 2218.930366] ? __warn+0x85/0x180 [ 2218.933597] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.938507] ? report_bug+0xfc/0x1d0 [ 2218.942098] ? prb_read_valid+0x1b/0x20 [ 2218.945949] ? handle_bug+0x44/0x70 [ 2218.949458] ? exc_invalid_op+0x17/0x60 [ 2218.953297] ? asm_exc_invalid_op+0x1a/0x20 [ 2218.957503] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.962397] ? __list_add_valid_or_report+0x7d/0xa0 [ 2218.967276] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2218.972166] ceph_add_cap+0x20b/0x620 [ceph] [ 2218.976477] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2218.981218] ceph_fill_trace+0x143/0xf30 [ceph] [ 2218.985753] ? __mutex_lock+0x97/0xce0 [ 2218.989516] ? __mutex_lock+0xaec/0xce0 [ 2218.993373] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2218.997914] ? lock_downgrade+0xfd/0x250 [ 2219.001877] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2219.006252] ceph_con_process_message+0x74/0x150 [libceph] [ 2219.011747] ? calc_signature+0xdb/0x100 [libceph] [ 2219.016567] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2219.021923] ? process_scheduled_works+0x22a/0x510 [ 2219.026733] ? process_scheduled_works+0x22a/0x510 [ 2219.031531] ceph_con_workfn+0x344/0x750 [libceph] [ 2219.036353] ? process_scheduled_works+0x22a/0x510 [ 2219.041142] process_scheduled_works+0x2a4/0x510 [ 2219.045794] worker_thread+0x188/0x340 [ 2219.049543] ? rescuer_thread+0x310/0x310 [ 2219.053571] kthread+0xf5/0x130 [ 2219.056736] ? kthread_complete_and_exit+0x20/0x20 [ 2219.061547] ret_from_fork+0x31/0x40 [ 2219.065141] ? kthread_complete_and_exit+0x20/0x20 [ 2219.069944] ret_from_fork_asm+0x11/0x20 [ 2219.073909] [ 2219.076097] irq event stamp: 10379135 [ 2219.079770] hardirqs last enabled at (10379145): [] console_unlock+0x106/0x120 [ 2219.088646] hardirqs last disabled at (10379154): [] console_unlock+0xeb/0x120 [ 2219.097444] softirqs last enabled at (10379092): [] __do_softirq+0x2be/0x371 [ 2219.106136] softirqs last disabled at (10379087): [] irq_exit_rcu+0xb0/0xd0 [ 2219.114672] ---[ end trace 0000000000000000 ]--- [ 2219.119455] ------------[ cut here ]------------ [ 2219.124070] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2219.135748] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2219.144445] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2219.199923] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2219.209064] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2219.216544] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2219.222039] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2219.227542] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2219.246305] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2219.251531] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2219.258672] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2219.265816] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003d89 [ 2219.272964] R10: 0000000000173248 R11: ffffc9000030b828 R12: ffff8881d1144bc8 [ 2219.280106] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2219.287255] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2219.295343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2219.301088] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2219.308236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2219.315363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2219.322496] Call Trace: [ 2219.324958] [ 2219.327081] ? __warn+0x85/0x180 [ 2219.330336] ? __list_add_valid_or_report+0x7d/0xa0 [ 2219.335237] ? report_bug+0xfc/0x1d0 [ 2219.338817] ? prb_read_valid+0x1b/0x20 [ 2219.342693] ? handle_bug+0x44/0x70 [ 2219.346183] ? exc_invalid_op+0x17/0x60 [ 2219.350040] ? asm_exc_invalid_op+0x1a/0x20 [ 2219.354255] ? __list_add_valid_or_report+0x7d/0xa0 [ 2219.359150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2219.364027] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2219.368953] ceph_add_cap+0x20b/0x620 [ceph] [ 2219.373252] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2219.377992] ? lock_acquire+0xcb/0x2b0 [ 2219.381763] ? lock_is_held_type+0xaa/0x120 [ 2219.385966] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2219.390421] ? __mutex_lock+0x97/0xce0 [ 2219.394174] ? __mutex_lock+0xaec/0xce0 [ 2219.398030] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2219.402578] ? lock_downgrade+0xfd/0x250 [ 2219.406546] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2219.410945] ceph_con_process_message+0x74/0x150 [libceph] [ 2219.416443] ? calc_signature+0xdb/0x100 [libceph] [ 2219.421248] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2219.426601] ? process_scheduled_works+0x22a/0x510 [ 2219.431397] ? process_scheduled_works+0x22a/0x510 [ 2219.436189] ceph_con_workfn+0x344/0x750 [libceph] [ 2219.441028] ? process_scheduled_works+0x22a/0x510 [ 2219.445825] process_scheduled_works+0x2a4/0x510 [ 2219.450471] worker_thread+0x188/0x340 [ 2219.454240] ? rescuer_thread+0x310/0x310 [ 2219.458257] kthread+0xf5/0x130 [ 2219.461409] ? kthread_complete_and_exit+0x20/0x20 [ 2219.466224] ret_from_fork+0x31/0x40 [ 2219.469805] ? kthread_complete_and_exit+0x20/0x20 [ 2219.474600] ret_from_fork_asm+0x11/0x20 [ 2219.478570] [ 2219.480762] irq event stamp: 10380275 [ 2219.484444] hardirqs last enabled at (10380285): [] console_unlock+0x106/0x120 [ 2219.493312] hardirqs last disabled at (10380294): [] console_unlock+0xeb/0x120 [ 2219.502090] softirqs last enabled at (10380210): [] __do_softirq+0x2be/0x371 [ 2219.510800] softirqs last disabled at (10380205): [] irq_exit_rcu+0xb0/0xd0 [ 2219.519338] ---[ end trace 0000000000000000 ]--- [ 2219.523976] ------------[ cut here ]------------ [ 2219.528595] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2219.540243] WARNING: CPU: 3 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2219.549196] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2219.604715] CPU: 3 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2219.613669] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2219.621147] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2219.626633] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2219.645383] RSP: 0018:ffffc90001f7fb98 EFLAGS: 00010282 [ 2219.650615] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2219.657747] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2219.664880] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003dc9 [ 2219.672014] R10: 0000000000174340 R11: ffffc90001f7fa28 R12: ffff8881c5357cb0 [ 2219.679146] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2219.686279] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2219.694364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2219.700112] CR2: 00007fb38d59b000 CR3: 0000000113532001 CR4: 00000000003706e0 [ 2219.707245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2219.714377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2219.721508] Call Trace: [ 2219.723963] [ 2219.726069] ? __warn+0x85/0x180 [ 2219.729302] ? __list_add_valid_or_report+0x7d/0xa0 [ 2219.734182] ? report_bug+0xfc/0x1d0 [ 2219.737762] ? prb_read_valid+0x1b/0x20 [ 2219.741611] ? handle_bug+0x44/0x70 [ 2219.745100] ? exc_invalid_op+0x17/0x60 [ 2219.748940] ? asm_exc_invalid_op+0x1a/0x20 [ 2219.753135] ? __list_add_valid_or_report+0x7d/0xa0 [ 2219.758016] ? __list_add_valid_or_report+0x7d/0xa0 [ 2219.762893] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2219.767795] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2219.772521] lookup_fast+0x6d/0xf0 [ 2219.775929] walk_component+0x26/0x160 [ 2219.779684] path_lookupat.isra.40+0x58/0x120 [ 2219.784045] filename_lookup+0xd4/0x190 [ 2219.787898] ? lock_acquire+0xcb/0x2b0 [ 2219.791651] ? __might_fault+0x45/0x80 [ 2219.795405] ? find_held_lock+0x2d/0x90 [ 2219.799243] ? __might_fault+0x45/0x80 [ 2219.803001] vfs_statx+0x89/0x150 [ 2219.806329] vfs_fstatat+0x5a/0xc0 [ 2219.809739] __do_sys_newstat+0x30/0x70 [ 2219.813578] ? lock_release+0xc5/0x260 [ 2219.817338] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2219.822217] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2219.827360] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2219.832509] do_syscall_64+0x38/0x80 [ 2219.836094] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2219.841147] RIP: 0033:0x7f48f0eeca79 [ 2219.844725] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2219.863472] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2219.871038] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2219.878172] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2219.885304] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2219.892436] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2219.899570] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2219.906719] [ 2219.908912] irq event stamp: 12919659 [ 2219.912578] hardirqs last enabled at (12919669): [] console_unlock+0x106/0x120 [ 2219.921444] hardirqs last disabled at (12919678): [] console_unlock+0xeb/0x120 [ 2219.930224] softirqs last enabled at (12919646): [] __do_softirq+0x2be/0x371 [ 2219.938923] softirqs last disabled at (12919697): [] irq_exit_rcu+0xb0/0xd0 [ 2219.947447] ---[ end trace 0000000000000000 ]--- [ 2219.952067] ------------[ cut here ]------------ [ 2219.956681] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2219.968327] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2219.977284] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2220.032753] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2220.041705] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2220.049182] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2220.054668] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2220.073413] RSP: 0018:ffffc90001fdfbf0 EFLAGS: 00010282 [ 2220.078641] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2220.085772] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2220.092897] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003e09 [ 2220.100031] R10: 0000000000175608 R11: ffffc90001fdfa80 R12: ffff888158a68750 [ 2220.107164] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2220.114297] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2220.122381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.128119] CR2: 00007f9bd2981028 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2220.135244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2220.142377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2220.149509] Call Trace: [ 2220.151963] [ 2220.154068] ? __warn+0x85/0x180 [ 2220.157301] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.162180] ? report_bug+0xfc/0x1d0 [ 2220.165760] ? prb_read_valid+0x1b/0x20 [ 2220.169602] ? handle_bug+0x44/0x70 [ 2220.173094] ? exc_invalid_op+0x17/0x60 [ 2220.176930] ? asm_exc_invalid_op+0x1a/0x20 [ 2220.181122] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.185999] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2220.190888] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2220.195611] lookup_fast+0x6d/0xf0 [ 2220.199063] walk_component+0x26/0x160 [ 2220.202815] link_path_walk+0x24d/0x380 [ 2220.206650] ? path_openat+0x80/0x9d0 [ 2220.210318] path_openat+0x8e/0x9d0 [ 2220.213812] do_filp_open+0xb8/0x120 [ 2220.217392] ? lock_release+0xc5/0x260 [ 2220.221142] ? _raw_spin_unlock+0x23/0x40 [ 2220.225154] ? alloc_fd+0x13f/0x200 [ 2220.228648] do_sys_openat2+0x249/0x290 [ 2220.232490] do_sys_open+0x4f/0x80 [ 2220.235894] do_syscall_64+0x38/0x80 [ 2220.239471] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2220.244524] RIP: 0033:0x7f48f0ef2805 [ 2220.248102] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2220.266849] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2220.274415] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2220.281550] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2220.288680] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2220.295814] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2220.302946] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2220.310087] [ 2220.312272] irq event stamp: 12820255 [ 2220.315939] hardirqs last enabled at (12820265): [] console_unlock+0x106/0x120 [ 2220.324806] hardirqs last disabled at (12820274): [] console_unlock+0xeb/0x120 [ 2220.333583] softirqs last enabled at (12820222): [] __do_softirq+0x2be/0x371 [ 2220.342277] softirqs last disabled at (12820217): [] irq_exit_rcu+0xb0/0xd0 [ 2220.350795] ---[ end trace 0000000000000000 ]--- [ 2220.355409] ------------[ cut here ]------------ [ 2220.360028] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2220.371676] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2220.380371] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2220.435859] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2220.444989] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2220.452469] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2220.457972] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2220.463456] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2220.482204] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2220.487430] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2220.494563] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2220.501700] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003e44 [ 2220.508828] R10: 0000000000176650 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2220.515961] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2220.523094] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2220.531182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.536925] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2220.544061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2220.551193] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2220.558325] Call Trace: [ 2220.560780] [ 2220.562884] ? __warn+0x85/0x180 [ 2220.566121] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.571009] ? report_bug+0xfc/0x1d0 [ 2220.574593] ? prb_read_valid+0x1b/0x20 [ 2220.578434] ? handle_bug+0x44/0x70 [ 2220.581927] ? exc_invalid_op+0x17/0x60 [ 2220.585766] ? asm_exc_invalid_op+0x1a/0x20 [ 2220.589954] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.594829] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.599713] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2220.604610] ceph_add_cap+0x20b/0x620 [ceph] [ 2220.608905] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2220.613630] ? lock_acquire+0xcb/0x2b0 [ 2220.617383] ? lock_is_held_type+0xaa/0x120 [ 2220.621569] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2220.626023] ? __mutex_lock+0x97/0xce0 [ 2220.629775] ? __mutex_lock+0xaec/0xce0 [ 2220.633615] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2220.638156] ? lock_downgrade+0xfd/0x250 [ 2220.642085] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2220.646471] ceph_con_process_message+0x74/0x150 [libceph] [ 2220.651971] ? calc_signature+0xdb/0x100 [libceph] [ 2220.656788] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2220.662119] ? process_scheduled_works+0x22a/0x510 [ 2220.666914] ? process_scheduled_works+0x22a/0x510 [ 2220.671706] ceph_con_workfn+0x344/0x750 [libceph] [ 2220.676518] ? process_scheduled_works+0x22a/0x510 [ 2220.681306] process_scheduled_works+0x2a4/0x510 [ 2220.685932] worker_thread+0x188/0x340 [ 2220.689690] ? rescuer_thread+0x310/0x310 [ 2220.693701] kthread+0xf5/0x130 [ 2220.696847] ? kthread_complete_and_exit+0x20/0x20 [ 2220.701643] ret_from_fork+0x31/0x40 [ 2220.705220] ? kthread_complete_and_exit+0x20/0x20 [ 2220.710015] ret_from_fork_asm+0x11/0x20 [ 2220.713944] [ 2220.716129] irq event stamp: 10383239 [ 2220.719798] hardirqs last enabled at (10383249): [] console_unlock+0x106/0x120 [ 2220.728662] hardirqs last disabled at (10383258): [] console_unlock+0xeb/0x120 [ 2220.737444] softirqs last enabled at (10383176): [] __do_softirq+0x2be/0x371 [ 2220.746136] softirqs last disabled at (10383171): [] irq_exit_rcu+0xb0/0xd0 [ 2220.754656] ---[ end trace 0000000000000000 ]--- [ 2220.759276] ------------[ cut here ]------------ [ 2220.763894] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2220.775541] WARNING: CPU: 0 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2220.784500] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2220.839985] CPU: 0 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2220.848951] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2220.856430] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2220.861918] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2220.880671] RSP: 0018:ffffc90001f7fb20 EFLAGS: 00010282 [ 2220.885898] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2220.893030] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2220.900164] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003e84 [ 2220.907295] R10: 0000000000177900 R11: ffffc90001f7f9b0 R12: ffff8881c5357cb0 [ 2220.914431] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2220.921562] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2220.929649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2220.935393] CR2: 00007fb38d5a5098 CR3: 0000000113532001 CR4: 00000000003706f0 [ 2220.942527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2220.949658] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2220.956793] Call Trace: [ 2220.959244] [ 2220.961357] ? __warn+0x85/0x180 [ 2220.964592] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.969474] ? report_bug+0xfc/0x1d0 [ 2220.973059] ? prb_read_valid+0x1b/0x20 [ 2220.976904] ? handle_bug+0x44/0x70 [ 2220.980401] ? exc_invalid_op+0x17/0x60 [ 2220.984243] ? asm_exc_invalid_op+0x1a/0x20 [ 2220.988441] ? __list_add_valid_or_report+0x7d/0xa0 [ 2220.993326] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2220.998223] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2221.003473] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2221.008284] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2221.013014] lookup_fast+0x6d/0xf0 [ 2221.016420] walk_component+0x26/0x160 [ 2221.020177] path_lookupat.isra.40+0x58/0x120 [ 2221.024543] filename_lookup+0xd4/0x190 [ 2221.028389] ? lock_acquire+0xcb/0x2b0 [ 2221.032142] ? __might_fault+0x45/0x80 [ 2221.035898] ? find_held_lock+0x2d/0x90 [ 2221.039743] ? __might_fault+0x45/0x80 [ 2221.043505] vfs_statx+0x89/0x150 [ 2221.046829] vfs_fstatat+0x5a/0xc0 [ 2221.050244] __do_sys_newstat+0x30/0x70 [ 2221.054087] ? lock_release+0xc5/0x260 [ 2221.057843] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2221.062726] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2221.067868] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2221.073006] do_syscall_64+0x38/0x80 [ 2221.076587] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2221.081646] RIP: 0033:0x7f48f0eeca79 [ 2221.085226] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2221.103975] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2221.111552] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2221.118687] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2221.125823] RBP: 00007ffc53d60520 R08: 0000000000002a1a R09: 00000000020767c0 [ 2221.132953] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2221.140087] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2221.147231] [ 2221.149420] irq event stamp: 12922603 [ 2221.153086] hardirqs last enabled at (12922613): [] console_unlock+0x106/0x120 [ 2221.161954] hardirqs last disabled at (12922622): [] console_unlock+0xeb/0x120 [ 2221.170731] softirqs last enabled at (12922540): [] __do_softirq+0x2be/0x371 [ 2221.179427] softirqs last disabled at (12922641): [] irq_exit_rcu+0xb0/0xd0 [ 2221.187952] ---[ end trace 0000000000000000 ]--- [ 2221.192574] ------------[ cut here ]------------ [ 2221.197192] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2221.208841] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2221.217533] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2221.273019] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2221.282142] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2221.289622] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2221.295126] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2221.300611] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2221.319359] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2221.324585] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2221.331719] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2221.338852] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003ec5 [ 2221.345983] R10: 0000000000178a50 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2221.353116] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2221.360251] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2221.368336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2221.374081] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2221.381213] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2221.388346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2221.395479] Call Trace: [ 2221.397933] [ 2221.400039] ? __warn+0x85/0x180 [ 2221.403274] ? __list_add_valid_or_report+0x7d/0xa0 [ 2221.408153] ? report_bug+0xfc/0x1d0 [ 2221.411729] ? prb_read_valid+0x1b/0x20 [ 2221.415569] ? handle_bug+0x44/0x70 [ 2221.419060] ? exc_invalid_op+0x17/0x60 [ 2221.422903] ? asm_exc_invalid_op+0x1a/0x20 [ 2221.427095] ? __list_add_valid_or_report+0x7d/0xa0 [ 2221.431977] ? __list_add_valid_or_report+0x7d/0xa0 [ 2221.436855] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2221.441747] ceph_add_cap+0x20b/0x620 [ceph] [ 2221.446041] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2221.450772] ceph_fill_trace+0x143/0xf30 [ceph] [ 2221.455317] ? __mutex_lock+0x97/0xce0 [ 2221.459068] ? __mutex_lock+0xaec/0xce0 [ 2221.462909] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2221.467462] ? lock_downgrade+0xfd/0x250 [ 2221.471392] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2221.475770] ceph_con_process_message+0x74/0x150 [libceph] [ 2221.481262] ? calc_signature+0xdb/0x100 [libceph] [ 2221.486072] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2221.491400] ? process_scheduled_works+0x22a/0x510 [ 2221.496201] ? process_scheduled_works+0x22a/0x510 [ 2221.500989] ceph_con_workfn+0x344/0x750 [libceph] [ 2221.505798] ? process_scheduled_works+0x22a/0x510 [ 2221.510593] process_scheduled_works+0x2a4/0x510 [ 2221.515222] worker_thread+0x188/0x340 [ 2221.518977] ? rescuer_thread+0x310/0x310 [ 2221.522994] kthread+0xf5/0x130 [ 2221.526142] ? kthread_complete_and_exit+0x20/0x20 [ 2221.530938] ret_from_fork+0x31/0x40 [ 2221.534522] ? kthread_complete_and_exit+0x20/0x20 [ 2221.539315] ret_from_fork_asm+0x11/0x20 [ 2221.543259] [ 2221.545449] irq event stamp: 10385281 [ 2221.549116] hardirqs last enabled at (10385291): [] console_unlock+0x106/0x120 [ 2221.557982] hardirqs last disabled at (10385300): [] console_unlock+0xeb/0x120 [ 2221.566762] softirqs last enabled at (10385236): [] __do_softirq+0x2be/0x371 [ 2221.575455] softirqs last disabled at (10385231): [] irq_exit_rcu+0xb0/0xd0 [ 2221.583974] ---[ end trace 0000000000000000 ]--- [ 2221.588594] ------------[ cut here ]------------ [ 2221.593212] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2221.604861] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2221.613813] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2221.669272] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2221.678225] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2221.685703] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2221.691190] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2221.709936] RSP: 0018:ffffc90001fdfbf0 EFLAGS: 00010282 [ 2221.715161] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2221.722285] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2221.729410] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003f03 [ 2221.736535] R10: 0000000000179ca0 R11: ffffc90001fdfa80 R12: ffff8881c5350750 [ 2221.743667] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2221.750800] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2221.758885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2221.764623] CR2: 000055a0beba2290 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2221.771747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2221.778871] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2221.786012] Call Trace: [ 2221.788459] [ 2221.790564] ? __warn+0x85/0x180 [ 2221.793796] ? __list_add_valid_or_report+0x7d/0xa0 [ 2221.798677] ? report_bug+0xfc/0x1d0 [ 2221.802256] ? prb_read_valid+0x1b/0x20 [ 2221.806097] ? handle_bug+0x44/0x70 [ 2221.809589] ? exc_invalid_op+0x17/0x60 [ 2221.813427] ? asm_exc_invalid_op+0x1a/0x20 [ 2221.817618] ? __list_add_valid_or_report+0x7d/0xa0 [ 2221.822494] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2221.827384] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2221.832100] lookup_fast+0x6d/0xf0 [ 2221.835501] walk_component+0x26/0x160 [ 2221.839256] link_path_walk+0x24d/0x380 [ 2221.843094] ? path_openat+0x80/0x9d0 [ 2221.846763] path_openat+0x8e/0x9d0 [ 2221.850256] do_filp_open+0xb8/0x120 [ 2221.853834] ? lock_release+0xc5/0x260 [ 2221.857587] ? _raw_spin_unlock+0x23/0x40 [ 2221.861597] ? alloc_fd+0x13f/0x200 [ 2221.865093] do_sys_openat2+0x249/0x290 [ 2221.868932] do_sys_open+0x4f/0x80 [ 2221.872337] do_syscall_64+0x38/0x80 [ 2221.875916] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2221.880967] RIP: 0033:0x7f48f0ef2805 [ 2221.884546] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2221.903294] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2221.910858] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2221.917982] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2221.925114] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2221.932243] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2221.939372] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2221.946503] [ 2221.948691] irq event stamp: 12824005 [ 2221.952356] hardirqs last enabled at (12824015): [] console_unlock+0x106/0x120 [ 2221.961223] hardirqs last disabled at (12824024): [] console_unlock+0xeb/0x120 [ 2221.970002] softirqs last enabled at (12823972): [] __do_softirq+0x2be/0x371 [ 2221.978694] softirqs last disabled at (12823967): [] irq_exit_rcu+0xb0/0xd0 [ 2221.987213] ---[ end trace 0000000000000000 ]--- [ 2221.991824] ------------[ cut here ]------------ [ 2221.996443] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2222.008094] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2222.016789] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2222.072276] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2222.081407] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2222.088887] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2222.094389] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2222.099875] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2222.118623] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2222.123848] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2222.130980] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2222.138117] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003f3e [ 2222.145246] R10: 000000000017acd8 R11: ffffc9000030b828 R12: ffff8881c2cefcb0 [ 2222.152378] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2222.159511] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2222.167598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2222.173345] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2222.180476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2222.187610] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2222.194741] Call Trace: [ 2222.197196] [ 2222.199302] ? __warn+0x85/0x180 [ 2222.202539] ? __list_add_valid_or_report+0x7d/0xa0 [ 2222.207424] ? report_bug+0xfc/0x1d0 [ 2222.211004] ? prb_read_valid+0x1b/0x20 [ 2222.214846] ? handle_bug+0x44/0x70 [ 2222.218334] ? exc_invalid_op+0x17/0x60 [ 2222.222174] ? asm_exc_invalid_op+0x1a/0x20 [ 2222.226370] ? __list_add_valid_or_report+0x7d/0xa0 [ 2222.231249] ? __list_add_valid_or_report+0x7d/0xa0 [ 2222.236128] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2222.241027] ceph_add_cap+0x20b/0x620 [ceph] [ 2222.245319] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2222.250040] ? lock_acquire+0xcb/0x2b0 [ 2222.253796] ? lock_is_held_type+0xaa/0x120 [ 2222.257991] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2222.262452] ? __mutex_lock+0x97/0xce0 [ 2222.266212] ? __mutex_lock+0xaec/0xce0 [ 2222.270052] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2222.274600] ? lock_downgrade+0xfd/0x250 [ 2222.278534] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2222.282914] ceph_con_process_message+0x74/0x150 [libceph] [ 2222.288414] ? calc_signature+0xdb/0x100 [libceph] [ 2222.293230] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2222.298558] ? process_scheduled_works+0x22a/0x510 [ 2222.303348] ? process_scheduled_works+0x22a/0x510 [ 2222.308141] ceph_con_workfn+0x344/0x750 [libceph] [ 2222.312954] ? process_scheduled_works+0x22a/0x510 [ 2222.317742] process_scheduled_works+0x2a4/0x510 [ 2222.322368] worker_thread+0x188/0x340 [ 2222.326127] ? rescuer_thread+0x310/0x310 [ 2222.330147] kthread+0xf5/0x130 [ 2222.333291] ? kthread_complete_and_exit+0x20/0x20 [ 2222.338086] ret_from_fork+0x31/0x40 [ 2222.341663] ? kthread_complete_and_exit+0x20/0x20 [ 2222.346457] ret_from_fork_asm+0x11/0x20 [ 2222.350389] [ 2222.352596] irq event stamp: 10387367 [ 2222.356265] hardirqs last enabled at (10387377): [] console_unlock+0x106/0x120 [ 2222.365131] hardirqs last disabled at (10387386): [] console_unlock+0xeb/0x120 [ 2222.373922] softirqs last enabled at (10387410): [] __do_softirq+0x2be/0x371 [ 2222.382614] softirqs last disabled at (10387397): [] irq_exit_rcu+0xb0/0xd0 [ 2222.391133] ---[ end trace 0000000000000000 ]--- [ 2222.395760] ------------[ cut here ]------------ [ 2222.400383] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2222.412030] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2222.420984] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2222.476512] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2222.485464] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2222.492944] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2222.498430] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2222.517177] RSP: 0018:ffffc90001ff7a78 EFLAGS: 00010286 [ 2222.522402] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2222.529539] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2222.536669] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003f7e [ 2222.543804] R10: 000000000017bf70 R11: ffffc90001ff7908 R12: ffff8881d1329ae0 [ 2222.550935] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2222.558067] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2222.566155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2222.571900] CR2: 00007fb38d5ba060 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2222.579044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2222.586182] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2222.593316] Call Trace: [ 2222.595768] [ 2222.597876] ? __warn+0x85/0x180 [ 2222.601115] ? __list_add_valid_or_report+0x7d/0xa0 [ 2222.606009] ? report_bug+0xfc/0x1d0 [ 2222.609590] ? prb_read_valid+0x1b/0x20 [ 2222.613433] ? handle_bug+0x44/0x70 [ 2222.616933] ? exc_invalid_op+0x17/0x60 [ 2222.620772] ? asm_exc_invalid_op+0x1a/0x20 [ 2222.624976] ? __list_add_valid_or_report+0x7d/0xa0 [ 2222.629857] ? __list_add_valid_or_report+0x7d/0xa0 [ 2222.634742] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2222.639653] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2222.644907] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2222.649722] ceph_lookup+0xf9/0x3f0 [ceph] [ 2222.653842] __lookup_slow+0xff/0x1a0 [ 2222.657528] walk_component+0x83/0x160 [ 2222.661283] path_lookupat.isra.40+0x58/0x120 [ 2222.665652] filename_lookup+0xd4/0x190 [ 2222.669504] ? lock_acquire+0xcb/0x2b0 [ 2222.673257] ? __might_fault+0x45/0x80 [ 2222.677018] ? find_held_lock+0x2d/0x90 [ 2222.680860] ? __might_fault+0x45/0x80 [ 2222.684627] vfs_statx+0x89/0x150 [ 2222.687955] vfs_fstatat+0x5a/0xc0 [ 2222.691366] __do_sys_newstat+0x30/0x70 [ 2222.695213] ? lock_release+0xc5/0x260 [ 2222.698970] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2222.703851] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2222.708992] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2222.714132] do_syscall_64+0x38/0x80 [ 2222.717719] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2222.722771] RIP: 0033:0x7f48f0eeca79 [ 2222.726350] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2222.745097] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2222.752664] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2222.759798] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2222.766928] RBP: 00007ffc53d60520 R08: 00000000000029f5 R09: 00000000020767c0 [ 2222.774060] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2222.781193] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2222.788341] [ 2222.790537] irq event stamp: 12882789 [ 2222.794202] hardirqs last enabled at (12882799): [] console_unlock+0x106/0x120 [ 2222.803068] hardirqs last disabled at (12882808): [] console_unlock+0xeb/0x120 [ 2222.811848] softirqs last enabled at (12882736): [] __do_softirq+0x2be/0x371 [ 2222.820549] softirqs last disabled at (12882731): [] irq_exit_rcu+0xb0/0xd0 [ 2222.829069] ---[ end trace 0000000000000000 ]--- [ 2222.833691] ------------[ cut here ]------------ [ 2222.838315] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2222.849980] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2222.858706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2222.914201] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2222.923329] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2222.930809] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2222.936320] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2222.941805] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2222.960558] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2222.965788] RAX: 0000000000000000 RBX: ffff8881a1d33ab0 RCX: 0000000000000000 [ 2222.972920] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2222.980055] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003fc0 [ 2222.987184] R10: 000000000017d110 R11: ffffc9000030b828 R12: ffff8881a1d34200 [ 2222.994318] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2223.001450] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2223.009536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2223.015285] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2223.022417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2223.029550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2223.036681] Call Trace: [ 2223.039135] [ 2223.041240] ? __warn+0x85/0x180 [ 2223.044475] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.049354] ? report_bug+0xfc/0x1d0 [ 2223.052932] ? prb_read_valid+0x1b/0x20 [ 2223.056773] ? handle_bug+0x44/0x70 [ 2223.060265] ? exc_invalid_op+0x17/0x60 [ 2223.064103] ? asm_exc_invalid_op+0x1a/0x20 [ 2223.068294] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.073170] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.078048] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2223.082939] ceph_add_cap+0x20b/0x620 [ceph] [ 2223.087225] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2223.091947] ? lock_acquire+0xcb/0x2b0 [ 2223.095707] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2223.100159] ? __mutex_lock+0x97/0xce0 [ 2223.103909] ? __mutex_lock+0xaec/0xce0 [ 2223.107759] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2223.112310] ? lock_downgrade+0xfd/0x250 [ 2223.116239] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2223.120620] ceph_con_process_message+0x74/0x150 [libceph] [ 2223.126115] ? calc_signature+0xdb/0x100 [libceph] [ 2223.130928] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2223.136260] ? process_scheduled_works+0x22a/0x510 [ 2223.141058] ? process_scheduled_works+0x22a/0x510 [ 2223.145849] ceph_con_workfn+0x344/0x750 [libceph] [ 2223.150654] ? process_scheduled_works+0x22a/0x510 [ 2223.155442] process_scheduled_works+0x2a4/0x510 [ 2223.160068] worker_thread+0x188/0x340 [ 2223.163827] ? rescuer_thread+0x310/0x310 [ 2223.167838] kthread+0xf5/0x130 [ 2223.170984] ? kthread_complete_and_exit+0x20/0x20 [ 2223.175776] ret_from_fork+0x31/0x40 [ 2223.179356] ? kthread_complete_and_exit+0x20/0x20 [ 2223.184147] ret_from_fork_asm+0x11/0x20 [ 2223.188082] [ 2223.190279] irq event stamp: 10389729 [ 2223.193939] hardirqs last enabled at (10389739): [] console_unlock+0x106/0x120 [ 2223.202808] hardirqs last disabled at (10389748): [] console_unlock+0xeb/0x120 [ 2223.211586] softirqs last enabled at (10389676): [] __do_softirq+0x2be/0x371 [ 2223.220282] softirqs last disabled at (10389671): [] irq_exit_rcu+0xb0/0xd0 [ 2223.228798] ---[ end trace 0000000000000000 ]--- [ 2223.233421] ------------[ cut here ]------------ [ 2223.238038] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2223.249690] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2223.258649] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2223.314173] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2223.323129] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2223.330610] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2223.336097] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2223.354841] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2223.360067] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2223.367202] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2223.374334] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100003fff [ 2223.381470] R10: 000000000017e368 R11: ffffc90001fd79c8 R12: ffff888158a68750 [ 2223.388599] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2223.395732] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2223.403818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2223.409564] CR2: 00007fb38d5c4000 CR3: 0000000120c1e004 CR4: 00000000003706f0 [ 2223.416697] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2223.423831] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2223.430964] Call Trace: [ 2223.433415] [ 2223.435523] ? __warn+0x85/0x180 [ 2223.438757] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.443636] ? report_bug+0xfc/0x1d0 [ 2223.447216] ? prb_read_valid+0x1b/0x20 [ 2223.451066] ? handle_bug+0x44/0x70 [ 2223.454564] ? exc_invalid_op+0x17/0x60 [ 2223.458403] ? asm_exc_invalid_op+0x1a/0x20 [ 2223.462597] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.467479] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.472356] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2223.477257] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2223.481985] lookup_fast+0x6d/0xf0 [ 2223.485392] walk_component+0x26/0x160 [ 2223.489149] link_path_walk+0x24d/0x380 [ 2223.492994] ? path_lookupat.isra.40+0x22/0x120 [ 2223.497531] path_lookupat.isra.40+0x6b/0x120 [ 2223.501898] filename_lookup+0xd4/0x190 [ 2223.505750] ? lock_acquire+0xcb/0x2b0 [ 2223.509505] ? __might_fault+0x45/0x80 [ 2223.513267] ? find_held_lock+0x2d/0x90 [ 2223.517112] ? __might_fault+0x45/0x80 [ 2223.520871] vfs_statx+0x89/0x150 [ 2223.524208] vfs_fstatat+0x5a/0xc0 [ 2223.527633] __do_sys_newstat+0x30/0x70 [ 2223.531474] ? lock_release+0xc5/0x260 [ 2223.535229] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2223.540112] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2223.545254] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2223.550395] do_syscall_64+0x38/0x80 [ 2223.553981] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2223.559032] RIP: 0033:0x7f48f0eeca79 [ 2223.562613] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2223.581362] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2223.588934] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2223.596066] RDX: 00007ffc53d5ffb0 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2223.603199] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2223.610332] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2223.617463] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2223.624611] [ 2223.626807] irq event stamp: 12944493 [ 2223.630473] hardirqs last enabled at (12944503): [] console_unlock+0x106/0x120 [ 2223.639339] hardirqs last disabled at (12944512): [] console_unlock+0xeb/0x120 [ 2223.648122] softirqs last enabled at (12944526): [] __do_softirq+0x2be/0x371 [ 2223.656820] softirqs last disabled at (12944521): [] irq_exit_rcu+0xb0/0xd0 [ 2223.665340] ---[ end trace 0000000000000000 ]--- [ 2223.669960] ------------[ cut here ]------------ [ 2223.674582] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2223.686239] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2223.694999] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2223.750547] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2223.759681] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2223.767160] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2223.772670] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2223.778156] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2223.796905] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2223.802139] RAX: 0000000000000000 RBX: ffff8881d09561d0 RCX: 0000000000000000 [ 2223.809271] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2223.816407] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004041 [ 2223.823546] R10: 000000000017f508 R11: ffffc9000030b828 R12: ffff8881d0956920 [ 2223.830679] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2223.837811] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2223.845897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2223.851656] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2223.858794] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2223.865926] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2223.873060] Call Trace: [ 2223.875512] [ 2223.877621] ? __warn+0x85/0x180 [ 2223.880863] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.885750] ? report_bug+0xfc/0x1d0 [ 2223.889336] ? prb_read_valid+0x1b/0x20 [ 2223.893189] ? handle_bug+0x44/0x70 [ 2223.896687] ? exc_invalid_op+0x17/0x60 [ 2223.900525] ? asm_exc_invalid_op+0x1a/0x20 [ 2223.904724] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.909610] ? __list_add_valid_or_report+0x7d/0xa0 [ 2223.914495] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2223.919398] ceph_add_cap+0x20b/0x620 [ceph] [ 2223.923705] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2223.928455] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2223.933808] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2223.938301] ceph_con_process_message+0x74/0x150 [libceph] [ 2223.943806] ? calc_signature+0xdb/0x100 [libceph] [ 2223.948630] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2223.953972] ? process_scheduled_works+0x22a/0x510 [ 2223.958772] ? process_scheduled_works+0x22a/0x510 [ 2223.963565] ceph_con_workfn+0x344/0x750 [libceph] [ 2223.968389] ? process_scheduled_works+0x22a/0x510 [ 2223.973189] process_scheduled_works+0x2a4/0x510 [ 2223.977828] worker_thread+0x188/0x340 [ 2223.981589] ? rescuer_thread+0x310/0x310 [ 2223.985607] kthread+0xf5/0x130 [ 2223.988763] ? kthread_complete_and_exit+0x20/0x20 [ 2223.993565] ret_from_fork+0x31/0x40 [ 2223.997150] ? kthread_complete_and_exit+0x20/0x20 [ 2224.001944] ret_from_fork_asm+0x11/0x20 [ 2224.005893] [ 2224.008091] irq event stamp: 10391839 [ 2224.011763] hardirqs last enabled at (10391849): [] console_unlock+0x106/0x120 [ 2224.020629] hardirqs last disabled at (10391858): [] console_unlock+0xeb/0x120 [ 2224.029418] softirqs last enabled at (10391836): [] __do_softirq+0x2be/0x371 [ 2224.038119] softirqs last disabled at (10391831): [] irq_exit_rcu+0xb0/0xd0 [ 2224.046646] ---[ end trace 0000000000000000 ]--- [ 2224.051267] ------------[ cut here ]------------ [ 2224.055895] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2224.067542] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2224.076493] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2224.132068] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2224.141066] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2224.148542] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2224.154027] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2224.172773] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2224.177998] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2224.185132] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2224.192265] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000407b [ 2224.199397] R10: 00000000001807f0 R11: ffffc90001fd79c8 R12: ffff8881c5350750 [ 2224.206530] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2224.213663] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2224.221750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2224.227495] CR2: 00007fb38d5d5000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2224.234629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2224.241762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2224.248893] Call Trace: [ 2224.251347] [ 2224.253454] ? __warn+0x85/0x180 [ 2224.256687] ? __list_add_valid_or_report+0x7d/0xa0 [ 2224.261566] ? report_bug+0xfc/0x1d0 [ 2224.265146] ? prb_read_valid+0x1b/0x20 [ 2224.268987] ? handle_bug+0x44/0x70 [ 2224.272477] ? exc_invalid_op+0x17/0x60 [ 2224.276316] ? asm_exc_invalid_op+0x1a/0x20 [ 2224.280513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2224.285392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2224.290270] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2224.295170] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2224.299901] lookup_fast+0x6d/0xf0 [ 2224.303307] walk_component+0x26/0x160 [ 2224.307061] link_path_walk+0x24d/0x380 [ 2224.310902] ? path_lookupat.isra.40+0x22/0x120 [ 2224.315444] path_lookupat.isra.40+0x6b/0x120 [ 2224.319810] filename_lookup+0xd4/0x190 [ 2224.323655] ? lock_acquire+0xcb/0x2b0 [ 2224.327408] ? __might_fault+0x45/0x80 [ 2224.331162] ? find_held_lock+0x2d/0x90 [ 2224.335008] ? __might_fault+0x45/0x80 [ 2224.338769] vfs_statx+0x89/0x150 [ 2224.342095] vfs_fstatat+0x5a/0xc0 [ 2224.345506] __do_sys_newstat+0x30/0x70 [ 2224.349344] ? lock_release+0xc5/0x260 [ 2224.353101] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2224.357983] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2224.363125] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2224.368266] do_syscall_64+0x38/0x80 [ 2224.371853] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2224.376905] RIP: 0033:0x7f48f0eeca79 [ 2224.380484] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2224.399230] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2224.406796] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2224.413929] RDX: 00007ffc53d5ffb0 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2224.421061] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2224.428194] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2224.435326] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2224.442477] [ 2224.444670] irq event stamp: 12946385 [ 2224.448334] hardirqs last enabled at (12946395): [] console_unlock+0x106/0x120 [ 2224.457202] hardirqs last disabled at (12946404): [] console_unlock+0xeb/0x120 [ 2224.465982] softirqs last enabled at (12946372): [] __do_softirq+0x2be/0x371 [ 2224.474674] softirqs last disabled at (12946365): [] irq_exit_rcu+0xb0/0xd0 [ 2224.483194] ---[ end trace 0000000000000000 ]--- [ 2224.487814] ------------[ cut here ]------------ [ 2224.492430] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2224.504080] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2224.512777] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2224.568319] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2224.577450] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2224.584935] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2224.590445] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2224.595931] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2224.614679] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2224.619906] RAX: 0000000000000000 RBX: ffff8881d10609c8 RCX: 0000000000000000 [ 2224.627039] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2224.634170] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000040bd [ 2224.641310] R10: 0000000000181960 R11: ffffc9000030b828 R12: ffff8881d1061118 [ 2224.648446] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2224.655578] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2224.663665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2224.669410] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2224.676544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2224.683675] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2224.690809] Call Trace: [ 2224.693262] [ 2224.695368] ? __warn+0x85/0x180 [ 2224.698602] ? __list_add_valid_or_report+0x7d/0xa0 [ 2224.703492] ? report_bug+0xfc/0x1d0 [ 2224.707078] ? prb_read_valid+0x1b/0x20 [ 2224.710930] ? handle_bug+0x44/0x70 [ 2224.714427] ? exc_invalid_op+0x17/0x60 [ 2224.718268] ? asm_exc_invalid_op+0x1a/0x20 [ 2224.722473] ? __list_add_valid_or_report+0x7d/0xa0 [ 2224.727361] ? __list_add_valid_or_report+0x7d/0xa0 [ 2224.732245] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2224.737146] ceph_add_cap+0x20b/0x620 [ceph] [ 2224.741454] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2224.746196] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2224.751548] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2224.756035] ceph_con_process_message+0x74/0x150 [libceph] [ 2224.761539] ? calc_signature+0xdb/0x100 [libceph] [ 2224.766362] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2224.771707] ? process_scheduled_works+0x22a/0x510 [ 2224.776512] ? process_scheduled_works+0x22a/0x510 [ 2224.781308] ceph_con_workfn+0x344/0x750 [libceph] [ 2224.786124] ? process_scheduled_works+0x22a/0x510 [ 2224.790920] process_scheduled_works+0x2a4/0x510 [ 2224.795551] worker_thread+0x188/0x340 [ 2224.799312] ? rescuer_thread+0x310/0x310 [ 2224.803331] kthread+0xf5/0x130 [ 2224.806486] ? kthread_complete_and_exit+0x20/0x20 [ 2224.811289] ret_from_fork+0x31/0x40 [ 2224.814873] ? kthread_complete_and_exit+0x20/0x20 [ 2224.819669] ret_from_fork_asm+0x11/0x20 [ 2224.823619] [ 2224.825811] irq event stamp: 10393857 [ 2224.829478] hardirqs last enabled at (10393867): [] console_unlock+0x106/0x120 [ 2224.838354] hardirqs last disabled at (10393882): [] console_unlock+0xeb/0x120 [ 2224.847141] softirqs last enabled at (10393880): [] __do_softirq+0x2be/0x371 [ 2224.855848] softirqs last disabled at (10393903): [] irq_exit_rcu+0xb0/0xd0 [ 2224.864372] ---[ end trace 0000000000000000 ]--- [ 2224.868990] ------------[ cut here ]------------ [ 2224.873606] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2224.885255] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2224.894209] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2224.949735] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2224.958690] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2224.966173] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2224.971663] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2224.990410] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2224.995641] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2225.002784] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2225.009918] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000040f7 [ 2225.017053] R10: 0000000000182ab8 R11: ffffc90001fd79c8 R12: ffff8881d10b72e8 [ 2225.024234] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2225.031417] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2225.039508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2225.045253] CR2: 00007fb38d5d5000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2225.052386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2225.059519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2225.066651] Call Trace: [ 2225.069106] [ 2225.071212] ? __warn+0x85/0x180 [ 2225.074445] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.079325] ? report_bug+0xfc/0x1d0 [ 2225.082903] ? prb_read_valid+0x1b/0x20 [ 2225.086745] ? handle_bug+0x44/0x70 [ 2225.090236] ? exc_invalid_op+0x17/0x60 [ 2225.094075] ? asm_exc_invalid_op+0x1a/0x20 [ 2225.098271] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.103150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.108028] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2225.112929] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2225.117657] lookup_fast+0x6d/0xf0 [ 2225.121066] walk_component+0x26/0x160 [ 2225.124819] link_path_walk+0x24d/0x380 [ 2225.128655] ? path_lookupat.isra.40+0x22/0x120 [ 2225.133194] path_lookupat.isra.40+0x6b/0x120 [ 2225.137560] filename_lookup+0xd4/0x190 [ 2225.141412] ? lock_acquire+0xcb/0x2b0 [ 2225.145165] ? __might_fault+0x45/0x80 [ 2225.148920] ? find_held_lock+0x2d/0x90 [ 2225.152758] ? __might_fault+0x45/0x80 [ 2225.156518] vfs_statx+0x89/0x150 [ 2225.159846] vfs_fstatat+0x5a/0xc0 [ 2225.163256] __do_sys_newstat+0x30/0x70 [ 2225.167093] ? lock_release+0xc5/0x260 [ 2225.170849] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2225.175733] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2225.180874] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2225.186016] do_syscall_64+0x38/0x80 [ 2225.189602] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2225.194652] RIP: 0033:0x7f48f0eeca79 [ 2225.198233] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2225.216980] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2225.224547] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2225.231678] RDX: 00007ffc53d5ffb0 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2225.238810] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2225.245943] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2225.253076] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2225.260223] [ 2225.262419] irq event stamp: 12948279 [ 2225.266086] hardirqs last enabled at (12948289): [] console_unlock+0x106/0x120 [ 2225.274950] hardirqs last disabled at (12948298): [] console_unlock+0xeb/0x120 [ 2225.283731] softirqs last enabled at (12948216): [] __do_softirq+0x2be/0x371 [ 2225.292423] softirqs last disabled at (12948211): [] irq_exit_rcu+0xb0/0xd0 [ 2225.300944] ---[ end trace 0000000000000000 ]--- [ 2225.305566] ------------[ cut here ]------------ [ 2225.310188] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2225.321834] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2225.330791] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2225.386257] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2225.395210] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2225.402689] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2225.408176] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2225.426924] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2225.432147] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2225.439280] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2225.446413] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004139 [ 2225.453545] R10: 0000000000183c98 R11: ffffc90001fdfb18 R12: ffff8881d1146920 [ 2225.460680] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2225.467810] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2225.475888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2225.481634] CR2: 00007f9bd2981028 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2225.488767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2225.495900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2225.503027] Call Trace: [ 2225.505477] [ 2225.507583] ? __warn+0x85/0x180 [ 2225.510817] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.515700] ? report_bug+0xfc/0x1d0 [ 2225.519285] ? prb_read_valid+0x1b/0x20 [ 2225.523126] ? handle_bug+0x44/0x70 [ 2225.526618] ? exc_invalid_op+0x17/0x60 [ 2225.530455] ? asm_exc_invalid_op+0x1a/0x20 [ 2225.534646] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.539522] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.544401] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2225.549290] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2225.554007] lookup_fast+0x6d/0xf0 [ 2225.557410] path_openat+0xe4/0x9d0 [ 2225.560905] do_filp_open+0xb8/0x120 [ 2225.564484] ? lock_release+0xc5/0x260 [ 2225.568236] ? _raw_spin_unlock+0x23/0x40 [ 2225.572246] ? alloc_fd+0x13f/0x200 [ 2225.575743] do_sys_openat2+0x249/0x290 [ 2225.579582] do_sys_open+0x4f/0x80 [ 2225.582987] do_syscall_64+0x38/0x80 [ 2225.586567] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2225.591620] RIP: 0033:0x7f48f0ef2805 [ 2225.595197] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2225.613944] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2225.621510] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2225.628643] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2225.635776] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2225.642908] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2225.650041] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2225.657181] [ 2225.659374] irq event stamp: 12832151 [ 2225.663039] hardirqs last enabled at (12832161): [] console_unlock+0x106/0x120 [ 2225.671906] hardirqs last disabled at (12832170): [] console_unlock+0xeb/0x120 [ 2225.680686] softirqs last enabled at (12832138): [] __do_softirq+0x2be/0x371 [ 2225.689381] softirqs last disabled at (12832189): [] irq_exit_rcu+0xb0/0xd0 [ 2225.697898] ---[ end trace 0000000000000000 ]--- [ 2225.702518] ------------[ cut here ]------------ [ 2225.707141] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2225.718787] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2225.727741] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2225.783263] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2225.792212] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2225.799693] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2225.805180] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2225.823926] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2225.829152] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2225.836283] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2225.843417] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004172 [ 2225.850550] R10: 0000000000184cd8 R11: ffffc90001fd79c8 R12: ffff8881d10b0750 [ 2225.857683] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2225.864814] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2225.872901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2225.878648] CR2: 00007fb38d5d5000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2225.885781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2225.892913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2225.900046] Call Trace: [ 2225.902499] [ 2225.904605] ? __warn+0x85/0x180 [ 2225.907838] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.912720] ? report_bug+0xfc/0x1d0 [ 2225.916306] ? prb_read_valid+0x1b/0x20 [ 2225.920148] ? handle_bug+0x44/0x70 [ 2225.923686] ? exc_invalid_op+0x17/0x60 [ 2225.927576] ? asm_exc_invalid_op+0x1a/0x20 [ 2225.931778] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.936657] ? __list_add_valid_or_report+0x7d/0xa0 [ 2225.941534] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2225.946436] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2225.951163] lookup_fast+0x6d/0xf0 [ 2225.954572] walk_component+0x26/0x160 [ 2225.958329] link_path_walk+0x24d/0x380 [ 2225.962170] ? path_lookupat.isra.40+0x22/0x120 [ 2225.966709] path_lookupat.isra.40+0x6b/0x120 [ 2225.971073] filename_lookup+0xd4/0x190 [ 2225.974919] ? lock_acquire+0xcb/0x2b0 [ 2225.978672] ? __might_fault+0x45/0x80 [ 2225.982427] ? find_held_lock+0x2d/0x90 [ 2225.986274] ? __might_fault+0x45/0x80 [ 2225.990033] vfs_statx+0x89/0x150 [ 2225.993363] vfs_fstatat+0x5a/0xc0 [ 2225.996772] __do_sys_newstat+0x30/0x70 [ 2226.000622] ? lock_release+0xc5/0x260 [ 2226.004383] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2226.009266] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2226.014408] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2226.019556] do_syscall_64+0x38/0x80 [ 2226.023144] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2226.028203] RIP: 0033:0x7f48f0eeca79 [ 2226.031781] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2226.050530] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2226.058103] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2226.065237] RDX: 00007ffc53d5ffb0 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2226.072368] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2226.079501] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2226.086634] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2226.093781] [ 2226.095978] irq event stamp: 12950175 [ 2226.099643] hardirqs last enabled at (12950185): [] console_unlock+0x106/0x120 [ 2226.108510] hardirqs last disabled at (12950194): [] console_unlock+0xeb/0x120 [ 2226.117289] softirqs last enabled at (12950096): [] __do_softirq+0x2be/0x371 [ 2226.125982] softirqs last disabled at (12950091): [] irq_exit_rcu+0xb0/0xd0 [ 2226.134501] ---[ end trace 0000000000000000 ]--- [ 2226.139121] ------------[ cut here ]------------ [ 2226.143737] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2226.155385] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2226.164081] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2226.219557] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2226.228681] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2226.236162] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2226.241658] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2226.247142] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2226.265888] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2226.271114] RAX: 0000000000000000 RBX: ffff8881d10b1390 RCX: 0000000000000000 [ 2226.278248] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2226.285382] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000041b4 [ 2226.292512] R10: 0000000000185f20 R11: ffffc9000030b828 R12: ffff8881d10b1ae0 [ 2226.299646] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2226.306779] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2226.314865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2226.320611] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2226.327744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2226.334877] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2226.342008] Call Trace: [ 2226.344463] [ 2226.346567] ? __warn+0x85/0x180 [ 2226.349803] ? __list_add_valid_or_report+0x7d/0xa0 [ 2226.354681] ? report_bug+0xfc/0x1d0 [ 2226.358259] ? prb_read_valid+0x1b/0x20 [ 2226.362101] ? handle_bug+0x44/0x70 [ 2226.365590] ? exc_invalid_op+0x17/0x60 [ 2226.369431] ? asm_exc_invalid_op+0x1a/0x20 [ 2226.373623] ? __list_add_valid_or_report+0x7d/0xa0 [ 2226.378507] ? __list_add_valid_or_report+0x7d/0xa0 [ 2226.383384] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2226.388275] ceph_add_cap+0x20b/0x620 [ceph] [ 2226.392560] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2226.397288] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2226.402616] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2226.407079] ceph_con_process_message+0x74/0x150 [libceph] [ 2226.412576] ? calc_signature+0xdb/0x100 [libceph] [ 2226.417388] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2226.422718] ? process_scheduled_works+0x22a/0x510 [ 2226.427509] ? process_scheduled_works+0x22a/0x510 [ 2226.432300] ceph_con_workfn+0x344/0x750 [libceph] [ 2226.437103] ? process_scheduled_works+0x22a/0x510 [ 2226.441894] process_scheduled_works+0x2a4/0x510 [ 2226.446519] worker_thread+0x188/0x340 [ 2226.450269] ? rescuer_thread+0x310/0x310 [ 2226.454279] kthread+0xf5/0x130 [ 2226.457424] ? kthread_complete_and_exit+0x20/0x20 [ 2226.462220] ret_from_fork+0x31/0x40 [ 2226.465798] ? kthread_complete_and_exit+0x20/0x20 [ 2226.470591] ret_from_fork_asm+0x11/0x20 [ 2226.474523] [ 2226.476718] irq event stamp: 10397903 [ 2226.480383] hardirqs last enabled at (10397913): [] console_unlock+0x106/0x120 [ 2226.489250] hardirqs last disabled at (10397922): [] console_unlock+0xeb/0x120 [ 2226.498031] softirqs last enabled at (10397936): [] __do_softirq+0x2be/0x371 [ 2226.506722] softirqs last disabled at (10397931): [] irq_exit_rcu+0xb0/0xd0 [ 2226.515242] ---[ end trace 0000000000000000 ]--- [ 2226.519860] ------------[ cut here ]------------ [ 2226.524494] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2226.536143] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2226.545107] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2226.600630] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2226.609581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2226.617062] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2226.622548] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2226.641293] RSP: 0018:ffffc90001fd7b98 EFLAGS: 00010282 [ 2226.646521] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2226.653655] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2226.660786] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000041ee [ 2226.667917] R10: 0000000000186f00 R11: ffffc90001fd7a28 R12: ffff8881c2cefcb0 [ 2226.675059] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2226.682194] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2226.690279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2226.696026] CR2: 00007fb38d5ba060 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2226.703157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2226.710291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2226.717424] Call Trace: [ 2226.719875] [ 2226.721982] ? __warn+0x85/0x180 [ 2226.725215] ? __list_add_valid_or_report+0x7d/0xa0 [ 2226.730095] ? report_bug+0xfc/0x1d0 [ 2226.733673] ? prb_read_valid+0x1b/0x20 [ 2226.737516] ? handle_bug+0x44/0x70 [ 2226.741005] ? exc_invalid_op+0x17/0x60 [ 2226.744846] ? asm_exc_invalid_op+0x1a/0x20 [ 2226.749041] ? __list_add_valid_or_report+0x7d/0xa0 [ 2226.753922] ? __list_add_valid_or_report+0x7d/0xa0 [ 2226.758807] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2226.763710] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2226.768435] lookup_fast+0x6d/0xf0 [ 2226.771844] walk_component+0x26/0x160 [ 2226.775599] path_lookupat.isra.40+0x58/0x120 [ 2226.779967] filename_lookup+0xd4/0x190 [ 2226.783819] ? lock_acquire+0xcb/0x2b0 [ 2226.787572] ? __might_fault+0x45/0x80 [ 2226.791328] ? find_held_lock+0x2d/0x90 [ 2226.795175] ? __might_fault+0x45/0x80 [ 2226.798934] vfs_statx+0x89/0x150 [ 2226.802261] vfs_fstatat+0x5a/0xc0 [ 2226.805672] __do_sys_newstat+0x30/0x70 [ 2226.809519] ? lock_release+0xc5/0x260 [ 2226.813274] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2226.818157] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2226.823299] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2226.828440] do_syscall_64+0x38/0x80 [ 2226.832025] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2226.837078] RIP: 0033:0x7f48f0eeca79 [ 2226.840658] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2226.859402] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2226.866970] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2226.874102] RDX: 00007ffc53d5ffb0 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2226.881233] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2226.888366] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2226.895499] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2226.902645] [ 2226.904834] irq event stamp: 12952061 [ 2226.908504] hardirqs last enabled at (12952077): [] console_unlock+0x106/0x120 [ 2226.917380] hardirqs last disabled at (12952092): [] console_unlock+0xeb/0x120 [ 2226.926163] softirqs last enabled at (12952090): [] __do_softirq+0x2be/0x371 [ 2226.934855] softirqs last disabled at (12952085): [] irq_exit_rcu+0xb0/0xd0 [ 2226.943376] ---[ end trace 0000000000000000 ]--- [ 2226.947998] ------------[ cut here ]------------ [ 2226.952622] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2226.964270] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2226.973229] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2227.028695] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2227.037653] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2227.045135] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2227.050629] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2227.069376] RSP: 0018:ffffc90001fffbf0 EFLAGS: 00010282 [ 2227.074609] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2227.081744] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2227.088875] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000422e [ 2227.096009] R10: 0000000000187ed8 R11: ffffc90001fffa80 R12: ffff888158a68750 [ 2227.103141] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2227.110272] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2227.118359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2227.124103] CR2: 000055fb2056a068 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2227.131237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2227.138379] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2227.145522] Call Trace: [ 2227.147973] [ 2227.150082] ? __warn+0x85/0x180 [ 2227.153321] ? __list_add_valid_or_report+0x7d/0xa0 [ 2227.158204] ? report_bug+0xfc/0x1d0 [ 2227.161790] ? prb_read_valid+0x1b/0x20 [ 2227.165632] ? handle_bug+0x44/0x70 [ 2227.169127] ? exc_invalid_op+0x17/0x60 [ 2227.172970] ? asm_exc_invalid_op+0x1a/0x20 [ 2227.177162] ? __list_add_valid_or_report+0x7d/0xa0 [ 2227.182047] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2227.186945] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2227.191679] lookup_fast+0x6d/0xf0 [ 2227.195087] walk_component+0x26/0x160 [ 2227.198843] link_path_walk+0x24d/0x380 [ 2227.202686] ? path_openat+0x80/0x9d0 [ 2227.206359] path_openat+0x8e/0x9d0 [ 2227.209860] do_filp_open+0xb8/0x120 [ 2227.213451] ? lock_release+0xc5/0x260 [ 2227.217207] ? _raw_spin_unlock+0x23/0x40 [ 2227.221227] ? alloc_fd+0x13f/0x200 [ 2227.224734] do_sys_openat2+0x249/0x290 [ 2227.228582] do_sys_open+0x4f/0x80 [ 2227.231991] do_syscall_64+0x38/0x80 [ 2227.235571] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2227.240621] RIP: 0033:0x7f48f0eed2f2 [ 2227.244201] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2227.262945] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2227.270514] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2227.277646] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2227.284780] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2227.291915] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2227.299046] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2227.306197] [ 2227.308389] irq event stamp: 12870917 [ 2227.312052] hardirqs last enabled at (12870927): [] console_unlock+0x106/0x120 [ 2227.320920] hardirqs last disabled at (12870936): [] console_unlock+0xeb/0x120 [ 2227.329703] softirqs last enabled at (12870952): [] __do_softirq+0x2be/0x371 [ 2227.338411] softirqs last disabled at (12870963): [] irq_exit_rcu+0xb0/0xd0 [ 2227.346937] ---[ end trace 0000000000000000 ]--- [ 2227.351558] ------------[ cut here ]------------ [ 2227.356175] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2227.367821] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2227.376775] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2227.432244] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2227.441201] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2227.448676] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2227.454161] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2227.472907] RSP: 0018:ffffc90001fdfd50 EFLAGS: 00010286 [ 2227.478135] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2227.485267] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2227.492401] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004269 [ 2227.499532] R10: 0000000000188eb0 R11: ffffc90001fdfbe0 R12: ffff8881d1144bc8 [ 2227.506665] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2227.513797] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2227.521883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2227.527631] CR2: 00007fb38e0dcd90 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2227.534763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2227.541896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2227.549028] Call Trace: [ 2227.551482] [ 2227.553589] ? __warn+0x85/0x180 [ 2227.556820] ? __list_add_valid_or_report+0x7d/0xa0 [ 2227.561700] ? report_bug+0xfc/0x1d0 [ 2227.565281] ? prb_read_valid+0x1b/0x20 [ 2227.569120] ? handle_bug+0x44/0x70 [ 2227.572612] ? exc_invalid_op+0x17/0x60 [ 2227.576452] ? asm_exc_invalid_op+0x1a/0x20 [ 2227.580640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2227.585518] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2227.590407] ceph_readdir+0xc0/0x1670 [ceph] [ 2227.594689] ? wrap_directory_iterator+0x38/0x70 [ 2227.599305] ? wrap_directory_iterator+0x30/0x70 [ 2227.603923] ? lock_is_held_type+0xaa/0x120 [ 2227.608113] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2227.612998] wrap_directory_iterator+0x47/0x70 [ 2227.617446] iterate_dir+0x8d/0x170 [ 2227.620940] __x64_sys_getdents64+0x87/0x140 [ 2227.625210] ? __x64_sys_getdents64+0x140/0x140 [ 2227.629749] do_syscall_64+0x38/0x80 [ 2227.633323] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2227.638374] RIP: 0033:0x7f48f0ec480b [ 2227.641953] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2227.660701] RSP: 002b:00007ffc53d5ff88 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2227.668267] RAX: ffffffffffffffda RBX: 000000000207e100 RCX: 00007f48f0ec480b [ 2227.675400] RDX: 0000000000010000 RSI: 000000000207e130 RDI: 0000000000000019 [ 2227.682532] RBP: 000000000207e130 R08: 00000000000000db R09: 0000000000000003 [ 2227.689665] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2227.696800] R13: 0000000000000002 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2227.703937] [ 2227.706123] irq event stamp: 12836761 [ 2227.709785] hardirqs last enabled at (12836771): [] console_unlock+0x106/0x120 [ 2227.718646] hardirqs last disabled at (12836780): [] console_unlock+0xeb/0x120 [ 2227.727427] softirqs last enabled at (12836738): [] __do_softirq+0x2be/0x371 [ 2227.736119] softirqs last disabled at (12836733): [] irq_exit_rcu+0xb0/0xd0 [ 2227.744638] ---[ end trace 0000000000000000 ]--- [ 2227.749260] ------------[ cut here ]------------ [ 2227.753894] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2227.765562] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2227.774527] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2227.830008] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2227.838973] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2227.846470] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2227.851976] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2227.870739] RSP: 0018:ffffc90001fffbf0 EFLAGS: 00010282 [ 2227.875968] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2227.883102] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2227.890234] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000042a1 [ 2227.897369] R10: 0000000000189e90 R11: ffffc90001fffa80 R12: ffff8881c5350750 [ 2227.904501] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2227.911631] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2227.919720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2227.925464] CR2: 00007f9bd2a2a000 CR3: 000000017eb7a003 CR4: 00000000003706f0 [ 2227.932599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2227.939734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2227.946864] Call Trace: [ 2227.949316] [ 2227.951423] ? __warn+0x85/0x180 [ 2227.954656] ? __list_add_valid_or_report+0x7d/0xa0 [ 2227.959536] ? report_bug+0xfc/0x1d0 [ 2227.963117] ? prb_read_valid+0x1b/0x20 [ 2227.966957] ? handle_bug+0x44/0x70 [ 2227.970455] ? exc_invalid_op+0x17/0x60 [ 2227.974310] ? asm_exc_invalid_op+0x1a/0x20 [ 2227.978506] ? __list_add_valid_or_report+0x7d/0xa0 [ 2227.983390] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2227.988286] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2227.993015] lookup_fast+0x6d/0xf0 [ 2227.996433] walk_component+0x26/0x160 [ 2228.000195] link_path_walk+0x24d/0x380 [ 2228.004039] ? path_openat+0x80/0x9d0 [ 2228.007712] path_openat+0x8e/0x9d0 [ 2228.011213] do_filp_open+0xb8/0x120 [ 2228.014804] ? lock_release+0xc5/0x260 [ 2228.018558] ? _raw_spin_unlock+0x23/0x40 [ 2228.022570] ? alloc_fd+0x13f/0x200 [ 2228.026068] do_sys_openat2+0x249/0x290 [ 2228.029914] do_sys_open+0x4f/0x80 [ 2228.033326] do_syscall_64+0x38/0x80 [ 2228.036906] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2228.041955] RIP: 0033:0x7f48f0eed2f2 [ 2228.045538] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2228.064302] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2228.071874] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2228.079005] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2228.086139] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2228.093275] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2228.100406] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2228.107551] [ 2228.109749] irq event stamp: 12873359 [ 2228.113414] hardirqs last enabled at (12873369): [] console_unlock+0x106/0x120 [ 2228.122281] hardirqs last disabled at (12873378): [] console_unlock+0xeb/0x120 [ 2228.131059] softirqs last enabled at (12873346): [] __do_softirq+0x2be/0x371 [ 2228.139753] softirqs last disabled at (12873341): [] irq_exit_rcu+0xb0/0xd0 [ 2228.148272] ---[ end trace 0000000000000000 ]--- [ 2228.152892] ------------[ cut here ]------------ [ 2228.157510] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2228.169161] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2228.177866] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2228.233421] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2228.242552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2228.250032] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2228.255543] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2228.261029] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2228.279778] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2228.285011] RAX: 0000000000000000 RBX: ffff8881a1d330e8 RCX: 0000000000000000 [ 2228.292142] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2228.299276] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000042dc [ 2228.306409] R10: 000000000018af28 R11: ffffc9000030b828 R12: ffff8881a1d33838 [ 2228.313545] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2228.320684] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2228.328770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2228.334513] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2228.341648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2228.348785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2228.355923] Call Trace: [ 2228.358375] [ 2228.360482] ? __warn+0x85/0x180 [ 2228.363714] ? __list_add_valid_or_report+0x7d/0xa0 [ 2228.368596] ? report_bug+0xfc/0x1d0 [ 2228.372183] ? prb_read_valid+0x1b/0x20 [ 2228.376033] ? handle_bug+0x44/0x70 [ 2228.379533] ? exc_invalid_op+0x17/0x60 [ 2228.383381] ? asm_exc_invalid_op+0x1a/0x20 [ 2228.387586] ? __list_add_valid_or_report+0x7d/0xa0 [ 2228.392474] ? __list_add_valid_or_report+0x7d/0xa0 [ 2228.397358] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2228.402268] ceph_add_cap+0x20b/0x620 [ceph] [ 2228.406576] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2228.411322] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2228.416669] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2228.421149] ceph_con_process_message+0x74/0x150 [libceph] [ 2228.426651] ? calc_signature+0xdb/0x100 [libceph] [ 2228.431475] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2228.436816] ? process_scheduled_works+0x22a/0x510 [ 2228.441621] ? process_scheduled_works+0x22a/0x510 [ 2228.446420] ceph_con_workfn+0x344/0x750 [libceph] [ 2228.451235] ? process_scheduled_works+0x22a/0x510 [ 2228.456033] process_scheduled_works+0x2a4/0x510 [ 2228.460673] worker_thread+0x188/0x340 [ 2228.464434] ? rescuer_thread+0x310/0x310 [ 2228.468452] kthread+0xf5/0x130 [ 2228.471600] ? kthread_complete_and_exit+0x20/0x20 [ 2228.476406] ret_from_fork+0x31/0x40 [ 2228.479987] ? kthread_complete_and_exit+0x20/0x20 [ 2228.484782] ret_from_fork_asm+0x11/0x20 [ 2228.488730] [ 2228.490926] irq event stamp: 10402627 [ 2228.494591] hardirqs last enabled at (10402637): [] console_unlock+0x106/0x120 [ 2228.503457] hardirqs last disabled at (10402646): [] console_unlock+0xeb/0x120 [ 2228.512241] softirqs last enabled at (10402662): [] __do_softirq+0x2be/0x371 [ 2228.520938] softirqs last disabled at (10402655): [] irq_exit_rcu+0xb0/0xd0 [ 2228.529459] ---[ end trace 0000000000000000 ]--- [ 2228.534078] ------------[ cut here ]------------ [ 2228.538694] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2228.550341] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2228.559298] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2228.614820] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2228.623779] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2228.631305] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2228.636840] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2228.655585] RSP: 0018:ffffc90001fd7bf8 EFLAGS: 00010282 [ 2228.660810] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2228.667943] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2228.675076] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004316 [ 2228.682209] R10: 000000000018c0f0 R11: ffffc90001fd7a88 R12: ffff888158a68750 [ 2228.689341] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2228.696476] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2228.704561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2228.710308] CR2: 00007fb38d5d5000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2228.717441] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2228.724574] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2228.731707] Call Trace: [ 2228.734161] [ 2228.736266] ? __warn+0x85/0x180 [ 2228.739500] ? __list_add_valid_or_report+0x7d/0xa0 [ 2228.744388] ? report_bug+0xfc/0x1d0 [ 2228.747966] ? prb_read_valid+0x1b/0x20 [ 2228.751807] ? handle_bug+0x44/0x70 [ 2228.755298] ? exc_invalid_op+0x17/0x60 [ 2228.759137] ? asm_exc_invalid_op+0x1a/0x20 [ 2228.763332] ? __list_add_valid_or_report+0x7d/0xa0 [ 2228.768212] ? __list_add_valid_or_report+0x7d/0xa0 [ 2228.773091] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2228.777991] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2228.782717] lookup_fast+0x6d/0xf0 [ 2228.786126] walk_component+0x26/0x160 [ 2228.789882] link_path_walk+0x24d/0x380 [ 2228.793726] ? path_parentat.isra.39+0x19/0x70 [ 2228.798177] path_parentat.isra.39+0x24/0x70 [ 2228.802456] __filename_parentat.isra.41+0xe6/0x1d0 [ 2228.807344] ? lock_acquire+0xcb/0x2b0 [ 2228.811094] ? __might_fault+0x45/0x80 [ 2228.814849] ? find_held_lock+0x2d/0x90 [ 2228.818700] do_unlinkat+0x60/0x2c0 [ 2228.822205] __x64_sys_unlink+0x23/0x30 [ 2228.826044] do_syscall_64+0x38/0x80 [ 2228.829626] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2228.834677] RIP: 0033:0x7f48f0eef16b [ 2228.838256] Code: f0 ff ff 73 01 c3 48 8b 0d 1a fd 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed fc 2c 00 f7 d8 64 89 01 48 [ 2228.857002] RSP: 002b:00007ffc53d60098 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 2228.864568] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eef16b [ 2228.871704] RDX: 0000000000000000 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2228.878834] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2228.885966] R10: 00007f48f0f711e0 R11: 0000000000000202 R12: 0000000000000004 [ 2228.893099] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2228.900245] [ 2228.902433] irq event stamp: 12956413 [ 2228.906100] hardirqs last enabled at (12956423): [] console_unlock+0x106/0x120 [ 2228.914966] hardirqs last disabled at (12956432): [] console_unlock+0xeb/0x120 [ 2228.923744] softirqs last enabled at (12956360): [] __do_softirq+0x2be/0x371 [ 2228.932437] softirqs last disabled at (12956355): [] irq_exit_rcu+0xb0/0xd0 [ 2228.940956] ---[ end trace 0000000000000000 ]--- [ 2228.945576] ------------[ cut here ]------------ [ 2228.950194] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2228.961845] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2228.970538] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2229.026015] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2229.035139] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2229.042618] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2229.048113] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2229.053598] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2229.072344] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2229.077571] RAX: 0000000000000000 RBX: ffff8881d09d1390 RCX: 0000000000000000 [ 2229.084704] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2229.091844] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004352 [ 2229.098980] R10: 000000000018d168 R11: ffffc9000030b828 R12: ffff8881d09d1ae0 [ 2229.106111] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2229.113244] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2229.121330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2229.127076] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2229.134209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2229.141340] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2229.148474] Call Trace: [ 2229.150927] [ 2229.153034] ? __warn+0x85/0x180 [ 2229.156267] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.161145] ? report_bug+0xfc/0x1d0 [ 2229.164725] ? prb_read_valid+0x1b/0x20 [ 2229.168565] ? handle_bug+0x44/0x70 [ 2229.172057] ? exc_invalid_op+0x17/0x60 [ 2229.175897] ? asm_exc_invalid_op+0x1a/0x20 [ 2229.180089] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.184973] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.189850] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2229.194741] ceph_add_cap+0x20b/0x620 [ceph] [ 2229.199025] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2229.203754] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2229.209082] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2229.213545] ceph_con_process_message+0x74/0x150 [libceph] [ 2229.219043] ? calc_signature+0xdb/0x100 [libceph] [ 2229.223852] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2229.229176] ? process_scheduled_works+0x22a/0x510 [ 2229.233967] ? process_scheduled_works+0x22a/0x510 [ 2229.238758] ceph_con_workfn+0x344/0x750 [libceph] [ 2229.243560] ? process_scheduled_works+0x22a/0x510 [ 2229.248351] process_scheduled_works+0x2a4/0x510 [ 2229.252975] worker_thread+0x188/0x340 [ 2229.256725] ? rescuer_thread+0x310/0x310 [ 2229.260735] kthread+0xf5/0x130 [ 2229.263882] ? kthread_complete_and_exit+0x20/0x20 [ 2229.268676] ret_from_fork+0x31/0x40 [ 2229.272253] ? kthread_complete_and_exit+0x20/0x20 [ 2229.277046] ret_from_fork_asm+0x11/0x20 [ 2229.280979] [ 2229.283166] irq event stamp: 10404649 [ 2229.286831] hardirqs last enabled at (10404659): [] console_unlock+0x106/0x120 [ 2229.295699] hardirqs last disabled at (10404668): [] console_unlock+0xeb/0x120 [ 2229.304478] softirqs last enabled at (10404646): [] __do_softirq+0x2be/0x371 [ 2229.313170] softirqs last disabled at (10404687): [] irq_exit_rcu+0xb0/0xd0 [ 2229.321690] ---[ end trace 0000000000000000 ]--- [ 2229.326310] ------------[ cut here ]------------ [ 2229.330937] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2229.342589] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2229.351550] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2229.407017] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2229.415970] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2229.423446] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2229.428934] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2229.447680] RSP: 0018:ffffc90001fd7bf8 EFLAGS: 00010282 [ 2229.452906] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2229.460037] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2229.467171] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000438c [ 2229.474307] R10: 000000000018e2e0 R11: ffffc90001fd7a88 R12: ffff8881c5350750 [ 2229.481439] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2229.488570] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2229.496657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2229.502402] CR2: 00007fb38d614000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2229.509535] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2229.516668] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2229.523802] Call Trace: [ 2229.526253] [ 2229.528362] ? __warn+0x85/0x180 [ 2229.531593] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.536476] ? report_bug+0xfc/0x1d0 [ 2229.540060] ? prb_read_valid+0x1b/0x20 [ 2229.543904] ? handle_bug+0x44/0x70 [ 2229.547401] ? exc_invalid_op+0x17/0x60 [ 2229.551242] ? asm_exc_invalid_op+0x1a/0x20 [ 2229.555433] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.560317] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.565203] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2229.570105] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2229.574829] lookup_fast+0x6d/0xf0 [ 2229.578240] walk_component+0x26/0x160 [ 2229.581992] link_path_walk+0x24d/0x380 [ 2229.585831] ? path_parentat.isra.39+0x19/0x70 [ 2229.590279] path_parentat.isra.39+0x24/0x70 [ 2229.594551] __filename_parentat.isra.41+0xe6/0x1d0 [ 2229.599441] ? lock_acquire+0xcb/0x2b0 [ 2229.603198] ? __might_fault+0x45/0x80 [ 2229.606952] ? find_held_lock+0x2d/0x90 [ 2229.610796] do_unlinkat+0x60/0x2c0 [ 2229.614298] __x64_sys_unlink+0x23/0x30 [ 2229.618139] do_syscall_64+0x38/0x80 [ 2229.621720] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2229.626772] RIP: 0033:0x7f48f0eef16b [ 2229.630349] Code: f0 ff ff 73 01 c3 48 8b 0d 1a fd 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed fc 2c 00 f7 d8 64 89 01 48 [ 2229.649099] RSP: 002b:00007ffc53d60098 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 2229.656672] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eef16b [ 2229.663808] RDX: 0000000000000000 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2229.670938] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2229.678072] R10: 00007f48f0f711e0 R11: 0000000000000202 R12: 0000000000000004 [ 2229.685202] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2229.692346] [ 2229.694538] irq event stamp: 12958287 [ 2229.698203] hardirqs last enabled at (12958297): [] console_unlock+0x106/0x120 [ 2229.707068] hardirqs last disabled at (12958306): [] console_unlock+0xeb/0x120 [ 2229.715849] softirqs last enabled at (12958234): [] __do_softirq+0x2be/0x371 [ 2229.724540] softirqs last disabled at (12958229): [] irq_exit_rcu+0xb0/0xd0 [ 2229.733065] ---[ end trace 0000000000000000 ]--- [ 2229.737680] ------------[ cut here ]------------ [ 2229.742299] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2229.753943] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2229.762899] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2229.818364] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2229.827320] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2229.834802] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2229.840286] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2229.859045] RSP: 0018:ffffc90001fffbf0 EFLAGS: 00010282 [ 2229.864278] RAX: 0000000000000000 RBX: ffff8881d11409c8 RCX: 0000000000000000 [ 2229.871407] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2229.878540] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000043c8 [ 2229.885666] R10: 000000000018f360 R11: ffffc90001fffa80 R12: ffff8881d1141118 [ 2229.892798] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2229.899931] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2229.908017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2229.913762] CR2: 00007f9bd2a34000 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2229.920888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2229.928022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2229.935151] Call Trace: [ 2229.937595] [ 2229.939694] ? __warn+0x85/0x180 [ 2229.942927] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.947808] ? report_bug+0xfc/0x1d0 [ 2229.951385] ? prb_read_valid+0x1b/0x20 [ 2229.955225] ? handle_bug+0x44/0x70 [ 2229.958716] ? exc_invalid_op+0x17/0x60 [ 2229.962559] ? asm_exc_invalid_op+0x1a/0x20 [ 2229.966748] ? __list_add_valid_or_report+0x7d/0xa0 [ 2229.971623] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2229.976514] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2229.981230] lookup_fast+0x6d/0xf0 [ 2229.984634] walk_component+0x26/0x160 [ 2229.988386] link_path_walk+0x24d/0x380 [ 2229.992223] ? path_openat+0x80/0x9d0 [ 2229.995891] path_openat+0x8e/0x9d0 [ 2229.999386] do_filp_open+0xb8/0x120 [ 2230.002965] ? lock_release+0xc5/0x260 [ 2230.006716] ? _raw_spin_unlock+0x23/0x40 [ 2230.010728] ? alloc_fd+0x13f/0x200 [ 2230.014224] do_sys_openat2+0x249/0x290 [ 2230.018060] do_sys_open+0x4f/0x80 [ 2230.021466] do_syscall_64+0x38/0x80 [ 2230.025047] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2230.030097] RIP: 0033:0x7f48f0eed2f2 [ 2230.033676] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2230.052422] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2230.059989] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2230.067122] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2230.074255] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2230.081388] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2230.088528] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2230.095660] [ 2230.097847] irq event stamp: 12878223 [ 2230.101511] hardirqs last enabled at (12878233): [] console_unlock+0x106/0x120 [ 2230.110378] hardirqs last disabled at (12878242): [] console_unlock+0xeb/0x120 [ 2230.119157] softirqs last enabled at (12878200): [] __do_softirq+0x2be/0x371 [ 2230.127854] softirqs last disabled at (12878263): [] irq_exit_rcu+0xb0/0xd0 [ 2230.136372] ---[ end trace 0000000000000000 ]--- [ 2230.140995] ------------[ cut here ]------------ [ 2230.145610] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2230.157257] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2230.166214] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2230.221682] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2230.230635] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2230.238112] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2230.243599] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2230.262344] RSP: 0018:ffffc90001fd7bf8 EFLAGS: 00010282 [ 2230.267570] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2230.274703] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2230.281839] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004403 [ 2230.288978] R10: 0000000000190510 R11: ffffc90001fd7a88 R12: ffff8881d10b72e8 [ 2230.296110] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2230.303243] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2230.311330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2230.317073] CR2: 00007fb38d614000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2230.324210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2230.331340] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2230.338473] Call Trace: [ 2230.340925] [ 2230.343035] ? __warn+0x85/0x180 [ 2230.346265] ? __list_add_valid_or_report+0x7d/0xa0 [ 2230.351147] ? report_bug+0xfc/0x1d0 [ 2230.354736] ? prb_read_valid+0x1b/0x20 [ 2230.358576] ? handle_bug+0x44/0x70 [ 2230.362074] ? exc_invalid_op+0x17/0x60 [ 2230.365917] ? asm_exc_invalid_op+0x1a/0x20 [ 2230.370115] ? __list_add_valid_or_report+0x7d/0xa0 [ 2230.374998] ? __list_add_valid_or_report+0x7d/0xa0 [ 2230.379876] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2230.384778] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2230.389503] lookup_fast+0x6d/0xf0 [ 2230.392914] walk_component+0x26/0x160 [ 2230.396675] link_path_walk+0x24d/0x380 [ 2230.400511] ? path_parentat.isra.39+0x19/0x70 [ 2230.404962] path_parentat.isra.39+0x24/0x70 [ 2230.409243] __filename_parentat.isra.41+0xe6/0x1d0 [ 2230.414135] ? lock_acquire+0xcb/0x2b0 [ 2230.417892] ? __might_fault+0x45/0x80 [ 2230.421651] ? find_held_lock+0x2d/0x90 [ 2230.425497] do_unlinkat+0x60/0x2c0 [ 2230.428996] __x64_sys_unlink+0x23/0x30 [ 2230.432838] do_syscall_64+0x38/0x80 [ 2230.436419] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2230.441472] RIP: 0033:0x7f48f0eef16b [ 2230.445048] Code: f0 ff ff 73 01 c3 48 8b 0d 1a fd 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed fc 2c 00 f7 d8 64 89 01 48 [ 2230.463797] RSP: 002b:00007ffc53d60098 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 2230.471362] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eef16b [ 2230.478496] RDX: 0000000000000000 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2230.485629] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2230.492760] R10: 00007f48f0f711e0 R11: 0000000000000202 R12: 0000000000000004 [ 2230.499892] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2230.507039] [ 2230.509226] irq event stamp: 12960207 [ 2230.512895] hardirqs last enabled at (12960217): [] console_unlock+0x106/0x120 [ 2230.521760] hardirqs last disabled at (12960226): [] console_unlock+0xeb/0x120 [ 2230.530540] softirqs last enabled at (12960164): [] __do_softirq+0x2be/0x371 [ 2230.539232] softirqs last disabled at (12960159): [] irq_exit_rcu+0xb0/0xd0 [ 2230.547753] ---[ end trace 0000000000000000 ]--- [ 2230.552376] ------------[ cut here ]------------ [ 2230.557011] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2230.568660] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2230.577625] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2230.633097] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2230.642058] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2230.649544] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2230.655038] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2230.673790] RSP: 0018:ffffc90001fffbf0 EFLAGS: 00010282 [ 2230.679026] RAX: 0000000000000000 RBX: ffff8881cccb8000 RCX: 0000000000000000 [ 2230.686167] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2230.693309] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000443f [ 2230.700450] R10: 0000000000191570 R11: ffffc90001fffa80 R12: ffff8881cccb8750 [ 2230.707580] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2230.714721] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2230.722811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2230.728564] CR2: 00007f9bd2a2a000 CR3: 000000017eb7a003 CR4: 00000000003706f0 [ 2230.735696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2230.742831] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2230.749977] Call Trace: [ 2230.752441] [ 2230.754547] ? __warn+0x85/0x180 [ 2230.757791] ? __list_add_valid_or_report+0x7d/0xa0 [ 2230.762678] ? report_bug+0xfc/0x1d0 [ 2230.766266] ? prb_read_valid+0x1b/0x20 [ 2230.770116] ? handle_bug+0x44/0x70 [ 2230.773615] ? exc_invalid_op+0x17/0x60 [ 2230.777454] ? asm_exc_invalid_op+0x1a/0x20 [ 2230.781653] ? __list_add_valid_or_report+0x7d/0xa0 [ 2230.786539] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2230.791435] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2230.796160] lookup_fast+0x6d/0xf0 [ 2230.799574] walk_component+0x26/0x160 [ 2230.803336] link_path_walk+0x24d/0x380 [ 2230.807183] ? path_openat+0x80/0x9d0 [ 2230.810861] path_openat+0x8e/0x9d0 [ 2230.814361] do_filp_open+0xb8/0x120 [ 2230.817949] ? lock_release+0xc5/0x260 [ 2230.821709] ? _raw_spin_unlock+0x23/0x40 [ 2230.825729] ? alloc_fd+0x13f/0x200 [ 2230.829233] do_sys_openat2+0x249/0x290 [ 2230.833080] do_sys_open+0x4f/0x80 [ 2230.836492] do_syscall_64+0x38/0x80 [ 2230.840081] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2230.845142] RIP: 0033:0x7f48f0eed2f2 [ 2230.848731] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2230.867495] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2230.875070] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2230.882213] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2230.889350] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2230.896486] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2230.903626] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2230.910765] [ 2230.912959] irq event stamp: 12883211 [ 2230.916626] hardirqs last enabled at (12883279): [] console_unlock+0x106/0x120 [ 2230.925500] hardirqs last disabled at (12883344): [] console_unlock+0xeb/0x120 [ 2230.934281] softirqs last enabled at (12883412): [] __do_softirq+0x2be/0x371 [ 2230.942981] softirqs last disabled at (12883423): [] irq_exit_rcu+0xb0/0xd0 [ 2230.951513] ---[ end trace 0000000000000000 ]--- [ 2230.956139] ------------[ cut here ]------------ [ 2230.960759] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2230.972406] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2230.981100] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2231.036579] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2231.045703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2231.053184] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2231.058686] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2231.064172] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2231.082918] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2231.088144] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2231.095277] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2231.102417] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000447a [ 2231.109553] R10: 0000000000192530 R11: ffffc9000030b828 R12: ffff8881d12d1118 [ 2231.116683] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2231.123817] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2231.131902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2231.137651] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2231.144783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2231.151916] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2231.159048] Call Trace: [ 2231.161500] [ 2231.163607] ? __warn+0x85/0x180 [ 2231.166840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.171719] ? report_bug+0xfc/0x1d0 [ 2231.175299] ? prb_read_valid+0x1b/0x20 [ 2231.179139] ? handle_bug+0x44/0x70 [ 2231.182631] ? exc_invalid_op+0x17/0x60 [ 2231.186469] ? asm_exc_invalid_op+0x1a/0x20 [ 2231.190661] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.195546] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.200423] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2231.205314] ceph_add_cap+0x20b/0x620 [ceph] [ 2231.209599] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2231.214319] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2231.219646] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2231.224111] ceph_con_process_message+0x74/0x150 [libceph] [ 2231.229605] ? calc_signature+0xdb/0x100 [libceph] [ 2231.234422] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2231.239748] ? process_scheduled_works+0x22a/0x510 [ 2231.244539] ? process_scheduled_works+0x22a/0x510 [ 2231.249330] ceph_con_workfn+0x344/0x750 [libceph] [ 2231.254135] ? process_scheduled_works+0x22a/0x510 [ 2231.258923] process_scheduled_works+0x2a4/0x510 [ 2231.263550] worker_thread+0x188/0x340 [ 2231.267306] ? rescuer_thread+0x310/0x310 [ 2231.271319] kthread+0xf5/0x130 [ 2231.274463] ? kthread_complete_and_exit+0x20/0x20 [ 2231.279257] ret_from_fork+0x31/0x40 [ 2231.282835] ? kthread_complete_and_exit+0x20/0x20 [ 2231.287629] ret_from_fork_asm+0x11/0x20 [ 2231.291561] [ 2231.293748] irq event stamp: 10409237 [ 2231.297413] hardirqs last enabled at (10409247): [] console_unlock+0x106/0x120 [ 2231.306279] hardirqs last disabled at (10409258): [] console_unlock+0xeb/0x120 [ 2231.315059] softirqs last enabled at (10409234): [] __do_softirq+0x2be/0x371 [ 2231.323753] softirqs last disabled at (10409229): [] irq_exit_rcu+0xb0/0xd0 [ 2231.332272] ---[ end trace 0000000000000000 ]--- [ 2231.336894] ------------[ cut here ]------------ [ 2231.341511] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2231.353159] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2231.362115] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2231.417585] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2231.426543] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2231.434022] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2231.439509] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2231.458258] RSP: 0018:ffffc90001fd7bf8 EFLAGS: 00010282 [ 2231.463488] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2231.470620] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2231.477756] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000044b4 [ 2231.484889] R10: 00000000001932f0 R11: ffffc90001fd7a88 R12: ffff8881d10b0750 [ 2231.492019] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2231.499152] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2231.507239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2231.512983] CR2: 00007fb38d614000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2231.520119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2231.527250] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2231.534384] Call Trace: [ 2231.536836] [ 2231.538943] ? __warn+0x85/0x180 [ 2231.542176] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.547057] ? report_bug+0xfc/0x1d0 [ 2231.550645] ? prb_read_valid+0x1b/0x20 [ 2231.554485] ? handle_bug+0x44/0x70 [ 2231.557975] ? exc_invalid_op+0x17/0x60 [ 2231.561817] ? asm_exc_invalid_op+0x1a/0x20 [ 2231.566015] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.570899] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.575776] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2231.580679] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2231.585403] lookup_fast+0x6d/0xf0 [ 2231.588815] walk_component+0x26/0x160 [ 2231.592575] link_path_walk+0x24d/0x380 [ 2231.596414] ? path_parentat.isra.39+0x19/0x70 [ 2231.600861] path_parentat.isra.39+0x24/0x70 [ 2231.605133] __filename_parentat.isra.41+0xe6/0x1d0 [ 2231.610021] ? lock_acquire+0xcb/0x2b0 [ 2231.613776] ? __might_fault+0x45/0x80 [ 2231.617535] ? find_held_lock+0x2d/0x90 [ 2231.621378] do_unlinkat+0x60/0x2c0 [ 2231.624881] __x64_sys_unlink+0x23/0x30 [ 2231.628723] do_syscall_64+0x38/0x80 [ 2231.632301] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2231.637354] RIP: 0033:0x7f48f0eef16b [ 2231.640934] Code: f0 ff ff 73 01 c3 48 8b 0d 1a fd 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed fc 2c 00 f7 d8 64 89 01 48 [ 2231.659682] RSP: 002b:00007ffc53d60098 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 2231.667255] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eef16b [ 2231.674392] RDX: 0000000000000000 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2231.681530] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2231.688661] R10: 00007f48f0f711e0 R11: 0000000000000202 R12: 0000000000000004 [ 2231.695793] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2231.702937] [ 2231.705126] irq event stamp: 12962965 [ 2231.708794] hardirqs last enabled at (12962975): [] console_unlock+0x106/0x120 [ 2231.717660] hardirqs last disabled at (12962984): [] console_unlock+0xeb/0x120 [ 2231.726442] softirqs last enabled at (12962922): [] __do_softirq+0x2be/0x371 [ 2231.735140] softirqs last disabled at (12962917): [] irq_exit_rcu+0xb0/0xd0 [ 2231.743661] ---[ end trace 0000000000000000 ]--- [ 2231.748285] ------------[ cut here ]------------ [ 2231.752982] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2231.764630] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2231.773591] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2231.829062] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2231.838015] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2231.845493] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2231.850991] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2231.869742] RSP: 0018:ffffc90001fffc88 EFLAGS: 00010286 [ 2231.874972] RAX: 0000000000000000 RBX: ffff8881a1d33ab0 RCX: 0000000000000000 [ 2231.882101] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2231.889225] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000044f0 [ 2231.896352] R10: 0000000000194300 R11: ffffc90001fffb18 R12: ffff8881a1d34200 [ 2231.903483] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2231.910609] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2231.918694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2231.924439] CR2: 00007f9bd2a34000 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2231.931566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2231.938697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2231.945829] Call Trace: [ 2231.948275] [ 2231.950381] ? __warn+0x85/0x180 [ 2231.953613] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.958497] ? report_bug+0xfc/0x1d0 [ 2231.962078] ? prb_read_valid+0x1b/0x20 [ 2231.965912] ? handle_bug+0x44/0x70 [ 2231.969411] ? exc_invalid_op+0x17/0x60 [ 2231.973243] ? asm_exc_invalid_op+0x1a/0x20 [ 2231.977439] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.982317] ? __list_add_valid_or_report+0x7d/0xa0 [ 2231.987202] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2231.992096] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2231.996812] lookup_fast+0x6d/0xf0 [ 2232.000213] path_openat+0xe4/0x9d0 [ 2232.003710] do_filp_open+0xb8/0x120 [ 2232.007289] ? lock_release+0xc5/0x260 [ 2232.011041] ? _raw_spin_unlock+0x23/0x40 [ 2232.015053] ? alloc_fd+0x13f/0x200 [ 2232.018546] do_sys_openat2+0x249/0x290 [ 2232.022386] do_sys_open+0x4f/0x80 [ 2232.025792] do_syscall_64+0x38/0x80 [ 2232.029371] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2232.034421] RIP: 0033:0x7f48f0eed2f2 [ 2232.038002] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2232.056749] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2232.064313] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2232.071438] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2232.078570] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2232.085696] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2232.092836] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2232.099977] [ 2232.102161] irq event stamp: 12887241 [ 2232.105820] hardirqs last enabled at (12887251): [] console_unlock+0x106/0x120 [ 2232.114686] hardirqs last disabled at (12887260): [] console_unlock+0xeb/0x120 [ 2232.123464] softirqs last enabled at (12887228): [] __do_softirq+0x2be/0x371 [ 2232.132156] softirqs last disabled at (12887223): [] irq_exit_rcu+0xb0/0xd0 [ 2232.140677] ---[ end trace 0000000000000000 ]--- [ 2232.145292] ------------[ cut here ]------------ [ 2232.149906] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2232.161546] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2232.170242] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2232.225718] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2232.234845] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2232.242324] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2232.247818] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2232.253302] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2232.272051] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2232.277276] RAX: 0000000000000000 RBX: ffff8881d0b5bab0 RCX: 0000000000000000 [ 2232.284410] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2232.291543] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004529 [ 2232.298674] R10: 0000000000195350 R11: ffffc9000030b828 R12: ffff8881d0b5c200 [ 2232.305807] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2232.312941] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2232.321027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.326772] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2232.333905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2232.341038] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2232.348171] Call Trace: [ 2232.350623] [ 2232.352730] ? __warn+0x85/0x180 [ 2232.355965] ? __list_add_valid_or_report+0x7d/0xa0 [ 2232.360843] ? report_bug+0xfc/0x1d0 [ 2232.364420] ? prb_read_valid+0x1b/0x20 [ 2232.368261] ? handle_bug+0x44/0x70 [ 2232.371753] ? exc_invalid_op+0x17/0x60 [ 2232.375593] ? asm_exc_invalid_op+0x1a/0x20 [ 2232.379786] ? __list_add_valid_or_report+0x7d/0xa0 [ 2232.384667] ? __list_add_valid_or_report+0x7d/0xa0 [ 2232.389547] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2232.394442] ceph_add_cap+0x20b/0x620 [ceph] [ 2232.398730] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2232.403455] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2232.408787] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2232.413249] ceph_con_process_message+0x74/0x150 [libceph] [ 2232.418746] ? calc_signature+0xdb/0x100 [libceph] [ 2232.423558] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2232.428889] ? process_scheduled_works+0x22a/0x510 [ 2232.433680] ? process_scheduled_works+0x22a/0x510 [ 2232.438469] ceph_con_workfn+0x344/0x750 [libceph] [ 2232.443276] ? process_scheduled_works+0x22a/0x510 [ 2232.448063] process_scheduled_works+0x2a4/0x510 [ 2232.452690] worker_thread+0x188/0x340 [ 2232.456447] ? rescuer_thread+0x310/0x310 [ 2232.460458] kthread+0xf5/0x130 [ 2232.463605] ? kthread_complete_and_exit+0x20/0x20 [ 2232.468399] ret_from_fork+0x31/0x40 [ 2232.471976] ? kthread_complete_and_exit+0x20/0x20 [ 2232.476770] ret_from_fork_asm+0x11/0x20 [ 2232.480702] [ 2232.482889] irq event stamp: 10412031 [ 2232.486554] hardirqs last enabled at (10412041): [] console_unlock+0x106/0x120 [ 2232.495421] hardirqs last disabled at (10412050): [] console_unlock+0xeb/0x120 [ 2232.504199] softirqs last enabled at (10412028): [] __do_softirq+0x2be/0x371 [ 2232.512892] softirqs last disabled at (10412023): [] irq_exit_rcu+0xb0/0xd0 [ 2232.521411] ---[ end trace 0000000000000000 ]--- [ 2232.526032] ------------[ cut here ]------------ [ 2232.530794] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2232.542452] WARNING: CPU: 3 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2232.551411] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2232.606947] CPU: 3 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2232.615900] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2232.623380] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2232.628865] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2232.647615] RSP: 0018:ffffc90001fd7dc8 EFLAGS: 00010286 [ 2232.652847] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2232.659979] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2232.667111] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004563 [ 2232.674244] R10: 0000000000196388 R11: ffffc90001fd7c58 R12: ffff8881c2cefcb0 [ 2232.681377] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2232.688509] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2232.696598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.702343] CR2: 00007fb38d638038 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2232.709476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2232.716607] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2232.723742] Call Trace: [ 2232.726194] [ 2232.728302] ? __warn+0x85/0x180 [ 2232.731533] ? __list_add_valid_or_report+0x7d/0xa0 [ 2232.736414] ? report_bug+0xfc/0x1d0 [ 2232.739991] ? prb_read_valid+0x1b/0x20 [ 2232.743834] ? handle_bug+0x44/0x70 [ 2232.747324] ? exc_invalid_op+0x17/0x60 [ 2232.751165] ? asm_exc_invalid_op+0x1a/0x20 [ 2232.755360] ? __list_add_valid_or_report+0x7d/0xa0 [ 2232.760239] ? __list_add_valid_or_report+0x7d/0xa0 [ 2232.765118] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2232.770017] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2232.774746] lookup_dcache+0x38/0x60 [ 2232.778327] lookup_one_qstr_excl+0x23/0xa0 [ 2232.782525] do_unlinkat+0x144/0x2c0 [ 2232.786119] __x64_sys_unlink+0x23/0x30 [ 2232.789955] do_syscall_64+0x38/0x80 [ 2232.793537] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2232.798598] RIP: 0033:0x7f48f0eef16b [ 2232.802176] Code: f0 ff ff 73 01 c3 48 8b 0d 1a fd 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed fc 2c 00 f7 d8 64 89 01 48 [ 2232.820928] RSP: 002b:00007ffc53d60098 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 2232.828498] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eef16b [ 2232.835629] RDX: 0000000000000000 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2232.842764] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2232.849894] R10: 00007f48f0f711e0 R11: 0000000000000202 R12: 0000000000000004 [ 2232.857027] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2232.864176] [ 2232.866370] irq event stamp: 12966081 [ 2232.870037] hardirqs last enabled at (12966091): [] console_unlock+0x106/0x120 [ 2232.878904] hardirqs last disabled at (12966100): [] console_unlock+0xeb/0x120 [ 2232.887682] softirqs last enabled at (12966018): [] __do_softirq+0x2be/0x371 [ 2232.896374] softirqs last disabled at (12966013): [] irq_exit_rcu+0xb0/0xd0 [ 2232.904899] ---[ end trace 0000000000000000 ]--- [ 2232.909518] ------------[ cut here ]------------ [ 2232.914132] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2232.925780] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2232.934474] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2232.989951] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2232.999077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2233.006556] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2233.012050] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2233.017535] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2233.036283] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2233.041508] RAX: 0000000000000000 RBX: ffff8881d0b16b98 RCX: 0000000000000000 [ 2233.048642] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2233.055775] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004598 [ 2233.062906] R10: 0000000000197460 R11: ffffc9000030b828 R12: ffff8881d0b172e8 [ 2233.070039] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2233.077172] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2233.085258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2233.091003] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2233.098138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2233.105271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2233.112404] Call Trace: [ 2233.114856] [ 2233.116961] ? __warn+0x85/0x180 [ 2233.120197] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.125075] ? report_bug+0xfc/0x1d0 [ 2233.128653] ? prb_read_valid+0x1b/0x20 [ 2233.132495] ? handle_bug+0x44/0x70 [ 2233.135984] ? exc_invalid_op+0x17/0x60 [ 2233.139825] ? asm_exc_invalid_op+0x1a/0x20 [ 2233.144017] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.148902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.153779] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2233.158668] ceph_add_cap+0x20b/0x620 [ceph] [ 2233.162963] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2233.167693] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2233.173021] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2233.177481] ceph_con_process_message+0x74/0x150 [libceph] [ 2233.182978] ? calc_signature+0xdb/0x100 [libceph] [ 2233.187791] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2233.193126] ? process_scheduled_works+0x22a/0x510 [ 2233.197926] ? process_scheduled_works+0x22a/0x510 [ 2233.202719] ceph_con_workfn+0x344/0x750 [libceph] [ 2233.207525] ? process_scheduled_works+0x22a/0x510 [ 2233.212313] process_scheduled_works+0x2a4/0x510 [ 2233.216940] worker_thread+0x188/0x340 [ 2233.220695] ? rescuer_thread+0x310/0x310 [ 2233.224708] kthread+0xf5/0x130 [ 2233.227853] ? kthread_complete_and_exit+0x20/0x20 [ 2233.232647] ret_from_fork+0x31/0x40 [ 2233.236225] ? kthread_complete_and_exit+0x20/0x20 [ 2233.241019] ret_from_fork_asm+0x11/0x20 [ 2233.244952] [ 2233.247139] irq event stamp: 10413921 [ 2233.250802] hardirqs last enabled at (10413931): [] console_unlock+0x106/0x120 [ 2233.259671] hardirqs last disabled at (10413940): [] console_unlock+0xeb/0x120 [ 2233.268448] softirqs last enabled at (10413918): [] __do_softirq+0x2be/0x371 [ 2233.277150] softirqs last disabled at (10413913): [] irq_exit_rcu+0xb0/0xd0 [ 2233.285670] ---[ end trace 0000000000000000 ]--- [ 2233.290291] ------------[ cut here ]------------ [ 2233.294993] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2233.306647] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2233.315609] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2233.371087] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2233.380046] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2233.387523] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2233.393010] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2233.411755] RSP: 0018:ffffc90001fd7d70 EFLAGS: 00010286 [ 2233.416983] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2233.424114] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2233.431249] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000045d2 [ 2233.438383] R10: 0000000000198478 R11: ffffc90001fd7c00 R12: ffff8881c2cefcb0 [ 2233.445515] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2233.452646] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2233.460734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2233.466479] CR2: 00007fb38d645000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2233.473611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2233.480744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2233.487878] Call Trace: [ 2233.490329] [ 2233.492437] ? __warn+0x85/0x180 [ 2233.495668] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.500557] ? report_bug+0xfc/0x1d0 [ 2233.504136] ? prb_read_valid+0x1b/0x20 [ 2233.507982] ? handle_bug+0x44/0x70 [ 2233.511477] ? exc_invalid_op+0x17/0x60 [ 2233.515319] ? asm_exc_invalid_op+0x1a/0x20 [ 2233.519519] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.524403] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2233.529299] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2233.534557] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2233.539363] ceph_unlink+0x167/0x4f0 [ceph] [ 2233.543572] vfs_unlink+0x10b/0x280 [ 2233.547068] do_unlinkat+0x21b/0x2c0 [ 2233.550661] __x64_sys_unlink+0x23/0x30 [ 2233.554499] do_syscall_64+0x38/0x80 [ 2233.558083] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2233.563140] RIP: 0033:0x7f48f0eef16b [ 2233.566724] Code: f0 ff ff 73 01 c3 48 8b 0d 1a fd 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed fc 2c 00 f7 d8 64 89 01 48 [ 2233.585474] RSP: 002b:00007ffc53d60098 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 2233.593041] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eef16b [ 2233.600172] RDX: 0000000000000000 RSI: 00007ffc53d5ffb0 RDI: 00007ffc53d60520 [ 2233.607306] RBP: 00000000020766e0 R08: 0000000002076673 R09: 0000000000000000 [ 2233.614437] R10: 00007f48f0f711e0 R11: 0000000000000202 R12: 0000000000000004 [ 2233.621572] R13: 00007f48f183d8a0 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2233.628718] [ 2233.630911] irq event stamp: 12968237 [ 2233.634580] hardirqs last enabled at (12968247): [] console_unlock+0x106/0x120 [ 2233.643448] hardirqs last disabled at (12968256): [] console_unlock+0xeb/0x120 [ 2233.652226] softirqs last enabled at (12968234): [] __do_softirq+0x2be/0x371 [ 2233.660918] softirqs last disabled at (12968229): [] irq_exit_rcu+0xb0/0xd0 [ 2233.669437] ---[ end trace 0000000000000000 ]--- [ 2233.674059] ------------[ cut here ]------------ [ 2233.678673] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2233.690313] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2233.699008] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2233.754485] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2233.763610] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2233.771092] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2233.776583] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2233.782070] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2233.800818] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2233.806043] RAX: 0000000000000000 RBX: ffff8881a1d31390 RCX: 0000000000000000 [ 2233.813177] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2233.820311] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004607 [ 2233.827442] R10: 00000000001994d0 R11: ffffc9000030b828 R12: ffff8881a1d31ae0 [ 2233.834576] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2233.841707] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2233.849793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2233.855538] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2233.862663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2233.869797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2233.876945] Call Trace: [ 2233.879400] [ 2233.881507] ? __warn+0x85/0x180 [ 2233.884741] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.889619] ? report_bug+0xfc/0x1d0 [ 2233.893196] ? prb_read_valid+0x1b/0x20 [ 2233.897039] ? handle_bug+0x44/0x70 [ 2233.900528] ? exc_invalid_op+0x17/0x60 [ 2233.904369] ? asm_exc_invalid_op+0x1a/0x20 [ 2233.908561] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.913444] ? __list_add_valid_or_report+0x7d/0xa0 [ 2233.918322] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2233.923213] ceph_add_cap+0x20b/0x620 [ceph] [ 2233.927499] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2233.932227] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2233.937555] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2233.942017] ceph_con_process_message+0x74/0x150 [libceph] [ 2233.947515] ? calc_signature+0xdb/0x100 [libceph] [ 2233.952325] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2233.957648] ? process_scheduled_works+0x22a/0x510 [ 2233.962440] ? process_scheduled_works+0x22a/0x510 [ 2233.967230] ceph_con_workfn+0x344/0x750 [libceph] [ 2233.972033] ? process_scheduled_works+0x22a/0x510 [ 2233.976824] process_scheduled_works+0x2a4/0x510 [ 2233.981448] worker_thread+0x188/0x340 [ 2233.985196] ? rescuer_thread+0x310/0x310 [ 2233.989209] kthread+0xf5/0x130 [ 2233.992356] ? kthread_complete_and_exit+0x20/0x20 [ 2233.997149] ret_from_fork+0x31/0x40 [ 2234.000728] ? kthread_complete_and_exit+0x20/0x20 [ 2234.005519] ret_from_fork_asm+0x11/0x20 [ 2234.009451] [ 2234.011639] irq event stamp: 10415785 [ 2234.015304] hardirqs last enabled at (10415795): [] console_unlock+0x106/0x120 [ 2234.024170] hardirqs last disabled at (10415804): [] console_unlock+0xeb/0x120 [ 2234.032950] softirqs last enabled at (10415782): [] __do_softirq+0x2be/0x371 [ 2234.041641] softirqs last disabled at (10415777): [] irq_exit_rcu+0xb0/0xd0 [ 2234.050162] ---[ end trace 0000000000000000 ]--- [ 2234.054782] ------------[ cut here ]------------ [ 2234.059400] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2234.071048] WARNING: CPU: 0 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2234.080003] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2234.135475] CPU: 0 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2234.144422] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2234.151902] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2234.157387] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2234.176134] RSP: 0018:ffffc90001fffc88 EFLAGS: 00010286 [ 2234.181359] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2234.188492] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2234.195625] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004641 [ 2234.202760] R10: 000000000019a478 R11: ffffc90001fffb18 R12: ffff8881d12c5590 [ 2234.209890] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2234.217015] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2234.225102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2234.230848] CR2: 00007f9bd2a2a000 CR3: 000000017eb7a003 CR4: 00000000003706f0 [ 2234.237981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.245114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2234.252247] Call Trace: [ 2234.254698] [ 2234.256805] ? __warn+0x85/0x180 [ 2234.260036] ? __list_add_valid_or_report+0x7d/0xa0 [ 2234.264917] ? report_bug+0xfc/0x1d0 [ 2234.268498] ? prb_read_valid+0x1b/0x20 [ 2234.272337] ? handle_bug+0x44/0x70 [ 2234.275828] ? exc_invalid_op+0x17/0x60 [ 2234.279668] ? asm_exc_invalid_op+0x1a/0x20 [ 2234.283859] ? __list_add_valid_or_report+0x7d/0xa0 [ 2234.288733] ? __list_add_valid_or_report+0x7d/0xa0 [ 2234.293612] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2234.298504] ceph_open+0x16b/0x440 [ceph] [ 2234.302526] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2234.307670] do_dentry_open+0x172/0x530 [ 2234.311512] path_openat+0x743/0x9d0 [ 2234.315092] do_filp_open+0xb8/0x120 [ 2234.318673] ? lock_release+0xc5/0x260 [ 2234.322423] ? _raw_spin_unlock+0x23/0x40 [ 2234.326435] ? alloc_fd+0x13f/0x200 [ 2234.329930] do_sys_openat2+0x249/0x290 [ 2234.333769] do_sys_open+0x4f/0x80 [ 2234.337172] do_syscall_64+0x38/0x80 [ 2234.340752] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2234.345803] RIP: 0033:0x7f48f0eed2f2 [ 2234.349384] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2234.368130] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2234.375697] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2234.382828] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2234.389961] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2234.397096] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2234.404228] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2234.411367] [ 2234.413552] irq event stamp: 12892535 [ 2234.417211] hardirqs last enabled at (12892545): [] console_unlock+0x106/0x120 [ 2234.426076] hardirqs last disabled at (12892554): [] console_unlock+0xeb/0x120 [ 2234.434854] softirqs last enabled at (12892502): [] __do_softirq+0x2be/0x371 [ 2234.443548] softirqs last disabled at (12892497): [] irq_exit_rcu+0xb0/0xd0 [ 2234.452068] ---[ end trace 0000000000000000 ]--- [ 2234.456689] ------------[ cut here ]------------ [ 2234.461308] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2234.472955] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2234.481651] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2234.537140] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2234.546269] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2234.553748] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2234.559253] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2234.564737] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2234.583487] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2234.588718] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2234.595854] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2234.602989] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000467b [ 2234.610119] R10: 000000000019b398 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2234.617250] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2234.624385] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2234.632470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2234.638214] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2234.645350] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.652484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2234.659615] Call Trace: [ 2234.662068] [ 2234.664175] ? __warn+0x85/0x180 [ 2234.667409] ? __list_add_valid_or_report+0x7d/0xa0 [ 2234.672296] ? report_bug+0xfc/0x1d0 [ 2234.675873] ? prb_read_valid+0x1b/0x20 [ 2234.679717] ? handle_bug+0x44/0x70 [ 2234.683208] ? exc_invalid_op+0x17/0x60 [ 2234.687046] ? asm_exc_invalid_op+0x1a/0x20 [ 2234.691240] ? __list_add_valid_or_report+0x7d/0xa0 [ 2234.696121] ? __list_add_valid_or_report+0x7d/0xa0 [ 2234.700999] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2234.705899] ceph_add_cap+0x20b/0x620 [ceph] [ 2234.710196] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2234.714918] ? lock_acquire+0xcb/0x2b0 [ 2234.718675] ? lock_is_held_type+0xaa/0x120 [ 2234.722870] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2234.727330] ? __mutex_lock+0x97/0xce0 [ 2234.731082] ? __mutex_lock+0xaec/0xce0 [ 2234.734923] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2234.739473] ? lock_downgrade+0xfd/0x250 [ 2234.743406] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2234.747784] ceph_con_process_message+0x74/0x150 [libceph] [ 2234.753286] ? calc_signature+0xdb/0x100 [libceph] [ 2234.758102] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2234.763434] ? process_scheduled_works+0x22a/0x510 [ 2234.768234] ? process_scheduled_works+0x22a/0x510 [ 2234.773028] ceph_con_workfn+0x344/0x750 [libceph] [ 2234.777834] ? process_scheduled_works+0x22a/0x510 [ 2234.782631] process_scheduled_works+0x2a4/0x510 [ 2234.787261] worker_thread+0x188/0x340 [ 2234.791016] ? rescuer_thread+0x310/0x310 [ 2234.795036] kthread+0xf5/0x130 [ 2234.798180] ? kthread_complete_and_exit+0x20/0x20 [ 2234.802974] ret_from_fork+0x31/0x40 [ 2234.806551] ? kthread_complete_and_exit+0x20/0x20 [ 2234.811347] ret_from_fork_asm+0x11/0x20 [ 2234.815283] [ 2234.817472] irq event stamp: 10417829 [ 2234.821138] hardirqs last enabled at (10417839): [] console_unlock+0x106/0x120 [ 2234.830005] hardirqs last disabled at (10417848): [] console_unlock+0xeb/0x120 [ 2234.838785] softirqs last enabled at (10417766): [] __do_softirq+0x2be/0x371 [ 2234.847477] softirqs last disabled at (10417761): [] irq_exit_rcu+0xb0/0xd0 [ 2234.856005] ---[ end trace 0000000000000000 ]--- [ 2234.860626] ------------[ cut here ]------------ [ 2234.865242] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2234.876896] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2234.885854] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2234.941324] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2234.950277] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2234.957753] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2234.963236] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2234.981986] RSP: 0018:ffffc90001fffaa0 EFLAGS: 00010286 [ 2234.987210] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2234.994344] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2235.001474] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000046bb [ 2235.008612] R10: 000000000019c598 R11: ffffc90001fff930 R12: ffff8881d12c5590 [ 2235.015747] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2235.022873] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2235.030960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2235.036706] CR2: 00007f9bd2a34000 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2235.043839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2235.050972] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2235.058103] Call Trace: [ 2235.060550] [ 2235.062657] ? __warn+0x85/0x180 [ 2235.065888] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.070769] ? report_bug+0xfc/0x1d0 [ 2235.074347] ? prb_read_valid+0x1b/0x20 [ 2235.078187] ? handle_bug+0x44/0x70 [ 2235.081678] ? exc_invalid_op+0x17/0x60 [ 2235.085519] ? asm_exc_invalid_op+0x1a/0x20 [ 2235.089712] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.094599] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2235.099499] ceph_check_caps+0x999/0xae0 [ceph] [ 2235.104054] ? lock_release+0xc5/0x260 [ 2235.107811] ceph_open+0x3cc/0x440 [ceph] [ 2235.111834] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2235.116979] do_dentry_open+0x172/0x530 [ 2235.120819] path_openat+0x743/0x9d0 [ 2235.124402] do_filp_open+0xb8/0x120 [ 2235.127979] ? lock_release+0xc5/0x260 [ 2235.131732] ? _raw_spin_unlock+0x23/0x40 [ 2235.135742] ? alloc_fd+0x13f/0x200 [ 2235.139240] do_sys_openat2+0x249/0x290 [ 2235.143081] do_sys_open+0x4f/0x80 [ 2235.146492] do_syscall_64+0x38/0x80 [ 2235.150074] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2235.155123] RIP: 0033:0x7f48f0eed2f2 [ 2235.158700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2235.177447] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2235.185012] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2235.192145] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2235.199278] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2235.206413] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2235.213545] R13: 00007f48f183e368 R14: 00000000020767c0 R15: 0000000000002a10 [ 2235.220689] [ 2235.222879] irq event stamp: 12894775 [ 2235.226544] hardirqs last enabled at (12894785): [] console_unlock+0x106/0x120 [ 2235.235418] hardirqs last disabled at (12894802): [] console_unlock+0xeb/0x120 [ 2235.244201] softirqs last enabled at (12894818): [] __do_softirq+0x2be/0x371 [ 2235.252891] softirqs last disabled at (12894811): [] irq_exit_rcu+0xb0/0xd0 [ 2235.261411] ---[ end trace 0000000000000000 ]--- [ 2235.266030] ------------[ cut here ]------------ [ 2235.270649] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2235.282299] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2235.291255] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2235.346723] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2235.355674] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2235.363152] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2235.368638] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2235.387383] RSP: 0018:ffffc90001fefbf0 EFLAGS: 00010282 [ 2235.392612] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2235.399743] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2235.406878] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000046f6 [ 2235.414011] R10: 000000000019d350 R11: ffffc90001fefa80 R12: ffff888158a68750 [ 2235.421142] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2235.428275] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2235.436362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2235.442107] CR2: 00007fb38d645000 CR3: 0000000108c72006 CR4: 00000000003706e0 [ 2235.449240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2235.456376] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2235.463507] Call Trace: [ 2235.465958] [ 2235.468066] ? __warn+0x85/0x180 [ 2235.471299] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.476179] ? report_bug+0xfc/0x1d0 [ 2235.479767] ? prb_read_valid+0x1b/0x20 [ 2235.483609] ? handle_bug+0x44/0x70 [ 2235.487104] ? exc_invalid_op+0x17/0x60 [ 2235.490947] ? asm_exc_invalid_op+0x1a/0x20 [ 2235.495140] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.500024] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2235.504920] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2235.509647] lookup_fast+0x6d/0xf0 [ 2235.513055] walk_component+0x26/0x160 [ 2235.516812] link_path_walk+0x24d/0x380 [ 2235.520656] ? path_openat+0x80/0x9d0 [ 2235.524327] path_openat+0x8e/0x9d0 [ 2235.527828] do_filp_open+0xb8/0x120 [ 2235.531418] ? lock_release+0xc5/0x260 [ 2235.535175] ? _raw_spin_unlock+0x23/0x40 [ 2235.539186] ? alloc_fd+0x13f/0x200 [ 2235.542684] do_sys_openat2+0x249/0x290 [ 2235.546530] do_sys_open+0x4f/0x80 [ 2235.549942] do_syscall_64+0x38/0x80 [ 2235.553522] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2235.558579] RIP: 0033:0x7f48f0ef2805 [ 2235.562162] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2235.580905] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2235.588473] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2235.595605] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2235.602738] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2235.609873] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2235.617004] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2235.624149] [ 2235.626338] irq event stamp: 12825025 [ 2235.630004] hardirqs last enabled at (12825035): [] console_unlock+0x106/0x120 [ 2235.638870] hardirqs last disabled at (12825044): [] console_unlock+0xeb/0x120 [ 2235.647650] softirqs last enabled at (12824992): [] __do_softirq+0x2be/0x371 [ 2235.656344] softirqs last disabled at (12824987): [] irq_exit_rcu+0xb0/0xd0 [ 2235.664861] ---[ end trace 0000000000000000 ]--- [ 2235.669483] ------------[ cut here ]------------ [ 2235.674100] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2235.685754] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2235.694455] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2235.750011] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2235.759142] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2235.766622] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2235.772133] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2235.777620] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2235.796366] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2235.801591] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2235.808724] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2235.815859] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004731 [ 2235.822992] R10: 000000000019e318 R11: ffffc9000030b828 R12: ffff8881d1295590 [ 2235.830124] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2235.837256] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2235.845342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2235.851087] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2235.858221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2235.865355] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2235.872489] Call Trace: [ 2235.874940] [ 2235.877048] ? __warn+0x85/0x180 [ 2235.880292] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.885179] ? report_bug+0xfc/0x1d0 [ 2235.888764] ? prb_read_valid+0x1b/0x20 [ 2235.892608] ? handle_bug+0x44/0x70 [ 2235.896104] ? exc_invalid_op+0x17/0x60 [ 2235.899945] ? asm_exc_invalid_op+0x1a/0x20 [ 2235.904150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.909039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2235.913923] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2235.918826] ceph_add_cap+0x20b/0x620 [ceph] [ 2235.923132] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2235.927870] ? lock_acquire+0xcb/0x2b0 [ 2235.931636] ? lock_is_held_type+0xaa/0x120 [ 2235.935830] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2235.940290] ? __mutex_lock+0x97/0xce0 [ 2235.944052] ? __mutex_lock+0xaec/0xce0 [ 2235.947902] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2235.952458] ? lock_downgrade+0xfd/0x250 [ 2235.956403] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2235.960802] ceph_con_process_message+0x74/0x150 [libceph] [ 2235.966307] ? calc_signature+0xdb/0x100 [libceph] [ 2235.971131] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2235.976472] ? process_scheduled_works+0x22a/0x510 [ 2235.981281] ? process_scheduled_works+0x22a/0x510 [ 2235.986074] ceph_con_workfn+0x344/0x750 [libceph] [ 2235.990888] ? process_scheduled_works+0x22a/0x510 [ 2235.995686] process_scheduled_works+0x2a4/0x510 [ 2236.000316] worker_thread+0x188/0x340 [ 2236.004077] ? rescuer_thread+0x310/0x310 [ 2236.008101] kthread+0xf5/0x130 [ 2236.011252] ? kthread_complete_and_exit+0x20/0x20 [ 2236.016055] ret_from_fork+0x31/0x40 [ 2236.019642] ? kthread_complete_and_exit+0x20/0x20 [ 2236.024444] ret_from_fork_asm+0x11/0x20 [ 2236.028392] [ 2236.030588] irq event stamp: 10420729 [ 2236.034252] hardirqs last enabled at (10420739): [] console_unlock+0x106/0x120 [ 2236.043119] hardirqs last disabled at (10420748): [] console_unlock+0xeb/0x120 [ 2236.051898] softirqs last enabled at (10420666): [] __do_softirq+0x2be/0x371 [ 2236.060592] softirqs last disabled at (10420661): [] irq_exit_rcu+0xb0/0xd0 [ 2236.069112] ---[ end trace 0000000000000000 ]--- [ 2236.073734] ------------[ cut here ]------------ [ 2236.078358] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2236.090006] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2236.098967] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2236.154487] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2236.163439] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2236.170917] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2236.176410] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2236.195160] RSP: 0018:ffffc90001fc7bf0 EFLAGS: 00010282 [ 2236.200390] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2236.207527] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2236.214659] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004771 [ 2236.221793] R10: 000000000019f570 R11: ffffc90001fc7a80 R12: ffff888158a68750 [ 2236.228925] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2236.236059] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2236.244144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.249890] CR2: 00007fb38d5d5000 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2236.257023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2236.264157] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2236.271290] Call Trace: [ 2236.273742] [ 2236.275849] ? __warn+0x85/0x180 [ 2236.279082] ? __list_add_valid_or_report+0x7d/0xa0 [ 2236.283968] ? report_bug+0xfc/0x1d0 [ 2236.287546] ? prb_read_valid+0x1b/0x20 [ 2236.291393] ? handle_bug+0x44/0x70 [ 2236.294889] ? exc_invalid_op+0x17/0x60 [ 2236.298728] ? asm_exc_invalid_op+0x1a/0x20 [ 2236.302924] ? __list_add_valid_or_report+0x7d/0xa0 [ 2236.307805] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2236.312712] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2236.317440] lookup_fast+0x6d/0xf0 [ 2236.320848] walk_component+0x26/0x160 [ 2236.324602] link_path_walk+0x24d/0x380 [ 2236.328439] ? path_openat+0x80/0x9d0 [ 2236.332110] path_openat+0x8e/0x9d0 [ 2236.335613] do_filp_open+0xb8/0x120 [ 2236.339203] ? lock_release+0xc5/0x260 [ 2236.342960] ? _raw_spin_unlock+0x23/0x40 [ 2236.346977] ? alloc_fd+0x13f/0x200 [ 2236.350476] do_sys_openat2+0x249/0x290 [ 2236.354321] do_sys_open+0x4f/0x80 [ 2236.357735] do_syscall_64+0x38/0x80 [ 2236.361323] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2236.366380] RIP: 0033:0x7f48f0ef2805 [ 2236.369962] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2236.388706] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2236.396272] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2236.403407] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2236.410538] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2236.417671] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2236.424804] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2236.431954] [ 2236.434147] irq event stamp: 12928783 [ 2236.437813] hardirqs last enabled at (12928793): [] console_unlock+0x106/0x120 [ 2236.446679] hardirqs last disabled at (12928802): [] console_unlock+0xeb/0x120 [ 2236.455458] softirqs last enabled at (12928770): [] __do_softirq+0x2be/0x371 [ 2236.464152] softirqs last disabled at (12928765): [] irq_exit_rcu+0xb0/0xd0 [ 2236.472670] ---[ end trace 0000000000000000 ]--- [ 2236.477291] ------------[ cut here ]------------ [ 2236.481910] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2236.493564] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2236.502263] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2236.557818] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2236.566951] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2236.574431] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2236.579950] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2236.585435] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2236.604184] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2236.609417] RAX: 0000000000000000 RBX: ffff8881d0bf4e40 RCX: 0000000000000000 [ 2236.616552] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2236.623687] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000047ac [ 2236.630826] R10: 00000000001a05a8 R11: ffffc9000030b828 R12: ffff8881d0bf5590 [ 2236.637959] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2236.645091] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2236.653178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.658923] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2236.666057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2236.673188] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2236.680323] Call Trace: [ 2236.682775] [ 2236.684882] ? __warn+0x85/0x180 [ 2236.688125] ? __list_add_valid_or_report+0x7d/0xa0 [ 2236.693013] ? report_bug+0xfc/0x1d0 [ 2236.696599] ? prb_read_valid+0x1b/0x20 [ 2236.700451] ? handle_bug+0x44/0x70 [ 2236.703949] ? exc_invalid_op+0x17/0x60 [ 2236.707789] ? asm_exc_invalid_op+0x1a/0x20 [ 2236.711994] ? __list_add_valid_or_report+0x7d/0xa0 [ 2236.716882] ? __list_add_valid_or_report+0x7d/0xa0 [ 2236.721768] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2236.726669] ceph_add_cap+0x20b/0x620 [ceph] [ 2236.730975] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2236.735708] ? lock_acquire+0xcb/0x2b0 [ 2236.739469] ? lock_is_held_type+0xaa/0x120 [ 2236.743664] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2236.748126] ? __mutex_lock+0x97/0xce0 [ 2236.751886] ? __mutex_lock+0xaec/0xce0 [ 2236.755736] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2236.760292] ? lock_downgrade+0xfd/0x250 [ 2236.764227] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2236.768621] ceph_con_process_message+0x74/0x150 [libceph] [ 2236.774132] ? calc_signature+0xdb/0x100 [libceph] [ 2236.778957] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2236.784297] ? process_scheduled_works+0x22a/0x510 [ 2236.789107] ? process_scheduled_works+0x22a/0x510 [ 2236.793900] ceph_con_workfn+0x344/0x750 [libceph] [ 2236.798715] ? process_scheduled_works+0x22a/0x510 [ 2236.803511] process_scheduled_works+0x2a4/0x510 [ 2236.808150] worker_thread+0x188/0x340 [ 2236.811913] ? rescuer_thread+0x310/0x310 [ 2236.815935] kthread+0xf5/0x130 [ 2236.819087] ? kthread_complete_and_exit+0x20/0x20 [ 2236.823891] ret_from_fork+0x31/0x40 [ 2236.827477] ? kthread_complete_and_exit+0x20/0x20 [ 2236.832279] ret_from_fork_asm+0x11/0x20 [ 2236.836230] [ 2236.838421] irq event stamp: 10432527 [ 2236.842089] hardirqs last enabled at (10432537): [] console_unlock+0x106/0x120 [ 2236.850962] hardirqs last disabled at (10432546): [] console_unlock+0xeb/0x120 [ 2236.859742] softirqs last enabled at (10432464): [] __do_softirq+0x2be/0x371 [ 2236.868443] softirqs last disabled at (10432459): [] irq_exit_rcu+0xb0/0xd0 [ 2236.876963] ---[ end trace 0000000000000000 ]--- [ 2236.881585] ------------[ cut here ]------------ [ 2236.886207] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2236.897858] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2236.906822] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2236.962306] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2236.971276] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2236.978755] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2236.984249] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2237.002993] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2237.008221] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2237.015354] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2237.022486] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000047ec [ 2237.029620] R10: 00000000001a1830 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2237.036752] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2237.043884] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2237.051970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2237.057714] CR2: 00007f9bd2a34000 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2237.064858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2237.071992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2237.079125] Call Trace: [ 2237.081577] [ 2237.083687] ? __warn+0x85/0x180 [ 2237.086926] ? __list_add_valid_or_report+0x7d/0xa0 [ 2237.091819] ? report_bug+0xfc/0x1d0 [ 2237.095402] ? prb_read_valid+0x1b/0x20 [ 2237.099244] ? handle_bug+0x44/0x70 [ 2237.102742] ? exc_invalid_op+0x17/0x60 [ 2237.106582] ? asm_exc_invalid_op+0x1a/0x20 [ 2237.110782] ? __list_add_valid_or_report+0x7d/0xa0 [ 2237.115667] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2237.120563] try_get_cap_refs+0x433/0x880 [ceph] [ 2237.125215] __ceph_get_caps+0xf8/0x760 [ceph] [ 2237.129694] ? ceph_read_iter+0x134/0x860 [ceph] [ 2237.134334] ceph_read_iter+0x134/0x860 [ceph] [ 2237.138805] ? selinux_file_permission+0x119/0x150 [ 2237.143611] ? vfs_read+0x20f/0x300 [ 2237.147105] vfs_read+0x20f/0x300 [ 2237.150437] ksys_pread64+0x65/0xa0 [ 2237.153937] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2237.159091] do_syscall_64+0x38/0x80 [ 2237.162674] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2237.167728] RIP: 0033:0x7f48f0eeb88b [ 2237.171315] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2237.190062] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2237.197644] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2237.204778] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2237.211907] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2237.219042] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2237.226176] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2237.233319] [ 2237.235508] irq event stamp: 12899771 [ 2237.239174] hardirqs last enabled at (12899781): [] console_unlock+0x106/0x120 [ 2237.248038] hardirqs last disabled at (12899790): [] console_unlock+0xeb/0x120 [ 2237.256821] softirqs last enabled at (12899748): [] __do_softirq+0x2be/0x371 [ 2237.265519] softirqs last disabled at (12899743): [] irq_exit_rcu+0xb0/0xd0 [ 2237.274048] ---[ end trace 0000000000000000 ]--- [ 2237.278669] ------------[ cut here ]------------ [ 2237.283292] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2237.294948] WARNING: CPU: 2 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2237.303907] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2237.359428] CPU: 2 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2237.368380] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2237.375860] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2237.381344] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2237.400090] RSP: 0018:ffffc90001fcfbf0 EFLAGS: 00010282 [ 2237.405316] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2237.412448] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2237.419583] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004824 [ 2237.426718] R10: 00000000001a2988 R11: ffffc90001fcfa80 R12: ffff888158a68750 [ 2237.433848] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2237.440982] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2237.449067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2237.454814] CR2: 00007fb38d645000 CR3: 000000011aeae004 CR4: 00000000003706e0 [ 2237.461946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2237.469079] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2237.476212] Call Trace: [ 2237.478666] [ 2237.480771] ? __warn+0x85/0x180 [ 2237.484004] ? __list_add_valid_or_report+0x7d/0xa0 [ 2237.488884] ? report_bug+0xfc/0x1d0 [ 2237.492464] ? prb_read_valid+0x1b/0x20 [ 2237.496305] ? handle_bug+0x44/0x70 [ 2237.499798] ? exc_invalid_op+0x17/0x60 [ 2237.503635] ? asm_exc_invalid_op+0x1a/0x20 [ 2237.507830] ? __list_add_valid_or_report+0x7d/0xa0 [ 2237.512710] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2237.517609] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2237.522337] lookup_fast+0x6d/0xf0 [ 2237.525751] walk_component+0x26/0x160 [ 2237.529508] link_path_walk+0x24d/0x380 [ 2237.533343] ? path_openat+0x80/0x9d0 [ 2237.537015] path_openat+0x8e/0x9d0 [ 2237.540521] do_filp_open+0xb8/0x120 [ 2237.544109] ? lock_release+0xc5/0x260 [ 2237.547865] ? _raw_spin_unlock+0x23/0x40 [ 2237.551882] ? alloc_fd+0x13f/0x200 [ 2237.555381] do_sys_openat2+0x249/0x290 [ 2237.559229] do_sys_open+0x4f/0x80 [ 2237.562641] do_syscall_64+0x38/0x80 [ 2237.566228] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2237.571280] RIP: 0033:0x7f48f0eed2f2 [ 2237.574859] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2237.593603] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2237.601171] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2237.608303] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2237.615435] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2237.622571] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2237.629700] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2237.636847] [ 2237.639036] irq event stamp: 12800285 [ 2237.642702] hardirqs last enabled at (12800295): [] console_unlock+0x106/0x120 [ 2237.651568] hardirqs last disabled at (12800304): [] console_unlock+0xeb/0x120 [ 2237.660348] softirqs last enabled at (12800252): [] __do_softirq+0x2be/0x371 [ 2237.669039] softirqs last disabled at (12800247): [] irq_exit_rcu+0xb0/0xd0 [ 2237.677560] ---[ end trace 0000000000000000 ]--- [ 2237.682180] ------------[ cut here ]------------ [ 2237.686797] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2237.698445] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2237.707399] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2237.762920] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2237.771873] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2237.779350] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2237.784838] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2237.803583] RSP: 0018:ffffc90001fefbf0 EFLAGS: 00010282 [ 2237.808810] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2237.815942] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2237.823075] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000485f [ 2237.830209] R10: 00000000001a39d0 R11: ffffc90001fefa80 R12: ffff8881c5350750 [ 2237.837340] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2237.844474] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2237.852560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2237.858307] CR2: 00007f9bd2a8f000 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2237.865440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2237.872572] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2237.879705] Call Trace: [ 2237.882158] [ 2237.884264] ? __warn+0x85/0x180 [ 2237.887498] ? __list_add_valid_or_report+0x7d/0xa0 [ 2237.892376] ? report_bug+0xfc/0x1d0 [ 2237.895957] ? prb_read_valid+0x1b/0x20 [ 2237.899797] ? handle_bug+0x44/0x70 [ 2237.903288] ? exc_invalid_op+0x17/0x60 [ 2237.907127] ? asm_exc_invalid_op+0x1a/0x20 [ 2237.911323] ? __list_add_valid_or_report+0x7d/0xa0 [ 2237.916203] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2237.921101] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2237.925828] lookup_fast+0x6d/0xf0 [ 2237.929238] walk_component+0x26/0x160 [ 2237.932993] link_path_walk+0x24d/0x380 [ 2237.936836] ? path_openat+0x80/0x9d0 [ 2237.940507] path_openat+0x8e/0x9d0 [ 2237.944013] do_filp_open+0xb8/0x120 [ 2237.947601] ? lock_release+0xc5/0x260 [ 2237.951359] ? _raw_spin_unlock+0x23/0x40 [ 2237.955374] ? alloc_fd+0x13f/0x200 [ 2237.958874] do_sys_openat2+0x249/0x290 [ 2237.962720] do_sys_open+0x4f/0x80 [ 2237.966133] do_syscall_64+0x38/0x80 [ 2237.969712] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2237.974763] RIP: 0033:0x7f48f0ef2805 [ 2237.978343] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2237.997089] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2238.004655] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2238.011786] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2238.018920] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2238.026054] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2238.033184] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2238.040332] [ 2238.042528] irq event stamp: 12830445 [ 2238.046195] hardirqs last enabled at (12830455): [] console_unlock+0x106/0x120 [ 2238.055060] hardirqs last disabled at (12830464): [] console_unlock+0xeb/0x120 [ 2238.063841] softirqs last enabled at (12830412): [] __do_softirq+0x2be/0x371 [ 2238.072532] softirqs last disabled at (12830407): [] irq_exit_rcu+0xb0/0xd0 [ 2238.081059] ---[ end trace 0000000000000000 ]--- [ 2238.085684] ------------[ cut here ]------------ [ 2238.090300] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2238.101948] WARNING: CPU: 0 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2238.110902] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2238.166426] CPU: 0 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2238.175395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2238.182872] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2238.188356] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2238.207112] RSP: 0018:ffffc90001fe7bf0 EFLAGS: 00010282 [ 2238.212338] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2238.219469] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2238.226604] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000489a [ 2238.233738] R10: 00000000001a4b78 R11: ffffc90001fe7a80 R12: ffff888158a68750 [ 2238.240867] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2238.248001] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2238.256088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2238.261832] CR2: 00007f9bd2aa9000 CR3: 0000000108c08005 CR4: 00000000003706f0 [ 2238.268967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2238.276098] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2238.283232] Call Trace: [ 2238.285686] [ 2238.287793] ? __warn+0x85/0x180 [ 2238.291024] ? __list_add_valid_or_report+0x7d/0xa0 [ 2238.295905] ? report_bug+0xfc/0x1d0 [ 2238.299484] ? prb_read_valid+0x1b/0x20 [ 2238.303326] ? handle_bug+0x44/0x70 [ 2238.306823] ? exc_invalid_op+0x17/0x60 [ 2238.310664] ? asm_exc_invalid_op+0x1a/0x20 [ 2238.314860] ? __list_add_valid_or_report+0x7d/0xa0 [ 2238.319748] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2238.324662] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2238.329401] lookup_fast+0x6d/0xf0 [ 2238.332808] walk_component+0x26/0x160 [ 2238.336564] link_path_walk+0x24d/0x380 [ 2238.340408] ? path_openat+0x80/0x9d0 [ 2238.344080] path_openat+0x8e/0x9d0 [ 2238.347584] do_filp_open+0xb8/0x120 [ 2238.351172] ? lock_release+0xc5/0x260 [ 2238.354930] ? _raw_spin_unlock+0x23/0x40 [ 2238.358945] ? alloc_fd+0x13f/0x200 [ 2238.362447] do_sys_openat2+0x249/0x290 [ 2238.366292] do_sys_open+0x4f/0x80 [ 2238.369705] do_syscall_64+0x38/0x80 [ 2238.373291] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2238.378342] RIP: 0033:0x7f48f0eed2f2 [ 2238.381921] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2238.400667] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2238.408235] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2238.415366] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2238.422499] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2238.429633] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2238.436765] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2238.443910] [ 2238.446099] irq event stamp: 12822701 [ 2238.449765] hardirqs last enabled at (12822711): [] console_unlock+0x106/0x120 [ 2238.458631] hardirqs last disabled at (12822720): [] console_unlock+0xeb/0x120 [ 2238.467411] softirqs last enabled at (12822668): [] __do_softirq+0x2be/0x371 [ 2238.476104] softirqs last disabled at (12822663): [] irq_exit_rcu+0xb0/0xd0 [ 2238.484623] ---[ end trace 0000000000000000 ]--- [ 2238.489243] ------------[ cut here ]------------ [ 2238.493861] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2238.505507] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2238.514463] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2238.570031] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2238.578990] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2238.586467] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2238.591953] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2238.610703] RSP: 0018:ffffc90001fc7bf0 EFLAGS: 00010282 [ 2238.615934] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2238.623066] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2238.630207] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000048d5 [ 2238.637341] R10: 00000000001a5bd8 R11: ffffc90001fc7a80 R12: ffff8881c5350750 [ 2238.644473] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2238.651606] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2238.659693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2238.665440] CR2: 00007fb38d5d5000 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2238.672572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2238.679705] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2238.686838] Call Trace: [ 2238.689290] [ 2238.691398] ? __warn+0x85/0x180 [ 2238.694629] ? __list_add_valid_or_report+0x7d/0xa0 [ 2238.699511] ? report_bug+0xfc/0x1d0 [ 2238.703087] ? prb_read_valid+0x1b/0x20 [ 2238.706930] ? handle_bug+0x44/0x70 [ 2238.710420] ? exc_invalid_op+0x17/0x60 [ 2238.714261] ? asm_exc_invalid_op+0x1a/0x20 [ 2238.718454] ? __list_add_valid_or_report+0x7d/0xa0 [ 2238.723335] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2238.728235] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2238.732962] lookup_fast+0x6d/0xf0 [ 2238.736370] walk_component+0x26/0x160 [ 2238.740130] link_path_walk+0x24d/0x380 [ 2238.743969] ? path_openat+0x80/0x9d0 [ 2238.747641] path_openat+0x8e/0x9d0 [ 2238.751144] do_filp_open+0xb8/0x120 [ 2238.754732] ? lock_release+0xc5/0x260 [ 2238.758490] ? _raw_spin_unlock+0x23/0x40 [ 2238.762508] ? alloc_fd+0x13f/0x200 [ 2238.766006] do_sys_openat2+0x249/0x290 [ 2238.769851] do_sys_open+0x4f/0x80 [ 2238.773257] do_syscall_64+0x38/0x80 [ 2238.776844] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2238.781895] RIP: 0033:0x7f48f0ef2805 [ 2238.785476] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2238.804221] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2238.811788] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2238.818920] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2238.826052] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2238.833187] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2238.840317] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2238.847464] [ 2238.849653] irq event stamp: 12933925 [ 2238.853318] hardirqs last enabled at (12933935): [] console_unlock+0x106/0x120 [ 2238.862183] hardirqs last disabled at (12933944): [] console_unlock+0xeb/0x120 [ 2238.870969] softirqs last enabled at (12933960): [] __do_softirq+0x2be/0x371 [ 2238.879668] softirqs last disabled at (12933971): [] irq_exit_rcu+0xb0/0xd0 [ 2238.888193] ---[ end trace 0000000000000000 ]--- [ 2238.892813] ------------[ cut here ]------------ [ 2238.897433] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2238.909088] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2238.917786] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2238.973342] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2238.982475] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2238.989954] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2238.995475] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2239.000968] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2239.019714] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2239.024942] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2239.032072] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2239.039209] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004910 [ 2239.046347] R10: 00000000001a6d48 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2239.053480] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2239.060612] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2239.068699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2239.074445] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2239.081578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2239.088711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2239.095844] Call Trace: [ 2239.098297] [ 2239.100403] ? __warn+0x85/0x180 [ 2239.103639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.108525] ? report_bug+0xfc/0x1d0 [ 2239.112114] ? prb_read_valid+0x1b/0x20 [ 2239.115963] ? handle_bug+0x44/0x70 [ 2239.119462] ? exc_invalid_op+0x17/0x60 [ 2239.123303] ? asm_exc_invalid_op+0x1a/0x20 [ 2239.127507] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.132395] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.137280] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2239.142181] ceph_add_cap+0x20b/0x620 [ceph] [ 2239.146490] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2239.151229] ? lock_acquire+0xcb/0x2b0 [ 2239.154992] ? lock_is_held_type+0xaa/0x120 [ 2239.159186] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2239.163646] ? __mutex_lock+0x97/0xce0 [ 2239.167408] ? __mutex_lock+0xaec/0xce0 [ 2239.171257] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2239.175813] ? lock_downgrade+0xfd/0x250 [ 2239.179749] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2239.184141] ceph_con_process_message+0x74/0x150 [libceph] [ 2239.189644] ? calc_signature+0xdb/0x100 [libceph] [ 2239.194470] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2239.199812] ? process_scheduled_works+0x22a/0x510 [ 2239.204619] ? process_scheduled_works+0x22a/0x510 [ 2239.209414] ceph_con_workfn+0x344/0x750 [libceph] [ 2239.214231] ? process_scheduled_works+0x22a/0x510 [ 2239.219026] process_scheduled_works+0x2a4/0x510 [ 2239.223657] worker_thread+0x188/0x340 [ 2239.227419] ? rescuer_thread+0x310/0x310 [ 2239.231440] kthread+0xf5/0x130 [ 2239.234593] ? kthread_complete_and_exit+0x20/0x20 [ 2239.239396] ret_from_fork+0x31/0x40 [ 2239.242982] ? kthread_complete_and_exit+0x20/0x20 [ 2239.247784] ret_from_fork_asm+0x11/0x20 [ 2239.251734] [ 2239.253927] irq event stamp: 10438097 [ 2239.257594] hardirqs last enabled at (10438107): [] console_unlock+0x106/0x120 [ 2239.266468] hardirqs last disabled at (10438116): [] console_unlock+0xeb/0x120 [ 2239.275247] softirqs last enabled at (10438034): [] __do_softirq+0x2be/0x371 [ 2239.283950] softirqs last disabled at (10438029): [] irq_exit_rcu+0xb0/0xd0 [ 2239.292478] ---[ end trace 0000000000000000 ]--- [ 2239.297099] ------------[ cut here ]------------ [ 2239.301712] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2239.313361] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2239.322317] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2239.377844] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2239.386797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2239.394280] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2239.399770] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2239.418517] RSP: 0018:ffffc90001fc7bf0 EFLAGS: 00010282 [ 2239.423742] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2239.430874] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2239.438008] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004950 [ 2239.445191] R10: 00000000001a7ed8 R11: ffffc90001fc7a80 R12: ffff8881d106df58 [ 2239.452325] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2239.459458] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2239.467544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2239.473290] CR2: 00007fb38d5d5000 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2239.480424] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2239.487558] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2239.494689] Call Trace: [ 2239.497142] [ 2239.499248] ? __warn+0x85/0x180 [ 2239.502482] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.507362] ? report_bug+0xfc/0x1d0 [ 2239.510940] ? prb_read_valid+0x1b/0x20 [ 2239.514785] ? handle_bug+0x44/0x70 [ 2239.518281] ? exc_invalid_op+0x17/0x60 [ 2239.522125] ? asm_exc_invalid_op+0x1a/0x20 [ 2239.526325] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.531205] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2239.536103] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2239.540831] lookup_fast+0x6d/0xf0 [ 2239.544239] walk_component+0x26/0x160 [ 2239.547994] link_path_walk+0x24d/0x380 [ 2239.551831] ? path_openat+0x80/0x9d0 [ 2239.555502] path_openat+0x8e/0x9d0 [ 2239.559006] do_filp_open+0xb8/0x120 [ 2239.562595] ? lock_release+0xc5/0x260 [ 2239.566351] ? _raw_spin_unlock+0x23/0x40 [ 2239.570367] ? alloc_fd+0x13f/0x200 [ 2239.573866] do_sys_openat2+0x249/0x290 [ 2239.577713] do_sys_open+0x4f/0x80 [ 2239.581127] do_syscall_64+0x38/0x80 [ 2239.584714] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2239.589766] RIP: 0033:0x7f48f0ef2805 [ 2239.593344] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2239.612091] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2239.619655] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2239.626789] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2239.633923] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2239.641054] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2239.648188] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2239.655338] [ 2239.657532] irq event stamp: 12935587 [ 2239.661195] hardirqs last enabled at (12935597): [] console_unlock+0x106/0x120 [ 2239.670063] hardirqs last disabled at (12935606): [] console_unlock+0xeb/0x120 [ 2239.678843] softirqs last enabled at (12935574): [] __do_softirq+0x2be/0x371 [ 2239.687534] softirqs last disabled at (12935567): [] irq_exit_rcu+0xb0/0xd0 [ 2239.696054] ---[ end trace 0000000000000000 ]--- [ 2239.700682] ------------[ cut here ]------------ [ 2239.705304] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2239.716955] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2239.725654] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2239.781202] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2239.790334] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2239.797813] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2239.803327] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2239.808819] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2239.827567] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2239.832803] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2239.839935] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2239.847069] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000498b [ 2239.854208] R10: 00000000001a8f80 R11: ffffc9000030b828 R12: ffff8881c5357cb0 [ 2239.861341] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2239.868474] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2239.876559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2239.882308] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2239.889441] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2239.896575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2239.903715] Call Trace: [ 2239.906167] [ 2239.908273] ? __warn+0x85/0x180 [ 2239.911509] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.916395] ? report_bug+0xfc/0x1d0 [ 2239.919981] ? prb_read_valid+0x1b/0x20 [ 2239.923826] ? handle_bug+0x44/0x70 [ 2239.927323] ? exc_invalid_op+0x17/0x60 [ 2239.931163] ? asm_exc_invalid_op+0x1a/0x20 [ 2239.935367] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.940256] ? __list_add_valid_or_report+0x7d/0xa0 [ 2239.945141] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2239.950044] ceph_add_cap+0x20b/0x620 [ceph] [ 2239.954350] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2239.959088] ? lock_acquire+0xcb/0x2b0 [ 2239.962852] ? lock_is_held_type+0xaa/0x120 [ 2239.967048] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2239.971517] ? __mutex_lock+0x97/0xce0 [ 2239.975278] ? __mutex_lock+0xaec/0xce0 [ 2239.979127] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2239.983682] ? lock_downgrade+0xfd/0x250 [ 2239.987618] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2239.992011] ceph_con_process_message+0x74/0x150 [libceph] [ 2239.997514] ? calc_signature+0xdb/0x100 [libceph] [ 2240.002340] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2240.007679] ? process_scheduled_works+0x22a/0x510 [ 2240.012482] ? process_scheduled_works+0x22a/0x510 [ 2240.017276] ceph_con_workfn+0x344/0x750 [libceph] [ 2240.022090] ? process_scheduled_works+0x22a/0x510 [ 2240.026886] process_scheduled_works+0x2a4/0x510 [ 2240.031517] worker_thread+0x188/0x340 [ 2240.035281] ? rescuer_thread+0x310/0x310 [ 2240.039300] kthread+0xf5/0x130 [ 2240.042452] ? kthread_complete_and_exit+0x20/0x20 [ 2240.047249] ret_from_fork+0x31/0x40 [ 2240.050832] ? kthread_complete_and_exit+0x20/0x20 [ 2240.055627] ret_from_fork_asm+0x11/0x20 [ 2240.059580] [ 2240.061768] irq event stamp: 10440905 [ 2240.065435] hardirqs last enabled at (10440915): [] console_unlock+0x106/0x120 [ 2240.074303] hardirqs last disabled at (10440924): [] console_unlock+0xeb/0x120 [ 2240.083091] softirqs last enabled at (10440840): [] __do_softirq+0x2be/0x371 [ 2240.091791] softirqs last disabled at (10440835): [] irq_exit_rcu+0xb0/0xd0 [ 2240.100321] ---[ end trace 0000000000000000 ]--- [ 2240.104943] ------------[ cut here ]------------ [ 2240.109566] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2240.121216] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2240.130177] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2240.185700] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2240.194656] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2240.202135] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2240.207622] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2240.226370] RSP: 0018:ffffc90001fc7c88 EFLAGS: 00010286 [ 2240.231594] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2240.238727] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2240.245861] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000049cb [ 2240.252995] R10: 00000000001aa0d8 R11: ffffc90001fc7b18 R12: ffff8881d0bec200 [ 2240.260125] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2240.267259] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2240.275346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2240.281091] CR2: 00007fb38d5d5000 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2240.288224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2240.295357] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2240.302490] Call Trace: [ 2240.304943] [ 2240.307050] ? __warn+0x85/0x180 [ 2240.310281] ? __list_add_valid_or_report+0x7d/0xa0 [ 2240.315165] ? report_bug+0xfc/0x1d0 [ 2240.318751] ? prb_read_valid+0x1b/0x20 [ 2240.322593] ? handle_bug+0x44/0x70 [ 2240.326090] ? exc_invalid_op+0x17/0x60 [ 2240.329929] ? asm_exc_invalid_op+0x1a/0x20 [ 2240.334125] ? __list_add_valid_or_report+0x7d/0xa0 [ 2240.339005] ? __list_add_valid_or_report+0x7d/0xa0 [ 2240.343884] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2240.348784] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2240.353509] lookup_fast+0x6d/0xf0 [ 2240.356918] path_openat+0xe4/0x9d0 [ 2240.360418] do_filp_open+0xb8/0x120 [ 2240.364006] ? lock_release+0xc5/0x260 [ 2240.367765] ? _raw_spin_unlock+0x23/0x40 [ 2240.371782] ? alloc_fd+0x13f/0x200 [ 2240.375280] do_sys_openat2+0x249/0x290 [ 2240.379127] do_sys_open+0x4f/0x80 [ 2240.382541] do_syscall_64+0x38/0x80 [ 2240.386126] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2240.391178] RIP: 0033:0x7f48f0ef2805 [ 2240.394757] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2240.413504] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2240.421069] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2240.428201] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2240.435335] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2240.442471] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2240.449603] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2240.456747] [ 2240.458943] irq event stamp: 12937483 [ 2240.462610] hardirqs last enabled at (12937493): [] console_unlock+0x106/0x120 [ 2240.471475] hardirqs last disabled at (12937502): [] console_unlock+0xeb/0x120 [ 2240.480253] softirqs last enabled at (12937460): [] __do_softirq+0x2be/0x371 [ 2240.488946] softirqs last disabled at (12937455): [] irq_exit_rcu+0xb0/0xd0 [ 2240.497466] ---[ end trace 0000000000000000 ]--- [ 2240.502087] ------------[ cut here ]------------ [ 2240.506706] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2240.518355] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2240.527049] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2240.582596] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2240.591730] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2240.599209] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2240.604721] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2240.610205] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2240.628952] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2240.634180] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2240.641312] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2240.648446] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004a04 [ 2240.655579] R10: 00000000001aade8 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2240.662718] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2240.669851] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2240.677949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2240.683701] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2240.690835] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2240.697967] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2240.705100] Call Trace: [ 2240.707552] [ 2240.709661] ? __warn+0x85/0x180 [ 2240.712904] ? __list_add_valid_or_report+0x7d/0xa0 [ 2240.717791] ? report_bug+0xfc/0x1d0 [ 2240.721376] ? prb_read_valid+0x1b/0x20 [ 2240.725220] ? handle_bug+0x44/0x70 [ 2240.728717] ? exc_invalid_op+0x17/0x60 [ 2240.732559] ? asm_exc_invalid_op+0x1a/0x20 [ 2240.736763] ? __list_add_valid_or_report+0x7d/0xa0 [ 2240.741651] ? __list_add_valid_or_report+0x7d/0xa0 [ 2240.746536] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2240.751438] ceph_add_cap+0x20b/0x620 [ceph] [ 2240.755745] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2240.760477] ? lock_acquire+0xcb/0x2b0 [ 2240.764239] ? lock_is_held_type+0xaa/0x120 [ 2240.768434] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2240.772903] ? __mutex_lock+0x97/0xce0 [ 2240.776665] ? __mutex_lock+0xaec/0xce0 [ 2240.780512] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2240.785069] ? lock_downgrade+0xfd/0x250 [ 2240.789005] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2240.793398] ceph_con_process_message+0x74/0x150 [libceph] [ 2240.798902] ? calc_signature+0xdb/0x100 [libceph] [ 2240.803724] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2240.809065] ? process_scheduled_works+0x22a/0x510 [ 2240.813867] ? process_scheduled_works+0x22a/0x510 [ 2240.818661] ceph_con_workfn+0x344/0x750 [libceph] [ 2240.823476] ? process_scheduled_works+0x22a/0x510 [ 2240.828273] process_scheduled_works+0x2a4/0x510 [ 2240.832903] worker_thread+0x188/0x340 [ 2240.836664] ? rescuer_thread+0x310/0x310 [ 2240.840687] kthread+0xf5/0x130 [ 2240.843840] ? kthread_complete_and_exit+0x20/0x20 [ 2240.848643] ret_from_fork+0x31/0x40 [ 2240.852229] ? kthread_complete_and_exit+0x20/0x20 [ 2240.857023] ret_from_fork_asm+0x11/0x20 [ 2240.860970] [ 2240.863166] irq event stamp: 10442967 [ 2240.866831] hardirqs last enabled at (10442977): [] console_unlock+0x106/0x120 [ 2240.875697] hardirqs last disabled at (10442986): [] console_unlock+0xeb/0x120 [ 2240.884477] softirqs last enabled at (10442904): [] __do_softirq+0x2be/0x371 [ 2240.893171] softirqs last disabled at (10442899): [] irq_exit_rcu+0xb0/0xd0 [ 2240.901699] ---[ end trace 0000000000000000 ]--- [ 2240.906321] ------------[ cut here ]------------ [ 2240.910943] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2240.922593] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2240.931544] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2240.987080] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2240.996035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2241.003513] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2241.008999] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2241.027745] RSP: 0018:ffffc90001fc7d50 EFLAGS: 00010286 [ 2241.032972] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2241.040104] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2241.047238] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004a44 [ 2241.054371] R10: 00000000001abfb8 R11: ffffc90001fc7be0 R12: ffff8881d800c200 [ 2241.061502] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2241.068637] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2241.076721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2241.082468] CR2: 00007fb38d5d5000 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2241.089602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2241.096734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2241.103867] Call Trace: [ 2241.106319] [ 2241.108426] ? __warn+0x85/0x180 [ 2241.111659] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.116547] ? report_bug+0xfc/0x1d0 [ 2241.120124] ? prb_read_valid+0x1b/0x20 [ 2241.123970] ? handle_bug+0x44/0x70 [ 2241.127466] ? exc_invalid_op+0x17/0x60 [ 2241.131307] ? asm_exc_invalid_op+0x1a/0x20 [ 2241.135501] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.140382] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2241.145282] ceph_readdir+0xc0/0x1670 [ceph] [ 2241.149570] ? wrap_directory_iterator+0x38/0x70 [ 2241.154197] ? wrap_directory_iterator+0x30/0x70 [ 2241.158814] ? lock_is_held_type+0xaa/0x120 [ 2241.163007] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2241.167899] wrap_directory_iterator+0x47/0x70 [ 2241.172347] iterate_dir+0x8d/0x170 [ 2241.175840] __x64_sys_getdents64+0x87/0x140 [ 2241.180118] ? __x64_sys_getdents64+0x140/0x140 [ 2241.184657] do_syscall_64+0x38/0x80 [ 2241.188239] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2241.193290] RIP: 0033:0x7f48f0ec480b [ 2241.196871] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2241.215616] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2241.223183] RAX: ffffffffffffffda RBX: 000000000207e000 RCX: 00007f48f0ec480b [ 2241.230315] RDX: 0000000000010000 RSI: 000000000207e030 RDI: 0000000000000008 [ 2241.237447] RBP: 000000000207e030 R08: 00000000000000db R09: 0000000000000003 [ 2241.244584] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2241.251713] R13: 0000000000000002 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2241.258859] [ 2241.261048] irq event stamp: 12939363 [ 2241.264713] hardirqs last enabled at (12939373): [] console_unlock+0x106/0x120 [ 2241.273579] hardirqs last disabled at (12939382): [] console_unlock+0xeb/0x120 [ 2241.282359] softirqs last enabled at (12939330): [] __do_softirq+0x2be/0x371 [ 2241.291052] softirqs last disabled at (12939323): [] irq_exit_rcu+0xb0/0xd0 [ 2241.299572] ---[ end trace 0000000000000000 ]--- [ 2241.304191] ------------[ cut here ]------------ [ 2241.308809] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2241.320454] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2241.329143] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2241.384630] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2241.393753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2241.401232] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2241.406729] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2241.412215] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2241.430964] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2241.436195] RAX: 0000000000000000 RBX: ffff8881d1182720 RCX: 0000000000000000 [ 2241.443327] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2241.450461] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004a7c [ 2241.457592] R10: 00000000001ace40 R11: ffffc9000030b828 R12: ffff8881d1182e70 [ 2241.464724] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2241.471860] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2241.479945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2241.485691] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2241.492824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2241.499957] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2241.507090] Call Trace: [ 2241.509542] [ 2241.511652] ? __warn+0x85/0x180 [ 2241.514886] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.519772] ? report_bug+0xfc/0x1d0 [ 2241.523349] ? prb_read_valid+0x1b/0x20 [ 2241.527193] ? handle_bug+0x44/0x70 [ 2241.530679] ? exc_invalid_op+0x17/0x60 [ 2241.534522] ? asm_exc_invalid_op+0x1a/0x20 [ 2241.538714] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.543594] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.548473] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2241.553367] ceph_add_cap+0x20b/0x620 [ceph] [ 2241.557665] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2241.562387] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2241.567720] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2241.572192] ceph_con_process_message+0x74/0x150 [libceph] [ 2241.577691] ? calc_signature+0xdb/0x100 [libceph] [ 2241.582501] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2241.587831] ? process_scheduled_works+0x22a/0x510 [ 2241.592629] ? process_scheduled_works+0x22a/0x510 [ 2241.597425] ceph_con_workfn+0x344/0x750 [libceph] [ 2241.602239] ? process_scheduled_works+0x22a/0x510 [ 2241.607036] process_scheduled_works+0x2a4/0x510 [ 2241.611666] worker_thread+0x188/0x340 [ 2241.615420] ? rescuer_thread+0x310/0x310 [ 2241.619427] kthread+0xf5/0x130 [ 2241.622577] ? kthread_complete_and_exit+0x20/0x20 [ 2241.627371] ret_from_fork+0x31/0x40 [ 2241.630946] ? kthread_complete_and_exit+0x20/0x20 [ 2241.635742] ret_from_fork_asm+0x11/0x20 [ 2241.639678] [ 2241.641869] irq event stamp: 10444965 [ 2241.645532] hardirqs last enabled at (10444975): [] console_unlock+0x106/0x120 [ 2241.654401] hardirqs last disabled at (10444984): [] console_unlock+0xeb/0x120 [ 2241.663179] softirqs last enabled at (10444962): [] __do_softirq+0x2be/0x371 [ 2241.671871] softirqs last disabled at (10444957): [] irq_exit_rcu+0xb0/0xd0 [ 2241.680393] ---[ end trace 0000000000000000 ]--- [ 2241.685014] ------------[ cut here ]------------ [ 2241.689634] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2241.701291] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2241.710253] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2241.765777] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2241.774732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2241.782211] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2241.787721] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2241.806478] RSP: 0018:ffffc90001fe7bf0 EFLAGS: 00010282 [ 2241.811705] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2241.818837] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2241.825971] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004ab6 [ 2241.833103] R10: 00000000001ade98 R11: ffffc90001fe7a80 R12: ffff8881c5350750 [ 2241.840239] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2241.847370] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2241.855455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2241.861200] CR2: 00007fb38d68d030 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2241.868335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2241.875468] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2241.882600] Call Trace: [ 2241.885053] [ 2241.887159] ? __warn+0x85/0x180 [ 2241.890392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.895271] ? report_bug+0xfc/0x1d0 [ 2241.898848] ? prb_read_valid+0x1b/0x20 [ 2241.902694] ? handle_bug+0x44/0x70 [ 2241.906193] ? exc_invalid_op+0x17/0x60 [ 2241.910032] ? asm_exc_invalid_op+0x1a/0x20 [ 2241.914234] ? __list_add_valid_or_report+0x7d/0xa0 [ 2241.919115] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2241.924014] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2241.928741] lookup_fast+0x6d/0xf0 [ 2241.932150] walk_component+0x26/0x160 [ 2241.935903] link_path_walk+0x24d/0x380 [ 2241.939743] ? path_openat+0x80/0x9d0 [ 2241.943410] path_openat+0x8e/0x9d0 [ 2241.946914] do_filp_open+0xb8/0x120 [ 2241.950504] ? lock_release+0xc5/0x260 [ 2241.954261] ? _raw_spin_unlock+0x23/0x40 [ 2241.958281] ? alloc_fd+0x13f/0x200 [ 2241.961784] do_sys_openat2+0x249/0x290 [ 2241.965633] do_sys_open+0x4f/0x80 [ 2241.969045] do_syscall_64+0x38/0x80 [ 2241.972635] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2241.977693] RIP: 0033:0x7f48f0eed2f2 [ 2241.981271] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2242.000017] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2242.007586] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2242.014717] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2242.021848] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2242.028981] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2242.036116] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2242.043265] [ 2242.045459] irq event stamp: 12831187 [ 2242.049125] hardirqs last enabled at (12831197): [] console_unlock+0x106/0x120 [ 2242.057990] hardirqs last disabled at (12831206): [] console_unlock+0xeb/0x120 [ 2242.066769] softirqs last enabled at (12831174): [] __do_softirq+0x2be/0x371 [ 2242.075464] softirqs last disabled at (12831225): [] irq_exit_rcu+0xb0/0xd0 [ 2242.083991] ---[ end trace 0000000000000000 ]--- [ 2242.088613] ------------[ cut here ]------------ [ 2242.093232] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2242.104877] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2242.113832] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2242.169351] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2242.178304] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2242.185782] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2242.191269] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2242.210014] RSP: 0018:ffffc90001fefc88 EFLAGS: 00010286 [ 2242.215242] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2242.222374] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2242.229505] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004af1 [ 2242.236640] R10: 00000000001aeeb0 R11: ffffc90001fefb18 R12: ffff88819f028750 [ 2242.243771] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2242.250905] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2242.258991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2242.264737] CR2: 00007fb38d645000 CR3: 0000000108c72006 CR4: 00000000003706e0 [ 2242.271869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2242.279002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2242.286136] Call Trace: [ 2242.288588] [ 2242.290695] ? __warn+0x85/0x180 [ 2242.293927] ? __list_add_valid_or_report+0x7d/0xa0 [ 2242.298809] ? report_bug+0xfc/0x1d0 [ 2242.302396] ? prb_read_valid+0x1b/0x20 [ 2242.306237] ? handle_bug+0x44/0x70 [ 2242.309728] ? exc_invalid_op+0x17/0x60 [ 2242.313568] ? asm_exc_invalid_op+0x1a/0x20 [ 2242.317762] ? __list_add_valid_or_report+0x7d/0xa0 [ 2242.322642] ? __list_add_valid_or_report+0x7d/0xa0 [ 2242.327521] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2242.332421] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2242.337148] lookup_fast+0x6d/0xf0 [ 2242.340557] path_openat+0xe4/0x9d0 [ 2242.344054] do_filp_open+0xb8/0x120 [ 2242.347643] ? lock_release+0xc5/0x260 [ 2242.351403] ? _raw_spin_unlock+0x23/0x40 [ 2242.355420] ? alloc_fd+0x13f/0x200 [ 2242.358917] do_sys_openat2+0x249/0x290 [ 2242.362763] do_sys_open+0x4f/0x80 [ 2242.366178] do_syscall_64+0x38/0x80 [ 2242.369763] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2242.374813] RIP: 0033:0x7f48f0ef2805 [ 2242.378394] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2242.397139] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2242.404706] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2242.411839] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2242.418972] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2242.426106] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2242.433240] R13: 00007ffc53d60520 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2242.440382] [ 2242.442572] irq event stamp: 12840809 [ 2242.446238] hardirqs last enabled at (12840819): [] console_unlock+0x106/0x120 [ 2242.455104] hardirqs last disabled at (12840828): [] console_unlock+0xeb/0x120 [ 2242.463883] softirqs last enabled at (12840786): [] __do_softirq+0x2be/0x371 [ 2242.472577] softirqs last disabled at (12840781): [] irq_exit_rcu+0xb0/0xd0 [ 2242.481097] ---[ end trace 0000000000000000 ]--- [ 2242.485715] ------------[ cut here ]------------ [ 2242.490337] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2242.501994] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2242.510955] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2242.566472] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2242.575425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2242.582902] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2242.588388] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2242.607135] RSP: 0018:ffffc90001fe7bf0 EFLAGS: 00010282 [ 2242.612361] RAX: 0000000000000000 RBX: ffff8881c277f560 RCX: 0000000000000000 [ 2242.619494] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2242.626627] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004b2a [ 2242.633762] R10: 00000000001b0038 R11: ffffc90001fe7a80 R12: ffff8881c277fcb0 [ 2242.640896] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2242.648026] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2242.656112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2242.661858] CR2: 00007fb38d68d030 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2242.668990] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2242.676123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2242.683255] Call Trace: [ 2242.685710] [ 2242.687816] ? __warn+0x85/0x180 [ 2242.691049] ? __list_add_valid_or_report+0x7d/0xa0 [ 2242.695929] ? report_bug+0xfc/0x1d0 [ 2242.699507] ? prb_read_valid+0x1b/0x20 [ 2242.703351] ? handle_bug+0x44/0x70 [ 2242.706841] ? exc_invalid_op+0x17/0x60 [ 2242.710680] ? asm_exc_invalid_op+0x1a/0x20 [ 2242.714873] ? __list_add_valid_or_report+0x7d/0xa0 [ 2242.719754] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2242.724655] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2242.729380] lookup_fast+0x6d/0xf0 [ 2242.732790] walk_component+0x26/0x160 [ 2242.736543] link_path_walk+0x24d/0x380 [ 2242.740379] ? path_openat+0x80/0x9d0 [ 2242.744051] path_openat+0x8e/0x9d0 [ 2242.747554] do_filp_open+0xb8/0x120 [ 2242.751143] ? lock_release+0xc5/0x260 [ 2242.754900] ? _raw_spin_unlock+0x23/0x40 [ 2242.758908] ? alloc_fd+0x13f/0x200 [ 2242.762406] do_sys_openat2+0x249/0x290 [ 2242.766246] do_sys_open+0x4f/0x80 [ 2242.769659] do_syscall_64+0x38/0x80 [ 2242.773237] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2242.778288] RIP: 0033:0x7f48f0eed2f2 [ 2242.781867] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2242.800614] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2242.808180] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2242.815314] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2242.822444] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2242.829577] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2242.836721] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2242.843867] [ 2242.846054] irq event stamp: 12833145 [ 2242.849721] hardirqs last enabled at (12833165): [] console_unlock+0x106/0x120 [ 2242.858587] hardirqs last disabled at (12833174): [] console_unlock+0xeb/0x120 [ 2242.867366] softirqs last enabled at (12833162): [] __do_softirq+0x2be/0x371 [ 2242.876065] softirqs last disabled at (12833193): [] irq_exit_rcu+0xb0/0xd0 [ 2242.884587] ---[ end trace 0000000000000000 ]--- [ 2242.889206] ------------[ cut here ]------------ [ 2242.893826] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2242.905473] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2242.914430] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2242.969953] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2242.978907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2242.986386] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2242.991874] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2243.010623] RSP: 0018:ffffc90001957bf0 EFLAGS: 00010282 [ 2243.015855] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2243.022986] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2243.030121] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004b65 [ 2243.037254] R10: 00000000001b1080 R11: ffffc90001957a80 R12: ffff888158a68750 [ 2243.044387] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2243.051519] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2243.059603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2243.065350] CR2: 00007f9bd2aa9000 CR3: 000000016696c001 CR4: 00000000003706f0 [ 2243.072483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2243.079618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2243.086748] Call Trace: [ 2243.089202] [ 2243.091310] ? __warn+0x85/0x180 [ 2243.094542] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.099421] ? report_bug+0xfc/0x1d0 [ 2243.102998] ? prb_read_valid+0x1b/0x20 [ 2243.106844] ? handle_bug+0x44/0x70 [ 2243.110337] ? exc_invalid_op+0x17/0x60 [ 2243.114181] ? asm_exc_invalid_op+0x1a/0x20 [ 2243.118375] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.123255] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2243.128155] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2243.132883] lookup_fast+0x6d/0xf0 [ 2243.136290] walk_component+0x26/0x160 [ 2243.140045] link_path_walk+0x24d/0x380 [ 2243.143883] ? path_openat+0x80/0x9d0 [ 2243.147553] path_openat+0x8e/0x9d0 [ 2243.151056] do_filp_open+0xb8/0x120 [ 2243.154645] ? lock_release+0xc5/0x260 [ 2243.158402] ? _raw_spin_unlock+0x23/0x40 [ 2243.162418] ? alloc_fd+0x13f/0x200 [ 2243.165919] do_sys_openat2+0x249/0x290 [ 2243.169765] do_sys_open+0x4f/0x80 [ 2243.173176] do_syscall_64+0x38/0x80 [ 2243.176756] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2243.181808] RIP: 0033:0x7f48f0ef2805 [ 2243.185387] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2243.204133] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2243.211700] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2243.218832] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2243.225964] RBP: 000000000207da30 R08: 000000000fffffff R09: 0000000000000000 [ 2243.233096] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2243.240231] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2243.247377] [ 2243.249564] irq event stamp: 12802643 [ 2243.253231] hardirqs last enabled at (12802653): [] console_unlock+0x106/0x120 [ 2243.262096] hardirqs last disabled at (12802662): [] console_unlock+0xeb/0x120 [ 2243.270881] softirqs last enabled at (12802678): [] __do_softirq+0x2be/0x371 [ 2243.279579] softirqs last disabled at (12802689): [] irq_exit_rcu+0xb0/0xd0 [ 2243.288097] ---[ end trace 0000000000000000 ]--- [ 2243.292719] ------------[ cut here ]------------ [ 2243.297337] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2243.308990] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2243.317688] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2243.373236] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2243.382368] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2243.389847] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2243.395358] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2243.400844] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2243.419591] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2243.424816] RAX: 0000000000000000 RBX: ffff8881d1358000 RCX: 0000000000000000 [ 2243.431950] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2243.439086] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004ba0 [ 2243.446224] R10: 00000000001b2228 R11: ffffc9000030b828 R12: ffff8881d1358750 [ 2243.453358] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2243.460490] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2243.468576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2243.474323] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2243.481456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2243.488589] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2243.495722] Call Trace: [ 2243.498173] [ 2243.500280] ? __warn+0x85/0x180 [ 2243.503515] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.508402] ? report_bug+0xfc/0x1d0 [ 2243.511981] ? prb_read_valid+0x1b/0x20 [ 2243.515824] ? handle_bug+0x44/0x70 [ 2243.519321] ? exc_invalid_op+0x17/0x60 [ 2243.523160] ? asm_exc_invalid_op+0x1a/0x20 [ 2243.527357] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.532238] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.537123] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2243.542024] ceph_add_cap+0x20b/0x620 [ceph] [ 2243.546330] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2243.551074] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2243.556416] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2243.560895] ceph_con_process_message+0x74/0x150 [libceph] [ 2243.566403] ? calc_signature+0xdb/0x100 [libceph] [ 2243.571224] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2243.576566] ? process_scheduled_works+0x22a/0x510 [ 2243.581365] ? process_scheduled_works+0x22a/0x510 [ 2243.586160] ceph_con_workfn+0x344/0x750 [libceph] [ 2243.590974] ? process_scheduled_works+0x22a/0x510 [ 2243.595771] process_scheduled_works+0x2a4/0x510 [ 2243.600403] worker_thread+0x188/0x340 [ 2243.604164] ? rescuer_thread+0x310/0x310 [ 2243.608182] kthread+0xf5/0x130 [ 2243.611329] ? kthread_complete_and_exit+0x20/0x20 [ 2243.616124] ret_from_fork+0x31/0x40 [ 2243.619709] ? kthread_complete_and_exit+0x20/0x20 [ 2243.624502] ret_from_fork_asm+0x11/0x20 [ 2243.628443] [ 2243.630643] irq event stamp: 10449435 [ 2243.634313] hardirqs last enabled at (10449445): [] console_unlock+0x106/0x120 [ 2243.643177] hardirqs last disabled at (10449454): [] console_unlock+0xeb/0x120 [ 2243.651957] softirqs last enabled at (10449430): [] __do_softirq+0x2be/0x371 [ 2243.660652] softirqs last disabled at (10449425): [] irq_exit_rcu+0xb0/0xd0 [ 2243.669169] ---[ end trace 0000000000000000 ]--- [ 2243.673789] ------------[ cut here ]------------ [ 2243.678408] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2243.690053] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2243.699009] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2243.754537] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2243.763495] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2243.770970] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2243.776456] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2243.795201] RSP: 0018:ffffc90001fefd50 EFLAGS: 00010286 [ 2243.800428] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2243.807561] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2243.814693] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004bda [ 2243.821827] R10: 00000000001b3260 R11: ffffc90001fefbe0 R12: ffff88819f02f2e8 [ 2243.828958] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2243.836091] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2243.844178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2243.849926] CR2: 00007f9bd2ae4008 CR3: 0000000108c72003 CR4: 00000000003706e0 [ 2243.857057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2243.864189] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2243.871321] Call Trace: [ 2243.873775] [ 2243.875883] ? __warn+0x85/0x180 [ 2243.879115] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.883994] ? report_bug+0xfc/0x1d0 [ 2243.887572] ? prb_read_valid+0x1b/0x20 [ 2243.891427] ? handle_bug+0x44/0x70 [ 2243.894923] ? exc_invalid_op+0x17/0x60 [ 2243.898762] ? asm_exc_invalid_op+0x1a/0x20 [ 2243.902958] ? __list_add_valid_or_report+0x7d/0xa0 [ 2243.907837] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2243.912738] ceph_readdir+0xc0/0x1670 [ceph] [ 2243.917028] ? wrap_directory_iterator+0x38/0x70 [ 2243.921653] ? wrap_directory_iterator+0x30/0x70 [ 2243.926269] ? lock_is_held_type+0xaa/0x120 [ 2243.930462] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2243.935355] wrap_directory_iterator+0x47/0x70 [ 2243.939803] iterate_dir+0x8d/0x170 [ 2243.943303] __x64_sys_getdents64+0x87/0x140 [ 2243.947572] ? __x64_sys_getdents64+0x140/0x140 [ 2243.952114] do_syscall_64+0x38/0x80 [ 2243.955696] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2243.960748] RIP: 0033:0x7f48f0ec480b [ 2243.964327] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2243.983073] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2243.990640] RAX: ffffffffffffffda RBX: 000000000207e100 RCX: 00007f48f0ec480b [ 2243.997770] RDX: 0000000000010000 RSI: 000000000207e130 RDI: 0000000000000019 [ 2244.004904] RBP: 000000000207e130 R08: 00000000000000db R09: 0000000000000003 [ 2244.012038] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2244.019171] R13: 0000000000000002 R14: 00007f48f183df18 R15: 00000000020767c0 [ 2244.026316] [ 2244.028503] irq event stamp: 12845101 [ 2244.032170] hardirqs last enabled at (12845111): [] console_unlock+0x106/0x120 [ 2244.041035] hardirqs last disabled at (12845120): [] console_unlock+0xeb/0x120 [ 2244.049815] softirqs last enabled at (12845078): [] __do_softirq+0x2be/0x371 [ 2244.058509] softirqs last disabled at (12845073): [] irq_exit_rcu+0xb0/0xd0 [ 2244.067027] ---[ end trace 0000000000000000 ]--- [ 2244.071649] ------------[ cut here ]------------ [ 2244.076272] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2244.087924] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2244.096617] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2244.152176] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2244.161307] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2244.168786] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2244.174299] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2244.179784] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2244.198532] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2244.203755] RAX: 0000000000000000 RBX: ffff8881d122b0e8 RCX: 0000000000000000 [ 2244.210889] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2244.218025] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004c12 [ 2244.225156] R10: 00000000001b42f8 R11: ffffc9000030b828 R12: ffff8881d122b838 [ 2244.232289] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2244.239422] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2244.247507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2244.253251] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2244.260386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2244.267518] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2244.274655] Call Trace: [ 2244.277107] [ 2244.279212] ? __warn+0x85/0x180 [ 2244.282448] ? __list_add_valid_or_report+0x7d/0xa0 [ 2244.287335] ? report_bug+0xfc/0x1d0 [ 2244.290919] ? prb_read_valid+0x1b/0x20 [ 2244.294764] ? handle_bug+0x44/0x70 [ 2244.298260] ? exc_invalid_op+0x17/0x60 [ 2244.302101] ? asm_exc_invalid_op+0x1a/0x20 [ 2244.306297] ? __list_add_valid_or_report+0x7d/0xa0 [ 2244.311177] ? __list_add_valid_or_report+0x7d/0xa0 [ 2244.316062] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2244.320962] ceph_add_cap+0x20b/0x620 [ceph] [ 2244.325261] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2244.330005] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2244.335350] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2244.339835] ceph_con_process_message+0x74/0x150 [libceph] [ 2244.345341] ? calc_signature+0xdb/0x100 [libceph] [ 2244.350164] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2244.355505] ? process_scheduled_works+0x22a/0x510 [ 2244.360304] ? process_scheduled_works+0x22a/0x510 [ 2244.365098] ceph_con_workfn+0x344/0x750 [libceph] [ 2244.369913] ? process_scheduled_works+0x22a/0x510 [ 2244.374711] process_scheduled_works+0x2a4/0x510 [ 2244.379342] worker_thread+0x188/0x340 [ 2244.383101] ? rescuer_thread+0x310/0x310 [ 2244.387113] kthread+0xf5/0x130 [ 2244.390260] ? kthread_complete_and_exit+0x20/0x20 [ 2244.395055] ret_from_fork+0x31/0x40 [ 2244.398641] ? kthread_complete_and_exit+0x20/0x20 [ 2244.403433] ret_from_fork_asm+0x11/0x20 [ 2244.407374] [ 2244.409571] irq event stamp: 10451331 [ 2244.413234] hardirqs last enabled at (10451341): [] console_unlock+0x106/0x120 [ 2244.422101] hardirqs last disabled at (10451350): [] console_unlock+0xeb/0x120 [ 2244.430880] softirqs last enabled at (10451328): [] __do_softirq+0x2be/0x371 [ 2244.439573] softirqs last disabled at (10451323): [] irq_exit_rcu+0xb0/0xd0 [ 2244.448092] ---[ end trace 0000000000000000 ]--- [ 2244.452711] ------------[ cut here ]------------ [ 2244.457330] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2244.468977] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2244.477929] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2244.533452] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2244.542413] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2244.549897] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2244.555386] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2244.574136] RSP: 0018:ffffc90001fcfbf0 EFLAGS: 00010282 [ 2244.579357] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2244.586491] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2244.593624] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004c4c [ 2244.600758] R10: 00000000001b5288 R11: ffffc90001fcfa80 R12: ffff8881c5350750 [ 2244.607891] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2244.615028] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2244.623116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2244.628864] CR2: 00007fb38d684000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2244.635996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2244.643130] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2244.650262] Call Trace: [ 2244.652721] [ 2244.654829] ? __warn+0x85/0x180 [ 2244.658064] ? __list_add_valid_or_report+0x7d/0xa0 [ 2244.662942] ? report_bug+0xfc/0x1d0 [ 2244.666522] ? prb_read_valid+0x1b/0x20 [ 2244.670371] ? handle_bug+0x44/0x70 [ 2244.673864] ? exc_invalid_op+0x17/0x60 [ 2244.677702] ? asm_exc_invalid_op+0x1a/0x20 [ 2244.681896] ? __list_add_valid_or_report+0x7d/0xa0 [ 2244.686777] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2244.691677] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2244.696402] lookup_fast+0x6d/0xf0 [ 2244.699811] walk_component+0x26/0x160 [ 2244.703567] link_path_walk+0x24d/0x380 [ 2244.707403] ? path_openat+0x80/0x9d0 [ 2244.711074] path_openat+0x8e/0x9d0 [ 2244.714577] do_filp_open+0xb8/0x120 [ 2244.718167] ? lock_release+0xc5/0x260 [ 2244.721925] ? _raw_spin_unlock+0x23/0x40 [ 2244.725940] ? alloc_fd+0x13f/0x200 [ 2244.729437] do_sys_openat2+0x249/0x290 [ 2244.733277] do_sys_open+0x4f/0x80 [ 2244.736682] do_syscall_64+0x38/0x80 [ 2244.740260] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2244.745311] RIP: 0033:0x7f48f0eed2f2 [ 2244.748892] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2244.767635] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2244.775202] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2244.782336] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2244.789467] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2244.796600] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2244.803735] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2244.810880] [ 2244.813068] irq event stamp: 12818177 [ 2244.816734] hardirqs last enabled at (12818187): [] console_unlock+0x106/0x120 [ 2244.825599] hardirqs last disabled at (12818196): [] console_unlock+0xeb/0x120 [ 2244.834379] softirqs last enabled at (12818154): [] __do_softirq+0x2be/0x371 [ 2244.843080] softirqs last disabled at (12818217): [] irq_exit_rcu+0xb0/0xd0 [ 2244.851604] ---[ end trace 0000000000000000 ]--- [ 2244.856233] ------------[ cut here ]------------ [ 2244.860858] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2244.872505] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2244.881210] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2244.936759] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2244.945889] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2244.953369] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2244.958879] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2244.964364] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2244.983119] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2244.988346] RAX: 0000000000000000 RBX: ffff8881d13d6b98 RCX: 0000000000000000 [ 2244.995481] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2245.002612] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004c87 [ 2245.009749] R10: 00000000001b6040 R11: ffffc9000030b828 R12: ffff8881d13d72e8 [ 2245.016881] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2245.024010] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2245.032097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2245.037843] CR2: 00007fb38f7e8000 CR3: 000000000243e004 CR4: 00000000003706e0 [ 2245.044976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2245.052108] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2245.059242] Call Trace: [ 2245.061694] [ 2245.063801] ? __warn+0x85/0x180 [ 2245.067036] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.071924] ? report_bug+0xfc/0x1d0 [ 2245.075511] ? prb_read_valid+0x1b/0x20 [ 2245.079352] ? handle_bug+0x44/0x70 [ 2245.082844] ? exc_invalid_op+0x17/0x60 [ 2245.086683] ? asm_exc_invalid_op+0x1a/0x20 [ 2245.090878] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.095768] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.100652] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2245.105562] ceph_add_cap+0x20b/0x620 [ceph] [ 2245.109870] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2245.114622] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2245.119971] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2245.124459] ceph_con_process_message+0x74/0x150 [libceph] [ 2245.129963] ? calc_signature+0xdb/0x100 [libceph] [ 2245.134787] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2245.140130] ? process_scheduled_works+0x22a/0x510 [ 2245.144928] ? process_scheduled_works+0x22a/0x510 [ 2245.149732] ceph_con_workfn+0x344/0x750 [libceph] [ 2245.154549] ? process_scheduled_works+0x22a/0x510 [ 2245.159346] process_scheduled_works+0x2a4/0x510 [ 2245.163976] worker_thread+0x188/0x340 [ 2245.167736] ? rescuer_thread+0x310/0x310 [ 2245.171755] kthread+0xf5/0x130 [ 2245.174904] ? kthread_complete_and_exit+0x20/0x20 [ 2245.179705] ret_from_fork+0x31/0x40 [ 2245.183295] ? kthread_complete_and_exit+0x20/0x20 [ 2245.188094] ret_from_fork_asm+0x11/0x20 [ 2245.192044] [ 2245.194238] irq event stamp: 10453259 [ 2245.197902] hardirqs last enabled at (10453269): [] console_unlock+0x106/0x120 [ 2245.206770] hardirqs last disabled at (10453278): [] console_unlock+0xeb/0x120 [ 2245.215557] softirqs last enabled at (10453294): [] __do_softirq+0x2be/0x371 [ 2245.224250] softirqs last disabled at (10453287): [] irq_exit_rcu+0xb0/0xd0 [ 2245.232779] ---[ end trace 0000000000000000 ]--- [ 2245.237400] ------------[ cut here ]------------ [ 2245.242028] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2245.253686] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2245.262642] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2245.318163] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2245.327114] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2245.334595] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2245.340080] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2245.358827] RSP: 0018:ffffc90001f7fbf0 EFLAGS: 00010282 [ 2245.364051] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2245.371186] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2245.378318] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004cc1 [ 2245.385455] R10: 00000000001b7010 R11: ffffc90001f7fa80 R12: ffff888158a68750 [ 2245.392583] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2245.399715] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2245.407803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2245.413548] CR2: 00007fb38d6b0000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2245.420681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2245.427813] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2245.434948] Call Trace: [ 2245.437400] [ 2245.439506] ? __warn+0x85/0x180 [ 2245.442751] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.447632] ? report_bug+0xfc/0x1d0 [ 2245.451218] ? prb_read_valid+0x1b/0x20 [ 2245.455058] ? handle_bug+0x44/0x70 [ 2245.458547] ? exc_invalid_op+0x17/0x60 [ 2245.462387] ? asm_exc_invalid_op+0x1a/0x20 [ 2245.466584] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.471464] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2245.476362] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2245.481088] lookup_fast+0x6d/0xf0 [ 2245.484497] walk_component+0x26/0x160 [ 2245.488251] link_path_walk+0x24d/0x380 [ 2245.492089] ? path_openat+0x80/0x9d0 [ 2245.495759] path_openat+0x8e/0x9d0 [ 2245.499255] do_filp_open+0xb8/0x120 [ 2245.502848] ? lock_release+0xc5/0x260 [ 2245.506601] ? _raw_spin_unlock+0x23/0x40 [ 2245.510619] ? alloc_fd+0x13f/0x200 [ 2245.514119] do_sys_openat2+0x249/0x290 [ 2245.517962] do_sys_open+0x4f/0x80 [ 2245.521367] do_syscall_64+0x38/0x80 [ 2245.524946] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2245.529995] RIP: 0033:0x7f48f0ef2805 [ 2245.533576] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2245.552323] RSP: 002b:00007ffc53d5fe80 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2245.559889] RAX: ffffffffffffffda RBX: 000000000209818f RCX: 00007f48f0ef2805 [ 2245.567021] RDX: 0000000000090800 RSI: 0000000002098170 RDI: 00000000ffffff9c [ 2245.574154] RBP: 0000000002098170 R08: 0000000000ffffff R09: 00000000020767c0 [ 2245.581289] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000040 [ 2245.588419] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2245.595566] [ 2245.597754] irq event stamp: 12930543 [ 2245.601421] hardirqs last enabled at (12930553): [] console_unlock+0x106/0x120 [ 2245.610287] hardirqs last disabled at (12930562): [] console_unlock+0xeb/0x120 [ 2245.619064] softirqs last enabled at (12930510): [] __do_softirq+0x2be/0x371 [ 2245.627759] softirqs last disabled at (12930505): [] irq_exit_rcu+0xb0/0xd0 [ 2245.636278] ---[ end trace 0000000000000000 ]--- [ 2245.640900] ------------[ cut here ]------------ [ 2245.645515] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2245.657163] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2245.666117] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2245.721631] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2245.730581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2245.738061] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2245.743545] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2245.762292] RSP: 0018:ffffc90001fe7bf0 EFLAGS: 00010282 [ 2245.767518] RAX: 0000000000000000 RBX: ffff8881c277a720 RCX: 0000000000000000 [ 2245.774651] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2245.781785] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004cfc [ 2245.788923] R10: 00000000001b80e0 R11: ffffc90001fe7a80 R12: ffff8881c277ae70 [ 2245.796073] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2245.803202] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2245.811288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2245.817034] CR2: 00007fb38d68d030 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2245.824172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2245.831307] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2245.838440] Call Trace: [ 2245.840893] [ 2245.842998] ? __warn+0x85/0x180 [ 2245.846232] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.851111] ? report_bug+0xfc/0x1d0 [ 2245.854693] ? prb_read_valid+0x1b/0x20 [ 2245.858534] ? handle_bug+0x44/0x70 [ 2245.862022] ? exc_invalid_op+0x17/0x60 [ 2245.865863] ? asm_exc_invalid_op+0x1a/0x20 [ 2245.870058] ? __list_add_valid_or_report+0x7d/0xa0 [ 2245.874937] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2245.879837] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2245.884564] lookup_fast+0x6d/0xf0 [ 2245.887973] walk_component+0x26/0x160 [ 2245.891728] link_path_walk+0x24d/0x380 [ 2245.895565] ? path_openat+0x80/0x9d0 [ 2245.899235] path_openat+0x8e/0x9d0 [ 2245.902739] do_filp_open+0xb8/0x120 [ 2245.906327] ? lock_release+0xc5/0x260 [ 2245.910084] ? _raw_spin_unlock+0x23/0x40 [ 2245.914101] ? alloc_fd+0x13f/0x200 [ 2245.917599] do_sys_openat2+0x249/0x290 [ 2245.921441] do_sys_open+0x4f/0x80 [ 2245.924850] do_syscall_64+0x38/0x80 [ 2245.928429] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2245.933480] RIP: 0033:0x7f48f0eed2f2 [ 2245.937061] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2245.955813] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2245.963382] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2245.970513] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2245.977647] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2245.984780] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2245.991915] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2245.999059] [ 2246.001247] irq event stamp: 12845297 [ 2246.004913] hardirqs last enabled at (12845307): [] console_unlock+0x106/0x120 [ 2246.013778] hardirqs last disabled at (12845316): [] console_unlock+0xeb/0x120 [ 2246.022558] softirqs last enabled at (12845274): [] __do_softirq+0x2be/0x371 [ 2246.031252] softirqs last disabled at (12845335): [] irq_exit_rcu+0xb0/0xd0 [ 2246.039770] ---[ end trace 0000000000000000 ]--- [ 2246.044390] ------------[ cut here ]------------ [ 2246.049009] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2246.060655] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2246.069614] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2246.125131] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2246.134086] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2246.141561] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2246.147048] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2246.165794] RSP: 0018:ffffc90001957bf0 EFLAGS: 00010282 [ 2246.171022] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2246.178154] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2246.185286] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004d37 [ 2246.192418] R10: 00000000001b91e8 R11: ffffc90001957a80 R12: ffff8881c5350750 [ 2246.199555] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2246.206685] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2246.214770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2246.220518] CR2: 00007f9bd2aa9000 CR3: 000000016696c001 CR4: 00000000003706f0 [ 2246.227653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2246.234782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2246.241914] Call Trace: [ 2246.244368] [ 2246.246476] ? __warn+0x85/0x180 [ 2246.249709] ? __list_add_valid_or_report+0x7d/0xa0 [ 2246.254588] ? report_bug+0xfc/0x1d0 [ 2246.258166] ? prb_read_valid+0x1b/0x20 [ 2246.262010] ? handle_bug+0x44/0x70 [ 2246.265499] ? exc_invalid_op+0x17/0x60 [ 2246.269339] ? asm_exc_invalid_op+0x1a/0x20 [ 2246.273533] ? __list_add_valid_or_report+0x7d/0xa0 [ 2246.278413] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2246.283312] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2246.288040] lookup_fast+0x6d/0xf0 [ 2246.291448] walk_component+0x26/0x160 [ 2246.295202] link_path_walk+0x24d/0x380 [ 2246.299038] ? path_openat+0x80/0x9d0 [ 2246.302711] path_openat+0x8e/0x9d0 [ 2246.306215] do_filp_open+0xb8/0x120 [ 2246.309802] ? lock_release+0xc5/0x260 [ 2246.313561] ? _raw_spin_unlock+0x23/0x40 [ 2246.317577] ? alloc_fd+0x13f/0x200 [ 2246.321074] do_sys_openat2+0x249/0x290 [ 2246.324914] do_sys_open+0x4f/0x80 [ 2246.328326] do_syscall_64+0x38/0x80 [ 2246.331904] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2246.336956] RIP: 0033:0x7f48f0ef2805 [ 2246.340536] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2246.359282] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2246.366847] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2246.373980] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2246.381114] RBP: 000000000207da30 R08: 000000000fffffff R09: 0000000000000000 [ 2246.388247] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2246.395379] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2246.402525] [ 2246.404714] irq event stamp: 12809737 [ 2246.408378] hardirqs last enabled at (12809747): [] console_unlock+0x106/0x120 [ 2246.417246] hardirqs last disabled at (12809756): [] console_unlock+0xeb/0x120 [ 2246.426024] softirqs last enabled at (12809704): [] __do_softirq+0x2be/0x371 [ 2246.434719] softirqs last disabled at (12809699): [] irq_exit_rcu+0xb0/0xd0 [ 2246.443236] ---[ end trace 0000000000000000 ]--- [ 2246.447856] ------------[ cut here ]------------ [ 2246.452489] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2246.464149] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2246.473103] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2246.528628] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2246.537584] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2246.545065] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2246.550549] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2246.569295] RSP: 0018:ffffc90001fcfbf0 EFLAGS: 00010282 [ 2246.574522] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2246.581655] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2246.588788] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004d72 [ 2246.595923] R10: 00000000001ba1d8 R11: ffffc90001fcfa80 R12: ffff8881d106b838 [ 2246.603061] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2246.610198] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2246.618287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2246.624036] CR2: 00007fb38d684000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2246.631177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2246.638313] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2246.645444] Call Trace: [ 2246.647896] [ 2246.650004] ? __warn+0x85/0x180 [ 2246.653235] ? __list_add_valid_or_report+0x7d/0xa0 [ 2246.658115] ? report_bug+0xfc/0x1d0 [ 2246.661697] ? prb_read_valid+0x1b/0x20 [ 2246.665537] ? handle_bug+0x44/0x70 [ 2246.669026] ? exc_invalid_op+0x17/0x60 [ 2246.672865] ? asm_exc_invalid_op+0x1a/0x20 [ 2246.677065] ? __list_add_valid_or_report+0x7d/0xa0 [ 2246.681949] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2246.686848] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2246.691575] lookup_fast+0x6d/0xf0 [ 2246.694985] walk_component+0x26/0x160 [ 2246.698738] link_path_walk+0x24d/0x380 [ 2246.702576] ? path_openat+0x80/0x9d0 [ 2246.706246] path_openat+0x8e/0x9d0 [ 2246.709742] do_filp_open+0xb8/0x120 [ 2246.713330] ? lock_release+0xc5/0x260 [ 2246.717092] ? _raw_spin_unlock+0x23/0x40 [ 2246.721105] ? alloc_fd+0x13f/0x200 [ 2246.724603] do_sys_openat2+0x249/0x290 [ 2246.728451] do_sys_open+0x4f/0x80 [ 2246.731862] do_syscall_64+0x38/0x80 [ 2246.735441] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2246.740493] RIP: 0033:0x7f48f0eed2f2 [ 2246.744070] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2246.762821] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2246.770397] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2246.777525] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2246.784658] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2246.791794] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2246.798923] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2246.806068] [ 2246.808257] irq event stamp: 12823223 [ 2246.811925] hardirqs last enabled at (12823233): [] console_unlock+0x106/0x120 [ 2246.820790] hardirqs last disabled at (12823242): [] console_unlock+0xeb/0x120 [ 2246.829570] softirqs last enabled at (12823190): [] __do_softirq+0x2be/0x371 [ 2246.838262] softirqs last disabled at (12823185): [] irq_exit_rcu+0xb0/0xd0 [ 2246.846787] ---[ end trace 0000000000000000 ]--- [ 2246.851414] ------------[ cut here ]------------ [ 2246.856032] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2246.867684] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2246.876639] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2246.932160] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2246.941112] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2246.948591] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2246.954077] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2246.972822] RSP: 0018:ffffc90001957bf0 EFLAGS: 00010282 [ 2246.978050] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2246.985183] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2246.992323] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004dad [ 2246.999459] R10: 00000000001bb3d8 R11: ffffc90001957a80 R12: ffff8881d12224a8 [ 2247.006589] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2247.013723] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2247.021807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2247.027554] CR2: 00007f9bd2b15000 CR3: 000000016696c001 CR4: 00000000003706f0 [ 2247.034687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2247.041819] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2247.048953] Call Trace: [ 2247.051405] [ 2247.053513] ? __warn+0x85/0x180 [ 2247.056744] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.061626] ? report_bug+0xfc/0x1d0 [ 2247.065205] ? prb_read_valid+0x1b/0x20 [ 2247.069046] ? handle_bug+0x44/0x70 [ 2247.072535] ? exc_invalid_op+0x17/0x60 [ 2247.076376] ? asm_exc_invalid_op+0x1a/0x20 [ 2247.080570] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.085450] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2247.090349] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2247.095078] lookup_fast+0x6d/0xf0 [ 2247.098486] walk_component+0x26/0x160 [ 2247.102240] link_path_walk+0x24d/0x380 [ 2247.106077] ? path_openat+0x80/0x9d0 [ 2247.109747] path_openat+0x8e/0x9d0 [ 2247.113244] do_filp_open+0xb8/0x120 [ 2247.116831] ? lock_release+0xc5/0x260 [ 2247.120597] ? _raw_spin_unlock+0x23/0x40 [ 2247.124605] ? alloc_fd+0x13f/0x200 [ 2247.128104] do_sys_openat2+0x249/0x290 [ 2247.131950] do_sys_open+0x4f/0x80 [ 2247.135354] do_syscall_64+0x38/0x80 [ 2247.138933] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2247.143984] RIP: 0033:0x7f48f0ef2805 [ 2247.147563] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2247.166311] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2247.173877] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2247.181009] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2247.188142] RBP: 000000000207da30 R08: 000000000fffffff R09: 0000000000000000 [ 2247.195277] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2247.202406] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2247.209555] [ 2247.211743] irq event stamp: 12811651 [ 2247.215408] hardirqs last enabled at (12811661): [] console_unlock+0x106/0x120 [ 2247.224275] hardirqs last disabled at (12811670): [] console_unlock+0xeb/0x120 [ 2247.233053] softirqs last enabled at (12811618): [] __do_softirq+0x2be/0x371 [ 2247.241746] softirqs last disabled at (12811613): [] irq_exit_rcu+0xb0/0xd0 [ 2247.250267] ---[ end trace 0000000000000000 ]--- [ 2247.254887] ------------[ cut here ]------------ [ 2247.259504] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2247.271151] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2247.280106] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2247.335631] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2247.344587] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2247.352066] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2247.357552] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2247.376303] RSP: 0018:ffffc90001fe7c88 EFLAGS: 00010286 [ 2247.381533] RAX: 0000000000000000 RBX: ffff8881d0bf4e40 RCX: 0000000000000000 [ 2247.388667] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2247.395801] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004de8 [ 2247.402931] R10: 00000000001bc420 R11: ffffc90001fe7b18 R12: ffff8881d0bf5590 [ 2247.410069] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2247.417198] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2247.425283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2247.431030] CR2: 00007fb38d68d030 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2247.438163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2247.445300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2247.452428] Call Trace: [ 2247.454881] [ 2247.456989] ? __warn+0x85/0x180 [ 2247.460221] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.465102] ? report_bug+0xfc/0x1d0 [ 2247.468678] ? prb_read_valid+0x1b/0x20 [ 2247.472521] ? handle_bug+0x44/0x70 [ 2247.476014] ? exc_invalid_op+0x17/0x60 [ 2247.479850] ? asm_exc_invalid_op+0x1a/0x20 [ 2247.484045] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.488928] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.493805] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2247.498708] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2247.503430] lookup_fast+0x6d/0xf0 [ 2247.506840] path_openat+0xe4/0x9d0 [ 2247.510341] do_filp_open+0xb8/0x120 [ 2247.513929] ? lock_release+0xc5/0x260 [ 2247.517685] ? _raw_spin_unlock+0x23/0x40 [ 2247.521697] ? alloc_fd+0x13f/0x200 [ 2247.525193] do_sys_openat2+0x249/0x290 [ 2247.529040] do_sys_open+0x4f/0x80 [ 2247.532452] do_syscall_64+0x38/0x80 [ 2247.536028] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2247.541083] RIP: 0033:0x7f48f0eed2f2 [ 2247.544660] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2247.563407] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2247.570973] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2247.578107] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2247.585238] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2247.592373] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2247.599504] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2247.606661] [ 2247.608856] irq event stamp: 12849085 [ 2247.612522] hardirqs last enabled at (12849095): [] console_unlock+0x106/0x120 [ 2247.621388] hardirqs last disabled at (12849104): [] console_unlock+0xeb/0x120 [ 2247.630168] softirqs last enabled at (12849072): [] __do_softirq+0x2be/0x371 [ 2247.638863] softirqs last disabled at (12849123): [] irq_exit_rcu+0xb0/0xd0 [ 2247.647381] ---[ end trace 0000000000000000 ]--- [ 2247.651999] ------------[ cut here ]------------ [ 2247.656618] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2247.668265] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2247.677224] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2247.732743] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2247.741693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2247.749172] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2247.754657] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2247.773403] RSP: 0018:ffffc90001f7fbf0 EFLAGS: 00010282 [ 2247.778629] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2247.785764] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2247.792895] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004e21 [ 2247.800030] R10: 00000000001bd560 R11: ffffc90001f7fa80 R12: ffff8881c5350750 [ 2247.807162] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2247.814295] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2247.822380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2247.828126] CR2: 00007fb38d6b0000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2247.835258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2247.842397] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2247.849524] Call Trace: [ 2247.851977] [ 2247.854085] ? __warn+0x85/0x180 [ 2247.857317] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.862197] ? report_bug+0xfc/0x1d0 [ 2247.865784] ? prb_read_valid+0x1b/0x20 [ 2247.869627] ? handle_bug+0x44/0x70 [ 2247.873117] ? exc_invalid_op+0x17/0x60 [ 2247.876955] ? asm_exc_invalid_op+0x1a/0x20 [ 2247.881153] ? __list_add_valid_or_report+0x7d/0xa0 [ 2247.886033] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2247.890929] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2247.895658] lookup_fast+0x6d/0xf0 [ 2247.899069] walk_component+0x26/0x160 [ 2247.902822] link_path_walk+0x24d/0x380 [ 2247.906667] ? path_openat+0x80/0x9d0 [ 2247.910338] path_openat+0x8e/0x9d0 [ 2247.913841] do_filp_open+0xb8/0x120 [ 2247.917430] ? lock_release+0xc5/0x260 [ 2247.921187] ? _raw_spin_unlock+0x23/0x40 [ 2247.925203] ? alloc_fd+0x13f/0x200 [ 2247.928702] do_sys_openat2+0x249/0x290 [ 2247.932557] do_sys_open+0x4f/0x80 [ 2247.935960] do_syscall_64+0x38/0x80 [ 2247.939540] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2247.944591] RIP: 0033:0x7f48f0ef2805 [ 2247.948170] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2247.966918] RSP: 002b:00007ffc53d5fe80 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2247.974490] RAX: ffffffffffffffda RBX: 000000000209818f RCX: 00007f48f0ef2805 [ 2247.981626] RDX: 0000000000090800 RSI: 0000000002098170 RDI: 00000000ffffff9c [ 2247.988757] RBP: 0000000002098170 R08: 0000000000ffffff R09: 00000000020767c0 [ 2247.995890] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000040 [ 2248.003022] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2248.010169] [ 2248.012357] irq event stamp: 12936067 [ 2248.016022] hardirqs last enabled at (12936077): [] console_unlock+0x106/0x120 [ 2248.024888] hardirqs last disabled at (12936086): [] console_unlock+0xeb/0x120 [ 2248.033670] softirqs last enabled at (12936100): [] __do_softirq+0x2be/0x371 [ 2248.042363] softirqs last disabled at (12936095): [] irq_exit_rcu+0xb0/0xd0 [ 2248.050881] ---[ end trace 0000000000000000 ]--- [ 2248.055500] ------------[ cut here ]------------ [ 2248.060123] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2248.071766] WARNING: CPU: 6 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2248.080724] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2248.136253] CPU: 6 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2248.145205] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2248.152681] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2248.158167] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2248.176915] RSP: 0018:ffffc90001fffb28 EFLAGS: 00010286 [ 2248.182140] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2248.189273] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2248.196406] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004e5c [ 2248.203538] R10: 00000000001be708 R11: ffffc90001fff9b8 R12: ffff8881d12c5590 [ 2248.210680] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2248.217813] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2248.225900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.231646] CR2: 00007fb38d68d030 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2248.238777] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2248.245914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2248.253044] Call Trace: [ 2248.255496] [ 2248.257603] ? __warn+0x85/0x180 [ 2248.260838] ? __list_add_valid_or_report+0x7d/0xa0 [ 2248.265716] ? report_bug+0xfc/0x1d0 [ 2248.269302] ? prb_read_valid+0x1b/0x20 [ 2248.273145] ? handle_bug+0x44/0x70 [ 2248.276636] ? exc_invalid_op+0x17/0x60 [ 2248.280476] ? asm_exc_invalid_op+0x1a/0x20 [ 2248.284670] ? __list_add_valid_or_report+0x7d/0xa0 [ 2248.289551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2248.294427] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2248.299329] ceph_check_caps+0x999/0xae0 [ceph] [ 2248.303879] ? lock_release+0xc5/0x260 [ 2248.307636] ? __lock_acquire+0xbfa/0x1e20 [ 2248.311744] ? __lock_acquire+0xbfa/0x1e20 [ 2248.315849] ? lock_acquire+0xcb/0x2b0 [ 2248.319605] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2248.324588] ? find_held_lock+0x2d/0x90 [ 2248.328428] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2248.333501] ? lock_release+0xc5/0x260 [ 2248.337258] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2248.342154] ceph_read_iter+0x26e/0x860 [ceph] [ 2248.346620] ? selinux_file_permission+0x119/0x150 [ 2248.351418] ? vfs_read+0x20f/0x300 [ 2248.354915] vfs_read+0x20f/0x300 [ 2248.358246] ksys_pread64+0x65/0xa0 [ 2248.361745] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2248.366885] do_syscall_64+0x38/0x80 [ 2248.370463] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2248.375522] RIP: 0033:0x7f48f0eeb88b [ 2248.379104] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2248.397850] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2248.405415] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2248.412551] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2248.419682] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2248.426813] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2248.433947] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2248.441093] [ 2248.443286] irq event stamp: 12903909 [ 2248.446946] hardirqs last enabled at (12903919): [] console_unlock+0x106/0x120 [ 2248.455814] hardirqs last disabled at (12903928): [] console_unlock+0xeb/0x120 [ 2248.464593] softirqs last enabled at (12903906): [] __do_softirq+0x2be/0x371 [ 2248.473285] softirqs last disabled at (12903899): [] irq_exit_rcu+0xb0/0xd0 [ 2248.481807] ---[ end trace 0000000000000000 ]--- [ 2248.486425] ------------[ cut here ]------------ [ 2248.491049] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2248.502702] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2248.511656] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2248.567178] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2248.576138] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2248.583613] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2248.589100] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2248.607847] RSP: 0018:ffffc90001f7fbf0 EFLAGS: 00010282 [ 2248.613071] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2248.620206] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2248.627337] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004e9c [ 2248.634469] R10: 00000000001bf860 R11: ffffc90001f7fa80 R12: ffff8881c5351118 [ 2248.641603] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2248.648737] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2248.656821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.662567] CR2: 00007fb38d6b0000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2248.669700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2248.676846] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2248.683975] Call Trace: [ 2248.686429] [ 2248.688536] ? __warn+0x85/0x180 [ 2248.691769] ? __list_add_valid_or_report+0x7d/0xa0 [ 2248.696648] ? report_bug+0xfc/0x1d0 [ 2248.700226] ? prb_read_valid+0x1b/0x20 [ 2248.704068] ? handle_bug+0x44/0x70 [ 2248.707558] ? exc_invalid_op+0x17/0x60 [ 2248.711402] ? asm_exc_invalid_op+0x1a/0x20 [ 2248.715603] ? __list_add_valid_or_report+0x7d/0xa0 [ 2248.720484] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2248.725381] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2248.730108] lookup_fast+0x6d/0xf0 [ 2248.733517] walk_component+0x26/0x160 [ 2248.737271] link_path_walk+0x24d/0x380 [ 2248.741107] ? path_openat+0x80/0x9d0 [ 2248.744779] path_openat+0x8e/0x9d0 [ 2248.748283] do_filp_open+0xb8/0x120 [ 2248.751870] ? lock_release+0xc5/0x260 [ 2248.755630] ? _raw_spin_unlock+0x23/0x40 [ 2248.759646] ? alloc_fd+0x13f/0x200 [ 2248.763143] do_sys_openat2+0x249/0x290 [ 2248.766983] do_sys_open+0x4f/0x80 [ 2248.770395] do_syscall_64+0x38/0x80 [ 2248.773976] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2248.779026] RIP: 0033:0x7f48f0ef2805 [ 2248.782605] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2248.801349] RSP: 002b:00007ffc53d5fe80 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2248.808917] RAX: ffffffffffffffda RBX: 000000000209818f RCX: 00007f48f0ef2805 [ 2248.816048] RDX: 0000000000090800 RSI: 0000000002098170 RDI: 00000000ffffff9c [ 2248.823182] RBP: 0000000002098170 R08: 0000000000ffffff R09: 00000000020767c0 [ 2248.830314] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000040 [ 2248.837451] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2248.844594] [ 2248.846782] irq event stamp: 12938081 [ 2248.850448] hardirqs last enabled at (12938091): [] console_unlock+0x106/0x120 [ 2248.859314] hardirqs last disabled at (12938100): [] console_unlock+0xeb/0x120 [ 2248.868094] softirqs last enabled at (12938058): [] __do_softirq+0x2be/0x371 [ 2248.876786] softirqs last disabled at (12938053): [] irq_exit_rcu+0xb0/0xd0 [ 2248.885304] ---[ end trace 0000000000000000 ]--- [ 2248.889926] ------------[ cut here ]------------ [ 2248.894543] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2248.906191] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2248.915147] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2248.970668] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2248.979617] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2248.987098] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2248.992582] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2249.011333] RSP: 0018:ffffc90001fe7c88 EFLAGS: 00010286 [ 2249.016557] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2249.023689] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2249.030820] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004ed7 [ 2249.037956] R10: 00000000001c0900 R11: ffffc90001fe7b18 R12: ffff8881d12c72e8 [ 2249.045087] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2249.052221] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2249.060307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2249.066052] CR2: 00007fb38d68d030 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2249.073184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2249.080317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2249.087452] Call Trace: [ 2249.089904] [ 2249.092010] ? __warn+0x85/0x180 [ 2249.095243] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.100124] ? report_bug+0xfc/0x1d0 [ 2249.103703] ? prb_read_valid+0x1b/0x20 [ 2249.107544] ? handle_bug+0x44/0x70 [ 2249.111034] ? exc_invalid_op+0x17/0x60 [ 2249.114873] ? asm_exc_invalid_op+0x1a/0x20 [ 2249.119070] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.123948] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.128827] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2249.133726] ceph_open+0x16b/0x440 [ceph] [ 2249.137759] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2249.142911] do_dentry_open+0x172/0x530 [ 2249.146753] path_openat+0x743/0x9d0 [ 2249.150337] do_filp_open+0xb8/0x120 [ 2249.153924] ? lock_release+0xc5/0x260 [ 2249.157683] ? _raw_spin_unlock+0x23/0x40 [ 2249.161700] ? alloc_fd+0x13f/0x200 [ 2249.165197] do_sys_openat2+0x249/0x290 [ 2249.169037] do_sys_open+0x4f/0x80 [ 2249.172450] do_syscall_64+0x38/0x80 [ 2249.176028] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2249.181079] RIP: 0033:0x7f48f0eed2f2 [ 2249.184657] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2249.203405] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2249.210970] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2249.218102] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2249.225235] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2249.232373] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2249.239500] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2249.246647] [ 2249.248836] irq event stamp: 12851025 [ 2249.252503] hardirqs last enabled at (12851035): [] console_unlock+0x106/0x120 [ 2249.261367] hardirqs last disabled at (12851044): [] console_unlock+0xeb/0x120 [ 2249.270151] softirqs last enabled at (12851060): [] __do_softirq+0x2be/0x371 [ 2249.278849] softirqs last disabled at (12851053): [] irq_exit_rcu+0xb0/0xd0 [ 2249.287367] ---[ end trace 0000000000000000 ]--- [ 2249.291988] ------------[ cut here ]------------ [ 2249.296611] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2249.308270] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2249.317235] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2249.372761] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2249.381716] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2249.389196] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2249.394682] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2249.413427] RSP: 0018:ffffc90001f7fc88 EFLAGS: 00010286 [ 2249.418653] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2249.425787] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2249.432919] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004f11 [ 2249.440059] R10: 00000000001c1868 R11: ffffc90001f7fb18 R12: ffff8881c5351ae0 [ 2249.447194] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2249.454325] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2249.462413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2249.468159] CR2: 00007fb38d6b0000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2249.475295] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2249.482424] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2249.489558] Call Trace: [ 2249.492010] [ 2249.494117] ? __warn+0x85/0x180 [ 2249.497351] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.502236] ? report_bug+0xfc/0x1d0 [ 2249.505816] ? prb_read_valid+0x1b/0x20 [ 2249.509659] ? handle_bug+0x44/0x70 [ 2249.513148] ? exc_invalid_op+0x17/0x60 [ 2249.516989] ? asm_exc_invalid_op+0x1a/0x20 [ 2249.521183] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.526066] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.530942] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2249.535842] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2249.540572] lookup_fast+0x6d/0xf0 [ 2249.543977] path_openat+0xe4/0x9d0 [ 2249.547478] do_filp_open+0xb8/0x120 [ 2249.551066] ? lock_release+0xc5/0x260 [ 2249.554824] ? _raw_spin_unlock+0x23/0x40 [ 2249.558840] ? alloc_fd+0x13f/0x200 [ 2249.562339] do_sys_openat2+0x249/0x290 [ 2249.566183] do_sys_open+0x4f/0x80 [ 2249.569589] do_syscall_64+0x38/0x80 [ 2249.573173] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2249.578228] RIP: 0033:0x7f48f0ef2805 [ 2249.581805] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2249.600553] RSP: 002b:00007ffc53d5fe80 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2249.608121] RAX: ffffffffffffffda RBX: 000000000209818f RCX: 00007f48f0ef2805 [ 2249.615251] RDX: 0000000000090800 RSI: 0000000002098170 RDI: 00000000ffffff9c [ 2249.622384] RBP: 0000000002098170 R08: 0000000000ffffff R09: 00000000020767c0 [ 2249.629521] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000040 [ 2249.636650] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2249.643795] [ 2249.645983] irq event stamp: 12939995 [ 2249.649651] hardirqs last enabled at (12940005): [] console_unlock+0x106/0x120 [ 2249.658516] hardirqs last disabled at (12940014): [] console_unlock+0xeb/0x120 [ 2249.667296] softirqs last enabled at (12939962): [] __do_softirq+0x2be/0x371 [ 2249.675992] softirqs last disabled at (12940033): [] irq_exit_rcu+0xb0/0xd0 [ 2249.684509] ---[ end trace 0000000000000000 ]--- [ 2249.689129] ------------[ cut here ]------------ [ 2249.693748] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2249.705394] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2249.714349] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2249.769872] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2249.778821] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2249.786299] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2249.791785] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2249.810540] RSP: 0018:ffffc90001fe7aa0 EFLAGS: 00010286 [ 2249.815767] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2249.822901] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2249.830032] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004f4a [ 2249.837167] R10: 00000000001c2648 R11: ffffc90001fe7930 R12: ffff8881d12c72e8 [ 2249.844299] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2249.851431] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2249.859517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2249.865263] CR2: 00007f9bd2b37020 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2249.872397] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2249.879533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2249.886661] Call Trace: [ 2249.889114] [ 2249.891222] ? __warn+0x85/0x180 [ 2249.894454] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.899333] ? report_bug+0xfc/0x1d0 [ 2249.902911] ? prb_read_valid+0x1b/0x20 [ 2249.906757] ? handle_bug+0x44/0x70 [ 2249.910250] ? exc_invalid_op+0x17/0x60 [ 2249.914084] ? asm_exc_invalid_op+0x1a/0x20 [ 2249.918279] ? __list_add_valid_or_report+0x7d/0xa0 [ 2249.923161] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2249.928059] ceph_check_caps+0x999/0xae0 [ceph] [ 2249.932638] ? lock_release+0xc5/0x260 [ 2249.936399] ceph_open+0x3cc/0x440 [ceph] [ 2249.940429] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2249.945584] do_dentry_open+0x172/0x530 [ 2249.949422] path_openat+0x743/0x9d0 [ 2249.953005] do_filp_open+0xb8/0x120 [ 2249.956595] ? lock_release+0xc5/0x260 [ 2249.960353] ? _raw_spin_unlock+0x23/0x40 [ 2249.964370] ? alloc_fd+0x13f/0x200 [ 2249.967866] do_sys_openat2+0x249/0x290 [ 2249.971714] do_sys_open+0x4f/0x80 [ 2249.975118] do_syscall_64+0x38/0x80 [ 2249.978696] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2249.983747] RIP: 0033:0x7f48f0eed2f2 [ 2249.987328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2250.006072] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2250.013642] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2250.020772] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2250.027906] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2250.035038] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2250.042172] R13: 00007f48f183dcf0 R14: 00000000020767c0 R15: 0000000000002a0f [ 2250.049323] [ 2250.051514] irq event stamp: 12852937 [ 2250.055180] hardirqs last enabled at (12852947): [] console_unlock+0x106/0x120 [ 2250.064047] hardirqs last disabled at (12852956): [] console_unlock+0xeb/0x120 [ 2250.072826] softirqs last enabled at (12852924): [] __do_softirq+0x2be/0x371 [ 2250.081521] softirqs last disabled at (12852975): [] irq_exit_rcu+0xb0/0xd0 [ 2250.090038] ---[ end trace 0000000000000000 ]--- [ 2250.094656] ------------[ cut here ]------------ [ 2250.099278] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2250.110924] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2250.119881] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2250.175411] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2250.184361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2250.191837] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2250.197324] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2250.216070] RSP: 0018:ffffc90001f7fd50 EFLAGS: 00010286 [ 2250.221297] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2250.228431] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2250.235563] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004f85 [ 2250.242694] R10: 00000000001c3630 R11: ffffc90001f7fbe0 R12: ffff8881c5357cb0 [ 2250.249836] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2250.256969] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2250.265055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2250.270801] CR2: 00007fb38d6b0000 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2250.277935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2250.285072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2250.292201] Call Trace: [ 2250.294653] [ 2250.296760] ? __warn+0x85/0x180 [ 2250.299993] ? __list_add_valid_or_report+0x7d/0xa0 [ 2250.304878] ? report_bug+0xfc/0x1d0 [ 2250.308460] ? prb_read_valid+0x1b/0x20 [ 2250.312304] ? handle_bug+0x44/0x70 [ 2250.315795] ? exc_invalid_op+0x17/0x60 [ 2250.319632] ? asm_exc_invalid_op+0x1a/0x20 [ 2250.323828] ? __list_add_valid_or_report+0x7d/0xa0 [ 2250.328708] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2250.333605] ceph_readdir+0xc0/0x1670 [ceph] [ 2250.337897] ? wrap_directory_iterator+0x38/0x70 [ 2250.342523] ? wrap_directory_iterator+0x30/0x70 [ 2250.347140] ? lock_is_held_type+0xaa/0x120 [ 2250.351332] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2250.356224] wrap_directory_iterator+0x47/0x70 [ 2250.360672] iterate_dir+0x8d/0x170 [ 2250.364164] __x64_sys_getdents64+0x87/0x140 [ 2250.368433] ? __x64_sys_getdents64+0x140/0x140 [ 2250.372976] do_syscall_64+0x38/0x80 [ 2250.376555] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2250.381612] RIP: 0033:0x7f48f0ec480b [ 2250.385189] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2250.403933] RSP: 002b:00007ffc53d5feb8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2250.411500] RAX: ffffffffffffffda RBX: 0000000002088130 RCX: 00007f48f0ec480b [ 2250.418632] RDX: 0000000000010000 RSI: 0000000002088160 RDI: 0000000000000006 [ 2250.425764] RBP: 0000000002088160 R08: 0000000002098160 R09: 0000000002088120 [ 2250.432903] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2250.440039] R13: 0000000000000002 R14: 00000000020767c0 R15: 0000000000002a1a [ 2250.447190] [ 2250.449382] irq event stamp: 12941947 [ 2250.453047] hardirqs last enabled at (12941957): [] console_unlock+0x106/0x120 [ 2250.461913] hardirqs last disabled at (12941966): [] console_unlock+0xeb/0x120 [ 2250.470694] softirqs last enabled at (12941934): [] __do_softirq+0x2be/0x371 [ 2250.479386] softirqs last disabled at (12941929): [] irq_exit_rcu+0xb0/0xd0 [ 2250.487907] ---[ end trace 0000000000000000 ]--- [ 2250.492527] ------------[ cut here ]------------ [ 2250.497148] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2250.508803] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2250.517755] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2250.573271] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2250.582229] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2250.589706] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2250.595195] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2250.613944] RSP: 0018:ffffc90001fe7bd0 EFLAGS: 00010282 [ 2250.619177] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2250.626305] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2250.633442] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004fbd [ 2250.640575] R10: 00000000001c45c8 R11: ffffc90001fe7a60 R12: ffff8881d12c72e8 [ 2250.647707] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2250.654837] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2250.662925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2250.668669] CR2: 00007f9bd2b37020 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2250.675804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2250.682936] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2250.690070] Call Trace: [ 2250.692523] [ 2250.694631] ? __warn+0x85/0x180 [ 2250.697870] ? __list_add_valid_or_report+0x7d/0xa0 [ 2250.702751] ? report_bug+0xfc/0x1d0 [ 2250.706331] ? prb_read_valid+0x1b/0x20 [ 2250.710173] ? handle_bug+0x44/0x70 [ 2250.713672] ? exc_invalid_op+0x17/0x60 [ 2250.717514] ? asm_exc_invalid_op+0x1a/0x20 [ 2250.721712] ? __list_add_valid_or_report+0x7d/0xa0 [ 2250.726596] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2250.731493] try_get_cap_refs+0x433/0x880 [ceph] [ 2250.736145] __ceph_get_caps+0xf8/0x760 [ceph] [ 2250.740623] ? ceph_read_iter+0x134/0x860 [ceph] [ 2250.745262] ceph_read_iter+0x134/0x860 [ceph] [ 2250.749731] ? selinux_file_permission+0x119/0x150 [ 2250.754531] ? vfs_read+0x20f/0x300 [ 2250.758025] vfs_read+0x20f/0x300 [ 2250.761355] ksys_pread64+0x65/0xa0 [ 2250.764845] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2250.769994] do_syscall_64+0x38/0x80 [ 2250.773574] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2250.778627] RIP: 0033:0x7f48f0eeb88b [ 2250.782204] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2250.800971] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2250.808537] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2250.815669] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2250.822802] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2250.829935] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2250.837069] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183dcf0 [ 2250.844212] [ 2250.846401] irq event stamp: 12854885 [ 2250.850067] hardirqs last enabled at (12854895): [] console_unlock+0x106/0x120 [ 2250.858933] hardirqs last disabled at (12854904): [] console_unlock+0xeb/0x120 [ 2250.867714] softirqs last enabled at (12854862): [] __do_softirq+0x2be/0x371 [ 2250.876406] softirqs last disabled at (12854857): [] irq_exit_rcu+0xb0/0xd0 [ 2250.884925] ---[ end trace 0000000000000000 ]--- [ 2250.889546] ------------[ cut here ]------------ [ 2250.894169] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2250.905831] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2250.914796] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2250.970319] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2250.979278] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2250.986762] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2250.992254] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2251.011002] RSP: 0018:ffffc90001ff7bf0 EFLAGS: 00010282 [ 2251.016228] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2251.023362] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2251.030493] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100004ff5 [ 2251.037635] R10: 00000000001c5720 R11: ffffc90001ff7a80 R12: ffff888158a68750 [ 2251.044780] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2251.051910] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2251.059996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2251.065743] CR2: 00007fb38d6c0038 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2251.072881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2251.080015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2251.087148] Call Trace: [ 2251.089603] [ 2251.091709] ? __warn+0x85/0x180 [ 2251.094943] ? __list_add_valid_or_report+0x7d/0xa0 [ 2251.099821] ? report_bug+0xfc/0x1d0 [ 2251.103400] ? prb_read_valid+0x1b/0x20 [ 2251.107247] ? handle_bug+0x44/0x70 [ 2251.110742] ? exc_invalid_op+0x17/0x60 [ 2251.114580] ? asm_exc_invalid_op+0x1a/0x20 [ 2251.118780] ? __list_add_valid_or_report+0x7d/0xa0 [ 2251.123663] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2251.128564] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2251.133290] lookup_fast+0x6d/0xf0 [ 2251.136700] walk_component+0x26/0x160 [ 2251.140453] link_path_walk+0x24d/0x380 [ 2251.144290] ? path_openat+0x80/0x9d0 [ 2251.147961] path_openat+0x8e/0x9d0 [ 2251.151465] do_filp_open+0xb8/0x120 [ 2251.155054] ? lock_release+0xc5/0x260 [ 2251.158811] ? _raw_spin_unlock+0x23/0x40 [ 2251.162829] ? alloc_fd+0x13f/0x200 [ 2251.166326] do_sys_openat2+0x249/0x290 [ 2251.170174] do_sys_open+0x4f/0x80 [ 2251.173586] do_syscall_64+0x38/0x80 [ 2251.177163] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2251.182216] RIP: 0033:0x7f48f0eed2f2 [ 2251.185796] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2251.204542] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2251.212107] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2251.219240] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2251.226374] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2251.233508] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2251.240643] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2251.247793] [ 2251.249983] irq event stamp: 12912929 [ 2251.253647] hardirqs last enabled at (12912939): [] console_unlock+0x106/0x120 [ 2251.262513] hardirqs last disabled at (12912948): [] console_unlock+0xeb/0x120 [ 2251.271292] softirqs last enabled at (12912906): [] __do_softirq+0x2be/0x371 [ 2251.279987] softirqs last disabled at (12912899): [] irq_exit_rcu+0xb0/0xd0 [ 2251.288506] ---[ end trace 0000000000000000 ]--- [ 2251.293126] ------------[ cut here ]------------ [ 2251.297878] list_del corruption. next->prev should be ffff8881a1d34bc8, but was ffff8881e4ee5590. (next=ffff8881e4ee5590) [ 2251.308839] WARNING: CPU: 7 PID: 78808 at lib/list_debug.c:67 __list_del_entry_valid_or_report+0xc3/0xd0 [ 2251.318319] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2251.373843] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2251.383234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2251.390715] Workqueue: events delayed_work [ceph] [ 2251.395445] RIP: 0010:__list_del_entry_valid_or_report+0xc3/0xd0 [ 2251.401450] Code: 0b 31 c0 c3 48 89 d1 48 89 f2 48 89 fe 48 c7 c7 58 9a 1d 82 e8 ae 0b b0 ff 0f 0b 31 c0 c3 48 c7 c7 a0 9a 1d 82 e8 9d 0b b0 ff <0f> 0b 31 c0 c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 d2 74 27 [ 2251.420197] RSP: 0018:ffffc90002117db0 EFLAGS: 00010286 [ 2251.425423] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 0000000000000000 [ 2251.432556] RDX: ffff88885fde9c00 RSI: ffff88885fddd880 RDI: ffff88885fddd880 [ 2251.439692] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005030 [ 2251.446821] R10: 00000000001c6768 R11: ffffc90002117c40 R12: 000000000000ea60 [ 2251.453954] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2251.461087] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2251.469174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2251.474919] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2251.482052] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2251.489184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2251.496320] Call Trace: [ 2251.498773] [ 2251.500884] ? __warn+0x85/0x180 [ 2251.504121] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2251.509519] ? report_bug+0xfc/0x1d0 [ 2251.513096] ? prb_read_valid+0x1b/0x20 [ 2251.516942] ? handle_bug+0x44/0x70 [ 2251.520439] ? exc_invalid_op+0x17/0x60 [ 2251.524279] ? asm_exc_invalid_op+0x1a/0x20 [ 2251.528473] ? __list_del_entry_valid_or_report+0xc3/0xd0 [ 2251.533873] ceph_check_delayed_caps+0x8f/0x270 [ceph] [ 2251.539034] delayed_work+0x1d8/0x280 [ceph] [ 2251.543332] ? process_scheduled_works+0x22a/0x510 [ 2251.548128] process_scheduled_works+0x2a4/0x510 [ 2251.552760] worker_thread+0x188/0x340 [ 2251.556521] ? rescuer_thread+0x310/0x310 [ 2251.560539] kthread+0xf5/0x130 [ 2251.563687] ? kthread_complete_and_exit+0x20/0x20 [ 2251.568482] ret_from_fork+0x31/0x40 [ 2251.572066] ? kthread_complete_and_exit+0x20/0x20 [ 2251.576860] ret_from_fork_asm+0x11/0x20 [ 2251.580809] [ 2251.583003] irq event stamp: 117565 [ 2251.586497] hardirqs last enabled at (117575): [] console_unlock+0x106/0x120 [ 2251.595189] hardirqs last disabled at (117584): [] console_unlock+0xeb/0x120 [ 2251.603796] softirqs last enabled at (117522): [] __do_softirq+0x2be/0x371 [ 2251.612316] softirqs last disabled at (117517): [] irq_exit_rcu+0xb0/0xd0 [ 2251.620660] ---[ end trace 0000000000000000 ]--- [ 2251.625281] ------------[ cut here ]------------ [ 2251.629898] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2251.641547] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2251.650504] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2251.706021] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2251.714973] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2251.722453] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2251.727940] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2251.746684] RSP: 0018:ffffc90001ff7bf0 EFLAGS: 00010282 [ 2251.751915] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2251.759045] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2251.766177] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005060 [ 2251.773311] R10: 00000000001c7708 R11: ffffc90001ff7a80 R12: ffff8881c5350750 [ 2251.780446] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2251.787577] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2251.795661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2251.801409] CR2: 00007fb38d726000 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2251.808542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2251.815680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2251.822816] Call Trace: [ 2251.825267] [ 2251.827374] ? __warn+0x85/0x180 [ 2251.830608] ? __list_add_valid_or_report+0x7d/0xa0 [ 2251.835487] ? report_bug+0xfc/0x1d0 [ 2251.839066] ? prb_read_valid+0x1b/0x20 [ 2251.842908] ? handle_bug+0x44/0x70 [ 2251.846399] ? exc_invalid_op+0x17/0x60 [ 2251.850239] ? asm_exc_invalid_op+0x1a/0x20 [ 2251.854434] ? __list_add_valid_or_report+0x7d/0xa0 [ 2251.859313] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2251.864212] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2251.868938] lookup_fast+0x6d/0xf0 [ 2251.872349] walk_component+0x26/0x160 [ 2251.876102] link_path_walk+0x24d/0x380 [ 2251.879944] ? path_openat+0x80/0x9d0 [ 2251.883619] path_openat+0x8e/0x9d0 [ 2251.887124] do_filp_open+0xb8/0x120 [ 2251.890710] ? lock_release+0xc5/0x260 [ 2251.894469] ? _raw_spin_unlock+0x23/0x40 [ 2251.898484] ? alloc_fd+0x13f/0x200 [ 2251.901982] do_sys_openat2+0x249/0x290 [ 2251.905829] do_sys_open+0x4f/0x80 [ 2251.909246] do_syscall_64+0x38/0x80 [ 2251.912830] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2251.917883] RIP: 0033:0x7f48f0eed2f2 [ 2251.921460] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2251.940207] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2251.947773] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2251.954907] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2251.962039] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2251.969172] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2251.976313] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2251.983458] [ 2251.985646] irq event stamp: 12914703 [ 2251.989314] hardirqs last enabled at (12914713): [] console_unlock+0x106/0x120 [ 2251.998179] hardirqs last disabled at (12914722): [] console_unlock+0xeb/0x120 [ 2252.006966] softirqs last enabled at (12914740): [] __do_softirq+0x2be/0x371 [ 2252.015660] softirqs last disabled at (12914731): [] irq_exit_rcu+0xb0/0xd0 [ 2252.024181] ---[ end trace 0000000000000000 ]--- [ 2252.028801] ------------[ cut here ]------------ [ 2252.033416] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2252.045066] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2252.054022] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2252.109501] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2252.118459] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2252.125937] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2252.131425] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2252.150176] RSP: 0018:ffffc90001fe7b28 EFLAGS: 00010286 [ 2252.155405] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2252.162536] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2252.169673] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000509b [ 2252.176807] R10: 00000000001c87b0 R11: ffffc90001fe79b8 R12: ffff8881d12c72e8 [ 2252.183944] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2252.191075] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2252.199162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2252.204909] CR2: 00007f9bd2b37020 CR3: 0000000108c08002 CR4: 00000000003706e0 [ 2252.212043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2252.219174] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2252.226309] Call Trace: [ 2252.228759] [ 2252.230868] ? __warn+0x85/0x180 [ 2252.234107] ? __list_add_valid_or_report+0x7d/0xa0 [ 2252.238991] ? report_bug+0xfc/0x1d0 [ 2252.242576] ? prb_read_valid+0x1b/0x20 [ 2252.246420] ? handle_bug+0x44/0x70 [ 2252.249914] ? exc_invalid_op+0x17/0x60 [ 2252.253756] ? asm_exc_invalid_op+0x1a/0x20 [ 2252.257950] ? __list_add_valid_or_report+0x7d/0xa0 [ 2252.262834] ? __list_add_valid_or_report+0x7d/0xa0 [ 2252.267716] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2252.272620] ceph_check_caps+0x999/0xae0 [ceph] [ 2252.277167] ? lock_release+0xc5/0x260 [ 2252.280926] ? __lock_acquire+0xbfa/0x1e20 [ 2252.285029] ? __lock_acquire+0xbfa/0x1e20 [ 2252.289139] ? lock_acquire+0xcb/0x2b0 [ 2252.292893] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2252.297880] ? find_held_lock+0x2d/0x90 [ 2252.301726] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2252.306803] ? lock_release+0xc5/0x260 [ 2252.310563] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2252.315472] ceph_read_iter+0x26e/0x860 [ceph] [ 2252.319935] ? selinux_file_permission+0x119/0x150 [ 2252.324734] ? vfs_read+0x20f/0x300 [ 2252.328230] vfs_read+0x20f/0x300 [ 2252.331558] ksys_pread64+0x65/0xa0 [ 2252.335049] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2252.340193] do_syscall_64+0x38/0x80 [ 2252.343779] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2252.348830] RIP: 0033:0x7f48f0eeb88b [ 2252.352410] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2252.371161] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2252.378729] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2252.385869] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2252.393006] RBP: 0000000000002a0f R08: 0000000000000100 R09: 0000000000000003 [ 2252.400139] R10: 000000000000f000 R11: 0000000000000246 R12: 0000000000001000 [ 2252.407269] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183dcf0 [ 2252.414415] [ 2252.416605] irq event stamp: 12859185 [ 2252.420271] hardirqs last enabled at (12859195): [] console_unlock+0x106/0x120 [ 2252.429138] hardirqs last disabled at (12859210): [] console_unlock+0xeb/0x120 [ 2252.437920] softirqs last enabled at (12859224): [] __do_softirq+0x2be/0x371 [ 2252.446617] softirqs last disabled at (12859219): [] irq_exit_rcu+0xb0/0xd0 [ 2252.455142] ---[ end trace 0000000000000000 ]--- [ 2252.459757] ------------[ cut here ]------------ [ 2252.464383] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2252.476035] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2252.484999] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2252.540519] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2252.549476] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2252.556956] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2252.562441] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2252.581193] RSP: 0018:ffffc90001fcfbf0 EFLAGS: 00010282 [ 2252.586429] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2252.593565] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2252.600697] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000050db [ 2252.607830] R10: 00000000001c9908 R11: ffffc90001fcfa80 R12: ffff8881d106cbc8 [ 2252.614967] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2252.622097] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2252.630182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2252.635928] CR2: 00007f9bd2b1a000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2252.643062] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2252.650195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2252.657326] Call Trace: [ 2252.659780] [ 2252.661887] ? __warn+0x85/0x180 [ 2252.665120] ? __list_add_valid_or_report+0x7d/0xa0 [ 2252.670000] ? report_bug+0xfc/0x1d0 [ 2252.673582] ? prb_read_valid+0x1b/0x20 [ 2252.677430] ? handle_bug+0x44/0x70 [ 2252.680929] ? exc_invalid_op+0x17/0x60 [ 2252.684767] ? asm_exc_invalid_op+0x1a/0x20 [ 2252.688968] ? __list_add_valid_or_report+0x7d/0xa0 [ 2252.693852] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2252.698752] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2252.703484] lookup_fast+0x6d/0xf0 [ 2252.706894] walk_component+0x26/0x160 [ 2252.710647] link_path_walk+0x24d/0x380 [ 2252.714489] ? path_openat+0x80/0x9d0 [ 2252.718165] path_openat+0x8e/0x9d0 [ 2252.721668] do_filp_open+0xb8/0x120 [ 2252.725255] ? lock_release+0xc5/0x260 [ 2252.729016] ? _raw_spin_unlock+0x23/0x40 [ 2252.733031] ? alloc_fd+0x13f/0x200 [ 2252.736530] do_sys_openat2+0x249/0x290 [ 2252.740377] do_sys_open+0x4f/0x80 [ 2252.743790] do_syscall_64+0x38/0x80 [ 2252.747378] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2252.752436] RIP: 0033:0x7f48f0eed2f2 [ 2252.756015] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2252.774760] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2252.782330] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2252.789460] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2252.796596] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2252.803727] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2252.810860] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2252.818007] [ 2252.820202] irq event stamp: 12838943 [ 2252.823868] hardirqs last enabled at (12838953): [] console_unlock+0x106/0x120 [ 2252.832734] hardirqs last disabled at (12838962): [] console_unlock+0xeb/0x120 [ 2252.841513] softirqs last enabled at (12838930): [] __do_softirq+0x2be/0x371 [ 2252.850211] softirqs last disabled at (12838981): [] irq_exit_rcu+0xb0/0xd0 [ 2252.858755] ---[ end trace 0000000000000000 ]--- [ 2252.863374] ------------[ cut here ]------------ [ 2252.867990] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2252.879639] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2252.888602] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2252.944124] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2252.953081] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2252.960563] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2252.966056] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2252.984806] RSP: 0018:ffffc90001957c88 EFLAGS: 00010286 [ 2252.990040] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2252.997171] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2253.004303] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005116 [ 2253.011436] R10: 00000000001ca6e8 R11: ffffc90001957b18 R12: ffff8881d1224bc8 [ 2253.018568] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2253.025702] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2253.033788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2253.039534] CR2: 00007f9bd2b15000 CR3: 000000016696c001 CR4: 00000000003706f0 [ 2253.046666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2253.053800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2253.060931] Call Trace: [ 2253.063385] [ 2253.065493] ? __warn+0x85/0x180 [ 2253.068726] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.073613] ? report_bug+0xfc/0x1d0 [ 2253.077193] ? prb_read_valid+0x1b/0x20 [ 2253.081033] ? handle_bug+0x44/0x70 [ 2253.084524] ? exc_invalid_op+0x17/0x60 [ 2253.088363] ? asm_exc_invalid_op+0x1a/0x20 [ 2253.092558] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.097438] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.102316] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2253.107217] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2253.111944] lookup_fast+0x6d/0xf0 [ 2253.115352] path_openat+0xe4/0x9d0 [ 2253.118853] do_filp_open+0xb8/0x120 [ 2253.122439] ? lock_release+0xc5/0x260 [ 2253.126199] ? _raw_spin_unlock+0x23/0x40 [ 2253.130214] ? alloc_fd+0x13f/0x200 [ 2253.133714] do_sys_openat2+0x249/0x290 [ 2253.137559] do_sys_open+0x4f/0x80 [ 2253.140974] do_syscall_64+0x38/0x80 [ 2253.144560] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2253.149611] RIP: 0033:0x7f48f0ef2805 [ 2253.153190] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2253.171937] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2253.179502] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2253.186636] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2253.193768] RBP: 000000000207da30 R08: 000000000fffffff R09: 0000000000000000 [ 2253.200901] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2253.208035] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2253.215180] [ 2253.217378] irq event stamp: 12828997 [ 2253.221044] hardirqs last enabled at (12829007): [] console_unlock+0x106/0x120 [ 2253.229908] hardirqs last disabled at (12829016): [] console_unlock+0xeb/0x120 [ 2253.238688] softirqs last enabled at (12828974): [] __do_softirq+0x2be/0x371 [ 2253.247385] softirqs last disabled at (12829035): [] irq_exit_rcu+0xb0/0xd0 [ 2253.255911] ---[ end trace 0000000000000000 ]--- [ 2253.260531] ------------[ cut here ]------------ [ 2253.265149] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2253.276793] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2253.285748] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2253.341287] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2253.350247] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2253.357726] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2253.363214] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2253.381960] RSP: 0018:ffffc90001fe7bd0 EFLAGS: 00010282 [ 2253.387184] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2253.394316] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2253.401450] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000514f [ 2253.408583] R10: 00000000001cb638 R11: ffffc90001fe7a60 R12: ffff8881d12c72e8 [ 2253.415716] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2253.422848] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2253.430935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2253.436682] CR2: 00007f9bd2b5e000 CR3: 0000000108c08003 CR4: 00000000003706e0 [ 2253.443818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2253.450951] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2253.458081] Call Trace: [ 2253.460533] [ 2253.462641] ? __warn+0x85/0x180 [ 2253.465873] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.470755] ? report_bug+0xfc/0x1d0 [ 2253.474340] ? prb_read_valid+0x1b/0x20 [ 2253.478186] ? handle_bug+0x44/0x70 [ 2253.481680] ? exc_invalid_op+0x17/0x60 [ 2253.485522] ? asm_exc_invalid_op+0x1a/0x20 [ 2253.489725] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.494604] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2253.499506] try_get_cap_refs+0x433/0x880 [ceph] [ 2253.504157] __ceph_get_caps+0xf8/0x760 [ceph] [ 2253.508636] ? ceph_read_iter+0x134/0x860 [ceph] [ 2253.513272] ceph_read_iter+0x134/0x860 [ceph] [ 2253.517745] ? selinux_file_permission+0x119/0x150 [ 2253.522542] ? vfs_read+0x20f/0x300 [ 2253.526037] vfs_read+0x20f/0x300 [ 2253.529370] ksys_pread64+0x65/0xa0 [ 2253.532867] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2253.538009] do_syscall_64+0x38/0x80 [ 2253.541595] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2253.546647] RIP: 0033:0x7f48f0eeb88b [ 2253.550230] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2253.568980] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2253.576547] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2253.583683] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2253.590812] RBP: 0000000000002a0f R08: 0000000000000063 R09: 0000000000000003 [ 2253.597951] R10: 000000000001f000 R11: 0000000000000246 R12: 0000000000001000 [ 2253.605087] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183dcf0 [ 2253.612235] [ 2253.614430] irq event stamp: 12861993 [ 2253.618097] hardirqs last enabled at (12862003): [] console_unlock+0x106/0x120 [ 2253.626961] hardirqs last disabled at (12862012): [] console_unlock+0xeb/0x120 [ 2253.635741] softirqs last enabled at (12861938): [] __do_softirq+0x2be/0x371 [ 2253.644435] softirqs last disabled at (12861933): [] irq_exit_rcu+0xb0/0xd0 [ 2253.652957] ---[ end trace 0000000000000000 ]--- [ 2253.657573] ------------[ cut here ]------------ [ 2253.662192] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2253.673848] WARNING: CPU: 6 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2253.682805] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2253.738345] CPU: 6 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2253.747300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2253.754780] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2253.760267] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2253.779010] RSP: 0018:ffffc90001957d50 EFLAGS: 00010286 [ 2253.784237] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2253.791371] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2253.798503] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005187 [ 2253.805644] R10: 00000000001cc5d0 R11: ffffc90001957be0 R12: ffff8881d0bb4bc8 [ 2253.812778] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2253.819911] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2253.827997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2253.833742] CR2: 00007f9bd2b67000 CR3: 000000016696c005 CR4: 00000000003706e0 [ 2253.840879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2253.848007] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2253.855142] Call Trace: [ 2253.857594] [ 2253.859702] ? __warn+0x85/0x180 [ 2253.862934] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.867814] ? report_bug+0xfc/0x1d0 [ 2253.871400] ? prb_read_valid+0x1b/0x20 [ 2253.875250] ? handle_bug+0x44/0x70 [ 2253.878741] ? exc_invalid_op+0x17/0x60 [ 2253.882581] ? asm_exc_invalid_op+0x1a/0x20 [ 2253.886778] ? __list_add_valid_or_report+0x7d/0xa0 [ 2253.891659] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2253.896565] ceph_readdir+0xc0/0x1670 [ceph] [ 2253.900855] ? wrap_directory_iterator+0x38/0x70 [ 2253.905480] ? wrap_directory_iterator+0x30/0x70 [ 2253.910099] ? lock_is_held_type+0xaa/0x120 [ 2253.914290] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2253.919181] wrap_directory_iterator+0x47/0x70 [ 2253.923631] iterate_dir+0x8d/0x170 [ 2253.927121] __x64_sys_getdents64+0x87/0x140 [ 2253.931394] ? __x64_sys_getdents64+0x140/0x140 [ 2253.935934] do_syscall_64+0x38/0x80 [ 2253.939515] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2253.944565] RIP: 0033:0x7f48f0ec480b [ 2253.948146] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2253.966891] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2253.974460] RAX: ffffffffffffffda RBX: 000000000207e080 RCX: 00007f48f0ec480b [ 2253.981591] RDX: 0000000000010000 RSI: 000000000207e0b0 RDI: 0000000000000007 [ 2253.988722] RBP: 000000000207e0b0 R08: 00000000000000db R09: 0000000000000003 [ 2253.995857] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2254.002987] R13: 0000000000000002 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2254.010143] [ 2254.012330] irq event stamp: 12830969 [ 2254.015996] hardirqs last enabled at (12830979): [] console_unlock+0x106/0x120 [ 2254.024864] hardirqs last disabled at (12830988): [] console_unlock+0xeb/0x120 [ 2254.033643] softirqs last enabled at (12830956): [] __do_softirq+0x2be/0x371 [ 2254.042337] softirqs last disabled at (12831007): [] irq_exit_rcu+0xb0/0xd0 [ 2254.050855] ---[ end trace 0000000000000000 ]--- [ 2254.055476] ------------[ cut here ]------------ [ 2254.060109] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2254.071760] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2254.080712] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2254.136221] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2254.145183] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2254.152670] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2254.158164] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2254.176921] RSP: 0018:ffffc90001fe7b28 EFLAGS: 00010286 [ 2254.182151] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2254.189285] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2254.196435] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000051bf [ 2254.203583] R10: 00000000001cd548 R11: ffffc90001fe79b8 R12: ffff8881d12c72e8 [ 2254.210716] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2254.217851] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2254.225942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2254.231689] CR2: 00007f9bd2b5e000 CR3: 0000000108c08003 CR4: 00000000003706e0 [ 2254.238823] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2254.245961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2254.253098] Call Trace: [ 2254.255551] [ 2254.257657] ? __warn+0x85/0x180 [ 2254.260890] ? __list_add_valid_or_report+0x7d/0xa0 [ 2254.265784] ? report_bug+0xfc/0x1d0 [ 2254.269368] ? prb_read_valid+0x1b/0x20 [ 2254.273215] ? handle_bug+0x44/0x70 [ 2254.276706] ? exc_invalid_op+0x17/0x60 [ 2254.280544] ? asm_exc_invalid_op+0x1a/0x20 [ 2254.284741] ? __list_add_valid_or_report+0x7d/0xa0 [ 2254.289620] ? __list_add_valid_or_report+0x7d/0xa0 [ 2254.294500] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2254.299407] ceph_check_caps+0x999/0xae0 [ceph] [ 2254.303966] ? lock_release+0xc5/0x260 [ 2254.307731] ? __lock_acquire+0xbfa/0x1e20 [ 2254.311841] ? __lock_acquire+0xbfa/0x1e20 [ 2254.315942] ? lock_acquire+0xcb/0x2b0 [ 2254.319696] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2254.324675] ? find_held_lock+0x2d/0x90 [ 2254.328519] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2254.333591] ? lock_release+0xc5/0x260 [ 2254.337355] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2254.342248] ceph_read_iter+0x26e/0x860 [ceph] [ 2254.346716] ? selinux_file_permission+0x119/0x150 [ 2254.351513] ? vfs_read+0x20f/0x300 [ 2254.355012] vfs_read+0x20f/0x300 [ 2254.358347] ksys_pread64+0x65/0xa0 [ 2254.361843] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2254.366991] do_syscall_64+0x38/0x80 [ 2254.370578] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2254.375637] RIP: 0033:0x7f48f0eeb88b [ 2254.379218] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2254.397968] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2254.405538] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb88b [ 2254.412672] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2254.419803] RBP: 0000000000002a0f R08: 0000000000000063 R09: 0000000000000003 [ 2254.426935] R10: 000000000001f000 R11: 0000000000000246 R12: 0000000000001000 [ 2254.434072] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183dcf0 [ 2254.441221] [ 2254.443414] irq event stamp: 12864021 [ 2254.447078] hardirqs last enabled at (12864031): [] console_unlock+0x106/0x120 [ 2254.455943] hardirqs last disabled at (12864040): [] console_unlock+0xeb/0x120 [ 2254.464725] softirqs last enabled at (12864054): [] __do_softirq+0x2be/0x371 [ 2254.473426] softirqs last disabled at (12864049): [] irq_exit_rcu+0xb0/0xd0 [ 2254.481951] ---[ end trace 0000000000000000 ]--- [ 2254.486575] ------------[ cut here ]------------ [ 2254.491191] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2254.502849] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2254.511812] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2254.567353] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2254.576311] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2254.583790] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2254.589275] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2254.608043] RSP: 0018:ffffc90001ff7bf0 EFLAGS: 00010282 [ 2254.613277] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2254.620415] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2254.627546] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000051ff [ 2254.634681] R10: 00000000001ce840 R11: ffffc90001ff7a80 R12: ffff8881c25d24a8 [ 2254.641815] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2254.648945] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2254.657031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2254.662778] CR2: 00007f9bd2b6c028 CR3: 0000000120cc8005 CR4: 00000000003706f0 [ 2254.669910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2254.677043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2254.684178] Call Trace: [ 2254.686629] [ 2254.688737] ? __warn+0x85/0x180 [ 2254.691970] ? __list_add_valid_or_report+0x7d/0xa0 [ 2254.696858] ? report_bug+0xfc/0x1d0 [ 2254.700444] ? prb_read_valid+0x1b/0x20 [ 2254.704289] ? handle_bug+0x44/0x70 [ 2254.707784] ? exc_invalid_op+0x17/0x60 [ 2254.711628] ? asm_exc_invalid_op+0x1a/0x20 [ 2254.715830] ? __list_add_valid_or_report+0x7d/0xa0 [ 2254.720717] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2254.725617] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2254.730344] lookup_fast+0x6d/0xf0 [ 2254.733752] walk_component+0x26/0x160 [ 2254.737510] link_path_walk+0x24d/0x380 [ 2254.741354] ? path_openat+0x80/0x9d0 [ 2254.745023] path_openat+0x8e/0x9d0 [ 2254.748528] do_filp_open+0xb8/0x120 [ 2254.752116] ? lock_release+0xc5/0x260 [ 2254.755874] ? _raw_spin_unlock+0x23/0x40 [ 2254.759891] ? alloc_fd+0x13f/0x200 [ 2254.763389] do_sys_openat2+0x249/0x290 [ 2254.767238] do_sys_open+0x4f/0x80 [ 2254.770650] do_syscall_64+0x38/0x80 [ 2254.774234] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2254.779287] RIP: 0033:0x7f48f0eed2f2 [ 2254.782866] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2254.801612] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2254.809177] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2254.816311] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2254.823443] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2254.830579] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2254.837710] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2254.844856] [ 2254.847053] irq event stamp: 12917609 [ 2254.850719] hardirqs last enabled at (12917619): [] console_unlock+0x106/0x120 [ 2254.859585] hardirqs last disabled at (12917628): [] console_unlock+0xeb/0x120 [ 2254.868365] softirqs last enabled at (12917576): [] __do_softirq+0x2be/0x371 [ 2254.877055] softirqs last disabled at (12917571): [] irq_exit_rcu+0xb0/0xd0 [ 2254.885576] ---[ end trace 0000000000000000 ]--- [ 2254.890196] ------------[ cut here ]------------ [ 2254.894814] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2254.906461] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2254.915420] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2254.970895] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2254.979860] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2254.987343] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2254.992842] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2255.011595] RSP: 0018:ffffc90001fcfc88 EFLAGS: 00010286 [ 2255.016824] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2255.023970] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2255.031154] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000523a [ 2255.038298] R10: 00000000001cf868 R11: ffffc90001fcfb18 R12: ffff8881d1295590 [ 2255.045439] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2255.052576] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2255.060664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2255.066407] CR2: 00007f9bd2b1a000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2255.073552] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2255.080682] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2255.087815] Call Trace: [ 2255.090271] [ 2255.092379] ? __warn+0x85/0x180 [ 2255.095617] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.100497] ? report_bug+0xfc/0x1d0 [ 2255.104084] ? prb_read_valid+0x1b/0x20 [ 2255.107930] ? handle_bug+0x44/0x70 [ 2255.111425] ? exc_invalid_op+0x17/0x60 [ 2255.115266] ? asm_exc_invalid_op+0x1a/0x20 [ 2255.119467] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.124349] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.129233] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2255.134137] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2255.138861] lookup_fast+0x6d/0xf0 [ 2255.142273] path_openat+0xe4/0x9d0 [ 2255.145777] do_filp_open+0xb8/0x120 [ 2255.149367] ? lock_release+0xc5/0x260 [ 2255.153123] ? _raw_spin_unlock+0x23/0x40 [ 2255.157144] ? alloc_fd+0x13f/0x200 [ 2255.160646] do_sys_openat2+0x249/0x290 [ 2255.164489] do_sys_open+0x4f/0x80 [ 2255.167900] do_syscall_64+0x38/0x80 [ 2255.171492] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2255.176545] RIP: 0033:0x7f48f0eed2f2 [ 2255.180129] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2255.198879] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2255.206456] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2255.213589] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2255.220722] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2255.227857] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2255.234991] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2255.242143] [ 2255.244350] irq event stamp: 12845383 [ 2255.248015] hardirqs last enabled at (12845393): [] console_unlock+0x106/0x120 [ 2255.256881] hardirqs last disabled at (12845402): [] console_unlock+0xeb/0x120 [ 2255.265668] softirqs last enabled at (12845380): [] __do_softirq+0x2be/0x371 [ 2255.274362] softirqs last disabled at (12845373): [] irq_exit_rcu+0xb0/0xd0 [ 2255.282887] ---[ end trace 0000000000000000 ]--- [ 2255.287511] ------------[ cut here ]------------ [ 2255.292135] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2255.303787] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2255.312752] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2255.368284] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2255.377235] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2255.384715] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2255.390202] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2255.408947] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2255.414175] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2255.421307] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2255.428439] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005273 [ 2255.435572] R10: 00000000001d09c0 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2255.442708] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2255.449838] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2255.457926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2255.463669] CR2: 00007fb38d730000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2255.470802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2255.477936] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2255.485069] Call Trace: [ 2255.487521] [ 2255.489628] ? __warn+0x85/0x180 [ 2255.492862] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.497742] ? report_bug+0xfc/0x1d0 [ 2255.501320] ? prb_read_valid+0x1b/0x20 [ 2255.505163] ? handle_bug+0x44/0x70 [ 2255.508660] ? exc_invalid_op+0x17/0x60 [ 2255.512502] ? asm_exc_invalid_op+0x1a/0x20 [ 2255.516695] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.521575] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2255.526476] try_get_cap_refs+0x433/0x880 [ceph] [ 2255.531128] __ceph_get_caps+0xf8/0x760 [ceph] [ 2255.535607] ? ceph_read_iter+0x134/0x860 [ceph] [ 2255.540242] ceph_read_iter+0x134/0x860 [ceph] [ 2255.544717] ? selinux_file_permission+0x119/0x150 [ 2255.549521] ? vfs_read+0x20f/0x300 [ 2255.553018] vfs_read+0x20f/0x300 [ 2255.556348] ksys_pread64+0x65/0xa0 [ 2255.559851] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2255.564995] do_syscall_64+0x38/0x80 [ 2255.568576] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2255.573632] RIP: 0033:0x7f48f0eeb88b [ 2255.577216] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2255.595962] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2255.603528] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2255.610660] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2255.617794] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2255.624924] R10: 000000000001f000 R11: 0000000000000246 R12: 0000000000001000 [ 2255.632059] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2255.639206] [ 2255.641401] irq event stamp: 12911779 [ 2255.645067] hardirqs last enabled at (12911789): [] console_unlock+0x106/0x120 [ 2255.653932] hardirqs last disabled at (12911798): [] console_unlock+0xeb/0x120 [ 2255.662713] softirqs last enabled at (12911680): [] __do_softirq+0x2be/0x371 [ 2255.671406] softirqs last disabled at (12911673): [] irq_exit_rcu+0xb0/0xd0 [ 2255.679925] ---[ end trace 0000000000000000 ]--- [ 2255.684546] ------------[ cut here ]------------ [ 2255.689163] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2255.700814] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2255.709774] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2255.765245] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2255.774207] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2255.781691] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2255.787189] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2255.805952] RSP: 0018:ffffc90001fcfc88 EFLAGS: 00010286 [ 2255.811194] RAX: 0000000000000000 RBX: ffff8881d12c09c8 RCX: 0000000000000000 [ 2255.818324] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2255.825459] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000052ab [ 2255.832595] R10: 00000000001d1b18 R11: ffffc90001fcfb18 R12: ffff8881d12c1118 [ 2255.839733] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2255.846865] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2255.854954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2255.860704] CR2: 00007f9bd2411000 CR3: 000000011aeae003 CR4: 00000000003706f0 [ 2255.867841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2255.874970] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2255.882104] Call Trace: [ 2255.884556] [ 2255.886667] ? __warn+0x85/0x180 [ 2255.889905] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.894786] ? report_bug+0xfc/0x1d0 [ 2255.898366] ? prb_read_valid+0x1b/0x20 [ 2255.902207] ? handle_bug+0x44/0x70 [ 2255.905705] ? exc_invalid_op+0x17/0x60 [ 2255.909545] ? asm_exc_invalid_op+0x1a/0x20 [ 2255.913736] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.918621] ? __list_add_valid_or_report+0x7d/0xa0 [ 2255.923509] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2255.928416] ceph_open+0x16b/0x440 [ceph] [ 2255.932446] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2255.937599] do_dentry_open+0x172/0x530 [ 2255.941441] path_openat+0x743/0x9d0 [ 2255.945026] do_filp_open+0xb8/0x120 [ 2255.948613] ? lock_release+0xc5/0x260 [ 2255.952371] ? _raw_spin_unlock+0x23/0x40 [ 2255.956387] ? alloc_fd+0x13f/0x200 [ 2255.959886] do_sys_openat2+0x249/0x290 [ 2255.963734] do_sys_open+0x4f/0x80 [ 2255.967146] do_syscall_64+0x38/0x80 [ 2255.970738] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2255.975794] RIP: 0033:0x7f48f0eed2f2 [ 2255.979371] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2255.998118] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2256.005684] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2256.012817] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2256.019948] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2256.027087] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2256.034227] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2256.041388] [ 2256.043586] irq event stamp: 12847393 [ 2256.047251] hardirqs last enabled at (12847403): [] console_unlock+0x106/0x120 [ 2256.056121] hardirqs last disabled at (12847420): [] console_unlock+0xeb/0x120 [ 2256.064906] softirqs last enabled at (12847418): [] __do_softirq+0x2be/0x371 [ 2256.073608] softirqs last disabled at (12847411): [] irq_exit_rcu+0xb0/0xd0 [ 2256.082126] ---[ end trace 0000000000000000 ]--- [ 2256.086746] ------------[ cut here ]------------ [ 2256.091364] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2256.103012] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2256.111966] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2256.167488] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2256.176440] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2256.183919] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2256.189405] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2256.208154] RSP: 0018:ffffc90001fffb28 EFLAGS: 00010286 [ 2256.213386] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2256.220518] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2256.227652] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000052e5 [ 2256.234784] R10: 00000000001d2a98 R11: ffffc90001fff9b8 R12: ffff8881d12c5590 [ 2256.241917] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2256.249050] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2256.257137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2256.262881] CR2: 00007fb38d730000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2256.270014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2256.277148] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2256.284280] Call Trace: [ 2256.286732] [ 2256.288841] ? __warn+0x85/0x180 [ 2256.292072] ? __list_add_valid_or_report+0x7d/0xa0 [ 2256.296952] ? report_bug+0xfc/0x1d0 [ 2256.300532] ? prb_read_valid+0x1b/0x20 [ 2256.304375] ? handle_bug+0x44/0x70 [ 2256.307875] ? exc_invalid_op+0x17/0x60 [ 2256.311712] ? asm_exc_invalid_op+0x1a/0x20 [ 2256.315908] ? __list_add_valid_or_report+0x7d/0xa0 [ 2256.320788] ? __list_add_valid_or_report+0x7d/0xa0 [ 2256.325673] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2256.330575] ceph_check_caps+0x999/0xae0 [ceph] [ 2256.335137] ? lock_release+0xc5/0x260 [ 2256.338898] ? __lock_acquire+0xbfa/0x1e20 [ 2256.343004] ? __lock_acquire+0xbfa/0x1e20 [ 2256.347110] ? lock_acquire+0xcb/0x2b0 [ 2256.350868] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2256.355852] ? find_held_lock+0x2d/0x90 [ 2256.359690] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2256.364762] ? lock_release+0xc5/0x260 [ 2256.368520] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2256.373425] ceph_read_iter+0x26e/0x860 [ceph] [ 2256.377901] ? selinux_file_permission+0x119/0x150 [ 2256.382697] ? vfs_read+0x20f/0x300 [ 2256.386193] vfs_read+0x20f/0x300 [ 2256.389525] ksys_pread64+0x65/0xa0 [ 2256.393023] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2256.398167] do_syscall_64+0x38/0x80 [ 2256.401750] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2256.406802] RIP: 0033:0x7f48f0eeb88b [ 2256.410382] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2256.429128] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2256.436696] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2256.443829] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2256.450960] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2256.458093] R10: 000000000001f000 R11: 0000000000000246 R12: 0000000000001000 [ 2256.465229] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2256.472373] [ 2256.474568] irq event stamp: 12913681 [ 2256.478234] hardirqs last enabled at (12913691): [] console_unlock+0x106/0x120 [ 2256.487102] hardirqs last disabled at (12913700): [] console_unlock+0xeb/0x120 [ 2256.495883] softirqs last enabled at (12913658): [] __do_softirq+0x2be/0x371 [ 2256.504582] softirqs last disabled at (12913651): [] irq_exit_rcu+0xb0/0xd0 [ 2256.513101] ---[ end trace 0000000000000000 ]--- [ 2256.517724] ------------[ cut here ]------------ [ 2256.522348] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2256.533998] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2256.542702] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2256.598247] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2256.607373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2256.614853] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2256.620382] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2256.625879] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2256.644632] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2256.649866] RAX: 0000000000000000 RBX: ffff8881d107ce40 RCX: 0000000000000000 [ 2256.657005] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2256.664141] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005325 [ 2256.671273] R10: 00000000001d3ca8 R11: ffffc9000030b828 R12: ffff8881d107d590 [ 2256.678404] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2256.685538] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2256.693624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2256.699369] CR2: 00007f9bd2b06000 CR3: 0000000108e80002 CR4: 00000000003706e0 [ 2256.706504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2256.713638] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2256.720769] Call Trace: [ 2256.723221] [ 2256.725329] ? __warn+0x85/0x180 [ 2256.728573] ? __list_add_valid_or_report+0x7d/0xa0 [ 2256.733459] ? report_bug+0xfc/0x1d0 [ 2256.737047] ? prb_read_valid+0x1b/0x20 [ 2256.740901] ? handle_bug+0x44/0x70 [ 2256.744397] ? exc_invalid_op+0x17/0x60 [ 2256.748246] ? asm_exc_invalid_op+0x1a/0x20 [ 2256.752449] ? __list_add_valid_or_report+0x7d/0xa0 [ 2256.757338] ? __list_add_valid_or_report+0x7d/0xa0 [ 2256.762222] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2256.767125] ceph_add_cap+0x20b/0x620 [ceph] [ 2256.771432] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2256.776184] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2256.781535] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2256.786022] ceph_con_process_message+0x74/0x150 [libceph] [ 2256.791528] ? calc_signature+0xdb/0x100 [libceph] [ 2256.796351] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2256.801694] ? process_scheduled_works+0x22a/0x510 [ 2256.806501] ? process_scheduled_works+0x22a/0x510 [ 2256.811295] ceph_con_workfn+0x344/0x750 [libceph] [ 2256.816110] ? process_scheduled_works+0x22a/0x510 [ 2256.820905] process_scheduled_works+0x2a4/0x510 [ 2256.825538] worker_thread+0x188/0x340 [ 2256.829300] ? rescuer_thread+0x310/0x310 [ 2256.833318] kthread+0xf5/0x130 [ 2256.836473] ? kthread_complete_and_exit+0x20/0x20 [ 2256.841277] ret_from_fork+0x31/0x40 [ 2256.844861] ? kthread_complete_and_exit+0x20/0x20 [ 2256.849655] ret_from_fork_asm+0x11/0x20 [ 2256.853605] [ 2256.855801] irq event stamp: 10478953 [ 2256.859473] hardirqs last enabled at (10478963): [] console_unlock+0x106/0x120 [ 2256.868346] hardirqs last disabled at (10478972): [] console_unlock+0xeb/0x120 [ 2256.877127] softirqs last enabled at (10478950): [] __do_softirq+0x2be/0x371 [ 2256.885828] softirqs last disabled at (10478945): [] irq_exit_rcu+0xb0/0xd0 [ 2256.894349] ---[ end trace 0000000000000000 ]--- [ 2256.898970] ------------[ cut here ]------------ [ 2256.903616] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2256.915281] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2256.924252] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2256.979781] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2256.988753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2256.996246] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2257.001750] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2257.020520] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2257.025758] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2257.032907] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2257.040060] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000535f [ 2257.047212] R10: 00000000001d4c00 R11: ffffc90001fef9c8 R12: ffff888158a68750 [ 2257.054360] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2257.061509] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2257.069612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2257.075375] CR2: 00007fb38d6ca008 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2257.082525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2257.089666] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2257.096810] Call Trace: [ 2257.099279] [ 2257.101403] ? __warn+0x85/0x180 [ 2257.104652] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.109553] ? report_bug+0xfc/0x1d0 [ 2257.113142] ? prb_read_valid+0x1b/0x20 [ 2257.116983] ? handle_bug+0x44/0x70 [ 2257.120473] ? exc_invalid_op+0x17/0x60 [ 2257.124312] ? asm_exc_invalid_op+0x1a/0x20 [ 2257.128507] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.133390] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.138275] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2257.143176] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2257.147907] lookup_fast+0x6d/0xf0 [ 2257.151319] walk_component+0x26/0x160 [ 2257.155074] link_path_walk+0x24d/0x380 [ 2257.158910] ? path_lookupat.isra.40+0x22/0x120 [ 2257.163449] path_lookupat.isra.40+0x6b/0x120 [ 2257.167816] filename_lookup+0xd4/0x190 [ 2257.171668] ? lock_acquire+0xcb/0x2b0 [ 2257.175420] ? __might_fault+0x45/0x80 [ 2257.179184] ? find_held_lock+0x2d/0x90 [ 2257.183030] ? __might_fault+0x45/0x80 [ 2257.186790] vfs_statx+0x89/0x150 [ 2257.190118] vfs_fstatat+0x5a/0xc0 [ 2257.193529] __do_sys_newstat+0x30/0x70 [ 2257.197375] ? lock_release+0xc5/0x260 [ 2257.201130] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2257.206013] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2257.211156] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2257.216304] do_syscall_64+0x38/0x80 [ 2257.219884] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2257.224943] RIP: 0033:0x7f48f0eeca79 [ 2257.228524] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2257.247268] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2257.254834] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2257.261967] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2257.269100] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2257.276237] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2257.283374] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2257.290520] [ 2257.292709] irq event stamp: 12875997 [ 2257.296375] hardirqs last enabled at (12876007): [] console_unlock+0x106/0x120 [ 2257.305241] hardirqs last disabled at (12876016): [] console_unlock+0xeb/0x120 [ 2257.314021] softirqs last enabled at (12875964): [] __do_softirq+0x2be/0x371 [ 2257.322713] softirqs last disabled at (12875957): [] irq_exit_rcu+0xb0/0xd0 [ 2257.331233] ---[ end trace 0000000000000000 ]--- [ 2257.335852] ------------[ cut here ]------------ [ 2257.340469] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2257.352118] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2257.360818] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2257.416401] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2257.425535] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2257.433019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2257.438529] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2257.444014] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2257.462761] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2257.467986] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2257.475120] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2257.482253] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000053a1 [ 2257.489390] R10: 00000000001d5e50 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2257.496527] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2257.503661] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2257.511752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2257.517500] CR2: 00007f9bd2b06000 CR3: 0000000108e80002 CR4: 00000000003706e0 [ 2257.524635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2257.531766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2257.538899] Call Trace: [ 2257.541353] [ 2257.543461] ? __warn+0x85/0x180 [ 2257.546701] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.551582] ? report_bug+0xfc/0x1d0 [ 2257.555169] ? prb_read_valid+0x1b/0x20 [ 2257.559020] ? handle_bug+0x44/0x70 [ 2257.562517] ? exc_invalid_op+0x17/0x60 [ 2257.566358] ? asm_exc_invalid_op+0x1a/0x20 [ 2257.570564] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.575452] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.580336] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2257.585239] ceph_add_cap+0x20b/0x620 [ceph] [ 2257.589546] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2257.594297] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2257.599648] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2257.604135] ceph_con_process_message+0x74/0x150 [libceph] [ 2257.609639] ? calc_signature+0xdb/0x100 [libceph] [ 2257.614463] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2257.619808] ? process_scheduled_works+0x22a/0x510 [ 2257.624614] ? process_scheduled_works+0x22a/0x510 [ 2257.629407] ceph_con_workfn+0x344/0x750 [libceph] [ 2257.634224] ? process_scheduled_works+0x22a/0x510 [ 2257.639020] process_scheduled_works+0x2a4/0x510 [ 2257.643659] worker_thread+0x188/0x340 [ 2257.647421] ? rescuer_thread+0x310/0x310 [ 2257.651439] kthread+0xf5/0x130 [ 2257.654586] ? kthread_complete_and_exit+0x20/0x20 [ 2257.659389] ret_from_fork+0x31/0x40 [ 2257.662975] ? kthread_complete_and_exit+0x20/0x20 [ 2257.667768] ret_from_fork_asm+0x11/0x20 [ 2257.671718] [ 2257.673912] irq event stamp: 10481053 [ 2257.677578] hardirqs last enabled at (10481063): [] console_unlock+0x106/0x120 [ 2257.686455] hardirqs last disabled at (10481078): [] console_unlock+0xeb/0x120 [ 2257.695240] softirqs last enabled at (10481076): [] __do_softirq+0x2be/0x371 [ 2257.703935] softirqs last disabled at (10481071): [] irq_exit_rcu+0xb0/0xd0 [ 2257.712463] ---[ end trace 0000000000000000 ]--- [ 2257.717081] ------------[ cut here ]------------ [ 2257.721698] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2257.733344] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2257.742302] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2257.797828] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2257.806784] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2257.814261] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2257.819746] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2257.838495] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2257.843719] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2257.850852] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2257.857985] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000053db [ 2257.865116] R10: 00000000001d6bd0 R11: ffffc90001fef9c8 R12: ffff8881c5350750 [ 2257.872250] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2257.879383] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2257.887469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2257.893216] CR2: 00007fb38d6ca008 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2257.900349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2257.907481] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2257.914613] Call Trace: [ 2257.917066] [ 2257.919174] ? __warn+0x85/0x180 [ 2257.922405] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.927291] ? report_bug+0xfc/0x1d0 [ 2257.930874] ? prb_read_valid+0x1b/0x20 [ 2257.934715] ? handle_bug+0x44/0x70 [ 2257.938206] ? exc_invalid_op+0x17/0x60 [ 2257.942047] ? asm_exc_invalid_op+0x1a/0x20 [ 2257.946240] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.951122] ? __list_add_valid_or_report+0x7d/0xa0 [ 2257.956008] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2257.960908] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2257.965634] lookup_fast+0x6d/0xf0 [ 2257.969043] walk_component+0x26/0x160 [ 2257.972799] link_path_walk+0x24d/0x380 [ 2257.976642] ? path_lookupat.isra.40+0x22/0x120 [ 2257.981180] path_lookupat.isra.40+0x6b/0x120 [ 2257.985547] filename_lookup+0xd4/0x190 [ 2257.989400] ? lock_acquire+0xcb/0x2b0 [ 2257.993153] ? __might_fault+0x45/0x80 [ 2257.996908] ? find_held_lock+0x2d/0x90 [ 2258.000755] ? __might_fault+0x45/0x80 [ 2258.004513] vfs_statx+0x89/0x150 [ 2258.007841] vfs_fstatat+0x5a/0xc0 [ 2258.011252] __do_sys_newstat+0x30/0x70 [ 2258.015089] ? lock_release+0xc5/0x260 [ 2258.018847] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2258.023729] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2258.028871] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2258.034012] do_syscall_64+0x38/0x80 [ 2258.037597] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2258.042649] RIP: 0033:0x7f48f0eeca79 [ 2258.046229] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2258.064978] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2258.072548] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2258.079683] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2258.086814] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2258.093947] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2258.101081] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2258.108226] [ 2258.110414] irq event stamp: 12877903 [ 2258.114081] hardirqs last enabled at (12877913): [] console_unlock+0x106/0x120 [ 2258.122947] hardirqs last disabled at (12877922): [] console_unlock+0xeb/0x120 [ 2258.131727] softirqs last enabled at (12877824): [] __do_softirq+0x2be/0x371 [ 2258.140420] softirqs last disabled at (12877817): [] irq_exit_rcu+0xb0/0xd0 [ 2258.148938] ---[ end trace 0000000000000000 ]--- [ 2258.153559] ------------[ cut here ]------------ [ 2258.158176] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2258.169825] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2258.178521] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2258.234068] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2258.243201] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2258.250680] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2258.256192] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2258.261678] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2258.280428] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2258.285660] RAX: 0000000000000000 RBX: ffff8881d0b58000 RCX: 0000000000000000 [ 2258.292791] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2258.299927] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000541d [ 2258.307057] R10: 00000000001d8088 R11: ffffc9000030b828 R12: ffff8881d0b58750 [ 2258.314192] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2258.321323] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2258.329409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2258.335155] CR2: 00007f9bd2b06000 CR3: 0000000108e80002 CR4: 00000000003706e0 [ 2258.342288] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2258.349421] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2258.356553] Call Trace: [ 2258.359007] [ 2258.361113] ? __warn+0x85/0x180 [ 2258.364348] ? __list_add_valid_or_report+0x7d/0xa0 [ 2258.369236] ? report_bug+0xfc/0x1d0 [ 2258.372822] ? prb_read_valid+0x1b/0x20 [ 2258.376665] ? handle_bug+0x44/0x70 [ 2258.380164] ? exc_invalid_op+0x17/0x60 [ 2258.384004] ? asm_exc_invalid_op+0x1a/0x20 [ 2258.388208] ? __list_add_valid_or_report+0x7d/0xa0 [ 2258.393097] ? __list_add_valid_or_report+0x7d/0xa0 [ 2258.397983] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2258.402885] ceph_add_cap+0x20b/0x620 [ceph] [ 2258.407197] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2258.411943] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2258.417295] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2258.421780] ceph_con_process_message+0x74/0x150 [libceph] [ 2258.427287] ? calc_signature+0xdb/0x100 [libceph] [ 2258.432108] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2258.437450] ? process_scheduled_works+0x22a/0x510 [ 2258.442248] ? process_scheduled_works+0x22a/0x510 [ 2258.447042] ceph_con_workfn+0x344/0x750 [libceph] [ 2258.451859] ? process_scheduled_works+0x22a/0x510 [ 2258.456656] process_scheduled_works+0x2a4/0x510 [ 2258.461287] worker_thread+0x188/0x340 [ 2258.465048] ? rescuer_thread+0x310/0x310 [ 2258.469068] kthread+0xf5/0x130 [ 2258.472214] ? kthread_complete_and_exit+0x20/0x20 [ 2258.477017] ret_from_fork+0x31/0x40 [ 2258.480604] ? kthread_complete_and_exit+0x20/0x20 [ 2258.485407] ret_from_fork_asm+0x11/0x20 [ 2258.489354] [ 2258.491550] irq event stamp: 10483039 [ 2258.495214] hardirqs last enabled at (10483049): [] console_unlock+0x106/0x120 [ 2258.504080] hardirqs last disabled at (10483058): [] console_unlock+0xeb/0x120 [ 2258.512861] softirqs last enabled at (10483036): [] __do_softirq+0x2be/0x371 [ 2258.521561] softirqs last disabled at (10483031): [] irq_exit_rcu+0xb0/0xd0 [ 2258.530081] ---[ end trace 0000000000000000 ]--- [ 2258.534702] ------------[ cut here ]------------ [ 2258.539320] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2258.550964] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2258.559919] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2258.615445] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2258.624400] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2258.631879] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2258.637368] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2258.656111] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2258.661340] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2258.668471] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2258.675604] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005457 [ 2258.682736] R10: 00000000001d8e50 R11: ffffc90001fef9c8 R12: ffff88819f028750 [ 2258.689870] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2258.697002] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2258.705089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2258.710835] CR2: 00007fb38d6ca008 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2258.717969] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2258.725100] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2258.732233] Call Trace: [ 2258.734686] [ 2258.736793] ? __warn+0x85/0x180 [ 2258.740026] ? __list_add_valid_or_report+0x7d/0xa0 [ 2258.744913] ? report_bug+0xfc/0x1d0 [ 2258.748492] ? prb_read_valid+0x1b/0x20 [ 2258.752335] ? handle_bug+0x44/0x70 [ 2258.755824] ? exc_invalid_op+0x17/0x60 [ 2258.759666] ? asm_exc_invalid_op+0x1a/0x20 [ 2258.763861] ? __list_add_valid_or_report+0x7d/0xa0 [ 2258.768740] ? __list_add_valid_or_report+0x7d/0xa0 [ 2258.773619] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2258.778519] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2258.783246] lookup_fast+0x6d/0xf0 [ 2258.786656] walk_component+0x26/0x160 [ 2258.790409] link_path_walk+0x24d/0x380 [ 2258.794255] ? path_lookupat.isra.40+0x22/0x120 [ 2258.798792] path_lookupat.isra.40+0x6b/0x120 [ 2258.803157] filename_lookup+0xd4/0x190 [ 2258.807001] ? lock_acquire+0xcb/0x2b0 [ 2258.810756] ? __might_fault+0x45/0x80 [ 2258.814510] ? find_held_lock+0x2d/0x90 [ 2258.818357] ? __might_fault+0x45/0x80 [ 2258.822115] vfs_statx+0x89/0x150 [ 2258.825443] vfs_fstatat+0x5a/0xc0 [ 2258.828855] __do_sys_newstat+0x30/0x70 [ 2258.832702] ? lock_release+0xc5/0x260 [ 2258.836457] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2258.841340] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2258.846482] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2258.851623] do_syscall_64+0x38/0x80 [ 2258.855210] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2258.860259] RIP: 0033:0x7f48f0eeca79 [ 2258.863839] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2258.882588] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2258.890160] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2258.897294] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2258.904425] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2258.911559] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2258.918691] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2258.925836] [ 2258.928025] irq event stamp: 12879779 [ 2258.931692] hardirqs last enabled at (12879789): [] console_unlock+0x106/0x120 [ 2258.940558] hardirqs last disabled at (12879798): [] console_unlock+0xeb/0x120 [ 2258.949338] softirqs last enabled at (12879700): [] __do_softirq+0x2be/0x371 [ 2258.958029] softirqs last disabled at (12879693): [] irq_exit_rcu+0xb0/0xd0 [ 2258.966550] ---[ end trace 0000000000000000 ]--- [ 2258.971169] ------------[ cut here ]------------ [ 2258.975789] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2258.987436] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2258.996134] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2259.051680] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2259.060813] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2259.068292] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2259.073804] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2259.079298] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2259.098044] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2259.103271] RAX: 0000000000000000 RBX: ffff8881c264e1d0 RCX: 0000000000000000 [ 2259.110402] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2259.117536] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005499 [ 2259.124671] R10: 00000000001da058 R11: ffffc9000030b828 R12: ffff8881c264e920 [ 2259.131810] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2259.138942] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2259.147029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2259.152775] CR2: 00007f9bd2b06000 CR3: 0000000108e80002 CR4: 00000000003706e0 [ 2259.159909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2259.167046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2259.174183] Call Trace: [ 2259.176635] [ 2259.178743] ? __warn+0x85/0x180 [ 2259.181983] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.186864] ? report_bug+0xfc/0x1d0 [ 2259.190454] ? prb_read_valid+0x1b/0x20 [ 2259.194302] ? handle_bug+0x44/0x70 [ 2259.197800] ? exc_invalid_op+0x17/0x60 [ 2259.201640] ? asm_exc_invalid_op+0x1a/0x20 [ 2259.205847] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.210735] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.215619] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2259.220522] ceph_add_cap+0x20b/0x620 [ceph] [ 2259.224826] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2259.229570] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2259.234927] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2259.239406] ceph_con_process_message+0x74/0x150 [libceph] [ 2259.244919] ? calc_signature+0xdb/0x100 [libceph] [ 2259.249737] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2259.255079] ? process_scheduled_works+0x22a/0x510 [ 2259.259888] ? process_scheduled_works+0x22a/0x510 [ 2259.264681] ceph_con_workfn+0x344/0x750 [libceph] [ 2259.269497] ? process_scheduled_works+0x22a/0x510 [ 2259.274293] process_scheduled_works+0x2a4/0x510 [ 2259.278928] worker_thread+0x188/0x340 [ 2259.282683] ? rescuer_thread+0x310/0x310 [ 2259.286695] kthread+0xf5/0x130 [ 2259.289841] ? kthread_complete_and_exit+0x20/0x20 [ 2259.294636] ret_from_fork+0x31/0x40 [ 2259.298222] ? kthread_complete_and_exit+0x20/0x20 [ 2259.303016] ret_from_fork_asm+0x11/0x20 [ 2259.306968] [ 2259.309160] irq event stamp: 10485033 [ 2259.312827] hardirqs last enabled at (10485043): [] console_unlock+0x106/0x120 [ 2259.321704] hardirqs last disabled at (10485058): [] console_unlock+0xeb/0x120 [ 2259.330489] softirqs last enabled at (10485056): [] __do_softirq+0x2be/0x371 [ 2259.339181] softirqs last disabled at (10485051): [] irq_exit_rcu+0xb0/0xd0 [ 2259.347709] ---[ end trace 0000000000000000 ]--- [ 2259.352329] ------------[ cut here ]------------ [ 2259.356945] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2259.368594] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2259.377548] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2259.433073] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2259.442032] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2259.449508] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2259.454996] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2259.473740] RSP: 0018:ffffc90001fefb38 EFLAGS: 00010282 [ 2259.478968] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2259.486100] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2259.493232] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000054d3 [ 2259.500365] R10: 00000000001db0e8 R11: ffffc90001fef9c8 R12: ffff88819f02f2e8 [ 2259.507499] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2259.514632] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2259.522718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2259.528462] CR2: 00007fb38d6ca008 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2259.535597] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2259.542730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2259.549862] Call Trace: [ 2259.552314] [ 2259.554422] ? __warn+0x85/0x180 [ 2259.557655] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.562538] ? report_bug+0xfc/0x1d0 [ 2259.566122] ? prb_read_valid+0x1b/0x20 [ 2259.569963] ? handle_bug+0x44/0x70 [ 2259.573454] ? exc_invalid_op+0x17/0x60 [ 2259.577293] ? asm_exc_invalid_op+0x1a/0x20 [ 2259.581488] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.586370] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.591256] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2259.596155] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2259.600882] lookup_fast+0x6d/0xf0 [ 2259.604291] walk_component+0x26/0x160 [ 2259.608045] link_path_walk+0x24d/0x380 [ 2259.611881] ? path_lookupat.isra.40+0x22/0x120 [ 2259.616420] path_lookupat.isra.40+0x6b/0x120 [ 2259.620787] filename_lookup+0xd4/0x190 [ 2259.624639] ? lock_acquire+0xcb/0x2b0 [ 2259.628391] ? __might_fault+0x45/0x80 [ 2259.632148] ? find_held_lock+0x2d/0x90 [ 2259.635993] ? __might_fault+0x45/0x80 [ 2259.639753] vfs_statx+0x89/0x150 [ 2259.643081] vfs_fstatat+0x5a/0xc0 [ 2259.646491] __do_sys_newstat+0x30/0x70 [ 2259.650330] ? lock_release+0xc5/0x260 [ 2259.654085] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2259.658968] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2259.664110] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2259.669249] do_syscall_64+0x38/0x80 [ 2259.672829] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2259.677880] RIP: 0033:0x7f48f0eeca79 [ 2259.681459] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2259.700208] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2259.707779] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2259.714913] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2259.722045] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2259.729179] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2259.736311] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2259.743456] [ 2259.745644] irq event stamp: 12881683 [ 2259.749311] hardirqs last enabled at (12881693): [] console_unlock+0x106/0x120 [ 2259.758177] hardirqs last disabled at (12881702): [] console_unlock+0xeb/0x120 [ 2259.766956] softirqs last enabled at (12881604): [] __do_softirq+0x2be/0x371 [ 2259.775649] softirqs last disabled at (12881597): [] irq_exit_rcu+0xb0/0xd0 [ 2259.784170] ---[ end trace 0000000000000000 ]--- [ 2259.788789] ------------[ cut here ]------------ [ 2259.793406] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2259.805055] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2259.814010] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2259.869530] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2259.878482] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2259.885962] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2259.891447] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2259.910193] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2259.915424] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2259.922552] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2259.929686] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005515 [ 2259.936817] R10: 00000000001dc2e0 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2259.943952] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2259.951085] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2259.959170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2259.964915] CR2: 00007fb38d730000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2259.972048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2259.979183] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2259.986314] Call Trace: [ 2259.988769] [ 2259.990875] ? __warn+0x85/0x180 [ 2259.994107] ? __list_add_valid_or_report+0x7d/0xa0 [ 2259.998988] ? report_bug+0xfc/0x1d0 [ 2260.002564] ? prb_read_valid+0x1b/0x20 [ 2260.006410] ? handle_bug+0x44/0x70 [ 2260.009905] ? exc_invalid_op+0x17/0x60 [ 2260.013746] ? asm_exc_invalid_op+0x1a/0x20 [ 2260.017944] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.022830] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2260.027729] try_get_cap_refs+0x433/0x880 [ceph] [ 2260.032372] __ceph_get_caps+0xf8/0x760 [ceph] [ 2260.036852] ? ceph_read_iter+0x134/0x860 [ceph] [ 2260.041488] ceph_read_iter+0x134/0x860 [ceph] [ 2260.045966] ? selinux_file_permission+0x119/0x150 [ 2260.050765] ? vfs_read+0x20f/0x300 [ 2260.054262] vfs_read+0x20f/0x300 [ 2260.057594] ksys_pread64+0x65/0xa0 [ 2260.061093] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2260.066235] do_syscall_64+0x38/0x80 [ 2260.069822] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2260.074874] RIP: 0033:0x7f48f0eeb88b [ 2260.078450] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2260.097197] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2260.104763] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2260.111898] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2260.119028] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2260.126168] R10: 000000000002f000 R11: 0000000000000246 R12: 0000000000001000 [ 2260.133304] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2260.140450] [ 2260.142647] irq event stamp: 12927597 [ 2260.146314] hardirqs last enabled at (12927607): [] console_unlock+0x106/0x120 [ 2260.155179] hardirqs last disabled at (12927616): [] console_unlock+0xeb/0x120 [ 2260.163959] softirqs last enabled at (12927564): [] __do_softirq+0x2be/0x371 [ 2260.172652] softirqs last disabled at (12927557): [] irq_exit_rcu+0xb0/0xd0 [ 2260.181176] ---[ end trace 0000000000000000 ]--- [ 2260.185801] ------------[ cut here ]------------ [ 2260.190417] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2260.202073] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2260.210795] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2260.266354] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2260.275484] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2260.282964] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2260.288477] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2260.293969] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2260.312718] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2260.317951] RAX: 0000000000000000 RBX: ffff8881d13d4e40 RCX: 0000000000000000 [ 2260.325086] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2260.332219] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000554d [ 2260.339351] R10: 00000000001dd290 R11: ffffc9000030b828 R12: ffff8881d13d5590 [ 2260.346483] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2260.353616] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2260.361701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.367448] CR2: 00007f9bd2b06000 CR3: 0000000108e80002 CR4: 00000000003706e0 [ 2260.374582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2260.381714] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2260.388845] Call Trace: [ 2260.391300] [ 2260.393406] ? __warn+0x85/0x180 [ 2260.396642] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.401530] ? report_bug+0xfc/0x1d0 [ 2260.405116] ? prb_read_valid+0x1b/0x20 [ 2260.408967] ? handle_bug+0x44/0x70 [ 2260.412464] ? exc_invalid_op+0x17/0x60 [ 2260.416305] ? asm_exc_invalid_op+0x1a/0x20 [ 2260.420509] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.425398] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.430282] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2260.435186] ceph_add_cap+0x20b/0x620 [ceph] [ 2260.439491] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2260.444235] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2260.449586] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2260.454074] ceph_con_process_message+0x74/0x150 [libceph] [ 2260.459581] ? calc_signature+0xdb/0x100 [libceph] [ 2260.464401] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2260.469744] ? process_scheduled_works+0x22a/0x510 [ 2260.474551] ? process_scheduled_works+0x22a/0x510 [ 2260.479345] ceph_con_workfn+0x344/0x750 [libceph] [ 2260.484161] ? process_scheduled_works+0x22a/0x510 [ 2260.488958] process_scheduled_works+0x2a4/0x510 [ 2260.493596] worker_thread+0x188/0x340 [ 2260.497359] ? rescuer_thread+0x310/0x310 [ 2260.501377] kthread+0xf5/0x130 [ 2260.504523] ? kthread_complete_and_exit+0x20/0x20 [ 2260.509320] ret_from_fork+0x31/0x40 [ 2260.512903] ? kthread_complete_and_exit+0x20/0x20 [ 2260.517699] ret_from_fork_asm+0x11/0x20 [ 2260.521646] [ 2260.523845] irq event stamp: 10488099 [ 2260.527515] hardirqs last enabled at (10488109): [] console_unlock+0x106/0x120 [ 2260.536382] hardirqs last disabled at (10488118): [] console_unlock+0xeb/0x120 [ 2260.545162] softirqs last enabled at (10488096): [] __do_softirq+0x2be/0x371 [ 2260.553864] softirqs last disabled at (10488091): [] irq_exit_rcu+0xb0/0xd0 [ 2260.562390] ---[ end trace 0000000000000000 ]--- [ 2260.567010] ------------[ cut here ]------------ [ 2260.571629] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2260.583282] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2260.592237] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2260.647764] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2260.656718] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2260.664197] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2260.669683] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2260.688432] RSP: 0018:ffffc90001fefa78 EFLAGS: 00010286 [ 2260.693656] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2260.700788] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2260.707922] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005587 [ 2260.715053] R10: 00000000001de440 R11: ffffc90001fef908 R12: ffff8881d12d1118 [ 2260.722189] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2260.729321] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2260.737407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.743153] CR2: 00007fb38d6ca008 CR3: 0000000108c72004 CR4: 00000000003706e0 [ 2260.750285] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2260.757420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2260.764550] Call Trace: [ 2260.767003] [ 2260.769110] ? __warn+0x85/0x180 [ 2260.772344] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.777227] ? report_bug+0xfc/0x1d0 [ 2260.780810] ? prb_read_valid+0x1b/0x20 [ 2260.784654] ? handle_bug+0x44/0x70 [ 2260.788143] ? exc_invalid_op+0x17/0x60 [ 2260.791981] ? asm_exc_invalid_op+0x1a/0x20 [ 2260.796177] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.801057] ? __list_add_valid_or_report+0x7d/0xa0 [ 2260.805935] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2260.810834] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2260.816080] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2260.820891] ceph_lookup+0xf9/0x3f0 [ceph] [ 2260.825008] __lookup_slow+0xff/0x1a0 [ 2260.828694] walk_component+0x83/0x160 [ 2260.832451] path_lookupat.isra.40+0x58/0x120 [ 2260.836810] filename_lookup+0xd4/0x190 [ 2260.840653] ? lock_acquire+0xcb/0x2b0 [ 2260.844407] ? __might_fault+0x45/0x80 [ 2260.848163] ? find_held_lock+0x2d/0x90 [ 2260.852000] ? __might_fault+0x45/0x80 [ 2260.855759] vfs_statx+0x89/0x150 [ 2260.859087] vfs_fstatat+0x5a/0xc0 [ 2260.862498] __do_sys_newstat+0x30/0x70 [ 2260.866335] ? lock_release+0xc5/0x260 [ 2260.870092] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2260.874975] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2260.880116] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2260.885259] do_syscall_64+0x38/0x80 [ 2260.888842] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2260.893894] RIP: 0033:0x7f48f0eeca79 [ 2260.897474] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2260.916224] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2260.923793] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2260.930928] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2260.938061] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2260.945193] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2260.952328] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2260.959471] [ 2260.961660] irq event stamp: 12884445 [ 2260.965326] hardirqs last enabled at (12884455): [] console_unlock+0x106/0x120 [ 2260.974192] hardirqs last disabled at (12884464): [] console_unlock+0xeb/0x120 [ 2260.982974] softirqs last enabled at (12884422): [] __do_softirq+0x2be/0x371 [ 2260.991664] softirqs last disabled at (12884417): [] irq_exit_rcu+0xb0/0xd0 [ 2261.000183] ---[ end trace 0000000000000000 ]--- [ 2261.004805] ------------[ cut here ]------------ [ 2261.009428] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2261.021084] WARNING: CPU: 5 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2261.030048] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2261.085520] CPU: 5 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2261.094477] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2261.101961] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2261.107448] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2261.126193] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2261.131420] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2261.138552] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2261.145685] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000055c9 [ 2261.152816] R10: 00000000001df590 R11: ffffc90001fd79c8 R12: ffff888158a68750 [ 2261.159958] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2261.167092] FS: 00007f48f1832740(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2261.175179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2261.180923] CR2: 00007f9bd2b06000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2261.188059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2261.195196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2261.202330] Call Trace: [ 2261.204799] [ 2261.206915] ? __warn+0x85/0x180 [ 2261.210166] ? __list_add_valid_or_report+0x7d/0xa0 [ 2261.215058] ? report_bug+0xfc/0x1d0 [ 2261.218655] ? prb_read_valid+0x1b/0x20 [ 2261.222505] ? handle_bug+0x44/0x70 [ 2261.226001] ? exc_invalid_op+0x17/0x60 [ 2261.229841] ? asm_exc_invalid_op+0x1a/0x20 [ 2261.234043] ? __list_add_valid_or_report+0x7d/0xa0 [ 2261.238927] ? __list_add_valid_or_report+0x7d/0xa0 [ 2261.243810] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2261.248713] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2261.253438] lookup_fast+0x6d/0xf0 [ 2261.256849] walk_component+0x26/0x160 [ 2261.260608] link_path_walk+0x24d/0x380 [ 2261.264448] ? path_lookupat.isra.40+0x22/0x120 [ 2261.268982] path_lookupat.isra.40+0x6b/0x120 [ 2261.273352] filename_lookup+0xd4/0x190 [ 2261.277200] ? lock_acquire+0xcb/0x2b0 [ 2261.280964] ? __might_fault+0x45/0x80 [ 2261.284719] ? find_held_lock+0x2d/0x90 [ 2261.288561] ? __might_fault+0x45/0x80 [ 2261.292327] vfs_statx+0x89/0x150 [ 2261.295654] vfs_fstatat+0x5a/0xc0 [ 2261.299062] __do_sys_newstat+0x30/0x70 [ 2261.302903] ? lock_release+0xc5/0x260 [ 2261.306656] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2261.311543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2261.316690] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2261.321833] do_syscall_64+0x38/0x80 [ 2261.325417] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2261.330471] RIP: 0033:0x7f48f0eeca79 [ 2261.334049] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2261.352798] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2261.360372] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2261.367504] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2261.374635] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2261.381770] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2261.388901] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2261.396047] [ 2261.398234] irq event stamp: 12970363 [ 2261.401903] hardirqs last enabled at (12970373): [] console_unlock+0x106/0x120 [ 2261.410775] hardirqs last disabled at (12970382): [] console_unlock+0xeb/0x120 [ 2261.419556] softirqs last enabled at (12970300): [] __do_softirq+0x2be/0x371 [ 2261.428252] softirqs last disabled at (12970401): [] irq_exit_rcu+0xb0/0xd0 [ 2261.436778] ---[ end trace 0000000000000000 ]--- [ 2261.441396] ------------[ cut here ]------------ [ 2261.446015] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2261.457661] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2261.466622] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2261.522146] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2261.531100] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2261.538576] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2261.544063] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2261.562808] RSP: 0018:ffffc90001fffb28 EFLAGS: 00010286 [ 2261.568035] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2261.575166] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2261.582300] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000560b [ 2261.589433] R10: 00000000001e08e8 R11: ffffc90001fff9b8 R12: ffff8881d12c5590 [ 2261.596568] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2261.603700] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2261.611785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2261.617529] CR2: 00007fb38d730000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2261.624664] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2261.631796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2261.638931] Call Trace: [ 2261.641381] [ 2261.643489] ? __warn+0x85/0x180 [ 2261.646723] ? __list_add_valid_or_report+0x7d/0xa0 [ 2261.651601] ? report_bug+0xfc/0x1d0 [ 2261.655181] ? prb_read_valid+0x1b/0x20 [ 2261.659022] ? handle_bug+0x44/0x70 [ 2261.662511] ? exc_invalid_op+0x17/0x60 [ 2261.666351] ? asm_exc_invalid_op+0x1a/0x20 [ 2261.670547] ? __list_add_valid_or_report+0x7d/0xa0 [ 2261.675429] ? __list_add_valid_or_report+0x7d/0xa0 [ 2261.680304] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2261.685206] ceph_check_caps+0x999/0xae0 [ceph] [ 2261.689755] ? lock_release+0xc5/0x260 [ 2261.693515] ? __lock_acquire+0xbfa/0x1e20 [ 2261.697618] ? __lock_acquire+0xbfa/0x1e20 [ 2261.701725] ? lock_acquire+0xcb/0x2b0 [ 2261.705473] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2261.710457] ? find_held_lock+0x2d/0x90 [ 2261.714297] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2261.719367] ? lock_release+0xc5/0x260 [ 2261.723128] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2261.728030] ceph_read_iter+0x26e/0x860 [ceph] [ 2261.732496] ? selinux_file_permission+0x119/0x150 [ 2261.737294] ? vfs_read+0x20f/0x300 [ 2261.740791] vfs_read+0x20f/0x300 [ 2261.744122] ksys_pread64+0x65/0xa0 [ 2261.747611] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2261.752754] do_syscall_64+0x38/0x80 [ 2261.756339] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2261.761391] RIP: 0033:0x7f48f0eeb88b [ 2261.764970] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2261.783716] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2261.791282] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2261.798416] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2261.805548] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2261.812681] R10: 000000000002f000 R11: 0000000000000246 R12: 0000000000001000 [ 2261.819814] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2261.826966] [ 2261.829157] irq event stamp: 12931279 [ 2261.832823] hardirqs last enabled at (12931289): [] console_unlock+0x106/0x120 [ 2261.841688] hardirqs last disabled at (12931298): [] console_unlock+0xeb/0x120 [ 2261.850471] softirqs last enabled at (12931314): [] __do_softirq+0x2be/0x371 [ 2261.859163] softirqs last disabled at (12931307): [] irq_exit_rcu+0xb0/0xd0 [ 2261.867681] ---[ end trace 0000000000000000 ]--- [ 2261.872301] ------------[ cut here ]------------ [ 2261.876924] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2261.888575] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2261.897530] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2261.953057] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2261.962018] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2261.969500] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2261.974985] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2261.993732] RSP: 0018:ffffc90001fcfaa0 EFLAGS: 00010286 [ 2261.998967] RAX: 0000000000000000 RBX: ffff8881d12c09c8 RCX: 0000000000000000 [ 2262.006100] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2262.013232] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000564b [ 2262.020368] R10: 00000000001e19d0 R11: ffffc90001fcf930 R12: ffff8881d12c1118 [ 2262.027503] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2262.034629] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2262.042717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2262.048467] CR2: 00007fb38d763000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2262.055596] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2262.062731] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2262.069862] Call Trace: [ 2262.072315] [ 2262.074421] ? __warn+0x85/0x180 [ 2262.077656] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.082535] ? report_bug+0xfc/0x1d0 [ 2262.086114] ? prb_read_valid+0x1b/0x20 [ 2262.089958] ? handle_bug+0x44/0x70 [ 2262.093453] ? exc_invalid_op+0x17/0x60 [ 2262.097295] ? asm_exc_invalid_op+0x1a/0x20 [ 2262.101494] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.106378] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2262.111279] ceph_check_caps+0x999/0xae0 [ceph] [ 2262.115886] ? lock_release+0xc5/0x260 [ 2262.119644] ceph_open+0x3cc/0x440 [ceph] [ 2262.123682] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2262.128832] do_dentry_open+0x172/0x530 [ 2262.132676] path_openat+0x743/0x9d0 [ 2262.136271] do_filp_open+0xb8/0x120 [ 2262.139858] ? lock_release+0xc5/0x260 [ 2262.143614] ? _raw_spin_unlock+0x23/0x40 [ 2262.147629] ? alloc_fd+0x13f/0x200 [ 2262.151134] do_sys_openat2+0x249/0x290 [ 2262.154976] do_sys_open+0x4f/0x80 [ 2262.158388] do_syscall_64+0x38/0x80 [ 2262.161969] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2262.167026] RIP: 0033:0x7f48f0eed2f2 [ 2262.170613] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2262.189359] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2262.196924] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2262.204061] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2262.211199] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2262.218336] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2262.225465] R13: 00007f48f183d678 R14: 00000000020767c0 R15: 00000000000029b2 [ 2262.232619] [ 2262.234808] irq event stamp: 12883513 [ 2262.238473] hardirqs last enabled at (12883529): [] console_unlock+0x106/0x120 [ 2262.247341] hardirqs last disabled at (12883544): [] console_unlock+0xeb/0x120 [ 2262.256119] softirqs last enabled at (12883558): [] __do_softirq+0x2be/0x371 [ 2262.264812] softirqs last disabled at (12883569): [] irq_exit_rcu+0xb0/0xd0 [ 2262.273334] ---[ end trace 0000000000000000 ]--- [ 2262.277953] ------------[ cut here ]------------ [ 2262.282572] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2262.294218] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2262.302914] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2262.358444] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2262.367569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2262.375048] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2262.380552] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2262.386045] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2262.404792] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2262.410017] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2262.417150] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2262.424286] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005686 [ 2262.431416] R10: 00000000001e2b88 R11: ffffc9000030b6a8 R12: ffff8881d12c5590 [ 2262.438550] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2262.445683] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2262.453768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2262.459512] CR2: 00007f9bd2b06000 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2262.466647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2262.473778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2262.480912] Call Trace: [ 2262.483366] [ 2262.485471] ? __warn+0x85/0x180 [ 2262.488707] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.493594] ? report_bug+0xfc/0x1d0 [ 2262.497172] ? prb_read_valid+0x1b/0x20 [ 2262.501014] ? handle_bug+0x44/0x70 [ 2262.504504] ? exc_invalid_op+0x17/0x60 [ 2262.508345] ? asm_exc_invalid_op+0x1a/0x20 [ 2262.512538] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.517419] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.522297] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2262.527197] ceph_check_caps+0x999/0xae0 [ceph] [ 2262.531777] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2262.536489] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2262.541226] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2262.546062] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2262.550440] ? lock_release+0xc5/0x260 [ 2262.554196] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2262.558910] ceph_con_process_message+0x74/0x150 [libceph] [ 2262.564417] ? calc_signature+0xdb/0x100 [libceph] [ 2262.569231] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2262.574572] ? process_scheduled_works+0x22a/0x510 [ 2262.579374] ? process_scheduled_works+0x22a/0x510 [ 2262.584169] ceph_con_workfn+0x344/0x750 [libceph] [ 2262.588982] ? process_scheduled_works+0x22a/0x510 [ 2262.593780] process_scheduled_works+0x2a4/0x510 [ 2262.598411] worker_thread+0x188/0x340 [ 2262.602164] ? rescuer_thread+0x310/0x310 [ 2262.606183] kthread+0xf5/0x130 [ 2262.609330] ? kthread_complete_and_exit+0x20/0x20 [ 2262.614123] ret_from_fork+0x31/0x40 [ 2262.617702] ? kthread_complete_and_exit+0x20/0x20 [ 2262.622496] ret_from_fork_asm+0x11/0x20 [ 2262.626432] [ 2262.628621] irq event stamp: 10491289 [ 2262.632287] hardirqs last enabled at (10491299): [] console_unlock+0x106/0x120 [ 2262.641156] hardirqs last disabled at (10491314): [] console_unlock+0xeb/0x120 [ 2262.649933] softirqs last enabled at (10491312): [] __do_softirq+0x2be/0x371 [ 2262.658624] softirqs last disabled at (10491307): [] irq_exit_rcu+0xb0/0xd0 [ 2262.667145] ---[ end trace 0000000000000000 ]--- [ 2262.671764] ------------[ cut here ]------------ [ 2262.676383] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2262.688029] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2262.696983] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2262.752509] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2262.761464] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2262.768945] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2262.774431] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2262.793177] RSP: 0000:ffffc90001fd7b38 EFLAGS: 00010282 [ 2262.798402] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2262.805535] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2262.812668] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000056c3 [ 2262.819801] R10: 00000000001e3dc0 R11: ffffc90001fd79c8 R12: ffff8881c5350750 [ 2262.826934] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2262.834066] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2262.842154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2262.847898] CR2: 00007f9bd2bcf000 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2262.855032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2262.862164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2262.869298] Call Trace: [ 2262.871750] [ 2262.873857] ? __warn+0x85/0x180 [ 2262.877090] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.881973] ? report_bug+0xfc/0x1d0 [ 2262.885556] ? prb_read_valid+0x1b/0x20 [ 2262.889399] ? handle_bug+0x44/0x70 [ 2262.892889] ? exc_invalid_op+0x17/0x60 [ 2262.896728] ? asm_exc_invalid_op+0x1a/0x20 [ 2262.900925] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.905803] ? __list_add_valid_or_report+0x7d/0xa0 [ 2262.910683] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2262.915584] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2262.920309] lookup_fast+0x6d/0xf0 [ 2262.923718] walk_component+0x26/0x160 [ 2262.927472] link_path_walk+0x24d/0x380 [ 2262.931310] ? path_lookupat.isra.40+0x22/0x120 [ 2262.935854] path_lookupat.isra.40+0x6b/0x120 [ 2262.940212] filename_lookup+0xd4/0x190 [ 2262.944057] ? lock_acquire+0xcb/0x2b0 [ 2262.947815] ? __might_fault+0x45/0x80 [ 2262.951574] ? find_held_lock+0x2d/0x90 [ 2262.955412] ? __might_fault+0x45/0x80 [ 2262.959172] vfs_statx+0x89/0x150 [ 2262.962498] vfs_fstatat+0x5a/0xc0 [ 2262.965909] __do_sys_newstat+0x30/0x70 [ 2262.969750] ? lock_release+0xc5/0x260 [ 2262.973504] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2262.978386] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2262.983527] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2262.988669] do_syscall_64+0x38/0x80 [ 2262.992245] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2262.997297] RIP: 0033:0x7f48f0eeca79 [ 2263.000883] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2263.019635] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2263.027198] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2263.034334] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2263.041464] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2263.048596] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2263.055731] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2263.062876] [ 2263.065063] irq event stamp: 12974071 [ 2263.068729] hardirqs last enabled at (12974081): [] console_unlock+0x106/0x120 [ 2263.077595] hardirqs last disabled at (12974090): [] console_unlock+0xeb/0x120 [ 2263.086375] softirqs last enabled at (12974018): [] __do_softirq+0x2be/0x371 [ 2263.095067] softirqs last disabled at (12974013): [] irq_exit_rcu+0xb0/0xd0 [ 2263.103590] ---[ end trace 0000000000000000 ]--- [ 2263.108208] ------------[ cut here ]------------ [ 2263.112827] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2263.124478] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2263.133438] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2263.188965] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2263.197918] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2263.205398] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2263.210883] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2263.229634] RSP: 0018:ffffc90001fdfbf0 EFLAGS: 00010282 [ 2263.234865] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2263.241997] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2263.249130] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005705 [ 2263.256266] R10: 00000000001e4f40 R11: ffffc90001fdfa80 R12: ffff888158a68750 [ 2263.263399] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2263.270528] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2263.278615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2263.284360] CR2: 00007f9bd2bc3000 CR3: 0000000179402001 CR4: 00000000003706f0 [ 2263.291494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2263.298632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2263.305758] Call Trace: [ 2263.308213] [ 2263.310318] ? __warn+0x85/0x180 [ 2263.313551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2263.318432] ? report_bug+0xfc/0x1d0 [ 2263.322009] ? prb_read_valid+0x1b/0x20 [ 2263.325854] ? handle_bug+0x44/0x70 [ 2263.329353] ? exc_invalid_op+0x17/0x60 [ 2263.333190] ? asm_exc_invalid_op+0x1a/0x20 [ 2263.337386] ? __list_add_valid_or_report+0x7d/0xa0 [ 2263.342266] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2263.347164] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2263.351891] lookup_fast+0x6d/0xf0 [ 2263.355299] walk_component+0x26/0x160 [ 2263.359056] link_path_walk+0x24d/0x380 [ 2263.362892] ? path_openat+0x80/0x9d0 [ 2263.366566] path_openat+0x8e/0x9d0 [ 2263.370066] do_filp_open+0xb8/0x120 [ 2263.373654] ? lock_release+0xc5/0x260 [ 2263.377412] ? _raw_spin_unlock+0x23/0x40 [ 2263.381430] ? alloc_fd+0x13f/0x200 [ 2263.384932] do_sys_openat2+0x249/0x290 [ 2263.388775] do_sys_open+0x4f/0x80 [ 2263.392188] do_syscall_64+0x38/0x80 [ 2263.395764] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2263.400816] RIP: 0033:0x7f48f0ef2805 [ 2263.404395] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2263.423141] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2263.430709] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2263.437842] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2263.444975] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2263.452106] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2263.459249] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2263.466395] [ 2263.468582] irq event stamp: 12843711 [ 2263.472250] hardirqs last enabled at (12843721): [] console_unlock+0x106/0x120 [ 2263.481115] hardirqs last disabled at (12843730): [] console_unlock+0xeb/0x120 [ 2263.489893] softirqs last enabled at (12843688): [] __do_softirq+0x2be/0x371 [ 2263.498590] softirqs last disabled at (12843749): [] irq_exit_rcu+0xb0/0xd0 [ 2263.507107] ---[ end trace 0000000000000000 ]--- [ 2263.511731] ------------[ cut here ]------------ [ 2263.516404] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2263.528061] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2263.537015] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2263.592541] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2263.601498] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2263.608976] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2263.614462] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2263.633210] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2263.638435] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2263.645567] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2263.652701] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005740 [ 2263.659837] R10: 00000000001e60f0 R11: ffffc90001fd79c8 R12: ffff8881d10b72e8 [ 2263.666965] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2263.674101] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2263.682186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2263.687932] CR2: 00007fb38d7a9070 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2263.695065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2263.702200] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2263.709328] Call Trace: [ 2263.711782] [ 2263.713889] ? __warn+0x85/0x180 [ 2263.717121] ? __list_add_valid_or_report+0x7d/0xa0 [ 2263.722003] ? report_bug+0xfc/0x1d0 [ 2263.725581] ? prb_read_valid+0x1b/0x20 [ 2263.729424] ? handle_bug+0x44/0x70 [ 2263.732913] ? exc_invalid_op+0x17/0x60 [ 2263.736757] ? asm_exc_invalid_op+0x1a/0x20 [ 2263.740957] ? __list_add_valid_or_report+0x7d/0xa0 [ 2263.745836] ? __list_add_valid_or_report+0x7d/0xa0 [ 2263.750715] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2263.755616] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2263.760342] lookup_fast+0x6d/0xf0 [ 2263.763749] walk_component+0x26/0x160 [ 2263.767512] link_path_walk+0x24d/0x380 [ 2263.771349] ? path_lookupat.isra.40+0x22/0x120 [ 2263.775889] path_lookupat.isra.40+0x6b/0x120 [ 2263.780253] filename_lookup+0xd4/0x190 [ 2263.784099] ? lock_acquire+0xcb/0x2b0 [ 2263.787854] ? __might_fault+0x45/0x80 [ 2263.791606] ? find_held_lock+0x2d/0x90 [ 2263.795443] ? __might_fault+0x45/0x80 [ 2263.799206] vfs_statx+0x89/0x150 [ 2263.802531] vfs_fstatat+0x5a/0xc0 [ 2263.805941] __do_sys_newstat+0x30/0x70 [ 2263.809780] ? lock_release+0xc5/0x260 [ 2263.813536] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2263.818419] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2263.823560] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2263.828700] do_syscall_64+0x38/0x80 [ 2263.832279] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2263.837330] RIP: 0033:0x7f48f0eeca79 [ 2263.840909] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2263.859655] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2263.867225] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2263.874354] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2263.881486] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2263.888620] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2263.895754] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2263.902903] [ 2263.905095] irq event stamp: 12976141 [ 2263.908763] hardirqs last enabled at (12976151): [] console_unlock+0x106/0x120 [ 2263.917631] hardirqs last disabled at (12976168): [] console_unlock+0xeb/0x120 [ 2263.926408] softirqs last enabled at (12976166): [] __do_softirq+0x2be/0x371 [ 2263.935106] softirqs last disabled at (12976187): [] irq_exit_rcu+0xb0/0xd0 [ 2263.943629] ---[ end trace 0000000000000000 ]--- [ 2263.948249] ------------[ cut here ]------------ [ 2263.952867] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2263.964515] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2263.973473] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2264.029010] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2264.037959] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2264.045439] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2264.050923] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2264.069671] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2264.074897] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2264.082028] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2264.089163] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005782 [ 2264.096295] R10: 00000000001e72b0 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2264.103427] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2264.110560] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2264.118645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.124392] CR2: 00007fb38d730000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2264.131527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2264.138658] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2264.145792] Call Trace: [ 2264.148244] [ 2264.150349] ? __warn+0x85/0x180 [ 2264.153582] ? __list_add_valid_or_report+0x7d/0xa0 [ 2264.158468] ? report_bug+0xfc/0x1d0 [ 2264.162050] ? prb_read_valid+0x1b/0x20 [ 2264.165895] ? handle_bug+0x44/0x70 [ 2264.169383] ? exc_invalid_op+0x17/0x60 [ 2264.173223] ? asm_exc_invalid_op+0x1a/0x20 [ 2264.177417] ? __list_add_valid_or_report+0x7d/0xa0 [ 2264.182298] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2264.187197] try_get_cap_refs+0x159/0x880 [ceph] [ 2264.191840] __ceph_get_caps+0xf8/0x760 [ceph] [ 2264.196321] ? ceph_read_iter+0x134/0x860 [ceph] [ 2264.200956] ceph_read_iter+0x134/0x860 [ceph] [ 2264.205428] ? selinux_file_permission+0x119/0x150 [ 2264.210225] ? vfs_read+0x20f/0x300 [ 2264.213723] vfs_read+0x20f/0x300 [ 2264.217052] ksys_pread64+0x65/0xa0 [ 2264.220543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2264.225685] do_syscall_64+0x38/0x80 [ 2264.229262] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2264.234315] RIP: 0033:0x7f48f0eeb88b [ 2264.237894] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2264.256639] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2264.264206] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2264.271339] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2264.278471] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2264.285608] R10: 000000000003f000 R11: 0000000000000246 R12: 0000000000001000 [ 2264.292738] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2264.299883] [ 2264.302070] irq event stamp: 12936625 [ 2264.305737] hardirqs last enabled at (12936635): [] console_unlock+0x106/0x120 [ 2264.314602] hardirqs last disabled at (12936644): [] console_unlock+0xeb/0x120 [ 2264.323382] softirqs last enabled at (12936592): [] __do_softirq+0x2be/0x371 [ 2264.332079] softirqs last disabled at (12936663): [] irq_exit_rcu+0xb0/0xd0 [ 2264.340596] ---[ end trace 0000000000000000 ]--- [ 2264.345216] ------------[ cut here ]------------ [ 2264.349836] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2264.361493] WARNING: CPU: 2 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2264.370452] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2264.425976] CPU: 2 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2264.434925] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2264.442405] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2264.447892] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2264.466640] RSP: 0018:ffffc90001ff7bf0 EFLAGS: 00010282 [ 2264.471872] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2264.479005] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2264.486137] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000057ba [ 2264.493269] R10: 00000000001e8258 R11: ffffc90001ff7a80 R12: ffff8881c25d1118 [ 2264.500405] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2264.507535] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2264.515621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.521367] CR2: 00007f9bd2b7d000 CR3: 0000000120cc8002 CR4: 00000000003706e0 [ 2264.528501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2264.535636] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2264.542766] Call Trace: [ 2264.545219] [ 2264.547327] ? __warn+0x85/0x180 [ 2264.550558] ? __list_add_valid_or_report+0x7d/0xa0 [ 2264.555440] ? report_bug+0xfc/0x1d0 [ 2264.559015] ? prb_read_valid+0x1b/0x20 [ 2264.562861] ? handle_bug+0x44/0x70 [ 2264.566352] ? exc_invalid_op+0x17/0x60 [ 2264.570189] ? asm_exc_invalid_op+0x1a/0x20 [ 2264.574385] ? __list_add_valid_or_report+0x7d/0xa0 [ 2264.579264] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2264.584162] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2264.588889] lookup_fast+0x6d/0xf0 [ 2264.592298] walk_component+0x26/0x160 [ 2264.596054] link_path_walk+0x24d/0x380 [ 2264.599892] ? path_openat+0x80/0x9d0 [ 2264.603562] path_openat+0x8e/0x9d0 [ 2264.607064] do_filp_open+0xb8/0x120 [ 2264.610652] ? lock_release+0xc5/0x260 [ 2264.614411] ? _raw_spin_unlock+0x23/0x40 [ 2264.618427] ? alloc_fd+0x13f/0x200 [ 2264.621926] do_sys_openat2+0x249/0x290 [ 2264.625774] do_sys_open+0x4f/0x80 [ 2264.629184] do_syscall_64+0x38/0x80 [ 2264.632765] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2264.637815] RIP: 0033:0x7f48f0eed2f2 [ 2264.641394] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2264.660141] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2264.667710] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2264.674840] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2264.681972] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2264.689105] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2264.696238] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2264.703386] [ 2264.705571] irq event stamp: 12942775 [ 2264.709237] hardirqs last enabled at (12942785): [] console_unlock+0x106/0x120 [ 2264.718103] hardirqs last disabled at (12942794): [] console_unlock+0xeb/0x120 [ 2264.726886] softirqs last enabled at (12942808): [] __do_softirq+0x2be/0x371 [ 2264.735579] softirqs last disabled at (12942819): [] irq_exit_rcu+0xb0/0xd0 [ 2264.744095] ---[ end trace 0000000000000000 ]--- [ 2264.748717] ------------[ cut here ]------------ [ 2264.753335] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2264.764978] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2264.773927] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2264.829400] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2264.838365] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2264.845845] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2264.851328] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2264.870066] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2264.875292] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2264.882418] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2264.889549] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000057f5 [ 2264.896676] R10: 00000000001e9438 R11: ffffc90001fe7a70 R12: ffff8881d12c72e8 [ 2264.903807] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2264.910930] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2264.919009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.924754] CR2: 00007f9bd2b76000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2264.931880] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2264.939010] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2264.946136] Call Trace: [ 2264.948581] [ 2264.950686] ? __warn+0x85/0x180 [ 2264.953918] ? __list_add_valid_or_report+0x7d/0xa0 [ 2264.958800] ? report_bug+0xfc/0x1d0 [ 2264.962378] ? prb_read_valid+0x1b/0x20 [ 2264.966217] ? handle_bug+0x44/0x70 [ 2264.969710] ? exc_invalid_op+0x17/0x60 [ 2264.973540] ? asm_exc_invalid_op+0x1a/0x20 [ 2264.977730] ? __list_add_valid_or_report+0x7d/0xa0 [ 2264.982608] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2264.987496] try_get_cap_refs+0x159/0x880 [ceph] [ 2264.992122] ? _raw_spin_unlock+0x23/0x40 [ 2264.996139] __ceph_get_caps+0xf8/0x760 [ceph] [ 2265.000591] ? __vfs_getxattr+0x8d/0xb0 [ 2265.004430] ? dentry_needs_remove_privs+0x31/0x50 [ 2265.009221] ? __file_remove_privs+0x89/0x110 [ 2265.013581] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2265.018297] ? lock_release+0xc5/0x260 [ 2265.022051] ? ceph_write_iter+0x260/0x770 [ceph] [ 2265.026764] ceph_write_iter+0x260/0x770 [ceph] [ 2265.031310] ? lock_is_held_type+0xaa/0x120 [ 2265.035495] vfs_write+0x324/0x4f0 [ 2265.038901] ksys_pwrite64+0x65/0xa0 [ 2265.042477] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2265.047618] do_syscall_64+0x38/0x80 [ 2265.051196] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2265.056248] RIP: 0033:0x7f48f0eeb93b [ 2265.059829] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2265.078572] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2265.086140] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2265.093273] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2265.100404] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2265.107530] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2265.114661] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2265.121792] [ 2265.123978] irq event stamp: 12887337 [ 2265.127637] hardirqs last enabled at (12887347): [] console_unlock+0x106/0x120 [ 2265.136503] hardirqs last disabled at (12887356): [] console_unlock+0xeb/0x120 [ 2265.145281] softirqs last enabled at (12887284): [] __do_softirq+0x2be/0x371 [ 2265.153975] softirqs last disabled at (12887279): [] irq_exit_rcu+0xb0/0xd0 [ 2265.162494] ---[ end trace 0000000000000000 ]--- [ 2265.167105] ------------[ cut here ]------------ [ 2265.171733] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2265.183387] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2265.192344] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2265.247865] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2265.256817] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2265.264296] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2265.269783] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2265.288530] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2265.293755] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2265.300888] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2265.308020] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005832 [ 2265.315158] R10: 00000000001ea4a0 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2265.322286] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2265.329419] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2265.337505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2265.343252] CR2: 00007fb38d730000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2265.350385] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2265.357516] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2265.364650] Call Trace: [ 2265.367103] [ 2265.369210] ? __warn+0x85/0x180 [ 2265.372444] ? __list_add_valid_or_report+0x7d/0xa0 [ 2265.377324] ? report_bug+0xfc/0x1d0 [ 2265.380900] ? prb_read_valid+0x1b/0x20 [ 2265.384745] ? handle_bug+0x44/0x70 [ 2265.388233] ? exc_invalid_op+0x17/0x60 [ 2265.392073] ? asm_exc_invalid_op+0x1a/0x20 [ 2265.396268] ? __list_add_valid_or_report+0x7d/0xa0 [ 2265.401149] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2265.406046] try_get_cap_refs+0x159/0x880 [ceph] [ 2265.410691] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2265.415251] ? wake_bit_function+0x70/0x70 [ 2265.419353] ? ceph_read_iter+0x134/0x860 [ceph] [ 2265.423984] ceph_read_iter+0x134/0x860 [ceph] [ 2265.428455] ? selinux_file_permission+0x119/0x150 [ 2265.433254] ? vfs_read+0x20f/0x300 [ 2265.436750] vfs_read+0x20f/0x300 [ 2265.440081] ksys_pread64+0x65/0xa0 [ 2265.443575] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2265.448720] do_syscall_64+0x38/0x80 [ 2265.452298] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2265.457349] RIP: 0033:0x7f48f0eeb88b [ 2265.460931] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2265.479678] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2265.487243] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2265.494375] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2265.501511] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2265.508641] R10: 000000000003f000 R11: 0000000000000246 R12: 0000000000001000 [ 2265.515772] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2265.522919] [ 2265.525108] irq event stamp: 12939349 [ 2265.528772] hardirqs last enabled at (12939359): [] console_unlock+0x106/0x120 [ 2265.537641] hardirqs last disabled at (12939368): [] console_unlock+0xeb/0x120 [ 2265.546418] softirqs last enabled at (12939220): [] __do_softirq+0x2be/0x371 [ 2265.555112] softirqs last disabled at (12939213): [] irq_exit_rcu+0xb0/0xd0 [ 2265.563631] ---[ end trace 0000000000000000 ]--- [ 2265.568252] ------------[ cut here ]------------ [ 2265.572873] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2265.584516] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2265.593472] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2265.649000] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2265.657952] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2265.665432] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2265.670917] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2265.689668] RSP: 0018:ffffc90001fd7b38 EFLAGS: 00010282 [ 2265.694891] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2265.702027] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2265.709165] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000586b [ 2265.716297] R10: 00000000001eb578 R11: ffffc90001fd79c8 R12: ffff8881d10b0750 [ 2265.723431] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2265.730562] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2265.738650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2265.744394] CR2: 00007fb38d7c0058 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2265.751528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2265.758665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2265.765794] Call Trace: [ 2265.768246] [ 2265.770353] ? __warn+0x85/0x180 [ 2265.773586] ? __list_add_valid_or_report+0x7d/0xa0 [ 2265.778466] ? report_bug+0xfc/0x1d0 [ 2265.782046] ? prb_read_valid+0x1b/0x20 [ 2265.785887] ? handle_bug+0x44/0x70 [ 2265.789383] ? exc_invalid_op+0x17/0x60 [ 2265.793227] ? asm_exc_invalid_op+0x1a/0x20 [ 2265.797421] ? __list_add_valid_or_report+0x7d/0xa0 [ 2265.802301] ? __list_add_valid_or_report+0x7d/0xa0 [ 2265.807178] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2265.812079] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2265.816806] lookup_fast+0x6d/0xf0 [ 2265.820216] walk_component+0x26/0x160 [ 2265.823969] link_path_walk+0x24d/0x380 [ 2265.827806] ? path_lookupat.isra.40+0x22/0x120 [ 2265.832348] path_lookupat.isra.40+0x6b/0x120 [ 2265.836710] filename_lookup+0xd4/0x190 [ 2265.840552] ? lock_acquire+0xcb/0x2b0 [ 2265.844307] ? __might_fault+0x45/0x80 [ 2265.848062] ? find_held_lock+0x2d/0x90 [ 2265.851900] ? __might_fault+0x45/0x80 [ 2265.855658] vfs_statx+0x89/0x150 [ 2265.858988] vfs_fstatat+0x5a/0xc0 [ 2265.862396] __do_sys_newstat+0x30/0x70 [ 2265.866234] ? lock_release+0xc5/0x260 [ 2265.869992] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2265.874873] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2265.880015] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2265.885161] do_syscall_64+0x38/0x80 [ 2265.888743] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2265.893793] RIP: 0033:0x7f48f0eeca79 [ 2265.897373] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2265.916125] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2265.923696] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2265.930828] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2265.937961] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2265.945093] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2265.952227] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2265.959378] [ 2265.961570] irq event stamp: 12980773 [ 2265.965235] hardirqs last enabled at (12980783): [] console_unlock+0x106/0x120 [ 2265.974102] hardirqs last disabled at (12980792): [] console_unlock+0xeb/0x120 [ 2265.982879] softirqs last enabled at (12980760): [] __do_softirq+0x2be/0x371 [ 2265.991573] softirqs last disabled at (12980753): [] irq_exit_rcu+0xb0/0xd0 [ 2266.000093] ---[ end trace 0000000000000000 ]--- [ 2266.004714] ------------[ cut here ]------------ [ 2266.009333] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2266.020983] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2266.029944] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2266.085487] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2266.094442] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2266.101922] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2266.107407] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2266.126159] RSP: 0018:ffffc90001fdfbf0 EFLAGS: 00010282 [ 2266.131389] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2266.138520] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2266.145654] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000058ad [ 2266.152788] R10: 00000000001ec7b0 R11: ffffc90001fdfa80 R12: ffff8881c5350750 [ 2266.159923] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2266.167052] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2266.175139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2266.180885] CR2: 00007f9bd2bc3000 CR3: 0000000179402001 CR4: 00000000003706f0 [ 2266.188016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2266.195157] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2266.202291] Call Trace: [ 2266.204745] [ 2266.206852] ? __warn+0x85/0x180 [ 2266.210085] ? __list_add_valid_or_report+0x7d/0xa0 [ 2266.214969] ? report_bug+0xfc/0x1d0 [ 2266.218551] ? prb_read_valid+0x1b/0x20 [ 2266.222394] ? handle_bug+0x44/0x70 [ 2266.225893] ? exc_invalid_op+0x17/0x60 [ 2266.229734] ? asm_exc_invalid_op+0x1a/0x20 [ 2266.233935] ? __list_add_valid_or_report+0x7d/0xa0 [ 2266.238818] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2266.243725] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2266.248450] lookup_fast+0x6d/0xf0 [ 2266.251859] walk_component+0x26/0x160 [ 2266.255614] link_path_walk+0x24d/0x380 [ 2266.259458] ? path_openat+0x80/0x9d0 [ 2266.263130] path_openat+0x8e/0x9d0 [ 2266.266634] do_filp_open+0xb8/0x120 [ 2266.270221] ? lock_release+0xc5/0x260 [ 2266.273980] ? _raw_spin_unlock+0x23/0x40 [ 2266.277998] ? alloc_fd+0x13f/0x200 [ 2266.281494] do_sys_openat2+0x249/0x290 [ 2266.285341] do_sys_open+0x4f/0x80 [ 2266.288754] do_syscall_64+0x38/0x80 [ 2266.292335] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2266.297393] RIP: 0033:0x7f48f0ef2805 [ 2266.300973] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2266.319718] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2266.327284] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2266.334418] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2266.341554] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2266.348691] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2266.355835] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2266.362980] [ 2266.365176] irq event stamp: 12850363 [ 2266.368842] hardirqs last enabled at (12850373): [] console_unlock+0x106/0x120 [ 2266.377708] hardirqs last disabled at (12850382): [] console_unlock+0xeb/0x120 [ 2266.386492] softirqs last enabled at (12850398): [] __do_softirq+0x2be/0x371 [ 2266.395189] softirqs last disabled at (12850391): [] irq_exit_rcu+0xb0/0xd0 [ 2266.403709] ---[ end trace 0000000000000000 ]--- [ 2266.408328] ------------[ cut here ]------------ [ 2266.412950] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2266.424604] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2266.433556] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2266.489082] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2266.498052] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2266.505536] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2266.511023] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2266.529768] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2266.534993] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2266.542127] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2266.549258] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000058e8 [ 2266.556394] R10: 00000000001ed930 R11: ffffc90001fcf9c8 R12: ffff888158a68750 [ 2266.563530] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2266.570667] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2266.578752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2266.584506] CR2: 00007fb38d763000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2266.591639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2266.598774] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2266.605911] Call Trace: [ 2266.608368] [ 2266.610475] ? __warn+0x85/0x180 [ 2266.613708] ? __list_add_valid_or_report+0x7d/0xa0 [ 2266.618588] ? report_bug+0xfc/0x1d0 [ 2266.622167] ? prb_read_valid+0x1b/0x20 [ 2266.626008] ? handle_bug+0x44/0x70 [ 2266.629498] ? exc_invalid_op+0x17/0x60 [ 2266.633337] ? asm_exc_invalid_op+0x1a/0x20 [ 2266.637532] ? __list_add_valid_or_report+0x7d/0xa0 [ 2266.642412] ? __list_add_valid_or_report+0x7d/0xa0 [ 2266.647290] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2266.652190] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2266.656918] lookup_fast+0x6d/0xf0 [ 2266.660327] walk_component+0x26/0x160 [ 2266.664081] link_path_walk+0x24d/0x380 [ 2266.667933] ? path_lookupat.isra.40+0x22/0x120 [ 2266.672472] path_lookupat.isra.40+0x6b/0x120 [ 2266.676840] filename_lookup+0xd4/0x190 [ 2266.680691] ? lock_acquire+0xcb/0x2b0 [ 2266.684444] ? __might_fault+0x45/0x80 [ 2266.688202] ? find_held_lock+0x2d/0x90 [ 2266.692050] ? __might_fault+0x45/0x80 [ 2266.695813] vfs_statx+0x89/0x150 [ 2266.699142] vfs_fstatat+0x5a/0xc0 [ 2266.702553] __do_sys_newstat+0x30/0x70 [ 2266.706401] ? lock_release+0xc5/0x260 [ 2266.710154] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2266.715044] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2266.720185] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2266.725329] do_syscall_64+0x38/0x80 [ 2266.728916] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2266.733968] RIP: 0033:0x7f48f0eeca79 [ 2266.737545] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2266.756295] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2266.763866] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2266.771000] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2266.778133] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2266.785283] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2266.792417] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2266.799562] [ 2266.801760] irq event stamp: 12895013 [ 2266.805425] hardirqs last enabled at (12895023): [] console_unlock+0x106/0x120 [ 2266.814291] hardirqs last disabled at (12895042): [] console_unlock+0xeb/0x120 [ 2266.823088] softirqs last enabled at (12895060): [] __do_softirq+0x2be/0x371 [ 2266.831787] softirqs last disabled at (12895071): [] irq_exit_rcu+0xb0/0xd0 [ 2266.840315] ---[ end trace 0000000000000000 ]--- [ 2266.844937] ------------[ cut here ]------------ [ 2266.849562] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2266.861214] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2266.870176] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2266.925699] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2266.934657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2266.942136] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2266.947620] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2266.966367] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2266.971592] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2266.978730] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2266.985859] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000592a [ 2266.992995] R10: 00000000001eec10 R11: ffffc90001ff7b18 R12: ffff8881d1329ae0 [ 2267.000123] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2267.007258] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2267.015345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2267.021091] CR2: 00007f9bd2bc3000 CR3: 0000000120cc8003 CR4: 00000000003706f0 [ 2267.028222] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2267.035356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2267.042489] Call Trace: [ 2267.044942] [ 2267.047049] ? __warn+0x85/0x180 [ 2267.050288] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.055172] ? report_bug+0xfc/0x1d0 [ 2267.058756] ? prb_read_valid+0x1b/0x20 [ 2267.062599] ? handle_bug+0x44/0x70 [ 2267.066087] ? exc_invalid_op+0x17/0x60 [ 2267.069928] ? asm_exc_invalid_op+0x1a/0x20 [ 2267.074125] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.079013] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.083890] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2267.088792] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2267.093519] lookup_fast+0x6d/0xf0 [ 2267.096925] path_openat+0xe4/0x9d0 [ 2267.100427] do_filp_open+0xb8/0x120 [ 2267.104014] ? lock_release+0xc5/0x260 [ 2267.107775] ? _raw_spin_unlock+0x23/0x40 [ 2267.111788] ? alloc_fd+0x13f/0x200 [ 2267.115287] do_sys_openat2+0x249/0x290 [ 2267.119144] do_sys_open+0x4f/0x80 [ 2267.122554] do_syscall_64+0x38/0x80 [ 2267.126132] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2267.131185] RIP: 0033:0x7f48f0eed2f2 [ 2267.134764] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2267.153515] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2267.161084] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2267.168219] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2267.175350] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2267.182484] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2267.189619] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2267.196761] [ 2267.198950] irq event stamp: 12944793 [ 2267.202616] hardirqs last enabled at (12944803): [] console_unlock+0x106/0x120 [ 2267.211481] hardirqs last disabled at (12944812): [] console_unlock+0xeb/0x120 [ 2267.220267] softirqs last enabled at (12944828): [] __do_softirq+0x2be/0x371 [ 2267.228964] softirqs last disabled at (12944821): [] irq_exit_rcu+0xb0/0xd0 [ 2267.237488] ---[ end trace 0000000000000000 ]--- [ 2267.242111] ------------[ cut here ]------------ [ 2267.246729] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2267.258378] WARNING: CPU: 4 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2267.267337] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2267.322858] CPU: 4 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2267.331813] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2267.339294] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2267.344779] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2267.363524] RSP: 0018:ffffc90001fdfc88 EFLAGS: 00010286 [ 2267.368750] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2267.375885] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2267.383017] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005963 [ 2267.390160] R10: 00000000001efb40 R11: ffffc90001fdfb18 R12: ffff8881d1146920 [ 2267.397298] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2267.404437] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2267.412528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2267.418273] CR2: 00007fb38d763000 CR3: 0000000179402004 CR4: 00000000003706e0 [ 2267.425407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2267.432539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2267.439671] Call Trace: [ 2267.442127] [ 2267.444232] ? __warn+0x85/0x180 [ 2267.447465] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.452345] ? report_bug+0xfc/0x1d0 [ 2267.455930] ? prb_read_valid+0x1b/0x20 [ 2267.459779] ? handle_bug+0x44/0x70 [ 2267.463271] ? exc_invalid_op+0x17/0x60 [ 2267.467113] ? asm_exc_invalid_op+0x1a/0x20 [ 2267.471308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.476186] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.481068] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2267.485966] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2267.490694] lookup_fast+0x6d/0xf0 [ 2267.494110] path_openat+0xe4/0x9d0 [ 2267.497608] do_filp_open+0xb8/0x120 [ 2267.501198] ? lock_release+0xc5/0x260 [ 2267.504954] ? _raw_spin_unlock+0x23/0x40 [ 2267.508972] ? alloc_fd+0x13f/0x200 [ 2267.512469] do_sys_openat2+0x249/0x290 [ 2267.516316] do_sys_open+0x4f/0x80 [ 2267.519731] do_syscall_64+0x38/0x80 [ 2267.523319] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2267.528378] RIP: 0033:0x7f48f0ef2805 [ 2267.531957] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2267.550706] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2267.558284] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2267.565420] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2267.572552] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2267.579684] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2267.586828] R13: 00007ffc53d60520 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2267.593974] [ 2267.596170] irq event stamp: 12852479 [ 2267.599835] hardirqs last enabled at (12852489): [] console_unlock+0x106/0x120 [ 2267.608699] hardirqs last disabled at (12852498): [] console_unlock+0xeb/0x120 [ 2267.617485] softirqs last enabled at (12852516): [] __do_softirq+0x2be/0x371 [ 2267.626191] softirqs last disabled at (12852527): [] irq_exit_rcu+0xb0/0xd0 [ 2267.634711] ---[ end trace 0000000000000000 ]--- [ 2267.639329] ------------[ cut here ]------------ [ 2267.643947] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2267.655596] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2267.664551] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2267.720029] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2267.728987] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2267.736466] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2267.741954] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2267.760698] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2267.765926] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2267.773057] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2267.780190] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000599c [ 2267.787326] R10: 00000000001f0c70 R11: ffffc90001fe7a70 R12: ffff8881d12c72e8 [ 2267.794457] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2267.801588] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2267.809675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2267.815421] CR2: 00007f9bd2b76000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2267.822553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2267.829686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2267.836819] Call Trace: [ 2267.839272] [ 2267.841381] ? __warn+0x85/0x180 [ 2267.844621] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.849503] ? report_bug+0xfc/0x1d0 [ 2267.853090] ? prb_read_valid+0x1b/0x20 [ 2267.856934] ? handle_bug+0x44/0x70 [ 2267.860430] ? exc_invalid_op+0x17/0x60 [ 2267.864270] ? asm_exc_invalid_op+0x1a/0x20 [ 2267.868472] ? __list_add_valid_or_report+0x7d/0xa0 [ 2267.873354] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2267.878250] try_get_cap_refs+0x159/0x880 [ceph] [ 2267.882894] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2267.887441] ? __file_remove_privs+0x89/0x110 [ 2267.891804] ? wake_bit_function+0x70/0x70 [ 2267.895907] ? ceph_write_iter+0x260/0x770 [ceph] [ 2267.900638] ceph_write_iter+0x260/0x770 [ceph] [ 2267.905190] ? lock_is_held_type+0xaa/0x120 [ 2267.909387] vfs_write+0x324/0x4f0 [ 2267.912802] ksys_pwrite64+0x65/0xa0 [ 2267.916383] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2267.921523] do_syscall_64+0x38/0x80 [ 2267.925103] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2267.930161] RIP: 0033:0x7f48f0eeb93b [ 2267.933741] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2267.952487] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2267.960053] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2267.967184] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2267.974320] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2267.981454] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2267.988586] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2267.995730] [ 2267.997920] irq event stamp: 12893689 [ 2268.001589] hardirqs last enabled at (12893711): [] console_unlock+0x106/0x120 [ 2268.010461] hardirqs last disabled at (12893720): [] console_unlock+0xeb/0x120 [ 2268.019248] softirqs last enabled at (12893708): [] __do_softirq+0x2be/0x371 [ 2268.027941] softirqs last disabled at (12893697): [] irq_exit_rcu+0xb0/0xd0 [ 2268.036459] ---[ end trace 0000000000000000 ]--- [ 2268.041081] ------------[ cut here ]------------ [ 2268.045697] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2268.057347] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2268.066302] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2268.121830] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2268.130786] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2268.138260] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2268.143746] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2268.162497] RSP: 0018:ffffc90001ff7c88 EFLAGS: 00010286 [ 2268.167727] RAX: 0000000000000000 RBX: ffff8881d12c3ab0 RCX: 0000000000000000 [ 2268.174861] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2268.181995] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000059d5 [ 2268.189126] R10: 00000000001f1b68 R11: ffffc90001ff7b18 R12: ffff8881d12c4200 [ 2268.196259] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2268.203393] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2268.211479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2268.217223] CR2: 00007f9bd2bc3000 CR3: 0000000120cc8003 CR4: 00000000003706f0 [ 2268.224358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2268.231500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2268.238632] Call Trace: [ 2268.241085] [ 2268.243190] ? __warn+0x85/0x180 [ 2268.246424] ? __list_add_valid_or_report+0x7d/0xa0 [ 2268.251305] ? report_bug+0xfc/0x1d0 [ 2268.254890] ? prb_read_valid+0x1b/0x20 [ 2268.258737] ? handle_bug+0x44/0x70 [ 2268.262236] ? exc_invalid_op+0x17/0x60 [ 2268.266081] ? asm_exc_invalid_op+0x1a/0x20 [ 2268.270275] ? __list_add_valid_or_report+0x7d/0xa0 [ 2268.275156] ? __list_add_valid_or_report+0x7d/0xa0 [ 2268.280042] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2268.284942] ceph_open+0x16b/0x440 [ceph] [ 2268.288974] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2268.294129] do_dentry_open+0x172/0x530 [ 2268.297977] path_openat+0x743/0x9d0 [ 2268.301569] do_filp_open+0xb8/0x120 [ 2268.305157] ? lock_release+0xc5/0x260 [ 2268.308915] ? _raw_spin_unlock+0x23/0x40 [ 2268.312933] ? alloc_fd+0x13f/0x200 [ 2268.316430] do_sys_openat2+0x249/0x290 [ 2268.320278] do_sys_open+0x4f/0x80 [ 2268.323691] do_syscall_64+0x38/0x80 [ 2268.327282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2268.332336] RIP: 0033:0x7f48f0eed2f2 [ 2268.335917] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2268.354663] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2268.362232] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2268.369361] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2268.376493] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2268.383628] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2268.390764] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2268.397906] [ 2268.400095] irq event stamp: 12947687 [ 2268.403761] hardirqs last enabled at (12947697): [] console_unlock+0x106/0x120 [ 2268.412627] hardirqs last disabled at (12947706): [] console_unlock+0xeb/0x120 [ 2268.421405] softirqs last enabled at (12947664): [] __do_softirq+0x2be/0x371 [ 2268.430100] softirqs last disabled at (12947725): [] irq_exit_rcu+0xb0/0xd0 [ 2268.438619] ---[ end trace 0000000000000000 ]--- [ 2268.443238] ------------[ cut here ]------------ [ 2268.447858] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2268.459506] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2268.468459] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2268.523992] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2268.532962] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2268.540446] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2268.545935] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2268.564689] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2268.569921] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2268.577054] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2268.584187] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005a0f [ 2268.591320] R10: 00000000001f2d48 R11: ffffc90001fcf9c8 R12: ffff8881c5350750 [ 2268.598454] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2268.605585] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2268.613672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2268.619417] CR2: 00007fb38d763000 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2268.626551] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2268.633686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2268.640824] Call Trace: [ 2268.643280] [ 2268.645385] ? __warn+0x85/0x180 [ 2268.648617] ? __list_add_valid_or_report+0x7d/0xa0 [ 2268.653498] ? report_bug+0xfc/0x1d0 [ 2268.657086] ? prb_read_valid+0x1b/0x20 [ 2268.660927] ? handle_bug+0x44/0x70 [ 2268.664429] ? exc_invalid_op+0x17/0x60 [ 2268.668274] ? asm_exc_invalid_op+0x1a/0x20 [ 2268.672468] ? __list_add_valid_or_report+0x7d/0xa0 [ 2268.677347] ? __list_add_valid_or_report+0x7d/0xa0 [ 2268.682225] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2268.687126] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2268.691854] lookup_fast+0x6d/0xf0 [ 2268.695267] walk_component+0x26/0x160 [ 2268.699025] link_path_walk+0x24d/0x380 [ 2268.702861] ? path_lookupat.isra.40+0x22/0x120 [ 2268.707399] path_lookupat.isra.40+0x6b/0x120 [ 2268.711765] filename_lookup+0xd4/0x190 [ 2268.715609] ? lock_acquire+0xcb/0x2b0 [ 2268.719371] ? __might_fault+0x45/0x80 [ 2268.723126] ? find_held_lock+0x2d/0x90 [ 2268.726969] ? __might_fault+0x45/0x80 [ 2268.730735] vfs_statx+0x89/0x150 [ 2268.734061] vfs_fstatat+0x5a/0xc0 [ 2268.737471] __do_sys_newstat+0x30/0x70 [ 2268.741307] ? lock_release+0xc5/0x260 [ 2268.745066] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2268.749956] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2268.755097] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2268.760242] do_syscall_64+0x38/0x80 [ 2268.763825] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2268.768877] RIP: 0033:0x7f48f0eeca79 [ 2268.772455] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2268.791207] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2268.798783] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2268.805922] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2268.813052] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2268.820184] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2268.827319] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2268.834465] [ 2268.836665] irq event stamp: 12898371 [ 2268.840335] hardirqs last enabled at (12898381): [] console_unlock+0x106/0x120 [ 2268.849214] hardirqs last disabled at (12898404): [] console_unlock+0xeb/0x120 [ 2268.857999] softirqs last enabled at (12898402): [] __do_softirq+0x2be/0x371 [ 2268.866693] softirqs last disabled at (12898423): [] irq_exit_rcu+0xb0/0xd0 [ 2268.875219] ---[ end trace 0000000000000000 ]--- [ 2268.879840] ------------[ cut here ]------------ [ 2268.884457] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2268.896104] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2268.905058] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2268.960586] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2268.969541] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2268.977019] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2268.982504] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2269.001250] RSP: 0018:ffffc90001ff7aa0 EFLAGS: 00010286 [ 2269.006477] RAX: 0000000000000000 RBX: ffff8881d12c3ab0 RCX: 0000000000000000 [ 2269.013609] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2269.020743] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005a51 [ 2269.027880] R10: 00000000001f40e8 R11: ffffc90001ff7930 R12: ffff8881d12c4200 [ 2269.035018] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2269.042150] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2269.050237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2269.055983] CR2: 00007f9bd2bc3000 CR3: 0000000120cc8003 CR4: 00000000003706f0 [ 2269.063115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2269.070247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2269.077381] Call Trace: [ 2269.079833] [ 2269.081940] ? __warn+0x85/0x180 [ 2269.085172] ? __list_add_valid_or_report+0x7d/0xa0 [ 2269.090054] ? report_bug+0xfc/0x1d0 [ 2269.093642] ? prb_read_valid+0x1b/0x20 [ 2269.097485] ? handle_bug+0x44/0x70 [ 2269.100981] ? exc_invalid_op+0x17/0x60 [ 2269.104822] ? asm_exc_invalid_op+0x1a/0x20 [ 2269.109015] ? __list_add_valid_or_report+0x7d/0xa0 [ 2269.113897] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2269.118796] ceph_check_caps+0x999/0xae0 [ceph] [ 2269.123381] ? lock_release+0xc5/0x260 [ 2269.127144] ceph_open+0x3cc/0x440 [ceph] [ 2269.131183] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2269.136333] do_dentry_open+0x172/0x530 [ 2269.140174] path_openat+0x743/0x9d0 [ 2269.143758] do_filp_open+0xb8/0x120 [ 2269.147349] ? lock_release+0xc5/0x260 [ 2269.151106] ? _raw_spin_unlock+0x23/0x40 [ 2269.155122] ? alloc_fd+0x13f/0x200 [ 2269.158623] do_sys_openat2+0x249/0x290 [ 2269.162468] do_sys_open+0x4f/0x80 [ 2269.165881] do_syscall_64+0x38/0x80 [ 2269.169467] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2269.174518] RIP: 0033:0x7f48f0eed2f2 [ 2269.178098] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2269.196844] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2269.204411] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2269.211544] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2269.218675] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2269.225811] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2269.232942] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f5 [ 2269.240088] [ 2269.242286] irq event stamp: 12949687 [ 2269.245951] hardirqs last enabled at (12949697): [] console_unlock+0x106/0x120 [ 2269.254818] hardirqs last disabled at (12949706): [] console_unlock+0xeb/0x120 [ 2269.263595] softirqs last enabled at (12949654): [] __do_softirq+0x2be/0x371 [ 2269.272289] softirqs last disabled at (12949649): [] irq_exit_rcu+0xb0/0xd0 [ 2269.280808] ---[ end trace 0000000000000000 ]--- [ 2269.285431] ------------[ cut here ]------------ [ 2269.290048] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2269.301699] WARNING: CPU: 4 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2269.310658] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2269.366192] CPU: 4 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2269.375147] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2269.382627] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2269.388121] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2269.406873] RSP: 0018:ffffc90001fdfd50 EFLAGS: 00010286 [ 2269.412104] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2269.419235] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2269.426369] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005a8c [ 2269.433500] R10: 00000000001f5030 R11: ffffc90001fdfbe0 R12: ffff8881d1144bc8 [ 2269.440637] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2269.447767] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2269.455854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2269.461599] CR2: 00007fb38d763000 CR3: 0000000179402004 CR4: 00000000003706e0 [ 2269.468737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2269.475882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2269.483016] Call Trace: [ 2269.485467] [ 2269.487574] ? __warn+0x85/0x180 [ 2269.490808] ? __list_add_valid_or_report+0x7d/0xa0 [ 2269.495688] ? report_bug+0xfc/0x1d0 [ 2269.499282] ? prb_read_valid+0x1b/0x20 [ 2269.503128] ? handle_bug+0x44/0x70 [ 2269.506629] ? exc_invalid_op+0x17/0x60 [ 2269.510472] ? asm_exc_invalid_op+0x1a/0x20 [ 2269.514668] ? __list_add_valid_or_report+0x7d/0xa0 [ 2269.519558] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2269.524472] ceph_readdir+0xc0/0x1670 [ceph] [ 2269.528763] ? wrap_directory_iterator+0x38/0x70 [ 2269.533389] ? wrap_directory_iterator+0x30/0x70 [ 2269.538019] ? lock_is_held_type+0xaa/0x120 [ 2269.542216] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2269.547117] wrap_directory_iterator+0x47/0x70 [ 2269.551571] iterate_dir+0x8d/0x170 [ 2269.555067] __x64_sys_getdents64+0x87/0x140 [ 2269.559345] ? __x64_sys_getdents64+0x140/0x140 [ 2269.563885] do_syscall_64+0x38/0x80 [ 2269.567465] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2269.572522] RIP: 0033:0x7f48f0ec480b [ 2269.576105] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2269.594854] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2269.602426] RAX: ffffffffffffffda RBX: 000000000207e100 RCX: 00007f48f0ec480b [ 2269.609563] RDX: 0000000000010000 RSI: 000000000207e130 RDI: 0000000000000019 [ 2269.616699] RBP: 000000000207e130 R08: 00000000000000db R09: 0000000000000003 [ 2269.623834] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2269.630967] R13: 0000000000000002 R14: 00007f48f183dac8 R15: 00000000020767c0 [ 2269.638124] [ 2269.640322] irq event stamp: 12854717 [ 2269.643993] hardirqs last enabled at (12854727): [] console_unlock+0x106/0x120 [ 2269.652859] hardirqs last disabled at (12854736): [] console_unlock+0xeb/0x120 [ 2269.661638] softirqs last enabled at (12854714): [] __do_softirq+0x2be/0x371 [ 2269.670331] softirqs last disabled at (12854707): [] irq_exit_rcu+0xb0/0xd0 [ 2269.678853] ---[ end trace 0000000000000000 ]--- [ 2269.683469] ------------[ cut here ]------------ [ 2269.688090] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2269.699735] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2269.708689] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2269.764213] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2269.773170] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2269.780651] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2269.786136] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2269.804887] RSP: 0018:ffffc90001ff7bd0 EFLAGS: 00010282 [ 2269.810118] RAX: 0000000000000000 RBX: ffff8881d12c3ab0 RCX: 0000000000000000 [ 2269.817251] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2269.824382] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005ac4 [ 2269.831518] R10: 00000000001f5da0 R11: ffffc90001ff7a60 R12: ffff8881d12c4200 [ 2269.838650] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2269.845782] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2269.853869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2269.859613] CR2: 00007f9bd2bc3000 CR3: 0000000120cc8003 CR4: 00000000003706f0 [ 2269.866748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2269.873880] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2269.881011] Call Trace: [ 2269.883466] [ 2269.885573] ? __warn+0x85/0x180 [ 2269.888805] ? __list_add_valid_or_report+0x7d/0xa0 [ 2269.893685] ? report_bug+0xfc/0x1d0 [ 2269.897264] ? prb_read_valid+0x1b/0x20 [ 2269.901111] ? handle_bug+0x44/0x70 [ 2269.904603] ? exc_invalid_op+0x17/0x60 [ 2269.908444] ? asm_exc_invalid_op+0x1a/0x20 [ 2269.912640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2269.917529] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2269.922427] try_get_cap_refs+0x433/0x880 [ceph] [ 2269.927080] __ceph_get_caps+0xf8/0x760 [ceph] [ 2269.931560] ? ceph_read_iter+0x134/0x860 [ceph] [ 2269.936195] ceph_read_iter+0x134/0x860 [ceph] [ 2269.940667] ? selinux_file_permission+0x119/0x150 [ 2269.945464] ? vfs_read+0x20f/0x300 [ 2269.948962] vfs_read+0x20f/0x300 [ 2269.952292] ksys_pread64+0x65/0xa0 [ 2269.955789] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2269.960935] do_syscall_64+0x38/0x80 [ 2269.964519] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2269.969571] RIP: 0033:0x7f48f0eeb88b [ 2269.973149] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2269.991900] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2269.999472] RAX: ffffffffffffffda RBX: 0000000002075870 RCX: 00007f48f0eeb88b [ 2270.006604] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000009 [ 2270.013737] RBP: 00000000000029f5 R08: 0000000000000063 R09: 0000000000000003 [ 2270.020869] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2270.028003] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2270.035149] [ 2270.037344] irq event stamp: 12951587 [ 2270.041009] hardirqs last enabled at (12951597): [] console_unlock+0x106/0x120 [ 2270.049876] hardirqs last disabled at (12951606): [] console_unlock+0xeb/0x120 [ 2270.058658] softirqs last enabled at (12951564): [] __do_softirq+0x2be/0x371 [ 2270.067353] softirqs last disabled at (12951625): [] irq_exit_rcu+0xb0/0xd0 [ 2270.075878] ---[ end trace 0000000000000000 ]--- [ 2270.080497] ------------[ cut here ]------------ [ 2270.085126] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2270.096845] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2270.105853] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2270.161375] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2270.170339] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2270.177817] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2270.183302] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2270.202053] RSP: 0018:ffffc90001fcfb38 EFLAGS: 00010282 [ 2270.207289] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2270.214426] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2270.221563] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005afc [ 2270.228700] R10: 00000000001f6da0 R11: ffffc90001fcf9c8 R12: ffff8881d106b838 [ 2270.235836] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2270.242974] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2270.251060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2270.256806] CR2: 00007fb38d763000 CR3: 000000011aeae001 CR4: 00000000003706e0 [ 2270.263938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2270.271077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2270.278214] Call Trace: [ 2270.280666] [ 2270.282774] ? __warn+0x85/0x180 [ 2270.286004] ? __list_add_valid_or_report+0x7d/0xa0 [ 2270.290885] ? report_bug+0xfc/0x1d0 [ 2270.294471] ? prb_read_valid+0x1b/0x20 [ 2270.298314] ? handle_bug+0x44/0x70 [ 2270.301808] ? exc_invalid_op+0x17/0x60 [ 2270.305653] ? asm_exc_invalid_op+0x1a/0x20 [ 2270.309847] ? __list_add_valid_or_report+0x7d/0xa0 [ 2270.314728] ? __list_add_valid_or_report+0x7d/0xa0 [ 2270.319615] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2270.324514] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2270.329244] lookup_fast+0x6d/0xf0 [ 2270.332655] walk_component+0x26/0x160 [ 2270.336415] link_path_walk+0x24d/0x380 [ 2270.340260] ? path_lookupat.isra.40+0x22/0x120 [ 2270.344796] path_lookupat.isra.40+0x6b/0x120 [ 2270.349164] filename_lookup+0xd4/0x190 [ 2270.353016] ? lock_acquire+0xcb/0x2b0 [ 2270.356768] ? __might_fault+0x45/0x80 [ 2270.360524] ? find_held_lock+0x2d/0x90 [ 2270.364373] ? __might_fault+0x45/0x80 [ 2270.368138] vfs_statx+0x89/0x150 [ 2270.371470] vfs_fstatat+0x5a/0xc0 [ 2270.374877] __do_sys_newstat+0x30/0x70 [ 2270.378715] ? lock_release+0xc5/0x260 [ 2270.382472] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2270.387354] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2270.392503] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2270.397648] do_syscall_64+0x38/0x80 [ 2270.401232] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2270.406283] RIP: 0033:0x7f48f0eeca79 [ 2270.409861] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2270.428610] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2270.436184] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2270.443315] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2270.450448] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2270.457581] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2270.464717] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2270.471860] [ 2270.474056] irq event stamp: 12900833 [ 2270.477722] hardirqs last enabled at (12900843): [] console_unlock+0x106/0x120 [ 2270.486590] hardirqs last disabled at (12900862): [] console_unlock+0xeb/0x120 [ 2270.495372] softirqs last enabled at (12900878): [] __do_softirq+0x2be/0x371 [ 2270.504077] softirqs last disabled at (12900891): [] irq_exit_rcu+0xb0/0xd0 [ 2270.512601] ---[ end trace 0000000000000000 ]--- [ 2270.517219] ------------[ cut here ]------------ [ 2270.521837] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2270.533487] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2270.542446] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2270.597972] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2270.606928] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2270.614407] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2270.619894] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2270.638639] RSP: 0018:ffffc90001fdfb38 EFLAGS: 00010282 [ 2270.643867] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2270.650999] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2270.658131] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005b3e [ 2270.665267] R10: 00000000001f7f40 R11: ffffc90001fdf9c8 R12: ffff888158a68750 [ 2270.672398] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2270.679531] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2270.687617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2270.693363] CR2: 00007f9bd2bc3000 CR3: 0000000179402004 CR4: 00000000003706f0 [ 2270.700497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2270.707628] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2270.714759] Call Trace: [ 2270.717215] [ 2270.719319] ? __warn+0x85/0x180 [ 2270.722553] ? __list_add_valid_or_report+0x7d/0xa0 [ 2270.727432] ? report_bug+0xfc/0x1d0 [ 2270.731015] ? prb_read_valid+0x1b/0x20 [ 2270.734864] ? handle_bug+0x44/0x70 [ 2270.738361] ? exc_invalid_op+0x17/0x60 [ 2270.742202] ? asm_exc_invalid_op+0x1a/0x20 [ 2270.746404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2270.751285] ? __list_add_valid_or_report+0x7d/0xa0 [ 2270.756163] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2270.761063] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2270.765790] lookup_fast+0x6d/0xf0 [ 2270.769200] walk_component+0x26/0x160 [ 2270.772954] link_path_walk+0x24d/0x380 [ 2270.776797] ? path_lookupat.isra.40+0x22/0x120 [ 2270.781337] path_lookupat.isra.40+0x6b/0x120 [ 2270.785702] filename_lookup+0xd4/0x190 [ 2270.789548] ? lock_acquire+0xcb/0x2b0 [ 2270.793303] ? __might_fault+0x45/0x80 [ 2270.797064] ? find_held_lock+0x2d/0x90 [ 2270.800909] ? __might_fault+0x45/0x80 [ 2270.804670] vfs_statx+0x89/0x150 [ 2270.807997] vfs_fstatat+0x5a/0xc0 [ 2270.811408] __do_sys_newstat+0x30/0x70 [ 2270.815253] ? lock_release+0xc5/0x260 [ 2270.819010] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2270.823894] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2270.829035] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2270.834174] do_syscall_64+0x38/0x80 [ 2270.837752] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2270.842805] RIP: 0033:0x7f48f0eeca79 [ 2270.846384] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2270.865133] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2270.872705] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2270.879837] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2270.886970] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2270.894103] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2270.901238] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2270.908382] [ 2270.910571] irq event stamp: 12856835 [ 2270.914237] hardirqs last enabled at (12856845): [] console_unlock+0x106/0x120 [ 2270.923103] hardirqs last disabled at (12856854): [] console_unlock+0xeb/0x120 [ 2270.931885] softirqs last enabled at (12856868): [] __do_softirq+0x2be/0x371 [ 2270.940584] softirqs last disabled at (12856863): [] irq_exit_rcu+0xb0/0xd0 [ 2270.949108] ---[ end trace 0000000000000000 ]--- [ 2270.953732] ------------[ cut here ]------------ [ 2270.958363] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2270.970015] WARNING: CPU: 4 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2270.978966] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2271.034433] CPU: 4 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2271.043394] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2271.050877] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2271.056361] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2271.075112] RSP: 0018:ffffc90001fcfb98 EFLAGS: 00010282 [ 2271.080332] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2271.087457] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2271.094586] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005b80 [ 2271.101715] R10: 00000000001f91e8 R11: ffffc90001fcfa28 R12: ffff8881d106cbc8 [ 2271.108848] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2271.115984] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2271.124071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2271.129811] CR2: 00007fb38d763000 CR3: 000000011aeae001 CR4: 00000000003706e0 [ 2271.136935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2271.144061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2271.151194] Call Trace: [ 2271.153645] [ 2271.155744] ? __warn+0x85/0x180 [ 2271.158977] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.163857] ? report_bug+0xfc/0x1d0 [ 2271.167435] ? prb_read_valid+0x1b/0x20 [ 2271.171281] ? handle_bug+0x44/0x70 [ 2271.174767] ? exc_invalid_op+0x17/0x60 [ 2271.178607] ? asm_exc_invalid_op+0x1a/0x20 [ 2271.182798] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.187673] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.192552] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2271.197444] ceph_d_revalidate+0x4bb/0x6d0 [ceph] [ 2271.202157] lookup_fast+0x6d/0xf0 [ 2271.205562] walk_component+0x26/0x160 [ 2271.209316] path_lookupat.isra.40+0x58/0x120 [ 2271.213673] filename_lookup+0xd4/0x190 [ 2271.217516] ? lock_acquire+0xcb/0x2b0 [ 2271.221263] ? __might_fault+0x45/0x80 [ 2271.225021] ? find_held_lock+0x2d/0x90 [ 2271.228858] ? __might_fault+0x45/0x80 [ 2271.232613] vfs_statx+0x89/0x150 [ 2271.235931] vfs_fstatat+0x5a/0xc0 [ 2271.239337] __do_sys_newstat+0x30/0x70 [ 2271.243175] ? lock_release+0xc5/0x260 [ 2271.246929] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2271.251808] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2271.256945] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2271.262087] do_syscall_64+0x38/0x80 [ 2271.265666] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2271.270717] RIP: 0033:0x7f48f0eeca79 [ 2271.274296] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2271.293043] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2271.300611] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2271.307741] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2271.314865] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2271.321995] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2271.329124] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2271.336262] [ 2271.338451] irq event stamp: 12903303 [ 2271.342115] hardirqs last enabled at (12903313): [] console_unlock+0x106/0x120 [ 2271.350981] hardirqs last disabled at (12903322): [] console_unlock+0xeb/0x120 [ 2271.359762] softirqs last enabled at (12903340): [] __do_softirq+0x2be/0x371 [ 2271.368454] softirqs last disabled at (12903351): [] irq_exit_rcu+0xb0/0xd0 [ 2271.376972] ---[ end trace 0000000000000000 ]--- [ 2271.381593] ------------[ cut here ]------------ [ 2271.386213] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2271.397863] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2271.406566] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2271.462096] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2271.471229] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2271.478709] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2271.484219] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2271.489705] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2271.508453] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2271.513678] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2271.520811] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2271.527946] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005bc0 [ 2271.535076] R10: 00000000001fa378 R11: ffffc9000030b6a8 R12: ffff8881d12c72e8 [ 2271.542209] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2271.549342] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2271.557428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2271.563172] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2271.570310] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2271.577440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2271.584574] Call Trace: [ 2271.587026] [ 2271.589138] ? __warn+0x85/0x180 [ 2271.592376] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.597264] ? report_bug+0xfc/0x1d0 [ 2271.600849] ? prb_read_valid+0x1b/0x20 [ 2271.604693] ? handle_bug+0x44/0x70 [ 2271.608190] ? exc_invalid_op+0x17/0x60 [ 2271.612031] ? asm_exc_invalid_op+0x1a/0x20 [ 2271.616227] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.621115] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.626000] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2271.630903] ceph_check_caps+0x999/0xae0 [ceph] [ 2271.635459] ? __lock_acquire+0xbfa/0x1e20 [ 2271.639594] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2271.644312] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2271.649055] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2271.653892] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2271.658273] ? lock_release+0xc5/0x260 [ 2271.662034] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2271.666749] ceph_con_process_message+0x74/0x150 [libceph] [ 2271.672253] ? calc_signature+0xdb/0x100 [libceph] [ 2271.677070] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2271.682411] ? process_scheduled_works+0x22a/0x510 [ 2271.687211] ? process_scheduled_works+0x22a/0x510 [ 2271.692007] ceph_con_workfn+0x344/0x750 [libceph] [ 2271.696822] ? process_scheduled_works+0x22a/0x510 [ 2271.701618] process_scheduled_works+0x2a4/0x510 [ 2271.706249] worker_thread+0x188/0x340 [ 2271.710011] ? rescuer_thread+0x310/0x310 [ 2271.714030] kthread+0xf5/0x130 [ 2271.717185] ? kthread_complete_and_exit+0x20/0x20 [ 2271.721982] ret_from_fork+0x31/0x40 [ 2271.725565] ? kthread_complete_and_exit+0x20/0x20 [ 2271.730358] ret_from_fork_asm+0x11/0x20 [ 2271.734308] [ 2271.736501] irq event stamp: 10509593 [ 2271.740169] hardirqs last enabled at (10509603): [] console_unlock+0x106/0x120 [ 2271.749033] hardirqs last disabled at (10509612): [] console_unlock+0xeb/0x120 [ 2271.757814] softirqs last enabled at (10509550): [] __do_softirq+0x2be/0x371 [ 2271.766507] softirqs last disabled at (10509545): [] irq_exit_rcu+0xb0/0xd0 [ 2271.775027] ---[ end trace 0000000000000000 ]--- [ 2271.779646] ------------[ cut here ]------------ [ 2271.784263] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2271.795912] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2271.804872] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2271.860397] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2271.869356] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2271.876834] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2271.882322] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2271.901117] RSP: 0018:ffffc90001fd7a78 EFLAGS: 00010286 [ 2271.906345] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2271.913477] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2271.920610] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005bfe [ 2271.927745] R10: 00000000001fb528 R11: ffffc90001fd7908 R12: ffff8881c2cefcb0 [ 2271.934880] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2271.942009] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2271.950094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2271.955841] CR2: 00007fb38d7e0020 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2271.962976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2271.970107] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2271.977239] Call Trace: [ 2271.979694] [ 2271.981799] ? __warn+0x85/0x180 [ 2271.985032] ? __list_add_valid_or_report+0x7d/0xa0 [ 2271.989914] ? report_bug+0xfc/0x1d0 [ 2271.993502] ? prb_read_valid+0x1b/0x20 [ 2271.997351] ? handle_bug+0x44/0x70 [ 2272.000849] ? exc_invalid_op+0x17/0x60 [ 2272.004689] ? asm_exc_invalid_op+0x1a/0x20 [ 2272.008885] ? __list_add_valid_or_report+0x7d/0xa0 [ 2272.013765] ? __list_add_valid_or_report+0x7d/0xa0 [ 2272.018651] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2272.023553] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2272.028804] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2272.033622] ceph_lookup+0xf9/0x3f0 [ceph] [ 2272.037742] __lookup_slow+0xff/0x1a0 [ 2272.041428] walk_component+0x83/0x160 [ 2272.045184] path_lookupat.isra.40+0x58/0x120 [ 2272.049551] filename_lookup+0xd4/0x190 [ 2272.053405] ? lock_acquire+0xcb/0x2b0 [ 2272.057158] ? __might_fault+0x45/0x80 [ 2272.060913] ? find_held_lock+0x2d/0x90 [ 2272.064764] ? __might_fault+0x45/0x80 [ 2272.068528] vfs_statx+0x89/0x150 [ 2272.071854] vfs_fstatat+0x5a/0xc0 [ 2272.075264] __do_sys_newstat+0x30/0x70 [ 2272.079101] ? lock_release+0xc5/0x260 [ 2272.082859] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2272.087742] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2272.092884] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2272.098036] do_syscall_64+0x38/0x80 [ 2272.101621] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2272.106679] RIP: 0033:0x7f48f0eeca79 [ 2272.110258] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2272.129015] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2272.136588] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2272.143722] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2272.150855] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2272.157987] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2272.165123] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2272.172267] [ 2272.174462] irq event stamp: 12994475 [ 2272.178128] hardirqs last enabled at (12994485): [] console_unlock+0x106/0x120 [ 2272.186995] hardirqs last disabled at (12994494): [] console_unlock+0xeb/0x120 [ 2272.195780] softirqs last enabled at (12994512): [] __do_softirq+0x2be/0x371 [ 2272.204475] softirqs last disabled at (12994503): [] irq_exit_rcu+0xb0/0xd0 [ 2272.212994] ---[ end trace 0000000000000000 ]--- [ 2272.217615] ------------[ cut here ]------------ [ 2272.222234] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2272.233882] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2272.242846] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2272.298323] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2272.307282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2272.314760] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2272.320248] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2272.338991] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2272.344220] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2272.351350] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2272.358485] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005c40 [ 2272.365618] R10: 00000000001fc758 R11: ffffc90001fe7a70 R12: ffff8881d12c72e8 [ 2272.372750] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2272.379881] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2272.387969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2272.393715] CR2: 00007f9bd2b76000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2272.400847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2272.407981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2272.415114] Call Trace: [ 2272.417566] [ 2272.419675] ? __warn+0x85/0x180 [ 2272.422915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2272.427796] ? report_bug+0xfc/0x1d0 [ 2272.431384] ? prb_read_valid+0x1b/0x20 [ 2272.435225] ? handle_bug+0x44/0x70 [ 2272.438722] ? exc_invalid_op+0x17/0x60 [ 2272.442562] ? asm_exc_invalid_op+0x1a/0x20 [ 2272.446756] ? __list_add_valid_or_report+0x7d/0xa0 [ 2272.451640] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2272.456535] try_get_cap_refs+0x159/0x880 [ceph] [ 2272.461178] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2272.465728] ? __file_remove_privs+0x89/0x110 [ 2272.470089] ? wake_bit_function+0x70/0x70 [ 2272.474192] ? ceph_write_iter+0x260/0x770 [ceph] [ 2272.478921] ceph_write_iter+0x260/0x770 [ceph] [ 2272.483477] ? lock_is_held_type+0xaa/0x120 [ 2272.487671] vfs_write+0x324/0x4f0 [ 2272.491090] ksys_pwrite64+0x65/0xa0 [ 2272.494679] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2272.499828] do_syscall_64+0x38/0x80 [ 2272.503413] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2272.508473] RIP: 0033:0x7f48f0eeb93b [ 2272.512054] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2272.530799] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2272.538364] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2272.545496] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2272.552631] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2272.559764] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2272.566897] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2272.574041] [ 2272.576230] irq event stamp: 12903733 [ 2272.579896] hardirqs last enabled at (12903743): [] console_unlock+0x106/0x120 [ 2272.588762] hardirqs last disabled at (12903752): [] console_unlock+0xeb/0x120 [ 2272.597542] softirqs last enabled at (12903700): [] __do_softirq+0x2be/0x371 [ 2272.606234] softirqs last disabled at (12903695): [] irq_exit_rcu+0xb0/0xd0 [ 2272.614752] ---[ end trace 0000000000000000 ]--- [ 2272.619374] ------------[ cut here ]------------ [ 2272.623991] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2272.635636] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2272.644591] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2272.700052] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2272.709002] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2272.716483] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2272.721969] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2272.740716] RSP: 0018:ffffc90001fcfb20 EFLAGS: 00010282 [ 2272.745941] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2272.753074] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2272.760207] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005c79 [ 2272.767341] R10: 00000000001fd780 R11: ffffc90001fcf9b0 R12: ffff8881d106cbc8 [ 2272.774473] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2272.781606] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2272.789693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2272.795437] CR2: 00007f9bd2bc3000 CR3: 000000011aeae001 CR4: 00000000003706f0 [ 2272.802569] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2272.809694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2272.816827] Call Trace: [ 2272.819279] [ 2272.821377] ? __warn+0x85/0x180 [ 2272.824611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2272.829491] ? report_bug+0xfc/0x1d0 [ 2272.833069] ? prb_read_valid+0x1b/0x20 [ 2272.836909] ? handle_bug+0x44/0x70 [ 2272.840400] ? exc_invalid_op+0x17/0x60 [ 2272.844240] ? asm_exc_invalid_op+0x1a/0x20 [ 2272.848431] ? __list_add_valid_or_report+0x7d/0xa0 [ 2272.853308] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2272.858197] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2272.863440] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2272.868240] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2272.872958] lookup_fast+0x6d/0xf0 [ 2272.876369] walk_component+0x26/0x160 [ 2272.880132] path_lookupat.isra.40+0x58/0x120 [ 2272.884491] filename_lookup+0xd4/0x190 [ 2272.888332] ? lock_acquire+0xcb/0x2b0 [ 2272.892082] ? __might_fault+0x45/0x80 [ 2272.895836] ? find_held_lock+0x2d/0x90 [ 2272.899673] ? __might_fault+0x45/0x80 [ 2272.903430] vfs_statx+0x89/0x150 [ 2272.906748] vfs_fstatat+0x5a/0xc0 [ 2272.910154] __do_sys_newstat+0x30/0x70 [ 2272.914000] ? lock_release+0xc5/0x260 [ 2272.917755] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2272.922631] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2272.927773] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2272.932912] do_syscall_64+0x38/0x80 [ 2272.936489] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2272.941543] RIP: 0033:0x7f48f0eeca79 [ 2272.945121] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2272.963871] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2272.971434] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2272.978566] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2272.985691] RBP: 00007ffc53d60520 R08: 000000000000000b R09: 00000000020767c0 [ 2272.992815] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2272.999939] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2273.007079] [ 2273.009264] irq event stamp: 12907225 [ 2273.012923] hardirqs last enabled at (12907235): [] console_unlock+0x106/0x120 [ 2273.021788] hardirqs last disabled at (12907244): [] console_unlock+0xeb/0x120 [ 2273.030569] softirqs last enabled at (12907258): [] __do_softirq+0x2be/0x371 [ 2273.039260] softirqs last disabled at (12907253): [] irq_exit_rcu+0xb0/0xd0 [ 2273.047782] ---[ end trace 0000000000000000 ]--- [ 2273.052400] ------------[ cut here ]------------ [ 2273.057069] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2273.068721] WARNING: CPU: 1 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2273.077677] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2273.133205] CPU: 1 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2273.142168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2273.149647] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2273.155140] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2273.173886] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2273.179112] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2273.186244] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2273.193378] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005cba [ 2273.200516] R10: 00000000001fe940 R11: ffffc90001fe7a70 R12: ffff8881d12c72e8 [ 2273.207650] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2273.214785] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2273.222871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2273.228617] CR2: 00007fb38cc0dc30 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2273.235750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2273.242882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2273.250014] Call Trace: [ 2273.252468] [ 2273.254575] ? __warn+0x85/0x180 [ 2273.257812] ? __list_add_valid_or_report+0x7d/0xa0 [ 2273.262697] ? report_bug+0xfc/0x1d0 [ 2273.266275] ? prb_read_valid+0x1b/0x20 [ 2273.270118] ? handle_bug+0x44/0x70 [ 2273.273617] ? exc_invalid_op+0x17/0x60 [ 2273.277457] ? asm_exc_invalid_op+0x1a/0x20 [ 2273.281661] ? __list_add_valid_or_report+0x7d/0xa0 [ 2273.286549] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2273.291448] try_get_cap_refs+0x159/0x880 [ceph] [ 2273.296100] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2273.300656] ? __file_remove_privs+0x89/0x110 [ 2273.305016] ? wake_bit_function+0x70/0x70 [ 2273.309122] ? ceph_write_iter+0x260/0x770 [ceph] [ 2273.313847] ceph_write_iter+0x260/0x770 [ceph] [ 2273.318405] ? lock_is_held_type+0xaa/0x120 [ 2273.322599] vfs_write+0x324/0x4f0 [ 2273.326018] ksys_pwrite64+0x65/0xa0 [ 2273.329602] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2273.334745] do_syscall_64+0x38/0x80 [ 2273.338333] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2273.343393] RIP: 0033:0x7f48f0eeb93b [ 2273.346972] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2273.365717] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2273.373282] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2273.380417] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2273.387550] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2273.394681] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2273.401814] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2273.408962] [ 2273.411158] irq event stamp: 12905631 [ 2273.414822] hardirqs last enabled at (12905641): [] console_unlock+0x106/0x120 [ 2273.423690] hardirqs last disabled at (12905650): [] console_unlock+0xeb/0x120 [ 2273.432470] softirqs last enabled at (12905332): [] __do_softirq+0x2be/0x371 [ 2273.441163] softirqs last disabled at (12905327): [] irq_exit_rcu+0xb0/0xd0 [ 2273.449689] ---[ end trace 0000000000000000 ]--- [ 2273.454310] ------------[ cut here ]------------ [ 2273.458927] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2273.470574] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2273.479269] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2273.534749] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2273.543870] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2273.551350] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2273.556844] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2273.562331] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2273.581077] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2273.586302] RAX: 0000000000000000 RBX: ffff8881e4d0b0e8 RCX: 0000000000000000 [ 2273.593435] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2273.600570] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005cf3 [ 2273.607703] R10: 00000000001ffa28 R11: ffffc9000030b828 R12: ffff8881e4d0b838 [ 2273.614840] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2273.621986] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2273.630080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2273.635827] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2273.642957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2273.650090] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2273.657224] Call Trace: [ 2273.659676] [ 2273.661784] ? __warn+0x85/0x180 [ 2273.665017] ? __list_add_valid_or_report+0x7d/0xa0 [ 2273.669895] ? report_bug+0xfc/0x1d0 [ 2273.673475] ? prb_read_valid+0x1b/0x20 [ 2273.677315] ? handle_bug+0x44/0x70 [ 2273.680807] ? exc_invalid_op+0x17/0x60 [ 2273.684647] ? asm_exc_invalid_op+0x1a/0x20 [ 2273.688837] ? __list_add_valid_or_report+0x7d/0xa0 [ 2273.693713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2273.698591] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2273.703483] ceph_add_cap+0x20b/0x620 [ceph] [ 2273.707777] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2273.712505] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2273.717833] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2273.722295] ceph_con_process_message+0x74/0x150 [libceph] [ 2273.727791] ? calc_signature+0xdb/0x100 [libceph] [ 2273.732603] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2273.737934] ? process_scheduled_works+0x22a/0x510 [ 2273.742726] ? process_scheduled_works+0x22a/0x510 [ 2273.747513] ceph_con_workfn+0x344/0x750 [libceph] [ 2273.752319] ? process_scheduled_works+0x22a/0x510 [ 2273.757109] process_scheduled_works+0x2a4/0x510 [ 2273.761735] worker_thread+0x188/0x340 [ 2273.765491] ? rescuer_thread+0x310/0x310 [ 2273.769503] kthread+0xf5/0x130 [ 2273.772650] ? kthread_complete_and_exit+0x20/0x20 [ 2273.777444] ret_from_fork+0x31/0x40 [ 2273.781022] ? kthread_complete_and_exit+0x20/0x20 [ 2273.785815] ret_from_fork_asm+0x11/0x20 [ 2273.789748] [ 2273.791943] irq event stamp: 10514613 [ 2273.795606] hardirqs last enabled at (10514623): [] console_unlock+0x106/0x120 [ 2273.804474] hardirqs last disabled at (10514632): [] console_unlock+0xeb/0x120 [ 2273.813252] softirqs last enabled at (10514610): [] __do_softirq+0x2be/0x371 [ 2273.821945] softirqs last disabled at (10514605): [] irq_exit_rcu+0xb0/0xd0 [ 2273.830465] ---[ end trace 0000000000000000 ]--- [ 2273.835100] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [ 2273.835394] ------------[ cut here ]------------ [ 2273.844431] libceph: mon1 (1)172.21.15.130:6789 session established [ 2273.847400] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2273.865245] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2273.873958] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2273.929478] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2273.938621] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2273.946114] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2273.951636] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2273.957147] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2273.975911] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2273.981154] RAX: 0000000000000000 RBX: ffff8881d12c09c8 RCX: 0000000000000000 [ 2273.988306] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2273.995458] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005d2f [ 2274.002603] R10: 0000000000200b08 R11: ffffc9000030b6a8 R12: ffff8881d12c1118 [ 2274.009745] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2274.016896] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2274.025001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2274.030762] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2274.037912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2274.045062] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2274.052211] Call Trace: [ 2274.054674] [ 2274.056791] ? __warn+0x85/0x180 [ 2274.060044] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.064938] ? report_bug+0xfc/0x1d0 [ 2274.068534] ? prb_read_valid+0x1b/0x20 [ 2274.072393] ? handle_bug+0x44/0x70 [ 2274.075901] ? exc_invalid_op+0x17/0x60 [ 2274.079757] ? asm_exc_invalid_op+0x1a/0x20 [ 2274.083965] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.088868] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.093763] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2274.098672] ceph_check_caps+0x999/0xae0 [ceph] [ 2274.103241] ? __lock_acquire+0xbfa/0x1e20 [ 2274.107391] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2274.112114] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2274.116867] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2274.121712] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2274.126104] ? lock_release+0xc5/0x260 [ 2274.129882] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2274.134606] ceph_con_process_message+0x74/0x150 [libceph] [ 2274.140142] ? calc_signature+0xdb/0x100 [libceph] [ 2274.144972] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2274.150327] ? process_scheduled_works+0x22a/0x510 [ 2274.155149] ? process_scheduled_works+0x22a/0x510 [ 2274.159961] ceph_con_workfn+0x344/0x750 [libceph] [ 2274.164783] ? process_scheduled_works+0x22a/0x510 [ 2274.169596] process_scheduled_works+0x2a4/0x510 [ 2274.174242] worker_thread+0x188/0x340 [ 2274.178006] ? rescuer_thread+0x310/0x310 [ 2274.182019] kthread+0xf5/0x130 [ 2274.185168] ? kthread_complete_and_exit+0x20/0x20 [ 2274.189972] ret_from_fork+0x31/0x40 [ 2274.193556] ? kthread_complete_and_exit+0x20/0x20 [ 2274.198357] ret_from_fork_asm+0x11/0x20 [ 2274.202296] [ 2274.204483] irq event stamp: 10519767 [ 2274.208150] hardirqs last enabled at (10519777): [] console_unlock+0x106/0x120 [ 2274.217017] hardirqs last disabled at (10519786): [] console_unlock+0xeb/0x120 [ 2274.225796] softirqs last enabled at (10519724): [] __do_softirq+0x2be/0x371 [ 2274.234488] softirqs last disabled at (10519719): [] irq_exit_rcu+0xb0/0xd0 [ 2274.243009] ---[ end trace 0000000000000000 ]--- [ 2274.247627] ------------[ cut here ]------------ [ 2274.252245] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2274.263891] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2274.272844] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2274.328316] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2274.337266] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2274.344744] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2274.350231] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2274.368977] RSP: 0018:ffffc90001ff7b28 EFLAGS: 00010286 [ 2274.374204] RAX: 0000000000000000 RBX: ffff8881d12c3ab0 RCX: 0000000000000000 [ 2274.381336] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2274.388468] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005d6d [ 2274.395593] R10: 0000000000201d68 R11: ffffc90001ff79b8 R12: ffff8881d12c4200 [ 2274.402725] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2274.409849] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2274.417937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2274.423682] CR2: 00007f9bd2b76000 CR3: 0000000120cc8004 CR4: 00000000003706e0 [ 2274.430816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2274.437948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2274.445079] Call Trace: [ 2274.447526] [ 2274.449632] ? __warn+0x85/0x180 [ 2274.452863] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.457743] ? report_bug+0xfc/0x1d0 [ 2274.461323] ? prb_read_valid+0x1b/0x20 [ 2274.465164] ? handle_bug+0x44/0x70 [ 2274.468655] ? exc_invalid_op+0x17/0x60 [ 2274.472495] ? asm_exc_invalid_op+0x1a/0x20 [ 2274.476685] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.481560] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.486440] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2274.491331] ceph_check_caps+0x999/0xae0 [ceph] [ 2274.495869] ? lock_release+0xc5/0x260 [ 2274.499626] ? __lock_acquire+0xbfa/0x1e20 [ 2274.503724] ? __lock_acquire+0xbfa/0x1e20 [ 2274.507821] ? lock_acquire+0xcb/0x2b0 [ 2274.511574] ? __ceph_put_cap_refs+0x39/0x4c0 [ceph] [ 2274.516547] ? find_held_lock+0x2d/0x90 [ 2274.520387] ? __ceph_put_cap_refs+0x15c/0x4c0 [ceph] [ 2274.525451] ? lock_release+0xc5/0x260 [ 2274.529204] __ceph_put_cap_refs+0x4a4/0x4c0 [ceph] [ 2274.534093] ceph_read_iter+0x26e/0x860 [ceph] [ 2274.538549] ? selinux_file_permission+0x119/0x150 [ 2274.543339] ? vfs_read+0x20f/0x300 [ 2274.546829] vfs_read+0x20f/0x300 [ 2274.550156] ksys_pread64+0x65/0xa0 [ 2274.553650] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2274.558790] do_syscall_64+0x38/0x80 [ 2274.562369] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2274.567422] RIP: 0033:0x7f48f0eeb88b [ 2274.571001] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2274.589749] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2274.597312] RAX: ffffffffffffffda RBX: 0000000002075870 RCX: 00007f48f0eeb88b [ 2274.604437] RDX: 0000000000001000 RSI: 000000000207e200 RDI: 0000000000000009 [ 2274.611572] RBP: 00000000000029f5 R08: 0000000000000063 R09: 0000000000000003 [ 2274.618702] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000 [ 2274.625826] R13: 0000000000001000 R14: 000000000207e200 R15: 00007f48f183e140 [ 2274.632958] [ 2274.635145] irq event stamp: 12956029 [ 2274.638809] hardirqs last enabled at (12956039): [] console_unlock+0x106/0x120 [ 2274.647675] hardirqs last disabled at (12956048): [] console_unlock+0xeb/0x120 [ 2274.656457] softirqs last enabled at (12956062): [] __do_softirq+0x2be/0x371 [ 2274.665150] softirqs last disabled at (12956057): [] irq_exit_rcu+0xb0/0xd0 [ 2274.673668] ---[ end trace 0000000000000000 ]--- [ 2274.678288] ------------[ cut here ]------------ [ 2274.682909] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2274.694561] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2274.703261] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2274.758766] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2274.767900] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2274.775384] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2274.780897] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2274.786382] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2274.805131] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2274.810354] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2274.817506] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2274.824641] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005dad [ 2274.831773] R10: 0000000000202eb0 R11: ffffc9000030b828 R12: ffff8881d12d1118 [ 2274.838902] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2274.846037] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2274.854123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2274.859869] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2274.867001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2274.874135] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2274.881268] Call Trace: [ 2274.883722] [ 2274.885827] ? __warn+0x85/0x180 [ 2274.889064] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.893949] ? report_bug+0xfc/0x1d0 [ 2274.897539] ? prb_read_valid+0x1b/0x20 [ 2274.901385] ? handle_bug+0x44/0x70 [ 2274.904877] ? exc_invalid_op+0x17/0x60 [ 2274.908716] ? asm_exc_invalid_op+0x1a/0x20 [ 2274.912909] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.917798] ? __list_add_valid_or_report+0x7d/0xa0 [ 2274.922679] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2274.927577] ceph_add_cap+0x20b/0x620 [ceph] [ 2274.931874] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2274.936598] ? lock_acquire+0xcb/0x2b0 [ 2274.940358] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2274.944815] ? __mutex_lock+0x97/0xce0 [ 2274.948565] ? __mutex_lock+0xaec/0xce0 [ 2274.952410] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2274.956967] ? lock_downgrade+0xfd/0x250 [ 2274.960906] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2274.965296] ceph_con_process_message+0x74/0x150 [libceph] [ 2274.970796] ? calc_signature+0xdb/0x100 [libceph] [ 2274.975621] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2274.980952] ? process_scheduled_works+0x22a/0x510 [ 2274.985751] ? process_scheduled_works+0x22a/0x510 [ 2274.990546] ceph_con_workfn+0x344/0x750 [libceph] [ 2274.995363] ? process_scheduled_works+0x22a/0x510 [ 2275.000159] process_scheduled_works+0x2a4/0x510 [ 2275.004788] worker_thread+0x188/0x340 [ 2275.008540] ? rescuer_thread+0x310/0x310 [ 2275.012554] kthread+0xf5/0x130 [ 2275.015702] ? kthread_complete_and_exit+0x20/0x20 [ 2275.020511] ret_from_fork+0x31/0x40 [ 2275.024087] ? kthread_complete_and_exit+0x20/0x20 [ 2275.028883] ret_from_fork_asm+0x11/0x20 [ 2275.032819] [ 2275.035010] irq event stamp: 10521927 [ 2275.038675] hardirqs last enabled at (10521937): [] console_unlock+0x106/0x120 [ 2275.047542] hardirqs last disabled at (10521946): [] console_unlock+0xeb/0x120 [ 2275.056321] softirqs last enabled at (10521874): [] __do_softirq+0x2be/0x371 [ 2275.065014] softirqs last disabled at (10521869): [] irq_exit_rcu+0xb0/0xd0 [ 2275.073532] ---[ end trace 0000000000000000 ]--- [ 2275.078167] ------------[ cut here ]------------ [ 2275.082791] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2275.094436] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2275.103129] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2275.158670] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2275.167803] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2275.175281] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2275.180794] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2275.186278] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2275.205027] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2275.210251] RAX: 0000000000000000 RBX: ffff8881c3b74478 RCX: 0000000000000000 [ 2275.217384] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2275.224521] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005dec [ 2275.231651] R10: 0000000000203fa0 R11: ffffc9000030b828 R12: ffff8881c3b74bc8 [ 2275.238781] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2275.245916] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2275.254002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2275.259746] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2275.266881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2275.274014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2275.281146] Call Trace: [ 2275.283601] [ 2275.285706] ? __warn+0x85/0x180 [ 2275.288942] ? __list_add_valid_or_report+0x7d/0xa0 [ 2275.293827] ? report_bug+0xfc/0x1d0 [ 2275.297408] ? prb_read_valid+0x1b/0x20 [ 2275.301257] ? handle_bug+0x44/0x70 [ 2275.304758] ? exc_invalid_op+0x17/0x60 [ 2275.308595] ? asm_exc_invalid_op+0x1a/0x20 [ 2275.312788] ? __list_add_valid_or_report+0x7d/0xa0 [ 2275.317669] ? __list_add_valid_or_report+0x7d/0xa0 [ 2275.322549] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2275.327448] ceph_add_cap+0x20b/0x620 [ceph] [ 2275.331752] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2275.336488] ceph_fill_trace+0x143/0xf30 [ceph] [ 2275.341053] ? __mutex_lock+0x97/0xce0 [ 2275.344803] ? __mutex_lock+0xaec/0xce0 [ 2275.348647] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2275.353197] ? lock_downgrade+0xfd/0x250 [ 2275.357149] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2275.361535] ceph_con_process_message+0x74/0x150 [libceph] [ 2275.367044] ? calc_signature+0xdb/0x100 [libceph] [ 2275.371858] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2275.377206] ? process_scheduled_works+0x22a/0x510 [ 2275.382019] ? process_scheduled_works+0x22a/0x510 [ 2275.386811] ceph_con_workfn+0x344/0x750 [libceph] [ 2275.391628] ? process_scheduled_works+0x22a/0x510 [ 2275.396422] process_scheduled_works+0x2a4/0x510 [ 2275.401052] worker_thread+0x188/0x340 [ 2275.404805] ? rescuer_thread+0x310/0x310 [ 2275.408817] kthread+0xf5/0x130 [ 2275.411963] ? kthread_complete_and_exit+0x20/0x20 [ 2275.416761] ret_from_fork+0x31/0x40 [ 2275.420344] ? kthread_complete_and_exit+0x20/0x20 [ 2275.425138] ret_from_fork_asm+0x11/0x20 [ 2275.429075] [ 2275.431265] irq event stamp: 10522985 [ 2275.434929] hardirqs last enabled at (10522995): [] console_unlock+0x106/0x120 [ 2275.443798] hardirqs last disabled at (10523004): [] console_unlock+0xeb/0x120 [ 2275.452576] softirqs last enabled at (10522942): [] __do_softirq+0x2be/0x371 [ 2275.461270] softirqs last disabled at (10522937): [] irq_exit_rcu+0xb0/0xd0 [ 2275.469795] ---[ end trace 0000000000000000 ]--- [ 2275.474454] ------------[ cut here ]------------ [ 2275.479078] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2275.490728] WARNING: CPU: 3 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2275.499689] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2275.555230] CPU: 3 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2275.564190] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2275.571685] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2275.577171] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2275.595916] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2275.601143] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2275.608276] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2275.615407] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005e2a [ 2275.622542] R10: 00000000002051c0 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2275.629673] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2275.636806] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2275.644892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2275.650638] CR2: 00007fb38cc89000 CR3: 000000017eb7a005 CR4: 00000000003706e0 [ 2275.657771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2275.664905] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2275.672038] Call Trace: [ 2275.674505] [ 2275.676615] ? __warn+0x85/0x180 [ 2275.679848] ? __list_add_valid_or_report+0x7d/0xa0 [ 2275.684726] ? report_bug+0xfc/0x1d0 [ 2275.688304] ? prb_read_valid+0x1b/0x20 [ 2275.692149] ? handle_bug+0x44/0x70 [ 2275.695645] ? exc_invalid_op+0x17/0x60 [ 2275.699488] ? asm_exc_invalid_op+0x1a/0x20 [ 2275.703681] ? __list_add_valid_or_report+0x7d/0xa0 [ 2275.708561] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2275.713461] try_get_cap_refs+0x159/0x880 [ceph] [ 2275.718111] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2275.722672] ? wake_bit_function+0x70/0x70 [ 2275.726779] ? ceph_read_iter+0x134/0x860 [ceph] [ 2275.731416] ceph_read_iter+0x134/0x860 [ceph] [ 2275.735886] ? selinux_file_permission+0x119/0x150 [ 2275.740683] ? vfs_read+0x20f/0x300 [ 2275.744180] vfs_read+0x20f/0x300 [ 2275.747526] ksys_pread64+0x65/0xa0 [ 2275.751018] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2275.756160] do_syscall_64+0x38/0x80 [ 2275.759737] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2275.764790] RIP: 0033:0x7f48f0eeb88b [ 2275.768370] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2275.787114] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2275.794682] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2275.801813] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2275.808946] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2275.816084] R10: 000000000003f000 R11: 0000000000000246 R12: 0000000000001000 [ 2275.823232] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2275.830388] [ 2275.832580] irq event stamp: 12940371 [ 2275.836247] hardirqs last enabled at (12940381): [] console_unlock+0x106/0x120 [ 2275.845114] hardirqs last disabled at (12940390): [] console_unlock+0xeb/0x120 [ 2275.853892] softirqs last enabled at (12940368): [] __do_softirq+0x2be/0x371 [ 2275.862585] softirqs last disabled at (12940363): [] irq_exit_rcu+0xb0/0xd0 [ 2275.871106] ---[ end trace 0000000000000000 ]--- [ 2275.875762] ------------[ cut here ]------------ [ 2275.880380] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2275.892023] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2275.900979] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2275.956473] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2275.965446] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2275.972922] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2275.978406] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2275.997153] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [ 2276.002384] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [ 2276.009511] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2276.016635] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005e63 [ 2276.023770] R10: 0000000000206328 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [ 2276.030902] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2276.038034] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2276.046119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.051864] CR2: 00007fb38d763000 CR3: 000000017eb7a004 CR4: 00000000003706e0 [ 2276.058994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2276.066124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2276.073256] Call Trace: [ 2276.075699] [ 2276.077798] ? __warn+0x85/0x180 [ 2276.081031] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.085911] ? report_bug+0xfc/0x1d0 [ 2276.089491] ? prb_read_valid+0x1b/0x20 [ 2276.093330] ? handle_bug+0x44/0x70 [ 2276.096821] ? exc_invalid_op+0x17/0x60 [ 2276.100661] ? asm_exc_invalid_op+0x1a/0x20 [ 2276.104851] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.109726] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2276.114617] try_get_cap_refs+0x159/0x880 [ceph] [ 2276.119248] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2276.123789] ? wake_bit_function+0x70/0x70 [ 2276.127888] ? ceph_read_iter+0x134/0x860 [ceph] [ 2276.132511] ceph_read_iter+0x134/0x860 [ceph] [ 2276.136970] ? selinux_file_permission+0x119/0x150 [ 2276.141762] ? vfs_read+0x20f/0x300 [ 2276.145252] vfs_read+0x20f/0x300 [ 2276.148577] ksys_pread64+0x65/0xa0 [ 2276.152063] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2276.157207] do_syscall_64+0x38/0x80 [ 2276.160784] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2276.165836] RIP: 0033:0x7f48f0eeb88b [ 2276.169414] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2276.188162] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [ 2276.195729] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [ 2276.202859] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [ 2276.209983] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [ 2276.217108] R10: 000000000003f000 R11: 0000000000000246 R12: 0000000000001000 [ 2276.224234] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [ 2276.231372] [ 2276.233563] irq event stamp: 12941483 [ 2276.237225] hardirqs last enabled at (12941493): [] console_unlock+0x106/0x120 [ 2276.246091] hardirqs last disabled at (12941502): [] console_unlock+0xeb/0x120 [ 2276.254870] softirqs last enabled at (12941480): [] __do_softirq+0x2be/0x371 [ 2276.263563] softirqs last disabled at (12941473): [] irq_exit_rcu+0xb0/0xd0 [ 2276.272081] ---[ end trace 0000000000000000 ]--- [ 2276.276695] ------------[ cut here ]------------ [ 2276.281314] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2276.292980] WARNING: CPU: 1 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2276.301934] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2276.357479] CPU: 1 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2276.366432] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2276.373912] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2276.379415] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2276.398161] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2276.403453] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2276.410652] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2276.417782] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005e9c [ 2276.424932] R10: 00000000002072f8 R11: ffffc90001fe7a70 R12: ffff8881d12c72e8 [ 2276.432083] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2276.439250] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2276.447337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.453082] CR2: 00007f9bd2083000 CR3: 0000000108c08006 CR4: 00000000003706e0 [ 2276.460250] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2276.467389] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2276.474533] Call Trace: [ 2276.477005] [ 2276.479128] ? __warn+0x85/0x180 [ 2276.482379] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.487266] ? report_bug+0xfc/0x1d0 [ 2276.490844] ? prb_read_valid+0x1b/0x20 [ 2276.494723] ? handle_bug+0x44/0x70 [ 2276.498219] ? exc_invalid_op+0x17/0x60 [ 2276.502075] ? asm_exc_invalid_op+0x1a/0x20 [ 2276.506273] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.511156] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2276.516077] try_get_cap_refs+0x159/0x880 [ceph] [ 2276.520740] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2276.525298] ? __file_remove_privs+0x89/0x110 [ 2276.529663] ? wake_bit_function+0x70/0x70 [ 2276.533803] ? ceph_write_iter+0x260/0x770 [ceph] [ 2276.538537] ceph_write_iter+0x260/0x770 [ceph] [ 2276.543132] ? lock_is_held_type+0xaa/0x120 [ 2276.547323] vfs_write+0x324/0x4f0 [ 2276.550761] ksys_pwrite64+0x65/0xa0 [ 2276.554346] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2276.559489] do_syscall_64+0x38/0x80 [ 2276.563075] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2276.568144] RIP: 0033:0x7f48f0eeb93b [ 2276.571740] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2276.590503] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2276.598070] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2276.605221] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2276.612377] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2276.619513] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2276.626644] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2276.633825] [ 2276.636014] irq event stamp: 12907457 [ 2276.639698] hardirqs last enabled at (12907467): [] console_unlock+0x106/0x120 [ 2276.648581] hardirqs last disabled at (12907476): [] console_unlock+0xeb/0x120 [ 2276.657378] softirqs last enabled at (12907226): [] __do_softirq+0x2be/0x371 [ 2276.666070] softirqs last disabled at (12907221): [] irq_exit_rcu+0xb0/0xd0 [ 2276.674607] ---[ end trace 0000000000000000 ]--- [ 2276.679256] ------------[ cut here ]------------ [ 2276.683887] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2276.695560] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2276.704268] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2276.759741] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2276.768904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2276.776380] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2276.781874] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2276.787377] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2276.806122] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2276.811367] RAX: 0000000000000000 RBX: ffff8881d12c3ab0 RCX: 0000000000000000 [ 2276.818499] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2276.825634] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005ed5 [ 2276.832784] R10: 0000000000208478 R11: ffffc9000030b6a8 R12: ffff8881d12c4200 [ 2276.839933] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2276.847084] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2276.855187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.860931] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2276.868073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2276.875225] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2276.882374] Call Trace: [ 2276.884818] [ 2276.886933] ? __warn+0x85/0x180 [ 2276.890183] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.895080] ? report_bug+0xfc/0x1d0 [ 2276.898678] ? prb_read_valid+0x1b/0x20 [ 2276.902545] ? handle_bug+0x44/0x70 [ 2276.906053] ? exc_invalid_op+0x17/0x60 [ 2276.909909] ? asm_exc_invalid_op+0x1a/0x20 [ 2276.914134] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.919010] ? __list_add_valid_or_report+0x7d/0xa0 [ 2276.923905] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2276.928813] ceph_check_caps+0x999/0xae0 [ceph] [ 2276.933374] ? find_held_lock+0x2d/0x90 [ 2276.937245] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2276.941952] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2276.946689] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2276.951505] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2276.955879] ? lock_release+0xc5/0x260 [ 2276.959650] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2276.964376] ceph_con_process_message+0x74/0x150 [libceph] [ 2276.969875] ? calc_signature+0xdb/0x100 [libceph] [ 2276.974695] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2276.980034] ? process_scheduled_works+0x22a/0x510 [ 2276.984844] ? process_scheduled_works+0x22a/0x510 [ 2276.989651] ceph_con_workfn+0x344/0x750 [libceph] [ 2276.994472] ? process_scheduled_works+0x22a/0x510 [ 2276.999264] process_scheduled_works+0x2a4/0x510 [ 2277.003904] worker_thread+0x188/0x340 [ 2277.007654] ? rescuer_thread+0x310/0x310 [ 2277.011683] kthread+0xf5/0x130 [ 2277.014846] ? kthread_complete_and_exit+0x20/0x20 [ 2277.019659] ret_from_fork+0x31/0x40 [ 2277.023260] ? kthread_complete_and_exit+0x20/0x20 [ 2277.028046] ret_from_fork_asm+0x11/0x20 [ 2277.032013] [ 2277.034199] irq event stamp: 10528719 [ 2277.037874] hardirqs last enabled at (10528729): [] console_unlock+0x106/0x120 [ 2277.046759] hardirqs last disabled at (10528738): [] console_unlock+0xeb/0x120 [ 2277.055555] softirqs last enabled at (10528676): [] __do_softirq+0x2be/0x371 [ 2277.064264] softirqs last disabled at (10528671): [] irq_exit_rcu+0xb0/0xd0 [ 2277.072785] ---[ end trace 0000000000000000 ]--- [ 2277.077585] ------------[ cut here ]------------ [ 2277.082208] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2277.093888] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2277.102583] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2277.158077] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2277.167220] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2277.174716] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2277.180228] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2277.185730] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2277.204477] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2277.209704] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2277.216854] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2277.224006] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005f13 [ 2277.231153] R10: 0000000000209578 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2277.238294] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2277.245428] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2277.253515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2277.259264] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2277.266394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2277.273542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2277.280685] Call Trace: [ 2277.283155] [ 2277.285278] ? __warn+0x85/0x180 [ 2277.288513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2277.293391] ? report_bug+0xfc/0x1d0 [ 2277.296968] ? prb_read_valid+0x1b/0x20 [ 2277.300827] ? handle_bug+0x44/0x70 [ 2277.304336] ? exc_invalid_op+0x17/0x60 [ 2277.308175] ? asm_exc_invalid_op+0x1a/0x20 [ 2277.312384] ? __list_add_valid_or_report+0x7d/0xa0 [ 2277.317269] ? __list_add_valid_or_report+0x7d/0xa0 [ 2277.322146] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2277.327056] ceph_add_cap+0x20b/0x620 [ceph] [ 2277.331357] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2277.336097] ? lock_acquire+0xcb/0x2b0 [ 2277.339868] ceph_fill_trace+0x143/0xf30 [ceph] [ 2277.344404] ? __mutex_lock+0x97/0xce0 [ 2277.348156] ? __mutex_lock+0xaec/0xce0 [ 2277.352033] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2277.356580] ? lock_downgrade+0xfd/0x250 [ 2277.360544] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2277.364950] ceph_con_process_message+0x74/0x150 [libceph] [ 2277.370446] ? calc_signature+0xdb/0x100 [libceph] [ 2277.375268] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2277.380606] ? process_scheduled_works+0x22a/0x510 [ 2277.385401] ? process_scheduled_works+0x22a/0x510 [ 2277.390189] ceph_con_workfn+0x344/0x750 [libceph] [ 2277.395012] ? process_scheduled_works+0x22a/0x510 [ 2277.399818] process_scheduled_works+0x2a4/0x510 [ 2277.404462] worker_thread+0x188/0x340 [ 2277.408218] ? rescuer_thread+0x310/0x310 [ 2277.412265] kthread+0xf5/0x130 [ 2277.415412] ? kthread_complete_and_exit+0x20/0x20 [ 2277.420204] ret_from_fork+0x31/0x40 [ 2277.423800] ? kthread_complete_and_exit+0x20/0x20 [ 2277.428610] ret_from_fork_asm+0x11/0x20 [ 2277.432579] [ 2277.434773] irq event stamp: 10529871 [ 2277.438455] hardirqs last enabled at (10529881): [] console_unlock+0x106/0x120 [ 2277.447323] hardirqs last disabled at (10529890): [] console_unlock+0xeb/0x120 [ 2277.456100] softirqs last enabled at (10529818): [] __do_softirq+0x2be/0x371 [ 2277.464812] softirqs last disabled at (10529813): [] irq_exit_rcu+0xb0/0xd0 [ 2277.473349] ---[ end trace 0000000000000000 ]--- [ 2277.478000] ------------[ cut here ]------------ [ 2277.482614] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2277.494257] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2277.502945] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2277.558424] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2277.567547] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2277.575028] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2277.580521] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2277.586007] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2277.604756] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2277.609981] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2277.617114] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2277.624249] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005f52 [ 2277.631379] R10: 000000000020a788 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2277.638513] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2277.645646] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2277.653731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2277.659476] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2277.666609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2277.673745] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2277.680875] Call Trace: [ 2277.683329] [ 2277.685434] ? __warn+0x85/0x180 [ 2277.688669] ? __list_add_valid_or_report+0x7d/0xa0 [ 2277.693547] ? report_bug+0xfc/0x1d0 [ 2277.697126] ? prb_read_valid+0x1b/0x20 [ 2277.700966] ? handle_bug+0x44/0x70 [ 2277.704458] ? exc_invalid_op+0x17/0x60 [ 2277.708298] ? asm_exc_invalid_op+0x1a/0x20 [ 2277.712490] ? __list_add_valid_or_report+0x7d/0xa0 [ 2277.717373] ? __list_add_valid_or_report+0x7d/0xa0 [ 2277.722252] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2277.727142] ceph_add_cap+0x20b/0x620 [ceph] [ 2277.731426] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2277.736141] ? lock_acquire+0xcb/0x2b0 [ 2277.739894] ceph_fill_trace+0x143/0xf30 [ceph] [ 2277.744429] ? __mutex_lock+0x97/0xce0 [ 2277.748182] ? __mutex_lock+0xaec/0xce0 [ 2277.752025] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2277.756570] ? lock_downgrade+0xfd/0x250 [ 2277.760503] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2277.764883] ceph_con_process_message+0x74/0x150 [libceph] [ 2277.770378] ? calc_signature+0xdb/0x100 [libceph] [ 2277.775182] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2277.780512] ? process_scheduled_works+0x22a/0x510 [ 2277.785305] ? process_scheduled_works+0x22a/0x510 [ 2277.790096] ceph_con_workfn+0x344/0x750 [libceph] [ 2277.794898] ? process_scheduled_works+0x22a/0x510 [ 2277.799688] process_scheduled_works+0x2a4/0x510 [ 2277.804315] worker_thread+0x188/0x340 [ 2277.808072] ? rescuer_thread+0x310/0x310 [ 2277.812082] kthread+0xf5/0x130 [ 2277.815230] ? kthread_complete_and_exit+0x20/0x20 [ 2277.820021] ret_from_fork+0x31/0x40 [ 2277.823601] ? kthread_complete_and_exit+0x20/0x20 [ 2277.828394] ret_from_fork_asm+0x11/0x20 [ 2277.832326] [ 2277.834512] irq event stamp: 10530983 [ 2277.838179] hardirqs last enabled at (10530993): [] console_unlock+0x106/0x120 [ 2277.847043] hardirqs last disabled at (10531002): [] console_unlock+0xeb/0x120 [ 2277.855823] softirqs last enabled at (10530930): [] __do_softirq+0x2be/0x371 [ 2277.864516] softirqs last disabled at (10530925): [] irq_exit_rcu+0xb0/0xd0 [ 2277.873035] ---[ end trace 0000000000000000 ]--- [ 2277.877657] ------------[ cut here ]------------ [ 2277.882274] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2277.893929] WARNING: CPU: 0 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2277.902894] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2277.958361] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2277.967316] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2277.974793] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2277.980278] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2277.999027] RSP: 0018:ffffc90001fdfac0 EFLAGS: 00010282 [ 2278.004253] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2278.011383] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2278.018516] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005f91 [ 2278.025651] R10: 000000000020b890 R11: ffffc90001fdf950 R12: ffff8881c5350750 [ 2278.032781] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2278.039906] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2278.047992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2278.053739] CR2: 00007f9bd2bc3000 CR3: 0000000179402006 CR4: 00000000003706f0 [ 2278.060871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2278.068004] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2278.075137] Call Trace: [ 2278.077589] [ 2278.079687] ? __warn+0x85/0x180 [ 2278.082921] ? __list_add_valid_or_report+0x7d/0xa0 [ 2278.087801] ? report_bug+0xfc/0x1d0 [ 2278.091379] ? prb_read_valid+0x1b/0x20 [ 2278.095221] ? handle_bug+0x44/0x70 [ 2278.098712] ? exc_invalid_op+0x17/0x60 [ 2278.102552] ? asm_exc_invalid_op+0x1a/0x20 [ 2278.106741] ? __list_add_valid_or_report+0x7d/0xa0 [ 2278.111616] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2278.116505] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2278.121741] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2278.126543] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2278.131261] ? __d_lookup+0xd6/0x200 [ 2278.134837] lookup_fast+0x6d/0xf0 [ 2278.138241] walk_component+0x26/0x160 [ 2278.141995] link_path_walk+0x24d/0x380 [ 2278.145833] ? path_lookupat.isra.40+0x22/0x120 [ 2278.150367] path_lookupat.isra.40+0x6b/0x120 [ 2278.154728] filename_lookup+0xd4/0x190 [ 2278.158567] ? lock_acquire+0xcb/0x2b0 [ 2278.162317] ? __might_fault+0x45/0x80 [ 2278.166070] ? find_held_lock+0x2d/0x90 [ 2278.169909] ? __might_fault+0x45/0x80 [ 2278.173663] vfs_statx+0x89/0x150 [ 2278.176985] vfs_fstatat+0x5a/0xc0 [ 2278.180389] __do_sys_newstat+0x30/0x70 [ 2278.184226] ? lock_release+0xc5/0x260 [ 2278.187981] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2278.192859] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2278.197998] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2278.203140] do_syscall_64+0x38/0x80 [ 2278.206718] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2278.211768] RIP: 0033:0x7f48f0eeca79 [ 2278.215351] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2278.234096] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2278.241661] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2278.248794] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2278.255926] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2278.263051] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2278.270183] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2278.277315] [ 2278.279500] irq event stamp: 12858909 [ 2278.283158] hardirqs last enabled at (12858919): [] console_unlock+0x106/0x120 [ 2278.292038] hardirqs last disabled at (12858952): [] console_unlock+0xeb/0x120 [ 2278.300815] softirqs last enabled at (12858966): [] __do_softirq+0x2be/0x371 [ 2278.309506] softirqs last disabled at (12858961): [] irq_exit_rcu+0xb0/0xd0 [ 2278.318023] ---[ end trace 0000000000000000 ]--- [ 2278.322636] ------------[ cut here ]------------ [ 2278.327277] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2278.338934] WARNING: CPU: 5 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2278.347883] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2278.403361] CPU: 5 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2278.412312] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2278.419790] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2278.425293] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2278.444042] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2278.449287] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2278.456416] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2278.463565] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100005fd5 [ 2278.470711] R10: 000000000020cbf8 R11: ffffc90001fc7950 R12: ffff888158a68750 [ 2278.477860] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2278.484992] FS: 00007f48f1832740(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2278.493095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2278.498857] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2278.506000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2278.513150] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2278.520291] Call Trace: [ 2278.522736] [ 2278.524860] ? __warn+0x85/0x180 [ 2278.528108] ? __list_add_valid_or_report+0x7d/0xa0 [ 2278.533006] ? report_bug+0xfc/0x1d0 [ 2278.536602] ? prb_read_valid+0x1b/0x20 [ 2278.540461] ? handle_bug+0x44/0x70 [ 2278.543953] ? exc_invalid_op+0x17/0x60 [ 2278.547810] ? asm_exc_invalid_op+0x1a/0x20 [ 2278.552034] ? __list_add_valid_or_report+0x7d/0xa0 [ 2278.556910] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2278.561816] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2278.567069] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2278.571887] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2278.576620] ? __d_lookup+0xd6/0x200 [ 2278.580216] lookup_fast+0x6d/0xf0 [ 2278.583637] walk_component+0x26/0x160 [ 2278.587410] link_path_walk+0x24d/0x380 [ 2278.591247] ? path_lookupat.isra.40+0x22/0x120 [ 2278.595800] path_lookupat.isra.40+0x6b/0x120 [ 2278.600177] filename_lookup+0xd4/0x190 [ 2278.604051] ? lock_acquire+0xcb/0x2b0 [ 2278.607799] ? __might_fault+0x45/0x80 [ 2278.611571] ? find_held_lock+0x2d/0x90 [ 2278.615427] ? __might_fault+0x45/0x80 [ 2278.619183] vfs_statx+0x89/0x150 [ 2278.622520] vfs_fstatat+0x5a/0xc0 [ 2278.625924] __do_sys_newstat+0x30/0x70 [ 2278.629780] ? lock_release+0xc5/0x260 [ 2278.633568] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2278.638464] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2278.643603] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2278.648762] do_syscall_64+0x38/0x80 [ 2278.652357] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2278.657410] RIP: 0033:0x7f48f0eeca79 [ 2278.660988] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2278.679753] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2278.687336] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2278.694468] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2278.701604] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2278.708734] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2278.715866] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2278.722997] [ 2278.725185] irq event stamp: 12945401 [ 2278.728849] hardirqs last enabled at (12945411): [] console_unlock+0x106/0x120 [ 2278.737717] hardirqs last disabled at (12945420): [] console_unlock+0xeb/0x120 [ 2278.746513] softirqs last enabled at (12945434): [] __do_softirq+0x2be/0x371 [ 2278.755205] softirqs last disabled at (12945429): [] irq_exit_rcu+0xb0/0xd0 [ 2278.763744] ---[ end trace 0000000000000000 ]--- [ 2278.768426] ------------[ cut here ]------------ [ 2278.773046] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2278.784723] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2278.793418] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2278.848896] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2278.858037] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2278.865536] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2278.871028] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2278.876533] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2278.895294] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2278.900523] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2278.907657] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2278.914806] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006019 [ 2278.921956] R10: 000000000020ddd8 R11: ffffc9000030b828 R12: ffff8881d106cbc8 [ 2278.929106] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2278.936256] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2278.944361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2278.950126] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2278.957258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2278.964406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2278.971539] Call Trace: [ 2278.973992] [ 2278.976114] ? __warn+0x85/0x180 [ 2278.979368] ? __list_add_valid_or_report+0x7d/0xa0 [ 2278.984246] ? report_bug+0xfc/0x1d0 [ 2278.987848] ? prb_read_valid+0x1b/0x20 [ 2278.991690] ? handle_bug+0x44/0x70 [ 2278.995200] ? exc_invalid_op+0x17/0x60 [ 2278.999057] ? asm_exc_invalid_op+0x1a/0x20 [ 2279.003286] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.008165] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.013045] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2279.017951] ceph_add_cap+0x20b/0x620 [ceph] [ 2279.022277] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2279.026993] ? lock_acquire+0xcb/0x2b0 [ 2279.030764] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2279.035214] ? __mutex_lock+0x97/0xce0 [ 2279.038983] ? __mutex_lock+0xaec/0xce0 [ 2279.042843] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2279.047399] ? lock_downgrade+0xfd/0x250 [ 2279.051330] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2279.055728] ceph_con_process_message+0x74/0x150 [libceph] [ 2279.061223] ? calc_signature+0xdb/0x100 [libceph] [ 2279.066045] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2279.071392] ? process_scheduled_works+0x22a/0x510 [ 2279.076202] ? process_scheduled_works+0x22a/0x510 [ 2279.080991] ceph_con_workfn+0x344/0x750 [libceph] [ 2279.085813] ? process_scheduled_works+0x22a/0x510 [ 2279.090621] process_scheduled_works+0x2a4/0x510 [ 2279.095284] worker_thread+0x188/0x340 [ 2279.099036] ? rescuer_thread+0x310/0x310 [ 2279.103050] kthread+0xf5/0x130 [ 2279.106215] ? kthread_complete_and_exit+0x20/0x20 [ 2279.111023] ret_from_fork+0x31/0x40 [ 2279.114619] ? kthread_complete_and_exit+0x20/0x20 [ 2279.119431] ret_from_fork_asm+0x11/0x20 [ 2279.123362] [ 2279.125549] irq event stamp: 10532171 [ 2279.129214] hardirqs last enabled at (10532181): [] console_unlock+0x106/0x120 [ 2279.138098] hardirqs last disabled at (10532190): [] console_unlock+0xeb/0x120 [ 2279.146896] softirqs last enabled at (10532118): [] __do_softirq+0x2be/0x371 [ 2279.155604] softirqs last disabled at (10532113): [] irq_exit_rcu+0xb0/0xd0 [ 2279.164141] ---[ end trace 0000000000000000 ]--- [ 2279.168950] ------------[ cut here ]------------ [ 2279.173571] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2279.185216] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2279.193914] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2279.249389] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2279.258514] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2279.265995] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2279.271489] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2279.276975] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2279.295721] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2279.300948] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2279.308081] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2279.315216] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006058 [ 2279.322346] R10: 000000000020f040 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2279.329479] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2279.336612] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2279.344698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2279.350443] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2279.357577] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2279.364709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2279.371842] Call Trace: [ 2279.374294] [ 2279.376401] ? __warn+0x85/0x180 [ 2279.379636] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.384514] ? report_bug+0xfc/0x1d0 [ 2279.388091] ? prb_read_valid+0x1b/0x20 [ 2279.391934] ? handle_bug+0x44/0x70 [ 2279.395425] ? exc_invalid_op+0x17/0x60 [ 2279.399265] ? asm_exc_invalid_op+0x1a/0x20 [ 2279.403454] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.408330] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.413209] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2279.418100] ceph_add_cap+0x20b/0x620 [ceph] [ 2279.422384] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2279.427099] ? lock_acquire+0xcb/0x2b0 [ 2279.430852] ceph_fill_trace+0x143/0xf30 [ceph] [ 2279.435387] ? __mutex_lock+0x97/0xce0 [ 2279.439140] ? __mutex_lock+0xaec/0xce0 [ 2279.442981] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2279.447521] ? lock_downgrade+0xfd/0x250 [ 2279.451453] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2279.455833] ceph_con_process_message+0x74/0x150 [libceph] [ 2279.461328] ? calc_signature+0xdb/0x100 [libceph] [ 2279.466132] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2279.471463] ? process_scheduled_works+0x22a/0x510 [ 2279.476254] ? process_scheduled_works+0x22a/0x510 [ 2279.481044] ceph_con_workfn+0x344/0x750 [libceph] [ 2279.485847] ? process_scheduled_works+0x22a/0x510 [ 2279.490637] process_scheduled_works+0x2a4/0x510 [ 2279.495263] worker_thread+0x188/0x340 [ 2279.499011] ? rescuer_thread+0x310/0x310 [ 2279.503023] kthread+0xf5/0x130 [ 2279.506170] ? kthread_complete_and_exit+0x20/0x20 [ 2279.510964] ret_from_fork+0x31/0x40 [ 2279.514541] ? kthread_complete_and_exit+0x20/0x20 [ 2279.519334] ret_from_fork_asm+0x11/0x20 [ 2279.523269] [ 2279.525462] irq event stamp: 10533279 [ 2279.529127] hardirqs last enabled at (10533289): [] console_unlock+0x106/0x120 [ 2279.537995] hardirqs last disabled at (10533298): [] console_unlock+0xeb/0x120 [ 2279.546774] softirqs last enabled at (10533226): [] __do_softirq+0x2be/0x371 [ 2279.555466] softirqs last disabled at (10533221): [] irq_exit_rcu+0xb0/0xd0 [ 2279.563985] ---[ end trace 0000000000000000 ]--- [ 2279.568631] ------------[ cut here ]------------ [ 2279.573252] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2279.584895] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2279.593592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2279.649069] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2279.658194] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2279.665675] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2279.671169] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2279.676654] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2279.695401] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2279.700626] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2279.707759] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2279.714895] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006097 [ 2279.722027] R10: 0000000000210178 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2279.729158] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2279.736291] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2279.744378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2279.750123] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2279.757257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2279.764390] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2279.771521] Call Trace: [ 2279.773976] [ 2279.776080] ? __warn+0x85/0x180 [ 2279.779316] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.784194] ? report_bug+0xfc/0x1d0 [ 2279.787772] ? prb_read_valid+0x1b/0x20 [ 2279.791613] ? handle_bug+0x44/0x70 [ 2279.795104] ? exc_invalid_op+0x17/0x60 [ 2279.798945] ? asm_exc_invalid_op+0x1a/0x20 [ 2279.803135] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.808011] ? __list_add_valid_or_report+0x7d/0xa0 [ 2279.812889] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2279.817780] ceph_add_cap+0x20b/0x620 [ceph] [ 2279.822073] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2279.826797] ? lock_acquire+0xcb/0x2b0 [ 2279.830557] ceph_fill_trace+0x143/0xf30 [ceph] [ 2279.835093] ? __mutex_lock+0x97/0xce0 [ 2279.838847] ? __mutex_lock+0xaec/0xce0 [ 2279.842688] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2279.847226] ? lock_downgrade+0xfd/0x250 [ 2279.851157] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2279.855538] ceph_con_process_message+0x74/0x150 [libceph] [ 2279.861032] ? calc_signature+0xdb/0x100 [libceph] [ 2279.865837] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2279.871169] ? process_scheduled_works+0x22a/0x510 [ 2279.875961] ? process_scheduled_works+0x22a/0x510 [ 2279.880749] ceph_con_workfn+0x344/0x750 [libceph] [ 2279.885555] ? process_scheduled_works+0x22a/0x510 [ 2279.890343] process_scheduled_works+0x2a4/0x510 [ 2279.894970] worker_thread+0x188/0x340 [ 2279.898726] ? rescuer_thread+0x310/0x310 [ 2279.902737] kthread+0xf5/0x130 [ 2279.905886] ? kthread_complete_and_exit+0x20/0x20 [ 2279.910678] ret_from_fork+0x31/0x40 [ 2279.914255] ? kthread_complete_and_exit+0x20/0x20 [ 2279.919049] ret_from_fork_asm+0x11/0x20 [ 2279.922982] [ 2279.925167] irq event stamp: 10534433 [ 2279.928832] hardirqs last enabled at (10534443): [] console_unlock+0x106/0x120 [ 2279.937701] hardirqs last disabled at (10534452): [] console_unlock+0xeb/0x120 [ 2279.946478] softirqs last enabled at (10534380): [] __do_softirq+0x2be/0x371 [ 2279.955172] softirqs last disabled at (10534375): [] irq_exit_rcu+0xb0/0xd0 [ 2279.963693] ---[ end trace 0000000000000000 ]--- [ 2279.968333] ------------[ cut here ]------------ [ 2279.972949] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2279.984594] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2279.993290] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2280.048776] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2280.057899] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2280.065379] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2280.070874] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2280.076360] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2280.095107] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2280.100335] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2280.107468] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2280.114603] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000060d6 [ 2280.121742] R10: 00000000002113c0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2280.128875] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2280.136009] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2280.144101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2280.149848] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2280.156981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2280.164114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2280.171247] Call Trace: [ 2280.173701] [ 2280.175805] ? __warn+0x85/0x180 [ 2280.179043] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.183928] ? report_bug+0xfc/0x1d0 [ 2280.187508] ? prb_read_valid+0x1b/0x20 [ 2280.191357] ? handle_bug+0x44/0x70 [ 2280.194856] ? exc_invalid_op+0x17/0x60 [ 2280.198694] ? asm_exc_invalid_op+0x1a/0x20 [ 2280.202894] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.207779] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.212659] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2280.217559] ceph_add_cap+0x20b/0x620 [ceph] [ 2280.221864] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2280.226595] ? lock_acquire+0xcb/0x2b0 [ 2280.230357] ceph_fill_trace+0x143/0xf30 [ceph] [ 2280.234896] ? __mutex_lock+0x97/0xce0 [ 2280.238650] ? __mutex_lock+0xaec/0xce0 [ 2280.242493] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2280.247049] ? lock_downgrade+0xfd/0x250 [ 2280.250990] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2280.255382] ceph_con_process_message+0x74/0x150 [libceph] [ 2280.260879] ? calc_signature+0xdb/0x100 [libceph] [ 2280.265704] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2280.271044] ? process_scheduled_works+0x22a/0x510 [ 2280.275847] ? process_scheduled_works+0x22a/0x510 [ 2280.280638] ceph_con_workfn+0x344/0x750 [libceph] [ 2280.285456] ? process_scheduled_works+0x22a/0x510 [ 2280.290250] process_scheduled_works+0x2a4/0x510 [ 2280.294884] worker_thread+0x188/0x340 [ 2280.298644] ? rescuer_thread+0x310/0x310 [ 2280.302664] kthread+0xf5/0x130 [ 2280.305819] ? kthread_complete_and_exit+0x20/0x20 [ 2280.310621] ret_from_fork+0x31/0x40 [ 2280.314206] ? kthread_complete_and_exit+0x20/0x20 [ 2280.319003] ret_from_fork_asm+0x11/0x20 [ 2280.322946] [ 2280.325144] irq event stamp: 10535567 [ 2280.328810] hardirqs last enabled at (10535577): [] console_unlock+0x106/0x120 [ 2280.337677] hardirqs last disabled at (10535586): [] console_unlock+0xeb/0x120 [ 2280.346455] softirqs last enabled at (10535514): [] __do_softirq+0x2be/0x371 [ 2280.355148] softirqs last disabled at (10535509): [] irq_exit_rcu+0xb0/0xd0 [ 2280.363668] ---[ end trace 0000000000000000 ]--- [ 2280.368326] ------------[ cut here ]------------ [ 2280.372948] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2280.384609] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2280.393328] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2280.448814] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2280.457946] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2280.465424] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2280.470938] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2280.476422] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2280.495169] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2280.500394] RAX: 0000000000000000 RBX: ffff8881c3b74478 RCX: 0000000000000000 [ 2280.507529] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2280.514664] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006115 [ 2280.521795] R10: 00000000002124f8 R11: ffffc9000030b828 R12: ffff8881c3b74bc8 [ 2280.528925] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2280.536060] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2280.544145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2280.549892] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2280.557023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2280.564160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2280.571289] Call Trace: [ 2280.573744] [ 2280.575848] ? __warn+0x85/0x180 [ 2280.579087] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.583969] ? report_bug+0xfc/0x1d0 [ 2280.587550] ? prb_read_valid+0x1b/0x20 [ 2280.591392] ? handle_bug+0x44/0x70 [ 2280.594891] ? exc_invalid_op+0x17/0x60 [ 2280.598728] ? asm_exc_invalid_op+0x1a/0x20 [ 2280.602923] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.607803] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.612684] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2280.617599] ceph_add_cap+0x20b/0x620 [ceph] [ 2280.621895] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2280.626619] ? lock_acquire+0xcb/0x2b0 [ 2280.630380] ceph_fill_trace+0x143/0xf30 [ceph] [ 2280.634922] ? __mutex_lock+0x97/0xce0 [ 2280.638675] ? __mutex_lock+0xaec/0xce0 [ 2280.642520] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2280.647074] ? lock_downgrade+0xfd/0x250 [ 2280.651008] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2280.655395] ceph_con_process_message+0x74/0x150 [libceph] [ 2280.660897] ? calc_signature+0xdb/0x100 [libceph] [ 2280.665721] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2280.671051] ? process_scheduled_works+0x22a/0x510 [ 2280.675851] ? process_scheduled_works+0x22a/0x510 [ 2280.680646] ceph_con_workfn+0x344/0x750 [libceph] [ 2280.685464] ? process_scheduled_works+0x22a/0x510 [ 2280.690258] process_scheduled_works+0x2a4/0x510 [ 2280.694886] worker_thread+0x188/0x340 [ 2280.698641] ? rescuer_thread+0x310/0x310 [ 2280.702654] kthread+0xf5/0x130 [ 2280.705802] ? kthread_complete_and_exit+0x20/0x20 [ 2280.710611] ret_from_fork+0x31/0x40 [ 2280.714188] ? kthread_complete_and_exit+0x20/0x20 [ 2280.718983] ret_from_fork_asm+0x11/0x20 [ 2280.722919] [ 2280.725109] irq event stamp: 10536683 [ 2280.728772] hardirqs last enabled at (10536693): [] console_unlock+0x106/0x120 [ 2280.737641] hardirqs last disabled at (10536702): [] console_unlock+0xeb/0x120 [ 2280.746419] softirqs last enabled at (10536630): [] __do_softirq+0x2be/0x371 [ 2280.755114] softirqs last disabled at (10536625): [] irq_exit_rcu+0xb0/0xd0 [ 2280.763632] ---[ end trace 0000000000000000 ]--- [ 2280.768317] ------------[ cut here ]------------ [ 2280.772940] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2280.784587] WARNING: CPU: 6 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2280.793542] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2280.849021] CPU: 6 PID: 78684 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2280.857981] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2280.865456] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2280.870946] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2280.889691] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [ 2280.894918] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000000 [ 2280.902049] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2280.909184] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006154 [ 2280.916318] R10: 0000000000213750 R11: ffffc90001fe7a70 R12: ffff8881d12c72e8 [ 2280.923448] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2280.930586] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2280.938677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2280.944421] CR2: 00007f9bd2b76000 CR3: 0000000108c08001 CR4: 00000000003706e0 [ 2280.951555] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2280.958687] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2280.965821] Call Trace: [ 2280.968273] [ 2280.970381] ? __warn+0x85/0x180 [ 2280.973611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2280.978494] ? report_bug+0xfc/0x1d0 [ 2280.982083] ? prb_read_valid+0x1b/0x20 [ 2280.985932] ? handle_bug+0x44/0x70 [ 2280.989428] ? exc_invalid_op+0x17/0x60 [ 2280.993270] ? asm_exc_invalid_op+0x1a/0x20 [ 2280.997463] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.002345] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2281.007241] try_get_cap_refs+0x159/0x880 [ceph] [ 2281.011888] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2281.016444] ? __file_remove_privs+0x89/0x110 [ 2281.020804] ? wake_bit_function+0x70/0x70 [ 2281.024907] ? ceph_write_iter+0x260/0x770 [ceph] [ 2281.029635] ceph_write_iter+0x260/0x770 [ceph] [ 2281.034190] ? lock_is_held_type+0xaa/0x120 [ 2281.038387] vfs_write+0x324/0x4f0 [ 2281.041803] ksys_pwrite64+0x65/0xa0 [ 2281.045385] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2281.050532] do_syscall_64+0x38/0x80 [ 2281.054112] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2281.059169] RIP: 0033:0x7f48f0eeb93b [ 2281.062750] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2281.081495] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2281.089064] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [ 2281.096195] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [ 2281.103327] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [ 2281.110464] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [ 2281.117594] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [ 2281.124738] [ 2281.126930] irq event stamp: 12908461 [ 2281.130593] hardirqs last enabled at (12908471): [] console_unlock+0x106/0x120 [ 2281.139461] hardirqs last disabled at (12908480): [] console_unlock+0xeb/0x120 [ 2281.148240] softirqs last enabled at (12908428): [] __do_softirq+0x2be/0x371 [ 2281.156933] softirqs last disabled at (12908423): [] irq_exit_rcu+0xb0/0xd0 [ 2281.165459] ---[ end trace 0000000000000000 ]--- [ 2281.170081] ------------[ cut here ]------------ [ 2281.174698] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2281.186345] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2281.195040] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2281.250534] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2281.259658] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2281.267155] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2281.272667] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2281.278171] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2281.296935] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2281.302178] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2281.309337] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2281.316471] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000618d [ 2281.323618] R10: 0000000000214890 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2281.330761] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2281.337912] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2281.346014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2281.351776] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2281.358920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2281.366070] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2281.373219] Call Trace: [ 2281.375681] [ 2281.377797] ? __warn+0x85/0x180 [ 2281.381049] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.385945] ? report_bug+0xfc/0x1d0 [ 2281.389539] ? prb_read_valid+0x1b/0x20 [ 2281.393382] ? handle_bug+0x44/0x70 [ 2281.396872] ? exc_invalid_op+0x17/0x60 [ 2281.400728] ? asm_exc_invalid_op+0x1a/0x20 [ 2281.404955] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.409839] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.414733] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2281.419641] ceph_add_cap+0x20b/0x620 [ceph] [ 2281.423960] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2281.428694] ? lock_acquire+0xcb/0x2b0 [ 2281.432447] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2281.436905] ? __mutex_lock+0x97/0xce0 [ 2281.440674] ? __mutex_lock+0xaec/0xce0 [ 2281.444533] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2281.449073] ? lock_downgrade+0xfd/0x250 [ 2281.453036] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2281.457410] ceph_con_process_message+0x74/0x150 [libceph] [ 2281.462906] ? calc_signature+0xdb/0x100 [libceph] [ 2281.467752] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2281.473102] ? process_scheduled_works+0x22a/0x510 [ 2281.477908] ? process_scheduled_works+0x22a/0x510 [ 2281.482700] ceph_con_workfn+0x344/0x750 [libceph] [ 2281.487520] ? process_scheduled_works+0x22a/0x510 [ 2281.492315] process_scheduled_works+0x2a4/0x510 [ 2281.496971] worker_thread+0x188/0x340 [ 2281.500727] ? rescuer_thread+0x310/0x310 [ 2281.504758] kthread+0xf5/0x130 [ 2281.507941] ? kthread_complete_and_exit+0x20/0x20 [ 2281.512731] ret_from_fork+0x31/0x40 [ 2281.516340] ? kthread_complete_and_exit+0x20/0x20 [ 2281.521128] ret_from_fork_asm+0x11/0x20 [ 2281.525078] [ 2281.527264] irq event stamp: 10539953 [ 2281.530947] hardirqs last enabled at (10539963): [] console_unlock+0x106/0x120 [ 2281.539831] hardirqs last disabled at (10539972): [] console_unlock+0xeb/0x120 [ 2281.548629] softirqs last enabled at (10539900): [] __do_softirq+0x2be/0x371 [ 2281.557343] softirqs last disabled at (10539895): [] irq_exit_rcu+0xb0/0xd0 [ 2281.565865] ---[ end trace 0000000000000000 ]--- [ 2281.570508] ------------[ cut here ]------------ [ 2281.575123] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2281.586802] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2281.595499] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2281.650977] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2281.660118] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2281.667622] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2281.673135] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2281.678637] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2281.697402] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2281.702631] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2281.709779] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2281.716930] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000061cc [ 2281.724080] R10: 0000000000215ae8 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2281.731230] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2281.738379] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2281.746466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2281.752213] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2281.759347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2281.766477] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2281.773626] Call Trace: [ 2281.776072] [ 2281.778194] ? __warn+0x85/0x180 [ 2281.781445] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.786343] ? report_bug+0xfc/0x1d0 [ 2281.789920] ? prb_read_valid+0x1b/0x20 [ 2281.793778] ? handle_bug+0x44/0x70 [ 2281.797289] ? exc_invalid_op+0x17/0x60 [ 2281.801146] ? asm_exc_invalid_op+0x1a/0x20 [ 2281.805354] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.810236] ? __list_add_valid_or_report+0x7d/0xa0 [ 2281.815132] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2281.820040] ceph_add_cap+0x20b/0x620 [ceph] [ 2281.824354] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2281.829099] ceph_fill_trace+0x143/0xf30 [ceph] [ 2281.833637] ? __mutex_lock+0x97/0xce0 [ 2281.837407] ? __mutex_lock+0xaec/0xce0 [ 2281.841246] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2281.845806] ? lock_downgrade+0xfd/0x250 [ 2281.849770] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2281.854167] ceph_con_process_message+0x74/0x150 [libceph] [ 2281.859662] ? calc_signature+0xdb/0x100 [libceph] [ 2281.864485] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2281.869833] ? process_scheduled_works+0x22a/0x510 [ 2281.874639] ? process_scheduled_works+0x22a/0x510 [ 2281.879448] ceph_con_workfn+0x344/0x750 [libceph] [ 2281.884253] ? process_scheduled_works+0x22a/0x510 [ 2281.889060] process_scheduled_works+0x2a4/0x510 [ 2281.893720] worker_thread+0x188/0x340 [ 2281.897469] ? rescuer_thread+0x310/0x310 [ 2281.901479] kthread+0xf5/0x130 [ 2281.904638] ? kthread_complete_and_exit+0x20/0x20 [ 2281.909448] ret_from_fork+0x31/0x40 [ 2281.913023] ? kthread_complete_and_exit+0x20/0x20 [ 2281.917834] ret_from_fork_asm+0x11/0x20 [ 2281.921803] [ 2281.923987] irq event stamp: 10541011 [ 2281.927662] hardirqs last enabled at (10541021): [] console_unlock+0x106/0x120 [ 2281.936545] hardirqs last disabled at (10541030): [] console_unlock+0xeb/0x120 [ 2281.945344] softirqs last enabled at (10540968): [] __do_softirq+0x2be/0x371 [ 2281.954058] softirqs last disabled at (10541049): [] irq_exit_rcu+0xb0/0xd0 [ 2281.962575] ---[ end trace 0000000000000000 ]--- [ 2281.967363] ------------[ cut here ]------------ [ 2281.971975] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2281.983656] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2281.992352] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2282.047829] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2282.056972] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2282.064470] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2282.069971] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2282.075474] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2282.094222] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2282.099465] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2282.106600] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2282.113731] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000620a [ 2282.120881] R10: 0000000000216bf8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2282.128032] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2282.135182] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2282.143286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2282.149047] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2282.156190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2282.163351] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2282.170481] Call Trace: [ 2282.172934] [ 2282.175057] ? __warn+0x85/0x180 [ 2282.178310] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.183214] ? report_bug+0xfc/0x1d0 [ 2282.186793] ? prb_read_valid+0x1b/0x20 [ 2282.190651] ? handle_bug+0x44/0x70 [ 2282.194160] ? exc_invalid_op+0x17/0x60 [ 2282.198015] ? asm_exc_invalid_op+0x1a/0x20 [ 2282.202241] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.207116] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.212013] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2282.216920] ceph_add_cap+0x20b/0x620 [ceph] [ 2282.221239] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2282.225972] ? lock_acquire+0xcb/0x2b0 [ 2282.229742] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2282.234192] ? __mutex_lock+0x97/0xce0 [ 2282.237961] ? __mutex_lock+0xaec/0xce0 [ 2282.241820] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2282.246377] ? lock_downgrade+0xfd/0x250 [ 2282.250307] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2282.254695] ceph_con_process_message+0x74/0x150 [libceph] [ 2282.260192] ? calc_signature+0xdb/0x100 [libceph] [ 2282.265013] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2282.270358] ? process_scheduled_works+0x22a/0x510 [ 2282.275154] ? process_scheduled_works+0x22a/0x510 [ 2282.279960] ceph_con_workfn+0x344/0x750 [libceph] [ 2282.284782] ? process_scheduled_works+0x22a/0x510 [ 2282.289588] process_scheduled_works+0x2a4/0x510 [ 2282.294232] worker_thread+0x188/0x340 [ 2282.297990] ? rescuer_thread+0x310/0x310 [ 2282.302019] kthread+0xf5/0x130 [ 2282.305182] ? kthread_complete_and_exit+0x20/0x20 [ 2282.309992] ret_from_fork+0x31/0x40 [ 2282.313588] ? kthread_complete_and_exit+0x20/0x20 [ 2282.318381] ret_from_fork_asm+0x11/0x20 [ 2282.322335] [ 2282.324525] irq event stamp: 10542141 [ 2282.328190] hardirqs last enabled at (10542151): [] console_unlock+0x106/0x120 [ 2282.337074] hardirqs last disabled at (10542160): [] console_unlock+0xeb/0x120 [ 2282.345876] softirqs last enabled at (10542182): [] __do_softirq+0x2be/0x371 [ 2282.354583] softirqs last disabled at (10542193): [] irq_exit_rcu+0xb0/0xd0 [ 2282.363103] ---[ end trace 0000000000000000 ]--- [ 2282.367762] ------------[ cut here ]------------ [ 2282.372375] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2282.384035] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2282.392725] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2282.448221] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2282.457361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2282.464842] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2282.470359] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2282.475839] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2282.494604] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2282.499828] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2282.506970] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2282.514115] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006249 [ 2282.521261] R10: 0000000000217e48 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2282.528404] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2282.535537] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2282.543639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2282.549402] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2282.556535] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2282.563670] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2282.570818] Call Trace: [ 2282.573280] [ 2282.575395] ? __warn+0x85/0x180 [ 2282.578641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.583534] ? report_bug+0xfc/0x1d0 [ 2282.587112] ? prb_read_valid+0x1b/0x20 [ 2282.590971] ? handle_bug+0x44/0x70 [ 2282.594479] ? exc_invalid_op+0x17/0x60 [ 2282.598318] ? asm_exc_invalid_op+0x1a/0x20 [ 2282.602527] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.607404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.612280] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2282.617190] ceph_add_cap+0x20b/0x620 [ceph] [ 2282.621494] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2282.626236] ceph_fill_trace+0x143/0xf30 [ceph] [ 2282.630778] ? __mutex_lock+0x97/0xce0 [ 2282.634547] ? __mutex_lock+0xaec/0xce0 [ 2282.638388] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2282.642928] ? lock_downgrade+0xfd/0x250 [ 2282.646892] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2282.651282] ceph_con_process_message+0x74/0x150 [libceph] [ 2282.656777] ? calc_signature+0xdb/0x100 [libceph] [ 2282.661598] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2282.666945] ? process_scheduled_works+0x22a/0x510 [ 2282.671738] ? process_scheduled_works+0x22a/0x510 [ 2282.676544] ceph_con_workfn+0x344/0x750 [libceph] [ 2282.681364] ? process_scheduled_works+0x22a/0x510 [ 2282.686157] process_scheduled_works+0x2a4/0x510 [ 2282.690816] worker_thread+0x188/0x340 [ 2282.694565] ? rescuer_thread+0x310/0x310 [ 2282.698577] kthread+0xf5/0x130 [ 2282.701724] ? kthread_complete_and_exit+0x20/0x20 [ 2282.706537] ret_from_fork+0x31/0x40 [ 2282.710113] ? kthread_complete_and_exit+0x20/0x20 [ 2282.714923] ret_from_fork_asm+0x11/0x20 [ 2282.718889] [ 2282.721075] irq event stamp: 10543231 [ 2282.724751] hardirqs last enabled at (10543241): [] console_unlock+0x106/0x120 [ 2282.733618] hardirqs last disabled at (10543250): [] console_unlock+0xeb/0x120 [ 2282.742397] softirqs last enabled at (10543188): [] __do_softirq+0x2be/0x371 [ 2282.751090] softirqs last disabled at (10543183): [] irq_exit_rcu+0xb0/0xd0 [ 2282.759626] ---[ end trace 0000000000000000 ]--- [ 2282.764296] ------------[ cut here ]------------ [ 2282.768917] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2282.780580] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2282.789276] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2282.844779] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2282.853921] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2282.861417] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2282.866912] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2282.872416] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2282.891163] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2282.896404] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2282.903539] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2282.910672] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006287 [ 2282.917821] R10: 0000000000219058 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2282.924972] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2282.932121] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2282.940226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2282.945988] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2282.953131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2282.960282] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2282.967430] Call Trace: [ 2282.969884] [ 2282.972006] ? __warn+0x85/0x180 [ 2282.975259] ? __list_add_valid_or_report+0x7d/0xa0 [ 2282.980154] ? report_bug+0xfc/0x1d0 [ 2282.983749] ? prb_read_valid+0x1b/0x20 [ 2282.987608] ? handle_bug+0x44/0x70 [ 2282.991100] ? exc_invalid_op+0x17/0x60 [ 2282.994957] ? asm_exc_invalid_op+0x1a/0x20 [ 2282.999182] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.004058] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.008952] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2283.013862] ceph_add_cap+0x20b/0x620 [ceph] [ 2283.018180] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2283.022912] ? lock_acquire+0xcb/0x2b0 [ 2283.026683] ceph_fill_trace+0x143/0xf30 [ceph] [ 2283.031219] ? __mutex_lock+0x97/0xce0 [ 2283.034988] ? __mutex_lock+0xaec/0xce0 [ 2283.038849] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2283.043412] ? lock_downgrade+0xfd/0x250 [ 2283.047365] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2283.051757] ceph_con_process_message+0x74/0x150 [libceph] [ 2283.057252] ? calc_signature+0xdb/0x100 [libceph] [ 2283.062074] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2283.067414] ? process_scheduled_works+0x22a/0x510 [ 2283.072206] ? process_scheduled_works+0x22a/0x510 [ 2283.077013] ceph_con_workfn+0x344/0x750 [libceph] [ 2283.081836] ? process_scheduled_works+0x22a/0x510 [ 2283.086647] process_scheduled_works+0x2a4/0x510 [ 2283.091301] worker_thread+0x188/0x340 [ 2283.095051] ? rescuer_thread+0x310/0x310 [ 2283.099080] kthread+0xf5/0x130 [ 2283.102245] ? kthread_complete_and_exit+0x20/0x20 [ 2283.107053] ret_from_fork+0x31/0x40 [ 2283.110659] ? kthread_complete_and_exit+0x20/0x20 [ 2283.115470] ret_from_fork_asm+0x11/0x20 [ 2283.119401] [ 2283.121586] irq event stamp: 10544345 [ 2283.125244] hardirqs last enabled at (10544355): [] console_unlock+0x106/0x120 [ 2283.134129] hardirqs last disabled at (10544364): [] console_unlock+0xeb/0x120 [ 2283.142924] softirqs last enabled at (10544292): [] __do_softirq+0x2be/0x371 [ 2283.151650] softirqs last disabled at (10544383): [] irq_exit_rcu+0xb0/0xd0 [ 2283.160190] ---[ end trace 0000000000000000 ]--- [ 2283.164868] ------------[ cut here ]------------ [ 2283.169489] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2283.181151] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2283.189847] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2283.245363] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2283.254486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2283.261964] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2283.267477] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2283.272961] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2283.291726] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2283.296968] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2283.304102] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2283.311253] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000062c6 [ 2283.318404] R10: 000000000021a2b0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2283.325535] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2283.332667] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2283.340772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2283.346534] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2283.353666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2283.360810] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2283.367959] Call Trace: [ 2283.370428] [ 2283.372536] ? __warn+0x85/0x180 [ 2283.375768] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.380664] ? report_bug+0xfc/0x1d0 [ 2283.384262] ? prb_read_valid+0x1b/0x20 [ 2283.388119] ? handle_bug+0x44/0x70 [ 2283.391627] ? exc_invalid_op+0x17/0x60 [ 2283.395469] ? asm_exc_invalid_op+0x1a/0x20 [ 2283.399685] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.404561] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.409440] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2283.414328] ceph_add_cap+0x20b/0x620 [ceph] [ 2283.418653] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2283.423381] ? lock_acquire+0xcb/0x2b0 [ 2283.427149] ceph_fill_trace+0x143/0xf30 [ceph] [ 2283.431686] ? __mutex_lock+0x97/0xce0 [ 2283.435455] ? __mutex_lock+0xaec/0xce0 [ 2283.439298] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2283.443854] ? lock_downgrade+0xfd/0x250 [ 2283.447819] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2283.452208] ceph_con_process_message+0x74/0x150 [libceph] [ 2283.457705] ? calc_signature+0xdb/0x100 [libceph] [ 2283.462526] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2283.467864] ? process_scheduled_works+0x22a/0x510 [ 2283.472671] ? process_scheduled_works+0x22a/0x510 [ 2283.477481] ceph_con_workfn+0x344/0x750 [libceph] [ 2283.482284] ? process_scheduled_works+0x22a/0x510 [ 2283.487092] process_scheduled_works+0x2a4/0x510 [ 2283.491753] worker_thread+0x188/0x340 [ 2283.495502] ? rescuer_thread+0x310/0x310 [ 2283.499514] kthread+0xf5/0x130 [ 2283.502660] ? kthread_complete_and_exit+0x20/0x20 [ 2283.507470] ret_from_fork+0x31/0x40 [ 2283.511048] ? kthread_complete_and_exit+0x20/0x20 [ 2283.515860] ret_from_fork_asm+0x11/0x20 [ 2283.519827] [ 2283.522013] irq event stamp: 10545465 [ 2283.525695] hardirqs last enabled at (10545475): [] console_unlock+0x106/0x120 [ 2283.534579] hardirqs last disabled at (10545484): [] console_unlock+0xeb/0x120 [ 2283.543375] softirqs last enabled at (10545412): [] __do_softirq+0x2be/0x371 [ 2283.552068] softirqs last disabled at (10545407): [] irq_exit_rcu+0xb0/0xd0 [ 2283.560605] ---[ end trace 0000000000000000 ]--- [ 2283.565256] ------------[ cut here ]------------ [ 2283.569873] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2283.581516] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2283.590213] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2283.645689] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2283.654814] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2283.662293] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2283.667788] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2283.673274] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2283.692020] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2283.697245] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2283.704379] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2283.711513] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006305 [ 2283.718646] R10: 000000000021b570 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2283.725779] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2283.732912] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2283.740998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2283.746742] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2283.753887] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2283.761021] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2283.768160] Call Trace: [ 2283.770614] [ 2283.772719] ? __warn+0x85/0x180 [ 2283.775952] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.780830] ? report_bug+0xfc/0x1d0 [ 2283.784408] ? prb_read_valid+0x1b/0x20 [ 2283.788257] ? handle_bug+0x44/0x70 [ 2283.791749] ? exc_invalid_op+0x17/0x60 [ 2283.795589] ? asm_exc_invalid_op+0x1a/0x20 [ 2283.799781] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.804656] ? __list_add_valid_or_report+0x7d/0xa0 [ 2283.809535] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2283.814426] ceph_add_cap+0x20b/0x620 [ceph] [ 2283.818710] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2283.823434] ? lock_acquire+0xcb/0x2b0 [ 2283.827192] ceph_fill_trace+0x143/0xf30 [ceph] [ 2283.831730] ? __mutex_lock+0x97/0xce0 [ 2283.835483] ? __mutex_lock+0xaec/0xce0 [ 2283.839325] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2283.843863] ? lock_downgrade+0xfd/0x250 [ 2283.847796] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2283.852173] ceph_con_process_message+0x74/0x150 [libceph] [ 2283.857669] ? calc_signature+0xdb/0x100 [libceph] [ 2283.862474] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2283.867804] ? process_scheduled_works+0x22a/0x510 [ 2283.872595] ? process_scheduled_works+0x22a/0x510 [ 2283.877387] ceph_con_workfn+0x344/0x750 [libceph] [ 2283.882205] ? process_scheduled_works+0x22a/0x510 [ 2283.886998] process_scheduled_works+0x2a4/0x510 [ 2283.891630] worker_thread+0x188/0x340 [ 2283.895384] ? rescuer_thread+0x310/0x310 [ 2283.899404] kthread+0xf5/0x130 [ 2283.902553] ? kthread_complete_and_exit+0x20/0x20 [ 2283.907359] ret_from_fork+0x31/0x40 [ 2283.910936] ? kthread_complete_and_exit+0x20/0x20 [ 2283.915731] ret_from_fork_asm+0x11/0x20 [ 2283.919668] [ 2283.921857] irq event stamp: 10546655 [ 2283.925527] hardirqs last enabled at (10546671): [] console_unlock+0x106/0x120 [ 2283.934400] hardirqs last disabled at (10546680): [] console_unlock+0xeb/0x120 [ 2283.943185] softirqs last enabled at (10546698): [] __do_softirq+0x2be/0x371 [ 2283.951879] softirqs last disabled at (10546689): [] irq_exit_rcu+0xb0/0xd0 [ 2283.960403] ---[ end trace 0000000000000000 ]--- [ 2283.965181] ------------[ cut here ]------------ [ 2283.969808] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2283.981461] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2283.990153] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2284.045668] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2284.054800] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2284.062279] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2284.067792] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2284.073284] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2284.092034] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2284.097266] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2284.104401] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2284.111533] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006344 [ 2284.118665] R10: 000000000021c730 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2284.125797] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2284.132931] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2284.141016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2284.146762] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2284.153894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2284.161029] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2284.168162] Call Trace: [ 2284.170615] [ 2284.172721] ? __warn+0x85/0x180 [ 2284.175958] ? __list_add_valid_or_report+0x7d/0xa0 [ 2284.180842] ? report_bug+0xfc/0x1d0 [ 2284.184422] ? prb_read_valid+0x1b/0x20 [ 2284.188272] ? handle_bug+0x44/0x70 [ 2284.191771] ? exc_invalid_op+0x17/0x60 [ 2284.195618] ? asm_exc_invalid_op+0x1a/0x20 [ 2284.199813] ? __list_add_valid_or_report+0x7d/0xa0 [ 2284.204692] ? __list_add_valid_or_report+0x7d/0xa0 [ 2284.209572] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2284.214472] ceph_add_cap+0x20b/0x620 [ceph] [ 2284.218777] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2284.223499] ? lock_acquire+0xcb/0x2b0 [ 2284.227260] ceph_fill_trace+0x143/0xf30 [ceph] [ 2284.231801] ? __mutex_lock+0x97/0xce0 [ 2284.235554] ? __mutex_lock+0xaec/0xce0 [ 2284.239399] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2284.243954] ? lock_downgrade+0xfd/0x250 [ 2284.247886] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2284.252276] ceph_con_process_message+0x74/0x150 [libceph] [ 2284.257774] ? calc_signature+0xdb/0x100 [libceph] [ 2284.262600] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2284.267932] ? process_scheduled_works+0x22a/0x510 [ 2284.272732] ? process_scheduled_works+0x22a/0x510 [ 2284.277526] ceph_con_workfn+0x344/0x750 [libceph] [ 2284.282343] ? process_scheduled_works+0x22a/0x510 [ 2284.287138] process_scheduled_works+0x2a4/0x510 [ 2284.291766] worker_thread+0x188/0x340 [ 2284.295523] ? rescuer_thread+0x310/0x310 [ 2284.299543] kthread+0xf5/0x130 [ 2284.302691] ? kthread_complete_and_exit+0x20/0x20 [ 2284.307492] ret_from_fork+0x31/0x40 [ 2284.311075] ? kthread_complete_and_exit+0x20/0x20 [ 2284.315872] ret_from_fork_asm+0x11/0x20 [ 2284.319808] [ 2284.321998] irq event stamp: 10547801 [ 2284.325661] hardirqs last enabled at (10547811): [] console_unlock+0x106/0x120 [ 2284.334530] hardirqs last disabled at (10547820): [] console_unlock+0xeb/0x120 [ 2284.343308] softirqs last enabled at (10547748): [] __do_softirq+0x2be/0x371 [ 2284.352003] softirqs last disabled at (10547743): [] irq_exit_rcu+0xb0/0xd0 [ 2284.360529] ---[ end trace 0000000000000000 ]--- [ 2284.365158] ------------[ cut here ]------------ [ 2284.369774] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2284.381421] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2284.390379] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2284.445849] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2284.454795] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2284.462275] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2284.467762] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2284.486508] RSP: 0018:ffffc90001f7fb78 EFLAGS: 00010286 [ 2284.491733] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2284.498858] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2284.505990] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006383 [ 2284.513124] R10: 000000000021d8d8 R11: ffffc90001f7fa08 R12: ffff888158a68750 [ 2284.520257] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2284.527390] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2284.535475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2284.541220] CR2: 00007f9bd2b76000 CR3: 0000000113532006 CR4: 00000000003706e0 [ 2284.548345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2284.555478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2284.562610] Call Trace: [ 2284.565057] [ 2284.567160] ? __warn+0x85/0x180 [ 2284.570394] ? __list_add_valid_or_report+0x7d/0xa0 [ 2284.575273] ? report_bug+0xfc/0x1d0 [ 2284.578854] ? prb_read_valid+0x1b/0x20 [ 2284.582694] ? handle_bug+0x44/0x70 [ 2284.586185] ? exc_invalid_op+0x17/0x60 [ 2284.590025] ? asm_exc_invalid_op+0x1a/0x20 [ 2284.594215] ? __list_add_valid_or_report+0x7d/0xa0 [ 2284.599090] ? __list_add_valid_or_report+0x7d/0xa0 [ 2284.603969] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2284.608859] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2284.614094] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2284.618896] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2284.623611] ? __d_lookup+0xd6/0x200 [ 2284.627188] lookup_fast+0x6d/0xf0 [ 2284.630593] walk_component+0x26/0x160 [ 2284.634348] link_path_walk+0x24d/0x380 [ 2284.638185] ? path_openat+0x80/0x9d0 [ 2284.641848] path_openat+0x8e/0x9d0 [ 2284.645341] do_filp_open+0xb8/0x120 [ 2284.648919] ? lock_release+0xc5/0x260 [ 2284.652671] ? _raw_spin_unlock+0x23/0x40 [ 2284.656681] ? alloc_fd+0x13f/0x200 [ 2284.660176] do_sys_openat2+0x249/0x290 [ 2284.664015] do_sys_open+0x4f/0x80 [ 2284.667421] do_syscall_64+0x38/0x80 [ 2284.670998] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2284.676051] RIP: 0033:0x7f48f0eed2f2 [ 2284.679631] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2284.698376] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2284.705944] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2284.713076] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2284.720209] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2284.727340] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2284.734465] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2284.741603] [ 2284.743791] irq event stamp: 12944847 [ 2284.747458] hardirqs last enabled at (12944857): [] console_unlock+0x106/0x120 [ 2284.756322] hardirqs last disabled at (12944866): [] console_unlock+0xeb/0x120 [ 2284.765104] softirqs last enabled at (12944784): [] __do_softirq+0x2be/0x371 [ 2284.773798] softirqs last disabled at (12944885): [] irq_exit_rcu+0xb0/0xd0 [ 2284.782316] ---[ end trace 0000000000000000 ]--- [ 2284.786938] ------------[ cut here ]------------ [ 2284.791554] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2284.803205] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2284.811903] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2284.867381] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2284.876514] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2284.883998] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2284.889509] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2284.894995] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2284.913743] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2284.918976] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2284.926110] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2284.933243] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000063c2 [ 2284.940383] R10: 000000000021eba0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2284.947517] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2284.954650] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2284.962737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2284.968482] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2284.975615] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2284.982747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2284.989879] Call Trace: [ 2284.992333] [ 2284.994439] ? __warn+0x85/0x180 [ 2284.997674] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.002560] ? report_bug+0xfc/0x1d0 [ 2285.006143] ? prb_read_valid+0x1b/0x20 [ 2285.009990] ? handle_bug+0x44/0x70 [ 2285.013489] ? exc_invalid_op+0x17/0x60 [ 2285.017329] ? asm_exc_invalid_op+0x1a/0x20 [ 2285.021526] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.026411] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.031290] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2285.036189] ceph_add_cap+0x20b/0x620 [ceph] [ 2285.040498] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2285.045233] ? lock_acquire+0xcb/0x2b0 [ 2285.049000] ceph_fill_trace+0x143/0xf30 [ceph] [ 2285.053548] ? __mutex_lock+0x97/0xce0 [ 2285.057311] ? __mutex_lock+0xaec/0xce0 [ 2285.061159] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2285.065718] ? lock_downgrade+0xfd/0x250 [ 2285.069662] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2285.074058] ceph_con_process_message+0x74/0x150 [libceph] [ 2285.079564] ? calc_signature+0xdb/0x100 [libceph] [ 2285.084393] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2285.089729] ? process_scheduled_works+0x22a/0x510 [ 2285.094536] ? process_scheduled_works+0x22a/0x510 [ 2285.099339] ceph_con_workfn+0x344/0x750 [libceph] [ 2285.104160] ? process_scheduled_works+0x22a/0x510 [ 2285.108952] process_scheduled_works+0x2a4/0x510 [ 2285.113584] worker_thread+0x188/0x340 [ 2285.117345] ? rescuer_thread+0x310/0x310 [ 2285.121366] kthread+0xf5/0x130 [ 2285.124522] ? kthread_complete_and_exit+0x20/0x20 [ 2285.129325] ret_from_fork+0x31/0x40 [ 2285.132912] ? kthread_complete_and_exit+0x20/0x20 [ 2285.137712] ret_from_fork_asm+0x11/0x20 [ 2285.141662] [ 2285.143858] irq event stamp: 10549825 [ 2285.147530] hardirqs last enabled at (10549835): [] console_unlock+0x106/0x120 [ 2285.156406] hardirqs last disabled at (10549844): [] console_unlock+0xeb/0x120 [ 2285.165192] softirqs last enabled at (10549792): [] __do_softirq+0x2be/0x371 [ 2285.173896] softirqs last disabled at (10549787): [] irq_exit_rcu+0xb0/0xd0 [ 2285.182421] ---[ end trace 0000000000000000 ]--- [ 2285.187062] ------------[ cut here ]------------ [ 2285.191689] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2285.203341] WARNING: CPU: 1 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2285.212297] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2285.267824] CPU: 1 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2285.276776] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2285.284257] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2285.289744] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2285.308493] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2285.313724] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2285.320857] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2285.327989] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006401 [ 2285.335123] R10: 000000000021fe40 R11: ffffc90001957950 R12: ffff888158a68750 [ 2285.342255] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2285.349388] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2285.357474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2285.363221] CR2: 00007f9bd20f7000 CR3: 000000016696c006 CR4: 00000000003706e0 [ 2285.370354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2285.377486] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2285.384618] Call Trace: [ 2285.387072] [ 2285.389179] ? __warn+0x85/0x180 [ 2285.392412] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.397292] ? report_bug+0xfc/0x1d0 [ 2285.400879] ? prb_read_valid+0x1b/0x20 [ 2285.404721] ? handle_bug+0x44/0x70 [ 2285.408220] ? exc_invalid_op+0x17/0x60 [ 2285.412058] ? asm_exc_invalid_op+0x1a/0x20 [ 2285.416255] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.421188] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2285.426144] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2285.431402] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2285.436217] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2285.440940] ? __d_lookup+0xd6/0x200 [ 2285.444527] lookup_fast+0x6d/0xf0 [ 2285.447939] walk_component+0x26/0x160 [ 2285.451696] link_path_walk+0x24d/0x380 [ 2285.455540] ? path_lookupat.isra.40+0x22/0x120 [ 2285.460076] path_lookupat.isra.40+0x6b/0x120 [ 2285.464443] filename_lookup+0xd4/0x190 [ 2285.468288] ? lock_acquire+0xcb/0x2b0 [ 2285.472042] ? __might_fault+0x45/0x80 [ 2285.475805] ? find_held_lock+0x2d/0x90 [ 2285.479651] ? __might_fault+0x45/0x80 [ 2285.483411] vfs_statx+0x89/0x150 [ 2285.486738] vfs_fstatat+0x5a/0xc0 [ 2285.490150] __do_sys_newstat+0x30/0x70 [ 2285.493994] ? lock_release+0xc5/0x260 [ 2285.497751] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2285.502639] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2285.507784] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2285.512926] do_syscall_64+0x38/0x80 [ 2285.516511] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2285.521563] RIP: 0033:0x7f48f0eeca79 [ 2285.525144] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2285.543887] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2285.551456] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2285.558589] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2285.565725] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2285.572862] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2285.579995] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2285.587142] [ 2285.589337] irq event stamp: 12835391 [ 2285.593004] hardirqs last enabled at (12835401): [] console_unlock+0x106/0x120 [ 2285.601868] hardirqs last disabled at (12835410): [] console_unlock+0xeb/0x120 [ 2285.610651] softirqs last enabled at (12835348): [] __do_softirq+0x2be/0x371 [ 2285.619341] softirqs last disabled at (12835343): [] irq_exit_rcu+0xb0/0xd0 [ 2285.627863] ---[ end trace 0000000000000000 ]--- [ 2285.632486] ------------[ cut here ]------------ [ 2285.637109] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2285.648755] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2285.657450] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2285.712926] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2285.722052] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2285.729532] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2285.735034] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2285.740520] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2285.759291] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2285.764523] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2285.771659] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2285.778795] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006445 [ 2285.785925] R10: 0000000000221580 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2285.793058] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2285.800191] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2285.808277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2285.814023] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2285.821156] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2285.828289] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2285.835421] Call Trace: [ 2285.837865] [ 2285.839973] ? __warn+0x85/0x180 [ 2285.843206] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.848085] ? report_bug+0xfc/0x1d0 [ 2285.851665] ? prb_read_valid+0x1b/0x20 [ 2285.855505] ? handle_bug+0x44/0x70 [ 2285.858997] ? exc_invalid_op+0x17/0x60 [ 2285.862835] ? asm_exc_invalid_op+0x1a/0x20 [ 2285.867027] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.871902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2285.876780] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2285.881672] ceph_add_cap+0x20b/0x620 [ceph] [ 2285.885957] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2285.890678] ? lock_acquire+0xcb/0x2b0 [ 2285.894432] ceph_fill_trace+0x143/0xf30 [ceph] [ 2285.898967] ? __mutex_lock+0x97/0xce0 [ 2285.902718] ? __mutex_lock+0xaec/0xce0 [ 2285.906562] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2285.911100] ? lock_downgrade+0xfd/0x250 [ 2285.915033] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2285.919411] ceph_con_process_message+0x74/0x150 [libceph] [ 2285.924912] ? calc_signature+0xdb/0x100 [libceph] [ 2285.929720] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2285.935053] ? process_scheduled_works+0x22a/0x510 [ 2285.939851] ? process_scheduled_works+0x22a/0x510 [ 2285.944641] ceph_con_workfn+0x344/0x750 [libceph] [ 2285.949444] ? process_scheduled_works+0x22a/0x510 [ 2285.954237] process_scheduled_works+0x2a4/0x510 [ 2285.958861] worker_thread+0x188/0x340 [ 2285.962617] ? rescuer_thread+0x310/0x310 [ 2285.966630] kthread+0xf5/0x130 [ 2285.969776] ? kthread_complete_and_exit+0x20/0x20 [ 2285.974568] ret_from_fork+0x31/0x40 [ 2285.978148] ? kthread_complete_and_exit+0x20/0x20 [ 2285.982941] ret_from_fork_asm+0x11/0x20 [ 2285.986874] [ 2285.989067] irq event stamp: 10551965 [ 2285.992732] hardirqs last enabled at (10551975): [] console_unlock+0x106/0x120 [ 2286.001601] hardirqs last disabled at (10551984): [] console_unlock+0xeb/0x120 [ 2286.010379] softirqs last enabled at (10551912): [] __do_softirq+0x2be/0x371 [ 2286.019073] softirqs last disabled at (10551907): [] irq_exit_rcu+0xb0/0xd0 [ 2286.027592] ---[ end trace 0000000000000000 ]--- [ 2286.032231] ------------[ cut here ]------------ [ 2286.036856] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2286.048503] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2286.057456] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2286.112918] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2286.121868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2286.129348] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2286.134834] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2286.153580] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2286.158807] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2286.165940] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2286.173071] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006484 [ 2286.180196] R10: 0000000000222830 R11: ffffc90001ff7950 R12: ffff888158a68750 [ 2286.187327] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2286.194453] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2286.202539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2286.208285] CR2: 00007f9bd2102000 CR3: 0000000120cc8003 CR4: 00000000003706e0 [ 2286.215417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2286.222542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2286.229674] Call Trace: [ 2286.232120] [ 2286.234225] ? __warn+0x85/0x180 [ 2286.237459] ? __list_add_valid_or_report+0x7d/0xa0 [ 2286.242338] ? report_bug+0xfc/0x1d0 [ 2286.245918] ? prb_read_valid+0x1b/0x20 [ 2286.249756] ? handle_bug+0x44/0x70 [ 2286.253247] ? exc_invalid_op+0x17/0x60 [ 2286.257079] ? asm_exc_invalid_op+0x1a/0x20 [ 2286.261268] ? __list_add_valid_or_report+0x7d/0xa0 [ 2286.266145] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2286.271035] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2286.276269] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2286.281070] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2286.285785] ? __d_lookup+0xd6/0x200 [ 2286.289364] lookup_fast+0x6d/0xf0 [ 2286.292770] walk_component+0x26/0x160 [ 2286.296523] link_path_walk+0x24d/0x380 [ 2286.300362] ? path_lookupat.isra.40+0x22/0x120 [ 2286.304898] path_lookupat.isra.40+0x6b/0x120 [ 2286.309257] filename_lookup+0xd4/0x190 [ 2286.313097] ? lock_acquire+0xcb/0x2b0 [ 2286.316845] ? __might_fault+0x45/0x80 [ 2286.320598] ? find_held_lock+0x2d/0x90 [ 2286.324437] ? __might_fault+0x45/0x80 [ 2286.328193] vfs_statx+0x89/0x150 [ 2286.331514] vfs_fstatat+0x5a/0xc0 [ 2286.334917] __do_sys_newstat+0x30/0x70 [ 2286.338754] ? lock_release+0xc5/0x260 [ 2286.342501] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2286.347380] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2286.352520] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2286.357659] do_syscall_64+0x38/0x80 [ 2286.361237] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2286.366289] RIP: 0033:0x7f48f0eeca79 [ 2286.369870] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2286.388617] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2286.396181] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2286.403305] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2286.410429] RBP: 00007ffc53d60520 R08: 00000000000029f6 R09: 00000000020767c0 [ 2286.417553] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2286.424686] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2286.431816] [ 2286.434010] irq event stamp: 12959083 [ 2286.437671] hardirqs last enabled at (12959101): [] console_unlock+0x106/0x120 [ 2286.446537] hardirqs last disabled at (12959122): [] console_unlock+0xeb/0x120 [ 2286.455318] softirqs last enabled at (12959136): [] __do_softirq+0x2be/0x371 [ 2286.464008] softirqs last disabled at (12959131): [] irq_exit_rcu+0xb0/0xd0 [ 2286.472528] ---[ end trace 0000000000000000 ]--- [ 2286.477148] ------------[ cut here ]------------ [ 2286.481769] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2286.493432] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2286.502126] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2286.557621] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2286.566746] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2286.574243] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2286.579753] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2286.585258] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2286.604023] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2286.609264] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2286.616425] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2286.623557] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000064c8 [ 2286.630704] R10: 0000000000223af0 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2286.637831] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2286.644981] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2286.653084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2286.658846] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2286.665989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2286.673131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2286.680282] Call Trace: [ 2286.682750] [ 2286.684865] ? __warn+0x85/0x180 [ 2286.688117] ? __list_add_valid_or_report+0x7d/0xa0 [ 2286.693014] ? report_bug+0xfc/0x1d0 [ 2286.696611] ? prb_read_valid+0x1b/0x20 [ 2286.700450] ? handle_bug+0x44/0x70 [ 2286.703942] ? exc_invalid_op+0x17/0x60 [ 2286.707797] ? asm_exc_invalid_op+0x1a/0x20 [ 2286.712023] ? __list_add_valid_or_report+0x7d/0xa0 [ 2286.716898] ? __list_add_valid_or_report+0x7d/0xa0 [ 2286.721795] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2286.726718] ceph_add_cap+0x20b/0x620 [ceph] [ 2286.731041] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2286.735780] ? lock_acquire+0xcb/0x2b0 [ 2286.739534] ceph_fill_trace+0x143/0xf30 [ceph] [ 2286.744069] ? __mutex_lock+0x97/0xce0 [ 2286.747837] ? __mutex_lock+0xaec/0xce0 [ 2286.751705] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2286.756262] ? lock_downgrade+0xfd/0x250 [ 2286.760228] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2286.764609] ceph_con_process_message+0x74/0x150 [libceph] [ 2286.770106] ? calc_signature+0xdb/0x100 [libceph] [ 2286.774933] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2286.780274] ? process_scheduled_works+0x22a/0x510 [ 2286.785083] ? process_scheduled_works+0x22a/0x510 [ 2286.789872] ceph_con_workfn+0x344/0x750 [libceph] [ 2286.794709] ? process_scheduled_works+0x22a/0x510 [ 2286.799519] process_scheduled_works+0x2a4/0x510 [ 2286.804160] worker_thread+0x188/0x340 [ 2286.807908] ? rescuer_thread+0x310/0x310 [ 2286.811939] kthread+0xf5/0x130 [ 2286.815104] ? kthread_complete_and_exit+0x20/0x20 [ 2286.819914] ret_from_fork+0x31/0x40 [ 2286.823509] ? kthread_complete_and_exit+0x20/0x20 [ 2286.828301] ret_from_fork_asm+0x11/0x20 [ 2286.832269] [ 2286.834454] irq event stamp: 10555173 [ 2286.838119] hardirqs last enabled at (10555183): [] console_unlock+0x106/0x120 [ 2286.847005] hardirqs last disabled at (10555192): [] console_unlock+0xeb/0x120 [ 2286.855801] softirqs last enabled at (10555120): [] __do_softirq+0x2be/0x371 [ 2286.864513] softirqs last disabled at (10555115): [] irq_exit_rcu+0xb0/0xd0 [ 2286.873031] ---[ end trace 0000000000000000 ]--- [ 2286.877695] ------------[ cut here ]------------ [ 2286.882314] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2286.893958] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2286.902654] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2286.958134] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2286.967258] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2286.974736] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2286.980231] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2286.985716] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2287.004466] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2287.009689] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2287.016823] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2287.023958] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006507 [ 2287.031089] R10: 0000000000224d48 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2287.038222] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2287.045353] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2287.053440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2287.059185] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2287.066319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2287.073453] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2287.080585] Call Trace: [ 2287.083037] [ 2287.085144] ? __warn+0x85/0x180 [ 2287.088377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.093255] ? report_bug+0xfc/0x1d0 [ 2287.096836] ? prb_read_valid+0x1b/0x20 [ 2287.100676] ? handle_bug+0x44/0x70 [ 2287.104166] ? exc_invalid_op+0x17/0x60 [ 2287.108007] ? asm_exc_invalid_op+0x1a/0x20 [ 2287.112199] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.117083] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.121961] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2287.126851] ceph_add_cap+0x20b/0x620 [ceph] [ 2287.131136] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2287.135860] ? lock_acquire+0xcb/0x2b0 [ 2287.139620] ceph_fill_trace+0x143/0xf30 [ceph] [ 2287.144156] ? __mutex_lock+0x97/0xce0 [ 2287.147907] ? __mutex_lock+0xaec/0xce0 [ 2287.151751] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2287.156298] ? lock_downgrade+0xfd/0x250 [ 2287.160229] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2287.164609] ceph_con_process_message+0x74/0x150 [libceph] [ 2287.170106] ? calc_signature+0xdb/0x100 [libceph] [ 2287.174916] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2287.180241] ? process_scheduled_works+0x22a/0x510 [ 2287.185041] ? process_scheduled_works+0x22a/0x510 [ 2287.189829] ceph_con_workfn+0x344/0x750 [libceph] [ 2287.194635] ? process_scheduled_works+0x22a/0x510 [ 2287.199424] process_scheduled_works+0x2a4/0x510 [ 2287.204048] worker_thread+0x188/0x340 [ 2287.207797] ? rescuer_thread+0x310/0x310 [ 2287.211810] kthread+0xf5/0x130 [ 2287.214957] ? kthread_complete_and_exit+0x20/0x20 [ 2287.219748] ret_from_fork+0x31/0x40 [ 2287.223327] ? kthread_complete_and_exit+0x20/0x20 [ 2287.228120] ret_from_fork_asm+0x11/0x20 [ 2287.232054] [ 2287.234248] irq event stamp: 10556287 [ 2287.237912] hardirqs last enabled at (10556297): [] console_unlock+0x106/0x120 [ 2287.246780] hardirqs last disabled at (10556306): [] console_unlock+0xeb/0x120 [ 2287.255559] softirqs last enabled at (10556234): [] __do_softirq+0x2be/0x371 [ 2287.264254] softirqs last disabled at (10556325): [] irq_exit_rcu+0xb0/0xd0 [ 2287.272772] ---[ end trace 0000000000000000 ]--- [ 2287.277392] ------------[ cut here ]------------ [ 2287.282008] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2287.293656] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2287.302610] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2287.358072] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2287.367023] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2287.374501] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2287.379987] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2287.398734] RSP: 0018:ffffc90001fdfac0 EFLAGS: 00010282 [ 2287.403961] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2287.411093] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2287.418226] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006546 [ 2287.425359] R10: 0000000000225fc8 R11: ffffc90001fdf950 R12: ffff8881d1146920 [ 2287.432491] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2287.439615] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2287.447702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2287.453447] CR2: 00007ff232607000 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2287.460571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2287.467695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2287.474825] Call Trace: [ 2287.477273] [ 2287.479379] ? __warn+0x85/0x180 [ 2287.482613] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.487494] ? report_bug+0xfc/0x1d0 [ 2287.491071] ? prb_read_valid+0x1b/0x20 [ 2287.494911] ? handle_bug+0x44/0x70 [ 2287.498401] ? exc_invalid_op+0x17/0x60 [ 2287.502233] ? asm_exc_invalid_op+0x1a/0x20 [ 2287.506426] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.511310] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2287.516199] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2287.521442] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2287.526241] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2287.530957] ? __d_lookup+0xd6/0x200 [ 2287.534537] lookup_fast+0x6d/0xf0 [ 2287.537942] walk_component+0x26/0x160 [ 2287.541698] link_path_walk+0x24d/0x380 [ 2287.545532] ? path_lookupat.isra.40+0x22/0x120 [ 2287.550069] path_lookupat.isra.40+0x6b/0x120 [ 2287.554428] filename_lookup+0xd4/0x190 [ 2287.558269] ? lock_acquire+0xcb/0x2b0 [ 2287.562016] ? __might_fault+0x45/0x80 [ 2287.565771] ? find_held_lock+0x2d/0x90 [ 2287.569609] ? __might_fault+0x45/0x80 [ 2287.573366] vfs_statx+0x89/0x150 [ 2287.576684] vfs_fstatat+0x5a/0xc0 [ 2287.580089] __do_sys_newstat+0x30/0x70 [ 2287.583927] ? lock_release+0xc5/0x260 [ 2287.587681] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2287.592557] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2287.597699] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2287.602838] do_syscall_64+0x38/0x80 [ 2287.606418] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2287.611468] RIP: 0033:0x7f48f0eeca79 [ 2287.615051] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2287.633798] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2287.641363] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2287.648495] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2287.655627] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2287.662759] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2287.669883] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2287.677014] [ 2287.679202] irq event stamp: 12860937 [ 2287.682866] hardirqs last enabled at (12860947): [] console_unlock+0x106/0x120 [ 2287.691733] hardirqs last disabled at (12860956): [] console_unlock+0xeb/0x120 [ 2287.700513] softirqs last enabled at (12860970): [] __do_softirq+0x2be/0x371 [ 2287.709205] softirqs last disabled at (12860965): [] irq_exit_rcu+0xb0/0xd0 [ 2287.717726] ---[ end trace 0000000000000000 ]--- [ 2287.722346] ------------[ cut here ]------------ [ 2287.726965] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2287.738615] WARNING: CPU: 1 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2287.747567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2287.803111] CPU: 1 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2287.812066] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2287.819562] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2287.825047] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2287.843811] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2287.849055] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2287.856206] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2287.863355] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000658a [ 2287.870506] R10: 00000000002272b8 R11: ffffc90001fc7950 R12: ffff8881c5350750 [ 2287.877640] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2287.884772] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2287.892875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2287.898638] CR2: 00007f9bd20f7000 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2287.905772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2287.912905] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2287.920054] Call Trace: [ 2287.922526] [ 2287.924633] ? __warn+0x85/0x180 [ 2287.927865] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.932745] ? report_bug+0xfc/0x1d0 [ 2287.936339] ? prb_read_valid+0x1b/0x20 [ 2287.940217] ? handle_bug+0x44/0x70 [ 2287.943706] ? exc_invalid_op+0x17/0x60 [ 2287.947548] ? asm_exc_invalid_op+0x1a/0x20 [ 2287.951773] ? __list_add_valid_or_report+0x7d/0xa0 [ 2287.956657] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2287.961557] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2287.966825] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2287.971636] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2287.976374] ? __d_lookup+0xd6/0x200 [ 2287.979973] lookup_fast+0x6d/0xf0 [ 2287.983386] walk_component+0x26/0x160 [ 2287.987174] link_path_walk+0x24d/0x380 [ 2287.991009] ? path_lookupat.isra.40+0x22/0x120 [ 2287.995566] path_lookupat.isra.40+0x6b/0x120 [ 2287.999949] filename_lookup+0xd4/0x190 [ 2288.003811] ? lock_acquire+0xcb/0x2b0 [ 2288.007565] ? __might_fault+0x45/0x80 [ 2288.011334] ? find_held_lock+0x2d/0x90 [ 2288.015174] ? __might_fault+0x45/0x80 [ 2288.018967] vfs_statx+0x89/0x150 [ 2288.022311] vfs_fstatat+0x5a/0xc0 [ 2288.025723] __do_sys_newstat+0x30/0x70 [ 2288.029562] ? lock_release+0xc5/0x260 [ 2288.033335] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2288.038218] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2288.043376] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2288.048536] do_syscall_64+0x38/0x80 [ 2288.052137] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2288.057190] RIP: 0033:0x7f48f0eeca79 [ 2288.060788] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2288.079549] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2288.087116] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2288.094266] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2288.101417] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2288.108566] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2288.115698] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2288.122911] [ 2288.125169] irq event stamp: 12947469 [ 2288.128838] hardirqs last enabled at (12947479): [] console_unlock+0x106/0x120 [ 2288.137737] hardirqs last disabled at (12947488): [] console_unlock+0xeb/0x120 [ 2288.146540] softirqs last enabled at (12947466): [] __do_softirq+0x2be/0x371 [ 2288.155237] softirqs last disabled at (12947453): [] irq_exit_rcu+0xb0/0xd0 [ 2288.163775] ---[ end trace 0000000000000000 ]--- [ 2288.168436] ------------[ cut here ]------------ [ 2288.173055] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2288.184700] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2288.193399] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2288.248897] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2288.258043] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2288.265540] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2288.271041] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2288.276546] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2288.295295] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2288.300535] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2288.307669] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2288.314801] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000065ce [ 2288.321953] R10: 0000000000228608 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2288.329101] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2288.336252] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2288.344355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.350117] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2288.357259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2288.364410] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2288.371561] Call Trace: [ 2288.374012] [ 2288.376129] ? __warn+0x85/0x180 [ 2288.379377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2288.384276] ? report_bug+0xfc/0x1d0 [ 2288.387871] ? prb_read_valid+0x1b/0x20 [ 2288.391746] ? handle_bug+0x44/0x70 [ 2288.395255] ? exc_invalid_op+0x17/0x60 [ 2288.399113] ? asm_exc_invalid_op+0x1a/0x20 [ 2288.403336] ? __list_add_valid_or_report+0x7d/0xa0 [ 2288.408212] ? __list_add_valid_or_report+0x7d/0xa0 [ 2288.413108] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2288.418015] ceph_add_cap+0x20b/0x620 [ceph] [ 2288.422336] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2288.427077] ? lock_acquire+0xcb/0x2b0 [ 2288.430846] ceph_fill_trace+0x143/0xf30 [ceph] [ 2288.435384] ? __mutex_lock+0x97/0xce0 [ 2288.439153] ? __mutex_lock+0xaec/0xce0 [ 2288.443011] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2288.447569] ? lock_downgrade+0xfd/0x250 [ 2288.451500] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2288.455897] ceph_con_process_message+0x74/0x150 [libceph] [ 2288.461392] ? calc_signature+0xdb/0x100 [libceph] [ 2288.466214] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2288.471563] ? process_scheduled_works+0x22a/0x510 [ 2288.476379] ? process_scheduled_works+0x22a/0x510 [ 2288.481170] ceph_con_workfn+0x344/0x750 [libceph] [ 2288.485990] ? process_scheduled_works+0x22a/0x510 [ 2288.490819] process_scheduled_works+0x2a4/0x510 [ 2288.495466] worker_thread+0x188/0x340 [ 2288.499214] ? rescuer_thread+0x310/0x310 [ 2288.503244] kthread+0xf5/0x130 [ 2288.506408] ? kthread_complete_and_exit+0x20/0x20 [ 2288.511218] ret_from_fork+0x31/0x40 [ 2288.514816] ? kthread_complete_and_exit+0x20/0x20 [ 2288.519626] ret_from_fork_asm+0x11/0x20 [ 2288.523557] [ 2288.525754] irq event stamp: 10559443 [ 2288.529434] hardirqs last enabled at (10559453): [] console_unlock+0x106/0x120 [ 2288.538316] hardirqs last disabled at (10559464): [] console_unlock+0xeb/0x120 [ 2288.547090] softirqs last enabled at (10559410): [] __do_softirq+0x2be/0x371 [ 2288.555799] softirqs last disabled at (10559483): [] irq_exit_rcu+0xb0/0xd0 [ 2288.564337] ---[ end trace 0000000000000000 ]--- [ 2288.569145] ------------[ cut here ]------------ [ 2288.573772] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2288.585462] WARNING: CPU: 1 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2288.594418] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2288.649958] CPU: 1 PID: 78682 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2288.658928] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2288.666425] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2288.671933] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2288.690684] RSP: 0018:ffffc90001fd7b78 EFLAGS: 00010286 [ 2288.695908] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2288.703059] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2288.710208] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000660d [ 2288.717358] R10: 0000000000229c88 R11: ffffc90001fd7a08 R12: ffff888158a68750 [ 2288.724509] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2288.731642] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2288.739729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.745492] CR2: 00007f9bd20f7000 CR3: 0000000120c1e004 CR4: 00000000003706e0 [ 2288.752625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2288.759762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2288.766908] Call Trace: [ 2288.769378] [ 2288.771502] ? __warn+0x85/0x180 [ 2288.774752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2288.779649] ? report_bug+0xfc/0x1d0 [ 2288.783228] ? prb_read_valid+0x1b/0x20 [ 2288.787105] ? handle_bug+0x44/0x70 [ 2288.790594] ? exc_invalid_op+0x17/0x60 [ 2288.794435] ? asm_exc_invalid_op+0x1a/0x20 [ 2288.798654] ? __list_add_valid_or_report+0x7d/0xa0 [ 2288.803537] ? __list_add_valid_or_report+0x7d/0xa0 [ 2288.808411] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2288.813330] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2288.818575] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2288.823400] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2288.828141] ? __d_lookup+0xd6/0x200 [ 2288.831722] lookup_fast+0x6d/0xf0 [ 2288.835144] walk_component+0x26/0x160 [ 2288.838914] link_path_walk+0x24d/0x380 [ 2288.842763] ? path_openat+0x80/0x9d0 [ 2288.846467] path_openat+0x8e/0x9d0 [ 2288.849988] do_filp_open+0xb8/0x120 [ 2288.853576] ? lock_release+0xc5/0x260 [ 2288.857351] ? _raw_spin_unlock+0x23/0x40 [ 2288.861366] ? alloc_fd+0x13f/0x200 [ 2288.864899] do_sys_openat2+0x249/0x290 [ 2288.868782] do_sys_open+0x4f/0x80 [ 2288.872213] do_syscall_64+0x38/0x80 [ 2288.875790] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2288.880860] RIP: 0033:0x7f48f0ef2805 [ 2288.884475] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2288.903218] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2288.910803] RAX: ffffffffffffffda RBX: 000000000207815f RCX: 00007f48f0ef2805 [ 2288.917954] RDX: 0000000000090800 RSI: 0000000002078140 RDI: 00000000ffffff9c [ 2288.925104] RBP: 0000000002078140 R08: 0000000002076674 R09: 0000000000000000 [ 2288.932252] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2288.939402] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2288.946567] [ 2288.948764] irq event stamp: 12995623 [ 2288.952446] hardirqs last enabled at (12995633): [] console_unlock+0x106/0x120 [ 2288.961329] hardirqs last disabled at (12995642): [] console_unlock+0xeb/0x120 [ 2288.970127] softirqs last enabled at (12995494): [] __do_softirq+0x2be/0x371 [ 2288.978837] softirqs last disabled at (12995489): [] irq_exit_rcu+0xb0/0xd0 [ 2288.987374] ---[ end trace 0000000000000000 ]--- [ 2288.992029] ------------[ cut here ]------------ [ 2288.996644] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2289.008298] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2289.016987] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2289.072482] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2289.081606] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2289.089085] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2289.094599] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2289.100084] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2289.118847] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2289.124090] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2289.131234] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2289.138383] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000664c [ 2289.145533] R10: 000000000022ae60 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2289.152667] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2289.159800] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2289.167903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2289.173665] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2289.180798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2289.187948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2289.195090] Call Trace: [ 2289.197552] [ 2289.199659] ? __warn+0x85/0x180 [ 2289.202892] ? __list_add_valid_or_report+0x7d/0xa0 [ 2289.207787] ? report_bug+0xfc/0x1d0 [ 2289.211385] ? prb_read_valid+0x1b/0x20 [ 2289.215242] ? handle_bug+0x44/0x70 [ 2289.218757] ? exc_invalid_op+0x17/0x60 [ 2289.222607] ? asm_exc_invalid_op+0x1a/0x20 [ 2289.226815] ? __list_add_valid_or_report+0x7d/0xa0 [ 2289.231692] ? __list_add_valid_or_report+0x7d/0xa0 [ 2289.236570] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2289.241477] ceph_add_cap+0x20b/0x620 [ceph] [ 2289.245780] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2289.250503] ? lock_acquire+0xcb/0x2b0 [ 2289.254273] ceph_fill_trace+0x143/0xf30 [ceph] [ 2289.258809] ? __mutex_lock+0x97/0xce0 [ 2289.262578] ? __mutex_lock+0xaec/0xce0 [ 2289.266421] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2289.270984] ? lock_downgrade+0xfd/0x250 [ 2289.274933] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2289.279322] ceph_con_process_message+0x74/0x150 [libceph] [ 2289.284816] ? calc_signature+0xdb/0x100 [libceph] [ 2289.289639] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2289.294980] ? process_scheduled_works+0x22a/0x510 [ 2289.299777] ? process_scheduled_works+0x22a/0x510 [ 2289.304586] ceph_con_workfn+0x344/0x750 [libceph] [ 2289.309390] ? process_scheduled_works+0x22a/0x510 [ 2289.314199] process_scheduled_works+0x2a4/0x510 [ 2289.318857] worker_thread+0x188/0x340 [ 2289.322607] ? rescuer_thread+0x310/0x310 [ 2289.326617] kthread+0xf5/0x130 [ 2289.329765] ? kthread_complete_and_exit+0x20/0x20 [ 2289.334575] ret_from_fork+0x31/0x40 [ 2289.338153] ? kthread_complete_and_exit+0x20/0x20 [ 2289.342963] ret_from_fork_asm+0x11/0x20 [ 2289.346932] [ 2289.349126] irq event stamp: 10561451 [ 2289.352808] hardirqs last enabled at (10561461): [] console_unlock+0x106/0x120 [ 2289.361692] hardirqs last disabled at (10561470): [] console_unlock+0xeb/0x120 [ 2289.370481] softirqs last enabled at (10561398): [] __do_softirq+0x2be/0x371 [ 2289.379173] softirqs last disabled at (10561393): [] irq_exit_rcu+0xb0/0xd0 [ 2289.387712] ---[ end trace 0000000000000000 ]--- [ 2289.392364] ------------[ cut here ]------------ [ 2289.396986] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2289.408629] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2289.417326] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2289.472827] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2289.481958] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2289.489434] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2289.494927] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2289.500414] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2289.519161] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2289.524386] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2289.531519] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2289.538654] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000668b [ 2289.545785] R10: 000000000022c150 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2289.552917] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2289.560050] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2289.568137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2289.573881] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2289.581014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2289.588149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2289.595280] Call Trace: [ 2289.597725] [ 2289.599831] ? __warn+0x85/0x180 [ 2289.603066] ? __list_add_valid_or_report+0x7d/0xa0 [ 2289.607944] ? report_bug+0xfc/0x1d0 [ 2289.611521] ? prb_read_valid+0x1b/0x20 [ 2289.615364] ? handle_bug+0x44/0x70 [ 2289.618856] ? exc_invalid_op+0x17/0x60 [ 2289.622696] ? asm_exc_invalid_op+0x1a/0x20 [ 2289.626886] ? __list_add_valid_or_report+0x7d/0xa0 [ 2289.631762] ? __list_add_valid_or_report+0x7d/0xa0 [ 2289.636639] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2289.641530] ceph_add_cap+0x20b/0x620 [ceph] [ 2289.645814] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2289.650529] ? lock_acquire+0xcb/0x2b0 [ 2289.654282] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2289.658733] ? __mutex_lock+0x97/0xce0 [ 2289.662484] ? __mutex_lock+0xaec/0xce0 [ 2289.666326] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2289.670875] ? lock_downgrade+0xfd/0x250 [ 2289.674805] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2289.679185] ceph_con_process_message+0x74/0x150 [libceph] [ 2289.684678] ? calc_signature+0xdb/0x100 [libceph] [ 2289.689482] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2289.694814] ? process_scheduled_works+0x22a/0x510 [ 2289.699607] ? process_scheduled_works+0x22a/0x510 [ 2289.704397] ceph_con_workfn+0x344/0x750 [libceph] [ 2289.709200] ? process_scheduled_works+0x22a/0x510 [ 2289.713990] process_scheduled_works+0x2a4/0x510 [ 2289.718616] worker_thread+0x188/0x340 [ 2289.722371] ? rescuer_thread+0x310/0x310 [ 2289.726384] kthread+0xf5/0x130 [ 2289.729533] ? kthread_complete_and_exit+0x20/0x20 [ 2289.734324] ret_from_fork+0x31/0x40 [ 2289.737901] ? kthread_complete_and_exit+0x20/0x20 [ 2289.742696] ret_from_fork_asm+0x11/0x20 [ 2289.746629] [ 2289.748822] irq event stamp: 10562617 [ 2289.752489] hardirqs last enabled at (10562627): [] console_unlock+0x106/0x120 [ 2289.761354] hardirqs last disabled at (10562636): [] console_unlock+0xeb/0x120 [ 2289.770134] softirqs last enabled at (10562564): [] __do_softirq+0x2be/0x371 [ 2289.778827] softirqs last disabled at (10562559): [] irq_exit_rcu+0xb0/0xd0 [ 2289.787345] ---[ end trace 0000000000000000 ]--- [ 2289.791975] ------------[ cut here ]------------ [ 2289.796596] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2289.808238] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2289.816927] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2289.872406] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2289.881529] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2289.889008] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2289.894504] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2289.899988] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2289.918736] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2289.923961] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2289.931096] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2289.938228] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000066ca [ 2289.945361] R10: 000000000022d258 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2289.952492] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2289.959628] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2289.967712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2289.973458] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2289.980589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2289.987714] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2289.994848] Call Trace: [ 2289.997300] [ 2289.999407] ? __warn+0x85/0x180 [ 2290.002641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.007520] ? report_bug+0xfc/0x1d0 [ 2290.011098] ? prb_read_valid+0x1b/0x20 [ 2290.014939] ? handle_bug+0x44/0x70 [ 2290.018431] ? exc_invalid_op+0x17/0x60 [ 2290.022269] ? asm_exc_invalid_op+0x1a/0x20 [ 2290.026469] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.031353] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.036231] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2290.041123] ceph_add_cap+0x20b/0x620 [ceph] [ 2290.045408] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2290.050137] ceph_fill_trace+0x143/0xf30 [ceph] [ 2290.054677] ? __mutex_lock+0x97/0xce0 [ 2290.058429] ? __mutex_lock+0xaec/0xce0 [ 2290.062268] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2290.066811] ? lock_downgrade+0xfd/0x250 [ 2290.070740] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2290.075119] ceph_con_process_message+0x74/0x150 [libceph] [ 2290.080615] ? calc_signature+0xdb/0x100 [libceph] [ 2290.085420] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2290.090749] ? process_scheduled_works+0x22a/0x510 [ 2290.095543] ? process_scheduled_works+0x22a/0x510 [ 2290.100333] ceph_con_workfn+0x344/0x750 [libceph] [ 2290.105135] ? process_scheduled_works+0x22a/0x510 [ 2290.109926] process_scheduled_works+0x2a4/0x510 [ 2290.114551] worker_thread+0x188/0x340 [ 2290.118309] ? rescuer_thread+0x310/0x310 [ 2290.122320] kthread+0xf5/0x130 [ 2290.125469] ? kthread_complete_and_exit+0x20/0x20 [ 2290.130262] ret_from_fork+0x31/0x40 [ 2290.133838] ? kthread_complete_and_exit+0x20/0x20 [ 2290.138632] ret_from_fork_asm+0x11/0x20 [ 2290.142565] [ 2290.144759] irq event stamp: 10563675 [ 2290.148424] hardirqs last enabled at (10563685): [] console_unlock+0x106/0x120 [ 2290.157290] hardirqs last disabled at (10563694): [] console_unlock+0xeb/0x120 [ 2290.166069] softirqs last enabled at (10563632): [] __do_softirq+0x2be/0x371 [ 2290.174764] softirqs last disabled at (10563627): [] irq_exit_rcu+0xb0/0xd0 [ 2290.183281] ---[ end trace 0000000000000000 ]--- [ 2290.187902] ------------[ cut here ]------------ [ 2290.192522] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2290.204172] WARNING: CPU: 2 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2290.213133] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2290.268603] CPU: 2 PID: 78681 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2290.277563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2290.285039] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2290.290528] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2290.309272] RSP: 0018:ffffc90001fcfb78 EFLAGS: 00010286 [ 2290.314499] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2290.321632] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2290.328765] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006708 [ 2290.335898] R10: 000000000022e470 R11: ffffc90001fcfa08 R12: ffff888158a68750 [ 2290.343030] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2290.350163] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2290.358250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2290.363994] CR2: 00007ff232607000 CR3: 000000011aeae003 CR4: 00000000003706e0 [ 2290.371128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2290.378260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2290.385394] Call Trace: [ 2290.387846] [ 2290.389954] ? __warn+0x85/0x180 [ 2290.393185] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.398068] ? report_bug+0xfc/0x1d0 [ 2290.401654] ? prb_read_valid+0x1b/0x20 [ 2290.405497] ? handle_bug+0x44/0x70 [ 2290.408992] ? exc_invalid_op+0x17/0x60 [ 2290.412834] ? asm_exc_invalid_op+0x1a/0x20 [ 2290.417027] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.421910] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.426799] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2290.431706] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2290.436948] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2290.441768] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2290.446489] ? __d_lookup+0xd6/0x200 [ 2290.450070] lookup_fast+0x6d/0xf0 [ 2290.453481] walk_component+0x26/0x160 [ 2290.457237] link_path_walk+0x24d/0x380 [ 2290.461080] ? path_openat+0x80/0x9d0 [ 2290.464755] path_openat+0x8e/0x9d0 [ 2290.468255] do_filp_open+0xb8/0x120 [ 2290.471846] ? lock_release+0xc5/0x260 [ 2290.475602] ? _raw_spin_unlock+0x23/0x40 [ 2290.479621] ? alloc_fd+0x13f/0x200 [ 2290.483116] do_sys_openat2+0x249/0x290 [ 2290.486964] do_sys_open+0x4f/0x80 [ 2290.490376] do_syscall_64+0x38/0x80 [ 2290.493958] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2290.499015] RIP: 0033:0x7f48f0ef2805 [ 2290.502597] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2290.521340] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2290.528910] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2290.536043] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2290.543174] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2290.550304] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2290.557440] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2290.564583] [ 2290.566773] irq event stamp: 12912207 [ 2290.570437] hardirqs last enabled at (12912217): [] console_unlock+0x106/0x120 [ 2290.579304] hardirqs last disabled at (12912226): [] console_unlock+0xeb/0x120 [ 2290.588082] softirqs last enabled at (12912154): [] __do_softirq+0x2be/0x371 [ 2290.596781] softirqs last disabled at (12912245): [] irq_exit_rcu+0xb0/0xd0 [ 2290.605304] ---[ end trace 0000000000000000 ]--- [ 2290.609929] ------------[ cut here ]------------ [ 2290.614542] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2290.626205] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2290.634904] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2290.690398] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2290.699540] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2290.707018] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2290.712529] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2290.718016] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2290.736785] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2290.742023] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2290.749173] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2290.756324] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006747 [ 2290.763497] R10: 000000000022f6b0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2290.770624] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2290.777757] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2290.785842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2290.791605] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2290.798739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2290.805873] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2290.813022] Call Trace: [ 2290.815506] [ 2290.817608] ? __warn+0x85/0x180 [ 2290.820839] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.825738] ? report_bug+0xfc/0x1d0 [ 2290.829319] ? prb_read_valid+0x1b/0x20 [ 2290.833173] ? handle_bug+0x44/0x70 [ 2290.836682] ? exc_invalid_op+0x17/0x60 [ 2290.840522] ? asm_exc_invalid_op+0x1a/0x20 [ 2290.844712] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.849588] ? __list_add_valid_or_report+0x7d/0xa0 [ 2290.854466] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2290.859382] ceph_add_cap+0x20b/0x620 [ceph] [ 2290.863669] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2290.868408] ? lock_acquire+0xcb/0x2b0 [ 2290.872177] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2290.876628] ? __mutex_lock+0x97/0xce0 [ 2290.880379] ? __mutex_lock+0xaec/0xce0 [ 2290.884238] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2290.888797] ? lock_downgrade+0xfd/0x250 [ 2290.892747] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2290.897143] ceph_con_process_message+0x74/0x150 [libceph] [ 2290.902638] ? calc_signature+0xdb/0x100 [libceph] [ 2290.907450] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2290.912797] ? process_scheduled_works+0x22a/0x510 [ 2290.917595] ? process_scheduled_works+0x22a/0x510 [ 2290.922388] ceph_con_workfn+0x344/0x750 [libceph] [ 2290.927209] ? process_scheduled_works+0x22a/0x510 [ 2290.932017] process_scheduled_works+0x2a4/0x510 [ 2290.936661] worker_thread+0x188/0x340 [ 2290.940418] ? rescuer_thread+0x310/0x310 [ 2290.944446] kthread+0xf5/0x130 [ 2290.947610] ? kthread_complete_and_exit+0x20/0x20 [ 2290.952402] ret_from_fork+0x31/0x40 [ 2290.955999] ? kthread_complete_and_exit+0x20/0x20 [ 2290.960810] ret_from_fork_asm+0x11/0x20 [ 2290.964760] [ 2290.966953] irq event stamp: 10565727 [ 2290.970627] hardirqs last enabled at (10565737): [] console_unlock+0x106/0x120 [ 2290.979509] hardirqs last disabled at (10565746): [] console_unlock+0xeb/0x120 [ 2290.988291] softirqs last enabled at (10565694): [] __do_softirq+0x2be/0x371 [ 2290.997001] softirqs last disabled at (10565689): [] irq_exit_rcu+0xb0/0xd0 [ 2291.005540] ---[ end trace 0000000000000000 ]--- [ 2291.010314] ------------[ cut here ]------------ [ 2291.014932] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2291.026598] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2291.035292] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2291.090796] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2291.099937] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2291.107433] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2291.112945] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2291.118448] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2291.137204] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2291.142449] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2291.149597] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2291.156732] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006786 [ 2291.163864] R10: 00000000002307b8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2291.171014] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2291.178165] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2291.186267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2291.192030] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2291.199182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2291.206332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2291.213482] Call Trace: [ 2291.215951] [ 2291.218057] ? __warn+0x85/0x180 [ 2291.221309] ? __list_add_valid_or_report+0x7d/0xa0 [ 2291.226204] ? report_bug+0xfc/0x1d0 [ 2291.229802] ? prb_read_valid+0x1b/0x20 [ 2291.233660] ? handle_bug+0x44/0x70 [ 2291.237151] ? exc_invalid_op+0x17/0x60 [ 2291.241007] ? asm_exc_invalid_op+0x1a/0x20 [ 2291.245233] ? __list_add_valid_or_report+0x7d/0xa0 [ 2291.250108] ? __list_add_valid_or_report+0x7d/0xa0 [ 2291.254987] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2291.259894] ceph_add_cap+0x20b/0x620 [ceph] [ 2291.264214] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2291.268960] ceph_fill_trace+0x143/0xf30 [ceph] [ 2291.273517] ? __mutex_lock+0x97/0xce0 [ 2291.277268] ? __mutex_lock+0xaec/0xce0 [ 2291.281127] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2291.285687] ? lock_downgrade+0xfd/0x250 [ 2291.289616] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2291.294012] ceph_con_process_message+0x74/0x150 [libceph] [ 2291.299514] ? calc_signature+0xdb/0x100 [libceph] [ 2291.304322] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2291.309671] ? process_scheduled_works+0x22a/0x510 [ 2291.314470] ? process_scheduled_works+0x22a/0x510 [ 2291.319286] ceph_con_workfn+0x344/0x750 [libceph] [ 2291.324091] ? process_scheduled_works+0x22a/0x510 [ 2291.328896] process_scheduled_works+0x2a4/0x510 [ 2291.333538] worker_thread+0x188/0x340 [ 2291.337287] ? rescuer_thread+0x310/0x310 [ 2291.341316] kthread+0xf5/0x130 [ 2291.344481] ? kthread_complete_and_exit+0x20/0x20 [ 2291.349290] ret_from_fork+0x31/0x40 [ 2291.352869] ? kthread_complete_and_exit+0x20/0x20 [ 2291.357679] ret_from_fork_asm+0x11/0x20 [ 2291.361615] [ 2291.363808] irq event stamp: 10566791 [ 2291.367508] hardirqs last enabled at (10566801): [] console_unlock+0x106/0x120 [ 2291.376374] hardirqs last disabled at (10566810): [] console_unlock+0xeb/0x120 [ 2291.385172] softirqs last enabled at (10566748): [] __do_softirq+0x2be/0x371 [ 2291.393882] softirqs last disabled at (10566743): [] irq_exit_rcu+0xb0/0xd0 [ 2291.402417] ---[ end trace 0000000000000000 ]--- [ 2291.407628] ------------[ cut here ]------------ [ 2291.412382] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2291.424053] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2291.432745] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2291.488221] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2291.497364] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2291.504860] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2291.510372] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2291.515876] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2291.534639] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2291.539866] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2291.547014] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2291.554167] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000067c4 [ 2291.561317] R10: 00000000002319c8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2291.568465] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2291.575617] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2291.583703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2291.589448] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2291.596597] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2291.603730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2291.610865] Call Trace: [ 2291.613335] [ 2291.615458] ? __warn+0x85/0x180 [ 2291.618709] ? __list_add_valid_or_report+0x7d/0xa0 [ 2291.623587] ? report_bug+0xfc/0x1d0 [ 2291.627166] ? prb_read_valid+0x1b/0x20 [ 2291.631024] ? handle_bug+0x44/0x70 [ 2291.634534] ? exc_invalid_op+0x17/0x60 [ 2291.638372] ? asm_exc_invalid_op+0x1a/0x20 [ 2291.642581] ? __list_add_valid_or_report+0x7d/0xa0 [ 2291.647457] ? __list_add_valid_or_report+0x7d/0xa0 [ 2291.652351] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2291.657261] ceph_add_cap+0x20b/0x620 [ceph] [ 2291.661572] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2291.666311] ? lock_acquire+0xcb/0x2b0 [ 2291.670080] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2291.674531] ? __mutex_lock+0x97/0xce0 [ 2291.678283] ? __mutex_lock+0xaec/0xce0 [ 2291.682161] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2291.686708] ? lock_downgrade+0xfd/0x250 [ 2291.690638] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2291.695034] ceph_con_process_message+0x74/0x150 [libceph] [ 2291.700531] ? calc_signature+0xdb/0x100 [libceph] [ 2291.705337] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2291.710685] ? process_scheduled_works+0x22a/0x510 [ 2291.715497] ? process_scheduled_works+0x22a/0x510 [ 2291.720291] ceph_con_workfn+0x344/0x750 [libceph] [ 2291.725095] ? process_scheduled_works+0x22a/0x510 [ 2291.729903] process_scheduled_works+0x2a4/0x510 [ 2291.734546] worker_thread+0x188/0x340 [ 2291.738302] ? rescuer_thread+0x310/0x310 [ 2291.742330] kthread+0xf5/0x130 [ 2291.745516] ? kthread_complete_and_exit+0x20/0x20 [ 2291.750305] ret_from_fork+0x31/0x40 [ 2291.753902] ? kthread_complete_and_exit+0x20/0x20 [ 2291.758711] ret_from_fork_asm+0x11/0x20 [ 2291.762645] [ 2291.764837] irq event stamp: 10568757 [ 2291.768512] hardirqs last enabled at (10568767): [] console_unlock+0x106/0x120 [ 2291.777378] hardirqs last disabled at (10568776): [] console_unlock+0xeb/0x120 [ 2291.786177] softirqs last enabled at (10568704): [] __do_softirq+0x2be/0x371 [ 2291.794887] softirqs last disabled at (10568699): [] irq_exit_rcu+0xb0/0xd0 [ 2291.803424] ---[ end trace 0000000000000000 ]--- [ 2291.808085] ------------[ cut here ]------------ [ 2291.812698] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2291.824357] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2291.833047] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2291.888540] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2291.897668] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2291.905146] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2291.910658] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2291.916143] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2291.934908] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2291.940174] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2291.947303] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2291.954452] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006803 [ 2291.961603] R10: 0000000000232ad0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2291.968736] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2291.975867] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2291.983970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2291.989733] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2291.996866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2292.004017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2292.011166] Call Trace: [ 2292.013637] [ 2292.015744] ? __warn+0x85/0x180 [ 2292.018977] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.023873] ? report_bug+0xfc/0x1d0 [ 2292.027468] ? prb_read_valid+0x1b/0x20 [ 2292.031329] ? handle_bug+0x44/0x70 [ 2292.034836] ? exc_invalid_op+0x17/0x60 [ 2292.038694] ? asm_exc_invalid_op+0x1a/0x20 [ 2292.042902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.047778] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.052656] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2292.057546] ceph_add_cap+0x20b/0x620 [ceph] [ 2292.061848] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2292.066578] ceph_fill_trace+0x143/0xf30 [ceph] [ 2292.071117] ? __mutex_lock+0x97/0xce0 [ 2292.074886] ? __mutex_lock+0xaec/0xce0 [ 2292.078744] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2292.083287] ? lock_downgrade+0xfd/0x250 [ 2292.087249] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2292.091630] ceph_con_process_message+0x74/0x150 [libceph] [ 2292.097126] ? calc_signature+0xdb/0x100 [libceph] [ 2292.101955] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2292.107322] ? process_scheduled_works+0x22a/0x510 [ 2292.112129] ? process_scheduled_works+0x22a/0x510 [ 2292.116920] ceph_con_workfn+0x344/0x750 [libceph] [ 2292.121741] ? process_scheduled_works+0x22a/0x510 [ 2292.126532] process_scheduled_works+0x2a4/0x510 [ 2292.131172] worker_thread+0x188/0x340 [ 2292.134921] ? rescuer_thread+0x310/0x310 [ 2292.138951] kthread+0xf5/0x130 [ 2292.142115] ? kthread_complete_and_exit+0x20/0x20 [ 2292.146927] ret_from_fork+0x31/0x40 [ 2292.150521] ? kthread_complete_and_exit+0x20/0x20 [ 2292.155313] ret_from_fork_asm+0x11/0x20 [ 2292.159283] [ 2292.161476] irq event stamp: 10569821 [ 2292.165160] hardirqs last enabled at (10569831): [] console_unlock+0x106/0x120 [ 2292.174042] hardirqs last disabled at (10569840): [] console_unlock+0xeb/0x120 [ 2292.182839] softirqs last enabled at (10569778): [] __do_softirq+0x2be/0x371 [ 2292.191550] softirqs last disabled at (10569773): [] irq_exit_rcu+0xb0/0xd0 [ 2292.200070] ---[ end trace 0000000000000000 ]--- [ 2292.204729] ------------[ cut here ]------------ [ 2292.209344] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2292.221024] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2292.229719] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2292.285197] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2292.294339] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2292.301835] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2292.307347] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2292.312850] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2292.331614] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2292.336840] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2292.343992] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2292.351143] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006841 [ 2292.358290] R10: 0000000000233d28 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2292.365432] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2292.372583] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2292.380668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2292.386413] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2292.393564] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2292.400697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2292.407830] Call Trace: [ 2292.410300] [ 2292.412416] ? __warn+0x85/0x180 [ 2292.415668] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.420546] ? report_bug+0xfc/0x1d0 [ 2292.424124] ? prb_read_valid+0x1b/0x20 [ 2292.427983] ? handle_bug+0x44/0x70 [ 2292.431491] ? exc_invalid_op+0x17/0x60 [ 2292.435347] ? asm_exc_invalid_op+0x1a/0x20 [ 2292.439556] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.444432] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.449327] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2292.454235] ceph_add_cap+0x20b/0x620 [ceph] [ 2292.458544] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2292.463277] ? lock_acquire+0xcb/0x2b0 [ 2292.467048] ceph_fill_trace+0x143/0xf30 [ceph] [ 2292.471586] ? __mutex_lock+0x97/0xce0 [ 2292.475336] ? __mutex_lock+0xaec/0xce0 [ 2292.479195] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2292.483753] ? lock_downgrade+0xfd/0x250 [ 2292.487684] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2292.492080] ceph_con_process_message+0x74/0x150 [libceph] [ 2292.497576] ? calc_signature+0xdb/0x100 [libceph] [ 2292.502405] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2292.507736] ? process_scheduled_works+0x22a/0x510 [ 2292.512540] ? process_scheduled_works+0x22a/0x510 [ 2292.517336] ceph_con_workfn+0x344/0x750 [libceph] [ 2292.522156] ? process_scheduled_works+0x22a/0x510 [ 2292.526965] process_scheduled_works+0x2a4/0x510 [ 2292.531606] worker_thread+0x188/0x340 [ 2292.535354] ? rescuer_thread+0x310/0x310 [ 2292.539383] kthread+0xf5/0x130 [ 2292.542551] ? kthread_complete_and_exit+0x20/0x20 [ 2292.547342] ret_from_fork+0x31/0x40 [ 2292.550936] ? kthread_complete_and_exit+0x20/0x20 [ 2292.555747] ret_from_fork_asm+0x11/0x20 [ 2292.559682] [ 2292.561875] irq event stamp: 10570935 [ 2292.565558] hardirqs last enabled at (10570945): [] console_unlock+0x106/0x120 [ 2292.574423] hardirqs last disabled at (10570954): [] console_unlock+0xeb/0x120 [ 2292.583224] softirqs last enabled at (10570976): [] __do_softirq+0x2be/0x371 [ 2292.591933] softirqs last disabled at (10570987): [] irq_exit_rcu+0xb0/0xd0 [ 2292.600467] ---[ end trace 0000000000000000 ]--- [ 2292.605139] ------------[ cut here ]------------ [ 2292.609762] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2292.621402] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2292.630093] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2292.685570] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2292.694694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2292.702174] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2292.707668] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2292.713153] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2292.731900] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2292.737126] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2292.744261] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2292.751393] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006880 [ 2292.758524] R10: 0000000000234f48 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2292.765657] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2292.772791] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2292.780877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2292.786623] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2292.793757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2292.800888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2292.808021] Call Trace: [ 2292.810473] [ 2292.812581] ? __warn+0x85/0x180 [ 2292.815814] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.820692] ? report_bug+0xfc/0x1d0 [ 2292.824272] ? prb_read_valid+0x1b/0x20 [ 2292.828112] ? handle_bug+0x44/0x70 [ 2292.831604] ? exc_invalid_op+0x17/0x60 [ 2292.835444] ? asm_exc_invalid_op+0x1a/0x20 [ 2292.839634] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.844515] ? __list_add_valid_or_report+0x7d/0xa0 [ 2292.849395] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2292.854287] ceph_add_cap+0x20b/0x620 [ceph] [ 2292.858572] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2292.863296] ? lock_acquire+0xcb/0x2b0 [ 2292.867047] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2292.871499] ? __mutex_lock+0x97/0xce0 [ 2292.875250] ? __mutex_lock+0xaec/0xce0 [ 2292.879091] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2292.883630] ? lock_downgrade+0xfd/0x250 [ 2292.887561] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2292.891943] ceph_con_process_message+0x74/0x150 [libceph] [ 2292.897437] ? calc_signature+0xdb/0x100 [libceph] [ 2292.902242] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2292.907571] ? process_scheduled_works+0x22a/0x510 [ 2292.912366] ? process_scheduled_works+0x22a/0x510 [ 2292.917154] ceph_con_workfn+0x344/0x750 [libceph] [ 2292.921957] ? process_scheduled_works+0x22a/0x510 [ 2292.926749] process_scheduled_works+0x2a4/0x510 [ 2292.931373] worker_thread+0x188/0x340 [ 2292.935131] ? rescuer_thread+0x310/0x310 [ 2292.939142] kthread+0xf5/0x130 [ 2292.942289] ? kthread_complete_and_exit+0x20/0x20 [ 2292.947083] ret_from_fork+0x31/0x40 [ 2292.950659] ? kthread_complete_and_exit+0x20/0x20 [ 2292.955452] ret_from_fork_asm+0x11/0x20 [ 2292.959385] [ 2292.961577] irq event stamp: 10572081 [ 2292.965237] hardirqs last enabled at (10572091): [] console_unlock+0x106/0x120 [ 2292.974103] hardirqs last disabled at (10572100): [] console_unlock+0xeb/0x120 [ 2292.982884] softirqs last enabled at (10572078): [] __do_softirq+0x2be/0x371 [ 2292.991575] softirqs last disabled at (10572067): [] irq_exit_rcu+0xb0/0xd0 [ 2293.000095] ---[ end trace 0000000000000000 ]--- [ 2293.004718] ------------[ cut here ]------------ [ 2293.009340] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2293.020982] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2293.029934] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2293.085394] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2293.094345] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2293.101818] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2293.107302] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2293.126050] RSP: 0018:ffffc90001fefb78 EFLAGS: 00010286 [ 2293.131275] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2293.138407] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2293.145531] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000068bf [ 2293.152659] R10: 0000000000236168 R11: ffffc90001fefa08 R12: ffff888158a68750 [ 2293.159788] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2293.166912] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2293.174989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2293.180727] CR2: 00007f9bd2118028 CR3: 0000000108c72005 CR4: 00000000003706f0 [ 2293.187852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2293.194975] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2293.202100] Call Trace: [ 2293.204544] [ 2293.206643] ? __warn+0x85/0x180 [ 2293.209875] ? __list_add_valid_or_report+0x7d/0xa0 [ 2293.214755] ? report_bug+0xfc/0x1d0 [ 2293.218335] ? prb_read_valid+0x1b/0x20 [ 2293.222175] ? handle_bug+0x44/0x70 [ 2293.225666] ? exc_invalid_op+0x17/0x60 [ 2293.229504] ? asm_exc_invalid_op+0x1a/0x20 [ 2293.233696] ? __list_add_valid_or_report+0x7d/0xa0 [ 2293.238571] ? __list_add_valid_or_report+0x7d/0xa0 [ 2293.243450] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2293.248342] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2293.253584] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2293.258386] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2293.263099] ? __d_lookup+0xd6/0x200 [ 2293.266680] lookup_fast+0x6d/0xf0 [ 2293.270085] walk_component+0x26/0x160 [ 2293.273838] link_path_walk+0x24d/0x380 [ 2293.277675] ? path_openat+0x80/0x9d0 [ 2293.281346] path_openat+0x8e/0x9d0 [ 2293.284839] do_filp_open+0xb8/0x120 [ 2293.288417] ? lock_release+0xc5/0x260 [ 2293.292167] ? _raw_spin_unlock+0x23/0x40 [ 2293.296179] ? alloc_fd+0x13f/0x200 [ 2293.299680] do_sys_openat2+0x249/0x290 [ 2293.303537] do_sys_open+0x4f/0x80 [ 2293.306946] do_syscall_64+0x38/0x80 [ 2293.310525] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2293.315575] RIP: 0033:0x7f48f0eed2f2 [ 2293.319155] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2293.337901] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2293.345468] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2293.352600] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2293.359732] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2293.366857] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2293.373980] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2293.381119] [ 2293.383306] irq event stamp: 12887457 [ 2293.386965] hardirqs last enabled at (12887467): [] console_unlock+0x106/0x120 [ 2293.395829] hardirqs last disabled at (12887476): [] console_unlock+0xeb/0x120 [ 2293.404609] softirqs last enabled at (12887394): [] __do_softirq+0x2be/0x371 [ 2293.413304] softirqs last disabled at (12887495): [] irq_exit_rcu+0xb0/0xd0 [ 2293.421821] ---[ end trace 0000000000000000 ]--- [ 2293.426433] ------------[ cut here ]------------ [ 2293.431053] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2293.442700] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2293.451397] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2293.506875] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2293.516000] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2293.523476] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2293.528971] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2293.534456] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2293.553203] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2293.558429] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2293.565562] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2293.572696] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000068fe [ 2293.579831] R10: 0000000000237390 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2293.586961] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2293.594095] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2293.602179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2293.607924] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2293.615050] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2293.622184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2293.629315] Call Trace: [ 2293.631767] [ 2293.633874] ? __warn+0x85/0x180 [ 2293.637108] ? __list_add_valid_or_report+0x7d/0xa0 [ 2293.641986] ? report_bug+0xfc/0x1d0 [ 2293.645568] ? prb_read_valid+0x1b/0x20 [ 2293.649408] ? handle_bug+0x44/0x70 [ 2293.652897] ? exc_invalid_op+0x17/0x60 [ 2293.656738] ? asm_exc_invalid_op+0x1a/0x20 [ 2293.660928] ? __list_add_valid_or_report+0x7d/0xa0 [ 2293.665804] ? __list_add_valid_or_report+0x7d/0xa0 [ 2293.670681] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2293.675573] ceph_add_cap+0x20b/0x620 [ceph] [ 2293.679858] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2293.684582] ? lock_acquire+0xcb/0x2b0 [ 2293.688342] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2293.692793] ? __mutex_lock+0x97/0xce0 [ 2293.696544] ? __mutex_lock+0xaec/0xce0 [ 2293.700384] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2293.704926] ? lock_downgrade+0xfd/0x250 [ 2293.708858] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2293.713237] ceph_con_process_message+0x74/0x150 [libceph] [ 2293.718730] ? calc_signature+0xdb/0x100 [libceph] [ 2293.723540] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2293.728866] ? process_scheduled_works+0x22a/0x510 [ 2293.733657] ? process_scheduled_works+0x22a/0x510 [ 2293.738447] ceph_con_workfn+0x344/0x750 [libceph] [ 2293.743251] ? process_scheduled_works+0x22a/0x510 [ 2293.748042] process_scheduled_works+0x2a4/0x510 [ 2293.752667] worker_thread+0x188/0x340 [ 2293.756415] ? rescuer_thread+0x310/0x310 [ 2293.760428] kthread+0xf5/0x130 [ 2293.763572] ? kthread_complete_and_exit+0x20/0x20 [ 2293.768366] ret_from_fork+0x31/0x40 [ 2293.771946] ? kthread_complete_and_exit+0x20/0x20 [ 2293.776739] ret_from_fork_asm+0x11/0x20 [ 2293.780672] [ 2293.782865] irq event stamp: 10574083 [ 2293.786534] hardirqs last enabled at (10574093): [] console_unlock+0x106/0x120 [ 2293.795399] hardirqs last disabled at (10574102): [] console_unlock+0xeb/0x120 [ 2293.804177] softirqs last enabled at (10574050): [] __do_softirq+0x2be/0x371 [ 2293.812871] softirqs last disabled at (10574045): [] irq_exit_rcu+0xb0/0xd0 [ 2293.821390] ---[ end trace 0000000000000000 ]--- [ 2293.826018] ------------[ cut here ]------------ [ 2293.830636] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2293.842299] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2293.850996] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2293.906490] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2293.915634] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2293.923112] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2293.928625] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2293.934110] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2293.952874] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2293.958099] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2293.965242] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2293.972393] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000693d [ 2293.979560] R10: 0000000000238640 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2293.986692] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2293.993841] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2294.001945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2294.007707] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2294.014849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2294.021992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2294.029141] Call Trace: [ 2294.031602] [ 2294.033702] ? __warn+0x85/0x180 [ 2294.036935] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.041847] ? report_bug+0xfc/0x1d0 [ 2294.045444] ? prb_read_valid+0x1b/0x20 [ 2294.049303] ? handle_bug+0x44/0x70 [ 2294.052811] ? exc_invalid_op+0x17/0x60 [ 2294.056651] ? asm_exc_invalid_op+0x1a/0x20 [ 2294.060866] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.065745] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.070622] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2294.075512] ceph_add_cap+0x20b/0x620 [ceph] [ 2294.079838] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2294.084576] ceph_fill_trace+0x143/0xf30 [ceph] [ 2294.089116] ? __mutex_lock+0x97/0xce0 [ 2294.092886] ? __mutex_lock+0xaec/0xce0 [ 2294.096745] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2294.101287] ? lock_downgrade+0xfd/0x250 [ 2294.105250] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2294.109632] ceph_con_process_message+0x74/0x150 [libceph] [ 2294.115125] ? calc_signature+0xdb/0x100 [libceph] [ 2294.119966] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2294.125313] ? process_scheduled_works+0x22a/0x510 [ 2294.130121] ? process_scheduled_works+0x22a/0x510 [ 2294.134912] ceph_con_workfn+0x344/0x750 [libceph] [ 2294.139733] ? process_scheduled_works+0x22a/0x510 [ 2294.144524] process_scheduled_works+0x2a4/0x510 [ 2294.149164] worker_thread+0x188/0x340 [ 2294.152915] ? rescuer_thread+0x310/0x310 [ 2294.156944] kthread+0xf5/0x130 [ 2294.160107] ? kthread_complete_and_exit+0x20/0x20 [ 2294.164918] ret_from_fork+0x31/0x40 [ 2294.168513] ? kthread_complete_and_exit+0x20/0x20 [ 2294.173323] ret_from_fork_asm+0x11/0x20 [ 2294.177292] [ 2294.179486] irq event stamp: 10575145 [ 2294.183168] hardirqs last enabled at (10575155): [] console_unlock+0x106/0x120 [ 2294.192053] hardirqs last disabled at (10575164): [] console_unlock+0xeb/0x120 [ 2294.200848] softirqs last enabled at (10575102): [] __do_softirq+0x2be/0x371 [ 2294.209568] softirqs last disabled at (10575097): [] irq_exit_rcu+0xb0/0xd0 [ 2294.218087] ---[ end trace 0000000000000000 ]--- [ 2294.222741] ------------[ cut here ]------------ [ 2294.227363] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2294.239041] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2294.247739] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2294.303215] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2294.312357] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2294.319858] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2294.325374] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2294.330876] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2294.349642] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2294.354867] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2294.362017] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2294.369168] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000697b [ 2294.376319] R10: 0000000000239758 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2294.383467] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2294.390619] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2294.398704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2294.404449] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2294.411592] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2294.418724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2294.425857] Call Trace: [ 2294.428328] [ 2294.430452] ? __warn+0x85/0x180 [ 2294.433703] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.438582] ? report_bug+0xfc/0x1d0 [ 2294.442160] ? prb_read_valid+0x1b/0x20 [ 2294.446017] ? handle_bug+0x44/0x70 [ 2294.449527] ? exc_invalid_op+0x17/0x60 [ 2294.453384] ? asm_exc_invalid_op+0x1a/0x20 [ 2294.457592] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.462466] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.467361] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2294.472270] ceph_add_cap+0x20b/0x620 [ceph] [ 2294.476582] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2294.481322] ? lock_acquire+0xcb/0x2b0 [ 2294.485090] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2294.489565] ? __mutex_lock+0x97/0xce0 [ 2294.493320] ? __mutex_lock+0xaec/0xce0 [ 2294.497196] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2294.501745] ? lock_downgrade+0xfd/0x250 [ 2294.505675] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2294.510071] ceph_con_process_message+0x74/0x150 [libceph] [ 2294.515573] ? calc_signature+0xdb/0x100 [libceph] [ 2294.520380] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2294.525721] ? process_scheduled_works+0x22a/0x510 [ 2294.530512] ? process_scheduled_works+0x22a/0x510 [ 2294.535321] ceph_con_workfn+0x344/0x750 [libceph] [ 2294.540139] ? process_scheduled_works+0x22a/0x510 [ 2294.544948] process_scheduled_works+0x2a4/0x510 [ 2294.549590] worker_thread+0x188/0x340 [ 2294.553337] ? rescuer_thread+0x310/0x310 [ 2294.557368] kthread+0xf5/0x130 [ 2294.560533] ? kthread_complete_and_exit+0x20/0x20 [ 2294.565349] ret_from_fork+0x31/0x40 [ 2294.568928] ? kthread_complete_and_exit+0x20/0x20 [ 2294.573740] ret_from_fork_asm+0x11/0x20 [ 2294.577673] [ 2294.579867] irq event stamp: 10576259 [ 2294.583567] hardirqs last enabled at (10576269): [] console_unlock+0x106/0x120 [ 2294.592434] hardirqs last disabled at (10576278): [] console_unlock+0xeb/0x120 [ 2294.601230] softirqs last enabled at (10576204): [] __do_softirq+0x2be/0x371 [ 2294.609940] softirqs last disabled at (10576199): [] irq_exit_rcu+0xb0/0xd0 [ 2294.618478] ---[ end trace 0000000000000000 ]--- [ 2294.623138] ------------[ cut here ]------------ [ 2294.627752] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2294.639416] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2294.648111] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2294.703603] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2294.712729] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2294.720208] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2294.725719] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2294.731205] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2294.749970] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2294.755214] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2294.762363] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2294.769517] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000069ba [ 2294.776665] R10: 000000000023a968 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2294.783798] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2294.790929] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2294.799034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2294.804796] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2294.811929] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2294.819079] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2294.826230] Call Trace: [ 2294.828699] [ 2294.830807] ? __warn+0x85/0x180 [ 2294.834041] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.838937] ? report_bug+0xfc/0x1d0 [ 2294.842532] ? prb_read_valid+0x1b/0x20 [ 2294.846390] ? handle_bug+0x44/0x70 [ 2294.849898] ? exc_invalid_op+0x17/0x60 [ 2294.853756] ? asm_exc_invalid_op+0x1a/0x20 [ 2294.857963] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.862840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2294.867736] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2294.872626] ceph_add_cap+0x20b/0x620 [ceph] [ 2294.876928] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2294.881650] ceph_fill_trace+0x143/0xf30 [ceph] [ 2294.886188] ? __mutex_lock+0x97/0xce0 [ 2294.889956] ? __mutex_lock+0xaec/0xce0 [ 2294.893816] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2294.898357] ? lock_downgrade+0xfd/0x250 [ 2294.902320] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2294.906701] ceph_con_process_message+0x74/0x150 [libceph] [ 2294.912197] ? calc_signature+0xdb/0x100 [libceph] [ 2294.917044] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2294.922390] ? process_scheduled_works+0x22a/0x510 [ 2294.927185] ? process_scheduled_works+0x22a/0x510 [ 2294.931991] ceph_con_workfn+0x344/0x750 [libceph] [ 2294.936812] ? process_scheduled_works+0x22a/0x510 [ 2294.941604] process_scheduled_works+0x2a4/0x510 [ 2294.946249] worker_thread+0x188/0x340 [ 2294.950003] ? rescuer_thread+0x310/0x310 [ 2294.954032] kthread+0xf5/0x130 [ 2294.957194] ? kthread_complete_and_exit+0x20/0x20 [ 2294.962008] ret_from_fork+0x31/0x40 [ 2294.965601] ? kthread_complete_and_exit+0x20/0x20 [ 2294.970395] ret_from_fork_asm+0x11/0x20 [ 2294.974362] [ 2294.976574] irq event stamp: 10577345 [ 2294.980240] hardirqs last enabled at (10577355): [] console_unlock+0x106/0x120 [ 2294.989122] hardirqs last disabled at (10577364): [] console_unlock+0xeb/0x120 [ 2294.997921] softirqs last enabled at (10577302): [] __do_softirq+0x2be/0x371 [ 2295.006631] softirqs last disabled at (10577297): [] irq_exit_rcu+0xb0/0xd0 [ 2295.015149] ---[ end trace 0000000000000000 ]--- [ 2295.019938] ------------[ cut here ]------------ [ 2295.024560] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2295.036207] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2295.044904] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2295.100381] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2295.109505] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2295.116985] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2295.122479] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2295.127965] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2295.146711] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2295.151937] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2295.159071] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2295.166205] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000069f8 [ 2295.173336] R10: 000000000023bb28 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2295.180469] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2295.187602] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2295.195689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2295.201434] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2295.208568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2295.215699] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2295.222832] Call Trace: [ 2295.225286] [ 2295.227391] ? __warn+0x85/0x180 [ 2295.230627] ? __list_add_valid_or_report+0x7d/0xa0 [ 2295.235505] ? report_bug+0xfc/0x1d0 [ 2295.239082] ? prb_read_valid+0x1b/0x20 [ 2295.242925] ? handle_bug+0x44/0x70 [ 2295.246416] ? exc_invalid_op+0x17/0x60 [ 2295.250256] ? asm_exc_invalid_op+0x1a/0x20 [ 2295.254445] ? __list_add_valid_or_report+0x7d/0xa0 [ 2295.259321] ? __list_add_valid_or_report+0x7d/0xa0 [ 2295.264201] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2295.269090] ceph_add_cap+0x20b/0x620 [ceph] [ 2295.273375] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2295.278098] ? lock_acquire+0xcb/0x2b0 [ 2295.281849] ceph_fill_trace+0x143/0xf30 [ceph] [ 2295.286387] ? __mutex_lock+0x97/0xce0 [ 2295.290138] ? __mutex_lock+0xaec/0xce0 [ 2295.293982] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2295.298531] ? lock_downgrade+0xfd/0x250 [ 2295.302461] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2295.306840] ceph_con_process_message+0x74/0x150 [libceph] [ 2295.312336] ? calc_signature+0xdb/0x100 [libceph] [ 2295.317148] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2295.322480] ? process_scheduled_works+0x22a/0x510 [ 2295.327271] ? process_scheduled_works+0x22a/0x510 [ 2295.332059] ceph_con_workfn+0x344/0x750 [libceph] [ 2295.336866] ? process_scheduled_works+0x22a/0x510 [ 2295.341654] process_scheduled_works+0x2a4/0x510 [ 2295.346279] worker_thread+0x188/0x340 [ 2295.350027] ? rescuer_thread+0x310/0x310 [ 2295.354039] kthread+0xf5/0x130 [ 2295.357192] ? kthread_complete_and_exit+0x20/0x20 [ 2295.361987] ret_from_fork+0x31/0x40 [ 2295.365567] ? kthread_complete_and_exit+0x20/0x20 [ 2295.370361] ret_from_fork_asm+0x11/0x20 [ 2295.374292] [ 2295.376477] irq event stamp: 10578453 [ 2295.380145] hardirqs last enabled at (10578463): [] console_unlock+0x106/0x120 [ 2295.389011] hardirqs last disabled at (10578472): [] console_unlock+0xeb/0x120 [ 2295.397791] softirqs last enabled at (10578400): [] __do_softirq+0x2be/0x371 [ 2295.406482] softirqs last disabled at (10578395): [] irq_exit_rcu+0xb0/0xd0 [ 2295.415003] ---[ end trace 0000000000000000 ]--- [ 2295.419641] ------------[ cut here ]------------ [ 2295.424260] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2295.435904] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2295.444600] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2295.500077] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2295.509201] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2295.516683] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2295.522184] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2295.527670] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2295.546418] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2295.551643] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2295.558776] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2295.565911] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006a37 [ 2295.573042] R10: 000000000023cc90 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2295.580176] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2295.587306] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2295.595394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2295.601139] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2295.608273] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2295.615407] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2295.622538] Call Trace: [ 2295.624990] [ 2295.627097] ? __warn+0x85/0x180 [ 2295.630332] ? __list_add_valid_or_report+0x7d/0xa0 [ 2295.635211] ? report_bug+0xfc/0x1d0 [ 2295.638788] ? prb_read_valid+0x1b/0x20 [ 2295.642629] ? handle_bug+0x44/0x70 [ 2295.646121] ? exc_invalid_op+0x17/0x60 [ 2295.649961] ? asm_exc_invalid_op+0x1a/0x20 [ 2295.654150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2295.659026] ? __list_add_valid_or_report+0x7d/0xa0 [ 2295.663905] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2295.668795] ceph_add_cap+0x20b/0x620 [ceph] [ 2295.673082] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2295.677803] ? lock_acquire+0xcb/0x2b0 [ 2295.681556] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2295.686014] ? __mutex_lock+0x97/0xce0 [ 2295.689767] ? __mutex_lock+0xaec/0xce0 [ 2295.693608] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2295.698147] ? lock_downgrade+0xfd/0x250 [ 2295.702080] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2295.706459] ceph_con_process_message+0x74/0x150 [libceph] [ 2295.711952] ? calc_signature+0xdb/0x100 [libceph] [ 2295.716756] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2295.722080] ? process_scheduled_works+0x22a/0x510 [ 2295.726870] ? process_scheduled_works+0x22a/0x510 [ 2295.731660] ceph_con_workfn+0x344/0x750 [libceph] [ 2295.736466] ? process_scheduled_works+0x22a/0x510 [ 2295.741256] process_scheduled_works+0x2a4/0x510 [ 2295.745880] worker_thread+0x188/0x340 [ 2295.749628] ? rescuer_thread+0x310/0x310 [ 2295.753641] kthread+0xf5/0x130 [ 2295.756788] ? kthread_complete_and_exit+0x20/0x20 [ 2295.761579] ret_from_fork+0x31/0x40 [ 2295.765159] ? kthread_complete_and_exit+0x20/0x20 [ 2295.769952] ret_from_fork_asm+0x11/0x20 [ 2295.773885] [ 2295.776071] irq event stamp: 10579567 [ 2295.779736] hardirqs last enabled at (10579577): [] console_unlock+0x106/0x120 [ 2295.788604] hardirqs last disabled at (10579586): [] console_unlock+0xeb/0x120 [ 2295.797383] softirqs last enabled at (10579514): [] __do_softirq+0x2be/0x371 [ 2295.806075] softirqs last disabled at (10579509): [] irq_exit_rcu+0xb0/0xd0 [ 2295.814594] ---[ end trace 0000000000000000 ]--- [ 2295.819214] ------------[ cut here ]------------ [ 2295.823832] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2295.835483] WARNING: CPU: 4 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2295.844434] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2295.899907] CPU: 4 PID: 78678 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2295.908854] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2295.916332] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2295.921819] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2295.940571] RSP: 0018:ffffc90001f7fb78 EFLAGS: 00010286 [ 2295.945797] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2295.952925] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2295.960062] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006a76 [ 2295.967194] R10: 000000000023df68 R11: ffffc90001f7fa08 R12: ffff8881c5350750 [ 2295.974331] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2295.981457] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2295.989542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2295.995288] CR2: 00007fb38d763000 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2296.002423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2296.009553] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2296.016677] Call Trace: [ 2296.019123] [ 2296.021229] ? __warn+0x85/0x180 [ 2296.024462] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.029342] ? report_bug+0xfc/0x1d0 [ 2296.032920] ? prb_read_valid+0x1b/0x20 [ 2296.036761] ? handle_bug+0x44/0x70 [ 2296.040253] ? exc_invalid_op+0x17/0x60 [ 2296.044092] ? asm_exc_invalid_op+0x1a/0x20 [ 2296.048283] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.053157] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.058036] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2296.062932] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2296.068172] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2296.072971] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2296.077687] ? __d_lookup+0xd6/0x200 [ 2296.081264] lookup_fast+0x6d/0xf0 [ 2296.084671] walk_component+0x26/0x160 [ 2296.088424] link_path_walk+0x24d/0x380 [ 2296.092260] ? path_openat+0x80/0x9d0 [ 2296.095924] path_openat+0x8e/0x9d0 [ 2296.099414] do_filp_open+0xb8/0x120 [ 2296.102994] ? lock_release+0xc5/0x260 [ 2296.106747] ? _raw_spin_unlock+0x23/0x40 [ 2296.110756] ? alloc_fd+0x13f/0x200 [ 2296.114252] do_sys_openat2+0x249/0x290 [ 2296.118091] do_sys_open+0x4f/0x80 [ 2296.121496] do_syscall_64+0x38/0x80 [ 2296.125075] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2296.130132] RIP: 0033:0x7f48f0eed2f2 [ 2296.133708] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2296.152457] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2296.160019] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2296.167151] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2296.174283] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2296.181409] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2296.188540] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2296.195679] [ 2296.197866] irq event stamp: 12948017 [ 2296.201524] hardirqs last enabled at (12948027): [] console_unlock+0x106/0x120 [ 2296.210390] hardirqs last disabled at (12948046): [] console_unlock+0xeb/0x120 [ 2296.219169] softirqs last enabled at (12948044): [] __do_softirq+0x2be/0x371 [ 2296.227864] softirqs last disabled at (12948065): [] irq_exit_rcu+0xb0/0xd0 [ 2296.236386] ---[ end trace 0000000000000000 ]--- [ 2296.241006] ------------[ cut here ]------------ [ 2296.245621] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2296.257284] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2296.265983] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2296.321480] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2296.330613] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2296.338088] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2296.343610] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2296.349095] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2296.367859] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2296.373103] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2296.380251] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2296.387393] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006ab5 [ 2296.394546] R10: 000000000023f158 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2296.401694] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2296.408828] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2296.416912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2296.422675] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2296.429810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2296.436942] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2296.444093] Call Trace: [ 2296.446564] [ 2296.448687] ? __warn+0x85/0x180 [ 2296.451918] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.456817] ? report_bug+0xfc/0x1d0 [ 2296.460393] ? prb_read_valid+0x1b/0x20 [ 2296.464252] ? handle_bug+0x44/0x70 [ 2296.467761] ? exc_invalid_op+0x17/0x60 [ 2296.471608] ? asm_exc_invalid_op+0x1a/0x20 [ 2296.475800] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.480675] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.485556] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2296.490479] ceph_add_cap+0x20b/0x620 [ceph] [ 2296.494765] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2296.499511] ceph_fill_trace+0x143/0xf30 [ceph] [ 2296.504049] ? __mutex_lock+0x97/0xce0 [ 2296.507819] ? __mutex_lock+0xaec/0xce0 [ 2296.511660] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2296.516201] ? lock_downgrade+0xfd/0x250 [ 2296.520168] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2296.524563] ceph_con_process_message+0x74/0x150 [libceph] [ 2296.530066] ? calc_signature+0xdb/0x100 [libceph] [ 2296.534870] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2296.540211] ? process_scheduled_works+0x22a/0x510 [ 2296.545002] ? process_scheduled_works+0x22a/0x510 [ 2296.549810] ceph_con_workfn+0x344/0x750 [libceph] [ 2296.554613] ? process_scheduled_works+0x22a/0x510 [ 2296.559405] process_scheduled_works+0x2a4/0x510 [ 2296.564063] worker_thread+0x188/0x340 [ 2296.567811] ? rescuer_thread+0x310/0x310 [ 2296.571825] kthread+0xf5/0x130 [ 2296.574970] ? kthread_complete_and_exit+0x20/0x20 [ 2296.579782] ret_from_fork+0x31/0x40 [ 2296.583361] ? kthread_complete_and_exit+0x20/0x20 [ 2296.588168] ret_from_fork_asm+0x11/0x20 [ 2296.592139] [ 2296.594332] irq event stamp: 10581513 [ 2296.597998] hardirqs last enabled at (10581523): [] console_unlock+0x106/0x120 [ 2296.606888] hardirqs last disabled at (10581532): [] console_unlock+0xeb/0x120 [ 2296.615688] softirqs last enabled at (10581480): [] __do_softirq+0x2be/0x371 [ 2296.624380] softirqs last disabled at (10581475): [] irq_exit_rcu+0xb0/0xd0 [ 2296.632916] ---[ end trace 0000000000000000 ]--- [ 2296.637609] ------------[ cut here ]------------ [ 2296.642223] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2296.653876] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2296.662567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2296.718055] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2296.727178] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2296.734658] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2296.740160] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2296.745645] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2296.764392] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2296.769618] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2296.776751] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2296.783887] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006af3 [ 2296.791018] R10: 0000000000240638 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2296.798151] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2296.805282] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2296.813369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2296.819114] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2296.826248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2296.833381] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2296.840512] Call Trace: [ 2296.842965] [ 2296.845072] ? __warn+0x85/0x180 [ 2296.848305] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.853184] ? report_bug+0xfc/0x1d0 [ 2296.856764] ? prb_read_valid+0x1b/0x20 [ 2296.860605] ? handle_bug+0x44/0x70 [ 2296.864097] ? exc_invalid_op+0x17/0x60 [ 2296.867934] ? asm_exc_invalid_op+0x1a/0x20 [ 2296.872127] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.877011] ? __list_add_valid_or_report+0x7d/0xa0 [ 2296.881889] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2296.886778] ceph_add_cap+0x20b/0x620 [ceph] [ 2296.891063] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2296.895779] ? lock_acquire+0xcb/0x2b0 [ 2296.899541] ceph_fill_trace+0x143/0xf30 [ceph] [ 2296.904086] ? __mutex_lock+0x97/0xce0 [ 2296.907837] ? __mutex_lock+0xaec/0xce0 [ 2296.911678] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2296.916217] ? lock_downgrade+0xfd/0x250 [ 2296.920148] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2296.924530] ceph_con_process_message+0x74/0x150 [libceph] [ 2296.930029] ? calc_signature+0xdb/0x100 [libceph] [ 2296.934837] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2296.940169] ? process_scheduled_works+0x22a/0x510 [ 2296.944968] ? process_scheduled_works+0x22a/0x510 [ 2296.949758] ceph_con_workfn+0x344/0x750 [libceph] [ 2296.954562] ? process_scheduled_works+0x22a/0x510 [ 2296.959353] process_scheduled_works+0x2a4/0x510 [ 2296.963980] worker_thread+0x188/0x340 [ 2296.967734] ? rescuer_thread+0x310/0x310 [ 2296.971745] kthread+0xf5/0x130 [ 2296.974895] ? kthread_complete_and_exit+0x20/0x20 [ 2296.979687] ret_from_fork+0x31/0x40 [ 2296.983265] ? kthread_complete_and_exit+0x20/0x20 [ 2296.988059] ret_from_fork_asm+0x11/0x20 [ 2296.991992] [ 2296.994186] irq event stamp: 10582659 [ 2296.997851] hardirqs last enabled at (10582669): [] console_unlock+0x106/0x120 [ 2297.006717] hardirqs last disabled at (10582678): [] console_unlock+0xeb/0x120 [ 2297.015497] softirqs last enabled at (10582606): [] __do_softirq+0x2be/0x371 [ 2297.024190] softirqs last disabled at (10582601): [] irq_exit_rcu+0xb0/0xd0 [ 2297.032708] ---[ end trace 0000000000000000 ]--- [ 2297.037505] ------------[ cut here ]------------ [ 2297.042124] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2297.053770] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2297.062720] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2297.118191] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2297.127140] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2297.134611] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2297.140099] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2297.158860] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2297.164091] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2297.171220] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2297.178344] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006b32 [ 2297.185469] R10: 00000000002419b0 R11: ffffc90001957950 R12: ffff8881c5350750 [ 2297.192601] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2297.199734] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2297.207821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2297.213566] CR2: 00007fd645a418a0 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2297.220691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2297.227823] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2297.234947] Call Trace: [ 2297.237393] [ 2297.239499] ? __warn+0x85/0x180 [ 2297.242730] ? __list_add_valid_or_report+0x7d/0xa0 [ 2297.247611] ? report_bug+0xfc/0x1d0 [ 2297.251190] ? prb_read_valid+0x1b/0x20 [ 2297.255029] ? handle_bug+0x44/0x70 [ 2297.258522] ? exc_invalid_op+0x17/0x60 [ 2297.262361] ? asm_exc_invalid_op+0x1a/0x20 [ 2297.266551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2297.271429] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2297.276327] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2297.281570] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2297.286370] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2297.291085] ? __d_lookup+0xd6/0x200 [ 2297.294663] lookup_fast+0x6d/0xf0 [ 2297.298070] walk_component+0x26/0x160 [ 2297.301825] link_path_walk+0x24d/0x380 [ 2297.305660] ? path_lookupat.isra.40+0x22/0x120 [ 2297.310195] path_lookupat.isra.40+0x6b/0x120 [ 2297.314558] filename_lookup+0xd4/0x190 [ 2297.318397] ? lock_acquire+0xcb/0x2b0 [ 2297.322144] ? __might_fault+0x45/0x80 [ 2297.325899] ? find_held_lock+0x2d/0x90 [ 2297.329737] ? __might_fault+0x45/0x80 [ 2297.333494] vfs_statx+0x89/0x150 [ 2297.336814] vfs_fstatat+0x5a/0xc0 [ 2297.340216] __do_sys_newstat+0x30/0x70 [ 2297.344056] ? lock_release+0xc5/0x260 [ 2297.347808] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2297.352686] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2297.357826] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2297.362967] do_syscall_64+0x38/0x80 [ 2297.366546] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2297.371599] RIP: 0033:0x7f48f0eeca79 [ 2297.375177] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2297.393923] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2297.401489] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2297.408621] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2297.415745] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2297.422871] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2297.430003] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2297.437142] [ 2297.439329] irq event stamp: 12836569 [ 2297.442985] hardirqs last enabled at (12836579): [] console_unlock+0x106/0x120 [ 2297.451852] hardirqs last disabled at (12836588): [] console_unlock+0xeb/0x120 [ 2297.460632] softirqs last enabled at (12836602): [] __do_softirq+0x2be/0x371 [ 2297.469324] softirqs last disabled at (12836597): [] irq_exit_rcu+0xb0/0xd0 [ 2297.477843] ---[ end trace 0000000000000000 ]--- [ 2297.482456] ------------[ cut here ]------------ [ 2297.487075] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2297.498722] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2297.507421] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2297.562920] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2297.572062] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2297.579558] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2297.585070] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2297.590573] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2297.609340] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2297.614580] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2297.621722] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2297.628858] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006b76 [ 2297.635988] R10: 0000000000242dc0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2297.643139] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2297.650289] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2297.658393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2297.664156] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2297.671305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2297.678456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2297.685625] Call Trace: [ 2297.688077] [ 2297.690200] ? __warn+0x85/0x180 [ 2297.693451] ? __list_add_valid_or_report+0x7d/0xa0 [ 2297.698346] ? report_bug+0xfc/0x1d0 [ 2297.701943] ? prb_read_valid+0x1b/0x20 [ 2297.705802] ? handle_bug+0x44/0x70 [ 2297.709292] ? exc_invalid_op+0x17/0x60 [ 2297.713150] ? asm_exc_invalid_op+0x1a/0x20 [ 2297.717374] ? __list_add_valid_or_report+0x7d/0xa0 [ 2297.722251] ? __list_add_valid_or_report+0x7d/0xa0 [ 2297.727145] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2297.732055] ceph_add_cap+0x20b/0x620 [ceph] [ 2297.736374] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2297.741106] ? lock_acquire+0xcb/0x2b0 [ 2297.744866] ceph_fill_trace+0x143/0xf30 [ceph] [ 2297.749403] ? __mutex_lock+0x97/0xce0 [ 2297.753173] ? __mutex_lock+0xaec/0xce0 [ 2297.757050] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2297.761598] ? lock_downgrade+0xfd/0x250 [ 2297.765552] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2297.769943] ceph_con_process_message+0x74/0x150 [libceph] [ 2297.775437] ? calc_signature+0xdb/0x100 [libceph] [ 2297.780258] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2297.785627] ? process_scheduled_works+0x22a/0x510 [ 2297.790425] ? process_scheduled_works+0x22a/0x510 [ 2297.795215] ceph_con_workfn+0x344/0x750 [libceph] [ 2297.800036] ? process_scheduled_works+0x22a/0x510 [ 2297.804846] process_scheduled_works+0x2a4/0x510 [ 2297.809485] worker_thread+0x188/0x340 [ 2297.813234] ? rescuer_thread+0x310/0x310 [ 2297.817263] kthread+0xf5/0x130 [ 2297.820429] ? kthread_complete_and_exit+0x20/0x20 [ 2297.825239] ret_from_fork+0x31/0x40 [ 2297.828833] ? kthread_complete_and_exit+0x20/0x20 [ 2297.833632] ret_from_fork_asm+0x11/0x20 [ 2297.837577] [ 2297.839762] irq event stamp: 10585811 [ 2297.843429] hardirqs last enabled at (10585821): [] console_unlock+0x106/0x120 [ 2297.852314] hardirqs last disabled at (10585830): [] console_unlock+0xeb/0x120 [ 2297.861109] softirqs last enabled at (10585758): [] __do_softirq+0x2be/0x371 [ 2297.869820] softirqs last disabled at (10585753): [] irq_exit_rcu+0xb0/0xd0 [ 2297.878339] ---[ end trace 0000000000000000 ]--- [ 2297.883033] ------------[ cut here ]------------ [ 2297.887652] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2297.899292] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2297.907990] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2297.963468] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2297.972591] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2297.980071] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2297.985566] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2297.991051] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2298.009797] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2298.015024] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2298.022158] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2298.029291] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006bb5 [ 2298.036422] R10: 0000000000243ff0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2298.043555] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2298.050689] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2298.058773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2298.064518] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2298.071653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2298.078786] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2298.085918] Call Trace: [ 2298.088370] [ 2298.090477] ? __warn+0x85/0x180 [ 2298.093711] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.098590] ? report_bug+0xfc/0x1d0 [ 2298.102168] ? prb_read_valid+0x1b/0x20 [ 2298.106009] ? handle_bug+0x44/0x70 [ 2298.109502] ? exc_invalid_op+0x17/0x60 [ 2298.113340] ? asm_exc_invalid_op+0x1a/0x20 [ 2298.117532] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.122417] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.127295] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2298.132184] ceph_add_cap+0x20b/0x620 [ceph] [ 2298.136470] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2298.141193] ? lock_acquire+0xcb/0x2b0 [ 2298.144953] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2298.149404] ? __mutex_lock+0x97/0xce0 [ 2298.153156] ? __mutex_lock+0xaec/0xce0 [ 2298.156997] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2298.161537] ? lock_downgrade+0xfd/0x250 [ 2298.165468] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2298.169847] ceph_con_process_message+0x74/0x150 [libceph] [ 2298.175346] ? calc_signature+0xdb/0x100 [libceph] [ 2298.180157] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2298.185489] ? process_scheduled_works+0x22a/0x510 [ 2298.190289] ? process_scheduled_works+0x22a/0x510 [ 2298.195077] ceph_con_workfn+0x344/0x750 [libceph] [ 2298.199882] ? process_scheduled_works+0x22a/0x510 [ 2298.204670] process_scheduled_works+0x2a4/0x510 [ 2298.209298] worker_thread+0x188/0x340 [ 2298.213054] ? rescuer_thread+0x310/0x310 [ 2298.217065] kthread+0xf5/0x130 [ 2298.220212] ? kthread_complete_and_exit+0x20/0x20 [ 2298.225006] ret_from_fork+0x31/0x40 [ 2298.228584] ? kthread_complete_and_exit+0x20/0x20 [ 2298.233377] ret_from_fork_asm+0x11/0x20 [ 2298.237311] [ 2298.239502] irq event stamp: 10586975 [ 2298.243170] hardirqs last enabled at (10586985): [] console_unlock+0x106/0x120 [ 2298.252036] hardirqs last disabled at (10586994): [] console_unlock+0xeb/0x120 [ 2298.260816] softirqs last enabled at (10586922): [] __do_softirq+0x2be/0x371 [ 2298.269508] softirqs last disabled at (10586917): [] irq_exit_rcu+0xb0/0xd0 [ 2298.278029] ---[ end trace 0000000000000000 ]--- [ 2298.282648] ------------[ cut here ]------------ [ 2298.287264] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2298.298912] WARNING: CPU: 0 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2298.307865] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2298.363335] CPU: 0 PID: 78686 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2298.372295] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2298.379774] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2298.385261] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2298.404007] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2298.409234] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2298.416368] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2298.423498] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006bf4 [ 2298.430633] R10: 0000000000245108 R11: ffffc90001ff7950 R12: ffff8881c5350750 [ 2298.437764] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2298.444896] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2298.452985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2298.458730] CR2: 00007fd645a418a0 CR3: 0000000120cc8005 CR4: 00000000003706f0 [ 2298.465862] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2298.472994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2298.480118] Call Trace: [ 2298.482564] [ 2298.484671] ? __warn+0x85/0x180 [ 2298.487902] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.492783] ? report_bug+0xfc/0x1d0 [ 2298.496361] ? prb_read_valid+0x1b/0x20 [ 2298.500201] ? handle_bug+0x44/0x70 [ 2298.503692] ? exc_invalid_op+0x17/0x60 [ 2298.507523] ? asm_exc_invalid_op+0x1a/0x20 [ 2298.511714] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.516592] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2298.521489] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2298.526732] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2298.531535] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2298.536247] ? __d_lookup+0xd6/0x200 [ 2298.539827] lookup_fast+0x6d/0xf0 [ 2298.543232] walk_component+0x26/0x160 [ 2298.546986] link_path_walk+0x24d/0x380 [ 2298.550824] ? path_lookupat.isra.40+0x22/0x120 [ 2298.555358] path_lookupat.isra.40+0x6b/0x120 [ 2298.559720] filename_lookup+0xd4/0x190 [ 2298.563559] ? lock_acquire+0xcb/0x2b0 [ 2298.567306] ? __might_fault+0x45/0x80 [ 2298.571061] ? find_held_lock+0x2d/0x90 [ 2298.574900] ? __might_fault+0x45/0x80 [ 2298.578656] vfs_statx+0x89/0x150 [ 2298.581976] vfs_fstatat+0x5a/0xc0 [ 2298.585378] __do_sys_newstat+0x30/0x70 [ 2298.589222] ? lock_release+0xc5/0x260 [ 2298.592972] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2298.597848] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2298.602989] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2298.608129] do_syscall_64+0x38/0x80 [ 2298.611709] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2298.616760] RIP: 0033:0x7f48f0eeca79 [ 2298.620339] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2298.639088] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2298.646652] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2298.653784] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2298.660908] RBP: 00007ffc53d60520 R08: 00000000000029f6 R09: 00000000020767c0 [ 2298.668034] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2298.675165] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2298.682296] [ 2298.684483] irq event stamp: 12961159 [ 2298.688150] hardirqs last enabled at (12961169): [] console_unlock+0x106/0x120 [ 2298.697014] hardirqs last disabled at (12961178): [] console_unlock+0xeb/0x120 [ 2298.705795] softirqs last enabled at (12961192): [] __do_softirq+0x2be/0x371 [ 2298.714488] softirqs last disabled at (12961187): [] irq_exit_rcu+0xb0/0xd0 [ 2298.723010] ---[ end trace 0000000000000000 ]--- [ 2298.727628] ------------[ cut here ]------------ [ 2298.732247] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2298.743896] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2298.752847] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2298.808315] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2298.817281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2298.824755] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2298.830241] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2298.848989] RSP: 0018:ffffc90001fdfac0 EFLAGS: 00010282 [ 2298.854213] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2298.861337] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2298.868470] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006c38 [ 2298.875595] R10: 00000000002464d8 R11: ffffc90001fdf950 R12: ffff8881d1144bc8 [ 2298.882733] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2298.889863] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2298.897947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2298.903692] CR2: 00007f9bd26f8000 CR3: 0000000179402002 CR4: 00000000003706e0 [ 2298.910825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2298.917957] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2298.925081] Call Trace: [ 2298.927528] [ 2298.929634] ? __warn+0x85/0x180 [ 2298.932867] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.937748] ? report_bug+0xfc/0x1d0 [ 2298.941325] ? prb_read_valid+0x1b/0x20 [ 2298.945167] ? handle_bug+0x44/0x70 [ 2298.948657] ? exc_invalid_op+0x17/0x60 [ 2298.952496] ? asm_exc_invalid_op+0x1a/0x20 [ 2298.956687] ? __list_add_valid_or_report+0x7d/0xa0 [ 2298.961576] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2298.966472] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2298.971712] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2298.976513] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2298.981230] ? __d_lookup+0xd6/0x200 [ 2298.984808] lookup_fast+0x6d/0xf0 [ 2298.988213] walk_component+0x26/0x160 [ 2298.991967] link_path_walk+0x24d/0x380 [ 2298.995803] ? path_lookupat.isra.40+0x22/0x120 [ 2299.000341] path_lookupat.isra.40+0x6b/0x120 [ 2299.004700] filename_lookup+0xd4/0x190 [ 2299.008540] ? lock_acquire+0xcb/0x2b0 [ 2299.012287] ? __might_fault+0x45/0x80 [ 2299.016042] ? find_held_lock+0x2d/0x90 [ 2299.019880] ? __might_fault+0x45/0x80 [ 2299.023637] vfs_statx+0x89/0x150 [ 2299.026954] vfs_fstatat+0x5a/0xc0 [ 2299.030360] __do_sys_newstat+0x30/0x70 [ 2299.034197] ? lock_release+0xc5/0x260 [ 2299.037954] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2299.042830] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2299.047969] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2299.053111] do_syscall_64+0x38/0x80 [ 2299.056688] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2299.061742] RIP: 0033:0x7f48f0eeca79 [ 2299.065322] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2299.084067] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2299.091633] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2299.098764] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2299.105889] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2299.113013] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2299.120139] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2299.127277] [ 2299.129464] irq event stamp: 12863981 [ 2299.133128] hardirqs last enabled at (12863991): [] console_unlock+0x106/0x120 [ 2299.141997] hardirqs last disabled at (12864000): [] console_unlock+0xeb/0x120 [ 2299.150777] softirqs last enabled at (12864014): [] __do_softirq+0x2be/0x371 [ 2299.159473] softirqs last disabled at (12864009): [] irq_exit_rcu+0xb0/0xd0 [ 2299.167996] ---[ end trace 0000000000000000 ]--- [ 2299.172608] ------------[ cut here ]------------ [ 2299.177226] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2299.188872] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2299.197571] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2299.253072] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2299.262197] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2299.269677] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2299.275172] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2299.280660] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2299.299404] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2299.304649] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2299.311779] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2299.318930] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006c7c [ 2299.326063] R10: 00000000002477b8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2299.333197] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2299.340328] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2299.348415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2299.354159] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2299.361284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2299.368418] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2299.375550] Call Trace: [ 2299.378003] [ 2299.380109] ? __warn+0x85/0x180 [ 2299.383344] ? __list_add_valid_or_report+0x7d/0xa0 [ 2299.388221] ? report_bug+0xfc/0x1d0 [ 2299.391801] ? prb_read_valid+0x1b/0x20 [ 2299.395659] ? handle_bug+0x44/0x70 [ 2299.399150] ? exc_invalid_op+0x17/0x60 [ 2299.402989] ? asm_exc_invalid_op+0x1a/0x20 [ 2299.407180] ? __list_add_valid_or_report+0x7d/0xa0 [ 2299.412057] ? __list_add_valid_or_report+0x7d/0xa0 [ 2299.416941] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2299.421835] ceph_add_cap+0x20b/0x620 [ceph] [ 2299.426120] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2299.430848] ceph_fill_trace+0x143/0xf30 [ceph] [ 2299.435386] ? __mutex_lock+0x97/0xce0 [ 2299.439138] ? __mutex_lock+0xaec/0xce0 [ 2299.442979] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2299.447521] ? lock_downgrade+0xfd/0x250 [ 2299.451452] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2299.455830] ceph_con_process_message+0x74/0x150 [libceph] [ 2299.461326] ? calc_signature+0xdb/0x100 [libceph] [ 2299.466130] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2299.471453] ? process_scheduled_works+0x22a/0x510 [ 2299.476245] ? process_scheduled_works+0x22a/0x510 [ 2299.481035] ceph_con_workfn+0x344/0x750 [libceph] [ 2299.485838] ? process_scheduled_works+0x22a/0x510 [ 2299.490629] process_scheduled_works+0x2a4/0x510 [ 2299.495277] worker_thread+0x188/0x340 [ 2299.499029] ? rescuer_thread+0x310/0x310 [ 2299.503040] kthread+0xf5/0x130 [ 2299.506185] ? kthread_complete_and_exit+0x20/0x20 [ 2299.510979] ret_from_fork+0x31/0x40 [ 2299.514557] ? kthread_complete_and_exit+0x20/0x20 [ 2299.519352] ret_from_fork_asm+0x11/0x20 [ 2299.523282] [ 2299.525470] irq event stamp: 10589935 [ 2299.529134] hardirqs last enabled at (10589945): [] console_unlock+0x106/0x120 [ 2299.538003] hardirqs last disabled at (10589954): [] console_unlock+0xeb/0x120 [ 2299.546781] softirqs last enabled at (10589892): [] __do_softirq+0x2be/0x371 [ 2299.555474] softirqs last disabled at (10589887): [] irq_exit_rcu+0xb0/0xd0 [ 2299.563994] ---[ end trace 0000000000000000 ]--- [ 2299.568765] ------------[ cut here ]------------ [ 2299.573380] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2299.585027] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2299.593721] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2299.649200] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2299.658324] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2299.665802] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2299.671297] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2299.676782] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2299.695530] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2299.700756] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2299.707890] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2299.715022] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006cba [ 2299.722155] R10: 00000000002488f8 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2299.729288] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2299.736421] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2299.744506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2299.750250] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2299.757376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2299.764508] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2299.771659] Call Trace: [ 2299.774113] [ 2299.776219] ? __warn+0x85/0x180 [ 2299.779453] ? __list_add_valid_or_report+0x7d/0xa0 [ 2299.784331] ? report_bug+0xfc/0x1d0 [ 2299.787910] ? prb_read_valid+0x1b/0x20 [ 2299.791750] ? handle_bug+0x44/0x70 [ 2299.795242] ? exc_invalid_op+0x17/0x60 [ 2299.799082] ? asm_exc_invalid_op+0x1a/0x20 [ 2299.803272] ? __list_add_valid_or_report+0x7d/0xa0 [ 2299.808148] ? __list_add_valid_or_report+0x7d/0xa0 [ 2299.813027] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2299.817916] ceph_add_cap+0x20b/0x620 [ceph] [ 2299.822218] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2299.826945] ? lock_acquire+0xcb/0x2b0 [ 2299.830703] ceph_fill_trace+0x143/0xf30 [ceph] [ 2299.835240] ? __mutex_lock+0x97/0xce0 [ 2299.838992] ? __mutex_lock+0xaec/0xce0 [ 2299.842834] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2299.847382] ? lock_downgrade+0xfd/0x250 [ 2299.851313] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2299.855692] ceph_con_process_message+0x74/0x150 [libceph] [ 2299.861187] ? calc_signature+0xdb/0x100 [libceph] [ 2299.865992] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2299.871323] ? process_scheduled_works+0x22a/0x510 [ 2299.876115] ? process_scheduled_works+0x22a/0x510 [ 2299.880904] ceph_con_workfn+0x344/0x750 [libceph] [ 2299.885708] ? process_scheduled_works+0x22a/0x510 [ 2299.890500] process_scheduled_works+0x2a4/0x510 [ 2299.895123] worker_thread+0x188/0x340 [ 2299.898873] ? rescuer_thread+0x310/0x310 [ 2299.902885] kthread+0xf5/0x130 [ 2299.906031] ? kthread_complete_and_exit+0x20/0x20 [ 2299.910823] ret_from_fork+0x31/0x40 [ 2299.914403] ? kthread_complete_and_exit+0x20/0x20 [ 2299.919195] ret_from_fork_asm+0x11/0x20 [ 2299.923128] [ 2299.925323] irq event stamp: 10591049 [ 2299.928988] hardirqs last enabled at (10591059): [] console_unlock+0x106/0x120 [ 2299.937855] hardirqs last disabled at (10591068): [] console_unlock+0xeb/0x120 [ 2299.946633] softirqs last enabled at (10590996): [] __do_softirq+0x2be/0x371 [ 2299.955351] softirqs last disabled at (10590991): [] irq_exit_rcu+0xb0/0xd0 [ 2299.963874] ---[ end trace 0000000000000000 ]--- [ 2299.968524] ------------[ cut here ]------------ [ 2299.973148] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2299.984790] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2299.993488] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2300.048963] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2300.058090] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2300.065569] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2300.071111] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2300.076605] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2300.095354] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2300.100587] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2300.107728] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2300.114865] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006cf9 [ 2300.122003] R10: 0000000000249bc8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2300.129145] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2300.136277] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2300.144363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.150108] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2300.157241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2300.164375] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2300.171503] Call Trace: [ 2300.173957] [ 2300.176062] ? __warn+0x85/0x180 [ 2300.179296] ? __list_add_valid_or_report+0x7d/0xa0 [ 2300.184174] ? report_bug+0xfc/0x1d0 [ 2300.187754] ? prb_read_valid+0x1b/0x20 [ 2300.191594] ? handle_bug+0x44/0x70 [ 2300.195086] ? exc_invalid_op+0x17/0x60 [ 2300.198927] ? asm_exc_invalid_op+0x1a/0x20 [ 2300.203116] ? __list_add_valid_or_report+0x7d/0xa0 [ 2300.207991] ? __list_add_valid_or_report+0x7d/0xa0 [ 2300.212869] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2300.217760] ceph_add_cap+0x20b/0x620 [ceph] [ 2300.222046] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2300.226769] ? lock_acquire+0xcb/0x2b0 [ 2300.230530] ceph_fill_trace+0x143/0xf30 [ceph] [ 2300.235066] ? __mutex_lock+0x97/0xce0 [ 2300.238817] ? __mutex_lock+0xaec/0xce0 [ 2300.242660] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2300.247209] ? lock_downgrade+0xfd/0x250 [ 2300.251140] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2300.255518] ceph_con_process_message+0x74/0x150 [libceph] [ 2300.261014] ? calc_signature+0xdb/0x100 [libceph] [ 2300.265817] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2300.271140] ? process_scheduled_works+0x22a/0x510 [ 2300.275933] ? process_scheduled_works+0x22a/0x510 [ 2300.280723] ceph_con_workfn+0x344/0x750 [libceph] [ 2300.285526] ? process_scheduled_works+0x22a/0x510 [ 2300.290316] process_scheduled_works+0x2a4/0x510 [ 2300.294940] worker_thread+0x188/0x340 [ 2300.298690] ? rescuer_thread+0x310/0x310 [ 2300.302701] kthread+0xf5/0x130 [ 2300.305850] ? kthread_complete_and_exit+0x20/0x20 [ 2300.310641] ret_from_fork+0x31/0x40 [ 2300.314219] ? kthread_complete_and_exit+0x20/0x20 [ 2300.319012] ret_from_fork_asm+0x11/0x20 [ 2300.322945] [ 2300.325130] irq event stamp: 10592229 [ 2300.328798] hardirqs last enabled at (10592239): [] console_unlock+0x106/0x120 [ 2300.337663] hardirqs last disabled at (10592248): [] console_unlock+0xeb/0x120 [ 2300.346442] softirqs last enabled at (10592174): [] __do_softirq+0x2be/0x371 [ 2300.355135] softirqs last disabled at (10592169): [] irq_exit_rcu+0xb0/0xd0 [ 2300.363654] ---[ end trace 0000000000000000 ]--- [ 2300.368276] ------------[ cut here ]------------ [ 2300.372894] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2300.384546] WARNING: CPU: 0 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2300.393494] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2300.448956] CPU: 0 PID: 78680 Comm: dbench Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2300.457905] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2300.465378] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2300.470862] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2300.489610] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2300.494846] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2300.501977] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2300.509118] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006d38 [ 2300.516244] R10: 000000000024b0e0 R11: ffffc90001fc7950 R12: ffff8881d106df58 [ 2300.523376] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2300.530509] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2300.538594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.544339] CR2: 00007fd645a418a0 CR3: 0000000113500002 CR4: 00000000003706f0 [ 2300.551464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2300.558596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2300.565722] Call Trace: [ 2300.568175] [ 2300.570282] ? __warn+0x85/0x180 [ 2300.573513] ? __list_add_valid_or_report+0x7d/0xa0 [ 2300.578393] ? report_bug+0xfc/0x1d0 [ 2300.581974] ? prb_read_valid+0x1b/0x20 [ 2300.585812] ? handle_bug+0x44/0x70 [ 2300.589305] ? exc_invalid_op+0x17/0x60 [ 2300.593145] ? asm_exc_invalid_op+0x1a/0x20 [ 2300.597333] ? __list_add_valid_or_report+0x7d/0xa0 [ 2300.602211] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2300.607100] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2300.612335] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2300.617134] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2300.621850] ? __d_lookup+0xd6/0x200 [ 2300.625430] lookup_fast+0x6d/0xf0 [ 2300.628835] walk_component+0x26/0x160 [ 2300.632589] link_path_walk+0x24d/0x380 [ 2300.636425] ? path_lookupat.isra.40+0x22/0x120 [ 2300.640961] path_lookupat.isra.40+0x6b/0x120 [ 2300.645322] filename_lookup+0xd4/0x190 [ 2300.649160] ? lock_acquire+0xcb/0x2b0 [ 2300.652910] ? __might_fault+0x45/0x80 [ 2300.656665] ? find_held_lock+0x2d/0x90 [ 2300.660503] ? __might_fault+0x45/0x80 [ 2300.664257] vfs_statx+0x89/0x150 [ 2300.667578] vfs_fstatat+0x5a/0xc0 [ 2300.670981] __do_sys_newstat+0x30/0x70 [ 2300.674819] ? lock_release+0xc5/0x260 [ 2300.678586] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2300.683465] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2300.688601] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2300.693740] do_syscall_64+0x38/0x80 [ 2300.697320] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2300.702371] RIP: 0033:0x7f48f0eeca79 [ 2300.705951] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2300.724698] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2300.732264] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2300.739397] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2300.746529] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2300.753662] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2300.760795] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2300.767933] [ 2300.770121] irq event stamp: 12949561 [ 2300.773786] hardirqs last enabled at (12949571): [] console_unlock+0x106/0x120 [ 2300.782657] hardirqs last disabled at (12949580): [] console_unlock+0xeb/0x120 [ 2300.791432] softirqs last enabled at (12949594): [] __do_softirq+0x2be/0x371 [ 2300.800123] softirqs last disabled at (12949589): [] irq_exit_rcu+0xb0/0xd0 [ 2300.808643] ---[ end trace 0000000000000000 ]--- [ 2300.813265] ------------[ cut here ]------------ [ 2300.817881] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2300.829519] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2300.838217] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2300.893694] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2300.902820] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2300.910299] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2300.915792] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2300.921277] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2300.940025] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2300.945250] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2300.952383] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2300.959518] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006d7c [ 2300.966654] R10: 000000000024c598 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2300.973789] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2300.980915] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2300.989002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2300.994746] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2301.001882] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2301.009014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2301.016147] Call Trace: [ 2301.018600] [ 2301.020705] ? __warn+0x85/0x180 [ 2301.023939] ? __list_add_valid_or_report+0x7d/0xa0 [ 2301.028818] ? report_bug+0xfc/0x1d0 [ 2301.032396] ? prb_read_valid+0x1b/0x20 [ 2301.036239] ? handle_bug+0x44/0x70 [ 2301.039728] ? exc_invalid_op+0x17/0x60 [ 2301.043569] ? asm_exc_invalid_op+0x1a/0x20 [ 2301.047759] ? __list_add_valid_or_report+0x7d/0xa0 [ 2301.052635] ? __list_add_valid_or_report+0x7d/0xa0 [ 2301.057513] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2301.062403] ceph_add_cap+0x20b/0x620 [ceph] [ 2301.066689] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2301.071413] ? lock_acquire+0xcb/0x2b0 [ 2301.075172] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2301.079624] ? __mutex_lock+0x97/0xce0 [ 2301.083373] ? __mutex_lock+0xaec/0xce0 [ 2301.087217] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2301.091764] ? lock_downgrade+0xfd/0x250 [ 2301.095695] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2301.100075] ceph_con_process_message+0x74/0x150 [libceph] [ 2301.105572] ? calc_signature+0xdb/0x100 [libceph] [ 2301.110383] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2301.115715] ? process_scheduled_works+0x22a/0x510 [ 2301.120507] ? process_scheduled_works+0x22a/0x510 [ 2301.125295] ceph_con_workfn+0x344/0x750 [libceph] [ 2301.130100] ? process_scheduled_works+0x22a/0x510 [ 2301.134891] process_scheduled_works+0x2a4/0x510 [ 2301.139515] worker_thread+0x188/0x340 [ 2301.143271] ? rescuer_thread+0x310/0x310 [ 2301.147285] kthread+0xf5/0x130 [ 2301.150432] ? kthread_complete_and_exit+0x20/0x20 [ 2301.155224] ret_from_fork+0x31/0x40 [ 2301.158802] ? kthread_complete_and_exit+0x20/0x20 [ 2301.163595] ret_from_fork_asm+0x11/0x20 [ 2301.167529] [ 2301.169723] irq event stamp: 10594301 [ 2301.173388] hardirqs last enabled at (10594311): [] console_unlock+0x106/0x120 [ 2301.182254] hardirqs last disabled at (10594320): [] console_unlock+0xeb/0x120 [ 2301.191033] softirqs last enabled at (10594248): [] __do_softirq+0x2be/0x371 [ 2301.199727] softirqs last disabled at (10594243): [] irq_exit_rcu+0xb0/0xd0 [ 2301.208248] ---[ end trace 0000000000000000 ]--- [ 2301.212872] ------------[ cut here ]------------ [ 2301.217485] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2301.229129] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2301.237827] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2301.293304] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2301.302430] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2301.309907] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2301.315404] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2301.320889] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2301.339635] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2301.344860] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2301.351994] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2301.359129] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006dbb [ 2301.366260] R10: 000000000024d850 R11: ffffc9000030b828 R12: ffff8881c5351118 [ 2301.373394] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2301.380525] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2301.388611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2301.394356] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2301.401490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2301.408623] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2301.415757] Call Trace: [ 2301.418209] [ 2301.420314] ? __warn+0x85/0x180 [ 2301.423549] ? __list_add_valid_or_report+0x7d/0xa0 [ 2301.428429] ? report_bug+0xfc/0x1d0 [ 2301.432007] ? prb_read_valid+0x1b/0x20 [ 2301.435847] ? handle_bug+0x44/0x70 [ 2301.439339] ? exc_invalid_op+0x17/0x60 [ 2301.443177] ? asm_exc_invalid_op+0x1a/0x20 [ 2301.447369] ? __list_add_valid_or_report+0x7d/0xa0 [ 2301.452245] ? __list_add_valid_or_report+0x7d/0xa0 [ 2301.457123] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2301.462014] ceph_add_cap+0x20b/0x620 [ceph] [ 2301.466299] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2301.471029] ceph_fill_trace+0x143/0xf30 [ceph] [ 2301.475567] ? __mutex_lock+0x97/0xce0 [ 2301.479318] ? __mutex_lock+0xaec/0xce0 [ 2301.483160] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2301.487709] ? lock_downgrade+0xfd/0x250 [ 2301.491639] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2301.496018] ceph_con_process_message+0x74/0x150 [libceph] [ 2301.501515] ? calc_signature+0xdb/0x100 [libceph] [ 2301.506299] watchdog: BUG: soft lockup - CPU#7 stuck for 26s! [kworker/7:4:78808] [ 2301.506331] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2301.513790] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [ 2301.519111] ? process_scheduled_works+0x22a/0x510 [ 2301.519113] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [ 2301.541609] ? process_scheduled_works+0x22a/0x510 [ 2301.541610] ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca [ 2301.566185] ceph_con_workfn+0x344/0x750 [libceph] [ 2301.586660] fuse [ 2301.586662] irq event stamp: 304706 [ 2301.586663] hardirqs last enabled at (304705): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2301.586669] hardirqs last disabled at (304706): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2301.591459] ? process_scheduled_works+0x22a/0x510 [ 2301.593376] softirqs last enabled at (304704): [] __do_softirq+0x2be/0x371 [ 2301.596864] process_scheduled_works+0x2a4/0x510 [ 2301.606846] softirqs last disabled at (304699): [] irq_exit_rcu+0xb0/0xd0 [ 2301.606850] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W 6.6.0-rc5-g5b73517e0e2f #1 [ 2301.616414] worker_thread+0x188/0x340 [ 2301.621189] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2301.621191] Workqueue: events delayed_work [ceph] [ 2301.629707] ? rescuer_thread+0x310/0x310 [ 2301.634310] [ 2301.634311] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [ 2301.642652] kthread+0xf5/0x130 [ 2301.652027] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [ 2301.652029] RSP: 0018:ffffc90002117d80 EFLAGS: 00000202 [ 2301.652031] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 2301.655783] ? kthread_complete_and_exit+0x20/0x20 [ 2301.663251] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [ 2301.663252] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 2301.663253] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2301.663254] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2301.663256] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2301.667961] ret_from_fork+0x31/0x40 [ 2301.671960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2301.671962] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2301.671963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2301.671965] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2301.673462] ? kthread_complete_and_exit+0x20/0x20 [ 2301.678937] Call Trace: [ 2301.678939] [ 2301.678941] ? watchdog_timer_fn+0x1b4/0x220 [ 2301.682087] ret_from_fork_asm+0x11/0x20 [ 2301.700822] ? __hrtimer_run_queues+0x1d3/0x360 [ 2301.700832] ? hrtimer_interrupt+0x104/0x240 [ 2301.706058] [ 2301.713171] ? __local_bh_enable+0x50/0x60 [ 2301.713180] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2301.717967] irq event stamp: 10595389 [ 2301.725087] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2301.725092] [ 2301.732214] hardirqs last enabled at (10595401): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2301.739336] [ 2301.739338] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2301.739350] ? queued_spin_lock_slowpath+0x18/0x2e0 [ 2301.746472] hardirqs last disabled at (10595412): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2301.754549] do_raw_spin_lock+0xb5/0xc0 [ 2301.758129] softirqs last enabled at (10595372): [] __do_softirq+0x2be/0x371 [ 2301.763863] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2301.770998] softirqs last disabled at (10595433): [] irq_exit_rcu+0xb0/0xd0 [ 2301.778123] delayed_work+0x1d8/0x280 [ceph] [ 2301.785255] ---[ end trace 0000000000000000 ]--- [ 2301.790039] ? process_scheduled_works+0x22a/0x510 [ 2301.792485] ------------[ cut here ]------------ [ 2301.794493] process_scheduled_works+0x2a4/0x510 [ 2301.794501] worker_thread+0x188/0x340 [ 2301.798767] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2301.802682] ? rescuer_thread+0x310/0x310 [ 2301.802686] kthread+0xf5/0x130 [ 2301.807225] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2301.811478] ? kthread_complete_and_exit+0x20/0x20 [ 2301.813664] Modules linked in: [ 2301.817754] ret_from_fork+0x31/0x40 [ 2301.822973] ceph [ 2301.826627] ? kthread_complete_and_exit+0x20/0x20 [ 2301.826632] ret_from_fork_asm+0x11/0x20 [ 2301.831588] libceph [ 2301.833694] [ 2301.843843] dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2302.025608] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2302.034559] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2302.042036] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2302.047524] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2302.066270] RSP: 0018:ffffc90001fd7b78 EFLAGS: 00010286 [ 2302.071494] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2302.078619] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2302.085743] RBP: ffff88817f867000 R08: 0000000000000001 R09: 0000000000000001 [ 2302.092868] R10: ffffffff811806bf R11: 0000000000000001 R12: ffff8881c5350750 [ 2302.100002] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2302.107133] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2302.115219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2302.120958] CR2: 00007fb38cdd10b0 CR3: 0000000120c1e002 CR4: 00000000003706e0 [ 2302.128090] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2302.135221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2302.142347] Call Trace: [ 2302.144790] [ 2302.146888] ? __warn+0x85/0x180 [ 2302.150123] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.155000] ? report_bug+0xfc/0x1d0 [ 2302.158583] ? handle_bug+0x44/0x70 [ 2302.162074] ? exc_invalid_op+0x17/0x60 [ 2302.165913] ? asm_exc_invalid_op+0x1a/0x20 [ 2302.170101] ? console_flush_all+0x1ff/0x5a0 [ 2302.174372] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.179252] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.184129] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2302.189019] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2302.194256] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2302.199056] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2302.203771] ? __d_lookup+0xd6/0x200 [ 2302.207351] lookup_fast+0x6d/0xf0 [ 2302.210754] walk_component+0x26/0x160 [ 2302.214508] link_path_walk+0x24d/0x380 [ 2302.218346] ? path_openat+0x80/0x9d0 [ 2302.222017] path_openat+0x8e/0x9d0 [ 2302.225509] do_filp_open+0xb8/0x120 [ 2302.229089] ? lock_release+0xc5/0x260 [ 2302.232838] ? _raw_spin_unlock+0x23/0x40 [ 2302.236850] ? alloc_fd+0x13f/0x200 [ 2302.240346] do_sys_openat2+0x249/0x290 [ 2302.244186] do_sys_open+0x4f/0x80 [ 2302.247589] do_syscall_64+0x38/0x80 [ 2302.251169] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2302.256219] RIP: 0033:0x7f48f0ef2805 [ 2302.259800] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2302.278546] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2302.286113] RAX: ffffffffffffffda RBX: 000000000207815f RCX: 00007f48f0ef2805 [ 2302.293245] RDX: 0000000000090800 RSI: 0000000002078140 RDI: 00000000ffffff9c [ 2302.300376] RBP: 0000000002078140 R08: 0000000002076674 R09: 0000000000000000 [ 2302.307500] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2302.314627] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2302.321765] [ 2302.323952] irq event stamp: 13000881 [ 2302.327616] hardirqs last enabled at (13000891): [] console_unlock+0x106/0x120 [ 2302.336483] hardirqs last disabled at (13000900): [] console_unlock+0xeb/0x120 [ 2302.345262] softirqs last enabled at (13000818): [] __do_softirq+0x2be/0x371 [ 2302.353958] softirqs last disabled at (13000919): [] irq_exit_rcu+0xb0/0xd0 [ 2302.362474] ---[ end trace 0000000000000000 ]--- [ 2302.367087] ------------[ cut here ]------------ [ 2302.371706] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2302.383353] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2302.392050] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2302.447526] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2302.456650] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2302.464130] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2302.469626] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2302.475110] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2302.493857] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2302.499084] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2302.506216] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2302.513351] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006e6f [ 2302.520483] R10: 0000000000250d80 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2302.527614] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2302.534748] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2302.542833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2302.548578] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2302.555712] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2302.562846] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2302.569978] Call Trace: [ 2302.572430] [ 2302.574537] ? __warn+0x85/0x180 [ 2302.577772] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.582651] ? report_bug+0xfc/0x1d0 [ 2302.586229] ? prb_read_valid+0x1b/0x20 [ 2302.590070] ? handle_bug+0x44/0x70 [ 2302.593560] ? exc_invalid_op+0x17/0x60 [ 2302.597400] ? asm_exc_invalid_op+0x1a/0x20 [ 2302.601591] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.606466] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.611346] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2302.616236] ceph_add_cap+0x20b/0x620 [ceph] [ 2302.620521] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2302.625245] ? lock_acquire+0xcb/0x2b0 [ 2302.629004] ceph_fill_trace+0x143/0xf30 [ceph] [ 2302.633542] ? __mutex_lock+0x97/0xce0 [ 2302.637293] ? __mutex_lock+0xaec/0xce0 [ 2302.641134] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2302.645682] ? lock_downgrade+0xfd/0x250 [ 2302.649612] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2302.653986] ceph_con_process_message+0x74/0x150 [libceph] [ 2302.659480] ? calc_signature+0xdb/0x100 [libceph] [ 2302.664285] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2302.669617] ? process_scheduled_works+0x22a/0x510 [ 2302.674417] ? process_scheduled_works+0x22a/0x510 [ 2302.679205] ceph_con_workfn+0x344/0x750 [libceph] [ 2302.684009] ? process_scheduled_works+0x22a/0x510 [ 2302.688800] process_scheduled_works+0x2a4/0x510 [ 2302.693425] worker_thread+0x188/0x340 [ 2302.697181] ? rescuer_thread+0x310/0x310 [ 2302.701195] kthread+0xf5/0x130 [ 2302.704341] ? kthread_complete_and_exit+0x20/0x20 [ 2302.709133] ret_from_fork+0x31/0x40 [ 2302.712711] ? kthread_complete_and_exit+0x20/0x20 [ 2302.717505] ret_from_fork_asm+0x11/0x20 [ 2302.721438] [ 2302.723633] irq event stamp: 10597737 [ 2302.727298] hardirqs last enabled at (10597747): [] console_unlock+0x106/0x120 [ 2302.736164] hardirqs last disabled at (10597756): [] console_unlock+0xeb/0x120 [ 2302.744944] softirqs last enabled at (10597684): [] __do_softirq+0x2be/0x371 [ 2302.753636] softirqs last disabled at (10597679): [] irq_exit_rcu+0xb0/0xd0 [ 2302.762156] ---[ end trace 0000000000000000 ]--- [ 2302.766930] ------------[ cut here ]------------ [ 2302.771556] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2302.783200] WARNING: CPU: 6 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2302.792166] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2302.847634] CPU: 6 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2302.856589] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2302.864069] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2302.869554] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2302.888301] RSP: 0018:ffffc90001fcfb78 EFLAGS: 00010286 [ 2302.893525] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2302.900651] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2302.907783] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006eae [ 2302.914916] R10: 0000000000252010 R11: ffffc90001fcfa08 R12: ffff8881c5350750 [ 2302.922048] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2302.929174] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2302.937259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2302.943006] CR2: 00007f9bd26f8000 CR3: 000000011aeae001 CR4: 00000000003706e0 [ 2302.950141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2302.957270] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2302.964407] Call Trace: [ 2302.966858] [ 2302.968964] ? __warn+0x85/0x180 [ 2302.972197] ? __list_add_valid_or_report+0x7d/0xa0 [ 2302.977076] ? report_bug+0xfc/0x1d0 [ 2302.980656] ? prb_read_valid+0x1b/0x20 [ 2302.984494] ? handle_bug+0x44/0x70 [ 2302.987985] ? exc_invalid_op+0x17/0x60 [ 2302.991817] ? asm_exc_invalid_op+0x1a/0x20 [ 2302.996007] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.000883] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.005762] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2303.010652] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2303.015887] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2303.020689] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2303.025410] ? __d_lookup+0xd6/0x200 [ 2303.028990] lookup_fast+0x6d/0xf0 [ 2303.032397] walk_component+0x26/0x160 [ 2303.036150] link_path_walk+0x24d/0x380 [ 2303.039987] ? path_openat+0x80/0x9d0 [ 2303.043658] path_openat+0x8e/0x9d0 [ 2303.047150] do_filp_open+0xb8/0x120 [ 2303.050730] ? lock_release+0xc5/0x260 [ 2303.054481] ? _raw_spin_unlock+0x23/0x40 [ 2303.058491] ? alloc_fd+0x13f/0x200 [ 2303.061987] do_sys_openat2+0x249/0x290 [ 2303.065826] do_sys_open+0x4f/0x80 [ 2303.069231] do_syscall_64+0x38/0x80 [ 2303.072808] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2303.077861] RIP: 0033:0x7f48f0ef2805 [ 2303.081441] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2303.100188] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2303.107755] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2303.114886] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2303.122018] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2303.129142] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2303.136266] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2303.143396] [ 2303.145583] irq event stamp: 12914257 [ 2303.149242] hardirqs last enabled at (12914267): [] console_unlock+0x106/0x120 [ 2303.158108] hardirqs last disabled at (12914276): [] console_unlock+0xeb/0x120 [ 2303.166886] softirqs last enabled at (12914194): [] __do_softirq+0x2be/0x371 [ 2303.175581] softirqs last disabled at (12914295): [] irq_exit_rcu+0xb0/0xd0 [ 2303.184099] ---[ end trace 0000000000000000 ]--- [ 2303.188720] ------------[ cut here ]------------ [ 2303.193339] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2303.205000] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2303.213720] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2303.269199] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2303.278327] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2303.285807] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2303.291308] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2303.296794] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2303.315541] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2303.320766] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2303.327901] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2303.335033] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006eed [ 2303.342168] R10: 0000000000253238 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2303.349299] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2303.356432] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2303.364518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2303.370262] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2303.377388] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2303.384521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2303.391654] Call Trace: [ 2303.394105] [ 2303.396214] ? __warn+0x85/0x180 [ 2303.399445] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.404326] ? report_bug+0xfc/0x1d0 [ 2303.407905] ? prb_read_valid+0x1b/0x20 [ 2303.411746] ? handle_bug+0x44/0x70 [ 2303.415235] ? exc_invalid_op+0x17/0x60 [ 2303.419075] ? asm_exc_invalid_op+0x1a/0x20 [ 2303.423266] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.428141] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.433020] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2303.437912] ceph_add_cap+0x20b/0x620 [ceph] [ 2303.442205] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2303.446927] ? lock_acquire+0xcb/0x2b0 [ 2303.450680] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2303.455129] ? __mutex_lock+0x97/0xce0 [ 2303.458883] ? __mutex_lock+0xaec/0xce0 [ 2303.462733] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2303.467279] ? lock_downgrade+0xfd/0x250 [ 2303.471213] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2303.475591] ceph_con_process_message+0x74/0x150 [libceph] [ 2303.481085] ? calc_signature+0xdb/0x100 [libceph] [ 2303.485890] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2303.491222] ? process_scheduled_works+0x22a/0x510 [ 2303.496016] ? process_scheduled_works+0x22a/0x510 [ 2303.500804] ceph_con_workfn+0x344/0x750 [libceph] [ 2303.505608] ? process_scheduled_works+0x22a/0x510 [ 2303.510398] process_scheduled_works+0x2a4/0x510 [ 2303.515023] worker_thread+0x188/0x340 [ 2303.518779] ? rescuer_thread+0x310/0x310 [ 2303.522793] kthread+0xf5/0x130 [ 2303.525938] ? kthread_complete_and_exit+0x20/0x20 [ 2303.530734] ret_from_fork+0x31/0x40 [ 2303.534311] ? kthread_complete_and_exit+0x20/0x20 [ 2303.539104] ret_from_fork_asm+0x11/0x20 [ 2303.543035] [ 2303.545229] irq event stamp: 10600685 [ 2303.548895] hardirqs last enabled at (10600695): [] console_unlock+0x106/0x120 [ 2303.557763] hardirqs last disabled at (10600704): [] console_unlock+0xeb/0x120 [ 2303.566542] softirqs last enabled at (10600652): [] __do_softirq+0x2be/0x371 [ 2303.575234] softirqs last disabled at (10600647): [] irq_exit_rcu+0xb0/0xd0 [ 2303.583753] ---[ end trace 0000000000000000 ]--- [ 2303.588384] ------------[ cut here ]------------ [ 2303.593016] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2303.604662] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2303.613350] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2303.668829] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2303.677953] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2303.685432] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2303.690926] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2303.696412] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2303.715161] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2303.720385] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2303.727520] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2303.734653] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006f2c [ 2303.741785] R10: 00000000002544a0 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2303.748917] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2303.756050] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2303.764137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2303.769882] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2303.777017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2303.784147] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2303.791281] Call Trace: [ 2303.793738] [ 2303.795839] ? __warn+0x85/0x180 [ 2303.799073] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.803952] ? report_bug+0xfc/0x1d0 [ 2303.807531] ? prb_read_valid+0x1b/0x20 [ 2303.811371] ? handle_bug+0x44/0x70 [ 2303.814864] ? exc_invalid_op+0x17/0x60 [ 2303.818702] ? asm_exc_invalid_op+0x1a/0x20 [ 2303.822894] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.827769] ? __list_add_valid_or_report+0x7d/0xa0 [ 2303.832648] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2303.837538] ceph_add_cap+0x20b/0x620 [ceph] [ 2303.841824] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2303.846551] ceph_fill_trace+0x143/0xf30 [ceph] [ 2303.851090] ? __mutex_lock+0x97/0xce0 [ 2303.854845] ? __mutex_lock+0xaec/0xce0 [ 2303.858685] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2303.863226] ? lock_downgrade+0xfd/0x250 [ 2303.867154] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2303.871526] ceph_con_process_message+0x74/0x150 [libceph] [ 2303.877031] ? calc_signature+0xdb/0x100 [libceph] [ 2303.881833] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2303.887156] ? process_scheduled_works+0x22a/0x510 [ 2303.891949] ? process_scheduled_works+0x22a/0x510 [ 2303.896739] ceph_con_workfn+0x344/0x750 [libceph] [ 2303.901542] ? process_scheduled_works+0x22a/0x510 [ 2303.906332] process_scheduled_works+0x2a4/0x510 [ 2303.910958] worker_thread+0x188/0x340 [ 2303.914733] ? rescuer_thread+0x310/0x310 [ 2303.918743] kthread+0xf5/0x130 [ 2303.921891] ? kthread_complete_and_exit+0x20/0x20 [ 2303.926684] ret_from_fork+0x31/0x40 [ 2303.930261] ? kthread_complete_and_exit+0x20/0x20 [ 2303.935054] ret_from_fork_asm+0x11/0x20 [ 2303.938988] [ 2303.941183] irq event stamp: 10601743 [ 2303.944848] hardirqs last enabled at (10601753): [] console_unlock+0x106/0x120 [ 2303.953733] hardirqs last disabled at (10601762): [] console_unlock+0xeb/0x120 [ 2303.962512] softirqs last enabled at (10601700): [] __do_softirq+0x2be/0x371 [ 2303.971203] softirqs last disabled at (10601695): [] irq_exit_rcu+0xb0/0xd0 [ 2303.979741] ---[ end trace 0000000000000000 ]--- [ 2303.984397] ------------[ cut here ]------------ [ 2303.989017] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2304.000657] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2304.009347] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2304.064823] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2304.073950] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2304.081430] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2304.086924] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2304.092408] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2304.111158] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2304.116381] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2304.123515] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2304.130650] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006f6a [ 2304.137781] R10: 00000000002556c0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2304.144912] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2304.152046] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2304.160133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.165882] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2304.173011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2304.180145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2304.187277] Call Trace: [ 2304.189729] [ 2304.191837] ? __warn+0x85/0x180 [ 2304.195070] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.199948] ? report_bug+0xfc/0x1d0 [ 2304.203526] ? prb_read_valid+0x1b/0x20 [ 2304.207369] ? handle_bug+0x44/0x70 [ 2304.210859] ? exc_invalid_op+0x17/0x60 [ 2304.214700] ? asm_exc_invalid_op+0x1a/0x20 [ 2304.218889] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.223766] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.228643] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2304.233535] ceph_add_cap+0x20b/0x620 [ceph] [ 2304.237829] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2304.242553] ? lock_acquire+0xcb/0x2b0 [ 2304.246312] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2304.250761] ? __mutex_lock+0x97/0xce0 [ 2304.254513] ? __mutex_lock+0xaec/0xce0 [ 2304.258356] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2304.262903] ? lock_downgrade+0xfd/0x250 [ 2304.266834] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2304.271216] ceph_con_process_message+0x74/0x150 [libceph] [ 2304.276710] ? calc_signature+0xdb/0x100 [libceph] [ 2304.281530] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2304.286854] ? process_scheduled_works+0x22a/0x510 [ 2304.291646] ? process_scheduled_works+0x22a/0x510 [ 2304.296434] ceph_con_workfn+0x344/0x750 [libceph] [ 2304.301238] ? process_scheduled_works+0x22a/0x510 [ 2304.306029] process_scheduled_works+0x2a4/0x510 [ 2304.310654] worker_thread+0x188/0x340 [ 2304.314402] ? rescuer_thread+0x310/0x310 [ 2304.318413] kthread+0xf5/0x130 [ 2304.321563] ? kthread_complete_and_exit+0x20/0x20 [ 2304.326355] ret_from_fork+0x31/0x40 [ 2304.329932] ? kthread_complete_and_exit+0x20/0x20 [ 2304.334726] ret_from_fork_asm+0x11/0x20 [ 2304.338659] [ 2304.340852] irq event stamp: 10602857 [ 2304.344518] hardirqs last enabled at (10602867): [] console_unlock+0x106/0x120 [ 2304.353385] hardirqs last disabled at (10602876): [] console_unlock+0xeb/0x120 [ 2304.362166] softirqs last enabled at (10602804): [] __do_softirq+0x2be/0x371 [ 2304.370859] softirqs last disabled at (10602895): [] irq_exit_rcu+0xb0/0xd0 [ 2304.379376] ---[ end trace 0000000000000000 ]--- [ 2304.384032] ------------[ cut here ]------------ [ 2304.388651] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2304.400295] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2304.408992] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2304.464487] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2304.473610] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2304.481090] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2304.486584] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2304.492072] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2304.510817] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2304.516046] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2304.523177] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2304.530312] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006fa9 [ 2304.537443] R10: 0000000000256928 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2304.544575] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2304.551708] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2304.559794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.565540] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2304.572674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2304.579805] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2304.586939] Call Trace: [ 2304.589392] [ 2304.591498] ? __warn+0x85/0x180 [ 2304.594749] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.599628] ? report_bug+0xfc/0x1d0 [ 2304.603207] ? prb_read_valid+0x1b/0x20 [ 2304.607048] ? handle_bug+0x44/0x70 [ 2304.610537] ? exc_invalid_op+0x17/0x60 [ 2304.614378] ? asm_exc_invalid_op+0x1a/0x20 [ 2304.618569] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.623444] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.628322] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2304.633213] ceph_add_cap+0x20b/0x620 [ceph] [ 2304.637499] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2304.642227] ceph_fill_trace+0x143/0xf30 [ceph] [ 2304.646767] ? __mutex_lock+0x97/0xce0 [ 2304.650517] ? __mutex_lock+0xaec/0xce0 [ 2304.654358] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2304.658900] ? lock_downgrade+0xfd/0x250 [ 2304.662832] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2304.667211] ceph_con_process_message+0x74/0x150 [libceph] [ 2304.672707] ? calc_signature+0xdb/0x100 [libceph] [ 2304.677535] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2304.682858] ? process_scheduled_works+0x22a/0x510 [ 2304.687650] ? process_scheduled_works+0x22a/0x510 [ 2304.692440] ceph_con_workfn+0x344/0x750 [libceph] [ 2304.697245] ? process_scheduled_works+0x22a/0x510 [ 2304.702035] process_scheduled_works+0x2a4/0x510 [ 2304.706659] worker_thread+0x188/0x340 [ 2304.710415] ? rescuer_thread+0x310/0x310 [ 2304.714428] kthread+0xf5/0x130 [ 2304.717574] ? kthread_complete_and_exit+0x20/0x20 [ 2304.722368] ret_from_fork+0x31/0x40 [ 2304.725945] ? kthread_complete_and_exit+0x20/0x20 [ 2304.730755] ret_from_fork_asm+0x11/0x20 [ 2304.734689] [ 2304.736873] irq event stamp: 10603917 [ 2304.740532] hardirqs last enabled at (10603927): [] console_unlock+0x106/0x120 [ 2304.749398] hardirqs last disabled at (10603936): [] console_unlock+0xeb/0x120 [ 2304.758177] softirqs last enabled at (10603874): [] __do_softirq+0x2be/0x371 [ 2304.766871] softirqs last disabled at (10603869): [] irq_exit_rcu+0xb0/0xd0 [ 2304.775389] ---[ end trace 0000000000000000 ]--- [ 2304.780062] ------------[ cut here ]------------ [ 2304.784681] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2304.796327] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2304.805037] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2304.860508] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2304.869636] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2304.877115] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2304.882616] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2304.888100] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2304.906848] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2304.912074] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2304.919206] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2304.926341] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100006fe7 [ 2304.933476] R10: 0000000000257c70 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2304.940606] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2304.947756] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2304.955843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.961590] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2304.968721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2304.975853] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2304.982988] Call Trace: [ 2304.985438] [ 2304.987545] ? __warn+0x85/0x180 [ 2304.990781] ? __list_add_valid_or_report+0x7d/0xa0 [ 2304.995657] ? report_bug+0xfc/0x1d0 [ 2304.999236] ? prb_read_valid+0x1b/0x20 [ 2305.003078] ? handle_bug+0x44/0x70 [ 2305.006569] ? exc_invalid_op+0x17/0x60 [ 2305.010408] ? asm_exc_invalid_op+0x1a/0x20 [ 2305.014600] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.019475] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.024352] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2305.029243] ceph_add_cap+0x20b/0x620 [ceph] [ 2305.033528] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2305.038247] ceph_fill_trace+0x143/0xf30 [ceph] [ 2305.042789] ? __mutex_lock+0x97/0xce0 [ 2305.046541] ? __mutex_lock+0xaec/0xce0 [ 2305.050380] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2305.054922] ? lock_downgrade+0xfd/0x250 [ 2305.058854] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2305.063232] ceph_con_process_message+0x74/0x150 [libceph] [ 2305.068727] ? calc_signature+0xdb/0x100 [libceph] [ 2305.073566] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2305.078887] ? process_scheduled_works+0x22a/0x510 [ 2305.083680] ? process_scheduled_works+0x22a/0x510 [ 2305.088471] ceph_con_workfn+0x344/0x750 [libceph] [ 2305.093275] ? process_scheduled_works+0x22a/0x510 [ 2305.098064] process_scheduled_works+0x2a4/0x510 [ 2305.102688] worker_thread+0x188/0x340 [ 2305.106437] ? rescuer_thread+0x310/0x310 [ 2305.110449] kthread+0xf5/0x130 [ 2305.113596] ? kthread_complete_and_exit+0x20/0x20 [ 2305.118390] ret_from_fork+0x31/0x40 [ 2305.121967] ? kthread_complete_and_exit+0x20/0x20 [ 2305.126761] ret_from_fork_asm+0x11/0x20 [ 2305.130694] [ 2305.132888] irq event stamp: 10605061 [ 2305.136553] hardirqs last enabled at (10605071): [] console_unlock+0x106/0x120 [ 2305.145421] hardirqs last disabled at (10605080): [] console_unlock+0xeb/0x120 [ 2305.154200] softirqs last enabled at (10605018): [] __do_softirq+0x2be/0x371 [ 2305.162892] softirqs last disabled at (10605013): [] irq_exit_rcu+0xb0/0xd0 [ 2305.171413] ---[ end trace 0000000000000000 ]--- [ 2305.176209] ------------[ cut here ]------------ [ 2305.180826] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2305.192471] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2305.201165] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2305.256645] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2305.265767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2305.273248] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2305.278761] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2305.284244] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2305.302992] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2305.308217] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2305.315350] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2305.322485] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007025 [ 2305.329616] R10: 0000000000258ee0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2305.336766] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2305.343900] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2305.351986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2305.357731] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2305.364881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2305.372016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2305.379163] Call Trace: [ 2305.381616] [ 2305.383714] ? __warn+0x85/0x180 [ 2305.386950] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.391829] ? report_bug+0xfc/0x1d0 [ 2305.395408] ? prb_read_valid+0x1b/0x20 [ 2305.399247] ? handle_bug+0x44/0x70 [ 2305.402758] ? exc_invalid_op+0x17/0x60 [ 2305.406596] ? asm_exc_invalid_op+0x1a/0x20 [ 2305.410787] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.415671] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.420548] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2305.425439] ceph_add_cap+0x20b/0x620 [ceph] [ 2305.429725] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2305.434462] ? lock_acquire+0xcb/0x2b0 [ 2305.438215] ceph_fill_trace+0x143/0xf30 [ceph] [ 2305.442769] ? __mutex_lock+0x97/0xce0 [ 2305.446522] ? __mutex_lock+0xaec/0xce0 [ 2305.450366] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2305.454912] ? lock_downgrade+0xfd/0x250 [ 2305.458843] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2305.463223] ceph_con_process_message+0x74/0x150 [libceph] [ 2305.468719] ? calc_signature+0xdb/0x100 [libceph] [ 2305.473547] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2305.478871] ? process_scheduled_works+0x22a/0x510 [ 2305.483665] ? process_scheduled_works+0x22a/0x510 [ 2305.488452] ceph_con_workfn+0x344/0x750 [libceph] [ 2305.493256] ? process_scheduled_works+0x22a/0x510 [ 2305.498047] process_scheduled_works+0x2a4/0x510 [ 2305.502673] worker_thread+0x188/0x340 [ 2305.506432] ? rescuer_thread+0x310/0x310 [ 2305.510441] kthread+0xf5/0x130 [ 2305.513587] ? kthread_complete_and_exit+0x20/0x20 [ 2305.518380] ret_from_fork+0x31/0x40 [ 2305.521960] ? kthread_complete_and_exit+0x20/0x20 [ 2305.526771] ret_from_fork_asm+0x11/0x20 [ 2305.530702] [ 2305.532887] irq event stamp: 10606181 [ 2305.536554] hardirqs last enabled at (10606191): [] console_unlock+0x106/0x120 [ 2305.545421] hardirqs last disabled at (10606200): [] console_unlock+0xeb/0x120 [ 2305.554201] softirqs last enabled at (10606128): [] __do_softirq+0x2be/0x371 [ 2305.562893] softirqs last disabled at (10606123): [] irq_exit_rcu+0xb0/0xd0 [ 2305.571411] ---[ end trace 0000000000000000 ]--- [ 2305.576088] ------------[ cut here ]------------ [ 2305.580704] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2305.592348] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2305.601043] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2305.656523] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2305.665646] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2305.673125] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2305.678620] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2305.684106] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2305.702852] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2305.708078] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2305.715211] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2305.722346] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007064 [ 2305.729477] R10: 000000000025a3f0 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2305.736610] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2305.743746] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2305.751830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2305.757575] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2305.764709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2305.771841] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2305.778974] Call Trace: [ 2305.781427] [ 2305.783532] ? __warn+0x85/0x180 [ 2305.786767] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.791646] ? report_bug+0xfc/0x1d0 [ 2305.795223] ? prb_read_valid+0x1b/0x20 [ 2305.799066] ? handle_bug+0x44/0x70 [ 2305.802555] ? exc_invalid_op+0x17/0x60 [ 2305.806396] ? asm_exc_invalid_op+0x1a/0x20 [ 2305.810588] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.815471] ? __list_add_valid_or_report+0x7d/0xa0 [ 2305.820349] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2305.825240] ceph_add_cap+0x20b/0x620 [ceph] [ 2305.829525] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2305.834248] ? lock_acquire+0xcb/0x2b0 [ 2305.837999] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2305.842449] ? __mutex_lock+0x97/0xce0 [ 2305.846203] ? __mutex_lock+0xaec/0xce0 [ 2305.850043] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2305.854584] ? lock_downgrade+0xfd/0x250 [ 2305.858515] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2305.862893] ceph_con_process_message+0x74/0x150 [libceph] [ 2305.868389] ? calc_signature+0xdb/0x100 [libceph] [ 2305.873194] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2305.878525] ? process_scheduled_works+0x22a/0x510 [ 2305.883317] ? process_scheduled_works+0x22a/0x510 [ 2305.888107] ceph_con_workfn+0x344/0x750 [libceph] [ 2305.892910] ? process_scheduled_works+0x22a/0x510 [ 2305.897700] process_scheduled_works+0x2a4/0x510 [ 2305.902327] worker_thread+0x188/0x340 [ 2305.906084] ? rescuer_thread+0x310/0x310 [ 2305.910094] kthread+0xf5/0x130 [ 2305.913243] ? kthread_complete_and_exit+0x20/0x20 [ 2305.918035] ret_from_fork+0x31/0x40 [ 2305.921612] ? kthread_complete_and_exit+0x20/0x20 [ 2305.926405] ret_from_fork_asm+0x11/0x20 [ 2305.930338] [ 2305.932523] irq event stamp: 10607295 [ 2305.936189] hardirqs last enabled at (10607305): [] console_unlock+0x106/0x120 [ 2305.945056] hardirqs last disabled at (10607314): [] console_unlock+0xeb/0x120 [ 2305.953836] softirqs last enabled at (10607242): [] __do_softirq+0x2be/0x371 [ 2305.962530] softirqs last disabled at (10607237): [] irq_exit_rcu+0xb0/0xd0 [ 2305.971048] ---[ end trace 0000000000000000 ]--- [ 2305.975669] ------------[ cut here ]------------ [ 2305.980286] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2305.991936] WARNING: CPU: 0 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2306.000887] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2306.056350] CPU: 0 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2306.065304] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2306.072779] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2306.078276] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2306.097027] RSP: 0018:ffffc90001fefb78 EFLAGS: 00010286 [ 2306.102255] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2306.109387] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2306.116520] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000070a3 [ 2306.123654] R10: 000000000025b720 R11: ffffc90001fefa08 R12: ffff8881c5350750 [ 2306.130784] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2306.137909] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2306.145995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2306.151741] CR2: 00007fb38ce0b000 CR3: 0000000108c72002 CR4: 00000000003706f0 [ 2306.158875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2306.166006] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2306.173130] Call Trace: [ 2306.175576] [ 2306.177682] ? __warn+0x85/0x180 [ 2306.180913] ? __list_add_valid_or_report+0x7d/0xa0 [ 2306.185795] ? report_bug+0xfc/0x1d0 [ 2306.189372] ? prb_read_valid+0x1b/0x20 [ 2306.193213] ? handle_bug+0x44/0x70 [ 2306.196704] ? exc_invalid_op+0x17/0x60 [ 2306.200536] ? asm_exc_invalid_op+0x1a/0x20 [ 2306.204727] ? __list_add_valid_or_report+0x7d/0xa0 [ 2306.209602] ? __list_add_valid_or_report+0x7d/0xa0 [ 2306.214481] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2306.219372] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2306.224615] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2306.229414] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2306.234130] ? __d_lookup+0xd6/0x200 [ 2306.237710] lookup_fast+0x6d/0xf0 [ 2306.241114] walk_component+0x26/0x160 [ 2306.244869] link_path_walk+0x24d/0x380 [ 2306.248706] ? path_openat+0x80/0x9d0 [ 2306.252375] path_openat+0x8e/0x9d0 [ 2306.255867] do_filp_open+0xb8/0x120 [ 2306.259447] ? lock_release+0xc5/0x260 [ 2306.263199] ? _raw_spin_unlock+0x23/0x40 [ 2306.267209] ? alloc_fd+0x13f/0x200 [ 2306.270705] do_sys_openat2+0x249/0x290 [ 2306.274544] do_sys_open+0x4f/0x80 [ 2306.277949] do_syscall_64+0x38/0x80 [ 2306.281526] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2306.286580] RIP: 0033:0x7f48f0eed2f2 [ 2306.290159] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2306.308905] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2306.316472] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2306.323604] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2306.330741] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2306.337870] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2306.345003] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2306.352143] [ 2306.354327] irq event stamp: 12890447 [ 2306.357987] hardirqs last enabled at (12890457): [] console_unlock+0x106/0x120 [ 2306.366851] hardirqs last disabled at (12890466): [] console_unlock+0xeb/0x120 [ 2306.375630] softirqs last enabled at (12890384): [] __do_softirq+0x2be/0x371 [ 2306.384326] softirqs last disabled at (12890485): [] irq_exit_rcu+0xb0/0xd0 [ 2306.392842] ---[ end trace 0000000000000000 ]--- [ 2306.397454] ------------[ cut here ]------------ [ 2306.402077] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2306.413720] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2306.422419] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2306.477892] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2306.487020] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2306.494497] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2306.499992] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2306.505477] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2306.524224] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2306.529451] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2306.536584] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2306.543716] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000070e2 [ 2306.550850] R10: 000000000025c980 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2306.557982] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2306.565115] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2306.573201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2306.578947] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2306.586082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2306.593213] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2306.600346] Call Trace: [ 2306.602798] [ 2306.604904] ? __warn+0x85/0x180 [ 2306.608138] ? __list_add_valid_or_report+0x7d/0xa0 [ 2306.613017] ? report_bug+0xfc/0x1d0 [ 2306.616596] ? prb_read_valid+0x1b/0x20 [ 2306.620438] ? handle_bug+0x44/0x70 [ 2306.623929] ? exc_invalid_op+0x17/0x60 [ 2306.627786] ? asm_exc_invalid_op+0x1a/0x20 [ 2306.631977] ? __list_add_valid_or_report+0x7d/0xa0 [ 2306.636851] ? __list_add_valid_or_report+0x7d/0xa0 [ 2306.641729] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2306.646621] ceph_add_cap+0x20b/0x620 [ceph] [ 2306.650907] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2306.655633] ceph_fill_trace+0x143/0xf30 [ceph] [ 2306.660174] ? __mutex_lock+0x97/0xce0 [ 2306.663925] ? __mutex_lock+0xaec/0xce0 [ 2306.667784] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2306.672323] ? lock_downgrade+0xfd/0x250 [ 2306.676255] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2306.680627] ceph_con_process_message+0x74/0x150 [libceph] [ 2306.686121] ? calc_signature+0xdb/0x100 [libceph] [ 2306.690926] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2306.696249] ? process_scheduled_works+0x22a/0x510 [ 2306.701039] ? process_scheduled_works+0x22a/0x510 [ 2306.705829] ceph_con_workfn+0x344/0x750 [libceph] [ 2306.710634] ? process_scheduled_works+0x22a/0x510 [ 2306.715423] process_scheduled_works+0x2a4/0x510 [ 2306.720050] worker_thread+0x188/0x340 [ 2306.723815] ? rescuer_thread+0x310/0x310 [ 2306.727828] kthread+0xf5/0x130 [ 2306.730974] ? kthread_complete_and_exit+0x20/0x20 [ 2306.735784] ret_from_fork+0x31/0x40 [ 2306.739363] ? kthread_complete_and_exit+0x20/0x20 [ 2306.744156] ret_from_fork_asm+0x11/0x20 [ 2306.748089] [ 2306.750281] irq event stamp: 10609231 [ 2306.753947] hardirqs last enabled at (10609241): [] console_unlock+0x106/0x120 [ 2306.762815] hardirqs last disabled at (10609250): [] console_unlock+0xeb/0x120 [ 2306.771593] softirqs last enabled at (10609208): [] __do_softirq+0x2be/0x371 [ 2306.780287] softirqs last disabled at (10609203): [] irq_exit_rcu+0xb0/0xd0 [ 2306.788806] ---[ end trace 0000000000000000 ]--- [ 2306.793458] ------------[ cut here ]------------ [ 2306.798080] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2306.809724] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2306.818412] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2306.873889] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2306.883014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2306.890494] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2306.895989] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2306.901473] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2306.920225] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2306.925449] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2306.932579] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2306.939714] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007120 [ 2306.946845] R10: 000000000025dbf0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2306.953980] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2306.961113] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2306.969199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2306.974942] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2306.982077] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2306.989209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2306.996341] Call Trace: [ 2306.998795] [ 2307.000901] ? __warn+0x85/0x180 [ 2307.004135] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.009013] ? report_bug+0xfc/0x1d0 [ 2307.012592] ? prb_read_valid+0x1b/0x20 [ 2307.016434] ? handle_bug+0x44/0x70 [ 2307.019925] ? exc_invalid_op+0x17/0x60 [ 2307.023786] ? asm_exc_invalid_op+0x1a/0x20 [ 2307.027972] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.032848] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.037727] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2307.042618] ceph_add_cap+0x20b/0x620 [ceph] [ 2307.046902] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2307.051624] ? lock_acquire+0xcb/0x2b0 [ 2307.055383] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2307.059845] ? __mutex_lock+0x97/0xce0 [ 2307.063595] ? __mutex_lock+0xaec/0xce0 [ 2307.067438] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2307.071977] ? lock_downgrade+0xfd/0x250 [ 2307.075909] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2307.080287] ceph_con_process_message+0x74/0x150 [libceph] [ 2307.085797] ? calc_signature+0xdb/0x100 [libceph] [ 2307.090605] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2307.095926] ? process_scheduled_works+0x22a/0x510 [ 2307.100720] ? process_scheduled_works+0x22a/0x510 [ 2307.105509] ceph_con_workfn+0x344/0x750 [libceph] [ 2307.110314] ? process_scheduled_works+0x22a/0x510 [ 2307.115104] process_scheduled_works+0x2a4/0x510 [ 2307.119728] worker_thread+0x188/0x340 [ 2307.123478] ? rescuer_thread+0x310/0x310 [ 2307.127490] kthread+0xf5/0x130 [ 2307.130637] ? kthread_complete_and_exit+0x20/0x20 [ 2307.135427] ret_from_fork+0x31/0x40 [ 2307.139007] ? kthread_complete_and_exit+0x20/0x20 [ 2307.143799] ret_from_fork_asm+0x11/0x20 [ 2307.147733] [ 2307.149917] irq event stamp: 10610363 [ 2307.153585] hardirqs last enabled at (10610373): [] console_unlock+0x106/0x120 [ 2307.162451] hardirqs last disabled at (10610382): [] console_unlock+0xeb/0x120 [ 2307.171230] softirqs last enabled at (10610310): [] __do_softirq+0x2be/0x371 [ 2307.179922] softirqs last disabled at (10610305): [] irq_exit_rcu+0xb0/0xd0 [ 2307.188441] ---[ end trace 0000000000000000 ]--- [ 2307.193083] ------------[ cut here ]------------ [ 2307.197699] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2307.209344] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2307.218041] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2307.273536] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2307.282660] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2307.290140] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2307.295633] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2307.301119] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2307.319865] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2307.325092] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2307.332225] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2307.339360] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000715f [ 2307.346489] R10: 000000000025ef08 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2307.353624] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2307.360756] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2307.368843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2307.374587] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2307.381720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2307.388854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2307.395987] Call Trace: [ 2307.398438] [ 2307.400545] ? __warn+0x85/0x180 [ 2307.403798] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.408676] ? report_bug+0xfc/0x1d0 [ 2307.412255] ? prb_read_valid+0x1b/0x20 [ 2307.416095] ? handle_bug+0x44/0x70 [ 2307.419587] ? exc_invalid_op+0x17/0x60 [ 2307.423426] ? asm_exc_invalid_op+0x1a/0x20 [ 2307.427616] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.432492] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.437371] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2307.442260] ceph_add_cap+0x20b/0x620 [ceph] [ 2307.446546] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2307.451267] ceph_fill_trace+0x143/0xf30 [ceph] [ 2307.455807] ? __mutex_lock+0x97/0xce0 [ 2307.459558] ? __mutex_lock+0xaec/0xce0 [ 2307.463397] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2307.467940] ? lock_downgrade+0xfd/0x250 [ 2307.471869] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2307.476248] ceph_con_process_message+0x74/0x150 [libceph] [ 2307.481746] ? calc_signature+0xdb/0x100 [libceph] [ 2307.486557] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2307.491879] ? process_scheduled_works+0x22a/0x510 [ 2307.496673] ? process_scheduled_works+0x22a/0x510 [ 2307.501462] ceph_con_workfn+0x344/0x750 [libceph] [ 2307.506265] ? process_scheduled_works+0x22a/0x510 [ 2307.511057] process_scheduled_works+0x2a4/0x510 [ 2307.515698] worker_thread+0x188/0x340 [ 2307.519445] ? rescuer_thread+0x310/0x310 [ 2307.523459] kthread+0xf5/0x130 [ 2307.526605] ? kthread_complete_and_exit+0x20/0x20 [ 2307.531400] ret_from_fork+0x31/0x40 [ 2307.534975] ? kthread_complete_and_exit+0x20/0x20 [ 2307.539771] ret_from_fork_asm+0x11/0x20 [ 2307.543727] [ 2307.545914] irq event stamp: 10611421 [ 2307.549579] hardirqs last enabled at (10611431): [] console_unlock+0x106/0x120 [ 2307.558446] hardirqs last disabled at (10611440): [] console_unlock+0xeb/0x120 [ 2307.567225] softirqs last enabled at (10611376): [] __do_softirq+0x2be/0x371 [ 2307.575919] softirqs last disabled at (10611371): [] irq_exit_rcu+0xb0/0xd0 [ 2307.584437] ---[ end trace 0000000000000000 ]--- [ 2307.589107] ------------[ cut here ]------------ [ 2307.593722] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2307.605367] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2307.614062] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2307.669558] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2307.678682] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2307.686161] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2307.691656] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2307.697142] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2307.715888] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2307.721113] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2307.728247] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2307.735380] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000719d [ 2307.742513] R10: 0000000000260120 R11: ffffc9000030b828 R12: ffff8881c5351118 [ 2307.749646] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2307.756800] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2307.764882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2307.770626] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2307.777760] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2307.784892] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2307.792025] Call Trace: [ 2307.794477] [ 2307.796576] ? __warn+0x85/0x180 [ 2307.799812] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.804687] ? report_bug+0xfc/0x1d0 [ 2307.808267] ? prb_read_valid+0x1b/0x20 [ 2307.812108] ? handle_bug+0x44/0x70 [ 2307.815598] ? exc_invalid_op+0x17/0x60 [ 2307.819446] ? asm_exc_invalid_op+0x1a/0x20 [ 2307.823640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.828524] ? __list_add_valid_or_report+0x7d/0xa0 [ 2307.833400] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2307.838293] ceph_add_cap+0x20b/0x620 [ceph] [ 2307.842577] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2307.847300] ? lock_acquire+0xcb/0x2b0 [ 2307.851052] ceph_fill_trace+0x143/0xf30 [ceph] [ 2307.855588] ? __mutex_lock+0x97/0xce0 [ 2307.859340] ? __mutex_lock+0xaec/0xce0 [ 2307.863184] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2307.867732] ? lock_downgrade+0xfd/0x250 [ 2307.871661] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2307.876033] ceph_con_process_message+0x74/0x150 [libceph] [ 2307.881528] ? calc_signature+0xdb/0x100 [libceph] [ 2307.886333] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2307.891664] ? process_scheduled_works+0x22a/0x510 [ 2307.896455] ? process_scheduled_works+0x22a/0x510 [ 2307.901245] ceph_con_workfn+0x344/0x750 [libceph] [ 2307.906050] ? process_scheduled_works+0x22a/0x510 [ 2307.910838] process_scheduled_works+0x2a4/0x510 [ 2307.915463] worker_thread+0x188/0x340 [ 2307.919211] ? rescuer_thread+0x310/0x310 [ 2307.923224] kthread+0xf5/0x130 [ 2307.926372] ? kthread_complete_and_exit+0x20/0x20 [ 2307.931164] ret_from_fork+0x31/0x40 [ 2307.934743] ? kthread_complete_and_exit+0x20/0x20 [ 2307.939541] ret_from_fork_asm+0x11/0x20 [ 2307.943469] [ 2307.945663] irq event stamp: 10612537 [ 2307.949327] hardirqs last enabled at (10612547): [] console_unlock+0x106/0x120 [ 2307.958195] hardirqs last disabled at (10612556): [] console_unlock+0xeb/0x120 [ 2307.966977] softirqs last enabled at (10612570): [] __do_softirq+0x2be/0x371 [ 2307.975667] softirqs last disabled at (10612565): [] irq_exit_rcu+0xb0/0xd0 [ 2307.984186] ---[ end trace 0000000000000000 ]--- [ 2307.988965] ------------[ cut here ]------------ [ 2307.993585] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2308.005228] WARNING: CPU: 0 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2308.014182] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2308.069657] CPU: 0 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2308.078613] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2308.086091] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2308.091576] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2308.110323] RSP: 0018:ffffc90001f7fb78 EFLAGS: 00010286 [ 2308.115548] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2308.122673] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2308.129805] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000071dc [ 2308.136930] R10: 00000000002613d0 R11: ffffc90001f7fa08 R12: ffff8881c5351118 [ 2308.144062] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2308.151194] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2308.159282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2308.165027] CR2: 00007fb38ce0b000 CR3: 0000000113532006 CR4: 00000000003706f0 [ 2308.172159] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2308.179292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2308.186416] Call Trace: [ 2308.188868] [ 2308.190968] ? __warn+0x85/0x180 [ 2308.194200] ? __list_add_valid_or_report+0x7d/0xa0 [ 2308.199080] ? report_bug+0xfc/0x1d0 [ 2308.202661] ? prb_read_valid+0x1b/0x20 [ 2308.206500] ? handle_bug+0x44/0x70 [ 2308.209991] ? exc_invalid_op+0x17/0x60 [ 2308.213830] ? asm_exc_invalid_op+0x1a/0x20 [ 2308.218020] ? __list_add_valid_or_report+0x7d/0xa0 [ 2308.222898] ? __list_add_valid_or_report+0x7d/0xa0 [ 2308.227777] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2308.232674] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2308.237909] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2308.242711] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2308.247426] ? __d_lookup+0xd6/0x200 [ 2308.251003] lookup_fast+0x6d/0xf0 [ 2308.254410] walk_component+0x26/0x160 [ 2308.258164] link_path_walk+0x24d/0x380 [ 2308.262001] ? path_openat+0x80/0x9d0 [ 2308.265670] path_openat+0x8e/0x9d0 [ 2308.269162] do_filp_open+0xb8/0x120 [ 2308.272745] ? lock_release+0xc5/0x260 [ 2308.276495] ? _raw_spin_unlock+0x23/0x40 [ 2308.280504] ? alloc_fd+0x13f/0x200 [ 2308.284000] do_sys_openat2+0x249/0x290 [ 2308.287839] do_sys_open+0x4f/0x80 [ 2308.291244] do_syscall_64+0x38/0x80 [ 2308.294821] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2308.299875] RIP: 0033:0x7f48f0eed2f2 [ 2308.303455] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2308.322200] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2308.329769] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2308.336898] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2308.344022] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2308.351149] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2308.358280] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2308.365418] [ 2308.367605] irq event stamp: 12951519 [ 2308.371262] hardirqs last enabled at (12951529): [] console_unlock+0x106/0x120 [ 2308.380128] hardirqs last disabled at (12951538): [] console_unlock+0xeb/0x120 [ 2308.388908] softirqs last enabled at (12951456): [] __do_softirq+0x2be/0x371 [ 2308.397604] softirqs last disabled at (12951557): [] irq_exit_rcu+0xb0/0xd0 [ 2308.406120] ---[ end trace 0000000000000000 ]--- [ 2308.410732] ------------[ cut here ]------------ [ 2308.415351] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2308.426998] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2308.435697] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2308.491171] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2308.500298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2308.507775] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2308.513295] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2308.518782] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2308.537530] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2308.542755] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2308.549887] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2308.557020] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000721b [ 2308.564155] R10: 0000000000262648 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2308.571285] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2308.578418] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2308.586504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2308.592251] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2308.599384] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2308.606518] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2308.613650] Call Trace: [ 2308.616102] [ 2308.618210] ? __warn+0x85/0x180 [ 2308.621441] ? __list_add_valid_or_report+0x7d/0xa0 [ 2308.626320] ? report_bug+0xfc/0x1d0 [ 2308.629901] ? prb_read_valid+0x1b/0x20 [ 2308.633742] ? handle_bug+0x44/0x70 [ 2308.637231] ? exc_invalid_op+0x17/0x60 [ 2308.641071] ? asm_exc_invalid_op+0x1a/0x20 [ 2308.645281] ? __list_add_valid_or_report+0x7d/0xa0 [ 2308.650164] ? __list_add_valid_or_report+0x7d/0xa0 [ 2308.655041] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2308.659934] ceph_add_cap+0x20b/0x620 [ceph] [ 2308.664217] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2308.668932] ? lock_acquire+0xcb/0x2b0 [ 2308.672684] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2308.677134] ? __mutex_lock+0x97/0xce0 [ 2308.680887] ? __mutex_lock+0xaec/0xce0 [ 2308.684728] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2308.689268] ? lock_downgrade+0xfd/0x250 [ 2308.693200] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2308.697579] ceph_con_process_message+0x74/0x150 [libceph] [ 2308.703074] ? calc_signature+0xdb/0x100 [libceph] [ 2308.707895] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2308.713216] ? process_scheduled_works+0x22a/0x510 [ 2308.718009] ? process_scheduled_works+0x22a/0x510 [ 2308.722819] ceph_con_workfn+0x344/0x750 [libceph] [ 2308.727629] ? process_scheduled_works+0x22a/0x510 [ 2308.732419] process_scheduled_works+0x2a4/0x510 [ 2308.737045] worker_thread+0x188/0x340 [ 2308.740820] ? rescuer_thread+0x310/0x310 [ 2308.744830] kthread+0xf5/0x130 [ 2308.747976] ? kthread_complete_and_exit+0x20/0x20 [ 2308.752771] ret_from_fork+0x31/0x40 [ 2308.756349] ? kthread_complete_and_exit+0x20/0x20 [ 2308.761143] ret_from_fork_asm+0x11/0x20 [ 2308.765076] [ 2308.767268] irq event stamp: 10615449 [ 2308.770935] hardirqs last enabled at (10615459): [] console_unlock+0x106/0x120 [ 2308.779819] hardirqs last disabled at (10615468): [] console_unlock+0xeb/0x120 [ 2308.788597] softirqs last enabled at (10615416): [] __do_softirq+0x2be/0x371 [ 2308.797291] softirqs last disabled at (10615411): [] irq_exit_rcu+0xb0/0xd0 [ 2308.805826] ---[ end trace 0000000000000000 ]--- [ 2308.810454] ------------[ cut here ]------------ [ 2308.815067] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2308.826711] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2308.835407] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2308.890884] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2308.900011] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2308.907490] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2308.912984] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2308.918469] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2308.937218] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2308.942443] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2308.949576] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2308.956710] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000725a [ 2308.963847] R10: 0000000000263870 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2308.970984] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2308.978114] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2308.986202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2308.991946] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2308.999081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2309.006231] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2309.013363] Call Trace: [ 2309.015832] [ 2309.017930] ? __warn+0x85/0x180 [ 2309.021166] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.026044] ? report_bug+0xfc/0x1d0 [ 2309.029621] ? prb_read_valid+0x1b/0x20 [ 2309.033463] ? handle_bug+0x44/0x70 [ 2309.036954] ? exc_invalid_op+0x17/0x60 [ 2309.040793] ? asm_exc_invalid_op+0x1a/0x20 [ 2309.044986] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.049860] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.054739] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2309.059629] ceph_add_cap+0x20b/0x620 [ceph] [ 2309.063915] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2309.068643] ceph_fill_trace+0x143/0xf30 [ceph] [ 2309.073182] ? __mutex_lock+0x97/0xce0 [ 2309.076934] ? __mutex_lock+0xaec/0xce0 [ 2309.080775] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2309.085317] ? lock_downgrade+0xfd/0x250 [ 2309.089246] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2309.093616] ceph_con_process_message+0x74/0x150 [libceph] [ 2309.099116] ? calc_signature+0xdb/0x100 [libceph] [ 2309.103926] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2309.109256] ? process_scheduled_works+0x22a/0x510 [ 2309.114048] ? process_scheduled_works+0x22a/0x510 [ 2309.118837] ceph_con_workfn+0x344/0x750 [libceph] [ 2309.123642] ? process_scheduled_works+0x22a/0x510 [ 2309.128431] process_scheduled_works+0x2a4/0x510 [ 2309.133057] worker_thread+0x188/0x340 [ 2309.136826] ? rescuer_thread+0x310/0x310 [ 2309.140835] kthread+0xf5/0x130 [ 2309.143982] ? kthread_complete_and_exit+0x20/0x20 [ 2309.148777] ret_from_fork+0x31/0x40 [ 2309.152354] ? kthread_complete_and_exit+0x20/0x20 [ 2309.157147] ret_from_fork_asm+0x11/0x20 [ 2309.161080] [ 2309.163283] irq event stamp: 10616521 [ 2309.166949] hardirqs last enabled at (10616531): [] console_unlock+0x106/0x120 [ 2309.175833] hardirqs last disabled at (10616540): [] console_unlock+0xeb/0x120 [ 2309.184611] softirqs last enabled at (10616478): [] __do_softirq+0x2be/0x371 [ 2309.193305] softirqs last disabled at (10616473): [] irq_exit_rcu+0xb0/0xd0 [ 2309.201832] ---[ end trace 0000000000000000 ]--- [ 2309.206486] ------------[ cut here ]------------ [ 2309.211117] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2309.222760] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2309.231457] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2309.286933] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2309.296057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2309.303537] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2309.309032] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2309.314517] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2309.333265] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2309.338491] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2309.345623] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2309.352758] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007298 [ 2309.359890] R10: 0000000000264a98 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2309.367021] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2309.374156] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2309.382241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2309.387986] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2309.395120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2309.402252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2309.409385] Call Trace: [ 2309.411838] [ 2309.413945] ? __warn+0x85/0x180 [ 2309.417178] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.422057] ? report_bug+0xfc/0x1d0 [ 2309.425636] ? prb_read_valid+0x1b/0x20 [ 2309.429478] ? handle_bug+0x44/0x70 [ 2309.432969] ? exc_invalid_op+0x17/0x60 [ 2309.436829] ? asm_exc_invalid_op+0x1a/0x20 [ 2309.441016] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.445892] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.450770] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2309.455660] ceph_add_cap+0x20b/0x620 [ceph] [ 2309.459946] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2309.464668] ? lock_acquire+0xcb/0x2b0 [ 2309.468421] ceph_fill_trace+0x143/0xf30 [ceph] [ 2309.472959] ? __mutex_lock+0x97/0xce0 [ 2309.476710] ? __mutex_lock+0xaec/0xce0 [ 2309.480551] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2309.485092] ? lock_downgrade+0xfd/0x250 [ 2309.489038] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2309.493417] ceph_con_process_message+0x74/0x150 [libceph] [ 2309.498915] ? calc_signature+0xdb/0x100 [libceph] [ 2309.503727] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2309.509060] ? process_scheduled_works+0x22a/0x510 [ 2309.513858] ? process_scheduled_works+0x22a/0x510 [ 2309.518649] ceph_con_workfn+0x344/0x750 [libceph] [ 2309.523453] ? process_scheduled_works+0x22a/0x510 [ 2309.528242] process_scheduled_works+0x2a4/0x510 [ 2309.532868] worker_thread+0x188/0x340 [ 2309.536624] ? rescuer_thread+0x310/0x310 [ 2309.540635] kthread+0xf5/0x130 [ 2309.543782] ? kthread_complete_and_exit+0x20/0x20 [ 2309.548576] ret_from_fork+0x31/0x40 [ 2309.552155] ? kthread_complete_and_exit+0x20/0x20 [ 2309.556947] ret_from_fork_asm+0x11/0x20 [ 2309.560880] [ 2309.563074] irq event stamp: 10617633 [ 2309.566740] hardirqs last enabled at (10617643): [] console_unlock+0x106/0x120 [ 2309.575607] hardirqs last disabled at (10617652): [] console_unlock+0xeb/0x120 [ 2309.584386] softirqs last enabled at (10617580): [] __do_softirq+0x2be/0x371 [ 2309.593079] softirqs last disabled at (10617575): [] irq_exit_rcu+0xb0/0xd0 [ 2309.601598] ---[ end trace 0000000000000000 ]--- [ 2309.606256] ------------[ cut here ]------------ [ 2309.610872] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2309.622516] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2309.631214] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2309.686692] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2309.695816] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2309.703296] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2309.708790] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2309.714275] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2309.733022] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2309.738248] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2309.745382] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2309.752514] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000072d7 [ 2309.759646] R10: 0000000000265df0 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2309.766779] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2309.773913] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2309.781998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2309.787743] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2309.794877] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2309.802011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2309.809142] Call Trace: [ 2309.811596] [ 2309.813701] ? __warn+0x85/0x180 [ 2309.816937] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.821822] ? report_bug+0xfc/0x1d0 [ 2309.825403] ? prb_read_valid+0x1b/0x20 [ 2309.829243] ? handle_bug+0x44/0x70 [ 2309.832735] ? exc_invalid_op+0x17/0x60 [ 2309.836574] ? asm_exc_invalid_op+0x1a/0x20 [ 2309.840766] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.845650] ? __list_add_valid_or_report+0x7d/0xa0 [ 2309.850528] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2309.855417] ceph_add_cap+0x20b/0x620 [ceph] [ 2309.859704] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2309.864427] ? lock_acquire+0xcb/0x2b0 [ 2309.868186] ceph_fill_trace+0x143/0xf30 [ceph] [ 2309.872724] ? __mutex_lock+0x97/0xce0 [ 2309.876476] ? __mutex_lock+0xaec/0xce0 [ 2309.880317] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2309.884856] ? lock_downgrade+0xfd/0x250 [ 2309.888787] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2309.893168] ceph_con_process_message+0x74/0x150 [libceph] [ 2309.898662] ? calc_signature+0xdb/0x100 [libceph] [ 2309.903467] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2309.908800] ? process_scheduled_works+0x22a/0x510 [ 2309.913599] ? process_scheduled_works+0x22a/0x510 [ 2309.918387] ceph_con_workfn+0x344/0x750 [libceph] [ 2309.923196] ? process_scheduled_works+0x22a/0x510 [ 2309.927992] process_scheduled_works+0x2a4/0x510 [ 2309.932617] worker_thread+0x188/0x340 [ 2309.936373] ? rescuer_thread+0x310/0x310 [ 2309.940386] kthread+0xf5/0x130 [ 2309.943533] ? kthread_complete_and_exit+0x20/0x20 [ 2309.948324] ret_from_fork+0x31/0x40 [ 2309.951902] ? kthread_complete_and_exit+0x20/0x20 [ 2309.956695] ret_from_fork_asm+0x11/0x20 [ 2309.960628] [ 2309.962819] irq event stamp: 10618751 [ 2309.966480] hardirqs last enabled at (10618761): [] console_unlock+0x106/0x120 [ 2309.975346] hardirqs last disabled at (10618770): [] console_unlock+0xeb/0x120 [ 2309.984126] softirqs last enabled at (10618698): [] __do_softirq+0x2be/0x371 [ 2309.992822] softirqs last disabled at (10618693): [] irq_exit_rcu+0xb0/0xd0 [ 2310.001339] ---[ end trace 0000000000000000 ]--- [ 2310.005959] ------------[ cut here ]------------ [ 2310.010578] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2310.022226] WARNING: CPU: 4 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2310.031183] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2310.086647] CPU: 4 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2310.095602] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2310.103077] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2310.108563] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2310.127311] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2310.132536] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2310.139675] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2310.146801] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007316 [ 2310.153941] R10: 0000000000267050 R11: ffffc90001957950 R12: ffff8881d12224a8 [ 2310.161071] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2310.168199] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2310.176287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2310.182031] CR2: 00007fb38ce40028 CR3: 000000016696c005 CR4: 00000000003706e0 [ 2310.189164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2310.196296] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2310.203429] Call Trace: [ 2310.205874] [ 2310.207972] ? __warn+0x85/0x180 [ 2310.211206] ? __list_add_valid_or_report+0x7d/0xa0 [ 2310.216085] ? report_bug+0xfc/0x1d0 [ 2310.219666] ? prb_read_valid+0x1b/0x20 [ 2310.223503] ? handle_bug+0x44/0x70 [ 2310.226994] ? exc_invalid_op+0x17/0x60 [ 2310.230828] ? asm_exc_invalid_op+0x1a/0x20 [ 2310.235017] ? __list_add_valid_or_report+0x7d/0xa0 [ 2310.239892] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2310.244783] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2310.250018] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2310.254817] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2310.259541] ? __d_lookup+0xd6/0x200 [ 2310.263121] lookup_fast+0x6d/0xf0 [ 2310.266526] walk_component+0x26/0x160 [ 2310.270279] link_path_walk+0x24d/0x380 [ 2310.274117] ? path_lookupat.isra.40+0x22/0x120 [ 2310.278652] path_lookupat.isra.40+0x6b/0x120 [ 2310.283013] filename_lookup+0xd4/0x190 [ 2310.286852] ? lock_acquire+0xcb/0x2b0 [ 2310.290602] ? __might_fault+0x45/0x80 [ 2310.294355] ? find_held_lock+0x2d/0x90 [ 2310.298193] ? __might_fault+0x45/0x80 [ 2310.301955] vfs_statx+0x89/0x150 [ 2310.305277] vfs_fstatat+0x5a/0xc0 [ 2310.308681] __do_sys_newstat+0x30/0x70 [ 2310.312521] ? lock_release+0xc5/0x260 [ 2310.316273] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2310.321151] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2310.326293] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2310.331432] do_syscall_64+0x38/0x80 [ 2310.335011] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2310.340063] RIP: 0033:0x7f48f0eeca79 [ 2310.343643] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2310.362390] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2310.369955] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2310.377093] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2310.384220] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2310.391352] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2310.398476] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2310.405608] [ 2310.407793] irq event stamp: 12839697 [ 2310.411450] hardirqs last enabled at (12839707): [] console_unlock+0x106/0x120 [ 2310.420317] hardirqs last disabled at (12839716): [] console_unlock+0xeb/0x120 [ 2310.429099] softirqs last enabled at (12839730): [] __do_softirq+0x2be/0x371 [ 2310.437790] softirqs last disabled at (12839725): [] irq_exit_rcu+0xb0/0xd0 [ 2310.446311] ---[ end trace 0000000000000000 ]--- [ 2310.450936] ------------[ cut here ]------------ [ 2310.455556] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2310.467203] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2310.475901] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2310.531377] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2310.540503] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2310.547982] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2310.553476] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2310.558961] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2310.577708] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2310.582933] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2310.590067] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2310.597202] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000735a [ 2310.604333] R10: 0000000000268390 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2310.611466] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2310.618598] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2310.626684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2310.632430] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2310.639564] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2310.646700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2310.653834] Call Trace: [ 2310.656281] [ 2310.658388] ? __warn+0x85/0x180 [ 2310.661622] ? __list_add_valid_or_report+0x7d/0xa0 [ 2310.666502] ? report_bug+0xfc/0x1d0 [ 2310.670080] ? prb_read_valid+0x1b/0x20 [ 2310.673921] ? handle_bug+0x44/0x70 [ 2310.677411] ? exc_invalid_op+0x17/0x60 [ 2310.681251] ? asm_exc_invalid_op+0x1a/0x20 [ 2310.685443] ? __list_add_valid_or_report+0x7d/0xa0 [ 2310.690327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2310.695205] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2310.700095] ceph_add_cap+0x20b/0x620 [ceph] [ 2310.704380] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2310.709103] ? lock_acquire+0xcb/0x2b0 [ 2310.712863] ceph_fill_trace+0x143/0xf30 [ceph] [ 2310.717402] ? __mutex_lock+0x97/0xce0 [ 2310.721153] ? __mutex_lock+0xaec/0xce0 [ 2310.724994] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2310.729533] ? lock_downgrade+0xfd/0x250 [ 2310.733464] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2310.737846] ceph_con_process_message+0x74/0x150 [libceph] [ 2310.743339] ? calc_signature+0xdb/0x100 [libceph] [ 2310.748143] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2310.753474] ? process_scheduled_works+0x22a/0x510 [ 2310.758266] ? process_scheduled_works+0x22a/0x510 [ 2310.763055] ceph_con_workfn+0x344/0x750 [libceph] [ 2310.767859] ? process_scheduled_works+0x22a/0x510 [ 2310.772650] process_scheduled_works+0x2a4/0x510 [ 2310.777276] worker_thread+0x188/0x340 [ 2310.781023] ? rescuer_thread+0x310/0x310 [ 2310.785036] kthread+0xf5/0x130 [ 2310.788183] ? kthread_complete_and_exit+0x20/0x20 [ 2310.792974] ret_from_fork+0x31/0x40 [ 2310.796554] ? kthread_complete_and_exit+0x20/0x20 [ 2310.801348] ret_from_fork_asm+0x11/0x20 [ 2310.805281] [ 2310.807475] irq event stamp: 10620803 [ 2310.811140] hardirqs last enabled at (10620813): [] console_unlock+0x106/0x120 [ 2310.820008] hardirqs last disabled at (10620822): [] console_unlock+0xeb/0x120 [ 2310.828785] softirqs last enabled at (10620750): [] __do_softirq+0x2be/0x371 [ 2310.837479] softirqs last disabled at (10620745): [] irq_exit_rcu+0xb0/0xd0 [ 2310.845999] ---[ end trace 0000000000000000 ]--- [ 2310.850619] ------------[ cut here ]------------ [ 2310.855237] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2310.866889] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2310.875849] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2310.931372] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2310.940331] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2310.947810] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2310.953296] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2310.972042] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2310.977269] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2310.984400] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2310.991534] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007399 [ 2310.998668] R10: 0000000000269550 R11: ffffc90001ff7950 R12: ffff8881c25d24a8 [ 2311.005800] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2311.012938] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2311.021032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2311.026782] CR2: 00007f9bd26f8000 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2311.033917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2311.041046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2311.048181] Call Trace: [ 2311.050634] [ 2311.052739] ? __warn+0x85/0x180 [ 2311.055973] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.060851] ? report_bug+0xfc/0x1d0 [ 2311.064430] ? prb_read_valid+0x1b/0x20 [ 2311.068276] ? handle_bug+0x44/0x70 [ 2311.071770] ? exc_invalid_op+0x17/0x60 [ 2311.075613] ? asm_exc_invalid_op+0x1a/0x20 [ 2311.079805] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.084687] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2311.089586] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2311.094842] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2311.099656] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2311.104378] ? __d_lookup+0xd6/0x200 [ 2311.107958] lookup_fast+0x6d/0xf0 [ 2311.111364] walk_component+0x26/0x160 [ 2311.115118] link_path_walk+0x24d/0x380 [ 2311.118964] ? path_lookupat.isra.40+0x22/0x120 [ 2311.123501] path_lookupat.isra.40+0x6b/0x120 [ 2311.127866] filename_lookup+0xd4/0x190 [ 2311.131714] ? lock_acquire+0xcb/0x2b0 [ 2311.135463] ? __might_fault+0x45/0x80 [ 2311.139218] ? find_held_lock+0x2d/0x90 [ 2311.143066] ? __might_fault+0x45/0x80 [ 2311.146824] vfs_statx+0x89/0x150 [ 2311.150152] vfs_fstatat+0x5a/0xc0 [ 2311.153562] __do_sys_newstat+0x30/0x70 [ 2311.157399] ? lock_release+0xc5/0x260 [ 2311.161158] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2311.166043] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2311.171191] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2311.176339] do_syscall_64+0x38/0x80 [ 2311.179916] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2311.184967] RIP: 0033:0x7f48f0eeca79 [ 2311.188547] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2311.207295] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2311.214861] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2311.221992] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2311.229125] RBP: 00007ffc53d60520 R08: 00000000000029f6 R09: 00000000020767c0 [ 2311.236259] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2311.243394] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2311.250537] [ 2311.252724] irq event stamp: 12964213 [ 2311.256390] hardirqs last enabled at (12964223): [] console_unlock+0x106/0x120 [ 2311.265257] hardirqs last disabled at (12964232): [] console_unlock+0xeb/0x120 [ 2311.274040] softirqs last enabled at (12964248): [] __do_softirq+0x2be/0x371 [ 2311.282741] softirqs last disabled at (12964259): [] irq_exit_rcu+0xb0/0xd0 [ 2311.291266] ---[ end trace 0000000000000000 ]--- [ 2311.295888] ------------[ cut here ]------------ [ 2311.300503] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2311.312151] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2311.320848] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2311.376332] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2311.385455] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2311.392935] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2311.398430] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2311.403915] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2311.422664] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2311.427887] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2311.435021] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2311.442157] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000073dd [ 2311.449288] R10: 000000000026ab48 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2311.456419] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2311.463553] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2311.471640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2311.477383] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2311.484518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2311.491652] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2311.498783] Call Trace: [ 2311.501236] [ 2311.503344] ? __warn+0x85/0x180 [ 2311.506577] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.511456] ? report_bug+0xfc/0x1d0 [ 2311.515034] ? prb_read_valid+0x1b/0x20 [ 2311.518875] ? handle_bug+0x44/0x70 [ 2311.522365] ? exc_invalid_op+0x17/0x60 [ 2311.526207] ? asm_exc_invalid_op+0x1a/0x20 [ 2311.530396] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.535272] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.540151] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2311.545041] ceph_add_cap+0x20b/0x620 [ceph] [ 2311.549326] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2311.554050] ? lock_acquire+0xcb/0x2b0 [ 2311.557809] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2311.562262] ? __mutex_lock+0x97/0xce0 [ 2311.566013] ? __mutex_lock+0xaec/0xce0 [ 2311.569855] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2311.574402] ? lock_downgrade+0xfd/0x250 [ 2311.578334] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2311.582713] ceph_con_process_message+0x74/0x150 [libceph] [ 2311.588207] ? calc_signature+0xdb/0x100 [libceph] [ 2311.593012] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2311.598344] ? process_scheduled_works+0x22a/0x510 [ 2311.603135] ? process_scheduled_works+0x22a/0x510 [ 2311.607923] ceph_con_workfn+0x344/0x750 [libceph] [ 2311.612729] ? process_scheduled_works+0x22a/0x510 [ 2311.617520] process_scheduled_works+0x2a4/0x510 [ 2311.622145] worker_thread+0x188/0x340 [ 2311.625901] ? rescuer_thread+0x310/0x310 [ 2311.629912] kthread+0xf5/0x130 [ 2311.633061] ? kthread_complete_and_exit+0x20/0x20 [ 2311.637854] ret_from_fork+0x31/0x40 [ 2311.641432] ? kthread_complete_and_exit+0x20/0x20 [ 2311.646225] ret_from_fork_asm+0x11/0x20 [ 2311.650156] [ 2311.652342] irq event stamp: 10622881 [ 2311.656007] hardirqs last enabled at (10622891): [] console_unlock+0x106/0x120 [ 2311.664875] hardirqs last disabled at (10622900): [] console_unlock+0xeb/0x120 [ 2311.673653] softirqs last enabled at (10622828): [] __do_softirq+0x2be/0x371 [ 2311.682347] softirqs last disabled at (10622823): [] irq_exit_rcu+0xb0/0xd0 [ 2311.690866] ---[ end trace 0000000000000000 ]--- [ 2311.695488] ------------[ cut here ]------------ [ 2311.700102] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2311.711742] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2311.720698] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2311.776167] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2311.785118] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2311.792598] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2311.798082] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2311.816829] RSP: 0018:ffffc90001fdfa78 EFLAGS: 00010286 [ 2311.822056] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2311.829188] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2311.836322] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000741c [ 2311.843455] R10: 000000000026bae0 R11: ffffc90001fdf908 R12: ffff8881d11624a8 [ 2311.850586] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2311.857719] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2311.865805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2311.871552] CR2: 00007fb38cdd10b0 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2311.878685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2311.885816] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2311.892948] Call Trace: [ 2311.895395] [ 2311.897501] ? __warn+0x85/0x180 [ 2311.900734] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.905612] ? report_bug+0xfc/0x1d0 [ 2311.909196] ? prb_read_valid+0x1b/0x20 [ 2311.913036] ? handle_bug+0x44/0x70 [ 2311.916524] ? exc_invalid_op+0x17/0x60 [ 2311.920364] ? asm_exc_invalid_op+0x1a/0x20 [ 2311.924555] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.929430] ? __list_add_valid_or_report+0x7d/0xa0 [ 2311.934307] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2311.939200] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2311.944443] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2311.949242] ceph_lookup+0xf9/0x3f0 [ceph] [ 2311.953354] __lookup_slow+0xff/0x1a0 [ 2311.957023] walk_component+0x83/0x160 [ 2311.960777] path_lookupat.isra.40+0x58/0x120 [ 2311.965137] filename_lookup+0xd4/0x190 [ 2311.968978] ? lock_acquire+0xcb/0x2b0 [ 2311.972730] ? __might_fault+0x45/0x80 [ 2311.976482] ? find_held_lock+0x2d/0x90 [ 2311.980320] ? __might_fault+0x45/0x80 [ 2311.984077] vfs_statx+0x89/0x150 [ 2311.987395] vfs_fstatat+0x5a/0xc0 [ 2311.990800] __do_sys_newstat+0x30/0x70 [ 2311.994638] ? lock_release+0xc5/0x260 [ 2311.998393] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2312.003271] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2312.008409] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2312.013550] do_syscall_64+0x38/0x80 [ 2312.017128] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2312.022180] RIP: 0033:0x7f48f0eeca79 [ 2312.025763] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2312.044509] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2312.052072] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2312.059204] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2312.066330] RBP: 00007ffc53d60520 R08: 00000000000029bc R09: 00000000020767c0 [ 2312.073462] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2312.080594] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2312.087727] [ 2312.089911] irq event stamp: 12866927 [ 2312.093570] hardirqs last enabled at (12866937): [] console_unlock+0x106/0x120 [ 2312.102436] hardirqs last disabled at (12866946): [] console_unlock+0xeb/0x120 [ 2312.111216] softirqs last enabled at (12866960): [] __do_softirq+0x2be/0x371 [ 2312.119909] softirqs last disabled at (12866955): [] irq_exit_rcu+0xb0/0xd0 [ 2312.128427] ---[ end trace 0000000000000000 ]--- [ 2312.133039] ------------[ cut here ]------------ [ 2312.137658] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2312.149304] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2312.158001] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2312.213478] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2312.222602] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2312.230082] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2312.235575] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2312.241063] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2312.259809] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2312.265033] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2312.272167] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2312.279303] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000745e [ 2312.286434] R10: 000000000026d060 R11: ffffc9000030b828 R12: ffff88819f028750 [ 2312.293566] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2312.300699] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2312.308786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2312.314529] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2312.321656] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2312.328789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2312.335920] Call Trace: [ 2312.338373] [ 2312.340480] ? __warn+0x85/0x180 [ 2312.343714] ? __list_add_valid_or_report+0x7d/0xa0 [ 2312.348592] ? report_bug+0xfc/0x1d0 [ 2312.352172] ? prb_read_valid+0x1b/0x20 [ 2312.356014] ? handle_bug+0x44/0x70 [ 2312.359503] ? exc_invalid_op+0x17/0x60 [ 2312.363342] ? asm_exc_invalid_op+0x1a/0x20 [ 2312.367533] ? __list_add_valid_or_report+0x7d/0xa0 [ 2312.372409] ? __list_add_valid_or_report+0x7d/0xa0 [ 2312.377287] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2312.382179] ceph_add_cap+0x20b/0x620 [ceph] [ 2312.386463] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2312.391182] ceph_fill_trace+0x143/0xf30 [ceph] [ 2312.395724] ? __mutex_lock+0x97/0xce0 [ 2312.399475] ? __mutex_lock+0xaec/0xce0 [ 2312.403315] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2312.407862] ? lock_downgrade+0xfd/0x250 [ 2312.411812] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2312.416182] ceph_con_process_message+0x74/0x150 [libceph] [ 2312.421680] ? calc_signature+0xdb/0x100 [libceph] [ 2312.426483] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2312.431805] ? process_scheduled_works+0x22a/0x510 [ 2312.436598] ? process_scheduled_works+0x22a/0x510 [ 2312.441386] ceph_con_workfn+0x344/0x750 [libceph] [ 2312.446190] ? process_scheduled_works+0x22a/0x510 [ 2312.450982] process_scheduled_works+0x2a4/0x510 [ 2312.455607] worker_thread+0x188/0x340 [ 2312.459363] ? rescuer_thread+0x310/0x310 [ 2312.463374] kthread+0xf5/0x130 [ 2312.466522] ? kthread_complete_and_exit+0x20/0x20 [ 2312.471316] ret_from_fork+0x31/0x40 [ 2312.474893] ? kthread_complete_and_exit+0x20/0x20 [ 2312.479687] ret_from_fork_asm+0x11/0x20 [ 2312.483618] [ 2312.485806] irq event stamp: 10624867 [ 2312.489472] hardirqs last enabled at (10624877): [] console_unlock+0x106/0x120 [ 2312.498338] hardirqs last disabled at (10624886): [] console_unlock+0xeb/0x120 [ 2312.507117] softirqs last enabled at (10624824): [] __do_softirq+0x2be/0x371 [ 2312.515810] softirqs last disabled at (10624819): [] irq_exit_rcu+0xb0/0xd0 [ 2312.524328] ---[ end trace 0000000000000000 ]--- [ 2312.529102] ------------[ cut here ]------------ [ 2312.533724] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2312.545369] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2312.554065] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2312.609543] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2312.618667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2312.626146] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2312.631658] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2312.637145] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2312.655908] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2312.661133] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2312.668268] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2312.675401] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000749c [ 2312.682534] R10: 000000000026e1a8 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2312.689666] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2312.696800] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2312.704895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2312.710638] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2312.717772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2312.724904] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2312.732037] Call Trace: [ 2312.734491] [ 2312.736598] ? __warn+0x85/0x180 [ 2312.739831] ? __list_add_valid_or_report+0x7d/0xa0 [ 2312.744710] ? report_bug+0xfc/0x1d0 [ 2312.748287] ? prb_read_valid+0x1b/0x20 [ 2312.752130] ? handle_bug+0x44/0x70 [ 2312.755620] ? exc_invalid_op+0x17/0x60 [ 2312.759460] ? asm_exc_invalid_op+0x1a/0x20 [ 2312.763651] ? __list_add_valid_or_report+0x7d/0xa0 [ 2312.768527] ? __list_add_valid_or_report+0x7d/0xa0 [ 2312.773405] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2312.778295] ceph_add_cap+0x20b/0x620 [ceph] [ 2312.782580] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2312.787295] ? lock_acquire+0xcb/0x2b0 [ 2312.791046] ceph_fill_trace+0x143/0xf30 [ceph] [ 2312.795584] ? __mutex_lock+0x97/0xce0 [ 2312.799337] ? __mutex_lock+0xaec/0xce0 [ 2312.803182] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2312.807726] ? lock_downgrade+0xfd/0x250 [ 2312.811656] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2312.816038] ceph_con_process_message+0x74/0x150 [libceph] [ 2312.821533] ? calc_signature+0xdb/0x100 [libceph] [ 2312.826344] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2312.831666] ? process_scheduled_works+0x22a/0x510 [ 2312.836458] ? process_scheduled_works+0x22a/0x510 [ 2312.841248] ceph_con_workfn+0x344/0x750 [libceph] [ 2312.846053] ? process_scheduled_works+0x22a/0x510 [ 2312.850843] process_scheduled_works+0x2a4/0x510 [ 2312.855466] worker_thread+0x188/0x340 [ 2312.859215] ? rescuer_thread+0x310/0x310 [ 2312.863229] kthread+0xf5/0x130 [ 2312.866374] ? kthread_complete_and_exit+0x20/0x20 [ 2312.871181] ret_from_fork+0x31/0x40 [ 2312.874756] ? kthread_complete_and_exit+0x20/0x20 [ 2312.879548] ret_from_fork_asm+0x11/0x20 [ 2312.883480] [ 2312.885665] irq event stamp: 10625975 [ 2312.889322] hardirqs last enabled at (10625985): [] console_unlock+0x106/0x120 [ 2312.898189] hardirqs last disabled at (10625994): [] console_unlock+0xeb/0x120 [ 2312.906969] softirqs last enabled at (10625922): [] __do_softirq+0x2be/0x371 [ 2312.915662] softirqs last disabled at (10625917): [] irq_exit_rcu+0xb0/0xd0 [ 2312.924181] ---[ end trace 0000000000000000 ]--- [ 2312.928833] ------------[ cut here ]------------ [ 2312.933448] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2312.945092] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2312.953789] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2313.009267] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2313.018392] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2313.025875] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2313.031372] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2313.036858] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2313.055605] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2313.060831] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2313.067967] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2313.075100] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000074db [ 2313.082231] R10: 000000000026f3e0 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2313.089362] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2313.096496] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2313.104582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2313.110328] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2313.117461] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2313.124593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2313.131727] Call Trace: [ 2313.134180] [ 2313.136285] ? __warn+0x85/0x180 [ 2313.139520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2313.144399] ? report_bug+0xfc/0x1d0 [ 2313.147977] ? prb_read_valid+0x1b/0x20 [ 2313.151817] ? handle_bug+0x44/0x70 [ 2313.155309] ? exc_invalid_op+0x17/0x60 [ 2313.159150] ? asm_exc_invalid_op+0x1a/0x20 [ 2313.163339] ? __list_add_valid_or_report+0x7d/0xa0 [ 2313.168216] ? __list_add_valid_or_report+0x7d/0xa0 [ 2313.173093] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2313.177983] ceph_add_cap+0x20b/0x620 [ceph] [ 2313.182268] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2313.186991] ? lock_acquire+0xcb/0x2b0 [ 2313.190745] ceph_fill_trace+0x143/0xf30 [ceph] [ 2313.195288] ? __mutex_lock+0x97/0xce0 [ 2313.199042] ? __mutex_lock+0xaec/0xce0 [ 2313.202883] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2313.207430] ? lock_downgrade+0xfd/0x250 [ 2313.211363] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2313.215742] ceph_con_process_message+0x74/0x150 [libceph] [ 2313.221236] ? calc_signature+0xdb/0x100 [libceph] [ 2313.226041] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2313.231372] ? process_scheduled_works+0x22a/0x510 [ 2313.236163] ? process_scheduled_works+0x22a/0x510 [ 2313.240953] ceph_con_workfn+0x344/0x750 [libceph] [ 2313.245759] ? process_scheduled_works+0x22a/0x510 [ 2313.250547] process_scheduled_works+0x2a4/0x510 [ 2313.255174] worker_thread+0x188/0x340 [ 2313.258930] ? rescuer_thread+0x310/0x310 [ 2313.262941] kthread+0xf5/0x130 [ 2313.266090] ? kthread_complete_and_exit+0x20/0x20 [ 2313.270883] ret_from_fork+0x31/0x40 [ 2313.274461] ? kthread_complete_and_exit+0x20/0x20 [ 2313.279254] ret_from_fork_asm+0x11/0x20 [ 2313.283187] [ 2313.285381] irq event stamp: 10627109 [ 2313.289046] hardirqs last enabled at (10627119): [] console_unlock+0x106/0x120 [ 2313.297914] hardirqs last disabled at (10627128): [] console_unlock+0xeb/0x120 [ 2313.306691] softirqs last enabled at (10627056): [] __do_softirq+0x2be/0x371 [ 2313.315385] softirqs last disabled at (10627051): [] irq_exit_rcu+0xb0/0xd0 [ 2313.323906] ---[ end trace 0000000000000000 ]--- [ 2313.328523] ------------[ cut here ]------------ [ 2313.333142] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2313.344791] WARNING: CPU: 4 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2313.353742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2313.409207] CPU: 4 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2313.418159] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2313.425636] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2313.431122] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2313.449868] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2313.455096] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2313.462226] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2313.469359] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000751a [ 2313.476492] R10: 00000000002705e8 R11: ffffc90001fc7950 R12: ffff8881d0bec200 [ 2313.483632] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2313.490769] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2313.498853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2313.504598] CR2: 00007fb38ce40028 CR3: 0000000113500002 CR4: 00000000003706e0 [ 2313.511731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2313.518864] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2313.525997] Call Trace: [ 2313.528448] [ 2313.530547] ? __warn+0x85/0x180 [ 2313.533780] ? __list_add_valid_or_report+0x7d/0xa0 [ 2313.538660] ? report_bug+0xfc/0x1d0 [ 2313.542238] ? prb_read_valid+0x1b/0x20 [ 2313.546078] ? handle_bug+0x44/0x70 [ 2313.549571] ? exc_invalid_op+0x17/0x60 [ 2313.553410] ? asm_exc_invalid_op+0x1a/0x20 [ 2313.557601] ? __list_add_valid_or_report+0x7d/0xa0 [ 2313.562477] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2313.567365] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2313.572600] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2313.577402] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2313.582115] ? __d_lookup+0xd6/0x200 [ 2313.585695] lookup_fast+0x6d/0xf0 [ 2313.589106] walk_component+0x26/0x160 [ 2313.592855] link_path_walk+0x24d/0x380 [ 2313.596693] ? path_lookupat.isra.40+0x22/0x120 [ 2313.601226] path_lookupat.isra.40+0x6b/0x120 [ 2313.605587] filename_lookup+0xd4/0x190 [ 2313.609428] ? lock_acquire+0xcb/0x2b0 [ 2313.613176] ? __might_fault+0x45/0x80 [ 2313.616930] ? find_held_lock+0x2d/0x90 [ 2313.620767] ? __might_fault+0x45/0x80 [ 2313.624524] vfs_statx+0x89/0x150 [ 2313.627844] vfs_fstatat+0x5a/0xc0 [ 2313.631247] __do_sys_newstat+0x30/0x70 [ 2313.635085] ? lock_release+0xc5/0x260 [ 2313.638831] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2313.643710] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2313.648848] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2313.653990] do_syscall_64+0x38/0x80 [ 2313.657567] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2313.662620] RIP: 0033:0x7f48f0eeca79 [ 2313.666199] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2313.684948] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2313.692511] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2313.699649] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2313.706779] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2313.713911] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2313.721043] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2313.728182] [ 2313.730371] irq event stamp: 12951779 [ 2313.734033] hardirqs last enabled at (12951789): [] console_unlock+0x106/0x120 [ 2313.742909] hardirqs last disabled at (12951808): [] console_unlock+0xeb/0x120 [ 2313.751690] softirqs last enabled at (12951822): [] __do_softirq+0x2be/0x371 [ 2313.760381] softirqs last disabled at (12951817): [] irq_exit_rcu+0xb0/0xd0 [ 2313.768900] ---[ end trace 0000000000000000 ]--- [ 2313.773512] ------------[ cut here ]------------ [ 2313.778131] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2313.789779] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2313.798475] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2313.853952] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2313.863077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2313.870556] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2313.876049] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2313.881536] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2313.900284] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2313.905510] RAX: 0000000000000000 RBX: ffff8881c53509c8 RCX: 0000000000000000 [ 2313.912641] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2313.919777] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000755e [ 2313.926908] R10: 0000000000271ab8 R11: ffffc9000030b828 R12: ffff8881c5351118 [ 2313.934039] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2313.941174] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2313.949261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2313.955003] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2313.962141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2313.969270] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2313.976405] Call Trace: [ 2313.978857] [ 2313.980962] ? __warn+0x85/0x180 [ 2313.984196] ? __list_add_valid_or_report+0x7d/0xa0 [ 2313.989075] ? report_bug+0xfc/0x1d0 [ 2313.992653] ? prb_read_valid+0x1b/0x20 [ 2313.996496] ? handle_bug+0x44/0x70 [ 2313.999986] ? exc_invalid_op+0x17/0x60 [ 2314.003827] ? asm_exc_invalid_op+0x1a/0x20 [ 2314.008016] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.012894] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.017769] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2314.022662] ceph_add_cap+0x20b/0x620 [ceph] [ 2314.026947] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2314.031671] ? lock_acquire+0xcb/0x2b0 [ 2314.035430] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2314.039882] ? __mutex_lock+0x97/0xce0 [ 2314.043631] ? __mutex_lock+0xaec/0xce0 [ 2314.047475] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2314.052023] ? lock_downgrade+0xfd/0x250 [ 2314.055954] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2314.060333] ceph_con_process_message+0x74/0x150 [libceph] [ 2314.065828] ? calc_signature+0xdb/0x100 [libceph] [ 2314.070633] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2314.075963] ? process_scheduled_works+0x22a/0x510 [ 2314.080755] ? process_scheduled_works+0x22a/0x510 [ 2314.085546] ceph_con_workfn+0x344/0x750 [libceph] [ 2314.090350] ? process_scheduled_works+0x22a/0x510 [ 2314.095138] process_scheduled_works+0x2a4/0x510 [ 2314.099763] worker_thread+0x188/0x340 [ 2314.103512] ? rescuer_thread+0x310/0x310 [ 2314.107525] kthread+0xf5/0x130 [ 2314.110672] ? kthread_complete_and_exit+0x20/0x20 [ 2314.115464] ret_from_fork+0x31/0x40 [ 2314.119042] ? kthread_complete_and_exit+0x20/0x20 [ 2314.123836] ret_from_fork_asm+0x11/0x20 [ 2314.127769] [ 2314.129963] irq event stamp: 10629981 [ 2314.133628] hardirqs last enabled at (10629991): [] console_unlock+0x106/0x120 [ 2314.142494] hardirqs last disabled at (10630000): [] console_unlock+0xeb/0x120 [ 2314.151275] softirqs last enabled at (10629928): [] __do_softirq+0x2be/0x371 [ 2314.159967] softirqs last disabled at (10629923): [] irq_exit_rcu+0xb0/0xd0 [ 2314.168486] ---[ end trace 0000000000000000 ]--- [ 2314.173109] ------------[ cut here ]------------ [ 2314.177724] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2314.189373] WARNING: CPU: 6 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2314.198326] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2314.253786] CPU: 6 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2314.262736] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2314.270216] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2314.275702] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2314.294449] RSP: 0018:ffffc90001fd7b78 EFLAGS: 00010286 [ 2314.299674] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2314.306798] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2314.313922] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000759d [ 2314.321049] R10: 0000000000272ba0 R11: ffffc90001fd7a08 R12: ffff8881d10b72e8 [ 2314.328179] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2314.335303] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2314.343382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2314.349129] CR2: 00007f9bd26f8000 CR3: 0000000120c1e006 CR4: 00000000003706e0 [ 2314.356260] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2314.363393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2314.370517] Call Trace: [ 2314.372963] [ 2314.375069] ? __warn+0x85/0x180 [ 2314.378302] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.383182] ? report_bug+0xfc/0x1d0 [ 2314.386762] ? prb_read_valid+0x1b/0x20 [ 2314.390601] ? handle_bug+0x44/0x70 [ 2314.394092] ? exc_invalid_op+0x17/0x60 [ 2314.397931] ? asm_exc_invalid_op+0x1a/0x20 [ 2314.402121] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.406998] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.411876] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2314.416767] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2314.422001] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2314.426803] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2314.431516] ? __d_lookup+0xd6/0x200 [ 2314.435095] lookup_fast+0x6d/0xf0 [ 2314.438501] walk_component+0x26/0x160 [ 2314.442255] link_path_walk+0x24d/0x380 [ 2314.446091] ? path_openat+0x80/0x9d0 [ 2314.449754] path_openat+0x8e/0x9d0 [ 2314.453246] do_filp_open+0xb8/0x120 [ 2314.456826] ? lock_release+0xc5/0x260 [ 2314.460577] ? _raw_spin_unlock+0x23/0x40 [ 2314.464586] ? alloc_fd+0x13f/0x200 [ 2314.468083] do_sys_openat2+0x249/0x290 [ 2314.471921] do_sys_open+0x4f/0x80 [ 2314.475326] do_syscall_64+0x38/0x80 [ 2314.478905] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2314.483958] RIP: 0033:0x7f48f0ef2805 [ 2314.487537] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2314.506284] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2314.513851] RAX: ffffffffffffffda RBX: 000000000207815f RCX: 00007f48f0ef2805 [ 2314.520982] RDX: 0000000000090800 RSI: 0000000002078140 RDI: 00000000ffffff9c [ 2314.528114] RBP: 0000000002078140 R08: 0000000002076674 R09: 0000000000000000 [ 2314.535243] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2314.542372] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2314.549510] [ 2314.551697] irq event stamp: 13003779 [ 2314.555354] hardirqs last enabled at (13003789): [] console_unlock+0x106/0x120 [ 2314.564220] hardirqs last disabled at (13003798): [] console_unlock+0xeb/0x120 [ 2314.572993] softirqs last enabled at (13003746): [] __do_softirq+0x2be/0x371 [ 2314.581686] softirqs last disabled at (13003817): [] irq_exit_rcu+0xb0/0xd0 [ 2314.590204] ---[ end trace 0000000000000000 ]--- [ 2314.594824] ------------[ cut here ]------------ [ 2314.599442] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2314.611090] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2314.619788] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2314.675262] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2314.684389] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2314.691868] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2314.697361] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2314.702846] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2314.721596] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2314.726821] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2314.733952] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2314.741087] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000075dc [ 2314.748219] R10: 0000000000273e08 R11: ffffc9000030b828 R12: ffff8881c5351ae0 [ 2314.755352] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2314.762486] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2314.770570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2314.776316] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2314.783449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2314.790582] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2314.797716] Call Trace: [ 2314.800167] [ 2314.802274] ? __warn+0x85/0x180 [ 2314.805508] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.810386] ? report_bug+0xfc/0x1d0 [ 2314.813968] ? prb_read_valid+0x1b/0x20 [ 2314.817807] ? handle_bug+0x44/0x70 [ 2314.821298] ? exc_invalid_op+0x17/0x60 [ 2314.825139] ? asm_exc_invalid_op+0x1a/0x20 [ 2314.829329] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.834214] ? __list_add_valid_or_report+0x7d/0xa0 [ 2314.839092] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2314.843982] ceph_add_cap+0x20b/0x620 [ceph] [ 2314.848266] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2314.852987] ceph_fill_trace+0x143/0xf30 [ceph] [ 2314.857527] ? __mutex_lock+0x97/0xce0 [ 2314.861277] ? __mutex_lock+0xaec/0xce0 [ 2314.865118] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2314.869657] ? lock_downgrade+0xfd/0x250 [ 2314.873589] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2314.877971] ceph_con_process_message+0x74/0x150 [libceph] [ 2314.883465] ? calc_signature+0xdb/0x100 [libceph] [ 2314.888268] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2314.893593] ? process_scheduled_works+0x22a/0x510 [ 2314.898391] ? process_scheduled_works+0x22a/0x510 [ 2314.903182] ceph_con_workfn+0x344/0x750 [libceph] [ 2314.908004] ? process_scheduled_works+0x22a/0x510 [ 2314.912793] process_scheduled_works+0x2a4/0x510 [ 2314.917427] worker_thread+0x188/0x340 [ 2314.921183] ? rescuer_thread+0x310/0x310 [ 2314.925211] kthread+0xf5/0x130 [ 2314.928358] ? kthread_complete_and_exit+0x20/0x20 [ 2314.933152] ret_from_fork+0x31/0x40 [ 2314.936730] ? kthread_complete_and_exit+0x20/0x20 [ 2314.941525] ret_from_fork_asm+0x11/0x20 [ 2314.945457] [ 2314.947650] irq event stamp: 10631943 [ 2314.951309] hardirqs last enabled at (10631953): [] console_unlock+0x106/0x120 [ 2314.960176] hardirqs last disabled at (10631962): [] console_unlock+0xeb/0x120 [ 2314.968958] softirqs last enabled at (10631920): [] __do_softirq+0x2be/0x371 [ 2314.977646] softirqs last disabled at (10631915): [] irq_exit_rcu+0xb0/0xd0 [ 2314.986168] ---[ end trace 0000000000000000 ]--- [ 2314.990959] ------------[ cut here ]------------ [ 2314.995584] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2315.007226] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2315.015939] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2315.071415] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2315.080539] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2315.088021] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2315.093523] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2315.099009] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2315.117759] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2315.122984] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2315.130114] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2315.137249] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000761a [ 2315.144380] R10: 0000000000275048 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2315.151511] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2315.158645] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2315.166737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2315.172489] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2315.179624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2315.186767] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2315.193935] Call Trace: [ 2315.196396] [ 2315.198512] ? __warn+0x85/0x180 [ 2315.201748] ? __list_add_valid_or_report+0x7d/0xa0 [ 2315.206627] ? report_bug+0xfc/0x1d0 [ 2315.210209] ? prb_read_valid+0x1b/0x20 [ 2315.214053] ? handle_bug+0x44/0x70 [ 2315.217556] ? exc_invalid_op+0x17/0x60 [ 2315.221400] ? asm_exc_invalid_op+0x1a/0x20 [ 2315.225594] ? __list_add_valid_or_report+0x7d/0xa0 [ 2315.230496] ? __list_add_valid_or_report+0x7d/0xa0 [ 2315.235375] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2315.240306] ceph_add_cap+0x20b/0x620 [ceph] [ 2315.244609] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2315.249347] ? lock_acquire+0xcb/0x2b0 [ 2315.253101] ceph_fill_trace+0x143/0xf30 [ceph] [ 2315.257640] ? __mutex_lock+0x97/0xce0 [ 2315.261407] ? __mutex_lock+0xaec/0xce0 [ 2315.265272] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2315.269827] ? lock_downgrade+0xfd/0x250 [ 2315.273790] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2315.278171] ceph_con_process_message+0x74/0x150 [libceph] [ 2315.283673] ? calc_signature+0xdb/0x100 [libceph] [ 2315.288510] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2315.293853] ? process_scheduled_works+0x22a/0x510 [ 2315.298668] ? process_scheduled_works+0x22a/0x510 [ 2315.303459] ceph_con_workfn+0x344/0x750 [libceph] [ 2315.308281] ? process_scheduled_works+0x22a/0x510 [ 2315.313089] process_scheduled_works+0x2a4/0x510 [ 2315.317730] worker_thread+0x188/0x340 [ 2315.321479] ? rescuer_thread+0x310/0x310 [ 2315.325508] kthread+0xf5/0x130 [ 2315.328691] ? kthread_complete_and_exit+0x20/0x20 [ 2315.333481] ret_from_fork+0x31/0x40 [ 2315.337079] ? kthread_complete_and_exit+0x20/0x20 [ 2315.341869] ret_from_fork_asm+0x11/0x20 [ 2315.345838] [ 2315.348034] irq event stamp: 10633095 [ 2315.351697] hardirqs last enabled at (10633105): [] console_unlock+0x106/0x120 [ 2315.360583] hardirqs last disabled at (10633114): [] console_unlock+0xeb/0x120 [ 2315.369380] softirqs last enabled at (10633042): [] __do_softirq+0x2be/0x371 [ 2315.378090] softirqs last disabled at (10633037): [] irq_exit_rcu+0xb0/0xd0 [ 2315.386608] ---[ end trace 0000000000000000 ]--- [ 2315.391302] ------------[ cut here ]------------ [ 2315.395922] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2315.407569] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2315.416259] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2315.471747] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2315.480870] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2315.488351] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2315.493854] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2315.499338] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2315.518084] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2315.523310] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2315.530444] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2315.537578] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007659 [ 2315.544709] R10: 00000000002762a8 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2315.551842] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2315.558975] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2315.567059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2315.572805] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2315.579939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2315.587072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2315.594206] Call Trace: [ 2315.596658] [ 2315.598765] ? __warn+0x85/0x180 [ 2315.601999] ? __list_add_valid_or_report+0x7d/0xa0 [ 2315.606877] ? report_bug+0xfc/0x1d0 [ 2315.610455] ? prb_read_valid+0x1b/0x20 [ 2315.614296] ? handle_bug+0x44/0x70 [ 2315.617788] ? exc_invalid_op+0x17/0x60 [ 2315.621626] ? asm_exc_invalid_op+0x1a/0x20 [ 2315.625819] ? __list_add_valid_or_report+0x7d/0xa0 [ 2315.630703] ? __list_add_valid_or_report+0x7d/0xa0 [ 2315.635580] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2315.640472] ceph_add_cap+0x20b/0x620 [ceph] [ 2315.644757] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2315.649480] ? lock_acquire+0xcb/0x2b0 [ 2315.653240] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2315.657691] ? __mutex_lock+0x97/0xce0 [ 2315.661442] ? __mutex_lock+0xaec/0xce0 [ 2315.665286] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2315.669832] ? lock_downgrade+0xfd/0x250 [ 2315.673763] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2315.678144] ceph_con_process_message+0x74/0x150 [libceph] [ 2315.683638] ? calc_signature+0xdb/0x100 [libceph] [ 2315.688442] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2315.693774] ? process_scheduled_works+0x22a/0x510 [ 2315.698566] ? process_scheduled_works+0x22a/0x510 [ 2315.703353] ceph_con_workfn+0x344/0x750 [libceph] [ 2315.708160] ? process_scheduled_works+0x22a/0x510 [ 2315.712948] process_scheduled_works+0x2a4/0x510 [ 2315.717576] worker_thread+0x188/0x340 [ 2315.721331] ? rescuer_thread+0x310/0x310 [ 2315.725344] kthread+0xf5/0x130 [ 2315.728491] ? kthread_complete_and_exit+0x20/0x20 [ 2315.733282] ret_from_fork+0x31/0x40 [ 2315.736862] ? kthread_complete_and_exit+0x20/0x20 [ 2315.741655] ret_from_fork_asm+0x11/0x20 [ 2315.745588] [ 2315.747781] irq event stamp: 10634211 [ 2315.751446] hardirqs last enabled at (10634221): [] console_unlock+0x106/0x120 [ 2315.760315] hardirqs last disabled at (10634230): [] console_unlock+0xeb/0x120 [ 2315.769094] softirqs last enabled at (10634158): [] __do_softirq+0x2be/0x371 [ 2315.777787] softirqs last disabled at (10634153): [] irq_exit_rcu+0xb0/0xd0 [ 2315.786306] ---[ end trace 0000000000000000 ]--- [ 2315.790928] ------------[ cut here ]------------ [ 2315.795543] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2315.807192] WARNING: CPU: 2 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2315.816145] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2315.871615] CPU: 2 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2315.880566] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2315.888043] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2315.893529] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2315.912268] RSP: 0018:ffffc90001fcfc10 EFLAGS: 00010286 [ 2315.917494] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2315.924625] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2315.931751] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007698 [ 2315.938890] R10: 0000000000277538 R11: ffffc90001fcfaa0 R12: ffff8881d106b838 [ 2315.946017] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2315.953151] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2315.961234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2315.966973] CR2: 00007f0d01c5d934 CR3: 000000011aeae005 CR4: 00000000003706e0 [ 2315.974105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2315.981228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2315.988354] Call Trace: [ 2315.990806] [ 2315.992904] ? __warn+0x85/0x180 [ 2315.996136] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.001018] ? report_bug+0xfc/0x1d0 [ 2316.004595] ? prb_read_valid+0x1b/0x20 [ 2316.008437] ? handle_bug+0x44/0x70 [ 2316.011929] ? exc_invalid_op+0x17/0x60 [ 2316.015766] ? asm_exc_invalid_op+0x1a/0x20 [ 2316.019957] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.024834] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2316.029723] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2316.034958] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2316.039760] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2316.044473] ? __d_lookup+0xd6/0x200 [ 2316.048054] lookup_fast+0x6d/0xf0 [ 2316.051457] path_openat+0xe4/0x9d0 [ 2316.054953] do_filp_open+0xb8/0x120 [ 2316.058533] ? lock_release+0xc5/0x260 [ 2316.062285] ? _raw_spin_unlock+0x23/0x40 [ 2316.066296] ? alloc_fd+0x13f/0x200 [ 2316.069789] do_sys_openat2+0x249/0x290 [ 2316.073630] do_sys_open+0x4f/0x80 [ 2316.077034] do_syscall_64+0x38/0x80 [ 2316.080615] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2316.085666] RIP: 0033:0x7f48f0ef2805 [ 2316.089244] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2316.107991] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2316.115557] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2316.122688] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2316.129816] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2316.136947] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2316.144078] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2316.151209] [ 2316.153397] irq event stamp: 12916157 [ 2316.157061] hardirqs last enabled at (12916167): [] console_unlock+0x106/0x120 [ 2316.165930] hardirqs last disabled at (12916176): [] console_unlock+0xeb/0x120 [ 2316.174708] softirqs last enabled at (12916114): [] __do_softirq+0x2be/0x371 [ 2316.183400] softirqs last disabled at (12916109): [] irq_exit_rcu+0xb0/0xd0 [ 2316.191920] ---[ end trace 0000000000000000 ]--- [ 2316.196541] ------------[ cut here ]------------ [ 2316.201162] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2316.212823] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2316.221518] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2316.277015] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2316.286139] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2316.293619] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2316.299130] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2316.304614] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2316.323372] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2316.328614] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2316.335763] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2316.342906] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000076d3 [ 2316.350057] R10: 00000000002785b0 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2316.357189] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2316.364321] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2316.372425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.378188] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2316.385321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2316.392472] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2316.399623] Call Trace: [ 2316.402091] [ 2316.404199] ? __warn+0x85/0x180 [ 2316.407432] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.412328] ? report_bug+0xfc/0x1d0 [ 2316.415924] ? prb_read_valid+0x1b/0x20 [ 2316.419780] ? handle_bug+0x44/0x70 [ 2316.423273] ? exc_invalid_op+0x17/0x60 [ 2316.427123] ? asm_exc_invalid_op+0x1a/0x20 [ 2316.431328] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.436205] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.441084] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2316.445975] ceph_add_cap+0x20b/0x620 [ceph] [ 2316.450287] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2316.455015] ceph_fill_trace+0x143/0xf30 [ceph] [ 2316.459554] ? __mutex_lock+0x97/0xce0 [ 2316.463322] ? __mutex_lock+0xaec/0xce0 [ 2316.467182] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2316.471732] ? lock_downgrade+0xfd/0x250 [ 2316.475695] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2316.480076] ceph_con_process_message+0x74/0x150 [libceph] [ 2316.485571] ? calc_signature+0xdb/0x100 [libceph] [ 2316.490391] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2316.495749] ? process_scheduled_works+0x22a/0x510 [ 2316.500540] ? process_scheduled_works+0x22a/0x510 [ 2316.505348] ceph_con_workfn+0x344/0x750 [libceph] [ 2316.510169] ? process_scheduled_works+0x22a/0x510 [ 2316.514963] process_scheduled_works+0x2a4/0x510 [ 2316.519601] worker_thread+0x188/0x340 [ 2316.523351] ? rescuer_thread+0x310/0x310 [ 2316.527379] kthread+0xf5/0x130 [ 2316.530541] ? kthread_complete_and_exit+0x20/0x20 [ 2316.535356] ret_from_fork+0x31/0x40 [ 2316.538931] ? kthread_complete_and_exit+0x20/0x20 [ 2316.543741] ret_from_fork_asm+0x11/0x20 [ 2316.547692] [ 2316.549878] irq event stamp: 10636213 [ 2316.553553] hardirqs last enabled at (10636223): [] console_unlock+0x106/0x120 [ 2316.562437] hardirqs last disabled at (10636232): [] console_unlock+0xeb/0x120 [ 2316.571246] softirqs last enabled at (10636168): [] __do_softirq+0x2be/0x371 [ 2316.579966] softirqs last disabled at (10636163): [] irq_exit_rcu+0xb0/0xd0 [ 2316.588489] ---[ end trace 0000000000000000 ]--- [ 2316.593151] ------------[ cut here ]------------ [ 2316.597772] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2316.609450] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2316.618149] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2316.673626] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2316.682767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2316.690263] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2316.695776] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2316.701277] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2316.720042] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2316.725270] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2316.732420] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2316.739570] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007711 [ 2316.746720] R10: 0000000000279760 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2316.753870] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2316.761020] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2316.769105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.774852] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2316.782003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2316.789135] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2316.796268] Call Trace: [ 2316.798738] [ 2316.800861] ? __warn+0x85/0x180 [ 2316.804112] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.808990] ? report_bug+0xfc/0x1d0 [ 2316.812569] ? prb_read_valid+0x1b/0x20 [ 2316.816427] ? handle_bug+0x44/0x70 [ 2316.819936] ? exc_invalid_op+0x17/0x60 [ 2316.823792] ? asm_exc_invalid_op+0x1a/0x20 [ 2316.827986] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.832868] ? __list_add_valid_or_report+0x7d/0xa0 [ 2316.837763] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2316.842671] ceph_add_cap+0x20b/0x620 [ceph] [ 2316.846978] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2316.851715] ? lock_acquire+0xcb/0x2b0 [ 2316.855483] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2316.859936] ? __mutex_lock+0x97/0xce0 [ 2316.863710] ? __mutex_lock+0xaec/0xce0 [ 2316.867564] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2316.872112] ? lock_downgrade+0xfd/0x250 [ 2316.876041] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2316.880439] ceph_con_process_message+0x74/0x150 [libceph] [ 2316.885935] ? calc_signature+0xdb/0x100 [libceph] [ 2316.890765] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2316.896088] ? process_scheduled_works+0x22a/0x510 [ 2316.900904] ? process_scheduled_works+0x22a/0x510 [ 2316.905695] ceph_con_workfn+0x344/0x750 [libceph] [ 2316.910515] ? process_scheduled_works+0x22a/0x510 [ 2316.915322] process_scheduled_works+0x2a4/0x510 [ 2316.919966] worker_thread+0x188/0x340 [ 2316.923724] ? rescuer_thread+0x310/0x310 [ 2316.927737] kthread+0xf5/0x130 [ 2316.930899] ? kthread_complete_and_exit+0x20/0x20 [ 2316.935709] ret_from_fork+0x31/0x40 [ 2316.939305] ? kthread_complete_and_exit+0x20/0x20 [ 2316.944115] ret_from_fork_asm+0x11/0x20 [ 2316.948049] [ 2316.950250] irq event stamp: 10637331 [ 2316.953925] hardirqs last enabled at (10637341): [] console_unlock+0x106/0x120 [ 2316.962816] hardirqs last disabled at (10637362): [] console_unlock+0xeb/0x120 [ 2316.971599] softirqs last enabled at (10637360): [] __do_softirq+0x2be/0x371 [ 2316.980290] softirqs last disabled at (10637351): [] irq_exit_rcu+0xb0/0xd0 [ 2316.988828] ---[ end trace 0000000000000000 ]--- [ 2316.993625] ------------[ cut here ]------------ [ 2316.998254] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2317.009907] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2317.018597] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2317.074093] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2317.083218] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2317.090698] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2317.096208] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2317.101693] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2317.120451] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2317.125693] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2317.132835] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2317.139988] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007750 [ 2317.147118] R10: 000000000027aa10 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2317.154253] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2317.161423] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2317.169506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2317.175268] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2317.182417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2317.189558] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2317.196700] Call Trace: [ 2317.199161] [ 2317.201269] ? __warn+0x85/0x180 [ 2317.204520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2317.209417] ? report_bug+0xfc/0x1d0 [ 2317.213012] ? prb_read_valid+0x1b/0x20 [ 2317.216853] ? handle_bug+0x44/0x70 [ 2317.220362] ? exc_invalid_op+0x17/0x60 [ 2317.224219] ? asm_exc_invalid_op+0x1a/0x20 [ 2317.228425] ? __list_add_valid_or_report+0x7d/0xa0 [ 2317.233301] ? __list_add_valid_or_report+0x7d/0xa0 [ 2317.238197] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2317.243087] ceph_add_cap+0x20b/0x620 [ceph] [ 2317.247389] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2317.252112] ceph_fill_trace+0x143/0xf30 [ceph] [ 2317.256650] ? __mutex_lock+0x97/0xce0 [ 2317.260418] ? __mutex_lock+0xaec/0xce0 [ 2317.264277] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2317.268836] ? lock_downgrade+0xfd/0x250 [ 2317.272799] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2317.277172] ceph_con_process_message+0x74/0x150 [libceph] [ 2317.282667] ? calc_signature+0xdb/0x100 [libceph] [ 2317.287504] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2317.292845] ? process_scheduled_works+0x22a/0x510 [ 2317.297638] ? process_scheduled_works+0x22a/0x510 [ 2317.302445] ceph_con_workfn+0x344/0x750 [libceph] [ 2317.307265] ? process_scheduled_works+0x22a/0x510 [ 2317.312056] process_scheduled_works+0x2a4/0x510 [ 2317.316698] worker_thread+0x188/0x340 [ 2317.320447] ? rescuer_thread+0x310/0x310 [ 2317.324475] kthread+0xf5/0x130 [ 2317.327639] ? kthread_complete_and_exit+0x20/0x20 [ 2317.332467] ret_from_fork+0x31/0x40 [ 2317.336045] ? kthread_complete_and_exit+0x20/0x20 [ 2317.340838] ret_from_fork_asm+0x11/0x20 [ 2317.344807] [ 2317.346992] irq event stamp: 10638421 [ 2317.350657] hardirqs last enabled at (10638431): [] console_unlock+0x106/0x120 [ 2317.359542] hardirqs last disabled at (10638440): [] console_unlock+0xeb/0x120 [ 2317.368337] softirqs last enabled at (10638378): [] __do_softirq+0x2be/0x371 [ 2317.377049] softirqs last disabled at (10638373): [] irq_exit_rcu+0xb0/0xd0 [ 2317.385568] ---[ end trace 0000000000000000 ]--- [ 2317.390222] ------------[ cut here ]------------ [ 2317.394843] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2317.406522] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2317.415219] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2317.470695] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2317.479839] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2317.487334] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2317.492847] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2317.498349] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2317.517118] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2317.522339] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2317.529480] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2317.536631] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000778e [ 2317.543782] R10: 000000000027bbc0 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2317.550932] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2317.558083] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2317.566168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2317.571912] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2317.579056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2317.586189] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2317.593320] Call Trace: [ 2317.595781] [ 2317.597896] ? __warn+0x85/0x180 [ 2317.601149] ? __list_add_valid_or_report+0x7d/0xa0 [ 2317.606028] ? report_bug+0xfc/0x1d0 [ 2317.609604] ? prb_read_valid+0x1b/0x20 [ 2317.613463] ? handle_bug+0x44/0x70 [ 2317.616986] ? exc_invalid_op+0x17/0x60 [ 2317.620822] ? asm_exc_invalid_op+0x1a/0x20 [ 2317.625029] ? __list_add_valid_or_report+0x7d/0xa0 [ 2317.629904] ? __list_add_valid_or_report+0x7d/0xa0 [ 2317.634801] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2317.639709] ceph_add_cap+0x20b/0x620 [ceph] [ 2317.644011] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2317.648751] ? lock_acquire+0xcb/0x2b0 [ 2317.652528] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2317.656987] ? __mutex_lock+0x97/0xce0 [ 2317.660739] ? __mutex_lock+0xaec/0xce0 [ 2317.664599] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2317.669157] ? lock_downgrade+0xfd/0x250 [ 2317.673087] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2317.677482] ceph_con_process_message+0x74/0x150 [libceph] [ 2317.682987] ? calc_signature+0xdb/0x100 [libceph] [ 2317.687794] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2317.693122] ? process_scheduled_works+0x22a/0x510 [ 2317.697932] ? process_scheduled_works+0x22a/0x510 [ 2317.702723] ceph_con_workfn+0x344/0x750 [libceph] [ 2317.707544] ? process_scheduled_works+0x22a/0x510 [ 2317.712350] process_scheduled_works+0x2a4/0x510 [ 2317.716993] worker_thread+0x188/0x340 [ 2317.720749] ? rescuer_thread+0x310/0x310 [ 2317.724778] kthread+0xf5/0x130 [ 2317.727945] ? kthread_complete_and_exit+0x20/0x20 [ 2317.732762] ret_from_fork+0x31/0x40 [ 2317.736341] ? kthread_complete_and_exit+0x20/0x20 [ 2317.741152] ret_from_fork_asm+0x11/0x20 [ 2317.745084] [ 2317.747269] irq event stamp: 10639537 [ 2317.750943] hardirqs last enabled at (10639547): [] console_unlock+0x106/0x120 [ 2317.759827] hardirqs last disabled at (10639558): [] console_unlock+0xeb/0x120 [ 2317.768607] softirqs last enabled at (10639482): [] __do_softirq+0x2be/0x371 [ 2317.777318] softirqs last disabled at (10639477): [] irq_exit_rcu+0xb0/0xd0 [ 2317.785855] ---[ end trace 0000000000000000 ]--- [ 2317.790503] ------------[ cut here ]------------ [ 2317.795119] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2317.806778] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2317.815469] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2317.870984] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2317.880108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2317.887587] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2317.893098] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2317.898583] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2317.917347] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2317.922592] RAX: 0000000000000000 RBX: ffff8881c3b761d0 RCX: 0000000000000000 [ 2317.929742] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2317.936894] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000077cd [ 2317.944042] R10: 000000000027ce80 R11: ffffc9000030b828 R12: ffff8881c3b76920 [ 2317.951173] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2317.958306] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2317.966411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2317.972174] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2317.979306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2317.986457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2317.993606] Call Trace: [ 2317.996067] [ 2317.998174] ? __warn+0x85/0x180 [ 2318.001408] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.006304] ? report_bug+0xfc/0x1d0 [ 2318.009900] ? prb_read_valid+0x1b/0x20 [ 2318.013759] ? handle_bug+0x44/0x70 [ 2318.017269] ? exc_invalid_op+0x17/0x60 [ 2318.021125] ? asm_exc_invalid_op+0x1a/0x20 [ 2318.025332] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.030221] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.035102] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2318.040006] ceph_add_cap+0x20b/0x620 [ceph] [ 2318.044315] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2318.049043] ceph_fill_trace+0x143/0xf30 [ceph] [ 2318.053581] ? __mutex_lock+0x97/0xce0 [ 2318.057352] ? __mutex_lock+0xaec/0xce0 [ 2318.061208] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2318.065750] ? lock_downgrade+0xfd/0x250 [ 2318.069714] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2318.074096] ceph_con_process_message+0x74/0x150 [libceph] [ 2318.079590] ? calc_signature+0xdb/0x100 [libceph] [ 2318.084429] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2318.089776] ? process_scheduled_works+0x22a/0x510 [ 2318.094587] ? process_scheduled_works+0x22a/0x510 [ 2318.099376] ceph_con_workfn+0x344/0x750 [libceph] [ 2318.104197] ? process_scheduled_works+0x22a/0x510 [ 2318.108996] process_scheduled_works+0x2a4/0x510 [ 2318.113639] worker_thread+0x188/0x340 [ 2318.117388] ? rescuer_thread+0x310/0x310 [ 2318.121416] kthread+0xf5/0x130 [ 2318.124579] ? kthread_complete_and_exit+0x20/0x20 [ 2318.129393] ret_from_fork+0x31/0x40 [ 2318.132992] ? kthread_complete_and_exit+0x20/0x20 [ 2318.137781] ret_from_fork_asm+0x11/0x20 [ 2318.141747] [ 2318.143933] irq event stamp: 10640661 [ 2318.147617] hardirqs last enabled at (10640671): [] console_unlock+0x106/0x120 [ 2318.156501] hardirqs last disabled at (10640680): [] console_unlock+0xeb/0x120 [ 2318.165297] softirqs last enabled at (10640618): [] __do_softirq+0x2be/0x371 [ 2318.174008] softirqs last disabled at (10640613): [] irq_exit_rcu+0xb0/0xd0 [ 2318.182527] ---[ end trace 0000000000000000 ]--- [ 2318.187184] ------------[ cut here ]------------ [ 2318.191800] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2318.203481] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2318.212177] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2318.267654] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2318.276797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2318.284293] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2318.289803] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2318.295307] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2318.314072] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2318.319298] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2318.326447] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2318.333592] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000780b [ 2318.340740] R10: 000000000027e088 R11: ffffc9000030b828 R12: ffff88819f028750 [ 2318.347890] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2318.355038] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2318.363127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2318.368871] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2318.376023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2318.383155] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2318.390288] Call Trace: [ 2318.392756] [ 2318.394872] ? __warn+0x85/0x180 [ 2318.398123] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.403003] ? report_bug+0xfc/0x1d0 [ 2318.406581] ? prb_read_valid+0x1b/0x20 [ 2318.410439] ? handle_bug+0x44/0x70 [ 2318.413947] ? exc_invalid_op+0x17/0x60 [ 2318.417795] ? asm_exc_invalid_op+0x1a/0x20 [ 2318.422006] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.426889] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.431783] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2318.436692] ceph_add_cap+0x20b/0x620 [ceph] [ 2318.441004] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2318.445743] ? lock_acquire+0xcb/0x2b0 [ 2318.449513] ceph_fill_trace+0x143/0xf30 [ceph] [ 2318.454049] ? __mutex_lock+0x97/0xce0 [ 2318.457803] ? __mutex_lock+0xaec/0xce0 [ 2318.461644] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2318.466210] ? lock_downgrade+0xfd/0x250 [ 2318.470139] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2318.474535] ceph_con_process_message+0x74/0x150 [libceph] [ 2318.480032] ? calc_signature+0xdb/0x100 [libceph] [ 2318.484836] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2318.490176] ? process_scheduled_works+0x22a/0x510 [ 2318.494991] ? process_scheduled_works+0x22a/0x510 [ 2318.499784] ceph_con_workfn+0x344/0x750 [libceph] [ 2318.504589] ? process_scheduled_works+0x22a/0x510 [ 2318.509396] process_scheduled_works+0x2a4/0x510 [ 2318.514039] worker_thread+0x188/0x340 [ 2318.517794] ? rescuer_thread+0x310/0x310 [ 2318.521823] kthread+0xf5/0x130 [ 2318.524999] ? kthread_complete_and_exit+0x20/0x20 [ 2318.529789] ret_from_fork+0x31/0x40 [ 2318.533385] ? kthread_complete_and_exit+0x20/0x20 [ 2318.538195] ret_from_fork_asm+0x11/0x20 [ 2318.542129] [ 2318.544321] irq event stamp: 10641777 [ 2318.548000] hardirqs last enabled at (10641787): [] console_unlock+0x106/0x120 [ 2318.556864] hardirqs last disabled at (10641796): [] console_unlock+0xeb/0x120 [ 2318.565660] softirqs last enabled at (10641724): [] __do_softirq+0x2be/0x371 [ 2318.574371] softirqs last disabled at (10641719): [] irq_exit_rcu+0xb0/0xd0 [ 2318.582908] ---[ end trace 0000000000000000 ]--- [ 2318.587605] ------------[ cut here ]------------ [ 2318.592231] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2318.603880] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2318.612833] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2318.668295] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2318.677244] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2318.684716] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2318.690202] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2318.708947] RSP: 0018:ffffc90001fefb78 EFLAGS: 00010286 [ 2318.714174] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2318.721306] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2318.728432] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000784a [ 2318.735564] R10: 000000000027f2f8 R11: ffffc90001fefa08 R12: ffff88819f028750 [ 2318.742697] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2318.749829] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2318.757915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2318.763662] CR2: 00007f0d01c5d934 CR3: 0000000108c72006 CR4: 00000000003706e0 [ 2318.770794] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2318.777919] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2318.785051] Call Trace: [ 2318.787505] [ 2318.789611] ? __warn+0x85/0x180 [ 2318.792844] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.797725] ? report_bug+0xfc/0x1d0 [ 2318.801303] ? prb_read_valid+0x1b/0x20 [ 2318.805142] ? handle_bug+0x44/0x70 [ 2318.808635] ? exc_invalid_op+0x17/0x60 [ 2318.812473] ? asm_exc_invalid_op+0x1a/0x20 [ 2318.816663] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.821539] ? __list_add_valid_or_report+0x7d/0xa0 [ 2318.826417] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2318.831310] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2318.836552] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2318.841354] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2318.846067] ? __d_lookup+0xd6/0x200 [ 2318.849646] lookup_fast+0x6d/0xf0 [ 2318.853053] walk_component+0x26/0x160 [ 2318.856806] link_path_walk+0x24d/0x380 [ 2318.860643] ? path_openat+0x80/0x9d0 [ 2318.864314] path_openat+0x8e/0x9d0 [ 2318.867806] do_filp_open+0xb8/0x120 [ 2318.871384] ? lock_release+0xc5/0x260 [ 2318.875136] ? _raw_spin_unlock+0x23/0x40 [ 2318.879147] ? alloc_fd+0x13f/0x200 [ 2318.882641] do_sys_openat2+0x249/0x290 [ 2318.886480] do_sys_open+0x4f/0x80 [ 2318.889887] do_syscall_64+0x38/0x80 [ 2318.893464] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2318.898516] RIP: 0033:0x7f48f0eed2f2 [ 2318.902090] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2318.920834] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2318.928404] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2318.935533] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2318.942656] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2318.949787] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2318.956918] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2318.964054] [ 2318.966239] irq event stamp: 12891579 [ 2318.969898] hardirqs last enabled at (12891589): [] console_unlock+0x106/0x120 [ 2318.978762] hardirqs last disabled at (12891598): [] console_unlock+0xeb/0x120 [ 2318.987535] softirqs last enabled at (12891556): [] __do_softirq+0x2be/0x371 [ 2318.996229] softirqs last disabled at (12891617): [] irq_exit_rcu+0xb0/0xd0 [ 2319.004747] ---[ end trace 0000000000000000 ]--- [ 2319.009366] ------------[ cut here ]------------ [ 2319.014012] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2319.025676] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2319.034375] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2319.089874] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2319.099019] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2319.106496] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2319.112011] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2319.117495] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2319.136260] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2319.141485] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2319.148635] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2319.155785] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007889 [ 2319.162936] R10: 00000000002805a0 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2319.170084] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2319.177217] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2319.185303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2319.191066] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2319.198201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2319.205354] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2319.212484] Call Trace: [ 2319.214954] [ 2319.217077] ? __warn+0x85/0x180 [ 2319.220310] ? __list_add_valid_or_report+0x7d/0xa0 [ 2319.225208] ? report_bug+0xfc/0x1d0 [ 2319.228784] ? prb_read_valid+0x1b/0x20 [ 2319.232644] ? handle_bug+0x44/0x70 [ 2319.236151] ? exc_invalid_op+0x17/0x60 [ 2319.240009] ? asm_exc_invalid_op+0x1a/0x20 [ 2319.244201] ? __list_add_valid_or_report+0x7d/0xa0 [ 2319.249075] ? __list_add_valid_or_report+0x7d/0xa0 [ 2319.253953] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2319.258870] ceph_add_cap+0x20b/0x620 [ceph] [ 2319.263157] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2319.267895] ? lock_acquire+0xcb/0x2b0 [ 2319.271664] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2319.276117] ? __mutex_lock+0x97/0xce0 [ 2319.279868] ? __mutex_lock+0xaec/0xce0 [ 2319.283725] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2319.288292] ? lock_downgrade+0xfd/0x250 [ 2319.292242] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2319.296636] ceph_con_process_message+0x74/0x150 [libceph] [ 2319.302132] ? calc_signature+0xdb/0x100 [libceph] [ 2319.306937] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2319.312276] ? process_scheduled_works+0x22a/0x510 [ 2319.317068] ? process_scheduled_works+0x22a/0x510 [ 2319.321858] ceph_con_workfn+0x344/0x750 [libceph] [ 2319.326680] ? process_scheduled_works+0x22a/0x510 [ 2319.331486] process_scheduled_works+0x2a4/0x510 [ 2319.336129] worker_thread+0x188/0x340 [ 2319.339877] ? rescuer_thread+0x310/0x310 [ 2319.343907] kthread+0xf5/0x130 [ 2319.347071] ? kthread_complete_and_exit+0x20/0x20 [ 2319.351865] ret_from_fork+0x31/0x40 [ 2319.355460] ? kthread_complete_and_exit+0x20/0x20 [ 2319.360285] ret_from_fork_asm+0x11/0x20 [ 2319.364221] [ 2319.366415] irq event stamp: 10644777 [ 2319.370098] hardirqs last enabled at (10644787): [] console_unlock+0x106/0x120 [ 2319.378965] hardirqs last disabled at (10644796): [] console_unlock+0xeb/0x120 [ 2319.387761] softirqs last enabled at (10644734): [] __do_softirq+0x2be/0x371 [ 2319.396471] softirqs last disabled at (10644729): [] irq_exit_rcu+0xb0/0xd0 [ 2319.405015] ---[ end trace 0000000000000000 ]--- [ 2319.409784] ------------[ cut here ]------------ [ 2319.414405] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2319.426065] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2319.434761] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2319.490255] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2319.499381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2319.506878] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2319.512390] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2319.517893] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2319.536658] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2319.541901] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2319.549052] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2319.556184] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000078c8 [ 2319.563316] R10: 00000000002817f8 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2319.570466] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2319.577615] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2319.585711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2319.591474] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2319.598626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2319.605776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2319.612908] Call Trace: [ 2319.615377] [ 2319.617493] ? __warn+0x85/0x180 [ 2319.620744] ? __list_add_valid_or_report+0x7d/0xa0 [ 2319.625641] ? report_bug+0xfc/0x1d0 [ 2319.629236] ? prb_read_valid+0x1b/0x20 [ 2319.633076] ? handle_bug+0x44/0x70 [ 2319.636569] ? exc_invalid_op+0x17/0x60 [ 2319.640424] ? asm_exc_invalid_op+0x1a/0x20 [ 2319.644650] ? __list_add_valid_or_report+0x7d/0xa0 [ 2319.649526] ? __list_add_valid_or_report+0x7d/0xa0 [ 2319.654420] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2319.659328] ceph_add_cap+0x20b/0x620 [ceph] [ 2319.663648] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2319.668387] ceph_fill_trace+0x143/0xf30 [ceph] [ 2319.672925] ? __mutex_lock+0x97/0xce0 [ 2319.676695] ? __mutex_lock+0xaec/0xce0 [ 2319.680582] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2319.685140] ? lock_downgrade+0xfd/0x250 [ 2319.689070] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2319.693465] ceph_con_process_message+0x74/0x150 [libceph] [ 2319.698961] ? calc_signature+0xdb/0x100 [libceph] [ 2319.703802] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2319.709132] ? process_scheduled_works+0x22a/0x510 [ 2319.713930] ? process_scheduled_works+0x22a/0x510 [ 2319.718737] ceph_con_workfn+0x344/0x750 [libceph] [ 2319.723559] ? process_scheduled_works+0x22a/0x510 [ 2319.728367] process_scheduled_works+0x2a4/0x510 [ 2319.733027] worker_thread+0x188/0x340 [ 2319.736784] ? rescuer_thread+0x310/0x310 [ 2319.740812] kthread+0xf5/0x130 [ 2319.743976] ? kthread_complete_and_exit+0x20/0x20 [ 2319.748788] ret_from_fork+0x31/0x40 [ 2319.752365] ? kthread_complete_and_exit+0x20/0x20 [ 2319.757177] ret_from_fork_asm+0x11/0x20 [ 2319.761110] [ 2319.763303] irq event stamp: 10645911 [ 2319.766978] hardirqs last enabled at (10645921): [] console_unlock+0x106/0x120 [ 2319.775861] hardirqs last disabled at (10645930): [] console_unlock+0xeb/0x120 [ 2319.784657] softirqs last enabled at (10645868): [] __do_softirq+0x2be/0x371 [ 2319.793369] softirqs last disabled at (10645863): [] irq_exit_rcu+0xb0/0xd0 [ 2319.801906] ---[ end trace 0000000000000000 ]--- [ 2319.806579] ------------[ cut here ]------------ [ 2319.811199] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2319.822855] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2319.831547] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2319.887040] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2319.896166] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2319.903646] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2319.909156] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2319.914641] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2319.933399] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2319.938642] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2319.945793] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2319.952944] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007906 [ 2319.960093] R10: 0000000000282a00 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2319.967226] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2319.974359] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2319.982460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2319.988224] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2319.995357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2320.002509] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2320.009657] Call Trace: [ 2320.012120] [ 2320.014227] ? __warn+0x85/0x180 [ 2320.017460] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.022355] ? report_bug+0xfc/0x1d0 [ 2320.025951] ? prb_read_valid+0x1b/0x20 [ 2320.029809] ? handle_bug+0x44/0x70 [ 2320.033318] ? exc_invalid_op+0x17/0x60 [ 2320.037176] ? asm_exc_invalid_op+0x1a/0x20 [ 2320.041383] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.046260] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.051137] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2320.056032] ceph_add_cap+0x20b/0x620 [ceph] [ 2320.060337] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2320.065055] ? lock_acquire+0xcb/0x2b0 [ 2320.068831] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2320.073297] ? __mutex_lock+0x97/0xce0 [ 2320.077051] ? __mutex_lock+0xaec/0xce0 [ 2320.080892] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2320.085448] ? lock_downgrade+0xfd/0x250 [ 2320.089414] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2320.093812] ceph_con_process_message+0x74/0x150 [libceph] [ 2320.099309] ? calc_signature+0xdb/0x100 [libceph] [ 2320.104137] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2320.109475] ? process_scheduled_works+0x22a/0x510 [ 2320.114269] ? process_scheduled_works+0x22a/0x510 [ 2320.119058] ceph_con_workfn+0x344/0x750 [libceph] [ 2320.123880] ? process_scheduled_works+0x22a/0x510 [ 2320.128669] process_scheduled_works+0x2a4/0x510 [ 2320.133332] worker_thread+0x188/0x340 [ 2320.137087] ? rescuer_thread+0x310/0x310 [ 2320.141099] kthread+0xf5/0x130 [ 2320.144247] ? kthread_complete_and_exit+0x20/0x20 [ 2320.149039] ret_from_fork+0x31/0x40 [ 2320.152617] ? kthread_complete_and_exit+0x20/0x20 [ 2320.157427] ret_from_fork_asm+0x11/0x20 [ 2320.161394] [ 2320.163579] irq event stamp: 10647023 [ 2320.167254] hardirqs last enabled at (10647033): [] console_unlock+0x106/0x120 [ 2320.176121] hardirqs last disabled at (10647042): [] console_unlock+0xeb/0x120 [ 2320.184902] softirqs last enabled at (10646970): [] __do_softirq+0x2be/0x371 [ 2320.193612] softirqs last disabled at (10646965): [] irq_exit_rcu+0xb0/0xd0 [ 2320.202148] ---[ end trace 0000000000000000 ]--- [ 2320.206789] ------------[ cut here ]------------ [ 2320.211406] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2320.223066] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2320.231764] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2320.287256] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2320.296381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2320.303871] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2320.309382] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2320.314884] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2320.333649] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2320.338893] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2320.346043] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2320.353178] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007945 [ 2320.360320] R10: 0000000000283c58 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2320.367468] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2320.374618] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2320.382706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2320.388465] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2320.395610] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2320.402758] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2320.409908] Call Trace: [ 2320.412369] [ 2320.414485] ? __warn+0x85/0x180 [ 2320.417766] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.422677] ? report_bug+0xfc/0x1d0 [ 2320.426255] ? prb_read_valid+0x1b/0x20 [ 2320.430097] ? handle_bug+0x44/0x70 [ 2320.433612] ? exc_invalid_op+0x17/0x60 [ 2320.437452] ? asm_exc_invalid_op+0x1a/0x20 [ 2320.441678] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.446553] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.451450] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2320.456358] ceph_add_cap+0x20b/0x620 [ceph] [ 2320.460677] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2320.465421] ceph_fill_trace+0x143/0xf30 [ceph] [ 2320.469963] ? __mutex_lock+0x97/0xce0 [ 2320.473730] ? __mutex_lock+0xaec/0xce0 [ 2320.477589] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2320.482148] ? lock_downgrade+0xfd/0x250 [ 2320.486078] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2320.490476] ceph_con_process_message+0x74/0x150 [libceph] [ 2320.495971] ? calc_signature+0xdb/0x100 [libceph] [ 2320.500800] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2320.506123] ? process_scheduled_works+0x22a/0x510 [ 2320.510930] ? process_scheduled_works+0x22a/0x510 [ 2320.515721] ceph_con_workfn+0x344/0x750 [libceph] [ 2320.520544] ? process_scheduled_works+0x22a/0x510 [ 2320.525350] process_scheduled_works+0x2a4/0x510 [ 2320.530016] worker_thread+0x188/0x340 [ 2320.533766] ? rescuer_thread+0x310/0x310 [ 2320.537779] kthread+0xf5/0x130 [ 2320.540943] ? kthread_complete_and_exit+0x20/0x20 [ 2320.545756] ret_from_fork+0x31/0x40 [ 2320.549350] ? kthread_complete_and_exit+0x20/0x20 [ 2320.554159] ret_from_fork_asm+0x11/0x20 [ 2320.558094] [ 2320.560286] irq event stamp: 10648145 [ 2320.563960] hardirqs last enabled at (10648155): [] console_unlock+0x106/0x120 [ 2320.572835] hardirqs last disabled at (10648164): [] console_unlock+0xeb/0x120 [ 2320.581633] softirqs last enabled at (10648102): [] __do_softirq+0x2be/0x371 [ 2320.590344] softirqs last disabled at (10648097): [] irq_exit_rcu+0xb0/0xd0 [ 2320.598881] ---[ end trace 0000000000000000 ]--- [ 2320.603552] ------------[ cut here ]------------ [ 2320.608175] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2320.619833] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2320.628531] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2320.684043] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2320.693166] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2320.700647] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2320.706158] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2320.711642] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2320.730408] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2320.735651] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2320.742802] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2320.749952] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007983 [ 2320.757102] R10: 0000000000284e70 R11: ffffc9000030b828 R12: ffff8881c5351ae0 [ 2320.764235] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2320.771367] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2320.779472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2320.785235] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2320.792366] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2320.799509] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2320.806659] Call Trace: [ 2320.809128] [ 2320.811234] ? __warn+0x85/0x180 [ 2320.814469] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.819365] ? report_bug+0xfc/0x1d0 [ 2320.822961] ? prb_read_valid+0x1b/0x20 [ 2320.826801] ? handle_bug+0x44/0x70 [ 2320.830321] ? exc_invalid_op+0x17/0x60 [ 2320.834177] ? asm_exc_invalid_op+0x1a/0x20 [ 2320.838385] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.843260] ? __list_add_valid_or_report+0x7d/0xa0 [ 2320.848139] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2320.853043] ceph_add_cap+0x20b/0x620 [ceph] [ 2320.857348] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2320.862063] ? lock_acquire+0xcb/0x2b0 [ 2320.865841] ceph_fill_trace+0x143/0xf30 [ceph] [ 2320.870377] ? __mutex_lock+0x97/0xce0 [ 2320.874148] ? __mutex_lock+0xaec/0xce0 [ 2320.877989] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2320.882553] ? lock_downgrade+0xfd/0x250 [ 2320.886502] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2320.890899] ceph_con_process_message+0x74/0x150 [libceph] [ 2320.896394] ? calc_signature+0xdb/0x100 [libceph] [ 2320.901217] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2320.906565] ? process_scheduled_works+0x22a/0x510 [ 2320.911357] ? process_scheduled_works+0x22a/0x510 [ 2320.916163] ceph_con_workfn+0x344/0x750 [libceph] [ 2320.920968] ? process_scheduled_works+0x22a/0x510 [ 2320.925775] process_scheduled_works+0x2a4/0x510 [ 2320.930434] worker_thread+0x188/0x340 [ 2320.934184] ? rescuer_thread+0x310/0x310 [ 2320.938195] kthread+0xf5/0x130 [ 2320.941343] ? kthread_complete_and_exit+0x20/0x20 [ 2320.946152] ret_from_fork+0x31/0x40 [ 2320.949731] ? kthread_complete_and_exit+0x20/0x20 [ 2320.954540] ret_from_fork_asm+0x11/0x20 [ 2320.958509] [ 2320.960695] irq event stamp: 10649253 [ 2320.964379] hardirqs last enabled at (10649263): [] console_unlock+0x106/0x120 [ 2320.973263] hardirqs last disabled at (10649272): [] console_unlock+0xeb/0x120 [ 2320.982042] softirqs last enabled at (10649200): [] __do_softirq+0x2be/0x371 [ 2320.990733] softirqs last disabled at (10649195): [] irq_exit_rcu+0xb0/0xd0 [ 2320.999270] ---[ end trace 0000000000000000 ]--- [ 2321.003925] ------------[ cut here ]------------ [ 2321.008548] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2321.020190] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2321.028885] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2321.084363] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2321.093487] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2321.100969] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2321.106469] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2321.111956] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2321.130702] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2321.135928] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2321.143061] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2321.150196] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000079c2 [ 2321.157327] R10: 00000000002860c8 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2321.164460] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2321.171594] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2321.179678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2321.185424] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2321.192558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2321.199692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2321.206824] Call Trace: [ 2321.209276] [ 2321.211383] ? __warn+0x85/0x180 [ 2321.214616] ? __list_add_valid_or_report+0x7d/0xa0 [ 2321.219495] ? report_bug+0xfc/0x1d0 [ 2321.223073] ? prb_read_valid+0x1b/0x20 [ 2321.226916] ? handle_bug+0x44/0x70 [ 2321.230406] ? exc_invalid_op+0x17/0x60 [ 2321.234246] ? asm_exc_invalid_op+0x1a/0x20 [ 2321.238436] ? __list_add_valid_or_report+0x7d/0xa0 [ 2321.243312] ? __list_add_valid_or_report+0x7d/0xa0 [ 2321.248190] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2321.253081] ceph_add_cap+0x20b/0x620 [ceph] [ 2321.257377] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2321.262101] ? lock_acquire+0xcb/0x2b0 [ 2321.265858] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2321.270310] ? __mutex_lock+0x97/0xce0 [ 2321.274061] ? __mutex_lock+0xaec/0xce0 [ 2321.277902] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2321.282452] ? lock_downgrade+0xfd/0x250 [ 2321.286383] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2321.290762] ceph_con_process_message+0x74/0x150 [libceph] [ 2321.296256] ? calc_signature+0xdb/0x100 [libceph] [ 2321.301061] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2321.306393] ? process_scheduled_works+0x22a/0x510 [ 2321.311184] ? process_scheduled_works+0x22a/0x510 [ 2321.315975] ceph_con_workfn+0x344/0x750 [libceph] [ 2321.320778] ? process_scheduled_works+0x22a/0x510 [ 2321.325567] process_scheduled_works+0x2a4/0x510 [ 2321.330194] worker_thread+0x188/0x340 [ 2321.333950] ? rescuer_thread+0x310/0x310 [ 2321.337961] kthread+0xf5/0x130 [ 2321.341111] ? kthread_complete_and_exit+0x20/0x20 [ 2321.345902] ret_from_fork+0x31/0x40 [ 2321.349480] ? kthread_complete_and_exit+0x20/0x20 [ 2321.354274] ret_from_fork_asm+0x11/0x20 [ 2321.358207] [ 2321.360401] irq event stamp: 10650391 [ 2321.364066] hardirqs last enabled at (10650401): [] console_unlock+0x106/0x120 [ 2321.372931] hardirqs last disabled at (10650410): [] console_unlock+0xeb/0x120 [ 2321.381712] softirqs last enabled at (10650338): [] __do_softirq+0x2be/0x371 [ 2321.390404] softirqs last disabled at (10650333): [] irq_exit_rcu+0xb0/0xd0 [ 2321.398923] ---[ end trace 0000000000000000 ]--- [ 2321.403544] ------------[ cut here ]------------ [ 2321.408161] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2321.419813] WARNING: CPU: 0 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2321.428773] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2321.484245] CPU: 0 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2321.493205] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2321.500684] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2321.506169] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2321.524925] RSP: 0018:ffffc90001f7fb78 EFLAGS: 00010286 [ 2321.530163] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2321.537290] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2321.544423] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007a01 [ 2321.551559] R10: 0000000000287420 R11: ffffc90001f7fa08 R12: ffff8881c5351ae0 [ 2321.558690] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2321.565821] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2321.573909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2321.579653] CR2: 00007fb38ced0000 CR3: 0000000113532001 CR4: 00000000003706f0 [ 2321.586786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2321.593919] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2321.601054] Call Trace: [ 2321.603507] [ 2321.605612] ? __warn+0x85/0x180 [ 2321.608846] ? __list_add_valid_or_report+0x7d/0xa0 [ 2321.613727] ? report_bug+0xfc/0x1d0 [ 2321.617313] ? prb_read_valid+0x1b/0x20 [ 2321.621156] ? handle_bug+0x44/0x70 [ 2321.624653] ? exc_invalid_op+0x17/0x60 [ 2321.628495] ? asm_exc_invalid_op+0x1a/0x20 [ 2321.632693] ? __list_add_valid_or_report+0x7d/0xa0 [ 2321.637578] ? __list_add_valid_or_report+0x7d/0xa0 [ 2321.642462] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2321.647366] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2321.652607] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2321.657419] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2321.662138] ? __d_lookup+0xd6/0x200 [ 2321.665719] lookup_fast+0x6d/0xf0 [ 2321.669123] walk_component+0x26/0x160 [ 2321.672881] link_path_walk+0x24d/0x380 [ 2321.676722] ? path_openat+0x80/0x9d0 [ 2321.680397] path_openat+0x8e/0x9d0 [ 2321.683897] do_filp_open+0xb8/0x120 [ 2321.687487] ? lock_release+0xc5/0x260 [ 2321.691244] ? _raw_spin_unlock+0x23/0x40 [ 2321.695263] ? alloc_fd+0x13f/0x200 [ 2321.698758] do_sys_openat2+0x249/0x290 [ 2321.702607] do_sys_open+0x4f/0x80 [ 2321.706017] do_syscall_64+0x38/0x80 [ 2321.709599] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2321.714657] RIP: 0033:0x7f48f0eed2f2 [ 2321.718237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2321.736983] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2321.744552] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2321.751680] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2321.758816] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2321.765947] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2321.773082] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2321.780225] [ 2321.782416] irq event stamp: 12953531 [ 2321.786079] hardirqs last enabled at (12953541): [] console_unlock+0x106/0x120 [ 2321.794948] hardirqs last disabled at (12953550): [] console_unlock+0xeb/0x120 [ 2321.803726] softirqs last enabled at (12953468): [] __do_softirq+0x2be/0x371 [ 2321.812421] softirqs last disabled at (12953569): [] irq_exit_rcu+0xb0/0xd0 [ 2321.820937] ---[ end trace 0000000000000000 ]--- [ 2321.825559] ------------[ cut here ]------------ [ 2321.830174] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2321.841837] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2321.850536] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2321.906054] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2321.915183] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2321.922660] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2321.928173] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2321.933657] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2321.952421] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2321.957684] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2321.964816] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2321.971966] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007a40 [ 2321.979117] R10: 0000000000288580 R11: ffffc9000030b828 R12: ffff8881d106cbc8 [ 2321.986247] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2321.993380] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2322.001484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2322.007247] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2322.014382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2322.021530] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2322.028681] Call Trace: [ 2322.031143] [ 2322.033250] ? __warn+0x85/0x180 [ 2322.036482] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.041378] ? report_bug+0xfc/0x1d0 [ 2322.044977] ? prb_read_valid+0x1b/0x20 [ 2322.048834] ? handle_bug+0x44/0x70 [ 2322.052343] ? exc_invalid_op+0x17/0x60 [ 2322.056199] ? asm_exc_invalid_op+0x1a/0x20 [ 2322.060406] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.065282] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.070161] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2322.075067] ceph_add_cap+0x20b/0x620 [ceph] [ 2322.079370] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2322.084092] ceph_fill_trace+0x143/0xf30 [ceph] [ 2322.088631] ? __mutex_lock+0x97/0xce0 [ 2322.092399] ? __mutex_lock+0xaec/0xce0 [ 2322.096259] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2322.100808] ? lock_downgrade+0xfd/0x250 [ 2322.104773] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2322.109155] ceph_con_process_message+0x74/0x150 [libceph] [ 2322.114647] ? calc_signature+0xdb/0x100 [libceph] [ 2322.119487] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2322.124835] ? process_scheduled_works+0x22a/0x510 [ 2322.129625] ? process_scheduled_works+0x22a/0x510 [ 2322.134433] ceph_con_workfn+0x344/0x750 [libceph] [ 2322.139255] ? process_scheduled_works+0x22a/0x510 [ 2322.144063] process_scheduled_works+0x2a4/0x510 [ 2322.148703] worker_thread+0x188/0x340 [ 2322.152454] ? rescuer_thread+0x310/0x310 [ 2322.156483] kthread+0xf5/0x130 [ 2322.159645] ? kthread_complete_and_exit+0x20/0x20 [ 2322.164455] ret_from_fork+0x31/0x40 [ 2322.168064] ? kthread_complete_and_exit+0x20/0x20 [ 2322.172856] ret_from_fork_asm+0x11/0x20 [ 2322.176822] [ 2322.179015] irq event stamp: 10652415 [ 2322.182691] hardirqs last enabled at (10652425): [] console_unlock+0x106/0x120 [ 2322.191575] hardirqs last disabled at (10652434): [] console_unlock+0xeb/0x120 [ 2322.200372] softirqs last enabled at (10652392): [] __do_softirq+0x2be/0x371 [ 2322.209082] softirqs last disabled at (10652387): [] irq_exit_rcu+0xb0/0xd0 [ 2322.217601] ---[ end trace 0000000000000000 ]--- [ 2322.222397] ------------[ cut here ]------------ [ 2322.227015] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2322.238694] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2322.247388] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2322.302884] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2322.312026] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2322.319522] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2322.325046] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2322.330529] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2322.349293] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2322.354517] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2322.361660] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2322.368813] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007a7e [ 2322.375962] R10: 0000000000289898 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2322.383110] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2322.390244] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2322.398345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2322.404101] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2322.411236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2322.418368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2322.425519] Call Trace: [ 2322.427987] [ 2322.430102] ? __warn+0x85/0x180 [ 2322.433350] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.438241] ? report_bug+0xfc/0x1d0 [ 2322.441821] ? prb_read_valid+0x1b/0x20 [ 2322.445679] ? handle_bug+0x44/0x70 [ 2322.449188] ? exc_invalid_op+0x17/0x60 [ 2322.453027] ? asm_exc_invalid_op+0x1a/0x20 [ 2322.457234] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.462110] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.466989] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2322.471896] ceph_add_cap+0x20b/0x620 [ceph] [ 2322.476200] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2322.480940] ? lock_acquire+0xcb/0x2b0 [ 2322.484707] ceph_fill_trace+0x143/0xf30 [ceph] [ 2322.489246] ? __mutex_lock+0x97/0xce0 [ 2322.492997] ? __mutex_lock+0xaec/0xce0 [ 2322.496856] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2322.501413] ? lock_downgrade+0xfd/0x250 [ 2322.505373] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2322.509766] ceph_con_process_message+0x74/0x150 [libceph] [ 2322.515263] ? calc_signature+0xdb/0x100 [libceph] [ 2322.520073] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2322.525414] ? process_scheduled_works+0x22a/0x510 [ 2322.530207] ? process_scheduled_works+0x22a/0x510 [ 2322.534998] ceph_con_workfn+0x344/0x750 [libceph] [ 2322.539818] ? process_scheduled_works+0x22a/0x510 [ 2322.544625] process_scheduled_works+0x2a4/0x510 [ 2322.549270] worker_thread+0x188/0x340 [ 2322.553024] ? rescuer_thread+0x310/0x310 [ 2322.557068] kthread+0xf5/0x130 [ 2322.560212] ? kthread_complete_and_exit+0x20/0x20 [ 2322.565003] ret_from_fork+0x31/0x40 [ 2322.568598] ? kthread_complete_and_exit+0x20/0x20 [ 2322.573408] ret_from_fork_asm+0x11/0x20 [ 2322.577375] [ 2322.579560] irq event stamp: 10653535 [ 2322.583235] hardirqs last enabled at (10653545): [] console_unlock+0x106/0x120 [ 2322.592104] hardirqs last disabled at (10653554): [] console_unlock+0xeb/0x120 [ 2322.600883] softirqs last enabled at (10653482): [] __do_softirq+0x2be/0x371 [ 2322.609592] softirqs last disabled at (10653477): [] irq_exit_rcu+0xb0/0xd0 [ 2322.618129] ---[ end trace 0000000000000000 ]--- [ 2322.622779] ------------[ cut here ]------------ [ 2322.627397] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2322.639046] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2322.647734] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2322.703213] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2322.712337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2322.719817] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2322.725312] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2322.730799] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2322.749546] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2322.754769] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2322.761904] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2322.769038] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007abd [ 2322.776169] R10: 000000000028abd0 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2322.783302] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2322.790436] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2322.798522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2322.804265] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2322.811399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2322.818533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2322.825664] Call Trace: [ 2322.828119] [ 2322.830223] ? __warn+0x85/0x180 [ 2322.833459] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.838337] ? report_bug+0xfc/0x1d0 [ 2322.841917] ? prb_read_valid+0x1b/0x20 [ 2322.845756] ? handle_bug+0x44/0x70 [ 2322.849248] ? exc_invalid_op+0x17/0x60 [ 2322.853086] ? asm_exc_invalid_op+0x1a/0x20 [ 2322.857279] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.862163] ? __list_add_valid_or_report+0x7d/0xa0 [ 2322.867040] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2322.871940] ceph_add_cap+0x20b/0x620 [ceph] [ 2322.876226] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2322.880949] ? lock_acquire+0xcb/0x2b0 [ 2322.884710] ceph_fill_trace+0x143/0xf30 [ceph] [ 2322.889254] ? __mutex_lock+0x97/0xce0 [ 2322.893005] ? __mutex_lock+0xaec/0xce0 [ 2322.896848] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2322.901387] ? lock_downgrade+0xfd/0x250 [ 2322.905318] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2322.909699] ceph_con_process_message+0x74/0x150 [libceph] [ 2322.915194] ? calc_signature+0xdb/0x100 [libceph] [ 2322.920007] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2322.925339] ? process_scheduled_works+0x22a/0x510 [ 2322.930129] ? process_scheduled_works+0x22a/0x510 [ 2322.934918] ceph_con_workfn+0x344/0x750 [libceph] [ 2322.939722] ? process_scheduled_works+0x22a/0x510 [ 2322.944513] process_scheduled_works+0x2a4/0x510 [ 2322.949138] worker_thread+0x188/0x340 [ 2322.952886] ? rescuer_thread+0x310/0x310 [ 2322.956899] kthread+0xf5/0x130 [ 2322.960050] ? kthread_complete_and_exit+0x20/0x20 [ 2322.964850] ret_from_fork+0x31/0x40 [ 2322.968426] ? kthread_complete_and_exit+0x20/0x20 [ 2322.973218] ret_from_fork_asm+0x11/0x20 [ 2322.977151] [ 2322.979337] irq event stamp: 10654653 [ 2322.983002] hardirqs last enabled at (10654663): [] console_unlock+0x106/0x120 [ 2322.991870] hardirqs last disabled at (10654672): [] console_unlock+0xeb/0x120 [ 2323.000648] softirqs last enabled at (10654598): [] __do_softirq+0x2be/0x371 [ 2323.009342] softirqs last disabled at (10654593): [] irq_exit_rcu+0xb0/0xd0 [ 2323.017862] ---[ end trace 0000000000000000 ]--- [ 2323.022482] ------------[ cut here ]------------ [ 2323.027099] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2323.038751] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2323.047714] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2323.103192] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2323.112163] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2323.119646] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2323.125133] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2323.143877] RSP: 0018:ffffc90001957ac0 EFLAGS: 00010282 [ 2323.149104] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2323.156237] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2323.163369] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007afc [ 2323.170502] R10: 000000000028bf78 R11: ffffc90001957950 R12: ffff8881d1224bc8 [ 2323.177634] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2323.184768] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2323.192855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2323.198598] CR2: 00007fb38ced0000 CR3: 000000016696c002 CR4: 00000000003706f0 [ 2323.205732] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2323.212866] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2323.219999] Call Trace: [ 2323.222452] [ 2323.224558] ? __warn+0x85/0x180 [ 2323.227791] ? __list_add_valid_or_report+0x7d/0xa0 [ 2323.232672] ? report_bug+0xfc/0x1d0 [ 2323.236260] ? prb_read_valid+0x1b/0x20 [ 2323.240101] ? handle_bug+0x44/0x70 [ 2323.243598] ? exc_invalid_op+0x17/0x60 [ 2323.247441] ? asm_exc_invalid_op+0x1a/0x20 [ 2323.251640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2323.256524] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2323.261421] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2323.266676] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2323.271481] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2323.276208] ? __d_lookup+0xd6/0x200 [ 2323.279793] lookup_fast+0x6d/0xf0 [ 2323.283199] walk_component+0x26/0x160 [ 2323.286961] link_path_walk+0x24d/0x380 [ 2323.290799] ? path_lookupat.isra.40+0x22/0x120 [ 2323.295336] path_lookupat.isra.40+0x6b/0x120 [ 2323.299706] filename_lookup+0xd4/0x190 [ 2323.303553] ? lock_acquire+0xcb/0x2b0 [ 2323.307309] ? __might_fault+0x45/0x80 [ 2323.311071] ? find_held_lock+0x2d/0x90 [ 2323.314920] ? __might_fault+0x45/0x80 [ 2323.318683] vfs_statx+0x89/0x150 [ 2323.322014] vfs_fstatat+0x5a/0xc0 [ 2323.325425] __do_sys_newstat+0x30/0x70 [ 2323.329264] ? lock_release+0xc5/0x260 [ 2323.333016] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2323.337902] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2323.343043] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2323.348186] do_syscall_64+0x38/0x80 [ 2323.351769] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2323.356822] RIP: 0033:0x7f48f0eeca79 [ 2323.360401] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2323.379152] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2323.386720] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2323.393857] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2323.400988] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2323.408121] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2323.415252] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2323.422396] [ 2323.424586] irq event stamp: 12841787 [ 2323.428254] hardirqs last enabled at (12841797): [] console_unlock+0x106/0x120 [ 2323.437117] hardirqs last disabled at (12841806): [] console_unlock+0xeb/0x120 [ 2323.445904] softirqs last enabled at (12841820): [] __do_softirq+0x2be/0x371 [ 2323.454600] softirqs last disabled at (12841815): [] irq_exit_rcu+0xb0/0xd0 [ 2323.463121] ---[ end trace 0000000000000000 ]--- [ 2323.467739] ------------[ cut here ]------------ [ 2323.472356] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2323.484006] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2323.492957] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2323.548419] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2323.557369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2323.564840] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2323.570327] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2323.589072] RSP: 0018:ffffc90001ff7ac0 EFLAGS: 00010282 [ 2323.594298] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2323.601422] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2323.608548] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007b40 [ 2323.615681] R10: 000000000028d538 R11: ffffc90001ff7950 R12: ffff8881c25d1118 [ 2323.622813] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2323.629944] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2323.638032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2323.643776] CR2: 00007f9bd22d7020 CR3: 0000000120cc8006 CR4: 00000000003706e0 [ 2323.650902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2323.658034] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2323.665157] Call Trace: [ 2323.667603] [ 2323.669709] ? __warn+0x85/0x180 [ 2323.672942] ? __list_add_valid_or_report+0x7d/0xa0 [ 2323.677821] ? report_bug+0xfc/0x1d0 [ 2323.681402] ? prb_read_valid+0x1b/0x20 [ 2323.685241] ? handle_bug+0x44/0x70 [ 2323.688733] ? exc_invalid_op+0x17/0x60 [ 2323.692572] ? asm_exc_invalid_op+0x1a/0x20 [ 2323.696762] ? __list_add_valid_or_report+0x7d/0xa0 [ 2323.701639] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2323.706528] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2323.711762] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2323.716563] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2323.721278] ? __d_lookup+0xd6/0x200 [ 2323.724857] lookup_fast+0x6d/0xf0 [ 2323.728262] walk_component+0x26/0x160 [ 2323.732015] link_path_walk+0x24d/0x380 [ 2323.735854] ? path_lookupat.isra.40+0x22/0x120 [ 2323.740388] path_lookupat.isra.40+0x6b/0x120 [ 2323.744749] filename_lookup+0xd4/0x190 [ 2323.748589] ? lock_acquire+0xcb/0x2b0 [ 2323.752339] ? __might_fault+0x45/0x80 [ 2323.756091] ? find_held_lock+0x2d/0x90 [ 2323.759931] ? __might_fault+0x45/0x80 [ 2323.763687] vfs_statx+0x89/0x150 [ 2323.767007] vfs_fstatat+0x5a/0xc0 [ 2323.770410] __do_sys_newstat+0x30/0x70 [ 2323.774249] ? lock_release+0xc5/0x260 [ 2323.778003] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2323.782881] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2323.788021] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2323.793160] do_syscall_64+0x38/0x80 [ 2323.796739] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2323.801792] RIP: 0033:0x7f48f0eeca79 [ 2323.805372] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2323.824117] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2323.831684] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2323.838815] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2323.845938] RBP: 00007ffc53d60520 R08: 00000000000029f6 R09: 00000000020767c0 [ 2323.853066] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2323.860197] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2323.867337] [ 2323.869523] irq event stamp: 12966337 [ 2323.873187] hardirqs last enabled at (12966347): [] console_unlock+0x106/0x120 [ 2323.882055] hardirqs last disabled at (12966356): [] console_unlock+0xeb/0x120 [ 2323.890834] softirqs last enabled at (12966370): [] __do_softirq+0x2be/0x371 [ 2323.899526] softirqs last disabled at (12966365): [] irq_exit_rcu+0xb0/0xd0 [ 2323.908047] ---[ end trace 0000000000000000 ]--- [ 2323.912666] ------------[ cut here ]------------ [ 2323.917284] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2323.928948] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2323.937644] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2323.993144] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2324.002274] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2324.009753] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2324.015265] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2324.020749] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2324.039516] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2324.044757] RAX: 0000000000000000 RBX: ffff8881c3b761d0 RCX: 0000000000000000 [ 2324.051900] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2324.059050] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007b84 [ 2324.066199] R10: 000000000028e8a0 R11: ffffc9000030b828 R12: ffff8881c3b76920 [ 2324.073333] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2324.080466] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2324.088568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2324.094322] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2324.101455] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2324.108606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2324.115756] Call Trace: [ 2324.118228] [ 2324.120334] ? __warn+0x85/0x180 [ 2324.123568] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.128463] ? report_bug+0xfc/0x1d0 [ 2324.132058] ? prb_read_valid+0x1b/0x20 [ 2324.135916] ? handle_bug+0x44/0x70 [ 2324.139427] ? exc_invalid_op+0x17/0x60 [ 2324.143282] ? asm_exc_invalid_op+0x1a/0x20 [ 2324.147489] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.152382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.157279] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2324.162169] ceph_add_cap+0x20b/0x620 [ceph] [ 2324.166473] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2324.171196] ? lock_acquire+0xcb/0x2b0 [ 2324.174973] ceph_fill_trace+0x143/0xf30 [ceph] [ 2324.179512] ? __mutex_lock+0x97/0xce0 [ 2324.183281] ? __mutex_lock+0xaec/0xce0 [ 2324.187122] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2324.191670] ? lock_downgrade+0xfd/0x250 [ 2324.195635] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2324.200032] ceph_con_process_message+0x74/0x150 [libceph] [ 2324.205527] ? calc_signature+0xdb/0x100 [libceph] [ 2324.210349] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2324.215697] ? process_scheduled_works+0x22a/0x510 [ 2324.220489] ? process_scheduled_works+0x22a/0x510 [ 2324.225295] ceph_con_workfn+0x344/0x750 [libceph] [ 2324.230104] ? process_scheduled_works+0x22a/0x510 [ 2324.234900] process_scheduled_works+0x2a4/0x510 [ 2324.239558] worker_thread+0x188/0x340 [ 2324.243307] ? rescuer_thread+0x310/0x310 [ 2324.247319] kthread+0xf5/0x130 [ 2324.250466] ? kthread_complete_and_exit+0x20/0x20 [ 2324.255276] ret_from_fork+0x31/0x40 [ 2324.258854] ? kthread_complete_and_exit+0x20/0x20 [ 2324.263665] ret_from_fork_asm+0x11/0x20 [ 2324.267614] [ 2324.269801] irq event stamp: 10657665 [ 2324.273483] hardirqs last enabled at (10657675): [] console_unlock+0x106/0x120 [ 2324.282385] hardirqs last disabled at (10657690): [] console_unlock+0xeb/0x120 [ 2324.291181] softirqs last enabled at (10657688): [] __do_softirq+0x2be/0x371 [ 2324.299876] softirqs last disabled at (10657683): [] irq_exit_rcu+0xb0/0xd0 [ 2324.308413] ---[ end trace 0000000000000000 ]--- [ 2324.313874] ------------[ cut here ]------------ [ 2324.318640] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2324.330289] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2324.338982] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2324.394484] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2324.403625] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2324.411114] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2324.416608] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2324.422112] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2324.440892] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2324.446118] RAX: 0000000000000000 RBX: ffff88819f028000 RCX: 0000000000000000 [ 2324.453260] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2324.460395] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007bc3 [ 2324.467544] R10: 000000000028faf0 R11: ffffc9000030b828 R12: ffff88819f028750 [ 2324.474692] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2324.481835] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2324.489939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2324.495700] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2324.502843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2324.509994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2324.517144] Call Trace: [ 2324.519596] [ 2324.521712] ? __warn+0x85/0x180 [ 2324.524961] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.529858] ? report_bug+0xfc/0x1d0 [ 2324.533453] ? prb_read_valid+0x1b/0x20 [ 2324.537312] ? handle_bug+0x44/0x70 [ 2324.540803] ? exc_invalid_op+0x17/0x60 [ 2324.544652] ? asm_exc_invalid_op+0x1a/0x20 [ 2324.548878] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.553752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.558650] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2324.563556] ceph_add_cap+0x20b/0x620 [ceph] [ 2324.567875] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2324.572607] ? lock_acquire+0xcb/0x2b0 [ 2324.576361] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2324.580821] ? __mutex_lock+0x97/0xce0 [ 2324.584589] ? __mutex_lock+0xaec/0xce0 [ 2324.588448] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2324.593013] ? lock_downgrade+0xfd/0x250 [ 2324.596978] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2324.601360] ceph_con_process_message+0x74/0x150 [libceph] [ 2324.606855] ? calc_signature+0xdb/0x100 [libceph] [ 2324.611701] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2324.617049] ? process_scheduled_works+0x22a/0x510 [ 2324.621841] ? process_scheduled_works+0x22a/0x510 [ 2324.626648] ceph_con_workfn+0x344/0x750 [libceph] [ 2324.631470] ? process_scheduled_works+0x22a/0x510 [ 2324.636279] process_scheduled_works+0x2a4/0x510 [ 2324.640918] worker_thread+0x188/0x340 [ 2324.644669] ? rescuer_thread+0x310/0x310 [ 2324.648697] kthread+0xf5/0x130 [ 2324.651862] ? kthread_complete_and_exit+0x20/0x20 [ 2324.656671] ret_from_fork+0x31/0x40 [ 2324.660267] ? kthread_complete_and_exit+0x20/0x20 [ 2324.665061] ret_from_fork_asm+0x11/0x20 [ 2324.669027] [ 2324.671212] irq event stamp: 10659753 [ 2324.674879] hardirqs last enabled at (10659763): [] console_unlock+0x106/0x120 [ 2324.683764] hardirqs last disabled at (10659772): [] console_unlock+0xeb/0x120 [ 2324.692561] softirqs last enabled at (10659700): [] __do_softirq+0x2be/0x371 [ 2324.701270] softirqs last disabled at (10659695): [] irq_exit_rcu+0xb0/0xd0 [ 2324.709790] ---[ end trace 0000000000000000 ]--- [ 2324.714453] ------------[ cut here ]------------ [ 2324.719072] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2324.730751] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2324.739449] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2324.794944] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2324.804103] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2324.811581] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2324.817112] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2324.822596] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2324.841361] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2324.846587] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2324.853738] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2324.860887] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007c02 [ 2324.868038] R10: 0000000000290c20 R11: ffffc9000030b828 R12: ffff88819f02f2e8 [ 2324.875187] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2324.882320] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2324.890406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2324.896167] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2324.903293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2324.910426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2324.917576] Call Trace: [ 2324.920037] [ 2324.922152] ? __warn+0x85/0x180 [ 2324.925392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.930291] ? report_bug+0xfc/0x1d0 [ 2324.933871] ? prb_read_valid+0x1b/0x20 [ 2324.937728] ? handle_bug+0x44/0x70 [ 2324.941238] ? exc_invalid_op+0x17/0x60 [ 2324.945076] ? asm_exc_invalid_op+0x1a/0x20 [ 2324.949285] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.954160] ? __list_add_valid_or_report+0x7d/0xa0 [ 2324.959039] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2324.963970] ceph_add_cap+0x20b/0x620 [ceph] [ 2324.968260] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2324.973003] ceph_fill_trace+0x143/0xf30 [ceph] [ 2324.977544] ? __mutex_lock+0x97/0xce0 [ 2324.981313] ? __mutex_lock+0xaec/0xce0 [ 2324.985152] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2324.989697] ? lock_downgrade+0xfd/0x250 [ 2324.993657] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2324.998046] ceph_con_process_message+0x74/0x150 [libceph] [ 2325.003542] ? calc_signature+0xdb/0x100 [libceph] [ 2325.008364] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2325.013713] ? process_scheduled_works+0x22a/0x510 [ 2325.018505] ? process_scheduled_works+0x22a/0x510 [ 2325.023312] ceph_con_workfn+0x344/0x750 [libceph] [ 2325.028118] ? process_scheduled_works+0x22a/0x510 [ 2325.032907] process_scheduled_works+0x2a4/0x510 [ 2325.037546] worker_thread+0x188/0x340 [ 2325.041297] ? rescuer_thread+0x310/0x310 [ 2325.045308] kthread+0xf5/0x130 [ 2325.048453] ? kthread_complete_and_exit+0x20/0x20 [ 2325.053268] ret_from_fork+0x31/0x40 [ 2325.056844] ? kthread_complete_and_exit+0x20/0x20 [ 2325.061653] ret_from_fork_asm+0x11/0x20 [ 2325.065622] [ 2325.067806] irq event stamp: 10660817 [ 2325.071480] hardirqs last enabled at (10660827): [] console_unlock+0x106/0x120 [ 2325.080366] hardirqs last disabled at (10660836): [] console_unlock+0xeb/0x120 [ 2325.089144] softirqs last enabled at (10660774): [] __do_softirq+0x2be/0x371 [ 2325.097839] softirqs last disabled at (10660769): [] irq_exit_rcu+0xb0/0xd0 [ 2325.106391] ---[ end trace 0000000000000000 ]--- [ 2325.111191] ------------[ cut here ]------------ [ 2325.115805] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2325.127467] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2325.136164] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2325.191705] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2325.200836] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2325.208331] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2325.213827] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2325.219328] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2325.238075] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2325.243317] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2325.250477] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2325.257612] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007c40 [ 2325.264792] R10: 0000000000291ba8 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2325.271921] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2325.279071] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2325.287173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2325.292919] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2325.300070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2325.307220] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2325.314354] Call Trace: [ 2325.316806] [ 2325.318929] ? __warn+0x85/0x180 [ 2325.322182] ? __list_add_valid_or_report+0x7d/0xa0 [ 2325.327059] ? report_bug+0xfc/0x1d0 [ 2325.330654] ? prb_read_valid+0x1b/0x20 [ 2325.334514] ? handle_bug+0x44/0x70 [ 2325.338022] ? exc_invalid_op+0x17/0x60 [ 2325.341879] ? asm_exc_invalid_op+0x1a/0x20 [ 2325.346112] ? __list_add_valid_or_report+0x7d/0xa0 [ 2325.350990] ? __list_add_valid_or_report+0x7d/0xa0 [ 2325.355868] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2325.360774] ceph_add_cap+0x20b/0x620 [ceph] [ 2325.365100] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2325.369815] ? lock_acquire+0xcb/0x2b0 [ 2325.373586] ceph_fill_trace+0x143/0xf30 [ceph] [ 2325.378124] ? __mutex_lock+0x97/0xce0 [ 2325.381875] ? __mutex_lock+0xaec/0xce0 [ 2325.385735] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2325.390293] ? lock_downgrade+0xfd/0x250 [ 2325.394222] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2325.398604] ceph_con_process_message+0x74/0x150 [libceph] [ 2325.404117] ? calc_signature+0xdb/0x100 [libceph] [ 2325.408929] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2325.414276] ? process_scheduled_works+0x22a/0x510 [ 2325.419068] ? process_scheduled_works+0x22a/0x510 [ 2325.423861] ceph_con_workfn+0x344/0x750 [libceph] [ 2325.428689] ? process_scheduled_works+0x22a/0x510 [ 2325.433496] process_scheduled_works+0x2a4/0x510 [ 2325.438138] worker_thread+0x188/0x340 [ 2325.441894] ? rescuer_thread+0x310/0x310 [ 2325.445924] kthread+0xf5/0x130 [ 2325.449088] ? kthread_complete_and_exit+0x20/0x20 [ 2325.453897] ret_from_fork+0x31/0x40 [ 2325.457476] ? kthread_complete_and_exit+0x20/0x20 [ 2325.462288] ret_from_fork_asm+0x11/0x20 [ 2325.466219] [ 2325.468410] irq event stamp: 10662115 [ 2325.472087] hardirqs last enabled at (10662125): [] console_unlock+0x106/0x120 [ 2325.480970] hardirqs last disabled at (10662136): [] console_unlock+0xeb/0x120 [ 2325.489742] softirqs last enabled at (10662060): [] __do_softirq+0x2be/0x371 [ 2325.498453] softirqs last disabled at (10662055): [] irq_exit_rcu+0xb0/0xd0 [ 2325.506991] ---[ end trace 0000000000000000 ]--- [ 2325.511650] ------------[ cut here ]------------ [ 2325.516264] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2325.527909] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2325.536607] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2325.592085] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2325.601206] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2325.608689] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2325.614191] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2325.619675] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2325.638424] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2325.643648] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2325.650781] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2325.657916] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007c7f [ 2325.665047] R10: 0000000000293190 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2325.672181] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2325.679312] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2325.687399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2325.693144] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2325.700279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2325.707411] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2325.714544] Call Trace: [ 2325.716997] [ 2325.719103] ? __warn+0x85/0x180 [ 2325.722337] ? __list_add_valid_or_report+0x7d/0xa0 [ 2325.727215] ? report_bug+0xfc/0x1d0 [ 2325.730795] ? prb_read_valid+0x1b/0x20 [ 2325.734636] ? handle_bug+0x44/0x70 [ 2325.738125] ? exc_invalid_op+0x17/0x60 [ 2325.741966] ? asm_exc_invalid_op+0x1a/0x20 [ 2325.746156] ? __list_add_valid_or_report+0x7d/0xa0 [ 2325.751032] ? __list_add_valid_or_report+0x7d/0xa0 [ 2325.755909] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2325.760802] ceph_add_cap+0x20b/0x620 [ceph] [ 2325.765087] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2325.769816] ? lock_acquire+0xcb/0x2b0 [ 2325.773571] ceph_fill_trace+0x143/0xf30 [ceph] [ 2325.778115] ? __mutex_lock+0x97/0xce0 [ 2325.781866] ? __mutex_lock+0xaec/0xce0 [ 2325.785710] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2325.790257] ? lock_downgrade+0xfd/0x250 [ 2325.794187] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2325.798569] ceph_con_process_message+0x74/0x150 [libceph] [ 2325.804063] ? calc_signature+0xdb/0x100 [libceph] [ 2325.808869] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2325.814198] ? process_scheduled_works+0x22a/0x510 [ 2325.818991] ? process_scheduled_works+0x22a/0x510 [ 2325.823779] ceph_con_workfn+0x344/0x750 [libceph] [ 2325.828583] ? process_scheduled_works+0x22a/0x510 [ 2325.833374] process_scheduled_works+0x2a4/0x510 [ 2325.837999] worker_thread+0x188/0x340 [ 2325.841755] ? rescuer_thread+0x310/0x310 [ 2325.845769] kthread+0xf5/0x130 [ 2325.848915] ? kthread_complete_and_exit+0x20/0x20 [ 2325.853707] ret_from_fork+0x31/0x40 [ 2325.857285] ? kthread_complete_and_exit+0x20/0x20 [ 2325.862079] ret_from_fork_asm+0x11/0x20 [ 2325.866012] [ 2325.868206] irq event stamp: 10663233 [ 2325.871871] hardirqs last enabled at (10663243): [] console_unlock+0x106/0x120 [ 2325.880739] hardirqs last disabled at (10663252): [] console_unlock+0xeb/0x120 [ 2325.889517] softirqs last enabled at (10663180): [] __do_softirq+0x2be/0x371 [ 2325.898212] softirqs last disabled at (10663175): [] irq_exit_rcu+0xb0/0xd0 [ 2325.906731] ---[ end trace 0000000000000000 ]--- [ 2325.911351] ------------[ cut here ]------------ [ 2325.915968] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2325.927619] WARNING: CPU: 0 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2325.936581] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2325.992055] CPU: 0 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2326.001010] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2326.008488] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2326.013975] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2326.032720] RSP: 0018:ffffc90001fc7b20 EFLAGS: 00010282 [ 2326.037948] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2326.045080] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2326.052213] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007cbe [ 2326.059347] R10: 0000000000294338 R11: ffffc90001fc79b0 R12: ffff8881d800c200 [ 2326.066479] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2326.073611] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2326.081698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2326.087442] CR2: 00007fb38ced0000 CR3: 0000000113500005 CR4: 00000000003706f0 [ 2326.094576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2326.101707] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2326.108841] Call Trace: [ 2326.111293] [ 2326.113403] ? __warn+0x85/0x180 [ 2326.116642] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.121525] ? report_bug+0xfc/0x1d0 [ 2326.125112] ? prb_read_valid+0x1b/0x20 [ 2326.128954] ? handle_bug+0x44/0x70 [ 2326.132451] ? exc_invalid_op+0x17/0x60 [ 2326.136291] ? asm_exc_invalid_op+0x1a/0x20 [ 2326.140485] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.145368] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2326.150265] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2326.155519] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2326.160325] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2326.165051] ? __d_lookup+0xd6/0x200 [ 2326.168636] lookup_fast+0x6d/0xf0 [ 2326.172044] walk_component+0x26/0x160 [ 2326.175803] path_lookupat.isra.40+0x58/0x120 [ 2326.180166] filename_lookup+0xd4/0x190 [ 2326.184038] ? lock_acquire+0xcb/0x2b0 [ 2326.187798] ? __might_fault+0x45/0x80 [ 2326.191561] ? find_held_lock+0x2d/0x90 [ 2326.195407] ? __might_fault+0x45/0x80 [ 2326.199165] vfs_statx+0x89/0x150 [ 2326.202493] vfs_fstatat+0x5a/0xc0 [ 2326.205903] __do_sys_newstat+0x30/0x70 [ 2326.209743] ? lock_release+0xc5/0x260 [ 2326.213496] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2326.218375] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2326.223514] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2326.228657] do_syscall_64+0x38/0x80 [ 2326.232240] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2326.237293] RIP: 0033:0x7f48f0eeca79 [ 2326.240871] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2326.259621] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2326.267192] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2326.274328] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2326.281459] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2326.288593] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2326.295723] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2326.302869] [ 2326.305059] irq event stamp: 12953839 [ 2326.308725] hardirqs last enabled at (12953849): [] console_unlock+0x106/0x120 [ 2326.317591] hardirqs last disabled at (12953858): [] console_unlock+0xeb/0x120 [ 2326.326372] softirqs last enabled at (12953872): [] __do_softirq+0x2be/0x371 [ 2326.335072] softirqs last disabled at (12953867): [] irq_exit_rcu+0xb0/0xd0 [ 2326.343593] ---[ end trace 0000000000000000 ]--- [ 2326.348212] ------------[ cut here ]------------ [ 2326.352827] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2326.364477] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2326.373430] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2326.428890] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2326.437841] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2326.445320] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2326.450807] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2326.469552] RSP: 0018:ffffc90001fd7c10 EFLAGS: 00010286 [ 2326.474779] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2326.481910] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2326.489034] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007d00 [ 2326.496161] R10: 00000000002957e0 R11: ffffc90001fd7aa0 R12: ffff8881d10b0750 [ 2326.503293] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2326.510424] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2326.518504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2326.524248] CR2: 00007fb38cf0a000 CR3: 0000000120c1e001 CR4: 00000000003706e0 [ 2326.531373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2326.538507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2326.545640] Call Trace: [ 2326.548091] [ 2326.550190] ? __warn+0x85/0x180 [ 2326.553423] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.558303] ? report_bug+0xfc/0x1d0 [ 2326.561880] ? prb_read_valid+0x1b/0x20 [ 2326.565723] ? handle_bug+0x44/0x70 [ 2326.569214] ? exc_invalid_op+0x17/0x60 [ 2326.573051] ? asm_exc_invalid_op+0x1a/0x20 [ 2326.577242] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.582120] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2326.587018] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2326.592252] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2326.597053] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2326.601766] ? __d_lookup+0xd6/0x200 [ 2326.605346] lookup_fast+0x6d/0xf0 [ 2326.608750] path_openat+0xe4/0x9d0 [ 2326.612248] do_filp_open+0xb8/0x120 [ 2326.615827] ? lock_release+0xc5/0x260 [ 2326.619579] ? _raw_spin_unlock+0x23/0x40 [ 2326.623588] ? alloc_fd+0x13f/0x200 [ 2326.627086] do_sys_openat2+0x249/0x290 [ 2326.630921] do_sys_open+0x4f/0x80 [ 2326.634329] do_syscall_64+0x38/0x80 [ 2326.637907] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2326.642958] RIP: 0033:0x7f48f0ef2805 [ 2326.646539] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2326.665284] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2326.672850] RAX: ffffffffffffffda RBX: 000000000207815f RCX: 00007f48f0ef2805 [ 2326.679974] RDX: 0000000000090800 RSI: 0000000002078140 RDI: 00000000ffffff9c [ 2326.687097] RBP: 0000000002078140 R08: 0000000002076674 R09: 0000000000000000 [ 2326.694224] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2326.701356] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2326.708494] [ 2326.710681] irq event stamp: 13005775 [ 2326.714337] hardirqs last enabled at (13005785): [] console_unlock+0x106/0x120 [ 2326.723205] hardirqs last disabled at (13005794): [] console_unlock+0xeb/0x120 [ 2326.731984] softirqs last enabled at (13005742): [] __do_softirq+0x2be/0x371 [ 2326.740676] softirqs last disabled at (13005737): [] irq_exit_rcu+0xb0/0xd0 [ 2326.749197] ---[ end trace 0000000000000000 ]--- [ 2326.753816] ------------[ cut here ]------------ [ 2326.758436] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2326.770098] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2326.778794] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2326.834290] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2326.843428] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2326.850920] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2326.856439] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2326.861943] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2326.880708] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2326.885950] RAX: 0000000000000000 RBX: ffff8881c5351390 RCX: 0000000000000000 [ 2326.893103] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2326.900253] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007d3b [ 2326.907384] R10: 0000000000296878 R11: ffffc9000030b828 R12: ffff8881c5351ae0 [ 2326.914518] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2326.921667] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2326.929771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2326.935533] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2326.942677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2326.949845] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2326.956977] Call Trace: [ 2326.959445] [ 2326.961561] ? __warn+0x85/0x180 [ 2326.964832] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.969708] ? report_bug+0xfc/0x1d0 [ 2326.973305] ? prb_read_valid+0x1b/0x20 [ 2326.977151] ? handle_bug+0x44/0x70 [ 2326.980645] ? exc_invalid_op+0x17/0x60 [ 2326.984502] ? asm_exc_invalid_op+0x1a/0x20 [ 2326.988726] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.993603] ? __list_add_valid_or_report+0x7d/0xa0 [ 2326.998500] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2327.003422] ceph_add_cap+0x20b/0x620 [ceph] [ 2327.007744] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2327.012484] ? lock_acquire+0xcb/0x2b0 [ 2327.016236] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2327.020695] ? __mutex_lock+0x97/0xce0 [ 2327.024463] ? __mutex_lock+0xaec/0xce0 [ 2327.028316] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2327.032862] ? lock_downgrade+0xfd/0x250 [ 2327.036827] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2327.041200] ceph_con_process_message+0x74/0x150 [libceph] [ 2327.046695] ? calc_signature+0xdb/0x100 [libceph] [ 2327.051516] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2327.056873] ? process_scheduled_works+0x22a/0x510 [ 2327.061682] ? process_scheduled_works+0x22a/0x510 [ 2327.066472] ceph_con_workfn+0x344/0x750 [libceph] [ 2327.071293] ? process_scheduled_works+0x22a/0x510 [ 2327.076083] process_scheduled_works+0x2a4/0x510 [ 2327.080744] worker_thread+0x188/0x340 [ 2327.084500] ? rescuer_thread+0x310/0x310 [ 2327.088529] kthread+0xf5/0x130 [ 2327.091695] ? kthread_complete_and_exit+0x20/0x20 [ 2327.096504] ret_from_fork+0x31/0x40 [ 2327.100100] ? kthread_complete_and_exit+0x20/0x20 [ 2327.104909] ret_from_fork_asm+0x11/0x20 [ 2327.108876] [ 2327.111062] irq event stamp: 10666189 [ 2327.114738] hardirqs last enabled at (10666199): [] console_unlock+0x106/0x120 [ 2327.123621] hardirqs last disabled at (10666208): [] console_unlock+0xeb/0x120 [ 2327.132433] softirqs last enabled at (10666136): [] __do_softirq+0x2be/0x371 [ 2327.141152] softirqs last disabled at (10666131): [] irq_exit_rcu+0xb0/0xd0 [ 2327.149675] ---[ end trace 0000000000000000 ]--- [ 2327.154452] ------------[ cut here ]------------ [ 2327.159068] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2327.170747] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2327.179437] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2327.234952] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2327.244072] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2327.251562] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2327.257072] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2327.262575] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2327.281353] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2327.286578] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2327.293727] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2327.300903] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007d7a [ 2327.308058] R10: 00000000002979c8 R11: ffffc9000030b828 R12: ffff8881c5357cb0 [ 2327.315185] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2327.322319] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2327.330403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2327.336155] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2327.343282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2327.350416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2327.357549] Call Trace: [ 2327.360018] [ 2327.362156] ? __warn+0x85/0x180 [ 2327.365385] ? __list_add_valid_or_report+0x7d/0xa0 [ 2327.370263] ? report_bug+0xfc/0x1d0 [ 2327.373842] ? prb_read_valid+0x1b/0x20 [ 2327.377700] ? handle_bug+0x44/0x70 [ 2327.381210] ? exc_invalid_op+0x17/0x60 [ 2327.385048] ? asm_exc_invalid_op+0x1a/0x20 [ 2327.389256] ? __list_add_valid_or_report+0x7d/0xa0 [ 2327.394150] ? __list_add_valid_or_report+0x7d/0xa0 [ 2327.399046] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2327.403936] ceph_add_cap+0x20b/0x620 [ceph] [ 2327.408237] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2327.412984] ceph_fill_trace+0x143/0xf30 [ceph] [ 2327.417524] ? __mutex_lock+0x97/0xce0 [ 2327.421291] ? __mutex_lock+0xaec/0xce0 [ 2327.425151] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2327.429693] ? lock_downgrade+0xfd/0x250 [ 2327.433656] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2327.438053] ceph_con_process_message+0x74/0x150 [libceph] [ 2327.443548] ? calc_signature+0xdb/0x100 [libceph] [ 2327.448370] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2327.453707] ? process_scheduled_works+0x22a/0x510 [ 2327.458518] ? process_scheduled_works+0x22a/0x510 [ 2327.463307] ceph_con_workfn+0x344/0x750 [libceph] [ 2327.468114] ? process_scheduled_works+0x22a/0x510 [ 2327.472927] process_scheduled_works+0x2a4/0x510 [ 2327.477569] worker_thread+0x188/0x340 [ 2327.481319] ? rescuer_thread+0x310/0x310 [ 2327.485333] kthread+0xf5/0x130 [ 2327.488478] ? kthread_complete_and_exit+0x20/0x20 [ 2327.493291] ret_from_fork+0x31/0x40 [ 2327.496866] ? kthread_complete_and_exit+0x20/0x20 [ 2327.501677] ret_from_fork_asm+0x11/0x20 [ 2327.505645] [ 2327.507829] irq event stamp: 10667337 [ 2327.511504] hardirqs last enabled at (10667347): [] console_unlock+0x106/0x120 [ 2327.520389] hardirqs last disabled at (10667356): [] console_unlock+0xeb/0x120 [ 2327.529170] softirqs last enabled at (10667294): [] __do_softirq+0x2be/0x371 [ 2327.537861] softirqs last disabled at (10667289): [] irq_exit_rcu+0xb0/0xd0 [ 2327.546399] ---[ end trace 0000000000000000 ]--- [ 2327.551055] ------------[ cut here ]------------ [ 2327.555671] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2327.567336] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2327.576032] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2327.631524] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2327.640667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2327.648165] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2327.653658] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2327.659162] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2327.677909] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2327.683164] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2327.690294] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2327.697443] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007db8 [ 2327.704575] R10: 0000000000298bd0 R11: ffffc9000030b828 R12: ffff8881d106cbc8 [ 2327.711717] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2327.718867] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2327.726970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2327.732725] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2327.739866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2327.747018] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2327.754168] Call Trace: [ 2327.756621] [ 2327.758745] ? __warn+0x85/0x180 [ 2327.761994] ? __list_add_valid_or_report+0x7d/0xa0 [ 2327.766893] ? report_bug+0xfc/0x1d0 [ 2327.770486] ? prb_read_valid+0x1b/0x20 [ 2327.774346] ? handle_bug+0x44/0x70 [ 2327.777837] ? exc_invalid_op+0x17/0x60 [ 2327.781693] ? asm_exc_invalid_op+0x1a/0x20 [ 2327.785920] ? __list_add_valid_or_report+0x7d/0xa0 [ 2327.790794] ? __list_add_valid_or_report+0x7d/0xa0 [ 2327.795691] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2327.800600] ceph_add_cap+0x20b/0x620 [ceph] [ 2327.804916] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2327.809648] ? lock_acquire+0xcb/0x2b0 [ 2327.813403] ceph_fill_trace+0x143/0xf30 [ceph] [ 2327.817949] ? __mutex_lock+0x97/0xce0 [ 2327.821718] ? __mutex_lock+0xaec/0xce0 [ 2327.825594] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2327.830159] ? lock_downgrade+0xfd/0x250 [ 2327.834106] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2327.838495] ceph_con_process_message+0x74/0x150 [libceph] [ 2327.843990] ? calc_signature+0xdb/0x100 [libceph] [ 2327.848828] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2327.854170] ? process_scheduled_works+0x22a/0x510 [ 2327.858963] ? process_scheduled_works+0x22a/0x510 [ 2327.863768] ceph_con_workfn+0x344/0x750 [libceph] [ 2327.868607] ? process_scheduled_works+0x22a/0x510 [ 2327.873397] process_scheduled_works+0x2a4/0x510 [ 2327.878039] worker_thread+0x188/0x340 [ 2327.881789] ? rescuer_thread+0x310/0x310 [ 2327.885816] kthread+0xf5/0x130 [ 2327.888983] ? kthread_complete_and_exit+0x20/0x20 [ 2327.893791] ret_from_fork+0x31/0x40 [ 2327.897387] ? kthread_complete_and_exit+0x20/0x20 [ 2327.902179] ret_from_fork_asm+0x11/0x20 [ 2327.906130] [ 2327.908315] irq event stamp: 10668449 [ 2327.911982] hardirqs last enabled at (10668459): [] console_unlock+0x106/0x120 [ 2327.920865] hardirqs last disabled at (10668468): [] console_unlock+0xeb/0x120 [ 2327.929663] softirqs last enabled at (10668396): [] __do_softirq+0x2be/0x371 [ 2327.938372] softirqs last disabled at (10668391): [] irq_exit_rcu+0xb0/0xd0 [ 2327.946892] ---[ end trace 0000000000000000 ]--- [ 2327.951563] ------------[ cut here ]------------ [ 2327.956184] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2327.967827] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2327.976526] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2328.032003] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2328.041126] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2328.048607] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2328.054101] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2328.059586] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2328.078332] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2328.083558] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2328.090692] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2328.097826] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007df7 [ 2328.104957] R10: 0000000000299ce8 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2328.112090] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2328.119223] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2328.127310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.133056] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2328.140189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2328.147320] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2328.154453] Call Trace: [ 2328.156905] [ 2328.159004] ? __warn+0x85/0x180 [ 2328.162237] ? __list_add_valid_or_report+0x7d/0xa0 [ 2328.167116] ? report_bug+0xfc/0x1d0 [ 2328.170696] ? prb_read_valid+0x1b/0x20 [ 2328.174536] ? handle_bug+0x44/0x70 [ 2328.178028] ? exc_invalid_op+0x17/0x60 [ 2328.181868] ? asm_exc_invalid_op+0x1a/0x20 [ 2328.186058] ? __list_add_valid_or_report+0x7d/0xa0 [ 2328.190933] ? __list_add_valid_or_report+0x7d/0xa0 [ 2328.195810] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2328.200702] ceph_add_cap+0x20b/0x620 [ceph] [ 2328.204987] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2328.209701] ? lock_acquire+0xcb/0x2b0 [ 2328.213453] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2328.217923] ? __mutex_lock+0x97/0xce0 [ 2328.221674] ? __mutex_lock+0xaec/0xce0 [ 2328.225515] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2328.230062] ? lock_downgrade+0xfd/0x250 [ 2328.233996] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2328.238383] ceph_con_process_message+0x74/0x150 [libceph] [ 2328.243880] ? calc_signature+0xdb/0x100 [libceph] [ 2328.248693] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2328.254022] ? process_scheduled_works+0x22a/0x510 [ 2328.258814] ? process_scheduled_works+0x22a/0x510 [ 2328.263605] ceph_con_workfn+0x344/0x750 [libceph] [ 2328.268408] ? process_scheduled_works+0x22a/0x510 [ 2328.273199] process_scheduled_works+0x2a4/0x510 [ 2328.277823] worker_thread+0x188/0x340 [ 2328.281581] ? rescuer_thread+0x310/0x310 [ 2328.285592] kthread+0xf5/0x130 [ 2328.288740] ? kthread_complete_and_exit+0x20/0x20 [ 2328.293532] ret_from_fork+0x31/0x40 [ 2328.297109] ? kthread_complete_and_exit+0x20/0x20 [ 2328.301902] ret_from_fork_asm+0x11/0x20 [ 2328.305835] [ 2328.308020] irq event stamp: 10669627 [ 2328.311688] hardirqs last enabled at (10669637): [] console_unlock+0x106/0x120 [ 2328.320553] hardirqs last disabled at (10669646): [] console_unlock+0xeb/0x120 [ 2328.329333] softirqs last enabled at (10669574): [] __do_softirq+0x2be/0x371 [ 2328.338026] softirqs last disabled at (10669569): [] irq_exit_rcu+0xb0/0xd0 [ 2328.346546] ---[ end trace 0000000000000000 ]--- [ 2328.351167] ------------[ cut here ]------------ [ 2328.355783] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2328.367432] WARNING: CPU: 6 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2328.376385] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2328.431844] CPU: 6 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2328.440796] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2328.448276] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2328.453763] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2328.472507] RSP: 0018:ffffc90001fcfaa0 EFLAGS: 00010286 [ 2328.477733] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2328.484856] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2328.491982] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007e36 [ 2328.499115] R10: 000000000029ae40 R11: ffffc90001fcf930 R12: ffff8881d106cbc8 [ 2328.506248] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2328.513381] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2328.521465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.527205] CR2: 00007f9bd22d7020 CR3: 000000011aeae004 CR4: 00000000003706e0 [ 2328.534336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2328.541460] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2328.548585] Call Trace: [ 2328.551040] [ 2328.553149] ? __warn+0x85/0x180 [ 2328.556377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2328.561257] ? report_bug+0xfc/0x1d0 [ 2328.564835] ? prb_read_valid+0x1b/0x20 [ 2328.568677] ? handle_bug+0x44/0x70 [ 2328.572167] ? exc_invalid_op+0x17/0x60 [ 2328.575998] ? asm_exc_invalid_op+0x1a/0x20 [ 2328.580188] ? __list_add_valid_or_report+0x7d/0xa0 [ 2328.585065] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2328.589954] ceph_check_caps+0x999/0xae0 [ceph] [ 2328.594500] ? __lock_acquire+0xbfa/0x1e20 [ 2328.598595] ? avc_has_perm+0x57/0x320 [ 2328.602347] ? lock_acquire+0xcb/0x2b0 [ 2328.606099] ? ceph_open+0x113/0x440 [ceph] [ 2328.610294] ? find_held_lock+0x2d/0x90 [ 2328.614133] ? ceph_open+0x175/0x440 [ceph] [ 2328.618327] ? lock_release+0xc5/0x260 [ 2328.622082] ceph_open+0x3cc/0x440 [ceph] [ 2328.626105] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2328.631249] do_dentry_open+0x172/0x530 [ 2328.635091] path_openat+0x743/0x9d0 [ 2328.638672] do_filp_open+0xb8/0x120 [ 2328.642251] ? lock_release+0xc5/0x260 [ 2328.646002] ? _raw_spin_unlock+0x23/0x40 [ 2328.650013] ? alloc_fd+0x13f/0x200 [ 2328.653508] do_sys_openat2+0x249/0x290 [ 2328.657346] do_sys_open+0x4f/0x80 [ 2328.660753] do_syscall_64+0x38/0x80 [ 2328.664332] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2328.669384] RIP: 0033:0x7f48f0ef2805 [ 2328.672963] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2328.691711] RSP: 002b:00007ffc53d5ff50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2328.699274] RAX: ffffffffffffffda RBX: 0000000002078089 RCX: 00007f48f0ef2805 [ 2328.706398] RDX: 0000000000090800 RSI: 0000000002078070 RDI: 00000000ffffff9c [ 2328.713525] RBP: 0000000002078070 R08: 00000000000007ff R09: 00000000020767c0 [ 2328.720656] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2328.727779] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2328.734911] [ 2328.737098] irq event stamp: 12918121 [ 2328.740764] hardirqs last enabled at (12918131): [] console_unlock+0x106/0x120 [ 2328.749630] hardirqs last disabled at (12918140): [] console_unlock+0xeb/0x120 [ 2328.758409] softirqs last enabled at (12918154): [] __do_softirq+0x2be/0x371 [ 2328.767103] softirqs last disabled at (12918149): [] irq_exit_rcu+0xb0/0xd0 [ 2328.775620] ---[ end trace 0000000000000000 ]--- [ 2328.780233] ------------[ cut here ]------------ [ 2328.784852] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2328.796515] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2328.805211] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2328.860730] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2328.869861] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2328.877355] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2328.882847] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2328.888352] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2328.907101] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2328.912340] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2328.919475] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2328.926625] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007e77 [ 2328.933777] R10: 000000000029c160 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2328.940924] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2328.948067] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2328.956185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2328.961950] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2328.969086] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2328.976235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2328.983365] Call Trace: [ 2328.985819] [ 2328.987943] ? __warn+0x85/0x180 [ 2328.991194] ? __list_add_valid_or_report+0x7d/0xa0 [ 2328.996072] ? report_bug+0xfc/0x1d0 [ 2328.999670] ? prb_read_valid+0x1b/0x20 [ 2329.003528] ? handle_bug+0x44/0x70 [ 2329.007036] ? exc_invalid_op+0x17/0x60 [ 2329.010893] ? asm_exc_invalid_op+0x1a/0x20 [ 2329.015118] ? __list_add_valid_or_report+0x7d/0xa0 [ 2329.019992] ? __list_add_valid_or_report+0x7d/0xa0 [ 2329.024890] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2329.029797] ceph_add_cap+0x20b/0x620 [ceph] [ 2329.034117] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2329.038862] ceph_fill_trace+0x143/0xf30 [ceph] [ 2329.043404] ? __mutex_lock+0x97/0xce0 [ 2329.047154] ? __mutex_lock+0xaec/0xce0 [ 2329.051012] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2329.055555] ? lock_downgrade+0xfd/0x250 [ 2329.059517] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2329.063916] ceph_con_process_message+0x74/0x150 [libceph] [ 2329.069409] ? calc_signature+0xdb/0x100 [libceph] [ 2329.074216] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2329.079563] ? process_scheduled_works+0x22a/0x510 [ 2329.084354] ? process_scheduled_works+0x22a/0x510 [ 2329.089144] ceph_con_workfn+0x344/0x750 [libceph] [ 2329.093973] ? process_scheduled_works+0x22a/0x510 [ 2329.098765] process_scheduled_works+0x2a4/0x510 [ 2329.103406] worker_thread+0x188/0x340 [ 2329.107177] ? rescuer_thread+0x310/0x310 [ 2329.111189] kthread+0xf5/0x130 [ 2329.114331] ? kthread_complete_and_exit+0x20/0x20 [ 2329.119125] ret_from_fork+0x31/0x40 [ 2329.122720] ? kthread_complete_and_exit+0x20/0x20 [ 2329.127531] ret_from_fork_asm+0x11/0x20 [ 2329.131501] [ 2329.133691] irq event stamp: 10671643 [ 2329.137367] hardirqs last enabled at (10671653): [] console_unlock+0x106/0x120 [ 2329.146233] hardirqs last disabled at (10671662): [] console_unlock+0xeb/0x120 [ 2329.155012] softirqs last enabled at (10671600): [] __do_softirq+0x2be/0x371 [ 2329.163722] softirqs last disabled at (10671595): [] irq_exit_rcu+0xb0/0xd0 [ 2329.172259] ---[ end trace 0000000000000000 ]--- [ 2329.177053] ------------[ cut here ]------------ [ 2329.181676] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2329.193334] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2329.202031] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2329.257525] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2329.266667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2329.274182] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2329.279675] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2329.285191] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2329.303934] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2329.309191] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2329.316319] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2329.323454] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007eb5 [ 2329.330585] R10: 000000000029d398 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2329.337734] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2329.344878] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2329.352981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2329.358742] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2329.365886] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2329.373035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2329.380192] Call Trace: [ 2329.382637] [ 2329.384753] ? __warn+0x85/0x180 [ 2329.388004] ? __list_add_valid_or_report+0x7d/0xa0 [ 2329.392900] ? report_bug+0xfc/0x1d0 [ 2329.396498] ? prb_read_valid+0x1b/0x20 [ 2329.400354] ? handle_bug+0x44/0x70 [ 2329.403845] ? exc_invalid_op+0x17/0x60 [ 2329.407695] ? asm_exc_invalid_op+0x1a/0x20 [ 2329.411918] ? __list_add_valid_or_report+0x7d/0xa0 [ 2329.416795] ? __list_add_valid_or_report+0x7d/0xa0 [ 2329.421692] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2329.426600] ceph_add_cap+0x20b/0x620 [ceph] [ 2329.430917] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2329.435650] ? lock_acquire+0xcb/0x2b0 [ 2329.439403] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2329.443852] ? __mutex_lock+0x97/0xce0 [ 2329.447641] ? __mutex_lock+0xaec/0xce0 [ 2329.451498] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2329.456038] ? lock_downgrade+0xfd/0x250 [ 2329.460002] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2329.464375] ceph_con_process_message+0x74/0x150 [libceph] [ 2329.469871] ? calc_signature+0xdb/0x100 [libceph] [ 2329.474701] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2329.480066] ? process_scheduled_works+0x22a/0x510 [ 2329.484874] ? process_scheduled_works+0x22a/0x510 [ 2329.489664] ceph_con_workfn+0x344/0x750 [libceph] [ 2329.494487] ? process_scheduled_works+0x22a/0x510 [ 2329.499293] process_scheduled_works+0x2a4/0x510 [ 2329.503935] worker_thread+0x188/0x340 [ 2329.506796] watchdog: BUG: soft lockup - CPU#7 stuck for 52s! [kworker/7:4:78808] [ 2329.507687] ? rescuer_thread+0x310/0x310 [ 2329.515149] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [ 2329.519193] kthread+0xf5/0x130 [ 2329.536894] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [ 2329.540051] ? kthread_complete_and_exit+0x20/0x20 [ 2329.540052] iTCO_vendor_support [ 2329.540056] ret_from_fork+0x31/0x40 [ 2329.553163] i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 [ 2329.557975] ? kthread_complete_and_exit+0x20/0x20 [ 2329.561197] acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe [ 2329.564778] ret_from_fork_asm+0x11/0x20 [ 2329.575712] igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca [ 2329.580524] [ 2329.590673] fuse [ 2329.590674] irq event stamp: 446844 [ 2329.590675] hardirqs last enabled at (446843): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2329.594609] irq event stamp: 10672783 [ 2329.601905] hardirqs last disabled at (446844): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2329.601908] softirqs last enabled at (446790): [] __do_softirq+0x2be/0x371 [ 2329.604116] hardirqs last enabled at (10672795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2329.606039] softirqs last disabled at (446785): [] irq_exit_rcu+0xb0/0xd0 [ 2329.606043] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2329.609541] hardirqs last disabled at (10672806): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2329.619525] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2329.619526] Workqueue: events delayed_work [ceph] [ 2329.623211] softirqs last enabled at (10672770): [] __do_softirq+0x2be/0x371 [ 2329.632759] [ 2329.632759] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [ 2329.632762] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [ 2329.641289] softirqs last disabled at (10672765): [] irq_exit_rcu+0xb0/0xd0 [ 2329.651444] RSP: 0018:ffffc90002117d80 EFLAGS: 00000202 [ 2329.651446] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 2329.651447] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [ 2329.651448] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 2329.651449] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2329.651450] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2329.659794] ---[ end trace 0000000000000000 ]--- [ 2329.669168] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2329.669187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2329.669188] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2329.669189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2329.669190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2329.678930] ------------[ cut here ]------------ [ 2329.686388] Call Trace: [ 2329.686389] [ 2329.686391] ? watchdog_timer_fn+0x1b4/0x220 [ 2329.686394] ? __hrtimer_run_queues+0x1d3/0x360 [ 2329.691096] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2329.699781] ? hrtimer_interrupt+0x104/0x240 [ 2329.701295] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2329.706765] ? __local_bh_enable+0x50/0x60 [ 2329.706770] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2329.725531] Modules linked in: [ 2329.734038] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2329.734041] [ 2329.734041] [ 2329.734042] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2329.739277] ceph [ 2329.746401] ? queued_spin_lock_slowpath+0x16/0x2e0 [ 2329.753533] libceph [ 2329.760657] do_raw_spin_lock+0xb5/0xc0 [ 2329.760661] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2329.767798] dns_resolver [ 2329.774923] delayed_work+0x1d8/0x280 [ceph] [ 2329.779550] fscache [ 2329.787628] ? process_scheduled_works+0x22a/0x510 [ 2329.793383] netfs [ 2329.800505] process_scheduled_works+0x2a4/0x510 [ 2329.800511] worker_thread+0x188/0x340 [ 2329.807649] veth [ 2329.814771] ? rescuer_thread+0x310/0x310 [ 2329.814774] kthread+0xf5/0x130 [ 2329.819400] nft_chain_nat [ 2329.821844] ? kthread_complete_and_exit+0x20/0x20 [ 2329.823874] xt_MASQUERADE [ 2329.828134] ret_from_fork+0x31/0x40 [ 2329.832679] nf_nat [ 2329.844308] ? kthread_complete_and_exit+0x20/0x20 [ 2329.844311] ret_from_fork_asm+0x11/0x20 [ 2329.848583] nf_conntrack [ 2329.857268] [ 2329.861366] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2330.010621] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2330.019764] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2330.027260] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2330.032754] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2330.038257] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2330.057004] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2330.062247] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2330.069380] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2330.076513] RBP: ffff88817f867000 R08: 0000000000000001 R09: 0000000000000001 [ 2330.083663] R10: ffffffff811806bf R11: 0000000000000001 R12: ffff8881d1329ae0 [ 2330.090813] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2330.097964] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2330.106067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2330.111828] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2330.118971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2330.126121] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2330.133264] Call Trace: [ 2330.135715] [ 2330.137831] ? __warn+0x85/0x180 [ 2330.141083] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.145978] ? report_bug+0xfc/0x1d0 [ 2330.149575] ? handle_bug+0x44/0x70 [ 2330.153083] ? exc_invalid_op+0x17/0x60 [ 2330.156931] ? asm_exc_invalid_op+0x1a/0x20 [ 2330.161138] ? console_flush_all+0x1ff/0x5a0 [ 2330.165428] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.170307] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.175202] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2330.180093] ceph_add_cap+0x20b/0x620 [ceph] [ 2330.184395] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2330.189140] ceph_fill_trace+0x143/0xf30 [ceph] [ 2330.193681] ? __mutex_lock+0x97/0xce0 [ 2330.197449] ? __mutex_lock+0xaec/0xce0 [ 2330.201290] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2330.205831] ? lock_downgrade+0xfd/0x250 [ 2330.209795] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2330.214202] ceph_con_process_message+0x74/0x150 [libceph] [ 2330.219696] ? calc_signature+0xdb/0x100 [libceph] [ 2330.224501] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2330.229857] ? process_scheduled_works+0x22a/0x510 [ 2330.234667] ? process_scheduled_works+0x22a/0x510 [ 2330.239457] ceph_con_workfn+0x344/0x750 [libceph] [ 2330.244261] ? process_scheduled_works+0x22a/0x510 [ 2330.249052] process_scheduled_works+0x2a4/0x510 [ 2330.253711] worker_thread+0x188/0x340 [ 2330.257460] ? rescuer_thread+0x310/0x310 [ 2330.261487] kthread+0xf5/0x130 [ 2330.264653] ? kthread_complete_and_exit+0x20/0x20 [ 2330.269481] ret_from_fork+0x31/0x40 [ 2330.273077] ? kthread_complete_and_exit+0x20/0x20 [ 2330.277886] ret_from_fork_asm+0x11/0x20 [ 2330.281854] [ 2330.284041] irq event stamp: 10674071 [ 2330.287723] hardirqs last enabled at (10674081): [] console_unlock+0x106/0x120 [ 2330.296606] hardirqs last disabled at (10674090): [] console_unlock+0xeb/0x120 [ 2330.305404] softirqs last enabled at (10674028): [] __do_softirq+0x2be/0x371 [ 2330.314097] softirqs last disabled at (10674023): [] irq_exit_rcu+0xb0/0xd0 [ 2330.322634] ---[ end trace 0000000000000000 ]--- [ 2330.327290] ------------[ cut here ]------------ [ 2330.331909] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2330.343587] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2330.352282] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2330.407762] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2330.416903] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2330.424398] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2330.429893] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2330.435397] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2330.454145] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2330.459387] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2330.466520] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2330.473673] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007f74 [ 2330.480821] R10: 00000000002a08e0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2330.487971] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2330.495121] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2330.503225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2330.508970] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2330.516121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2330.523271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2330.530403] Call Trace: [ 2330.532854] [ 2330.534970] ? __warn+0x85/0x180 [ 2330.538223] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.543102] ? report_bug+0xfc/0x1d0 [ 2330.546697] ? prb_read_valid+0x1b/0x20 [ 2330.550556] ? handle_bug+0x44/0x70 [ 2330.554064] ? exc_invalid_op+0x17/0x60 [ 2330.557920] ? asm_exc_invalid_op+0x1a/0x20 [ 2330.562145] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.567022] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.571918] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2330.576826] ceph_add_cap+0x20b/0x620 [ceph] [ 2330.581146] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2330.585884] ? lock_acquire+0xcb/0x2b0 [ 2330.589655] ceph_fill_trace+0x143/0xf30 [ceph] [ 2330.594213] ? __mutex_lock+0x97/0xce0 [ 2330.597962] ? __mutex_lock+0xaec/0xce0 [ 2330.601837] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2330.606386] ? lock_downgrade+0xfd/0x250 [ 2330.610317] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2330.614713] ceph_con_process_message+0x74/0x150 [libceph] [ 2330.620216] ? calc_signature+0xdb/0x100 [libceph] [ 2330.625040] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2330.630370] ? process_scheduled_works+0x22a/0x510 [ 2330.635162] ? process_scheduled_works+0x22a/0x510 [ 2330.639976] ceph_con_workfn+0x344/0x750 [libceph] [ 2330.644783] ? process_scheduled_works+0x22a/0x510 [ 2330.649589] process_scheduled_works+0x2a4/0x510 [ 2330.654231] worker_thread+0x188/0x340 [ 2330.657979] ? rescuer_thread+0x310/0x310 [ 2330.662012] kthread+0xf5/0x130 [ 2330.665175] ? kthread_complete_and_exit+0x20/0x20 [ 2330.669991] ret_from_fork+0x31/0x40 [ 2330.673571] ? kthread_complete_and_exit+0x20/0x20 [ 2330.678382] ret_from_fork_asm+0x11/0x20 [ 2330.682315] [ 2330.684506] irq event stamp: 10675187 [ 2330.688205] hardirqs last enabled at (10675197): [] console_unlock+0x106/0x120 [ 2330.697075] hardirqs last disabled at (10675206): [] console_unlock+0xeb/0x120 [ 2330.705872] softirqs last enabled at (10675132): [] __do_softirq+0x2be/0x371 [ 2330.714582] softirqs last disabled at (10675127): [] irq_exit_rcu+0xb0/0xd0 [ 2330.723119] ---[ end trace 0000000000000000 ]--- [ 2330.728534] ------------[ cut here ]------------ [ 2330.733319] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2330.744987] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2330.753679] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2330.809174] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2330.818322] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2330.825795] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2330.831305] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2330.836791] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2330.855556] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2330.860799] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2330.867951] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2330.875102] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007fb3 [ 2330.882250] R10: 00000000002a19d0 R11: ffffc9000030b828 R12: ffff88819f02f2e8 [ 2330.889383] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2330.896516] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2330.904620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2330.910383] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2330.917518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2330.924665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2330.931816] Call Trace: [ 2330.934278] [ 2330.936385] ? __warn+0x85/0x180 [ 2330.939617] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.944514] ? report_bug+0xfc/0x1d0 [ 2330.948109] ? prb_read_valid+0x1b/0x20 [ 2330.951968] ? handle_bug+0x44/0x70 [ 2330.955493] ? exc_invalid_op+0x17/0x60 [ 2330.959350] ? asm_exc_invalid_op+0x1a/0x20 [ 2330.963561] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.968446] ? __list_add_valid_or_report+0x7d/0xa0 [ 2330.973321] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2330.978225] ceph_add_cap+0x20b/0x620 [ceph] [ 2330.982534] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2330.987256] ? lock_acquire+0xcb/0x2b0 [ 2330.991031] ceph_fill_trace+0x143/0xf30 [ceph] [ 2330.995570] ? __mutex_lock+0x97/0xce0 [ 2330.999340] ? __mutex_lock+0xaec/0xce0 [ 2331.003202] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2331.007745] ? lock_downgrade+0xfd/0x250 [ 2331.011693] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2331.016083] ceph_con_process_message+0x74/0x150 [libceph] [ 2331.021579] ? calc_signature+0xdb/0x100 [libceph] [ 2331.026399] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2331.031737] ? process_scheduled_works+0x22a/0x510 [ 2331.036532] ? process_scheduled_works+0x22a/0x510 [ 2331.041337] ceph_con_workfn+0x344/0x750 [libceph] [ 2331.046141] ? process_scheduled_works+0x22a/0x510 [ 2331.050951] process_scheduled_works+0x2a4/0x510 [ 2331.055608] worker_thread+0x188/0x340 [ 2331.059359] ? rescuer_thread+0x310/0x310 [ 2331.063369] kthread+0xf5/0x130 [ 2331.066517] ? kthread_complete_and_exit+0x20/0x20 [ 2331.071327] ret_from_fork+0x31/0x40 [ 2331.074936] ? kthread_complete_and_exit+0x20/0x20 [ 2331.079765] ret_from_fork_asm+0x11/0x20 [ 2331.083717] [ 2331.085930] irq event stamp: 10677277 [ 2331.089596] hardirqs last enabled at (10677295): [] console_unlock+0x106/0x120 [ 2331.098494] hardirqs last disabled at (10677304): [] console_unlock+0xeb/0x120 [ 2331.107276] softirqs last enabled at (10677292): [] __do_softirq+0x2be/0x371 [ 2331.115968] softirqs last disabled at (10677287): [] irq_exit_rcu+0xb0/0xd0 [ 2331.124512] ---[ end trace 0000000000000000 ]--- [ 2331.129182] ------------[ cut here ]------------ [ 2331.133800] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2331.145441] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2331.154139] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2331.209624] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2331.218750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2331.226229] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2331.231724] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2331.237208] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2331.255954] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2331.261179] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2331.268306] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2331.275440] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100007ff2 [ 2331.282571] R10: 00000000002a2c40 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2331.289705] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2331.296836] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2331.304923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2331.310669] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2331.317802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2331.324935] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2331.332066] Call Trace: [ 2331.334521] [ 2331.336625] ? __warn+0x85/0x180 [ 2331.339861] ? __list_add_valid_or_report+0x7d/0xa0 [ 2331.344739] ? report_bug+0xfc/0x1d0 [ 2331.348318] ? prb_read_valid+0x1b/0x20 [ 2331.352158] ? handle_bug+0x44/0x70 [ 2331.355649] ? exc_invalid_op+0x17/0x60 [ 2331.359490] ? asm_exc_invalid_op+0x1a/0x20 [ 2331.363680] ? __list_add_valid_or_report+0x7d/0xa0 [ 2331.368556] ? __list_add_valid_or_report+0x7d/0xa0 [ 2331.373435] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2331.378324] ceph_add_cap+0x20b/0x620 [ceph] [ 2331.382609] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2331.387324] ? lock_acquire+0xcb/0x2b0 [ 2331.391077] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2331.395536] ? __mutex_lock+0x97/0xce0 [ 2331.399287] ? __mutex_lock+0xaec/0xce0 [ 2331.403130] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2331.407678] ? lock_downgrade+0xfd/0x250 [ 2331.411608] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2331.415989] ceph_con_process_message+0x74/0x150 [libceph] [ 2331.421482] ? calc_signature+0xdb/0x100 [libceph] [ 2331.426287] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2331.431618] ? process_scheduled_works+0x22a/0x510 [ 2331.436409] ? process_scheduled_works+0x22a/0x510 [ 2331.441203] ceph_con_workfn+0x344/0x750 [libceph] [ 2331.446011] ? process_scheduled_works+0x22a/0x510 [ 2331.450802] process_scheduled_works+0x2a4/0x510 [ 2331.455429] worker_thread+0x188/0x340 [ 2331.459184] ? rescuer_thread+0x310/0x310 [ 2331.463197] kthread+0xf5/0x130 [ 2331.466351] ? kthread_complete_and_exit+0x20/0x20 [ 2331.471145] ret_from_fork+0x31/0x40 [ 2331.474722] ? kthread_complete_and_exit+0x20/0x20 [ 2331.479517] ret_from_fork_asm+0x11/0x20 [ 2331.483449] [ 2331.485643] irq event stamp: 10678403 [ 2331.489309] hardirqs last enabled at (10678413): [] console_unlock+0x106/0x120 [ 2331.498178] hardirqs last disabled at (10678436): [] console_unlock+0xeb/0x120 [ 2331.506956] softirqs last enabled at (10678434): [] __do_softirq+0x2be/0x371 [ 2331.515647] softirqs last disabled at (10678423): [] irq_exit_rcu+0xb0/0xd0 [ 2331.524169] ---[ end trace 0000000000000000 ]--- [ 2331.528786] ------------[ cut here ]------------ [ 2331.533406] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2331.545052] WARNING: CPU: 6 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2331.554007] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2331.609467] CPU: 6 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2331.618416] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2331.625888] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2331.631374] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2331.650121] RSP: 0018:ffffc90001fefb78 EFLAGS: 00010286 [ 2331.655348] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2331.662479] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2331.669605] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008031 [ 2331.676738] R10: 00000000002a3e88 R11: ffffc90001fefa08 R12: ffff88819f02f2e8 [ 2331.683869] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2331.690995] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2331.699080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2331.704825] CR2: 00007f9bd22d7020 CR3: 0000000108c72001 CR4: 00000000003706e0 [ 2331.711951] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2331.719083] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2331.726216] Call Trace: [ 2331.728661] [ 2331.730767] ? __warn+0x85/0x180 [ 2331.733999] ? __list_add_valid_or_report+0x7d/0xa0 [ 2331.738878] ? report_bug+0xfc/0x1d0 [ 2331.742458] ? prb_read_valid+0x1b/0x20 [ 2331.746299] ? handle_bug+0x44/0x70 [ 2331.749790] ? exc_invalid_op+0x17/0x60 [ 2331.753629] ? asm_exc_invalid_op+0x1a/0x20 [ 2331.757818] ? __list_add_valid_or_report+0x7d/0xa0 [ 2331.762696] ? __list_add_valid_or_report+0x7d/0xa0 [ 2331.767575] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2331.772464] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2331.777699] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2331.782500] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2331.787215] ? __d_lookup+0xd6/0x200 [ 2331.790794] lookup_fast+0x6d/0xf0 [ 2331.794198] walk_component+0x26/0x160 [ 2331.797952] link_path_walk+0x24d/0x380 [ 2331.801792] ? path_openat+0x80/0x9d0 [ 2331.805462] path_openat+0x8e/0x9d0 [ 2331.808953] do_filp_open+0xb8/0x120 [ 2331.812533] ? lock_release+0xc5/0x260 [ 2331.816283] ? _raw_spin_unlock+0x23/0x40 [ 2331.820295] ? alloc_fd+0x13f/0x200 [ 2331.823790] do_sys_openat2+0x249/0x290 [ 2331.827627] do_sys_open+0x4f/0x80 [ 2331.831032] do_syscall_64+0x38/0x80 [ 2331.834612] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2331.839664] RIP: 0033:0x7f48f0eed2f2 [ 2331.843246] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2331.861991] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2331.869559] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2331.876688] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2331.883812] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2331.890936] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2331.898063] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2331.905202] [ 2331.907387] irq event stamp: 12893531 [ 2331.911043] hardirqs last enabled at (12893541): [] console_unlock+0x106/0x120 [ 2331.919910] hardirqs last disabled at (12893550): [] console_unlock+0xeb/0x120 [ 2331.928691] softirqs last enabled at (12893518): [] __do_softirq+0x2be/0x371 [ 2331.937385] softirqs last disabled at (12893569): [] irq_exit_rcu+0xb0/0xd0 [ 2331.945904] ---[ end trace 0000000000000000 ]--- [ 2331.950523] ------------[ cut here ]------------ [ 2331.955141] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2331.966798] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2331.975511] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2332.031004] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2332.040148] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2332.047645] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2332.053157] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2332.058659] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2332.077424] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2332.082649] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2332.089799] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2332.096949] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008070 [ 2332.104100] R10: 00000000002a4fa0 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2332.111250] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2332.118382] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2332.126469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.132232] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2332.139365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2332.146514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2332.153666] Call Trace: [ 2332.156134] [ 2332.158250] ? __warn+0x85/0x180 [ 2332.161483] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.166365] ? report_bug+0xfc/0x1d0 [ 2332.169941] ? prb_read_valid+0x1b/0x20 [ 2332.173799] ? handle_bug+0x44/0x70 [ 2332.177308] ? exc_invalid_op+0x17/0x60 [ 2332.181147] ? asm_exc_invalid_op+0x1a/0x20 [ 2332.185355] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.190243] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.195117] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2332.200027] ceph_add_cap+0x20b/0x620 [ceph] [ 2332.204328] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2332.209059] ? lock_acquire+0xcb/0x2b0 [ 2332.212828] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2332.217279] ? __mutex_lock+0x97/0xce0 [ 2332.221032] ? __mutex_lock+0xaec/0xce0 [ 2332.224889] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2332.229450] ? lock_downgrade+0xfd/0x250 [ 2332.233378] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2332.237775] ceph_con_process_message+0x74/0x150 [libceph] [ 2332.243271] ? calc_signature+0xdb/0x100 [libceph] [ 2332.248075] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2332.253415] ? process_scheduled_works+0x22a/0x510 [ 2332.258228] ? process_scheduled_works+0x22a/0x510 [ 2332.263022] ceph_con_workfn+0x344/0x750 [libceph] [ 2332.267826] ? process_scheduled_works+0x22a/0x510 [ 2332.272634] process_scheduled_works+0x2a4/0x510 [ 2332.277276] worker_thread+0x188/0x340 [ 2332.281023] ? rescuer_thread+0x310/0x310 [ 2332.285053] kthread+0xf5/0x130 [ 2332.288235] ? kthread_complete_and_exit+0x20/0x20 [ 2332.293031] ret_from_fork+0x31/0x40 [ 2332.296625] ? kthread_complete_and_exit+0x20/0x20 [ 2332.301433] ret_from_fork_asm+0x11/0x20 [ 2332.305368] [ 2332.307562] irq event stamp: 10680427 [ 2332.311245] hardirqs last enabled at (10680437): [] console_unlock+0x106/0x120 [ 2332.320112] hardirqs last disabled at (10680446): [] console_unlock+0xeb/0x120 [ 2332.328908] softirqs last enabled at (10680384): [] __do_softirq+0x2be/0x371 [ 2332.337619] softirqs last disabled at (10680379): [] irq_exit_rcu+0xb0/0xd0 [ 2332.346154] ---[ end trace 0000000000000000 ]--- [ 2332.350929] ------------[ cut here ]------------ [ 2332.355550] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2332.367232] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2332.375925] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2332.431402] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2332.440529] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2332.448025] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2332.453540] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2332.459038] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2332.477796] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2332.483038] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2332.490189] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2332.497339] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000080af [ 2332.504473] R10: 00000000002a60b8 R11: ffffc9000030b828 R12: ffff8881c2cefcb0 [ 2332.511604] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2332.518746] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2332.526849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.532613] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2332.539763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2332.546915] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2332.554064] Call Trace: [ 2332.556533] [ 2332.558649] ? __warn+0x85/0x180 [ 2332.561899] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.566797] ? report_bug+0xfc/0x1d0 [ 2332.570391] ? prb_read_valid+0x1b/0x20 [ 2332.574250] ? handle_bug+0x44/0x70 [ 2332.577740] ? exc_invalid_op+0x17/0x60 [ 2332.581590] ? asm_exc_invalid_op+0x1a/0x20 [ 2332.585815] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.590689] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.595585] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2332.600495] ceph_add_cap+0x20b/0x620 [ceph] [ 2332.604796] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2332.609549] ceph_fill_trace+0x143/0xf30 [ceph] [ 2332.614091] ? __mutex_lock+0x97/0xce0 [ 2332.617859] ? __mutex_lock+0xaec/0xce0 [ 2332.621708] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2332.626268] ? lock_downgrade+0xfd/0x250 [ 2332.630214] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2332.634593] ceph_con_process_message+0x74/0x150 [libceph] [ 2332.640091] ? calc_signature+0xdb/0x100 [libceph] [ 2332.644911] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2332.650252] ? process_scheduled_works+0x22a/0x510 [ 2332.655043] ? process_scheduled_works+0x22a/0x510 [ 2332.659850] ceph_con_workfn+0x344/0x750 [libceph] [ 2332.664670] ? process_scheduled_works+0x22a/0x510 [ 2332.669480] process_scheduled_works+0x2a4/0x510 [ 2332.674119] worker_thread+0x188/0x340 [ 2332.677870] ? rescuer_thread+0x310/0x310 [ 2332.681899] kthread+0xf5/0x130 [ 2332.685064] ? kthread_complete_and_exit+0x20/0x20 [ 2332.689876] ret_from_fork+0x31/0x40 [ 2332.693469] ? kthread_complete_and_exit+0x20/0x20 [ 2332.698263] ret_from_fork_asm+0x11/0x20 [ 2332.702212] [ 2332.704398] irq event stamp: 10681489 [ 2332.708064] hardirqs last enabled at (10681499): [] console_unlock+0x106/0x120 [ 2332.716947] hardirqs last disabled at (10681508): [] console_unlock+0xeb/0x120 [ 2332.725743] softirqs last enabled at (10681446): [] __do_softirq+0x2be/0x371 [ 2332.734455] softirqs last disabled at (10681441): [] irq_exit_rcu+0xb0/0xd0 [ 2332.742973] ---[ end trace 0000000000000000 ]--- [ 2332.747636] ------------[ cut here ]------------ [ 2332.752259] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2332.763916] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2332.772606] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2332.828101] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2332.837255] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2332.844732] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2332.850254] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2332.855737] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2332.874493] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2332.879720] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2332.886869] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2332.894049] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000080ed [ 2332.901204] R10: 00000000002a71a8 R11: ffffc9000030b828 R12: ffff8881c5357cb0 [ 2332.908343] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2332.915477] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2332.923564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.929327] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2332.936462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2332.943594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2332.950735] Call Trace: [ 2332.953195] [ 2332.955313] ? __warn+0x85/0x180 [ 2332.958545] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.963443] ? report_bug+0xfc/0x1d0 [ 2332.967021] ? prb_read_valid+0x1b/0x20 [ 2332.970878] ? handle_bug+0x44/0x70 [ 2332.974387] ? exc_invalid_op+0x17/0x60 [ 2332.978247] ? asm_exc_invalid_op+0x1a/0x20 [ 2332.982434] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.987311] ? __list_add_valid_or_report+0x7d/0xa0 [ 2332.992189] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2332.997096] ceph_add_cap+0x20b/0x620 [ceph] [ 2333.001398] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2333.006130] ? lock_acquire+0xcb/0x2b0 [ 2333.009900] ceph_fill_trace+0x143/0xf30 [ceph] [ 2333.014437] ? __mutex_lock+0x97/0xce0 [ 2333.018188] ? __mutex_lock+0xaec/0xce0 [ 2333.022057] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2333.026595] ? lock_downgrade+0xfd/0x250 [ 2333.030543] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2333.034950] ceph_con_process_message+0x74/0x150 [libceph] [ 2333.040445] ? calc_signature+0xdb/0x100 [libceph] [ 2333.045259] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2333.050606] ? process_scheduled_works+0x22a/0x510 [ 2333.055399] ? process_scheduled_works+0x22a/0x510 [ 2333.060191] ceph_con_workfn+0x344/0x750 [libceph] [ 2333.065009] ? process_scheduled_works+0x22a/0x510 [ 2333.069817] process_scheduled_works+0x2a4/0x510 [ 2333.074460] worker_thread+0x188/0x340 [ 2333.078207] ? rescuer_thread+0x310/0x310 [ 2333.082253] kthread+0xf5/0x130 [ 2333.085402] ? kthread_complete_and_exit+0x20/0x20 [ 2333.090195] ret_from_fork+0x31/0x40 [ 2333.093790] ? kthread_complete_and_exit+0x20/0x20 [ 2333.098600] ret_from_fork_asm+0x11/0x20 [ 2333.102566] [ 2333.104752] irq event stamp: 10682677 [ 2333.108429] hardirqs last enabled at (10682687): [] console_unlock+0x106/0x120 [ 2333.117295] hardirqs last disabled at (10682696): [] console_unlock+0xeb/0x120 [ 2333.126074] softirqs last enabled at (10682624): [] __do_softirq+0x2be/0x371 [ 2333.134785] softirqs last disabled at (10682619): [] irq_exit_rcu+0xb0/0xd0 [ 2333.143321] ---[ end trace 0000000000000000 ]--- [ 2333.147975] ------------[ cut here ]------------ [ 2333.152597] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2333.164241] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2333.172935] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2333.228413] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2333.237538] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2333.245017] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2333.250512] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2333.255996] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2333.274744] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2333.279969] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2333.287105] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2333.294239] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000812c [ 2333.301368] R10: 00000000002a82c0 R11: ffffc9000030b828 R12: ffff8881d106cbc8 [ 2333.308503] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2333.315634] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2333.323720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2333.329465] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2333.336590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2333.343723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2333.350856] Call Trace: [ 2333.353308] [ 2333.355415] ? __warn+0x85/0x180 [ 2333.358650] ? __list_add_valid_or_report+0x7d/0xa0 [ 2333.363528] ? report_bug+0xfc/0x1d0 [ 2333.367107] ? prb_read_valid+0x1b/0x20 [ 2333.370947] ? handle_bug+0x44/0x70 [ 2333.374439] ? exc_invalid_op+0x17/0x60 [ 2333.378279] ? asm_exc_invalid_op+0x1a/0x20 [ 2333.382471] ? __list_add_valid_or_report+0x7d/0xa0 [ 2333.387354] ? __list_add_valid_or_report+0x7d/0xa0 [ 2333.392237] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2333.397130] ceph_add_cap+0x20b/0x620 [ceph] [ 2333.401417] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2333.406139] ? lock_acquire+0xcb/0x2b0 [ 2333.409892] ceph_fill_trace+0x143/0xf30 [ceph] [ 2333.414430] ? __mutex_lock+0x97/0xce0 [ 2333.418180] ? __mutex_lock+0xaec/0xce0 [ 2333.422022] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2333.426561] ? lock_downgrade+0xfd/0x250 [ 2333.430493] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2333.434871] ceph_con_process_message+0x74/0x150 [libceph] [ 2333.440366] ? calc_signature+0xdb/0x100 [libceph] [ 2333.445173] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2333.450503] ? process_scheduled_works+0x22a/0x510 [ 2333.455294] ? process_scheduled_works+0x22a/0x510 [ 2333.460084] ceph_con_workfn+0x344/0x750 [libceph] [ 2333.464889] ? process_scheduled_works+0x22a/0x510 [ 2333.469679] process_scheduled_works+0x2a4/0x510 [ 2333.474303] worker_thread+0x188/0x340 [ 2333.478051] ? rescuer_thread+0x310/0x310 [ 2333.482064] kthread+0xf5/0x130 [ 2333.485212] ? kthread_complete_and_exit+0x20/0x20 [ 2333.490003] ret_from_fork+0x31/0x40 [ 2333.493580] ? kthread_complete_and_exit+0x20/0x20 [ 2333.498376] ret_from_fork_asm+0x11/0x20 [ 2333.502307] [ 2333.504494] irq event stamp: 10683783 [ 2333.508160] hardirqs last enabled at (10683793): [] console_unlock+0x106/0x120 [ 2333.517025] hardirqs last disabled at (10683802): [] console_unlock+0xeb/0x120 [ 2333.525804] softirqs last enabled at (10683730): [] __do_softirq+0x2be/0x371 [ 2333.534498] softirqs last disabled at (10683725): [] irq_exit_rcu+0xb0/0xd0 [ 2333.543023] ---[ end trace 0000000000000000 ]--- [ 2333.547640] ------------[ cut here ]------------ [ 2333.552257] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2333.563903] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2333.572858] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2333.628326] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2333.637281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2333.644758] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2333.650244] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2333.668991] RSP: 0018:ffffc90001f7fb30 EFLAGS: 00010282 [ 2333.674214] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2333.681339] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2333.688472] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000816b [ 2333.695606] R10: 00000000002a93c0 R11: ffffc90001f7f9c0 R12: ffff8881c5357cb0 [ 2333.702736] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2333.709869] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2333.717958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2333.723703] CR2: 00007f9bd22d7020 CR3: 0000000113532003 CR4: 00000000003706e0 [ 2333.730834] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2333.737958] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2333.745083] Call Trace: [ 2333.747527] [ 2333.749626] ? __warn+0x85/0x180 [ 2333.752859] ? __list_add_valid_or_report+0x7d/0xa0 [ 2333.757737] ? report_bug+0xfc/0x1d0 [ 2333.761318] ? prb_read_valid+0x1b/0x20 [ 2333.765159] ? handle_bug+0x44/0x70 [ 2333.768650] ? exc_invalid_op+0x17/0x60 [ 2333.772488] ? asm_exc_invalid_op+0x1a/0x20 [ 2333.776679] ? __list_add_valid_or_report+0x7d/0xa0 [ 2333.781555] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2333.786444] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2333.791680] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2333.796480] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2333.801196] lookup_open+0x15d/0x630 [ 2333.804777] ? verify_cpu+0x20/0x100 [ 2333.808354] ? verify_cpu+0x20/0x100 [ 2333.811933] path_openat+0x25a/0x9d0 [ 2333.815515] do_filp_open+0xb8/0x120 [ 2333.819095] ? lock_release+0xc5/0x260 [ 2333.822845] ? _raw_spin_unlock+0x23/0x40 [ 2333.826856] ? alloc_fd+0x13f/0x200 [ 2333.830351] do_sys_openat2+0x249/0x290 [ 2333.834190] do_sys_open+0x4f/0x80 [ 2333.837594] do_syscall_64+0x38/0x80 [ 2333.841174] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2333.846225] RIP: 0033:0x7f48f0eed2f2 [ 2333.849807] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2333.868552] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2333.876118] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2333.883254] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2333.890385] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2333.897516] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2333.904654] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2333.911788] [ 2333.913974] irq event stamp: 12955459 [ 2333.917641] hardirqs last enabled at (12955469): [] console_unlock+0x106/0x120 [ 2333.926509] hardirqs last disabled at (12955484): [] console_unlock+0xeb/0x120 [ 2333.935286] softirqs last enabled at (12955482): [] __do_softirq+0x2be/0x371 [ 2333.943979] softirqs last disabled at (12955477): [] irq_exit_rcu+0xb0/0xd0 [ 2333.952498] ---[ end trace 0000000000000000 ]--- [ 2333.957111] ------------[ cut here ]------------ [ 2333.961731] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2333.973380] WARNING: CPU: 3 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2333.982342] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2334.037869] CPU: 3 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2334.046822] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2334.054305] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2334.059798] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2334.078550] RSP: 0018:ffffc90001fcfd50 EFLAGS: 00010286 [ 2334.083796] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2334.090931] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2334.098078] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000081a7 [ 2334.105209] R10: 00000000002aa5c8 R11: ffffc90001fcfbe0 R12: ffff8881d106cbc8 [ 2334.112344] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2334.119476] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2334.127564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2334.133308] CR2: 00007fb38cf81060 CR3: 000000011aeae006 CR4: 00000000003706e0 [ 2334.140441] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2334.147575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2334.154707] Call Trace: [ 2334.157160] [ 2334.159280] ? __warn+0x85/0x180 [ 2334.162519] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.167406] ? report_bug+0xfc/0x1d0 [ 2334.170983] ? prb_read_valid+0x1b/0x20 [ 2334.174828] ? handle_bug+0x44/0x70 [ 2334.178326] ? exc_invalid_op+0x17/0x60 [ 2334.182164] ? asm_exc_invalid_op+0x1a/0x20 [ 2334.186360] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.191239] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2334.196163] ceph_readdir+0xc0/0x1670 [ceph] [ 2334.200456] ? wrap_directory_iterator+0x38/0x70 [ 2334.205081] ? wrap_directory_iterator+0x30/0x70 [ 2334.209697] ? lock_is_held_type+0xaa/0x120 [ 2334.213890] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2334.218782] wrap_directory_iterator+0x47/0x70 [ 2334.223231] iterate_dir+0x8d/0x170 [ 2334.226731] __x64_sys_getdents64+0x87/0x140 [ 2334.231000] ? __x64_sys_getdents64+0x140/0x140 [ 2334.235557] do_syscall_64+0x38/0x80 [ 2334.239140] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2334.244195] RIP: 0033:0x7f48f0ec480b [ 2334.247781] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2334.266543] RSP: 002b:00007ffc53d5ff88 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2334.274110] RAX: ffffffffffffffda RBX: 000000000207dfc0 RCX: 00007f48f0ec480b [ 2334.281276] RDX: 0000000000010000 RSI: 000000000207dff0 RDI: 0000000000000014 [ 2334.288409] RBP: 000000000207dff0 R08: 00000000000000db R09: 0000000000000003 [ 2334.295557] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2334.302708] R13: 0000000000000002 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2334.309893] [ 2334.312088] irq event stamp: 12919991 [ 2334.315771] hardirqs last enabled at (12920001): [] console_unlock+0x106/0x120 [ 2334.324653] hardirqs last disabled at (12920010): [] console_unlock+0xeb/0x120 [ 2334.333450] softirqs last enabled at (12919978): [] __do_softirq+0x2be/0x371 [ 2334.342143] softirqs last disabled at (12919973): [] irq_exit_rcu+0xb0/0xd0 [ 2334.350681] ---[ end trace 0000000000000000 ]--- [ 2334.355320] ------------[ cut here ]------------ [ 2334.359935] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2334.371600] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2334.380294] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2334.435772] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2334.444913] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2334.452410] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2334.457904] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2334.463407] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2334.482155] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2334.487397] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2334.494532] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2334.501665] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000081df [ 2334.508815] R10: 00000000002ab5b8 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2334.515964] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2334.523113] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2334.531216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2334.536972] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2334.544121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2334.551280] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2334.558415] Call Trace: [ 2334.560866] [ 2334.562982] ? __warn+0x85/0x180 [ 2334.566233] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.571129] ? report_bug+0xfc/0x1d0 [ 2334.574707] ? prb_read_valid+0x1b/0x20 [ 2334.578566] ? handle_bug+0x44/0x70 [ 2334.582073] ? exc_invalid_op+0x17/0x60 [ 2334.585921] ? asm_exc_invalid_op+0x1a/0x20 [ 2334.590148] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.595026] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.599919] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2334.604826] ceph_add_cap+0x20b/0x620 [ceph] [ 2334.609147] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2334.613886] ? lock_acquire+0xcb/0x2b0 [ 2334.617655] ceph_fill_trace+0x143/0xf30 [ceph] [ 2334.622195] ? __mutex_lock+0x97/0xce0 [ 2334.625963] ? __mutex_lock+0xaec/0xce0 [ 2334.629822] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2334.634378] ? lock_downgrade+0xfd/0x250 [ 2334.638310] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2334.642706] ceph_con_process_message+0x74/0x150 [libceph] [ 2334.648202] ? calc_signature+0xdb/0x100 [libceph] [ 2334.653040] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2334.658363] ? process_scheduled_works+0x22a/0x510 [ 2334.663155] ? process_scheduled_works+0x22a/0x510 [ 2334.667962] ceph_con_workfn+0x344/0x750 [libceph] [ 2334.672784] ? process_scheduled_works+0x22a/0x510 [ 2334.677591] process_scheduled_works+0x2a4/0x510 [ 2334.682250] worker_thread+0x188/0x340 [ 2334.685999] ? rescuer_thread+0x310/0x310 [ 2334.690029] kthread+0xf5/0x130 [ 2334.693194] ? kthread_complete_and_exit+0x20/0x20 [ 2334.698002] ret_from_fork+0x31/0x40 [ 2334.701597] ? kthread_complete_and_exit+0x20/0x20 [ 2334.706400] ret_from_fork_asm+0x11/0x20 [ 2334.710334] [ 2334.712528] irq event stamp: 10686617 [ 2334.716194] hardirqs last enabled at (10686627): [] console_unlock+0x106/0x120 [ 2334.725076] hardirqs last disabled at (10686636): [] console_unlock+0xeb/0x120 [ 2334.733874] softirqs last enabled at (10686564): [] __do_softirq+0x2be/0x371 [ 2334.742583] softirqs last disabled at (10686559): [] irq_exit_rcu+0xb0/0xd0 [ 2334.751120] ---[ end trace 0000000000000000 ]--- [ 2334.755785] ------------[ cut here ]------------ [ 2334.760403] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2334.772048] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2334.780745] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2334.836223] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2334.845346] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2334.852827] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2334.858320] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2334.863806] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2334.882553] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2334.887778] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2334.894912] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2334.902046] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000821e [ 2334.909179] R10: 00000000002ac6c8 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2334.916310] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2334.923443] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2334.931530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2334.937275] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2334.944408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2334.951548] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2334.958674] Call Trace: [ 2334.961126] [ 2334.963237] ? __warn+0x85/0x180 [ 2334.966469] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.971345] ? report_bug+0xfc/0x1d0 [ 2334.974925] ? prb_read_valid+0x1b/0x20 [ 2334.978765] ? handle_bug+0x44/0x70 [ 2334.982257] ? exc_invalid_op+0x17/0x60 [ 2334.986094] ? asm_exc_invalid_op+0x1a/0x20 [ 2334.990286] ? __list_add_valid_or_report+0x7d/0xa0 [ 2334.995162] ? __list_add_valid_or_report+0x7d/0xa0 [ 2335.000041] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2335.004933] ceph_add_cap+0x20b/0x620 [ceph] [ 2335.009217] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2335.013940] ? lock_acquire+0xcb/0x2b0 [ 2335.017700] ceph_fill_trace+0x143/0xf30 [ceph] [ 2335.022238] ? __mutex_lock+0x97/0xce0 [ 2335.025989] ? __mutex_lock+0xaec/0xce0 [ 2335.029832] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2335.034379] ? lock_downgrade+0xfd/0x250 [ 2335.038310] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2335.042691] ceph_con_process_message+0x74/0x150 [libceph] [ 2335.048187] ? calc_signature+0xdb/0x100 [libceph] [ 2335.052999] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2335.058330] ? process_scheduled_works+0x22a/0x510 [ 2335.063122] ? process_scheduled_works+0x22a/0x510 [ 2335.067910] ceph_con_workfn+0x344/0x750 [libceph] [ 2335.072714] ? process_scheduled_works+0x22a/0x510 [ 2335.077504] process_scheduled_works+0x2a4/0x510 [ 2335.082131] worker_thread+0x188/0x340 [ 2335.085886] ? rescuer_thread+0x310/0x310 [ 2335.089899] kthread+0xf5/0x130 [ 2335.093045] ? kthread_complete_and_exit+0x20/0x20 [ 2335.097839] ret_from_fork+0x31/0x40 [ 2335.101417] ? kthread_complete_and_exit+0x20/0x20 [ 2335.106211] ret_from_fork_asm+0x11/0x20 [ 2335.110143] [ 2335.112337] irq event stamp: 10687741 [ 2335.116002] hardirqs last enabled at (10687751): [] console_unlock+0x106/0x120 [ 2335.124869] hardirqs last disabled at (10687760): [] console_unlock+0xeb/0x120 [ 2335.133650] softirqs last enabled at (10687718): [] __do_softirq+0x2be/0x371 [ 2335.142341] softirqs last disabled at (10687713): [] irq_exit_rcu+0xb0/0xd0 [ 2335.150862] ---[ end trace 0000000000000000 ]--- [ 2335.155481] ------------[ cut here ]------------ [ 2335.160114] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2335.171765] WARNING: CPU: 4 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2335.180730] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2335.236199] CPU: 4 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2335.245159] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2335.252638] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2335.258125] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2335.276873] RSP: 0018:ffffc90001957b20 EFLAGS: 00010282 [ 2335.282105] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2335.289236] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2335.296370] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000825d [ 2335.303505] R10: 00000000002ad7e0 R11: ffffc900019579b0 R12: ffff8881d0bb4bc8 [ 2335.310637] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2335.317768] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2335.325856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2335.331599] CR2: 00007f9bd237b128 CR3: 000000016696c006 CR4: 00000000003706e0 [ 2335.338734] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2335.345864] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2335.353008] Call Trace: [ 2335.355459] [ 2335.357567] ? __warn+0x85/0x180 [ 2335.360800] ? __list_add_valid_or_report+0x7d/0xa0 [ 2335.365681] ? report_bug+0xfc/0x1d0 [ 2335.369260] ? prb_read_valid+0x1b/0x20 [ 2335.373103] ? handle_bug+0x44/0x70 [ 2335.376597] ? exc_invalid_op+0x17/0x60 [ 2335.380442] ? asm_exc_invalid_op+0x1a/0x20 [ 2335.384639] ? __list_add_valid_or_report+0x7d/0xa0 [ 2335.389523] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2335.394420] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2335.399676] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2335.404482] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2335.409207] ? __d_lookup+0xd6/0x200 [ 2335.412794] lookup_fast+0x6d/0xf0 [ 2335.416199] walk_component+0x26/0x160 [ 2335.419951] path_lookupat.isra.40+0x58/0x120 [ 2335.424313] filename_lookup+0xd4/0x190 [ 2335.428164] ? lock_acquire+0xcb/0x2b0 [ 2335.431922] ? __might_fault+0x45/0x80 [ 2335.435681] ? find_held_lock+0x2d/0x90 [ 2335.439521] ? __might_fault+0x45/0x80 [ 2335.443280] vfs_statx+0x89/0x150 [ 2335.446609] vfs_fstatat+0x5a/0xc0 [ 2335.450017] __do_sys_newstat+0x30/0x70 [ 2335.453858] ? lock_release+0xc5/0x260 [ 2335.457618] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2335.462504] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2335.467645] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2335.472785] do_syscall_64+0x38/0x80 [ 2335.476364] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2335.481416] RIP: 0033:0x7f48f0eeca79 [ 2335.484996] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2335.503744] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2335.511315] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2335.518450] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2335.525590] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2335.532725] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2335.539855] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2335.547000] [ 2335.549189] irq event stamp: 12843993 [ 2335.552857] hardirqs last enabled at (12844003): [] console_unlock+0x106/0x120 [ 2335.561729] hardirqs last disabled at (12844012): [] console_unlock+0xeb/0x120 [ 2335.570514] softirqs last enabled at (12844026): [] __do_softirq+0x2be/0x371 [ 2335.579210] softirqs last disabled at (12844021): [] irq_exit_rcu+0xb0/0xd0 [ 2335.587736] ---[ end trace 0000000000000000 ]--- [ 2335.592360] ------------[ cut here ]------------ [ 2335.596976] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2335.608622] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2335.617578] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2335.673049] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2335.681997] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2335.689469] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2335.694956] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2335.713701] RSP: 0018:ffffc90001ff7a78 EFLAGS: 00010286 [ 2335.718926] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2335.726050] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2335.733174] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000829f [ 2335.740302] R10: 00000000002ae980 R11: ffffc90001ff7908 R12: ffff8881d1329ae0 [ 2335.747433] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2335.754565] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2335.762650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2335.768387] CR2: 00007f9bd22d7020 CR3: 0000000120cc8005 CR4: 00000000003706e0 [ 2335.775512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2335.782646] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2335.789782] Call Trace: [ 2335.792233] [ 2335.794337] ? __warn+0x85/0x180 [ 2335.797571] ? __list_add_valid_or_report+0x7d/0xa0 [ 2335.802451] ? report_bug+0xfc/0x1d0 [ 2335.806031] ? prb_read_valid+0x1b/0x20 [ 2335.809869] ? handle_bug+0x44/0x70 [ 2335.813360] ? exc_invalid_op+0x17/0x60 [ 2335.817191] ? asm_exc_invalid_op+0x1a/0x20 [ 2335.821382] ? __list_add_valid_or_report+0x7d/0xa0 [ 2335.826259] ? __list_add_valid_or_report+0x7d/0xa0 [ 2335.831136] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2335.836028] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2335.841263] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2335.846071] ceph_lookup+0xf9/0x3f0 [ceph] [ 2335.850179] __lookup_slow+0xff/0x1a0 [ 2335.853852] walk_component+0x83/0x160 [ 2335.857606] path_lookupat.isra.40+0x58/0x120 [ 2335.861966] filename_lookup+0xd4/0x190 [ 2335.865809] ? lock_acquire+0xcb/0x2b0 [ 2335.869557] ? __might_fault+0x45/0x80 [ 2335.873309] ? find_held_lock+0x2d/0x90 [ 2335.877150] ? __might_fault+0x45/0x80 [ 2335.880904] vfs_statx+0x89/0x150 [ 2335.884224] vfs_fstatat+0x5a/0xc0 [ 2335.887628] __do_sys_newstat+0x30/0x70 [ 2335.891468] ? lock_release+0xc5/0x260 [ 2335.895222] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2335.900097] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2335.905238] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2335.910379] do_syscall_64+0x38/0x80 [ 2335.913958] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2335.919009] RIP: 0033:0x7f48f0eeca79 [ 2335.922591] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2335.941337] RSP: 002b:00007ffc53d5fee8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2335.948903] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2335.956033] RDX: 00007ffc53d5fef0 RSI: 00007ffc53d5fef0 RDI: 00007ffc53d60520 [ 2335.963157] RBP: 00007ffc53d60520 R08: 00000000000029f6 R09: 00000000020767c0 [ 2335.970285] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000040 [ 2335.977415] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2335.984554] [ 2335.986742] irq event stamp: 12968429 [ 2335.990406] hardirqs last enabled at (12968439): [] console_unlock+0x106/0x120 [ 2335.999273] hardirqs last disabled at (12968448): [] console_unlock+0xeb/0x120 [ 2336.008054] softirqs last enabled at (12968462): [] __do_softirq+0x2be/0x371 [ 2336.016746] softirqs last disabled at (12968457): [] irq_exit_rcu+0xb0/0xd0 [ 2336.025265] ---[ end trace 0000000000000000 ]--- [ 2336.029886] ------------[ cut here ]------------ [ 2336.034505] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2336.046169] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2336.054863] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2336.110358] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2336.119483] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2336.126962] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2336.132474] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2336.137959] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2336.156725] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2336.161968] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2336.169116] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2336.176259] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000082e1 [ 2336.183409] R10: 00000000002afb28 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2336.190543] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2336.197674] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2336.205769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2336.211522] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2336.218658] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2336.225807] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2336.232957] Call Trace: [ 2336.235419] [ 2336.237525] ? __warn+0x85/0x180 [ 2336.240760] ? __list_add_valid_or_report+0x7d/0xa0 [ 2336.245653] ? report_bug+0xfc/0x1d0 [ 2336.249251] ? prb_read_valid+0x1b/0x20 [ 2336.253110] ? handle_bug+0x44/0x70 [ 2336.256617] ? exc_invalid_op+0x17/0x60 [ 2336.260465] ? asm_exc_invalid_op+0x1a/0x20 [ 2336.264673] ? __list_add_valid_or_report+0x7d/0xa0 [ 2336.269549] ? __list_add_valid_or_report+0x7d/0xa0 [ 2336.274428] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2336.279320] ceph_add_cap+0x20b/0x620 [ceph] [ 2336.283620] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2336.288337] ? lock_acquire+0xcb/0x2b0 [ 2336.292112] ceph_fill_trace+0x143/0xf30 [ceph] [ 2336.296651] ? __mutex_lock+0x97/0xce0 [ 2336.300419] ? __mutex_lock+0xaec/0xce0 [ 2336.304261] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2336.308826] ? lock_downgrade+0xfd/0x250 [ 2336.312775] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2336.317171] ceph_con_process_message+0x74/0x150 [libceph] [ 2336.322667] ? calc_signature+0xdb/0x100 [libceph] [ 2336.327489] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2336.332828] ? process_scheduled_works+0x22a/0x510 [ 2336.337638] ? process_scheduled_works+0x22a/0x510 [ 2336.342426] ceph_con_workfn+0x344/0x750 [libceph] [ 2336.347231] ? process_scheduled_works+0x22a/0x510 [ 2336.352039] process_scheduled_works+0x2a4/0x510 [ 2336.356699] worker_thread+0x188/0x340 [ 2336.360457] ? rescuer_thread+0x310/0x310 [ 2336.364467] kthread+0xf5/0x130 [ 2336.367616] ? kthread_complete_and_exit+0x20/0x20 [ 2336.372424] ret_from_fork+0x31/0x40 [ 2336.376002] ? kthread_complete_and_exit+0x20/0x20 [ 2336.380814] ret_from_fork_asm+0x11/0x20 [ 2336.384780] [ 2336.386965] irq event stamp: 10690723 [ 2336.390642] hardirqs last enabled at (10690733): [] console_unlock+0x106/0x120 [ 2336.399524] hardirqs last disabled at (10690742): [] console_unlock+0xeb/0x120 [ 2336.408319] softirqs last enabled at (10690670): [] __do_softirq+0x2be/0x371 [ 2336.417015] softirqs last disabled at (10690665): [] irq_exit_rcu+0xb0/0xd0 [ 2336.425553] ---[ end trace 0000000000000000 ]--- [ 2336.430229] ------------[ cut here ]------------ [ 2336.434859] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2336.446507] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2336.455460] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2336.510922] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2336.519872] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2336.527352] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2336.532837] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2336.551582] RSP: 0018:ffffc90001fdfb78 EFLAGS: 00010286 [ 2336.556800] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2336.563924] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2336.571048] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008320 [ 2336.578173] R10: 00000000002b0c40 R11: ffffc90001fdfa08 R12: ffff888158a68750 [ 2336.585305] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2336.592431] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2336.600516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2336.606252] CR2: 00007fb38cfaa020 CR3: 0000000179402003 CR4: 00000000003706e0 [ 2336.613379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2336.620511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2336.627643] Call Trace: [ 2336.630088] [ 2336.632194] ? __warn+0x85/0x180 [ 2336.635427] ? __list_add_valid_or_report+0x7d/0xa0 [ 2336.640309] ? report_bug+0xfc/0x1d0 [ 2336.643885] ? prb_read_valid+0x1b/0x20 [ 2336.647726] ? handle_bug+0x44/0x70 [ 2336.651216] ? exc_invalid_op+0x17/0x60 [ 2336.655047] ? asm_exc_invalid_op+0x1a/0x20 [ 2336.659239] ? __list_add_valid_or_report+0x7d/0xa0 [ 2336.664114] ? __list_add_valid_or_report+0x7d/0xa0 [ 2336.668993] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2336.673884] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2336.679118] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2336.683920] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2336.688635] ? __d_lookup+0xd6/0x200 [ 2336.692214] lookup_fast+0x6d/0xf0 [ 2336.695618] walk_component+0x26/0x160 [ 2336.699372] link_path_walk+0x24d/0x380 [ 2336.703211] ? path_openat+0x80/0x9d0 [ 2336.706880] path_openat+0x8e/0x9d0 [ 2336.710372] do_filp_open+0xb8/0x120 [ 2336.713951] ? lock_release+0xc5/0x260 [ 2336.717702] ? _raw_spin_unlock+0x23/0x40 [ 2336.721713] ? alloc_fd+0x13f/0x200 [ 2336.725207] do_sys_openat2+0x249/0x290 [ 2336.729047] do_sys_open+0x4f/0x80 [ 2336.732452] do_syscall_64+0x38/0x80 [ 2336.736031] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2336.741083] RIP: 0033:0x7f48f0eed2f2 [ 2336.744664] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2336.763408] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2336.770975] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2336.778108] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2336.785231] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2336.792357] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2336.799488] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2336.806620] [ 2336.808807] irq event stamp: 12868091 [ 2336.812473] hardirqs last enabled at (12868101): [] console_unlock+0x106/0x120 [ 2336.821338] hardirqs last disabled at (12868110): [] console_unlock+0xeb/0x120 [ 2336.830116] softirqs last enabled at (12868028): [] __do_softirq+0x2be/0x371 [ 2336.838811] softirqs last disabled at (12868129): [] irq_exit_rcu+0xb0/0xd0 [ 2336.847329] ---[ end trace 0000000000000000 ]--- [ 2336.851942] ------------[ cut here ]------------ [ 2336.856561] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2336.868223] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2336.876922] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2336.932417] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2336.941542] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2336.949020] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2336.954531] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2336.960018] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2336.978763] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2336.984007] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2336.991156] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2336.998319] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000835f [ 2337.005449] R10: 00000000002b1d58 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2337.012581] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2337.019724] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2337.027826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2337.033588] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2337.040723] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2337.047872] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2337.055013] Call Trace: [ 2337.057476] [ 2337.059583] ? __warn+0x85/0x180 [ 2337.062833] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.067730] ? report_bug+0xfc/0x1d0 [ 2337.071330] ? prb_read_valid+0x1b/0x20 [ 2337.075166] ? handle_bug+0x44/0x70 [ 2337.078673] ? exc_invalid_op+0x17/0x60 [ 2337.082524] ? asm_exc_invalid_op+0x1a/0x20 [ 2337.086732] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.091609] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.096503] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2337.101393] ceph_add_cap+0x20b/0x620 [ceph] [ 2337.105695] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2337.110411] ? lock_acquire+0xcb/0x2b0 [ 2337.114188] ceph_fill_trace+0x143/0xf30 [ceph] [ 2337.118723] ? __mutex_lock+0x97/0xce0 [ 2337.122495] ? __mutex_lock+0xaec/0xce0 [ 2337.126335] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2337.130875] ? lock_downgrade+0xfd/0x250 [ 2337.134841] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2337.139238] ceph_con_process_message+0x74/0x150 [libceph] [ 2337.144732] ? calc_signature+0xdb/0x100 [libceph] [ 2337.149553] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2337.154895] ? process_scheduled_works+0x22a/0x510 [ 2337.159686] ? process_scheduled_works+0x22a/0x510 [ 2337.164492] ceph_con_workfn+0x344/0x750 [libceph] [ 2337.169297] ? process_scheduled_works+0x22a/0x510 [ 2337.174105] process_scheduled_works+0x2a4/0x510 [ 2337.178747] worker_thread+0x188/0x340 [ 2337.182495] ? rescuer_thread+0x310/0x310 [ 2337.186506] kthread+0xf5/0x130 [ 2337.189653] ? kthread_complete_and_exit+0x20/0x20 [ 2337.194465] ret_from_fork+0x31/0x40 [ 2337.198044] ? kthread_complete_and_exit+0x20/0x20 [ 2337.202853] ret_from_fork_asm+0x11/0x20 [ 2337.206820] [ 2337.209006] irq event stamp: 10693683 [ 2337.212689] hardirqs last enabled at (10693693): [] console_unlock+0x106/0x120 [ 2337.221573] hardirqs last disabled at (10693702): [] console_unlock+0xeb/0x120 [ 2337.230353] softirqs last enabled at (10693650): [] __do_softirq+0x2be/0x371 [ 2337.239047] softirqs last disabled at (10693645): [] irq_exit_rcu+0xb0/0xd0 [ 2337.247582] ---[ end trace 0000000000000000 ]--- [ 2337.252372] ------------[ cut here ]------------ [ 2337.256986] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2337.268650] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2337.277344] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2337.332821] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2337.341963] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2337.349451] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2337.354947] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2337.360450] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2337.379197] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2337.384441] RAX: 0000000000000000 RBX: ffff88819f02eb98 RCX: 0000000000000000 [ 2337.391574] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2337.398706] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000839e [ 2337.405857] R10: 00000000002b2e68 R11: ffffc9000030b828 R12: ffff88819f02f2e8 [ 2337.413006] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2337.420147] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2337.428250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2337.434004] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2337.441148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2337.448296] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2337.455439] Call Trace: [ 2337.457891] [ 2337.460007] ? __warn+0x85/0x180 [ 2337.463258] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.468154] ? report_bug+0xfc/0x1d0 [ 2337.471749] ? prb_read_valid+0x1b/0x20 [ 2337.475622] ? handle_bug+0x44/0x70 [ 2337.479126] ? exc_invalid_op+0x17/0x60 [ 2337.482983] ? asm_exc_invalid_op+0x1a/0x20 [ 2337.487207] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.492082] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.496979] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2337.501888] ceph_add_cap+0x20b/0x620 [ceph] [ 2337.506207] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2337.510939] ? lock_acquire+0xcb/0x2b0 [ 2337.514708] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2337.519157] ? __mutex_lock+0x97/0xce0 [ 2337.522927] ? __mutex_lock+0xaec/0xce0 [ 2337.526803] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2337.531342] ? lock_downgrade+0xfd/0x250 [ 2337.535291] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2337.539669] ceph_con_process_message+0x74/0x150 [libceph] [ 2337.545166] ? calc_signature+0xdb/0x100 [libceph] [ 2337.549987] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2337.555341] ? process_scheduled_works+0x22a/0x510 [ 2337.560136] ? process_scheduled_works+0x22a/0x510 [ 2337.564943] ceph_con_workfn+0x344/0x750 [libceph] [ 2337.569766] ? process_scheduled_works+0x22a/0x510 [ 2337.574574] process_scheduled_works+0x2a4/0x510 [ 2337.579214] worker_thread+0x188/0x340 [ 2337.582962] ? rescuer_thread+0x310/0x310 [ 2337.586992] kthread+0xf5/0x130 [ 2337.590156] ? kthread_complete_and_exit+0x20/0x20 [ 2337.594967] ret_from_fork+0x31/0x40 [ 2337.598562] ? kthread_complete_and_exit+0x20/0x20 [ 2337.603356] ret_from_fork_asm+0x11/0x20 [ 2337.607311] [ 2337.609498] irq event stamp: 10694795 [ 2337.613158] hardirqs last enabled at (10694805): [] console_unlock+0x106/0x120 [ 2337.622042] hardirqs last disabled at (10694814): [] console_unlock+0xeb/0x120 [ 2337.630838] softirqs last enabled at (10694740): [] __do_softirq+0x2be/0x371 [ 2337.639553] softirqs last disabled at (10694835): [] irq_exit_rcu+0xb0/0xd0 [ 2337.648071] ---[ end trace 0000000000000000 ]--- [ 2337.652733] ------------[ cut here ]------------ [ 2337.657350] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2337.669012] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2337.677699] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2337.733195] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2337.742341] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2337.749815] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2337.755340] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2337.760822] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2337.779578] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2337.784803] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2337.791946] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2337.799098] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000083dd [ 2337.806246] R10: 00000000002b40b8 R11: ffffc9000030b828 R12: ffff8881d12d1118 [ 2337.813396] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2337.820528] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2337.828622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2337.834379] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2337.841511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2337.848643] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2337.855785] Call Trace: [ 2337.858247] [ 2337.860361] ? __warn+0x85/0x180 [ 2337.863595] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.868475] ? report_bug+0xfc/0x1d0 [ 2337.872052] ? prb_read_valid+0x1b/0x20 [ 2337.875911] ? handle_bug+0x44/0x70 [ 2337.879419] ? exc_invalid_op+0x17/0x60 [ 2337.883259] ? asm_exc_invalid_op+0x1a/0x20 [ 2337.887469] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.892352] ? __list_add_valid_or_report+0x7d/0xa0 [ 2337.897231] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2337.902138] ceph_add_cap+0x20b/0x620 [ceph] [ 2337.906442] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2337.911186] ceph_fill_trace+0x143/0xf30 [ceph] [ 2337.915725] ? __mutex_lock+0x97/0xce0 [ 2337.919496] ? __mutex_lock+0xaec/0xce0 [ 2337.923346] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2337.927894] ? lock_downgrade+0xfd/0x250 [ 2337.931859] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2337.936246] ceph_con_process_message+0x74/0x150 [libceph] [ 2337.941743] ? calc_signature+0xdb/0x100 [libceph] [ 2337.946564] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2337.951886] ? process_scheduled_works+0x22a/0x510 [ 2337.956719] ? process_scheduled_works+0x22a/0x510 [ 2337.961511] ceph_con_workfn+0x344/0x750 [libceph] [ 2337.966338] ? process_scheduled_works+0x22a/0x510 [ 2337.971132] process_scheduled_works+0x2a4/0x510 [ 2337.975792] worker_thread+0x188/0x340 [ 2337.979541] ? rescuer_thread+0x310/0x310 [ 2337.983553] kthread+0xf5/0x130 [ 2337.986699] ? kthread_complete_and_exit+0x20/0x20 [ 2337.991513] ret_from_fork+0x31/0x40 [ 2337.995088] ? kthread_complete_and_exit+0x20/0x20 [ 2337.999899] ret_from_fork_asm+0x11/0x20 [ 2338.003864] [ 2338.006050] irq event stamp: 10695877 [ 2338.009725] hardirqs last enabled at (10695887): [] console_unlock+0x106/0x120 [ 2338.018609] hardirqs last disabled at (10695896): [] console_unlock+0xeb/0x120 [ 2338.027389] softirqs last enabled at (10695834): [] __do_softirq+0x2be/0x371 [ 2338.036081] softirqs last disabled at (10695829): [] irq_exit_rcu+0xb0/0xd0 [ 2338.044634] ---[ end trace 0000000000000000 ]--- [ 2338.049299] ------------[ cut here ]------------ [ 2338.053919] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2338.065582] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2338.074278] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2338.129780] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2338.138923] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2338.146419] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2338.151914] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2338.157417] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2338.176164] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2338.181406] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2338.188541] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2338.195674] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000841b [ 2338.202823] R10: 00000000002b5190 R11: ffffc9000030b828 R12: ffff8881c2cefcb0 [ 2338.209973] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2338.217122] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2338.225219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2338.230979] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2338.238122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2338.245263] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2338.252405] Call Trace: [ 2338.254858] [ 2338.256983] ? __warn+0x85/0x180 [ 2338.260233] ? __list_add_valid_or_report+0x7d/0xa0 [ 2338.265129] ? report_bug+0xfc/0x1d0 [ 2338.268726] ? prb_read_valid+0x1b/0x20 [ 2338.272584] ? handle_bug+0x44/0x70 [ 2338.276074] ? exc_invalid_op+0x17/0x60 [ 2338.279922] ? asm_exc_invalid_op+0x1a/0x20 [ 2338.284147] ? __list_add_valid_or_report+0x7d/0xa0 [ 2338.289025] ? __list_add_valid_or_report+0x7d/0xa0 [ 2338.293921] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2338.298828] ceph_add_cap+0x20b/0x620 [ceph] [ 2338.303139] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2338.307887] ? lock_acquire+0xcb/0x2b0 [ 2338.311656] ceph_fill_trace+0x143/0xf30 [ceph] [ 2338.316193] ? __mutex_lock+0x97/0xce0 [ 2338.319963] ? __mutex_lock+0xaec/0xce0 [ 2338.323823] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2338.328380] ? lock_downgrade+0xfd/0x250 [ 2338.332331] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2338.336706] ceph_con_process_message+0x74/0x150 [libceph] [ 2338.342203] ? calc_signature+0xdb/0x100 [libceph] [ 2338.347025] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2338.352372] ? process_scheduled_works+0x22a/0x510 [ 2338.357165] ? process_scheduled_works+0x22a/0x510 [ 2338.361970] ceph_con_workfn+0x344/0x750 [libceph] [ 2338.366792] ? process_scheduled_works+0x22a/0x510 [ 2338.371601] process_scheduled_works+0x2a4/0x510 [ 2338.376241] worker_thread+0x188/0x340 [ 2338.379991] ? rescuer_thread+0x310/0x310 [ 2338.384019] kthread+0xf5/0x130 [ 2338.387184] ? kthread_complete_and_exit+0x20/0x20 [ 2338.391994] ret_from_fork+0x31/0x40 [ 2338.395589] ? kthread_complete_and_exit+0x20/0x20 [ 2338.400384] ret_from_fork_asm+0x11/0x20 [ 2338.404336] [ 2338.406528] irq event stamp: 10696995 [ 2338.410192] hardirqs last enabled at (10697005): [] console_unlock+0x106/0x120 [ 2338.419076] hardirqs last disabled at (10697014): [] console_unlock+0xeb/0x120 [ 2338.427875] softirqs last enabled at (10696940): [] __do_softirq+0x2be/0x371 [ 2338.436584] softirqs last disabled at (10696935): [] irq_exit_rcu+0xb0/0xd0 [ 2338.445105] ---[ end trace 0000000000000000 ]--- [ 2338.449773] ------------[ cut here ]------------ [ 2338.454387] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2338.466046] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2338.474737] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2338.530233] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2338.539374] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2338.546852] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2338.552365] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2338.557849] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2338.576605] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2338.581832] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2338.588982] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2338.596126] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000845a [ 2338.603274] R10: 00000000002b6400 R11: ffffc9000030b828 R12: ffff8881c5357cb0 [ 2338.610423] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2338.617556] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2338.625643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2338.631404] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2338.638553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2338.645682] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2338.652829] Call Trace: [ 2338.655292] [ 2338.657416] ? __warn+0x85/0x180 [ 2338.660650] ? __list_add_valid_or_report+0x7d/0xa0 [ 2338.665545] ? report_bug+0xfc/0x1d0 [ 2338.669125] ? prb_read_valid+0x1b/0x20 [ 2338.672982] ? handle_bug+0x44/0x70 [ 2338.676490] ? exc_invalid_op+0x17/0x60 [ 2338.680349] ? asm_exc_invalid_op+0x1a/0x20 [ 2338.684538] ? __list_add_valid_or_report+0x7d/0xa0 [ 2338.689415] ? __list_add_valid_or_report+0x7d/0xa0 [ 2338.694293] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2338.699200] ceph_add_cap+0x20b/0x620 [ceph] [ 2338.703503] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2338.708242] ? lock_acquire+0xcb/0x2b0 [ 2338.712012] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2338.716463] ? __mutex_lock+0x97/0xce0 [ 2338.720214] ? __mutex_lock+0xaec/0xce0 [ 2338.724074] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2338.728634] ? lock_downgrade+0xfd/0x250 [ 2338.732579] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2338.736974] ceph_con_process_message+0x74/0x150 [libceph] [ 2338.742471] ? calc_signature+0xdb/0x100 [libceph] [ 2338.747291] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2338.752615] ? process_scheduled_works+0x22a/0x510 [ 2338.757407] ? process_scheduled_works+0x22a/0x510 [ 2338.762197] ceph_con_workfn+0x344/0x750 [libceph] [ 2338.767019] ? process_scheduled_works+0x22a/0x510 [ 2338.771826] process_scheduled_works+0x2a4/0x510 [ 2338.776469] worker_thread+0x188/0x340 [ 2338.780224] ? rescuer_thread+0x310/0x310 [ 2338.784254] kthread+0xf5/0x130 [ 2338.787419] ? kthread_complete_and_exit+0x20/0x20 [ 2338.792211] ret_from_fork+0x31/0x40 [ 2338.795807] ? kthread_complete_and_exit+0x20/0x20 [ 2338.800618] ret_from_fork_asm+0x11/0x20 [ 2338.804549] [ 2338.806735] irq event stamp: 10698121 [ 2338.810411] hardirqs last enabled at (10698131): [] console_unlock+0x106/0x120 [ 2338.819277] hardirqs last disabled at (10698140): [] console_unlock+0xeb/0x120 [ 2338.828072] softirqs last enabled at (10698068): [] __do_softirq+0x2be/0x371 [ 2338.836784] softirqs last disabled at (10698063): [] irq_exit_rcu+0xb0/0xd0 [ 2338.845320] ---[ end trace 0000000000000000 ]--- [ 2338.849984] ------------[ cut here ]------------ [ 2338.854607] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2338.866261] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2338.874952] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2338.930437] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2338.939564] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2338.947042] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2338.952553] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2338.958040] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2338.976805] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2338.982046] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2338.989188] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2338.996352] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008499 [ 2339.003481] R10: 00000000002b7500 R11: ffffc9000030b6a8 R12: ffff8881d106cbc8 [ 2339.010614] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2339.017755] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2339.025850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2339.031612] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2339.038738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2339.045888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2339.053028] Call Trace: [ 2339.055490] [ 2339.057597] ? __warn+0x85/0x180 [ 2339.060830] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.065725] ? report_bug+0xfc/0x1d0 [ 2339.069323] ? prb_read_valid+0x1b/0x20 [ 2339.073180] ? handle_bug+0x44/0x70 [ 2339.076690] ? exc_invalid_op+0x17/0x60 [ 2339.080537] ? asm_exc_invalid_op+0x1a/0x20 [ 2339.084745] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.089620] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.094518] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2339.099408] ceph_check_caps+0x999/0xae0 [ceph] [ 2339.103981] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2339.108689] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2339.113428] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2339.118261] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2339.122646] ? lock_release+0xc5/0x260 [ 2339.126414] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2339.131120] ceph_con_process_message+0x74/0x150 [libceph] [ 2339.136630] ? calc_signature+0xdb/0x100 [libceph] [ 2339.141433] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2339.146754] ? process_scheduled_works+0x22a/0x510 [ 2339.151566] ? process_scheduled_works+0x22a/0x510 [ 2339.156365] ceph_con_workfn+0x344/0x750 [libceph] [ 2339.161167] ? process_scheduled_works+0x22a/0x510 [ 2339.165975] process_scheduled_works+0x2a4/0x510 [ 2339.170618] worker_thread+0x188/0x340 [ 2339.174374] ? rescuer_thread+0x310/0x310 [ 2339.178386] kthread+0xf5/0x130 [ 2339.181532] ? kthread_complete_and_exit+0x20/0x20 [ 2339.186326] ret_from_fork+0x31/0x40 [ 2339.189921] ? kthread_complete_and_exit+0x20/0x20 [ 2339.194713] ret_from_fork_asm+0x11/0x20 [ 2339.198678] [ 2339.200868] irq event stamp: 10699211 [ 2339.204543] hardirqs last enabled at (10699227): [] console_unlock+0x106/0x120 [ 2339.213410] hardirqs last disabled at (10699236): [] console_unlock+0xeb/0x120 [ 2339.222189] softirqs last enabled at (10699224): [] __do_softirq+0x2be/0x371 [ 2339.230897] softirqs last disabled at (10699219): [] irq_exit_rcu+0xb0/0xd0 [ 2339.239434] ---[ end trace 0000000000000000 ]--- [ 2339.244075] ------------[ cut here ]------------ [ 2339.248691] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2339.260373] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2339.269067] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2339.324563] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2339.333688] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2339.341184] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2339.346694] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2339.352189] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2339.370955] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2339.376198] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2339.383365] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2339.390500] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000084d6 [ 2339.397645] R10: 00000000002b86d8 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2339.404789] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2339.411932] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2339.420034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2339.425788] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2339.432931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2339.440073] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2339.447222] Call Trace: [ 2339.449693] [ 2339.451818] ? __warn+0x85/0x180 [ 2339.455067] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.459963] ? report_bug+0xfc/0x1d0 [ 2339.463560] ? prb_read_valid+0x1b/0x20 [ 2339.467401] ? handle_bug+0x44/0x70 [ 2339.470891] ? exc_invalid_op+0x17/0x60 [ 2339.474740] ? asm_exc_invalid_op+0x1a/0x20 [ 2339.478964] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.483840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.488735] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2339.493655] ceph_add_cap+0x20b/0x620 [ceph] [ 2339.497973] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2339.502705] ? lock_acquire+0xcb/0x2b0 [ 2339.506458] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2339.510917] ? __mutex_lock+0x97/0xce0 [ 2339.514685] ? __mutex_lock+0xaec/0xce0 [ 2339.518544] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2339.523083] ? lock_downgrade+0xfd/0x250 [ 2339.527048] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2339.531419] ceph_con_process_message+0x74/0x150 [libceph] [ 2339.536916] ? calc_signature+0xdb/0x100 [libceph] [ 2339.541736] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2339.547093] ? process_scheduled_works+0x22a/0x510 [ 2339.551885] ? process_scheduled_works+0x22a/0x510 [ 2339.556694] ceph_con_workfn+0x344/0x750 [libceph] [ 2339.561514] ? process_scheduled_works+0x22a/0x510 [ 2339.566304] process_scheduled_works+0x2a4/0x510 [ 2339.570964] worker_thread+0x188/0x340 [ 2339.574712] ? rescuer_thread+0x310/0x310 [ 2339.578742] kthread+0xf5/0x130 [ 2339.581908] ? kthread_complete_and_exit+0x20/0x20 [ 2339.586717] ret_from_fork+0x31/0x40 [ 2339.590312] ? kthread_complete_and_exit+0x20/0x20 [ 2339.595123] ret_from_fork_asm+0x11/0x20 [ 2339.599090] [ 2339.601284] irq event stamp: 10700315 [ 2339.604958] hardirqs last enabled at (10700325): [] console_unlock+0x106/0x120 [ 2339.613842] hardirqs last disabled at (10700334): [] console_unlock+0xeb/0x120 [ 2339.622654] softirqs last enabled at (10700262): [] __do_softirq+0x2be/0x371 [ 2339.631375] softirqs last disabled at (10700257): [] irq_exit_rcu+0xb0/0xd0 [ 2339.639895] ---[ end trace 0000000000000000 ]--- [ 2339.644555] ------------[ cut here ]------------ [ 2339.649170] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2339.660846] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2339.669535] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2339.725035] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2339.734158] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2339.741655] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2339.747174] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2339.752674] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2339.771432] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2339.776657] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2339.783808] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2339.790958] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008515 [ 2339.798108] R10: 00000000002b97f0 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2339.805258] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2339.812409] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2339.820494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2339.826238] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2339.833381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2339.840514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2339.847657] Call Trace: [ 2339.850127] [ 2339.852250] ? __warn+0x85/0x180 [ 2339.855501] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.860379] ? report_bug+0xfc/0x1d0 [ 2339.863957] ? prb_read_valid+0x1b/0x20 [ 2339.867817] ? handle_bug+0x44/0x70 [ 2339.871324] ? exc_invalid_op+0x17/0x60 [ 2339.875174] ? asm_exc_invalid_op+0x1a/0x20 [ 2339.879385] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.884266] ? __list_add_valid_or_report+0x7d/0xa0 [ 2339.889162] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2339.894069] ceph_add_cap+0x20b/0x620 [ceph] [ 2339.898384] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2339.903119] ? lock_acquire+0xcb/0x2b0 [ 2339.906890] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2339.911339] ? __mutex_lock+0x97/0xce0 [ 2339.915109] ? __mutex_lock+0xaec/0xce0 [ 2339.918961] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2339.923525] ? lock_downgrade+0xfd/0x250 [ 2339.927457] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2339.931852] ceph_con_process_message+0x74/0x150 [libceph] [ 2339.937369] ? calc_signature+0xdb/0x100 [libceph] [ 2339.942201] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2339.947529] ? process_scheduled_works+0x22a/0x510 [ 2339.952346] ? process_scheduled_works+0x22a/0x510 [ 2339.957142] ceph_con_workfn+0x344/0x750 [libceph] [ 2339.961948] ? process_scheduled_works+0x22a/0x510 [ 2339.966756] process_scheduled_works+0x2a4/0x510 [ 2339.971398] worker_thread+0x188/0x340 [ 2339.975154] ? rescuer_thread+0x310/0x310 [ 2339.979183] kthread+0xf5/0x130 [ 2339.982369] ? kthread_complete_and_exit+0x20/0x20 [ 2339.987157] ret_from_fork+0x31/0x40 [ 2339.990754] ? kthread_complete_and_exit+0x20/0x20 [ 2339.995565] ret_from_fork_asm+0x11/0x20 [ 2339.999497] [ 2340.001691] irq event stamp: 10701457 [ 2340.005383] hardirqs last enabled at (10701467): [] console_unlock+0x106/0x120 [ 2340.014249] hardirqs last disabled at (10701476): [] console_unlock+0xeb/0x120 [ 2340.023046] softirqs last enabled at (10701402): [] __do_softirq+0x2be/0x371 [ 2340.031757] softirqs last disabled at (10701397): [] irq_exit_rcu+0xb0/0xd0 [ 2340.040277] ---[ end trace 0000000000000000 ]--- [ 2340.045836] ------------[ cut here ]------------ [ 2340.050609] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2340.062275] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2340.070965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2340.126461] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2340.135585] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2340.143066] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2340.148575] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2340.154063] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2340.172828] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2340.178069] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2340.185211] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2340.192380] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008554 [ 2340.199513] R10: 00000000002ba918 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2340.206661] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2340.213804] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2340.221905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2340.227676] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2340.234819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2340.241963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2340.249114] Call Trace: [ 2340.251581] [ 2340.253690] ? __warn+0x85/0x180 [ 2340.256939] ? __list_add_valid_or_report+0x7d/0xa0 [ 2340.261836] ? report_bug+0xfc/0x1d0 [ 2340.265433] ? prb_read_valid+0x1b/0x20 [ 2340.269272] ? handle_bug+0x44/0x70 [ 2340.272780] ? exc_invalid_op+0x17/0x60 [ 2340.276630] ? asm_exc_invalid_op+0x1a/0x20 [ 2340.280838] ? __list_add_valid_or_report+0x7d/0xa0 [ 2340.285712] ? __list_add_valid_or_report+0x7d/0xa0 [ 2340.290610] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2340.295500] ceph_add_cap+0x20b/0x620 [ceph] [ 2340.299802] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2340.304524] ? lock_acquire+0xcb/0x2b0 [ 2340.308294] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2340.312743] ? __mutex_lock+0x97/0xce0 [ 2340.316513] ? __mutex_lock+0xaec/0xce0 [ 2340.320376] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2340.324919] ? lock_downgrade+0xfd/0x250 [ 2340.328861] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2340.333256] ceph_con_process_message+0x74/0x150 [libceph] [ 2340.338752] ? calc_signature+0xdb/0x100 [libceph] [ 2340.343574] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2340.348914] ? process_scheduled_works+0x22a/0x510 [ 2340.353705] ? process_scheduled_works+0x22a/0x510 [ 2340.358511] ceph_con_workfn+0x344/0x750 [libceph] [ 2340.363319] ? process_scheduled_works+0x22a/0x510 [ 2340.368125] process_scheduled_works+0x2a4/0x510 [ 2340.372785] worker_thread+0x188/0x340 [ 2340.376533] ? rescuer_thread+0x310/0x310 [ 2340.380544] kthread+0xf5/0x130 [ 2340.383692] ? kthread_complete_and_exit+0x20/0x20 [ 2340.388502] ret_from_fork+0x31/0x40 [ 2340.392078] ? kthread_complete_and_exit+0x20/0x20 [ 2340.396881] ret_from_fork_asm+0x11/0x20 [ 2340.400848] [ 2340.403034] irq event stamp: 10703489 [ 2340.406718] hardirqs last enabled at (10703499): [] console_unlock+0x106/0x120 [ 2340.415603] hardirqs last disabled at (10703508): [] console_unlock+0xeb/0x120 [ 2340.424389] softirqs last enabled at (10703436): [] __do_softirq+0x2be/0x371 [ 2340.433082] softirqs last disabled at (10703431): [] irq_exit_rcu+0xb0/0xd0 [ 2340.441620] ---[ end trace 0000000000000000 ]--- [ 2340.446264] ------------[ cut here ]------------ [ 2340.450887] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2340.462547] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2340.471243] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2340.526736] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2340.535880] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2340.543390] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2340.548887] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2340.554393] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2340.573138] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2340.578392] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2340.585523] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2340.592672] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008593 [ 2340.599815] R10: 00000000002bbac8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2340.606964] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2340.614107] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2340.622209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2340.627963] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2340.635105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2340.642248] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2340.649398] Call Trace: [ 2340.651851] [ 2340.653975] ? __warn+0x85/0x180 [ 2340.657227] ? __list_add_valid_or_report+0x7d/0xa0 [ 2340.662122] ? report_bug+0xfc/0x1d0 [ 2340.665717] ? prb_read_valid+0x1b/0x20 [ 2340.669576] ? handle_bug+0x44/0x70 [ 2340.673068] ? exc_invalid_op+0x17/0x60 [ 2340.676924] ? asm_exc_invalid_op+0x1a/0x20 [ 2340.681149] ? __list_add_valid_or_report+0x7d/0xa0 [ 2340.686025] ? __list_add_valid_or_report+0x7d/0xa0 [ 2340.690921] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2340.695829] ceph_add_cap+0x20b/0x620 [ceph] [ 2340.700148] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2340.704894] ceph_fill_trace+0x143/0xf30 [ceph] [ 2340.709433] ? __mutex_lock+0x97/0xce0 [ 2340.713186] ? __mutex_lock+0xaec/0xce0 [ 2340.717045] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2340.721602] ? lock_downgrade+0xfd/0x250 [ 2340.725532] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2340.729919] ceph_con_process_message+0x74/0x150 [libceph] [ 2340.735416] ? calc_signature+0xdb/0x100 [libceph] [ 2340.740220] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2340.745559] ? process_scheduled_works+0x22a/0x510 [ 2340.750353] ? process_scheduled_works+0x22a/0x510 [ 2340.755167] ceph_con_workfn+0x344/0x750 [libceph] [ 2340.759972] ? process_scheduled_works+0x22a/0x510 [ 2340.764778] process_scheduled_works+0x2a4/0x510 [ 2340.769412] worker_thread+0x188/0x340 [ 2340.773160] ? rescuer_thread+0x310/0x310 [ 2340.777189] kthread+0xf5/0x130 [ 2340.780354] ? kthread_complete_and_exit+0x20/0x20 [ 2340.785164] ret_from_fork+0x31/0x40 [ 2340.788742] ? kthread_complete_and_exit+0x20/0x20 [ 2340.793554] ret_from_fork_asm+0x11/0x20 [ 2340.797488] [ 2340.799681] irq event stamp: 10704551 [ 2340.803345] hardirqs last enabled at (10704561): [] console_unlock+0x106/0x120 [ 2340.812216] hardirqs last disabled at (10704570): [] console_unlock+0xeb/0x120 [ 2340.821009] softirqs last enabled at (10704508): [] __do_softirq+0x2be/0x371 [ 2340.829719] softirqs last disabled at (10704503): [] irq_exit_rcu+0xb0/0xd0 [ 2340.838255] ---[ end trace 0000000000000000 ]--- [ 2340.842919] ------------[ cut here ]------------ [ 2340.847542] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2340.859183] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2340.867872] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2340.923373] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2340.932510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2340.939991] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2340.945492] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2340.950976] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2340.969724] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2340.974949] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2340.982082] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2340.989216] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000085d1 [ 2340.996349] R10: 00000000002bcbc8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2341.003480] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2341.010615] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2341.018701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2341.024446] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2341.031579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2341.038713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2341.045846] Call Trace: [ 2341.048297] [ 2341.050404] ? __warn+0x85/0x180 [ 2341.053637] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.058515] ? report_bug+0xfc/0x1d0 [ 2341.062095] ? prb_read_valid+0x1b/0x20 [ 2341.065935] ? handle_bug+0x44/0x70 [ 2341.069427] ? exc_invalid_op+0x17/0x60 [ 2341.073267] ? asm_exc_invalid_op+0x1a/0x20 [ 2341.077459] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.082342] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.087220] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2341.092112] ceph_add_cap+0x20b/0x620 [ceph] [ 2341.096396] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2341.101118] ? lock_acquire+0xcb/0x2b0 [ 2341.104872] ceph_fill_trace+0x143/0xf30 [ceph] [ 2341.109407] ? __mutex_lock+0x97/0xce0 [ 2341.113160] ? __mutex_lock+0xaec/0xce0 [ 2341.117003] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2341.121550] ? lock_downgrade+0xfd/0x250 [ 2341.125480] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2341.129860] ceph_con_process_message+0x74/0x150 [libceph] [ 2341.135356] ? calc_signature+0xdb/0x100 [libceph] [ 2341.140169] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2341.145499] ? process_scheduled_works+0x22a/0x510 [ 2341.150293] ? process_scheduled_works+0x22a/0x510 [ 2341.155081] ceph_con_workfn+0x344/0x750 [libceph] [ 2341.159884] ? process_scheduled_works+0x22a/0x510 [ 2341.164674] process_scheduled_works+0x2a4/0x510 [ 2341.169301] worker_thread+0x188/0x340 [ 2341.173059] ? rescuer_thread+0x310/0x310 [ 2341.177069] kthread+0xf5/0x130 [ 2341.180218] ? kthread_complete_and_exit+0x20/0x20 [ 2341.185010] ret_from_fork+0x31/0x40 [ 2341.188588] ? kthread_complete_and_exit+0x20/0x20 [ 2341.193381] ret_from_fork_asm+0x11/0x20 [ 2341.197314] [ 2341.199507] irq event stamp: 10705733 [ 2341.203172] hardirqs last enabled at (10705743): [] console_unlock+0x106/0x120 [ 2341.212040] hardirqs last disabled at (10705752): [] console_unlock+0xeb/0x120 [ 2341.220819] softirqs last enabled at (10705680): [] __do_softirq+0x2be/0x371 [ 2341.229512] softirqs last disabled at (10705675): [] irq_exit_rcu+0xb0/0xd0 [ 2341.238031] ---[ end trace 0000000000000000 ]--- [ 2341.242670] ------------[ cut here ]------------ [ 2341.247288] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2341.258934] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2341.267630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2341.323109] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2341.332232] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2341.339710] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2341.345207] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2341.350690] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2341.369438] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2341.374663] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2341.381796] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2341.388930] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008610 [ 2341.396063] R10: 00000000002bdce0 R11: ffffc9000030b828 R12: ffff8881d12d1118 [ 2341.403195] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2341.410328] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2341.418414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2341.424160] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2341.431293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2341.438426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2341.445559] Call Trace: [ 2341.448011] [ 2341.450118] ? __warn+0x85/0x180 [ 2341.453353] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.458230] ? report_bug+0xfc/0x1d0 [ 2341.461810] ? prb_read_valid+0x1b/0x20 [ 2341.465650] ? handle_bug+0x44/0x70 [ 2341.469142] ? exc_invalid_op+0x17/0x60 [ 2341.472982] ? asm_exc_invalid_op+0x1a/0x20 [ 2341.477171] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.482046] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.486926] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2341.491817] ceph_add_cap+0x20b/0x620 [ceph] [ 2341.496101] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2341.500816] ? lock_acquire+0xcb/0x2b0 [ 2341.504578] ceph_fill_trace+0x143/0xf30 [ceph] [ 2341.509125] ? __mutex_lock+0x97/0xce0 [ 2341.512873] ? __mutex_lock+0xaec/0xce0 [ 2341.516716] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2341.521263] ? lock_downgrade+0xfd/0x250 [ 2341.525194] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2341.529575] ceph_con_process_message+0x74/0x150 [libceph] [ 2341.535069] ? calc_signature+0xdb/0x100 [libceph] [ 2341.539875] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2341.545206] ? process_scheduled_works+0x22a/0x510 [ 2341.549997] ? process_scheduled_works+0x22a/0x510 [ 2341.554787] ceph_con_workfn+0x344/0x750 [libceph] [ 2341.559590] ? process_scheduled_works+0x22a/0x510 [ 2341.564382] process_scheduled_works+0x2a4/0x510 [ 2341.569007] worker_thread+0x188/0x340 [ 2341.572763] ? rescuer_thread+0x310/0x310 [ 2341.576774] kthread+0xf5/0x130 [ 2341.579923] ? kthread_complete_and_exit+0x20/0x20 [ 2341.584714] ret_from_fork+0x31/0x40 [ 2341.588291] ? kthread_complete_and_exit+0x20/0x20 [ 2341.593085] ret_from_fork_asm+0x11/0x20 [ 2341.597020] [ 2341.599213] irq event stamp: 10706845 [ 2341.602878] hardirqs last enabled at (10706855): [] console_unlock+0x106/0x120 [ 2341.611746] hardirqs last disabled at (10706864): [] console_unlock+0xeb/0x120 [ 2341.620525] softirqs last enabled at (10706792): [] __do_softirq+0x2be/0x371 [ 2341.629217] softirqs last disabled at (10706787): [] irq_exit_rcu+0xb0/0xd0 [ 2341.637736] ---[ end trace 0000000000000000 ]--- [ 2341.642357] ------------[ cut here ]------------ [ 2341.646976] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2341.658627] WARNING: CPU: 6 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2341.667588] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2341.723110] CPU: 6 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2341.732070] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2341.739547] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2341.745034] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2341.763780] RSP: 0018:ffffc90001fc7b78 EFLAGS: 00010286 [ 2341.769007] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2341.776139] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2341.783271] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000864f [ 2341.790406] R10: 00000000002bef28 R11: ffffc90001fc7a08 R12: ffff888158a68750 [ 2341.797538] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2341.804669] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2341.812756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2341.818502] CR2: 00007f9bd22d7020 CR3: 0000000113500001 CR4: 00000000003706e0 [ 2341.825636] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2341.832768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2341.839901] Call Trace: [ 2341.842354] [ 2341.844461] ? __warn+0x85/0x180 [ 2341.847694] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.852573] ? report_bug+0xfc/0x1d0 [ 2341.856154] ? prb_read_valid+0x1b/0x20 [ 2341.859994] ? handle_bug+0x44/0x70 [ 2341.863484] ? exc_invalid_op+0x17/0x60 [ 2341.867325] ? asm_exc_invalid_op+0x1a/0x20 [ 2341.871520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.876400] ? __list_add_valid_or_report+0x7d/0xa0 [ 2341.881285] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2341.886186] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2341.891440] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2341.896257] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2341.900980] ? __d_lookup+0xd6/0x200 [ 2341.904561] lookup_fast+0x6d/0xf0 [ 2341.907972] walk_component+0x26/0x160 [ 2341.911727] link_path_walk+0x24d/0x380 [ 2341.915563] ? path_openat+0x80/0x9d0 [ 2341.919237] path_openat+0x8e/0x9d0 [ 2341.922738] do_filp_open+0xb8/0x120 [ 2341.926328] ? lock_release+0xc5/0x260 [ 2341.930084] ? _raw_spin_unlock+0x23/0x40 [ 2341.934100] ? alloc_fd+0x13f/0x200 [ 2341.937598] do_sys_openat2+0x249/0x290 [ 2341.941447] do_sys_open+0x4f/0x80 [ 2341.944860] do_syscall_64+0x38/0x80 [ 2341.948446] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2341.953496] RIP: 0033:0x7f48f0ef2805 [ 2341.957078] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2341.975823] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2341.983391] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2341.990522] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2341.997656] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2342.004786] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2342.011920] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2342.019066] [ 2342.021253] irq event stamp: 12955857 [ 2342.024921] hardirqs last enabled at (12955867): [] console_unlock+0x106/0x120 [ 2342.033787] hardirqs last disabled at (12955876): [] console_unlock+0xeb/0x120 [ 2342.042567] softirqs last enabled at (12955794): [] __do_softirq+0x2be/0x371 [ 2342.051261] softirqs last disabled at (12955895): [] irq_exit_rcu+0xb0/0xd0 [ 2342.059786] ---[ end trace 0000000000000000 ]--- [ 2342.064406] ------------[ cut here ]------------ [ 2342.069101] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2342.080751] WARNING: CPU: 2 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2342.089702] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2342.145172] CPU: 2 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2342.154122] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2342.161595] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2342.167079] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2342.185825] RSP: 0018:ffffc90001fefab8 EFLAGS: 00010286 [ 2342.191051] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2342.198175] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2342.205299] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000868e [ 2342.212425] R10: 00000000002c0050 R11: ffffc90001fef948 R12: ffff8881d12d1118 [ 2342.219556] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2342.226680] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2342.234758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2342.240497] CR2: 00007fb38cfaa020 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2342.247629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2342.254761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2342.261885] Call Trace: [ 2342.264331] [ 2342.266437] ? __warn+0x85/0x180 [ 2342.269668] ? __list_add_valid_or_report+0x7d/0xa0 [ 2342.274549] ? report_bug+0xfc/0x1d0 [ 2342.278129] ? prb_read_valid+0x1b/0x20 [ 2342.281968] ? handle_bug+0x44/0x70 [ 2342.285459] ? exc_invalid_op+0x17/0x60 [ 2342.289299] ? asm_exc_invalid_op+0x1a/0x20 [ 2342.293490] ? __list_add_valid_or_report+0x7d/0xa0 [ 2342.298365] ? __list_add_valid_or_report+0x7d/0xa0 [ 2342.303244] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2342.308135] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2342.313370] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2342.318171] ceph_atomic_open+0x28d/0xe80 [ceph] [ 2342.322806] lookup_open+0x37b/0x630 [ 2342.326400] path_openat+0x25a/0x9d0 [ 2342.329976] do_filp_open+0xb8/0x120 [ 2342.333555] ? lock_release+0xc5/0x260 [ 2342.337307] ? _raw_spin_unlock+0x23/0x40 [ 2342.341318] ? alloc_fd+0x13f/0x200 [ 2342.344812] do_sys_openat2+0x249/0x290 [ 2342.348651] do_sys_open+0x4f/0x80 [ 2342.352056] do_syscall_64+0x38/0x80 [ 2342.355635] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2342.360688] RIP: 0033:0x7f48f0eed2f2 [ 2342.364267] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2342.383014] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2342.390580] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2342.397713] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2342.404847] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2342.411979] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2342.419111] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2342.426249] [ 2342.428437] irq event stamp: 12895561 [ 2342.432101] hardirqs last enabled at (12895571): [] console_unlock+0x106/0x120 [ 2342.440969] hardirqs last disabled at (12895580): [] console_unlock+0xeb/0x120 [ 2342.449749] softirqs last enabled at (12895518): [] __do_softirq+0x2be/0x371 [ 2342.458440] softirqs last disabled at (12895513): [] irq_exit_rcu+0xb0/0xd0 [ 2342.466959] ---[ end trace 0000000000000000 ]--- [ 2342.471572] ------------[ cut here ]------------ [ 2342.476191] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2342.487853] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2342.496552] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2342.552028] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2342.561170] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2342.568666] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2342.574161] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2342.579664] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2342.598426] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2342.603655] RAX: 0000000000000000 RBX: ffff8881d0951390 RCX: 0000000000000000 [ 2342.610786] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2342.617931] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000086c9 [ 2342.625080] R10: 00000000002c0dd0 R11: ffffc9000030b828 R12: ffff8881d0951ae0 [ 2342.632229] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2342.639380] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2342.647482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2342.653227] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2342.660370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2342.667519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2342.674653] Call Trace: [ 2342.677106] [ 2342.679230] ? __warn+0x85/0x180 [ 2342.682482] ? __list_add_valid_or_report+0x7d/0xa0 [ 2342.687359] ? report_bug+0xfc/0x1d0 [ 2342.690955] ? prb_read_valid+0x1b/0x20 [ 2342.694815] ? handle_bug+0x44/0x70 [ 2342.698321] ? exc_invalid_op+0x17/0x60 [ 2342.702170] ? asm_exc_invalid_op+0x1a/0x20 [ 2342.706395] ? __list_add_valid_or_report+0x7d/0xa0 [ 2342.711279] ? __list_add_valid_or_report+0x7d/0xa0 [ 2342.716158] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2342.721066] ceph_add_cap+0x20b/0x620 [ceph] [ 2342.725387] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2342.730112] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2342.735444] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2342.739922] ceph_con_process_message+0x74/0x150 [libceph] [ 2342.745434] ? calc_signature+0xdb/0x100 [libceph] [ 2342.750240] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2342.755587] ? process_scheduled_works+0x22a/0x510 [ 2342.760380] ? process_scheduled_works+0x22a/0x510 [ 2342.765193] ceph_con_workfn+0x344/0x750 [libceph] [ 2342.769998] ? process_scheduled_works+0x22a/0x510 [ 2342.774807] process_scheduled_works+0x2a4/0x510 [ 2342.779449] worker_thread+0x188/0x340 [ 2342.783207] ? rescuer_thread+0x310/0x310 [ 2342.787234] kthread+0xf5/0x130 [ 2342.790398] ? kthread_complete_and_exit+0x20/0x20 [ 2342.795209] ret_from_fork+0x31/0x40 [ 2342.798789] ? kthread_complete_and_exit+0x20/0x20 [ 2342.803601] ret_from_fork_asm+0x11/0x20 [ 2342.807532] [ 2342.809726] irq event stamp: 10709741 [ 2342.813407] hardirqs last enabled at (10709751): [] console_unlock+0x106/0x120 [ 2342.822274] hardirqs last disabled at (10709760): [] console_unlock+0xeb/0x120 [ 2342.831063] softirqs last enabled at (10709738): [] __do_softirq+0x2be/0x371 [ 2342.839773] softirqs last disabled at (10709733): [] irq_exit_rcu+0xb0/0xd0 [ 2342.848309] ---[ end trace 0000000000000000 ]--- [ 2342.853097] ------------[ cut here ]------------ [ 2342.857713] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2342.869391] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2342.878084] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2342.933574] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2342.942717] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2342.950205] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2342.955724] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2342.961221] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2342.979987] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2342.985230] RAX: 0000000000000000 RBX: ffff8881d1393ab0 RCX: 0000000000000000 [ 2342.992379] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2342.999528] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008703 [ 2343.006664] R10: 00000000002c1d80 R11: ffffc9000030b828 R12: ffff8881d1394200 [ 2343.013796] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2343.020944] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2343.029048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2343.034801] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2343.041945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2343.049093] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2343.056236] Call Trace: [ 2343.058712] [ 2343.060829] ? __warn+0x85/0x180 [ 2343.064080] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.068976] ? report_bug+0xfc/0x1d0 [ 2343.072576] ? prb_read_valid+0x1b/0x20 [ 2343.076432] ? handle_bug+0x44/0x70 [ 2343.079922] ? exc_invalid_op+0x17/0x60 [ 2343.083771] ? asm_exc_invalid_op+0x1a/0x20 [ 2343.087996] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.092872] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.097767] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2343.102675] ceph_add_cap+0x20b/0x620 [ceph] [ 2343.106978] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2343.111707] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2343.117051] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2343.121515] ceph_con_process_message+0x74/0x150 [libceph] [ 2343.127010] ? calc_signature+0xdb/0x100 [libceph] [ 2343.131849] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2343.137198] ? process_scheduled_works+0x22a/0x510 [ 2343.141997] ? process_scheduled_works+0x22a/0x510 [ 2343.146803] ceph_con_workfn+0x344/0x750 [libceph] [ 2343.151627] ? process_scheduled_works+0x22a/0x510 [ 2343.156434] process_scheduled_works+0x2a4/0x510 [ 2343.161075] worker_thread+0x188/0x340 [ 2343.164825] ? rescuer_thread+0x310/0x310 [ 2343.168853] kthread+0xf5/0x130 [ 2343.172017] ? kthread_complete_and_exit+0x20/0x20 [ 2343.176828] ret_from_fork+0x31/0x40 [ 2343.180438] ? kthread_complete_and_exit+0x20/0x20 [ 2343.185234] ret_from_fork_asm+0x11/0x20 [ 2343.189199] [ 2343.191385] irq event stamp: 10710749 [ 2343.195061] hardirqs last enabled at (10710759): [] console_unlock+0x106/0x120 [ 2343.203947] hardirqs last disabled at (10710774): [] console_unlock+0xeb/0x120 [ 2343.212740] softirqs last enabled at (10710772): [] __do_softirq+0x2be/0x371 [ 2343.221452] softirqs last disabled at (10710767): [] irq_exit_rcu+0xb0/0xd0 [ 2343.229972] ---[ end trace 0000000000000000 ]--- [ 2343.234614] ------------[ cut here ]------------ [ 2343.239226] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2343.250896] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2343.259585] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2343.315065] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2343.324205] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2343.331711] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2343.337224] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2343.342738] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2343.361500] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2343.366726] RAX: 0000000000000000 RBX: ffff8881d096a720 RCX: 0000000000000000 [ 2343.373875] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2343.381018] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000873d [ 2343.388166] R10: 00000000002c2e68 R11: ffffc9000030b828 R12: ffff8881d096ae70 [ 2343.395317] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2343.402466] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2343.410552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2343.416298] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2343.423443] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2343.430573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2343.437719] Call Trace: [ 2343.440186] [ 2343.442309] ? __warn+0x85/0x180 [ 2343.445560] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.450443] ? report_bug+0xfc/0x1d0 [ 2343.454017] ? prb_read_valid+0x1b/0x20 [ 2343.457875] ? handle_bug+0x44/0x70 [ 2343.461384] ? exc_invalid_op+0x17/0x60 [ 2343.465240] ? asm_exc_invalid_op+0x1a/0x20 [ 2343.469449] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.474324] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.479219] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2343.484127] ceph_add_cap+0x20b/0x620 [ceph] [ 2343.488449] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2343.493195] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2343.498521] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2343.503001] ceph_con_process_message+0x74/0x150 [libceph] [ 2343.508499] ? calc_signature+0xdb/0x100 [libceph] [ 2343.513312] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2343.518657] ? process_scheduled_works+0x22a/0x510 [ 2343.523451] ? process_scheduled_works+0x22a/0x510 [ 2343.528240] ceph_con_workfn+0x344/0x750 [libceph] [ 2343.533060] ? process_scheduled_works+0x22a/0x510 [ 2343.537869] process_scheduled_works+0x2a4/0x510 [ 2343.542511] worker_thread+0x188/0x340 [ 2343.546268] ? rescuer_thread+0x310/0x310 [ 2343.550298] kthread+0xf5/0x130 [ 2343.553461] ? kthread_complete_and_exit+0x20/0x20 [ 2343.558255] ret_from_fork+0x31/0x40 [ 2343.561850] ? kthread_complete_and_exit+0x20/0x20 [ 2343.566662] ret_from_fork_asm+0x11/0x20 [ 2343.570595] [ 2343.572788] irq event stamp: 10711761 [ 2343.576471] hardirqs last enabled at (10711771): [] console_unlock+0x106/0x120 [ 2343.585336] hardirqs last disabled at (10711780): [] console_unlock+0xeb/0x120 [ 2343.594134] softirqs last enabled at (10711758): [] __do_softirq+0x2be/0x371 [ 2343.602844] softirqs last disabled at (10711753): [] irq_exit_rcu+0xb0/0xd0 [ 2343.611381] ---[ end trace 0000000000000000 ]--- [ 2343.616044] ------------[ cut here ]------------ [ 2343.620662] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2343.632322] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2343.641014] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2343.696507] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2343.705633] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2343.713112] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2343.718623] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2343.724108] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2343.742864] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2343.748108] RAX: 0000000000000000 RBX: ffff8881d8075808 RCX: 0000000000000000 [ 2343.755251] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2343.762399] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008777 [ 2343.769544] R10: 00000000002c3eb0 R11: ffffc9000030b828 R12: ffff8881d8075f58 [ 2343.776676] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2343.783810] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2343.791912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2343.797674] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2343.804806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2343.811948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2343.819091] Call Trace: [ 2343.821560] [ 2343.823666] ? __warn+0x85/0x180 [ 2343.826901] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.831796] ? report_bug+0xfc/0x1d0 [ 2343.835394] ? prb_read_valid+0x1b/0x20 [ 2343.839251] ? handle_bug+0x44/0x70 [ 2343.842760] ? exc_invalid_op+0x17/0x60 [ 2343.846617] ? asm_exc_invalid_op+0x1a/0x20 [ 2343.850823] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.855701] ? __list_add_valid_or_report+0x7d/0xa0 [ 2343.860578] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2343.865468] ceph_add_cap+0x20b/0x620 [ceph] [ 2343.869771] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2343.874499] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2343.879830] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2343.884316] ceph_con_process_message+0x74/0x150 [libceph] [ 2343.889811] ? calc_signature+0xdb/0x100 [libceph] [ 2343.894632] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2343.899974] ? process_scheduled_works+0x22a/0x510 [ 2343.904763] ? process_scheduled_works+0x22a/0x510 [ 2343.909572] ceph_con_workfn+0x344/0x750 [libceph] [ 2343.914377] ? process_scheduled_works+0x22a/0x510 [ 2343.919182] process_scheduled_works+0x2a4/0x510 [ 2343.923842] worker_thread+0x188/0x340 [ 2343.927591] ? rescuer_thread+0x310/0x310 [ 2343.931603] kthread+0xf5/0x130 [ 2343.934748] ? kthread_complete_and_exit+0x20/0x20 [ 2343.939561] ret_from_fork+0x31/0x40 [ 2343.943138] ? kthread_complete_and_exit+0x20/0x20 [ 2343.947949] ret_from_fork_asm+0x11/0x20 [ 2343.951916] [ 2343.954111] irq event stamp: 10712805 [ 2343.957794] hardirqs last enabled at (10712815): [] console_unlock+0x106/0x120 [ 2343.966681] hardirqs last disabled at (10712830): [] console_unlock+0xeb/0x120 [ 2343.975457] softirqs last enabled at (10712828): [] __do_softirq+0x2be/0x371 [ 2343.984151] softirqs last disabled at (10712823): [] irq_exit_rcu+0xb0/0xd0 [ 2343.992687] ---[ end trace 0000000000000000 ]--- [ 2343.997330] ------------[ cut here ]------------ [ 2344.001944] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2344.013588] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2344.022284] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2344.077778] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2344.086920] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2344.094408] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2344.099950] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2344.105453] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2344.124197] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2344.129441] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2344.136574] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2344.143724] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000087b1 [ 2344.150855] R10: 00000000002c4ef0 R11: ffffc9000030b828 R12: ffff8881d1295590 [ 2344.157999] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2344.165149] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2344.173250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.179006] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2344.186155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2344.193297] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2344.200456] Call Trace: [ 2344.202908] [ 2344.205025] ? __warn+0x85/0x180 [ 2344.208275] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.213170] ? report_bug+0xfc/0x1d0 [ 2344.216768] ? prb_read_valid+0x1b/0x20 [ 2344.220626] ? handle_bug+0x44/0x70 [ 2344.224116] ? exc_invalid_op+0x17/0x60 [ 2344.227966] ? asm_exc_invalid_op+0x1a/0x20 [ 2344.232191] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.237065] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.241960] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2344.246871] ceph_add_cap+0x20b/0x620 [ceph] [ 2344.251189] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2344.255927] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2344.261271] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2344.265761] ceph_con_process_message+0x74/0x150 [libceph] [ 2344.271257] ? calc_signature+0xdb/0x100 [libceph] [ 2344.276087] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2344.281458] ? process_scheduled_works+0x22a/0x510 [ 2344.286269] ? process_scheduled_works+0x22a/0x510 [ 2344.291060] ceph_con_workfn+0x344/0x750 [libceph] [ 2344.295880] ? process_scheduled_works+0x22a/0x510 [ 2344.300688] process_scheduled_works+0x2a4/0x510 [ 2344.305330] worker_thread+0x188/0x340 [ 2344.309079] ? rescuer_thread+0x310/0x310 [ 2344.313108] kthread+0xf5/0x130 [ 2344.316272] ? kthread_complete_and_exit+0x20/0x20 [ 2344.321081] ret_from_fork+0x31/0x40 [ 2344.324677] ? kthread_complete_and_exit+0x20/0x20 [ 2344.329472] ret_from_fork_asm+0x11/0x20 [ 2344.333419] [ 2344.335607] irq event stamp: 10713835 [ 2344.339274] hardirqs last enabled at (10713851): [] console_unlock+0x106/0x120 [ 2344.348158] hardirqs last disabled at (10713860): [] console_unlock+0xeb/0x120 [ 2344.356953] softirqs last enabled at (10713848): [] __do_softirq+0x2be/0x371 [ 2344.365664] softirqs last disabled at (10713843): [] irq_exit_rcu+0xb0/0xd0 [ 2344.374183] ---[ end trace 0000000000000000 ]--- [ 2344.378843] ------------[ cut here ]------------ [ 2344.383460] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2344.395118] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2344.403808] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2344.459299] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2344.468460] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2344.475939] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2344.481461] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2344.486945] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2344.505701] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2344.510929] RAX: 0000000000000000 RBX: ffff8881d1291390 RCX: 0000000000000000 [ 2344.518078] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2344.525227] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000087eb [ 2344.532371] R10: 00000000002c5f10 R11: ffffc9000030b828 R12: ffff8881d1291ae0 [ 2344.539521] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2344.546652] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2344.554751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.560510] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2344.567644] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2344.574777] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2344.581925] Call Trace: [ 2344.584387] [ 2344.586504] ? __warn+0x85/0x180 [ 2344.589752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.594652] ? report_bug+0xfc/0x1d0 [ 2344.598229] ? prb_read_valid+0x1b/0x20 [ 2344.602086] ? handle_bug+0x44/0x70 [ 2344.605596] ? exc_invalid_op+0x17/0x60 [ 2344.609455] ? asm_exc_invalid_op+0x1a/0x20 [ 2344.613643] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.618518] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.623398] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2344.628306] ceph_add_cap+0x20b/0x620 [ceph] [ 2344.632609] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2344.637353] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2344.642681] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2344.647160] ceph_con_process_message+0x74/0x150 [libceph] [ 2344.652656] ? calc_signature+0xdb/0x100 [libceph] [ 2344.657467] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2344.662807] ? process_scheduled_works+0x22a/0x510 [ 2344.667600] ? process_scheduled_works+0x22a/0x510 [ 2344.672391] ceph_con_workfn+0x344/0x750 [libceph] [ 2344.677213] ? process_scheduled_works+0x22a/0x510 [ 2344.682021] process_scheduled_works+0x2a4/0x510 [ 2344.686663] worker_thread+0x188/0x340 [ 2344.690418] ? rescuer_thread+0x310/0x310 [ 2344.694463] kthread+0xf5/0x130 [ 2344.697611] ? kthread_complete_and_exit+0x20/0x20 [ 2344.702406] ret_from_fork+0x31/0x40 [ 2344.706001] ? kthread_complete_and_exit+0x20/0x20 [ 2344.710811] ret_from_fork_asm+0x11/0x20 [ 2344.714778] [ 2344.716963] irq event stamp: 10714849 [ 2344.720648] hardirqs last enabled at (10714865): [] console_unlock+0x106/0x120 [ 2344.729513] hardirqs last disabled at (10714874): [] console_unlock+0xeb/0x120 [ 2344.738292] softirqs last enabled at (10714862): [] __do_softirq+0x2be/0x371 [ 2344.747003] softirqs last disabled at (10714857): [] irq_exit_rcu+0xb0/0xd0 [ 2344.755541] ---[ end trace 0000000000000000 ]--- [ 2344.760181] ------------[ cut here ]------------ [ 2344.764796] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2344.776471] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2344.785166] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2344.840658] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2344.849784] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2344.857280] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2344.862793] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2344.868294] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2344.887060] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2344.892301] RAX: 0000000000000000 RBX: ffff8881d0bec478 RCX: 0000000000000000 [ 2344.899462] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2344.906593] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008825 [ 2344.913723] R10: 00000000002c6f40 R11: ffffc9000030b828 R12: ffff8881d0becbc8 [ 2344.920850] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2344.927992] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2344.936108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.941850] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2344.948993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2344.956143] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2344.963295] Call Trace: [ 2344.965764] [ 2344.967887] ? __warn+0x85/0x180 [ 2344.971136] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.976034] ? report_bug+0xfc/0x1d0 [ 2344.979632] ? prb_read_valid+0x1b/0x20 [ 2344.983472] ? handle_bug+0x44/0x70 [ 2344.986961] ? exc_invalid_op+0x17/0x60 [ 2344.990811] ? asm_exc_invalid_op+0x1a/0x20 [ 2344.995036] ? __list_add_valid_or_report+0x7d/0xa0 [ 2344.999912] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.004806] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2345.009716] ceph_add_cap+0x20b/0x620 [ceph] [ 2345.014017] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2345.018777] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2345.024125] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2345.028587] ceph_con_process_message+0x74/0x150 [libceph] [ 2345.034084] ? calc_signature+0xdb/0x100 [libceph] [ 2345.038887] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2345.044246] ? process_scheduled_works+0x22a/0x510 [ 2345.049036] ? process_scheduled_works+0x22a/0x510 [ 2345.053844] ceph_con_workfn+0x344/0x750 [libceph] [ 2345.058664] ? process_scheduled_works+0x22a/0x510 [ 2345.063470] process_scheduled_works+0x2a4/0x510 [ 2345.068106] worker_thread+0x188/0x340 [ 2345.071856] ? rescuer_thread+0x310/0x310 [ 2345.075884] kthread+0xf5/0x130 [ 2345.079047] ? kthread_complete_and_exit+0x20/0x20 [ 2345.083859] ret_from_fork+0x31/0x40 [ 2345.087455] ? kthread_complete_and_exit+0x20/0x20 [ 2345.092248] ret_from_fork_asm+0x11/0x20 [ 2345.096215] [ 2345.098401] irq event stamp: 10715871 [ 2345.102082] hardirqs last enabled at (10715881): [] console_unlock+0x106/0x120 [ 2345.110968] hardirqs last disabled at (10715896): [] console_unlock+0xeb/0x120 [ 2345.119764] softirqs last enabled at (10715894): [] __do_softirq+0x2be/0x371 [ 2345.128479] softirqs last disabled at (10715915): [] irq_exit_rcu+0xb0/0xd0 [ 2345.136993] ---[ end trace 0000000000000000 ]--- [ 2345.141762] ------------[ cut here ]------------ [ 2345.146381] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2345.158058] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2345.166755] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2345.222252] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2345.231392] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2345.238889] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2345.244401] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2345.249903] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2345.268661] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2345.273887] RAX: 0000000000000000 RBX: ffff8881c2e430e8 RCX: 0000000000000000 [ 2345.281037] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2345.288195] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000885f [ 2345.295333] R10: 00000000002c7f80 R11: ffffc9000030b828 R12: ffff8881c2e43838 [ 2345.302467] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2345.309613] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2345.317709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2345.323475] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2345.330615] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2345.337747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2345.344879] Call Trace: [ 2345.347333] [ 2345.349438] ? __warn+0x85/0x180 [ 2345.352696] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.357574] ? report_bug+0xfc/0x1d0 [ 2345.361153] ? prb_read_valid+0x1b/0x20 [ 2345.365009] ? handle_bug+0x44/0x70 [ 2345.368520] ? exc_invalid_op+0x17/0x60 [ 2345.372358] ? asm_exc_invalid_op+0x1a/0x20 [ 2345.376567] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.381470] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.386356] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2345.391273] ceph_add_cap+0x20b/0x620 [ceph] [ 2345.395574] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2345.400320] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2345.405647] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2345.410128] ceph_con_process_message+0x74/0x150 [libceph] [ 2345.415634] ? calc_signature+0xdb/0x100 [libceph] [ 2345.420454] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2345.425783] ? process_scheduled_works+0x22a/0x510 [ 2345.430576] ? process_scheduled_works+0x22a/0x510 [ 2345.435365] ceph_con_workfn+0x344/0x750 [libceph] [ 2345.440188] ? process_scheduled_works+0x22a/0x510 [ 2345.444995] process_scheduled_works+0x2a4/0x510 [ 2345.449640] worker_thread+0x188/0x340 [ 2345.453394] ? rescuer_thread+0x310/0x310 [ 2345.457424] kthread+0xf5/0x130 [ 2345.460588] ? kthread_complete_and_exit+0x20/0x20 [ 2345.465380] ret_from_fork+0x31/0x40 [ 2345.468977] ? kthread_complete_and_exit+0x20/0x20 [ 2345.473787] ret_from_fork_asm+0x11/0x20 [ 2345.477754] [ 2345.479960] irq event stamp: 10716911 [ 2345.483641] hardirqs last enabled at (10716921): [] console_unlock+0x106/0x120 [ 2345.492506] hardirqs last disabled at (10716930): [] console_unlock+0xeb/0x120 [ 2345.501287] softirqs last enabled at (10716908): [] __do_softirq+0x2be/0x371 [ 2345.509982] softirqs last disabled at (10716903): [] irq_exit_rcu+0xb0/0xd0 [ 2345.518517] ---[ end trace 0000000000000000 ]--- [ 2345.523206] ------------[ cut here ]------------ [ 2345.527824] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2345.539495] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2345.548194] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2345.603702] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2345.612829] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2345.620326] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2345.625836] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2345.631341] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2345.650105] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2345.655347] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2345.662488] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2345.669622] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008899 [ 2345.676771] R10: 00000000002c8f90 R11: ffffc9000030b828 R12: ffff8881c5357cb0 [ 2345.683913] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2345.691055] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2345.699159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2345.704922] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2345.712071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2345.719221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2345.726363] Call Trace: [ 2345.728825] [ 2345.730940] ? __warn+0x85/0x180 [ 2345.734189] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.739086] ? report_bug+0xfc/0x1d0 [ 2345.742684] ? prb_read_valid+0x1b/0x20 [ 2345.746525] ? handle_bug+0x44/0x70 [ 2345.750015] ? exc_invalid_op+0x17/0x60 [ 2345.753873] ? asm_exc_invalid_op+0x1a/0x20 [ 2345.758098] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.762974] ? __list_add_valid_or_report+0x7d/0xa0 [ 2345.767869] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2345.772796] ceph_add_cap+0x20b/0x620 [ceph] [ 2345.777097] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2345.781836] ? lock_acquire+0xcb/0x2b0 [ 2345.785589] ceph_fill_trace+0x143/0xf30 [ceph] [ 2345.790127] ? __mutex_lock+0x97/0xce0 [ 2345.793896] ? __mutex_lock+0xaec/0xce0 [ 2345.797765] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2345.802329] ? lock_downgrade+0xfd/0x250 [ 2345.806294] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2345.810673] ceph_con_process_message+0x74/0x150 [libceph] [ 2345.816169] ? calc_signature+0xdb/0x100 [libceph] [ 2345.820992] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2345.826356] ? process_scheduled_works+0x22a/0x510 [ 2345.831149] ? process_scheduled_works+0x22a/0x510 [ 2345.835978] ceph_con_workfn+0x344/0x750 [libceph] [ 2345.840804] ? process_scheduled_works+0x22a/0x510 [ 2345.845593] process_scheduled_works+0x2a4/0x510 [ 2345.850236] worker_thread+0x188/0x340 [ 2345.853992] ? rescuer_thread+0x310/0x310 [ 2345.858021] kthread+0xf5/0x130 [ 2345.861185] ? kthread_complete_and_exit+0x20/0x20 [ 2345.865996] ret_from_fork+0x31/0x40 [ 2345.869594] ? kthread_complete_and_exit+0x20/0x20 [ 2345.874387] ret_from_fork_asm+0x11/0x20 [ 2345.878352] [ 2345.880537] irq event stamp: 10718095 [ 2345.884203] hardirqs last enabled at (10718105): [] console_unlock+0x106/0x120 [ 2345.893087] hardirqs last disabled at (10718114): [] console_unlock+0xeb/0x120 [ 2345.901883] softirqs last enabled at (10718072): [] __do_softirq+0x2be/0x371 [ 2345.910595] softirqs last disabled at (10718067): [] irq_exit_rcu+0xb0/0xd0 [ 2345.919115] ---[ end trace 0000000000000000 ]--- [ 2345.923803] ------------[ cut here ]------------ [ 2345.928424] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2345.940071] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2345.948764] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2346.004243] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2346.013366] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2346.020847] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2346.026350] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2346.031834] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2346.050581] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2346.055807] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2346.062939] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2346.070074] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000088d8 [ 2346.077205] R10: 00000000002ca098 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2346.084339] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2346.091472] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2346.099556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2346.105304] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2346.112438] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2346.119569] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2346.126702] Call Trace: [ 2346.129154] [ 2346.131261] ? __warn+0x85/0x180 [ 2346.134495] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.139374] ? report_bug+0xfc/0x1d0 [ 2346.142952] ? prb_read_valid+0x1b/0x20 [ 2346.146792] ? handle_bug+0x44/0x70 [ 2346.150284] ? exc_invalid_op+0x17/0x60 [ 2346.154125] ? asm_exc_invalid_op+0x1a/0x20 [ 2346.158314] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.163191] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.168068] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2346.172960] ceph_add_cap+0x20b/0x620 [ceph] [ 2346.177253] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2346.181977] ? lock_acquire+0xcb/0x2b0 [ 2346.185737] ceph_fill_trace+0x143/0xf30 [ceph] [ 2346.190275] ? __mutex_lock+0x97/0xce0 [ 2346.194026] ? __mutex_lock+0xaec/0xce0 [ 2346.197867] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2346.202415] ? lock_downgrade+0xfd/0x250 [ 2346.206345] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2346.210726] ceph_con_process_message+0x74/0x150 [libceph] [ 2346.216223] ? calc_signature+0xdb/0x100 [libceph] [ 2346.221036] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2346.226365] ? process_scheduled_works+0x22a/0x510 [ 2346.231158] ? process_scheduled_works+0x22a/0x510 [ 2346.235946] ceph_con_workfn+0x344/0x750 [libceph] [ 2346.240750] ? process_scheduled_works+0x22a/0x510 [ 2346.245541] process_scheduled_works+0x2a4/0x510 [ 2346.250165] worker_thread+0x188/0x340 [ 2346.253914] ? rescuer_thread+0x310/0x310 [ 2346.257927] kthread+0xf5/0x130 [ 2346.261075] ? kthread_complete_and_exit+0x20/0x20 [ 2346.265866] ret_from_fork+0x31/0x40 [ 2346.269444] ? kthread_complete_and_exit+0x20/0x20 [ 2346.274237] ret_from_fork_asm+0x11/0x20 [ 2346.278170] [ 2346.280356] irq event stamp: 10719201 [ 2346.284021] hardirqs last enabled at (10719211): [] console_unlock+0x106/0x120 [ 2346.292889] hardirqs last disabled at (10719220): [] console_unlock+0xeb/0x120 [ 2346.301667] softirqs last enabled at (10719148): [] __do_softirq+0x2be/0x371 [ 2346.310361] softirqs last disabled at (10719143): [] irq_exit_rcu+0xb0/0xd0 [ 2346.318881] ---[ end trace 0000000000000000 ]--- [ 2346.323501] ------------[ cut here ]------------ [ 2346.328120] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2346.339767] WARNING: CPU: 2 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2346.348718] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2346.404189] CPU: 2 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2346.413140] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2346.420617] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2346.426106] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2346.444852] RSP: 0018:ffffc90001f7fab8 EFLAGS: 00010286 [ 2346.450076] RAX: 0000000000000000 RBX: ffff8881c5357560 RCX: 0000000000000000 [ 2346.457209] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2346.464333] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008917 [ 2346.471460] R10: 00000000002cb2f0 R11: ffffc90001f7f948 R12: ffff8881c5357cb0 [ 2346.478590] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2346.485715] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2346.493802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2346.499547] CR2: 00007f04ff65d934 CR3: 0000000113532002 CR4: 00000000003706e0 [ 2346.506671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2346.513795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2346.520921] Call Trace: [ 2346.523373] [ 2346.525479] ? __warn+0x85/0x180 [ 2346.528713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.533594] ? report_bug+0xfc/0x1d0 [ 2346.537170] ? prb_read_valid+0x1b/0x20 [ 2346.541012] ? handle_bug+0x44/0x70 [ 2346.544503] ? exc_invalid_op+0x17/0x60 [ 2346.548343] ? asm_exc_invalid_op+0x1a/0x20 [ 2346.552533] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.557408] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.562286] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2346.567178] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2346.572420] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2346.577222] ceph_atomic_open+0x28d/0xe80 [ceph] [ 2346.581856] lookup_open+0x37b/0x630 [ 2346.585441] ? verify_cpu+0x20/0x100 [ 2346.589017] ? verify_cpu+0x20/0x100 [ 2346.592598] path_openat+0x25a/0x9d0 [ 2346.596178] do_filp_open+0xb8/0x120 [ 2346.599760] ? lock_release+0xc5/0x260 [ 2346.603510] ? _raw_spin_unlock+0x23/0x40 [ 2346.607519] ? alloc_fd+0x13f/0x200 [ 2346.611013] do_sys_openat2+0x249/0x290 [ 2346.614853] do_sys_open+0x4f/0x80 [ 2346.618258] do_syscall_64+0x38/0x80 [ 2346.621839] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2346.626890] RIP: 0033:0x7f48f0eed2f2 [ 2346.630475] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2346.649214] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2346.656782] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2346.663915] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2346.671045] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2346.678170] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2346.685294] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2346.692425] [ 2346.694613] irq event stamp: 12957407 [ 2346.698277] hardirqs last enabled at (12957417): [] console_unlock+0x106/0x120 [ 2346.707143] hardirqs last disabled at (12957426): [] console_unlock+0xeb/0x120 [ 2346.715922] softirqs last enabled at (12957354): [] __do_softirq+0x2be/0x371 [ 2346.724616] softirqs last disabled at (12957349): [] irq_exit_rcu+0xb0/0xd0 [ 2346.733136] ---[ end trace 0000000000000000 ]--- [ 2346.737755] ------------[ cut here ]------------ [ 2346.742373] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2346.754039] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2346.762736] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2346.818214] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2346.827355] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2346.834853] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2346.840371] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2346.845873] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2346.864639] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2346.869865] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2346.877015] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2346.884165] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008954 [ 2346.891314] R10: 00000000002cc3b8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2346.898455] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2346.905609] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2346.913700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2346.919492] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2346.926627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2346.933756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2346.940899] Call Trace: [ 2346.943368] [ 2346.945506] ? __warn+0x85/0x180 [ 2346.948744] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.953621] ? report_bug+0xfc/0x1d0 [ 2346.957200] ? prb_read_valid+0x1b/0x20 [ 2346.961058] ? handle_bug+0x44/0x70 [ 2346.964570] ? exc_invalid_op+0x17/0x60 [ 2346.968407] ? asm_exc_invalid_op+0x1a/0x20 [ 2346.972616] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.977508] ? __list_add_valid_or_report+0x7d/0xa0 [ 2346.982386] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2346.987293] ceph_add_cap+0x20b/0x620 [ceph] [ 2346.991597] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2346.996336] ? lock_acquire+0xcb/0x2b0 [ 2347.000104] ceph_fill_trace+0x143/0xf30 [ceph] [ 2347.004644] ? __mutex_lock+0x97/0xce0 [ 2347.008394] ? __mutex_lock+0xaec/0xce0 [ 2347.012262] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2347.016803] ? lock_downgrade+0xfd/0x250 [ 2347.020751] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2347.025147] ceph_con_process_message+0x74/0x150 [libceph] [ 2347.030643] ? calc_signature+0xdb/0x100 [libceph] [ 2347.035447] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2347.040794] ? process_scheduled_works+0x22a/0x510 [ 2347.045588] ? process_scheduled_works+0x22a/0x510 [ 2347.050378] ceph_con_workfn+0x344/0x750 [libceph] [ 2347.055198] ? process_scheduled_works+0x22a/0x510 [ 2347.060005] process_scheduled_works+0x2a4/0x510 [ 2347.064649] worker_thread+0x188/0x340 [ 2347.068407] ? rescuer_thread+0x310/0x310 [ 2347.072415] kthread+0xf5/0x130 [ 2347.075582] ? kthread_complete_and_exit+0x20/0x20 [ 2347.080374] ret_from_fork+0x31/0x40 [ 2347.083970] ? kthread_complete_and_exit+0x20/0x20 [ 2347.088788] ret_from_fork_asm+0x11/0x20 [ 2347.092741] [ 2347.094934] irq event stamp: 10722245 [ 2347.098617] hardirqs last enabled at (10722255): [] console_unlock+0x106/0x120 [ 2347.107500] hardirqs last disabled at (10722264): [] console_unlock+0xeb/0x120 [ 2347.116279] softirqs last enabled at (10722232): [] __do_softirq+0x2be/0x371 [ 2347.124991] softirqs last disabled at (10722227): [] irq_exit_rcu+0xb0/0xd0 [ 2347.133527] ---[ end trace 0000000000000000 ]--- [ 2347.138335] ------------[ cut here ]------------ [ 2347.142959] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2347.154601] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2347.163299] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2347.218776] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2347.227901] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2347.235380] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2347.240875] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2347.246359] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2347.265106] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2347.270332] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2347.277467] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2347.284599] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008993 [ 2347.291730] R10: 00000000002cd4d0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2347.298864] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2347.305998] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2347.314082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2347.319828] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2347.326962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2347.334096] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2347.341228] Call Trace: [ 2347.343680] [ 2347.345788] ? __warn+0x85/0x180 [ 2347.349021] ? __list_add_valid_or_report+0x7d/0xa0 [ 2347.353900] ? report_bug+0xfc/0x1d0 [ 2347.357478] ? prb_read_valid+0x1b/0x20 [ 2347.361326] ? handle_bug+0x44/0x70 [ 2347.364819] ? exc_invalid_op+0x17/0x60 [ 2347.368658] ? asm_exc_invalid_op+0x1a/0x20 [ 2347.372850] ? __list_add_valid_or_report+0x7d/0xa0 [ 2347.377734] ? __list_add_valid_or_report+0x7d/0xa0 [ 2347.382612] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2347.387502] ceph_add_cap+0x20b/0x620 [ceph] [ 2347.391787] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2347.396511] ? lock_acquire+0xcb/0x2b0 [ 2347.400273] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2347.404729] ? __mutex_lock+0x97/0xce0 [ 2347.408487] ? __mutex_lock+0xaec/0xce0 [ 2347.412325] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2347.416872] ? lock_downgrade+0xfd/0x250 [ 2347.420803] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2347.425182] ceph_con_process_message+0x74/0x150 [libceph] [ 2347.430676] ? calc_signature+0xdb/0x100 [libceph] [ 2347.435487] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2347.440820] ? process_scheduled_works+0x22a/0x510 [ 2347.445615] ? process_scheduled_works+0x22a/0x510 [ 2347.450403] ceph_con_workfn+0x344/0x750 [libceph] [ 2347.455207] ? process_scheduled_works+0x22a/0x510 [ 2347.459998] process_scheduled_works+0x2a4/0x510 [ 2347.464623] worker_thread+0x188/0x340 [ 2347.468379] ? rescuer_thread+0x310/0x310 [ 2347.472390] kthread+0xf5/0x130 [ 2347.475539] ? kthread_complete_and_exit+0x20/0x20 [ 2347.480332] ret_from_fork+0x31/0x40 [ 2347.483910] ? kthread_complete_and_exit+0x20/0x20 [ 2347.488702] ret_from_fork_asm+0x11/0x20 [ 2347.492636] [ 2347.494830] irq event stamp: 10723407 [ 2347.498495] hardirqs last enabled at (10723417): [] console_unlock+0x106/0x120 [ 2347.507363] hardirqs last disabled at (10723426): [] console_unlock+0xeb/0x120 [ 2347.516142] softirqs last enabled at (10723354): [] __do_softirq+0x2be/0x371 [ 2347.524833] softirqs last disabled at (10723349): [] irq_exit_rcu+0xb0/0xd0 [ 2347.533353] ---[ end trace 0000000000000000 ]--- [ 2347.537974] ------------[ cut here ]------------ [ 2347.542593] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2347.554238] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2347.563193] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2347.618663] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2347.627612] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2347.635093] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2347.640577] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2347.659325] RSP: 0018:ffffc90001fdfb78 EFLAGS: 00010286 [ 2347.664550] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2347.671674] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2347.678800] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000089d2 [ 2347.685933] R10: 00000000002ce5e8 R11: ffffc90001fdfa08 R12: ffff8881c5350750 [ 2347.693064] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2347.700198] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2347.708283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2347.714029] CR2: 00007f9bd22d7020 CR3: 0000000179402005 CR4: 00000000003706e0 [ 2347.721153] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2347.728277] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2347.735401] Call Trace: [ 2347.737847] [ 2347.739954] ? __warn+0x85/0x180 [ 2347.743185] ? __list_add_valid_or_report+0x7d/0xa0 [ 2347.748065] ? report_bug+0xfc/0x1d0 [ 2347.751645] ? prb_read_valid+0x1b/0x20 [ 2347.755485] ? handle_bug+0x44/0x70 [ 2347.758976] ? exc_invalid_op+0x17/0x60 [ 2347.762815] ? asm_exc_invalid_op+0x1a/0x20 [ 2347.767006] ? __list_add_valid_or_report+0x7d/0xa0 [ 2347.771883] ? __list_add_valid_or_report+0x7d/0xa0 [ 2347.776760] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2347.781652] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2347.786897] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2347.791704] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2347.796420] ? __d_lookup+0xd6/0x200 [ 2347.799997] lookup_fast+0x6d/0xf0 [ 2347.803404] walk_component+0x26/0x160 [ 2347.807158] link_path_walk+0x24d/0x380 [ 2347.810996] ? path_openat+0x80/0x9d0 [ 2347.814665] path_openat+0x8e/0x9d0 [ 2347.818158] do_filp_open+0xb8/0x120 [ 2347.821737] ? lock_release+0xc5/0x260 [ 2347.825486] ? _raw_spin_unlock+0x23/0x40 [ 2347.829496] ? alloc_fd+0x13f/0x200 [ 2347.832984] do_sys_openat2+0x249/0x290 [ 2347.836823] do_sys_open+0x4f/0x80 [ 2347.840228] do_syscall_64+0x38/0x80 [ 2347.843807] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2347.848860] RIP: 0033:0x7f48f0eed2f2 [ 2347.852441] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2347.871185] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2347.878756] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2347.885886] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2347.893017] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2347.900149] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2347.907273] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2347.914403] [ 2347.916590] irq event stamp: 12870933 [ 2347.920249] hardirqs last enabled at (12870943): [] console_unlock+0x106/0x120 [ 2347.929115] hardirqs last disabled at (12870952): [] console_unlock+0xeb/0x120 [ 2347.937894] softirqs last enabled at (12870870): [] __do_softirq+0x2be/0x371 [ 2347.946588] softirqs last disabled at (12870971): [] irq_exit_rcu+0xb0/0xd0 [ 2347.955107] ---[ end trace 0000000000000000 ]--- [ 2347.959727] ------------[ cut here ]------------ [ 2347.964355] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2347.976003] WARNING: CPU: 0 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2347.984956] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2348.040425] CPU: 0 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2348.049377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2348.056856] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2348.062340] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2348.081087] RSP: 0018:ffffc90001fd7d50 EFLAGS: 00010286 [ 2348.086312] RAX: 0000000000000000 RBX: ffff8881c2cef560 RCX: 0000000000000000 [ 2348.093437] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2348.100569] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008a11 [ 2348.107705] R10: 00000000002cf700 R11: ffffc90001fd7be0 R12: ffff8881c2cefcb0 [ 2348.114845] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2348.121978] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2348.130063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2348.135809] CR2: 00007f4f3ea07000 CR3: 0000000120c1e005 CR4: 00000000003706f0 [ 2348.142942] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2348.150074] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2348.157199] Call Trace: [ 2348.159643] [ 2348.161741] ? __warn+0x85/0x180 [ 2348.164973] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.169854] ? report_bug+0xfc/0x1d0 [ 2348.173432] ? prb_read_valid+0x1b/0x20 [ 2348.177275] ? handle_bug+0x44/0x70 [ 2348.180764] ? exc_invalid_op+0x17/0x60 [ 2348.184604] ? asm_exc_invalid_op+0x1a/0x20 [ 2348.188795] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.193671] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2348.198560] ceph_readdir+0xc0/0x1670 [ceph] [ 2348.202842] ? wrap_directory_iterator+0x38/0x70 [ 2348.207459] ? wrap_directory_iterator+0x30/0x70 [ 2348.212077] ? lock_is_held_type+0xaa/0x120 [ 2348.216266] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2348.221152] wrap_directory_iterator+0x47/0x70 [ 2348.225601] iterate_dir+0x8d/0x170 [ 2348.229092] __x64_sys_getdents64+0x87/0x140 [ 2348.233365] ? __x64_sys_getdents64+0x140/0x140 [ 2348.237901] do_syscall_64+0x38/0x80 [ 2348.241476] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2348.246529] RIP: 0033:0x7f48f0ec480b [ 2348.250108] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2348.268853] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2348.276420] RAX: ffffffffffffffda RBX: 000000000207e000 RCX: 00007f48f0ec480b [ 2348.283553] RDX: 0000000000010000 RSI: 000000000207e030 RDI: 0000000000000005 [ 2348.290686] RBP: 000000000207e030 R08: 00000000000000db R09: 0000000000000003 [ 2348.297818] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2348.304952] R13: 0000000000000002 R14: 00007f48f183d8a0 R15: 00000000020766e0 [ 2348.312091] [ 2348.314276] irq event stamp: 13008683 [ 2348.317933] hardirqs last enabled at (13008693): [] console_unlock+0x106/0x120 [ 2348.326801] hardirqs last disabled at (13008702): [] console_unlock+0xeb/0x120 [ 2348.335580] softirqs last enabled at (13008660): [] __do_softirq+0x2be/0x371 [ 2348.344273] softirqs last disabled at (13008655): [] irq_exit_rcu+0xb0/0xd0 [ 2348.352792] ---[ end trace 0000000000000000 ]--- [ 2348.357413] ------------[ cut here ]------------ [ 2348.362033] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2348.373683] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2348.382385] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2348.437862] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2348.446996] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2348.454473] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2348.459978] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2348.465470] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2348.484220] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2348.489443] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2348.496580] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2348.503713] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008a49 [ 2348.510845] R10: 00000000002d0708 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2348.517976] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2348.525111] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2348.533203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2348.538951] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2348.546083] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2348.553217] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2348.560346] Call Trace: [ 2348.562815] [ 2348.564925] ? __warn+0x85/0x180 [ 2348.568162] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.573046] ? report_bug+0xfc/0x1d0 [ 2348.576626] ? prb_read_valid+0x1b/0x20 [ 2348.580467] ? handle_bug+0x44/0x70 [ 2348.583957] ? exc_invalid_op+0x17/0x60 [ 2348.587810] ? asm_exc_invalid_op+0x1a/0x20 [ 2348.592008] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.596888] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.601769] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2348.606686] ceph_add_cap+0x20b/0x620 [ceph] [ 2348.610989] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2348.615724] ceph_fill_trace+0x143/0xf30 [ceph] [ 2348.620273] ? __mutex_lock+0x97/0xce0 [ 2348.624022] ? __mutex_lock+0xaec/0xce0 [ 2348.627868] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2348.632416] ? lock_downgrade+0xfd/0x250 [ 2348.636357] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2348.640747] ceph_con_process_message+0x74/0x150 [libceph] [ 2348.646254] ? calc_signature+0xdb/0x100 [libceph] [ 2348.651067] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2348.656409] ? process_scheduled_works+0x22a/0x510 [ 2348.661209] ? process_scheduled_works+0x22a/0x510 [ 2348.666007] ceph_con_workfn+0x344/0x750 [libceph] [ 2348.670829] ? process_scheduled_works+0x22a/0x510 [ 2348.675626] process_scheduled_works+0x2a4/0x510 [ 2348.680253] worker_thread+0x188/0x340 [ 2348.684010] ? rescuer_thread+0x310/0x310 [ 2348.688026] kthread+0xf5/0x130 [ 2348.691177] ? kthread_complete_and_exit+0x20/0x20 [ 2348.695980] ret_from_fork+0x31/0x40 [ 2348.699565] ? kthread_complete_and_exit+0x20/0x20 [ 2348.704355] ret_from_fork_asm+0x11/0x20 [ 2348.708292] [ 2348.710482] irq event stamp: 10726217 [ 2348.714151] hardirqs last enabled at (10726227): [] console_unlock+0x106/0x120 [ 2348.723023] hardirqs last disabled at (10726236): [] console_unlock+0xeb/0x120 [ 2348.731819] softirqs last enabled at (10726174): [] __do_softirq+0x2be/0x371 [ 2348.740532] softirqs last disabled at (10726169): [] irq_exit_rcu+0xb0/0xd0 [ 2348.749052] ---[ end trace 0000000000000000 ]--- [ 2348.753715] ------------[ cut here ]------------ [ 2348.758335] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2348.769981] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2348.778676] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2348.834161] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2348.843285] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2348.850763] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2348.856270] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2348.861752] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2348.880501] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2348.885751] RAX: 0000000000000000 RBX: ffff8881d12d09c8 RCX: 0000000000000000 [ 2348.892886] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2348.900022] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008a87 [ 2348.907150] R10: 00000000002d17e0 R11: ffffc9000030b828 R12: ffff8881d12d1118 [ 2348.914285] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2348.921417] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2348.929503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2348.935271] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2348.942413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2348.949562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2348.956710] Call Trace: [ 2348.959164] [ 2348.961267] ? __warn+0x85/0x180 [ 2348.964520] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.969405] ? report_bug+0xfc/0x1d0 [ 2348.973000] ? prb_read_valid+0x1b/0x20 [ 2348.976857] ? handle_bug+0x44/0x70 [ 2348.980365] ? exc_invalid_op+0x17/0x60 [ 2348.984214] ? asm_exc_invalid_op+0x1a/0x20 [ 2348.988440] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.993317] ? __list_add_valid_or_report+0x7d/0xa0 [ 2348.998212] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2349.003119] ceph_add_cap+0x20b/0x620 [ceph] [ 2349.007439] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2349.012169] ? lock_acquire+0xcb/0x2b0 [ 2349.015940] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2349.020389] ? __mutex_lock+0x97/0xce0 [ 2349.024159] ? __mutex_lock+0xaec/0xce0 [ 2349.028011] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2349.032575] ? lock_downgrade+0xfd/0x250 [ 2349.036528] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2349.040902] ceph_con_process_message+0x74/0x150 [libceph] [ 2349.046399] ? calc_signature+0xdb/0x100 [libceph] [ 2349.051237] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2349.056560] ? process_scheduled_works+0x22a/0x510 [ 2349.061368] ? process_scheduled_works+0x22a/0x510 [ 2349.066159] ceph_con_workfn+0x344/0x750 [libceph] [ 2349.070963] ? process_scheduled_works+0x22a/0x510 [ 2349.075771] process_scheduled_works+0x2a4/0x510 [ 2349.080411] worker_thread+0x188/0x340 [ 2349.084162] ? rescuer_thread+0x310/0x310 [ 2349.088189] kthread+0xf5/0x130 [ 2349.091354] ? kthread_complete_and_exit+0x20/0x20 [ 2349.096165] ret_from_fork+0x31/0x40 [ 2349.099761] ? kthread_complete_and_exit+0x20/0x20 [ 2349.104555] ret_from_fork_asm+0x11/0x20 [ 2349.108503] [ 2349.110690] irq event stamp: 10727427 [ 2349.114356] hardirqs last enabled at (10727437): [] console_unlock+0x106/0x120 [ 2349.123238] hardirqs last disabled at (10727446): [] console_unlock+0xeb/0x120 [ 2349.132036] softirqs last enabled at (10727374): [] __do_softirq+0x2be/0x371 [ 2349.140746] softirqs last disabled at (10727369): [] irq_exit_rcu+0xb0/0xd0 [ 2349.149267] ---[ end trace 0000000000000000 ]--- [ 2349.154064] ------------[ cut here ]------------ [ 2349.158678] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2349.170337] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2349.179027] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2349.234539] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2349.243664] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2349.251143] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2349.256656] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2349.262142] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2349.280907] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2349.286150] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [ 2349.293300] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2349.300451] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008ac6 [ 2349.307599] R10: 00000000002d2a50 R11: ffffc9000030b828 R12: ffff8881c3b77cb0 [ 2349.314731] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2349.321864] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2349.329961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2349.335722] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2349.342857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2349.350007] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2349.357155] Call Trace: [ 2349.359618] [ 2349.361725] ? __warn+0x85/0x180 [ 2349.364957] ? __list_add_valid_or_report+0x7d/0xa0 [ 2349.369855] ? report_bug+0xfc/0x1d0 [ 2349.373450] ? prb_read_valid+0x1b/0x20 [ 2349.377308] ? handle_bug+0x44/0x70 [ 2349.380826] ? exc_invalid_op+0x17/0x60 [ 2349.384682] ? asm_exc_invalid_op+0x1a/0x20 [ 2349.388890] ? __list_add_valid_or_report+0x7d/0xa0 [ 2349.393766] ? __list_add_valid_or_report+0x7d/0xa0 [ 2349.398646] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2349.403550] ceph_add_cap+0x20b/0x620 [ceph] [ 2349.407853] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2349.412575] ceph_fill_trace+0x143/0xf30 [ceph] [ 2349.417114] ? __mutex_lock+0x97/0xce0 [ 2349.420883] ? __mutex_lock+0xaec/0xce0 [ 2349.424724] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2349.429265] ? lock_downgrade+0xfd/0x250 [ 2349.433231] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2349.437611] ceph_con_process_message+0x74/0x150 [libceph] [ 2349.443107] ? calc_signature+0xdb/0x100 [libceph] [ 2349.447926] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2349.453283] ? process_scheduled_works+0x22a/0x510 [ 2349.458075] ? process_scheduled_works+0x22a/0x510 [ 2349.462881] ceph_con_workfn+0x344/0x750 [libceph] [ 2349.467704] ? process_scheduled_works+0x22a/0x510 [ 2349.472495] process_scheduled_works+0x2a4/0x510 [ 2349.477152] worker_thread+0x188/0x340 [ 2349.480903] ? rescuer_thread+0x310/0x310 [ 2349.484931] kthread+0xf5/0x130 [ 2349.488095] ? kthread_complete_and_exit+0x20/0x20 [ 2349.492907] ret_from_fork+0x31/0x40 [ 2349.496502] ? kthread_complete_and_exit+0x20/0x20 [ 2349.501313] ret_from_fork_asm+0x11/0x20 [ 2349.505262] [ 2349.507451] irq event stamp: 10728493 [ 2349.511114] hardirqs last enabled at (10728503): [] console_unlock+0x106/0x120 [ 2349.519996] hardirqs last disabled at (10728512): [] console_unlock+0xeb/0x120 [ 2349.528794] softirqs last enabled at (10728448): [] __do_softirq+0x2be/0x371 [ 2349.537486] softirqs last disabled at (10728443): [] irq_exit_rcu+0xb0/0xd0 [ 2349.546025] ---[ end trace 0000000000000000 ]--- [ 2349.550662] ------------[ cut here ]------------ [ 2349.555279] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2349.566950] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2349.575638] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2349.631115] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2349.640259] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2349.647754] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2349.653251] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2349.658755] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2349.677500] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2349.682742] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [ 2349.689875] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2349.697019] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008b04 [ 2349.704167] R10: 00000000002d3b40 R11: ffffc9000030b8d8 R12: ffff8881c3b77cb0 [ 2349.711317] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2349.718467] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2349.726571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2349.732318] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2349.739466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2349.746608] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2349.753743] Call Trace: [ 2349.756195] [ 2349.758318] ? __warn+0x85/0x180 [ 2349.761569] ? __list_add_valid_or_report+0x7d/0xa0 [ 2349.766448] ? report_bug+0xfc/0x1d0 [ 2349.770044] ? prb_read_valid+0x1b/0x20 [ 2349.773902] ? handle_bug+0x44/0x70 [ 2349.777412] ? exc_invalid_op+0x17/0x60 [ 2349.781268] ? asm_exc_invalid_op+0x1a/0x20 [ 2349.785478] ? __list_add_valid_or_report+0x7d/0xa0 [ 2349.790360] ? __list_add_valid_or_report+0x7d/0xa0 [ 2349.795255] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2349.800164] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2349.804818] ceph_fill_trace+0x143/0xf30 [ceph] [ 2349.809357] ? __mutex_lock+0x97/0xce0 [ 2349.813126] ? __mutex_lock+0xaec/0xce0 [ 2349.816985] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2349.821557] ? lock_downgrade+0xfd/0x250 [ 2349.825499] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2349.829896] ceph_con_process_message+0x74/0x150 [libceph] [ 2349.835393] ? calc_signature+0xdb/0x100 [libceph] [ 2349.840231] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2349.845562] ? process_scheduled_works+0x22a/0x510 [ 2349.850370] ? process_scheduled_works+0x22a/0x510 [ 2349.855161] ceph_con_workfn+0x344/0x750 [libceph] [ 2349.859983] ? process_scheduled_works+0x22a/0x510 [ 2349.864790] process_scheduled_works+0x2a4/0x510 [ 2349.869432] worker_thread+0x188/0x340 [ 2349.873179] ? rescuer_thread+0x310/0x310 [ 2349.877209] kthread+0xf5/0x130 [ 2349.880373] ? kthread_complete_and_exit+0x20/0x20 [ 2349.885183] ret_from_fork+0x31/0x40 [ 2349.888781] ? kthread_complete_and_exit+0x20/0x20 [ 2349.893574] ret_from_fork_asm+0x11/0x20 [ 2349.897521] [ 2349.899707] irq event stamp: 10729537 [ 2349.903364] hardirqs last enabled at (10729547): [] console_unlock+0x106/0x120 [ 2349.912250] hardirqs last disabled at (10729556): [] console_unlock+0xeb/0x120 [ 2349.921047] softirqs last enabled at (10729570): [] __do_softirq+0x2be/0x371 [ 2349.929763] softirqs last disabled at (10729583): [] irq_exit_rcu+0xb0/0xd0 [ 2349.938284] ---[ end trace 0000000000000000 ]--- [ 2349.943095] ------------[ cut here ]------------ [ 2349.947719] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2349.959389] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2349.968084] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2350.023648] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2350.032774] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2350.040252] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2350.045772] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2350.051258] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2350.070024] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2350.075268] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2350.082417] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2350.089570] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008b41 [ 2350.096709] R10: 00000000002d4bc8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2350.103843] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2350.110990] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2350.119094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2350.124858] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2350.132008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2350.139158] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2350.146309] Call Trace: [ 2350.148779] [ 2350.150887] ? __warn+0x85/0x180 [ 2350.154154] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.159058] ? report_bug+0xfc/0x1d0 [ 2350.162639] ? prb_read_valid+0x1b/0x20 [ 2350.166506] ? handle_bug+0x44/0x70 [ 2350.170003] ? exc_invalid_op+0x17/0x60 [ 2350.173861] ? asm_exc_invalid_op+0x1a/0x20 [ 2350.178094] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.182997] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.187883] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2350.192803] ceph_add_cap+0x20b/0x620 [ceph] [ 2350.197127] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2350.201873] ? lock_acquire+0xcb/0x2b0 [ 2350.205637] ceph_fill_trace+0x143/0xf30 [ceph] [ 2350.210186] ? __mutex_lock+0x97/0xce0 [ 2350.213954] ? __mutex_lock+0xaec/0xce0 [ 2350.217817] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2350.222370] ? lock_downgrade+0xfd/0x250 [ 2350.226339] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2350.230731] ceph_con_process_message+0x74/0x150 [libceph] [ 2350.236237] ? calc_signature+0xdb/0x100 [libceph] [ 2350.241094] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2350.246453] ? process_scheduled_works+0x22a/0x510 [ 2350.251269] ? process_scheduled_works+0x22a/0x510 [ 2350.256065] ceph_con_workfn+0x344/0x750 [libceph] [ 2350.260914] ? process_scheduled_works+0x22a/0x510 [ 2350.265713] process_scheduled_works+0x2a4/0x510 [ 2350.270366] worker_thread+0x188/0x340 [ 2350.274137] ? rescuer_thread+0x310/0x310 [ 2350.278148] kthread+0xf5/0x130 [ 2350.281333] ? kthread_complete_and_exit+0x20/0x20 [ 2350.286149] ret_from_fork+0x31/0x40 [ 2350.289738] ? kthread_complete_and_exit+0x20/0x20 [ 2350.294558] ret_from_fork_asm+0x11/0x20 [ 2350.298522] [ 2350.300717] irq event stamp: 10730673 [ 2350.304383] hardirqs last enabled at (10730683): [] console_unlock+0x106/0x120 [ 2350.313265] hardirqs last disabled at (10730692): [] console_unlock+0xeb/0x120 [ 2350.322065] softirqs last enabled at (10730620): [] __do_softirq+0x2be/0x371 [ 2350.330764] softirqs last disabled at (10730615): [] irq_exit_rcu+0xb0/0xd0 [ 2350.339285] ---[ end trace 0000000000000000 ]--- [ 2350.343941] ------------[ cut here ]------------ [ 2350.348565] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2350.360213] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2350.369167] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2350.424695] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2350.433649] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2350.441129] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2350.446615] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2350.465358] RSP: 0018:ffffc90001fefaa8 EFLAGS: 00010282 [ 2350.470589] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [ 2350.477727] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2350.484860] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008b80 [ 2350.491993] R10: 00000000002d5d68 R11: ffffc90001fef938 R12: ffff8881c3b77cb0 [ 2350.499125] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2350.506259] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2350.514343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2350.520091] CR2: 00007f9bd183f000 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2350.527223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2350.534356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2350.541489] Call Trace: [ 2350.543942] [ 2350.546049] ? __warn+0x85/0x180 [ 2350.549283] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.554170] ? report_bug+0xfc/0x1d0 [ 2350.557748] ? prb_read_valid+0x1b/0x20 [ 2350.561592] ? handle_bug+0x44/0x70 [ 2350.565090] ? exc_invalid_op+0x17/0x60 [ 2350.568929] ? asm_exc_invalid_op+0x1a/0x20 [ 2350.573125] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.578004] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.582883] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2350.587783] ceph_open+0x16b/0x440 [ceph] [ 2350.591813] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2350.596967] do_dentry_open+0x172/0x530 [ 2350.600810] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2350.605459] lookup_open+0x37b/0x630 [ 2350.609059] path_openat+0x25a/0x9d0 [ 2350.612653] do_filp_open+0xb8/0x120 [ 2350.616240] ? lock_release+0xc5/0x260 [ 2350.619999] ? _raw_spin_unlock+0x23/0x40 [ 2350.624015] ? alloc_fd+0x13f/0x200 [ 2350.627514] do_sys_openat2+0x249/0x290 [ 2350.631361] do_sys_open+0x4f/0x80 [ 2350.634774] do_syscall_64+0x38/0x80 [ 2350.638359] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2350.643412] RIP: 0033:0x7f48f0eed2f2 [ 2350.646990] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2350.665736] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2350.673304] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2350.680434] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2350.687569] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2350.694701] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2350.701833] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2350.708981] [ 2350.711177] irq event stamp: 12897391 [ 2350.714844] hardirqs last enabled at (12897401): [] console_unlock+0x106/0x120 [ 2350.723708] hardirqs last disabled at (12897410): [] console_unlock+0xeb/0x120 [ 2350.732487] softirqs last enabled at (12896932): [] __do_softirq+0x2be/0x371 [ 2350.741182] softirqs last disabled at (12896927): [] irq_exit_rcu+0xb0/0xd0 [ 2350.749700] ---[ end trace 0000000000000000 ]--- [ 2350.754322] ------------[ cut here ]------------ [ 2350.758939] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2350.770588] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2350.779284] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2350.834830] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2350.843964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2350.851443] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2350.856954] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2350.862440] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2350.881186] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2350.886414] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2350.893549] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2350.900690] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008bbc [ 2350.907828] R10: 00000000002d6d90 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2350.914961] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2350.922095] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2350.930182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2350.935925] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2350.943061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2350.950196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2350.957334] Call Trace: [ 2350.959786] [ 2350.961894] ? __warn+0x85/0x180 [ 2350.965137] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.970024] ? report_bug+0xfc/0x1d0 [ 2350.973609] ? prb_read_valid+0x1b/0x20 [ 2350.977454] ? handle_bug+0x44/0x70 [ 2350.980952] ? exc_invalid_op+0x17/0x60 [ 2350.984791] ? asm_exc_invalid_op+0x1a/0x20 [ 2350.988997] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.993883] ? __list_add_valid_or_report+0x7d/0xa0 [ 2350.998771] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2351.003671] ceph_add_cap+0x20b/0x620 [ceph] [ 2351.007978] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2351.012709] ? lock_acquire+0xcb/0x2b0 [ 2351.016478] ceph_fill_trace+0x143/0xf30 [ceph] [ 2351.021027] ? __mutex_lock+0x97/0xce0 [ 2351.024779] ? __mutex_lock+0xaec/0xce0 [ 2351.028623] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2351.033179] ? lock_downgrade+0xfd/0x250 [ 2351.037122] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2351.041513] ceph_con_process_message+0x74/0x150 [libceph] [ 2351.047018] ? calc_signature+0xdb/0x100 [libceph] [ 2351.051842] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2351.057181] ? process_scheduled_works+0x22a/0x510 [ 2351.061981] ? process_scheduled_works+0x22a/0x510 [ 2351.066778] ceph_con_workfn+0x344/0x750 [libceph] [ 2351.071592] ? process_scheduled_works+0x22a/0x510 [ 2351.076388] process_scheduled_works+0x2a4/0x510 [ 2351.081020] worker_thread+0x188/0x340 [ 2351.084781] ? rescuer_thread+0x310/0x310 [ 2351.088801] kthread+0xf5/0x130 [ 2351.091949] ? kthread_complete_and_exit+0x20/0x20 [ 2351.096751] ret_from_fork+0x31/0x40 [ 2351.100337] ? kthread_complete_and_exit+0x20/0x20 [ 2351.105129] ret_from_fork_asm+0x11/0x20 [ 2351.109070] [ 2351.111265] irq event stamp: 10732663 [ 2351.114931] hardirqs last enabled at (10732673): [] console_unlock+0x106/0x120 [ 2351.123805] hardirqs last disabled at (10732682): [] console_unlock+0xeb/0x120 [ 2351.132586] softirqs last enabled at (10732610): [] __do_softirq+0x2be/0x371 [ 2351.141286] softirqs last disabled at (10732605): [] irq_exit_rcu+0xb0/0xd0 [ 2351.149806] ---[ end trace 0000000000000000 ]--- [ 2351.154425] ------------[ cut here ]------------ [ 2351.159044] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2351.170695] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2351.179653] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2351.235229] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2351.244237] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2351.251719] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2351.257203] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2351.275950] RSP: 0018:ffffc90001fef8c0 EFLAGS: 00010282 [ 2351.281176] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [ 2351.288310] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2351.295443] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008bfb [ 2351.302575] R10: 00000000002d7eb8 R11: ffffc90001fef750 R12: ffff8881c3b77cb0 [ 2351.309709] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2351.316840] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2351.324926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2351.330673] CR2: 00007f9bd183f000 CR3: 0000000108c72002 CR4: 00000000003706e0 [ 2351.337807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2351.344939] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2351.352070] Call Trace: [ 2351.354524] [ 2351.356630] ? __warn+0x85/0x180 [ 2351.359864] ? __list_add_valid_or_report+0x7d/0xa0 [ 2351.364744] ? report_bug+0xfc/0x1d0 [ 2351.368321] ? prb_read_valid+0x1b/0x20 [ 2351.372166] ? handle_bug+0x44/0x70 [ 2351.375663] ? exc_invalid_op+0x17/0x60 [ 2351.379503] ? asm_exc_invalid_op+0x1a/0x20 [ 2351.383699] ? __list_add_valid_or_report+0x7d/0xa0 [ 2351.388589] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2351.393496] ceph_check_caps+0x999/0xae0 [ceph] [ 2351.398068] ? handle_bug+0x44/0x70 [ 2351.401559] ? exc_invalid_op+0x17/0x60 [ 2351.405400] ? asm_exc_invalid_op+0x1a/0x20 [ 2351.409591] ? find_held_lock+0x2d/0x90 [ 2351.413433] ? ceph_open+0x175/0x440 [ceph] [ 2351.417637] ? lock_release+0xc5/0x260 [ 2351.421395] ceph_open+0x3cc/0x440 [ceph] [ 2351.425425] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2351.430577] do_dentry_open+0x172/0x530 [ 2351.434417] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2351.439069] lookup_open+0x37b/0x630 [ 2351.442671] path_openat+0x25a/0x9d0 [ 2351.446263] do_filp_open+0xb8/0x120 [ 2351.449850] ? lock_release+0xc5/0x260 [ 2351.453609] ? _raw_spin_unlock+0x23/0x40 [ 2351.457625] ? alloc_fd+0x13f/0x200 [ 2351.461123] do_sys_openat2+0x249/0x290 [ 2351.464970] do_sys_open+0x4f/0x80 [ 2351.468383] do_syscall_64+0x38/0x80 [ 2351.471969] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2351.477020] RIP: 0033:0x7f48f0eed2f2 [ 2351.480600] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2351.499346] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2351.506911] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2351.514046] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2351.521178] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2351.528310] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2351.535443] R13: 00007f48f183df18 R14: 00000000020767c0 R15: 00000000000029bc [ 2351.542590] [ 2351.544787] irq event stamp: 12899275 [ 2351.548453] hardirqs last enabled at (12899285): [] console_unlock+0x106/0x120 [ 2351.557322] hardirqs last disabled at (12899300): [] console_unlock+0xeb/0x120 [ 2351.566107] softirqs last enabled at (12899298): [] __do_softirq+0x2be/0x371 [ 2351.574800] softirqs last disabled at (12899293): [] irq_exit_rcu+0xb0/0xd0 [ 2351.583318] ---[ end trace 0000000000000000 ]--- [ 2351.587939] ------------[ cut here ]------------ [ 2351.592559] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2351.604205] WARNING: CPU: 6 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2351.613156] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2351.668627] CPU: 6 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2351.677578] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2351.685056] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2351.690543] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2351.709289] RSP: 0018:ffffc90001957b78 EFLAGS: 00010286 [ 2351.714513] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2351.721638] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2351.728762] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008c3d [ 2351.735888] R10: 00000000002d9168 R11: ffffc90001957a08 R12: ffff888158a68750 [ 2351.743021] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2351.750153] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2351.758238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2351.763975] CR2: 00007f9bd22d7020 CR3: 000000016696c005 CR4: 00000000003706e0 [ 2351.771109] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2351.778242] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2351.785376] Call Trace: [ 2351.787829] [ 2351.789933] ? __warn+0x85/0x180 [ 2351.793168] ? __list_add_valid_or_report+0x7d/0xa0 [ 2351.798046] ? report_bug+0xfc/0x1d0 [ 2351.801626] ? prb_read_valid+0x1b/0x20 [ 2351.805465] ? handle_bug+0x44/0x70 [ 2351.808956] ? exc_invalid_op+0x17/0x60 [ 2351.812788] ? asm_exc_invalid_op+0x1a/0x20 [ 2351.816979] ? __list_add_valid_or_report+0x7d/0xa0 [ 2351.821854] ? __list_add_valid_or_report+0x7d/0xa0 [ 2351.826734] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2351.831624] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2351.836859] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2351.841659] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2351.846374] ? __d_lookup+0xd6/0x200 [ 2351.849952] lookup_fast+0x6d/0xf0 [ 2351.853357] walk_component+0x26/0x160 [ 2351.857111] link_path_walk+0x24d/0x380 [ 2351.860950] ? path_openat+0x80/0x9d0 [ 2351.864619] path_openat+0x8e/0x9d0 [ 2351.868111] do_filp_open+0xb8/0x120 [ 2351.871691] ? lock_release+0xc5/0x260 [ 2351.875443] ? _raw_spin_unlock+0x23/0x40 [ 2351.879453] ? alloc_fd+0x13f/0x200 [ 2351.882947] do_sys_openat2+0x249/0x290 [ 2351.886786] do_sys_open+0x4f/0x80 [ 2351.890191] do_syscall_64+0x38/0x80 [ 2351.893771] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2351.898824] RIP: 0033:0x7f48f0ef2805 [ 2351.902403] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2351.921150] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2351.928715] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2351.935847] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2351.942971] RBP: 000000000207da30 R08: 0000000003ffffff R09: 0000000000000000 [ 2351.950097] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2351.957229] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2351.964368] [ 2351.966555] irq event stamp: 12846089 [ 2351.970219] hardirqs last enabled at (12846099): [] console_unlock+0x106/0x120 [ 2351.979089] hardirqs last disabled at (12846120): [] console_unlock+0xeb/0x120 [ 2351.987866] softirqs last enabled at (12846118): [] __do_softirq+0x2be/0x371 [ 2351.996560] softirqs last disabled at (12846139): [] irq_exit_rcu+0xb0/0xd0 [ 2352.005077] ---[ end trace 0000000000000000 ]--- [ 2352.009690] ------------[ cut here ]------------ [ 2352.014310] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2352.025971] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2352.034671] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2352.090146] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2352.099289] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2352.106788] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2352.112287] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2352.117792] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2352.136538] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2352.141779] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2352.148906] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2352.156054] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008c7c [ 2352.163199] R10: 00000000002da058 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2352.170346] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2352.177488] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2352.185599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2352.191338] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2352.198487] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2352.205630] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2352.212764] Call Trace: [ 2352.215216] [ 2352.217338] ? __warn+0x85/0x180 [ 2352.220599] ? __list_add_valid_or_report+0x7d/0xa0 [ 2352.225477] ? report_bug+0xfc/0x1d0 [ 2352.229075] ? prb_read_valid+0x1b/0x20 [ 2352.232932] ? handle_bug+0x44/0x70 [ 2352.236439] ? exc_invalid_op+0x17/0x60 [ 2352.240289] ? asm_exc_invalid_op+0x1a/0x20 [ 2352.244514] ? __list_add_valid_or_report+0x7d/0xa0 [ 2352.249391] ? __list_add_valid_or_report+0x7d/0xa0 [ 2352.254284] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2352.259192] ceph_add_cap+0x20b/0x620 [ceph] [ 2352.263513] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2352.268254] ? lock_acquire+0xcb/0x2b0 [ 2352.272022] ceph_fill_trace+0x143/0xf30 [ceph] [ 2352.276561] ? __mutex_lock+0x97/0xce0 [ 2352.280329] ? __mutex_lock+0xaec/0xce0 [ 2352.284169] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2352.288727] ? lock_downgrade+0xfd/0x250 [ 2352.292661] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2352.297055] ceph_con_process_message+0x74/0x150 [libceph] [ 2352.302551] ? calc_signature+0xdb/0x100 [libceph] [ 2352.307380] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2352.312704] ? process_scheduled_works+0x22a/0x510 [ 2352.317496] ? process_scheduled_works+0x22a/0x510 [ 2352.322302] ceph_con_workfn+0x344/0x750 [libceph] [ 2352.327124] ? process_scheduled_works+0x22a/0x510 [ 2352.331930] process_scheduled_works+0x2a4/0x510 [ 2352.336594] worker_thread+0x188/0x340 [ 2352.340347] ? rescuer_thread+0x310/0x310 [ 2352.344376] kthread+0xf5/0x130 [ 2352.347539] ? kthread_complete_and_exit+0x20/0x20 [ 2352.352352] ret_from_fork+0x31/0x40 [ 2352.355949] ? kthread_complete_and_exit+0x20/0x20 [ 2352.360757] ret_from_fork_asm+0x11/0x20 [ 2352.364690] [ 2352.366879] irq event stamp: 10735679 [ 2352.370559] hardirqs last enabled at (10735689): [] console_unlock+0x106/0x120 [ 2352.379442] hardirqs last disabled at (10735698): [] console_unlock+0xeb/0x120 [ 2352.388240] softirqs last enabled at (10735646): [] __do_softirq+0x2be/0x371 [ 2352.396950] softirqs last disabled at (10735641): [] irq_exit_rcu+0xb0/0xd0 [ 2352.405486] ---[ end trace 0000000000000000 ]--- [ 2352.410697] ------------[ cut here ]------------ [ 2352.415448] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2352.427113] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2352.435805] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2352.491283] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2352.500423] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2352.507912] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2352.513424] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2352.518927] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2352.537691] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2352.542916] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [ 2352.550066] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2352.557208] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008cbb [ 2352.564359] R10: 00000000002db3c8 R11: ffffc9000030b828 R12: ffff8881e4f8df58 [ 2352.571509] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2352.578659] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2352.586745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2352.592490] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2352.599634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2352.606766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2352.613897] Call Trace: [ 2352.616359] [ 2352.618475] ? __warn+0x85/0x180 [ 2352.621727] ? __list_add_valid_or_report+0x7d/0xa0 [ 2352.626609] ? report_bug+0xfc/0x1d0 [ 2352.630183] ? prb_read_valid+0x1b/0x20 [ 2352.634041] ? handle_bug+0x44/0x70 [ 2352.637549] ? exc_invalid_op+0x17/0x60 [ 2352.641399] ? asm_exc_invalid_op+0x1a/0x20 [ 2352.645611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2352.650491] ? __list_add_valid_or_report+0x7d/0xa0 [ 2352.655387] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2352.660295] ceph_add_cap+0x20b/0x620 [ceph] [ 2352.664610] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2352.669352] ceph_fill_trace+0x143/0xf30 [ceph] [ 2352.673891] ? __mutex_lock+0x97/0xce0 [ 2352.677659] ? __mutex_lock+0xaec/0xce0 [ 2352.681501] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2352.686059] ? lock_downgrade+0xfd/0x250 [ 2352.690024] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2352.694419] ceph_con_process_message+0x74/0x150 [libceph] [ 2352.699917] ? calc_signature+0xdb/0x100 [libceph] [ 2352.704738] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2352.710085] ? process_scheduled_works+0x22a/0x510 [ 2352.714894] ? process_scheduled_works+0x22a/0x510 [ 2352.719701] ceph_con_workfn+0x344/0x750 [libceph] [ 2352.724507] ? process_scheduled_works+0x22a/0x510 [ 2352.729314] process_scheduled_works+0x2a4/0x510 [ 2352.733972] worker_thread+0x188/0x340 [ 2352.737722] ? rescuer_thread+0x310/0x310 [ 2352.741735] kthread+0xf5/0x130 [ 2352.744895] ? kthread_complete_and_exit+0x20/0x20 [ 2352.749709] ret_from_fork+0x31/0x40 [ 2352.753286] ? kthread_complete_and_exit+0x20/0x20 [ 2352.758096] ret_from_fork_asm+0x11/0x20 [ 2352.762063] [ 2352.764249] irq event stamp: 10737879 [ 2352.767924] hardirqs last enabled at (10737889): [] console_unlock+0x106/0x120 [ 2352.776807] hardirqs last disabled at (10737898): [] console_unlock+0xeb/0x120 [ 2352.785606] softirqs last enabled at (10737836): [] __do_softirq+0x2be/0x371 [ 2352.794299] softirqs last disabled at (10737831): [] irq_exit_rcu+0xb0/0xd0 [ 2352.802834] ---[ end trace 0000000000000000 ]--- [ 2352.807456] ------------[ cut here ]------------ [ 2352.812091] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2352.823754] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2352.832449] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2352.887944] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2352.897085] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2352.904576] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2352.910094] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2352.915610] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2352.934356] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2352.939610] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [ 2352.946740] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2352.953888] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008cf9 [ 2352.961029] R10: 00000000002dc260 R11: ffffc9000030b8d8 R12: ffff8881e4f8df58 [ 2352.968175] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2352.975321] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2352.983426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2352.989189] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2352.996339] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2353.003490] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2353.010640] Call Trace: [ 2353.013093] [ 2353.015215] ? __warn+0x85/0x180 [ 2353.018469] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.023363] ? report_bug+0xfc/0x1d0 [ 2353.026958] ? prb_read_valid+0x1b/0x20 [ 2353.030818] ? handle_bug+0x44/0x70 [ 2353.034309] ? exc_invalid_op+0x17/0x60 [ 2353.038164] ? asm_exc_invalid_op+0x1a/0x20 [ 2353.042373] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.047248] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.052143] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2353.057053] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2353.061706] ceph_fill_trace+0x143/0xf30 [ceph] [ 2353.066247] ? __mutex_lock+0x97/0xce0 [ 2353.070015] ? __mutex_lock+0xaec/0xce0 [ 2353.073889] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2353.078450] ? lock_downgrade+0xfd/0x250 [ 2353.082416] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2353.086794] ceph_con_process_message+0x74/0x150 [libceph] [ 2353.092291] ? calc_signature+0xdb/0x100 [libceph] [ 2353.097102] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2353.102441] ? process_scheduled_works+0x22a/0x510 [ 2353.107234] ? process_scheduled_works+0x22a/0x510 [ 2353.112040] ceph_con_workfn+0x344/0x750 [libceph] [ 2353.116862] ? process_scheduled_works+0x22a/0x510 [ 2353.121652] process_scheduled_works+0x2a4/0x510 [ 2353.126294] worker_thread+0x188/0x340 [ 2353.130042] ? rescuer_thread+0x310/0x310 [ 2353.134073] kthread+0xf5/0x130 [ 2353.137235] ? kthread_complete_and_exit+0x20/0x20 [ 2353.142048] ret_from_fork+0x31/0x40 [ 2353.145645] ? kthread_complete_and_exit+0x20/0x20 [ 2353.150436] ret_from_fork_asm+0x11/0x20 [ 2353.154402] [ 2353.156608] irq event stamp: 10738931 [ 2353.160272] hardirqs last enabled at (10738941): [] console_unlock+0x106/0x120 [ 2353.169155] hardirqs last disabled at (10738950): [] console_unlock+0xeb/0x120 [ 2353.177954] softirqs last enabled at (10738898): [] __do_softirq+0x2be/0x371 [ 2353.186644] softirqs last disabled at (10738893): [] irq_exit_rcu+0xb0/0xd0 [ 2353.195165] ---[ end trace 0000000000000000 ]--- [ 2353.199957] ------------[ cut here ]------------ [ 2353.204578] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2353.216223] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2353.224918] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2353.280397] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2353.289520] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2353.297001] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2353.302495] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2353.307979] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2353.326728] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2353.331953] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2353.339087] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2353.346220] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008d36 [ 2353.353352] R10: 00000000002dd298 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2353.360483] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2353.367616] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2353.375702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2353.381448] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2353.388573] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2353.395705] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2353.402838] Call Trace: [ 2353.405282] [ 2353.407381] ? __warn+0x85/0x180 [ 2353.410616] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.415494] ? report_bug+0xfc/0x1d0 [ 2353.419071] ? prb_read_valid+0x1b/0x20 [ 2353.422913] ? handle_bug+0x44/0x70 [ 2353.426404] ? exc_invalid_op+0x17/0x60 [ 2353.430245] ? asm_exc_invalid_op+0x1a/0x20 [ 2353.434434] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.439310] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.444189] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2353.449080] ceph_add_cap+0x20b/0x620 [ceph] [ 2353.453366] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2353.458089] ? lock_acquire+0xcb/0x2b0 [ 2353.461848] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2353.466298] ? __mutex_lock+0x97/0xce0 [ 2353.470051] ? __mutex_lock+0xaec/0xce0 [ 2353.473894] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2353.478440] ? lock_downgrade+0xfd/0x250 [ 2353.482372] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2353.486750] ceph_con_process_message+0x74/0x150 [libceph] [ 2353.492246] ? calc_signature+0xdb/0x100 [libceph] [ 2353.497050] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2353.502375] ? process_scheduled_works+0x22a/0x510 [ 2353.507174] ? process_scheduled_works+0x22a/0x510 [ 2353.511966] ceph_con_workfn+0x344/0x750 [libceph] [ 2353.516766] ? process_scheduled_works+0x22a/0x510 [ 2353.521556] process_scheduled_works+0x2a4/0x510 [ 2353.526182] worker_thread+0x188/0x340 [ 2353.529930] ? rescuer_thread+0x310/0x310 [ 2353.533942] kthread+0xf5/0x130 [ 2353.537091] ? kthread_complete_and_exit+0x20/0x20 [ 2353.541882] ret_from_fork+0x31/0x40 [ 2353.545461] ? kthread_complete_and_exit+0x20/0x20 [ 2353.550254] ret_from_fork_asm+0x11/0x20 [ 2353.554186] [ 2353.556372] irq event stamp: 10740049 [ 2353.560036] hardirqs last enabled at (10740059): [] console_unlock+0x106/0x120 [ 2353.568904] hardirqs last disabled at (10740068): [] console_unlock+0xeb/0x120 [ 2353.577684] softirqs last enabled at (10739996): [] __do_softirq+0x2be/0x371 [ 2353.586378] softirqs last disabled at (10739991): [] irq_exit_rcu+0xb0/0xd0 [ 2353.594897] ---[ end trace 0000000000000000 ]--- [ 2353.599517] ------------[ cut here ]------------ [ 2353.604135] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2353.615782] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2353.624736] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2353.680205] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2353.689158] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2353.696636] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2353.702121] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2353.720869] RSP: 0018:ffffc90001f7faa8 EFLAGS: 00010282 [ 2353.726094] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [ 2353.733227] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2353.740359] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008d75 [ 2353.747493] R10: 00000000002de450 R11: ffffc90001f7f938 R12: ffff8881e4f8df58 [ 2353.754626] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2353.761758] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2353.769845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2353.775590] CR2: 00007f9bd22d7020 CR3: 0000000113532006 CR4: 00000000003706e0 [ 2353.782724] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2353.789856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2353.796989] Call Trace: [ 2353.799440] [ 2353.801539] ? __warn+0x85/0x180 [ 2353.804771] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.809652] ? report_bug+0xfc/0x1d0 [ 2353.813230] ? prb_read_valid+0x1b/0x20 [ 2353.817072] ? handle_bug+0x44/0x70 [ 2353.820561] ? exc_invalid_op+0x17/0x60 [ 2353.824401] ? asm_exc_invalid_op+0x1a/0x20 [ 2353.828598] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.833475] ? __list_add_valid_or_report+0x7d/0xa0 [ 2353.838354] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2353.843247] ceph_open+0x16b/0x440 [ceph] [ 2353.847268] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2353.852414] do_dentry_open+0x172/0x530 [ 2353.856253] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2353.860888] lookup_open+0x37b/0x630 [ 2353.864471] ? verify_cpu+0x20/0x100 [ 2353.868052] ? verify_cpu+0x20/0x100 [ 2353.871631] path_openat+0x25a/0x9d0 [ 2353.875212] do_filp_open+0xb8/0x120 [ 2353.878789] ? lock_release+0xc5/0x260 [ 2353.882543] ? _raw_spin_unlock+0x23/0x40 [ 2353.886551] ? alloc_fd+0x13f/0x200 [ 2353.890046] do_sys_openat2+0x249/0x290 [ 2353.893886] do_sys_open+0x4f/0x80 [ 2353.897291] do_syscall_64+0x38/0x80 [ 2353.900871] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2353.905923] RIP: 0033:0x7f48f0eed2f2 [ 2353.909502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2353.928246] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2353.935813] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2353.942947] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2353.950078] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2353.957203] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2353.964337] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2353.971476] [ 2353.973661] irq event stamp: 12959335 [ 2353.977328] hardirqs last enabled at (12959345): [] console_unlock+0x106/0x120 [ 2353.986194] hardirqs last disabled at (12959354): [] console_unlock+0xeb/0x120 [ 2353.994974] softirqs last enabled at (12959282): [] __do_softirq+0x2be/0x371 [ 2354.003667] softirqs last disabled at (12959277): [] irq_exit_rcu+0xb0/0xd0 [ 2354.012187] ---[ end trace 0000000000000000 ]--- [ 2354.016806] ------------[ cut here ]------------ [ 2354.021425] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2354.033073] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2354.041770] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2354.097253] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2354.106378] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2354.113858] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2354.119361] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2354.124846] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2354.143593] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2354.148818] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2354.155951] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2354.163084] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008db3 [ 2354.170218] R10: 00000000002df508 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2354.177349] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2354.184482] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2354.192568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2354.198314] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2354.205439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2354.212571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2354.219706] Call Trace: [ 2354.222158] [ 2354.224265] ? __warn+0x85/0x180 [ 2354.227496] ? __list_add_valid_or_report+0x7d/0xa0 [ 2354.232376] ? report_bug+0xfc/0x1d0 [ 2354.235957] ? prb_read_valid+0x1b/0x20 [ 2354.239797] ? handle_bug+0x44/0x70 [ 2354.243286] ? exc_invalid_op+0x17/0x60 [ 2354.247126] ? asm_exc_invalid_op+0x1a/0x20 [ 2354.251317] ? __list_add_valid_or_report+0x7d/0xa0 [ 2354.256193] ? __list_add_valid_or_report+0x7d/0xa0 [ 2354.261072] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2354.265963] ceph_add_cap+0x20b/0x620 [ceph] [ 2354.270248] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2354.274976] ceph_fill_trace+0x143/0xf30 [ceph] [ 2354.279516] ? __mutex_lock+0x97/0xce0 [ 2354.283267] ? __mutex_lock+0xaec/0xce0 [ 2354.287109] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2354.291650] ? lock_downgrade+0xfd/0x250 [ 2354.295579] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2354.299962] ceph_con_process_message+0x74/0x150 [libceph] [ 2354.305455] ? calc_signature+0xdb/0x100 [libceph] [ 2354.310268] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2354.315599] ? process_scheduled_works+0x22a/0x510 [ 2354.320398] ? process_scheduled_works+0x22a/0x510 [ 2354.325248] ceph_con_workfn+0x344/0x750 [libceph] [ 2354.330056] ? process_scheduled_works+0x22a/0x510 [ 2354.334852] process_scheduled_works+0x2a4/0x510 [ 2354.339476] worker_thread+0x188/0x340 [ 2354.343226] ? rescuer_thread+0x310/0x310 [ 2354.347237] kthread+0xf5/0x130 [ 2354.350384] ? kthread_complete_and_exit+0x20/0x20 [ 2354.355177] ret_from_fork+0x31/0x40 [ 2354.358755] ? kthread_complete_and_exit+0x20/0x20 [ 2354.363550] ret_from_fork_asm+0x11/0x20 [ 2354.367481] [ 2354.369667] irq event stamp: 10742129 [ 2354.373331] hardirqs last enabled at (10742139): [] console_unlock+0x106/0x120 [ 2354.382199] hardirqs last disabled at (10742148): [] console_unlock+0xeb/0x120 [ 2354.390978] softirqs last enabled at (10742106): [] __do_softirq+0x2be/0x371 [ 2354.399670] softirqs last disabled at (10742101): [] irq_exit_rcu+0xb0/0xd0 [ 2354.408191] ---[ end trace 0000000000000000 ]--- [ 2354.412812] ------------[ cut here ]------------ [ 2354.417429] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2354.429074] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2354.438022] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2354.493491] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2354.502442] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2354.509922] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2354.515407] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2354.534154] RSP: 0018:ffffc90001f7f8c0 EFLAGS: 00010282 [ 2354.539379] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [ 2354.546512] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2354.553637] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008df1 [ 2354.560771] R10: 00000000002e08f8 R11: ffffc90001f7f750 R12: ffff8881e4f8df58 [ 2354.567901] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2354.575027] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2354.583113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2354.588858] CR2: 00007f9bd22d7020 CR3: 0000000113532006 CR4: 00000000003706e0 [ 2354.595991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2354.603124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2354.610249] Call Trace: [ 2354.612700] [ 2354.614799] ? __warn+0x85/0x180 [ 2354.618033] ? __list_add_valid_or_report+0x7d/0xa0 [ 2354.622910] ? report_bug+0xfc/0x1d0 [ 2354.626490] ? prb_read_valid+0x1b/0x20 [ 2354.630333] ? handle_bug+0x44/0x70 [ 2354.633821] ? exc_invalid_op+0x17/0x60 [ 2354.637653] ? asm_exc_invalid_op+0x1a/0x20 [ 2354.641844] ? __list_add_valid_or_report+0x7d/0xa0 [ 2354.646720] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2354.651612] ceph_check_caps+0x999/0xae0 [ceph] [ 2354.656166] ? handle_bug+0x44/0x70 [ 2354.659657] ? exc_invalid_op+0x17/0x60 [ 2354.663497] ? asm_exc_invalid_op+0x1a/0x20 [ 2354.667687] ? find_held_lock+0x2d/0x90 [ 2354.671523] ? ceph_open+0x175/0x440 [ceph] [ 2354.675718] ? lock_release+0xc5/0x260 [ 2354.679474] ceph_open+0x3cc/0x440 [ceph] [ 2354.683494] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2354.688643] do_dentry_open+0x172/0x530 [ 2354.692482] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2354.697116] lookup_open+0x37b/0x630 [ 2354.700699] ? verify_cpu+0x20/0x100 [ 2354.704278] ? verify_cpu+0x20/0x100 [ 2354.707857] path_openat+0x25a/0x9d0 [ 2354.711437] do_filp_open+0xb8/0x120 [ 2354.715018] ? lock_release+0xc5/0x260 [ 2354.718768] ? _raw_spin_unlock+0x23/0x40 [ 2354.722778] ? alloc_fd+0x13f/0x200 [ 2354.726274] do_sys_openat2+0x249/0x290 [ 2354.730114] do_sys_open+0x4f/0x80 [ 2354.733517] do_syscall_64+0x38/0x80 [ 2354.737097] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2354.742148] RIP: 0033:0x7f48f0eed2f2 [ 2354.745727] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2354.764475] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2354.772039] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2354.779172] RDX: 0000000000000042 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2354.786297] RBP: 00007ffc53d60520 R08: 0000000000000042 R09: 0000000000000002 [ 2354.793422] R10: 0000000000000180 R11: 0000000000000246 R12: 0000000000000000 [ 2354.800554] R13: 00007f48f183d000 R14: 00000000020767c0 R15: 0000000000002a1a [ 2354.807694] [ 2354.809879] irq event stamp: 12961295 [ 2354.813538] hardirqs last enabled at (12961305): [] console_unlock+0x106/0x120 [ 2354.822403] hardirqs last disabled at (12961314): [] console_unlock+0xeb/0x120 [ 2354.831183] softirqs last enabled at (12961328): [] __do_softirq+0x2be/0x371 [ 2354.839877] softirqs last disabled at (12961323): [] irq_exit_rcu+0xb0/0xd0 [ 2354.848396] ---[ end trace 0000000000000000 ]--- [ 2354.853017] ------------[ cut here ]------------ [ 2354.857644] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2354.869305] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2354.878002] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2354.933499] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2354.942647] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2354.950129] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2354.955639] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2354.961123] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2354.979882] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2354.985107] RAX: 0000000000000000 RBX: ffff8881c30309c8 RCX: 0000000000000000 [ 2354.992257] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2354.999408] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008e35 [ 2355.006556] R10: 00000000002e1b78 R11: ffffc9000030b828 R12: ffff8881c3031118 [ 2355.013698] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2355.020832] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2355.028928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2355.034688] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2355.041815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2355.048945] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2355.056088] Call Trace: [ 2355.058557] [ 2355.060675] ? __warn+0x85/0x180 [ 2355.063907] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.068786] ? report_bug+0xfc/0x1d0 [ 2355.072365] ? prb_read_valid+0x1b/0x20 [ 2355.076221] ? handle_bug+0x44/0x70 [ 2355.079732] ? exc_invalid_op+0x17/0x60 [ 2355.083571] ? asm_exc_invalid_op+0x1a/0x20 [ 2355.087779] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.092655] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.097531] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2355.102439] ceph_add_cap+0x20b/0x620 [ceph] [ 2355.106744] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2355.111488] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2355.116818] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2355.121295] ceph_con_process_message+0x74/0x150 [libceph] [ 2355.126793] ? calc_signature+0xdb/0x100 [libceph] [ 2355.131605] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2355.136950] ? process_scheduled_works+0x22a/0x510 [ 2355.141744] ? process_scheduled_works+0x22a/0x510 [ 2355.146535] ceph_con_workfn+0x344/0x750 [libceph] [ 2355.151355] ? process_scheduled_works+0x22a/0x510 [ 2355.156165] process_scheduled_works+0x2a4/0x510 [ 2355.160806] worker_thread+0x188/0x340 [ 2355.164564] ? rescuer_thread+0x310/0x310 [ 2355.168591] kthread+0xf5/0x130 [ 2355.171756] ? kthread_complete_and_exit+0x20/0x20 [ 2355.176550] ret_from_fork+0x31/0x40 [ 2355.180145] ? kthread_complete_and_exit+0x20/0x20 [ 2355.184956] ret_from_fork_asm+0x11/0x20 [ 2355.188906] [ 2355.191092] irq event stamp: 10744133 [ 2355.194774] hardirqs last enabled at (10744143): [] console_unlock+0x106/0x120 [ 2355.203651] hardirqs last disabled at (10744152): [] console_unlock+0xeb/0x120 [ 2355.212427] softirqs last enabled at (10744130): [] __do_softirq+0x2be/0x371 [ 2355.221138] softirqs last disabled at (10744125): [] irq_exit_rcu+0xb0/0xd0 [ 2355.229676] ---[ end trace 0000000000000000 ]--- [ 2355.234351] ------------[ cut here ]------------ [ 2355.238967] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2355.250650] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2355.259345] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2355.314836] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2355.323964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2355.331459] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2355.336970] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2355.342472] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2355.361229] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2355.366472] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2355.373614] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2355.380766] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008e6f [ 2355.387899] R10: 00000000002e2980 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2355.395030] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2355.402170] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2355.410267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2355.416029] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2355.423170] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2355.430311] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2355.437454] Call Trace: [ 2355.439932] [ 2355.442047] ? __warn+0x85/0x180 [ 2355.445297] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.450196] ? report_bug+0xfc/0x1d0 [ 2355.453792] ? prb_read_valid+0x1b/0x20 [ 2355.457650] ? handle_bug+0x44/0x70 [ 2355.461141] ? exc_invalid_op+0x17/0x60 [ 2355.464998] ? asm_exc_invalid_op+0x1a/0x20 [ 2355.469224] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.474099] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.478994] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2355.483904] ceph_add_cap+0x20b/0x620 [ceph] [ 2355.488214] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2355.492959] ? lock_acquire+0xcb/0x2b0 [ 2355.496715] ceph_fill_trace+0x143/0xf30 [ceph] [ 2355.501250] ? __mutex_lock+0x97/0xce0 [ 2355.505021] ? __mutex_lock+0xaec/0xce0 [ 2355.508880] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2355.513419] ? lock_downgrade+0xfd/0x250 [ 2355.517385] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2355.521765] ceph_con_process_message+0x74/0x150 [libceph] [ 2355.527261] ? calc_signature+0xdb/0x100 [libceph] [ 2355.532081] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2355.537438] ? process_scheduled_works+0x22a/0x510 [ 2355.542230] ? process_scheduled_works+0x22a/0x510 [ 2355.547037] ceph_con_workfn+0x344/0x750 [libceph] [ 2355.551860] ? process_scheduled_works+0x22a/0x510 [ 2355.556657] process_scheduled_works+0x2a4/0x510 [ 2355.561299] worker_thread+0x188/0x340 [ 2355.565048] ? rescuer_thread+0x310/0x310 [ 2355.569077] kthread+0xf5/0x130 [ 2355.572240] ? kthread_complete_and_exit+0x20/0x20 [ 2355.577052] ret_from_fork+0x31/0x40 [ 2355.580661] ? kthread_complete_and_exit+0x20/0x20 [ 2355.585449] ret_from_fork_asm+0x11/0x20 [ 2355.589415] [ 2355.591603] irq event stamp: 10745275 [ 2355.595285] hardirqs last enabled at (10745285): [] console_unlock+0x106/0x120 [ 2355.604169] hardirqs last disabled at (10745294): [] console_unlock+0xeb/0x120 [ 2355.612966] softirqs last enabled at (10745242): [] __do_softirq+0x2be/0x371 [ 2355.621676] softirqs last disabled at (10745237): [] irq_exit_rcu+0xb0/0xd0 [ 2355.630195] ---[ end trace 0000000000000000 ]--- [ 2355.634838] ------------[ cut here ]------------ [ 2355.639453] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2355.651097] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2355.659794] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2355.715271] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2355.724395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2355.731875] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2355.737369] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2355.742856] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2355.761601] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2355.766828] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2355.773962] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2355.781094] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008eae [ 2355.788226] R10: 00000000002e3aa0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2355.795359] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2355.802492] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2355.810578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2355.816323] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2355.823448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2355.830582] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2355.837715] Call Trace: [ 2355.840168] [ 2355.842275] ? __warn+0x85/0x180 [ 2355.845507] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.850387] ? report_bug+0xfc/0x1d0 [ 2355.853964] ? prb_read_valid+0x1b/0x20 [ 2355.857807] ? handle_bug+0x44/0x70 [ 2355.861298] ? exc_invalid_op+0x17/0x60 [ 2355.865135] ? asm_exc_invalid_op+0x1a/0x20 [ 2355.869327] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.874202] ? __list_add_valid_or_report+0x7d/0xa0 [ 2355.879082] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2355.883973] ceph_add_cap+0x20b/0x620 [ceph] [ 2355.888257] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2355.892971] ? lock_acquire+0xcb/0x2b0 [ 2355.896722] ceph_fill_trace+0x143/0xf30 [ceph] [ 2355.901259] ? __mutex_lock+0x97/0xce0 [ 2355.905013] ? __mutex_lock+0xaec/0xce0 [ 2355.908855] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2355.913401] ? lock_downgrade+0xfd/0x250 [ 2355.917334] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2355.921712] ceph_con_process_message+0x74/0x150 [libceph] [ 2355.927208] ? calc_signature+0xdb/0x100 [libceph] [ 2355.932012] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2355.937343] ? process_scheduled_works+0x22a/0x510 [ 2355.942140] ? process_scheduled_works+0x22a/0x510 [ 2355.946932] ceph_con_workfn+0x344/0x750 [libceph] [ 2355.951738] ? process_scheduled_works+0x22a/0x510 [ 2355.956528] process_scheduled_works+0x2a4/0x510 [ 2355.961152] worker_thread+0x188/0x340 [ 2355.964903] ? rescuer_thread+0x310/0x310 [ 2355.968912] kthread+0xf5/0x130 [ 2355.972061] ? kthread_complete_and_exit+0x20/0x20 [ 2355.976852] ret_from_fork+0x31/0x40 [ 2355.980432] ? kthread_complete_and_exit+0x20/0x20 [ 2355.985225] ret_from_fork_asm+0x11/0x20 [ 2355.989157] [ 2355.991342] irq event stamp: 10746397 [ 2355.995008] hardirqs last enabled at (10746407): [] console_unlock+0x106/0x120 [ 2356.003876] hardirqs last disabled at (10746416): [] console_unlock+0xeb/0x120 [ 2356.012655] softirqs last enabled at (10746344): [] __do_softirq+0x2be/0x371 [ 2356.021346] softirqs last disabled at (10746339): [] irq_exit_rcu+0xb0/0xd0 [ 2356.029869] ---[ end trace 0000000000000000 ]--- [ 2356.034486] ------------[ cut here ]------------ [ 2356.039109] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2356.050756] WARNING: CPU: 6 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2356.059705] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2356.115170] CPU: 6 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2356.124119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2356.131597] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2356.137082] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2356.155830] RSP: 0018:ffffc90001fc7b78 EFLAGS: 00010286 [ 2356.161055] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2356.168188] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2356.175321] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008eed [ 2356.182454] R10: 00000000002e4ce0 R11: ffffc90001fc7a08 R12: ffff8881c5350750 [ 2356.189586] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2356.196719] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2356.204804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2356.210544] CR2: 00007f9bd22d7020 CR3: 0000000113500001 CR4: 00000000003706e0 [ 2356.217676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2356.224809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2356.231941] Call Trace: [ 2356.234385] [ 2356.236483] ? __warn+0x85/0x180 [ 2356.239717] ? __list_add_valid_or_report+0x7d/0xa0 [ 2356.244596] ? report_bug+0xfc/0x1d0 [ 2356.248175] ? prb_read_valid+0x1b/0x20 [ 2356.252017] ? handle_bug+0x44/0x70 [ 2356.255508] ? exc_invalid_op+0x17/0x60 [ 2356.259347] ? asm_exc_invalid_op+0x1a/0x20 [ 2356.263538] ? __list_add_valid_or_report+0x7d/0xa0 [ 2356.268412] ? __list_add_valid_or_report+0x7d/0xa0 [ 2356.273291] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2356.278182] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2356.283425] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2356.288225] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2356.292942] ? __d_lookup+0xd6/0x200 [ 2356.296519] lookup_fast+0x6d/0xf0 [ 2356.299924] walk_component+0x26/0x160 [ 2356.303678] link_path_walk+0x24d/0x380 [ 2356.307518] ? path_openat+0x80/0x9d0 [ 2356.311187] path_openat+0x8e/0x9d0 [ 2356.314679] do_filp_open+0xb8/0x120 [ 2356.318258] ? lock_release+0xc5/0x260 [ 2356.322011] ? _raw_spin_unlock+0x23/0x40 [ 2356.326025] ? alloc_fd+0x13f/0x200 [ 2356.329521] do_sys_openat2+0x249/0x290 [ 2356.333375] do_sys_open+0x4f/0x80 [ 2356.336796] do_syscall_64+0x38/0x80 [ 2356.340393] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2356.345459] RIP: 0033:0x7f48f0ef2805 [ 2356.349047] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2356.367798] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2356.375365] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2356.382493] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2356.389626] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2356.396758] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2356.403890] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2356.411023] [ 2356.413208] irq event stamp: 12957825 [ 2356.416866] hardirqs last enabled at (12957835): [] console_unlock+0x106/0x120 [ 2356.425732] hardirqs last disabled at (12957844): [] console_unlock+0xeb/0x120 [ 2356.434510] softirqs last enabled at (12957762): [] __do_softirq+0x2be/0x371 [ 2356.443206] softirqs last disabled at (12957863): [] irq_exit_rcu+0xb0/0xd0 [ 2356.451728] ---[ end trace 0000000000000000 ]--- [ 2356.456345] ------------[ cut here ]------------ [ 2356.460961] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2356.472615] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2356.481576] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2356.537049] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2356.546003] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2356.553499] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2356.558986] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2356.577730] RSP: 0018:ffffc90001fcfb78 EFLAGS: 00010286 [ 2356.582958] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2356.590090] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2356.597223] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008f2c [ 2356.604355] R10: 00000000002e5df8 R11: ffffc90001fcfa08 R12: ffff888158a68750 [ 2356.611489] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2356.618620] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2356.626706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2356.632454] CR2: 00007f4f3ea07000 CR3: 000000011aeae004 CR4: 00000000003706f0 [ 2356.639585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2356.646718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2356.653851] Call Trace: [ 2356.656295] [ 2356.658394] ? __warn+0x85/0x180 [ 2356.661626] ? __list_add_valid_or_report+0x7d/0xa0 [ 2356.666506] ? report_bug+0xfc/0x1d0 [ 2356.670084] ? prb_read_valid+0x1b/0x20 [ 2356.673928] ? handle_bug+0x44/0x70 [ 2356.677417] ? exc_invalid_op+0x17/0x60 [ 2356.681256] ? asm_exc_invalid_op+0x1a/0x20 [ 2356.685447] ? __list_add_valid_or_report+0x7d/0xa0 [ 2356.690323] ? __list_add_valid_or_report+0x7d/0xa0 [ 2356.695201] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2356.700101] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2356.705346] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2356.710154] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2356.714869] ? __d_lookup+0xd6/0x200 [ 2356.718448] lookup_fast+0x6d/0xf0 [ 2356.721853] walk_component+0x26/0x160 [ 2356.725606] link_path_walk+0x24d/0x380 [ 2356.729445] ? path_openat+0x80/0x9d0 [ 2356.733111] path_openat+0x8e/0x9d0 [ 2356.736607] do_filp_open+0xb8/0x120 [ 2356.740187] ? lock_release+0xc5/0x260 [ 2356.743936] ? _raw_spin_unlock+0x23/0x40 [ 2356.747947] ? alloc_fd+0x13f/0x200 [ 2356.751443] do_sys_openat2+0x249/0x290 [ 2356.755281] do_sys_open+0x4f/0x80 [ 2356.758688] do_syscall_64+0x38/0x80 [ 2356.762266] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2356.767316] RIP: 0033:0x7f48f0ef2805 [ 2356.770897] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2356.789645] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2356.797219] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2356.804354] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2356.811484] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2356.818616] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2356.825749] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2356.832889] [ 2356.835074] irq event stamp: 12923881 [ 2356.838731] hardirqs last enabled at (12923891): [] console_unlock+0x106/0x120 [ 2356.847599] hardirqs last disabled at (12923900): [] console_unlock+0xeb/0x120 [ 2356.856376] softirqs last enabled at (12923828): [] __do_softirq+0x2be/0x371 [ 2356.865071] softirqs last disabled at (12923823): [] irq_exit_rcu+0xb0/0xd0 [ 2356.873591] ---[ end trace 0000000000000000 ]--- [ 2356.878213] ------------[ cut here ]------------ [ 2356.882829] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2356.894480] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2356.903183] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2356.958686] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2356.967814] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2356.975299] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2356.980820] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2356.986311] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2357.005062] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2357.010294] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2357.017427] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2357.024562] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008f6b [ 2357.031693] R10: 00000000002e7168 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2357.038825] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2357.045967] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2357.054056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2357.059799] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2357.066931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2357.074065] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2357.081198] Call Trace: [ 2357.083675] [ 2357.085780] ? __warn+0x85/0x180 [ 2357.089020] ? __list_add_valid_or_report+0x7d/0xa0 [ 2357.093907] ? report_bug+0xfc/0x1d0 [ 2357.097491] ? prb_read_valid+0x1b/0x20 [ 2357.101336] ? handle_bug+0x44/0x70 [ 2357.104833] ? exc_invalid_op+0x17/0x60 [ 2357.108691] ? asm_exc_invalid_op+0x1a/0x20 [ 2357.112885] ? __list_add_valid_or_report+0x7d/0xa0 [ 2357.117768] ? __list_add_valid_or_report+0x7d/0xa0 [ 2357.122650] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2357.127574] ceph_add_cap+0x20b/0x620 [ceph] [ 2357.131866] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2357.136598] ? lock_acquire+0xcb/0x2b0 [ 2357.140356] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2357.144825] ? __mutex_lock+0x97/0xce0 [ 2357.148582] ? __mutex_lock+0xaec/0xce0 [ 2357.152427] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2357.156971] ? lock_downgrade+0xfd/0x250 [ 2357.160908] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2357.165298] ceph_con_process_message+0x74/0x150 [libceph] [ 2357.170805] ? calc_signature+0xdb/0x100 [libceph] [ 2357.175619] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2357.180979] ? process_scheduled_works+0x22a/0x510 [ 2357.185777] ? process_scheduled_works+0x22a/0x510 [ 2357.190576] ceph_con_workfn+0x344/0x750 [libceph] [ 2357.195386] ? process_scheduled_works+0x22a/0x510 [ 2357.200186] process_scheduled_works+0x2a4/0x510 [ 2357.204814] worker_thread+0x188/0x340 [ 2357.208571] ? rescuer_thread+0x310/0x310 [ 2357.212587] kthread+0xf5/0x130 [ 2357.215738] ? kthread_complete_and_exit+0x20/0x20 [ 2357.220536] ret_from_fork+0x31/0x40 [ 2357.224124] ? kthread_complete_and_exit+0x20/0x20 [ 2357.228915] ret_from_fork_asm+0x11/0x20 [ 2357.232853] [ 2357.235041] irq event stamp: 10749367 [ 2357.238709] hardirqs last enabled at (10749377): [] console_unlock+0x106/0x120 [ 2357.247575] hardirqs last disabled at (10749386): [] console_unlock+0xeb/0x120 [ 2357.256356] softirqs last enabled at (10749314): [] __do_softirq+0x2be/0x371 [ 2357.265056] softirqs last disabled at (10749309): [] irq_exit_rcu+0xb0/0xd0 [ 2357.273575] ---[ end trace 0000000000000000 ]--- [ 2357.278205] ------------[ cut here ]------------ [ 2357.282828] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2357.294475] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2357.303174] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2357.358654] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2357.367800] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2357.375280] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2357.380784] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2357.386270] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2357.405018] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2357.410250] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2357.417384] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2357.424519] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100008faa [ 2357.431652] R10: 00000000002e8280 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2357.438782] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2357.445916] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2357.454000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2357.459748] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2357.466880] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2357.474015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2357.481145] Call Trace: [ 2357.483598] [ 2357.485705] ? __warn+0x85/0x180 [ 2357.488942] ? __list_add_valid_or_report+0x7d/0xa0 [ 2357.493845] ? report_bug+0xfc/0x1d0 [ 2357.497424] ? prb_read_valid+0x1b/0x20 [ 2357.501263] ? handle_bug+0x44/0x70 [ 2357.504756] ? exc_invalid_op+0x17/0x60 [ 2357.507294] watchdog: BUG: soft lockup - CPU#7 stuck for 78s! [kworker/7:4:78808] [ 2357.508597] ? asm_exc_invalid_op+0x1a/0x20 [ 2357.516059] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [ 2357.520253] ? __list_add_valid_or_report+0x7d/0xa0 [ 2357.547136] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio [ 2357.552024] ? __list_add_valid_or_report+0x7d/0xa0 [ 2357.576569] crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2357.576574] irq event stamp: 508058 [ 2357.576575] hardirqs last enabled at (508057): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2357.581452] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2357.588398] hardirqs last disabled at (508058): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2357.588403] softirqs last enabled at (508002): [] __do_softirq+0x2be/0x371 [ 2357.588406] softirqs last disabled at (507997): [] irq_exit_rcu+0xb0/0xd0 [ 2357.591900] ceph_add_cap+0x20b/0x620 [ceph] [ 2357.601876] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2357.601879] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2357.601880] Workqueue: events delayed_work [ceph] [ 2357.606767] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2357.616307] [ 2357.616307] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [ 2357.616310] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [ 2357.616312] RSP: 0018:ffffc90002117d80 EFLAGS: 00000202 [ 2357.616313] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 2357.616315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [ 2357.616316] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 2357.616317] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2357.624844] ceph_fill_trace+0x143/0xf30 [ceph] [ 2357.633163] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2357.633166] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2357.633167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2357.633168] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2357.633169] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2357.633170] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2357.637443] ? __mutex_lock+0x97/0xce0 [ 2357.646814] Call Trace: [ 2357.646815] [ 2357.646817] ? watchdog_timer_fn+0x1b4/0x220 [ 2357.646821] ? __hrtimer_run_queues+0x1d3/0x360 [ 2357.654297] ? __mutex_lock+0xaec/0xce0 [ 2357.658993] ? hrtimer_interrupt+0x104/0x240 [ 2357.658994] ? __local_bh_enable+0x50/0x60 [ 2357.663703] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2357.665189] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2357.665193] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2357.670694] ? lock_downgrade+0xfd/0x250 [ 2357.689430] [ 2357.689430] [ 2357.689432] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2357.689438] ? queued_spin_lock_slowpath+0x2d/0x2e0 [ 2357.694689] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2357.701781] do_raw_spin_lock+0xb5/0xc0 [ 2357.701785] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2357.708924] ceph_con_process_message+0x74/0x150 [libceph] [ 2357.716039] delayed_work+0x1d8/0x280 [ceph] [ 2357.723174] ? calc_signature+0xdb/0x100 [libceph] [ 2357.727695] ? process_scheduled_works+0x22a/0x510 [ 2357.727698] process_scheduled_works+0x2a4/0x510 [ 2357.734832] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2357.742908] worker_thread+0x188/0x340 [ 2357.742912] ? rescuer_thread+0x310/0x310 [ 2357.748658] ? process_scheduled_works+0x22a/0x510 [ 2357.755773] kthread+0xf5/0x130 [ 2357.755776] ? kthread_complete_and_exit+0x20/0x20 [ 2357.755779] ret_from_fork+0x31/0x40 [ 2357.755783] ? kthread_complete_and_exit+0x20/0x20 [ 2357.762914] ? process_scheduled_works+0x22a/0x510 [ 2357.770030] ret_from_fork_asm+0x11/0x20 [ 2357.770038] [ 2357.773787] ceph_con_workfn+0x344/0x750 [libceph] [ 2357.919340] ? process_scheduled_works+0x22a/0x510 [ 2357.924140] process_scheduled_works+0x2a4/0x510 [ 2357.928774] worker_thread+0x188/0x340 [ 2357.932533] ? rescuer_thread+0x310/0x310 [ 2357.936546] kthread+0xf5/0x130 [ 2357.939702] ? kthread_complete_and_exit+0x20/0x20 [ 2357.944503] ret_from_fork+0x31/0x40 [ 2357.948079] ? kthread_complete_and_exit+0x20/0x20 [ 2357.952878] ret_from_fork_asm+0x11/0x20 [ 2357.956821] [ 2357.959017] irq event stamp: 10750635 [ 2357.962700] hardirqs last enabled at (10750645): [] console_unlock+0x106/0x120 [ 2357.971569] hardirqs last disabled at (10750654): [] console_unlock+0xeb/0x120 [ 2357.980353] softirqs last enabled at (10750612): [] __do_softirq+0x2be/0x371 [ 2357.989048] softirqs last disabled at (10750607): [] irq_exit_rcu+0xb0/0xd0 [ 2357.997575] ---[ end trace 0000000000000000 ]--- [ 2358.002384] ------------[ cut here ]------------ [ 2358.007007] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2358.018654] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2358.027347] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2358.082853] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2358.091988] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2358.099471] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2358.104991] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2358.110478] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2358.129227] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2358.134462] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2358.141594] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2358.148727] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000901b [ 2358.155861] R10: 00000000002e9ee0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2358.162991] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2358.170126] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2358.178210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2358.183959] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2358.191091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2358.198224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2358.205363] Call Trace: [ 2358.207816] [ 2358.209923] ? __warn+0x85/0x180 [ 2358.213159] ? __list_add_valid_or_report+0x7d/0xa0 [ 2358.218043] ? report_bug+0xfc/0x1d0 [ 2358.221624] ? prb_read_valid+0x1b/0x20 [ 2358.225463] ? handle_bug+0x44/0x70 [ 2358.228957] ? exc_invalid_op+0x17/0x60 [ 2358.232811] ? asm_exc_invalid_op+0x1a/0x20 [ 2358.237006] ? __list_add_valid_or_report+0x7d/0xa0 [ 2358.241888] ? __list_add_valid_or_report+0x7d/0xa0 [ 2358.246775] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2358.251678] ceph_add_cap+0x20b/0x620 [ceph] [ 2358.255998] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2358.260727] ? lock_acquire+0xcb/0x2b0 [ 2358.264488] ceph_fill_trace+0x143/0xf30 [ceph] [ 2358.269031] ? __mutex_lock+0x97/0xce0 [ 2358.272784] ? __mutex_lock+0xaec/0xce0 [ 2358.276626] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2358.281182] ? lock_downgrade+0xfd/0x250 [ 2358.285115] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2358.289502] ceph_con_process_message+0x74/0x150 [libceph] [ 2358.295002] ? calc_signature+0xdb/0x100 [libceph] [ 2358.299820] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2358.305150] ? process_scheduled_works+0x22a/0x510 [ 2358.309951] ? process_scheduled_works+0x22a/0x510 [ 2358.314745] ceph_con_workfn+0x344/0x750 [libceph] [ 2358.319563] ? process_scheduled_works+0x22a/0x510 [ 2358.324357] process_scheduled_works+0x2a4/0x510 [ 2358.328999] worker_thread+0x188/0x340 [ 2358.332758] ? rescuer_thread+0x310/0x310 [ 2358.336770] kthread+0xf5/0x130 [ 2358.339919] ? kthread_complete_and_exit+0x20/0x20 [ 2358.344720] ret_from_fork+0x31/0x40 [ 2358.348305] ? kthread_complete_and_exit+0x20/0x20 [ 2358.353102] ret_from_fork_asm+0x11/0x20 [ 2358.357044] [ 2358.359235] irq event stamp: 10751771 [ 2358.362899] hardirqs last enabled at (10751781): [] console_unlock+0x106/0x120 [ 2358.371766] hardirqs last disabled at (10751790): [] console_unlock+0xeb/0x120 [ 2358.380544] softirqs last enabled at (10751718): [] __do_softirq+0x2be/0x371 [ 2358.389242] softirqs last disabled at (10751809): [] irq_exit_rcu+0xb0/0xd0 [ 2358.397767] ---[ end trace 0000000000000000 ]--- [ 2358.402409] ------------[ cut here ]------------ [ 2358.407034] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2358.418700] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2358.427655] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2358.483197] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2358.492147] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2358.499628] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2358.505112] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2358.523860] RSP: 0018:ffffc90001ff7b78 EFLAGS: 00010286 [ 2358.529085] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2358.536218] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2358.543351] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000905a [ 2358.550484] R10: 00000000002eaff8 R11: ffffc90001ff7a08 R12: ffff888158a68750 [ 2358.557616] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2358.564751] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2358.572835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2358.578583] CR2: 00007fb38c4fb000 CR3: 0000000120cc8004 CR4: 00000000003706e0 [ 2358.585716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2358.592847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2358.599992] Call Trace: [ 2358.602441] [ 2358.604550] ? __warn+0x85/0x180 [ 2358.607783] ? __list_add_valid_or_report+0x7d/0xa0 [ 2358.612661] ? report_bug+0xfc/0x1d0 [ 2358.616240] ? prb_read_valid+0x1b/0x20 [ 2358.620083] ? handle_bug+0x44/0x70 [ 2358.623571] ? exc_invalid_op+0x17/0x60 [ 2358.627411] ? asm_exc_invalid_op+0x1a/0x20 [ 2358.631606] ? __list_add_valid_or_report+0x7d/0xa0 [ 2358.636487] ? __list_add_valid_or_report+0x7d/0xa0 [ 2358.641368] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2358.646266] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2358.651518] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2358.656326] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2358.661051] ? __d_lookup+0xd6/0x200 [ 2358.664637] lookup_fast+0x6d/0xf0 [ 2358.668042] walk_component+0x26/0x160 [ 2358.671797] link_path_walk+0x24d/0x380 [ 2358.675634] ? path_openat+0x80/0x9d0 [ 2358.679304] path_openat+0x8e/0x9d0 [ 2358.682809] do_filp_open+0xb8/0x120 [ 2358.686398] ? lock_release+0xc5/0x260 [ 2358.690155] ? _raw_spin_unlock+0x23/0x40 [ 2358.694172] ? alloc_fd+0x13f/0x200 [ 2358.697669] do_sys_openat2+0x249/0x290 [ 2358.701515] do_sys_open+0x4f/0x80 [ 2358.704921] do_syscall_64+0x38/0x80 [ 2358.708508] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2358.713559] RIP: 0033:0x7f48f0eed2f2 [ 2358.717139] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2358.735886] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2358.743451] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2358.750583] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2358.757716] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2358.764849] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2358.771998] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2358.779145] [ 2358.781334] irq event stamp: 12969569 [ 2358.784999] hardirqs last enabled at (12969579): [] console_unlock+0x106/0x120 [ 2358.793866] hardirqs last disabled at (12969588): [] console_unlock+0xeb/0x120 [ 2358.802645] softirqs last enabled at (12969290): [] __do_softirq+0x2be/0x371 [ 2358.811337] softirqs last disabled at (12969285): [] irq_exit_rcu+0xb0/0xd0 [ 2358.819857] ---[ end trace 0000000000000000 ]--- [ 2358.824477] ------------[ cut here ]------------ [ 2358.829095] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2358.840744] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2358.849444] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2358.904925] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2358.914053] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2358.921528] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2358.927032] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2358.932516] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2358.951269] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2358.956498] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2358.963634] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2358.970766] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009099 [ 2358.977904] R10: 00000000002ec110 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2358.985039] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2358.992174] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2359.000257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2359.006020] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2359.013154] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2359.020290] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2359.027418] Call Trace: [ 2359.029876] [ 2359.031996] ? __warn+0x85/0x180 [ 2359.035232] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.040117] ? report_bug+0xfc/0x1d0 [ 2359.043720] ? prb_read_valid+0x1b/0x20 [ 2359.047564] ? handle_bug+0x44/0x70 [ 2359.051056] ? exc_invalid_op+0x17/0x60 [ 2359.054894] ? asm_exc_invalid_op+0x1a/0x20 [ 2359.059088] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.063969] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.068866] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2359.073765] ceph_add_cap+0x20b/0x620 [ceph] [ 2359.078070] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2359.082794] ? lock_acquire+0xcb/0x2b0 [ 2359.086554] ceph_fill_trace+0x143/0xf30 [ceph] [ 2359.091096] ? __mutex_lock+0x97/0xce0 [ 2359.094848] ? __mutex_lock+0xaec/0xce0 [ 2359.098694] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2359.103263] ? lock_downgrade+0xfd/0x250 [ 2359.107202] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2359.111594] ceph_con_process_message+0x74/0x150 [libceph] [ 2359.117094] ? calc_signature+0xdb/0x100 [libceph] [ 2359.121911] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2359.127240] ? process_scheduled_works+0x22a/0x510 [ 2359.132044] ? process_scheduled_works+0x22a/0x510 [ 2359.136837] ceph_con_workfn+0x344/0x750 [libceph] [ 2359.141656] ? process_scheduled_works+0x22a/0x510 [ 2359.146449] process_scheduled_works+0x2a4/0x510 [ 2359.151079] worker_thread+0x188/0x340 [ 2359.154832] ? rescuer_thread+0x310/0x310 [ 2359.158847] kthread+0xf5/0x130 [ 2359.162005] ? kthread_complete_and_exit+0x20/0x20 [ 2359.166803] ret_from_fork+0x31/0x40 [ 2359.170391] ? kthread_complete_and_exit+0x20/0x20 [ 2359.175192] ret_from_fork_asm+0x11/0x20 [ 2359.179128] [ 2359.181319] irq event stamp: 10754929 [ 2359.184982] hardirqs last enabled at (10754939): [] console_unlock+0x106/0x120 [ 2359.193868] hardirqs last disabled at (10754948): [] console_unlock+0xeb/0x120 [ 2359.202646] softirqs last enabled at (10754896): [] __do_softirq+0x2be/0x371 [ 2359.211341] softirqs last disabled at (10754891): [] irq_exit_rcu+0xb0/0xd0 [ 2359.219867] ---[ end trace 0000000000000000 ]--- [ 2359.224535] ------------[ cut here ]------------ [ 2359.229162] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2359.240801] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2359.249759] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2359.305227] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2359.314177] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2359.321657] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2359.327140] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2359.345889] RSP: 0018:ffffc90001fdfb78 EFLAGS: 00010286 [ 2359.351113] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2359.358240] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2359.365372] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000090d8 [ 2359.372505] R10: 00000000002ed358 R11: ffffc90001fdfa08 R12: ffff8881d1146920 [ 2359.379636] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2359.386769] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2359.394856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2359.400601] CR2: 00007fb38c506000 CR3: 0000000179402004 CR4: 00000000003706e0 [ 2359.407725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2359.414849] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2359.421975] Call Trace: [ 2359.424428] [ 2359.426534] ? __warn+0x85/0x180 [ 2359.429767] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.434646] ? report_bug+0xfc/0x1d0 [ 2359.438226] ? prb_read_valid+0x1b/0x20 [ 2359.442065] ? handle_bug+0x44/0x70 [ 2359.445557] ? exc_invalid_op+0x17/0x60 [ 2359.449397] ? asm_exc_invalid_op+0x1a/0x20 [ 2359.453586] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.458463] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.463342] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2359.468232] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2359.473468] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2359.478267] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2359.482983] ? __d_lookup+0xd6/0x200 [ 2359.486560] lookup_fast+0x6d/0xf0 [ 2359.489967] walk_component+0x26/0x160 [ 2359.493720] link_path_walk+0x24d/0x380 [ 2359.497557] ? path_openat+0x80/0x9d0 [ 2359.501228] path_openat+0x8e/0x9d0 [ 2359.504720] do_filp_open+0xb8/0x120 [ 2359.508300] ? lock_release+0xc5/0x260 [ 2359.512051] ? _raw_spin_unlock+0x23/0x40 [ 2359.516060] ? alloc_fd+0x13f/0x200 [ 2359.519557] do_sys_openat2+0x249/0x290 [ 2359.523395] do_sys_open+0x4f/0x80 [ 2359.526800] do_syscall_64+0x38/0x80 [ 2359.530379] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2359.535431] RIP: 0033:0x7f48f0eed2f2 [ 2359.539014] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2359.557758] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2359.565325] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2359.572456] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2359.579589] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2359.586712] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2359.593836] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2359.600968] [ 2359.603153] irq event stamp: 12872031 [ 2359.606811] hardirqs last enabled at (12872041): [] console_unlock+0x106/0x120 [ 2359.615677] hardirqs last disabled at (12872050): [] console_unlock+0xeb/0x120 [ 2359.624458] softirqs last enabled at (12871968): [] __do_softirq+0x2be/0x371 [ 2359.633152] softirqs last disabled at (12872069): [] irq_exit_rcu+0xb0/0xd0 [ 2359.641669] ---[ end trace 0000000000000000 ]--- [ 2359.646280] ------------[ cut here ]------------ [ 2359.650901] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2359.662550] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2359.671245] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2359.726735] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2359.735864] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2359.743343] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2359.748857] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2359.754348] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2359.773097] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2359.778321] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2359.785457] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2359.792590] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009117 [ 2359.799735] R10: 00000000002ee460 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2359.806871] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2359.814021] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2359.822108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2359.827853] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2359.834987] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2359.842138] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2359.849269] Call Trace: [ 2359.851738] [ 2359.853846] ? __warn+0x85/0x180 [ 2359.857082] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.861966] ? report_bug+0xfc/0x1d0 [ 2359.865545] ? prb_read_valid+0x1b/0x20 [ 2359.869387] ? handle_bug+0x44/0x70 [ 2359.872879] ? exc_invalid_op+0x17/0x60 [ 2359.876734] ? asm_exc_invalid_op+0x1a/0x20 [ 2359.880930] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.885811] ? __list_add_valid_or_report+0x7d/0xa0 [ 2359.890689] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2359.895610] ceph_add_cap+0x20b/0x620 [ceph] [ 2359.899902] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2359.904634] ? lock_acquire+0xcb/0x2b0 [ 2359.908394] ceph_fill_trace+0x143/0xf30 [ceph] [ 2359.912947] ? __mutex_lock+0x97/0xce0 [ 2359.916696] ? __mutex_lock+0xaec/0xce0 [ 2359.920541] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2359.925088] ? lock_downgrade+0xfd/0x250 [ 2359.929026] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2359.933412] ceph_con_process_message+0x74/0x150 [libceph] [ 2359.938919] ? calc_signature+0xdb/0x100 [libceph] [ 2359.943743] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2359.949085] ? process_scheduled_works+0x22a/0x510 [ 2359.953882] ? process_scheduled_works+0x22a/0x510 [ 2359.958682] ceph_con_workfn+0x344/0x750 [libceph] [ 2359.963492] ? process_scheduled_works+0x22a/0x510 [ 2359.968292] process_scheduled_works+0x2a4/0x510 [ 2359.972936] worker_thread+0x188/0x340 [ 2359.976693] ? rescuer_thread+0x310/0x310 [ 2359.980731] kthread+0xf5/0x130 [ 2359.983888] ? kthread_complete_and_exit+0x20/0x20 [ 2359.988685] ret_from_fork+0x31/0x40 [ 2359.992264] ? kthread_complete_and_exit+0x20/0x20 [ 2359.997056] ret_from_fork_asm+0x11/0x20 [ 2360.001009] [ 2360.003201] irq event stamp: 10756985 [ 2360.006867] hardirqs last enabled at (10756995): [] console_unlock+0x106/0x120 [ 2360.015733] hardirqs last disabled at (10757004): [] console_unlock+0xeb/0x120 [ 2360.024512] softirqs last enabled at (10756932): [] __do_softirq+0x2be/0x371 [ 2360.033205] softirqs last disabled at (10756927): [] irq_exit_rcu+0xb0/0xd0 [ 2360.041742] ---[ end trace 0000000000000000 ]--- [ 2360.046550] ------------[ cut here ]------------ [ 2360.051176] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2360.062828] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2360.071521] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2360.127042] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2360.136169] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2360.143647] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2360.149158] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2360.154642] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2360.173391] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2360.178615] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2360.185751] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2360.192886] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009156 [ 2360.200032] R10: 00000000002ef578 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2360.207164] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2360.214298] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2360.222385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.228132] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2360.235267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2360.242400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2360.249528] Call Trace: [ 2360.251982] [ 2360.254088] ? __warn+0x85/0x180 [ 2360.257325] ? __list_add_valid_or_report+0x7d/0xa0 [ 2360.262211] ? report_bug+0xfc/0x1d0 [ 2360.265790] ? prb_read_valid+0x1b/0x20 [ 2360.269631] ? handle_bug+0x44/0x70 [ 2360.273129] ? exc_invalid_op+0x17/0x60 [ 2360.276970] ? asm_exc_invalid_op+0x1a/0x20 [ 2360.281171] ? __list_add_valid_or_report+0x7d/0xa0 [ 2360.286053] ? __list_add_valid_or_report+0x7d/0xa0 [ 2360.290933] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2360.295831] ceph_add_cap+0x20b/0x620 [ceph] [ 2360.300135] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2360.304861] ? lock_acquire+0xcb/0x2b0 [ 2360.308628] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2360.313085] ? __mutex_lock+0x97/0xce0 [ 2360.316836] ? __mutex_lock+0xaec/0xce0 [ 2360.320681] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2360.325235] ? lock_downgrade+0xfd/0x250 [ 2360.329169] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2360.333556] ceph_con_process_message+0x74/0x150 [libceph] [ 2360.339056] ? calc_signature+0xdb/0x100 [libceph] [ 2360.343873] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2360.349204] ? process_scheduled_works+0x22a/0x510 [ 2360.354005] ? process_scheduled_works+0x22a/0x510 [ 2360.358816] ceph_con_workfn+0x344/0x750 [libceph] [ 2360.363634] ? process_scheduled_works+0x22a/0x510 [ 2360.368427] process_scheduled_works+0x2a4/0x510 [ 2360.373056] worker_thread+0x188/0x340 [ 2360.376813] ? rescuer_thread+0x310/0x310 [ 2360.380832] kthread+0xf5/0x130 [ 2360.383981] ? kthread_complete_and_exit+0x20/0x20 [ 2360.388784] ret_from_fork+0x31/0x40 [ 2360.392365] ? kthread_complete_and_exit+0x20/0x20 [ 2360.397162] ret_from_fork_asm+0x11/0x20 [ 2360.401100] [ 2360.403297] irq event stamp: 10758169 [ 2360.406962] hardirqs last enabled at (10758179): [] console_unlock+0x106/0x120 [ 2360.415830] hardirqs last disabled at (10758188): [] console_unlock+0xeb/0x120 [ 2360.424616] softirqs last enabled at (10758116): [] __do_softirq+0x2be/0x371 [ 2360.433309] softirqs last disabled at (10758111): [] irq_exit_rcu+0xb0/0xd0 [ 2360.441828] ---[ end trace 0000000000000000 ]--- [ 2360.446456] ------------[ cut here ]------------ [ 2360.451079] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2360.462742] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2360.471433] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2360.526940] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2360.536074] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2360.543551] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2360.549055] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2360.554540] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2360.573289] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2360.578521] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2360.585656] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2360.592790] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009195 [ 2360.599920] R10: 00000000002f0690 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2360.607053] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2360.614186] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2360.622271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.628017] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2360.635167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2360.642301] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2360.649433] Call Trace: [ 2360.651890] [ 2360.653994] ? __warn+0x85/0x180 [ 2360.657248] ? __list_add_valid_or_report+0x7d/0xa0 [ 2360.662132] ? report_bug+0xfc/0x1d0 [ 2360.665712] ? prb_read_valid+0x1b/0x20 [ 2360.669575] ? handle_bug+0x44/0x70 [ 2360.673069] ? exc_invalid_op+0x17/0x60 [ 2360.676911] ? asm_exc_invalid_op+0x1a/0x20 [ 2360.681111] ? __list_add_valid_or_report+0x7d/0xa0 [ 2360.685994] ? __list_add_valid_or_report+0x7d/0xa0 [ 2360.690878] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2360.695781] ceph_add_cap+0x20b/0x620 [ceph] [ 2360.700075] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2360.704813] ceph_fill_trace+0x143/0xf30 [ceph] [ 2360.709358] ? __mutex_lock+0x97/0xce0 [ 2360.713108] ? __mutex_lock+0xaec/0xce0 [ 2360.716950] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2360.721512] ? lock_downgrade+0xfd/0x250 [ 2360.725450] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2360.729840] ceph_con_process_message+0x74/0x150 [libceph] [ 2360.735339] ? calc_signature+0xdb/0x100 [libceph] [ 2360.740155] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2360.745487] ? process_scheduled_works+0x22a/0x510 [ 2360.750287] ? process_scheduled_works+0x22a/0x510 [ 2360.755082] ceph_con_workfn+0x344/0x750 [libceph] [ 2360.759897] ? process_scheduled_works+0x22a/0x510 [ 2360.764693] process_scheduled_works+0x2a4/0x510 [ 2360.769321] worker_thread+0x188/0x340 [ 2360.773076] ? rescuer_thread+0x310/0x310 [ 2360.777088] kthread+0xf5/0x130 [ 2360.780234] ? kthread_complete_and_exit+0x20/0x20 [ 2360.785032] ret_from_fork+0x31/0x40 [ 2360.788614] ? kthread_complete_and_exit+0x20/0x20 [ 2360.793410] ret_from_fork_asm+0x11/0x20 [ 2360.797346] [ 2360.799544] irq event stamp: 10759221 [ 2360.803210] hardirqs last enabled at (10759231): [] console_unlock+0x106/0x120 [ 2360.812077] hardirqs last disabled at (10759240): [] console_unlock+0xeb/0x120 [ 2360.820855] softirqs last enabled at (10759178): [] __do_softirq+0x2be/0x371 [ 2360.829548] softirqs last disabled at (10759173): [] irq_exit_rcu+0xb0/0xd0 [ 2360.838068] ---[ end trace 0000000000000000 ]--- [ 2360.842736] ------------[ cut here ]------------ [ 2360.847355] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2360.859008] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2360.867747] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2360.923228] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2360.932356] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2360.939834] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2360.945345] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2360.950831] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2360.969580] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2360.974810] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2360.981947] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2360.989081] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000091d3 [ 2360.996212] R10: 00000000002f1780 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2361.003342] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2361.010477] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2361.018563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2361.024309] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2361.031440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2361.038576] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2361.045709] Call Trace: [ 2361.048160] [ 2361.050265] ? __warn+0x85/0x180 [ 2361.053505] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.058388] ? report_bug+0xfc/0x1d0 [ 2361.061967] ? prb_read_valid+0x1b/0x20 [ 2361.065807] ? handle_bug+0x44/0x70 [ 2361.069299] ? exc_invalid_op+0x17/0x60 [ 2361.073137] ? asm_exc_invalid_op+0x1a/0x20 [ 2361.077331] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.082213] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.087093] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2361.091991] ceph_add_cap+0x20b/0x620 [ceph] [ 2361.096304] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2361.101043] ? lock_acquire+0xcb/0x2b0 [ 2361.104808] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2361.109262] ? __mutex_lock+0x97/0xce0 [ 2361.113014] ? __mutex_lock+0xaec/0xce0 [ 2361.116876] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2361.121431] ? lock_downgrade+0xfd/0x250 [ 2361.125364] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2361.129770] ceph_con_process_message+0x74/0x150 [libceph] [ 2361.135269] ? calc_signature+0xdb/0x100 [libceph] [ 2361.140085] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2361.145417] ? process_scheduled_works+0x22a/0x510 [ 2361.150217] ? process_scheduled_works+0x22a/0x510 [ 2361.155013] ceph_con_workfn+0x344/0x750 [libceph] [ 2361.159845] ? process_scheduled_works+0x22a/0x510 [ 2361.164640] process_scheduled_works+0x2a4/0x510 [ 2361.169271] worker_thread+0x188/0x340 [ 2361.173039] ? rescuer_thread+0x310/0x310 [ 2361.177054] kthread+0xf5/0x130 [ 2361.180202] ? kthread_complete_and_exit+0x20/0x20 [ 2361.185005] ret_from_fork+0x31/0x40 [ 2361.188588] ? kthread_complete_and_exit+0x20/0x20 [ 2361.193383] ret_from_fork_asm+0x11/0x20 [ 2361.197319] [ 2361.199510] irq event stamp: 10760335 [ 2361.203172] hardirqs last enabled at (10760345): [] console_unlock+0x106/0x120 [ 2361.212041] hardirqs last disabled at (10760354): [] console_unlock+0xeb/0x120 [ 2361.220819] softirqs last enabled at (10760282): [] __do_softirq+0x2be/0x371 [ 2361.229513] softirqs last disabled at (10760277): [] irq_exit_rcu+0xb0/0xd0 [ 2361.238039] ---[ end trace 0000000000000000 ]--- [ 2361.242667] ------------[ cut here ]------------ [ 2361.247290] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2361.258936] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2361.267629] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2361.323170] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2361.332303] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2361.339781] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2361.345294] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2361.350779] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2361.369525] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2361.374760] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2361.381895] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2361.389044] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009212 [ 2361.396176] R10: 00000000002f29c8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2361.403308] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2361.410442] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2361.418527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2361.424274] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2361.431405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2361.438540] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2361.445671] Call Trace: [ 2361.448126] [ 2361.450231] ? __warn+0x85/0x180 [ 2361.453469] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.458353] ? report_bug+0xfc/0x1d0 [ 2361.461933] ? prb_read_valid+0x1b/0x20 [ 2361.465775] ? handle_bug+0x44/0x70 [ 2361.469273] ? exc_invalid_op+0x17/0x60 [ 2361.473112] ? asm_exc_invalid_op+0x1a/0x20 [ 2361.477306] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.482187] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.487067] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2361.491966] ceph_add_cap+0x20b/0x620 [ceph] [ 2361.496270] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2361.501005] ceph_fill_trace+0x143/0xf30 [ceph] [ 2361.505572] ? __mutex_lock+0x97/0xce0 [ 2361.509332] ? __mutex_lock+0xaec/0xce0 [ 2361.513195] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2361.517773] ? lock_downgrade+0xfd/0x250 [ 2361.521725] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2361.526125] ceph_con_process_message+0x74/0x150 [libceph] [ 2361.531633] ? calc_signature+0xdb/0x100 [libceph] [ 2361.536489] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2361.541832] ? process_scheduled_works+0x22a/0x510 [ 2361.546656] ? process_scheduled_works+0x22a/0x510 [ 2361.551453] ceph_con_workfn+0x344/0x750 [libceph] [ 2361.556296] ? process_scheduled_works+0x22a/0x510 [ 2361.561090] process_scheduled_works+0x2a4/0x510 [ 2361.565759] worker_thread+0x188/0x340 [ 2361.569515] ? rescuer_thread+0x310/0x310 [ 2361.573537] kthread+0xf5/0x130 [ 2361.576700] ? kthread_complete_and_exit+0x20/0x20 [ 2361.581541] ret_from_fork+0x31/0x40 [ 2361.585124] ? kthread_complete_and_exit+0x20/0x20 [ 2361.589935] ret_from_fork_asm+0x11/0x20 [ 2361.593883] [ 2361.596077] irq event stamp: 10761391 [ 2361.599767] hardirqs last enabled at (10761401): [] console_unlock+0x106/0x120 [ 2361.608636] hardirqs last disabled at (10761410): [] console_unlock+0xeb/0x120 [ 2361.617430] softirqs last enabled at (10761348): [] __do_softirq+0x2be/0x371 [ 2361.626140] softirqs last disabled at (10761343): [] irq_exit_rcu+0xb0/0xd0 [ 2361.634676] ---[ end trace 0000000000000000 ]--- [ 2361.639355] ------------[ cut here ]------------ [ 2361.643978] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2361.655620] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2361.664318] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2361.719793] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2361.728919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2361.736400] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2361.741893] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2361.747379] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2361.766127] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2361.771352] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2361.778485] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2361.785619] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009250 [ 2361.792750] R10: 00000000002f3ac8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2361.799882] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2361.807017] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2361.815101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2361.820848] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2361.827982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2361.835115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2361.842247] Call Trace: [ 2361.844699] [ 2361.846806] ? __warn+0x85/0x180 [ 2361.850040] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.854918] ? report_bug+0xfc/0x1d0 [ 2361.858497] ? prb_read_valid+0x1b/0x20 [ 2361.862337] ? handle_bug+0x44/0x70 [ 2361.865829] ? exc_invalid_op+0x17/0x60 [ 2361.869669] ? asm_exc_invalid_op+0x1a/0x20 [ 2361.873861] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.878746] ? __list_add_valid_or_report+0x7d/0xa0 [ 2361.883622] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2361.888514] ceph_add_cap+0x20b/0x620 [ceph] [ 2361.892798] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2361.897520] ? lock_acquire+0xcb/0x2b0 [ 2361.901273] ceph_fill_trace+0x143/0xf30 [ceph] [ 2361.905811] ? __mutex_lock+0x97/0xce0 [ 2361.909561] ? __mutex_lock+0xaec/0xce0 [ 2361.913405] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2361.917952] ? lock_downgrade+0xfd/0x250 [ 2361.921882] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2361.926263] ceph_con_process_message+0x74/0x150 [libceph] [ 2361.931758] ? calc_signature+0xdb/0x100 [libceph] [ 2361.936562] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2361.941893] ? process_scheduled_works+0x22a/0x510 [ 2361.946684] ? process_scheduled_works+0x22a/0x510 [ 2361.951474] ceph_con_workfn+0x344/0x750 [libceph] [ 2361.956285] ? process_scheduled_works+0x22a/0x510 [ 2361.961079] process_scheduled_works+0x2a4/0x510 [ 2361.965705] worker_thread+0x188/0x340 [ 2361.969459] ? rescuer_thread+0x310/0x310 [ 2361.973472] kthread+0xf5/0x130 [ 2361.976619] ? kthread_complete_and_exit+0x20/0x20 [ 2361.981410] ret_from_fork+0x31/0x40 [ 2361.984990] ? kthread_complete_and_exit+0x20/0x20 [ 2361.989783] ret_from_fork_asm+0x11/0x20 [ 2361.993716] [ 2361.995901] irq event stamp: 10762515 [ 2361.999566] hardirqs last enabled at (10762525): [] console_unlock+0x106/0x120 [ 2362.008433] hardirqs last disabled at (10762534): [] console_unlock+0xeb/0x120 [ 2362.017214] softirqs last enabled at (10762462): [] __do_softirq+0x2be/0x371 [ 2362.025904] softirqs last disabled at (10762457): [] irq_exit_rcu+0xb0/0xd0 [ 2362.034424] ---[ end trace 0000000000000000 ]--- [ 2362.039195] ------------[ cut here ]------------ [ 2362.043816] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2362.055457] WARNING: CPU: 2 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2362.064412] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2362.119872] CPU: 2 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2362.128824] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2362.136304] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2362.141789] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2362.160534] RSP: 0018:ffffc90001957b78 EFLAGS: 00010286 [ 2362.165760] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2362.172885] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2362.180008] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000928f [ 2362.187133] R10: 00000000002f4bd0 R11: ffffc90001957a08 R12: ffff8881c5350750 [ 2362.194257] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2362.201380] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2362.209459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2362.215204] CR2: 00007fb38c506000 CR3: 000000016696c002 CR4: 00000000003706e0 [ 2362.222330] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2362.229461] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2362.236585] Call Trace: [ 2362.239032] [ 2362.241137] ? __warn+0x85/0x180 [ 2362.244371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2362.249248] ? report_bug+0xfc/0x1d0 [ 2362.252830] ? prb_read_valid+0x1b/0x20 [ 2362.256668] ? handle_bug+0x44/0x70 [ 2362.260159] ? exc_invalid_op+0x17/0x60 [ 2362.263990] ? asm_exc_invalid_op+0x1a/0x20 [ 2362.268182] ? __list_add_valid_or_report+0x7d/0xa0 [ 2362.273057] ? __list_add_valid_or_report+0x7d/0xa0 [ 2362.277938] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2362.282827] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2362.288070] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2362.292871] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2362.297586] ? __d_lookup+0xd6/0x200 [ 2362.301165] lookup_fast+0x6d/0xf0 [ 2362.304570] walk_component+0x26/0x160 [ 2362.308324] link_path_walk+0x24d/0x380 [ 2362.312160] ? path_openat+0x80/0x9d0 [ 2362.315822] path_openat+0x8e/0x9d0 [ 2362.319314] do_filp_open+0xb8/0x120 [ 2362.322895] ? lock_release+0xc5/0x260 [ 2362.326646] ? _raw_spin_unlock+0x23/0x40 [ 2362.330657] ? alloc_fd+0x13f/0x200 [ 2362.334152] do_sys_openat2+0x249/0x290 [ 2362.337989] do_sys_open+0x4f/0x80 [ 2362.341394] do_syscall_64+0x38/0x80 [ 2362.344973] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2362.350025] RIP: 0033:0x7f48f0ef2805 [ 2362.353599] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2362.372342] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2362.379909] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2362.387041] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2362.394166] RBP: 000000000207da30 R08: 0000000003ffffff R09: 0000000000000000 [ 2362.401300] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2362.408431] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2362.415563] [ 2362.417752] irq event stamp: 12847259 [ 2362.421415] hardirqs last enabled at (12847269): [] console_unlock+0x106/0x120 [ 2362.430281] hardirqs last disabled at (12847278): [] console_unlock+0xeb/0x120 [ 2362.439061] softirqs last enabled at (12847196): [] __do_softirq+0x2be/0x371 [ 2362.447756] softirqs last disabled at (12847297): [] irq_exit_rcu+0xb0/0xd0 [ 2362.456273] ---[ end trace 0000000000000000 ]--- [ 2362.460885] ------------[ cut here ]------------ [ 2362.465505] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2362.477167] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2362.485865] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2362.541348] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2362.550493] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2362.557988] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2362.563483] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2362.568987] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2362.587731] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2362.592976] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2362.600108] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2362.607259] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000092ce [ 2362.614409] R10: 00000000002f5ce8 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2362.621559] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2362.628710] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2362.636812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2362.642558] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2362.649700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2362.656843] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2362.663976] Call Trace: [ 2362.666428] [ 2362.668552] ? __warn+0x85/0x180 [ 2362.671800] ? __list_add_valid_or_report+0x7d/0xa0 [ 2362.676679] ? report_bug+0xfc/0x1d0 [ 2362.680279] ? prb_read_valid+0x1b/0x20 [ 2362.684135] ? handle_bug+0x44/0x70 [ 2362.687644] ? exc_invalid_op+0x17/0x60 [ 2362.691500] ? asm_exc_invalid_op+0x1a/0x20 [ 2362.695725] ? __list_add_valid_or_report+0x7d/0xa0 [ 2362.700601] ? __list_add_valid_or_report+0x7d/0xa0 [ 2362.705498] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2362.710407] ceph_add_cap+0x20b/0x620 [ceph] [ 2362.714708] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2362.719447] ? lock_acquire+0xcb/0x2b0 [ 2362.723217] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2362.727668] ? __mutex_lock+0x97/0xce0 [ 2362.731438] ? __mutex_lock+0xaec/0xce0 [ 2362.735296] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2362.739852] ? lock_downgrade+0xfd/0x250 [ 2362.743792] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2362.748188] ceph_con_process_message+0x74/0x150 [libceph] [ 2362.753686] ? calc_signature+0xdb/0x100 [libceph] [ 2362.758532] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2362.763854] ? process_scheduled_works+0x22a/0x510 [ 2362.768647] ? process_scheduled_works+0x22a/0x510 [ 2362.773455] ceph_con_workfn+0x344/0x750 [libceph] [ 2362.778275] ? process_scheduled_works+0x22a/0x510 [ 2362.783083] process_scheduled_works+0x2a4/0x510 [ 2362.787741] worker_thread+0x188/0x340 [ 2362.791491] ? rescuer_thread+0x310/0x310 [ 2362.795519] kthread+0xf5/0x130 [ 2362.798682] ? kthread_complete_and_exit+0x20/0x20 [ 2362.803494] ret_from_fork+0x31/0x40 [ 2362.807090] ? kthread_complete_and_exit+0x20/0x20 [ 2362.811900] ret_from_fork_asm+0x11/0x20 [ 2362.815833] [ 2362.818019] irq event stamp: 10765459 [ 2362.821685] hardirqs last enabled at (10765469): [] console_unlock+0x106/0x120 [ 2362.830568] hardirqs last disabled at (10765478): [] console_unlock+0xeb/0x120 [ 2362.839364] softirqs last enabled at (10765426): [] __do_softirq+0x2be/0x371 [ 2362.848075] softirqs last disabled at (10765421): [] irq_exit_rcu+0xb0/0xd0 [ 2362.856611] ---[ end trace 0000000000000000 ]--- [ 2362.861263] ------------[ cut here ]------------ [ 2362.865878] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2362.877539] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2362.886235] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2362.941730] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2362.950873] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2362.958352] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2362.963864] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2362.969350] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2362.988114] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2362.993356] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2363.000498] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2363.007650] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000930d [ 2363.014798] R10: 00000000002f6e00 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2363.021931] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2363.029070] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2363.037169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2363.042930] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2363.050080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2363.057224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2363.064372] Call Trace: [ 2363.066835] [ 2363.068942] ? __warn+0x85/0x180 [ 2363.072176] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.077072] ? report_bug+0xfc/0x1d0 [ 2363.080666] ? prb_read_valid+0x1b/0x20 [ 2363.084525] ? handle_bug+0x44/0x70 [ 2363.088035] ? exc_invalid_op+0x17/0x60 [ 2363.091873] ? asm_exc_invalid_op+0x1a/0x20 [ 2363.096079] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.100973] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.105852] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2363.110742] ceph_add_cap+0x20b/0x620 [ceph] [ 2363.115069] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2363.119803] ceph_fill_trace+0x143/0xf30 [ceph] [ 2363.124341] ? __mutex_lock+0x97/0xce0 [ 2363.128108] ? __mutex_lock+0xaec/0xce0 [ 2363.131958] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2363.136500] ? lock_downgrade+0xfd/0x250 [ 2363.140462] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2363.144836] ceph_con_process_message+0x74/0x150 [libceph] [ 2363.150331] ? calc_signature+0xdb/0x100 [libceph] [ 2363.155168] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2363.160509] ? process_scheduled_works+0x22a/0x510 [ 2363.165319] ? process_scheduled_works+0x22a/0x510 [ 2363.170107] ceph_con_workfn+0x344/0x750 [libceph] [ 2363.174929] ? process_scheduled_works+0x22a/0x510 [ 2363.179720] process_scheduled_works+0x2a4/0x510 [ 2363.184379] worker_thread+0x188/0x340 [ 2363.188128] ? rescuer_thread+0x310/0x310 [ 2363.192157] kthread+0xf5/0x130 [ 2363.195319] ? kthread_complete_and_exit+0x20/0x20 [ 2363.200133] ret_from_fork+0x31/0x40 [ 2363.203727] ? kthread_complete_and_exit+0x20/0x20 [ 2363.208536] ret_from_fork_asm+0x11/0x20 [ 2363.212505] [ 2363.214690] irq event stamp: 10766525 [ 2363.218363] hardirqs last enabled at (10766535): [] console_unlock+0x106/0x120 [ 2363.227249] hardirqs last disabled at (10766544): [] console_unlock+0xeb/0x120 [ 2363.236047] softirqs last enabled at (10766482): [] __do_softirq+0x2be/0x371 [ 2363.244756] softirqs last disabled at (10766477): [] irq_exit_rcu+0xb0/0xd0 [ 2363.253291] ---[ end trace 0000000000000000 ]--- [ 2363.257937] ------------[ cut here ]------------ [ 2363.262558] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2363.274238] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2363.282933] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2363.338412] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2363.347552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2363.355042] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2363.360536] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2363.366039] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2363.384797] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2363.390018] RAX: 0000000000000000 RBX: ffff8881d11461d0 RCX: 0000000000000000 [ 2363.397145] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2363.404295] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000934b [ 2363.411444] R10: 00000000002f7f90 R11: ffffc9000030b828 R12: ffff8881d1146920 [ 2363.418596] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2363.425745] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2363.433848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2363.439593] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2363.446736] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2363.453887] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2363.461019] Call Trace: [ 2363.463472] [ 2363.465595] ? __warn+0x85/0x180 [ 2363.468846] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.473724] ? report_bug+0xfc/0x1d0 [ 2363.477321] ? prb_read_valid+0x1b/0x20 [ 2363.481180] ? handle_bug+0x44/0x70 [ 2363.484688] ? exc_invalid_op+0x17/0x60 [ 2363.488545] ? asm_exc_invalid_op+0x1a/0x20 [ 2363.492777] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.497654] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.502534] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2363.507440] ceph_add_cap+0x20b/0x620 [ceph] [ 2363.511759] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2363.516482] ? lock_acquire+0xcb/0x2b0 [ 2363.520251] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2363.524703] ? __mutex_lock+0x97/0xce0 [ 2363.528472] ? __mutex_lock+0xaec/0xce0 [ 2363.532331] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2363.536888] ? lock_downgrade+0xfd/0x250 [ 2363.540821] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2363.545217] ceph_con_process_message+0x74/0x150 [libceph] [ 2363.550713] ? calc_signature+0xdb/0x100 [libceph] [ 2363.555533] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2363.560874] ? process_scheduled_works+0x22a/0x510 [ 2363.565664] ? process_scheduled_works+0x22a/0x510 [ 2363.570471] ceph_con_workfn+0x344/0x750 [libceph] [ 2363.575295] ? process_scheduled_works+0x22a/0x510 [ 2363.580102] process_scheduled_works+0x2a4/0x510 [ 2363.584760] worker_thread+0x188/0x340 [ 2363.588510] ? rescuer_thread+0x310/0x310 [ 2363.592539] kthread+0xf5/0x130 [ 2363.595702] ? kthread_complete_and_exit+0x20/0x20 [ 2363.600514] ret_from_fork+0x31/0x40 [ 2363.604109] ? kthread_complete_and_exit+0x20/0x20 [ 2363.608918] ret_from_fork_asm+0x11/0x20 [ 2363.612852] [ 2363.615046] irq event stamp: 10767693 [ 2363.618712] hardirqs last enabled at (10767703): [] console_unlock+0x106/0x120 [ 2363.627596] hardirqs last disabled at (10767712): [] console_unlock+0xeb/0x120 [ 2363.636392] softirqs last enabled at (10767640): [] __do_softirq+0x2be/0x371 [ 2363.645103] softirqs last disabled at (10767731): [] irq_exit_rcu+0xb0/0xd0 [ 2363.653639] ---[ end trace 0000000000000000 ]--- [ 2363.658297] ------------[ cut here ]------------ [ 2363.662913] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2363.674573] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2363.683264] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2363.738758] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2363.747907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2363.755389] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2363.760900] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2363.766384] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2363.785150] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2363.790393] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2363.797544] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2363.804695] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000938a [ 2363.811844] R10: 00000000002f9168 R11: ffffc9000030b828 R12: ffff8881d1144bc8 [ 2363.818977] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2363.826110] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2363.834213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2363.839976] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2363.847109] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2363.854260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2363.861410] Call Trace: [ 2363.863880] [ 2363.865984] ? __warn+0x85/0x180 [ 2363.869218] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.874116] ? report_bug+0xfc/0x1d0 [ 2363.877711] ? prb_read_valid+0x1b/0x20 [ 2363.881569] ? handle_bug+0x44/0x70 [ 2363.885094] ? exc_invalid_op+0x17/0x60 [ 2363.888944] ? asm_exc_invalid_op+0x1a/0x20 [ 2363.893152] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.898028] ? __list_add_valid_or_report+0x7d/0xa0 [ 2363.902907] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2363.907809] ceph_add_cap+0x20b/0x620 [ceph] [ 2363.912114] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2363.916838] ceph_fill_trace+0x143/0xf30 [ceph] [ 2363.921376] ? __mutex_lock+0x97/0xce0 [ 2363.925145] ? __mutex_lock+0xaec/0xce0 [ 2363.929004] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2363.933554] ? lock_downgrade+0xfd/0x250 [ 2363.937517] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2363.941902] ceph_con_process_message+0x74/0x150 [libceph] [ 2363.947401] ? calc_signature+0xdb/0x100 [libceph] [ 2363.952240] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2363.957589] ? process_scheduled_works+0x22a/0x510 [ 2363.962381] ? process_scheduled_works+0x22a/0x510 [ 2363.967190] ceph_con_workfn+0x344/0x750 [libceph] [ 2363.972010] ? process_scheduled_works+0x22a/0x510 [ 2363.976808] process_scheduled_works+0x2a4/0x510 [ 2363.981450] worker_thread+0x188/0x340 [ 2363.985206] ? rescuer_thread+0x310/0x310 [ 2363.989237] kthread+0xf5/0x130 [ 2363.992400] ? kthread_complete_and_exit+0x20/0x20 [ 2363.997230] ret_from_fork+0x31/0x40 [ 2364.000808] ? kthread_complete_and_exit+0x20/0x20 [ 2364.005600] ret_from_fork_asm+0x11/0x20 [ 2364.009567] [ 2364.011753] irq event stamp: 10768767 [ 2364.015435] hardirqs last enabled at (10768777): [] console_unlock+0x106/0x120 [ 2364.024319] hardirqs last disabled at (10768786): [] console_unlock+0xeb/0x120 [ 2364.033115] softirqs last enabled at (10768724): [] __do_softirq+0x2be/0x371 [ 2364.041828] softirqs last disabled at (10768719): [] irq_exit_rcu+0xb0/0xd0 [ 2364.050345] ---[ end trace 0000000000000000 ]--- [ 2364.055127] ------------[ cut here ]------------ [ 2364.059742] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2364.071420] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2364.080116] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2364.135612] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2364.144754] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2364.152251] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2364.157763] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2364.163272] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2364.182020] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2364.187246] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2364.194389] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2364.201539] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000093c8 [ 2364.208690] R10: 00000000002fa1f0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2364.215840] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2364.222972] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2364.231059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2364.236810] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2364.243938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2364.251069] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2364.258202] Call Trace: [ 2364.260663] [ 2364.262778] ? __warn+0x85/0x180 [ 2364.266029] ? __list_add_valid_or_report+0x7d/0xa0 [ 2364.270908] ? report_bug+0xfc/0x1d0 [ 2364.274488] ? prb_read_valid+0x1b/0x20 [ 2364.278345] ? handle_bug+0x44/0x70 [ 2364.281854] ? exc_invalid_op+0x17/0x60 [ 2364.285693] ? asm_exc_invalid_op+0x1a/0x20 [ 2364.289903] ? __list_add_valid_or_report+0x7d/0xa0 [ 2364.294806] ? __list_add_valid_or_report+0x7d/0xa0 [ 2364.299682] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2364.304590] ceph_add_cap+0x20b/0x620 [ceph] [ 2364.308893] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2364.313632] ? lock_acquire+0xcb/0x2b0 [ 2364.317402] ceph_fill_trace+0x143/0xf30 [ceph] [ 2364.321939] ? __mutex_lock+0x97/0xce0 [ 2364.325691] ? __mutex_lock+0xaec/0xce0 [ 2364.329569] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2364.334117] ? lock_downgrade+0xfd/0x250 [ 2364.338064] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2364.342459] ceph_con_process_message+0x74/0x150 [libceph] [ 2364.347955] ? calc_signature+0xdb/0x100 [libceph] [ 2364.352760] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2364.358111] ? process_scheduled_works+0x22a/0x510 [ 2364.362909] ? process_scheduled_works+0x22a/0x510 [ 2364.367698] ceph_con_workfn+0x344/0x750 [libceph] [ 2364.372519] ? process_scheduled_works+0x22a/0x510 [ 2364.377328] process_scheduled_works+0x2a4/0x510 [ 2364.381970] worker_thread+0x188/0x340 [ 2364.385718] ? rescuer_thread+0x310/0x310 [ 2364.389748] kthread+0xf5/0x130 [ 2364.392914] ? kthread_complete_and_exit+0x20/0x20 [ 2364.397704] ret_from_fork+0x31/0x40 [ 2364.401299] ? kthread_complete_and_exit+0x20/0x20 [ 2364.406110] ret_from_fork_asm+0x11/0x20 [ 2364.410060] [ 2364.412245] irq event stamp: 10769905 [ 2364.415922] hardirqs last enabled at (10769921): [] console_unlock+0x106/0x120 [ 2364.424807] hardirqs last disabled at (10769930): [] console_unlock+0xeb/0x120 [ 2364.433583] softirqs last enabled at (10769918): [] __do_softirq+0x2be/0x371 [ 2364.442293] softirqs last disabled at (10769913): [] irq_exit_rcu+0xb0/0xd0 [ 2364.450831] ---[ end trace 0000000000000000 ]--- [ 2364.455500] ------------[ cut here ]------------ [ 2364.460127] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2364.471767] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2364.480463] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2364.535940] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2364.545067] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2364.552546] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2364.558040] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2364.563524] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2364.582273] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2364.587497] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2364.594631] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2364.601765] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009407 [ 2364.608897] R10: 00000000002fb4a8 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2364.616031] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2364.623162] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2364.631248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2364.636995] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2364.644128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2364.651261] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2364.658393] Call Trace: [ 2364.660845] [ 2364.662953] ? __warn+0x85/0x180 [ 2364.666186] ? __list_add_valid_or_report+0x7d/0xa0 [ 2364.671065] ? report_bug+0xfc/0x1d0 [ 2364.674642] ? prb_read_valid+0x1b/0x20 [ 2364.678485] ? handle_bug+0x44/0x70 [ 2364.681974] ? exc_invalid_op+0x17/0x60 [ 2364.685816] ? asm_exc_invalid_op+0x1a/0x20 [ 2364.690006] ? __list_add_valid_or_report+0x7d/0xa0 [ 2364.694882] ? __list_add_valid_or_report+0x7d/0xa0 [ 2364.699759] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2364.704650] ceph_add_cap+0x20b/0x620 [ceph] [ 2364.708935] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2364.713650] ? lock_acquire+0xcb/0x2b0 [ 2364.717401] ceph_fill_trace+0x143/0xf30 [ceph] [ 2364.721940] ? __mutex_lock+0x97/0xce0 [ 2364.725690] ? __mutex_lock+0xaec/0xce0 [ 2364.729533] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2364.734080] ? lock_downgrade+0xfd/0x250 [ 2364.738013] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2364.742391] ceph_con_process_message+0x74/0x150 [libceph] [ 2364.747887] ? calc_signature+0xdb/0x100 [libceph] [ 2364.752692] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2364.758023] ? process_scheduled_works+0x22a/0x510 [ 2364.762822] ? process_scheduled_works+0x22a/0x510 [ 2364.767612] ceph_con_workfn+0x344/0x750 [libceph] [ 2364.772416] ? process_scheduled_works+0x22a/0x510 [ 2364.777206] process_scheduled_works+0x2a4/0x510 [ 2364.781830] worker_thread+0x188/0x340 [ 2364.785581] ? rescuer_thread+0x310/0x310 [ 2364.789591] kthread+0xf5/0x130 [ 2364.792740] ? kthread_complete_and_exit+0x20/0x20 [ 2364.797531] ret_from_fork+0x31/0x40 [ 2364.801109] ? kthread_complete_and_exit+0x20/0x20 [ 2364.805902] ret_from_fork_asm+0x11/0x20 [ 2364.809835] [ 2364.812020] irq event stamp: 10771073 [ 2364.815688] hardirqs last enabled at (10771083): [] console_unlock+0x106/0x120 [ 2364.824553] hardirqs last disabled at (10771092): [] console_unlock+0xeb/0x120 [ 2364.833333] softirqs last enabled at (10771020): [] __do_softirq+0x2be/0x371 [ 2364.842025] softirqs last disabled at (10771015): [] irq_exit_rcu+0xb0/0xd0 [ 2364.850547] ---[ end trace 0000000000000000 ]--- [ 2364.855166] ------------[ cut here ]------------ [ 2364.859786] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2364.871435] WARNING: CPU: 6 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2364.880397] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2364.935919] CPU: 6 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2364.944876] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2364.952355] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2364.957843] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2364.976594] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2364.981824] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2364.988955] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2364.996089] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009446 [ 2365.003224] R10: 00000000002fc560 R11: ffffc90001fd7950 R12: ffff888158a68750 [ 2365.010354] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2365.017487] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2365.025572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2365.031320] CR2: 00007f9bd22d7020 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2365.038455] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2365.045586] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2365.052718] Call Trace: [ 2365.055172] [ 2365.057278] ? __warn+0x85/0x180 [ 2365.060510] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.065390] ? report_bug+0xfc/0x1d0 [ 2365.068968] ? prb_read_valid+0x1b/0x20 [ 2365.072813] ? handle_bug+0x44/0x70 [ 2365.076308] ? exc_invalid_op+0x17/0x60 [ 2365.080150] ? asm_exc_invalid_op+0x1a/0x20 [ 2365.084344] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.089224] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2365.094128] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2365.099378] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2365.104194] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2365.108918] ? __d_lookup+0xd6/0x200 [ 2365.112505] lookup_fast+0x6d/0xf0 [ 2365.115909] walk_component+0x26/0x160 [ 2365.119663] link_path_walk+0x24d/0x380 [ 2365.123501] ? path_lookupat.isra.40+0x22/0x120 [ 2365.128039] path_lookupat.isra.40+0x6b/0x120 [ 2365.132404] filename_lookup+0xd4/0x190 [ 2365.136249] ? lock_acquire+0xcb/0x2b0 [ 2365.140001] ? __might_fault+0x45/0x80 [ 2365.143758] ? find_held_lock+0x2d/0x90 [ 2365.147604] ? __might_fault+0x45/0x80 [ 2365.151364] vfs_statx+0x89/0x150 [ 2365.154691] vfs_fstatat+0x5a/0xc0 [ 2365.158102] __do_sys_newstat+0x30/0x70 [ 2365.161947] ? lock_release+0xc5/0x260 [ 2365.165703] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2365.170589] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2365.175736] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2365.180878] do_syscall_64+0x38/0x80 [ 2365.184464] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2365.189515] RIP: 0033:0x7f48f0eeca79 [ 2365.193097] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2365.211842] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2365.219408] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2365.226541] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2365.233673] RBP: 00007ffc53d60520 R08: 0000000000000000 R09: 00000000020767c0 [ 2365.240804] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2365.247937] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020767c0 [ 2365.255084] [ 2365.257273] irq event stamp: 13010743 [ 2365.260938] hardirqs last enabled at (13010753): [] console_unlock+0x106/0x120 [ 2365.269805] hardirqs last disabled at (13010762): [] console_unlock+0xeb/0x120 [ 2365.278584] softirqs last enabled at (13010700): [] __do_softirq+0x2be/0x371 [ 2365.287277] softirqs last disabled at (13010695): [] irq_exit_rcu+0xb0/0xd0 [ 2365.295795] ---[ end trace 0000000000000000 ]--- [ 2365.300417] ------------[ cut here ]------------ [ 2365.305048] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2365.316700] WARNING: CPU: 2 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2365.325653] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2365.381175] CPU: 2 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2365.390126] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2365.397606] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2365.403091] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2365.421838] RSP: 0018:ffffc90001fc7b78 EFLAGS: 00010286 [ 2365.427065] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2365.434196] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2365.441329] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000948a [ 2365.448464] R10: 00000000002fd918 R11: ffffc90001fc7a08 R12: ffff8881d106df58 [ 2365.455595] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2365.462729] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2365.470815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2365.476560] CR2: 00007fb38c506000 CR3: 0000000113500006 CR4: 00000000003706e0 [ 2365.483693] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2365.490827] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2365.497960] Call Trace: [ 2365.500412] [ 2365.502519] ? __warn+0x85/0x180 [ 2365.505752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.510631] ? report_bug+0xfc/0x1d0 [ 2365.514211] ? prb_read_valid+0x1b/0x20 [ 2365.518053] ? handle_bug+0x44/0x70 [ 2365.521551] ? exc_invalid_op+0x17/0x60 [ 2365.525391] ? asm_exc_invalid_op+0x1a/0x20 [ 2365.529586] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.534465] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.539343] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2365.544245] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2365.549498] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2365.554314] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2365.559038] ? __d_lookup+0xd6/0x200 [ 2365.562627] lookup_fast+0x6d/0xf0 [ 2365.566040] walk_component+0x26/0x160 [ 2365.569794] link_path_walk+0x24d/0x380 [ 2365.573638] ? path_openat+0x80/0x9d0 [ 2365.577311] path_openat+0x8e/0x9d0 [ 2365.580816] do_filp_open+0xb8/0x120 [ 2365.584399] ? lock_release+0xc5/0x260 [ 2365.588159] ? _raw_spin_unlock+0x23/0x40 [ 2365.592176] ? alloc_fd+0x13f/0x200 [ 2365.595674] do_sys_openat2+0x249/0x290 [ 2365.599520] do_sys_open+0x4f/0x80 [ 2365.602935] do_syscall_64+0x38/0x80 [ 2365.606521] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2365.611572] RIP: 0033:0x7f48f0ef2805 [ 2365.615153] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2365.633899] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2365.641466] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2365.648597] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2365.655730] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2365.662864] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2365.669996] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2365.677142] [ 2365.679337] irq event stamp: 12959875 [ 2365.683004] hardirqs last enabled at (12959885): [] console_unlock+0x106/0x120 [ 2365.691870] hardirqs last disabled at (12959894): [] console_unlock+0xeb/0x120 [ 2365.700650] softirqs last enabled at (12959812): [] __do_softirq+0x2be/0x371 [ 2365.709345] softirqs last disabled at (12959913): [] irq_exit_rcu+0xb0/0xd0 [ 2365.717871] ---[ end trace 0000000000000000 ]--- [ 2365.722491] ------------[ cut here ]------------ [ 2365.727106] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2365.738756] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2365.747451] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2365.802934] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2365.812062] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2365.819541] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2365.825044] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2365.830528] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2365.849275] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2365.854501] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2365.861635] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2365.868768] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000094c9 [ 2365.875900] R10: 00000000002fea30 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2365.883034] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2365.890165] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2365.898252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2365.903997] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2365.911131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2365.918265] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2365.925399] Call Trace: [ 2365.927849] [ 2365.929956] ? __warn+0x85/0x180 [ 2365.933190] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.938069] ? report_bug+0xfc/0x1d0 [ 2365.941646] ? prb_read_valid+0x1b/0x20 [ 2365.945487] ? handle_bug+0x44/0x70 [ 2365.948981] ? exc_invalid_op+0x17/0x60 [ 2365.952819] ? asm_exc_invalid_op+0x1a/0x20 [ 2365.957009] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.961885] ? __list_add_valid_or_report+0x7d/0xa0 [ 2365.966766] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2365.971663] ceph_add_cap+0x20b/0x620 [ceph] [ 2365.975948] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2365.980672] ? lock_acquire+0xcb/0x2b0 [ 2365.984431] ceph_fill_trace+0x143/0xf30 [ceph] [ 2365.988968] ? __mutex_lock+0x97/0xce0 [ 2365.992719] ? __mutex_lock+0xaec/0xce0 [ 2365.996562] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2366.001109] ? lock_downgrade+0xfd/0x250 [ 2366.005040] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2366.009422] ceph_con_process_message+0x74/0x150 [libceph] [ 2366.014915] ? calc_signature+0xdb/0x100 [libceph] [ 2366.019720] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2366.025052] ? process_scheduled_works+0x22a/0x510 [ 2366.029843] ? process_scheduled_works+0x22a/0x510 [ 2366.034634] ceph_con_workfn+0x344/0x750 [libceph] [ 2366.039438] ? process_scheduled_works+0x22a/0x510 [ 2366.044226] process_scheduled_works+0x2a4/0x510 [ 2366.048851] worker_thread+0x188/0x340 [ 2366.052602] ? rescuer_thread+0x310/0x310 [ 2366.056612] kthread+0xf5/0x130 [ 2366.059761] ? kthread_complete_and_exit+0x20/0x20 [ 2366.064553] ret_from_fork+0x31/0x40 [ 2366.068131] ? kthread_complete_and_exit+0x20/0x20 [ 2366.072923] ret_from_fork_asm+0x11/0x20 [ 2366.076856] [ 2366.079042] irq event stamp: 10774049 [ 2366.082707] hardirqs last enabled at (10774059): [] console_unlock+0x106/0x120 [ 2366.091574] hardirqs last disabled at (10774068): [] console_unlock+0xeb/0x120 [ 2366.100354] softirqs last enabled at (10773996): [] __do_softirq+0x2be/0x371 [ 2366.109046] softirqs last disabled at (10773991): [] irq_exit_rcu+0xb0/0xd0 [ 2366.117566] ---[ end trace 0000000000000000 ]--- [ 2366.122340] ------------[ cut here ]------------ [ 2366.126964] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2366.138610] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2366.147565] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2366.203055] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2366.212010] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2366.219490] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2366.224976] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2366.243723] RSP: 0018:ffffc90001fcfb78 EFLAGS: 00010286 [ 2366.248947] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2366.256078] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2366.263213] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009508 [ 2366.270348] R10: 00000000002ffb48 R11: ffffc90001fcfa08 R12: ffff8881c5350750 [ 2366.277479] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2366.284610] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2366.292697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2366.298444] CR2: 00007f4f3ea07000 CR3: 000000011aeae004 CR4: 00000000003706f0 [ 2366.305577] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2366.312709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2366.319843] Call Trace: [ 2366.322296] [ 2366.324403] ? __warn+0x85/0x180 [ 2366.327633] ? __list_add_valid_or_report+0x7d/0xa0 [ 2366.332515] ? report_bug+0xfc/0x1d0 [ 2366.336095] ? prb_read_valid+0x1b/0x20 [ 2366.339936] ? handle_bug+0x44/0x70 [ 2366.343433] ? exc_invalid_op+0x17/0x60 [ 2366.347276] ? asm_exc_invalid_op+0x1a/0x20 [ 2366.351474] ? __list_add_valid_or_report+0x7d/0xa0 [ 2366.356358] ? __list_add_valid_or_report+0x7d/0xa0 [ 2366.361245] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2366.366147] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2366.371387] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2366.376207] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2366.380929] ? __d_lookup+0xd6/0x200 [ 2366.384509] lookup_fast+0x6d/0xf0 [ 2366.387921] walk_component+0x26/0x160 [ 2366.391676] link_path_walk+0x24d/0x380 [ 2366.395520] ? path_openat+0x80/0x9d0 [ 2366.399195] path_openat+0x8e/0x9d0 [ 2366.402695] do_filp_open+0xb8/0x120 [ 2366.406285] ? lock_release+0xc5/0x260 [ 2366.410041] ? _raw_spin_unlock+0x23/0x40 [ 2366.414062] ? alloc_fd+0x13f/0x200 [ 2366.417564] do_sys_openat2+0x249/0x290 [ 2366.421413] do_sys_open+0x4f/0x80 [ 2366.424827] do_syscall_64+0x38/0x80 [ 2366.428414] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2366.433471] RIP: 0033:0x7f48f0ef2805 [ 2366.437054] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2366.455797] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2366.463367] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2366.470497] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2366.477631] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2366.484762] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2366.491895] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2366.499044] [ 2366.501240] irq event stamp: 12925875 [ 2366.504902] hardirqs last enabled at (12925885): [] console_unlock+0x106/0x120 [ 2366.513772] hardirqs last disabled at (12925894): [] console_unlock+0xeb/0x120 [ 2366.522550] softirqs last enabled at (12925862): [] __do_softirq+0x2be/0x371 [ 2366.531245] softirqs last disabled at (12925913): [] irq_exit_rcu+0xb0/0xd0 [ 2366.539770] ---[ end trace 0000000000000000 ]--- [ 2366.544392] ------------[ cut here ]------------ [ 2366.549007] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2366.560671] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2366.569369] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2366.624861] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2366.633990] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2366.641467] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2366.646979] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2366.652463] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2366.671229] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2366.676471] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2366.683613] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2366.690763] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009547 [ 2366.697906] R10: 0000000000300c40 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2366.705038] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2366.712171] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2366.720276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2366.726038] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2366.733170] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2366.740319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2366.747469] Call Trace: [ 2366.749932] [ 2366.752039] ? __warn+0x85/0x180 [ 2366.755272] ? __list_add_valid_or_report+0x7d/0xa0 [ 2366.760167] ? report_bug+0xfc/0x1d0 [ 2366.763765] ? prb_read_valid+0x1b/0x20 [ 2366.767623] ? handle_bug+0x44/0x70 [ 2366.771146] ? exc_invalid_op+0x17/0x60 [ 2366.774998] ? asm_exc_invalid_op+0x1a/0x20 [ 2366.779204] ? __list_add_valid_or_report+0x7d/0xa0 [ 2366.784081] ? __list_add_valid_or_report+0x7d/0xa0 [ 2366.788958] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2366.793862] ceph_add_cap+0x20b/0x620 [ceph] [ 2366.798169] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2366.802892] ? lock_acquire+0xcb/0x2b0 [ 2366.806660] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2366.811113] ? __mutex_lock+0x97/0xce0 [ 2366.814882] ? __mutex_lock+0xaec/0xce0 [ 2366.818721] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2366.823280] ? lock_downgrade+0xfd/0x250 [ 2366.827245] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2366.831641] ceph_con_process_message+0x74/0x150 [libceph] [ 2366.837150] ? calc_signature+0xdb/0x100 [libceph] [ 2366.841976] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2366.847314] ? process_scheduled_works+0x22a/0x510 [ 2366.852108] ? process_scheduled_works+0x22a/0x510 [ 2366.856897] ceph_con_workfn+0x344/0x750 [libceph] [ 2366.861701] ? process_scheduled_works+0x22a/0x510 [ 2366.866510] process_scheduled_works+0x2a4/0x510 [ 2366.871169] worker_thread+0x188/0x340 [ 2366.874919] ? rescuer_thread+0x310/0x310 [ 2366.878929] kthread+0xf5/0x130 [ 2366.882076] ? kthread_complete_and_exit+0x20/0x20 [ 2366.886870] ret_from_fork+0x31/0x40 [ 2366.890450] ? kthread_complete_and_exit+0x20/0x20 [ 2366.895258] ret_from_fork_asm+0x11/0x20 [ 2366.899224] [ 2366.901411] irq event stamp: 10776979 [ 2366.905093] hardirqs last enabled at (10776989): [] console_unlock+0x106/0x120 [ 2366.913961] hardirqs last disabled at (10776998): [] console_unlock+0xeb/0x120 [ 2366.922740] softirqs last enabled at (10776946): [] __do_softirq+0x2be/0x371 [ 2366.931449] softirqs last disabled at (10776941): [] irq_exit_rcu+0xb0/0xd0 [ 2366.939988] ---[ end trace 0000000000000000 ]--- [ 2366.944631] ------------[ cut here ]------------ [ 2366.949244] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2366.960907] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2366.969603] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2367.025097] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2367.034221] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2367.041709] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2367.047220] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2367.052722] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2367.071480] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2367.076723] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2367.083865] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2367.090999] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009586 [ 2367.098143] R10: 0000000000301f00 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2367.105289] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2367.112438] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2367.120535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2367.126295] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2367.133439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2367.140580] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2367.147721] Call Trace: [ 2367.150183] [ 2367.152299] ? __warn+0x85/0x180 [ 2367.155549] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.160446] ? report_bug+0xfc/0x1d0 [ 2367.164041] ? prb_read_valid+0x1b/0x20 [ 2367.167882] ? handle_bug+0x44/0x70 [ 2367.171373] ? exc_invalid_op+0x17/0x60 [ 2367.175223] ? asm_exc_invalid_op+0x1a/0x20 [ 2367.179446] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.184324] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.189218] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2367.194127] ceph_add_cap+0x20b/0x620 [ceph] [ 2367.198428] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2367.203175] ceph_fill_trace+0x143/0xf30 [ceph] [ 2367.207714] ? __mutex_lock+0x97/0xce0 [ 2367.211483] ? __mutex_lock+0xaec/0xce0 [ 2367.215340] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2367.219902] ? lock_downgrade+0xfd/0x250 [ 2367.223854] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2367.228233] ceph_con_process_message+0x74/0x150 [libceph] [ 2367.233731] ? calc_signature+0xdb/0x100 [libceph] [ 2367.238569] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2367.243894] ? process_scheduled_works+0x22a/0x510 [ 2367.248684] ? process_scheduled_works+0x22a/0x510 [ 2367.253490] ceph_con_workfn+0x344/0x750 [libceph] [ 2367.258313] ? process_scheduled_works+0x22a/0x510 [ 2367.263119] process_scheduled_works+0x2a4/0x510 [ 2367.267778] worker_thread+0x188/0x340 [ 2367.271527] ? rescuer_thread+0x310/0x310 [ 2367.275557] kthread+0xf5/0x130 [ 2367.278720] ? kthread_complete_and_exit+0x20/0x20 [ 2367.283533] ret_from_fork+0x31/0x40 [ 2367.287144] ? kthread_complete_and_exit+0x20/0x20 [ 2367.291955] ret_from_fork_asm+0x11/0x20 [ 2367.295890] [ 2367.298083] irq event stamp: 10778033 [ 2367.301749] hardirqs last enabled at (10778043): [] console_unlock+0x106/0x120 [ 2367.310632] hardirqs last disabled at (10778052): [] console_unlock+0xeb/0x120 [ 2367.319428] softirqs last enabled at (10777990): [] __do_softirq+0x2be/0x371 [ 2367.328145] softirqs last disabled at (10777985): [] irq_exit_rcu+0xb0/0xd0 [ 2367.336685] ---[ end trace 0000000000000000 ]--- [ 2367.341354] ------------[ cut here ]------------ [ 2367.345977] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2367.357619] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2367.366309] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2367.421787] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2367.430911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2367.438390] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2367.443884] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2367.449372] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2367.468117] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2367.473343] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2367.480477] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2367.487612] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000095c4 [ 2367.494742] R10: 0000000000302fb0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2367.501875] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2367.509009] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2367.517092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2367.522846] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2367.529981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2367.537114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2367.544248] Call Trace: [ 2367.546700] [ 2367.548807] ? __warn+0x85/0x180 [ 2367.552041] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.556919] ? report_bug+0xfc/0x1d0 [ 2367.560496] ? prb_read_valid+0x1b/0x20 [ 2367.564338] ? handle_bug+0x44/0x70 [ 2367.567830] ? exc_invalid_op+0x17/0x60 [ 2367.571670] ? asm_exc_invalid_op+0x1a/0x20 [ 2367.575859] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.580735] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.585612] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2367.590504] ceph_add_cap+0x20b/0x620 [ceph] [ 2367.594788] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2367.599504] ? lock_acquire+0xcb/0x2b0 [ 2367.603263] ceph_fill_trace+0x143/0xf30 [ceph] [ 2367.607801] ? __mutex_lock+0x97/0xce0 [ 2367.611552] ? __mutex_lock+0xaec/0xce0 [ 2367.615395] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2367.619944] ? lock_downgrade+0xfd/0x250 [ 2367.623875] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2367.628253] ceph_con_process_message+0x74/0x150 [libceph] [ 2367.633749] ? calc_signature+0xdb/0x100 [libceph] [ 2367.638563] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2367.643893] ? process_scheduled_works+0x22a/0x510 [ 2367.648685] ? process_scheduled_works+0x22a/0x510 [ 2367.653476] ceph_con_workfn+0x344/0x750 [libceph] [ 2367.658279] ? process_scheduled_works+0x22a/0x510 [ 2367.663068] process_scheduled_works+0x2a4/0x510 [ 2367.667695] worker_thread+0x188/0x340 [ 2367.671450] ? rescuer_thread+0x310/0x310 [ 2367.675463] kthread+0xf5/0x130 [ 2367.678609] ? kthread_complete_and_exit+0x20/0x20 [ 2367.683401] ret_from_fork+0x31/0x40 [ 2367.686981] ? kthread_complete_and_exit+0x20/0x20 [ 2367.691774] ret_from_fork_asm+0x11/0x20 [ 2367.695706] [ 2367.697892] irq event stamp: 10779145 [ 2367.701557] hardirqs last enabled at (10779155): [] console_unlock+0x106/0x120 [ 2367.710424] hardirqs last disabled at (10779164): [] console_unlock+0xeb/0x120 [ 2367.719204] softirqs last enabled at (10779092): [] __do_softirq+0x2be/0x371 [ 2367.727896] softirqs last disabled at (10779087): [] irq_exit_rcu+0xb0/0xd0 [ 2367.736415] ---[ end trace 0000000000000000 ]--- [ 2367.741058] ------------[ cut here ]------------ [ 2367.745686] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2367.757339] WARNING: CPU: 3 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2367.766292] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2367.821814] CPU: 3 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2367.830766] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2367.838245] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2367.843731] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2367.862477] RSP: 0018:ffffc90001ff7b78 EFLAGS: 00010286 [ 2367.867702] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2367.874836] RDX: ffff88885fce9c00 RSI: ffff88885fcdd880 RDI: ffff88885fcdd880 [ 2367.881968] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009603 [ 2367.889100] R10: 0000000000304238 R11: ffffc90001ff7a08 R12: ffff8881c5350750 [ 2367.896234] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2367.903367] FS: 00007f48f1832740(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000 [ 2367.911456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2367.917200] CR2: 00007fb38c572008 CR3: 0000000120cc8004 CR4: 00000000003706e0 [ 2367.924335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2367.931464] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2367.938597] Call Trace: [ 2367.941050] [ 2367.943156] ? __warn+0x85/0x180 [ 2367.946390] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.951271] ? report_bug+0xfc/0x1d0 [ 2367.954858] ? prb_read_valid+0x1b/0x20 [ 2367.958701] ? handle_bug+0x44/0x70 [ 2367.962197] ? exc_invalid_op+0x17/0x60 [ 2367.966037] ? asm_exc_invalid_op+0x1a/0x20 [ 2367.970232] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.975119] ? __list_add_valid_or_report+0x7d/0xa0 [ 2367.979999] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2367.984900] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2367.990160] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2367.994970] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2367.999693] ? __d_lookup+0xd6/0x200 [ 2368.003273] lookup_fast+0x6d/0xf0 [ 2368.006687] walk_component+0x26/0x160 [ 2368.010442] link_path_walk+0x24d/0x380 [ 2368.014286] ? path_openat+0x80/0x9d0 [ 2368.017956] path_openat+0x8e/0x9d0 [ 2368.021460] do_filp_open+0xb8/0x120 [ 2368.025049] ? lock_release+0xc5/0x260 [ 2368.028807] ? _raw_spin_unlock+0x23/0x40 [ 2368.032825] ? alloc_fd+0x13f/0x200 [ 2368.036321] do_sys_openat2+0x249/0x290 [ 2368.040169] do_sys_open+0x4f/0x80 [ 2368.043582] do_syscall_64+0x38/0x80 [ 2368.047160] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2368.052212] RIP: 0033:0x7f48f0eed2f2 [ 2368.055791] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2368.074537] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2368.082102] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2368.089236] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2368.096373] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2368.103511] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2368.110642] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2368.117789] [ 2368.119976] irq event stamp: 12970679 [ 2368.123643] hardirqs last enabled at (12970689): [] console_unlock+0x106/0x120 [ 2368.132510] hardirqs last disabled at (12970698): [] console_unlock+0xeb/0x120 [ 2368.141289] softirqs last enabled at (12970430): [] __do_softirq+0x2be/0x371 [ 2368.149982] softirqs last disabled at (12970425): [] irq_exit_rcu+0xb0/0xd0 [ 2368.158501] ---[ end trace 0000000000000000 ]--- [ 2368.163125] ------------[ cut here ]------------ [ 2368.167746] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2368.179392] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2368.188089] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2368.243564] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2368.252693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2368.260170] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2368.265665] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2368.271150] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2368.289896] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2368.295122] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2368.302256] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2368.309391] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009642 [ 2368.316523] R10: 0000000000305350 R11: ffffc9000030b828 R12: ffff8881d1144bc8 [ 2368.323654] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2368.330789] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2368.338873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2368.344619] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2368.351753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2368.358884] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2368.366019] Call Trace: [ 2368.368469] [ 2368.370576] ? __warn+0x85/0x180 [ 2368.373808] ? __list_add_valid_or_report+0x7d/0xa0 [ 2368.378689] ? report_bug+0xfc/0x1d0 [ 2368.382271] ? prb_read_valid+0x1b/0x20 [ 2368.386109] ? handle_bug+0x44/0x70 [ 2368.389602] ? exc_invalid_op+0x17/0x60 [ 2368.393439] ? asm_exc_invalid_op+0x1a/0x20 [ 2368.397632] ? __list_add_valid_or_report+0x7d/0xa0 [ 2368.402516] ? __list_add_valid_or_report+0x7d/0xa0 [ 2368.407394] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2368.412283] ceph_add_cap+0x20b/0x620 [ceph] [ 2368.416569] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2368.421292] ? lock_acquire+0xcb/0x2b0 [ 2368.425054] ceph_fill_trace+0x143/0xf30 [ceph] [ 2368.429600] ? __mutex_lock+0x97/0xce0 [ 2368.433349] ? __mutex_lock+0xaec/0xce0 [ 2368.437190] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2368.441732] ? lock_downgrade+0xfd/0x250 [ 2368.445665] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2368.450042] ceph_con_process_message+0x74/0x150 [libceph] [ 2368.455539] ? calc_signature+0xdb/0x100 [libceph] [ 2368.460350] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2368.465682] ? process_scheduled_works+0x22a/0x510 [ 2368.470482] ? process_scheduled_works+0x22a/0x510 [ 2368.475270] ceph_con_workfn+0x344/0x750 [libceph] [ 2368.480075] ? process_scheduled_works+0x22a/0x510 [ 2368.484866] process_scheduled_works+0x2a4/0x510 [ 2368.489491] worker_thread+0x188/0x340 [ 2368.493247] ? rescuer_thread+0x310/0x310 [ 2368.497261] kthread+0xf5/0x130 [ 2368.500406] ? kthread_complete_and_exit+0x20/0x20 [ 2368.505200] ret_from_fork+0x31/0x40 [ 2368.508780] ? kthread_complete_and_exit+0x20/0x20 [ 2368.513571] ret_from_fork_asm+0x11/0x20 [ 2368.517505] [ 2368.519698] irq event stamp: 10781237 [ 2368.523363] hardirqs last enabled at (10781247): [] console_unlock+0x106/0x120 [ 2368.532229] hardirqs last disabled at (10781256): [] console_unlock+0xeb/0x120 [ 2368.541011] softirqs last enabled at (10781204): [] __do_softirq+0x2be/0x371 [ 2368.549703] softirqs last disabled at (10781199): [] irq_exit_rcu+0xb0/0xd0 [ 2368.558222] ---[ end trace 0000000000000000 ]--- [ 2368.562843] ------------[ cut here ]------------ [ 2368.567473] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2368.579117] WARNING: CPU: 0 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2368.588072] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2368.643552] CPU: 0 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2368.652510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2368.659988] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2368.665475] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2368.684222] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [ 2368.689447] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [ 2368.696580] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2368.703714] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009681 [ 2368.710850] R10: 0000000000306468 R11: ffffc90001f7fa70 R12: ffff8881e4f8df58 [ 2368.717980] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2368.725112] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2368.733198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2368.738942] CR2: 00007f4f3ea07000 CR3: 0000000113532002 CR4: 00000000003706f0 [ 2368.746078] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2368.753210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2368.760342] Call Trace: [ 2368.762796] [ 2368.764903] ? __warn+0x85/0x180 [ 2368.768135] ? __list_add_valid_or_report+0x7d/0xa0 [ 2368.773019] ? report_bug+0xfc/0x1d0 [ 2368.776601] ? prb_read_valid+0x1b/0x20 [ 2368.780446] ? handle_bug+0x44/0x70 [ 2368.783941] ? exc_invalid_op+0x17/0x60 [ 2368.787784] ? asm_exc_invalid_op+0x1a/0x20 [ 2368.791984] ? __list_add_valid_or_report+0x7d/0xa0 [ 2368.796871] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2368.801772] try_get_cap_refs+0x159/0x880 [ceph] [ 2368.806410] ? _raw_spin_unlock+0x23/0x40 [ 2368.810433] __ceph_get_caps+0xf8/0x760 [ceph] [ 2368.814903] ? __vfs_getxattr+0x8d/0xb0 [ 2368.818753] ? dentry_needs_remove_privs+0x31/0x50 [ 2368.823551] ? __file_remove_privs+0x89/0x110 [ 2368.827909] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2368.832635] ? lock_release+0xc5/0x260 [ 2368.836401] ? ceph_write_iter+0x260/0x770 [ceph] [ 2368.841127] ceph_write_iter+0x260/0x770 [ceph] [ 2368.845684] ? lock_is_held_type+0xaa/0x120 [ 2368.849878] vfs_write+0x324/0x4f0 [ 2368.853296] ksys_pwrite64+0x65/0xa0 [ 2368.856876] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2368.862024] do_syscall_64+0x38/0x80 [ 2368.865602] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2368.870652] RIP: 0033:0x7f48f0eeb93b [ 2368.874235] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2368.892979] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2368.900548] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2368.907700] RDX: 0000000000000001 RSI: 000000000207bcc0 RDI: 0000000000000006 [ 2368.914838] RBP: 0000000000002a1a R08: 0000000000000001 R09: 0000000002074010 [ 2368.921969] R10: 00000000000005ff R11: 0000000000000246 R12: 00007f48f183d000 [ 2368.929103] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207bcc0 [ 2368.936248] [ 2368.938437] irq event stamp: 12964299 [ 2368.942101] hardirqs last enabled at (12964309): [] console_unlock+0x106/0x120 [ 2368.950973] hardirqs last disabled at (12964324): [] console_unlock+0xeb/0x120 [ 2368.959755] softirqs last enabled at (12964322): [] __do_softirq+0x2be/0x371 [ 2368.968455] softirqs last disabled at (12964343): [] irq_exit_rcu+0xb0/0xd0 [ 2368.976977] ---[ end trace 0000000000000000 ]--- [ 2368.981599] ------------[ cut here ]------------ [ 2368.986217] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2368.997869] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2369.006825] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2369.062356] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2369.071309] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2369.078788] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2369.084273] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2369.103022] RSP: 0018:ffffc90001fefbe0 EFLAGS: 00010286 [ 2369.108246] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [ 2369.115380] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2369.122512] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000096be [ 2369.129646] R10: 00000000003074e0 R11: ffffc90001fefa70 R12: ffff8881c3b77cb0 [ 2369.136778] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2369.143910] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2369.151997] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2369.157742] CR2: 00007f9bd2629000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2369.164877] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2369.172009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2369.179141] Call Trace: [ 2369.181595] [ 2369.183702] ? __warn+0x85/0x180 [ 2369.186935] ? __list_add_valid_or_report+0x7d/0xa0 [ 2369.191815] ? report_bug+0xfc/0x1d0 [ 2369.195402] ? prb_read_valid+0x1b/0x20 [ 2369.199245] ? handle_bug+0x44/0x70 [ 2369.202742] ? exc_invalid_op+0x17/0x60 [ 2369.206580] ? asm_exc_invalid_op+0x1a/0x20 [ 2369.210777] ? __list_add_valid_or_report+0x7d/0xa0 [ 2369.215657] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2369.220555] try_get_cap_refs+0x159/0x880 [ceph] [ 2369.225190] ? _raw_spin_unlock+0x23/0x40 [ 2369.229210] __ceph_get_caps+0xf8/0x760 [ceph] [ 2369.233675] ? __vfs_getxattr+0x8d/0xb0 [ 2369.237517] ? dentry_needs_remove_privs+0x31/0x50 [ 2369.242315] ? __file_remove_privs+0x89/0x110 [ 2369.246674] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2369.251399] ? lock_release+0xc5/0x260 [ 2369.255154] ? ceph_write_iter+0x260/0x770 [ceph] [ 2369.259878] ceph_write_iter+0x260/0x770 [ceph] [ 2369.264433] ? lock_is_held_type+0xaa/0x120 [ 2369.268624] vfs_write+0x324/0x4f0 [ 2369.272044] ksys_pwrite64+0x65/0xa0 [ 2369.275628] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2369.280772] do_syscall_64+0x38/0x80 [ 2369.284357] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2369.289410] RIP: 0033:0x7f48f0eeb93b [ 2369.292988] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2369.311735] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2369.319300] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [ 2369.326435] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [ 2369.333567] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [ 2369.340700] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183df18 [ 2369.347831] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2369.354980] [ 2369.357174] irq event stamp: 12901251 [ 2369.360841] hardirqs last enabled at (12901261): [] console_unlock+0x106/0x120 [ 2369.369708] hardirqs last disabled at (12901270): [] console_unlock+0xeb/0x120 [ 2369.378488] softirqs last enabled at (12900932): [] __do_softirq+0x2be/0x371 [ 2369.387180] softirqs last disabled at (12900927): [] irq_exit_rcu+0xb0/0xd0 [ 2369.395699] ---[ end trace 0000000000000000 ]--- [ 2369.400318] ------------[ cut here ]------------ [ 2369.404937] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2369.416586] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2369.425280] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2369.480828] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2369.489961] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2369.497440] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2369.502951] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2369.508439] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2369.527185] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2369.532420] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2369.539551] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2369.546688] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000096fb [ 2369.553827] R10: 0000000000308728 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2369.560961] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2369.568091] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2369.576179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2369.581923] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2369.589059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2369.596189] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2369.603322] Call Trace: [ 2369.605776] [ 2369.607885] ? __warn+0x85/0x180 [ 2369.611126] ? __list_add_valid_or_report+0x7d/0xa0 [ 2369.616013] ? report_bug+0xfc/0x1d0 [ 2369.619600] ? prb_read_valid+0x1b/0x20 [ 2369.623443] ? handle_bug+0x44/0x70 [ 2369.626940] ? exc_invalid_op+0x17/0x60 [ 2369.630780] ? asm_exc_invalid_op+0x1a/0x20 [ 2369.634977] ? __list_add_valid_or_report+0x7d/0xa0 [ 2369.639867] ? __list_add_valid_or_report+0x7d/0xa0 [ 2369.644750] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2369.649653] ceph_add_cap+0x20b/0x620 [ceph] [ 2369.653960] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2369.658689] ? lock_acquire+0xcb/0x2b0 [ 2369.662459] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2369.666922] ? __mutex_lock+0x97/0xce0 [ 2369.670682] ? __mutex_lock+0xaec/0xce0 [ 2369.674526] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2369.679081] ? lock_downgrade+0xfd/0x250 [ 2369.683018] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2369.687417] ceph_con_process_message+0x74/0x150 [libceph] [ 2369.692920] ? calc_signature+0xdb/0x100 [libceph] [ 2369.697745] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2369.703086] ? process_scheduled_works+0x22a/0x510 [ 2369.707890] ? process_scheduled_works+0x22a/0x510 [ 2369.712689] ceph_con_workfn+0x344/0x750 [libceph] [ 2369.717505] ? process_scheduled_works+0x22a/0x510 [ 2369.722301] process_scheduled_works+0x2a4/0x510 [ 2369.726932] worker_thread+0x188/0x340 [ 2369.730695] ? rescuer_thread+0x310/0x310 [ 2369.734713] kthread+0xf5/0x130 [ 2369.737869] ? kthread_complete_and_exit+0x20/0x20 [ 2369.742672] ret_from_fork+0x31/0x40 [ 2369.746258] ? kthread_complete_and_exit+0x20/0x20 [ 2369.751060] ret_from_fork_asm+0x11/0x20 [ 2369.755008] [ 2369.757203] irq event stamp: 10784177 [ 2369.760868] hardirqs last enabled at (10784187): [] console_unlock+0x106/0x120 [ 2369.769742] hardirqs last disabled at (10784196): [] console_unlock+0xeb/0x120 [ 2369.778523] softirqs last enabled at (10784124): [] __do_softirq+0x2be/0x371 [ 2369.787214] softirqs last disabled at (10784119): [] irq_exit_rcu+0xb0/0xd0 [ 2369.795736] ---[ end trace 0000000000000000 ]--- [ 2369.800357] ------------[ cut here ]------------ [ 2369.804981] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2369.816627] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2369.825581] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2369.881112] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2369.890062] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2369.897544] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2369.903028] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2369.921776] RSP: 0018:ffffc90001fefbe0 EFLAGS: 00010286 [ 2369.927000] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [ 2369.934135] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2369.941266] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000973a [ 2369.948400] R10: 0000000000309840 R11: ffffc90001fefa70 R12: ffff8881c3b77cb0 [ 2369.955533] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2369.962665] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2369.970750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2369.976498] CR2: 00007f9bd2629000 CR3: 0000000108c72005 CR4: 00000000003706e0 [ 2369.983631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2369.990762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2369.997895] Call Trace: [ 2370.000349] [ 2370.002457] ? __warn+0x85/0x180 [ 2370.005689] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.010569] ? report_bug+0xfc/0x1d0 [ 2370.014147] ? prb_read_valid+0x1b/0x20 [ 2370.017991] ? handle_bug+0x44/0x70 [ 2370.021488] ? exc_invalid_op+0x17/0x60 [ 2370.025328] ? asm_exc_invalid_op+0x1a/0x20 [ 2370.029524] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.034411] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2370.039312] try_get_cap_refs+0x159/0x880 [ceph] [ 2370.043962] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2370.048520] ? __file_remove_privs+0x89/0x110 [ 2370.052879] ? wake_bit_function+0x70/0x70 [ 2370.056991] ? ceph_write_iter+0x260/0x770 [ceph] [ 2370.061711] ceph_write_iter+0x260/0x770 [ceph] [ 2370.066270] ? lock_is_held_type+0xaa/0x120 [ 2370.070463] vfs_write+0x324/0x4f0 [ 2370.073880] ksys_pwrite64+0x65/0xa0 [ 2370.077464] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2370.082610] do_syscall_64+0x38/0x80 [ 2370.086195] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2370.091247] RIP: 0033:0x7f48f0eeb93b [ 2370.094826] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2370.113572] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2370.121139] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [ 2370.128271] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [ 2370.135404] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [ 2370.142537] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183df18 [ 2370.149670] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2370.156823] [ 2370.159013] irq event stamp: 12903043 [ 2370.162679] hardirqs last enabled at (12903053): [] console_unlock+0x106/0x120 [ 2370.171544] hardirqs last disabled at (12903062): [] console_unlock+0xeb/0x120 [ 2370.180325] softirqs last enabled at (12903030): [] __do_softirq+0x2be/0x371 [ 2370.189017] softirqs last disabled at (12903023): [] irq_exit_rcu+0xb0/0xd0 [ 2370.197536] ---[ end trace 0000000000000000 ]--- [ 2370.202156] ------------[ cut here ]------------ [ 2370.206774] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2370.218417] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2370.227115] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2370.282594] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2370.291718] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2370.299198] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2370.304692] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2370.310176] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2370.328925] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2370.334150] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2370.341283] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2370.348416] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009773 [ 2370.355550] R10: 000000000030a838 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2370.362681] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2370.369815] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2370.377901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2370.383644] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2370.390778] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2370.397911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2370.405044] Call Trace: [ 2370.407496] [ 2370.409604] ? __warn+0x85/0x180 [ 2370.412837] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.417715] ? report_bug+0xfc/0x1d0 [ 2370.421293] ? prb_read_valid+0x1b/0x20 [ 2370.425136] ? handle_bug+0x44/0x70 [ 2370.428626] ? exc_invalid_op+0x17/0x60 [ 2370.432466] ? asm_exc_invalid_op+0x1a/0x20 [ 2370.436658] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.441541] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.446419] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2370.451310] ceph_add_cap+0x20b/0x620 [ceph] [ 2370.455597] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2370.460325] ceph_fill_trace+0x143/0xf30 [ceph] [ 2370.464863] ? __mutex_lock+0x97/0xce0 [ 2370.468614] ? __mutex_lock+0xaec/0xce0 [ 2370.472458] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2370.477008] ? lock_downgrade+0xfd/0x250 [ 2370.480936] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2370.485317] ceph_con_process_message+0x74/0x150 [libceph] [ 2370.490813] ? calc_signature+0xdb/0x100 [libceph] [ 2370.495624] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2370.500954] ? process_scheduled_works+0x22a/0x510 [ 2370.505749] ? process_scheduled_works+0x22a/0x510 [ 2370.510536] ceph_con_workfn+0x344/0x750 [libceph] [ 2370.515340] ? process_scheduled_works+0x22a/0x510 [ 2370.520130] process_scheduled_works+0x2a4/0x510 [ 2370.524757] worker_thread+0x188/0x340 [ 2370.528512] ? rescuer_thread+0x310/0x310 [ 2370.532526] kthread+0xf5/0x130 [ 2370.535671] ? kthread_complete_and_exit+0x20/0x20 [ 2370.540466] ret_from_fork+0x31/0x40 [ 2370.544044] ? kthread_complete_and_exit+0x20/0x20 [ 2370.548837] ret_from_fork_asm+0x11/0x20 [ 2370.552770] [ 2370.554962] irq event stamp: 10786103 [ 2370.558629] hardirqs last enabled at (10786113): [] console_unlock+0x106/0x120 [ 2370.567494] hardirqs last disabled at (10786122): [] console_unlock+0xeb/0x120 [ 2370.576274] softirqs last enabled at (10786060): [] __do_softirq+0x2be/0x371 [ 2370.584968] softirqs last disabled at (10786055): [] irq_exit_rcu+0xb0/0xd0 [ 2370.593488] ---[ end trace 0000000000000000 ]--- [ 2370.598106] ------------[ cut here ]------------ [ 2370.602727] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2370.614373] WARNING: CPU: 2 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2370.623327] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2370.678787] CPU: 2 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2370.687738] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2370.695216] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2370.700703] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2370.719450] RSP: 0018:ffffc90001fdfb78 EFLAGS: 00010286 [ 2370.724676] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2370.731809] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2370.738944] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000097b1 [ 2370.746076] R10: 000000000030b920 R11: ffffc90001fdfa08 R12: ffff8881d1144bc8 [ 2370.753208] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2370.760340] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2370.768427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2370.774172] CR2: 00007fb38c57d000 CR3: 0000000179402005 CR4: 00000000003706e0 [ 2370.781305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2370.788437] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2370.795570] Call Trace: [ 2370.798013] [ 2370.800113] ? __warn+0x85/0x180 [ 2370.803345] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.808225] ? report_bug+0xfc/0x1d0 [ 2370.811805] ? prb_read_valid+0x1b/0x20 [ 2370.815645] ? handle_bug+0x44/0x70 [ 2370.819136] ? exc_invalid_op+0x17/0x60 [ 2370.822974] ? asm_exc_invalid_op+0x1a/0x20 [ 2370.827166] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.832042] ? __list_add_valid_or_report+0x7d/0xa0 [ 2370.836921] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2370.841810] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2370.847045] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2370.851847] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2370.856562] ? __d_lookup+0xd6/0x200 [ 2370.860139] lookup_fast+0x6d/0xf0 [ 2370.863544] walk_component+0x26/0x160 [ 2370.867298] link_path_walk+0x24d/0x380 [ 2370.871135] ? path_openat+0x80/0x9d0 [ 2370.874797] path_openat+0x8e/0x9d0 [ 2370.878289] do_filp_open+0xb8/0x120 [ 2370.881869] ? lock_release+0xc5/0x260 [ 2370.885622] ? _raw_spin_unlock+0x23/0x40 [ 2370.889632] ? alloc_fd+0x13f/0x200 [ 2370.893126] do_sys_openat2+0x249/0x290 [ 2370.896967] do_sys_open+0x4f/0x80 [ 2370.900370] do_syscall_64+0x38/0x80 [ 2370.903950] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2370.909001] RIP: 0033:0x7f48f0eed2f2 [ 2370.912583] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2370.931328] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2370.938894] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2370.946027] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2370.953159] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2370.960292] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2370.967427] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2370.974565] [ 2370.976749] irq event stamp: 12877483 [ 2370.980409] hardirqs last enabled at (12877493): [] console_unlock+0x106/0x120 [ 2370.989274] hardirqs last disabled at (12877502): [] console_unlock+0xeb/0x120 [ 2370.998056] softirqs last enabled at (12877516): [] __do_softirq+0x2be/0x371 [ 2371.006749] softirqs last disabled at (12877527): [] irq_exit_rcu+0xb0/0xd0 [ 2371.015265] ---[ end trace 0000000000000000 ]--- [ 2371.019885] ------------[ cut here ]------------ [ 2371.024506] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2371.036151] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2371.044848] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2371.100331] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2371.109458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2371.116938] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2371.122434] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2371.127918] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2371.146663] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2371.151889] RAX: 0000000000000000 RBX: ffff8881d106d808 RCX: 0000000000000000 [ 2371.159023] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2371.166156] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000097f0 [ 2371.173290] R10: 000000000030ca40 R11: ffffc9000030b828 R12: ffff8881d106df58 [ 2371.180423] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2371.187555] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2371.195641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2371.201386] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2371.208520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2371.215654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2371.222785] Call Trace: [ 2371.225238] [ 2371.227345] ? __warn+0x85/0x180 [ 2371.230577] ? __list_add_valid_or_report+0x7d/0xa0 [ 2371.235458] ? report_bug+0xfc/0x1d0 [ 2371.239037] ? prb_read_valid+0x1b/0x20 [ 2371.242877] ? handle_bug+0x44/0x70 [ 2371.246366] ? exc_invalid_op+0x17/0x60 [ 2371.250208] ? asm_exc_invalid_op+0x1a/0x20 [ 2371.254397] ? __list_add_valid_or_report+0x7d/0xa0 [ 2371.259273] ? __list_add_valid_or_report+0x7d/0xa0 [ 2371.264153] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2371.269041] ceph_add_cap+0x20b/0x620 [ceph] [ 2371.273328] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2371.278051] ? lock_acquire+0xcb/0x2b0 [ 2371.281813] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2371.286271] ? __mutex_lock+0x97/0xce0 [ 2371.290022] ? __mutex_lock+0xaec/0xce0 [ 2371.293863] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2371.298402] ? lock_downgrade+0xfd/0x250 [ 2371.302335] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2371.306714] ceph_con_process_message+0x74/0x150 [libceph] [ 2371.312210] ? calc_signature+0xdb/0x100 [libceph] [ 2371.317021] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2371.322344] ? process_scheduled_works+0x22a/0x510 [ 2371.327135] ? process_scheduled_works+0x22a/0x510 [ 2371.331925] ceph_con_workfn+0x344/0x750 [libceph] [ 2371.336731] ? process_scheduled_works+0x22a/0x510 [ 2371.341521] process_scheduled_works+0x2a4/0x510 [ 2371.346145] worker_thread+0x188/0x340 [ 2371.349893] ? rescuer_thread+0x310/0x310 [ 2371.353906] kthread+0xf5/0x130 [ 2371.357051] ? kthread_complete_and_exit+0x20/0x20 [ 2371.361847] ret_from_fork+0x31/0x40 [ 2371.365426] ? kthread_complete_and_exit+0x20/0x20 [ 2371.370216] ret_from_fork_asm+0x11/0x20 [ 2371.374149] [ 2371.376335] irq event stamp: 10788125 [ 2371.380001] hardirqs last enabled at (10788141): [] console_unlock+0x106/0x120 [ 2371.388868] hardirqs last disabled at (10788150): [] console_unlock+0xeb/0x120 [ 2371.397648] softirqs last enabled at (10788138): [] __do_softirq+0x2be/0x371 [ 2371.406339] softirqs last disabled at (10788133): [] irq_exit_rcu+0xb0/0xd0 [ 2371.414861] ---[ end trace 0000000000000000 ]--- [ 2371.419479] ------------[ cut here ]------------ [ 2371.424098] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2371.435746] WARNING: CPU: 0 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2371.444703] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2371.500181] CPU: 0 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2371.509141] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2371.516618] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2371.522105] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2371.540851] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [ 2371.546078] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [ 2371.553207] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2371.560341] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000982f [ 2371.567477] R10: 000000000030db58 R11: ffffc90001f7fa70 R12: ffff8881e4f8df58 [ 2371.574607] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2371.581738] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2371.589838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2371.595593] CR2: 00007f4f3ea07000 CR3: 0000000113532002 CR4: 00000000003706f0 [ 2371.602731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2371.609863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2371.616998] Call Trace: [ 2371.619451] [ 2371.621557] ? __warn+0x85/0x180 [ 2371.624790] ? __list_add_valid_or_report+0x7d/0xa0 [ 2371.629672] ? report_bug+0xfc/0x1d0 [ 2371.633258] ? prb_read_valid+0x1b/0x20 [ 2371.637102] ? handle_bug+0x44/0x70 [ 2371.640596] ? exc_invalid_op+0x17/0x60 [ 2371.644439] ? asm_exc_invalid_op+0x1a/0x20 [ 2371.648641] ? __list_add_valid_or_report+0x7d/0xa0 [ 2371.653524] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2371.658421] try_get_cap_refs+0x159/0x880 [ceph] [ 2371.663073] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2371.667629] ? __file_remove_privs+0x89/0x110 [ 2371.671989] ? wake_bit_function+0x70/0x70 [ 2371.676092] ? ceph_write_iter+0x260/0x770 [ceph] [ 2371.680822] ceph_write_iter+0x260/0x770 [ceph] [ 2371.685378] ? lock_is_held_type+0xaa/0x120 [ 2371.689572] vfs_write+0x324/0x4f0 [ 2371.692987] ksys_pwrite64+0x65/0xa0 [ 2371.696572] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2371.701719] do_syscall_64+0x38/0x80 [ 2371.705306] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2371.710364] RIP: 0033:0x7f48f0eeb93b [ 2371.713946] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2371.732699] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2371.740266] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [ 2371.747397] RDX: 0000000000000001 RSI: 000000000207bcc0 RDI: 0000000000000006 [ 2371.754532] RBP: 0000000000002a1a R08: 0000000000000001 R09: 0000000002074010 [ 2371.761667] R10: 00000000000005ff R11: 0000000000000246 R12: 00007f48f183d000 [ 2371.768796] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207bcc0 [ 2371.775942] [ 2371.778133] irq event stamp: 12970773 [ 2371.781796] hardirqs last enabled at (12970783): [] console_unlock+0x106/0x120 [ 2371.790666] hardirqs last disabled at (12970792): [] console_unlock+0xeb/0x120 [ 2371.799451] softirqs last enabled at (12970740): [] __do_softirq+0x2be/0x371 [ 2371.808145] softirqs last disabled at (12970735): [] irq_exit_rcu+0xb0/0xd0 [ 2371.816672] ---[ end trace 0000000000000000 ]--- [ 2371.821294] ------------[ cut here ]------------ [ 2371.825909] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2371.837570] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2371.846260] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2371.901755] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2371.910896] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2371.918393] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2371.923905] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2371.929408] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2371.948173] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2371.953397] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2371.960541] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2371.967691] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009868 [ 2371.974841] R10: 000000000030eb60 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2371.981990] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2371.989124] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2371.997225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.002979] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2372.010105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2372.017237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2372.024379] Call Trace: [ 2372.026839] [ 2372.028956] ? __warn+0x85/0x180 [ 2372.032191] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.037069] ? report_bug+0xfc/0x1d0 [ 2372.040646] ? prb_read_valid+0x1b/0x20 [ 2372.044505] ? handle_bug+0x44/0x70 [ 2372.048014] ? exc_invalid_op+0x17/0x60 [ 2372.051854] ? asm_exc_invalid_op+0x1a/0x20 [ 2372.056061] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.060951] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.065833] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2372.070740] ceph_add_cap+0x20b/0x620 [ceph] [ 2372.075044] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2372.079789] ceph_fill_trace+0x143/0xf30 [ceph] [ 2372.084329] ? __mutex_lock+0x97/0xce0 [ 2372.088099] ? __mutex_lock+0xaec/0xce0 [ 2372.091953] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2372.096498] ? lock_downgrade+0xfd/0x250 [ 2372.100460] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2372.104859] ceph_con_process_message+0x74/0x150 [libceph] [ 2372.110353] ? calc_signature+0xdb/0x100 [libceph] [ 2372.115175] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2372.120498] ? process_scheduled_works+0x22a/0x510 [ 2372.125306] ? process_scheduled_works+0x22a/0x510 [ 2372.130113] ceph_con_workfn+0x344/0x750 [libceph] [ 2372.134940] ? process_scheduled_works+0x22a/0x510 [ 2372.139733] process_scheduled_works+0x2a4/0x510 [ 2372.144375] worker_thread+0x188/0x340 [ 2372.148127] ? rescuer_thread+0x310/0x310 [ 2372.152137] kthread+0xf5/0x130 [ 2372.155284] ? kthread_complete_and_exit+0x20/0x20 [ 2372.160096] ret_from_fork+0x31/0x40 [ 2372.163673] ? kthread_complete_and_exit+0x20/0x20 [ 2372.168484] ret_from_fork_asm+0x11/0x20 [ 2372.172451] [ 2372.174634] irq event stamp: 10790031 [ 2372.178311] hardirqs last enabled at (10790041): [] console_unlock+0x106/0x120 [ 2372.187194] hardirqs last disabled at (10790050): [] console_unlock+0xeb/0x120 [ 2372.195974] softirqs last enabled at (10789988): [] __do_softirq+0x2be/0x371 [ 2372.204667] softirqs last disabled at (10789983): [] irq_exit_rcu+0xb0/0xd0 [ 2372.213202] ---[ end trace 0000000000000000 ]--- [ 2372.218491] ------------[ cut here ]------------ [ 2372.223249] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2372.234916] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2372.243608] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2372.299103] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2372.308234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2372.315725] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2372.321250] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2372.326747] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2372.345511] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2372.350754] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2372.357895] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2372.365038] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000098a6 [ 2372.372172] R10: 000000000030fd80 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2372.379303] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2372.386446] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2372.394548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.400311] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2372.407453] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2372.414603] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2372.421745] Call Trace: [ 2372.424207] [ 2372.426312] ? __warn+0x85/0x180 [ 2372.429563] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.434458] ? report_bug+0xfc/0x1d0 [ 2372.438056] ? prb_read_valid+0x1b/0x20 [ 2372.441897] ? handle_bug+0x44/0x70 [ 2372.445406] ? exc_invalid_op+0x17/0x60 [ 2372.449262] ? asm_exc_invalid_op+0x1a/0x20 [ 2372.453486] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.458362] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.463259] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2372.468167] ceph_add_cap+0x20b/0x620 [ceph] [ 2372.472468] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2372.477184] ? lock_acquire+0xcb/0x2b0 [ 2372.480954] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2372.485411] ? __mutex_lock+0x97/0xce0 [ 2372.489181] ? __mutex_lock+0xaec/0xce0 [ 2372.493023] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2372.497561] ? lock_downgrade+0xfd/0x250 [ 2372.501518] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2372.505907] ceph_con_process_message+0x74/0x150 [libceph] [ 2372.511404] ? calc_signature+0xdb/0x100 [libceph] [ 2372.516225] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2372.521572] ? process_scheduled_works+0x22a/0x510 [ 2372.526380] ? process_scheduled_works+0x22a/0x510 [ 2372.531172] ceph_con_workfn+0x344/0x750 [libceph] [ 2372.535976] ? process_scheduled_works+0x22a/0x510 [ 2372.540766] process_scheduled_works+0x2a4/0x510 [ 2372.545424] worker_thread+0x188/0x340 [ 2372.549176] ? rescuer_thread+0x310/0x310 [ 2372.553187] kthread+0xf5/0x130 [ 2372.556332] ? kthread_complete_and_exit+0x20/0x20 [ 2372.561142] ret_from_fork+0x31/0x40 [ 2372.564721] ? kthread_complete_and_exit+0x20/0x20 [ 2372.569532] ret_from_fork_asm+0x11/0x20 [ 2372.573498] [ 2372.575683] irq event stamp: 10791953 [ 2372.579360] hardirqs last enabled at (10791963): [] console_unlock+0x106/0x120 [ 2372.588243] hardirqs last disabled at (10791972): [] console_unlock+0xeb/0x120 [ 2372.597039] softirqs last enabled at (10791900): [] __do_softirq+0x2be/0x371 [ 2372.605732] softirqs last disabled at (10791895): [] irq_exit_rcu+0xb0/0xd0 [ 2372.614270] ---[ end trace 0000000000000000 ]--- [ 2372.618937] ------------[ cut here ]------------ [ 2372.623552] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2372.635198] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2372.643892] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2372.699397] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2372.708540] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2372.716036] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2372.721531] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2372.727032] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2372.745781] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2372.751021] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2372.758147] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2372.765280] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000098e5 [ 2372.772429] R10: 0000000000310e98 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2372.779571] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2372.786721] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2372.794816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.800571] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2372.807721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2372.814863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2372.822014] Call Trace: [ 2372.824467] [ 2372.826588] ? __warn+0x85/0x180 [ 2372.829842] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.834738] ? report_bug+0xfc/0x1d0 [ 2372.838332] ? prb_read_valid+0x1b/0x20 [ 2372.842183] ? handle_bug+0x44/0x70 [ 2372.845673] ? exc_invalid_op+0x17/0x60 [ 2372.849522] ? asm_exc_invalid_op+0x1a/0x20 [ 2372.853746] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.858624] ? __list_add_valid_or_report+0x7d/0xa0 [ 2372.863519] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2372.868426] ceph_add_cap+0x20b/0x620 [ceph] [ 2372.872745] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2372.877483] ceph_fill_trace+0x143/0xf30 [ceph] [ 2372.882023] ? __mutex_lock+0x97/0xce0 [ 2372.885774] ? __mutex_lock+0xaec/0xce0 [ 2372.889625] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2372.894185] ? lock_downgrade+0xfd/0x250 [ 2372.898114] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2372.902510] ceph_con_process_message+0x74/0x150 [libceph] [ 2372.908006] ? calc_signature+0xdb/0x100 [libceph] [ 2372.912826] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2372.918148] ? process_scheduled_works+0x22a/0x510 [ 2372.922960] ? process_scheduled_works+0x22a/0x510 [ 2372.927750] ceph_con_workfn+0x344/0x750 [libceph] [ 2372.932553] ? process_scheduled_works+0x22a/0x510 [ 2372.937359] process_scheduled_works+0x2a4/0x510 [ 2372.942001] worker_thread+0x188/0x340 [ 2372.945750] ? rescuer_thread+0x310/0x310 [ 2372.949779] kthread+0xf5/0x130 [ 2372.952961] ? kthread_complete_and_exit+0x20/0x20 [ 2372.957755] ret_from_fork+0x31/0x40 [ 2372.961350] ? kthread_complete_and_exit+0x20/0x20 [ 2372.966159] ret_from_fork_asm+0x11/0x20 [ 2372.970094] [ 2372.972287] irq event stamp: 10793009 [ 2372.975969] hardirqs last enabled at (10793019): [] console_unlock+0x106/0x120 [ 2372.984837] hardirqs last disabled at (10793028): [] console_unlock+0xeb/0x120 [ 2372.993634] softirqs last enabled at (10792966): [] __do_softirq+0x2be/0x371 [ 2373.002343] softirqs last disabled at (10792961): [] irq_exit_rcu+0xb0/0xd0 [ 2373.010879] ---[ end trace 0000000000000000 ]--- [ 2373.015543] ------------[ cut here ]------------ [ 2373.020166] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2373.031824] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2373.040512] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2373.096007] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2373.105134] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2373.112612] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2373.118124] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2373.123609] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2373.142374] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2373.147616] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2373.154759] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2373.161910] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009923 [ 2373.169060] R10: 0000000000311f88 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2373.176193] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2373.183325] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2373.191428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2373.197190] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2373.204323] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2373.211465] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2373.218607] Call Trace: [ 2373.221070] [ 2373.223174] ? __warn+0x85/0x180 [ 2373.226409] ? __list_add_valid_or_report+0x7d/0xa0 [ 2373.231304] ? report_bug+0xfc/0x1d0 [ 2373.234901] ? prb_read_valid+0x1b/0x20 [ 2373.238760] ? handle_bug+0x44/0x70 [ 2373.242269] ? exc_invalid_op+0x17/0x60 [ 2373.246125] ? asm_exc_invalid_op+0x1a/0x20 [ 2373.250332] ? __list_add_valid_or_report+0x7d/0xa0 [ 2373.255209] ? __list_add_valid_or_report+0x7d/0xa0 [ 2373.260086] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2373.264977] ceph_add_cap+0x20b/0x620 [ceph] [ 2373.269282] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2373.274004] ? lock_acquire+0xcb/0x2b0 [ 2373.277779] ceph_fill_trace+0x143/0xf30 [ceph] [ 2373.282317] ? __mutex_lock+0x97/0xce0 [ 2373.286086] ? __mutex_lock+0xaec/0xce0 [ 2373.289930] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2373.294494] ? lock_downgrade+0xfd/0x250 [ 2373.298441] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2373.302829] ceph_con_process_message+0x74/0x150 [libceph] [ 2373.308326] ? calc_signature+0xdb/0x100 [libceph] [ 2373.313147] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2373.318470] ? process_scheduled_works+0x22a/0x510 [ 2373.323295] ? process_scheduled_works+0x22a/0x510 [ 2373.328086] ceph_con_workfn+0x344/0x750 [libceph] [ 2373.332890] ? process_scheduled_works+0x22a/0x510 [ 2373.337698] process_scheduled_works+0x2a4/0x510 [ 2373.342357] worker_thread+0x188/0x340 [ 2373.346106] ? rescuer_thread+0x310/0x310 [ 2373.350118] kthread+0xf5/0x130 [ 2373.353264] ? kthread_complete_and_exit+0x20/0x20 [ 2373.358076] ret_from_fork+0x31/0x40 [ 2373.361652] ? kthread_complete_and_exit+0x20/0x20 [ 2373.366464] ret_from_fork_asm+0x11/0x20 [ 2373.370430] [ 2373.372618] irq event stamp: 10794123 [ 2373.376299] hardirqs last enabled at (10794133): [] console_unlock+0x106/0x120 [ 2373.385185] hardirqs last disabled at (10794142): [] console_unlock+0xeb/0x120 [ 2373.393974] softirqs last enabled at (10794070): [] __do_softirq+0x2be/0x371 [ 2373.402664] softirqs last disabled at (10794065): [] irq_exit_rcu+0xb0/0xd0 [ 2373.411201] ---[ end trace 0000000000000000 ]--- [ 2373.415853] ------------[ cut here ]------------ [ 2373.420467] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2373.432110] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2373.440799] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2373.496277] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2373.505403] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2373.512884] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2373.518385] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2373.523870] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2373.542616] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2373.547843] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2373.554977] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2373.562109] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009962 [ 2373.569241] R10: 00000000003130a0 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2373.576374] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2373.583507] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2373.591593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2373.597339] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2373.604471] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2373.611605] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2373.618738] Call Trace: [ 2373.621190] [ 2373.623297] ? __warn+0x85/0x180 [ 2373.626529] ? __list_add_valid_or_report+0x7d/0xa0 [ 2373.631410] ? report_bug+0xfc/0x1d0 [ 2373.634987] ? prb_read_valid+0x1b/0x20 [ 2373.638828] ? handle_bug+0x44/0x70 [ 2373.642319] ? exc_invalid_op+0x17/0x60 [ 2373.646159] ? asm_exc_invalid_op+0x1a/0x20 [ 2373.650351] ? __list_add_valid_or_report+0x7d/0xa0 [ 2373.655227] ? __list_add_valid_or_report+0x7d/0xa0 [ 2373.660103] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2373.664995] ceph_add_cap+0x20b/0x620 [ceph] [ 2373.669279] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2373.673994] ? lock_acquire+0xcb/0x2b0 [ 2373.677747] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2373.682196] ? __mutex_lock+0x97/0xce0 [ 2373.685953] ? __mutex_lock+0xaec/0xce0 [ 2373.689790] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2373.694331] ? lock_downgrade+0xfd/0x250 [ 2373.698261] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2373.702639] ceph_con_process_message+0x74/0x150 [libceph] [ 2373.708134] ? calc_signature+0xdb/0x100 [libceph] [ 2373.712941] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2373.718270] ? process_scheduled_works+0x22a/0x510 [ 2373.723063] ? process_scheduled_works+0x22a/0x510 [ 2373.727853] ceph_con_workfn+0x344/0x750 [libceph] [ 2373.732656] ? process_scheduled_works+0x22a/0x510 [ 2373.737446] process_scheduled_works+0x2a4/0x510 [ 2373.742073] worker_thread+0x188/0x340 [ 2373.745830] ? rescuer_thread+0x310/0x310 [ 2373.749840] kthread+0xf5/0x130 [ 2373.752989] ? kthread_complete_and_exit+0x20/0x20 [ 2373.757780] ret_from_fork+0x31/0x40 [ 2373.761360] ? kthread_complete_and_exit+0x20/0x20 [ 2373.766153] ret_from_fork_asm+0x11/0x20 [ 2373.770086] [ 2373.772279] irq event stamp: 10795231 [ 2373.775946] hardirqs last enabled at (10795241): [] console_unlock+0x106/0x120 [ 2373.784811] hardirqs last disabled at (10795252): [] console_unlock+0xeb/0x120 [ 2373.793590] softirqs last enabled at (10795178): [] __do_softirq+0x2be/0x371 [ 2373.802283] softirqs last disabled at (10795173): [] irq_exit_rcu+0xb0/0xd0 [ 2373.810803] ---[ end trace 0000000000000000 ]--- [ 2373.815423] ------------[ cut here ]------------ [ 2373.820042] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2373.831686] WARNING: CPU: 2 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2373.840643] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2373.896105] CPU: 2 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2373.905053] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2373.912523] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2373.918010] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2373.936764] RSP: 0018:ffffc90001957b78 EFLAGS: 00010286 [ 2373.941994] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2373.949125] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2373.956257] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000099a1 [ 2373.963382] R10: 00000000003141c0 R11: ffffc90001957a08 R12: ffff8881d12224a8 [ 2373.970516] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2373.977647] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2373.985733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2373.991478] CR2: 00007fb38c57d000 CR3: 000000016696c003 CR4: 00000000003706e0 [ 2373.998603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2374.005736] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2374.012869] Call Trace: [ 2374.015312] [ 2374.017411] ? __warn+0x85/0x180 [ 2374.020644] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.025524] ? report_bug+0xfc/0x1d0 [ 2374.029103] ? prb_read_valid+0x1b/0x20 [ 2374.032944] ? handle_bug+0x44/0x70 [ 2374.036433] ? exc_invalid_op+0x17/0x60 [ 2374.040265] ? asm_exc_invalid_op+0x1a/0x20 [ 2374.044455] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.049330] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.054211] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2374.059100] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2374.064335] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2374.069137] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2374.073850] ? __d_lookup+0xd6/0x200 [ 2374.077431] lookup_fast+0x6d/0xf0 [ 2374.080835] walk_component+0x26/0x160 [ 2374.084588] link_path_walk+0x24d/0x380 [ 2374.088428] ? path_openat+0x80/0x9d0 [ 2374.092097] path_openat+0x8e/0x9d0 [ 2374.095589] do_filp_open+0xb8/0x120 [ 2374.099169] ? lock_release+0xc5/0x260 [ 2374.102919] ? _raw_spin_unlock+0x23/0x40 [ 2374.106929] ? alloc_fd+0x13f/0x200 [ 2374.110427] do_sys_openat2+0x249/0x290 [ 2374.114263] do_sys_open+0x4f/0x80 [ 2374.117669] do_syscall_64+0x38/0x80 [ 2374.121248] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2374.126302] RIP: 0033:0x7f48f0ef2805 [ 2374.129881] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2374.148627] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2374.156193] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2374.163325] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2374.170457] RBP: 000000000207da30 R08: 0000000003ffffff R09: 0000000000000000 [ 2374.177581] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2374.184705] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2374.191836] [ 2374.194024] irq event stamp: 12850059 [ 2374.197687] hardirqs last enabled at (12850069): [] console_unlock+0x106/0x120 [ 2374.206554] hardirqs last disabled at (12850078): [] console_unlock+0xeb/0x120 [ 2374.215334] softirqs last enabled at (12849996): [] __do_softirq+0x2be/0x371 [ 2374.224028] softirqs last disabled at (12850097): [] irq_exit_rcu+0xb0/0xd0 [ 2374.232546] ---[ end trace 0000000000000000 ]--- [ 2374.237158] ------------[ cut here ]------------ [ 2374.241778] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2374.253441] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2374.262139] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2374.317613] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2374.326757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2374.334268] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2374.339783] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2374.345287] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2374.364051] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2374.369276] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2374.376425] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2374.383567] RBP: ffff88817f867000 R08: 0000000000000000 R09: c0000001000099e0 [ 2374.390718] R10: 00000000003152d8 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2374.397867] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2374.405009] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2374.413095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2374.418841] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2374.425992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2374.433123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2374.440273] Call Trace: [ 2374.442736] [ 2374.444849] ? __warn+0x85/0x180 [ 2374.448093] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.452991] ? report_bug+0xfc/0x1d0 [ 2374.456567] ? prb_read_valid+0x1b/0x20 [ 2374.460425] ? handle_bug+0x44/0x70 [ 2374.463933] ? exc_invalid_op+0x17/0x60 [ 2374.467782] ? asm_exc_invalid_op+0x1a/0x20 [ 2374.472000] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.476875] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.481771] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2374.486679] ceph_add_cap+0x20b/0x620 [ceph] [ 2374.490999] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2374.495744] ceph_fill_trace+0x143/0xf30 [ceph] [ 2374.500283] ? __mutex_lock+0x97/0xce0 [ 2374.504054] ? __mutex_lock+0xaec/0xce0 [ 2374.507894] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2374.512451] ? lock_downgrade+0xfd/0x250 [ 2374.516419] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2374.520814] ceph_con_process_message+0x74/0x150 [libceph] [ 2374.526310] ? calc_signature+0xdb/0x100 [libceph] [ 2374.531130] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2374.536470] ? process_scheduled_works+0x22a/0x510 [ 2374.541262] ? process_scheduled_works+0x22a/0x510 [ 2374.546052] ceph_con_workfn+0x344/0x750 [libceph] [ 2374.550859] ? process_scheduled_works+0x22a/0x510 [ 2374.555663] process_scheduled_works+0x2a4/0x510 [ 2374.560315] worker_thread+0x188/0x340 [ 2374.564065] ? rescuer_thread+0x310/0x310 [ 2374.568079] kthread+0xf5/0x130 [ 2374.571223] ? kthread_complete_and_exit+0x20/0x20 [ 2374.576016] ret_from_fork+0x31/0x40 [ 2374.579593] ? kthread_complete_and_exit+0x20/0x20 [ 2374.584403] ret_from_fork_asm+0x11/0x20 [ 2374.588370] [ 2374.590557] irq event stamp: 10797219 [ 2374.594231] hardirqs last enabled at (10797229): [] console_unlock+0x106/0x120 [ 2374.603099] hardirqs last disabled at (10797238): [] console_unlock+0xeb/0x120 [ 2374.611876] softirqs last enabled at (10797186): [] __do_softirq+0x2be/0x371 [ 2374.620588] softirqs last disabled at (10797181): [] irq_exit_rcu+0xb0/0xd0 [ 2374.629123] ---[ end trace 0000000000000000 ]--- [ 2374.633912] ------------[ cut here ]------------ [ 2374.638529] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2374.650193] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2374.658885] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2374.714381] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2374.723523] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2374.731012] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2374.736507] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2374.742010] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2374.760756] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2374.765998] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2374.773132] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2374.780281] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009a1e [ 2374.787423] R10: 00000000003163b8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2374.794566] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2374.801716] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2374.809820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2374.815581] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2374.822725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2374.829875] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2374.837025] Call Trace: [ 2374.839477] [ 2374.841601] ? __warn+0x85/0x180 [ 2374.844853] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.849748] ? report_bug+0xfc/0x1d0 [ 2374.853343] ? prb_read_valid+0x1b/0x20 [ 2374.857195] ? handle_bug+0x44/0x70 [ 2374.860685] ? exc_invalid_op+0x17/0x60 [ 2374.864542] ? asm_exc_invalid_op+0x1a/0x20 [ 2374.868767] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.873642] ? __list_add_valid_or_report+0x7d/0xa0 [ 2374.878537] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2374.883447] ceph_add_cap+0x20b/0x620 [ceph] [ 2374.887766] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2374.892497] ? lock_acquire+0xcb/0x2b0 [ 2374.896250] ceph_fill_trace+0x143/0xf30 [ceph] [ 2374.900787] ? __mutex_lock+0x97/0xce0 [ 2374.904556] ? __mutex_lock+0xaec/0xce0 [ 2374.908415] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2374.912991] ? lock_downgrade+0xfd/0x250 [ 2374.916936] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2374.921323] ceph_con_process_message+0x74/0x150 [libceph] [ 2374.926822] ? calc_signature+0xdb/0x100 [libceph] [ 2374.931659] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2374.936999] ? process_scheduled_works+0x22a/0x510 [ 2374.941807] ? process_scheduled_works+0x22a/0x510 [ 2374.946599] ceph_con_workfn+0x344/0x750 [libceph] [ 2374.951418] ? process_scheduled_works+0x22a/0x510 [ 2374.956227] process_scheduled_works+0x2a4/0x510 [ 2374.960870] worker_thread+0x188/0x340 [ 2374.964619] ? rescuer_thread+0x310/0x310 [ 2374.968646] kthread+0xf5/0x130 [ 2374.971813] ? kthread_complete_and_exit+0x20/0x20 [ 2374.976621] ret_from_fork+0x31/0x40 [ 2374.980217] ? kthread_complete_and_exit+0x20/0x20 [ 2374.985012] ret_from_fork_asm+0x11/0x20 [ 2374.988958] [ 2374.991147] irq event stamp: 10798335 [ 2374.994812] hardirqs last enabled at (10798345): [] console_unlock+0x106/0x120 [ 2375.003697] hardirqs last disabled at (10798354): [] console_unlock+0xeb/0x120 [ 2375.012493] softirqs last enabled at (10798282): [] __do_softirq+0x2be/0x371 [ 2375.021202] softirqs last disabled at (10798277): [] irq_exit_rcu+0xb0/0xd0 [ 2375.029723] ---[ end trace 0000000000000000 ]--- [ 2375.034392] ------------[ cut here ]------------ [ 2375.039006] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2375.050650] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2375.059345] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2375.114825] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2375.123948] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2375.131427] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2375.136923] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2375.142408] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2375.161155] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2375.166380] RAX: 0000000000000000 RBX: ffff8881d1144478 RCX: 0000000000000000 [ 2375.173515] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2375.180647] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009a5d [ 2375.187781] R10: 0000000000317608 R11: ffffc9000030b828 R12: ffff8881d1144bc8 [ 2375.194911] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2375.202044] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2375.210131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2375.215877] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2375.223010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2375.230144] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2375.237277] Call Trace: [ 2375.239729] [ 2375.241835] ? __warn+0x85/0x180 [ 2375.245069] ? __list_add_valid_or_report+0x7d/0xa0 [ 2375.249947] ? report_bug+0xfc/0x1d0 [ 2375.253525] ? prb_read_valid+0x1b/0x20 [ 2375.257368] ? handle_bug+0x44/0x70 [ 2375.260857] ? exc_invalid_op+0x17/0x60 [ 2375.264698] ? asm_exc_invalid_op+0x1a/0x20 [ 2375.268889] ? __list_add_valid_or_report+0x7d/0xa0 [ 2375.273765] ? __list_add_valid_or_report+0x7d/0xa0 [ 2375.278642] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2375.283534] ceph_add_cap+0x20b/0x620 [ceph] [ 2375.287818] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2375.292541] ? lock_acquire+0xcb/0x2b0 [ 2375.296292] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2375.300744] ? __mutex_lock+0x97/0xce0 [ 2375.304495] ? __mutex_lock+0xaec/0xce0 [ 2375.308338] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2375.312884] ? lock_downgrade+0xfd/0x250 [ 2375.316817] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2375.321196] ceph_con_process_message+0x74/0x150 [libceph] [ 2375.326692] ? calc_signature+0xdb/0x100 [libceph] [ 2375.331511] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2375.336837] ? process_scheduled_works+0x22a/0x510 [ 2375.341637] ? process_scheduled_works+0x22a/0x510 [ 2375.346425] ceph_con_workfn+0x344/0x750 [libceph] [ 2375.351229] ? process_scheduled_works+0x22a/0x510 [ 2375.356020] process_scheduled_works+0x2a4/0x510 [ 2375.360645] worker_thread+0x188/0x340 [ 2375.364393] ? rescuer_thread+0x310/0x310 [ 2375.368406] kthread+0xf5/0x130 [ 2375.371553] ? kthread_complete_and_exit+0x20/0x20 [ 2375.376345] ret_from_fork+0x31/0x40 [ 2375.379923] ? kthread_complete_and_exit+0x20/0x20 [ 2375.384716] ret_from_fork_asm+0x11/0x20 [ 2375.388648] [ 2375.390834] irq event stamp: 10799457 [ 2375.394499] hardirqs last enabled at (10799467): [] console_unlock+0x106/0x120 [ 2375.403373] hardirqs last disabled at (10799476): [] console_unlock+0xeb/0x120 [ 2375.412157] softirqs last enabled at (10799404): [] __do_softirq+0x2be/0x371 [ 2375.420847] softirqs last disabled at (10799399): [] irq_exit_rcu+0xb0/0xd0 [ 2375.429369] ---[ end trace 0000000000000000 ]--- [ 2375.433994] ------------[ cut here ]------------ [ 2375.438614] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2375.450279] WARNING: CPU: 6 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2375.459251] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2375.514779] CPU: 6 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2375.523732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2375.531210] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2375.536702] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2375.555444] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2375.560669] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2375.567800] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2375.574924] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009a9c [ 2375.582052] R10: 0000000000318728 R11: ffffc90001fd7950 R12: ffff8881c5350750 [ 2375.589181] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2375.596305] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2375.604394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2375.610138] CR2: 00007f9bd22d7020 CR3: 0000000120c1e003 CR4: 00000000003706e0 [ 2375.617262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2375.624386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2375.631510] Call Trace: [ 2375.633957] [ 2375.636064] ? __warn+0x85/0x180 [ 2375.639293] ? __list_add_valid_or_report+0x7d/0xa0 [ 2375.644174] ? report_bug+0xfc/0x1d0 [ 2375.647755] ? prb_read_valid+0x1b/0x20 [ 2375.651595] ? handle_bug+0x44/0x70 [ 2375.655085] ? exc_invalid_op+0x17/0x60 [ 2375.658924] ? asm_exc_invalid_op+0x1a/0x20 [ 2375.663114] ? __list_add_valid_or_report+0x7d/0xa0 [ 2375.667993] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2375.672889] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2375.678125] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2375.682924] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2375.687640] ? __d_lookup+0xd6/0x200 [ 2375.691218] lookup_fast+0x6d/0xf0 [ 2375.694623] walk_component+0x26/0x160 [ 2375.698377] link_path_walk+0x24d/0x380 [ 2375.702217] ? path_lookupat.isra.40+0x22/0x120 [ 2375.706751] path_lookupat.isra.40+0x6b/0x120 [ 2375.711111] filename_lookup+0xd4/0x190 [ 2375.714952] ? lock_acquire+0xcb/0x2b0 [ 2375.718701] ? __might_fault+0x45/0x80 [ 2375.722452] ? find_held_lock+0x2d/0x90 [ 2375.726291] ? __might_fault+0x45/0x80 [ 2375.730048] vfs_statx+0x89/0x150 [ 2375.733367] vfs_fstatat+0x5a/0xc0 [ 2375.736773] __do_sys_newstat+0x30/0x70 [ 2375.740610] ? lock_release+0xc5/0x260 [ 2375.744364] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2375.749241] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2375.754382] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2375.759521] do_syscall_64+0x38/0x80 [ 2375.763100] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2375.768154] RIP: 0033:0x7f48f0eeca79 [ 2375.771733] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2375.790478] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2375.798046] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2375.805178] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2375.812309] RBP: 00007ffc53d60520 R08: 0000000000000000 R09: 00000000020767c0 [ 2375.819433] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2375.826557] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020767c0 [ 2375.833691] [ 2375.835885] irq event stamp: 13014577 [ 2375.839549] hardirqs last enabled at (13014587): [] console_unlock+0x106/0x120 [ 2375.848422] hardirqs last disabled at (13014602): [] console_unlock+0xeb/0x120 [ 2375.857199] softirqs last enabled at (13014616): [] __do_softirq+0x2be/0x371 [ 2375.865888] softirqs last disabled at (13014611): [] irq_exit_rcu+0xb0/0xd0 [ 2375.874407] ---[ end trace 0000000000000000 ]--- [ 2375.879029] ------------[ cut here ]------------ [ 2375.883648] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2375.895333] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2375.904025] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2375.959511] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2375.968653] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2375.976150] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2375.981660] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2375.987164] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2376.005910] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2376.011154] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2376.018301] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2376.025446] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009ae0 [ 2376.032596] R10: 0000000000319918 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2376.039746] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2376.046896] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2376.055018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2376.060761] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2376.067904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2376.075054] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2376.082187] Call Trace: [ 2376.084640] [ 2376.086765] ? __warn+0x85/0x180 [ 2376.090024] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.094903] ? report_bug+0xfc/0x1d0 [ 2376.098498] ? prb_read_valid+0x1b/0x20 [ 2376.102358] ? handle_bug+0x44/0x70 [ 2376.105864] ? exc_invalid_op+0x17/0x60 [ 2376.109714] ? asm_exc_invalid_op+0x1a/0x20 [ 2376.113937] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.118814] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.123710] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2376.128619] ceph_add_cap+0x20b/0x620 [ceph] [ 2376.132936] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2376.137674] ceph_fill_trace+0x143/0xf30 [ceph] [ 2376.142216] ? __mutex_lock+0x97/0xce0 [ 2376.145967] ? __mutex_lock+0xaec/0xce0 [ 2376.149823] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2376.154383] ? lock_downgrade+0xfd/0x250 [ 2376.158347] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2376.162743] ceph_con_process_message+0x74/0x150 [libceph] [ 2376.168240] ? calc_signature+0xdb/0x100 [libceph] [ 2376.173044] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2376.178394] ? process_scheduled_works+0x22a/0x510 [ 2376.183193] ? process_scheduled_works+0x22a/0x510 [ 2376.187983] ceph_con_workfn+0x344/0x750 [libceph] [ 2376.192811] ? process_scheduled_works+0x22a/0x510 [ 2376.197603] process_scheduled_works+0x2a4/0x510 [ 2376.202245] worker_thread+0x188/0x340 [ 2376.206015] ? rescuer_thread+0x310/0x310 [ 2376.210025] kthread+0xf5/0x130 [ 2376.213170] ? kthread_complete_and_exit+0x20/0x20 [ 2376.217981] ret_from_fork+0x31/0x40 [ 2376.221557] ? kthread_complete_and_exit+0x20/0x20 [ 2376.226369] ret_from_fork_asm+0x11/0x20 [ 2376.230336] [ 2376.232521] irq event stamp: 10801459 [ 2376.236196] hardirqs last enabled at (10801469): [] console_unlock+0x106/0x120 [ 2376.245063] hardirqs last disabled at (10801478): [] console_unlock+0xeb/0x120 [ 2376.253843] softirqs last enabled at (10801416): [] __do_softirq+0x2be/0x371 [ 2376.262551] softirqs last disabled at (10801411): [] irq_exit_rcu+0xb0/0xd0 [ 2376.271089] ---[ end trace 0000000000000000 ]--- [ 2376.275758] ------------[ cut here ]------------ [ 2376.280374] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2376.292036] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2376.300730] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2376.356225] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2376.365348] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2376.372839] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2376.378349] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2376.383851] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2376.402610] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2376.407852] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2376.415002] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2376.422154] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009b1e [ 2376.429285] R10: 000000000031a9e0 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2376.436418] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2376.443568] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2376.451671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2376.457434] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2376.464576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2376.471726] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2376.478877] Call Trace: [ 2376.481346] [ 2376.483461] ? __warn+0x85/0x180 [ 2376.486713] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.491609] ? report_bug+0xfc/0x1d0 [ 2376.495204] ? prb_read_valid+0x1b/0x20 [ 2376.499046] ? handle_bug+0x44/0x70 [ 2376.502536] ? exc_invalid_op+0x17/0x60 [ 2376.506386] ? asm_exc_invalid_op+0x1a/0x20 [ 2376.510609] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.515486] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.520382] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2376.525289] ceph_add_cap+0x20b/0x620 [ceph] [ 2376.529575] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2376.534334] ? lock_acquire+0xcb/0x2b0 [ 2376.538094] ceph_fill_trace+0x143/0xf30 [ceph] [ 2376.542630] ? __mutex_lock+0x97/0xce0 [ 2376.546399] ? __mutex_lock+0xaec/0xce0 [ 2376.550250] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2376.554790] ? lock_downgrade+0xfd/0x250 [ 2376.558754] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2376.563136] ceph_con_process_message+0x74/0x150 [libceph] [ 2376.568630] ? calc_signature+0xdb/0x100 [libceph] [ 2376.573467] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2376.578808] ? process_scheduled_works+0x22a/0x510 [ 2376.583600] ? process_scheduled_works+0x22a/0x510 [ 2376.588408] ceph_con_workfn+0x344/0x750 [libceph] [ 2376.593229] ? process_scheduled_works+0x22a/0x510 [ 2376.598031] process_scheduled_works+0x2a4/0x510 [ 2376.602668] worker_thread+0x188/0x340 [ 2376.606419] ? rescuer_thread+0x310/0x310 [ 2376.610447] kthread+0xf5/0x130 [ 2376.613612] ? kthread_complete_and_exit+0x20/0x20 [ 2376.618442] ret_from_fork+0x31/0x40 [ 2376.622033] ? kthread_complete_and_exit+0x20/0x20 [ 2376.626828] ret_from_fork_asm+0x11/0x20 [ 2376.630795] [ 2376.632981] irq event stamp: 10802591 [ 2376.636656] hardirqs last enabled at (10802601): [] console_unlock+0x106/0x120 [ 2376.645539] hardirqs last disabled at (10802610): [] console_unlock+0xeb/0x120 [ 2376.654337] softirqs last enabled at (10802558): [] __do_softirq+0x2be/0x371 [ 2376.663047] softirqs last disabled at (10802629): [] irq_exit_rcu+0xb0/0xd0 [ 2376.671566] ---[ end trace 0000000000000000 ]--- [ 2376.676227] ------------[ cut here ]------------ [ 2376.680851] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2376.692495] WARNING: CPU: 6 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2376.701448] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2376.756910] CPU: 6 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2376.765860] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2376.773339] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2376.778825] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2376.797574] RSP: 0018:ffffc90001fc7c10 EFLAGS: 00010286 [ 2376.802797] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2376.809923] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2376.817054] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009b5d [ 2376.824181] R10: 000000000031bc50 R11: ffffc90001fc7aa0 R12: ffff8881d0bec200 [ 2376.831310] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2376.838436] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2376.846521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2376.852260] CR2: 00007f9bd22d7020 CR3: 0000000113500003 CR4: 00000000003706e0 [ 2376.859393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2376.866524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2376.873657] Call Trace: [ 2376.876103] [ 2376.878209] ? __warn+0x85/0x180 [ 2376.881442] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.886321] ? report_bug+0xfc/0x1d0 [ 2376.889901] ? prb_read_valid+0x1b/0x20 [ 2376.893741] ? handle_bug+0x44/0x70 [ 2376.897231] ? exc_invalid_op+0x17/0x60 [ 2376.901070] ? asm_exc_invalid_op+0x1a/0x20 [ 2376.905263] ? __list_add_valid_or_report+0x7d/0xa0 [ 2376.910138] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2376.915028] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2376.920262] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2376.925064] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2376.929778] ? __d_lookup+0xd6/0x200 [ 2376.933356] lookup_fast+0x6d/0xf0 [ 2376.936763] path_openat+0xe4/0x9d0 [ 2376.940258] do_filp_open+0xb8/0x120 [ 2376.943838] ? lock_release+0xc5/0x260 [ 2376.947587] ? _raw_spin_unlock+0x23/0x40 [ 2376.951599] ? alloc_fd+0x13f/0x200 [ 2376.955094] do_sys_openat2+0x249/0x290 [ 2376.958933] do_sys_open+0x4f/0x80 [ 2376.962339] do_syscall_64+0x38/0x80 [ 2376.965917] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2376.970970] RIP: 0033:0x7f48f0ef2805 [ 2376.974549] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2376.993295] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2377.000861] RAX: ffffffffffffffda RBX: 000000000207dfdf RCX: 00007f48f0ef2805 [ 2377.007993] RDX: 0000000000090800 RSI: 000000000207dfc0 RDI: 00000000ffffff9c [ 2377.015125] RBP: 000000000207dfc0 R08: 0000000002076674 R09: 0000000000000000 [ 2377.022252] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2377.029384] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2377.036521] [ 2377.038707] irq event stamp: 12961591 [ 2377.042366] hardirqs last enabled at (12961601): [] console_unlock+0x106/0x120 [ 2377.051233] hardirqs last disabled at (12961610): [] console_unlock+0xeb/0x120 [ 2377.060012] softirqs last enabled at (12961558): [] __do_softirq+0x2be/0x371 [ 2377.068704] softirqs last disabled at (12961553): [] irq_exit_rcu+0xb0/0xd0 [ 2377.077224] ---[ end trace 0000000000000000 ]--- [ 2377.081844] ------------[ cut here ]------------ [ 2377.086463] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2377.098110] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2377.106805] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2377.162316] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2377.171460] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2377.178959] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2377.184477] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2377.189980] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2377.208745] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2377.213988] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2377.221138] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2377.228273] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009b98 [ 2377.235404] R10: 000000000031cde8 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2377.242553] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2377.249695] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2377.257797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2377.263552] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2377.270694] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2377.277836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2377.284986] Call Trace: [ 2377.287446] [ 2377.289564] ? __warn+0x85/0x180 [ 2377.292815] ? __list_add_valid_or_report+0x7d/0xa0 [ 2377.297709] ? report_bug+0xfc/0x1d0 [ 2377.301323] ? prb_read_valid+0x1b/0x20 [ 2377.305166] ? handle_bug+0x44/0x70 [ 2377.308656] ? exc_invalid_op+0x17/0x60 [ 2377.312513] ? asm_exc_invalid_op+0x1a/0x20 [ 2377.316738] ? __list_add_valid_or_report+0x7d/0xa0 [ 2377.321613] ? __list_add_valid_or_report+0x7d/0xa0 [ 2377.326509] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2377.331416] ceph_add_cap+0x20b/0x620 [ceph] [ 2377.335737] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2377.340476] ? lock_acquire+0xcb/0x2b0 [ 2377.344229] ceph_fill_trace+0x143/0xf30 [ceph] [ 2377.348766] ? __mutex_lock+0x97/0xce0 [ 2377.352535] ? __mutex_lock+0xaec/0xce0 [ 2377.356395] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2377.360952] ? lock_downgrade+0xfd/0x250 [ 2377.364900] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2377.369281] ceph_con_process_message+0x74/0x150 [libceph] [ 2377.374775] ? calc_signature+0xdb/0x100 [libceph] [ 2377.379614] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2377.384961] ? process_scheduled_works+0x22a/0x510 [ 2377.389769] ? process_scheduled_works+0x22a/0x510 [ 2377.394561] ceph_con_workfn+0x344/0x750 [libceph] [ 2377.399381] ? process_scheduled_works+0x22a/0x510 [ 2377.404190] process_scheduled_works+0x2a4/0x510 [ 2377.408830] worker_thread+0x188/0x340 [ 2377.412580] ? rescuer_thread+0x310/0x310 [ 2377.416610] kthread+0xf5/0x130 [ 2377.419774] ? kthread_complete_and_exit+0x20/0x20 [ 2377.424585] ret_from_fork+0x31/0x40 [ 2377.428179] ? kthread_complete_and_exit+0x20/0x20 [ 2377.432974] ret_from_fork_asm+0x11/0x20 [ 2377.436942] [ 2377.439136] irq event stamp: 10805281 [ 2377.442801] hardirqs last enabled at (10805291): [] console_unlock+0x106/0x120 [ 2377.451684] hardirqs last disabled at (10805300): [] console_unlock+0xeb/0x120 [ 2377.460481] softirqs last enabled at (10805268): [] __do_softirq+0x2be/0x371 [ 2377.469190] softirqs last disabled at (10805263): [] irq_exit_rcu+0xb0/0xd0 [ 2377.477710] ---[ end trace 0000000000000000 ]--- [ 2377.482519] ------------[ cut here ]------------ [ 2377.487144] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2377.498794] WARNING: CPU: 0 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2377.507754] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2377.563222] CPU: 0 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2377.572181] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2377.579660] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2377.585146] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2377.603892] RSP: 0018:ffffc90001fcfc10 EFLAGS: 00010286 [ 2377.609119] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2377.616250] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2377.623384] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009bd7 [ 2377.630518] R10: 000000000031df00 R11: ffffc90001fcfaa0 R12: ffff8881d106b838 [ 2377.637650] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2377.644781] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2377.652870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2377.658613] CR2: 00007f9bd19a7000 CR3: 000000011aeae003 CR4: 00000000003706f0 [ 2377.665749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2377.672879] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2377.680013] Call Trace: [ 2377.682465] [ 2377.684576] ? __warn+0x85/0x180 [ 2377.687815] ? __list_add_valid_or_report+0x7d/0xa0 [ 2377.692697] ? report_bug+0xfc/0x1d0 [ 2377.696283] ? prb_read_valid+0x1b/0x20 [ 2377.700124] ? handle_bug+0x44/0x70 [ 2377.703621] ? exc_invalid_op+0x17/0x60 [ 2377.707464] ? asm_exc_invalid_op+0x1a/0x20 [ 2377.711663] ? __list_add_valid_or_report+0x7d/0xa0 [ 2377.716546] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2377.721444] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2377.726701] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2377.731515] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2377.736239] ? __d_lookup+0xd6/0x200 [ 2377.739824] lookup_fast+0x6d/0xf0 [ 2377.743244] path_openat+0xe4/0x9d0 [ 2377.746757] do_filp_open+0xb8/0x120 [ 2377.750343] ? lock_release+0xc5/0x260 [ 2377.754101] ? _raw_spin_unlock+0x23/0x40 [ 2377.758118] ? alloc_fd+0x13f/0x200 [ 2377.761619] do_sys_openat2+0x249/0x290 [ 2377.765463] do_sys_open+0x4f/0x80 [ 2377.768876] do_syscall_64+0x38/0x80 [ 2377.772455] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2377.777505] RIP: 0033:0x7f48f0ef2805 [ 2377.781088] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2377.799832] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2377.807399] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2377.814530] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2377.821664] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2377.828798] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2377.835928] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2377.843074] [ 2377.845265] irq event stamp: 12926955 [ 2377.848929] hardirqs last enabled at (12926965): [] console_unlock+0x106/0x120 [ 2377.857797] hardirqs last disabled at (12926974): [] console_unlock+0xeb/0x120 [ 2377.866575] softirqs last enabled at (12926922): [] __do_softirq+0x2be/0x371 [ 2377.875269] softirqs last disabled at (12926917): [] irq_exit_rcu+0xb0/0xd0 [ 2377.883787] ---[ end trace 0000000000000000 ]--- [ 2377.888408] ------------[ cut here ]------------ [ 2377.893027] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2377.904704] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2377.913400] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2377.968895] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2377.978055] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2377.985534] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2377.991058] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2377.996540] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2378.015306] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2378.020532] RAX: 0000000000000000 RBX: ffff8881d1221d58 RCX: 0000000000000000 [ 2378.027682] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2378.034832] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009c12 [ 2378.041982] R10: 000000000031ef78 R11: ffffc9000030b828 R12: ffff8881d12224a8 [ 2378.049131] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2378.056264] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2378.064352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2378.070114] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2378.077248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2378.084380] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2378.091531] Call Trace: [ 2378.094000] [ 2378.096115] ? __warn+0x85/0x180 [ 2378.099349] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.104245] ? report_bug+0xfc/0x1d0 [ 2378.107822] ? prb_read_valid+0x1b/0x20 [ 2378.111681] ? handle_bug+0x44/0x70 [ 2378.115190] ? exc_invalid_op+0x17/0x60 [ 2378.119051] ? asm_exc_invalid_op+0x1a/0x20 [ 2378.123238] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.128113] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.132992] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2378.137901] ceph_add_cap+0x20b/0x620 [ceph] [ 2378.142202] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2378.146942] ? lock_acquire+0xcb/0x2b0 [ 2378.150710] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2378.155162] ? __mutex_lock+0x97/0xce0 [ 2378.158913] ? __mutex_lock+0xaec/0xce0 [ 2378.162765] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2378.167336] ? lock_downgrade+0xfd/0x250 [ 2378.171280] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2378.175673] ceph_con_process_message+0x74/0x150 [libceph] [ 2378.181171] ? calc_signature+0xdb/0x100 [libceph] [ 2378.185999] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2378.191323] ? process_scheduled_works+0x22a/0x510 [ 2378.196114] ? process_scheduled_works+0x22a/0x510 [ 2378.200906] ceph_con_workfn+0x344/0x750 [libceph] [ 2378.205725] ? process_scheduled_works+0x22a/0x510 [ 2378.210535] process_scheduled_works+0x2a4/0x510 [ 2378.215176] worker_thread+0x188/0x340 [ 2378.218934] ? rescuer_thread+0x310/0x310 [ 2378.222963] kthread+0xf5/0x130 [ 2378.226127] ? kthread_complete_and_exit+0x20/0x20 [ 2378.230920] ret_from_fork+0x31/0x40 [ 2378.234516] ? kthread_complete_and_exit+0x20/0x20 [ 2378.239341] ret_from_fork_asm+0x11/0x20 [ 2378.243293] [ 2378.245487] irq event stamp: 10807245 [ 2378.249161] hardirqs last enabled at (10807255): [] console_unlock+0x106/0x120 [ 2378.258029] hardirqs last disabled at (10807264): [] console_unlock+0xeb/0x120 [ 2378.266824] softirqs last enabled at (10807192): [] __do_softirq+0x2be/0x371 [ 2378.275517] softirqs last disabled at (10807187): [] irq_exit_rcu+0xb0/0xd0 [ 2378.284062] ---[ end trace 0000000000000000 ]--- [ 2378.288710] ------------[ cut here ]------------ [ 2378.293342] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2378.305015] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2378.313704] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2378.369199] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2378.378339] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2378.385829] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2378.391346] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2378.396844] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2378.415609] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2378.420852] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2378.428001] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2378.435146] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009c51 [ 2378.442275] R10: 00000000003200b8 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2378.449409] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2378.456558] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2378.464663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2378.470426] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2378.477578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2378.484729] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2378.491875] Call Trace: [ 2378.494346] [ 2378.496463] ? __warn+0x85/0x180 [ 2378.499714] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.504608] ? report_bug+0xfc/0x1d0 [ 2378.508204] ? prb_read_valid+0x1b/0x20 [ 2378.512045] ? handle_bug+0x44/0x70 [ 2378.515536] ? exc_invalid_op+0x17/0x60 [ 2378.519386] ? asm_exc_invalid_op+0x1a/0x20 [ 2378.523610] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.528486] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.533382] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2378.538289] ceph_add_cap+0x20b/0x620 [ceph] [ 2378.542591] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2378.547312] ceph_fill_trace+0x143/0xf30 [ceph] [ 2378.551853] ? __mutex_lock+0x97/0xce0 [ 2378.555621] ? __mutex_lock+0xaec/0xce0 [ 2378.559479] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2378.564063] ? lock_downgrade+0xfd/0x250 [ 2378.568011] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2378.572407] ceph_con_process_message+0x74/0x150 [libceph] [ 2378.577905] ? calc_signature+0xdb/0x100 [libceph] [ 2378.582734] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2378.588083] ? process_scheduled_works+0x22a/0x510 [ 2378.592875] ? process_scheduled_works+0x22a/0x510 [ 2378.597680] ceph_con_workfn+0x344/0x750 [libceph] [ 2378.602503] ? process_scheduled_works+0x22a/0x510 [ 2378.607309] process_scheduled_works+0x2a4/0x510 [ 2378.611951] worker_thread+0x188/0x340 [ 2378.615701] ? rescuer_thread+0x310/0x310 [ 2378.619731] kthread+0xf5/0x130 [ 2378.622893] ? kthread_complete_and_exit+0x20/0x20 [ 2378.627704] ret_from_fork+0x31/0x40 [ 2378.631299] ? kthread_complete_and_exit+0x20/0x20 [ 2378.636092] ret_from_fork_asm+0x11/0x20 [ 2378.640051] [ 2378.642235] irq event stamp: 10808307 [ 2378.645895] hardirqs last enabled at (10808317): [] console_unlock+0x106/0x120 [ 2378.654778] hardirqs last disabled at (10808326): [] console_unlock+0xeb/0x120 [ 2378.663577] softirqs last enabled at (10808342): [] __do_softirq+0x2be/0x371 [ 2378.672287] softirqs last disabled at (10808353): [] irq_exit_rcu+0xb0/0xd0 [ 2378.680806] ---[ end trace 0000000000000000 ]--- [ 2378.685478] ------------[ cut here ]------------ [ 2378.690099] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2378.701756] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2378.710445] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2378.765940] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2378.775081] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2378.782561] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2378.788075] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2378.793559] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2378.812325] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2378.817549] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2378.824693] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2378.831842] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009c8f [ 2378.838991] R10: 0000000000321180 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2378.846134] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2378.853266] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2378.861351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2378.867106] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2378.874240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2378.881372] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2378.888523] Call Trace: [ 2378.890993] [ 2378.893115] ? __warn+0x85/0x180 [ 2378.896352] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.901245] ? report_bug+0xfc/0x1d0 [ 2378.904824] ? prb_read_valid+0x1b/0x20 [ 2378.908684] ? handle_bug+0x44/0x70 [ 2378.912192] ? exc_invalid_op+0x17/0x60 [ 2378.916030] ? asm_exc_invalid_op+0x1a/0x20 [ 2378.920239] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.925114] ? __list_add_valid_or_report+0x7d/0xa0 [ 2378.929993] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2378.934900] ceph_add_cap+0x20b/0x620 [ceph] [ 2378.939204] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2378.943943] ? lock_acquire+0xcb/0x2b0 [ 2378.947712] ceph_fill_trace+0x143/0xf30 [ceph] [ 2378.952250] ? __mutex_lock+0x97/0xce0 [ 2378.956001] ? __mutex_lock+0xaec/0xce0 [ 2378.959869] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2378.964410] ? lock_downgrade+0xfd/0x250 [ 2378.968384] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2378.972779] ceph_con_process_message+0x74/0x150 [libceph] [ 2378.978276] ? calc_signature+0xdb/0x100 [libceph] [ 2378.983079] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2378.988427] ? process_scheduled_works+0x22a/0x510 [ 2378.993220] ? process_scheduled_works+0x22a/0x510 [ 2378.998011] ceph_con_workfn+0x344/0x750 [libceph] [ 2379.002838] ? process_scheduled_works+0x22a/0x510 [ 2379.007629] process_scheduled_works+0x2a4/0x510 [ 2379.012272] worker_thread+0x188/0x340 [ 2379.016022] ? rescuer_thread+0x310/0x310 [ 2379.020077] kthread+0xf5/0x130 [ 2379.023225] ? kthread_complete_and_exit+0x20/0x20 [ 2379.028015] ret_from_fork+0x31/0x40 [ 2379.031612] ? kthread_complete_and_exit+0x20/0x20 [ 2379.036421] ret_from_fork_asm+0x11/0x20 [ 2379.040388] [ 2379.042575] irq event stamp: 10809445 [ 2379.046251] hardirqs last enabled at (10809455): [] console_unlock+0x106/0x120 [ 2379.055116] hardirqs last disabled at (10809464): [] console_unlock+0xeb/0x120 [ 2379.063895] softirqs last enabled at (10809392): [] __do_softirq+0x2be/0x371 [ 2379.072606] softirqs last disabled at (10809387): [] irq_exit_rcu+0xb0/0xd0 [ 2379.081143] ---[ end trace 0000000000000000 ]--- [ 2379.085794] ------------[ cut here ]------------ [ 2379.090409] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2379.102056] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2379.110748] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2379.166225] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2379.175351] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2379.182830] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2379.188324] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2379.193810] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2379.212559] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2379.217784] RAX: 0000000000000000 RBX: ffff8881c5350000 RCX: 0000000000000000 [ 2379.224918] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2379.232055] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009cce [ 2379.239182] R10: 00000000003222a8 R11: ffffc9000030b828 R12: ffff8881c5350750 [ 2379.246315] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2379.253449] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2379.261535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2379.267280] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2379.274414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2379.281548] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2379.288679] Call Trace: [ 2379.291131] [ 2379.293239] ? __warn+0x85/0x180 [ 2379.296471] ? __list_add_valid_or_report+0x7d/0xa0 [ 2379.301350] ? report_bug+0xfc/0x1d0 [ 2379.304930] ? prb_read_valid+0x1b/0x20 [ 2379.308770] ? handle_bug+0x44/0x70 [ 2379.312262] ? exc_invalid_op+0x17/0x60 [ 2379.316100] ? asm_exc_invalid_op+0x1a/0x20 [ 2379.320292] ? __list_add_valid_or_report+0x7d/0xa0 [ 2379.325174] ? __list_add_valid_or_report+0x7d/0xa0 [ 2379.330055] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2379.334944] ceph_add_cap+0x20b/0x620 [ceph] [ 2379.339231] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2379.343954] ? lock_acquire+0xcb/0x2b0 [ 2379.347712] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2379.352164] ? __mutex_lock+0x97/0xce0 [ 2379.355915] ? __mutex_lock+0xaec/0xce0 [ 2379.359758] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2379.364296] ? lock_downgrade+0xfd/0x250 [ 2379.368228] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2379.372608] ceph_con_process_message+0x74/0x150 [libceph] [ 2379.378101] ? calc_signature+0xdb/0x100 [libceph] [ 2379.382906] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2379.388230] ? process_scheduled_works+0x22a/0x510 [ 2379.393022] ? process_scheduled_works+0x22a/0x510 [ 2379.397810] ceph_con_workfn+0x344/0x750 [libceph] [ 2379.402616] ? process_scheduled_works+0x22a/0x510 [ 2379.407404] process_scheduled_works+0x2a4/0x510 [ 2379.412030] worker_thread+0x188/0x340 [ 2379.415788] ? rescuer_thread+0x310/0x310 [ 2379.419799] kthread+0xf5/0x130 [ 2379.422948] ? kthread_complete_and_exit+0x20/0x20 [ 2379.427739] ret_from_fork+0x31/0x40 [ 2379.431316] ? kthread_complete_and_exit+0x20/0x20 [ 2379.436110] ret_from_fork_asm+0x11/0x20 [ 2379.440043] [ 2379.442228] irq event stamp: 10810557 [ 2379.445896] hardirqs last enabled at (10810567): [] console_unlock+0x106/0x120 [ 2379.454761] hardirqs last disabled at (10810576): [] console_unlock+0xeb/0x120 [ 2379.463541] softirqs last enabled at (10810504): [] __do_softirq+0x2be/0x371 [ 2379.472232] softirqs last disabled at (10810499): [] irq_exit_rcu+0xb0/0xd0 [ 2379.480752] ---[ end trace 0000000000000000 ]--- [ 2379.485373] ------------[ cut here ]------------ [ 2379.489991] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2379.501639] WARNING: CPU: 2 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2379.510592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2379.566057] CPU: 2 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2379.575004] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2379.582481] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2379.587970] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2379.606714] RSP: 0018:ffffc90001ff7b78 EFLAGS: 00010286 [ 2379.611941] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2379.619064] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2379.626192] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009d0d [ 2379.633322] R10: 0000000000323498 R11: ffffc90001ff7a08 R12: ffff8881c25d24a8 [ 2379.640446] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2379.647580] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2379.655665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2379.661412] CR2: 00007fb38c57d000 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2379.668543] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2379.675667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2379.682793] Call Trace: [ 2379.685245] [ 2379.687343] ? __warn+0x85/0x180 [ 2379.690576] ? __list_add_valid_or_report+0x7d/0xa0 [ 2379.695455] ? report_bug+0xfc/0x1d0 [ 2379.699035] ? prb_read_valid+0x1b/0x20 [ 2379.702874] ? handle_bug+0x44/0x70 [ 2379.706367] ? exc_invalid_op+0x17/0x60 [ 2379.710207] ? asm_exc_invalid_op+0x1a/0x20 [ 2379.714395] ? __list_add_valid_or_report+0x7d/0xa0 [ 2379.719271] ? __list_add_valid_or_report+0x7d/0xa0 [ 2379.724150] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2379.729041] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2379.734282] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2379.739086] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2379.743801] ? __d_lookup+0xd6/0x200 [ 2379.747378] lookup_fast+0x6d/0xf0 [ 2379.750785] walk_component+0x26/0x160 [ 2379.754539] link_path_walk+0x24d/0x380 [ 2379.758376] ? path_openat+0x80/0x9d0 [ 2379.762046] path_openat+0x8e/0x9d0 [ 2379.765537] do_filp_open+0xb8/0x120 [ 2379.769119] ? lock_release+0xc5/0x260 [ 2379.772868] ? _raw_spin_unlock+0x23/0x40 [ 2379.776879] ? alloc_fd+0x13f/0x200 [ 2379.780376] do_sys_openat2+0x249/0x290 [ 2379.784214] do_sys_open+0x4f/0x80 [ 2379.787619] do_syscall_64+0x38/0x80 [ 2379.791198] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2379.796250] RIP: 0033:0x7f48f0eed2f2 [ 2379.799831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2379.818575] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2379.826142] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2379.833274] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2379.840397] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2379.847523] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2379.854655] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2379.861785] [ 2379.863971] irq event stamp: 12972619 [ 2379.867630] hardirqs last enabled at (12972629): [] console_unlock+0x106/0x120 [ 2379.876496] hardirqs last disabled at (12972638): [] console_unlock+0xeb/0x120 [ 2379.885274] softirqs last enabled at (12972556): [] __do_softirq+0x2be/0x371 [ 2379.893969] softirqs last disabled at (12972657): [] irq_exit_rcu+0xb0/0xd0 [ 2379.902487] ---[ end trace 0000000000000000 ]--- [ 2379.907098] ------------[ cut here ]------------ [ 2379.911718] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2379.923398] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2379.932093] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2379.987591] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2379.993060] rcu: INFO: rcu_preempt self-detected stall on CPU [ 2379.996715] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2380.002466] rcu: 7-....: (64372 ticks this GP) idle=6464/1/0x4000000000000000 softirq=393959/398742 fqs=12901 [ 2380.009958] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2380.020025] rcu: (t=65023 jiffies g=1213253 q=59909 ncpus=8) [ 2380.020027] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2380.020029] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2380.020031] Workqueue: events delayed_work [ceph] [ 2380.025532] [ 2380.031263] [ 2380.031264] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [ 2380.031268] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [ 2380.031270] RSP: 0018:ffffc90002117d80 EFLAGS: 00000202 [ 2380.040656] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2380.048122] [ 2380.048122] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 2380.048123] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [ 2380.048125] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 2380.048126] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2380.048127] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2380.048128] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2380.048129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2380.048131] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2380.048132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2380.052832] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2380.054318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2380.054320] Call Trace: [ 2380.054321] [ 2380.054322] ? rcu_dump_cpu_stacks+0xd1/0x180 [ 2380.055823] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2380.061295] ? rcu_sched_clock_irq+0x633/0x1390 [ 2380.061300] ? lock_is_held_type+0xaa/0x120 [ 2380.061304] ? lock_is_held_type+0xaa/0x120 [ 2380.080047] [ 2380.085260] ? tick_sched_do_timer+0xa0/0xa0 [ 2380.090750] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2380.092237] ? rcu_read_lock_sched_held+0x56/0x90 [ 2380.092243] ? tick_sched_do_timer+0xa0/0xa0 [ 2380.099376] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2380.106494] ? update_process_times+0x5f/0x90 [ 2380.106498] ? tick_sched_handle.isra.16+0x1e/0x50 [ 2380.106501] ? tick_sched_timer+0x6f/0x80 [ 2380.113648] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009d4c [ 2380.120768] ? __hrtimer_run_queues+0x1d3/0x360 [ 2380.120774] ? hrtimer_interrupt+0x104/0x240 [ 2380.127925] R10: 0000000000324760 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2380.135995] ? __local_bh_enable+0x50/0x60 [ 2380.136001] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2380.141762] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2380.148882] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2380.148885] [ 2380.148886] [ 2380.148887] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2380.156037] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2380.174775] ? queued_spin_lock_slowpath+0x18/0x2e0 [ 2380.174797] do_raw_spin_lock+0xb5/0xc0 [ 2380.181926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2380.184371] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2380.186405] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2380.190754] delayed_work+0x1d8/0x280 [ceph] [ 2380.196001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2380.200521] ? process_scheduled_works+0x22a/0x510 [ 2380.200524] process_scheduled_works+0x2a4/0x510 [ 2380.200548] worker_thread+0x188/0x340 [ 2380.204727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2380.208901] ? rescuer_thread+0x310/0x310 [ 2380.208904] kthread+0xf5/0x130 [ 2380.208907] ? kthread_complete_and_exit+0x20/0x20 [ 2380.210422] Call Trace: [ 2380.214681] ret_from_fork+0x31/0x40 [ 2380.214684] ? kthread_complete_and_exit+0x20/0x20 [ 2380.214687] ret_from_fork_asm+0x11/0x20 [ 2380.221838] [ 2380.226535] [ 2380.402485] ? __warn+0x85/0x180 [ 2380.405721] ? __list_add_valid_or_report+0x7d/0xa0 [ 2380.410617] ? report_bug+0xfc/0x1d0 [ 2380.414213] ? prb_read_valid+0x1b/0x20 [ 2380.418055] ? handle_bug+0x44/0x70 [ 2380.421562] ? exc_invalid_op+0x17/0x60 [ 2380.425401] ? asm_exc_invalid_op+0x1a/0x20 [ 2380.429626] ? __list_add_valid_or_report+0x7d/0xa0 [ 2380.434504] ? __list_add_valid_or_report+0x7d/0xa0 [ 2380.439397] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2380.444307] ceph_add_cap+0x20b/0x620 [ceph] [ 2380.448610] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2380.453338] ceph_fill_trace+0x143/0xf30 [ceph] [ 2380.457878] ? __mutex_lock+0x97/0xce0 [ 2380.461646] ? __mutex_lock+0xaec/0xce0 [ 2380.465503] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2380.470064] ? lock_downgrade+0xfd/0x250 [ 2380.474008] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2380.478405] ceph_con_process_message+0x74/0x150 [libceph] [ 2380.483903] ? calc_signature+0xdb/0x100 [libceph] [ 2380.488741] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2380.494097] ? process_scheduled_works+0x22a/0x510 [ 2380.498889] ? process_scheduled_works+0x22a/0x510 [ 2380.503680] ceph_con_workfn+0x344/0x750 [libceph] [ 2380.508503] ? process_scheduled_works+0x22a/0x510 [ 2380.513310] process_scheduled_works+0x2a4/0x510 [ 2380.517950] worker_thread+0x188/0x340 [ 2380.521699] ? rescuer_thread+0x310/0x310 [ 2380.525728] kthread+0xf5/0x130 [ 2380.528893] ? kthread_complete_and_exit+0x20/0x20 [ 2380.533703] ret_from_fork+0x31/0x40 [ 2380.537300] ? kthread_complete_and_exit+0x20/0x20 [ 2380.542103] ret_from_fork_asm+0x11/0x20 [ 2380.546050] [ 2380.548238] irq event stamp: 10812747 [ 2380.551903] hardirqs last enabled at (10812757): [] console_unlock+0x106/0x120 [ 2380.560788] hardirqs last disabled at (10812772): [] console_unlock+0xeb/0x120 [ 2380.569584] softirqs last enabled at (10812770): [] __do_softirq+0x2be/0x371 [ 2380.578296] softirqs last disabled at (10812791): [] irq_exit_rcu+0xb0/0xd0 [ 2380.586813] ---[ end trace 0000000000000000 ]--- [ 2380.591618] ------------[ cut here ]------------ [ 2380.596235] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2380.607894] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2380.616592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2380.672105] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2380.681231] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2380.688709] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2380.694221] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2380.699706] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2380.718456] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2380.723697] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2380.730846] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2380.737997] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009dc3 [ 2380.745148] R10: 0000000000326790 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2380.752279] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2380.759412] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2380.767516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2380.773278] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2380.780403] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2380.787544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2380.794686] Call Trace: [ 2380.797147] [ 2380.799254] ? __warn+0x85/0x180 [ 2380.802489] ? __list_add_valid_or_report+0x7d/0xa0 [ 2380.807396] ? report_bug+0xfc/0x1d0 [ 2380.810988] ? prb_read_valid+0x1b/0x20 [ 2380.814847] ? handle_bug+0x44/0x70 [ 2380.818356] ? exc_invalid_op+0x17/0x60 [ 2380.822196] ? asm_exc_invalid_op+0x1a/0x20 [ 2380.826404] ? __list_add_valid_or_report+0x7d/0xa0 [ 2380.831279] ? __list_add_valid_or_report+0x7d/0xa0 [ 2380.836157] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2380.841047] ceph_add_cap+0x20b/0x620 [ceph] [ 2380.845359] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2380.850092] ? lock_acquire+0xcb/0x2b0 [ 2380.853867] ceph_fill_trace+0x143/0xf30 [ceph] [ 2380.858406] ? __mutex_lock+0x97/0xce0 [ 2380.862175] ? __mutex_lock+0xaec/0xce0 [ 2380.866018] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2380.870582] ? lock_downgrade+0xfd/0x250 [ 2380.874548] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2380.878945] ceph_con_process_message+0x74/0x150 [libceph] [ 2380.884439] ? calc_signature+0xdb/0x100 [libceph] [ 2380.889262] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2380.894608] ? process_scheduled_works+0x22a/0x510 [ 2380.899418] ? process_scheduled_works+0x22a/0x510 [ 2380.904208] ceph_con_workfn+0x344/0x750 [libceph] [ 2380.909013] ? process_scheduled_works+0x22a/0x510 [ 2380.913820] process_scheduled_works+0x2a4/0x510 [ 2380.918479] worker_thread+0x188/0x340 [ 2380.922229] ? rescuer_thread+0x310/0x310 [ 2380.926240] kthread+0xf5/0x130 [ 2380.929396] ? kthread_complete_and_exit+0x20/0x20 [ 2380.934207] ret_from_fork+0x31/0x40 [ 2380.937784] ? kthread_complete_and_exit+0x20/0x20 [ 2380.942595] ret_from_fork_asm+0x11/0x20 [ 2380.946560] [ 2380.948747] irq event stamp: 10813889 [ 2380.952423] hardirqs last enabled at (10813899): [] console_unlock+0x106/0x120 [ 2380.961306] hardirqs last disabled at (10813908): [] console_unlock+0xeb/0x120 [ 2380.970088] softirqs last enabled at (10813836): [] __do_softirq+0x2be/0x371 [ 2380.978778] softirqs last disabled at (10813831): [] irq_exit_rcu+0xb0/0xd0 [ 2380.987315] ---[ end trace 0000000000000000 ]--- [ 2380.992002] ------------[ cut here ]------------ [ 2380.996627] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2381.008271] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2381.017225] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2381.072693] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2381.081644] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2381.089116] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2381.094602] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2381.113347] RSP: 0018:ffffc90001fdfab8 EFLAGS: 00010286 [ 2381.118574] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2381.125706] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2381.132829] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009e02 [ 2381.139954] R10: 00000000003278a8 R11: ffffc90001fdf948 R12: ffff8881d11624a8 [ 2381.147087] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2381.154211] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2381.162298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2381.168043] CR2: 00007f9bd22d7020 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2381.175167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2381.182291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2381.189415] Call Trace: [ 2381.191861] [ 2381.193967] ? __warn+0x85/0x180 [ 2381.197200] ? __list_add_valid_or_report+0x7d/0xa0 [ 2381.202080] ? report_bug+0xfc/0x1d0 [ 2381.205667] ? prb_read_valid+0x1b/0x20 [ 2381.209507] ? handle_bug+0x44/0x70 [ 2381.212998] ? exc_invalid_op+0x17/0x60 [ 2381.216829] ? asm_exc_invalid_op+0x1a/0x20 [ 2381.221020] ? __list_add_valid_or_report+0x7d/0xa0 [ 2381.225895] ? __list_add_valid_or_report+0x7d/0xa0 [ 2381.230774] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2381.235665] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2381.240901] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2381.245700] ceph_atomic_open+0x28d/0xe80 [ceph] [ 2381.250336] lookup_open+0x37b/0x630 [ 2381.253920] ? verify_cpu+0x20/0x100 [ 2381.257496] ? verify_cpu+0x20/0x100 [ 2381.261077] path_openat+0x25a/0x9d0 [ 2381.264656] do_filp_open+0xb8/0x120 [ 2381.268237] ? lock_release+0xc5/0x260 [ 2381.271988] ? _raw_spin_unlock+0x23/0x40 [ 2381.275997] ? alloc_fd+0x13f/0x200 [ 2381.279494] do_sys_openat2+0x249/0x290 [ 2381.283331] do_sys_open+0x4f/0x80 [ 2381.286736] do_syscall_64+0x38/0x80 [ 2381.290315] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2381.295368] RIP: 0033:0x7f48f0eed2f2 [ 2381.298949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2381.317694] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2381.325261] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2381.332394] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2381.339526] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2381.346657] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2381.353790] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2381.360921] [ 2381.363109] irq event stamp: 12878587 [ 2381.366773] hardirqs last enabled at (12878597): [] console_unlock+0x106/0x120 [ 2381.375641] hardirqs last disabled at (12878606): [] console_unlock+0xeb/0x120 [ 2381.384420] softirqs last enabled at (12878534): [] __do_softirq+0x2be/0x371 [ 2381.393112] softirqs last disabled at (12878529): [] irq_exit_rcu+0xb0/0xd0 [ 2381.401633] ---[ end trace 0000000000000000 ]--- [ 2381.406252] ------------[ cut here ]------------ [ 2381.410870] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2381.422535] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2381.431230] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2381.486709] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2381.495850] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2381.503331] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2381.508836] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2381.514335] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2381.533082] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2381.538327] RAX: 0000000000000000 RBX: ffff8881d0bebab0 RCX: 0000000000000000 [ 2381.545458] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2381.552602] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009e3f [ 2381.559752] R10: 0000000000328930 R11: ffffc9000030b828 R12: ffff8881d0bec200 [ 2381.566901] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2381.574051] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2381.582154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2381.587900] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2381.595041] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2381.602184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2381.609316] Call Trace: [ 2381.611768] [ 2381.613883] ? __warn+0x85/0x180 [ 2381.617137] ? __list_add_valid_or_report+0x7d/0xa0 [ 2381.622013] ? report_bug+0xfc/0x1d0 [ 2381.625610] ? prb_read_valid+0x1b/0x20 [ 2381.629468] ? handle_bug+0x44/0x70 [ 2381.632976] ? exc_invalid_op+0x17/0x60 [ 2381.636834] ? asm_exc_invalid_op+0x1a/0x20 [ 2381.641058] ? __list_add_valid_or_report+0x7d/0xa0 [ 2381.645934] ? __list_add_valid_or_report+0x7d/0xa0 [ 2381.650831] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2381.655738] ceph_add_cap+0x20b/0x620 [ceph] [ 2381.660057] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2381.664789] ? lock_acquire+0xcb/0x2b0 [ 2381.668559] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2381.673009] ? __mutex_lock+0x97/0xce0 [ 2381.676779] ? __mutex_lock+0xaec/0xce0 [ 2381.680656] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2381.685204] ? lock_downgrade+0xfd/0x250 [ 2381.689133] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2381.693532] ceph_con_process_message+0x74/0x150 [libceph] [ 2381.699026] ? calc_signature+0xdb/0x100 [libceph] [ 2381.703866] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2381.709197] ? process_scheduled_works+0x22a/0x510 [ 2381.714006] ? process_scheduled_works+0x22a/0x510 [ 2381.718795] ceph_con_workfn+0x344/0x750 [libceph] [ 2381.723616] ? process_scheduled_works+0x22a/0x510 [ 2381.728423] process_scheduled_works+0x2a4/0x510 [ 2381.733082] worker_thread+0x188/0x340 [ 2381.736832] ? rescuer_thread+0x310/0x310 [ 2381.740862] kthread+0xf5/0x130 [ 2381.744026] ? kthread_complete_and_exit+0x20/0x20 [ 2381.748836] ret_from_fork+0x31/0x40 [ 2381.752432] ? kthread_complete_and_exit+0x20/0x20 [ 2381.757243] ret_from_fork_asm+0x11/0x20 [ 2381.761175] [ 2381.763369] irq event stamp: 10816681 [ 2381.767034] hardirqs last enabled at (10816691): [] console_unlock+0x106/0x120 [ 2381.775919] hardirqs last disabled at (10816700): [] console_unlock+0xeb/0x120 [ 2381.784716] softirqs last enabled at (10816628): [] __do_softirq+0x2be/0x371 [ 2381.793425] softirqs last disabled at (10816623): [] irq_exit_rcu+0xb0/0xd0 [ 2381.801963] ---[ end trace 0000000000000000 ]--- [ 2381.806629] ------------[ cut here ]------------ [ 2381.811245] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2381.822905] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2381.831594] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2381.887090] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2381.896222] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2381.903702] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2381.909215] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2381.914700] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2381.933464] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2381.938706] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2381.945848] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2381.952983] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009e7e [ 2381.960132] R10: 0000000000329b18 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2381.967265] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2381.974408] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2381.982511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2381.988282] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2381.995415] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2382.002564] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2382.009715] Call Trace: [ 2382.012185] [ 2382.014293] ? __warn+0x85/0x180 [ 2382.017526] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.022422] ? report_bug+0xfc/0x1d0 [ 2382.026017] ? prb_read_valid+0x1b/0x20 [ 2382.029875] ? handle_bug+0x44/0x70 [ 2382.033383] ? exc_invalid_op+0x17/0x60 [ 2382.037225] ? asm_exc_invalid_op+0x1a/0x20 [ 2382.041431] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.046308] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.051186] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2382.056075] ceph_add_cap+0x20b/0x620 [ceph] [ 2382.060402] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2382.065142] ceph_fill_trace+0x143/0xf30 [ceph] [ 2382.069681] ? __mutex_lock+0x97/0xce0 [ 2382.073450] ? __mutex_lock+0xaec/0xce0 [ 2382.077309] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2382.081850] ? lock_downgrade+0xfd/0x250 [ 2382.085813] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2382.090185] ceph_con_process_message+0x74/0x150 [libceph] [ 2382.095681] ? calc_signature+0xdb/0x100 [libceph] [ 2382.100519] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2382.105868] ? process_scheduled_works+0x22a/0x510 [ 2382.110676] ? process_scheduled_works+0x22a/0x510 [ 2382.115467] ceph_con_workfn+0x344/0x750 [libceph] [ 2382.120289] ? process_scheduled_works+0x22a/0x510 [ 2382.125079] process_scheduled_works+0x2a4/0x510 [ 2382.129738] worker_thread+0x188/0x340 [ 2382.133486] ? rescuer_thread+0x310/0x310 [ 2382.137517] kthread+0xf5/0x130 [ 2382.140680] ? kthread_complete_and_exit+0x20/0x20 [ 2382.145491] ret_from_fork+0x31/0x40 [ 2382.149085] ? kthread_complete_and_exit+0x20/0x20 [ 2382.153897] ret_from_fork_asm+0x11/0x20 [ 2382.157864] [ 2382.160059] irq event stamp: 10817773 [ 2382.163741] hardirqs last enabled at (10817783): [] console_unlock+0x106/0x120 [ 2382.172626] hardirqs last disabled at (10817792): [] console_unlock+0xeb/0x120 [ 2382.181423] softirqs last enabled at (10817730): [] __do_softirq+0x2be/0x371 [ 2382.190133] softirqs last disabled at (10817725): [] irq_exit_rcu+0xb0/0xd0 [ 2382.198651] ---[ end trace 0000000000000000 ]--- [ 2382.203298] ------------[ cut here ]------------ [ 2382.207917] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2382.219596] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2382.228293] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2382.283770] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2382.292913] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2382.300422] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2382.305938] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2382.311441] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2382.330206] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2382.335430] RAX: 0000000000000000 RBX: ffff8881d106b0e8 RCX: 0000000000000000 [ 2382.342573] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2382.349724] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009ebc [ 2382.356874] R10: 000000000032ac08 R11: ffffc9000030b828 R12: ffff8881d106b838 [ 2382.364023] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2382.371174] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2382.379260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2382.385004] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2382.392147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2382.399279] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2382.406425] Call Trace: [ 2382.408890] [ 2382.411007] ? __warn+0x85/0x180 [ 2382.414257] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.419136] ? report_bug+0xfc/0x1d0 [ 2382.422714] ? prb_read_valid+0x1b/0x20 [ 2382.426574] ? handle_bug+0x44/0x70 [ 2382.430083] ? exc_invalid_op+0x17/0x60 [ 2382.433940] ? asm_exc_invalid_op+0x1a/0x20 [ 2382.438147] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.443023] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.447919] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2382.452826] ceph_add_cap+0x20b/0x620 [ceph] [ 2382.457142] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2382.461877] ? lock_acquire+0xcb/0x2b0 [ 2382.465646] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2382.470096] ? __mutex_lock+0x97/0xce0 [ 2382.473865] ? __mutex_lock+0xaec/0xce0 [ 2382.477724] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2382.482265] ? lock_downgrade+0xfd/0x250 [ 2382.486196] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2382.490593] ceph_con_process_message+0x74/0x150 [libceph] [ 2382.496090] ? calc_signature+0xdb/0x100 [libceph] [ 2382.500918] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2382.506242] ? process_scheduled_works+0x22a/0x510 [ 2382.511034] ? process_scheduled_works+0x22a/0x510 [ 2382.515839] ceph_con_workfn+0x344/0x750 [libceph] [ 2382.520662] ? process_scheduled_works+0x22a/0x510 [ 2382.525469] process_scheduled_works+0x2a4/0x510 [ 2382.530135] worker_thread+0x188/0x340 [ 2382.533885] ? rescuer_thread+0x310/0x310 [ 2382.537913] kthread+0xf5/0x130 [ 2382.541080] ? kthread_complete_and_exit+0x20/0x20 [ 2382.545888] ret_from_fork+0x31/0x40 [ 2382.549485] ? kthread_complete_and_exit+0x20/0x20 [ 2382.554296] ret_from_fork_asm+0x11/0x20 [ 2382.558228] [ 2382.560426] irq event stamp: 10818889 [ 2382.564104] hardirqs last enabled at (10818899): [] console_unlock+0x106/0x120 [ 2382.572987] hardirqs last disabled at (10818910): [] console_unlock+0xeb/0x120 [ 2382.581768] softirqs last enabled at (10818836): [] __do_softirq+0x2be/0x371 [ 2382.590480] softirqs last disabled at (10818831): [] irq_exit_rcu+0xb0/0xd0 [ 2382.599017] ---[ end trace 0000000000000000 ]--- [ 2382.603693] ------------[ cut here ]------------ [ 2382.608307] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2382.619966] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2382.628657] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2382.684150] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2382.693277] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2382.700756] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2382.706268] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2382.711753] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2382.730518] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2382.735761] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2382.742911] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2382.750063] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009efb [ 2382.757214] R10: 000000000032bb60 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2382.764347] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2382.771477] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2382.779582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2382.785343] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2382.792477] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2382.799626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2382.806777] Call Trace: [ 2382.809247] [ 2382.811354] ? __warn+0x85/0x180 [ 2382.814587] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.819483] ? report_bug+0xfc/0x1d0 [ 2382.823078] ? prb_read_valid+0x1b/0x20 [ 2382.826937] ? handle_bug+0x44/0x70 [ 2382.830447] ? exc_invalid_op+0x17/0x60 [ 2382.834302] ? asm_exc_invalid_op+0x1a/0x20 [ 2382.838511] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.843386] ? __list_add_valid_or_report+0x7d/0xa0 [ 2382.848264] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2382.853157] ceph_add_cap+0x20b/0x620 [ceph] [ 2382.857457] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2382.862172] ? lock_acquire+0xcb/0x2b0 [ 2382.865940] ceph_fill_trace+0x143/0xf30 [ceph] [ 2382.870479] ? __mutex_lock+0x97/0xce0 [ 2382.874249] ? __mutex_lock+0xaec/0xce0 [ 2382.878090] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2382.882655] ? lock_downgrade+0xfd/0x250 [ 2382.886603] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2382.891000] ceph_con_process_message+0x74/0x150 [libceph] [ 2382.896495] ? calc_signature+0xdb/0x100 [libceph] [ 2382.901318] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2382.906665] ? process_scheduled_works+0x22a/0x510 [ 2382.911456] ? process_scheduled_works+0x22a/0x510 [ 2382.916263] ceph_con_workfn+0x344/0x750 [libceph] [ 2382.921069] ? process_scheduled_works+0x22a/0x510 [ 2382.925875] process_scheduled_works+0x2a4/0x510 [ 2382.930535] worker_thread+0x188/0x340 [ 2382.934284] ? rescuer_thread+0x310/0x310 [ 2382.938296] kthread+0xf5/0x130 [ 2382.941444] ? kthread_complete_and_exit+0x20/0x20 [ 2382.946253] ret_from_fork+0x31/0x40 [ 2382.949852] ? kthread_complete_and_exit+0x20/0x20 [ 2382.954642] ret_from_fork_asm+0x11/0x20 [ 2382.958609] [ 2382.960793] irq event stamp: 10820033 [ 2382.964468] hardirqs last enabled at (10820043): [] console_unlock+0x106/0x120 [ 2382.973354] hardirqs last disabled at (10820052): [] console_unlock+0xeb/0x120 [ 2382.982144] softirqs last enabled at (10820000): [] __do_softirq+0x2be/0x371 [ 2382.990833] softirqs last disabled at (10819991): [] irq_exit_rcu+0xb0/0xd0 [ 2382.999373] ---[ end trace 0000000000000000 ]--- [ 2383.004157] ------------[ cut here ]------------ [ 2383.008776] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2383.020426] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2383.029386] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2383.084857] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2383.093817] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2383.101293] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2383.106782] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2383.125542] RSP: 0018:ffffc90001957c10 EFLAGS: 00010286 [ 2383.130787] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2383.137919] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2383.145053] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009f3a [ 2383.152189] R10: 000000000032caf0 R11: ffffc90001957aa0 R12: ffff8881d1224bc8 [ 2383.159319] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2383.166449] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2383.174539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2383.180282] CR2: 00007f9bd19a7000 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2383.187417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2383.194549] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2383.201682] Call Trace: [ 2383.204136] [ 2383.206241] ? __warn+0x85/0x180 [ 2383.209473] ? __list_add_valid_or_report+0x7d/0xa0 [ 2383.214355] ? report_bug+0xfc/0x1d0 [ 2383.217935] ? prb_read_valid+0x1b/0x20 [ 2383.221786] ? handle_bug+0x44/0x70 [ 2383.225282] ? exc_invalid_op+0x17/0x60 [ 2383.229127] ? asm_exc_invalid_op+0x1a/0x20 [ 2383.233323] ? __list_add_valid_or_report+0x7d/0xa0 [ 2383.238206] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2383.243104] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2383.248353] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2383.253166] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2383.257890] ? __d_lookup+0xd6/0x200 [ 2383.261472] lookup_fast+0x6d/0xf0 [ 2383.264884] path_openat+0xe4/0x9d0 [ 2383.268379] do_filp_open+0xb8/0x120 [ 2383.271969] ? lock_release+0xc5/0x260 [ 2383.275725] ? _raw_spin_unlock+0x23/0x40 [ 2383.279736] ? alloc_fd+0x13f/0x200 [ 2383.283233] do_sys_openat2+0x249/0x290 [ 2383.287082] do_sys_open+0x4f/0x80 [ 2383.290493] do_syscall_64+0x38/0x80 [ 2383.294072] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2383.299125] RIP: 0033:0x7f48f0ef2805 [ 2383.302704] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2383.321449] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2383.329015] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2383.336146] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2383.343279] RBP: 000000000207da30 R08: 0000000003ffffff R09: 0000000000000000 [ 2383.350415] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2383.357548] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2383.364691] [ 2383.366882] irq event stamp: 12851167 [ 2383.370544] hardirqs last enabled at (12851177): [] console_unlock+0x106/0x120 [ 2383.379412] hardirqs last disabled at (12851186): [] console_unlock+0xeb/0x120 [ 2383.388190] softirqs last enabled at (12851134): [] __do_softirq+0x2be/0x371 [ 2383.396885] softirqs last disabled at (12851129): [] irq_exit_rcu+0xb0/0xd0 [ 2383.405403] ---[ end trace 0000000000000000 ]--- [ 2383.410026] ------------[ cut here ]------------ [ 2383.414640] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2383.426279] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2383.434975] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2383.490468] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2383.499612] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2383.507110] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2383.512640] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2383.518122] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2383.536886] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2383.542113] RAX: 0000000000000000 RBX: ffff8881c25d1d58 RCX: 0000000000000000 [ 2383.549264] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2383.556397] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009f75 [ 2383.563528] R10: 000000000032daf0 R11: ffffc9000030b828 R12: ffff8881c25d24a8 [ 2383.570680] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2383.577830] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2383.585932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2383.591686] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2383.598828] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2383.605979] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2383.613122] Call Trace: [ 2383.615590] [ 2383.617713] ? __warn+0x85/0x180 [ 2383.620957] ? __list_add_valid_or_report+0x7d/0xa0 [ 2383.625852] ? report_bug+0xfc/0x1d0 [ 2383.629448] ? prb_read_valid+0x1b/0x20 [ 2383.633309] ? handle_bug+0x44/0x70 [ 2383.636798] ? exc_invalid_op+0x17/0x60 [ 2383.640655] ? asm_exc_invalid_op+0x1a/0x20 [ 2383.644880] ? __list_add_valid_or_report+0x7d/0xa0 [ 2383.649756] ? __list_add_valid_or_report+0x7d/0xa0 [ 2383.654652] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2383.659560] ceph_add_cap+0x20b/0x620 [ceph] [ 2383.663880] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2383.668619] ? lock_acquire+0xcb/0x2b0 [ 2383.672373] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2383.676822] ? __mutex_lock+0x97/0xce0 [ 2383.680591] ? __mutex_lock+0xaec/0xce0 [ 2383.684467] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2383.689008] ? lock_downgrade+0xfd/0x250 [ 2383.692973] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2383.697354] ceph_con_process_message+0x74/0x150 [libceph] [ 2383.702850] ? calc_signature+0xdb/0x100 [libceph] [ 2383.707678] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2383.713036] ? process_scheduled_works+0x22a/0x510 [ 2383.717827] ? process_scheduled_works+0x22a/0x510 [ 2383.722635] ceph_con_workfn+0x344/0x750 [libceph] [ 2383.727440] ? process_scheduled_works+0x22a/0x510 [ 2383.732246] process_scheduled_works+0x2a4/0x510 [ 2383.736888] worker_thread+0x188/0x340 [ 2383.740636] ? rescuer_thread+0x310/0x310 [ 2383.744665] kthread+0xf5/0x130 [ 2383.747830] ? kthread_complete_and_exit+0x20/0x20 [ 2383.752639] ret_from_fork+0x31/0x40 [ 2383.756235] ? kthread_complete_and_exit+0x20/0x20 [ 2383.761030] ret_from_fork_asm+0x11/0x20 [ 2383.764996] [ 2383.767182] irq event stamp: 10821991 [ 2383.770847] hardirqs last enabled at (10822001): [] console_unlock+0x106/0x120 [ 2383.779731] hardirqs last disabled at (10822010): [] console_unlock+0xeb/0x120 [ 2383.788529] softirqs last enabled at (10821938): [] __do_softirq+0x2be/0x371 [ 2383.797238] softirqs last disabled at (10821933): [] irq_exit_rcu+0xb0/0xd0 [ 2383.805757] ---[ end trace 0000000000000000 ]--- [ 2383.810392] ------------[ cut here ]------------ [ 2383.815007] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2383.826685] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2383.835383] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2383.890860] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2383.900000] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2383.907489] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2383.913001] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2383.918505] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2383.937268] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2383.942494] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2383.949644] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2383.956788] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009fb4 [ 2383.963937] R10: 000000000032ec08 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2383.971087] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2383.978236] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2383.986322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2383.992069] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2383.999220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2384.006353] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2384.013484] Call Trace: [ 2384.015945] [ 2384.018060] ? __warn+0x85/0x180 [ 2384.021314] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.026191] ? report_bug+0xfc/0x1d0 [ 2384.029769] ? prb_read_valid+0x1b/0x20 [ 2384.033628] ? handle_bug+0x44/0x70 [ 2384.037155] ? exc_invalid_op+0x17/0x60 [ 2384.040994] ? asm_exc_invalid_op+0x1a/0x20 [ 2384.045202] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.050077] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.054971] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2384.059880] ceph_add_cap+0x20b/0x620 [ceph] [ 2384.064183] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2384.068921] ceph_fill_trace+0x143/0xf30 [ceph] [ 2384.073460] ? __mutex_lock+0x97/0xce0 [ 2384.077230] ? __mutex_lock+0xaec/0xce0 [ 2384.081070] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2384.085629] ? lock_downgrade+0xfd/0x250 [ 2384.089592] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2384.093982] ceph_con_process_message+0x74/0x150 [libceph] [ 2384.099477] ? calc_signature+0xdb/0x100 [libceph] [ 2384.104297] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2384.109638] ? process_scheduled_works+0x22a/0x510 [ 2384.114444] ? process_scheduled_works+0x22a/0x510 [ 2384.119255] ceph_con_workfn+0x344/0x750 [libceph] [ 2384.124059] ? process_scheduled_works+0x22a/0x510 [ 2384.128865] process_scheduled_works+0x2a4/0x510 [ 2384.133524] worker_thread+0x188/0x340 [ 2384.137274] ? rescuer_thread+0x310/0x310 [ 2384.141286] kthread+0xf5/0x130 [ 2384.144447] ? kthread_complete_and_exit+0x20/0x20 [ 2384.149245] ret_from_fork+0x31/0x40 [ 2384.152821] ? kthread_complete_and_exit+0x20/0x20 [ 2384.157633] ret_from_fork_asm+0x11/0x20 [ 2384.161600] [ 2384.163794] irq event stamp: 10823047 [ 2384.167477] hardirqs last enabled at (10823057): [] console_unlock+0x106/0x120 [ 2384.176362] hardirqs last disabled at (10823066): [] console_unlock+0xeb/0x120 [ 2384.185159] softirqs last enabled at (10823004): [] __do_softirq+0x2be/0x371 [ 2384.193850] softirqs last disabled at (10822999): [] irq_exit_rcu+0xb0/0xd0 [ 2384.202386] ---[ end trace 0000000000000000 ]--- [ 2384.207040] ------------[ cut here ]------------ [ 2384.211662] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2384.223325] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2384.232019] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2384.287513] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2384.296655] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2384.304162] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2384.309656] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2384.315170] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2384.333915] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2384.339173] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2384.346300] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2384.353433] RBP: ffff88817f867000 R08: 0000000000000000 R09: c000000100009ff2 [ 2384.360565] R10: 000000000032faa0 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2384.367716] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2384.374866] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2384.382970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2384.388731] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2384.395873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2384.403015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2384.410169] Call Trace: [ 2384.412617] [ 2384.414734] ? __warn+0x85/0x180 [ 2384.417984] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.422880] ? report_bug+0xfc/0x1d0 [ 2384.426476] ? prb_read_valid+0x1b/0x20 [ 2384.430335] ? handle_bug+0x44/0x70 [ 2384.433825] ? exc_invalid_op+0x17/0x60 [ 2384.437675] ? asm_exc_invalid_op+0x1a/0x20 [ 2384.441899] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.446775] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.451669] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2384.456595] ceph_add_cap+0x20b/0x620 [ceph] [ 2384.460916] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2384.465655] ? lock_acquire+0xcb/0x2b0 [ 2384.469408] ceph_fill_trace+0x143/0xf30 [ceph] [ 2384.473945] ? __mutex_lock+0x97/0xce0 [ 2384.477714] ? __mutex_lock+0xaec/0xce0 [ 2384.481573] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2384.486131] ? lock_downgrade+0xfd/0x250 [ 2384.490096] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2384.494492] ceph_con_process_message+0x74/0x150 [libceph] [ 2384.499988] ? calc_signature+0xdb/0x100 [libceph] [ 2384.504826] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2384.510172] ? process_scheduled_works+0x22a/0x510 [ 2384.514985] ? process_scheduled_works+0x22a/0x510 [ 2384.519773] ceph_con_workfn+0x344/0x750 [libceph] [ 2384.524596] ? process_scheduled_works+0x22a/0x510 [ 2384.529403] process_scheduled_works+0x2a4/0x510 [ 2384.534045] worker_thread+0x188/0x340 [ 2384.537795] ? rescuer_thread+0x310/0x310 [ 2384.541823] kthread+0xf5/0x130 [ 2384.544987] ? kthread_complete_and_exit+0x20/0x20 [ 2384.549796] ret_from_fork+0x31/0x40 [ 2384.553392] ? kthread_complete_and_exit+0x20/0x20 [ 2384.558188] ret_from_fork_asm+0x11/0x20 [ 2384.562136] [ 2384.564322] irq event stamp: 10824153 [ 2384.567987] hardirqs last enabled at (10824163): [] console_unlock+0x106/0x120 [ 2384.576872] hardirqs last disabled at (10824172): [] console_unlock+0xeb/0x120 [ 2384.585668] softirqs last enabled at (10824100): [] __do_softirq+0x2be/0x371 [ 2384.594380] softirqs last disabled at (10824095): [] irq_exit_rcu+0xb0/0xd0 [ 2384.602897] ---[ end trace 0000000000000000 ]--- [ 2384.607583] ------------[ cut here ]------------ [ 2384.612200] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2384.623846] WARNING: CPU: 2 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2384.632799] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2384.688262] CPU: 2 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2384.697210] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2384.704689] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2384.710175] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2384.728922] RSP: 0018:ffffc90001fd7ac0 EFLAGS: 00010282 [ 2384.734150] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2384.741280] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2384.748412] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a031 [ 2384.755540] R10: 0000000000330bc0 R11: ffffc90001fd7950 R12: ffff8881d10b72e8 [ 2384.762668] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2384.769794] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2384.777881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2384.783625] CR2: 00007fb38c57d000 CR3: 0000000120c1e005 CR4: 00000000003706e0 [ 2384.790751] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2384.797882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2384.805011] Call Trace: [ 2384.807460] [ 2384.809558] ? __warn+0x85/0x180 [ 2384.812791] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.817671] ? report_bug+0xfc/0x1d0 [ 2384.821250] ? prb_read_valid+0x1b/0x20 [ 2384.825089] ? handle_bug+0x44/0x70 [ 2384.828582] ? exc_invalid_op+0x17/0x60 [ 2384.832421] ? asm_exc_invalid_op+0x1a/0x20 [ 2384.836611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2384.841487] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2384.846376] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2384.851612] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2384.856413] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2384.861128] ? __d_lookup+0xd6/0x200 [ 2384.864705] lookup_fast+0x6d/0xf0 [ 2384.868112] walk_component+0x26/0x160 [ 2384.871866] link_path_walk+0x24d/0x380 [ 2384.875703] ? path_lookupat.isra.40+0x22/0x120 [ 2384.880237] path_lookupat.isra.40+0x6b/0x120 [ 2384.884598] filename_lookup+0xd4/0x190 [ 2384.888439] ? lock_acquire+0xcb/0x2b0 [ 2384.892188] ? __might_fault+0x45/0x80 [ 2384.895940] ? find_held_lock+0x2d/0x90 [ 2384.899781] ? __might_fault+0x45/0x80 [ 2384.903536] vfs_statx+0x89/0x150 [ 2384.906856] vfs_fstatat+0x5a/0xc0 [ 2384.910259] __do_sys_newstat+0x30/0x70 [ 2384.914096] ? lock_release+0xc5/0x260 [ 2384.917853] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2384.922727] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2384.927868] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2384.933012] do_syscall_64+0x38/0x80 [ 2384.936589] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2384.941639] RIP: 0033:0x7f48f0eeca79 [ 2384.945221] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2384.963967] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2384.971531] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2384.978654] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2384.985780] RBP: 00007ffc53d60520 R08: 0000000000000000 R09: 00000000020767c0 [ 2384.992912] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2385.000038] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020767c0 [ 2385.007175] [ 2385.009363] irq event stamp: 13015743 [ 2385.013029] hardirqs last enabled at (13015753): [] console_unlock+0x106/0x120 [ 2385.021894] hardirqs last disabled at (13015762): [] console_unlock+0xeb/0x120 [ 2385.030674] softirqs last enabled at (13015776): [] __do_softirq+0x2be/0x371 [ 2385.039366] softirqs last disabled at (13015771): [] irq_exit_rcu+0xb0/0xd0 [ 2385.047886] ---[ end trace 0000000000000000 ]--- [ 2385.052497] ------------[ cut here ]------------ [ 2385.057119] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2385.068780] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2385.077477] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2385.132972] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2385.142113] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2385.149602] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2385.155112] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2385.160616] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2385.179372] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2385.184598] RAX: 0000000000000000 RBX: ffff8881d1161d58 RCX: 0000000000000000 [ 2385.191749] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2385.198901] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a075 [ 2385.206048] R10: 00000000003321b0 R11: ffffc9000030b828 R12: ffff8881d11624a8 [ 2385.213191] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2385.220324] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2385.228409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2385.234178] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2385.241315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2385.248462] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2385.255605] Call Trace: [ 2385.258065] [ 2385.260186] ? __warn+0x85/0x180 [ 2385.263415] ? __list_add_valid_or_report+0x7d/0xa0 [ 2385.268294] ? report_bug+0xfc/0x1d0 [ 2385.271873] ? prb_read_valid+0x1b/0x20 [ 2385.275732] ? handle_bug+0x44/0x70 [ 2385.279240] ? exc_invalid_op+0x17/0x60 [ 2385.283080] ? asm_exc_invalid_op+0x1a/0x20 [ 2385.287287] ? __list_add_valid_or_report+0x7d/0xa0 [ 2385.292181] ? __list_add_valid_or_report+0x7d/0xa0 [ 2385.297059] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2385.301966] ceph_add_cap+0x20b/0x620 [ceph] [ 2385.306271] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2385.311010] ? lock_acquire+0xcb/0x2b0 [ 2385.314779] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2385.319229] ? __mutex_lock+0x97/0xce0 [ 2385.322980] ? __mutex_lock+0xaec/0xce0 [ 2385.326857] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2385.331398] ? lock_downgrade+0xfd/0x250 [ 2385.335327] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2385.339715] ceph_con_process_message+0x74/0x150 [libceph] [ 2385.345211] ? calc_signature+0xdb/0x100 [libceph] [ 2385.350017] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2385.355364] ? process_scheduled_works+0x22a/0x510 [ 2385.360178] ? process_scheduled_works+0x22a/0x510 [ 2385.364971] ceph_con_workfn+0x344/0x750 [libceph] [ 2385.369776] ? process_scheduled_works+0x22a/0x510 [ 2385.374582] process_scheduled_works+0x2a4/0x510 [ 2385.379227] worker_thread+0x188/0x340 [ 2385.382982] ? rescuer_thread+0x310/0x310 [ 2385.387012] kthread+0xf5/0x130 [ 2385.390191] ? kthread_complete_and_exit+0x20/0x20 [ 2385.394987] ret_from_fork+0x31/0x40 [ 2385.398582] ? kthread_complete_and_exit+0x20/0x20 [ 2385.403393] ret_from_fork_asm+0x11/0x20 [ 2385.407325] [ 2385.409510] irq event stamp: 10826925 [ 2385.413190] hardirqs last enabled at (10826935): [] console_unlock+0x106/0x120 [ 2385.422052] hardirqs last disabled at (10826944): [] console_unlock+0xeb/0x120 [ 2385.430848] softirqs last enabled at (10826872): [] __do_softirq+0x2be/0x371 [ 2385.439559] softirqs last disabled at (10826867): [] irq_exit_rcu+0xb0/0xd0 [ 2385.448097] ---[ end trace 0000000000000000 ]--- [ 2385.452889] ------------[ cut here ]------------ [ 2385.457509] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2385.469188] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2385.477884] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2385.533363] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2385.542487] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2385.549984] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2385.555494] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2385.560998] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2385.579765] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2385.585004] RAX: 0000000000000000 RBX: ffff8881d0967560 RCX: 0000000000000000 [ 2385.592146] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2385.599290] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a0b4 [ 2385.606423] R10: 00000000003332c8 R11: ffffc9000030b828 R12: ffff8881d0967cb0 [ 2385.613554] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2385.620704] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2385.628809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2385.634570] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2385.641713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2385.648862] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2385.656012] Call Trace: [ 2385.658479] [ 2385.660598] ? __warn+0x85/0x180 [ 2385.663849] ? __list_add_valid_or_report+0x7d/0xa0 [ 2385.668745] ? report_bug+0xfc/0x1d0 [ 2385.672341] ? prb_read_valid+0x1b/0x20 [ 2385.676196] ? handle_bug+0x44/0x70 [ 2385.679683] ? exc_invalid_op+0x17/0x60 [ 2385.683540] ? asm_exc_invalid_op+0x1a/0x20 [ 2385.687763] ? __list_add_valid_or_report+0x7d/0xa0 [ 2385.692640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2385.697535] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2385.702444] ceph_add_cap+0x20b/0x620 [ceph] [ 2385.706747] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2385.711504] ceph_fill_trace+0x143/0xf30 [ceph] [ 2385.716040] ? __mutex_lock+0x97/0xce0 [ 2385.719810] ? __mutex_lock+0xaec/0xce0 [ 2385.723667] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2385.728226] ? lock_downgrade+0xfd/0x250 [ 2385.732181] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2385.736561] ceph_con_process_message+0x74/0x150 [libceph] [ 2385.742057] ? calc_signature+0xdb/0x100 [libceph] [ 2385.746878] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2385.752218] ? process_scheduled_works+0x22a/0x510 [ 2385.757026] ? process_scheduled_works+0x22a/0x510 [ 2385.761816] ceph_con_workfn+0x344/0x750 [libceph] [ 2385.766637] ? process_scheduled_works+0x22a/0x510 [ 2385.771446] process_scheduled_works+0x2a4/0x510 [ 2385.776088] worker_thread+0x188/0x340 [ 2385.779836] ? rescuer_thread+0x310/0x310 [ 2385.783867] kthread+0xf5/0x130 [ 2385.787030] ? kthread_complete_and_exit+0x20/0x20 [ 2385.791841] ret_from_fork+0x31/0x40 [ 2385.795437] ? kthread_complete_and_exit+0x20/0x20 [ 2385.800233] ret_from_fork_asm+0x11/0x20 [ 2385.804188] [ 2385.806382] irq event stamp: 10827987 [ 2385.810046] hardirqs last enabled at (10827997): [] console_unlock+0x106/0x120 [ 2385.818931] hardirqs last disabled at (10828006): [] console_unlock+0xeb/0x120 [ 2385.827729] softirqs last enabled at (10827942): [] __do_softirq+0x2be/0x371 [ 2385.836441] softirqs last disabled at (10828025): [] irq_exit_rcu+0xb0/0xd0 [ 2385.844958] ---[ end trace 0000000000000000 ]--- [ 2385.849596] ------------[ cut here ]------------ [ 2385.854232] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2385.865891] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2385.874581] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2385.930078] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2385.939218] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2385.946700] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2385.952210] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2385.957695] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2385.976467] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2385.981694] RAX: 0000000000000000 RBX: ffff8881d0967560 RCX: 0000000000000000 [ 2385.988836] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2385.995989] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a0f2 [ 2386.003137] R10: 00000000003343b8 R11: ffffc9000030b8d8 R12: ffff8881d0967cb0 [ 2386.010287] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2386.017420] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2386.025504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2386.031260] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2386.038394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2386.045525] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2386.052668] Call Trace: [ 2386.055138] [ 2386.057261] ? __warn+0x85/0x180 [ 2386.060494] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.065392] ? report_bug+0xfc/0x1d0 [ 2386.068970] ? prb_read_valid+0x1b/0x20 [ 2386.072827] ? handle_bug+0x44/0x70 [ 2386.076337] ? exc_invalid_op+0x17/0x60 [ 2386.080195] ? asm_exc_invalid_op+0x1a/0x20 [ 2386.084384] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.089261] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.094140] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2386.099054] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2386.103707] ceph_fill_trace+0x143/0xf30 [ceph] [ 2386.108248] ? __mutex_lock+0x97/0xce0 [ 2386.112001] ? __mutex_lock+0xaec/0xce0 [ 2386.115857] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2386.120415] ? lock_downgrade+0xfd/0x250 [ 2386.124350] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2386.128743] ceph_con_process_message+0x74/0x150 [libceph] [ 2386.134240] ? calc_signature+0xdb/0x100 [libceph] [ 2386.139044] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2386.144384] ? process_scheduled_works+0x22a/0x510 [ 2386.149199] ? process_scheduled_works+0x22a/0x510 [ 2386.153990] ceph_con_workfn+0x344/0x750 [libceph] [ 2386.158811] ? process_scheduled_works+0x22a/0x510 [ 2386.163620] process_scheduled_works+0x2a4/0x510 [ 2386.168262] worker_thread+0x188/0x340 [ 2386.172020] ? rescuer_thread+0x310/0x310 [ 2386.176049] kthread+0xf5/0x130 [ 2386.179212] ? kthread_complete_and_exit+0x20/0x20 [ 2386.184006] ret_from_fork+0x31/0x40 [ 2386.187602] ? kthread_complete_and_exit+0x20/0x20 [ 2386.192411] ret_from_fork_asm+0x11/0x20 [ 2386.196344] [ 2386.198529] irq event stamp: 10829039 [ 2386.202205] hardirqs last enabled at (10829049): [] console_unlock+0x106/0x120 [ 2386.211072] hardirqs last disabled at (10829058): [] console_unlock+0xeb/0x120 [ 2386.219869] softirqs last enabled at (10829006): [] __do_softirq+0x2be/0x371 [ 2386.228577] softirqs last disabled at (10829001): [] irq_exit_rcu+0xb0/0xd0 [ 2386.237114] ---[ end trace 0000000000000000 ]--- [ 2386.241783] ------------[ cut here ]------------ [ 2386.246399] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2386.258041] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2386.266739] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2386.322215] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2386.331341] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2386.338820] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2386.344314] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2386.349801] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2386.368548] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2386.373773] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2386.380907] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2386.388039] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a12f [ 2386.395172] R10: 0000000000335458 R11: ffffc9000030b828 R12: ffff8881d800c200 [ 2386.402303] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2386.409437] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2386.417523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2386.423268] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2386.430402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2386.437535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2386.444668] Call Trace: [ 2386.447120] [ 2386.449227] ? __warn+0x85/0x180 [ 2386.452460] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.457338] ? report_bug+0xfc/0x1d0 [ 2386.460918] ? prb_read_valid+0x1b/0x20 [ 2386.464761] ? handle_bug+0x44/0x70 [ 2386.468250] ? exc_invalid_op+0x17/0x60 [ 2386.472090] ? asm_exc_invalid_op+0x1a/0x20 [ 2386.476282] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.481165] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.486043] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2386.490934] ceph_add_cap+0x20b/0x620 [ceph] [ 2386.495219] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2386.499942] ? lock_acquire+0xcb/0x2b0 [ 2386.503693] ceph_fill_trace+0x143/0xf30 [ceph] [ 2386.508232] ? __mutex_lock+0x97/0xce0 [ 2386.511983] ? __mutex_lock+0xaec/0xce0 [ 2386.515826] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2386.520373] ? lock_downgrade+0xfd/0x250 [ 2386.524303] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2386.528685] ceph_con_process_message+0x74/0x150 [libceph] [ 2386.534184] ? calc_signature+0xdb/0x100 [libceph] [ 2386.538992] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2386.544324] ? process_scheduled_works+0x22a/0x510 [ 2386.549115] ? process_scheduled_works+0x22a/0x510 [ 2386.553905] ceph_con_workfn+0x344/0x750 [libceph] [ 2386.558708] ? process_scheduled_works+0x22a/0x510 [ 2386.563499] process_scheduled_works+0x2a4/0x510 [ 2386.568123] worker_thread+0x188/0x340 [ 2386.571873] ? rescuer_thread+0x310/0x310 [ 2386.575883] kthread+0xf5/0x130 [ 2386.579032] ? kthread_complete_and_exit+0x20/0x20 [ 2386.583823] ret_from_fork+0x31/0x40 [ 2386.587401] ? kthread_complete_and_exit+0x20/0x20 [ 2386.592195] ret_from_fork_asm+0x11/0x20 [ 2386.596128] [ 2386.598314] irq event stamp: 10830155 [ 2386.601978] hardirqs last enabled at (10830165): [] console_unlock+0x106/0x120 [ 2386.610848] hardirqs last disabled at (10830180): [] console_unlock+0xeb/0x120 [ 2386.619625] softirqs last enabled at (10830178): [] __do_softirq+0x2be/0x371 [ 2386.628317] softirqs last disabled at (10830173): [] irq_exit_rcu+0xb0/0xd0 [ 2386.636837] ---[ end trace 0000000000000000 ]--- [ 2386.641459] ------------[ cut here ]------------ [ 2386.646080] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2386.657737] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2386.666696] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2386.722227] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2386.731179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2386.738663] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2386.744152] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2386.762899] RSP: 0018:ffffc90001fdfaa8 EFLAGS: 00010282 [ 2386.768125] RAX: 0000000000000000 RBX: ffff8881d0967560 RCX: 0000000000000000 [ 2386.775256] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2386.782391] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a16e [ 2386.789529] R10: 0000000000336570 R11: ffffc90001fdf938 R12: ffff8881d0967cb0 [ 2386.796664] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2386.803796] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2386.811884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2386.817628] CR2: 00007f9bd22d7020 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2386.824763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2386.831895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2386.839028] Call Trace: [ 2386.841481] [ 2386.843589] ? __warn+0x85/0x180 [ 2386.846822] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.851705] ? report_bug+0xfc/0x1d0 [ 2386.855289] ? prb_read_valid+0x1b/0x20 [ 2386.859130] ? handle_bug+0x44/0x70 [ 2386.862628] ? exc_invalid_op+0x17/0x60 [ 2386.866469] ? asm_exc_invalid_op+0x1a/0x20 [ 2386.870664] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.875551] ? __list_add_valid_or_report+0x7d/0xa0 [ 2386.880430] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2386.885330] ceph_open+0x16b/0x440 [ceph] [ 2386.889360] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2386.894515] do_dentry_open+0x172/0x530 [ 2386.898357] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2386.903009] lookup_open+0x37b/0x630 [ 2386.906603] ? verify_cpu+0x20/0x100 [ 2386.910193] ? verify_cpu+0x20/0x100 [ 2386.913776] path_openat+0x25a/0x9d0 [ 2386.917367] do_filp_open+0xb8/0x120 [ 2386.920957] ? lock_release+0xc5/0x260 [ 2386.924714] ? _raw_spin_unlock+0x23/0x40 [ 2386.928730] ? alloc_fd+0x13f/0x200 [ 2386.932229] do_sys_openat2+0x249/0x290 [ 2386.936075] do_sys_open+0x4f/0x80 [ 2386.939487] do_syscall_64+0x38/0x80 [ 2386.943074] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2386.948125] RIP: 0033:0x7f48f0eed2f2 [ 2386.951705] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2386.970453] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2386.978018] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2386.985153] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2386.992283] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2386.999415] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2387.006548] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2387.013695] [ 2387.015895] irq event stamp: 12880527 [ 2387.019559] hardirqs last enabled at (12880537): [] console_unlock+0x106/0x120 [ 2387.028425] hardirqs last disabled at (12880546): [] console_unlock+0xeb/0x120 [ 2387.037204] softirqs last enabled at (12880524): [] __do_softirq+0x2be/0x371 [ 2387.045897] softirqs last disabled at (12880517): [] irq_exit_rcu+0xb0/0xd0 [ 2387.054427] ---[ end trace 0000000000000000 ]--- [ 2387.059046] ------------[ cut here ]------------ [ 2387.063664] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2387.075312] WARNING: CPU: 2 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2387.084274] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2387.139797] CPU: 2 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2387.148753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2387.156235] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2387.161719] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2387.180465] RSP: 0018:ffffc90001fcfaa0 EFLAGS: 00010286 [ 2387.185693] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2387.192824] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2387.199957] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a1ac [ 2387.207093] R10: 0000000000337848 R11: ffffc90001fcf930 R12: ffff8881d106cbc8 [ 2387.214227] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2387.221365] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2387.229451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2387.235197] CR2: 00007fb38c57d000 CR3: 000000011aeae002 CR4: 00000000003706e0 [ 2387.242329] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2387.249463] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2387.256596] Call Trace: [ 2387.259048] [ 2387.261155] ? __warn+0x85/0x180 [ 2387.264389] ? __list_add_valid_or_report+0x7d/0xa0 [ 2387.269269] ? report_bug+0xfc/0x1d0 [ 2387.272844] ? prb_read_valid+0x1b/0x20 [ 2387.276688] ? handle_bug+0x44/0x70 [ 2387.280179] ? exc_invalid_op+0x17/0x60 [ 2387.284017] ? asm_exc_invalid_op+0x1a/0x20 [ 2387.288213] ? __list_add_valid_or_report+0x7d/0xa0 [ 2387.293094] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2387.297993] ceph_check_caps+0x999/0xae0 [ceph] [ 2387.302553] ? __lock_acquire+0xbfa/0x1e20 [ 2387.306660] ? avc_has_perm+0x57/0x320 [ 2387.310421] ? lock_acquire+0xcb/0x2b0 [ 2387.314178] ? ceph_open+0x113/0x440 [ceph] [ 2387.318379] ? find_held_lock+0x2d/0x90 [ 2387.322221] ? ceph_open+0x175/0x440 [ceph] [ 2387.326426] ? lock_release+0xc5/0x260 [ 2387.330184] ceph_open+0x3cc/0x440 [ceph] [ 2387.334221] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2387.339374] do_dentry_open+0x172/0x530 [ 2387.343214] path_openat+0x743/0x9d0 [ 2387.346800] do_filp_open+0xb8/0x120 [ 2387.350387] ? lock_release+0xc5/0x260 [ 2387.354145] ? _raw_spin_unlock+0x23/0x40 [ 2387.358162] ? alloc_fd+0x13f/0x200 [ 2387.361660] do_sys_openat2+0x249/0x290 [ 2387.365506] do_sys_open+0x4f/0x80 [ 2387.368921] do_syscall_64+0x38/0x80 [ 2387.372507] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2387.377559] RIP: 0033:0x7f48f0ef2805 [ 2387.381138] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2387.399886] RSP: 002b:00007ffc53d60040 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2387.407451] RAX: ffffffffffffffda RBX: 0000000000000556 RCX: 00007f48f0ef2805 [ 2387.414583] RDX: 0000000000090800 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2387.421716] RBP: 00007ffc53d60520 R08: 0000000007ffffff R09: 0000000000000002 [ 2387.428849] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000006 [ 2387.435981] R13: 00007ffc53d60520 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2387.443127] [ 2387.445315] irq event stamp: 12928965 [ 2387.448981] hardirqs last enabled at (12928975): [] console_unlock+0x106/0x120 [ 2387.457846] hardirqs last disabled at (12928984): [] console_unlock+0xeb/0x120 [ 2387.466628] softirqs last enabled at (12928998): [] __do_softirq+0x2be/0x371 [ 2387.475319] softirqs last disabled at (12928993): [] irq_exit_rcu+0xb0/0xd0 [ 2387.483839] ---[ end trace 0000000000000000 ]--- [ 2387.488459] ------------[ cut here ]------------ [ 2387.493075] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2387.504720] WARNING: CPU: 6 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2387.513669] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2387.569138] CPU: 6 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2387.578088] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2387.585568] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2387.591054] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2387.609800] RSP: 0018:ffffc90001fdf8c0 EFLAGS: 00010282 [ 2387.615025] RAX: 0000000000000000 RBX: ffff8881d0967560 RCX: 0000000000000000 [ 2387.622151] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2387.629283] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a1ed [ 2387.636416] R10: 0000000000338c40 R11: ffffc90001fdf750 R12: ffff8881d0967cb0 [ 2387.643548] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2387.650672] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2387.658749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2387.664488] CR2: 00007f9bd22d7020 CR3: 0000000179402001 CR4: 00000000003706e0 [ 2387.671620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2387.678746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2387.685877] Call Trace: [ 2387.688320] [ 2387.690419] ? __warn+0x85/0x180 [ 2387.693652] ? __list_add_valid_or_report+0x7d/0xa0 [ 2387.698531] ? report_bug+0xfc/0x1d0 [ 2387.702111] ? prb_read_valid+0x1b/0x20 [ 2387.705953] ? handle_bug+0x44/0x70 [ 2387.709443] ? exc_invalid_op+0x17/0x60 [ 2387.713283] ? asm_exc_invalid_op+0x1a/0x20 [ 2387.717473] ? __list_add_valid_or_report+0x7d/0xa0 [ 2387.722350] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2387.727237] ceph_check_caps+0x999/0xae0 [ceph] [ 2387.731787] ? handle_bug+0x44/0x70 [ 2387.735280] ? exc_invalid_op+0x17/0x60 [ 2387.739119] ? asm_exc_invalid_op+0x1a/0x20 [ 2387.743309] ? find_held_lock+0x2d/0x90 [ 2387.747144] ? ceph_open+0x175/0x440 [ceph] [ 2387.751340] ? lock_release+0xc5/0x260 [ 2387.755095] ceph_open+0x3cc/0x440 [ceph] [ 2387.759115] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2387.764263] do_dentry_open+0x172/0x530 [ 2387.768101] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2387.772736] lookup_open+0x37b/0x630 [ 2387.776318] ? verify_cpu+0x20/0x100 [ 2387.779897] ? verify_cpu+0x20/0x100 [ 2387.783477] path_openat+0x25a/0x9d0 [ 2387.787059] do_filp_open+0xb8/0x120 [ 2387.790639] ? lock_release+0xc5/0x260 [ 2387.794388] ? _raw_spin_unlock+0x23/0x40 [ 2387.798398] ? alloc_fd+0x13f/0x200 [ 2387.801894] do_sys_openat2+0x249/0x290 [ 2387.805733] do_sys_open+0x4f/0x80 [ 2387.809138] do_syscall_64+0x38/0x80 [ 2387.812717] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2387.817770] RIP: 0033:0x7f48f0eed2f2 [ 2387.821348] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2387.840096] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2387.847662] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2387.854793] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2387.861917] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2387.869052] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2387.876184] R13: 00007f48f183dac8 R14: 00000000020767c0 R15: 00000000000029bc [ 2387.883323] [ 2387.885510] irq event stamp: 12882613 [ 2387.889176] hardirqs last enabled at (12882623): [] console_unlock+0x106/0x120 [ 2387.898042] hardirqs last disabled at (12882632): [] console_unlock+0xeb/0x120 [ 2387.906821] softirqs last enabled at (12882646): [] __do_softirq+0x2be/0x371 [ 2387.915514] softirqs last disabled at (12882641): [] irq_exit_rcu+0xb0/0xd0 [ 2387.924034] ---[ end trace 0000000000000000 ]--- [ 2387.928653] ------------[ cut here ]------------ [ 2387.933270] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2387.944931] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2387.953625] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2388.009119] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2388.018260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2388.025741] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2388.031251] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2388.036737] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2388.055514] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2388.060752] RAX: 0000000000000000 RBX: ffff8881d1224478 RCX: 0000000000000000 [ 2388.067894] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2388.075046] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a231 [ 2388.082195] R10: 0000000000339e78 R11: ffffc9000030b828 R12: ffff8881d1224bc8 [ 2388.089344] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2388.096478] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2388.104563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.110318] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2388.117452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2388.124584] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2388.131726] Call Trace: [ 2388.134195] [ 2388.136310] ? __warn+0x85/0x180 [ 2388.139546] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.144443] ? report_bug+0xfc/0x1d0 [ 2388.148021] ? prb_read_valid+0x1b/0x20 [ 2388.151878] ? handle_bug+0x44/0x70 [ 2388.155387] ? exc_invalid_op+0x17/0x60 [ 2388.159237] ? asm_exc_invalid_op+0x1a/0x20 [ 2388.163427] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.168311] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.173188] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2388.178103] ceph_add_cap+0x20b/0x620 [ceph] [ 2388.182389] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2388.187120] ? lock_acquire+0xcb/0x2b0 [ 2388.190891] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2388.195342] ? __mutex_lock+0x97/0xce0 [ 2388.199093] ? __mutex_lock+0xaec/0xce0 [ 2388.202953] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2388.207529] ? lock_downgrade+0xfd/0x250 [ 2388.211474] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2388.215871] ceph_con_process_message+0x74/0x150 [libceph] [ 2388.221368] ? calc_signature+0xdb/0x100 [libceph] [ 2388.226180] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2388.231526] ? process_scheduled_works+0x22a/0x510 [ 2388.236321] ? process_scheduled_works+0x22a/0x510 [ 2388.241109] ceph_con_workfn+0x344/0x750 [libceph] [ 2388.245930] ? process_scheduled_works+0x22a/0x510 [ 2388.250738] process_scheduled_works+0x2a4/0x510 [ 2388.255381] worker_thread+0x188/0x340 [ 2388.259130] ? rescuer_thread+0x310/0x310 [ 2388.263158] kthread+0xf5/0x130 [ 2388.266322] ? kthread_complete_and_exit+0x20/0x20 [ 2388.271115] ret_from_fork+0x31/0x40 [ 2388.274712] ? kthread_complete_and_exit+0x20/0x20 [ 2388.279521] ret_from_fork_asm+0x11/0x20 [ 2388.283474] [ 2388.285666] irq event stamp: 10834081 [ 2388.289340] hardirqs last enabled at (10834091): [] console_unlock+0x106/0x120 [ 2388.298230] hardirqs last disabled at (10834100): [] console_unlock+0xeb/0x120 [ 2388.307013] softirqs last enabled at (10834028): [] __do_softirq+0x2be/0x371 [ 2388.315722] softirqs last disabled at (10834023): [] irq_exit_rcu+0xb0/0xd0 [ 2388.324259] ---[ end trace 0000000000000000 ]--- [ 2388.329034] ------------[ cut here ]------------ [ 2388.333657] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2388.345317] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2388.354013] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2388.409522] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2388.418667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2388.426156] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2388.431666] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2388.437168] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2388.455927] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2388.461169] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2388.468321] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2388.475455] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a270 [ 2388.482585] R10: 000000000033af70 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2388.489736] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2388.496886] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2388.504988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.510733] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2388.517876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2388.525025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2388.532167] Call Trace: [ 2388.534629] [ 2388.536752] ? __warn+0x85/0x180 [ 2388.540005] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.544900] ? report_bug+0xfc/0x1d0 [ 2388.548495] ? prb_read_valid+0x1b/0x20 [ 2388.552338] ? handle_bug+0x44/0x70 [ 2388.555829] ? exc_invalid_op+0x17/0x60 [ 2388.559686] ? asm_exc_invalid_op+0x1a/0x20 [ 2388.563910] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.568786] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.573683] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2388.578589] ceph_add_cap+0x20b/0x620 [ceph] [ 2388.582910] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2388.587655] ceph_fill_trace+0x143/0xf30 [ceph] [ 2388.592196] ? __mutex_lock+0x97/0xce0 [ 2388.595972] ? __mutex_lock+0xaec/0xce0 [ 2388.599814] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2388.604374] ? lock_downgrade+0xfd/0x250 [ 2388.608302] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2388.612699] ceph_con_process_message+0x74/0x150 [libceph] [ 2388.618195] ? calc_signature+0xdb/0x100 [libceph] [ 2388.623024] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2388.628347] ? process_scheduled_works+0x22a/0x510 [ 2388.633156] ? process_scheduled_works+0x22a/0x510 [ 2388.637946] ceph_con_workfn+0x344/0x750 [libceph] [ 2388.642766] ? process_scheduled_works+0x22a/0x510 [ 2388.647575] process_scheduled_works+0x2a4/0x510 [ 2388.652242] worker_thread+0x188/0x340 [ 2388.655992] ? rescuer_thread+0x310/0x310 [ 2388.660005] kthread+0xf5/0x130 [ 2388.663167] ? kthread_complete_and_exit+0x20/0x20 [ 2388.667980] ret_from_fork+0x31/0x40 [ 2388.671575] ? kthread_complete_and_exit+0x20/0x20 [ 2388.676385] ret_from_fork_asm+0x11/0x20 [ 2388.680317] [ 2388.682518] irq event stamp: 10835139 [ 2388.686178] hardirqs last enabled at (10835149): [] console_unlock+0x106/0x120 [ 2388.695060] hardirqs last disabled at (10835158): [] console_unlock+0xeb/0x120 [ 2388.703858] softirqs last enabled at (10835096): [] __do_softirq+0x2be/0x371 [ 2388.712568] softirqs last disabled at (10835091): [] irq_exit_rcu+0xb0/0xd0 [ 2388.721104] ---[ end trace 0000000000000000 ]--- [ 2388.725777] ------------[ cut here ]------------ [ 2388.730398] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2388.742057] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2388.750746] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2388.806249] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2388.815375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2388.822852] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2388.828364] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2388.833851] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2388.852614] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2388.857859] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2388.865009] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2388.872161] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a2ae [ 2388.879310] R10: 000000000033c1b0 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2388.886443] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2388.893574] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2388.901681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.907443] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2388.914574] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2388.921724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2388.928867] Call Trace: [ 2388.931337] [ 2388.933443] ? __warn+0x85/0x180 [ 2388.936677] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.941571] ? report_bug+0xfc/0x1d0 [ 2388.945168] ? prb_read_valid+0x1b/0x20 [ 2388.949026] ? handle_bug+0x44/0x70 [ 2388.952539] ? exc_invalid_op+0x17/0x60 [ 2388.956392] ? asm_exc_invalid_op+0x1a/0x20 [ 2388.960599] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.965476] ? __list_add_valid_or_report+0x7d/0xa0 [ 2388.970355] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2388.975254] ceph_add_cap+0x20b/0x620 [ceph] [ 2388.979556] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2388.984280] ? lock_acquire+0xcb/0x2b0 [ 2388.988056] ceph_fill_trace+0x143/0xf30 [ceph] [ 2388.992594] ? __mutex_lock+0x97/0xce0 [ 2388.996363] ? __mutex_lock+0xaec/0xce0 [ 2389.000204] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2389.004768] ? lock_downgrade+0xfd/0x250 [ 2389.008710] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2389.013107] ceph_con_process_message+0x74/0x150 [libceph] [ 2389.018603] ? calc_signature+0xdb/0x100 [libceph] [ 2389.023425] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2389.028770] ? process_scheduled_works+0x22a/0x510 [ 2389.033564] ? process_scheduled_works+0x22a/0x510 [ 2389.038371] ceph_con_workfn+0x344/0x750 [libceph] [ 2389.043175] ? process_scheduled_works+0x22a/0x510 [ 2389.047982] process_scheduled_works+0x2a4/0x510 [ 2389.052642] worker_thread+0x188/0x340 [ 2389.056390] ? rescuer_thread+0x310/0x310 [ 2389.060403] kthread+0xf5/0x130 [ 2389.063550] ? kthread_complete_and_exit+0x20/0x20 [ 2389.068360] ret_from_fork+0x31/0x40 [ 2389.071937] ? kthread_complete_and_exit+0x20/0x20 [ 2389.076747] ret_from_fork_asm+0x11/0x20 [ 2389.080714] [ 2389.082901] irq event stamp: 10836273 [ 2389.086577] hardirqs last enabled at (10836283): [] console_unlock+0x106/0x120 [ 2389.095460] hardirqs last disabled at (10836292): [] console_unlock+0xeb/0x120 [ 2389.104254] softirqs last enabled at (10836220): [] __do_softirq+0x2be/0x371 [ 2389.112949] softirqs last disabled at (10836215): [] irq_exit_rcu+0xb0/0xd0 [ 2389.121487] ---[ end trace 0000000000000000 ]--- [ 2389.126163] ------------[ cut here ]------------ [ 2389.130779] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2389.142425] WARNING: CPU: 2 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2389.151376] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2389.206837] CPU: 2 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2389.215790] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2389.223267] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2389.228753] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2389.247491] RSP: 0018:ffffc90001ff7b78 EFLAGS: 00010286 [ 2389.252718] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2389.259841] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2389.266967] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a2ed [ 2389.274100] R10: 000000000033d3a8 R11: ffffc90001ff7a08 R12: ffff8881c25d1118 [ 2389.281232] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2389.288363] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2389.296443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2389.302187] CR2: 00007fb38c57d000 CR3: 0000000120cc8004 CR4: 00000000003706e0 [ 2389.309313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2389.316445] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2389.323577] Call Trace: [ 2389.326022] [ 2389.328128] ? __warn+0x85/0x180 [ 2389.331361] ? __list_add_valid_or_report+0x7d/0xa0 [ 2389.336241] ? report_bug+0xfc/0x1d0 [ 2389.339820] ? prb_read_valid+0x1b/0x20 [ 2389.343661] ? handle_bug+0x44/0x70 [ 2389.347151] ? exc_invalid_op+0x17/0x60 [ 2389.350991] ? asm_exc_invalid_op+0x1a/0x20 [ 2389.355183] ? __list_add_valid_or_report+0x7d/0xa0 [ 2389.360057] ? __list_add_valid_or_report+0x7d/0xa0 [ 2389.364935] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2389.369826] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2389.375062] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2389.379863] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2389.384578] ? __d_lookup+0xd6/0x200 [ 2389.388157] lookup_fast+0x6d/0xf0 [ 2389.391560] walk_component+0x26/0x160 [ 2389.395314] link_path_walk+0x24d/0x380 [ 2389.399153] ? path_openat+0x80/0x9d0 [ 2389.402824] path_openat+0x8e/0x9d0 [ 2389.406315] do_filp_open+0xb8/0x120 [ 2389.409894] ? lock_release+0xc5/0x260 [ 2389.413645] ? _raw_spin_unlock+0x23/0x40 [ 2389.417656] ? alloc_fd+0x13f/0x200 [ 2389.421152] do_sys_openat2+0x249/0x290 [ 2389.424990] do_sys_open+0x4f/0x80 [ 2389.428395] do_syscall_64+0x38/0x80 [ 2389.431974] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2389.437026] RIP: 0033:0x7f48f0eed2f2 [ 2389.440599] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2389.459346] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2389.466910] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2389.474044] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2389.481175] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2389.488298] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2389.495424] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2389.502564] [ 2389.504748] irq event stamp: 12973713 [ 2389.508408] hardirqs last enabled at (12973723): [] console_unlock+0x106/0x120 [ 2389.517272] hardirqs last disabled at (12973732): [] console_unlock+0xeb/0x120 [ 2389.526044] softirqs last enabled at (12973650): [] __do_softirq+0x2be/0x371 [ 2389.534738] softirqs last disabled at (12973751): [] irq_exit_rcu+0xb0/0xd0 [ 2389.543255] ---[ end trace 0000000000000000 ]--- [ 2389.547866] ------------[ cut here ]------------ [ 2389.552485] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2389.564150] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2389.572848] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2389.628339] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2389.637468] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2389.644945] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2389.650458] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2389.655943] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2389.674689] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2389.679932] RAX: 0000000000000000 RBX: ffff8881d10b6b98 RCX: 0000000000000000 [ 2389.687082] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2389.694254] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a32c [ 2389.701384] R10: 000000000033e4d8 R11: ffffc9000030b828 R12: ffff8881d10b72e8 [ 2389.708516] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2389.715648] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2389.723752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2389.729515] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2389.736650] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2389.743798] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2389.750949] Call Trace: [ 2389.753411] [ 2389.755517] ? __warn+0x85/0x180 [ 2389.758750] ? __list_add_valid_or_report+0x7d/0xa0 [ 2389.763646] ? report_bug+0xfc/0x1d0 [ 2389.767261] ? prb_read_valid+0x1b/0x20 [ 2389.771101] ? handle_bug+0x44/0x70 [ 2389.774610] ? exc_invalid_op+0x17/0x60 [ 2389.778468] ? asm_exc_invalid_op+0x1a/0x20 [ 2389.782675] ? __list_add_valid_or_report+0x7d/0xa0 [ 2389.787550] ? __list_add_valid_or_report+0x7d/0xa0 [ 2389.792446] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2389.797337] ceph_add_cap+0x20b/0x620 [ceph] [ 2389.801639] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2389.806363] ? lock_acquire+0xcb/0x2b0 [ 2389.810140] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2389.814591] ? __mutex_lock+0x97/0xce0 [ 2389.818359] ? __mutex_lock+0xaec/0xce0 [ 2389.822200] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2389.826759] ? lock_downgrade+0xfd/0x250 [ 2389.830724] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2389.835121] ceph_con_process_message+0x74/0x150 [libceph] [ 2389.840616] ? calc_signature+0xdb/0x100 [libceph] [ 2389.845437] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2389.850785] ? process_scheduled_works+0x22a/0x510 [ 2389.855594] ? process_scheduled_works+0x22a/0x510 [ 2389.860384] ceph_con_workfn+0x344/0x750 [libceph] [ 2389.865189] ? process_scheduled_works+0x22a/0x510 [ 2389.869996] process_scheduled_works+0x2a4/0x510 [ 2389.874656] worker_thread+0x188/0x340 [ 2389.878404] ? rescuer_thread+0x310/0x310 [ 2389.882417] kthread+0xf5/0x130 [ 2389.885562] ? kthread_complete_and_exit+0x20/0x20 [ 2389.890373] ret_from_fork+0x31/0x40 [ 2389.893952] ? kthread_complete_and_exit+0x20/0x20 [ 2389.898761] ret_from_fork_asm+0x11/0x20 [ 2389.902730] [ 2389.904914] irq event stamp: 10839071 [ 2389.908589] hardirqs last enabled at (10839081): [] console_unlock+0x106/0x120 [ 2389.917474] hardirqs last disabled at (10839090): [] console_unlock+0xeb/0x120 [ 2389.926268] softirqs last enabled at (10839038): [] __do_softirq+0x2be/0x371 [ 2389.934962] softirqs last disabled at (10839033): [] irq_exit_rcu+0xb0/0xd0 [ 2389.943500] ---[ end trace 0000000000000000 ]--- [ 2389.948145] ------------[ cut here ]------------ [ 2389.952765] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2389.964429] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2389.973124] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2390.028619] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2390.037760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2390.045270] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2390.050769] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2390.056272] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2390.075019] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2390.080273] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2390.087405] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2390.094553] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a36b [ 2390.101697] R10: 000000000033f708 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2390.108845] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2390.115997] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2390.124099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2390.129861] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2390.137003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2390.144154] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2390.151304] Call Trace: [ 2390.153757] [ 2390.155873] ? __warn+0x85/0x180 [ 2390.159124] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.164019] ? report_bug+0xfc/0x1d0 [ 2390.167616] ? prb_read_valid+0x1b/0x20 [ 2390.171475] ? handle_bug+0x44/0x70 [ 2390.174964] ? exc_invalid_op+0x17/0x60 [ 2390.178814] ? asm_exc_invalid_op+0x1a/0x20 [ 2390.183039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.187915] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.192810] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2390.197718] ceph_add_cap+0x20b/0x620 [ceph] [ 2390.202038] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2390.206783] ceph_fill_trace+0x143/0xf30 [ceph] [ 2390.211324] ? __mutex_lock+0x97/0xce0 [ 2390.215073] ? __mutex_lock+0xaec/0xce0 [ 2390.218923] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2390.223484] ? lock_downgrade+0xfd/0x250 [ 2390.227413] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2390.231809] ceph_con_process_message+0x74/0x150 [libceph] [ 2390.237306] ? calc_signature+0xdb/0x100 [libceph] [ 2390.242117] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2390.247458] ? process_scheduled_works+0x22a/0x510 [ 2390.252271] ? process_scheduled_works+0x22a/0x510 [ 2390.257067] ceph_con_workfn+0x344/0x750 [libceph] [ 2390.261887] ? process_scheduled_works+0x22a/0x510 [ 2390.266693] process_scheduled_works+0x2a4/0x510 [ 2390.271336] worker_thread+0x188/0x340 [ 2390.275086] ? rescuer_thread+0x310/0x310 [ 2390.279115] kthread+0xf5/0x130 [ 2390.282277] ? kthread_complete_and_exit+0x20/0x20 [ 2390.287073] ret_from_fork+0x31/0x40 [ 2390.290668] ? kthread_complete_and_exit+0x20/0x20 [ 2390.295478] ret_from_fork_asm+0x11/0x20 [ 2390.299411] [ 2390.301605] irq event stamp: 10840135 [ 2390.305287] hardirqs last enabled at (10840145): [] console_unlock+0x106/0x120 [ 2390.314155] hardirqs last disabled at (10840154): [] console_unlock+0xeb/0x120 [ 2390.322951] softirqs last enabled at (10840092): [] __do_softirq+0x2be/0x371 [ 2390.331661] softirqs last disabled at (10840087): [] irq_exit_rcu+0xb0/0xd0 [ 2390.340198] ---[ end trace 0000000000000000 ]--- [ 2390.344888] ------------[ cut here ]------------ [ 2390.349509] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2390.361157] WARNING: CPU: 0 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2390.370119] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2390.425619] CPU: 0 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2390.434574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2390.442050] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2390.447538] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2390.466283] RSP: 0018:ffffc90001fc7d50 EFLAGS: 00010286 [ 2390.471511] RAX: 0000000000000000 RBX: ffff8881d800bab0 RCX: 0000000000000000 [ 2390.478642] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2390.485776] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a3a9 [ 2390.492910] R10: 0000000000340800 R11: ffffc90001fc7be0 R12: ffff8881d800c200 [ 2390.500041] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2390.507172] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2390.515260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2390.521007] CR2: 00007f9bd19a7000 CR3: 0000000113500005 CR4: 00000000003706f0 [ 2390.528141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2390.535271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2390.542406] Call Trace: [ 2390.544858] [ 2390.546966] ? __warn+0x85/0x180 [ 2390.550196] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.555080] ? report_bug+0xfc/0x1d0 [ 2390.558665] ? prb_read_valid+0x1b/0x20 [ 2390.562509] ? handle_bug+0x44/0x70 [ 2390.566004] ? exc_invalid_op+0x17/0x60 [ 2390.569846] ? asm_exc_invalid_op+0x1a/0x20 [ 2390.574039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.578923] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2390.583819] ceph_readdir+0xc0/0x1670 [ceph] [ 2390.588110] ? wrap_directory_iterator+0x38/0x70 [ 2390.592736] ? wrap_directory_iterator+0x30/0x70 [ 2390.597354] ? lock_is_held_type+0xaa/0x120 [ 2390.601545] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2390.606447] wrap_directory_iterator+0x47/0x70 [ 2390.610892] iterate_dir+0x8d/0x170 [ 2390.614387] __x64_sys_getdents64+0x87/0x140 [ 2390.618665] ? __x64_sys_getdents64+0x140/0x140 [ 2390.623205] do_syscall_64+0x38/0x80 [ 2390.626785] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2390.631839] RIP: 0033:0x7f48f0ec480b [ 2390.635417] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2390.654165] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2390.661730] RAX: ffffffffffffffda RBX: 000000000207e000 RCX: 00007f48f0ec480b [ 2390.668862] RDX: 0000000000010000 RSI: 000000000207e030 RDI: 0000000000000008 [ 2390.675995] RBP: 000000000207e030 R08: 00000000000000db R09: 0000000000000003 [ 2390.683130] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2390.690263] R13: 0000000000000002 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2390.697405] [ 2390.699595] irq event stamp: 12962651 [ 2390.703262] hardirqs last enabled at (12962661): [] console_unlock+0x106/0x120 [ 2390.712127] hardirqs last disabled at (12962670): [] console_unlock+0xeb/0x120 [ 2390.720907] softirqs last enabled at (12962628): [] __do_softirq+0x2be/0x371 [ 2390.729599] softirqs last disabled at (12962623): [] irq_exit_rcu+0xb0/0xd0 [ 2390.738120] ---[ end trace 0000000000000000 ]--- [ 2390.742739] ------------[ cut here ]------------ [ 2390.747357] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2390.759023] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2390.767719] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2390.823284] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2390.832416] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2390.839895] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2390.845399] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2390.850884] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2390.869631] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2390.874874] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2390.882023] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2390.889176] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a3e1 [ 2390.896326] R10: 0000000000341690 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2390.903458] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2390.910589] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2390.918693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2390.924457] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2390.931591] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2390.938740] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2390.945890] Call Trace: [ 2390.948368] [ 2390.950479] ? __warn+0x85/0x180 [ 2390.953738] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.958641] ? report_bug+0xfc/0x1d0 [ 2390.962228] ? prb_read_valid+0x1b/0x20 [ 2390.966105] ? handle_bug+0x44/0x70 [ 2390.969604] ? exc_invalid_op+0x17/0x60 [ 2390.973462] ? asm_exc_invalid_op+0x1a/0x20 [ 2390.977684] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.982591] ? __list_add_valid_or_report+0x7d/0xa0 [ 2390.987475] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2390.992375] ceph_add_cap+0x20b/0x620 [ceph] [ 2390.996699] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2391.001433] ? lock_acquire+0xcb/0x2b0 [ 2391.005217] ceph_fill_trace+0x143/0xf30 [ceph] [ 2391.009766] ? __mutex_lock+0x97/0xce0 [ 2391.013535] ? __mutex_lock+0xaec/0xce0 [ 2391.017382] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2391.021934] ? lock_downgrade+0xfd/0x250 [ 2391.025903] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2391.030314] ceph_con_process_message+0x74/0x150 [libceph] [ 2391.035817] ? calc_signature+0xdb/0x100 [libceph] [ 2391.040665] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2391.046026] ? process_scheduled_works+0x22a/0x510 [ 2391.050842] ? process_scheduled_works+0x22a/0x510 [ 2391.055638] ceph_con_workfn+0x344/0x750 [libceph] [ 2391.060470] ? process_scheduled_works+0x22a/0x510 [ 2391.065285] process_scheduled_works+0x2a4/0x510 [ 2391.069931] worker_thread+0x188/0x340 [ 2391.073702] ? rescuer_thread+0x310/0x310 [ 2391.077721] kthread+0xf5/0x130 [ 2391.080906] ? kthread_complete_and_exit+0x20/0x20 [ 2391.085723] ret_from_fork+0x31/0x40 [ 2391.089310] ? kthread_complete_and_exit+0x20/0x20 [ 2391.094102] ret_from_fork_asm+0x11/0x20 [ 2391.098078] [ 2391.100290] irq event stamp: 10842269 [ 2391.103957] hardirqs last enabled at (10842279): [] console_unlock+0x106/0x120 [ 2391.112839] hardirqs last disabled at (10842288): [] console_unlock+0xeb/0x120 [ 2391.121635] softirqs last enabled at (10842216): [] __do_softirq+0x2be/0x371 [ 2391.130347] softirqs last disabled at (10842211): [] irq_exit_rcu+0xb0/0xd0 [ 2391.138873] ---[ end trace 0000000000000000 ]--- [ 2391.143514] ------------[ cut here ]------------ [ 2391.148139] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2391.159787] WARNING: CPU: 1 PID: 78681 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2391.168750] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2391.224338] CPU: 1 PID: 78681 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2391.233296] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2391.240779] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2391.246268] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2391.265020] RSP: 0018:ffffc90001fcfd50 EFLAGS: 00010286 [ 2391.270245] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2391.277378] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2391.284512] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a420 [ 2391.291645] R10: 00000000003427b8 R11: ffffc90001fcfbe0 R12: ffff8881d106cbc8 [ 2391.298778] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2391.305912] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2391.313996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2391.319741] CR2: 000055a0beb810f0 CR3: 000000011aeae001 CR4: 00000000003706e0 [ 2391.326874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2391.334007] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2391.341140] Call Trace: [ 2391.343593] [ 2391.345699] ? __warn+0x85/0x180 [ 2391.348936] ? __list_add_valid_or_report+0x7d/0xa0 [ 2391.353821] ? report_bug+0xfc/0x1d0 [ 2391.357401] ? prb_read_valid+0x1b/0x20 [ 2391.361244] ? handle_bug+0x44/0x70 [ 2391.364740] ? exc_invalid_op+0x17/0x60 [ 2391.368581] ? asm_exc_invalid_op+0x1a/0x20 [ 2391.372778] ? __list_add_valid_or_report+0x7d/0xa0 [ 2391.377665] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2391.382565] ceph_readdir+0xc0/0x1670 [ceph] [ 2391.386864] ? wrap_directory_iterator+0x38/0x70 [ 2391.391490] ? wrap_directory_iterator+0x30/0x70 [ 2391.396119] ? lock_is_held_type+0xaa/0x120 [ 2391.400316] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2391.405214] wrap_directory_iterator+0x47/0x70 [ 2391.409663] iterate_dir+0x8d/0x170 [ 2391.413164] __x64_sys_getdents64+0x87/0x140 [ 2391.417435] ? __x64_sys_getdents64+0x140/0x140 [ 2391.421976] do_syscall_64+0x38/0x80 [ 2391.425556] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2391.430607] RIP: 0033:0x7f48f0ec480b [ 2391.434188] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2391.452936] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2391.460508] RAX: ffffffffffffffda RBX: 000000000207dfc0 RCX: 00007f48f0ec480b [ 2391.467641] RDX: 0000000000010000 RSI: 000000000207dff0 RDI: 0000000000000014 [ 2391.474775] RBP: 000000000207dff0 R08: 00000000000000db R09: 0000000000000003 [ 2391.481906] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2391.489039] R13: 0000000000000002 R14: 00007f48f183d678 R15: 00000000020767c0 [ 2391.496186] [ 2391.498382] irq event stamp: 12931685 [ 2391.502049] hardirqs last enabled at (12931695): [] console_unlock+0x106/0x120 [ 2391.510916] hardirqs last disabled at (12931704): [] console_unlock+0xeb/0x120 [ 2391.519694] softirqs last enabled at (12931606): [] __do_softirq+0x2be/0x371 [ 2391.528389] softirqs last disabled at (12931601): [] irq_exit_rcu+0xb0/0xd0 [ 2391.536915] ---[ end trace 0000000000000000 ]--- [ 2391.541535] ------------[ cut here ]------------ [ 2391.546152] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2391.557799] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2391.566494] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2391.621970] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2391.631095] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2391.638575] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2391.644069] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2391.649556] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2391.668302] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2391.673528] RAX: 0000000000000000 RBX: ffff8881c25d09c8 RCX: 0000000000000000 [ 2391.680661] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2391.687795] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a458 [ 2391.694928] R10: 0000000000343440 R11: ffffc9000030b828 R12: ffff8881c25d1118 [ 2391.702059] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2391.709191] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2391.717278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2391.723022] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2391.730148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2391.737282] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2391.744415] Call Trace: [ 2391.746867] [ 2391.748974] ? __warn+0x85/0x180 [ 2391.752207] ? __list_add_valid_or_report+0x7d/0xa0 [ 2391.757085] ? report_bug+0xfc/0x1d0 [ 2391.760665] ? prb_read_valid+0x1b/0x20 [ 2391.764505] ? handle_bug+0x44/0x70 [ 2391.767997] ? exc_invalid_op+0x17/0x60 [ 2391.771837] ? asm_exc_invalid_op+0x1a/0x20 [ 2391.776027] ? __list_add_valid_or_report+0x7d/0xa0 [ 2391.780903] ? __list_add_valid_or_report+0x7d/0xa0 [ 2391.785780] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2391.790672] ceph_add_cap+0x20b/0x620 [ceph] [ 2391.794956] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2391.799671] ? lock_acquire+0xcb/0x2b0 [ 2391.803422] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2391.807873] ? __mutex_lock+0x97/0xce0 [ 2391.811624] ? __mutex_lock+0xaec/0xce0 [ 2391.815467] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2391.820016] ? lock_downgrade+0xfd/0x250 [ 2391.823946] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2391.828327] ceph_con_process_message+0x74/0x150 [libceph] [ 2391.833823] ? calc_signature+0xdb/0x100 [libceph] [ 2391.838634] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2391.843967] ? process_scheduled_works+0x22a/0x510 [ 2391.848766] ? process_scheduled_works+0x22a/0x510 [ 2391.853556] ceph_con_workfn+0x344/0x750 [libceph] [ 2391.858361] ? process_scheduled_works+0x22a/0x510 [ 2391.863149] process_scheduled_works+0x2a4/0x510 [ 2391.867774] worker_thread+0x188/0x340 [ 2391.871524] ? rescuer_thread+0x310/0x310 [ 2391.875535] kthread+0xf5/0x130 [ 2391.878683] ? kthread_complete_and_exit+0x20/0x20 [ 2391.883475] ret_from_fork+0x31/0x40 [ 2391.887052] ? kthread_complete_and_exit+0x20/0x20 [ 2391.891846] ret_from_fork_asm+0x11/0x20 [ 2391.895778] [ 2391.897964] irq event stamp: 10844793 [ 2391.901629] hardirqs last enabled at (10844803): [] console_unlock+0x106/0x120 [ 2391.910498] hardirqs last disabled at (10844812): [] console_unlock+0xeb/0x120 [ 2391.919279] softirqs last enabled at (10844740): [] __do_softirq+0x2be/0x371 [ 2391.927971] softirqs last disabled at (10844831): [] irq_exit_rcu+0xb0/0xd0 [ 2391.936487] ---[ end trace 0000000000000000 ]--- [ 2391.941108] ------------[ cut here ]------------ [ 2391.945728] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2391.957387] WARNING: CPU: 0 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2391.966360] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2392.021838] CPU: 0 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2392.030800] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2392.038283] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2392.043778] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2392.062522] RSP: 0018:ffffc90001957aa0 EFLAGS: 00010286 [ 2392.067749] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2392.074880] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2392.082013] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a497 [ 2392.089146] R10: 0000000000344558 R11: ffffc90001957930 R12: ffff8881d0bb4bc8 [ 2392.096285] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2392.103421] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2392.111509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2392.117252] CR2: 00007f9bd19a7000 CR3: 000000016696c004 CR4: 00000000003706f0 [ 2392.124386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2392.131517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2392.138652] Call Trace: [ 2392.141105] [ 2392.143214] ? __warn+0x85/0x180 [ 2392.146452] ? __list_add_valid_or_report+0x7d/0xa0 [ 2392.151334] ? report_bug+0xfc/0x1d0 [ 2392.154911] ? prb_read_valid+0x1b/0x20 [ 2392.158755] ? handle_bug+0x44/0x70 [ 2392.162254] ? exc_invalid_op+0x17/0x60 [ 2392.166094] ? asm_exc_invalid_op+0x1a/0x20 [ 2392.170295] ? __list_add_valid_or_report+0x7d/0xa0 [ 2392.175177] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2392.180073] ceph_check_caps+0x999/0xae0 [ceph] [ 2392.184633] ? __lock_acquire+0xbfa/0x1e20 [ 2392.188734] ? avc_has_perm+0x57/0x320 [ 2392.192495] ? lock_acquire+0xcb/0x2b0 [ 2392.196252] ? ceph_open+0x113/0x440 [ceph] [ 2392.200455] ? find_held_lock+0x2d/0x90 [ 2392.204296] ? ceph_open+0x175/0x440 [ceph] [ 2392.208501] ? lock_release+0xc5/0x260 [ 2392.212263] ceph_open+0x3cc/0x440 [ceph] [ 2392.216298] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2392.221446] do_dentry_open+0x172/0x530 [ 2392.225295] path_openat+0x743/0x9d0 [ 2392.228881] do_filp_open+0xb8/0x120 [ 2392.232469] ? lock_release+0xc5/0x260 [ 2392.236225] ? _raw_spin_unlock+0x23/0x40 [ 2392.240237] ? alloc_fd+0x13f/0x200 [ 2392.243732] do_sys_openat2+0x249/0x290 [ 2392.247573] do_sys_open+0x4f/0x80 [ 2392.250985] do_syscall_64+0x38/0x80 [ 2392.254564] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2392.259622] RIP: 0033:0x7f48f0ef2805 [ 2392.263204] Code: 44 24 18 31 c0 41 83 e2 40 75 42 89 f0 25 00 00 41 00 3d 00 00 41 00 74 34 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 43 48 8b 4c 24 18 64 48 33 0c 25 28 00 00 00 [ 2392.281949] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 [ 2392.289520] RAX: ffffffffffffffda RBX: 000000000207da51 RCX: 00007f48f0ef2805 [ 2392.296656] RDX: 0000000000090800 RSI: 000000000207da30 RDI: 00000000ffffff9c [ 2392.303791] RBP: 000000000207da30 R08: 0000000003ffffff R09: 0000000000000000 [ 2392.310921] R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000004 [ 2392.318055] R13: 00007ffc53d60520 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2392.325198] [ 2392.327390] irq event stamp: 12854095 [ 2392.331055] hardirqs last enabled at (12854105): [] console_unlock+0x106/0x120 [ 2392.339922] hardirqs last disabled at (12854114): [] console_unlock+0xeb/0x120 [ 2392.348701] softirqs last enabled at (12854032): [] __do_softirq+0x2be/0x371 [ 2392.357398] softirqs last disabled at (12854133): [] irq_exit_rcu+0xb0/0xd0 [ 2392.365920] ---[ end trace 0000000000000000 ]--- [ 2392.370543] ------------[ cut here ]------------ [ 2392.375159] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2392.386823] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2392.395520] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2392.450995] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2392.460138] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2392.467636] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2392.473148] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2392.478648] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2392.497405] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2392.502629] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2392.509772] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2392.516914] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a4d8 [ 2392.524063] R10: 00000000003459f0 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2392.531205] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2392.538347] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2392.546433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2392.552178] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2392.559321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2392.566455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2392.573596] Call Trace: [ 2392.576066] [ 2392.578189] ? __warn+0x85/0x180 [ 2392.581440] ? __list_add_valid_or_report+0x7d/0xa0 [ 2392.586319] ? report_bug+0xfc/0x1d0 [ 2392.589896] ? prb_read_valid+0x1b/0x20 [ 2392.593748] ? handle_bug+0x44/0x70 [ 2392.597256] ? exc_invalid_op+0x17/0x60 [ 2392.601113] ? asm_exc_invalid_op+0x1a/0x20 [ 2392.605323] ? __list_add_valid_or_report+0x7d/0xa0 [ 2392.610205] ? __list_add_valid_or_report+0x7d/0xa0 [ 2392.615101] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2392.620009] ceph_add_cap+0x20b/0x620 [ceph] [ 2392.624324] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2392.629066] ceph_fill_trace+0x143/0xf30 [ceph] [ 2392.633605] ? __mutex_lock+0x97/0xce0 [ 2392.637374] ? __mutex_lock+0xaec/0xce0 [ 2392.641215] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2392.645773] ? lock_downgrade+0xfd/0x250 [ 2392.649738] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2392.654134] ceph_con_process_message+0x74/0x150 [libceph] [ 2392.659631] ? calc_signature+0xdb/0x100 [libceph] [ 2392.664451] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2392.669791] ? process_scheduled_works+0x22a/0x510 [ 2392.674583] ? process_scheduled_works+0x22a/0x510 [ 2392.679373] ceph_con_workfn+0x344/0x750 [libceph] [ 2392.684177] ? process_scheduled_works+0x22a/0x510 [ 2392.688984] process_scheduled_works+0x2a4/0x510 [ 2392.693645] worker_thread+0x188/0x340 [ 2392.697394] ? rescuer_thread+0x310/0x310 [ 2392.701406] kthread+0xf5/0x130 [ 2392.704551] ? kthread_complete_and_exit+0x20/0x20 [ 2392.709347] ret_from_fork+0x31/0x40 [ 2392.712924] ? kthread_complete_and_exit+0x20/0x20 [ 2392.717734] ret_from_fork_asm+0x11/0x20 [ 2392.721700] [ 2392.723885] irq event stamp: 10846759 [ 2392.727560] hardirqs last enabled at (10846769): [] console_unlock+0x106/0x120 [ 2392.736428] hardirqs last disabled at (10846778): [] console_unlock+0xeb/0x120 [ 2392.745206] softirqs last enabled at (10846716): [] __do_softirq+0x2be/0x371 [ 2392.753918] softirqs last disabled at (10846711): [] irq_exit_rcu+0xb0/0xd0 [ 2392.762455] ---[ end trace 0000000000000000 ]--- [ 2392.767108] ------------[ cut here ]------------ [ 2392.771727] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2392.783382] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2392.792078] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2392.847572] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2392.856697] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2392.864194] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2392.869705] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2392.875209] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2392.893974] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2392.899217] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2392.906367] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2392.913500] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a516 [ 2392.920632] R10: 0000000000346aa0 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2392.927772] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2392.934936] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2392.943020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2392.948782] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2392.955934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2392.963082] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2392.970234] Call Trace: [ 2392.972701] [ 2392.974819] ? __warn+0x85/0x180 [ 2392.978068] ? __list_add_valid_or_report+0x7d/0xa0 [ 2392.982966] ? report_bug+0xfc/0x1d0 [ 2392.986560] ? prb_read_valid+0x1b/0x20 [ 2392.990402] ? handle_bug+0x44/0x70 [ 2392.993892] ? exc_invalid_op+0x17/0x60 [ 2392.997740] ? asm_exc_invalid_op+0x1a/0x20 [ 2393.001970] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.006868] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.011746] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2393.016656] ceph_add_cap+0x20b/0x620 [ceph] [ 2393.020975] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2393.025715] ? lock_acquire+0xcb/0x2b0 [ 2393.029468] ceph_fill_trace+0x143/0xf30 [ceph] [ 2393.034011] ? __mutex_lock+0x97/0xce0 [ 2393.037781] ? __mutex_lock+0xaec/0xce0 [ 2393.041658] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2393.046217] ? lock_downgrade+0xfd/0x250 [ 2393.050162] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2393.054533] ceph_con_process_message+0x74/0x150 [libceph] [ 2393.060028] ? calc_signature+0xdb/0x100 [libceph] [ 2393.064867] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2393.070207] ? process_scheduled_works+0x22a/0x510 [ 2393.075017] ? process_scheduled_works+0x22a/0x510 [ 2393.079808] ceph_con_workfn+0x344/0x750 [libceph] [ 2393.084628] ? process_scheduled_works+0x22a/0x510 [ 2393.089435] process_scheduled_works+0x2a4/0x510 [ 2393.094077] worker_thread+0x188/0x340 [ 2393.097826] ? rescuer_thread+0x310/0x310 [ 2393.101857] kthread+0xf5/0x130 [ 2393.105020] ? kthread_complete_and_exit+0x20/0x20 [ 2393.109830] ret_from_fork+0x31/0x40 [ 2393.113427] ? kthread_complete_and_exit+0x20/0x20 [ 2393.118219] ret_from_fork_asm+0x11/0x20 [ 2393.122186] [ 2393.124371] irq event stamp: 10847901 [ 2393.128037] hardirqs last enabled at (10847911): [] console_unlock+0x106/0x120 [ 2393.136922] hardirqs last disabled at (10847920): [] console_unlock+0xeb/0x120 [ 2393.145719] softirqs last enabled at (10847848): [] __do_softirq+0x2be/0x371 [ 2393.154430] softirqs last disabled at (10847843): [] irq_exit_rcu+0xb0/0xd0 [ 2393.162948] ---[ end trace 0000000000000000 ]--- [ 2393.167788] ------------[ cut here ]------------ [ 2393.172407] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2393.184051] WARNING: CPU: 6 PID: 78682 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2393.193003] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2393.248466] CPU: 6 PID: 78682 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2393.257416] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2393.264896] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2393.270381] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2393.289129] RSP: 0018:ffffc90001fd7b20 EFLAGS: 00010282 [ 2393.294353] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2393.301485] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2393.308611] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a555 [ 2393.315744] R10: 0000000000347bb8 R11: ffffc90001fd79b0 R12: ffff8881d10b0750 [ 2393.322875] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2393.330000] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2393.338085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2393.343823] CR2: 00007f9bd22d7020 CR3: 0000000120c1e006 CR4: 00000000003706e0 [ 2393.350957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2393.358088] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2393.365214] Call Trace: [ 2393.367666] [ 2393.369764] ? __warn+0x85/0x180 [ 2393.372996] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.377876] ? report_bug+0xfc/0x1d0 [ 2393.381455] ? prb_read_valid+0x1b/0x20 [ 2393.385295] ? handle_bug+0x44/0x70 [ 2393.388787] ? exc_invalid_op+0x17/0x60 [ 2393.392626] ? asm_exc_invalid_op+0x1a/0x20 [ 2393.396816] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.401693] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2393.406584] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2393.411826] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2393.416628] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2393.421340] ? __d_lookup+0xd6/0x200 [ 2393.424919] lookup_fast+0x6d/0xf0 [ 2393.428327] walk_component+0x26/0x160 [ 2393.432080] path_lookupat.isra.40+0x58/0x120 [ 2393.436441] filename_lookup+0xd4/0x190 [ 2393.440281] ? lock_acquire+0xcb/0x2b0 [ 2393.444030] ? __might_fault+0x45/0x80 [ 2393.447784] ? find_held_lock+0x2d/0x90 [ 2393.451621] ? __might_fault+0x45/0x80 [ 2393.455377] vfs_statx+0x89/0x150 [ 2393.458696] vfs_fstatat+0x5a/0xc0 [ 2393.462101] __do_sys_newstat+0x30/0x70 [ 2393.465940] ? lock_release+0xc5/0x260 [ 2393.469694] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2393.474571] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2393.479710] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2393.484851] do_syscall_64+0x38/0x80 [ 2393.488430] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2393.493483] RIP: 0033:0x7f48f0eeca79 [ 2393.497061] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2393.515811] RSP: 002b:00007ffc53d5ffb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2393.523375] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2393.530507] RDX: 00007ffc53d5ffc0 RSI: 00007ffc53d5ffc0 RDI: 00007ffc53d60520 [ 2393.537638] RBP: 00007ffc53d60520 R08: 0000000000000000 R09: 00000000020767c0 [ 2393.544765] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000006 [ 2393.551897] R13: 00007ffc53d60520 R14: 00007f48f183d8a0 R15: 00000000020767c0 [ 2393.559035] [ 2393.561221] irq event stamp: 13016879 [ 2393.564881] hardirqs last enabled at (13016889): [] console_unlock+0x106/0x120 [ 2393.573746] hardirqs last disabled at (13016898): [] console_unlock+0xeb/0x120 [ 2393.582526] softirqs last enabled at (13016912): [] __do_softirq+0x2be/0x371 [ 2393.591219] softirqs last disabled at (13016907): [] irq_exit_rcu+0xb0/0xd0 [ 2393.599737] ---[ end trace 0000000000000000 ]--- [ 2393.604349] ------------[ cut here ]------------ [ 2393.608970] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2393.620643] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2393.629336] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2393.684805] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2393.693945] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2393.701435] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2393.706929] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2393.712431] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2393.731181] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2393.736424] RAX: 0000000000000000 RBX: ffff8881d106c478 RCX: 0000000000000000 [ 2393.743555] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2393.750691] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a597 [ 2393.757838] R10: 0000000000348dc8 R11: ffffc9000030b6a8 R12: ffff8881d106cbc8 [ 2393.764979] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2393.772120] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2393.780216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2393.785970] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2393.793113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2393.800263] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2393.807414] Call Trace: [ 2393.809866] [ 2393.811989] ? __warn+0x85/0x180 [ 2393.815240] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.820136] ? report_bug+0xfc/0x1d0 [ 2393.823733] ? prb_read_valid+0x1b/0x20 [ 2393.827591] ? handle_bug+0x44/0x70 [ 2393.831083] ? exc_invalid_op+0x17/0x60 [ 2393.834938] ? asm_exc_invalid_op+0x1a/0x20 [ 2393.839163] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.844039] ? __list_add_valid_or_report+0x7d/0xa0 [ 2393.848935] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2393.853842] ceph_check_caps+0x999/0xae0 [ceph] [ 2393.858403] ? __dev_queue_xmit+0x82e/0x11b0 [ 2393.862701] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2393.867408] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2393.872144] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2393.876977] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2393.881352] ? lock_release+0xc5/0x260 [ 2393.885105] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2393.889828] ceph_con_process_message+0x74/0x150 [libceph] [ 2393.895340] ? calc_signature+0xdb/0x100 [libceph] [ 2393.900150] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2393.905490] ? process_scheduled_works+0x22a/0x510 [ 2393.910282] ? process_scheduled_works+0x22a/0x510 [ 2393.915090] ceph_con_workfn+0x344/0x750 [libceph] [ 2393.919911] ? process_scheduled_works+0x22a/0x510 [ 2393.924719] process_scheduled_works+0x2a4/0x510 [ 2393.929360] worker_thread+0x188/0x340 [ 2393.933107] ? rescuer_thread+0x310/0x310 [ 2393.937137] kthread+0xf5/0x130 [ 2393.940302] ? kthread_complete_and_exit+0x20/0x20 [ 2393.945113] ret_from_fork+0x31/0x40 [ 2393.948690] ? kthread_complete_and_exit+0x20/0x20 [ 2393.953502] ret_from_fork_asm+0x11/0x20 [ 2393.957433] [ 2393.959620] irq event stamp: 10850719 [ 2393.963302] hardirqs last enabled at (10850729): [] console_unlock+0x106/0x120 [ 2393.972186] hardirqs last disabled at (10850738): [] console_unlock+0xeb/0x120 [ 2393.980983] softirqs last enabled at (10850676): [] __do_softirq+0x2be/0x371 [ 2393.989693] softirqs last disabled at (10850671): [] irq_exit_rcu+0xb0/0xd0 [ 2393.998232] ---[ end trace 0000000000000000 ]--- [ 2394.002889] ------------[ cut here ]------------ [ 2394.007509] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2394.019163] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2394.027854] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2394.083350] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2394.092474] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2394.099953] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2394.105463] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2394.110949] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2394.129706] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2394.134950] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2394.142099] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2394.149242] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a5d5 [ 2394.156391] R10: 0000000000349eb8 R11: ffffc9000030b828 R12: ffff8881d0bb4bc8 [ 2394.163525] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2394.170658] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2394.178759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2394.184515] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2394.191648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2394.198797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2394.205940] Call Trace: [ 2394.208411] [ 2394.210517] ? __warn+0x85/0x180 [ 2394.213751] ? __list_add_valid_or_report+0x7d/0xa0 [ 2394.218647] ? report_bug+0xfc/0x1d0 [ 2394.222242] ? prb_read_valid+0x1b/0x20 [ 2394.226101] ? handle_bug+0x44/0x70 [ 2394.229623] ? exc_invalid_op+0x17/0x60 [ 2394.233474] ? asm_exc_invalid_op+0x1a/0x20 [ 2394.237682] ? __list_add_valid_or_report+0x7d/0xa0 [ 2394.242559] ? __list_add_valid_or_report+0x7d/0xa0 [ 2394.247436] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2394.252345] ceph_add_cap+0x20b/0x620 [ceph] [ 2394.256654] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2394.261371] ? lock_acquire+0xcb/0x2b0 [ 2394.265147] ceph_fill_trace+0x143/0xf30 [ceph] [ 2394.269686] ? __mutex_lock+0x97/0xce0 [ 2394.273454] ? __mutex_lock+0xaec/0xce0 [ 2394.277295] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2394.281861] ? lock_downgrade+0xfd/0x250 [ 2394.285810] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2394.290207] ceph_con_process_message+0x74/0x150 [libceph] [ 2394.295703] ? calc_signature+0xdb/0x100 [libceph] [ 2394.300523] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2394.305863] ? process_scheduled_works+0x22a/0x510 [ 2394.310671] ? process_scheduled_works+0x22a/0x510 [ 2394.315462] ceph_con_workfn+0x344/0x750 [libceph] [ 2394.320266] ? process_scheduled_works+0x22a/0x510 [ 2394.325075] process_scheduled_works+0x2a4/0x510 [ 2394.329733] worker_thread+0x188/0x340 [ 2394.333481] ? rescuer_thread+0x310/0x310 [ 2394.337494] kthread+0xf5/0x130 [ 2394.340642] ? kthread_complete_and_exit+0x20/0x20 [ 2394.345450] ret_from_fork+0x31/0x40 [ 2394.349027] ? kthread_complete_and_exit+0x20/0x20 [ 2394.353829] ret_from_fork_asm+0x11/0x20 [ 2394.357798] [ 2394.359984] irq event stamp: 10851833 [ 2394.363668] hardirqs last enabled at (10851843): [] console_unlock+0x106/0x120 [ 2394.372550] hardirqs last disabled at (10851852): [] console_unlock+0xeb/0x120 [ 2394.381348] softirqs last enabled at (10851780): [] __do_softirq+0x2be/0x371 [ 2394.390041] softirqs last disabled at (10851775): [] irq_exit_rcu+0xb0/0xd0 [ 2394.398578] ---[ end trace 0000000000000000 ]--- [ 2394.403230] ------------[ cut here ]------------ [ 2394.407845] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2394.419486] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2394.428184] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2394.483660] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2394.492786] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2394.500265] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2394.505760] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2394.511248] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2394.529992] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2394.535218] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2394.542352] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2394.549486] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a614 [ 2394.556617] R10: 000000000034afd0 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2394.563751] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2394.570882] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2394.578969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2394.584714] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2394.591848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2394.598979] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2394.606113] Call Trace: [ 2394.608566] [ 2394.610673] ? __warn+0x85/0x180 [ 2394.613907] ? __list_add_valid_or_report+0x7d/0xa0 [ 2394.618786] ? report_bug+0xfc/0x1d0 [ 2394.622364] ? prb_read_valid+0x1b/0x20 [ 2394.626204] ? handle_bug+0x44/0x70 [ 2394.629696] ? exc_invalid_op+0x17/0x60 [ 2394.633534] ? asm_exc_invalid_op+0x1a/0x20 [ 2394.637727] ? __list_add_valid_or_report+0x7d/0xa0 [ 2394.642611] ? __list_add_valid_or_report+0x7d/0xa0 [ 2394.647489] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2394.652378] ceph_add_cap+0x20b/0x620 [ceph] [ 2394.656666] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2394.661389] ? lock_acquire+0xcb/0x2b0 [ 2394.665148] ceph_fill_trace+0x143/0xf30 [ceph] [ 2394.669684] ? __mutex_lock+0x97/0xce0 [ 2394.673438] ? __mutex_lock+0xaec/0xce0 [ 2394.677279] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2394.681828] ? lock_downgrade+0xfd/0x250 [ 2394.685759] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2394.690138] ceph_con_process_message+0x74/0x150 [libceph] [ 2394.695632] ? calc_signature+0xdb/0x100 [libceph] [ 2394.700436] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2394.705759] ? process_scheduled_works+0x22a/0x510 [ 2394.710550] ? process_scheduled_works+0x22a/0x510 [ 2394.715340] ceph_con_workfn+0x344/0x750 [libceph] [ 2394.720146] ? process_scheduled_works+0x22a/0x510 [ 2394.724937] process_scheduled_works+0x2a4/0x510 [ 2394.729559] worker_thread+0x188/0x340 [ 2394.733310] ? rescuer_thread+0x310/0x310 [ 2394.737320] kthread+0xf5/0x130 [ 2394.740477] ? kthread_complete_and_exit+0x20/0x20 [ 2394.745269] ret_from_fork+0x31/0x40 [ 2394.748846] ? kthread_complete_and_exit+0x20/0x20 [ 2394.753640] ret_from_fork_asm+0x11/0x20 [ 2394.757573] [ 2394.759759] irq event stamp: 10852947 [ 2394.763424] hardirqs last enabled at (10852957): [] console_unlock+0x106/0x120 [ 2394.772293] hardirqs last disabled at (10852972): [] console_unlock+0xeb/0x120 [ 2394.781070] softirqs last enabled at (10852970): [] __do_softirq+0x2be/0x371 [ 2394.789764] softirqs last disabled at (10852965): [] irq_exit_rcu+0xb0/0xd0 [ 2394.798284] ---[ end trace 0000000000000000 ]--- [ 2394.802902] ------------[ cut here ]------------ [ 2394.807522] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2394.819171] WARNING: CPU: 4 PID: 78679 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2394.828135] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2394.883613] CPU: 4 PID: 78679 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2394.892572] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2394.900048] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2394.905540] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2394.924294] RSP: 0018:ffffc90001957d50 EFLAGS: 00010286 [ 2394.929527] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2394.936661] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [ 2394.943792] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a653 [ 2394.950926] R10: 000000000034c0e0 R11: ffffc90001957be0 R12: ffff8881d0bb4bc8 [ 2394.958057] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2394.965188] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [ 2394.973290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2394.979042] CR2: 00007f9bd1a78000 CR3: 000000016696c002 CR4: 00000000003706e0 [ 2394.986173] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2394.993305] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2395.000445] Call Trace: [ 2395.002899] [ 2395.005008] ? __warn+0x85/0x180 [ 2395.008246] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.013131] ? report_bug+0xfc/0x1d0 [ 2395.016717] ? prb_read_valid+0x1b/0x20 [ 2395.020559] ? handle_bug+0x44/0x70 [ 2395.024056] ? exc_invalid_op+0x17/0x60 [ 2395.027896] ? asm_exc_invalid_op+0x1a/0x20 [ 2395.032090] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.036972] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2395.041869] ceph_readdir+0xc0/0x1670 [ceph] [ 2395.046166] ? wrap_directory_iterator+0x38/0x70 [ 2395.050795] ? wrap_directory_iterator+0x30/0x70 [ 2395.055421] ? lock_is_held_type+0xaa/0x120 [ 2395.059612] ? ceph_d_revalidate+0x6d0/0x6d0 [ceph] [ 2395.064514] wrap_directory_iterator+0x47/0x70 [ 2395.068970] iterate_dir+0x8d/0x170 [ 2395.072473] __x64_sys_getdents64+0x87/0x140 [ 2395.076748] ? __x64_sys_getdents64+0x140/0x140 [ 2395.081291] do_syscall_64+0x38/0x80 [ 2395.084871] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2395.089924] RIP: 0033:0x7f48f0ec480b [ 2395.093502] Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 a6 2f 00 f7 d8 [ 2395.112250] RSP: 002b:00007ffc53d5ffa8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [ 2395.119823] RAX: ffffffffffffffda RBX: 000000000207e080 RCX: 00007f48f0ec480b [ 2395.126957] RDX: 0000000000010000 RSI: 000000000207e0b0 RDI: 0000000000000007 [ 2395.134088] RBP: 000000000207e0b0 R08: 00000000000000db R09: 0000000000000003 [ 2395.141222] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff80 [ 2395.148361] R13: 0000000000000002 R14: 00007f48f183d228 R15: 00000000020766e0 [ 2395.155512] [ 2395.157705] irq event stamp: 12856171 [ 2395.161373] hardirqs last enabled at (12856181): [] console_unlock+0x106/0x120 [ 2395.170245] hardirqs last disabled at (12856190): [] console_unlock+0xeb/0x120 [ 2395.179027] softirqs last enabled at (12856158): [] __do_softirq+0x2be/0x371 [ 2395.187726] softirqs last disabled at (12856149): [] irq_exit_rcu+0xb0/0xd0 [ 2395.196248] ---[ end trace 0000000000000000 ]--- [ 2395.200868] ------------[ cut here ]------------ [ 2395.205484] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2395.217130] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2395.226086] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2395.281554] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2395.290504] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2395.297978] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2395.303462] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2395.322208] RSP: 0018:ffffc90001ff7ab8 EFLAGS: 00010286 [ 2395.327436] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2395.334567] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2395.341691] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a68b [ 2395.348816] R10: 000000000034cea8 R11: ffffc90001ff7948 R12: ffff8881d1329ae0 [ 2395.355947] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2395.363073] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2395.371158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2395.376905] CR2: 00007f9bd22d7020 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2395.384037] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2395.391171] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2395.398304] Call Trace: [ 2395.400758] [ 2395.402864] ? __warn+0x85/0x180 [ 2395.406094] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.410974] ? report_bug+0xfc/0x1d0 [ 2395.414554] ? prb_read_valid+0x1b/0x20 [ 2395.418396] ? handle_bug+0x44/0x70 [ 2395.421886] ? exc_invalid_op+0x17/0x60 [ 2395.425725] ? asm_exc_invalid_op+0x1a/0x20 [ 2395.429916] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.434793] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.439669] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2395.444561] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2395.449804] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2395.454606] ceph_atomic_open+0x28d/0xe80 [ceph] [ 2395.459239] lookup_open+0x37b/0x630 [ 2395.462823] ? verify_cpu+0x20/0x100 [ 2395.466402] ? verify_cpu+0x20/0x100 [ 2395.469980] path_openat+0x25a/0x9d0 [ 2395.473562] do_filp_open+0xb8/0x120 [ 2395.477142] ? lock_release+0xc5/0x260 [ 2395.480893] ? _raw_spin_unlock+0x23/0x40 [ 2395.484903] ? alloc_fd+0x13f/0x200 [ 2395.488399] do_sys_openat2+0x249/0x290 [ 2395.492237] do_sys_open+0x4f/0x80 [ 2395.495642] do_syscall_64+0x38/0x80 [ 2395.499221] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2395.504273] RIP: 0033:0x7f48f0eed2f2 [ 2395.507851] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2395.526598] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2395.534164] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2395.541296] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2395.548432] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2395.555563] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2395.562694] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2395.569827] [ 2395.572011] irq event stamp: 12975659 [ 2395.575670] hardirqs last enabled at (12975669): [] console_unlock+0x106/0x120 [ 2395.584536] hardirqs last disabled at (12975678): [] console_unlock+0xeb/0x120 [ 2395.593315] softirqs last enabled at (12975626): [] __do_softirq+0x2be/0x371 [ 2395.602009] softirqs last disabled at (12975621): [] irq_exit_rcu+0xb0/0xd0 [ 2395.610527] ---[ end trace 0000000000000000 ]--- [ 2395.615140] ------------[ cut here ]------------ [ 2395.619759] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2395.631408] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2395.640102] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2395.695595] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2395.704719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2395.712209] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2395.717720] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2395.723222] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2395.741979] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2395.747221] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2395.754373] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2395.761506] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a6c8 [ 2395.768652] R10: 000000000034e108 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2395.775778] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2395.782920] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2395.791023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2395.796779] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2395.803928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2395.811080] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2395.818229] Call Trace: [ 2395.820699] [ 2395.822814] ? __warn+0x85/0x180 [ 2395.826066] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.830961] ? report_bug+0xfc/0x1d0 [ 2395.834558] ? prb_read_valid+0x1b/0x20 [ 2395.838399] ? handle_bug+0x44/0x70 [ 2395.841889] ? exc_invalid_op+0x17/0x60 [ 2395.845738] ? asm_exc_invalid_op+0x1a/0x20 [ 2395.849946] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.854821] ? __list_add_valid_or_report+0x7d/0xa0 [ 2395.859716] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2395.864625] ceph_add_cap+0x20b/0x620 [ceph] [ 2395.868926] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2395.873642] ? lock_acquire+0xcb/0x2b0 [ 2395.877393] ceph_fill_trace+0x143/0xf30 [ceph] [ 2395.881932] ? __mutex_lock+0x97/0xce0 [ 2395.885683] ? __mutex_lock+0xaec/0xce0 [ 2395.889527] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2395.894074] ? lock_downgrade+0xfd/0x250 [ 2395.898004] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2395.902384] ceph_con_process_message+0x74/0x150 [libceph] [ 2395.907880] ? calc_signature+0xdb/0x100 [libceph] [ 2395.912693] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2395.918023] ? process_scheduled_works+0x22a/0x510 [ 2395.922815] ? process_scheduled_works+0x22a/0x510 [ 2395.927604] ceph_con_workfn+0x344/0x750 [libceph] [ 2395.932408] ? process_scheduled_works+0x22a/0x510 [ 2395.937199] process_scheduled_works+0x2a4/0x510 [ 2395.941863] worker_thread+0x188/0x340 [ 2395.945617] ? rescuer_thread+0x310/0x310 [ 2395.949626] kthread+0xf5/0x130 [ 2395.952774] ? kthread_complete_and_exit+0x20/0x20 [ 2395.957584] ret_from_fork+0x31/0x40 [ 2395.961162] ? kthread_complete_and_exit+0x20/0x20 [ 2395.965972] ret_from_fork_asm+0x11/0x20 [ 2395.969939] [ 2395.972126] irq event stamp: 10855865 [ 2395.975809] hardirqs last enabled at (10855875): [] console_unlock+0x106/0x120 [ 2395.984694] hardirqs last disabled at (10855884): [] console_unlock+0xeb/0x120 [ 2395.993490] softirqs last enabled at (10855812): [] __do_softirq+0x2be/0x371 [ 2396.002182] softirqs last disabled at (10855807): [] irq_exit_rcu+0xb0/0xd0 [ 2396.010719] ---[ end trace 0000000000000000 ]--- [ 2396.016299] ------------[ cut here ]------------ [ 2396.021062] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2396.032756] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2396.041455] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2396.096940] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2396.106082] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2396.113579] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2396.119082] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2396.124584] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2396.143330] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2396.148573] RAX: 0000000000000000 RBX: ffff8881d0951390 RCX: 0000000000000000 [ 2396.155708] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2396.162858] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a707 [ 2396.170006] R10: 000000000034f530 R11: ffffc9000030b828 R12: ffff8881d0951ae0 [ 2396.177149] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2396.184298] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2396.192402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2396.198149] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2396.205297] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2396.212440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2396.219574] Call Trace: [ 2396.222025] [ 2396.224149] ? __warn+0x85/0x180 [ 2396.227400] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.232280] ? report_bug+0xfc/0x1d0 [ 2396.235874] ? prb_read_valid+0x1b/0x20 [ 2396.239733] ? handle_bug+0x44/0x70 [ 2396.243242] ? exc_invalid_op+0x17/0x60 [ 2396.247099] ? asm_exc_invalid_op+0x1a/0x20 [ 2396.251323] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.256200] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.261096] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2396.266003] ceph_add_cap+0x20b/0x620 [ceph] [ 2396.270321] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2396.275059] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2396.280391] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2396.284875] ceph_con_process_message+0x74/0x150 [libceph] [ 2396.290385] ? calc_signature+0xdb/0x100 [libceph] [ 2396.295195] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2396.300544] ? process_scheduled_works+0x22a/0x510 [ 2396.305364] ? process_scheduled_works+0x22a/0x510 [ 2396.310156] ceph_con_workfn+0x344/0x750 [libceph] [ 2396.314962] ? process_scheduled_works+0x22a/0x510 [ 2396.319769] process_scheduled_works+0x2a4/0x510 [ 2396.324412] worker_thread+0x188/0x340 [ 2396.328172] ? rescuer_thread+0x310/0x310 [ 2396.332198] kthread+0xf5/0x130 [ 2396.335380] ? kthread_complete_and_exit+0x20/0x20 [ 2396.340172] ret_from_fork+0x31/0x40 [ 2396.343767] ? kthread_complete_and_exit+0x20/0x20 [ 2396.348580] ret_from_fork_asm+0x11/0x20 [ 2396.352513] [ 2396.354708] irq event stamp: 10857745 [ 2396.358389] hardirqs last enabled at (10857755): [] console_unlock+0x106/0x120 [ 2396.367255] hardirqs last disabled at (10857764): [] console_unlock+0xeb/0x120 [ 2396.376052] softirqs last enabled at (10857742): [] __do_softirq+0x2be/0x371 [ 2396.384762] softirqs last disabled at (10857737): [] irq_exit_rcu+0xb0/0xd0 [ 2396.393299] ---[ end trace 0000000000000000 ]--- [ 2396.397963] ------------[ cut here ]------------ [ 2396.402582] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2396.414241] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2396.422933] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2396.478424] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2396.487551] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2396.495031] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2396.500542] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2396.506027] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2396.524785] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2396.530027] RAX: 0000000000000000 RBX: ffff8881d1393ab0 RCX: 0000000000000000 [ 2396.537176] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2396.544319] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a741 [ 2396.551471] R10: 0000000000350580 R11: ffffc9000030b828 R12: ffff8881d1394200 [ 2396.558602] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2396.565733] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2396.573830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2396.579592] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2396.586726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2396.593874] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2396.601016] Call Trace: [ 2396.603479] [ 2396.605586] ? __warn+0x85/0x180 [ 2396.608818] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.613714] ? report_bug+0xfc/0x1d0 [ 2396.617311] ? prb_read_valid+0x1b/0x20 [ 2396.621168] ? handle_bug+0x44/0x70 [ 2396.624677] ? exc_invalid_op+0x17/0x60 [ 2396.628534] ? asm_exc_invalid_op+0x1a/0x20 [ 2396.632741] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.637618] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.642497] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2396.647391] ceph_add_cap+0x20b/0x620 [ceph] [ 2396.651697] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2396.656419] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2396.661762] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2396.666241] ceph_con_process_message+0x74/0x150 [libceph] [ 2396.671739] ? calc_signature+0xdb/0x100 [libceph] [ 2396.676558] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2396.681900] ? process_scheduled_works+0x22a/0x510 [ 2396.686690] ? process_scheduled_works+0x22a/0x510 [ 2396.691497] ceph_con_workfn+0x344/0x750 [libceph] [ 2396.696303] ? process_scheduled_works+0x22a/0x510 [ 2396.701110] process_scheduled_works+0x2a4/0x510 [ 2396.705768] worker_thread+0x188/0x340 [ 2396.709517] ? rescuer_thread+0x310/0x310 [ 2396.713529] kthread+0xf5/0x130 [ 2396.716677] ? kthread_complete_and_exit+0x20/0x20 [ 2396.721488] ret_from_fork+0x31/0x40 [ 2396.725064] ? kthread_complete_and_exit+0x20/0x20 [ 2396.729876] ret_from_fork_asm+0x11/0x20 [ 2396.733842] [ 2396.736029] irq event stamp: 10858759 [ 2396.739711] hardirqs last enabled at (10858769): [] console_unlock+0x106/0x120 [ 2396.748596] hardirqs last disabled at (10858784): [] console_unlock+0xeb/0x120 [ 2396.757392] softirqs last enabled at (10858782): [] __do_softirq+0x2be/0x371 [ 2396.766086] softirqs last disabled at (10858777): [] irq_exit_rcu+0xb0/0xd0 [ 2396.774623] ---[ end trace 0000000000000000 ]--- [ 2396.779264] ------------[ cut here ]------------ [ 2396.783877] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2396.795531] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2396.804228] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2396.859723] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2396.868867] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2396.876384] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2396.881882] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2396.887398] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2396.906140] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2396.911394] RAX: 0000000000000000 RBX: ffff8881d096a720 RCX: 0000000000000000 [ 2396.918527] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2396.925674] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a77b [ 2396.932817] R10: 00000000003515d0 R11: ffffc9000030b828 R12: ffff8881d096ae70 [ 2396.939959] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2396.947109] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2396.955213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2396.960975] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2396.968119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2396.975269] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2396.982417] Call Trace: [ 2396.984871] [ 2396.986995] ? __warn+0x85/0x180 [ 2396.990245] ? __list_add_valid_or_report+0x7d/0xa0 [ 2396.995141] ? report_bug+0xfc/0x1d0 [ 2396.998736] ? prb_read_valid+0x1b/0x20 [ 2397.002596] ? handle_bug+0x44/0x70 [ 2397.006087] ? exc_invalid_op+0x17/0x60 [ 2397.009943] ? asm_exc_invalid_op+0x1a/0x20 [ 2397.014168] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.019043] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.023940] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2397.028848] ceph_add_cap+0x20b/0x620 [ceph] [ 2397.033149] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2397.037888] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2397.043234] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2397.047713] ceph_con_process_message+0x74/0x150 [libceph] [ 2397.053210] ? calc_signature+0xdb/0x100 [libceph] [ 2397.058056] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2397.063401] ? process_scheduled_works+0x22a/0x510 [ 2397.068214] ? process_scheduled_works+0x22a/0x510 [ 2397.073003] ceph_con_workfn+0x344/0x750 [libceph] [ 2397.077826] ? process_scheduled_works+0x22a/0x510 [ 2397.082631] process_scheduled_works+0x2a4/0x510 [ 2397.087273] worker_thread+0x188/0x340 [ 2397.091024] ? rescuer_thread+0x310/0x310 [ 2397.095052] kthread+0xf5/0x130 [ 2397.098217] ? kthread_complete_and_exit+0x20/0x20 [ 2397.103046] ret_from_fork+0x31/0x40 [ 2397.106621] ? kthread_complete_and_exit+0x20/0x20 [ 2397.111415] ret_from_fork_asm+0x11/0x20 [ 2397.115371] [ 2397.117559] irq event stamp: 10859799 [ 2397.121226] hardirqs last enabled at (10859809): [] console_unlock+0x106/0x120 [ 2397.130111] hardirqs last disabled at (10859824): [] console_unlock+0xeb/0x120 [ 2397.138907] softirqs last enabled at (10859822): [] __do_softirq+0x2be/0x371 [ 2397.147618] softirqs last disabled at (10859817): [] irq_exit_rcu+0xb0/0xd0 [ 2397.156135] ---[ end trace 0000000000000000 ]--- [ 2397.160789] ------------[ cut here ]------------ [ 2397.165409] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2397.177069] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2397.185762] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2397.241261] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2397.250406] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2397.257885] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2397.263399] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2397.268881] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2397.287647] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2397.292871] RAX: 0000000000000000 RBX: ffff8881d8075808 RCX: 0000000000000000 [ 2397.300015] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2397.307163] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a7b5 [ 2397.314306] R10: 0000000000352620 R11: ffffc9000030b828 R12: ffff8881d8075f58 [ 2397.321455] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2397.328587] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2397.336690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2397.342446] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2397.349580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2397.356712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2397.363863] Call Trace: [ 2397.366331] [ 2397.368446] ? __warn+0x85/0x180 [ 2397.371694] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.376587] ? report_bug+0xfc/0x1d0 [ 2397.380164] ? prb_read_valid+0x1b/0x20 [ 2397.384022] ? handle_bug+0x44/0x70 [ 2397.387534] ? exc_invalid_op+0x17/0x60 [ 2397.391394] ? asm_exc_invalid_op+0x1a/0x20 [ 2397.395587] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.400463] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.405343] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2397.410257] ceph_add_cap+0x20b/0x620 [ceph] [ 2397.414545] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2397.419288] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2397.424618] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2397.429096] ceph_con_process_message+0x74/0x150 [libceph] [ 2397.434591] ? calc_signature+0xdb/0x100 [libceph] [ 2397.439408] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2397.444752] ? process_scheduled_works+0x22a/0x510 [ 2397.449544] ? process_scheduled_works+0x22a/0x510 [ 2397.454334] ceph_con_workfn+0x344/0x750 [libceph] [ 2397.459156] ? process_scheduled_works+0x22a/0x510 [ 2397.463963] process_scheduled_works+0x2a4/0x510 [ 2397.468606] worker_thread+0x188/0x340 [ 2397.472355] ? rescuer_thread+0x310/0x310 [ 2397.476402] kthread+0xf5/0x130 [ 2397.479547] ? kthread_complete_and_exit+0x20/0x20 [ 2397.484340] ret_from_fork+0x31/0x40 [ 2397.487936] ? kthread_complete_and_exit+0x20/0x20 [ 2397.492748] ret_from_fork_asm+0x11/0x20 [ 2397.496713] [ 2397.498900] irq event stamp: 10860827 [ 2397.502586] hardirqs last enabled at (10860843): [] console_unlock+0x106/0x120 [ 2397.511450] hardirqs last disabled at (10860852): [] console_unlock+0xeb/0x120 [ 2397.520230] softirqs last enabled at (10860840): [] __do_softirq+0x2be/0x371 [ 2397.528939] softirqs last disabled at (10860835): [] irq_exit_rcu+0xb0/0xd0 [ 2397.537477] ---[ end trace 0000000000000000 ]--- [ 2397.542121] ------------[ cut here ]------------ [ 2397.546741] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2397.558410] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2397.567102] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2397.622594] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2397.631721] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2397.639215] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2397.644728] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2397.650231] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2397.668995] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2397.674237] RAX: 0000000000000000 RBX: ffff8881d1294e40 RCX: 0000000000000000 [ 2397.681398] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2397.688530] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a7ef [ 2397.695665] R10: 0000000000353670 R11: ffffc9000030b828 R12: ffff8881d1295590 [ 2397.702797] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2397.709944] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2397.718041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2397.723802] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2397.730946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2397.738097] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2397.745245] Call Trace: [ 2397.747706] [ 2397.749822] ? __warn+0x85/0x180 [ 2397.753055] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.757951] ? report_bug+0xfc/0x1d0 [ 2397.761550] ? prb_read_valid+0x1b/0x20 [ 2397.765407] ? handle_bug+0x44/0x70 [ 2397.768897] ? exc_invalid_op+0x17/0x60 [ 2397.772755] ? asm_exc_invalid_op+0x1a/0x20 [ 2397.776978] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.781855] ? __list_add_valid_or_report+0x7d/0xa0 [ 2397.786751] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2397.791659] ceph_add_cap+0x20b/0x620 [ceph] [ 2397.795960] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2397.800681] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2397.806027] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2397.810489] ceph_con_process_message+0x74/0x150 [libceph] [ 2397.815984] ? calc_signature+0xdb/0x100 [libceph] [ 2397.820824] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2397.826174] ? process_scheduled_works+0x22a/0x510 [ 2397.830989] ? process_scheduled_works+0x22a/0x510 [ 2397.835780] ceph_con_workfn+0x344/0x750 [libceph] [ 2397.840600] ? process_scheduled_works+0x22a/0x510 [ 2397.845407] process_scheduled_works+0x2a4/0x510 [ 2397.850050] worker_thread+0x188/0x340 [ 2397.853799] ? rescuer_thread+0x310/0x310 [ 2397.857828] kthread+0xf5/0x130 [ 2397.860991] ? kthread_complete_and_exit+0x20/0x20 [ 2397.865803] ret_from_fork+0x31/0x40 [ 2397.869408] ? kthread_complete_and_exit+0x20/0x20 [ 2397.874200] ret_from_fork_asm+0x11/0x20 [ 2397.878165] [ 2397.880352] irq event stamp: 10861847 [ 2397.884026] hardirqs last enabled at (10861857): [] console_unlock+0x106/0x120 [ 2397.892914] hardirqs last disabled at (10861872): [] console_unlock+0xeb/0x120 [ 2397.901709] softirqs last enabled at (10861886): [] __do_softirq+0x2be/0x371 [ 2397.910417] softirqs last disabled at (10861881): [] irq_exit_rcu+0xb0/0xd0 [ 2397.918939] ---[ end trace 0000000000000000 ]--- [ 2397.923719] ------------[ cut here ]------------ [ 2397.928331] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2397.940002] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2397.948693] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2398.004187] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2398.013328] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2398.020824] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2398.026336] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2398.031840] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2398.050605] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2398.055829] RAX: 0000000000000000 RBX: ffff8881d1291390 RCX: 0000000000000000 [ 2398.062971] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2398.070114] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a829 [ 2398.077264] R10: 00000000003546c0 R11: ffffc9000030b828 R12: ffff8881d1291ae0 [ 2398.084413] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2398.091545] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2398.099631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2398.105379] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2398.112526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2398.119653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2398.126786] Call Trace: [ 2398.129254] [ 2398.131369] ? __warn+0x85/0x180 [ 2398.134622] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.139500] ? report_bug+0xfc/0x1d0 [ 2398.143078] ? prb_read_valid+0x1b/0x20 [ 2398.146938] ? handle_bug+0x44/0x70 [ 2398.150445] ? exc_invalid_op+0x17/0x60 [ 2398.154286] ? asm_exc_invalid_op+0x1a/0x20 [ 2398.158495] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.163377] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.168272] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2398.173164] ceph_add_cap+0x20b/0x620 [ceph] [ 2398.177466] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2398.182213] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2398.187541] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2398.192021] ceph_con_process_message+0x74/0x150 [libceph] [ 2398.197517] ? calc_signature+0xdb/0x100 [libceph] [ 2398.202328] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2398.207668] ? process_scheduled_works+0x22a/0x510 [ 2398.212460] ? process_scheduled_works+0x22a/0x510 [ 2398.217251] ceph_con_workfn+0x344/0x750 [libceph] [ 2398.222070] ? process_scheduled_works+0x22a/0x510 [ 2398.226878] process_scheduled_works+0x2a4/0x510 [ 2398.231520] worker_thread+0x188/0x340 [ 2398.235269] ? rescuer_thread+0x310/0x310 [ 2398.239298] kthread+0xf5/0x130 [ 2398.242461] ? kthread_complete_and_exit+0x20/0x20 [ 2398.247255] ret_from_fork+0x31/0x40 [ 2398.250849] ? kthread_complete_and_exit+0x20/0x20 [ 2398.255663] ret_from_fork_asm+0x11/0x20 [ 2398.259594] [ 2398.261781] irq event stamp: 10862889 [ 2398.265464] hardirqs last enabled at (10862899): [] console_unlock+0x106/0x120 [ 2398.274329] hardirqs last disabled at (10862908): [] console_unlock+0xeb/0x120 [ 2398.283126] softirqs last enabled at (10862886): [] __do_softirq+0x2be/0x371 [ 2398.291835] softirqs last disabled at (10862881): [] irq_exit_rcu+0xb0/0xd0 [ 2398.300373] ---[ end trace 0000000000000000 ]--- [ 2398.305031] ------------[ cut here ]------------ [ 2398.309648] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2398.321307] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2398.330008] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2398.385497] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2398.394625] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2398.402104] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2398.407616] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2398.413102] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2398.431867] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2398.437110] RAX: 0000000000000000 RBX: ffff8881d0bec478 RCX: 0000000000000000 [ 2398.444261] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2398.451422] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a863 [ 2398.458553] R10: 0000000000355710 R11: ffffc9000030b828 R12: ffff8881d0becbc8 [ 2398.465700] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2398.472843] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2398.480945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2398.486702] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2398.493850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2398.501002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2398.508151] Call Trace: [ 2398.510612] [ 2398.512709] ? __warn+0x85/0x180 [ 2398.515960] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.520859] ? report_bug+0xfc/0x1d0 [ 2398.524456] ? prb_read_valid+0x1b/0x20 [ 2398.528294] ? handle_bug+0x44/0x70 [ 2398.531804] ? exc_invalid_op+0x17/0x60 [ 2398.535662] ? asm_exc_invalid_op+0x1a/0x20 [ 2398.539869] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.544745] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.549641] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2398.554532] ceph_add_cap+0x20b/0x620 [ceph] [ 2398.558840] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2398.563562] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2398.568890] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2398.573368] ceph_con_process_message+0x74/0x150 [libceph] [ 2398.578864] ? calc_signature+0xdb/0x100 [libceph] [ 2398.583701] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2398.589061] ? process_scheduled_works+0x22a/0x510 [ 2398.593868] ? process_scheduled_works+0x22a/0x510 [ 2398.598658] ceph_con_workfn+0x344/0x750 [libceph] [ 2398.603463] ? process_scheduled_works+0x22a/0x510 [ 2398.608253] process_scheduled_works+0x2a4/0x510 [ 2398.612911] worker_thread+0x188/0x340 [ 2398.616663] ? rescuer_thread+0x310/0x310 [ 2398.620672] kthread+0xf5/0x130 [ 2398.623820] ? kthread_complete_and_exit+0x20/0x20 [ 2398.628630] ret_from_fork+0x31/0x40 [ 2398.632208] ? kthread_complete_and_exit+0x20/0x20 [ 2398.637019] ret_from_fork_asm+0x11/0x20 [ 2398.640987] [ 2398.643181] irq event stamp: 10863901 [ 2398.646863] hardirqs last enabled at (10863911): [] console_unlock+0x106/0x120 [ 2398.655750] hardirqs last disabled at (10863926): [] console_unlock+0xeb/0x120 [ 2398.664544] softirqs last enabled at (10863924): [] __do_softirq+0x2be/0x371 [ 2398.673238] softirqs last disabled at (10863919): [] irq_exit_rcu+0xb0/0xd0 [ 2398.681775] ---[ end trace 0000000000000000 ]--- [ 2398.686431] ------------[ cut here ]------------ [ 2398.691048] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2398.702727] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2398.711427] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2398.766910] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2398.776051] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2398.783548] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2398.789042] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2398.794547] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2398.813294] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2398.818536] RAX: 0000000000000000 RBX: ffff8881c2e430e8 RCX: 0000000000000000 [ 2398.825670] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2398.832804] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a89d [ 2398.839953] R10: 0000000000356760 R11: ffffc9000030b828 R12: ffff8881c2e43838 [ 2398.847103] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2398.854252] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2398.862348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2398.868110] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2398.875260] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2398.882429] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2398.889562] Call Trace: [ 2398.892014] [ 2398.894130] ? __warn+0x85/0x180 [ 2398.897379] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.902274] ? report_bug+0xfc/0x1d0 [ 2398.905854] ? prb_read_valid+0x1b/0x20 [ 2398.909713] ? handle_bug+0x44/0x70 [ 2398.913222] ? exc_invalid_op+0x17/0x60 [ 2398.917079] ? asm_exc_invalid_op+0x1a/0x20 [ 2398.921304] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.926179] ? __list_add_valid_or_report+0x7d/0xa0 [ 2398.931076] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2398.935983] ceph_add_cap+0x20b/0x620 [ceph] [ 2398.940303] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2398.945042] ceph_readdir_prepopulate+0x34f/0xf70 [ceph] [ 2398.950386] mds_dispatch+0x1f1b/0x1fa0 [ceph] [ 2398.954855] ceph_con_process_message+0x74/0x150 [libceph] [ 2398.960354] ? calc_signature+0xdb/0x100 [libceph] [ 2398.965199] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2398.970531] ? process_scheduled_works+0x22a/0x510 [ 2398.975342] ? process_scheduled_works+0x22a/0x510 [ 2398.980138] ceph_con_workfn+0x344/0x750 [libceph] [ 2398.984943] ? process_scheduled_works+0x22a/0x510 [ 2398.989750] process_scheduled_works+0x2a4/0x510 [ 2398.994416] worker_thread+0x188/0x340 [ 2398.998167] ? rescuer_thread+0x310/0x310 [ 2399.002177] kthread+0xf5/0x130 [ 2399.005325] ? kthread_complete_and_exit+0x20/0x20 [ 2399.010135] ret_from_fork+0x31/0x40 [ 2399.013730] ? kthread_complete_and_exit+0x20/0x20 [ 2399.018541] ret_from_fork_asm+0x11/0x20 [ 2399.022475] [ 2399.024668] irq event stamp: 10864931 [ 2399.028335] hardirqs last enabled at (10864941): [] console_unlock+0x106/0x120 [ 2399.037217] hardirqs last disabled at (10864950): [] console_unlock+0xeb/0x120 [ 2399.046015] softirqs last enabled at (10864918): [] __do_softirq+0x2be/0x371 [ 2399.054725] softirqs last disabled at (10864913): [] irq_exit_rcu+0xb0/0xd0 [ 2399.063262] ---[ end trace 0000000000000000 ]--- [ 2399.067943] ------------[ cut here ]------------ [ 2399.072563] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2399.084222] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2399.092912] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2399.148430] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2399.157560] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2399.165038] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2399.170547] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2399.176033] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2399.194799] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2399.200043] RAX: 0000000000000000 RBX: ffff8881d10b0000 RCX: 0000000000000000 [ 2399.207193] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2399.214343] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a8d7 [ 2399.221494] R10: 00000000003574a0 R11: ffffc9000030b828 R12: ffff8881d10b0750 [ 2399.228625] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2399.235759] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2399.243860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2399.249614] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2399.256750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2399.263898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2399.271039] Call Trace: [ 2399.273501] [ 2399.275608] ? __warn+0x85/0x180 [ 2399.278840] ? __list_add_valid_or_report+0x7d/0xa0 [ 2399.283737] ? report_bug+0xfc/0x1d0 [ 2399.287336] ? prb_read_valid+0x1b/0x20 [ 2399.291192] ? handle_bug+0x44/0x70 [ 2399.294715] ? exc_invalid_op+0x17/0x60 [ 2399.298566] ? asm_exc_invalid_op+0x1a/0x20 [ 2399.302774] ? __list_add_valid_or_report+0x7d/0xa0 [ 2399.307650] ? __list_add_valid_or_report+0x7d/0xa0 [ 2399.312527] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2399.317437] ceph_add_cap+0x20b/0x620 [ceph] [ 2399.321744] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2399.326462] ? lock_acquire+0xcb/0x2b0 [ 2399.330230] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2399.334682] ? __mutex_lock+0x97/0xce0 [ 2399.338435] ? __mutex_lock+0xaec/0xce0 [ 2399.342273] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2399.346832] ? lock_downgrade+0xfd/0x250 [ 2399.350798] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2399.355194] ceph_con_process_message+0x74/0x150 [libceph] [ 2399.360690] ? calc_signature+0xdb/0x100 [libceph] [ 2399.365494] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2399.370841] ? process_scheduled_works+0x22a/0x510 [ 2399.375635] ? process_scheduled_works+0x22a/0x510 [ 2399.380436] ceph_con_workfn+0x344/0x750 [libceph] [ 2399.385238] ? process_scheduled_works+0x22a/0x510 [ 2399.390027] process_scheduled_works+0x2a4/0x510 [ 2399.394669] worker_thread+0x188/0x340 [ 2399.398435] ? rescuer_thread+0x310/0x310 [ 2399.402446] kthread+0xf5/0x130 [ 2399.405592] ? kthread_complete_and_exit+0x20/0x20 [ 2399.410387] ret_from_fork+0x31/0x40 [ 2399.413991] ? kthread_complete_and_exit+0x20/0x20 [ 2399.418785] ret_from_fork_asm+0x11/0x20 [ 2399.422752] [ 2399.424937] irq event stamp: 10866103 [ 2399.428620] hardirqs last enabled at (10866113): [] console_unlock+0x106/0x120 [ 2399.437487] hardirqs last disabled at (10866122): [] console_unlock+0xeb/0x120 [ 2399.446265] softirqs last enabled at (10866070): [] __do_softirq+0x2be/0x371 [ 2399.454977] softirqs last disabled at (10866141): [] irq_exit_rcu+0xb0/0xd0 [ 2399.463514] ---[ end trace 0000000000000000 ]--- [ 2399.468184] ------------[ cut here ]------------ [ 2399.472810] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2399.484481] WARNING: CPU: 1 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2399.493443] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2399.548970] CPU: 1 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2399.557939] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2399.565440] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2399.570932] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2399.589695] RSP: 0018:ffffc90001fdfbe0 EFLAGS: 00010286 [ 2399.594921] RAX: 0000000000000000 RBX: ffff8881d0967560 RCX: 0000000000000000 [ 2399.602072] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2399.609220] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a916 [ 2399.616390] R10: 00000000003585b8 R11: ffffc90001fdfa70 R12: ffff8881d0967cb0 [ 2399.623521] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2399.630655] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2399.638742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2399.644505] CR2: 000055a0beb810f0 CR3: 0000000179402003 CR4: 00000000003706e0 [ 2399.651636] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2399.658770] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2399.665919] Call Trace: [ 2399.668390] [ 2399.670516] ? __warn+0x85/0x180 [ 2399.673747] ? __list_add_valid_or_report+0x7d/0xa0 [ 2399.678645] ? report_bug+0xfc/0x1d0 [ 2399.682231] ? prb_read_valid+0x1b/0x20 [ 2399.686110] ? handle_bug+0x44/0x70 [ 2399.689608] ? exc_invalid_op+0x17/0x60 [ 2399.693446] ? asm_exc_invalid_op+0x1a/0x20 [ 2399.697644] ? __list_add_valid_or_report+0x7d/0xa0 [ 2399.702531] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2399.707447] try_get_cap_refs+0x159/0x880 [ceph] [ 2399.712107] ? _raw_spin_unlock+0x23/0x40 [ 2399.716127] __ceph_get_caps+0xf8/0x760 [ceph] [ 2399.720592] ? __vfs_getxattr+0x8d/0xb0 [ 2399.724436] ? dentry_needs_remove_privs+0x31/0x50 [ 2399.729233] ? __file_remove_privs+0x89/0x110 [ 2399.733607] ? ceph_write_iter+0x1d5/0x770 [ceph] [ 2399.738332] ? lock_release+0xc5/0x260 [ 2399.742125] ? ceph_write_iter+0x260/0x770 [ceph] [ 2399.746843] ceph_write_iter+0x260/0x770 [ceph] [ 2399.751439] ? lock_is_held_type+0xaa/0x120 [ 2399.755635] vfs_write+0x324/0x4f0 [ 2399.759073] ksys_pwrite64+0x65/0xa0 [ 2399.762652] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2399.767811] do_syscall_64+0x38/0x80 [ 2399.771412] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2399.776472] RIP: 0033:0x7f48f0eeb93b [ 2399.780053] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2399.798835] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2399.806400] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [ 2399.813549] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [ 2399.820683] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [ 2399.827817] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [ 2399.834965] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2399.842146] [ 2399.844335] irq event stamp: 12883733 [ 2399.848019] hardirqs last enabled at (12883743): [] console_unlock+0x106/0x120 [ 2399.856901] hardirqs last disabled at (12883752): [] console_unlock+0xeb/0x120 [ 2399.865699] softirqs last enabled at (12883652): [] __do_softirq+0x2be/0x371 [ 2399.874391] softirqs last disabled at (12883647): [] irq_exit_rcu+0xb0/0xd0 [ 2399.882936] ---[ end trace 0000000000000000 ]--- [ 2399.887559] ------------[ cut here ]------------ [ 2399.892175] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2399.903823] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2399.912517] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2399.968065] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2399.977212] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2399.984697] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2399.990208] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2399.995709] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2400.014458] RSP: 0018:ffffc9000030b818 EFLAGS: 00010286 [ 2400.019684] RAX: 0000000000000000 RBX: ffff8881d0bb4478 RCX: 0000000000000000 [ 2400.026815] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2400.033984] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a953 [ 2400.041115] R10: 00000000003597b8 R11: ffffc9000030b6a8 R12: ffff8881d0bb4bc8 [ 2400.048265] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2400.055437] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2400.063530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.069275] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2400.076441] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2400.083576] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2400.090707] Call Trace: [ 2400.093162] [ 2400.095284] ? __warn+0x85/0x180 [ 2400.098536] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.103445] ? report_bug+0xfc/0x1d0 [ 2400.107028] ? prb_read_valid+0x1b/0x20 [ 2400.110905] ? handle_bug+0x44/0x70 [ 2400.114397] ? exc_invalid_op+0x17/0x60 [ 2400.118235] ? asm_exc_invalid_op+0x1a/0x20 [ 2400.122464] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.127361] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.132239] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2400.137174] ceph_check_caps+0x999/0xae0 [ceph] [ 2400.141784] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2400.146494] handle_cap_grant+0xc5b/0x1010 [ceph] [ 2400.151254] ceph_handle_caps+0x1303/0x2010 [ceph] [ 2400.156107] mds_dispatch+0x21e/0x1fa0 [ceph] [ 2400.160488] ? lock_release+0xc5/0x260 [ 2400.164268] ? __mutex_unlock_slowpath+0x43/0x2b0 [ 2400.169001] ceph_con_process_message+0x74/0x150 [libceph] [ 2400.174509] ? calc_signature+0xdb/0x100 [libceph] [ 2400.179348] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2400.184689] ? process_scheduled_works+0x22a/0x510 [ 2400.189489] ? process_scheduled_works+0x22a/0x510 [ 2400.194285] ceph_con_workfn+0x344/0x750 [libceph] [ 2400.199133] ? process_scheduled_works+0x22a/0x510 [ 2400.203930] process_scheduled_works+0x2a4/0x510 [ 2400.208580] worker_thread+0x188/0x340 [ 2400.212340] ? rescuer_thread+0x310/0x310 [ 2400.216367] kthread+0xf5/0x130 [ 2400.219532] ? kthread_complete_and_exit+0x20/0x20 [ 2400.224346] ret_from_fork+0x31/0x40 [ 2400.227947] ? kthread_complete_and_exit+0x20/0x20 [ 2400.232740] ret_from_fork_asm+0x11/0x20 [ 2400.236697] [ 2400.238892] irq event stamp: 10868143 [ 2400.242578] hardirqs last enabled at (10868153): [] console_unlock+0x106/0x120 [ 2400.251455] hardirqs last disabled at (10868162): [] console_unlock+0xeb/0x120 [ 2400.260240] softirqs last enabled at (10868110): [] __do_softirq+0x2be/0x371 [ 2400.268948] softirqs last disabled at (10868105): [] irq_exit_rcu+0xb0/0xd0 [ 2400.277486] ---[ end trace 0000000000000000 ]--- [ 2400.282105] ------------[ cut here ]------------ [ 2400.286748] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2400.298413] WARNING: CPU: 1 PID: 78683 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2400.307369] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2400.362923] CPU: 1 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2400.371894] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2400.379391] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2400.384892] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2400.403657] RSP: 0018:ffffc90001fdfbe0 EFLAGS: 00010286 [ 2400.408884] RAX: 0000000000000000 RBX: ffff8881d0967560 RCX: 0000000000000000 [ 2400.416034] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [ 2400.423185] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a990 [ 2400.430333] R10: 000000000035a820 R11: ffffc90001fdfa70 R12: ffff8881d0967cb0 [ 2400.437485] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2400.444617] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [ 2400.452704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.458460] CR2: 000055a0beb810f0 CR3: 0000000179402003 CR4: 00000000003706e0 [ 2400.465590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2400.472724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2400.479855] Call Trace: [ 2400.482327] [ 2400.484452] ? __warn+0x85/0x180 [ 2400.487683] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.492563] ? report_bug+0xfc/0x1d0 [ 2400.496140] ? prb_read_valid+0x1b/0x20 [ 2400.500005] ? handle_bug+0x44/0x70 [ 2400.503501] ? exc_invalid_op+0x17/0x60 [ 2400.507340] ? asm_exc_invalid_op+0x1a/0x20 [ 2400.511554] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.516458] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2400.521373] try_get_cap_refs+0x159/0x880 [ceph] [ 2400.526035] __ceph_get_caps+0x4a1/0x760 [ceph] [ 2400.530593] ? __file_remove_privs+0x89/0x110 [ 2400.534952] ? wake_bit_function+0x70/0x70 [ 2400.539090] ? ceph_write_iter+0x260/0x770 [ceph] [ 2400.543810] ceph_write_iter+0x260/0x770 [ceph] [ 2400.548401] ? lock_is_held_type+0xaa/0x120 [ 2400.552595] vfs_write+0x324/0x4f0 [ 2400.556029] ksys_pwrite64+0x65/0xa0 [ 2400.559608] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2400.564772] do_syscall_64+0x38/0x80 [ 2400.568354] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2400.573424] RIP: 0033:0x7f48f0eeb93b [ 2400.577028] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [ 2400.595776] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [ 2400.603358] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [ 2400.610509] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [ 2400.617642] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [ 2400.624788] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [ 2400.631923] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [ 2400.639106] [ 2400.641300] irq event stamp: 12885509 [ 2400.644985] hardirqs last enabled at (12885519): [] console_unlock+0x106/0x120 [ 2400.653869] hardirqs last disabled at (12885528): [] console_unlock+0xeb/0x120 [ 2400.662666] softirqs last enabled at (12885238): [] __do_softirq+0x2be/0x371 [ 2400.671358] softirqs last disabled at (12885233): [] irq_exit_rcu+0xb0/0xd0 [ 2400.679896] ---[ end trace 0000000000000000 ]--- [ 2400.684533] ------------[ cut here ]------------ [ 2400.689149] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2400.700812] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2400.709509] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2400.764986] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2400.774128] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2400.781624] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2400.787118] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2400.792622] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2400.811368] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2400.816612] RAX: 0000000000000000 RBX: ffff8881d1329390 RCX: 0000000000000000 [ 2400.823749] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2400.830896] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000a9c9 [ 2400.838044] R10: 000000000035b760 R11: ffffc9000030b828 R12: ffff8881d1329ae0 [ 2400.845186] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2400.852327] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2400.860424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.866185] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2400.873329] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2400.880480] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2400.887612] Call Trace: [ 2400.890064] [ 2400.892188] ? __warn+0x85/0x180 [ 2400.895458] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.900334] ? report_bug+0xfc/0x1d0 [ 2400.903930] ? prb_read_valid+0x1b/0x20 [ 2400.907790] ? handle_bug+0x44/0x70 [ 2400.911296] ? exc_invalid_op+0x17/0x60 [ 2400.915147] ? asm_exc_invalid_op+0x1a/0x20 [ 2400.919371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.924247] ? __list_add_valid_or_report+0x7d/0xa0 [ 2400.929142] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2400.934050] ceph_add_cap+0x20b/0x620 [ceph] [ 2400.938370] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2400.943102] ? lock_acquire+0xcb/0x2b0 [ 2400.946870] ceph_fill_trace+0xa8/0xf30 [ceph] [ 2400.951321] ? __mutex_lock+0x97/0xce0 [ 2400.955090] ? __mutex_lock+0xaec/0xce0 [ 2400.958941] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2400.963499] ? lock_downgrade+0xfd/0x250 [ 2400.967450] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2400.971824] ceph_con_process_message+0x74/0x150 [libceph] [ 2400.977321] ? calc_signature+0xdb/0x100 [libceph] [ 2400.982160] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2400.987482] ? process_scheduled_works+0x22a/0x510 [ 2400.992276] ? process_scheduled_works+0x22a/0x510 [ 2400.997082] ceph_con_workfn+0x344/0x750 [libceph] [ 2401.001903] ? process_scheduled_works+0x22a/0x510 [ 2401.006710] process_scheduled_works+0x2a4/0x510 [ 2401.011352] worker_thread+0x188/0x340 [ 2401.015102] ? rescuer_thread+0x310/0x310 [ 2401.019130] kthread+0xf5/0x130 [ 2401.022295] ? kthread_complete_and_exit+0x20/0x20 [ 2401.027104] ret_from_fork+0x31/0x40 [ 2401.030701] ? kthread_complete_and_exit+0x20/0x20 [ 2401.035494] ret_from_fork_asm+0x11/0x20 [ 2401.039427] [ 2401.041612] irq event stamp: 10870095 [ 2401.045277] hardirqs last enabled at (10870105): [] console_unlock+0x106/0x120 [ 2401.054145] hardirqs last disabled at (10870114): [] console_unlock+0xeb/0x120 [ 2401.062922] softirqs last enabled at (10870042): [] __do_softirq+0x2be/0x371 [ 2401.071634] softirqs last disabled at (10870037): [] irq_exit_rcu+0xb0/0xd0 [ 2401.080154] ---[ end trace 0000000000000000 ]--- [ 2401.084815] ------------[ cut here ]------------ [ 2401.089456] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2401.101113] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2401.109803] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2401.165299] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2401.174460] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2401.181937] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2401.187465] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2401.192953] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2401.211719] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2401.216942] RAX: 0000000000000000 RBX: ffff8881d0966b98 RCX: 0000000000000000 [ 2401.224091] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2401.231244] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000aa08 [ 2401.238392] R10: 000000000035c9a8 R11: ffffc9000030b828 R12: ffff8881d09672e8 [ 2401.245542] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2401.252676] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2401.260761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2401.266523] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2401.273650] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2401.280782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2401.287931] Call Trace: [ 2401.290393] [ 2401.292516] ? __warn+0x85/0x180 [ 2401.295752] ? __list_add_valid_or_report+0x7d/0xa0 [ 2401.300646] ? report_bug+0xfc/0x1d0 [ 2401.304226] ? prb_read_valid+0x1b/0x20 [ 2401.308085] ? handle_bug+0x44/0x70 [ 2401.311592] ? exc_invalid_op+0x17/0x60 [ 2401.315433] ? asm_exc_invalid_op+0x1a/0x20 [ 2401.319640] ? __list_add_valid_or_report+0x7d/0xa0 [ 2401.324517] ? __list_add_valid_or_report+0x7d/0xa0 [ 2401.329395] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2401.334301] ceph_add_cap+0x20b/0x620 [ceph] [ 2401.338603] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2401.343341] ceph_fill_trace+0x143/0xf30 [ceph] [ 2401.347880] ? __mutex_lock+0x97/0xce0 [ 2401.351651] ? __mutex_lock+0xaec/0xce0 [ 2401.355491] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2401.360033] ? lock_downgrade+0xfd/0x250 [ 2401.363996] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2401.368384] ceph_con_process_message+0x74/0x150 [libceph] [ 2401.373879] ? calc_signature+0xdb/0x100 [libceph] [ 2401.378703] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2401.384050] ? process_scheduled_works+0x22a/0x510 [ 2401.388859] ? process_scheduled_works+0x22a/0x510 [ 2401.393649] ceph_con_workfn+0x344/0x750 [libceph] [ 2401.398470] ? process_scheduled_works+0x22a/0x510 [ 2401.403260] process_scheduled_works+0x2a4/0x510 [ 2401.407921] worker_thread+0x188/0x340 [ 2401.411678] ? rescuer_thread+0x310/0x310 [ 2401.415689] kthread+0xf5/0x130 [ 2401.418836] ? kthread_complete_and_exit+0x20/0x20 [ 2401.423648] ret_from_fork+0x31/0x40 [ 2401.427226] ? kthread_complete_and_exit+0x20/0x20 [ 2401.432036] ret_from_fork_asm+0x11/0x20 [ 2401.436001] [ 2401.438188] irq event stamp: 10871173 [ 2401.441863] hardirqs last enabled at (10871183): [] console_unlock+0x106/0x120 [ 2401.450753] hardirqs last disabled at (10871192): [] console_unlock+0xeb/0x120 [ 2401.459543] softirqs last enabled at (10871130): [] __do_softirq+0x2be/0x371 [ 2401.468238] softirqs last disabled at (10871125): [] irq_exit_rcu+0xb0/0xd0 [ 2401.476772] ---[ end trace 0000000000000000 ]--- [ 2401.481402] ------------[ cut here ]------------ [ 2401.486038] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2401.497703] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2401.506397] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2401.561919] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2401.571043] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2401.578538] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2401.584034] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2401.589538] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2401.608284] RSP: 0018:ffffc9000030ba48 EFLAGS: 00010282 [ 2401.613527] RAX: 0000000000000000 RBX: ffff8881d0966b98 RCX: 0000000000000000 [ 2401.620660] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2401.627794] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000aa46 [ 2401.634944] R10: 000000000035dcd8 R11: ffffc9000030b8d8 R12: ffff8881d09672e8 [ 2401.642094] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2401.649226] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2401.657329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2401.663083] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2401.670225] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2401.677367] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2401.684518] Call Trace: [ 2401.686969] [ 2401.689085] ? __warn+0x85/0x180 [ 2401.692336] ? __list_add_valid_or_report+0x7d/0xa0 [ 2401.697233] ? report_bug+0xfc/0x1d0 [ 2401.700829] ? prb_read_valid+0x1b/0x20 [ 2401.704687] ? handle_bug+0x44/0x70 [ 2401.708178] ? exc_invalid_op+0x17/0x60 [ 2401.712035] ? asm_exc_invalid_op+0x1a/0x20 [ 2401.716260] ? __list_add_valid_or_report+0x7d/0xa0 [ 2401.721135] ? __list_add_valid_or_report+0x7d/0xa0 [ 2401.726030] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2401.730939] ceph_fill_inode+0xa88/0x1640 [ceph] [ 2401.735592] ceph_fill_trace+0x143/0xf30 [ceph] [ 2401.740134] ? __mutex_lock+0x97/0xce0 [ 2401.743901] ? __mutex_lock+0xaec/0xce0 [ 2401.747760] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2401.752318] ? lock_downgrade+0xfd/0x250 [ 2401.756284] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2401.760663] ceph_con_process_message+0x74/0x150 [libceph] [ 2401.766158] ? calc_signature+0xdb/0x100 [libceph] [ 2401.770998] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2401.776346] ? process_scheduled_works+0x22a/0x510 [ 2401.781156] ? process_scheduled_works+0x22a/0x510 [ 2401.785944] ceph_con_workfn+0x344/0x750 [libceph] [ 2401.790765] ? process_scheduled_works+0x22a/0x510 [ 2401.795572] process_scheduled_works+0x2a4/0x510 [ 2401.800215] worker_thread+0x188/0x340 [ 2401.803964] ? rescuer_thread+0x310/0x310 [ 2401.807993] kthread+0xf5/0x130 [ 2401.811158] ? kthread_complete_and_exit+0x20/0x20 [ 2401.815968] ret_from_fork+0x31/0x40 [ 2401.819564] ? kthread_complete_and_exit+0x20/0x20 [ 2401.824355] ret_from_fork_asm+0x11/0x20 [ 2401.828323] [ 2401.830510] irq event stamp: 10872213 [ 2401.834176] hardirqs last enabled at (10872223): [] console_unlock+0x106/0x120 [ 2401.843058] hardirqs last disabled at (10872232): [] console_unlock+0xeb/0x120 [ 2401.851848] softirqs last enabled at (10872180): [] __do_softirq+0x2be/0x371 [ 2401.860557] softirqs last disabled at (10872175): [] irq_exit_rcu+0xb0/0xd0 [ 2401.869077] ---[ end trace 0000000000000000 ]--- [ 2401.873725] ------------[ cut here ]------------ [ 2401.878343] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2401.889991] WARNING: CPU: 6 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2401.898942] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2401.954413] CPU: 6 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2401.963369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2401.970844] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2401.976328] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2401.995074] RSP: 0018:ffffc90001ff7aa8 EFLAGS: 00010282 [ 2402.000302] RAX: 0000000000000000 RBX: ffff8881d0966b98 RCX: 0000000000000000 [ 2402.007434] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [ 2402.014566] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000aa83 [ 2402.021700] R10: 000000000035ed98 R11: ffffc90001ff7938 R12: ffff8881d09672e8 [ 2402.028831] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2402.035966] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [ 2402.044050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2402.049798] CR2: 00007f9bd22d7020 CR3: 0000000120cc8001 CR4: 00000000003706e0 [ 2402.056931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2402.064063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2402.071195] Call Trace: [ 2402.073639] [ 2402.075737] ? __warn+0x85/0x180 [ 2402.078970] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.083850] ? report_bug+0xfc/0x1d0 [ 2402.087431] ? prb_read_valid+0x1b/0x20 [ 2402.091270] ? handle_bug+0x44/0x70 [ 2402.094760] ? exc_invalid_op+0x17/0x60 [ 2402.098600] ? asm_exc_invalid_op+0x1a/0x20 [ 2402.102791] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.107666] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.112545] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2402.117436] ceph_open+0x16b/0x440 [ceph] [ 2402.121462] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2402.126612] do_dentry_open+0x172/0x530 [ 2402.130458] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2402.135095] lookup_open+0x37b/0x630 [ 2402.138678] ? verify_cpu+0x20/0x100 [ 2402.142257] ? verify_cpu+0x20/0x100 [ 2402.145838] path_openat+0x25a/0x9d0 [ 2402.149417] do_filp_open+0xb8/0x120 [ 2402.152999] ? lock_release+0xc5/0x260 [ 2402.156749] ? _raw_spin_unlock+0x23/0x40 [ 2402.160758] ? alloc_fd+0x13f/0x200 [ 2402.164253] do_sys_openat2+0x249/0x290 [ 2402.168092] do_sys_open+0x4f/0x80 [ 2402.171498] do_syscall_64+0x38/0x80 [ 2402.175078] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2402.180128] RIP: 0033:0x7f48f0eed2f2 [ 2402.183709] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2402.202454] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2402.210020] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2402.217153] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2402.224286] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2402.231409] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2402.238534] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2402.245665] [ 2402.247851] irq event stamp: 12976739 [ 2402.251507] hardirqs last enabled at (12976749): [] console_unlock+0x106/0x120 [ 2402.260373] hardirqs last disabled at (12976758): [] console_unlock+0xeb/0x120 [ 2402.269154] softirqs last enabled at (12976686): [] __do_softirq+0x2be/0x371 [ 2402.277847] softirqs last disabled at (12976681): [] irq_exit_rcu+0xb0/0xd0 [ 2402.286367] ---[ end trace 0000000000000000 ]--- [ 2402.291001] ------------[ cut here ]------------ [ 2402.295622] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2402.307269] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2402.315965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2402.371443] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2402.380567] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2402.388047] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2402.393541] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2402.399028] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2402.417775] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2402.423001] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2402.430134] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2402.437266] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000aac1 [ 2402.444398] R10: 000000000035fe50 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2402.451532] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2402.458665] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2402.466750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2402.472496] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2402.479629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2402.486762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2402.493896] Call Trace: [ 2402.496348] [ 2402.498455] ? __warn+0x85/0x180 [ 2402.501688] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.506566] ? report_bug+0xfc/0x1d0 [ 2402.510144] ? prb_read_valid+0x1b/0x20 [ 2402.513987] ? handle_bug+0x44/0x70 [ 2402.517476] ? exc_invalid_op+0x17/0x60 [ 2402.521316] ? asm_exc_invalid_op+0x1a/0x20 [ 2402.525509] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.530392] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.535269] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2402.540161] ceph_add_cap+0x20b/0x620 [ceph] [ 2402.544445] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2402.549161] ? lock_acquire+0xcb/0x2b0 [ 2402.552920] ceph_fill_trace+0x143/0xf30 [ceph] [ 2402.557458] ? __mutex_lock+0x97/0xce0 [ 2402.561217] ? __mutex_lock+0xaec/0xce0 [ 2402.565060] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2402.569610] ? lock_downgrade+0xfd/0x250 [ 2402.573540] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2402.577919] ceph_con_process_message+0x74/0x150 [libceph] [ 2402.583415] ? calc_signature+0xdb/0x100 [libceph] [ 2402.588228] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2402.593559] ? process_scheduled_works+0x22a/0x510 [ 2402.598351] ? process_scheduled_works+0x22a/0x510 [ 2402.603141] ceph_con_workfn+0x344/0x750 [libceph] [ 2402.607946] ? process_scheduled_works+0x22a/0x510 [ 2402.612743] process_scheduled_works+0x2a4/0x510 [ 2402.617368] worker_thread+0x188/0x340 [ 2402.621126] ? rescuer_thread+0x310/0x310 [ 2402.625136] kthread+0xf5/0x130 [ 2402.628285] ? kthread_complete_and_exit+0x20/0x20 [ 2402.633077] ret_from_fork+0x31/0x40 [ 2402.636654] ? kthread_complete_and_exit+0x20/0x20 [ 2402.641448] ret_from_fork_asm+0x11/0x20 [ 2402.645381] [ 2402.647575] irq event stamp: 10874241 [ 2402.651240] hardirqs last enabled at (10874251): [] console_unlock+0x106/0x120 [ 2402.660108] hardirqs last disabled at (10874260): [] console_unlock+0xeb/0x120 [ 2402.668888] softirqs last enabled at (10874188): [] __do_softirq+0x2be/0x371 [ 2402.677579] softirqs last disabled at (10874183): [] irq_exit_rcu+0xb0/0xd0 [ 2402.686099] ---[ end trace 0000000000000000 ]--- [ 2402.690721] ------------[ cut here ]------------ [ 2402.695336] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2402.706983] WARNING: CPU: 2 PID: 78686 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2402.715939] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2402.771408] CPU: 2 PID: 78686 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2402.780359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2402.787837] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2402.793324] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2402.812069] RSP: 0018:ffffc90001ff78c0 EFLAGS: 00010282 [ 2402.817296] RAX: 0000000000000000 RBX: ffff8881d0966b98 RCX: 0000000000000000 [ 2402.824428] RDX: ffff88885fca9c00 RSI: ffff88885fc9d880 RDI: ffff88885fc9d880 [ 2402.831553] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000ab00 [ 2402.838688] R10: 0000000000360fa0 R11: ffffc90001ff7750 R12: ffff8881d09672e8 [ 2402.845819] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2402.852951] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [ 2402.861036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2402.866776] CR2: 00007fb38c57d000 CR3: 0000000120cc8004 CR4: 00000000003706e0 [ 2402.873909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2402.881041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2402.888173] Call Trace: [ 2402.890617] [ 2402.892715] ? __warn+0x85/0x180 [ 2402.895949] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.900828] ? report_bug+0xfc/0x1d0 [ 2402.904408] ? prb_read_valid+0x1b/0x20 [ 2402.908248] ? handle_bug+0x44/0x70 [ 2402.911739] ? exc_invalid_op+0x17/0x60 [ 2402.915579] ? asm_exc_invalid_op+0x1a/0x20 [ 2402.919769] ? __list_add_valid_or_report+0x7d/0xa0 [ 2402.924646] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2402.929534] ceph_check_caps+0x999/0xae0 [ceph] [ 2402.934090] ? lock_release+0xc5/0x260 [ 2402.937846] ceph_open+0x3cc/0x440 [ceph] [ 2402.941867] ? ceph_copy_file_range+0x870/0x870 [ceph] [ 2402.947014] do_dentry_open+0x172/0x530 [ 2402.950853] ceph_atomic_open+0x8d2/0xe80 [ceph] [ 2402.955488] lookup_open+0x37b/0x630 [ 2402.959071] ? verify_cpu+0x20/0x100 [ 2402.962649] ? verify_cpu+0x20/0x100 [ 2402.966230] path_openat+0x25a/0x9d0 [ 2402.969811] do_filp_open+0xb8/0x120 [ 2402.973391] ? lock_release+0xc5/0x260 [ 2402.977140] ? _raw_spin_unlock+0x23/0x40 [ 2402.981157] ? alloc_fd+0x13f/0x200 [ 2402.984646] do_sys_openat2+0x249/0x290 [ 2402.988486] do_sys_open+0x4f/0x80 [ 2402.991891] do_syscall_64+0x38/0x80 [ 2402.995476] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2403.000521] RIP: 0033:0x7f48f0eed2f2 [ 2403.004102] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 b5 72 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2403.022848] RSP: 002b:00007ffc53d5ff70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2403.030415] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eed2f2 [ 2403.037548] RDX: 0000000000000002 RSI: 00007ffc53d60520 RDI: 00000000ffffff9c [ 2403.044680] RBP: 00007ffc53d60520 R08: 0000000000000002 R09: 00000000020767c0 [ 2403.051812] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2403.058945] R13: 00007f48f183e140 R14: 00000000020767c0 R15: 00000000000029f6 [ 2403.066084] [ 2403.068271] irq event stamp: 12978699 [ 2403.071936] hardirqs last enabled at (12978709): [] console_unlock+0x106/0x120 [ 2403.080802] hardirqs last disabled at (12978718): [] console_unlock+0xeb/0x120 [ 2403.089581] softirqs last enabled at (12978646): [] __do_softirq+0x2be/0x371 [ 2403.098275] softirqs last disabled at (12978641): [] irq_exit_rcu+0xb0/0xd0 [ 2403.106795] ---[ end trace 0000000000000000 ]--- [ 2403.111414] ------------[ cut here ]------------ [ 2403.116034] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2403.127686] WARNING: CPU: 0 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2403.136645] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2403.192127] CPU: 0 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2403.201098] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2403.208581] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2403.214067] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2403.232813] RSP: 0018:ffffc90001fc7ac0 EFLAGS: 00010282 [ 2403.238040] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2403.245172] RDX: ffff88885fc29c00 RSI: ffff88885fc1d880 RDI: ffff88885fc1d880 [ 2403.252304] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000ab3f [ 2403.259438] R10: 00000000003620c0 R11: ffffc90001fc7950 R12: ffff888158a68750 [ 2403.266570] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2403.273701] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [ 2403.281790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2403.287534] CR2: 00007f9bd1ab4000 CR3: 0000000113500005 CR4: 00000000003706f0 [ 2403.294669] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2403.301799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2403.308934] Call Trace: [ 2403.311386] [ 2403.313494] ? __warn+0x85/0x180 [ 2403.316726] ? __list_add_valid_or_report+0x7d/0xa0 [ 2403.321609] ? report_bug+0xfc/0x1d0 [ 2403.325191] ? prb_read_valid+0x1b/0x20 [ 2403.329035] ? handle_bug+0x44/0x70 [ 2403.332524] ? exc_invalid_op+0x17/0x60 [ 2403.336366] ? asm_exc_invalid_op+0x1a/0x20 [ 2403.340556] ? __list_add_valid_or_report+0x7d/0xa0 [ 2403.345440] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2403.350339] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [ 2403.355592] ceph_mdsc_do_request+0x33/0xf0 [ceph] [ 2403.360398] ceph_d_revalidate+0x316/0x6d0 [ceph] [ 2403.365124] ? __d_lookup+0xd6/0x200 [ 2403.368711] lookup_fast+0x6d/0xf0 [ 2403.372116] walk_component+0x26/0x160 [ 2403.375869] link_path_walk+0x24d/0x380 [ 2403.379710] ? path_lookupat.isra.40+0x22/0x120 [ 2403.384251] path_lookupat.isra.40+0x6b/0x120 [ 2403.388616] filename_lookup+0xd4/0x190 [ 2403.392463] ? lock_acquire+0xcb/0x2b0 [ 2403.396218] ? __might_fault+0x45/0x80 [ 2403.399971] ? find_held_lock+0x2d/0x90 [ 2403.403819] ? __might_fault+0x45/0x80 [ 2403.407575] vfs_statx+0x89/0x150 [ 2403.410907] vfs_fstatat+0x5a/0xc0 [ 2403.414317] __do_sys_newstat+0x30/0x70 [ 2403.418155] ? lock_release+0xc5/0x260 [ 2403.421908] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [ 2403.426795] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2403.431942] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [ 2403.437085] do_syscall_64+0x38/0x80 [ 2403.440669] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 2403.445722] RIP: 0033:0x7f48f0eeca79 [ 2403.449300] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [ 2403.468050] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 2403.475620] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [ 2403.482755] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [ 2403.489887] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [ 2403.497022] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [ 2403.504152] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [ 2403.511297] [ 2403.513488] irq event stamp: 12964847 [ 2403.517153] hardirqs last enabled at (12964857): [] console_unlock+0x106/0x120 [ 2403.526020] hardirqs last disabled at (12964866): [] console_unlock+0xeb/0x120 [ 2403.534801] softirqs last enabled at (12964880): [] __do_softirq+0x2be/0x371 [ 2403.543498] softirqs last disabled at (12964875): [] irq_exit_rcu+0xb0/0xd0 [ 2403.552019] ---[ end trace 0000000000000000 ]--- [ 2403.556641] ------------[ cut here ]------------ [ 2403.561257] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2403.572925] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2403.581631] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2403.637112] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2403.646253] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2403.653750] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2403.659245] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2403.664749] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2403.683511] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2403.688737] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2403.695871] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2403.703022] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000ab83 [ 2403.710170] R10: 00000000003635f0 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2403.717313] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2403.724461] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2403.732565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2403.738310] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2403.745455] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2403.752604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2403.759737] Call Trace: [ 2403.762190] [ 2403.764314] ? __warn+0x85/0x180 [ 2403.767565] ? __list_add_valid_or_report+0x7d/0xa0 [ 2403.772443] ? report_bug+0xfc/0x1d0 [ 2403.776038] ? prb_read_valid+0x1b/0x20 [ 2403.779896] ? handle_bug+0x44/0x70 [ 2403.783405] ? exc_invalid_op+0x17/0x60 [ 2403.787263] ? asm_exc_invalid_op+0x1a/0x20 [ 2403.791495] ? __list_add_valid_or_report+0x7d/0xa0 [ 2403.796371] ? __list_add_valid_or_report+0x7d/0xa0 [ 2403.801250] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2403.806158] ceph_add_cap+0x20b/0x620 [ceph] [ 2403.810476] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2403.815200] ? lock_acquire+0xcb/0x2b0 [ 2403.818971] ceph_fill_trace+0x143/0xf30 [ceph] [ 2403.823517] ? __mutex_lock+0x97/0xce0 [ 2403.827269] ? __mutex_lock+0xaec/0xce0 [ 2403.831128] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2403.835694] ? lock_downgrade+0xfd/0x250 [ 2403.839623] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2403.844020] ceph_con_process_message+0x74/0x150 [libceph] [ 2403.849518] ? calc_signature+0xdb/0x100 [libceph] [ 2403.854346] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2403.859678] ? process_scheduled_works+0x22a/0x510 [ 2403.864491] ? process_scheduled_works+0x22a/0x510 [ 2403.869284] ceph_con_workfn+0x344/0x750 [libceph] [ 2403.874088] ? process_scheduled_works+0x22a/0x510 [ 2403.878897] process_scheduled_works+0x2a4/0x510 [ 2403.883539] worker_thread+0x188/0x340 [ 2403.887296] ? rescuer_thread+0x310/0x310 [ 2403.891326] kthread+0xf5/0x130 [ 2403.894511] ? kthread_complete_and_exit+0x20/0x20 [ 2403.899299] ret_from_fork+0x31/0x40 [ 2403.902894] ? kthread_complete_and_exit+0x20/0x20 [ 2403.907706] ret_from_fork_asm+0x11/0x20 [ 2403.911638] [ 2403.913824] irq event stamp: 10878081 [ 2403.917518] hardirqs last enabled at (10878091): [] console_unlock+0x106/0x120 [ 2403.926383] hardirqs last disabled at (10878100): [] console_unlock+0xeb/0x120 [ 2403.935178] softirqs last enabled at (10878028): [] __do_softirq+0x2be/0x371 [ 2403.943890] softirqs last disabled at (10878023): [] irq_exit_rcu+0xb0/0xd0 [ 2403.952427] ---[ end trace 0000000000000000 ]--- [ 2403.957344] ------------[ cut here ]------------ [ 2403.961965] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [ 2403.973606] WARNING: CPU: 5 PID: 74 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [ 2403.982303] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2404.037788] CPU: 5 PID: 74 Comm: kworker/5:1 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2404.046912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2404.054393] Workqueue: ceph-msgr ceph_con_workfn [libceph] [ 2404.059893] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [ 2404.065380] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [ 2404.084127] RSP: 0018:ffffc9000030b998 EFLAGS: 00010282 [ 2404.089353] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [ 2404.096487] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [ 2404.103620] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000abc2 [ 2404.110751] R10: 0000000000364910 R11: ffffc9000030b828 R12: ffff888158a68750 [ 2404.117885] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [ 2404.125017] FS: 0000000000000000(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [ 2404.133104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2404.138849] CR2: 00007f9bd1ff5000 CR3: 0000000113500004 CR4: 00000000003706e0 [ 2404.145983] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2404.153117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2404.160249] Call Trace: [ 2404.162701] [ 2404.164809] ? __warn+0x85/0x180 [ 2404.168042] ? __list_add_valid_or_report+0x7d/0xa0 [ 2404.172921] ? report_bug+0xfc/0x1d0 [ 2404.176499] ? prb_read_valid+0x1b/0x20 [ 2404.180339] ? handle_bug+0x44/0x70 [ 2404.183832] ? exc_invalid_op+0x17/0x60 [ 2404.187670] ? asm_exc_invalid_op+0x1a/0x20 [ 2404.191860] ? __list_add_valid_or_report+0x7d/0xa0 [ 2404.196737] ? __list_add_valid_or_report+0x7d/0xa0 [ 2404.201614] __cap_delay_requeue+0x140/0x1d0 [ceph] [ 2404.206506] ceph_add_cap+0x20b/0x620 [ceph] [ 2404.210798] ceph_fill_inode+0x10f8/0x1640 [ceph] [ 2404.215514] ? lock_acquire+0xcb/0x2b0 [ 2404.219276] ceph_fill_trace+0x143/0xf30 [ceph] [ 2404.223822] ? __mutex_lock+0x97/0xce0 [ 2404.227571] ? __mutex_lock+0xaec/0xce0 [ 2404.231414] ? mds_dispatch+0xb5a/0x1fa0 [ceph] [ 2404.235962] ? lock_downgrade+0xfd/0x250 [ 2404.239893] mds_dispatch+0xb9d/0x1fa0 [ceph] [ 2404.244272] ceph_con_process_message+0x74/0x150 [libceph] [ 2404.249768] ? calc_signature+0xdb/0x100 [libceph] [ 2404.254581] ceph_con_v1_try_read+0xcea/0x1b80 [libceph] [ 2404.259911] ? process_scheduled_works+0x22a/0x510 [ 2404.264702] ? process_scheduled_works+0x22a/0x510 [ 2404.269499] ceph_con_workfn+0x344/0x750 [libceph] [ 2404.274307] ? process_scheduled_works+0x22a/0x510 [ 2404.279096] process_scheduled_works+0x2a4/0x510 [ 2404.283721] worker_thread+0x188/0x340 [ 2404.287471] ? rescuer_thread+0x310/0x310 [ 2404.291482] kthread+0xf5/0x130 [ 2404.294629] ? kthread_complete_and_exit+0x20/0x20 [ 2404.299420] ret_from_fork+0x31/0x40 [ 2404.302998] ? kthread_complete_and_exit+0x20/0x20 [ 2404.307792] ret_from_fork_asm+0x11/0x20 [ 2404.311725] [ 2404.313919] irq event stamp: 10879755 [ 2404.317584] hardirqs last enabled at (10879765): [] console_unlock+0x106/0x120 [ 2404.326452] hardirqs last disabled at (10879774): [] console_unlock+0xeb/0x120 [ 2404.335231] softirqs last enabled at (10879702): [] __do_softirq+0x2be/0x371 [ 2404.343925] softirqs last disabled at (10879697): [] irq_exit_rcu+0xb0/0xd0 [ 2404.352443] ---[ end trace 0000000000000000 ]--- [ 2405.508152] watchdog: BUG: soft lockup - CPU#7 stuck for 123s! [kworker/7:4:78808] [ 2405.515718] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2405.571167] irq event stamp: 786130 [ 2405.574652] hardirqs last enabled at (786129): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2405.584645] hardirqs last disabled at (786130): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2405.594205] softirqs last enabled at (786080): [] __do_softirq+0x2be/0x371 [ 2405.602724] softirqs last disabled at (786075): [] irq_exit_rcu+0xb0/0xd0 [ 2405.611071] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2405.620447] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2405.627918] Workqueue: events delayed_work [ceph] [ 2405.632643] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2405.637002] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2405.655738] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 2405.660955] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 2405.668080] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 2405.675203] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 2405.682327] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 2405.689453] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2405.696576] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2405.704653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2405.710390] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2405.717515] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2405.724641] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2405.731763] Call Trace: [ 2405.734210] [ 2405.736221] ? watchdog_timer_fn+0x1b4/0x220 [ 2405.740492] ? __hrtimer_run_queues+0x1d3/0x360 [ 2405.745018] ? hrtimer_interrupt+0x104/0x240 [ 2405.749278] ? __local_bh_enable+0x50/0x60 [ 2405.753372] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2405.758596] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2405.763553] [ 2405.765652] [ 2405.767750] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2405.773057] ? lock_acquire+0xf0/0x2b0 [ 2405.776807] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2405.782213] ? lock_release+0xc5/0x260 [ 2405.785958] _raw_spin_lock+0x30/0x40 [ 2405.789616] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2405.795015] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2405.800243] delayed_work+0x1d8/0x280 [ceph] [ 2405.804526] ? process_scheduled_works+0x22a/0x510 [ 2405.809315] process_scheduled_works+0x2a4/0x510 [ 2405.813930] worker_thread+0x188/0x340 [ 2405.817680] ? rescuer_thread+0x310/0x310 [ 2405.821682] kthread+0xf5/0x130 [ 2405.824820] ? kthread_complete_and_exit+0x20/0x20 [ 2405.829604] ret_from_fork+0x31/0x40 [ 2405.833175] ? kthread_complete_and_exit+0x20/0x20 [ 2405.837959] ret_from_fork_asm+0x11/0x20 [ 2405.841882] [ 2433.508655] watchdog: BUG: soft lockup - CPU#7 stuck for 149s! [kworker/7:4:78808] [ 2433.516221] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2433.571671] irq event stamp: 847648 [ 2433.575154] hardirqs last enabled at (847647): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2433.585146] hardirqs last disabled at (847648): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2433.594697] softirqs last enabled at (847632): [] __do_softirq+0x2be/0x371 [ 2433.603216] softirqs last disabled at (847627): [] irq_exit_rcu+0xb0/0xd0 [ 2433.611556] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2433.620933] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2433.628402] Workqueue: events delayed_work [ceph] [ 2433.633125] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2433.637477] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2433.656215] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 2433.661431] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 2433.668555] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 2433.675680] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 2433.682805] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 2433.689927] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2433.697054] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2433.705129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2433.710868] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2433.717992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2433.725115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2433.732240] Call Trace: [ 2433.734685] [ 2433.736696] ? watchdog_timer_fn+0x1b4/0x220 [ 2433.740959] ? __hrtimer_run_queues+0x1d3/0x360 [ 2433.745496] ? hrtimer_interrupt+0x104/0x240 [ 2433.749765] ? __local_bh_enable+0x50/0x60 [ 2433.753858] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2433.759080] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2433.764038] [ 2433.766136] [ 2433.768233] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2433.773541] ? lock_acquire+0xf0/0x2b0 [ 2433.777292] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 2433.782006] ? lock_release+0xc5/0x260 [ 2433.785751] _raw_spin_lock+0x30/0x40 [ 2433.789407] ? ceph_check_caps+0x63/0xae0 [ceph] [ 2433.794025] ceph_check_caps+0x63/0xae0 [ceph] [ 2433.798478] ? __lock_acquire+0xbfa/0x1e20 [ 2433.802576] ? lock_acquire+0xcb/0x2b0 [ 2433.806323] ? igrab+0x1d/0x50 [ 2433.809376] ? find_held_lock+0x2d/0x90 [ 2433.813215] ? find_held_lock+0x2d/0x90 [ 2433.817046] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2433.822444] ? lock_release+0xc5/0x260 [ 2433.826190] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2433.831418] delayed_work+0x1d8/0x280 [ceph] [ 2433.835699] ? process_scheduled_works+0x22a/0x510 [ 2433.840488] process_scheduled_works+0x2a4/0x510 [ 2433.845103] worker_thread+0x188/0x340 [ 2433.848852] ? rescuer_thread+0x310/0x310 [ 2433.852857] kthread+0xf5/0x130 [ 2433.855994] ? kthread_complete_and_exit+0x20/0x20 [ 2433.860779] ret_from_fork+0x31/0x40 [ 2433.864358] ? kthread_complete_and_exit+0x20/0x20 [ 2433.869141] ret_from_fork_asm+0x11/0x20 [ 2433.873064] [ 2461.509143] watchdog: BUG: soft lockup - CPU#7 stuck for 175s! [kworker/7:4:78808] [ 2461.516706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2461.572156] irq event stamp: 908534 [ 2461.575638] hardirqs last enabled at (908533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2461.585633] hardirqs last disabled at (908534): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2461.595182] softirqs last enabled at (908454): [] __do_softirq+0x2be/0x371 [ 2461.603695] softirqs last disabled at (908449): [] irq_exit_rcu+0xb0/0xd0 [ 2461.612042] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2461.621419] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2461.628891] Workqueue: events delayed_work [ceph] [ 2461.633611] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2461.637965] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2461.656700] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 2461.661918] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 2461.669043] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 2461.676165] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 2461.683289] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 2461.690415] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2461.697539] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2461.705615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2461.711354] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2461.718478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2461.725600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2461.732727] Call Trace: [ 2461.735171] [ 2461.737182] ? watchdog_timer_fn+0x1b4/0x220 [ 2461.741454] ? __hrtimer_run_queues+0x1d3/0x360 [ 2461.745981] ? hrtimer_interrupt+0x104/0x240 [ 2461.750250] ? __local_bh_enable+0x50/0x60 [ 2461.754344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2461.759567] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2461.764523] [ 2461.766621] [ 2461.768719] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2461.774026] ? lock_acquire+0xf0/0x2b0 [ 2461.777769] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2461.783175] ? lock_release+0xc5/0x260 [ 2461.786920] _raw_spin_lock+0x30/0x40 [ 2461.790578] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2461.795976] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2461.801206] delayed_work+0x1d8/0x280 [ceph] [ 2461.805487] ? process_scheduled_works+0x22a/0x510 [ 2461.810277] process_scheduled_works+0x2a4/0x510 [ 2461.814892] worker_thread+0x188/0x340 [ 2461.818640] ? rescuer_thread+0x310/0x310 [ 2461.822643] kthread+0xf5/0x130 [ 2461.825781] ? kthread_complete_and_exit+0x20/0x20 [ 2461.830566] ret_from_fork+0x31/0x40 [ 2461.834137] ? kthread_complete_and_exit+0x20/0x20 [ 2461.838921] ret_from_fork_asm+0x11/0x20 [ 2461.842844] [ 2485.924986] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [ 2485.933889] libceph: mon0 (1)172.21.15.50:6789 session established [ 2489.509643] watchdog: BUG: soft lockup - CPU#7 stuck for 201s! [kworker/7:4:78808] [ 2489.517213] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2489.572660] irq event stamp: 970632 [ 2489.576143] hardirqs last enabled at (970631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2489.586136] hardirqs last disabled at (970632): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2489.595687] softirqs last enabled at (970616): [] __do_softirq+0x2be/0x371 [ 2489.604198] softirqs last disabled at (970611): [] irq_exit_rcu+0xb0/0xd0 [ 2489.612538] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2489.621914] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2489.629387] Workqueue: events delayed_work [ceph] [ 2489.634108] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2489.638460] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2489.657196] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 2489.662415] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 2489.669537] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 2489.676662] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 2489.683786] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 2489.690911] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2489.698035] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2489.706111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2489.711850] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2489.718972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2489.726099] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2489.733222] Call Trace: [ 2489.735667] [ 2489.737677] ? watchdog_timer_fn+0x1b4/0x220 [ 2489.741942] ? __hrtimer_run_queues+0x1d3/0x360 [ 2489.746468] ? hrtimer_interrupt+0x104/0x240 [ 2489.750738] ? __local_bh_enable+0x50/0x60 [ 2489.754832] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2489.760055] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2489.765013] [ 2489.767108] [ 2489.769206] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2489.774513] ? lock_acquire+0xf0/0x2b0 [ 2489.778256] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 2489.782969] ? lock_release+0xc5/0x260 [ 2489.786716] _raw_spin_lock+0x30/0x40 [ 2489.790382] ? ceph_check_caps+0x63/0xae0 [ceph] [ 2489.794999] ceph_check_caps+0x63/0xae0 [ceph] [ 2489.799451] ? __lock_acquire+0xbfa/0x1e20 [ 2489.803550] ? lock_acquire+0xcb/0x2b0 [ 2489.807299] ? igrab+0x1d/0x50 [ 2489.810348] ? find_held_lock+0x2d/0x90 [ 2489.814179] ? find_held_lock+0x2d/0x90 [ 2489.818011] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2489.823411] ? lock_release+0xc5/0x260 [ 2489.827155] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2489.832381] delayed_work+0x1d8/0x280 [ceph] [ 2489.836662] ? process_scheduled_works+0x22a/0x510 [ 2489.841452] process_scheduled_works+0x2a4/0x510 [ 2489.846067] worker_thread+0x188/0x340 [ 2489.849818] ? rescuer_thread+0x310/0x310 [ 2489.853822] kthread+0xf5/0x130 [ 2489.856957] ? kthread_complete_and_exit+0x20/0x20 [ 2489.861744] ret_from_fork+0x31/0x40 [ 2489.865323] ? kthread_complete_and_exit+0x20/0x20 [ 2489.870107] ret_from_fork_asm+0x11/0x20 [ 2489.874038] [ 2517.510141] watchdog: BUG: soft lockup - CPU#7 stuck for 227s! [kworker/7:4:78808] [ 2517.517709] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2517.573157] irq event stamp: 1032220 [ 2517.576727] hardirqs last enabled at (1032219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2517.586807] hardirqs last disabled at (1032220): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2517.596444] softirqs last enabled at (1032188): [] __do_softirq+0x2be/0x371 [ 2517.605042] softirqs last disabled at (1032183): [] irq_exit_rcu+0xb0/0xd0 [ 2517.613478] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2517.622863] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2517.630333] Workqueue: events delayed_work [ceph] [ 2517.635057] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2517.639417] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2517.658153] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 2517.663369] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 2517.670493] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 2517.677619] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 2517.684742] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 2517.691866] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2517.698990] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2517.707067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2517.712806] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2517.719931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2517.727052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2517.734179] Call Trace: [ 2517.736624] [ 2517.738633] ? watchdog_timer_fn+0x1b4/0x220 [ 2517.742898] ? __hrtimer_run_queues+0x1d3/0x360 [ 2517.747425] ? hrtimer_interrupt+0x104/0x240 [ 2517.751694] ? __local_bh_enable+0x50/0x60 [ 2517.755788] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2517.761010] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2517.765970] [ 2517.768064] [ 2517.770164] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2517.775471] ? lock_acquire+0xf0/0x2b0 [ 2517.779221] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 2517.783935] ? lock_release+0xc5/0x260 [ 2517.787680] _raw_spin_lock+0x30/0x40 [ 2517.791337] ? ceph_check_caps+0x63/0xae0 [ceph] [ 2517.795955] ceph_check_caps+0x63/0xae0 [ceph] [ 2517.800408] ? __lock_acquire+0xbfa/0x1e20 [ 2517.804505] ? lock_acquire+0xcb/0x2b0 [ 2517.808252] ? igrab+0x1d/0x50 [ 2517.811305] ? find_held_lock+0x2d/0x90 [ 2517.815144] ? find_held_lock+0x2d/0x90 [ 2517.818975] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2517.824374] ? lock_release+0xc5/0x260 [ 2517.828120] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2517.833348] delayed_work+0x1d8/0x280 [ceph] [ 2517.837629] ? process_scheduled_works+0x22a/0x510 [ 2517.842419] process_scheduled_works+0x2a4/0x510 [ 2517.847035] worker_thread+0x188/0x340 [ 2517.850784] ? rescuer_thread+0x310/0x310 [ 2517.854794] kthread+0xf5/0x130 [ 2517.857933] ? kthread_complete_and_exit+0x20/0x20 [ 2517.862717] ret_from_fork+0x31/0x40 [ 2517.866296] ? kthread_complete_and_exit+0x20/0x20 [ 2517.871078] ret_from_fork_asm+0x11/0x20 [ 2517.875001] [ 2522.981653] ceph: [cde022fc-3e6c-431e-bc4a-204d5c5d32a6 4445] check_session_state: mds0 hung [ 2545.510650] watchdog: BUG: soft lockup - CPU#7 stuck for 253s! [kworker/7:4:78808] [ 2545.518214] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2545.573663] irq event stamp: 1095398 [ 2545.577232] hardirqs last enabled at (1095397): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2545.587313] hardirqs last disabled at (1095398): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2545.596959] softirqs last enabled at (1095318): [] __do_softirq+0x2be/0x371 [ 2545.605556] softirqs last disabled at (1095313): [] irq_exit_rcu+0xb0/0xd0 [ 2545.613990] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2545.623366] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2545.630838] Workqueue: events delayed_work [ceph] [ 2545.635560] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2545.639912] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2545.658649] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 2545.663866] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 2545.670991] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 2545.678115] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 2545.685240] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 2545.692363] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2545.699486] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2545.707564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2545.713303] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2545.720426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2545.727549] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2545.734673] Call Trace: [ 2545.737121] [ 2545.739131] ? watchdog_timer_fn+0x1b4/0x220 [ 2545.743402] ? __hrtimer_run_queues+0x1d3/0x360 [ 2545.747930] ? hrtimer_interrupt+0x104/0x240 [ 2545.752200] ? __local_bh_enable+0x50/0x60 [ 2545.756291] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2545.761515] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2545.766472] [ 2545.768571] [ 2545.770669] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2545.775975] ? lock_acquire+0xf0/0x2b0 [ 2545.779727] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2545.785135] _raw_spin_lock+0x30/0x40 [ 2545.788791] ? igrab+0x1d/0x50 [ 2545.791842] igrab+0x1d/0x50 [ 2545.794719] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 2545.799862] delayed_work+0x1d8/0x280 [ceph] [ 2545.804142] ? process_scheduled_works+0x22a/0x510 [ 2545.808933] process_scheduled_works+0x2a4/0x510 [ 2545.813548] worker_thread+0x188/0x340 [ 2545.817297] ? rescuer_thread+0x310/0x310 [ 2545.821300] kthread+0xf5/0x130 [ 2545.824447] ? kthread_complete_and_exit+0x20/0x20 [ 2545.829230] ret_from_fork+0x31/0x40 [ 2545.832808] ? kthread_complete_and_exit+0x20/0x20 [ 2545.837592] ret_from_fork_asm+0x11/0x20 [ 2545.841516] [ 2573.511138] watchdog: BUG: soft lockup - CPU#7 stuck for 279s! [kworker/7:4:78808] [ 2573.518701] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2573.574150] irq event stamp: 1157226 [ 2573.577720] hardirqs last enabled at (1157225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2573.587799] hardirqs last disabled at (1157226): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2573.597437] softirqs last enabled at (1157204): [] __do_softirq+0x2be/0x371 [ 2573.606034] softirqs last disabled at (1157199): [] irq_exit_rcu+0xb0/0xd0 [ 2573.614460] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2573.623838] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2573.631309] Workqueue: events delayed_work [ceph] [ 2573.636032] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2573.640384] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2573.659119] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 2573.664337] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 2573.671462] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 2573.678586] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 2573.685710] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 2573.692835] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2573.699957] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2573.708035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2573.713772] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2573.720897] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2573.728020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2573.735144] Call Trace: [ 2573.737592] [ 2573.739602] ? watchdog_timer_fn+0x1b4/0x220 [ 2573.743874] ? __hrtimer_run_queues+0x1d3/0x360 [ 2573.748409] ? hrtimer_interrupt+0x104/0x240 [ 2573.752678] ? __local_bh_enable+0x50/0x60 [ 2573.756771] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2573.761995] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2573.766953] [ 2573.769051] [ 2573.771147] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2573.776455] ? lock_acquire+0xf0/0x2b0 [ 2573.780205] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 2573.784918] ? lock_release+0xc5/0x260 [ 2573.788663] _raw_spin_lock+0x30/0x40 [ 2573.792322] ? ceph_check_caps+0x63/0xae0 [ceph] [ 2573.796939] ceph_check_caps+0x63/0xae0 [ceph] [ 2573.801390] ? __lock_acquire+0xbfa/0x1e20 [ 2573.805488] ? lock_acquire+0xcb/0x2b0 [ 2573.809230] ? igrab+0x1d/0x50 [ 2573.812289] ? find_held_lock+0x2d/0x90 [ 2573.816128] ? find_held_lock+0x2d/0x90 [ 2573.819959] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2573.825359] ? lock_release+0xc5/0x260 [ 2573.829105] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2573.834340] delayed_work+0x1d8/0x280 [ceph] [ 2573.838621] ? process_scheduled_works+0x22a/0x510 [ 2573.843412] process_scheduled_works+0x2a4/0x510 [ 2573.848027] worker_thread+0x188/0x340 [ 2573.851775] ? rescuer_thread+0x310/0x310 [ 2573.855780] kthread+0xf5/0x130 [ 2573.858915] ? kthread_complete_and_exit+0x20/0x20 [ 2573.863701] ret_from_fork+0x31/0x40 [ 2573.867280] ? kthread_complete_and_exit+0x20/0x20 [ 2573.872063] ret_from_fork_asm+0x11/0x20 [ 2573.875988] [ 2601.511651] watchdog: BUG: soft lockup - CPU#7 stuck for 305s! [kworker/7:4:78808] [ 2601.519214] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2601.574664] irq event stamp: 1219216 [ 2601.578236] hardirqs last enabled at (1219215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2601.588314] hardirqs last disabled at (1219216): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2601.597953] softirqs last enabled at (1219136): [] __do_softirq+0x2be/0x371 [ 2601.606557] softirqs last disabled at (1219131): [] irq_exit_rcu+0xb0/0xd0 [ 2601.614984] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2601.624361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2601.631831] Workqueue: events delayed_work [ceph] [ 2601.636554] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2601.640907] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2601.659643] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 2601.664860] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 2601.671984] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 2601.679108] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 2601.686233] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 2601.693355] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2601.700480] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2601.708558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2601.714296] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2601.721420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2601.728543] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2601.735669] Call Trace: [ 2601.738114] [ 2601.740124] ? watchdog_timer_fn+0x1b4/0x220 [ 2601.744387] ? __hrtimer_run_queues+0x1d3/0x360 [ 2601.748913] ? hrtimer_interrupt+0x104/0x240 [ 2601.753175] ? __local_bh_enable+0x50/0x60 [ 2601.757269] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2601.762492] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2601.767450] [ 2601.769545] [ 2601.771643] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2601.776952] ? lock_acquire+0xf0/0x2b0 [ 2601.780703] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2601.786109] ? lock_release+0xc5/0x260 [ 2601.789855] _raw_spin_lock+0x30/0x40 [ 2601.793510] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2601.798911] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2601.804139] delayed_work+0x1d8/0x280 [ceph] [ 2601.808419] ? process_scheduled_works+0x22a/0x510 [ 2601.813209] process_scheduled_works+0x2a4/0x510 [ 2601.817826] worker_thread+0x188/0x340 [ 2601.821575] ? rescuer_thread+0x310/0x310 [ 2601.825578] kthread+0xf5/0x130 [ 2601.828715] ? kthread_complete_and_exit+0x20/0x20 [ 2601.833500] ret_from_fork+0x31/0x40 [ 2601.837079] ? kthread_complete_and_exit+0x20/0x20 [ 2601.841864] ret_from_fork_asm+0x11/0x20 [ 2601.845795] [ 2629.512130] watchdog: BUG: soft lockup - CPU#7 stuck for 331s! [kworker/7:4:78808] [ 2629.519694] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2629.575143] irq event stamp: 1280180 [ 2629.578715] hardirqs last enabled at (1280179): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2629.588793] hardirqs last disabled at (1280180): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2629.598431] softirqs last enabled at (1280164): [] __do_softirq+0x2be/0x371 [ 2629.607028] softirqs last disabled at (1280159): [] irq_exit_rcu+0xb0/0xd0 [ 2629.615456] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2629.624840] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2629.632311] Workqueue: events delayed_work [ceph] [ 2629.637033] RIP: 0010:__ceph_caps_issued+0x0/0x110 [ceph] [ 2629.642435] Code: 48 8d bf c0 04 00 00 be ff ff ff ff e8 09 67 06 e1 85 c0 0f 85 b8 fe ff ff 0f 0b e9 b1 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 <66> 0f 1f 00 0f 1f 44 00 00 41 57 41 56 41 55 49 89 fd 41 54 55 48 [ 2629.661171] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [ 2629.666388] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: 0000000000000000 [ 2629.673512] RDX: 0000000000000000 RSI: ffffc90002117c64 RDI: ffff8881a1d34478 [ 2629.680638] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 2629.687761] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000ea60 [ 2629.694886] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888100ee2000 [ 2629.702008] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2629.710087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2629.715824] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2629.722948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2629.730071] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2629.737197] Call Trace: [ 2629.739643] [ 2629.741652] ? watchdog_timer_fn+0x1b4/0x220 [ 2629.745925] ? __hrtimer_run_queues+0x1d3/0x360 [ 2629.750461] ? hrtimer_interrupt+0x104/0x240 [ 2629.754730] ? __local_bh_enable+0x50/0x60 [ 2629.758822] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2629.764045] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2629.769003] [ 2629.771100] [ 2629.773198] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2629.778506] ? change_auth_cap_ses+0x180/0x180 [ceph] [ 2629.783565] ceph_check_caps+0x154/0xae0 [ceph] [ 2629.788107] ? lock_acquire+0xcb/0x2b0 [ 2629.791850] ? igrab+0x1d/0x50 [ 2629.794900] ? find_held_lock+0x2d/0x90 [ 2629.798732] ? find_held_lock+0x2d/0x90 [ 2629.802563] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2629.807961] ? lock_release+0xc5/0x260 [ 2629.811706] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2629.816934] delayed_work+0x1d8/0x280 [ceph] [ 2629.821215] ? process_scheduled_works+0x22a/0x510 [ 2629.826004] process_scheduled_works+0x2a4/0x510 [ 2629.830622] worker_thread+0x188/0x340 [ 2629.834371] ? rescuer_thread+0x310/0x310 [ 2629.838374] kthread+0xf5/0x130 [ 2629.841519] ? kthread_complete_and_exit+0x20/0x20 [ 2629.846304] ret_from_fork+0x31/0x40 [ 2629.849883] ? kthread_complete_and_exit+0x20/0x20 [ 2629.854668] ret_from_fork_asm+0x11/0x20 [ 2629.858589] [ 2657.512635] watchdog: BUG: soft lockup - CPU#7 stuck for 358s! [kworker/7:4:78808] [ 2657.520200] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2657.575650] irq event stamp: 1341804 [ 2657.579220] hardirqs last enabled at (1341803): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2657.589299] hardirqs last disabled at (1341804): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2657.598936] softirqs last enabled at (1341688): [] __do_softirq+0x2be/0x371 [ 2657.607534] softirqs last disabled at (1341683): [] irq_exit_rcu+0xb0/0xd0 [ 2657.615960] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2657.625338] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2657.632809] Workqueue: events delayed_work [ceph] [ 2657.637532] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2657.641881] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2657.660620] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 2657.665836] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 2657.672961] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 2657.680084] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 2657.687208] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 2657.694333] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2657.701458] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2657.709534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2657.715272] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2657.722398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2657.729520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2657.736644] Call Trace: [ 2657.739091] [ 2657.741101] ? watchdog_timer_fn+0x1b4/0x220 [ 2657.745364] ? __hrtimer_run_queues+0x1d3/0x360 [ 2657.749890] ? hrtimer_interrupt+0x104/0x240 [ 2657.754152] ? __local_bh_enable+0x50/0x60 [ 2657.758244] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2657.763468] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2657.768425] [ 2657.770523] [ 2657.772620] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2657.777927] ? lock_acquire+0xf0/0x2b0 [ 2657.781670] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 2657.786384] ? lock_release+0xc5/0x260 [ 2657.790129] _raw_spin_lock+0x30/0x40 [ 2657.793786] ? ceph_check_caps+0x63/0xae0 [ceph] [ 2657.798405] ceph_check_caps+0x63/0xae0 [ceph] [ 2657.802857] ? __lock_acquire+0xbfa/0x1e20 [ 2657.806954] ? lock_acquire+0xcb/0x2b0 [ 2657.810703] ? igrab+0x1d/0x50 [ 2657.813753] ? find_held_lock+0x2d/0x90 [ 2657.817585] ? find_held_lock+0x2d/0x90 [ 2657.821416] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2657.826814] ? lock_release+0xc5/0x260 [ 2657.830560] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2657.835788] delayed_work+0x1d8/0x280 [ceph] [ 2657.840069] ? process_scheduled_works+0x22a/0x510 [ 2657.844859] process_scheduled_works+0x2a4/0x510 [ 2657.849474] worker_thread+0x188/0x340 [ 2657.853222] ? rescuer_thread+0x310/0x310 [ 2657.857227] kthread+0xf5/0x130 [ 2657.860365] ? kthread_complete_and_exit+0x20/0x20 [ 2657.865150] ret_from_fork+0x31/0x40 [ 2657.868728] ? kthread_complete_and_exit+0x20/0x20 [ 2657.873513] ret_from_fork_asm+0x11/0x20 [ 2657.877444] [ 2685.513116] watchdog: BUG: soft lockup - CPU#7 stuck for 384s! [kworker/7:4:78808] [ 2685.520681] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2685.576129] irq event stamp: 1403414 [ 2685.579701] hardirqs last enabled at (1403413): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2685.589780] hardirqs last disabled at (1403414): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2685.599425] softirqs last enabled at (1403334): [] __do_softirq+0x2be/0x371 [ 2685.608032] softirqs last disabled at (1403329): [] irq_exit_rcu+0xb0/0xd0 [ 2685.616459] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2685.625844] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2685.633314] Workqueue: events delayed_work [ceph] [ 2685.638036] RIP: 0010:iput+0x45/0x220 [ 2685.641694] Code: 00 00 00 40 0f 85 c8 01 00 00 48 89 fb 48 8d af 88 00 00 00 4c 8d a7 08 02 00 00 eb 0d be 01 00 00 00 48 89 df e8 6b c9 01 00 <48> 89 ee 4c 89 e7 e8 b0 13 7b 00 85 c0 0f 84 1c 01 00 00 48 8b 83 [ 2685.660432] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [ 2685.665649] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 2685.672772] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff8881a1d34478 [ 2685.679897] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [ 2685.687022] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [ 2685.694145] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2685.701269] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2685.709348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2685.715085] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2685.722210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2685.729333] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2685.736458] Call Trace: [ 2685.738902] [ 2685.740913] ? watchdog_timer_fn+0x1b4/0x220 [ 2685.745177] ? __hrtimer_run_queues+0x1d3/0x360 [ 2685.749710] ? hrtimer_interrupt+0x104/0x240 [ 2685.753972] ? __local_bh_enable+0x50/0x60 [ 2685.758066] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2685.763290] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2685.768247] [ 2685.770345] [ 2685.772442] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2685.777748] ? iput+0x45/0x220 [ 2685.780799] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 2685.786035] delayed_work+0x1d8/0x280 [ceph] [ 2685.790316] ? process_scheduled_works+0x22a/0x510 [ 2685.795105] process_scheduled_works+0x2a4/0x510 [ 2685.799721] worker_thread+0x188/0x340 [ 2685.803471] ? rescuer_thread+0x310/0x310 [ 2685.807473] kthread+0xf5/0x130 [ 2685.810612] ? kthread_complete_and_exit+0x20/0x20 [ 2685.815395] ret_from_fork+0x31/0x40 [ 2685.818967] ? kthread_complete_and_exit+0x20/0x20 [ 2685.823751] ret_from_fork_asm+0x11/0x20 [ 2685.827682] [ 2695.336671] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [ 2713.513609] watchdog: BUG: soft lockup - CPU#7 stuck for 410s! [kworker/7:4:78808] [ 2713.521177] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2713.576628] irq event stamp: 1466988 [ 2713.580197] hardirqs last enabled at (1466987): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2713.590278] hardirqs last disabled at (1466988): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2713.599924] softirqs last enabled at (1466934): [] __do_softirq+0x2be/0x371 [ 2713.608529] softirqs last disabled at (1466929): [] irq_exit_rcu+0xb0/0xd0 [ 2713.616963] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2713.626340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2713.633811] Workqueue: events delayed_work [ceph] [ 2713.638535] RIP: 0010:iput+0x25/0x220 [ 2713.642200] Code: 01 cc 7d 00 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 dd 01 00 00 41 55 41 54 55 53 f6 87 d8 00 00 00 40 0f 85 c8 01 00 00 <48> 89 fb 48 8d af 88 00 00 00 4c 8d a7 08 02 00 00 eb 0d be 01 00 [ 2713.660938] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [ 2713.666154] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 2713.673279] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff8881a1d34478 [ 2713.680403] RBP: ffff88817f867000 R08: 0000000000000000 R09: 0000000000000000 [ 2713.687527] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000ea60 [ 2713.694652] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2713.701775] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2713.709854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2713.715589] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2713.722715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2713.729840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2713.736964] Call Trace: [ 2713.739409] [ 2713.741420] ? watchdog_timer_fn+0x1b4/0x220 [ 2713.745690] ? __hrtimer_run_queues+0x1d3/0x360 [ 2713.750218] ? hrtimer_interrupt+0x104/0x240 [ 2713.754488] ? __local_bh_enable+0x50/0x60 [ 2713.758581] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2713.763805] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2713.768762] [ 2713.770860] [ 2713.772957] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2713.778265] ? iput+0x25/0x220 [ 2713.781321] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 2713.786556] delayed_work+0x1d8/0x280 [ceph] [ 2713.790838] ? process_scheduled_works+0x22a/0x510 [ 2713.795620] process_scheduled_works+0x2a4/0x510 [ 2713.800235] worker_thread+0x188/0x340 [ 2713.803986] ? rescuer_thread+0x310/0x310 [ 2713.807997] kthread+0xf5/0x130 [ 2713.811135] ? kthread_complete_and_exit+0x20/0x20 [ 2713.815918] ret_from_fork+0x31/0x40 [ 2713.819499] ? kthread_complete_and_exit+0x20/0x20 [ 2713.824283] ret_from_fork_asm+0x11/0x20 [ 2713.828214] [ 2741.514099] watchdog: BUG: soft lockup - CPU#7 stuck for 436s! [kworker/7:4:78808] [ 2741.521666] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2741.577115] irq event stamp: 1529436 [ 2741.580686] hardirqs last enabled at (1529435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2741.590766] hardirqs last disabled at (1529436): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2741.600410] softirqs last enabled at (1529382): [] __do_softirq+0x2be/0x371 [ 2741.609008] softirqs last disabled at (1529377): [] irq_exit_rcu+0xb0/0xd0 [ 2741.617434] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2741.626811] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2741.634284] Workqueue: events delayed_work [ceph] [ 2741.639005] RIP: 0010:ceph_put_mds_session+0x0/0x90 [ceph] [ 2741.644491] Code: 41 0f b1 08 74 de eb ec 31 c0 5b c3 31 f6 4c 89 c7 e8 64 d7 a8 e0 eb d7 31 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 <66> 0f 1f 00 0f 1f 44 00 00 48 85 ff 74 6e 48 81 ff 00 f0 ff ff 77 [ 2741.663230] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [ 2741.668447] RAX: ffffffffffffffff RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 2741.675570] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: 0000000000000000 [ 2741.682694] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 2741.689818] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 2741.696943] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 2741.704066] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2741.712143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2741.717882] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2741.725006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2741.732130] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2741.739253] Call Trace: [ 2741.741701] [ 2741.743712] ? watchdog_timer_fn+0x1b4/0x220 [ 2741.747982] ? __hrtimer_run_queues+0x1d3/0x360 [ 2741.752508] ? hrtimer_interrupt+0x104/0x240 [ 2741.756769] ? __local_bh_enable+0x50/0x60 [ 2741.760862] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2741.766087] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2741.771045] [ 2741.773143] [ 2741.775238] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2741.780548] ? ceph_get_mds_session+0x70/0x70 [ceph] [ 2741.785518] ceph_check_caps+0x275/0xae0 [ceph] [ 2741.790060] ? lock_acquire+0xcb/0x2b0 [ 2741.793802] ? igrab+0x1d/0x50 [ 2741.796854] ? find_held_lock+0x2d/0x90 [ 2741.800685] ? find_held_lock+0x2d/0x90 [ 2741.804516] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2741.809915] ? lock_release+0xc5/0x260 [ 2741.813661] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2741.818889] delayed_work+0x1d8/0x280 [ceph] [ 2741.823170] ? process_scheduled_works+0x22a/0x510 [ 2741.827959] process_scheduled_works+0x2a4/0x510 [ 2741.832575] worker_thread+0x188/0x340 [ 2741.836325] ? rescuer_thread+0x310/0x310 [ 2741.840336] kthread+0xf5/0x130 [ 2741.843474] ? kthread_complete_and_exit+0x20/0x20 [ 2741.848257] ret_from_fork+0x31/0x40 [ 2741.851829] ? kthread_complete_and_exit+0x20/0x20 [ 2741.856613] ret_from_fork_asm+0x11/0x20 [ 2741.860543] [ 2769.514587] watchdog: BUG: soft lockup - CPU#7 stuck for 462s! [kworker/7:4:78808] [ 2769.522156] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2769.577606] irq event stamp: 1590862 [ 2769.581174] hardirqs last enabled at (1590861): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2769.591253] hardirqs last disabled at (1590862): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2769.600892] softirqs last enabled at (1590746): [] __do_softirq+0x2be/0x371 [ 2769.609498] softirqs last disabled at (1590741): [] irq_exit_rcu+0xb0/0xd0 [ 2769.617932] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2769.627308] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2769.634781] Workqueue: events delayed_work [ceph] [ 2769.639502] RIP: 0010:igrab+0x2d/0x50 [ 2769.643159] Code: fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 <48> 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 [ 2769.661899] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [ 2769.667115] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 2769.674238] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 2769.681363] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 2769.688487] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2769.695612] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2769.702735] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2769.710814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2769.716551] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2769.723676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2769.730797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2769.737922] Call Trace: [ 2769.740369] [ 2769.742379] ? watchdog_timer_fn+0x1b4/0x220 [ 2769.746643] ? __hrtimer_run_queues+0x1d3/0x360 [ 2769.751178] ? hrtimer_interrupt+0x104/0x240 [ 2769.755447] ? __local_bh_enable+0x50/0x60 [ 2769.759541] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2769.764765] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2769.769722] [ 2769.771820] [ 2769.773917] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2769.779225] ? igrab+0x2d/0x50 [ 2769.782280] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 2769.787431] delayed_work+0x1d8/0x280 [ceph] [ 2769.791711] ? process_scheduled_works+0x22a/0x510 [ 2769.796502] process_scheduled_works+0x2a4/0x510 [ 2769.801118] worker_thread+0x188/0x340 [ 2769.804867] ? rescuer_thread+0x310/0x310 [ 2769.808870] kthread+0xf5/0x130 [ 2769.812009] ? kthread_complete_and_exit+0x20/0x20 [ 2769.816792] ret_from_fork+0x31/0x40 [ 2769.820371] ? kthread_complete_and_exit+0x20/0x20 [ 2769.825155] ret_from_fork_asm+0x11/0x20 [ 2769.829078] [ 2797.515074] watchdog: BUG: soft lockup - CPU#7 stuck for 488s! [kworker/7:4:78808] [ 2797.522661] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2797.578111] irq event stamp: 1651508 [ 2797.581680] hardirqs last enabled at (1651507): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2797.591760] hardirqs last disabled at (1651508): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2797.601396] softirqs last enabled at (1651454): [] __do_softirq+0x2be/0x371 [ 2797.609994] softirqs last disabled at (1651449): [] irq_exit_rcu+0xb0/0xd0 [ 2797.618421] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2797.627798] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2797.635267] Workqueue: events delayed_work [ceph] [ 2797.639991] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2797.644344] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2797.663080] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 2797.668298] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 2797.675422] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 2797.682545] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 2797.689670] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 2797.696794] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2797.703918] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2797.711994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2797.717732] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2797.724857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2797.731979] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2797.739104] Call Trace: [ 2797.741550] [ 2797.743561] ? watchdog_timer_fn+0x1b4/0x220 [ 2797.747825] ? __hrtimer_run_queues+0x1d3/0x360 [ 2797.752350] ? hrtimer_interrupt+0x104/0x240 [ 2797.756612] ? __local_bh_enable+0x50/0x60 [ 2797.760705] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2797.765930] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2797.770885] [ 2797.772983] [ 2797.775081] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2797.780389] ? lock_acquire+0xf0/0x2b0 [ 2797.784140] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2797.789546] _raw_spin_lock+0x30/0x40 [ 2797.793205] ? igrab+0x1d/0x50 [ 2797.796253] igrab+0x1d/0x50 [ 2797.799133] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 2797.804275] delayed_work+0x1d8/0x280 [ceph] [ 2797.808554] ? process_scheduled_works+0x22a/0x510 [ 2797.813336] process_scheduled_works+0x2a4/0x510 [ 2797.817952] worker_thread+0x188/0x340 [ 2797.821701] ? rescuer_thread+0x310/0x310 [ 2797.825713] kthread+0xf5/0x130 [ 2797.828852] ? kthread_complete_and_exit+0x20/0x20 [ 2797.833634] ret_from_fork+0x31/0x40 [ 2797.837215] ? kthread_complete_and_exit+0x20/0x20 [ 2797.841998] ret_from_fork_asm+0x11/0x20 [ 2797.845931] [ 2825.515562] watchdog: BUG: soft lockup - CPU#7 stuck for 514s! [kworker/7:4:78808] [ 2825.523133] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2825.578581] irq event stamp: 1713510 [ 2825.582151] hardirqs last enabled at (1713509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2825.592231] hardirqs last disabled at (1713510): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2825.601868] softirqs last enabled at (1713488): [] __do_softirq+0x2be/0x371 [ 2825.610474] softirqs last disabled at (1713483): [] irq_exit_rcu+0xb0/0xd0 [ 2825.618900] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2825.628276] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2825.635747] Workqueue: events delayed_work [ceph] [ 2825.640471] RIP: 0010:_raw_spin_unlock+0x0/0x40 [ 2825.644995] Code: 18 bb 01 00 00 00 45 31 c9 41 b8 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 [ 2825.663732] RSP: 0018:ffffc90002117db0 EFLAGS: 00000286 [ 2825.668950] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [ 2825.676074] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88817f867488 [ 2825.683199] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 2825.690321] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2825.697446] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2825.704571] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2825.712649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2825.718387] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2825.725509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2825.732634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2825.739759] Call Trace: [ 2825.742204] [ 2825.744215] ? watchdog_timer_fn+0x1b4/0x220 [ 2825.748487] ? __hrtimer_run_queues+0x1d3/0x360 [ 2825.753012] ? hrtimer_interrupt+0x104/0x240 [ 2825.757274] ? __local_bh_enable+0x50/0x60 [ 2825.761366] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2825.766590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2825.771548] [ 2825.773645] [ 2825.775743] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2825.781051] ? _raw_spin_trylock_bh+0x70/0x70 [ 2825.785406] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2825.790637] delayed_work+0x1d8/0x280 [ceph] [ 2825.794917] ? process_scheduled_works+0x22a/0x510 [ 2825.799707] process_scheduled_works+0x2a4/0x510 [ 2825.804322] worker_thread+0x188/0x340 [ 2825.808072] ? rescuer_thread+0x310/0x310 [ 2825.812075] kthread+0xf5/0x130 [ 2825.815213] ? kthread_complete_and_exit+0x20/0x20 [ 2825.819997] ret_from_fork+0x31/0x40 [ 2825.823576] ? kthread_complete_and_exit+0x20/0x20 [ 2825.828360] ret_from_fork_asm+0x11/0x20 [ 2825.832292] [ 2853.516050] watchdog: BUG: soft lockup - CPU#7 stuck for 540s! [kworker/7:4:78808] [ 2853.523613] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2853.579062] irq event stamp: 1776198 [ 2853.582632] hardirqs last enabled at (1776197): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2853.592713] hardirqs last disabled at (1776198): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2853.602358] softirqs last enabled at (1776148): [] __do_softirq+0x2be/0x371 [ 2853.610963] softirqs last disabled at (1776143): [] irq_exit_rcu+0xb0/0xd0 [ 2853.619398] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2853.628774] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2853.636247] Workqueue: events delayed_work [ceph] [ 2853.640969] RIP: 0010:igrab+0x2d/0x50 [ 2853.644635] Code: fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 <48> 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 [ 2853.663372] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [ 2853.668590] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 2853.675715] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 2853.682838] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 2853.689961] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2853.697086] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2853.704209] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2853.712288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2853.718025] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2853.725150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2853.732273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2853.739396] Call Trace: [ 2853.741843] [ 2853.743854] ? watchdog_timer_fn+0x1b4/0x220 [ 2853.748126] ? __hrtimer_run_queues+0x1d3/0x360 [ 2853.752660] ? hrtimer_interrupt+0x104/0x240 [ 2853.756921] ? __local_bh_enable+0x50/0x60 [ 2853.761014] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2853.766239] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2853.771197] [ 2853.773294] [ 2853.775392] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2853.780698] ? igrab+0x2d/0x50 [ 2853.783748] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 2853.788897] delayed_work+0x1d8/0x280 [ceph] [ 2853.793178] ? process_scheduled_works+0x22a/0x510 [ 2853.797968] process_scheduled_works+0x2a4/0x510 [ 2853.802583] worker_thread+0x188/0x340 [ 2853.806333] ? rescuer_thread+0x310/0x310 [ 2853.810336] kthread+0xf5/0x130 [ 2853.813475] ? kthread_complete_and_exit+0x20/0x20 [ 2853.818267] ret_from_fork+0x31/0x40 [ 2853.821847] ? kthread_complete_and_exit+0x20/0x20 [ 2853.826629] ret_from_fork_asm+0x11/0x20 [ 2853.830553] [ 2881.516539] watchdog: BUG: soft lockup - CPU#7 stuck for 566s! [kworker/7:4:78808] [ 2881.524102] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2881.579550] irq event stamp: 1836816 [ 2881.583119] hardirqs last enabled at (1836815): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2881.593200] hardirqs last disabled at (1836816): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2881.602836] softirqs last enabled at (1836762): [] __do_softirq+0x2be/0x371 [ 2881.611434] softirqs last disabled at (1836757): [] irq_exit_rcu+0xb0/0xd0 [ 2881.619869] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2881.629247] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2881.636717] Workqueue: events delayed_work [ceph] [ 2881.641440] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2881.645791] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2881.664529] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 2881.669746] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 2881.676870] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 2881.683995] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 2881.691116] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 2881.698241] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2881.705366] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2881.713444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2881.719182] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2881.726304] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2881.733429] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2881.740554] Call Trace: [ 2881.742999] [ 2881.745012] ? watchdog_timer_fn+0x1b4/0x220 [ 2881.749283] ? __hrtimer_run_queues+0x1d3/0x360 [ 2881.753818] ? hrtimer_interrupt+0x104/0x240 [ 2881.758087] ? __local_bh_enable+0x50/0x60 [ 2881.762179] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2881.767403] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2881.772360] [ 2881.774457] [ 2881.776557] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2881.781863] ? lock_acquire+0xf0/0x2b0 [ 2881.785613] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2881.791013] ? lock_release+0xc5/0x260 [ 2881.794758] _raw_spin_lock+0x30/0x40 [ 2881.798424] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2881.803823] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 2881.809052] delayed_work+0x1d8/0x280 [ceph] [ 2881.813332] ? process_scheduled_works+0x22a/0x510 [ 2881.818122] process_scheduled_works+0x2a4/0x510 [ 2881.822737] worker_thread+0x188/0x340 [ 2881.826487] ? rescuer_thread+0x310/0x310 [ 2881.830489] kthread+0xf5/0x130 [ 2881.833627] ? kthread_complete_and_exit+0x20/0x20 [ 2881.838410] ret_from_fork+0x31/0x40 [ 2881.841984] ? kthread_complete_and_exit+0x20/0x20 [ 2881.846775] ret_from_fork_asm+0x11/0x20 [ 2881.850706] [ 2909.517031] watchdog: BUG: soft lockup - CPU#7 stuck for 592s! [kworker/7:4:78808] [ 2909.524599] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2909.580047] irq event stamp: 1898370 [ 2909.583619] hardirqs last enabled at (1898369): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2909.593697] hardirqs last disabled at (1898370): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2909.603334] softirqs last enabled at (1898354): [] __do_softirq+0x2be/0x371 [ 2909.611940] softirqs last disabled at (1898349): [] irq_exit_rcu+0xb0/0xd0 [ 2909.620368] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2909.629752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2909.637222] Workqueue: events delayed_work [ceph] [ 2909.641946] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2909.646307] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2909.665042] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 2909.670259] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 2909.677384] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 2909.684509] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 2909.691633] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 2909.698756] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2909.705880] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2909.713958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2909.719695] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2909.726821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2909.733943] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2909.741069] Call Trace: [ 2909.743513] [ 2909.745526] ? watchdog_timer_fn+0x1b4/0x220 [ 2909.749796] ? __hrtimer_run_queues+0x1d3/0x360 [ 2909.754322] ? hrtimer_interrupt+0x104/0x240 [ 2909.758584] ? __local_bh_enable+0x50/0x60 [ 2909.762677] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2909.767902] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2909.772858] [ 2909.774955] [ 2909.777053] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2909.782362] ? lock_acquire+0xf0/0x2b0 [ 2909.786112] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2909.791521] _raw_spin_lock+0x30/0x40 [ 2909.795187] ? igrab+0x1d/0x50 [ 2909.798243] igrab+0x1d/0x50 [ 2909.801122] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 2909.806265] delayed_work+0x1d8/0x280 [ceph] [ 2909.810544] ? process_scheduled_works+0x22a/0x510 [ 2909.815333] process_scheduled_works+0x2a4/0x510 [ 2909.819951] worker_thread+0x188/0x340 [ 2909.823699] ? rescuer_thread+0x310/0x310 [ 2909.827703] kthread+0xf5/0x130 [ 2909.830848] ? kthread_complete_and_exit+0x20/0x20 [ 2909.835634] ret_from_fork+0x31/0x40 [ 2909.839213] ? kthread_complete_and_exit+0x20/0x20 [ 2909.843997] ret_from_fork_asm+0x11/0x20 [ 2909.847928] [ 2937.517518] watchdog: BUG: soft lockup - CPU#7 stuck for 618s! [kworker/7:4:78808] [ 2937.525087] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2937.580536] irq event stamp: 1959904 [ 2937.584106] hardirqs last enabled at (1959903): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2937.594186] hardirqs last disabled at (1959904): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2937.603824] softirqs last enabled at (1959882): [] __do_softirq+0x2be/0x371 [ 2937.612430] softirqs last disabled at (1959877): [] irq_exit_rcu+0xb0/0xd0 [ 2937.620857] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2937.630239] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2937.637713] Workqueue: events delayed_work [ceph] [ 2937.642435] RIP: 0010:ceph_check_caps+0x2a8/0xae0 [ceph] [ 2937.647757] Code: 85 24 06 00 00 80 bc 24 8e 00 00 00 00 0f 85 04 06 00 00 48 8b 84 24 a8 01 00 00 65 48 33 04 25 28 00 00 00 0f 85 2d 08 00 00 <48> 81 c4 b0 01 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 c7 84 24 88 [ 2937.666493] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [ 2937.671712] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 2937.678836] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: 0000000000000000 [ 2937.685959] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 2937.693084] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 2937.700207] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 2937.707331] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2937.715410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2937.721146] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2937.728271] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2937.735394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2937.742518] Call Trace: [ 2937.744964] [ 2937.746977] ? watchdog_timer_fn+0x1b4/0x220 [ 2937.751248] ? __hrtimer_run_queues+0x1d3/0x360 [ 2937.755781] ? hrtimer_interrupt+0x104/0x240 [ 2937.760042] ? __local_bh_enable+0x50/0x60 [ 2937.764136] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2937.769361] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2937.774319] [ 2937.776416] [ 2937.778512] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2937.783820] ? ceph_check_caps+0x2a8/0xae0 [ceph] [ 2937.788539] ? lock_acquire+0xcb/0x2b0 [ 2937.792285] ? igrab+0x1d/0x50 [ 2937.795336] ? find_held_lock+0x2d/0x90 [ 2937.799174] ? find_held_lock+0x2d/0x90 [ 2937.803004] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2937.808404] ? lock_release+0xc5/0x260 [ 2937.812150] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2937.817387] delayed_work+0x1d8/0x280 [ceph] [ 2937.821667] ? process_scheduled_works+0x22a/0x510 [ 2937.826457] process_scheduled_works+0x2a4/0x510 [ 2937.831073] worker_thread+0x188/0x340 [ 2937.834821] ? rescuer_thread+0x310/0x310 [ 2937.838825] kthread+0xf5/0x130 [ 2937.841963] ? kthread_complete_and_exit+0x20/0x20 [ 2937.846747] ret_from_fork+0x31/0x40 [ 2937.850318] ? kthread_complete_and_exit+0x20/0x20 [ 2937.855101] ret_from_fork_asm+0x11/0x20 [ 2937.859024] [ 2965.518007] watchdog: BUG: soft lockup - CPU#7 stuck for 644s! [kworker/7:4:78808] [ 2965.525575] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2965.581026] irq event stamp: 2021620 [ 2965.584594] hardirqs last enabled at (2021619): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2965.594674] hardirqs last disabled at (2021620): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2965.604312] softirqs last enabled at (2021494): [] __do_softirq+0x2be/0x371 [ 2965.612910] softirqs last disabled at (2021489): [] irq_exit_rcu+0xb0/0xd0 [ 2965.621336] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2965.630713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2965.638183] Workqueue: events delayed_work [ceph] [ 2965.642906] RIP: 0010:igrab+0x39/0x50 [ 2965.646563] Code: 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 48 89 ef e8 8b 60 7f 00 48 89 d8 5b <5d> c3 48 89 ef e8 7d 60 7f 00 31 c0 5b 5d c3 0f 1f 84 00 00 00 00 [ 2965.665299] RSP: 0018:ffffc90002117da8 EFLAGS: 00000202 [ 2965.670517] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [ 2965.677642] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [ 2965.684766] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 2965.691890] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 2965.699015] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 2965.706140] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2965.714217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2965.719955] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2965.727077] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2965.734203] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2965.741327] Call Trace: [ 2965.743772] [ 2965.745785] ? watchdog_timer_fn+0x1b4/0x220 [ 2965.750055] ? __hrtimer_run_queues+0x1d3/0x360 [ 2965.754580] ? hrtimer_interrupt+0x104/0x240 [ 2965.758841] ? __local_bh_enable+0x50/0x60 [ 2965.762935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2965.768159] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2965.773116] [ 2965.775214] [ 2965.777312] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2965.782618] ? igrab+0x39/0x50 [ 2965.785667] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 2965.790819] delayed_work+0x1d8/0x280 [ceph] [ 2965.795099] ? process_scheduled_works+0x22a/0x510 [ 2965.799887] process_scheduled_works+0x2a4/0x510 [ 2965.804505] worker_thread+0x188/0x340 [ 2965.808254] ? rescuer_thread+0x310/0x310 [ 2965.812264] kthread+0xf5/0x130 [ 2965.815402] ? kthread_complete_and_exit+0x20/0x20 [ 2965.820187] ret_from_fork+0x31/0x40 [ 2965.823766] ? kthread_complete_and_exit+0x20/0x20 [ 2965.828550] ret_from_fork_asm+0x11/0x20 [ 2965.832472] [ 2993.518494] watchdog: BUG: soft lockup - CPU#7 stuck for 670s! [kworker/7:4:78808] [ 2993.526057] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 2993.581507] irq event stamp: 2082822 [ 2993.585074] hardirqs last enabled at (2082821): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2993.595154] hardirqs last disabled at (2082822): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 2993.604793] softirqs last enabled at (2082820): [] __do_softirq+0x2be/0x371 [ 2993.613399] softirqs last disabled at (2082811): [] irq_exit_rcu+0xb0/0xd0 [ 2993.621833] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 2993.631209] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 2993.638680] Workqueue: events delayed_work [ceph] [ 2993.643405] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 2993.647763] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 2993.666502] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 2993.671719] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 2993.678842] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 2993.685966] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 2993.693091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 2993.700214] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 2993.707337] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 2993.715415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2993.721153] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 2993.728276] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2993.735402] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2993.742526] Call Trace: [ 2993.744971] [ 2993.746982] ? watchdog_timer_fn+0x1b4/0x220 [ 2993.751245] ? __hrtimer_run_queues+0x1d3/0x360 [ 2993.755773] ? hrtimer_interrupt+0x104/0x240 [ 2993.760042] ? __local_bh_enable+0x50/0x60 [ 2993.764135] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 2993.769358] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 2993.774315] [ 2993.776413] [ 2993.778511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 2993.783820] ? lock_acquire+0xf0/0x2b0 [ 2993.787569] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 2993.792283] ? lock_release+0xc5/0x260 [ 2993.796029] _raw_spin_lock+0x30/0x40 [ 2993.799693] ? ceph_check_caps+0x63/0xae0 [ceph] [ 2993.804313] ceph_check_caps+0x63/0xae0 [ceph] [ 2993.808763] ? __lock_acquire+0xbfa/0x1e20 [ 2993.812862] ? lock_acquire+0xcb/0x2b0 [ 2993.816611] ? igrab+0x1d/0x50 [ 2993.819662] ? find_held_lock+0x2d/0x90 [ 2993.823501] ? find_held_lock+0x2d/0x90 [ 2993.827331] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 2993.832731] ? lock_release+0xc5/0x260 [ 2993.836477] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 2993.841703] delayed_work+0x1d8/0x280 [ceph] [ 2993.845985] ? process_scheduled_works+0x22a/0x510 [ 2993.850774] process_scheduled_works+0x2a4/0x510 [ 2993.855390] worker_thread+0x188/0x340 [ 2993.859138] ? rescuer_thread+0x310/0x310 [ 2993.863143] kthread+0xf5/0x130 [ 2993.866280] ? kthread_complete_and_exit+0x20/0x20 [ 2993.871065] ret_from_fork+0x31/0x40 [ 2993.874645] ? kthread_complete_and_exit+0x20/0x20 [ 2993.879427] ret_from_fork_asm+0x11/0x20 [ 2993.883360] [ 3021.518979] watchdog: BUG: soft lockup - CPU#7 stuck for 697s! [kworker/7:4:78808] [ 3021.526545] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3021.581994] irq event stamp: 2143610 [ 3021.585563] hardirqs last enabled at (2143609): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3021.595644] hardirqs last disabled at (2143610): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3021.605290] softirqs last enabled at (2143530): [] __do_softirq+0x2be/0x371 [ 3021.613889] softirqs last disabled at (2143525): [] irq_exit_rcu+0xb0/0xd0 [ 3021.622323] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3021.631698] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3021.639170] Workqueue: events delayed_work [ceph] [ 3021.643892] RIP: 0010:rcu_is_watching+0x1c/0x40 [ 3021.648416] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 [ 3021.667153] RSP: 0018:ffffc90002117d10 EFLAGS: 00000286 [ 3021.672372] RAX: 0000000001f06464 RBX: ffff8881a1d34500 RCX: 0000000000000000 [ 3021.679497] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [ 3021.686619] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 3021.693744] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3021.700868] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3021.707992] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3021.716069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3021.721807] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3021.728930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3021.736056] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3021.743180] Call Trace: [ 3021.745626] [ 3021.747635] ? watchdog_timer_fn+0x1b4/0x220 [ 3021.751899] ? __hrtimer_run_queues+0x1d3/0x360 [ 3021.756425] ? hrtimer_interrupt+0x104/0x240 [ 3021.760687] ? __local_bh_enable+0x50/0x60 [ 3021.764780] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3021.770005] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3021.774960] [ 3021.777058] [ 3021.779155] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3021.784464] ? rcu_is_watching+0x1c/0x40 [ 3021.788388] lock_acquire+0x184/0x2b0 [ 3021.792054] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3021.797461] _raw_spin_lock+0x30/0x40 [ 3021.801119] ? igrab+0x1d/0x50 [ 3021.804169] igrab+0x1d/0x50 [ 3021.807047] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3021.812188] delayed_work+0x1d8/0x280 [ceph] [ 3021.816468] ? process_scheduled_works+0x22a/0x510 [ 3021.821259] process_scheduled_works+0x2a4/0x510 [ 3021.825874] worker_thread+0x188/0x340 [ 3021.829625] ? rescuer_thread+0x310/0x310 [ 3021.833626] kthread+0xf5/0x130 [ 3021.836766] ? kthread_complete_and_exit+0x20/0x20 [ 3021.841549] ret_from_fork+0x31/0x40 [ 3021.845128] ? kthread_complete_and_exit+0x20/0x20 [ 3021.849913] ret_from_fork_asm+0x11/0x20 [ 3021.853844] [ 3049.519464] watchdog: BUG: soft lockup - CPU#7 stuck for 723s! [kworker/7:4:78808] [ 3049.527033] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3049.582483] irq event stamp: 2205996 [ 3049.586052] hardirqs last enabled at (2205995): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3049.596132] hardirqs last disabled at (2205996): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3049.605770] softirqs last enabled at (2205982): [] __do_softirq+0x2be/0x371 [ 3049.614376] softirqs last disabled at (2205977): [] irq_exit_rcu+0xb0/0xd0 [ 3049.622812] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3049.632195] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3049.639666] Workqueue: events delayed_work [ceph] [ 3049.644390] RIP: 0010:lock_release+0x14e/0x260 [ 3049.648829] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 3049.667566] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [ 3049.672782] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [ 3049.679906] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [ 3049.687031] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [ 3049.694156] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9425 [ 3049.701279] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [ 3049.708403] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3049.716480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3049.722218] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3049.729342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3049.736467] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3049.743589] Call Trace: [ 3049.746036] [ 3049.748047] ? watchdog_timer_fn+0x1b4/0x220 [ 3049.752310] ? __hrtimer_run_queues+0x1d3/0x360 [ 3049.756836] ? hrtimer_interrupt+0x104/0x240 [ 3049.761098] ? __local_bh_enable+0x50/0x60 [ 3049.765190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3049.770414] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3049.775371] [ 3049.777469] [ 3049.779567] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3049.784870] ? igrab+0x35/0x50 [ 3049.787923] ? lock_release+0x14e/0x260 [ 3049.791752] ? lock_release+0xc5/0x260 [ 3049.795498] _raw_spin_unlock+0x1b/0x40 [ 3049.799327] igrab+0x35/0x50 [ 3049.802205] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3049.807356] delayed_work+0x1d8/0x280 [ceph] [ 3049.811637] ? process_scheduled_works+0x22a/0x510 [ 3049.816427] process_scheduled_works+0x2a4/0x510 [ 3049.821042] worker_thread+0x188/0x340 [ 3049.824790] ? rescuer_thread+0x310/0x310 [ 3049.828795] kthread+0xf5/0x130 [ 3049.831933] ? kthread_complete_and_exit+0x20/0x20 [ 3049.836716] ret_from_fork+0x31/0x40 [ 3049.840286] ? kthread_complete_and_exit+0x20/0x20 [ 3049.845070] ret_from_fork_asm+0x11/0x20 [ 3049.848993] [ 3077.519949] watchdog: BUG: soft lockup - CPU#7 stuck for 749s! [kworker/7:4:78808] [ 3077.527515] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3077.582963] irq event stamp: 2267040 [ 3077.586534] hardirqs last enabled at (2267039): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3077.596612] hardirqs last disabled at (2267040): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3077.606250] softirqs last enabled at (2267016): [] __do_softirq+0x2be/0x371 [ 3077.614848] softirqs last disabled at (2267011): [] irq_exit_rcu+0xb0/0xd0 [ 3077.623284] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3077.632667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3077.640137] Workqueue: events delayed_work [ceph] [ 3077.644861] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3077.649212] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3077.667951] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3077.673167] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3077.680292] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3077.687415] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3077.694541] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3077.701663] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3077.708788] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3077.716865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3077.722602] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3077.729728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3077.736850] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3077.743975] Call Trace: [ 3077.746423] [ 3077.748431] ? watchdog_timer_fn+0x1b4/0x220 [ 3077.752695] ? __hrtimer_run_queues+0x1d3/0x360 [ 3077.757221] ? hrtimer_interrupt+0x104/0x240 [ 3077.761482] ? __local_bh_enable+0x50/0x60 [ 3077.765577] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3077.770800] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3077.775758] [ 3077.777853] [ 3077.779951] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3077.785260] ? lock_acquire+0xf0/0x2b0 [ 3077.789008] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3077.794417] ? lock_release+0xc5/0x260 [ 3077.798161] _raw_spin_lock+0x30/0x40 [ 3077.801818] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3077.807218] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3077.812446] delayed_work+0x1d8/0x280 [ceph] [ 3077.816727] ? process_scheduled_works+0x22a/0x510 [ 3077.821517] process_scheduled_works+0x2a4/0x510 [ 3077.826133] worker_thread+0x188/0x340 [ 3077.829881] ? rescuer_thread+0x310/0x310 [ 3077.833884] kthread+0xf5/0x130 [ 3077.837024] ? kthread_complete_and_exit+0x20/0x20 [ 3077.841808] ret_from_fork+0x31/0x40 [ 3077.845386] ? kthread_complete_and_exit+0x20/0x20 [ 3077.850169] ret_from_fork_asm+0x11/0x20 [ 3077.854093] [ 3105.520433] watchdog: BUG: soft lockup - CPU#7 stuck for 775s! [kworker/7:4:78808] [ 3105.528004] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3105.583454] irq event stamp: 2328150 [ 3105.587023] hardirqs last enabled at (2328149): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3105.597102] hardirqs last disabled at (2328150): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3105.606741] softirqs last enabled at (2328134): [] __do_softirq+0x2be/0x371 [ 3105.615346] softirqs last disabled at (2328129): [] irq_exit_rcu+0xb0/0xd0 [ 3105.623780] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3105.633157] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3105.640629] Workqueue: events delayed_work [ceph] [ 3105.645350] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3105.649702] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3105.668438] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 3105.673656] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 3105.680781] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 3105.687905] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 3105.695027] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3105.702153] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3105.709277] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3105.717353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3105.723092] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3105.730215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3105.737341] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3105.744465] Call Trace: [ 3105.746910] [ 3105.748921] ? watchdog_timer_fn+0x1b4/0x220 [ 3105.753193] ? __hrtimer_run_queues+0x1d3/0x360 [ 3105.757718] ? hrtimer_interrupt+0x104/0x240 [ 3105.761980] ? __local_bh_enable+0x50/0x60 [ 3105.766074] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3105.771298] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3105.776255] [ 3105.778350] [ 3105.780448] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3105.785756] ? lock_acquire+0xf0/0x2b0 [ 3105.789508] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3105.794915] _raw_spin_lock+0x30/0x40 [ 3105.798572] ? igrab+0x1d/0x50 [ 3105.801621] igrab+0x1d/0x50 [ 3105.804502] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3105.809652] delayed_work+0x1d8/0x280 [ceph] [ 3105.813933] ? process_scheduled_works+0x22a/0x510 [ 3105.818722] process_scheduled_works+0x2a4/0x510 [ 3105.823337] worker_thread+0x188/0x340 [ 3105.827086] ? rescuer_thread+0x310/0x310 [ 3105.831089] kthread+0xf5/0x130 [ 3105.834227] ? kthread_complete_and_exit+0x20/0x20 [ 3105.839012] ret_from_fork+0x31/0x40 [ 3105.842592] ? kthread_complete_and_exit+0x20/0x20 [ 3105.847374] ret_from_fork_asm+0x11/0x20 [ 3105.851298] [ 3133.520918] watchdog: BUG: soft lockup - CPU#7 stuck for 801s! [kworker/7:4:78808] [ 3133.528500] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3133.583950] irq event stamp: 2390450 [ 3133.587519] hardirqs last enabled at (2390449): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3133.597601] hardirqs last disabled at (2390450): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3133.607245] softirqs last enabled at (2390432): [] __do_softirq+0x2be/0x371 [ 3133.615843] softirqs last disabled at (2390415): [] irq_exit_rcu+0xb0/0xd0 [ 3133.624278] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3133.633664] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3133.641134] Workqueue: events delayed_work [ceph] [ 3133.645857] RIP: 0010:__ceph_caps_issued+0x70/0x110 [ceph] [ 3133.651344] Code: 49 89 c7 48 85 c0 74 32 49 8d 5f f8 48 89 df e8 c6 dc ff ff 85 c0 74 12 66 90 45 0b 67 38 48 85 ed 74 07 41 8b 47 3c 09 45 00 <4c> 89 ff e8 98 fc 04 e1 49 89 c7 48 85 c0 75 ce 49 8b 95 b0 06 00 [ 3133.670080] RSP: 0018:ffffc90002117b98 EFLAGS: 00000246 [ 3133.675298] RAX: 0000000000000000 RBX: ffff88810da9e000 RCX: 00000001002b37a0 [ 3133.682421] RDX: ffff888157b28000 RSI: ffffc90002117c64 RDI: ffff88810da9e000 [ 3133.689546] RBP: ffffc90002117c64 R08: 0000000000000000 R09: 0000000000000000 [ 3133.696670] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 3133.703794] R13: ffff8881a1d34478 R14: ffff888100ee2000 R15: ffff88810da9e008 [ 3133.710918] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3133.718996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3133.724732] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3133.731857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3133.738982] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3133.746107] Call Trace: [ 3133.748553] [ 3133.750563] ? watchdog_timer_fn+0x1b4/0x220 [ 3133.754833] ? __hrtimer_run_queues+0x1d3/0x360 [ 3133.759359] ? hrtimer_interrupt+0x104/0x240 [ 3133.763620] ? __local_bh_enable+0x50/0x60 [ 3133.767714] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3133.772939] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3133.777895] [ 3133.779993] [ 3133.782091] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3133.787399] ? __ceph_caps_issued+0x70/0x110 [ceph] [ 3133.792283] ? __ceph_caps_issued+0x5a/0x110 [ceph] [ 3133.797182] ceph_check_caps+0x154/0xae0 [ceph] [ 3133.801730] ? lock_acquire+0xcb/0x2b0 [ 3133.805474] ? igrab+0x1d/0x50 [ 3133.808524] ? find_held_lock+0x2d/0x90 [ 3133.812356] ? find_held_lock+0x2d/0x90 [ 3133.816185] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3133.821585] ? lock_release+0xc5/0x260 [ 3133.825332] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3133.830558] delayed_work+0x1d8/0x280 [ceph] [ 3133.834839] ? process_scheduled_works+0x22a/0x510 [ 3133.839628] process_scheduled_works+0x2a4/0x510 [ 3133.844246] worker_thread+0x188/0x340 [ 3133.847995] ? rescuer_thread+0x310/0x310 [ 3133.851996] kthread+0xf5/0x130 [ 3133.855135] ? kthread_complete_and_exit+0x20/0x20 [ 3133.859919] ret_from_fork+0x31/0x40 [ 3133.863498] ? kthread_complete_and_exit+0x20/0x20 [ 3133.868282] ret_from_fork_asm+0x11/0x20 [ 3133.872207] [ 3161.521403] watchdog: BUG: soft lockup - CPU#7 stuck for 827s! [kworker/7:4:78808] [ 3161.528972] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3161.584421] irq event stamp: 2452714 [ 3161.587994] hardirqs last enabled at (2452713): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3161.598070] hardirqs last disabled at (2452714): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3161.607708] softirqs last enabled at (2452608): [] __do_softirq+0x2be/0x371 [ 3161.616305] softirqs last disabled at (2452603): [] irq_exit_rcu+0xb0/0xd0 [ 3161.624731] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3161.634110] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3161.641580] Workqueue: events delayed_work [ceph] [ 3161.646303] RIP: 0010:lock_release+0x1d2/0x260 [ 3161.650741] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [ 3161.669476] RSP: 0018:ffffc90002117b78 EFLAGS: 00000247 [ 3161.674693] RAX: 0000000000000007 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 3161.681818] RDX: ffff8881a1d34bc8 RSI: ffffffffa0b3407b RDI: ffff8881a1d34938 [ 3161.688943] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 3161.696066] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 3161.703191] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 3161.710314] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3161.718392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3161.724131] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3161.731255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3161.738378] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3161.745503] Call Trace: [ 3161.747948] [ 3161.749959] ? watchdog_timer_fn+0x1b4/0x220 [ 3161.754221] ? __hrtimer_run_queues+0x1d3/0x360 [ 3161.758747] ? hrtimer_interrupt+0x104/0x240 [ 3161.763010] ? __local_bh_enable+0x50/0x60 [ 3161.767103] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3161.772327] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3161.777283] [ 3161.779380] [ 3161.781478] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3161.786783] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3161.791499] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3161.796205] ? lock_release+0x1d2/0x260 [ 3161.800038] _raw_spin_unlock+0x1b/0x40 [ 3161.803875] ceph_check_caps+0x26b/0xae0 [ceph] [ 3161.808418] ? lock_acquire+0xcb/0x2b0 [ 3161.812160] ? igrab+0x1d/0x50 [ 3161.815220] ? find_held_lock+0x2d/0x90 [ 3161.819058] ? find_held_lock+0x2d/0x90 [ 3161.822891] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3161.828297] ? lock_release+0xc5/0x260 [ 3161.832043] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3161.837270] delayed_work+0x1d8/0x280 [ceph] [ 3161.841552] ? process_scheduled_works+0x22a/0x510 [ 3161.846341] process_scheduled_works+0x2a4/0x510 [ 3161.850957] worker_thread+0x188/0x340 [ 3161.854705] ? rescuer_thread+0x310/0x310 [ 3161.858709] kthread+0xf5/0x130 [ 3161.861848] ? kthread_complete_and_exit+0x20/0x20 [ 3161.866631] ret_from_fork+0x31/0x40 [ 3161.870211] ? kthread_complete_and_exit+0x20/0x20 [ 3161.874993] ret_from_fork_asm+0x11/0x20 [ 3161.878917] [ 3189.521888] watchdog: BUG: soft lockup - CPU#7 stuck for 853s! [kworker/7:4:78808] [ 3189.529450] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3189.584893] irq event stamp: 2515404 [ 3189.588462] hardirqs last enabled at (2515403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3189.598534] hardirqs last disabled at (2515404): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3189.608172] softirqs last enabled at (2515310): [] __do_softirq+0x2be/0x371 [ 3189.616768] softirqs last disabled at (2515305): [] irq_exit_rcu+0xb0/0xd0 [ 3189.625194] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3189.634572] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3189.642040] Workqueue: events delayed_work [ceph] [ 3189.646757] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3189.651106] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3189.669844] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 3189.675063] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 3189.682187] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 3189.689311] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 3189.696433] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3189.703558] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3189.710683] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3189.718759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3189.724499] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3189.731621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3189.738746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3189.745870] Call Trace: [ 3189.748315] [ 3189.750324] ? watchdog_timer_fn+0x1b4/0x220 [ 3189.754590] ? __hrtimer_run_queues+0x1d3/0x360 [ 3189.759116] ? hrtimer_interrupt+0x104/0x240 [ 3189.763377] ? __local_bh_enable+0x50/0x60 [ 3189.767469] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3189.772686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3189.777644] [ 3189.779739] [ 3189.781836] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3189.787146] ? lock_acquire+0xf0/0x2b0 [ 3189.790897] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3189.796304] _raw_spin_lock+0x30/0x40 [ 3189.799961] ? igrab+0x1d/0x50 [ 3189.803011] igrab+0x1d/0x50 [ 3189.805890] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3189.811031] delayed_work+0x1d8/0x280 [ceph] [ 3189.815312] ? process_scheduled_works+0x22a/0x510 [ 3189.820102] process_scheduled_works+0x2a4/0x510 [ 3189.824716] worker_thread+0x188/0x340 [ 3189.828468] ? rescuer_thread+0x310/0x310 [ 3189.832479] kthread+0xf5/0x130 [ 3189.835615] ? kthread_complete_and_exit+0x20/0x20 [ 3189.840401] ret_from_fork+0x31/0x40 [ 3189.843970] ? kthread_complete_and_exit+0x20/0x20 [ 3189.848754] ret_from_fork_asm+0x11/0x20 [ 3189.852676] [ 3217.522375] watchdog: BUG: soft lockup - CPU#7 stuck for 879s! [kworker/7:4:78808] [ 3217.529939] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3217.585390] irq event stamp: 2579364 [ 3217.588962] hardirqs last enabled at (2579363): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3217.599040] hardirqs last disabled at (2579364): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3217.608686] softirqs last enabled at (2579342): [] __do_softirq+0x2be/0x371 [ 3217.617293] softirqs last disabled at (2579337): [] irq_exit_rcu+0xb0/0xd0 [ 3217.625727] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3217.635103] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3217.642574] Workqueue: events delayed_work [ceph] [ 3217.647297] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3217.651649] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3217.670385] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 3217.675604] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 3217.682726] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 3217.689850] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 3217.696975] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 3217.704100] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3217.711224] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3217.719300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3217.725038] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3217.732162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3217.739285] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3217.746411] Call Trace: [ 3217.748856] [ 3217.750867] ? watchdog_timer_fn+0x1b4/0x220 [ 3217.755130] ? __hrtimer_run_queues+0x1d3/0x360 [ 3217.759657] ? hrtimer_interrupt+0x104/0x240 [ 3217.763917] ? __local_bh_enable+0x50/0x60 [ 3217.768012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3217.773234] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3217.778193] [ 3217.780290] [ 3217.782388] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3217.787694] ? lock_acquire+0xf0/0x2b0 [ 3217.791444] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3217.796149] ? lock_release+0xc5/0x260 [ 3217.799895] _raw_spin_lock+0x30/0x40 [ 3217.803553] ? ceph_check_caps+0x63/0xae0 [ceph] [ 3217.808170] ceph_check_caps+0x63/0xae0 [ceph] [ 3217.812624] ? __lock_acquire+0xbfa/0x1e20 [ 3217.816719] ? lock_acquire+0xcb/0x2b0 [ 3217.820460] ? igrab+0x1d/0x50 [ 3217.823510] ? find_held_lock+0x2d/0x90 [ 3217.827342] ? find_held_lock+0x2d/0x90 [ 3217.831173] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3217.836572] ? lock_release+0xc5/0x260 [ 3217.840319] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3217.845544] delayed_work+0x1d8/0x280 [ceph] [ 3217.849826] ? process_scheduled_works+0x22a/0x510 [ 3217.854616] process_scheduled_works+0x2a4/0x510 [ 3217.859232] worker_thread+0x188/0x340 [ 3217.862981] ? rescuer_thread+0x310/0x310 [ 3217.866984] kthread+0xf5/0x130 [ 3217.870123] ? kthread_complete_and_exit+0x20/0x20 [ 3217.874905] ret_from_fork+0x31/0x40 [ 3217.878486] ? kthread_complete_and_exit+0x20/0x20 [ 3217.883268] ret_from_fork_asm+0x11/0x20 [ 3217.887193] [ 3245.522860] watchdog: BUG: soft lockup - CPU#7 stuck for 905s! [kworker/7:4:78808] [ 3245.530431] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3245.585880] irq event stamp: 2641948 [ 3245.589451] hardirqs last enabled at (2641947): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3245.599529] hardirqs last disabled at (2641948): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3245.609165] softirqs last enabled at (2641868): [] __do_softirq+0x2be/0x371 [ 3245.617765] softirqs last disabled at (2641863): [] irq_exit_rcu+0xb0/0xd0 [ 3245.626199] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3245.635584] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3245.643053] Workqueue: events delayed_work [ceph] [ 3245.647777] RIP: 0010:lock_release+0x14e/0x260 [ 3245.652216] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 3245.670952] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [ 3245.676168] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [ 3245.683292] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [ 3245.690418] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 3245.697541] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 3245.704666] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [ 3245.711791] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3245.719867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3245.725605] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3245.732729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3245.739855] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3245.746978] Call Trace: [ 3245.749423] [ 3245.751434] ? watchdog_timer_fn+0x1b4/0x220 [ 3245.755698] ? __hrtimer_run_queues+0x1d3/0x360 [ 3245.760222] ? hrtimer_interrupt+0x104/0x240 [ 3245.764483] ? __local_bh_enable+0x50/0x60 [ 3245.768576] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3245.773803] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3245.778758] [ 3245.780855] [ 3245.782954] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3245.788258] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3245.792968] ? lock_release+0x14e/0x260 [ 3245.796804] ? lock_release+0xc5/0x260 [ 3245.800550] _raw_spin_unlock+0x1b/0x40 [ 3245.804388] ceph_check_caps+0x26b/0xae0 [ceph] [ 3245.808930] ? lock_acquire+0xcb/0x2b0 [ 3245.812672] ? igrab+0x1d/0x50 [ 3245.815723] ? find_held_lock+0x2d/0x90 [ 3245.819554] ? find_held_lock+0x2d/0x90 [ 3245.823384] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3245.828785] ? lock_release+0xc5/0x260 [ 3245.832531] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3245.837756] delayed_work+0x1d8/0x280 [ceph] [ 3245.842039] ? process_scheduled_works+0x22a/0x510 [ 3245.846829] process_scheduled_works+0x2a4/0x510 [ 3245.851444] worker_thread+0x188/0x340 [ 3245.855193] ? rescuer_thread+0x310/0x310 [ 3245.859197] kthread+0xf5/0x130 [ 3245.862344] ? kthread_complete_and_exit+0x20/0x20 [ 3245.867126] ret_from_fork+0x31/0x40 [ 3245.870706] ? kthread_complete_and_exit+0x20/0x20 [ 3245.875490] ret_from_fork_asm+0x11/0x20 [ 3245.879413] [ 3273.523346] watchdog: BUG: soft lockup - CPU#7 stuck for 931s! [kworker/7:4:78808] [ 3273.530909] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3273.586359] irq event stamp: 2705256 [ 3273.589930] hardirqs last enabled at (2705255): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3273.600010] hardirqs last disabled at (2705256): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3273.609656] softirqs last enabled at (2705160): [] __do_softirq+0x2be/0x371 [ 3273.618261] softirqs last disabled at (2705155): [] irq_exit_rcu+0xb0/0xd0 [ 3273.626697] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3273.636082] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3273.643553] Workqueue: events delayed_work [ceph] [ 3273.648276] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3273.652626] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3273.671363] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 3273.676581] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 3273.683705] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 3273.690828] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 3273.697953] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 3273.705078] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3273.712201] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3273.720279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3273.726016] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3273.733141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3273.740264] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3273.747389] Call Trace: [ 3273.749836] [ 3273.751847] ? watchdog_timer_fn+0x1b4/0x220 [ 3273.756116] ? __hrtimer_run_queues+0x1d3/0x360 [ 3273.760642] ? hrtimer_interrupt+0x104/0x240 [ 3273.764903] ? __local_bh_enable+0x50/0x60 [ 3273.768997] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3273.774221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3273.779178] [ 3273.781277] [ 3273.783374] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3273.788682] ? lock_acquire+0xf0/0x2b0 [ 3273.792432] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3273.797145] ? lock_release+0xc5/0x260 [ 3273.800890] _raw_spin_lock+0x30/0x40 [ 3273.804548] ? ceph_check_caps+0x63/0xae0 [ceph] [ 3273.809166] ceph_check_caps+0x63/0xae0 [ceph] [ 3273.813616] ? __lock_acquire+0xbfa/0x1e20 [ 3273.817715] ? lock_acquire+0xcb/0x2b0 [ 3273.821463] ? igrab+0x1d/0x50 [ 3273.824514] ? find_held_lock+0x2d/0x90 [ 3273.828345] ? find_held_lock+0x2d/0x90 [ 3273.832177] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3273.837575] ? lock_release+0xc5/0x260 [ 3273.841322] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3273.846548] delayed_work+0x1d8/0x280 [ceph] [ 3273.850830] ? process_scheduled_works+0x22a/0x510 [ 3273.855620] process_scheduled_works+0x2a4/0x510 [ 3273.860235] worker_thread+0x188/0x340 [ 3273.863985] ? rescuer_thread+0x310/0x310 [ 3273.867986] kthread+0xf5/0x130 [ 3273.871127] ? kthread_complete_and_exit+0x20/0x20 [ 3273.875910] ret_from_fork+0x31/0x40 [ 3273.879489] ? kthread_complete_and_exit+0x20/0x20 [ 3273.884273] ret_from_fork_asm+0x11/0x20 [ 3273.888196] [ 3301.523832] watchdog: BUG: soft lockup - CPU#7 stuck for 957s! [kworker/7:4:78808] [ 3301.531399] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3301.586847] irq event stamp: 2767400 [ 3301.590419] hardirqs last enabled at (2767399): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3301.600498] hardirqs last disabled at (2767400): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3301.610136] softirqs last enabled at (2767320): [] __do_softirq+0x2be/0x371 [ 3301.618742] softirqs last disabled at (2767315): [] irq_exit_rcu+0xb0/0xd0 [ 3301.627175] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3301.636553] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3301.644025] Workqueue: events delayed_work [ceph] [ 3301.648746] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3301.653097] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3301.671834] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3301.677052] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3301.684177] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3301.691300] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3301.698423] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3301.705547] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3301.712672] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3301.720751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3301.726486] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3301.733611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3301.740736] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3301.747859] Call Trace: [ 3301.750305] [ 3301.752319] ? watchdog_timer_fn+0x1b4/0x220 [ 3301.756590] ? __hrtimer_run_queues+0x1d3/0x360 [ 3301.761124] ? hrtimer_interrupt+0x104/0x240 [ 3301.765394] ? __local_bh_enable+0x50/0x60 [ 3301.769486] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3301.774710] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3301.779669] [ 3301.781764] [ 3301.783862] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3301.789169] ? lock_acquire+0xf0/0x2b0 [ 3301.792910] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3301.798320] ? lock_release+0xc5/0x260 [ 3301.802063] _raw_spin_lock+0x30/0x40 [ 3301.805722] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3301.811118] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3301.816350] delayed_work+0x1d8/0x280 [ceph] [ 3301.820630] ? process_scheduled_works+0x22a/0x510 [ 3301.825419] process_scheduled_works+0x2a4/0x510 [ 3301.830035] worker_thread+0x188/0x340 [ 3301.833784] ? rescuer_thread+0x310/0x310 [ 3301.837787] kthread+0xf5/0x130 [ 3301.840925] ? kthread_complete_and_exit+0x20/0x20 [ 3301.845710] ret_from_fork+0x31/0x40 [ 3301.849289] ? kthread_complete_and_exit+0x20/0x20 [ 3301.854072] ret_from_fork_asm+0x11/0x20 [ 3301.858004] [ 3329.524318] watchdog: BUG: soft lockup - CPU#7 stuck for 983s! [kworker/7:4:78808] [ 3329.531906] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3329.587355] irq event stamp: 2830438 [ 3329.590924] hardirqs last enabled at (2830437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3329.601013] hardirqs last disabled at (2830438): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3329.610650] softirqs last enabled at (2830422): [] __do_softirq+0x2be/0x371 [ 3329.619247] softirqs last disabled at (2830417): [] irq_exit_rcu+0xb0/0xd0 [ 3329.627673] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3329.637049] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3329.644522] Workqueue: events delayed_work [ceph] [ 3329.649243] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3329.653596] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3329.672333] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3329.677550] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3329.684674] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3329.691799] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3329.698921] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3329.706045] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3329.713171] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3329.721248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3329.726984] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3329.734111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3329.741234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3329.748358] Call Trace: [ 3329.750804] [ 3329.752815] ? watchdog_timer_fn+0x1b4/0x220 [ 3329.757086] ? __hrtimer_run_queues+0x1d3/0x360 [ 3329.761622] ? hrtimer_interrupt+0x104/0x240 [ 3329.765890] ? __local_bh_enable+0x50/0x60 [ 3329.769985] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3329.775207] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3329.780164] [ 3329.782262] [ 3329.784359] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3329.789667] ? lock_acquire+0xf0/0x2b0 [ 3329.793410] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3329.798817] ? lock_release+0xc5/0x260 [ 3329.802563] _raw_spin_lock+0x30/0x40 [ 3329.806226] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3329.811627] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3329.816856] delayed_work+0x1d8/0x280 [ceph] [ 3329.821137] ? process_scheduled_works+0x22a/0x510 [ 3329.825926] process_scheduled_works+0x2a4/0x510 [ 3329.830540] worker_thread+0x188/0x340 [ 3329.834289] ? rescuer_thread+0x310/0x310 [ 3329.838292] kthread+0xf5/0x130 [ 3329.841431] ? kthread_complete_and_exit+0x20/0x20 [ 3329.846215] ret_from_fork+0x31/0x40 [ 3329.849786] ? kthread_complete_and_exit+0x20/0x20 [ 3329.854570] ret_from_fork_asm+0x11/0x20 [ 3329.858502] [ 3357.524803] watchdog: BUG: soft lockup - CPU#7 stuck for 1009s! [kworker/7:4:78808] [ 3357.532475] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3357.587922] irq event stamp: 2895126 [ 3357.591493] hardirqs last enabled at (2895125): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3357.601571] hardirqs last disabled at (2895126): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3357.611218] softirqs last enabled at (2895112): [] __do_softirq+0x2be/0x371 [ 3357.619824] softirqs last disabled at (2895099): [] irq_exit_rcu+0xb0/0xd0 [ 3357.628256] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3357.637634] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3357.645106] Workqueue: events delayed_work [ceph] [ 3357.649827] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3357.654179] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3357.672917] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3357.678133] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3357.685258] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3357.692383] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3357.699507] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3357.706629] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3357.713754] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3357.721831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3357.727570] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3357.734695] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3357.741817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3357.748942] Call Trace: [ 3357.751387] [ 3357.753399] ? watchdog_timer_fn+0x1b4/0x220 [ 3357.757661] ? __hrtimer_run_queues+0x1d3/0x360 [ 3357.762204] ? hrtimer_interrupt+0x104/0x240 [ 3357.766483] ? __local_bh_enable+0x50/0x60 [ 3357.770577] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3357.775802] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3357.780758] [ 3357.782856] [ 3357.784954] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3357.790261] ? lock_acquire+0xf0/0x2b0 [ 3357.794011] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3357.799419] ? lock_release+0xc5/0x260 [ 3357.803163] _raw_spin_lock+0x30/0x40 [ 3357.806820] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3357.812219] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3357.817447] delayed_work+0x1d8/0x280 [ceph] [ 3357.821728] ? process_scheduled_works+0x22a/0x510 [ 3357.826517] process_scheduled_works+0x2a4/0x510 [ 3357.831135] worker_thread+0x188/0x340 [ 3357.834882] ? rescuer_thread+0x310/0x310 [ 3357.838886] kthread+0xf5/0x130 [ 3357.842025] ? kthread_complete_and_exit+0x20/0x20 [ 3357.846809] ret_from_fork+0x31/0x40 [ 3357.850389] ? kthread_complete_and_exit+0x20/0x20 [ 3357.855172] ret_from_fork_asm+0x11/0x20 [ 3357.859103] [ 3385.525289] watchdog: BUG: soft lockup - CPU#7 stuck for 1036s! [kworker/7:4:78808] [ 3385.532943] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3385.588393] irq event stamp: 2958172 [ 3385.591963] hardirqs last enabled at (2958171): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3385.602041] hardirqs last disabled at (2958172): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3385.611680] softirqs last enabled at (2958158): [] __do_softirq+0x2be/0x371 [ 3385.620287] softirqs last disabled at (2958153): [] irq_exit_rcu+0xb0/0xd0 [ 3385.628720] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3385.638097] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3385.645567] Workqueue: events delayed_work [ceph] [ 3385.650291] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3385.654643] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3385.673380] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3385.678596] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3385.685722] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3385.692845] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3385.699969] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3385.707094] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3385.714218] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3385.722295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3385.728033] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3385.735157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3385.742280] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3385.749405] Call Trace: [ 3385.751851] [ 3385.753862] ? watchdog_timer_fn+0x1b4/0x220 [ 3385.758133] ? __hrtimer_run_queues+0x1d3/0x360 [ 3385.762658] ? hrtimer_interrupt+0x104/0x240 [ 3385.766921] ? __local_bh_enable+0x50/0x60 [ 3385.771013] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3385.776237] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3385.781196] [ 3385.783291] [ 3385.785389] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3385.790696] ? lock_acquire+0xf0/0x2b0 [ 3385.794438] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3385.799845] ? lock_release+0xc5/0x260 [ 3385.803590] _raw_spin_lock+0x30/0x40 [ 3385.807249] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3385.812647] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3385.817877] delayed_work+0x1d8/0x280 [ceph] [ 3385.822155] ? process_scheduled_works+0x22a/0x510 [ 3385.826938] process_scheduled_works+0x2a4/0x510 [ 3385.831554] worker_thread+0x188/0x340 [ 3385.835302] ? rescuer_thread+0x310/0x310 [ 3385.839305] kthread+0xf5/0x130 [ 3385.842444] ? kthread_complete_and_exit+0x20/0x20 [ 3385.847229] ret_from_fork+0x31/0x40 [ 3385.850807] ? kthread_complete_and_exit+0x20/0x20 [ 3385.855591] ret_from_fork_asm+0x11/0x20 [ 3385.859515] [ 3413.525774] watchdog: BUG: soft lockup - CPU#7 stuck for 1062s! [kworker/7:4:78808] [ 3413.533424] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3413.588873] irq event stamp: 3020182 [ 3413.592443] hardirqs last enabled at (3020181): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3413.602523] hardirqs last disabled at (3020182): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3413.612160] softirqs last enabled at (3020170): [] __do_softirq+0x2be/0x371 [ 3413.620758] softirqs last disabled at (3020165): [] irq_exit_rcu+0xb0/0xd0 [ 3413.629193] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3413.638570] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3413.646040] Workqueue: events delayed_work [ceph] [ 3413.650764] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3413.655115] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3413.673850] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 3413.679069] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 3413.686193] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 3413.693318] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 3413.700441] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3413.707566] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3413.714690] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3413.722767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3413.728504] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3413.735628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3413.742751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3413.749877] Call Trace: [ 3413.752322] [ 3413.754332] ? watchdog_timer_fn+0x1b4/0x220 [ 3413.758596] ? __hrtimer_run_queues+0x1d3/0x360 [ 3413.763124] ? hrtimer_interrupt+0x104/0x240 [ 3413.767392] ? __local_bh_enable+0x50/0x60 [ 3413.771486] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3413.776710] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3413.781667] [ 3413.783764] [ 3413.785862] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3413.791169] ? lock_acquire+0xf0/0x2b0 [ 3413.794920] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3413.800327] _raw_spin_lock+0x30/0x40 [ 3413.803985] ? igrab+0x1d/0x50 [ 3413.807042] igrab+0x1d/0x50 [ 3413.809921] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3413.815062] delayed_work+0x1d8/0x280 [ceph] [ 3413.819343] ? process_scheduled_works+0x22a/0x510 [ 3413.824134] process_scheduled_works+0x2a4/0x510 [ 3413.828749] worker_thread+0x188/0x340 [ 3413.832499] ? rescuer_thread+0x310/0x310 [ 3413.836502] kthread+0xf5/0x130 [ 3413.839647] ? kthread_complete_and_exit+0x20/0x20 [ 3413.844433] ret_from_fork+0x31/0x40 [ 3413.848011] ? kthread_complete_and_exit+0x20/0x20 [ 3413.852795] ret_from_fork_asm+0x11/0x20 [ 3413.856719] [ 3439.287062] libceph: mon1 (1)172.21.15.130:6789 session established [ 3441.526260] watchdog: BUG: soft lockup - CPU#7 stuck for 1088s! [kworker/7:4:78808] [ 3441.533913] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3441.589362] irq event stamp: 3080916 [ 3441.592932] hardirqs last enabled at (3080915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3441.603013] hardirqs last disabled at (3080916): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3441.612659] softirqs last enabled at (3080894): [] __do_softirq+0x2be/0x371 [ 3441.621264] softirqs last disabled at (3080889): [] irq_exit_rcu+0xb0/0xd0 [ 3441.629699] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3441.639075] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3441.646545] Workqueue: events delayed_work [ceph] [ 3441.651269] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 3441.655881] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 3441.674616] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [ 3441.679835] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [ 3441.686958] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 3441.694084] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 3441.701206] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 3441.708330] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 3441.715456] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3441.723534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3441.729269] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3441.736395] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3441.743519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3441.750641] Call Trace: [ 3441.753088] [ 3441.755100] ? watchdog_timer_fn+0x1b4/0x220 [ 3441.759372] ? __hrtimer_run_queues+0x1d3/0x360 [ 3441.763905] ? hrtimer_interrupt+0x104/0x240 [ 3441.768167] ? __local_bh_enable+0x50/0x60 [ 3441.772259] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3441.777483] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3441.782440] [ 3441.784538] [ 3441.786636] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3441.791944] ? do_raw_spin_lock+0x42/0xc0 [ 3441.795955] igrab+0x1d/0x50 [ 3441.798841] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3441.803989] delayed_work+0x1d8/0x280 [ceph] [ 3441.808272] ? process_scheduled_works+0x22a/0x510 [ 3441.813061] process_scheduled_works+0x2a4/0x510 [ 3441.817677] worker_thread+0x188/0x340 [ 3441.821424] ? rescuer_thread+0x310/0x310 [ 3441.825430] kthread+0xf5/0x130 [ 3441.828568] ? kthread_complete_and_exit+0x20/0x20 [ 3441.833350] ret_from_fork+0x31/0x40 [ 3441.836930] ? kthread_complete_and_exit+0x20/0x20 [ 3441.841715] ret_from_fork_asm+0x11/0x20 [ 3441.845647] [ 3469.526749] watchdog: BUG: soft lockup - CPU#7 stuck for 1114s! [kworker/7:4:78808] [ 3469.534402] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3469.589851] irq event stamp: 3142006 [ 3469.593422] hardirqs last enabled at (3142005): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3469.603500] hardirqs last disabled at (3142006): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3469.613138] softirqs last enabled at (3141962): [] __do_softirq+0x2be/0x371 [ 3469.621744] softirqs last disabled at (3141957): [] irq_exit_rcu+0xb0/0xd0 [ 3469.630178] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3469.639555] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3469.647025] Workqueue: events delayed_work [ceph] [ 3469.651749] RIP: 0010:lock_release+0x14e/0x260 [ 3469.656188] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 3469.674925] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [ 3469.680143] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [ 3469.687265] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [ 3469.694390] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 3469.701513] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 3469.708639] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [ 3469.715761] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3469.723840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3469.729578] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3469.736700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3469.743825] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3469.750949] Call Trace: [ 3469.753396] [ 3469.755408] ? watchdog_timer_fn+0x1b4/0x220 [ 3469.759679] ? __hrtimer_run_queues+0x1d3/0x360 [ 3469.764214] ? hrtimer_interrupt+0x104/0x240 [ 3469.768483] ? __local_bh_enable+0x50/0x60 [ 3469.772575] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3469.777799] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3469.782756] [ 3469.784853] [ 3469.786951] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3469.792257] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3469.797666] ? lock_release+0x14e/0x260 [ 3469.801498] _raw_spin_unlock+0x1b/0x40 [ 3469.805336] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3469.810572] delayed_work+0x1d8/0x280 [ceph] [ 3469.814853] ? process_scheduled_works+0x22a/0x510 [ 3469.819643] process_scheduled_works+0x2a4/0x510 [ 3469.824259] worker_thread+0x188/0x340 [ 3469.828007] ? rescuer_thread+0x310/0x310 [ 3469.832012] kthread+0xf5/0x130 [ 3469.835157] ? kthread_complete_and_exit+0x20/0x20 [ 3469.839942] ret_from_fork+0x31/0x40 [ 3469.843521] ? kthread_complete_and_exit+0x20/0x20 [ 3469.848306] ret_from_fork_asm+0x11/0x20 [ 3469.852237] [ 3497.527237] watchdog: BUG: soft lockup - CPU#7 stuck for 1140s! [kworker/7:4:78808] [ 3497.534889] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3497.590340] irq event stamp: 3203738 [ 3497.593909] hardirqs last enabled at (3203737): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3497.603988] hardirqs last disabled at (3203738): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3497.613628] softirqs last enabled at (3203706): [] __do_softirq+0x2be/0x371 [ 3497.622233] softirqs last disabled at (3203701): [] irq_exit_rcu+0xb0/0xd0 [ 3497.630661] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3497.640045] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3497.647515] Workqueue: events delayed_work [ceph] [ 3497.652238] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3497.656590] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3497.675325] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3497.680545] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3497.687667] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3497.694791] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3497.701916] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3497.709041] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3497.716165] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3497.724242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3497.729980] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3497.737103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3497.744229] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3497.751351] Call Trace: [ 3497.753799] [ 3497.755810] ? watchdog_timer_fn+0x1b4/0x220 [ 3497.760079] ? __hrtimer_run_queues+0x1d3/0x360 [ 3497.764606] ? hrtimer_interrupt+0x104/0x240 [ 3497.768876] ? __local_bh_enable+0x50/0x60 [ 3497.772968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3497.778192] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3497.783151] [ 3497.785247] [ 3497.787345] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3497.792654] ? lock_acquire+0xf0/0x2b0 [ 3497.796404] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3497.801811] ? lock_release+0xc5/0x260 [ 3497.805556] _raw_spin_lock+0x30/0x40 [ 3497.809212] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3497.814610] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3497.819840] delayed_work+0x1d8/0x280 [ceph] [ 3497.824120] ? process_scheduled_works+0x22a/0x510 [ 3497.828903] process_scheduled_works+0x2a4/0x510 [ 3497.833519] worker_thread+0x188/0x340 [ 3497.837268] ? rescuer_thread+0x310/0x310 [ 3497.841271] kthread+0xf5/0x130 [ 3497.844418] ? kthread_complete_and_exit+0x20/0x20 [ 3497.849200] ret_from_fork+0x31/0x40 [ 3497.852781] ? kthread_complete_and_exit+0x20/0x20 [ 3497.857563] ret_from_fork_asm+0x11/0x20 [ 3497.861487] [ 3525.527726] watchdog: BUG: soft lockup - CPU#7 stuck for 1166s! [kworker/7:4:78808] [ 3525.535379] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3525.590829] irq event stamp: 3264094 [ 3525.594398] hardirqs last enabled at (3264093): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3525.604479] hardirqs last disabled at (3264094): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3525.614116] softirqs last enabled at (3264080): [] __do_softirq+0x2be/0x371 [ 3525.622723] softirqs last disabled at (3264075): [] irq_exit_rcu+0xb0/0xd0 [ 3525.631157] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3525.640534] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3525.648006] Workqueue: events delayed_work [ceph] [ 3525.652727] RIP: 0010:do_raw_spin_lock+0xd/0xc0 [ 3525.657252] Code: 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 <48> 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 [ 3525.675989] RSP: 0018:ffffc90002117da8 EFLAGS: 00000282 [ 3525.681205] RAX: 00000000dead4ead RBX: ffff8881a1d34478 RCX: 00000000de0313a6 [ 3525.688331] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88817f867488 [ 3525.695454] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 3525.702578] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 3525.709703] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 3525.716827] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3525.724904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3525.730642] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3525.737765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3525.744891] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3525.752014] Call Trace: [ 3525.754459] [ 3525.756473] ? watchdog_timer_fn+0x1b4/0x220 [ 3525.760744] ? __hrtimer_run_queues+0x1d3/0x360 [ 3525.765276] ? hrtimer_interrupt+0x104/0x240 [ 3525.769539] ? __local_bh_enable+0x50/0x60 [ 3525.773631] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3525.778856] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3525.783813] [ 3525.785911] [ 3525.788009] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3525.793316] ? do_raw_spin_lock+0xd/0xc0 [ 3525.797239] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3525.802647] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3525.807876] delayed_work+0x1d8/0x280 [ceph] [ 3525.812156] ? process_scheduled_works+0x22a/0x510 [ 3525.816946] process_scheduled_works+0x2a4/0x510 [ 3525.821561] worker_thread+0x188/0x340 [ 3525.825312] ? rescuer_thread+0x310/0x310 [ 3525.829322] kthread+0xf5/0x130 [ 3525.832460] ? kthread_complete_and_exit+0x20/0x20 [ 3525.837246] ret_from_fork+0x31/0x40 [ 3525.840825] ? kthread_complete_and_exit+0x20/0x20 [ 3525.845608] ret_from_fork_asm+0x11/0x20 [ 3525.849539] [ 3553.528227] watchdog: BUG: soft lockup - CPU#7 stuck for 1192s! [kworker/7:4:78808] [ 3553.535877] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3553.591325] irq event stamp: 3324452 [ 3553.594898] hardirqs last enabled at (3324451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3553.604977] hardirqs last disabled at (3324452): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3553.614622] softirqs last enabled at (3324412): [] __do_softirq+0x2be/0x371 [ 3553.623228] softirqs last disabled at (3324407): [] irq_exit_rcu+0xb0/0xd0 [ 3553.631654] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3553.641032] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3553.648501] Workqueue: events delayed_work [ceph] [ 3553.653226] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3553.657585] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3553.676323] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 3553.681539] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 3553.688664] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 3553.695786] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 3553.702911] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 3553.710036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3553.717159] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3553.725238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3553.730975] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3553.738098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3553.745223] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3553.752347] Call Trace: [ 3553.754795] [ 3553.756805] ? watchdog_timer_fn+0x1b4/0x220 [ 3553.761076] ? __hrtimer_run_queues+0x1d3/0x360 [ 3553.765611] ? hrtimer_interrupt+0x104/0x240 [ 3553.769881] ? __local_bh_enable+0x50/0x60 [ 3553.773972] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3553.779197] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3553.784154] [ 3553.786251] [ 3553.788349] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3553.793658] ? lock_acquire+0xf0/0x2b0 [ 3553.797408] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3553.802122] ? lock_release+0xc5/0x260 [ 3553.805865] _raw_spin_lock+0x30/0x40 [ 3553.809525] ? ceph_check_caps+0x63/0xae0 [ceph] [ 3553.814152] ceph_check_caps+0x63/0xae0 [ceph] [ 3553.818602] ? __lock_acquire+0xbfa/0x1e20 [ 3553.822701] ? lock_acquire+0xcb/0x2b0 [ 3553.826448] ? igrab+0x1d/0x50 [ 3553.829499] ? find_held_lock+0x2d/0x90 [ 3553.833330] ? find_held_lock+0x2d/0x90 [ 3553.837162] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3553.842562] ? lock_release+0xc5/0x260 [ 3553.846306] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3553.851536] delayed_work+0x1d8/0x280 [ceph] [ 3553.855813] ? process_scheduled_works+0x22a/0x510 [ 3553.860604] process_scheduled_works+0x2a4/0x510 [ 3553.865220] worker_thread+0x188/0x340 [ 3553.868969] ? rescuer_thread+0x310/0x310 [ 3553.872972] kthread+0xf5/0x130 [ 3553.876111] ? kthread_complete_and_exit+0x20/0x20 [ 3553.880894] ret_from_fork+0x31/0x40 [ 3553.884474] ? kthread_complete_and_exit+0x20/0x20 [ 3553.889256] ret_from_fork_asm+0x11/0x20 [ 3553.893180] [ 3581.528705] watchdog: BUG: soft lockup - CPU#7 stuck for 1218s! [kworker/7:4:78808] [ 3581.536357] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3581.591807] irq event stamp: 3384806 [ 3581.595376] hardirqs last enabled at (3384805): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3581.605455] hardirqs last disabled at (3384806): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3581.615094] softirqs last enabled at (3384726): [] __do_softirq+0x2be/0x371 [ 3581.623700] softirqs last disabled at (3384721): [] irq_exit_rcu+0xb0/0xd0 [ 3581.632136] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3581.641520] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3581.648990] Workqueue: events delayed_work [ceph] [ 3581.653714] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3581.658066] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3581.676803] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3581.682019] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3581.689143] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3581.696267] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3581.703391] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3581.710516] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3581.717641] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3581.725717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3581.731454] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3581.738580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3581.745704] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3581.752828] Call Trace: [ 3581.755275] [ 3581.757285] ? watchdog_timer_fn+0x1b4/0x220 [ 3581.761557] ? __hrtimer_run_queues+0x1d3/0x360 [ 3581.766092] ? hrtimer_interrupt+0x104/0x240 [ 3581.770360] ? __local_bh_enable+0x50/0x60 [ 3581.774453] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3581.779679] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3581.784636] [ 3581.786731] [ 3581.788829] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3581.794136] ? lock_acquire+0xf0/0x2b0 [ 3581.797878] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3581.803287] ? lock_release+0xc5/0x260 [ 3581.807030] _raw_spin_lock+0x30/0x40 [ 3581.810687] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3581.816087] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3581.821315] delayed_work+0x1d8/0x280 [ceph] [ 3581.825598] ? process_scheduled_works+0x22a/0x510 [ 3581.830387] process_scheduled_works+0x2a4/0x510 [ 3581.835001] worker_thread+0x188/0x340 [ 3581.838751] ? rescuer_thread+0x310/0x310 [ 3581.842755] kthread+0xf5/0x130 [ 3581.845893] ? kthread_complete_and_exit+0x20/0x20 [ 3581.850678] ret_from_fork+0x31/0x40 [ 3581.854256] ? kthread_complete_and_exit+0x20/0x20 [ 3581.859040] ret_from_fork_asm+0x11/0x20 [ 3581.862963] [ 3609.529199] watchdog: BUG: soft lockup - CPU#7 stuck for 1244s! [kworker/7:4:78808] [ 3609.536854] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3609.592305] irq event stamp: 3445450 [ 3609.595874] hardirqs last enabled at (3445449): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3609.605953] hardirqs last disabled at (3445450): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3609.615592] softirqs last enabled at (3445436): [] __do_softirq+0x2be/0x371 [ 3609.624199] softirqs last disabled at (3445431): [] irq_exit_rcu+0xb0/0xd0 [ 3609.632633] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3609.642010] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3609.649480] Workqueue: events delayed_work [ceph] [ 3609.654204] RIP: 0010:rcu_is_watching+0xb/0x40 [ 3609.658649] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [ 3609.677385] RSP: 0018:ffffc90002117d28 EFLAGS: 00000203 [ 3609.682603] RAX: 0000000000000007 RBX: ffff88817f867488 RCX: 0000000000000000 [ 3609.689729] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88817f8674a0 [ 3609.696852] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 3609.703977] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [ 3609.711100] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3609.718225] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3609.726303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3609.732038] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3609.739164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3609.746286] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3609.753411] Call Trace: [ 3609.755858] [ 3609.757869] ? watchdog_timer_fn+0x1b4/0x220 [ 3609.762140] ? __hrtimer_run_queues+0x1d3/0x360 [ 3609.766666] ? hrtimer_interrupt+0x104/0x240 [ 3609.770927] ? __local_bh_enable+0x50/0x60 [ 3609.775021] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3609.780245] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3609.785201] [ 3609.787298] [ 3609.789397] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3609.794704] ? rcu_is_watching+0xb/0x40 [ 3609.798540] ? find_held_lock+0x2d/0x90 [ 3609.802379] lock_acquire+0x184/0x2b0 [ 3609.806038] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3609.811444] ? lock_release+0xc5/0x260 [ 3609.815189] _raw_spin_lock+0x30/0x40 [ 3609.818847] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3609.824246] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3609.829475] delayed_work+0x1d8/0x280 [ceph] [ 3609.833756] ? process_scheduled_works+0x22a/0x510 [ 3609.838545] process_scheduled_works+0x2a4/0x510 [ 3609.843161] worker_thread+0x188/0x340 [ 3609.846910] ? rescuer_thread+0x310/0x310 [ 3609.850913] kthread+0xf5/0x130 [ 3609.854052] ? kthread_complete_and_exit+0x20/0x20 [ 3609.858835] ret_from_fork+0x31/0x40 [ 3609.862406] ? kthread_complete_and_exit+0x20/0x20 [ 3609.867190] ret_from_fork_asm+0x11/0x20 [ 3609.871112] [ 3637.529694] watchdog: BUG: soft lockup - CPU#7 stuck for 1270s! [kworker/7:4:78808] [ 3637.537345] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3637.592793] irq event stamp: 3507828 [ 3637.596362] hardirqs last enabled at (3507827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3637.606443] hardirqs last disabled at (3507828): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3637.616081] softirqs last enabled at (3507702): [] __do_softirq+0x2be/0x371 [ 3637.624686] softirqs last disabled at (3507697): [] irq_exit_rcu+0xb0/0xd0 [ 3637.633121] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3637.642498] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3637.649970] Workqueue: events delayed_work [ceph] [ 3637.654691] RIP: 0010:igrab+0x2d/0x50 [ 3637.658351] Code: fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 <48> 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 [ 3637.677087] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [ 3637.682304] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 3637.689428] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 3637.696551] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 3637.703676] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 3637.710800] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 3637.717925] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3637.726003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3637.731739] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3637.738864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3637.745987] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3637.753112] Call Trace: [ 3637.755557] [ 3637.757568] ? watchdog_timer_fn+0x1b4/0x220 [ 3637.761840] ? __hrtimer_run_queues+0x1d3/0x360 [ 3637.766367] ? hrtimer_interrupt+0x104/0x240 [ 3637.770628] ? __local_bh_enable+0x50/0x60 [ 3637.774721] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3637.779945] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3637.784902] [ 3637.786999] [ 3637.789098] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3637.794403] ? igrab+0x2d/0x50 [ 3637.797454] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3637.802602] delayed_work+0x1d8/0x280 [ceph] [ 3637.806883] ? process_scheduled_works+0x22a/0x510 [ 3637.811673] process_scheduled_works+0x2a4/0x510 [ 3637.816290] worker_thread+0x188/0x340 [ 3637.820039] ? rescuer_thread+0x310/0x310 [ 3637.824042] kthread+0xf5/0x130 [ 3637.827189] ? kthread_complete_and_exit+0x20/0x20 [ 3637.831971] ret_from_fork+0x31/0x40 [ 3637.835551] ? kthread_complete_and_exit+0x20/0x20 [ 3637.840335] ret_from_fork_asm+0x11/0x20 [ 3637.844259] [ 3665.530188] watchdog: BUG: soft lockup - CPU#7 stuck for 1296s! [kworker/7:4:78808] [ 3665.537841] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3665.593291] irq event stamp: 3569222 [ 3665.596861] hardirqs last enabled at (3569221): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3665.606941] hardirqs last disabled at (3569222): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3665.616577] softirqs last enabled at (3569168): [] __do_softirq+0x2be/0x371 [ 3665.625175] softirqs last disabled at (3569163): [] irq_exit_rcu+0xb0/0xd0 [ 3665.633600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3665.642978] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3665.650449] Workqueue: events delayed_work [ceph] [ 3665.655171] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3665.659524] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3665.678260] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 3665.683477] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 3665.690602] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 3665.697726] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 3665.704849] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3665.711973] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3665.719099] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3665.727177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3665.732914] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3665.740037] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3665.747162] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3665.754285] Call Trace: [ 3665.756732] [ 3665.758742] ? watchdog_timer_fn+0x1b4/0x220 [ 3665.763004] ? __hrtimer_run_queues+0x1d3/0x360 [ 3665.767531] ? hrtimer_interrupt+0x104/0x240 [ 3665.771794] ? __local_bh_enable+0x50/0x60 [ 3665.775885] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3665.781109] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3665.786066] [ 3665.788165] [ 3665.790262] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3665.795571] ? lock_acquire+0xf0/0x2b0 [ 3665.799322] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3665.804727] _raw_spin_lock+0x30/0x40 [ 3665.808385] ? igrab+0x1d/0x50 [ 3665.811434] igrab+0x1d/0x50 [ 3665.814313] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3665.819454] delayed_work+0x1d8/0x280 [ceph] [ 3665.823736] ? process_scheduled_works+0x22a/0x510 [ 3665.828525] process_scheduled_works+0x2a4/0x510 [ 3665.833141] worker_thread+0x188/0x340 [ 3665.836891] ? rescuer_thread+0x310/0x310 [ 3665.840893] kthread+0xf5/0x130 [ 3665.844033] ? kthread_complete_and_exit+0x20/0x20 [ 3665.848815] ret_from_fork+0x31/0x40 [ 3665.852395] ? kthread_complete_and_exit+0x20/0x20 [ 3665.857188] ret_from_fork_asm+0x11/0x20 [ 3665.861118] [ 3693.530682] watchdog: BUG: soft lockup - CPU#7 stuck for 1322s! [kworker/7:4:78808] [ 3693.538356] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3693.593805] irq event stamp: 3632326 [ 3693.597377] hardirqs last enabled at (3632325): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3693.607456] hardirqs last disabled at (3632326): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3693.617094] softirqs last enabled at (3632304): [] __do_softirq+0x2be/0x371 [ 3693.625700] softirqs last disabled at (3632299): [] irq_exit_rcu+0xb0/0xd0 [ 3693.634134] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3693.643512] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3693.650982] Workqueue: events delayed_work [ceph] [ 3693.655705] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3693.660063] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3693.678801] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3693.684019] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3693.691144] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3693.698267] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3693.705390] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3693.712516] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3693.719639] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3693.727715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3693.733455] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3693.740578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3693.747701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3693.754826] Call Trace: [ 3693.757273] [ 3693.759283] ? watchdog_timer_fn+0x1b4/0x220 [ 3693.763547] ? __hrtimer_run_queues+0x1d3/0x360 [ 3693.768080] ? hrtimer_interrupt+0x104/0x240 [ 3693.772341] ? __local_bh_enable+0x50/0x60 [ 3693.776434] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3693.781658] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3693.786616] [ 3693.788713] [ 3693.790812] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3693.796118] ? lock_acquire+0xf0/0x2b0 [ 3693.799861] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3693.805267] ? lock_release+0xc5/0x260 [ 3693.809013] _raw_spin_lock+0x30/0x40 [ 3693.812670] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3693.818077] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3693.823307] delayed_work+0x1d8/0x280 [ceph] [ 3693.827589] ? process_scheduled_works+0x22a/0x510 [ 3693.832378] process_scheduled_works+0x2a4/0x510 [ 3693.836993] worker_thread+0x188/0x340 [ 3693.840741] ? rescuer_thread+0x310/0x310 [ 3693.844746] kthread+0xf5/0x130 [ 3693.847883] ? kthread_complete_and_exit+0x20/0x20 [ 3693.852668] ret_from_fork+0x31/0x40 [ 3693.856246] ? kthread_complete_and_exit+0x20/0x20 [ 3693.861031] ret_from_fork_asm+0x11/0x20 [ 3693.864962] [ 3721.531178] watchdog: BUG: soft lockup - CPU#7 stuck for 1348s! [kworker/7:4:78808] [ 3721.538827] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3721.594278] irq event stamp: 3697692 [ 3721.597848] hardirqs last enabled at (3697691): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3721.607927] hardirqs last disabled at (3697692): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3721.617564] softirqs last enabled at (3697678): [] __do_softirq+0x2be/0x371 [ 3721.626161] softirqs last disabled at (3697673): [] irq_exit_rcu+0xb0/0xd0 [ 3721.634586] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3721.643964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3721.651436] Workqueue: events delayed_work [ceph] [ 3721.656158] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3721.660510] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3721.679247] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3721.684463] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3721.691588] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3721.698711] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3721.705837] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3721.712962] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3721.720084] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3721.728161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3721.733900] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3721.741023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3721.748148] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3721.755273] Call Trace: [ 3721.757719] [ 3721.759729] ? watchdog_timer_fn+0x1b4/0x220 [ 3721.763993] ? __hrtimer_run_queues+0x1d3/0x360 [ 3721.768526] ? hrtimer_interrupt+0x104/0x240 [ 3721.772788] ? __local_bh_enable+0x50/0x60 [ 3721.776882] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3721.782104] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3721.787062] [ 3721.789160] [ 3721.791258] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3721.796565] ? lock_acquire+0xf0/0x2b0 [ 3721.800315] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3721.805723] ? lock_release+0xc5/0x260 [ 3721.809469] _raw_spin_lock+0x30/0x40 [ 3721.813133] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3721.818541] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3721.823769] delayed_work+0x1d8/0x280 [ceph] [ 3721.828050] ? process_scheduled_works+0x22a/0x510 [ 3721.832840] process_scheduled_works+0x2a4/0x510 [ 3721.837455] worker_thread+0x188/0x340 [ 3721.841205] ? rescuer_thread+0x310/0x310 [ 3721.845208] kthread+0xf5/0x130 [ 3721.848345] ? kthread_complete_and_exit+0x20/0x20 [ 3721.853129] ret_from_fork+0x31/0x40 [ 3721.856701] ? kthread_complete_and_exit+0x20/0x20 [ 3721.861486] ret_from_fork_asm+0x11/0x20 [ 3721.865417] [ 3722.426604] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [ 3722.435719] libceph: mon0 (1)172.21.15.50:6789 session established [ 3749.531673] watchdog: BUG: soft lockup - CPU#7 stuck for 1375s! [kworker/7:4:78808] [ 3749.539326] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3749.594775] irq event stamp: 3760258 [ 3749.598346] hardirqs last enabled at (3760257): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3749.608426] hardirqs last disabled at (3760258): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3749.618070] softirqs last enabled at (3760244): [] __do_softirq+0x2be/0x371 [ 3749.626669] softirqs last disabled at (3760239): [] irq_exit_rcu+0xb0/0xd0 [ 3749.635095] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3749.644470] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3749.651940] Workqueue: events delayed_work [ceph] [ 3749.656664] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3749.661016] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3749.679754] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 3749.684971] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 3749.692094] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 3749.699219] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 3749.706356] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3749.713485] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3749.720610] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3749.728686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3749.734422] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3749.741548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3749.748671] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3749.755795] Call Trace: [ 3749.758242] [ 3749.760254] ? watchdog_timer_fn+0x1b4/0x220 [ 3749.764523] ? __hrtimer_run_queues+0x1d3/0x360 [ 3749.769050] ? hrtimer_interrupt+0x104/0x240 [ 3749.773312] ? __local_bh_enable+0x50/0x60 [ 3749.777404] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3749.782628] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3749.787585] [ 3749.789682] [ 3749.791780] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3749.797088] ? lock_acquire+0xf0/0x2b0 [ 3749.800840] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3749.806248] _raw_spin_lock+0x30/0x40 [ 3749.809912] ? igrab+0x1d/0x50 [ 3749.812961] igrab+0x1d/0x50 [ 3749.815842] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3749.820990] delayed_work+0x1d8/0x280 [ceph] [ 3749.825272] ? process_scheduled_works+0x22a/0x510 [ 3749.830074] process_scheduled_works+0x2a4/0x510 [ 3749.834694] worker_thread+0x188/0x340 [ 3749.838445] ? rescuer_thread+0x310/0x310 [ 3749.842447] kthread+0xf5/0x130 [ 3749.845594] ? kthread_complete_and_exit+0x20/0x20 [ 3749.850377] ret_from_fork+0x31/0x40 [ 3749.853949] ? kthread_complete_and_exit+0x20/0x20 [ 3749.858731] ret_from_fork_asm+0x11/0x20 [ 3749.862654] [ 3777.532169] watchdog: BUG: soft lockup - CPU#7 stuck for 1401s! [kworker/7:4:78808] [ 3777.539840] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3777.595307] irq event stamp: 3822858 [ 3777.598878] hardirqs last enabled at (3822857): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3777.608959] hardirqs last disabled at (3822858): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3777.618603] softirqs last enabled at (3822836): [] __do_softirq+0x2be/0x371 [ 3777.627201] softirqs last disabled at (3822831): [] irq_exit_rcu+0xb0/0xd0 [ 3777.635635] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3777.645013] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3777.652484] Workqueue: events delayed_work [ceph] [ 3777.657207] RIP: 0010:ceph_check_caps+0xb/0xae0 [ceph] [ 3777.662353] Code: c0 48 39 f2 76 0c 48 01 c9 48 01 f2 48 39 d1 0f 93 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 66 0f 1f 00 0f 1f 44 00 00 41 57 <41> 56 41 55 41 54 55 89 f5 53 48 89 fb 48 81 ec b0 01 00 00 65 48 [ 3777.681091] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [ 3777.686309] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d6c [ 3777.693433] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8881a1d34478 [ 3777.700556] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 3777.707682] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 3777.714805] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 3777.721928] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3777.730007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3777.735743] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3777.742869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3777.749991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3777.757116] Call Trace: [ 3777.759584] [ 3777.761601] ? watchdog_timer_fn+0x1b4/0x220 [ 3777.765872] ? __hrtimer_run_queues+0x1d3/0x360 [ 3777.770406] ? hrtimer_interrupt+0x104/0x240 [ 3777.774676] ? __local_bh_enable+0x50/0x60 [ 3777.778769] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3777.783992] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3777.788950] [ 3777.791047] [ 3777.793144] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3777.798453] ? ceph_check_caps+0xb/0xae0 [ceph] [ 3777.802992] ? _raw_spin_unlock+0x23/0x40 [ 3777.806995] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3777.812224] delayed_work+0x1d8/0x280 [ceph] [ 3777.816504] ? process_scheduled_works+0x22a/0x510 [ 3777.821295] process_scheduled_works+0x2a4/0x510 [ 3777.825909] worker_thread+0x188/0x340 [ 3777.829660] ? rescuer_thread+0x310/0x310 [ 3777.833661] kthread+0xf5/0x130 [ 3777.836800] ? kthread_complete_and_exit+0x20/0x20 [ 3777.841584] ret_from_fork+0x31/0x40 [ 3777.845164] ? kthread_complete_and_exit+0x20/0x20 [ 3777.849948] ret_from_fork_asm+0x11/0x20 [ 3777.853880] [ 3796.155904] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [ 3805.532665] watchdog: BUG: soft lockup - CPU#7 stuck for 1427s! [kworker/7:4:78808] [ 3805.540331] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3805.595780] irq event stamp: 3884954 [ 3805.599350] hardirqs last enabled at (3884953): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3805.609428] hardirqs last disabled at (3884954): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3805.619065] softirqs last enabled at (3884914): [] __do_softirq+0x2be/0x371 [ 3805.627664] softirqs last disabled at (3884909): [] irq_exit_rcu+0xb0/0xd0 [ 3805.636097] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3805.645475] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3805.652946] Workqueue: events delayed_work [ceph] [ 3805.657667] RIP: 0010:_atomic_dec_and_lock+0x14/0x40 [ 3805.662626] Code: 63 c9 48 83 f9 07 76 ba b8 08 00 00 00 eb d2 44 89 d0 5b c3 66 90 f3 0f 1e fa 55 53 8b 07 83 f8 01 74 0e 8d 50 ff f0 0f b1 17 <75> f2 31 c0 5b 5d c3 48 89 fb 48 89 f7 48 89 f5 e8 07 2b 04 00 f0 [ 3805.681364] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [ 3805.686581] RAX: 0000000000000002 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 3805.693705] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [ 3805.700828] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [ 3805.707954] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [ 3805.715077] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 3805.722202] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3805.730278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3805.736016] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3805.743140] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3805.750265] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3805.757388] Call Trace: [ 3805.759834] [ 3805.761845] ? watchdog_timer_fn+0x1b4/0x220 [ 3805.766110] ? __hrtimer_run_queues+0x1d3/0x360 [ 3805.770643] ? hrtimer_interrupt+0x104/0x240 [ 3805.774904] ? __local_bh_enable+0x50/0x60 [ 3805.778998] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3805.784223] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3805.789180] [ 3805.791277] [ 3805.793375] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3805.798681] ? _atomic_dec_and_lock+0x14/0x40 [ 3805.803038] iput+0x50/0x220 [ 3805.805926] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 3805.811162] delayed_work+0x1d8/0x280 [ceph] [ 3805.815442] ? process_scheduled_works+0x22a/0x510 [ 3805.820232] process_scheduled_works+0x2a4/0x510 [ 3805.824847] worker_thread+0x188/0x340 [ 3805.828597] ? rescuer_thread+0x310/0x310 [ 3805.832601] kthread+0xf5/0x130 [ 3805.835739] ? kthread_complete_and_exit+0x20/0x20 [ 3805.840523] ret_from_fork+0x31/0x40 [ 3805.844102] ? kthread_complete_and_exit+0x20/0x20 [ 3805.848886] ret_from_fork_asm+0x11/0x20 [ 3805.852816] [ 3833.533160] watchdog: BUG: soft lockup - CPU#7 stuck for 1453s! [kworker/7:4:78808] [ 3833.540809] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3833.596260] irq event stamp: 3945618 [ 3833.599829] hardirqs last enabled at (3945617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3833.609909] hardirqs last disabled at (3945618): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3833.619556] softirqs last enabled at (3945574): [] __do_softirq+0x2be/0x371 [ 3833.628160] softirqs last disabled at (3945569): [] irq_exit_rcu+0xb0/0xd0 [ 3833.636587] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3833.645964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3833.653434] Workqueue: events delayed_work [ceph] [ 3833.658159] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3833.662518] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3833.681255] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 3833.686472] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 3833.693597] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 3833.700721] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 3833.707843] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 3833.714969] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3833.722094] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3833.730172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3833.735907] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3833.743033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3833.750155] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3833.757281] Call Trace: [ 3833.759725] [ 3833.761738] ? watchdog_timer_fn+0x1b4/0x220 [ 3833.766009] ? __hrtimer_run_queues+0x1d3/0x360 [ 3833.770544] ? hrtimer_interrupt+0x104/0x240 [ 3833.774813] ? __local_bh_enable+0x50/0x60 [ 3833.778905] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3833.784131] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3833.789088] [ 3833.791184] [ 3833.793281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3833.798591] ? lock_acquire+0xf0/0x2b0 [ 3833.802341] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3833.807749] _raw_spin_lock+0x30/0x40 [ 3833.811413] ? igrab+0x1d/0x50 [ 3833.814465] igrab+0x1d/0x50 [ 3833.817342] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 3833.822483] delayed_work+0x1d8/0x280 [ceph] [ 3833.826765] ? process_scheduled_works+0x22a/0x510 [ 3833.831555] process_scheduled_works+0x2a4/0x510 [ 3833.836170] worker_thread+0x188/0x340 [ 3833.839919] ? rescuer_thread+0x310/0x310 [ 3833.843923] kthread+0xf5/0x130 [ 3833.847061] ? kthread_complete_and_exit+0x20/0x20 [ 3833.851846] ret_from_fork+0x31/0x40 [ 3833.855424] ? kthread_complete_and_exit+0x20/0x20 [ 3833.860206] ret_from_fork_asm+0x11/0x20 [ 3833.864130] [ 3861.533663] watchdog: BUG: soft lockup - CPU#7 stuck for 1479s! [kworker/7:4:78808] [ 3861.541315] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3861.596765] irq event stamp: 4006496 [ 3861.600337] hardirqs last enabled at (4006495): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3861.610416] hardirqs last disabled at (4006496): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3861.620063] softirqs last enabled at (4006474): [] __do_softirq+0x2be/0x371 [ 3861.628668] softirqs last disabled at (4006469): [] irq_exit_rcu+0xb0/0xd0 [ 3861.637092] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3861.646469] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3861.653940] Workqueue: events delayed_work [ceph] [ 3861.658663] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3861.663016] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3861.681751] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 3861.686970] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 3861.694093] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 3861.701218] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 3861.708341] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 3861.715466] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3861.722589] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3861.730667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3861.736406] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3861.743529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3861.750653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3861.757778] Call Trace: [ 3861.760225] [ 3861.762235] ? watchdog_timer_fn+0x1b4/0x220 [ 3861.766506] ? __hrtimer_run_queues+0x1d3/0x360 [ 3861.771042] ? hrtimer_interrupt+0x104/0x240 [ 3861.775311] ? __local_bh_enable+0x50/0x60 [ 3861.779404] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3861.784628] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3861.789585] [ 3861.791683] [ 3861.793779] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3861.799088] ? lock_acquire+0xf0/0x2b0 [ 3861.802839] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 3861.807552] ? lock_release+0xc5/0x260 [ 3861.811297] _raw_spin_lock+0x30/0x40 [ 3861.814954] ? ceph_check_caps+0x63/0xae0 [ceph] [ 3861.819573] ceph_check_caps+0x63/0xae0 [ceph] [ 3861.824023] ? __lock_acquire+0xbfa/0x1e20 [ 3861.828121] ? lock_acquire+0xcb/0x2b0 [ 3861.831870] ? igrab+0x1d/0x50 [ 3861.834921] ? find_held_lock+0x2d/0x90 [ 3861.838753] ? find_held_lock+0x2d/0x90 [ 3861.842582] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3861.847982] ? lock_release+0xc5/0x260 [ 3861.851728] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 3861.856956] delayed_work+0x1d8/0x280 [ceph] [ 3861.861236] ? process_scheduled_works+0x22a/0x510 [ 3861.866026] process_scheduled_works+0x2a4/0x510 [ 3861.870642] worker_thread+0x188/0x340 [ 3861.874390] ? rescuer_thread+0x310/0x310 [ 3861.878394] kthread+0xf5/0x130 [ 3861.881532] ? kthread_complete_and_exit+0x20/0x20 [ 3861.886317] ret_from_fork+0x31/0x40 [ 3861.889895] ? kthread_complete_and_exit+0x20/0x20 [ 3861.894679] ret_from_fork_asm+0x11/0x20 [ 3861.898603] [ 3889.534161] watchdog: BUG: soft lockup - CPU#7 stuck for 1505s! [kworker/7:4:78808] [ 3889.541813] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3889.597263] irq event stamp: 4066946 [ 3889.600833] hardirqs last enabled at (4066945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3889.610912] hardirqs last disabled at (4066946): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3889.620551] softirqs last enabled at (4066866): [] __do_softirq+0x2be/0x371 [ 3889.629157] softirqs last disabled at (4066861): [] irq_exit_rcu+0xb0/0xd0 [ 3889.637592] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3889.646969] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3889.654438] Workqueue: events delayed_work [ceph] [ 3889.659161] RIP: 0010:lock_release+0x14e/0x260 [ 3889.663600] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 3889.682337] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [ 3889.687553] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [ 3889.694679] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [ 3889.701801] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 3889.708925] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 3889.716049] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [ 3889.723176] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3889.731252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3889.736988] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3889.744113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3889.751238] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3889.758362] Call Trace: [ 3889.760807] [ 3889.762818] ? watchdog_timer_fn+0x1b4/0x220 [ 3889.767082] ? __hrtimer_run_queues+0x1d3/0x360 [ 3889.771615] ? hrtimer_interrupt+0x104/0x240 [ 3889.775876] ? __local_bh_enable+0x50/0x60 [ 3889.779971] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3889.785193] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3889.790151] [ 3889.792249] [ 3889.794347] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3889.799651] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3889.805061] ? lock_release+0x14e/0x260 [ 3889.808893] _raw_spin_unlock+0x1b/0x40 [ 3889.812732] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3889.817959] delayed_work+0x1d8/0x280 [ceph] [ 3889.822238] ? process_scheduled_works+0x22a/0x510 [ 3889.827030] process_scheduled_works+0x2a4/0x510 [ 3889.831644] worker_thread+0x188/0x340 [ 3889.835393] ? rescuer_thread+0x310/0x310 [ 3889.839396] kthread+0xf5/0x130 [ 3889.842535] ? kthread_complete_and_exit+0x20/0x20 [ 3889.847320] ret_from_fork+0x31/0x40 [ 3889.850898] ? kthread_complete_and_exit+0x20/0x20 [ 3889.855683] ret_from_fork_asm+0x11/0x20 [ 3889.859614] [ 3917.534654] watchdog: BUG: soft lockup - CPU#7 stuck for 1531s! [kworker/7:4:78808] [ 3917.542301] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3917.597752] irq event stamp: 4130232 [ 3917.601322] hardirqs last enabled at (4130231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3917.611403] hardirqs last disabled at (4130232): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3917.621047] softirqs last enabled at (4130200): [] __do_softirq+0x2be/0x371 [ 3917.629645] softirqs last disabled at (4130195): [] irq_exit_rcu+0xb0/0xd0 [ 3917.638073] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3917.647457] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3917.654927] Workqueue: events delayed_work [ceph] [ 3917.659650] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3917.664000] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3917.682738] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3917.687955] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3917.695081] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3917.702205] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3917.709328] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3917.716453] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3917.723576] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3917.731654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3917.737392] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3917.744516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3917.751640] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3917.758765] Call Trace: [ 3917.761210] [ 3917.763221] ? watchdog_timer_fn+0x1b4/0x220 [ 3917.767484] ? __hrtimer_run_queues+0x1d3/0x360 [ 3917.772018] ? hrtimer_interrupt+0x104/0x240 [ 3917.776280] ? __local_bh_enable+0x50/0x60 [ 3917.780372] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3917.785597] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3917.790553] [ 3917.792651] [ 3917.794749] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3917.800056] ? lock_acquire+0xf0/0x2b0 [ 3917.803807] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3917.809214] ? lock_release+0xc5/0x260 [ 3917.812960] _raw_spin_lock+0x30/0x40 [ 3917.816626] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3917.822022] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3917.827254] delayed_work+0x1d8/0x280 [ceph] [ 3917.831532] ? process_scheduled_works+0x22a/0x510 [ 3917.836315] process_scheduled_works+0x2a4/0x510 [ 3917.840931] worker_thread+0x188/0x340 [ 3917.844679] ? rescuer_thread+0x310/0x310 [ 3917.848682] kthread+0xf5/0x130 [ 3917.851822] ? kthread_complete_and_exit+0x20/0x20 [ 3917.856604] ret_from_fork+0x31/0x40 [ 3917.860174] ? kthread_complete_and_exit+0x20/0x20 [ 3917.864958] ret_from_fork_asm+0x11/0x20 [ 3917.868882] [ 3945.535141] watchdog: BUG: soft lockup - CPU#7 stuck for 1557s! [kworker/7:4:78808] [ 3945.542792] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3945.598242] irq event stamp: 4192520 [ 3945.601810] hardirqs last enabled at (4192519): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3945.611891] hardirqs last disabled at (4192520): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3945.621537] softirqs last enabled at (4192506): [] __do_softirq+0x2be/0x371 [ 3945.630143] softirqs last disabled at (4192501): [] irq_exit_rcu+0xb0/0xd0 [ 3945.638578] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3945.647955] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3945.655427] Workqueue: events delayed_work [ceph] [ 3945.660149] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 3945.664499] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 3945.683235] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 3945.688454] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 3945.695577] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 3945.702702] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 3945.709827] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 3945.716950] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 3945.724074] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3945.732152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3945.737888] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3945.745013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3945.752138] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3945.759260] Call Trace: [ 3945.761707] [ 3945.763717] ? watchdog_timer_fn+0x1b4/0x220 [ 3945.767981] ? __hrtimer_run_queues+0x1d3/0x360 [ 3945.772508] ? hrtimer_interrupt+0x104/0x240 [ 3945.776768] ? __local_bh_enable+0x50/0x60 [ 3945.780860] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3945.786085] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3945.791042] [ 3945.793139] [ 3945.795237] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3945.800546] ? lock_acquire+0xf0/0x2b0 [ 3945.804295] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 3945.809704] ? lock_release+0xc5/0x260 [ 3945.813447] _raw_spin_lock+0x30/0x40 [ 3945.817106] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3945.822505] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 3945.827732] delayed_work+0x1d8/0x280 [ceph] [ 3945.832014] ? process_scheduled_works+0x22a/0x510 [ 3945.836804] process_scheduled_works+0x2a4/0x510 [ 3945.841419] worker_thread+0x188/0x340 [ 3945.845167] ? rescuer_thread+0x310/0x310 [ 3945.849172] kthread+0xf5/0x130 [ 3945.852310] ? kthread_complete_and_exit+0x20/0x20 [ 3945.857092] ret_from_fork+0x31/0x40 [ 3945.860663] ? kthread_complete_and_exit+0x20/0x20 [ 3945.865449] ret_from_fork_asm+0x11/0x20 [ 3945.869380] [ 3973.535640] watchdog: BUG: soft lockup - CPU#7 stuck for 1583s! [kworker/7:4:78808] [ 3973.543287] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 3973.598731] irq event stamp: 4252910 [ 3973.602301] hardirqs last enabled at (4252909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3973.612379] hardirqs last disabled at (4252910): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 3973.622017] softirqs last enabled at (4252896): [] __do_softirq+0x2be/0x371 [ 3973.630616] softirqs last disabled at (4252891): [] irq_exit_rcu+0xb0/0xd0 [ 3973.639048] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 3973.648426] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 3973.655898] Workqueue: events delayed_work [ceph] [ 3973.660611] RIP: 0010:_atomic_dec_and_lock+0x14/0x40 [ 3973.665569] Code: 63 c9 48 83 f9 07 76 ba b8 08 00 00 00 eb d2 44 89 d0 5b c3 66 90 f3 0f 1e fa 55 53 8b 07 83 f8 01 74 0e 8d 50 ff f0 0f b1 17 <75> f2 31 c0 5b 5d c3 48 89 fb 48 89 f7 48 89 f5 e8 07 2b 04 00 f0 [ 3973.684306] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [ 3973.689522] RAX: 0000000000000002 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 3973.696648] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [ 3973.703771] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [ 3973.710897] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [ 3973.718019] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 3973.725145] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 3973.733220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3973.738960] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 3973.746083] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3973.753206] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3973.760330] Call Trace: [ 3973.762778] [ 3973.764787] ? watchdog_timer_fn+0x1b4/0x220 [ 3973.769050] ? __hrtimer_run_queues+0x1d3/0x360 [ 3973.773578] ? hrtimer_interrupt+0x104/0x240 [ 3973.777846] ? __local_bh_enable+0x50/0x60 [ 3973.781941] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 3973.787163] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 3973.792121] [ 3973.794219] [ 3973.796317] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 3973.801623] ? _atomic_dec_and_lock+0x14/0x40 [ 3973.805982] iput+0x50/0x220 [ 3973.808867] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 3973.814102] delayed_work+0x1d8/0x280 [ceph] [ 3973.818384] ? process_scheduled_works+0x22a/0x510 [ 3973.823175] process_scheduled_works+0x2a4/0x510 [ 3973.827789] worker_thread+0x188/0x340 [ 3973.831538] ? rescuer_thread+0x310/0x310 [ 3973.835541] kthread+0xf5/0x130 [ 3973.838680] ? kthread_complete_and_exit+0x20/0x20 [ 3973.843464] ret_from_fork+0x31/0x40 [ 3973.847044] ? kthread_complete_and_exit+0x20/0x20 [ 3973.851828] ret_from_fork_asm+0x11/0x20 [ 3973.855759] [ 4001.536129] watchdog: BUG: soft lockup - CPU#7 stuck for 1609s! [kworker/7:4:78808] [ 4001.543778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4001.599228] irq event stamp: 4313868 [ 4001.602798] hardirqs last enabled at (4313867): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4001.612878] hardirqs last disabled at (4313868): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4001.622522] softirqs last enabled at (4313824): [] __do_softirq+0x2be/0x371 [ 4001.631121] softirqs last disabled at (4313819): [] irq_exit_rcu+0xb0/0xd0 [ 4001.639546] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4001.648924] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4001.656393] Workqueue: events delayed_work [ceph] [ 4001.661117] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4001.665468] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4001.684205] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4001.689423] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4001.696548] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4001.703672] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4001.710796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4001.717918] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4001.725042] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4001.733120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4001.738857] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4001.745984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4001.753107] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4001.760231] Call Trace: [ 4001.762678] [ 4001.764687] ? watchdog_timer_fn+0x1b4/0x220 [ 4001.768958] ? __hrtimer_run_queues+0x1d3/0x360 [ 4001.773485] ? hrtimer_interrupt+0x104/0x240 [ 4001.777747] ? __local_bh_enable+0x50/0x60 [ 4001.781840] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4001.787064] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4001.792020] [ 4001.794118] [ 4001.796215] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4001.801525] ? lock_acquire+0xf0/0x2b0 [ 4001.805273] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4001.809978] ? lock_release+0xc5/0x260 [ 4001.813723] _raw_spin_lock+0x30/0x40 [ 4001.817379] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4001.822001] ceph_check_caps+0x63/0xae0 [ceph] [ 4001.826450] ? __lock_acquire+0xbfa/0x1e20 [ 4001.830548] ? lock_acquire+0xcb/0x2b0 [ 4001.834297] ? igrab+0x1d/0x50 [ 4001.837349] ? find_held_lock+0x2d/0x90 [ 4001.841187] ? find_held_lock+0x2d/0x90 [ 4001.845020] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4001.850419] ? lock_release+0xc5/0x260 [ 4001.854165] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4001.859400] delayed_work+0x1d8/0x280 [ceph] [ 4001.863680] ? process_scheduled_works+0x22a/0x510 [ 4001.868471] process_scheduled_works+0x2a4/0x510 [ 4001.873086] worker_thread+0x188/0x340 [ 4001.876837] ? rescuer_thread+0x310/0x310 [ 4001.880847] kthread+0xf5/0x130 [ 4001.883985] ? kthread_complete_and_exit+0x20/0x20 [ 4001.888770] ret_from_fork+0x31/0x40 [ 4001.892349] ? kthread_complete_and_exit+0x20/0x20 [ 4001.897133] ret_from_fork_asm+0x11/0x20 [ 4001.901063] [ 4029.536629] watchdog: BUG: soft lockup - CPU#7 stuck for 1635s! [kworker/7:4:78808] [ 4029.544284] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4029.599735] irq event stamp: 4376452 [ 4029.603304] hardirqs last enabled at (4376451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4029.613383] hardirqs last disabled at (4376452): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4029.623020] softirqs last enabled at (4376372): [] __do_softirq+0x2be/0x371 [ 4029.631618] softirqs last disabled at (4376367): [] irq_exit_rcu+0xb0/0xd0 [ 4029.640045] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4029.649420] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4029.656894] Workqueue: events delayed_work [ceph] [ 4029.661614] RIP: 0010:lock_acquire+0x4a/0x2b0 [ 4029.665966] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [ 4029.684704] RSP: 0018:ffffc90002117b48 EFLAGS: 00000247 [ 4029.689921] RAX: 0000000000000007 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 4029.697045] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34938 [ 4029.704169] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 4029.711293] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4029.718416] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4029.725541] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4029.733619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4029.739355] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4029.746480] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4029.753603] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4029.760727] Call Trace: [ 4029.763175] [ 4029.765186] ? watchdog_timer_fn+0x1b4/0x220 [ 4029.769457] ? __hrtimer_run_queues+0x1d3/0x360 [ 4029.773993] ? hrtimer_interrupt+0x104/0x240 [ 4029.778261] ? __local_bh_enable+0x50/0x60 [ 4029.782353] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4029.787579] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4029.792536] [ 4029.794634] [ 4029.796732] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4029.802038] ? lock_acquire+0x4a/0x2b0 [ 4029.805790] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4029.810502] ? lock_release+0xc5/0x260 [ 4029.814247] _raw_spin_lock+0x30/0x40 [ 4029.817905] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4029.822523] ceph_check_caps+0x63/0xae0 [ceph] [ 4029.826975] ? __lock_acquire+0xbfa/0x1e20 [ 4029.831073] ? lock_acquire+0xcb/0x2b0 [ 4029.834822] ? igrab+0x1d/0x50 [ 4029.837872] ? find_held_lock+0x2d/0x90 [ 4029.841703] ? find_held_lock+0x2d/0x90 [ 4029.845533] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4029.850932] ? lock_release+0xc5/0x260 [ 4029.854678] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4029.859907] delayed_work+0x1d8/0x280 [ceph] [ 4029.864187] ? process_scheduled_works+0x22a/0x510 [ 4029.868977] process_scheduled_works+0x2a4/0x510 [ 4029.873592] worker_thread+0x188/0x340 [ 4029.877343] ? rescuer_thread+0x310/0x310 [ 4029.881352] kthread+0xf5/0x130 [ 4029.884492] ? kthread_complete_and_exit+0x20/0x20 [ 4029.889275] ret_from_fork+0x31/0x40 [ 4029.892855] ? kthread_complete_and_exit+0x20/0x20 [ 4029.897639] ret_from_fork_asm+0x11/0x20 [ 4029.901571] [ 4057.537119] watchdog: BUG: soft lockup - CPU#7 stuck for 1661s! [kworker/7:4:78808] [ 4057.544790] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4057.600241] irq event stamp: 4438818 [ 4057.603810] hardirqs last enabled at (4438817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4057.613890] hardirqs last disabled at (4438818): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4057.623528] softirqs last enabled at (4438738): [] __do_softirq+0x2be/0x371 [ 4057.632134] softirqs last disabled at (4438733): [] irq_exit_rcu+0xb0/0xd0 [ 4057.640567] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4057.649945] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4057.657417] Workqueue: events delayed_work [ceph] [ 4057.662139] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4057.666499] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4057.685235] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4057.690451] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4057.697577] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4057.704701] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4057.711825] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4057.718950] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4057.726072] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4057.734150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4057.739889] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4057.747011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4057.754137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4057.761261] Call Trace: [ 4057.763707] [ 4057.765718] ? watchdog_timer_fn+0x1b4/0x220 [ 4057.769990] ? __hrtimer_run_queues+0x1d3/0x360 [ 4057.774524] ? hrtimer_interrupt+0x104/0x240 [ 4057.778801] ? __local_bh_enable+0x50/0x60 [ 4057.782895] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4057.788120] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4057.793078] [ 4057.795173] [ 4057.797271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4057.802579] ? lock_acquire+0xf0/0x2b0 [ 4057.806330] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4057.811044] ? lock_release+0xc5/0x260 [ 4057.814788] _raw_spin_lock+0x30/0x40 [ 4057.818462] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4057.823082] ceph_check_caps+0x63/0xae0 [ceph] [ 4057.827532] ? __lock_acquire+0xbfa/0x1e20 [ 4057.831631] ? lock_acquire+0xcb/0x2b0 [ 4057.835378] ? igrab+0x1d/0x50 [ 4057.838431] ? find_held_lock+0x2d/0x90 [ 4057.842271] ? find_held_lock+0x2d/0x90 [ 4057.846100] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4057.851500] ? lock_release+0xc5/0x260 [ 4057.855247] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4057.860483] delayed_work+0x1d8/0x280 [ceph] [ 4057.864763] ? process_scheduled_works+0x22a/0x510 [ 4057.869552] process_scheduled_works+0x2a4/0x510 [ 4057.874168] worker_thread+0x188/0x340 [ 4057.877917] ? rescuer_thread+0x310/0x310 [ 4057.881920] kthread+0xf5/0x130 [ 4057.885059] ? kthread_complete_and_exit+0x20/0x20 [ 4057.889843] ret_from_fork+0x31/0x40 [ 4057.893423] ? kthread_complete_and_exit+0x20/0x20 [ 4057.898206] ret_from_fork_asm+0x11/0x20 [ 4057.902138] [ 4085.537614] watchdog: BUG: soft lockup - CPU#7 stuck for 1687s! [kworker/7:4:78808] [ 4085.545280] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4085.600729] irq event stamp: 4500630 [ 4085.604298] hardirqs last enabled at (4500629): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4085.614379] hardirqs last disabled at (4500630): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4085.624016] softirqs last enabled at (4500550): [] __do_softirq+0x2be/0x371 [ 4085.632613] softirqs last disabled at (4500545): [] irq_exit_rcu+0xb0/0xd0 [ 4085.641039] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4085.650416] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4085.657888] Workqueue: events delayed_work [ceph] [ 4085.662609] RIP: 0010:lock_release+0x16a/0x260 [ 4085.667048] Code: f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 c4 18 5b 5d 41 5c <41> 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 48 0f a3 05 18 bf 53 01 [ 4085.685786] RSP: 0018:ffffc90002117d78 EFLAGS: 00000296 [ 4085.691003] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [ 4085.698126] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [ 4085.705249] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 4085.712375] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 4085.719500] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [ 4085.726624] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4085.734701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4085.740437] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4085.747561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4085.754686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4085.761811] Call Trace: [ 4085.764256] [ 4085.766265] ? watchdog_timer_fn+0x1b4/0x220 [ 4085.770529] ? __hrtimer_run_queues+0x1d3/0x360 [ 4085.775055] ? hrtimer_interrupt+0x104/0x240 [ 4085.779317] ? __local_bh_enable+0x50/0x60 [ 4085.783412] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4085.788634] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4085.793592] [ 4085.795690] [ 4085.797787] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4085.803094] ? lock_release+0x16a/0x260 [ 4085.806932] _raw_spin_unlock+0x1b/0x40 [ 4085.810761] igrab+0x35/0x50 [ 4085.813640] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4085.818788] delayed_work+0x1d8/0x280 [ceph] [ 4085.823070] ? process_scheduled_works+0x22a/0x510 [ 4085.827860] process_scheduled_works+0x2a4/0x510 [ 4085.832476] worker_thread+0x188/0x340 [ 4085.836224] ? rescuer_thread+0x310/0x310 [ 4085.840227] kthread+0xf5/0x130 [ 4085.843366] ? kthread_complete_and_exit+0x20/0x20 [ 4085.848150] ret_from_fork+0x31/0x40 [ 4085.851729] ? kthread_complete_and_exit+0x20/0x20 [ 4085.856513] ret_from_fork_asm+0x11/0x20 [ 4085.860437] [ 4113.538126] watchdog: BUG: soft lockup - CPU#7 stuck for 1714s! [kworker/7:4:78808] [ 4113.545776] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4113.601226] irq event stamp: 4562446 [ 4113.604797] hardirqs last enabled at (4562445): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4113.614878] hardirqs last disabled at (4562446): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4113.624522] softirqs last enabled at (4562408): [] __do_softirq+0x2be/0x371 [ 4113.633120] softirqs last disabled at (4562403): [] irq_exit_rcu+0xb0/0xd0 [ 4113.641555] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4113.650932] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4113.658401] Workqueue: events delayed_work [ceph] [ 4113.663124] RIP: 0010:rcu_is_watching+0x29/0x40 [ 4113.667650] Code: 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e <74> 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [ 4113.686388] RSP: 0018:ffffc90002117d58 EFLAGS: 00000202 [ 4113.691604] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: ffffc90002117d54 [ 4113.698728] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [ 4113.705851] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 4113.712977] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 4113.720099] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 4113.727225] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4113.735301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4113.741039] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4113.748165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4113.755288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4113.762413] Call Trace: [ 4113.764857] [ 4113.766868] ? watchdog_timer_fn+0x1b4/0x220 [ 4113.771131] ? __hrtimer_run_queues+0x1d3/0x360 [ 4113.775658] ? hrtimer_interrupt+0x104/0x240 [ 4113.779921] ? __local_bh_enable+0x50/0x60 [ 4113.784013] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4113.789237] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4113.794192] [ 4113.796289] [ 4113.798387] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4113.803692] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4113.809103] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4113.814499] ? rcu_is_watching+0x29/0x40 [ 4113.818419] lock_release+0x1d7/0x260 [ 4113.822085] _raw_spin_unlock+0x1b/0x40 [ 4113.825915] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4113.831143] delayed_work+0x1d8/0x280 [ceph] [ 4113.835423] ? process_scheduled_works+0x22a/0x510 [ 4113.840205] process_scheduled_works+0x2a4/0x510 [ 4113.844820] worker_thread+0x188/0x340 [ 4113.848570] ? rescuer_thread+0x310/0x310 [ 4113.852573] kthread+0xf5/0x130 [ 4113.855711] ? kthread_complete_and_exit+0x20/0x20 [ 4113.860497] ret_from_fork+0x31/0x40 [ 4113.864074] ? kthread_complete_and_exit+0x20/0x20 [ 4113.868858] ret_from_fork_asm+0x11/0x20 [ 4113.872782] [ 4141.538620] watchdog: BUG: soft lockup - CPU#7 stuck for 1740s! [kworker/7:4:78808] [ 4141.546275] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4141.601725] irq event stamp: 4624424 [ 4141.605294] hardirqs last enabled at (4624423): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4141.615375] hardirqs last disabled at (4624424): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4141.625020] softirqs last enabled at (4624410): [] __do_softirq+0x2be/0x371 [ 4141.633617] softirqs last disabled at (4624405): [] irq_exit_rcu+0xb0/0xd0 [ 4141.642045] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4141.651429] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4141.658900] Workqueue: events delayed_work [ceph] [ 4141.663622] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4141.667975] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4141.686710] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4141.691929] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4141.699051] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4141.706175] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4141.713300] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4141.720424] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4141.727548] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4141.735627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4141.741365] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4141.748488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4141.755613] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4141.762736] Call Trace: [ 4141.765182] [ 4141.767192] ? watchdog_timer_fn+0x1b4/0x220 [ 4141.771456] ? __hrtimer_run_queues+0x1d3/0x360 [ 4141.775991] ? hrtimer_interrupt+0x104/0x240 [ 4141.780253] ? __local_bh_enable+0x50/0x60 [ 4141.784344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4141.789568] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4141.794525] [ 4141.796623] [ 4141.798721] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4141.804030] ? lock_acquire+0xf0/0x2b0 [ 4141.807779] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4141.812493] ? lock_release+0xc5/0x260 [ 4141.816238] _raw_spin_lock+0x30/0x40 [ 4141.819895] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4141.824514] ceph_check_caps+0x63/0xae0 [ceph] [ 4141.828965] ? __lock_acquire+0xbfa/0x1e20 [ 4141.833063] ? lock_acquire+0xcb/0x2b0 [ 4141.836812] ? igrab+0x1d/0x50 [ 4141.839863] ? find_held_lock+0x2d/0x90 [ 4141.843694] ? find_held_lock+0x2d/0x90 [ 4141.847525] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4141.852924] ? lock_release+0xc5/0x260 [ 4141.856670] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4141.861896] delayed_work+0x1d8/0x280 [ceph] [ 4141.866177] ? process_scheduled_works+0x22a/0x510 [ 4141.870967] process_scheduled_works+0x2a4/0x510 [ 4141.875582] worker_thread+0x188/0x340 [ 4141.879333] ? rescuer_thread+0x310/0x310 [ 4141.883334] kthread+0xf5/0x130 [ 4141.886472] ? kthread_complete_and_exit+0x20/0x20 [ 4141.891257] ret_from_fork+0x31/0x40 [ 4141.894828] ? kthread_complete_and_exit+0x20/0x20 [ 4141.899612] ret_from_fork_asm+0x11/0x20 [ 4141.903535] [ 4169.539097] watchdog: BUG: soft lockup - CPU#7 stuck for 1766s! [kworker/7:4:78808] [ 4169.546748] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4169.602197] irq event stamp: 4686428 [ 4169.605766] hardirqs last enabled at (4686427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4169.615847] hardirqs last disabled at (4686428): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4169.625493] softirqs last enabled at (4686348): [] __do_softirq+0x2be/0x371 [ 4169.634100] softirqs last disabled at (4686343): [] irq_exit_rcu+0xb0/0xd0 [ 4169.642531] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4169.651908] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4169.659381] Workqueue: events delayed_work [ceph] [ 4169.664102] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4169.668453] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4169.687192] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4169.692409] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4169.699531] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4169.706656] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4169.713782] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4169.720904] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4169.728029] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4169.736107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4169.741844] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4169.748969] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4169.756092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4169.763216] Call Trace: [ 4169.765664] [ 4169.767674] ? watchdog_timer_fn+0x1b4/0x220 [ 4169.771945] ? __hrtimer_run_queues+0x1d3/0x360 [ 4169.776472] ? hrtimer_interrupt+0x104/0x240 [ 4169.780742] ? __local_bh_enable+0x50/0x60 [ 4169.784834] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4169.790058] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4169.795016] [ 4169.797112] [ 4169.799210] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4169.804517] ? lock_acquire+0xf0/0x2b0 [ 4169.808260] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4169.812972] ? lock_release+0xc5/0x260 [ 4169.816718] _raw_spin_lock+0x30/0x40 [ 4169.820375] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4169.824993] ceph_check_caps+0x63/0xae0 [ceph] [ 4169.829444] ? __lock_acquire+0xbfa/0x1e20 [ 4169.833542] ? lock_acquire+0xcb/0x2b0 [ 4169.837283] ? igrab+0x1d/0x50 [ 4169.840334] ? find_held_lock+0x2d/0x90 [ 4169.844165] ? find_held_lock+0x2d/0x90 [ 4169.847997] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4169.853397] ? lock_release+0xc5/0x260 [ 4169.857143] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4169.862369] delayed_work+0x1d8/0x280 [ceph] [ 4169.866650] ? process_scheduled_works+0x22a/0x510 [ 4169.871440] process_scheduled_works+0x2a4/0x510 [ 4169.876055] worker_thread+0x188/0x340 [ 4169.879804] ? rescuer_thread+0x310/0x310 [ 4169.883806] kthread+0xf5/0x130 [ 4169.886945] ? kthread_complete_and_exit+0x20/0x20 [ 4169.891730] ret_from_fork+0x31/0x40 [ 4169.895309] ? kthread_complete_and_exit+0x20/0x20 [ 4169.900093] ret_from_fork_asm+0x11/0x20 [ 4169.904014] [ 4197.539591] watchdog: BUG: soft lockup - CPU#7 stuck for 1792s! [kworker/7:4:78808] [ 4197.547245] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4197.602693] irq event stamp: 4747694 [ 4197.606265] hardirqs last enabled at (4747693): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4197.616343] hardirqs last disabled at (4747694): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4197.625981] softirqs last enabled at (4747614): [] __do_softirq+0x2be/0x371 [ 4197.634588] softirqs last disabled at (4747609): [] irq_exit_rcu+0xb0/0xd0 [ 4197.643022] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4197.652408] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4197.659878] Workqueue: events delayed_work [ceph] [ 4197.664602] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4197.668961] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4197.687698] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4197.692914] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4197.700039] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4197.707162] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4197.714288] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4197.721413] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4197.728537] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4197.736614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4197.742351] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4197.749474] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4197.756598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4197.763722] Call Trace: [ 4197.766170] [ 4197.768181] ? watchdog_timer_fn+0x1b4/0x220 [ 4197.772452] ? __hrtimer_run_queues+0x1d3/0x360 [ 4197.776985] ? hrtimer_interrupt+0x104/0x240 [ 4197.781248] ? __local_bh_enable+0x50/0x60 [ 4197.785340] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4197.790563] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4197.795523] [ 4197.797618] [ 4197.799716] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4197.805024] ? lock_acquire+0xf0/0x2b0 [ 4197.808776] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4197.814183] _raw_spin_lock+0x30/0x40 [ 4197.817840] ? igrab+0x1d/0x50 [ 4197.820891] igrab+0x1d/0x50 [ 4197.823769] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4197.828919] delayed_work+0x1d8/0x280 [ceph] [ 4197.833200] ? process_scheduled_works+0x22a/0x510 [ 4197.837988] process_scheduled_works+0x2a4/0x510 [ 4197.842604] worker_thread+0x188/0x340 [ 4197.846355] ? rescuer_thread+0x310/0x310 [ 4197.850366] kthread+0xf5/0x130 [ 4197.853512] ? kthread_complete_and_exit+0x20/0x20 [ 4197.858296] ret_from_fork+0x31/0x40 [ 4197.861875] ? kthread_complete_and_exit+0x20/0x20 [ 4197.866660] ret_from_fork_asm+0x11/0x20 [ 4197.870590] [ 4225.540085] watchdog: BUG: soft lockup - CPU#7 stuck for 1818s! [kworker/7:4:78808] [ 4225.547742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4225.603200] irq event stamp: 4809450 [ 4225.606769] hardirqs last enabled at (4809449): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4225.616849] hardirqs last disabled at (4809450): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4225.626488] softirqs last enabled at (4809428): [] __do_softirq+0x2be/0x371 [ 4225.635094] softirqs last disabled at (4809423): [] irq_exit_rcu+0xb0/0xd0 [ 4225.643527] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4225.652904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4225.660375] Workqueue: events delayed_work [ceph] [ 4225.665097] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4225.669451] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4225.688185] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4225.693402] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4225.700529] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4225.707652] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4225.714776] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4225.721901] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4225.729025] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4225.737103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4225.742840] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4225.749965] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4225.757089] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4225.764211] Call Trace: [ 4225.766658] [ 4225.768668] ? watchdog_timer_fn+0x1b4/0x220 [ 4225.772931] ? __hrtimer_run_queues+0x1d3/0x360 [ 4225.777458] ? hrtimer_interrupt+0x104/0x240 [ 4225.781728] ? __local_bh_enable+0x50/0x60 [ 4225.785820] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4225.791043] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4225.796001] [ 4225.798100] [ 4225.800196] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4225.805505] ? lock_acquire+0xf0/0x2b0 [ 4225.809257] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4225.814662] _raw_spin_lock+0x30/0x40 [ 4225.818320] ? igrab+0x1d/0x50 [ 4225.821369] igrab+0x1d/0x50 [ 4225.824248] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4225.829390] delayed_work+0x1d8/0x280 [ceph] [ 4225.833671] ? process_scheduled_works+0x22a/0x510 [ 4225.838459] process_scheduled_works+0x2a4/0x510 [ 4225.843077] worker_thread+0x188/0x340 [ 4225.846826] ? rescuer_thread+0x310/0x310 [ 4225.850828] kthread+0xf5/0x130 [ 4225.853967] ? kthread_complete_and_exit+0x20/0x20 [ 4225.858749] ret_from_fork+0x31/0x40 [ 4225.862330] ? kthread_complete_and_exit+0x20/0x20 [ 4225.867114] ret_from_fork_asm+0x11/0x20 [ 4225.871045] [ 4253.540578] watchdog: BUG: soft lockup - CPU#7 stuck for 1844s! [kworker/7:4:78808] [ 4253.548249] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4253.603699] irq event stamp: 4870132 [ 4253.607268] hardirqs last enabled at (4870131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4253.617347] hardirqs last disabled at (4870132): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4253.626993] softirqs last enabled at (4870110): [] __do_softirq+0x2be/0x371 [ 4253.635592] softirqs last disabled at (4870105): [] irq_exit_rcu+0xb0/0xd0 [ 4253.644025] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4253.653401] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4253.660873] Workqueue: events delayed_work [ceph] [ 4253.665597] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4253.669957] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4253.688694] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4253.693910] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4253.701034] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4253.708159] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4253.715282] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4253.722405] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4253.729532] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4253.737609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4253.743346] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4253.750469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4253.757595] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4253.764718] Call Trace: [ 4253.767163] [ 4253.769174] ? watchdog_timer_fn+0x1b4/0x220 [ 4253.773439] ? __hrtimer_run_queues+0x1d3/0x360 [ 4253.777972] ? hrtimer_interrupt+0x104/0x240 [ 4253.782234] ? __local_bh_enable+0x50/0x60 [ 4253.786326] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4253.791551] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4253.796507] [ 4253.798604] [ 4253.800702] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4253.806010] ? lock_acquire+0xf0/0x2b0 [ 4253.809753] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4253.815161] _raw_spin_lock+0x30/0x40 [ 4253.818818] ? igrab+0x1d/0x50 [ 4253.821866] igrab+0x1d/0x50 [ 4253.824746] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4253.829888] delayed_work+0x1d8/0x280 [ceph] [ 4253.834169] ? process_scheduled_works+0x22a/0x510 [ 4253.838959] process_scheduled_works+0x2a4/0x510 [ 4253.843575] worker_thread+0x188/0x340 [ 4253.847324] ? rescuer_thread+0x310/0x310 [ 4253.851327] kthread+0xf5/0x130 [ 4253.854465] ? kthread_complete_and_exit+0x20/0x20 [ 4253.859248] ret_from_fork+0x31/0x40 [ 4253.862827] ? kthread_complete_and_exit+0x20/0x20 [ 4253.867612] ret_from_fork_asm+0x11/0x20 [ 4253.871544] [ 4281.541072] watchdog: BUG: soft lockup - CPU#7 stuck for 1870s! [kworker/7:4:78808] [ 4281.548728] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4281.604178] irq event stamp: 4933394 [ 4281.607749] hardirqs last enabled at (4933393): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4281.617828] hardirqs last disabled at (4933394): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4281.627474] softirqs last enabled at (4933372): [] __do_softirq+0x2be/0x371 [ 4281.636079] softirqs last disabled at (4933367): [] irq_exit_rcu+0xb0/0xd0 [ 4281.644513] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4281.653891] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4281.661362] Workqueue: events delayed_work [ceph] [ 4281.666084] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4281.670435] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4281.689173] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4281.694390] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4281.701513] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4281.708639] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4281.715763] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4281.722886] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4281.730012] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4281.738088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4281.743827] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4281.750950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4281.758074] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4281.765199] Call Trace: [ 4281.767646] [ 4281.769654] ? watchdog_timer_fn+0x1b4/0x220 [ 4281.773917] ? __hrtimer_run_queues+0x1d3/0x360 [ 4281.778444] ? hrtimer_interrupt+0x104/0x240 [ 4281.782705] ? __local_bh_enable+0x50/0x60 [ 4281.786798] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4281.792022] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4281.796980] [ 4281.799076] [ 4281.801174] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4281.806483] ? lock_acquire+0xf0/0x2b0 [ 4281.810233] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4281.815641] ? lock_release+0xc5/0x260 [ 4281.819384] _raw_spin_lock+0x30/0x40 [ 4281.823042] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4281.828449] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4281.833677] delayed_work+0x1d8/0x280 [ceph] [ 4281.837959] ? process_scheduled_works+0x22a/0x510 [ 4281.842749] process_scheduled_works+0x2a4/0x510 [ 4281.847366] worker_thread+0x188/0x340 [ 4281.851114] ? rescuer_thread+0x310/0x310 [ 4281.855117] kthread+0xf5/0x130 [ 4281.858256] ? kthread_complete_and_exit+0x20/0x20 [ 4281.863038] ret_from_fork+0x31/0x40 [ 4281.866609] ? kthread_complete_and_exit+0x20/0x20 [ 4281.871394] ret_from_fork_asm+0x11/0x20 [ 4281.875326] [ 4309.541566] watchdog: BUG: soft lockup - CPU#7 stuck for 1896s! [kworker/7:4:78808] [ 4309.549243] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4309.604694] irq event stamp: 4995142 [ 4309.608263] hardirqs last enabled at (4995141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4309.618342] hardirqs last disabled at (4995142): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4309.627980] softirqs last enabled at (4995128): [] __do_softirq+0x2be/0x371 [ 4309.636587] softirqs last disabled at (4995123): [] irq_exit_rcu+0xb0/0xd0 [ 4309.645022] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4309.654406] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4309.661877] Workqueue: events delayed_work [ceph] [ 4309.666599] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4309.670950] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4309.689688] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4309.694905] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4309.702031] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4309.709154] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4309.716277] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4309.723401] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4309.730527] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4309.738604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4309.744340] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4309.751466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4309.758589] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4309.765714] Call Trace: [ 4309.768159] [ 4309.770170] ? watchdog_timer_fn+0x1b4/0x220 [ 4309.774443] ? __hrtimer_run_queues+0x1d3/0x360 [ 4309.778976] ? hrtimer_interrupt+0x104/0x240 [ 4309.783238] ? __local_bh_enable+0x50/0x60 [ 4309.787330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4309.792555] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4309.797512] [ 4309.799610] [ 4309.801708] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4309.807015] ? lock_acquire+0xf0/0x2b0 [ 4309.810766] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4309.816173] ? lock_release+0xc5/0x260 [ 4309.819916] _raw_spin_lock+0x30/0x40 [ 4309.823573] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4309.828974] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4309.834202] delayed_work+0x1d8/0x280 [ceph] [ 4309.838482] ? process_scheduled_works+0x22a/0x510 [ 4309.843272] process_scheduled_works+0x2a4/0x510 [ 4309.847889] worker_thread+0x188/0x340 [ 4309.851637] ? rescuer_thread+0x310/0x310 [ 4309.855641] kthread+0xf5/0x130 [ 4309.858780] ? kthread_complete_and_exit+0x20/0x20 [ 4309.863562] ret_from_fork+0x31/0x40 [ 4309.867141] ? kthread_complete_and_exit+0x20/0x20 [ 4309.871926] ret_from_fork_asm+0x11/0x20 [ 4309.875848] [ 4337.542059] watchdog: BUG: soft lockup - CPU#7 stuck for 1922s! [kworker/7:4:78808] [ 4337.549730] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4337.605183] irq event stamp: 5056634 [ 4337.608752] hardirqs last enabled at (5056633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4337.618830] hardirqs last disabled at (5056634): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4337.628470] softirqs last enabled at (5056628): [] __do_softirq+0x2be/0x371 [ 4337.637075] softirqs last disabled at (5056623): [] irq_exit_rcu+0xb0/0xd0 [ 4337.645509] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4337.654886] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4337.662357] Workqueue: events delayed_work [ceph] [ 4337.667081] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4337.671455] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4337.690194] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4337.695412] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4337.702537] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4337.709661] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4337.716784] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4337.723908] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4337.731032] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4337.739109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4337.744847] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4337.751972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4337.759096] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4337.766220] Call Trace: [ 4337.768664] [ 4337.770675] ? watchdog_timer_fn+0x1b4/0x220 [ 4337.774938] ? __hrtimer_run_queues+0x1d3/0x360 [ 4337.779465] ? hrtimer_interrupt+0x104/0x240 [ 4337.783741] ? __local_bh_enable+0x50/0x60 [ 4337.787836] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4337.793062] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4337.798018] [ 4337.800116] [ 4337.802213] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4337.807521] ? lock_acquire+0xf0/0x2b0 [ 4337.811272] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4337.816679] ? lock_release+0xc5/0x260 [ 4337.820424] _raw_spin_lock+0x30/0x40 [ 4337.824090] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4337.829496] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4337.834741] delayed_work+0x1d8/0x280 [ceph] [ 4337.839023] ? process_scheduled_works+0x22a/0x510 [ 4337.843812] process_scheduled_works+0x2a4/0x510 [ 4337.848431] worker_thread+0x188/0x340 [ 4337.852195] ? rescuer_thread+0x310/0x310 [ 4337.856199] kthread+0xf5/0x130 [ 4337.859337] ? kthread_complete_and_exit+0x20/0x20 [ 4337.864120] ret_from_fork+0x31/0x40 [ 4337.867692] ? kthread_complete_and_exit+0x20/0x20 [ 4337.872475] ret_from_fork_asm+0x11/0x20 [ 4337.876397] [ 4365.542551] watchdog: BUG: soft lockup - CPU#7 stuck for 1948s! [kworker/7:4:78808] [ 4365.550204] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4365.605652] irq event stamp: 5116974 [ 4365.609224] hardirqs last enabled at (5116973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4365.619304] hardirqs last disabled at (5116974): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4365.628940] softirqs last enabled at (5116960): [] __do_softirq+0x2be/0x371 [ 4365.637539] softirqs last disabled at (5116955): [] irq_exit_rcu+0xb0/0xd0 [ 4365.645962] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4365.655341] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4365.662811] Workqueue: events delayed_work [ceph] [ 4365.667525] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4365.671876] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4365.690613] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4365.695831] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4365.702956] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4365.710079] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4365.717204] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4365.724329] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4365.731450] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4365.739529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4365.745267] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4365.752391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4365.759514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4365.766638] Call Trace: [ 4365.769083] [ 4365.771094] ? watchdog_timer_fn+0x1b4/0x220 [ 4365.775359] ? __hrtimer_run_queues+0x1d3/0x360 [ 4365.779886] ? hrtimer_interrupt+0x104/0x240 [ 4365.784155] ? __local_bh_enable+0x50/0x60 [ 4365.788247] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4365.793472] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4365.798429] [ 4365.800525] [ 4365.802623] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4365.807932] ? lock_acquire+0xf0/0x2b0 [ 4365.811684] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4365.817091] _raw_spin_lock+0x30/0x40 [ 4365.820746] ? igrab+0x1d/0x50 [ 4365.823798] igrab+0x1d/0x50 [ 4365.826675] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4365.831818] delayed_work+0x1d8/0x280 [ceph] [ 4365.836097] ? process_scheduled_works+0x22a/0x510 [ 4365.840887] process_scheduled_works+0x2a4/0x510 [ 4365.845503] worker_thread+0x188/0x340 [ 4365.849253] ? rescuer_thread+0x310/0x310 [ 4365.853257] kthread+0xf5/0x130 [ 4365.856401] ? kthread_complete_and_exit+0x20/0x20 [ 4365.861186] ret_from_fork+0x31/0x40 [ 4365.864758] ? kthread_complete_and_exit+0x20/0x20 [ 4365.869541] ret_from_fork_asm+0x11/0x20 [ 4365.873464] [ 4393.543044] watchdog: BUG: soft lockup - CPU#7 stuck for 1974s! [kworker/7:4:78808] [ 4393.550692] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4393.606143] irq event stamp: 5177232 [ 4393.609712] hardirqs last enabled at (5177231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4393.619792] hardirqs last disabled at (5177232): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4393.629439] softirqs last enabled at (5177210): [] __do_softirq+0x2be/0x371 [ 4393.638045] softirqs last disabled at (5177205): [] irq_exit_rcu+0xb0/0xd0 [ 4393.646478] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4393.655855] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4393.663327] Workqueue: events delayed_work [ceph] [ 4393.668050] RIP: 0010:lock_release+0x77/0x260 [ 4393.672400] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [ 4393.691137] RSP: 0018:ffffc90002117b70 EFLAGS: 00000246 [ 4393.696354] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 0000000000000000 [ 4393.703480] RDX: ffff8881a1d34bc8 RSI: 0000000000000000 RDI: 0000000000000001 [ 4393.710603] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 4393.717727] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 4393.724852] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 4393.731975] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4393.740053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4393.745789] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4393.752913] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4393.760039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4393.767162] Call Trace: [ 4393.769607] [ 4393.771619] ? watchdog_timer_fn+0x1b4/0x220 [ 4393.775883] ? __hrtimer_run_queues+0x1d3/0x360 [ 4393.780416] ? hrtimer_interrupt+0x104/0x240 [ 4393.784678] ? __local_bh_enable+0x50/0x60 [ 4393.788771] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4393.793995] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4393.798952] [ 4393.801050] [ 4393.803147] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4393.808463] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4393.813178] ? lock_release+0x77/0x260 [ 4393.816921] _raw_spin_unlock+0x1b/0x40 [ 4393.820750] ceph_check_caps+0x26b/0xae0 [ceph] [ 4393.825293] ? lock_acquire+0xcb/0x2b0 [ 4393.829036] ? igrab+0x1d/0x50 [ 4393.832087] ? find_held_lock+0x2d/0x90 [ 4393.835926] ? find_held_lock+0x2d/0x90 [ 4393.839757] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4393.845156] ? lock_release+0xc5/0x260 [ 4393.848902] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4393.854127] delayed_work+0x1d8/0x280 [ceph] [ 4393.858410] ? process_scheduled_works+0x22a/0x510 [ 4393.863199] process_scheduled_works+0x2a4/0x510 [ 4393.867816] worker_thread+0x188/0x340 [ 4393.871564] ? rescuer_thread+0x310/0x310 [ 4393.875567] kthread+0xf5/0x130 [ 4393.878705] ? kthread_complete_and_exit+0x20/0x20 [ 4393.883490] ret_from_fork+0x31/0x40 [ 4393.887069] ? kthread_complete_and_exit+0x20/0x20 [ 4393.891853] ret_from_fork_asm+0x11/0x20 [ 4393.895776] [ 4421.543536] watchdog: BUG: soft lockup - CPU#7 stuck for 2000s! [kworker/7:4:78808] [ 4421.551190] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4421.606641] irq event stamp: 5238464 [ 4421.610210] hardirqs last enabled at (5238463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4421.620291] hardirqs last disabled at (5238464): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4421.629937] softirqs last enabled at (5238360): [] __do_softirq+0x2be/0x371 [ 4421.638544] softirqs last disabled at (5238355): [] irq_exit_rcu+0xb0/0xd0 [ 4421.646976] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4421.656354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4421.663824] Workqueue: events delayed_work [ceph] [ 4421.668548] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4421.672906] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4421.691643] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4421.696861] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4421.703985] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4421.711109] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4421.718233] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4421.725357] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4421.732483] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4421.740558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4421.746297] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4421.753422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4421.760544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4421.767670] Call Trace: [ 4421.770113] [ 4421.772126] ? watchdog_timer_fn+0x1b4/0x220 [ 4421.776396] ? __hrtimer_run_queues+0x1d3/0x360 [ 4421.780923] ? hrtimer_interrupt+0x104/0x240 [ 4421.785186] ? __local_bh_enable+0x50/0x60 [ 4421.789279] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4421.794503] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4421.799459] [ 4421.801557] [ 4421.803653] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4421.808961] ? lock_acquire+0xf0/0x2b0 [ 4421.812711] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4421.817426] ? lock_release+0xc5/0x260 [ 4421.821171] _raw_spin_lock+0x30/0x40 [ 4421.824827] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4421.829456] ceph_check_caps+0x63/0xae0 [ceph] [ 4421.833906] ? __lock_acquire+0xbfa/0x1e20 [ 4421.838005] ? lock_acquire+0xcb/0x2b0 [ 4421.841754] ? igrab+0x1d/0x50 [ 4421.844805] ? find_held_lock+0x2d/0x90 [ 4421.848643] ? find_held_lock+0x2d/0x90 [ 4421.852473] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4421.857875] ? lock_release+0xc5/0x260 [ 4421.861620] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4421.866854] delayed_work+0x1d8/0x280 [ceph] [ 4421.871136] ? process_scheduled_works+0x22a/0x510 [ 4421.875927] process_scheduled_works+0x2a4/0x510 [ 4421.880543] worker_thread+0x188/0x340 [ 4421.884291] ? rescuer_thread+0x310/0x310 [ 4421.888294] kthread+0xf5/0x130 [ 4421.891442] ? kthread_complete_and_exit+0x20/0x20 [ 4421.896233] ret_from_fork+0x31/0x40 [ 4421.899814] ? kthread_complete_and_exit+0x20/0x20 [ 4421.904595] ret_from_fork_asm+0x11/0x20 [ 4421.908520] [ 4449.544042] watchdog: BUG: soft lockup - CPU#7 stuck for 2026s! [kworker/7:4:78808] [ 4449.551696] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4449.607146] irq event stamp: 5300302 [ 4449.610717] hardirqs last enabled at (5300301): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4449.620794] hardirqs last disabled at (5300302): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4449.630434] softirqs last enabled at (5300222): [] __do_softirq+0x2be/0x371 [ 4449.639039] softirqs last disabled at (5300217): [] irq_exit_rcu+0xb0/0xd0 [ 4449.647465] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4449.656841] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4449.664312] Workqueue: events delayed_work [ceph] [ 4449.669036] RIP: 0010:do_raw_spin_unlock+0x42/0xc0 [ 4449.673819] Code: 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 8b 05 4a bf eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 10 <89> 43 08 c6 03 00 5b c3 e8 41 21 45 00 85 c0 74 c3 48 c7 c6 29 e2 [ 4449.692558] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [ 4449.697775] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [ 4449.704897] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [ 4449.712022] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 4449.719148] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 4449.726270] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 4449.733395] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4449.741473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4449.747209] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4449.754335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4449.761458] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4449.768582] Call Trace: [ 4449.771029] [ 4449.773040] ? watchdog_timer_fn+0x1b4/0x220 [ 4449.777310] ? __hrtimer_run_queues+0x1d3/0x360 [ 4449.781838] ? hrtimer_interrupt+0x104/0x240 [ 4449.786107] ? __local_bh_enable+0x50/0x60 [ 4449.790199] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4449.795425] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4449.800380] [ 4449.802477] [ 4449.804575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4449.809884] ? do_raw_spin_unlock+0x42/0xc0 [ 4449.814068] _raw_spin_unlock+0x23/0x40 [ 4449.817899] igrab+0x35/0x50 [ 4449.820775] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4449.825926] delayed_work+0x1d8/0x280 [ceph] [ 4449.830207] ? process_scheduled_works+0x22a/0x510 [ 4449.834997] process_scheduled_works+0x2a4/0x510 [ 4449.839613] worker_thread+0x188/0x340 [ 4449.843360] ? rescuer_thread+0x310/0x310 [ 4449.847364] kthread+0xf5/0x130 [ 4449.850502] ? kthread_complete_and_exit+0x20/0x20 [ 4449.855286] ret_from_fork+0x31/0x40 [ 4449.858865] ? kthread_complete_and_exit+0x20/0x20 [ 4449.863649] ret_from_fork_asm+0x11/0x20 [ 4449.867572] [ 4477.544521] watchdog: BUG: soft lockup - CPU#7 stuck for 2053s! [kworker/7:4:78808] [ 4477.552177] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4477.607626] irq event stamp: 5360860 [ 4477.611198] hardirqs last enabled at (5360859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4477.621276] hardirqs last disabled at (5360860): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4477.630923] softirqs last enabled at (5360824): [] __do_softirq+0x2be/0x371 [ 4477.639529] softirqs last disabled at (5360819): [] irq_exit_rcu+0xb0/0xd0 [ 4477.647964] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4477.657348] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4477.664819] Workqueue: events delayed_work [ceph] [ 4477.669540] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4477.673892] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4477.692630] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4477.697849] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4477.704971] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4477.712096] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4477.719219] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4477.726344] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4477.733468] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4477.741544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4477.747282] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4477.754408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4477.761531] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4477.768656] Call Trace: [ 4477.771101] [ 4477.773112] ? watchdog_timer_fn+0x1b4/0x220 [ 4477.777375] ? __hrtimer_run_queues+0x1d3/0x360 [ 4477.781902] ? hrtimer_interrupt+0x104/0x240 [ 4477.786171] ? __local_bh_enable+0x50/0x60 [ 4477.790264] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4477.795487] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4477.800444] [ 4477.802541] [ 4477.804639] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4477.809949] ? lock_acquire+0xf0/0x2b0 [ 4477.813699] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4477.819108] _raw_spin_lock+0x30/0x40 [ 4477.822771] ? igrab+0x1d/0x50 [ 4477.825822] igrab+0x1d/0x50 [ 4477.828702] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4477.833850] delayed_work+0x1d8/0x280 [ceph] [ 4477.838132] ? process_scheduled_works+0x22a/0x510 [ 4477.842922] process_scheduled_works+0x2a4/0x510 [ 4477.847537] worker_thread+0x188/0x340 [ 4477.851286] ? rescuer_thread+0x310/0x310 [ 4477.855289] kthread+0xf5/0x130 [ 4477.858427] ? kthread_complete_and_exit+0x20/0x20 [ 4477.863212] ret_from_fork+0x31/0x40 [ 4477.866790] ? kthread_complete_and_exit+0x20/0x20 [ 4477.871574] ret_from_fork_asm+0x11/0x20 [ 4477.875496] [ 4505.545013] watchdog: BUG: soft lockup - CPU#7 stuck for 2079s! [kworker/7:4:78808] [ 4505.552666] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4505.608115] irq event stamp: 5421294 [ 4505.611684] hardirqs last enabled at (5421293): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4505.621764] hardirqs last disabled at (5421294): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4505.631401] softirqs last enabled at (5421272): [] __do_softirq+0x2be/0x371 [ 4505.640000] softirqs last disabled at (5421267): [] irq_exit_rcu+0xb0/0xd0 [ 4505.648435] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4505.657821] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4505.665292] Workqueue: events delayed_work [ceph] [ 4505.670014] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4505.674365] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4505.693101] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4505.698319] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4505.705443] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4505.712566] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4505.719691] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4505.726815] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4505.733939] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4505.742018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4505.747755] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4505.754879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4505.762003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4505.769127] Call Trace: [ 4505.771572] [ 4505.773585] ? watchdog_timer_fn+0x1b4/0x220 [ 4505.777856] ? __hrtimer_run_queues+0x1d3/0x360 [ 4505.782391] ? hrtimer_interrupt+0x104/0x240 [ 4505.786660] ? __local_bh_enable+0x50/0x60 [ 4505.790752] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4505.795978] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4505.800934] [ 4505.803032] [ 4505.805130] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4505.810438] ? lock_acquire+0xf0/0x2b0 [ 4505.814188] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4505.819594] ? lock_release+0xc5/0x260 [ 4505.823339] _raw_spin_lock+0x30/0x40 [ 4505.826996] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4505.832396] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4505.837624] delayed_work+0x1d8/0x280 [ceph] [ 4505.841905] ? process_scheduled_works+0x22a/0x510 [ 4505.846695] process_scheduled_works+0x2a4/0x510 [ 4505.851310] worker_thread+0x188/0x340 [ 4505.855061] ? rescuer_thread+0x310/0x310 [ 4505.859072] kthread+0xf5/0x130 [ 4505.862210] ? kthread_complete_and_exit+0x20/0x20 [ 4505.866994] ret_from_fork+0x31/0x40 [ 4505.870564] ? kthread_complete_and_exit+0x20/0x20 [ 4505.875347] ret_from_fork_asm+0x11/0x20 [ 4505.879271] [ 4533.545505] watchdog: BUG: soft lockup - CPU#7 stuck for 2105s! [kworker/7:4:78808] [ 4533.553155] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4533.608605] irq event stamp: 5481736 [ 4533.612174] hardirqs last enabled at (5481735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4533.622253] hardirqs last disabled at (5481736): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4533.631890] softirqs last enabled at (5481722): [] __do_softirq+0x2be/0x371 [ 4533.640489] softirqs last disabled at (5481717): [] irq_exit_rcu+0xb0/0xd0 [ 4533.648922] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4533.658300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4533.665771] Workqueue: events delayed_work [ceph] [ 4533.670494] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4533.674845] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4533.693583] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4533.698800] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4533.705923] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4533.713048] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4533.720172] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4533.727296] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4533.734419] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4533.742497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4533.748236] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4533.755359] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4533.762484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4533.769608] Call Trace: [ 4533.772052] [ 4533.774065] ? watchdog_timer_fn+0x1b4/0x220 [ 4533.778335] ? __hrtimer_run_queues+0x1d3/0x360 [ 4533.782863] ? hrtimer_interrupt+0x104/0x240 [ 4533.787132] ? __local_bh_enable+0x50/0x60 [ 4533.791224] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4533.796450] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4533.801405] [ 4533.803503] [ 4533.805601] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4533.810908] ? lock_acquire+0xf0/0x2b0 [ 4533.814650] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4533.820057] ? lock_release+0xc5/0x260 [ 4533.823803] _raw_spin_lock+0x30/0x40 [ 4533.827460] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4533.832858] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4533.838087] delayed_work+0x1d8/0x280 [ceph] [ 4533.842368] ? process_scheduled_works+0x22a/0x510 [ 4533.847158] process_scheduled_works+0x2a4/0x510 [ 4533.851773] worker_thread+0x188/0x340 [ 4533.855524] ? rescuer_thread+0x310/0x310 [ 4533.859526] kthread+0xf5/0x130 [ 4533.862663] ? kthread_complete_and_exit+0x20/0x20 [ 4533.867447] ret_from_fork+0x31/0x40 [ 4533.871019] ? kthread_complete_and_exit+0x20/0x20 [ 4533.875802] ret_from_fork_asm+0x11/0x20 [ 4533.879735] [ 4561.546014] watchdog: BUG: soft lockup - CPU#7 stuck for 2131s! [kworker/7:4:78808] [ 4561.553669] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4561.609120] irq event stamp: 5542412 [ 4561.612690] hardirqs last enabled at (5542411): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4561.622770] hardirqs last disabled at (5542412): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4561.632416] softirqs last enabled at (5542398): [] __do_softirq+0x2be/0x371 [ 4561.641022] softirqs last disabled at (5542393): [] irq_exit_rcu+0xb0/0xd0 [ 4561.649457] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4561.658841] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4561.666313] Workqueue: events delayed_work [ceph] [ 4561.671036] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4561.675394] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4561.694132] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4561.699350] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4561.706473] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4561.713597] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4561.720720] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4561.727846] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4561.734970] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4561.743046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4561.748785] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4561.755910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4561.763032] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4561.770157] Call Trace: [ 4561.772604] [ 4561.774615] ? watchdog_timer_fn+0x1b4/0x220 [ 4561.778886] ? __hrtimer_run_queues+0x1d3/0x360 [ 4561.783421] ? hrtimer_interrupt+0x104/0x240 [ 4561.787690] ? __local_bh_enable+0x50/0x60 [ 4561.791784] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4561.797007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4561.801964] [ 4561.804061] [ 4561.806159] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4561.811467] ? lock_acquire+0xf0/0x2b0 [ 4561.815218] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4561.819932] ? lock_release+0xc5/0x260 [ 4561.823675] _raw_spin_lock+0x30/0x40 [ 4561.827334] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4561.831952] ceph_check_caps+0x63/0xae0 [ceph] [ 4561.836404] ? __lock_acquire+0xbfa/0x1e20 [ 4561.840502] ? lock_acquire+0xcb/0x2b0 [ 4561.844251] ? igrab+0x1d/0x50 [ 4561.847301] ? find_held_lock+0x2d/0x90 [ 4561.851139] ? find_held_lock+0x2d/0x90 [ 4561.854971] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4561.860371] ? lock_release+0xc5/0x260 [ 4561.864116] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4561.869344] delayed_work+0x1d8/0x280 [ceph] [ 4561.873625] ? process_scheduled_works+0x22a/0x510 [ 4561.878415] process_scheduled_works+0x2a4/0x510 [ 4561.883031] worker_thread+0x188/0x340 [ 4561.886779] ? rescuer_thread+0x310/0x310 [ 4561.890783] kthread+0xf5/0x130 [ 4561.893929] ? kthread_complete_and_exit+0x20/0x20 [ 4561.898713] ret_from_fork+0x31/0x40 [ 4561.902283] ? kthread_complete_and_exit+0x20/0x20 [ 4561.907067] ret_from_fork_asm+0x11/0x20 [ 4561.910989] [ 4589.546489] watchdog: BUG: soft lockup - CPU#7 stuck for 2157s! [kworker/7:4:78808] [ 4589.554142] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4589.609592] irq event stamp: 5603168 [ 4589.613161] hardirqs last enabled at (5603167): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4589.623241] hardirqs last disabled at (5603168): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4589.632879] softirqs last enabled at (5603088): [] __do_softirq+0x2be/0x371 [ 4589.641483] softirqs last disabled at (5603083): [] irq_exit_rcu+0xb0/0xd0 [ 4589.649910] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4589.659287] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4589.666758] Workqueue: events delayed_work [ceph] [ 4589.671479] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4589.675832] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4589.694568] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4589.699786] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4589.706909] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4589.714035] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4589.721159] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4589.728282] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4589.735408] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4589.743484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4589.749222] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4589.756344] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4589.763470] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4589.770593] Call Trace: [ 4589.773042] [ 4589.775052] ? watchdog_timer_fn+0x1b4/0x220 [ 4589.779322] ? __hrtimer_run_queues+0x1d3/0x360 [ 4589.783848] ? hrtimer_interrupt+0x104/0x240 [ 4589.788109] ? __local_bh_enable+0x50/0x60 [ 4589.792202] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4589.797427] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4589.802385] [ 4589.804482] [ 4589.806578] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4589.811885] ? lock_acquire+0xf0/0x2b0 [ 4589.815628] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4589.821034] ? lock_release+0xc5/0x260 [ 4589.824781] _raw_spin_lock+0x30/0x40 [ 4589.828437] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4589.833837] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4589.839064] delayed_work+0x1d8/0x280 [ceph] [ 4589.843346] ? process_scheduled_works+0x22a/0x510 [ 4589.848137] process_scheduled_works+0x2a4/0x510 [ 4589.852752] worker_thread+0x188/0x340 [ 4589.856501] ? rescuer_thread+0x310/0x310 [ 4589.860504] kthread+0xf5/0x130 [ 4589.863642] ? kthread_complete_and_exit+0x20/0x20 [ 4589.868425] ret_from_fork+0x31/0x40 [ 4589.871996] ? kthread_complete_and_exit+0x20/0x20 [ 4589.876780] ret_from_fork_asm+0x11/0x20 [ 4589.880704] [ 4617.546981] watchdog: BUG: soft lockup - CPU#7 stuck for 2183s! [kworker/7:4:78808] [ 4617.554632] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4617.610080] irq event stamp: 5663366 [ 4617.613651] hardirqs last enabled at (5663365): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4617.623730] hardirqs last disabled at (5663366): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4617.633375] softirqs last enabled at (5663352): [] __do_softirq+0x2be/0x371 [ 4617.641982] softirqs last disabled at (5663347): [] irq_exit_rcu+0xb0/0xd0 [ 4617.650417] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4617.659794] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4617.667264] Workqueue: events delayed_work [ceph] [ 4617.671988] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4617.676346] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4617.695083] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4617.700302] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4617.707426] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4617.714549] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4617.721672] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4617.728799] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4617.735921] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4617.743999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4617.749737] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4617.756861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4617.763985] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4617.771109] Call Trace: [ 4617.773554] [ 4617.775567] ? watchdog_timer_fn+0x1b4/0x220 [ 4617.779839] ? __hrtimer_run_queues+0x1d3/0x360 [ 4617.784372] ? hrtimer_interrupt+0x104/0x240 [ 4617.788632] ? __local_bh_enable+0x50/0x60 [ 4617.792727] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4617.797951] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4617.802906] [ 4617.805006] [ 4617.807103] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4617.812410] ? lock_acquire+0xf0/0x2b0 [ 4617.816162] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4617.821570] _raw_spin_lock+0x30/0x40 [ 4617.825234] ? igrab+0x1d/0x50 [ 4617.828284] igrab+0x1d/0x50 [ 4617.831163] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4617.836304] delayed_work+0x1d8/0x280 [ceph] [ 4617.840584] ? process_scheduled_works+0x22a/0x510 [ 4617.845375] process_scheduled_works+0x2a4/0x510 [ 4617.849990] worker_thread+0x188/0x340 [ 4617.853739] ? rescuer_thread+0x310/0x310 [ 4617.857742] kthread+0xf5/0x130 [ 4617.860880] ? kthread_complete_and_exit+0x20/0x20 [ 4617.865665] ret_from_fork+0x31/0x40 [ 4617.869244] ? kthread_complete_and_exit+0x20/0x20 [ 4617.874028] ret_from_fork_asm+0x11/0x20 [ 4617.877951] [ 4645.547474] watchdog: BUG: soft lockup - CPU#7 stuck for 2209s! [kworker/7:4:78808] [ 4645.555128] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4645.610577] irq event stamp: 5723834 [ 4645.614146] hardirqs last enabled at (5723833): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4645.624228] hardirqs last disabled at (5723834): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4645.633872] softirqs last enabled at (5723812): [] __do_softirq+0x2be/0x371 [ 4645.642471] softirqs last disabled at (5723807): [] irq_exit_rcu+0xb0/0xd0 [ 4645.650905] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4645.660282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4645.667753] Workqueue: events delayed_work [ceph] [ 4645.672478] RIP: 0010:lock_release+0x14e/0x260 [ 4645.676923] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 4645.695659] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [ 4645.700878] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [ 4645.708000] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [ 4645.715126] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 4645.722248] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 4645.729374] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [ 4645.736496] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4645.744576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4645.750313] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4645.757437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4645.764560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4645.771685] Call Trace: [ 4645.774131] [ 4645.776142] ? watchdog_timer_fn+0x1b4/0x220 [ 4645.780412] ? __hrtimer_run_queues+0x1d3/0x360 [ 4645.784940] ? hrtimer_interrupt+0x104/0x240 [ 4645.789210] ? __local_bh_enable+0x50/0x60 [ 4645.793302] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4645.798525] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4645.803484] [ 4645.805582] [ 4645.807678] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4645.812982] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4645.817699] ? lock_release+0x14e/0x260 [ 4645.821528] ? lock_release+0xc5/0x260 [ 4645.825274] _raw_spin_unlock+0x1b/0x40 [ 4645.829113] ceph_check_caps+0x26b/0xae0 [ceph] [ 4645.833653] ? lock_acquire+0xcb/0x2b0 [ 4645.837396] ? igrab+0x1d/0x50 [ 4645.840447] ? find_held_lock+0x2d/0x90 [ 4645.844287] ? find_held_lock+0x2d/0x90 [ 4645.848117] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4645.853517] ? lock_release+0xc5/0x260 [ 4645.857263] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4645.862491] delayed_work+0x1d8/0x280 [ceph] [ 4645.866772] ? process_scheduled_works+0x22a/0x510 [ 4645.871562] process_scheduled_works+0x2a4/0x510 [ 4645.876178] worker_thread+0x188/0x340 [ 4645.879927] ? rescuer_thread+0x310/0x310 [ 4645.883937] kthread+0xf5/0x130 [ 4645.887075] ? kthread_complete_and_exit+0x20/0x20 [ 4645.891861] ret_from_fork+0x31/0x40 [ 4645.895439] ? kthread_complete_and_exit+0x20/0x20 [ 4645.900223] ret_from_fork_asm+0x11/0x20 [ 4645.904155] [ 4673.547981] watchdog: BUG: soft lockup - CPU#7 stuck for 2235s! [kworker/7:4:78808] [ 4673.555632] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4673.611083] irq event stamp: 5785084 [ 4673.614654] hardirqs last enabled at (5785083): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4673.624733] hardirqs last disabled at (5785084): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4673.634372] softirqs last enabled at (5784988): [] __do_softirq+0x2be/0x371 [ 4673.642977] softirqs last disabled at (5784983): [] irq_exit_rcu+0xb0/0xd0 [ 4673.651411] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4673.660788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4673.668259] Workqueue: events delayed_work [ceph] [ 4673.672982] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4673.677332] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4673.696070] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4673.701287] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4673.708411] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4673.715537] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4673.722660] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4673.729783] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4673.736909] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4673.744985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4673.750724] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4673.757848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4673.764971] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4673.772096] Call Trace: [ 4673.774544] [ 4673.776563] ? watchdog_timer_fn+0x1b4/0x220 [ 4673.780834] ? __hrtimer_run_queues+0x1d3/0x360 [ 4673.785367] ? hrtimer_interrupt+0x104/0x240 [ 4673.789628] ? __local_bh_enable+0x50/0x60 [ 4673.793721] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4673.798946] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4673.803904] [ 4673.805999] [ 4673.808099] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4673.813406] ? lock_acquire+0xf0/0x2b0 [ 4673.817158] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4673.822566] _raw_spin_lock+0x30/0x40 [ 4673.826229] ? igrab+0x1d/0x50 [ 4673.829280] igrab+0x1d/0x50 [ 4673.832159] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4673.837309] delayed_work+0x1d8/0x280 [ceph] [ 4673.841589] ? process_scheduled_works+0x22a/0x510 [ 4673.846378] process_scheduled_works+0x2a4/0x510 [ 4673.850993] worker_thread+0x188/0x340 [ 4673.854743] ? rescuer_thread+0x310/0x310 [ 4673.858747] kthread+0xf5/0x130 [ 4673.861895] ? kthread_complete_and_exit+0x20/0x20 [ 4673.866687] ret_from_fork+0x31/0x40 [ 4673.870265] ? kthread_complete_and_exit+0x20/0x20 [ 4673.875050] ret_from_fork_asm+0x11/0x20 [ 4673.878980] [ 4701.548459] watchdog: BUG: soft lockup - CPU#7 stuck for 2261s! [kworker/7:4:78808] [ 4701.556115] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4701.611564] irq event stamp: 5846522 [ 4701.615133] hardirqs last enabled at (5846521): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4701.625213] hardirqs last disabled at (5846522): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4701.634852] softirqs last enabled at (5846500): [] __do_softirq+0x2be/0x371 [ 4701.643457] softirqs last disabled at (5846495): [] irq_exit_rcu+0xb0/0xd0 [ 4701.651891] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4701.661267] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4701.668740] Workqueue: events delayed_work [ceph] [ 4701.673463] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4701.677814] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4701.696550] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4701.701767] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4701.708892] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4701.716015] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4701.723140] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4701.730264] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4701.737389] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4701.745465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4701.751202] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4701.758328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4701.765451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4701.772576] Call Trace: [ 4701.775023] [ 4701.777035] ? watchdog_timer_fn+0x1b4/0x220 [ 4701.781304] ? __hrtimer_run_queues+0x1d3/0x360 [ 4701.785830] ? hrtimer_interrupt+0x104/0x240 [ 4701.790092] ? __local_bh_enable+0x50/0x60 [ 4701.794183] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4701.799408] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4701.804366] [ 4701.806463] [ 4701.808561] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4701.813869] ? lock_acquire+0xf0/0x2b0 [ 4701.817619] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4701.823027] _raw_spin_lock+0x30/0x40 [ 4701.826684] ? igrab+0x1d/0x50 [ 4701.829734] igrab+0x1d/0x50 [ 4701.832612] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4701.837753] delayed_work+0x1d8/0x280 [ceph] [ 4701.842034] ? process_scheduled_works+0x22a/0x510 [ 4701.846824] process_scheduled_works+0x2a4/0x510 [ 4701.851441] worker_thread+0x188/0x340 [ 4701.855190] ? rescuer_thread+0x310/0x310 [ 4701.859192] kthread+0xf5/0x130 [ 4701.862331] ? kthread_complete_and_exit+0x20/0x20 [ 4701.867116] ret_from_fork+0x31/0x40 [ 4701.870695] ? kthread_complete_and_exit+0x20/0x20 [ 4701.875485] ret_from_fork_asm+0x11/0x20 [ 4701.879410] [ 4729.548951] watchdog: BUG: soft lockup - CPU#7 stuck for 2287s! [kworker/7:4:78808] [ 4729.556602] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4729.612043] irq event stamp: 5907224 [ 4729.615615] hardirqs last enabled at (5907223): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4729.625686] hardirqs last disabled at (5907224): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4729.635324] softirqs last enabled at (5907202): [] __do_softirq+0x2be/0x371 [ 4729.643919] softirqs last disabled at (5907197): [] irq_exit_rcu+0xb0/0xd0 [ 4729.652344] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4729.661721] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4729.669193] Workqueue: events delayed_work [ceph] [ 4729.673908] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4729.678259] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4729.696995] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4729.702214] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4729.709338] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4729.716463] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4729.723587] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4729.730710] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4729.737835] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4729.745911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4729.751648] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4729.758774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4729.765896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4729.773021] Call Trace: [ 4729.775468] [ 4729.777477] ? watchdog_timer_fn+0x1b4/0x220 [ 4729.781742] ? __hrtimer_run_queues+0x1d3/0x360 [ 4729.786266] ? hrtimer_interrupt+0x104/0x240 [ 4729.790528] ? __local_bh_enable+0x50/0x60 [ 4729.794623] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4729.799846] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4729.804802] [ 4729.806899] [ 4729.808999] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4729.814305] ? lock_acquire+0xf0/0x2b0 [ 4729.818055] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4729.823464] ? lock_release+0xc5/0x260 [ 4729.827208] _raw_spin_lock+0x30/0x40 [ 4729.830864] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4729.836262] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4729.841492] delayed_work+0x1d8/0x280 [ceph] [ 4729.845772] ? process_scheduled_works+0x22a/0x510 [ 4729.850554] process_scheduled_works+0x2a4/0x510 [ 4729.855169] worker_thread+0x188/0x340 [ 4729.858920] ? rescuer_thread+0x310/0x310 [ 4729.862922] kthread+0xf5/0x130 [ 4729.866062] ? kthread_complete_and_exit+0x20/0x20 [ 4729.870844] ret_from_fork+0x31/0x40 [ 4729.874416] ? kthread_complete_and_exit+0x20/0x20 [ 4729.879199] ret_from_fork_asm+0x11/0x20 [ 4729.883122] [ 4757.549443] watchdog: BUG: soft lockup - CPU#7 stuck for 2313s! [kworker/7:4:78808] [ 4757.557115] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4757.612567] irq event stamp: 5968774 [ 4757.616137] hardirqs last enabled at (5968773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4757.626216] hardirqs last disabled at (5968774): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4757.635856] softirqs last enabled at (5968758): [] __do_softirq+0x2be/0x371 [ 4757.644461] softirqs last disabled at (5968753): [] irq_exit_rcu+0xb0/0xd0 [ 4757.652896] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4757.662271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4757.669744] Workqueue: events delayed_work [ceph] [ 4757.674465] RIP: 0010:do_raw_spin_unlock+0x42/0xc0 [ 4757.679251] Code: 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 8b 05 4a bf eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 10 <89> 43 08 c6 03 00 5b c3 e8 41 21 45 00 85 c0 74 c3 48 c7 c6 29 e2 [ 4757.697988] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000246 [ 4757.703206] RAX: ffffffffffffffff RBX: ffff8881a1d34920 RCX: ffffc90002117b84 [ 4757.710329] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff8881a1d34920 [ 4757.717453] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 4757.724576] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 4757.731700] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 4757.738838] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4757.746921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4757.752657] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4757.759781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4757.766905] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4757.774030] Call Trace: [ 4757.776478] [ 4757.778486] ? watchdog_timer_fn+0x1b4/0x220 [ 4757.782750] ? __hrtimer_run_queues+0x1d3/0x360 [ 4757.787285] ? hrtimer_interrupt+0x104/0x240 [ 4757.791546] ? __local_bh_enable+0x50/0x60 [ 4757.795640] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4757.800864] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4757.805838] [ 4757.807935] [ 4757.810033] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4757.815341] ? do_raw_spin_unlock+0x42/0xc0 [ 4757.819523] _raw_spin_unlock+0x23/0x40 [ 4757.823356] ceph_check_caps+0x26b/0xae0 [ceph] [ 4757.827905] ? lock_acquire+0xcb/0x2b0 [ 4757.831648] ? igrab+0x1d/0x50 [ 4757.834699] ? find_held_lock+0x2d/0x90 [ 4757.838537] ? find_held_lock+0x2d/0x90 [ 4757.842368] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4757.847767] ? lock_release+0xc5/0x260 [ 4757.851513] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4757.856741] delayed_work+0x1d8/0x280 [ceph] [ 4757.861023] ? process_scheduled_works+0x22a/0x510 [ 4757.865811] process_scheduled_works+0x2a4/0x510 [ 4757.870428] worker_thread+0x188/0x340 [ 4757.874177] ? rescuer_thread+0x310/0x310 [ 4757.878187] kthread+0xf5/0x130 [ 4757.881328] ? kthread_complete_and_exit+0x20/0x20 [ 4757.886120] ret_from_fork+0x31/0x40 [ 4757.889698] ? kthread_complete_and_exit+0x20/0x20 [ 4757.894483] ret_from_fork_asm+0x11/0x20 [ 4757.898415] [ 4785.549934] watchdog: BUG: soft lockup - CPU#7 stuck for 2339s! [kworker/7:4:78808] [ 4785.557589] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4785.613038] irq event stamp: 6029952 [ 4785.616609] hardirqs last enabled at (6029951): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4785.626689] hardirqs last disabled at (6029952): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4785.636326] softirqs last enabled at (6029840): [] __do_softirq+0x2be/0x371 [ 4785.644923] softirqs last disabled at (6029835): [] irq_exit_rcu+0xb0/0xd0 [ 4785.653349] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4785.662726] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4785.670196] Workqueue: events delayed_work [ceph] [ 4785.674922] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4785.679281] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4785.698017] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4785.703234] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4785.710358] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4785.717483] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4785.724608] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4785.731731] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4785.738856] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4785.746933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4785.752671] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4785.759793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4785.766917] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4785.774043] Call Trace: [ 4785.776490] [ 4785.778500] ? watchdog_timer_fn+0x1b4/0x220 [ 4785.782771] ? __hrtimer_run_queues+0x1d3/0x360 [ 4785.787296] ? hrtimer_interrupt+0x104/0x240 [ 4785.791558] ? __local_bh_enable+0x50/0x60 [ 4785.795652] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4785.800876] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4785.805831] [ 4785.807929] [ 4785.810027] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4785.815336] ? lock_acquire+0xf0/0x2b0 [ 4785.819085] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4785.824492] ? lock_release+0xc5/0x260 [ 4785.828237] _raw_spin_lock+0x30/0x40 [ 4785.831893] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4785.837294] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4785.842523] delayed_work+0x1d8/0x280 [ceph] [ 4785.846803] ? process_scheduled_works+0x22a/0x510 [ 4785.851593] process_scheduled_works+0x2a4/0x510 [ 4785.856208] worker_thread+0x188/0x340 [ 4785.859959] ? rescuer_thread+0x310/0x310 [ 4785.863960] kthread+0xf5/0x130 [ 4785.867099] ? kthread_complete_and_exit+0x20/0x20 [ 4785.871883] ret_from_fork+0x31/0x40 [ 4785.875453] ? kthread_complete_and_exit+0x20/0x20 [ 4785.880238] ret_from_fork_asm+0x11/0x20 [ 4785.884161] [ 4813.550426] watchdog: BUG: soft lockup - CPU#7 stuck for 2365s! [kworker/7:4:78808] [ 4813.558079] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4813.613529] irq event stamp: 6091916 [ 4813.617099] hardirqs last enabled at (6091915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4813.627177] hardirqs last disabled at (6091916): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4813.636816] softirqs last enabled at (6091902): [] __do_softirq+0x2be/0x371 [ 4813.645423] softirqs last disabled at (6091897): [] irq_exit_rcu+0xb0/0xd0 [ 4813.653857] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4813.663232] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4813.670703] Workqueue: events delayed_work [ceph] [ 4813.675428] RIP: 0010:ceph_check_caps+0x30/0xae0 [ceph] [ 4813.680661] Code: 44 00 00 41 57 41 56 41 55 41 54 55 89 f5 53 48 89 fb 48 81 ec b0 01 00 00 65 48 8b 04 25 28 00 00 00 48 89 84 24 a8 01 00 00 <31> c0 48 8b 47 28 48 8b 80 10 06 00 00 48 8b 48 40 48 8b 40 20 48 [ 4813.699398] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000282 [ 4813.704616] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34478 RCX: ffffc90002117d6c [ 4813.711741] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8881a1d34478 [ 4813.718863] RBP: 0000000000000000 R08: 0000000000005c53 R09: 0000000000000003 [ 4813.725987] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 4813.733112] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 4813.740236] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4813.748313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4813.754051] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4813.761175] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4813.768300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4813.775423] Call Trace: [ 4813.777870] [ 4813.779880] ? watchdog_timer_fn+0x1b4/0x220 [ 4813.784145] ? __hrtimer_run_queues+0x1d3/0x360 [ 4813.788669] ? hrtimer_interrupt+0x104/0x240 [ 4813.792930] ? __local_bh_enable+0x50/0x60 [ 4813.797023] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4813.802249] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4813.807206] [ 4813.809301] [ 4813.811402] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4813.816707] ? ceph_check_caps+0x30/0xae0 [ceph] [ 4813.821338] ? __lock_acquire+0xbfa/0x1e20 [ 4813.825435] ? lock_acquire+0xcb/0x2b0 [ 4813.829186] ? igrab+0x1d/0x50 [ 4813.832235] ? find_held_lock+0x2d/0x90 [ 4813.836066] ? find_held_lock+0x2d/0x90 [ 4813.839896] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4813.845295] ? lock_release+0xc5/0x260 [ 4813.849043] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4813.854279] delayed_work+0x1d8/0x280 [ceph] [ 4813.858560] ? process_scheduled_works+0x22a/0x510 [ 4813.863350] process_scheduled_works+0x2a4/0x510 [ 4813.867965] worker_thread+0x188/0x340 [ 4813.871714] ? rescuer_thread+0x310/0x310 [ 4813.875716] kthread+0xf5/0x130 [ 4813.878854] ? kthread_complete_and_exit+0x20/0x20 [ 4813.883638] ret_from_fork+0x31/0x40 [ 4813.887208] ? kthread_complete_and_exit+0x20/0x20 [ 4813.891994] ret_from_fork_asm+0x11/0x20 [ 4813.895926] [ 4841.550918] watchdog: BUG: soft lockup - CPU#7 stuck for 2392s! [kworker/7:4:78808] [ 4841.558568] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4841.614018] irq event stamp: 6153314 [ 4841.617588] hardirqs last enabled at (6153313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4841.627667] hardirqs last disabled at (6153314): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4841.637313] softirqs last enabled at (6153196): [] __do_softirq+0x2be/0x371 [ 4841.645920] softirqs last disabled at (6153191): [] irq_exit_rcu+0xb0/0xd0 [ 4841.654354] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4841.663730] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4841.671201] Workqueue: events delayed_work [ceph] [ 4841.675924] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4841.680275] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4841.699013] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4841.704229] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4841.711353] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4841.718477] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4841.725603] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4841.732725] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4841.739850] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4841.747927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4841.753666] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4841.760789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4841.767912] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4841.775038] Call Trace: [ 4841.777482] [ 4841.779495] ? watchdog_timer_fn+0x1b4/0x220 [ 4841.783765] ? __hrtimer_run_queues+0x1d3/0x360 [ 4841.788291] ? hrtimer_interrupt+0x104/0x240 [ 4841.792553] ? __local_bh_enable+0x50/0x60 [ 4841.796645] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4841.801870] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4841.806829] [ 4841.808926] [ 4841.811022] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4841.816331] ? lock_acquire+0xf0/0x2b0 [ 4841.820080] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4841.825489] ? lock_release+0xc5/0x260 [ 4841.829234] _raw_spin_lock+0x30/0x40 [ 4841.832899] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4841.838298] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4841.843527] delayed_work+0x1d8/0x280 [ceph] [ 4841.847807] ? process_scheduled_works+0x22a/0x510 [ 4841.852598] process_scheduled_works+0x2a4/0x510 [ 4841.857213] worker_thread+0x188/0x340 [ 4841.860961] ? rescuer_thread+0x310/0x310 [ 4841.864964] kthread+0xf5/0x130 [ 4841.868103] ? kthread_complete_and_exit+0x20/0x20 [ 4841.872888] ret_from_fork+0x31/0x40 [ 4841.876468] ? kthread_complete_and_exit+0x20/0x20 [ 4841.881258] ret_from_fork_asm+0x11/0x20 [ 4841.885181] [ 4869.551412] watchdog: BUG: soft lockup - CPU#7 stuck for 2418s! [kworker/7:4:78808] [ 4869.559065] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4869.614514] irq event stamp: 6216898 [ 4869.618085] hardirqs last enabled at (6216897): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4869.628164] hardirqs last disabled at (6216898): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4869.637803] softirqs last enabled at (6216884): [] __do_softirq+0x2be/0x371 [ 4869.646407] softirqs last disabled at (6216879): [] irq_exit_rcu+0xb0/0xd0 [ 4869.654834] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4869.664210] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4869.671681] Workqueue: events delayed_work [ceph] [ 4869.676406] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4869.680764] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4869.699502] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 4869.704719] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 4869.711843] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 4869.718965] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 4869.726091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 4869.733216] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4869.740338] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4869.748416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4869.754155] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4869.761278] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4869.768401] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4869.775527] Call Trace: [ 4869.777972] [ 4869.779985] ? watchdog_timer_fn+0x1b4/0x220 [ 4869.784254] ? __hrtimer_run_queues+0x1d3/0x360 [ 4869.788781] ? hrtimer_interrupt+0x104/0x240 [ 4869.793042] ? __local_bh_enable+0x50/0x60 [ 4869.797134] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4869.802359] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4869.807317] [ 4869.809414] [ 4869.811511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4869.816818] ? lock_acquire+0xf0/0x2b0 [ 4869.820562] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4869.825969] _raw_spin_lock+0x30/0x40 [ 4869.829635] ? igrab+0x1d/0x50 [ 4869.832685] igrab+0x1d/0x50 [ 4869.835563] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 4869.840704] delayed_work+0x1d8/0x280 [ceph] [ 4869.844986] ? process_scheduled_works+0x22a/0x510 [ 4869.849776] process_scheduled_works+0x2a4/0x510 [ 4869.854391] worker_thread+0x188/0x340 [ 4869.858139] ? rescuer_thread+0x310/0x310 [ 4869.862143] kthread+0xf5/0x130 [ 4869.865282] ? kthread_complete_and_exit+0x20/0x20 [ 4869.870065] ret_from_fork+0x31/0x40 [ 4869.873645] ? kthread_complete_and_exit+0x20/0x20 [ 4869.878429] ret_from_fork_asm+0x11/0x20 [ 4869.882359] [ 4897.551902] watchdog: BUG: soft lockup - CPU#7 stuck for 2444s! [kworker/7:4:78808] [ 4897.559555] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4897.615003] irq event stamp: 6279234 [ 4897.618572] hardirqs last enabled at (6279233): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4897.628654] hardirqs last disabled at (6279234): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4897.638299] softirqs last enabled at (6279212): [] __do_softirq+0x2be/0x371 [ 4897.646896] softirqs last disabled at (6279207): [] irq_exit_rcu+0xb0/0xd0 [ 4897.655331] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4897.664706] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4897.672179] Workqueue: events delayed_work [ceph] [ 4897.676904] RIP: 0010:__ceph_caps_issued+0x55/0x110 [ceph] [ 4897.682396] Code: 4c 8b 70 20 48 85 f6 74 06 c7 06 00 00 00 00 49 8d bd a8 06 00 00 e8 8a fc 04 e1 49 89 c7 48 85 c0 74 32 49 8d 5f f8 48 89 df c6 dc ff ff 85 c0 74 12 66 90 45 0b 67 38 48 85 ed 74 07 41 8b [ 4897.701135] RSP: 0018:ffffc90002117b98 EFLAGS: 00000282 [ 4897.706351] RAX: ffff88810da9e008 RBX: ffff88810da9e000 RCX: 0000000000000000 [ 4897.713474] RDX: 0000000000000000 RSI: ffffc90002117c64 RDI: ffff88810da9e000 [ 4897.720599] RBP: ffffc90002117c64 R08: 0000000000000000 R09: 0000000000000000 [ 4897.727724] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 4897.734847] R13: ffff8881a1d34478 R14: ffff888100ee2000 R15: ffff88810da9e008 [ 4897.741971] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4897.750050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4897.755788] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4897.762912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4897.770035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4897.777159] Call Trace: [ 4897.779607] [ 4897.781615] ? watchdog_timer_fn+0x1b4/0x220 [ 4897.785878] ? __hrtimer_run_queues+0x1d3/0x360 [ 4897.790405] ? hrtimer_interrupt+0x104/0x240 [ 4897.794667] ? __local_bh_enable+0x50/0x60 [ 4897.798760] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4897.803983] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4897.808939] [ 4897.811037] [ 4897.813135] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4897.818444] ? __ceph_caps_issued+0x55/0x110 [ceph] [ 4897.823328] ? __ceph_caps_issued+0x46/0x110 [ceph] [ 4897.828207] ceph_check_caps+0x154/0xae0 [ceph] [ 4897.832749] ? lock_acquire+0xcb/0x2b0 [ 4897.836493] ? igrab+0x1d/0x50 [ 4897.839543] ? find_held_lock+0x2d/0x90 [ 4897.843375] ? find_held_lock+0x2d/0x90 [ 4897.847205] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4897.852605] ? lock_release+0xc5/0x260 [ 4897.856350] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4897.861578] delayed_work+0x1d8/0x280 [ceph] [ 4897.865858] ? process_scheduled_works+0x22a/0x510 [ 4897.870649] process_scheduled_works+0x2a4/0x510 [ 4897.875263] worker_thread+0x188/0x340 [ 4897.879014] ? rescuer_thread+0x310/0x310 [ 4897.883017] kthread+0xf5/0x130 [ 4897.886155] ? kthread_complete_and_exit+0x20/0x20 [ 4897.890937] ret_from_fork+0x31/0x40 [ 4897.894508] ? kthread_complete_and_exit+0x20/0x20 [ 4897.899292] ret_from_fork_asm+0x11/0x20 [ 4897.903215] [ 4925.552408] watchdog: BUG: soft lockup - CPU#7 stuck for 2470s! [kworker/7:4:78808] [ 4925.560061] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4925.615510] irq event stamp: 6341392 [ 4925.619081] hardirqs last enabled at (6341391): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4925.629158] hardirqs last disabled at (6341392): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4925.638797] softirqs last enabled at (6341286): [] __do_softirq+0x2be/0x371 [ 4925.647395] softirqs last disabled at (6341281): [] irq_exit_rcu+0xb0/0xd0 [ 4925.655829] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4925.665203] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4925.672678] Workqueue: events delayed_work [ceph] [ 4925.677400] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4925.681751] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4925.700488] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 4925.705706] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 4925.712829] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 4925.719952] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 4925.727077] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 4925.734201] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4925.741325] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4925.749402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4925.755141] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4925.762264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4925.769388] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4925.776513] Call Trace: [ 4925.778960] [ 4925.780971] ? watchdog_timer_fn+0x1b4/0x220 [ 4925.785241] ? __hrtimer_run_queues+0x1d3/0x360 [ 4925.789769] ? hrtimer_interrupt+0x104/0x240 [ 4925.794038] ? __local_bh_enable+0x50/0x60 [ 4925.798131] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4925.803356] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4925.808311] [ 4925.810408] [ 4925.812508] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4925.817821] ? lock_acquire+0xf0/0x2b0 [ 4925.821565] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4925.826278] ? lock_release+0xc5/0x260 [ 4925.830023] _raw_spin_lock+0x30/0x40 [ 4925.833680] ? ceph_check_caps+0x63/0xae0 [ceph] [ 4925.838299] ceph_check_caps+0x63/0xae0 [ceph] [ 4925.842751] ? __lock_acquire+0xbfa/0x1e20 [ 4925.846847] ? lock_acquire+0xcb/0x2b0 [ 4925.850589] ? igrab+0x1d/0x50 [ 4925.853639] ? find_held_lock+0x2d/0x90 [ 4925.857479] ? find_held_lock+0x2d/0x90 [ 4925.861311] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4925.866718] ? lock_release+0xc5/0x260 [ 4925.870462] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4925.875690] delayed_work+0x1d8/0x280 [ceph] [ 4925.879972] ? process_scheduled_works+0x22a/0x510 [ 4925.884762] process_scheduled_works+0x2a4/0x510 [ 4925.889376] worker_thread+0x188/0x340 [ 4925.893128] ? rescuer_thread+0x310/0x310 [ 4925.897137] kthread+0xf5/0x130 [ 4925.900277] ? kthread_complete_and_exit+0x20/0x20 [ 4925.905061] ret_from_fork+0x31/0x40 [ 4925.908641] ? kthread_complete_and_exit+0x20/0x20 [ 4925.913431] ret_from_fork_asm+0x11/0x20 [ 4925.917354] [ 4953.552887] watchdog: BUG: soft lockup - CPU#7 stuck for 2496s! [kworker/7:4:78808] [ 4953.560541] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4953.615989] irq event stamp: 6402900 [ 4953.619560] hardirqs last enabled at (6402899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4953.629640] hardirqs last disabled at (6402900): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4953.639285] softirqs last enabled at (6402820): [] __do_softirq+0x2be/0x371 [ 4953.647883] softirqs last disabled at (6402815): [] irq_exit_rcu+0xb0/0xd0 [ 4953.656317] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4953.665694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4953.673164] Workqueue: events delayed_work [ceph] [ 4953.677889] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 4953.682240] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 4953.700976] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 4953.706194] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 4953.713317] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 4953.720441] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 4953.727565] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 4953.734691] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4953.741815] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4953.749891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4953.755628] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4953.762753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4953.769879] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4953.777001] Call Trace: [ 4953.779449] [ 4953.781459] ? watchdog_timer_fn+0x1b4/0x220 [ 4953.785729] ? __hrtimer_run_queues+0x1d3/0x360 [ 4953.790256] ? hrtimer_interrupt+0x104/0x240 [ 4953.794517] ? __local_bh_enable+0x50/0x60 [ 4953.798610] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4953.803834] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4953.808793] [ 4953.810888] [ 4953.812987] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4953.818294] ? lock_acquire+0xf0/0x2b0 [ 4953.822044] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4953.827453] ? lock_release+0xc5/0x260 [ 4953.831198] _raw_spin_lock+0x30/0x40 [ 4953.834862] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4953.840261] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 4953.845489] delayed_work+0x1d8/0x280 [ceph] [ 4953.849771] ? process_scheduled_works+0x22a/0x510 [ 4953.854562] process_scheduled_works+0x2a4/0x510 [ 4953.859177] worker_thread+0x188/0x340 [ 4953.862927] ? rescuer_thread+0x310/0x310 [ 4953.866928] kthread+0xf5/0x130 [ 4953.870068] ? kthread_complete_and_exit+0x20/0x20 [ 4953.874850] ret_from_fork+0x31/0x40 [ 4953.878422] ? kthread_complete_and_exit+0x20/0x20 [ 4953.883206] ret_from_fork_asm+0x11/0x20 [ 4953.887138] [ 4981.553386] watchdog: BUG: soft lockup - CPU#7 stuck for 2522s! [kworker/7:4:78808] [ 4981.561038] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 4981.616487] irq event stamp: 6463950 [ 4981.620056] hardirqs last enabled at (6463949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4981.630137] hardirqs last disabled at (6463950): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 4981.639775] softirqs last enabled at (6463936): [] __do_softirq+0x2be/0x371 [ 4981.648381] softirqs last disabled at (6463931): [] irq_exit_rcu+0xb0/0xd0 [ 4981.656814] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 4981.666191] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 4981.673664] Workqueue: events delayed_work [ceph] [ 4981.678386] RIP: 0010:lock_release+0x0/0x260 [ 4981.682651] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [ 4981.701386] RSP: 0018:ffffc90002117bb8 EFLAGS: 00000282 [ 4981.706604] RAX: 0000000000000000 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 4981.713729] RDX: ffff8881a1d34bc8 RSI: ffffffffa0b3407b RDI: ffff8881a1d34938 [ 4981.720852] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 4981.727977] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 4981.735101] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 4981.742224] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 4981.750302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4981.756040] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 4981.763163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4981.770287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4981.777413] Call Trace: [ 4981.779858] [ 4981.781869] ? watchdog_timer_fn+0x1b4/0x220 [ 4981.786133] ? __hrtimer_run_queues+0x1d3/0x360 [ 4981.790667] ? hrtimer_interrupt+0x104/0x240 [ 4981.794929] ? __local_bh_enable+0x50/0x60 [ 4981.799021] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 4981.804246] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 4981.809202] [ 4981.811301] [ 4981.813397] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 4981.818704] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 4981.823415] ? lock_downgrade+0x250/0x250 [ 4981.827421] _raw_spin_unlock+0x1b/0x40 [ 4981.831254] ceph_check_caps+0x26b/0xae0 [ceph] [ 4981.835794] ? lock_acquire+0xcb/0x2b0 [ 4981.839537] ? igrab+0x1d/0x50 [ 4981.842588] ? find_held_lock+0x2d/0x90 [ 4981.846428] ? find_held_lock+0x2d/0x90 [ 4981.850257] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 4981.855658] ? lock_release+0xc5/0x260 [ 4981.859402] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 4981.864634] delayed_work+0x1d8/0x280 [ceph] [ 4981.868910] ? process_scheduled_works+0x22a/0x510 [ 4981.873701] process_scheduled_works+0x2a4/0x510 [ 4981.878318] worker_thread+0x188/0x340 [ 4981.882066] ? rescuer_thread+0x310/0x310 [ 4981.886069] kthread+0xf5/0x130 [ 4981.889208] ? kthread_complete_and_exit+0x20/0x20 [ 4981.893990] ret_from_fork+0x31/0x40 [ 4981.897571] ? kthread_complete_and_exit+0x20/0x20 [ 4981.902355] ret_from_fork_asm+0x11/0x20 [ 4981.906277] [ 5009.553870] watchdog: BUG: soft lockup - CPU#7 stuck for 2548s! [kworker/7:4:78808] [ 5009.561545] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5009.616992] irq event stamp: 6525952 [ 5009.620566] hardirqs last enabled at (6525951): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5009.630644] hardirqs last disabled at (6525952): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5009.640280] softirqs last enabled at (6525848): [] __do_softirq+0x2be/0x371 [ 5009.648877] softirqs last disabled at (6525843): [] irq_exit_rcu+0xb0/0xd0 [ 5009.657303] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5009.666679] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5009.674153] Workqueue: events delayed_work [ceph] [ 5009.678875] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5009.683227] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5009.701962] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5009.707180] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5009.714304] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5009.721430] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5009.728554] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5009.735677] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5009.742801] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5009.750880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5009.756617] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5009.763739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5009.770863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5009.777988] Call Trace: [ 5009.780434] [ 5009.782445] ? watchdog_timer_fn+0x1b4/0x220 [ 5009.786717] ? __hrtimer_run_queues+0x1d3/0x360 [ 5009.791251] ? hrtimer_interrupt+0x104/0x240 [ 5009.795513] ? __local_bh_enable+0x50/0x60 [ 5009.799606] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5009.804831] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5009.809786] [ 5009.811883] [ 5009.813982] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5009.819289] ? lock_acquire+0xf0/0x2b0 [ 5009.823032] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5009.827745] ? lock_release+0xc5/0x260 [ 5009.831490] _raw_spin_lock+0x30/0x40 [ 5009.835156] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5009.839775] ceph_check_caps+0x63/0xae0 [ceph] [ 5009.844226] ? __lock_acquire+0xbfa/0x1e20 [ 5009.848324] ? lock_acquire+0xcb/0x2b0 [ 5009.852073] ? igrab+0x1d/0x50 [ 5009.855124] ? find_held_lock+0x2d/0x90 [ 5009.858963] ? find_held_lock+0x2d/0x90 [ 5009.862794] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5009.868194] ? lock_release+0xc5/0x260 [ 5009.871940] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5009.877175] delayed_work+0x1d8/0x280 [ceph] [ 5009.881455] ? process_scheduled_works+0x22a/0x510 [ 5009.886245] process_scheduled_works+0x2a4/0x510 [ 5009.890860] worker_thread+0x188/0x340 [ 5009.894611] ? rescuer_thread+0x310/0x310 [ 5009.898614] kthread+0xf5/0x130 [ 5009.901752] ? kthread_complete_and_exit+0x20/0x20 [ 5009.906535] ret_from_fork+0x31/0x40 [ 5009.910115] ? kthread_complete_and_exit+0x20/0x20 [ 5009.914899] ret_from_fork_asm+0x11/0x20 [ 5009.918831] [ 5037.554368] watchdog: BUG: soft lockup - CPU#7 stuck for 2574s! [kworker/7:4:78808] [ 5037.562024] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5037.617473] irq event stamp: 6587084 [ 5037.621043] hardirqs last enabled at (6587083): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5037.631123] hardirqs last disabled at (6587084): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5037.640762] softirqs last enabled at (6587004): [] __do_softirq+0x2be/0x371 [ 5037.649366] softirqs last disabled at (6586999): [] irq_exit_rcu+0xb0/0xd0 [ 5037.657794] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5037.667168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5037.674641] Workqueue: events delayed_work [ceph] [ 5037.679363] RIP: 0010:lock_release+0x6e/0x260 [ 5037.683715] Code: d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 <85> f6 0f 85 d8 00 00 00 9c 58 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 [ 5037.702452] RSP: 0018:ffffc90002117d60 EFLAGS: 00000246 [ 5037.707669] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [ 5037.714794] RDX: 0000000000000003 RSI: 0000000000000000 RDI: 0000000000000001 [ 5037.721917] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 5037.729040] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 5037.736165] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5037.743290] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5037.751368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5037.757103] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5037.764230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5037.771352] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5037.778478] Call Trace: [ 5037.780923] [ 5037.782934] ? watchdog_timer_fn+0x1b4/0x220 [ 5037.787206] ? __hrtimer_run_queues+0x1d3/0x360 [ 5037.791739] ? hrtimer_interrupt+0x104/0x240 [ 5037.796001] ? __local_bh_enable+0x50/0x60 [ 5037.800093] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5037.805318] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5037.810275] [ 5037.812372] [ 5037.814472] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5037.819776] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5037.825186] ? lock_release+0x6e/0x260 [ 5037.828939] _raw_spin_unlock+0x1b/0x40 [ 5037.832775] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5037.838005] delayed_work+0x1d8/0x280 [ceph] [ 5037.842285] ? process_scheduled_works+0x22a/0x510 [ 5037.847074] process_scheduled_works+0x2a4/0x510 [ 5037.851690] worker_thread+0x188/0x340 [ 5037.855441] ? rescuer_thread+0x310/0x310 [ 5037.859452] kthread+0xf5/0x130 [ 5037.862600] ? kthread_complete_and_exit+0x20/0x20 [ 5037.867392] ret_from_fork+0x31/0x40 [ 5037.870970] ? kthread_complete_and_exit+0x20/0x20 [ 5037.875754] ret_from_fork_asm+0x11/0x20 [ 5037.879678] [ 5065.554853] watchdog: BUG: soft lockup - CPU#7 stuck for 2600s! [kworker/7:4:78808] [ 5065.562504] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5065.617954] irq event stamp: 6648922 [ 5065.621524] hardirqs last enabled at (6648921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5065.631605] hardirqs last disabled at (6648922): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5065.641251] softirqs last enabled at (6648890): [] __do_softirq+0x2be/0x371 [ 5065.649856] softirqs last disabled at (6648885): [] irq_exit_rcu+0xb0/0xd0 [ 5065.658291] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5065.667666] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5065.675138] Workqueue: events delayed_work [ceph] [ 5065.679863] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5065.684211] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5065.702949] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5065.708168] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5065.715289] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5065.722415] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5065.729540] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5065.736663] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5065.743787] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5065.751865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5065.757602] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5065.764728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5065.771850] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5065.778974] Call Trace: [ 5065.781421] [ 5065.783432] ? watchdog_timer_fn+0x1b4/0x220 [ 5065.787695] ? __hrtimer_run_queues+0x1d3/0x360 [ 5065.792230] ? hrtimer_interrupt+0x104/0x240 [ 5065.796499] ? __local_bh_enable+0x50/0x60 [ 5065.800593] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5065.805817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5065.810773] [ 5065.812870] [ 5065.814969] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5065.820276] ? lock_acquire+0xf0/0x2b0 [ 5065.824026] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5065.828741] ? lock_release+0xc5/0x260 [ 5065.832484] _raw_spin_lock+0x30/0x40 [ 5065.836142] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5065.840761] ceph_check_caps+0x63/0xae0 [ceph] [ 5065.845213] ? __lock_acquire+0xbfa/0x1e20 [ 5065.849309] ? lock_acquire+0xcb/0x2b0 [ 5065.853050] ? igrab+0x1d/0x50 [ 5065.856100] ? find_held_lock+0x2d/0x90 [ 5065.859931] ? find_held_lock+0x2d/0x90 [ 5065.863763] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5065.869161] ? lock_release+0xc5/0x260 [ 5065.872909] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5065.878145] delayed_work+0x1d8/0x280 [ceph] [ 5065.882424] ? process_scheduled_works+0x22a/0x510 [ 5065.887214] process_scheduled_works+0x2a4/0x510 [ 5065.891830] worker_thread+0x188/0x340 [ 5065.895579] ? rescuer_thread+0x310/0x310 [ 5065.899581] kthread+0xf5/0x130 [ 5065.902721] ? kthread_complete_and_exit+0x20/0x20 [ 5065.907506] ret_from_fork+0x31/0x40 [ 5065.911084] ? kthread_complete_and_exit+0x20/0x20 [ 5065.915868] ret_from_fork_asm+0x11/0x20 [ 5065.919800] [ 5093.555345] watchdog: BUG: soft lockup - CPU#7 stuck for 2626s! [kworker/7:4:78808] [ 5093.562993] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5093.618442] irq event stamp: 6709624 [ 5093.622013] hardirqs last enabled at (6709623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5093.632093] hardirqs last disabled at (6709624): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5093.641730] softirqs last enabled at (6709544): [] __do_softirq+0x2be/0x371 [ 5093.650327] softirqs last disabled at (6709539): [] irq_exit_rcu+0xb0/0xd0 [ 5093.658762] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5093.668138] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5093.675610] Workqueue: events delayed_work [ceph] [ 5093.680333] RIP: 0010:do_raw_spin_unlock+0x9/0xc0 [ 5093.685030] Code: eb 7e 89 47 08 65 48 8b 04 25 c0 e5 02 00 48 89 47 10 b8 01 00 00 00 c3 31 c0 c3 66 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 <53> 81 7f 04 ad 4e ad de 48 89 fb 75 34 8b 03 85 c0 74 48 65 48 8b [ 5093.703769] RSP: 0018:ffffc90002117bb8 EFLAGS: 00000296 [ 5093.708985] RAX: 0000000000000000 RBX: ffff8881a1d34920 RCX: ffffc90002117b84 [ 5093.716110] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff8881a1d34920 [ 5093.723234] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 5093.730358] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 5093.737481] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 5093.744606] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5093.752682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5093.758420] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5093.765546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5093.772668] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5093.779793] Call Trace: [ 5093.782240] [ 5093.784251] ? watchdog_timer_fn+0x1b4/0x220 [ 5093.788522] ? __hrtimer_run_queues+0x1d3/0x360 [ 5093.793056] ? hrtimer_interrupt+0x104/0x240 [ 5093.797317] ? __local_bh_enable+0x50/0x60 [ 5093.801411] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5093.806634] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5093.811592] [ 5093.813689] [ 5093.815787] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5093.821094] ? do_raw_spin_unlock+0x9/0xc0 [ 5093.825192] _raw_spin_unlock+0x23/0x40 [ 5093.829032] ceph_check_caps+0x26b/0xae0 [ceph] [ 5093.833580] ? lock_acquire+0xcb/0x2b0 [ 5093.837322] ? igrab+0x1d/0x50 [ 5093.840375] ? find_held_lock+0x2d/0x90 [ 5093.844214] ? find_held_lock+0x2d/0x90 [ 5093.848045] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5093.853454] ? lock_release+0xc5/0x260 [ 5093.857200] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5093.862435] delayed_work+0x1d8/0x280 [ceph] [ 5093.866715] ? process_scheduled_works+0x22a/0x510 [ 5093.871505] process_scheduled_works+0x2a4/0x510 [ 5093.876122] worker_thread+0x188/0x340 [ 5093.879872] ? rescuer_thread+0x310/0x310 [ 5093.883881] kthread+0xf5/0x130 [ 5093.887020] ? kthread_complete_and_exit+0x20/0x20 [ 5093.891803] ret_from_fork+0x31/0x40 [ 5093.895383] ? kthread_complete_and_exit+0x20/0x20 [ 5093.900166] ret_from_fork_asm+0x11/0x20 [ 5093.904090] [ 5121.555836] watchdog: BUG: soft lockup - CPU#7 stuck for 2652s! [kworker/7:4:78808] [ 5121.563492] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5121.618941] irq event stamp: 6771478 [ 5121.622511] hardirqs last enabled at (6771477): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5121.632589] hardirqs last disabled at (6771478): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5121.642227] softirqs last enabled at (6771366): [] __do_softirq+0x2be/0x371 [ 5121.650825] softirqs last disabled at (6771361): [] irq_exit_rcu+0xb0/0xd0 [ 5121.659259] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5121.668635] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5121.676108] Workqueue: events delayed_work [ceph] [ 5121.680830] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5121.685181] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5121.703918] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5121.709135] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5121.716261] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5121.723383] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5121.730508] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5121.737632] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5121.744757] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5121.752834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5121.758571] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5121.765697] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5121.772820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5121.779943] Call Trace: [ 5121.782391] [ 5121.784401] ? watchdog_timer_fn+0x1b4/0x220 [ 5121.788673] ? __hrtimer_run_queues+0x1d3/0x360 [ 5121.793208] ? hrtimer_interrupt+0x104/0x240 [ 5121.797478] ? __local_bh_enable+0x50/0x60 [ 5121.801572] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5121.806795] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5121.811751] [ 5121.813848] [ 5121.815945] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5121.821252] ? lock_acquire+0xf0/0x2b0 [ 5121.824995] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5121.830403] ? lock_release+0xc5/0x260 [ 5121.834148] _raw_spin_lock+0x30/0x40 [ 5121.837813] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5121.843212] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5121.848440] delayed_work+0x1d8/0x280 [ceph] [ 5121.852720] ? process_scheduled_works+0x22a/0x510 [ 5121.857512] process_scheduled_works+0x2a4/0x510 [ 5121.862127] worker_thread+0x188/0x340 [ 5121.865876] ? rescuer_thread+0x310/0x310 [ 5121.869879] kthread+0xf5/0x130 [ 5121.873018] ? kthread_complete_and_exit+0x20/0x20 [ 5121.877803] ret_from_fork+0x31/0x40 [ 5121.881383] ? kthread_complete_and_exit+0x20/0x20 [ 5121.886165] ret_from_fork_asm+0x11/0x20 [ 5121.890087] [ 5149.556325] watchdog: BUG: soft lockup - CPU#7 stuck for 2678s! [kworker/7:4:78808] [ 5149.563997] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5149.619446] irq event stamp: 6834376 [ 5149.623018] hardirqs last enabled at (6834375): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5149.633098] hardirqs last disabled at (6834376): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5149.642743] softirqs last enabled at (6834362): [] __do_softirq+0x2be/0x371 [ 5149.651348] softirqs last disabled at (6834357): [] irq_exit_rcu+0xb0/0xd0 [ 5149.659775] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5149.669150] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5149.676622] Workqueue: events delayed_work [ceph] [ 5149.681346] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5149.685697] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5149.704434] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 5149.709651] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 5149.716774] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 5149.723900] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 5149.731023] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 5149.738146] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5149.745271] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5149.753349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5149.759087] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5149.766210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5149.773333] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5149.780458] Call Trace: [ 5149.782906] [ 5149.784916] ? watchdog_timer_fn+0x1b4/0x220 [ 5149.789188] ? __hrtimer_run_queues+0x1d3/0x360 [ 5149.793721] ? hrtimer_interrupt+0x104/0x240 [ 5149.797984] ? __local_bh_enable+0x50/0x60 [ 5149.802075] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5149.807299] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5149.812257] [ 5149.814355] [ 5149.816452] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5149.821759] ? lock_acquire+0xf0/0x2b0 [ 5149.825504] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5149.830909] _raw_spin_lock+0x30/0x40 [ 5149.834568] ? igrab+0x1d/0x50 [ 5149.837625] igrab+0x1d/0x50 [ 5149.840503] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5149.845646] delayed_work+0x1d8/0x280 [ceph] [ 5149.849927] ? process_scheduled_works+0x22a/0x510 [ 5149.854716] process_scheduled_works+0x2a4/0x510 [ 5149.859332] worker_thread+0x188/0x340 [ 5149.863082] ? rescuer_thread+0x310/0x310 [ 5149.867084] kthread+0xf5/0x130 [ 5149.870231] ? kthread_complete_and_exit+0x20/0x20 [ 5149.875016] ret_from_fork+0x31/0x40 [ 5149.878593] ? kthread_complete_and_exit+0x20/0x20 [ 5149.883379] ret_from_fork_asm+0x11/0x20 [ 5149.887310] [ 5177.556816] watchdog: BUG: soft lockup - CPU#7 stuck for 2704s! [kworker/7:4:78808] [ 5177.564485] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5177.619935] irq event stamp: 6896658 [ 5177.623507] hardirqs last enabled at (6896657): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5177.633586] hardirqs last disabled at (6896658): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5177.643222] softirqs last enabled at (6896636): [] __do_softirq+0x2be/0x371 [ 5177.651820] softirqs last disabled at (6896631): [] irq_exit_rcu+0xb0/0xd0 [ 5177.660253] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5177.669631] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5177.677102] Workqueue: events delayed_work [ceph] [ 5177.681826] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 5177.686437] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 5177.705175] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [ 5177.710400] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 5177.717525] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f867488 [ 5177.724650] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 5177.731781] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 5177.738906] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5177.746029] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5177.754107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5177.759843] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5177.766969] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5177.774092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5177.781216] Call Trace: [ 5177.783662] [ 5177.785675] ? watchdog_timer_fn+0x1b4/0x220 [ 5177.789945] ? __hrtimer_run_queues+0x1d3/0x360 [ 5177.794480] ? hrtimer_interrupt+0x104/0x240 [ 5177.798748] ? __local_bh_enable+0x50/0x60 [ 5177.802841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5177.808068] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5177.813032] [ 5177.815129] [ 5177.817226] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5177.822535] ? do_raw_spin_lock+0x42/0xc0 [ 5177.826547] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5177.831960] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5177.837190] delayed_work+0x1d8/0x280 [ceph] [ 5177.841501] ? process_scheduled_works+0x22a/0x510 [ 5177.846286] process_scheduled_works+0x2a4/0x510 [ 5177.850902] worker_thread+0x188/0x340 [ 5177.854652] ? rescuer_thread+0x310/0x310 [ 5177.858653] kthread+0xf5/0x130 [ 5177.861793] ? kthread_complete_and_exit+0x20/0x20 [ 5177.866577] ret_from_fork+0x31/0x40 [ 5177.870158] ? kthread_complete_and_exit+0x20/0x20 [ 5177.874939] ret_from_fork_asm+0x11/0x20 [ 5177.878863] [ 5205.557305] watchdog: BUG: soft lockup - CPU#7 stuck for 2731s! [kworker/7:4:78808] [ 5205.564958] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5205.620408] irq event stamp: 6960346 [ 5205.623977] hardirqs last enabled at (6960345): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5205.634057] hardirqs last disabled at (6960346): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5205.643694] softirqs last enabled at (6960306): [] __do_softirq+0x2be/0x371 [ 5205.652292] softirqs last disabled at (6960301): [] irq_exit_rcu+0xb0/0xd0 [ 5205.660725] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5205.670102] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5205.677575] Workqueue: events delayed_work [ceph] [ 5205.682298] RIP: 0010:iput+0x12/0x220 [ 5205.685955] Code: 9c 7e 00 48 8d 7c 24 08 e8 bb fd ff ff e9 81 fe ff ff e8 01 cc 7d 00 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 dd 01 00 00 <41> 55 41 54 55 53 f6 87 d8 00 00 00 40 0f 85 c8 01 00 00 48 89 fb [ 5205.704691] RSP: 0018:ffffc90002117db0 EFLAGS: 00000286 [ 5205.709910] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 5205.717033] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff8881a1d34478 [ 5205.724158] RBP: ffff88817f867000 R08: 0000000000000000 R09: 0000000000000000 [ 5205.731280] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000ea60 [ 5205.738406] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5205.745529] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5205.753606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5205.759344] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5205.766468] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5205.773593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5205.780718] Call Trace: [ 5205.783163] [ 5205.785174] ? watchdog_timer_fn+0x1b4/0x220 [ 5205.789438] ? __hrtimer_run_queues+0x1d3/0x360 [ 5205.793971] ? hrtimer_interrupt+0x104/0x240 [ 5205.798233] ? __local_bh_enable+0x50/0x60 [ 5205.802325] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5205.807549] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5205.812507] [ 5205.814605] [ 5205.816701] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5205.822009] ? iput+0x12/0x220 [ 5205.825059] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 5205.830293] delayed_work+0x1d8/0x280 [ceph] [ 5205.834574] ? process_scheduled_works+0x22a/0x510 [ 5205.839366] process_scheduled_works+0x2a4/0x510 [ 5205.843981] worker_thread+0x188/0x340 [ 5205.847730] ? rescuer_thread+0x310/0x310 [ 5205.851734] kthread+0xf5/0x130 [ 5205.854881] ? kthread_complete_and_exit+0x20/0x20 [ 5205.859664] ret_from_fork+0x31/0x40 [ 5205.863245] ? kthread_complete_and_exit+0x20/0x20 [ 5205.868027] ret_from_fork_asm+0x11/0x20 [ 5205.871960] [ 5233.557805] watchdog: BUG: soft lockup - CPU#7 stuck for 2757s! [kworker/7:4:78808] [ 5233.565455] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5233.620904] irq event stamp: 7022158 [ 5233.624476] hardirqs last enabled at (7022157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5233.634553] hardirqs last disabled at (7022158): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5233.644193] softirqs last enabled at (7022104): [] __do_softirq+0x2be/0x371 [ 5233.652797] softirqs last disabled at (7022099): [] irq_exit_rcu+0xb0/0xd0 [ 5233.661224] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5233.670599] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5233.678072] Workqueue: events delayed_work [ceph] [ 5233.682797] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5233.687155] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5233.705890] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5233.711109] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5233.718233] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5233.725357] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5233.732480] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5233.739606] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5233.746728] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5233.754807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5233.760543] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5233.767668] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5233.774794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5233.781917] Call Trace: [ 5233.784363] [ 5233.786374] ? watchdog_timer_fn+0x1b4/0x220 [ 5233.790646] ? __hrtimer_run_queues+0x1d3/0x360 [ 5233.795181] ? hrtimer_interrupt+0x104/0x240 [ 5233.799451] ? __local_bh_enable+0x50/0x60 [ 5233.803543] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5233.808766] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5233.813725] [ 5233.815820] [ 5233.817919] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5233.823228] ? lock_acquire+0xf0/0x2b0 [ 5233.826976] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5233.831691] ? lock_release+0xc5/0x260 [ 5233.835437] _raw_spin_lock+0x30/0x40 [ 5233.839103] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5233.843720] ceph_check_caps+0x63/0xae0 [ceph] [ 5233.848172] ? __lock_acquire+0xbfa/0x1e20 [ 5233.852270] ? lock_acquire+0xcb/0x2b0 [ 5233.856017] ? igrab+0x1d/0x50 [ 5233.859068] ? find_held_lock+0x2d/0x90 [ 5233.862899] ? find_held_lock+0x2d/0x90 [ 5233.866731] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5233.872131] ? lock_release+0xc5/0x260 [ 5233.875876] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5233.881104] delayed_work+0x1d8/0x280 [ceph] [ 5233.885383] ? process_scheduled_works+0x22a/0x510 [ 5233.890175] process_scheduled_works+0x2a4/0x510 [ 5233.894791] worker_thread+0x188/0x340 [ 5233.898539] ? rescuer_thread+0x310/0x310 [ 5233.902542] kthread+0xf5/0x130 [ 5233.905681] ? kthread_complete_and_exit+0x20/0x20 [ 5233.910464] ret_from_fork+0x31/0x40 [ 5233.914035] ? kthread_complete_and_exit+0x20/0x20 [ 5233.918819] ret_from_fork_asm+0x11/0x20 [ 5233.922751] [ 5261.558288] watchdog: BUG: soft lockup - CPU#7 stuck for 2783s! [kworker/7:4:78808] [ 5261.565942] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5261.621393] irq event stamp: 7083330 [ 5261.624965] hardirqs last enabled at (7083329): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5261.635043] hardirqs last disabled at (7083330): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5261.644681] softirqs last enabled at (7083250): [] __do_softirq+0x2be/0x371 [ 5261.653279] softirqs last disabled at (7083245): [] irq_exit_rcu+0xb0/0xd0 [ 5261.661712] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5261.671089] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5261.678560] Workqueue: events delayed_work [ceph] [ 5261.683285] RIP: 0010:igrab+0x2d/0x50 [ 5261.686941] Code: fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 <48> 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 [ 5261.705679] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [ 5261.710895] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 5261.718020] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 5261.725143] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 5261.732269] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 5261.739392] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5261.746516] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5261.754594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5261.760330] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5261.767456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5261.774578] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5261.781703] Call Trace: [ 5261.784149] [ 5261.786162] ? watchdog_timer_fn+0x1b4/0x220 [ 5261.790432] ? __hrtimer_run_queues+0x1d3/0x360 [ 5261.794959] ? hrtimer_interrupt+0x104/0x240 [ 5261.799227] ? __local_bh_enable+0x50/0x60 [ 5261.803321] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5261.808545] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5261.813502] [ 5261.815600] [ 5261.817697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5261.823006] ? igrab+0x2d/0x50 [ 5261.826061] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5261.831212] delayed_work+0x1d8/0x280 [ceph] [ 5261.835494] ? process_scheduled_works+0x22a/0x510 [ 5261.840284] process_scheduled_works+0x2a4/0x510 [ 5261.844899] worker_thread+0x188/0x340 [ 5261.848648] ? rescuer_thread+0x310/0x310 [ 5261.852649] kthread+0xf5/0x130 [ 5261.855788] ? kthread_complete_and_exit+0x20/0x20 [ 5261.860574] ret_from_fork+0x31/0x40 [ 5261.864152] ? kthread_complete_and_exit+0x20/0x20 [ 5261.868937] ret_from_fork_asm+0x11/0x20 [ 5261.872867] [ 5289.558776] watchdog: BUG: soft lockup - CPU#7 stuck for 2809s! [kworker/7:4:78808] [ 5289.566424] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5289.621873] irq event stamp: 7144372 [ 5289.625445] hardirqs last enabled at (7144371): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5289.635524] hardirqs last disabled at (7144372): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5289.645162] softirqs last enabled at (7144318): [] __do_softirq+0x2be/0x371 [ 5289.653768] softirqs last disabled at (7144313): [] irq_exit_rcu+0xb0/0xd0 [ 5289.662202] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5289.671577] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5289.679050] Workqueue: events delayed_work [ceph] [ 5289.683773] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5289.688125] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5289.706860] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5289.712078] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5289.719201] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5289.726326] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5289.733450] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5289.740574] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5289.747698] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5289.755775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5289.761513] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5289.768637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5289.775761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5289.782887] Call Trace: [ 5289.785332] [ 5289.787343] ? watchdog_timer_fn+0x1b4/0x220 [ 5289.791615] ? __hrtimer_run_queues+0x1d3/0x360 [ 5289.796148] ? hrtimer_interrupt+0x104/0x240 [ 5289.800411] ? __local_bh_enable+0x50/0x60 [ 5289.804503] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5289.809727] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5289.814684] [ 5289.816782] [ 5289.818881] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5289.824187] ? lock_acquire+0xf0/0x2b0 [ 5289.827938] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5289.832651] ? lock_release+0xc5/0x260 [ 5289.836396] _raw_spin_lock+0x30/0x40 [ 5289.840063] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5289.844682] ceph_check_caps+0x63/0xae0 [ceph] [ 5289.849132] ? __lock_acquire+0xbfa/0x1e20 [ 5289.853229] ? lock_acquire+0xcb/0x2b0 [ 5289.856978] ? igrab+0x1d/0x50 [ 5289.860031] ? find_held_lock+0x2d/0x90 [ 5289.863870] ? find_held_lock+0x2d/0x90 [ 5289.867708] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5289.873108] ? lock_release+0xc5/0x260 [ 5289.876853] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5289.882082] delayed_work+0x1d8/0x280 [ceph] [ 5289.886363] ? process_scheduled_works+0x22a/0x510 [ 5289.891152] process_scheduled_works+0x2a4/0x510 [ 5289.895767] worker_thread+0x188/0x340 [ 5289.899518] ? rescuer_thread+0x310/0x310 [ 5289.903521] kthread+0xf5/0x130 [ 5289.906660] ? kthread_complete_and_exit+0x20/0x20 [ 5289.911450] ret_from_fork+0x31/0x40 [ 5289.915030] ? kthread_complete_and_exit+0x20/0x20 [ 5289.919815] ret_from_fork_asm+0x11/0x20 [ 5289.923744] [ 5317.559267] watchdog: BUG: soft lockup - CPU#7 stuck for 2835s! [kworker/7:4:78808] [ 5317.566921] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5317.622371] irq event stamp: 7206466 [ 5317.625942] hardirqs last enabled at (7206465): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5317.636022] hardirqs last disabled at (7206466): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5317.645668] softirqs last enabled at (7206386): [] __do_softirq+0x2be/0x371 [ 5317.654273] softirqs last disabled at (7206381): [] irq_exit_rcu+0xb0/0xd0 [ 5317.662699] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5317.672076] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5317.679547] Workqueue: events delayed_work [ceph] [ 5317.684271] RIP: 0010:lock_acquire+0x86/0x2b0 [ 5317.688623] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [ 5317.707359] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [ 5317.712575] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [ 5317.719701] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [ 5317.726823] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 5317.733947] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [ 5317.741071] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5317.748197] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5317.756273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5317.762010] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5317.769135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5317.776260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5317.783383] Call Trace: [ 5317.785831] [ 5317.787842] ? watchdog_timer_fn+0x1b4/0x220 [ 5317.792112] ? __hrtimer_run_queues+0x1d3/0x360 [ 5317.796639] ? hrtimer_interrupt+0x104/0x240 [ 5317.800908] ? __local_bh_enable+0x50/0x60 [ 5317.805002] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5317.810226] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5317.815182] [ 5317.817279] [ 5317.819378] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5317.824685] ? lock_acquire+0x86/0x2b0 [ 5317.828437] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5317.833844] _raw_spin_lock+0x30/0x40 [ 5317.837510] ? igrab+0x1d/0x50 [ 5317.840567] igrab+0x1d/0x50 [ 5317.843448] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5317.848595] delayed_work+0x1d8/0x280 [ceph] [ 5317.852876] ? process_scheduled_works+0x22a/0x510 [ 5317.857666] process_scheduled_works+0x2a4/0x510 [ 5317.862284] worker_thread+0x188/0x340 [ 5317.866033] ? rescuer_thread+0x310/0x310 [ 5317.870035] kthread+0xf5/0x130 [ 5317.873183] ? kthread_complete_and_exit+0x20/0x20 [ 5317.877975] ret_from_fork+0x31/0x40 [ 5317.881553] ? kthread_complete_and_exit+0x20/0x20 [ 5317.886338] ret_from_fork_asm+0x11/0x20 [ 5317.890269] [ 5345.559756] watchdog: BUG: soft lockup - CPU#7 stuck for 2861s! [kworker/7:4:78808] [ 5345.567429] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5345.622877] irq event stamp: 7268940 [ 5345.626449] hardirqs last enabled at (7268939): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5345.636527] hardirqs last disabled at (7268940): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5345.646164] softirqs last enabled at (7268918): [] __do_softirq+0x2be/0x371 [ 5345.654762] softirqs last disabled at (7268913): [] irq_exit_rcu+0xb0/0xd0 [ 5345.663188] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5345.672563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5345.680037] Workqueue: events delayed_work [ceph] [ 5345.684761] RIP: 0010:_atomic_dec_and_lock+0x14/0x40 [ 5345.689726] Code: 63 c9 48 83 f9 07 76 ba b8 08 00 00 00 eb d2 44 89 d0 5b c3 66 90 f3 0f 1e fa 55 53 8b 07 83 f8 01 74 0e 8d 50 ff f0 0f b1 17 <75> f2 31 c0 5b 5d c3 48 89 fb 48 89 f7 48 89 f5 e8 07 2b 04 00 f0 [ 5345.708462] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [ 5345.713680] RAX: 0000000000000002 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 5345.720805] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [ 5345.727929] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [ 5345.735053] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [ 5345.742177] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5345.749299] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5345.757379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5345.763116] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5345.770240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5345.777363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5345.784488] Call Trace: [ 5345.786933] [ 5345.788945] ? watchdog_timer_fn+0x1b4/0x220 [ 5345.793208] ? __hrtimer_run_queues+0x1d3/0x360 [ 5345.797741] ? hrtimer_interrupt+0x104/0x240 [ 5345.802004] ? __local_bh_enable+0x50/0x60 [ 5345.806098] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5345.811321] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5345.816278] [ 5345.818375] [ 5345.820472] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5345.825781] ? _atomic_dec_and_lock+0x14/0x40 [ 5345.830139] iput+0x50/0x220 [ 5345.833025] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 5345.838259] delayed_work+0x1d8/0x280 [ceph] [ 5345.842542] ? process_scheduled_works+0x22a/0x510 [ 5345.847330] process_scheduled_works+0x2a4/0x510 [ 5345.851948] worker_thread+0x188/0x340 [ 5345.855697] ? rescuer_thread+0x310/0x310 [ 5345.859699] kthread+0xf5/0x130 [ 5345.862837] ? kthread_complete_and_exit+0x20/0x20 [ 5345.867621] ret_from_fork+0x31/0x40 [ 5345.871199] ? kthread_complete_and_exit+0x20/0x20 [ 5345.875984] ret_from_fork_asm+0x11/0x20 [ 5345.879907] [ 5373.560263] watchdog: BUG: soft lockup - CPU#7 stuck for 2887s! [kworker/7:4:78808] [ 5373.567916] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5373.623366] irq event stamp: 7330778 [ 5373.626938] hardirqs last enabled at (7330777): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5373.637017] hardirqs last disabled at (7330778): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5373.646663] softirqs last enabled at (7330734): [] __do_softirq+0x2be/0x371 [ 5373.655269] softirqs last disabled at (7330729): [] irq_exit_rcu+0xb0/0xd0 [ 5373.663696] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5373.673079] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5373.680550] Workqueue: events delayed_work [ceph] [ 5373.685274] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5373.689625] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5373.708361] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5373.713580] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5373.720703] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5373.727828] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5373.734951] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5373.742076] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5373.749199] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5373.757277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5373.763016] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5373.770140] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5373.777264] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5373.784387] Call Trace: [ 5373.786835] [ 5373.788844] ? watchdog_timer_fn+0x1b4/0x220 [ 5373.793108] ? __hrtimer_run_queues+0x1d3/0x360 [ 5373.797643] ? hrtimer_interrupt+0x104/0x240 [ 5373.801910] ? __local_bh_enable+0x50/0x60 [ 5373.806004] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5373.811229] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5373.816185] [ 5373.818284] [ 5373.820382] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5373.825690] ? lock_acquire+0xf0/0x2b0 [ 5373.829440] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5373.834152] ? lock_release+0xc5/0x260 [ 5373.837898] _raw_spin_lock+0x30/0x40 [ 5373.841554] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5373.846173] ceph_check_caps+0x63/0xae0 [ceph] [ 5373.850625] ? __lock_acquire+0xbfa/0x1e20 [ 5373.854724] ? lock_acquire+0xcb/0x2b0 [ 5373.858471] ? igrab+0x1d/0x50 [ 5373.861522] ? find_held_lock+0x2d/0x90 [ 5373.865354] ? find_held_lock+0x2d/0x90 [ 5373.869183] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5373.874584] ? lock_release+0xc5/0x260 [ 5373.878331] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5373.883565] delayed_work+0x1d8/0x280 [ceph] [ 5373.887846] ? process_scheduled_works+0x22a/0x510 [ 5373.892635] process_scheduled_works+0x2a4/0x510 [ 5373.897253] worker_thread+0x188/0x340 [ 5373.901000] ? rescuer_thread+0x310/0x310 [ 5373.905004] kthread+0xf5/0x130 [ 5373.908143] ? kthread_complete_and_exit+0x20/0x20 [ 5373.912926] ret_from_fork+0x31/0x40 [ 5373.916506] ? kthread_complete_and_exit+0x20/0x20 [ 5373.921289] ret_from_fork_asm+0x11/0x20 [ 5373.925221] [ 5401.560736] watchdog: BUG: soft lockup - CPU#7 stuck for 2913s! [kworker/7:4:78808] [ 5401.568406] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5401.623857] irq event stamp: 7393200 [ 5401.627427] hardirqs last enabled at (7393199): [] asm_common_interrupt+0x26/0x40 [ 5401.636552] hardirqs last disabled at (7393200): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5401.646199] softirqs last enabled at (7393198): [] __do_softirq+0x2be/0x371 [ 5401.654805] softirqs last disabled at (7393191): [] irq_exit_rcu+0xb0/0xd0 [ 5401.663238] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5401.672616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5401.680088] Workqueue: events delayed_work [ceph] [ 5401.684810] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5401.689161] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5401.707896] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5401.713116] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5401.720240] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5401.727363] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5401.734486] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5401.741612] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5401.748735] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5401.756814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5401.762551] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5401.769674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5401.776798] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5401.783923] Call Trace: [ 5401.786369] [ 5401.788380] ? watchdog_timer_fn+0x1b4/0x220 [ 5401.792651] ? __hrtimer_run_queues+0x1d3/0x360 [ 5401.797178] ? hrtimer_interrupt+0x104/0x240 [ 5401.801446] ? __local_bh_enable+0x50/0x60 [ 5401.805540] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5401.810765] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5401.815723] [ 5401.817818] [ 5401.819918] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5401.825225] ? lock_acquire+0xf0/0x2b0 [ 5401.828974] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5401.833688] ? lock_release+0xc5/0x260 [ 5401.837434] _raw_spin_lock+0x30/0x40 [ 5401.841101] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5401.845728] ceph_check_caps+0x63/0xae0 [ceph] [ 5401.850186] ? __lock_acquire+0xbfa/0x1e20 [ 5401.854285] ? lock_acquire+0xcb/0x2b0 [ 5401.858033] ? igrab+0x1d/0x50 [ 5401.861085] ? find_held_lock+0x2d/0x90 [ 5401.864925] ? find_held_lock+0x2d/0x90 [ 5401.868755] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5401.874164] ? lock_release+0xc5/0x260 [ 5401.877908] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5401.883145] delayed_work+0x1d8/0x280 [ceph] [ 5401.887424] ? process_scheduled_works+0x22a/0x510 [ 5401.892214] process_scheduled_works+0x2a4/0x510 [ 5401.896831] worker_thread+0x188/0x340 [ 5401.900580] ? rescuer_thread+0x310/0x310 [ 5401.904584] kthread+0xf5/0x130 [ 5401.907721] ? kthread_complete_and_exit+0x20/0x20 [ 5401.912504] ret_from_fork+0x31/0x40 [ 5401.916074] ? kthread_complete_and_exit+0x20/0x20 [ 5401.920858] ret_from_fork_asm+0x11/0x20 [ 5401.924783] [ 5423.321574] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5429.561235] watchdog: BUG: soft lockup - CPU#7 stuck for 2939s! [kworker/7:4:78808] [ 5429.568888] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5429.624337] irq event stamp: 7454528 [ 5429.627907] hardirqs last enabled at (7454527): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5429.637985] hardirqs last disabled at (7454528): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5429.647623] softirqs last enabled at (7454446): [] __do_softirq+0x2be/0x371 [ 5429.656222] softirqs last disabled at (7454441): [] irq_exit_rcu+0xb0/0xd0 [ 5429.664655] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5429.674031] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5429.681504] Workqueue: events delayed_work [ceph] [ 5429.686226] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5429.690577] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5429.709314] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5429.714532] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5429.721655] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5429.728779] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5429.735904] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5429.743028] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5429.750153] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5429.758229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5429.763966] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5429.771093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5429.778216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5429.785341] Call Trace: [ 5429.787786] [ 5429.789796] ? watchdog_timer_fn+0x1b4/0x220 [ 5429.794068] ? __hrtimer_run_queues+0x1d3/0x360 [ 5429.798595] ? hrtimer_interrupt+0x104/0x240 [ 5429.802864] ? __local_bh_enable+0x50/0x60 [ 5429.806957] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5429.812181] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5429.817138] [ 5429.819235] [ 5429.821334] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5429.826641] ? lock_acquire+0xf0/0x2b0 [ 5429.830391] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5429.835797] ? lock_release+0xc5/0x260 [ 5429.839544] _raw_spin_lock+0x30/0x40 [ 5429.843209] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5429.848609] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5429.853837] delayed_work+0x1d8/0x280 [ceph] [ 5429.858118] ? process_scheduled_works+0x22a/0x510 [ 5429.862908] process_scheduled_works+0x2a4/0x510 [ 5429.867522] worker_thread+0x188/0x340 [ 5429.871273] ? rescuer_thread+0x310/0x310 [ 5429.875275] kthread+0xf5/0x130 [ 5429.878413] ? kthread_complete_and_exit+0x20/0x20 [ 5429.883198] ret_from_fork+0x31/0x40 [ 5429.886778] ? kthread_complete_and_exit+0x20/0x20 [ 5429.891561] ret_from_fork_asm+0x11/0x20 [ 5429.895492] [ 5434.073146] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5444.313242] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5455.065700] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5457.561717] watchdog: BUG: soft lockup - CPU#7 stuck for 2965s! [kworker/7:4:78808] [ 5457.569366] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5457.624816] irq event stamp: 7515082 [ 5457.628387] hardirqs last enabled at (7515081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5457.638465] hardirqs last disabled at (7515082): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5457.648104] softirqs last enabled at (7515068): [] __do_softirq+0x2be/0x371 [ 5457.656711] softirqs last disabled at (7515063): [] irq_exit_rcu+0xb0/0xd0 [ 5457.665145] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5457.674521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5457.681993] Workqueue: events delayed_work [ceph] [ 5457.686714] RIP: 0010:_raw_spin_unlock+0x16/0x40 [ 5457.691326] Code: 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 [ 5457.710063] RSP: 0018:ffffc90002117da8 EFLAGS: 00000296 [ 5457.715280] RAX: ffff8881a1d34478 RBX: ffff88817f867488 RCX: ffffc90002117d54 [ 5457.722405] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [ 5457.729529] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 5457.736653] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 5457.743777] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5457.750902] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5457.758979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5457.764717] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5457.771840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5457.778964] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5457.786088] Call Trace: [ 5457.788535] [ 5457.790544] ? watchdog_timer_fn+0x1b4/0x220 [ 5457.794808] ? __hrtimer_run_queues+0x1d3/0x360 [ 5457.799334] ? hrtimer_interrupt+0x104/0x240 [ 5457.803596] ? __local_bh_enable+0x50/0x60 [ 5457.807688] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5457.812912] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5457.817870] [ 5457.819967] [ 5457.822066] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5457.827372] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5457.832777] ? _raw_spin_unlock+0x16/0x40 [ 5457.836781] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5457.842011] delayed_work+0x1d8/0x280 [ceph] [ 5457.846292] ? process_scheduled_works+0x22a/0x510 [ 5457.851082] process_scheduled_works+0x2a4/0x510 [ 5457.855697] worker_thread+0x188/0x340 [ 5457.859445] ? rescuer_thread+0x310/0x310 [ 5457.863450] kthread+0xf5/0x130 [ 5457.866588] ? kthread_complete_and_exit+0x20/0x20 [ 5457.871370] ret_from_fork+0x31/0x40 [ 5457.874941] ? kthread_complete_and_exit+0x20/0x20 [ 5457.879725] ret_from_fork_asm+0x11/0x20 [ 5457.883648] [ 5465.305793] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5476.057993] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5485.562207] watchdog: BUG: soft lockup - CPU#7 stuck for 2991s! [kworker/7:4:78808] [ 5485.569858] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5485.625305] irq event stamp: 7575376 [ 5485.628875] hardirqs last enabled at (7575375): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5485.638956] hardirqs last disabled at (7575376): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5485.648601] softirqs last enabled at (7575336): [] __do_softirq+0x2be/0x371 [ 5485.657199] softirqs last disabled at (7575331): [] irq_exit_rcu+0xb0/0xd0 [ 5485.665632] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5485.675011] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5485.682481] Workqueue: events delayed_work [ceph] [ 5485.687204] RIP: 0010:lock_acquire+0x65/0x2b0 [ 5485.691556] Code: 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 0f 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b 05 db d9 ea 7e <85> c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 [ 5485.710292] RSP: 0018:ffffc90002117d30 EFLAGS: 00000202 [ 5485.715508] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 0000000000000000 [ 5485.722633] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f8674a0 [ 5485.729757] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 5485.736883] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [ 5485.744006] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5485.751131] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5485.759208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5485.764945] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5485.772071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5485.779194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5485.786318] Call Trace: [ 5485.788765] [ 5485.790775] ? watchdog_timer_fn+0x1b4/0x220 [ 5485.795047] ? __hrtimer_run_queues+0x1d3/0x360 [ 5485.799573] ? hrtimer_interrupt+0x104/0x240 [ 5485.803843] ? __local_bh_enable+0x50/0x60 [ 5485.807935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5485.813159] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5485.818117] [ 5485.820214] [ 5485.822313] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5485.827618] ? lock_acquire+0x65/0x2b0 [ 5485.831360] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5485.836769] ? lock_release+0xc5/0x260 [ 5485.840514] _raw_spin_lock+0x30/0x40 [ 5485.844179] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5485.849577] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5485.854807] delayed_work+0x1d8/0x280 [ceph] [ 5485.859086] ? process_scheduled_works+0x22a/0x510 [ 5485.863876] process_scheduled_works+0x2a4/0x510 [ 5485.868494] worker_thread+0x188/0x340 [ 5485.872242] ? rescuer_thread+0x310/0x310 [ 5485.876245] kthread+0xf5/0x130 [ 5485.879383] ? kthread_complete_and_exit+0x20/0x20 [ 5485.884167] ret_from_fork+0x31/0x40 [ 5485.887746] ? kthread_complete_and_exit+0x20/0x20 [ 5485.892530] ret_from_fork_asm+0x11/0x20 [ 5485.896453] [ 5486.297984] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5497.050304] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5507.290455] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5513.562697] watchdog: BUG: soft lockup - CPU#7 stuck for 3017s! [kworker/7:4:78808] [ 5513.570363] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5513.625812] irq event stamp: 7635954 [ 5513.629383] hardirqs last enabled at (7635953): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5513.639463] hardirqs last disabled at (7635954): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5513.649101] softirqs last enabled at (7635940): [] __do_softirq+0x2be/0x371 [ 5513.657706] softirqs last disabled at (7635935): [] irq_exit_rcu+0xb0/0xd0 [ 5513.666140] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5513.675517] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5513.682989] Workqueue: events delayed_work [ceph] [ 5513.687702] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5513.692052] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5513.710790] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 5513.716008] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 5513.723131] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 5513.730255] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 5513.737378] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 5513.744505] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5513.751627] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5513.759705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5513.765443] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5513.772568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5513.779691] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5513.786815] Call Trace: [ 5513.789260] [ 5513.791271] ? watchdog_timer_fn+0x1b4/0x220 [ 5513.795535] ? __hrtimer_run_queues+0x1d3/0x360 [ 5513.800061] ? hrtimer_interrupt+0x104/0x240 [ 5513.804323] ? __local_bh_enable+0x50/0x60 [ 5513.808416] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5513.813640] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5513.818597] [ 5513.820694] [ 5513.822792] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5513.828099] ? lock_acquire+0xf0/0x2b0 [ 5513.831849] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5513.837258] _raw_spin_lock+0x30/0x40 [ 5513.840916] ? igrab+0x1d/0x50 [ 5513.843964] igrab+0x1d/0x50 [ 5513.846845] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5513.851993] delayed_work+0x1d8/0x280 [ceph] [ 5513.856274] ? process_scheduled_works+0x22a/0x510 [ 5513.861065] process_scheduled_works+0x2a4/0x510 [ 5513.865680] worker_thread+0x188/0x340 [ 5513.869430] ? rescuer_thread+0x310/0x310 [ 5513.873433] kthread+0xf5/0x130 [ 5513.876579] ? kthread_complete_and_exit+0x20/0x20 [ 5513.881362] ret_from_fork+0x31/0x40 [ 5513.884941] ? kthread_complete_and_exit+0x20/0x20 [ 5513.889725] ret_from_fork_asm+0x11/0x20 [ 5513.893650] [ 5518.042641] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5528.282728] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5539.035017] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5541.563189] watchdog: BUG: soft lockup - CPU#7 stuck for 3043s! [kworker/7:4:78808] [ 5541.570842] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5541.626291] irq event stamp: 7697388 [ 5541.629864] hardirqs last enabled at (7697387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5541.639942] hardirqs last disabled at (7697388): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5541.649581] softirqs last enabled at (7697366): [] __do_softirq+0x2be/0x371 [ 5541.658185] softirqs last disabled at (7697361): [] irq_exit_rcu+0xb0/0xd0 [ 5541.666620] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5541.675997] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5541.683468] Workqueue: events delayed_work [ceph] [ 5541.688192] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5541.692543] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5541.711279] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5541.716497] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5541.723620] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5541.730745] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5541.737868] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5541.744994] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5541.752118] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5541.760193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5541.765931] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5541.773058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5541.780180] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5541.787304] Call Trace: [ 5541.789751] [ 5541.791762] ? watchdog_timer_fn+0x1b4/0x220 [ 5541.796033] ? __hrtimer_run_queues+0x1d3/0x360 [ 5541.800567] ? hrtimer_interrupt+0x104/0x240 [ 5541.804828] ? __local_bh_enable+0x50/0x60 [ 5541.808922] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5541.814147] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5541.819102] [ 5541.821200] [ 5541.823297] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5541.828607] ? lock_acquire+0xf0/0x2b0 [ 5541.832357] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5541.837763] ? lock_release+0xc5/0x260 [ 5541.841509] _raw_spin_lock+0x30/0x40 [ 5541.845172] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5541.850572] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5541.855801] delayed_work+0x1d8/0x280 [ceph] [ 5541.860082] ? process_scheduled_works+0x22a/0x510 [ 5541.864871] process_scheduled_works+0x2a4/0x510 [ 5541.869489] worker_thread+0x188/0x340 [ 5541.873237] ? rescuer_thread+0x310/0x310 [ 5541.877240] kthread+0xf5/0x130 [ 5541.880378] ? kthread_complete_and_exit+0x20/0x20 [ 5541.885163] ret_from_fork+0x31/0x40 [ 5541.888742] ? kthread_complete_and_exit+0x20/0x20 [ 5541.893526] ret_from_fork_asm+0x11/0x20 [ 5541.897456] [ 5549.275162] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5560.027204] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5569.563699] watchdog: BUG: soft lockup - CPU#7 stuck for 3070s! [kworker/7:4:78808] [ 5569.571349] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5569.626800] irq event stamp: 7758322 [ 5569.630368] hardirqs last enabled at (7758321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5569.640448] hardirqs last disabled at (7758322): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5569.650087] softirqs last enabled at (7758308): [] __do_softirq+0x2be/0x371 [ 5569.658691] softirqs last disabled at (7758303): [] irq_exit_rcu+0xb0/0xd0 [ 5569.667118] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5569.676494] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5569.683966] Workqueue: events delayed_work [ceph] [ 5569.688689] RIP: 0010:ceph_check_caps+0x974/0xae0 [ceph] [ 5569.694009] Code: a0 b8 a0 48 c7 c7 68 f5 b5 a0 48 8b 81 e0 03 00 00 ff 70 28 51 48 c7 c1 ac 4e b9 a0 e8 d5 0e ae e0 48 83 c4 18 e9 ed f9 ff ff <89> c8 83 e0 fe 0f 84 e2 f8 ff ff 41 81 e7 00 3a 00 00 0f 85 d5 f8 [ 5569.712748] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [ 5569.717964] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: 0000000000000000 [ 5569.725090] RDX: ffff8881a1d34bc8 RSI: 0000000000000000 RDI: ffff88810da9e008 [ 5569.732212] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 5569.739337] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 5569.746461] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 5569.753587] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5569.761663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5569.767400] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5569.774525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5569.781648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5569.788773] Call Trace: [ 5569.791220] [ 5569.793231] ? watchdog_timer_fn+0x1b4/0x220 [ 5569.797502] ? __hrtimer_run_queues+0x1d3/0x360 [ 5569.802037] ? hrtimer_interrupt+0x104/0x240 [ 5569.806306] ? __local_bh_enable+0x50/0x60 [ 5569.810398] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5569.815615] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5569.820572] [ 5569.822670] [ 5569.824768] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5569.830074] ? ceph_check_caps+0x974/0xae0 [ceph] [ 5569.834796] ? lock_acquire+0xcb/0x2b0 [ 5569.838546] ? igrab+0x1d/0x50 [ 5569.841597] ? find_held_lock+0x2d/0x90 [ 5569.845429] ? find_held_lock+0x2d/0x90 [ 5569.849260] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5569.854659] ? lock_release+0xc5/0x260 [ 5569.858406] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5569.863639] delayed_work+0x1d8/0x280 [ceph] [ 5569.867922] ? process_scheduled_works+0x22a/0x510 [ 5569.872711] process_scheduled_works+0x2a4/0x510 [ 5569.877326] worker_thread+0x188/0x340 [ 5569.881075] ? rescuer_thread+0x310/0x310 [ 5569.885080] kthread+0xf5/0x130 [ 5569.888227] ? kthread_complete_and_exit+0x20/0x20 [ 5569.893009] ret_from_fork+0x31/0x40 [ 5569.896579] ? kthread_complete_and_exit+0x20/0x20 [ 5569.901365] ret_from_fork_asm+0x11/0x20 [ 5569.905296] [ 5570.267680] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5581.019636] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5591.259998] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5597.564171] watchdog: BUG: soft lockup - CPU#7 stuck for 3096s! [kworker/7:4:78808] [ 5597.571838] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5597.627287] irq event stamp: 7819676 [ 5597.630858] hardirqs last enabled at (7819675): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5597.640937] hardirqs last disabled at (7819676): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5597.650576] softirqs last enabled at (7819550): [] __do_softirq+0x2be/0x371 [ 5597.659181] softirqs last disabled at (7819545): [] irq_exit_rcu+0xb0/0xd0 [ 5597.667608] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5597.676983] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5597.684456] Workqueue: events delayed_work [ceph] [ 5597.689178] RIP: 0010:rcu_is_watching+0xb/0x40 [ 5597.693617] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [ 5597.712351] RSP: 0018:ffffc90002117b70 EFLAGS: 00000203 [ 5597.717570] RAX: 0000000000000007 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 5597.724694] RDX: ffff8881a1d34bc8 RSI: ffffffffa0b3407b RDI: ffff8881a1d34938 [ 5597.731819] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 5597.738942] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 5597.746065] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 5597.753191] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5597.761269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5597.767006] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5597.774130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5597.781254] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5597.788378] Call Trace: [ 5597.790824] [ 5597.792835] ? watchdog_timer_fn+0x1b4/0x220 [ 5597.797099] ? __hrtimer_run_queues+0x1d3/0x360 [ 5597.801632] ? hrtimer_interrupt+0x104/0x240 [ 5597.805893] ? __local_bh_enable+0x50/0x60 [ 5597.809986] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5597.815212] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5597.820167] [ 5597.822265] [ 5597.824363] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5597.829667] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5597.834374] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5597.839080] ? rcu_is_watching+0xb/0x40 [ 5597.842910] lock_release+0x1d7/0x260 [ 5597.846571] _raw_spin_unlock+0x1b/0x40 [ 5597.850409] ceph_check_caps+0x26b/0xae0 [ceph] [ 5597.854948] ? lock_acquire+0xcb/0x2b0 [ 5597.858693] ? igrab+0x1d/0x50 [ 5597.861752] ? find_held_lock+0x2d/0x90 [ 5597.865592] ? find_held_lock+0x2d/0x90 [ 5597.869421] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5597.874822] ? lock_release+0xc5/0x260 [ 5597.878568] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5597.883802] delayed_work+0x1d8/0x280 [ceph] [ 5597.888083] ? process_scheduled_works+0x22a/0x510 [ 5597.892873] process_scheduled_works+0x2a4/0x510 [ 5597.897490] worker_thread+0x188/0x340 [ 5597.901241] ? rescuer_thread+0x310/0x310 [ 5597.905250] kthread+0xf5/0x130 [ 5597.908389] ? kthread_complete_and_exit+0x20/0x20 [ 5597.913181] ret_from_fork+0x31/0x40 [ 5597.916760] ? kthread_complete_and_exit+0x20/0x20 [ 5597.921545] ret_from_fork_asm+0x11/0x20 [ 5597.925484] [ 5602.012128] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5612.252409] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5623.004472] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5625.564662] watchdog: BUG: soft lockup - CPU#7 stuck for 3122s! [kworker/7:4:78808] [ 5625.572310] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5625.627760] irq event stamp: 7881472 [ 5625.631330] hardirqs last enabled at (7881471): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5625.641409] hardirqs last disabled at (7881472): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5625.651047] softirqs last enabled at (7881386): [] __do_softirq+0x2be/0x371 [ 5625.659652] softirqs last disabled at (7881381): [] irq_exit_rcu+0xb0/0xd0 [ 5625.668079] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5625.677456] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5625.684927] Workqueue: events delayed_work [ceph] [ 5625.689650] RIP: 0010:lock_acquire+0x24/0x2b0 [ 5625.694009] Code: 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 <65> 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 [ 5625.712746] RSP: 0018:ffffc90002117d30 EFLAGS: 00000286 [ 5625.717962] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 0000000000000000 [ 5625.725087] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88817f8674a0 [ 5625.732213] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 5625.739336] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [ 5625.746461] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5625.753583] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5625.761661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5625.767400] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5625.774522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5625.781648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5625.788772] Call Trace: [ 5625.791219] [ 5625.793230] ? watchdog_timer_fn+0x1b4/0x220 [ 5625.797500] ? __hrtimer_run_queues+0x1d3/0x360 [ 5625.802026] ? hrtimer_interrupt+0x104/0x240 [ 5625.806287] ? __local_bh_enable+0x50/0x60 [ 5625.810380] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5625.815605] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5625.820561] [ 5625.822658] [ 5625.824756] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5625.830063] ? lock_acquire+0x24/0x2b0 [ 5625.833806] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5625.839212] ? lock_release+0xc5/0x260 [ 5625.842957] _raw_spin_lock+0x30/0x40 [ 5625.846615] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5625.852014] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5625.857243] delayed_work+0x1d8/0x280 [ceph] [ 5625.861524] ? process_scheduled_works+0x22a/0x510 [ 5625.866314] process_scheduled_works+0x2a4/0x510 [ 5625.870929] worker_thread+0x188/0x340 [ 5625.874677] ? rescuer_thread+0x310/0x310 [ 5625.878680] kthread+0xf5/0x130 [ 5625.881820] ? kthread_complete_and_exit+0x20/0x20 [ 5625.886603] ret_from_fork+0x31/0x40 [ 5625.890184] ? kthread_complete_and_exit+0x20/0x20 [ 5625.894965] ret_from_fork_asm+0x11/0x20 [ 5625.898889] [ 5633.244677] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5643.996935] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5653.565172] watchdog: BUG: soft lockup - CPU#7 stuck for 3148s! [kworker/7:4:78808] [ 5653.572825] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5653.628274] irq event stamp: 7942556 [ 5653.631844] hardirqs last enabled at (7942555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5653.641924] hardirqs last disabled at (7942556): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5653.651569] softirqs last enabled at (7942542): [] __do_softirq+0x2be/0x371 [ 5653.660166] softirqs last disabled at (7942537): [] irq_exit_rcu+0xb0/0xd0 [ 5653.668594] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5653.677968] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5653.685443] Workqueue: events delayed_work [ceph] [ 5653.690165] RIP: 0010:rcu_is_watching+0xb/0x40 [ 5653.694610] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [ 5653.713348] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [ 5653.718565] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [ 5653.725689] RDX: 0000000000000001 RSI: ffffffff813b9425 RDI: ffff8881a1d34518 [ 5653.732812] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [ 5653.739936] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9425 [ 5653.747061] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5653.754185] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5653.762262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5653.768001] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5653.775125] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5653.782249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5653.789372] Call Trace: [ 5653.791818] [ 5653.793829] ? watchdog_timer_fn+0x1b4/0x220 [ 5653.798100] ? __hrtimer_run_queues+0x1d3/0x360 [ 5653.802626] ? hrtimer_interrupt+0x104/0x240 [ 5653.806889] ? __local_bh_enable+0x50/0x60 [ 5653.810980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5653.816206] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5653.821163] [ 5653.823259] [ 5653.825358] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5653.830662] ? igrab+0x35/0x50 [ 5653.833714] ? igrab+0x35/0x50 [ 5653.836762] ? rcu_is_watching+0xb/0x40 [ 5653.840595] lock_release+0x1d7/0x260 [ 5653.844263] _raw_spin_unlock+0x1b/0x40 [ 5653.848100] igrab+0x35/0x50 [ 5653.850978] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5653.856129] delayed_work+0x1d8/0x280 [ceph] [ 5653.860409] ? process_scheduled_works+0x22a/0x510 [ 5653.865199] process_scheduled_works+0x2a4/0x510 [ 5653.869814] worker_thread+0x188/0x340 [ 5653.873563] ? rescuer_thread+0x310/0x310 [ 5653.877567] kthread+0xf5/0x130 [ 5653.880715] ? kthread_complete_and_exit+0x20/0x20 [ 5653.885505] ret_from_fork+0x31/0x40 [ 5653.889085] ? kthread_complete_and_exit+0x20/0x20 [ 5653.893870] ret_from_fork_asm+0x11/0x20 [ 5653.897800] [ 5654.237038] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5664.989318] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5675.229372] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5681.565644] watchdog: BUG: soft lockup - CPU#7 stuck for 3174s! [kworker/7:4:78808] [ 5681.573313] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5681.628762] irq event stamp: 8002950 [ 5681.632334] hardirqs last enabled at (8002949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5681.642411] hardirqs last disabled at (8002950): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5681.652049] softirqs last enabled at (8002932): [] __do_softirq+0x2be/0x371 [ 5681.660647] softirqs last disabled at (8002927): [] irq_exit_rcu+0xb0/0xd0 [ 5681.669082] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5681.678458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5681.685930] Workqueue: events delayed_work [ceph] [ 5681.690652] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5681.695003] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5681.713740] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5681.718957] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5681.726081] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5681.733205] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5681.740329] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5681.747455] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5681.754579] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5681.762655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5681.768393] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5681.775517] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5681.782642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5681.789765] Call Trace: [ 5681.792211] [ 5681.794222] ? watchdog_timer_fn+0x1b4/0x220 [ 5681.798486] ? __hrtimer_run_queues+0x1d3/0x360 [ 5681.803011] ? hrtimer_interrupt+0x104/0x240 [ 5681.807273] ? __local_bh_enable+0x50/0x60 [ 5681.811365] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5681.816590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5681.821548] [ 5681.823643] [ 5681.825742] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5681.831051] ? lock_acquire+0xf0/0x2b0 [ 5681.834801] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5681.839515] ? lock_release+0xc5/0x260 [ 5681.843259] _raw_spin_lock+0x30/0x40 [ 5681.846925] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5681.851543] ceph_check_caps+0x63/0xae0 [ceph] [ 5681.855994] ? __lock_acquire+0xbfa/0x1e20 [ 5681.860092] ? lock_acquire+0xcb/0x2b0 [ 5681.863841] ? igrab+0x1d/0x50 [ 5681.866891] ? find_held_lock+0x2d/0x90 [ 5681.870722] ? find_held_lock+0x2d/0x90 [ 5681.874553] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5681.879953] ? lock_release+0xc5/0x260 [ 5681.883699] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5681.888925] delayed_work+0x1d8/0x280 [ceph] [ 5681.893207] ? process_scheduled_works+0x22a/0x510 [ 5681.897996] process_scheduled_works+0x2a4/0x510 [ 5681.902611] worker_thread+0x188/0x340 [ 5681.906362] ? rescuer_thread+0x310/0x310 [ 5681.910366] kthread+0xf5/0x130 [ 5681.913504] ? kthread_complete_and_exit+0x20/0x20 [ 5681.918286] ret_from_fork+0x31/0x40 [ 5681.921858] ? kthread_complete_and_exit+0x20/0x20 [ 5681.926642] ret_from_fork_asm+0x11/0x20 [ 5681.930574] [ 5685.981679] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5696.221673] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5706.461947] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5709.566135] watchdog: BUG: soft lockup - CPU#7 stuck for 3200s! [kworker/7:4:78808] [ 5709.573785] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5709.629235] irq event stamp: 8064588 [ 5709.632804] hardirqs last enabled at (8064587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5709.642883] hardirqs last disabled at (8064588): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5709.652520] softirqs last enabled at (8064508): [] __do_softirq+0x2be/0x371 [ 5709.661120] softirqs last disabled at (8064503): [] irq_exit_rcu+0xb0/0xd0 [ 5709.669554] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5709.678929] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5709.686401] Workqueue: events delayed_work [ceph] [ 5709.691123] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5709.695474] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5709.714211] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 5709.719428] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 5709.726553] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 5709.733676] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 5709.740802] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 5709.747926] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5709.755049] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5709.763126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5709.768864] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5709.775989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5709.783113] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5709.790238] Call Trace: [ 5709.792683] [ 5709.794694] ? watchdog_timer_fn+0x1b4/0x220 [ 5709.798965] ? __hrtimer_run_queues+0x1d3/0x360 [ 5709.803493] ? hrtimer_interrupt+0x104/0x240 [ 5709.807760] ? __local_bh_enable+0x50/0x60 [ 5709.811854] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5709.817079] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5709.822035] [ 5709.824134] [ 5709.826230] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5709.831537] ? lock_acquire+0xf0/0x2b0 [ 5709.835280] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5709.840688] _raw_spin_lock+0x30/0x40 [ 5709.844345] ? igrab+0x1d/0x50 [ 5709.847396] igrab+0x1d/0x50 [ 5709.850274] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5709.855414] delayed_work+0x1d8/0x280 [ceph] [ 5709.859696] ? process_scheduled_works+0x22a/0x510 [ 5709.864486] process_scheduled_works+0x2a4/0x510 [ 5709.869103] worker_thread+0x188/0x340 [ 5709.872851] ? rescuer_thread+0x310/0x310 [ 5709.876854] kthread+0xf5/0x130 [ 5709.879993] ? kthread_complete_and_exit+0x20/0x20 [ 5709.884785] ret_from_fork+0x31/0x40 [ 5709.888363] ? kthread_complete_and_exit+0x20/0x20 [ 5709.893147] ret_from_fork_asm+0x11/0x20 [ 5709.897071] [ 5737.566648] watchdog: BUG: soft lockup - CPU#7 stuck for 3226s! [kworker/7:4:78808] [ 5737.574298] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5737.629748] irq event stamp: 8126634 [ 5737.633319] hardirqs last enabled at (8126633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5737.643398] hardirqs last disabled at (8126634): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5737.653035] softirqs last enabled at (8126612): [] __do_softirq+0x2be/0x371 [ 5737.661641] softirqs last disabled at (8126607): [] irq_exit_rcu+0xb0/0xd0 [ 5737.670069] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5737.679445] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5737.686917] Workqueue: events delayed_work [ceph] [ 5737.691639] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5737.695990] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5737.714726] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5737.719945] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5737.727069] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5737.734191] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5737.741316] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5737.748441] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5737.755564] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5737.763641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5737.769379] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5737.776505] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5737.783627] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5737.790751] Call Trace: [ 5737.793198] [ 5737.795209] ? watchdog_timer_fn+0x1b4/0x220 [ 5737.799480] ? __hrtimer_run_queues+0x1d3/0x360 [ 5737.804016] ? hrtimer_interrupt+0x104/0x240 [ 5737.808285] ? __local_bh_enable+0x50/0x60 [ 5737.812378] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5737.817602] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5737.822559] [ 5737.824657] [ 5737.826753] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5737.832062] ? lock_acquire+0xf0/0x2b0 [ 5737.835812] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5737.841219] ? lock_release+0xc5/0x260 [ 5737.844964] _raw_spin_lock+0x30/0x40 [ 5737.848622] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5737.854020] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5737.859249] delayed_work+0x1d8/0x280 [ceph] [ 5737.863531] ? process_scheduled_works+0x22a/0x510 [ 5737.868320] process_scheduled_works+0x2a4/0x510 [ 5737.872935] worker_thread+0x188/0x340 [ 5737.876683] ? rescuer_thread+0x310/0x310 [ 5737.880688] kthread+0xf5/0x130 [ 5737.883827] ? kthread_complete_and_exit+0x20/0x20 [ 5737.888609] ret_from_fork+0x31/0x40 [ 5737.892180] ? kthread_complete_and_exit+0x20/0x20 [ 5737.896964] ret_from_fork_asm+0x11/0x20 [ 5737.900896] [ 5738.206113] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [ 5738.214693] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5748.446655] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5759.198847] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 5765.567122] watchdog: BUG: soft lockup - CPU#7 stuck for 3252s! [kworker/7:4:78808] [ 5765.574769] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5765.630221] irq event stamp: 8190392 [ 5765.633790] hardirqs last enabled at (8190391): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5765.643870] hardirqs last disabled at (8190392): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5765.653507] softirqs last enabled at (8190378): [] __do_softirq+0x2be/0x371 [ 5765.662114] softirqs last disabled at (8190373): [] irq_exit_rcu+0xb0/0xd0 [ 5765.670547] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5765.679924] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5765.687395] Workqueue: events delayed_work [ceph] [ 5765.692119] RIP: 0010:do_raw_spin_lock+0x24/0xc0 [ 5765.696730] Code: 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 <48> 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba [ 5765.715466] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [ 5765.720684] RAX: ffff88810debaa80 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 5765.727807] RDX: ffffffffffffffff RSI: 0000000000000000 RDI: ffff88817f867488 [ 5765.734932] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 5765.742056] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 5765.749180] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5765.756304] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5765.764382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5765.770120] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5765.777245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5765.784367] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5765.791491] Call Trace: [ 5765.793939] [ 5765.795948] ? watchdog_timer_fn+0x1b4/0x220 [ 5765.800212] ? __hrtimer_run_queues+0x1d3/0x360 [ 5765.804746] ? hrtimer_interrupt+0x104/0x240 [ 5765.809007] ? __local_bh_enable+0x50/0x60 [ 5765.813100] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5765.818324] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5765.823283] [ 5765.825378] [ 5765.827477] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5765.832785] ? do_raw_spin_lock+0x24/0xc0 [ 5765.836794] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5765.842194] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5765.847423] delayed_work+0x1d8/0x280 [ceph] [ 5765.851704] ? process_scheduled_works+0x22a/0x510 [ 5765.856494] process_scheduled_works+0x2a4/0x510 [ 5765.861108] worker_thread+0x188/0x340 [ 5765.864859] ? rescuer_thread+0x310/0x310 [ 5765.868871] kthread+0xf5/0x130 [ 5765.872018] ? kthread_complete_and_exit+0x20/0x20 [ 5765.876800] ret_from_fork+0x31/0x40 [ 5765.880380] ? kthread_complete_and_exit+0x20/0x20 [ 5765.885164] ret_from_fork_asm+0x11/0x20 [ 5765.889096] [ 5790.430962] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [ 5793.567609] watchdog: BUG: soft lockup - CPU#7 stuck for 3278s! [kworker/7:4:78808] [ 5793.575258] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5793.630708] irq event stamp: 8253898 [ 5793.634280] hardirqs last enabled at (8253897): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5793.644358] hardirqs last disabled at (8253898): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5793.653995] softirqs last enabled at (8253858): [] __do_softirq+0x2be/0x371 [ 5793.662603] softirqs last disabled at (8253853): [] irq_exit_rcu+0xb0/0xd0 [ 5793.671036] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5793.680413] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5793.687886] Workqueue: events delayed_work [ceph] [ 5793.692608] RIP: 0010:lock_release+0x3c/0x260 [ 5793.696958] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [ 5793.715696] RSP: 0018:ffffc90002117d60 EFLAGS: 00000247 [ 5793.720913] RAX: 0000000000000007 RBX: ffff88817f867488 RCX: ffffc90002117d54 [ 5793.728037] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [ 5793.735161] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 5793.742284] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 5793.749410] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5793.756533] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5793.764611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5793.770348] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5793.777473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5793.784597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5793.791720] Call Trace: [ 5793.794166] [ 5793.796178] ? watchdog_timer_fn+0x1b4/0x220 [ 5793.800440] ? __hrtimer_run_queues+0x1d3/0x360 [ 5793.804976] ? hrtimer_interrupt+0x104/0x240 [ 5793.809244] ? __local_bh_enable+0x50/0x60 [ 5793.813337] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5793.818562] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5793.823520] [ 5793.825618] [ 5793.827715] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5793.833019] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5793.838430] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5793.843836] ? lock_release+0x3c/0x260 [ 5793.847581] _raw_spin_unlock+0x1b/0x40 [ 5793.851420] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5793.856655] delayed_work+0x1d8/0x280 [ceph] [ 5793.860937] ? process_scheduled_works+0x22a/0x510 [ 5793.865727] process_scheduled_works+0x2a4/0x510 [ 5793.870342] worker_thread+0x188/0x340 [ 5793.874091] ? rescuer_thread+0x310/0x310 [ 5793.878096] kthread+0xf5/0x130 [ 5793.881243] ? kthread_complete_and_exit+0x20/0x20 [ 5793.886033] ret_from_fork+0x31/0x40 [ 5793.889614] ? kthread_complete_and_exit+0x20/0x20 [ 5793.894398] ret_from_fork_asm+0x11/0x20 [ 5793.898328] [ 5821.568101] watchdog: BUG: soft lockup - CPU#7 stuck for 3304s! [kworker/7:4:78808] [ 5821.575755] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5821.631207] irq event stamp: 8316084 [ 5821.634776] hardirqs last enabled at (8316083): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5821.644856] hardirqs last disabled at (8316084): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5821.654493] softirqs last enabled at (8316062): [] __do_softirq+0x2be/0x371 [ 5821.663091] softirqs last disabled at (8316057): [] irq_exit_rcu+0xb0/0xd0 [ 5821.671527] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5821.680911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5821.688382] Workqueue: events delayed_work [ceph] [ 5821.693105] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5821.697456] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5821.716192] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5821.721410] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5821.728534] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5821.735659] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5821.742783] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5821.749906] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5821.757031] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5821.765107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5821.770845] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5821.777972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5821.785093] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5821.792217] Call Trace: [ 5821.794666] [ 5821.796676] ? watchdog_timer_fn+0x1b4/0x220 [ 5821.800947] ? __hrtimer_run_queues+0x1d3/0x360 [ 5821.805480] ? hrtimer_interrupt+0x104/0x240 [ 5821.809743] ? __local_bh_enable+0x50/0x60 [ 5821.813835] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5821.819059] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5821.824016] [ 5821.826114] [ 5821.828212] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5821.833519] ? lock_acquire+0xf0/0x2b0 [ 5821.837269] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5821.842676] ? lock_release+0xc5/0x260 [ 5821.846421] _raw_spin_lock+0x30/0x40 [ 5821.850079] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5821.855478] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5821.860707] delayed_work+0x1d8/0x280 [ceph] [ 5821.864989] ? process_scheduled_works+0x22a/0x510 [ 5821.869779] process_scheduled_works+0x2a4/0x510 [ 5821.874393] worker_thread+0x188/0x340 [ 5821.878142] ? rescuer_thread+0x310/0x310 [ 5821.882146] kthread+0xf5/0x130 [ 5821.885293] ? kthread_complete_and_exit+0x20/0x20 [ 5821.890075] ret_from_fork+0x31/0x40 [ 5821.893655] ? kthread_complete_and_exit+0x20/0x20 [ 5821.898439] ret_from_fork_asm+0x11/0x20 [ 5821.902370] [ 5849.568601] watchdog: BUG: soft lockup - CPU#7 stuck for 3330s! [kworker/7:4:78808] [ 5849.576254] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5849.631705] irq event stamp: 8378252 [ 5849.635275] hardirqs last enabled at (8378251): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5849.645353] hardirqs last disabled at (8378252): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5849.654991] softirqs last enabled at (8378238): [] __do_softirq+0x2be/0x371 [ 5849.663598] softirqs last disabled at (8378233): [] irq_exit_rcu+0xb0/0xd0 [ 5849.672032] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5849.681407] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5849.688879] Workqueue: events delayed_work [ceph] [ 5849.693602] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5849.697953] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5849.716690] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5849.721908] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5849.729033] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5849.736155] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5849.743280] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5849.750404] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5849.757530] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5849.765606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5849.771342] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5849.778469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5849.785591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5849.792716] Call Trace: [ 5849.795163] [ 5849.797174] ? watchdog_timer_fn+0x1b4/0x220 [ 5849.801443] ? __hrtimer_run_queues+0x1d3/0x360 [ 5849.805970] ? hrtimer_interrupt+0x104/0x240 [ 5849.810232] ? __local_bh_enable+0x50/0x60 [ 5849.814324] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5849.819549] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5849.824506] [ 5849.826604] [ 5849.828700] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5849.834007] ? lock_acquire+0xf0/0x2b0 [ 5849.837751] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5849.842465] ? lock_release+0xc5/0x260 [ 5849.846210] _raw_spin_lock+0x30/0x40 [ 5849.849875] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5849.854493] ceph_check_caps+0x63/0xae0 [ceph] [ 5849.858944] ? __lock_acquire+0xbfa/0x1e20 [ 5849.863043] ? lock_acquire+0xcb/0x2b0 [ 5849.866792] ? igrab+0x1d/0x50 [ 5849.869843] ? find_held_lock+0x2d/0x90 [ 5849.873682] ? find_held_lock+0x2d/0x90 [ 5849.877512] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5849.882912] ? lock_release+0xc5/0x260 [ 5849.886659] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5849.891884] delayed_work+0x1d8/0x280 [ceph] [ 5849.896165] ? process_scheduled_works+0x22a/0x510 [ 5849.900956] process_scheduled_works+0x2a4/0x510 [ 5849.905572] worker_thread+0x188/0x340 [ 5849.909320] ? rescuer_thread+0x310/0x310 [ 5849.913324] kthread+0xf5/0x130 [ 5849.916462] ? kthread_complete_and_exit+0x20/0x20 [ 5849.921247] ret_from_fork+0x31/0x40 [ 5849.924825] ? kthread_complete_and_exit+0x20/0x20 [ 5849.929610] ret_from_fork_asm+0x11/0x20 [ 5849.933541] [ 5877.569101] watchdog: BUG: soft lockup - CPU#7 stuck for 3356s! [kworker/7:4:78808] [ 5877.576752] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5877.632200] irq event stamp: 8441380 [ 5877.635773] hardirqs last enabled at (8441379): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5877.645852] hardirqs last disabled at (8441380): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5877.655496] softirqs last enabled at (8441332): [] __do_softirq+0x2be/0x371 [ 5877.664097] softirqs last disabled at (8441325): [] irq_exit_rcu+0xb0/0xd0 [ 5877.672530] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5877.681907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5877.689378] Workqueue: events delayed_work [ceph] [ 5877.694101] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5877.698459] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5877.717197] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 5877.722415] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 5877.729538] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 5877.736663] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 5877.743788] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 5877.750912] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5877.758035] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5877.766112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5877.771849] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5877.778975] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5877.786098] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5877.793222] Call Trace: [ 5877.795668] [ 5877.797679] ? watchdog_timer_fn+0x1b4/0x220 [ 5877.801950] ? __hrtimer_run_queues+0x1d3/0x360 [ 5877.806477] ? hrtimer_interrupt+0x104/0x240 [ 5877.810746] ? __local_bh_enable+0x50/0x60 [ 5877.814841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5877.820066] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5877.825020] [ 5877.827118] [ 5877.829215] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5877.834524] ? lock_acquire+0xf0/0x2b0 [ 5877.838273] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 5877.842988] ? lock_release+0xc5/0x260 [ 5877.846732] _raw_spin_lock+0x30/0x40 [ 5877.850390] ? ceph_check_caps+0x63/0xae0 [ceph] [ 5877.855009] ceph_check_caps+0x63/0xae0 [ceph] [ 5877.859462] ? __lock_acquire+0xbfa/0x1e20 [ 5877.863575] ? lock_acquire+0xcb/0x2b0 [ 5877.867325] ? igrab+0x1d/0x50 [ 5877.870373] ? find_held_lock+0x2d/0x90 [ 5877.874206] ? find_held_lock+0x2d/0x90 [ 5877.878036] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5877.883436] ? lock_release+0xc5/0x260 [ 5877.887181] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5877.892409] delayed_work+0x1d8/0x280 [ceph] [ 5877.896689] ? process_scheduled_works+0x22a/0x510 [ 5877.901479] process_scheduled_works+0x2a4/0x510 [ 5877.906095] worker_thread+0x188/0x340 [ 5877.909845] ? rescuer_thread+0x310/0x310 [ 5877.913846] kthread+0xf5/0x130 [ 5877.916987] ? kthread_complete_and_exit+0x20/0x20 [ 5877.921771] ret_from_fork+0x31/0x40 [ 5877.925350] ? kthread_complete_and_exit+0x20/0x20 [ 5877.930133] ret_from_fork_asm+0x11/0x20 [ 5877.934065] [ 5905.569579] watchdog: BUG: soft lockup - CPU#7 stuck for 3382s! [kworker/7:4:78808] [ 5905.577233] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5905.632682] irq event stamp: 8503118 [ 5905.636251] hardirqs last enabled at (8503117): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5905.646332] hardirqs last disabled at (8503118): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5905.655977] softirqs last enabled at (8503038): [] __do_softirq+0x2be/0x371 [ 5905.664575] softirqs last disabled at (8503033): [] irq_exit_rcu+0xb0/0xd0 [ 5905.673000] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5905.682377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5905.689848] Workqueue: events delayed_work [ceph] [ 5905.694573] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5905.698931] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5905.717669] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5905.722886] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5905.730009] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5905.737134] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5905.744258] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5905.751383] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5905.758507] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5905.766582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.772320] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5905.779445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.786570] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.793694] Call Trace: [ 5905.796140] [ 5905.798151] ? watchdog_timer_fn+0x1b4/0x220 [ 5905.802423] ? __hrtimer_run_queues+0x1d3/0x360 [ 5905.806956] ? hrtimer_interrupt+0x104/0x240 [ 5905.811217] ? __local_bh_enable+0x50/0x60 [ 5905.815312] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5905.820534] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5905.825492] [ 5905.827590] [ 5905.829688] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5905.834995] ? lock_acquire+0xf0/0x2b0 [ 5905.838745] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5905.844152] ? lock_release+0xc5/0x260 [ 5905.847898] _raw_spin_lock+0x30/0x40 [ 5905.851562] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5905.856961] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5905.862190] delayed_work+0x1d8/0x280 [ceph] [ 5905.866471] ? process_scheduled_works+0x22a/0x510 [ 5905.871261] process_scheduled_works+0x2a4/0x510 [ 5905.875878] worker_thread+0x188/0x340 [ 5905.879626] ? rescuer_thread+0x310/0x310 [ 5905.883629] kthread+0xf5/0x130 [ 5905.886769] ? kthread_complete_and_exit+0x20/0x20 [ 5905.891552] ret_from_fork+0x31/0x40 [ 5905.895130] ? kthread_complete_and_exit+0x20/0x20 [ 5905.899916] ret_from_fork_asm+0x11/0x20 [ 5905.903846] [ 5933.570071] watchdog: BUG: soft lockup - CPU#7 stuck for 3409s! [kworker/7:4:78808] [ 5933.577722] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5933.633170] irq event stamp: 8564054 [ 5933.636740] hardirqs last enabled at (8564053): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5933.646820] hardirqs last disabled at (8564054): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5933.656458] softirqs last enabled at (8564040): [] __do_softirq+0x2be/0x371 [ 5933.665063] softirqs last disabled at (8564035): [] irq_exit_rcu+0xb0/0xd0 [ 5933.673490] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5933.682868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5933.690338] Workqueue: events delayed_work [ceph] [ 5933.695062] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 5933.699679] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 5933.718417] RSP: 0018:ffffc90002117bc0 EFLAGS: 00000246 [ 5933.723634] RAX: 0000000000000000 RBX: ffff8881a1d34920 RCX: 00000000d3c6b569 [ 5933.730759] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34920 [ 5933.737882] RBP: 0000000000000000 R08: 00000000000079a2 R09: 0000000000000002 [ 5933.745008] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 5933.752130] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 5933.759255] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5933.767332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5933.773070] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5933.780194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5933.787317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5933.794442] Call Trace: [ 5933.796888] [ 5933.798899] ? watchdog_timer_fn+0x1b4/0x220 [ 5933.803163] ? __hrtimer_run_queues+0x1d3/0x360 [ 5933.807696] ? hrtimer_interrupt+0x104/0x240 [ 5933.811958] ? __local_bh_enable+0x50/0x60 [ 5933.816051] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5933.821275] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5933.826234] [ 5933.828329] [ 5933.830427] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5933.835735] ? do_raw_spin_lock+0x42/0xc0 [ 5933.839745] ceph_check_caps+0x63/0xae0 [ceph] [ 5933.844204] ? __lock_acquire+0xbfa/0x1e20 [ 5933.848302] ? lock_acquire+0xcb/0x2b0 [ 5933.852051] ? igrab+0x1d/0x50 [ 5933.855102] ? find_held_lock+0x2d/0x90 [ 5933.858934] ? find_held_lock+0x2d/0x90 [ 5933.862763] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5933.868164] ? lock_release+0xc5/0x260 [ 5933.871909] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 5933.877135] delayed_work+0x1d8/0x280 [ceph] [ 5933.881418] ? process_scheduled_works+0x22a/0x510 [ 5933.886207] process_scheduled_works+0x2a4/0x510 [ 5933.890823] worker_thread+0x188/0x340 [ 5933.894572] ? rescuer_thread+0x310/0x310 [ 5933.898584] kthread+0xf5/0x130 [ 5933.901732] ? kthread_complete_and_exit+0x20/0x20 [ 5933.906523] ret_from_fork+0x31/0x40 [ 5933.910102] ? kthread_complete_and_exit+0x20/0x20 [ 5933.914885] ret_from_fork_asm+0x11/0x20 [ 5933.918810] [ 5961.570564] watchdog: BUG: soft lockup - CPU#7 stuck for 3435s! [kworker/7:4:78808] [ 5961.578219] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5961.633668] irq event stamp: 8625200 [ 5961.637238] hardirqs last enabled at (8625199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5961.647317] hardirqs last disabled at (8625200): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5961.656955] softirqs last enabled at (8625088): [] __do_softirq+0x2be/0x371 [ 5961.665554] softirqs last disabled at (8625083): [] irq_exit_rcu+0xb0/0xd0 [ 5961.673989] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5961.683371] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5961.690845] Workqueue: events delayed_work [ceph] [ 5961.695567] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5961.699919] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5961.718654] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 5961.723872] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 5961.730997] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 5961.738119] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 5961.745243] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 5961.752369] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5961.759492] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5961.767570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5961.773307] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5961.780434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5961.787556] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5961.794679] Call Trace: [ 5961.797126] [ 5961.799137] ? watchdog_timer_fn+0x1b4/0x220 [ 5961.803401] ? __hrtimer_run_queues+0x1d3/0x360 [ 5961.807926] ? hrtimer_interrupt+0x104/0x240 [ 5961.812187] ? __local_bh_enable+0x50/0x60 [ 5961.816281] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5961.821503] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5961.826462] [ 5961.828560] [ 5961.830656] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5961.835965] ? lock_acquire+0xf0/0x2b0 [ 5961.839713] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5961.845122] ? lock_release+0xc5/0x260 [ 5961.848867] _raw_spin_lock+0x30/0x40 [ 5961.852533] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5961.857932] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 5961.863159] delayed_work+0x1d8/0x280 [ceph] [ 5961.867440] ? process_scheduled_works+0x22a/0x510 [ 5961.872232] process_scheduled_works+0x2a4/0x510 [ 5961.876847] worker_thread+0x188/0x340 [ 5961.880595] ? rescuer_thread+0x310/0x310 [ 5961.884598] kthread+0xf5/0x130 [ 5961.887737] ? kthread_complete_and_exit+0x20/0x20 [ 5961.892522] ret_from_fork+0x31/0x40 [ 5961.896100] ? kthread_complete_and_exit+0x20/0x20 [ 5961.900883] ret_from_fork_asm+0x11/0x20 [ 5961.904807] [ 5989.571055] watchdog: BUG: soft lockup - CPU#7 stuck for 3461s! [kworker/7:4:78808] [ 5989.578707] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 5989.634157] irq event stamp: 8686474 [ 5989.637727] hardirqs last enabled at (8686473): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5989.647806] hardirqs last disabled at (8686474): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 5989.657444] softirqs last enabled at (8686460): [] __do_softirq+0x2be/0x371 [ 5989.666043] softirqs last disabled at (8686455): [] irq_exit_rcu+0xb0/0xd0 [ 5989.674476] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 5989.683853] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 5989.691323] Workqueue: events delayed_work [ceph] [ 5989.696047] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 5989.700399] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 5989.719134] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 5989.724353] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 5989.731477] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 5989.738600] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 5989.745725] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 5989.752848] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 5989.759973] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 5989.768050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5989.773788] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 5989.780912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5989.788037] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5989.795161] Call Trace: [ 5989.797608] [ 5989.799619] ? watchdog_timer_fn+0x1b4/0x220 [ 5989.803890] ? __hrtimer_run_queues+0x1d3/0x360 [ 5989.808423] ? hrtimer_interrupt+0x104/0x240 [ 5989.812685] ? __local_bh_enable+0x50/0x60 [ 5989.816779] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 5989.822001] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 5989.826959] [ 5989.829058] [ 5989.831154] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 5989.836461] ? lock_acquire+0xf0/0x2b0 [ 5989.840213] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 5989.845620] _raw_spin_lock+0x30/0x40 [ 5989.849277] ? igrab+0x1d/0x50 [ 5989.852328] igrab+0x1d/0x50 [ 5989.855207] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 5989.860355] delayed_work+0x1d8/0x280 [ceph] [ 5989.864636] ? process_scheduled_works+0x22a/0x510 [ 5989.869427] process_scheduled_works+0x2a4/0x510 [ 5989.874041] worker_thread+0x188/0x340 [ 5989.877793] ? rescuer_thread+0x310/0x310 [ 5989.881803] kthread+0xf5/0x130 [ 5989.884952] ? kthread_complete_and_exit+0x20/0x20 [ 5989.889742] ret_from_fork+0x31/0x40 [ 5989.893313] ? kthread_complete_and_exit+0x20/0x20 [ 5989.898097] ret_from_fork_asm+0x11/0x20 [ 5989.902021] [ 6017.571548] watchdog: BUG: soft lockup - CPU#7 stuck for 3487s! [kworker/7:4:78808] [ 6017.579196] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6017.634645] irq event stamp: 8750334 [ 6017.638216] hardirqs last enabled at (8750333): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6017.648296] hardirqs last disabled at (8750334): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6017.657934] softirqs last enabled at (8750312): [] __do_softirq+0x2be/0x371 [ 6017.666541] softirqs last disabled at (8750307): [] irq_exit_rcu+0xb0/0xd0 [ 6017.674973] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6017.684351] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6017.691822] Workqueue: events delayed_work [ceph] [ 6017.696547] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6017.700904] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6017.719641] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 6017.724858] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 6017.731983] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 6017.739106] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 6017.746230] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6017.753356] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6017.760479] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6017.768557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6017.774295] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6017.781420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6017.788542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6017.795667] Call Trace: [ 6017.798113] [ 6017.800123] ? watchdog_timer_fn+0x1b4/0x220 [ 6017.804387] ? __hrtimer_run_queues+0x1d3/0x360 [ 6017.808921] ? hrtimer_interrupt+0x104/0x240 [ 6017.813182] ? __local_bh_enable+0x50/0x60 [ 6017.817277] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6017.822501] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6017.827458] [ 6017.829555] [ 6017.831652] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6017.836960] ? lock_acquire+0xf0/0x2b0 [ 6017.840711] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6017.845424] ? lock_release+0xc5/0x260 [ 6017.849169] _raw_spin_lock+0x30/0x40 [ 6017.852833] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6017.857453] ceph_check_caps+0x63/0xae0 [ceph] [ 6017.861903] ? __lock_acquire+0xbfa/0x1e20 [ 6017.866003] ? lock_acquire+0xcb/0x2b0 [ 6017.869752] ? igrab+0x1d/0x50 [ 6017.872812] ? find_held_lock+0x2d/0x90 [ 6017.876650] ? find_held_lock+0x2d/0x90 [ 6017.880482] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6017.885879] ? lock_release+0xc5/0x260 [ 6017.889627] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6017.894863] delayed_work+0x1d8/0x280 [ceph] [ 6017.899142] ? process_scheduled_works+0x22a/0x510 [ 6017.903932] process_scheduled_works+0x2a4/0x510 [ 6017.908548] worker_thread+0x188/0x340 [ 6017.912297] ? rescuer_thread+0x310/0x310 [ 6017.916301] kthread+0xf5/0x130 [ 6017.919440] ? kthread_complete_and_exit+0x20/0x20 [ 6017.924231] ret_from_fork+0x31/0x40 [ 6017.927813] ? kthread_complete_and_exit+0x20/0x20 [ 6017.932595] ret_from_fork_asm+0x11/0x20 [ 6017.936527] [ 6045.572046] watchdog: BUG: soft lockup - CPU#7 stuck for 3513s! [kworker/7:4:78808] [ 6045.579695] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6045.635145] irq event stamp: 8812836 [ 6045.638715] hardirqs last enabled at (8812835): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6045.648793] hardirqs last disabled at (8812836): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6045.658432] softirqs last enabled at (8812756): [] __do_softirq+0x2be/0x371 [ 6045.667037] softirqs last disabled at (8812751): [] irq_exit_rcu+0xb0/0xd0 [ 6045.675463] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6045.684839] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6045.692310] Workqueue: events delayed_work [ceph] [ 6045.697035] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6045.701384] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6045.720122] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6045.725338] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6045.732463] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6045.739588] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6045.746712] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6045.753835] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6045.760959] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6045.769037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6045.774776] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6045.781900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6045.789023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6045.796148] Call Trace: [ 6045.798595] [ 6045.800606] ? watchdog_timer_fn+0x1b4/0x220 [ 6045.804876] ? __hrtimer_run_queues+0x1d3/0x360 [ 6045.809410] ? hrtimer_interrupt+0x104/0x240 [ 6045.813690] ? __local_bh_enable+0x50/0x60 [ 6045.817782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6045.823007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6045.827963] [ 6045.830062] [ 6045.832158] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6045.837466] ? lock_acquire+0xf0/0x2b0 [ 6045.841217] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6045.846624] ? lock_release+0xc5/0x260 [ 6045.850369] _raw_spin_lock+0x30/0x40 [ 6045.854032] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6045.859433] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6045.864661] delayed_work+0x1d8/0x280 [ceph] [ 6045.868943] ? process_scheduled_works+0x22a/0x510 [ 6045.873734] process_scheduled_works+0x2a4/0x510 [ 6045.878349] worker_thread+0x188/0x340 [ 6045.882097] ? rescuer_thread+0x310/0x310 [ 6045.886099] kthread+0xf5/0x130 [ 6045.889238] ? kthread_complete_and_exit+0x20/0x20 [ 6045.894024] ret_from_fork+0x31/0x40 [ 6045.897602] ? kthread_complete_and_exit+0x20/0x20 [ 6045.902385] ret_from_fork_asm+0x11/0x20 [ 6045.906308] [ 6073.572536] watchdog: BUG: soft lockup - CPU#7 stuck for 3539s! [kworker/7:4:78808] [ 6073.580191] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6073.635642] irq event stamp: 8873950 [ 6073.639212] hardirqs last enabled at (8873949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6073.649291] hardirqs last disabled at (8873950): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6073.658929] softirqs last enabled at (8873948): [] __do_softirq+0x2be/0x371 [ 6073.667528] softirqs last disabled at (8873933): [] irq_exit_rcu+0xb0/0xd0 [ 6073.675960] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6073.685336] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6073.692810] Workqueue: events delayed_work [ceph] [ 6073.697531] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6073.701883] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6073.720621] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 6073.725837] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 6073.732960] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 6073.740086] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 6073.747209] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6073.754335] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6073.761458] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6073.769534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6073.775273] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6073.782396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6073.789521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6073.796646] Call Trace: [ 6073.799091] [ 6073.801101] ? watchdog_timer_fn+0x1b4/0x220 [ 6073.805365] ? __hrtimer_run_queues+0x1d3/0x360 [ 6073.809901] ? hrtimer_interrupt+0x104/0x240 [ 6073.814170] ? __local_bh_enable+0x50/0x60 [ 6073.818264] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6073.823488] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6073.828443] [ 6073.830540] [ 6073.832639] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6073.837946] ? lock_acquire+0xf0/0x2b0 [ 6073.841696] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6073.846409] ? lock_release+0xc5/0x260 [ 6073.850155] _raw_spin_lock+0x30/0x40 [ 6073.853812] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6073.858431] ceph_check_caps+0x63/0xae0 [ceph] [ 6073.862883] ? __lock_acquire+0xbfa/0x1e20 [ 6073.866981] ? lock_acquire+0xcb/0x2b0 [ 6073.870729] ? igrab+0x1d/0x50 [ 6073.873779] ? find_held_lock+0x2d/0x90 [ 6073.877610] ? find_held_lock+0x2d/0x90 [ 6073.881443] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6073.886840] ? lock_release+0xc5/0x260 [ 6073.890587] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6073.895813] delayed_work+0x1d8/0x280 [ceph] [ 6073.900095] ? process_scheduled_works+0x22a/0x510 [ 6073.904884] process_scheduled_works+0x2a4/0x510 [ 6073.909502] worker_thread+0x188/0x340 [ 6073.913250] ? rescuer_thread+0x310/0x310 [ 6073.917253] kthread+0xf5/0x130 [ 6073.920392] ? kthread_complete_and_exit+0x20/0x20 [ 6073.925176] ret_from_fork+0x31/0x40 [ 6073.928754] ? kthread_complete_and_exit+0x20/0x20 [ 6073.933537] ret_from_fork_asm+0x11/0x20 [ 6073.937461] [ 6101.573028] watchdog: BUG: soft lockup - CPU#7 stuck for 3565s! [kworker/7:4:78808] [ 6101.580682] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6101.636130] irq event stamp: 8936960 [ 6101.639702] hardirqs last enabled at (8936959): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6101.649782] hardirqs last disabled at (8936960): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6101.659427] softirqs last enabled at (8936928): [] __do_softirq+0x2be/0x371 [ 6101.668024] softirqs last disabled at (8936915): [] irq_exit_rcu+0xb0/0xd0 [ 6101.676458] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6101.685836] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6101.693309] Workqueue: events delayed_work [ceph] [ 6101.698030] RIP: 0010:lock_acquire+0x85/0x2b0 [ 6101.702380] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [ 6101.721118] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [ 6101.726334] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [ 6101.733458] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [ 6101.740582] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 6101.747708] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6101.754831] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6101.761956] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6101.770032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6101.775771] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6101.782895] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6101.790018] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6101.797141] Call Trace: [ 6101.799589] [ 6101.801599] ? watchdog_timer_fn+0x1b4/0x220 [ 6101.805864] ? __hrtimer_run_queues+0x1d3/0x360 [ 6101.810390] ? hrtimer_interrupt+0x104/0x240 [ 6101.814659] ? __local_bh_enable+0x50/0x60 [ 6101.818751] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6101.823977] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6101.828934] [ 6101.831029] [ 6101.833128] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6101.838436] ? lock_acquire+0x85/0x2b0 [ 6101.842187] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6101.847595] _raw_spin_lock+0x30/0x40 [ 6101.851261] ? igrab+0x1d/0x50 [ 6101.854317] igrab+0x1d/0x50 [ 6101.857198] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6101.862347] delayed_work+0x1d8/0x280 [ceph] [ 6101.866627] ? process_scheduled_works+0x22a/0x510 [ 6101.871419] process_scheduled_works+0x2a4/0x510 [ 6101.876034] worker_thread+0x188/0x340 [ 6101.879783] ? rescuer_thread+0x310/0x310 [ 6101.883784] kthread+0xf5/0x130 [ 6101.886923] ? kthread_complete_and_exit+0x20/0x20 [ 6101.891706] ret_from_fork+0x31/0x40 [ 6101.895279] ? kthread_complete_and_exit+0x20/0x20 [ 6101.900062] ret_from_fork_asm+0x11/0x20 [ 6101.903985] [ 6129.573528] watchdog: BUG: soft lockup - CPU#7 stuck for 3591s! [kworker/7:4:78808] [ 6129.581180] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6129.636629] irq event stamp: 8998816 [ 6129.640198] hardirqs last enabled at (8998815): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6129.650278] hardirqs last disabled at (8998816): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6129.659916] softirqs last enabled at (8998794): [] __do_softirq+0x2be/0x371 [ 6129.668521] softirqs last disabled at (8998789): [] irq_exit_rcu+0xb0/0xd0 [ 6129.676948] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6129.686324] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6129.693795] Workqueue: events delayed_work [ceph] [ 6129.698520] RIP: 0010:lock_release+0x14e/0x260 [ 6129.702966] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 6129.721702] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [ 6129.726918] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [ 6129.734044] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [ 6129.741167] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 6129.748292] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 6129.755417] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [ 6129.762539] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6129.770618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6129.776354] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6129.783479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6129.790604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6129.797726] Call Trace: [ 6129.800174] [ 6129.802184] ? watchdog_timer_fn+0x1b4/0x220 [ 6129.806455] ? __hrtimer_run_queues+0x1d3/0x360 [ 6129.810983] ? hrtimer_interrupt+0x104/0x240 [ 6129.815252] ? __local_bh_enable+0x50/0x60 [ 6129.819344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6129.824568] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6129.829526] [ 6129.831623] [ 6129.833722] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6129.839025] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6129.844435] ? lock_release+0x14e/0x260 [ 6129.848265] _raw_spin_unlock+0x1b/0x40 [ 6129.852105] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6129.857332] delayed_work+0x1d8/0x280 [ceph] [ 6129.861613] ? process_scheduled_works+0x22a/0x510 [ 6129.866402] process_scheduled_works+0x2a4/0x510 [ 6129.871020] worker_thread+0x188/0x340 [ 6129.874768] ? rescuer_thread+0x310/0x310 [ 6129.878770] kthread+0xf5/0x130 [ 6129.881909] ? kthread_complete_and_exit+0x20/0x20 [ 6129.886692] ret_from_fork+0x31/0x40 [ 6129.890264] ? kthread_complete_and_exit+0x20/0x20 [ 6129.895047] ret_from_fork_asm+0x11/0x20 [ 6129.898980] [ 6157.574004] watchdog: BUG: soft lockup - CPU#7 stuck for 3617s! [kworker/7:4:78808] [ 6157.581658] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6157.637109] irq event stamp: 9061166 [ 6157.640679] hardirqs last enabled at (9061165): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6157.650759] hardirqs last disabled at (9061166): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6157.660397] softirqs last enabled at (9061134): [] __do_softirq+0x2be/0x371 [ 6157.669003] softirqs last disabled at (9061129): [] irq_exit_rcu+0xb0/0xd0 [ 6157.677436] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6157.686814] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6157.694285] Workqueue: events delayed_work [ceph] [ 6157.699007] RIP: 0010:lock_release+0x14e/0x260 [ 6157.703445] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 6157.722183] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [ 6157.727401] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [ 6157.734524] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [ 6157.741648] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 6157.748771] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 6157.755897] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [ 6157.763020] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6157.771099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6157.776835] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6157.783960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6157.791084] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6157.798207] Call Trace: [ 6157.800655] [ 6157.802665] ? watchdog_timer_fn+0x1b4/0x220 [ 6157.806936] ? __hrtimer_run_queues+0x1d3/0x360 [ 6157.811463] ? hrtimer_interrupt+0x104/0x240 [ 6157.815733] ? __local_bh_enable+0x50/0x60 [ 6157.819826] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6157.825050] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6157.830005] [ 6157.832103] [ 6157.834201] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6157.839505] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6157.844907] ? lock_release+0x14e/0x260 [ 6157.848739] _raw_spin_unlock+0x1b/0x40 [ 6157.852576] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6157.857805] delayed_work+0x1d8/0x280 [ceph] [ 6157.862085] ? process_scheduled_works+0x22a/0x510 [ 6157.866874] process_scheduled_works+0x2a4/0x510 [ 6157.871490] worker_thread+0x188/0x340 [ 6157.875241] ? rescuer_thread+0x310/0x310 [ 6157.879241] kthread+0xf5/0x130 [ 6157.882382] ? kthread_complete_and_exit+0x20/0x20 [ 6157.887174] ret_from_fork+0x31/0x40 [ 6157.890754] ? kthread_complete_and_exit+0x20/0x20 [ 6157.895535] ret_from_fork_asm+0x11/0x20 [ 6157.899459] [ 6185.574501] watchdog: BUG: soft lockup - CPU#7 stuck for 3643s! [kworker/7:4:78808] [ 6185.582148] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6185.637597] irq event stamp: 9123384 [ 6185.641169] hardirqs last enabled at (9123383): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6185.651247] hardirqs last disabled at (9123384): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6185.660884] softirqs last enabled at (9123352): [] __do_softirq+0x2be/0x371 [ 6185.669483] softirqs last disabled at (9123347): [] irq_exit_rcu+0xb0/0xd0 [ 6185.677918] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6185.687292] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6185.694766] Workqueue: events delayed_work [ceph] [ 6185.699490] RIP: 0010:lock_release+0x14e/0x260 [ 6185.703934] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 6185.722670] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [ 6185.727889] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [ 6185.735012] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [ 6185.742136] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 6185.749262] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 6185.756385] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [ 6185.763509] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6185.771586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6185.777325] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6185.784450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6185.791573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6185.798696] Call Trace: [ 6185.801143] [ 6185.803154] ? watchdog_timer_fn+0x1b4/0x220 [ 6185.807425] ? __hrtimer_run_queues+0x1d3/0x360 [ 6185.811951] ? hrtimer_interrupt+0x104/0x240 [ 6185.816213] ? __local_bh_enable+0x50/0x60 [ 6185.820307] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6185.825530] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6185.830486] [ 6185.832584] [ 6185.834683] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6185.839987] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6185.844702] ? lock_release+0x14e/0x260 [ 6185.848533] ? lock_release+0xc5/0x260 [ 6185.852277] _raw_spin_unlock+0x1b/0x40 [ 6185.856117] ceph_check_caps+0x26b/0xae0 [ceph] [ 6185.860658] ? lock_acquire+0xcb/0x2b0 [ 6185.864400] ? igrab+0x1d/0x50 [ 6185.867451] ? find_held_lock+0x2d/0x90 [ 6185.871282] ? find_held_lock+0x2d/0x90 [ 6185.875113] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6185.880511] ? lock_release+0xc5/0x260 [ 6185.884258] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6185.889484] delayed_work+0x1d8/0x280 [ceph] [ 6185.893768] ? process_scheduled_works+0x22a/0x510 [ 6185.898556] process_scheduled_works+0x2a4/0x510 [ 6185.903173] worker_thread+0x188/0x340 [ 6185.906922] ? rescuer_thread+0x310/0x310 [ 6185.910925] kthread+0xf5/0x130 [ 6185.914063] ? kthread_complete_and_exit+0x20/0x20 [ 6185.918845] ret_from_fork+0x31/0x40 [ 6185.922418] ? kthread_complete_and_exit+0x20/0x20 [ 6185.927200] ret_from_fork_asm+0x11/0x20 [ 6185.931124] [ 6213.574987] watchdog: BUG: soft lockup - CPU#7 stuck for 3669s! [kworker/7:4:78808] [ 6213.582637] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6213.638087] irq event stamp: 9186590 [ 6213.641657] hardirqs last enabled at (9186589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6213.651738] hardirqs last disabled at (9186590): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6213.661374] softirqs last enabled at (9186494): [] __do_softirq+0x2be/0x371 [ 6213.669972] softirqs last disabled at (9186489): [] irq_exit_rcu+0xb0/0xd0 [ 6213.678407] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6213.687783] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6213.695255] Workqueue: events delayed_work [ceph] [ 6213.699977] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6213.704328] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6213.723066] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 6213.728284] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 6213.735406] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 6213.742531] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 6213.749656] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6213.756778] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6213.763902] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6213.771981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6213.777720] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6213.784841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6213.791967] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6213.799091] Call Trace: [ 6213.801537] [ 6213.803547] ? watchdog_timer_fn+0x1b4/0x220 [ 6213.807811] ? __hrtimer_run_queues+0x1d3/0x360 [ 6213.812346] ? hrtimer_interrupt+0x104/0x240 [ 6213.816616] ? __local_bh_enable+0x50/0x60 [ 6213.820708] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6213.825933] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6213.830889] [ 6213.832986] [ 6213.835085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6213.840391] ? lock_acquire+0xf0/0x2b0 [ 6213.844133] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6213.848847] ? lock_release+0xc5/0x260 [ 6213.852592] _raw_spin_lock+0x30/0x40 [ 6213.856250] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6213.860868] ceph_check_caps+0x63/0xae0 [ceph] [ 6213.865320] ? __lock_acquire+0xbfa/0x1e20 [ 6213.869418] ? lock_acquire+0xcb/0x2b0 [ 6213.873165] ? igrab+0x1d/0x50 [ 6213.876216] ? find_held_lock+0x2d/0x90 [ 6213.880047] ? find_held_lock+0x2d/0x90 [ 6213.883877] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6213.889279] ? lock_release+0xc5/0x260 [ 6213.893023] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6213.898250] delayed_work+0x1d8/0x280 [ceph] [ 6213.902532] ? process_scheduled_works+0x22a/0x510 [ 6213.907322] process_scheduled_works+0x2a4/0x510 [ 6213.911937] worker_thread+0x188/0x340 [ 6213.915688] ? rescuer_thread+0x310/0x310 [ 6213.919698] kthread+0xf5/0x130 [ 6213.922838] ? kthread_complete_and_exit+0x20/0x20 [ 6213.927620] ret_from_fork+0x31/0x40 [ 6213.931200] ? kthread_complete_and_exit+0x20/0x20 [ 6213.935984] ret_from_fork_asm+0x11/0x20 [ 6213.939915] [ 6241.575491] watchdog: BUG: soft lockup - CPU#7 stuck for 3695s! [kworker/7:4:78808] [ 6241.583144] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6241.638593] irq event stamp: 9248700 [ 6241.642164] hardirqs last enabled at (9248699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6241.652243] hardirqs last disabled at (9248700): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6241.661880] softirqs last enabled at (9248620): [] __do_softirq+0x2be/0x371 [ 6241.670479] softirqs last disabled at (9248615): [] irq_exit_rcu+0xb0/0xd0 [ 6241.678913] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6241.688290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6241.695762] Workqueue: events delayed_work [ceph] [ 6241.700485] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6241.704844] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6241.723581] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6241.728797] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6241.735922] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6241.743047] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6241.750170] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6241.757295] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6241.764417] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6241.772495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6241.778233] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6241.785357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6241.792482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6241.799605] Call Trace: [ 6241.802053] [ 6241.804064] ? watchdog_timer_fn+0x1b4/0x220 [ 6241.808335] ? __hrtimer_run_queues+0x1d3/0x360 [ 6241.812870] ? hrtimer_interrupt+0x104/0x240 [ 6241.817139] ? __local_bh_enable+0x50/0x60 [ 6241.821233] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6241.826457] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6241.831413] [ 6241.833511] [ 6241.835609] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6241.840917] ? lock_acquire+0xf0/0x2b0 [ 6241.844667] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6241.850073] ? lock_release+0xc5/0x260 [ 6241.853819] _raw_spin_lock+0x30/0x40 [ 6241.857482] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6241.862884] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6241.868113] delayed_work+0x1d8/0x280 [ceph] [ 6241.872392] ? process_scheduled_works+0x22a/0x510 [ 6241.877182] process_scheduled_works+0x2a4/0x510 [ 6241.881798] worker_thread+0x188/0x340 [ 6241.885548] ? rescuer_thread+0x310/0x310 [ 6241.889551] kthread+0xf5/0x130 [ 6241.892689] ? kthread_complete_and_exit+0x20/0x20 [ 6241.897472] ret_from_fork+0x31/0x40 [ 6241.901045] ? kthread_complete_and_exit+0x20/0x20 [ 6241.905828] ret_from_fork_asm+0x11/0x20 [ 6241.909750] [ 6269.575978] watchdog: BUG: soft lockup - CPU#7 stuck for 3721s! [kworker/7:4:78808] [ 6269.583635] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6269.639083] irq event stamp: 9310020 [ 6269.642654] hardirqs last enabled at (9310019): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6269.652734] hardirqs last disabled at (9310020): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6269.662378] softirqs last enabled at (9310006): [] __do_softirq+0x2be/0x371 [ 6269.670978] softirqs last disabled at (9310001): [] irq_exit_rcu+0xb0/0xd0 [ 6269.679411] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6269.688788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6269.696259] Workqueue: events delayed_work [ceph] [ 6269.700982] RIP: 0010:do_raw_spin_lock+0xd/0xc0 [ 6269.705506] Code: 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 <48> 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 [ 6269.724242] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [ 6269.729460] RAX: 00000000dead4ead RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 6269.736585] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 6269.743709] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 6269.750832] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 6269.757957] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 6269.765082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6269.773161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6269.778904] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6269.786029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6269.793155] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6269.800287] Call Trace: [ 6269.802732] [ 6269.804741] ? watchdog_timer_fn+0x1b4/0x220 [ 6269.809004] ? __hrtimer_run_queues+0x1d3/0x360 [ 6269.813532] ? hrtimer_interrupt+0x104/0x240 [ 6269.817791] ? __local_bh_enable+0x50/0x60 [ 6269.821887] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6269.827110] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6269.832066] [ 6269.834165] [ 6269.836263] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6269.841578] ? do_raw_spin_lock+0xd/0xc0 [ 6269.845501] igrab+0x1d/0x50 [ 6269.848382] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6269.853539] delayed_work+0x1d8/0x280 [ceph] [ 6269.857819] ? process_scheduled_works+0x22a/0x510 [ 6269.862608] process_scheduled_works+0x2a4/0x510 [ 6269.867223] worker_thread+0x188/0x340 [ 6269.870973] ? rescuer_thread+0x310/0x310 [ 6269.874978] kthread+0xf5/0x130 [ 6269.878125] ? kthread_complete_and_exit+0x20/0x20 [ 6269.882916] ret_from_fork+0x31/0x40 [ 6269.886494] ? kthread_complete_and_exit+0x20/0x20 [ 6269.891280] ret_from_fork_asm+0x11/0x20 [ 6269.895202] [ 6297.576469] watchdog: BUG: soft lockup - CPU#7 stuck for 3748s! [kworker/7:4:78808] [ 6297.584123] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6297.639572] irq event stamp: 9370766 [ 6297.643142] hardirqs last enabled at (9370765): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6297.653223] hardirqs last disabled at (9370766): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6297.662867] softirqs last enabled at (9370722): [] __do_softirq+0x2be/0x371 [ 6297.671464] softirqs last disabled at (9370717): [] irq_exit_rcu+0xb0/0xd0 [ 6297.679892] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6297.689268] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6297.696739] Workqueue: events delayed_work [ceph] [ 6297.701463] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6297.705814] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6297.724549] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6297.729767] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6297.736891] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6297.744017] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6297.751139] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6297.758265] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6297.765388] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6297.773465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6297.779203] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6297.786326] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6297.793452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6297.800576] Call Trace: [ 6297.803022] [ 6297.805032] ? watchdog_timer_fn+0x1b4/0x220 [ 6297.809295] ? __hrtimer_run_queues+0x1d3/0x360 [ 6297.813830] ? hrtimer_interrupt+0x104/0x240 [ 6297.818091] ? __local_bh_enable+0x50/0x60 [ 6297.822185] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6297.827408] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6297.832366] [ 6297.834463] [ 6297.836561] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6297.841868] ? lock_acquire+0xf0/0x2b0 [ 6297.845618] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6297.851027] ? lock_release+0xc5/0x260 [ 6297.854771] _raw_spin_lock+0x30/0x40 [ 6297.858428] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6297.863827] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6297.869057] delayed_work+0x1d8/0x280 [ceph] [ 6297.873338] ? process_scheduled_works+0x22a/0x510 [ 6297.878125] process_scheduled_works+0x2a4/0x510 [ 6297.882742] worker_thread+0x188/0x340 [ 6297.886491] ? rescuer_thread+0x310/0x310 [ 6297.890493] kthread+0xf5/0x130 [ 6297.893632] ? kthread_complete_and_exit+0x20/0x20 [ 6297.898417] ret_from_fork+0x31/0x40 [ 6297.901997] ? kthread_complete_and_exit+0x20/0x20 [ 6297.906779] ret_from_fork_asm+0x11/0x20 [ 6297.910711] [ 6325.576964] watchdog: BUG: soft lockup - CPU#7 stuck for 3774s! [kworker/7:4:78808] [ 6325.584619] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6325.640071] irq event stamp: 9432188 [ 6325.643641] hardirqs last enabled at (9432187): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6325.653720] hardirqs last disabled at (9432188): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6325.663365] softirqs last enabled at (9432174): [] __do_softirq+0x2be/0x371 [ 6325.671962] softirqs last disabled at (9432169): [] irq_exit_rcu+0xb0/0xd0 [ 6325.680390] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6325.689765] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6325.697237] Workqueue: events delayed_work [ceph] [ 6325.701959] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6325.706310] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6325.725048] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 6325.730266] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 6325.737390] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 6325.744513] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 6325.751637] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6325.758763] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6325.765886] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6325.773965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6325.779700] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6325.786824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6325.793950] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6325.801073] Call Trace: [ 6325.803521] [ 6325.805532] ? watchdog_timer_fn+0x1b4/0x220 [ 6325.809803] ? __hrtimer_run_queues+0x1d3/0x360 [ 6325.814337] ? hrtimer_interrupt+0x104/0x240 [ 6325.818599] ? __local_bh_enable+0x50/0x60 [ 6325.822690] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6325.827916] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6325.832871] [ 6325.834969] [ 6325.837067] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6325.842375] ? lock_acquire+0xf0/0x2b0 [ 6325.846126] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6325.851534] _raw_spin_lock+0x30/0x40 [ 6325.855199] ? igrab+0x1d/0x50 [ 6325.858248] igrab+0x1d/0x50 [ 6325.861128] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6325.866278] delayed_work+0x1d8/0x280 [ceph] [ 6325.870559] ? process_scheduled_works+0x22a/0x510 [ 6325.875348] process_scheduled_works+0x2a4/0x510 [ 6325.879964] worker_thread+0x188/0x340 [ 6325.883713] ? rescuer_thread+0x310/0x310 [ 6325.887716] kthread+0xf5/0x130 [ 6325.890855] ? kthread_complete_and_exit+0x20/0x20 [ 6325.895638] ret_from_fork+0x31/0x40 [ 6325.899218] ? kthread_complete_and_exit+0x20/0x20 [ 6325.904000] ret_from_fork_asm+0x11/0x20 [ 6325.907924] [ 6353.577445] watchdog: BUG: soft lockup - CPU#7 stuck for 3800s! [kworker/7:4:78808] [ 6353.585100] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6353.640551] irq event stamp: 9493842 [ 6353.644120] hardirqs last enabled at (9493841): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6353.654200] hardirqs last disabled at (9493842): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6353.663839] softirqs last enabled at (9493820): [] __do_softirq+0x2be/0x371 [ 6353.672445] softirqs last disabled at (9493815): [] irq_exit_rcu+0xb0/0xd0 [ 6353.680879] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6353.690254] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6353.697725] Workqueue: events delayed_work [ceph] [ 6353.702450] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6353.706800] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6353.725536] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 6353.730755] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 6353.737879] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 6353.745002] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 6353.752126] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6353.759250] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6353.766375] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6353.774452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6353.780191] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6353.787315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6353.794438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6353.801561] Call Trace: [ 6353.804007] [ 6353.806019] ? watchdog_timer_fn+0x1b4/0x220 [ 6353.810281] ? __hrtimer_run_queues+0x1d3/0x360 [ 6353.814808] ? hrtimer_interrupt+0x104/0x240 [ 6353.819069] ? __local_bh_enable+0x50/0x60 [ 6353.823162] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6353.828387] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6353.833344] [ 6353.835440] [ 6353.837539] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6353.842847] ? lock_acquire+0xf0/0x2b0 [ 6353.846597] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6353.852006] _raw_spin_lock+0x30/0x40 [ 6353.855672] ? igrab+0x1d/0x50 [ 6353.858720] igrab+0x1d/0x50 [ 6353.861599] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6353.866741] delayed_work+0x1d8/0x280 [ceph] [ 6353.871021] ? process_scheduled_works+0x22a/0x510 [ 6353.875810] process_scheduled_works+0x2a4/0x510 [ 6353.880427] worker_thread+0x188/0x340 [ 6353.884176] ? rescuer_thread+0x310/0x310 [ 6353.888178] kthread+0xf5/0x130 [ 6353.891318] ? kthread_complete_and_exit+0x20/0x20 [ 6353.896100] ret_from_fork+0x31/0x40 [ 6353.899681] ? kthread_complete_and_exit+0x20/0x20 [ 6353.904463] ret_from_fork_asm+0x11/0x20 [ 6353.908388] [ 6381.577949] watchdog: BUG: soft lockup - CPU#7 stuck for 3826s! [kworker/7:4:78808] [ 6381.585598] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6381.641048] irq event stamp: 9555476 [ 6381.644619] hardirqs last enabled at (9555475): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6381.654697] hardirqs last disabled at (9555476): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6381.664336] softirqs last enabled at (9555454): [] __do_softirq+0x2be/0x371 [ 6381.672941] softirqs last disabled at (9555449): [] irq_exit_rcu+0xb0/0xd0 [ 6381.681368] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6381.690742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6381.698214] Workqueue: events delayed_work [ceph] [ 6381.702938] RIP: 0010:ceph_check_caps+0x2f5/0xae0 [ceph] [ 6381.708259] Code: ff ff 48 85 c0 0f 84 55 01 00 00 48 39 d8 0f 84 4c 01 00 00 44 8b 60 40 41 f7 d4 44 23 64 24 20 41 8b 6f 38 f7 d5 41 23 6f 3c <0f> 1f 44 00 00 85 ed 74 1c 41 85 ec 0f 84 0b 06 00 00 41 0f b7 45 [ 6381.726997] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [ 6381.732214] RAX: ffff88810da9e000 RBX: ffff88810da9e000 RCX: ffff88817f867000 [ 6381.739338] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34b20 [ 6381.746463] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 6381.753588] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 6381.760710] R13: ffff8881a1d34478 R14: 0000000000000000 R15: ffff88810da9e008 [ 6381.767836] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6381.775911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6381.781650] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6381.788775] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6381.795897] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6381.803022] Call Trace: [ 6381.805470] [ 6381.807478] ? watchdog_timer_fn+0x1b4/0x220 [ 6381.811742] ? __hrtimer_run_queues+0x1d3/0x360 [ 6381.816267] ? hrtimer_interrupt+0x104/0x240 [ 6381.820528] ? __local_bh_enable+0x50/0x60 [ 6381.824622] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6381.829846] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6381.834804] [ 6381.836901] [ 6381.838997] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6381.844306] ? ceph_check_caps+0x2f5/0xae0 [ceph] [ 6381.849027] ? lock_acquire+0xcb/0x2b0 [ 6381.852778] ? igrab+0x1d/0x50 [ 6381.855828] ? find_held_lock+0x2d/0x90 [ 6381.859659] ? find_held_lock+0x2d/0x90 [ 6381.863491] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6381.868890] ? lock_release+0xc5/0x260 [ 6381.872636] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6381.877862] delayed_work+0x1d8/0x280 [ceph] [ 6381.882145] ? process_scheduled_works+0x22a/0x510 [ 6381.886933] process_scheduled_works+0x2a4/0x510 [ 6381.891549] worker_thread+0x188/0x340 [ 6381.895299] ? rescuer_thread+0x310/0x310 [ 6381.899302] kthread+0xf5/0x130 [ 6381.902441] ? kthread_complete_and_exit+0x20/0x20 [ 6381.907225] ret_from_fork+0x31/0x40 [ 6381.910803] ? kthread_complete_and_exit+0x20/0x20 [ 6381.915588] ret_from_fork_asm+0x11/0x20 [ 6381.919519] [ 6409.578437] watchdog: BUG: soft lockup - CPU#7 stuck for 3852s! [kworker/7:4:78808] [ 6409.586089] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6409.641536] irq event stamp: 9616482 [ 6409.645108] hardirqs last enabled at (9616481): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6409.655187] hardirqs last disabled at (9616482): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6409.664825] softirqs last enabled at (9616356): [] __do_softirq+0x2be/0x371 [ 6409.673430] softirqs last disabled at (9616351): [] irq_exit_rcu+0xb0/0xd0 [ 6409.681855] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6409.691233] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6409.698705] Workqueue: events delayed_work [ceph] [ 6409.703428] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6409.707778] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6409.726516] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 6409.731733] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 6409.738857] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 6409.745981] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 6409.753105] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6409.760229] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6409.767352] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6409.775430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6409.781168] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6409.788292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6409.795416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6409.802540] Call Trace: [ 6409.804987] [ 6409.806998] ? watchdog_timer_fn+0x1b4/0x220 [ 6409.811270] ? __hrtimer_run_queues+0x1d3/0x360 [ 6409.815805] ? hrtimer_interrupt+0x104/0x240 [ 6409.820074] ? __local_bh_enable+0x50/0x60 [ 6409.824166] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6409.829390] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6409.834348] [ 6409.836445] [ 6409.838543] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6409.843851] ? lock_acquire+0xf0/0x2b0 [ 6409.847602] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6409.853009] _raw_spin_lock+0x30/0x40 [ 6409.856674] ? igrab+0x1d/0x50 [ 6409.859723] igrab+0x1d/0x50 [ 6409.862604] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6409.867753] delayed_work+0x1d8/0x280 [ceph] [ 6409.872034] ? process_scheduled_works+0x22a/0x510 [ 6409.876824] process_scheduled_works+0x2a4/0x510 [ 6409.881438] worker_thread+0x188/0x340 [ 6409.885189] ? rescuer_thread+0x310/0x310 [ 6409.889191] kthread+0xf5/0x130 [ 6409.892332] ? kthread_complete_and_exit+0x20/0x20 [ 6409.897123] ret_from_fork+0x31/0x40 [ 6409.900702] ? kthread_complete_and_exit+0x20/0x20 [ 6409.905486] ret_from_fork_asm+0x11/0x20 [ 6409.909418] [ 6437.578918] watchdog: BUG: soft lockup - CPU#7 stuck for 3878s! [kworker/7:4:78808] [ 6437.586568] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6437.642018] irq event stamp: 9677720 [ 6437.645588] hardirqs last enabled at (9677719): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6437.655668] hardirqs last disabled at (9677720): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6437.665315] softirqs last enabled at (9677698): [] __do_softirq+0x2be/0x371 [ 6437.673920] softirqs last disabled at (9677693): [] irq_exit_rcu+0xb0/0xd0 [ 6437.682353] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6437.691731] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6437.699203] Workqueue: events delayed_work [ceph] [ 6437.703925] RIP: 0010:lock_acquire+0x4a/0x2b0 [ 6437.708277] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [ 6437.727012] RSP: 0018:ffffc90002117b48 EFLAGS: 00000247 [ 6437.732230] RAX: 0000000000000007 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 6437.739354] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34938 [ 6437.746478] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 6437.753604] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6437.760726] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6437.767851] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6437.775928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6437.781665] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6437.788789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6437.795914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6437.803037] Call Trace: [ 6437.805485] [ 6437.807495] ? watchdog_timer_fn+0x1b4/0x220 [ 6437.811757] ? __hrtimer_run_queues+0x1d3/0x360 [ 6437.816283] ? hrtimer_interrupt+0x104/0x240 [ 6437.820545] ? __local_bh_enable+0x50/0x60 [ 6437.824639] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6437.829863] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6437.834821] [ 6437.836916] [ 6437.839015] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6437.844322] ? lock_acquire+0x4a/0x2b0 [ 6437.848072] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6437.852786] ? lock_release+0xc5/0x260 [ 6437.856531] _raw_spin_lock+0x30/0x40 [ 6437.860190] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6437.864817] ceph_check_caps+0x63/0xae0 [ceph] [ 6437.869267] ? __lock_acquire+0xbfa/0x1e20 [ 6437.873365] ? lock_acquire+0xcb/0x2b0 [ 6437.877113] ? igrab+0x1d/0x50 [ 6437.880164] ? find_held_lock+0x2d/0x90 [ 6437.883994] ? find_held_lock+0x2d/0x90 [ 6437.887826] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6437.893226] ? lock_release+0xc5/0x260 [ 6437.896972] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6437.902199] delayed_work+0x1d8/0x280 [ceph] [ 6437.906480] ? process_scheduled_works+0x22a/0x510 [ 6437.911270] process_scheduled_works+0x2a4/0x510 [ 6437.915884] worker_thread+0x188/0x340 [ 6437.919634] ? rescuer_thread+0x310/0x310 [ 6437.923639] kthread+0xf5/0x130 [ 6437.926777] ? kthread_complete_and_exit+0x20/0x20 [ 6437.931559] ret_from_fork+0x31/0x40 [ 6437.935130] ? kthread_complete_and_exit+0x20/0x20 [ 6437.939913] ret_from_fork_asm+0x11/0x20 [ 6437.943836] [ 6465.579409] watchdog: BUG: soft lockup - CPU#7 stuck for 3904s! [kworker/7:4:78808] [ 6465.587065] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6465.642516] irq event stamp: 9738964 [ 6465.646084] hardirqs last enabled at (9738963): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6465.656165] hardirqs last disabled at (9738964): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6465.665803] softirqs last enabled at (9738884): [] __do_softirq+0x2be/0x371 [ 6465.674410] softirqs last disabled at (9738879): [] irq_exit_rcu+0xb0/0xd0 [ 6465.682842] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6465.692220] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6465.699691] Workqueue: events delayed_work [ceph] [ 6465.704413] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6465.708766] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6465.727501] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6465.732719] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6465.739843] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6465.746967] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6465.754092] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6465.761216] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6465.768339] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6465.776418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6465.782154] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6465.789280] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6465.796403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6465.803527] Call Trace: [ 6465.805972] [ 6465.807985] ? watchdog_timer_fn+0x1b4/0x220 [ 6465.812257] ? __hrtimer_run_queues+0x1d3/0x360 [ 6465.816782] ? hrtimer_interrupt+0x104/0x240 [ 6465.821044] ? __local_bh_enable+0x50/0x60 [ 6465.825137] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6465.830360] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6465.835317] [ 6465.837415] [ 6465.839511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6465.844821] ? lock_acquire+0xf0/0x2b0 [ 6465.848571] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6465.853978] ? lock_release+0xc5/0x260 [ 6465.857724] _raw_spin_lock+0x30/0x40 [ 6465.861389] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6465.866786] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6465.872015] delayed_work+0x1d8/0x280 [ceph] [ 6465.876297] ? process_scheduled_works+0x22a/0x510 [ 6465.881087] process_scheduled_works+0x2a4/0x510 [ 6465.885703] worker_thread+0x188/0x340 [ 6465.889451] ? rescuer_thread+0x310/0x310 [ 6465.893455] kthread+0xf5/0x130 [ 6465.896594] ? kthread_complete_and_exit+0x20/0x20 [ 6465.901376] ret_from_fork+0x31/0x40 [ 6465.904947] ? kthread_complete_and_exit+0x20/0x20 [ 6465.909732] ret_from_fork_asm+0x11/0x20 [ 6465.913663] [ 6493.579901] watchdog: BUG: soft lockup - CPU#7 stuck for 3930s! [kworker/7:4:78808] [ 6493.587554] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6493.643004] irq event stamp: 9801244 [ 6493.646574] hardirqs last enabled at (9801243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6493.656654] hardirqs last disabled at (9801244): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6493.666292] softirqs last enabled at (9801230): [] __do_softirq+0x2be/0x371 [ 6493.674888] softirqs last disabled at (9801225): [] irq_exit_rcu+0xb0/0xd0 [ 6493.683315] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6493.692690] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6493.700163] Workqueue: events delayed_work [ceph] [ 6493.704885] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6493.709237] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6493.727975] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 6493.733191] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 6493.740316] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 6493.747440] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 6493.754564] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6493.761688] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6493.768811] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6493.776889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6493.782628] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6493.789750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6493.796875] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6493.803998] Call Trace: [ 6493.806446] [ 6493.808456] ? watchdog_timer_fn+0x1b4/0x220 [ 6493.812727] ? __hrtimer_run_queues+0x1d3/0x360 [ 6493.817253] ? hrtimer_interrupt+0x104/0x240 [ 6493.821515] ? __local_bh_enable+0x50/0x60 [ 6493.825607] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6493.830831] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6493.835790] [ 6493.837887] [ 6493.839985] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6493.845291] ? lock_acquire+0xf0/0x2b0 [ 6493.849034] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6493.853747] ? lock_release+0xc5/0x260 [ 6493.857493] _raw_spin_lock+0x30/0x40 [ 6493.861156] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6493.865775] ceph_check_caps+0x63/0xae0 [ceph] [ 6493.870227] ? __lock_acquire+0xbfa/0x1e20 [ 6493.874325] ? lock_acquire+0xcb/0x2b0 [ 6493.878066] ? igrab+0x1d/0x50 [ 6493.881118] ? find_held_lock+0x2d/0x90 [ 6493.884957] ? find_held_lock+0x2d/0x90 [ 6493.888788] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6493.894186] ? lock_release+0xc5/0x260 [ 6493.897932] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6493.903158] delayed_work+0x1d8/0x280 [ceph] [ 6493.907439] ? process_scheduled_works+0x22a/0x510 [ 6493.912221] process_scheduled_works+0x2a4/0x510 [ 6493.916837] worker_thread+0x188/0x340 [ 6493.920588] ? rescuer_thread+0x310/0x310 [ 6493.924597] kthread+0xf5/0x130 [ 6493.927737] ? kthread_complete_and_exit+0x20/0x20 [ 6493.932521] ret_from_fork+0x31/0x40 [ 6493.936101] ? kthread_complete_and_exit+0x20/0x20 [ 6493.940882] ret_from_fork_asm+0x11/0x20 [ 6493.944806] [ 6521.580391] watchdog: BUG: soft lockup - CPU#7 stuck for 3956s! [kworker/7:4:78808] [ 6521.588044] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6521.643492] irq event stamp: 9863624 [ 6521.647063] hardirqs last enabled at (9863623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6521.657143] hardirqs last disabled at (9863624): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6521.666780] softirqs last enabled at (9863544): [] __do_softirq+0x2be/0x371 [ 6521.675378] softirqs last disabled at (9863539): [] irq_exit_rcu+0xb0/0xd0 [ 6521.683812] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6521.693190] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6521.700660] Workqueue: events delayed_work [ceph] [ 6521.705382] RIP: 0010:lock_release+0x14e/0x260 [ 6521.709822] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 6521.728558] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [ 6521.733776] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [ 6521.740899] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [ 6521.748023] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 6521.755148] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 6521.762272] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [ 6521.769395] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6521.777473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6521.783211] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6521.790336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6521.797460] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6521.804582] Call Trace: [ 6521.807030] [ 6521.809040] ? watchdog_timer_fn+0x1b4/0x220 [ 6521.813312] ? __hrtimer_run_queues+0x1d3/0x360 [ 6521.817847] ? hrtimer_interrupt+0x104/0x240 [ 6521.822107] ? __local_bh_enable+0x50/0x60 [ 6521.826202] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6521.831424] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6521.836382] [ 6521.838479] [ 6521.840578] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6521.845882] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6521.850597] ? lock_release+0x14e/0x260 [ 6521.854427] ? lock_release+0xc5/0x260 [ 6521.858172] _raw_spin_unlock+0x1b/0x40 [ 6521.862011] ceph_check_caps+0x26b/0xae0 [ceph] [ 6521.866553] ? lock_acquire+0xcb/0x2b0 [ 6521.870295] ? igrab+0x1d/0x50 [ 6521.873345] ? find_held_lock+0x2d/0x90 [ 6521.877178] ? find_held_lock+0x2d/0x90 [ 6521.881008] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6521.886407] ? lock_release+0xc5/0x260 [ 6521.890152] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6521.895379] delayed_work+0x1d8/0x280 [ceph] [ 6521.899660] ? process_scheduled_works+0x22a/0x510 [ 6521.904450] process_scheduled_works+0x2a4/0x510 [ 6521.909068] worker_thread+0x188/0x340 [ 6521.912816] ? rescuer_thread+0x310/0x310 [ 6521.916818] kthread+0xf5/0x130 [ 6521.919958] ? kthread_complete_and_exit+0x20/0x20 [ 6521.924740] ret_from_fork+0x31/0x40 [ 6521.928312] ? kthread_complete_and_exit+0x20/0x20 [ 6521.933096] ret_from_fork_asm+0x11/0x20 [ 6521.937027] [ 6549.580883] watchdog: BUG: soft lockup - CPU#7 stuck for 3982s! [kworker/7:4:78808] [ 6549.588532] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6549.643981] irq event stamp: 9924308 [ 6549.647553] hardirqs last enabled at (9924307): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6549.657632] hardirqs last disabled at (9924308): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6549.667269] softirqs last enabled at (9924212): [] __do_softirq+0x2be/0x371 [ 6549.675875] softirqs last disabled at (9924207): [] irq_exit_rcu+0xb0/0xd0 [ 6549.684311] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6549.693694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6549.701166] Workqueue: events delayed_work [ceph] [ 6549.705891] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6549.710249] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6549.728986] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6549.734204] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6549.741327] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6549.748451] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6549.755575] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6549.762700] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6549.769824] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6549.777900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6549.783640] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6549.790763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6549.797888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6549.805012] Call Trace: [ 6549.807456] [ 6549.809469] ? watchdog_timer_fn+0x1b4/0x220 [ 6549.813740] ? __hrtimer_run_queues+0x1d3/0x360 [ 6549.818267] ? hrtimer_interrupt+0x104/0x240 [ 6549.822536] ? __local_bh_enable+0x50/0x60 [ 6549.826628] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6549.831852] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6549.836809] [ 6549.838906] [ 6549.841005] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6549.846313] ? lock_acquire+0xf0/0x2b0 [ 6549.850063] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6549.855470] ? lock_release+0xc5/0x260 [ 6549.859215] _raw_spin_lock+0x30/0x40 [ 6549.862872] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6549.868270] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6549.873501] delayed_work+0x1d8/0x280 [ceph] [ 6549.877782] ? process_scheduled_works+0x22a/0x510 [ 6549.882570] process_scheduled_works+0x2a4/0x510 [ 6549.887186] worker_thread+0x188/0x340 [ 6549.890934] ? rescuer_thread+0x310/0x310 [ 6549.894937] kthread+0xf5/0x130 [ 6549.898076] ? kthread_complete_and_exit+0x20/0x20 [ 6549.902862] ret_from_fork+0x31/0x40 [ 6549.906441] ? kthread_complete_and_exit+0x20/0x20 [ 6549.911224] ret_from_fork_asm+0x11/0x20 [ 6549.915155] [ 6577.581375] watchdog: BUG: soft lockup - CPU#7 stuck for 4008s! [kworker/7:4:78808] [ 6577.589028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6577.644478] irq event stamp: 9984730 [ 6577.648051] hardirqs last enabled at (9984729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6577.658128] hardirqs last disabled at (9984730): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6577.667765] softirqs last enabled at (9984716): [] __do_softirq+0x2be/0x371 [ 6577.676363] softirqs last disabled at (9984711): [] irq_exit_rcu+0xb0/0xd0 [ 6577.684798] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6577.694174] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6577.701648] Workqueue: events delayed_work [ceph] [ 6577.706370] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6577.710721] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6577.729456] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 6577.734674] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 6577.741799] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 6577.748921] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 6577.756046] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6577.763171] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6577.770295] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6577.778372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6577.784110] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6577.791235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6577.798359] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6577.805481] Call Trace: [ 6577.807928] [ 6577.809939] ? watchdog_timer_fn+0x1b4/0x220 [ 6577.814210] ? __hrtimer_run_queues+0x1d3/0x360 [ 6577.818736] ? hrtimer_interrupt+0x104/0x240 [ 6577.822997] ? __local_bh_enable+0x50/0x60 [ 6577.827091] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6577.832315] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6577.837274] [ 6577.839370] [ 6577.841468] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6577.846774] ? lock_acquire+0xf0/0x2b0 [ 6577.850517] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6577.855230] ? lock_release+0xc5/0x260 [ 6577.858976] _raw_spin_lock+0x30/0x40 [ 6577.862632] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6577.867251] ceph_check_caps+0x63/0xae0 [ceph] [ 6577.871702] ? __lock_acquire+0xbfa/0x1e20 [ 6577.875800] ? lock_acquire+0xcb/0x2b0 [ 6577.879549] ? igrab+0x1d/0x50 [ 6577.882600] ? find_held_lock+0x2d/0x90 [ 6577.886431] ? find_held_lock+0x2d/0x90 [ 6577.890261] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6577.895661] ? lock_release+0xc5/0x260 [ 6577.899407] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6577.904635] delayed_work+0x1d8/0x280 [ceph] [ 6577.908914] ? process_scheduled_works+0x22a/0x510 [ 6577.913706] process_scheduled_works+0x2a4/0x510 [ 6577.918321] worker_thread+0x188/0x340 [ 6577.922069] ? rescuer_thread+0x310/0x310 [ 6577.926072] kthread+0xf5/0x130 [ 6577.929211] ? kthread_complete_and_exit+0x20/0x20 [ 6577.933994] ret_from_fork+0x31/0x40 [ 6577.937576] ? kthread_complete_and_exit+0x20/0x20 [ 6577.942357] ret_from_fork_asm+0x11/0x20 [ 6577.946281] [ 6605.581865] watchdog: BUG: soft lockup - CPU#7 stuck for 4034s! [kworker/7:4:78808] [ 6605.589519] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6605.644968] irq event stamp: 10045652 [ 6605.648626] hardirqs last enabled at (10045651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6605.658791] hardirqs last disabled at (10045652): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6605.668526] softirqs last enabled at (10045572): [] __do_softirq+0x2be/0x371 [ 6605.677217] softirqs last disabled at (10045567): [] irq_exit_rcu+0xb0/0xd0 [ 6605.685736] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6605.695113] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6605.702585] Workqueue: events delayed_work [ceph] [ 6605.707310] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6605.711669] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6605.730404] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6605.735623] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6605.742748] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6605.749872] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6605.756996] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6605.764118] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6605.771243] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6605.779320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6605.785057] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6605.792184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6605.799307] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6605.806430] Call Trace: [ 6605.808878] [ 6605.810888] ? watchdog_timer_fn+0x1b4/0x220 [ 6605.815158] ? __hrtimer_run_queues+0x1d3/0x360 [ 6605.819686] ? hrtimer_interrupt+0x104/0x240 [ 6605.823955] ? __local_bh_enable+0x50/0x60 [ 6605.828048] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6605.833273] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6605.838228] [ 6605.840326] [ 6605.842424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6605.847732] ? lock_acquire+0xf0/0x2b0 [ 6605.851483] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6605.856890] ? lock_release+0xc5/0x260 [ 6605.860635] _raw_spin_lock+0x30/0x40 [ 6605.864300] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6605.869699] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6605.874927] delayed_work+0x1d8/0x280 [ceph] [ 6605.879208] ? process_scheduled_works+0x22a/0x510 [ 6605.883998] process_scheduled_works+0x2a4/0x510 [ 6605.888615] worker_thread+0x188/0x340 [ 6605.892364] ? rescuer_thread+0x310/0x310 [ 6605.896368] kthread+0xf5/0x130 [ 6605.899506] ? kthread_complete_and_exit+0x20/0x20 [ 6605.904297] ret_from_fork+0x31/0x40 [ 6605.907868] ? kthread_complete_and_exit+0x20/0x20 [ 6605.912652] ret_from_fork_asm+0x11/0x20 [ 6605.916584] [ 6633.582372] watchdog: BUG: soft lockup - CPU#7 stuck for 4061s! [kworker/7:4:78808] [ 6633.590025] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6633.645474] irq event stamp: 10107238 [ 6633.649130] hardirqs last enabled at (10107237): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6633.659297] hardirqs last disabled at (10107238): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6633.669021] softirqs last enabled at (10107224): [] __do_softirq+0x2be/0x371 [ 6633.677714] softirqs last disabled at (10107219): [] irq_exit_rcu+0xb0/0xd0 [ 6633.686234] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6633.695611] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6633.703083] Workqueue: events delayed_work [ceph] [ 6633.707806] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6633.712157] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6633.730893] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6633.736111] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6633.743234] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6633.750361] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6633.757483] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6633.764608] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6633.771731] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6633.779809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6633.785546] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6633.792672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6633.799796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6633.806919] Call Trace: [ 6633.809366] [ 6633.811376] ? watchdog_timer_fn+0x1b4/0x220 [ 6633.815640] ? __hrtimer_run_queues+0x1d3/0x360 [ 6633.820175] ? hrtimer_interrupt+0x104/0x240 [ 6633.824444] ? __local_bh_enable+0x50/0x60 [ 6633.828536] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6633.833760] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6633.838718] [ 6633.840816] [ 6633.842912] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6633.848221] ? lock_acquire+0xf0/0x2b0 [ 6633.851972] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6633.857379] ? lock_release+0xc5/0x260 [ 6633.861122] _raw_spin_lock+0x30/0x40 [ 6633.864779] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6633.870177] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6633.875408] delayed_work+0x1d8/0x280 [ceph] [ 6633.879689] ? process_scheduled_works+0x22a/0x510 [ 6633.884478] process_scheduled_works+0x2a4/0x510 [ 6633.889095] worker_thread+0x188/0x340 [ 6633.892843] ? rescuer_thread+0x310/0x310 [ 6633.896846] kthread+0xf5/0x130 [ 6633.899984] ? kthread_complete_and_exit+0x20/0x20 [ 6633.904768] ret_from_fork+0x31/0x40 [ 6633.908340] ? kthread_complete_and_exit+0x20/0x20 [ 6633.913124] ret_from_fork_asm+0x11/0x20 [ 6633.917054] [ 6638.480757] perf: interrupt took too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [ 6665.582918] watchdog: BUG: soft lockup - CPU#7 stuck for 4090s! [kworker/7:4:78808] [ 6665.590567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6665.646015] irq event stamp: 10177906 [ 6665.649672] hardirqs last enabled at (10177905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6665.659838] hardirqs last disabled at (10177906): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6665.669562] softirqs last enabled at (10177830): [] __do_softirq+0x2be/0x371 [ 6665.678255] softirqs last disabled at (10177825): [] irq_exit_rcu+0xb0/0xd0 [ 6665.686767] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6665.696145] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6665.703615] Workqueue: events delayed_work [ceph] [ 6665.708338] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 6665.712949] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 6665.731687] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [ 6665.736903] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 6665.744028] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f867488 [ 6665.751151] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 6665.758276] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 6665.765400] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 6665.772525] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6665.780602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6665.786340] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6665.793464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6665.800588] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6665.807711] Call Trace: [ 6665.810159] [ 6665.812169] ? watchdog_timer_fn+0x1b4/0x220 [ 6665.816440] ? __hrtimer_run_queues+0x1d3/0x360 [ 6665.820966] ? hrtimer_interrupt+0x104/0x240 [ 6665.825237] ? __local_bh_enable+0x50/0x60 [ 6665.829330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6665.834552] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6665.839510] [ 6665.841608] [ 6665.843706] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6665.849012] ? do_raw_spin_lock+0x42/0xc0 [ 6665.853024] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6665.858430] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6665.863659] delayed_work+0x1d8/0x280 [ceph] [ 6665.867941] ? process_scheduled_works+0x22a/0x510 [ 6665.872729] process_scheduled_works+0x2a4/0x510 [ 6665.877345] worker_thread+0x188/0x340 [ 6665.881095] ? rescuer_thread+0x310/0x310 [ 6665.885098] kthread+0xf5/0x130 [ 6665.888237] ? kthread_complete_and_exit+0x20/0x20 [ 6665.893020] ret_from_fork+0x31/0x40 [ 6665.896601] ? kthread_complete_and_exit+0x20/0x20 [ 6665.901382] ret_from_fork_asm+0x11/0x20 [ 6665.905306] [ 6693.583409] watchdog: BUG: soft lockup - CPU#7 stuck for 4116s! [kworker/7:4:78808] [ 6693.591063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6693.646513] irq event stamp: 10239282 [ 6693.650170] hardirqs last enabled at (10239281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6693.660336] hardirqs last disabled at (10239282): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6693.670060] softirqs last enabled at (10239244): [] __do_softirq+0x2be/0x371 [ 6693.678754] softirqs last disabled at (10239239): [] irq_exit_rcu+0xb0/0xd0 [ 6693.687274] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6693.696650] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6693.704122] Workqueue: events delayed_work [ceph] [ 6693.708846] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6693.713205] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6693.731942] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 6693.737160] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 6693.744283] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 6693.751408] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 6693.758532] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 6693.765656] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6693.772779] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6693.780857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6693.786594] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6693.793720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6693.800844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6693.807966] Call Trace: [ 6693.810413] [ 6693.812424] ? watchdog_timer_fn+0x1b4/0x220 [ 6693.816688] ? __hrtimer_run_queues+0x1d3/0x360 [ 6693.821221] ? hrtimer_interrupt+0x104/0x240 [ 6693.825482] ? __local_bh_enable+0x50/0x60 [ 6693.829575] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6693.834800] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6693.839757] [ 6693.841854] [ 6693.843952] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6693.849259] ? lock_acquire+0xf0/0x2b0 [ 6693.853009] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 6693.857714] ? lock_release+0xc5/0x260 [ 6693.861460] _raw_spin_lock+0x30/0x40 [ 6693.865116] ? ceph_check_caps+0x63/0xae0 [ceph] [ 6693.869737] ceph_check_caps+0x63/0xae0 [ceph] [ 6693.874187] ? __lock_acquire+0xbfa/0x1e20 [ 6693.878285] ? lock_acquire+0xcb/0x2b0 [ 6693.882034] ? igrab+0x1d/0x50 [ 6693.885084] ? find_held_lock+0x2d/0x90 [ 6693.888916] ? find_held_lock+0x2d/0x90 [ 6693.892745] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6693.898145] ? lock_release+0xc5/0x260 [ 6693.901892] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6693.907128] delayed_work+0x1d8/0x280 [ceph] [ 6693.911407] ? process_scheduled_works+0x22a/0x510 [ 6693.916198] process_scheduled_works+0x2a4/0x510 [ 6693.920814] worker_thread+0x188/0x340 [ 6693.924563] ? rescuer_thread+0x310/0x310 [ 6693.928566] kthread+0xf5/0x130 [ 6693.931705] ? kthread_complete_and_exit+0x20/0x20 [ 6693.936488] ret_from_fork+0x31/0x40 [ 6693.940068] ? kthread_complete_and_exit+0x20/0x20 [ 6693.944850] ret_from_fork_asm+0x11/0x20 [ 6693.948775] [ 6721.583901] watchdog: BUG: soft lockup - CPU#7 stuck for 4142s! [kworker/7:4:78808] [ 6721.591552] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6721.647000] irq event stamp: 10301718 [ 6721.650659] hardirqs last enabled at (10301717): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6721.660825] hardirqs last disabled at (10301718): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6721.670548] softirqs last enabled at (10301640): [] __do_softirq+0x2be/0x371 [ 6721.679234] softirqs last disabled at (10301635): [] irq_exit_rcu+0xb0/0xd0 [ 6721.687753] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6721.697130] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6721.704603] Workqueue: events delayed_work [ceph] [ 6721.709316] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6721.713667] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6721.732405] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6721.737621] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6721.744746] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6721.751870] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6721.758994] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6721.766118] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6721.773243] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6721.781321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6721.787057] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6721.794180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6721.801305] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6721.808430] Call Trace: [ 6721.810876] [ 6721.812884] ? watchdog_timer_fn+0x1b4/0x220 [ 6721.817150] ? __hrtimer_run_queues+0x1d3/0x360 [ 6721.821686] ? hrtimer_interrupt+0x104/0x240 [ 6721.825953] ? __local_bh_enable+0x50/0x60 [ 6721.830047] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6721.835270] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6721.840228] [ 6721.842325] [ 6721.844423] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6721.849732] ? lock_acquire+0xf0/0x2b0 [ 6721.853480] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6721.858890] ? lock_release+0xc5/0x260 [ 6721.862635] _raw_spin_lock+0x30/0x40 [ 6721.866300] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6721.871698] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6721.876925] delayed_work+0x1d8/0x280 [ceph] [ 6721.881207] ? process_scheduled_works+0x22a/0x510 [ 6721.885999] process_scheduled_works+0x2a4/0x510 [ 6721.890614] worker_thread+0x188/0x340 [ 6721.894362] ? rescuer_thread+0x310/0x310 [ 6721.898365] kthread+0xf5/0x130 [ 6721.901504] ? kthread_complete_and_exit+0x20/0x20 [ 6721.906287] ret_from_fork+0x31/0x40 [ 6721.909868] ? kthread_complete_and_exit+0x20/0x20 [ 6721.914651] ret_from_fork_asm+0x11/0x20 [ 6721.918581] [ 6749.584410] watchdog: BUG: soft lockup - CPU#7 stuck for 4169s! [kworker/7:4:78808] [ 6749.592068] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6749.647517] irq event stamp: 10362762 [ 6749.651173] hardirqs last enabled at (10362761): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6749.661340] hardirqs last disabled at (10362762): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6749.671072] softirqs last enabled at (10362748): [] __do_softirq+0x2be/0x371 [ 6749.679756] softirqs last disabled at (10362743): [] irq_exit_rcu+0xb0/0xd0 [ 6749.688276] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6749.697656] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6749.705126] Workqueue: events delayed_work [ceph] [ 6749.709849] RIP: 0010:rb_next+0xc/0x50 [ 6749.713601] Code: 1e fa 48 8b 07 48 85 c0 75 05 eb 0d 48 89 d0 48 8b 50 08 48 85 d2 75 f4 c3 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 39 cf 74 39 <48> 8b 47 08 48 85 c0 74 22 48 8b 50 10 48 85 d2 74 0c 48 89 d0 48 [ 6749.732340] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000282 [ 6749.737557] RAX: 0000000000000000 RBX: ffff88810da9e000 RCX: 0000000000000001 [ 6749.744680] RDX: 0000000000000155 RSI: 0000000000000000 RDI: ffff88810da9e008 [ 6749.751803] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 6749.758929] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 6749.766052] R13: ffff8881a1d34478 R14: 0000000000000000 R15: ffff88810da9e008 [ 6749.773177] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6749.781254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6749.786992] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6749.794116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6749.801240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6749.808364] Call Trace: [ 6749.810810] [ 6749.812821] ? watchdog_timer_fn+0x1b4/0x220 [ 6749.817093] ? __hrtimer_run_queues+0x1d3/0x360 [ 6749.821627] ? hrtimer_interrupt+0x104/0x240 [ 6749.825887] ? __local_bh_enable+0x50/0x60 [ 6749.829982] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6749.835206] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6749.840162] [ 6749.842260] [ 6749.844357] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6749.849665] ? rb_next+0xc/0x50 [ 6749.852808] ceph_check_caps+0x228/0xae0 [ceph] [ 6749.857358] ? lock_acquire+0xcb/0x2b0 [ 6749.861102] ? igrab+0x1d/0x50 [ 6749.864153] ? find_held_lock+0x2d/0x90 [ 6749.867992] ? find_held_lock+0x2d/0x90 [ 6749.871824] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6749.877231] ? lock_release+0xc5/0x260 [ 6749.880977] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 6749.886214] delayed_work+0x1d8/0x280 [ceph] [ 6749.890493] ? process_scheduled_works+0x22a/0x510 [ 6749.895283] process_scheduled_works+0x2a4/0x510 [ 6749.899899] worker_thread+0x188/0x340 [ 6749.903649] ? rescuer_thread+0x310/0x310 [ 6749.907651] kthread+0xf5/0x130 [ 6749.910792] ? kthread_complete_and_exit+0x20/0x20 [ 6749.915583] ret_from_fork+0x31/0x40 [ 6749.919161] ? kthread_complete_and_exit+0x20/0x20 [ 6749.923946] ret_from_fork_asm+0x11/0x20 [ 6749.927877] [ 6777.584883] watchdog: BUG: soft lockup - CPU#7 stuck for 4195s! [kworker/7:4:78808] [ 6777.592537] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6777.647988] irq event stamp: 10424230 [ 6777.651644] hardirqs last enabled at (10424229): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6777.661812] hardirqs last disabled at (10424230): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6777.671544] softirqs last enabled at (10424108): [] __do_softirq+0x2be/0x371 [ 6777.680230] softirqs last disabled at (10424103): [] irq_exit_rcu+0xb0/0xd0 [ 6777.688750] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6777.698125] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6777.705598] Workqueue: events delayed_work [ceph] [ 6777.710320] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 6777.714932] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 6777.733667] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [ 6777.738887] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 00000000de0313a6 [ 6777.746011] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f867488 [ 6777.753133] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 6777.760259] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 6777.767383] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 6777.774506] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6777.782583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6777.788321] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6777.795445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6777.802569] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6777.809694] Call Trace: [ 6777.812139] [ 6777.814149] ? watchdog_timer_fn+0x1b4/0x220 [ 6777.818414] ? __hrtimer_run_queues+0x1d3/0x360 [ 6777.822950] ? hrtimer_interrupt+0x104/0x240 [ 6777.827219] ? __local_bh_enable+0x50/0x60 [ 6777.831311] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6777.836535] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6777.841492] [ 6777.843590] [ 6777.845688] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6777.850997] ? do_raw_spin_lock+0x42/0xc0 [ 6777.855006] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6777.860413] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6777.865641] delayed_work+0x1d8/0x280 [ceph] [ 6777.869923] ? process_scheduled_works+0x22a/0x510 [ 6777.874711] process_scheduled_works+0x2a4/0x510 [ 6777.879329] worker_thread+0x188/0x340 [ 6777.883078] ? rescuer_thread+0x310/0x310 [ 6777.887080] kthread+0xf5/0x130 [ 6777.890219] ? kthread_complete_and_exit+0x20/0x20 [ 6777.895001] ret_from_fork+0x31/0x40 [ 6777.898582] ? kthread_complete_and_exit+0x20/0x20 [ 6777.903364] ret_from_fork_asm+0x11/0x20 [ 6777.907288] [ 6805.585381] watchdog: BUG: soft lockup - CPU#7 stuck for 4221s! [kworker/7:4:78808] [ 6805.593037] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6805.648485] irq event stamp: 10488182 [ 6805.652142] hardirqs last enabled at (10488181): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6805.662310] hardirqs last disabled at (10488182): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6805.672041] softirqs last enabled at (10488144): [] __do_softirq+0x2be/0x371 [ 6805.680727] softirqs last disabled at (10488139): [] irq_exit_rcu+0xb0/0xd0 [ 6805.689246] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6805.698624] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6805.706095] Workqueue: events delayed_work [ceph] [ 6805.710819] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6805.715178] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6805.733914] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6805.739133] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6805.746256] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6805.753380] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6805.760505] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6805.767628] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6805.774754] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6805.782829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6805.788568] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6805.795692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6805.802815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6805.809939] Call Trace: [ 6805.812386] [ 6805.814396] ? watchdog_timer_fn+0x1b4/0x220 [ 6805.818659] ? __hrtimer_run_queues+0x1d3/0x360 [ 6805.823185] ? hrtimer_interrupt+0x104/0x240 [ 6805.827446] ? __local_bh_enable+0x50/0x60 [ 6805.831541] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6805.836763] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6805.841720] [ 6805.843817] [ 6805.845917] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6805.851224] ? lock_acquire+0xf0/0x2b0 [ 6805.854974] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6805.860381] ? lock_release+0xc5/0x260 [ 6805.864126] _raw_spin_lock+0x30/0x40 [ 6805.867782] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6805.873183] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6805.878410] delayed_work+0x1d8/0x280 [ceph] [ 6805.882693] ? process_scheduled_works+0x22a/0x510 [ 6805.887481] process_scheduled_works+0x2a4/0x510 [ 6805.892099] worker_thread+0x188/0x340 [ 6805.895847] ? rescuer_thread+0x310/0x310 [ 6805.899850] kthread+0xf5/0x130 [ 6805.902989] ? kthread_complete_and_exit+0x20/0x20 [ 6805.907771] ret_from_fork+0x31/0x40 [ 6805.911342] ? kthread_complete_and_exit+0x20/0x20 [ 6805.916126] ret_from_fork_asm+0x11/0x20 [ 6805.920050] [ 6833.585879] watchdog: BUG: soft lockup - CPU#7 stuck for 4247s! [kworker/7:4:78808] [ 6833.593533] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6833.648984] irq event stamp: 10550612 [ 6833.652641] hardirqs last enabled at (10550611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6833.662807] hardirqs last disabled at (10550612): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6833.672532] softirqs last enabled at (10550598): [] __do_softirq+0x2be/0x371 [ 6833.681224] softirqs last disabled at (10550593): [] irq_exit_rcu+0xb0/0xd0 [ 6833.689744] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6833.699122] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6833.706594] Workqueue: events delayed_work [ceph] [ 6833.711316] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6833.715666] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6833.734403] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 6833.739622] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 6833.746745] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 6833.753868] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 6833.760993] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6833.768119] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6833.775241] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6833.783319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6833.789057] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6833.796180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6833.803304] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6833.810429] Call Trace: [ 6833.812877] [ 6833.814885] ? watchdog_timer_fn+0x1b4/0x220 [ 6833.819149] ? __hrtimer_run_queues+0x1d3/0x360 [ 6833.823674] ? hrtimer_interrupt+0x104/0x240 [ 6833.827935] ? __local_bh_enable+0x50/0x60 [ 6833.832030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6833.837252] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6833.842209] [ 6833.844307] [ 6833.846405] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6833.851714] ? lock_acquire+0xf0/0x2b0 [ 6833.855465] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6833.860873] _raw_spin_lock+0x30/0x40 [ 6833.864539] ? igrab+0x1d/0x50 [ 6833.867596] igrab+0x1d/0x50 [ 6833.870474] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6833.875615] delayed_work+0x1d8/0x280 [ceph] [ 6833.879896] ? process_scheduled_works+0x22a/0x510 [ 6833.884686] process_scheduled_works+0x2a4/0x510 [ 6833.889302] worker_thread+0x188/0x340 [ 6833.893052] ? rescuer_thread+0x310/0x310 [ 6833.897055] kthread+0xf5/0x130 [ 6833.900202] ? kthread_complete_and_exit+0x20/0x20 [ 6833.904984] ret_from_fork+0x31/0x40 [ 6833.908565] ? kthread_complete_and_exit+0x20/0x20 [ 6833.913356] ret_from_fork_asm+0x11/0x20 [ 6833.917279] [ 6861.586357] watchdog: BUG: soft lockup - CPU#7 stuck for 4273s! [kworker/7:4:78808] [ 6861.594007] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6861.649456] irq event stamp: 10611834 [ 6861.653114] hardirqs last enabled at (10611833): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6861.663280] hardirqs last disabled at (10611834): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6861.673002] softirqs last enabled at (10611814): [] __do_softirq+0x2be/0x371 [ 6861.681688] softirqs last disabled at (10611809): [] irq_exit_rcu+0xb0/0xd0 [ 6861.690209] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6861.699594] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6861.707063] Workqueue: events delayed_work [ceph] [ 6861.711787] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6861.716138] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6861.734876] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 6861.740092] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 6861.747217] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 6861.754340] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 6861.761466] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6861.768589] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6861.775714] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6861.783791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6861.789529] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6861.796651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6861.803777] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6861.810900] Call Trace: [ 6861.813349] [ 6861.815359] ? watchdog_timer_fn+0x1b4/0x220 [ 6861.819629] ? __hrtimer_run_queues+0x1d3/0x360 [ 6861.824156] ? hrtimer_interrupt+0x104/0x240 [ 6861.828426] ? __local_bh_enable+0x50/0x60 [ 6861.832518] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6861.837743] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6861.842699] [ 6861.844796] [ 6861.846895] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6861.852201] ? lock_acquire+0xf0/0x2b0 [ 6861.855945] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6861.861351] _raw_spin_lock+0x30/0x40 [ 6861.865009] ? igrab+0x1d/0x50 [ 6861.868058] igrab+0x1d/0x50 [ 6861.870938] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6861.876088] delayed_work+0x1d8/0x280 [ceph] [ 6861.880368] ? process_scheduled_works+0x22a/0x510 [ 6861.885158] process_scheduled_works+0x2a4/0x510 [ 6861.889773] worker_thread+0x188/0x340 [ 6861.893522] ? rescuer_thread+0x310/0x310 [ 6861.897525] kthread+0xf5/0x130 [ 6861.900664] ? kthread_complete_and_exit+0x20/0x20 [ 6861.905449] ret_from_fork+0x31/0x40 [ 6861.909028] ? kthread_complete_and_exit+0x20/0x20 [ 6861.913812] ret_from_fork_asm+0x11/0x20 [ 6861.917734] [ 6889.586849] watchdog: BUG: soft lockup - CPU#7 stuck for 4299s! [kworker/7:4:78808] [ 6889.594522] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6889.649970] irq event stamp: 10674530 [ 6889.653627] hardirqs last enabled at (10674529): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6889.663793] hardirqs last disabled at (10674530): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6889.673518] softirqs last enabled at (10674510): [] __do_softirq+0x2be/0x371 [ 6889.682211] softirqs last disabled at (10674505): [] irq_exit_rcu+0xb0/0xd0 [ 6889.690733] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6889.700108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6889.707581] Workqueue: events delayed_work [ceph] [ 6889.712303] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6889.716654] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6889.735391] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 6889.740607] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 6889.747733] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 6889.754855] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 6889.761980] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6889.769104] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6889.776230] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6889.784305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6889.790043] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6889.797168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6889.804291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6889.811416] Call Trace: [ 6889.813862] [ 6889.815873] ? watchdog_timer_fn+0x1b4/0x220 [ 6889.820136] ? __hrtimer_run_queues+0x1d3/0x360 [ 6889.824671] ? hrtimer_interrupt+0x104/0x240 [ 6889.828931] ? __local_bh_enable+0x50/0x60 [ 6889.833025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6889.838249] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6889.843206] [ 6889.845304] [ 6889.847402] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6889.852707] ? lock_acquire+0xf0/0x2b0 [ 6889.856452] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6889.861858] _raw_spin_lock+0x30/0x40 [ 6889.865517] ? igrab+0x1d/0x50 [ 6889.868575] igrab+0x1d/0x50 [ 6889.871452] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6889.876594] delayed_work+0x1d8/0x280 [ceph] [ 6889.880875] ? process_scheduled_works+0x22a/0x510 [ 6889.885665] process_scheduled_works+0x2a4/0x510 [ 6889.890281] worker_thread+0x188/0x340 [ 6889.894029] ? rescuer_thread+0x310/0x310 [ 6889.898032] kthread+0xf5/0x130 [ 6889.901171] ? kthread_complete_and_exit+0x20/0x20 [ 6889.905955] ret_from_fork+0x31/0x40 [ 6889.909534] ? kthread_complete_and_exit+0x20/0x20 [ 6889.914316] ret_from_fork_asm+0x11/0x20 [ 6889.918241] [ 6902.003698] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6912.243017] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6917.587340] watchdog: BUG: soft lockup - CPU#7 stuck for 4325s! [kworker/7:4:78808] [ 6917.595011] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6917.650459] irq event stamp: 10736030 [ 6917.654116] hardirqs last enabled at (10736029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6917.664283] hardirqs last disabled at (10736030): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6917.674006] softirqs last enabled at (10736010): [] __do_softirq+0x2be/0x371 [ 6917.682691] softirqs last disabled at (10736005): [] irq_exit_rcu+0xb0/0xd0 [ 6917.691202] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6917.700580] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6917.708051] Workqueue: events delayed_work [ceph] [ 6917.712774] RIP: 0010:lock_release+0x14e/0x260 [ 6917.717212] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 6917.735949] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [ 6917.741166] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [ 6917.748292] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [ 6917.755415] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [ 6917.762540] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9425 [ 6917.769662] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [ 6917.776788] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6917.784865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6917.790603] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6917.797726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6917.804850] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6917.811975] Call Trace: [ 6917.814420] [ 6917.816431] ? watchdog_timer_fn+0x1b4/0x220 [ 6917.820695] ? __hrtimer_run_queues+0x1d3/0x360 [ 6917.825230] ? hrtimer_interrupt+0x104/0x240 [ 6917.829500] ? __local_bh_enable+0x50/0x60 [ 6917.833592] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6917.838817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6917.843774] [ 6917.845870] [ 6917.847969] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6917.853273] ? igrab+0x35/0x50 [ 6917.856327] ? lock_release+0x14e/0x260 [ 6917.860163] ? lock_release+0xc5/0x260 [ 6917.863907] _raw_spin_unlock+0x1b/0x40 [ 6917.867742] igrab+0x35/0x50 [ 6917.870626] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6917.875776] delayed_work+0x1d8/0x280 [ceph] [ 6917.880056] ? process_scheduled_works+0x22a/0x510 [ 6917.884844] process_scheduled_works+0x2a4/0x510 [ 6917.889460] worker_thread+0x188/0x340 [ 6917.893211] ? rescuer_thread+0x310/0x310 [ 6917.897214] kthread+0xf5/0x130 [ 6917.900352] ? kthread_complete_and_exit+0x20/0x20 [ 6917.905135] ret_from_fork+0x31/0x40 [ 6917.908715] ? kthread_complete_and_exit+0x20/0x20 [ 6917.913499] ret_from_fork_asm+0x11/0x20 [ 6917.917430] [ 6922.995616] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6933.235526] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6943.475673] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6945.587831] watchdog: BUG: soft lockup - CPU#7 stuck for 4351s! [kworker/7:4:78808] [ 6945.595501] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6945.650950] irq event stamp: 10796672 [ 6945.654606] hardirqs last enabled at (10796671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6945.664773] hardirqs last disabled at (10796672): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6945.674505] softirqs last enabled at (10796648): [] __do_softirq+0x2be/0x371 [ 6945.683190] softirqs last disabled at (10796643): [] irq_exit_rcu+0xb0/0xd0 [ 6945.691711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6945.701086] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6945.708559] Workqueue: events delayed_work [ceph] [ 6945.713282] RIP: 0010:lock_acquire+0x86/0x2b0 [ 6945.717632] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [ 6945.736369] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [ 6945.741587] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [ 6945.748710] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [ 6945.755835] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 6945.762958] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [ 6945.770082] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6945.777207] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6945.785286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6945.791023] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6945.798145] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6945.805271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6945.812395] Call Trace: [ 6945.814840] [ 6945.816851] ? watchdog_timer_fn+0x1b4/0x220 [ 6945.821115] ? __hrtimer_run_queues+0x1d3/0x360 [ 6945.825639] ? hrtimer_interrupt+0x104/0x240 [ 6945.829901] ? __local_bh_enable+0x50/0x60 [ 6945.833993] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6945.839220] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6945.844202] [ 6945.846300] [ 6945.848396] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6945.853705] ? lock_acquire+0x86/0x2b0 [ 6945.857455] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6945.862862] _raw_spin_lock+0x30/0x40 [ 6945.866519] ? igrab+0x1d/0x50 [ 6945.869568] igrab+0x1d/0x50 [ 6945.872450] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 6945.877598] delayed_work+0x1d8/0x280 [ceph] [ 6945.881880] ? process_scheduled_works+0x22a/0x510 [ 6945.886669] process_scheduled_works+0x2a4/0x510 [ 6945.891286] worker_thread+0x188/0x340 [ 6945.895034] ? rescuer_thread+0x310/0x310 [ 6945.899038] kthread+0xf5/0x130 [ 6945.902176] ? kthread_complete_and_exit+0x20/0x20 [ 6945.906959] ret_from_fork+0x31/0x40 [ 6945.910538] ? kthread_complete_and_exit+0x20/0x20 [ 6945.915323] ret_from_fork_asm+0x11/0x20 [ 6945.919254] [ 6954.227914] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6964.468019] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6973.588323] watchdog: BUG: soft lockup - CPU#7 stuck for 4377s! [kworker/7:4:78808] [ 6973.595972] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 6973.651421] irq event stamp: 10858314 [ 6973.655077] hardirqs last enabled at (10858313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6973.665243] hardirqs last disabled at (10858314): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 6973.674969] softirqs last enabled at (10858294): [] __do_softirq+0x2be/0x371 [ 6973.683662] softirqs last disabled at (10858289): [] irq_exit_rcu+0xb0/0xd0 [ 6973.692183] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 6973.701557] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 6973.709031] Workqueue: events delayed_work [ceph] [ 6973.713754] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 6973.718105] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 6973.736840] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 6973.742058] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 6973.749182] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 6973.756307] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 6973.763432] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 6973.770554] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6973.777679] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 6973.785755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6973.791494] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 6973.798618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6973.805742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6973.812866] Call Trace: [ 6973.815312] [ 6973.817322] ? watchdog_timer_fn+0x1b4/0x220 [ 6973.821586] ? __hrtimer_run_queues+0x1d3/0x360 [ 6973.826111] ? hrtimer_interrupt+0x104/0x240 [ 6973.830374] ? __local_bh_enable+0x50/0x60 [ 6973.834466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 6973.839690] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 6973.844648] [ 6973.846745] [ 6973.848843] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 6973.854150] ? lock_acquire+0xf0/0x2b0 [ 6973.857900] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 6973.863307] ? lock_release+0xc5/0x260 [ 6973.867053] _raw_spin_lock+0x30/0x40 [ 6973.870711] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6973.876107] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 6973.881337] delayed_work+0x1d8/0x280 [ceph] [ 6973.885618] ? process_scheduled_works+0x22a/0x510 [ 6973.890410] process_scheduled_works+0x2a4/0x510 [ 6973.895024] worker_thread+0x188/0x340 [ 6973.898772] ? rescuer_thread+0x310/0x310 [ 6973.902777] kthread+0xf5/0x130 [ 6973.905915] ? kthread_complete_and_exit+0x20/0x20 [ 6973.910700] ret_from_fork+0x31/0x40 [ 6973.914279] ? kthread_complete_and_exit+0x20/0x20 [ 6973.919063] ret_from_fork_asm+0x11/0x20 [ 6973.922985] [ 6975.220203] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 6985.460417] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7001.588818] watchdog: BUG: soft lockup - CPU#7 stuck for 4403s! [kworker/7:4:78808] [ 7001.596470] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7001.651919] irq event stamp: 10921686 [ 7001.655576] hardirqs last enabled at (10921685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7001.665743] hardirqs last disabled at (10921686): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7001.675467] softirqs last enabled at (10921672): [] __do_softirq+0x2be/0x371 [ 7001.684158] softirqs last disabled at (10921667): [] irq_exit_rcu+0xb0/0xd0 [ 7001.692681] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7001.702057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7001.709529] Workqueue: events delayed_work [ceph] [ 7001.714251] RIP: 0010:lock_acquire+0x4a/0x2b0 [ 7001.718602] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [ 7001.737338] RSP: 0018:ffffc90002117b48 EFLAGS: 00000247 [ 7001.742555] RAX: 0000000000000007 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 7001.749680] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34938 [ 7001.756803] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 7001.763928] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34938 [ 7001.771051] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7001.778177] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7001.786254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7001.791992] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7001.799116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7001.806241] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7001.813364] Call Trace: [ 7001.815811] [ 7001.817823] ? watchdog_timer_fn+0x1b4/0x220 [ 7001.822094] ? __hrtimer_run_queues+0x1d3/0x360 [ 7001.826626] ? hrtimer_interrupt+0x104/0x240 [ 7001.830888] ? __local_bh_enable+0x50/0x60 [ 7001.834983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7001.840205] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7001.845163] [ 7001.847261] [ 7001.849357] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7001.854665] ? lock_acquire+0x4a/0x2b0 [ 7001.858407] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 7001.863122] ? lock_release+0xc5/0x260 [ 7001.866865] _raw_spin_lock+0x30/0x40 [ 7001.870523] ? ceph_check_caps+0x63/0xae0 [ceph] [ 7001.875141] ceph_check_caps+0x63/0xae0 [ceph] [ 7001.879593] ? __lock_acquire+0xbfa/0x1e20 [ 7001.883692] ? lock_acquire+0xcb/0x2b0 [ 7001.887441] ? igrab+0x1d/0x50 [ 7001.890500] ? find_held_lock+0x2d/0x90 [ 7001.894331] ? find_held_lock+0x2d/0x90 [ 7001.898162] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7001.903560] ? lock_release+0xc5/0x260 [ 7001.907306] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7001.912534] delayed_work+0x1d8/0x280 [ceph] [ 7001.916814] ? process_scheduled_works+0x22a/0x510 [ 7001.921604] process_scheduled_works+0x2a4/0x510 [ 7001.926219] worker_thread+0x188/0x340 [ 7001.929968] ? rescuer_thread+0x310/0x310 [ 7001.933972] kthread+0xf5/0x130 [ 7001.937110] ? kthread_complete_and_exit+0x20/0x20 [ 7001.941893] ret_from_fork+0x31/0x40 [ 7001.945464] ? kthread_complete_and_exit+0x20/0x20 [ 7001.950248] ret_from_fork_asm+0x11/0x20 [ 7001.954181] [ 7017.204501] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [ 7017.213056] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7027.445265] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7029.589323] watchdog: BUG: soft lockup - CPU#7 stuck for 4429s! [kworker/7:4:78808] [ 7029.596978] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7029.652426] irq event stamp: 10986390 [ 7029.656082] hardirqs last enabled at (10986389): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7029.666249] hardirqs last disabled at (10986390): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7029.675982] softirqs last enabled at (10986312): [] __do_softirq+0x2be/0x371 [ 7029.684675] softirqs last disabled at (10986307): [] irq_exit_rcu+0xb0/0xd0 [ 7029.693197] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7029.702581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7029.710052] Workqueue: events delayed_work [ceph] [ 7029.714774] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7029.719125] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7029.737863] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 7029.743079] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 7029.750205] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 7029.757330] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 7029.764453] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 7029.771576] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7029.778702] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7029.786778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7029.792515] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7029.799641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7029.806765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7029.813889] Call Trace: [ 7029.816334] [ 7029.818344] ? watchdog_timer_fn+0x1b4/0x220 [ 7029.822609] ? __hrtimer_run_queues+0x1d3/0x360 [ 7029.827142] ? hrtimer_interrupt+0x104/0x240 [ 7029.831404] ? __local_bh_enable+0x50/0x60 [ 7029.835496] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7029.840722] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7029.845678] [ 7029.847776] [ 7029.849874] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7029.855181] ? lock_acquire+0xf0/0x2b0 [ 7029.858932] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 7029.863644] ? lock_release+0xc5/0x260 [ 7029.867389] _raw_spin_lock+0x30/0x40 [ 7029.871047] ? ceph_check_caps+0x63/0xae0 [ceph] [ 7029.875665] ceph_check_caps+0x63/0xae0 [ceph] [ 7029.880117] ? __lock_acquire+0xbfa/0x1e20 [ 7029.884215] ? lock_acquire+0xcb/0x2b0 [ 7029.887964] ? igrab+0x1d/0x50 [ 7029.891015] ? find_held_lock+0x2d/0x90 [ 7029.894855] ? find_held_lock+0x2d/0x90 [ 7029.898686] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7029.904086] ? lock_release+0xc5/0x260 [ 7029.907830] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7029.913056] delayed_work+0x1d8/0x280 [ceph] [ 7029.917338] ? process_scheduled_works+0x22a/0x510 [ 7029.922129] process_scheduled_works+0x2a4/0x510 [ 7029.926745] worker_thread+0x188/0x340 [ 7029.930493] ? rescuer_thread+0x310/0x310 [ 7029.934496] kthread+0xf5/0x130 [ 7029.937636] ? kthread_complete_and_exit+0x20/0x20 [ 7029.942426] ret_from_fork+0x31/0x40 [ 7029.946006] ? kthread_complete_and_exit+0x20/0x20 [ 7029.950790] ret_from_fork_asm+0x11/0x20 [ 7029.954722] [ 7038.197373] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7048.437606] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7057.589797] watchdog: BUG: soft lockup - CPU#7 stuck for 4455s! [kworker/7:4:78808] [ 7057.597448] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7057.652896] irq event stamp: 11050846 [ 7057.656555] hardirqs last enabled at (11050845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7057.666719] hardirqs last disabled at (11050846): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7057.676445] softirqs last enabled at (11050768): [] __do_softirq+0x2be/0x371 [ 7057.685139] softirqs last disabled at (11050763): [] irq_exit_rcu+0xb0/0xd0 [ 7057.693659] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7057.703035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7057.710506] Workqueue: events delayed_work [ceph] [ 7057.715232] RIP: 0010:lock_release+0x16a/0x260 [ 7057.719675] Code: f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 c4 18 5b 5d 41 5c <41> 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 48 0f a3 05 18 bf 53 01 [ 7057.738412] RSP: 0018:ffffc90002117d78 EFLAGS: 00000296 [ 7057.743631] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [ 7057.750753] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [ 7057.757879] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 7057.765002] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7057.772126] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [ 7057.779250] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7057.787327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7057.793065] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7057.800191] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7057.807313] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7057.814438] Call Trace: [ 7057.816885] [ 7057.818895] ? watchdog_timer_fn+0x1b4/0x220 [ 7057.823167] ? __hrtimer_run_queues+0x1d3/0x360 [ 7057.827703] ? hrtimer_interrupt+0x104/0x240 [ 7057.831972] ? __local_bh_enable+0x50/0x60 [ 7057.836065] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7057.841289] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7057.846246] [ 7057.848341] [ 7057.850439] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7057.855749] ? lock_release+0x16a/0x260 [ 7057.859586] _raw_spin_unlock+0x1b/0x40 [ 7057.863424] igrab+0x35/0x50 [ 7057.866301] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7057.871453] delayed_work+0x1d8/0x280 [ceph] [ 7057.875733] ? process_scheduled_works+0x22a/0x510 [ 7057.880523] process_scheduled_works+0x2a4/0x510 [ 7057.885137] worker_thread+0x188/0x340 [ 7057.888887] ? rescuer_thread+0x310/0x310 [ 7057.892891] kthread+0xf5/0x130 [ 7057.896028] ? kthread_complete_and_exit+0x20/0x20 [ 7057.900813] ret_from_fork+0x31/0x40 [ 7057.904393] ? kthread_complete_and_exit+0x20/0x20 [ 7057.909176] ret_from_fork_asm+0x11/0x20 [ 7057.913098] [ 7059.189939] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7069.429823] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7080.182015] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7085.590288] watchdog: BUG: soft lockup - CPU#7 stuck for 4481s! [kworker/7:4:78808] [ 7085.597935] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7085.653386] irq event stamp: 11113436 [ 7085.657043] hardirqs last enabled at (11113435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7085.667210] hardirqs last disabled at (11113436): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7085.676934] softirqs last enabled at (11113400): [] __do_softirq+0x2be/0x371 [ 7085.685635] softirqs last disabled at (11113395): [] irq_exit_rcu+0xb0/0xd0 [ 7085.694147] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7085.703524] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7085.710995] Workqueue: events delayed_work [ceph] [ 7085.715719] RIP: 0010:_raw_spin_unlock+0xf/0x40 [ 7085.720242] Code: 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 <48> 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff [ 7085.738979] RSP: 0018:ffffc90002117da8 EFLAGS: 00000286 [ 7085.744198] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [ 7085.751320] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f867488 [ 7085.758445] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 7085.765570] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7085.772694] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7085.779817] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7085.787896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7085.793633] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7085.800758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7085.807880] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7085.815006] Call Trace: [ 7085.817451] [ 7085.819464] ? watchdog_timer_fn+0x1b4/0x220 [ 7085.823735] ? __hrtimer_run_queues+0x1d3/0x360 [ 7085.828268] ? hrtimer_interrupt+0x104/0x240 [ 7085.832529] ? __local_bh_enable+0x50/0x60 [ 7085.836622] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7085.841847] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7085.846803] [ 7085.848901] [ 7085.850999] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7085.856306] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7085.861710] ? _raw_spin_unlock+0xf/0x40 [ 7085.865629] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7085.870857] delayed_work+0x1d8/0x280 [ceph] [ 7085.875138] ? process_scheduled_works+0x22a/0x510 [ 7085.879928] process_scheduled_works+0x2a4/0x510 [ 7085.884544] worker_thread+0x188/0x340 [ 7085.888295] ? rescuer_thread+0x310/0x310 [ 7085.892304] kthread+0xf5/0x130 [ 7085.895443] ? kthread_complete_and_exit+0x20/0x20 [ 7085.900227] ret_from_fork+0x31/0x40 [ 7085.903806] ? kthread_complete_and_exit+0x20/0x20 [ 7085.908590] ret_from_fork_asm+0x11/0x20 [ 7085.912521] [ 7090.422326] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7101.174482] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7113.590780] watchdog: BUG: soft lockup - CPU#7 stuck for 4508s! [kworker/7:4:78808] [ 7113.598426] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7113.653876] irq event stamp: 11173764 [ 7113.657532] hardirqs last enabled at (11173763): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7113.667698] hardirqs last disabled at (11173764): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7113.677424] softirqs last enabled at (11173726): [] __do_softirq+0x2be/0x371 [ 7113.686117] softirqs last disabled at (11173721): [] irq_exit_rcu+0xb0/0xd0 [ 7113.694637] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7113.704013] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7113.711485] Workqueue: events delayed_work [ceph] [ 7113.716209] RIP: 0010:lock_release+0x6/0x260 [ 7113.720473] Code: df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 <41> 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 [ 7113.739208] RSP: 0018:ffffc90002117d98 EFLAGS: 00000296 [ 7113.744426] RAX: ffff8881a1d34478 RBX: ffff88817f867488 RCX: ffffc90002117d54 [ 7113.751552] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [ 7113.758674] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 7113.765800] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7113.772922] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7113.780046] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7113.788126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7113.793863] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7113.800986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7113.808110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7113.815236] Call Trace: [ 7113.817681] [ 7113.819691] ? watchdog_timer_fn+0x1b4/0x220 [ 7113.823955] ? __hrtimer_run_queues+0x1d3/0x360 [ 7113.828488] ? hrtimer_interrupt+0x104/0x240 [ 7113.832749] ? __local_bh_enable+0x50/0x60 [ 7113.836843] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7113.842068] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7113.847025] [ 7113.849122] [ 7113.851221] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7113.856527] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7113.861931] ? lock_release+0x6/0x260 [ 7113.865589] _raw_spin_unlock+0x1b/0x40 [ 7113.869421] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7113.874648] delayed_work+0x1d8/0x280 [ceph] [ 7113.878931] ? process_scheduled_works+0x22a/0x510 [ 7113.883720] process_scheduled_works+0x2a4/0x510 [ 7113.888336] worker_thread+0x188/0x340 [ 7113.892084] ? rescuer_thread+0x310/0x310 [ 7113.896087] kthread+0xf5/0x130 [ 7113.899226] ? kthread_complete_and_exit+0x20/0x20 [ 7113.904011] ret_from_fork+0x31/0x40 [ 7113.907589] ? kthread_complete_and_exit+0x20/0x20 [ 7113.912373] ret_from_fork_asm+0x11/0x20 [ 7113.916305] [ 7141.591272] watchdog: BUG: soft lockup - CPU#7 stuck for 4534s! [kworker/7:4:78808] [ 7141.598942] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7141.654390] irq event stamp: 11234686 [ 7141.658047] hardirqs last enabled at (11234685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7141.668213] hardirqs last disabled at (11234686): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7141.677938] softirqs last enabled at (11234654): [] __do_softirq+0x2be/0x371 [ 7141.686622] softirqs last disabled at (11234649): [] irq_exit_rcu+0xb0/0xd0 [ 7141.695134] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7141.704511] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7141.711983] Workqueue: events delayed_work [ceph] [ 7141.716706] RIP: 0010:_raw_spin_lock+0x0/0x40 [ 7141.721056] Code: 48 c7 c7 7c ca 15 82 e8 4e a8 57 ff e8 69 5d ff ff 48 89 ea 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 [ 7141.739794] RSP: 0018:ffffc90002117d98 EFLAGS: 00000286 [ 7141.745010] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [ 7141.752134] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [ 7141.759259] RBP: ffff8881a1d34500 R08: 000000000000694b R09: 0000000000000002 [ 7141.766383] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7141.773508] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7141.780633] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7141.788710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7141.794445] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7141.801572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7141.808694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7141.815819] Call Trace: [ 7141.818264] [ 7141.820277] ? watchdog_timer_fn+0x1b4/0x220 [ 7141.824549] ? __hrtimer_run_queues+0x1d3/0x360 [ 7141.829082] ? hrtimer_interrupt+0x104/0x240 [ 7141.833345] ? __local_bh_enable+0x50/0x60 [ 7141.837436] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7141.842661] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7141.847618] [ 7141.849715] [ 7141.851813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7141.857123] ? __sched_text_end+0xa/0xa [ 7141.860958] igrab+0x1d/0x50 [ 7141.863845] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7141.868993] delayed_work+0x1d8/0x280 [ceph] [ 7141.873275] ? process_scheduled_works+0x22a/0x510 [ 7141.878066] process_scheduled_works+0x2a4/0x510 [ 7141.882680] worker_thread+0x188/0x340 [ 7141.886429] ? rescuer_thread+0x310/0x310 [ 7141.890432] kthread+0xf5/0x130 [ 7141.893571] ? kthread_complete_and_exit+0x20/0x20 [ 7141.898354] ret_from_fork+0x31/0x40 [ 7141.901935] ? kthread_complete_and_exit+0x20/0x20 [ 7141.906717] ret_from_fork_asm+0x11/0x20 [ 7141.910649] [ 7143.158698] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [ 7143.167261] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7153.399437] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7164.151540] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 7169.591762] watchdog: BUG: soft lockup - CPU#7 stuck for 4560s! [kworker/7:4:78808] [ 7169.599414] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7169.654863] irq event stamp: 11296274 [ 7169.658520] hardirqs last enabled at (11296273): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7169.668686] hardirqs last disabled at (11296274): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7169.678411] softirqs last enabled at (11296230): [] __do_softirq+0x2be/0x371 [ 7169.687104] softirqs last disabled at (11296225): [] irq_exit_rcu+0xb0/0xd0 [ 7169.695625] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7169.705001] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7169.712471] Workqueue: events delayed_work [ceph] [ 7169.717196] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7169.721547] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7169.740284] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 7169.745499] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 7169.752625] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 7169.759748] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 7169.766874] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 7169.773997] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7169.781122] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7169.789200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7169.794936] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7169.802059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7169.809184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7169.816307] Call Trace: [ 7169.818754] [ 7169.820767] ? watchdog_timer_fn+0x1b4/0x220 [ 7169.825036] ? __hrtimer_run_queues+0x1d3/0x360 [ 7169.829562] ? hrtimer_interrupt+0x104/0x240 [ 7169.833823] ? __local_bh_enable+0x50/0x60 [ 7169.837918] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7169.843142] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7169.848099] [ 7169.850197] [ 7169.852294] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7169.857600] ? lock_acquire+0xf0/0x2b0 [ 7169.861351] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 7169.866065] ? lock_release+0xc5/0x260 [ 7169.869811] _raw_spin_lock+0x30/0x40 [ 7169.873476] ? ceph_check_caps+0x63/0xae0 [ceph] [ 7169.878095] ceph_check_caps+0x63/0xae0 [ceph] [ 7169.882546] ? __lock_acquire+0xbfa/0x1e20 [ 7169.886643] ? lock_acquire+0xcb/0x2b0 [ 7169.890392] ? igrab+0x1d/0x50 [ 7169.893443] ? find_held_lock+0x2d/0x90 [ 7169.897274] ? find_held_lock+0x2d/0x90 [ 7169.901104] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7169.906504] ? lock_release+0xc5/0x260 [ 7169.910250] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7169.915478] delayed_work+0x1d8/0x280 [ceph] [ 7169.919759] ? process_scheduled_works+0x22a/0x510 [ 7169.924547] process_scheduled_works+0x2a4/0x510 [ 7169.929165] worker_thread+0x188/0x340 [ 7169.932914] ? rescuer_thread+0x310/0x310 [ 7169.936917] kthread+0xf5/0x130 [ 7169.940056] ? kthread_complete_and_exit+0x20/0x20 [ 7169.944838] ret_from_fork+0x31/0x40 [ 7169.948419] ? kthread_complete_and_exit+0x20/0x20 [ 7169.953202] ret_from_fork_asm+0x11/0x20 [ 7169.957133] [ 7195.383614] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [ 7197.592259] watchdog: BUG: soft lockup - CPU#7 stuck for 4586s! [kworker/7:4:78808] [ 7197.599912] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7197.655362] irq event stamp: 11357900 [ 7197.659019] hardirqs last enabled at (11357899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7197.669183] hardirqs last disabled at (11357900): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7197.678908] softirqs last enabled at (11357822): [] __do_softirq+0x2be/0x371 [ 7197.687593] softirqs last disabled at (11357817): [] irq_exit_rcu+0xb0/0xd0 [ 7197.696113] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7197.705490] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7197.712963] Workqueue: events delayed_work [ceph] [ 7197.717685] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7197.722036] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7197.740772] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7197.745989] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7197.753115] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7197.760238] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7197.767363] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7197.774487] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7197.781611] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7197.789689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7197.795425] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7197.802549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7197.809674] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7197.816798] Call Trace: [ 7197.819242] [ 7197.821255] ? watchdog_timer_fn+0x1b4/0x220 [ 7197.825526] ? __hrtimer_run_queues+0x1d3/0x360 [ 7197.830051] ? hrtimer_interrupt+0x104/0x240 [ 7197.834313] ? __local_bh_enable+0x50/0x60 [ 7197.838405] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7197.843630] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7197.848588] [ 7197.850685] [ 7197.852783] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7197.858090] ? lock_acquire+0xf0/0x2b0 [ 7197.861842] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7197.867250] _raw_spin_lock+0x30/0x40 [ 7197.870914] ? igrab+0x1d/0x50 [ 7197.873963] igrab+0x1d/0x50 [ 7197.876843] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7197.881984] delayed_work+0x1d8/0x280 [ceph] [ 7197.886265] ? process_scheduled_works+0x22a/0x510 [ 7197.891055] process_scheduled_works+0x2a4/0x510 [ 7197.895670] worker_thread+0x188/0x340 [ 7197.899420] ? rescuer_thread+0x310/0x310 [ 7197.903424] kthread+0xf5/0x130 [ 7197.906570] ? kthread_complete_and_exit+0x20/0x20 [ 7197.911353] ret_from_fork+0x31/0x40 [ 7197.914926] ? kthread_complete_and_exit+0x20/0x20 [ 7197.919718] ret_from_fork_asm+0x11/0x20 [ 7197.923649] [ 7225.592762] watchdog: BUG: soft lockup - CPU#7 stuck for 4612s! [kworker/7:4:78808] [ 7225.600409] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7225.655857] irq event stamp: 11419138 [ 7225.659515] hardirqs last enabled at (11419137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7225.669683] hardirqs last disabled at (11419138): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7225.679416] softirqs last enabled at (11419118): [] __do_softirq+0x2be/0x371 [ 7225.688108] softirqs last disabled at (11419113): [] irq_exit_rcu+0xb0/0xd0 [ 7225.696630] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7225.706005] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7225.713477] Workqueue: events delayed_work [ceph] [ 7225.718201] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7225.722558] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7225.741295] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 7225.746513] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 7225.753638] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 7225.760761] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 7225.767885] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 7225.775011] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7225.782133] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7225.790212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7225.795950] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7225.803073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7225.810197] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7225.817322] Call Trace: [ 7225.819769] [ 7225.821780] ? watchdog_timer_fn+0x1b4/0x220 [ 7225.826051] ? __hrtimer_run_queues+0x1d3/0x360 [ 7225.830586] ? hrtimer_interrupt+0x104/0x240 [ 7225.834855] ? __local_bh_enable+0x50/0x60 [ 7225.838946] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7225.844173] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7225.849128] [ 7225.851226] [ 7225.853323] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7225.858633] ? lock_acquire+0xf0/0x2b0 [ 7225.862382] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7225.867790] ? lock_release+0xc5/0x260 [ 7225.871535] _raw_spin_lock+0x30/0x40 [ 7225.875201] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7225.880599] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7225.885828] delayed_work+0x1d8/0x280 [ceph] [ 7225.890108] ? process_scheduled_works+0x22a/0x510 [ 7225.894899] process_scheduled_works+0x2a4/0x510 [ 7225.899513] worker_thread+0x188/0x340 [ 7225.903263] ? rescuer_thread+0x310/0x310 [ 7225.907266] kthread+0xf5/0x130 [ 7225.910405] ? kthread_complete_and_exit+0x20/0x20 [ 7225.915205] ret_from_fork+0x31/0x40 [ 7225.918785] ? kthread_complete_and_exit+0x20/0x20 [ 7225.923569] ret_from_fork_asm+0x11/0x20 [ 7225.927499] [ 7253.593237] watchdog: BUG: soft lockup - CPU#7 stuck for 4638s! [kworker/7:4:78808] [ 7253.600889] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7253.656340] irq event stamp: 11479788 [ 7253.659997] hardirqs last enabled at (11479787): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7253.670163] hardirqs last disabled at (11479788): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7253.679887] softirqs last enabled at (11479774): [] __do_softirq+0x2be/0x371 [ 7253.688579] softirqs last disabled at (11479769): [] irq_exit_rcu+0xb0/0xd0 [ 7253.697099] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7253.706476] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7253.713948] Workqueue: events delayed_work [ceph] [ 7253.718671] RIP: 0010:iput+0x0/0x220 [ 7253.722244] Code: e7 02 00 00 48 c7 c7 7c 70 14 82 e8 fa e4 d6 ff e8 95 9c 7e 00 48 8d 7c 24 08 e8 bb fd ff ff e9 81 fe ff ff e8 01 cc 7d 00 90 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 dd 01 00 00 41 55 41 54 [ 7253.740980] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [ 7253.746196] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 7253.753322] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff8881a1d34478 [ 7253.760446] RBP: ffff88817f867000 R08: 0000000000000000 R09: 0000000000000000 [ 7253.767570] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000ea60 [ 7253.774694] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7253.781816] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7253.789895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7253.795634] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7253.802756] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7253.809880] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7253.817005] Call Trace: [ 7253.819450] [ 7253.821463] ? watchdog_timer_fn+0x1b4/0x220 [ 7253.825733] ? __hrtimer_run_queues+0x1d3/0x360 [ 7253.830258] ? hrtimer_interrupt+0x104/0x240 [ 7253.834521] ? __local_bh_enable+0x50/0x60 [ 7253.838614] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7253.843838] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7253.848795] [ 7253.850893] [ 7253.852989] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7253.858296] ? evict_inodes+0x1d0/0x1d0 [ 7253.862126] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 7253.867362] delayed_work+0x1d8/0x280 [ceph] [ 7253.871644] ? process_scheduled_works+0x22a/0x510 [ 7253.876433] process_scheduled_works+0x2a4/0x510 [ 7253.881048] worker_thread+0x188/0x340 [ 7253.884797] ? rescuer_thread+0x310/0x310 [ 7253.888802] kthread+0xf5/0x130 [ 7253.891939] ? kthread_complete_and_exit+0x20/0x20 [ 7253.896724] ret_from_fork+0x31/0x40 [ 7253.900302] ? kthread_complete_and_exit+0x20/0x20 [ 7253.905087] ret_from_fork_asm+0x11/0x20 [ 7253.909019] [ 7281.593729] watchdog: BUG: soft lockup - CPU#7 stuck for 4664s! [kworker/7:4:78808] [ 7281.601380] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7281.656830] irq event stamp: 11540672 [ 7281.660486] hardirqs last enabled at (11540671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7281.670652] hardirqs last disabled at (11540672): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7281.680377] softirqs last enabled at (11540620): [] __do_softirq+0x2be/0x371 [ 7281.689069] softirqs last disabled at (11540615): [] irq_exit_rcu+0xb0/0xd0 [ 7281.697581] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7281.706958] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7281.714429] Workqueue: events delayed_work [ceph] [ 7281.719153] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7281.723504] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7281.742240] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7281.747457] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7281.754582] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7281.761706] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7281.768831] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7281.775955] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7281.783079] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7281.791155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7281.796894] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7281.804017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7281.811142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7281.818266] Call Trace: [ 7281.820711] [ 7281.822722] ? watchdog_timer_fn+0x1b4/0x220 [ 7281.826986] ? __hrtimer_run_queues+0x1d3/0x360 [ 7281.831519] ? hrtimer_interrupt+0x104/0x240 [ 7281.835781] ? __local_bh_enable+0x50/0x60 [ 7281.839875] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7281.845097] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7281.850056] [ 7281.852154] [ 7281.854250] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7281.859559] ? lock_acquire+0xf0/0x2b0 [ 7281.863311] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7281.868716] _raw_spin_lock+0x30/0x40 [ 7281.872375] ? igrab+0x1d/0x50 [ 7281.875425] igrab+0x1d/0x50 [ 7281.878304] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7281.883453] delayed_work+0x1d8/0x280 [ceph] [ 7281.887734] ? process_scheduled_works+0x22a/0x510 [ 7281.892523] process_scheduled_works+0x2a4/0x510 [ 7281.897139] worker_thread+0x188/0x340 [ 7281.900888] ? rescuer_thread+0x310/0x310 [ 7281.904890] kthread+0xf5/0x130 [ 7281.908031] ? kthread_complete_and_exit+0x20/0x20 [ 7281.912821] ret_from_fork+0x31/0x40 [ 7281.916392] ? kthread_complete_and_exit+0x20/0x20 [ 7281.921177] ret_from_fork_asm+0x11/0x20 [ 7281.925108] [ 7309.594220] watchdog: BUG: soft lockup - CPU#7 stuck for 4690s! [kworker/7:4:78808] [ 7309.601870] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7309.657318] irq event stamp: 11602392 [ 7309.660976] hardirqs last enabled at (11602391): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7309.671141] hardirqs last disabled at (11602392): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7309.680867] softirqs last enabled at (11602372): [] __do_softirq+0x2be/0x371 [ 7309.689560] softirqs last disabled at (11602367): [] irq_exit_rcu+0xb0/0xd0 [ 7309.698081] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7309.707456] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7309.714927] Workqueue: events delayed_work [ceph] [ 7309.719652] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7309.724010] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7309.742746] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7309.747965] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7309.755088] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7309.762213] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7309.769338] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7309.776460] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7309.783585] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7309.791662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7309.797400] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7309.804525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7309.811648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7309.818772] Call Trace: [ 7309.821219] [ 7309.823229] ? watchdog_timer_fn+0x1b4/0x220 [ 7309.827501] ? __hrtimer_run_queues+0x1d3/0x360 [ 7309.832037] ? hrtimer_interrupt+0x104/0x240 [ 7309.836306] ? __local_bh_enable+0x50/0x60 [ 7309.840398] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7309.845623] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7309.850581] [ 7309.852678] [ 7309.854776] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7309.860082] ? lock_acquire+0xf0/0x2b0 [ 7309.863834] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7309.869241] _raw_spin_lock+0x30/0x40 [ 7309.872898] ? igrab+0x1d/0x50 [ 7309.875948] igrab+0x1d/0x50 [ 7309.878827] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7309.883968] delayed_work+0x1d8/0x280 [ceph] [ 7309.888249] ? process_scheduled_works+0x22a/0x510 [ 7309.893037] process_scheduled_works+0x2a4/0x510 [ 7309.897655] worker_thread+0x188/0x340 [ 7309.901404] ? rescuer_thread+0x310/0x310 [ 7309.905407] kthread+0xf5/0x130 [ 7309.908554] ? kthread_complete_and_exit+0x20/0x20 [ 7309.913336] ret_from_fork+0x31/0x40 [ 7309.916917] ? kthread_complete_and_exit+0x20/0x20 [ 7309.921701] ret_from_fork_asm+0x11/0x20 [ 7309.925632] [ 7337.594713] watchdog: BUG: soft lockup - CPU#7 stuck for 4716s! [kworker/7:4:78808] [ 7337.602366] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7337.657816] irq event stamp: 11663464 [ 7337.661473] hardirqs last enabled at (11663463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7337.671640] hardirqs last disabled at (11663464): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7337.681373] softirqs last enabled at (11663444): [] __do_softirq+0x2be/0x371 [ 7337.690064] softirqs last disabled at (11663439): [] irq_exit_rcu+0xb0/0xd0 [ 7337.698587] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7337.707962] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7337.715433] Workqueue: events delayed_work [ceph] [ 7337.720156] RIP: 0010:ceph_check_caps+0x0/0xae0 [ceph] [ 7337.725296] Code: ca 76 1a 48 8b b7 68 08 00 00 31 c0 48 39 f2 76 0c 48 01 c9 48 01 f2 48 39 d1 0f 93 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 <66> 0f 1f 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 89 f5 53 48 [ 7337.744033] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [ 7337.749250] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d6c [ 7337.756375] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8881a1d34478 [ 7337.763500] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 7337.770623] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7337.777747] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7337.784872] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7337.792949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7337.798686] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7337.805810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7337.812934] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7337.820060] Call Trace: [ 7337.822505] [ 7337.824516] ? watchdog_timer_fn+0x1b4/0x220 [ 7337.828788] ? __hrtimer_run_queues+0x1d3/0x360 [ 7337.833321] ? hrtimer_interrupt+0x104/0x240 [ 7337.837584] ? __local_bh_enable+0x50/0x60 [ 7337.841676] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7337.846901] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7337.851859] [ 7337.853954] [ 7337.856052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7337.861362] ? __ceph_should_report_size+0x50/0x50 [ceph] [ 7337.866765] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7337.871994] delayed_work+0x1d8/0x280 [ceph] [ 7337.876274] ? process_scheduled_works+0x22a/0x510 [ 7337.881065] process_scheduled_works+0x2a4/0x510 [ 7337.885681] worker_thread+0x188/0x340 [ 7337.889429] ? rescuer_thread+0x310/0x310 [ 7337.893432] kthread+0xf5/0x130 [ 7337.896570] ? kthread_complete_and_exit+0x20/0x20 [ 7337.901356] ret_from_fork+0x31/0x40 [ 7337.904933] ? kthread_complete_and_exit+0x20/0x20 [ 7337.909717] ret_from_fork_asm+0x11/0x20 [ 7337.913642] [ 7365.595205] watchdog: BUG: soft lockup - CPU#7 stuck for 4742s! [kworker/7:4:78808] [ 7365.602857] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7365.658306] irq event stamp: 11724740 [ 7365.661963] hardirqs last enabled at (11724739): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7365.672129] hardirqs last disabled at (11724740): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7365.681862] softirqs last enabled at (11724694): [] __do_softirq+0x2be/0x371 [ 7365.690556] softirqs last disabled at (11724689): [] irq_exit_rcu+0xb0/0xd0 [ 7365.699074] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7365.708451] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7365.715923] Workqueue: events delayed_work [ceph] [ 7365.720648] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7365.725005] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7365.743744] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 7365.748960] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 7365.756084] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 7365.763210] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 7365.770333] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 7365.777457] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7365.784581] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7365.792658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7365.798396] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7365.805521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7365.812643] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7365.819767] Call Trace: [ 7365.822214] [ 7365.824225] ? watchdog_timer_fn+0x1b4/0x220 [ 7365.828488] ? __hrtimer_run_queues+0x1d3/0x360 [ 7365.833014] ? hrtimer_interrupt+0x104/0x240 [ 7365.837276] ? __local_bh_enable+0x50/0x60 [ 7365.841367] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7365.846594] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7365.851549] [ 7365.853646] [ 7365.855744] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7365.861053] ? lock_acquire+0xf0/0x2b0 [ 7365.864804] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 7365.869517] ? lock_release+0xc5/0x260 [ 7365.873261] _raw_spin_lock+0x30/0x40 [ 7365.876917] ? ceph_check_caps+0x63/0xae0 [ceph] [ 7365.881536] ceph_check_caps+0x63/0xae0 [ceph] [ 7365.885988] ? __lock_acquire+0xbfa/0x1e20 [ 7365.890087] ? lock_acquire+0xcb/0x2b0 [ 7365.893835] ? igrab+0x1d/0x50 [ 7365.896886] ? find_held_lock+0x2d/0x90 [ 7365.900717] ? find_held_lock+0x2d/0x90 [ 7365.904547] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7365.909946] ? lock_release+0xc5/0x260 [ 7365.913692] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7365.918920] delayed_work+0x1d8/0x280 [ceph] [ 7365.923200] ? process_scheduled_works+0x22a/0x510 [ 7365.927990] process_scheduled_works+0x2a4/0x510 [ 7365.932607] worker_thread+0x188/0x340 [ 7365.936355] ? rescuer_thread+0x310/0x310 [ 7365.940359] kthread+0xf5/0x130 [ 7365.943498] ? kthread_complete_and_exit+0x20/0x20 [ 7365.948280] ret_from_fork+0x31/0x40 [ 7365.951851] ? kthread_complete_and_exit+0x20/0x20 [ 7365.956634] ret_from_fork_asm+0x11/0x20 [ 7365.960559] [ 7393.595711] watchdog: BUG: soft lockup - CPU#7 stuck for 4768s! [kworker/7:4:78808] [ 7393.603363] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7393.658812] irq event stamp: 11785564 [ 7393.662470] hardirqs last enabled at (11785563): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7393.672636] hardirqs last disabled at (11785564): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7393.682361] softirqs last enabled at (11785486): [] __do_softirq+0x2be/0x371 [ 7393.691054] softirqs last disabled at (11785481): [] irq_exit_rcu+0xb0/0xd0 [ 7393.699575] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7393.708957] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7393.716429] Workqueue: events delayed_work [ceph] [ 7393.721155] RIP: 0010:lock_acquire+0x85/0x2b0 [ 7393.725513] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [ 7393.744251] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [ 7393.749467] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [ 7393.756590] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [ 7393.763715] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 7393.770840] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7393.777964] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7393.785088] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7393.793164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7393.798903] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7393.806027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7393.813150] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7393.820275] Call Trace: [ 7393.822723] [ 7393.824733] ? watchdog_timer_fn+0x1b4/0x220 [ 7393.829003] ? __hrtimer_run_queues+0x1d3/0x360 [ 7393.833530] ? hrtimer_interrupt+0x104/0x240 [ 7393.837800] ? __local_bh_enable+0x50/0x60 [ 7393.841892] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7393.847117] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7393.852075] [ 7393.854170] [ 7393.856269] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7393.861576] ? lock_acquire+0x85/0x2b0 [ 7393.865327] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7393.870736] _raw_spin_lock+0x30/0x40 [ 7393.874400] ? igrab+0x1d/0x50 [ 7393.877450] igrab+0x1d/0x50 [ 7393.880328] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7393.885471] delayed_work+0x1d8/0x280 [ceph] [ 7393.889752] ? process_scheduled_works+0x22a/0x510 [ 7393.894540] process_scheduled_works+0x2a4/0x510 [ 7393.899156] worker_thread+0x188/0x340 [ 7393.902904] ? rescuer_thread+0x310/0x310 [ 7393.906908] kthread+0xf5/0x130 [ 7393.910046] ? kthread_complete_and_exit+0x20/0x20 [ 7393.914830] ret_from_fork+0x31/0x40 [ 7393.918400] ? kthread_complete_and_exit+0x20/0x20 [ 7393.923185] ret_from_fork_asm+0x11/0x20 [ 7393.927108] [ 7421.596192] watchdog: BUG: soft lockup - CPU#7 stuck for 4794s! [kworker/7:4:78808] [ 7421.603843] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7421.659292] irq event stamp: 11846552 [ 7421.662949] hardirqs last enabled at (11846551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7421.673117] hardirqs last disabled at (11846552): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7421.682848] softirqs last enabled at (11846532): [] __do_softirq+0x2be/0x371 [ 7421.691532] softirqs last disabled at (11846527): [] irq_exit_rcu+0xb0/0xd0 [ 7421.700055] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7421.709439] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7421.716912] Workqueue: events delayed_work [ceph] [ 7421.721635] RIP: 0010:rcu_is_watching+0x0/0x40 [ 7421.726080] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [ 7421.744816] RSP: 0018:ffffc90002117d28 EFLAGS: 00000247 [ 7421.750034] RAX: 0000000000000007 RBX: ffff88817f867488 RCX: 0000000000000000 [ 7421.757159] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88817f8674a0 [ 7421.764283] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 7421.771406] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [ 7421.778532] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7421.785655] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7421.793731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7421.799470] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7421.806595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7421.813718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7421.820842] Call Trace: [ 7421.823287] [ 7421.825299] ? watchdog_timer_fn+0x1b4/0x220 [ 7421.829570] ? __hrtimer_run_queues+0x1d3/0x360 [ 7421.834096] ? hrtimer_interrupt+0x104/0x240 [ 7421.838359] ? __local_bh_enable+0x50/0x60 [ 7421.842452] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7421.847675] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7421.852632] [ 7421.854730] [ 7421.856827] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7421.862135] ? rcutorture_get_gp_data+0x30/0x30 [ 7421.866664] ? find_held_lock+0x2d/0x90 [ 7421.870502] lock_acquire+0x184/0x2b0 [ 7421.874161] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7421.879570] ? lock_release+0xc5/0x260 [ 7421.883314] _raw_spin_lock+0x30/0x40 [ 7421.886972] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7421.892380] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7421.897616] delayed_work+0x1d8/0x280 [ceph] [ 7421.901899] ? process_scheduled_works+0x22a/0x510 [ 7421.906688] process_scheduled_works+0x2a4/0x510 [ 7421.911303] worker_thread+0x188/0x340 [ 7421.915052] ? rescuer_thread+0x310/0x310 [ 7421.919056] kthread+0xf5/0x130 [ 7421.922202] ? kthread_complete_and_exit+0x20/0x20 [ 7421.926987] ret_from_fork+0x31/0x40 [ 7421.930565] ? kthread_complete_and_exit+0x20/0x20 [ 7421.935349] ret_from_fork_asm+0x11/0x20 [ 7421.939280] [ 7449.596684] watchdog: BUG: soft lockup - CPU#7 stuck for 4820s! [kworker/7:4:78808] [ 7449.604349] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7449.659800] irq event stamp: 11907832 [ 7449.663457] hardirqs last enabled at (11907831): [] asm_common_interrupt+0x26/0x40 [ 7449.672668] hardirqs last disabled at (11907832): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7449.682394] softirqs last enabled at (11907830): [] __do_softirq+0x2be/0x371 [ 7449.691078] softirqs last disabled at (11907825): [] irq_exit_rcu+0xb0/0xd0 [ 7449.699600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7449.708976] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7449.716447] Workqueue: events delayed_work [ceph] [ 7449.721170] RIP: 0010:_raw_spin_lock+0x14/0x40 [ 7449.725608] Code: 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 [ 7449.744343] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [ 7449.749562] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [ 7449.756685] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [ 7449.763809] RBP: ffff8881a1d34500 R08: 000000000000694b R09: 0000000000000002 [ 7449.770933] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7449.778057] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7449.785181] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7449.793259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7449.798998] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7449.806123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7449.813246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7449.820380] Call Trace: [ 7449.822825] [ 7449.824835] ? watchdog_timer_fn+0x1b4/0x220 [ 7449.829107] ? __hrtimer_run_queues+0x1d3/0x360 [ 7449.833641] ? hrtimer_interrupt+0x104/0x240 [ 7449.837903] ? __local_bh_enable+0x50/0x60 [ 7449.841996] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7449.847220] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7449.852176] [ 7449.854274] [ 7449.856382] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7449.861687] ? _raw_spin_lock+0x14/0x40 [ 7449.865526] igrab+0x1d/0x50 [ 7449.868413] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7449.873563] delayed_work+0x1d8/0x280 [ceph] [ 7449.877841] ? process_scheduled_works+0x22a/0x510 [ 7449.882623] process_scheduled_works+0x2a4/0x510 [ 7449.887240] worker_thread+0x188/0x340 [ 7449.890987] ? rescuer_thread+0x310/0x310 [ 7449.894991] kthread+0xf5/0x130 [ 7449.898129] ? kthread_complete_and_exit+0x20/0x20 [ 7449.902914] ret_from_fork+0x31/0x40 [ 7449.906494] ? kthread_complete_and_exit+0x20/0x20 [ 7449.911277] ret_from_fork_asm+0x11/0x20 [ 7449.915209] [ 7477.597177] watchdog: BUG: soft lockup - CPU#7 stuck for 4847s! [kworker/7:4:78808] [ 7477.604831] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7477.660280] irq event stamp: 11969568 [ 7477.663938] hardirqs last enabled at (11969567): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7477.674102] hardirqs last disabled at (11969568): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7477.683829] softirqs last enabled at (11969566): [] __do_softirq+0x2be/0x371 [ 7477.692523] softirqs last disabled at (11969557): [] irq_exit_rcu+0xb0/0xd0 [ 7477.701043] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7477.710427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7477.717899] Workqueue: events delayed_work [ceph] [ 7477.722621] RIP: 0010:rcu_is_watching+0x0/0x40 [ 7477.727059] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [ 7477.745795] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [ 7477.751012] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [ 7477.758138] RDX: 0000000000000001 RSI: ffffffff813b9425 RDI: ffff8881a1d34518 [ 7477.765260] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [ 7477.772385] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9425 [ 7477.779509] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7477.786634] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7477.794711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7477.800450] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7477.807572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7477.814697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7477.821822] Call Trace: [ 7477.824267] [ 7477.826278] ? watchdog_timer_fn+0x1b4/0x220 [ 7477.830540] ? __hrtimer_run_queues+0x1d3/0x360 [ 7477.835068] ? hrtimer_interrupt+0x104/0x240 [ 7477.839336] ? __local_bh_enable+0x50/0x60 [ 7477.843430] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7477.848655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7477.853610] [ 7477.855709] [ 7477.857805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7477.863110] ? igrab+0x35/0x50 [ 7477.866163] ? igrab+0x35/0x50 [ 7477.869211] ? rcutorture_get_gp_data+0x30/0x30 [ 7477.873737] lock_release+0x1d7/0x260 [ 7477.877405] _raw_spin_unlock+0x1b/0x40 [ 7477.881241] igrab+0x35/0x50 [ 7477.884119] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7477.889269] delayed_work+0x1d8/0x280 [ceph] [ 7477.893553] ? process_scheduled_works+0x22a/0x510 [ 7477.898341] process_scheduled_works+0x2a4/0x510 [ 7477.902955] worker_thread+0x188/0x340 [ 7477.906706] ? rescuer_thread+0x310/0x310 [ 7477.910709] kthread+0xf5/0x130 [ 7477.913855] ? kthread_complete_and_exit+0x20/0x20 [ 7477.918640] ret_from_fork+0x31/0x40 [ 7477.922220] ? kthread_complete_and_exit+0x20/0x20 [ 7477.927011] ret_from_fork_asm+0x11/0x20 [ 7477.930944] [ 7505.597670] watchdog: BUG: soft lockup - CPU#7 stuck for 4873s! [kworker/7:4:78808] [ 7505.605320] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7505.660768] irq event stamp: 12031848 [ 7505.664428] hardirqs last enabled at (12031847): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7505.674592] hardirqs last disabled at (12031848): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7505.684316] softirqs last enabled at (12031832): [] __do_softirq+0x2be/0x371 [ 7505.693001] softirqs last disabled at (12031827): [] irq_exit_rcu+0xb0/0xd0 [ 7505.701514] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7505.710889] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7505.718361] Workqueue: events delayed_work [ceph] [ 7505.723075] RIP: 0010:lock_release+0x14e/0x260 [ 7505.727514] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 7505.746251] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [ 7505.751468] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [ 7505.758592] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [ 7505.765717] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 7505.772841] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 7505.779963] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [ 7505.787088] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7505.795165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7505.800902] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7505.808026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7505.815151] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7505.822274] Call Trace: [ 7505.824720] [ 7505.826731] ? watchdog_timer_fn+0x1b4/0x220 [ 7505.830995] ? __hrtimer_run_queues+0x1d3/0x360 [ 7505.835522] ? hrtimer_interrupt+0x104/0x240 [ 7505.839790] ? __local_bh_enable+0x50/0x60 [ 7505.843884] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7505.849108] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7505.854067] [ 7505.856164] [ 7505.858261] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7505.863565] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7505.868976] ? lock_release+0x14e/0x260 [ 7505.872805] _raw_spin_unlock+0x1b/0x40 [ 7505.876643] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7505.881872] delayed_work+0x1d8/0x280 [ceph] [ 7505.886154] ? process_scheduled_works+0x22a/0x510 [ 7505.890943] process_scheduled_works+0x2a4/0x510 [ 7505.895558] worker_thread+0x188/0x340 [ 7505.899308] ? rescuer_thread+0x310/0x310 [ 7505.903312] kthread+0xf5/0x130 [ 7505.906450] ? kthread_complete_and_exit+0x20/0x20 [ 7505.911232] ret_from_fork+0x31/0x40 [ 7505.914804] ? kthread_complete_and_exit+0x20/0x20 [ 7505.919588] ret_from_fork_asm+0x11/0x20 [ 7505.923510] [ 7533.598180] watchdog: BUG: soft lockup - CPU#7 stuck for 4899s! [kworker/7:4:78808] [ 7533.605836] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7533.661285] irq event stamp: 12094000 [ 7533.664942] hardirqs last enabled at (12093999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7533.675107] hardirqs last disabled at (12094000): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7533.684833] softirqs last enabled at (12093968): [] __do_softirq+0x2be/0x371 [ 7533.693524] softirqs last disabled at (12093963): [] irq_exit_rcu+0xb0/0xd0 [ 7533.702038] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7533.711413] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7533.718885] Workqueue: events delayed_work [ceph] [ 7533.723609] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7533.727959] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7533.746697] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 7533.751913] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 7533.759037] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 7533.766161] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 7533.773286] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 7533.780411] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7533.787533] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7533.795612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7533.801349] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7533.808473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7533.815597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7533.822722] Call Trace: [ 7533.825169] [ 7533.827179] ? watchdog_timer_fn+0x1b4/0x220 [ 7533.831451] ? __hrtimer_run_queues+0x1d3/0x360 [ 7533.835985] ? hrtimer_interrupt+0x104/0x240 [ 7533.840254] ? __local_bh_enable+0x50/0x60 [ 7533.844347] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7533.849571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7533.854528] [ 7533.856626] [ 7533.858724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7533.864031] ? lock_acquire+0xf0/0x2b0 [ 7533.867782] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7533.873189] ? lock_release+0xc5/0x260 [ 7533.876934] _raw_spin_lock+0x30/0x40 [ 7533.880591] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7533.885989] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7533.891219] delayed_work+0x1d8/0x280 [ceph] [ 7533.895499] ? process_scheduled_works+0x22a/0x510 [ 7533.900291] process_scheduled_works+0x2a4/0x510 [ 7533.904906] worker_thread+0x188/0x340 [ 7533.908654] ? rescuer_thread+0x310/0x310 [ 7533.912658] kthread+0xf5/0x130 [ 7533.915796] ? kthread_complete_and_exit+0x20/0x20 [ 7533.920578] ret_from_fork+0x31/0x40 [ 7533.924152] ? kthread_complete_and_exit+0x20/0x20 [ 7533.928934] ret_from_fork_asm+0x11/0x20 [ 7533.932857] [ 7561.598656] watchdog: BUG: soft lockup - CPU#7 stuck for 4925s! [kworker/7:4:78808] [ 7561.606326] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7561.661775] irq event stamp: 12154992 [ 7561.665430] hardirqs last enabled at (12154991): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7561.675597] hardirqs last disabled at (12154992): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7561.685321] softirqs last enabled at (12154978): [] __do_softirq+0x2be/0x371 [ 7561.694006] softirqs last disabled at (12154973): [] irq_exit_rcu+0xb0/0xd0 [ 7561.702516] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7561.711893] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7561.719365] Workqueue: events delayed_work [ceph] [ 7561.724091] RIP: 0010:igrab+0x2d/0x50 [ 7561.727755] Code: fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 <48> 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 [ 7561.746491] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [ 7561.751711] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 7561.758834] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 7561.765958] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 7561.773082] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7561.780205] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7561.787329] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7561.795408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7561.801146] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7561.808269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7561.815394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7561.822517] Call Trace: [ 7561.824963] [ 7561.826975] ? watchdog_timer_fn+0x1b4/0x220 [ 7561.831245] ? __hrtimer_run_queues+0x1d3/0x360 [ 7561.835773] ? hrtimer_interrupt+0x104/0x240 [ 7561.840043] ? __local_bh_enable+0x50/0x60 [ 7561.844136] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7561.849360] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7561.854316] [ 7561.856414] [ 7561.858512] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7561.863819] ? igrab+0x2d/0x50 [ 7561.866876] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7561.872026] delayed_work+0x1d8/0x280 [ceph] [ 7561.876307] ? process_scheduled_works+0x22a/0x510 [ 7561.881096] process_scheduled_works+0x2a4/0x510 [ 7561.885711] worker_thread+0x188/0x340 [ 7561.889461] ? rescuer_thread+0x310/0x310 [ 7561.893464] kthread+0xf5/0x130 [ 7561.896604] ? kthread_complete_and_exit+0x20/0x20 [ 7561.901395] ret_from_fork+0x31/0x40 [ 7561.904975] ? kthread_complete_and_exit+0x20/0x20 [ 7561.909757] ret_from_fork_asm+0x11/0x20 [ 7561.913682] [ 7589.599150] watchdog: BUG: soft lockup - CPU#7 stuck for 4951s! [kworker/7:4:78808] [ 7589.606806] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7589.662256] irq event stamp: 12217818 [ 7589.665912] hardirqs last enabled at (12217817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7589.676078] hardirqs last disabled at (12217818): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7589.685801] softirqs last enabled at (12217764): [] __do_softirq+0x2be/0x371 [ 7589.694488] softirqs last disabled at (12217759): [] irq_exit_rcu+0xb0/0xd0 [ 7589.703006] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7589.712382] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7589.719855] Workqueue: events delayed_work [ceph] [ 7589.724580] RIP: 0010:_raw_spin_unlock+0x0/0x40 [ 7589.729112] Code: 18 bb 01 00 00 00 45 31 c9 41 b8 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 [ 7589.747848] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [ 7589.753066] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: 0000000000000000 [ 7589.760189] RDX: ffff8881a1d34bc8 RSI: 0000000000000000 RDI: ffff8881a1d34920 [ 7589.767314] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 7589.774438] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 7589.781563] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 7589.788686] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7589.796765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7589.802500] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7589.809625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7589.816749] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7589.823873] Call Trace: [ 7589.826319] [ 7589.828332] ? watchdog_timer_fn+0x1b4/0x220 [ 7589.832601] ? __hrtimer_run_queues+0x1d3/0x360 [ 7589.837129] ? hrtimer_interrupt+0x104/0x240 [ 7589.841390] ? __local_bh_enable+0x50/0x60 [ 7589.845482] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7589.850708] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7589.855663] [ 7589.857760] [ 7589.859858] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7589.865167] ? _raw_spin_trylock_bh+0x70/0x70 [ 7589.869523] ceph_check_caps+0x26b/0xae0 [ceph] [ 7589.874074] ? lock_acquire+0xcb/0x2b0 [ 7589.877816] ? igrab+0x1d/0x50 [ 7589.880868] ? find_held_lock+0x2d/0x90 [ 7589.884698] ? find_held_lock+0x2d/0x90 [ 7589.888530] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7589.893929] ? lock_release+0xc5/0x260 [ 7589.897673] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7589.902902] delayed_work+0x1d8/0x280 [ceph] [ 7589.907182] ? process_scheduled_works+0x22a/0x510 [ 7589.911973] process_scheduled_works+0x2a4/0x510 [ 7589.916588] worker_thread+0x188/0x340 [ 7589.920338] ? rescuer_thread+0x310/0x310 [ 7589.924349] kthread+0xf5/0x130 [ 7589.927495] ? kthread_complete_and_exit+0x20/0x20 [ 7589.932280] ret_from_fork+0x31/0x40 [ 7589.935858] ? kthread_complete_and_exit+0x20/0x20 [ 7589.940641] ret_from_fork_asm+0x11/0x20 [ 7589.944566] [ 7617.599655] watchdog: BUG: soft lockup - CPU#7 stuck for 4977s! [kworker/7:4:78808] [ 7617.607303] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7617.662754] irq event stamp: 12278594 [ 7617.666411] hardirqs last enabled at (12278593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7617.676577] hardirqs last disabled at (12278594): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7617.686310] softirqs last enabled at (12278476): [] __do_softirq+0x2be/0x371 [ 7617.695002] softirqs last disabled at (12278471): [] irq_exit_rcu+0xb0/0xd0 [ 7617.703523] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7617.712900] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7617.720371] Workqueue: events delayed_work [ceph] [ 7617.725095] RIP: 0010:ceph_put_mds_session+0x0/0x90 [ceph] [ 7617.730588] Code: 41 0f b1 08 74 de eb ec 31 c0 5b c3 31 f6 4c 89 c7 e8 64 d7 a8 e0 eb d7 31 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 <66> 0f 1f 00 0f 1f 44 00 00 48 85 ff 74 6e 48 81 ff 00 f0 ff ff 77 [ 7617.749325] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [ 7617.754543] RAX: ffffffffffffffff RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 7617.761666] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: 0000000000000000 [ 7617.768791] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 7617.775916] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 7617.783040] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 7617.790163] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7617.798241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7617.803979] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7617.811103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7617.818228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7617.825352] Call Trace: [ 7617.827797] [ 7617.829808] ? watchdog_timer_fn+0x1b4/0x220 [ 7617.834072] ? __hrtimer_run_queues+0x1d3/0x360 [ 7617.838605] ? hrtimer_interrupt+0x104/0x240 [ 7617.842866] ? __local_bh_enable+0x50/0x60 [ 7617.846961] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7617.852183] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7617.857140] [ 7617.859238] [ 7617.861335] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7617.866643] ? ceph_get_mds_session+0x70/0x70 [ceph] [ 7617.871616] ceph_check_caps+0x275/0xae0 [ceph] [ 7617.876159] ? lock_acquire+0xcb/0x2b0 [ 7617.879901] ? igrab+0x1d/0x50 [ 7617.882950] ? find_held_lock+0x2d/0x90 [ 7617.886781] ? find_held_lock+0x2d/0x90 [ 7617.890613] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7617.896013] ? lock_release+0xc5/0x260 [ 7617.899759] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7617.904986] delayed_work+0x1d8/0x280 [ceph] [ 7617.909266] ? process_scheduled_works+0x22a/0x510 [ 7617.914056] process_scheduled_works+0x2a4/0x510 [ 7617.918672] worker_thread+0x188/0x340 [ 7617.922421] ? rescuer_thread+0x310/0x310 [ 7617.926423] kthread+0xf5/0x130 [ 7617.929562] ? kthread_complete_and_exit+0x20/0x20 [ 7617.934346] ret_from_fork+0x31/0x40 [ 7617.937926] ? kthread_complete_and_exit+0x20/0x20 [ 7617.942708] ret_from_fork_asm+0x11/0x20 [ 7617.946641] [ 7645.600136] watchdog: BUG: soft lockup - CPU#7 stuck for 5003s! [kworker/7:4:78808] [ 7645.607803] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7645.663252] irq event stamp: 12340010 [ 7645.666908] hardirqs last enabled at (12340009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7645.677074] hardirqs last disabled at (12340010): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7645.686799] softirqs last enabled at (12339894): [] __do_softirq+0x2be/0x371 [ 7645.695483] softirqs last disabled at (12339889): [] irq_exit_rcu+0xb0/0xd0 [ 7645.704005] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7645.713379] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7645.720850] Workqueue: events delayed_work [ceph] [ 7645.725574] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7645.729927] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7645.748662] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7645.753880] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7645.761003] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7645.768127] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7645.775252] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7645.782376] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7645.789500] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7645.797577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7645.803316] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7645.810440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7645.817564] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7645.824689] Call Trace: [ 7645.827134] [ 7645.829145] ? watchdog_timer_fn+0x1b4/0x220 [ 7645.833417] ? __hrtimer_run_queues+0x1d3/0x360 [ 7645.837943] ? hrtimer_interrupt+0x104/0x240 [ 7645.842213] ? __local_bh_enable+0x50/0x60 [ 7645.846304] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7645.851528] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7645.856486] [ 7645.858584] [ 7645.860681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7645.865989] ? lock_acquire+0xf0/0x2b0 [ 7645.869740] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7645.875148] _raw_spin_lock+0x30/0x40 [ 7645.878812] ? igrab+0x1d/0x50 [ 7645.881864] igrab+0x1d/0x50 [ 7645.884743] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7645.889891] delayed_work+0x1d8/0x280 [ceph] [ 7645.894172] ? process_scheduled_works+0x22a/0x510 [ 7645.898963] process_scheduled_works+0x2a4/0x510 [ 7645.903578] worker_thread+0x188/0x340 [ 7645.907328] ? rescuer_thread+0x310/0x310 [ 7645.911338] kthread+0xf5/0x130 [ 7645.914478] ? kthread_complete_and_exit+0x20/0x20 [ 7645.919261] ret_from_fork+0x31/0x40 [ 7645.922840] ? kthread_complete_and_exit+0x20/0x20 [ 7645.927623] ret_from_fork_asm+0x11/0x20 [ 7645.931547] [ 7673.600628] watchdog: BUG: soft lockup - CPU#7 stuck for 5029s! [kworker/7:4:78808] [ 7673.608300] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7673.663750] irq event stamp: 12400980 [ 7673.667405] hardirqs last enabled at (12400979): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7673.677573] hardirqs last disabled at (12400980): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7673.687297] softirqs last enabled at (12400960): [] __do_softirq+0x2be/0x371 [ 7673.695990] softirqs last disabled at (12400955): [] irq_exit_rcu+0xb0/0xd0 [ 7673.704502] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7673.713878] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7673.721349] Workqueue: events delayed_work [ceph] [ 7673.726073] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7673.730423] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7673.749160] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 7673.754379] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 7673.761502] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 7673.768627] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 7673.775751] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 7673.782873] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7673.789997] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7673.798076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7673.803813] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7673.810939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7673.818061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7673.825185] Call Trace: [ 7673.827632] [ 7673.829643] ? watchdog_timer_fn+0x1b4/0x220 [ 7673.833915] ? __hrtimer_run_queues+0x1d3/0x360 [ 7673.838449] ? hrtimer_interrupt+0x104/0x240 [ 7673.842711] ? __local_bh_enable+0x50/0x60 [ 7673.846803] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7673.852028] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7673.856984] [ 7673.859082] [ 7673.861181] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7673.866488] ? lock_acquire+0xf0/0x2b0 [ 7673.870238] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7673.875644] ? lock_release+0xc5/0x260 [ 7673.879389] _raw_spin_lock+0x30/0x40 [ 7673.883047] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7673.888445] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7673.893674] delayed_work+0x1d8/0x280 [ceph] [ 7673.897955] ? process_scheduled_works+0x22a/0x510 [ 7673.902744] process_scheduled_works+0x2a4/0x510 [ 7673.907360] worker_thread+0x188/0x340 [ 7673.911111] ? rescuer_thread+0x310/0x310 [ 7673.915122] kthread+0xf5/0x130 [ 7673.918267] ? kthread_complete_and_exit+0x20/0x20 [ 7673.923052] ret_from_fork+0x31/0x40 [ 7673.926632] ? kthread_complete_and_exit+0x20/0x20 [ 7673.931417] ret_from_fork_asm+0x11/0x20 [ 7673.935348] [ 7701.601128] watchdog: BUG: soft lockup - CPU#7 stuck for 5055s! [kworker/7:4:78808] [ 7701.608782] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7701.664229] irq event stamp: 12462574 [ 7701.667886] hardirqs last enabled at (12462573): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7701.678053] hardirqs last disabled at (12462574): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7701.687778] softirqs last enabled at (12462560): [] __do_softirq+0x2be/0x371 [ 7701.696469] softirqs last disabled at (12462555): [] irq_exit_rcu+0xb0/0xd0 [ 7701.704982] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7701.714359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7701.721832] Workqueue: events delayed_work [ceph] [ 7701.726552] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7701.730903] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7701.749640] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7701.754859] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7701.761982] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7701.769107] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7701.776232] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7701.783354] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7701.790478] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7701.798557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7701.804295] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7701.811419] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7701.818542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7701.825667] Call Trace: [ 7701.828114] [ 7701.830125] ? watchdog_timer_fn+0x1b4/0x220 [ 7701.834395] ? __hrtimer_run_queues+0x1d3/0x360 [ 7701.838920] ? hrtimer_interrupt+0x104/0x240 [ 7701.843182] ? __local_bh_enable+0x50/0x60 [ 7701.847275] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7701.852499] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7701.857456] [ 7701.859553] [ 7701.861651] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7701.866960] ? lock_acquire+0xf0/0x2b0 [ 7701.870711] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7701.876118] _raw_spin_lock+0x30/0x40 [ 7701.879773] ? igrab+0x1d/0x50 [ 7701.882824] igrab+0x1d/0x50 [ 7701.885704] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7701.890855] delayed_work+0x1d8/0x280 [ceph] [ 7701.895134] ? process_scheduled_works+0x22a/0x510 [ 7701.899923] process_scheduled_works+0x2a4/0x510 [ 7701.904540] worker_thread+0x188/0x340 [ 7701.908288] ? rescuer_thread+0x310/0x310 [ 7701.912291] kthread+0xf5/0x130 [ 7701.915431] ? kthread_complete_and_exit+0x20/0x20 [ 7701.920213] ret_from_fork+0x31/0x40 [ 7701.923784] ? kthread_complete_and_exit+0x20/0x20 [ 7701.928567] ret_from_fork_asm+0x11/0x20 [ 7701.932491] [ 7729.601626] watchdog: BUG: soft lockup - CPU#7 stuck for 5081s! [kworker/7:4:78808] [ 7729.609279] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7729.664726] irq event stamp: 12524904 [ 7729.668385] hardirqs last enabled at (12524903): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7729.678551] hardirqs last disabled at (12524904): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7729.688275] softirqs last enabled at (12524884): [] __do_softirq+0x2be/0x371 [ 7729.696968] softirqs last disabled at (12524879): [] irq_exit_rcu+0xb0/0xd0 [ 7729.705479] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7729.714857] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7729.722327] Workqueue: events delayed_work [ceph] [ 7729.727051] RIP: 0010:__cap_is_valid+0x44/0xc0 [ceph] [ 7729.732105] Code: 00 44 39 e5 72 1e 48 8b 0d c9 b8 8d e1 b8 01 00 00 00 48 3b 8a e8 08 00 00 79 09 5b 5d 41 5c 41 5d 41 5e c3 66 90 5b 31 c0 5d <41> 5c 41 5d 41 5e c3 48 8b 02 4c 8b 2f 48 89 fb 8b 7f 40 48 8b 00 [ 7729.750841] RSP: 0018:ffffc90002117b78 EFLAGS: 00000246 [ 7729.756057] RAX: 0000000000000000 RBX: ffff88810da9e000 RCX: 00000001007158c0 [ 7729.763181] RDX: ffff888157b28000 RSI: ffffc90002117c64 RDI: ffff88810da9e000 [ 7729.770307] RBP: ffffc90002117c64 R08: 0000000000000000 R09: 0000000000000000 [ 7729.777430] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 7729.784554] R13: ffff8881a1d34478 R14: ffff888100ee2000 R15: ffff88810da9e008 [ 7729.791680] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7729.799755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7729.805495] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7729.812619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7729.819741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7729.826867] Call Trace: [ 7729.829312] [ 7729.831322] ? watchdog_timer_fn+0x1b4/0x220 [ 7729.835587] ? __hrtimer_run_queues+0x1d3/0x360 [ 7729.840121] ? hrtimer_interrupt+0x104/0x240 [ 7729.844390] ? __local_bh_enable+0x50/0x60 [ 7729.848484] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7729.853708] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7729.858664] [ 7729.860762] [ 7729.862861] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7729.868168] ? __cap_is_valid+0x44/0xc0 [ceph] [ 7729.872619] __ceph_caps_issued+0x5a/0x110 [ceph] [ 7729.877328] ceph_check_caps+0x154/0xae0 [ceph] [ 7729.881876] ? lock_acquire+0xcb/0x2b0 [ 7729.885619] ? igrab+0x1d/0x50 [ 7729.888669] ? find_held_lock+0x2d/0x90 [ 7729.892500] ? find_held_lock+0x2d/0x90 [ 7729.896333] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7729.901739] ? lock_release+0xc5/0x260 [ 7729.905484] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7729.910713] delayed_work+0x1d8/0x280 [ceph] [ 7729.914994] ? process_scheduled_works+0x22a/0x510 [ 7729.919782] process_scheduled_works+0x2a4/0x510 [ 7729.924399] worker_thread+0x188/0x340 [ 7729.928147] ? rescuer_thread+0x310/0x310 [ 7729.932151] kthread+0xf5/0x130 [ 7729.935289] ? kthread_complete_and_exit+0x20/0x20 [ 7729.940074] ret_from_fork+0x31/0x40 [ 7729.943654] ? kthread_complete_and_exit+0x20/0x20 [ 7729.948437] ret_from_fork_asm+0x11/0x20 [ 7729.952367] [ 7757.602107] watchdog: BUG: soft lockup - CPU#7 stuck for 5107s! [kworker/7:4:78808] [ 7757.609760] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7757.665208] irq event stamp: 12586116 [ 7757.668866] hardirqs last enabled at (12586115): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7757.679032] hardirqs last disabled at (12586116): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7757.688765] softirqs last enabled at (12586008): [] __do_softirq+0x2be/0x371 [ 7757.697457] softirqs last disabled at (12586003): [] irq_exit_rcu+0xb0/0xd0 [ 7757.705978] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7757.715355] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7757.722827] Workqueue: events delayed_work [ceph] [ 7757.727549] RIP: 0010:lock_release+0x14e/0x260 [ 7757.731987] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 7757.750725] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [ 7757.755941] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [ 7757.763066] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [ 7757.770189] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 7757.777313] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 7757.784438] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [ 7757.791562] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7757.799638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7757.805376] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7757.812501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7757.819625] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7757.826748] Call Trace: [ 7757.829196] [ 7757.831206] ? watchdog_timer_fn+0x1b4/0x220 [ 7757.835470] ? __hrtimer_run_queues+0x1d3/0x360 [ 7757.839996] ? hrtimer_interrupt+0x104/0x240 [ 7757.844264] ? __local_bh_enable+0x50/0x60 [ 7757.848357] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7757.853582] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7757.858540] [ 7757.860636] [ 7757.862734] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7757.868040] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 7757.872755] ? lock_release+0x14e/0x260 [ 7757.876584] ? lock_release+0xc5/0x260 [ 7757.880330] _raw_spin_unlock+0x1b/0x40 [ 7757.884170] ceph_check_caps+0x26b/0xae0 [ceph] [ 7757.888717] ? lock_acquire+0xcb/0x2b0 [ 7757.892462] ? igrab+0x1d/0x50 [ 7757.895512] ? find_held_lock+0x2d/0x90 [ 7757.899352] ? find_held_lock+0x2d/0x90 [ 7757.903182] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7757.908583] ? lock_release+0xc5/0x260 [ 7757.912329] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7757.917555] delayed_work+0x1d8/0x280 [ceph] [ 7757.921835] ? process_scheduled_works+0x22a/0x510 [ 7757.926626] process_scheduled_works+0x2a4/0x510 [ 7757.931240] worker_thread+0x188/0x340 [ 7757.934989] ? rescuer_thread+0x310/0x310 [ 7757.938994] kthread+0xf5/0x130 [ 7757.942133] ? kthread_complete_and_exit+0x20/0x20 [ 7757.946925] ret_from_fork+0x31/0x40 [ 7757.950496] ? kthread_complete_and_exit+0x20/0x20 [ 7757.955280] ret_from_fork_asm+0x11/0x20 [ 7757.959210] [ 7785.602601] watchdog: BUG: soft lockup - CPU#7 stuck for 5133s! [kworker/7:4:78808] [ 7785.610258] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7785.665705] irq event stamp: 12646742 [ 7785.669364] hardirqs last enabled at (12646741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7785.679528] hardirqs last disabled at (12646742): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7785.689253] softirqs last enabled at (12646646): [] __do_softirq+0x2be/0x371 [ 7785.697937] softirqs last disabled at (12646641): [] irq_exit_rcu+0xb0/0xd0 [ 7785.706457] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7785.715835] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7785.723307] Workqueue: events delayed_work [ceph] [ 7785.728030] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7785.732380] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7785.751118] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 7785.756334] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 7785.763459] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 7785.770583] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 7785.777708] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 7785.784831] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7785.791955] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7785.800034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7785.805770] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7785.812896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7785.820019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7785.827144] Call Trace: [ 7785.829589] [ 7785.831600] ? watchdog_timer_fn+0x1b4/0x220 [ 7785.835872] ? __hrtimer_run_queues+0x1d3/0x360 [ 7785.840405] ? hrtimer_interrupt+0x104/0x240 [ 7785.844667] ? __local_bh_enable+0x50/0x60 [ 7785.848761] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7785.853986] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7785.858940] [ 7785.861038] [ 7785.863137] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7785.868445] ? lock_acquire+0xf0/0x2b0 [ 7785.872195] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7785.877602] ? lock_release+0xc5/0x260 [ 7785.881347] _raw_spin_lock+0x30/0x40 [ 7785.885013] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7785.890412] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 7785.895639] delayed_work+0x1d8/0x280 [ceph] [ 7785.899922] ? process_scheduled_works+0x22a/0x510 [ 7785.904712] process_scheduled_works+0x2a4/0x510 [ 7785.909326] worker_thread+0x188/0x340 [ 7785.913077] ? rescuer_thread+0x310/0x310 [ 7785.917080] kthread+0xf5/0x130 [ 7785.920226] ? kthread_complete_and_exit+0x20/0x20 [ 7785.925010] ret_from_fork+0x31/0x40 [ 7785.928588] ? kthread_complete_and_exit+0x20/0x20 [ 7785.933372] ret_from_fork_asm+0x11/0x20 [ 7785.937295] [ 7813.603100] watchdog: BUG: soft lockup - CPU#7 stuck for 5159s! [kworker/7:4:78808] [ 7813.610755] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7813.666203] irq event stamp: 12707216 [ 7813.669861] hardirqs last enabled at (12707215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7813.680026] hardirqs last disabled at (12707216): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7813.689751] softirqs last enabled at (12707202): [] __do_softirq+0x2be/0x371 [ 7813.698446] softirqs last disabled at (12707197): [] irq_exit_rcu+0xb0/0xd0 [ 7813.706966] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7813.716342] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7813.723813] Workqueue: events delayed_work [ceph] [ 7813.728535] RIP: 0010:__list_del_entry_valid_or_report+0x49/0xd0 [ 7813.734535] Code: 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 48 39 f2 75 73 b8 01 00 00 00 48 89 fe 48 c7 c7 98 99 1d 82 e8 07 0c b0 ff 0f 0b 31 c0 c3 48 [ 7813.753270] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [ 7813.758488] RAX: 0000000000000001 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [ 7813.765612] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [ 7813.772736] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 7813.779861] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7813.786985] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7813.794108] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7813.802187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7813.807923] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7813.815048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7813.822172] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7813.829297] Call Trace: [ 7813.831741] [ 7813.833754] ? watchdog_timer_fn+0x1b4/0x220 [ 7813.838023] ? __hrtimer_run_queues+0x1d3/0x360 [ 7813.842551] ? hrtimer_interrupt+0x104/0x240 [ 7813.846821] ? __local_bh_enable+0x50/0x60 [ 7813.850913] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7813.856138] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7813.861095] [ 7813.863193] [ 7813.865290] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7813.870596] ? __list_del_entry_valid_or_report+0x49/0xd0 [ 7813.875995] ceph_check_delayed_caps+0x8f/0x270 [ceph] [ 7813.881146] delayed_work+0x1d8/0x280 [ceph] [ 7813.885426] ? process_scheduled_works+0x22a/0x510 [ 7813.890216] process_scheduled_works+0x2a4/0x510 [ 7813.894831] worker_thread+0x188/0x340 [ 7813.898580] ? rescuer_thread+0x310/0x310 [ 7813.902583] kthread+0xf5/0x130 [ 7813.905722] ? kthread_complete_and_exit+0x20/0x20 [ 7813.910505] ret_from_fork+0x31/0x40 [ 7813.914085] ? kthread_complete_and_exit+0x20/0x20 [ 7813.918869] ret_from_fork_asm+0x11/0x20 [ 7813.922800] [ 7841.603587] watchdog: BUG: soft lockup - CPU#7 stuck for 5186s! [kworker/7:4:78808] [ 7841.611234] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7841.666686] irq event stamp: 12767780 [ 7841.670342] hardirqs last enabled at (12767779): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7841.680509] hardirqs last disabled at (12767780): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7841.690242] softirqs last enabled at (12767736): [] __do_softirq+0x2be/0x371 [ 7841.698935] softirqs last disabled at (12767731): [] irq_exit_rcu+0xb0/0xd0 [ 7841.707456] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7841.716830] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7841.724303] Workqueue: events delayed_work [ceph] [ 7841.729027] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7841.733386] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7841.752123] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 7841.757339] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 7841.764462] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 7841.771589] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 7841.778712] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 7841.785836] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7841.792959] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7841.801038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7841.806776] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7841.813899] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7841.821022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7841.828146] Call Trace: [ 7841.830594] [ 7841.832605] ? watchdog_timer_fn+0x1b4/0x220 [ 7841.836875] ? __hrtimer_run_queues+0x1d3/0x360 [ 7841.841403] ? hrtimer_interrupt+0x104/0x240 [ 7841.845672] ? __local_bh_enable+0x50/0x60 [ 7841.849765] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7841.854989] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7841.859947] [ 7841.862042] [ 7841.864141] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7841.869448] ? lock_acquire+0xf0/0x2b0 [ 7841.873191] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 7841.877903] ? lock_release+0xc5/0x260 [ 7841.881649] _raw_spin_lock+0x30/0x40 [ 7841.885305] ? ceph_check_caps+0x63/0xae0 [ceph] [ 7841.889925] ceph_check_caps+0x63/0xae0 [ceph] [ 7841.894377] ? __lock_acquire+0xbfa/0x1e20 [ 7841.898473] ? lock_acquire+0xcb/0x2b0 [ 7841.902214] ? igrab+0x1d/0x50 [ 7841.905264] ? find_held_lock+0x2d/0x90 [ 7841.909095] ? find_held_lock+0x2d/0x90 [ 7841.912927] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7841.918327] ? lock_release+0xc5/0x260 [ 7841.922072] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7841.927300] delayed_work+0x1d8/0x280 [ceph] [ 7841.931580] ? process_scheduled_works+0x22a/0x510 [ 7841.936371] process_scheduled_works+0x2a4/0x510 [ 7841.940987] worker_thread+0x188/0x340 [ 7841.944735] ? rescuer_thread+0x310/0x310 [ 7841.948738] kthread+0xf5/0x130 [ 7841.951877] ? kthread_complete_and_exit+0x20/0x20 [ 7841.956668] ret_from_fork+0x31/0x40 [ 7841.960239] ? kthread_complete_and_exit+0x20/0x20 [ 7841.965023] ret_from_fork_asm+0x11/0x20 [ 7841.968946] [ 7869.604079] watchdog: BUG: soft lockup - CPU#7 stuck for 5212s! [kworker/7:4:78808] [ 7869.611724] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7869.667174] irq event stamp: 12828110 [ 7869.670830] hardirqs last enabled at (12828109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7869.680987] hardirqs last disabled at (12828110): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7869.690712] softirqs last enabled at (12828032): [] __do_softirq+0x2be/0x371 [ 7869.699397] softirqs last disabled at (12828027): [] irq_exit_rcu+0xb0/0xd0 [ 7869.707909] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7869.717286] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7869.724756] Workqueue: events delayed_work [ceph] [ 7869.729470] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7869.733823] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7869.752559] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7869.757775] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7869.764901] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7869.772043] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7869.779169] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7869.786301] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7869.793423] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7869.801502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7869.807239] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7869.814364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7869.821496] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7869.828619] Call Trace: [ 7869.831067] [ 7869.833085] ? watchdog_timer_fn+0x1b4/0x220 [ 7869.837356] ? __hrtimer_run_queues+0x1d3/0x360 [ 7869.841884] ? hrtimer_interrupt+0x104/0x240 [ 7869.846153] ? __local_bh_enable+0x50/0x60 [ 7869.850245] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7869.855477] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7869.860437] [ 7869.862533] [ 7869.864629] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7869.869937] ? lock_acquire+0xf0/0x2b0 [ 7869.873690] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7869.879105] _raw_spin_lock+0x30/0x40 [ 7869.882772] ? igrab+0x1d/0x50 [ 7869.885821] igrab+0x1d/0x50 [ 7869.888699] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7869.893840] delayed_work+0x1d8/0x280 [ceph] [ 7869.898122] ? process_scheduled_works+0x22a/0x510 [ 7869.902912] process_scheduled_works+0x2a4/0x510 [ 7869.907527] worker_thread+0x188/0x340 [ 7869.911276] ? rescuer_thread+0x310/0x310 [ 7869.915279] kthread+0xf5/0x130 [ 7869.918419] ? kthread_complete_and_exit+0x20/0x20 [ 7869.923211] ret_from_fork+0x31/0x40 [ 7869.926788] ? kthread_complete_and_exit+0x20/0x20 [ 7869.931574] ret_from_fork_asm+0x11/0x20 [ 7869.935505] [ 7897.604572] watchdog: BUG: soft lockup - CPU#7 stuck for 5238s! [kworker/7:4:78808] [ 7897.612222] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7897.667672] irq event stamp: 12888688 [ 7897.671329] hardirqs last enabled at (12888687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7897.681496] hardirqs last disabled at (12888688): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7897.691227] softirqs last enabled at (12888668): [] __do_softirq+0x2be/0x371 [ 7897.699912] softirqs last disabled at (12888663): [] irq_exit_rcu+0xb0/0xd0 [ 7897.708432] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7897.717810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7897.725281] Workqueue: events delayed_work [ceph] [ 7897.730006] RIP: 0010:igrab+0x2d/0x50 [ 7897.733669] Code: fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 <48> 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 [ 7897.752407] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [ 7897.757625] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [ 7897.764749] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 7897.771874] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 7897.778996] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 7897.786120] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 7897.793244] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7897.801323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7897.807061] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7897.814183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7897.821308] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7897.828432] Call Trace: [ 7897.830880] [ 7897.832889] ? watchdog_timer_fn+0x1b4/0x220 [ 7897.837153] ? __hrtimer_run_queues+0x1d3/0x360 [ 7897.841688] ? hrtimer_interrupt+0x104/0x240 [ 7897.845957] ? __local_bh_enable+0x50/0x60 [ 7897.850049] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7897.855273] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7897.860232] [ 7897.862327] [ 7897.864428] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7897.869733] ? igrab+0x2d/0x50 [ 7897.872783] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7897.877932] delayed_work+0x1d8/0x280 [ceph] [ 7897.882212] ? process_scheduled_works+0x22a/0x510 [ 7897.887003] process_scheduled_works+0x2a4/0x510 [ 7897.891620] worker_thread+0x188/0x340 [ 7897.895369] ? rescuer_thread+0x310/0x310 [ 7897.899380] kthread+0xf5/0x130 [ 7897.902527] ? kthread_complete_and_exit+0x20/0x20 [ 7897.907312] ret_from_fork+0x31/0x40 [ 7897.910889] ? kthread_complete_and_exit+0x20/0x20 [ 7897.915673] ret_from_fork_asm+0x11/0x20 [ 7897.919606] [ 7925.605065] watchdog: BUG: soft lockup - CPU#7 stuck for 5264s! [kworker/7:4:78808] [ 7925.612721] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7925.668169] irq event stamp: 12950798 [ 7925.671826] hardirqs last enabled at (12950797): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7925.681993] hardirqs last disabled at (12950798): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7925.691718] softirqs last enabled at (12950744): [] __do_softirq+0x2be/0x371 [ 7925.700411] softirqs last disabled at (12950739): [] irq_exit_rcu+0xb0/0xd0 [ 7925.708932] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7925.718306] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7925.725778] Workqueue: events delayed_work [ceph] [ 7925.730503] RIP: 0010:lock_acquire+0x18/0x2b0 [ 7925.734862] Code: c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 <49> 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 [ 7925.753599] RSP: 0018:ffffc90002117b90 EFLAGS: 00000246 [ 7925.758815] RAX: ffff8881a1d34920 RBX: ffff8881a1d34920 RCX: 0000000000000000 [ 7925.765940] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34938 [ 7925.773065] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000 [ 7925.780188] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [ 7925.787313] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7925.794436] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7925.802515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7925.808252] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7925.815374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7925.822500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7925.829624] Call Trace: [ 7925.832071] [ 7925.834082] ? watchdog_timer_fn+0x1b4/0x220 [ 7925.838353] ? __hrtimer_run_queues+0x1d3/0x360 [ 7925.842888] ? hrtimer_interrupt+0x104/0x240 [ 7925.847158] ? __local_bh_enable+0x50/0x60 [ 7925.851250] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7925.856475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7925.861431] [ 7925.863528] [ 7925.865625] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7925.870934] ? lock_acquire+0x18/0x2b0 [ 7925.874685] _raw_spin_lock+0x30/0x40 [ 7925.878348] ? ceph_check_caps+0x63/0xae0 [ceph] [ 7925.882970] ceph_check_caps+0x63/0xae0 [ceph] [ 7925.887421] ? __lock_acquire+0xbfa/0x1e20 [ 7925.891517] ? lock_acquire+0xcb/0x2b0 [ 7925.895266] ? igrab+0x1d/0x50 [ 7925.898317] ? find_held_lock+0x2d/0x90 [ 7925.902148] ? find_held_lock+0x2d/0x90 [ 7925.905979] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7925.911378] ? lock_release+0xc5/0x260 [ 7925.915124] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 7925.920351] delayed_work+0x1d8/0x280 [ceph] [ 7925.924632] ? process_scheduled_works+0x22a/0x510 [ 7925.929421] process_scheduled_works+0x2a4/0x510 [ 7925.934039] worker_thread+0x188/0x340 [ 7925.937787] ? rescuer_thread+0x310/0x310 [ 7925.941789] kthread+0xf5/0x130 [ 7925.944929] ? kthread_complete_and_exit+0x20/0x20 [ 7925.949713] ret_from_fork+0x31/0x40 [ 7925.953291] ? kthread_complete_and_exit+0x20/0x20 [ 7925.958076] ret_from_fork_asm+0x11/0x20 [ 7925.962006] [ 7953.605559] watchdog: BUG: soft lockup - CPU#7 stuck for 5290s! [kworker/7:4:78808] [ 7953.613211] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7953.668659] irq event stamp: 13011572 [ 7953.672316] hardirqs last enabled at (13011571): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7953.682481] hardirqs last disabled at (13011572): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7953.692206] softirqs last enabled at (13011476): [] __do_softirq+0x2be/0x371 [ 7953.700890] softirqs last disabled at (13011471): [] irq_exit_rcu+0xb0/0xd0 [ 7953.709410] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7953.718787] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7953.726259] Workqueue: events delayed_work [ceph] [ 7953.730984] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7953.735343] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7953.754079] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7953.759297] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7953.766421] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7953.773544] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7953.780668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7953.787793] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7953.794918] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7953.802995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7953.808732] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7953.815857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7953.822981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7953.830104] Call Trace: [ 7953.832552] [ 7953.834563] ? watchdog_timer_fn+0x1b4/0x220 [ 7953.838833] ? __hrtimer_run_queues+0x1d3/0x360 [ 7953.843360] ? hrtimer_interrupt+0x104/0x240 [ 7953.847629] ? __local_bh_enable+0x50/0x60 [ 7953.851721] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7953.856946] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7953.861903] [ 7953.864000] [ 7953.866099] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7953.871405] ? lock_acquire+0xf0/0x2b0 [ 7953.875156] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7953.880564] _raw_spin_lock+0x30/0x40 [ 7953.884221] ? igrab+0x1d/0x50 [ 7953.887270] igrab+0x1d/0x50 [ 7953.890150] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7953.895292] delayed_work+0x1d8/0x280 [ceph] [ 7953.899573] ? process_scheduled_works+0x22a/0x510 [ 7953.904362] process_scheduled_works+0x2a4/0x510 [ 7953.908977] worker_thread+0x188/0x340 [ 7953.912726] ? rescuer_thread+0x310/0x310 [ 7953.916729] kthread+0xf5/0x130 [ 7953.919869] ? kthread_complete_and_exit+0x20/0x20 [ 7953.924652] ret_from_fork+0x31/0x40 [ 7953.928231] ? kthread_complete_and_exit+0x20/0x20 [ 7953.933013] ret_from_fork_asm+0x11/0x20 [ 7953.936937] [ 7981.606065] watchdog: BUG: soft lockup - CPU#7 stuck for 5316s! [kworker/7:4:78808] [ 7981.613718] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 7981.669166] irq event stamp: 13072030 [ 7981.672823] hardirqs last enabled at (13072029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7981.682989] hardirqs last disabled at (13072030): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 7981.692721] softirqs last enabled at (13072010): [] __do_softirq+0x2be/0x371 [ 7981.701407] softirqs last disabled at (13072005): [] irq_exit_rcu+0xb0/0xd0 [ 7981.709926] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 7981.719303] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 7981.726775] Workqueue: events delayed_work [ceph] [ 7981.731498] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 7981.735848] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 7981.754586] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 7981.759804] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 7981.766928] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 7981.774051] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 7981.781177] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 7981.788301] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 7981.795423] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 7981.803500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7981.809238] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 7981.816364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7981.823487] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7981.830612] Call Trace: [ 7981.833057] [ 7981.835068] ? watchdog_timer_fn+0x1b4/0x220 [ 7981.839338] ? __hrtimer_run_queues+0x1d3/0x360 [ 7981.843867] ? hrtimer_interrupt+0x104/0x240 [ 7981.848136] ? __local_bh_enable+0x50/0x60 [ 7981.852229] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 7981.857453] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 7981.862409] [ 7981.864507] [ 7981.866604] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 7981.871912] ? lock_acquire+0xf0/0x2b0 [ 7981.875664] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 7981.881072] _raw_spin_lock+0x30/0x40 [ 7981.884736] ? igrab+0x1d/0x50 [ 7981.887788] igrab+0x1d/0x50 [ 7981.890666] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 7981.895806] delayed_work+0x1d8/0x280 [ceph] [ 7981.900087] ? process_scheduled_works+0x22a/0x510 [ 7981.904878] process_scheduled_works+0x2a4/0x510 [ 7981.909493] worker_thread+0x188/0x340 [ 7981.913243] ? rescuer_thread+0x310/0x310 [ 7981.917253] kthread+0xf5/0x130 [ 7981.920393] ? kthread_complete_and_exit+0x20/0x20 [ 7981.925183] ret_from_fork+0x31/0x40 [ 7981.928755] ? kthread_complete_and_exit+0x20/0x20 [ 7981.933538] ret_from_fork_asm+0x11/0x20 [ 7981.937461] [ 8009.606544] watchdog: BUG: soft lockup - CPU#7 stuck for 5342s! [kworker/7:4:78808] [ 8009.614197] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8009.669645] irq event stamp: 13132624 [ 8009.673304] hardirqs last enabled at (13132623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8009.683469] hardirqs last disabled at (13132624): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8009.693195] softirqs last enabled at (13132604): [] __do_softirq+0x2be/0x371 [ 8009.701886] softirqs last disabled at (13132599): [] irq_exit_rcu+0xb0/0xd0 [ 8009.710399] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8009.719775] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8009.727246] Workqueue: events delayed_work [ceph] [ 8009.731971] RIP: 0010:lock_release+0x14e/0x260 [ 8009.736415] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 8009.755152] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [ 8009.760370] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [ 8009.767496] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [ 8009.774620] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [ 8009.781743] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [ 8009.788866] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [ 8009.795991] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8009.804068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8009.809805] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8009.816929] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8009.824055] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8009.831179] Call Trace: [ 8009.833623] [ 8009.835636] ? watchdog_timer_fn+0x1b4/0x220 [ 8009.839906] ? __hrtimer_run_queues+0x1d3/0x360 [ 8009.844432] ? hrtimer_interrupt+0x104/0x240 [ 8009.848694] ? __local_bh_enable+0x50/0x60 [ 8009.852787] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8009.858012] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8009.862967] [ 8009.865065] [ 8009.867164] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8009.872467] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8009.877184] ? lock_release+0x14e/0x260 [ 8009.881014] ? lock_release+0xc5/0x260 [ 8009.884759] _raw_spin_unlock+0x1b/0x40 [ 8009.888598] ceph_check_caps+0x26b/0xae0 [ceph] [ 8009.893139] ? lock_acquire+0xcb/0x2b0 [ 8009.896881] ? igrab+0x1d/0x50 [ 8009.899931] ? find_held_lock+0x2d/0x90 [ 8009.903762] ? find_held_lock+0x2d/0x90 [ 8009.907594] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8009.912995] ? lock_release+0xc5/0x260 [ 8009.916738] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8009.921966] delayed_work+0x1d8/0x280 [ceph] [ 8009.926248] ? process_scheduled_works+0x22a/0x510 [ 8009.931038] process_scheduled_works+0x2a4/0x510 [ 8009.935653] worker_thread+0x188/0x340 [ 8009.939401] ? rescuer_thread+0x310/0x310 [ 8009.943406] kthread+0xf5/0x130 [ 8009.946546] ? kthread_complete_and_exit+0x20/0x20 [ 8009.951335] ret_from_fork+0x31/0x40 [ 8009.954906] ? kthread_complete_and_exit+0x20/0x20 [ 8009.959690] ret_from_fork_asm+0x11/0x20 [ 8009.963614] [ 8037.607037] watchdog: BUG: soft lockup - CPU#7 stuck for 5368s! [kworker/7:4:78808] [ 8037.614687] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8037.670136] irq event stamp: 13192832 [ 8037.673794] hardirqs last enabled at (13192831): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8037.683959] hardirqs last disabled at (13192832): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8037.693682] softirqs last enabled at (13192736): [] __do_softirq+0x2be/0x371 [ 8037.702368] softirqs last disabled at (13192731): [] irq_exit_rcu+0xb0/0xd0 [ 8037.710887] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8037.720265] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8037.727737] Workqueue: events delayed_work [ceph] [ 8037.732458] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8037.736811] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8037.755547] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8037.760764] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8037.767887] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8037.775013] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8037.782136] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8037.789261] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8037.796385] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8037.804462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8037.810199] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8037.817325] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8037.824449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8037.831571] Call Trace: [ 8037.834019] [ 8037.836030] ? watchdog_timer_fn+0x1b4/0x220 [ 8037.840302] ? __hrtimer_run_queues+0x1d3/0x360 [ 8037.844826] ? hrtimer_interrupt+0x104/0x240 [ 8037.849089] ? __local_bh_enable+0x50/0x60 [ 8037.853181] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8037.858406] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8037.863362] [ 8037.865460] [ 8037.867558] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8037.872866] ? lock_acquire+0xf0/0x2b0 [ 8037.876616] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8037.882022] ? lock_release+0xc5/0x260 [ 8037.885768] _raw_spin_lock+0x30/0x40 [ 8037.889435] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8037.894841] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8037.900069] delayed_work+0x1d8/0x280 [ceph] [ 8037.904350] ? process_scheduled_works+0x22a/0x510 [ 8037.909140] process_scheduled_works+0x2a4/0x510 [ 8037.913756] worker_thread+0x188/0x340 [ 8037.917505] ? rescuer_thread+0x310/0x310 [ 8037.921507] kthread+0xf5/0x130 [ 8037.924648] ? kthread_complete_and_exit+0x20/0x20 [ 8037.929430] ret_from_fork+0x31/0x40 [ 8037.933002] ? kthread_complete_and_exit+0x20/0x20 [ 8037.937784] ret_from_fork_asm+0x11/0x20 [ 8037.941715] [ 8065.607530] watchdog: BUG: soft lockup - CPU#7 stuck for 5394s! [kworker/7:4:78808] [ 8065.615184] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8065.670633] irq event stamp: 13253244 [ 8065.674289] hardirqs last enabled at (13253243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8065.684456] hardirqs last disabled at (13253244): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8065.694181] softirqs last enabled at (13253230): [] __do_softirq+0x2be/0x371 [ 8065.702866] softirqs last disabled at (13253225): [] irq_exit_rcu+0xb0/0xd0 [ 8065.711386] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8065.720761] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8065.728233] Workqueue: events delayed_work [ceph] [ 8065.732956] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 8065.737568] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 8065.756303] RSP: 0018:ffffc90002117bc0 EFLAGS: 00000246 [ 8065.761521] RAX: 0000000000000000 RBX: ffff8881a1d34920 RCX: 00000000d3c6b569 [ 8065.768645] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34920 [ 8065.775770] RBP: 0000000000000000 R08: 00000000000079a2 R09: 0000000000000002 [ 8065.782894] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 8065.790019] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 8065.797143] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8065.805221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8065.810957] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8065.818081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8065.825205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8065.832329] Call Trace: [ 8065.834776] [ 8065.836787] ? watchdog_timer_fn+0x1b4/0x220 [ 8065.841059] ? __hrtimer_run_queues+0x1d3/0x360 [ 8065.845592] ? hrtimer_interrupt+0x104/0x240 [ 8065.849854] ? __local_bh_enable+0x50/0x60 [ 8065.853949] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8065.859171] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8065.864129] [ 8065.866227] [ 8065.868323] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8065.873633] ? do_raw_spin_lock+0x42/0xc0 [ 8065.877642] ceph_check_caps+0x63/0xae0 [ceph] [ 8065.882100] ? __lock_acquire+0xbfa/0x1e20 [ 8065.886199] ? lock_acquire+0xcb/0x2b0 [ 8065.889948] ? igrab+0x1d/0x50 [ 8065.892999] ? find_held_lock+0x2d/0x90 [ 8065.896830] ? find_held_lock+0x2d/0x90 [ 8065.900659] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8065.906060] ? lock_release+0xc5/0x260 [ 8065.909806] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8065.915041] delayed_work+0x1d8/0x280 [ceph] [ 8065.919322] ? process_scheduled_works+0x22a/0x510 [ 8065.924112] process_scheduled_works+0x2a4/0x510 [ 8065.928728] worker_thread+0x188/0x340 [ 8065.932477] ? rescuer_thread+0x310/0x310 [ 8065.936479] kthread+0xf5/0x130 [ 8065.939618] ? kthread_complete_and_exit+0x20/0x20 [ 8065.944401] ret_from_fork+0x31/0x40 [ 8065.947972] ? kthread_complete_and_exit+0x20/0x20 [ 8065.952756] ret_from_fork_asm+0x11/0x20 [ 8065.956680] [ 8093.608022] watchdog: BUG: soft lockup - CPU#7 stuck for 5420s! [kworker/7:4:78808] [ 8093.615673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8093.671122] irq event stamp: 13313706 [ 8093.674779] hardirqs last enabled at (13313705): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8093.684947] hardirqs last disabled at (13313706): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8093.694678] softirqs last enabled at (13313594): [] __do_softirq+0x2be/0x371 [ 8093.703364] softirqs last disabled at (13313589): [] irq_exit_rcu+0xb0/0xd0 [ 8093.711883] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8093.721261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8093.728733] Workqueue: events delayed_work [ceph] [ 8093.733455] RIP: 0010:lock_release+0x1d2/0x260 [ 8093.737894] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [ 8093.756630] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [ 8093.761847] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [ 8093.768970] RDX: 0000000000000001 RSI: ffffffff813b9425 RDI: ffff8881a1d34518 [ 8093.776095] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [ 8093.783219] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9425 [ 8093.790343] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 8093.797468] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8093.805546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8093.811282] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8093.818407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8093.825532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8093.832654] Call Trace: [ 8093.835101] [ 8093.837112] ? watchdog_timer_fn+0x1b4/0x220 [ 8093.841382] ? __hrtimer_run_queues+0x1d3/0x360 [ 8093.845909] ? hrtimer_interrupt+0x104/0x240 [ 8093.850171] ? __local_bh_enable+0x50/0x60 [ 8093.854263] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8093.859488] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8093.864445] [ 8093.866542] [ 8093.868640] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8093.873943] ? igrab+0x35/0x50 [ 8093.876995] ? igrab+0x35/0x50 [ 8093.880047] ? lock_release+0x1d2/0x260 [ 8093.883878] _raw_spin_unlock+0x1b/0x40 [ 8093.887717] igrab+0x35/0x50 [ 8093.890595] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 8093.895743] delayed_work+0x1d8/0x280 [ceph] [ 8093.900024] ? process_scheduled_works+0x22a/0x510 [ 8093.904814] process_scheduled_works+0x2a4/0x510 [ 8093.909432] worker_thread+0x188/0x340 [ 8093.913180] ? rescuer_thread+0x310/0x310 [ 8093.917183] kthread+0xf5/0x130 [ 8093.920321] ? kthread_complete_and_exit+0x20/0x20 [ 8093.925104] ret_from_fork+0x31/0x40 [ 8093.928675] ? kthread_complete_and_exit+0x20/0x20 [ 8093.933458] ret_from_fork_asm+0x11/0x20 [ 8093.937383] [ 8121.608520] watchdog: BUG: soft lockup - CPU#7 stuck for 5446s! [kworker/7:4:78808] [ 8121.616173] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8121.671622] irq event stamp: 13375312 [ 8121.675278] hardirqs last enabled at (13375311): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8121.685444] hardirqs last disabled at (13375312): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8121.695169] softirqs last enabled at (13375288): [] __do_softirq+0x2be/0x371 [ 8121.703861] softirqs last disabled at (13375283): [] irq_exit_rcu+0xb0/0xd0 [ 8121.712382] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8121.721757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8121.729229] Workqueue: events delayed_work [ceph] [ 8121.733953] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8121.738304] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8121.757041] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8121.762259] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8121.769383] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8121.776505] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8121.783629] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8121.790754] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8121.797878] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8121.805957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8121.811695] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8121.818817] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8121.825942] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8121.833067] Call Trace: [ 8121.835511] [ 8121.837524] ? watchdog_timer_fn+0x1b4/0x220 [ 8121.841796] ? __hrtimer_run_queues+0x1d3/0x360 [ 8121.846330] ? hrtimer_interrupt+0x104/0x240 [ 8121.850599] ? __local_bh_enable+0x50/0x60 [ 8121.854693] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8121.859917] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8121.864875] [ 8121.866971] [ 8121.869069] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8121.874375] ? lock_acquire+0xf0/0x2b0 [ 8121.878117] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8121.882823] ? lock_release+0xc5/0x260 [ 8121.886568] _raw_spin_lock+0x30/0x40 [ 8121.890225] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8121.894844] ceph_check_caps+0x63/0xae0 [ceph] [ 8121.899294] ? __lock_acquire+0xbfa/0x1e20 [ 8121.903393] ? lock_acquire+0xcb/0x2b0 [ 8121.907141] ? igrab+0x1d/0x50 [ 8121.910192] ? find_held_lock+0x2d/0x90 [ 8121.914023] ? find_held_lock+0x2d/0x90 [ 8121.917856] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8121.923253] ? lock_release+0xc5/0x260 [ 8121.927000] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8121.932226] delayed_work+0x1d8/0x280 [ceph] [ 8121.936509] ? process_scheduled_works+0x22a/0x510 [ 8121.941297] process_scheduled_works+0x2a4/0x510 [ 8121.945912] worker_thread+0x188/0x340 [ 8121.949662] ? rescuer_thread+0x310/0x310 [ 8121.953666] kthread+0xf5/0x130 [ 8121.956813] ? kthread_complete_and_exit+0x20/0x20 [ 8121.961596] ret_from_fork+0x31/0x40 [ 8121.965177] ? kthread_complete_and_exit+0x20/0x20 [ 8121.969959] ret_from_fork_asm+0x11/0x20 [ 8121.973882] [ 8149.609011] watchdog: BUG: soft lockup - CPU#7 stuck for 5472s! [kworker/7:4:78808] [ 8149.616661] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8149.672109] irq event stamp: 13438044 [ 8149.675767] hardirqs last enabled at (13438043): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8149.685934] hardirqs last disabled at (13438044): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8149.695657] softirqs last enabled at (13438012): [] __do_softirq+0x2be/0x371 [ 8149.704341] softirqs last disabled at (13438007): [] irq_exit_rcu+0xb0/0xd0 [ 8149.712863] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8149.722238] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8149.729710] Workqueue: events delayed_work [ceph] [ 8149.734434] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8149.738785] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8149.757522] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8149.762739] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8149.769863] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8149.776988] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8149.784111] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8149.791234] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8149.798361] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8149.806438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8149.812175] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8149.819299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8149.826423] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8149.833546] Call Trace: [ 8149.835994] [ 8149.838005] ? watchdog_timer_fn+0x1b4/0x220 [ 8149.842276] ? __hrtimer_run_queues+0x1d3/0x360 [ 8149.846802] ? hrtimer_interrupt+0x104/0x240 [ 8149.851071] ? __local_bh_enable+0x50/0x60 [ 8149.855164] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8149.860388] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8149.865345] [ 8149.867442] [ 8149.869541] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8149.874848] ? lock_acquire+0xf0/0x2b0 [ 8149.878599] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8149.884006] ? lock_release+0xc5/0x260 [ 8149.887751] _raw_spin_lock+0x30/0x40 [ 8149.891417] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8149.896814] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8149.902043] delayed_work+0x1d8/0x280 [ceph] [ 8149.906325] ? process_scheduled_works+0x22a/0x510 [ 8149.911114] process_scheduled_works+0x2a4/0x510 [ 8149.915729] worker_thread+0x188/0x340 [ 8149.919481] ? rescuer_thread+0x310/0x310 [ 8149.923483] kthread+0xf5/0x130 [ 8149.926621] ? kthread_complete_and_exit+0x20/0x20 [ 8149.931405] ret_from_fork+0x31/0x40 [ 8149.934985] ? kthread_complete_and_exit+0x20/0x20 [ 8149.939767] ret_from_fork_asm+0x11/0x20 [ 8149.943690] [ 8177.609501] watchdog: BUG: soft lockup - CPU#7 stuck for 5498s! [kworker/7:4:78808] [ 8177.617151] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8177.672600] irq event stamp: 13499544 [ 8177.676256] hardirqs last enabled at (13499543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8177.686422] hardirqs last disabled at (13499544): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8177.696146] softirqs last enabled at (13499530): [] __do_softirq+0x2be/0x371 [ 8177.704832] softirqs last disabled at (13499525): [] irq_exit_rcu+0xb0/0xd0 [ 8177.713351] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8177.722728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8177.730201] Workqueue: events delayed_work [ceph] [ 8177.734924] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8177.739275] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8177.758010] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8177.763228] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8177.770351] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8177.777477] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8177.784600] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8177.791725] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8177.798848] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8177.806927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8177.812665] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8177.819790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8177.826911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8177.834037] Call Trace: [ 8177.836482] [ 8177.838493] ? watchdog_timer_fn+0x1b4/0x220 [ 8177.842757] ? __hrtimer_run_queues+0x1d3/0x360 [ 8177.847290] ? hrtimer_interrupt+0x104/0x240 [ 8177.851552] ? __local_bh_enable+0x50/0x60 [ 8177.855645] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8177.860870] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8177.865827] [ 8177.867922] [ 8177.870022] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8177.875329] ? lock_acquire+0xf0/0x2b0 [ 8177.879079] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8177.883793] ? lock_release+0xc5/0x260 [ 8177.887537] _raw_spin_lock+0x30/0x40 [ 8177.891195] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8177.895814] ceph_check_caps+0x63/0xae0 [ceph] [ 8177.900265] ? __lock_acquire+0xbfa/0x1e20 [ 8177.904362] ? lock_acquire+0xcb/0x2b0 [ 8177.908111] ? igrab+0x1d/0x50 [ 8177.911163] ? find_held_lock+0x2d/0x90 [ 8177.915002] ? find_held_lock+0x2d/0x90 [ 8177.918833] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8177.924232] ? lock_release+0xc5/0x260 [ 8177.927978] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8177.933204] delayed_work+0x1d8/0x280 [ceph] [ 8177.937485] ? process_scheduled_works+0x22a/0x510 [ 8177.942276] process_scheduled_works+0x2a4/0x510 [ 8177.946892] worker_thread+0x188/0x340 [ 8177.950641] ? rescuer_thread+0x310/0x310 [ 8177.954644] kthread+0xf5/0x130 [ 8177.957793] ? kthread_complete_and_exit+0x20/0x20 [ 8177.962583] ret_from_fork+0x31/0x40 [ 8177.966163] ? kthread_complete_and_exit+0x20/0x20 [ 8177.970947] ret_from_fork_asm+0x11/0x20 [ 8177.974879] [ 8205.609994] watchdog: BUG: soft lockup - CPU#7 stuck for 5525s! [kworker/7:4:78808] [ 8205.617648] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8205.673098] irq event stamp: 13560940 [ 8205.676755] hardirqs last enabled at (13560939): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8205.686920] hardirqs last disabled at (13560940): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8205.696644] softirqs last enabled at (13560862): [] __do_softirq+0x2be/0x371 [ 8205.705329] softirqs last disabled at (13560857): [] irq_exit_rcu+0xb0/0xd0 [ 8205.713848] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8205.723227] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8205.730699] Workqueue: events delayed_work [ceph] [ 8205.735421] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8205.739772] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8205.758510] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8205.763725] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8205.770851] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8205.777974] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8205.785098] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8205.792224] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8205.799346] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8205.807425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8205.813162] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8205.820286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8205.827410] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8205.834534] Call Trace: [ 8205.836982] [ 8205.838993] ? watchdog_timer_fn+0x1b4/0x220 [ 8205.843263] ? __hrtimer_run_queues+0x1d3/0x360 [ 8205.847790] ? hrtimer_interrupt+0x104/0x240 [ 8205.852059] ? __local_bh_enable+0x50/0x60 [ 8205.856151] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8205.861377] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8205.866333] [ 8205.868430] [ 8205.870529] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8205.875836] ? lock_acquire+0xf0/0x2b0 [ 8205.879586] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8205.884301] ? lock_release+0xc5/0x260 [ 8205.888045] _raw_spin_lock+0x30/0x40 [ 8205.891702] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8205.896328] ceph_check_caps+0x63/0xae0 [ceph] [ 8205.900779] ? __lock_acquire+0xbfa/0x1e20 [ 8205.904879] ? lock_acquire+0xcb/0x2b0 [ 8205.908628] ? igrab+0x1d/0x50 [ 8205.911678] ? find_held_lock+0x2d/0x90 [ 8205.915518] ? find_held_lock+0x2d/0x90 [ 8205.919347] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8205.924747] ? lock_release+0xc5/0x260 [ 8205.928493] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8205.933720] delayed_work+0x1d8/0x280 [ceph] [ 8205.938001] ? process_scheduled_works+0x22a/0x510 [ 8205.942792] process_scheduled_works+0x2a4/0x510 [ 8205.947407] worker_thread+0x188/0x340 [ 8205.951156] ? rescuer_thread+0x310/0x310 [ 8205.955160] kthread+0xf5/0x130 [ 8205.958307] ? kthread_complete_and_exit+0x20/0x20 [ 8205.963090] ret_from_fork+0x31/0x40 [ 8205.966668] ? kthread_complete_and_exit+0x20/0x20 [ 8205.971452] ret_from_fork_asm+0x11/0x20 [ 8205.975375] [ 8233.610487] watchdog: BUG: soft lockup - CPU#7 stuck for 5551s! [kworker/7:4:78808] [ 8233.618137] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8233.673587] irq event stamp: 13621826 [ 8233.677245] hardirqs last enabled at (13621825): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8233.687410] hardirqs last disabled at (13621826): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8233.697143] softirqs last enabled at (13621748): [] __do_softirq+0x2be/0x371 [ 8233.705836] softirqs last disabled at (13621743): [] irq_exit_rcu+0xb0/0xd0 [ 8233.714357] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8233.723734] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8233.731204] Workqueue: events delayed_work [ceph] [ 8233.735929] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8233.740288] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8233.759023] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8233.764241] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8233.771365] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8233.778491] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8233.785615] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8233.792737] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8233.799861] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8233.807941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8233.813677] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8233.820801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8233.827926] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8233.835049] Call Trace: [ 8233.837495] [ 8233.839506] ? watchdog_timer_fn+0x1b4/0x220 [ 8233.843769] ? __hrtimer_run_queues+0x1d3/0x360 [ 8233.848294] ? hrtimer_interrupt+0x104/0x240 [ 8233.852555] ? __local_bh_enable+0x50/0x60 [ 8233.856650] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8233.861875] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8233.866831] [ 8233.868929] [ 8233.871027] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8233.876333] ? lock_acquire+0xf0/0x2b0 [ 8233.880083] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8233.884797] ? lock_release+0xc5/0x260 [ 8233.888543] _raw_spin_lock+0x30/0x40 [ 8233.892208] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8233.896828] ceph_check_caps+0x63/0xae0 [ceph] [ 8233.901277] ? __lock_acquire+0xbfa/0x1e20 [ 8233.905376] ? lock_acquire+0xcb/0x2b0 [ 8233.909124] ? igrab+0x1d/0x50 [ 8233.912176] ? find_held_lock+0x2d/0x90 [ 8233.916006] ? find_held_lock+0x2d/0x90 [ 8233.919838] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8233.925245] ? lock_release+0xc5/0x260 [ 8233.928990] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8233.934219] delayed_work+0x1d8/0x280 [ceph] [ 8233.938498] ? process_scheduled_works+0x22a/0x510 [ 8233.943288] process_scheduled_works+0x2a4/0x510 [ 8233.947906] worker_thread+0x188/0x340 [ 8233.951655] ? rescuer_thread+0x310/0x310 [ 8233.955657] kthread+0xf5/0x130 [ 8233.958796] ? kthread_complete_and_exit+0x20/0x20 [ 8233.963580] ret_from_fork+0x31/0x40 [ 8233.967158] ? kthread_complete_and_exit+0x20/0x20 [ 8233.971942] ret_from_fork_asm+0x11/0x20 [ 8233.975866] [ 8261.610985] watchdog: BUG: soft lockup - CPU#7 stuck for 5577s! [kworker/7:4:78808] [ 8261.618636] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8261.674083] irq event stamp: 13683214 [ 8261.677741] hardirqs last enabled at (13683213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8261.687907] hardirqs last disabled at (13683214): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8261.697632] softirqs last enabled at (13683136): [] __do_softirq+0x2be/0x371 [ 8261.706325] softirqs last disabled at (13683131): [] irq_exit_rcu+0xb0/0xd0 [ 8261.714847] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8261.724222] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8261.731693] Workqueue: events delayed_work [ceph] [ 8261.736417] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8261.740769] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8261.759505] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8261.764721] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8261.771847] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8261.778970] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8261.786094] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8261.793219] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8261.800343] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8261.808420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8261.814158] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8261.821282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8261.828407] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8261.835530] Call Trace: [ 8261.837977] [ 8261.839988] ? watchdog_timer_fn+0x1b4/0x220 [ 8261.844258] ? __hrtimer_run_queues+0x1d3/0x360 [ 8261.848785] ? hrtimer_interrupt+0x104/0x240 [ 8261.853047] ? __local_bh_enable+0x50/0x60 [ 8261.857140] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8261.862363] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8261.867320] [ 8261.869418] [ 8261.871516] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8261.876823] ? lock_acquire+0xf0/0x2b0 [ 8261.880574] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8261.885286] ? lock_release+0xc5/0x260 [ 8261.889031] _raw_spin_lock+0x30/0x40 [ 8261.892689] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8261.897309] ceph_check_caps+0x63/0xae0 [ceph] [ 8261.901757] ? __lock_acquire+0xbfa/0x1e20 [ 8261.905856] ? lock_acquire+0xcb/0x2b0 [ 8261.909606] ? igrab+0x1d/0x50 [ 8261.912656] ? find_held_lock+0x2d/0x90 [ 8261.916487] ? find_held_lock+0x2d/0x90 [ 8261.920319] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8261.925719] ? lock_release+0xc5/0x260 [ 8261.929463] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8261.934691] delayed_work+0x1d8/0x280 [ceph] [ 8261.938972] ? process_scheduled_works+0x22a/0x510 [ 8261.943760] process_scheduled_works+0x2a4/0x510 [ 8261.948378] worker_thread+0x188/0x340 [ 8261.952127] ? rescuer_thread+0x310/0x310 [ 8261.956130] kthread+0xf5/0x130 [ 8261.959277] ? kthread_complete_and_exit+0x20/0x20 [ 8261.964060] ret_from_fork+0x31/0x40 [ 8261.967632] ? kthread_complete_and_exit+0x20/0x20 [ 8261.972424] ret_from_fork_asm+0x11/0x20 [ 8261.976355] [ 8289.611486] watchdog: BUG: soft lockup - CPU#7 stuck for 5603s! [kworker/7:4:78808] [ 8289.619141] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8289.674590] irq event stamp: 13744782 [ 8289.678248] hardirqs last enabled at (13744781): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8289.688413] hardirqs last disabled at (13744782): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8289.698138] softirqs last enabled at (13744704): [] __do_softirq+0x2be/0x371 [ 8289.706823] softirqs last disabled at (13744699): [] irq_exit_rcu+0xb0/0xd0 [ 8289.715343] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8289.724719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8289.732190] Workqueue: events delayed_work [ceph] [ 8289.736915] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8289.741265] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8289.760002] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8289.765219] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8289.772345] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8289.779468] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8289.786592] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8289.793716] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8289.800840] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8289.808918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8289.814656] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8289.821779] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8289.828903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8289.836027] Call Trace: [ 8289.838475] [ 8289.840487] ? watchdog_timer_fn+0x1b4/0x220 [ 8289.844755] ? __hrtimer_run_queues+0x1d3/0x360 [ 8289.849284] ? hrtimer_interrupt+0x104/0x240 [ 8289.853553] ? __local_bh_enable+0x50/0x60 [ 8289.857645] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8289.862870] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8289.867825] [ 8289.869924] [ 8289.872023] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8289.877329] ? lock_acquire+0xf0/0x2b0 [ 8289.881079] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8289.886486] ? lock_release+0xc5/0x260 [ 8289.890232] _raw_spin_lock+0x30/0x40 [ 8289.893888] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8289.899288] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8289.904517] delayed_work+0x1d8/0x280 [ceph] [ 8289.908798] ? process_scheduled_works+0x22a/0x510 [ 8289.913588] process_scheduled_works+0x2a4/0x510 [ 8289.918203] worker_thread+0x188/0x340 [ 8289.921951] ? rescuer_thread+0x310/0x310 [ 8289.925954] kthread+0xf5/0x130 [ 8289.929093] ? kthread_complete_and_exit+0x20/0x20 [ 8289.933878] ret_from_fork+0x31/0x40 [ 8289.937457] ? kthread_complete_and_exit+0x20/0x20 [ 8289.942241] ret_from_fork_asm+0x11/0x20 [ 8289.946172] [ 8317.611966] watchdog: BUG: soft lockup - CPU#7 stuck for 5629s! [kworker/7:4:78808] [ 8317.619623] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8317.675072] irq event stamp: 13805828 [ 8317.678728] hardirqs last enabled at (13805827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8317.688895] hardirqs last disabled at (13805828): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8317.698618] softirqs last enabled at (13805814): [] __do_softirq+0x2be/0x371 [ 8317.707303] softirqs last disabled at (13805809): [] irq_exit_rcu+0xb0/0xd0 [ 8317.715826] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8317.725210] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8317.732681] Workqueue: events delayed_work [ceph] [ 8317.737405] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8317.741755] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8317.760493] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 8317.765710] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 8317.772834] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 8317.779958] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 8317.787081] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 8317.794205] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8317.801329] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8317.809406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8317.815144] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8317.822268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8317.829392] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8317.836518] Call Trace: [ 8317.838963] [ 8317.840974] ? watchdog_timer_fn+0x1b4/0x220 [ 8317.845238] ? __hrtimer_run_queues+0x1d3/0x360 [ 8317.849772] ? hrtimer_interrupt+0x104/0x240 [ 8317.854042] ? __local_bh_enable+0x50/0x60 [ 8317.858134] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8317.863358] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8317.868316] [ 8317.870413] [ 8317.872511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8317.877818] ? lock_acquire+0xf0/0x2b0 [ 8317.881569] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8317.886977] _raw_spin_lock+0x30/0x40 [ 8317.890643] ? igrab+0x1d/0x50 [ 8317.893693] igrab+0x1d/0x50 [ 8317.896571] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 8317.901711] delayed_work+0x1d8/0x280 [ceph] [ 8317.905992] ? process_scheduled_works+0x22a/0x510 [ 8317.910782] process_scheduled_works+0x2a4/0x510 [ 8317.915400] worker_thread+0x188/0x340 [ 8317.919148] ? rescuer_thread+0x310/0x310 [ 8317.923151] kthread+0xf5/0x130 [ 8317.926289] ? kthread_complete_and_exit+0x20/0x20 [ 8317.931073] ret_from_fork+0x31/0x40 [ 8317.934644] ? kthread_complete_and_exit+0x20/0x20 [ 8317.939428] ret_from_fork_asm+0x11/0x20 [ 8317.943359] [ 8345.612458] watchdog: BUG: soft lockup - CPU#7 stuck for 5655s! [kworker/7:4:78808] [ 8345.620112] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8345.675561] irq event stamp: 13866362 [ 8345.679218] hardirqs last enabled at (13866361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8345.689384] hardirqs last disabled at (13866362): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8345.699110] softirqs last enabled at (13866342): [] __do_softirq+0x2be/0x371 [ 8345.707802] softirqs last disabled at (13866337): [] irq_exit_rcu+0xb0/0xd0 [ 8345.716322] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8345.725699] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8345.733171] Workqueue: events delayed_work [ceph] [ 8345.737894] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8345.742245] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8345.760981] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8345.766199] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8345.773322] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8345.780448] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8345.787572] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8345.794696] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8345.801819] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8345.809897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8345.815634] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8345.822758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8345.829882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8345.837005] Call Trace: [ 8345.839453] [ 8345.841464] ? watchdog_timer_fn+0x1b4/0x220 [ 8345.845736] ? __hrtimer_run_queues+0x1d3/0x360 [ 8345.850262] ? hrtimer_interrupt+0x104/0x240 [ 8345.854532] ? __local_bh_enable+0x50/0x60 [ 8345.858624] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8345.863849] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8345.868805] [ 8345.870903] [ 8345.873001] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8345.878308] ? lock_acquire+0xf0/0x2b0 [ 8345.882059] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8345.887466] ? lock_release+0xc5/0x260 [ 8345.891209] _raw_spin_lock+0x30/0x40 [ 8345.894867] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8345.900266] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8345.905496] delayed_work+0x1d8/0x280 [ceph] [ 8345.909777] ? process_scheduled_works+0x22a/0x510 [ 8345.914565] process_scheduled_works+0x2a4/0x510 [ 8345.919182] worker_thread+0x188/0x340 [ 8345.922930] ? rescuer_thread+0x310/0x310 [ 8345.926934] kthread+0xf5/0x130 [ 8345.930073] ? kthread_complete_and_exit+0x20/0x20 [ 8345.934858] ret_from_fork+0x31/0x40 [ 8345.938437] ? kthread_complete_and_exit+0x20/0x20 [ 8345.943218] ret_from_fork_asm+0x11/0x20 [ 8345.947151] [ 8373.612949] watchdog: BUG: soft lockup - CPU#7 stuck for 5681s! [kworker/7:4:78808] [ 8373.620602] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8373.676050] irq event stamp: 13927728 [ 8373.679707] hardirqs last enabled at (13927727): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8373.689873] hardirqs last disabled at (13927728): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8373.699598] softirqs last enabled at (13927714): [] __do_softirq+0x2be/0x371 [ 8373.708290] softirqs last disabled at (13927709): [] irq_exit_rcu+0xb0/0xd0 [ 8373.716803] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8373.726180] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8373.733650] Workqueue: events delayed_work [ceph] [ 8373.738375] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8373.742734] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8373.761469] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8373.766688] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8373.773812] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8373.780935] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8373.788061] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8373.795184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8373.802307] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8373.810386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8373.816122] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8373.823248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8373.830373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8373.837495] Call Trace: [ 8373.839943] [ 8373.841953] ? watchdog_timer_fn+0x1b4/0x220 [ 8373.846216] ? __hrtimer_run_queues+0x1d3/0x360 [ 8373.850750] ? hrtimer_interrupt+0x104/0x240 [ 8373.855011] ? __local_bh_enable+0x50/0x60 [ 8373.859103] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8373.864327] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8373.869285] [ 8373.871382] [ 8373.873481] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8373.878789] ? lock_acquire+0xf0/0x2b0 [ 8373.882538] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8373.887945] ? lock_release+0xc5/0x260 [ 8373.891692] _raw_spin_lock+0x30/0x40 [ 8373.895358] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8373.900754] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8373.905984] delayed_work+0x1d8/0x280 [ceph] [ 8373.910264] ? process_scheduled_works+0x22a/0x510 [ 8373.915055] process_scheduled_works+0x2a4/0x510 [ 8373.919670] worker_thread+0x188/0x340 [ 8373.923419] ? rescuer_thread+0x310/0x310 [ 8373.927423] kthread+0xf5/0x130 [ 8373.930563] ? kthread_complete_and_exit+0x20/0x20 [ 8373.935353] ret_from_fork+0x31/0x40 [ 8373.938924] ? kthread_complete_and_exit+0x20/0x20 [ 8373.943707] ret_from_fork_asm+0x11/0x20 [ 8373.947631] [ 8401.613439] watchdog: BUG: soft lockup - CPU#7 stuck for 5707s! [kworker/7:4:78808] [ 8401.621090] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8401.676540] irq event stamp: 13989722 [ 8401.680198] hardirqs last enabled at (13989721): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8401.690363] hardirqs last disabled at (13989722): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8401.700087] softirqs last enabled at (13989708): [] __do_softirq+0x2be/0x371 [ 8401.708772] softirqs last disabled at (13989703): [] irq_exit_rcu+0xb0/0xd0 [ 8401.717292] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8401.726667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8401.734141] Workqueue: events delayed_work [ceph] [ 8401.738864] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8401.743215] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8401.761952] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8401.767169] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8401.774292] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8401.781415] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8401.788541] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8401.795665] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8401.802789] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8401.810865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8401.816603] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8401.823729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8401.830853] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8401.837976] Call Trace: [ 8401.840423] [ 8401.842433] ? watchdog_timer_fn+0x1b4/0x220 [ 8401.846696] ? __hrtimer_run_queues+0x1d3/0x360 [ 8401.851230] ? hrtimer_interrupt+0x104/0x240 [ 8401.855491] ? __local_bh_enable+0x50/0x60 [ 8401.859585] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8401.864810] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8401.869767] [ 8401.871865] [ 8401.873962] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8401.879269] ? lock_acquire+0xf0/0x2b0 [ 8401.883019] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8401.888426] ? lock_release+0xc5/0x260 [ 8401.892171] _raw_spin_lock+0x30/0x40 [ 8401.895829] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8401.901227] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8401.906457] delayed_work+0x1d8/0x280 [ceph] [ 8401.910738] ? process_scheduled_works+0x22a/0x510 [ 8401.915528] process_scheduled_works+0x2a4/0x510 [ 8401.920143] worker_thread+0x188/0x340 [ 8401.923891] ? rescuer_thread+0x310/0x310 [ 8401.927894] kthread+0xf5/0x130 [ 8401.931033] ? kthread_complete_and_exit+0x20/0x20 [ 8401.935816] ret_from_fork+0x31/0x40 [ 8401.939396] ? kthread_complete_and_exit+0x20/0x20 [ 8401.944181] ret_from_fork_asm+0x11/0x20 [ 8401.948112] [ 8429.613947] watchdog: BUG: soft lockup - CPU#7 stuck for 5733s! [kworker/7:4:78808] [ 8429.621597] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8429.677047] irq event stamp: 14050870 [ 8429.680703] hardirqs last enabled at (14050869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8429.690870] hardirqs last disabled at (14050870): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8429.700593] softirqs last enabled at (14050856): [] __do_softirq+0x2be/0x371 [ 8429.709279] softirqs last disabled at (14050851): [] irq_exit_rcu+0xb0/0xd0 [ 8429.717800] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8429.727175] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8429.734647] Workqueue: events delayed_work [ceph] [ 8429.739371] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8429.743730] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8429.762466] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8429.767683] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8429.774809] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8429.781931] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8429.789057] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8429.796181] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8429.803305] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8429.811383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8429.817118] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8429.824244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8429.831366] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8429.838491] Call Trace: [ 8429.840939] [ 8429.842949] ? watchdog_timer_fn+0x1b4/0x220 [ 8429.847221] ? __hrtimer_run_queues+0x1d3/0x360 [ 8429.851754] ? hrtimer_interrupt+0x104/0x240 [ 8429.856015] ? __local_bh_enable+0x50/0x60 [ 8429.860110] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8429.865332] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8429.870289] [ 8429.872387] [ 8429.874485] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8429.879792] ? lock_acquire+0xf0/0x2b0 [ 8429.883535] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8429.888941] ? lock_release+0xc5/0x260 [ 8429.892687] _raw_spin_lock+0x30/0x40 [ 8429.896353] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8429.901752] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8429.906981] delayed_work+0x1d8/0x280 [ceph] [ 8429.911262] ? process_scheduled_works+0x22a/0x510 [ 8429.916051] process_scheduled_works+0x2a4/0x510 [ 8429.920666] worker_thread+0x188/0x340 [ 8429.924416] ? rescuer_thread+0x310/0x310 [ 8429.928420] kthread+0xf5/0x130 [ 8429.931556] ? kthread_complete_and_exit+0x20/0x20 [ 8429.936343] ret_from_fork+0x31/0x40 [ 8429.939922] ? kthread_complete_and_exit+0x20/0x20 [ 8429.944712] ret_from_fork_asm+0x11/0x20 [ 8429.948637] [ 8457.614442] watchdog: BUG: soft lockup - CPU#7 stuck for 5759s! [kworker/7:4:78808] [ 8457.622096] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8457.677543] irq event stamp: 14111370 [ 8457.681202] hardirqs last enabled at (14111369): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8457.691367] hardirqs last disabled at (14111370): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8457.701092] softirqs last enabled at (14111356): [] __do_softirq+0x2be/0x371 [ 8457.709785] softirqs last disabled at (14111351): [] irq_exit_rcu+0xb0/0xd0 [ 8457.718305] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8457.727681] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8457.735152] Workqueue: events delayed_work [ceph] [ 8457.739876] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8457.744228] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8457.762964] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8457.768181] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8457.775306] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8457.782429] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8457.789554] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8457.796677] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8457.803802] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8457.811878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8457.817616] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8457.824741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8457.831866] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8457.838988] Call Trace: [ 8457.841436] [ 8457.843446] ? watchdog_timer_fn+0x1b4/0x220 [ 8457.847710] ? __hrtimer_run_queues+0x1d3/0x360 [ 8457.852243] ? hrtimer_interrupt+0x104/0x240 [ 8457.856506] ? __local_bh_enable+0x50/0x60 [ 8457.860597] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8457.865823] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8457.870780] [ 8457.872877] [ 8457.874976] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8457.880282] ? lock_acquire+0xf0/0x2b0 [ 8457.884031] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8457.888737] ? lock_release+0xc5/0x260 [ 8457.892482] _raw_spin_lock+0x30/0x40 [ 8457.896141] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8457.900758] ceph_check_caps+0x63/0xae0 [ceph] [ 8457.905210] ? __lock_acquire+0xbfa/0x1e20 [ 8457.909307] ? lock_acquire+0xcb/0x2b0 [ 8457.913057] ? igrab+0x1d/0x50 [ 8457.916108] ? find_held_lock+0x2d/0x90 [ 8457.919947] ? find_held_lock+0x2d/0x90 [ 8457.923777] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8457.929176] ? lock_release+0xc5/0x260 [ 8457.932922] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8457.938151] delayed_work+0x1d8/0x280 [ceph] [ 8457.942430] ? process_scheduled_works+0x22a/0x510 [ 8457.947220] process_scheduled_works+0x2a4/0x510 [ 8457.951836] worker_thread+0x188/0x340 [ 8457.955586] ? rescuer_thread+0x310/0x310 [ 8457.959589] kthread+0xf5/0x130 [ 8457.962728] ? kthread_complete_and_exit+0x20/0x20 [ 8457.967511] ret_from_fork+0x31/0x40 [ 8457.971091] ? kthread_complete_and_exit+0x20/0x20 [ 8457.975873] ret_from_fork_asm+0x11/0x20 [ 8457.979796] [ 8485.614922] watchdog: BUG: soft lockup - CPU#7 stuck for 5785s! [kworker/7:4:78808] [ 8485.622577] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8485.678024] irq event stamp: 14172614 [ 8485.681681] hardirqs last enabled at (14172613): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8485.691847] hardirqs last disabled at (14172614): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8485.701573] softirqs last enabled at (14172536): [] __do_softirq+0x2be/0x371 [ 8485.710266] softirqs last disabled at (14172531): [] irq_exit_rcu+0xb0/0xd0 [ 8485.718786] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8485.728162] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8485.735634] Workqueue: events delayed_work [ceph] [ 8485.740358] RIP: 0010:ceph_check_caps+0x351/0xae0 [ceph] [ 8485.745678] Code: d0 01 00 00 41 8b 47 44 89 c1 f7 d1 44 85 f1 74 19 09 d0 f7 d0 44 85 f0 75 1a 48 89 df e8 b7 b7 ff ff 85 c0 74 0e 41 8b 57 38 <85> 54 24 18 0f 84 c5 fe ff ff 48 89 dd 4c 89 eb 48 8b 7c 24 10 e8 [ 8485.764415] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [ 8485.769632] RAX: 0000000000000000 RBX: ffff88810da9e000 RCX: 00000000ffffffff [ 8485.776757] RDX: 0000000000000155 RSI: 0000000000000000 RDI: ffff8881a1d34b20 [ 8485.783880] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 8485.791005] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 8485.798129] R13: ffff8881a1d34478 R14: 0000000000000000 R15: ffff88810da9e008 [ 8485.805254] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8485.813331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8485.819067] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8485.826193] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8485.833317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8485.840442] Call Trace: [ 8485.842887] [ 8485.844898] ? watchdog_timer_fn+0x1b4/0x220 [ 8485.849170] ? __hrtimer_run_queues+0x1d3/0x360 [ 8485.853696] ? hrtimer_interrupt+0x104/0x240 [ 8485.857965] ? __local_bh_enable+0x50/0x60 [ 8485.862058] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8485.867283] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8485.872240] [ 8485.874337] [ 8485.876435] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8485.881742] ? ceph_check_caps+0x351/0xae0 [ceph] [ 8485.886463] ? lock_acquire+0xcb/0x2b0 [ 8485.890214] ? igrab+0x1d/0x50 [ 8485.893265] ? find_held_lock+0x2d/0x90 [ 8485.897095] ? find_held_lock+0x2d/0x90 [ 8485.900927] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8485.906326] ? lock_release+0xc5/0x260 [ 8485.910072] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8485.915300] delayed_work+0x1d8/0x280 [ceph] [ 8485.919581] ? process_scheduled_works+0x22a/0x510 [ 8485.924371] process_scheduled_works+0x2a4/0x510 [ 8485.928986] worker_thread+0x188/0x340 [ 8485.932736] ? rescuer_thread+0x310/0x310 [ 8485.936738] kthread+0xf5/0x130 [ 8485.939877] ? kthread_complete_and_exit+0x20/0x20 [ 8485.944660] ret_from_fork+0x31/0x40 [ 8485.948231] ? kthread_complete_and_exit+0x20/0x20 [ 8485.953014] ret_from_fork_asm+0x11/0x20 [ 8485.956937] [ 8490.255503] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8500.494911] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8511.247191] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8513.615408] watchdog: BUG: soft lockup - CPU#7 stuck for 5811s! [kworker/7:4:78808] [ 8513.623056] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8513.678504] irq event stamp: 14232946 [ 8513.682163] hardirqs last enabled at (14232945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8513.692330] hardirqs last disabled at (14232946): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8513.702061] softirqs last enabled at (14232820): [] __do_softirq+0x2be/0x371 [ 8513.710744] softirqs last disabled at (14232815): [] irq_exit_rcu+0xb0/0xd0 [ 8513.719257] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8513.728635] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8513.736105] Workqueue: events delayed_work [ceph] [ 8513.740830] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8513.745189] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8513.763924] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8513.769142] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8513.776267] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8513.783390] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8513.790515] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8513.797640] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8513.804764] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8513.812840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8513.818578] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8513.825702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8513.832827] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8513.839952] Call Trace: [ 8513.842397] [ 8513.844407] ? watchdog_timer_fn+0x1b4/0x220 [ 8513.848669] ? __hrtimer_run_queues+0x1d3/0x360 [ 8513.853196] ? hrtimer_interrupt+0x104/0x240 [ 8513.857457] ? __local_bh_enable+0x50/0x60 [ 8513.861552] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8513.866776] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8513.871731] [ 8513.873829] [ 8513.875927] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8513.881234] ? lock_acquire+0xf0/0x2b0 [ 8513.884984] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8513.890383] ? lock_release+0xc5/0x260 [ 8513.894128] _raw_spin_lock+0x30/0x40 [ 8513.897786] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8513.903184] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8513.908414] delayed_work+0x1d8/0x280 [ceph] [ 8513.912694] ? process_scheduled_works+0x22a/0x510 [ 8513.917484] process_scheduled_works+0x2a4/0x510 [ 8513.922099] worker_thread+0x188/0x340 [ 8513.925850] ? rescuer_thread+0x310/0x310 [ 8513.929852] kthread+0xf5/0x130 [ 8513.932991] ? kthread_complete_and_exit+0x20/0x20 [ 8513.937775] ret_from_fork+0x31/0x40 [ 8513.941354] ? kthread_complete_and_exit+0x20/0x20 [ 8513.946138] ret_from_fork_asm+0x11/0x20 [ 8513.950069] [ 8521.487446] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8532.239412] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8541.615892] watchdog: BUG: soft lockup - CPU#7 stuck for 5837s! [kworker/7:4:78808] [ 8541.623545] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8541.678994] irq event stamp: 14293370 [ 8541.682653] hardirqs last enabled at (14293369): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8541.692818] hardirqs last disabled at (14293370): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8541.702542] softirqs last enabled at (14293356): [] __do_softirq+0x2be/0x371 [ 8541.711227] softirqs last disabled at (14293351): [] irq_exit_rcu+0xb0/0xd0 [ 8541.719748] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8541.729124] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8541.736595] Workqueue: events delayed_work [ceph] [ 8541.741318] RIP: 0010:lock_release+0x14e/0x260 [ 8541.745757] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 8541.764492] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [ 8541.769710] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [ 8541.776834] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [ 8541.783958] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 8541.791081] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 8541.798205] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [ 8541.805330] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8541.813407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8541.819144] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8541.826269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8541.833394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8541.840517] Call Trace: [ 8541.842965] [ 8541.844976] ? watchdog_timer_fn+0x1b4/0x220 [ 8541.849248] ? __hrtimer_run_queues+0x1d3/0x360 [ 8541.853781] ? hrtimer_interrupt+0x104/0x240 [ 8541.858042] ? __local_bh_enable+0x50/0x60 [ 8541.862135] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8541.867360] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8541.872318] [ 8541.874413] [ 8541.876513] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8541.881817] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8541.887226] ? lock_release+0x14e/0x260 [ 8541.891057] _raw_spin_unlock+0x1b/0x40 [ 8541.894897] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8541.900133] delayed_work+0x1d8/0x280 [ceph] [ 8541.904414] ? process_scheduled_works+0x22a/0x510 [ 8541.909202] process_scheduled_works+0x2a4/0x510 [ 8541.913817] worker_thread+0x188/0x340 [ 8541.917568] ? rescuer_thread+0x310/0x310 [ 8541.921569] kthread+0xf5/0x130 [ 8541.924708] ? kthread_complete_and_exit+0x20/0x20 [ 8541.929493] ret_from_fork+0x31/0x40 [ 8541.933072] ? kthread_complete_and_exit+0x20/0x20 [ 8541.937857] ret_from_fork_asm+0x11/0x20 [ 8541.941786] [ 8542.479675] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8553.231916] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8563.472182] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8569.616382] watchdog: BUG: soft lockup - CPU#7 stuck for 5864s! [kworker/7:4:78808] [ 8569.624035] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8569.679484] irq event stamp: 14353634 [ 8569.683141] hardirqs last enabled at (14353633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8569.693307] hardirqs last disabled at (14353634): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8569.703032] softirqs last enabled at (14353602): [] __do_softirq+0x2be/0x371 [ 8569.711725] softirqs last disabled at (14353597): [] irq_exit_rcu+0xb0/0xd0 [ 8569.720244] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8569.729622] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8569.737092] Workqueue: events delayed_work [ceph] [ 8569.741817] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8569.746168] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8569.764904] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8569.770121] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8569.777244] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8569.784369] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8569.791494] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8569.798617] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8569.805742] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8569.813819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8569.819556] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8569.826681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8569.833804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8569.840929] Call Trace: [ 8569.843377] [ 8569.845386] ? watchdog_timer_fn+0x1b4/0x220 [ 8569.849649] ? __hrtimer_run_queues+0x1d3/0x360 [ 8569.854176] ? hrtimer_interrupt+0x104/0x240 [ 8569.858444] ? __local_bh_enable+0x50/0x60 [ 8569.862537] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8569.867762] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8569.872720] [ 8569.874817] [ 8569.876915] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8569.882221] ? lock_acquire+0xf0/0x2b0 [ 8569.885963] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8569.890676] ? lock_release+0xc5/0x260 [ 8569.894423] _raw_spin_lock+0x30/0x40 [ 8569.898079] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8569.902698] ceph_check_caps+0x63/0xae0 [ceph] [ 8569.907150] ? __lock_acquire+0xbfa/0x1e20 [ 8569.911248] ? lock_acquire+0xcb/0x2b0 [ 8569.914996] ? igrab+0x1d/0x50 [ 8569.918060] ? find_held_lock+0x2d/0x90 [ 8569.921895] ? find_held_lock+0x2d/0x90 [ 8569.925726] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8569.931127] ? lock_release+0xc5/0x260 [ 8569.934871] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8569.940099] delayed_work+0x1d8/0x280 [ceph] [ 8569.944380] ? process_scheduled_works+0x22a/0x510 [ 8569.949170] process_scheduled_works+0x2a4/0x510 [ 8569.953786] worker_thread+0x188/0x340 [ 8569.957534] ? rescuer_thread+0x310/0x310 [ 8569.961537] kthread+0xf5/0x130 [ 8569.964676] ? kthread_complete_and_exit+0x20/0x20 [ 8569.969459] ret_from_fork+0x31/0x40 [ 8569.973041] ? kthread_complete_and_exit+0x20/0x20 [ 8569.977830] ret_from_fork_asm+0x11/0x20 [ 8569.981753] [ 8574.224193] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8584.464428] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8595.216602] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8597.616877] watchdog: BUG: soft lockup - CPU#7 stuck for 5890s! [kworker/7:4:78808] [ 8597.624524] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8597.679972] irq event stamp: 14414158 [ 8597.683630] hardirqs last enabled at (14414157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8597.693796] hardirqs last disabled at (14414158): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8597.703522] softirqs last enabled at (14414080): [] __do_softirq+0x2be/0x371 [ 8597.712213] softirqs last disabled at (14414075): [] irq_exit_rcu+0xb0/0xd0 [ 8597.720726] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8597.730103] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8597.737575] Workqueue: events delayed_work [ceph] [ 8597.742297] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8597.746648] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8597.765383] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8597.770602] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8597.777726] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8597.784849] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8597.791974] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8597.799099] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8597.806223] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8597.814299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8597.820038] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8597.827163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8597.834286] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8597.841411] Call Trace: [ 8597.843856] [ 8597.845867] ? watchdog_timer_fn+0x1b4/0x220 [ 8597.850138] ? __hrtimer_run_queues+0x1d3/0x360 [ 8597.854672] ? hrtimer_interrupt+0x104/0x240 [ 8597.858934] ? __local_bh_enable+0x50/0x60 [ 8597.863027] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8597.868250] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8597.873210] [ 8597.875305] [ 8597.877404] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8597.882712] ? lock_acquire+0xf0/0x2b0 [ 8597.886462] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8597.891176] ? lock_release+0xc5/0x260 [ 8597.894919] _raw_spin_lock+0x30/0x40 [ 8597.898577] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8597.903196] ceph_check_caps+0x63/0xae0 [ceph] [ 8597.907647] ? __lock_acquire+0xbfa/0x1e20 [ 8597.911745] ? lock_acquire+0xcb/0x2b0 [ 8597.915493] ? igrab+0x1d/0x50 [ 8597.918544] ? find_held_lock+0x2d/0x90 [ 8597.922375] ? find_held_lock+0x2d/0x90 [ 8597.926207] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8597.931606] ? lock_release+0xc5/0x260 [ 8597.935351] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8597.940578] delayed_work+0x1d8/0x280 [ceph] [ 8597.944860] ? process_scheduled_works+0x22a/0x510 [ 8597.949650] process_scheduled_works+0x2a4/0x510 [ 8597.954265] worker_thread+0x188/0x340 [ 8597.958016] ? rescuer_thread+0x310/0x310 [ 8597.962026] kthread+0xf5/0x130 [ 8597.965165] ? kthread_complete_and_exit+0x20/0x20 [ 8597.969949] ret_from_fork+0x31/0x40 [ 8597.973529] ? kthread_complete_and_exit+0x20/0x20 [ 8597.978321] ret_from_fork_asm+0x11/0x20 [ 8597.982251] [ 8605.456986] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8616.209157] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8625.617362] watchdog: BUG: soft lockup - CPU#7 stuck for 5916s! [kworker/7:4:78808] [ 8625.625028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8625.680478] irq event stamp: 14474220 [ 8625.684136] hardirqs last enabled at (14474219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8625.694301] hardirqs last disabled at (14474220): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8625.704026] softirqs last enabled at (14474142): [] __do_softirq+0x2be/0x371 [ 8625.712711] softirqs last disabled at (14474137): [] irq_exit_rcu+0xb0/0xd0 [ 8625.721233] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8625.730608] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8625.738080] Workqueue: events delayed_work [ceph] [ 8625.742802] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8625.747153] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8625.765889] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8625.771108] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8625.778232] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8625.785355] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8625.792481] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8625.799605] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8625.806727] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8625.814806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8625.820544] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8625.827667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8625.834792] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8625.841916] Call Trace: [ 8625.844363] [ 8625.846374] ? watchdog_timer_fn+0x1b4/0x220 [ 8625.850645] ? __hrtimer_run_queues+0x1d3/0x360 [ 8625.855170] ? hrtimer_interrupt+0x104/0x240 [ 8625.859432] ? __local_bh_enable+0x50/0x60 [ 8625.863524] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8625.868748] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8625.873705] [ 8625.875803] [ 8625.877901] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8625.883209] ? lock_acquire+0xf0/0x2b0 [ 8625.886960] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8625.892366] ? lock_release+0xc5/0x260 [ 8625.896112] _raw_spin_lock+0x30/0x40 [ 8625.899777] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8625.905174] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8625.910405] delayed_work+0x1d8/0x280 [ceph] [ 8625.914685] ? process_scheduled_works+0x22a/0x510 [ 8625.919476] process_scheduled_works+0x2a4/0x510 [ 8625.924090] worker_thread+0x188/0x340 [ 8625.927839] ? rescuer_thread+0x310/0x310 [ 8625.931844] kthread+0xf5/0x130 [ 8625.934991] ? kthread_complete_and_exit+0x20/0x20 [ 8625.939774] ret_from_fork+0x31/0x40 [ 8625.943353] ? kthread_complete_and_exit+0x20/0x20 [ 8625.948136] ret_from_fork_asm+0x11/0x20 [ 8625.952061] [ 8626.449289] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8637.201381] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8647.441607] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8653.617852] watchdog: BUG: soft lockup - CPU#7 stuck for 5942s! [kworker/7:4:78808] [ 8653.625500] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8653.680941] irq event stamp: 14534090 [ 8653.684599] hardirqs last enabled at (14534089): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8653.694756] hardirqs last disabled at (14534090): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8653.704481] softirqs last enabled at (14534076): [] __do_softirq+0x2be/0x371 [ 8653.713165] softirqs last disabled at (14534071): [] irq_exit_rcu+0xb0/0xd0 [ 8653.721676] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8653.731053] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8653.738525] Workqueue: events delayed_work [ceph] [ 8653.743238] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8653.747589] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8653.766326] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8653.771545] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8653.778670] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8653.785792] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8653.792917] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8653.800040] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8653.807165] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8653.815243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8653.820979] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8653.828104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8653.835229] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8653.842352] Call Trace: [ 8653.844797] [ 8653.846809] ? watchdog_timer_fn+0x1b4/0x220 [ 8653.851081] ? __hrtimer_run_queues+0x1d3/0x360 [ 8653.855608] ? hrtimer_interrupt+0x104/0x240 [ 8653.859878] ? __local_bh_enable+0x50/0x60 [ 8653.863970] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8653.869193] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8653.874150] [ 8653.876248] [ 8653.878347] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8653.883654] ? lock_acquire+0xf0/0x2b0 [ 8653.887404] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8653.892812] ? lock_release+0xc5/0x260 [ 8653.896557] _raw_spin_lock+0x30/0x40 [ 8653.900220] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8653.905619] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8653.910848] delayed_work+0x1d8/0x280 [ceph] [ 8653.915129] ? process_scheduled_works+0x22a/0x510 [ 8653.919912] process_scheduled_works+0x2a4/0x510 [ 8653.924527] worker_thread+0x188/0x340 [ 8653.928276] ? rescuer_thread+0x310/0x310 [ 8653.932278] kthread+0xf5/0x130 [ 8653.935418] ? kthread_complete_and_exit+0x20/0x20 [ 8653.940201] ret_from_fork+0x31/0x40 [ 8653.943770] ? kthread_complete_and_exit+0x20/0x20 [ 8653.948555] ret_from_fork_asm+0x11/0x20 [ 8653.952478] [ 8658.193844] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8668.433881] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8679.186216] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8681.618341] watchdog: BUG: soft lockup - CPU#7 stuck for 5968s! [kworker/7:4:78808] [ 8681.625990] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8681.681439] irq event stamp: 14594098 [ 8681.685095] hardirqs last enabled at (14594097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8681.695262] hardirqs last disabled at (14594098): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8681.704986] softirqs last enabled at (14594084): [] __do_softirq+0x2be/0x371 [ 8681.713672] softirqs last disabled at (14594079): [] irq_exit_rcu+0xb0/0xd0 [ 8681.722191] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8681.731568] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8681.739038] Workqueue: events delayed_work [ceph] [ 8681.743763] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8681.748114] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8681.766852] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8681.772068] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8681.779192] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8681.786315] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8681.793441] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8681.800563] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8681.807688] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8681.815766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8681.821503] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8681.828628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8681.835752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8681.842874] Call Trace: [ 8681.845322] [ 8681.847332] ? watchdog_timer_fn+0x1b4/0x220 [ 8681.851596] ? __hrtimer_run_queues+0x1d3/0x360 [ 8681.856123] ? hrtimer_interrupt+0x104/0x240 [ 8681.860391] ? __local_bh_enable+0x50/0x60 [ 8681.864486] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8681.869708] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8681.874667] [ 8681.876762] [ 8681.878860] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8681.884169] ? lock_acquire+0xf0/0x2b0 [ 8681.887919] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8681.892632] ? lock_release+0xc5/0x260 [ 8681.896378] _raw_spin_lock+0x30/0x40 [ 8681.900035] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8681.904654] ceph_check_caps+0x63/0xae0 [ceph] [ 8681.909103] ? __lock_acquire+0xbfa/0x1e20 [ 8681.913201] ? lock_acquire+0xcb/0x2b0 [ 8681.916943] ? igrab+0x1d/0x50 [ 8681.919994] ? find_held_lock+0x2d/0x90 [ 8681.923833] ? find_held_lock+0x2d/0x90 [ 8681.927664] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8681.933063] ? lock_release+0xc5/0x260 [ 8681.936809] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8681.942035] delayed_work+0x1d8/0x280 [ceph] [ 8681.946318] ? process_scheduled_works+0x22a/0x510 [ 8681.951106] process_scheduled_works+0x2a4/0x510 [ 8681.955724] worker_thread+0x188/0x340 [ 8681.959472] ? rescuer_thread+0x310/0x310 [ 8681.963474] kthread+0xf5/0x130 [ 8681.966614] ? kthread_complete_and_exit+0x20/0x20 [ 8681.971396] ret_from_fork+0x31/0x40 [ 8681.974967] ? kthread_complete_and_exit+0x20/0x20 [ 8681.979752] ret_from_fork_asm+0x11/0x20 [ 8681.983675] [ 8689.426310] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8700.178555] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8709.618838] watchdog: BUG: soft lockup - CPU#7 stuck for 5994s! [kworker/7:4:78808] [ 8709.626488] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8709.681937] irq event stamp: 14653938 [ 8709.685592] hardirqs last enabled at (14653937): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8709.695760] hardirqs last disabled at (14653938): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8709.705491] softirqs last enabled at (14653860): [] __do_softirq+0x2be/0x371 [ 8709.714176] softirqs last disabled at (14653855): [] irq_exit_rcu+0xb0/0xd0 [ 8709.722698] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8709.732073] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8709.739546] Workqueue: events delayed_work [ceph] [ 8709.744270] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8709.748627] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8709.767366] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8709.772582] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8709.779706] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8709.786830] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8709.793955] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8709.801078] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8709.808204] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8709.816281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8709.822017] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8709.829143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8709.836266] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8709.843391] Call Trace: [ 8709.845836] [ 8709.847847] ? watchdog_timer_fn+0x1b4/0x220 [ 8709.852109] ? __hrtimer_run_queues+0x1d3/0x360 [ 8709.856637] ? hrtimer_interrupt+0x104/0x240 [ 8709.860906] ? __local_bh_enable+0x50/0x60 [ 8709.864999] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8709.870225] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8709.875180] [ 8709.877277] [ 8709.879375] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8709.884684] ? lock_acquire+0xf0/0x2b0 [ 8709.888432] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8709.893842] ? lock_release+0xc5/0x260 [ 8709.897587] _raw_spin_lock+0x30/0x40 [ 8709.901253] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8709.906650] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8709.911879] delayed_work+0x1d8/0x280 [ceph] [ 8709.916159] ? process_scheduled_works+0x22a/0x510 [ 8709.920949] process_scheduled_works+0x2a4/0x510 [ 8709.925567] worker_thread+0x188/0x340 [ 8709.929316] ? rescuer_thread+0x310/0x310 [ 8709.933327] kthread+0xf5/0x130 [ 8709.936466] ? kthread_complete_and_exit+0x20/0x20 [ 8709.941250] ret_from_fork+0x31/0x40 [ 8709.944829] ? kthread_complete_and_exit+0x20/0x20 [ 8709.949613] ret_from_fork_asm+0x11/0x20 [ 8709.953543] [ 8710.418725] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8721.170868] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8731.411162] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8737.619320] watchdog: BUG: soft lockup - CPU#7 stuck for 6020s! [kworker/7:4:78808] [ 8737.626967] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8737.682416] irq event stamp: 14714278 [ 8737.686073] hardirqs last enabled at (14714277): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8737.696240] hardirqs last disabled at (14714278): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8737.705964] softirqs last enabled at (14714264): [] __do_softirq+0x2be/0x371 [ 8737.714649] softirqs last disabled at (14714259): [] irq_exit_rcu+0xb0/0xd0 [ 8737.723170] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8737.732546] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8737.740016] Workqueue: events delayed_work [ceph] [ 8737.744741] RIP: 0010:lock_release+0x14e/0x260 [ 8737.749178] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [ 8737.767916] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [ 8737.773132] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [ 8737.780256] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [ 8737.787380] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [ 8737.794506] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9425 [ 8737.801629] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [ 8737.808752] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8737.816831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8737.822566] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8737.829691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8737.836815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8737.843940] Call Trace: [ 8737.846388] [ 8737.848399] ? watchdog_timer_fn+0x1b4/0x220 [ 8737.852668] ? __hrtimer_run_queues+0x1d3/0x360 [ 8737.857195] ? hrtimer_interrupt+0x104/0x240 [ 8737.861457] ? __local_bh_enable+0x50/0x60 [ 8737.865550] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8737.870773] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8737.875729] [ 8737.877827] [ 8737.879925] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8737.885230] ? igrab+0x35/0x50 [ 8737.888284] ? lock_release+0x14e/0x260 [ 8737.892121] ? lock_release+0xc5/0x260 [ 8737.895866] _raw_spin_unlock+0x1b/0x40 [ 8737.899706] igrab+0x35/0x50 [ 8737.902583] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 8737.907732] delayed_work+0x1d8/0x280 [ceph] [ 8737.912012] ? process_scheduled_works+0x22a/0x510 [ 8737.916802] process_scheduled_works+0x2a4/0x510 [ 8737.921418] worker_thread+0x188/0x340 [ 8737.925166] ? rescuer_thread+0x310/0x310 [ 8737.929169] kthread+0xf5/0x130 [ 8737.932308] ? kthread_complete_and_exit+0x20/0x20 [ 8737.937093] ret_from_fork+0x31/0x40 [ 8737.940673] ? kthread_complete_and_exit+0x20/0x20 [ 8737.945455] ret_from_fork_asm+0x11/0x20 [ 8737.949387] [ 8742.163301] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8752.403460] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8763.155601] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8765.619810] watchdog: BUG: soft lockup - CPU#7 stuck for 6046s! [kworker/7:4:78808] [ 8765.627465] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8765.682914] irq event stamp: 14774880 [ 8765.686571] hardirqs last enabled at (14774879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8765.696737] hardirqs last disabled at (14774880): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8765.706461] softirqs last enabled at (14774856): [] __do_softirq+0x2be/0x371 [ 8765.715146] softirqs last disabled at (14774851): [] irq_exit_rcu+0xb0/0xd0 [ 8765.723667] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8765.733044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8765.740515] Workqueue: events delayed_work [ceph] [ 8765.745238] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8765.749589] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8765.768327] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 8765.773543] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 8765.780666] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 8765.787792] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 8765.794915] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 8765.802041] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8765.809164] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8765.817240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8765.822979] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8765.830104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8765.837227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8765.844350] Call Trace: [ 8765.846798] [ 8765.848809] ? watchdog_timer_fn+0x1b4/0x220 [ 8765.853081] ? __hrtimer_run_queues+0x1d3/0x360 [ 8765.857614] ? hrtimer_interrupt+0x104/0x240 [ 8765.861875] ? __local_bh_enable+0x50/0x60 [ 8765.865970] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8765.871194] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8765.876149] [ 8765.878248] [ 8765.880346] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8765.885651] ? lock_acquire+0xf0/0x2b0 [ 8765.889394] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8765.894803] _raw_spin_lock+0x30/0x40 [ 8765.898468] ? igrab+0x1d/0x50 [ 8765.901519] igrab+0x1d/0x50 [ 8765.904398] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 8765.909547] delayed_work+0x1d8/0x280 [ceph] [ 8765.913828] ? process_scheduled_works+0x22a/0x510 [ 8765.918616] process_scheduled_works+0x2a4/0x510 [ 8765.923234] worker_thread+0x188/0x340 [ 8765.926982] ? rescuer_thread+0x310/0x310 [ 8765.930985] kthread+0xf5/0x130 [ 8765.934124] ? kthread_complete_and_exit+0x20/0x20 [ 8765.938906] ret_from_fork+0x31/0x40 [ 8765.942486] ? kthread_complete_and_exit+0x20/0x20 [ 8765.947270] ret_from_fork_asm+0x11/0x20 [ 8765.951202] [ 8773.395654] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8784.148162] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8793.620300] watchdog: BUG: soft lockup - CPU#7 stuck for 6072s! [kworker/7:4:78808] [ 8793.627954] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8793.683404] irq event stamp: 14834902 [ 8793.687061] hardirqs last enabled at (14834901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8793.697228] hardirqs last disabled at (14834902): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8793.706959] softirqs last enabled at (14834882): [] __do_softirq+0x2be/0x371 [ 8793.715644] softirqs last disabled at (14834877): [] irq_exit_rcu+0xb0/0xd0 [ 8793.724166] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8793.733550] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8793.741020] Workqueue: events delayed_work [ceph] [ 8793.745743] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8793.750095] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8793.768832] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8793.774049] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8793.781172] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8793.788299] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8793.795422] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8793.802547] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8793.809670] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8793.817749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8793.823484] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8793.830610] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8793.837733] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8793.844856] Call Trace: [ 8793.847304] [ 8793.849315] ? watchdog_timer_fn+0x1b4/0x220 [ 8793.853587] ? __hrtimer_run_queues+0x1d3/0x360 [ 8793.858120] ? hrtimer_interrupt+0x104/0x240 [ 8793.862382] ? __local_bh_enable+0x50/0x60 [ 8793.866475] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8793.871698] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8793.876656] [ 8793.878754] [ 8793.880852] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8793.886158] ? lock_acquire+0xf0/0x2b0 [ 8793.889901] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8793.894616] ? lock_release+0xc5/0x260 [ 8793.898359] _raw_spin_lock+0x30/0x40 [ 8793.902016] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8793.906635] ceph_check_caps+0x63/0xae0 [ceph] [ 8793.911085] ? __lock_acquire+0xbfa/0x1e20 [ 8793.915183] ? lock_acquire+0xcb/0x2b0 [ 8793.918925] ? igrab+0x1d/0x50 [ 8793.921975] ? find_held_lock+0x2d/0x90 [ 8793.925815] ? find_held_lock+0x2d/0x90 [ 8793.929645] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8793.935045] ? lock_release+0xc5/0x260 [ 8793.938791] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8793.944019] delayed_work+0x1d8/0x280 [ceph] [ 8793.948299] ? process_scheduled_works+0x22a/0x510 [ 8793.953089] process_scheduled_works+0x2a4/0x510 [ 8793.957705] worker_thread+0x188/0x340 [ 8793.961455] ? rescuer_thread+0x310/0x310 [ 8793.965466] kthread+0xf5/0x130 [ 8793.968613] ? kthread_complete_and_exit+0x20/0x20 [ 8793.973397] ret_from_fork+0x31/0x40 [ 8793.976975] ? kthread_complete_and_exit+0x20/0x20 [ 8793.981759] ret_from_fork_asm+0x11/0x20 [ 8793.985691] [ 8794.391295] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8805.140320] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8815.380537] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8821.620789] watchdog: BUG: soft lockup - CPU#7 stuck for 6098s! [kworker/7:4:78808] [ 8821.628443] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8821.683892] irq event stamp: 14894512 [ 8821.687550] hardirqs last enabled at (14894511): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8821.697716] hardirqs last disabled at (14894512): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8821.707440] softirqs last enabled at (14894434): [] __do_softirq+0x2be/0x371 [ 8821.716133] softirqs last disabled at (14894429): [] irq_exit_rcu+0xb0/0xd0 [ 8821.724652] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8821.734030] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8821.741501] Workqueue: events delayed_work [ceph] [ 8821.746224] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8821.750574] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8821.769313] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 8821.774529] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 8821.781654] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 8821.788778] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 8821.795903] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 8821.803026] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8821.810152] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8821.818228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8821.823964] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8821.831089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8821.838214] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8821.845338] Call Trace: [ 8821.847784] [ 8821.849795] ? watchdog_timer_fn+0x1b4/0x220 [ 8821.854066] ? __hrtimer_run_queues+0x1d3/0x360 [ 8821.858592] ? hrtimer_interrupt+0x104/0x240 [ 8821.862854] ? __local_bh_enable+0x50/0x60 [ 8821.866947] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8821.872170] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8821.877128] [ 8821.879225] [ 8821.881323] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8821.886629] ? lock_acquire+0xf0/0x2b0 [ 8821.890372] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 8821.895086] ? lock_release+0xc5/0x260 [ 8821.898832] _raw_spin_lock+0x30/0x40 [ 8821.902496] ? ceph_check_caps+0x63/0xae0 [ceph] [ 8821.907115] ceph_check_caps+0x63/0xae0 [ceph] [ 8821.911566] ? __lock_acquire+0xbfa/0x1e20 [ 8821.915664] ? lock_acquire+0xcb/0x2b0 [ 8821.919412] ? igrab+0x1d/0x50 [ 8821.922465] ? find_held_lock+0x2d/0x90 [ 8821.926304] ? find_held_lock+0x2d/0x90 [ 8821.930134] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8821.935533] ? lock_release+0xc5/0x260 [ 8821.939279] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8821.944506] delayed_work+0x1d8/0x280 [ceph] [ 8821.948787] ? process_scheduled_works+0x22a/0x510 [ 8821.953577] process_scheduled_works+0x2a4/0x510 [ 8821.958193] worker_thread+0x188/0x340 [ 8821.961944] ? rescuer_thread+0x310/0x310 [ 8821.965954] kthread+0xf5/0x130 [ 8821.969094] ? kthread_complete_and_exit+0x20/0x20 [ 8821.973876] ret_from_fork+0x31/0x40 [ 8821.977455] ? kthread_complete_and_exit+0x20/0x20 [ 8821.982239] ret_from_fork_asm+0x11/0x20 [ 8821.986171] [ 8826.132699] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8836.372867] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8847.125164] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8849.621279] watchdog: BUG: soft lockup - CPU#7 stuck for 6124s! [kworker/7:4:78808] [ 8849.628933] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8849.684381] irq event stamp: 14954622 [ 8849.688038] hardirqs last enabled at (14954621): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8849.698206] hardirqs last disabled at (14954622): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8849.707929] softirqs last enabled at (14954544): [] __do_softirq+0x2be/0x371 [ 8849.716612] softirqs last disabled at (14954539): [] irq_exit_rcu+0xb0/0xd0 [ 8849.725126] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8849.734501] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8849.741973] Workqueue: events delayed_work [ceph] [ 8849.746696] RIP: 0010:lock_release+0x24/0x260 [ 8849.751047] Code: 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 <31> c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 [ 8849.769785] RSP: 0018:ffffc90002117d60 EFLAGS: 00000286 [ 8849.775001] RAX: c5e6ef7cc8fa9e00 RBX: ffff88817f867488 RCX: ffffc90002117d54 [ 8849.782126] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [ 8849.789249] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [ 8849.796373] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [ 8849.803498] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 8849.810622] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8849.818699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8849.824438] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8849.831562] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8849.838684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8849.845810] Call Trace: [ 8849.848255] [ 8849.850266] ? watchdog_timer_fn+0x1b4/0x220 [ 8849.854530] ? __hrtimer_run_queues+0x1d3/0x360 [ 8849.859063] ? hrtimer_interrupt+0x104/0x240 [ 8849.863325] ? __local_bh_enable+0x50/0x60 [ 8849.867418] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8849.872643] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8849.877600] [ 8849.879697] [ 8849.881796] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8849.887099] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8849.892508] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8849.897907] ? lock_release+0x24/0x260 [ 8849.901652] _raw_spin_unlock+0x1b/0x40 [ 8849.905492] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8849.910720] delayed_work+0x1d8/0x280 [ceph] [ 8849.914999] ? process_scheduled_works+0x22a/0x510 [ 8849.919790] process_scheduled_works+0x2a4/0x510 [ 8849.924405] worker_thread+0x188/0x340 [ 8849.928154] ? rescuer_thread+0x310/0x310 [ 8849.932157] kthread+0xf5/0x130 [ 8849.935296] ? kthread_complete_and_exit+0x20/0x20 [ 8849.940079] ret_from_fork+0x31/0x40 [ 8849.943660] ? kthread_complete_and_exit+0x20/0x20 [ 8849.948442] ret_from_fork_asm+0x11/0x20 [ 8849.952375] [ 8857.365291] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8868.117394] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8877.621769] watchdog: BUG: soft lockup - CPU#7 stuck for 6150s! [kworker/7:4:78808] [ 8877.629422] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8877.684870] irq event stamp: 15014610 [ 8877.688527] hardirqs last enabled at (15014609): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8877.698692] hardirqs last disabled at (15014610): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8877.708418] softirqs last enabled at (15014588): [] __do_softirq+0x2be/0x371 [ 8877.717101] softirqs last disabled at (15014583): [] irq_exit_rcu+0xb0/0xd0 [ 8877.725614] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8877.734991] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8877.742461] Workqueue: events delayed_work [ceph] [ 8877.747186] RIP: 0010:ceph_check_caps+0x835/0xae0 [ceph] [ 8877.752506] Code: 48 8b 82 e0 03 00 00 ff 70 28 52 ba 03 00 00 00 e8 20 10 ae e0 41 5b 41 5d e9 38 fb ff ff 0d 55 0d 00 00 45 31 f6 89 44 24 04 64 f9 ff ff 41 f7 c4 00 01 00 00 0f 84 6c fe ff ff 48 8b 93 18 [ 8877.771243] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [ 8877.776461] RAX: 0000000000000d55 RBX: ffff8881a1d34478 RCX: ffff88817f867000 [ 8877.783584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810da9e008 [ 8877.790710] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 8877.797833] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 8877.804959] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 [ 8877.812082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8877.820158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8877.825896] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8877.833020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8877.840145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8877.847269] Call Trace: [ 8877.849715] [ 8877.851726] ? watchdog_timer_fn+0x1b4/0x220 [ 8877.855999] ? __hrtimer_run_queues+0x1d3/0x360 [ 8877.860525] ? hrtimer_interrupt+0x104/0x240 [ 8877.864794] ? __local_bh_enable+0x50/0x60 [ 8877.868886] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8877.874111] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8877.879069] [ 8877.881165] [ 8877.883263] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8877.888570] ? ceph_check_caps+0x835/0xae0 [ceph] [ 8877.893290] ? lock_acquire+0xcb/0x2b0 [ 8877.897035] ? igrab+0x1d/0x50 [ 8877.900085] ? find_held_lock+0x2d/0x90 [ 8877.903916] ? find_held_lock+0x2d/0x90 [ 8877.907746] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8877.913146] ? lock_release+0xc5/0x260 [ 8877.916892] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8877.922119] delayed_work+0x1d8/0x280 [ceph] [ 8877.926400] ? process_scheduled_works+0x22a/0x510 [ 8877.931191] process_scheduled_works+0x2a4/0x510 [ 8877.935805] worker_thread+0x188/0x340 [ 8877.939554] ? rescuer_thread+0x310/0x310 [ 8877.943557] kthread+0xf5/0x130 [ 8877.946696] ? kthread_complete_and_exit+0x20/0x20 [ 8877.951480] ret_from_fork+0x31/0x40 [ 8877.955060] ? kthread_complete_and_exit+0x20/0x20 [ 8877.959844] ret_from_fork_asm+0x11/0x20 [ 8877.963774] [ 8878.357513] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8889.109776] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8899.350006] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8905.622260] watchdog: BUG: soft lockup - CPU#7 stuck for 6177s! [kworker/7:4:78808] [ 8905.629910] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8905.685360] irq event stamp: 15074654 [ 8905.689017] hardirqs last enabled at (15074653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8905.699184] hardirqs last disabled at (15074654): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8905.708916] softirqs last enabled at (15074528): [] __do_softirq+0x2be/0x371 [ 8905.717609] softirqs last disabled at (15074523): [] irq_exit_rcu+0xb0/0xd0 [ 8905.726129] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8905.735506] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8905.742978] Workqueue: events delayed_work [ceph] [ 8905.747701] RIP: 0010:lock_acquire+0x13/0x2b0 [ 8905.752052] Code: 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 <41> 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 [ 8905.770787] RSP: 0018:ffffc90002117d68 EFLAGS: 00000246 [ 8905.776005] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [ 8905.783129] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [ 8905.790255] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [ 8905.797379] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [ 8905.804501] R13: ffff88817f867478 R14: 0000000000000000 R15: 0000000000000000 [ 8905.811626] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8905.819704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8905.825442] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8905.832565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8905.839690] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8905.846813] Call Trace: [ 8905.849260] [ 8905.851271] ? watchdog_timer_fn+0x1b4/0x220 [ 8905.855543] ? __hrtimer_run_queues+0x1d3/0x360 [ 8905.860077] ? hrtimer_interrupt+0x104/0x240 [ 8905.864339] ? __local_bh_enable+0x50/0x60 [ 8905.868431] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8905.873655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8905.878612] [ 8905.880709] [ 8905.882808] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8905.888114] ? lock_acquire+0x13/0x2b0 [ 8905.891857] _raw_spin_lock+0x30/0x40 [ 8905.895513] ? igrab+0x1d/0x50 [ 8905.898563] igrab+0x1d/0x50 [ 8905.901442] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 8905.906591] delayed_work+0x1d8/0x280 [ceph] [ 8905.910872] ? process_scheduled_works+0x22a/0x510 [ 8905.915662] process_scheduled_works+0x2a4/0x510 [ 8905.920279] worker_thread+0x188/0x340 [ 8905.924027] ? rescuer_thread+0x310/0x310 [ 8905.928030] kthread+0xf5/0x130 [ 8905.931170] ? kthread_complete_and_exit+0x20/0x20 [ 8905.935952] ret_from_fork+0x31/0x40 [ 8905.939533] ? kthread_complete_and_exit+0x20/0x20 [ 8905.944316] ret_from_fork_asm+0x11/0x20 [ 8905.948239] [ 8910.102230] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8920.342357] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8931.094672] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8933.622750] watchdog: BUG: soft lockup - CPU#7 stuck for 6203s! [kworker/7:4:78808] [ 8933.630400] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8933.685847] irq event stamp: 15134510 [ 8933.689507] hardirqs last enabled at (15134509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8933.699671] hardirqs last disabled at (15134510): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8933.709396] softirqs last enabled at (15134478): [] __do_softirq+0x2be/0x371 [ 8933.718089] softirqs last disabled at (15134473): [] irq_exit_rcu+0xb0/0xd0 [ 8933.726610] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8933.735986] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8933.743459] Workqueue: events delayed_work [ceph] [ 8933.748182] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 8933.752533] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 8933.771267] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 8933.776486] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 8933.783609] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 8933.790734] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 8933.797859] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 8933.804982] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8933.812107] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8933.820184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8933.825921] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8933.833046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8933.840170] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8933.847295] Call Trace: [ 8933.849740] [ 8933.851751] ? watchdog_timer_fn+0x1b4/0x220 [ 8933.856014] ? __hrtimer_run_queues+0x1d3/0x360 [ 8933.860539] ? hrtimer_interrupt+0x104/0x240 [ 8933.864802] ? __local_bh_enable+0x50/0x60 [ 8933.868894] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8933.874119] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8933.879077] [ 8933.881172] [ 8933.883271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8933.888577] ? lock_acquire+0xf0/0x2b0 [ 8933.892328] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8933.897736] ? lock_release+0xc5/0x260 [ 8933.901481] _raw_spin_lock+0x30/0x40 [ 8933.905147] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8933.910546] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 8933.915773] delayed_work+0x1d8/0x280 [ceph] [ 8933.920056] ? process_scheduled_works+0x22a/0x510 [ 8933.924844] process_scheduled_works+0x2a4/0x510 [ 8933.929462] worker_thread+0x188/0x340 [ 8933.933210] ? rescuer_thread+0x310/0x310 [ 8933.937212] kthread+0xf5/0x130 [ 8933.940350] ? kthread_complete_and_exit+0x20/0x20 [ 8933.945135] ret_from_fork+0x31/0x40 [ 8933.948705] ? kthread_complete_and_exit+0x20/0x20 [ 8933.953489] ret_from_fork_asm+0x11/0x20 [ 8933.957413] [ 8941.334699] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8952.086977] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8961.623241] watchdog: BUG: soft lockup - CPU#7 stuck for 6229s! [kworker/7:4:78808] [ 8961.630899] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8961.686348] irq event stamp: 15194524 [ 8961.690003] hardirqs last enabled at (15194523): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8961.700169] hardirqs last disabled at (15194524): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8961.709893] softirqs last enabled at (15194510): [] __do_softirq+0x2be/0x371 [ 8961.718579] softirqs last disabled at (15194505): [] irq_exit_rcu+0xb0/0xd0 [ 8961.727099] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8961.736476] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8961.743946] Workqueue: events delayed_work [ceph] [ 8961.748669] RIP: 0010:ceph_check_caps+0x6bb/0xae0 [ceph] [ 8961.753982] Code: 00 00 0f 45 d0 88 94 24 8d 00 00 00 e9 78 fc ff ff 0f b7 03 66 25 00 f0 66 3d 00 40 0f 84 87 01 00 00 48 83 bb 60 08 00 00 00 <8b> 44 24 20 0f 84 64 01 00 00 0d 55 01 00 00 45 31 f6 89 44 24 04 [ 8961.772719] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [ 8961.777938] RAX: 0000000000004000 RBX: ffff8881a1d34478 RCX: ffff88817f867000 [ 8961.785061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810da9e008 [ 8961.792186] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 8961.799310] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 8961.806433] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 [ 8961.813558] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8961.821636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8961.827374] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8961.834497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8961.841620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8961.848745] Call Trace: [ 8961.851192] [ 8961.853203] ? watchdog_timer_fn+0x1b4/0x220 [ 8961.857473] ? __hrtimer_run_queues+0x1d3/0x360 [ 8961.862000] ? hrtimer_interrupt+0x104/0x240 [ 8961.866270] ? __local_bh_enable+0x50/0x60 [ 8961.870363] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8961.875587] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8961.880545] [ 8961.882640] [ 8961.884740] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8961.890046] ? ceph_check_caps+0x6bb/0xae0 [ceph] [ 8961.894768] ? lock_acquire+0xcb/0x2b0 [ 8961.898510] ? igrab+0x1d/0x50 [ 8961.901561] ? find_held_lock+0x2d/0x90 [ 8961.905393] ? find_held_lock+0x2d/0x90 [ 8961.909223] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 8961.914624] ? lock_release+0xc5/0x260 [ 8961.918368] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 8961.923596] delayed_work+0x1d8/0x280 [ceph] [ 8961.927876] ? process_scheduled_works+0x22a/0x510 [ 8961.932665] process_scheduled_works+0x2a4/0x510 [ 8961.937281] worker_thread+0x188/0x340 [ 8961.941032] ? rescuer_thread+0x310/0x310 [ 8961.945034] kthread+0xf5/0x130 [ 8961.948173] ? kthread_complete_and_exit+0x20/0x20 [ 8961.952955] ret_from_fork+0x31/0x40 [ 8961.956535] ? kthread_complete_and_exit+0x20/0x20 [ 8961.961318] ret_from_fork_asm+0x11/0x20 [ 8961.965242] [ 8962.327171] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8973.079542] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8983.319500] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 8989.623732] watchdog: BUG: soft lockup - CPU#7 stuck for 6255s! [kworker/7:4:78808] [ 8989.631404] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 8989.686853] irq event stamp: 15254898 [ 8989.690510] hardirqs last enabled at (15254897): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8989.700677] hardirqs last disabled at (15254898): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 8989.710408] softirqs last enabled at (15254772): [] __do_softirq+0x2be/0x371 [ 8989.719093] softirqs last disabled at (15254767): [] irq_exit_rcu+0xb0/0xd0 [ 8989.727606] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 8989.736981] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 8989.744453] Workqueue: events delayed_work [ceph] [ 8989.749175] RIP: 0010:do_raw_spin_lock+0x24/0xc0 [ 8989.753787] Code: 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 <48> 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba [ 8989.772525] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [ 8989.777742] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [ 8989.784865] RDX: ffffffffffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 8989.791991] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 8989.799114] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 8989.806237] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 8989.813361] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 8989.821439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8989.827178] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 8989.834302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8989.841426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8989.848550] Call Trace: [ 8989.850997] [ 8989.853007] ? watchdog_timer_fn+0x1b4/0x220 [ 8989.857279] ? __hrtimer_run_queues+0x1d3/0x360 [ 8989.861812] ? hrtimer_interrupt+0x104/0x240 [ 8989.866073] ? __local_bh_enable+0x50/0x60 [ 8989.870168] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 8989.875392] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 8989.880349] [ 8989.882447] [ 8989.884542] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 8989.889852] ? do_raw_spin_lock+0x24/0xc0 [ 8989.893861] igrab+0x1d/0x50 [ 8989.896739] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 8989.901890] delayed_work+0x1d8/0x280 [ceph] [ 8989.906171] ? process_scheduled_works+0x22a/0x510 [ 8989.910960] process_scheduled_works+0x2a4/0x510 [ 8989.915576] worker_thread+0x188/0x340 [ 8989.919324] ? rescuer_thread+0x310/0x310 [ 8989.923327] kthread+0xf5/0x130 [ 8989.926465] ? kthread_complete_and_exit+0x20/0x20 [ 8989.931250] ret_from_fork+0x31/0x40 [ 8989.934830] ? kthread_complete_and_exit+0x20/0x20 [ 8989.939613] ret_from_fork_asm+0x11/0x20 [ 8989.943535] [ 8994.071676] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9004.311775] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9015.064180] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9017.624223] watchdog: BUG: soft lockup - CPU#7 stuck for 6281s! [kworker/7:4:78808] [ 9017.631875] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9017.687324] irq event stamp: 15315860 [ 9017.690981] hardirqs last enabled at (15315859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9017.701148] hardirqs last disabled at (15315860): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9017.710873] softirqs last enabled at (15315858): [] __do_softirq+0x2be/0x371 [ 9017.719565] softirqs last disabled at (15315849): [] irq_exit_rcu+0xb0/0xd0 [ 9017.728078] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9017.737454] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9017.744925] Workqueue: events delayed_work [ceph] [ 9017.749648] RIP: 0010:rb_next+0x0/0x50 [ 9017.753392] Code: 85 d2 75 f4 c3 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 07 48 85 c0 75 05 eb 0d 48 89 d0 48 8b 50 08 48 85 d2 75 f4 c3 c3 0f 1f 40 00 0f 1e fa 48 8b 0f 48 39 cf 74 39 48 8b 47 08 48 85 c0 74 22 48 [ 9017.772129] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [ 9017.777348] RAX: 0000000000000000 RBX: ffff88810da9e000 RCX: 00000000ffffffff [ 9017.784472] RDX: 0000000000000155 RSI: 0000000000000000 RDI: ffff88810da9e008 [ 9017.791594] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 9017.798719] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 9017.805844] R13: ffff8881a1d34478 R14: 0000000000000000 R15: ffff88810da9e008 [ 9017.812966] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9017.821046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9017.826781] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9017.833908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9017.841030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9017.848155] Call Trace: [ 9017.850600] [ 9017.852613] ? watchdog_timer_fn+0x1b4/0x220 [ 9017.856882] ? __hrtimer_run_queues+0x1d3/0x360 [ 9017.861409] ? hrtimer_interrupt+0x104/0x240 [ 9017.865671] ? __local_bh_enable+0x50/0x60 [ 9017.869763] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9017.874988] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9017.879945] [ 9017.882042] [ 9017.884140] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9017.889448] ? rb_last+0x20/0x20 [ 9017.892679] ceph_check_caps+0x228/0xae0 [ceph] [ 9017.897228] ? lock_acquire+0xcb/0x2b0 [ 9017.900972] ? igrab+0x1d/0x50 [ 9017.904020] ? find_held_lock+0x2d/0x90 [ 9017.907851] ? find_held_lock+0x2d/0x90 [ 9017.911683] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9017.917084] ? lock_release+0xc5/0x260 [ 9017.920830] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9017.926064] delayed_work+0x1d8/0x280 [ceph] [ 9017.930346] ? process_scheduled_works+0x22a/0x510 [ 9017.935134] process_scheduled_works+0x2a4/0x510 [ 9017.939751] worker_thread+0x188/0x340 [ 9017.943501] ? rescuer_thread+0x310/0x310 [ 9017.947504] kthread+0xf5/0x130 [ 9017.950651] ? kthread_complete_and_exit+0x20/0x20 [ 9017.955433] ret_from_fork+0x31/0x40 [ 9017.959005] ? kthread_complete_and_exit+0x20/0x20 [ 9017.963789] ret_from_fork_asm+0x11/0x20 [ 9017.967721] [ 9025.304383] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9036.056464] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9045.624713] watchdog: BUG: soft lockup - CPU#7 stuck for 6307s! [kworker/7:4:78808] [ 9045.632364] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9045.687815] irq event stamp: 15376352 [ 9045.691470] hardirqs last enabled at (15376351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9045.701636] hardirqs last disabled at (15376352): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9045.711362] softirqs last enabled at (15376230): [] __do_softirq+0x2be/0x371 [ 9045.720053] softirqs last disabled at (15376225): [] irq_exit_rcu+0xb0/0xd0 [ 9045.728566] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9045.737943] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9045.745414] Workqueue: events delayed_work [ceph] [ 9045.750137] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9045.754489] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9045.773225] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9045.778442] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9045.785567] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9045.792691] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9045.799815] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9045.806939] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9045.814064] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9045.822141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9045.827879] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9045.835003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9045.842126] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9045.849251] Call Trace: [ 9045.851696] [ 9045.853707] ? watchdog_timer_fn+0x1b4/0x220 [ 9045.857971] ? __hrtimer_run_queues+0x1d3/0x360 [ 9045.862507] ? hrtimer_interrupt+0x104/0x240 [ 9045.866774] ? __local_bh_enable+0x50/0x60 [ 9045.870869] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9045.876092] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9045.881049] [ 9045.883148] [ 9045.885245] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9045.890551] ? lock_acquire+0xf0/0x2b0 [ 9045.894303] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9045.899710] _raw_spin_lock+0x30/0x40 [ 9045.903367] ? igrab+0x1d/0x50 [ 9045.906417] igrab+0x1d/0x50 [ 9045.909296] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9045.914438] delayed_work+0x1d8/0x280 [ceph] [ 9045.918718] ? process_scheduled_works+0x22a/0x510 [ 9045.923507] process_scheduled_works+0x2a4/0x510 [ 9045.928125] worker_thread+0x188/0x340 [ 9045.931874] ? rescuer_thread+0x310/0x310 [ 9045.935876] kthread+0xf5/0x130 [ 9045.939023] ? kthread_complete_and_exit+0x20/0x20 [ 9045.943806] ret_from_fork+0x31/0x40 [ 9045.947386] ? kthread_complete_and_exit+0x20/0x20 [ 9045.952169] ret_from_fork_asm+0x11/0x20 [ 9045.956093] [ 9046.296521] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9057.048925] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9067.289041] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9073.625204] watchdog: BUG: soft lockup - CPU#7 stuck for 6333s! [kworker/7:4:78808] [ 9073.632855] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9073.688304] irq event stamp: 15436464 [ 9073.691960] hardirqs last enabled at (15436463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9073.702128] hardirqs last disabled at (15436464): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9073.711859] softirqs last enabled at (15436444): [] __do_softirq+0x2be/0x371 [ 9073.720542] softirqs last disabled at (15436439): [] irq_exit_rcu+0xb0/0xd0 [ 9073.729057] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9073.738440] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9073.745912] Workqueue: events delayed_work [ceph] [ 9073.750636] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9073.754987] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9073.773724] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9073.778942] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9073.786063] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9073.793188] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9073.800314] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9073.807438] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9073.814561] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9073.822637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9073.828375] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9073.835501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9073.842624] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9073.849749] Call Trace: [ 9073.852195] [ 9073.854205] ? watchdog_timer_fn+0x1b4/0x220 [ 9073.858468] ? __hrtimer_run_queues+0x1d3/0x360 [ 9073.862994] ? hrtimer_interrupt+0x104/0x240 [ 9073.867255] ? __local_bh_enable+0x50/0x60 [ 9073.871350] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9073.876574] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9073.881529] [ 9073.883627] [ 9073.885725] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9073.891033] ? lock_acquire+0xf0/0x2b0 [ 9073.894783] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9073.899496] ? lock_release+0xc5/0x260 [ 9073.903242] _raw_spin_lock+0x30/0x40 [ 9073.906898] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9073.911516] ceph_check_caps+0x63/0xae0 [ceph] [ 9073.915968] ? __lock_acquire+0xbfa/0x1e20 [ 9073.920068] ? lock_acquire+0xcb/0x2b0 [ 9073.923816] ? igrab+0x1d/0x50 [ 9073.926866] ? find_held_lock+0x2d/0x90 [ 9073.930705] ? find_held_lock+0x2d/0x90 [ 9073.934537] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9073.939936] ? lock_release+0xc5/0x260 [ 9073.943682] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9073.948908] delayed_work+0x1d8/0x280 [ceph] [ 9073.953190] ? process_scheduled_works+0x22a/0x510 [ 9073.957981] process_scheduled_works+0x2a4/0x510 [ 9073.962596] worker_thread+0x188/0x340 [ 9073.966345] ? rescuer_thread+0x310/0x310 [ 9073.970347] kthread+0xf5/0x130 [ 9073.973486] ? kthread_complete_and_exit+0x20/0x20 [ 9073.978269] ret_from_fork+0x31/0x40 [ 9073.981849] ? kthread_complete_and_exit+0x20/0x20 [ 9073.986633] ret_from_fork_asm+0x11/0x20 [ 9073.990555] [ 9078.041115] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9101.625698] watchdog: BUG: soft lockup - CPU#7 stuck for 6359s! [kworker/7:4:78808] [ 9101.633352] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9101.688802] irq event stamp: 15497494 [ 9101.692458] hardirqs last enabled at (15497493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9101.702625] hardirqs last disabled at (15497494): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9101.712356] softirqs last enabled at (15497416): [] __do_softirq+0x2be/0x371 [ 9101.721042] softirqs last disabled at (15497411): [] irq_exit_rcu+0xb0/0xd0 [ 9101.729561] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9101.738938] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9101.746410] Workqueue: events delayed_work [ceph] [ 9101.751134] RIP: 0010:_raw_spin_unlock+0xa/0x40 [ 9101.755656] Code: b8 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 <48> 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d [ 9101.774393] RSP: 0018:ffffc90002117da8 EFLAGS: 00000286 [ 9101.779613] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [ 9101.786735] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88817f867488 [ 9101.793859] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 9101.800983] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9101.808108] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9101.815231] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9101.823310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9101.829048] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9101.836171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9101.843296] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9101.850419] Call Trace: [ 9101.852866] [ 9101.854875] ? watchdog_timer_fn+0x1b4/0x220 [ 9101.859140] ? __hrtimer_run_queues+0x1d3/0x360 [ 9101.863675] ? hrtimer_interrupt+0x104/0x240 [ 9101.867945] ? __local_bh_enable+0x50/0x60 [ 9101.872038] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9101.877260] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9101.882218] [ 9101.884315] [ 9101.886413] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9101.891720] ? _raw_spin_unlock+0xa/0x40 [ 9101.895643] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9101.900882] delayed_work+0x1d8/0x280 [ceph] [ 9101.905161] ? process_scheduled_works+0x22a/0x510 [ 9101.909951] process_scheduled_works+0x2a4/0x510 [ 9101.914569] worker_thread+0x188/0x340 [ 9101.918317] ? rescuer_thread+0x310/0x310 [ 9101.922322] kthread+0xf5/0x130 [ 9101.925459] ? kthread_complete_and_exit+0x20/0x20 [ 9101.930241] ret_from_fork+0x31/0x40 [ 9101.933822] ? kthread_complete_and_exit+0x20/0x20 [ 9101.938606] ret_from_fork_asm+0x11/0x20 [ 9101.942537] [ 9119.513405] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [ 9119.521889] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9129.626189] watchdog: BUG: soft lockup - CPU#7 stuck for 6385s! [kworker/7:4:78808] [ 9129.633841] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9129.689290] irq event stamp: 15557336 [ 9129.692948] hardirqs last enabled at (15557335): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9129.703114] hardirqs last disabled at (15557336): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9129.712857] softirqs last enabled at (15557286): [] __do_softirq+0x2be/0x371 [ 9129.721549] softirqs last disabled at (15557281): [] irq_exit_rcu+0xb0/0xd0 [ 9129.730085] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9129.739463] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9129.746934] Workqueue: events delayed_work [ceph] [ 9129.751656] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9129.756008] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9129.774744] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9129.779963] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9129.787086] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9129.794211] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9129.801335] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9129.808459] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9129.815582] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9129.823661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9129.829398] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9129.836521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9129.843645] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9129.850770] Call Trace: [ 9129.853215] [ 9129.855228] ? watchdog_timer_fn+0x1b4/0x220 [ 9129.859497] ? __hrtimer_run_queues+0x1d3/0x360 [ 9129.864026] ? hrtimer_interrupt+0x104/0x240 [ 9129.868294] ? __local_bh_enable+0x50/0x60 [ 9129.872389] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9129.877611] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9129.882583] [ 9129.884684] [ 9129.886781] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9129.892088] ? lock_acquire+0xf0/0x2b0 [ 9129.895831] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9129.900579] ? lock_release+0xc5/0x260 [ 9129.904324] _raw_spin_lock+0x30/0x40 [ 9129.907981] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9129.912598] ceph_check_caps+0x63/0xae0 [ceph] [ 9129.917052] ? __lock_acquire+0xbfa/0x1e20 [ 9129.921148] ? lock_acquire+0xcb/0x2b0 [ 9129.924888] ? igrab+0x1d/0x50 [ 9129.927939] ? find_held_lock+0x2d/0x90 [ 9129.931770] ? find_held_lock+0x2d/0x90 [ 9129.935602] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9129.941002] ? lock_release+0xc5/0x260 [ 9129.944748] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9129.949982] delayed_work+0x1d8/0x280 [ceph] [ 9129.954264] ? process_scheduled_works+0x22a/0x510 [ 9129.959053] process_scheduled_works+0x2a4/0x510 [ 9129.963669] worker_thread+0x188/0x340 [ 9129.967419] ? rescuer_thread+0x310/0x310 [ 9129.971421] kthread+0xf5/0x130 [ 9129.974585] ? kthread_complete_and_exit+0x20/0x20 [ 9129.979369] ret_from_fork+0x31/0x40 [ 9129.982949] ? kthread_complete_and_exit+0x20/0x20 [ 9129.987733] ret_from_fork_asm+0x11/0x20 [ 9129.991664] [ 9130.265951] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9140.506204] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9157.626681] watchdog: BUG: soft lockup - CPU#7 stuck for 6411s! [kworker/7:4:78808] [ 9157.634330] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9157.689780] irq event stamp: 15618602 [ 9157.693436] hardirqs last enabled at (15618601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9157.703603] hardirqs last disabled at (15618602): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9157.713334] softirqs last enabled at (15618524): [] __do_softirq+0x2be/0x371 [ 9157.722020] softirqs last disabled at (15618519): [] irq_exit_rcu+0xb0/0xd0 [ 9157.730542] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9157.739916] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9157.747386] Workqueue: events delayed_work [ceph] [ 9157.752111] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9157.756462] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9157.775199] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9157.780416] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9157.787541] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9157.794665] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9157.801790] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9157.808913] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9157.816037] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9157.824115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9157.829851] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9157.836977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9157.844100] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9157.851224] Call Trace: [ 9157.853671] [ 9157.855681] ? watchdog_timer_fn+0x1b4/0x220 [ 9157.859945] ? __hrtimer_run_queues+0x1d3/0x360 [ 9157.864478] ? hrtimer_interrupt+0x104/0x240 [ 9157.868740] ? __local_bh_enable+0x50/0x60 [ 9157.872833] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9157.878057] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9157.883016] [ 9157.885111] [ 9157.887210] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9157.892517] ? lock_acquire+0xf0/0x2b0 [ 9157.896259] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9157.900974] ? lock_release+0xc5/0x260 [ 9157.904717] _raw_spin_lock+0x30/0x40 [ 9157.908376] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9157.912992] ceph_check_caps+0x63/0xae0 [ceph] [ 9157.917445] ? __lock_acquire+0xbfa/0x1e20 [ 9157.921544] ? lock_acquire+0xcb/0x2b0 [ 9157.925293] ? igrab+0x1d/0x50 [ 9157.928344] ? find_held_lock+0x2d/0x90 [ 9157.932173] ? find_held_lock+0x2d/0x90 [ 9157.936003] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9157.941403] ? lock_release+0xc5/0x260 [ 9157.945149] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9157.950378] delayed_work+0x1d8/0x280 [ceph] [ 9157.954656] ? process_scheduled_works+0x22a/0x510 [ 9157.959446] process_scheduled_works+0x2a4/0x510 [ 9157.964064] worker_thread+0x188/0x340 [ 9157.967813] ? rescuer_thread+0x310/0x310 [ 9157.971815] kthread+0xf5/0x130 [ 9157.974953] ? kthread_complete_and_exit+0x20/0x20 [ 9157.979736] ret_from_fork+0x31/0x40 [ 9157.983317] ? kthread_complete_and_exit+0x20/0x20 [ 9157.988101] ret_from_fork_asm+0x11/0x20 [ 9157.992031] [ 9172.250333] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [ 9172.258934] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9182.490964] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9185.627168] watchdog: BUG: soft lockup - CPU#7 stuck for 6437s! [kworker/7:4:78808] [ 9185.634819] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9185.690269] irq event stamp: 15682464 [ 9185.693926] hardirqs last enabled at (15682463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9185.704093] hardirqs last disabled at (15682464): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9185.713824] softirqs last enabled at (15682404): [] __do_softirq+0x2be/0x371 [ 9185.722508] softirqs last disabled at (15682397): [] irq_exit_rcu+0xb0/0xd0 [ 9185.731022] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9185.740397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9185.747870] Workqueue: events delayed_work [ceph] [ 9185.752591] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9185.756942] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9185.775679] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 9185.780898] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 9185.788021] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 9185.795145] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 9185.802271] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 9185.809395] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9185.816517] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9185.824595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9185.830333] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9185.837458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9185.844582] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9185.851704] Call Trace: [ 9185.854152] [ 9185.856162] ? watchdog_timer_fn+0x1b4/0x220 [ 9185.860425] ? __hrtimer_run_queues+0x1d3/0x360 [ 9185.864950] ? hrtimer_interrupt+0x104/0x240 [ 9185.869212] ? __local_bh_enable+0x50/0x60 [ 9185.873304] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9185.878529] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9185.883486] [ 9185.885583] [ 9185.887681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9185.892989] ? lock_acquire+0xf0/0x2b0 [ 9185.896739] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9185.902139] ? lock_release+0xc5/0x260 [ 9185.905883] _raw_spin_lock+0x30/0x40 [ 9185.909540] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9185.914939] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9185.920169] delayed_work+0x1d8/0x280 [ceph] [ 9185.924450] ? process_scheduled_works+0x22a/0x510 [ 9185.929240] process_scheduled_works+0x2a4/0x510 [ 9185.933855] worker_thread+0x188/0x340 [ 9185.937603] ? rescuer_thread+0x310/0x310 [ 9185.941606] kthread+0xf5/0x130 [ 9185.944744] ? kthread_complete_and_exit+0x20/0x20 [ 9185.949529] ret_from_fork+0x31/0x40 [ 9185.953109] ? kthread_complete_and_exit+0x20/0x20 [ 9185.957892] ret_from_fork_asm+0x11/0x20 [ 9185.961815] [ 9193.243192] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9203.483278] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9213.627659] watchdog: BUG: soft lockup - CPU#7 stuck for 6463s! [kworker/7:4:78808] [ 9213.635308] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9213.690759] irq event stamp: 15744524 [ 9213.694415] hardirqs last enabled at (15744523): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9213.704582] hardirqs last disabled at (15744524): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9213.714314] softirqs last enabled at (15744510): [] __do_softirq+0x2be/0x371 [ 9213.723008] softirqs last disabled at (15744505): [] irq_exit_rcu+0xb0/0xd0 [ 9213.731528] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9213.740905] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9213.748376] Workqueue: events delayed_work [ceph] [ 9213.753098] RIP: 0010:do_raw_spin_unlock+0x2b/0xc0 [ 9213.757882] Code: 0f 1f 00 0f 1f 44 00 00 53 81 7f 04 ad 4e ad de 48 89 fb 75 34 8b 03 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 <65> 8b 05 4a bf eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 [ 9213.776621] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [ 9213.781838] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [ 9213.788961] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [ 9213.796086] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 9213.803209] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9213.810334] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9213.817458] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9213.825535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9213.831273] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9213.838398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9213.845522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9213.852645] Call Trace: [ 9213.855093] [ 9213.857103] ? watchdog_timer_fn+0x1b4/0x220 [ 9213.861374] ? __hrtimer_run_queues+0x1d3/0x360 [ 9213.865901] ? hrtimer_interrupt+0x104/0x240 [ 9213.870170] ? __local_bh_enable+0x50/0x60 [ 9213.874262] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9213.879486] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9213.884446] [ 9213.886542] [ 9213.888640] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9213.893947] ? do_raw_spin_unlock+0x2b/0xc0 [ 9213.898130] _raw_spin_unlock+0x23/0x40 [ 9213.901961] igrab+0x35/0x50 [ 9213.904839] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9213.909988] delayed_work+0x1d8/0x280 [ceph] [ 9213.914270] ? process_scheduled_works+0x22a/0x510 [ 9213.919058] process_scheduled_works+0x2a4/0x510 [ 9213.923676] worker_thread+0x188/0x340 [ 9213.927425] ? rescuer_thread+0x310/0x310 [ 9213.931426] kthread+0xf5/0x130 [ 9213.934565] ? kthread_complete_and_exit+0x20/0x20 [ 9213.939349] ret_from_fork+0x31/0x40 [ 9213.942929] ? kthread_complete_and_exit+0x20/0x20 [ 9213.947711] ret_from_fork_asm+0x11/0x20 [ 9213.951636] [ 9214.235440] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9224.475586] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9235.227834] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9241.628150] watchdog: BUG: soft lockup - CPU#7 stuck for 6489s! [kworker/7:4:78808] [ 9241.635816] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9241.691265] irq event stamp: 15805972 [ 9241.694922] hardirqs last enabled at (15805971): [] asm_common_interrupt+0x26/0x40 [ 9241.704135] hardirqs last disabled at (15805972): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9241.713858] softirqs last enabled at (15805970): [] __do_softirq+0x2be/0x371 [ 9241.722566] softirqs last disabled at (15805963): [] irq_exit_rcu+0xb0/0xd0 [ 9241.731082] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9241.740467] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9241.747938] Workqueue: events delayed_work [ceph] [ 9241.752659] RIP: 0010:ceph_check_caps+0x0/0xae0 [ceph] [ 9241.757801] Code: ca 76 1a 48 8b b7 68 08 00 00 31 c0 48 39 f2 76 0c 48 01 c9 48 01 f2 48 39 d1 0f 93 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 <66> 0f 1f 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 89 f5 53 48 [ 9241.776536] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [ 9241.781755] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d6c [ 9241.788879] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8881a1d34478 [ 9241.796001] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 9241.803126] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9241.810251] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9241.817375] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9241.825453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9241.831190] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9241.838315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9241.845437] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9241.852561] Call Trace: [ 9241.855008] [ 9241.857019] ? watchdog_timer_fn+0x1b4/0x220 [ 9241.861283] ? __hrtimer_run_queues+0x1d3/0x360 [ 9241.865817] ? hrtimer_interrupt+0x104/0x240 [ 9241.870077] ? __local_bh_enable+0x50/0x60 [ 9241.874171] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9241.879397] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9241.884353] [ 9241.886449] [ 9241.888548] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9241.893854] ? __ceph_should_report_size+0x50/0x50 [ceph] [ 9241.899261] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9241.904489] delayed_work+0x1d8/0x280 [ceph] [ 9241.908769] ? process_scheduled_works+0x22a/0x510 [ 9241.913558] process_scheduled_works+0x2a4/0x510 [ 9241.918176] worker_thread+0x188/0x340 [ 9241.921925] ? rescuer_thread+0x310/0x310 [ 9241.925928] kthread+0xf5/0x130 [ 9241.929066] ? kthread_complete_and_exit+0x20/0x20 [ 9241.933850] ret_from_fork+0x31/0x40 [ 9241.937429] ? kthread_complete_and_exit+0x20/0x20 [ 9241.942212] ret_from_fork_asm+0x11/0x20 [ 9241.946136] [ 9245.468016] libceph: auth protocol 'cephx' mauth authentication failed: -13 [ 9269.628642] watchdog: BUG: soft lockup - CPU#7 stuck for 6516s! [kworker/7:4:78808] [ 9269.636296] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9269.691745] irq event stamp: 15866818 [ 9269.695402] hardirqs last enabled at (15866817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9269.705568] hardirqs last disabled at (15866818): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9269.715292] softirqs last enabled at (15866770): [] __do_softirq+0x2be/0x371 [ 9269.723976] softirqs last disabled at (15866765): [] irq_exit_rcu+0xb0/0xd0 [ 9269.732489] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9269.741865] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9269.749336] Workqueue: events delayed_work [ceph] [ 9269.754060] RIP: 0010:iput+0x17a/0x220 [ 9269.757805] Code: f2 ff 84 c0 74 08 65 48 ff 0d 1a 02 c7 7e 48 89 ef e8 da 40 7f 00 48 89 df 5b 5d 41 5c 41 5d e9 cc fa ff ff 5b 5d 41 5c 41 5d 48 8b 83 d8 00 00 00 a9 00 00 01 00 75 08 41 f6 44 24 53 40 75 [ 9269.776540] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [ 9269.781759] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 9269.788882] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [ 9269.796008] RBP: ffff88817f867000 R08: 0000000000000000 R09: 0000000000000000 [ 9269.803130] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000ea60 [ 9269.810254] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9269.817379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9269.825457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9269.831194] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9269.838318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9269.845442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9269.852566] Call Trace: [ 9269.855014] [ 9269.857023] ? watchdog_timer_fn+0x1b4/0x220 [ 9269.861294] ? __hrtimer_run_queues+0x1d3/0x360 [ 9269.865821] ? hrtimer_interrupt+0x104/0x240 [ 9269.870081] ? __local_bh_enable+0x50/0x60 [ 9269.874176] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9269.879400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9269.884356] [ 9269.886453] [ 9269.888551] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9269.893858] ? iput+0x17a/0x220 [ 9269.896994] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 9269.902230] delayed_work+0x1d8/0x280 [ceph] [ 9269.906511] ? process_scheduled_works+0x22a/0x510 [ 9269.911301] process_scheduled_works+0x2a4/0x510 [ 9269.915918] worker_thread+0x188/0x340 [ 9269.919666] ? rescuer_thread+0x310/0x310 [ 9269.923668] kthread+0xf5/0x130 [ 9269.926807] ? kthread_complete_and_exit+0x20/0x20 [ 9269.931592] ret_from_fork+0x31/0x40 [ 9269.935171] ? kthread_complete_and_exit+0x20/0x20 [ 9269.939953] ret_from_fork_asm+0x11/0x20 [ 9269.943877] [ 9277.212188] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [ 9297.629133] watchdog: BUG: soft lockup - CPU#7 stuck for 6542s! [kworker/7:4:78808] [ 9297.636784] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9297.692235] irq event stamp: 15927376 [ 9297.695891] hardirqs last enabled at (15927375): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9297.706059] hardirqs last disabled at (15927376): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9297.715782] softirqs last enabled at (15927324): [] __do_softirq+0x2be/0x371 [ 9297.724474] softirqs last disabled at (15927319): [] irq_exit_rcu+0xb0/0xd0 [ 9297.732995] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9297.742373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9297.749844] Workqueue: events delayed_work [ceph] [ 9297.754565] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9297.758919] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9297.777654] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9297.782872] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9297.789997] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9297.797120] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9297.804245] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9297.811368] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9297.818493] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9297.826569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9297.832307] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9297.839431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9297.846557] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9297.853680] Call Trace: [ 9297.856125] [ 9297.858135] ? watchdog_timer_fn+0x1b4/0x220 [ 9297.862400] ? __hrtimer_run_queues+0x1d3/0x360 [ 9297.866935] ? hrtimer_interrupt+0x104/0x240 [ 9297.871204] ? __local_bh_enable+0x50/0x60 [ 9297.875296] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9297.880521] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9297.885478] [ 9297.887575] [ 9297.889674] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9297.894982] ? lock_acquire+0xf0/0x2b0 [ 9297.898732] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9297.903446] ? lock_release+0xc5/0x260 [ 9297.907190] _raw_spin_lock+0x30/0x40 [ 9297.910846] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9297.915465] ceph_check_caps+0x63/0xae0 [ceph] [ 9297.919918] ? __lock_acquire+0xbfa/0x1e20 [ 9297.924014] ? lock_acquire+0xcb/0x2b0 [ 9297.927755] ? igrab+0x1d/0x50 [ 9297.930806] ? find_held_lock+0x2d/0x90 [ 9297.934636] ? find_held_lock+0x2d/0x90 [ 9297.938469] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9297.943867] ? lock_release+0xc5/0x260 [ 9297.947613] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9297.952840] delayed_work+0x1d8/0x280 [ceph] [ 9297.957121] ? process_scheduled_works+0x22a/0x510 [ 9297.961911] process_scheduled_works+0x2a4/0x510 [ 9297.966527] worker_thread+0x188/0x340 [ 9297.970276] ? rescuer_thread+0x310/0x310 [ 9297.974281] kthread+0xf5/0x130 [ 9297.977426] ? kthread_complete_and_exit+0x20/0x20 [ 9297.982209] ret_from_fork+0x31/0x40 [ 9297.985789] ? kthread_complete_and_exit+0x20/0x20 [ 9297.990574] ret_from_fork_asm+0x11/0x20 [ 9297.994506] [ 9325.629624] watchdog: BUG: soft lockup - CPU#7 stuck for 6568s! [kworker/7:4:78808] [ 9325.637292] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9325.692741] irq event stamp: 15989334 [ 9325.696397] hardirqs last enabled at (15989333): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9325.706564] hardirqs last disabled at (15989334): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9325.716296] softirqs last enabled at (15989270): [] __do_softirq+0x2be/0x371 [ 9325.724980] softirqs last disabled at (15989257): [] irq_exit_rcu+0xb0/0xd0 [ 9325.733494] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9325.742871] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9325.750343] Workqueue: events delayed_work [ceph] [ 9325.755065] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [ 9325.759685] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [ 9325.778420] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [ 9325.783638] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [ 9325.790762] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [ 9325.797887] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 9325.805010] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9325.812134] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9325.819259] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9325.827337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9325.833073] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9325.840199] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9325.847322] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9325.854447] Call Trace: [ 9325.856894] [ 9325.858904] ? watchdog_timer_fn+0x1b4/0x220 [ 9325.863175] ? __hrtimer_run_queues+0x1d3/0x360 [ 9325.867701] ? hrtimer_interrupt+0x104/0x240 [ 9325.871962] ? __local_bh_enable+0x50/0x60 [ 9325.876054] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9325.881280] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9325.886237] [ 9325.888334] [ 9325.890433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9325.895738] ? do_raw_spin_lock+0x42/0xc0 [ 9325.899749] igrab+0x1d/0x50 [ 9325.902636] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9325.907787] delayed_work+0x1d8/0x280 [ceph] [ 9325.912067] ? process_scheduled_works+0x22a/0x510 [ 9325.916857] process_scheduled_works+0x2a4/0x510 [ 9325.921472] worker_thread+0x188/0x340 [ 9325.925222] ? rescuer_thread+0x310/0x310 [ 9325.929223] kthread+0xf5/0x130 [ 9325.932362] ? kthread_complete_and_exit+0x20/0x20 [ 9325.937147] ret_from_fork+0x31/0x40 [ 9325.940725] ? kthread_complete_and_exit+0x20/0x20 [ 9325.945509] ret_from_fork_asm+0x11/0x20 [ 9325.949434] [ 9353.630115] watchdog: BUG: soft lockup - CPU#7 stuck for 6594s! [kworker/7:4:78808] [ 9353.637764] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9353.693213] irq event stamp: 16053410 [ 9353.696871] hardirqs last enabled at (16053409): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9353.707037] hardirqs last disabled at (16053410): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9353.716768] softirqs last enabled at (16053366): [] __do_softirq+0x2be/0x371 [ 9353.725454] softirqs last disabled at (16053361): [] irq_exit_rcu+0xb0/0xd0 [ 9353.733973] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9353.743350] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9353.750821] Workqueue: events delayed_work [ceph] [ 9353.755544] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9353.759895] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9353.778633] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9353.783849] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9353.790975] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9353.798099] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9353.805223] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9353.812347] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9353.819472] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9353.827548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9353.833285] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9353.840411] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9353.847535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9353.854657] Call Trace: [ 9353.857104] [ 9353.859115] ? watchdog_timer_fn+0x1b4/0x220 [ 9353.863379] ? __hrtimer_run_queues+0x1d3/0x360 [ 9353.867905] ? hrtimer_interrupt+0x104/0x240 [ 9353.872173] ? __local_bh_enable+0x50/0x60 [ 9353.876268] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9353.881492] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9353.886450] [ 9353.888547] [ 9353.890644] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9353.895951] ? lock_acquire+0xf0/0x2b0 [ 9353.899693] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9353.905102] _raw_spin_lock+0x30/0x40 [ 9353.908766] ? igrab+0x1d/0x50 [ 9353.911816] igrab+0x1d/0x50 [ 9353.914696] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9353.919844] delayed_work+0x1d8/0x280 [ceph] [ 9353.924126] ? process_scheduled_works+0x22a/0x510 [ 9353.928916] process_scheduled_works+0x2a4/0x510 [ 9353.933532] worker_thread+0x188/0x340 [ 9353.937282] ? rescuer_thread+0x310/0x310 [ 9353.941294] kthread+0xf5/0x130 [ 9353.944440] ? kthread_complete_and_exit+0x20/0x20 [ 9353.949222] ret_from_fork+0x31/0x40 [ 9353.952803] ? kthread_complete_and_exit+0x20/0x20 [ 9353.957587] ret_from_fork_asm+0x11/0x20 [ 9353.961517] [ 9381.630607] watchdog: BUG: soft lockup - CPU#7 stuck for 6620s! [kworker/7:4:78808] [ 9381.638263] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9381.693711] irq event stamp: 16114642 [ 9381.697368] hardirqs last enabled at (16114641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9381.707534] hardirqs last disabled at (16114642): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9381.717259] softirqs last enabled at (16114622): [] __do_softirq+0x2be/0x371 [ 9381.725953] softirqs last disabled at (16114617): [] irq_exit_rcu+0xb0/0xd0 [ 9381.734472] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9381.743850] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9381.751319] Workqueue: events delayed_work [ceph] [ 9381.756044] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9381.760402] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9381.779140] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 9381.784357] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 9381.791481] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 9381.798607] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 9381.805730] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 9381.812853] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9381.819979] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9381.828055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9381.833794] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9381.840918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9381.848041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9381.855164] Call Trace: [ 9381.857613] [ 9381.859621] ? watchdog_timer_fn+0x1b4/0x220 [ 9381.863884] ? __hrtimer_run_queues+0x1d3/0x360 [ 9381.868411] ? hrtimer_interrupt+0x104/0x240 [ 9381.872671] ? __local_bh_enable+0x50/0x60 [ 9381.876765] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9381.881989] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9381.886947] [ 9381.889043] [ 9381.891141] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9381.896449] ? lock_acquire+0xf0/0x2b0 [ 9381.900199] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9381.905606] ? lock_release+0xc5/0x260 [ 9381.909352] _raw_spin_lock+0x30/0x40 [ 9381.913008] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9381.918417] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9381.923655] delayed_work+0x1d8/0x280 [ceph] [ 9381.927933] ? process_scheduled_works+0x22a/0x510 [ 9381.932715] process_scheduled_works+0x2a4/0x510 [ 9381.937331] worker_thread+0x188/0x340 [ 9381.941079] ? rescuer_thread+0x310/0x310 [ 9381.945083] kthread+0xf5/0x130 [ 9381.948221] ? kthread_complete_and_exit+0x20/0x20 [ 9381.953007] ret_from_fork+0x31/0x40 [ 9381.956587] ? kthread_complete_and_exit+0x20/0x20 [ 9381.961377] ret_from_fork_asm+0x11/0x20 [ 9381.965309] [ 9409.631101] watchdog: BUG: soft lockup - CPU#7 stuck for 6646s! [kworker/7:4:78808] [ 9409.638778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9409.694227] irq event stamp: 16177874 [ 9409.697884] hardirqs last enabled at (16177873): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9409.708049] hardirqs last disabled at (16177874): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9409.717773] softirqs last enabled at (16177860): [] __do_softirq+0x2be/0x371 [ 9409.726458] softirqs last disabled at (16177855): [] irq_exit_rcu+0xb0/0xd0 [ 9409.734979] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9409.744356] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9409.751825] Workqueue: events delayed_work [ceph] [ 9409.756549] RIP: 0010:_atomic_dec_and_lock+0x14/0x40 [ 9409.761508] Code: 63 c9 48 83 f9 07 76 ba b8 08 00 00 00 eb d2 44 89 d0 5b c3 66 90 f3 0f 1e fa 55 53 8b 07 83 f8 01 74 0e 8d 50 ff f0 0f b1 17 <75> f2 31 c0 5b 5d c3 48 89 fb 48 89 f7 48 89 f5 e8 07 2b 04 00 f0 [ 9409.780244] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [ 9409.785461] RAX: 0000000000000002 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 9409.792585] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [ 9409.799710] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [ 9409.806835] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [ 9409.813958] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9409.821082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9409.829160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9409.834896] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9409.842022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9409.849146] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9409.856270] Call Trace: [ 9409.858715] [ 9409.860728] ? watchdog_timer_fn+0x1b4/0x220 [ 9409.864997] ? __hrtimer_run_queues+0x1d3/0x360 [ 9409.869525] ? hrtimer_interrupt+0x104/0x240 [ 9409.873786] ? __local_bh_enable+0x50/0x60 [ 9409.877879] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9409.883103] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9409.888061] [ 9409.890156] [ 9409.892254] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9409.897562] ? _atomic_dec_and_lock+0x14/0x40 [ 9409.901920] iput+0x50/0x220 [ 9409.904806] ceph_check_delayed_caps+0x1be/0x270 [ceph] [ 9409.910043] delayed_work+0x1d8/0x280 [ceph] [ 9409.914323] ? process_scheduled_works+0x22a/0x510 [ 9409.919113] process_scheduled_works+0x2a4/0x510 [ 9409.923728] worker_thread+0x188/0x340 [ 9409.927476] ? rescuer_thread+0x310/0x310 [ 9409.931482] kthread+0xf5/0x130 [ 9409.934620] ? kthread_complete_and_exit+0x20/0x20 [ 9409.939402] ret_from_fork+0x31/0x40 [ 9409.942974] ? kthread_complete_and_exit+0x20/0x20 [ 9409.947756] ret_from_fork_asm+0x11/0x20 [ 9409.951680] [ 9437.631595] watchdog: BUG: soft lockup - CPU#7 stuck for 6672s! [kworker/7:4:78808] [ 9437.639267] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9437.694714] irq event stamp: 16241572 [ 9437.698373] hardirqs last enabled at (16241571): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9437.708538] hardirqs last disabled at (16241572): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9437.718264] softirqs last enabled at (16241530): [] __do_softirq+0x2be/0x371 [ 9437.726957] softirqs last disabled at (16241525): [] irq_exit_rcu+0xb0/0xd0 [ 9437.735477] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9437.744855] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9437.752325] Workqueue: events delayed_work [ceph] [ 9437.757037] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9437.761389] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9437.780127] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 9437.785344] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 9437.792468] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 9437.799593] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 9437.806716] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 9437.813840] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9437.820965] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9437.829044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9437.834780] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9437.841905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9437.849028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9437.856153] Call Trace: [ 9437.858598] [ 9437.860610] ? watchdog_timer_fn+0x1b4/0x220 [ 9437.864881] ? __hrtimer_run_queues+0x1d3/0x360 [ 9437.869407] ? hrtimer_interrupt+0x104/0x240 [ 9437.873669] ? __local_bh_enable+0x50/0x60 [ 9437.877762] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9437.882984] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9437.887941] [ 9437.890040] [ 9437.892137] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9437.897444] ? lock_acquire+0xf0/0x2b0 [ 9437.901187] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9437.906593] ? lock_release+0xc5/0x260 [ 9437.910339] _raw_spin_lock+0x30/0x40 [ 9437.913997] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9437.919394] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9437.924623] delayed_work+0x1d8/0x280 [ceph] [ 9437.928904] ? process_scheduled_works+0x22a/0x510 [ 9437.933694] process_scheduled_works+0x2a4/0x510 [ 9437.938311] worker_thread+0x188/0x340 [ 9437.942061] ? rescuer_thread+0x310/0x310 [ 9437.946071] kthread+0xf5/0x130 [ 9437.949210] ? kthread_complete_and_exit+0x20/0x20 [ 9437.953994] ret_from_fork+0x31/0x40 [ 9437.957572] ? kthread_complete_and_exit+0x20/0x20 [ 9437.962356] ret_from_fork_asm+0x11/0x20 [ 9437.966280] [ 9465.632090] watchdog: BUG: soft lockup - CPU#7 stuck for 6698s! [kworker/7:4:78808] [ 9465.639738] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9465.695189] irq event stamp: 16304056 [ 9465.698844] hardirqs last enabled at (16304055): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9465.709011] hardirqs last disabled at (16304056): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9465.718742] softirqs last enabled at (16304042): [] __do_softirq+0x2be/0x371 [ 9465.727428] softirqs last disabled at (16304037): [] irq_exit_rcu+0xb0/0xd0 [ 9465.735949] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9465.745326] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9465.752794] Workqueue: events delayed_work [ceph] [ 9465.757520] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9465.761880] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9465.780616] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9465.785833] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9465.792956] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9465.800082] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9465.807205] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9465.814330] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9465.821455] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9465.829531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9465.835270] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9465.842392] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9465.849518] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9465.856642] Call Trace: [ 9465.859088] [ 9465.861099] ? watchdog_timer_fn+0x1b4/0x220 [ 9465.865371] ? __hrtimer_run_queues+0x1d3/0x360 [ 9465.869904] ? hrtimer_interrupt+0x104/0x240 [ 9465.874165] ? __local_bh_enable+0x50/0x60 [ 9465.878258] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9465.883484] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9465.888441] [ 9465.890537] [ 9465.892635] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9465.897942] ? lock_acquire+0xf0/0x2b0 [ 9465.901685] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9465.906399] ? lock_release+0xc5/0x260 [ 9465.910145] _raw_spin_lock+0x30/0x40 [ 9465.913809] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9465.918426] ceph_check_caps+0x63/0xae0 [ceph] [ 9465.922879] ? __lock_acquire+0xbfa/0x1e20 [ 9465.926978] ? lock_acquire+0xcb/0x2b0 [ 9465.930727] ? igrab+0x1d/0x50 [ 9465.933786] ? find_held_lock+0x2d/0x90 [ 9465.937625] ? find_held_lock+0x2d/0x90 [ 9465.941457] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9465.946856] ? lock_release+0xc5/0x260 [ 9465.950600] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9465.955829] delayed_work+0x1d8/0x280 [ceph] [ 9465.960108] ? process_scheduled_works+0x22a/0x510 [ 9465.964899] process_scheduled_works+0x2a4/0x510 [ 9465.969514] worker_thread+0x188/0x340 [ 9465.973264] ? rescuer_thread+0x310/0x310 [ 9465.977266] kthread+0xf5/0x130 [ 9465.980405] ? kthread_complete_and_exit+0x20/0x20 [ 9465.985190] ret_from_fork+0x31/0x40 [ 9465.988768] ? kthread_complete_and_exit+0x20/0x20 [ 9465.993551] ret_from_fork_asm+0x11/0x20 [ 9465.997474] [ 9493.632589] watchdog: BUG: soft lockup - CPU#7 stuck for 6724s! [kworker/7:4:78808] [ 9493.640246] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9493.695693] irq event stamp: 16366722 [ 9493.699351] hardirqs last enabled at (16366721): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9493.709518] hardirqs last disabled at (16366722): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9493.719250] softirqs last enabled at (16366644): [] __do_softirq+0x2be/0x371 [ 9493.727933] softirqs last disabled at (16366639): [] irq_exit_rcu+0xb0/0xd0 [ 9493.736447] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9493.745823] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9493.753293] Workqueue: events delayed_work [ceph] [ 9493.758018] RIP: 0010:_raw_spin_unlock+0x1e/0x40 [ 9493.762629] Code: 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b [ 9493.781364] RSP: 0018:ffffc90002117da8 EFLAGS: 00000286 [ 9493.786584] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: ffffc90002117d6c [ 9493.793707] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88817f867488 [ 9493.800831] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 9493.807954] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9493.815079] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9493.822203] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9493.830281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9493.836018] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9493.843141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9493.850267] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9493.857390] Call Trace: [ 9493.859836] [ 9493.861847] ? watchdog_timer_fn+0x1b4/0x220 [ 9493.866110] ? __hrtimer_run_queues+0x1d3/0x360 [ 9493.870637] ? hrtimer_interrupt+0x104/0x240 [ 9493.874908] ? __local_bh_enable+0x50/0x60 [ 9493.879001] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9493.884223] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9493.889181] [ 9493.891279] [ 9493.893375] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9493.898685] ? _raw_spin_unlock+0x1e/0x40 [ 9493.902694] ? _raw_spin_unlock+0x1b/0x40 [ 9493.906696] ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9493.911935] delayed_work+0x1d8/0x280 [ceph] [ 9493.916215] ? process_scheduled_works+0x22a/0x510 [ 9493.921004] process_scheduled_works+0x2a4/0x510 [ 9493.925622] worker_thread+0x188/0x340 [ 9493.929371] ? rescuer_thread+0x310/0x310 [ 9493.933372] kthread+0xf5/0x130 [ 9493.936510] ? kthread_complete_and_exit+0x20/0x20 [ 9493.941295] ret_from_fork+0x31/0x40 [ 9493.944866] ? kthread_complete_and_exit+0x20/0x20 [ 9493.949650] ret_from_fork_asm+0x11/0x20 [ 9493.953571] [ 9521.633091] watchdog: BUG: soft lockup - CPU#7 stuck for 6750s! [kworker/7:4:78808] [ 9521.640744] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9521.696193] irq event stamp: 16429796 [ 9521.699849] hardirqs last enabled at (16429795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9521.710016] hardirqs last disabled at (16429796): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9521.719748] softirqs last enabled at (16429754): [] __do_softirq+0x2be/0x371 [ 9521.728442] softirqs last disabled at (16429749): [] irq_exit_rcu+0xb0/0xd0 [ 9521.736960] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9521.746338] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9521.753808] Workqueue: events delayed_work [ceph] [ 9521.758532] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9521.762883] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9521.781621] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9521.786837] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9521.793963] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9521.801085] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9521.808210] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9521.815335] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9521.822460] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9521.830537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9521.836273] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9521.843399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9521.850521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9521.857645] Call Trace: [ 9521.860092] [ 9521.862103] ? watchdog_timer_fn+0x1b4/0x220 [ 9521.866366] ? __hrtimer_run_queues+0x1d3/0x360 [ 9521.870901] ? hrtimer_interrupt+0x104/0x240 [ 9521.875170] ? __local_bh_enable+0x50/0x60 [ 9521.879264] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9521.884488] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9521.889444] [ 9521.891541] [ 9521.893639] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9521.898948] ? lock_acquire+0xf0/0x2b0 [ 9521.902697] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9521.907411] ? lock_release+0xc5/0x260 [ 9521.911156] _raw_spin_lock+0x30/0x40 [ 9521.914813] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9521.919432] ceph_check_caps+0x63/0xae0 [ceph] [ 9521.923884] ? __lock_acquire+0xbfa/0x1e20 [ 9521.927982] ? lock_acquire+0xcb/0x2b0 [ 9521.931731] ? igrab+0x1d/0x50 [ 9521.934782] ? find_held_lock+0x2d/0x90 [ 9521.938621] ? find_held_lock+0x2d/0x90 [ 9521.942451] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9521.947851] ? lock_release+0xc5/0x260 [ 9521.951597] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9521.956825] delayed_work+0x1d8/0x280 [ceph] [ 9521.961104] ? process_scheduled_works+0x22a/0x510 [ 9521.965894] process_scheduled_works+0x2a4/0x510 [ 9521.970511] worker_thread+0x188/0x340 [ 9521.974259] ? rescuer_thread+0x310/0x310 [ 9521.978262] kthread+0xf5/0x130 [ 9521.981400] ? kthread_complete_and_exit+0x20/0x20 [ 9521.986184] ret_from_fork+0x31/0x40 [ 9521.989755] ? kthread_complete_and_exit+0x20/0x20 [ 9521.994539] ret_from_fork_asm+0x11/0x20 [ 9521.998471] [ 9549.633573] watchdog: BUG: soft lockup - CPU#7 stuck for 6776s! [kworker/7:4:78808] [ 9549.641223] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9549.696674] irq event stamp: 16492012 [ 9549.700330] hardirqs last enabled at (16492011): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9549.710497] hardirqs last disabled at (16492012): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9549.720228] softirqs last enabled at (16491934): [] __do_softirq+0x2be/0x371 [ 9549.728912] softirqs last disabled at (16491929): [] irq_exit_rcu+0xb0/0xd0 [ 9549.737434] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9549.746810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9549.754280] Workqueue: events delayed_work [ceph] [ 9549.759005] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9549.763356] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9549.782093] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9549.787310] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9549.794433] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9549.801560] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9549.808682] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9549.815805] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9549.822929] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9549.831008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9549.836747] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9549.843869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9549.850992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9549.858119] Call Trace: [ 9549.860564] [ 9549.862573] ? watchdog_timer_fn+0x1b4/0x220 [ 9549.866837] ? __hrtimer_run_queues+0x1d3/0x360 [ 9549.871364] ? hrtimer_interrupt+0x104/0x240 [ 9549.875624] ? __local_bh_enable+0x50/0x60 [ 9549.879719] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9549.884943] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9549.889900] [ 9549.891997] [ 9549.894095] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9549.899403] ? lock_acquire+0xf0/0x2b0 [ 9549.903154] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9549.908560] _raw_spin_lock+0x30/0x40 [ 9549.912216] ? igrab+0x1d/0x50 [ 9549.915268] igrab+0x1d/0x50 [ 9549.918146] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9549.923287] delayed_work+0x1d8/0x280 [ceph] [ 9549.927568] ? process_scheduled_works+0x22a/0x510 [ 9549.932359] process_scheduled_works+0x2a4/0x510 [ 9549.936973] worker_thread+0x188/0x340 [ 9549.940722] ? rescuer_thread+0x310/0x310 [ 9549.944725] kthread+0xf5/0x130 [ 9549.947864] ? kthread_complete_and_exit+0x20/0x20 [ 9549.952649] ret_from_fork+0x31/0x40 [ 9549.956228] ? kthread_complete_and_exit+0x20/0x20 [ 9549.961019] ret_from_fork_asm+0x11/0x20 [ 9549.964944] [ 9577.634067] watchdog: BUG: soft lockup - CPU#7 stuck for 6802s! [kworker/7:4:78808] [ 9577.641736] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9577.697187] irq event stamp: 16555604 [ 9577.700844] hardirqs last enabled at (16555603): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9577.711012] hardirqs last disabled at (16555604): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9577.720744] softirqs last enabled at (16555584): [] __do_softirq+0x2be/0x371 [ 9577.729437] softirqs last disabled at (16555575): [] irq_exit_rcu+0xb0/0xd0 [ 9577.737957] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9577.747333] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9577.754805] Workqueue: events delayed_work [ceph] [ 9577.759529] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9577.763881] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9577.782617] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9577.787833] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9577.794958] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9577.802083] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9577.809205] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9577.816329] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9577.823453] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9577.831531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9577.837269] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9577.844394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9577.851517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9577.858642] Call Trace: [ 9577.861087] [ 9577.863098] ? watchdog_timer_fn+0x1b4/0x220 [ 9577.867363] ? __hrtimer_run_queues+0x1d3/0x360 [ 9577.871896] ? hrtimer_interrupt+0x104/0x240 [ 9577.876157] ? __local_bh_enable+0x50/0x60 [ 9577.880251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9577.885476] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9577.890431] [ 9577.892528] [ 9577.894627] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9577.899934] ? lock_acquire+0xf0/0x2b0 [ 9577.903685] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9577.909094] _raw_spin_lock+0x30/0x40 [ 9577.912750] ? igrab+0x1d/0x50 [ 9577.915800] igrab+0x1d/0x50 [ 9577.918679] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9577.923821] delayed_work+0x1d8/0x280 [ceph] [ 9577.928102] ? process_scheduled_works+0x22a/0x510 [ 9577.932890] process_scheduled_works+0x2a4/0x510 [ 9577.937505] worker_thread+0x188/0x340 [ 9577.941254] ? rescuer_thread+0x310/0x310 [ 9577.945259] kthread+0xf5/0x130 [ 9577.948406] ? kthread_complete_and_exit+0x20/0x20 [ 9577.953189] ret_from_fork+0x31/0x40 [ 9577.956769] ? kthread_complete_and_exit+0x20/0x20 [ 9577.961551] ret_from_fork_asm+0x11/0x20 [ 9577.965475] [ 9605.634568] watchdog: BUG: soft lockup - CPU#7 stuck for 6828s! [kworker/7:4:78808] [ 9605.642217] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9605.697668] irq event stamp: 16621114 [ 9605.701324] hardirqs last enabled at (16621113): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9605.711491] hardirqs last disabled at (16621114): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9605.721217] softirqs last enabled at (16621094): [] __do_softirq+0x2be/0x371 [ 9605.729909] softirqs last disabled at (16621089): [] irq_exit_rcu+0xb0/0xd0 [ 9605.738430] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9605.747807] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9605.755278] Workqueue: events delayed_work [ceph] [ 9605.760000] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9605.764352] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9605.783089] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9605.788306] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9605.795429] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9605.802555] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9605.809679] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9605.816801] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9605.823925] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9605.832005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9605.837740] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9605.844864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9605.851990] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9605.859114] Call Trace: [ 9605.861559] [ 9605.863570] ? watchdog_timer_fn+0x1b4/0x220 [ 9605.867833] ? __hrtimer_run_queues+0x1d3/0x360 [ 9605.872361] ? hrtimer_interrupt+0x104/0x240 [ 9605.876631] ? __local_bh_enable+0x50/0x60 [ 9605.880723] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9605.885947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9605.890903] [ 9605.893001] [ 9605.895099] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9605.900406] ? lock_acquire+0xf0/0x2b0 [ 9605.904157] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9605.908872] ? lock_release+0xc5/0x260 [ 9605.912616] _raw_spin_lock+0x30/0x40 [ 9605.916272] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9605.920891] ceph_check_caps+0x63/0xae0 [ceph] [ 9605.925343] ? __lock_acquire+0xbfa/0x1e20 [ 9605.929439] ? lock_acquire+0xcb/0x2b0 [ 9605.933180] ? igrab+0x1d/0x50 [ 9605.936231] ? find_held_lock+0x2d/0x90 [ 9605.940062] ? find_held_lock+0x2d/0x90 [ 9605.943894] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9605.949291] ? lock_release+0xc5/0x260 [ 9605.953038] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9605.958264] delayed_work+0x1d8/0x280 [ceph] [ 9605.962546] ? process_scheduled_works+0x22a/0x510 [ 9605.967335] process_scheduled_works+0x2a4/0x510 [ 9605.971953] worker_thread+0x188/0x340 [ 9605.975701] ? rescuer_thread+0x310/0x310 [ 9605.979704] kthread+0xf5/0x130 [ 9605.982842] ? kthread_complete_and_exit+0x20/0x20 [ 9605.987625] ret_from_fork+0x31/0x40 [ 9605.991196] ? kthread_complete_and_exit+0x20/0x20 [ 9605.995980] ret_from_fork_asm+0x11/0x20 [ 9605.999904] [ 9633.635056] watchdog: BUG: soft lockup - CPU#7 stuck for 6855s! [kworker/7:4:78808] [ 9633.642707] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9633.698158] irq event stamp: 16683938 [ 9633.701815] hardirqs last enabled at (16683937): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9633.711981] hardirqs last disabled at (16683938): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9633.721713] softirqs last enabled at (16683860): [] __do_softirq+0x2be/0x371 [ 9633.730397] softirqs last disabled at (16683855): [] irq_exit_rcu+0xb0/0xd0 [ 9633.738912] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9633.748295] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9633.755767] Workqueue: events delayed_work [ceph] [ 9633.760490] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9633.764840] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9633.783578] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9633.788794] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9633.795919] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9633.803044] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9633.810166] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9633.817292] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9633.824417] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9633.832492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9633.838231] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9633.845354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9633.852479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9633.859603] Call Trace: [ 9633.862050] [ 9633.864069] ? watchdog_timer_fn+0x1b4/0x220 [ 9633.868340] ? __hrtimer_run_queues+0x1d3/0x360 [ 9633.872875] ? hrtimer_interrupt+0x104/0x240 [ 9633.877144] ? __local_bh_enable+0x50/0x60 [ 9633.881237] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9633.886462] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9633.891420] [ 9633.893515] [ 9633.895613] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9633.900923] ? lock_acquire+0xf0/0x2b0 [ 9633.904674] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9633.910080] _raw_spin_lock+0x30/0x40 [ 9633.913736] ? igrab+0x1d/0x50 [ 9633.916787] igrab+0x1d/0x50 [ 9633.919665] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9633.924808] delayed_work+0x1d8/0x280 [ceph] [ 9633.929086] ? process_scheduled_works+0x22a/0x510 [ 9633.933869] process_scheduled_works+0x2a4/0x510 [ 9633.938484] worker_thread+0x188/0x340 [ 9633.942233] ? rescuer_thread+0x310/0x310 [ 9633.946236] kthread+0xf5/0x130 [ 9633.949375] ? kthread_complete_and_exit+0x20/0x20 [ 9633.954159] ret_from_fork+0x31/0x40 [ 9633.957739] ? kthread_complete_and_exit+0x20/0x20 [ 9633.962530] ret_from_fork_asm+0x11/0x20 [ 9633.966455] [ 9661.635550] watchdog: BUG: soft lockup - CPU#7 stuck for 6881s! [kworker/7:4:78808] [ 9661.643205] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9661.698655] irq event stamp: 16746968 [ 9661.702314] hardirqs last enabled at (16746967): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9661.712480] hardirqs last disabled at (16746968): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9661.722211] softirqs last enabled at (16746948): [] __do_softirq+0x2be/0x371 [ 9661.730896] softirqs last disabled at (16746943): [] irq_exit_rcu+0xb0/0xd0 [ 9661.739417] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9661.748793] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9661.756265] Workqueue: events delayed_work [ceph] [ 9661.760987] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9661.765339] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9661.784076] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9661.789291] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9661.796418] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9661.803540] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9661.810664] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9661.817788] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9661.824913] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9661.832990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9661.838727] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9661.845852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9661.852977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9661.860101] Call Trace: [ 9661.862545] [ 9661.864558] ? watchdog_timer_fn+0x1b4/0x220 [ 9661.868830] ? __hrtimer_run_queues+0x1d3/0x360 [ 9661.873365] ? hrtimer_interrupt+0x104/0x240 [ 9661.877634] ? __local_bh_enable+0x50/0x60 [ 9661.881726] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9661.886950] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9661.891909] [ 9661.894006] [ 9661.896104] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9661.901410] ? lock_acquire+0xf0/0x2b0 [ 9661.905152] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9661.910560] _raw_spin_lock+0x30/0x40 [ 9661.914218] ? igrab+0x1d/0x50 [ 9661.917268] igrab+0x1d/0x50 [ 9661.920145] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9661.925286] delayed_work+0x1d8/0x280 [ceph] [ 9661.929569] ? process_scheduled_works+0x22a/0x510 [ 9661.934357] process_scheduled_works+0x2a4/0x510 [ 9661.938975] worker_thread+0x188/0x340 [ 9661.942724] ? rescuer_thread+0x310/0x310 [ 9661.946727] kthread+0xf5/0x130 [ 9661.949874] ? kthread_complete_and_exit+0x20/0x20 [ 9661.954664] ret_from_fork+0x31/0x40 [ 9661.958245] ? kthread_complete_and_exit+0x20/0x20 [ 9661.963029] ret_from_fork_asm+0x11/0x20 [ 9661.966962] [ 9689.636046] watchdog: BUG: soft lockup - CPU#7 stuck for 6907s! [kworker/7:4:78808] [ 9689.643696] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9689.699144] irq event stamp: 16807956 [ 9689.702801] hardirqs last enabled at (16807955): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9689.712968] hardirqs last disabled at (16807956): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9689.722692] softirqs last enabled at (16807936): [] __do_softirq+0x2be/0x371 [ 9689.731386] softirqs last disabled at (16807931): [] irq_exit_rcu+0xb0/0xd0 [ 9689.739907] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9689.749290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9689.756762] Workqueue: events delayed_work [ceph] [ 9689.761484] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9689.765835] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9689.784574] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9689.789791] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9689.796914] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9689.804040] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9689.811163] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9689.818287] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9689.825412] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9689.833488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9689.839225] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9689.846349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9689.853474] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9689.860599] Call Trace: [ 9689.863046] [ 9689.865056] ? watchdog_timer_fn+0x1b4/0x220 [ 9689.869328] ? __hrtimer_run_queues+0x1d3/0x360 [ 9689.873861] ? hrtimer_interrupt+0x104/0x240 [ 9689.878123] ? __local_bh_enable+0x50/0x60 [ 9689.882216] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9689.887441] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9689.892398] [ 9689.894493] [ 9689.896592] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9689.901899] ? lock_acquire+0xf0/0x2b0 [ 9689.905642] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9689.911050] _raw_spin_lock+0x30/0x40 [ 9689.914707] ? igrab+0x1d/0x50 [ 9689.917756] igrab+0x1d/0x50 [ 9689.920636] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9689.925778] delayed_work+0x1d8/0x280 [ceph] [ 9689.930057] ? process_scheduled_works+0x22a/0x510 [ 9689.934847] process_scheduled_works+0x2a4/0x510 [ 9689.939462] worker_thread+0x188/0x340 [ 9689.943214] ? rescuer_thread+0x310/0x310 [ 9689.947223] kthread+0xf5/0x130 [ 9689.950362] ? kthread_complete_and_exit+0x20/0x20 [ 9689.955147] ret_from_fork+0x31/0x40 [ 9689.958726] ? kthread_complete_and_exit+0x20/0x20 [ 9689.963509] ret_from_fork_asm+0x11/0x20 [ 9689.967441] [ 9717.636547] watchdog: BUG: soft lockup - CPU#7 stuck for 6933s! [kworker/7:4:78808] [ 9717.644201] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9717.699653] irq event stamp: 16869472 [ 9717.703308] hardirqs last enabled at (16869471): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9717.713476] hardirqs last disabled at (16869472): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9717.723208] softirqs last enabled at (16869452): [] __do_softirq+0x2be/0x371 [ 9717.731899] softirqs last disabled at (16869447): [] irq_exit_rcu+0xb0/0xd0 [ 9717.740412] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9717.749789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9717.757259] Workqueue: events delayed_work [ceph] [ 9717.761983] RIP: 0010:ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9717.767999] Code: ff 70 28 41 57 e8 85 bc ad e0 5f 41 58 e9 b6 fe ff ff 66 90 48 89 d0 e9 41 ff ff ff 48 8b 3c 24 48 89 44 24 08 e8 c5 5a 07 e1 <48> 8b 44 24 08 66 90 31 f6 48 89 df 48 89 44 24 08 e8 0f a4 ff ff [ 9717.786734] RSP: 0018:ffffc90002117db8 EFLAGS: 00000246 [ 9717.791953] RAX: ffffffffffffffff RBX: ffff8881a1d34478 RCX: ffffc90002117d6c [ 9717.799077] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88817f867488 [ 9717.806200] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [ 9717.813324] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9717.820449] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9717.827572] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9717.835651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9717.841389] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9717.848511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9717.855636] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9717.862761] Call Trace: [ 9717.865206] [ 9717.867217] ? watchdog_timer_fn+0x1b4/0x220 [ 9717.871481] ? __hrtimer_run_queues+0x1d3/0x360 [ 9717.876016] ? hrtimer_interrupt+0x104/0x240 [ 9717.880286] ? __local_bh_enable+0x50/0x60 [ 9717.884377] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9717.889601] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9717.894559] [ 9717.896656] [ 9717.898754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9717.904061] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9717.909466] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9717.914867] delayed_work+0x1d8/0x280 [ceph] [ 9717.919150] ? process_scheduled_works+0x22a/0x510 [ 9717.923940] process_scheduled_works+0x2a4/0x510 [ 9717.928554] worker_thread+0x188/0x340 [ 9717.932304] ? rescuer_thread+0x310/0x310 [ 9717.936307] kthread+0xf5/0x130 [ 9717.939445] ? kthread_complete_and_exit+0x20/0x20 [ 9717.944230] ret_from_fork+0x31/0x40 [ 9717.947809] ? kthread_complete_and_exit+0x20/0x20 [ 9717.952592] ret_from_fork_asm+0x11/0x20 [ 9717.956515] [ 9745.637034] watchdog: BUG: soft lockup - CPU#7 stuck for 6959s! [kworker/7:4:78808] [ 9745.644683] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9745.700132] irq event stamp: 16931292 [ 9745.703788] hardirqs last enabled at (16931291): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9745.713954] hardirqs last disabled at (16931292): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9745.723678] softirqs last enabled at (16931248): [] __do_softirq+0x2be/0x371 [ 9745.732363] softirqs last disabled at (16931243): [] irq_exit_rcu+0xb0/0xd0 [ 9745.740885] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9745.750269] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9745.757740] Workqueue: events delayed_work [ceph] [ 9745.762464] RIP: 0010:_raw_spin_unlock+0x23/0x40 [ 9745.767074] Code: 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff <65> ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b c3 66 66 2e 0f [ 9745.785811] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [ 9745.791029] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [ 9745.798152] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [ 9745.805278] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [ 9745.812401] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9745.819525] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9745.826650] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9745.834727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9745.840465] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9745.847587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9745.854714] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9745.861838] Call Trace: [ 9745.864283] [ 9745.866293] ? watchdog_timer_fn+0x1b4/0x220 [ 9745.870556] ? __hrtimer_run_queues+0x1d3/0x360 [ 9745.875083] ? hrtimer_interrupt+0x104/0x240 [ 9745.879344] ? __local_bh_enable+0x50/0x60 [ 9745.883436] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9745.888660] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9745.893617] [ 9745.895715] [ 9745.897813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9745.903121] ? _raw_spin_unlock+0x23/0x40 [ 9745.907132] igrab+0x35/0x50 [ 9745.910018] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9745.915167] delayed_work+0x1d8/0x280 [ceph] [ 9745.919449] ? process_scheduled_works+0x22a/0x510 [ 9745.924238] process_scheduled_works+0x2a4/0x510 [ 9745.928853] worker_thread+0x188/0x340 [ 9745.932601] ? rescuer_thread+0x310/0x310 [ 9745.936605] kthread+0xf5/0x130 [ 9745.939743] ? kthread_complete_and_exit+0x20/0x20 [ 9745.944528] ret_from_fork+0x31/0x40 [ 9745.948098] ? kthread_complete_and_exit+0x20/0x20 [ 9745.952883] ret_from_fork_asm+0x11/0x20 [ 9745.956806] [ 9773.637529] watchdog: BUG: soft lockup - CPU#7 stuck for 6985s! [kworker/7:4:78808] [ 9773.645180] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9773.700631] irq event stamp: 16993106 [ 9773.704287] hardirqs last enabled at (16993105): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9773.714453] hardirqs last disabled at (16993106): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9773.724187] softirqs last enabled at (16993062): [] __do_softirq+0x2be/0x371 [ 9773.732877] softirqs last disabled at (16993057): [] irq_exit_rcu+0xb0/0xd0 [ 9773.741390] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9773.750767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9773.758240] Workqueue: events delayed_work [ceph] [ 9773.762962] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9773.767321] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9773.786058] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [ 9773.791275] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [ 9773.798400] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [ 9773.805523] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [ 9773.812648] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [ 9773.819772] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9773.826896] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9773.834974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9773.840712] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9773.847836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9773.854958] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9773.862084] Call Trace: [ 9773.864530] [ 9773.866541] ? watchdog_timer_fn+0x1b4/0x220 [ 9773.870812] ? __hrtimer_run_queues+0x1d3/0x360 [ 9773.875348] ? hrtimer_interrupt+0x104/0x240 [ 9773.879617] ? __local_bh_enable+0x50/0x60 [ 9773.883710] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9773.888934] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9773.893892] [ 9773.895987] [ 9773.898085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9773.903395] ? lock_acquire+0xf0/0x2b0 [ 9773.907145] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9773.912552] ? lock_release+0xc5/0x260 [ 9773.916297] _raw_spin_lock+0x30/0x40 [ 9773.919961] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9773.925360] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9773.930590] delayed_work+0x1d8/0x280 [ceph] [ 9773.934871] ? process_scheduled_works+0x22a/0x510 [ 9773.939659] process_scheduled_works+0x2a4/0x510 [ 9773.944276] worker_thread+0x188/0x340 [ 9773.948025] ? rescuer_thread+0x310/0x310 [ 9773.952027] kthread+0xf5/0x130 [ 9773.955167] ? kthread_complete_and_exit+0x20/0x20 [ 9773.959950] ret_from_fork+0x31/0x40 [ 9773.963529] ? kthread_complete_and_exit+0x20/0x20 [ 9773.968315] ret_from_fork_asm+0x11/0x20 [ 9773.972245] [ 9801.638023] watchdog: BUG: soft lockup - CPU#7 stuck for 7011s! [kworker/7:4:78808] [ 9801.645696] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9801.701146] irq event stamp: 17055492 [ 9801.704800] hardirqs last enabled at (17055491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9801.714967] hardirqs last disabled at (17055492): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9801.724693] softirqs last enabled at (17055478): [] __do_softirq+0x2be/0x371 [ 9801.733386] softirqs last disabled at (17055473): [] irq_exit_rcu+0xb0/0xd0 [ 9801.741905] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9801.751282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9801.758753] Workqueue: events delayed_work [ceph] [ 9801.763477] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9801.767827] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9801.786564] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9801.791781] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9801.798906] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9801.806030] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9801.813154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9801.820279] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9801.827402] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9801.835480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9801.841218] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9801.848342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9801.855465] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9801.862589] Call Trace: [ 9801.865038] [ 9801.867048] ? watchdog_timer_fn+0x1b4/0x220 [ 9801.871319] ? __hrtimer_run_queues+0x1d3/0x360 [ 9801.875846] ? hrtimer_interrupt+0x104/0x240 [ 9801.880113] ? __local_bh_enable+0x50/0x60 [ 9801.884208] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9801.889431] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9801.894388] [ 9801.896486] [ 9801.898583] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9801.903892] ? lock_acquire+0xf0/0x2b0 [ 9801.907642] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9801.913049] _raw_spin_lock+0x30/0x40 [ 9801.916707] ? igrab+0x1d/0x50 [ 9801.919756] igrab+0x1d/0x50 [ 9801.922635] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9801.927775] delayed_work+0x1d8/0x280 [ceph] [ 9801.932059] ? process_scheduled_works+0x22a/0x510 [ 9801.936846] process_scheduled_works+0x2a4/0x510 [ 9801.941461] worker_thread+0x188/0x340 [ 9801.945214] ? rescuer_thread+0x310/0x310 [ 9801.949225] kthread+0xf5/0x130 [ 9801.952370] ? kthread_complete_and_exit+0x20/0x20 [ 9801.957156] ret_from_fork+0x31/0x40 [ 9801.960735] ? kthread_complete_and_exit+0x20/0x20 [ 9801.965518] ret_from_fork_asm+0x11/0x20 [ 9801.969449] [ 9829.638518] watchdog: BUG: soft lockup - CPU#7 stuck for 7037s! [kworker/7:4:78808] [ 9829.646184] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9829.701636] irq event stamp: 17119672 [ 9829.705293] hardirqs last enabled at (17119671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9829.715457] hardirqs last disabled at (17119672): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9829.725181] softirqs last enabled at (17119664): [] __do_softirq+0x2be/0x371 [ 9829.733866] softirqs last disabled at (17119653): [] irq_exit_rcu+0xb0/0xd0 [ 9829.742377] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9829.751754] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9829.759225] Workqueue: events delayed_work [ceph] [ 9829.763941] RIP: 0010:lock_acquire+0x7c/0x2b0 [ 9829.768290] Code: c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 <85> c0 0f 85 13 01 00 00 9c 58 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 [ 9829.787029] RSP: 0018:ffffc90002117d30 EFLAGS: 00000246 [ 9829.792246] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 0000000000000000 [ 9829.799369] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f8674a0 [ 9829.806492] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [ 9829.813618] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [ 9829.820743] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9829.827866] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9829.835943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9829.841681] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9829.848804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9829.855929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9829.863054] Call Trace: [ 9829.865499] [ 9829.867509] ? watchdog_timer_fn+0x1b4/0x220 [ 9829.871773] ? __hrtimer_run_queues+0x1d3/0x360 [ 9829.876300] ? hrtimer_interrupt+0x104/0x240 [ 9829.880570] ? __local_bh_enable+0x50/0x60 [ 9829.884661] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9829.889885] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9829.894843] [ 9829.896940] [ 9829.899038] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9829.904346] ? lock_acquire+0x7c/0x2b0 [ 9829.908096] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9829.913503] ? lock_release+0xc5/0x260 [ 9829.917248] _raw_spin_lock+0x30/0x40 [ 9829.920905] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9829.926304] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [ 9829.931532] delayed_work+0x1d8/0x280 [ceph] [ 9829.935812] ? process_scheduled_works+0x22a/0x510 [ 9829.940596] process_scheduled_works+0x2a4/0x510 [ 9829.945211] worker_thread+0x188/0x340 [ 9829.948959] ? rescuer_thread+0x310/0x310 [ 9829.952962] kthread+0xf5/0x130 [ 9829.956101] ? kthread_complete_and_exit+0x20/0x20 [ 9829.960886] ret_from_fork+0x31/0x40 [ 9829.964465] ? kthread_complete_and_exit+0x20/0x20 [ 9829.969248] ret_from_fork_asm+0x11/0x20 [ 9829.973171] [ 9857.639021] watchdog: BUG: soft lockup - CPU#7 stuck for 7063s! [kworker/7:4:78808] [ 9857.646674] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9857.702124] irq event stamp: 17182422 [ 9857.705780] hardirqs last enabled at (17182421): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9857.715947] hardirqs last disabled at (17182422): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9857.725680] softirqs last enabled at (17182408): [] __do_softirq+0x2be/0x371 [ 9857.734365] softirqs last disabled at (17182403): [] irq_exit_rcu+0xb0/0xd0 [ 9857.742884] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9857.752261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9857.759733] Workqueue: events delayed_work [ceph] [ 9857.764455] RIP: 0010:ceph_put_mds_session+0x7c/0x90 [ceph] [ 9857.770036] Code: 61 e0 84 c0 75 25 48 8d bb 08 0a 00 00 e8 4c 3c 04 e1 48 89 df 5b e9 b3 2b 79 e0 c3 be 03 00 00 00 48 89 d7 5b e9 d4 d6 a8 e0 0f 0b eb d7 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f [ 9857.788775] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [ 9857.793990] RAX: ffffffffffffffff RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [ 9857.801116] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: 0000000000000000 [ 9857.808240] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 9857.815365] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 9857.822487] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [ 9857.829613] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9857.837689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9857.843426] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9857.850551] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9857.857675] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9857.864800] Call Trace: [ 9857.867245] [ 9857.869255] ? watchdog_timer_fn+0x1b4/0x220 [ 9857.873518] ? __hrtimer_run_queues+0x1d3/0x360 [ 9857.878046] ? hrtimer_interrupt+0x104/0x240 [ 9857.882314] ? __local_bh_enable+0x50/0x60 [ 9857.886407] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9857.891632] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9857.896589] [ 9857.898687] [ 9857.900784] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9857.906093] ? ceph_put_mds_session+0x7c/0x90 [ceph] [ 9857.911063] ceph_check_caps+0x275/0xae0 [ceph] [ 9857.915605] ? lock_acquire+0xcb/0x2b0 [ 9857.919349] ? igrab+0x1d/0x50 [ 9857.922400] ? find_held_lock+0x2d/0x90 [ 9857.926239] ? find_held_lock+0x2d/0x90 [ 9857.930069] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9857.935469] ? lock_release+0xc5/0x260 [ 9857.939215] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9857.944442] delayed_work+0x1d8/0x280 [ceph] [ 9857.948722] ? process_scheduled_works+0x22a/0x510 [ 9857.953512] process_scheduled_works+0x2a4/0x510 [ 9857.958127] worker_thread+0x188/0x340 [ 9857.961879] ? rescuer_thread+0x310/0x310 [ 9857.965890] kthread+0xf5/0x130 [ 9857.969027] ? kthread_complete_and_exit+0x20/0x20 [ 9857.973813] ret_from_fork+0x31/0x40 [ 9857.977391] ? kthread_complete_and_exit+0x20/0x20 [ 9857.982174] ret_from_fork_asm+0x11/0x20 [ 9857.986099] [ 9885.639511] watchdog: BUG: soft lockup - CPU#7 stuck for 7089s! [kworker/7:4:78808] [ 9885.647163] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9885.702614] irq event stamp: 17245402 [ 9885.706270] hardirqs last enabled at (17245401): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9885.716436] hardirqs last disabled at (17245402): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9885.726169] softirqs last enabled at (17245286): [] __do_softirq+0x2be/0x371 [ 9885.734860] softirqs last disabled at (17245281): [] irq_exit_rcu+0xb0/0xd0 [ 9885.743374] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9885.752750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9885.760222] Workqueue: events delayed_work [ceph] [ 9885.764944] RIP: 0010:ceph_check_delayed_caps+0xb2/0x270 [ceph] [ 9885.770872] Code: 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 48 8b 85 78 04 00 00 49 [ 9885.789609] RSP: 0018:ffffc90002117db8 EFLAGS: 00000286 [ 9885.794828] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [ 9885.801951] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [ 9885.809074] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [ 9885.816199] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [ 9885.823325] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [ 9885.830448] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9885.838524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9885.844263] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9885.851387] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9885.858511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9885.865635] Call Trace: [ 9885.868082] [ 9885.870092] ? watchdog_timer_fn+0x1b4/0x220 [ 9885.874364] ? __hrtimer_run_queues+0x1d3/0x360 [ 9885.878891] ? hrtimer_interrupt+0x104/0x240 [ 9885.883160] ? __local_bh_enable+0x50/0x60 [ 9885.887252] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9885.892477] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9885.897434] [ 9885.899531] [ 9885.901629] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9885.906937] ? ceph_check_delayed_caps+0xb2/0x270 [ceph] [ 9885.912256] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [ 9885.917568] delayed_work+0x1d8/0x280 [ceph] [ 9885.921843] ? process_scheduled_works+0x22a/0x510 [ 9885.926633] process_scheduled_works+0x2a4/0x510 [ 9885.931249] worker_thread+0x188/0x340 [ 9885.934997] ? rescuer_thread+0x310/0x310 [ 9885.938999] kthread+0xf5/0x130 [ 9885.942139] ? kthread_complete_and_exit+0x20/0x20 [ 9885.946922] ret_from_fork+0x31/0x40 [ 9885.950492] ? kthread_complete_and_exit+0x20/0x20 [ 9885.955276] ret_from_fork_asm+0x11/0x20 [ 9885.959199] [ 9913.640002] watchdog: BUG: soft lockup - CPU#7 stuck for 7115s! [kworker/7:4:78808] [ 9913.647653] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9913.703102] irq event stamp: 17308020 [ 9913.706758] hardirqs last enabled at (17308019): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9913.716925] hardirqs last disabled at (17308020): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9913.726658] softirqs last enabled at (17307976): [] __do_softirq+0x2be/0x371 [ 9913.735351] softirqs last disabled at (17307971): [] irq_exit_rcu+0xb0/0xd0 [ 9913.743872] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9913.753248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9913.760720] Workqueue: events delayed_work [ceph] [ 9913.765442] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9913.769792] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9913.788531] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [ 9913.793748] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [ 9913.800872] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [ 9913.807995] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [ 9913.815121] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [ 9913.822244] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9913.829367] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9913.837447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9913.843183] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9913.850307] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9913.857432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9913.864556] Call Trace: [ 9913.867002] [ 9913.869013] ? watchdog_timer_fn+0x1b4/0x220 [ 9913.873284] ? __hrtimer_run_queues+0x1d3/0x360 [ 9913.877811] ? hrtimer_interrupt+0x104/0x240 [ 9913.882079] ? __local_bh_enable+0x50/0x60 [ 9913.886173] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9913.891398] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9913.896355] [ 9913.898451] [ 9913.900549] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9913.905856] ? lock_acquire+0xf0/0x2b0 [ 9913.909599] ? ceph_check_caps+0x26b/0xae0 [ceph] [ 9913.914311] ? lock_release+0xc5/0x260 [ 9913.918057] _raw_spin_lock+0x30/0x40 [ 9913.921713] ? ceph_check_caps+0x63/0xae0 [ceph] [ 9913.926333] ceph_check_caps+0x63/0xae0 [ceph] [ 9913.930785] ? __lock_acquire+0xbfa/0x1e20 [ 9913.934883] ? lock_acquire+0xcb/0x2b0 [ 9913.938631] ? igrab+0x1d/0x50 [ 9913.941683] ? find_held_lock+0x2d/0x90 [ 9913.945521] ? find_held_lock+0x2d/0x90 [ 9913.949353] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9913.954752] ? lock_release+0xc5/0x260 [ 9913.958498] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9913.963724] delayed_work+0x1d8/0x280 [ceph] [ 9913.968006] ? process_scheduled_works+0x22a/0x510 [ 9913.972797] process_scheduled_works+0x2a4/0x510 [ 9913.977413] worker_thread+0x188/0x340 [ 9913.981161] ? rescuer_thread+0x310/0x310 [ 9913.985162] kthread+0xf5/0x130 [ 9913.988302] ? kthread_complete_and_exit+0x20/0x20 [ 9913.993087] ret_from_fork+0x31/0x40 [ 9913.996663] ? kthread_complete_and_exit+0x20/0x20 [ 9914.001450] ret_from_fork_asm+0x11/0x20 [ 9914.005381] [ 9941.640497] watchdog: BUG: soft lockup - CPU#7 stuck for 7141s! [kworker/7:4:78808] [ 9941.648168] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9941.703617] irq event stamp: 17369668 [ 9941.707275] hardirqs last enabled at (17369667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9941.717440] hardirqs last disabled at (17369668): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9941.727164] softirqs last enabled at (17369636): [] __do_softirq+0x2be/0x371 [ 9941.735849] softirqs last disabled at (17369631): [] irq_exit_rcu+0xb0/0xd0 [ 9941.744369] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9941.753746] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9941.761217] Workqueue: events delayed_work [ceph] [ 9941.765939] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9941.770290] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9941.789029] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9941.794247] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9941.801369] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9941.808493] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9941.815617] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9941.822743] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9941.829865] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9941.837943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9941.843682] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9941.850806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9941.857928] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9941.865054] Call Trace: [ 9941.867499] [ 9941.869512] ? watchdog_timer_fn+0x1b4/0x220 [ 9941.873782] ? __hrtimer_run_queues+0x1d3/0x360 [ 9941.878309] ? hrtimer_interrupt+0x104/0x240 [ 9941.882579] ? __local_bh_enable+0x50/0x60 [ 9941.886671] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9941.891896] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9941.896852] [ 9941.898949] [ 9941.901046] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9941.906356] ? lock_acquire+0xf0/0x2b0 [ 9941.910107] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9941.915515] _raw_spin_lock+0x30/0x40 [ 9941.919178] ? igrab+0x1d/0x50 [ 9941.922228] igrab+0x1d/0x50 [ 9941.925109] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9941.930249] delayed_work+0x1d8/0x280 [ceph] [ 9941.934530] ? process_scheduled_works+0x22a/0x510 [ 9941.939320] process_scheduled_works+0x2a4/0x510 [ 9941.943935] worker_thread+0x188/0x340 [ 9941.947685] ? rescuer_thread+0x310/0x310 [ 9941.951687] kthread+0xf5/0x130 [ 9941.954826] ? kthread_complete_and_exit+0x20/0x20 [ 9941.959609] ret_from_fork+0x31/0x40 [ 9941.963189] ? kthread_complete_and_exit+0x20/0x20 [ 9941.967972] ret_from_fork_asm+0x11/0x20 [ 9941.971896] [ 9969.641008] watchdog: BUG: soft lockup - CPU#7 stuck for 7167s! [kworker/7:4:78808] [ 9969.648656] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9969.704107] irq event stamp: 17435592 [ 9969.707763] hardirqs last enabled at (17435591): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9969.717930] hardirqs last disabled at (17435592): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9969.727663] softirqs last enabled at (17435572): [] __do_softirq+0x2be/0x371 [ 9969.736356] softirqs last disabled at (17435567): [] irq_exit_rcu+0xb0/0xd0 [ 9969.744876] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9969.754252] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9969.761723] Workqueue: events delayed_work [ceph] [ 9969.766448] RIP: 0010:lock_acquire+0xf0/0x2b0 [ 9969.770806] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [ 9969.789544] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [ 9969.794761] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [ 9969.801885] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [ 9969.809008] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [ 9969.816134] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [ 9969.823258] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 9969.830381] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9969.838460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9969.844197] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9969.851320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9969.858444] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9969.865570] Call Trace: [ 9969.868016] [ 9969.870027] ? watchdog_timer_fn+0x1b4/0x220 [ 9969.874297] ? __hrtimer_run_queues+0x1d3/0x360 [ 9969.878832] ? hrtimer_interrupt+0x104/0x240 [ 9969.883093] ? __local_bh_enable+0x50/0x60 [ 9969.887187] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9969.892410] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9969.897367] [ 9969.899466] [ 9969.901562] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9969.906869] ? lock_acquire+0xf0/0x2b0 [ 9969.910613] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9969.916019] _raw_spin_lock+0x30/0x40 [ 9969.919676] ? igrab+0x1d/0x50 [ 9969.922728] igrab+0x1d/0x50 [ 9969.925606] ceph_check_delayed_caps+0xb7/0x270 [ceph] [ 9969.930757] delayed_work+0x1d8/0x280 [ceph] [ 9969.935037] ? process_scheduled_works+0x22a/0x510 [ 9969.939826] process_scheduled_works+0x2a4/0x510 [ 9969.944441] worker_thread+0x188/0x340 [ 9969.948190] ? rescuer_thread+0x310/0x310 [ 9969.952194] kthread+0xf5/0x130 [ 9969.955332] ? kthread_complete_and_exit+0x20/0x20 [ 9969.960117] ret_from_fork+0x31/0x40 [ 9969.963696] ? kthread_complete_and_exit+0x20/0x20 [ 9969.968478] ret_from_fork_asm+0x11/0x20 [ 9969.972402] [ 9997.641487] watchdog: BUG: soft lockup - CPU#7 stuck for 7194s! [kworker/7:4:78808] [ 9997.649138] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [ 9997.704588] irq event stamp: 17498230 [ 9997.708244] hardirqs last enabled at (17498229): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9997.718409] hardirqs last disabled at (17498230): [] sysvec_apic_timer_interrupt+0xe/0x90 [ 9997.728135] softirqs last enabled at (17498210): [] __do_softirq+0x2be/0x371 [ 9997.736827] softirqs last disabled at (17498205): [] irq_exit_rcu+0xb0/0xd0 [ 9997.745341] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [ 9997.754725] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [ 9997.762195] Workqueue: events delayed_work [ceph] [ 9997.766919] RIP: 0010:__ceph_caps_issued+0x55/0x110 [ceph] [ 9997.772415] Code: 4c 8b 70 20 48 85 f6 74 06 c7 06 00 00 00 00 49 8d bd a8 06 00 00 e8 8a fc 04 e1 49 89 c7 48 85 c0 74 32 49 8d 5f f8 48 89 df c6 dc ff ff 85 c0 74 12 66 90 45 0b 67 38 48 85 ed 74 07 41 8b [ 9997.791150] RSP: 0018:ffffc90002117b98 EFLAGS: 00000282 [ 9997.796369] RAX: ffff88810da9e008 RBX: ffff88810da9e000 RCX: 0000000000000000 [ 9997.803491] RDX: 0000000000000000 RSI: ffffc90002117c64 RDI: ffff88810da9e000 [ 9997.810616] RBP: ffffc90002117c64 R08: 0000000000000000 R09: 0000000000000000 [ 9997.817741] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 9997.824865] R13: ffff8881a1d34478 R14: ffff888100ee2000 R15: ffff88810da9e008 [ 9997.831990] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [ 9997.840066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9997.845804] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [ 9997.852927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9997.860052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9997.867177] Call Trace: [ 9997.869623] [ 9997.871633] ? watchdog_timer_fn+0x1b4/0x220 [ 9997.875906] ? __hrtimer_run_queues+0x1d3/0x360 [ 9997.880431] ? hrtimer_interrupt+0x104/0x240 [ 9997.884700] ? __local_bh_enable+0x50/0x60 [ 9997.888793] ? __sysvec_apic_timer_interrupt+0x61/0x160 [ 9997.894017] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 9997.898974] [ 9997.901072] [ 9997.903169] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 9997.908478] ? __ceph_caps_issued+0x55/0x110 [ceph] [ 9997.913364] ? __ceph_caps_issued+0x46/0x110 [ceph] [ 9997.918244] ceph_check_caps+0x154/0xae0 [ceph] [ 9997.922792] ? lock_acquire+0xcb/0x2b0 [ 9997.926536] ? igrab+0x1d/0x50 [ 9997.929586] ? find_held_lock+0x2d/0x90 [ 9997.933418] ? find_held_lock+0x2d/0x90 [ 9997.937247] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [ 9997.942648] ? lock_release+0xc5/0x260 [ 9997.946394] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [ 9997.951629] delayed_work+0x1d8/0x280 [ceph] [ 9997.955909] ? process_scheduled_works+0x22a/0x510 [ 9997.960700] process_scheduled_works+0x2a4/0x510 [ 9997.965316] worker_thread+0x188/0x340 [ 9997.969065] ? rescuer_thread+0x310/0x310 [ 9997.973067] kthread+0xf5/0x130 [ 9997.976207] ? kthread_complete_and_exit+0x20/0x20 [ 9997.980989] ret_from_fork+0x31/0x40 [ 9997.984560] ? kthread_complete_and_exit+0x20/0x20 [ 9997.989344] ret_from_fork_asm+0x11/0x20 [ 9997.993268] [10025.641981] watchdog: BUG: soft lockup - CPU#7 stuck for 7220s! [kworker/7:4:78808] [10025.649654] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10025.705103] irq event stamp: 17560304 [10025.708759] hardirqs last enabled at (17560303): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10025.718926] hardirqs last disabled at (17560304): [] sysvec_apic_timer_interrupt+0xe/0x90 [10025.728658] softirqs last enabled at (17560286): [] __do_softirq+0x2be/0x371 [10025.737351] softirqs last disabled at (17560281): [] irq_exit_rcu+0xb0/0xd0 [10025.745873] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10025.755258] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10025.762728] Workqueue: events delayed_work [ceph] [10025.767451] RIP: 0010:lock_acquire+0xf0/0x2b0 [10025.771803] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10025.790538] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [10025.795755] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [10025.802880] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [10025.810004] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [10025.817130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [10025.824253] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10025.831377] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10025.839455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10025.845191] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10025.852317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10025.859441] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10025.866565] Call Trace: [10025.869011] [10025.871021] ? watchdog_timer_fn+0x1b4/0x220 [10025.875294] ? __hrtimer_run_queues+0x1d3/0x360 [10025.879829] ? hrtimer_interrupt+0x104/0x240 [10025.884098] ? __local_bh_enable+0x50/0x60 [10025.888190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10025.893416] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10025.898380] [10025.900478] [10025.902575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10025.907884] ? lock_acquire+0xf0/0x2b0 [10025.911634] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10025.917043] _raw_spin_lock+0x30/0x40 [10025.920708] ? igrab+0x1d/0x50 [10025.923758] igrab+0x1d/0x50 [10025.926636] ceph_check_delayed_caps+0xb7/0x270 [ceph] [10025.931778] delayed_work+0x1d8/0x280 [ceph] [10025.936059] ? process_scheduled_works+0x22a/0x510 [10025.940848] process_scheduled_works+0x2a4/0x510 [10025.945464] worker_thread+0x188/0x340 [10025.949214] ? rescuer_thread+0x310/0x310 [10025.953224] kthread+0xf5/0x130 [10025.956363] ? kthread_complete_and_exit+0x20/0x20 [10025.961147] ret_from_fork+0x31/0x40 [10025.964726] ? kthread_complete_and_exit+0x20/0x20 [10025.969509] ret_from_fork_asm+0x11/0x20 [10025.973433] [10053.642476] watchdog: BUG: soft lockup - CPU#7 stuck for 7246s! [kworker/7:4:78808] [10053.650134] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10053.705582] irq event stamp: 17621816 [10053.709240] hardirqs last enabled at (17621815): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10053.719406] hardirqs last disabled at (17621816): [] sysvec_apic_timer_interrupt+0xe/0x90 [10053.729138] softirqs last enabled at (17621796): [] __do_softirq+0x2be/0x371 [10053.737823] softirqs last disabled at (17621791): [] irq_exit_rcu+0xb0/0xd0 [10053.746344] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10053.755722] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10053.763190] Workqueue: events delayed_work [ceph] [10053.767914] RIP: 0010:_raw_spin_unlock+0x1e/0x40 [10053.772525] Code: 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b [10053.791263] RSP: 0018:ffffc90002117da8 EFLAGS: 00000286 [10053.796479] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: ffffc90002117d6c [10053.803604] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88817f867488 [10053.810729] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [10053.817852] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [10053.824977] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10053.832099] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10053.840177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10053.845916] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10053.853039] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10053.860164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10053.867286] Call Trace: [10053.869734] [10053.871744] ? watchdog_timer_fn+0x1b4/0x220 [10053.876009] ? __hrtimer_run_queues+0x1d3/0x360 [10053.880542] ? hrtimer_interrupt+0x104/0x240 [10053.884804] ? __local_bh_enable+0x50/0x60 [10053.888897] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10053.894121] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10053.899079] [10053.901174] [10053.903272] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10053.908580] ? _raw_spin_unlock+0x1e/0x40 [10053.912591] ? _raw_spin_unlock+0x1b/0x40 [10053.916594] ceph_check_delayed_caps+0x19b/0x270 [ceph] [10053.921825] delayed_work+0x1d8/0x280 [ceph] [10053.926104] ? process_scheduled_works+0x22a/0x510 [10053.930894] process_scheduled_works+0x2a4/0x510 [10053.935509] worker_thread+0x188/0x340 [10053.939257] ? rescuer_thread+0x310/0x310 [10053.943261] kthread+0xf5/0x130 [10053.946400] ? kthread_complete_and_exit+0x20/0x20 [10053.951185] ret_from_fork+0x31/0x40 [10053.954764] ? kthread_complete_and_exit+0x20/0x20 [10053.959546] ret_from_fork_asm+0x11/0x20 [10053.963469] [10081.642985] watchdog: BUG: soft lockup - CPU#7 stuck for 7272s! [kworker/7:4:78808] [10081.650640] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10081.706091] irq event stamp: 17683736 [10081.709747] hardirqs last enabled at (17683735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10081.719913] hardirqs last disabled at (17683736): [] sysvec_apic_timer_interrupt+0xe/0x90 [10081.729645] softirqs last enabled at (17683694): [] __do_softirq+0x2be/0x371 [10081.738338] softirqs last disabled at (17683689): [] irq_exit_rcu+0xb0/0xd0 [10081.746849] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10081.756228] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10081.763698] Workqueue: events delayed_work [ceph] [10081.768421] RIP: 0010:ceph_check_caps+0x251/0xae0 [ceph] [10081.773735] Code: e1 49 89 c7 48 85 c0 75 c7 8b 44 24 54 44 8b 7c 24 20 4c 89 eb 89 c1 48 8b 83 a8 06 00 00 48 85 c0 74 17 48 8b 93 50 07 00 00 <48> 8d 83 50 07 00 00 48 39 c2 0f 84 13 07 00 00 48 8b 7c 24 60 e8 [10081.792470] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000282 [10081.797687] RAX: ffff88810da9e008 RBX: ffff8881a1d34478 RCX: 0000000000000000 [10081.804812] RDX: ffff8881a1d34bc8 RSI: 0000000000000000 RDI: ffff88810da9e008 [10081.811937] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [10081.819060] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [10081.826186] R13: ffff8881a1d34478 R14: 0000000000000000 R15: 0000000000000000 [10081.833308] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10081.841387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10081.847123] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10081.854248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10081.861372] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10081.868495] Call Trace: [10081.870943] [10081.872953] ? watchdog_timer_fn+0x1b4/0x220 [10081.877216] ? __hrtimer_run_queues+0x1d3/0x360 [10081.881742] ? hrtimer_interrupt+0x104/0x240 [10081.886003] ? __local_bh_enable+0x50/0x60 [10081.890098] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10081.895320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10081.900277] [10081.902375] [10081.904472] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10081.909781] ? ceph_check_caps+0x251/0xae0 [ceph] [10081.914501] ? lock_acquire+0xcb/0x2b0 [10081.918245] ? igrab+0x1d/0x50 [10081.921296] ? find_held_lock+0x2d/0x90 [10081.925135] ? find_held_lock+0x2d/0x90 [10081.928965] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10081.934366] ? lock_release+0xc5/0x260 [10081.938112] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10081.943345] delayed_work+0x1d8/0x280 [ceph] [10081.947628] ? process_scheduled_works+0x22a/0x510 [10081.952416] process_scheduled_works+0x2a4/0x510 [10081.957032] worker_thread+0x188/0x340 [10081.960781] ? rescuer_thread+0x310/0x310 [10081.964784] kthread+0xf5/0x130 [10081.967923] ? kthread_complete_and_exit+0x20/0x20 [10081.972706] ret_from_fork+0x31/0x40 [10081.976286] ? kthread_complete_and_exit+0x20/0x20 [10081.981071] ret_from_fork_asm+0x11/0x20 [10081.985003] [10109.643466] watchdog: BUG: soft lockup - CPU#7 stuck for 7298s! [kworker/7:4:78808] [10109.651119] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10109.706570] irq event stamp: 17747382 [10109.710226] hardirqs last enabled at (17747381): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10109.720392] hardirqs last disabled at (17747382): [] sysvec_apic_timer_interrupt+0xe/0x90 [10109.730118] softirqs last enabled at (17747256): [] __do_softirq+0x2be/0x371 [10109.738811] softirqs last disabled at (17747251): [] irq_exit_rcu+0xb0/0xd0 [10109.747333] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10109.756716] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10109.764186] Workqueue: events delayed_work [ceph] [10109.768911] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [10109.773520] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [10109.792259] RSP: 0018:ffffc90002117bc0 EFLAGS: 00000246 [10109.797476] RAX: 0000000000000000 RBX: ffff8881a1d34920 RCX: 00000000d3c6b569 [10109.804600] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34920 [10109.811725] RBP: 0000000000000000 R08: 00000000000079a2 R09: 0000000000000002 [10109.818849] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [10109.825971] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10109.833095] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10109.841175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10109.846911] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10109.854035] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10109.861160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10109.868285] Call Trace: [10109.870729] [10109.872742] ? watchdog_timer_fn+0x1b4/0x220 [10109.877011] ? __hrtimer_run_queues+0x1d3/0x360 [10109.881538] ? hrtimer_interrupt+0x104/0x240 [10109.885800] ? __local_bh_enable+0x50/0x60 [10109.889891] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10109.895117] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10109.900074] [10109.902172] [10109.904269] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10109.909576] ? do_raw_spin_lock+0x42/0xc0 [10109.913588] ceph_check_caps+0x63/0xae0 [ceph] [10109.918046] ? __lock_acquire+0xbfa/0x1e20 [10109.922143] ? lock_acquire+0xcb/0x2b0 [10109.925892] ? igrab+0x1d/0x50 [10109.928943] ? find_held_lock+0x2d/0x90 [10109.932774] ? find_held_lock+0x2d/0x90 [10109.936604] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10109.942004] ? lock_release+0xc5/0x260 [10109.945750] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10109.950978] delayed_work+0x1d8/0x280 [ceph] [10109.955257] ? process_scheduled_works+0x22a/0x510 [10109.960048] process_scheduled_works+0x2a4/0x510 [10109.964665] worker_thread+0x188/0x340 [10109.968414] ? rescuer_thread+0x310/0x310 [10109.972423] kthread+0xf5/0x130 [10109.975564] ? kthread_complete_and_exit+0x20/0x20 [10109.980346] ret_from_fork+0x31/0x40 [10109.983926] ? kthread_complete_and_exit+0x20/0x20 [10109.988709] ret_from_fork_asm+0x11/0x20 [10109.992632] [10137.643962] watchdog: BUG: soft lockup - CPU#7 stuck for 7324s! [kworker/7:4:78808] [10137.651626] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10137.707076] irq event stamp: 17810594 [10137.710734] hardirqs last enabled at (17810593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10137.720901] hardirqs last disabled at (17810594): [] sysvec_apic_timer_interrupt+0xe/0x90 [10137.730633] softirqs last enabled at (17810578): [] __do_softirq+0x2be/0x371 [10137.739333] softirqs last disabled at (17810565): [] irq_exit_rcu+0xb0/0xd0 [10137.747845] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10137.757224] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10137.764693] Workqueue: events delayed_work [ceph] [10137.769416] RIP: 0010:lock_acquire+0xf0/0x2b0 [10137.773767] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10137.792505] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10137.797722] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10137.804848] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10137.811971] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10137.819095] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10137.826219] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10137.833344] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10137.841421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10137.847158] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10137.854283] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10137.861405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10137.868531] Call Trace: [10137.870977] [10137.872986] ? watchdog_timer_fn+0x1b4/0x220 [10137.877250] ? __hrtimer_run_queues+0x1d3/0x360 [10137.881777] ? hrtimer_interrupt+0x104/0x240 [10137.886047] ? __local_bh_enable+0x50/0x60 [10137.890139] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10137.895364] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10137.900321] [10137.902419] [10137.904515] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10137.909824] ? lock_acquire+0xf0/0x2b0 [10137.913572] ? ceph_check_caps+0x26b/0xae0 [ceph] [10137.918280] ? lock_release+0xc5/0x260 [10137.922024] _raw_spin_lock+0x30/0x40 [10137.925680] ? ceph_check_caps+0x63/0xae0 [ceph] [10137.930299] ceph_check_caps+0x63/0xae0 [ceph] [10137.934750] ? __lock_acquire+0xbfa/0x1e20 [10137.938848] ? lock_acquire+0xcb/0x2b0 [10137.942596] ? igrab+0x1d/0x50 [10137.945648] ? find_held_lock+0x2d/0x90 [10137.949479] ? find_held_lock+0x2d/0x90 [10137.953310] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10137.958710] ? lock_release+0xc5/0x260 [10137.962454] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10137.967683] delayed_work+0x1d8/0x280 [ceph] [10137.971962] ? process_scheduled_works+0x22a/0x510 [10137.976753] process_scheduled_works+0x2a4/0x510 [10137.981369] worker_thread+0x188/0x340 [10137.985117] ? rescuer_thread+0x310/0x310 [10137.989121] kthread+0xf5/0x130 [10137.992260] ? kthread_complete_and_exit+0x20/0x20 [10137.997042] ret_from_fork+0x31/0x40 [10138.000613] ? kthread_complete_and_exit+0x20/0x20 [10138.005397] ret_from_fork_asm+0x11/0x20 [10138.009322] [10165.644456] watchdog: BUG: soft lockup - CPU#7 stuck for 7350s! [kworker/7:4:78808] [10165.652132] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10165.707583] irq event stamp: 17872340 [10165.711241] hardirqs last enabled at (17872339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10165.721406] hardirqs last disabled at (17872340): [] sysvec_apic_timer_interrupt+0xe/0x90 [10165.731130] softirqs last enabled at (17872262): [] __do_softirq+0x2be/0x371 [10165.739824] softirqs last disabled at (17872257): [] irq_exit_rcu+0xb0/0xd0 [10165.748345] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10165.757720] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10165.765192] Workqueue: events delayed_work [ceph] [10165.769914] RIP: 0010:lock_acquire+0xf0/0x2b0 [10165.774265] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10165.793003] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10165.798219] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10165.805343] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10165.812469] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10165.819592] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10165.826717] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10165.833842] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10165.841919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10165.847655] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10165.854781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10165.861904] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10165.869027] Call Trace: [10165.871476] [10165.873485] ? watchdog_timer_fn+0x1b4/0x220 [10165.877748] ? __hrtimer_run_queues+0x1d3/0x360 [10165.882273] ? hrtimer_interrupt+0x104/0x240 [10165.886536] ? __local_bh_enable+0x50/0x60 [10165.890629] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10165.895852] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10165.900809] [10165.902906] [10165.905004] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10165.910313] ? lock_acquire+0xf0/0x2b0 [10165.914063] ? ceph_check_caps+0x26b/0xae0 [ceph] [10165.918778] ? lock_release+0xc5/0x260 [10165.922522] _raw_spin_lock+0x30/0x40 [10165.926187] ? ceph_check_caps+0x63/0xae0 [ceph] [10165.930806] ceph_check_caps+0x63/0xae0 [ceph] [10165.935257] ? __lock_acquire+0xbfa/0x1e20 [10165.939355] ? lock_acquire+0xcb/0x2b0 [10165.943104] ? igrab+0x1d/0x50 [10165.946155] ? find_held_lock+0x2d/0x90 [10165.949995] ? find_held_lock+0x2d/0x90 [10165.953825] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10165.959225] ? lock_release+0xc5/0x260 [10165.962971] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10165.968197] delayed_work+0x1d8/0x280 [ceph] [10165.972478] ? process_scheduled_works+0x22a/0x510 [10165.977267] process_scheduled_works+0x2a4/0x510 [10165.981884] worker_thread+0x188/0x340 [10165.985635] ? rescuer_thread+0x310/0x310 [10165.989644] kthread+0xf5/0x130 [10165.992784] ? kthread_complete_and_exit+0x20/0x20 [10165.997567] ret_from_fork+0x31/0x40 [10166.001147] ? kthread_complete_and_exit+0x20/0x20 [10166.005932] ret_from_fork_asm+0x11/0x20 [10166.009863] [10193.644951] watchdog: BUG: soft lockup - CPU#7 stuck for 7376s! [kworker/7:4:78808] [10193.652604] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10193.708074] irq event stamp: 17934438 [10193.711737] hardirqs last enabled at (17934437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10193.721905] hardirqs last disabled at (17934438): [] sysvec_apic_timer_interrupt+0xe/0x90 [10193.731636] softirqs last enabled at (17934360): [] __do_softirq+0x2be/0x371 [10193.740322] softirqs last disabled at (17934355): [] irq_exit_rcu+0xb0/0xd0 [10193.748858] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10193.758236] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10193.765708] Workqueue: events delayed_work [ceph] [10193.770430] RIP: 0010:lock_release+0x14e/0x260 [10193.774869] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [10193.793604] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [10193.798839] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [10193.805963] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [10193.813088] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [10193.820211] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [10193.827347] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [10193.834469] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10193.842547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10193.848284] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10193.855408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10193.862532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10193.869656] Call Trace: [10193.872101] [10193.874112] ? watchdog_timer_fn+0x1b4/0x220 [10193.878377] ? __hrtimer_run_queues+0x1d3/0x360 [10193.882912] ? hrtimer_interrupt+0x104/0x240 [10193.887182] ? __local_bh_enable+0x50/0x60 [10193.891273] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10193.896498] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10193.901456] [10193.903553] [10193.905649] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10193.910955] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10193.916364] ? lock_release+0x14e/0x260 [10193.920196] _raw_spin_unlock+0x1b/0x40 [10193.924034] ceph_check_delayed_caps+0x19b/0x270 [ceph] [10193.929261] delayed_work+0x1d8/0x280 [ceph] [10193.933543] ? process_scheduled_works+0x22a/0x510 [10193.938348] process_scheduled_works+0x2a4/0x510 [10193.942965] worker_thread+0x188/0x340 [10193.946716] ? rescuer_thread+0x310/0x310 [10193.950727] kthread+0xf5/0x130 [10193.953872] ? kthread_complete_and_exit+0x20/0x20 [10193.958657] ret_from_fork+0x31/0x40 [10193.962236] ? kthread_complete_and_exit+0x20/0x20 [10193.967021] ret_from_fork_asm+0x11/0x20 [10193.970952] [10221.645446] watchdog: BUG: soft lockup - CPU#7 stuck for 7402s! [kworker/7:4:78808] [10221.653096] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10221.708546] irq event stamp: 17996074 [10221.712202] hardirqs last enabled at (17996073): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10221.722368] hardirqs last disabled at (17996074): [] sysvec_apic_timer_interrupt+0xe/0x90 [10221.732101] softirqs last enabled at (17996042): [] __do_softirq+0x2be/0x371 [10221.740793] softirqs last disabled at (17996037): [] irq_exit_rcu+0xb0/0xd0 [10221.749314] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10221.758690] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10221.766161] Workqueue: events delayed_work [ceph] [10221.770885] RIP: 0010:lock_acquire+0xf0/0x2b0 [10221.775244] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10221.793982] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10221.799197] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10221.806322] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10221.813448] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10221.820570] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10221.827695] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10221.834819] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10221.842897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10221.848635] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10221.855758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10221.862882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10221.870006] Call Trace: [10221.872451] [10221.874462] ? watchdog_timer_fn+0x1b4/0x220 [10221.878727] ? __hrtimer_run_queues+0x1d3/0x360 [10221.883252] ? hrtimer_interrupt+0x104/0x240 [10221.887515] ? __local_bh_enable+0x50/0x60 [10221.891608] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10221.896843] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10221.901804] [10221.903903] [10221.906001] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10221.911307] ? lock_acquire+0xf0/0x2b0 [10221.915049] ? ceph_check_caps+0x26b/0xae0 [ceph] [10221.919764] ? lock_release+0xc5/0x260 [10221.923508] _raw_spin_lock+0x30/0x40 [10221.927165] ? ceph_check_caps+0x63/0xae0 [ceph] [10221.931784] ceph_check_caps+0x63/0xae0 [ceph] [10221.936235] ? __lock_acquire+0xbfa/0x1e20 [10221.940334] ? lock_acquire+0xcb/0x2b0 [10221.944083] ? igrab+0x1d/0x50 [10221.947133] ? find_held_lock+0x2d/0x90 [10221.950964] ? find_held_lock+0x2d/0x90 [10221.954795] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10221.960195] ? lock_release+0xc5/0x260 [10221.963942] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10221.969175] delayed_work+0x1d8/0x280 [ceph] [10221.973456] ? process_scheduled_works+0x22a/0x510 [10221.978247] process_scheduled_works+0x2a4/0x510 [10221.982862] worker_thread+0x188/0x340 [10221.986612] ? rescuer_thread+0x310/0x310 [10221.990614] kthread+0xf5/0x130 [10221.993753] ? kthread_complete_and_exit+0x20/0x20 [10221.998536] ret_from_fork+0x31/0x40 [10222.002118] ? kthread_complete_and_exit+0x20/0x20 [10222.006899] ret_from_fork_asm+0x11/0x20 [10222.010822] [10249.645940] watchdog: BUG: soft lockup - CPU#7 stuck for 7428s! [kworker/7:4:78808] [10249.653592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10249.709043] irq event stamp: 18057888 [10249.712699] hardirqs last enabled at (18057887): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10249.722866] hardirqs last disabled at (18057888): [] sysvec_apic_timer_interrupt+0xe/0x90 [10249.732597] softirqs last enabled at (18057810): [] __do_softirq+0x2be/0x371 [10249.741282] softirqs last disabled at (18057805): [] irq_exit_rcu+0xb0/0xd0 [10249.749804] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10249.759180] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10249.766651] Workqueue: events delayed_work [ceph] [10249.771372] RIP: 0010:ceph_check_delayed_caps+0x8a/0x270 [ceph] [10249.777303] Code: 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 [10249.796038] RSP: 0018:ffffc90002117db8 EFLAGS: 00000216 [10249.801258] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 000000010097cc80 [10249.808381] RDX: 00000001001ae57b RSI: 0000000000000000 RDI: ffff8881a1d34bc8 [10249.815506] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [10249.822629] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [10249.829752] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10249.836878] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10249.844954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10249.850691] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10249.857817] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10249.864941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10249.872064] Call Trace: [10249.874510] [10249.876522] ? watchdog_timer_fn+0x1b4/0x220 [10249.880794] ? __hrtimer_run_queues+0x1d3/0x360 [10249.885319] ? hrtimer_interrupt+0x104/0x240 [10249.889580] ? __local_bh_enable+0x50/0x60 [10249.893673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10249.898899] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10249.903854] [10249.905952] [10249.908051] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10249.913358] ? ceph_check_delayed_caps+0x8a/0x270 [ceph] [10249.918675] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10249.924084] delayed_work+0x1d8/0x280 [ceph] [10249.928369] ? process_scheduled_works+0x22a/0x510 [10249.933156] process_scheduled_works+0x2a4/0x510 [10249.937773] worker_thread+0x188/0x340 [10249.941522] ? rescuer_thread+0x310/0x310 [10249.945526] kthread+0xf5/0x130 [10249.948664] ? kthread_complete_and_exit+0x20/0x20 [10249.953446] ret_from_fork+0x31/0x40 [10249.957017] ? kthread_complete_and_exit+0x20/0x20 [10249.961801] ret_from_fork_asm+0x11/0x20 [10249.965725] [10277.646437] watchdog: BUG: soft lockup - CPU#7 stuck for 7454s! [kworker/7:4:78808] [10277.654092] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10277.709540] irq event stamp: 18120484 [10277.713197] hardirqs last enabled at (18120483): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10277.723364] hardirqs last disabled at (18120484): [] sysvec_apic_timer_interrupt+0xe/0x90 [10277.733096] softirqs last enabled at (18120440): [] __do_softirq+0x2be/0x371 [10277.741788] softirqs last disabled at (18120435): [] irq_exit_rcu+0xb0/0xd0 [10277.750301] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10277.759679] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10277.767150] Workqueue: events delayed_work [ceph] [10277.771871] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [10277.776484] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [10277.795220] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [10277.800438] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [10277.807562] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [10277.814686] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [10277.821811] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [10277.828934] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10277.836057] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10277.844136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10277.849874] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10277.856998] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10277.864120] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10277.871245] Call Trace: [10277.873692] [10277.875703] ? watchdog_timer_fn+0x1b4/0x220 [10277.879975] ? __hrtimer_run_queues+0x1d3/0x360 [10277.884507] ? hrtimer_interrupt+0x104/0x240 [10277.888769] ? __local_bh_enable+0x50/0x60 [10277.892863] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10277.898086] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10277.903044] [10277.905141] [10277.907239] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10277.912548] ? do_raw_spin_lock+0x42/0xc0 [10277.916556] igrab+0x1d/0x50 [10277.919435] ceph_check_delayed_caps+0xb7/0x270 [ceph] [10277.924584] delayed_work+0x1d8/0x280 [ceph] [10277.928864] ? process_scheduled_works+0x22a/0x510 [10277.933655] process_scheduled_works+0x2a4/0x510 [10277.938272] worker_thread+0x188/0x340 [10277.942022] ? rescuer_thread+0x310/0x310 [10277.946033] kthread+0xf5/0x130 [10277.949180] ? kthread_complete_and_exit+0x20/0x20 [10277.953962] ret_from_fork+0x31/0x40 [10277.957543] ? kthread_complete_and_exit+0x20/0x20 [10277.962325] ret_from_fork_asm+0x11/0x20 [10277.966249] [10305.646930] watchdog: BUG: soft lockup - CPU#7 stuck for 7480s! [kworker/7:4:78808] [10305.654580] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10305.710029] irq event stamp: 18182878 [10305.713686] hardirqs last enabled at (18182877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10305.723853] hardirqs last disabled at (18182878): [] sysvec_apic_timer_interrupt+0xe/0x90 [10305.733585] softirqs last enabled at (18182834): [] __do_softirq+0x2be/0x371 [10305.742278] softirqs last disabled at (18182829): [] irq_exit_rcu+0xb0/0xd0 [10305.750799] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10305.760175] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10305.767647] Workqueue: events delayed_work [ceph] [10305.772368] RIP: 0010:lock_release+0x78/0x260 [10305.776720] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [10305.795458] RSP: 0018:ffffc90002117d60 EFLAGS: 00000246 [10305.800675] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [10305.807798] RDX: 0000000000000003 RSI: 0000000000000000 RDI: 0000000000000001 [10305.814924] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [10305.822048] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [10305.829172] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10305.836295] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10305.844372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10305.850110] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10305.857234] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10305.864358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10305.871482] Call Trace: [10305.873929] [10305.875941] ? watchdog_timer_fn+0x1b4/0x220 [10305.880211] ? __hrtimer_run_queues+0x1d3/0x360 [10305.884738] ? hrtimer_interrupt+0x104/0x240 [10305.888998] ? __local_bh_enable+0x50/0x60 [10305.893091] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10305.898328] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10305.903292] [10305.905387] [10305.907486] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10305.912790] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10305.918200] ? lock_release+0x78/0x260 [10305.921945] _raw_spin_unlock+0x1b/0x40 [10305.925783] ceph_check_delayed_caps+0x19b/0x270 [ceph] [10305.931009] delayed_work+0x1d8/0x280 [ceph] [10305.935292] ? process_scheduled_works+0x22a/0x510 [10305.940082] process_scheduled_works+0x2a4/0x510 [10305.944697] worker_thread+0x188/0x340 [10305.948447] ? rescuer_thread+0x310/0x310 [10305.952449] kthread+0xf5/0x130 [10305.955587] ? kthread_complete_and_exit+0x20/0x20 [10305.960370] ret_from_fork+0x31/0x40 [10305.963950] ? kthread_complete_and_exit+0x20/0x20 [10305.968734] ret_from_fork_asm+0x11/0x20 [10305.972658] [10333.647425] watchdog: BUG: soft lockup - CPU#7 stuck for 7506s! [kworker/7:4:78808] [10333.655077] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10333.710527] irq event stamp: 18244676 [10333.714183] hardirqs last enabled at (18244675): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10333.724350] hardirqs last disabled at (18244676): [] sysvec_apic_timer_interrupt+0xe/0x90 [10333.734075] softirqs last enabled at (18244644): [] __do_softirq+0x2be/0x371 [10333.742767] softirqs last disabled at (18244639): [] irq_exit_rcu+0xb0/0xd0 [10333.751279] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10333.760656] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10333.768126] Workqueue: events delayed_work [ceph] [10333.772850] RIP: 0010:_raw_spin_unlock+0x2a/0x40 [10333.777462] Code: f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 47 7e <74> 02 5b c3 0f 1f 44 00 00 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 [10333.796198] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [10333.801415] RAX: ffffffffffffffff RBX: ffff88817f867488 RCX: ffffc90002117d6c [10333.808541] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88817f867488 [10333.815664] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [10333.822787] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [10333.829912] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10333.837036] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10333.845114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10333.850852] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10333.857974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10333.865099] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10333.872223] Call Trace: [10333.874669] [10333.876679] ? watchdog_timer_fn+0x1b4/0x220 [10333.880944] ? __hrtimer_run_queues+0x1d3/0x360 [10333.885469] ? hrtimer_interrupt+0x104/0x240 [10333.889731] ? __local_bh_enable+0x50/0x60 [10333.893823] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10333.899048] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10333.904006] [10333.906103] [10333.908200] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10333.913506] ? _raw_spin_unlock+0x2a/0x40 [10333.917510] ? _raw_spin_unlock+0x23/0x40 [10333.921522] ceph_check_delayed_caps+0x19b/0x270 [ceph] [10333.926758] delayed_work+0x1d8/0x280 [ceph] [10333.931040] ? process_scheduled_works+0x22a/0x510 [10333.935829] process_scheduled_works+0x2a4/0x510 [10333.940444] worker_thread+0x188/0x340 [10333.944193] ? rescuer_thread+0x310/0x310 [10333.948198] kthread+0xf5/0x130 [10333.951336] ? kthread_complete_and_exit+0x20/0x20 [10333.956118] ret_from_fork+0x31/0x40 [10333.959689] ? kthread_complete_and_exit+0x20/0x20 [10333.964474] ret_from_fork_asm+0x11/0x20 [10333.968406] [10361.647920] watchdog: BUG: soft lockup - CPU#7 stuck for 7533s! [kworker/7:4:78808] [10361.655575] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10361.711025] irq event stamp: 18306194 [10361.714682] hardirqs last enabled at (18306193): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10361.724849] hardirqs last disabled at (18306194): [] sysvec_apic_timer_interrupt+0xe/0x90 [10361.734581] softirqs last enabled at (18306152): [] __do_softirq+0x2be/0x371 [10361.743273] softirqs last disabled at (18306147): [] irq_exit_rcu+0xb0/0xd0 [10361.751786] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10361.761164] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10361.768633] Workqueue: events delayed_work [ceph] [10361.773357] RIP: 0010:lock_acquire+0xf0/0x2b0 [10361.777716] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10361.796453] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [10361.801670] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [10361.808795] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [10361.815919] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [10361.823044] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [10361.830166] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10361.837291] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10361.845369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10361.851107] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10361.858230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10361.865356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10361.872478] Call Trace: [10361.874925] [10361.876936] ? watchdog_timer_fn+0x1b4/0x220 [10361.881208] ? __hrtimer_run_queues+0x1d3/0x360 [10361.885742] ? hrtimer_interrupt+0x104/0x240 [10361.890004] ? __local_bh_enable+0x50/0x60 [10361.894096] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10361.899320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10361.904277] [10361.906375] [10361.908474] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10361.913780] ? lock_acquire+0xf0/0x2b0 [10361.917532] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10361.922938] _raw_spin_lock+0x30/0x40 [10361.926595] ? igrab+0x1d/0x50 [10361.929647] igrab+0x1d/0x50 [10361.932523] ceph_check_delayed_caps+0xb7/0x270 [ceph] [10361.937665] delayed_work+0x1d8/0x280 [ceph] [10361.941946] ? process_scheduled_works+0x22a/0x510 [10361.946737] process_scheduled_works+0x2a4/0x510 [10361.951352] worker_thread+0x188/0x340 [10361.955100] ? rescuer_thread+0x310/0x310 [10361.959103] kthread+0xf5/0x130 [10361.962243] ? kthread_complete_and_exit+0x20/0x20 [10361.967025] ret_from_fork+0x31/0x40 [10361.970595] ? kthread_complete_and_exit+0x20/0x20 [10361.975380] ret_from_fork_asm+0x11/0x20 [10361.979304] [10389.648415] watchdog: BUG: soft lockup - CPU#7 stuck for 7559s! [kworker/7:4:78808] [10389.656081] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10389.711531] irq event stamp: 18368394 [10389.715187] hardirqs last enabled at (18368393): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10389.725355] hardirqs last disabled at (18368394): [] sysvec_apic_timer_interrupt+0xe/0x90 [10389.735087] softirqs last enabled at (18368374): [] __do_softirq+0x2be/0x371 [10389.743780] softirqs last disabled at (18368369): [] irq_exit_rcu+0xb0/0xd0 [10389.752292] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10389.761669] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10389.769140] Workqueue: events delayed_work [ceph] [10389.773864] RIP: 0010:lock_acquire+0xf0/0x2b0 [10389.778214] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10389.796951] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10389.802169] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10389.809292] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10389.816418] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10389.823541] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10389.830664] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10389.837789] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10389.845867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10389.851605] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10389.858728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10389.865852] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10389.872977] Call Trace: [10389.875424] [10389.877434] ? watchdog_timer_fn+0x1b4/0x220 [10389.881706] ? __hrtimer_run_queues+0x1d3/0x360 [10389.886239] ? hrtimer_interrupt+0x104/0x240 [10389.890501] ? __local_bh_enable+0x50/0x60 [10389.894593] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10389.899817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10389.904777] [10389.906872] [10389.908972] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10389.914278] ? lock_acquire+0xf0/0x2b0 [10389.918028] ? ceph_check_caps+0x26b/0xae0 [ceph] [10389.922742] ? lock_release+0xc5/0x260 [10389.926487] _raw_spin_lock+0x30/0x40 [10389.930152] ? ceph_check_caps+0x63/0xae0 [ceph] [10389.934771] ceph_check_caps+0x63/0xae0 [ceph] [10389.939223] ? __lock_acquire+0xbfa/0x1e20 [10389.943321] ? lock_acquire+0xcb/0x2b0 [10389.947070] ? igrab+0x1d/0x50 [10389.950120] ? find_held_lock+0x2d/0x90 [10389.953951] ? find_held_lock+0x2d/0x90 [10389.957781] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10389.963182] ? lock_release+0xc5/0x260 [10389.966928] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10389.972154] delayed_work+0x1d8/0x280 [ceph] [10389.976436] ? process_scheduled_works+0x22a/0x510 [10389.981225] process_scheduled_works+0x2a4/0x510 [10389.985841] worker_thread+0x188/0x340 [10389.989592] ? rescuer_thread+0x310/0x310 [10389.993603] kthread+0xf5/0x130 [10389.996749] ? kthread_complete_and_exit+0x20/0x20 [10390.001533] ret_from_fork+0x31/0x40 [10390.005113] ? kthread_complete_and_exit+0x20/0x20 [10390.009894] ret_from_fork_asm+0x11/0x20 [10390.013819] [10417.648910] watchdog: BUG: soft lockup - CPU#7 stuck for 7585s! [kworker/7:4:78808] [10417.656563] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10417.712011] irq event stamp: 18432344 [10417.715668] hardirqs last enabled at (18432343): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10417.725836] hardirqs last disabled at (18432344): [] sysvec_apic_timer_interrupt+0xe/0x90 [10417.735568] softirqs last enabled at (18432266): [] __do_softirq+0x2be/0x371 [10417.744261] softirqs last disabled at (18432261): [] irq_exit_rcu+0xb0/0xd0 [10417.752783] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10417.762159] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10417.769630] Workqueue: events delayed_work [ceph] [10417.774352] RIP: 0010:rb_first+0x0/0x20 [10417.778183] Code: ff d2 48 8b 4d 10 48 89 e8 48 8b 14 24 e9 08 ff ff ff 48 c7 03 01 00 00 00 e9 52 ff ff ff 48 c7 07 01 00 00 00 c3 0f 1f 40 00 0f 1e fa 48 8b 07 48 85 c0 75 05 eb 0d 48 89 d0 48 8b 50 10 48 [10417.796921] RSP: 0018:ffffc90002117b90 EFLAGS: 00000282 [10417.802137] RAX: ffff888104426000 RBX: ffff8881a1d34478 RCX: 0000000000000000 [10417.809262] RDX: 0000000000000000 RSI: ffffc90002117c64 RDI: ffff8881a1d34b20 [10417.816387] RBP: ffffc90002117c64 R08: 0000000000000000 R09: 0000000000000000 [10417.823510] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [10417.830635] R13: ffff8881a1d34478 R14: ffff888100ee2000 R15: ffff888100ee2000 [10417.837758] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10417.845837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10417.851574] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10417.858699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10417.865821] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10417.872945] Call Trace: [10417.875392] [10417.877402] ? watchdog_timer_fn+0x1b4/0x220 [10417.881666] ? __hrtimer_run_queues+0x1d3/0x360 [10417.886193] ? hrtimer_interrupt+0x104/0x240 [10417.890461] ? __local_bh_enable+0x50/0x60 [10417.894555] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10417.899779] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10417.904736] [10417.906833] [10417.908932] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10417.914239] ? __rb_insert_augmented+0x1c0/0x1c0 [10417.918856] __ceph_caps_issued+0x46/0x110 [ceph] [10417.923572] ceph_check_caps+0x154/0xae0 [ceph] [10417.928120] ? lock_acquire+0xcb/0x2b0 [10417.931864] ? igrab+0x1d/0x50 [10417.934915] ? find_held_lock+0x2d/0x90 [10417.938753] ? find_held_lock+0x2d/0x90 [10417.942585] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10417.947985] ? lock_release+0xc5/0x260 [10417.951731] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10417.956964] delayed_work+0x1d8/0x280 [ceph] [10417.961247] ? process_scheduled_works+0x22a/0x510 [10417.966037] process_scheduled_works+0x2a4/0x510 [10417.970652] worker_thread+0x188/0x340 [10417.974402] ? rescuer_thread+0x310/0x310 [10417.978413] kthread+0xf5/0x130 [10417.981550] ? kthread_complete_and_exit+0x20/0x20 [10417.986335] ret_from_fork+0x31/0x40 [10417.989913] ? kthread_complete_and_exit+0x20/0x20 [10417.994697] ret_from_fork_asm+0x11/0x20 [10417.998620] [10445.649405] watchdog: BUG: soft lockup - CPU#7 stuck for 7611s! [kworker/7:4:78808] [10445.657061] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10445.712511] irq event stamp: 18494306 [10445.716166] hardirqs last enabled at (18494305): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10445.726333] hardirqs last disabled at (18494306): [] sysvec_apic_timer_interrupt+0xe/0x90 [10445.736057] softirqs last enabled at (18494196): [] __do_softirq+0x2be/0x371 [10445.744740] softirqs last disabled at (18494191): [] irq_exit_rcu+0xb0/0xd0 [10445.753253] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10445.762630] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10445.770101] Workqueue: events delayed_work [ceph] [10445.774825] RIP: 0010:lock_acquire+0xf0/0x2b0 [10445.779177] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10445.797912] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10445.803130] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10445.810255] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10445.817378] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10445.824503] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10445.831625] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10445.838750] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10445.846829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10445.852566] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10445.859689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10445.866814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10445.873937] Call Trace: [10445.876384] [10445.878396] ? watchdog_timer_fn+0x1b4/0x220 [10445.882665] ? __hrtimer_run_queues+0x1d3/0x360 [10445.887192] ? hrtimer_interrupt+0x104/0x240 [10445.891454] ? __local_bh_enable+0x50/0x60 [10445.895547] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10445.900770] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10445.905728] [10445.907826] [10445.909924] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10445.915230] ? lock_acquire+0xf0/0x2b0 [10445.918981] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10445.924389] ? lock_release+0xc5/0x260 [10445.928134] _raw_spin_lock+0x30/0x40 [10445.931798] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10445.937196] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10445.942427] delayed_work+0x1d8/0x280 [ceph] [10445.946708] ? process_scheduled_works+0x22a/0x510 [10445.951498] process_scheduled_works+0x2a4/0x510 [10445.956114] worker_thread+0x188/0x340 [10445.959861] ? rescuer_thread+0x310/0x310 [10445.963865] kthread+0xf5/0x130 [10445.967003] ? kthread_complete_and_exit+0x20/0x20 [10445.971786] ret_from_fork+0x31/0x40 [10445.975367] ? kthread_complete_and_exit+0x20/0x20 [10445.980149] ret_from_fork_asm+0x11/0x20 [10445.984073] [10473.649901] watchdog: BUG: soft lockup - CPU#7 stuck for 7637s! [kworker/7:4:78808] [10473.657550] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10473.713000] irq event stamp: 18555944 [10473.716656] hardirqs last enabled at (18555943): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10473.726823] hardirqs last disabled at (18555944): [] sysvec_apic_timer_interrupt+0xe/0x90 [10473.736556] softirqs last enabled at (18555930): [] __do_softirq+0x2be/0x371 [10473.745248] softirqs last disabled at (18555925): [] irq_exit_rcu+0xb0/0xd0 [10473.753770] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10473.763145] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10473.770616] Workqueue: events delayed_work [ceph] [10473.775340] RIP: 0010:_atomic_dec_and_lock+0x14/0x40 [10473.780306] Code: 63 c9 48 83 f9 07 76 ba b8 08 00 00 00 eb d2 44 89 d0 5b c3 66 90 f3 0f 1e fa 55 53 8b 07 83 f8 01 74 0e 8d 50 ff f0 0f b1 17 <75> f2 31 c0 5b 5d c3 48 89 fb 48 89 f7 48 89 f5 e8 07 2b 04 00 f0 [10473.799043] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [10473.804261] RAX: 0000000000000002 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [10473.811385] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [10473.818508] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [10473.825631] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [10473.832756] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10473.839881] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10473.847959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10473.853697] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10473.860820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10473.867943] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10473.875068] Call Trace: [10473.877513] [10473.879527] ? watchdog_timer_fn+0x1b4/0x220 [10473.883796] ? __hrtimer_run_queues+0x1d3/0x360 [10473.888323] ? hrtimer_interrupt+0x104/0x240 [10473.892584] ? __local_bh_enable+0x50/0x60 [10473.896676] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10473.901902] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10473.906858] [10473.908955] [10473.911053] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10473.916360] ? _atomic_dec_and_lock+0x14/0x40 [10473.920717] iput+0x50/0x220 [10473.923604] ceph_check_delayed_caps+0x1be/0x270 [ceph] [10473.928840] delayed_work+0x1d8/0x280 [ceph] [10473.933123] ? process_scheduled_works+0x22a/0x510 [10473.937911] process_scheduled_works+0x2a4/0x510 [10473.942529] worker_thread+0x188/0x340 [10473.946278] ? rescuer_thread+0x310/0x310 [10473.950287] kthread+0xf5/0x130 [10473.953427] ? kthread_complete_and_exit+0x20/0x20 [10473.958209] ret_from_fork+0x31/0x40 [10473.961781] ? kthread_complete_and_exit+0x20/0x20 [10473.966565] ret_from_fork_asm+0x11/0x20 [10473.970486] [10501.650396] watchdog: BUG: soft lockup - CPU#7 stuck for 7663s! [kworker/7:4:78808] [10501.658046] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10501.713498] irq event stamp: 18618588 [10501.717154] hardirqs last enabled at (18618587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10501.727320] hardirqs last disabled at (18618588): [] sysvec_apic_timer_interrupt+0xe/0x90 [10501.737054] softirqs last enabled at (18618546): [] __do_softirq+0x2be/0x371 [10501.745746] softirqs last disabled at (18618541): [] irq_exit_rcu+0xb0/0xd0 [10501.754267] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10501.763643] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10501.771115] Workqueue: events delayed_work [ceph] [10501.775837] RIP: 0010:lock_acquire+0xf0/0x2b0 [10501.780188] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10501.798925] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10501.804142] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10501.811266] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10501.818391] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10501.825515] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10501.832639] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10501.839765] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10501.847840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10501.853577] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10501.860702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10501.867827] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10501.874950] Call Trace: [10501.877397] [10501.879407] ? watchdog_timer_fn+0x1b4/0x220 [10501.883670] ? __hrtimer_run_queues+0x1d3/0x360 [10501.888198] ? hrtimer_interrupt+0x104/0x240 [10501.892467] ? __local_bh_enable+0x50/0x60 [10501.896560] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10501.901784] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10501.906741] [10501.908838] [10501.910935] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10501.916245] ? lock_acquire+0xf0/0x2b0 [10501.919993] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10501.925401] ? lock_release+0xc5/0x260 [10501.929146] _raw_spin_lock+0x30/0x40 [10501.932804] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10501.938201] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10501.943432] delayed_work+0x1d8/0x280 [ceph] [10501.947713] ? process_scheduled_works+0x22a/0x510 [10501.952501] process_scheduled_works+0x2a4/0x510 [10501.957118] worker_thread+0x188/0x340 [10501.960867] ? rescuer_thread+0x310/0x310 [10501.964870] kthread+0xf5/0x130 [10501.968017] ? kthread_complete_and_exit+0x20/0x20 [10501.972799] ret_from_fork+0x31/0x40 [10501.976380] ? kthread_complete_and_exit+0x20/0x20 [10501.981164] ret_from_fork_asm+0x11/0x20 [10501.985094] [10529.650891] watchdog: BUG: soft lockup - CPU#7 stuck for 7689s! [kworker/7:4:78808] [10529.658563] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10529.714011] irq event stamp: 18681582 [10529.717670] hardirqs last enabled at (18681581): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10529.727834] hardirqs last disabled at (18681582): [] sysvec_apic_timer_interrupt+0xe/0x90 [10529.737559] softirqs last enabled at (18681580): [] __do_softirq+0x2be/0x371 [10529.746252] softirqs last disabled at (18681575): [] irq_exit_rcu+0xb0/0xd0 [10529.754772] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10529.764151] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10529.771621] Workqueue: events delayed_work [ceph] [10529.776344] RIP: 0010:lock_acquire+0xf0/0x2b0 [10529.780694] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10529.799431] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10529.804649] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10529.811773] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10529.818899] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10529.826022] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10529.833145] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10529.840271] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10529.848347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10529.854084] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10529.861209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10529.868334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10529.875458] Call Trace: [10529.877905] [10529.879913] ? watchdog_timer_fn+0x1b4/0x220 [10529.884177] ? __hrtimer_run_queues+0x1d3/0x360 [10529.888702] ? hrtimer_interrupt+0x104/0x240 [10529.892964] ? __local_bh_enable+0x50/0x60 [10529.897057] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10529.902283] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10529.907239] [10529.909336] [10529.911433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10529.916740] ? lock_acquire+0xf0/0x2b0 [10529.920484] ? ceph_check_caps+0x26b/0xae0 [ceph] [10529.925198] ? lock_release+0xc5/0x260 [10529.928943] _raw_spin_lock+0x30/0x40 [10529.932599] ? ceph_check_caps+0x63/0xae0 [ceph] [10529.937218] ceph_check_caps+0x63/0xae0 [ceph] [10529.941668] ? __lock_acquire+0xbfa/0x1e20 [10529.945766] ? lock_acquire+0xcb/0x2b0 [10529.949516] ? igrab+0x1d/0x50 [10529.952566] ? find_held_lock+0x2d/0x90 [10529.956397] ? find_held_lock+0x2d/0x90 [10529.960229] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10529.965628] ? lock_release+0xc5/0x260 [10529.969375] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10529.974609] delayed_work+0x1d8/0x280 [ceph] [10529.978890] ? process_scheduled_works+0x22a/0x510 [10529.983679] process_scheduled_works+0x2a4/0x510 [10529.988295] worker_thread+0x188/0x340 [10529.992045] ? rescuer_thread+0x310/0x310 [10529.996048] kthread+0xf5/0x130 [10529.999186] ? kthread_complete_and_exit+0x20/0x20 [10530.003971] ret_from_fork+0x31/0x40 [10530.007548] ? kthread_complete_and_exit+0x20/0x20 [10530.012334] ret_from_fork_asm+0x11/0x20 [10530.016264] [10557.651387] watchdog: BUG: soft lockup - CPU#7 stuck for 7715s! [kworker/7:4:78808] [10557.659033] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10557.714484] irq event stamp: 18744854 [10557.718140] hardirqs last enabled at (18744853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10557.728306] hardirqs last disabled at (18744854): [] sysvec_apic_timer_interrupt+0xe/0x90 [10557.738032] softirqs last enabled at (18744776): [] __do_softirq+0x2be/0x371 [10557.746723] softirqs last disabled at (18744771): [] irq_exit_rcu+0xb0/0xd0 [10557.755238] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10557.764623] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10557.772094] Workqueue: events delayed_work [ceph] [10557.776817] RIP: 0010:lock_acquire+0xf0/0x2b0 [10557.781165] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10557.799903] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10557.805122] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10557.812245] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10557.819369] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10557.826494] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10557.833617] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10557.840742] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10557.848818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10557.854556] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10557.861680] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10557.868806] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10557.875929] Call Trace: [10557.878376] [10557.880387] ? watchdog_timer_fn+0x1b4/0x220 [10557.884657] ? __hrtimer_run_queues+0x1d3/0x360 [10557.889184] ? hrtimer_interrupt+0x104/0x240 [10557.893454] ? __local_bh_enable+0x50/0x60 [10557.897546] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10557.902772] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10557.907729] [10557.909826] [10557.911922] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10557.917230] ? lock_acquire+0xf0/0x2b0 [10557.920972] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10557.926379] ? lock_release+0xc5/0x260 [10557.930125] _raw_spin_lock+0x30/0x40 [10557.933782] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10557.939180] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10557.944408] delayed_work+0x1d8/0x280 [ceph] [10557.948690] ? process_scheduled_works+0x22a/0x510 [10557.953480] process_scheduled_works+0x2a4/0x510 [10557.958095] worker_thread+0x188/0x340 [10557.961843] ? rescuer_thread+0x310/0x310 [10557.965847] kthread+0xf5/0x130 [10557.968985] ? kthread_complete_and_exit+0x20/0x20 [10557.973770] ret_from_fork+0x31/0x40 [10557.977339] ? kthread_complete_and_exit+0x20/0x20 [10557.982125] ret_from_fork_asm+0x11/0x20 [10557.986046] [10585.651882] watchdog: BUG: soft lockup - CPU#7 stuck for 7741s! [kworker/7:4:78808] [10585.659532] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10585.714981] irq event stamp: 18805394 [10585.718637] hardirqs last enabled at (18805393): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10585.728804] hardirqs last disabled at (18805394): [] sysvec_apic_timer_interrupt+0xe/0x90 [10585.738528] softirqs last enabled at (18805380): [] __do_softirq+0x2be/0x371 [10585.747213] softirqs last disabled at (18805375): [] irq_exit_rcu+0xb0/0xd0 [10585.755734] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10585.765110] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10585.772582] Workqueue: events delayed_work [ceph] [10585.777305] RIP: 0010:lock_release+0x14e/0x260 [10585.781752] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [10585.800487] RSP: 0018:ffffc90002117b78 EFLAGS: 00000206 [10585.805706] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [10585.812831] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [10585.819952] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [10585.827077] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [10585.834202] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [10585.841327] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10585.849403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10585.855141] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10585.862266] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10585.869388] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10585.876513] Call Trace: [10585.878960] [10585.880971] ? watchdog_timer_fn+0x1b4/0x220 [10585.885242] ? __hrtimer_run_queues+0x1d3/0x360 [10585.889769] ? hrtimer_interrupt+0x104/0x240 [10585.894038] ? __local_bh_enable+0x50/0x60 [10585.898130] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10585.903354] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10585.908313] [10585.910408] [10585.912508] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10585.917812] ? ceph_check_caps+0x26b/0xae0 [ceph] [10585.922530] ? lock_release+0x14e/0x260 [10585.926366] ? lock_release+0xc5/0x260 [10585.930111] _raw_spin_unlock+0x1b/0x40 [10585.933941] ceph_check_caps+0x26b/0xae0 [ceph] [10585.938483] ? lock_acquire+0xcb/0x2b0 [10585.942225] ? igrab+0x1d/0x50 [10585.945277] ? find_held_lock+0x2d/0x90 [10585.949116] ? find_held_lock+0x2d/0x90 [10585.952949] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10585.958366] ? lock_release+0xc5/0x260 [10585.962118] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10585.967355] delayed_work+0x1d8/0x280 [ceph] [10585.971646] ? process_scheduled_works+0x22a/0x510 [10585.976435] process_scheduled_works+0x2a4/0x510 [10585.981058] worker_thread+0x188/0x340 [10585.984807] ? rescuer_thread+0x310/0x310 [10585.988819] kthread+0xf5/0x130 [10585.991966] ? kthread_complete_and_exit+0x20/0x20 [10585.996751] ret_from_fork+0x31/0x40 [10586.000328] ? kthread_complete_and_exit+0x20/0x20 [10586.005113] ret_from_fork_asm+0x11/0x20 [10586.009046] [10613.652377] watchdog: BUG: soft lockup - CPU#7 stuck for 7767s! [kworker/7:4:78808] [10613.660031] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10613.715481] irq event stamp: 18867182 [10613.719137] hardirqs last enabled at (18867181): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10613.729302] hardirqs last disabled at (18867182): [] sysvec_apic_timer_interrupt+0xe/0x90 [10613.739027] softirqs last enabled at (18867086): [] __do_softirq+0x2be/0x371 [10613.747720] softirqs last disabled at (18867081): [] irq_exit_rcu+0xb0/0xd0 [10613.756240] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10613.765618] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10613.773089] Workqueue: events delayed_work [ceph] [10613.777811] RIP: 0010:lock_acquire+0xf0/0x2b0 [10613.782163] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10613.800899] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10613.806118] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10613.813242] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10613.820366] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10613.827488] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10613.834613] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10613.841739] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10613.849816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10613.855552] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10613.862676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10613.869800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10613.876925] Call Trace: [10613.879370] [10613.881380] ? watchdog_timer_fn+0x1b4/0x220 [10613.885644] ? __hrtimer_run_queues+0x1d3/0x360 [10613.890172] ? hrtimer_interrupt+0x104/0x240 [10613.894441] ? __local_bh_enable+0x50/0x60 [10613.898534] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10613.903758] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10613.908715] [10613.910811] [10613.912909] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10613.918218] ? lock_acquire+0xf0/0x2b0 [10613.921967] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10613.927376] ? lock_release+0xc5/0x260 [10613.931119] _raw_spin_lock+0x30/0x40 [10613.934777] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10613.940175] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10613.945404] delayed_work+0x1d8/0x280 [ceph] [10613.949686] ? process_scheduled_works+0x22a/0x510 [10613.954475] process_scheduled_works+0x2a4/0x510 [10613.959091] worker_thread+0x188/0x340 [10613.962839] ? rescuer_thread+0x310/0x310 [10613.966844] kthread+0xf5/0x130 [10613.969991] ? kthread_complete_and_exit+0x20/0x20 [10613.974773] ret_from_fork+0x31/0x40 [10613.978353] ? kthread_complete_and_exit+0x20/0x20 [10613.983138] ret_from_fork_asm+0x11/0x20 [10613.987070] [10641.652872] watchdog: BUG: soft lockup - CPU#7 stuck for 7793s! [kworker/7:4:78808] [10641.660529] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10641.715977] irq event stamp: 18929744 [10641.719633] hardirqs last enabled at (18929743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10641.729801] hardirqs last disabled at (18929744): [] sysvec_apic_timer_interrupt+0xe/0x90 [10641.739533] softirqs last enabled at (18929730): [] __do_softirq+0x2be/0x371 [10641.748226] softirqs last disabled at (18929725): [] irq_exit_rcu+0xb0/0xd0 [10641.756747] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10641.766124] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10641.773595] Workqueue: events delayed_work [ceph] [10641.778316] RIP: 0010:lock_acquire+0xf0/0x2b0 [10641.782668] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10641.801405] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10641.806622] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10641.813748] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10641.820871] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10641.827994] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10641.835119] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10641.842245] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10641.850321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10641.856059] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10641.863184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10641.870306] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10641.877431] Call Trace: [10641.879878] [10641.881887] ? watchdog_timer_fn+0x1b4/0x220 [10641.886151] ? __hrtimer_run_queues+0x1d3/0x360 [10641.890686] ? hrtimer_interrupt+0x104/0x240 [10641.894955] ? __local_bh_enable+0x50/0x60 [10641.899047] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10641.904265] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10641.909222] [10641.911318] [10641.913416] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10641.918724] ? lock_acquire+0xf0/0x2b0 [10641.922474] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10641.927882] ? lock_release+0xc5/0x260 [10641.931626] _raw_spin_lock+0x30/0x40 [10641.935283] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10641.940683] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10641.945911] delayed_work+0x1d8/0x280 [ceph] [10641.950193] ? process_scheduled_works+0x22a/0x510 [10641.954982] process_scheduled_works+0x2a4/0x510 [10641.959599] worker_thread+0x188/0x340 [10641.963347] ? rescuer_thread+0x310/0x310 [10641.967350] kthread+0xf5/0x130 [10641.970488] ? kthread_complete_and_exit+0x20/0x20 [10641.975273] ret_from_fork+0x31/0x40 [10641.978852] ? kthread_complete_and_exit+0x20/0x20 [10641.983636] ret_from_fork_asm+0x11/0x20 [10641.987567] [10669.653368] watchdog: BUG: soft lockup - CPU#7 stuck for 7819s! [kworker/7:4:78808] [10669.661018] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10669.716468] irq event stamp: 18990490 [10669.720125] hardirqs last enabled at (18990489): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10669.730290] hardirqs last disabled at (18990490): [] sysvec_apic_timer_interrupt+0xe/0x90 [10669.740021] softirqs last enabled at (18990476): [] __do_softirq+0x2be/0x371 [10669.748708] softirqs last disabled at (18990471): [] irq_exit_rcu+0xb0/0xd0 [10669.757227] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10669.766603] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10669.774075] Workqueue: events delayed_work [ceph] [10669.778799] RIP: 0010:lock_acquire+0x85/0x2b0 [10669.783158] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [10669.801896] RSP: 0018:ffffc90002117d28 EFLAGS: 00000246 [10669.807112] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 0000000000000000 [10669.814237] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f8674a0 [10669.821361] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [10669.828484] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [10669.835610] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10669.842733] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10669.850811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10669.856548] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10669.863672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10669.870796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10669.877919] Call Trace: [10669.880367] [10669.882377] ? watchdog_timer_fn+0x1b4/0x220 [10669.886649] ? __hrtimer_run_queues+0x1d3/0x360 [10669.891176] ? hrtimer_interrupt+0x104/0x240 [10669.895444] ? __local_bh_enable+0x50/0x60 [10669.899538] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10669.904763] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10669.909718] [10669.911817] [10669.913913] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10669.919223] ? lock_acquire+0x85/0x2b0 [10669.922972] ? find_held_lock+0x2d/0x90 [10669.926809] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10669.932211] ? lock_release+0xc5/0x260 [10669.935955] _raw_spin_lock+0x30/0x40 [10669.939611] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10669.945010] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10669.950241] delayed_work+0x1d8/0x280 [ceph] [10669.954521] ? process_scheduled_works+0x22a/0x510 [10669.959311] process_scheduled_works+0x2a4/0x510 [10669.963926] worker_thread+0x188/0x340 [10669.967676] ? rescuer_thread+0x310/0x310 [10669.971679] kthread+0xf5/0x130 [10669.974826] ? kthread_complete_and_exit+0x20/0x20 [10669.979609] ret_from_fork+0x31/0x40 [10669.983189] ? kthread_complete_and_exit+0x20/0x20 [10669.987972] ret_from_fork_asm+0x11/0x20 [10669.991896] [10697.653863] watchdog: BUG: soft lockup - CPU#7 stuck for 7845s! [kworker/7:4:78808] [10697.661515] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10697.716966] irq event stamp: 19050818 [10697.720620] hardirqs last enabled at (19050817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10697.730788] hardirqs last disabled at (19050818): [] sysvec_apic_timer_interrupt+0xe/0x90 [10697.740520] softirqs last enabled at (19050812): [] __do_softirq+0x2be/0x371 [10697.749213] softirqs last disabled at (19050807): [] irq_exit_rcu+0xb0/0xd0 [10697.757724] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10697.767102] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10697.774574] Workqueue: events delayed_work [ceph] [10697.779296] RIP: 0010:ceph_check_delayed_caps+0x8a/0x270 [ceph] [10697.785225] Code: 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 [10697.803962] RSP: 0018:ffffc90002117db8 EFLAGS: 00000216 [10697.809178] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 00000001009ea280 [10697.816303] RDX: 00000001001ae57b RSI: 0000000000000000 RDI: ffff8881a1d34bc8 [10697.823427] RBP: ffff88817f867000 R08: 000000000000694b R09: 0000000000000002 [10697.830552] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [10697.837674] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10697.844800] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10697.852876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10697.858615] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10697.865737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10697.872862] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10697.879987] Call Trace: [10697.882435] [10697.884453] ? watchdog_timer_fn+0x1b4/0x220 [10697.888725] ? __hrtimer_run_queues+0x1d3/0x360 [10697.893258] ? hrtimer_interrupt+0x104/0x240 [10697.897519] ? __local_bh_enable+0x50/0x60 [10697.901614] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10697.906838] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10697.911793] [10697.913891] [10697.915988] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10697.921297] ? ceph_check_delayed_caps+0x8a/0x270 [ceph] [10697.926616] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10697.932017] delayed_work+0x1d8/0x280 [ceph] [10697.936297] ? process_scheduled_works+0x22a/0x510 [10697.941087] process_scheduled_works+0x2a4/0x510 [10697.945705] worker_thread+0x188/0x340 [10697.949452] ? rescuer_thread+0x310/0x310 [10697.953455] kthread+0xf5/0x130 [10697.956594] ? kthread_complete_and_exit+0x20/0x20 [10697.961379] ret_from_fork+0x31/0x40 [10697.964959] ? kthread_complete_and_exit+0x20/0x20 [10697.969741] ret_from_fork_asm+0x11/0x20 [10697.973664] [10725.654358] watchdog: BUG: soft lockup - CPU#7 stuck for 7872s! [kworker/7:4:78808] [10725.662011] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10725.717462] irq event stamp: 19111854 [10725.721120] hardirqs last enabled at (19111853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10725.731286] hardirqs last disabled at (19111854): [] sysvec_apic_timer_interrupt+0xe/0x90 [10725.741017] softirqs last enabled at (19111810): [] __do_softirq+0x2be/0x371 [10725.749702] softirqs last disabled at (19111805): [] irq_exit_rcu+0xb0/0xd0 [10725.758223] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10725.767601] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10725.775071] Workqueue: events delayed_work [ceph] [10725.779794] RIP: 0010:lock_acquire+0xf0/0x2b0 [10725.784145] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10725.802880] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10725.808099] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10725.815224] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10725.822349] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10725.829472] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10725.836595] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10725.843721] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10725.851798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10725.857535] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10725.864659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10725.871782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10725.878907] Call Trace: [10725.881354] [10725.883365] ? watchdog_timer_fn+0x1b4/0x220 [10725.887637] ? __hrtimer_run_queues+0x1d3/0x360 [10725.892169] ? hrtimer_interrupt+0x104/0x240 [10725.896433] ? __local_bh_enable+0x50/0x60 [10725.900526] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10725.905750] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10725.910707] [10725.912804] [10725.914901] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10725.920209] ? lock_acquire+0xf0/0x2b0 [10725.923959] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10725.929365] ? lock_release+0xc5/0x260 [10725.933111] _raw_spin_lock+0x30/0x40 [10725.936769] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10725.942168] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10725.947395] delayed_work+0x1d8/0x280 [ceph] [10725.951677] ? process_scheduled_works+0x22a/0x510 [10725.956467] process_scheduled_works+0x2a4/0x510 [10725.961083] worker_thread+0x188/0x340 [10725.964831] ? rescuer_thread+0x310/0x310 [10725.968834] kthread+0xf5/0x130 [10725.971973] ? kthread_complete_and_exit+0x20/0x20 [10725.976756] ret_from_fork+0x31/0x40 [10725.980327] ? kthread_complete_and_exit+0x20/0x20 [10725.985110] ret_from_fork_asm+0x11/0x20 [10725.989033] [10753.654853] watchdog: BUG: soft lockup - CPU#7 stuck for 7898s! [kworker/7:4:78808] [10753.662503] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10753.717951] irq event stamp: 19174618 [10753.721608] hardirqs last enabled at (19174617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10753.731774] hardirqs last disabled at (19174618): [] sysvec_apic_timer_interrupt+0xe/0x90 [10753.741497] softirqs last enabled at (19174604): [] __do_softirq+0x2be/0x371 [10753.750183] softirqs last disabled at (19174599): [] irq_exit_rcu+0xb0/0xd0 [10753.758704] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10753.768080] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10753.775551] Workqueue: events delayed_work [ceph] [10753.780275] RIP: 0010:lock_acquire+0xf0/0x2b0 [10753.784626] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10753.803363] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [10753.808581] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [10753.815705] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [10753.822828] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [10753.829953] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [10753.837075] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10753.844201] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10753.852279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10753.858016] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10753.865139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10753.872264] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10753.879387] Call Trace: [10753.881835] [10753.883842] ? watchdog_timer_fn+0x1b4/0x220 [10753.888107] ? __hrtimer_run_queues+0x1d3/0x360 [10753.892632] ? hrtimer_interrupt+0x104/0x240 [10753.896895] ? __local_bh_enable+0x50/0x60 [10753.900988] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10753.906212] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10753.911170] [10753.913265] [10753.915364] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10753.920671] ? lock_acquire+0xf0/0x2b0 [10753.924424] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10753.929830] _raw_spin_lock+0x30/0x40 [10753.933487] ? igrab+0x1d/0x50 [10753.936537] igrab+0x1d/0x50 [10753.939416] ceph_check_delayed_caps+0xb7/0x270 [ceph] [10753.944557] delayed_work+0x1d8/0x280 [ceph] [10753.948838] ? process_scheduled_works+0x22a/0x510 [10753.953627] process_scheduled_works+0x2a4/0x510 [10753.958243] worker_thread+0x188/0x340 [10753.962011] ? rescuer_thread+0x310/0x310 [10753.966021] kthread+0xf5/0x130 [10753.969159] ? kthread_complete_and_exit+0x20/0x20 [10753.973943] ret_from_fork+0x31/0x40 [10753.977524] ? kthread_complete_and_exit+0x20/0x20 [10753.982308] ret_from_fork_asm+0x11/0x20 [10753.986229] [10781.495947] libceph: auth protocol 'cephx' mauth authentication failed: -13 [10781.655367] watchdog: BUG: soft lockup - CPU#7 stuck for 7924s! [kworker/7:4:78808] [10781.663015] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10781.718468] irq event stamp: 19236322 [10781.722123] hardirqs last enabled at (19236321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10781.732290] hardirqs last disabled at (19236322): [] sysvec_apic_timer_interrupt+0xe/0x90 [10781.742013] softirqs last enabled at (19236302): [] __do_softirq+0x2be/0x371 [10781.750698] softirqs last disabled at (19236297): [] irq_exit_rcu+0xb0/0xd0 [10781.759220] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10781.768603] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10781.776074] Workqueue: events delayed_work [ceph] [10781.780798] RIP: 0010:lock_acquire+0xf0/0x2b0 [10781.785150] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10781.803885] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [10781.809102] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [10781.816227] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [10781.823351] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [10781.830476] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [10781.837600] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10781.844725] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10781.852802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10781.858539] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10781.865662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10781.872787] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10781.879912] Call Trace: [10781.882357] [10781.884368] ? watchdog_timer_fn+0x1b4/0x220 [10781.888630] ? __hrtimer_run_queues+0x1d3/0x360 [10781.893157] ? hrtimer_interrupt+0x104/0x240 [10781.897419] ? __local_bh_enable+0x50/0x60 [10781.901511] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10781.906735] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10781.911693] [10781.913789] [10781.915887] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10781.921195] ? lock_acquire+0xf0/0x2b0 [10781.924946] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10781.930352] ? lock_release+0xc5/0x260 [10781.934098] _raw_spin_lock+0x30/0x40 [10781.937756] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10781.943155] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [10781.948382] delayed_work+0x1d8/0x280 [ceph] [10781.952664] ? process_scheduled_works+0x22a/0x510 [10781.957454] process_scheduled_works+0x2a4/0x510 [10781.962069] worker_thread+0x188/0x340 [10781.965817] ? rescuer_thread+0x310/0x310 [10781.969823] kthread+0xf5/0x130 [10781.972969] ? kthread_complete_and_exit+0x20/0x20 [10781.977753] ret_from_fork+0x31/0x40 [10781.981332] ? kthread_complete_and_exit+0x20/0x20 [10781.986114] ret_from_fork_asm+0x11/0x20 [10781.990038] [10792.247274] libceph: auth protocol 'cephx' mauth authentication failed: -13 [10802.487759] libceph: auth protocol 'cephx' mauth authentication failed: -13 [10809.655843] watchdog: BUG: soft lockup - CPU#7 stuck for 7950s! [kworker/7:4:78808] [10809.663499] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10809.718947] irq event stamp: 19297206 [10809.722603] hardirqs last enabled at (19297205): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10809.732770] hardirqs last disabled at (19297206): [] sysvec_apic_timer_interrupt+0xe/0x90 [10809.742495] softirqs last enabled at (19297192): [] __do_softirq+0x2be/0x371 [10809.751187] softirqs last disabled at (19297187): [] irq_exit_rcu+0xb0/0xd0 [10809.759698] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10809.769077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10809.776546] Workqueue: events delayed_work [ceph] [10809.781270] RIP: 0010:lock_acquire+0xf0/0x2b0 [10809.785620] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10809.804357] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10809.809574] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10809.816699] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10809.823825] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10809.830946] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10809.838072] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10809.845196] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10809.853275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10809.859010] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10809.866136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10809.873259] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10809.880383] Call Trace: [10809.882830] [10809.884840] ? watchdog_timer_fn+0x1b4/0x220 [10809.889111] ? __hrtimer_run_queues+0x1d3/0x360 [10809.893638] ? hrtimer_interrupt+0x104/0x240 [10809.897900] ? __local_bh_enable+0x50/0x60 [10809.901992] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10809.907217] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10809.912175] [10809.914270] [10809.916369] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10809.921676] ? lock_acquire+0xf0/0x2b0 [10809.925427] ? ceph_check_caps+0x26b/0xae0 [ceph] [10809.930141] ? lock_release+0xc5/0x260 [10809.933887] _raw_spin_lock+0x30/0x40 [10809.937550] ? ceph_check_caps+0x63/0xae0 [ceph] [10809.942170] ceph_check_caps+0x63/0xae0 [ceph] [10809.946621] ? __lock_acquire+0xbfa/0x1e20 [10809.950719] ? lock_acquire+0xcb/0x2b0 [10809.954467] ? igrab+0x1d/0x50 [10809.957519] ? find_held_lock+0x2d/0x90 [10809.961350] ? find_held_lock+0x2d/0x90 [10809.965180] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10809.970579] ? lock_release+0xc5/0x260 [10809.974326] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10809.979553] delayed_work+0x1d8/0x280 [ceph] [10809.983835] ? process_scheduled_works+0x22a/0x510 [10809.988623] process_scheduled_works+0x2a4/0x510 [10809.993240] worker_thread+0x188/0x340 [10809.996988] ? rescuer_thread+0x310/0x310 [10810.000990] kthread+0xf5/0x130 [10810.004128] ? kthread_complete_and_exit+0x20/0x20 [10810.008913] ret_from_fork+0x31/0x40 [10810.012492] ? kthread_complete_and_exit+0x20/0x20 [10810.017276] ret_from_fork_asm+0x11/0x20 [10810.021199] [10813.239646] libceph: auth protocol 'cephx' mauth authentication failed: -13 [10823.480040] libceph: auth protocol 'cephx' mauth authentication failed: -13 [10834.232201] libceph: auth protocol 'cephx' mauth authentication failed: -13 [10837.656338] watchdog: BUG: soft lockup - CPU#7 stuck for 7976s! [kworker/7:4:78808] [10837.663987] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10837.719437] irq event stamp: 19358814 [10837.723093] hardirqs last enabled at (19358813): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10837.733258] hardirqs last disabled at (19358814): [] sysvec_apic_timer_interrupt+0xe/0x90 [10837.742984] softirqs last enabled at (19358736): [] __do_softirq+0x2be/0x371 [10837.751677] softirqs last disabled at (19358731): [] irq_exit_rcu+0xb0/0xd0 [10837.760198] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10837.769582] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10837.777054] Workqueue: events delayed_work [ceph] [10837.781775] RIP: 0010:lock_acquire+0xf0/0x2b0 [10837.786127] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10837.804865] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10837.810083] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10837.817205] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10837.824330] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10837.831454] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10837.838579] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10837.845701] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10837.853780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10837.859518] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10837.866641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10837.873765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10837.880889] Call Trace: [10837.883336] [10837.885347] ? watchdog_timer_fn+0x1b4/0x220 [10837.889619] ? __hrtimer_run_queues+0x1d3/0x360 [10837.894154] ? hrtimer_interrupt+0x104/0x240 [10837.898423] ? __local_bh_enable+0x50/0x60 [10837.902516] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10837.907739] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10837.912698] [10837.914795] [10837.916893] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10837.922200] ? lock_acquire+0xf0/0x2b0 [10837.925949] ? ceph_check_caps+0x26b/0xae0 [ceph] [10837.930663] ? lock_release+0xc5/0x260 [10837.934408] _raw_spin_lock+0x30/0x40 [10837.938067] ? ceph_check_caps+0x63/0xae0 [ceph] [10837.942694] ceph_check_caps+0x63/0xae0 [ceph] [10837.947144] ? __lock_acquire+0xbfa/0x1e20 [10837.951242] ? lock_acquire+0xcb/0x2b0 [10837.954991] ? igrab+0x1d/0x50 [10837.958041] ? find_held_lock+0x2d/0x90 [10837.961873] ? find_held_lock+0x2d/0x90 [10837.965705] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10837.971104] ? lock_release+0xc5/0x260 [10837.974848] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10837.980076] delayed_work+0x1d8/0x280 [ceph] [10837.984358] ? process_scheduled_works+0x22a/0x510 [10837.989146] process_scheduled_works+0x2a4/0x510 [10837.993762] worker_thread+0x188/0x340 [10837.997513] ? rescuer_thread+0x310/0x310 [10838.001524] kthread+0xf5/0x130 [10838.004672] ? kthread_complete_and_exit+0x20/0x20 [10838.009455] ret_from_fork+0x31/0x40 [10838.013035] ? kthread_complete_and_exit+0x20/0x20 [10838.017817] ret_from_fork_asm+0x11/0x20 [10838.021741] [10865.464232] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [10865.656854] watchdog: BUG: soft lockup - CPU#7 stuck for 8002s! [kworker/7:4:78808] [10865.664500] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10865.719950] irq event stamp: 19420634 [10865.723607] hardirqs last enabled at (19420633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10865.733774] hardirqs last disabled at (19420634): [] sysvec_apic_timer_interrupt+0xe/0x90 [10865.743508] softirqs last enabled at (19420556): [] __do_softirq+0x2be/0x371 [10865.752200] softirqs last disabled at (19420551): [] irq_exit_rcu+0xb0/0xd0 [10865.760719] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10865.770096] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10865.777567] Workqueue: events delayed_work [ceph] [10865.782291] RIP: 0010:lock_acquire+0xf0/0x2b0 [10865.786643] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10865.805379] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [10865.810595] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [10865.817721] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [10865.824845] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [10865.831970] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [10865.839092] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10865.846217] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10865.854296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10865.860033] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10865.867157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10865.874280] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10865.881404] Call Trace: [10865.883851] [10865.885862] ? watchdog_timer_fn+0x1b4/0x220 [10865.890133] ? __hrtimer_run_queues+0x1d3/0x360 [10865.894659] ? hrtimer_interrupt+0x104/0x240 [10865.898921] ? __local_bh_enable+0x50/0x60 [10865.903014] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10865.908237] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10865.913194] [10865.915292] [10865.917390] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10865.922698] ? lock_acquire+0xf0/0x2b0 [10865.926449] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10865.931858] _raw_spin_lock+0x30/0x40 [10865.935520] ? igrab+0x1d/0x50 [10865.938571] igrab+0x1d/0x50 [10865.941450] ceph_check_delayed_caps+0xb7/0x270 [ceph] [10865.946592] delayed_work+0x1d8/0x280 [ceph] [10865.950873] ? process_scheduled_works+0x22a/0x510 [10865.955663] process_scheduled_works+0x2a4/0x510 [10865.960278] worker_thread+0x188/0x340 [10865.964027] ? rescuer_thread+0x310/0x310 [10865.968031] kthread+0xf5/0x130 [10865.971169] ? kthread_complete_and_exit+0x20/0x20 [10865.975952] ret_from_fork+0x31/0x40 [10865.979533] ? kthread_complete_and_exit+0x20/0x20 [10865.984323] ret_from_fork_asm+0x11/0x20 [10865.988247] [10893.657345] watchdog: BUG: soft lockup - CPU#7 stuck for 8028s! [kworker/7:4:78808] [10893.664998] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10893.720448] irq event stamp: 19482816 [10893.724106] hardirqs last enabled at (19482815): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10893.734273] hardirqs last disabled at (19482816): [] sysvec_apic_timer_interrupt+0xe/0x90 [10893.744006] softirqs last enabled at (19482796): [] __do_softirq+0x2be/0x371 [10893.752697] softirqs last disabled at (19482791): [] irq_exit_rcu+0xb0/0xd0 [10893.761218] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10893.770596] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10893.778065] Workqueue: events delayed_work [ceph] [10893.782788] RIP: 0010:lock_release+0x3c/0x260 [10893.787140] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [10893.805877] RSP: 0018:ffffc90002117d60 EFLAGS: 00000247 [10893.811096] RAX: 0000000000000007 RBX: ffff88817f867488 RCX: ffffc90002117d54 [10893.818219] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [10893.825343] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [10893.832466] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [10893.839590] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [10893.846716] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10893.854791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10893.860529] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10893.867653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10893.874777] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10893.881902] Call Trace: [10893.884349] [10893.886359] ? watchdog_timer_fn+0x1b4/0x220 [10893.890623] ? __hrtimer_run_queues+0x1d3/0x360 [10893.895158] ? hrtimer_interrupt+0x104/0x240 [10893.899427] ? __local_bh_enable+0x50/0x60 [10893.903519] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10893.908743] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10893.913701] [10893.915798] [10893.917896] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10893.923200] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10893.928603] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10893.934000] ? lock_release+0x3c/0x260 [10893.937747] _raw_spin_unlock+0x1b/0x40 [10893.941584] ceph_check_delayed_caps+0x19b/0x270 [ceph] [10893.946812] delayed_work+0x1d8/0x280 [ceph] [10893.951093] ? process_scheduled_works+0x22a/0x510 [10893.955882] process_scheduled_works+0x2a4/0x510 [10893.960497] worker_thread+0x188/0x340 [10893.964247] ? rescuer_thread+0x310/0x310 [10893.968251] kthread+0xf5/0x130 [10893.971388] ? kthread_complete_and_exit+0x20/0x20 [10893.976174] ret_from_fork+0x31/0x40 [10893.979751] ? kthread_complete_and_exit+0x20/0x20 [10893.984534] ret_from_fork_asm+0x11/0x20 [10893.988458] [10921.657823] watchdog: BUG: soft lockup - CPU#7 stuck for 8054s! [kworker/7:4:78808] [10921.665481] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10921.720928] irq event stamp: 19546136 [10921.724587] hardirqs last enabled at (19546135): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10921.734752] hardirqs last disabled at (19546136): [] sysvec_apic_timer_interrupt+0xe/0x90 [10921.744475] softirqs last enabled at (19546114): [] __do_softirq+0x2be/0x371 [10921.753170] softirqs last disabled at (19546109): [] irq_exit_rcu+0xb0/0xd0 [10921.761689] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10921.771066] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10921.778537] Workqueue: events delayed_work [ceph] [10921.783262] RIP: 0010:ceph_check_caps+0x18c/0xae0 [ceph] [10921.788583] Code: 89 ee 44 09 e8 8b 91 b0 01 00 00 f7 d5 23 ac 24 94 00 00 00 89 c6 83 ce 01 89 74 24 04 85 d2 75 1b 44 8b 7b 48 45 85 ff 74 12 <45> 85 ed 0f 84 0d 05 00 00 0d fd bf 00 00 89 44 24 04 0f 1f 44 00 [10921.807319] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000202 [10921.812536] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffff88817f867000 [10921.819662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810da9e008 [10921.826785] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [10921.833909] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [10921.841034] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 [10921.848159] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10921.856234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10921.861972] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10921.869098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10921.876220] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10921.883345] Call Trace: [10921.885793] [10921.887811] ? watchdog_timer_fn+0x1b4/0x220 [10921.892083] ? __hrtimer_run_queues+0x1d3/0x360 [10921.896618] ? hrtimer_interrupt+0x104/0x240 [10921.900887] ? __local_bh_enable+0x50/0x60 [10921.904980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10921.910204] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10921.915161] [10921.917258] [10921.919356] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10921.924665] ? ceph_check_caps+0x18c/0xae0 [ceph] [10921.929384] ? lock_acquire+0xcb/0x2b0 [10921.933128] ? igrab+0x1d/0x50 [10921.936178] ? find_held_lock+0x2d/0x90 [10921.940008] ? find_held_lock+0x2d/0x90 [10921.943840] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10921.949240] ? lock_release+0xc5/0x260 [10921.952985] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10921.958220] delayed_work+0x1d8/0x280 [ceph] [10921.962502] ? process_scheduled_works+0x22a/0x510 [10921.967293] process_scheduled_works+0x2a4/0x510 [10921.971907] worker_thread+0x188/0x340 [10921.975656] ? rescuer_thread+0x310/0x310 [10921.979659] kthread+0xf5/0x130 [10921.982798] ? kthread_complete_and_exit+0x20/0x20 [10921.987581] ret_from_fork+0x31/0x40 [10921.991160] ? kthread_complete_and_exit+0x20/0x20 [10921.995944] ret_from_fork_asm+0x11/0x20 [10921.999867] [10949.658319] watchdog: BUG: soft lockup - CPU#7 stuck for 8080s! [kworker/7:4:78808] [10949.665969] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10949.721418] irq event stamp: 19609218 [10949.725074] hardirqs last enabled at (19609217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10949.735242] hardirqs last disabled at (19609218): [] sysvec_apic_timer_interrupt+0xe/0x90 [10949.744974] softirqs last enabled at (19609092): [] __do_softirq+0x2be/0x371 [10949.753667] softirqs last disabled at (19609087): [] irq_exit_rcu+0xb0/0xd0 [10949.762179] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10949.771557] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10949.779027] Workqueue: events delayed_work [ceph] [10949.783750] RIP: 0010:ceph_check_caps+0x2f5/0xae0 [ceph] [10949.789071] Code: ff ff 48 85 c0 0f 84 55 01 00 00 48 39 d8 0f 84 4c 01 00 00 44 8b 60 40 41 f7 d4 44 23 64 24 20 41 8b 6f 38 f7 d5 41 23 6f 3c <0f> 1f 44 00 00 85 ed 74 1c 41 85 ec 0f 84 0b 06 00 00 41 0f b7 45 [10949.807809] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [10949.813025] RAX: ffff88810da9e000 RBX: ffff88810da9e000 RCX: ffff88817f867000 [10949.820150] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34b20 [10949.827273] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [10949.834399] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [10949.841522] R13: ffff8881a1d34478 R14: 0000000000000000 R15: ffff88810da9e008 [10949.848648] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10949.856724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10949.862462] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10949.869587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10949.876711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10949.883834] Call Trace: [10949.886280] [10949.888291] ? watchdog_timer_fn+0x1b4/0x220 [10949.892563] ? __hrtimer_run_queues+0x1d3/0x360 [10949.897097] ? hrtimer_interrupt+0x104/0x240 [10949.901359] ? __local_bh_enable+0x50/0x60 [10949.905450] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10949.910675] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10949.915634] [10949.917731] [10949.919827] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10949.925136] ? ceph_check_caps+0x2f5/0xae0 [ceph] [10949.929855] ? lock_acquire+0xcb/0x2b0 [10949.933599] ? igrab+0x1d/0x50 [10949.936649] ? find_held_lock+0x2d/0x90 [10949.940481] ? find_held_lock+0x2d/0x90 [10949.944311] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10949.949712] ? lock_release+0xc5/0x260 [10949.953457] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10949.958684] delayed_work+0x1d8/0x280 [ceph] [10949.962966] ? process_scheduled_works+0x22a/0x510 [10949.967756] process_scheduled_works+0x2a4/0x510 [10949.972371] worker_thread+0x188/0x340 [10949.976120] ? rescuer_thread+0x310/0x310 [10949.980123] kthread+0xf5/0x130 [10949.983261] ? kthread_complete_and_exit+0x20/0x20 [10949.988045] ret_from_fork+0x31/0x40 [10949.991624] ? kthread_complete_and_exit+0x20/0x20 [10949.996409] ret_from_fork_asm+0x11/0x20 [10950.000339] [10977.658836] watchdog: BUG: soft lockup - CPU#7 stuck for 8106s! [kworker/7:4:78808] [10977.666484] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [10977.721935] irq event stamp: 19671214 [10977.725590] hardirqs last enabled at (19671213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10977.735756] hardirqs last disabled at (19671214): [] sysvec_apic_timer_interrupt+0xe/0x90 [10977.745482] softirqs last enabled at (19671088): [] __do_softirq+0x2be/0x371 [10977.754174] softirqs last disabled at (19671083): [] irq_exit_rcu+0xb0/0xd0 [10977.762685] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [10977.772062] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [10977.779534] Workqueue: events delayed_work [ceph] [10977.784247] RIP: 0010:lock_acquire+0xf0/0x2b0 [10977.788598] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [10977.807335] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [10977.812553] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [10977.819678] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [10977.826801] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [10977.833924] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [10977.841049] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [10977.848173] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [10977.856251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10977.861990] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [10977.869112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10977.876237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [10977.883360] Call Trace: [10977.885806] [10977.887818] ? watchdog_timer_fn+0x1b4/0x220 [10977.892090] ? __hrtimer_run_queues+0x1d3/0x360 [10977.896616] ? hrtimer_interrupt+0x104/0x240 [10977.900877] ? __local_bh_enable+0x50/0x60 [10977.904969] ? __sysvec_apic_timer_interrupt+0x61/0x160 [10977.910194] ? sysvec_apic_timer_interrupt+0x6d/0x90 [10977.915151] [10977.917249] [10977.919347] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [10977.924654] ? lock_acquire+0xf0/0x2b0 [10977.928404] ? ceph_check_caps+0x26b/0xae0 [ceph] [10977.933118] ? lock_release+0xc5/0x260 [10977.936863] _raw_spin_lock+0x30/0x40 [10977.940519] ? ceph_check_caps+0x63/0xae0 [ceph] [10977.945139] ceph_check_caps+0x63/0xae0 [ceph] [10977.949590] ? __lock_acquire+0xbfa/0x1e20 [10977.953688] ? lock_acquire+0xcb/0x2b0 [10977.957437] ? igrab+0x1d/0x50 [10977.960486] ? find_held_lock+0x2d/0x90 [10977.964320] ? find_held_lock+0x2d/0x90 [10977.968149] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [10977.973549] ? lock_release+0xc5/0x260 [10977.977295] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [10977.982522] delayed_work+0x1d8/0x280 [ceph] [10977.986801] ? process_scheduled_works+0x22a/0x510 [10977.991584] process_scheduled_works+0x2a4/0x510 [10977.996199] worker_thread+0x188/0x340 [10977.999948] ? rescuer_thread+0x310/0x310 [10978.003951] kthread+0xf5/0x130 [10978.007089] ? kthread_complete_and_exit+0x20/0x20 [10978.011874] ret_from_fork+0x31/0x40 [10978.015453] ? kthread_complete_and_exit+0x20/0x20 [10978.020238] ret_from_fork_asm+0x11/0x20 [10978.024170] [11005.659310] watchdog: BUG: soft lockup - CPU#7 stuck for 8132s! [kworker/7:4:78808] [11005.666965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11005.722413] irq event stamp: 19732058 [11005.726071] hardirqs last enabled at (19732057): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11005.736236] hardirqs last disabled at (19732058): [] sysvec_apic_timer_interrupt+0xe/0x90 [11005.745961] softirqs last enabled at (19731980): [] __do_softirq+0x2be/0x371 [11005.754654] softirqs last disabled at (19731975): [] irq_exit_rcu+0xb0/0xd0 [11005.763175] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11005.772551] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11005.780021] Workqueue: events delayed_work [ceph] [11005.784746] RIP: 0010:lock_acquire+0xf0/0x2b0 [11005.789097] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11005.807832] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11005.813050] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11005.820174] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11005.827299] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11005.834424] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11005.841548] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11005.848672] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11005.856748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11005.862487] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11005.869610] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11005.876734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11005.883859] Call Trace: [11005.886306] [11005.888317] ? watchdog_timer_fn+0x1b4/0x220 [11005.892586] ? __hrtimer_run_queues+0x1d3/0x360 [11005.897114] ? hrtimer_interrupt+0x104/0x240 [11005.901383] ? __local_bh_enable+0x50/0x60 [11005.905475] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11005.910702] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11005.915658] [11005.917755] [11005.919853] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11005.925161] ? lock_acquire+0xf0/0x2b0 [11005.928909] ? ceph_check_caps+0x26b/0xae0 [ceph] [11005.933623] ? lock_release+0xc5/0x260 [11005.937368] _raw_spin_lock+0x30/0x40 [11005.941027] ? ceph_check_caps+0x63/0xae0 [ceph] [11005.945644] ceph_check_caps+0x63/0xae0 [ceph] [11005.950097] ? __lock_acquire+0xbfa/0x1e20 [11005.954194] ? lock_acquire+0xcb/0x2b0 [11005.957942] ? igrab+0x1d/0x50 [11005.960994] ? find_held_lock+0x2d/0x90 [11005.964834] ? find_held_lock+0x2d/0x90 [11005.968665] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11005.974073] ? lock_release+0xc5/0x260 [11005.977818] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11005.983045] delayed_work+0x1d8/0x280 [ceph] [11005.987326] ? process_scheduled_works+0x22a/0x510 [11005.992116] process_scheduled_works+0x2a4/0x510 [11005.996731] worker_thread+0x188/0x340 [11006.000481] ? rescuer_thread+0x310/0x310 [11006.004484] kthread+0xf5/0x130 [11006.007622] ? kthread_complete_and_exit+0x20/0x20 [11006.012406] ret_from_fork+0x31/0x40 [11006.015977] ? kthread_complete_and_exit+0x20/0x20 [11006.020762] ret_from_fork_asm+0x11/0x20 [11006.024693] [11033.659803] watchdog: BUG: soft lockup - CPU#7 stuck for 8158s! [kworker/7:4:78808] [11033.667471] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11033.722920] irq event stamp: 19792842 [11033.726577] hardirqs last enabled at (19792841): [] asm_common_interrupt+0x26/0x40 [11033.735789] hardirqs last disabled at (19792842): [] sysvec_apic_timer_interrupt+0xe/0x90 [11033.745513] softirqs last enabled at (19792840): [] __do_softirq+0x2be/0x371 [11033.754208] softirqs last disabled at (19792835): [] irq_exit_rcu+0xb0/0xd0 [11033.762726] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11033.772105] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11033.779575] Workqueue: events delayed_work [ceph] [11033.784299] RIP: 0010:lock_acquire+0xf0/0x2b0 [11033.788659] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11033.807395] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [11033.812612] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [11033.819736] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [11033.826859] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [11033.833984] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [11033.841109] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11033.848234] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11033.856311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11033.862049] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11033.869171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11033.876295] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11033.883420] Call Trace: [11033.885868] [11033.887879] ? watchdog_timer_fn+0x1b4/0x220 [11033.892150] ? __hrtimer_run_queues+0x1d3/0x360 [11033.896683] ? hrtimer_interrupt+0x104/0x240 [11033.900945] ? __local_bh_enable+0x50/0x60 [11033.905037] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11033.910261] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11033.915219] [11033.917317] [11033.919415] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11033.924721] ? lock_acquire+0xf0/0x2b0 [11033.928473] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11033.933882] _raw_spin_lock+0x30/0x40 [11033.937545] ? igrab+0x1d/0x50 [11033.940595] igrab+0x1d/0x50 [11033.943474] ceph_check_delayed_caps+0xb7/0x270 [ceph] [11033.948615] delayed_work+0x1d8/0x280 [ceph] [11033.952896] ? process_scheduled_works+0x22a/0x510 [11033.957688] process_scheduled_works+0x2a4/0x510 [11033.962302] worker_thread+0x188/0x340 [11033.966051] ? rescuer_thread+0x310/0x310 [11033.970055] kthread+0xf5/0x130 [11033.973193] ? kthread_complete_and_exit+0x20/0x20 [11033.977975] ret_from_fork+0x31/0x40 [11033.981546] ? kthread_complete_and_exit+0x20/0x20 [11033.986331] ret_from_fork_asm+0x11/0x20 [11033.990262] [11061.660298] watchdog: BUG: soft lockup - CPU#7 stuck for 8184s! [kworker/7:4:78808] [11061.667951] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11061.723401] irq event stamp: 19855036 [11061.727057] hardirqs last enabled at (19855035): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11061.737224] hardirqs last disabled at (19855036): [] sysvec_apic_timer_interrupt+0xe/0x90 [11061.746956] softirqs last enabled at (19855014): [] __do_softirq+0x2be/0x371 [11061.755648] softirqs last disabled at (19855009): [] irq_exit_rcu+0xb0/0xd0 [11061.764162] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11061.773546] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11061.781018] Workqueue: events delayed_work [ceph] [11061.785742] RIP: 0010:lock_acquire+0xf0/0x2b0 [11061.790102] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11061.808838] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11061.814055] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11061.821178] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11061.828302] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11061.835427] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11061.842551] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11061.849693] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11061.857770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11061.863509] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11061.870631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11061.877756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11061.884879] Call Trace: [11061.887326] [11061.889337] ? watchdog_timer_fn+0x1b4/0x220 [11061.893601] ? __hrtimer_run_queues+0x1d3/0x360 [11061.898135] ? hrtimer_interrupt+0x104/0x240 [11061.902403] ? __local_bh_enable+0x50/0x60 [11061.906497] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11061.911721] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11061.916679] [11061.918776] [11061.920874] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11061.926181] ? lock_acquire+0xf0/0x2b0 [11061.929931] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11061.935340] ? lock_release+0xc5/0x260 [11061.939085] _raw_spin_lock+0x30/0x40 [11061.942750] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11061.948157] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11061.953386] delayed_work+0x1d8/0x280 [ceph] [11061.957668] ? process_scheduled_works+0x22a/0x510 [11061.962474] process_scheduled_works+0x2a4/0x510 [11061.967091] worker_thread+0x188/0x340 [11061.970840] ? rescuer_thread+0x310/0x310 [11061.974842] kthread+0xf5/0x130 [11061.977996] ? kthread_complete_and_exit+0x20/0x20 [11061.982782] ret_from_fork+0x31/0x40 [11061.986361] ? kthread_complete_and_exit+0x20/0x20 [11061.991145] ret_from_fork_asm+0x11/0x20 [11061.995067] [11089.660793] watchdog: BUG: soft lockup - CPU#7 stuck for 8211s! [kworker/7:4:78808] [11089.668448] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11089.723899] irq event stamp: 19918104 [11089.727556] hardirqs last enabled at (19918103): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11089.737722] hardirqs last disabled at (19918104): [] sysvec_apic_timer_interrupt+0xe/0x90 [11089.747453] softirqs last enabled at (19918090): [] __do_softirq+0x2be/0x371 [11089.756138] softirqs last disabled at (19918085): [] irq_exit_rcu+0xb0/0xd0 [11089.764660] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11089.774035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11089.781508] Workqueue: events delayed_work [ceph] [11089.786229] RIP: 0010:lock_acquire+0xf0/0x2b0 [11089.790581] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11089.809318] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11089.814536] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11089.821659] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11089.828784] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11089.835907] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11089.843033] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11089.850155] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11089.858234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11089.863970] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11089.871095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11089.878220] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11089.885343] Call Trace: [11089.887790] [11089.889800] ? watchdog_timer_fn+0x1b4/0x220 [11089.894072] ? __hrtimer_run_queues+0x1d3/0x360 [11089.898607] ? hrtimer_interrupt+0x104/0x240 [11089.902877] ? __local_bh_enable+0x50/0x60 [11089.906969] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11089.912193] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11089.917150] [11089.919247] [11089.921345] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11089.926654] ? lock_acquire+0xf0/0x2b0 [11089.930404] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11089.935810] ? lock_release+0xc5/0x260 [11089.939555] _raw_spin_lock+0x30/0x40 [11089.943213] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11089.948611] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11089.953841] delayed_work+0x1d8/0x280 [ceph] [11089.958121] ? process_scheduled_works+0x22a/0x510 [11089.962910] process_scheduled_works+0x2a4/0x510 [11089.967527] worker_thread+0x188/0x340 [11089.971275] ? rescuer_thread+0x310/0x310 [11089.975278] kthread+0xf5/0x130 [11089.978417] ? kthread_complete_and_exit+0x20/0x20 [11089.983203] ret_from_fork+0x31/0x40 [11089.986781] ? kthread_complete_and_exit+0x20/0x20 [11089.991563] ret_from_fork_asm+0x11/0x20 [11089.995487] [11117.661288] watchdog: BUG: soft lockup - CPU#7 stuck for 8237s! [kworker/7:4:78808] [11117.668956] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11117.724405] irq event stamp: 19984418 [11117.728062] hardirqs last enabled at (19984417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11117.738227] hardirqs last disabled at (19984418): [] sysvec_apic_timer_interrupt+0xe/0x90 [11117.747950] softirqs last enabled at (19984404): [] __do_softirq+0x2be/0x371 [11117.756636] softirqs last disabled at (19984399): [] irq_exit_rcu+0xb0/0xd0 [11117.765155] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11117.774533] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11117.782004] Workqueue: events delayed_work [ceph] [11117.786727] RIP: 0010:lock_acquire+0xf0/0x2b0 [11117.791080] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11117.809815] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11117.815033] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11117.822158] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11117.829281] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11117.836406] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11117.843530] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11117.850655] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11117.858731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11117.864467] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11117.871593] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11117.878716] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11117.885840] Call Trace: [11117.888286] [11117.890299] ? watchdog_timer_fn+0x1b4/0x220 [11117.894569] ? __hrtimer_run_queues+0x1d3/0x360 [11117.899095] ? hrtimer_interrupt+0x104/0x240 [11117.903356] ? __local_bh_enable+0x50/0x60 [11117.907450] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11117.912675] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11117.917632] [11117.919728] [11117.921826] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11117.927133] ? lock_acquire+0xf0/0x2b0 [11117.930883] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11117.936292] ? lock_release+0xc5/0x260 [11117.940037] _raw_spin_lock+0x30/0x40 [11117.943702] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11117.949100] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11117.954329] delayed_work+0x1d8/0x280 [ceph] [11117.958609] ? process_scheduled_works+0x22a/0x510 [11117.963392] process_scheduled_works+0x2a4/0x510 [11117.968006] worker_thread+0x188/0x340 [11117.971757] ? rescuer_thread+0x310/0x310 [11117.975758] kthread+0xf5/0x130 [11117.978898] ? kthread_complete_and_exit+0x20/0x20 [11117.983682] ret_from_fork+0x31/0x40 [11117.987251] ? kthread_complete_and_exit+0x20/0x20 [11117.992037] ret_from_fork_asm+0x11/0x20 [11117.995967] [11145.661784] watchdog: BUG: soft lockup - CPU#7 stuck for 8263s! [kworker/7:4:78808] [11145.669436] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11145.724883] irq event stamp: 20049472 [11145.728541] hardirqs last enabled at (20049471): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11145.738707] hardirqs last disabled at (20049472): [] sysvec_apic_timer_interrupt+0xe/0x90 [11145.748432] softirqs last enabled at (20049458): [] __do_softirq+0x2be/0x371 [11145.757126] softirqs last disabled at (20049453): [] irq_exit_rcu+0xb0/0xd0 [11145.765646] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11145.775024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11145.782494] Workqueue: events delayed_work [ceph] [11145.787216] RIP: 0010:lock_acquire+0xf0/0x2b0 [11145.791568] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11145.810306] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11145.815522] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11145.822645] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11145.829769] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11145.836893] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11145.844019] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11145.851142] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11145.859219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11145.864958] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11145.872082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11145.879205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11145.886330] Call Trace: [11145.888776] [11145.890786] ? watchdog_timer_fn+0x1b4/0x220 [11145.895050] ? __hrtimer_run_queues+0x1d3/0x360 [11145.899576] ? hrtimer_interrupt+0x104/0x240 [11145.903845] ? __local_bh_enable+0x50/0x60 [11145.907939] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11145.913163] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11145.918120] [11145.920217] [11145.922315] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11145.927622] ? lock_acquire+0xf0/0x2b0 [11145.931364] ? ceph_check_caps+0x26b/0xae0 [ceph] [11145.936077] ? lock_release+0xc5/0x260 [11145.939823] _raw_spin_lock+0x30/0x40 [11145.943480] ? ceph_check_caps+0x63/0xae0 [ceph] [11145.948099] ceph_check_caps+0x63/0xae0 [ceph] [11145.952551] ? __lock_acquire+0xbfa/0x1e20 [11145.956648] ? lock_acquire+0xcb/0x2b0 [11145.960397] ? igrab+0x1d/0x50 [11145.963456] ? find_held_lock+0x2d/0x90 [11145.967287] ? find_held_lock+0x2d/0x90 [11145.971117] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11145.976517] ? lock_release+0xc5/0x260 [11145.980263] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11145.985489] delayed_work+0x1d8/0x280 [ceph] [11145.989770] ? process_scheduled_works+0x22a/0x510 [11145.994552] process_scheduled_works+0x2a4/0x510 [11145.999168] worker_thread+0x188/0x340 [11146.002918] ? rescuer_thread+0x310/0x310 [11146.006922] kthread+0xf5/0x130 [11146.010059] ? kthread_complete_and_exit+0x20/0x20 [11146.014843] ret_from_fork+0x31/0x40 [11146.018423] ? kthread_complete_and_exit+0x20/0x20 [11146.023207] ret_from_fork_asm+0x11/0x20 [11146.027138] [11173.662278] watchdog: BUG: soft lockup - CPU#7 stuck for 8289s! [kworker/7:4:78808] [11173.669934] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11173.725383] irq event stamp: 20114718 [11173.729040] hardirqs last enabled at (20114717): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11173.739204] hardirqs last disabled at (20114718): [] sysvec_apic_timer_interrupt+0xe/0x90 [11173.748930] softirqs last enabled at (20114640): [] __do_softirq+0x2be/0x371 [11173.757614] softirqs last disabled at (20114635): [] irq_exit_rcu+0xb0/0xd0 [11173.766135] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11173.775512] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11173.782984] Workqueue: events delayed_work [ceph] [11173.787706] RIP: 0010:ceph_check_caps+0x1be/0xae0 [ceph] [11173.793028] Code: 00 0d fd bf 00 00 89 44 24 04 0f 1f 44 00 00 44 8b 64 24 50 45 85 e4 74 11 48 8b 44 24 08 44 8b 98 b0 01 00 00 45 85 db 74 11 <0f> b7 03 66 25 00 f0 66 3d 00 80 0f 84 17 04 00 00 48 8b 7c 24 48 [11173.811765] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [11173.816983] RAX: 0000000000000d55 RBX: ffff8881a1d34478 RCX: ffff88817f867000 [11173.824106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810da9e008 [11173.831229] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [11173.838353] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11173.845479] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 [11173.852603] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11173.860680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11173.866417] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11173.873541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11173.880667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11173.887791] Call Trace: [11173.890237] [11173.892247] ? watchdog_timer_fn+0x1b4/0x220 [11173.896517] ? __hrtimer_run_queues+0x1d3/0x360 [11173.901045] ? hrtimer_interrupt+0x104/0x240 [11173.905306] ? __local_bh_enable+0x50/0x60 [11173.909398] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11173.914622] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11173.919580] [11173.921676] [11173.923776] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11173.929083] ? ceph_check_caps+0x1be/0xae0 [ceph] [11173.933804] ? lock_acquire+0xcb/0x2b0 [11173.937547] ? igrab+0x1d/0x50 [11173.940605] ? find_held_lock+0x2d/0x90 [11173.944438] ? find_held_lock+0x2d/0x90 [11173.948268] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11173.953667] ? lock_release+0xc5/0x260 [11173.957413] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11173.962638] delayed_work+0x1d8/0x280 [ceph] [11173.966921] ? process_scheduled_works+0x22a/0x510 [11173.971712] process_scheduled_works+0x2a4/0x510 [11173.976326] worker_thread+0x188/0x340 [11173.980074] ? rescuer_thread+0x310/0x310 [11173.984078] kthread+0xf5/0x130 [11173.987217] ? kthread_complete_and_exit+0x20/0x20 [11173.992000] ret_from_fork+0x31/0x40 [11173.995571] ? kthread_complete_and_exit+0x20/0x20 [11174.000355] ret_from_fork_asm+0x11/0x20 [11174.004277] [11201.662784] watchdog: BUG: soft lockup - CPU#7 stuck for 8315s! [kworker/7:4:78808] [11201.670439] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11201.725889] irq event stamp: 20178420 [11201.729547] hardirqs last enabled at (20178419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11201.739713] hardirqs last disabled at (20178420): [] sysvec_apic_timer_interrupt+0xe/0x90 [11201.749446] softirqs last enabled at (20178294): [] __do_softirq+0x2be/0x371 [11201.758138] softirqs last disabled at (20178289): [] irq_exit_rcu+0xb0/0xd0 [11201.766658] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11201.776035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11201.783506] Workqueue: events delayed_work [ceph] [11201.788229] RIP: 0010:lock_acquire+0xf0/0x2b0 [11201.792580] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11201.811317] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11201.816534] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11201.823659] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11201.830783] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11201.837906] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11201.845031] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11201.852155] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11201.860234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11201.865969] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11201.873095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11201.880219] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11201.887344] Call Trace: [11201.889789] [11201.891801] ? watchdog_timer_fn+0x1b4/0x220 [11201.896070] ? __hrtimer_run_queues+0x1d3/0x360 [11201.900598] ? hrtimer_interrupt+0x104/0x240 [11201.904866] ? __local_bh_enable+0x50/0x60 [11201.908959] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11201.914185] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11201.919143] [11201.921239] [11201.923336] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11201.928644] ? lock_acquire+0xf0/0x2b0 [11201.932394] ? ceph_check_caps+0x26b/0xae0 [ceph] [11201.937109] ? lock_release+0xc5/0x260 [11201.940854] _raw_spin_lock+0x30/0x40 [11201.944518] ? ceph_check_caps+0x63/0xae0 [ceph] [11201.949138] ceph_check_caps+0x63/0xae0 [ceph] [11201.953605] ? __lock_acquire+0xbfa/0x1e20 [11201.957705] ? lock_acquire+0xcb/0x2b0 [11201.961454] ? igrab+0x1d/0x50 [11201.964504] ? find_held_lock+0x2d/0x90 [11201.968344] ? find_held_lock+0x2d/0x90 [11201.972173] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11201.977575] ? lock_release+0xc5/0x260 [11201.981320] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11201.986545] delayed_work+0x1d8/0x280 [ceph] [11201.990828] ? process_scheduled_works+0x22a/0x510 [11201.995618] process_scheduled_works+0x2a4/0x510 [11202.000232] worker_thread+0x188/0x340 [11202.003981] ? rescuer_thread+0x310/0x310 [11202.007985] kthread+0xf5/0x130 [11202.011123] ? kthread_complete_and_exit+0x20/0x20 [11202.015907] ret_from_fork+0x31/0x40 [11202.019487] ? kthread_complete_and_exit+0x20/0x20 [11202.024270] ret_from_fork_asm+0x11/0x20 [11202.028193] [11229.663268] watchdog: BUG: soft lockup - CPU#7 stuck for 8341s! [kworker/7:4:78808] [11229.670947] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11229.726396] irq event stamp: 20242490 [11229.730053] hardirqs last enabled at (20242489): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11229.740219] hardirqs last disabled at (20242490): [] sysvec_apic_timer_interrupt+0xe/0x90 [11229.749943] softirqs last enabled at (20242412): [] __do_softirq+0x2be/0x371 [11229.758637] softirqs last disabled at (20242407): [] irq_exit_rcu+0xb0/0xd0 [11229.767156] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11229.776533] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11229.784004] Workqueue: events delayed_work [ceph] [11229.788727] RIP: 0010:lock_acquire+0xf0/0x2b0 [11229.793079] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11229.811815] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11229.817032] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11229.824158] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11229.831280] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11229.838404] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11229.845530] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11229.852652] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11229.860729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11229.866468] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11229.873592] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11229.880717] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11229.887839] Call Trace: [11229.890286] [11229.892296] ? watchdog_timer_fn+0x1b4/0x220 [11229.896561] ? __hrtimer_run_queues+0x1d3/0x360 [11229.901096] ? hrtimer_interrupt+0x104/0x240 [11229.905365] ? __local_bh_enable+0x50/0x60 [11229.909457] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11229.914682] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11229.919640] [11229.921735] [11229.923833] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11229.929143] ? lock_acquire+0xf0/0x2b0 [11229.932893] ? ceph_check_caps+0x26b/0xae0 [ceph] [11229.937606] ? lock_release+0xc5/0x260 [11229.941350] _raw_spin_lock+0x30/0x40 [11229.945009] ? ceph_check_caps+0x63/0xae0 [ceph] [11229.949626] ceph_check_caps+0x63/0xae0 [ceph] [11229.954078] ? __lock_acquire+0xbfa/0x1e20 [11229.958176] ? lock_acquire+0xcb/0x2b0 [11229.961925] ? igrab+0x1d/0x50 [11229.964975] ? find_held_lock+0x2d/0x90 [11229.968806] ? find_held_lock+0x2d/0x90 [11229.972637] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11229.978037] ? lock_release+0xc5/0x260 [11229.981782] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11229.987009] delayed_work+0x1d8/0x280 [ceph] [11229.991291] ? process_scheduled_works+0x22a/0x510 [11229.996080] process_scheduled_works+0x2a4/0x510 [11230.000696] worker_thread+0x188/0x340 [11230.004445] ? rescuer_thread+0x310/0x310 [11230.008448] kthread+0xf5/0x130 [11230.011587] ? kthread_complete_and_exit+0x20/0x20 [11230.016371] ret_from_fork+0x31/0x40 [11230.019941] ? kthread_complete_and_exit+0x20/0x20 [11230.024725] ret_from_fork_asm+0x11/0x20 [11230.028648] [11257.663763] watchdog: BUG: soft lockup - CPU#7 stuck for 8367s! [kworker/7:4:78808] [11257.671418] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11257.726867] irq event stamp: 20305266 [11257.730523] hardirqs last enabled at (20305265): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11257.740691] hardirqs last disabled at (20305266): [] sysvec_apic_timer_interrupt+0xe/0x90 [11257.750423] softirqs last enabled at (20305188): [] __do_softirq+0x2be/0x371 [11257.759116] softirqs last disabled at (20305183): [] irq_exit_rcu+0xb0/0xd0 [11257.767638] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11257.777022] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11257.784493] Workqueue: events delayed_work [ceph] [11257.789216] RIP: 0010:ceph_check_caps+0x31a/0xae0 [ceph] [11257.794538] Code: d5 41 23 6f 3c 0f 1f 44 00 00 85 ed 74 1c 41 85 ec 0f 84 0b 06 00 00 41 0f b7 45 00 66 25 00 f0 66 3d 00 80 0f 84 57 03 00 00 <41> 8b 57 38 49 39 9d b0 06 00 00 0f 84 d0 01 00 00 41 8b 47 44 89 [11257.813275] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000246 [11257.818492] RAX: ffff88810da9e000 RBX: ffff88810da9e000 RCX: ffff88817f867000 [11257.825615] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34b20 [11257.832741] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [11257.839866] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11257.846988] R13: ffff8881a1d34478 R14: 0000000000000000 R15: ffff88810da9e008 [11257.854112] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11257.862191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11257.867928] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11257.875053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11257.882177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11257.889301] Call Trace: [11257.891746] [11257.893756] ? watchdog_timer_fn+0x1b4/0x220 [11257.898020] ? __hrtimer_run_queues+0x1d3/0x360 [11257.902547] ? hrtimer_interrupt+0x104/0x240 [11257.906815] ? __local_bh_enable+0x50/0x60 [11257.910910] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11257.916133] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11257.921091] [11257.923199] [11257.925293] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11257.930601] ? ceph_check_caps+0x31a/0xae0 [ceph] [11257.935323] ? lock_acquire+0xcb/0x2b0 [11257.939066] ? igrab+0x1d/0x50 [11257.942125] ? find_held_lock+0x2d/0x90 [11257.945956] ? find_held_lock+0x2d/0x90 [11257.949786] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11257.955185] ? lock_release+0xc5/0x260 [11257.958933] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11257.964167] delayed_work+0x1d8/0x280 [ceph] [11257.968450] ? process_scheduled_works+0x22a/0x510 [11257.973238] process_scheduled_works+0x2a4/0x510 [11257.977856] worker_thread+0x188/0x340 [11257.981604] ? rescuer_thread+0x310/0x310 [11257.985606] kthread+0xf5/0x130 [11257.988745] ? kthread_complete_and_exit+0x20/0x20 [11257.993528] ret_from_fork+0x31/0x40 [11257.997108] ? kthread_complete_and_exit+0x20/0x20 [11258.001891] ret_from_fork_asm+0x11/0x20 [11258.005814] [11285.664266] watchdog: BUG: soft lockup - CPU#7 stuck for 8393s! [kworker/7:4:78808] [11285.671914] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11285.727364] irq event stamp: 20369012 [11285.731021] hardirqs last enabled at (20369011): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11285.741187] hardirqs last disabled at (20369012): [] sysvec_apic_timer_interrupt+0xe/0x90 [11285.750911] softirqs last enabled at (20368886): [] __do_softirq+0x2be/0x371 [11285.759597] softirqs last disabled at (20368881): [] irq_exit_rcu+0xb0/0xd0 [11285.768117] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11285.777494] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11285.784965] Workqueue: events delayed_work [ceph] [11285.789688] RIP: 0010:lock_acquire+0xf0/0x2b0 [11285.794038] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11285.812775] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11285.817994] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11285.825118] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11285.832242] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11285.839366] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11285.846490] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11285.853613] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11285.861692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11285.867430] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11285.874554] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11285.881676] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11285.888802] Call Trace: [11285.891246] [11285.893257] ? watchdog_timer_fn+0x1b4/0x220 [11285.897522] ? __hrtimer_run_queues+0x1d3/0x360 [11285.902056] ? hrtimer_interrupt+0x104/0x240 [11285.906318] ? __local_bh_enable+0x50/0x60 [11285.910410] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11285.915633] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11285.920591] [11285.922688] [11285.924787] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11285.930093] ? lock_acquire+0xf0/0x2b0 [11285.933844] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11285.939251] ? lock_release+0xc5/0x260 [11285.942997] _raw_spin_lock+0x30/0x40 [11285.946653] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11285.952053] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11285.957283] delayed_work+0x1d8/0x280 [ceph] [11285.961563] ? process_scheduled_works+0x22a/0x510 [11285.966351] process_scheduled_works+0x2a4/0x510 [11285.970968] worker_thread+0x188/0x340 [11285.974718] ? rescuer_thread+0x310/0x310 [11285.978720] kthread+0xf5/0x130 [11285.981858] ? kthread_complete_and_exit+0x20/0x20 [11285.986642] ret_from_fork+0x31/0x40 [11285.990221] ? kthread_complete_and_exit+0x20/0x20 [11285.995005] ret_from_fork_asm+0x11/0x20 [11285.998937] [11313.664775] watchdog: BUG: soft lockup - CPU#7 stuck for 8419s! [kworker/7:4:78808] [11313.672432] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11313.727881] irq event stamp: 20432448 [11313.731537] hardirqs last enabled at (20432447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11313.741702] hardirqs last disabled at (20432448): [] sysvec_apic_timer_interrupt+0xe/0x90 [11313.751428] softirqs last enabled at (20432434): [] __do_softirq+0x2be/0x371 [11313.760120] softirqs last disabled at (20432429): [] irq_exit_rcu+0xb0/0xd0 [11313.768641] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11313.778017] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11313.785487] Workqueue: events delayed_work [ceph] [11313.790211] RIP: 0010:lock_release+0x162/0x260 [11313.794649] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [11313.813386] RSP: 0018:ffffc90002117b78 EFLAGS: 00000246 [11313.818604] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117b84 [11313.825727] RDX: 0000000000000002 RSI: ffff8881a1d34938 RDI: ffff88810debb528 [11313.832853] RBP: ffff8881a1d34938 R08: 0000000000000000 R09: 0000000000000000 [11313.839976] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0b3407b [11313.847099] R13: 0000000000000246 R14: 0000000000000003 R15: 0000000000000000 [11313.854225] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11313.862302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11313.868039] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11313.875164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11313.882287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11313.889411] Call Trace: [11313.891858] [11313.893869] ? watchdog_timer_fn+0x1b4/0x220 [11313.898132] ? __hrtimer_run_queues+0x1d3/0x360 [11313.902656] ? hrtimer_interrupt+0x104/0x240 [11313.906918] ? __local_bh_enable+0x50/0x60 [11313.911012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11313.916237] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11313.921193] [11313.923291] [11313.925389] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11313.930693] ? ceph_check_caps+0x26b/0xae0 [ceph] [11313.935409] ? lock_release+0x162/0x260 [11313.939238] ? lock_release+0xc5/0x260 [11313.942982] _raw_spin_unlock+0x1b/0x40 [11313.946812] ceph_check_caps+0x26b/0xae0 [ceph] [11313.951355] ? lock_acquire+0xcb/0x2b0 [11313.955099] ? igrab+0x1d/0x50 [11313.958149] ? find_held_lock+0x2d/0x90 [11313.961979] ? find_held_lock+0x2d/0x90 [11313.965811] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11313.971211] ? lock_release+0xc5/0x260 [11313.974954] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11313.980184] delayed_work+0x1d8/0x280 [ceph] [11313.984464] ? process_scheduled_works+0x22a/0x510 [11313.989252] process_scheduled_works+0x2a4/0x510 [11313.993868] worker_thread+0x188/0x340 [11313.997620] ? rescuer_thread+0x310/0x310 [11314.001630] kthread+0xf5/0x130 [11314.004769] ? kthread_complete_and_exit+0x20/0x20 [11314.009552] ret_from_fork+0x31/0x40 [11314.013132] ? kthread_complete_and_exit+0x20/0x20 [11314.017915] ret_from_fork_asm+0x11/0x20 [11314.021839] [11341.665248] watchdog: BUG: soft lockup - CPU#7 stuck for 8445s! [kworker/7:4:78808] [11341.672918] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11341.728368] irq event stamp: 20497120 [11341.732025] hardirqs last enabled at (20497119): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11341.742192] hardirqs last disabled at (20497120): [] sysvec_apic_timer_interrupt+0xe/0x90 [11341.751925] softirqs last enabled at (20497024): [] __do_softirq+0x2be/0x371 [11341.760617] softirqs last disabled at (20497019): [] irq_exit_rcu+0xb0/0xd0 [11341.769129] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11341.778508] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11341.785977] Workqueue: events delayed_work [ceph] [11341.790701] RIP: 0010:lock_acquire+0x86/0x2b0 [11341.795061] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [11341.813798] RSP: 0018:ffffc90002117d30 EFLAGS: 00000246 [11341.819015] RAX: 0000000000000246 RBX: ffff88817f867488 RCX: 0000000000000000 [11341.826138] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f8674a0 [11341.833262] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [11341.840387] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [11341.847510] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11341.854636] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11341.862714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11341.868449] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11341.875575] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11341.882698] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11341.889822] Call Trace: [11341.892268] [11341.894280] ? watchdog_timer_fn+0x1b4/0x220 [11341.898552] ? __hrtimer_run_queues+0x1d3/0x360 [11341.903086] ? hrtimer_interrupt+0x104/0x240 [11341.907356] ? __local_bh_enable+0x50/0x60 [11341.911448] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11341.916673] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11341.921629] [11341.923726] [11341.925825] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11341.931132] ? lock_acquire+0x86/0x2b0 [11341.934875] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11341.940282] ? lock_release+0xc5/0x260 [11341.944028] _raw_spin_lock+0x30/0x40 [11341.947692] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11341.953091] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11341.958322] delayed_work+0x1d8/0x280 [ceph] [11341.962602] ? process_scheduled_works+0x22a/0x510 [11341.967391] process_scheduled_works+0x2a4/0x510 [11341.972007] worker_thread+0x188/0x340 [11341.975754] ? rescuer_thread+0x310/0x310 [11341.979760] kthread+0xf5/0x130 [11341.982898] ? kthread_complete_and_exit+0x20/0x20 [11341.987680] ret_from_fork+0x31/0x40 [11341.991250] ? kthread_complete_and_exit+0x20/0x20 [11341.996034] ret_from_fork_asm+0x11/0x20 [11341.999959] [11369.665742] watchdog: BUG: soft lockup - CPU#7 stuck for 8471s! [kworker/7:4:78808] [11369.673390] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11369.728840] irq event stamp: 20561786 [11369.732497] hardirqs last enabled at (20561785): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11369.742665] hardirqs last disabled at (20561786): [] sysvec_apic_timer_interrupt+0xe/0x90 [11369.752396] softirqs last enabled at (20561772): [] __do_softirq+0x2be/0x371 [11369.761082] softirqs last disabled at (20561767): [] irq_exit_rcu+0xb0/0xd0 [11369.769602] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11369.778979] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11369.786449] Workqueue: events delayed_work [ceph] [11369.791172] RIP: 0010:lock_acquire+0x85/0x2b0 [11369.795524] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [11369.814259] RSP: 0018:ffffc90002117d28 EFLAGS: 00000246 [11369.819477] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 0000000000000000 [11369.826603] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f8674a0 [11369.833726] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [11369.840849] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [11369.847975] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11369.855098] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11369.863174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11369.868912] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11369.876038] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11369.883161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11369.890285] Call Trace: [11369.892733] [11369.894743] ? watchdog_timer_fn+0x1b4/0x220 [11369.899014] ? __hrtimer_run_queues+0x1d3/0x360 [11369.903539] ? hrtimer_interrupt+0x104/0x240 [11369.907802] ? __local_bh_enable+0x50/0x60 [11369.911893] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11369.917118] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11369.922076] [11369.924173] [11369.926272] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11369.931578] ? lock_acquire+0x85/0x2b0 [11369.935328] ? find_held_lock+0x2d/0x90 [11369.939168] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11369.944576] ? lock_release+0xc5/0x260 [11369.948321] _raw_spin_lock+0x30/0x40 [11369.951977] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11369.957375] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11369.962606] delayed_work+0x1d8/0x280 [ceph] [11369.966885] ? process_scheduled_works+0x22a/0x510 [11369.971675] process_scheduled_works+0x2a4/0x510 [11369.976293] worker_thread+0x188/0x340 [11369.980041] ? rescuer_thread+0x310/0x310 [11369.984042] kthread+0xf5/0x130 [11369.987183] ? kthread_complete_and_exit+0x20/0x20 [11369.991966] ret_from_fork+0x31/0x40 [11369.995545] ? kthread_complete_and_exit+0x20/0x20 [11370.000328] ret_from_fork_asm+0x11/0x20 [11370.004253] [11397.666244] watchdog: BUG: soft lockup - CPU#7 stuck for 8497s! [kworker/7:4:78808] [11397.673897] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11397.729347] irq event stamp: 20624980 [11397.733003] hardirqs last enabled at (20624979): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11397.743169] hardirqs last disabled at (20624980): [] sysvec_apic_timer_interrupt+0xe/0x90 [11397.752894] softirqs last enabled at (20624974): [] __do_softirq+0x2be/0x371 [11397.761579] softirqs last disabled at (20624969): [] irq_exit_rcu+0xb0/0xd0 [11397.770100] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11397.779485] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11397.786956] Workqueue: events delayed_work [ceph] [11397.791677] RIP: 0010:rcu_is_watching+0x1f/0x40 [11397.796203] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [11397.814940] RSP: 0018:ffffc90002117d58 EFLAGS: 00000202 [11397.820157] RAX: 00000000007c1919 RBX: ffff88817f867488 RCX: ffffc90002117d54 [11397.827281] RDX: 0000000000000003 RSI: ffffffffa0b399eb RDI: ffff88817f8674a0 [11397.834405] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [11397.841530] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [11397.848653] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [11397.855777] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11397.863855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11397.869594] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11397.876718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11397.883841] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11397.890966] Call Trace: [11397.893411] [11397.895422] ? watchdog_timer_fn+0x1b4/0x220 [11397.899685] ? __hrtimer_run_queues+0x1d3/0x360 [11397.904212] ? hrtimer_interrupt+0x104/0x240 [11397.908482] ? __local_bh_enable+0x50/0x60 [11397.912574] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11397.917798] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11397.922756] [11397.924853] [11397.926951] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11397.932255] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11397.937665] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11397.943071] ? rcu_is_watching+0x1f/0x40 [11397.946988] lock_release+0x1d7/0x260 [11397.950647] _raw_spin_unlock+0x1b/0x40 [11397.954477] ceph_check_delayed_caps+0x19b/0x270 [ceph] [11397.959706] delayed_work+0x1d8/0x280 [ceph] [11397.963986] ? process_scheduled_works+0x22a/0x510 [11397.968775] process_scheduled_works+0x2a4/0x510 [11397.973392] worker_thread+0x188/0x340 [11397.977141] ? rescuer_thread+0x310/0x310 [11397.981144] kthread+0xf5/0x130 [11397.984282] ? kthread_complete_and_exit+0x20/0x20 [11397.989065] ret_from_fork+0x31/0x40 [11397.992647] ? kthread_complete_and_exit+0x20/0x20 [11397.997430] ret_from_fork_asm+0x11/0x20 [11398.001361] [11425.666732] watchdog: BUG: soft lockup - CPU#7 stuck for 8523s! [kworker/7:4:78808] [11425.674404] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11425.729853] irq event stamp: 20688526 [11425.733509] hardirqs last enabled at (20688525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11425.743675] hardirqs last disabled at (20688526): [] sysvec_apic_timer_interrupt+0xe/0x90 [11425.753406] softirqs last enabled at (20688512): [] __do_softirq+0x2be/0x371 [11425.762092] softirqs last disabled at (20688507): [] irq_exit_rcu+0xb0/0xd0 [11425.770607] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11425.779992] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11425.787462] Workqueue: events delayed_work [ceph] [11425.792185] RIP: 0010:lock_acquire+0xf0/0x2b0 [11425.796543] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11425.815281] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11425.820498] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11425.827623] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11425.834747] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11425.841871] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11425.848996] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11425.856119] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11425.864197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11425.869935] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11425.877060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11425.884182] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11425.891308] Call Trace: [11425.893752] [11425.895763] ? watchdog_timer_fn+0x1b4/0x220 [11425.900026] ? __hrtimer_run_queues+0x1d3/0x360 [11425.904554] ? hrtimer_interrupt+0x104/0x240 [11425.908821] ? __local_bh_enable+0x50/0x60 [11425.912914] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11425.918141] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11425.923096] [11425.925195] [11425.927292] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11425.932600] ? lock_acquire+0xf0/0x2b0 [11425.936349] ? ceph_check_caps+0x26b/0xae0 [ceph] [11425.941063] ? lock_release+0xc5/0x260 [11425.944809] _raw_spin_lock+0x30/0x40 [11425.948466] ? ceph_check_caps+0x63/0xae0 [ceph] [11425.953084] ceph_check_caps+0x63/0xae0 [ceph] [11425.957535] ? __lock_acquire+0xbfa/0x1e20 [11425.961633] ? lock_acquire+0xcb/0x2b0 [11425.965373] ? igrab+0x1d/0x50 [11425.968424] ? find_held_lock+0x2d/0x90 [11425.972257] ? find_held_lock+0x2d/0x90 [11425.976085] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11425.981487] ? lock_release+0xc5/0x260 [11425.985232] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11425.990467] delayed_work+0x1d8/0x280 [ceph] [11425.994748] ? process_scheduled_works+0x22a/0x510 [11425.999539] process_scheduled_works+0x2a4/0x510 [11426.004153] worker_thread+0x188/0x340 [11426.007903] ? rescuer_thread+0x310/0x310 [11426.011905] kthread+0xf5/0x130 [11426.015045] ? kthread_complete_and_exit+0x20/0x20 [11426.019827] ret_from_fork+0x31/0x40 [11426.023400] ? kthread_complete_and_exit+0x20/0x20 [11426.028182] ret_from_fork_asm+0x11/0x20 [11426.032107] [11453.667228] watchdog: BUG: soft lockup - CPU#7 stuck for 8550s! [kworker/7:4:78808] [11453.674883] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11453.730335] irq event stamp: 20751890 [11453.733990] hardirqs last enabled at (20751889): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11453.744157] hardirqs last disabled at (20751890): [] sysvec_apic_timer_interrupt+0xe/0x90 [11453.753881] softirqs last enabled at (20751812): [] __do_softirq+0x2be/0x371 [11453.762574] softirqs last disabled at (20751807): [] irq_exit_rcu+0xb0/0xd0 [11453.771096] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11453.780479] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11453.787950] Workqueue: events delayed_work [ceph] [11453.792674] RIP: 0010:lock_acquire+0xf0/0x2b0 [11453.797026] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11453.815761] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11453.820978] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11453.828103] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11453.835228] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11453.842352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11453.849475] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11453.856600] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11453.864678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11453.870414] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11453.877540] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11453.884662] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11453.891787] Call Trace: [11453.894234] [11453.896245] ? watchdog_timer_fn+0x1b4/0x220 [11453.900516] ? __hrtimer_run_queues+0x1d3/0x360 [11453.905051] ? hrtimer_interrupt+0x104/0x240 [11453.909321] ? __local_bh_enable+0x50/0x60 [11453.913413] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11453.918638] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11453.923594] [11453.925691] [11453.927790] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11453.933097] ? lock_acquire+0xf0/0x2b0 [11453.936847] ? ceph_check_caps+0x26b/0xae0 [ceph] [11453.941562] ? lock_release+0xc5/0x260 [11453.945307] _raw_spin_lock+0x30/0x40 [11453.948971] ? ceph_check_caps+0x63/0xae0 [ceph] [11453.953591] ceph_check_caps+0x63/0xae0 [ceph] [11453.958043] ? __lock_acquire+0xbfa/0x1e20 [11453.962139] ? lock_acquire+0xcb/0x2b0 [11453.965881] ? igrab+0x1d/0x50 [11453.968932] ? find_held_lock+0x2d/0x90 [11453.972771] ? find_held_lock+0x2d/0x90 [11453.976600] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11453.982000] ? lock_release+0xc5/0x260 [11453.985747] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11453.990973] delayed_work+0x1d8/0x280 [ceph] [11453.995253] ? process_scheduled_works+0x22a/0x510 [11454.000035] process_scheduled_works+0x2a4/0x510 [11454.004651] worker_thread+0x188/0x340 [11454.008401] ? rescuer_thread+0x310/0x310 [11454.012403] kthread+0xf5/0x130 [11454.015543] ? kthread_complete_and_exit+0x20/0x20 [11454.020326] ret_from_fork+0x31/0x40 [11454.023905] ? kthread_complete_and_exit+0x20/0x20 [11454.028690] ret_from_fork_asm+0x11/0x20 [11454.032619] [11481.667724] watchdog: BUG: soft lockup - CPU#7 stuck for 8576s! [kworker/7:4:78808] [11481.675374] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11481.730823] irq event stamp: 20814232 [11481.734478] hardirqs last enabled at (20814231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11481.744645] hardirqs last disabled at (20814232): [] sysvec_apic_timer_interrupt+0xe/0x90 [11481.754370] softirqs last enabled at (20814154): [] __do_softirq+0x2be/0x371 [11481.763064] softirqs last disabled at (20814149): [] irq_exit_rcu+0xb0/0xd0 [11481.771584] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11481.780959] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11481.788431] Workqueue: events delayed_work [ceph] [11481.793154] RIP: 0010:_atomic_dec_and_lock+0x14/0x40 [11481.798112] Code: 63 c9 48 83 f9 07 76 ba b8 08 00 00 00 eb d2 44 89 d0 5b c3 66 90 f3 0f 1e fa 55 53 8b 07 83 f8 01 74 0e 8d 50 ff f0 0f b1 17 <75> f2 31 c0 5b 5d c3 48 89 fb 48 89 f7 48 89 f5 e8 07 2b 04 00 f0 [11481.816849] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [11481.822066] RAX: 0000000000000002 RBX: ffff8881a1d34478 RCX: ffffc90002117b84 [11481.829191] RDX: 0000000000000001 RSI: ffff8881a1d34500 RDI: ffff8881a1d34680 [11481.836315] RBP: ffff8881a1d34500 R08: 0000000000000000 R09: 0000000000000000 [11481.843437] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881a1d34680 [11481.850563] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [11481.857687] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11481.865765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11481.871500] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11481.878626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11481.885750] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11481.892875] Call Trace: [11481.895320] [11481.897331] ? watchdog_timer_fn+0x1b4/0x220 [11481.901603] ? __hrtimer_run_queues+0x1d3/0x360 [11481.906130] ? hrtimer_interrupt+0x104/0x240 [11481.910399] ? __local_bh_enable+0x50/0x60 [11481.914491] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11481.919715] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11481.924675] [11481.926769] [11481.928869] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11481.934176] ? _atomic_dec_and_lock+0x14/0x40 [11481.938532] iput+0x50/0x220 [11481.941419] ceph_check_delayed_caps+0x1be/0x270 [ceph] [11481.946655] delayed_work+0x1d8/0x280 [ceph] [11481.950936] ? process_scheduled_works+0x22a/0x510 [11481.955727] process_scheduled_works+0x2a4/0x510 [11481.960343] worker_thread+0x188/0x340 [11481.964092] ? rescuer_thread+0x310/0x310 [11481.968095] kthread+0xf5/0x130 [11481.971241] ? kthread_complete_and_exit+0x20/0x20 [11481.976024] ret_from_fork+0x31/0x40 [11481.979604] ? kthread_complete_and_exit+0x20/0x20 [11481.984388] ret_from_fork_asm+0x11/0x20 [11481.988312] [11509.668229] watchdog: BUG: soft lockup - CPU#7 stuck for 8602s! [kworker/7:4:78808] [11509.675880] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11509.731330] irq event stamp: 20877418 [11509.734986] hardirqs last enabled at (20877417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11509.745152] hardirqs last disabled at (20877418): [] sysvec_apic_timer_interrupt+0xe/0x90 [11509.754877] softirqs last enabled at (20877376): [] __do_softirq+0x2be/0x371 [11509.763570] softirqs last disabled at (20877371): [] irq_exit_rcu+0xb0/0xd0 [11509.772090] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11509.781465] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11509.788938] Workqueue: events delayed_work [ceph] [11509.793660] RIP: 0010:lock_acquire+0xf0/0x2b0 [11509.798011] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11509.816749] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11509.821965] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11509.829091] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11509.836213] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11509.843338] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11509.850463] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11509.857586] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11509.865663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11509.871402] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11509.878527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11509.885649] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11509.892773] Call Trace: [11509.895220] [11509.897230] ? watchdog_timer_fn+0x1b4/0x220 [11509.901494] ? __hrtimer_run_queues+0x1d3/0x360 [11509.906020] ? hrtimer_interrupt+0x104/0x240 [11509.910289] ? __local_bh_enable+0x50/0x60 [11509.914383] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11509.919614] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11509.924572] [11509.926670] [11509.928767] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11509.934075] ? lock_acquire+0xf0/0x2b0 [11509.937826] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11509.943234] ? lock_release+0xc5/0x260 [11509.946977] _raw_spin_lock+0x30/0x40 [11509.950634] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11509.956035] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11509.961263] delayed_work+0x1d8/0x280 [ceph] [11509.965543] ? process_scheduled_works+0x22a/0x510 [11509.970332] process_scheduled_works+0x2a4/0x510 [11509.974950] worker_thread+0x188/0x340 [11509.978698] ? rescuer_thread+0x310/0x310 [11509.982701] kthread+0xf5/0x130 [11509.985840] ? kthread_complete_and_exit+0x20/0x20 [11509.990623] ret_from_fork+0x31/0x40 [11509.994202] ? kthread_complete_and_exit+0x20/0x20 [11509.998986] ret_from_fork_asm+0x11/0x20 [11510.002909] [11537.668712] watchdog: BUG: soft lockup - CPU#7 stuck for 8628s! [kworker/7:4:78808] [11537.676368] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11537.731817] irq event stamp: 20940082 [11537.735475] hardirqs last enabled at (20940081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11537.745641] hardirqs last disabled at (20940082): [] sysvec_apic_timer_interrupt+0xe/0x90 [11537.755374] softirqs last enabled at (20940068): [] __do_softirq+0x2be/0x371 [11537.764067] softirqs last disabled at (20940063): [] irq_exit_rcu+0xb0/0xd0 [11537.772578] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11537.781956] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11537.789426] Workqueue: events delayed_work [ceph] [11537.794149] RIP: 0010:lock_acquire+0xf0/0x2b0 [11537.798501] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11537.817236] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [11537.822456] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [11537.829580] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [11537.836703] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [11537.843828] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [11537.850953] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11537.858075] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11537.866152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11537.871891] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11537.879014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11537.886139] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11537.893262] Call Trace: [11537.895709] [11537.897721] ? watchdog_timer_fn+0x1b4/0x220 [11537.901990] ? __hrtimer_run_queues+0x1d3/0x360 [11537.906519] ? hrtimer_interrupt+0x104/0x240 [11537.910786] ? __local_bh_enable+0x50/0x60 [11537.914880] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11537.920104] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11537.925062] [11537.927160] [11537.929257] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11537.934564] ? lock_acquire+0xf0/0x2b0 [11537.938315] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11537.943723] _raw_spin_lock+0x30/0x40 [11537.947379] ? igrab+0x1d/0x50 [11537.950430] igrab+0x1d/0x50 [11537.953310] ceph_check_delayed_caps+0xb7/0x270 [ceph] [11537.958458] delayed_work+0x1d8/0x280 [ceph] [11537.962739] ? process_scheduled_works+0x22a/0x510 [11537.967528] process_scheduled_works+0x2a4/0x510 [11537.972144] worker_thread+0x188/0x340 [11537.975894] ? rescuer_thread+0x310/0x310 [11537.979897] kthread+0xf5/0x130 [11537.983036] ? kthread_complete_and_exit+0x20/0x20 [11537.987820] ret_from_fork+0x31/0x40 [11537.991398] ? kthread_complete_and_exit+0x20/0x20 [11537.996182] ret_from_fork_asm+0x11/0x20 [11538.000106] [11565.669227] watchdog: BUG: soft lockup - CPU#7 stuck for 8654s! [kworker/7:4:78808] [11565.676885] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11565.732333] irq event stamp: 21001790 [11565.735992] hardirqs last enabled at (21001789): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11565.746156] hardirqs last disabled at (21001790): [] sysvec_apic_timer_interrupt+0xe/0x90 [11565.755879] softirqs last enabled at (21001770): [] __do_softirq+0x2be/0x371 [11565.764564] softirqs last disabled at (21001765): [] irq_exit_rcu+0xb0/0xd0 [11565.773077] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11565.782453] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11565.789925] Workqueue: events delayed_work [ceph] [11565.794647] RIP: 0010:__ceph_caps_used+0x9/0x90 [ceph] [11565.799787] Code: 00 00 ff 70 28 41 56 e8 55 33 ae e0 44 89 e0 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e c3 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 <44> 8b 97 80 08 00 00 44 8b 9f 84 08 00 00 31 c0 45 85 d2 0f 95 c0 [11565.818523] RSP: 0018:ffffc90002117bc8 EFLAGS: 00000246 [11565.823742] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: 0000000000000000 [11565.830867] RDX: 0000000100aaf6c0 RSI: 0000000100abe120 RDI: ffff8881a1d34478 [11565.837990] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [11565.845115] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [11565.852238] R13: 0000000000000000 R14: 00000001001d239b R15: ffff888100ee2000 [11565.859363] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11565.867440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11565.873176] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11565.880302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11565.887424] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11565.894550] Call Trace: [11565.896997] [11565.899015] ? watchdog_timer_fn+0x1b4/0x220 [11565.903287] ? __hrtimer_run_queues+0x1d3/0x360 [11565.907814] ? hrtimer_interrupt+0x104/0x240 [11565.912082] ? __local_bh_enable+0x50/0x60 [11565.916176] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11565.921400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11565.926357] [11565.928453] [11565.930551] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11565.935861] ? __ceph_caps_used+0x9/0x90 [ceph] [11565.940398] ceph_check_caps+0x13d/0xae0 [ceph] [11565.944940] ? lock_acquire+0xcb/0x2b0 [11565.948684] ? igrab+0x1d/0x50 [11565.951733] ? find_held_lock+0x2d/0x90 [11565.955564] ? find_held_lock+0x2d/0x90 [11565.959395] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11565.964795] ? lock_release+0xc5/0x260 [11565.968541] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11565.973775] delayed_work+0x1d8/0x280 [ceph] [11565.978057] ? process_scheduled_works+0x22a/0x510 [11565.982846] process_scheduled_works+0x2a4/0x510 [11565.987464] worker_thread+0x188/0x340 [11565.991213] ? rescuer_thread+0x310/0x310 [11565.995222] kthread+0xf5/0x130 [11565.998362] ? kthread_complete_and_exit+0x20/0x20 [11566.003147] ret_from_fork+0x31/0x40 [11566.006725] ? kthread_complete_and_exit+0x20/0x20 [11566.011509] ret_from_fork_asm+0x11/0x20 [11566.015431] [11580.229887] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11590.469640] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11593.669701] watchdog: BUG: soft lockup - CPU#7 stuck for 8680s! [kworker/7:4:78808] [11593.677356] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11593.732805] irq event stamp: 21063210 [11593.736461] hardirqs last enabled at (21063209): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11593.746629] hardirqs last disabled at (21063210): [] sysvec_apic_timer_interrupt+0xe/0x90 [11593.756360] softirqs last enabled at (21063094): [] __do_softirq+0x2be/0x371 [11593.765044] softirqs last disabled at (21063089): [] irq_exit_rcu+0xb0/0xd0 [11593.773556] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11593.782934] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11593.790406] Workqueue: events delayed_work [ceph] [11593.795128] RIP: 0010:lock_acquire+0xf0/0x2b0 [11593.799480] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11593.818216] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11593.823433] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11593.830558] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11593.837682] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11593.844806] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11593.851930] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11593.859055] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11593.867132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11593.872867] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11593.879992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11593.887117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11593.894242] Call Trace: [11593.896687] [11593.898698] ? watchdog_timer_fn+0x1b4/0x220 [11593.902961] ? __hrtimer_run_queues+0x1d3/0x360 [11593.907487] ? hrtimer_interrupt+0x104/0x240 [11593.911749] ? __local_bh_enable+0x50/0x60 [11593.915841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11593.921065] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11593.926022] [11593.928119] [11593.930217] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11593.935526] ? lock_acquire+0xf0/0x2b0 [11593.939277] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11593.944683] ? lock_release+0xc5/0x260 [11593.948429] _raw_spin_lock+0x30/0x40 [11593.952092] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11593.957491] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11593.962720] delayed_work+0x1d8/0x280 [ceph] [11593.967003] ? process_scheduled_works+0x22a/0x510 [11593.971792] process_scheduled_works+0x2a4/0x510 [11593.976408] worker_thread+0x188/0x340 [11593.980156] ? rescuer_thread+0x310/0x310 [11593.984161] kthread+0xf5/0x130 [11593.987299] ? kthread_complete_and_exit+0x20/0x20 [11593.992082] ret_from_fork+0x31/0x40 [11593.995661] ? kthread_complete_and_exit+0x20/0x20 [11594.000444] ret_from_fork_asm+0x11/0x20 [11594.004367] [11601.221935] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11611.461922] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11621.670196] watchdog: BUG: soft lockup - CPU#7 stuck for 8706s! [kworker/7:4:78808] [11621.677868] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11621.733321] irq event stamp: 21124022 [11621.736977] hardirqs last enabled at (21124021): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11621.747141] hardirqs last disabled at (21124022): [] sysvec_apic_timer_interrupt+0xe/0x90 [11621.756867] softirqs last enabled at (21124018): [] __do_softirq+0x2be/0x371 [11621.765562] softirqs last disabled at (21124011): [] irq_exit_rcu+0xb0/0xd0 [11621.774079] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11621.783457] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11621.790928] Workqueue: events delayed_work [ceph] [11621.795651] RIP: 0010:_raw_spin_unlock+0x0/0x40 [11621.800176] Code: 18 bb 01 00 00 00 45 31 c9 41 b8 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 [11621.818913] RSP: 0018:ffffc90002117db0 EFLAGS: 00000286 [11621.824129] RAX: ffff8881a1d34478 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [11621.831254] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88817f867488 [11621.838379] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [11621.845501] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [11621.852627] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [11621.859752] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11621.867828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11621.873566] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11621.880689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11621.887814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11621.894938] Call Trace: [11621.897385] [11621.899396] ? watchdog_timer_fn+0x1b4/0x220 [11621.903668] ? __hrtimer_run_queues+0x1d3/0x360 [11621.908200] ? hrtimer_interrupt+0x104/0x240 [11621.912462] ? __local_bh_enable+0x50/0x60 [11621.916555] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11621.921780] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11621.926736] [11621.928834] [11621.930932] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11621.936239] ? _raw_spin_trylock_bh+0x70/0x70 [11621.940597] ceph_check_delayed_caps+0x19b/0x270 [ceph] [11621.945834] delayed_work+0x1d8/0x280 [ceph] [11621.950113] ? process_scheduled_works+0x22a/0x510 [11621.954903] process_scheduled_works+0x2a4/0x510 [11621.959521] worker_thread+0x188/0x340 [11621.963268] ? rescuer_thread+0x310/0x310 [11621.967271] kthread+0xf5/0x130 [11621.970409] ? kthread_complete_and_exit+0x20/0x20 [11621.975194] ret_from_fork+0x31/0x40 [11621.978773] ? kthread_complete_and_exit+0x20/0x20 [11621.983556] ret_from_fork_asm+0x11/0x20 [11621.987479] [11622.214169] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11632.454295] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11643.206417] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11649.670694] watchdog: BUG: soft lockup - CPU#7 stuck for 8732s! [kworker/7:4:78808] [11649.678342] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11649.733791] irq event stamp: 21184486 [11649.737448] hardirqs last enabled at (21184485): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11649.747614] hardirqs last disabled at (21184486): [] sysvec_apic_timer_interrupt+0xe/0x90 [11649.757338] softirqs last enabled at (21184438): [] __do_softirq+0x2be/0x371 [11649.766023] softirqs last disabled at (21184433): [] irq_exit_rcu+0xb0/0xd0 [11649.774544] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11649.783920] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11649.791393] Workqueue: events delayed_work [ceph] [11649.796115] RIP: 0010:lock_acquire+0xf0/0x2b0 [11649.800475] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11649.819211] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [11649.824428] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [11649.831552] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [11649.838676] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [11649.845801] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [11649.852926] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11649.860049] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11649.868127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11649.873865] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11649.880987] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11649.888112] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11649.895236] Call Trace: [11649.897683] [11649.899693] ? watchdog_timer_fn+0x1b4/0x220 [11649.903966] ? __hrtimer_run_queues+0x1d3/0x360 [11649.908499] ? hrtimer_interrupt+0x104/0x240 [11649.912763] ? __local_bh_enable+0x50/0x60 [11649.916853] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11649.922078] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11649.927035] [11649.929133] [11649.931229] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11649.936538] ? lock_acquire+0xf0/0x2b0 [11649.940290] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11649.945697] _raw_spin_lock+0x30/0x40 [11649.949354] ? igrab+0x1d/0x50 [11649.952412] igrab+0x1d/0x50 [11649.955290] ceph_check_delayed_caps+0xb7/0x270 [ceph] [11649.960432] delayed_work+0x1d8/0x280 [ceph] [11649.964713] ? process_scheduled_works+0x22a/0x510 [11649.969501] process_scheduled_works+0x2a4/0x510 [11649.974117] worker_thread+0x188/0x340 [11649.977868] ? rescuer_thread+0x310/0x310 [11649.981869] kthread+0xf5/0x130 [11649.985009] ? kthread_complete_and_exit+0x20/0x20 [11649.989793] ret_from_fork+0x31/0x40 [11649.993372] ? kthread_complete_and_exit+0x20/0x20 [11649.998155] ret_from_fork_asm+0x11/0x20 [11650.002079] [11653.446747] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11664.198793] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11674.439097] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11677.671186] watchdog: BUG: soft lockup - CPU#7 stuck for 8758s! [kworker/7:4:78808] [11677.678840] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11677.734289] irq event stamp: 21244868 [11677.737945] hardirqs last enabled at (21244867): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11677.748112] hardirqs last disabled at (21244868): [] sysvec_apic_timer_interrupt+0xe/0x90 [11677.757837] softirqs last enabled at (21244848): [] __do_softirq+0x2be/0x371 [11677.766529] softirqs last disabled at (21244843): [] irq_exit_rcu+0xb0/0xd0 [11677.775042] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11677.784418] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11677.791889] Workqueue: events delayed_work [ceph] [11677.796613] RIP: 0010:ceph_check_caps+0x1d4/0xae0 [ceph] [11677.801933] Code: e4 74 11 48 8b 44 24 08 44 8b 98 b0 01 00 00 45 85 db 74 11 0f b7 03 66 25 00 f0 66 3d 00 80 0f 84 17 04 00 00 48 8b 7c 24 48 d7 d8 04 e1 49 89 c7 48 85 c0 0f 84 a3 06 00 00 8b 44 24 04 44 [11677.820672] RSP: 0018:ffffc90002117bd0 EFLAGS: 00000a87 [11677.825888] RAX: 0000000000004000 RBX: ffff8881a1d34478 RCX: ffff88817f867000 [11677.833013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8881a1d34b20 [11677.840137] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [11677.847260] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [11677.854384] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 [11677.861509] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11677.869586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11677.875325] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11677.882449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11677.889572] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11677.896695] Call Trace: [11677.899143] [11677.901154] ? watchdog_timer_fn+0x1b4/0x220 [11677.905426] ? __hrtimer_run_queues+0x1d3/0x360 [11677.909959] ? hrtimer_interrupt+0x104/0x240 [11677.914221] ? __local_bh_enable+0x50/0x60 [11677.918313] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11677.923538] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11677.928496] [11677.930593] [11677.932691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11677.937998] ? ceph_check_caps+0x1d4/0xae0 [ceph] [11677.942719] ? lock_acquire+0xcb/0x2b0 [11677.946471] ? igrab+0x1d/0x50 [11677.949522] ? find_held_lock+0x2d/0x90 [11677.953361] ? find_held_lock+0x2d/0x90 [11677.957191] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11677.962591] ? lock_release+0xc5/0x260 [11677.966337] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11677.971564] delayed_work+0x1d8/0x280 [ceph] [11677.975845] ? process_scheduled_works+0x22a/0x510 [11677.980635] process_scheduled_works+0x2a4/0x510 [11677.985251] worker_thread+0x188/0x340 [11677.989000] ? rescuer_thread+0x310/0x310 [11677.993002] kthread+0xf5/0x130 [11677.996141] ? kthread_complete_and_exit+0x20/0x20 [11678.000925] ret_from_fork+0x31/0x40 [11678.004506] ? kthread_complete_and_exit+0x20/0x20 [11678.009287] ret_from_fork_asm+0x11/0x20 [11678.013211] [11685.191099] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11695.431398] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11705.671681] watchdog: BUG: soft lockup - CPU#7 stuck for 8784s! [kworker/7:4:78808] [11705.679336] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11705.734787] irq event stamp: 21305124 [11705.738443] hardirqs last enabled at (21305123): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11705.748610] hardirqs last disabled at (21305124): [] sysvec_apic_timer_interrupt+0xe/0x90 [11705.758335] softirqs last enabled at (21304998): [] __do_softirq+0x2be/0x371 [11705.767027] softirqs last disabled at (21304993): [] irq_exit_rcu+0xb0/0xd0 [11705.775547] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11705.784926] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11705.792396] Workqueue: events delayed_work [ceph] [11705.797119] RIP: 0010:lock_acquire+0xf0/0x2b0 [11705.801471] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11705.820206] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11705.825424] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11705.832548] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11705.839673] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11705.846796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11705.853919] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11705.861044] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11705.869123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11705.874859] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11705.881985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11705.889107] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11705.896232] Call Trace: [11705.898678] [11705.900689] ? watchdog_timer_fn+0x1b4/0x220 [11705.904953] ? __hrtimer_run_queues+0x1d3/0x360 [11705.909485] ? hrtimer_interrupt+0x104/0x240 [11705.913747] ? __local_bh_enable+0x50/0x60 [11705.917841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11705.923064] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11705.928022] [11705.930119] [11705.932217] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11705.937525] ? lock_acquire+0xf0/0x2b0 [11705.941275] ? ceph_check_caps+0x26b/0xae0 [ceph] [11705.945988] ? lock_release+0xc5/0x260 [11705.949733] _raw_spin_lock+0x30/0x40 [11705.953390] ? ceph_check_caps+0x63/0xae0 [ceph] [11705.958009] ceph_check_caps+0x63/0xae0 [ceph] [11705.962460] ? __lock_acquire+0xbfa/0x1e20 [11705.966558] ? lock_acquire+0xcb/0x2b0 [11705.970307] ? igrab+0x1d/0x50 [11705.973358] ? find_held_lock+0x2d/0x90 [11705.977188] ? find_held_lock+0x2d/0x90 [11705.981020] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11705.986419] ? lock_release+0xc5/0x260 [11705.990165] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11705.995391] delayed_work+0x1d8/0x280 [ceph] [11705.999673] ? process_scheduled_works+0x22a/0x510 [11706.004463] process_scheduled_works+0x2a4/0x510 [11706.009079] worker_thread+0x188/0x340 [11706.012828] ? rescuer_thread+0x310/0x310 [11706.016832] kthread+0xf5/0x130 [11706.019979] ? kthread_complete_and_exit+0x20/0x20 [11706.024764] ret_from_fork+0x31/0x40 [11706.028342] ? kthread_complete_and_exit+0x20/0x20 [11706.033124] ret_from_fork_asm+0x11/0x20 [11706.037048] [11706.183459] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11716.423798] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11727.175929] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11733.672175] watchdog: BUG: soft lockup - CPU#7 stuck for 8810s! [kworker/7:4:78808] [11733.679827] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11733.735277] irq event stamp: 21365878 [11733.738933] hardirqs last enabled at (21365877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11733.749099] hardirqs last disabled at (21365878): [] sysvec_apic_timer_interrupt+0xe/0x90 [11733.758824] softirqs last enabled at (21365846): [] __do_softirq+0x2be/0x371 [11733.767517] softirqs last disabled at (21365841): [] irq_exit_rcu+0xb0/0xd0 [11733.776036] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11733.785414] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11733.792885] Workqueue: events delayed_work [ceph] [11733.797607] RIP: 0010:lock_release+0x14e/0x260 [11733.802045] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [11733.820783] RSP: 0018:ffffc90002117d60 EFLAGS: 00000206 [11733.826000] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d6c [11733.833125] RDX: 0000000000000002 RSI: ffff88817f8674a0 RDI: ffff88810debb528 [11733.840249] RBP: ffff88817f8674a0 R08: 0000000000005c53 R09: 0000000000000003 [11733.847372] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa0b399eb [11733.854497] R13: 0000000000000246 R14: 0000000000000003 R15: ffff888100ee2000 [11733.861620] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11733.869699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11733.875435] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11733.882559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11733.889684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11733.896808] Call Trace: [11733.899255] [11733.901265] ? watchdog_timer_fn+0x1b4/0x220 [11733.905537] ? __hrtimer_run_queues+0x1d3/0x360 [11733.910070] ? hrtimer_interrupt+0x104/0x240 [11733.914331] ? __local_bh_enable+0x50/0x60 [11733.918426] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11733.923650] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11733.928607] [11733.930704] [11733.932803] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11733.938107] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11733.943517] ? lock_release+0x14e/0x260 [11733.947356] _raw_spin_unlock+0x1b/0x40 [11733.951194] ceph_check_delayed_caps+0x19b/0x270 [ceph] [11733.956422] delayed_work+0x1d8/0x280 [ceph] [11733.960703] ? process_scheduled_works+0x22a/0x510 [11733.965491] process_scheduled_works+0x2a4/0x510 [11733.970109] worker_thread+0x188/0x340 [11733.973858] ? rescuer_thread+0x310/0x310 [11733.977860] kthread+0xf5/0x130 [11733.981000] ? kthread_complete_and_exit+0x20/0x20 [11733.985782] ret_from_fork+0x31/0x40 [11733.989353] ? kthread_complete_and_exit+0x20/0x20 [11733.994137] ret_from_fork_asm+0x11/0x20 [11733.998061] [11737.416133] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11748.168363] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11758.408407] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11761.672669] watchdog: BUG: soft lockup - CPU#7 stuck for 8836s! [kworker/7:4:78808] [11761.680315] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11761.735766] irq event stamp: 21426090 [11761.739422] hardirqs last enabled at (21426089): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11761.749589] hardirqs last disabled at (21426090): [] sysvec_apic_timer_interrupt+0xe/0x90 [11761.759313] softirqs last enabled at (21426058): [] __do_softirq+0x2be/0x371 [11761.767997] softirqs last disabled at (21426053): [] irq_exit_rcu+0xb0/0xd0 [11761.776508] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11761.785886] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11761.793357] Workqueue: events delayed_work [ceph] [11761.798072] RIP: 0010:lock_acquire+0xf0/0x2b0 [11761.802421] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11761.821159] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11761.826377] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11761.833502] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11761.840625] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11761.847749] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11761.854872] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11761.861997] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11761.870075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11761.875811] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11761.882936] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11761.890061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11761.897185] Call Trace: [11761.899632] [11761.901641] ? watchdog_timer_fn+0x1b4/0x220 [11761.905904] ? __hrtimer_run_queues+0x1d3/0x360 [11761.910431] ? hrtimer_interrupt+0x104/0x240 [11761.914701] ? __local_bh_enable+0x50/0x60 [11761.918794] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11761.924018] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11761.928974] [11761.931071] [11761.933169] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11761.938477] ? lock_acquire+0xf0/0x2b0 [11761.942228] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11761.947634] ? lock_release+0xc5/0x260 [11761.951379] _raw_spin_lock+0x30/0x40 [11761.955037] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11761.960434] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11761.965664] delayed_work+0x1d8/0x280 [ceph] [11761.969943] ? process_scheduled_works+0x22a/0x510 [11761.974725] process_scheduled_works+0x2a4/0x510 [11761.979341] worker_thread+0x188/0x340 [11761.983092] ? rescuer_thread+0x310/0x310 [11761.987095] kthread+0xf5/0x130 [11761.990231] ? kthread_complete_and_exit+0x20/0x20 [11761.995018] ret_from_fork+0x31/0x40 [11761.998596] ? kthread_complete_and_exit+0x20/0x20 [11762.003379] ret_from_fork_asm+0x11/0x20 [11762.007302] [11769.160754] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11779.400997] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11789.673166] watchdog: BUG: soft lockup - CPU#7 stuck for 8862s! [kworker/7:4:78808] [11789.680822] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11789.736272] irq event stamp: 21486420 [11789.739929] hardirqs last enabled at (21486419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11789.750094] hardirqs last disabled at (21486420): [] sysvec_apic_timer_interrupt+0xe/0x90 [11789.759819] softirqs last enabled at (21486406): [] __do_softirq+0x2be/0x371 [11789.768512] softirqs last disabled at (21486401): [] irq_exit_rcu+0xb0/0xd0 [11789.777032] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11789.786410] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11789.793880] Workqueue: events delayed_work [ceph] [11789.798602] RIP: 0010:lock_acquire+0xf0/0x2b0 [11789.802956] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11789.821691] RSP: 0018:ffffc90002117d30 EFLAGS: 00000206 [11789.826908] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000de0313a6 [11789.834033] RDX: b656a521d3820f2c RSI: 0000000000000000 RDI: ffff88810debb528 [11789.841157] RBP: 0000000000000001 R08: 000000000000694b R09: 0000000000000002 [11789.848281] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88817f8674a0 [11789.855406] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11789.862530] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11789.870608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11789.876345] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11789.883468] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11789.890592] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11789.897717] Call Trace: [11789.900163] [11789.902175] ? watchdog_timer_fn+0x1b4/0x220 [11789.906446] ? __hrtimer_run_queues+0x1d3/0x360 [11789.910981] ? hrtimer_interrupt+0x104/0x240 [11789.915250] ? __local_bh_enable+0x50/0x60 [11789.919344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11789.924567] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11789.929524] [11789.931622] [11789.933720] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11789.939026] ? lock_acquire+0xf0/0x2b0 [11789.942777] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11789.948184] ? lock_release+0xc5/0x260 [11789.951929] _raw_spin_lock+0x30/0x40 [11789.955585] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11789.960984] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11789.966215] delayed_work+0x1d8/0x280 [ceph] [11789.970494] ? process_scheduled_works+0x22a/0x510 [11789.975284] process_scheduled_works+0x2a4/0x510 [11789.979900] worker_thread+0x188/0x340 [11789.983651] ? rescuer_thread+0x310/0x310 [11789.987662] kthread+0xf5/0x130 [11789.990800] ? kthread_complete_and_exit+0x20/0x20 [11789.995583] ret_from_fork+0x31/0x40 [11789.999163] ? kthread_complete_and_exit+0x20/0x20 [11790.003948] ret_from_fork_asm+0x11/0x20 [11790.007879] [11790.152995] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11800.393359] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11817.673657] watchdog: BUG: soft lockup - CPU#7 stuck for 8889s! [kworker/7:4:78808] [11817.681310] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11817.736760] irq event stamp: 21546668 [11817.740417] hardirqs last enabled at (21546667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11817.750585] hardirqs last disabled at (21546668): [] sysvec_apic_timer_interrupt+0xe/0x90 [11817.760316] softirqs last enabled at (21546654): [] __do_softirq+0x2be/0x371 [11817.769001] softirqs last disabled at (21546649): [] irq_exit_rcu+0xb0/0xd0 [11817.777522] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11817.786900] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11817.794370] Workqueue: events delayed_work [ceph] [11817.799092] RIP: 0010:lock_acquire+0xf0/0x2b0 [11817.803443] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11817.822180] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11817.827399] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11817.834523] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11817.841646] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11817.848770] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11817.855894] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11817.863020] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11817.871096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11817.876832] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11817.883958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11817.891081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11817.898206] Call Trace: [11817.900652] [11817.902662] ? watchdog_timer_fn+0x1b4/0x220 [11817.906927] ? __hrtimer_run_queues+0x1d3/0x360 [11817.911460] ? hrtimer_interrupt+0x104/0x240 [11817.915722] ? __local_bh_enable+0x50/0x60 [11817.919815] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11817.925038] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11817.929995] [11817.932092] [11817.934191] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11817.939497] ? lock_acquire+0xf0/0x2b0 [11817.943250] ? ceph_check_caps+0x26b/0xae0 [ceph] [11817.947964] ? lock_release+0xc5/0x260 [11817.951707] _raw_spin_lock+0x30/0x40 [11817.955366] ? ceph_check_caps+0x63/0xae0 [ceph] [11817.959984] ceph_check_caps+0x63/0xae0 [ceph] [11817.964435] ? __lock_acquire+0xbfa/0x1e20 [11817.968533] ? lock_acquire+0xcb/0x2b0 [11817.972283] ? igrab+0x1d/0x50 [11817.975341] ? find_held_lock+0x2d/0x90 [11817.979180] ? find_held_lock+0x2d/0x90 [11817.983011] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11817.988411] ? lock_release+0xc5/0x260 [11817.992157] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11817.997393] delayed_work+0x1d8/0x280 [ceph] [11818.001674] ? process_scheduled_works+0x22a/0x510 [11818.006462] process_scheduled_works+0x2a4/0x510 [11818.011080] worker_thread+0x188/0x340 [11818.014828] ? rescuer_thread+0x310/0x310 [11818.018832] kthread+0xf5/0x130 [11818.021968] ? kthread_complete_and_exit+0x20/0x20 [11818.026752] ret_from_fork+0x31/0x40 [11818.030324] ? kthread_complete_and_exit+0x20/0x20 [11818.035108] ret_from_fork_asm+0x11/0x20 [11818.039040] [11832.137322] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [11832.145746] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11842.377984] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11845.674151] watchdog: BUG: soft lockup - CPU#7 stuck for 8915s! [kworker/7:4:78808] [11845.681801] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11845.737249] irq event stamp: 21607592 [11845.740907] hardirqs last enabled at (21607591): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11845.751073] hardirqs last disabled at (21607592): [] sysvec_apic_timer_interrupt+0xe/0x90 [11845.760798] softirqs last enabled at (21607514): [] __do_softirq+0x2be/0x371 [11845.769482] softirqs last disabled at (21607509): [] irq_exit_rcu+0xb0/0xd0 [11845.778003] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11845.787387] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11845.794858] Workqueue: events delayed_work [ceph] [11845.799580] RIP: 0010:lock_acquire+0xf0/0x2b0 [11845.803934] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11845.822670] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [11845.827887] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [11845.835010] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [11845.842136] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [11845.849259] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [11845.856385] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11845.863508] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11845.871584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11845.877323] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11845.884447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11845.891570] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11845.898696] Call Trace: [11845.901141] [11845.903152] ? watchdog_timer_fn+0x1b4/0x220 [11845.907423] ? __hrtimer_run_queues+0x1d3/0x360 [11845.911949] ? hrtimer_interrupt+0x104/0x240 [11845.916212] ? __local_bh_enable+0x50/0x60 [11845.920303] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11845.925529] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11845.930486] [11845.932582] [11845.934681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11845.939988] ? lock_acquire+0xf0/0x2b0 [11845.943740] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11845.949148] _raw_spin_lock+0x30/0x40 [11845.952812] ? igrab+0x1d/0x50 [11845.955862] igrab+0x1d/0x50 [11845.958740] ceph_check_delayed_caps+0xb7/0x270 [ceph] [11845.963890] delayed_work+0x1d8/0x280 [ceph] [11845.968170] ? process_scheduled_works+0x22a/0x510 [11845.972962] process_scheduled_works+0x2a4/0x510 [11845.977577] worker_thread+0x188/0x340 [11845.981325] ? rescuer_thread+0x310/0x310 [11845.985328] kthread+0xf5/0x130 [11845.988467] ? kthread_complete_and_exit+0x20/0x20 [11845.993250] ret_from_fork+0x31/0x40 [11845.996823] ? kthread_complete_and_exit+0x20/0x20 [11846.001605] ret_from_fork_asm+0x11/0x20 [11846.005528] [11853.130016] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11863.370413] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11873.674659] watchdog: BUG: soft lockup - CPU#7 stuck for 8941s! [kworker/7:4:78808] [11873.682307] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11873.737756] irq event stamp: 21668730 [11873.741412] hardirqs last enabled at (21668729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11873.751580] hardirqs last disabled at (21668730): [] sysvec_apic_timer_interrupt+0xe/0x90 [11873.761311] softirqs last enabled at (21668710): [] __do_softirq+0x2be/0x371 [11873.769998] softirqs last disabled at (21668705): [] irq_exit_rcu+0xb0/0xd0 [11873.778519] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11873.787895] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11873.795366] Workqueue: events delayed_work [ceph] [11873.800087] RIP: 0010:lock_acquire+0xf0/0x2b0 [11873.804439] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11873.823175] RSP: 0018:ffffc90002117b48 EFLAGS: 00000206 [11873.828394] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 00000000d3c6b569 [11873.835516] RDX: d83d95b1e75052a7 RSI: 0000000000000000 RDI: ffff88810debb528 [11873.842641] RBP: 0000000000000001 R08: 00000000000079a2 R09: 0000000000000002 [11873.849767] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34938 [11873.856891] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11873.864014] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11873.872091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11873.877829] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11873.884953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11873.892078] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11873.899202] Call Trace: [11873.901647] [11873.903658] ? watchdog_timer_fn+0x1b4/0x220 [11873.907920] ? __hrtimer_run_queues+0x1d3/0x360 [11873.912447] ? hrtimer_interrupt+0x104/0x240 [11873.916707] ? __local_bh_enable+0x50/0x60 [11873.920802] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11873.926027] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11873.930982] [11873.933080] [11873.935177] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11873.940486] ? lock_acquire+0xf0/0x2b0 [11873.944235] ? ceph_check_caps+0x26b/0xae0 [ceph] [11873.948949] ? lock_release+0xc5/0x260 [11873.952696] _raw_spin_lock+0x30/0x40 [11873.956353] ? ceph_check_caps+0x63/0xae0 [ceph] [11873.960971] ceph_check_caps+0x63/0xae0 [ceph] [11873.965422] ? __lock_acquire+0xbfa/0x1e20 [11873.969521] ? lock_acquire+0xcb/0x2b0 [11873.973268] ? igrab+0x1d/0x50 [11873.976318] ? find_held_lock+0x2d/0x90 [11873.980151] ? find_held_lock+0x2d/0x90 [11873.983981] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11873.989381] ? lock_release+0xc5/0x260 [11873.993127] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11873.998362] delayed_work+0x1d8/0x280 [ceph] [11874.002642] ? process_scheduled_works+0x22a/0x510 [11874.007432] process_scheduled_works+0x2a4/0x510 [11874.012048] worker_thread+0x188/0x340 [11874.015799] ? rescuer_thread+0x310/0x310 [11874.019802] kthread+0xf5/0x130 [11874.022947] ? kthread_complete_and_exit+0x20/0x20 [11874.027732] ret_from_fork+0x31/0x40 [11874.031311] ? kthread_complete_and_exit+0x20/0x20 [11874.036094] ret_from_fork_asm+0x11/0x20 [11874.040018] [11874.122573] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11884.363041] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11895.114813] libceph: auth protocol 'cephx' mauth authentication failed: -13 [11901.675139] watchdog: BUG: soft lockup - CPU#7 stuck for 8967s! [kworker/7:4:78808] [11901.682786] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11901.738237] irq event stamp: 21730208 [11901.741893] hardirqs last enabled at (21730207): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11901.752059] hardirqs last disabled at (21730208): [] sysvec_apic_timer_interrupt+0xe/0x90 [11901.761785] softirqs last enabled at (21730130): [] __do_softirq+0x2be/0x371 [11901.770477] softirqs last disabled at (21730125): [] irq_exit_rcu+0xb0/0xd0 [11901.778999] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11901.788375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11901.795845] Workqueue: events delayed_work [ceph] [11901.800567] RIP: 0010:lock_acquire+0x85/0x2b0 [11901.804919] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [11901.823656] RSP: 0018:ffffc90002117d28 EFLAGS: 00000246 [11901.828873] RAX: 0000000000000000 RBX: ffff88817f867488 RCX: 0000000000000000 [11901.835997] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88817f8674a0 [11901.843122] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [11901.850247] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88817f8674a0 [11901.857370] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11901.864495] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11901.872572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11901.878309] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11901.885433] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11901.892558] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11901.899682] Call Trace: [11901.902129] [11901.904140] ? watchdog_timer_fn+0x1b4/0x220 [11901.908411] ? __hrtimer_run_queues+0x1d3/0x360 [11901.912936] ? hrtimer_interrupt+0x104/0x240 [11901.917197] ? __local_bh_enable+0x50/0x60 [11901.921290] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11901.926514] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11901.931472] [11901.933569] [11901.935668] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11901.940975] ? lock_acquire+0x85/0x2b0 [11901.944725] ? find_held_lock+0x2d/0x90 [11901.948562] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11901.953962] ? lock_release+0xc5/0x260 [11901.957708] _raw_spin_lock+0x30/0x40 [11901.961364] ? ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11901.966764] ceph_check_delayed_caps+0x1c7/0x270 [ceph] [11901.971994] delayed_work+0x1d8/0x280 [ceph] [11901.976273] ? process_scheduled_works+0x22a/0x510 [11901.981063] process_scheduled_works+0x2a4/0x510 [11901.985680] worker_thread+0x188/0x340 [11901.989429] ? rescuer_thread+0x310/0x310 [11901.993441] kthread+0xf5/0x130 [11901.996586] ? kthread_complete_and_exit+0x20/0x20 [11902.001372] ret_from_fork+0x31/0x40 [11902.004951] ? kthread_complete_and_exit+0x20/0x20 [11902.009733] ret_from_fork_asm+0x11/0x20 [11902.013656] [11929.675634] watchdog: BUG: soft lockup - CPU#7 stuck for 8993s! [kworker/7:4:78808] [11929.683286] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11929.738735] irq event stamp: 21790544 [11929.742392] hardirqs last enabled at (21790543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11929.752558] hardirqs last disabled at (21790544): [] sysvec_apic_timer_interrupt+0xe/0x90 [11929.762290] softirqs last enabled at (21790538): [] __do_softirq+0x2be/0x371 [11929.770976] softirqs last disabled at (21790533): [] irq_exit_rcu+0xb0/0xd0 [11929.779496] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11929.788871] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11929.796343] Workqueue: events delayed_work [ceph] [11929.801065] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [11929.805678] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [11929.824414] RSP: 0018:ffffc90002117bc0 EFLAGS: 00000246 [11929.829632] RAX: 0000000000000000 RBX: ffff8881a1d34920 RCX: 00000000d3c6b569 [11929.836756] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34920 [11929.843879] RBP: 0000000000000000 R08: 00000000000079a2 R09: 0000000000000002 [11929.851003] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [11929.858129] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [11929.865253] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11929.873329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11929.879068] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11929.886191] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11929.893315] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11929.900440] Call Trace: [11929.902885] [11929.904898] ? watchdog_timer_fn+0x1b4/0x220 [11929.909169] ? __hrtimer_run_queues+0x1d3/0x360 [11929.913702] ? hrtimer_interrupt+0x104/0x240 [11929.917965] ? __local_bh_enable+0x50/0x60 [11929.922057] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11929.927282] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11929.932239] [11929.934335] [11929.936433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11929.941740] ? do_raw_spin_lock+0x42/0xc0 [11929.945751] ceph_check_caps+0x63/0xae0 [ceph] [11929.950211] ? __lock_acquire+0xbfa/0x1e20 [11929.954309] ? lock_acquire+0xcb/0x2b0 [11929.958057] ? igrab+0x1d/0x50 [11929.961107] ? find_held_lock+0x2d/0x90 [11929.964939] ? find_held_lock+0x2d/0x90 [11929.968768] ? ceph_check_delayed_caps+0x19b/0x270 [ceph] [11929.974170] ? lock_release+0xc5/0x260 [11929.977914] ceph_check_delayed_caps+0x1b1/0x270 [ceph] [11929.983141] delayed_work+0x1d8/0x280 [ceph] [11929.987422] ? process_scheduled_works+0x22a/0x510 [11929.992214] process_scheduled_works+0x2a4/0x510 [11929.996830] worker_thread+0x188/0x340 [11930.000579] ? rescuer_thread+0x310/0x310 [11930.004589] kthread+0xf5/0x130 [11930.007728] ? kthread_complete_and_exit+0x20/0x20 [11930.012511] ret_from_fork+0x31/0x40 [11930.016084] ? kthread_complete_and_exit+0x20/0x20 [11930.020873] ret_from_fork_asm+0x11/0x20 [11930.024797] [11936.225126] ------------[ cut here ]------------ [11936.229750] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [11936.241428] WARNING: CPU: 4 PID: 78687 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [11936.250389] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11936.305927] CPU: 4 PID: 78687 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11936.314888] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11936.322367] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [11936.327851] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [11936.346599] RSP: 0018:ffffc90001fffbd0 EFLAGS: 00010282 [11936.351825] RAX: 0000000000000000 RBX: ffff8881d12c4e40 RCX: 0000000000000000 [11936.358962] RDX: ffff88885fd29c00 RSI: ffff88885fd1d880 RDI: ffff88885fd1d880 [11936.366106] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000f2e7 [11936.373240] R10: 00000000004a2a40 R11: ffffc90001fffa60 R12: ffff8881d12c5590 [11936.380372] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [11936.387506] FS: 00007f48f1832740(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 [11936.395597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11936.401346] CR2: 00007f808760c1a0 CR3: 000000017eb7a004 CR4: 00000000003706e0 [11936.408478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11936.415613] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11936.422745] Call Trace: [11936.425200] [11936.427308] ? __warn+0x85/0x180 [11936.430547] ? __list_add_valid_or_report+0x7d/0xa0 [11936.435428] ? report_bug+0xfc/0x1d0 [11936.439013] ? prb_read_valid+0x1b/0x20 [11936.442857] ? handle_bug+0x44/0x70 [11936.446354] ? exc_invalid_op+0x17/0x60 [11936.450195] ? asm_exc_invalid_op+0x1a/0x20 [11936.454392] ? __list_add_valid_or_report+0x7d/0xa0 [11936.459278] __cap_delay_requeue+0x140/0x1d0 [ceph] [11936.464187] try_get_cap_refs+0x159/0x880 [ceph] [11936.468838] __ceph_get_caps+0x4a1/0x760 [ceph] [11936.473399] ? wake_bit_function+0x70/0x70 [11936.477508] ? ceph_read_iter+0x134/0x860 [ceph] [11936.482147] ceph_read_iter+0x134/0x860 [ceph] [11936.486622] ? selinux_file_permission+0x119/0x150 [11936.491428] ? vfs_read+0x20f/0x300 [11936.494922] vfs_read+0x20f/0x300 [11936.498254] ksys_pread64+0x65/0xa0 [11936.501760] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11936.506904] do_syscall_64+0x38/0x80 [11936.510489] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11936.515547] RIP: 0033:0x7f48f0eeb88b [11936.519129] Code: 00 48 8d 3d f5 cc 09 00 e8 c2 41 f4 ff 66 90 f3 0f 1e fa 48 8d 05 15 8d 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [11936.537876] RSP: 002b:00007ffc53d60078 EFLAGS: 00000246 ORIG_RAX: 0000000000000011 [11936.545458] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb88b [11936.552591] RDX: 0000000000001000 RSI: 000000000207e000 RDI: 0000000000000006 [11936.559729] RBP: 0000000000002a10 R08: 0000000000000063 R09: 0000000000000003 [11936.566866] R10: 000000000003f000 R11: 0000000000000246 R12: 0000000000001000 [11936.573999] R13: 0000000000001000 R14: 000000000207e000 R15: 00007f48f183e368 [11936.581144] [11936.583343] irq event stamp: 12942621 [11936.587007] hardirqs last enabled at (12942631): [] console_unlock+0x106/0x120 [11936.595876] hardirqs last disabled at (12942650): [] console_unlock+0xeb/0x120 [11936.604661] softirqs last enabled at (12942648): [] __do_softirq+0x2be/0x371 [11936.613355] softirqs last disabled at (12942639): [] irq_exit_rcu+0xb0/0xd0 [11936.621875] ---[ end trace 0000000000000000 ]--- [11936.626497] ------------[ cut here ]------------ [11936.631120] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [11936.642781] WARNING: CPU: 5 PID: 78680 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [11936.651741] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11936.707271] CPU: 5 PID: 78680 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11936.716222] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11936.723701] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [11936.729185] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [11936.747934] RSP: 0018:ffffc90001fc7a18 EFLAGS: 00010286 [11936.753159] RAX: 0000000000000000 RBX: ffff888158a68000 RCX: 0000000000000000 [11936.760292] RDX: ffff88885fd69c00 RSI: ffff88885fd5d880 RDI: ffff88885fd5d880 [11936.767425] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000f320 [11936.774559] R10: 00000000004a3a20 R11: ffffc90001fc78a8 R12: ffff888158a68750 [11936.781690] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [11936.788824] FS: 00007f48f1832740(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000 [11936.796910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11936.802655] CR2: 00007fa9f121b340 CR3: 0000000113500002 CR4: 00000000003706e0 [11936.809787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11936.816922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11936.824054] Call Trace: [11936.826508] [11936.828614] ? __warn+0x85/0x180 [11936.831846] ? __list_add_valid_or_report+0x7d/0xa0 [11936.836726] ? report_bug+0xfc/0x1d0 [11936.840306] ? prb_read_valid+0x1b/0x20 [11936.844147] ? handle_bug+0x44/0x70 [11936.847645] ? exc_invalid_op+0x17/0x60 [11936.851486] ? asm_exc_invalid_op+0x1a/0x20 [11936.855681] ? __list_add_valid_or_report+0x7d/0xa0 [11936.860562] ? __list_add_valid_or_report+0x7d/0xa0 [11936.865447] __cap_delay_requeue+0x140/0x1d0 [ceph] [11936.870349] ceph_mdsc_submit_request+0x86/0x4d0 [ceph] [11936.875600] ceph_mdsc_do_request+0x33/0xf0 [ceph] [11936.880417] ceph_lookup+0xf9/0x3f0 [ceph] [11936.884538] __lookup_slow+0xff/0x1a0 [11936.888225] walk_component+0x83/0x160 [11936.891982] link_path_walk+0x24d/0x380 [11936.895825] ? path_lookupat.isra.40+0x22/0x120 [11936.900363] path_lookupat.isra.40+0x6b/0x120 [11936.904732] filename_lookup+0xd4/0x190 [11936.908581] ? lock_acquire+0xcb/0x2b0 [11936.912335] ? __might_fault+0x45/0x80 [11936.916090] ? find_held_lock+0x2d/0x90 [11936.919935] ? __might_fault+0x45/0x80 [11936.923694] vfs_statx+0x89/0x150 [11936.927023] vfs_fstatat+0x5a/0xc0 [11936.930433] __do_sys_newstat+0x30/0x70 [11936.934271] ? lock_release+0xc5/0x260 [11936.938027] ? ktime_get_coarse_real_ts64+0xc5/0xd0 [11936.942910] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11936.948051] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11936.953194] do_syscall_64+0x38/0x80 [11936.956779] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11936.961830] RIP: 0033:0x7f48f0eeca79 [11936.965411] Code: 00 00 00 75 05 48 83 c4 18 c3 e8 a2 08 02 00 66 90 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 04 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 d9 23 2d 00 [11936.984157] RSP: 002b:00007ffc53d5ffd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [11936.991721] RAX: ffffffffffffffda RBX: 00007ffc53d60520 RCX: 00007f48f0eeca79 [11936.998855] RDX: 00007ffc53d5ffe0 RSI: 00007ffc53d5ffe0 RDI: 00007ffc53d60520 [11937.005988] RBP: 0000000002076550 R08: 0000000002076674 R09: 0000000000000000 [11937.013120] R10: 00007f48f0f711e0 R11: 0000000000000246 R12: 0000000000000004 [11937.020256] R13: 00007ffc53d60520 R14: 00007f48f183d450 R15: 00000000020766e0 [11937.027400] [11937.029598] irq event stamp: 12970917 [11937.033261] hardirqs last enabled at (12970927): [] console_unlock+0x106/0x120 [11937.042129] hardirqs last disabled at (12970936): [] console_unlock+0xeb/0x120 [11937.050916] softirqs last enabled at (12970952): [] __do_softirq+0x2be/0x371 [11937.059609] softirqs last disabled at (12970947): [] irq_exit_rcu+0xb0/0xd0 [11937.068133] ---[ end trace 0000000000000000 ]--- [11937.072750] ------------[ cut here ]------------ [11937.077368] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [11937.089012] WARNING: CPU: 1 PID: 78685 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [11937.097965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11937.153444] CPU: 1 PID: 78685 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11937.162397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11937.169875] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [11937.175364] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [11937.194106] RSP: 0018:ffffc90001fefbe0 EFLAGS: 00010286 [11937.199333] RAX: 0000000000000000 RBX: ffff8881c3b77560 RCX: 0000000000000000 [11937.206465] RDX: ffff88885fc69c00 RSI: ffff88885fc5d880 RDI: ffff88885fc5d880 [11937.213599] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000f364 [11937.220733] R10: 00000000004a4b80 R11: ffffc90001fefa70 R12: ffff8881c3b77cb0 [11937.227865] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [11937.234999] FS: 00007f48f1832740(0000) GS:ffff88885fc40000(0000) knlGS:0000000000000000 [11937.243082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11937.248822] CR2: 000055c8933c2010 CR3: 0000000108c72004 CR4: 00000000003706e0 [11937.255954] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11937.263085] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11937.270209] Call Trace: [11937.272656] [11937.274763] ? __warn+0x85/0x180 [11937.277995] ? __list_add_valid_or_report+0x7d/0xa0 [11937.282873] ? report_bug+0xfc/0x1d0 [11937.286455] ? prb_read_valid+0x1b/0x20 [11937.290295] ? handle_bug+0x44/0x70 [11937.293783] ? exc_invalid_op+0x17/0x60 [11937.297624] ? asm_exc_invalid_op+0x1a/0x20 [11937.301815] ? __list_add_valid_or_report+0x7d/0xa0 [11937.306691] __cap_delay_requeue+0x140/0x1d0 [ceph] [11937.311579] try_get_cap_refs+0x159/0x880 [ceph] [11937.316214] __ceph_get_caps+0x4a1/0x760 [ceph] [11937.320760] ? __file_remove_privs+0x89/0x110 [11937.325116] ? wake_bit_function+0x70/0x70 [11937.329219] ? ceph_write_iter+0x260/0x770 [ceph] [11937.333929] ceph_write_iter+0x260/0x770 [ceph] [11937.338476] ? lock_is_held_type+0xaa/0x120 [11937.342660] vfs_write+0x324/0x4f0 [11937.346069] ksys_pwrite64+0x65/0xa0 [11937.349643] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11937.354783] do_syscall_64+0x38/0x80 [11937.358360] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11937.363414] RIP: 0033:0x7f48f0eeb93b [11937.366994] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [11937.385740] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [11937.393305] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [11937.400437] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [11937.407570] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [11937.414696] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183df18 [11937.421828] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [11937.428968] [11937.431156] irq event stamp: 12905887 [11937.434821] hardirqs last enabled at (12905897): [] console_unlock+0x106/0x120 [11937.443685] hardirqs last disabled at (12905906): [] console_unlock+0xeb/0x120 [11937.452465] softirqs last enabled at (12905884): [] __do_softirq+0x2be/0x371 [11937.461158] softirqs last disabled at (12905877): [] irq_exit_rcu+0xb0/0xd0 [11937.469677] ---[ end trace 0000000000000000 ]--- [11937.474298] ------------[ cut here ]------------ [11937.474316] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [11937.478916] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [11937.478932] WARNING: CPU: 2 PID: 78684 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [11937.507010] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11937.562546] CPU: 2 PID: 78684 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11937.571500] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11937.578978] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [11937.584464] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [11937.603209] RSP: 0018:ffffc90001fe7be0 EFLAGS: 00010286 [11937.608436] RAX: 0000000000000000 RBX: ffff8881d12c6b98 RCX: 0000000000000027 [11937.615569] RDX: 0000000000000027 RSI: ffff88885fc9d880 RDI: ffff88885fc9d888 [11937.622703] RBP: ffff88817f867000 R08: 0000000000000001 R09: 0000000000000001 [11937.629838] R10: ffffc90001fe79d8 R11: ffffffff83115580 R12: ffff8881d12c72e8 [11937.636967] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [11937.644101] FS: 00007f48f1832740(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000 [11937.652187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11937.657933] CR2: 000056400e01ab90 CR3: 0000000108c08005 CR4: 00000000003706e0 [11937.665065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11937.672201] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11937.679331] Call Trace: [11937.681783] [11937.683891] ? __warn+0x85/0x180 [11937.687123] ? __list_add_valid_or_report+0x7d/0xa0 [11937.692003] ? report_bug+0xfc/0x1d0 [11937.695584] ? mark_held_locks+0x48/0x70 [11937.699511] ? handle_bug+0x44/0x70 [11937.703011] ? exc_invalid_op+0x17/0x60 [11937.706848] ? asm_exc_invalid_op+0x1a/0x20 [11937.711046] ? __list_add_valid_or_report+0x7d/0xa0 [11937.715934] __cap_delay_requeue+0x140/0x1d0 [ceph] [11937.720834] try_get_cap_refs+0x159/0x880 [ceph] [11937.725487] __ceph_get_caps+0x4a1/0x760 [ceph] [11937.730044] ? __file_remove_privs+0x89/0x110 [11937.734409] ? wake_bit_function+0x70/0x70 [11937.738516] ? ceph_write_iter+0x260/0x770 [ceph] [11937.743240] ceph_write_iter+0x260/0x770 [ceph] [11937.747798] ? lock_is_held_type+0xaa/0x120 [11937.751993] vfs_write+0x324/0x4f0 [11937.755411] ksys_pwrite64+0x65/0xa0 [11937.759001] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11937.764151] do_syscall_64+0x38/0x80 [11937.767735] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11937.772787] RIP: 0033:0x7f48f0eeb93b [11937.776366] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [11937.795111] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [11937.802677] RAX: ffffffffffffffda RBX: 0000000002075830 RCX: 00007f48f0eeb93b [11937.809809] RDX: 0000000000010000 RSI: 000000000207e000 RDI: 0000000000000005 [11937.816943] RBP: 0000000000002a0f R08: 0000000000010000 R09: 0000000000000003 [11937.824078] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f48f183dcf0 [11937.831208] R13: 0000000000010000 R14: 0000000000010000 R15: 000000000207e000 [11937.838355] [11937.840550] irq event stamp: 12912261 [11937.844217] hardirqs last enabled at (12912271): [] console_unlock+0x106/0x120 [11937.853083] hardirqs last disabled at (12912280): [] console_unlock+0xeb/0x120 [11937.861864] softirqs last enabled at (12912228): [] __do_softirq+0x2be/0x371 [11937.870555] softirqs last disabled at (12912223): [] irq_exit_rcu+0xb0/0xd0 [11937.879075] ---[ end trace 0000000000000000 ]--- [11937.883696] ------------[ cut here ]------------ [11937.888314] list_add corruption. prev->next should be next (ffff88817f867478), but was ffff8881a1d36920. (prev=ffff8881a1d36920). [11937.899959] WARNING: CPU: 6 PID: 78678 at lib/list_debug.c:34 __list_add_valid_or_report+0x7d/0xa0 [11937.908913] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11937.964384] CPU: 6 PID: 78678 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11937.973333] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11937.980813] RIP: 0010:__list_add_valid_or_report+0x7d/0xa0 [11937.986299] Code: c3 48 89 c1 4c 89 c2 48 c7 c7 c0 98 1d 82 e8 9a 0c b0 ff 0f 0b 31 c0 c3 4c 89 c1 48 89 c6 48 c7 c7 10 99 1d 82 e8 83 0c b0 ff <0f> 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 [11938.005045] RSP: 0018:ffffc90001f7fbe0 EFLAGS: 00010286 [11938.010270] RAX: 0000000000000000 RBX: ffff8881e4f8d808 RCX: 0000000000000000 [11938.017402] RDX: ffff88885fda9c00 RSI: ffff88885fd9d880 RDI: ffff88885fd9d880 [11938.024526] RBP: ffff88817f867000 R08: 0000000000000000 R09: c00000010000f3d7 [11938.031652] R10: 00000000004a6ab8 R11: ffffc90001f7fa70 R12: ffff8881e4f8df58 [11938.038785] R13: ffff88817f867488 R14: ffff8881a1d36920 R15: ffff88817f867478 [11938.045917] FS: 00007f48f1832740(0000) GS:ffff88885fd80000(0000) knlGS:0000000000000000 [11938.054002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11938.059742] CR2: 00007f7e9865d934 CR3: 0000000113532001 CR4: 00000000003706e0 [11938.066874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11938.074006] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11938.081131] Call Trace: [11938.083585] [11938.085689] ? __warn+0x85/0x180 [11938.088923] ? __list_add_valid_or_report+0x7d/0xa0 [11938.093802] ? report_bug+0xfc/0x1d0 [11938.097383] ? prb_read_valid+0x1b/0x20 [11938.101222] ? handle_bug+0x44/0x70 [11938.104713] ? exc_invalid_op+0x17/0x60 [11938.108553] ? asm_exc_invalid_op+0x1a/0x20 [11938.112744] ? __list_add_valid_or_report+0x7d/0xa0 [11938.117619] __cap_delay_requeue+0x140/0x1d0 [ceph] [11938.122508] try_get_cap_refs+0x159/0x880 [ceph] [11938.127140] __ceph_get_caps+0x4a1/0x760 [ceph] [11938.131688] ? __file_remove_privs+0x89/0x110 [11938.136045] ? wake_bit_function+0x70/0x70 [11938.140146] ? ceph_write_iter+0x260/0x770 [ceph] [11938.144867] ceph_write_iter+0x260/0x770 [ceph] [11938.149414] ? lock_is_held_type+0xaa/0x120 [11938.153597] vfs_write+0x324/0x4f0 [11938.157005] ksys_pwrite64+0x65/0xa0 [11938.160580] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11938.165720] do_syscall_64+0x38/0x80 [11938.169300] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11938.174351] RIP: 0033:0x7f48f0eeb93b [11938.177932] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [11938.196678] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [11938.204244] RAX: ffffffffffffffda RBX: 0000000002075840 RCX: 00007f48f0eeb93b [11938.211376] RDX: 0000000000000001 RSI: 000000000207bcc0 RDI: 0000000000000006 [11938.218509] RBP: 0000000000002a1a R08: 0000000000000001 R09: 0000000002074010 [11938.225642] R10: 00000000000005ff R11: 0000000000000246 R12: 00007f48f183d000 [11938.232774] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207bcc0 [11938.239913] [11938.242100] irq event stamp: 12975433 [11938.245765] hardirqs last enabled at (12975443): [] console_unlock+0x106/0x120 [11938.254636] hardirqs last disabled at (12975464): [] console_unlock+0xeb/0x120 [11938.263413] softirqs last enabled at (12975462): [] __do_softirq+0x2be/0x371 [11938.272103] softirqs last disabled at (12975451): [] irq_exit_rcu+0xb0/0xd0 [11938.280622] ---[ end trace 0000000000000000 ]--- [11957.676127] watchdog: BUG: soft lockup - CPU#7 stuck for 9019s! [kworker/7:4:78808] [11957.683802] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11957.739250] irq event stamp: 21850652 [11957.742907] hardirqs last enabled at (21850651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11957.753072] hardirqs last disabled at (21850652): [] sysvec_apic_timer_interrupt+0xe/0x90 [11957.762802] softirqs last enabled at (21850540): [] __do_softirq+0x2be/0x371 [11957.771489] softirqs last disabled at (21850535): [] irq_exit_rcu+0xb0/0xd0 [11957.780029] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11957.789405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11957.796876] Workqueue: events delayed_work [ceph] [11957.801599] RIP: 0010:lock_acquire+0xf0/0x2b0 [11957.805957] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11957.824694] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [11957.829912] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [11957.837037] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [11957.844160] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [11957.851286] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [11957.858408] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11957.865535] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11957.873611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11957.879348] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11957.886473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11957.893596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11957.900720] Call Trace: [11957.903169] [11957.905184] ? watchdog_timer_fn+0x1b4/0x220 [11957.909448] ? __hrtimer_run_queues+0x1d3/0x360 [11957.913975] ? hrtimer_interrupt+0x104/0x240 [11957.918235] ? __local_bh_enable+0x50/0x60 [11957.922330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11957.927555] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11957.932527] [11957.934624] [11957.936723] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11957.942031] ? lock_acquire+0xf0/0x2b0 [11957.945782] ? igrab+0x43/0x50 [11957.948839] ? lock_release+0xc5/0x260 [11957.952584] _raw_spin_lock+0x30/0x40 [11957.956243] ? igrab+0x1d/0x50 [11957.959291] igrab+0x1d/0x50 [11957.962172] ceph_check_delayed_caps+0xb7/0x270 [ceph] [11957.967321] delayed_work+0x1d8/0x280 [ceph] [11957.971601] ? process_scheduled_works+0x22a/0x510 [11957.976391] process_scheduled_works+0x2a4/0x510 [11957.981007] worker_thread+0x188/0x340 [11957.984757] ? rescuer_thread+0x310/0x310 [11957.988768] kthread+0xf5/0x130 [11957.991906] ? kthread_complete_and_exit+0x20/0x20 [11957.996690] ret_from_fork+0x31/0x40 [11958.000269] ? kthread_complete_and_exit+0x20/0x20 [11958.005053] ret_from_fork_asm+0x11/0x20 [11958.008983] [11961.664199] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [dbench:78683] [11961.671241] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11961.726691] irq event stamp: 12939340 [11961.730347] hardirqs last enabled at (12939339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11961.740515] hardirqs last disabled at (12939340): [] sysvec_apic_timer_interrupt+0xe/0x90 [11961.750247] softirqs last enabled at (12939310): [] __do_softirq+0x2be/0x371 [11961.758931] softirqs last disabled at (12939305): [] irq_exit_rcu+0xb0/0xd0 [11961.767452] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11961.776397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11961.783883] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [11961.789362] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [11961.808100] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [11961.813315] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [11961.820439] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [11961.827563] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [11961.834689] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [11961.841813] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [11961.848937] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [11961.857014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11961.862751] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [11961.869882] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11961.877010] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11961.884134] Call Trace: [11961.886596] [11961.888608] ? watchdog_timer_fn+0x1b4/0x220 [11961.892880] ? __hrtimer_run_queues+0x1d3/0x360 [11961.897415] ? hrtimer_interrupt+0x104/0x240 [11961.901684] ? __local_bh_enable+0x50/0x60 [11961.905778] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11961.911002] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11961.915957] [11961.918055] [11961.920153] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11961.925460] ? queued_spin_lock_slowpath+0x18/0x2e0 [11961.930338] do_raw_spin_lock+0xb5/0xc0 [11961.934170] __cap_delay_requeue+0xa7/0x1d0 [ceph] [11961.938979] try_get_cap_refs+0x159/0x880 [ceph] [11961.943611] __ceph_get_caps+0x4a1/0x760 [ceph] [11961.948150] ? __file_remove_privs+0x89/0x110 [11961.952507] ? wake_bit_function+0x70/0x70 [11961.956599] ? ceph_write_iter+0x260/0x770 [ceph] [11961.961312] ceph_write_iter+0x260/0x770 [ceph] [11961.965848] ? lock_is_held_type+0xaa/0x120 [11961.970033] vfs_write+0x324/0x4f0 [11961.973439] ksys_pwrite64+0x65/0xa0 [11961.977017] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11961.982156] do_syscall_64+0x38/0x80 [11961.985734] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11961.990780] RIP: 0033:0x7f48f0eeb93b [11961.994349] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [11962.013087] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [11962.020645] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [11962.027767] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [11962.034893] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [11962.042016] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [11962.049141] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [11962.056271] [11985.676621] watchdog: BUG: soft lockup - CPU#7 stuck for 9045s! [kworker/7:4:78808] [11985.684302] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11985.739746] irq event stamp: 21911000 [11985.743403] hardirqs last enabled at (21910999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11985.753562] hardirqs last disabled at (21911000): [] sysvec_apic_timer_interrupt+0xe/0x90 [11985.763294] softirqs last enabled at (21910982): [] __do_softirq+0x2be/0x371 [11985.771979] softirqs last disabled at (21910977): [] irq_exit_rcu+0xb0/0xd0 [11985.780489] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11985.789867] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11985.797337] Workqueue: events delayed_work [ceph] [11985.802053] RIP: 0010:lock_acquire+0xf0/0x2b0 [11985.806403] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [11985.825141] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [11985.830358] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [11985.837481] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [11985.844607] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [11985.851730] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [11985.858854] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [11985.865978] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [11985.874055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11985.879793] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [11985.886917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11985.894042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11985.901164] Call Trace: [11985.903611] [11985.905621] ? watchdog_timer_fn+0x1b4/0x220 [11985.909885] ? __hrtimer_run_queues+0x1d3/0x360 [11985.914411] ? hrtimer_interrupt+0x104/0x240 [11985.918672] ? __local_bh_enable+0x50/0x60 [11985.922765] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11985.927989] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11985.932946] [11985.935044] [11985.937142] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11985.942450] ? lock_acquire+0xf0/0x2b0 [11985.946200] ? igrab+0x43/0x50 [11985.949250] ? lock_release+0xc5/0x260 [11985.952996] _raw_spin_lock+0x30/0x40 [11985.956677] ? igrab+0x1d/0x50 [11985.959727] igrab+0x1d/0x50 [11985.962606] ceph_check_delayed_caps+0xb7/0x270 [ceph] [11985.967755] delayed_work+0x1d8/0x280 [ceph] [11985.972038] ? process_scheduled_works+0x22a/0x510 [11985.976827] process_scheduled_works+0x2a4/0x510 [11985.981442] worker_thread+0x188/0x340 [11985.985192] ? rescuer_thread+0x310/0x310 [11985.989195] kthread+0xf5/0x130 [11985.992342] ? kthread_complete_and_exit+0x20/0x20 [11985.997125] ret_from_fork+0x31/0x40 [11986.000706] ? kthread_complete_and_exit+0x20/0x20 [11986.005498] ret_from_fork_asm+0x11/0x20 [11986.009429] [11989.664692] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [dbench:78683] [11989.671729] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [11989.727170] irq event stamp: 12997656 [11989.730827] hardirqs last enabled at (12997655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11989.740984] hardirqs last disabled at (12997656): [] sysvec_apic_timer_interrupt+0xe/0x90 [11989.750709] softirqs last enabled at (12997530): [] __do_softirq+0x2be/0x371 [11989.759394] softirqs last disabled at (12997525): [] irq_exit_rcu+0xb0/0xd0 [11989.767905] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [11989.776849] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [11989.784321] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [11989.789807] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [11989.808543] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [11989.813761] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [11989.820885] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [11989.828010] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [11989.835132] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [11989.842257] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [11989.849380] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [11989.857458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11989.863196] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [11989.870321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11989.877445] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [11989.884569] Call Trace: [11989.887013] [11989.889025] ? watchdog_timer_fn+0x1b4/0x220 [11989.893289] ? __hrtimer_run_queues+0x1d3/0x360 [11989.897815] ? hrtimer_interrupt+0x104/0x240 [11989.902089] ? __local_bh_enable+0x50/0x60 [11989.906186] ? __sysvec_apic_timer_interrupt+0x61/0x160 [11989.911410] ? sysvec_apic_timer_interrupt+0x6d/0x90 [11989.916372] [11989.918473] [11989.920571] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [11989.925880] ? queued_spin_lock_slowpath+0x18/0x2e0 [11989.930757] do_raw_spin_lock+0xb5/0xc0 [11989.934588] __cap_delay_requeue+0xa7/0x1d0 [ceph] [11989.939392] try_get_cap_refs+0x159/0x880 [ceph] [11989.944019] __ceph_get_caps+0x4a1/0x760 [ceph] [11989.948561] ? __file_remove_privs+0x89/0x110 [11989.952917] ? wake_bit_function+0x70/0x70 [11989.957010] ? ceph_write_iter+0x260/0x770 [ceph] [11989.961720] ceph_write_iter+0x260/0x770 [ceph] [11989.966257] ? lock_is_held_type+0xaa/0x120 [11989.970442] vfs_write+0x324/0x4f0 [11989.973850] ksys_pwrite64+0x65/0xa0 [11989.977426] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [11989.982558] do_syscall_64+0x38/0x80 [11989.986136] entry_SYSCALL_64_after_hwframe+0x63/0xcd [11989.991179] RIP: 0033:0x7f48f0eeb93b [11989.994750] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [11990.013488] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [11990.021047] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [11990.028170] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [11990.035294] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [11990.042419] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [11990.049542] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [11990.056671] [12001.627269] rcu: INFO: rcu_preempt self-detected stall on CPU [12001.633033] rcu: 0-....: (64211 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=16211 [12001.643018] rcu: (t=65015 jiffies g=1309609 q=6958 ncpus=8) [12001.648668] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12001.657611] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12001.665082] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12001.670561] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12001.689297] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12001.694515] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12001.701638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12001.708764] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12001.715887] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12001.723012] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12001.730135] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12001.738214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12001.743950] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12001.751075] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12001.758199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12001.765322] Call Trace: [12001.767767] [12001.769778] ? rcu_dump_cpu_stacks+0xd1/0x180 [12001.774139] ? rcu_sched_clock_irq+0x633/0x1390 [12001.778671] ? lock_is_held_type+0xaa/0x120 [12001.782849] ? lock_is_held_type+0xaa/0x120 [12001.787026] ? tick_sched_do_timer+0xa0/0xa0 [12001.791298] ? rcu_read_lock_sched_held+0x56/0x90 [12001.795996] ? tick_sched_do_timer+0xa0/0xa0 [12001.800259] ? update_process_times+0x5f/0x90 [12001.804611] ? tick_sched_handle.isra.16+0x1e/0x50 [12001.809394] ? tick_sched_timer+0x6f/0x80 [12001.813400] ? __hrtimer_run_queues+0x1d3/0x360 [12001.817927] ? hrtimer_interrupt+0x104/0x240 [12001.822194] ? __local_bh_enable+0x50/0x60 [12001.826290] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12001.831511] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12001.836469] [12001.838567] [12001.840666] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12001.845982] ? queued_spin_lock_slowpath+0x18/0x2e0 [12001.850859] do_raw_spin_lock+0xb5/0xc0 [12001.854696] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12001.859498] try_get_cap_refs+0x159/0x880 [ceph] [12001.864131] __ceph_get_caps+0x4a1/0x760 [ceph] [12001.868671] ? __file_remove_privs+0x89/0x110 [12001.873026] ? wake_bit_function+0x70/0x70 [12001.877118] ? ceph_write_iter+0x260/0x770 [ceph] [12001.881832] ceph_write_iter+0x260/0x770 [ceph] [12001.886367] ? lock_is_held_type+0xaa/0x120 [12001.890554] vfs_write+0x324/0x4f0 [12001.893962] ksys_pwrite64+0x65/0xa0 [12001.897535] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12001.902667] do_syscall_64+0x38/0x80 [12001.906237] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12001.911281] RIP: 0033:0x7f48f0eeb93b [12001.914851] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12001.933589] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12001.941146] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12001.948272] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12001.955396] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12001.962518] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12001.969644] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12001.976773] [12001.978959] Sending NMI from CPU 0 to CPUs 7: [12001.983314] NMI backtrace for cpu 7 [12001.983315] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12001.983317] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12001.983318] Workqueue: events delayed_work [ceph] [12001.983334] RIP: 0010:__lock_acquire+0xfb/0x1e20 [12001.983336] Code: c1 fc 04 48 89 34 24 4c 0f af e0 8b 44 24 10 4c 8d 04 80 49 c1 e0 03 85 c0 74 25 41 f6 c5 01 75 1f 49 8d 74 30 d8 0f b7 46 20 <25> ff 1f 00 00 44 39 e0 75 0c 48 83 7c 24 08 00 0f 85 0c 07 00 00 [12001.983338] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [12001.983339] RAX: 0000000000000702 RBX: ffff88810debaa80 RCX: 0000000000000000 [12001.983340] RDX: 0000000000000000 RSI: ffff88810debb528 RDI: ffff8881a1d34518 [12001.983340] RBP: 0000000000000001 R08: 0000000000000078 R09: 0000000000000000 [12001.983341] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [12001.983342] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8881a1d34518 [12001.983342] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12001.983343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12001.983344] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12001.983345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12001.983346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12001.983346] Call Trace: [12001.983347] [12001.983348] ? nmi_cpu_backtrace+0xad/0x120 [12001.983352] ? nmi_cpu_backtrace_handler+0x11/0x20 [12001.983353] ? nmi_handle+0xd4/0x260 [12001.983357] ? default_do_nmi+0x49/0xf0 [12001.983359] ? exc_nmi+0x11e/0x150 [12001.983360] ? end_repeat_nmi+0x16/0x67 [12001.983364] ? __lock_acquire+0xfb/0x1e20 [12001.983365] ? __lock_acquire+0xfb/0x1e20 [12001.983367] ? __lock_acquire+0xfb/0x1e20 [12001.983368] [12001.983369] [12001.983370] lock_acquire+0xcb/0x2b0 [12001.983372] ? igrab+0x1d/0x50 [12001.983373] ? igrab+0x43/0x50 [12001.983375] ? lock_release+0xc5/0x260 [12001.983376] _raw_spin_lock+0x30/0x40 [12001.983378] ? igrab+0x1d/0x50 [12001.983379] igrab+0x1d/0x50 [12001.983381] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12001.983395] delayed_work+0x1d8/0x280 [ceph] [12001.983408] ? process_scheduled_works+0x22a/0x510 [12001.983411] process_scheduled_works+0x2a4/0x510 [12001.983414] worker_thread+0x188/0x340 [12001.983417] ? rescuer_thread+0x310/0x310 [12001.983419] kthread+0xf5/0x130 [12001.983421] ? kthread_complete_and_exit+0x20/0x20 [12001.983424] ret_from_fork+0x31/0x40 [12001.983426] ? kthread_complete_and_exit+0x20/0x20 [12001.983428] ret_from_fork_asm+0x11/0x20 [12001.983432] [12025.665327] watchdog: BUG: soft lockup - CPU#0 stuck for 85s! [dbench:78683] [12025.672375] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12025.727824] irq event stamp: 13072004 [12025.731481] hardirqs last enabled at (13072003): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12025.741648] hardirqs last disabled at (13072004): [] sysvec_apic_timer_interrupt+0xe/0x90 [12025.751373] softirqs last enabled at (13071904): [] __do_softirq+0x2be/0x371 [12025.760064] softirqs last disabled at (13071899): [] irq_exit_rcu+0xb0/0xd0 [12025.768587] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12025.777538] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12025.785006] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12025.790485] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12025.809224] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12025.814439] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12025.821564] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12025.828688] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12025.835812] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12025.842937] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12025.850060] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12025.858139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12025.863876] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12025.871016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12025.878141] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12025.885264] Call Trace: [12025.887727] [12025.889740] ? watchdog_timer_fn+0x1b4/0x220 [12025.894014] ? __hrtimer_run_queues+0x1d3/0x360 [12025.898547] ? hrtimer_interrupt+0x104/0x240 [12025.902815] ? __local_bh_enable+0x50/0x60 [12025.906909] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12025.912133] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12025.917091] [12025.919186] [12025.921285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12025.926592] ? queued_spin_lock_slowpath+0x18/0x2e0 [12025.931470] do_raw_spin_lock+0xb5/0xc0 [12025.935309] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12025.940113] try_get_cap_refs+0x159/0x880 [ceph] [12025.944744] __ceph_get_caps+0x4a1/0x760 [ceph] [12025.949284] ? __file_remove_privs+0x89/0x110 [12025.953638] ? wake_bit_function+0x70/0x70 [12025.957732] ? ceph_write_iter+0x260/0x770 [ceph] [12025.962443] ceph_write_iter+0x260/0x770 [ceph] [12025.966981] ? lock_is_held_type+0xaa/0x120 [12025.971166] vfs_write+0x324/0x4f0 [12025.974574] ksys_pwrite64+0x65/0xa0 [12025.978148] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12025.983281] do_syscall_64+0x38/0x80 [12025.986857] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12025.991904] RIP: 0033:0x7f48f0eeb93b [12025.995475] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12026.014211] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12026.021767] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12026.028891] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12026.036018] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12026.043139] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12026.050263] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12026.057395] [12029.677398] watchdog: BUG: soft lockup - CPU#7 stuck for 9086s! [kworker/7:4:78808] [12029.685078] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12029.740521] irq event stamp: 22005830 [12029.744178] hardirqs last enabled at (22005829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12029.754337] hardirqs last disabled at (22005830): [] sysvec_apic_timer_interrupt+0xe/0x90 [12029.764073] softirqs last enabled at (22005734): [] __do_softirq+0x2be/0x371 [12029.772760] softirqs last disabled at (22005729): [] irq_exit_rcu+0xb0/0xd0 [12029.781271] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12029.790651] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12029.798123] Workqueue: events delayed_work [ceph] [12029.802836] RIP: 0010:do_raw_spin_unlock+0x42/0xc0 [12029.807619] Code: 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 8b 05 4a bf eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 10 <89> 43 08 c6 03 00 5b c3 e8 41 21 45 00 85 c0 74 c3 48 c7 c6 29 e2 [12029.826356] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [12029.831576] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [12029.838699] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [12029.845823] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [12029.852947] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12029.860072] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12029.867194] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12029.875273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12029.881010] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12029.888135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12029.895257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12029.902383] Call Trace: [12029.904827] [12029.906838] ? watchdog_timer_fn+0x1b4/0x220 [12029.911102] ? __hrtimer_run_queues+0x1d3/0x360 [12029.915628] ? hrtimer_interrupt+0x104/0x240 [12029.919889] ? __local_bh_enable+0x50/0x60 [12029.923983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12029.929207] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12029.934164] [12029.936260] [12029.938358] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12029.943668] ? do_raw_spin_unlock+0x42/0xc0 [12029.947851] _raw_spin_unlock+0x23/0x40 [12029.951689] igrab+0x43/0x50 [12029.954567] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12029.959717] delayed_work+0x1d8/0x280 [ceph] [12029.963999] ? process_scheduled_works+0x22a/0x510 [12029.968796] process_scheduled_works+0x2a4/0x510 [12029.973412] worker_thread+0x188/0x340 [12029.977161] ? rescuer_thread+0x310/0x310 [12029.981165] kthread+0xf5/0x130 [12029.984311] ? kthread_complete_and_exit+0x20/0x20 [12029.989094] ret_from_fork+0x31/0x40 [12029.992674] ? kthread_complete_and_exit+0x20/0x20 [12029.997459] ret_from_fork_asm+0x11/0x20 [12030.001389] [12053.665821] watchdog: BUG: soft lockup - CPU#0 stuck for 111s! [dbench:78683] [12053.672948] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12053.728389] irq event stamp: 13130152 [12053.732046] hardirqs last enabled at (13130151): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12053.742215] hardirqs last disabled at (13130152): [] sysvec_apic_timer_interrupt+0xe/0x90 [12053.751937] softirqs last enabled at (13130150): [] __do_softirq+0x2be/0x371 [12053.760621] softirqs last disabled at (13130145): [] irq_exit_rcu+0xb0/0xd0 [12053.769133] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12053.778076] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12053.785546] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12053.791024] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12053.809761] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12053.814979] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12053.822103] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12053.829228] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12053.836352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12053.843476] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12053.850600] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12053.858677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12053.864414] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12053.871540] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12053.878664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12053.885786] Call Trace: [12053.888232] [12053.890243] ? watchdog_timer_fn+0x1b4/0x220 [12053.894517] ? __hrtimer_run_queues+0x1d3/0x360 [12053.899041] ? hrtimer_interrupt+0x104/0x240 [12053.903303] ? __local_bh_enable+0x50/0x60 [12053.907395] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12053.912620] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12053.917579] [12053.919674] [12053.921773] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12053.927079] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12053.931959] do_raw_spin_lock+0xb5/0xc0 [12053.935796] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12053.940599] try_get_cap_refs+0x159/0x880 [ceph] [12053.945228] __ceph_get_caps+0x4a1/0x760 [ceph] [12053.949769] ? __file_remove_privs+0x89/0x110 [12053.954126] ? wake_bit_function+0x70/0x70 [12053.958220] ? ceph_write_iter+0x260/0x770 [ceph] [12053.962931] ceph_write_iter+0x260/0x770 [ceph] [12053.967467] ? lock_is_held_type+0xaa/0x120 [12053.971651] vfs_write+0x324/0x4f0 [12053.975061] ksys_pwrite64+0x65/0xa0 [12053.978636] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12053.983767] do_syscall_64+0x38/0x80 [12053.987338] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12053.992381] RIP: 0033:0x7f48f0eeb93b [12053.995953] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12054.014690] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12054.022246] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12054.029369] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12054.036503] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12054.043628] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12054.050751] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12054.057880] [12057.677892] watchdog: BUG: soft lockup - CPU#7 stuck for 9112s! [kworker/7:4:78808] [12057.685542] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12057.741010] irq event stamp: 22065724 [12057.744666] hardirqs last enabled at (22065723): [] asm_common_interrupt+0x26/0x40 [12057.753878] hardirqs last disabled at (22065724): [] sysvec_apic_timer_interrupt+0xe/0x90 [12057.763602] softirqs last enabled at (22065722): [] __do_softirq+0x2be/0x371 [12057.772296] softirqs last disabled at (22065717): [] irq_exit_rcu+0xb0/0xd0 [12057.780815] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12057.790192] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12057.797664] Workqueue: events delayed_work [ceph] [12057.802389] RIP: 0010:lock_acquire+0xf0/0x2b0 [12057.806747] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12057.825483] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12057.830701] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12057.837826] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12057.844948] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12057.852074] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12057.859198] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12057.866322] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12057.874399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12057.880135] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12057.887263] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12057.894402] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12057.901526] Call Trace: [12057.903972] [12057.905982] ? watchdog_timer_fn+0x1b4/0x220 [12057.910245] ? __hrtimer_run_queues+0x1d3/0x360 [12057.914773] ? hrtimer_interrupt+0x104/0x240 [12057.919041] ? __local_bh_enable+0x50/0x60 [12057.923137] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12057.928360] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12057.933317] [12057.935415] [12057.937512] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12057.942818] ? lock_acquire+0xf0/0x2b0 [12057.946572] ? igrab+0x43/0x50 [12057.949627] ? lock_release+0xc5/0x260 [12057.953373] _raw_spin_lock+0x30/0x40 [12057.957031] ? igrab+0x1d/0x50 [12057.960081] igrab+0x1d/0x50 [12057.962959] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12057.968110] delayed_work+0x1d8/0x280 [ceph] [12057.972390] ? process_scheduled_works+0x22a/0x510 [12057.977179] process_scheduled_works+0x2a4/0x510 [12057.981795] worker_thread+0x188/0x340 [12057.985545] ? rescuer_thread+0x310/0x310 [12057.989570] kthread+0xf5/0x130 [12057.992713] ? kthread_complete_and_exit+0x20/0x20 [12057.997503] ret_from_fork+0x31/0x40 [12058.001073] ? kthread_complete_and_exit+0x20/0x20 [12058.005858] ret_from_fork_asm+0x11/0x20 [12058.009790] [12081.666316] watchdog: BUG: soft lockup - CPU#0 stuck for 137s! [dbench:78683] [12081.673443] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12081.728895] irq event stamp: 13188464 [12081.732552] hardirqs last enabled at (13188463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12081.742717] hardirqs last disabled at (13188464): [] sysvec_apic_timer_interrupt+0xe/0x90 [12081.752441] softirqs last enabled at (13188462): [] __do_softirq+0x2be/0x371 [12081.761126] softirqs last disabled at (13188457): [] irq_exit_rcu+0xb0/0xd0 [12081.769647] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12081.778590] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12081.786060] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12081.791539] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12081.810275] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12081.815493] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12081.822617] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12081.829741] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12081.836865] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12081.844000] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12081.851122] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12081.859200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12081.864936] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12081.872060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12081.879187] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12081.886309] Call Trace: [12081.888756] [12081.890767] ? watchdog_timer_fn+0x1b4/0x220 [12081.895037] ? __hrtimer_run_queues+0x1d3/0x360 [12081.899563] ? hrtimer_interrupt+0x104/0x240 [12081.903824] ? __local_bh_enable+0x50/0x60 [12081.907918] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12081.913143] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12081.918101] [12081.920197] [12081.922295] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12081.927603] ? queued_spin_lock_slowpath+0x16/0x2e0 [12081.932480] do_raw_spin_lock+0xb5/0xc0 [12081.936319] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12081.941131] try_get_cap_refs+0x159/0x880 [ceph] [12081.945761] __ceph_get_caps+0x4a1/0x760 [ceph] [12081.950302] ? __file_remove_privs+0x89/0x110 [12081.954656] ? wake_bit_function+0x70/0x70 [12081.958749] ? ceph_write_iter+0x260/0x770 [ceph] [12081.963461] ceph_write_iter+0x260/0x770 [ceph] [12081.967999] ? lock_is_held_type+0xaa/0x120 [12081.972183] vfs_write+0x324/0x4f0 [12081.975591] ksys_pwrite64+0x65/0xa0 [12081.979167] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12081.984307] do_syscall_64+0x38/0x80 [12081.987887] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12081.992930] RIP: 0033:0x7f48f0eeb93b [12081.996500] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12082.015237] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12082.022795] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12082.029917] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12082.037043] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12082.044167] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12082.051291] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12082.058420] [12085.678386] watchdog: BUG: soft lockup - CPU#7 stuck for 9138s! [kworker/7:4:78808] [12085.686059] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12085.741506] irq event stamp: 22125930 [12085.745161] hardirqs last enabled at (22125929): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12085.755327] hardirqs last disabled at (22125930): [] sysvec_apic_timer_interrupt+0xe/0x90 [12085.765060] softirqs last enabled at (22125910): [] __do_softirq+0x2be/0x371 [12085.773745] softirqs last disabled at (22125905): [] irq_exit_rcu+0xb0/0xd0 [12085.782255] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12085.791634] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12085.799105] Workqueue: events delayed_work [ceph] [12085.803818] RIP: 0010:rcu_is_watching+0xb/0x40 [12085.808255] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [12085.826994] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [12085.832211] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12085.839334] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [12085.846460] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [12085.853584] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [12085.860707] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12085.867832] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12085.875911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12085.881648] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12085.888782] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12085.895903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12085.903027] Call Trace: [12085.905472] [12085.907485] ? watchdog_timer_fn+0x1b4/0x220 [12085.911749] ? __hrtimer_run_queues+0x1d3/0x360 [12085.916274] ? hrtimer_interrupt+0x104/0x240 [12085.920545] ? __local_bh_enable+0x50/0x60 [12085.924637] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12085.929860] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12085.934817] [12085.936914] [12085.939012] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12085.944317] ? igrab+0x43/0x50 [12085.947369] ? igrab+0x43/0x50 [12085.950419] ? rcu_is_watching+0xb/0x40 [12085.954249] lock_release+0x1d7/0x260 [12085.957909] _raw_spin_unlock+0x1b/0x40 [12085.961747] igrab+0x43/0x50 [12085.964625] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12085.969787] delayed_work+0x1d8/0x280 [ceph] [12085.974067] ? process_scheduled_works+0x22a/0x510 [12085.978855] process_scheduled_works+0x2a4/0x510 [12085.983471] worker_thread+0x188/0x340 [12085.987220] ? rescuer_thread+0x310/0x310 [12085.991222] kthread+0xf5/0x130 [12085.994360] ? kthread_complete_and_exit+0x20/0x20 [12085.999143] ret_from_fork+0x31/0x40 [12086.002724] ? kthread_complete_and_exit+0x20/0x20 [12086.007515] ret_from_fork_asm+0x11/0x20 [12086.011437] [12109.666809] watchdog: BUG: soft lockup - CPU#0 stuck for 164s! [dbench:78683] [12109.673941] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12109.729383] irq event stamp: 13246770 [12109.733039] hardirqs last enabled at (13246769): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12109.743204] hardirqs last disabled at (13246770): [] sysvec_apic_timer_interrupt+0xe/0x90 [12109.752928] softirqs last enabled at (13246768): [] __do_softirq+0x2be/0x371 [12109.761613] softirqs last disabled at (13246763): [] irq_exit_rcu+0xb0/0xd0 [12109.770126] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12109.779069] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12109.786540] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12109.792018] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12109.810755] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12109.815972] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12109.823094] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12109.830219] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12109.837344] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12109.844485] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12109.851610] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12109.859688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12109.865424] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12109.872549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12109.879672] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12109.886797] Call Trace: [12109.889241] [12109.891252] ? watchdog_timer_fn+0x1b4/0x220 [12109.895517] ? __hrtimer_run_queues+0x1d3/0x360 [12109.900043] ? hrtimer_interrupt+0x104/0x240 [12109.904305] ? __local_bh_enable+0x50/0x60 [12109.908397] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12109.913622] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12109.918580] [12109.920676] [12109.922775] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12109.928091] ? queued_spin_lock_slowpath+0x16/0x2e0 [12109.932968] do_raw_spin_lock+0xb5/0xc0 [12109.936804] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12109.941609] try_get_cap_refs+0x159/0x880 [ceph] [12109.946238] __ceph_get_caps+0x4a1/0x760 [ceph] [12109.950778] ? __file_remove_privs+0x89/0x110 [12109.955134] ? wake_bit_function+0x70/0x70 [12109.959227] ? ceph_write_iter+0x260/0x770 [ceph] [12109.963941] ceph_write_iter+0x260/0x770 [ceph] [12109.968493] ? lock_is_held_type+0xaa/0x120 [12109.972670] vfs_write+0x324/0x4f0 [12109.976078] ksys_pwrite64+0x65/0xa0 [12109.979653] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12109.984784] do_syscall_64+0x38/0x80 [12109.988356] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12109.993399] RIP: 0033:0x7f48f0eeb93b [12109.996970] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12110.015706] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12110.023265] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12110.030390] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12110.037514] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12110.044637] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12110.051761] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12110.058891] [12113.678880] watchdog: BUG: soft lockup - CPU#7 stuck for 9164s! [kworker/7:4:78808] [12113.686554] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12113.742002] irq event stamp: 22186372 [12113.745657] hardirqs last enabled at (22186371): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12113.755816] hardirqs last disabled at (22186372): [] sysvec_apic_timer_interrupt+0xe/0x90 [12113.765554] softirqs last enabled at (22186354): [] __do_softirq+0x2be/0x371 [12113.774240] softirqs last disabled at (22186349): [] irq_exit_rcu+0xb0/0xd0 [12113.782753] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12113.792129] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12113.799600] Workqueue: events delayed_work [ceph] [12113.804316] RIP: 0010:lock_acquire+0xf0/0x2b0 [12113.808666] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12113.827403] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12113.832621] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12113.839744] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12113.846870] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12113.853993] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12113.861118] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12113.868240] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12113.876320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12113.882056] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12113.889180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12113.896304] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12113.903429] Call Trace: [12113.905872] [12113.907884] ? watchdog_timer_fn+0x1b4/0x220 [12113.912147] ? __hrtimer_run_queues+0x1d3/0x360 [12113.916676] ? hrtimer_interrupt+0x104/0x240 [12113.920946] ? __local_bh_enable+0x50/0x60 [12113.925037] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12113.930254] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12113.935226] [12113.937325] [12113.939422] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12113.944729] ? lock_acquire+0xf0/0x2b0 [12113.948480] ? igrab+0x43/0x50 [12113.951531] ? lock_release+0xc5/0x260 [12113.955278] _raw_spin_lock+0x30/0x40 [12113.958940] ? igrab+0x1d/0x50 [12113.961992] igrab+0x1d/0x50 [12113.964869] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12113.970018] delayed_work+0x1d8/0x280 [ceph] [12113.974301] ? process_scheduled_works+0x22a/0x510 [12113.979089] process_scheduled_works+0x2a4/0x510 [12113.983705] worker_thread+0x188/0x340 [12113.987454] ? rescuer_thread+0x310/0x310 [12113.991459] kthread+0xf5/0x130 [12113.994596] ? kthread_complete_and_exit+0x20/0x20 [12113.999379] ret_from_fork+0x31/0x40 [12114.002951] ? kthread_complete_and_exit+0x20/0x20 [12114.007735] ret_from_fork_asm+0x11/0x20 [12114.011658] [12137.667303] watchdog: BUG: soft lockup - CPU#0 stuck for 190s! [dbench:78683] [12137.674435] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12137.729877] irq event stamp: 13305010 [12137.733535] hardirqs last enabled at (13305009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12137.743698] hardirqs last disabled at (13305010): [] sysvec_apic_timer_interrupt+0xe/0x90 [12137.753426] softirqs last enabled at (13305008): [] __do_softirq+0x2be/0x371 [12137.762108] softirqs last disabled at (13305003): [] irq_exit_rcu+0xb0/0xd0 [12137.770619] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12137.779563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12137.787035] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12137.792513] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12137.811251] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12137.816468] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12137.823592] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12137.830716] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12137.837840] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12137.844965] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12137.852107] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12137.860183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12137.865921] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12137.873045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12137.880169] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12137.887294] Call Trace: [12137.889738] [12137.891748] ? watchdog_timer_fn+0x1b4/0x220 [12137.896012] ? __hrtimer_run_queues+0x1d3/0x360 [12137.900538] ? hrtimer_interrupt+0x104/0x240 [12137.904809] ? __local_bh_enable+0x50/0x60 [12137.908903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12137.914126] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12137.919082] [12137.921181] [12137.923278] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12137.928586] ? queued_spin_lock_slowpath+0x18/0x2e0 [12137.933464] do_raw_spin_lock+0xb5/0xc0 [12137.937304] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12137.942104] try_get_cap_refs+0x159/0x880 [ceph] [12137.946734] __ceph_get_caps+0x4a1/0x760 [ceph] [12137.951277] ? __file_remove_privs+0x89/0x110 [12137.955631] ? wake_bit_function+0x70/0x70 [12137.959724] ? ceph_write_iter+0x260/0x770 [ceph] [12137.964436] ceph_write_iter+0x260/0x770 [ceph] [12137.968990] ? lock_is_held_type+0xaa/0x120 [12137.973175] vfs_write+0x324/0x4f0 [12137.976583] ksys_pwrite64+0x65/0xa0 [12137.980159] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12137.985289] do_syscall_64+0x38/0x80 [12137.988861] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12137.993903] RIP: 0033:0x7f48f0eeb93b [12137.997474] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12138.016211] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12138.023769] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12138.030894] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12138.038017] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12138.045142] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12138.052266] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12138.059395] [12141.679374] watchdog: BUG: soft lockup - CPU#7 stuck for 9190s! [kworker/7:4:78808] [12141.687048] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12141.742497] irq event stamp: 22247064 [12141.746153] hardirqs last enabled at (22247063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12141.756321] hardirqs last disabled at (22247064): [] sysvec_apic_timer_interrupt+0xe/0x90 [12141.766054] softirqs last enabled at (22247046): [] __do_softirq+0x2be/0x371 [12141.774745] softirqs last disabled at (22247041): [] irq_exit_rcu+0xb0/0xd0 [12141.783276] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12141.792651] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12141.800124] Workqueue: events delayed_work [ceph] [12141.804845] RIP: 0010:rcu_is_watching+0x1c/0x40 [12141.809371] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 [12141.828106] RSP: 0018:ffffc90002117d10 EFLAGS: 00000286 [12141.833324] RAX: 0000000001f06464 RBX: ffff8881a1d34500 RCX: 0000000000000000 [12141.840449] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [12141.847573] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [12141.854696] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [12141.861822] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12141.868947] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12141.877023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12141.882777] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12141.889902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12141.897026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12141.904168] Call Trace: [12141.906613] [12141.908624] ? watchdog_timer_fn+0x1b4/0x220 [12141.912895] ? __hrtimer_run_queues+0x1d3/0x360 [12141.917422] ? hrtimer_interrupt+0x104/0x240 [12141.921683] ? __local_bh_enable+0x50/0x60 [12141.925777] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12141.931001] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12141.935958] [12141.938055] [12141.940152] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12141.945460] ? rcu_is_watching+0x1c/0x40 [12141.949384] lock_acquire+0x184/0x2b0 [12141.953055] ? igrab+0x43/0x50 [12141.956109] ? lock_release+0xc5/0x260 [12141.959855] _raw_spin_lock+0x30/0x40 [12141.963519] ? igrab+0x1d/0x50 [12141.966570] igrab+0x1d/0x50 [12141.969449] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12141.974599] delayed_work+0x1d8/0x280 [ceph] [12141.978879] ? process_scheduled_works+0x22a/0x510 [12141.983668] process_scheduled_works+0x2a4/0x510 [12141.988284] worker_thread+0x188/0x340 [12141.992049] ? rescuer_thread+0x310/0x310 [12141.996055] kthread+0xf5/0x130 [12141.999200] ? kthread_complete_and_exit+0x20/0x20 [12142.003984] ret_from_fork+0x31/0x40 [12142.007565] ? kthread_complete_and_exit+0x20/0x20 [12142.012348] ret_from_fork_asm+0x11/0x20 [12142.016280] [12165.667797] watchdog: BUG: soft lockup - CPU#0 stuck for 216s! [dbench:78683] [12165.674925] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12165.730374] irq event stamp: 13363246 [12165.734030] hardirqs last enabled at (13363245): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12165.744192] hardirqs last disabled at (13363246): [] sysvec_apic_timer_interrupt+0xe/0x90 [12165.753921] softirqs last enabled at (13363244): [] __do_softirq+0x2be/0x371 [12165.762606] softirqs last disabled at (13363239): [] irq_exit_rcu+0xb0/0xd0 [12165.771116] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12165.780063] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12165.787532] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12165.793010] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12165.811747] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12165.816965] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12165.824088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12165.831213] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12165.838336] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12165.845461] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12165.852602] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12165.860680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12165.866417] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12165.873541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12165.880665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12165.887789] Call Trace: [12165.890234] [12165.892244] ? watchdog_timer_fn+0x1b4/0x220 [12165.896509] ? __hrtimer_run_queues+0x1d3/0x360 [12165.901036] ? hrtimer_interrupt+0x104/0x240 [12165.905304] ? __local_bh_enable+0x50/0x60 [12165.909397] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12165.914623] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12165.919579] [12165.921677] [12165.923774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12165.929081] ? queued_spin_lock_slowpath+0x18/0x2e0 [12165.933961] do_raw_spin_lock+0xb5/0xc0 [12165.937799] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12165.942601] try_get_cap_refs+0x159/0x880 [ceph] [12165.947231] __ceph_get_caps+0x4a1/0x760 [ceph] [12165.951774] ? __file_remove_privs+0x89/0x110 [12165.956127] ? wake_bit_function+0x70/0x70 [12165.960221] ? ceph_write_iter+0x260/0x770 [ceph] [12165.964933] ceph_write_iter+0x260/0x770 [ceph] [12165.969469] ? lock_is_held_type+0xaa/0x120 [12165.973672] vfs_write+0x324/0x4f0 [12165.977080] ksys_pwrite64+0x65/0xa0 [12165.980656] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12165.985796] do_syscall_64+0x38/0x80 [12165.989375] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12165.994417] RIP: 0033:0x7f48f0eeb93b [12165.997990] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12166.016726] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12166.024282] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12166.031406] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12166.038532] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12166.045656] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12166.052780] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12166.059908] [12167.503249] INFO: task jbd2/sda1-8:470 blocked for more than 122 seconds. [12167.510152] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12167.516931] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12167.524759] task:jbd2/sda1-8 state:D stack:0 pid:470 ppid:2 flags:0x00004000 [12167.533103] Call Trace: [12167.535548] [12167.537655] __schedule+0x379/0xe70 [12167.541146] ? find_held_lock+0x2d/0x90 [12167.544988] ? out_of_line_wait_on_bit_timeout+0xd0/0xd0 [12167.550297] schedule+0x54/0xc0 [12167.553434] io_schedule+0x16/0x40 [12167.556834] bit_wait_io+0x11/0x60 [12167.560239] __wait_on_bit+0x75/0x90 [12167.563822] out_of_line_wait_on_bit+0x98/0xc0 [12167.568269] ? __wake_up_locked_sync_key+0x20/0x20 [12167.573058] jbd2_journal_commit_transaction+0x1761/0x1bd0 [12167.578558] kjournald2+0xcd/0x2a0 [12167.581959] ? sched_numa_warn+0xf0/0xf0 [12167.585885] ? jbd2_seq_info_stop+0x10/0x10 [12167.590070] kthread+0xf5/0x130 [12167.593218] ? kthread_complete_and_exit+0x20/0x20 [12167.598011] ret_from_fork+0x31/0x40 [12167.601587] ? kthread_complete_and_exit+0x20/0x20 [12167.606381] ret_from_fork_asm+0x11/0x20 [12167.610312] [12167.612503] INFO: task NetworkManager:806 blocked for more than 122 seconds. [12167.619546] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12167.626333] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12167.634161] task:NetworkManager state:D stack:0 pid:806 ppid:1 flags:0x00004000 [12167.642504] Call Trace: [12167.644948] [12167.647048] __schedule+0x379/0xe70 [12167.650544] schedule+0x54/0xc0 [12167.653683] jbd2_log_wait_commit+0x124/0x190 [12167.658038] ? sched_numa_warn+0xf0/0xf0 [12167.661965] ext4_fc_commit+0x2b9/0x8d0 [12167.665803] ? jbd2_trans_will_send_data_barrier+0x35/0xa0 [12167.671288] ? find_held_lock+0x2d/0x90 [12167.675126] ? jbd2_trans_will_send_data_barrier+0x83/0xa0 [12167.680614] ? lock_release+0xc5/0x260 [12167.684370] ext4_sync_file+0x224/0x3d0 [12167.688207] ? __fget_files+0xce/0x1b0 [12167.691959] do_fsync+0x38/0x70 [12167.695105] __x64_sys_fsync+0x14/0x20 [12167.698856] do_syscall_64+0x38/0x80 [12167.702436] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12167.707489] RIP: 0033:0x7f8d54c11e57 [12167.711068] RSP: 002b:00007ffef3f2a520 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [12167.718635] RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 00007f8d54c11e57 [12167.725767] RDX: 0000000000000000 RSI: 00007ffef3f2a550 RDI: 0000000000000018 [12167.732898] RBP: 0000000000000018 R08: 0000000000000000 R09: 0000000000000006 [12167.740031] R10: 00000000000000fd R11: 0000000000000293 R12: 00007f8d58b7e088 [12167.747155] R13: 000056016a932050 R14: 000056016a8d5f4d R15: 0000000000000000 [12167.754285] [12167.756478] INFO: task logrotate:85883 blocked for more than 123 seconds. [12167.763260] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12167.770045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12167.777873] task:logrotate state:D stack:0 pid:85883 ppid:85867 flags:0x00004000 [12167.786217] Call Trace: [12167.788662] [12167.790762] __schedule+0x379/0xe70 [12167.794255] schedule+0x54/0xc0 [12167.797402] jbd2_log_wait_commit+0x124/0x190 [12167.801760] ? sched_numa_warn+0xf0/0xf0 [12167.805687] ext4_fc_commit+0x2b9/0x8d0 [12167.809524] ? jbd2_trans_will_send_data_barrier+0x35/0xa0 [12167.815008] ? find_held_lock+0x2d/0x90 [12167.818849] ? jbd2_trans_will_send_data_barrier+0x83/0xa0 [12167.824336] ? lock_release+0xc5/0x260 [12167.828093] ext4_sync_file+0x224/0x3d0 [12167.831929] do_fsync+0x38/0x70 [12167.835074] __x64_sys_fsync+0x14/0x20 [12167.838824] do_syscall_64+0x38/0x80 [12167.842406] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12167.847457] RIP: 0033:0x7ff2606f4478 [12167.851037] RSP: 002b:00007ffcb78e6c28 EFLAGS: 00000246 ORIG_RAX: 000000000000004a [12167.858603] RAX: ffffffffffffffda RBX: 00005593995f6f0d RCX: 00007ff2606f4478 [12167.865726] RDX: 0000000000000000 RSI: 00005593995eb3c0 RDI: 0000000000000003 [12167.872850] RBP: 0000000000000000 R08: 00005593995eb3c0 R09: 00007ff261a04800 [12167.879975] R10: 0000000000000000 R11: 0000000000000246 R12: 00005593995eb2e0 [12167.887099] R13: 00007ffcb78e6ca0 R14: 0000000000000040 R15: 0000000000000000 [12167.894231] [12167.896417] [12167.896417] Showing all locks held in the system: [12167.902596] 1 lock held by khungtaskd/57: [12167.906609] #0: ffffffff8255ece0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x36/0x1c0 [12167.915563] 1 lock held by systemd-journal/568: [12167.920094] #0: ffff88885fcef018 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0xf7/0xe70 [12167.928011] 4 locks held by dbench/78683: [12167.932020] 3 locks held by kworker/7:4/78808: [12167.936466] [12167.937964] ============================================= [12167.937964] [12193.668292] watchdog: BUG: soft lockup - CPU#0 stuck for 242s! [dbench:78683] [12193.675421] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12193.680291] watchdog: BUG: soft lockup - CPU#7 stuck for 9239s! [kworker/7:4:78808] [12193.730861] irq event stamp: 13421360 [12193.730862] hardirqs last enabled at (13421359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12193.738505] Modules linked in: [12193.742161] hardirqs last disabled at (13421360): [] sysvec_apic_timer_interrupt+0xe/0x90 [12193.752321] ceph [12193.755372] softirqs last enabled at (13421334): [] __do_softirq+0x2be/0x371 [12193.765095] libceph [12193.767019] softirqs last disabled at (13421329): [] irq_exit_rcu+0xb0/0xd0 [12193.775704] dns_resolver [12193.777888] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12193.786397] fscache netfs [12193.789015] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12193.797975] veth nft_chain_nat [12193.800685] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12193.808152] xt_MASQUERADE nf_nat [12193.811290] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12193.816766] nf_conntrack nf_defrag_ipv6 [12193.820077] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12193.838815] nf_defrag_ipv4 nft_compat [12193.842732] [12193.842732] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12193.847966] nf_tables nfnetlink [12193.851712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12193.853201] bridge stp [12193.860327] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12193.863551] llc xfs [12193.870688] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12193.873136] libcrc32c [12193.880259] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12193.882443] sunrpc [12193.889569] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12193.891925] intel_rapl_msr intel_rapl_common [12193.899048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12193.901146] sb_edac x86_pkg_temp_thermal [12193.909225] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12193.913576] coretemp kvm_intel [12193.919313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12193.923318] kvm irqbypass [12193.930442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12193.933578] i2c_i801 iTCO_wdt [12193.940702] Call Trace: [12193.940703] [12193.943407] iTCO_vendor_support i2c_smbus [12193.950533] ? watchdog_timer_fn+0x1b4/0x220 [12193.953580] crct10dif_pclmul joydev [12193.956026] ? __hrtimer_run_queues+0x1d3/0x360 [12193.958037] crc32_pclmul ipmi_ssif [12193.962129] ? hrtimer_interrupt+0x104/0x240 [12193.966391] ghash_clmulni_intel [12193.969962] ? __local_bh_enable+0x50/0x60 [12193.974503] mei_me mei [12193.977990] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12193.982250] lpc_ich sha512_ssse3 [12193.985477] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12193.989566] acpi_ipmi [12193.992009] [12193.997226] mfd_core wmi [12194.000539] [12194.000540] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12194.005494] ipmi_si ipmi_devintf [12194.007856] ? queued_spin_lock_slowpath+0x16/0x2e0 [12194.009951] ipmi_msghandler ioatdma [12194.012569] do_raw_spin_lock+0xb5/0xc0 [12194.014665] acpi_pad acpi_power_meter [12194.019988] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12194.023297] ixgbe igb [12194.028169] try_get_cap_refs+0x159/0x880 [ceph] [12194.031738] mdio crc32c_intel [12194.035575] __ceph_get_caps+0x4a1/0x760 [ceph] [12194.039314] nvme ptp [12194.044102] ? __file_remove_privs+0x89/0x110 [12194.046454] nvme_core i2c_algo_bit [12194.051066] ? wake_bit_function+0x70/0x70 [12194.054117] pps_core dca [12194.058644] ? ceph_write_iter+0x260/0x770 [ceph] [12194.060911] fuse [12194.065262] ceph_write_iter+0x260/0x770 [ceph] [12194.068746] [12194.072840] ? lock_is_held_type+0xaa/0x120 [12194.075454] irq event stamp: 22359514 [12194.080154] vfs_write+0x324/0x4f0 [12194.082076] hardirqs last enabled at (22359513): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12194.086605] ksys_pwrite64+0x65/0xa0 [12194.088090] hardirqs last disabled at (22359514): [] sysvec_apic_timer_interrupt+0xe/0x90 [12194.092268] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12194.095924] softirqs last enabled at (22359510): [] __do_softirq+0x2be/0x371 [12194.099323] do_syscall_64+0x38/0x80 [12194.109481] softirqs last disabled at (22359505): [] irq_exit_rcu+0xb0/0xd0 [12194.113051] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12194.122776] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12194.127905] RIP: 0033:0x7f48f0eeb93b [12194.136589] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12194.140161] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12194.148690] Workqueue: events delayed_work [ceph] [12194.153734] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12194.163110] [12194.163111] RIP: 0010:_raw_spin_lock+0x30/0x40 [12194.166696] ORIG_RAX: 0000000000000012 [12194.174171] Code: 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c 5c ff <48> 89 df 58 5b e9 76 33 5c ff 66 0f 1f 44 00 00 f3 0f 1e fa 0f 1f [12194.192908] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12194.197605] RSP: 0018:ffffc90002117d88 EFLAGS: 00000296 [12194.202821] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12194.204313] [12194.204314] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12194.208749] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12194.212580] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12194.231318] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12194.238442] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [12194.243660] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12194.250783] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12194.252281] [12194.259399] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12194.311453] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12194.319530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12194.325266] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12194.332391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12194.339515] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12194.346640] Call Trace: [12194.349085] [12194.351096] ? watchdog_timer_fn+0x1b4/0x220 [12194.355358] ? __hrtimer_run_queues+0x1d3/0x360 [12194.359885] ? hrtimer_interrupt+0x104/0x240 [12194.364146] ? __local_bh_enable+0x50/0x60 [12194.368239] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12194.373464] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12194.378420] [12194.380518] [12194.382616] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12194.387923] ? _raw_spin_lock+0x30/0x40 [12194.391761] ? _raw_spin_lock+0x30/0x40 [12194.395598] ? igrab+0x1d/0x50 [12194.398649] igrab+0x1d/0x50 [12194.401528] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12194.406694] delayed_work+0x1d8/0x280 [ceph] [12194.410981] ? process_scheduled_works+0x22a/0x510 [12194.415765] process_scheduled_works+0x2a4/0x510 [12194.420381] worker_thread+0x188/0x340 [12194.424132] ? rescuer_thread+0x310/0x310 [12194.428143] kthread+0xf5/0x130 [12194.431281] ? kthread_complete_and_exit+0x20/0x20 [12194.436066] ret_from_fork+0x31/0x40 [12194.439643] ? kthread_complete_and_exit+0x20/0x20 [12194.444429] ret_from_fork_asm+0x11/0x20 [12194.448359] [12196.990717] rcu: INFO: rcu_preempt self-detected stall on CPU [12196.996479] rcu: 7-....: (257019 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/489288 fqs=99367 [12197.006549] rcu: (t=260375 jiffies g=1309609 q=69652 ncpus=8) [12197.012373] Sending NMI from CPU 7 to CPUs 0: [12197.016725] NMI backtrace for cpu 0 [12197.016727] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12197.016729] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12197.016729] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12197.016733] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12197.016734] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12197.016736] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12197.016737] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12197.016737] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12197.016738] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12197.016739] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12197.016740] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12197.016741] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12197.016742] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12197.016743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12197.016744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12197.016744] Call Trace: [12197.016745] [12197.016747] ? nmi_cpu_backtrace+0xad/0x120 [12197.016750] ? nmi_cpu_backtrace_handler+0x11/0x20 [12197.016752] ? nmi_handle+0xd4/0x260 [12197.016756] ? default_do_nmi+0x49/0xf0 [12197.016758] ? exc_nmi+0x11e/0x150 [12197.016759] ? end_repeat_nmi+0x16/0x67 [12197.016764] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12197.016766] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12197.016769] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12197.016771] [12197.016771] [12197.016772] do_raw_spin_lock+0xb5/0xc0 [12197.016774] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12197.016789] try_get_cap_refs+0x159/0x880 [ceph] [12197.016804] __ceph_get_caps+0x4a1/0x760 [ceph] [12197.016817] ? __file_remove_privs+0x89/0x110 [12197.016818] ? wake_bit_function+0x70/0x70 [12197.016821] ? ceph_write_iter+0x260/0x770 [ceph] [12197.016832] ceph_write_iter+0x260/0x770 [ceph] [12197.016844] ? lock_is_held_type+0xaa/0x120 [12197.016846] vfs_write+0x324/0x4f0 [12197.016850] ksys_pwrite64+0x65/0xa0 [12197.016853] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12197.016855] do_syscall_64+0x38/0x80 [12197.016858] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12197.016860] RIP: 0033:0x7f48f0eeb93b [12197.016861] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12197.016862] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12197.016863] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12197.016864] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12197.016865] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12197.016865] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12197.016866] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12197.016869] [12197.017724] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12197.322065] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12197.329536] Workqueue: events delayed_work [ceph] [12197.334250] RIP: 0010:lock_acquire+0xf0/0x2b0 [12197.338601] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12197.357339] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12197.362557] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12197.369680] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12197.376805] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12197.383928] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12197.391053] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12197.398176] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12197.406255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12197.411991] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12197.419117] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12197.426239] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12197.433365] Call Trace: [12197.435810] [12197.437820] ? rcu_dump_cpu_stacks+0xd1/0x180 [12197.442180] ? rcu_sched_clock_irq+0x633/0x1390 [12197.446710] ? lock_release+0xc5/0x260 [12197.450456] ? lock_is_held_type+0xaa/0x120 [12197.454634] ? lock_is_held_type+0xaa/0x120 [12197.458811] ? tick_sched_do_timer+0xa0/0xa0 [12197.463082] ? rcu_read_lock_sched_held+0x56/0x90 [12197.467782] ? tick_sched_do_timer+0xa0/0xa0 [12197.472053] ? update_process_times+0x5f/0x90 [12197.476403] ? tick_sched_handle.isra.16+0x1e/0x50 [12197.481189] ? tick_sched_timer+0x6f/0x80 [12197.485193] ? __hrtimer_run_queues+0x1d3/0x360 [12197.489720] ? hrtimer_interrupt+0x104/0x240 [12197.493990] ? __local_bh_enable+0x50/0x60 [12197.498082] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12197.503306] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12197.508262] [12197.510361] [12197.512458] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12197.517767] ? lock_acquire+0xf0/0x2b0 [12197.521515] ? igrab+0x43/0x50 [12197.524567] ? lock_release+0xc5/0x260 [12197.528312] _raw_spin_lock+0x30/0x40 [12197.531977] ? igrab+0x1d/0x50 [12197.535025] igrab+0x1d/0x50 [12197.537903] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12197.543054] delayed_work+0x1d8/0x280 [ceph] [12197.547336] ? process_scheduled_works+0x22a/0x510 [12197.552125] process_scheduled_works+0x2a4/0x510 [12197.556741] worker_thread+0x188/0x340 [12197.560491] ? rescuer_thread+0x310/0x310 [12197.564503] kthread+0xf5/0x130 [12197.567639] ? kthread_complete_and_exit+0x20/0x20 [12197.572423] ret_from_fork+0x31/0x40 [12197.575994] ? kthread_complete_and_exit+0x20/0x20 [12197.580778] ret_from_fork_asm+0x11/0x20 [12197.584702] [12221.668785] watchdog: BUG: soft lockup - CPU#0 stuck for 268s! [dbench:78683] [12221.675916] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12221.680785] watchdog: BUG: soft lockup - CPU#7 stuck for 9265s! [kworker/7:4:78808] [12221.731368] irq event stamp: 13479212 [12221.739011] Modules linked in: ceph libceph dns_resolver fscache netfs veth [12221.742667] hardirqs last enabled at (13479211): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12221.749628] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12221.759783] hardirqs last disabled at (13479212): [] sysvec_apic_timer_interrupt+0xe/0x90 [12221.766735] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [12221.776470] softirqs last enabled at (13479128): [] __do_softirq+0x2be/0x371 [12221.776475] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [12221.791938] softirqs last disabled at (13479123): [] irq_exit_rcu+0xb0/0xd0 [12221.791942] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [12221.807324] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12221.814274] iTCO_wdt iTCO_vendor_support i2c_smbus [12221.823219] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12221.828088] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [12221.835559] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12221.835561] ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 [12221.846602] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12221.852598] acpi_ipmi mfd_core wmi [12221.871336] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12221.871337] ipmi_si ipmi_devintf [12221.874821] [12221.880038] ipmi_msghandler ioatdma [12221.883347] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12221.884837] acpi_pad acpi_power_meter [12221.888409] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12221.895533] ixgbe igb [12221.899277] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12221.906401] mdio [12221.908758] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12221.915883] crc32c_intel [12221.917807] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12221.924930] nvme [12221.927548] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12221.934673] ptp nvme_core [12221.936598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12221.944673] i2c_algo_bit [12221.947377] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12221.953116] pps_core [12221.955733] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12221.962856] dca fuse [12221.965128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12221.972253] [12221.974522] Call Trace: [12221.974524] [12221.981646] irq event stamp: 22417148 [12221.983141] ? watchdog_timer_fn+0x1b4/0x220 [12221.985582] hardirqs last enabled at (22417147): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12221.987594] ? __hrtimer_run_queues+0x1d3/0x360 [12221.991248] hardirqs last disabled at (22417148): [] sysvec_apic_timer_interrupt+0xe/0x90 [12221.995517] ? hrtimer_interrupt+0x104/0x240 [12222.005671] softirqs last enabled at (22417092): [] __do_softirq+0x2be/0x371 [12222.010194] ? __local_bh_enable+0x50/0x60 [12222.019919] softirqs last disabled at (22417087): [] irq_exit_rcu+0xb0/0xd0 [12222.024186] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12222.032869] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12222.036958] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12222.045482] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12222.050694] [12222.060074] Workqueue: events delayed_work [ceph] [12222.065031] [12222.065032] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12222.072500] [12222.074603] ? queued_spin_lock_slowpath+0x16/0x2e0 [12222.079296] RIP: 0010:ceph_check_delayed_caps+0xa1/0x270 [ceph] [12222.081394] do_raw_spin_lock+0xb5/0xc0 [12222.086698] Code: e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 <48> 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 [12222.088189] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12222.093058] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [12222.098973] try_get_cap_refs+0x159/0x880 [ceph] [12222.102800] [12222.102801] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [12222.121545] __ceph_get_caps+0x4a1/0x760 [ceph] [12222.126323] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [12222.131544] ? __file_remove_privs+0x89/0x110 [12222.136151] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [12222.137641] ? wake_bit_function+0x70/0x70 [12222.144766] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12222.149293] ? ceph_write_iter+0x260/0x770 [ceph] [12222.156413] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12222.160765] ceph_write_iter+0x260/0x770 [ceph] [12222.167888] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12222.171982] ? lock_is_held_type+0xaa/0x120 [12222.179103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12222.183802] vfs_write+0x324/0x4f0 [12222.190924] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12222.195453] ksys_pwrite64+0x65/0xa0 [12222.203527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12222.207705] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12222.213441] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12222.216858] do_syscall_64+0x38/0x80 [12222.223981] Call Trace: [12222.227552] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12222.234677] [12222.239806] RIP: 0033:0x7f48f0eeb93b [12222.246930] ? watchdog_timer_fn+0x1b4/0x220 [12222.250502] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12222.252947] ? __hrtimer_run_queues+0x1d3/0x360 [12222.257987] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12222.260002] ? hrtimer_interrupt+0x104/0x240 [12222.263570] ORIG_RAX: 0000000000000012 [12222.267835] ? __local_bh_enable+0x50/0x60 [12222.286571] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12222.291097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12222.296312] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12222.300579] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12222.304407] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12222.308499] [12222.315624] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12222.320840] [12222.320842] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12222.327965] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12222.332926] ? ceph_check_delayed_caps+0xa1/0x270 [ceph] [12222.340050] [12222.342144] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [12222.376588] delayed_work+0x1d8/0x280 [ceph] [12222.380871] ? process_scheduled_works+0x22a/0x510 [12222.385659] process_scheduled_works+0x2a4/0x510 [12222.390275] worker_thread+0x188/0x340 [12222.394026] ? rescuer_thread+0x310/0x310 [12222.398036] kthread+0xf5/0x130 [12222.401190] ? kthread_complete_and_exit+0x20/0x20 [12222.405975] ret_from_fork+0x31/0x40 [12222.409556] ? kthread_complete_and_exit+0x20/0x20 [12222.414339] ret_from_fork_asm+0x11/0x20 [12222.418270] [12249.669277] watchdog: BUG: soft lockup - CPU#0 stuck for 294s! [dbench:78683] [12249.676405] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12249.681277] watchdog: BUG: soft lockup - CPU#7 stuck for 9291s! [kworker/7:4:78808] [12249.731846] irq event stamp: 13536744 [12249.731847] hardirqs last enabled at (13536743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12249.739489] Modules linked in: [12249.743147] hardirqs last disabled at (13536744): [] sysvec_apic_timer_interrupt+0xe/0x90 [12249.753304] ceph [12249.756354] softirqs last enabled at (13536696): [] __do_softirq+0x2be/0x371 [12249.766078] libceph [12249.768002] softirqs last disabled at (13536691): [] irq_exit_rcu+0xb0/0xd0 [12249.776687] dns_resolver [12249.778872] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12249.787382] fscache netfs [12249.789999] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12249.798953] veth nft_chain_nat [12249.801675] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12249.809144] xt_MASQUERADE nf_nat [12249.812281] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12249.817758] nf_conntrack nf_defrag_ipv6 [12249.821069] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12249.839807] nf_defrag_ipv4 nft_compat [12249.843726] [12249.843727] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12249.848953] nf_tables [12249.852696] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12249.854186] nfnetlink bridge [12249.861310] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12249.863676] stp llc [12249.870800] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12249.873763] xfs [12249.880889] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12249.883073] libcrc32c sunrpc [12249.890196] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12249.892034] intel_rapl_msr intel_rapl_common [12249.899159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12249.902123] sb_edac x86_pkg_temp_thermal [12249.910198] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12249.914549] coretemp kvm_intel [12249.920288] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12249.924292] kvm [12249.931417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12249.934552] irqbypass i2c_i801 [12249.941678] Call Trace: [12249.943516] iTCO_wdt iTCO_vendor_support [12249.950639] [12249.953777] i2c_smbus crct10dif_pclmul [12249.956223] ? watchdog_timer_fn+0x1b4/0x220 [12249.960223] joydev crc32_pclmul [12249.962237] ? __hrtimer_run_queues+0x1d3/0x360 [12249.966067] ipmi_ssif [12249.970333] ? hrtimer_interrupt+0x104/0x240 [12249.973554] ghash_clmulni_intel mei_me [12249.978079] ? __local_bh_enable+0x50/0x60 [12249.980435] mei lpc_ich [12249.984702] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12249.988531] sha512_ssse3 acpi_ipmi [12249.992621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12249.995152] mfd_core wmi [12250.000370] [12250.003853] ipmi_si ipmi_devintf [12250.008811] [12250.008813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12250.011428] ipmi_msghandler [12250.013528] ? queued_spin_lock_slowpath+0x18/0x2e0 [12250.016837] ioatdma acpi_pad [12250.018936] do_raw_spin_lock+0xb5/0xc0 [12250.024238] acpi_power_meter [12250.027117] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12250.031986] ixgbe igb [12250.034970] try_get_cap_refs+0x159/0x880 [ceph] [12250.038799] mdio [12250.041768] __ceph_get_caps+0x4a1/0x760 [ceph] [12250.046545] crc32c_intel nvme [12250.048907] ? __file_remove_privs+0x89/0x110 [12250.053515] ptp nvme_core [12250.055439] ? wake_bit_function+0x70/0x70 [12250.059962] i2c_algo_bit pps_core [12250.063015] ? ceph_write_iter+0x260/0x770 [ceph] [12250.067363] dca [12250.070067] ceph_write_iter+0x260/0x770 [ceph] [12250.074157] fuse [12250.077561] ? lock_is_held_type+0xaa/0x120 [12250.082254] irq event stamp: 22476888 [12250.084093] vfs_write+0x324/0x4f0 [12250.088614] hardirqs last enabled at (22476887): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12250.090542] ksys_pwrite64+0x65/0xa0 [12250.094717] hardirqs last disabled at (22476888): [] sysvec_apic_timer_interrupt+0xe/0x90 [12250.098373] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12250.101771] softirqs last enabled at (22476876): [] __do_softirq+0x2be/0x371 [12250.111929] do_syscall_64+0x38/0x80 [12250.115500] softirqs last disabled at (22476863): [] irq_exit_rcu+0xb0/0xd0 [12250.125225] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12250.130355] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12250.139038] RIP: 0033:0x7f48f0eeb93b [12250.142610] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12250.151121] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12250.156163] Workqueue: events delayed_work [ceph] [12250.165542] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12250.169113] [12250.169114] RIP: 0010:lock_acquire+0xf0/0x2b0 [12250.176584] ORIG_RAX: 0000000000000012 [12250.195322] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12250.200017] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12250.205237] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12250.206727] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12250.211077] [12250.214909] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12250.233645] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12250.240768] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12250.245987] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12250.253110] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12250.254603] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12250.261732] [12250.268850] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12250.306681] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12250.313805] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12250.321884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12250.327620] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12250.334746] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12250.341867] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12250.348994] Call Trace: [12250.351439] [12250.353447] ? watchdog_timer_fn+0x1b4/0x220 [12250.357714] ? __hrtimer_run_queues+0x1d3/0x360 [12250.362238] ? hrtimer_interrupt+0x104/0x240 [12250.366500] ? __local_bh_enable+0x50/0x60 [12250.370595] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12250.375817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12250.380774] [12250.382871] [12250.384970] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12250.390277] ? lock_acquire+0xf0/0x2b0 [12250.394027] ? igrab+0x43/0x50 [12250.397078] ? lock_release+0xc5/0x260 [12250.400822] _raw_spin_lock+0x30/0x40 [12250.404480] ? igrab+0x1d/0x50 [12250.407528] igrab+0x1d/0x50 [12250.410408] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12250.415557] delayed_work+0x1d8/0x280 [ceph] [12250.419839] ? process_scheduled_works+0x22a/0x510 [12250.424629] process_scheduled_works+0x2a4/0x510 [12250.429245] worker_thread+0x188/0x340 [12250.432993] ? rescuer_thread+0x310/0x310 [12250.436996] kthread+0xf5/0x130 [12250.440133] ? kthread_complete_and_exit+0x20/0x20 [12250.444917] ret_from_fork+0x31/0x40 [12250.448487] ? kthread_complete_and_exit+0x20/0x20 [12250.453273] ret_from_fork_asm+0x11/0x20 [12250.457194] [12277.669769] watchdog: BUG: soft lockup - CPU#0 stuck for 320s! [dbench:78683] [12277.676901] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12277.681769] watchdog: BUG: soft lockup - CPU#7 stuck for 9317s! [kworker/7:4:78808] [12277.732351] irq event stamp: 13594556 [12277.739994] Modules linked in: ceph libceph dns_resolver [12277.743652] hardirqs last enabled at (13594555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12277.743656] fscache [12277.748965] netfs veth [12277.759122] hardirqs last disabled at (13594556): [] sysvec_apic_timer_interrupt+0xe/0x90 [12277.761305] nft_chain_nat [12277.763749] softirqs last enabled at (13594508): [] __do_softirq+0x2be/0x371 [12277.773475] xt_MASQUERADE nf_nat [12277.776178] softirqs last disabled at (13594503): [] irq_exit_rcu+0xb0/0xd0 [12277.784862] nf_conntrack [12277.788174] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12277.796683] nf_defrag_ipv6 nf_defrag_ipv4 [12277.799302] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12277.808244] nft_compat [12277.812337] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12277.819807] nf_tables nfnetlink [12277.822251] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12277.827728] bridge [12277.830953] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12277.849689] stp llc [12277.851787] [12277.857005] xfs [12277.859188] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12277.860679] libcrc32c sunrpc [12277.862517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12277.869640] intel_rapl_msr intel_rapl_common [12277.872604] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12277.879728] sb_edac x86_pkg_temp_thermal [12277.884079] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12277.891203] coretemp kvm_intel [12277.895208] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12277.902332] kvm irqbypass [12277.905471] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12277.912593] i2c_i801 iTCO_wdt [12277.915297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12277.923374] iTCO_vendor_support i2c_smbus [12277.926426] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12277.932167] crct10dif_pclmul [12277.936254] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12277.943378] joydev crc32_pclmul [12277.946344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12277.953467] ipmi_ssif ghash_clmulni_intel [12277.956690] Call Trace: [12277.956693] [12277.963815] mei_me [12277.967908] ? watchdog_timer_fn+0x1b4/0x220 [12277.970350] mei lpc_ich [12277.972362] ? __hrtimer_run_queues+0x1d3/0x360 [12277.974456] sha512_ssse3 acpi_ipmi [12277.978725] ? hrtimer_interrupt+0x104/0x240 [12277.981253] mfd_core wmi [12277.985777] ? __local_bh_enable+0x50/0x60 [12277.989262] ipmi_si ipmi_devintf [12277.993527] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12277.996143] ipmi_msghandler ioatdma [12278.000252] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12278.003560] acpi_pad acpi_power_meter [12278.008777] [12278.012350] ixgbe igb [12278.017307] [12278.017308] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12278.021051] mdio crc32c_intel [12278.023170] ? queued_spin_lock_slowpath+0x18/0x2e0 [12278.025523] nvme [12278.027622] do_raw_spin_lock+0xb5/0xc0 [12278.032923] ptp nvme_core [12278.035977] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12278.040845] i2c_algo_bit pps_core [12278.042774] try_get_cap_refs+0x159/0x880 [ceph] [12278.046599] dca [12278.049311] __ceph_get_caps+0x4a1/0x760 [ceph] [12278.054089] fuse [12278.057491] ? __file_remove_privs+0x89/0x110 [12278.062098] [12278.063935] ? wake_bit_function+0x70/0x70 [12278.068459] irq event stamp: 22536348 [12278.070386] ? ceph_write_iter+0x260/0x770 [ceph] [12278.074734] hardirqs last enabled at (22536347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12278.076225] ceph_write_iter+0x260/0x770 [ceph] [12278.080314] hardirqs last disabled at (22536348): [] sysvec_apic_timer_interrupt+0xe/0x90 [12278.083977] ? lock_is_held_type+0xaa/0x120 [12278.088668] softirqs last enabled at (22536334): [] __do_softirq+0x2be/0x371 [12278.098828] vfs_write+0x324/0x4f0 [12278.103351] softirqs last disabled at (22536329): [] irq_exit_rcu+0xb0/0xd0 [12278.113080] ksys_pwrite64+0x65/0xa0 [12278.117251] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12278.125937] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12278.129333] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12278.137847] do_syscall_64+0x38/0x80 [12278.141417] Workqueue: events delayed_work [ceph] [12278.150794] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12278.155924] [12278.155925] RIP: 0010:lock_acquire+0xf0/0x2b0 [12278.163394] RIP: 0033:0x7f48f0eeb93b [12278.166966] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12278.171662] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12278.176706] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12278.178198] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12278.182549] [12278.182550] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12278.186118] ORIG_RAX: 0000000000000012 [12278.204857] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12278.223593] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12278.228812] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12278.234028] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12278.235519] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12278.242643] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12278.246475] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12278.253599] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12278.260722] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12278.267847] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12278.274972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12278.282102] [12278.289220] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12278.333716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12278.340838] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12278.347963] Call Trace: [12278.350410] [12278.352420] ? watchdog_timer_fn+0x1b4/0x220 [12278.356683] ? __hrtimer_run_queues+0x1d3/0x360 [12278.361210] ? hrtimer_interrupt+0x104/0x240 [12278.365480] ? __local_bh_enable+0x50/0x60 [12278.369571] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12278.374789] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12278.379746] [12278.381843] [12278.383939] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12278.389248] ? lock_acquire+0xf0/0x2b0 [12278.392998] ? igrab+0x43/0x50 [12278.396048] ? lock_release+0xc5/0x260 [12278.399792] _raw_spin_lock+0x30/0x40 [12278.403457] ? igrab+0x1d/0x50 [12278.406507] igrab+0x1d/0x50 [12278.409387] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12278.414535] delayed_work+0x1d8/0x280 [ceph] [12278.418818] ? process_scheduled_works+0x22a/0x510 [12278.423608] process_scheduled_works+0x2a4/0x510 [12278.428223] worker_thread+0x188/0x340 [12278.431973] ? rescuer_thread+0x310/0x310 [12278.435983] kthread+0xf5/0x130 [12278.439123] ? kthread_complete_and_exit+0x20/0x20 [12278.443908] ret_from_fork+0x31/0x40 [12278.447485] ? kthread_complete_and_exit+0x20/0x20 [12278.452270] ret_from_fork_asm+0x11/0x20 [12278.456201] [12290.385409] INFO: task jbd2/sda1-8:470 blocked for more than 245 seconds. [12290.392199] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12290.398982] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12290.406807] task:jbd2/sda1-8 state:D stack:0 pid:470 ppid:2 flags:0x00004000 [12290.415153] Call Trace: [12290.417598] [12290.419706] __schedule+0x379/0xe70 [12290.423196] ? find_held_lock+0x2d/0x90 [12290.427038] ? out_of_line_wait_on_bit_timeout+0xd0/0xd0 [12290.432349] schedule+0x54/0xc0 [12290.435493] io_schedule+0x16/0x40 [12290.438893] bit_wait_io+0x11/0x60 [12290.442299] __wait_on_bit+0x75/0x90 [12290.445879] out_of_line_wait_on_bit+0x98/0xc0 [12290.450328] ? __wake_up_locked_sync_key+0x20/0x20 [12290.455116] jbd2_journal_commit_transaction+0x1761/0x1bd0 [12290.460616] kjournald2+0xcd/0x2a0 [12290.464018] ? sched_numa_warn+0xf0/0xf0 [12290.467943] ? jbd2_seq_info_stop+0x10/0x10 [12290.472127] kthread+0xf5/0x130 [12290.475274] ? kthread_complete_and_exit+0x20/0x20 [12290.480069] ret_from_fork+0x31/0x40 [12290.483648] ? kthread_complete_and_exit+0x20/0x20 [12290.488439] ret_from_fork_asm+0x11/0x20 [12290.492374] [12290.494562] INFO: task NetworkManager:806 blocked for more than 245 seconds. [12290.501604] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12290.508384] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12290.516209] task:NetworkManager state:D stack:0 pid:806 ppid:1 flags:0x00004000 [12290.524554] Call Trace: [12290.526999] [12290.529105] __schedule+0x379/0xe70 [12290.532601] schedule+0x54/0xc0 [12290.535743] jbd2_log_wait_commit+0x124/0x190 [12290.540105] ? sched_numa_warn+0xf0/0xf0 [12290.544030] ext4_fc_commit+0x2b9/0x8d0 [12290.547869] ? jbd2_trans_will_send_data_barrier+0x35/0xa0 [12290.553355] ? find_held_lock+0x2d/0x90 [12290.557193] ? jbd2_trans_will_send_data_barrier+0x83/0xa0 [12290.562679] ? lock_release+0xc5/0x260 [12290.566437] ext4_sync_file+0x224/0x3d0 [12290.570273] ? __fget_files+0xce/0x1b0 [12290.574025] do_fsync+0x38/0x70 [12290.577173] __x64_sys_fsync+0x14/0x20 [12290.580924] do_syscall_64+0x38/0x80 [12290.584502] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12290.589555] RIP: 0033:0x7f8d54c11e57 [12290.593136] RSP: 002b:00007ffef3f2a520 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [12290.600701] RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 00007f8d54c11e57 [12290.607833] RDX: 0000000000000000 RSI: 00007ffef3f2a550 RDI: 0000000000000018 [12290.614958] RBP: 0000000000000018 R08: 0000000000000000 R09: 0000000000000006 [12290.622089] R10: 00000000000000fd R11: 0000000000000293 R12: 00007f8d58b7e088 [12290.629215] R13: 000056016a932050 R14: 000056016a8d5f4d R15: 0000000000000000 [12290.636352] [12290.638546] INFO: task logrotate:85883 blocked for more than 246 seconds. [12290.645326] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12290.652102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12290.659921] task:logrotate state:D stack:0 pid:85883 ppid:85867 flags:0x00004000 [12290.668269] Call Trace: [12290.670721] [12290.672829] __schedule+0x379/0xe70 [12290.676324] schedule+0x54/0xc0 [12290.679464] jbd2_log_wait_commit+0x124/0x190 [12290.683816] ? sched_numa_warn+0xf0/0xf0 [12290.687744] ext4_fc_commit+0x2b9/0x8d0 [12290.691582] ? jbd2_trans_will_send_data_barrier+0x35/0xa0 [12290.697069] ? find_held_lock+0x2d/0x90 [12290.700907] ? jbd2_trans_will_send_data_barrier+0x83/0xa0 [12290.706394] ? lock_release+0xc5/0x260 [12290.710152] ext4_sync_file+0x224/0x3d0 [12290.713988] do_fsync+0x38/0x70 [12290.717132] __x64_sys_fsync+0x14/0x20 [12290.720883] do_syscall_64+0x38/0x80 [12290.724462] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12290.729516] RIP: 0033:0x7ff2606f4478 [12290.733095] RSP: 002b:00007ffcb78e6c28 EFLAGS: 00000246 ORIG_RAX: 000000000000004a [12290.740662] RAX: ffffffffffffffda RBX: 00005593995f6f0d RCX: 00007ff2606f4478 [12290.747794] RDX: 0000000000000000 RSI: 00005593995eb3c0 RDI: 0000000000000003 [12290.754926] RBP: 0000000000000000 R08: 00005593995eb3c0 R09: 00007ff261a04800 [12290.762051] R10: 0000000000000000 R11: 0000000000000246 R12: 00005593995eb2e0 [12290.769177] R13: 00007ffcb78e6ca0 R14: 0000000000000040 R15: 0000000000000000 [12290.776314] [12290.778504] [12290.778504] Showing all locks held in the system: [12290.784681] 5 locks held by systemd/1: [12290.788433] #0: ffff8881047af3f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x60/0xe0 [12290.796346] #1: ffff888105868088 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x108/0x1d0 [12290.805211] #2: ffffffff8256e028 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_kn_lock_live+0x51/0x240 [12290.813990] #3: ffffffff82466ef0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x7a/0x280 [12290.823552] #4: ffffffff8256de90 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x86/0x280 [12290.833807] 1 lock held by khungtaskd/57: [12290.837818] #0: ffffffff8255ece0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x36/0x1c0 [12290.846776] 4 locks held by dbench/78683: [12290.850782] 3 locks held by kworker/7:4/78808: [12290.855219] 3 locks held by (sa1)/85950: [12290.859137] #0: ffff8881047af3f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x60/0xe0 [12290.867050] #1: ffff8881178f8888 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x108/0x1d0 [12290.875915] #2: ffffffff8256e028 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_kn_lock_live+0x51/0x240 [12290.884697] [12290.886195] ============================================= [12290.886195] [12317.670472] watchdog: BUG: soft lockup - CPU#0 stuck for 357s! [dbench:78683] [12317.677606] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12317.682472] watchdog: BUG: soft lockup - CPU#7 stuck for 9354s! [kworker/7:4:78808] [12317.733055] irq event stamp: 13677560 [12317.740700] Modules linked in: ceph libceph dns_resolver fscache [12317.744356] hardirqs last enabled at (13677559): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12317.744360] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [12317.760518] hardirqs last disabled at (13677560): [] sysvec_apic_timer_interrupt+0xe/0x90 [12317.765997] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [12317.775720] softirqs last enabled at (13677444): [] __do_softirq+0x2be/0x371 [12317.775723] nfnetlink bridge stp llc xfs libcrc32c [12317.791442] softirqs last disabled at (13677439): [] irq_exit_rcu+0xb0/0xd0 [12317.796313] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [12317.804827] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12317.812295] coretemp kvm_intel [12317.821240] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12317.824376] kvm irqbypass i2c_i801 [12317.831865] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12317.835348] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [12317.840827] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12317.847171] joydev [12317.865908] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12317.865909] crc32_pclmul [12317.865910] [12317.868006] ipmi_ssif ghash_clmulni_intel [12317.873224] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12317.875839] mei_me mei [12317.877330] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12317.881422] lpc_ich sha512_ssse3 [12317.888545] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12317.890988] acpi_ipmi mfd_core [12317.898113] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12317.901423] wmi ipmi_si [12317.908549] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12317.911687] ipmi_devintf ipmi_msghandler [12317.918809] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12317.921340] ioatdma acpi_pad [12317.928464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12317.932470] acpi_power_meter [12317.940547] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12317.943510] ixgbe [12317.949248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12317.952213] igb mdio [12317.959337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12317.961348] crc32c_intel [12317.968472] Call Trace: [12317.968474] [12317.970743] nvme ptp [12317.977872] ? watchdog_timer_fn+0x1b4/0x220 [12317.980492] nvme_core [12317.982937] ? __hrtimer_run_queues+0x1d3/0x360 [12317.984947] i2c_algo_bit pps_core [12317.987219] ? hrtimer_interrupt+0x104/0x240 [12317.991482] dca fuse [12317.993838] ? __local_bh_enable+0x50/0x60 [12317.998363] [12317.998364] irq event stamp: 22622148 [12318.001763] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12318.006023] hardirqs last enabled at (22622147): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12318.008295] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12318.012385] hardirqs last disabled at (22622148): [] sysvec_apic_timer_interrupt+0xe/0x90 [12318.013878] [12318.017535] softirqs last enabled at (22622072): [] __do_softirq+0x2be/0x371 [12318.022750] [12318.022752] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12318.032908] softirqs last disabled at (22622067): [] irq_exit_rcu+0xb0/0xd0 [12318.037875] ? queued_spin_lock_slowpath+0x16/0x2e0 [12318.047599] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12318.049699] do_raw_spin_lock+0xb5/0xc0 [12318.058381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12318.060480] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12318.065781] Workqueue: events delayed_work [ceph] [12318.074296] try_get_cap_refs+0x159/0x880 [ceph] [12318.079164] [12318.079165] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [12318.088548] __ceph_get_caps+0x4a1/0x760 [ceph] [12318.092372] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [12318.099849] ? __file_remove_privs+0x89/0x110 [12318.104627] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [12318.109325] ? wake_bit_function+0x70/0x70 [12318.113934] [12318.113935] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12318.115428] ? ceph_write_iter+0x260/0x770 [ceph] [12318.120038] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [12318.124561] ceph_write_iter+0x260/0x770 [ceph] [12318.143299] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [12318.147655] ? lock_is_held_type+0xaa/0x120 [12318.152874] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12318.156970] vfs_write+0x324/0x4f0 [12318.158456] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12318.165585] ksys_pwrite64+0x65/0xa0 [12318.170278] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12318.177402] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12318.181927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12318.189051] do_syscall_64+0x38/0x80 [12318.193229] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12318.200354] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12318.203749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12318.210877] RIP: 0033:0x7f48f0eeb93b [12318.214444] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12318.222523] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12318.227653] Call Trace: [12318.227654] [12318.233390] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12318.236961] ? watchdog_timer_fn+0x1b4/0x220 [12318.244084] ORIG_RAX: 0000000000000012 [12318.249131] ? __hrtimer_run_queues+0x1d3/0x360 [12318.256270] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12318.259845] ? hrtimer_interrupt+0x104/0x240 [12318.266965] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12318.285702] ? __local_bh_enable+0x50/0x60 [12318.288157] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12318.290169] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12318.295383] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12318.299648] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12318.303478] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12318.308003] [12318.315133] [12318.319390] [12318.319392] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12318.373877] ? do_raw_spin_lock+0x42/0xc0 [12318.377885] igrab+0x1d/0x50 [12318.380764] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12318.385913] delayed_work+0x1d8/0x280 [ceph] [12318.390192] ? process_scheduled_works+0x22a/0x510 [12318.394984] process_scheduled_works+0x2a4/0x510 [12318.399597] worker_thread+0x188/0x340 [12318.403347] ? rescuer_thread+0x310/0x310 [12318.407351] kthread+0xf5/0x130 [12318.410489] ? kthread_complete_and_exit+0x20/0x20 [12318.415274] ret_from_fork+0x31/0x40 [12318.418852] ? kthread_complete_and_exit+0x20/0x20 [12318.423636] ret_from_fork_asm+0x11/0x20 [12318.427559] [12345.670964] watchdog: BUG: soft lockup - CPU#0 stuck for 383s! [dbench:78683] [12345.678092] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12345.682964] watchdog: BUG: soft lockup - CPU#7 stuck for 9380s! [kworker/7:4:78808] [12345.733541] irq event stamp: 13735238 [12345.741186] Modules linked in: ceph libceph dns_resolver fscache netfs veth [12345.744841] hardirqs last enabled at (13735237): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12345.751792] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12345.761952] hardirqs last disabled at (13735238): [] sysvec_apic_timer_interrupt+0xe/0x90 [12345.768910] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [12345.778638] softirqs last enabled at (13735228): [] __do_softirq+0x2be/0x371 [12345.785422] xfs libcrc32c sunrpc intel_rapl_msr [12345.794106] softirqs last disabled at (13735223): [] irq_exit_rcu+0xb0/0xd0 [12345.798725] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [12345.807234] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12345.813579] kvm_intel kvm [12345.822522] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12345.822523] irqbypass i2c_i801 [12345.825227] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12345.832697] iTCO_wdt [12345.835837] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12345.841311] iTCO_vendor_support [12345.843583] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12345.862322] i2c_smbus [12345.865547] [12345.870763] crct10dif_pclmul [12345.873121] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12345.874609] joydev crc32_pclmul [12345.877574] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12345.884699] ipmi_ssif [12345.887922] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12345.895048] ghash_clmulni_intel [12345.897403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12345.904530] mei_me mei [12345.907753] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12345.914876] lpc_ich sha512_ssse3 [12345.917322] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12345.924446] acpi_ipmi mfd_core [12345.927754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12345.935833] wmi ipmi_si [12345.938971] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12345.944709] ipmi_devintf ipmi_msghandler ioatdma [12345.947238] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12345.954363] acpi_pad acpi_power_meter [12345.959059] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12345.966185] ixgbe [12345.969929] Call Trace: [12345.969931] [12345.977051] igb mdio [12345.979065] ? watchdog_timer_fn+0x1b4/0x220 [12345.981506] crc32c_intel nvme [12345.983520] ? __hrtimer_run_queues+0x1d3/0x360 [12345.985788] ptp nvme_core [12345.990055] ? hrtimer_interrupt+0x104/0x240 [12345.993104] i2c_algo_bit pps_core [12345.997644] ? __local_bh_enable+0x50/0x60 [12346.000361] dca fuse [12346.004623] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12346.008019] [12346.008020] irq event stamp: 22681696 [12346.012111] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12346.014380] hardirqs last enabled at (22681695): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12346.019597] [12346.021088] hardirqs last disabled at (22681696): [] sysvec_apic_timer_interrupt+0xe/0x90 [12346.024746] [12346.024747] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12346.029703] softirqs last enabled at (22681620): [] __do_softirq+0x2be/0x371 [12346.039864] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12346.041959] softirqs last disabled at (22681615): [] irq_exit_rcu+0xb0/0xd0 [12346.051684] do_raw_spin_lock+0xb5/0xc0 [12346.053781] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12346.059088] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12346.067769] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12346.072659] try_get_cap_refs+0x159/0x880 [ceph] [12346.081167] Workqueue: events delayed_work [ceph] [12346.085004] __ceph_get_caps+0x4a1/0x760 [ceph] [12346.094374] [12346.099166] ? __file_remove_privs+0x89/0x110 [12346.106645] RIP: 0010:lock_acquire+0xf0/0x2b0 [12346.111260] ? wake_bit_function+0x70/0x70 [12346.115955] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12346.120482] ? ceph_write_iter+0x260/0x770 [ceph] [12346.121971] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12346.126323] ceph_write_iter+0x260/0x770 [ceph] [12346.130672] [12346.134767] ? lock_is_held_type+0xaa/0x120 [12346.153500] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12346.158199] vfs_write+0x324/0x4f0 [12346.163415] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12346.167945] ksys_pwrite64+0x65/0xa0 [12346.169430] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12346.173608] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12346.180732] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12346.184130] do_syscall_64+0x38/0x80 [12346.191254] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12346.194824] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12346.201949] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12346.207079] RIP: 0033:0x7f48f0eeb93b [12346.214203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12346.217775] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12346.224897] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12346.229942] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12346.238020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12346.241589] ORIG_RAX: 0000000000000012 [12346.247327] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12346.266067] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12346.273190] Call Trace: [12346.273191] [12346.278408] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12346.285531] ? watchdog_timer_fn+0x1b4/0x220 [12346.289367] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12346.296496] ? __hrtimer_run_queues+0x1d3/0x360 [12346.303619] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12346.306066] ? hrtimer_interrupt+0x104/0x240 [12346.308073] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12346.315197] ? __local_bh_enable+0x50/0x60 [12346.319466] [12346.326589] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12346.361116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12346.366072] [12346.368168] [12346.370266] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12346.375576] ? lock_acquire+0xf0/0x2b0 [12346.379324] ? igrab+0x43/0x50 [12346.382375] ? lock_release+0xc5/0x260 [12346.386120] _raw_spin_lock+0x30/0x40 [12346.389777] ? igrab+0x1d/0x50 [12346.392826] igrab+0x1d/0x50 [12346.395705] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12346.400856] delayed_work+0x1d8/0x280 [ceph] [12346.405135] ? process_scheduled_works+0x22a/0x510 [12346.409927] process_scheduled_works+0x2a4/0x510 [12346.414543] worker_thread+0x188/0x340 [12346.418292] ? rescuer_thread+0x310/0x310 [12346.422302] kthread+0xf5/0x130 [12346.425440] ? kthread_complete_and_exit+0x20/0x20 [12346.430225] ret_from_fork+0x31/0x40 [12346.433805] ? kthread_complete_and_exit+0x20/0x20 [12346.438588] ret_from_fork_asm+0x11/0x20 [12346.442520] [12373.671456] watchdog: BUG: soft lockup - CPU#0 stuck for 409s! [dbench:78683] [12373.678586] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12373.683456] watchdog: BUG: soft lockup - CPU#7 stuck for 9406s! [kworker/7:4:78808] [12373.734026] irq event stamp: 13792926 [12373.734028] hardirqs last enabled at (13792925): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12373.741671] Modules linked in: [12373.745328] hardirqs last disabled at (13792926): [] sysvec_apic_timer_interrupt+0xe/0x90 [12373.755487] ceph [12373.758538] softirqs last enabled at (13792838): [] __do_softirq+0x2be/0x371 [12373.768262] libceph [12373.770186] softirqs last disabled at (13792833): [] irq_exit_rcu+0xb0/0xd0 [12373.778870] dns_resolver fscache [12373.781053] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12373.789563] netfs veth [12373.792875] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12373.801820] nft_chain_nat xt_MASQUERADE [12373.804263] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12373.811734] nf_nat nf_conntrack [12373.815652] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12373.821131] nf_defrag_ipv6 nf_defrag_ipv4 [12373.824354] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12373.843089] nft_compat nf_tables [12373.847182] [12373.852399] nfnetlink bridge [12373.855710] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12373.857201] stp llc [12373.860164] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12373.867290] xfs libcrc32c [12373.869472] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12373.876596] sunrpc [12373.879301] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12373.886425] intel_rapl_msr [12373.888522] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12373.895646] intel_rapl_common [12373.898437] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12373.905562] sb_edac x86_pkg_temp_thermal [12373.908611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12373.916690] coretemp [12373.920694] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12373.926431] kvm_intel kvm [12373.928703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12373.935826] irqbypass i2c_i801 [12373.938529] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12373.945653] iTCO_wdt iTCO_vendor_support i2c_smbus [12373.948792] Call Trace: [12373.955915] crct10dif_pclmul joydev [12373.960787] [12373.963231] crc32_pclmul [12373.966802] ? watchdog_timer_fn+0x1b4/0x220 [12373.968811] ipmi_ssif ghash_clmulni_intel [12373.971429] ? __hrtimer_run_queues+0x1d3/0x360 [12373.975694] mei_me [12373.979788] ? hrtimer_interrupt+0x104/0x240 [12373.984308] mei lpc_ich [12373.986406] ? __local_bh_enable+0x50/0x60 [12373.990668] sha512_ssse3 acpi_ipmi [12373.993203] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12373.997290] mfd_core wmi [12374.000774] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12374.005991] ipmi_si ipmi_devintf [12374.008610] [12374.013568] ipmi_msghandler ioatdma [12374.016878] [12374.018975] acpi_pad acpi_power_meter [12374.022546] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12374.024644] ixgbe igb [12374.028392] ? queued_spin_lock_slowpath+0x16/0x2e0 [12374.033692] mdio crc32c_intel [12374.036052] do_raw_spin_lock+0xb5/0xc0 [12374.040919] nvme ptp [12374.043971] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12374.047802] nvme_core i2c_algo_bit [12374.050076] try_get_cap_refs+0x159/0x880 [ceph] [12374.054855] pps_core dca [12374.058344] __ceph_get_caps+0x4a1/0x760 [ceph] [12374.062950] fuse [12374.065572] ? __file_remove_privs+0x89/0x110 [12374.070093] [12374.070094] irq event stamp: 22741332 [12374.072016] ? wake_bit_function+0x70/0x70 [12374.076367] hardirqs last enabled at (22741331): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12374.077859] ? ceph_write_iter+0x260/0x770 [ceph] [12374.081514] hardirqs last disabled at (22741332): [] sysvec_apic_timer_interrupt+0xe/0x90 [12374.085605] ceph_write_iter+0x260/0x770 [ceph] [12374.095764] softirqs last enabled at (22741286): [] __do_softirq+0x2be/0x371 [12374.100464] ? lock_is_held_type+0xaa/0x120 [12374.110185] softirqs last disabled at (22741281): [] irq_exit_rcu+0xb0/0xd0 [12374.114712] vfs_write+0x324/0x4f0 [12374.123392] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12374.127575] ksys_pwrite64+0x65/0xa0 [12374.136081] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12374.139480] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12374.148858] Workqueue: events delayed_work [ceph] [12374.152428] do_syscall_64+0x38/0x80 [12374.159898] [12374.159899] RIP: 0010:lock_acquire+0xf0/0x2b0 [12374.165031] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12374.169726] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12374.173298] RIP: 0033:0x7f48f0eeb93b [12374.174789] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12374.179139] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12374.184182] [12374.184182] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12374.202919] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12374.206492] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12374.211707] ORIG_RAX: 0000000000000012 [12374.230445] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12374.231936] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12374.239061] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12374.244278] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12374.251401] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12374.255233] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12374.262358] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12374.269480] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12374.276605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12374.283730] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12374.290861] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12374.297992] [12374.306065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12374.342482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12374.349606] Call Trace: [12374.352050] [12374.354063] ? watchdog_timer_fn+0x1b4/0x220 [12374.358326] ? __hrtimer_run_queues+0x1d3/0x360 [12374.362864] ? hrtimer_interrupt+0x104/0x240 [12374.367140] ? __local_bh_enable+0x50/0x60 [12374.371233] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12374.376455] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12374.381415] [12374.383512] [12374.385608] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12374.390917] ? lock_acquire+0xf0/0x2b0 [12374.394667] ? igrab+0x43/0x50 [12374.397716] ? lock_release+0xc5/0x260 [12374.401463] _raw_spin_lock+0x30/0x40 [12374.405143] ? igrab+0x1d/0x50 [12374.408195] igrab+0x1d/0x50 [12374.411072] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12374.416222] delayed_work+0x1d8/0x280 [ceph] [12374.420504] ? process_scheduled_works+0x22a/0x510 [12374.425294] process_scheduled_works+0x2a4/0x510 [12374.429910] worker_thread+0x188/0x340 [12374.433658] ? rescuer_thread+0x310/0x310 [12374.437661] kthread+0xf5/0x130 [12374.440799] ? kthread_complete_and_exit+0x20/0x20 [12374.445585] ret_from_fork+0x31/0x40 [12374.449164] ? kthread_complete_and_exit+0x20/0x20 [12374.453946] ret_from_fork_asm+0x11/0x20 [12374.457870] [12392.592155] rcu: INFO: rcu_preempt self-detected stall on CPU [12392.597921] rcu: 7-....: (447447 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/489288 fqs=195484 [12392.608077] rcu: (t=455973 jiffies g=1309609 q=77620 ncpus=8) [12392.613904] Sending NMI from CPU 7 to CPUs 0: [12392.618255] NMI backtrace for cpu 0 [12392.618257] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12392.618260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12392.618261] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12392.618267] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12392.618268] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12392.618270] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12392.618271] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12392.618271] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12392.618272] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12392.618273] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12392.618274] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12392.618275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12392.618276] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12392.618277] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12392.618277] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12392.618278] Call Trace: [12392.618279] [12392.618281] ? nmi_cpu_backtrace+0xad/0x120 [12392.618285] ? nmi_cpu_backtrace_handler+0x11/0x20 [12392.618287] ? nmi_handle+0xd4/0x260 [12392.618291] ? default_do_nmi+0x49/0xf0 [12392.618293] ? exc_nmi+0x11e/0x150 [12392.618294] ? end_repeat_nmi+0x16/0x67 [12392.618299] ? queued_spin_lock_slowpath+0x16/0x2e0 [12392.618301] ? queued_spin_lock_slowpath+0x16/0x2e0 [12392.618304] ? queued_spin_lock_slowpath+0x16/0x2e0 [12392.618306] [12392.618307] [12392.618307] do_raw_spin_lock+0xb5/0xc0 [12392.618311] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12392.618334] try_get_cap_refs+0x159/0x880 [ceph] [12392.618351] __ceph_get_caps+0x4a1/0x760 [ceph] [12392.618368] ? __file_remove_privs+0x89/0x110 [12392.618370] ? wake_bit_function+0x70/0x70 [12392.618373] ? ceph_write_iter+0x260/0x770 [ceph] [12392.618386] ceph_write_iter+0x260/0x770 [ceph] [12392.618400] ? lock_is_held_type+0xaa/0x120 [12392.618403] vfs_write+0x324/0x4f0 [12392.618408] ksys_pwrite64+0x65/0xa0 [12392.618410] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12392.618413] do_syscall_64+0x38/0x80 [12392.618416] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12392.618418] RIP: 0033:0x7f48f0eeb93b [12392.618419] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12392.618421] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12392.618422] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12392.618423] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12392.618424] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12392.618424] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12392.618425] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12392.618428] [12392.619254] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12392.923499] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12392.930969] Workqueue: events delayed_work [ceph] [12392.935684] RIP: 0010:igrab+0xa/0x50 [12392.939254] Code: 1f 44 00 00 c3 b8 00 04 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 55 <48> 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 [12392.957993] RSP: 0018:ffffc90002117da8 EFLAGS: 00000286 [12392.963209] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [12392.970333] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [12392.977458] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [12392.984583] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12392.991707] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12392.998831] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12393.006907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12393.012645] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12393.019770] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12393.026893] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12393.034018] Call Trace: [12393.036463] [12393.038473] ? rcu_dump_cpu_stacks+0xd1/0x180 [12393.042835] ? rcu_sched_clock_irq+0x633/0x1390 [12393.047364] ? lock_release+0xc5/0x260 [12393.051108] ? lock_is_held_type+0xaa/0x120 [12393.055288] ? lock_is_held_type+0xaa/0x120 [12393.059465] ? tick_sched_do_timer+0xa0/0xa0 [12393.063737] ? rcu_read_lock_sched_held+0x56/0x90 [12393.068435] ? tick_sched_do_timer+0xa0/0xa0 [12393.072698] ? update_process_times+0x5f/0x90 [12393.077050] ? tick_sched_handle.isra.16+0x1e/0x50 [12393.081834] ? tick_sched_timer+0x6f/0x80 [12393.085837] ? __hrtimer_run_queues+0x1d3/0x360 [12393.090365] ? hrtimer_interrupt+0x104/0x240 [12393.094634] ? __local_bh_enable+0x50/0x60 [12393.098736] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12393.103960] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12393.108916] [12393.111015] [12393.113112] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12393.118419] ? igrab+0xa/0x50 [12393.121390] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12393.126539] delayed_work+0x1d8/0x280 [ceph] [12393.130821] ? process_scheduled_works+0x22a/0x510 [12393.135612] process_scheduled_works+0x2a4/0x510 [12393.140227] worker_thread+0x188/0x340 [12393.143975] ? rescuer_thread+0x310/0x310 [12393.147978] kthread+0xf5/0x130 [12393.151118] ? kthread_complete_and_exit+0x20/0x20 [12393.155909] ret_from_fork+0x31/0x40 [12393.159489] ? kthread_complete_and_exit+0x20/0x20 [12393.164273] ret_from_fork_asm+0x11/0x20 [12393.168205] [12413.267562] INFO: task systemd:1 blocked for more than 122 seconds. [12413.273827] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12413.280613] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12413.288438] task:systemd state:D stack:0 pid:1 ppid:0 flags:0x00000000 [12413.296782] Call Trace: [12413.299229] [12413.301334] __schedule+0x379/0xe70 [12413.304832] schedule+0x54/0xc0 [12413.307972] schedule_timeout+0x2e5/0x360 [12413.311987] ? mark_held_locks+0x48/0x70 [12413.315913] wait_for_completion+0xb7/0x110 [12413.320101] __wait_rcu_gp+0x11c/0x140 [12413.323852] synchronize_rcu+0x1bf/0x1d0 [12413.327777] ? __call_rcu_common.constprop.56+0x5f0/0x5f0 [12413.333178] ? rcu_tasks_pregp_step+0x10/0x10 [12413.337537] rcu_sync_enter+0x58/0xf0 [12413.341205] percpu_down_write+0x6c/0x240 [12413.345215] cgroup_procs_write_start+0x86/0x280 [12413.349835] __cgroup1_procs_write.constprop.11+0x57/0x140 [12413.355321] cgroup_file_write+0x8b/0x240 [12413.359335] kernfs_fop_write_iter+0x13b/0x1d0 [12413.363779] vfs_write+0x324/0x4f0 [12413.367190] ksys_write+0x60/0xe0 [12413.370504] do_syscall_64+0x38/0x80 [12413.374084] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12413.379135] RIP: 0033:0x7fa4e52ed67f [12413.382715] RSP: 002b:00007ffe80080920 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 [12413.390280] RAX: ffffffffffffffda RBX: 0000000000000017 RCX: 00007fa4e52ed67f [12413.397406] RDX: 0000000000000006 RSI: 000055a0bec99fd0 RDI: 0000000000000017 [12413.404538] RBP: 000055a0bec99fd0 R08: 0000000000000000 R09: 000055a0bec99fc0 [12413.411670] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000006 [12413.418803] R13: 0000000000000006 R14: 00007fa4e55bb880 R15: 0000000000000006 [12413.425932] [12413.428130] INFO: task jbd2/sda1-8:470 blocked for more than 368 seconds. [12413.434916] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12413.441701] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12413.449519] task:jbd2/sda1-8 state:D stack:0 pid:470 ppid:2 flags:0x00004000 [12413.457864] Call Trace: [12413.460312] [12413.462416] __schedule+0x379/0xe70 [12413.465908] ? find_held_lock+0x2d/0x90 [12413.469751] ? out_of_line_wait_on_bit_timeout+0xd0/0xd0 [12413.475060] schedule+0x54/0xc0 [12413.478205] io_schedule+0x16/0x40 [12413.481613] bit_wait_io+0x11/0x60 [12413.485020] __wait_on_bit+0x75/0x90 [12413.488600] out_of_line_wait_on_bit+0x98/0xc0 [12413.493045] ? __wake_up_locked_sync_key+0x20/0x20 [12413.497837] jbd2_journal_commit_transaction+0x1761/0x1bd0 [12413.503338] kjournald2+0xcd/0x2a0 [12413.506747] ? sched_numa_warn+0xf0/0xf0 [12413.510674] ? jbd2_seq_info_stop+0x10/0x10 [12413.514857] kthread+0xf5/0x130 [12413.518005] ? kthread_complete_and_exit+0x20/0x20 [12413.522798] ret_from_fork+0x31/0x40 [12413.526375] ? kthread_complete_and_exit+0x20/0x20 [12413.531169] ret_from_fork_asm+0x11/0x20 [12413.535102] [12413.537291] INFO: task NetworkManager:806 blocked for more than 368 seconds. [12413.544333] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12413.551110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12413.558929] task:NetworkManager state:D stack:0 pid:806 ppid:1 flags:0x00004000 [12413.567274] Call Trace: [12413.569720] [12413.571827] __schedule+0x379/0xe70 [12413.575323] schedule+0x54/0xc0 [12413.578463] jbd2_log_wait_commit+0x124/0x190 [12413.582825] ? sched_numa_warn+0xf0/0xf0 [12413.586751] ext4_fc_commit+0x2b9/0x8d0 [12413.590590] ? jbd2_trans_will_send_data_barrier+0x35/0xa0 [12413.596076] ? find_held_lock+0x2d/0x90 [12413.599914] ? jbd2_trans_will_send_data_barrier+0x83/0xa0 [12413.605402] ? lock_release+0xc5/0x260 [12413.609158] ext4_sync_file+0x224/0x3d0 [12413.612994] ? __fget_files+0xce/0x1b0 [12413.616746] do_fsync+0x38/0x70 [12413.619893] __x64_sys_fsync+0x14/0x20 [12413.623643] do_syscall_64+0x38/0x80 [12413.627224] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12413.632277] RIP: 0033:0x7f8d54c11e57 [12413.635855] RSP: 002b:00007ffef3f2a520 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [12413.643421] RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 00007f8d54c11e57 [12413.650554] RDX: 0000000000000000 RSI: 00007ffef3f2a550 RDI: 0000000000000018 [12413.657688] RBP: 0000000000000018 R08: 0000000000000000 R09: 0000000000000006 [12413.664819] R10: 00000000000000fd R11: 0000000000000293 R12: 00007f8d58b7e088 [12413.671943] R13: 000056016a932050 R14: 000056016a8d5f4d R15: 0000000000000000 [12413.679075] [12413.681269] INFO: task logrotate:85883 blocked for more than 369 seconds. [12413.688055] Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12413.694832] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12413.702651] task:logrotate state:D stack:0 pid:85883 ppid:85867 flags:0x00004000 [12413.710996] Call Trace: [12413.713443] [12413.715549] __schedule+0x379/0xe70 [12413.719043] schedule+0x54/0xc0 [12413.722186] jbd2_log_wait_commit+0x124/0x190 [12413.726546] ? sched_numa_warn+0xf0/0xf0 [12413.730474] ext4_fc_commit+0x2b9/0x8d0 [12413.734311] ? jbd2_trans_will_send_data_barrier+0x35/0xa0 [12413.739796] ? find_held_lock+0x2d/0x90 [12413.743636] ? jbd2_trans_will_send_data_barrier+0x83/0xa0 [12413.749123] ? lock_release+0xc5/0x260 [12413.752878] ext4_sync_file+0x224/0x3d0 [12413.756717] do_fsync+0x38/0x70 [12413.759862] __x64_sys_fsync+0x14/0x20 [12413.763612] do_syscall_64+0x38/0x80 [12413.767194] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12413.772245] RIP: 0033:0x7ff2606f4478 [12413.775825] RSP: 002b:00007ffcb78e6c28 EFLAGS: 00000246 ORIG_RAX: 000000000000004a [12413.783393] RAX: ffffffffffffffda RBX: 00005593995f6f0d RCX: 00007ff2606f4478 [12413.790523] RDX: 0000000000000000 RSI: 00005593995eb3c0 RDI: 0000000000000003 [12413.797648] RBP: 0000000000000000 R08: 00005593995eb3c0 R09: 00007ff261a04800 [12413.804780] R10: 0000000000000000 R11: 0000000000000246 R12: 00005593995eb2e0 [12413.811904] R13: 00007ffcb78e6ca0 R14: 0000000000000040 R15: 0000000000000000 [12413.819034] [12413.821221] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [12413.829395] [12413.829395] Showing all locks held in the system: [12413.835575] 5 locks held by systemd/1: [12413.839326] #0: ffff8881047af3f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x60/0xe0 [12413.847241] #1: ffff888105868088 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x108/0x1d0 [12413.856105] #2: ffffffff8256e028 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_kn_lock_live+0x51/0x240 [12413.864884] #3: ffffffff82466ef0 (cpu_hotplug_lock){++++}-{0:0}, at: cgroup_procs_write_start+0x7a/0x280 [12413.874444] #4: ffffffff8256de90 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_procs_write_start+0x86/0x280 [12413.884699] 1 lock held by khungtaskd/57: [12413.888709] #0: ffffffff8255ece0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x36/0x1c0 [12413.897664] 1 lock held by systemd-journal/568: [12413.902195] #0: ffff88885fcef018 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0xf7/0xe70 [12413.910112] 4 locks held by dbench/78683: [12413.914122] 3 locks held by kworker/7:4/78808: [12413.918568] 3 locks held by (sa1)/85950: [12413.922493] #0: ffff8881047af3f8 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x60/0xe0 [12413.930407] #1: ffff8881178f8888 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x108/0x1d0 [12413.939273] #2: ffffffff8256e028 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_kn_lock_live+0x51/0x240 [12413.948052] [12413.949551] ============================================= [12413.949551] [12441.672651] watchdog: BUG: soft lockup - CPU#0 stuck for 473s! [dbench:78683] [12441.679786] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12441.684651] watchdog: BUG: soft lockup - CPU#7 stuck for 9470s! [kworker/7:4:78808] [12441.735236] irq event stamp: 13934850 [12441.735237] hardirqs last enabled at (13934849): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12441.742880] Modules linked in: [12441.746537] hardirqs last disabled at (13934850): [] sysvec_apic_timer_interrupt+0xe/0x90 [12441.756695] ceph [12441.759745] softirqs last enabled at (13934742): [] __do_softirq+0x2be/0x371 [12441.769470] libceph [12441.771394] softirqs last disabled at (13934737): [] irq_exit_rcu+0xb0/0xd0 [12441.780078] dns_resolver [12441.782264] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12441.790773] fscache netfs [12441.793392] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12441.802335] veth nft_chain_nat [12441.805047] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12441.812517] xt_MASQUERADE nf_nat [12441.815657] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12441.821133] nf_conntrack nf_defrag_ipv6 [12441.824444] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12441.843182] nf_defrag_ipv4 nft_compat [12441.847099] [12441.852326] nf_tables nfnetlink [12441.856068] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12441.857560] bridge stp [12441.860783] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12441.867906] llc xfs [12441.870350] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12441.877478] libcrc32c [12441.879661] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12441.886783] sunrpc intel_rapl_msr [12441.889141] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12441.896265] intel_rapl_common sb_edac [12441.899662] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12441.906788] x86_pkg_temp_thermal [12441.910531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12441.918607] coretemp kvm_intel [12441.921919] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12441.927658] kvm irqbypass [12441.930793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12441.937917] i2c_i801 iTCO_wdt [12441.940622] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12441.947747] iTCO_vendor_support i2c_smbus [12441.950797] Call Trace: [12441.957922] crct10dif_pclmul joydev [12441.962015] [12441.964455] crc32_pclmul ipmi_ssif [12441.968046] ? watchdog_timer_fn+0x1b4/0x220 [12441.970055] ghash_clmulni_intel [12441.973540] ? __hrtimer_run_queues+0x1d3/0x360 [12441.977804] mei_me [12441.981049] ? hrtimer_interrupt+0x104/0x240 [12441.985568] mei lpc_ich [12441.987666] ? __local_bh_enable+0x50/0x60 [12441.991931] sha512_ssse3 acpi_ipmi [12441.994463] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12441.998551] mfd_core wmi [12442.002048] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12442.007261] ipmi_si [12442.009878] [12442.009879] [12442.014838] ipmi_devintf [12442.017020] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12442.019118] ipmi_msghandler ioatdma [12442.021220] ? queued_spin_lock_slowpath+0x18/0x2e0 [12442.023832] acpi_pad acpi_power_meter [12442.029140] do_raw_spin_lock+0xb5/0xc0 [12442.032708] ixgbe igb [12442.037580] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12442.041338] mdio crc32c_intel [12442.045173] try_get_cap_refs+0x159/0x880 [ceph] [12442.047528] nvme ptp [12442.052317] __ceph_get_caps+0x4a1/0x760 [ceph] [12442.055362] nvme_core i2c_algo_bit [12442.059979] ? __file_remove_privs+0x89/0x110 [12442.062244] pps_core dca [12442.066767] ? wake_bit_function+0x70/0x70 [12442.070252] fuse [12442.074605] ? ceph_write_iter+0x260/0x770 [ceph] [12442.077220] [12442.077221] irq event stamp: 22920666 [12442.081311] ceph_write_iter+0x260/0x770 [ceph] [12442.083235] hardirqs last enabled at (22920665): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12442.087936] ? lock_is_held_type+0xaa/0x120 [12442.089423] hardirqs last disabled at (22920666): [] sysvec_apic_timer_interrupt+0xe/0x90 [12442.093084] vfs_write+0x324/0x4f0 [12442.097603] softirqs last enabled at (22920664): [] __do_softirq+0x2be/0x371 [12442.107767] ksys_pwrite64+0x65/0xa0 [12442.111940] softirqs last disabled at (22920659): [] irq_exit_rcu+0xb0/0xd0 [12442.121665] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12442.125062] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12442.133747] do_syscall_64+0x38/0x80 [12442.137332] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12442.145845] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12442.150975] Workqueue: events delayed_work [ceph] [12442.160353] RIP: 0033:0x7f48f0eeb93b [12442.163922] [12442.163923] RIP: 0010:lock_release+0x16a/0x260 [12442.171394] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12442.176437] Code: f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 c4 18 5b 5d 41 5c <41> 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 48 0f a3 05 18 bf 53 01 [12442.181135] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12442.184706] RSP: 0018:ffffc90002117d78 EFLAGS: 00000296 [12442.186196] ORIG_RAX: 0000000000000012 [12442.190634] [12442.190635] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [12442.209372] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12442.228110] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [12442.233341] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12442.238562] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [12442.242392] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12442.243883] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12442.251008] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12442.258131] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [12442.265256] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12442.272379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12442.279508] [12442.286628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12442.331125] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12442.338247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12442.345373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12442.352496] Call Trace: [12442.354942] [12442.356952] ? watchdog_timer_fn+0x1b4/0x220 [12442.361216] ? __hrtimer_run_queues+0x1d3/0x360 [12442.365743] ? hrtimer_interrupt+0x104/0x240 [12442.370011] ? __local_bh_enable+0x50/0x60 [12442.374106] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12442.379330] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12442.384304] [12442.386402] [12442.388499] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12442.393808] ? lock_release+0x16a/0x260 [12442.397644] _raw_spin_unlock+0x1b/0x40 [12442.401474] igrab+0x43/0x50 [12442.404357] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12442.409502] delayed_work+0x1d8/0x280 [ceph] [12442.413783] ? process_scheduled_works+0x22a/0x510 [12442.418571] process_scheduled_works+0x2a4/0x510 [12442.423190] worker_thread+0x188/0x340 [12442.426938] ? rescuer_thread+0x310/0x310 [12442.430939] kthread+0xf5/0x130 [12442.434078] ? kthread_complete_and_exit+0x20/0x20 [12442.438863] ret_from_fork+0x31/0x40 [12442.442441] ? kthread_complete_and_exit+0x20/0x20 [12442.447225] ret_from_fork_asm+0x11/0x20 [12442.451159] [12469.673143] watchdog: BUG: soft lockup - CPU#0 stuck for 499s! [dbench:78683] [12469.680274] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12469.685143] watchdog: BUG: soft lockup - CPU#7 stuck for 9496s! [kworker/7:4:78808] [12469.735716] irq event stamp: 13992626 [12469.743358] Modules linked in: ceph [12469.747016] hardirqs last enabled at (13992625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12469.747018] libceph dns_resolver fscache [12469.760656] hardirqs last disabled at (13992626): [] sysvec_apic_timer_interrupt+0xe/0x90 [12469.760659] softirqs last enabled at (13992550): [] __do_softirq+0x2be/0x371 [12469.764662] netfs veth [12469.774385] softirqs last disabled at (13992545): [] irq_exit_rcu+0xb0/0xd0 [12469.783070] nft_chain_nat [12469.785514] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12469.794042] xt_MASQUERADE nf_nat [12469.796745] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12469.805690] nf_conntrack nf_defrag_ipv6 [12469.809001] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12469.816472] nf_defrag_ipv4 nft_compat [12469.820391] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12469.825866] nf_tables nfnetlink [12469.829611] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12469.848348] bridge stp [12469.851571] [12469.856791] llc xfs [12469.859233] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12469.860724] libcrc32c sunrpc [12469.862908] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12469.870034] intel_rapl_msr [12469.872998] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12469.880121] intel_rapl_common sb_edac [12469.882913] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12469.890035] x86_pkg_temp_thermal [12469.893780] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12469.900903] coretemp kvm_intel [12469.904216] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12469.911339] kvm irqbypass [12469.914476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12469.922555] i2c_i801 iTCO_wdt [12469.925258] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12469.930995] iTCO_vendor_support i2c_smbus [12469.934045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12469.941169] crct10dif_pclmul joydev [12469.945260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12469.952385] crc32_pclmul ipmi_ssif ghash_clmulni_intel [12469.955955] Call Trace: [12469.963079] mei_me mei [12469.968298] [12469.970741] lpc_ich [12469.973186] ? watchdog_timer_fn+0x1b4/0x220 [12469.975197] sha512_ssse3 acpi_ipmi [12469.977380] ? __hrtimer_run_queues+0x1d3/0x360 [12469.981644] mfd_core wmi [12469.985131] ? hrtimer_interrupt+0x104/0x240 [12469.989653] ipmi_si [12469.992269] ? __local_bh_enable+0x50/0x60 [12469.996540] ipmi_devintf [12469.998729] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12470.002829] ipmi_msghandler ioatdma [12470.005444] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12470.010660] acpi_pad [12470.014232] [12470.019188] acpi_power_meter ixgbe [12470.021460] [12470.023557] igb mdio [12470.027042] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12470.029137] crc32c_intel [12470.031414] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12470.036714] nvme ptp [12470.039334] do_raw_spin_lock+0xb5/0xc0 [12470.044201] nvme_core i2c_algo_bit [12470.046473] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12470.050303] pps_core dca [12470.053788] try_get_cap_refs+0x159/0x880 [ceph] [12470.058570] fuse [12470.061193] __ceph_get_caps+0x4a1/0x760 [ceph] [12470.065800] irq event stamp: 23142314 [12470.067746] ? __file_remove_privs+0x89/0x110 [12470.072264] hardirqs last enabled at (23142313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12470.075923] ? wake_bit_function+0x70/0x70 [12470.080272] hardirqs last disabled at (23142314): [] sysvec_apic_timer_interrupt+0xe/0x90 [12470.090434] ? ceph_write_iter+0x260/0x770 [ceph] [12470.094520] softirqs last enabled at (23142312): [] __do_softirq+0x2be/0x371 [12470.104263] ceph_write_iter+0x260/0x770 [ceph] [12470.108959] softirqs last disabled at (23142307): [] irq_exit_rcu+0xb0/0xd0 [12470.117647] ? lock_is_held_type+0xaa/0x120 [12470.122168] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12470.130681] vfs_write+0x324/0x4f0 [12470.134857] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12470.144239] ksys_pwrite64+0x65/0xa0 [12470.147632] Workqueue: events delayed_work [ceph] [12470.155104] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12470.158673] [12470.163372] do_syscall_64+0x38/0x80 [12470.168501] RIP: 0010:lock_acquire+0xf0/0x2b0 [12470.169993] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12470.173562] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12470.177914] RIP: 0033:0x7f48f0eeb93b [12470.182958] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12470.201695] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12470.205266] [12470.205266] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12470.210482] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12470.229221] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12470.230712] ORIG_RAX: 0000000000000012 [12470.237836] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12470.243054] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12470.250177] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12470.254009] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12470.261133] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12470.268257] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12470.275381] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12470.282503] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12470.289629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12470.296753] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12470.304829] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12470.311959] [12470.317693] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12470.341267] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12470.348389] Call Trace: [12470.350835] [12470.352845] ? watchdog_timer_fn+0x1b4/0x220 [12470.357111] ? __hrtimer_run_queues+0x1d3/0x360 [12470.361638] ? hrtimer_interrupt+0x104/0x240 [12470.365907] ? __local_bh_enable+0x50/0x60 [12470.370000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12470.375222] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12470.380181] [12470.382278] [12470.384375] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12470.389684] ? lock_acquire+0xf0/0x2b0 [12470.393434] ? igrab+0x43/0x50 [12470.396483] ? lock_release+0xc5/0x260 [12470.400228] _raw_spin_lock+0x30/0x40 [12470.403885] ? igrab+0x1d/0x50 [12470.406934] igrab+0x1d/0x50 [12470.409813] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12470.414979] delayed_work+0x1d8/0x280 [ceph] [12470.419263] ? process_scheduled_works+0x22a/0x510 [12470.424051] process_scheduled_works+0x2a4/0x510 [12470.428666] worker_thread+0x188/0x340 [12470.432417] ? rescuer_thread+0x310/0x310 [12470.436418] kthread+0xf5/0x130 [12470.439558] ? kthread_complete_and_exit+0x20/0x20 [12470.444342] ret_from_fork+0x31/0x40 [12470.447912] ? kthread_complete_and_exit+0x20/0x20 [12470.452695] ret_from_fork_asm+0x11/0x20 [12470.456618] [12497.673634] watchdog: BUG: soft lockup - CPU#0 stuck for 525s! [dbench:78683] [12497.680761] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12497.685634] watchdog: BUG: soft lockup - CPU#7 stuck for 9522s! [kworker/7:4:78808] [12497.736202] irq event stamp: 14050244 [12497.736203] hardirqs last enabled at (14050243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12497.743846] Modules linked in: ceph [12497.747503] hardirqs last disabled at (14050244): [] sysvec_apic_timer_interrupt+0xe/0x90 [12497.757661] libceph [12497.761143] softirqs last enabled at (14050238): [] __do_softirq+0x2be/0x371 [12497.770869] dns_resolver fscache [12497.773052] softirqs last disabled at (14050233): [] irq_exit_rcu+0xb0/0xd0 [12497.781737] netfs veth [12497.785049] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12497.793557] nft_chain_nat xt_MASQUERADE [12497.796002] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12497.804964] nf_nat nf_conntrack [12497.808882] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12497.816352] nf_defrag_ipv6 [12497.819576] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12497.825055] nf_defrag_ipv4 nft_compat [12497.827844] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12497.846581] nf_tables nfnetlink [12497.850326] [12497.850327] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12497.855544] bridge stp [12497.858769] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12497.860259] llc xfs [12497.867383] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12497.869827] libcrc32c sunrpc [12497.876951] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12497.879136] intel_rapl_msr intel_rapl_common [12497.886259] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12497.889224] sb_edac x86_pkg_temp_thermal [12497.896347] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12497.900697] coretemp kvm_intel [12497.907821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12497.911825] kvm [12497.919905] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12497.923041] irqbypass i2c_i801 [12497.928778] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12497.930615] iTCO_wdt iTCO_vendor_support [12497.937741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12497.940877] i2c_smbus crct10dif_pclmul [12497.948002] Call Trace: [12497.948003] [12497.952007] joydev crc32_pclmul [12497.959148] ? watchdog_timer_fn+0x1b4/0x220 [12497.962978] ipmi_ssif ghash_clmulni_intel [12497.965422] ? __hrtimer_run_queues+0x1d3/0x360 [12497.967433] mei_me mei [12497.970661] ? hrtimer_interrupt+0x104/0x240 [12497.974920] lpc_ich sha512_ssse3 [12497.979030] ? __local_bh_enable+0x50/0x60 [12497.983552] acpi_ipmi mfd_core [12497.985999] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12497.990260] wmi ipmi_si [12497.993572] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12497.997662] ipmi_devintf ipmi_msghandler [12498.000801] [12498.006033] ioatdma [12498.008564] [12498.013523] acpi_pad acpi_power_meter [12498.017526] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12498.019624] ixgbe igb [12498.021813] ? queued_spin_lock_slowpath+0x18/0x2e0 [12498.023906] mdio crc32c_intel [12498.027651] do_raw_spin_lock+0xb5/0xc0 [12498.032954] nvme [12498.035329] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12498.040200] ptp nvme_core [12498.043251] try_get_cap_refs+0x159/0x880 [ceph] [12498.047081] i2c_algo_bit pps_core [12498.049030] __ceph_get_caps+0x4a1/0x760 [ceph] [12498.053790] dca fuse [12498.056496] ? __file_remove_privs+0x89/0x110 [12498.061104] [12498.061105] irq event stamp: 23364502 [12498.064502] ? wake_bit_function+0x70/0x70 [12498.069032] hardirqs last enabled at (23364501): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12498.071317] ? ceph_write_iter+0x260/0x770 [ceph] [12498.075664] hardirqs last disabled at (23364502): [] sysvec_apic_timer_interrupt+0xe/0x90 [12498.077155] ceph_write_iter+0x260/0x770 [ceph] [12498.080811] softirqs last enabled at (23364500): [] __do_softirq+0x2be/0x371 [12498.084906] ? lock_is_held_type+0xaa/0x120 [12498.095060] softirqs last disabled at (23364495): [] irq_exit_rcu+0xb0/0xd0 [12498.099759] vfs_write+0x324/0x4f0 [12498.109483] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12498.114031] ksys_pwrite64+0x65/0xa0 [12498.122708] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12498.126885] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12498.135397] Workqueue: events delayed_work [ceph] [12498.138795] do_syscall_64+0x38/0x80 [12498.148171] [12498.148172] RIP: 0010:lock_release+0x14e/0x260 [12498.151743] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12498.159212] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [12498.164344] RIP: 0033:0x7f48f0eeb93b [12498.169040] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [12498.172611] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12498.174101] [12498.174102] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [12498.178539] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12498.183583] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [12498.202326] ORIG_RAX: 0000000000000012 [12498.205901] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [12498.211117] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12498.229854] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [12498.231347] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12498.238471] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [12498.243688] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12498.250811] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12498.254642] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12498.261766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12498.268891] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12498.276014] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12498.283162] [12498.290281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12498.341901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12498.349038] Call Trace: [12498.351486] [12498.353497] ? watchdog_timer_fn+0x1b4/0x220 [12498.357761] ? __hrtimer_run_queues+0x1d3/0x360 [12498.362289] ? hrtimer_interrupt+0x104/0x240 [12498.366558] ? __local_bh_enable+0x50/0x60 [12498.370650] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12498.375867] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12498.380822] [12498.382922] [12498.385018] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12498.390340] ? igrab+0x43/0x50 [12498.393394] ? lock_release+0x14e/0x260 [12498.397229] ? lock_release+0xc5/0x260 [12498.400975] _raw_spin_unlock+0x1b/0x40 [12498.404804] igrab+0x43/0x50 [12498.407684] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12498.412833] delayed_work+0x1d8/0x280 [ceph] [12498.417113] ? process_scheduled_works+0x22a/0x510 [12498.421905] process_scheduled_works+0x2a4/0x510 [12498.426520] worker_thread+0x188/0x340 [12498.430268] ? rescuer_thread+0x310/0x310 [12498.434272] kthread+0xf5/0x130 [12498.437409] ? kthread_complete_and_exit+0x20/0x20 [12498.442195] ret_from_fork+0x31/0x40 [12498.445773] ? kthread_complete_and_exit+0x20/0x20 [12498.450557] ret_from_fork_asm+0x11/0x20 [12498.454479] [12525.674125] watchdog: BUG: soft lockup - CPU#0 stuck for 551s! [dbench:78683] [12525.681258] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12525.686125] watchdog: BUG: soft lockup - CPU#7 stuck for 9548s! [kworker/7:4:78808] [12525.736708] irq event stamp: 14108020 [12525.744351] Modules linked in: ceph libceph dns_resolver fscache netfs [12525.748007] hardirqs last enabled at (14108019): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12525.754526] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12525.764682] hardirqs last disabled at (14108020): [] sysvec_apic_timer_interrupt+0xe/0x90 [12525.772067] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [12525.781801] softirqs last enabled at (14107954): [] __do_softirq+0x2be/0x371 [12525.788587] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [12525.797269] softirqs last disabled at (14107949): [] irq_exit_rcu+0xb0/0xd0 [12525.804133] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [12525.812647] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12525.819596] iTCO_wdt iTCO_vendor_support i2c_smbus [12525.828541] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12525.828543] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12525.833412] crct10dif_pclmul joydev [12525.840884] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12525.846359] crc32_pclmul ipmi_ssif [12525.849931] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12525.868668] ghash_clmulni_intel mei_me [12525.872151] [12525.872152] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12525.877370] mei lpc_ich [12525.881200] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12525.882690] sha512_ssse3 [12525.889815] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12525.892345] acpi_ipmi [12525.899470] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12525.902086] mfd_core wmi [12525.909211] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12525.911568] ipmi_si ipmi_devintf [12525.918694] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12525.921309] ipmi_msghandler [12525.928434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12525.931745] ioatdma acpi_pad [12525.939823] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12525.942699] acpi_power_meter [12525.948437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12525.951400] ixgbe igb [12525.958526] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12525.961489] mdio [12525.968613] Call Trace: [12525.968615] [12525.970971] crc32c_intel [12525.978098] ? watchdog_timer_fn+0x1b4/0x220 [12525.980019] nvme ptp [12525.982463] ? __hrtimer_run_queues+0x1d3/0x360 [12525.984475] nvme_core [12525.987094] ? hrtimer_interrupt+0x104/0x240 [12525.991354] i2c_algo_bit pps_core [12525.993625] ? __local_bh_enable+0x50/0x60 [12525.998150] dca fuse [12526.000525] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12526.004790] [12526.004790] irq event stamp: 23586602 [12526.008203] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12526.012294] hardirqs last enabled at (23586601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12526.014566] [12526.019783] hardirqs last disabled at (23586602): [] sysvec_apic_timer_interrupt+0xe/0x90 [12526.021290] [12526.021292] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12526.024948] softirqs last enabled at (23586600): [] __do_softirq+0x2be/0x371 [12526.029910] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12526.040062] softirqs last disabled at (23586595): [] irq_exit_rcu+0xb0/0xd0 [12526.042162] do_raw_spin_lock+0xb5/0xc0 [12526.051883] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12526.053985] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12526.059286] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12526.067973] try_get_cap_refs+0x159/0x880 [ceph] [12526.072842] Workqueue: events delayed_work [ceph] [12526.081358] __ceph_get_caps+0x4a1/0x760 [ceph] [12526.085182] [12526.094563] ? __file_remove_privs+0x89/0x110 [12526.099343] RIP: 0010:lock_acquire+0xf0/0x2b0 [12526.106816] ? wake_bit_function+0x70/0x70 [12526.111425] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12526.116126] ? ceph_write_iter+0x260/0x770 [ceph] [12526.120647] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12526.122137] ceph_write_iter+0x260/0x770 [ceph] [12526.126488] [12526.126489] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12526.130843] ? lock_is_held_type+0xaa/0x120 [12526.134929] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12526.153670] vfs_write+0x324/0x4f0 [12526.158365] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12526.163586] ksys_pwrite64+0x65/0xa0 [12526.168107] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12526.169598] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12526.176721] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12526.180900] do_syscall_64+0x38/0x80 [12526.188024] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12526.191420] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12526.198544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12526.202116] RIP: 0033:0x7f48f0eeb93b [12526.209238] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12526.214370] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12526.221494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12526.225064] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12526.233142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12526.238186] ORIG_RAX: 0000000000000012 [12526.243924] Call Trace: [12526.243926] [12526.247494] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12526.254619] ? watchdog_timer_fn+0x1b4/0x220 [12526.273356] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12526.280481] ? __hrtimer_run_queues+0x1d3/0x360 [12526.285697] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12526.292827] ? hrtimer_interrupt+0x104/0x240 [12526.296653] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12526.299096] ? __local_bh_enable+0x50/0x60 [12526.301107] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12526.308235] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12526.312500] [12526.319621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12526.366230] [12526.368327] [12526.370426] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12526.375732] ? lock_acquire+0xf0/0x2b0 [12526.379484] ? igrab+0x43/0x50 [12526.382534] ? lock_release+0xc5/0x260 [12526.386278] _raw_spin_lock+0x30/0x40 [12526.389936] ? igrab+0x1d/0x50 [12526.392984] igrab+0x1d/0x50 [12526.395864] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12526.401013] delayed_work+0x1d8/0x280 [ceph] [12526.405296] ? process_scheduled_works+0x22a/0x510 [12526.410083] process_scheduled_works+0x2a4/0x510 [12526.414701] worker_thread+0x188/0x340 [12526.418450] ? rescuer_thread+0x310/0x310 [12526.422451] kthread+0xf5/0x130 [12526.425589] ? kthread_complete_and_exit+0x20/0x20 [12526.430375] ret_from_fork+0x31/0x40 [12526.433954] ? kthread_complete_and_exit+0x20/0x20 [12526.438737] ret_from_fork_asm+0x11/0x20 [12526.442661] [12553.674616] watchdog: BUG: soft lockup - CPU#0 stuck for 577s! [dbench:78683] [12553.681743] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12553.686616] watchdog: BUG: soft lockup - CPU#7 stuck for 9574s! [kworker/7:4:78808] [12553.737184] irq event stamp: 14165652 [12553.737185] hardirqs last enabled at (14165651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12553.744829] Modules linked in: [12553.748486] hardirqs last disabled at (14165652): [] sysvec_apic_timer_interrupt+0xe/0x90 [12553.758644] ceph [12553.761695] softirqs last enabled at (14165632): [] __do_softirq+0x2be/0x371 [12553.771419] libceph [12553.773342] softirqs last disabled at (14165627): [] irq_exit_rcu+0xb0/0xd0 [12553.782027] dns_resolver fscache [12553.784211] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12553.792723] netfs veth [12553.796034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12553.804977] nft_chain_nat xt_MASQUERADE [12553.807422] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12553.814891] nf_nat nf_conntrack [12553.818808] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12553.824291] nf_defrag_ipv6 nf_defrag_ipv4 [12553.827510] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12553.846248] nft_compat nf_tables [12553.850340] [12553.855557] nfnetlink bridge [12553.858868] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12553.860359] stp llc [12553.863322] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12553.870447] xfs libcrc32c [12553.872629] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12553.879756] sunrpc [12553.882458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12553.889582] intel_rapl_msr intel_rapl_common [12553.891680] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12553.898805] sb_edac x86_pkg_temp_thermal [12553.903155] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12553.910293] coretemp kvm_intel [12553.914301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12553.922378] kvm irqbypass [12553.925516] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12553.931254] i2c_i801 iTCO_wdt [12553.933957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12553.941081] iTCO_vendor_support i2c_smbus [12553.944131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12553.951254] crct10dif_pclmul joydev [12553.955346] Call Trace: [12553.955348] [12553.962471] crc32_pclmul ipmi_ssif ghash_clmulni_intel [12553.966042] ? watchdog_timer_fn+0x1b4/0x220 [12553.968485] mei_me mei [12553.970499] ? __hrtimer_run_queues+0x1d3/0x360 [12553.975714] lpc_ich sha512_ssse3 [12553.979981] ? hrtimer_interrupt+0x104/0x240 [12553.982422] acpi_ipmi mfd_core [12553.986946] ? __local_bh_enable+0x50/0x60 [12553.990256] wmi ipmi_si [12553.994522] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12553.997657] ipmi_devintf ipmi_msghandler [12554.001750] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12554.004296] ioatdma acpi_pad [12554.009513] [12554.009514] [12554.013519] acpi_power_meter [12554.018476] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12554.021441] ixgbe igb [12554.023542] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12554.025636] mdio crc32c_intel [12554.028601] do_raw_spin_lock+0xb5/0xc0 [12554.033904] nvme [12554.036261] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12554.041131] ptp nvme_core [12554.044183] try_get_cap_refs+0x159/0x880 [ceph] [12554.048016] i2c_algo_bit pps_core [12554.049942] __ceph_get_caps+0x4a1/0x760 [ceph] [12554.054719] dca fuse [12554.057428] ? __file_remove_privs+0x89/0x110 [12554.062036] [12554.062037] irq event stamp: 23810270 [12554.065434] ? wake_bit_function+0x70/0x70 [12554.069955] hardirqs last enabled at (23810269): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12554.072231] ? ceph_write_iter+0x260/0x770 [ceph] [12554.076577] hardirqs last disabled at (23810270): [] sysvec_apic_timer_interrupt+0xe/0x90 [12554.078068] ceph_write_iter+0x260/0x770 [ceph] [12554.081725] softirqs last enabled at (23810268): [] __do_softirq+0x2be/0x371 [12554.085822] ? lock_is_held_type+0xaa/0x120 [12554.095974] softirqs last disabled at (23810263): [] irq_exit_rcu+0xb0/0xd0 [12554.100675] vfs_write+0x324/0x4f0 [12554.110396] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12554.114923] ksys_pwrite64+0x65/0xa0 [12554.123605] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12554.127782] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12554.136293] Workqueue: events delayed_work [ceph] [12554.139691] do_syscall_64+0x38/0x80 [12554.149066] [12554.149067] RIP: 0010:lock_acquire+0xf0/0x2b0 [12554.152641] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12554.160109] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12554.165240] RIP: 0033:0x7f48f0eeb93b [12554.169937] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12554.173509] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12554.175015] [12554.175016] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12554.179367] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12554.184410] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12554.203149] ORIG_RAX: 0000000000000012 [12554.206720] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12554.211937] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12554.230675] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12554.232165] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12554.239300] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12554.244515] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12554.251639] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12554.255471] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12554.262594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12554.269719] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12554.276843] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12554.283973] [12554.291091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12554.342745] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12554.349868] Call Trace: [12554.352314] [12554.354325] ? watchdog_timer_fn+0x1b4/0x220 [12554.358591] ? __hrtimer_run_queues+0x1d3/0x360 [12554.363115] ? hrtimer_interrupt+0x104/0x240 [12554.367377] ? __local_bh_enable+0x50/0x60 [12554.371471] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12554.376695] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12554.381651] [12554.383748] [12554.385845] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12554.391154] ? lock_acquire+0xf0/0x2b0 [12554.394905] ? igrab+0x43/0x50 [12554.397954] ? lock_release+0xc5/0x260 [12554.401700] _raw_spin_lock+0x30/0x40 [12554.405364] ? igrab+0x1d/0x50 [12554.408414] igrab+0x1d/0x50 [12554.411293] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12554.416460] delayed_work+0x1d8/0x280 [ceph] [12554.420741] ? process_scheduled_works+0x22a/0x510 [12554.425531] process_scheduled_works+0x2a4/0x510 [12554.430147] worker_thread+0x188/0x340 [12554.433894] ? rescuer_thread+0x310/0x310 [12554.437898] kthread+0xf5/0x130 [12554.441035] ? kthread_complete_and_exit+0x20/0x20 [12554.445820] ret_from_fork+0x31/0x40 [12554.449400] ? kthread_complete_and_exit+0x20/0x20 [12554.454185] ret_from_fork_asm+0x11/0x20 [12554.458107] [12581.675107] watchdog: BUG: soft lockup - CPU#0 stuck for 603s! [dbench:78683] [12581.682239] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12581.687107] watchdog: BUG: soft lockup - CPU#7 stuck for 9600s! [kworker/7:4:78808] [12581.737691] irq event stamp: 14223476 [12581.745335] Modules linked in: ceph libceph dns_resolver fscache netfs [12581.748993] hardirqs last enabled at (14223475): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12581.755516] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [12581.765674] hardirqs last disabled at (14223476): [] sysvec_apic_timer_interrupt+0xe/0x90 [12581.771758] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [12581.781500] softirqs last enabled at (14223410): [] __do_softirq+0x2be/0x371 [12581.789231] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [12581.797917] softirqs last disabled at (14223405): [] irq_exit_rcu+0xb0/0xd0 [12581.804441] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [12581.812955] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12581.812955] irqbypass [12581.812957] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12581.818949] i2c_i801 iTCO_wdt [12581.827896] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12581.830252] iTCO_vendor_support i2c_smbus [12581.837722] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12581.840788] crct10dif_pclmul joydev [12581.846267] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12581.850358] crc32_pclmul [12581.869098] [12581.872668] ipmi_ssif ghash_clmulni_intel [12581.877884] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12581.880503] mei_me [12581.881992] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12581.886083] mei lpc_ich [12581.893208] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12581.895304] sha512_ssse3 acpi_ipmi [12581.902430] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12581.904960] mfd_core wmi [12581.912083] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12581.915568] ipmi_si ipmi_devintf [12581.922692] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12581.925310] ipmi_msghandler ioatdma [12581.932432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12581.935744] acpi_pad acpi_power_meter [12581.943823] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12581.947393] ixgbe igb [12581.953130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12581.956874] mdio crc32c_intel [12581.963999] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12581.966355] nvme ptp [12581.973480] Call Trace: [12581.976548] nvme_core [12581.983673] [12581.985942] i2c_algo_bit pps_core [12581.988389] ? watchdog_timer_fn+0x1b4/0x220 [12581.990743] dca fuse [12581.992755] ? __hrtimer_run_queues+0x1d3/0x360 [12581.996152] [12581.996153] irq event stamp: 24033368 [12582.000420] ? hrtimer_interrupt+0x104/0x240 [12582.002686] hardirqs last enabled at (24033367): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12582.007212] ? __local_bh_enable+0x50/0x60 [12582.008702] hardirqs last disabled at (24033368): [] sysvec_apic_timer_interrupt+0xe/0x90 [12582.012360] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12582.016623] softirqs last enabled at (24033366): [] __do_softirq+0x2be/0x371 [12582.026781] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12582.030888] softirqs last disabled at (24033361): [] irq_exit_rcu+0xb0/0xd0 [12582.040613] [12582.045831] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12582.054513] [12582.059472] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12582.067982] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12582.070081] Workqueue: events delayed_work [ceph] [12582.079462] ? queued_spin_lock_slowpath+0x16/0x2e0 [12582.081556] [12582.081557] RIP: 0010:lock_acquire+0xf0/0x2b0 [12582.089027] do_raw_spin_lock+0xb5/0xc0 [12582.094331] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12582.099029] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12582.103898] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12582.105391] try_get_cap_refs+0x159/0x880 [ceph] [12582.109741] [12582.109742] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12582.113577] __ceph_get_caps+0x4a1/0x760 [ceph] [12582.132309] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12582.137095] ? __file_remove_privs+0x89/0x110 [12582.142310] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12582.146922] ? wake_bit_function+0x70/0x70 [12582.148412] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12582.155537] ? ceph_write_iter+0x260/0x770 [ceph] [12582.160059] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12582.167185] ceph_write_iter+0x260/0x770 [ceph] [12582.171535] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12582.178663] ? lock_is_held_type+0xaa/0x120 [12582.182748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12582.189877] vfs_write+0x324/0x4f0 [12582.194571] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12582.201700] ksys_pwrite64+0x65/0xa0 [12582.206218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12582.214298] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12582.218472] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12582.224213] do_syscall_64+0x38/0x80 [12582.227608] Call Trace: [12582.227610] [12582.234734] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12582.238305] ? watchdog_timer_fn+0x1b4/0x220 [12582.245428] RIP: 0033:0x7f48f0eeb93b [12582.250560] ? __hrtimer_run_queues+0x1d3/0x360 [12582.257682] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12582.261256] ? hrtimer_interrupt+0x104/0x240 [12582.263698] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12582.265709] ? __local_bh_enable+0x50/0x60 [12582.270752] ORIG_RAX: 0000000000000012 [12582.275018] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12582.278586] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12582.283113] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12582.301848] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12582.306113] [12582.311330] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12582.315420] [12582.319250] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12582.324471] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12582.331593] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12582.336554] ? lock_acquire+0xf0/0x2b0 [12582.343681] [12582.345790] ? igrab+0x43/0x50 [12582.383561] ? lock_release+0xc5/0x260 [12582.387315] _raw_spin_lock+0x30/0x40 [12582.390979] ? igrab+0x1d/0x50 [12582.394029] igrab+0x1d/0x50 [12582.396909] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12582.402057] delayed_work+0x1d8/0x280 [ceph] [12582.406339] ? process_scheduled_works+0x22a/0x510 [12582.411129] process_scheduled_works+0x2a4/0x510 [12582.415745] worker_thread+0x188/0x340 [12582.419511] ? rescuer_thread+0x310/0x310 [12582.423513] kthread+0xf5/0x130 [12582.426653] ? kthread_complete_and_exit+0x20/0x20 [12582.431436] ret_from_fork+0x31/0x40 [12582.435016] ? kthread_complete_and_exit+0x20/0x20 [12582.439800] ret_from_fork_asm+0x11/0x20 [12582.443733] [12588.173587] rcu: INFO: rcu_preempt self-detected stall on CPU [12588.179344] rcu: 0-....: (639722 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=291581 [12588.189502] rcu: (t=651551 jiffies g=1309609 q=84990 ncpus=8) [12588.195326] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12588.204271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12588.211741] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12588.217225] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12588.235963] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12588.241180] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12588.248305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12588.255429] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12588.262554] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12588.269679] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12588.276803] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12588.284896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12588.290633] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12588.297758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12588.304898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12588.312025] Call Trace: [12588.314469] [12588.316479] ? rcu_dump_cpu_stacks+0xd1/0x180 [12588.320840] ? rcu_sched_clock_irq+0x633/0x1390 [12588.325374] ? lock_is_held_type+0xaa/0x120 [12588.329560] ? lock_is_held_type+0xaa/0x120 [12588.333746] ? tick_sched_do_timer+0xa0/0xa0 [12588.338015] ? rcu_read_lock_sched_held+0x56/0x90 [12588.342718] ? tick_sched_do_timer+0xa0/0xa0 [12588.346987] ? update_process_times+0x5f/0x90 [12588.351338] ? tick_sched_handle.isra.16+0x1e/0x50 [12588.356131] ? tick_sched_timer+0x6f/0x80 [12588.360145] ? __hrtimer_run_queues+0x1d3/0x360 [12588.364683] ? hrtimer_interrupt+0x104/0x240 [12588.368948] ? __local_bh_enable+0x50/0x60 [12588.373044] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12588.378265] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12588.383223] [12588.385318] [12588.387418] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12588.392739] ? queued_spin_lock_slowpath+0x16/0x2e0 [12588.397622] do_raw_spin_lock+0xb5/0xc0 [12588.401458] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12588.406262] try_get_cap_refs+0x159/0x880 [ceph] [12588.410919] __ceph_get_caps+0x4a1/0x760 [ceph] [12588.415471] ? __file_remove_privs+0x89/0x110 [12588.419824] ? wake_bit_function+0x70/0x70 [12588.423927] ? ceph_write_iter+0x260/0x770 [ceph] [12588.428638] ceph_write_iter+0x260/0x770 [ceph] [12588.433184] ? lock_is_held_type+0xaa/0x120 [12588.437369] vfs_write+0x324/0x4f0 [12588.440780] ksys_pwrite64+0x65/0xa0 [12588.444359] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12588.449500] do_syscall_64+0x38/0x80 [12588.453079] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12588.458130] RIP: 0033:0x7f48f0eeb93b [12588.461702] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12588.480448] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12588.488014] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12588.495136] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12588.502261] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12588.509385] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12588.516510] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12588.523647] [12588.525835] Sending NMI from CPU 0 to CPUs 7: [12588.530187] NMI backtrace for cpu 7 [12588.530189] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12588.530191] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12588.530191] Workqueue: events delayed_work [ceph] [12588.530207] RIP: 0010:mark_lock+0x8f/0x4c0 [12588.530209] Code: 04 90 48 c1 e0 04 48 05 20 55 f7 82 41 bc 01 00 00 00 4c 85 70 60 0f 84 98 00 00 00 48 8b 4c 24 68 65 48 33 0c 25 28 00 00 00 <44> 89 e0 0f 85 1e 04 00 00 48 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f [12588.530210] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000046 [12588.530211] RAX: ffffffff82fcfc30 RBX: 0000000000000006 RCX: 0000000000000000 [12588.530212] RDX: 00000000000014df RSI: ffff88810debb550 RDI: ffff88810debaa80 [12588.530213] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [12588.530214] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000001 [12588.530214] R13: ffff88810debb550 R14: 0000000000000040 R15: ffff88810debaa80 [12588.530215] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12588.530216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12588.530217] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12588.530218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12588.530218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12588.530219] Call Trace: [12588.530220] [12588.530221] ? nmi_cpu_backtrace+0xad/0x120 [12588.530224] ? nmi_cpu_backtrace_handler+0x11/0x20 [12588.530226] ? nmi_handle+0xd4/0x260 [12588.530230] ? default_do_nmi+0x49/0xf0 [12588.530231] ? exc_nmi+0x11e/0x150 [12588.530232] ? end_repeat_nmi+0x16/0x67 [12588.530236] ? mark_lock+0x8f/0x4c0 [12588.530237] ? mark_lock+0x8f/0x4c0 [12588.530239] ? mark_lock+0x8f/0x4c0 [12588.530240] [12588.530240] [12588.530241] ? _raw_spin_unlock+0x23/0x40 [12588.530243] ? ceph_check_caps+0x275/0xae0 [ceph] [12588.530256] ? mark_held_locks+0x48/0x70 [12588.530257] __lock_acquire+0x4be/0x1e20 [12588.530260] lock_acquire+0xcb/0x2b0 [12588.530261] ? igrab+0x1d/0x50 [12588.530263] ? igrab+0x43/0x50 [12588.530264] ? lock_release+0xc5/0x260 [12588.530265] _raw_spin_lock+0x30/0x40 [12588.530267] ? igrab+0x1d/0x50 [12588.530268] igrab+0x1d/0x50 [12588.530269] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12588.530282] delayed_work+0x1d8/0x280 [ceph] [12588.530294] ? process_scheduled_works+0x22a/0x510 [12588.530296] process_scheduled_works+0x2a4/0x510 [12588.530300] worker_thread+0x188/0x340 [12588.530302] ? rescuer_thread+0x310/0x310 [12588.530304] kthread+0xf5/0x130 [12588.530306] ? kthread_complete_and_exit+0x20/0x20 [12588.530308] ret_from_fork+0x31/0x40 [12588.530310] ? kthread_complete_and_exit+0x20/0x20 [12588.530312] ret_from_fork_asm+0x11/0x20 [12588.530316] [12613.675668] watchdog: BUG: soft lockup - CPU#0 stuck for 633s! [dbench:78683] [12613.682797] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12613.687668] watchdog: BUG: soft lockup - CPU#7 stuck for 9630s! [kworker/7:4:78808] [12613.738238] irq event stamp: 14289010 [12613.745883] Modules linked in: ceph libceph dns_resolver fscache [12613.749541] hardirqs last enabled at (14289009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12613.755546] netfs veth [12613.765704] hardirqs last disabled at (14289010): [] sysvec_apic_timer_interrupt+0xe/0x90 [12613.768148] nft_chain_nat xt_MASQUERADE [12613.777872] softirqs last enabled at (14289006): [] __do_softirq+0x2be/0x371 [12613.777874] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [12613.790472] softirqs last disabled at (14289001): [] irq_exit_rcu+0xb0/0xd0 [12613.797250] nf_tables nfnetlink bridge [12613.805762] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12613.805764] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12613.809593] stp llc [12613.818537] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12613.826007] xfs libcrc32c [12613.828193] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12613.833668] sunrpc intel_rapl_msr [12613.836373] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12613.855111] intel_rapl_common [12613.858508] [12613.858509] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12613.863726] sb_edac [12613.866776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12613.868267] x86_pkg_temp_thermal [12613.875392] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12613.877575] coretemp kvm_intel [12613.884700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12613.888009] kvm [12613.895135] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12613.898271] irqbypass [12613.905396] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12613.907234] i2c_i801 iTCO_wdt [12613.914358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12613.916714] iTCO_vendor_support i2c_smbus [12613.924791] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12613.927843] crct10dif_pclmul joydev [12613.933580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12613.937670] crc32_pclmul ipmi_ssif [12613.944795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12613.948365] ghash_clmulni_intel mei_me [12613.955490] Call Trace: [12613.955491] [12613.958974] mei lpc_ich [12613.966099] ? watchdog_timer_fn+0x1b4/0x220 [12613.969929] sha512_ssse3 acpi_ipmi [12613.972375] ? __hrtimer_run_queues+0x1d3/0x360 [12613.974384] mfd_core [12613.976917] ? hrtimer_interrupt+0x104/0x240 [12613.981179] wmi ipmi_si [12613.984663] ? __local_bh_enable+0x50/0x60 [12613.989187] ipmi_devintf ipmi_msghandler [12613.991460] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12613.995721] ioatdma acpi_pad [12613.998253] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12614.002342] acpi_power_meter ixgbe [12614.006348] [12614.011565] igb [12614.014528] [12614.014530] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12614.019485] mdio crc32c_intel [12614.022973] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12614.025068] nvme ptp [12614.026907] do_raw_spin_lock+0xb5/0xc0 [12614.029003] nvme_core i2c_algo_bit [12614.034308] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12614.037357] pps_core dca [12614.042230] try_get_cap_refs+0x159/0x880 [ceph] [12614.044497] fuse [12614.048350] __ceph_get_caps+0x4a1/0x760 [ceph] [12614.051830] irq event stamp: 24286054 [12614.056619] ? __file_remove_privs+0x89/0x110 [12614.059232] hardirqs last enabled at (24286053): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12614.063844] ? wake_bit_function+0x70/0x70 [12614.065767] hardirqs last disabled at (24286054): [] sysvec_apic_timer_interrupt+0xe/0x90 [12614.070294] ? ceph_write_iter+0x260/0x770 [ceph] [12614.073948] softirqs last enabled at (24286052): [] __do_softirq+0x2be/0x371 [12614.078298] ceph_write_iter+0x260/0x770 [ceph] [12614.088457] softirqs last disabled at (24286047): [] irq_exit_rcu+0xb0/0xd0 [12614.092551] ? lock_is_held_type+0xaa/0x120 [12614.102270] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12614.106972] vfs_write+0x324/0x4f0 [12614.115652] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12614.120181] ksys_pwrite64+0x65/0xa0 [12614.128688] Workqueue: events delayed_work [ceph] [12614.132866] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12614.142244] [12614.142245] RIP: 0010:lock_acquire+0xf0/0x2b0 [12614.145642] do_syscall_64+0x38/0x80 [12614.153112] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12614.156684] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12614.161380] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12614.166509] RIP: 0033:0x7f48f0eeb93b [12614.168000] [12614.172363] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12614.175929] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12614.194669] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12614.199713] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12614.204929] ORIG_RAX: 0000000000000012 [12614.208500] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12614.209990] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12614.228728] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12614.235853] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12614.241070] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12614.248195] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12614.252025] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12614.259149] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12614.266274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12614.273397] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12614.280523] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12614.287652] [12614.294769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12614.339267] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12614.346389] Call Trace: [12614.348834] [12614.350845] ? watchdog_timer_fn+0x1b4/0x220 [12614.355109] ? __hrtimer_run_queues+0x1d3/0x360 [12614.359638] ? hrtimer_interrupt+0x104/0x240 [12614.363906] ? __local_bh_enable+0x50/0x60 [12614.367999] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12614.373223] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12614.378181] [12614.380276] [12614.382376] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12614.387684] ? lock_acquire+0xf0/0x2b0 [12614.391434] ? igrab+0x43/0x50 [12614.394482] ? lock_release+0xc5/0x260 [12614.398230] _raw_spin_lock+0x30/0x40 [12614.401892] ? igrab+0x1d/0x50 [12614.404943] igrab+0x1d/0x50 [12614.407822] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12614.412972] delayed_work+0x1d8/0x280 [ceph] [12614.417253] ? process_scheduled_works+0x22a/0x510 [12614.422042] process_scheduled_works+0x2a4/0x510 [12614.426658] worker_thread+0x188/0x340 [12614.430409] ? rescuer_thread+0x310/0x310 [12614.434420] kthread+0xf5/0x130 [12614.437557] ? kthread_complete_and_exit+0x20/0x20 [12614.442357] ret_from_fork+0x31/0x40 [12614.445929] ? kthread_complete_and_exit+0x20/0x20 [12614.450714] ret_from_fork_asm+0x11/0x20 [12614.454635] [12641.676159] watchdog: BUG: soft lockup - CPU#0 stuck for 659s! [dbench:78683] [12641.683293] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12641.688159] watchdog: BUG: soft lockup - CPU#7 stuck for 9656s! [kworker/7:4:78808] [12641.738745] irq event stamp: 14347092 [12641.746387] Modules linked in: ceph libceph dns_resolver fscache netfs veth [12641.750045] hardirqs last enabled at (14347091): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12641.757005] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12641.767163] hardirqs last disabled at (14347092): [] sysvec_apic_timer_interrupt+0xe/0x90 [12641.774112] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [12641.783837] softirqs last enabled at (14347036): [] __do_softirq+0x2be/0x371 [12641.790266] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [12641.798951] softirqs last disabled at (14347031): [] irq_exit_rcu+0xb0/0xd0 [12641.805468] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [12641.813981] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12641.819632] kvm irqbypass i2c_i801 [12641.828574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12641.832059] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [12641.839530] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12641.846498] crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me [12641.851977] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12641.857798] mei lpc_ich [12641.876555] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12641.876556] sha512_ssse3 [12641.876557] [12641.879085] acpi_ipmi [12641.884304] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12641.886920] mfd_core wmi [12641.888410] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12641.890767] ipmi_si ipmi_devintf [12641.897892] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12641.900511] ipmi_msghandler ioatdma [12641.907635] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12641.910945] acpi_pad acpi_power_meter [12641.918070] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12641.921640] ixgbe igb [12641.928765] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12641.932507] mdio crc32c_intel [12641.939632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12641.941988] nvme ptp [12641.950067] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12641.953118] nvme_core i2c_algo_bit [12641.958854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12641.961127] pps_core [12641.968249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12641.971733] dca fuse [12641.978857] Call Trace: [12641.978859] [12641.981127] [12641.981128] irq event stamp: 24508418 [12641.988254] ? watchdog_timer_fn+0x1b4/0x220 [12641.990524] hardirqs last enabled at (24508417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12641.992968] ? __hrtimer_run_queues+0x1d3/0x360 [12641.994977] hardirqs last disabled at (24508418): [] sysvec_apic_timer_interrupt+0xe/0x90 [12641.996471] ? hrtimer_interrupt+0x104/0x240 [12642.000127] softirqs last enabled at (24508416): [] __do_softirq+0x2be/0x371 [12642.004391] ? __local_bh_enable+0x50/0x60 [12642.014558] softirqs last disabled at (24508411): [] irq_exit_rcu+0xb0/0xd0 [12642.019083] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12642.028806] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12642.033069] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12642.041753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12642.045844] [12642.054355] Workqueue: events delayed_work [ceph] [12642.059571] [12642.059573] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12642.068950] [12642.073911] ? queued_spin_lock_slowpath+0x18/0x2e0 [12642.081377] RIP: 0010:lock_acquire+0xf0/0x2b0 [12642.083476] do_raw_spin_lock+0xb5/0xc0 [12642.088173] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12642.090272] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12642.095573] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12642.097068] try_get_cap_refs+0x159/0x880 [ceph] [12642.101935] [12642.106292] __ceph_get_caps+0x4a1/0x760 [ceph] [12642.110117] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12642.128860] ? __file_remove_privs+0x89/0x110 [12642.133639] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12642.138857] ? wake_bit_function+0x70/0x70 [12642.143466] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12642.144959] ? ceph_write_iter+0x260/0x770 [ceph] [12642.149480] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12642.156605] ceph_write_iter+0x260/0x770 [ceph] [12642.160957] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12642.168086] ? lock_is_held_type+0xaa/0x120 [12642.172172] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12642.179297] vfs_write+0x324/0x4f0 [12642.183992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12642.191123] ksys_pwrite64+0x65/0xa0 [12642.195642] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12642.202766] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12642.206943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12642.215021] do_syscall_64+0x38/0x80 [12642.218418] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12642.224158] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12642.227726] Call Trace: [12642.227728] [12642.234851] RIP: 0033:0x7f48f0eeb93b [12642.239982] ? watchdog_timer_fn+0x1b4/0x220 [12642.247106] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12642.250677] ? __hrtimer_run_queues+0x1d3/0x360 [12642.257799] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12642.262862] ? hrtimer_interrupt+0x104/0x240 [12642.265305] ORIG_RAX: 0000000000000012 [12642.267317] ? __local_bh_enable+0x50/0x60 [12642.270888] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12642.275152] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12642.293887] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12642.298412] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12642.303630] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12642.307894] [12642.311723] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12642.315814] [12642.315816] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12642.322940] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12642.328162] ? lock_acquire+0xf0/0x2b0 [12642.335287] [12642.340240] ? igrab+0x43/0x50 [12642.380090] ? lock_release+0xc5/0x260 [12642.383847] _raw_spin_lock+0x30/0x40 [12642.387508] ? igrab+0x1d/0x50 [12642.390565] igrab+0x1d/0x50 [12642.393447] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12642.398596] delayed_work+0x1d8/0x280 [ceph] [12642.402878] ? process_scheduled_works+0x22a/0x510 [12642.407666] process_scheduled_works+0x2a4/0x510 [12642.412283] worker_thread+0x188/0x340 [12642.416033] ? rescuer_thread+0x310/0x310 [12642.420044] kthread+0xf5/0x130 [12642.423191] ? kthread_complete_and_exit+0x20/0x20 [12642.427973] ret_from_fork+0x31/0x40 [12642.431568] ? kthread_complete_and_exit+0x20/0x20 [12642.436354] ret_from_fork_asm+0x11/0x20 [12642.440278] [12669.676650] watchdog: BUG: soft lockup - CPU#0 stuck for 685s! [dbench:78683] [12669.683780] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12669.688650] watchdog: BUG: soft lockup - CPU#7 stuck for 9682s! [kworker/7:4:78808] [12669.739221] irq event stamp: 14405028 [12669.739222] hardirqs last enabled at (14405027): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12669.746865] Modules linked in: [12669.750523] hardirqs last disabled at (14405028): [] sysvec_apic_timer_interrupt+0xe/0x90 [12669.760682] ceph [12669.763731] softirqs last enabled at (14404960): [] __do_softirq+0x2be/0x371 [12669.773456] libceph dns_resolver [12669.775379] softirqs last disabled at (14404955): [] irq_exit_rcu+0xb0/0xd0 [12669.784063] fscache netfs [12669.787374] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12669.795885] veth nft_chain_nat [12669.798588] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12669.807533] xt_MASQUERADE nf_nat [12669.810672] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12669.818142] nf_conntrack [12669.821454] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12669.826931] nf_defrag_ipv6 nf_defrag_ipv4 [12669.829548] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12669.848286] nft_compat nf_tables [12669.852375] [12669.852376] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12669.857594] nfnetlink bridge [12669.860904] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12669.862394] stp llc [12669.869519] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12669.872482] xfs libcrc32c [12669.879607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12669.881790] sunrpc intel_rapl_msr [12669.888916] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12669.891620] intel_rapl_common [12669.898744] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12669.902141] sb_edac x86_pkg_temp_thermal [12669.909265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12669.912328] coretemp kvm_intel [12669.920403] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12669.924404] kvm irqbypass [12669.930144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12669.933280] i2c_i801 iTCO_wdt [12669.940404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12669.943109] iTCO_vendor_support i2c_smbus [12669.950233] Call Trace: [12669.950234] [12669.953284] crct10dif_pclmul joydev [12669.960408] ? watchdog_timer_fn+0x1b4/0x220 [12669.964499] crc32_pclmul ipmi_ssif [12669.966943] ? __hrtimer_run_queues+0x1d3/0x360 [12669.968954] ghash_clmulni_intel mei_me mei [12669.972528] ? hrtimer_interrupt+0x104/0x240 [12669.976787] lpc_ich [12669.980271] ? __local_bh_enable+0x50/0x60 [12669.984795] sha512_ssse3 acpi_ipmi [12669.988977] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12669.993239] mfd_core wmi [12669.995422] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12669.999512] ipmi_si ipmi_devintf [12670.002997] [12670.008213] ipmi_msghandler ioatdma [12670.010830] [12670.015788] acpi_pad acpi_power_meter [12670.019099] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12670.021196] ixgbe igb [12670.024772] ? queued_spin_lock_slowpath+0x18/0x2e0 [12670.026865] mdio crc32c_intel [12670.030611] do_raw_spin_lock+0xb5/0xc0 [12670.035914] nvme [12670.038271] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12670.043142] ptp [12670.046194] try_get_cap_refs+0x159/0x880 [ceph] [12670.050022] nvme_core i2c_algo_bit [12670.051951] __ceph_get_caps+0x4a1/0x760 [ceph] [12670.056730] pps_core dca [12670.058572] ? __file_remove_privs+0x89/0x110 [12670.063179] fuse [12670.066664] ? wake_bit_function+0x70/0x70 [12670.071187] [12670.071188] irq event stamp: 24729590 [12670.073807] ? ceph_write_iter+0x260/0x770 [ceph] [12670.078156] hardirqs last enabled at (24729589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12670.080080] ceph_write_iter+0x260/0x770 [ceph] [12670.084170] hardirqs last disabled at (24729590): [] sysvec_apic_timer_interrupt+0xe/0x90 [12670.085666] ? lock_is_held_type+0xaa/0x120 [12670.089329] softirqs last enabled at (24729588): [] __do_softirq+0x2be/0x371 [12670.094029] vfs_write+0x324/0x4f0 [12670.104199] softirqs last disabled at (24729583): [] irq_exit_rcu+0xb0/0xd0 [12670.108727] ksys_pwrite64+0x65/0xa0 [12670.118448] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12670.122625] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12670.131309] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12670.134708] do_syscall_64+0x38/0x80 [12670.143218] Workqueue: events delayed_work [ceph] [12670.146791] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12670.156165] [12670.156166] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [12670.161296] RIP: 0033:0x7f48f0eeb93b [12670.168767] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [12670.172339] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12670.177052] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [12670.182096] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12670.183586] [12670.183587] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12670.188197] ORIG_RAX: 0000000000000012 [12670.191769] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [12670.210507] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12670.229243] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [12670.234463] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12670.239679] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12670.241170] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12670.248294] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12670.252125] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12670.259248] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12670.266372] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12670.273497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12670.280627] [12670.287746] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12670.339383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12670.346507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12670.353632] Call Trace: [12670.356075] [12670.358086] ? watchdog_timer_fn+0x1b4/0x220 [12670.362351] ? __hrtimer_run_queues+0x1d3/0x360 [12670.366878] ? hrtimer_interrupt+0x104/0x240 [12670.371146] ? __local_bh_enable+0x50/0x60 [12670.375241] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12670.380464] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12670.385420] [12670.387519] [12670.389616] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12670.394923] ? do_raw_spin_lock+0x42/0xc0 [12670.398934] igrab+0x1d/0x50 [12670.401811] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12670.406960] delayed_work+0x1d8/0x280 [ceph] [12670.411243] ? process_scheduled_works+0x22a/0x510 [12670.416031] process_scheduled_works+0x2a4/0x510 [12670.420664] worker_thread+0x188/0x340 [12670.424414] ? rescuer_thread+0x310/0x310 [12670.428418] kthread+0xf5/0x130 [12670.431556] ? kthread_complete_and_exit+0x20/0x20 [12670.436339] ret_from_fork+0x31/0x40 [12670.439910] ? kthread_complete_and_exit+0x20/0x20 [12670.444694] ret_from_fork_asm+0x11/0x20 [12670.448617] [12697.677141] watchdog: BUG: soft lockup - CPU#0 stuck for 711s! [dbench:78683] [12697.684278] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12697.689141] watchdog: BUG: soft lockup - CPU#7 stuck for 9708s! [kworker/7:4:78808] [12697.739726] irq event stamp: 14462978 [12697.747370] Modules linked in: ceph libceph dns_resolver fscache netfs [12697.751028] hardirqs last enabled at (14462977): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12697.751033] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12697.767712] hardirqs last disabled at (14462978): [] sysvec_apic_timer_interrupt+0xe/0x90 [12697.775094] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [12697.784818] softirqs last enabled at (14462910): [] __do_softirq+0x2be/0x371 [12697.791596] xfs libcrc32c sunrpc intel_rapl_msr [12697.800281] softirqs last disabled at (14462905): [] irq_exit_rcu+0xb0/0xd0 [12697.800285] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [12697.813403] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12697.821826] i2c_i801 iTCO_wdt iTCO_vendor_support [12697.830771] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12697.830772] i2c_smbus [12697.830773] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12697.835555] crct10dif_pclmul joydev [12697.843026] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12697.845382] crc32_pclmul ipmi_ssif [12697.850861] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12697.854430] ghash_clmulni_intel mei_me [12697.873171] [12697.876652] mei lpc_ich [12697.881872] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12697.885702] sha512_ssse3 acpi_ipmi [12697.887193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12697.889722] mfd_core wmi [12697.896848] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12697.900331] ipmi_si ipmi_devintf [12697.907456] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12697.910073] ipmi_msghandler ioatdma [12697.917197] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12697.920506] acpi_pad acpi_power_meter [12697.927632] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12697.931203] ixgbe igb [12697.938327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12697.942071] mdio crc32c_intel [12697.950149] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12697.952504] nvme ptp [12697.958242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12697.961295] nvme_core i2c_algo_bit [12697.968417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12697.970687] pps_core dca [12697.977829] Call Trace: [12697.981314] fuse [12697.988439] [12697.991055] [12697.991056] irq event stamp: 24950670 [12697.993501] ? watchdog_timer_fn+0x1b4/0x220 [12697.995424] hardirqs last enabled at (24950669): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12697.997435] ? __hrtimer_run_queues+0x1d3/0x360 [12697.998924] hardirqs last disabled at (24950670): [] sysvec_apic_timer_interrupt+0xe/0x90 [12698.002584] ? hrtimer_interrupt+0x104/0x240 [12698.006846] softirqs last enabled at (24950668): [] __do_softirq+0x2be/0x371 [12698.017005] ? __local_bh_enable+0x50/0x60 [12698.021540] softirqs last disabled at (24950663): [] irq_exit_rcu+0xb0/0xd0 [12698.031263] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12698.035541] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12698.044227] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12698.048317] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12698.056828] [12698.062046] Workqueue: events delayed_work [ceph] [12698.071422] [12698.071424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12698.076381] [12698.083855] ? queued_spin_lock_slowpath+0x18/0x2e0 [12698.085949] RIP: 0010:rcu_is_watching+0x1f/0x40 [12698.090647] do_raw_spin_lock+0xb5/0xc0 [12698.092744] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [12698.098050] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12698.099542] RSP: 0018:ffffc90002117d40 EFLAGS: 00000202 [12698.104421] try_get_cap_refs+0x159/0x880 [ceph] [12698.108941] [12698.108941] RAX: 00000000007c1919 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12698.112778] __ceph_get_caps+0x4a1/0x760 [ceph] [12698.131511] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [12698.136300] ? __file_remove_privs+0x89/0x110 [12698.141512] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [12698.146122] ? wake_bit_function+0x70/0x70 [12698.147612] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [12698.154740] ? ceph_write_iter+0x260/0x770 [ceph] [12698.159262] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12698.166385] ceph_write_iter+0x260/0x770 [ceph] [12698.170737] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12698.177864] ? lock_is_held_type+0xaa/0x120 [12698.181950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12698.189078] vfs_write+0x324/0x4f0 [12698.193772] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12698.200901] ksys_pwrite64+0x65/0xa0 [12698.205420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12698.213500] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12698.217676] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12698.223414] do_syscall_64+0x38/0x80 [12698.226824] Call Trace: [12698.226825] [12698.233953] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12698.237541] ? watchdog_timer_fn+0x1b4/0x220 [12698.244664] RIP: 0033:0x7f48f0eeb93b [12698.249797] ? __hrtimer_run_queues+0x1d3/0x360 [12698.256919] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12698.260494] ? hrtimer_interrupt+0x104/0x240 [12698.262934] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12698.264944] ? __local_bh_enable+0x50/0x60 [12698.269987] ORIG_RAX: 0000000000000012 [12698.274257] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12698.277825] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12698.282348] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12698.301085] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12698.305350] [12698.310566] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12698.314657] [12698.314658] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12698.318488] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12698.323706] ? igrab+0x43/0x50 [12698.330829] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12698.335791] ? igrab+0x43/0x50 [12698.342918] [12698.345008] ? rcu_is_watching+0x1f/0x40 [12698.385997] lock_release+0x1d7/0x260 [12698.389663] _raw_spin_unlock+0x1b/0x40 [12698.393501] igrab+0x43/0x50 [12698.396378] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12698.401546] delayed_work+0x1d8/0x280 [ceph] [12698.405830] ? process_scheduled_works+0x22a/0x510 [12698.410615] process_scheduled_works+0x2a4/0x510 [12698.415233] worker_thread+0x188/0x340 [12698.418983] ? rescuer_thread+0x310/0x310 [12698.422993] kthread+0xf5/0x130 [12698.426131] ? kthread_complete_and_exit+0x20/0x20 [12698.430914] ret_from_fork+0x31/0x40 [12698.434496] ? kthread_complete_and_exit+0x20/0x20 [12698.439278] ret_from_fork_asm+0x11/0x20 [12698.443202] [12725.677632] watchdog: BUG: soft lockup - CPU#0 stuck for 737s! [dbench:78683] [12725.684763] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12725.689632] watchdog: BUG: soft lockup - CPU#7 stuck for 9734s! [kworker/7:4:78808] [12725.740204] irq event stamp: 14520874 [12725.740206] hardirqs last enabled at (14520873): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12725.747849] Modules linked in: [12725.751507] hardirqs last disabled at (14520874): [] sysvec_apic_timer_interrupt+0xe/0x90 [12725.761664] ceph [12725.764715] softirqs last enabled at (14520816): [] __do_softirq+0x2be/0x371 [12725.774439] libceph [12725.776361] softirqs last disabled at (14520811): [] irq_exit_rcu+0xb0/0xd0 [12725.785048] dns_resolver [12725.787230] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12725.795742] fscache netfs [12725.798359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12725.798360] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12725.807316] veth nft_chain_nat [12725.810034] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12725.817504] xt_MASQUERADE nf_nat [12725.822980] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12725.826118] nf_conntrack [12725.844856] [12725.848168] nf_defrag_ipv6 nf_defrag_ipv4 [12725.853386] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12725.856003] nft_compat [12725.857510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12725.861601] nf_tables nfnetlink [12725.868724] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12725.871169] bridge [12725.878310] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12725.881535] stp llc [12725.888658] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12725.890755] xfs libcrc32c [12725.897880] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12725.900065] sunrpc intel_rapl_msr [12725.907189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12725.909892] intel_rapl_common sb_edac [12725.917969] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12725.921367] x86_pkg_temp_thermal [12725.927103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12725.930848] coretemp kvm_intel [12725.937971] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12725.941282] kvm irqbypass [12725.948407] Call Trace: [12725.948408] [12725.951544] i2c_i801 iTCO_wdt [12725.958670] ? watchdog_timer_fn+0x1b4/0x220 [12725.961372] iTCO_vendor_support i2c_smbus [12725.963820] ? __hrtimer_run_queues+0x1d3/0x360 [12725.965828] crct10dif_pclmul joydev [12725.968881] ? hrtimer_interrupt+0x104/0x240 [12725.973142] crc32_pclmul ipmi_ssif [12725.977234] ? __local_bh_enable+0x50/0x60 [12725.981758] ghash_clmulni_intel mei_me [12725.985332] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12725.989593] mei lpc_ich [12725.993078] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12725.997167] sha512_ssse3 acpi_ipmi [12726.000999] [12726.006217] mfd_core wmi [12726.008747] [12726.008748] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12726.013705] ipmi_si [12726.017192] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12726.019284] ipmi_devintf ipmi_msghandler [12726.021903] do_raw_spin_lock+0xb5/0xc0 [12726.024001] ioatdma acpi_pad [12726.029322] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12726.031506] acpi_power_meter [12726.036380] try_get_cap_refs+0x159/0x880 [ceph] [12726.040380] ixgbe igb [12726.044217] __ceph_get_caps+0x4a1/0x760 [ceph] [12726.047173] mdio crc32c_intel [12726.051964] ? __file_remove_privs+0x89/0x110 [12726.054923] nvme ptp [12726.059534] ? wake_bit_function+0x70/0x70 [12726.061891] nvme_core i2c_algo_bit [12726.066418] ? ceph_write_iter+0x260/0x770 [ceph] [12726.069466] pps_core dca [12726.073818] ceph_write_iter+0x260/0x770 [ceph] [12726.076087] fuse [12726.080181] ? lock_is_held_type+0xaa/0x120 [12726.083663] irq event stamp: 25171678 [12726.088361] vfs_write+0x324/0x4f0 [12726.090976] hardirqs last enabled at (25171677): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12726.095504] ksys_pwrite64+0x65/0xa0 [12726.097425] hardirqs last disabled at (25171678): [] sysvec_apic_timer_interrupt+0xe/0x90 [12726.101603] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12726.105259] softirqs last enabled at (25171676): [] __do_softirq+0x2be/0x371 [12726.108659] do_syscall_64+0x38/0x80 [12726.118815] softirqs last disabled at (25171671): [] irq_exit_rcu+0xb0/0xd0 [12726.122387] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12726.132110] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12726.137241] RIP: 0033:0x7f48f0eeb93b [12726.145925] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12726.149496] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12726.158005] Workqueue: events delayed_work [ceph] [12726.163051] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12726.172429] [12726.172429] RIP: 0010:lock_release+0x5f/0x260 [12726.175999] ORIG_RAX: 0000000000000012 [12726.183469] Code: 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 <65> 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 [12726.202207] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12726.206903] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [12726.212123] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12726.213612] [12726.213613] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12726.217964] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12726.221794] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: 0000000000000001 [12726.240533] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12726.247656] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [12726.252874] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12726.259997] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [12726.261494] [12726.268612] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12726.320666] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12726.328744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12726.334480] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12726.341605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12726.348730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12726.355854] Call Trace: [12726.358296] [12726.360325] ? watchdog_timer_fn+0x1b4/0x220 [12726.364590] ? __hrtimer_run_queues+0x1d3/0x360 [12726.369119] ? hrtimer_interrupt+0x104/0x240 [12726.373386] ? __local_bh_enable+0x50/0x60 [12726.377479] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12726.382704] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12726.387661] [12726.389757] [12726.391856] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12726.397159] ? igrab+0x43/0x50 [12726.400213] ? igrab+0x43/0x50 [12726.403261] ? lock_release+0x5f/0x260 [12726.407006] ? lock_release+0x1d7/0x260 [12726.410838] _raw_spin_unlock+0x1b/0x40 [12726.414675] igrab+0x43/0x50 [12726.417554] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12726.422701] delayed_work+0x1d8/0x280 [ceph] [12726.426984] ? process_scheduled_works+0x22a/0x510 [12726.431774] process_scheduled_works+0x2a4/0x510 [12726.436391] worker_thread+0x188/0x340 [12726.440139] ? rescuer_thread+0x310/0x310 [12726.444142] kthread+0xf5/0x130 [12726.447280] ? kthread_complete_and_exit+0x20/0x20 [12726.452065] ret_from_fork+0x31/0x40 [12726.455642] ? kthread_complete_and_exit+0x20/0x20 [12726.460428] ret_from_fork_asm+0x11/0x20 [12726.464350] [12753.678124] watchdog: BUG: soft lockup - CPU#0 stuck for 763s! [dbench:78683] [12753.685260] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12753.690124] watchdog: BUG: soft lockup - CPU#7 stuck for 9760s! [kworker/7:4:78808] [12753.740710] irq event stamp: 14578870 [12753.748354] Modules linked in: ceph libceph dns_resolver fscache [12753.752011] hardirqs last enabled at (14578869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12753.758016] netfs veth nft_chain_nat xt_MASQUERADE [12753.768175] hardirqs last disabled at (14578870): [] sysvec_apic_timer_interrupt+0xe/0x90 [12753.773045] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [12753.782769] softirqs last enabled at (14578812): [] __do_softirq+0x2be/0x371 [12753.782773] nfnetlink [12753.790422] bridge [12753.799106] softirqs last disabled at (14578807): [] irq_exit_rcu+0xb0/0xd0 [12753.801462] stp llc [12753.803562] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12753.812072] xfs [12753.814257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12753.823200] libcrc32c sunrpc [12753.825037] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12753.832520] intel_rapl_msr intel_rapl_common [12753.835483] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12753.840958] sb_edac x86_pkg_temp_thermal [12753.845308] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12753.864047] coretemp [12753.868050] [12753.873269] kvm_intel kvm [12753.875539] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12753.877028] irqbypass i2c_i801 [12753.879732] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12753.879734] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12753.886857] iTCO_wdt [12753.889996] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12753.897120] iTCO_vendor_support i2c_smbus [12753.904243] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12753.906520] crct10dif_pclmul [12753.913646] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12753.917738] joydev crc32_pclmul [12753.924861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12753.927825] ipmi_ssif ghash_clmulni_intel [12753.935902] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12753.939128] mei_me mei [12753.944865] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12753.948956] lpc_ich sha512_ssse3 [12753.956079] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12753.958524] acpi_ipmi [12753.965648] Call Trace: [12753.965651] [12753.968958] mfd_core [12753.976084] ? watchdog_timer_fn+0x1b4/0x220 [12753.978441] wmi ipmi_si [12753.980884] ? __hrtimer_run_queues+0x1d3/0x360 [12753.982896] ipmi_devintf [12753.985170] ? hrtimer_interrupt+0x104/0x240 [12753.989429] ipmi_msghandler ioatdma [12753.991959] ? __local_bh_enable+0x50/0x60 [12753.996485] acpi_pad [12753.999103] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12754.003366] acpi_power_meter ixgbe [12754.006936] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12754.011026] igb [12754.013297] [12754.018523] mdio crc32c_intel [12754.022008] [12754.022010] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12754.026967] nvme [12754.028810] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12754.030900] ptp nvme_core [12754.033953] do_raw_spin_lock+0xb5/0xc0 [12754.036050] i2c_algo_bit [12754.041356] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12754.043277] pps_core dca [12754.048151] try_get_cap_refs+0x159/0x880 [ceph] [12754.050850] fuse [12754.054687] __ceph_get_caps+0x4a1/0x760 [ceph] [12754.057298] irq event stamp: 25393662 [12754.062087] ? __file_remove_privs+0x89/0x110 [12754.064701] hardirqs last enabled at (25393661): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12754.069311] ? wake_bit_function+0x70/0x70 [12754.071235] hardirqs last disabled at (25393662): [] sysvec_apic_timer_interrupt+0xe/0x90 [12754.075762] ? ceph_write_iter+0x260/0x770 [ceph] [12754.079416] softirqs last enabled at (25393660): [] __do_softirq+0x2be/0x371 [12754.083768] ceph_write_iter+0x260/0x770 [ceph] [12754.093925] softirqs last disabled at (25393655): [] irq_exit_rcu+0xb0/0xd0 [12754.098020] ? lock_is_held_type+0xaa/0x120 [12754.107739] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12754.112440] vfs_write+0x324/0x4f0 [12754.121122] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12754.125651] ksys_pwrite64+0x65/0xa0 [12754.134159] Workqueue: events delayed_work [ceph] [12754.138334] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12754.147711] [12754.151112] do_syscall_64+0x38/0x80 [12754.158580] RIP: 0010:lock_acquire+0xf0/0x2b0 [12754.162152] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12754.166848] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12754.171981] RIP: 0033:0x7f48f0eeb93b [12754.173469] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12754.177043] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12754.181390] [12754.181391] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12754.186435] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12754.205173] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12754.208743] ORIG_RAX: 0000000000000012 [12754.213962] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12754.232699] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12754.234190] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12754.241313] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12754.246531] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12754.253656] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12754.257486] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12754.264611] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12754.271734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12754.278859] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12754.285983] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12754.293111] [12754.300231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12754.344726] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12754.351850] Call Trace: [12754.354296] [12754.356306] ? watchdog_timer_fn+0x1b4/0x220 [12754.360570] ? __hrtimer_run_queues+0x1d3/0x360 [12754.365097] ? hrtimer_interrupt+0x104/0x240 [12754.369365] ? __local_bh_enable+0x50/0x60 [12754.373459] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12754.378683] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12754.383641] [12754.385739] [12754.387836] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12754.393145] ? lock_acquire+0xf0/0x2b0 [12754.396893] ? igrab+0x43/0x50 [12754.399944] ? lock_release+0xc5/0x260 [12754.403688] _raw_spin_lock+0x30/0x40 [12754.407346] ? igrab+0x1d/0x50 [12754.410396] igrab+0x1d/0x50 [12754.413275] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12754.418424] delayed_work+0x1d8/0x280 [ceph] [12754.422706] ? process_scheduled_works+0x22a/0x510 [12754.427493] process_scheduled_works+0x2a4/0x510 [12754.432110] worker_thread+0x188/0x340 [12754.435860] ? rescuer_thread+0x310/0x310 [12754.439861] kthread+0xf5/0x130 [12754.443001] ? kthread_complete_and_exit+0x20/0x20 [12754.447785] ret_from_fork+0x31/0x40 [12754.451383] ? kthread_complete_and_exit+0x20/0x20 [12754.456165] ret_from_fork_asm+0x11/0x20 [12754.460089] [12781.678616] watchdog: BUG: soft lockup - CPU#0 stuck for 789s! [dbench:78683] [12781.685746] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12781.690615] watchdog: BUG: soft lockup - CPU#7 stuck for 9786s! [kworker/7:4:78808] [12781.741187] irq event stamp: 14636878 [12781.748831] Modules linked in: ceph libceph [12781.752488] hardirqs last enabled at (14636877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12781.756666] dns_resolver fscache netfs [12781.766824] hardirqs last disabled at (14636878): [] sysvec_apic_timer_interrupt+0xe/0x90 [12781.766826] veth [12781.770653] softirqs last enabled at (14636822): [] __do_softirq+0x2be/0x371 [12781.780377] nft_chain_nat xt_MASQUERADE [12781.782303] softirqs last disabled at (14636817): [] irq_exit_rcu+0xb0/0xd0 [12781.790986] nf_nat nf_conntrack [12781.794921] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12781.803431] nf_defrag_ipv6 nf_defrag_ipv4 [12781.806656] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12781.815600] nft_compat nf_tables [12781.819692] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12781.827163] nfnetlink bridge [12781.830474] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12781.835950] stp llc [12781.838915] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12781.857652] xfs libcrc32c [12781.859836] [12781.859837] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12781.865055] sunrpc [12781.867758] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12781.869248] intel_rapl_msr intel_rapl_common [12781.876373] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12781.878469] sb_edac x86_pkg_temp_thermal [12781.885593] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12781.889944] coretemp kvm_intel [12781.897068] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12781.901074] kvm irqbypass [12781.908196] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12781.911334] i2c_i801 iTCO_wdt [12781.918458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12781.921163] iTCO_vendor_support i2c_smbus [12781.929240] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12781.932293] crct10dif_pclmul joydev [12781.938029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12781.942118] crc32_pclmul ipmi_ssif [12781.949242] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12781.952814] ghash_clmulni_intel mei_me [12781.959938] Call Trace: [12781.959940] [12781.963422] mei lpc_ich [12781.970548] ? watchdog_timer_fn+0x1b4/0x220 [12781.974377] sha512_ssse3 [12781.976821] ? __hrtimer_run_queues+0x1d3/0x360 [12781.978832] acpi_ipmi mfd_core [12781.981367] ? hrtimer_interrupt+0x104/0x240 [12781.985627] wmi ipmi_si [12781.988244] ? __local_bh_enable+0x50/0x60 [12781.992768] ipmi_devintf ipmi_msghandler [12781.995908] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12782.000168] ioatdma acpi_pad [12782.002700] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12782.006790] acpi_power_meter ixgbe [12782.010796] [12782.016014] igb mdio [12782.018977] [12782.018978] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12782.023933] crc32c_intel nvme [12782.027421] ? queued_spin_lock_slowpath+0x18/0x2e0 [12782.029515] ptp nvme_core [12782.031787] do_raw_spin_lock+0xb5/0xc0 [12782.033882] i2c_algo_bit [12782.039189] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12782.042238] pps_core [12782.047112] try_get_cap_refs+0x159/0x880 [ceph] [12782.049812] dca fuse [12782.053648] __ceph_get_caps+0x4a1/0x760 [ceph] [12782.056261] [12782.056262] irq event stamp: 25616392 [12782.061049] ? __file_remove_privs+0x89/0x110 [12782.063315] hardirqs last enabled at (25616391): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12782.067927] ? wake_bit_function+0x70/0x70 [12782.070197] hardirqs last disabled at (25616392): [] sysvec_apic_timer_interrupt+0xe/0x90 [12782.074724] ? ceph_write_iter+0x260/0x770 [ceph] [12782.076212] softirqs last enabled at (25616390): [] __do_softirq+0x2be/0x371 [12782.079871] ceph_write_iter+0x260/0x770 [ceph] [12782.084219] softirqs last disabled at (25616385): [] irq_exit_rcu+0xb0/0xd0 [12782.094382] ? lock_is_held_type+0xaa/0x120 [12782.098470] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12782.108195] vfs_write+0x324/0x4f0 [12782.112891] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12782.121578] ksys_pwrite64+0x65/0xa0 [12782.126098] Workqueue: events delayed_work [ceph] [12782.134609] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12782.138787] [12782.148165] do_syscall_64+0x38/0x80 [12782.151561] RIP: 0010:lock_acquire+0xf0/0x2b0 [12782.159033] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12782.162603] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12782.167308] RIP: 0033:0x7f48f0eeb93b [12782.172441] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12782.173930] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12782.177500] [12782.177501] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12782.181851] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12782.186896] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12782.205634] ORIG_RAX: 0000000000000012 [12782.209205] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12782.214421] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12782.233160] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12782.234649] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12782.241774] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12782.246993] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12782.254134] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12782.257963] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12782.265088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12782.272212] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12782.279337] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12782.286467] [12782.293585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12782.345205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12782.352330] Call Trace: [12782.354774] [12782.356785] ? watchdog_timer_fn+0x1b4/0x220 [12782.361048] ? __hrtimer_run_queues+0x1d3/0x360 [12782.365575] ? hrtimer_interrupt+0x104/0x240 [12782.369845] ? __local_bh_enable+0x50/0x60 [12782.373937] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12782.379162] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12782.384120] [12782.386216] [12782.388314] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12782.393622] ? lock_acquire+0xf0/0x2b0 [12782.397371] ? igrab+0x43/0x50 [12782.400421] ? lock_release+0xc5/0x260 [12782.404168] _raw_spin_lock+0x30/0x40 [12782.407832] ? igrab+0x1d/0x50 [12782.410884] igrab+0x1d/0x50 [12782.413760] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12782.418911] delayed_work+0x1d8/0x280 [ceph] [12782.423191] ? process_scheduled_works+0x22a/0x510 [12782.427980] process_scheduled_works+0x2a4/0x510 [12782.432598] worker_thread+0x188/0x340 [12782.436347] ? rescuer_thread+0x310/0x310 [12782.440349] kthread+0xf5/0x130 [12782.443487] ? kthread_complete_and_exit+0x20/0x20 [12782.448271] ret_from_fork+0x31/0x40 [12782.451850] ? kthread_complete_and_exit+0x20/0x20 [12782.456634] ret_from_fork_asm+0x11/0x20 [12782.460557] [12783.537015] rcu: INFO: rcu_preempt self-detected stall on CPU [12783.542772] rcu: 7-....: (827599 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/826462 fqs=387394 [12783.552936] rcu: (t=846911 jiffies g=1309609 q=92173 ncpus=8) [12783.558761] Sending NMI from CPU 7 to CPUs 0: [12783.563111] NMI backtrace for cpu 0 [12783.563113] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12783.563115] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12783.563115] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12783.563119] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12783.563120] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12783.563122] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12783.563123] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12783.563123] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12783.563124] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12783.563125] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12783.563126] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12783.563127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12783.563128] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12783.563129] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12783.563129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12783.563130] Call Trace: [12783.563131] [12783.563132] ? nmi_cpu_backtrace+0xad/0x120 [12783.563136] ? nmi_cpu_backtrace_handler+0x11/0x20 [12783.563137] ? nmi_handle+0xd4/0x260 [12783.563142] ? default_do_nmi+0x49/0xf0 [12783.563143] ? exc_nmi+0x11e/0x150 [12783.563145] ? end_repeat_nmi+0x16/0x67 [12783.563149] ? queued_spin_lock_slowpath+0x18/0x2e0 [12783.563151] ? queued_spin_lock_slowpath+0x18/0x2e0 [12783.563154] ? queued_spin_lock_slowpath+0x18/0x2e0 [12783.563156] [12783.563157] [12783.563157] do_raw_spin_lock+0xb5/0xc0 [12783.563160] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12783.563174] try_get_cap_refs+0x159/0x880 [ceph] [12783.563188] __ceph_get_caps+0x4a1/0x760 [ceph] [12783.563201] ? __file_remove_privs+0x89/0x110 [12783.563203] ? wake_bit_function+0x70/0x70 [12783.563205] ? ceph_write_iter+0x260/0x770 [ceph] [12783.563216] ceph_write_iter+0x260/0x770 [ceph] [12783.563227] ? lock_is_held_type+0xaa/0x120 [12783.563230] vfs_write+0x324/0x4f0 [12783.563234] ksys_pwrite64+0x65/0xa0 [12783.563236] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12783.563239] do_syscall_64+0x38/0x80 [12783.563241] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12783.563243] RIP: 0033:0x7f48f0eeb93b [12783.563244] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12783.563246] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12783.563247] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12783.563248] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12783.563248] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12783.563249] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12783.563250] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12783.563253] [12783.564109] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12783.868366] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12783.875835] Workqueue: events delayed_work [ceph] [12783.880551] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [12783.885160] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [12783.903898] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [12783.909116] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12783.916239] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [12783.923364] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [12783.930487] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [12783.937612] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12783.944738] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12783.952814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12783.958552] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12783.965675] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12783.972800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12783.979924] Call Trace: [12783.982369] [12783.984379] ? rcu_dump_cpu_stacks+0xd1/0x180 [12783.988739] ? rcu_sched_clock_irq+0x633/0x1390 [12783.993263] ? lock_release+0xc5/0x260 [12783.997006] ? lock_is_held_type+0xaa/0x120 [12784.001184] ? lock_is_held_type+0xaa/0x120 [12784.005364] ? tick_sched_do_timer+0xa0/0xa0 [12784.009635] ? rcu_read_lock_sched_held+0x56/0x90 [12784.014333] ? tick_sched_do_timer+0xa0/0xa0 [12784.018595] ? update_process_times+0x5f/0x90 [12784.022948] ? tick_sched_handle.isra.16+0x1e/0x50 [12784.027731] ? tick_sched_timer+0x6f/0x80 [12784.031736] ? __hrtimer_run_queues+0x1d3/0x360 [12784.036263] ? hrtimer_interrupt+0x104/0x240 [12784.040533] ? __local_bh_enable+0x50/0x60 [12784.044635] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12784.049856] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12784.054813] [12784.056912] [12784.059009] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12784.064317] ? do_raw_spin_lock+0x42/0xc0 [12784.068328] igrab+0x1d/0x50 [12784.071214] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12784.076363] delayed_work+0x1d8/0x280 [ceph] [12784.080642] ? process_scheduled_works+0x22a/0x510 [12784.085426] process_scheduled_works+0x2a4/0x510 [12784.090040] worker_thread+0x188/0x340 [12784.093790] ? rescuer_thread+0x310/0x310 [12784.097793] kthread+0xf5/0x130 [12784.100930] ? kthread_complete_and_exit+0x20/0x20 [12784.105715] ret_from_fork+0x31/0x40 [12784.109285] ? kthread_complete_and_exit+0x20/0x20 [12784.114070] ret_from_fork_asm+0x11/0x20 [12784.117994] [12809.679107] watchdog: BUG: soft lockup - CPU#0 stuck for 815s! [dbench:78683] [12809.686242] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12809.691107] watchdog: BUG: soft lockup - CPU#7 stuck for 9812s! [kworker/7:4:78808] [12809.741692] irq event stamp: 14694838 [12809.749335] Modules linked in: ceph libceph dns_resolver fscache netfs veth [12809.752994] hardirqs last enabled at (14694837): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12809.759954] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12809.770111] hardirqs last disabled at (14694838): [] sysvec_apic_timer_interrupt+0xe/0x90 [12809.770114] nf_defrag_ipv4 [12809.770115] softirqs last enabled at (14694810): [] __do_softirq+0x2be/0x371 [12809.777070] nft_compat nf_tables [12809.786794] softirqs last disabled at (14694805): [] irq_exit_rcu+0xb0/0xd0 [12809.789585] nfnetlink [12809.798270] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12809.801579] bridge stp [12809.810090] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12809.812446] llc xfs [12809.821392] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12809.823835] libcrc32c sunrpc [12809.831308] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12809.833503] intel_rapl_msr intel_rapl_common [12809.838976] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12809.841940] sb_edac x86_pkg_temp_thermal [12809.860678] [12809.865030] coretemp kvm_intel [12809.870247] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12809.870249] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12809.874252] kvm [12809.875742] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12809.878878] irqbypass i2c_i801 [12809.886004] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12809.893127] iTCO_wdt iTCO_vendor_support [12809.894964] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12809.902088] i2c_smbus crct10dif_pclmul [12809.905226] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12809.912351] joydev crc32_pclmul [12809.916355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12809.923478] ipmi_ssif ghash_clmulni_intel [12809.927327] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12809.935402] mei_me mei [12809.938629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12809.944367] lpc_ich [12809.948456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12809.955579] sha512_ssse3 [12809.958026] Call Trace: [12809.965149] acpi_ipmi [12809.967335] [12809.974456] mfd_core [12809.977077] ? watchdog_timer_fn+0x1b4/0x220 [12809.979518] wmi [12809.981878] ? __hrtimer_run_queues+0x1d3/0x360 [12809.983886] ipmi_si ipmi_devintf [12809.986160] ? hrtimer_interrupt+0x104/0x240 [12809.990420] ipmi_msghandler ioatdma [12809.992259] ? __local_bh_enable+0x50/0x60 [12809.996784] acpi_pad acpi_power_meter [12810.000097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12810.004356] ixgbe igb [12810.007927] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12810.012017] mdio crc32c_intel [12810.015762] [12810.020979] nvme ptp [12810.023339] [12810.023340] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12810.028296] nvme_core i2c_algo_bit [12810.031350] ? queued_spin_lock_slowpath+0x18/0x2e0 [12810.033443] pps_core dca [12810.035716] do_raw_spin_lock+0xb5/0xc0 [12810.037811] fuse [12810.043118] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12810.046598] irq event stamp: 25834184 [12810.051473] try_get_cap_refs+0x159/0x880 [ceph] [12810.054088] hardirqs last enabled at (25834183): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12810.057923] __ceph_get_caps+0x4a1/0x760 [ceph] [12810.059843] hardirqs last disabled at (25834184): [] sysvec_apic_timer_interrupt+0xe/0x90 [12810.064630] ? __file_remove_privs+0x89/0x110 [12810.068282] softirqs last enabled at (25834182): [] __do_softirq+0x2be/0x371 [12810.072895] ? wake_bit_function+0x70/0x70 [12810.083052] softirqs last disabled at (25834177): [] irq_exit_rcu+0xb0/0xd0 [12810.087578] ? ceph_write_iter+0x260/0x770 [ceph] [12810.097299] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12810.101652] ceph_write_iter+0x260/0x770 [ceph] [12810.110336] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12810.114431] ? lock_is_held_type+0xaa/0x120 [12810.122938] Workqueue: events delayed_work [ceph] [12810.127637] vfs_write+0x324/0x4f0 [12810.137012] [12810.141540] ksys_pwrite64+0x65/0xa0 [12810.149007] RIP: 0010:lock_release+0x78/0x260 [12810.153186] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12810.157882] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [12810.161280] do_syscall_64+0x38/0x80 [12810.162785] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [12810.166360] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12810.170707] [12810.170708] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [12810.175839] RIP: 0033:0x7f48f0eeb93b [12810.194575] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [12810.198147] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12810.203365] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [12810.208408] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12810.209899] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [12810.217025] ORIG_RAX: 0000000000000012 [12810.220594] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12810.227718] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12810.246457] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12810.253581] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12810.258798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12810.265921] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12810.269752] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12810.276878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12810.284000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12810.292079] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12810.299203] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12810.304946] [12810.312065] Call Trace: [12810.312067] [12810.354325] ? watchdog_timer_fn+0x1b4/0x220 [12810.358588] ? __hrtimer_run_queues+0x1d3/0x360 [12810.363115] ? hrtimer_interrupt+0x104/0x240 [12810.367376] ? __local_bh_enable+0x50/0x60 [12810.371469] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12810.376692] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12810.381652] [12810.383748] [12810.385846] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12810.391149] ? igrab+0x43/0x50 [12810.394203] ? lock_release+0x78/0x260 [12810.397958] _raw_spin_unlock+0x1b/0x40 [12810.401797] igrab+0x43/0x50 [12810.404681] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12810.409830] delayed_work+0x1d8/0x280 [ceph] [12810.414109] ? process_scheduled_works+0x22a/0x510 [12810.418900] process_scheduled_works+0x2a4/0x510 [12810.423518] worker_thread+0x188/0x340 [12810.427265] ? rescuer_thread+0x310/0x310 [12810.431268] kthread+0xf5/0x130 [12810.434407] ? kthread_complete_and_exit+0x20/0x20 [12810.439189] ret_from_fork+0x31/0x40 [12810.442771] ? kthread_complete_and_exit+0x20/0x20 [12810.447570] ret_from_fork_asm+0x11/0x20 [12810.451513] [12837.679599] watchdog: BUG: soft lockup - CPU#0 stuck for 842s! [dbench:78683] [12837.686728] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12837.691599] watchdog: BUG: soft lockup - CPU#7 stuck for 9839s! [kworker/7:4:78808] [12837.742169] irq event stamp: 14752838 [12837.742171] hardirqs last enabled at (14752837): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12837.749815] Modules linked in: [12837.753473] hardirqs last disabled at (14752838): [] sysvec_apic_timer_interrupt+0xe/0x90 [12837.763628] ceph [12837.766679] softirqs last enabled at (14752802): [] __do_softirq+0x2be/0x371 [12837.776404] libceph [12837.778329] softirqs last disabled at (14752797): [] irq_exit_rcu+0xb0/0xd0 [12837.787011] dns_resolver [12837.789197] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12837.797707] fscache netfs [12837.800323] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12837.809274] veth nft_chain_nat [12837.811994] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12837.819460] xt_MASQUERADE nf_nat [12837.822598] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12837.828075] nf_conntrack nf_defrag_ipv6 [12837.831386] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12837.850123] nf_defrag_ipv4 nft_compat [12837.854042] [12837.854043] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12837.859275] nf_tables nfnetlink [12837.863021] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12837.864512] bridge stp [12837.871635] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12837.874860] llc xfs [12837.881996] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12837.884435] libcrc32c sunrpc [12837.891561] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12837.893744] intel_rapl_msr intel_rapl_common [12837.900868] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12837.903832] sb_edac [12837.910956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12837.915306] x86_pkg_temp_thermal coretemp [12837.923383] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12837.925568] kvm_intel kvm [12837.931306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12837.935398] irqbypass i2c_i801 [12837.942520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12837.945225] iTCO_wdt [12837.952348] Call Trace: [12837.952349] [12837.955486] iTCO_vendor_support i2c_smbus [12837.962611] ? watchdog_timer_fn+0x1b4/0x220 [12837.964883] crct10dif_pclmul joydev [12837.967327] ? __hrtimer_run_queues+0x1d3/0x360 [12837.969335] crc32_pclmul ipmi_ssif [12837.973431] ? hrtimer_interrupt+0x104/0x240 [12837.977692] ghash_clmulni_intel mei_me [12837.981263] ? __local_bh_enable+0x50/0x60 [12837.985786] mei lpc_ich [12837.989288] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12837.993551] sha512_ssse3 acpi_ipmi [12837.997382] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12838.001473] mfd_core wmi [12838.004002] [12838.009221] ipmi_si ipmi_devintf [12838.012705] [12838.017663] ipmi_msghandler ioatdma [12838.020281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12838.022377] acpi_pad acpi_power_meter [12838.025692] ? queued_spin_lock_slowpath+0x16/0x2e0 [12838.027784] ixgbe igb [12838.031358] do_raw_spin_lock+0xb5/0xc0 [12838.036659] mdio crc32c_intel [12838.040405] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12838.045276] nvme ptp [12838.047635] try_get_cap_refs+0x159/0x880 [ceph] [12838.051462] nvme_core i2c_algo_bit [12838.054519] __ceph_get_caps+0x4a1/0x760 [ceph] [12838.059299] pps_core dca [12838.061571] ? __file_remove_privs+0x89/0x110 [12838.066180] fuse [12838.069665] ? wake_bit_function+0x70/0x70 [12838.074187] irq event stamp: 26056522 [12838.076807] ? ceph_write_iter+0x260/0x770 [ceph] [12838.081154] hardirqs last enabled at (26056521): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12838.083079] ceph_write_iter+0x260/0x770 [ceph] [12838.087171] hardirqs last disabled at (26056522): [] sysvec_apic_timer_interrupt+0xe/0x90 [12838.090831] ? lock_is_held_type+0xaa/0x120 [12838.095526] softirqs last enabled at (26056520): [] __do_softirq+0x2be/0x371 [12838.105686] vfs_write+0x324/0x4f0 [12838.110207] softirqs last disabled at (26056515): [] irq_exit_rcu+0xb0/0xd0 [12838.119935] ksys_pwrite64+0x65/0xa0 [12838.124108] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12838.132792] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12838.136191] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12838.136192] Workqueue: events delayed_work [ceph] [12838.144701] do_syscall_64+0x38/0x80 [12838.148280] [12838.148281] RIP: 0010:lock_acquire+0xf0/0x2b0 [12838.157658] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12838.162789] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12838.170260] RIP: 0033:0x7f48f0eeb93b [12838.174956] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12838.178529] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12838.180017] [12838.180018] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12838.184370] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12838.189413] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12838.208152] ORIG_RAX: 0000000000000012 [12838.211720] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12838.216939] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12838.235677] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12838.237168] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12838.244290] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12838.249508] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12838.256632] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12838.260464] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12838.267589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12838.274712] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12838.281835] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12838.288967] [12838.296085] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12838.347730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12838.354855] Call Trace: [12838.357299] [12838.359310] ? watchdog_timer_fn+0x1b4/0x220 [12838.363573] ? __hrtimer_run_queues+0x1d3/0x360 [12838.368102] ? hrtimer_interrupt+0x104/0x240 [12838.372369] ? __local_bh_enable+0x50/0x60 [12838.376463] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12838.381688] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12838.386646] [12838.388740] [12838.390840] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12838.396149] ? lock_acquire+0xf0/0x2b0 [12838.399898] ? igrab+0x43/0x50 [12838.402947] ? lock_release+0xc5/0x260 [12838.406694] _raw_spin_lock+0x30/0x40 [12838.410358] ? igrab+0x1d/0x50 [12838.413408] igrab+0x1d/0x50 [12838.416302] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12838.421453] delayed_work+0x1d8/0x280 [ceph] [12838.425735] ? process_scheduled_works+0x22a/0x510 [12838.430525] process_scheduled_works+0x2a4/0x510 [12838.435141] worker_thread+0x188/0x340 [12838.438888] ? rescuer_thread+0x310/0x310 [12838.442893] kthread+0xf5/0x130 [12838.446030] ? kthread_complete_and_exit+0x20/0x20 [12838.450815] ret_from_fork+0x31/0x40 [12838.454392] ? kthread_complete_and_exit+0x20/0x20 [12838.459177] ret_from_fork_asm+0x11/0x20 [12838.463100] [12865.680091] watchdog: BUG: soft lockup - CPU#0 stuck for 868s! [dbench:78683] [12865.687226] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12865.692091] watchdog: BUG: soft lockup - CPU#7 stuck for 9865s! [kworker/7:4:78808] [12865.742676] irq event stamp: 14810788 [12865.750320] Modules linked in: ceph libceph dns_resolver fscache netfs [12865.753976] hardirqs last enabled at (14810787): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12865.760502] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [12865.770660] hardirqs last disabled at (14810788): [] sysvec_apic_timer_interrupt+0xe/0x90 [12865.776743] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [12865.786486] softirqs last enabled at (14810740): [] __do_softirq+0x2be/0x371 [12865.793869] stp llc xfs libcrc32c sunrpc [12865.802553] softirqs last disabled at (14810735): [] irq_exit_rcu+0xb0/0xd0 [12865.802557] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [12865.815071] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12865.815072] kvm [12865.815073] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12865.823580] irqbypass i2c_i801 [12865.832523] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12865.834362] iTCO_wdt iTCO_vendor_support [12865.841833] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12865.844968] i2c_smbus crct10dif_pclmul [12865.850446] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12865.854451] joydev crc32_pclmul [12865.873189] [12865.877019] ipmi_ssif ghash_clmulni_intel [12865.882237] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12865.885461] mei_me mei [12865.886967] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12865.891059] lpc_ich [12865.898184] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12865.900628] sha512_ssse3 acpi_ipmi [12865.907767] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12865.909953] mfd_core wmi [12865.917078] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12865.920560] ipmi_si ipmi_devintf [12865.927686] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12865.930302] ipmi_msghandler [12865.937428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12865.940738] ioatdma acpi_pad [12865.948814] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12865.951692] acpi_power_meter ixgbe [12865.957430] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12865.960393] igb mdio [12865.967517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12865.971001] crc32c_intel nvme [12865.978126] Call Trace: [12865.978129] [12865.980396] ptp nvme_core [12865.987525] ? watchdog_timer_fn+0x1b4/0x220 [12865.990571] i2c_algo_bit pps_core [12865.993017] ? __hrtimer_run_queues+0x1d3/0x360 [12865.995026] dca fuse [12865.997735] ? hrtimer_interrupt+0x104/0x240 [12866.001993] [12866.001994] irq event stamp: 26277742 [12866.005392] ? __local_bh_enable+0x50/0x60 [12866.009916] hardirqs last enabled at (26277741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12866.012190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12866.016451] hardirqs last disabled at (26277742): [] sysvec_apic_timer_interrupt+0xe/0x90 [12866.017942] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12866.021598] softirqs last enabled at (26277740): [] __do_softirq+0x2be/0x371 [12866.025691] [12866.035848] softirqs last disabled at (26277735): [] irq_exit_rcu+0xb0/0xd0 [12866.041064] [12866.050789] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12866.055746] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12866.064448] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12866.066548] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12866.075057] Workqueue: events delayed_work [ceph] [12866.077156] do_raw_spin_lock+0xb5/0xc0 [12866.086531] [12866.091835] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12866.099305] RIP: 0010:lock_acquire+0x32/0x2b0 [12866.104181] try_get_cap_refs+0x159/0x880 [ceph] [12866.108875] Code: ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 <31> c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd [12866.112711] __ceph_get_caps+0x4a1/0x760 [ceph] [12866.114196] RSP: 0018:ffffc90002117d18 EFLAGS: 00000296 [12866.118982] ? __file_remove_privs+0x89/0x110 [12866.123329] [12866.123330] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 0000000000000000 [12866.127942] ? wake_bit_function+0x70/0x70 [12866.146679] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [12866.151205] ? ceph_write_iter+0x260/0x770 [ceph] [12866.156421] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [12866.160782] ceph_write_iter+0x260/0x770 [ceph] [12866.162270] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [12866.169398] ? lock_is_held_type+0xaa/0x120 [12866.173493] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12866.180620] vfs_write+0x324/0x4f0 [12866.185314] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12866.192444] ksys_pwrite64+0x65/0xa0 [12866.196963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12866.204089] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12866.208264] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12866.215391] do_syscall_64+0x38/0x80 [12866.218787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12866.226867] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12866.230435] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12866.236174] RIP: 0033:0x7f48f0eeb93b [12866.241303] Call Trace: [12866.241305] [12866.248427] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12866.251998] ? watchdog_timer_fn+0x1b4/0x220 [12866.259122] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12866.264166] ? __hrtimer_run_queues+0x1d3/0x360 [12866.271290] ORIG_RAX: 0000000000000012 [12866.274863] ? hrtimer_interrupt+0x104/0x240 [12866.277304] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12866.279315] ? __local_bh_enable+0x50/0x60 [12866.298053] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12866.302321] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12866.307533] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12866.312060] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12866.315889] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12866.320154] [12866.327277] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12866.331369] [12866.331370] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12866.338500] [12866.343714] ? lock_acquire+0x32/0x2b0 [12866.385472] ? igrab+0x43/0x50 [12866.388544] ? lock_release+0xc5/0x260 [12866.392289] _raw_spin_lock+0x30/0x40 [12866.395945] ? igrab+0x1d/0x50 [12866.398996] igrab+0x1d/0x50 [12866.401875] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12866.407026] delayed_work+0x1d8/0x280 [ceph] [12866.411306] ? process_scheduled_works+0x22a/0x510 [12866.416094] process_scheduled_works+0x2a4/0x510 [12866.420714] worker_thread+0x188/0x340 [12866.424461] ? rescuer_thread+0x310/0x310 [12866.428463] kthread+0xf5/0x130 [12866.431601] ? kthread_complete_and_exit+0x20/0x20 [12866.436385] ret_from_fork+0x31/0x40 [12866.439965] ? kthread_complete_and_exit+0x20/0x20 [12866.444750] ret_from_fork_asm+0x11/0x20 [12866.448690] [12893.680582] watchdog: BUG: soft lockup - CPU#0 stuck for 894s! [dbench:78683] [12893.687711] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12893.692582] watchdog: BUG: soft lockup - CPU#7 stuck for 9891s! [kworker/7:4:78808] [12893.743152] irq event stamp: 14868724 [12893.750796] Modules linked in: ceph libceph [12893.754454] hardirqs last enabled at (14868723): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12893.758632] dns_resolver fscache netfs [12893.768788] hardirqs last disabled at (14868724): [] sysvec_apic_timer_interrupt+0xe/0x90 [12893.772618] veth nft_chain_nat xt_MASQUERADE nf_nat [12893.782342] softirqs last enabled at (14868676): [] __do_softirq+0x2be/0x371 [12893.787301] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [12893.795985] softirqs last disabled at (14868671): [] irq_exit_rcu+0xb0/0xd0 [12893.802157] nf_tables nfnetlink bridge [12893.810666] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12893.810667] stp [12893.810668] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12893.814497] llc xfs [12893.823441] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [12893.825279] libcrc32c sunrpc [12893.832749] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [12893.834935] intel_rapl_msr intel_rapl_common [12893.840412] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12893.843376] sb_edac [12893.862113] [12893.862114] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12893.866464] x86_pkg_temp_thermal coretemp [12893.871682] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12893.873866] kvm_intel kvm [12893.875357] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12893.882481] irqbypass i2c_i801 [12893.886571] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12893.893695] iTCO_wdt iTCO_vendor_support [12893.896398] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12893.903523] i2c_smbus crct10dif_pclmul [12893.906660] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12893.913784] joydev crc32_pclmul [12893.917788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12893.924913] ipmi_ssif ghash_clmulni_intel [12893.928744] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12893.936821] mei_me mei [12893.940045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12893.945784] lpc_ich sha512_ssse3 [12893.949874] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12893.956998] acpi_ipmi mfd_core [12893.959442] Call Trace: [12893.959444] [12893.966566] wmi ipmi_si [12893.969878] ? watchdog_timer_fn+0x1b4/0x220 [12893.977000] ipmi_devintf ipmi_msghandler [12893.980139] ? __hrtimer_run_queues+0x1d3/0x360 [12893.982581] ioatdma acpi_pad [12893.984597] ? hrtimer_interrupt+0x104/0x240 [12893.987124] acpi_power_meter [12893.991388] ? __local_bh_enable+0x50/0x60 [12893.995392] ixgbe igb [12893.999917] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12894.002881] mdio crc32c_intel [12894.007144] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12894.010109] nvme ptp [12894.014200] [12894.016557] nvme_core i2c_algo_bit [12894.021774] [12894.021776] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12894.024823] pps_core dca [12894.029785] ? queued_spin_lock_slowpath+0x18/0x2e0 [12894.032052] fuse [12894.034151] do_raw_spin_lock+0xb5/0xc0 [12894.037634] irq event stamp: 26500002 [12894.039733] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12894.045035] hardirqs last enabled at (26500001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12894.047655] try_get_cap_refs+0x159/0x880 [ceph] [12894.052524] hardirqs last disabled at (26500002): [] sysvec_apic_timer_interrupt+0xe/0x90 [12894.054452] __ceph_get_caps+0x4a1/0x760 [ceph] [12894.058277] softirqs last enabled at (26500000): [] __do_softirq+0x2be/0x371 [12894.061939] ? __file_remove_privs+0x89/0x110 [12894.066719] softirqs last disabled at (26499995): [] irq_exit_rcu+0xb0/0xd0 [12894.076877] ? wake_bit_function+0x70/0x70 [12894.081488] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12894.091214] ? ceph_write_iter+0x260/0x770 [ceph] [12894.095737] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12894.095738] Workqueue: events delayed_work [ceph] [12894.104420] ceph_write_iter+0x260/0x770 [ceph] [12894.108772] [12894.108773] RIP: 0010:lock_acquire+0xf0/0x2b0 [12894.117287] ? lock_is_held_type+0xaa/0x120 [12894.121374] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12894.130752] vfs_write+0x324/0x4f0 [12894.135447] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12894.142923] ksys_pwrite64+0x65/0xa0 [12894.147616] [12894.147617] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12894.152141] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12894.153629] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12894.157988] do_syscall_64+0x38/0x80 [12894.162168] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12894.180908] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12894.184303] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12894.189520] RIP: 0033:0x7f48f0eeb93b [12894.193091] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12894.194583] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12894.201706] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12894.206836] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12894.213961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12894.217550] ORIG_RAX: 0000000000000012 [12894.224673] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12894.229717] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12894.236842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12894.240412] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12894.247535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12894.266272] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12894.274350] Call Trace: [12894.274352] [12894.279569] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12894.285306] ? watchdog_timer_fn+0x1b4/0x220 [12894.289137] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12894.296276] ? __hrtimer_run_queues+0x1d3/0x360 [12894.303408] [12894.310530] ? hrtimer_interrupt+0x104/0x240 [12894.365837] ? __local_bh_enable+0x50/0x60 [12894.369933] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12894.375154] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12894.380113] [12894.382211] [12894.384307] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12894.389614] ? lock_acquire+0xf0/0x2b0 [12894.393365] ? igrab+0x43/0x50 [12894.396417] ? lock_release+0xc5/0x260 [12894.400162] _raw_spin_lock+0x30/0x40 [12894.403826] ? igrab+0x1d/0x50 [12894.406875] igrab+0x1d/0x50 [12894.409754] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12894.414904] delayed_work+0x1d8/0x280 [ceph] [12894.419184] ? process_scheduled_works+0x22a/0x510 [12894.423988] process_scheduled_works+0x2a4/0x510 [12894.428607] worker_thread+0x188/0x340 [12894.432358] ? rescuer_thread+0x310/0x310 [12894.436359] kthread+0xf5/0x130 [12894.439498] ? kthread_complete_and_exit+0x20/0x20 [12894.444281] ret_from_fork+0x31/0x40 [12894.447853] ? kthread_complete_and_exit+0x20/0x20 [12894.452638] ret_from_fork_asm+0x11/0x20 [12894.456559] [12921.681076] watchdog: BUG: soft lockup - CPU#0 stuck for 920s! [dbench:78683] [12921.688208] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12921.693074] watchdog: BUG: soft lockup - CPU#7 stuck for 9917s! [kworker/7:4:78808] [12921.743659] irq event stamp: 14926638 [12921.751301] Modules linked in: ceph libceph dns_resolver fscache netfs [12921.754957] hardirqs last enabled at (14926637): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12921.754962] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [12921.771641] hardirqs last disabled at (14926638): [] sysvec_apic_timer_interrupt+0xe/0x90 [12921.781280] nf_tables nfnetlink bridge stp llc xfs [12921.791004] softirqs last enabled at (14926602): [] __do_softirq+0x2be/0x371 [12921.795883] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [12921.804567] softirqs last disabled at (14926597): [] irq_exit_rcu+0xb0/0xd0 [12921.804570] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [12921.821417] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12921.821418] iTCO_vendor_support [12921.821419] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12921.827326] i2c_smbus [12921.836271] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [12921.839495] crct10dif_pclmul joydev [12921.846965] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [12921.849323] crc32_pclmul ipmi_ssif [12921.854799] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12921.858370] ghash_clmulni_intel mei_me [12921.877110] [12921.880592] mei lpc_ich [12921.885809] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12921.889640] sha512_ssse3 acpi_ipmi [12921.891132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12921.893662] mfd_core wmi [12921.900785] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12921.904271] ipmi_si ipmi_devintf [12921.911395] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12921.914011] ipmi_msghandler ioatdma [12921.921135] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12921.924448] acpi_pad acpi_power_meter [12921.931587] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12921.935159] ixgbe igb [12921.942283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12921.946026] mdio crc32c_intel [12921.954103] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12921.956472] nvme [12921.962208] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12921.965258] ptp nvme_core [12921.972382] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12921.974307] i2c_algo_bit pps_core [12921.981430] Call Trace: [12921.981431] [12921.984136] dca fuse [12921.991263] ? watchdog_timer_fn+0x1b4/0x220 [12921.994657] [12921.994658] irq event stamp: 26721408 [12921.997102] ? __hrtimer_run_queues+0x1d3/0x360 [12921.999110] hardirqs last enabled at (26721407): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12922.001385] ? hrtimer_interrupt+0x104/0x240 [12922.005646] hardirqs last disabled at (26721408): [] sysvec_apic_timer_interrupt+0xe/0x90 [12922.007136] ? __local_bh_enable+0x50/0x60 [12922.010794] softirqs last enabled at (26721406): [] __do_softirq+0x2be/0x371 [12922.015322] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12922.025475] softirqs last disabled at (26721401): [] irq_exit_rcu+0xb0/0xd0 [12922.029755] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12922.039482] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12922.043573] [12922.052257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12922.052259] Workqueue: events delayed_work [ceph] [12922.057474] [12922.065985] [12922.065986] RIP: 0010:lock_acquire+0xf0/0x2b0 [12922.070942] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12922.080319] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12922.082421] ? queued_spin_lock_slowpath+0x16/0x2e0 [12922.089886] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12922.094588] do_raw_spin_lock+0xb5/0xc0 [12922.096682] [12922.096683] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12922.098176] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12922.102523] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12922.107833] try_get_cap_refs+0x159/0x880 [ceph] [12922.126565] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12922.131443] __ceph_get_caps+0x4a1/0x760 [ceph] [12922.136653] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12922.140491] ? __file_remove_privs+0x89/0x110 [12922.141974] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12922.149100] ? wake_bit_function+0x70/0x70 [12922.153883] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12922.161010] ? ceph_write_iter+0x260/0x770 [ceph] [12922.165618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12922.172744] ceph_write_iter+0x260/0x770 [ceph] [12922.177266] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12922.184394] ? lock_is_held_type+0xaa/0x120 [12922.188740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12922.195886] vfs_write+0x324/0x4f0 [12922.199972] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12922.208055] ksys_pwrite64+0x65/0xa0 [12922.212748] Call Trace: [12922.212766] [12922.218503] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12922.223028] ? watchdog_timer_fn+0x1b4/0x220 [12922.230152] do_syscall_64+0x38/0x80 [12922.234330] ? __hrtimer_run_queues+0x1d3/0x360 [12922.241455] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12922.244870] ? hrtimer_interrupt+0x104/0x240 [12922.251993] RIP: 0033:0x7f48f0eeb93b [12922.255563] ? __local_bh_enable+0x50/0x60 [12922.258006] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12922.260021] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12922.265148] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12922.269414] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12922.272982] ORIG_RAX: 0000000000000012 [12922.277507] [12922.282552] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12922.286814] [12922.290386] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12922.294478] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12922.313214] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12922.318437] ? lock_acquire+0xf0/0x2b0 [12922.323650] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12922.328608] ? igrab+0x43/0x50 [12922.332438] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12922.334535] ? lock_release+0xc5/0x260 [12922.341665] [12922.343770] _raw_spin_lock+0x30/0x40 [12922.393946] ? igrab+0x1d/0x50 [12922.396998] igrab+0x1d/0x50 [12922.399876] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12922.405026] delayed_work+0x1d8/0x280 [ceph] [12922.409307] ? process_scheduled_works+0x22a/0x510 [12922.414097] process_scheduled_works+0x2a4/0x510 [12922.418713] worker_thread+0x188/0x340 [12922.422478] ? rescuer_thread+0x310/0x310 [12922.426483] kthread+0xf5/0x130 [12922.429629] ? kthread_complete_and_exit+0x20/0x20 [12922.434412] ret_from_fork+0x31/0x40 [12922.437991] ? kthread_complete_and_exit+0x20/0x20 [12922.442776] ret_from_fork_asm+0x11/0x20 [12922.446708] [12949.681566] watchdog: BUG: soft lockup - CPU#0 stuck for 946s! [dbench:78683] [12949.688694] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12949.693566] watchdog: BUG: soft lockup - CPU#7 stuck for 9943s! [kworker/7:4:78808] [12949.744134] irq event stamp: 14984570 [12949.744135] hardirqs last enabled at (14984569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12949.751779] Modules linked in: [12949.755436] hardirqs last disabled at (14984570): [] sysvec_apic_timer_interrupt+0xe/0x90 [12949.765594] ceph [12949.768645] softirqs last enabled at (14984522): [] __do_softirq+0x2be/0x371 [12949.778369] libceph [12949.780291] softirqs last disabled at (14984517): [] irq_exit_rcu+0xb0/0xd0 [12949.788978] dns_resolver [12949.791160] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12949.799670] fscache netfs [12949.802289] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12949.802290] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12949.811241] veth nft_chain_nat [12949.813959] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12949.821435] xt_MASQUERADE nf_nat [12949.826911] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12949.830049] nf_conntrack nf_defrag_ipv6 [12949.848786] [12949.848787] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12949.852097] nf_defrag_ipv4 nft_compat [12949.857314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12949.861243] nf_tables [12949.862732] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12949.869855] nfnetlink bridge [12949.873598] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12949.880723] stp llc [12949.883080] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12949.890204] xfs libcrc32c [12949.893169] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12949.900294] sunrpc intel_rapl_msr [12949.902477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12949.909602] intel_rapl_common [12949.912307] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12949.920383] sb_edac x86_pkg_temp_thermal [12949.923781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12949.929517] coretemp [12949.932568] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12949.939693] kvm_intel [12949.943696] Call Trace: [12949.943698] [12949.950822] kvm irqbypass [12949.953091] ? watchdog_timer_fn+0x1b4/0x220 [12949.960216] i2c_i801 [12949.962573] ? __hrtimer_run_queues+0x1d3/0x360 [12949.965017] iTCO_wdt [12949.967032] ? hrtimer_interrupt+0x104/0x240 [12949.969731] iTCO_vendor_support i2c_smbus [12949.973997] ? __local_bh_enable+0x50/0x60 [12949.976268] crct10dif_pclmul joydev [12949.980794] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12949.983061] crc32_pclmul [12949.987327] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12949.991417] ipmi_ssif ghash_clmulni_intel [12949.995507] [12949.999077] mei_me mei [12950.004295] [12950.006914] lpc_ich [12950.011871] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12950.015963] sha512_ssse3 [12950.018062] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12950.020501] acpi_ipmi mfd_core [12950.022601] do_raw_spin_lock+0xb5/0xc0 [12950.024784] wmi [12950.030089] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12950.032706] ipmi_si ipmi_devintf [12950.037580] try_get_cap_refs+0x159/0x880 [ceph] [12950.040712] ipmi_msghandler ioatdma [12950.044550] __ceph_get_caps+0x4a1/0x760 [ceph] [12950.046380] acpi_pad acpi_power_meter [12950.051170] ? __file_remove_privs+0x89/0x110 [12950.054475] ixgbe igb [12950.059087] ? wake_bit_function+0x70/0x70 [12950.062657] mdio crc32c_intel [12950.067184] ? ceph_write_iter+0x260/0x770 [ceph] [12950.070925] nvme ptp [12950.075277] ceph_write_iter+0x260/0x770 [ceph] [12950.077633] nvme_core i2c_algo_bit [12950.081728] ? lock_is_held_type+0xaa/0x120 [12950.084781] pps_core dca [12950.089484] vfs_write+0x324/0x4f0 [12950.091753] fuse [12950.096281] ksys_pwrite64+0x65/0xa0 [12950.099760] irq event stamp: 26943508 [12950.103939] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12950.106554] hardirqs last enabled at (26943507): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12950.109969] do_syscall_64+0x38/0x80 [12950.111895] hardirqs last disabled at (26943508): [] sysvec_apic_timer_interrupt+0xe/0x90 [12950.115466] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12950.119122] softirqs last enabled at (26943506): [] __do_softirq+0x2be/0x371 [12950.124253] RIP: 0033:0x7f48f0eeb93b [12950.134409] softirqs last disabled at (26943501): [] irq_exit_rcu+0xb0/0xd0 [12950.137983] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12950.147705] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12950.152750] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12950.161434] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12950.165005] ORIG_RAX: 0000000000000012 [12950.173514] Workqueue: events delayed_work [ceph] [12950.192252] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12950.201631] [12950.206847] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12950.214319] RIP: 0010:lock_acquire+0xf0/0x2b0 [12950.218150] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12950.222846] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [12950.229972] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12950.231462] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [12950.238586] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12950.242935] [12950.242936] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [12950.250067] [12950.268797] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [12950.306188] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [12950.313311] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [12950.320435] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [12950.327560] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12950.335637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12950.341374] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12950.348498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12950.355622] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12950.362746] Call Trace: [12950.365190] [12950.367203] ? watchdog_timer_fn+0x1b4/0x220 [12950.371466] ? __hrtimer_run_queues+0x1d3/0x360 [12950.375992] ? hrtimer_interrupt+0x104/0x240 [12950.380253] ? __local_bh_enable+0x50/0x60 [12950.384347] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12950.389571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12950.394529] [12950.396625] [12950.398724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12950.404031] ? lock_acquire+0xf0/0x2b0 [12950.407781] ? igrab+0x43/0x50 [12950.410830] ? lock_release+0xc5/0x260 [12950.414576] _raw_spin_lock+0x30/0x40 [12950.418234] ? igrab+0x1d/0x50 [12950.421299] igrab+0x1d/0x50 [12950.424177] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12950.429329] delayed_work+0x1d8/0x280 [ceph] [12950.433609] ? process_scheduled_works+0x22a/0x510 [12950.438399] process_scheduled_works+0x2a4/0x510 [12950.443016] worker_thread+0x188/0x340 [12950.446764] ? rescuer_thread+0x310/0x310 [12950.450768] kthread+0xf5/0x130 [12950.453904] ? kthread_complete_and_exit+0x20/0x20 [12950.458690] ret_from_fork+0x31/0x40 [12950.462268] ? kthread_complete_and_exit+0x20/0x20 [12950.467052] ret_from_fork_asm+0x11/0x20 [12950.470979] [12977.682058] watchdog: BUG: soft lockup - CPU#0 stuck for 972s! [dbench:78683] [12977.689190] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [12977.694058] watchdog: BUG: soft lockup - CPU#7 stuck for 9969s! [kworker/7:4:78808] [12977.744638] irq event stamp: 15042680 [12977.752282] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [12977.755939] hardirqs last enabled at (15042679): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12977.764112] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [12977.774269] hardirqs last disabled at (15042680): [] sysvec_apic_timer_interrupt+0xe/0x90 [12977.780009] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [12977.789732] softirqs last enabled at (15042632): [] __do_softirq+0x2be/0x371 [12977.796518] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [12977.805203] softirqs last disabled at (15042627): [] irq_exit_rcu+0xb0/0xd0 [12977.813896] coretemp kvm_intel kvm irqbypass i2c_i801 [12977.822407] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12977.827537] iTCO_wdt iTCO_vendor_support i2c_smbus [12977.836481] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12977.836482] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12977.841351] crct10dif_pclmul joydev [12977.848824] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12977.854299] crc32_pclmul ipmi_ssif [12977.857871] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12977.876609] ghash_clmulni_intel mei_me [12977.880093] [12977.885309] mei lpc_ich [12977.889142] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12977.890631] sha512_ssse3 acpi_ipmi [12977.893161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12977.900287] mfd_core wmi [12977.903769] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12977.910894] ipmi_si ipmi_devintf [12977.913512] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12977.920635] ipmi_msghandler ioatdma [12977.923948] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12977.931070] acpi_pad acpi_power_meter [12977.934643] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12977.941766] ixgbe [12977.945508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12977.953587] igb mdio [12977.955599] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12977.961334] crc32c_intel nvme [12977.963606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12977.970735] ptp nvme_core [12977.973789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12977.980914] i2c_algo_bit pps_core [12977.983617] Call Trace: [12977.990741] dca fuse [12977.994140] [12977.996582] [12977.996583] irq event stamp: 27166428 [12977.998856] ? watchdog_timer_fn+0x1b4/0x220 [12978.000864] hardirqs last enabled at (27166427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12978.002356] ? __hrtimer_run_queues+0x1d3/0x360 [12978.006013] hardirqs last disabled at (27166428): [] sysvec_apic_timer_interrupt+0xe/0x90 [12978.010279] ? hrtimer_interrupt+0x104/0x240 [12978.020452] softirqs last enabled at (27166426): [] __do_softirq+0x2be/0x371 [12978.024975] ? __local_bh_enable+0x50/0x60 [12978.034701] softirqs last disabled at (27166421): [] irq_exit_rcu+0xb0/0xd0 [12978.038968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12978.047648] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12978.051740] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12978.060249] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12978.065468] [12978.074845] Workqueue: events delayed_work [ceph] [12978.079803] [12978.087274] [12978.087275] RIP: 0010:lock_release+0x32/0x260 [12978.089371] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12978.094068] Code: 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e <89> c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 [12978.096169] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12978.097655] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [12978.102008] do_raw_spin_lock+0xb5/0xc0 [12978.107310] [12978.107310] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [12978.126051] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12978.130919] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [12978.136141] try_get_cap_refs+0x159/0x880 [ceph] [12978.139967] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [12978.141464] __ceph_get_caps+0x4a1/0x760 [ceph] [12978.148583] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [12978.153372] ? __file_remove_privs+0x89/0x110 [12978.160489] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [12978.165101] ? wake_bit_function+0x70/0x70 [12978.172226] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12978.176752] ? ceph_write_iter+0x260/0x770 [ceph] [12978.183872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12978.188225] ceph_write_iter+0x260/0x770 [ceph] [12978.195347] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12978.199461] ? lock_is_held_type+0xaa/0x120 [12978.207533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12978.212233] vfs_write+0x324/0x4f0 [12978.217970] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12978.222497] ksys_pwrite64+0x65/0xa0 [12978.229618] Call Trace: [12978.229619] [12978.233795] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12978.240919] ? watchdog_timer_fn+0x1b4/0x220 [12978.244318] do_syscall_64+0x38/0x80 [12978.251459] ? __hrtimer_run_queues+0x1d3/0x360 [12978.255029] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12978.257477] ? hrtimer_interrupt+0x104/0x240 [12978.259485] RIP: 0033:0x7f48f0eeb93b [12978.264615] ? __local_bh_enable+0x50/0x60 [12978.268879] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12978.272462] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12978.276982] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [12978.282026] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12978.286290] ORIG_RAX: 0000000000000012 [12978.289861] [12978.293952] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12978.312689] [12978.312691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12978.317905] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12978.323124] ? igrab+0x43/0x50 [12978.328080] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12978.331914] ? igrab+0x43/0x50 [12978.334008] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12978.341132] ? lock_release+0x32/0x260 [12978.343229] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12978.348534] ? lock_release+0xc5/0x260 [12978.355665] [12978.358711] _raw_spin_unlock+0x1b/0x40 [12978.396638] igrab+0x43/0x50 [12978.399523] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12978.404673] delayed_work+0x1d8/0x280 [ceph] [12978.408953] ? process_scheduled_works+0x22a/0x510 [12978.413755] process_scheduled_works+0x2a4/0x510 [12978.418377] worker_thread+0x188/0x340 [12978.422127] ? rescuer_thread+0x310/0x310 [12978.426129] kthread+0xf5/0x130 [12978.429266] ? kthread_complete_and_exit+0x20/0x20 [12978.434051] ret_from_fork+0x31/0x40 [12978.437630] ? kthread_complete_and_exit+0x20/0x20 [12978.442415] ret_from_fork_asm+0x11/0x20 [12978.446346] [12979.123450] rcu: INFO: rcu_preempt self-detected stall on CPU [12979.129201] rcu: 0-....: (1021360 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=483288 [12979.139454] rcu: (t=1042495 jiffies g=1309609 q=111054 ncpus=8) [12979.145452] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12979.154405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12979.161882] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [12979.167369] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [12979.186117] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [12979.191339] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [12979.198463] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [12979.205590] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [12979.212724] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [12979.219855] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [12979.226988] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [12979.235073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12979.240811] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [12979.247935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12979.255058] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12979.262183] Call Trace: [12979.264629] [12979.266639] ? rcu_dump_cpu_stacks+0xd1/0x180 [12979.271000] ? rcu_sched_clock_irq+0x633/0x1390 [12979.275531] ? lock_is_held_type+0xaa/0x120 [12979.279718] ? lock_is_held_type+0xaa/0x120 [12979.283894] ? tick_sched_do_timer+0xa0/0xa0 [12979.288167] ? rcu_read_lock_sched_held+0x56/0x90 [12979.292876] ? tick_sched_do_timer+0xa0/0xa0 [12979.297145] ? update_process_times+0x5f/0x90 [12979.301498] ? tick_sched_handle.isra.16+0x1e/0x50 [12979.306289] ? tick_sched_timer+0x6f/0x80 [12979.310294] ? __hrtimer_run_queues+0x1d3/0x360 [12979.314821] ? hrtimer_interrupt+0x104/0x240 [12979.319090] ? __local_bh_enable+0x50/0x60 [12979.323192] ? __sysvec_apic_timer_interrupt+0x61/0x160 [12979.328418] ? sysvec_apic_timer_interrupt+0x6d/0x90 [12979.333381] [12979.335477] [12979.337575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [12979.342885] ? queued_spin_lock_slowpath+0x2d/0x2e0 [12979.347762] do_raw_spin_lock+0xb5/0xc0 [12979.351601] __cap_delay_requeue+0xa7/0x1d0 [ceph] [12979.356410] try_get_cap_refs+0x159/0x880 [ceph] [12979.361043] __ceph_get_caps+0x4a1/0x760 [ceph] [12979.365582] ? __file_remove_privs+0x89/0x110 [12979.369938] ? wake_bit_function+0x70/0x70 [12979.374031] ? ceph_write_iter+0x260/0x770 [ceph] [12979.378743] ceph_write_iter+0x260/0x770 [ceph] [12979.383280] ? lock_is_held_type+0xaa/0x120 [12979.387469] vfs_write+0x324/0x4f0 [12979.390875] ksys_pwrite64+0x65/0xa0 [12979.394448] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [12979.399589] do_syscall_64+0x38/0x80 [12979.403167] entry_SYSCALL_64_after_hwframe+0x63/0xcd [12979.408210] RIP: 0033:0x7f48f0eeb93b [12979.411782] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [12979.430518] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [12979.438076] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [12979.445200] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [12979.452323] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [12979.459450] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [12979.466573] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [12979.473703] [12979.475890] Sending NMI from CPU 0 to CPUs 7: [12979.480243] NMI backtrace for cpu 7 [12979.480245] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [12979.480247] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [12979.480248] Workqueue: events delayed_work [ceph] [12979.480272] RIP: 0010:__lock_acquire+0xfb/0x1e20 [12979.480275] Code: c1 fc 04 48 89 34 24 4c 0f af e0 8b 44 24 10 4c 8d 04 80 49 c1 e0 03 85 c0 74 25 41 f6 c5 01 75 1f 49 8d 74 30 d8 0f b7 46 20 <25> ff 1f 00 00 44 39 e0 75 0c 48 83 7c 24 08 00 0f 85 0c 07 00 00 [12979.480276] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [12979.480277] RAX: 0000000000000702 RBX: ffff88810debaa80 RCX: 0000000000000000 [12979.480278] RDX: 0000000000000000 RSI: ffff88810debb528 RDI: ffff8881a1d34518 [12979.480279] RBP: 0000000000000001 R08: 0000000000000078 R09: 0000000000000000 [12979.480280] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [12979.480280] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8881a1d34518 [12979.480281] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [12979.480282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12979.480283] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [12979.480284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12979.480284] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [12979.480285] Call Trace: [12979.480286] [12979.480287] ? nmi_cpu_backtrace+0xad/0x120 [12979.480291] ? nmi_cpu_backtrace_handler+0x11/0x20 [12979.480293] ? nmi_handle+0xd4/0x260 [12979.480297] ? default_do_nmi+0x49/0xf0 [12979.480299] ? exc_nmi+0x11e/0x150 [12979.480300] ? end_repeat_nmi+0x16/0x67 [12979.480304] ? __lock_acquire+0xfb/0x1e20 [12979.480306] ? __lock_acquire+0xfb/0x1e20 [12979.480307] ? __lock_acquire+0xfb/0x1e20 [12979.480309] [12979.480309] [12979.480311] lock_acquire+0xcb/0x2b0 [12979.480313] ? igrab+0x1d/0x50 [12979.480315] ? igrab+0x43/0x50 [12979.480316] ? lock_release+0xc5/0x260 [12979.480318] _raw_spin_lock+0x30/0x40 [12979.480321] ? igrab+0x1d/0x50 [12979.480322] igrab+0x1d/0x50 [12979.480324] ceph_check_delayed_caps+0xb7/0x270 [ceph] [12979.480339] delayed_work+0x1d8/0x280 [ceph] [12979.480352] ? process_scheduled_works+0x22a/0x510 [12979.480356] process_scheduled_works+0x2a4/0x510 [12979.480360] worker_thread+0x188/0x340 [12979.480363] ? rescuer_thread+0x310/0x310 [12979.480365] kthread+0xf5/0x130 [12979.480367] ? kthread_complete_and_exit+0x20/0x20 [12979.480370] ret_from_fork+0x31/0x40 [12979.480372] ? kthread_complete_and_exit+0x20/0x20 [12979.480374] ret_from_fork_asm+0x11/0x20 [12979.480378] [13005.682550] watchdog: BUG: soft lockup - CPU#0 stuck for 998s! [dbench:78683] [13005.689685] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13005.694549] watchdog: BUG: soft lockup - CPU#7 stuck for 9995s! [kworker/7:4:78808] [13005.745135] irq event stamp: 15099870 [13005.752778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [13005.756436] hardirqs last enabled at (15099869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13005.764608] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13005.774765] hardirqs last disabled at (15099870): [] sysvec_apic_timer_interrupt+0xe/0x90 [13005.774768] nft_compat [13005.774769] softirqs last enabled at (15099832): [] __do_softirq+0x2be/0x371 [13005.781803] nf_tables [13005.791526] softirqs last disabled at (15099827): [] irq_exit_rcu+0xb0/0xd0 [13005.793972] nfnetlink bridge [13005.802657] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13005.805013] stp [13005.813526] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13005.816487] llc xfs [13005.825432] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13005.827270] libcrc32c sunrpc [13005.834741] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13005.836941] intel_rapl_msr [13005.842419] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13005.845384] intel_rapl_common sb_edac [13005.864122] [13005.866912] x86_pkg_temp_thermal [13005.872130] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13005.875872] coretemp [13005.877365] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13005.880675] kvm_intel kvm [13005.887798] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13005.890069] irqbypass i2c_i801 [13005.897192] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13005.899897] iTCO_wdt [13005.907022] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13005.910159] iTCO_vendor_support [13005.917283] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13005.919554] i2c_smbus crct10dif_pclmul [13005.926678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13005.929901] joydev [13005.937979] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13005.941811] crc32_pclmul ipmi_ssif [13005.947548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13005.949645] ghash_clmulni_intel mei_me [13005.956769] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13005.960254] mei [13005.967377] Call Trace: [13005.967380] [13005.971208] lpc_ich sha512_ssse3 [13005.978334] ? watchdog_timer_fn+0x1b4/0x220 [13005.980169] acpi_ipmi mfd_core [13005.982614] ? __hrtimer_run_queues+0x1d3/0x360 [13005.984625] wmi ipmi_si [13005.987953] ? hrtimer_interrupt+0x104/0x240 [13005.992229] ipmi_devintf [13005.995361] ? __local_bh_enable+0x50/0x60 [13005.999886] ipmi_msghandler ioatdma [13006.002421] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13006.006683] acpi_pad acpi_power_meter [13006.009298] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13006.013390] ixgbe [13006.016959] [13006.022178] igb mdio [13006.025922] [13006.025924] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13006.030897] crc32c_intel nvme [13006.032912] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13006.035005] ptp nvme_core [13006.037278] do_raw_spin_lock+0xb5/0xc0 [13006.039372] i2c_algo_bit pps_core [13006.044678] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13006.047727] dca fuse [13006.052601] try_get_cap_refs+0x159/0x880 [ceph] [13006.055304] [13006.055304] irq event stamp: 27387874 [13006.059140] __ceph_get_caps+0x4a1/0x760 [ceph] [13006.062530] hardirqs last enabled at (27387873): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13006.067320] ? __file_remove_privs+0x89/0x110 [13006.069586] hardirqs last disabled at (27387874): [] sysvec_apic_timer_interrupt+0xe/0x90 [13006.074196] ? wake_bit_function+0x70/0x70 [13006.075685] softirqs last enabled at (27387872): [] __do_softirq+0x2be/0x371 [13006.079346] ? ceph_write_iter+0x260/0x770 [ceph] [13006.083869] softirqs last disabled at (27387867): [] irq_exit_rcu+0xb0/0xd0 [13006.094025] ceph_write_iter+0x260/0x770 [ceph] [13006.098376] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13006.108104] ? lock_is_held_type+0xaa/0x120 [13006.112191] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13006.120879] vfs_write+0x324/0x4f0 [13006.125574] Workqueue: events delayed_work [ceph] [13006.134088] ksys_pwrite64+0x65/0xa0 [13006.138609] [13006.138610] RIP: 0010:lock_acquire+0xf0/0x2b0 [13006.147986] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13006.152163] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13006.159636] do_syscall_64+0x38/0x80 [13006.163032] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13006.167730] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13006.171300] [13006.171301] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13006.172793] RIP: 0033:0x7f48f0eeb93b [13006.177141] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13006.182272] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13006.201010] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13006.204579] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13006.209798] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13006.214842] ORIG_RAX: 0000000000000012 [13006.216333] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13006.223456] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13006.227026] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13006.234151] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13006.252888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13006.260014] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13006.265243] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13006.272363] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13006.276193] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13006.283319] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13006.290442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13006.298525] [13006.305643] Call Trace: [13006.358756] [13006.360765] ? watchdog_timer_fn+0x1b4/0x220 [13006.365031] ? __hrtimer_run_queues+0x1d3/0x360 [13006.369555] ? hrtimer_interrupt+0x104/0x240 [13006.373817] ? __local_bh_enable+0x50/0x60 [13006.377910] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13006.383136] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13006.388091] [13006.390188] [13006.392286] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13006.397595] ? lock_acquire+0xf0/0x2b0 [13006.401345] ? igrab+0x43/0x50 [13006.404393] ? lock_release+0xc5/0x260 [13006.408139] _raw_spin_lock+0x30/0x40 [13006.411795] ? igrab+0x1d/0x50 [13006.414847] igrab+0x1d/0x50 [13006.417725] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13006.422875] delayed_work+0x1d8/0x280 [ceph] [13006.427155] ? process_scheduled_works+0x22a/0x510 [13006.431957] process_scheduled_works+0x2a4/0x510 [13006.436579] worker_thread+0x188/0x340 [13006.440329] ? rescuer_thread+0x310/0x310 [13006.444340] kthread+0xf5/0x130 [13006.447486] ? kthread_complete_and_exit+0x20/0x20 [13006.452270] ret_from_fork+0x31/0x40 [13006.455851] ? kthread_complete_and_exit+0x20/0x20 [13006.460641] ret_from_fork_asm+0x11/0x20 [13006.464567] [13033.683041] watchdog: BUG: soft lockup - CPU#0 stuck for 1024s! [dbench:78683] [13033.690259] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13033.695041] watchdog: BUG: soft lockup - CPU#7 stuck for 10021s! [kworker/7:4:78808] [13033.745724] irq event stamp: 15161000 [13033.753457] Modules linked in: ceph libceph dns_resolver fscache [13033.757113] hardirqs last enabled at (15160999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13033.763120] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [13033.773277] hardirqs last disabled at (15161000): [] sysvec_apic_timer_interrupt+0xe/0x90 [13033.778755] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13033.788479] softirqs last enabled at (15160962): [] __do_softirq+0x2be/0x371 [13033.794648] nf_tables nfnetlink bridge stp llc xfs libcrc32c [13033.803332] softirqs last disabled at (15160957): [] irq_exit_rcu+0xb0/0xd0 [13033.809072] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [13033.817583] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13033.825052] coretemp kvm_intel [13033.833998] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13033.837133] kvm irqbypass i2c_i801 iTCO_wdt [13033.844604] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [13033.848869] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [13033.854348] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [13033.860518] crc32_pclmul ipmi_ssif [13033.879255] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13033.882739] ghash_clmulni_intel mei_me [13033.887957] [13033.887957] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13033.891787] mei lpc_ich [13033.893278] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13033.900401] sha512_ssse3 acpi_ipmi [13033.902932] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13033.910056] mfd_core wmi [13033.913541] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13033.920664] ipmi_si [13033.923282] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13033.930406] ipmi_devintf ipmi_msghandler [13033.932591] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13033.939715] ioatdma acpi_pad [13033.943735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13033.951814] acpi_power_meter ixgbe [13033.954778] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13033.960514] igb [13033.963999] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13033.971123] mdio [13033.972961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13033.980086] crc32c_intel nvme [13033.982008] Call Trace: [13033.989134] ptp nvme_core [13033.992184] [13033.994627] i2c_algo_bit pps_core [13033.997335] ? watchdog_timer_fn+0x1b4/0x220 [13033.999343] dca fuse [13034.002740] ? __hrtimer_run_queues+0x1d3/0x360 [13034.007004] [13034.007004] irq event stamp: 27609744 [13034.009279] ? hrtimer_interrupt+0x104/0x240 [13034.013800] hardirqs last enabled at (27609743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13034.015290] ? __local_bh_enable+0x50/0x60 [13034.018947] hardirqs last disabled at (27609744): [] sysvec_apic_timer_interrupt+0xe/0x90 [13034.023213] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13034.033369] softirqs last enabled at (27609742): [] __do_softirq+0x2be/0x371 [13034.037459] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13034.047183] softirqs last disabled at (27609737): [] irq_exit_rcu+0xb0/0xd0 [13034.052401] [13034.061085] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13034.066043] [13034.066044] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13034.074554] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13034.074556] Workqueue: events delayed_work [ceph] [13034.076654] ? queued_spin_lock_slowpath+0x2b/0x2e0 [13034.086029] [13034.088128] do_raw_spin_lock+0xb5/0xc0 [13034.093442] RIP: 0010:lock_acquire+0xf0/0x2b0 [13034.100911] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13034.105605] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13034.110481] try_get_cap_refs+0x159/0x880 [ceph] [13034.111969] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13034.115804] __ceph_get_caps+0x4a1/0x760 [ceph] [13034.120149] [13034.120150] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13034.124937] ? __file_remove_privs+0x89/0x110 [13034.143670] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13034.148281] ? wake_bit_function+0x70/0x70 [13034.153499] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13034.158027] ? ceph_write_iter+0x260/0x770 [ceph] [13034.159513] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13034.166639] ceph_write_iter+0x260/0x770 [ceph] [13034.170990] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13034.178117] ? lock_is_held_type+0xaa/0x120 [13034.182205] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13034.189331] vfs_write+0x324/0x4f0 [13034.194025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13034.201153] ksys_pwrite64+0x65/0xa0 [13034.205675] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13034.212800] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13034.216976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13034.225054] do_syscall_64+0x38/0x80 [13034.228451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13034.234188] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13034.237759] Call Trace: [13034.237761] [13034.244884] RIP: 0033:0x7f48f0eeb93b [13034.250014] ? watchdog_timer_fn+0x1b4/0x220 [13034.257139] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13034.260711] ? __hrtimer_run_queues+0x1d3/0x360 [13034.267849] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13034.272898] ? hrtimer_interrupt+0x104/0x240 [13034.275337] ORIG_RAX: 0000000000000012 [13034.277349] ? __local_bh_enable+0x50/0x60 [13034.280919] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13034.285186] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13034.303921] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13034.308448] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13034.313663] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13034.317927] [13034.321757] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13034.325848] [13034.325849] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13034.332972] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13034.338194] ? lock_acquire+0xf0/0x2b0 [13034.345320] [13034.350272] ? igrab+0x43/0x50 [13034.390132] ? lock_release+0xc5/0x260 [13034.393875] _raw_spin_lock+0x30/0x40 [13034.397532] ? igrab+0x1d/0x50 [13034.400582] igrab+0x1d/0x50 [13034.403462] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13034.408612] delayed_work+0x1d8/0x280 [ceph] [13034.412892] ? process_scheduled_works+0x22a/0x510 [13034.417681] process_scheduled_works+0x2a4/0x510 [13034.422298] worker_thread+0x188/0x340 [13034.426047] ? rescuer_thread+0x310/0x310 [13034.430050] kthread+0xf5/0x130 [13034.433187] ? kthread_complete_and_exit+0x20/0x20 [13034.437971] ret_from_fork+0x31/0x40 [13034.441542] ? kthread_complete_and_exit+0x20/0x20 [13034.446328] ret_from_fork_asm+0x11/0x20 [13034.450258] [13061.683533] watchdog: BUG: soft lockup - CPU#0 stuck for 1050s! [dbench:78683] [13061.690754] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13061.695533] watchdog: BUG: soft lockup - CPU#7 stuck for 10047s! [kworker/7:4:78808] [13061.746217] irq event stamp: 15218848 [13061.753952] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [13061.757608] hardirqs last enabled at (15218847): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13061.765783] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13061.775939] hardirqs last disabled at (15218848): [] sysvec_apic_timer_interrupt+0xe/0x90 [13061.783931] nf_tables nfnetlink bridge stp llc xfs libcrc32c [13061.793653] softirqs last enabled at (15218810): [] __do_softirq+0x2be/0x371 [13061.799391] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [13061.808075] softirqs last disabled at (15218805): [] irq_exit_rcu+0xb0/0xd0 [13061.816326] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [13061.824838] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13061.824840] iTCO_vendor_support [13061.824841] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13061.829969] i2c_smbus crct10dif_pclmul [13061.838929] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13061.842153] joydev crc32_pclmul [13061.849627] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13061.853455] ipmi_ssif ghash_clmulni_intel [13061.858934] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13061.862158] mei_me mei [13061.880895] [13061.884985] lpc_ich [13061.890203] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13061.892665] sha512_ssse3 acpi_ipmi [13061.894156] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13061.896339] mfd_core wmi [13061.903463] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13061.906948] ipmi_si [13061.914072] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13061.916688] ipmi_devintf [13061.923814] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13061.925996] ipmi_msghandler ioatdma [13061.933122] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13061.935738] acpi_pad [13061.942862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13061.946434] acpi_power_meter [13061.954510] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13061.956782] ixgbe igb [13061.962518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13061.965484] mdio [13061.972607] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13061.974965] crc32c_intel nvme [13061.982088] Call Trace: [13061.984012] ptp nvme_core [13061.991138] [13061.994186] i2c_algo_bit pps_core [13061.996651] ? watchdog_timer_fn+0x1b4/0x220 [13061.999353] dca [13062.001364] ? __hrtimer_run_queues+0x1d3/0x360 [13062.004762] fuse [13062.004763] irq event stamp: 27831828 [13062.009030] ? hrtimer_interrupt+0x104/0x240 [13062.010862] hardirqs last enabled at (27831827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13062.015386] ? __local_bh_enable+0x50/0x60 [13062.017310] hardirqs last disabled at (27831828): [] sysvec_apic_timer_interrupt+0xe/0x90 [13062.020972] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13062.025232] softirqs last enabled at (27831826): [] __do_softirq+0x2be/0x371 [13062.035391] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13062.039481] softirqs last disabled at (27831821): [] irq_exit_rcu+0xb0/0xd0 [13062.049222] [13062.054440] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13062.063124] [13062.068081] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13062.076592] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13062.078691] Workqueue: events delayed_work [ceph] [13062.088072] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13062.090165] [13062.090166] RIP: 0010:lock_acquire+0xf0/0x2b0 [13062.097638] do_raw_spin_lock+0xb5/0xc0 [13062.102938] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13062.107639] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13062.112507] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13062.114000] try_get_cap_refs+0x159/0x880 [ceph] [13062.118348] [13062.118349] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13062.122186] __ceph_get_caps+0x4a1/0x760 [ceph] [13062.140934] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13062.145724] ? __file_remove_privs+0x89/0x110 [13062.150935] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13062.155547] ? wake_bit_function+0x70/0x70 [13062.157037] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13062.164164] ? ceph_write_iter+0x260/0x770 [ceph] [13062.168686] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13062.175809] ceph_write_iter+0x260/0x770 [ceph] [13062.180159] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13062.187289] ? lock_is_held_type+0xaa/0x120 [13062.191374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13062.198502] vfs_write+0x324/0x4f0 [13062.203213] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13062.210344] ksys_pwrite64+0x65/0xa0 [13062.214862] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13062.222942] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13062.227117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13062.232856] do_syscall_64+0x38/0x80 [13062.236253] Call Trace: [13062.236255] [13062.243378] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13062.246949] ? watchdog_timer_fn+0x1b4/0x220 [13062.254073] RIP: 0033:0x7f48f0eeb93b [13062.259215] ? __hrtimer_run_queues+0x1d3/0x360 [13062.266336] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13062.269908] ? hrtimer_interrupt+0x104/0x240 [13062.272350] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13062.274359] ? __local_bh_enable+0x50/0x60 [13062.279404] ORIG_RAX: 0000000000000012 [13062.283670] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13062.287239] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13062.291764] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13062.310500] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13062.314764] [13062.319981] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13062.324074] [13062.327903] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13062.333121] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13062.340245] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13062.345208] ? lock_acquire+0xf0/0x2b0 [13062.352332] [13062.354427] ? igrab+0x43/0x50 [13062.392196] ? lock_release+0xc5/0x260 [13062.395960] _raw_spin_lock+0x30/0x40 [13062.399623] ? igrab+0x1d/0x50 [13062.402673] igrab+0x1d/0x50 [13062.405553] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13062.410701] delayed_work+0x1d8/0x280 [ceph] [13062.414983] ? process_scheduled_works+0x22a/0x510 [13062.419773] process_scheduled_works+0x2a4/0x510 [13062.424390] worker_thread+0x188/0x340 [13062.428138] ? rescuer_thread+0x310/0x310 [13062.432141] kthread+0xf5/0x130 [13062.435286] ? kthread_complete_and_exit+0x20/0x20 [13062.440072] ret_from_fork+0x31/0x40 [13062.443652] ? kthread_complete_and_exit+0x20/0x20 [13062.448443] ret_from_fork_asm+0x11/0x20 [13062.452374] [13089.684024] watchdog: BUG: soft lockup - CPU#0 stuck for 1076s! [dbench:78683] [13089.691241] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13089.696024] watchdog: BUG: soft lockup - CPU#7 stuck for 10073s! [kworker/7:4:78808] [13089.746706] irq event stamp: 15276660 [13089.754438] Modules linked in: ceph libceph dns_resolver fscache netfs [13089.758096] hardirqs last enabled at (15276659): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13089.764623] veth nft_chain_nat xt_MASQUERADE nf_nat [13089.774779] hardirqs last disabled at (15276660): [] sysvec_apic_timer_interrupt+0xe/0x90 [13089.774781] nf_conntrack [13089.774782] softirqs last enabled at (15276622): [] __do_softirq+0x2be/0x371 [13089.779737] nf_defrag_ipv6 nf_defrag_ipv4 [13089.789461] softirqs last disabled at (15276617): [] irq_exit_rcu+0xb0/0xd0 [13089.792077] nft_compat nf_tables [13089.800763] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13089.804853] nfnetlink bridge [13089.813365] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13089.816676] stp llc [13089.825620] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13089.828584] xfs libcrc32c [13089.836055] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13089.838237] sunrpc intel_rapl_msr [13089.843715] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13089.846421] intel_rapl_common sb_edac [13089.865158] [13089.868553] x86_pkg_temp_thermal coretemp [13089.873771] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13089.877517] kvm_intel kvm [13089.879005] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13089.883097] irqbypass i2c_i801 [13089.890222] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13089.892927] iTCO_wdt [13089.900051] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13089.903186] iTCO_vendor_support i2c_smbus [13089.910311] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13089.912583] crct10dif_pclmul joydev [13089.919707] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13089.923798] crc32_pclmul ipmi_ssif [13089.930921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13089.934492] ghash_clmulni_intel mei_me [13089.942568] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13089.946053] mei lpc_ich [13089.951791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13089.955622] sha512_ssse3 [13089.962746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13089.965277] acpi_ipmi mfd_core [13089.972401] Call Trace: [13089.975034] wmi ipmi_si ipmi_devintf [13089.982160] [13089.985296] ipmi_msghandler ioatdma [13089.987743] ? watchdog_timer_fn+0x1b4/0x220 [13089.991399] acpi_pad acpi_power_meter [13089.993426] ? __hrtimer_run_queues+0x1d3/0x360 [13089.996996] ixgbe igb [13090.001265] ? hrtimer_interrupt+0x104/0x240 [13090.005005] mdio crc32c_intel [13090.009529] ? __local_bh_enable+0x50/0x60 [13090.011886] nvme ptp [13090.016154] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13090.019201] nvme_core i2c_algo_bit [13090.023293] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13090.025563] pps_core dca [13090.030779] [13090.034264] fuse [13090.039221] [13090.041838] irq event stamp: 28054794 [13090.041839] hardirqs last enabled at (28054793): [] asm_common_interrupt+0x26/0x40 [13090.043937] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13090.045860] hardirqs last disabled at (28054794): [] sysvec_apic_timer_interrupt+0xe/0x90 [13090.047962] ? queued_spin_lock_slowpath+0x16/0x2e0 [13090.051615] softirqs last enabled at (28054792): [] __do_softirq+0x2be/0x371 [13090.060821] do_raw_spin_lock+0xb5/0xc0 [13090.066124] softirqs last disabled at (28054775): [] irq_exit_rcu+0xb0/0xd0 [13090.075850] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13090.080719] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13090.089422] try_get_cap_refs+0x159/0x880 [ceph] [13090.093251] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13090.101767] __ceph_get_caps+0x4a1/0x760 [ceph] [13090.106546] Workqueue: events delayed_work [ceph] [13090.115929] ? __file_remove_privs+0x89/0x110 [13090.120534] [13090.120535] RIP: 0010:lock_release+0x14e/0x260 [13090.128005] ? wake_bit_function+0x70/0x70 [13090.132528] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [13090.137230] ? ceph_write_iter+0x260/0x770 [ceph] [13090.141577] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [13090.143067] ceph_write_iter+0x260/0x770 [ceph] [13090.147504] [13090.147504] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [13090.151601] ? lock_is_held_type+0xaa/0x120 [13090.170333] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [13090.175033] vfs_write+0x324/0x4f0 [13090.180247] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [13090.184777] ksys_pwrite64+0x65/0xa0 [13090.186264] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [13090.193388] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13090.197565] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [13090.204706] do_syscall_64+0x38/0x80 [13090.208104] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13090.215229] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13090.218797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13090.225922] RIP: 0033:0x7f48f0eeb93b [13090.231053] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13090.238177] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13090.241747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13090.249825] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13090.254871] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13090.260606] ORIG_RAX: 0000000000000012 [13090.264177] Call Trace: [13090.264178] [13090.271303] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13090.290040] ? watchdog_timer_fn+0x1b4/0x220 [13090.297164] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13090.302383] ? __hrtimer_run_queues+0x1d3/0x360 [13090.309505] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13090.313339] ? hrtimer_interrupt+0x104/0x240 [13090.315781] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13090.317790] ? __local_bh_enable+0x50/0x60 [13090.324916] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13090.329182] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13090.336309] [13090.340827] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13090.382913] [13090.385012] [13090.387109] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13090.392431] ? igrab+0x43/0x50 [13090.395483] ? lock_release+0x14e/0x260 [13090.399321] ? lock_release+0xc5/0x260 [13090.403066] _raw_spin_unlock+0x1b/0x40 [13090.406904] igrab+0x43/0x50 [13090.409782] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13090.414934] delayed_work+0x1d8/0x280 [ceph] [13090.419214] ? process_scheduled_works+0x22a/0x510 [13090.424004] process_scheduled_works+0x2a4/0x510 [13090.428619] worker_thread+0x188/0x340 [13090.432369] ? rescuer_thread+0x310/0x310 [13090.436379] kthread+0xf5/0x130 [13090.439517] ? kthread_complete_and_exit+0x20/0x20 [13090.444303] ret_from_fork+0x31/0x40 [13090.447882] ? kthread_complete_and_exit+0x20/0x20 [13090.452666] ret_from_fork_asm+0x11/0x20 [13090.456597] [13117.684516] watchdog: BUG: soft lockup - CPU#0 stuck for 1102s! [dbench:78683] [13117.691737] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13117.696516] watchdog: BUG: soft lockup - CPU#7 stuck for 10099s! [kworker/7:4:78808] [13117.747198] irq event stamp: 15334500 [13117.747200] hardirqs last enabled at (15334499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13117.754933] Modules linked in: [13117.758590] hardirqs last disabled at (15334500): [] sysvec_apic_timer_interrupt+0xe/0x90 [13117.768748] ceph [13117.771799] softirqs last enabled at (15334462): [] __do_softirq+0x2be/0x371 [13117.781525] libceph [13117.783449] softirqs last disabled at (15334457): [] irq_exit_rcu+0xb0/0xd0 [13117.792132] dns_resolver fscache [13117.794318] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13117.802828] netfs veth [13117.806137] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13117.815080] nft_chain_nat xt_MASQUERADE [13117.817526] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13117.824996] nf_nat nf_conntrack [13117.828916] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13117.834391] nf_defrag_ipv6 [13117.837616] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13117.856354] nf_defrag_ipv4 [13117.859146] [13117.864361] nft_compat [13117.867151] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13117.868642] nf_tables nfnetlink [13117.871088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13117.878210] bridge stp [13117.881435] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13117.888558] llc [13117.891004] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13117.898128] xfs libcrc32c [13117.899964] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13117.907089] sunrpc intel_rapl_msr [13117.909793] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13117.916917] intel_rapl_common [13117.920314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13117.928392] sb_edac [13117.931443] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13117.937196] x86_pkg_temp_thermal [13117.939381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13117.946505] coretemp [13117.949817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13117.956941] kvm_intel [13117.959211] Call Trace: [13117.959212] [13117.966334] kvm irqbypass [13117.968694] ? watchdog_timer_fn+0x1b4/0x220 [13117.971137] i2c_i801 iTCO_wdt [13117.973149] ? __hrtimer_run_queues+0x1d3/0x360 [13117.975851] iTCO_vendor_support i2c_smbus [13117.980119] ? hrtimer_interrupt+0x104/0x240 [13117.983167] crct10dif_pclmul [13117.987691] ? __local_bh_enable+0x50/0x60 [13117.991781] joydev crc32_pclmul [13117.996049] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13117.999008] ipmi_ssif ghash_clmulni_intel [13118.003101] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13118.006324] mei_me mei [13118.011543] [13118.015633] lpc_ich [13118.020590] [13118.023034] sha512_ssse3 acpi_ipmi [13118.025132] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13118.027315] mfd_core wmi [13118.029416] ? queued_spin_lock_slowpath+0x16/0x2e0 [13118.032914] ipmi_si ipmi_devintf [13118.038219] do_raw_spin_lock+0xb5/0xc0 [13118.040835] ipmi_msghandler ioatdma [13118.045708] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13118.049017] acpi_pad acpi_power_meter [13118.052852] try_get_cap_refs+0x159/0x880 [ceph] [13118.056418] ixgbe igb [13118.061217] __ceph_get_caps+0x4a1/0x760 [ceph] [13118.064955] mdio crc32c_intel [13118.069570] ? __file_remove_privs+0x89/0x110 [13118.071924] nvme [13118.076447] ? wake_bit_function+0x70/0x70 [13118.079498] ptp nvme_core [13118.083852] ? ceph_write_iter+0x260/0x770 [ceph] [13118.085771] i2c_algo_bit [13118.089864] ceph_write_iter+0x260/0x770 [ceph] [13118.092568] pps_core dca [13118.097269] ? lock_is_held_type+0xaa/0x120 [13118.099882] fuse [13118.104408] vfs_write+0x324/0x4f0 [13118.107023] irq event stamp: 28277164 [13118.111216] ksys_pwrite64+0x65/0xa0 [13118.113133] hardirqs last enabled at (28277163): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13118.116532] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13118.120205] hardirqs last disabled at (28277164): [] sysvec_apic_timer_interrupt+0xe/0x90 [13118.123778] do_syscall_64+0x38/0x80 [13118.133933] softirqs last enabled at (28277162): [] __do_softirq+0x2be/0x371 [13118.139068] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13118.148790] softirqs last disabled at (28277157): [] irq_exit_rcu+0xb0/0xd0 [13118.152362] RIP: 0033:0x7f48f0eeb93b [13118.161044] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13118.166089] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13118.174600] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13118.178171] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13118.187566] Workqueue: events delayed_work [ceph] [13118.206302] ORIG_RAX: 0000000000000012 [13118.213772] [13118.213773] RIP: 0010:lock_acquire+0xf0/0x2b0 [13118.218991] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13118.223688] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13118.227518] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13118.229009] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13118.233360] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13118.240485] [13118.240485] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13118.259221] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13118.266346] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13118.271564] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13118.278687] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13118.280199] [13118.287320] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13118.325126] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13118.332250] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13118.340327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13118.346062] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13118.353202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13118.360330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13118.367454] Call Trace: [13118.369916] [13118.371927] ? watchdog_timer_fn+0x1b4/0x220 [13118.376192] ? __hrtimer_run_queues+0x1d3/0x360 [13118.380735] ? hrtimer_interrupt+0x104/0x240 [13118.385004] ? __local_bh_enable+0x50/0x60 [13118.389097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13118.394322] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13118.399279] [13118.401375] [13118.403474] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13118.408780] ? lock_acquire+0xf0/0x2b0 [13118.412531] ? igrab+0x43/0x50 [13118.415580] ? lock_release+0xc5/0x260 [13118.419326] _raw_spin_lock+0x30/0x40 [13118.422990] ? igrab+0x1d/0x50 [13118.426042] igrab+0x1d/0x50 [13118.428922] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13118.434069] delayed_work+0x1d8/0x280 [ceph] [13118.438350] ? process_scheduled_works+0x22a/0x510 [13118.443142] process_scheduled_works+0x2a4/0x510 [13118.447757] worker_thread+0x188/0x340 [13118.451506] ? rescuer_thread+0x310/0x310 [13118.455509] kthread+0xf5/0x130 [13118.458656] ? kthread_complete_and_exit+0x20/0x20 [13118.463441] ret_from_fork+0x31/0x40 [13118.467020] ? kthread_complete_and_exit+0x20/0x20 [13118.471802] ret_from_fork_asm+0x11/0x20 [13118.475727] [13145.685007] watchdog: BUG: soft lockup - CPU#0 stuck for 1128s! [dbench:78683] [13145.692223] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13145.697007] watchdog: BUG: soft lockup - CPU#7 stuck for 10125s! [kworker/7:4:78808] [13145.747680] irq event stamp: 15392482 [13145.747682] hardirqs last enabled at (15392481): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13145.755413] Modules linked in: [13145.759070] hardirqs last disabled at (15392482): [] sysvec_apic_timer_interrupt+0xe/0x90 [13145.769226] ceph [13145.772278] softirqs last enabled at (15392444): [] __do_softirq+0x2be/0x371 [13145.782003] libceph [13145.783927] softirqs last disabled at (15392439): [] irq_exit_rcu+0xb0/0xd0 [13145.792611] dns_resolver fscache [13145.794796] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13145.803305] netfs [13145.806615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13145.815560] veth nft_chain_nat [13145.817571] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13145.825041] xt_MASQUERADE nf_nat [13145.828180] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13145.833657] nf_conntrack nf_defrag_ipv6 [13145.836983] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13145.855722] nf_defrag_ipv4 nft_compat [13145.859641] [13145.864857] nf_tables nfnetlink [13145.868600] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13145.870092] bridge [13145.873316] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13145.880439] stp llc [13145.882537] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13145.889661] xfs libcrc32c [13145.891862] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13145.898987] sunrpc [13145.901701] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13145.908823] intel_rapl_msr [13145.910921] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13145.918045] intel_rapl_common sb_edac [13145.920835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13145.928913] x86_pkg_temp_thermal coretemp [13145.932657] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13145.938405] kvm_intel [13145.942494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13145.949618] kvm irqbypass [13145.951977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13145.959099] i2c_i801 [13145.961804] Call Trace: [13145.961806] [13145.968929] iTCO_wdt iTCO_vendor_support [13145.971200] ? watchdog_timer_fn+0x1b4/0x220 [13145.973644] i2c_smbus crct10dif_pclmul [13145.975654] ? __hrtimer_run_queues+0x1d3/0x360 [13145.979657] joydev crc32_pclmul [13145.983925] ? hrtimer_interrupt+0x104/0x240 [13145.987753] ipmi_ssif [13145.992277] ? __local_bh_enable+0x50/0x60 [13145.995502] ghash_clmulni_intel mei_me [13145.999767] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13146.002122] mei lpc_ich [13146.006214] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13146.010044] sha512_ssse3 acpi_ipmi [13146.015261] [13146.017791] mfd_core wmi [13146.022750] [13146.026233] ipmi_si ipmi_devintf [13146.028332] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13146.030949] ipmi_msghandler [13146.033049] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13146.036356] ioatdma acpi_pad [13146.041663] do_raw_spin_lock+0xb5/0xc0 [13146.044536] acpi_power_meter [13146.049410] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13146.052372] ixgbe igb [13146.056206] try_get_cap_refs+0x159/0x880 [ceph] [13146.059167] mdio crc32c_intel [13146.063956] __ceph_get_caps+0x4a1/0x760 [ceph] [13146.066309] nvme ptp [13146.070925] ? __file_remove_privs+0x89/0x110 [13146.073971] nvme_core [13146.078494] ? wake_bit_function+0x70/0x70 [13146.080764] i2c_algo_bit pps_core [13146.085117] ? ceph_write_iter+0x260/0x770 [ceph] [13146.087473] dca fuse [13146.091563] ceph_write_iter+0x260/0x770 [ceph] [13146.094962] [13146.094962] irq event stamp: 28498862 [13146.099664] ? lock_is_held_type+0xaa/0x120 [13146.101929] hardirqs last enabled at (28498861): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13146.106456] vfs_write+0x324/0x4f0 [13146.107945] hardirqs last disabled at (28498862): [] sysvec_apic_timer_interrupt+0xe/0x90 [13146.111605] ksys_pwrite64+0x65/0xa0 [13146.115779] softirqs last enabled at (28498860): [] __do_softirq+0x2be/0x371 [13146.125938] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13146.129335] softirqs last disabled at (28498855): [] irq_exit_rcu+0xb0/0xd0 [13146.139059] do_syscall_64+0x38/0x80 [13146.142629] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13146.151315] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13146.156442] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13146.164954] RIP: 0033:0x7f48f0eeb93b [13146.168527] Workqueue: events delayed_work [ceph] [13146.177905] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13146.182947] [13146.182948] RIP: 0010:lock_acquire+0xf0/0x2b0 [13146.190418] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13146.193988] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13146.198691] ORIG_RAX: 0000000000000012 [13146.217432] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13146.218923] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13146.223273] [13146.223274] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13146.228490] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13146.247228] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13146.251059] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13146.256277] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13146.263410] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13146.264901] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13146.272024] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13146.279149] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13146.286277] [13146.293411] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13146.339296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13146.345033] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13146.352158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13146.359284] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13146.366412] Call Trace: [13146.368861] [13146.370871] ? watchdog_timer_fn+0x1b4/0x220 [13146.375134] ? __hrtimer_run_queues+0x1d3/0x360 [13146.379662] ? hrtimer_interrupt+0x104/0x240 [13146.383931] ? __local_bh_enable+0x50/0x60 [13146.388023] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13146.393248] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13146.398205] [13146.400303] [13146.402415] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13146.407725] ? lock_acquire+0xf0/0x2b0 [13146.411475] ? igrab+0x43/0x50 [13146.414525] ? lock_release+0xc5/0x260 [13146.418270] _raw_spin_lock+0x30/0x40 [13146.421926] ? igrab+0x1d/0x50 [13146.424978] igrab+0x1d/0x50 [13146.427855] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13146.433005] delayed_work+0x1d8/0x280 [ceph] [13146.437288] ? process_scheduled_works+0x22a/0x510 [13146.442075] process_scheduled_works+0x2a4/0x510 [13146.446709] worker_thread+0x188/0x340 [13146.450458] ? rescuer_thread+0x310/0x310 [13146.454461] kthread+0xf5/0x130 [13146.457600] ? kthread_complete_and_exit+0x20/0x20 [13146.462384] ret_from_fork+0x31/0x40 [13146.465980] ? kthread_complete_and_exit+0x20/0x20 [13146.470764] ret_from_fork_asm+0x11/0x20 [13146.474701] [13173.685499] watchdog: BUG: soft lockup - CPU#0 stuck for 1154s! [dbench:78683] [13173.692720] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13173.697499] watchdog: BUG: soft lockup - CPU#7 stuck for 10151s! [kworker/7:4:78808] [13173.748176] irq event stamp: 15450428 [13173.755908] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [13173.759567] hardirqs last enabled at (15450427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13173.767738] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [13173.777896] hardirqs last disabled at (15450428): [] sysvec_apic_timer_interrupt+0xe/0x90 [13173.777899] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [13173.793357] softirqs last enabled at (15450390): [] __do_softirq+0x2be/0x371 [13173.800482] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [13173.809176] softirqs last disabled at (15450385): [] irq_exit_rcu+0xb0/0xd0 [13173.815701] x86_pkg_temp_thermal coretemp kvm_intel kvm [13173.824213] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13173.829514] irqbypass i2c_i801 iTCO_wdt [13173.838461] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13173.838463] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13173.842376] iTCO_vendor_support [13173.849848] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13173.855325] i2c_smbus crct10dif_pclmul [13173.858549] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13173.877287] joydev [13173.881118] [13173.886334] crc32_pclmul ipmi_ssif [13173.888431] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13173.889923] ghash_clmulni_intel [13173.893407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13173.900531] mei_me mei [13173.903756] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13173.910897] lpc_ich [13173.913341] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13173.920465] sha512_ssse3 [13173.922650] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13173.929774] acpi_ipmi [13173.932390] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13173.939516] mfd_core wmi [13173.941872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13173.949968] ipmi_si [13173.952585] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13173.958320] ipmi_devintf ipmi_msghandler [13173.960505] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13173.967629] ioatdma acpi_pad [13173.971633] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13173.978758] acpi_power_meter ixgbe [13173.981721] Call Trace: [13173.981723] [13173.988846] igb mdio [13173.992333] ? watchdog_timer_fn+0x1b4/0x220 [13173.994775] crc32c_intel [13173.996785] ? __hrtimer_run_queues+0x1d3/0x360 [13173.999057] nvme ptp [13174.003323] ? hrtimer_interrupt+0x104/0x240 [13174.005936] nvme_core i2c_algo_bit [13174.010461] ? __local_bh_enable+0x50/0x60 [13174.012733] pps_core dca [13174.016999] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13174.020481] fuse [13174.024571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13174.027193] irq event stamp: 28720150 [13174.032415] [13174.034339] hardirqs last enabled at (28720149): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13174.039295] [13174.039297] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13174.042953] hardirqs last disabled at (28720150): [] sysvec_apic_timer_interrupt+0xe/0x90 [13174.045056] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13174.055208] softirqs last enabled at (28720148): [] __do_softirq+0x2be/0x371 [13174.057307] do_raw_spin_lock+0xb5/0xc0 [13174.062610] softirqs last disabled at (28720143): [] irq_exit_rcu+0xb0/0xd0 [13174.072336] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13174.077205] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13174.085904] try_get_cap_refs+0x159/0x880 [ceph] [13174.089729] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13174.098247] __ceph_get_caps+0x4a1/0x760 [ceph] [13174.103023] Workqueue: events delayed_work [ceph] [13174.112406] ? __file_remove_privs+0x89/0x110 [13174.117012] [13174.117013] RIP: 0010:lock_acquire+0xf0/0x2b0 [13174.124484] ? wake_bit_function+0x70/0x70 [13174.129005] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13174.133706] ? ceph_write_iter+0x260/0x770 [ceph] [13174.138053] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13174.139546] ceph_write_iter+0x260/0x770 [ceph] [13174.143900] [13174.143901] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13174.148001] ? lock_is_held_type+0xaa/0x120 [13174.166732] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13174.171432] vfs_write+0x324/0x4f0 [13174.176649] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13174.181186] ksys_pwrite64+0x65/0xa0 [13174.182672] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13174.189796] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13174.193972] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13174.201098] do_syscall_64+0x38/0x80 [13174.204495] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13174.211621] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13174.215191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13174.222315] RIP: 0033:0x7f48f0eeb93b [13174.227445] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13174.234568] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13174.238139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13174.246216] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13174.251259] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13174.256998] ORIG_RAX: 0000000000000012 [13174.260568] Call Trace: [13174.260570] [13174.267694] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13174.286430] ? watchdog_timer_fn+0x1b4/0x220 [13174.293554] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13174.298775] ? __hrtimer_run_queues+0x1d3/0x360 [13174.305903] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13174.309740] ? hrtimer_interrupt+0x104/0x240 [13174.312196] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13174.314208] ? __local_bh_enable+0x50/0x60 [13174.321332] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13174.325598] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13174.332726] [13174.337244] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13174.379339] [13174.381437] [13174.383534] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13174.388842] ? lock_acquire+0xf0/0x2b0 [13174.392592] ? igrab+0x43/0x50 [13174.395642] ? lock_release+0xc5/0x260 [13174.399388] _raw_spin_lock+0x30/0x40 [13174.403053] ? igrab+0x1d/0x50 [13174.406103] igrab+0x1d/0x50 [13174.408981] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13174.414132] delayed_work+0x1d8/0x280 [ceph] [13174.418411] ? process_scheduled_works+0x22a/0x510 [13174.423203] process_scheduled_works+0x2a4/0x510 [13174.427819] worker_thread+0x188/0x340 [13174.431568] ? rescuer_thread+0x310/0x310 [13174.435577] kthread+0xf5/0x130 [13174.438716] ? kthread_complete_and_exit+0x20/0x20 [13174.443501] ret_from_fork+0x31/0x40 [13174.447080] ? kthread_complete_and_exit+0x20/0x20 [13174.451863] ret_from_fork_asm+0x11/0x20 [13174.455787] [13174.486880] rcu: INFO: rcu_preempt self-detected stall on CPU [13174.492641] rcu: 7-....: (1207007 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/1205864 fqs=579105 [13174.502971] rcu: (t=1237854 jiffies g=1309609 q=119136 ncpus=8) [13174.508969] Sending NMI from CPU 7 to CPUs 0: [13174.513321] NMI backtrace for cpu 0 [13174.513322] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13174.513324] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13174.513324] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13174.513327] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13174.513329] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13174.513330] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13174.513331] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13174.513331] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13174.513332] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13174.513333] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13174.513333] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13174.513334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13174.513335] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13174.513336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13174.513337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13174.513338] Call Trace: [13174.513338] [13174.513339] ? nmi_cpu_backtrace+0xad/0x120 [13174.513343] ? nmi_cpu_backtrace_handler+0x11/0x20 [13174.513344] ? nmi_handle+0xd4/0x260 [13174.513349] ? default_do_nmi+0x49/0xf0 [13174.513350] ? exc_nmi+0x11e/0x150 [13174.513352] ? end_repeat_nmi+0x16/0x67 [13174.513356] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13174.513358] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13174.513361] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13174.513363] [13174.513364] [13174.513365] do_raw_spin_lock+0xb5/0xc0 [13174.513367] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13174.513381] try_get_cap_refs+0x159/0x880 [ceph] [13174.513395] __ceph_get_caps+0x4a1/0x760 [ceph] [13174.513408] ? __file_remove_privs+0x89/0x110 [13174.513410] ? wake_bit_function+0x70/0x70 [13174.513412] ? ceph_write_iter+0x260/0x770 [ceph] [13174.513423] ceph_write_iter+0x260/0x770 [ceph] [13174.513435] ? lock_is_held_type+0xaa/0x120 [13174.513437] vfs_write+0x324/0x4f0 [13174.513441] ksys_pwrite64+0x65/0xa0 [13174.513444] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13174.513446] do_syscall_64+0x38/0x80 [13174.513448] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13174.513450] RIP: 0033:0x7f48f0eeb93b [13174.513451] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13174.513452] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [13174.513454] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13174.513454] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13174.513455] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13174.513456] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13174.513456] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13174.513459] [13174.514319] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13174.818227] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13174.825698] Workqueue: events delayed_work [ceph] [13174.830413] RIP: 0010:lock_release+0x14e/0x260 [13174.834849] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [13174.853588] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [13174.858805] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [13174.865928] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [13174.873053] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [13174.880179] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [13174.887302] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [13174.894428] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13174.902503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13174.908240] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13174.915365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13174.922490] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13174.929612] Call Trace: [13174.932058] [13174.934068] ? rcu_dump_cpu_stacks+0xd1/0x180 [13174.938429] ? rcu_sched_clock_irq+0x633/0x1390 [13174.942953] ? lock_release+0xc5/0x260 [13174.946696] ? lock_is_held_type+0xaa/0x120 [13174.950873] ? lock_is_held_type+0xaa/0x120 [13174.955051] ? tick_sched_do_timer+0xa0/0xa0 [13174.959322] ? rcu_read_lock_sched_held+0x56/0x90 [13174.964022] ? tick_sched_do_timer+0xa0/0xa0 [13174.968285] ? update_process_times+0x5f/0x90 [13174.972636] ? tick_sched_handle.isra.16+0x1e/0x50 [13174.977420] ? tick_sched_timer+0x6f/0x80 [13174.981425] ? __hrtimer_run_queues+0x1d3/0x360 [13174.985952] ? hrtimer_interrupt+0x104/0x240 [13174.990220] ? __local_bh_enable+0x50/0x60 [13174.994315] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13174.999539] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13175.004495] [13175.006593] [13175.008690] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13175.013993] ? igrab+0x43/0x50 [13175.017048] ? lock_release+0x14e/0x260 [13175.020885] ? lock_release+0xc5/0x260 [13175.024631] _raw_spin_unlock+0x1b/0x40 [13175.028468] igrab+0x43/0x50 [13175.031347] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13175.036495] delayed_work+0x1d8/0x280 [ceph] [13175.040775] ? process_scheduled_works+0x22a/0x510 [13175.045558] process_scheduled_works+0x2a4/0x510 [13175.050175] worker_thread+0x188/0x340 [13175.053922] ? rescuer_thread+0x310/0x310 [13175.057925] kthread+0xf5/0x130 [13175.061063] ? kthread_complete_and_exit+0x20/0x20 [13175.065849] ret_from_fork+0x31/0x40 [13175.069419] ? kthread_complete_and_exit+0x20/0x20 [13175.074202] ret_from_fork_asm+0x11/0x20 [13175.078126] [13201.685991] watchdog: BUG: soft lockup - CPU#0 stuck for 1181s! [dbench:78683] [13201.693205] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13201.697991] watchdog: BUG: soft lockup - CPU#7 stuck for 10178s! [kworker/7:4:78808] [13201.748662] irq event stamp: 15508288 [13201.756395] Modules linked in: ceph libceph dns_resolver fscache netfs veth [13201.760051] hardirqs last enabled at (15508287): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13201.767003] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13201.777162] hardirqs last disabled at (15508288): [] sysvec_apic_timer_interrupt+0xe/0x90 [13201.785420] nft_compat nf_tables nfnetlink bridge [13201.795145] softirqs last enabled at (15508250): [] __do_softirq+0x2be/0x371 [13201.799928] stp llc xfs libcrc32c sunrpc [13201.808613] softirqs last disabled at (15508245): [] irq_exit_rcu+0xb0/0xd0 [13201.812625] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [13201.821136] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13201.828779] kvm_intel kvm [13201.837724] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13201.840429] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support [13201.847898] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13201.853550] i2c_smbus crct10dif_pclmul joydev [13201.859028] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13201.863464] crc32_pclmul ipmi_ssif ghash_clmulni_intel [13201.882202] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13201.887420] mei_me mei [13201.892636] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13201.892638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13201.895083] lpc_ich [13201.902206] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13201.909329] sha512_ssse3 [13201.911515] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13201.918637] acpi_ipmi mfd_core [13201.921255] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13201.928386] wmi ipmi_si [13201.931525] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13201.938665] ipmi_devintf ipmi_msghandler [13201.941197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13201.949275] ioatdma acpi_pad [13201.953279] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13201.959016] acpi_power_meter [13201.961980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13201.969105] ixgbe [13201.972068] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13201.979192] igb [13201.981203] Call Trace: [13201.981205] [13201.988329] mdio crc32c_intel [13201.990166] ? watchdog_timer_fn+0x1b4/0x220 [13201.992608] nvme ptp [13201.994621] ? __hrtimer_run_queues+0x1d3/0x360 [13201.997682] nvme_core i2c_algo_bit [13202.001946] ? hrtimer_interrupt+0x104/0x240 [13202.004213] pps_core dca [13202.008737] ? __local_bh_enable+0x50/0x60 [13202.012221] fuse [13202.016488] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13202.019103] irq event stamp: 28936928 [13202.019104] hardirqs last enabled at (28936927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13202.023196] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13202.025118] hardirqs last disabled at (28936928): [] sysvec_apic_timer_interrupt+0xe/0x90 [13202.030337] [13202.033994] softirqs last enabled at (28936926): [] __do_softirq+0x2be/0x371 [13202.044151] [13202.049109] softirqs last disabled at (28936921): [] irq_exit_rcu+0xb0/0xd0 [13202.058834] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13202.060930] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13202.069619] ? queued_spin_lock_slowpath+0x18/0x2e0 [13202.071710] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13202.080224] do_raw_spin_lock+0xb5/0xc0 [13202.085526] Workqueue: events delayed_work [ceph] [13202.094905] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13202.099775] [13202.107248] try_get_cap_refs+0x159/0x880 [ceph] [13202.111076] RIP: 0010:lock_acquire+0xf0/0x2b0 [13202.115778] __ceph_get_caps+0x4a1/0x760 [ceph] [13202.120558] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13202.122053] ? __file_remove_privs+0x89/0x110 [13202.126672] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13202.131019] ? wake_bit_function+0x70/0x70 [13202.135542] [13202.135543] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13202.154283] ? ceph_write_iter+0x260/0x770 [ceph] [13202.158630] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13202.163849] ceph_write_iter+0x260/0x770 [ceph] [13202.167940] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13202.169434] ? lock_is_held_type+0xaa/0x120 [13202.176553] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13202.181254] vfs_write+0x324/0x4f0 [13202.188391] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13202.192922] ksys_pwrite64+0x65/0xa0 [13202.200041] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13202.204219] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13202.211343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13202.214740] do_syscall_64+0x38/0x80 [13202.221864] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13202.225435] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13202.233513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13202.238644] RIP: 0033:0x7f48f0eeb93b [13202.244393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13202.247968] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13202.255093] Call Trace: [13202.255094] [13202.260135] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13202.267262] ? watchdog_timer_fn+0x1b4/0x220 [13202.270830] ORIG_RAX: 0000000000000012 [13202.277958] ? __hrtimer_run_queues+0x1d3/0x360 [13202.296694] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13202.299142] ? hrtimer_interrupt+0x104/0x240 [13202.301149] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13202.306367] ? __local_bh_enable+0x50/0x60 [13202.310630] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13202.314464] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13202.318985] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13202.326108] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13202.330374] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13202.337514] [13202.337515] [13202.341611] [13202.348729] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13202.384839] ? lock_acquire+0xf0/0x2b0 [13202.388589] ? igrab+0x43/0x50 [13202.391640] ? lock_release+0xc5/0x260 [13202.395399] _raw_spin_lock+0x30/0x40 [13202.399059] ? igrab+0x1d/0x50 [13202.402108] igrab+0x1d/0x50 [13202.404989] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13202.410138] delayed_work+0x1d8/0x280 [ceph] [13202.414418] ? process_scheduled_works+0x22a/0x510 [13202.419207] process_scheduled_works+0x2a4/0x510 [13202.423822] worker_thread+0x188/0x340 [13202.427574] ? rescuer_thread+0x310/0x310 [13202.431584] kthread+0xf5/0x130 [13202.434722] ? kthread_complete_and_exit+0x20/0x20 [13202.439505] ret_from_fork+0x31/0x40 [13202.443075] ? kthread_complete_and_exit+0x20/0x20 [13202.447863] ret_from_fork_asm+0x11/0x20 [13202.451794] [13229.686482] watchdog: BUG: soft lockup - CPU#0 stuck for 1207s! [dbench:78683] [13229.693701] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13229.698482] watchdog: BUG: soft lockup - CPU#7 stuck for 10204s! [kworker/7:4:78808] [13229.749156] irq event stamp: 15566128 [13229.749157] hardirqs last enabled at (15566127): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13229.756891] Modules linked in: [13229.760548] hardirqs last disabled at (15566128): [] sysvec_apic_timer_interrupt+0xe/0x90 [13229.770705] ceph [13229.773754] softirqs last enabled at (15566090): [] __do_softirq+0x2be/0x371 [13229.783480] libceph dns_resolver [13229.785403] softirqs last disabled at (15566085): [] irq_exit_rcu+0xb0/0xd0 [13229.794088] fscache netfs [13229.797400] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13229.805910] veth [13229.808615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13229.817557] nft_chain_nat [13229.819483] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13229.826952] xt_MASQUERADE nf_nat [13229.829657] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13229.835133] nf_conntrack [13229.838461] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13229.857200] nf_defrag_ipv6 [13229.859817] [13229.865034] nf_defrag_ipv4 [13229.867825] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13229.869316] nft_compat nf_tables [13229.872106] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13229.879231] nfnetlink bridge [13229.882541] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13229.889667] stp llc [13229.892629] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13229.899754] xfs [13229.901939] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13229.909063] libcrc32c [13229.910900] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13229.918025] sunrpc intel_rapl_msr [13229.920382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13229.928459] intel_rapl_common [13229.931857] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13229.937610] sb_edac x86_pkg_temp_thermal [13229.940662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13229.947787] coretemp [13229.951789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13229.958914] kvm_intel [13229.961183] Call Trace: [13229.961185] [13229.968308] kvm [13229.970670] ? watchdog_timer_fn+0x1b4/0x220 [13229.973109] irqbypass i2c_i801 [13229.975122] ? __hrtimer_run_queues+0x1d3/0x360 [13229.976957] iTCO_wdt iTCO_vendor_support [13229.981227] ? hrtimer_interrupt+0x104/0x240 [13229.984359] i2c_smbus crct10dif_pclmul [13229.988885] ? __local_bh_enable+0x50/0x60 [13229.992888] joydev crc32_pclmul [13229.997170] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13230.001000] ipmi_ssif ghash_clmulni_intel [13230.005092] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13230.008316] mei_me mei [13230.013531] [13230.017623] lpc_ich [13230.022579] [13230.022581] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13230.025024] sha512_ssse3 acpi_ipmi [13230.027126] ? queued_spin_lock_slowpath+0x16/0x2e0 [13230.029305] mfd_core [13230.031404] do_raw_spin_lock+0xb5/0xc0 [13230.036707] wmi ipmi_si [13230.040194] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13230.045061] ipmi_devintf ipmi_msghandler ioatdma [13230.047337] try_get_cap_refs+0x159/0x880 [ceph] [13230.051174] acpi_pad acpi_power_meter [13230.053710] __ceph_get_caps+0x4a1/0x760 [ceph] [13230.058488] ixgbe igb [13230.063188] ? __file_remove_privs+0x89/0x110 [13230.067795] mdio crc32c_intel [13230.071539] ? wake_bit_function+0x70/0x70 [13230.076062] nvme ptp [13230.078423] ? ceph_write_iter+0x260/0x770 [ceph] [13230.082771] nvme_core [13230.085822] ceph_write_iter+0x260/0x770 [ceph] [13230.089911] i2c_algo_bit [13230.092188] ? lock_is_held_type+0xaa/0x120 [13230.096881] pps_core dca [13230.099241] vfs_write+0x324/0x4f0 [13230.103763] fuse [13230.106385] ksys_pwrite64+0x65/0xa0 [13230.110557] irq event stamp: 29158780 [13230.113181] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13230.116581] hardirqs last enabled at (29158779): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13230.118506] do_syscall_64+0x38/0x80 [13230.122074] hardirqs last disabled at (29158780): [] sysvec_apic_timer_interrupt+0xe/0x90 [13230.125736] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13230.130881] softirqs last enabled at (29158778): [] __do_softirq+0x2be/0x371 [13230.141039] RIP: 0033:0x7f48f0eeb93b [13230.144608] softirqs last disabled at (29158773): [] irq_exit_rcu+0xb0/0xd0 [13230.154334] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13230.159378] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13230.168061] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13230.171631] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13230.171633] Workqueue: events delayed_work [ceph] [13230.180160] ORIG_RAX: 0000000000000012 [13230.198899] [13230.198900] RIP: 0010:ceph_check_delayed_caps+0x63/0x270 [ceph] [13230.208275] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13230.213494] Code: 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 <48> 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 [13230.220963] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13230.225662] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [13230.229492] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13230.230983] [13230.230984] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [13230.236893] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13230.244018] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [13230.262755] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13230.269885] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [13230.275112] [13230.282231] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13230.328649] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13230.335773] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13230.343850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13230.349589] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13230.356712] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13230.363836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13230.370961] Call Trace: [13230.373407] [13230.375416] ? watchdog_timer_fn+0x1b4/0x220 [13230.379682] ? __hrtimer_run_queues+0x1d3/0x360 [13230.384206] ? hrtimer_interrupt+0x104/0x240 [13230.388469] ? __local_bh_enable+0x50/0x60 [13230.392561] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13230.397786] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13230.402744] [13230.404841] [13230.406936] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13230.412247] ? ceph_check_delayed_caps+0x63/0x270 [ceph] [13230.417564] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [13230.422889] delayed_work+0x1d8/0x280 [ceph] [13230.427171] ? process_scheduled_works+0x22a/0x510 [13230.431957] process_scheduled_works+0x2a4/0x510 [13230.436574] worker_thread+0x188/0x340 [13230.440323] ? rescuer_thread+0x310/0x310 [13230.444327] kthread+0xf5/0x130 [13230.447474] ? kthread_complete_and_exit+0x20/0x20 [13230.452258] ret_from_fork+0x31/0x40 [13230.455837] ? kthread_complete_and_exit+0x20/0x20 [13230.460621] ret_from_fork_asm+0x11/0x20 [13230.464551] [13257.686974] watchdog: BUG: soft lockup - CPU#0 stuck for 1233s! [dbench:78683] [13257.694187] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13257.698974] watchdog: BUG: soft lockup - CPU#7 stuck for 10230s! [kworker/7:4:78808] [13257.749645] irq event stamp: 15624164 [13257.749646] hardirqs last enabled at (15624163): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13257.757378] Modules linked in: [13257.761034] hardirqs last disabled at (15624164): [] sysvec_apic_timer_interrupt+0xe/0x90 [13257.771191] ceph [13257.774243] softirqs last enabled at (15624126): [] __do_softirq+0x2be/0x371 [13257.783968] libceph dns_resolver [13257.785891] softirqs last disabled at (15624121): [] irq_exit_rcu+0xb0/0xd0 [13257.794575] fscache [13257.797886] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13257.806396] netfs veth [13257.808582] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13257.817523] nft_chain_nat xt_MASQUERADE [13257.819970] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13257.827439] nf_nat nf_conntrack [13257.831369] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13257.836844] nf_defrag_ipv6 nf_defrag_ipv4 [13257.840066] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13257.858805] nft_compat nf_tables [13257.862896] [13257.862897] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13257.868112] nfnetlink bridge [13257.871424] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13257.872914] stp llc [13257.880039] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13257.883002] xfs libcrc32c [13257.890127] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13257.892311] sunrpc intel_rapl_msr [13257.899434] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13257.902140] intel_rapl_common [13257.909262] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13257.912659] sb_edac x86_pkg_temp_thermal [13257.919784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13257.922835] coretemp kvm_intel [13257.930914] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13257.934917] kvm irqbypass [13257.940653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13257.943791] i2c_i801 iTCO_wdt [13257.950916] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13257.953619] iTCO_vendor_support [13257.960744] Call Trace: [13257.960746] [13257.963796] i2c_smbus [13257.970922] ? watchdog_timer_fn+0x1b4/0x220 [13257.974142] crct10dif_pclmul joydev [13257.976588] ? __hrtimer_run_queues+0x1d3/0x360 [13257.978598] crc32_pclmul ipmi_ssif [13257.980959] ? hrtimer_interrupt+0x104/0x240 [13257.985219] ghash_clmulni_intel [13257.988790] ? __local_bh_enable+0x50/0x60 [13257.993314] mei_me mei [13257.996801] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13258.001062] lpc_ich sha512_ssse3 [13258.004286] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13258.008378] acpi_ipmi mfd_core [13258.010820] [13258.016039] wmi [13258.019348] [13258.019350] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13258.024324] ipmi_si [13258.027466] ? queued_spin_lock_slowpath+0x18/0x2e0 [13258.029558] ipmi_devintf ipmi_msghandler [13258.031399] do_raw_spin_lock+0xb5/0xc0 [13258.033494] ioatdma acpi_pad [13258.038800] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13258.040981] acpi_power_meter ixgbe [13258.045855] try_get_cap_refs+0x159/0x880 [ceph] [13258.049857] igb mdio [13258.053691] __ceph_get_caps+0x4a1/0x760 [ceph] [13258.056653] crc32c_intel nvme [13258.061442] ? __file_remove_privs+0x89/0x110 [13258.064920] ptp nvme_core [13258.069529] ? wake_bit_function+0x70/0x70 [13258.071801] i2c_algo_bit pps_core [13258.076328] ? ceph_write_iter+0x260/0x770 [ceph] [13258.079376] dca fuse [13258.083726] ceph_write_iter+0x260/0x770 [ceph] [13258.086429] [13258.086430] irq event stamp: 29380862 [13258.090524] ? lock_is_held_type+0xaa/0x120 [13258.093919] hardirqs last enabled at (29380861): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13258.098619] vfs_write+0x324/0x4f0 [13258.100887] hardirqs last disabled at (29380862): [] sysvec_apic_timer_interrupt+0xe/0x90 [13258.105416] ksys_pwrite64+0x65/0xa0 [13258.106901] softirqs last enabled at (29380860): [] __do_softirq+0x2be/0x371 [13258.110559] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13258.114735] softirqs last disabled at (29380855): [] irq_exit_rcu+0xb0/0xd0 [13258.124894] do_syscall_64+0x38/0x80 [13258.128291] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13258.138016] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13258.141586] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13258.150272] RIP: 0033:0x7f48f0eeb93b [13258.155403] Workqueue: events delayed_work [ceph] [13258.163912] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13258.167481] [13258.167482] RIP: 0010:lock_acquire+0xf0/0x2b0 [13258.176860] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13258.181904] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13258.189375] ORIG_RAX: 0000000000000012 [13258.192944] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13258.197658] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13258.216397] [13258.217888] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13258.222239] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13258.227456] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13258.246195] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13258.250026] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13258.255242] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13258.262379] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13258.263865] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13258.270997] [13258.278115] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13258.330185] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13258.338263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13258.344001] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13258.351124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13258.358249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13258.365380] Call Trace: [13258.367827] [13258.369837] ? watchdog_timer_fn+0x1b4/0x220 [13258.374102] ? __hrtimer_run_queues+0x1d3/0x360 [13258.378628] ? hrtimer_interrupt+0x104/0x240 [13258.382896] ? __local_bh_enable+0x50/0x60 [13258.386989] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13258.392205] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13258.397161] [13258.399261] [13258.401375] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13258.406684] ? lock_acquire+0xf0/0x2b0 [13258.410432] ? igrab+0x43/0x50 [13258.413482] ? lock_release+0xc5/0x260 [13258.417228] _raw_spin_lock+0x30/0x40 [13258.420894] ? igrab+0x1d/0x50 [13258.423942] igrab+0x1d/0x50 [13258.426821] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13258.431971] delayed_work+0x1d8/0x280 [ceph] [13258.436253] ? process_scheduled_works+0x22a/0x510 [13258.441043] process_scheduled_works+0x2a4/0x510 [13258.445667] worker_thread+0x188/0x340 [13258.449415] ? rescuer_thread+0x310/0x310 [13258.453419] kthread+0xf5/0x130 [13258.456557] ? kthread_complete_and_exit+0x20/0x20 [13258.461341] ret_from_fork+0x31/0x40 [13258.464922] ? kthread_complete_and_exit+0x20/0x20 [13258.469703] ret_from_fork_asm+0x11/0x20 [13258.473629] [13285.687466] watchdog: BUG: soft lockup - CPU#0 stuck for 1259s! [dbench:78683] [13285.694683] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13285.699465] watchdog: BUG: soft lockup - CPU#7 stuck for 10256s! [kworker/7:4:78808] [13285.750140] irq event stamp: 15682208 [13285.757873] Modules linked in: ceph libceph dns_resolver fscache netfs veth [13285.761529] hardirqs last enabled at (15682207): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13285.768490] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [13285.778648] hardirqs last disabled at (15682208): [] sysvec_apic_timer_interrupt+0xe/0x90 [13285.785608] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [13285.795331] softirqs last enabled at (15682170): [] __do_softirq+0x2be/0x371 [13285.795335] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [13285.810453] softirqs last disabled at (15682165): [] irq_exit_rcu+0xb0/0xd0 [13285.810457] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [13285.826176] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13285.832347] i2c_i801 [13285.841290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13285.841292] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13285.843562] iTCO_wdt iTCO_vendor_support [13285.851034] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13285.856510] i2c_smbus [13285.860513] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13285.879253] crct10dif_pclmul [13285.881609] [13285.881610] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13285.886827] joydev crc32_pclmul [13285.889789] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13285.891280] ipmi_ssif ghash_clmulni_intel [13285.898406] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13285.901630] mei_me mei [13285.908754] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13285.912860] lpc_ich [13285.919985] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13285.922430] sha512_ssse3 [13285.929553] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13285.931738] acpi_ipmi [13285.938864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13285.941481] mfd_core [13285.949556] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13285.951914] wmi ipmi_si [13285.957653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13285.959922] ipmi_devintf [13285.967048] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13285.969577] ipmi_msghandler [13285.976700] Call Trace: [13285.976702] [13285.979319] ioatdma acpi_pad [13285.986447] ? watchdog_timer_fn+0x1b4/0x220 [13285.989321] acpi_power_meter ixgbe [13285.991766] ? __hrtimer_run_queues+0x1d3/0x360 [13285.993774] igb mdio [13285.996744] ? hrtimer_interrupt+0x104/0x240 [13286.001003] crc32c_intel nvme [13286.004487] ? __local_bh_enable+0x50/0x60 [13286.009010] ptp nvme_core [13286.011284] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13286.015547] i2c_algo_bit pps_core [13286.018598] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13286.022689] dca [13286.025393] [13286.030609] fuse [13286.034007] [13286.034009] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13286.038964] irq event stamp: 29603906 [13286.040805] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13286.042899] hardirqs last enabled at (29603905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13286.044826] do_raw_spin_lock+0xb5/0xc0 [13286.046919] hardirqs last disabled at (29603906): [] sysvec_apic_timer_interrupt+0xe/0x90 [13286.052227] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13286.055880] softirqs last enabled at (29603904): [] __do_softirq+0x2be/0x371 [13286.060755] try_get_cap_refs+0x159/0x880 [ceph] [13286.070910] softirqs last disabled at (29603899): [] irq_exit_rcu+0xb0/0xd0 [13286.074745] __ceph_get_caps+0x4a1/0x760 [ceph] [13286.084464] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13286.089255] ? __file_remove_privs+0x89/0x110 [13286.097932] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13286.102545] ? wake_bit_function+0x70/0x70 [13286.111055] Workqueue: events delayed_work [ceph] [13286.115580] ? ceph_write_iter+0x260/0x770 [ceph] [13286.124957] [13286.129309] ceph_write_iter+0x260/0x770 [ceph] [13286.136777] RIP: 0010:lock_acquire+0xf0/0x2b0 [13286.140873] ? lock_is_held_type+0xaa/0x120 [13286.145565] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13286.150267] vfs_write+0x324/0x4f0 [13286.151754] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13286.156284] ksys_pwrite64+0x65/0xa0 [13286.160628] [13286.164806] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13286.183543] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13286.186943] do_syscall_64+0x38/0x80 [13286.192160] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13286.195730] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13286.197219] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13286.202351] RIP: 0033:0x7f48f0eeb93b [13286.209476] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13286.213046] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13286.220169] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13286.225214] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13286.232338] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13286.235909] ORIG_RAX: 0000000000000012 [13286.243034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13286.261771] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13286.268895] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13286.274111] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13286.282189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13286.286020] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13286.291758] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13286.298882] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13286.306006] Call Trace: [13286.306008] [13286.313130] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13286.320254] ? watchdog_timer_fn+0x1b4/0x220 [13286.327385] [13286.334505] ? __hrtimer_run_queues+0x1d3/0x360 [13286.364182] ? hrtimer_interrupt+0x104/0x240 [13286.368450] ? __local_bh_enable+0x50/0x60 [13286.372544] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13286.377767] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13286.382726] [13286.384822] [13286.386922] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13286.392228] ? lock_acquire+0xf0/0x2b0 [13286.395977] ? igrab+0x43/0x50 [13286.399027] ? lock_release+0xc5/0x260 [13286.402775] _raw_spin_lock+0x30/0x40 [13286.406438] ? igrab+0x1d/0x50 [13286.409490] igrab+0x1d/0x50 [13286.412367] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13286.417517] delayed_work+0x1d8/0x280 [ceph] [13286.421799] ? process_scheduled_works+0x22a/0x510 [13286.426588] process_scheduled_works+0x2a4/0x510 [13286.431203] worker_thread+0x188/0x340 [13286.434955] ? rescuer_thread+0x310/0x310 [13286.438965] kthread+0xf5/0x130 [13286.442111] ? kthread_complete_and_exit+0x20/0x20 [13286.446895] ret_from_fork+0x31/0x40 [13286.450475] ? kthread_complete_and_exit+0x20/0x20 [13286.455259] ret_from_fork_asm+0x11/0x20 [13286.459190] [13313.687957] watchdog: BUG: soft lockup - CPU#0 stuck for 1285s! [dbench:78683] [13313.695170] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13313.699957] watchdog: BUG: soft lockup - CPU#7 stuck for 10282s! [kworker/7:4:78808] [13313.750637] irq event stamp: 15740126 [13313.750638] hardirqs last enabled at (15740125): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13313.758369] Modules linked in: [13313.762025] hardirqs last disabled at (15740126): [] sysvec_apic_timer_interrupt+0xe/0x90 [13313.772183] ceph [13313.775234] softirqs last enabled at (15740116): [] __do_softirq+0x2be/0x371 [13313.784959] libceph [13313.786881] softirqs last disabled at (15740111): [] irq_exit_rcu+0xb0/0xd0 [13313.795566] dns_resolver fscache [13313.797750] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13313.806262] netfs veth [13313.809572] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13313.818517] nft_chain_nat [13313.820961] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13313.828430] xt_MASQUERADE nf_nat [13313.831136] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13313.836613] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13313.839941] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13313.858679] nft_compat nf_tables [13313.863897] [13313.869114] nfnetlink bridge [13313.872424] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13313.873913] stp llc [13313.876879] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13313.884003] xfs [13313.886187] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13313.893311] libcrc32c [13313.895148] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13313.902272] sunrpc [13313.904629] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13313.911771] intel_rapl_msr intel_rapl_common [13313.913870] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13313.920992] sb_edac x86_pkg_temp_thermal [13313.925353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13313.933430] coretemp kvm_intel [13313.937434] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13313.943171] kvm irqbypass [13313.946309] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13313.953433] i2c_i801 iTCO_wdt [13313.956136] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13313.963262] iTCO_vendor_support i2c_smbus [13313.966311] Call Trace: [13313.966312] [13313.973436] crct10dif_pclmul [13313.977529] ? watchdog_timer_fn+0x1b4/0x220 [13313.979970] joydev crc32_pclmul [13313.981983] ? __hrtimer_run_queues+0x1d3/0x360 [13313.984947] ipmi_ssif ghash_clmulni_intel [13313.989213] ? hrtimer_interrupt+0x104/0x240 [13313.992433] mei_me mei [13313.996957] ? __local_bh_enable+0x50/0x60 [13314.001049] lpc_ich sha512_ssse3 [13314.005315] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13314.007757] acpi_ipmi [13314.011849] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13314.015158] mfd_core wmi [13314.020376] [13314.022733] ipmi_si [13314.027691] [13314.030307] ipmi_devintf [13314.032406] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13314.034590] ipmi_msghandler [13314.036692] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13314.039305] ioatdma acpi_pad [13314.044611] do_raw_spin_lock+0xb5/0xc0 [13314.047486] acpi_power_meter ixgbe [13314.052357] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13314.055321] igb mdio [13314.059154] try_get_cap_refs+0x159/0x880 [ceph] [13314.062637] crc32c_intel nvme [13314.067425] __ceph_get_caps+0x4a1/0x760 [ceph] [13314.069690] ptp nvme_core [13314.074306] ? __file_remove_privs+0x89/0x110 [13314.077357] i2c_algo_bit [13314.081885] ? wake_bit_function+0x70/0x70 [13314.084588] pps_core dca [13314.088940] ? ceph_write_iter+0x260/0x770 [ceph] [13314.091555] fuse [13314.095647] ceph_write_iter+0x260/0x770 [ceph] [13314.098264] irq event stamp: 29827160 [13314.102965] ? lock_is_held_type+0xaa/0x120 [13314.104885] hardirqs last enabled at (29827159): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13314.109411] vfs_write+0x324/0x4f0 [13314.113066] hardirqs last disabled at (29827160): [] sysvec_apic_timer_interrupt+0xe/0x90 [13314.117250] ksys_pwrite64+0x65/0xa0 [13314.127403] softirqs last enabled at (29827158): [] __do_softirq+0x2be/0x371 [13314.130800] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13314.140523] softirqs last disabled at (29827153): [] irq_exit_rcu+0xb0/0xd0 [13314.144095] do_syscall_64+0x38/0x80 [13314.152779] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13314.157909] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13314.166421] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13314.169992] RIP: 0033:0x7f48f0eeb93b [13314.179369] Workqueue: events delayed_work [ceph] [13314.184413] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13314.191883] [13314.191884] RIP: 0010:rcu_is_watching+0xb/0x40 [13314.195453] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13314.200151] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [13314.218889] ORIG_RAX: 0000000000000012 [13314.220379] RSP: 0018:ffffc90002117d10 EFLAGS: 00000207 [13314.224818] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13314.230036] [13314.230036] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13314.248773] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13314.252604] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13314.257820] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13314.264944] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [13314.266436] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13314.273560] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [13314.280684] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13314.287809] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13314.294938] [13314.302057] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13314.340815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13314.346551] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13314.353675] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13314.360799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13314.367924] Call Trace: [13314.370369] [13314.372380] ? watchdog_timer_fn+0x1b4/0x220 [13314.376644] ? __hrtimer_run_queues+0x1d3/0x360 [13314.381169] ? hrtimer_interrupt+0x104/0x240 [13314.385431] ? __local_bh_enable+0x50/0x60 [13314.389525] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13314.394748] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13314.399705] [13314.401801] [13314.403902] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13314.409208] ? rcu_is_watching+0xb/0x40 [13314.413045] lock_acquire+0x184/0x2b0 [13314.416701] ? igrab+0x43/0x50 [13314.419752] ? lock_release+0xc5/0x260 [13314.423498] _raw_spin_lock+0x30/0x40 [13314.427163] ? igrab+0x1d/0x50 [13314.430214] igrab+0x1d/0x50 [13314.433093] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13314.438242] delayed_work+0x1d8/0x280 [ceph] [13314.442523] ? process_scheduled_works+0x22a/0x510 [13314.447313] process_scheduled_works+0x2a4/0x510 [13314.451928] worker_thread+0x188/0x340 [13314.455678] ? rescuer_thread+0x310/0x310 [13314.459690] kthread+0xf5/0x130 [13314.462837] ? kthread_complete_and_exit+0x20/0x20 [13314.467645] ret_from_fork+0x31/0x40 [13314.471227] ? kthread_complete_and_exit+0x20/0x20 [13314.476016] ret_from_fork_asm+0x11/0x20 [13314.479939] [13341.688449] watchdog: BUG: soft lockup - CPU#0 stuck for 1311s! [dbench:78683] [13341.695667] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13341.700448] watchdog: BUG: soft lockup - CPU#7 stuck for 10308s! [kworker/7:4:78808] [13341.751132] irq event stamp: 15798132 [13341.758865] Modules linked in: ceph libceph dns_resolver fscache netfs [13341.762522] hardirqs last enabled at (15798131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13341.769048] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [13341.779205] hardirqs last disabled at (15798132): [] sysvec_apic_timer_interrupt+0xe/0x90 [13341.786588] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [13341.796315] softirqs last enabled at (15798094): [] __do_softirq+0x2be/0x371 [13341.803100] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [13341.811784] softirqs last disabled at (15798089): [] irq_exit_rcu+0xb0/0xd0 [13341.820469] coretemp kvm_intel kvm [13341.828979] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13341.832463] irqbypass i2c_i801 [13341.841407] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13341.844543] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [13341.852017] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13341.860969] ghash_clmulni_intel mei_me mei [13341.866448] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13341.870623] lpc_ich [13341.889360] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13341.889360] sha512_ssse3 acpi_ipmi [13341.891545] [13341.896761] mfd_core [13341.900247] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13341.901736] wmi ipmi_si [13341.904008] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13341.911132] ipmi_devintf [13341.913662] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13341.920787] ipmi_msghandler [13341.923403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13341.930529] ioatdma acpi_pad [13341.933407] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13341.940531] acpi_power_meter [13341.943494] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13341.950617] ixgbe igb [13341.953581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13341.961659] mdio crc32c_intel [13341.964017] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13341.969755] nvme [13341.972805] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13341.979930] ptp nvme_core [13341.981854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13341.988978] i2c_algo_bit [13341.991682] Call Trace: [13341.998805] pps_core [13342.001426] [13342.003866] dca [13342.006139] ? watchdog_timer_fn+0x1b4/0x220 [13342.008149] fuse [13342.009986] ? __hrtimer_run_queues+0x1d3/0x360 [13342.014251] irq event stamp: 30049418 [13342.016178] ? hrtimer_interrupt+0x104/0x240 [13342.020699] hardirqs last enabled at (30049417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13342.024356] ? __local_bh_enable+0x50/0x60 [13342.028619] hardirqs last disabled at (30049418): [] sysvec_apic_timer_interrupt+0xe/0x90 [13342.038779] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13342.042867] softirqs last enabled at (30049416): [] __do_softirq+0x2be/0x371 [13342.052594] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13342.057808] softirqs last disabled at (30049411): [] irq_exit_rcu+0xb0/0xd0 [13342.066494] [13342.071452] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13342.079963] [13342.082059] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13342.091437] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13342.093536] Workqueue: events delayed_work [ceph] [13342.101008] ? queued_spin_lock_slowpath+0x16/0x2e0 [13342.106309] [13342.106310] RIP: 0010:lock_acquire+0x85/0x2b0 [13342.111007] do_raw_spin_lock+0xb5/0xc0 [13342.115877] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [13342.117370] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13342.121717] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [13342.125552] try_get_cap_refs+0x159/0x880 [ceph] [13342.144286] [13342.144287] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13342.149076] __ceph_get_caps+0x4a1/0x760 [ceph] [13342.154288] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13342.158904] ? __file_remove_privs+0x89/0x110 [13342.160389] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [13342.167514] ? wake_bit_function+0x70/0x70 [13342.172038] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [13342.179165] ? ceph_write_iter+0x260/0x770 [ceph] [13342.183514] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13342.190637] ceph_write_iter+0x260/0x770 [ceph] [13342.194728] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13342.201856] ? lock_is_held_type+0xaa/0x120 [13342.206549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13342.213675] vfs_write+0x324/0x4f0 [13342.218196] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13342.226282] ksys_pwrite64+0x65/0xa0 [13342.230453] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13342.236190] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13342.239588] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13342.246713] do_syscall_64+0x38/0x80 [13342.250282] Call Trace: [13342.250284] [13342.257407] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13342.262538] ? watchdog_timer_fn+0x1b4/0x220 [13342.269662] RIP: 0033:0x7f48f0eeb93b [13342.273233] ? __hrtimer_run_queues+0x1d3/0x360 [13342.275679] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13342.277691] ? hrtimer_interrupt+0x104/0x240 [13342.282729] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13342.286994] ? __local_bh_enable+0x50/0x60 [13342.290566] ORIG_RAX: 0000000000000012 [13342.295094] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13342.313828] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13342.318109] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13342.323327] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13342.327418] [13342.331248] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13342.336465] [13342.336467] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13342.343589] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13342.348552] ? lock_acquire+0x85/0x2b0 [13342.355671] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13342.357771] ? igrab+0x43/0x50 [13342.364898] [13342.366991] ? lock_release+0xc5/0x260 [13342.399268] _raw_spin_lock+0x30/0x40 [13342.402932] ? igrab+0x1d/0x50 [13342.405983] igrab+0x1d/0x50 [13342.408863] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13342.414010] delayed_work+0x1d8/0x280 [ceph] [13342.418291] ? process_scheduled_works+0x22a/0x510 [13342.423081] process_scheduled_works+0x2a4/0x510 [13342.427698] worker_thread+0x188/0x340 [13342.431446] ? rescuer_thread+0x310/0x310 [13342.435448] kthread+0xf5/0x130 [13342.438589] ? kthread_complete_and_exit+0x20/0x20 [13342.443382] ret_from_fork+0x31/0x40 [13342.446961] ? kthread_complete_and_exit+0x20/0x20 [13342.451752] ret_from_fork_asm+0x11/0x20 [13342.455684] [13369.688940] watchdog: BUG: soft lockup - CPU#0 stuck for 1337s! [dbench:78683] [13369.696155] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13369.700939] watchdog: BUG: soft lockup - CPU#7 stuck for 10334s! [kworker/7:4:78808] [13369.751595] irq event stamp: 15855948 [13369.759335] Modules linked in: ceph libceph [13369.762991] hardirqs last enabled at (15855947): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13369.767169] dns_resolver fscache netfs [13369.777335] hardirqs last disabled at (15855948): [] sysvec_apic_timer_interrupt+0xe/0x90 [13369.781166] veth nft_chain_nat xt_MASQUERADE nf_nat [13369.790890] softirqs last enabled at (15855910): [] __do_softirq+0x2be/0x371 [13369.795849] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13369.804533] softirqs last disabled at (15855905): [] irq_exit_rcu+0xb0/0xd0 [13369.810704] nf_tables nfnetlink bridge [13369.819215] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13369.823045] stp llc [13369.831987] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13369.834173] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [13369.841643] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13369.847815] sb_edac x86_pkg_temp_thermal [13369.853291] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13369.853292] coretemp kvm_intel kvm [13369.876034] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13369.879518] irqbypass i2c_i801 [13369.884735] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13369.884736] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13369.887872] iTCO_wdt iTCO_vendor_support [13369.894996] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13369.902120] i2c_smbus crct10dif_pclmul [13369.906125] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13369.913249] joydev crc32_pclmul [13369.917078] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13369.924204] ipmi_ssif ghash_clmulni_intel [13369.927428] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13369.934551] mei_me mei [13369.938643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13369.946721] lpc_ich [13369.949164] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13369.954902] sha512_ssse3 [13369.957084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13369.964209] acpi_ipmi mfd_core [13369.966826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13369.973952] wmi ipmi_si [13369.977088] Call Trace: [13369.977091] [13369.984212] ipmi_devintf [13369.986744] ? watchdog_timer_fn+0x1b4/0x220 [13369.989187] ipmi_msghandler [13369.991201] ? __hrtimer_run_queues+0x1d3/0x360 [13369.993816] ioatdma acpi_pad [13369.998083] ? hrtimer_interrupt+0x104/0x240 [13370.000957] acpi_power_meter ixgbe [13370.005481] ? __local_bh_enable+0x50/0x60 [13370.008446] igb mdio [13370.012712] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13370.016194] crc32c_intel nvme [13370.020283] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13370.022556] ptp nvme_core i2c_algo_bit [13370.027773] [13370.030824] pps_core dca [13370.035781] [13370.039620] fuse [13370.041718] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13370.044339] irq event stamp: 30271198 [13370.046445] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13370.048364] hardirqs last enabled at (30271197): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13370.053670] do_raw_spin_lock+0xb5/0xc0 [13370.057340] hardirqs last disabled at (30271198): [] sysvec_apic_timer_interrupt+0xe/0x90 [13370.062215] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13370.072370] softirqs last enabled at (30271196): [] __do_softirq+0x2be/0x371 [13370.076204] try_get_cap_refs+0x159/0x880 [ceph] [13370.085314] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [13370.085317] rcu: 0-....: (1403221 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=675107 [13370.085339] rcu: 7-....: (1397028 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/1395880 fqs=675107 [13370.085342] rcu: (detected by 4, t=1433434 jiffies, g=1309609, q=124812 ncpus=8) [13370.085345] Sending NMI from CPU 4 to CPUs 0: [13370.085348] NMI backtrace for cpu 0 [13370.085349] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13370.085351] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13370.085352] RIP: 0010:vprintk_emit+0x1ee/0x340 [13370.085356] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [13370.085358] RSP: 0018:ffffc90000003d30 EFLAGS: 00000002 [13370.085360] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [13370.085361] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [13370.085362] RBP: 0000000000000024 R08: 0000000000005ef4 R09: 0000000000000004 [13370.085363] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [13370.085365] R13: 0000000000000000 R14: ffffffff82123e9c R15: ffffc90000003d80 [13370.085366] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13370.085368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13370.085369] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13370.085370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13370.085371] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13370.085372] Call Trace: [13370.085373] [13370.085374] ? nmi_cpu_backtrace+0xad/0x120 [13370.085379] ? nmi_cpu_backtrace_handler+0x11/0x20 [13370.085381] ? nmi_handle+0xd4/0x260 [13370.085386] ? default_do_nmi+0x49/0xf0 [13370.085389] ? exc_nmi+0x11e/0x150 [13370.085391] ? end_repeat_nmi+0x16/0x67 [13370.085398] ? vprintk_emit+0x1ee/0x340 [13370.085400] ? vprintk_emit+0x1ee/0x340 [13370.085404] ? vprintk_emit+0x1ee/0x340 [13370.085406] [13370.085407] [13370.085408] ? try_get_cap_refs+0x159/0x880 [ceph] [13370.085426] ? try_get_cap_refs+0x159/0x880 [ceph] [13370.085441] ? try_get_cap_refs+0x159/0x880 [ceph] [13370.085456] _printk+0x5c/0x80 [13370.085459] ? is_module_text_address+0x15/0x30 [13370.085462] ? try_get_cap_refs+0x159/0x880 [ceph] [13370.085477] show_trace_log_lvl+0x264/0x3f0 [13370.085485] ? try_get_cap_refs+0x159/0x880 [ceph] [13370.085501] ? touch_softlockup_watchdog+0x20/0x20 [13370.085503] watchdog_timer_fn+0x1b4/0x220 [13370.085506] __hrtimer_run_queues+0x1d3/0x360 [13370.085511] hrtimer_interrupt+0x104/0x240 [13370.085513] ? __local_bh_enable+0x50/0x60 [13370.085517] __sysvec_apic_timer_interrupt+0x61/0x160 [13370.085521] sysvec_apic_timer_interrupt+0x6d/0x90 [13370.085523] [13370.085524] [13370.085525] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13370.085527] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13370.085530] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13370.085532] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13370.085534] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13370.085535] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13370.085536] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13370.085537] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13370.085538] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13370.085545] do_raw_spin_lock+0xb5/0xc0 [13370.085547] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13370.085565] try_get_cap_refs+0x159/0x880 [ceph] [13370.085584] __ceph_get_caps+0x4a1/0x760 [ceph] [13370.085603] ? __file_remove_privs+0x89/0x110 [13370.085606] ? wake_bit_function+0x70/0x70 [13370.085610] ? ceph_write_iter+0x260/0x770 [ceph] [13370.085626] ceph_write_iter+0x260/0x770 [ceph] [13370.085644] ? lock_is_held_type+0xaa/0x120 [13370.085648] vfs_write+0x324/0x4f0 [13370.085655] ksys_pwrite64+0x65/0xa0 [13370.085658] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13370.085663] do_syscall_64+0x38/0x80 [13370.085666] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13370.085668] RIP: 0033:0x7f48f0eeb93b [13370.085670] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13370.085671] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [13370.085673] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13370.085674] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13370.085675] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13370.085677] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13370.085678] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13370.085684] [13370.085927] softirqs last disabled at (30271191): [] irq_exit_rcu+0xb0/0xd0 [13370.086347] Sending NMI from CPU 4 to CPUs 7: [13370.086350] NMI backtrace for cpu 7 [13370.086351] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13370.086353] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13370.086354] Workqueue: events delayed_work [ceph] [13370.086370] RIP: 0010:vprintk_emit+0x1ee/0x340 [13370.086373] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [13370.086374] RSP: 0018:ffffc90000264e38 EFLAGS: 00000002 [13370.086375] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [13370.086376] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb550 [13370.086376] RBP: 0000000000000051 R08: 0000000000005ef4 R09: 0000000000000003 [13370.086377] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [13370.086378] R13: 0000000000000000 R14: ffffffff8219c3c8 R15: ffffc90000264e88 [13370.086379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13370.086380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13370.086381] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13370.086382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13370.086382] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13370.086383] Call Trace: [13370.086384] [13370.086385] ? nmi_cpu_backtrace+0xad/0x120 [13370.086387] ? nmi_cpu_backtrace_handler+0x11/0x20 [13370.086389] ? nmi_handle+0xd4/0x260 [13370.086392] ? default_do_nmi+0x49/0xf0 [13370.086394] ? exc_nmi+0x11e/0x150 [13370.086395] ? end_repeat_nmi+0x16/0x67 [13370.086399] ? vprintk_emit+0x1ee/0x340 [13370.086401] ? vprintk_emit+0x1ee/0x340 [13370.086403] ? vprintk_emit+0x1ee/0x340 [13370.086405] [13370.086405] [13370.086406] ? touch_softlockup_watchdog+0x20/0x20 [13370.086408] _printk+0x5c/0x80 [13370.086409] ? __do_softirq+0x2be/0x371 [13370.086412] ? irq_exit_rcu+0xb0/0xd0 [13370.086414] ? irq_exit_rcu+0xb0/0xd0 [13370.086416] watchdog_timer_fn+0x1a7/0x220 [13370.086418] __hrtimer_run_queues+0x1d3/0x360 [13370.086421] hrtimer_interrupt+0x104/0x240 [13370.086422] ? __local_bh_enable+0x50/0x60 [13370.086425] __sysvec_apic_timer_interrupt+0x61/0x160 [13370.086428] sysvec_apic_timer_interrupt+0x6d/0x90 [13370.086430] [13370.086430] [13370.086431] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13370.086433] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [13370.086436] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [13370.086437] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [13370.086438] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [13370.086439] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [13370.086439] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [13370.086440] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13370.086441] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13370.086444] ceph_check_delayed_caps+0x8f/0x270 [ceph] [13370.086458] delayed_work+0x1d8/0x280 [ceph] [13370.086472] ? process_scheduled_works+0x22a/0x510 [13370.086475] process_scheduled_works+0x2a4/0x510 [13370.086479] worker_thread+0x188/0x340 [13370.086481] ? rescuer_thread+0x310/0x310 [13370.086484] kthread+0xf5/0x130 [13370.086486] ? kthread_complete_and_exit+0x20/0x20 [13370.086488] ret_from_fork+0x31/0x40 [13370.086490] ? kthread_complete_and_exit+0x20/0x20 [13370.086492] ret_from_fork_asm+0x11/0x20 [13370.086497] [13370.090716] __ceph_get_caps+0x4a1/0x760 [ceph] [13370.099394] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13370.104010] ? __file_remove_privs+0x89/0x110 [13370.110089] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13370.120342] ? wake_bit_function+0x70/0x70 [13370.130674] Workqueue: events delayed_work [ceph] [13370.138146] ? ceph_write_iter+0x260/0x770 [ceph] [13370.142495] [13370.142495] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [13370.145980] ceph_write_iter+0x260/0x770 [ceph] [13370.154924] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [13370.162399] ? lock_is_held_type+0xaa/0x120 [13370.166831] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [13370.185571] vfs_write+0x324/0x4f0 [13370.190787] [13370.190787] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [13370.197915] ksys_pwrite64+0x65/0xa0 [13370.205035] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [13370.212159] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13370.219282] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [13370.226409] do_syscall_64+0x38/0x80 [13370.234484] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13370.240222] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13370.247346] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13370.254472] RIP: 0033:0x7f48f0eeb93b [13370.261594] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13370.264038] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13370.266050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13370.270227] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13370.275011] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13370.278582] ORIG_RAX: 0000000000000012 [13370.282414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13370.285810] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13370.289641] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13370.293471] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13370.297302] Call Trace: [13370.297303] [13370.301134] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13370.303230] ? watchdog_timer_fn+0x1b4/0x220 [13370.305239] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13370.310025] ? __hrtimer_run_queues+0x1d3/0x360 [13370.314808] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13370.319597] ? hrtimer_interrupt+0x104/0x240 [13370.322648] [13370.327168] ? __local_bh_enable+0x50/0x60 [13371.187673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13371.192898] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13371.197856] [13371.199952] [13371.202049] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13371.207363] ? __list_del_entry_valid_or_report+0x10/0xd0 [13371.212755] ceph_check_delayed_caps+0x8f/0x270 [ceph] [13371.217903] delayed_work+0x1d8/0x280 [ceph] [13371.222184] ? process_scheduled_works+0x22a/0x510 [13371.226976] process_scheduled_works+0x2a4/0x510 [13371.231590] worker_thread+0x188/0x340 [13371.235359] ? rescuer_thread+0x310/0x310 [13371.239368] kthread+0xf5/0x130 [13371.242506] ? kthread_complete_and_exit+0x20/0x20 [13371.247290] ret_from_fork+0x31/0x40 [13371.250862] ? kthread_complete_and_exit+0x20/0x20 [13371.255661] ret_from_fork_asm+0x11/0x20 [13371.259585] [13384.549701] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13395.301403] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13397.689431] watchdog: BUG: soft lockup - CPU#0 stuck for 1363s! [dbench:78683] [13397.696650] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13397.701431] watchdog: BUG: soft lockup - CPU#7 stuck for 10360s! [kworker/7:4:78808] [13397.752108] irq event stamp: 15913878 [13397.759840] Modules linked in: ceph libceph dns_resolver [13397.763496] hardirqs last enabled at (15913877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13397.768802] fscache netfs veth [13397.778976] hardirqs last disabled at (15913878): [] sysvec_apic_timer_interrupt+0xe/0x90 [13397.782115] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [13397.791837] softirqs last enabled at (15913840): [] __do_softirq+0x2be/0x371 [13397.797487] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13397.806171] softirqs last disabled at (15913835): [] irq_exit_rcu+0xb0/0xd0 [13397.806172] nf_tables [13397.806174] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13397.811217] nfnetlink bridge [13397.819728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13397.822084] stp llc [13397.831046] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13397.834010] xfs libcrc32c [13397.841481] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13397.843664] sunrpc intel_rapl_msr [13397.849142] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13397.851847] intel_rapl_common sb_edac [13397.870583] [13397.873982] x86_pkg_temp_thermal [13397.879199] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13397.882943] coretemp kvm_intel [13397.884434] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13397.887744] kvm irqbypass [13397.894868] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13397.898005] i2c_i801 iTCO_wdt [13397.905130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13397.907834] iTCO_vendor_support i2c_smbus [13397.914957] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13397.918008] crct10dif_pclmul [13397.925132] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13397.929223] joydev crc32_pclmul [13397.936349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13397.939311] ipmi_ssif ghash_clmulni_intel [13397.947389] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13397.950614] mei_me mei [13397.956352] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13397.960441] lpc_ich [13397.967565] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13397.970009] sha512_ssse3 acpi_ipmi [13397.977133] Call Trace: [13397.977134] [13397.979317] mfd_core wmi [13397.986442] ? watchdog_timer_fn+0x1b4/0x220 [13397.989926] ipmi_si ipmi_devintf [13397.992373] ? __hrtimer_run_queues+0x1d3/0x360 [13397.994380] ipmi_msghandler ioatdma [13397.997002] ? hrtimer_interrupt+0x104/0x240 [13398.001261] acpi_pad acpi_power_meter [13398.004573] ? __local_bh_enable+0x50/0x60 [13398.009109] ixgbe igb [13398.012680] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13398.016939] mdio crc32c_intel [13398.020686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13398.024775] nvme ptp [13398.027133] [13398.032350] nvme_core i2c_algo_bit [13398.035402] [13398.035403] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13398.040359] pps_core dca [13398.042632] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13398.044727] fuse [13398.048212] do_raw_spin_lock+0xb5/0xc0 [13398.050308] irq event stamp: 30486056 [13398.050309] hardirqs last enabled at (30486055): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13398.055614] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13398.058230] hardirqs last disabled at (30486056): [] sysvec_apic_timer_interrupt+0xe/0x90 [13398.063113] try_get_cap_refs+0x159/0x880 [ceph] [13398.065033] softirqs last enabled at (30486054): [] __do_softirq+0x2be/0x371 [13398.068867] __ceph_get_caps+0x4a1/0x760 [ceph] [13398.072521] softirqs last disabled at (30486049): [] irq_exit_rcu+0xb0/0xd0 [13398.082684] ? __file_remove_privs+0x89/0x110 [13398.087462] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13398.097187] ? wake_bit_function+0x70/0x70 [13398.101797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13398.110484] ? ceph_write_iter+0x260/0x770 [ceph] [13398.115006] Workqueue: events delayed_work [ceph] [13398.123518] ceph_write_iter+0x260/0x770 [ceph] [13398.127867] [13398.137249] ? lock_is_held_type+0xaa/0x120 [13398.141334] RIP: 0010:lock_acquire+0xf0/0x2b0 [13398.148828] vfs_write+0x324/0x4f0 [13398.153520] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13398.158224] ksys_pwrite64+0x65/0xa0 [13398.162744] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13398.164234] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13398.168409] [13398.168410] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13398.172762] do_syscall_64+0x38/0x80 [13398.176159] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13398.194897] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13398.198468] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13398.203685] RIP: 0033:0x7f48f0eeb93b [13398.208831] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13398.210323] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13398.217447] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13398.221019] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13398.228143] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13398.233186] ORIG_RAX: 0000000000000012 [13398.240311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13398.243880] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13398.251005] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13398.269743] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13398.276869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13398.282086] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13398.290162] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13398.293992] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13398.299731] Call Trace: [13398.299732] [13398.306855] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13398.313980] ? watchdog_timer_fn+0x1b4/0x220 [13398.321108] [13398.328228] ? __hrtimer_run_queues+0x1d3/0x360 [13398.372155] ? hrtimer_interrupt+0x104/0x240 [13398.376423] ? __local_bh_enable+0x50/0x60 [13398.380518] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13398.385741] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13398.390697] [13398.392794] [13398.394892] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13398.400200] ? lock_acquire+0xf0/0x2b0 [13398.403952] ? igrab+0x43/0x50 [13398.407001] ? lock_release+0xc5/0x260 [13398.410747] _raw_spin_lock+0x30/0x40 [13398.414410] ? igrab+0x1d/0x50 [13398.417460] igrab+0x1d/0x50 [13398.420340] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13398.425490] delayed_work+0x1d8/0x280 [ceph] [13398.429770] ? process_scheduled_works+0x22a/0x510 [13398.434561] process_scheduled_works+0x2a4/0x510 [13398.439175] worker_thread+0x188/0x340 [13398.442924] ? rescuer_thread+0x310/0x310 [13398.446929] kthread+0xf5/0x130 [13398.450067] ? kthread_complete_and_exit+0x20/0x20 [13398.454851] ret_from_fork+0x31/0x40 [13398.458420] ? kthread_complete_and_exit+0x20/0x20 [13398.463205] ret_from_fork_asm+0x11/0x20 [13398.467127] [13405.541448] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13416.293627] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13425.689922] watchdog: BUG: soft lockup - CPU#0 stuck for 1389s! [dbench:78683] [13425.697136] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13425.701922] watchdog: BUG: soft lockup - CPU#7 stuck for 10386s! [kworker/7:4:78808] [13425.752597] irq event stamp: 15971776 [13425.760326] Modules linked in: ceph libceph dns_resolver fscache netfs veth [13425.763983] hardirqs last enabled at (15971775): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13425.770942] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13425.781101] hardirqs last disabled at (15971776): [] sysvec_apic_timer_interrupt+0xe/0x90 [13425.789352] nft_compat nf_tables nfnetlink bridge stp [13425.799075] softirqs last enabled at (15971750): [] __do_softirq+0x2be/0x371 [13425.799078] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [13425.812889] softirqs last disabled at (15971745): [] irq_exit_rcu+0xb0/0xd0 [13425.812894] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [13425.828613] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13425.834784] i2c_i801 iTCO_wdt [13425.843728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13425.843729] iTCO_vendor_support [13425.846778] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13425.854250] i2c_smbus crct10dif_pclmul [13425.857472] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13425.862951] joydev crc32_pclmul [13425.866780] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13425.885519] ipmi_ssif ghash_clmulni_intel [13425.888742] [13425.893961] mei_me [13425.898051] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13425.899540] mei lpc_ich [13425.901638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13425.908763] sha512_ssse3 acpi_ipmi [13425.911294] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13425.918435] mfd_core wmi [13425.921919] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13425.929043] ipmi_si [13425.931661] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13425.938784] ipmi_devintf ipmi_msghandler [13425.940968] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13425.948093] ioatdma [13425.952096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13425.960174] acpi_pad [13425.962359] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13425.968096] acpi_power_meter [13425.970366] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13425.977491] ixgbe igb [13425.980455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13425.987580] mdio crc32c_intel [13425.989953] Call Trace: [13425.989955] [13425.997079] nvme [13426.000131] ? watchdog_timer_fn+0x1b4/0x220 [13426.002571] ptp nvme_core [13426.004586] ? __hrtimer_run_queues+0x1d3/0x360 [13426.006507] i2c_algo_bit [13426.010775] ? hrtimer_interrupt+0x104/0x240 [13426.013476] pps_core dca [13426.017999] ? __local_bh_enable+0x50/0x60 [13426.020616] fuse [13426.024883] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13426.027498] [13426.027499] irq event stamp: 30708236 [13426.031589] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13426.033514] hardirqs last enabled at (30708235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13426.038730] [13426.040221] hardirqs last disabled at (30708236): [] sysvec_apic_timer_interrupt+0xe/0x90 [13426.043878] [13426.048837] softirqs last enabled at (30708234): [] __do_softirq+0x2be/0x371 [13426.058996] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13426.061090] softirqs last disabled at (30708229): [] irq_exit_rcu+0xb0/0xd0 [13426.070818] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13426.072913] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13426.081616] do_raw_spin_lock+0xb5/0xc0 [13426.086917] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13426.095430] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13426.100301] Workqueue: events delayed_work [ceph] [13426.109697] try_get_cap_refs+0x159/0x880 [ceph] [13426.113525] [13426.113525] RIP: 0010:lock_acquire+0xf0/0x2b0 [13426.121001] __ceph_get_caps+0x4a1/0x760 [ceph] [13426.125781] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13426.130483] ? __file_remove_privs+0x89/0x110 [13426.135088] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13426.136579] ? wake_bit_function+0x70/0x70 [13426.140929] [13426.140930] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13426.145457] ? ceph_write_iter+0x260/0x770 [ceph] [13426.164192] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13426.168542] ceph_write_iter+0x260/0x770 [ceph] [13426.173761] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13426.177855] ? lock_is_held_type+0xaa/0x120 [13426.179342] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13426.186467] vfs_write+0x324/0x4f0 [13426.191162] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13426.198292] ksys_pwrite64+0x65/0xa0 [13426.202812] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13426.209936] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13426.214113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13426.221237] do_syscall_64+0x38/0x80 [13426.224634] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13426.231760] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13426.235328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13426.243408] RIP: 0033:0x7f48f0eeb93b [13426.248538] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13426.254275] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13426.257844] Call Trace: [13426.257846] [13426.264968] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13426.270014] ? watchdog_timer_fn+0x1b4/0x220 [13426.277138] ORIG_RAX: 0000000000000012 [13426.280710] ? __hrtimer_run_queues+0x1d3/0x360 [13426.287833] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13426.306574] ? hrtimer_interrupt+0x104/0x240 [13426.309013] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13426.311024] ? __local_bh_enable+0x50/0x60 [13426.316241] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13426.320508] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13426.324338] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13426.328862] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13426.335986] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13426.340250] [13426.347378] [13426.351464] [13426.351465] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13426.394700] ? lock_acquire+0xf0/0x2b0 [13426.398449] ? igrab+0x43/0x50 [13426.401499] ? lock_release+0xc5/0x260 [13426.405245] _raw_spin_lock+0x30/0x40 [13426.408910] ? igrab+0x1d/0x50 [13426.411960] igrab+0x1d/0x50 [13426.414838] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13426.419989] delayed_work+0x1d8/0x280 [ceph] [13426.424269] ? process_scheduled_works+0x22a/0x510 [13426.429059] process_scheduled_works+0x2a4/0x510 [13426.433675] worker_thread+0x188/0x340 [13426.437422] ? rescuer_thread+0x310/0x310 [13426.441425] kthread+0xf5/0x130 [13426.444565] ? kthread_complete_and_exit+0x20/0x20 [13426.449349] ret_from_fork+0x31/0x40 [13426.452926] ? kthread_complete_and_exit+0x20/0x20 [13426.457711] ret_from_fork_asm+0x11/0x20 [13426.461636] [13426.533801] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13437.286143] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13447.526135] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13453.690414] watchdog: BUG: soft lockup - CPU#0 stuck for 1415s! [dbench:78683] [13453.697634] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13453.702414] watchdog: BUG: soft lockup - CPU#7 stuck for 10412s! [kworker/7:4:78808] [13453.753088] irq event stamp: 16029628 [13453.753090] hardirqs last enabled at (16029627): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13453.760821] Modules linked in: [13453.764478] hardirqs last disabled at (16029628): [] sysvec_apic_timer_interrupt+0xe/0x90 [13453.774637] ceph [13453.777687] softirqs last enabled at (16029590): [] __do_softirq+0x2be/0x371 [13453.787411] libceph [13453.789336] softirqs last disabled at (16029585): [] irq_exit_rcu+0xb0/0xd0 [13453.798019] dns_resolver fscache [13453.800203] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13453.808715] netfs [13453.812026] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13453.812028] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13453.820970] veth nft_chain_nat [13453.822981] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13453.830450] xt_MASQUERADE nf_nat [13453.835928] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13453.839066] nf_conntrack nf_defrag_ipv6 [13453.857820] [13453.861132] nf_defrag_ipv4 nft_compat [13453.866348] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13453.870265] nf_tables nfnetlink [13453.871756] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13453.875500] bridge stp llc [13453.882626] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13453.885850] xfs [13453.892974] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13453.895764] libcrc32c sunrpc [13453.902888] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13453.904726] intel_rapl_msr [13453.911849] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13453.914815] intel_rapl_common [13453.921938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13453.924728] sb_edac x86_pkg_temp_thermal [13453.932812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13453.935866] coretemp kvm_intel [13453.941602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13453.945608] kvm irqbypass [13453.952732] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13453.955868] i2c_i801 iTCO_wdt [13453.962991] Call Trace: [13453.962993] [13453.965696] iTCO_vendor_support i2c_smbus [13453.972821] ? watchdog_timer_fn+0x1b4/0x220 [13453.975870] crct10dif_pclmul joydev [13453.978318] ? __hrtimer_run_queues+0x1d3/0x360 [13453.980325] crc32_pclmul ipmi_ssif [13453.984419] ? hrtimer_interrupt+0x104/0x240 [13453.988680] ghash_clmulni_intel mei_me [13453.992252] ? __local_bh_enable+0x50/0x60 [13453.996776] mei lpc_ich [13454.000262] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13454.004523] sha512_ssse3 acpi_ipmi [13454.008355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13454.012447] mfd_core [13454.014977] [13454.020193] wmi ipmi_si [13454.023677] [13454.028634] ipmi_devintf ipmi_msghandler [13454.030907] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13454.033002] ioatdma acpi_pad [13454.035538] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13454.037632] acpi_power_meter [13454.041637] do_raw_spin_lock+0xb5/0xc0 [13454.046940] ixgbe igb mdio [13454.049904] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13454.054774] crc32c_intel nvme [13454.057740] try_get_cap_refs+0x159/0x880 [ceph] [13454.061568] ptp nvme_core [13454.064365] __ceph_get_caps+0x4a1/0x760 [ceph] [13454.069144] i2c_algo_bit pps_core [13454.072199] ? __file_remove_privs+0x89/0x110 [13454.076812] dca fuse [13454.079519] ? wake_bit_function+0x70/0x70 [13454.084042] [13454.084043] irq event stamp: 30931004 [13454.087442] ? ceph_write_iter+0x260/0x770 [ceph] [13454.091808] hardirqs last enabled at (30931003): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13454.094079] ceph_write_iter+0x260/0x770 [ceph] [13454.098186] hardirqs last disabled at (30931004): [] sysvec_apic_timer_interrupt+0xe/0x90 [13454.099680] ? lock_is_held_type+0xaa/0x120 [13454.103335] softirqs last enabled at (30931002): [] __do_softirq+0x2be/0x371 [13454.108034] vfs_write+0x324/0x4f0 [13454.118189] softirqs last disabled at (30930997): [] irq_exit_rcu+0xb0/0xd0 [13454.122719] ksys_pwrite64+0x65/0xa0 [13454.132437] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13454.136614] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13454.145298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13454.145299] Workqueue: events delayed_work [ceph] [13454.148697] do_syscall_64+0x38/0x80 [13454.157207] [13454.157208] RIP: 0010:lock_acquire+0x9/0x2b0 [13454.160778] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13454.170156] Code: 01 75 0e 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca <41> 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 [13454.175287] RIP: 0033:0x7f48f0eeb93b [13454.182758] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [13454.187454] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13454.191024] [13454.191025] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13454.192516] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13454.196779] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13454.201824] ORIG_RAX: 0000000000000012 [13454.220561] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [13454.224133] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13454.229351] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [13454.248103] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13454.249596] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13454.256720] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13454.261937] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13454.269062] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13454.272893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13454.280016] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13454.287141] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13454.294269] [13454.301388] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13454.360132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13454.367256] Call Trace: [13454.369701] [13454.371713] ? watchdog_timer_fn+0x1b4/0x220 [13454.375978] ? __hrtimer_run_queues+0x1d3/0x360 [13454.380503] ? hrtimer_interrupt+0x104/0x240 [13454.384773] ? __local_bh_enable+0x50/0x60 [13454.388867] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13454.394105] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13454.399063] [13454.401161] [13454.403260] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13454.408568] ? lock_acquire+0x9/0x2b0 [13454.412230] _raw_spin_lock+0x30/0x40 [13454.415887] ? igrab+0x1d/0x50 [13454.418937] igrab+0x1d/0x50 [13454.421820] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13454.426966] delayed_work+0x1d8/0x280 [ceph] [13454.431246] ? process_scheduled_works+0x22a/0x510 [13454.436037] process_scheduled_works+0x2a4/0x510 [13454.440652] worker_thread+0x188/0x340 [13454.444402] ? rescuer_thread+0x310/0x310 [13454.448404] kthread+0xf5/0x130 [13454.451543] ? kthread_complete_and_exit+0x20/0x20 [13454.456330] ret_from_fork+0x31/0x40 [13454.459906] ? kthread_complete_and_exit+0x20/0x20 [13454.464700] ret_from_fork_asm+0x11/0x20 [13454.468630] [13458.278483] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13468.518507] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13479.270739] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13481.690906] watchdog: BUG: soft lockup - CPU#0 stuck for 1441s! [dbench:78683] [13481.698127] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13481.702905] watchdog: BUG: soft lockup - CPU#7 stuck for 10438s! [kworker/7:4:78808] [13481.753586] irq event stamp: 16087650 [13481.761319] Modules linked in: ceph libceph dns_resolver fscache netfs veth [13481.764973] hardirqs last enabled at (16087649): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13481.764979] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13481.782090] hardirqs last disabled at (16087650): [] sysvec_apic_timer_interrupt+0xe/0x90 [13481.791300] nf_tables nfnetlink bridge stp llc xfs [13481.801027] softirqs last enabled at (16087612): [] __do_softirq+0x2be/0x371 [13481.805899] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [13481.814583] softirqs last disabled at (16087607): [] irq_exit_rcu+0xb0/0xd0 [13481.823709] kvm_intel kvm irqbypass i2c_i801 [13481.832222] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13481.836582] iTCO_wdt [13481.845523] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13481.845525] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13481.847794] iTCO_vendor_support i2c_smbus [13481.855266] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13481.860742] crct10dif_pclmul [13481.864832] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13481.883580] joydev crc32_pclmul [13481.886544] [13481.886545] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13481.891760] ipmi_ssif [13481.894985] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13481.896477] ghash_clmulni_intel [13481.903601] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13481.905957] mei_me [13481.913081] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13481.916305] mei [13481.923429] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13481.925528] lpc_ich [13481.932652] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13481.934489] sha512_ssse3 [13481.941614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13481.943797] acpi_ipmi mfd_core [13481.951874] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13481.954492] wmi ipmi_si [13481.960229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13481.963367] ipmi_devintf ipmi_msghandler [13481.970490] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13481.973021] ioatdma acpi_pad [13481.980145] Call Trace: [13481.980148] [13481.984150] acpi_power_meter [13481.991278] ? watchdog_timer_fn+0x1b4/0x220 [13481.994237] ixgbe igb [13481.996682] ? __hrtimer_run_queues+0x1d3/0x360 [13481.998693] mdio crc32c_intel [13482.001659] ? hrtimer_interrupt+0x104/0x240 [13482.005922] nvme [13482.008277] ? __local_bh_enable+0x50/0x60 [13482.012819] ptp [13482.015872] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13482.020133] nvme_core i2c_algo_bit [13482.022060] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13482.026150] pps_core [13482.027986] [13482.033205] dca fuse [13482.036688] [13482.036689] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13482.041646] [13482.043921] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13482.046014] irq event stamp: 31152898 [13482.046015] hardirqs last enabled at (31152897): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13482.048305] do_raw_spin_lock+0xb5/0xc0 [13482.050399] hardirqs last disabled at (31152898): [] sysvec_apic_timer_interrupt+0xe/0x90 [13482.055704] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13482.057195] softirqs last enabled at (31152896): [] __do_softirq+0x2be/0x371 [13482.062068] try_get_cap_refs+0x159/0x880 [ceph] [13482.065723] softirqs last disabled at (31152891): [] irq_exit_rcu+0xb0/0xd0 [13482.075884] __ceph_get_caps+0x4a1/0x760 [ceph] [13482.079710] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13482.089438] ? __file_remove_privs+0x89/0x110 [13482.094219] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13482.102902] ? wake_bit_function+0x70/0x70 [13482.107514] Workqueue: events delayed_work [ceph] [13482.116027] ? ceph_write_iter+0x260/0x770 [ceph] [13482.120549] [13482.120550] RIP: 0010:lock_acquire+0xf0/0x2b0 [13482.129928] ceph_write_iter+0x260/0x770 [ceph] [13482.134276] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13482.141752] ? lock_is_held_type+0xaa/0x120 [13482.145838] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13482.150537] vfs_write+0x324/0x4f0 [13482.155233] [13482.155234] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13482.156729] ksys_pwrite64+0x65/0xa0 [13482.161074] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13482.165599] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13482.184335] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13482.188515] do_syscall_64+0x38/0x80 [13482.193730] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13482.197129] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13482.198619] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13482.205746] RIP: 0033:0x7f48f0eeb93b [13482.209314] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13482.216438] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13482.221568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13482.228693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13482.232262] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13482.239388] ORIG_RAX: 0000000000000012 [13482.244431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13482.251555] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13482.255127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13482.263205] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13482.281943] Call Trace: [13482.281945] [13482.287679] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13482.292899] ? watchdog_timer_fn+0x1b4/0x220 [13482.300021] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13482.303852] ? __hrtimer_run_queues+0x1d3/0x360 [13482.310977] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13482.318104] ? hrtimer_interrupt+0x104/0x240 [13482.325229] [13482.332349] ? __local_bh_enable+0x50/0x60 [13482.377505] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13482.382730] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13482.387685] [13482.389784] [13482.391882] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13482.397189] ? lock_acquire+0xf0/0x2b0 [13482.400940] ? igrab+0x43/0x50 [13482.403999] ? lock_release+0xc5/0x260 [13482.407742] _raw_spin_lock+0x30/0x40 [13482.411400] ? igrab+0x1d/0x50 [13482.414450] igrab+0x1d/0x50 [13482.417330] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13482.422477] delayed_work+0x1d8/0x280 [ceph] [13482.426759] ? process_scheduled_works+0x22a/0x510 [13482.431548] process_scheduled_works+0x2a4/0x510 [13482.436165] worker_thread+0x188/0x340 [13482.439915] ? rescuer_thread+0x310/0x310 [13482.443925] kthread+0xf5/0x130 [13482.447064] ? kthread_complete_and_exit+0x20/0x20 [13482.451848] ret_from_fork+0x31/0x40 [13482.455428] ? kthread_complete_and_exit+0x20/0x20 [13482.460211] ret_from_fork_asm+0x11/0x20 [13482.464143] [13509.691397] watchdog: BUG: soft lockup - CPU#0 stuck for 1467s! [dbench:78683] [13509.698614] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13509.703397] watchdog: BUG: soft lockup - CPU#7 stuck for 10464s! [kworker/7:4:78808] [13509.754056] irq event stamp: 16145568 [13509.761803] Modules linked in: ceph libceph [13509.765462] hardirqs last enabled at (16145567): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13509.769639] dns_resolver fscache netfs [13509.779797] hardirqs last disabled at (16145568): [] sysvec_apic_timer_interrupt+0xe/0x90 [13509.783626] veth nft_chain_nat xt_MASQUERADE [13509.793352] softirqs last enabled at (16145542): [] __do_softirq+0x2be/0x371 [13509.793353] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13509.806386] softirqs last disabled at (16145537): [] irq_exit_rcu+0xb0/0xd0 [13509.806388] nft_compat [13509.812209] nf_tables nfnetlink [13509.820722] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13509.823165] bridge stp [13509.826388] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13509.835333] llc xfs [13509.837794] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13509.845264] libcrc32c sunrpc [13509.847449] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13509.852927] intel_rapl_msr intel_rapl_common [13509.855891] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13509.874627] sb_edac x86_pkg_temp_thermal [13509.878980] [13509.878981] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13509.884197] coretemp kvm_intel [13509.888201] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13509.889692] kvm irqbypass [13509.896816] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13509.899953] i2c_i801 iTCO_wdt [13509.907077] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13509.909793] iTCO_vendor_support [13509.916914] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13509.919966] i2c_smbus crct10dif_pclmul [13509.927089] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13509.930312] joydev crc32_pclmul [13509.937436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13509.941267] ipmi_ssif ghash_clmulni_intel [13509.949345] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13509.952570] mei_me [13509.958308] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13509.962398] mei lpc_ich [13509.969523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13509.971620] sha512_ssse3 acpi_ipmi [13509.978744] Call Trace: [13509.978746] [13509.981275] mfd_core wmi [13509.988400] ? watchdog_timer_fn+0x1b4/0x220 [13509.991882] ipmi_si ipmi_devintf [13509.994327] ? __hrtimer_run_queues+0x1d3/0x360 [13509.996338] ipmi_msghandler ioatdma [13509.998957] ? hrtimer_interrupt+0x104/0x240 [13510.003217] acpi_pad acpi_power_meter [13510.006530] ? __local_bh_enable+0x50/0x60 [13510.011052] ixgbe igb [13510.014627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13510.018889] mdio crc32c_intel [13510.022632] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13510.026724] nvme ptp [13510.029081] [13510.034298] nvme_core i2c_algo_bit [13510.037348] [13510.042306] pps_core dca [13510.044576] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13510.046675] fuse [13510.046676] irq event stamp: 31375202 [13510.050163] ? queued_spin_lock_slowpath+0x16/0x2e0 [13510.052255] hardirqs last enabled at (31375201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13510.054874] do_raw_spin_lock+0xb5/0xc0 [13510.060177] hardirqs last disabled at (31375202): [] sysvec_apic_timer_interrupt+0xe/0x90 [13510.062102] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13510.065758] softirqs last enabled at (31375200): [] __do_softirq+0x2be/0x371 [13510.070632] try_get_cap_refs+0x159/0x880 [ceph] [13510.080798] softirqs last disabled at (31375195): [] irq_exit_rcu+0xb0/0xd0 [13510.084630] __ceph_get_caps+0x4a1/0x760 [ceph] [13510.094350] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13510.099137] ? __file_remove_privs+0x89/0x110 [13510.107817] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13510.112429] ? wake_bit_function+0x70/0x70 [13510.120941] Workqueue: events delayed_work [ceph] [13510.125468] ? ceph_write_iter+0x260/0x770 [ceph] [13510.134842] [13510.139193] ceph_write_iter+0x260/0x770 [ceph] [13510.146664] RIP: 0010:lock_acquire+0xf0/0x2b0 [13510.150757] ? lock_is_held_type+0xaa/0x120 [13510.155451] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13510.160152] vfs_write+0x324/0x4f0 [13510.161638] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13510.166169] ksys_pwrite64+0x65/0xa0 [13510.170514] [13510.170515] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13510.174691] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13510.193430] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13510.196827] do_syscall_64+0x38/0x80 [13510.202043] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13510.205614] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13510.207107] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13510.214230] RIP: 0033:0x7f48f0eeb93b [13510.219359] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13510.226485] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13510.230056] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13510.237178] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13510.242222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13510.249348] ORIG_RAX: 0000000000000012 [13510.252917] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13510.260043] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13510.278798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13510.286876] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13510.292092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13510.297828] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13510.301662] Call Trace: [13510.301663] [13510.308801] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13510.315926] ? watchdog_timer_fn+0x1b4/0x220 [13510.323051] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13510.330176] ? __hrtimer_run_queues+0x1d3/0x360 [13510.337304] [13510.344428] ? hrtimer_interrupt+0x104/0x240 [13510.378363] ? __local_bh_enable+0x50/0x60 [13510.382455] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13510.387678] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13510.392637] [13510.394734] [13510.396831] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13510.402140] ? lock_acquire+0xf0/0x2b0 [13510.405889] ? igrab+0x43/0x50 [13510.408939] ? lock_release+0xc5/0x260 [13510.412685] _raw_spin_lock+0x30/0x40 [13510.416350] ? igrab+0x1d/0x50 [13510.419399] igrab+0x1d/0x50 [13510.422278] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13510.427426] delayed_work+0x1d8/0x280 [ceph] [13510.431709] ? process_scheduled_works+0x22a/0x510 [13510.436500] process_scheduled_works+0x2a4/0x510 [13510.441114] worker_thread+0x188/0x340 [13510.444863] ? rescuer_thread+0x310/0x310 [13510.448867] kthread+0xf5/0x130 [13510.452004] ? kthread_complete_and_exit+0x20/0x20 [13510.456807] ret_from_fork+0x31/0x40 [13510.460386] ? kthread_complete_and_exit+0x20/0x20 [13510.465169] ret_from_fork_asm+0x11/0x20 [13510.469092] [13510.502826] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [13510.511280] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13521.255538] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13531.495621] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13537.691888] watchdog: BUG: soft lockup - CPU#0 stuck for 1493s! [dbench:78683] [13537.699102] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13537.703888] watchdog: BUG: soft lockup - CPU#7 stuck for 10490s! [kworker/7:4:78808] [13537.754562] irq event stamp: 16203624 [13537.762292] Modules linked in: ceph libceph dns_resolver fscache netfs veth [13537.765947] hardirqs last enabled at (16203623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13537.772909] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [13537.783066] hardirqs last disabled at (16203624): [] sysvec_apic_timer_interrupt+0xe/0x90 [13537.788717] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [13537.798442] softirqs last enabled at (16203616): [] __do_softirq+0x2be/0x371 [13537.805832] stp llc xfs libcrc32c sunrpc intel_rapl_msr [13537.814517] softirqs last disabled at (16203611): [] irq_exit_rcu+0xb0/0xd0 [13537.819821] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [13537.828333] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13537.834676] kvm_intel kvm [13537.843619] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13537.846325] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [13537.853798] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13537.853799] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [13537.865792] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13537.865793] ghash_clmulni_intel [13537.865794] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13537.871354] mei_me mei [13537.890092] [13537.890093] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13537.893316] lpc_ich [13537.898534] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13537.900977] sha512_ssse3 [13537.902467] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13537.909592] acpi_ipmi mfd_core [13537.911776] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13537.918902] wmi ipmi_si [13537.921518] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13537.928641] ipmi_devintf ipmi_msghandler [13537.931778] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13537.938903] ioatdma acpi_pad [13537.941435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13537.948557] acpi_power_meter [13537.952565] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13537.960639] ixgbe igb [13537.963604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13537.969342] mdio crc32c_intel [13537.972306] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13537.979431] nvme [13537.981787] Call Trace: [13537.981789] [13537.988911] ptp nvme_core [13537.991966] ? watchdog_timer_fn+0x1b4/0x220 [13537.999086] i2c_algo_bit pps_core [13538.001012] ? __hrtimer_run_queues+0x1d3/0x360 [13538.003454] dca [13538.005468] ? hrtimer_interrupt+0x104/0x240 [13538.008170] fuse [13538.012432] ? __local_bh_enable+0x50/0x60 [13538.015831] irq event stamp: 31597882 [13538.020357] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13538.022191] hardirqs last enabled at (31597881): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13538.026456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13538.028379] hardirqs last disabled at (31597882): [] sysvec_apic_timer_interrupt+0xe/0x90 [13538.032471] [13538.036128] softirqs last enabled at (31597880): [] __do_softirq+0x2be/0x371 [13538.041344] [13538.041346] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13538.051504] softirqs last disabled at (31597875): [] irq_exit_rcu+0xb0/0xd0 [13538.056464] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13538.066184] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13538.068290] do_raw_spin_lock+0xb5/0xc0 [13538.076974] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13538.079073] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13538.084376] Workqueue: events delayed_work [ceph] [13538.092891] try_get_cap_refs+0x159/0x880 [ceph] [13538.097757] [13538.097758] RIP: 0010:lock_acquire+0xf0/0x2b0 [13538.107141] __ceph_get_caps+0x4a1/0x760 [ceph] [13538.110966] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13538.118441] ? __file_remove_privs+0x89/0x110 [13538.123221] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13538.127919] ? wake_bit_function+0x70/0x70 [13538.132529] [13538.132530] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13538.134021] ? ceph_write_iter+0x260/0x770 [ceph] [13538.138371] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13538.142894] ceph_write_iter+0x260/0x770 [ceph] [13538.161632] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13538.165988] ? lock_is_held_type+0xaa/0x120 [13538.171202] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13538.175295] vfs_write+0x324/0x4f0 [13538.176782] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13538.183911] ksys_pwrite64+0x65/0xa0 [13538.188603] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13538.195728] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13538.200251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13538.207376] do_syscall_64+0x38/0x80 [13538.211568] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13538.218695] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13538.222093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13538.229218] RIP: 0033:0x7f48f0eeb93b [13538.232788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13538.240866] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13538.245994] Call Trace: [13538.245997] [13538.251733] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13538.255305] ? watchdog_timer_fn+0x1b4/0x220 [13538.262428] ORIG_RAX: 0000000000000012 [13538.267473] ? __hrtimer_run_queues+0x1d3/0x360 [13538.274595] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13538.278169] ? hrtimer_interrupt+0x104/0x240 [13538.285291] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13538.304027] ? __local_bh_enable+0x50/0x60 [13538.306471] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13538.308486] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13538.313702] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13538.317966] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13538.321794] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13538.326320] [13538.333449] [13538.337709] [13538.337710] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13538.392166] ? lock_acquire+0xf0/0x2b0 [13538.395916] ? igrab+0x43/0x50 [13538.398966] ? lock_release+0xc5/0x260 [13538.402711] _raw_spin_lock+0x30/0x40 [13538.406375] ? igrab+0x1d/0x50 [13538.409427] igrab+0x1d/0x50 [13538.412306] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13538.417455] delayed_work+0x1d8/0x280 [ceph] [13538.421736] ? process_scheduled_works+0x22a/0x510 [13538.426526] process_scheduled_works+0x2a4/0x510 [13538.431141] worker_thread+0x188/0x340 [13538.434890] ? rescuer_thread+0x310/0x310 [13538.438894] kthread+0xf5/0x130 [13538.442042] ? kthread_complete_and_exit+0x20/0x20 [13538.446832] ret_from_fork+0x31/0x40 [13538.450412] ? kthread_complete_and_exit+0x20/0x20 [13538.455195] ret_from_fork_asm+0x11/0x20 [13538.459118] [13542.248028] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13552.488014] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13563.240322] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13565.091734] rcu: INFO: rcu_preempt self-detected stall on CPU [13565.097496] rcu: 0-....: (1594099 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=770848 [13565.107741] rcu: (t=1628453 jiffies g=1309609 q=130806 ncpus=8) [13565.113738] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13565.122698] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13565.130169] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13565.135647] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13565.154385] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13565.159601] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13565.166725] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13565.173850] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13565.180973] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13565.188099] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13565.195222] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13565.203299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13565.209037] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13565.216161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13565.223287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13565.230412] Call Trace: [13565.232855] [13565.234865] ? rcu_dump_cpu_stacks+0xd1/0x180 [13565.239218] ? rcu_sched_clock_irq+0x633/0x1390 [13565.243742] ? lock_is_held_type+0xaa/0x120 [13565.247928] ? lock_is_held_type+0xaa/0x120 [13565.252106] ? tick_sched_do_timer+0xa0/0xa0 [13565.256377] ? rcu_read_lock_sched_held+0x56/0x90 [13565.261074] ? tick_sched_do_timer+0xa0/0xa0 [13565.265338] ? update_process_times+0x5f/0x90 [13565.269688] ? tick_sched_handle.isra.16+0x1e/0x50 [13565.274474] ? tick_sched_timer+0x6f/0x80 [13565.278478] ? __hrtimer_run_queues+0x1d3/0x360 [13565.283003] ? hrtimer_interrupt+0x104/0x240 [13565.287274] ? __local_bh_enable+0x50/0x60 [13565.291367] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13565.296590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13565.301548] [13565.303645] [13565.305742] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13565.311052] ? queued_spin_lock_slowpath+0x16/0x2e0 [13565.315927] do_raw_spin_lock+0xb5/0xc0 [13565.319758] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13565.324561] try_get_cap_refs+0x159/0x880 [ceph] [13565.329191] __ceph_get_caps+0x4a1/0x760 [ceph] [13565.333732] ? __file_remove_privs+0x89/0x110 [13565.338087] ? wake_bit_function+0x70/0x70 [13565.342180] ? ceph_write_iter+0x260/0x770 [ceph] [13565.346893] ceph_write_iter+0x260/0x770 [ceph] [13565.351429] ? lock_is_held_type+0xaa/0x120 [13565.355614] vfs_write+0x324/0x4f0 [13565.359021] ksys_pwrite64+0x65/0xa0 [13565.362597] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13565.367728] do_syscall_64+0x38/0x80 [13565.371299] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13565.376343] RIP: 0033:0x7f48f0eeb93b [13565.379913] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13565.398650] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [13565.406208] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13565.413332] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13565.420455] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13565.427581] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13565.434705] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13565.441834] [13565.444021] Sending NMI from CPU 0 to CPUs 7: [13565.448373] NMI backtrace for cpu 7 [13565.448375] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13565.448377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13565.448378] Workqueue: events delayed_work [ceph] [13565.448394] RIP: 0010:rcu_is_watching+0x1f/0x40 [13565.448397] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [13565.448398] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [13565.448399] RAX: 00000000007c1919 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13565.448400] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13565.448401] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [13565.448402] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [13565.448402] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13565.448403] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13565.448404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13565.448405] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13565.448406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13565.448406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13565.448407] Call Trace: [13565.448408] [13565.448409] ? nmi_cpu_backtrace+0xad/0x120 [13565.448413] ? nmi_cpu_backtrace_handler+0x11/0x20 [13565.448414] ? nmi_handle+0xd4/0x260 [13565.448418] ? default_do_nmi+0x49/0xf0 [13565.448420] ? exc_nmi+0x11e/0x150 [13565.448422] ? end_repeat_nmi+0x16/0x67 [13565.448426] ? rcu_is_watching+0x1f/0x40 [13565.448428] ? rcu_is_watching+0x1f/0x40 [13565.448430] ? rcu_is_watching+0x1f/0x40 [13565.448432] [13565.448432] [13565.448433] lock_acquire+0x184/0x2b0 [13565.448434] ? igrab+0x43/0x50 [13565.448436] ? lock_release+0xc5/0x260 [13565.448438] _raw_spin_lock+0x30/0x40 [13565.448440] ? igrab+0x1d/0x50 [13565.448441] igrab+0x1d/0x50 [13565.448442] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13565.448456] delayed_work+0x1d8/0x280 [ceph] [13565.448470] ? process_scheduled_works+0x22a/0x510 [13565.448472] process_scheduled_works+0x2a4/0x510 [13565.448476] worker_thread+0x188/0x340 [13565.448479] ? rescuer_thread+0x310/0x310 [13565.448481] kthread+0xf5/0x130 [13565.448483] ? kthread_complete_and_exit+0x20/0x20 [13565.448486] ret_from_fork+0x31/0x40 [13565.448488] ? kthread_complete_and_exit+0x20/0x20 [13565.448490] ret_from_fork_asm+0x11/0x20 [13565.448494] [13573.480454] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13584.232669] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13589.692800] watchdog: BUG: soft lockup - CPU#0 stuck for 1542s! [dbench:78683] [13589.700016] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13589.704800] watchdog: BUG: soft lockup - CPU#7 stuck for 10539s! [kworker/7:4:78808] [13589.755474] irq event stamp: 16311632 [13589.763204] Modules linked in: ceph libceph dns_resolver fscache netfs [13589.766862] hardirqs last enabled at (16311631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13589.766867] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13589.783545] hardirqs last disabled at (16311632): [] sysvec_apic_timer_interrupt+0xe/0x90 [13589.792231] nft_compat nf_tables nfnetlink bridge stp llc [13589.801953] softirqs last enabled at (16311540): [] __do_softirq+0x2be/0x371 [13589.807430] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [13589.816116] softirqs last disabled at (16311535): [] irq_exit_rcu+0xb0/0xd0 [13589.816120] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [13589.831499] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13589.837670] i2c_i801 iTCO_wdt [13589.846613] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13589.846615] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13589.849665] iTCO_vendor_support i2c_smbus [13589.857136] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13589.862613] crct10dif_pclmul joydev [13589.866704] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13589.885442] crc32_pclmul [13589.889011] [13589.889012] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13589.894228] ipmi_ssif [13589.896846] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13589.898337] ghash_clmulni_intel [13589.905477] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13589.907837] mei_me mei [13589.914960] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13589.918196] lpc_ich [13589.925317] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13589.927762] sha512_ssse3 [13589.934885] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13589.937069] acpi_ipmi mfd_core [13589.944197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13589.946820] wmi ipmi_si [13589.954896] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13589.958033] ipmi_devintf ipmi_msghandler [13589.963773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13589.966301] ioatdma acpi_pad [13589.973426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13589.977431] acpi_power_meter [13589.984556] Call Trace: [13589.987520] ixgbe igb [13589.994645] [13589.997606] mdio crc32c_intel [13590.000055] ? watchdog_timer_fn+0x1b4/0x220 [13590.002409] nvme [13590.004420] ? __hrtimer_run_queues+0x1d3/0x360 [13590.007478] ptp nvme_core [13590.011746] ? hrtimer_interrupt+0x104/0x240 [13590.013666] i2c_algo_bit pps_core [13590.018197] ? __local_bh_enable+0x50/0x60 [13590.020903] dca fuse [13590.025169] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13590.028566] [13590.028566] irq event stamp: 32013026 [13590.032658] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13590.034927] hardirqs last enabled at (32013025): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13590.040144] [13590.041634] hardirqs last disabled at (32013026): [] sysvec_apic_timer_interrupt+0xe/0x90 [13590.045293] [13590.045294] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13590.050249] softirqs last enabled at (32013024): [] __do_softirq+0x2be/0x371 [13590.060413] ? queued_spin_lock_slowpath+0x18/0x2e0 [13590.062504] softirqs last disabled at (32013019): [] irq_exit_rcu+0xb0/0xd0 [13590.072230] do_raw_spin_lock+0xb5/0xc0 [13590.074327] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13590.079632] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13590.088314] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13590.093201] try_get_cap_refs+0x159/0x880 [ceph] [13590.101714] Workqueue: events delayed_work [ceph] [13590.105549] __ceph_get_caps+0x4a1/0x760 [ceph] [13590.114921] [13590.119712] ? __file_remove_privs+0x89/0x110 [13590.127178] RIP: 0010:ceph_check_delayed_caps+0x93/0x270 [ceph] [13590.131805] ? wake_bit_function+0x70/0x70 [13590.136503] Code: 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e <48> 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 [13590.141028] ? ceph_write_iter+0x260/0x770 [ceph] [13590.142516] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [13590.146868] ceph_write_iter+0x260/0x770 [ceph] [13590.152778] [13590.152779] RAX: 0000000000000001 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [13590.156873] ? lock_is_held_type+0xaa/0x120 [13590.175606] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [13590.180305] vfs_write+0x324/0x4f0 [13590.185522] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [13590.190050] ksys_pwrite64+0x65/0xa0 [13590.191537] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13590.198661] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13590.202838] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13590.209964] do_syscall_64+0x38/0x80 [13590.213358] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13590.220493] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13590.224062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13590.231204] RIP: 0033:0x7f48f0eeb93b [13590.236334] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13590.243460] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13590.247030] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13590.255108] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13590.260151] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13590.265888] ORIG_RAX: 0000000000000012 [13590.269460] Call Trace: [13590.269461] [13590.276585] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13590.295323] ? watchdog_timer_fn+0x1b4/0x220 [13590.302445] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13590.307663] ? __hrtimer_run_queues+0x1d3/0x360 [13590.314788] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13590.318620] ? hrtimer_interrupt+0x104/0x240 [13590.321062] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13590.323072] ? __local_bh_enable+0x50/0x60 [13590.330203] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13590.334487] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13590.341617] [13590.346135] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13590.388223] [13590.390320] [13590.392417] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13590.397725] ? ceph_check_delayed_caps+0x93/0x270 [ceph] [13590.403045] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [13590.408364] delayed_work+0x1d8/0x280 [ceph] [13590.412648] ? process_scheduled_works+0x22a/0x510 [13590.417436] process_scheduled_works+0x2a4/0x510 [13590.422054] worker_thread+0x188/0x340 [13590.425804] ? rescuer_thread+0x310/0x310 [13590.429812] kthread+0xf5/0x130 [13590.432953] ? kthread_complete_and_exit+0x20/0x20 [13590.437746] ret_from_fork+0x31/0x40 [13590.441324] ? kthread_complete_and_exit+0x20/0x20 [13590.446107] ret_from_fork_asm+0x11/0x20 [13590.450030] [13594.472802] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13605.224863] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13617.693291] watchdog: BUG: soft lockup - CPU#0 stuck for 1568s! [dbench:78683] [13617.700501] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13617.705291] watchdog: BUG: soft lockup - CPU#7 stuck for 10565s! [kworker/7:4:78808] [13617.755943] irq event stamp: 16369588 [13617.755944] hardirqs last enabled at (16369587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13617.763691] Modules linked in: [13617.767349] hardirqs last disabled at (16369588): [] sysvec_apic_timer_interrupt+0xe/0x90 [13617.777505] ceph [13617.780556] softirqs last enabled at (16369500): [] __do_softirq+0x2be/0x371 [13617.790281] libceph [13617.792204] softirqs last disabled at (16369495): [] irq_exit_rcu+0xb0/0xd0 [13617.800889] dns_resolver fscache [13617.803074] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13617.811585] netfs veth [13617.814896] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13617.823839] nft_chain_nat xt_MASQUERADE [13617.826284] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13617.833753] nf_nat [13617.837685] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13617.843166] nf_conntrack nf_defrag_ipv6 [13617.845264] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13617.864000] nf_defrag_ipv4 [13617.867919] [13617.867920] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13617.873135] nft_compat nf_tables [13617.875926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13617.877416] nfnetlink [13617.884540] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13617.887851] bridge stp [13617.894985] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13617.897342] llc xfs [13617.904466] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13617.906910] libcrc32c sunrpc [13617.914036] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13617.916219] intel_rapl_msr intel_rapl_common [13617.923343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13617.926306] sb_edac x86_pkg_temp_thermal [13617.934385] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13617.938736] coretemp kvm_intel [13617.944472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13617.948477] kvm irqbypass [13617.955600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13617.958738] i2c_i801 iTCO_wdt [13617.965863] Call Trace: [13617.965864] [13617.968566] iTCO_vendor_support i2c_smbus [13617.975690] ? watchdog_timer_fn+0x1b4/0x220 [13617.978741] crct10dif_pclmul joydev [13617.981185] ? __hrtimer_run_queues+0x1d3/0x360 [13617.983195] crc32_pclmul ipmi_ssif [13617.987289] ? hrtimer_interrupt+0x104/0x240 [13617.991552] ghash_clmulni_intel mei_me [13617.995122] ? __local_bh_enable+0x50/0x60 [13617.999645] mei lpc_ich [13618.003133] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13618.007395] sha512_ssse3 acpi_ipmi [13618.011226] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13618.015315] mfd_core wmi [13618.017846] [13618.023064] ipmi_si ipmi_devintf [13618.026565] [13618.026566] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13618.031523] ipmi_msghandler ioatdma acpi_pad [13618.034143] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13618.036236] acpi_power_meter ixgbe [13618.039551] do_raw_spin_lock+0xb5/0xc0 [13618.041646] igb [13618.046950] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13618.051299] mdio crc32c_intel [13618.056174] try_get_cap_refs+0x159/0x880 [ceph] [13618.059655] nvme ptp [13618.063489] __ceph_get_caps+0x4a1/0x760 [ceph] [13618.065322] nvme_core i2c_algo_bit [13618.070111] ? __file_remove_privs+0x89/0x110 [13618.073158] pps_core [13618.077769] ? wake_bit_function+0x70/0x70 [13618.080039] dca fuse [13618.084565] ? ceph_write_iter+0x260/0x770 [ceph] [13618.088047] [13618.088047] irq event stamp: 32235630 [13618.092399] ceph_write_iter+0x260/0x770 [ceph] [13618.094668] hardirqs last enabled at (32235629): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13618.098780] ? lock_is_held_type+0xaa/0x120 [13618.101048] hardirqs last disabled at (32235630): [] sysvec_apic_timer_interrupt+0xe/0x90 [13618.105747] vfs_write+0x324/0x4f0 [13618.107235] softirqs last enabled at (32235628): [] __do_softirq+0x2be/0x371 [13618.110897] ksys_pwrite64+0x65/0xa0 [13618.115417] softirqs last disabled at (32235623): [] irq_exit_rcu+0xb0/0xd0 [13618.125574] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13618.129753] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13618.139477] do_syscall_64+0x38/0x80 [13618.142873] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13618.151558] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13618.155128] Workqueue: events delayed_work [ceph] [13618.163639] RIP: 0033:0x7f48f0eeb93b [13618.168769] [13618.168770] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [13618.178148] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13618.181719] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [13618.189188] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13618.194234] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [13618.198930] ORIG_RAX: 0000000000000012 [13618.202502] [13618.202503] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [13618.203992] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13618.209902] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [13618.228641] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13618.247377] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [13618.252596] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13618.257812] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13618.261645] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13618.263133] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13618.270259] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13618.277384] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13618.284512] [13618.291630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13618.350375] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13618.357500] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13618.364623] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13618.371747] Call Trace: [13618.374191] [13618.376204] ? watchdog_timer_fn+0x1b4/0x220 [13618.380469] ? __hrtimer_run_queues+0x1d3/0x360 [13618.384997] ? hrtimer_interrupt+0x104/0x240 [13618.389263] ? __local_bh_enable+0x50/0x60 [13618.393357] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13618.398580] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13618.403538] [13618.405634] [13618.407732] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13618.413041] ? ceph_check_delayed_caps+0x9e/0x270 [ceph] [13618.418358] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [13618.423674] delayed_work+0x1d8/0x280 [ceph] [13618.427979] ? process_scheduled_works+0x22a/0x510 [13618.432761] process_scheduled_works+0x2a4/0x510 [13618.437377] worker_thread+0x188/0x340 [13618.441128] ? rescuer_thread+0x310/0x310 [13618.445129] kthread+0xf5/0x130 [13618.448267] ? kthread_complete_and_exit+0x20/0x20 [13618.453052] ret_from_fork+0x31/0x40 [13618.456621] ? kthread_complete_and_exit+0x20/0x20 [13618.461407] ret_from_fork_asm+0x11/0x20 [13618.465330] [13636.457030] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [13636.465530] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13645.693783] watchdog: BUG: soft lockup - CPU#0 stuck for 1594s! [dbench:78683] [13645.700999] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13645.705782] watchdog: BUG: soft lockup - CPU#7 stuck for 10591s! [kworker/7:4:78808] [13645.756457] irq event stamp: 16427720 [13645.756458] hardirqs last enabled at (16427719): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13645.764187] Modules linked in: [13645.767846] hardirqs last disabled at (16427720): [] sysvec_apic_timer_interrupt+0xe/0x90 [13645.778003] ceph [13645.781054] softirqs last enabled at (16427632): [] __do_softirq+0x2be/0x371 [13645.790777] libceph [13645.792700] softirqs last disabled at (16427627): [] irq_exit_rcu+0xb0/0xd0 [13645.801385] dns_resolver fscache [13645.803571] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13645.812079] netfs veth [13645.815390] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13645.824334] nft_chain_nat xt_MASQUERADE [13645.826781] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13645.834250] nf_nat [13645.838178] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13645.843653] nf_conntrack nf_defrag_ipv6 [13645.845752] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13645.864488] nf_defrag_ipv4 nft_compat [13645.868405] [13645.873623] nf_tables nfnetlink [13645.877367] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13645.878859] bridge [13645.882082] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13645.889206] stp llc [13645.891303] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13645.898427] xfs libcrc32c [13645.900613] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13645.907735] sunrpc intel_rapl_msr [13645.910440] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13645.917566] intel_rapl_common [13645.920961] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13645.928086] sb_edac x86_pkg_temp_thermal [13645.931137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13645.939214] coretemp kvm_intel [13645.943217] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13645.948956] kvm irqbypass [13645.952092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13645.959217] i2c_i801 iTCO_wdt [13645.961920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13645.969045] iTCO_vendor_support i2c_smbus [13645.972095] Call Trace: [13645.979220] crct10dif_pclmul joydev [13645.983312] [13645.985754] crc32_pclmul ipmi_ssif [13645.989328] ? watchdog_timer_fn+0x1b4/0x220 [13645.991335] ghash_clmulni_intel [13645.994821] ? __hrtimer_run_queues+0x1d3/0x360 [13645.999085] mei_me mei [13646.002312] ? hrtimer_interrupt+0x104/0x240 [13646.006833] lpc_ich sha512_ssse3 [13646.009276] ? __local_bh_enable+0x50/0x60 [13646.013541] acpi_ipmi mfd_core [13646.016853] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13646.020941] wmi ipmi_si [13646.024080] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13646.029297] ipmi_devintf ipmi_msghandler [13646.031828] [13646.036785] ioatdma acpi_pad [13646.040789] [13646.040790] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13646.042886] acpi_power_meter ixgbe [13646.045855] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13646.047947] igb mdio [13646.053254] do_raw_spin_lock+0xb5/0xc0 [13646.056737] crc32c_intel nvme [13646.061609] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13646.063876] ptp nvme_core [13646.067711] try_get_cap_refs+0x159/0x880 [ceph] [13646.070759] i2c_algo_bit [13646.075550] __ceph_get_caps+0x4a1/0x760 [ceph] [13646.078247] pps_core [13646.082863] ? __file_remove_privs+0x89/0x110 [13646.085480] dca fuse [13646.090007] ? wake_bit_function+0x70/0x70 [13646.092278] [13646.096632] ? ceph_write_iter+0x260/0x770 [ceph] [13646.098901] irq event stamp: 32458242 [13646.102992] ceph_write_iter+0x260/0x770 [ceph] [13646.104480] hardirqs last enabled at (32458241): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13646.109186] ? lock_is_held_type+0xaa/0x120 [13646.112837] hardirqs last disabled at (32458242): [] sysvec_apic_timer_interrupt+0xe/0x90 [13646.117364] vfs_write+0x324/0x4f0 [13646.127517] softirqs last enabled at (32458240): [] __do_softirq+0x2be/0x371 [13646.131701] ksys_pwrite64+0x65/0xa0 [13646.141419] softirqs last disabled at (32458235): [] irq_exit_rcu+0xb0/0xd0 [13646.144819] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13646.153500] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13646.157072] do_syscall_64+0x38/0x80 [13646.165583] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13646.170715] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13646.180090] Workqueue: events delayed_work [ceph] [13646.183662] RIP: 0033:0x7f48f0eeb93b [13646.191132] [13646.191133] RIP: 0010:lock_acquire+0xf0/0x2b0 [13646.196185] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13646.200884] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13646.204465] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13646.205952] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13646.210303] ORIG_RAX: 0000000000000012 [13646.229042] [13646.229042] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13646.247780] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13646.252996] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13646.258214] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13646.262043] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13646.263535] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13646.270660] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13646.277783] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13646.284907] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13646.292032] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13646.299157] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13646.306285] [13646.313404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13646.350776] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13646.357899] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13646.365025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13646.372149] Call Trace: [13646.374595] [13646.376603] ? watchdog_timer_fn+0x1b4/0x220 [13646.380868] ? __hrtimer_run_queues+0x1d3/0x360 [13646.385395] ? hrtimer_interrupt+0x104/0x240 [13646.389663] ? __local_bh_enable+0x50/0x60 [13646.393756] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13646.398973] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13646.403930] [13646.406027] [13646.408124] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13646.413432] ? lock_acquire+0xf0/0x2b0 [13646.417189] ? igrab+0x43/0x50 [13646.420240] ? lock_release+0xc5/0x260 [13646.423985] _raw_spin_lock+0x30/0x40 [13646.427642] ? igrab+0x1d/0x50 [13646.430694] igrab+0x1d/0x50 [13646.433571] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13646.438721] delayed_work+0x1d8/0x280 [ceph] [13646.443003] ? process_scheduled_works+0x22a/0x510 [13646.447792] process_scheduled_works+0x2a4/0x510 [13646.452408] worker_thread+0x188/0x340 [13646.456156] ? rescuer_thread+0x310/0x310 [13646.460160] kthread+0xf5/0x130 [13646.463325] ? kthread_complete_and_exit+0x20/0x20 [13646.468110] ret_from_fork+0x31/0x40 [13646.471687] ? kthread_complete_and_exit+0x20/0x20 [13646.476471] ret_from_fork_asm+0x11/0x20 [13646.480396] [13647.209632] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13657.449828] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13673.694273] watchdog: BUG: soft lockup - CPU#0 stuck for 1620s! [dbench:78683] [13673.701485] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13673.706273] watchdog: BUG: soft lockup - CPU#7 stuck for 10617s! [kworker/7:4:78808] [13673.756925] irq event stamp: 16485652 [13673.764675] Modules linked in: ceph libceph [13673.768333] hardirqs last enabled at (16485651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13673.772508] dns_resolver fscache netfs [13673.782668] hardirqs last disabled at (16485652): [] sysvec_apic_timer_interrupt+0xe/0x90 [13673.786498] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [13673.796221] softirqs last enabled at (16485596): [] __do_softirq+0x2be/0x371 [13673.802306] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13673.810991] softirqs last disabled at (16485591): [] irq_exit_rcu+0xb0/0xd0 [13673.810992] nf_tables [13673.810994] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13673.816033] nfnetlink bridge [13673.824545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13673.826902] stp llc [13673.835847] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13673.838810] xfs [13673.846282] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13673.848465] libcrc32c sunrpc [13673.853959] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13673.855796] intel_rapl_msr intel_rapl_common [13673.874535] [13673.874536] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13673.877500] sb_edac x86_pkg_temp_thermal [13673.882718] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13673.887067] coretemp [13673.888559] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13673.895681] kvm_intel [13673.899687] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13673.906810] kvm irqbypass [13673.909080] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13673.916204] i2c_i801 iTCO_wdt [13673.918563] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13673.925686] iTCO_vendor_support i2c_smbus [13673.928391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13673.935516] crct10dif_pclmul joydev [13673.938566] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13673.946642] crc32_pclmul ipmi_ssif [13673.950733] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13673.956472] ghash_clmulni_intel mei_me [13673.960043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13673.967165] mei lpc_ich [13673.970668] Call Trace: [13673.970669] [13673.977792] sha512_ssse3 acpi_ipmi [13673.981623] ? watchdog_timer_fn+0x1b4/0x220 [13673.988748] mfd_core wmi [13673.991277] ? __hrtimer_run_queues+0x1d3/0x360 [13673.993720] ipmi_si ipmi_devintf [13673.995734] ? hrtimer_interrupt+0x104/0x240 [13673.999215] ipmi_msghandler ioatdma [13674.003479] ? __local_bh_enable+0x50/0x60 [13674.006097] acpi_pad acpi_power_meter [13674.010624] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13674.013933] ixgbe igb [13674.018215] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13674.021785] mdio [13674.025874] [13674.029620] crc32c_intel nvme [13674.034836] [13674.034837] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13674.037193] ptp nvme_core [13674.042156] ? queued_spin_lock_slowpath+0x18/0x2e0 [13674.044074] i2c_algo_bit pps_core [13674.046174] do_raw_spin_lock+0xb5/0xc0 [13674.049224] dca fuse [13674.051323] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13674.056626] irq event stamp: 32680222 [13674.059332] try_get_cap_refs+0x159/0x880 [ceph] [13674.064199] hardirqs last enabled at (32680221): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13674.067603] __ceph_get_caps+0x4a1/0x760 [ceph] [13674.071428] hardirqs last disabled at (32680222): [] sysvec_apic_timer_interrupt+0xe/0x90 [13674.073703] ? __file_remove_privs+0x89/0x110 [13674.078482] softirqs last enabled at (32680220): [] __do_softirq+0x2be/0x371 [13674.082141] ? wake_bit_function+0x70/0x70 [13674.086750] softirqs last disabled at (32680215): [] irq_exit_rcu+0xb0/0xd0 [13674.096912] ? ceph_write_iter+0x260/0x770 [ceph] [13674.101432] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13674.111157] ceph_write_iter+0x260/0x770 [ceph] [13674.115507] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13674.124195] ? lock_is_held_type+0xaa/0x120 [13674.128282] Workqueue: events delayed_work [ceph] [13674.136796] vfs_write+0x324/0x4f0 [13674.141489] [13674.150874] ksys_pwrite64+0x65/0xa0 [13674.155393] RIP: 0010:lock_acquire+0xf0/0x2b0 [13674.162864] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13674.167041] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13674.171740] do_syscall_64+0x38/0x80 [13674.175135] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13674.176627] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13674.180198] [13674.180199] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13674.184547] RIP: 0033:0x7f48f0eeb93b [13674.189679] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13674.208416] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13674.211988] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13674.217205] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13674.222248] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13674.223738] ORIG_RAX: 0000000000000012 [13674.230862] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13674.234434] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13674.241559] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13674.260295] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13674.267418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13674.272636] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13674.279761] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13674.283591] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13674.290715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13674.297840] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13674.305917] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13674.313047] [13674.318778] Call Trace: [13674.318779] [13674.368164] ? watchdog_timer_fn+0x1b4/0x220 [13674.372427] ? __hrtimer_run_queues+0x1d3/0x360 [13674.376960] ? hrtimer_interrupt+0x104/0x240 [13674.381225] ? __local_bh_enable+0x50/0x60 [13674.385318] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13674.390541] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13674.395499] [13674.397595] [13674.399695] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13674.405002] ? lock_acquire+0xf0/0x2b0 [13674.408751] ? igrab+0x43/0x50 [13674.411802] ? lock_release+0xc5/0x260 [13674.415547] _raw_spin_lock+0x30/0x40 [13674.419210] ? igrab+0x1d/0x50 [13674.422262] igrab+0x1d/0x50 [13674.425139] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13674.430290] delayed_work+0x1d8/0x280 [ceph] [13674.434569] ? process_scheduled_works+0x22a/0x510 [13674.439353] process_scheduled_works+0x2a4/0x510 [13674.443968] worker_thread+0x188/0x340 [13674.447716] ? rescuer_thread+0x310/0x310 [13674.451721] kthread+0xf5/0x130 [13674.454857] ? kthread_complete_and_exit+0x20/0x20 [13674.459643] ret_from_fork+0x31/0x40 [13674.463221] ? kthread_complete_and_exit+0x20/0x20 [13674.468005] ret_from_fork_asm+0x11/0x20 [13674.471929] [13689.193954] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [13701.694764] watchdog: BUG: soft lockup - CPU#0 stuck for 1646s! [dbench:78683] [13701.701981] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13701.706764] watchdog: BUG: soft lockup - CPU#7 stuck for 10643s! [kworker/7:4:78808] [13701.757438] irq event stamp: 16543560 [13701.757439] hardirqs last enabled at (16543559): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13701.765170] Modules linked in: [13701.768827] hardirqs last disabled at (16543560): [] sysvec_apic_timer_interrupt+0xe/0x90 [13701.778984] ceph [13701.782036] softirqs last enabled at (16543514): [] __do_softirq+0x2be/0x371 [13701.791761] libceph [13701.793683] softirqs last disabled at (16543509): [] irq_exit_rcu+0xb0/0xd0 [13701.802367] dns_resolver [13701.804553] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13701.813064] fscache [13701.815679] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13701.815680] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13701.824625] netfs [13701.826809] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13701.834281] veth [13701.839756] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13701.841768] nft_chain_nat [13701.860505] [13701.860506] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13701.862428] xt_MASQUERADE nf_nat [13701.867664] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13701.870368] nf_conntrack [13701.871859] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13701.878982] nf_defrag_ipv6 [13701.882294] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13701.889416] nf_defrag_ipv4 [13701.892052] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13701.899179] nft_compat [13701.901968] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13701.909091] nf_tables [13701.911882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13701.919006] nfnetlink [13701.921452] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13701.929527] bridge [13701.931885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13701.937621] stp llc [13701.939980] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13701.947106] xfs [13701.949202] Call Trace: [13701.956324] libcrc32c [13701.958512] [13701.965634] sunrpc [13701.967475] ? watchdog_timer_fn+0x1b4/0x220 [13701.969915] intel_rapl_msr [13701.972273] ? __hrtimer_run_queues+0x1d3/0x360 [13701.974284] intel_rapl_common [13701.976385] ? hrtimer_interrupt+0x104/0x240 [13701.980644] sb_edac [13701.983443] ? __local_bh_enable+0x50/0x60 [13701.987970] x86_pkg_temp_thermal [13701.991020] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13701.995282] coretemp kvm_intel [13701.997468] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13702.001558] kvm [13702.004869] [13702.010086] irqbypass i2c_i801 [13702.013223] [13702.018179] iTCO_wdt [13702.020019] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13702.022116] iTCO_vendor_support [13702.025258] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13702.027350] i2c_smbus crct10dif_pclmul [13702.029624] do_raw_spin_lock+0xb5/0xc0 [13702.034926] joydev [13702.038167] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13702.043039] crc32_pclmul [13702.046872] try_get_cap_refs+0x159/0x880 [ceph] [13702.050698] ipmi_ssif ghash_clmulni_intel [13702.052802] __ceph_get_caps+0x4a1/0x760 [ceph] [13702.057579] mei_me mei [13702.060203] ? __file_remove_privs+0x89/0x110 [13702.064808] lpc_ich sha512_ssse3 [13702.068900] ? wake_bit_function+0x70/0x70 [13702.073424] acpi_ipmi [13702.075871] ? ceph_write_iter+0x260/0x770 [ceph] [13702.080220] mfd_core [13702.083528] ceph_write_iter+0x260/0x770 [ceph] [13702.087621] wmi [13702.089980] ? lock_is_held_type+0xaa/0x120 [13702.094675] ipmi_si [13702.096948] vfs_write+0x324/0x4f0 [13702.101468] ipmi_devintf ipmi_msghandler [13702.103312] ksys_pwrite64+0x65/0xa0 [13702.107483] ioatdma acpi_pad [13702.109669] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13702.113065] acpi_power_meter ixgbe [13702.117069] do_syscall_64+0x38/0x80 [13702.120642] igb [13702.123606] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13702.128735] mdio crc32c_intel [13702.132220] RIP: 0033:0x7f48f0eeb93b [13702.135789] nvme ptp [13702.137628] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13702.142670] nvme_core i2c_algo_bit [13702.145722] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13702.149292] pps_core dca [13702.151562] ORIG_RAX: 0000000000000012 [13702.170301] fuse [13702.173785] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13702.179002] irq event stamp: 32902860 [13702.181618] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13702.185449] hardirqs last enabled at (32902859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13702.187374] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13702.194498] hardirqs last disabled at (32902860): [] sysvec_apic_timer_interrupt+0xe/0x90 [13702.198167] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13702.205287] softirqs last enabled at (32902858): [] __do_softirq+0x2be/0x371 [13702.215446] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13702.222570] softirqs last disabled at (32902853): [] irq_exit_rcu+0xb0/0xd0 [13702.232301] [13702.239420] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13702.275301] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13702.282771] Workqueue: events delayed_work [ceph] [13702.287486] RIP: 0010:lock_release+0x42/0x260 [13702.291837] Code: 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 <8b> 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 [13702.310573] RSP: 0018:ffffc90002117d48 EFLAGS: 00000202 [13702.315791] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [13702.322916] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [13702.330040] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [13702.337169] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [13702.344295] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13702.351419] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13702.359499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13702.365234] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13702.372360] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13702.379484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13702.386607] Call Trace: [13702.389053] [13702.391064] ? watchdog_timer_fn+0x1b4/0x220 [13702.395327] ? __hrtimer_run_queues+0x1d3/0x360 [13702.399854] ? hrtimer_interrupt+0x104/0x240 [13702.404124] ? __local_bh_enable+0x50/0x60 [13702.408217] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13702.413450] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13702.418408] [13702.420504] [13702.422602] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13702.427905] ? igrab+0x43/0x50 [13702.430957] ? igrab+0x43/0x50 [13702.434008] ? lock_release+0x42/0x260 [13702.437752] ? lock_release+0x1d7/0x260 [13702.441582] _raw_spin_unlock+0x1b/0x40 [13702.445414] igrab+0x43/0x50 [13702.448291] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13702.453453] delayed_work+0x1d8/0x280 [ceph] [13702.457730] ? process_scheduled_works+0x22a/0x510 [13702.462521] process_scheduled_works+0x2a4/0x510 [13702.467138] worker_thread+0x188/0x340 [13702.470885] ? rescuer_thread+0x310/0x310 [13702.474888] kthread+0xf5/0x130 [13702.478027] ? kthread_complete_and_exit+0x20/0x20 [13702.482811] ret_from_fork+0x31/0x40 [13702.486390] ? kthread_complete_and_exit+0x20/0x20 [13702.491175] ret_from_fork_asm+0x11/0x20 [13702.495107] [13729.695256] watchdog: BUG: soft lockup - CPU#0 stuck for 1672s! [dbench:78683] [13729.702466] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13729.707255] watchdog: BUG: soft lockup - CPU#7 stuck for 10669s! [kworker/7:4:78808] [13729.757909] irq event stamp: 16601668 [13729.765657] Modules linked in: ceph libceph [13729.769313] hardirqs last enabled at (16601667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13729.773492] dns_resolver fscache [13729.783652] hardirqs last disabled at (16601668): [] sysvec_apic_timer_interrupt+0xe/0x90 [13729.786969] netfs veth nft_chain_nat [13729.796692] softirqs last enabled at (16601592): [] __do_softirq+0x2be/0x371 [13729.796694] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [13729.809035] softirqs last disabled at (16601587): [] irq_exit_rcu+0xb0/0xd0 [13729.809038] nf_tables nfnetlink bridge stp [13729.825537] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13729.829713] llc xfs [13729.838657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13729.838658] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13729.840841] libcrc32c sunrpc [13729.848313] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13729.853790] intel_rapl_msr intel_rapl_common [13729.856754] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13729.875491] sb_edac x86_pkg_temp_thermal [13729.879842] [13729.879843] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13729.885060] coretemp kvm_intel [13729.889063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13729.890554] kvm irqbypass [13729.897678] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13729.900816] i2c_i801 iTCO_wdt [13729.907939] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13729.910653] iTCO_vendor_support [13729.917778] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13729.920826] i2c_smbus [13729.927951] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13729.931174] crct10dif_pclmul joydev [13729.938299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13729.940658] crc32_pclmul ipmi_ssif [13729.948735] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13729.952305] ghash_clmulni_intel [13729.958043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13729.961527] mei_me [13729.968653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13729.971875] mei lpc_ich [13729.978999] Call Trace: [13729.979000] [13729.981098] sha512_ssse3 acpi_ipmi [13729.988223] ? watchdog_timer_fn+0x1b4/0x220 [13729.990752] mfd_core wmi [13729.993198] ? __hrtimer_run_queues+0x1d3/0x360 [13729.995206] ipmi_si ipmi_devintf [13729.998692] ? hrtimer_interrupt+0x104/0x240 [13730.002954] ipmi_msghandler [13730.005570] ? __local_bh_enable+0x50/0x60 [13730.010095] ioatdma acpi_pad [13730.013409] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13730.017670] acpi_power_meter [13730.020549] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13730.024653] ixgbe igb [13730.027621] [13730.032837] mdio crc32c_intel [13730.035802] [13730.035803] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13730.040758] nvme ptp [13730.043121] ? queued_spin_lock_slowpath+0x18/0x2e0 [13730.045213] nvme_core i2c_algo_bit [13730.048267] do_raw_spin_lock+0xb5/0xc0 [13730.050361] pps_core dca fuse [13730.055667] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13730.057943] [13730.057944] irq event stamp: 33125620 [13730.062818] try_get_cap_refs+0x159/0x880 [ceph] [13730.066299] hardirqs last enabled at (33125619): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13730.070137] __ceph_get_caps+0x4a1/0x760 [ceph] [13730.073182] hardirqs last disabled at (33125620): [] sysvec_apic_timer_interrupt+0xe/0x90 [13730.077970] ? __file_remove_privs+0x89/0x110 [13730.079456] softirqs last enabled at (33125618): [] __do_softirq+0x2be/0x371 [13730.083114] ? wake_bit_function+0x70/0x70 [13730.087723] softirqs last disabled at (33125613): [] irq_exit_rcu+0xb0/0xd0 [13730.097886] ? ceph_write_iter+0x260/0x770 [ceph] [13730.102407] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13730.112132] ceph_write_iter+0x260/0x770 [ceph] [13730.116480] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13730.125170] ? lock_is_held_type+0xaa/0x120 [13730.129257] Workqueue: events delayed_work [ceph] [13730.137770] vfs_write+0x324/0x4f0 [13730.142464] [13730.142465] RIP: 0010:rcu_is_watching+0x29/0x40 [13730.151847] ksys_pwrite64+0x65/0xa0 [13730.156365] Code: 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e <74> 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [13730.163837] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13730.168014] RSP: 0018:ffffc90002117d40 EFLAGS: 00000202 [13730.172713] do_syscall_64+0x38/0x80 [13730.176108] [13730.177600] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13730.182123] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [13730.185695] RIP: 0033:0x7f48f0eeb93b [13730.204433] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [13730.209564] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13730.214781] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [13730.218351] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13730.219842] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [13730.224886] ORIG_RAX: 0000000000000012 [13730.232011] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13730.235582] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13730.242703] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13730.261441] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13730.268566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13730.273785] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13730.280907] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13730.284737] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13730.291862] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13730.298988] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13730.307064] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13730.314194] [13730.319926] Call Trace: [13730.319943] [13730.369329] ? watchdog_timer_fn+0x1b4/0x220 [13730.373592] ? __hrtimer_run_queues+0x1d3/0x360 [13730.378118] ? hrtimer_interrupt+0x104/0x240 [13730.382379] ? __local_bh_enable+0x50/0x60 [13730.386473] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13730.391696] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13730.396663] [13730.398759] [13730.400858] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13730.406161] ? igrab+0x43/0x50 [13730.409215] ? igrab+0x43/0x50 [13730.412262] ? rcu_is_watching+0x29/0x40 [13730.416180] lock_release+0x1d7/0x260 [13730.419841] _raw_spin_unlock+0x1b/0x40 [13730.423676] igrab+0x43/0x50 [13730.426555] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13730.431706] delayed_work+0x1d8/0x280 [ceph] [13730.435985] ? process_scheduled_works+0x22a/0x510 [13730.440778] process_scheduled_works+0x2a4/0x510 [13730.445391] worker_thread+0x188/0x340 [13730.449142] ? rescuer_thread+0x310/0x310 [13730.453143] kthread+0xf5/0x130 [13730.456282] ? kthread_complete_and_exit+0x20/0x20 [13730.461065] ret_from_fork+0x31/0x40 [13730.464663] ? kthread_complete_and_exit+0x20/0x20 [13730.469447] ret_from_fork_asm+0x11/0x20 [13730.473370] [13757.695747] watchdog: BUG: soft lockup - CPU#0 stuck for 1698s! [dbench:78683] [13757.702961] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13757.707747] watchdog: BUG: soft lockup - CPU#7 stuck for 10695s! [kworker/7:4:78808] [13757.758419] irq event stamp: 16659592 [13757.758421] hardirqs last enabled at (16659591): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13757.766152] Modules linked in: [13757.769810] hardirqs last disabled at (16659592): [] sysvec_apic_timer_interrupt+0xe/0x90 [13757.779966] ceph [13757.783017] softirqs last enabled at (16659546): [] __do_softirq+0x2be/0x371 [13757.792741] libceph [13757.794664] softirqs last disabled at (16659541): [] irq_exit_rcu+0xb0/0xd0 [13757.803350] dns_resolver fscache [13757.805534] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13757.814045] netfs veth [13757.817354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13757.817356] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13757.826300] nft_chain_nat xt_MASQUERADE [13757.828743] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13757.836214] nf_nat [13757.841692] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13757.845609] nf_conntrack nf_defrag_ipv6 [13757.864348] [13757.864348] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13757.866445] nf_defrag_ipv4 [13757.871660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13757.875578] nft_compat [13757.877068] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13757.884194] nf_tables [13757.886984] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13757.894108] nfnetlink [13757.896553] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13757.903678] bridge stp [13757.906033] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13757.913158] llc xfs [13757.915514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13757.922641] libcrc32c [13757.925083] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13757.933161] sunrpc [13757.935346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13757.941083] intel_rapl_msr intel_rapl_common [13757.943440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13757.950564] sb_edac [13757.952662] Call Trace: [13757.952663] [13757.959786] x86_pkg_temp_thermal coretemp [13757.964144] ? watchdog_timer_fn+0x1b4/0x220 [13757.971269] kvm_intel kvm [13757.973453] ? __hrtimer_run_queues+0x1d3/0x360 [13757.975897] irqbypass i2c_i801 [13757.977911] ? hrtimer_interrupt+0x104/0x240 [13757.981999] iTCO_wdt iTCO_vendor_support [13757.986263] ? __local_bh_enable+0x50/0x60 [13757.988967] i2c_smbus crct10dif_pclmul [13757.993495] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13757.996630] joydev crc32_pclmul [13758.000893] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13758.004897] ipmi_ssif [13758.008989] [13758.012819] ghash_clmulni_intel mei_me [13758.018034] [13758.018036] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13758.021259] mei [13758.026221] ? queued_spin_lock_slowpath+0x16/0x2e0 [13758.028574] lpc_ich sha512_ssse3 [13758.030674] do_raw_spin_lock+0xb5/0xc0 [13758.034503] acpi_ipmi mfd_core [13758.036601] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13758.041903] wmi ipmi_si [13758.043742] try_get_cap_refs+0x159/0x880 [ceph] [13758.048611] ipmi_devintf ipmi_msghandler [13758.051928] __ceph_get_caps+0x4a1/0x760 [ceph] [13758.055753] ioatdma acpi_pad [13758.058894] ? __file_remove_privs+0x89/0x110 [13758.063675] acpi_power_meter ixgbe [13758.066207] ? wake_bit_function+0x70/0x70 [13758.070815] igb mdio [13758.074823] ? ceph_write_iter+0x260/0x770 [ceph] [13758.079344] crc32c_intel [13758.082308] ceph_write_iter+0x260/0x770 [ceph] [13758.086660] nvme [13758.090148] ? lock_is_held_type+0xaa/0x120 [13758.094235] ptp nvme_core [13758.096509] vfs_write+0x324/0x4f0 [13758.101202] i2c_algo_bit pps_core [13758.103823] ksys_pwrite64+0x65/0xa0 [13758.108343] dca [13758.110267] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13758.114446] fuse [13758.117150] do_syscall_64+0x38/0x80 [13758.120546] [13758.120547] irq event stamp: 33348172 [13758.123944] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13758.127514] hardirqs last enabled at (33348171): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13758.129352] RIP: 0033:0x7f48f0eeb93b [13758.134483] hardirqs last disabled at (33348172): [] sysvec_apic_timer_interrupt+0xe/0x90 [13758.136424] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13758.139996] softirqs last enabled at (33348170): [] __do_softirq+0x2be/0x371 [13758.141485] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13758.145149] softirqs last disabled at (33348165): [] irq_exit_rcu+0xb0/0xd0 [13758.150197] ORIG_RAX: 0000000000000012 [13758.160355] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13758.163923] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13758.173648] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13758.192387] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13758.201070] Workqueue: events delayed_work [ceph] [13758.206288] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13758.214798] [13758.214799] RIP: 0010:lock_acquire+0xc/0x2b0 [13758.218629] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13758.228007] Code: 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 <41> 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 [13758.235149] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13758.242619] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [13758.249750] [13758.254440] [13758.254441] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13758.316321] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13758.323447] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [13758.330571] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [13758.337695] R13: ffff88817f867478 R14: 00000001001d239b R15: 0000000000000000 [13758.344819] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13758.352895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13758.358634] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13758.365757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13758.372881] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13758.380005] Call Trace: [13758.382451] [13758.384461] ? watchdog_timer_fn+0x1b4/0x220 [13758.388726] ? __hrtimer_run_queues+0x1d3/0x360 [13758.393253] ? hrtimer_interrupt+0x104/0x240 [13758.397520] ? __local_bh_enable+0x50/0x60 [13758.401614] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13758.406831] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13758.411786] [13758.413884] [13758.415983] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13758.421292] ? lock_acquire+0xc/0x2b0 [13758.424954] _raw_spin_lock+0x30/0x40 [13758.428611] ? igrab+0x1d/0x50 [13758.431660] igrab+0x1d/0x50 [13758.434537] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13758.439689] delayed_work+0x1d8/0x280 [ceph] [13758.443969] ? process_scheduled_works+0x22a/0x510 [13758.448760] process_scheduled_works+0x2a4/0x510 [13758.453375] worker_thread+0x188/0x340 [13758.457124] ? rescuer_thread+0x310/0x310 [13758.461155] kthread+0xf5/0x130 [13758.464292] ? kthread_complete_and_exit+0x20/0x20 [13758.469075] ret_from_fork+0x31/0x40 [13758.472645] ? kthread_complete_and_exit+0x20/0x20 [13758.477445] ret_from_fork_asm+0x11/0x20 [13758.481369] [13760.455162] rcu: INFO: rcu_preempt self-detected stall on CPU [13760.460921] rcu: 7-....: (1776014 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/1774856 fqs=866661 [13760.471259] rcu: (t=1823812 jiffies g=1309609 q=135565 ncpus=8) [13760.477257] Sending NMI from CPU 7 to CPUs 0: [13760.481608] NMI backtrace for cpu 0 [13760.481610] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13760.481612] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13760.481613] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13760.481616] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13760.481618] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13760.481619] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13760.481620] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13760.481621] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13760.481622] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13760.481622] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13760.481623] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13760.481624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13760.481625] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13760.481626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13760.481626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13760.481627] Call Trace: [13760.481628] [13760.481629] ? nmi_cpu_backtrace+0xad/0x120 [13760.481633] ? nmi_cpu_backtrace_handler+0x11/0x20 [13760.481634] ? nmi_handle+0xd4/0x260 [13760.481639] ? default_do_nmi+0x49/0xf0 [13760.481640] ? exc_nmi+0x11e/0x150 [13760.481642] ? end_repeat_nmi+0x16/0x67 [13760.481646] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13760.481648] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13760.481651] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13760.481653] [13760.481654] [13760.481654] do_raw_spin_lock+0xb5/0xc0 [13760.481657] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13760.481671] try_get_cap_refs+0x159/0x880 [ceph] [13760.481686] __ceph_get_caps+0x4a1/0x760 [ceph] [13760.481698] ? __file_remove_privs+0x89/0x110 [13760.481700] ? wake_bit_function+0x70/0x70 [13760.481703] ? ceph_write_iter+0x260/0x770 [ceph] [13760.481713] ceph_write_iter+0x260/0x770 [ceph] [13760.481725] ? lock_is_held_type+0xaa/0x120 [13760.481727] vfs_write+0x324/0x4f0 [13760.481732] ksys_pwrite64+0x65/0xa0 [13760.481734] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13760.481736] do_syscall_64+0x38/0x80 [13760.481739] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13760.481741] RIP: 0033:0x7f48f0eeb93b [13760.481742] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13760.481743] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [13760.481744] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13760.481745] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13760.481746] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13760.481747] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13760.481747] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13760.481750] [13760.482607] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13760.786507] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13760.793976] Workqueue: events delayed_work [ceph] [13760.798692] RIP: 0010:lock_acquire+0xf0/0x2b0 [13760.803042] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13760.821780] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13760.826998] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13760.834120] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13760.841246] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13760.848371] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13760.855493] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13760.862618] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13760.870697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13760.876432] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13760.883559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13760.890682] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13760.897804] Call Trace: [13760.900249] [13760.902262] ? rcu_dump_cpu_stacks+0xd1/0x180 [13760.906622] ? rcu_sched_clock_irq+0x633/0x1390 [13760.911143] ? lock_release+0xc5/0x260 [13760.914887] ? lock_is_held_type+0xaa/0x120 [13760.919067] ? lock_is_held_type+0xaa/0x120 [13760.923245] ? tick_sched_do_timer+0xa0/0xa0 [13760.927516] ? rcu_read_lock_sched_held+0x56/0x90 [13760.932213] ? tick_sched_do_timer+0xa0/0xa0 [13760.936478] ? update_process_times+0x5f/0x90 [13760.940828] ? tick_sched_handle.isra.16+0x1e/0x50 [13760.945613] ? tick_sched_timer+0x6f/0x80 [13760.949616] ? __hrtimer_run_queues+0x1d3/0x360 [13760.954143] ? hrtimer_interrupt+0x104/0x240 [13760.958413] ? __local_bh_enable+0x50/0x60 [13760.962505] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13760.967729] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13760.972686] [13760.974783] [13760.976882] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13760.982190] ? lock_acquire+0xf0/0x2b0 [13760.985940] ? igrab+0x43/0x50 [13760.988989] ? lock_release+0xc5/0x260 [13760.992734] _raw_spin_lock+0x30/0x40 [13760.996391] ? igrab+0x1d/0x50 [13760.999442] igrab+0x1d/0x50 [13761.002318] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13761.007469] delayed_work+0x1d8/0x280 [ceph] [13761.011751] ? process_scheduled_works+0x22a/0x510 [13761.016542] process_scheduled_works+0x2a4/0x510 [13761.021156] worker_thread+0x188/0x340 [13761.024905] ? rescuer_thread+0x310/0x310 [13761.028910] kthread+0xf5/0x130 [13761.032047] ? kthread_complete_and_exit+0x20/0x20 [13761.036830] ret_from_fork+0x31/0x40 [13761.040401] ? kthread_complete_and_exit+0x20/0x20 [13761.045186] ret_from_fork_asm+0x11/0x20 [13761.049116] [13785.696238] watchdog: BUG: soft lockup - CPU#0 stuck for 1724s! [dbench:78683] [13785.703459] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13785.708237] watchdog: BUG: soft lockup - CPU#7 stuck for 10721s! [kworker/7:4:78808] [13785.758910] irq event stamp: 16717652 [13785.766640] Modules linked in: ceph libceph dns_resolver fscache netfs [13785.770295] hardirqs last enabled at (16717651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13785.776821] veth nft_chain_nat xt_MASQUERADE nf_nat [13785.786979] hardirqs last disabled at (16717652): [] sysvec_apic_timer_interrupt+0xe/0x90 [13785.791937] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [13785.801660] softirqs last enabled at (16717632): [] __do_softirq+0x2be/0x371 [13785.810173] stp llc xfs libcrc32c sunrpc intel_rapl_msr [13785.818855] softirqs last disabled at (16717627): [] irq_exit_rcu+0xb0/0xd0 [13785.818859] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [13785.832673] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13785.832674] kvm [13785.832675] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13785.839882] irqbypass i2c_i801 [13785.848826] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13785.850663] iTCO_wdt iTCO_vendor_support [13785.858134] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13785.861273] i2c_smbus crct10dif_pclmul [13785.866749] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13785.870751] joydev crc32_pclmul [13785.889490] [13785.889491] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13785.893322] ipmi_ssif [13785.898540] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13785.901765] ghash_clmulni_intel [13785.903252] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13785.910378] mei_me [13785.912736] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13785.919859] mei lpc_ich [13785.923082] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13785.930208] sha512_ssse3 acpi_ipmi [13785.932303] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13785.939430] mfd_core wmi [13785.941959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13785.949083] ipmi_si ipmi_devintf [13785.952567] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13785.960645] ipmi_msghandler [13785.963263] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13785.968999] ioatdma acpi_pad [13785.972310] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13785.979434] acpi_power_meter [13785.982312] Call Trace: [13785.989436] ixgbe igb [13785.992403] [13785.999525] mdio crc32c_intel [13786.002492] ? watchdog_timer_fn+0x1b4/0x220 [13786.004933] nvme [13786.007292] ? __hrtimer_run_queues+0x1d3/0x360 [13786.009300] ptp [13786.012355] ? hrtimer_interrupt+0x104/0x240 [13786.016633] nvme_core [13786.018558] ? __local_bh_enable+0x50/0x60 [13786.023080] i2c_algo_bit pps_core [13786.024920] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13786.029182] dca fuse [13786.031541] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13786.035634] [13786.035635] irq event stamp: 33566022 [13786.039029] [13786.044247] hardirqs last enabled at (33566021): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13786.046517] [13786.051473] hardirqs last disabled at (33566022): [] sysvec_apic_timer_interrupt+0xe/0x90 [13786.052965] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13786.056636] softirqs last enabled at (33566020): [] __do_softirq+0x2be/0x371 [13786.058740] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13786.068894] softirqs last disabled at (33566015): [] irq_exit_rcu+0xb0/0xd0 [13786.070994] do_raw_spin_lock+0xb5/0xc0 [13786.080716] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13786.086023] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13786.094703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13786.099577] try_get_cap_refs+0x159/0x880 [ceph] [13786.108085] Workqueue: events delayed_work [ceph] [13786.111925] __ceph_get_caps+0x4a1/0x760 [ceph] [13786.121293] [13786.121294] RIP: 0010:lock_acquire+0xf0/0x2b0 [13786.126083] ? __file_remove_privs+0x89/0x110 [13786.133548] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13786.138159] ? wake_bit_function+0x70/0x70 [13786.142857] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13786.147382] ? ceph_write_iter+0x260/0x770 [ceph] [13786.148872] [13786.148872] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13786.153222] ceph_write_iter+0x260/0x770 [ceph] [13786.157573] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13786.176314] ? lock_is_held_type+0xaa/0x120 [13786.180400] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13786.185638] vfs_write+0x324/0x4f0 [13786.190334] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13786.191829] ksys_pwrite64+0x65/0xa0 [13786.198949] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13786.203474] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13786.210598] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13786.214775] do_syscall_64+0x38/0x80 [13786.221898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13786.221916] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13786.225315] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13786.232437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13786.236010] RIP: 0033:0x7f48f0eeb93b [13786.243132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13786.248265] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13786.256340] Call Trace: [13786.259925] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13786.265667] [13786.272789] ORIG_RAX: 0000000000000012 [13786.277833] ? watchdog_timer_fn+0x1b4/0x220 [13786.284957] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13786.288529] ? __hrtimer_run_queues+0x1d3/0x360 [13786.295651] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13786.314395] ? hrtimer_interrupt+0x104/0x240 [13786.316834] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13786.322051] ? __local_bh_enable+0x50/0x60 [13786.324061] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13786.327897] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13786.332156] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13786.339281] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13786.343812] [13786.350930] [13786.395113] [13786.397211] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13786.402519] ? lock_acquire+0xf0/0x2b0 [13786.406268] ? igrab+0x43/0x50 [13786.409319] ? lock_release+0xc5/0x260 [13786.413063] _raw_spin_lock+0x30/0x40 [13786.416720] ? igrab+0x1d/0x50 [13786.419772] igrab+0x1d/0x50 [13786.422651] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13786.427800] delayed_work+0x1d8/0x280 [ceph] [13786.432080] ? process_scheduled_works+0x22a/0x510 [13786.436871] process_scheduled_works+0x2a4/0x510 [13786.441486] worker_thread+0x188/0x340 [13786.445235] ? rescuer_thread+0x310/0x310 [13786.449238] kthread+0xf5/0x130 [13786.452376] ? kthread_complete_and_exit+0x20/0x20 [13786.457161] ret_from_fork+0x31/0x40 [13786.460739] ? kthread_complete_and_exit+0x20/0x20 [13786.465524] ret_from_fork_asm+0x11/0x20 [13786.469456] [13813.696726] watchdog: BUG: soft lockup - CPU#0 stuck for 1751s! [dbench:78683] [13813.703936] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13813.708726] watchdog: BUG: soft lockup - CPU#7 stuck for 10748s! [kworker/7:4:78808] [13813.759377] irq event stamp: 16775512 [13813.759379] hardirqs last enabled at (16775511): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13813.767126] Modules linked in: [13813.770782] hardirqs last disabled at (16775512): [] sysvec_apic_timer_interrupt+0xe/0x90 [13813.780939] ceph [13813.783992] softirqs last enabled at (16775424): [] __do_softirq+0x2be/0x371 [13813.793716] libceph [13813.795638] softirqs last disabled at (16775419): [] irq_exit_rcu+0xb0/0xd0 [13813.804324] dns_resolver fscache [13813.806509] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13813.815019] netfs veth [13813.818328] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13813.827273] nft_chain_nat xt_MASQUERADE [13813.829719] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13813.837189] nf_nat nf_conntrack [13813.841120] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13813.846599] nf_defrag_ipv6 nf_defrag_ipv4 [13813.849823] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13813.868561] nft_compat nf_tables [13813.872652] [13813.872653] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13813.877870] nfnetlink bridge [13813.881180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13813.882670] stp llc [13813.889796] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13813.892760] xfs libcrc32c [13813.899883] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13813.902067] sunrpc intel_rapl_msr [13813.909193] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13813.911896] intel_rapl_common sb_edac [13813.919021] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13813.922417] x86_pkg_temp_thermal coretemp [13813.929541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13813.933287] kvm_intel kvm [13813.941363] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13813.945455] irqbypass i2c_i801 [13813.951192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13813.953894] iTCO_wdt iTCO_vendor_support [13813.961019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13813.964157] i2c_smbus crct10dif_pclmul [13813.971281] Call Trace: [13813.975284] joydev crc32_pclmul [13813.982409] [13813.986240] ipmi_ssif ghash_clmulni_intel [13813.988685] ? watchdog_timer_fn+0x1b4/0x220 [13813.991909] mei_me mei [13813.993919] ? __hrtimer_run_queues+0x1d3/0x360 [13813.998010] lpc_ich sha512_ssse3 [13814.002278] ? hrtimer_interrupt+0x104/0x240 [13814.004718] acpi_ipmi mfd_core [13814.009242] ? __local_bh_enable+0x50/0x60 [13814.012551] wmi ipmi_si [13814.016820] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13814.019955] ipmi_devintf ipmi_msghandler [13814.024046] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13814.026576] ioatdma acpi_pad [13814.031793] [13814.035798] acpi_power_meter ixgbe [13814.040753] [13814.043719] igb mdio [13814.045817] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13814.049300] crc32c_intel nvme [13814.051410] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13814.053675] ptp nvme_core [13814.058982] do_raw_spin_lock+0xb5/0xc0 [13814.062031] i2c_algo_bit [13814.066903] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13814.069606] pps_core dca [13814.073439] try_get_cap_refs+0x159/0x880 [ceph] [13814.076056] fuse [13814.080843] __ceph_get_caps+0x4a1/0x760 [ceph] [13814.083455] irq event stamp: 33789000 [13814.088071] ? __file_remove_privs+0x89/0x110 [13814.089991] hardirqs last enabled at (33788999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13814.094514] ? wake_bit_function+0x70/0x70 [13814.098173] hardirqs last disabled at (33789000): [] sysvec_apic_timer_interrupt+0xe/0x90 [13814.102526] ? ceph_write_iter+0x260/0x770 [ceph] [13814.112680] softirqs last enabled at (33788998): [] __do_softirq+0x2be/0x371 [13814.116772] ceph_write_iter+0x260/0x770 [ceph] [13814.126495] softirqs last disabled at (33788993): [] irq_exit_rcu+0xb0/0xd0 [13814.131196] ? lock_is_held_type+0xaa/0x120 [13814.139876] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13814.144410] vfs_write+0x324/0x4f0 [13814.152920] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13814.157104] ksys_pwrite64+0x65/0xa0 [13814.166477] Workqueue: events delayed_work [ceph] [13814.169872] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13814.177343] [13814.180916] do_syscall_64+0x38/0x80 [13814.185612] RIP: 0010:lock_acquire+0xf0/0x2b0 [13814.190744] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13814.192234] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13814.195804] RIP: 0033:0x7f48f0eeb93b [13814.200154] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13814.205199] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13814.223936] [13814.223937] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13814.227507] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13814.232724] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13814.251463] ORIG_RAX: 0000000000000012 [13814.252952] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13814.260077] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13814.265294] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13814.272417] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13814.276249] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13814.283374] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13814.290497] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13814.297621] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13814.304747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13814.311871] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13814.318995] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13814.327078] [13814.334196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13814.363490] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13814.370614] Call Trace: [13814.373058] [13814.375070] ? watchdog_timer_fn+0x1b4/0x220 [13814.379335] ? __hrtimer_run_queues+0x1d3/0x360 [13814.383862] ? hrtimer_interrupt+0x104/0x240 [13814.388131] ? __local_bh_enable+0x50/0x60 [13814.392232] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13814.397456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13814.402412] [13814.404510] [13814.406608] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13814.411915] ? lock_acquire+0xf0/0x2b0 [13814.415666] ? igrab+0x43/0x50 [13814.418716] ? lock_release+0xc5/0x260 [13814.422461] _raw_spin_lock+0x30/0x40 [13814.426133] ? igrab+0x1d/0x50 [13814.429185] igrab+0x1d/0x50 [13814.432064] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13814.437214] delayed_work+0x1d8/0x280 [ceph] [13814.441494] ? process_scheduled_works+0x22a/0x510 [13814.446284] process_scheduled_works+0x2a4/0x510 [13814.450900] worker_thread+0x188/0x340 [13814.454649] ? rescuer_thread+0x310/0x310 [13814.458652] kthread+0xf5/0x130 [13814.461789] ? kthread_complete_and_exit+0x20/0x20 [13814.466574] ret_from_fork+0x31/0x40 [13814.470144] ? kthread_complete_and_exit+0x20/0x20 [13814.474929] ret_from_fork_asm+0x11/0x20 [13814.478852] [13841.697215] watchdog: BUG: soft lockup - CPU#0 stuck for 1777s! [dbench:78683] [13841.704432] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13841.709214] watchdog: BUG: soft lockup - CPU#7 stuck for 10774s! [kworker/7:4:78808] [13841.759891] irq event stamp: 16833450 [13841.767621] Modules linked in: ceph libceph dns_resolver fscache [13841.771279] hardirqs last enabled at (16833449): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13841.777275] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [13841.787433] hardirqs last disabled at (16833450): [] sysvec_apic_timer_interrupt+0xe/0x90 [13841.792910] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [13841.802636] softirqs last enabled at (16833424): [] __do_softirq+0x2be/0x371 [13841.811154] stp llc xfs libcrc32c sunrpc intel_rapl_msr [13841.819839] softirqs last disabled at (16833419): [] irq_exit_rcu+0xb0/0xd0 [13841.825143] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [13841.833654] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13841.833655] kvm_intel kvm irqbypass [13841.848942] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13841.852512] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [13841.859984] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13841.867108] joydev crc32_pclmul ipmi_ssif [13841.872586] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13841.872587] ghash_clmulni_intel mei_me mei [13841.895432] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13841.895433] lpc_ich sha512_ssse3 [13841.899611] [13841.904826] acpi_ipmi mfd_core [13841.908135] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13841.909628] wmi ipmi_si [13841.912765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13841.919893] ipmi_devintf ipmi_msghandler [13841.922428] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13841.929551] ioatdma acpi_pad [13841.933557] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13841.940680] acpi_power_meter [13841.943645] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13841.950768] ixgbe igb [13841.953733] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13841.960857] mdio crc32c_intel [13841.963215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13841.971293] nvme ptp [13841.974343] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13841.980080] nvme_core [13841.982351] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13841.989476] i2c_algo_bit [13841.991831] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13841.998955] pps_core [13842.001574] Call Trace: [13842.001576] [13842.008697] dca fuse [13842.010971] ? watchdog_timer_fn+0x1b4/0x220 [13842.013413] [13842.013414] irq event stamp: 34011878 [13842.015425] ? __hrtimer_run_queues+0x1d3/0x360 [13842.017695] hardirqs last enabled at (34011877): [] asm_common_interrupt+0x26/0x40 [13842.021962] ? hrtimer_interrupt+0x104/0x240 [13842.023450] hardirqs last disabled at (34011878): [] sysvec_apic_timer_interrupt+0xe/0x90 [13842.027105] ? __local_bh_enable+0x50/0x60 [13842.031629] softirqs last enabled at (34011876): [] __do_softirq+0x2be/0x371 [13842.040836] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13842.045097] softirqs last disabled at (34011869): [] irq_exit_rcu+0xb0/0xd0 [13842.054822] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13842.058912] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13842.067614] [13842.072833] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13842.081343] [13842.086301] Workqueue: events delayed_work [ceph] [13842.095679] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13842.097774] [13842.097775] RIP: 0010:ceph_check_delayed_caps+0xb2/0x270 [ceph] [13842.105249] ? queued_spin_lock_slowpath+0x16/0x2e0 [13842.107344] Code: 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 48 8b 85 78 04 00 00 49 [13842.112042] do_raw_spin_lock+0xb5/0xc0 [13842.117344] RSP: 0018:ffffc90002117db8 EFLAGS: 00000286 [13842.118837] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13842.124745] [13842.124746] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [13842.129621] try_get_cap_refs+0x159/0x880 [ceph] [13842.148355] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [13842.148356] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [13842.152191] __ceph_get_caps+0x4a1/0x760 [ceph] [13842.157404] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13842.162191] ? __file_remove_privs+0x89/0x110 [13842.163677] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13842.170802] ? wake_bit_function+0x70/0x70 [13842.175412] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13842.182540] ? ceph_write_iter+0x260/0x770 [ceph] [13842.189660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13842.194185] ceph_write_iter+0x260/0x770 [ceph] [13842.201310] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13842.205664] ? lock_is_held_type+0xaa/0x120 [13842.212785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13842.216895] vfs_write+0x324/0x4f0 [13842.224970] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13842.229674] ksys_pwrite64+0x65/0xa0 [13842.235405] Call Trace: [13842.235407] [13842.239928] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13842.247053] ? watchdog_timer_fn+0x1b4/0x220 [13842.251232] do_syscall_64+0x38/0x80 [13842.258355] ? __hrtimer_run_queues+0x1d3/0x360 [13842.261753] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13842.268878] ? hrtimer_interrupt+0x104/0x240 [13842.272447] RIP: 0033:0x7f48f0eeb93b [13842.274905] ? __local_bh_enable+0x50/0x60 [13842.276921] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13842.282053] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13842.286314] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13842.289901] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13842.294425] ORIG_RAX: 0000000000000012 [13842.299470] [13842.303733] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13842.307306] [13842.307307] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13842.311394] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13842.330137] ? ceph_check_delayed_caps+0xb2/0x270 [ceph] [13842.335351] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13842.340570] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [13842.345526] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13842.349359] delayed_work+0x1d8/0x280 [ceph] [13842.351452] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13842.358580] ? process_scheduled_works+0x22a/0x510 [13842.360679] [13842.365978] process_scheduled_works+0x2a4/0x510 [13842.420931] worker_thread+0x188/0x340 [13842.424679] ? rescuer_thread+0x310/0x310 [13842.428684] kthread+0xf5/0x130 [13842.431832] ? kthread_complete_and_exit+0x20/0x20 [13842.436620] ret_from_fork+0x31/0x40 [13842.440194] ? kthread_complete_and_exit+0x20/0x20 [13842.444979] ret_from_fork_asm+0x11/0x20 [13842.448921] [13853.549928] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13869.697703] watchdog: BUG: soft lockup - CPU#0 stuck for 1803s! [dbench:78683] [13869.704918] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13869.709703] watchdog: BUG: soft lockup - CPU#7 stuck for 10800s! [kworker/7:4:78808] [13869.760376] irq event stamp: 16891418 [13869.760377] hardirqs last enabled at (16891417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13869.768108] Modules linked in: [13869.771764] hardirqs last disabled at (16891418): [] sysvec_apic_timer_interrupt+0xe/0x90 [13869.781922] ceph [13869.784973] softirqs last enabled at (16891360): [] __do_softirq+0x2be/0x371 [13869.794698] libceph dns_resolver [13869.796622] softirqs last disabled at (16891355): [] irq_exit_rcu+0xb0/0xd0 [13869.805306] fscache netfs [13869.808617] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13869.817128] veth nft_chain_nat [13869.819830] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13869.819831] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13869.828776] xt_MASQUERADE nf_nat [13869.831912] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13869.839396] nf_conntrack nf_defrag_ipv6 [13869.844870] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13869.848181] nf_defrag_ipv4 nft_compat [13869.866918] [13869.866919] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13869.870836] nf_tables nfnetlink [13869.876054] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13869.879797] bridge stp [13869.881287] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13869.888412] llc xfs [13869.891636] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13869.898761] libcrc32c [13869.901204] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13869.908329] sunrpc intel_rapl_msr [13869.910514] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13869.917635] intel_rapl_common [13869.919994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13869.927117] sb_edac x86_pkg_temp_thermal [13869.930515] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13869.938594] coretemp [13869.941645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13869.947382] kvm_intel kvm [13869.951386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13869.958509] irqbypass i2c_i801 [13869.960781] Call Trace: [13869.960784] [13869.967905] iTCO_wdt iTCO_vendor_support [13869.970609] ? watchdog_timer_fn+0x1b4/0x220 [13869.977732] i2c_smbus [13869.980870] ? __hrtimer_run_queues+0x1d3/0x360 [13869.983314] crct10dif_pclmul joydev [13869.985329] ? hrtimer_interrupt+0x104/0x240 [13869.989329] crc32_pclmul ipmi_ssif [13869.993594] ? __local_bh_enable+0x50/0x60 [13869.995949] ghash_clmulni_intel [13870.000478] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13870.004044] mei_me mei [13870.008309] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13870.011792] lpc_ich sha512_ssse3 [13870.015885] [13870.019106] acpi_ipmi mfd_core [13870.024325] [13870.024327] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13870.026769] wmi ipmi_si [13870.031732] ? queued_spin_lock_slowpath+0x18/0x2e0 [13870.035036] ipmi_devintf ipmi_msghandler [13870.037138] do_raw_spin_lock+0xb5/0xc0 [13870.040272] ioatdma [13870.042383] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13870.047692] acpi_pad acpi_power_meter [13870.050224] try_get_cap_refs+0x159/0x880 [ceph] [13870.055102] ixgbe igb [13870.059111] __ceph_get_caps+0x4a1/0x760 [ceph] [13870.062937] mdio crc32c_intel [13870.065124] ? __file_remove_privs+0x89/0x110 [13870.069904] nvme ptp [13870.073649] ? wake_bit_function+0x70/0x70 [13870.078259] nvme_core i2c_algo_bit [13870.080619] ? ceph_write_iter+0x260/0x770 [ceph] [13870.085140] pps_core dca [13870.088191] ceph_write_iter+0x260/0x770 [ceph] [13870.092541] fuse [13870.094815] ? lock_is_held_type+0xaa/0x120 [13870.098904] irq event stamp: 34233936 [13870.102395] vfs_write+0x324/0x4f0 [13870.107104] hardirqs last enabled at (34233935): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13870.109723] ksys_pwrite64+0x65/0xa0 [13870.114244] hardirqs last disabled at (34233936): [] sysvec_apic_timer_interrupt+0xe/0x90 [13870.116167] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13870.120345] softirqs last enabled at (34233934): [] __do_softirq+0x2be/0x371 [13870.124003] do_syscall_64+0x38/0x80 [13870.127399] softirqs last disabled at (34233929): [] irq_exit_rcu+0xb0/0xd0 [13870.137559] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13870.141128] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13870.150852] RIP: 0033:0x7f48f0eeb93b [13870.155982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13870.164668] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13870.168239] Workqueue: events delayed_work [ceph] [13870.176748] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13870.181793] [13870.181794] RIP: 0010:lock_acquire+0xf0/0x2b0 [13870.191171] ORIG_RAX: 0000000000000012 [13870.194741] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13870.202212] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13870.220949] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13870.225647] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13870.230864] [13870.230865] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13870.232354] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13870.236705] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13870.240536] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13870.259274] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13870.266397] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13870.271615] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13870.278744] [13870.280229] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13870.339417] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13870.347494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13870.353232] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13870.360354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13870.367480] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13870.374603] Call Trace: [13870.377049] [13870.379059] ? watchdog_timer_fn+0x1b4/0x220 [13870.383325] ? __hrtimer_run_queues+0x1d3/0x360 [13870.387859] ? hrtimer_interrupt+0x104/0x240 [13870.392127] ? __local_bh_enable+0x50/0x60 [13870.396221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13870.401446] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13870.406403] [13870.408501] [13870.410597] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13870.415907] ? lock_acquire+0xf0/0x2b0 [13870.419655] ? igrab+0x43/0x50 [13870.422705] ? lock_release+0xc5/0x260 [13870.426452] _raw_spin_lock+0x30/0x40 [13870.430115] ? igrab+0x1d/0x50 [13870.433167] igrab+0x1d/0x50 [13870.436046] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13870.441193] delayed_work+0x1d8/0x280 [ceph] [13870.445474] ? process_scheduled_works+0x22a/0x510 [13870.450257] process_scheduled_works+0x2a4/0x510 [13870.454873] worker_thread+0x188/0x340 [13870.458622] ? rescuer_thread+0x310/0x310 [13870.462632] kthread+0xf5/0x130 [13870.465772] ? kthread_complete_and_exit+0x20/0x20 [13870.470555] ret_from_fork+0x31/0x40 [13870.474125] ? kthread_complete_and_exit+0x20/0x20 [13870.478909] ret_from_fork_asm+0x11/0x20 [13870.482833] [13885.293387] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [13885.301991] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13895.534021] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13897.698192] watchdog: BUG: soft lockup - CPU#0 stuck for 1829s! [dbench:78683] [13897.705406] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13897.710192] watchdog: BUG: soft lockup - CPU#7 stuck for 10826s! [kworker/7:4:78808] [13897.760867] irq event stamp: 16949578 [13897.768602] Modules linked in: ceph libceph dns_resolver fscache netfs veth [13897.772261] hardirqs last enabled at (16949577): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13897.779220] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [13897.789376] hardirqs last disabled at (16949578): [] sysvec_apic_timer_interrupt+0xe/0x90 [13897.797629] nft_compat nf_tables nfnetlink bridge stp [13897.807352] softirqs last enabled at (16949572): [] __do_softirq+0x2be/0x371 [13897.807355] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [13897.821168] softirqs last disabled at (16949567): [] irq_exit_rcu+0xb0/0xd0 [13897.821170] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [13897.836198] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13897.836199] irqbypass [13897.836200] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13897.842194] i2c_i801 iTCO_wdt [13897.851138] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13897.853495] iTCO_vendor_support [13897.860966] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13897.864017] i2c_smbus [13897.869494] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13897.872716] crct10dif_pclmul joydev [13897.891456] [13897.893811] crc32_pclmul [13897.899031] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13897.902603] ipmi_ssif [13897.904091] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13897.906711] ghash_clmulni_intel [13897.913833] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13897.916189] mei_me mei [13897.923314] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13897.926538] lpc_ich [13897.933663] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13897.936108] sha512_ssse3 acpi_ipmi [13897.943232] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13897.945415] mfd_core wmi [13897.952539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13897.956023] ipmi_si ipmi_devintf [13897.964102] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13897.966717] ipmi_msghandler ioatdma [13897.972456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13897.975766] acpi_pad acpi_power_meter [13897.982891] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13897.986462] ixgbe igb [13897.993590] Call Trace: [13897.993591] [13897.997337] mdio [13898.004466] ? watchdog_timer_fn+0x1b4/0x220 [13898.006820] crc32c_intel nvme [13898.009265] ? __hrtimer_run_queues+0x1d3/0x360 [13898.011274] ptp nvme_core [13898.013201] ? hrtimer_interrupt+0x104/0x240 [13898.017464] i2c_algo_bit [13898.020514] ? __local_bh_enable+0x50/0x60 [13898.025037] pps_core dca [13898.027743] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13898.032004] fuse [13898.034623] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13898.038713] irq event stamp: 34456828 [13898.041332] [13898.046549] hardirqs last enabled at (34456827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13898.048473] [13898.053429] hardirqs last disabled at (34456828): [] sysvec_apic_timer_interrupt+0xe/0x90 [13898.057089] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13898.059184] softirqs last enabled at (34456826): [] __do_softirq+0x2be/0x371 [13898.069346] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13898.071439] softirqs last disabled at (34456821): [] irq_exit_rcu+0xb0/0xd0 [13898.081167] do_raw_spin_lock+0xb5/0xc0 [13898.086468] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13898.095152] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13898.100022] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13898.108536] try_get_cap_refs+0x159/0x880 [ceph] [13898.112363] Workqueue: events delayed_work [ceph] [13898.121746] __ceph_get_caps+0x4a1/0x760 [ceph] [13898.126526] [13898.126527] RIP: 0010:lock_acquire+0xf0/0x2b0 [13898.134000] ? __file_remove_privs+0x89/0x110 [13898.138607] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [13898.143306] ? wake_bit_function+0x70/0x70 [13898.147828] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [13898.149323] ? ceph_write_iter+0x260/0x770 [ceph] [13898.153669] [13898.153670] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [13898.158020] ceph_write_iter+0x260/0x770 [ceph] [13898.176758] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [13898.180853] ? lock_is_held_type+0xaa/0x120 [13898.186084] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [13898.190784] vfs_write+0x324/0x4f0 [13898.192272] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [13898.199403] ksys_pwrite64+0x65/0xa0 [13898.203920] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13898.211046] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13898.215222] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13898.222347] do_syscall_64+0x38/0x80 [13898.225743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13898.232875] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13898.236447] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13898.243590] RIP: 0033:0x7f48f0eeb93b [13898.248718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13898.256798] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13898.260368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13898.266104] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13898.271149] Call Trace: [13898.278273] ORIG_RAX: 0000000000000012 [13898.281844] [13898.288968] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13898.307706] ? watchdog_timer_fn+0x1b4/0x220 [13898.314829] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13898.320048] ? __hrtimer_run_queues+0x1d3/0x360 [13898.322490] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13898.326326] ? hrtimer_interrupt+0x104/0x240 [13898.328332] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13898.335457] ? __local_bh_enable+0x50/0x60 [13898.339720] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13898.346849] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13898.351374] [13898.358493] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13898.393465] [13898.395562] [13898.397661] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13898.402967] ? lock_acquire+0xf0/0x2b0 [13898.406718] ? igrab+0x43/0x50 [13898.409775] ? lock_release+0xc5/0x260 [13898.413520] _raw_spin_lock+0x30/0x40 [13898.417176] ? igrab+0x1d/0x50 [13898.420227] igrab+0x1d/0x50 [13898.423105] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13898.428254] delayed_work+0x1d8/0x280 [ceph] [13898.432536] ? process_scheduled_works+0x22a/0x510 [13898.437326] process_scheduled_works+0x2a4/0x510 [13898.441941] worker_thread+0x188/0x340 [13898.445693] ? rescuer_thread+0x310/0x310 [13898.449703] kthread+0xf5/0x130 [13898.452842] ? kthread_complete_and_exit+0x20/0x20 [13898.457625] ret_from_fork+0x31/0x40 [13898.461205] ? kthread_complete_and_exit+0x20/0x20 [13898.465988] ret_from_fork_asm+0x11/0x20 [13898.469910] [13906.286348] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13916.526418] libceph: auth protocol 'cephx' mauth authentication failed: -13 [13925.698681] watchdog: BUG: soft lockup - CPU#0 stuck for 1855s! [dbench:78683] [13925.705892] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13925.710681] watchdog: BUG: soft lockup - CPU#7 stuck for 10852s! [kworker/7:4:78808] [13925.761332] irq event stamp: 17007510 [13925.761333] hardirqs last enabled at (17007509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13925.769081] Modules linked in: [13925.772737] hardirqs last disabled at (17007510): [] sysvec_apic_timer_interrupt+0xe/0x90 [13925.782896] ceph [13925.785946] softirqs last enabled at (17007452): [] __do_softirq+0x2be/0x371 [13925.795670] libceph [13925.797595] softirqs last disabled at (17007447): [] irq_exit_rcu+0xb0/0xd0 [13925.806279] dns_resolver fscache [13925.808464] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13925.816975] netfs [13925.820284] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13925.829228] veth nft_chain_nat [13925.831240] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [13925.838711] xt_MASQUERADE nf_nat [13925.841847] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [13925.847325] nf_conntrack nf_defrag_ipv6 [13925.850635] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13925.869374] nf_defrag_ipv4 [13925.873291] [13925.878508] nft_compat nf_tables [13925.881299] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13925.882789] nfnetlink bridge [13925.886099] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13925.893225] stp llc [13925.896189] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13925.903312] xfs libcrc32c [13925.905497] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13925.912622] sunrpc intel_rapl_msr [13925.915325] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13925.922448] intel_rapl_common [13925.925846] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13925.932970] sb_edac x86_pkg_temp_thermal [13925.936021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13925.944099] coretemp [13925.948103] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13925.953840] kvm_intel [13925.956110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13925.963236] kvm irqbypass [13925.965592] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13925.972718] i2c_i801 iTCO_wdt [13925.975420] Call Trace: [13925.975421] [13925.982544] iTCO_vendor_support [13925.985596] ? watchdog_timer_fn+0x1b4/0x220 [13925.988039] i2c_smbus crct10dif_pclmul [13925.990054] ? __hrtimer_run_queues+0x1d3/0x360 [13925.993291] joydev [13925.997559] ? hrtimer_interrupt+0x104/0x240 [13926.001386] crc32_pclmul ipmi_ssif [13926.005910] ? __local_bh_enable+0x50/0x60 [13926.008009] ghash_clmulni_intel mei_me [13926.012275] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13926.015756] mei lpc_ich [13926.019848] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13926.023678] sha512_ssse3 [13926.028896] [13926.031426] acpi_ipmi mfd_core [13926.036384] [13926.039001] wmi ipmi_si [13926.041098] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13926.044235] ipmi_devintf ipmi_msghandler [13926.046337] ? queued_spin_lock_slowpath+0x16/0x2e0 [13926.048864] ioatdma acpi_pad [13926.054171] do_raw_spin_lock+0xb5/0xc0 [13926.058172] acpi_power_meter ixgbe [13926.063044] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13926.066006] igb mdio [13926.069839] try_get_cap_refs+0x159/0x880 [ceph] [13926.073322] crc32c_intel nvme [13926.078110] __ceph_get_caps+0x4a1/0x760 [ceph] [13926.080376] ptp nvme_core [13926.084991] ? __file_remove_privs+0x89/0x110 [13926.088039] i2c_algo_bit pps_core [13926.092562] ? wake_bit_function+0x70/0x70 [13926.095266] dca fuse [13926.099620] ? ceph_write_iter+0x260/0x770 [ceph] [13926.103014] [13926.103015] irq event stamp: 34679566 [13926.107104] ceph_write_iter+0x260/0x770 [ceph] [13926.109375] hardirqs last enabled at (34679565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13926.114093] ? lock_is_held_type+0xaa/0x120 [13926.115580] hardirqs last disabled at (34679566): [] sysvec_apic_timer_interrupt+0xe/0x90 [13926.119240] vfs_write+0x324/0x4f0 [13926.123763] softirqs last enabled at (34679564): [] __do_softirq+0x2be/0x371 [13926.133924] ksys_pwrite64+0x65/0xa0 [13926.138097] softirqs last disabled at (34679559): [] irq_exit_rcu+0xb0/0xd0 [13926.147821] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13926.151219] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13926.159904] do_syscall_64+0x38/0x80 [13926.163475] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13926.171985] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13926.177115] Workqueue: events delayed_work [ceph] [13926.186494] RIP: 0033:0x7f48f0eeb93b [13926.190081] [13926.197552] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13926.202596] RIP: 0010:lock_release+0x14e/0x260 [13926.207292] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13926.210865] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [13926.212363] ORIG_RAX: 0000000000000012 [13926.231101] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [13926.235539] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13926.240755] [13926.240756] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [13926.259494] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13926.263325] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [13926.268542] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13926.275666] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [13926.277157] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13926.284282] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [13926.291405] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13926.298529] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [13926.305657] [13926.312777] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13926.351535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13926.357272] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13926.364398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13926.371521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13926.378644] Call Trace: [13926.381091] [13926.383101] ? watchdog_timer_fn+0x1b4/0x220 [13926.387369] ? __hrtimer_run_queues+0x1d3/0x360 [13926.391901] ? hrtimer_interrupt+0x104/0x240 [13926.396170] ? __local_bh_enable+0x50/0x60 [13926.400262] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13926.405479] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13926.410435] [13926.412532] [13926.414632] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13926.419936] ? igrab+0x43/0x50 [13926.422989] ? lock_release+0x14e/0x260 [13926.426824] ? lock_release+0xc5/0x260 [13926.430570] _raw_spin_unlock+0x1b/0x40 [13926.434399] igrab+0x43/0x50 [13926.437276] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13926.442427] delayed_work+0x1d8/0x280 [ceph] [13926.446707] ? process_scheduled_works+0x22a/0x510 [13926.451490] process_scheduled_works+0x2a4/0x510 [13926.456106] worker_thread+0x188/0x340 [13926.459855] ? rescuer_thread+0x310/0x310 [13926.463857] kthread+0xf5/0x130 [13926.466994] ? kthread_complete_and_exit+0x20/0x20 [13926.471781] ret_from_fork+0x31/0x40 [13926.475375] ? kthread_complete_and_exit+0x20/0x20 [13926.480160] ret_from_fork_asm+0x11/0x20 [13926.484095] [13948.270485] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [13953.699170] watchdog: BUG: soft lockup - CPU#0 stuck for 1881s! [dbench:78683] [13953.706387] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13953.711170] watchdog: BUG: soft lockup - CPU#7 stuck for 10878s! [kworker/7:4:78808] [13953.761845] irq event stamp: 17065494 [13953.769577] Modules linked in: ceph libceph dns_resolver fscache [13953.773235] hardirqs last enabled at (17065493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13953.779240] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [13953.789399] hardirqs last disabled at (17065494): [] sysvec_apic_timer_interrupt+0xe/0x90 [13953.794884] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [13953.804609] softirqs last enabled at (17065436): [] __do_softirq+0x2be/0x371 [13953.804612] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [13953.821205] softirqs last disabled at (17065431): [] irq_exit_rcu+0xb0/0xd0 [13953.827117] intel_rapl_common sb_edac [13953.835626] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13953.835627] x86_pkg_temp_thermal [13953.835628] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13953.839370] coretemp kvm_intel [13953.848316] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13953.851626] kvm irqbypass [13953.859098] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13953.862234] i2c_i801 [13953.867711] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13953.870416] iTCO_wdt [13953.889152] [13953.891424] iTCO_vendor_support i2c_smbus [13953.896642] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13953.898911] crct10dif_pclmul joydev [13953.900401] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13953.904492] crc32_pclmul [13953.911616] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13953.915188] ipmi_ssif ghash_clmulni_intel [13953.922313] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13953.924929] mei_me mei lpc_ich [13953.932055] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13953.936145] sha512_ssse3 acpi_ipmi [13953.943268] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13953.946406] mfd_core [13953.953530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13953.957013] wmi ipmi_si [13953.965090] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13953.967361] ipmi_devintf ipmi_msghandler [13953.973100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13953.975631] ioatdma acpi_pad [13953.982754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13953.986759] acpi_power_meter ixgbe [13953.993882] Call Trace: [13953.996849] igb [13954.003974] [13954.007454] mdio crc32c_intel [13954.009903] ? watchdog_timer_fn+0x1b4/0x220 [13954.011736] nvme [13954.013747] ? __hrtimer_run_queues+0x1d3/0x360 [13954.016799] ptp nvme_core [13954.021065] ? hrtimer_interrupt+0x104/0x240 [13954.022987] i2c_algo_bit [13954.027509] ? __local_bh_enable+0x50/0x60 [13954.030215] pps_core dca [13954.034481] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13954.037095] fuse [13954.041186] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13954.043803] [13954.043803] irq event stamp: 34903160 [13954.049021] [13954.050946] hardirqs last enabled at (34903159): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13954.055903] [13954.057394] hardirqs last disabled at (34903160): [] sysvec_apic_timer_interrupt+0xe/0x90 [13954.061051] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13954.063147] softirqs last enabled at (34903158): [] __do_softirq+0x2be/0x371 [13954.073310] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13954.075403] softirqs last disabled at (34903153): [] irq_exit_rcu+0xb0/0xd0 [13954.085130] do_raw_spin_lock+0xb5/0xc0 [13954.090432] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13954.099118] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13954.103985] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13954.112501] try_get_cap_refs+0x159/0x880 [ceph] [13954.116328] Workqueue: events delayed_work [ceph] [13954.125712] __ceph_get_caps+0x4a1/0x760 [ceph] [13954.130489] [13954.130490] RIP: 0010:do_raw_spin_lock+0xd/0xc0 [13954.137964] ? __file_remove_privs+0x89/0x110 [13954.142570] Code: 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 <48> 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 [13954.147269] ? wake_bit_function+0x70/0x70 [13954.151793] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [13954.153285] ? ceph_write_iter+0x260/0x770 [ceph] [13954.157806] [13954.162160] ceph_write_iter+0x260/0x770 [ceph] [13954.180895] RAX: 00000000dead4ead RBX: ffff8881a1d34478 RCX: 000000005553a634 [13954.184991] ? lock_is_held_type+0xaa/0x120 [13954.190204] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [13954.194903] vfs_write+0x324/0x4f0 [13954.196393] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [13954.200921] ksys_pwrite64+0x65/0xa0 [13954.208039] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [13954.212217] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13954.219342] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [13954.222741] do_syscall_64+0x38/0x80 [13954.229863] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13954.233435] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13954.240570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13954.245699] RIP: 0033:0x7f48f0eeb93b [13954.252823] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13954.256392] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13954.264470] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13954.269513] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13954.275251] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13954.278821] ORIG_RAX: 0000000000000012 [13954.285945] Call Trace: [13954.285946] [13954.304683] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13954.311809] ? watchdog_timer_fn+0x1b4/0x220 [13954.317025] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13954.324150] ? __hrtimer_run_queues+0x1d3/0x360 [13954.327981] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13954.330429] ? hrtimer_interrupt+0x104/0x240 [13954.332435] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13954.339573] ? __local_bh_enable+0x50/0x60 [13954.343853] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13954.350976] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13954.355502] [13954.362621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13954.397583] [13954.399680] [13954.401778] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13954.407088] ? do_raw_spin_lock+0xd/0xc0 [13954.411010] igrab+0x1d/0x50 [13954.413888] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13954.419038] delayed_work+0x1d8/0x280 [ceph] [13954.423319] ? process_scheduled_works+0x22a/0x510 [13954.428108] process_scheduled_works+0x2a4/0x510 [13954.432724] worker_thread+0x188/0x340 [13954.436474] ? rescuer_thread+0x310/0x310 [13954.440475] kthread+0xf5/0x130 [13954.443615] ? kthread_complete_and_exit+0x20/0x20 [13954.448397] ret_from_fork+0x31/0x40 [13954.451978] ? kthread_complete_and_exit+0x20/0x20 [13954.456762] ret_from_fork_asm+0x11/0x20 [13954.460695] [13956.056577] rcu: INFO: rcu_preempt self-detected stall on CPU [13956.062336] rcu: 0-....: (1975914 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=962555 [13956.072586] rcu: (t=2019411 jiffies g=1309609 q=141522 ncpus=8) [13956.078603] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13956.087545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13956.095016] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [13956.100494] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [13956.119233] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13956.124457] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13956.131581] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13956.138705] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13956.145830] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13956.152955] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13956.160078] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13956.168156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13956.173892] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13956.181019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13956.188142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13956.195266] Call Trace: [13956.197709] [13956.199721] ? rcu_dump_cpu_stacks+0xd1/0x180 [13956.204081] ? rcu_sched_clock_irq+0x633/0x1390 [13956.208613] ? lock_is_held_type+0xaa/0x120 [13956.212791] ? lock_is_held_type+0xaa/0x120 [13956.216969] ? tick_sched_do_timer+0xa0/0xa0 [13956.221240] ? rcu_read_lock_sched_held+0x56/0x90 [13956.225939] ? tick_sched_do_timer+0xa0/0xa0 [13956.230203] ? update_process_times+0x5f/0x90 [13956.234552] ? tick_sched_handle.isra.16+0x1e/0x50 [13956.239336] ? tick_sched_timer+0x6f/0x80 [13956.243342] ? __hrtimer_run_queues+0x1d3/0x360 [13956.247869] ? hrtimer_interrupt+0x104/0x240 [13956.252137] ? __local_bh_enable+0x50/0x60 [13956.256230] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13956.261455] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13956.266413] [13956.268510] [13956.270607] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13956.275915] ? queued_spin_lock_slowpath+0x18/0x2e0 [13956.280792] do_raw_spin_lock+0xb5/0xc0 [13956.284629] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13956.289432] try_get_cap_refs+0x159/0x880 [ceph] [13956.294064] __ceph_get_caps+0x4a1/0x760 [ceph] [13956.298604] ? __file_remove_privs+0x89/0x110 [13956.302960] ? wake_bit_function+0x70/0x70 [13956.307053] ? ceph_write_iter+0x260/0x770 [ceph] [13956.311765] ceph_write_iter+0x260/0x770 [ceph] [13956.316302] ? lock_is_held_type+0xaa/0x120 [13956.320486] vfs_write+0x324/0x4f0 [13956.323895] ksys_pwrite64+0x65/0xa0 [13956.327470] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13956.332600] do_syscall_64+0x38/0x80 [13956.336170] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13956.341215] RIP: 0033:0x7f48f0eeb93b [13956.344785] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13956.363522] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [13956.371081] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13956.378205] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13956.385328] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13956.392452] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13956.399578] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13956.406707] [13956.408894] Sending NMI from CPU 0 to CPUs 7: [13956.413246] NMI backtrace for cpu 7 [13956.413248] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13956.413250] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13956.413251] Workqueue: events delayed_work [ceph] [13956.413267] RIP: 0010:lock_acquire+0x86/0x2b0 [13956.413269] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [13956.413270] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [13956.413271] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13956.413273] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13956.413273] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [13956.413274] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [13956.413274] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13956.413275] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13956.413276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13956.413277] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13956.413278] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13956.413278] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13956.413279] Call Trace: [13956.413279] [13956.413281] ? nmi_cpu_backtrace+0xad/0x120 [13956.413284] ? nmi_cpu_backtrace_handler+0x11/0x20 [13956.413285] ? nmi_handle+0xd4/0x260 [13956.413290] ? default_do_nmi+0x49/0xf0 [13956.413291] ? exc_nmi+0x11e/0x150 [13956.413293] ? end_repeat_nmi+0x16/0x67 [13956.413297] ? lock_acquire+0x86/0x2b0 [13956.413298] ? lock_acquire+0x86/0x2b0 [13956.413300] ? lock_acquire+0x86/0x2b0 [13956.413302] [13956.413302] [13956.413303] ? igrab+0x43/0x50 [13956.413304] ? lock_release+0xc5/0x260 [13956.413306] _raw_spin_lock+0x30/0x40 [13956.413308] ? igrab+0x1d/0x50 [13956.413309] igrab+0x1d/0x50 [13956.413311] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13956.413325] delayed_work+0x1d8/0x280 [ceph] [13956.413339] ? process_scheduled_works+0x22a/0x510 [13956.413341] process_scheduled_works+0x2a4/0x510 [13956.413345] worker_thread+0x188/0x340 [13956.413348] ? rescuer_thread+0x310/0x310 [13956.413350] kthread+0xf5/0x130 [13956.413352] ? kthread_complete_and_exit+0x20/0x20 [13956.413355] ret_from_fork+0x31/0x40 [13956.413357] ? kthread_complete_and_exit+0x20/0x20 [13956.413359] ret_from_fork_asm+0x11/0x20 [13956.413363] [13981.699659] watchdog: BUG: soft lockup - CPU#0 stuck for 1907s! [dbench:78683] [13981.706875] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [13981.711659] watchdog: BUG: soft lockup - CPU#7 stuck for 10904s! [kworker/7:4:78808] [13981.762315] irq event stamp: 17122598 [13981.762317] hardirqs last enabled at (17122597): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13981.770063] Modules linked in: [13981.773721] hardirqs last disabled at (17122598): [] sysvec_apic_timer_interrupt+0xe/0x90 [13981.783879] ceph [13981.786930] softirqs last enabled at (17122564): [] __do_softirq+0x2be/0x371 [13981.796653] libceph [13981.798578] softirqs last disabled at (17122559): [] irq_exit_rcu+0xb0/0xd0 [13981.807263] dns_resolver fscache [13981.809447] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13981.817956] netfs [13981.821267] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13981.830211] veth nft_chain_nat [13981.832222] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [13981.839693] xt_MASQUERADE nf_nat [13981.842831] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [13981.848307] nf_conntrack [13981.851637] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [13981.870373] nf_defrag_ipv6 nf_defrag_ipv4 [13981.872990] [13981.872991] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [13981.878208] nft_compat nf_tables [13981.882299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [13981.883790] nfnetlink bridge [13981.890913] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [13981.894223] stp llc [13981.901353] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [13981.904314] xfs libcrc32c [13981.911455] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [13981.913639] sunrpc intel_rapl_msr [13981.920762] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [13981.923468] intel_rapl_common [13981.930591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13981.933987] sb_edac x86_pkg_temp_thermal [13981.942065] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [13981.945116] coretemp kvm_intel [13981.950855] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13981.954858] kvm irqbypass [13981.961982] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13981.965120] i2c_i801 [13981.972243] Call Trace: [13981.972244] [13981.974947] iTCO_wdt [13981.982073] ? watchdog_timer_fn+0x1b4/0x220 [13981.984343] iTCO_vendor_support i2c_smbus [13981.986787] ? __hrtimer_run_queues+0x1d3/0x360 [13981.988797] crct10dif_pclmul [13981.991071] ? hrtimer_interrupt+0x104/0x240 [13981.995338] joydev crc32_pclmul [13981.999431] ? __local_bh_enable+0x50/0x60 [13982.003956] ipmi_ssif [13982.006923] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13982.011185] ghash_clmulni_intel [13982.014407] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13982.018499] mei_me mei [13982.020856] [13982.026072] lpc_ich sha512_ssse3 [13982.029298] [13982.029299] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13982.034254] acpi_ipmi [13982.036702] ? queued_spin_lock_slowpath+0x2d/0x2e0 [13982.038796] mfd_core wmi [13982.042109] do_raw_spin_lock+0xb5/0xc0 [13982.044204] ipmi_si ipmi_devintf [13982.049509] __cap_delay_requeue+0xa7/0x1d0 [ceph] [13982.051866] ipmi_msghandler ioatdma [13982.056740] try_get_cap_refs+0x159/0x880 [ceph] [13982.059354] acpi_pad acpi_power_meter [13982.063190] __ceph_get_caps+0x4a1/0x760 [ceph] [13982.066496] ixgbe igb [13982.071283] ? __file_remove_privs+0x89/0x110 [13982.074850] mdio crc32c_intel [13982.079462] ? wake_bit_function+0x70/0x70 [13982.083204] nvme ptp [13982.087731] ? ceph_write_iter+0x260/0x770 [ceph] [13982.090087] nvme_core i2c_algo_bit [13982.094437] ceph_write_iter+0x260/0x770 [ceph] [13982.097487] pps_core [13982.101582] ? lock_is_held_type+0xaa/0x120 [13982.103849] dca fuse [13982.108548] vfs_write+0x324/0x4f0 [13982.112031] [13982.112032] irq event stamp: 35123772 [13982.116575] ksys_pwrite64+0x65/0xa0 [13982.118844] hardirqs last enabled at (35123771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13982.123021] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [13982.125290] hardirqs last disabled at (35123772): [] sysvec_apic_timer_interrupt+0xe/0x90 [13982.128689] do_syscall_64+0x38/0x80 [13982.130180] softirqs last enabled at (35123770): [] __do_softirq+0x2be/0x371 [13982.133837] entry_SYSCALL_64_after_hwframe+0x63/0xcd [13982.137408] softirqs last disabled at (35123765): [] irq_exit_rcu+0xb0/0xd0 [13982.147566] RIP: 0033:0x7f48f0eeb93b [13982.152695] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [13982.162419] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [13982.165991] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [13982.165993] Workqueue: events delayed_work [ceph] [13982.174675] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [13982.179719] [13982.188229] ORIG_RAX: 0000000000000012 [13982.191800] RIP: 0010:rcu_is_watching+0x1f/0x40 [13982.201178] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [13982.219916] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [13982.227386] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [13982.232085] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [13982.237302] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [13982.238793] [13982.238794] RAX: 00000000007c1919 RBX: ffff8881a1d34500 RCX: 0000000000000000 [13982.242623] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [13982.247147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [13982.254270] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [13982.273008] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [13982.280139] [13982.285351] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [13982.338912] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [13982.346035] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [13982.354130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13982.359866] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [13982.366992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13982.374115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13982.381240] Call Trace: [13982.383686] [13982.385695] ? watchdog_timer_fn+0x1b4/0x220 [13982.389960] ? __hrtimer_run_queues+0x1d3/0x360 [13982.394488] ? hrtimer_interrupt+0x104/0x240 [13982.398755] ? __local_bh_enable+0x50/0x60 [13982.402850] ? __sysvec_apic_timer_interrupt+0x61/0x160 [13982.408072] ? sysvec_apic_timer_interrupt+0x6d/0x90 [13982.413029] [13982.415126] [13982.417224] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [13982.422532] ? rcu_is_watching+0x1f/0x40 [13982.426456] lock_acquire+0x184/0x2b0 [13982.430113] ? igrab+0x43/0x50 [13982.433165] ? lock_release+0xc5/0x260 [13982.436909] _raw_spin_lock+0x30/0x40 [13982.440564] ? igrab+0x1d/0x50 [13982.443614] igrab+0x1d/0x50 [13982.446492] ceph_check_delayed_caps+0xb7/0x270 [ceph] [13982.451643] delayed_work+0x1d8/0x280 [ceph] [13982.455924] ? process_scheduled_works+0x22a/0x510 [13982.460716] process_scheduled_works+0x2a4/0x510 [13982.465346] worker_thread+0x188/0x340 [13982.469096] ? rescuer_thread+0x310/0x310 [13982.473100] kthread+0xf5/0x130 [13982.476237] ? kthread_complete_and_exit+0x20/0x20 [13982.481023] ret_from_fork+0x31/0x40 [13982.484601] ? kthread_complete_and_exit+0x20/0x20 [13982.489385] ret_from_fork_asm+0x11/0x20 [13982.493308] [14009.700148] watchdog: BUG: soft lockup - CPU#0 stuck for 1933s! [dbench:78683] [14009.707361] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14009.712147] watchdog: BUG: soft lockup - CPU#7 stuck for 10930s! [kworker/7:4:78808] [14009.762824] irq event stamp: 17180594 [14009.770561] Modules linked in: ceph libceph dns_resolver fscache netfs veth [14009.774217] hardirqs last enabled at (17180593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14009.774222] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [14009.791335] hardirqs last disabled at (17180594): [] sysvec_apic_timer_interrupt+0xe/0x90 [14009.800541] nf_tables nfnetlink bridge stp llc xfs [14009.810262] softirqs last enabled at (17180546): [] __do_softirq+0x2be/0x371 [14009.810265] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [14009.823833] softirqs last disabled at (17180541): [] irq_exit_rcu+0xb0/0xd0 [14009.832173] coretemp kvm_intel kvm [14009.840684] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14009.840686] irqbypass i2c_i801 iTCO_wdt [14009.853111] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14009.853113] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14009.857030] iTCO_vendor_support i2c_smbus [14009.864500] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14009.869977] crct10dif_pclmul joydev [14009.874067] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14009.892804] crc32_pclmul ipmi_ssif [14009.896377] [14009.896377] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14009.901593] ghash_clmulni_intel mei_me [14009.905077] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14009.906567] mei lpc_ich [14009.913691] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14009.917540] sha512_ssse3 [14009.924663] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14009.927196] acpi_ipmi mfd_core [14009.934320] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14009.936938] wmi ipmi_si [14009.944060] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14009.947197] ipmi_devintf ipmi_msghandler [14009.954321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14009.956854] ioatdma acpi_pad [14009.964931] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14009.968934] acpi_power_meter [14009.974671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14009.977637] ixgbe igb [14009.984760] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14009.987724] mdio crc32c_intel [14009.994848] Call Trace: [14009.994851] [14009.997208] nvme [14010.004332] ? watchdog_timer_fn+0x1b4/0x220 [14010.007380] ptp nvme_core [14010.009828] ? __hrtimer_run_queues+0x1d3/0x360 [14010.011842] i2c_algo_bit [14010.013770] ? hrtimer_interrupt+0x104/0x240 [14010.018033] pps_core dca [14010.020736] ? __local_bh_enable+0x50/0x60 [14010.025261] fuse [14010.027881] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14010.032142] [14010.032143] irq event stamp: 35346090 [14010.034760] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14010.038849] hardirqs last enabled at (35346089): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14010.040774] [14010.045992] hardirqs last disabled at (35346090): [] sysvec_apic_timer_interrupt+0xe/0x90 [14010.047481] [14010.047483] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14010.051140] softirqs last enabled at (35346088): [] __do_softirq+0x2be/0x371 [14010.056101] ? queued_spin_lock_slowpath+0x16/0x2e0 [14010.066255] softirqs last disabled at (35346079): [] irq_exit_rcu+0xb0/0xd0 [14010.068355] do_raw_spin_lock+0xb5/0xc0 [14010.078075] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14010.080176] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14010.085478] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14010.085479] Workqueue: events delayed_work [ceph] [14010.094166] try_get_cap_refs+0x159/0x880 [ceph] [14010.099033] [14010.107555] __ceph_get_caps+0x4a1/0x760 [ceph] [14010.111384] RIP: 0010:lock_acquire+0xf0/0x2b0 [14010.120765] ? __file_remove_privs+0x89/0x110 [14010.125548] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14010.133016] ? wake_bit_function+0x70/0x70 [14010.137712] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14010.142325] ? ceph_write_iter+0x260/0x770 [ceph] [14010.143826] [14010.143827] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14010.148346] ceph_write_iter+0x260/0x770 [ceph] [14010.152698] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14010.157053] ? lock_is_held_type+0xaa/0x120 [14010.175787] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14010.179880] vfs_write+0x324/0x4f0 [14010.185095] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14010.189796] ksys_pwrite64+0x65/0xa0 [14010.191283] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14010.198408] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14010.202929] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14010.210056] do_syscall_64+0x38/0x80 [14010.214231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14010.221357] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14010.224753] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14010.231878] RIP: 0033:0x7f48f0eeb93b [14010.235448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14010.242573] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14010.247704] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14010.255782] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14010.259350] Call Trace: [14010.265090] ORIG_RAX: 0000000000000012 [14010.270135] [14010.277257] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14010.280831] ? watchdog_timer_fn+0x1b4/0x220 [14010.287952] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14010.306690] ? __hrtimer_run_queues+0x1d3/0x360 [14010.313830] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14010.319051] ? hrtimer_interrupt+0x104/0x240 [14010.321493] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14010.325322] ? __local_bh_enable+0x50/0x60 [14010.327335] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14010.334461] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14010.338729] [14010.345848] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14010.392457] [14010.394554] [14010.396652] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14010.401959] ? lock_acquire+0xf0/0x2b0 [14010.405709] ? igrab+0x43/0x50 [14010.408759] ? lock_release+0xc5/0x260 [14010.412505] _raw_spin_lock+0x30/0x40 [14010.416170] ? igrab+0x1d/0x50 [14010.419219] igrab+0x1d/0x50 [14010.422098] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14010.427249] delayed_work+0x1d8/0x280 [ceph] [14010.431530] ? process_scheduled_works+0x22a/0x510 [14010.436337] process_scheduled_works+0x2a4/0x510 [14010.440953] worker_thread+0x188/0x340 [14010.444701] ? rescuer_thread+0x310/0x310 [14010.448705] kthread+0xf5/0x130 [14010.451851] ? kthread_complete_and_exit+0x20/0x20 [14010.456636] ret_from_fork+0x31/0x40 [14010.460214] ? kthread_complete_and_exit+0x20/0x20 [14010.464998] ret_from_fork_asm+0x11/0x20 [14010.468931] [14037.700637] watchdog: BUG: soft lockup - CPU#0 stuck for 1959s! [dbench:78683] [14037.707850] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14037.712636] watchdog: BUG: soft lockup - CPU#7 stuck for 10956s! [kworker/7:4:78808] [14037.763289] irq event stamp: 17238478 [14037.763291] hardirqs last enabled at (17238477): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14037.771031] Modules linked in: [14037.774688] hardirqs last disabled at (17238478): [] sysvec_apic_timer_interrupt+0xe/0x90 [14037.784845] ceph [14037.787896] softirqs last enabled at (17238458): [] __do_softirq+0x2be/0x371 [14037.797619] libceph [14037.799542] softirqs last disabled at (17238453): [] irq_exit_rcu+0xb0/0xd0 [14037.808227] dns_resolver fscache [14037.810413] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14037.818922] netfs [14037.822233] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14037.831176] veth nft_chain_nat [14037.833189] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [14037.840658] xt_MASQUERADE [14037.843797] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [14037.849274] nf_nat nf_conntrack [14037.851977] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14037.870715] nf_defrag_ipv6 nf_defrag_ipv4 [14037.873939] [14037.873940] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14037.879157] nft_compat nf_tables [14037.883248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14037.884738] nfnetlink bridge [14037.891861] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14037.895174] stp llc [14037.902297] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14037.905261] xfs libcrc32c [14037.912386] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14037.914569] sunrpc intel_rapl_msr [14037.921692] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14037.924397] intel_rapl_common sb_edac [14037.931521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14037.934917] x86_pkg_temp_thermal coretemp [14037.942996] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14037.946740] kvm_intel [14037.952479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14037.956568] kvm irqbypass [14037.963692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14037.966050] i2c_i801 iTCO_wdt [14037.973175] Call Trace: [14037.973177] [14037.975879] iTCO_vendor_support i2c_smbus [14037.983003] ? watchdog_timer_fn+0x1b4/0x220 [14037.986053] crct10dif_pclmul joydev [14037.988499] ? __hrtimer_run_queues+0x1d3/0x360 [14037.990507] crc32_pclmul ipmi_ssif [14037.994601] ? hrtimer_interrupt+0x104/0x240 [14037.998863] ghash_clmulni_intel [14038.002434] ? __local_bh_enable+0x50/0x60 [14038.006958] mei_me mei [14038.010443] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14038.014707] lpc_ich sha512_ssse3 [14038.017929] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14038.022035] acpi_ipmi [14038.024481] [14038.029698] mfd_core wmi [14038.033010] [14038.037985] ipmi_si [14038.040343] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14038.042439] ipmi_devintf ipmi_msghandler [14038.045060] ? queued_spin_lock_slowpath+0x2b/0x2e0 [14038.047155] ioatdma acpi_pad [14038.049340] do_raw_spin_lock+0xb5/0xc0 [14038.054641] acpi_power_meter ixgbe [14038.058649] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14038.063517] igb mdio [14038.066484] try_get_cap_refs+0x159/0x880 [ceph] [14038.070317] crc32c_intel nvme [14038.073809] __ceph_get_caps+0x4a1/0x760 [ceph] [14038.078590] ptp nvme_core [14038.080864] ? __file_remove_privs+0x89/0x110 [14038.085470] i2c_algo_bit pps_core [14038.088522] ? wake_bit_function+0x70/0x70 [14038.093046] dca [14038.095752] ? ceph_write_iter+0x260/0x770 [ceph] [14038.100099] fuse [14038.103497] ceph_write_iter+0x260/0x770 [ceph] [14038.107589] irq event stamp: 35568752 [14038.109429] ? lock_is_held_type+0xaa/0x120 [14038.114123] hardirqs last enabled at (35568751): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14038.116049] vfs_write+0x324/0x4f0 [14038.120571] hardirqs last disabled at (35568752): [] sysvec_apic_timer_interrupt+0xe/0x90 [14038.124234] ksys_pwrite64+0x65/0xa0 [14038.128407] softirqs last enabled at (35568750): [] __do_softirq+0x2be/0x371 [14038.138564] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14038.141960] softirqs last disabled at (35568745): [] irq_exit_rcu+0xb0/0xd0 [14038.151687] do_syscall_64+0x38/0x80 [14038.155257] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14038.163942] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14038.169071] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14038.177582] RIP: 0033:0x7f48f0eeb93b [14038.181151] Workqueue: events delayed_work [ceph] [14038.190530] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14038.195573] [14038.195574] RIP: 0010:lock_acquire+0xf0/0x2b0 [14038.203044] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14038.206616] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14038.211318] ORIG_RAX: 0000000000000012 [14038.230060] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14038.231550] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14038.235900] [14038.235901] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14038.241119] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14038.259855] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14038.263686] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14038.268902] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14038.276039] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14038.277526] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14038.284652] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14038.291775] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14038.298905] [14038.306041] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14038.351923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14038.357662] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14038.364786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14038.371910] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14038.379039] Call Trace: [14038.381488] [14038.383498] ? watchdog_timer_fn+0x1b4/0x220 [14038.387763] ? __hrtimer_run_queues+0x1d3/0x360 [14038.392289] ? hrtimer_interrupt+0x104/0x240 [14038.396558] ? __local_bh_enable+0x50/0x60 [14038.400651] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14038.405875] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14038.410833] [14038.412929] [14038.415041] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14038.420352] ? lock_acquire+0xf0/0x2b0 [14038.424101] ? igrab+0x43/0x50 [14038.427152] ? lock_release+0xc5/0x260 [14038.430896] _raw_spin_lock+0x30/0x40 [14038.434552] ? igrab+0x1d/0x50 [14038.437603] igrab+0x1d/0x50 [14038.440482] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14038.445631] delayed_work+0x1d8/0x280 [ceph] [14038.449914] ? process_scheduled_works+0x22a/0x510 [14038.454703] process_scheduled_works+0x2a4/0x510 [14038.459325] worker_thread+0x188/0x340 [14038.463078] ? rescuer_thread+0x310/0x310 [14038.467088] kthread+0xf5/0x130 [14038.470225] ? kthread_complete_and_exit+0x20/0x20 [14038.475012] ret_from_fork+0x31/0x40 [14038.478607] ? kthread_complete_and_exit+0x20/0x20 [14038.483390] ret_from_fork_asm+0x11/0x20 [14038.487315] [14065.701125] watchdog: BUG: soft lockup - CPU#0 stuck for 1985s! [dbench:78683] [14065.708344] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14065.713124] watchdog: BUG: soft lockup - CPU#7 stuck for 10982s! [kworker/7:4:78808] [14065.763799] irq event stamp: 17296454 [14065.771532] Modules linked in: ceph libceph dns_resolver fscache netfs veth [14065.775191] hardirqs last enabled at (17296453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14065.782150] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [14065.792309] hardirqs last disabled at (17296454): [] sysvec_apic_timer_interrupt+0xe/0x90 [14065.800558] nft_compat nf_tables nfnetlink bridge stp llc xfs [14065.810283] softirqs last enabled at (17296406): [] __do_softirq+0x2be/0x371 [14065.816114] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [14065.824801] softirqs last disabled at (17296401): [] irq_exit_rcu+0xb0/0xd0 [14065.831325] x86_pkg_temp_thermal coretemp kvm_intel [14065.839836] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14065.839837] kvm irqbypass i2c_i801 [14065.853754] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14065.853755] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14065.857239] iTCO_wdt [14065.864711] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14065.870188] iTCO_vendor_support [14065.872459] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14065.891197] i2c_smbus crct10dif_pclmul [14065.894421] [14065.894422] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14065.899636] joydev crc32_pclmul [14065.903469] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14065.904960] ipmi_ssif ghash_clmulni_intel [14065.912083] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14065.915307] mei_me mei [14065.922431] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14065.926522] lpc_ich [14065.933646] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14065.936090] sha512_ssse3 acpi_ipmi [14065.943213] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14065.945400] mfd_core [14065.952521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14065.956006] wmi ipmi_si [14065.964083] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14065.966356] ipmi_devintf ipmi_msghandler [14065.972092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14065.974623] ioatdma acpi_pad [14065.981748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14065.985751] acpi_power_meter [14065.992876] Call Trace: [14065.992878] [14065.995839] ixgbe igb [14066.002966] ? watchdog_timer_fn+0x1b4/0x220 [14066.005927] mdio crc32c_intel [14066.008373] ? __hrtimer_run_queues+0x1d3/0x360 [14066.010382] nvme [14066.012744] ? hrtimer_interrupt+0x104/0x240 [14066.017003] ptp nvme_core [14066.020054] ? __local_bh_enable+0x50/0x60 [14066.024579] i2c_algo_bit [14066.026522] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14066.030798] pps_core dca [14066.033524] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14066.037612] fuse [14066.040230] [14066.045447] irq event stamp: 35791240 [14066.048067] [14066.048068] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14066.053023] hardirqs last enabled at (35791239): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14066.054951] ? queued_spin_lock_slowpath+0x16/0x2e0 [14066.057045] hardirqs last disabled at (35791240): [] sysvec_apic_timer_interrupt+0xe/0x90 [14066.060703] do_raw_spin_lock+0xb5/0xc0 [14066.062800] softirqs last enabled at (35791238): [] __do_softirq+0x2be/0x371 [14066.068124] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14066.078277] softirqs last disabled at (35791233): [] irq_exit_rcu+0xb0/0xd0 [14066.083151] try_get_cap_refs+0x159/0x880 [ceph] [14066.092873] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14066.096711] __ceph_get_caps+0x4a1/0x760 [ceph] [14066.105387] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14066.110177] ? __file_remove_privs+0x89/0x110 [14066.118684] Workqueue: events delayed_work [ceph] [14066.123293] ? wake_bit_function+0x70/0x70 [14066.132672] [14066.132673] RIP: 0010:lock_acquire+0xf0/0x2b0 [14066.137197] ? ceph_write_iter+0x260/0x770 [ceph] [14066.144666] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14066.149017] ceph_write_iter+0x260/0x770 [ceph] [14066.153713] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14066.157823] ? lock_is_held_type+0xaa/0x120 [14066.159313] [14066.159313] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14066.163667] vfs_write+0x324/0x4f0 [14066.168360] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14066.187105] ksys_pwrite64+0x65/0xa0 [14066.191623] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14066.196840] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14066.201017] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14066.202527] do_syscall_64+0x38/0x80 [14066.209649] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14066.213047] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14066.220171] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14066.223744] RIP: 0033:0x7f48f0eeb93b [14066.230865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14066.235998] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14066.243121] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14066.246693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14066.253815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14066.258859] ORIG_RAX: 0000000000000012 [14066.266938] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14066.270527] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14066.276263] Call Trace: [14066.276265] [14066.295000] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14066.295001] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14066.302127] ? watchdog_timer_fn+0x1b4/0x220 [14066.307342] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14066.314467] ? __hrtimer_run_queues+0x1d3/0x360 [14066.318296] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14066.325425] ? hrtimer_interrupt+0x104/0x240 [14066.332552] [14066.334989] ? __local_bh_enable+0x50/0x60 [14066.384845] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14066.390067] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14066.395025] [14066.397123] [14066.399221] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14066.404535] ? lock_acquire+0xf0/0x2b0 [14066.408278] ? igrab+0x43/0x50 [14066.411329] ? lock_release+0xc5/0x260 [14066.415073] _raw_spin_lock+0x30/0x40 [14066.418737] ? igrab+0x1d/0x50 [14066.421805] igrab+0x1d/0x50 [14066.424685] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14066.429834] delayed_work+0x1d8/0x280 [ceph] [14066.434115] ? process_scheduled_works+0x22a/0x510 [14066.438905] process_scheduled_works+0x2a4/0x510 [14066.443537] worker_thread+0x188/0x340 [14066.447287] ? rescuer_thread+0x310/0x310 [14066.451289] kthread+0xf5/0x130 [14066.454428] ? kthread_complete_and_exit+0x20/0x20 [14066.459213] ret_from_fork+0x31/0x40 [14066.462791] ? kthread_complete_and_exit+0x20/0x20 [14066.467574] ret_from_fork_asm+0x11/0x20 [14066.471498] [14093.701612] watchdog: BUG: soft lockup - CPU#0 stuck for 2011s! [dbench:78683] [14093.708831] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14093.713612] watchdog: BUG: soft lockup - CPU#7 stuck for 11008s! [kworker/7:4:78808] [14093.764289] irq event stamp: 17354374 [14093.772021] Modules linked in: ceph libceph [14093.775676] hardirqs last enabled at (17354373): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14093.779855] dns_resolver fscache netfs veth [14093.790011] hardirqs last disabled at (17354374): [] sysvec_apic_timer_interrupt+0xe/0x90 [14093.794289] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [14093.804010] softirqs last enabled at (17354328): [] __do_softirq+0x2be/0x371 [14093.809659] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [14093.818344] softirqs last disabled at (17354323): [] irq_exit_rcu+0xb0/0xd0 [14093.818345] nf_tables nfnetlink bridge [14093.831899] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14093.831901] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14093.835729] stp llc [14093.844674] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14093.852144] xfs libcrc32c [14093.854330] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14093.859805] sunrpc [14093.862510] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14093.881247] intel_rapl_msr intel_rapl_common [14093.883344] [14093.883345] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14093.888564] sb_edac x86_pkg_temp_thermal [14093.892913] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14093.894404] coretemp kvm_intel [14093.901528] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14093.905531] kvm irqbypass [14093.912658] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14093.915793] i2c_i801 iTCO_wdt [14093.922917] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14093.925622] iTCO_vendor_support i2c_smbus [14093.932747] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14093.935798] crct10dif_pclmul joydev [14093.942921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14093.947012] crc32_pclmul ipmi_ssif [14093.955088] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14093.958660] ghash_clmulni_intel [14093.964398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14093.967880] mei_me mei [14093.975008] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14093.978228] lpc_ich sha512_ssse3 [14093.985353] Call Trace: [14093.985355] [14093.987798] acpi_ipmi mfd_core [14093.994924] ? watchdog_timer_fn+0x1b4/0x220 [14093.998233] wmi ipmi_si [14094.000677] ? __hrtimer_run_queues+0x1d3/0x360 [14094.002689] ipmi_devintf [14094.005827] ? hrtimer_interrupt+0x104/0x240 [14094.010089] ipmi_msghandler ioatdma [14094.012620] ? __local_bh_enable+0x50/0x60 [14094.017143] acpi_pad [14094.019763] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14094.024026] acpi_power_meter [14094.027597] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14094.031686] ixgbe igb [14094.033957] [14094.039174] mdio crc32c_intel [14094.042139] [14094.047097] nvme [14094.049455] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14094.051551] ptp [14094.054605] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14094.056700] nvme_core i2c_algo_bit pps_core [14094.058626] do_raw_spin_lock+0xb5/0xc0 [14094.063928] dca fuse [14094.065766] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14094.070634] [14094.070635] irq event stamp: 36014110 [14094.074903] try_get_cap_refs+0x159/0x880 [ceph] [14094.078730] hardirqs last enabled at (36014109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14094.081017] __ceph_get_caps+0x4a1/0x760 [ceph] [14094.085793] hardirqs last disabled at (36014110): [] sysvec_apic_timer_interrupt+0xe/0x90 [14094.087298] ? __file_remove_privs+0x89/0x110 [14094.090951] softirqs last enabled at (36014108): [] __do_softirq+0x2be/0x371 [14094.095563] ? wake_bit_function+0x70/0x70 [14094.105720] softirqs last disabled at (36014103): [] irq_exit_rcu+0xb0/0xd0 [14094.110245] ? ceph_write_iter+0x260/0x770 [ceph] [14094.119968] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14094.124317] ceph_write_iter+0x260/0x770 [ceph] [14094.133014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14094.137104] ? lock_is_held_type+0xaa/0x120 [14094.145612] Workqueue: events delayed_work [ceph] [14094.150311] vfs_write+0x324/0x4f0 [14094.159687] [14094.164217] ksys_pwrite64+0x65/0xa0 [14094.171681] RIP: 0010:lock_acquire+0xf0/0x2b0 [14094.175859] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14094.180557] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14094.183955] do_syscall_64+0x38/0x80 [14094.185444] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14094.189017] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14094.193365] [14094.193366] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14094.198499] RIP: 0033:0x7f48f0eeb93b [14094.217235] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14094.220806] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14094.226023] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14094.231067] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14094.232557] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14094.239682] ORIG_RAX: 0000000000000012 [14094.243252] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14094.250378] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14094.269115] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14094.276238] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14094.281455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14094.288579] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14094.292410] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14094.299535] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14094.306658] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14094.314735] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14094.321860] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14094.327605] [14094.334721] Call Trace: [14094.334722] [14094.376983] ? watchdog_timer_fn+0x1b4/0x220 [14094.381246] ? __hrtimer_run_queues+0x1d3/0x360 [14094.385773] ? hrtimer_interrupt+0x104/0x240 [14094.390042] ? __local_bh_enable+0x50/0x60 [14094.394136] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14094.399359] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14094.404318] [14094.406414] [14094.408514] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14094.413828] ? lock_acquire+0xf0/0x2b0 [14094.417579] ? igrab+0x43/0x50 [14094.420629] ? lock_release+0xc5/0x260 [14094.424375] _raw_spin_lock+0x30/0x40 [14094.428039] ? igrab+0x1d/0x50 [14094.431089] igrab+0x1d/0x50 [14094.433966] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14094.439117] delayed_work+0x1d8/0x280 [ceph] [14094.443399] ? process_scheduled_works+0x22a/0x510 [14094.448188] process_scheduled_works+0x2a4/0x510 [14094.452803] worker_thread+0x188/0x340 [14094.456552] ? rescuer_thread+0x310/0x310 [14094.460556] kthread+0xf5/0x130 [14094.463696] ? kthread_complete_and_exit+0x20/0x20 [14094.468478] ret_from_fork+0x31/0x40 [14094.472050] ? kthread_complete_and_exit+0x20/0x20 [14094.476833] ret_from_fork_asm+0x11/0x20 [14094.480756] [14121.702100] watchdog: BUG: soft lockup - CPU#0 stuck for 2037s! [dbench:78683] [14121.709317] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14121.714100] watchdog: BUG: soft lockup - CPU#7 stuck for 11034s! [kworker/7:4:78808] [14121.764776] irq event stamp: 17412282 [14121.764777] hardirqs last enabled at (17412281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14121.772506] Modules linked in: [14121.776164] hardirqs last disabled at (17412282): [] sysvec_apic_timer_interrupt+0xe/0x90 [14121.786321] ceph [14121.789372] softirqs last enabled at (17412246): [] __do_softirq+0x2be/0x371 [14121.799096] libceph [14121.801020] softirqs last disabled at (17412241): [] irq_exit_rcu+0xb0/0xd0 [14121.809706] dns_resolver fscache [14121.811890] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14121.820400] netfs veth [14121.823711] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14121.832655] nft_chain_nat [14121.835098] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14121.842570] xt_MASQUERADE nf_nat [14121.845274] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14121.850752] nf_conntrack nf_defrag_ipv6 [14121.854061] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14121.872800] nf_defrag_ipv4 [14121.876717] [14121.881934] nft_compat nf_tables [14121.884724] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14121.886216] nfnetlink [14121.889527] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14121.896649] bridge stp [14121.899009] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14121.906133] llc xfs [14121.908575] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14121.915699] libcrc32c [14121.917885] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14121.925007] sunrpc intel_rapl_msr [14121.927365] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14121.934496] intel_rapl_common [14121.937896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14121.945973] sb_edac [14121.949023] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14121.954778] x86_pkg_temp_thermal coretemp [14121.956962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14121.964087] kvm_intel kvm [14121.968177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14121.975303] irqbypass i2c_i801 [14121.978005] Call Trace: [14121.978006] [14121.985129] iTCO_wdt iTCO_vendor_support [14121.988268] ? watchdog_timer_fn+0x1b4/0x220 [14121.990712] i2c_smbus [14121.992723] ? __hrtimer_run_queues+0x1d3/0x360 [14121.996728] crct10dif_pclmul [14122.000995] ? hrtimer_interrupt+0x104/0x240 [14122.003347] joydev crc32_pclmul [14122.007872] ? __local_bh_enable+0x50/0x60 [14122.010835] ipmi_ssif ghash_clmulni_intel [14122.015102] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14122.018325] mei_me [14122.022416] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14122.026506] mei lpc_ich [14122.031723] [14122.033820] sha512_ssse3 acpi_ipmi [14122.038785] [14122.038787] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14122.041318] mfd_core [14122.043419] ? queued_spin_lock_slowpath+0x16/0x2e0 [14122.046898] wmi ipmi_si [14122.048998] do_raw_spin_lock+0xb5/0xc0 [14122.054300] ipmi_devintf ipmi_msghandler [14122.056574] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14122.061441] ioatdma [14122.063975] try_get_cap_refs+0x159/0x880 [ceph] [14122.067803] acpi_pad acpi_power_meter [14122.071813] __ceph_get_caps+0x4a1/0x760 [ceph] [14122.076592] ixgbe [14122.078791] ? __file_remove_privs+0x89/0x110 [14122.083395] igb mdio [14122.087139] ? wake_bit_function+0x70/0x70 [14122.091662] crc32c_intel [14122.093675] ? ceph_write_iter+0x260/0x770 [ceph] [14122.098024] nvme [14122.100295] ceph_write_iter+0x260/0x770 [ceph] [14122.104386] ptp [14122.107007] ? lock_is_held_type+0xaa/0x120 [14122.111701] nvme_core [14122.113626] vfs_write+0x324/0x4f0 [14122.118150] i2c_algo_bit [14122.119992] ksys_pwrite64+0x65/0xa0 [14122.124163] pps_core [14122.126522] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14122.129918] dca fuse [14122.132538] do_syscall_64+0x38/0x80 [14122.136107] [14122.138377] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14122.143508] irq event stamp: 36236154 [14122.145784] RIP: 0033:0x7f48f0eeb93b [14122.149349] hardirqs last enabled at (36236153): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14122.150840] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14122.155885] hardirqs last disabled at (36236154): [] sysvec_apic_timer_interrupt+0xe/0x90 [14122.159540] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14122.163112] softirqs last enabled at (36236152): [] __do_softirq+0x2be/0x371 [14122.173269] ORIG_RAX: 0000000000000012 [14122.192006] softirqs last disabled at (36236147): [] irq_exit_rcu+0xb0/0xd0 [14122.201731] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14122.206949] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14122.215633] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14122.219463] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14122.227975] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14122.235100] Workqueue: events delayed_work [ceph] [14122.244475] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14122.251618] [14122.251618] RIP: 0010:lock_acquire+0x4a/0x2b0 [14122.259088] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14122.266212] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [14122.270916] [14122.278034] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [14122.317140] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [14122.324263] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [14122.331388] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [14122.338511] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [14122.345635] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14122.352761] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14122.360836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14122.366575] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14122.373700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14122.380823] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14122.387948] Call Trace: [14122.390393] [14122.392403] ? watchdog_timer_fn+0x1b4/0x220 [14122.396667] ? __hrtimer_run_queues+0x1d3/0x360 [14122.401193] ? hrtimer_interrupt+0x104/0x240 [14122.405462] ? __local_bh_enable+0x50/0x60 [14122.409556] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14122.414796] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14122.419755] [14122.421851] [14122.423950] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14122.429259] ? lock_acquire+0x4a/0x2b0 [14122.433008] ? igrab+0x43/0x50 [14122.436057] ? lock_release+0xc5/0x260 [14122.439802] _raw_spin_lock+0x30/0x40 [14122.443469] ? igrab+0x1d/0x50 [14122.446519] igrab+0x1d/0x50 [14122.449398] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14122.454545] delayed_work+0x1d8/0x280 [ceph] [14122.458828] ? process_scheduled_works+0x22a/0x510 [14122.463618] process_scheduled_works+0x2a4/0x510 [14122.468235] worker_thread+0x188/0x340 [14122.471981] ? rescuer_thread+0x310/0x310 [14122.475984] kthread+0xf5/0x130 [14122.479123] ? kthread_complete_and_exit+0x20/0x20 [14122.483908] ret_from_fork+0x31/0x40 [14122.487504] ? kthread_complete_and_exit+0x20/0x20 [14122.492288] ret_from_fork_asm+0x11/0x20 [14122.496220] [14149.702588] watchdog: BUG: soft lockup - CPU#0 stuck for 2063s! [dbench:78683] [14149.709805] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14149.714588] watchdog: BUG: soft lockup - CPU#7 stuck for 11060s! [kworker/7:4:78808] [14149.765262] irq event stamp: 17470384 [14149.772995] Modules linked in: ceph libceph dns_resolver fscache netfs [14149.776652] hardirqs last enabled at (17470383): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14149.783169] veth nft_chain_nat xt_MASQUERADE [14149.793326] hardirqs last disabled at (17470384): [] sysvec_apic_timer_interrupt+0xe/0x90 [14149.797678] nf_nat nf_conntrack nf_defrag_ipv6 [14149.807400] softirqs last enabled at (17470358): [] __do_softirq+0x2be/0x371 [14149.807402] nf_defrag_ipv4 [14149.807403] softirqs last disabled at (17470353): [] irq_exit_rcu+0xb0/0xd0 [14149.811925] nft_compat nf_tables [14149.820612] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14149.823401] nfnetlink bridge [14149.831912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14149.835222] stp llc [14149.844165] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14149.847130] xfs libcrc32c [14149.854600] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14149.856785] sunrpc intel_rapl_msr [14149.862263] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14149.864984] intel_rapl_common [14149.883721] [14149.883722] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14149.887119] sb_edac x86_pkg_temp_thermal [14149.892337] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14149.895386] coretemp kvm_intel [14149.896877] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14149.904001] kvm irqbypass [14149.908005] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14149.915129] i2c_i801 iTCO_wdt [14149.918267] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14149.925391] iTCO_vendor_support [14149.928095] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14149.935219] i2c_smbus crct10dif_pclmul [14149.938269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14149.945395] joydev crc32_pclmul [14149.948618] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14149.956696] ipmi_ssif ghash_clmulni_intel [14149.960528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14149.966274] mei_me mei [14149.969496] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14149.976622] lpc_ich sha512_ssse3 [14149.980712] Call Trace: [14149.980713] [14149.987837] acpi_ipmi mfd_core [14149.990283] ? watchdog_timer_fn+0x1b4/0x220 [14149.997404] wmi ipmi_si [14150.000715] ? __hrtimer_run_queues+0x1d3/0x360 [14150.003160] ipmi_devintf ipmi_msghandler [14150.005174] ? hrtimer_interrupt+0x104/0x240 [14150.008307] ioatdma acpi_pad [14150.012571] ? __local_bh_enable+0x50/0x60 [14150.015101] acpi_power_meter ixgbe [14150.019629] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14150.023630] igb mdio [14150.027896] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14150.030859] crc32c_intel nvme [14150.034949] [14150.038433] ptp [14150.043650] [14150.043652] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14150.045922] nvme_core i2c_algo_bit [14150.050882] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14150.053929] pps_core dca [14150.056028] do_raw_spin_lock+0xb5/0xc0 [14150.057864] fuse [14150.059964] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14150.065283] irq event stamp: 36458496 [14150.068769] try_get_cap_refs+0x159/0x880 [ceph] [14150.073637] hardirqs last enabled at (36458495): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14150.076276] __ceph_get_caps+0x4a1/0x760 [ceph] [14150.080102] hardirqs last disabled at (36458496): [] sysvec_apic_timer_interrupt+0xe/0x90 [14150.082030] ? __file_remove_privs+0x89/0x110 [14150.086812] softirqs last enabled at (36458494): [] __do_softirq+0x2be/0x371 [14150.090468] ? wake_bit_function+0x70/0x70 [14150.095079] softirqs last disabled at (36458485): [] irq_exit_rcu+0xb0/0xd0 [14150.105241] ? ceph_write_iter+0x260/0x770 [ceph] [14150.109760] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14150.119485] ceph_write_iter+0x260/0x770 [ceph] [14150.123835] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14150.132524] ? lock_is_held_type+0xaa/0x120 [14150.136610] Workqueue: events delayed_work [ceph] [14150.145125] vfs_write+0x324/0x4f0 [14150.149820] [14150.149821] RIP: 0010:lock_acquire+0xf0/0x2b0 [14150.159201] ksys_pwrite64+0x65/0xa0 [14150.163722] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14150.171193] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14150.175369] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14150.180067] do_syscall_64+0x38/0x80 [14150.183464] [14150.183465] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14150.184955] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14150.189306] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14150.192877] RIP: 0033:0x7f48f0eeb93b [14150.211615] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14150.216745] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14150.221961] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14150.225533] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14150.227023] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14150.234146] ORIG_RAX: 0000000000000012 [14150.239190] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14150.246316] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14150.249886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14150.257011] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14150.275749] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14150.282872] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14150.288091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14150.295214] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14150.299044] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14150.307122] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14150.314245] Call Trace: [14150.314247] [14150.319998] [14150.327117] ? watchdog_timer_fn+0x1b4/0x220 [14150.380766] ? __hrtimer_run_queues+0x1d3/0x360 [14150.385292] ? hrtimer_interrupt+0x104/0x240 [14150.389562] ? __local_bh_enable+0x50/0x60 [14150.393655] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14150.398879] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14150.403835] [14150.405932] [14150.408031] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14150.413340] ? lock_acquire+0xf0/0x2b0 [14150.417088] ? igrab+0x43/0x50 [14150.420139] ? lock_release+0xc5/0x260 [14150.423884] _raw_spin_lock+0x30/0x40 [14150.427540] ? igrab+0x1d/0x50 [14150.430590] igrab+0x1d/0x50 [14150.433469] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14150.438619] delayed_work+0x1d8/0x280 [ceph] [14150.442900] ? process_scheduled_works+0x22a/0x510 [14150.447690] process_scheduled_works+0x2a4/0x510 [14150.452305] worker_thread+0x188/0x340 [14150.456054] ? rescuer_thread+0x310/0x310 [14150.460058] kthread+0xf5/0x130 [14150.463205] ? kthread_complete_and_exit+0x20/0x20 [14150.467996] ret_from_fork+0x31/0x40 [14150.471566] ? kthread_complete_and_exit+0x20/0x20 [14150.476351] ret_from_fork_asm+0x11/0x20 [14150.480284] [14151.419984] rcu: INFO: rcu_preempt self-detected stall on CPU [14151.425743] rcu: 0-....: (2166583 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=1058370 [14151.436081] rcu: (t=2214771 jiffies g=1309609 q=146091 ncpus=8) [14151.442080] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14151.451031] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14151.458502] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14151.463980] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14151.482716] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14151.487934] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14151.495059] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14151.502182] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14151.509305] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14151.516433] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14151.523554] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14151.531633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14151.537369] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14151.544493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14151.551616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14151.558743] Call Trace: [14151.561185] [14151.563199] ? rcu_dump_cpu_stacks+0xd1/0x180 [14151.567557] ? rcu_sched_clock_irq+0x633/0x1390 [14151.572080] ? lock_is_held_type+0xaa/0x120 [14151.576258] ? lock_is_held_type+0xaa/0x120 [14151.580435] ? tick_sched_do_timer+0xa0/0xa0 [14151.584709] ? rcu_read_lock_sched_held+0x56/0x90 [14151.589406] ? tick_sched_do_timer+0xa0/0xa0 [14151.593670] ? update_process_times+0x5f/0x90 [14151.598020] ? tick_sched_handle.isra.16+0x1e/0x50 [14151.602804] ? tick_sched_timer+0x6f/0x80 [14151.606810] ? __hrtimer_run_queues+0x1d3/0x360 [14151.611335] ? hrtimer_interrupt+0x104/0x240 [14151.615597] ? __local_bh_enable+0x50/0x60 [14151.619690] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14151.624915] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14151.629871] [14151.631969] [14151.634066] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14151.639373] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14151.644251] do_raw_spin_lock+0xb5/0xc0 [14151.648080] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14151.652884] try_get_cap_refs+0x159/0x880 [ceph] [14151.657514] __ceph_get_caps+0x4a1/0x760 [ceph] [14151.662055] ? __file_remove_privs+0x89/0x110 [14151.666411] ? wake_bit_function+0x70/0x70 [14151.670503] ? ceph_write_iter+0x260/0x770 [ceph] [14151.675216] ceph_write_iter+0x260/0x770 [ceph] [14151.679752] ? lock_is_held_type+0xaa/0x120 [14151.683938] vfs_write+0x324/0x4f0 [14151.687344] ksys_pwrite64+0x65/0xa0 [14151.690919] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14151.696052] do_syscall_64+0x38/0x80 [14151.699623] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14151.704673] RIP: 0033:0x7f48f0eeb93b [14151.708245] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14151.726982] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [14151.734540] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14151.741665] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14151.748789] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14151.755912] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14151.763036] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14151.770166] [14151.772353] Sending NMI from CPU 0 to CPUs 7: [14151.776707] NMI backtrace for cpu 7 [14151.776709] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14151.776711] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14151.776712] Workqueue: events delayed_work [ceph] [14151.776727] RIP: 0010:mark_lock+0x5d/0x4c0 [14151.776729] Code: f5 89 d3 83 fa 08 74 6c 41 be 01 00 00 00 89 d1 41 d3 e6 4d 63 f6 41 0f b7 45 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 d3 7e e0 01 <73> 79 48 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 41 bc [14151.776730] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000007 [14151.776731] RAX: 00000000000006f5 RBX: 0000000000000002 RCX: 0000000000000002 [14151.776732] RDX: 0000000000000002 RSI: ffff88810debb550 RDI: ffff88810debaa80 [14151.776733] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [14151.776733] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [14151.776734] R13: ffff88810debb550 R14: 0000000000000004 R15: ffff88810debaa80 [14151.776735] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14151.776736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14151.776737] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14151.776737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14151.776738] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14151.776739] Call Trace: [14151.776739] [14151.776740] ? nmi_cpu_backtrace+0xad/0x120 [14151.776743] ? nmi_cpu_backtrace_handler+0x11/0x20 [14151.776745] ? nmi_handle+0xd4/0x260 [14151.776749] ? default_do_nmi+0x49/0xf0 [14151.776751] ? exc_nmi+0x11e/0x150 [14151.776752] ? end_repeat_nmi+0x16/0x67 [14151.776755] ? mark_lock+0x5d/0x4c0 [14151.776757] ? mark_lock+0x5d/0x4c0 [14151.776758] ? mark_lock+0x5d/0x4c0 [14151.776760] [14151.776760] [14151.776761] ? _raw_spin_unlock+0x23/0x40 [14151.776763] ? ceph_check_caps+0x275/0xae0 [ceph] [14151.776776] ? mark_held_locks+0x48/0x70 [14151.776778] __lock_acquire+0x49a/0x1e20 [14151.776781] lock_acquire+0xcb/0x2b0 [14151.776782] ? igrab+0x1d/0x50 [14151.776784] ? igrab+0x43/0x50 [14151.776785] ? lock_release+0xc5/0x260 [14151.776787] _raw_spin_lock+0x30/0x40 [14151.776789] ? igrab+0x1d/0x50 [14151.776790] igrab+0x1d/0x50 [14151.776791] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14151.776804] delayed_work+0x1d8/0x280 [ceph] [14151.776817] ? process_scheduled_works+0x22a/0x510 [14151.776819] process_scheduled_works+0x2a4/0x510 [14151.776823] worker_thread+0x188/0x340 [14151.776825] ? rescuer_thread+0x310/0x310 [14151.776828] kthread+0xf5/0x130 [14151.776830] ? kthread_complete_and_exit+0x20/0x20 [14151.776832] ret_from_fork+0x31/0x40 [14151.776834] ? kthread_complete_and_exit+0x20/0x20 [14151.776836] ret_from_fork_asm+0x11/0x20 [14151.776840] [14177.703076] watchdog: BUG: soft lockup - CPU#0 stuck for 2090s! [dbench:78683] [14177.710291] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14177.715076] watchdog: BUG: soft lockup - CPU#7 stuck for 11087s! [kworker/7:4:78808] [14177.765750] irq event stamp: 17527584 [14177.773482] Modules linked in: ceph libceph dns_resolver [14177.777138] hardirqs last enabled at (17527583): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14177.782443] fscache netfs veth [14177.792618] hardirqs last disabled at (17527584): [] sysvec_apic_timer_interrupt+0xe/0x90 [14177.795754] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [14177.805479] softirqs last enabled at (17527542): [] __do_softirq+0x2be/0x371 [14177.811129] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [14177.819815] softirqs last disabled at (17527537): [] irq_exit_rcu+0xb0/0xd0 [14177.825726] nfnetlink bridge stp [14177.834235] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14177.837547] llc xfs [14177.846491] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14177.848675] libcrc32c sunrpc intel_rapl_msr [14177.856147] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14177.856148] intel_rapl_common [14177.856150] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14177.860411] sb_edac x86_pkg_temp_thermal [14177.865887] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14177.868937] coretemp [14177.887676] [14177.887677] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14177.891678] kvm_intel kvm [14177.896897] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14177.899167] irqbypass i2c_i801 [14177.900657] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14177.907782] iTCO_wdt iTCO_vendor_support [14177.910488] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14177.917611] i2c_smbus crct10dif_pclmul [14177.920755] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14177.927880] joydev [14177.931884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14177.939010] crc32_pclmul ipmi_ssif [14177.942840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14177.949964] ghash_clmulni_intel mei_me [14177.952062] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14177.960140] mei [14177.963624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14177.969361] lpc_ich sha512_ssse3 [14177.973190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14177.980316] acpi_ipmi mfd_core [14177.982152] Call Trace: [14177.982154] [14177.989277] wmi ipmi_si [14177.992588] ? watchdog_timer_fn+0x1b4/0x220 [14177.999711] ipmi_devintf ipmi_msghandler [14178.002851] ? __hrtimer_run_queues+0x1d3/0x360 [14178.005293] ioatdma acpi_pad [14178.007308] ? hrtimer_interrupt+0x104/0x240 [14178.009834] acpi_power_meter ixgbe [14178.014099] ? __local_bh_enable+0x50/0x60 [14178.018102] igb mdio [14178.022628] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14178.025591] crc32c_intel nvme [14178.029856] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14178.033339] ptp nvme_core [14178.037429] [14178.039702] i2c_algo_bit [14178.044917] [14178.044918] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14178.047969] pps_core dca [14178.052930] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14178.055629] fuse [14178.057729] do_raw_spin_lock+0xb5/0xc0 [14178.060344] irq event stamp: 36679478 [14178.062443] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14178.067756] hardirqs last enabled at (36679477): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14178.070375] try_get_cap_refs+0x159/0x880 [ceph] [14178.075244] hardirqs last disabled at (36679478): [] sysvec_apic_timer_interrupt+0xe/0x90 [14178.077171] __ceph_get_caps+0x4a1/0x760 [ceph] [14178.080999] softirqs last enabled at (36679476): [] __do_softirq+0x2be/0x371 [14178.084660] ? __file_remove_privs+0x89/0x110 [14178.089439] softirqs last disabled at (36679471): [] irq_exit_rcu+0xb0/0xd0 [14178.099598] ? wake_bit_function+0x70/0x70 [14178.104208] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14178.113935] ? ceph_write_iter+0x260/0x770 [ceph] [14178.118474] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14178.127159] ceph_write_iter+0x260/0x770 [ceph] [14178.131508] Workqueue: events delayed_work [ceph] [14178.140024] ? lock_is_held_type+0xaa/0x120 [14178.144111] [14178.144112] RIP: 0010:lock_acquire+0xf0/0x2b0 [14178.153489] vfs_write+0x324/0x4f0 [14178.158183] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14178.165660] ksys_pwrite64+0x65/0xa0 [14178.170179] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14178.174878] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14178.179054] [14178.179055] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14178.180547] do_syscall_64+0x38/0x80 [14178.184895] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14178.188295] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14178.207032] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14178.210602] RIP: 0033:0x7f48f0eeb93b [14178.215820] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14178.220950] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14178.222441] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14178.229564] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14178.233134] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14178.240259] ORIG_RAX: 0000000000000012 [14178.245304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14178.252428] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14178.256000] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14178.263121] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14178.281860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14178.288985] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14178.294201] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14178.302280] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14178.306109] Call Trace: [14178.306110] [14178.311849] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14178.318974] ? watchdog_timer_fn+0x1b4/0x220 [14178.326102] [14178.333222] ? __hrtimer_run_queues+0x1d3/0x360 [14178.384280] ? hrtimer_interrupt+0x104/0x240 [14178.388548] ? __local_bh_enable+0x50/0x60 [14178.392643] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14178.397865] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14178.402825] [14178.404920] [14178.407019] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14178.412327] ? lock_acquire+0xf0/0x2b0 [14178.416078] ? igrab+0x43/0x50 [14178.419126] ? lock_release+0xc5/0x260 [14178.422871] _raw_spin_lock+0x30/0x40 [14178.426535] ? igrab+0x1d/0x50 [14178.429588] igrab+0x1d/0x50 [14178.432482] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14178.437632] delayed_work+0x1d8/0x280 [ceph] [14178.441913] ? process_scheduled_works+0x22a/0x510 [14178.446704] process_scheduled_works+0x2a4/0x510 [14178.451320] worker_thread+0x188/0x340 [14178.455069] ? rescuer_thread+0x310/0x310 [14178.459071] kthread+0xf5/0x130 [14178.462208] ? kthread_complete_and_exit+0x20/0x20 [14178.466995] ret_from_fork+0x31/0x40 [14178.470571] ? kthread_complete_and_exit+0x20/0x20 [14178.475356] ret_from_fork_asm+0x11/0x20 [14178.479280] [14204.276222] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14205.703564] watchdog: BUG: soft lockup - CPU#0 stuck for 2116s! [dbench:78683] [14205.710781] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14205.715564] watchdog: BUG: soft lockup - CPU#7 stuck for 11113s! [kworker/7:4:78808] [14205.766242] irq event stamp: 17585546 [14205.773977] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [14205.777636] hardirqs last enabled at (17585545): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14205.785808] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [14205.795966] hardirqs last disabled at (17585546): [] sysvec_apic_timer_interrupt+0xe/0x90 [14205.803964] nf_tables nfnetlink bridge stp llc xfs [14205.813689] softirqs last enabled at (17585510): [] __do_softirq+0x2be/0x371 [14205.818559] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [14205.827248] softirqs last disabled at (17585505): [] irq_exit_rcu+0xb0/0xd0 [14205.827251] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [14205.842281] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14205.842282] i2c_i801 iTCO_wdt [14205.857396] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14205.860446] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul [14205.867919] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14205.875214] ipmi_ssif ghash_clmulni_intel [14205.880694] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14205.880695] mei_me [14205.880697] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14205.884784] mei lpc_ich [14205.903520] [14205.905617] sha512_ssse3 acpi_ipmi [14205.910836] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14205.913367] mfd_core wmi [14205.914856] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14205.918341] ipmi_si [14205.925464] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14205.928083] ipmi_devintf ipmi_msghandler [14205.935206] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14205.937391] ioatdma acpi_pad [14205.944513] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14205.948517] acpi_power_meter ixgbe [14205.955643] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14205.958608] igb mdio [14205.965730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14205.969216] crc32c_intel nvme [14205.977293] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14205.979564] ptp nvme_core [14205.985302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14205.988351] i2c_algo_bit pps_core [14205.995475] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14205.998180] dca fuse [14206.005304] Call Trace: [14206.005307] [14206.008700] [14206.008701] irq event stamp: 36901434 [14206.015828] ? watchdog_timer_fn+0x1b4/0x220 [14206.018095] hardirqs last enabled at (36901433): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14206.020542] ? __hrtimer_run_queues+0x1d3/0x360 [14206.022550] hardirqs last disabled at (36901434): [] sysvec_apic_timer_interrupt+0xe/0x90 [14206.024044] ? hrtimer_interrupt+0x104/0x240 [14206.027698] softirqs last enabled at (36901432): [] __do_softirq+0x2be/0x371 [14206.031963] ? __local_bh_enable+0x50/0x60 [14206.042121] softirqs last disabled at (36901427): [] irq_exit_rcu+0xb0/0xd0 [14206.046647] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14206.056370] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14206.060633] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14206.069316] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14206.073407] [14206.081919] Workqueue: events delayed_work [ceph] [14206.087137] [14206.096512] [14206.096513] RIP: 0010:lock_release+0x0/0x260 [14206.101472] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14206.108942] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [14206.111062] ? queued_spin_lock_slowpath+0x16/0x2e0 [14206.115753] RSP: 0018:ffffc90002117d88 EFLAGS: 00000286 [14206.117853] do_raw_spin_lock+0xb5/0xc0 [14206.119341] [14206.119342] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [14206.123608] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14206.128911] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [14206.147651] try_get_cap_refs+0x159/0x880 [ceph] [14206.152517] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [14206.157742] __ceph_get_caps+0x4a1/0x760 [ceph] [14206.161565] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [14206.163063] ? __file_remove_privs+0x89/0x110 [14206.170181] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [14206.174968] ? wake_bit_function+0x70/0x70 [14206.182090] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14206.186702] ? ceph_write_iter+0x260/0x770 [ceph] [14206.193825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14206.198348] ceph_write_iter+0x260/0x770 [ceph] [14206.205473] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14206.209828] ? lock_is_held_type+0xaa/0x120 [14206.216964] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14206.221057] vfs_write+0x324/0x4f0 [14206.229133] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14206.233835] ksys_pwrite64+0x65/0xa0 [14206.239567] Call Trace: [14206.244094] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14206.251217] [14206.255396] do_syscall_64+0x38/0x80 [14206.262519] ? watchdog_timer_fn+0x1b4/0x220 [14206.265917] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14206.273041] ? __hrtimer_run_queues+0x1d3/0x360 [14206.276612] RIP: 0033:0x7f48f0eeb93b [14206.279057] ? hrtimer_interrupt+0x104/0x240 [14206.284186] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14206.286195] ? __local_bh_enable+0x50/0x60 [14206.289766] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14206.294034] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14206.299075] ORIG_RAX: 0000000000000012 [14206.303599] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14206.307170] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14206.311433] [14206.330170] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14206.334263] [14206.339479] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14206.344697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14206.348528] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14206.353488] ? igrab+0x43/0x50 [14206.360610] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14206.362706] ? lock_downgrade+0x250/0x250 [14206.369835] [14206.371929] ? _raw_spin_lock+0x30/0x40 [14206.411674] _raw_spin_unlock+0x1b/0x40 [14206.415507] igrab+0x43/0x50 [14206.418384] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14206.423535] delayed_work+0x1d8/0x280 [ceph] [14206.427815] ? process_scheduled_works+0x22a/0x510 [14206.432604] process_scheduled_works+0x2a4/0x510 [14206.437220] worker_thread+0x188/0x340 [14206.440971] ? rescuer_thread+0x310/0x310 [14206.444981] kthread+0xf5/0x130 [14206.448120] ? kthread_complete_and_exit+0x20/0x20 [14206.452902] ret_from_fork+0x31/0x40 [14206.456482] ? kthread_complete_and_exit+0x20/0x20 [14206.461267] ret_from_fork_asm+0x11/0x20 [14206.465200] [14214.515594] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14225.267700] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14233.704052] watchdog: BUG: soft lockup - CPU#0 stuck for 2142s! [dbench:78683] [14233.711266] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14233.716052] watchdog: BUG: soft lockup - CPU#7 stuck for 11139s! [kworker/7:4:78808] [14233.766725] irq event stamp: 17643436 [14233.774457] Modules linked in: ceph libceph [14233.778113] hardirqs last enabled at (17643435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14233.782290] dns_resolver fscache [14233.792447] hardirqs last disabled at (17643436): [] sysvec_apic_timer_interrupt+0xe/0x90 [14233.795759] netfs veth nft_chain_nat xt_MASQUERADE [14233.805484] softirqs last enabled at (17643398): [] __do_softirq+0x2be/0x371 [14233.810354] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [14233.819039] softirqs last disabled at (17643393): [] irq_exit_rcu+0xb0/0xd0 [14233.824863] nft_compat nf_tables nfnetlink [14233.833372] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14233.837550] bridge stp [14233.846495] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14233.846496] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [14233.848939] llc xfs [14233.856408] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [14233.861886] libcrc32c sunrpc [14233.864070] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14233.882809] intel_rapl_msr intel_rapl_common [14233.885772] [14233.890991] sb_edac x86_pkg_temp_thermal [14233.895341] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14233.896831] coretemp kvm_intel [14233.900836] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14233.907959] kvm irqbypass [14233.911098] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14233.918221] i2c_i801 iTCO_wdt [14233.920926] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14233.928049] iTCO_vendor_support i2c_smbus [14233.931101] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14233.938224] crct10dif_pclmul joydev [14233.942316] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14233.949451] crc32_pclmul ipmi_ssif [14233.953019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14233.961096] ghash_clmulni_intel mei_me [14233.964581] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14233.970317] mei lpc_ich [14233.974149] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14233.981272] sha512_ssse3 acpi_ipmi [14233.983803] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14233.990927] mfd_core wmi [14233.994412] Call Trace: [14233.994414] [14234.001534] ipmi_si ipmi_devintf [14234.004154] ? watchdog_timer_fn+0x1b4/0x220 [14234.006596] ipmi_msghandler ioatdma [14234.008608] ? __hrtimer_run_queues+0x1d3/0x360 [14234.011919] acpi_pad [14234.016187] ? hrtimer_interrupt+0x104/0x240 [14234.019753] acpi_power_meter ixgbe [14234.024277] ? __local_bh_enable+0x50/0x60 [14234.026547] igb mdio [14234.030814] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14234.034295] crc32c_intel nvme [14234.038388] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14234.040659] ptp [14234.045876] [14234.048926] nvme_core i2c_algo_bit [14234.053883] [14234.055721] pps_core dca [14234.057837] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14234.061320] fuse [14234.063422] ? queued_spin_lock_slowpath+0x2b/0x2e0 [14234.066034] irq event stamp: 37122562 [14234.071341] do_raw_spin_lock+0xb5/0xc0 [14234.073261] hardirqs last enabled at (37122561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14234.078135] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14234.081790] hardirqs last disabled at (37122562): [] sysvec_apic_timer_interrupt+0xe/0x90 [14234.085623] try_get_cap_refs+0x159/0x880 [ceph] [14234.095779] softirqs last enabled at (37122560): [] __do_softirq+0x2be/0x371 [14234.100567] __ceph_get_caps+0x4a1/0x760 [ceph] [14234.110286] softirqs last disabled at (37122555): [] irq_exit_rcu+0xb0/0xd0 [14234.114900] ? __file_remove_privs+0x89/0x110 [14234.123583] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14234.128107] ? wake_bit_function+0x70/0x70 [14234.136617] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14234.140970] ? ceph_write_iter+0x260/0x770 [ceph] [14234.150345] Workqueue: events delayed_work [ceph] [14234.154452] ceph_write_iter+0x260/0x770 [ceph] [14234.161922] [14234.161923] RIP: 0010:lock_acquire+0xf0/0x2b0 [14234.166625] ? lock_is_held_type+0xaa/0x120 [14234.171318] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14234.175846] vfs_write+0x324/0x4f0 [14234.177334] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14234.181689] ksys_pwrite64+0x65/0xa0 [14234.185861] [14234.185862] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14234.204599] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14234.207996] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14234.213215] do_syscall_64+0x38/0x80 [14234.216784] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14234.218277] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14234.225400] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14234.230529] RIP: 0033:0x7f48f0eeb93b [14234.237655] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14234.241224] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14234.248349] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14234.253394] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14234.260516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14234.264088] ORIG_RAX: 0000000000000012 [14234.271211] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14234.289951] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14234.298028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14234.303245] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14234.308983] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14234.312812] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14234.319936] Call Trace: [14234.319938] [14234.327062] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14234.334186] ? watchdog_timer_fn+0x1b4/0x220 [14234.341309] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14234.348454] ? __hrtimer_run_queues+0x1d3/0x360 [14234.355582] [14234.358022] ? hrtimer_interrupt+0x104/0x240 [14234.389515] ? __local_bh_enable+0x50/0x60 [14234.393608] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14234.398831] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14234.403789] [14234.405888] [14234.407985] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14234.413292] ? lock_acquire+0xf0/0x2b0 [14234.417043] ? igrab+0x43/0x50 [14234.420093] ? lock_release+0xc5/0x260 [14234.423838] _raw_spin_lock+0x30/0x40 [14234.427504] ? igrab+0x1d/0x50 [14234.430551] igrab+0x1d/0x50 [14234.433432] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14234.438599] delayed_work+0x1d8/0x280 [ceph] [14234.442880] ? process_scheduled_works+0x22a/0x510 [14234.447669] process_scheduled_works+0x2a4/0x510 [14234.452284] worker_thread+0x188/0x340 [14234.456034] ? rescuer_thread+0x310/0x310 [14234.460037] kthread+0xf5/0x130 [14234.463175] ? kthread_complete_and_exit+0x20/0x20 [14234.467960] ret_from_fork+0x31/0x40 [14234.471529] ? kthread_complete_and_exit+0x20/0x20 [14234.476314] ret_from_fork_asm+0x11/0x20 [14234.480237] [14235.507853] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14246.260228] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14256.500331] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14261.704540] watchdog: BUG: soft lockup - CPU#0 stuck for 2168s! [dbench:78683] [14261.711755] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14261.716540] watchdog: BUG: soft lockup - CPU#7 stuck for 11165s! [kworker/7:4:78808] [14261.767217] irq event stamp: 17701372 [14261.774951] Modules linked in: ceph libceph dns_resolver fscache netfs [14261.778609] hardirqs last enabled at (17701371): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14261.785134] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [14261.795292] hardirqs last disabled at (17701372): [] sysvec_apic_timer_interrupt+0xe/0x90 [14261.801376] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [14261.811100] softirqs last enabled at (17701362): [] __do_softirq+0x2be/0x371 [14261.817012] nfnetlink bridge stp llc xfs libcrc32c [14261.825694] softirqs last disabled at (17701357): [] irq_exit_rcu+0xb0/0xd0 [14261.830567] sunrpc intel_rapl_msr intel_rapl_common [14261.839078] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14261.844034] sb_edac x86_pkg_temp_thermal [14261.852978] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14261.852979] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14261.856982] coretemp [14261.864454] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14261.869935] kvm_intel kvm [14261.872216] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14261.890957] irqbypass i2c_i801 [14261.893661] [14261.893662] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14261.898879] iTCO_wdt iTCO_vendor_support [14261.902016] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14261.903505] i2c_smbus crct10dif_pclmul [14261.910629] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14261.914634] joydev crc32_pclmul [14261.921758] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14261.925589] ipmi_ssif [14261.932714] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14261.935938] ghash_clmulni_intel [14261.943062] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14261.945420] mei_me [14261.952542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14261.955767] mei [14261.963845] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14261.965942] lpc_ich [14261.971681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14261.973517] sha512_ssse3 acpi_ipmi [14261.980642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14261.982824] mfd_core wmi [14261.989948] Call Trace: [14261.993433] ipmi_si [14262.000560] [14262.003175] ipmi_devintf ipmi_msghandler [14262.005621] ? watchdog_timer_fn+0x1b4/0x220 [14262.007803] ioatdma acpi_pad [14262.009814] ? __hrtimer_run_queues+0x1d3/0x360 [14262.013818] acpi_power_meter [14262.018085] ? hrtimer_interrupt+0x104/0x240 [14262.021046] ixgbe igb [14262.025571] ? __local_bh_enable+0x50/0x60 [14262.028533] mdio crc32c_intel [14262.032801] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14262.035157] nvme [14262.039245] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14262.042298] ptp nvme_core [14262.047514] [14262.049437] i2c_algo_bit pps_core [14262.054396] [14262.054397] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14262.057100] dca fuse [14262.059203] ? queued_spin_lock_slowpath+0x18/0x2e0 [14262.062595] [14262.062596] irq event stamp: 37343602 [14262.064694] do_raw_spin_lock+0xb5/0xc0 [14262.069997] hardirqs last enabled at (37343601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14262.072270] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14262.077139] hardirqs last disabled at (37343602): [] sysvec_apic_timer_interrupt+0xe/0x90 [14262.078631] try_get_cap_refs+0x159/0x880 [ceph] [14262.082284] softirqs last enabled at (37343600): [] __do_softirq+0x2be/0x371 [14262.086122] __ceph_get_caps+0x4a1/0x760 [ceph] [14262.096274] softirqs last disabled at (37343595): [] irq_exit_rcu+0xb0/0xd0 [14262.101064] ? __file_remove_privs+0x89/0x110 [14262.110783] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14262.115394] ? wake_bit_function+0x70/0x70 [14262.124076] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14262.128605] ? ceph_write_iter+0x260/0x770 [ceph] [14262.137111] Workqueue: events delayed_work [ceph] [14262.141463] ceph_write_iter+0x260/0x770 [ceph] [14262.150839] [14262.150840] RIP: 0010:lock_acquire+0xf0/0x2b0 [14262.154946] ? lock_is_held_type+0xaa/0x120 [14262.162411] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14262.167110] vfs_write+0x324/0x4f0 [14262.171806] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14262.176333] ksys_pwrite64+0x65/0xa0 [14262.177820] [14262.182171] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14262.186347] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14262.205088] do_syscall_64+0x38/0x80 [14262.208484] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14262.213701] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14262.217272] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14262.218763] RIP: 0033:0x7f48f0eeb93b [14262.223894] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14262.231019] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14262.234588] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14262.241713] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14262.246757] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14262.253881] ORIG_RAX: 0000000000000012 [14262.257451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14262.264574] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14262.283314] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14262.290436] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14262.295655] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14262.303732] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14262.307563] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14262.313300] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14262.320424] Call Trace: [14262.320425] [14262.327548] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14262.334674] ? watchdog_timer_fn+0x1b4/0x220 [14262.341801] [14262.348939] ? __hrtimer_run_queues+0x1d3/0x360 [14262.385758] ? hrtimer_interrupt+0x104/0x240 [14262.390028] ? __local_bh_enable+0x50/0x60 [14262.394122] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14262.399346] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14262.404304] [14262.406399] [14262.408496] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14262.413804] ? lock_acquire+0xf0/0x2b0 [14262.417555] ? igrab+0x43/0x50 [14262.420604] ? lock_release+0xc5/0x260 [14262.424350] _raw_spin_lock+0x30/0x40 [14262.428006] ? igrab+0x1d/0x50 [14262.431057] igrab+0x1d/0x50 [14262.433947] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14262.439094] delayed_work+0x1d8/0x280 [ceph] [14262.443374] ? process_scheduled_works+0x22a/0x510 [14262.448164] process_scheduled_works+0x2a4/0x510 [14262.452782] worker_thread+0x188/0x340 [14262.456530] ? rescuer_thread+0x310/0x310 [14262.460540] kthread+0xf5/0x130 [14262.463681] ? kthread_complete_and_exit+0x20/0x20 [14262.468471] ret_from_fork+0x31/0x40 [14262.472051] ? kthread_complete_and_exit+0x20/0x20 [14262.476836] ret_from_fork_asm+0x11/0x20 [14262.480767] [14267.252435] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14277.492759] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14288.244832] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14289.705028] watchdog: BUG: soft lockup - CPU#0 stuck for 2194s! [dbench:78683] [14289.712241] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14289.717028] watchdog: BUG: soft lockup - CPU#7 stuck for 11191s! [kworker/7:4:78808] [14289.767698] irq event stamp: 17759312 [14289.767700] hardirqs last enabled at (17759311): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14289.775430] Modules linked in: [14289.779086] hardirqs last disabled at (17759312): [] sysvec_apic_timer_interrupt+0xe/0x90 [14289.789245] ceph [14289.792296] softirqs last enabled at (17759302): [] __do_softirq+0x2be/0x371 [14289.802019] libceph [14289.803943] softirqs last disabled at (17759297): [] irq_exit_rcu+0xb0/0xd0 [14289.812626] dns_resolver fscache [14289.814811] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14289.823323] netfs veth [14289.826633] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14289.826634] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14289.835577] nft_chain_nat [14289.838022] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14289.845492] xt_MASQUERADE nf_nat [14289.850969] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14289.853673] nf_conntrack nf_defrag_ipv6 [14289.872424] [14289.872424] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14289.875730] nf_defrag_ipv4 nft_compat [14289.880948] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14289.884865] nf_tables nfnetlink [14289.886355] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14289.893480] bridge stp [14289.897223] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14289.904350] llc xfs [14289.907573] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14289.914704] libcrc32c [14289.917150] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14289.924273] sunrpc intel_rapl_msr [14289.926458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14289.933582] intel_rapl_common sb_edac [14289.935940] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14289.944015] x86_pkg_temp_thermal coretemp [14289.947425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14289.953161] kvm_intel kvm [14289.956903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14289.964028] irqbypass i2c_i801 [14289.968119] Call Trace: [14289.968121] [14289.975243] iTCO_wdt iTCO_vendor_support [14289.977948] ? watchdog_timer_fn+0x1b4/0x220 [14289.985071] i2c_smbus crct10dif_pclmul [14289.988210] ? __hrtimer_run_queues+0x1d3/0x360 [14289.990654] joydev crc32_pclmul [14289.992666] ? hrtimer_interrupt+0x104/0x240 [14289.996666] ipmi_ssif ghash_clmulni_intel [14290.000931] ? __local_bh_enable+0x50/0x60 [14290.004763] mei_me mei [14290.009289] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14290.012510] lpc_ich sha512_ssse3 [14290.016775] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14290.020865] acpi_ipmi mfd_core [14290.024957] [14290.027400] wmi [14290.032635] [14290.032636] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14290.035947] ipmi_si ipmi_devintf [14290.040907] ? queued_spin_lock_slowpath+0x18/0x2e0 [14290.044040] ipmi_msghandler [14290.046141] do_raw_spin_lock+0xb5/0xc0 [14290.047975] ioatdma [14290.050075] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14290.055377] acpi_pad acpi_power_meter [14290.058706] try_get_cap_refs+0x159/0x880 [ceph] [14290.063575] ixgbe igb [14290.066459] __ceph_get_caps+0x4a1/0x760 [ceph] [14290.070283] mdio crc32c_intel [14290.072472] ? __file_remove_privs+0x89/0x110 [14290.077253] nvme ptp [14290.080997] ? wake_bit_function+0x70/0x70 [14290.085606] nvme_core i2c_algo_bit [14290.087966] ? ceph_write_iter+0x260/0x770 [ceph] [14290.092488] pps_core [14290.095540] ceph_write_iter+0x260/0x770 [ceph] [14290.099890] dca fuse [14290.102163] ? lock_is_held_type+0xaa/0x120 [14290.106251] [14290.106252] irq event stamp: 37565254 [14290.109739] vfs_write+0x324/0x4f0 [14290.114433] hardirqs last enabled at (37565253): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14290.116724] ksys_pwrite64+0x65/0xa0 [14290.121244] hardirqs last disabled at (37565254): [] sysvec_apic_timer_interrupt+0xe/0x90 [14290.123516] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14290.127709] softirqs last enabled at (37565252): [] __do_softirq+0x2be/0x371 [14290.129202] do_syscall_64+0x38/0x80 [14290.132858] softirqs last disabled at (37565247): [] irq_exit_rcu+0xb0/0xd0 [14290.136257] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14290.146430] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14290.150003] RIP: 0033:0x7f48f0eeb93b [14290.159724] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14290.159726] Workqueue: events delayed_work [ceph] [14290.164856] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14290.173539] [14290.173540] RIP: 0010:lock_acquire+0xf0/0x2b0 [14290.177111] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14290.185622] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14290.190665] ORIG_RAX: 0000000000000012 [14290.200043] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14290.203614] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14290.211086] [14290.211086] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14290.215783] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14290.234519] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14290.236010] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14290.240361] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14290.245579] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14290.264315] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14290.268147] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14290.273364] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14290.280493] [14290.281979] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14290.356359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14290.362095] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14290.369221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14290.376344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14290.383468] Call Trace: [14290.385914] [14290.387924] ? watchdog_timer_fn+0x1b4/0x220 [14290.392189] ? __hrtimer_run_queues+0x1d3/0x360 [14290.396731] ? hrtimer_interrupt+0x104/0x240 [14290.400992] ? __local_bh_enable+0x50/0x60 [14290.405087] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14290.410310] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14290.415269] [14290.417365] [14290.419463] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14290.424771] ? lock_acquire+0xf0/0x2b0 [14290.428519] ? igrab+0x43/0x50 [14290.431569] ? lock_release+0xc5/0x260 [14290.435316] _raw_spin_lock+0x30/0x40 [14290.438980] ? igrab+0x1d/0x50 [14290.442031] igrab+0x1d/0x50 [14290.444907] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14290.450050] delayed_work+0x1d8/0x280 [ceph] [14290.454331] ? process_scheduled_works+0x22a/0x510 [14290.459121] process_scheduled_works+0x2a4/0x510 [14290.463737] worker_thread+0x188/0x340 [14290.467487] ? rescuer_thread+0x310/0x310 [14290.471497] kthread+0xf5/0x130 [14290.474635] ? kthread_complete_and_exit+0x20/0x20 [14290.479436] ret_from_fork+0x31/0x40 [14290.483007] ? kthread_complete_and_exit+0x20/0x20 [14290.487791] ret_from_fork_asm+0x11/0x20 [14290.491730] [14298.485174] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14317.705516] watchdog: BUG: soft lockup - CPU#0 stuck for 2220s! [dbench:78683] [14317.712735] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14317.717516] watchdog: BUG: soft lockup - CPU#7 stuck for 11217s! [kworker/7:4:78808] [14317.768192] irq event stamp: 17817404 [14317.775926] Modules linked in: ceph libceph dns_resolver fscache netfs [14317.779582] hardirqs last enabled at (17817403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14317.786109] veth nft_chain_nat xt_MASQUERADE nf_nat [14317.796267] hardirqs last disabled at (17817404): [] sysvec_apic_timer_interrupt+0xe/0x90 [14317.801224] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [14317.810948] softirqs last enabled at (17817388): [] __do_softirq+0x2be/0x371 [14317.817986] nfnetlink bridge stp llc xfs libcrc32c sunrpc [14317.826669] softirqs last disabled at (17817383): [] irq_exit_rcu+0xb0/0xd0 [14317.826673] intel_rapl_msr intel_rapl_common sb_edac [14317.840666] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14317.845711] x86_pkg_temp_thermal [14317.854654] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14317.854656] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14317.857965] coretemp kvm_intel [14317.865438] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14317.870914] kvm irqbypass [14317.874050] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14317.892788] i2c_i801 iTCO_wdt [14317.895494] [14317.895495] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14317.900710] iTCO_vendor_support i2c_smbus [14317.903762] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14317.905250] crct10dif_pclmul [14317.912377] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14317.916465] joydev crc32_pclmul [14317.923591] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14317.926556] ipmi_ssif [14317.933679] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14317.936912] ghash_clmulni_intel [14317.944037] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14317.946393] mei_me mei [14317.953517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14317.956742] lpc_ich sha512_ssse3 [14317.964817] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14317.967262] acpi_ipmi mfd_core [14317.973002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14317.976310] wmi ipmi_si [14317.983435] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14317.986572] ipmi_devintf ipmi_msghandler [14317.993697] Call Trace: [14317.996227] ioatdma acpi_pad [14318.003355] [14318.007356] acpi_power_meter [14318.009802] ? watchdog_timer_fn+0x1b4/0x220 [14318.012763] ixgbe igb [14318.014776] ? __hrtimer_run_queues+0x1d3/0x360 [14318.017737] mdio crc32c_intel [14318.022007] ? hrtimer_interrupt+0x104/0x240 [14318.024359] nvme [14318.028884] ? __local_bh_enable+0x50/0x60 [14318.031935] ptp nvme_core [14318.036201] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14318.038121] i2c_algo_bit pps_core [14318.042215] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14318.044919] dca [14318.050136] [14318.053532] fuse [14318.058489] [14318.060327] irq event stamp: 37787294 [14318.060329] hardirqs last enabled at (37787293): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14318.062426] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14318.064348] hardirqs last disabled at (37787294): [] sysvec_apic_timer_interrupt+0xe/0x90 [14318.066449] ? queued_spin_lock_slowpath+0x16/0x2e0 [14318.070103] softirqs last enabled at (37787292): [] __do_softirq+0x2be/0x371 [14318.080262] do_raw_spin_lock+0xb5/0xc0 [14318.085565] softirqs last disabled at (37787287): [] irq_exit_rcu+0xb0/0xd0 [14318.095292] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14318.100159] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14318.108848] try_get_cap_refs+0x159/0x880 [ceph] [14318.112675] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14318.121201] __ceph_get_caps+0x4a1/0x760 [ceph] [14318.125978] Workqueue: events delayed_work [ceph] [14318.135361] ? __file_remove_privs+0x89/0x110 [14318.139966] [14318.139967] RIP: 0010:lock_acquire+0xc/0x2b0 [14318.147438] ? wake_bit_function+0x70/0x70 [14318.151961] Code: 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 <41> 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 [14318.156661] ? ceph_write_iter+0x260/0x770 [ceph] [14318.161010] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [14318.162501] ceph_write_iter+0x260/0x770 [ceph] [14318.166765] [14318.170859] ? lock_is_held_type+0xaa/0x120 [14318.189594] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [14318.194292] vfs_write+0x324/0x4f0 [14318.199507] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [14318.204036] ksys_pwrite64+0x65/0xa0 [14318.205523] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [14318.209702] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14318.216823] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [14318.220223] do_syscall_64+0x38/0x80 [14318.227345] R13: ffff88817f867478 R14: 00000001001d239b R15: 0000000000000000 [14318.230919] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14318.238041] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14318.243174] RIP: 0033:0x7f48f0eeb93b [14318.250295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14318.253867] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14318.260990] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14318.266035] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14318.274113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14318.277681] ORIG_RAX: 0000000000000012 [14318.283421] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14318.302158] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14318.309282] Call Trace: [14318.314499] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14318.321623] [14318.325453] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14318.332579] ? watchdog_timer_fn+0x1b4/0x220 [14318.339702] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14318.342148] ? __hrtimer_run_queues+0x1d3/0x360 [14318.349271] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14318.351284] ? hrtimer_interrupt+0x104/0x240 [14318.358411] [14318.362670] ? __local_bh_enable+0x50/0x60 [14318.391983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14318.397198] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14318.402173] [14318.404270] [14318.406369] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14318.411676] ? lock_acquire+0xc/0x2b0 [14318.415338] _raw_spin_lock+0x30/0x40 [14318.418996] ? igrab+0x1d/0x50 [14318.422046] igrab+0x1d/0x50 [14318.424931] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14318.430075] delayed_work+0x1d8/0x280 [ceph] [14318.434355] ? process_scheduled_works+0x22a/0x510 [14318.439145] process_scheduled_works+0x2a4/0x510 [14318.443762] worker_thread+0x188/0x340 [14318.447510] ? rescuer_thread+0x310/0x310 [14318.451514] kthread+0xf5/0x130 [14318.454660] ? kthread_complete_and_exit+0x20/0x20 [14318.459444] ret_from_fork+0x31/0x40 [14318.463025] ? kthread_complete_and_exit+0x20/0x20 [14318.467816] ret_from_fork_asm+0x11/0x20 [14318.471747] [14330.229143] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [14330.237667] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14340.469734] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14345.706005] watchdog: BUG: soft lockup - CPU#0 stuck for 2246s! [dbench:78683] [14345.713223] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14345.718005] watchdog: BUG: soft lockup - CPU#7 stuck for 11243s! [kworker/7:4:78808] [14345.768680] irq event stamp: 17875420 [14345.768681] hardirqs last enabled at (17875419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14345.776411] Modules linked in: [14345.780070] hardirqs last disabled at (17875420): [] sysvec_apic_timer_interrupt+0xe/0x90 [14345.790226] ceph [14345.793276] softirqs last enabled at (17875382): [] __do_softirq+0x2be/0x371 [14345.803003] libceph dns_resolver [14345.804927] softirqs last disabled at (17875377): [] irq_exit_rcu+0xb0/0xd0 [14345.813611] fscache netfs [14345.816920] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14345.825431] veth [14345.828136] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14345.837079] nft_chain_nat [14345.839006] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14345.846476] xt_MASQUERADE [14345.849179] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14345.854656] nf_nat nf_conntrack [14345.857377] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14345.876133] nf_defrag_ipv6 [14345.879355] [14345.879356] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14345.884573] nf_defrag_ipv4 nft_compat [14345.887366] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14345.888856] nf_tables [14345.895980] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14345.899723] nfnetlink bridge [14345.906848] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14345.909205] stp llc [14345.916330] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14345.919292] xfs [14345.926417] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14345.928602] libcrc32c sunrpc [14345.935725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14345.937563] intel_rapl_msr intel_rapl_common [14345.945641] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14345.948604] sb_edac x86_pkg_temp_thermal [14345.954343] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14345.958691] coretemp kvm_intel [14345.965817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14345.969821] kvm irqbypass [14345.976946] Call Trace: [14345.976947] [14345.980081] i2c_i801 [14345.987208] ? watchdog_timer_fn+0x1b4/0x220 [14345.989911] iTCO_wdt iTCO_vendor_support [14345.992357] ? __hrtimer_run_queues+0x1d3/0x360 [14345.994366] i2c_smbus crct10dif_pclmul [14345.996640] ? hrtimer_interrupt+0x104/0x240 [14346.000899] joydev crc32_pclmul [14346.004904] ? __local_bh_enable+0x50/0x60 [14346.009429] ipmi_ssif [14346.013262] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14346.017524] ghash_clmulni_intel mei_me [14346.020747] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14346.024837] mei lpc_ich [14346.027196] [14346.032413] sha512_ssse3 acpi_ipmi [14346.036243] [14346.041201] mfd_core [14346.043731] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14346.045829] wmi ipmi_si [14346.049318] ? queued_spin_lock_slowpath+0x16/0x2e0 [14346.051411] ipmi_devintf ipmi_msghandler [14346.053688] do_raw_spin_lock+0xb5/0xc0 [14346.058993] ioatdma acpi_pad [14346.061526] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14346.066405] acpi_power_meter ixgbe [14346.070411] try_get_cap_refs+0x159/0x880 [ceph] [14346.074239] igb mdio [14346.077208] __ceph_get_caps+0x4a1/0x760 [ceph] [14346.081986] crc32c_intel nvme [14346.085475] ? __file_remove_privs+0x89/0x110 [14346.090082] ptp nvme_core [14346.092353] ? wake_bit_function+0x70/0x70 [14346.096877] i2c_algo_bit [14346.099931] ? ceph_write_iter+0x260/0x770 [ceph] [14346.104279] pps_core dca [14346.106983] ceph_write_iter+0x260/0x770 [ceph] [14346.111073] fuse [14346.113693] ? lock_is_held_type+0xaa/0x120 [14346.118403] irq event stamp: 38010050 [14346.121024] vfs_write+0x324/0x4f0 [14346.125547] hardirqs last enabled at (38010049): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14346.127476] ksys_pwrite64+0x65/0xa0 [14346.131648] hardirqs last disabled at (38010050): [] sysvec_apic_timer_interrupt+0xe/0x90 [14346.135305] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14346.138702] softirqs last enabled at (38010048): [] __do_softirq+0x2be/0x371 [14346.148862] do_syscall_64+0x38/0x80 [14346.152431] softirqs last disabled at (38010043): [] irq_exit_rcu+0xb0/0xd0 [14346.162155] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14346.167287] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14346.175971] RIP: 0033:0x7f48f0eeb93b [14346.179540] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14346.188053] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14346.193096] Workqueue: events delayed_work [ceph] [14346.202472] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14346.206045] [14346.206046] RIP: 0010:lock_acquire+0xf0/0x2b0 [14346.213515] ORIG_RAX: 0000000000000012 [14346.232251] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14346.236951] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14346.242167] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14346.243658] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14346.248009] [14346.248010] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14346.251839] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14346.270576] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14346.277703] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14346.282918] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14346.290043] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14346.291533] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14346.298664] [14346.305781] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14346.350720] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14346.358797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14346.364534] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14346.371659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14346.378781] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14346.385907] Call Trace: [14346.388350] [14346.390363] ? watchdog_timer_fn+0x1b4/0x220 [14346.394627] ? __hrtimer_run_queues+0x1d3/0x360 [14346.399152] ? hrtimer_interrupt+0x104/0x240 [14346.403417] ? __local_bh_enable+0x50/0x60 [14346.407516] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14346.412740] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14346.417708] [14346.419802] [14346.421901] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14346.427209] ? lock_acquire+0xf0/0x2b0 [14346.430957] ? igrab+0x43/0x50 [14346.434008] ? lock_release+0xc5/0x260 [14346.437755] _raw_spin_lock+0x30/0x40 [14346.441418] ? igrab+0x1d/0x50 [14346.444470] igrab+0x1d/0x50 [14346.447347] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14346.452496] delayed_work+0x1d8/0x280 [ceph] [14346.456777] ? process_scheduled_works+0x22a/0x510 [14346.461567] process_scheduled_works+0x2a4/0x510 [14346.466184] worker_thread+0x188/0x340 [14346.469933] ? rescuer_thread+0x310/0x310 [14346.473936] kthread+0xf5/0x130 [14346.477074] ? kthread_complete_and_exit+0x20/0x20 [14346.481857] ret_from_fork+0x31/0x40 [14346.485428] ? kthread_complete_and_exit+0x20/0x20 [14346.490212] ret_from_fork_asm+0x11/0x20 [14346.494137] [14346.783391] rcu: INFO: rcu_preempt self-detected stall on CPU [14346.789149] rcu: 7-....: (2345164 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/2343988 fqs=1154188 [14346.799575] rcu: (t=2410130 jiffies g=1309609 q=150722 ncpus=8) [14346.805573] Sending NMI from CPU 7 to CPUs 0: [14346.809923] NMI backtrace for cpu 0 [14346.809926] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14346.809927] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14346.809928] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14346.809931] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14346.809932] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14346.809933] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14346.809934] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14346.809935] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14346.809935] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14346.809936] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14346.809937] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14346.809938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14346.809939] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14346.809939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14346.809940] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14346.809941] Call Trace: [14346.809941] [14346.809942] ? nmi_cpu_backtrace+0xad/0x120 [14346.809946] ? nmi_cpu_backtrace_handler+0x11/0x20 [14346.809947] ? nmi_handle+0xd4/0x260 [14346.809952] ? default_do_nmi+0x49/0xf0 [14346.809953] ? exc_nmi+0x11e/0x150 [14346.809955] ? end_repeat_nmi+0x16/0x67 [14346.809959] ? queued_spin_lock_slowpath+0x16/0x2e0 [14346.809962] ? queued_spin_lock_slowpath+0x16/0x2e0 [14346.809964] ? queued_spin_lock_slowpath+0x16/0x2e0 [14346.809967] [14346.809967] [14346.809968] do_raw_spin_lock+0xb5/0xc0 [14346.809970] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14346.809984] try_get_cap_refs+0x159/0x880 [ceph] [14346.809998] __ceph_get_caps+0x4a1/0x760 [ceph] [14346.810011] ? __file_remove_privs+0x89/0x110 [14346.810012] ? wake_bit_function+0x70/0x70 [14346.810015] ? ceph_write_iter+0x260/0x770 [ceph] [14346.810025] ceph_write_iter+0x260/0x770 [ceph] [14346.810037] ? lock_is_held_type+0xaa/0x120 [14346.810039] vfs_write+0x324/0x4f0 [14346.810043] ksys_pwrite64+0x65/0xa0 [14346.810046] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14346.810048] do_syscall_64+0x38/0x80 [14346.810050] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14346.810052] RIP: 0033:0x7f48f0eeb93b [14346.810053] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14346.810055] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [14346.810056] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14346.810057] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14346.810057] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14346.810058] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14346.810059] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14346.810062] [14346.810922] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14347.114735] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14347.122207] Workqueue: events delayed_work [ceph] [14347.126921] RIP: 0010:lock_acquire+0xf0/0x2b0 [14347.131272] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14347.150007] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14347.155225] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14347.162350] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14347.169473] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14347.176599] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14347.183724] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14347.190846] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14347.198925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14347.204661] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14347.211787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14347.218909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14347.226034] Call Trace: [14347.228479] [14347.230489] ? rcu_dump_cpu_stacks+0xd1/0x180 [14347.234849] ? rcu_sched_clock_irq+0x633/0x1390 [14347.239373] ? lock_release+0xc5/0x260 [14347.243117] ? lock_is_held_type+0xaa/0x120 [14347.247294] ? lock_is_held_type+0xaa/0x120 [14347.251474] ? tick_sched_do_timer+0xa0/0xa0 [14347.255744] ? rcu_read_lock_sched_held+0x56/0x90 [14347.260444] ? tick_sched_do_timer+0xa0/0xa0 [14347.264714] ? update_process_times+0x5f/0x90 [14347.269066] ? tick_sched_handle.isra.16+0x1e/0x50 [14347.273849] ? tick_sched_timer+0x6f/0x80 [14347.277856] ? __hrtimer_run_queues+0x1d3/0x360 [14347.282382] ? hrtimer_interrupt+0x104/0x240 [14347.286650] ? __local_bh_enable+0x50/0x60 [14347.290745] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14347.295966] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14347.300925] [14347.303021] [14347.305121] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14347.310437] ? lock_acquire+0xf0/0x2b0 [14347.314185] ? igrab+0x43/0x50 [14347.317237] ? lock_release+0xc5/0x260 [14347.320982] _raw_spin_lock+0x30/0x40 [14347.324644] ? igrab+0x1d/0x50 [14347.327696] igrab+0x1d/0x50 [14347.330574] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14347.335724] delayed_work+0x1d8/0x280 [ceph] [14347.340006] ? process_scheduled_works+0x22a/0x510 [14347.344797] process_scheduled_works+0x2a4/0x510 [14347.349413] worker_thread+0x188/0x340 [14347.353161] ? rescuer_thread+0x310/0x310 [14347.357163] kthread+0xf5/0x130 [14347.360300] ? kthread_complete_and_exit+0x20/0x20 [14347.365085] ret_from_fork+0x31/0x40 [14347.368655] ? kthread_complete_and_exit+0x20/0x20 [14347.373441] ret_from_fork_asm+0x11/0x20 [14347.377364] [14351.221921] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14361.462217] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14372.214354] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14373.706494] watchdog: BUG: soft lockup - CPU#0 stuck for 2272s! [dbench:78683] [14373.713709] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14373.718494] watchdog: BUG: soft lockup - CPU#7 stuck for 11269s! [kworker/7:4:78808] [14373.769175] irq event stamp: 17933468 [14373.776908] Modules linked in: ceph libceph dns_resolver fscache netfs [14373.780563] hardirqs last enabled at (17933467): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14373.787082] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [14373.797239] hardirqs last disabled at (17933468): [] sysvec_apic_timer_interrupt+0xe/0x90 [14373.803324] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [14373.813048] softirqs last enabled at (17933432): [] __do_softirq+0x2be/0x371 [14373.819826] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [14373.828508] softirqs last disabled at (17933427): [] irq_exit_rcu+0xb0/0xd0 [14373.828513] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [14373.842931] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14373.842932] kvm irqbypass i2c_i801 [14373.859086] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14373.862569] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul [14373.870042] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14373.878118] ipmi_ssif ghash_clmulni_intel mei_me mei [14373.883596] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14373.888639] lpc_ich sha512_ssse3 [14373.907377] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14373.907378] acpi_ipmi mfd_core [14373.910689] [14373.910689] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14373.915905] wmi [14373.919042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14373.920534] ipmi_si ipmi_devintf [14373.927658] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14373.929494] ipmi_msghandler [14373.936620] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14373.939931] ioatdma acpi_pad [14373.947054] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14373.949931] acpi_power_meter [14373.957055] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14373.960019] ixgbe igb [14373.967144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14373.970108] mdio crc32c_intel [14373.978191] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14373.980552] nvme [14373.986289] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14373.989339] ptp nvme_core [14373.996464] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14373.998388] i2c_algo_bit pps_core [14374.005512] Call Trace: [14374.008215] dca fuse [14374.015342] [14374.018736] [14374.018737] irq event stamp: 38227842 [14374.021185] ? watchdog_timer_fn+0x1b4/0x220 [14374.023453] hardirqs last enabled at (38227841): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14374.025465] ? __hrtimer_run_queues+0x1d3/0x360 [14374.026953] hardirqs last disabled at (38227842): [] sysvec_apic_timer_interrupt+0xe/0x90 [14374.030615] ? hrtimer_interrupt+0x104/0x240 [14374.034891] softirqs last enabled at (38227840): [] __do_softirq+0x2be/0x371 [14374.045049] ? __local_bh_enable+0x50/0x60 [14374.049575] softirqs last disabled at (38227835): [] irq_exit_rcu+0xb0/0xd0 [14374.059301] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14374.063563] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14374.072247] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14374.076337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14374.084847] [14374.090066] Workqueue: events delayed_work [ceph] [14374.099443] [14374.104399] [14374.104400] RIP: 0010:lock_acquire+0xf0/0x2b0 [14374.111890] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14374.113986] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14374.118688] ? queued_spin_lock_slowpath+0x18/0x2e0 [14374.120780] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14374.122275] do_raw_spin_lock+0xb5/0xc0 [14374.126622] [14374.126623] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14374.131929] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14374.150663] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14374.155538] try_get_cap_refs+0x159/0x880 [ceph] [14374.160751] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14374.164588] __ceph_get_caps+0x4a1/0x760 [ceph] [14374.166072] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14374.173202] ? __file_remove_privs+0x89/0x110 [14374.177982] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14374.185107] ? wake_bit_function+0x70/0x70 [14374.189716] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14374.196844] ? ceph_write_iter+0x260/0x770 [ceph] [14374.201365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14374.208489] ceph_write_iter+0x260/0x770 [ceph] [14374.212841] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14374.219968] ? lock_is_held_type+0xaa/0x120 [14374.224055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14374.232136] vfs_write+0x324/0x4f0 [14374.236830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14374.242573] ksys_pwrite64+0x65/0xa0 [14374.247091] Call Trace: [14374.247093] [14374.254216] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14374.258394] ? watchdog_timer_fn+0x1b4/0x220 [14374.265519] do_syscall_64+0x38/0x80 [14374.268916] ? __hrtimer_run_queues+0x1d3/0x360 [14374.276039] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14374.279614] ? hrtimer_interrupt+0x104/0x240 [14374.282054] RIP: 0033:0x7f48f0eeb93b [14374.284065] ? __local_bh_enable+0x50/0x60 [14374.289195] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14374.293462] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14374.297030] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14374.301555] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14374.306597] ORIG_RAX: 0000000000000012 [14374.310862] [14374.314431] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14374.318523] [14374.337261] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14374.342479] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14374.347696] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14374.352657] ? lock_acquire+0xf0/0x2b0 [14374.356485] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14374.358599] ? igrab+0x43/0x50 [14374.365723] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14374.367821] ? lock_release+0xc5/0x260 [14374.374949] [14374.380251] _raw_spin_lock+0x30/0x40 [14374.418018] ? igrab+0x1d/0x50 [14374.421069] igrab+0x1d/0x50 [14374.423950] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14374.429097] delayed_work+0x1d8/0x280 [ceph] [14374.433378] ? process_scheduled_works+0x22a/0x510 [14374.438168] process_scheduled_works+0x2a4/0x510 [14374.442785] worker_thread+0x188/0x340 [14374.446532] ? rescuer_thread+0x310/0x310 [14374.450535] kthread+0xf5/0x130 [14374.453676] ? kthread_complete_and_exit+0x20/0x20 [14374.458459] ret_from_fork+0x31/0x40 [14374.462040] ? kthread_complete_and_exit+0x20/0x20 [14374.466831] ret_from_fork_asm+0x11/0x20 [14374.470763] [14382.454538] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14393.206710] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14401.706983] watchdog: BUG: soft lockup - CPU#0 stuck for 2298s! [dbench:78683] [14401.714194] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14401.718983] watchdog: BUG: soft lockup - CPU#7 stuck for 11295s! [kworker/7:4:78808] [14401.769638] irq event stamp: 17991384 [14401.769639] hardirqs last enabled at (17991383): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14401.777377] Modules linked in: [14401.781033] hardirqs last disabled at (17991384): [] sysvec_apic_timer_interrupt+0xe/0x90 [14401.791191] ceph [14401.794241] softirqs last enabled at (17991356): [] __do_softirq+0x2be/0x371 [14401.803966] libceph [14401.805891] softirqs last disabled at (17991351): [] irq_exit_rcu+0xb0/0xd0 [14401.814573] dns_resolver fscache [14401.816759] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14401.825269] netfs veth [14401.828581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14401.837524] nft_chain_nat xt_MASQUERADE [14401.839969] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14401.847440] nf_nat nf_conntrack [14401.851357] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14401.856850] nf_defrag_ipv6 nf_defrag_ipv4 [14401.860074] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14401.878812] nft_compat nf_tables [14401.882904] [14401.882905] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14401.888121] nfnetlink [14401.891430] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14401.892922] bridge stp [14401.900046] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14401.902402] llc xfs [14401.909529] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14401.911971] libcrc32c sunrpc [14401.919095] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14401.921280] intel_rapl_msr intel_rapl_common [14401.928403] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14401.931381] sb_edac x86_pkg_temp_thermal [14401.938500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14401.942852] coretemp kvm_intel [14401.950930] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14401.954932] kvm irqbypass [14401.960672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14401.963807] i2c_i801 iTCO_wdt [14401.970933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14401.973636] iTCO_vendor_support i2c_smbus [14401.980761] Call Trace: [14401.980763] [14401.983810] crct10dif_pclmul [14401.990935] ? watchdog_timer_fn+0x1b4/0x220 [14401.995026] joydev crc32_pclmul [14401.997470] ? __hrtimer_run_queues+0x1d3/0x360 [14401.999480] ipmi_ssif ghash_clmulni_intel [14402.002449] ? hrtimer_interrupt+0x104/0x240 [14402.006709] mei_me mei [14402.009934] ? __local_bh_enable+0x50/0x60 [14402.014457] lpc_ich sha512_ssse3 [14402.018551] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14402.022813] acpi_ipmi mfd_core [14402.025256] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14402.029346] wmi ipmi_si [14402.032662] [14402.037883] ipmi_devintf ipmi_msghandler [14402.041022] [14402.041023] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14402.045979] ioatdma [14402.048513] ? queued_spin_lock_slowpath+0x16/0x2e0 [14402.050607] acpi_pad acpi_power_meter [14402.054613] do_raw_spin_lock+0xb5/0xc0 [14402.056707] ixgbe igb [14402.062014] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14402.064195] mdio crc32c_intel [14402.069069] try_get_cap_refs+0x159/0x880 [ceph] [14402.072810] nvme ptp [14402.076663] __ceph_get_caps+0x4a1/0x760 [ceph] [14402.079015] nvme_core i2c_algo_bit [14402.083805] ? __file_remove_privs+0x89/0x110 [14402.086850] pps_core dca [14402.091462] ? wake_bit_function+0x70/0x70 [14402.093731] fuse [14402.098260] ? ceph_write_iter+0x260/0x770 [ceph] [14402.101739] irq event stamp: 38451106 [14402.106093] ceph_write_iter+0x260/0x770 [ceph] [14402.108710] hardirqs last enabled at (38451105): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14402.112802] ? lock_is_held_type+0xaa/0x120 [14402.114723] hardirqs last disabled at (38451106): [] sysvec_apic_timer_interrupt+0xe/0x90 [14402.119424] vfs_write+0x324/0x4f0 [14402.123078] softirqs last enabled at (38451104): [] __do_softirq+0x2be/0x371 [14402.127608] ksys_pwrite64+0x65/0xa0 [14402.137760] softirqs last disabled at (38451099): [] irq_exit_rcu+0xb0/0xd0 [14402.141938] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14402.151674] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14402.155069] do_syscall_64+0x38/0x80 [14402.163751] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14402.167322] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14402.175832] Workqueue: events delayed_work [ceph] [14402.180965] RIP: 0033:0x7f48f0eeb93b [14402.190342] [14402.193914] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14402.201385] RIP: 0010:lock_acquire+0xf0/0x2b0 [14402.206427] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14402.211123] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14402.214694] ORIG_RAX: 0000000000000012 [14402.216185] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14402.234924] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14402.239275] [14402.239276] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14402.244492] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14402.263231] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14402.267060] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14402.272277] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14402.279401] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14402.280891] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14402.288017] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14402.295141] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14402.302272] [14402.309389] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14402.355272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14402.361009] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14402.368133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14402.375257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14402.382388] Call Trace: [14402.384835] [14402.386846] ? watchdog_timer_fn+0x1b4/0x220 [14402.391110] ? __hrtimer_run_queues+0x1d3/0x360 [14402.395637] ? hrtimer_interrupt+0x104/0x240 [14402.399905] ? __local_bh_enable+0x50/0x60 [14402.404000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14402.409222] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14402.414179] [14402.416277] [14402.418390] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14402.423702] ? lock_acquire+0xf0/0x2b0 [14402.427450] ? igrab+0x43/0x50 [14402.430499] ? lock_release+0xc5/0x260 [14402.434245] _raw_spin_lock+0x30/0x40 [14402.437900] ? igrab+0x1d/0x50 [14402.440952] igrab+0x1d/0x50 [14402.443830] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14402.448980] delayed_work+0x1d8/0x280 [ceph] [14402.453261] ? process_scheduled_works+0x22a/0x510 [14402.458051] process_scheduled_works+0x2a4/0x510 [14402.462676] worker_thread+0x188/0x340 [14402.466426] ? rescuer_thread+0x310/0x310 [14402.470437] kthread+0xf5/0x130 [14402.473574] ? kthread_complete_and_exit+0x20/0x20 [14402.478359] ret_from_fork+0x31/0x40 [14402.481953] ? kthread_complete_and_exit+0x20/0x20 [14402.486740] ret_from_fork_asm+0x11/0x20 [14402.490673] [14403.446862] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14414.199125] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14429.707473] watchdog: BUG: soft lockup - CPU#0 stuck for 2324s! [dbench:78683] [14429.714690] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14429.719472] watchdog: BUG: soft lockup - CPU#7 stuck for 11321s! [kworker/7:4:78808] [14429.770146] irq event stamp: 18049436 [14429.770147] hardirqs last enabled at (18049435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14429.777879] Modules linked in: [14429.781536] hardirqs last disabled at (18049436): [] sysvec_apic_timer_interrupt+0xe/0x90 [14429.791694] ceph [14429.794745] softirqs last enabled at (18049408): [] __do_softirq+0x2be/0x371 [14429.804470] libceph dns_resolver [14429.806393] softirqs last disabled at (18049403): [] irq_exit_rcu+0xb0/0xd0 [14429.815077] fscache netfs [14429.818389] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14429.826900] veth nft_chain_nat [14429.829602] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14429.838546] xt_MASQUERADE nf_nat [14429.841686] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14429.849165] nf_conntrack nf_defrag_ipv6 [14429.852477] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14429.857953] nf_defrag_ipv4 nft_compat [14429.861869] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14429.880606] nf_tables [14429.884351] [14429.884352] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14429.889569] nfnetlink [14429.891926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14429.893417] bridge stp [14429.900540] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14429.902899] llc [14429.910023] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14429.912467] xfs libcrc32c [14429.919590] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14429.921429] sunrpc intel_rapl_msr [14429.928553] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14429.931257] intel_rapl_common [14429.938380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14429.941777] sb_edac x86_pkg_temp_thermal [14429.949867] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14429.952916] coretemp kvm_intel [14429.958652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14429.962656] kvm irqbypass [14429.969780] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14429.972919] i2c_i801 iTCO_wdt [14429.980041] Call Trace: [14429.980043] [14429.982747] iTCO_vendor_support i2c_smbus [14429.989873] ? watchdog_timer_fn+0x1b4/0x220 [14429.992920] crct10dif_pclmul joydev [14429.995367] ? __hrtimer_run_queues+0x1d3/0x360 [14429.997375] crc32_pclmul ipmi_ssif [14430.001469] ? hrtimer_interrupt+0x104/0x240 [14430.005731] ghash_clmulni_intel [14430.009300] ? __local_bh_enable+0x50/0x60 [14430.013825] mei_me mei [14430.017311] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14430.021574] lpc_ich sha512_ssse3 [14430.024798] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14430.028888] acpi_ipmi [14430.031333] [14430.036548] mfd_core wmi [14430.039872] [14430.044827] ipmi_si [14430.047183] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14430.049281] ipmi_devintf ipmi_msghandler [14430.051904] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14430.053995] ioatdma [14430.056182] do_raw_spin_lock+0xb5/0xc0 [14430.061483] acpi_pad acpi_power_meter [14430.065490] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14430.070358] ixgbe [14430.072546] try_get_cap_refs+0x159/0x880 [ceph] [14430.076375] igb [14430.080124] __ceph_get_caps+0x4a1/0x760 [ceph] [14430.084901] mdio crc32c_intel [14430.086917] ? __file_remove_privs+0x89/0x110 [14430.091524] nvme ptp [14430.093361] ? wake_bit_function+0x70/0x70 [14430.097885] nvme_core [14430.100939] ? ceph_write_iter+0x260/0x770 [ceph] [14430.105287] i2c_algo_bit [14430.107557] ceph_write_iter+0x260/0x770 [ceph] [14430.111648] pps_core dca [14430.114010] ? lock_is_held_type+0xaa/0x120 [14430.118702] fuse [14430.121321] vfs_write+0x324/0x4f0 [14430.125843] irq event stamp: 38673974 [14430.128467] ksys_pwrite64+0x65/0xa0 [14430.132639] hardirqs last enabled at (38673973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14430.134563] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14430.137960] hardirqs last disabled at (38673974): [] sysvec_apic_timer_interrupt+0xe/0x90 [14430.141618] do_syscall_64+0x38/0x80 [14430.145187] softirqs last enabled at (38673972): [] __do_softirq+0x2be/0x371 [14430.155347] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14430.160476] softirqs last disabled at (38673967): [] irq_exit_rcu+0xb0/0xd0 [14430.170202] RIP: 0033:0x7f48f0eeb93b [14430.173771] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14430.182456] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14430.187501] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14430.196012] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14430.199580] Workqueue: events delayed_work [ceph] [14430.208958] ORIG_RAX: 0000000000000012 [14430.208959] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14430.227696] [14430.227697] RIP: 0010:lock_acquire+0xf0/0x2b0 [14430.235168] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14430.240385] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14430.245083] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14430.248913] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14430.256037] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14430.257528] [14430.261879] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14430.269003] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14430.287745] [14430.294875] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14430.332280] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14430.339403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14430.346527] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14430.353651] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14430.361728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14430.367467] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14430.374590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14430.381715] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14430.388838] Call Trace: [14430.391285] [14430.393294] ? watchdog_timer_fn+0x1b4/0x220 [14430.397559] ? __hrtimer_run_queues+0x1d3/0x360 [14430.402085] ? hrtimer_interrupt+0x104/0x240 [14430.406354] ? __local_bh_enable+0x50/0x60 [14430.410449] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14430.415672] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14430.420629] [14430.422726] [14430.424823] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14430.430131] ? lock_acquire+0xf0/0x2b0 [14430.433891] ? igrab+0x43/0x50 [14430.436942] ? lock_release+0xc5/0x260 [14430.440686] _raw_spin_lock+0x30/0x40 [14430.444342] ? igrab+0x1d/0x50 [14430.447393] igrab+0x1d/0x50 [14430.450273] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14430.455419] delayed_work+0x1d8/0x280 [ceph] [14430.459702] ? process_scheduled_works+0x22a/0x510 [14430.464491] process_scheduled_works+0x2a4/0x510 [14430.469109] worker_thread+0x188/0x340 [14430.472874] ? rescuer_thread+0x310/0x310 [14430.476882] kthread+0xf5/0x130 [14430.480025] ? kthread_complete_and_exit+0x20/0x20 [14430.484816] ret_from_fork+0x31/0x40 [14430.488395] ? kthread_complete_and_exit+0x20/0x20 [14430.493180] ret_from_fork_asm+0x11/0x20 [14430.497110] [14445.431160] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [14445.439855] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14456.183896] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14457.707961] watchdog: BUG: soft lockup - CPU#0 stuck for 2350s! [dbench:78683] [14457.715177] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14457.719961] watchdog: BUG: soft lockup - CPU#7 stuck for 11347s! [kworker/7:4:78808] [14457.770634] irq event stamp: 18107528 [14457.770635] hardirqs last enabled at (18107527): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14457.778367] Modules linked in: [14457.782022] hardirqs last disabled at (18107528): [] sysvec_apic_timer_interrupt+0xe/0x90 [14457.792180] ceph [14457.795232] softirqs last enabled at (18107500): [] __do_softirq+0x2be/0x371 [14457.804955] libceph dns_resolver [14457.806879] softirqs last disabled at (18107495): [] irq_exit_rcu+0xb0/0xd0 [14457.815563] fscache [14457.818875] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14457.827387] netfs [14457.829569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14457.838513] veth nft_chain_nat [14457.840525] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14457.847995] xt_MASQUERADE nf_nat [14457.851134] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14457.856610] nf_conntrack nf_defrag_ipv6 [14457.859920] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14457.878658] nf_defrag_ipv4 nft_compat [14457.882575] [14457.882576] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14457.887793] nf_tables nfnetlink [14457.891538] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14457.893028] bridge stp [14457.900153] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14457.903376] llc xfs [14457.910501] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14457.912943] libcrc32c [14457.920069] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14457.922253] sunrpc intel_rapl_msr [14457.929378] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14457.931735] intel_rapl_common sb_edac [14457.938857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14457.942255] x86_pkg_temp_thermal coretemp [14457.950334] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14457.954095] kvm_intel [14457.959833] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14457.963923] kvm irqbypass [14457.971046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14457.973405] i2c_i801 iTCO_wdt [14457.980527] Call Trace: [14457.980528] [14457.983233] iTCO_vendor_support i2c_smbus [14457.990360] ? watchdog_timer_fn+0x1b4/0x220 [14457.993407] crct10dif_pclmul joydev [14457.995853] ? __hrtimer_run_queues+0x1d3/0x360 [14457.997861] crc32_pclmul ipmi_ssif [14458.001957] ? hrtimer_interrupt+0x104/0x240 [14458.006217] ghash_clmulni_intel mei_me [14458.009788] ? __local_bh_enable+0x50/0x60 [14458.014312] mei lpc_ich [14458.017797] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14458.022058] sha512_ssse3 acpi_ipmi [14458.025892] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14458.029982] mfd_core wmi [14458.032512] [14458.037730] ipmi_si ipmi_devintf [14458.041214] [14458.041216] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14458.046170] ipmi_msghandler [14458.048791] ? queued_spin_lock_slowpath+0x18/0x2e0 [14458.050885] ioatdma [14458.054199] do_raw_spin_lock+0xb5/0xc0 [14458.056295] acpi_pad acpi_power_meter [14458.061598] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14458.064475] ixgbe [14458.069364] try_get_cap_refs+0x159/0x880 [ceph] [14458.071546] igb mdio [14458.075382] __ceph_get_caps+0x4a1/0x760 [ceph] [14458.079122] crc32c_intel nvme [14458.083911] ? __file_remove_privs+0x89/0x110 [14458.085917] ptp nvme_core [14458.090528] ? wake_bit_function+0x70/0x70 [14458.092797] i2c_algo_bit pps_core [14458.097324] ? ceph_write_iter+0x260/0x770 [ceph] [14458.100373] dca fuse [14458.104723] ceph_write_iter+0x260/0x770 [ceph] [14458.107429] [14458.107429] irq event stamp: 38896492 [14458.111523] ? lock_is_held_type+0xaa/0x120 [14458.114917] hardirqs last enabled at (38896491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14458.119616] vfs_write+0x324/0x4f0 [14458.121884] hardirqs last disabled at (38896492): [] sysvec_apic_timer_interrupt+0xe/0x90 [14458.126413] ksys_pwrite64+0x65/0xa0 [14458.127899] softirqs last enabled at (38896490): [] __do_softirq+0x2be/0x371 [14458.131556] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14458.135734] softirqs last disabled at (38896485): [] irq_exit_rcu+0xb0/0xd0 [14458.145894] do_syscall_64+0x38/0x80 [14458.149288] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14458.159014] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14458.162585] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14458.171270] RIP: 0033:0x7f48f0eeb93b [14458.176399] Workqueue: events delayed_work [ceph] [14458.184910] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14458.188479] [14458.188480] RIP: 0010:lock_release+0x11/0x260 [14458.197857] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14458.202901] Code: 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd <53> 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f [14458.210371] ORIG_RAX: 0000000000000012 [14458.213942] RSP: 0018:ffffc90002117d68 EFLAGS: 00000286 [14458.218641] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14458.237378] [14458.237379] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [14458.238870] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14458.243221] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [14458.248438] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14458.267176] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [14458.271005] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14458.276222] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [14458.283363] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14458.284855] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [14458.291986] [14458.299102] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14458.359268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14458.365005] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14458.372132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14458.379255] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14458.386378] Call Trace: [14458.388824] [14458.390835] ? watchdog_timer_fn+0x1b4/0x220 [14458.395098] ? __hrtimer_run_queues+0x1d3/0x360 [14458.399625] ? hrtimer_interrupt+0x104/0x240 [14458.403893] ? __local_bh_enable+0x50/0x60 [14458.407988] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14458.413211] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14458.418170] [14458.420266] [14458.422367] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14458.427676] ? igrab+0x43/0x50 [14458.430730] ? igrab+0x43/0x50 [14458.433786] ? lock_release+0x11/0x260 [14458.437534] _raw_spin_unlock+0x1b/0x40 [14458.441372] igrab+0x43/0x50 [14458.444247] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14458.449398] delayed_work+0x1d8/0x280 [ceph] [14458.453678] ? process_scheduled_works+0x22a/0x510 [14458.458461] process_scheduled_works+0x2a4/0x510 [14458.463075] worker_thread+0x188/0x340 [14458.466824] ? rescuer_thread+0x310/0x310 [14458.470828] kthread+0xf5/0x130 [14458.473966] ? kthread_complete_and_exit+0x20/0x20 [14458.478752] ret_from_fork+0x31/0x40 [14458.482328] ? kthread_complete_and_exit+0x20/0x20 [14458.487113] ret_from_fork_asm+0x11/0x20 [14458.491037] [14466.423963] libceph: auth protocol 'cephx' mauth authentication failed: -13 [14485.708451] watchdog: BUG: soft lockup - CPU#0 stuck for 2376s! [dbench:78683] [14485.715664] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14485.720450] watchdog: BUG: soft lockup - CPU#7 stuck for 11373s! [kworker/7:4:78808] [14485.771125] irq event stamp: 18165582 [14485.778860] Modules linked in: ceph libceph dns_resolver fscache netfs veth [14485.782518] hardirqs last enabled at (18165581): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14485.789479] nft_chain_nat xt_MASQUERADE [14485.799635] hardirqs last disabled at (18165582): [] sysvec_apic_timer_interrupt+0xe/0x90 [14485.803552] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [14485.813276] softirqs last enabled at (18165554): [] __do_softirq+0x2be/0x371 [14485.813280] nfnetlink bridge stp llc xfs [14485.829606] softirqs last disabled at (18165549): [] irq_exit_rcu+0xb0/0xd0 [14485.833609] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [14485.842129] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14485.847953] sb_edac x86_pkg_temp_thermal [14485.856898] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14485.856900] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14485.860901] coretemp [14485.868373] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14485.873851] kvm_intel [14485.876125] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14485.894866] kvm irqbypass [14485.897224] [14485.897225] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14485.902441] i2c_i801 [14485.905146] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14485.906637] iTCO_wdt iTCO_vendor_support [14485.913761] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14485.916030] i2c_smbus crct10dif_pclmul [14485.923155] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14485.927159] joydev crc32_pclmul [14485.934285] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14485.938124] ipmi_ssif [14485.945246] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14485.948470] ghash_clmulni_intel [14485.955595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14485.957952] mei_me mei [14485.966031] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14485.969254] lpc_ich sha512_ssse3 [14485.974991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14485.977435] acpi_ipmi mfd_core [14485.984561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14485.987870] wmi [14485.994994] Call Trace: [14485.994996] [14485.998131] ipmi_si ipmi_devintf [14486.005257] ? watchdog_timer_fn+0x1b4/0x220 [14486.007094] ipmi_msghandler [14486.009538] ? __hrtimer_run_queues+0x1d3/0x360 [14486.011549] ioatdma acpi_pad [14486.014861] ? hrtimer_interrupt+0x104/0x240 [14486.019128] acpi_power_meter [14486.022009] ? __local_bh_enable+0x50/0x60 [14486.026532] ixgbe igb [14486.029499] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14486.033761] mdio [14486.036725] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14486.040817] crc32c_intel [14486.043173] [14486.043173] [14486.048390] nvme [14486.050314] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14486.055271] ptp nvme_core [14486.057894] ? queued_spin_lock_slowpath+0x18/0x2e0 [14486.059988] i2c_algo_bit pps_core [14486.062088] do_raw_spin_lock+0xb5/0xc0 [14486.064009] dca fuse [14486.069315] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14486.072016] [14486.076891] try_get_cap_refs+0x159/0x880 [ceph] [14486.080286] irq event stamp: 39118724 [14486.084135] __ceph_get_caps+0x4a1/0x760 [ceph] [14486.086403] hardirqs last enabled at (39118723): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14486.091191] ? __file_remove_privs+0x89/0x110 [14486.092677] hardirqs last disabled at (39118724): [] sysvec_apic_timer_interrupt+0xe/0x90 [14486.097288] ? wake_bit_function+0x70/0x70 [14486.100946] softirqs last enabled at (39118722): [] __do_softirq+0x2be/0x371 [14486.105474] ? ceph_write_iter+0x260/0x770 [ceph] [14486.115627] softirqs last disabled at (39118717): [] irq_exit_rcu+0xb0/0xd0 [14486.119980] ceph_write_iter+0x260/0x770 [ceph] [14486.129703] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14486.133799] ? lock_is_held_type+0xaa/0x120 [14486.142477] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14486.147177] vfs_write+0x324/0x4f0 [14486.155685] Workqueue: events delayed_work [ceph] [14486.160216] ksys_pwrite64+0x65/0xa0 [14486.169587] [14486.173765] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14486.181237] RIP: 0010:lock_acquire+0xf0/0x2b0 [14486.184634] do_syscall_64+0x38/0x80 [14486.189331] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14486.192903] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14486.194393] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14486.199524] RIP: 0033:0x7f48f0eeb93b [14486.203874] [14486.203875] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14486.207445] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14486.226182] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14486.231225] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14486.236443] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14486.240015] ORIG_RAX: 0000000000000012 [14486.241504] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14486.248628] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14486.267367] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14486.274491] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14486.279707] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14486.286850] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14486.290681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14486.297804] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14486.304928] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14486.312052] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14486.319178] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14486.327260] [14486.334379] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14486.377921] Call Trace: [14486.380366] [14486.382377] ? watchdog_timer_fn+0x1b4/0x220 [14486.386640] ? __hrtimer_run_queues+0x1d3/0x360 [14486.391168] ? hrtimer_interrupt+0x104/0x240 [14486.395437] ? __local_bh_enable+0x50/0x60 [14486.399529] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14486.404747] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14486.409701] [14486.411801] [14486.413897] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14486.419205] ? lock_acquire+0xf0/0x2b0 [14486.422955] ? igrab+0x43/0x50 [14486.426005] ? lock_release+0xc5/0x260 [14486.429751] _raw_spin_lock+0x30/0x40 [14486.433417] ? igrab+0x1d/0x50 [14486.436466] igrab+0x1d/0x50 [14486.439346] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14486.444493] delayed_work+0x1d8/0x280 [ceph] [14486.448775] ? process_scheduled_works+0x22a/0x510 [14486.453565] process_scheduled_works+0x2a4/0x510 [14486.458181] worker_thread+0x188/0x340 [14486.461931] ? rescuer_thread+0x310/0x310 [14486.465942] kthread+0xf5/0x130 [14486.469080] ? kthread_complete_and_exit+0x20/0x20 [14486.473865] ret_from_fork+0x31/0x40 [14486.477444] ? kthread_complete_and_exit+0x20/0x20 [14486.482226] ret_from_fork_asm+0x11/0x20 [14486.486150] [14498.168077] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [14513.708939] watchdog: BUG: soft lockup - CPU#0 stuck for 2402s! [dbench:78683] [14513.716150] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14513.720939] watchdog: BUG: soft lockup - CPU#7 stuck for 11399s! [kworker/7:4:78808] [14513.771591] irq event stamp: 18223566 [14513.771592] hardirqs last enabled at (18223565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14513.779340] Modules linked in: [14513.782997] hardirqs last disabled at (18223566): [] sysvec_apic_timer_interrupt+0xe/0x90 [14513.793154] ceph [14513.796204] softirqs last enabled at (18223538): [] __do_softirq+0x2be/0x371 [14513.805930] libceph [14513.807854] softirqs last disabled at (18223533): [] irq_exit_rcu+0xb0/0xd0 [14513.816537] dns_resolver fscache [14513.818722] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14513.827232] netfs veth [14513.830544] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14513.830545] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14513.839487] nft_chain_nat [14513.841932] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14513.849403] xt_MASQUERADE [14513.854878] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14513.857584] nf_nat nf_conntrack [14513.876332] [14513.876333] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14513.879035] nf_defrag_ipv6 nf_defrag_ipv4 [14513.884252] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14513.887474] nft_compat [14513.888966] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14513.896089] nf_tables nfnetlink [14513.900180] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14513.907306] bridge [14513.909750] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14513.916873] stp llc [14513.920097] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14513.927221] xfs libcrc32c [14513.929333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14513.936460] sunrpc intel_rapl_msr [14513.938645] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14513.946722] intel_rapl_common sb_edac [14513.949425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14513.955164] x86_pkg_temp_thermal coretemp [14513.958560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14513.965684] kvm_intel [14513.969428] Call Trace: [14513.969430] [14513.976554] kvm irqbypass [14513.980644] ? watchdog_timer_fn+0x1b4/0x220 [14513.987767] i2c_i801 iTCO_wdt [14513.990126] ? __hrtimer_run_queues+0x1d3/0x360 [14513.992568] iTCO_vendor_support i2c_smbus [14513.994583] ? hrtimer_interrupt+0x104/0x240 [14513.997284] crct10dif_pclmul joydev [14514.001549] ? __local_bh_enable+0x50/0x60 [14514.004614] crc32_pclmul ipmi_ssif [14514.009142] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14514.013232] ghash_clmulni_intel mei_me [14514.017495] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14514.021066] mei lpc_ich [14514.025158] [14514.028642] sha512_ssse3 acpi_ipmi [14514.033857] [14514.033859] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14514.037690] mfd_core wmi [14514.042651] ? queued_spin_lock_slowpath+0x18/0x2e0 [14514.045176] ipmi_si ipmi_devintf [14514.047276] do_raw_spin_lock+0xb5/0xc0 [14514.050759] ipmi_msghandler [14514.052857] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14514.058160] ioatdma acpi_pad [14514.060780] try_get_cap_refs+0x159/0x880 [ceph] [14514.065648] acpi_power_meter ixgbe [14514.068963] __ceph_get_caps+0x4a1/0x760 [ceph] [14514.072789] igb mdio [14514.075672] ? __file_remove_privs+0x89/0x110 [14514.080452] crc32c_intel nvme [14514.083416] ? wake_bit_function+0x70/0x70 [14514.088026] ptp nvme_core [14514.091511] ? ceph_write_iter+0x260/0x770 [ceph] [14514.096033] i2c_algo_bit pps_core [14514.098305] ceph_write_iter+0x260/0x770 [ceph] [14514.102654] dca fuse [14514.105709] ? lock_is_held_type+0xaa/0x120 [14514.109797] [14514.112503] vfs_write+0x324/0x4f0 [14514.117197] irq event stamp: 39341114 [14514.120600] ksys_pwrite64+0x65/0xa0 [14514.125119] hardirqs last enabled at (39341113): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14514.127391] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14514.131567] hardirqs last disabled at (39341114): [] sysvec_apic_timer_interrupt+0xe/0x90 [14514.133060] do_syscall_64+0x38/0x80 [14514.136456] softirqs last enabled at (39341112): [] __do_softirq+0x2be/0x371 [14514.140113] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14514.143685] softirqs last disabled at (39341107): [] irq_exit_rcu+0xb0/0xd0 [14514.153843] RIP: 0033:0x7f48f0eeb93b [14514.158971] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14514.168696] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14514.172268] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14514.180950] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14514.185997] Workqueue: events delayed_work [ceph] [14514.194506] ORIG_RAX: 0000000000000012 [14514.198076] [14514.198077] RIP: 0010:rcu_is_watching+0xb/0x40 [14514.207454] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14514.226192] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [14514.233664] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14514.238880] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [14514.243577] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14514.247408] [14514.247408] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [14514.248899] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14514.253342] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [14514.260468] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14514.279208] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [14514.286348] [14514.291558] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [14514.345100] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [14514.352226] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14514.360302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14514.366041] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14514.373164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14514.380288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14514.387413] Call Trace: [14514.389858] [14514.391869] ? watchdog_timer_fn+0x1b4/0x220 [14514.396133] ? __hrtimer_run_queues+0x1d3/0x360 [14514.400659] ? hrtimer_interrupt+0x104/0x240 [14514.404929] ? __local_bh_enable+0x50/0x60 [14514.409022] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14514.414245] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14514.419204] [14514.421302] [14514.423398] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14514.428703] ? igrab+0x43/0x50 [14514.431755] ? igrab+0x43/0x50 [14514.434802] ? rcu_is_watching+0xb/0x40 [14514.438634] lock_release+0x1d7/0x260 [14514.442294] _raw_spin_unlock+0x1b/0x40 [14514.446131] igrab+0x43/0x50 [14514.449009] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14514.454158] delayed_work+0x1d8/0x280 [ceph] [14514.458441] ? process_scheduled_works+0x22a/0x510 [14514.463231] process_scheduled_works+0x2a4/0x510 [14514.467847] worker_thread+0x188/0x340 [14514.471596] ? rescuer_thread+0x310/0x310 [14514.475608] kthread+0xf5/0x130 [14514.478761] ? kthread_complete_and_exit+0x20/0x20 [14514.483546] ret_from_fork+0x31/0x40 [14514.487124] ? kthread_complete_and_exit+0x20/0x20 [14514.491911] ret_from_fork_asm+0x11/0x20 [14514.495841] [14541.709427] watchdog: BUG: soft lockup - CPU#0 stuck for 2429s! [dbench:78683] [14541.716646] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14541.721427] watchdog: BUG: soft lockup - CPU#7 stuck for 11426s! [kworker/7:4:78808] [14541.772101] irq event stamp: 18281594 [14541.772102] hardirqs last enabled at (18281593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14541.779836] Modules linked in: [14541.783492] hardirqs last disabled at (18281594): [] sysvec_apic_timer_interrupt+0xe/0x90 [14541.793651] ceph [14541.796700] softirqs last enabled at (18281588): [] __do_softirq+0x2be/0x371 [14541.806424] libceph [14541.808350] softirqs last disabled at (18281583): [] irq_exit_rcu+0xb0/0xd0 [14541.817033] dns_resolver fscache [14541.819217] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14541.827729] netfs veth [14541.831038] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14541.839983] nft_chain_nat xt_MASQUERADE [14541.842428] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14541.849897] nf_nat nf_conntrack [14541.853823] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14541.859302] nf_defrag_ipv6 nf_defrag_ipv4 [14541.862526] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14541.881263] nft_compat nf_tables [14541.885353] [14541.885354] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14541.890570] nfnetlink bridge [14541.893882] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14541.895374] stp llc [14541.902497] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14541.905460] xfs libcrc32c [14541.912586] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14541.914768] sunrpc [14541.921892] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14541.924597] intel_rapl_msr intel_rapl_common [14541.931721] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14541.933824] sb_edac x86_pkg_temp_thermal [14541.940950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14541.945301] coretemp kvm_intel [14541.953380] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14541.957384] kvm irqbypass [14541.963121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14541.966259] i2c_i801 iTCO_wdt [14541.973383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14541.976086] iTCO_vendor_support i2c_smbus [14541.983210] Call Trace: [14541.983212] [14541.986261] crct10dif_pclmul [14541.993385] ? watchdog_timer_fn+0x1b4/0x220 [14541.997477] joydev crc32_pclmul [14541.999921] ? __hrtimer_run_queues+0x1d3/0x360 [14542.001930] ipmi_ssif ghash_clmulni_intel [14542.004898] ? hrtimer_interrupt+0x104/0x240 [14542.009159] mei_me mei [14542.012384] ? __local_bh_enable+0x50/0x60 [14542.016907] lpc_ich sha512_ssse3 [14542.021002] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14542.025264] acpi_ipmi mfd_core [14542.027706] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14542.031798] wmi ipmi_si [14542.035108] [14542.040325] ipmi_devintf [14542.043464] [14542.048419] ipmi_msghandler ioatdma [14542.050950] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14542.053047] acpi_pad acpi_power_meter [14542.055670] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14542.057762] ixgbe [14542.061336] do_raw_spin_lock+0xb5/0xc0 [14542.066638] igb [14542.070381] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14542.075254] mdio crc32c_intel [14542.077267] try_get_cap_refs+0x159/0x880 [ceph] [14542.081094] nvme ptp [14542.082935] __ceph_get_caps+0x4a1/0x760 [ceph] [14542.087716] nvme_core [14542.090769] ? __file_remove_privs+0x89/0x110 [14542.095376] i2c_algo_bit [14542.097647] ? wake_bit_function+0x70/0x70 [14542.102172] pps_core [14542.104532] ? ceph_write_iter+0x260/0x770 [ceph] [14542.108880] dca fuse [14542.111498] ceph_write_iter+0x260/0x770 [ceph] [14542.115588] [14542.115589] irq event stamp: 39563928 [14542.117862] ? lock_is_held_type+0xaa/0x120 [14542.122556] hardirqs last enabled at (39563927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14542.124830] vfs_write+0x324/0x4f0 [14542.129350] hardirqs last disabled at (39563928): [] sysvec_apic_timer_interrupt+0xe/0x90 [14542.130846] ksys_pwrite64+0x65/0xa0 [14542.134499] softirqs last enabled at (39563926): [] __do_softirq+0x2be/0x371 [14542.138678] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14542.148835] softirqs last disabled at (39563921): [] irq_exit_rcu+0xb0/0xd0 [14542.152232] do_syscall_64+0x38/0x80 [14542.161957] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14542.165528] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14542.174211] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14542.179343] RIP: 0033:0x7f48f0eeb93b [14542.187853] Workqueue: events delayed_work [ceph] [14542.191423] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14542.191425] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14542.200800] [14542.205846] ORIG_RAX: 0000000000000012 [14542.205847] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14542.213314] RIP: 0010:rcu_is_watching+0x29/0x40 [14542.216886] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14542.216888] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14542.221583] Code: 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e <74> 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [14542.240321] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14542.245538] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [14542.247028] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14542.250859] [14542.257991] [14542.262508] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 0000000000000000 [14542.325758] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [14542.332884] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [14542.340006] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [14542.347130] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14542.354256] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14542.362332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14542.368071] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14542.375195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14542.382319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14542.386805] rcu: INFO: rcu_preempt self-detected stall on CPU [14542.389443] Call Trace: [14542.389445] [14542.395181] rcu: 0-....: (2548459 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=1250102 [14542.397626] ? watchdog_timer_fn+0x1b4/0x220 [14542.399636] rcu: (t=2605724 jiffies g=1309609 q=156655 ncpus=8) [14542.409967] ? __hrtimer_run_queues+0x1d3/0x360 [14542.414231] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14542.420231] ? hrtimer_interrupt+0x104/0x240 [14542.424752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14542.433696] ? __local_bh_enable+0x50/0x60 [14542.437961] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14542.445433] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14542.449520] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14542.454998] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14542.460217] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14542.478953] [14542.483910] [14542.483911] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14542.489129] [14542.491225] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14542.492717] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14542.499839] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14542.501942] ? rcu_is_watching+0x29/0x40 [14542.509063] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14542.514368] lock_acquire+0x184/0x2b0 [14542.521489] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14542.525409] ? igrab+0x43/0x50 [14542.532533] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14542.536191] ? lock_release+0xc5/0x260 [14542.543312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14542.546367] _raw_spin_lock+0x30/0x40 [14542.554440] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14542.558186] ? igrab+0x1d/0x50 [14542.563923] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14542.567580] igrab+0x1d/0x50 [14542.574704] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14542.577758] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14542.584881] Call Trace: [14542.584882] [14542.587761] delayed_work+0x1d8/0x280 [ceph] [14542.594883] ? rcu_dump_cpu_stacks+0xd1/0x180 [14542.600013] ? process_scheduled_works+0x22a/0x510 [14542.602456] ? rcu_sched_clock_irq+0x633/0x1390 [14542.604467] process_scheduled_works+0x2a4/0x510 [14542.608732] ? lock_is_held_type+0xaa/0x120 [14542.613087] worker_thread+0x188/0x340 [14542.617866] ? lock_is_held_type+0xaa/0x120 [14542.622390] ? rescuer_thread+0x310/0x310 [14542.627002] ? tick_sched_do_timer+0xa0/0xa0 [14542.631178] kthread+0xf5/0x130 [14542.634923] ? rcu_read_lock_sched_held+0x56/0x90 [14542.639101] ? kthread_complete_and_exit+0x20/0x20 [14542.643104] ? tick_sched_do_timer+0xa0/0xa0 [14542.647369] ret_from_fork+0x31/0x40 [14542.650506] ? update_process_times+0x5f/0x90 [14542.655203] ? kthread_complete_and_exit+0x20/0x20 [14542.659986] ? tick_sched_handle.isra.16+0x1e/0x50 [14542.664251] ret_from_fork_asm+0x11/0x20 [14542.667821] ? tick_sched_timer+0x6f/0x80 [14542.672179] [14542.676957] ? __hrtimer_run_queues+0x1d3/0x360 [14542.696374] ? hrtimer_interrupt+0x104/0x240 [14542.700642] ? __local_bh_enable+0x50/0x60 [14542.704735] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14542.709961] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14542.714917] [14542.717014] [14542.719113] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14542.724421] ? queued_spin_lock_slowpath+0x16/0x2e0 [14542.729298] do_raw_spin_lock+0xb5/0xc0 [14542.733135] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14542.737939] try_get_cap_refs+0x159/0x880 [ceph] [14542.742570] __ceph_get_caps+0x4a1/0x760 [ceph] [14542.747111] ? __file_remove_privs+0x89/0x110 [14542.751466] ? wake_bit_function+0x70/0x70 [14542.755559] ? ceph_write_iter+0x260/0x770 [ceph] [14542.760272] ceph_write_iter+0x260/0x770 [ceph] [14542.764808] ? lock_is_held_type+0xaa/0x120 [14542.768991] vfs_write+0x324/0x4f0 [14542.772401] ksys_pwrite64+0x65/0xa0 [14542.775975] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14542.781105] do_syscall_64+0x38/0x80 [14542.784678] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14542.789719] RIP: 0033:0x7f48f0eeb93b [14542.793292] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14542.812027] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [14542.819587] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14542.826710] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14542.833834] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14542.840957] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14542.848083] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14542.855212] [14542.857401] Sending NMI from CPU 0 to CPUs 7: [14542.861754] NMI backtrace for cpu 7 [14542.861755] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14542.861757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14542.861758] Workqueue: events delayed_work [ceph] [14542.861774] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [14542.861776] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [14542.861777] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [14542.861778] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [14542.861779] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [14542.861780] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [14542.861781] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [14542.861781] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [14542.861782] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14542.861783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14542.861784] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14542.861785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14542.861785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14542.861786] Call Trace: [14542.861787] [14542.861788] ? nmi_cpu_backtrace+0xad/0x120 [14542.861791] ? nmi_cpu_backtrace_handler+0x11/0x20 [14542.861793] ? nmi_handle+0xd4/0x260 [14542.861797] ? default_do_nmi+0x49/0xf0 [14542.861799] ? exc_nmi+0x11e/0x150 [14542.861800] ? end_repeat_nmi+0x16/0x67 [14542.861804] ? do_raw_spin_lock+0x42/0xc0 [14542.861806] ? do_raw_spin_lock+0x42/0xc0 [14542.861808] ? do_raw_spin_lock+0x42/0xc0 [14542.861810] [14542.861811] [14542.861811] igrab+0x1d/0x50 [14542.861813] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14542.861829] delayed_work+0x1d8/0x280 [ceph] [14542.861842] ? process_scheduled_works+0x22a/0x510 [14542.861845] process_scheduled_works+0x2a4/0x510 [14542.861849] worker_thread+0x188/0x340 [14542.861852] ? rescuer_thread+0x310/0x310 [14542.861854] kthread+0xf5/0x130 [14542.861856] ? kthread_complete_and_exit+0x20/0x20 [14542.861858] ret_from_fork+0x31/0x40 [14542.861861] ? kthread_complete_and_exit+0x20/0x20 [14542.861863] ret_from_fork_asm+0x11/0x20 [14542.861867] [14569.709916] watchdog: BUG: soft lockup - CPU#0 stuck for 2455s! [dbench:78683] [14569.717132] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14569.721916] watchdog: BUG: soft lockup - CPU#7 stuck for 11452s! [kworker/7:4:78808] [14569.772575] irq event stamp: 18338690 [14569.780323] Modules linked in: ceph libceph dns_resolver [14569.783979] hardirqs last enabled at (18338689): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14569.789285] fscache netfs [14569.799458] hardirqs last disabled at (18338690): [] sysvec_apic_timer_interrupt+0xe/0x90 [14569.802162] veth nft_chain_nat xt_MASQUERADE [14569.811886] softirqs last enabled at (18338682): [] __do_softirq+0x2be/0x371 [14569.811887] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [14569.824923] softirqs last disabled at (18338677): [] irq_exit_rcu+0xb0/0xd0 [14569.831701] nf_tables nfnetlink bridge [14569.840210] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14569.844041] stp [14569.852984] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14569.854823] llc xfs libcrc32c sunrpc [14569.862310] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14569.865969] intel_rapl_msr [14569.871444] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14569.871445] intel_rapl_common [14569.874235] sb_edac x86_pkg_temp_thermal [14569.892974] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14569.896025] coretemp kvm_intel [14569.900028] [14569.900029] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14569.905246] kvm [14569.908384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14569.909875] irqbypass i2c_i801 [14569.917000] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14569.918835] iTCO_wdt iTCO_vendor_support [14569.925959] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14569.929096] i2c_smbus crct10dif_pclmul [14569.936221] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14569.940225] joydev crc32_pclmul [14569.947350] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14569.951181] ipmi_ssif ghash_clmulni_intel [14569.958315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14569.961537] mei_me mei [14569.969614] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14569.973706] lpc_ich sha512_ssse3 [14569.979443] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14569.981887] acpi_ipmi [14569.989010] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14569.992323] mfd_core wmi [14569.999447] Call Trace: [14569.999448] [14570.001802] ipmi_si ipmi_devintf [14570.008929] ? watchdog_timer_fn+0x1b4/0x220 [14570.011544] ipmi_msghandler [14570.013989] ? __hrtimer_run_queues+0x1d3/0x360 [14570.016000] ioatdma acpi_pad [14570.019317] ? hrtimer_interrupt+0x104/0x240 [14570.023574] acpi_power_meter ixgbe [14570.026469] ? __local_bh_enable+0x50/0x60 [14570.030994] igb mdio [14570.033960] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14570.038221] crc32c_intel nvme [14570.041707] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14570.045797] ptp [14570.048068] [14570.048069] [14570.053285] nvme_core i2c_algo_bit [14570.056336] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14570.061293] pps_core dca [14570.063150] ? queued_spin_lock_slowpath+0x16/0x2e0 [14570.065244] fuse [14570.067345] do_raw_spin_lock+0xb5/0xc0 [14570.070827] [14570.070827] irq event stamp: 39782774 [14570.076131] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14570.078747] hardirqs last enabled at (39782773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14570.083621] try_get_cap_refs+0x159/0x880 [ceph] [14570.085543] hardirqs last disabled at (39782774): [] sysvec_apic_timer_interrupt+0xe/0x90 [14570.089379] __ceph_get_caps+0x4a1/0x760 [ceph] [14570.090864] softirqs last enabled at (39782772): [] __do_softirq+0x2be/0x371 [14570.094525] ? __file_remove_privs+0x89/0x110 [14570.099315] softirqs last disabled at (39782767): [] irq_exit_rcu+0xb0/0xd0 [14570.109471] ? wake_bit_function+0x70/0x70 [14570.114082] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14570.123810] ? ceph_write_iter+0x260/0x770 [ceph] [14570.128330] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14570.137016] ceph_write_iter+0x260/0x770 [ceph] [14570.141366] Workqueue: events delayed_work [ceph] [14570.149880] ? lock_is_held_type+0xaa/0x120 [14570.153968] [14570.163348] vfs_write+0x324/0x4f0 [14570.168042] RIP: 0010:lock_acquire+0xf0/0x2b0 [14570.175519] ksys_pwrite64+0x65/0xa0 [14570.180036] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14570.184733] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14570.188912] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14570.190403] do_syscall_64+0x38/0x80 [14570.193800] [14570.193800] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14570.198151] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14570.201721] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14570.220458] RIP: 0033:0x7f48f0eeb93b [14570.225597] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14570.230816] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14570.234385] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14570.235877] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14570.243002] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14570.248045] ORIG_RAX: 0000000000000012 [14570.255171] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14570.258741] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14570.265865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14570.284608] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14570.291735] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14570.296953] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14570.304077] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14570.307908] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14570.315985] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14570.323107] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14570.328845] Call Trace: [14570.328846] [14570.335977] [14570.343095] ? watchdog_timer_fn+0x1b4/0x220 [14570.389619] ? __hrtimer_run_queues+0x1d3/0x360 [14570.394145] ? hrtimer_interrupt+0x104/0x240 [14570.398407] ? __local_bh_enable+0x50/0x60 [14570.402501] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14570.407724] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14570.412681] [14570.414777] [14570.416876] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14570.422185] ? lock_acquire+0xf0/0x2b0 [14570.425935] ? igrab+0x43/0x50 [14570.428983] ? lock_release+0xc5/0x260 [14570.432729] _raw_spin_lock+0x30/0x40 [14570.436394] ? igrab+0x1d/0x50 [14570.439444] igrab+0x1d/0x50 [14570.442324] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14570.447471] delayed_work+0x1d8/0x280 [ceph] [14570.451754] ? process_scheduled_works+0x22a/0x510 [14570.456543] process_scheduled_works+0x2a4/0x510 [14570.461159] worker_thread+0x188/0x340 [14570.464907] ? rescuer_thread+0x310/0x310 [14570.468913] kthread+0xf5/0x130 [14570.472047] ? kthread_complete_and_exit+0x20/0x20 [14570.476835] ret_from_fork+0x31/0x40 [14570.480412] ? kthread_complete_and_exit+0x20/0x20 [14570.485196] ret_from_fork_asm+0x11/0x20 [14570.489118] [14597.710405] watchdog: BUG: soft lockup - CPU#0 stuck for 2481s! [dbench:78683] [14597.717622] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14597.722405] watchdog: BUG: soft lockup - CPU#7 stuck for 11478s! [kworker/7:4:78808] [14597.773079] irq event stamp: 18396596 [14597.773080] hardirqs last enabled at (18396595): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14597.780810] Modules linked in: [14597.784468] hardirqs last disabled at (18396596): [] sysvec_apic_timer_interrupt+0xe/0x90 [14597.794624] ceph [14597.797676] softirqs last enabled at (18396560): [] __do_softirq+0x2be/0x371 [14597.807399] libceph [14597.809324] softirqs last disabled at (18396555): [] irq_exit_rcu+0xb0/0xd0 [14597.818009] dns_resolver [14597.820194] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14597.828704] fscache netfs [14597.831321] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14597.840266] veth nft_chain_nat [14597.842969] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14597.850440] xt_MASQUERADE nf_nat [14597.853577] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14597.859055] nf_conntrack [14597.862383] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14597.881121] nf_defrag_ipv6 [14597.883738] [14597.883738] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14597.888953] nf_defrag_ipv4 nft_compat [14597.891746] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14597.893235] nf_tables nfnetlink [14597.900361] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14597.904105] bridge [14597.911227] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14597.914453] stp [14597.921577] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14597.923674] llc xfs [14597.930804] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14597.932644] libcrc32c sunrpc [14597.939769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14597.941951] intel_rapl_msr intel_rapl_common [14597.950029] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14597.952993] sb_edac x86_pkg_temp_thermal [14597.958731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14597.963082] coretemp kvm_intel [14597.970205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14597.974210] kvm irqbypass [14597.981334] Call Trace: [14597.981335] [14597.984472] i2c_i801 iTCO_wdt [14597.991596] ? watchdog_timer_fn+0x1b4/0x220 [14597.994300] iTCO_vendor_support i2c_smbus [14597.996744] ? __hrtimer_run_queues+0x1d3/0x360 [14597.998755] crct10dif_pclmul [14598.001810] ? hrtimer_interrupt+0x104/0x240 [14598.006069] joydev crc32_pclmul [14598.010177] ? __local_bh_enable+0x50/0x60 [14598.014702] ipmi_ssif [14598.017669] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14598.021931] ghash_clmulni_intel mei_me [14598.025154] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14598.029243] mei lpc_ich [14598.031602] [14598.036820] sha512_ssse3 acpi_ipmi [14598.040651] [14598.045607] mfd_core [14598.048138] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14598.050236] wmi ipmi_si [14598.053725] ? queued_spin_lock_slowpath+0x16/0x2e0 [14598.055818] ipmi_devintf [14598.058090] do_raw_spin_lock+0xb5/0xc0 [14598.063391] ipmi_msghandler ioatdma [14598.065924] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14598.070805] acpi_pad [14598.073420] try_get_cap_refs+0x159/0x880 [ceph] [14598.077251] acpi_power_meter ixgbe [14598.080827] __ceph_get_caps+0x4a1/0x760 [ceph] [14598.085604] igb [14598.087879] ? __file_remove_privs+0x89/0x110 [14598.092486] mdio crc32c_intel [14598.095971] ? wake_bit_function+0x70/0x70 [14598.100494] nvme ptp [14598.102336] ? ceph_write_iter+0x260/0x770 [ceph] [14598.106683] nvme_core i2c_algo_bit [14598.109734] ceph_write_iter+0x260/0x770 [ceph] [14598.113824] pps_core [14598.116097] ? lock_is_held_type+0xaa/0x120 [14598.120804] dca fuse [14598.124289] vfs_write+0x324/0x4f0 [14598.128810] [14598.131089] ksys_pwrite64+0x65/0xa0 [14598.135266] irq event stamp: 40005202 [14598.137537] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14598.140935] hardirqs last enabled at (40005201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14598.142426] do_syscall_64+0x38/0x80 [14598.145995] hardirqs last disabled at (40005202): [] sysvec_apic_timer_interrupt+0xe/0x90 [14598.149653] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14598.154801] softirqs last enabled at (40005200): [] __do_softirq+0x2be/0x371 [14598.164958] RIP: 0033:0x7f48f0eeb93b [14598.168529] softirqs last disabled at (40005195): [] irq_exit_rcu+0xb0/0xd0 [14598.178253] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14598.183297] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14598.191983] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14598.195553] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14598.204064] ORIG_RAX: 0000000000000012 [14598.222819] Workqueue: events delayed_work [ceph] [14598.232197] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14598.237413] [14598.237414] RIP: 0010:_raw_spin_lock+0x18/0x40 [14598.244883] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14598.248714] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [14598.253413] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14598.260536] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [14598.262026] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14598.266465] [14598.266466] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [14598.273588] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14598.292327] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [14598.299456] [14598.304668] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [14598.343980] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [14598.351105] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [14598.358229] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14598.366305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14598.372043] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14598.379168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14598.386291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14598.393417] Call Trace: [14598.395859] [14598.397873] ? watchdog_timer_fn+0x1b4/0x220 [14598.402137] ? __hrtimer_run_queues+0x1d3/0x360 [14598.406664] ? hrtimer_interrupt+0x104/0x240 [14598.410931] ? __local_bh_enable+0x50/0x60 [14598.415025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14598.420251] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14598.425205] [14598.427304] [14598.429401] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14598.434710] ? _raw_spin_lock+0x18/0x40 [14598.438546] ? igrab+0x1d/0x50 [14598.441596] igrab+0x1d/0x50 [14598.444475] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14598.449624] delayed_work+0x1d8/0x280 [ceph] [14598.453904] ? process_scheduled_works+0x22a/0x510 [14598.458686] process_scheduled_works+0x2a4/0x510 [14598.463302] worker_thread+0x188/0x340 [14598.467051] ? rescuer_thread+0x310/0x310 [14598.471054] kthread+0xf5/0x130 [14598.474192] ? kthread_complete_and_exit+0x20/0x20 [14598.478976] ret_from_fork+0x31/0x40 [14598.482545] ? kthread_complete_and_exit+0x20/0x20 [14598.487332] ret_from_fork_asm+0x11/0x20 [14598.491261] [14625.710894] watchdog: BUG: soft lockup - CPU#0 stuck for 2507s! [dbench:78683] [14625.718110] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14625.722894] watchdog: BUG: soft lockup - CPU#7 stuck for 11504s! [kworker/7:4:78808] [14625.773570] irq event stamp: 18454584 [14625.781307] Modules linked in: ceph libceph dns_resolver fscache netfs [14625.784964] hardirqs last enabled at (18454583): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14625.784969] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [14625.801647] hardirqs last disabled at (18454584): [] sysvec_apic_timer_interrupt+0xe/0x90 [14625.801650] nf_defrag_ipv4 [14625.801651] softirqs last enabled at (18454478): [] __do_softirq+0x2be/0x371 [14625.809031] nft_compat nf_tables [14625.818755] softirqs last disabled at (18454473): [] irq_exit_rcu+0xb0/0xd0 [14625.821547] nfnetlink bridge [14625.830232] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14625.833540] stp llc [14625.842053] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14625.845017] xfs libcrc32c [14625.853961] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14625.856144] sunrpc intel_rapl_msr [14625.863615] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14625.866319] intel_rapl_common [14625.871796] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14625.875193] sb_edac x86_pkg_temp_thermal [14625.893933] [14625.893933] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14625.896982] coretemp kvm_intel [14625.902200] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14625.906203] kvm irqbypass [14625.907694] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14625.914819] i2c_i801 [14625.917956] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14625.925079] iTCO_wdt iTCO_vendor_support [14625.927784] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14625.934909] i2c_smbus crct10dif_pclmul [14625.937180] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14625.944302] joydev crc32_pclmul [14625.948307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14625.955432] ipmi_ssif ghash_clmulni_intel [14625.959263] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14625.967357] mei_me [14625.970580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14625.976319] mei lpc_ich [14625.980408] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14625.987533] sha512_ssse3 [14625.989631] Call Trace: [14625.996755] acpi_ipmi mfd_core [14625.999294] [14626.006418] wmi ipmi_si [14626.009039] ? watchdog_timer_fn+0x1b4/0x220 [14626.011479] ipmi_devintf ipmi_msghandler [14626.014619] ? __hrtimer_run_queues+0x1d3/0x360 [14626.016628] ioatdma acpi_pad [14626.019161] ? hrtimer_interrupt+0x104/0x240 [14626.023423] acpi_power_meter ixgbe [14626.027426] ? __local_bh_enable+0x50/0x60 [14626.031951] igb mdio [14626.034917] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14626.039180] crc32c_intel nvme [14626.042664] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14626.046753] ptp nvme_core [14626.049024] [14626.054241] i2c_algo_bit [14626.057295] [14626.062251] pps_core dca [14626.064955] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14626.067051] fuse [14626.067052] irq event stamp: 40228400 [14626.069672] ? queued_spin_lock_slowpath+0x18/0x2e0 [14626.071767] hardirqs last enabled at (40228399): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14626.074387] do_raw_spin_lock+0xb5/0xc0 [14626.079689] hardirqs last disabled at (40228400): [] sysvec_apic_timer_interrupt+0xe/0x90 [14626.081613] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14626.085268] softirqs last enabled at (40228398): [] __do_softirq+0x2be/0x371 [14626.090161] try_get_cap_refs+0x159/0x880 [ceph] [14626.100314] softirqs last disabled at (40228393): [] irq_exit_rcu+0xb0/0xd0 [14626.104152] __ceph_get_caps+0x4a1/0x760 [ceph] [14626.113870] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14626.118659] ? __file_remove_privs+0x89/0x110 [14626.127337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14626.131949] ? wake_bit_function+0x70/0x70 [14626.140458] Workqueue: events delayed_work [ceph] [14626.144987] ? ceph_write_iter+0x260/0x770 [ceph] [14626.154361] [14626.154362] RIP: 0010:lock_acquire+0x2d/0x2b0 [14626.158712] ceph_write_iter+0x260/0x770 [ceph] [14626.166182] Code: fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 <48> 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 [14626.170295] ? lock_is_held_type+0xaa/0x120 [14626.174989] RSP: 0018:ffffc90002117d18 EFLAGS: 00000296 [14626.179689] vfs_write+0x324/0x4f0 [14626.181176] [14626.181177] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 0000000000000000 [14626.185532] ksys_pwrite64+0x65/0xa0 [14626.190051] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [14626.208789] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14626.212967] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [14626.218185] do_syscall_64+0x38/0x80 [14626.221582] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [14626.223073] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14626.230197] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14626.233767] RIP: 0033:0x7f48f0eeb93b [14626.240891] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14626.246022] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14626.253145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14626.256716] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14626.263840] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14626.268885] ORIG_RAX: 0000000000000012 [14626.276009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14626.279579] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14626.287657] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14626.306394] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14626.312133] Call Trace: [14626.317349] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14626.324475] [14626.328305] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14626.335430] ? watchdog_timer_fn+0x1b4/0x220 [14626.342552] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14626.349678] ? __hrtimer_run_queues+0x1d3/0x360 [14626.356807] [14626.359249] ? hrtimer_interrupt+0x104/0x240 [14626.397865] ? __local_bh_enable+0x50/0x60 [14626.401958] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14626.407182] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14626.412139] [14626.414235] [14626.416335] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14626.421643] ? lock_acquire+0x2d/0x2b0 [14626.425392] ? igrab+0x43/0x50 [14626.428443] ? lock_release+0xc5/0x260 [14626.432187] _raw_spin_lock+0x30/0x40 [14626.435843] ? igrab+0x1d/0x50 [14626.438894] igrab+0x1d/0x50 [14626.441772] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14626.446922] delayed_work+0x1d8/0x280 [ceph] [14626.451203] ? process_scheduled_works+0x22a/0x510 [14626.455992] process_scheduled_works+0x2a4/0x510 [14626.460610] worker_thread+0x188/0x340 [14626.464358] ? rescuer_thread+0x310/0x310 [14626.468361] kthread+0xf5/0x130 [14626.471499] ? kthread_complete_and_exit+0x20/0x20 [14626.476299] ret_from_fork+0x31/0x40 [14626.479870] ? kthread_complete_and_exit+0x20/0x20 [14626.484655] ret_from_fork_asm+0x11/0x20 [14626.488586] [14653.711382] watchdog: BUG: soft lockup - CPU#0 stuck for 2533s! [dbench:78683] [14653.718597] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14653.723382] watchdog: BUG: soft lockup - CPU#7 stuck for 11530s! [kworker/7:4:78808] [14653.774053] irq event stamp: 18512440 [14653.774054] hardirqs last enabled at (18512439): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14653.781785] Modules linked in: [14653.785442] hardirqs last disabled at (18512440): [] sysvec_apic_timer_interrupt+0xe/0x90 [14653.795601] ceph [14653.798650] softirqs last enabled at (18512432): [] __do_softirq+0x2be/0x371 [14653.808375] libceph dns_resolver [14653.810299] softirqs last disabled at (18512427): [] irq_exit_rcu+0xb0/0xd0 [14653.818983] fscache [14653.822294] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14653.830805] netfs veth [14653.832990] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14653.841934] nft_chain_nat [14653.844378] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14653.851848] xt_MASQUERADE nf_nat [14653.854551] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14653.860029] nf_conntrack nf_defrag_ipv6 [14653.863341] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14653.882078] nf_defrag_ipv4 nft_compat [14653.885994] [14653.885995] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14653.891214] nf_tables [14653.894957] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14653.896448] nfnetlink bridge [14653.903571] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14653.905929] stp llc [14653.913053] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14653.916035] xfs libcrc32c [14653.923176] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14653.925361] sunrpc intel_rapl_msr [14653.932485] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14653.935189] intel_rapl_common [14653.942313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14653.945709] sb_edac x86_pkg_temp_thermal [14653.953786] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14653.956838] coretemp kvm_intel [14653.962576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14653.966579] kvm irqbypass [14653.973703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14653.976840] i2c_i801 iTCO_wdt [14653.983965] Call Trace: [14653.983967] [14653.986668] iTCO_vendor_support i2c_smbus [14653.993795] ? watchdog_timer_fn+0x1b4/0x220 [14653.996845] crct10dif_pclmul joydev [14653.999290] ? __hrtimer_run_queues+0x1d3/0x360 [14654.001298] crc32_pclmul ipmi_ssif [14654.005392] ? hrtimer_interrupt+0x104/0x240 [14654.009655] ghash_clmulni_intel mei_me [14654.013225] ? __local_bh_enable+0x50/0x60 [14654.017749] mei lpc_ich [14654.021234] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14654.025495] sha512_ssse3 acpi_ipmi [14654.029327] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14654.033419] mfd_core wmi [14654.035950] [14654.041166] ipmi_si ipmi_devintf [14654.044651] [14654.044652] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14654.049608] ipmi_msghandler ioatdma [14654.052229] ? queued_spin_lock_slowpath+0x18/0x2e0 [14654.054323] acpi_pad acpi_power_meter [14654.057634] do_raw_spin_lock+0xb5/0xc0 [14654.059731] ixgbe igb [14654.065036] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14654.068606] mdio crc32c_intel nvme [14654.073479] try_get_cap_refs+0x159/0x880 [ceph] [14654.077219] ptp nvme_core [14654.081070] __ceph_get_caps+0x4a1/0x760 [ceph] [14654.083427] i2c_algo_bit pps_core [14654.088214] ? __file_remove_privs+0x89/0x110 [14654.091693] dca fuse [14654.096305] ? wake_bit_function+0x70/0x70 [14654.099007] [14654.099008] irq event stamp: 40451670 [14654.103536] ? ceph_write_iter+0x260/0x770 [ceph] [14654.106930] hardirqs last enabled at (40451669): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14654.111281] ceph_write_iter+0x260/0x770 [ceph] [14654.113551] hardirqs last disabled at (40451670): [] sysvec_apic_timer_interrupt+0xe/0x90 [14654.117645] ? lock_is_held_type+0xaa/0x120 [14654.119133] softirqs last enabled at (40451668): [] __do_softirq+0x2be/0x371 [14654.122792] vfs_write+0x324/0x4f0 [14654.127487] softirqs last disabled at (40451663): [] irq_exit_rcu+0xb0/0xd0 [14654.137651] ksys_pwrite64+0x65/0xa0 [14654.142169] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14654.151894] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14654.156070] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14654.164758] do_syscall_64+0x38/0x80 [14654.168170] Workqueue: events delayed_work [ceph] [14654.176681] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14654.180253] [14654.180253] RIP: 0010:lock_acquire+0xf0/0x2b0 [14654.189631] RIP: 0033:0x7f48f0eeb93b [14654.194760] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14654.202247] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14654.205818] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14654.210516] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14654.215561] [14654.215561] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14654.217051] ORIG_RAX: 0000000000000012 [14654.221419] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14654.224990] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14654.243727] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14654.262465] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14654.267683] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14654.272899] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14654.274391] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14654.281515] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14654.285346] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14654.292470] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14654.299594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14654.306722] [14654.313842] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14654.372587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14654.379710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14654.386834] Call Trace: [14654.389280] [14654.391290] ? watchdog_timer_fn+0x1b4/0x220 [14654.395554] ? __hrtimer_run_queues+0x1d3/0x360 [14654.400081] ? hrtimer_interrupt+0x104/0x240 [14654.404350] ? __local_bh_enable+0x50/0x60 [14654.408443] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14654.413667] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14654.418625] [14654.420721] [14654.422819] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14654.428127] ? lock_acquire+0xf0/0x2b0 [14654.431877] ? igrab+0x43/0x50 [14654.434928] ? lock_release+0xc5/0x260 [14654.438672] _raw_spin_lock+0x30/0x40 [14654.442328] ? igrab+0x1d/0x50 [14654.445380] igrab+0x1d/0x50 [14654.448258] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14654.453406] delayed_work+0x1d8/0x280 [ceph] [14654.457688] ? process_scheduled_works+0x22a/0x510 [14654.462479] process_scheduled_works+0x2a4/0x510 [14654.467094] worker_thread+0x188/0x340 [14654.470844] ? rescuer_thread+0x310/0x310 [14654.474854] kthread+0xf5/0x130 [14654.477995] ? kthread_complete_and_exit+0x20/0x20 [14654.482789] ret_from_fork+0x31/0x40 [14654.486365] ? kthread_complete_and_exit+0x20/0x20 [14654.491149] ret_from_fork_asm+0x11/0x20 [14654.495075] [14681.711872] watchdog: BUG: soft lockup - CPU#0 stuck for 2559s! [dbench:78683] [14681.719092] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14681.723871] watchdog: BUG: soft lockup - CPU#7 stuck for 11556s! [kworker/7:4:78808] [14681.774546] irq event stamp: 18570438 [14681.782282] Modules linked in: ceph libceph dns_resolver [14681.785938] hardirqs last enabled at (18570437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14681.791243] fscache netfs veth nft_chain_nat xt_MASQUERADE [14681.801418] hardirqs last disabled at (18570438): [] sysvec_apic_timer_interrupt+0xe/0x90 [14681.806981] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [14681.816707] softirqs last enabled at (18570342): [] __do_softirq+0x2be/0x371 [14681.824359] nfnetlink bridge stp llc xfs libcrc32c [14681.833044] softirqs last disabled at (18570337): [] irq_exit_rcu+0xb0/0xd0 [14681.833047] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [14681.846435] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14681.853904] coretemp kvm_intel [14681.862848] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14681.862850] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14681.865985] kvm irqbypass [14681.873458] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14681.878933] i2c_i801 iTCO_wdt [14681.881637] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14681.900375] iTCO_vendor_support i2c_smbus [14681.903427] [14681.903428] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14681.908644] crct10dif_pclmul joydev [14681.912734] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14681.914225] crc32_pclmul ipmi_ssif [14681.921350] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14681.924919] ghash_clmulni_intel [14681.932045] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14681.935528] mei_me mei lpc_ich [14681.942669] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14681.945894] sha512_ssse3 acpi_ipmi [14681.953017] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14681.956155] mfd_core wmi [14681.963280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14681.966763] ipmi_si ipmi_devintf [14681.974841] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14681.977459] ipmi_msghandler ioatdma [14681.983196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14681.986508] acpi_pad acpi_power_meter [14681.993631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14681.997202] ixgbe igb [14682.004326] Call Trace: [14682.008069] mdio crc32c_intel [14682.015197] [14682.017551] nvme ptp [14682.019997] ? watchdog_timer_fn+0x1b4/0x220 [14682.023046] nvme_core i2c_algo_bit [14682.025058] ? __hrtimer_run_queues+0x1d3/0x360 [14682.027328] pps_core dca [14682.031594] ? hrtimer_interrupt+0x104/0x240 [14682.035076] fuse [14682.039600] ? __local_bh_enable+0x50/0x60 [14682.042218] irq event stamp: 40674212 [14682.046483] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14682.048405] hardirqs last enabled at (40674211): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14682.052497] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14682.056154] hardirqs last disabled at (40674212): [] sysvec_apic_timer_interrupt+0xe/0x90 [14682.061371] [14682.071528] softirqs last enabled at (40674210): [] __do_softirq+0x2be/0x371 [14682.076502] [14682.076504] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14682.086226] softirqs last disabled at (40674205): [] irq_exit_rcu+0xb0/0xd0 [14682.088329] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14682.097008] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14682.099107] do_raw_spin_lock+0xb5/0xc0 [14682.104410] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14682.112923] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14682.117791] Workqueue: events delayed_work [ceph] [14682.127173] try_get_cap_refs+0x159/0x880 [ceph] [14682.130999] [14682.131000] RIP: 0010:lock_acquire+0xf0/0x2b0 [14682.138477] __ceph_get_caps+0x4a1/0x760 [ceph] [14682.143271] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14682.147974] ? __file_remove_privs+0x89/0x110 [14682.152580] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14682.154072] ? wake_bit_function+0x70/0x70 [14682.158422] [14682.158423] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14682.162948] ? ceph_write_iter+0x260/0x770 [ceph] [14682.181684] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14682.186034] ceph_write_iter+0x260/0x770 [ceph] [14682.191251] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14682.195363] ? lock_is_held_type+0xaa/0x120 [14682.196850] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14682.203978] vfs_write+0x324/0x4f0 [14682.208673] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14682.215800] ksys_pwrite64+0x65/0xa0 [14682.220320] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14682.227444] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14682.231623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14682.238747] do_syscall_64+0x38/0x80 [14682.242142] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14682.249274] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14682.252847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14682.260927] RIP: 0033:0x7f48f0eeb93b [14682.266056] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14682.271794] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14682.275363] Call Trace: [14682.282487] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14682.287532] [14682.294672] ORIG_RAX: 0000000000000012 [14682.298245] ? watchdog_timer_fn+0x1b4/0x220 [14682.305367] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14682.324107] ? __hrtimer_run_queues+0x1d3/0x360 [14682.326565] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14682.331788] ? hrtimer_interrupt+0x104/0x240 [14682.333794] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14682.337626] ? __local_bh_enable+0x50/0x60 [14682.341891] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14682.349017] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14682.353538] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14682.360663] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14682.364932] [14682.372051] [14682.404856] [14682.406953] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14682.412279] ? lock_acquire+0xf0/0x2b0 [14682.416029] ? igrab+0x43/0x50 [14682.419087] ? lock_release+0xc5/0x260 [14682.422832] _raw_spin_lock+0x30/0x40 [14682.426498] ? igrab+0x1d/0x50 [14682.429555] igrab+0x1d/0x50 [14682.432434] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14682.437583] delayed_work+0x1d8/0x280 [ceph] [14682.441866] ? process_scheduled_works+0x22a/0x510 [14682.446654] process_scheduled_works+0x2a4/0x510 [14682.451284] worker_thread+0x188/0x340 [14682.455029] ? rescuer_thread+0x310/0x310 [14682.459040] kthread+0xf5/0x130 [14682.462178] ? kthread_complete_and_exit+0x20/0x20 [14682.466961] ret_from_fork+0x31/0x40 [14682.470558] ? kthread_complete_and_exit+0x20/0x20 [14682.475344] ret_from_fork_asm+0x11/0x20 [14682.479282] [14709.712360] watchdog: BUG: soft lockup - CPU#0 stuck for 2585s! [dbench:78683] [14709.719570] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14709.724360] watchdog: BUG: soft lockup - CPU#7 stuck for 11582s! [kworker/7:4:78808] [14709.775013] irq event stamp: 18628310 [14709.775014] hardirqs last enabled at (18628309): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14709.782760] Modules linked in: [14709.786417] hardirqs last disabled at (18628310): [] sysvec_apic_timer_interrupt+0xe/0x90 [14709.796574] ceph [14709.799626] softirqs last enabled at (18628232): [] __do_softirq+0x2be/0x371 [14709.809349] libceph [14709.811273] softirqs last disabled at (18628227): [] irq_exit_rcu+0xb0/0xd0 [14709.819958] dns_resolver [14709.822143] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14709.830654] fscache netfs [14709.833271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14709.842214] veth nft_chain_nat [14709.844917] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14709.852391] xt_MASQUERADE nf_nat [14709.855527] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14709.861005] nf_conntrack nf_defrag_ipv6 [14709.864316] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14709.883053] nf_defrag_ipv4 nft_compat [14709.886969] [14709.886970] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14709.892188] nf_tables [14709.895932] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14709.897423] nfnetlink bridge [14709.904547] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14709.906904] stp llc [14709.914028] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14709.916993] xfs libcrc32c [14709.924116] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14709.926300] sunrpc [14709.933424] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14709.936129] intel_rapl_msr [14709.943253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14709.945349] intel_rapl_common sb_edac [14709.953428] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14709.956219] x86_pkg_temp_thermal coretemp [14709.961956] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14709.965699] kvm_intel kvm [14709.972825] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14709.976914] irqbypass i2c_i801 [14709.984049] Call Trace: [14709.984051] [14709.986756] iTCO_wdt iTCO_vendor_support [14709.993885] ? watchdog_timer_fn+0x1b4/0x220 [14709.997038] i2c_smbus crct10dif_pclmul [14709.999484] ? __hrtimer_run_queues+0x1d3/0x360 [14710.001493] joydev crc32_pclmul [14710.005500] ? hrtimer_interrupt+0x104/0x240 [14710.009762] ipmi_ssif [14710.013593] ? __local_bh_enable+0x50/0x60 [14710.018116] ghash_clmulni_intel mei_me [14710.021342] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14710.025604] mei lpc_ich [14710.027962] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14710.032053] sha512_ssse3 acpi_ipmi [14710.035883] [14710.041102] mfd_core wmi [14710.043631] [14710.043632] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14710.048590] ipmi_si ipmi_devintf [14710.052077] ? queued_spin_lock_slowpath+0x16/0x2e0 [14710.054170] ipmi_msghandler [14710.056791] do_raw_spin_lock+0xb5/0xc0 [14710.058886] ioatdma acpi_pad [14710.064190] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14710.067500] acpi_power_meter ixgbe igb [14710.072374] try_get_cap_refs+0x159/0x880 [ceph] [14710.075247] mdio crc32c_intel [14710.079084] __ceph_get_caps+0x4a1/0x760 [ceph] [14710.082043] nvme ptp [14710.086831] ? __file_remove_privs+0x89/0x110 [14710.090658] nvme_core i2c_algo_bit [14710.095268] ? wake_bit_function+0x70/0x70 [14710.098319] pps_core dca [14710.102846] ? ceph_write_iter+0x260/0x770 [ceph] [14710.105114] fuse [14710.109465] ceph_write_iter+0x260/0x770 [ceph] [14710.112950] irq event stamp: 40896694 [14710.117044] ? lock_is_held_type+0xaa/0x120 [14710.119656] hardirqs last enabled at (40896693): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14710.124357] vfs_write+0x324/0x4f0 [14710.126280] hardirqs last disabled at (40896694): [] sysvec_apic_timer_interrupt+0xe/0x90 [14710.130808] ksys_pwrite64+0x65/0xa0 [14710.134461] softirqs last enabled at (40896692): [] __do_softirq+0x2be/0x371 [14710.138638] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14710.148796] softirqs last disabled at (40896687): [] irq_exit_rcu+0xb0/0xd0 [14710.152193] do_syscall_64+0x38/0x80 [14710.161916] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14710.165488] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14710.174171] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14710.174172] Workqueue: events delayed_work [ceph] [14710.179303] RIP: 0033:0x7f48f0eeb93b [14710.187813] [14710.187814] RIP: 0010:lock_acquire+0xf0/0x2b0 [14710.191384] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14710.200762] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14710.205804] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14710.213277] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14710.217975] ORIG_RAX: 0000000000000012 [14710.221545] [14710.221545] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14710.223043] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14710.227393] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14710.246132] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14710.264869] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14710.270087] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14710.275304] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14710.279136] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14710.280626] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14710.287761] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14710.294883] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14710.302012] [14710.309132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14710.367883] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14710.375007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14710.382133] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14710.389255] Call Trace: [14710.391702] [14710.393713] ? watchdog_timer_fn+0x1b4/0x220 [14710.397977] ? __hrtimer_run_queues+0x1d3/0x360 [14710.402503] ? hrtimer_interrupt+0x104/0x240 [14710.406771] ? __local_bh_enable+0x50/0x60 [14710.410865] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14710.416081] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14710.421047] [14710.423144] [14710.425241] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14710.430550] ? lock_acquire+0xf0/0x2b0 [14710.434301] ? igrab+0x43/0x50 [14710.437350] ? lock_release+0xc5/0x260 [14710.441095] _raw_spin_lock+0x30/0x40 [14710.444766] ? igrab+0x1d/0x50 [14710.447819] igrab+0x1d/0x50 [14710.450698] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14710.455848] delayed_work+0x1d8/0x280 [ceph] [14710.460128] ? process_scheduled_works+0x22a/0x510 [14710.464919] process_scheduled_works+0x2a4/0x510 [14710.469533] worker_thread+0x188/0x340 [14710.473282] ? rescuer_thread+0x310/0x310 [14710.477285] kthread+0xf5/0x130 [14710.480423] ? kthread_complete_and_exit+0x20/0x20 [14710.485208] ret_from_fork+0x31/0x40 [14710.488788] ? kthread_complete_and_exit+0x20/0x20 [14710.493570] ret_from_fork_asm+0x11/0x20 [14710.497494] [14737.712849] watchdog: BUG: soft lockup - CPU#0 stuck for 2611s! [dbench:78683] [14737.720067] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14737.724848] watchdog: BUG: soft lockup - CPU#7 stuck for 11608s! [kworker/7:4:78808] [14737.775525] irq event stamp: 18686418 [14737.783257] Modules linked in: ceph libceph dns_resolver fscache netfs [14737.786912] hardirqs last enabled at (18686417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14737.786917] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [14737.803598] hardirqs last disabled at (18686418): [] sysvec_apic_timer_interrupt+0xe/0x90 [14737.803601] nf_defrag_ipv4 [14737.803602] softirqs last enabled at (18686340): [] __do_softirq+0x2be/0x371 [14737.810989] nft_compat nf_tables [14737.820715] softirqs last disabled at (18686335): [] irq_exit_rcu+0xb0/0xd0 [14737.823520] nfnetlink [14737.832206] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14737.835533] bridge stp [14737.844046] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14737.846402] llc xfs [14737.855348] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14737.857789] libcrc32c sunrpc [14737.865263] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14737.867447] intel_rapl_msr intel_rapl_common [14737.872922] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14737.874218] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [14737.874221] rcu: 0-....: (2739537 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=1346109 [14737.874243] rcu: 7-....: (2725094 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/2723912 fqs=1346109 [14737.874246] rcu: (detected by 3, t=2801199 jiffies, g=1309609, q=161508 ncpus=8) [14737.874248] Sending NMI from CPU 3 to CPUs 0: [14737.874251] NMI backtrace for cpu 0 [14737.874252] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14737.874254] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14737.874254] RIP: 0010:vprintk_emit+0x1ee/0x340 [14737.874257] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [14737.874258] RSP: 0018:ffffc90000003db8 EFLAGS: 00000002 [14737.874259] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [14737.874260] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [14737.874261] RBP: 000000000000002b R08: 0000000000005ef4 R09: 0000000000000004 [14737.874262] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [14737.874262] R13: 0000000000000000 R14: ffffffff82188e50 R15: ffffc90000003e08 [14737.874263] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14737.874264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14737.874265] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14737.874266] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14737.874267] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14737.874267] Call Trace: [14737.874269] [14737.874270] ? nmi_cpu_backtrace+0xad/0x120 [14737.874274] ? nmi_cpu_backtrace_handler+0x11/0x20 [14737.874275] ? nmi_handle+0xd4/0x260 [14737.874280] ? default_do_nmi+0x49/0xf0 [14737.874281] ? exc_nmi+0x11e/0x150 [14737.874283] ? end_repeat_nmi+0x16/0x67 [14737.874287] ? vprintk_emit+0x1ee/0x340 [14737.874289] ? vprintk_emit+0x1ee/0x340 [14737.874291] ? vprintk_emit+0x1ee/0x340 [14737.874293] [14737.874293] [14737.874294] ? touch_softlockup_watchdog+0x20/0x20 [14737.874296] _printk+0x5c/0x80 [14737.874299] __show_regs+0x28/0x2e0 [14737.874302] ? touch_softlockup_watchdog+0x20/0x20 [14737.874303] show_regs+0x38/0x60 [14737.874305] watchdog_timer_fn+0x1b4/0x220 [14737.874307] __hrtimer_run_queues+0x1d3/0x360 [14737.874310] hrtimer_interrupt+0x104/0x240 [14737.874312] ? __local_bh_enable+0x50/0x60 [14737.874315] __sysvec_apic_timer_interrupt+0x61/0x160 [14737.874318] sysvec_apic_timer_interrupt+0x6d/0x90 [14737.874320] [14737.874320] [14737.874321] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14737.874323] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14737.874325] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14737.874327] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14737.874327] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14737.874328] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14737.874329] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14737.874329] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14737.874330] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14737.874334] do_raw_spin_lock+0xb5/0xc0 [14737.874336] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14737.874356] try_get_cap_refs+0x159/0x880 [ceph] [14737.874371] __ceph_get_caps+0x4a1/0x760 [ceph] [14737.874384] ? __file_remove_privs+0x89/0x110 [14737.874386] ? wake_bit_function+0x70/0x70 [14737.874389] ? ceph_write_iter+0x260/0x770 [ceph] [14737.874400] ceph_write_iter+0x260/0x770 [ceph] [14737.874411] ? lock_is_held_type+0xaa/0x120 [14737.874414] vfs_write+0x324/0x4f0 [14737.874418] ksys_pwrite64+0x65/0xa0 [14737.874421] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14737.874423] do_syscall_64+0x38/0x80 [14737.874426] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14737.874428] RIP: 0033:0x7f48f0eeb93b [14737.874429] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14737.874430] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [14737.874431] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14737.874432] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14737.874433] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14737.874433] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14737.874434] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14737.874437] [14737.875250] Sending NMI from CPU 3 to CPUs 7: [14737.875254] NMI backtrace for cpu 7 [14737.875255] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14737.875257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14737.875258] Workqueue: events delayed_work [ceph] [14737.875280] RIP: 0010:io_serial_in+0x19/0x20 [14737.875283] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [14737.875285] RSP: 0018:ffffc90000264c98 EFLAGS: 00000002 [14737.875287] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [14737.875288] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [14737.875289] RBP: 000000000000257b R08: 00003083aadb73c4 R09: 0000000000000007 [14737.875291] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [14737.875292] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [14737.875293] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14737.875295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14737.875296] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14737.875298] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14737.875299] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14737.875300] Call Trace: [14737.875301] [14737.875302] ? nmi_cpu_backtrace+0xad/0x120 [14737.875307] ? nmi_cpu_backtrace_handler+0x11/0x20 [14737.875309] ? nmi_handle+0xd4/0x260 [14737.875314] ? default_do_nmi+0x49/0xf0 [14737.875316] ? exc_nmi+0x11e/0x150 [14737.875318] ? end_repeat_nmi+0x16/0x67 [14737.875325] ? io_serial_in+0x19/0x20 [14737.875327] ? io_serial_in+0x19/0x20 [14737.875331] ? io_serial_in+0x19/0x20 [14737.875333] [14737.875334] [14737.875334] wait_for_lsr+0x45/0x70 [14737.875337] serial8250_console_write+0x341/0x3f0 [14737.875341] ? lock_release+0xc5/0x260 [14737.875346] console_flush_all+0x227/0x5a0 [14737.875348] ? console_flush_all+0x1ff/0x5a0 [14737.875350] ? console_flush_all+0x1ff/0x5a0 [14737.875354] console_unlock+0x7b/0x120 [14737.875358] vprintk_emit+0x246/0x340 [14737.875360] ? irq_work_queue+0x2c/0x50 [14737.875365] ? touch_softlockup_watchdog+0x20/0x20 [14737.875367] _printk+0x5c/0x80 [14737.875372] print_modules+0x6e/0xe0 [14737.875375] watchdog_timer_fn+0x19f/0x220 [14737.875377] __hrtimer_run_queues+0x1d3/0x360 [14737.875382] hrtimer_interrupt+0x104/0x240 [14737.875384] ? __local_bh_enable+0x50/0x60 [14737.875389] __sysvec_apic_timer_interrupt+0x61/0x160 [14737.875393] sysvec_apic_timer_interrupt+0x6d/0x90 [14737.875395] [14737.875396] [14737.875397] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14737.875399] RIP: 0010:lock_acquire+0x104/0x2b0 [14737.875401] Code: ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 <48> 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 b2 e6 eb 7e [14737.875403] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [14737.875405] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 000000005553a634 [14737.875406] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14737.875407] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14737.875408] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14737.875409] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14737.875415] ? igrab+0x43/0x50 [14737.875418] ? lock_release+0xc5/0x260 [14737.875421] _raw_spin_lock+0x30/0x40 [14737.875424] ? igrab+0x1d/0x50 [14737.875425] igrab+0x1d/0x50 [14737.875428] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14737.875448] delayed_work+0x1d8/0x280 [ceph] [14737.875467] ? process_scheduled_works+0x22a/0x510 [14737.875471] process_scheduled_works+0x2a4/0x510 [14737.875477] worker_thread+0x188/0x340 [14737.875481] ? rescuer_thread+0x310/0x310 [14737.875484] kthread+0xf5/0x130 [14737.875487] ? kthread_complete_and_exit+0x20/0x20 [14737.875490] ret_from_fork+0x31/0x40 [14737.875493] ? kthread_complete_and_exit+0x20/0x20 [14737.875496] ret_from_fork_asm+0x11/0x20 [14737.875503] [14737.875887] sb_edac [14737.894624] [14737.894625] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14737.898974] x86_pkg_temp_thermal coretemp [14737.904192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14737.910277] kvm_intel kvm [14737.920607] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14737.931025] irqbypass i2c_i801 [14737.938495] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14737.942847] iTCO_wdt iTCO_vendor_support [14737.946331] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14737.955274] i2c_smbus crct10dif_pclmul [14737.962746] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14737.967183] joydev [14737.985921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14737.991138] crc32_pclmul [14737.998262] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14738.005385] ipmi_ssif ghash_clmulni_intel mei_me [14738.012511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14738.019653] mei lpc_ich [14738.026776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14738.034853] sha512_ssse3 [14738.040592] Call Trace: [14738.040593] [14738.047716] acpi_ipmi mfd_core [14738.054839] ? watchdog_timer_fn+0x1b4/0x220 [14738.061964] wmi ipmi_si [14738.064410] ? __hrtimer_run_queues+0x1d3/0x360 [14738.066419] ipmi_devintf ipmi_msghandler [14738.070601] ? hrtimer_interrupt+0x104/0x240 [14738.075381] ioatdma acpi_pad [14738.078950] ? __local_bh_enable+0x50/0x60 [14738.082781] acpi_power_meter [14738.086181] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14738.090008] ixgbe igb [14738.093840] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14738.097671] mdio crc32c_intel [14738.101501] [14738.103601] nvme [14738.105609] [14738.110393] ptp nvme_core [14738.113445] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14738.116929] i2c_algo_bit pps_core [14738.121717] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14738.124937] dca fuse [14738.129030] do_raw_spin_lock+0xb5/0xc0 [14738.133377] [14738.133378] irq event stamp: 41118220 [14738.137471] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14738.141561] hardirqs last enabled at (41118219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14738.146606] try_get_cap_refs+0x159/0x880 [ceph] [14738.151387] hardirqs last disabled at (41118220): [] sysvec_apic_timer_interrupt+0xe/0x90 [14738.153490] __ceph_get_caps+0x4a1/0x760 [ceph] [14738.155582] softirqs last enabled at (41118218): [] __do_softirq+0x2be/0x371 [14738.160716] ? __file_remove_privs+0x89/0x110 [14738.166190] softirqs last disabled at (41118213): [] irq_exit_rcu+0xb0/0xd0 [14738.184929] ? wake_bit_function+0x70/0x70 [14738.190146] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14738.197273] ? ceph_write_iter+0x260/0x770 [ceph] [14738.204393] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14738.211519] ceph_write_iter+0x260/0x770 [ceph] [14738.218660] Workqueue: events delayed_work [ceph] [14738.225787] ? lock_is_held_type+0xaa/0x120 [14738.229614] [14738.229615] RIP: 0010:lock_acquire+0x104/0x2b0 [14738.234401] vfs_write+0x324/0x4f0 [14738.239008] Code: ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 <48> 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 b2 e6 eb 7e [14738.243537] ksys_pwrite64+0x65/0xa0 [14738.247885] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [14738.251976] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14738.256672] [14738.256672] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 000000005553a634 [14738.261199] do_syscall_64+0x38/0x80 [14738.265373] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14738.268771] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14738.272343] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14738.277474] RIP: 0033:0x7f48f0eeb93b [14738.281045] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14738.286087] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14738.289659] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14738.308396] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14738.315954] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14738.323077] ORIG_RAX: 0000000000000012 [14738.330202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14738.337327] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14738.344451] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14738.351575] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14738.353758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14738.358110] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14738.361594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14738.370972] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14738.378442] Call Trace: [14738.383140] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14738.387404] [14738.406147] [14738.411359] ? watchdog_timer_fn+0x1b4/0x220 [14739.226043] ? __hrtimer_run_queues+0x1d3/0x360 [14739.230577] ? hrtimer_interrupt+0x104/0x240 [14739.234845] ? __local_bh_enable+0x50/0x60 [14739.238939] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14739.244164] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14739.249120] [14739.251216] [14739.253316] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14739.258632] ? lock_acquire+0x104/0x2b0 [14739.262470] ? igrab+0x43/0x50 [14739.265527] ? lock_release+0xc5/0x260 [14739.269273] _raw_spin_lock+0x30/0x40 [14739.272937] ? igrab+0x1d/0x50 [14739.275989] igrab+0x1d/0x50 [14739.278865] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14739.284017] delayed_work+0x1d8/0x280 [ceph] [14739.288295] ? process_scheduled_works+0x22a/0x510 [14739.293079] process_scheduled_works+0x2a4/0x510 [14739.297694] worker_thread+0x188/0x340 [14739.301441] ? rescuer_thread+0x310/0x310 [14739.305444] kthread+0xf5/0x130 [14739.308584] ? kthread_complete_and_exit+0x20/0x20 [14739.313368] ret_from_fork+0x31/0x40 [14739.316945] ? kthread_complete_and_exit+0x20/0x20 [14739.321730] ret_from_fork_asm+0x11/0x20 [14739.325653] [14765.713337] watchdog: BUG: soft lockup - CPU#0 stuck for 2637s! [dbench:78683] [14765.720554] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14765.725337] watchdog: BUG: soft lockup - CPU#7 stuck for 11634s! [kworker/7:4:78808] [14765.776011] irq event stamp: 18744358 [14765.776012] hardirqs last enabled at (18744357): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14765.783744] Modules linked in: [14765.787401] hardirqs last disabled at (18744358): [] sysvec_apic_timer_interrupt+0xe/0x90 [14765.797558] ceph [14765.800607] softirqs last enabled at (18744248): [] __do_softirq+0x2be/0x371 [14765.810332] libceph dns_resolver [14765.812258] softirqs last disabled at (18744243): [] irq_exit_rcu+0xb0/0xd0 [14765.820941] fscache [14765.824251] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14765.832761] netfs veth [14765.834946] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14765.834947] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14765.843889] nft_chain_nat [14765.846333] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14765.853804] xt_MASQUERADE [14765.859284] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14765.861986] nf_nat nf_conntrack [14765.880733] [14765.880734] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14765.883437] nf_defrag_ipv6 nf_defrag_ipv4 [14765.888654] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14765.891878] nft_compat nf_tables [14765.893370] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14765.900495] nfnetlink bridge [14765.904585] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14765.911708] stp [14765.915035] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14765.922161] llc xfs [14765.925124] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14765.932250] libcrc32c sunrpc [14765.934087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14765.941210] intel_rapl_msr [14765.943394] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14765.951473] intel_rapl_common [14765.954435] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14765.960174] sb_edac x86_pkg_temp_thermal [14765.962964] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14765.970087] coretemp [14765.973139] Call Trace: [14765.973140] [14765.980262] kvm_intel [14765.984269] ? watchdog_timer_fn+0x1b4/0x220 [14765.991392] kvm irqbypass [14765.993664] ? __hrtimer_run_queues+0x1d3/0x360 [14765.996107] i2c_i801 iTCO_wdt [14765.998120] ? hrtimer_interrupt+0x104/0x240 [14766.000474] iTCO_vendor_support [14766.004738] ? __local_bh_enable+0x50/0x60 [14766.007443] i2c_smbus crct10dif_pclmul [14766.011968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14766.015031] joydev crc32_pclmul [14766.019298] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14766.022522] ipmi_ssif ghash_clmulni_intel [14766.026613] [14766.030444] mei_me mei [14766.035663] [14766.038886] lpc_ich [14766.043845] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14766.047933] sha512_ssse3 acpi_ipmi [14766.050037] ? queued_spin_lock_slowpath+0x18/0x2e0 [14766.052474] mfd_core wmi [14766.054574] do_raw_spin_lock+0xb5/0xc0 [14766.056756] ipmi_si ipmi_devintf [14766.062061] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14766.065544] ipmi_msghandler ioatdma [14766.070417] try_get_cap_refs+0x159/0x880 [ceph] [14766.073032] acpi_pad acpi_power_meter [14766.076869] __ceph_get_caps+0x4a1/0x760 [ceph] [14766.080175] ixgbe igb [14766.084963] ? __file_remove_privs+0x89/0x110 [14766.088529] mdio crc32c_intel [14766.093141] ? wake_bit_function+0x70/0x70 [14766.096885] nvme ptp [14766.101410] ? ceph_write_iter+0x260/0x770 [ceph] [14766.103766] nvme_core i2c_algo_bit [14766.108116] ceph_write_iter+0x260/0x770 [ceph] [14766.111166] pps_core dca [14766.115262] ? lock_is_held_type+0xaa/0x120 [14766.117529] fuse [14766.122227] vfs_write+0x324/0x4f0 [14766.125711] irq event stamp: 41332930 [14766.130256] ksys_pwrite64+0x65/0xa0 [14766.132869] hardirqs last enabled at (41332929): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14766.137047] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14766.138969] hardirqs last disabled at (41332930): [] sysvec_apic_timer_interrupt+0xe/0x90 [14766.142368] do_syscall_64+0x38/0x80 [14766.146024] softirqs last enabled at (41332928): [] __do_softirq+0x2be/0x371 [14766.149597] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14766.159754] softirqs last disabled at (41332919): [] irq_exit_rcu+0xb0/0xd0 [14766.164884] RIP: 0033:0x7f48f0eeb93b [14766.174607] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14766.178178] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14766.186862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14766.191906] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14766.200417] Workqueue: events delayed_work [ceph] [14766.203989] ORIG_RAX: 0000000000000012 [14766.213366] [14766.213367] RIP: 0010:lock_acquire+0xf0/0x2b0 [14766.232105] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14766.239575] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14766.244792] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14766.249489] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14766.253319] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14766.254811] [14766.254812] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14766.259161] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14766.266284] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14766.285037] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14766.292165] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14766.297387] [14766.304506] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14766.350944] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14766.358067] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14766.366144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14766.371881] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14766.379020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14766.386148] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14766.393271] Call Trace: [14766.395717] [14766.397747] ? watchdog_timer_fn+0x1b4/0x220 [14766.402024] ? __hrtimer_run_queues+0x1d3/0x360 [14766.406552] ? hrtimer_interrupt+0x104/0x240 [14766.410813] ? __local_bh_enable+0x50/0x60 [14766.414907] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14766.420131] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14766.425089] [14766.427185] [14766.429282] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14766.434592] ? lock_acquire+0xf0/0x2b0 [14766.438340] ? igrab+0x43/0x50 [14766.441391] ? lock_release+0xc5/0x260 [14766.445135] _raw_spin_lock+0x30/0x40 [14766.448792] ? igrab+0x1d/0x50 [14766.451844] igrab+0x1d/0x50 [14766.454740] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14766.459887] delayed_work+0x1d8/0x280 [ceph] [14766.464170] ? process_scheduled_works+0x22a/0x510 [14766.468960] process_scheduled_works+0x2a4/0x510 [14766.473575] worker_thread+0x188/0x340 [14766.477325] ? rescuer_thread+0x310/0x310 [14766.481329] kthread+0xf5/0x130 [14766.484464] ? kthread_complete_and_exit+0x20/0x20 [14766.489250] ret_from_fork+0x31/0x40 [14766.492827] ? kthread_complete_and_exit+0x20/0x20 [14766.497613] ret_from_fork_asm+0x11/0x20 [14766.501536] [14793.713827] watchdog: BUG: soft lockup - CPU#0 stuck for 2663s! [dbench:78683] [14793.721041] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14793.725826] watchdog: BUG: soft lockup - CPU#7 stuck for 11660s! [kworker/7:4:78808] [14793.776506] irq event stamp: 18802522 [14793.784238] Modules linked in: ceph libceph dns_resolver fscache netfs [14793.787895] hardirqs last enabled at (18802521): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14793.794421] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [14793.804580] hardirqs last disabled at (18802522): [] sysvec_apic_timer_interrupt+0xe/0x90 [14793.804583] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [14793.820395] softirqs last enabled at (18802400): [] __do_softirq+0x2be/0x371 [14793.828127] llc xfs libcrc32c sunrpc intel_rapl_msr [14793.836812] softirqs last disabled at (18802395): [] irq_exit_rcu+0xb0/0xd0 [14793.836815] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [14793.850288] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14793.857503] kvm irqbypass [14793.866444] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14793.866445] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [14793.869147] i2c_i801 iTCO_wdt [14793.876618] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [14793.882096] iTCO_vendor_support i2c_smbus [14793.885146] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14793.903884] crct10dif_pclmul [14793.907973] [14793.907974] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14793.913191] joydev crc32_pclmul [14793.916155] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14793.917646] ipmi_ssif ghash_clmulni_intel [14793.924772] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14793.927996] mei_me mei [14793.935118] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14793.939225] lpc_ich sha512_ssse3 [14793.946352] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14793.948794] acpi_ipmi mfd_core [14793.955919] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14793.959230] wmi ipmi_si [14793.966353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14793.969504] ipmi_devintf ipmi_msghandler [14793.977579] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14793.980108] ioatdma acpi_pad [14793.985847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14793.989850] acpi_power_meter ixgbe [14793.996975] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14793.999938] igb [14794.007061] Call Trace: [14794.007063] [14794.010547] mdio crc32c_intel [14794.017674] ? watchdog_timer_fn+0x1b4/0x220 [14794.019509] nvme [14794.021952] ? __hrtimer_run_queues+0x1d3/0x360 [14794.023963] ptp nvme_core [14794.027017] ? hrtimer_interrupt+0x104/0x240 [14794.031277] i2c_algo_bit [14794.033201] ? __local_bh_enable+0x50/0x60 [14794.037742] pps_core [14794.040450] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14794.044712] dca fuse [14794.047329] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14794.051419] [14794.051420] irq event stamp: 41554962 [14794.053690] [14794.058908] hardirqs last enabled at (41554961): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14794.061177] [14794.066136] hardirqs last disabled at (41554962): [] sysvec_apic_timer_interrupt+0xe/0x90 [14794.067626] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14794.071284] softirqs last enabled at (41554960): [] __do_softirq+0x2be/0x371 [14794.073386] ? queued_spin_lock_slowpath+0x2b/0x2e0 [14794.083539] softirqs last disabled at (41554955): [] irq_exit_rcu+0xb0/0xd0 [14794.085638] do_raw_spin_lock+0xb5/0xc0 [14794.095359] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14794.100665] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14794.109348] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14794.114230] try_get_cap_refs+0x159/0x880 [ceph] [14794.122737] Workqueue: events delayed_work [ceph] [14794.126575] __ceph_get_caps+0x4a1/0x760 [ceph] [14794.135946] [14794.135947] RIP: 0010:lock_acquire+0xf0/0x2b0 [14794.140735] ? __file_remove_privs+0x89/0x110 [14794.148201] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14794.152831] ? wake_bit_function+0x70/0x70 [14794.157526] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14794.162053] ? ceph_write_iter+0x260/0x770 [ceph] [14794.163540] [14794.163541] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14794.167893] ceph_write_iter+0x260/0x770 [ceph] [14794.172244] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14794.190986] ? lock_is_held_type+0xaa/0x120 [14794.195071] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14794.200291] vfs_write+0x324/0x4f0 [14794.204987] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14794.206483] ksys_pwrite64+0x65/0xa0 [14794.213601] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14794.218126] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14794.225249] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14794.229428] do_syscall_64+0x38/0x80 [14794.236551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14794.239950] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14794.247072] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14794.250645] RIP: 0033:0x7f48f0eeb93b [14794.257768] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14794.262898] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14794.270977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14794.274547] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14794.280284] Call Trace: [14794.280285] [14794.285328] ORIG_RAX: 0000000000000012 [14794.292453] ? watchdog_timer_fn+0x1b4/0x220 [14794.296024] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14794.303149] ? __hrtimer_run_queues+0x1d3/0x360 [14794.321884] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14794.329012] ? hrtimer_interrupt+0x104/0x240 [14794.334229] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14794.336670] ? __local_bh_enable+0x50/0x60 [14794.338682] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14794.342515] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14794.346775] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14794.353902] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14794.358428] [14794.365548] [14794.409733] [14794.411829] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14794.417138] ? lock_acquire+0xf0/0x2b0 [14794.420887] ? igrab+0x43/0x50 [14794.423938] ? lock_release+0xc5/0x260 [14794.427684] _raw_spin_lock+0x30/0x40 [14794.431348] ? igrab+0x1d/0x50 [14794.434398] igrab+0x1d/0x50 [14794.437277] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14794.442427] delayed_work+0x1d8/0x280 [ceph] [14794.446709] ? process_scheduled_works+0x22a/0x510 [14794.451511] process_scheduled_works+0x2a4/0x510 [14794.456131] worker_thread+0x188/0x340 [14794.459878] ? rescuer_thread+0x310/0x310 [14794.463883] kthread+0xf5/0x130 [14794.467029] ? kthread_complete_and_exit+0x20/0x20 [14794.471814] ret_from_fork+0x31/0x40 [14794.475392] ? kthread_complete_and_exit+0x20/0x20 [14794.480177] ret_from_fork_asm+0x11/0x20 [14794.484108] [14821.714314] watchdog: BUG: soft lockup - CPU#0 stuck for 2689s! [dbench:78683] [14821.721527] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14821.726314] watchdog: BUG: soft lockup - CPU#7 stuck for 11686s! [kworker/7:4:78808] [14821.776967] irq event stamp: 18860542 [14821.776968] hardirqs last enabled at (18860541): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14821.784715] Modules linked in: [14821.788374] hardirqs last disabled at (18860542): [] sysvec_apic_timer_interrupt+0xe/0x90 [14821.798531] ceph [14821.801582] softirqs last enabled at (18860532): [] __do_softirq+0x2be/0x371 [14821.811307] libceph [14821.813230] softirqs last disabled at (18860527): [] irq_exit_rcu+0xb0/0xd0 [14821.821913] dns_resolver [14821.824099] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14821.832610] fscache netfs [14821.835227] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14821.844171] veth nft_chain_nat [14821.846875] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14821.854345] xt_MASQUERADE nf_nat [14821.857483] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14821.862961] nf_conntrack [14821.866272] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14821.885009] nf_defrag_ipv6 nf_defrag_ipv4 [14821.887626] [14821.887627] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14821.892843] nft_compat nf_tables [14821.896933] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14821.898425] nfnetlink bridge [14821.905550] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14821.908859] stp llc [14821.915983] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14821.918965] xfs libcrc32c [14821.926089] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14821.928273] sunrpc intel_rapl_msr [14821.935399] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14821.938101] intel_rapl_common sb_edac [14821.945225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14821.948624] x86_pkg_temp_thermal coretemp [14821.956711] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14821.960453] kvm_intel kvm [14821.966191] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14821.970282] irqbypass i2c_i801 [14821.977405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14821.980109] iTCO_wdt iTCO_vendor_support [14821.987233] Call Trace: [14821.990372] i2c_smbus crct10dif_pclmul [14821.997497] [14822.001500] joydev crc32_pclmul [14822.003945] ? watchdog_timer_fn+0x1b4/0x220 [14822.007773] ipmi_ssif ghash_clmulni_intel [14822.009785] ? __hrtimer_run_queues+0x1d3/0x360 [14822.013010] mei_me mei [14822.017276] ? hrtimer_interrupt+0x104/0x240 [14822.021363] lpc_ich sha512_ssse3 [14822.025889] ? __local_bh_enable+0x50/0x60 [14822.028333] acpi_ipmi mfd_core [14822.032599] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14822.035908] wmi ipmi_si [14822.039998] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14822.043134] ipmi_devintf ipmi_msghandler [14822.048353] [14822.050883] ioatdma acpi_pad [14822.055839] [14822.059845] acpi_power_meter [14822.061942] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14822.064906] ixgbe igb [14822.067006] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14822.069966] mdio crc32c_intel [14822.075273] do_raw_spin_lock+0xb5/0xc0 [14822.077630] nvme ptp [14822.082500] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14822.085550] nvme_core i2c_algo_bit [14822.089384] try_get_cap_refs+0x159/0x880 [ceph] [14822.091651] pps_core dca [14822.096441] __ceph_get_caps+0x4a1/0x760 [ceph] [14822.099920] fuse [14822.104536] ? __file_remove_privs+0x89/0x110 [14822.107149] irq event stamp: 41776590 [14822.111672] ? wake_bit_function+0x70/0x70 [14822.113597] hardirqs last enabled at (41776589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14822.117950] ? ceph_write_iter+0x260/0x770 [ceph] [14822.121605] hardirqs last disabled at (41776590): [] sysvec_apic_timer_interrupt+0xe/0x90 [14822.125713] ceph_write_iter+0x260/0x770 [ceph] [14822.135871] softirqs last enabled at (41776588): [] __do_softirq+0x2be/0x371 [14822.140572] ? lock_is_held_type+0xaa/0x120 [14822.150291] softirqs last disabled at (41776583): [] irq_exit_rcu+0xb0/0xd0 [14822.154818] vfs_write+0x324/0x4f0 [14822.163500] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14822.167683] ksys_pwrite64+0x65/0xa0 [14822.176187] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14822.179588] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14822.188962] Workqueue: events delayed_work [ceph] [14822.192535] do_syscall_64+0x38/0x80 [14822.200005] [14822.200006] RIP: 0010:lock_acquire+0x17b/0x2b0 [14822.205137] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14822.209833] Code: c8 89 d1 89 f2 4c 89 e6 e8 62 5b ff ff 59 5e 4c 8b 14 24 65 ff 0d 5d e6 eb 7e 0f 85 c8 fe ff ff 0f 1f 44 00 00 e9 be fe ff ff <4c> 89 14 24 e8 9c 8a 02 00 4c 8b 14 24 84 c0 0f 85 c0 fe ff ff 0f [14822.213405] RIP: 0033:0x7f48f0eeb93b [14822.214894] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [14822.219333] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14822.224376] [14822.224377] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [14822.243113] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14822.246683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [14822.251901] ORIG_RAX: 0000000000000012 [14822.270638] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [14822.272131] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14822.279254] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [14822.284473] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14822.291595] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14822.295426] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14822.302551] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14822.309673] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14822.316800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14822.323924] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14822.331048] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14822.338177] [14822.346249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14822.382666] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14822.389790] Call Trace: [14822.392236] [14822.394246] ? watchdog_timer_fn+0x1b4/0x220 [14822.398511] ? __hrtimer_run_queues+0x1d3/0x360 [14822.403038] ? hrtimer_interrupt+0x104/0x240 [14822.407308] ? __local_bh_enable+0x50/0x60 [14822.411399] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14822.416616] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14822.421572] [14822.423671] [14822.425767] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14822.431075] ? lock_acquire+0x17b/0x2b0 [14822.434913] ? igrab+0x43/0x50 [14822.437961] ? lock_release+0xc5/0x260 [14822.441722] _raw_spin_lock+0x30/0x40 [14822.445380] ? igrab+0x1d/0x50 [14822.448431] igrab+0x1d/0x50 [14822.451310] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14822.456458] delayed_work+0x1d8/0x280 [ceph] [14822.460742] ? process_scheduled_works+0x22a/0x510 [14822.465531] process_scheduled_works+0x2a4/0x510 [14822.470145] worker_thread+0x188/0x340 [14822.473895] ? rescuer_thread+0x310/0x310 [14822.477898] kthread+0xf5/0x130 [14822.481036] ? kthread_complete_and_exit+0x20/0x20 [14822.485822] ret_from_fork+0x31/0x40 [14822.489399] ? kthread_complete_and_exit+0x20/0x20 [14822.494184] ret_from_fork_asm+0x11/0x20 [14822.498114] [14849.714803] watchdog: BUG: soft lockup - CPU#0 stuck for 2715s! [dbench:78683] [14849.722022] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14849.726802] watchdog: BUG: soft lockup - CPU#7 stuck for 11712s! [kworker/7:4:78808] [14849.777475] irq event stamp: 18918632 [14849.785203] Modules linked in: ceph libceph dns_resolver fscache netfs [14849.788862] hardirqs last enabled at (18918631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14849.795387] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [14849.805544] hardirqs last disabled at (18918632): [] sysvec_apic_timer_interrupt+0xe/0x90 [14849.811629] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [14849.821353] softirqs last enabled at (18918596): [] __do_softirq+0x2be/0x371 [14849.829092] llc xfs libcrc32c sunrpc intel_rapl_msr [14849.837777] softirqs last disabled at (18918591): [] irq_exit_rcu+0xb0/0xd0 [14849.837780] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [14849.851246] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14849.858457] kvm irqbypass [14849.867400] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14849.870104] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [14849.877575] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14849.877577] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me [14849.890195] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14849.896625] mei lpc_ich [14849.915360] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14849.915361] sha512_ssse3 [14849.915362] [14849.917893] acpi_ipmi mfd_core [14849.923110] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14849.925727] wmi ipmi_si [14849.927217] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14849.930356] ipmi_devintf ipmi_msghandler [14849.937480] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14849.940011] ioatdma acpi_pad [14849.947135] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14849.951139] acpi_power_meter [14849.958262] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14849.961228] ixgbe [14849.968350] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14849.971314] igb mdio [14849.978440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14849.980450] crc32c_intel nvme [14849.988528] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14849.990797] ptp nvme_core [14849.996535] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14849.999587] i2c_algo_bit [14850.006711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14850.009415] pps_core dca [14850.016538] Call Trace: [14850.016540] [14850.019156] fuse [14850.026281] ? watchdog_timer_fn+0x1b4/0x220 [14850.028897] irq event stamp: 41998184 [14850.031343] ? __hrtimer_run_queues+0x1d3/0x360 [14850.033352] hardirqs last enabled at (41998183): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14850.035280] ? hrtimer_interrupt+0x104/0x240 [14850.039540] hardirqs last disabled at (41998184): [] sysvec_apic_timer_interrupt+0xe/0x90 [14850.043202] ? __local_bh_enable+0x50/0x60 [14850.047730] softirqs last enabled at (41998182): [] __do_softirq+0x2be/0x371 [14850.057891] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14850.062152] softirqs last disabled at (41998177): [] irq_exit_rcu+0xb0/0xd0 [14850.071877] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14850.075968] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14850.084651] [14850.089868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14850.098380] [14850.098381] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14850.103335] Workqueue: events delayed_work [ceph] [14850.112718] ? queued_spin_lock_slowpath+0x18/0x2e0 [14850.114810] [14850.114811] RIP: 0010:lock_acquire+0xf0/0x2b0 [14850.122284] do_raw_spin_lock+0xb5/0xc0 [14850.124380] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14850.129686] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14850.134382] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14850.139256] try_get_cap_refs+0x159/0x880 [ceph] [14850.140744] [14850.145100] __ceph_get_caps+0x4a1/0x760 [ceph] [14850.148924] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14850.167667] ? __file_remove_privs+0x89/0x110 [14850.172445] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14850.177665] ? wake_bit_function+0x70/0x70 [14850.182274] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14850.183766] ? ceph_write_iter+0x260/0x770 [ceph] [14850.188289] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14850.195414] ceph_write_iter+0x260/0x770 [ceph] [14850.199764] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14850.206891] ? lock_is_held_type+0xaa/0x120 [14850.210979] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14850.218104] vfs_write+0x324/0x4f0 [14850.222800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14850.229929] ksys_pwrite64+0x65/0xa0 [14850.234448] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14850.241575] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14850.245749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14850.253830] do_syscall_64+0x38/0x80 [14850.257224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14850.262962] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14850.266532] Call Trace: [14850.266534] [14850.273658] RIP: 0033:0x7f48f0eeb93b [14850.278789] ? watchdog_timer_fn+0x1b4/0x220 [14850.285914] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14850.289494] ? __hrtimer_run_queues+0x1d3/0x360 [14850.296615] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14850.301664] ? hrtimer_interrupt+0x104/0x240 [14850.304104] ORIG_RAX: 0000000000000012 [14850.306113] ? __local_bh_enable+0x50/0x60 [14850.309686] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14850.313952] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14850.332686] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14850.337212] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14850.342428] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14850.346691] [14850.350524] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14850.354613] [14850.354615] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14850.361738] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14850.366960] ? lock_acquire+0xf0/0x2b0 [14850.374085] [14850.379039] ? igrab+0x43/0x50 [14850.418905] ? lock_release+0xc5/0x260 [14850.422652] _raw_spin_lock+0x30/0x40 [14850.426317] ? igrab+0x1d/0x50 [14850.429366] igrab+0x1d/0x50 [14850.432245] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14850.437394] delayed_work+0x1d8/0x280 [ceph] [14850.441675] ? process_scheduled_works+0x22a/0x510 [14850.446465] process_scheduled_works+0x2a4/0x510 [14850.451081] worker_thread+0x188/0x340 [14850.454829] ? rescuer_thread+0x310/0x310 [14850.458833] kthread+0xf5/0x130 [14850.461972] ? kthread_complete_and_exit+0x20/0x20 [14850.466755] ret_from_fork+0x31/0x40 [14850.470336] ? kthread_complete_and_exit+0x20/0x20 [14850.475119] ret_from_fork_asm+0x11/0x20 [14850.479050] [14877.715290] watchdog: BUG: soft lockup - CPU#0 stuck for 2741s! [dbench:78683] [14877.722501] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14877.727290] watchdog: BUG: soft lockup - CPU#7 stuck for 11738s! [kworker/7:4:78808] [14877.777943] irq event stamp: 18976626 [14877.785690] Modules linked in: ceph libceph [14877.789348] hardirqs last enabled at (18976625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14877.793525] dns_resolver fscache [14877.803683] hardirqs last disabled at (18976626): [] sysvec_apic_timer_interrupt+0xe/0x90 [14877.806994] netfs veth nft_chain_nat xt_MASQUERADE [14877.816717] softirqs last enabled at (18976558): [] __do_softirq+0x2be/0x371 [14877.821588] nf_nat nf_conntrack nf_defrag_ipv6 [14877.830273] softirqs last disabled at (18976553): [] irq_exit_rcu+0xb0/0xd0 [14877.834797] nf_defrag_ipv4 nft_compat nf_tables [14877.843308] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14877.847919] nfnetlink bridge [14877.856862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14877.856864] stp [14877.859828] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14877.867296] llc [14877.869134] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14877.874611] xfs libcrc32c [14877.876449] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14877.895188] sunrpc intel_rapl_msr [14877.897891] [14877.897892] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14877.903110] intel_rapl_common sb_edac [14877.906507] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14877.907997] x86_pkg_temp_thermal coretemp [14877.915122] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14877.918866] kvm_intel kvm [14877.925989] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14877.930080] irqbypass i2c_i801 [14877.937204] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14877.939908] iTCO_wdt iTCO_vendor_support [14877.947032] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14877.950170] i2c_smbus crct10dif_pclmul [14877.957294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14877.961298] joydev crc32_pclmul [14877.969374] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14877.973206] ipmi_ssif ghash_clmulni_intel [14877.978943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14877.982184] mei_me [14877.989310] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14877.993399] mei lpc_ich [14878.000523] Call Trace: [14878.000524] [14878.002623] sha512_ssse3 acpi_ipmi [14878.009747] ? watchdog_timer_fn+0x1b4/0x220 [14878.012276] mfd_core wmi [14878.014721] ? __hrtimer_run_queues+0x1d3/0x360 [14878.016731] ipmi_si [14878.020219] ? hrtimer_interrupt+0x104/0x240 [14878.024480] ipmi_devintf ipmi_msghandler [14878.027095] ? __local_bh_enable+0x50/0x60 [14878.031619] ioatdma acpi_pad [14878.033807] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14878.038067] acpi_power_meter ixgbe [14878.042074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14878.046163] igb mdio [14878.049129] [14878.054345] crc32c_intel nvme [14878.057830] [14878.057831] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14878.062787] ptp nvme_core [14878.065061] ? queued_spin_lock_slowpath+0x18/0x2e0 [14878.067154] i2c_algo_bit pps_core [14878.070206] do_raw_spin_lock+0xb5/0xc0 [14878.072301] dca fuse [14878.077608] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14878.080311] [14878.080311] irq event stamp: 42219534 [14878.085184] try_get_cap_refs+0x159/0x880 [ceph] [14878.088578] hardirqs last enabled at (42219533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14878.092415] __ceph_get_caps+0x4a1/0x760 [ceph] [14878.094690] hardirqs last disabled at (42219534): [] sysvec_apic_timer_interrupt+0xe/0x90 [14878.099477] ? __file_remove_privs+0x89/0x110 [14878.100977] softirqs last enabled at (42219532): [] __do_softirq+0x2be/0x371 [14878.104639] ? wake_bit_function+0x70/0x70 [14878.109250] softirqs last disabled at (42219527): [] irq_exit_rcu+0xb0/0xd0 [14878.119410] ? ceph_write_iter+0x260/0x770 [ceph] [14878.123931] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14878.133655] ceph_write_iter+0x260/0x770 [ceph] [14878.138006] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14878.146694] ? lock_is_held_type+0xaa/0x120 [14878.150780] Workqueue: events delayed_work [ceph] [14878.159293] vfs_write+0x324/0x4f0 [14878.163990] [14878.163991] RIP: 0010:lock_release+0x14e/0x260 [14878.173370] ksys_pwrite64+0x65/0xa0 [14878.177890] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [14878.185361] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14878.189539] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [14878.194238] do_syscall_64+0x38/0x80 [14878.197632] [14878.197633] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [14878.199126] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14878.203562] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [14878.207133] RIP: 0033:0x7f48f0eeb93b [14878.225869] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [14878.231002] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14878.236218] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [14878.239789] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14878.241280] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [14878.248403] ORIG_RAX: 0000000000000012 [14878.253447] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14878.260572] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14878.264143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14878.271267] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14878.290005] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14878.297128] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14878.302347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14878.309471] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14878.313302] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14878.321379] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14878.328502] Call Trace: [14878.328503] [14878.334246] [14878.341370] ? watchdog_timer_fn+0x1b4/0x220 [14878.395021] ? __hrtimer_run_queues+0x1d3/0x360 [14878.399549] ? hrtimer_interrupt+0x104/0x240 [14878.403818] ? __local_bh_enable+0x50/0x60 [14878.407910] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14878.413127] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14878.418083] [14878.420180] [14878.422278] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14878.427582] ? igrab+0x43/0x50 [14878.430636] ? lock_release+0x14e/0x260 [14878.434472] ? lock_release+0xc5/0x260 [14878.438217] _raw_spin_unlock+0x1b/0x40 [14878.442050] igrab+0x43/0x50 [14878.444926] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14878.450074] delayed_work+0x1d8/0x280 [ceph] [14878.454356] ? process_scheduled_works+0x22a/0x510 [14878.459147] process_scheduled_works+0x2a4/0x510 [14878.463763] worker_thread+0x188/0x340 [14878.467512] ? rescuer_thread+0x310/0x310 [14878.471514] kthread+0xf5/0x130 [14878.474653] ? kthread_complete_and_exit+0x20/0x20 [14878.479438] ret_from_fork+0x31/0x40 [14878.483016] ? kthread_complete_and_exit+0x20/0x20 [14878.487799] ret_from_fork_asm+0x11/0x20 [14878.491723] [14905.715779] watchdog: BUG: soft lockup - CPU#0 stuck for 2768s! [dbench:78683] [14905.722998] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14905.727778] watchdog: BUG: soft lockup - CPU#7 stuck for 11765s! [kworker/7:4:78808] [14905.778456] irq event stamp: 19034696 [14905.786188] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [14905.789845] hardirqs last enabled at (19034695): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14905.798017] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [14905.808174] hardirqs last disabled at (19034696): [] sysvec_apic_timer_interrupt+0xe/0x90 [14905.815212] nft_compat nf_tables nfnetlink bridge stp llc xfs [14905.824937] softirqs last enabled at (19034650): [] __do_softirq+0x2be/0x371 [14905.830769] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [14905.839454] softirqs last disabled at (19034645): [] irq_exit_rcu+0xb0/0xd0 [14905.847798] coretemp kvm_intel kvm irqbypass [14905.856311] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14905.860661] i2c_i801 iTCO_wdt [14905.869605] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14905.869607] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14905.872655] iTCO_vendor_support i2c_smbus [14905.880128] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14905.885604] crct10dif_pclmul joydev [14905.889694] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14905.908433] crc32_pclmul ipmi_ssif [14905.912021] [14905.912022] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14905.917238] ghash_clmulni_intel [14905.920721] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14905.922213] mei_me mei [14905.929336] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14905.932561] lpc_ich sha512_ssse3 [14905.939686] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14905.942130] acpi_ipmi mfd_core [14905.949254] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14905.952564] wmi ipmi_si [14905.959689] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14905.962824] ipmi_devintf ipmi_msghandler [14905.969950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14905.972480] ioatdma acpi_pad [14905.980558] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14905.984561] acpi_power_meter [14905.990301] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14905.993265] ixgbe igb [14906.000388] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14906.003351] mdio crc32c_intel [14906.010476] Call Trace: [14906.012833] nvme [14906.019960] [14906.023008] ptp nvme_core [14906.025460] ? watchdog_timer_fn+0x1b4/0x220 [14906.027383] i2c_algo_bit pps_core [14906.029396] ? __hrtimer_run_queues+0x1d3/0x360 [14906.032099] dca [14906.036367] ? hrtimer_interrupt+0x104/0x240 [14906.039761] fuse [14906.044284] ? __local_bh_enable+0x50/0x60 [14906.046123] irq event stamp: 42441460 [14906.050388] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14906.052310] hardirqs last enabled at (42441459): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14906.056403] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14906.060058] hardirqs last disabled at (42441460): [] sysvec_apic_timer_interrupt+0xe/0x90 [14906.065276] [14906.075434] softirqs last enabled at (42441458): [] __do_softirq+0x2be/0x371 [14906.080409] [14906.090132] softirqs last disabled at (42441453): [] irq_exit_rcu+0xb0/0xd0 [14906.092231] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14906.100913] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14906.103016] ? queued_spin_lock_slowpath+0x16/0x2e0 [14906.111521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14906.111523] Workqueue: events delayed_work [ceph] [14906.116827] do_raw_spin_lock+0xb5/0xc0 [14906.126204] [14906.126205] RIP: 0010:lock_acquire+0xf0/0x2b0 [14906.131077] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14906.138544] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14906.143246] try_get_cap_refs+0x159/0x880 [ceph] [14906.147073] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14906.148568] __ceph_get_caps+0x4a1/0x760 [ceph] [14906.152914] [14906.152915] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14906.157702] ? __file_remove_privs+0x89/0x110 [14906.176437] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14906.176454] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14906.181067] ? wake_bit_function+0x70/0x70 [14906.186283] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14906.190810] ? ceph_write_iter+0x260/0x770 [ceph] [14906.192297] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14906.199421] ceph_write_iter+0x260/0x770 [ceph] [14906.203773] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14906.210899] ? lock_is_held_type+0xaa/0x120 [14906.218020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14906.222113] vfs_write+0x324/0x4f0 [14906.229236] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14906.233939] ksys_pwrite64+0x65/0xa0 [14906.241056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14906.245583] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14906.253657] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14906.257838] do_syscall_64+0x38/0x80 [14906.263574] Call Trace: [14906.263576] [14906.266971] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14906.274094] ? watchdog_timer_fn+0x1b4/0x220 [14906.277666] RIP: 0033:0x7f48f0eeb93b [14906.284789] ? __hrtimer_run_queues+0x1d3/0x360 [14906.289921] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14906.297048] ? hrtimer_interrupt+0x104/0x240 [14906.300614] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14906.303059] ? __local_bh_enable+0x50/0x60 [14906.305069] ORIG_RAX: 0000000000000012 [14906.310115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14906.314378] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14906.317948] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14906.322483] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14906.341219] [14906.345482] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14906.350700] [14906.350702] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14906.354792] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14906.358628] ? lock_acquire+0xf0/0x2b0 [14906.363838] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14906.370966] ? igrab+0x43/0x50 [14906.375926] [14906.383046] ? lock_release+0xc5/0x260 [14906.426641] _raw_spin_lock+0x30/0x40 [14906.430297] ? igrab+0x1d/0x50 [14906.433347] igrab+0x1d/0x50 [14906.436226] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14906.441377] delayed_work+0x1d8/0x280 [ceph] [14906.445658] ? process_scheduled_works+0x22a/0x510 [14906.450464] process_scheduled_works+0x2a4/0x510 [14906.455080] worker_thread+0x188/0x340 [14906.458831] ? rescuer_thread+0x310/0x310 [14906.462841] kthread+0xf5/0x130 [14906.465981] ? kthread_complete_and_exit+0x20/0x20 [14906.470763] ret_from_fork+0x31/0x40 [14906.474343] ? kthread_complete_and_exit+0x20/0x20 [14906.479127] ret_from_fork_asm+0x11/0x20 [14906.483059] [14932.880619] rcu: INFO: rcu_preempt self-detected stall on CPU [14932.886375] rcu: 7-....: (2914057 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/2912873 fqs=1441796 [14932.896801] rcu: (t=2996217 jiffies g=1309609 q=166109 ncpus=8) [14932.902799] Sending NMI from CPU 7 to CPUs 0: [14932.907149] NMI backtrace for cpu 0 [14932.907151] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14932.907153] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14932.907153] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [14932.907157] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [14932.907158] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14932.907160] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14932.907161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14932.907161] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14932.907162] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14932.907163] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14932.907164] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14932.907165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14932.907166] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14932.907167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14932.907167] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14932.907168] Call Trace: [14932.907169] [14932.907170] ? nmi_cpu_backtrace+0xad/0x120 [14932.907173] ? nmi_cpu_backtrace_handler+0x11/0x20 [14932.907175] ? nmi_handle+0xd4/0x260 [14932.907180] ? default_do_nmi+0x49/0xf0 [14932.907181] ? exc_nmi+0x11e/0x150 [14932.907183] ? end_repeat_nmi+0x16/0x67 [14932.907187] ? queued_spin_lock_slowpath+0x18/0x2e0 [14932.907189] ? queued_spin_lock_slowpath+0x18/0x2e0 [14932.907192] ? queued_spin_lock_slowpath+0x18/0x2e0 [14932.907194] [14932.907195] [14932.907196] do_raw_spin_lock+0xb5/0xc0 [14932.907198] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14932.907213] try_get_cap_refs+0x159/0x880 [ceph] [14932.907228] __ceph_get_caps+0x4a1/0x760 [ceph] [14932.907240] ? __file_remove_privs+0x89/0x110 [14932.907242] ? wake_bit_function+0x70/0x70 [14932.907245] ? ceph_write_iter+0x260/0x770 [ceph] [14932.907256] ceph_write_iter+0x260/0x770 [ceph] [14932.907267] ? lock_is_held_type+0xaa/0x120 [14932.907270] vfs_write+0x324/0x4f0 [14932.907274] ksys_pwrite64+0x65/0xa0 [14932.907277] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14932.907279] do_syscall_64+0x38/0x80 [14932.907282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14932.907284] RIP: 0033:0x7f48f0eeb93b [14932.907285] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14932.907286] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [14932.907288] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14932.907289] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14932.907289] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14932.907290] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14932.907291] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14932.907294] [14932.908147] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14933.211968] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14933.219439] Workqueue: events delayed_work [ceph] [14933.224155] RIP: 0010:lock_acquire+0xf0/0x2b0 [14933.228505] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14933.247242] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14933.252460] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14933.259585] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14933.266707] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14933.273833] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14933.280958] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14933.288080] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14933.296159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14933.301897] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14933.309020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14933.316145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14933.323268] Call Trace: [14933.325714] [14933.327724] ? rcu_dump_cpu_stacks+0xd1/0x180 [14933.332082] ? rcu_sched_clock_irq+0x633/0x1390 [14933.336606] ? lock_release+0xc5/0x260 [14933.340352] ? lock_is_held_type+0xaa/0x120 [14933.344529] ? lock_is_held_type+0xaa/0x120 [14933.348707] ? tick_sched_do_timer+0xa0/0xa0 [14933.352979] ? rcu_read_lock_sched_held+0x56/0x90 [14933.357676] ? tick_sched_do_timer+0xa0/0xa0 [14933.361939] ? update_process_times+0x5f/0x90 [14933.366291] ? tick_sched_handle.isra.16+0x1e/0x50 [14933.371074] ? tick_sched_timer+0x6f/0x80 [14933.375080] ? __hrtimer_run_queues+0x1d3/0x360 [14933.379607] ? hrtimer_interrupt+0x104/0x240 [14933.383876] ? __local_bh_enable+0x50/0x60 [14933.387970] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14933.393194] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14933.398150] [14933.400248] [14933.402344] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14933.407654] ? lock_acquire+0xf0/0x2b0 [14933.411402] ? igrab+0x43/0x50 [14933.414452] ? lock_release+0xc5/0x260 [14933.418197] _raw_spin_lock+0x30/0x40 [14933.421855] ? igrab+0x1d/0x50 [14933.424904] igrab+0x1d/0x50 [14933.427782] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14933.432933] delayed_work+0x1d8/0x280 [ceph] [14933.437214] ? process_scheduled_works+0x22a/0x510 [14933.442003] process_scheduled_works+0x2a4/0x510 [14933.446619] worker_thread+0x188/0x340 [14933.450368] ? rescuer_thread+0x310/0x310 [14933.454373] kthread+0xf5/0x130 [14933.457511] ? kthread_complete_and_exit+0x20/0x20 [14933.462295] ret_from_fork+0x31/0x40 [14933.465865] ? kthread_complete_and_exit+0x20/0x20 [14933.470649] ret_from_fork_asm+0x11/0x20 [14933.474573] [14957.716685] watchdog: BUG: soft lockup - CPU#0 stuck for 2816s! [dbench:78683] [14957.723904] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14957.728685] watchdog: BUG: soft lockup - CPU#7 stuck for 11813s! [kworker/7:4:78808] [14957.779345] irq event stamp: 19143610 [14957.779346] hardirqs last enabled at (19143609): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14957.787093] Modules linked in: [14957.790749] hardirqs last disabled at (19143610): [] sysvec_apic_timer_interrupt+0xe/0x90 [14957.800907] ceph [14957.803957] softirqs last enabled at (19143548): [] __do_softirq+0x2be/0x371 [14957.813682] libceph [14957.815605] softirqs last disabled at (19143543): [] irq_exit_rcu+0xb0/0xd0 [14957.824291] dns_resolver fscache [14957.826475] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14957.834985] netfs veth [14957.838296] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14957.847241] nft_chain_nat xt_MASQUERADE [14957.849685] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [14957.857156] nf_nat nf_conntrack [14957.861082] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [14957.866560] nf_defrag_ipv6 nf_defrag_ipv4 [14957.869782] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14957.888521] nft_compat [14957.892611] [14957.892612] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14957.897827] nf_tables nfnetlink [14957.900271] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14957.901763] bridge stp [14957.908888] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14957.912111] llc [14957.919234] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14957.921681] xfs libcrc32c [14957.928803] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14957.930642] sunrpc intel_rapl_msr [14957.937764] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14957.940469] intel_rapl_common sb_edac [14957.947594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14957.950991] x86_pkg_temp_thermal coretemp [14957.959083] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14957.962830] kvm_intel kvm [14957.968567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14957.972658] irqbypass i2c_i801 [14957.979782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14957.982486] iTCO_wdt iTCO_vendor_support [14957.989611] Call Trace: [14957.989612] [14957.992748] i2c_smbus crct10dif_pclmul [14957.999872] ? watchdog_timer_fn+0x1b4/0x220 [14958.003875] joydev crc32_pclmul [14958.006320] ? __hrtimer_run_queues+0x1d3/0x360 [14958.008331] ipmi_ssif ghash_clmulni_intel [14958.012164] ? hrtimer_interrupt+0x104/0x240 [14958.016424] mei_me mei [14958.019649] ? __local_bh_enable+0x50/0x60 [14958.024173] lpc_ich sha512_ssse3 [14958.028267] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14958.032529] acpi_ipmi mfd_core [14958.034972] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14958.039081] wmi ipmi_si [14958.042391] [14958.047609] ipmi_devintf [14958.050746] [14958.055702] ipmi_msghandler ioatdma [14958.058235] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14958.060331] acpi_pad acpi_power_meter [14958.062954] ? queued_spin_lock_slowpath+0x2d/0x2e0 [14958.065047] ixgbe igb [14958.068618] do_raw_spin_lock+0xb5/0xc0 [14958.073922] mdio crc32c_intel [14958.077666] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14958.082535] nvme [14958.084896] try_get_cap_refs+0x159/0x880 [ceph] [14958.088724] ptp [14958.091781] __ceph_get_caps+0x4a1/0x760 [ceph] [14958.096558] nvme_core i2c_algo_bit [14958.098487] ? __file_remove_privs+0x89/0x110 [14958.103094] pps_core dca [14958.104932] ? wake_bit_function+0x70/0x70 [14958.109455] fuse [14958.112942] ? ceph_write_iter+0x260/0x770 [ceph] [14958.117290] irq event stamp: 42854034 [14958.119908] ceph_write_iter+0x260/0x770 [ceph] [14958.123999] hardirqs last enabled at (42854033): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14958.125926] ? lock_is_held_type+0xaa/0x120 [14958.130620] hardirqs last disabled at (42854034): [] sysvec_apic_timer_interrupt+0xe/0x90 [14958.134279] vfs_write+0x324/0x4f0 [14958.138801] softirqs last enabled at (42854032): [] __do_softirq+0x2be/0x371 [14958.148962] ksys_pwrite64+0x65/0xa0 [14958.153137] softirqs last disabled at (42854027): [] irq_exit_rcu+0xb0/0xd0 [14958.162860] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14958.166258] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14958.174942] do_syscall_64+0x38/0x80 [14958.178512] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14958.187024] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14958.192155] Workqueue: events delayed_work [ceph] [14958.201533] RIP: 0033:0x7f48f0eeb93b [14958.205103] [14958.205104] RIP: 0010:lock_acquire+0xf0/0x2b0 [14958.212574] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14958.217618] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [14958.222313] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14958.225886] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [14958.227376] ORIG_RAX: 0000000000000012 [14958.231727] [14958.231728] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [14958.250466] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14958.269201] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [14958.274421] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14958.279636] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [14958.283469] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14958.284957] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [14958.292088] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14958.299215] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [14958.306338] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14958.313463] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14958.320593] [14958.327712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14958.372208] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14958.379332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14958.386456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14958.393579] Call Trace: [14958.396023] [14958.398037] ? watchdog_timer_fn+0x1b4/0x220 [14958.402300] ? __hrtimer_run_queues+0x1d3/0x360 [14958.406827] ? hrtimer_interrupt+0x104/0x240 [14958.411095] ? __local_bh_enable+0x50/0x60 [14958.415190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14958.420412] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14958.425381] [14958.427476] [14958.429574] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14958.434883] ? lock_acquire+0xf0/0x2b0 [14958.438631] ? igrab+0x43/0x50 [14958.441682] ? lock_release+0xc5/0x260 [14958.445426] _raw_spin_lock+0x30/0x40 [14958.449092] ? igrab+0x1d/0x50 [14958.452141] igrab+0x1d/0x50 [14958.455020] ceph_check_delayed_caps+0xb7/0x270 [ceph] [14958.460169] delayed_work+0x1d8/0x280 [ceph] [14958.464450] ? process_scheduled_works+0x22a/0x510 [14958.469234] process_scheduled_works+0x2a4/0x510 [14958.473848] worker_thread+0x188/0x340 [14958.477596] ? rescuer_thread+0x310/0x310 [14958.481600] kthread+0xf5/0x130 [14958.484739] ? kthread_complete_and_exit+0x20/0x20 [14958.489523] ret_from_fork+0x31/0x40 [14958.493092] ? kthread_complete_and_exit+0x20/0x20 [14958.497876] ret_from_fork_asm+0x11/0x20 [14958.501800] [14985.717174] watchdog: BUG: soft lockup - CPU#0 stuck for 2842s! [dbench:78683] [14985.724392] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [14985.729173] watchdog: BUG: soft lockup - CPU#7 stuck for 11839s! [kworker/7:4:78808] [14985.779850] irq event stamp: 19201730 [14985.779851] hardirqs last enabled at (19201729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14985.787582] Modules linked in: [14985.791237] hardirqs last disabled at (19201730): [] sysvec_apic_timer_interrupt+0xe/0x90 [14985.801396] ceph [14985.804447] softirqs last enabled at (19201654): [] __do_softirq+0x2be/0x371 [14985.814171] libceph [14985.816094] softirqs last disabled at (19201649): [] irq_exit_rcu+0xb0/0xd0 [14985.824779] dns_resolver fscache [14985.826965] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14985.835474] netfs [14985.838785] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14985.847729] veth nft_chain_nat [14985.849740] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [14985.857209] xt_MASQUERADE nf_nat [14985.860349] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [14985.865825] nf_conntrack nf_defrag_ipv6 [14985.869136] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [14985.887874] nf_defrag_ipv4 nft_compat [14985.891789] [14985.891790] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [14985.897008] nf_tables [14985.900751] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [14985.902242] nfnetlink [14985.909368] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [14985.911725] bridge stp [14985.918865] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [14985.921223] llc xfs [14985.928347] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [14985.930791] libcrc32c [14985.937915] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [14985.940098] sunrpc intel_rapl_msr [14985.947223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14985.949582] intel_rapl_common [14985.957657] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [14985.961057] sb_edac x86_pkg_temp_thermal [14985.966792] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14985.969851] coretemp kvm_intel [14985.976977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14985.980980] kvm irqbypass [14985.988104] Call Trace: [14985.988107] [14985.991241] i2c_i801 iTCO_wdt [14985.998368] ? watchdog_timer_fn+0x1b4/0x220 [14986.001071] iTCO_vendor_support i2c_smbus [14986.003515] ? __hrtimer_run_queues+0x1d3/0x360 [14986.005525] crct10dif_pclmul [14986.008579] ? hrtimer_interrupt+0x104/0x240 [14986.012840] joydev crc32_pclmul ipmi_ssif [14986.016948] ? __local_bh_enable+0x50/0x60 [14986.021473] ghash_clmulni_intel mei_me [14986.024438] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14986.028699] mei lpc_ich [14986.032792] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14986.036882] sha512_ssse3 acpi_ipmi [14986.040712] [14986.045929] mfd_core wmi [14986.048460] [14986.053418] ipmi_si ipmi_devintf [14986.056904] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14986.059000] ipmi_msghandler [14986.061621] ? queued_spin_lock_slowpath+0x16/0x2e0 [14986.063715] ioatdma acpi_pad [14986.067027] do_raw_spin_lock+0xb5/0xc0 [14986.072329] acpi_power_meter [14986.075209] __cap_delay_requeue+0xa7/0x1d0 [ceph] [14986.080077] ixgbe igb [14986.083044] try_get_cap_refs+0x159/0x880 [ceph] [14986.086872] mdio crc32c_intel [14986.089857] __ceph_get_caps+0x4a1/0x760 [ceph] [14986.094638] nvme [14986.096999] ? __file_remove_privs+0x89/0x110 [14986.101606] ptp nvme_core [14986.104657] ? wake_bit_function+0x70/0x70 [14986.109180] i2c_algo_bit pps_core [14986.111109] ? ceph_write_iter+0x260/0x770 [ceph] [14986.115457] dca [14986.118160] ceph_write_iter+0x260/0x770 [ceph] [14986.122249] fuse [14986.125653] ? lock_is_held_type+0xaa/0x120 [14986.130345] irq event stamp: 43075604 [14986.132184] vfs_write+0x324/0x4f0 [14986.136706] hardirqs last enabled at (43075603): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14986.138635] ksys_pwrite64+0x65/0xa0 [14986.142809] hardirqs last disabled at (43075604): [] sysvec_apic_timer_interrupt+0xe/0x90 [14986.146466] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [14986.149862] softirqs last enabled at (43075602): [] __do_softirq+0x2be/0x371 [14986.160021] do_syscall_64+0x38/0x80 [14986.163590] softirqs last disabled at (43075597): [] irq_exit_rcu+0xb0/0xd0 [14986.173315] entry_SYSCALL_64_after_hwframe+0x63/0xcd [14986.178446] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [14986.187131] RIP: 0033:0x7f48f0eeb93b [14986.190701] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [14986.199213] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [14986.204256] Workqueue: events delayed_work [ceph] [14986.213632] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [14986.217204] [14986.217205] RIP: 0010:ceph_check_delayed_caps+0x85/0x270 [ceph] [14986.224675] ORIG_RAX: 0000000000000012 [14986.243413] Code: 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 <78> 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 [14986.248110] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [14986.253328] RSP: 0018:ffffc90002117db8 EFLAGS: 00000216 [14986.254818] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [14986.260727] [14986.260728] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 0000000100e01080 [14986.264575] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [14986.283313] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [14986.290438] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [14986.295656] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [14986.302779] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [14986.304271] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [14986.311401] [14986.318519] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [14986.363449] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [14986.371524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14986.377264] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [14986.384386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14986.391510] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [14986.398635] Call Trace: [14986.401081] [14986.403091] ? watchdog_timer_fn+0x1b4/0x220 [14986.407354] ? __hrtimer_run_queues+0x1d3/0x360 [14986.411881] ? hrtimer_interrupt+0x104/0x240 [14986.416141] ? __local_bh_enable+0x50/0x60 [14986.420235] ? __sysvec_apic_timer_interrupt+0x61/0x160 [14986.425452] ? sysvec_apic_timer_interrupt+0x6d/0x90 [14986.430407] [14986.432505] [14986.434602] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [14986.439911] ? ceph_check_delayed_caps+0x85/0x270 [ceph] [14986.445230] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [14986.450543] delayed_work+0x1d8/0x280 [ceph] [14986.454859] ? process_scheduled_works+0x22a/0x510 [14986.459651] process_scheduled_works+0x2a4/0x510 [14986.464266] worker_thread+0x188/0x340 [14986.468014] ? rescuer_thread+0x310/0x310 [14986.472016] kthread+0xf5/0x130 [14986.475157] ? kthread_complete_and_exit+0x20/0x20 [14986.479947] ret_from_fork+0x31/0x40 [14986.483529] ? kthread_complete_and_exit+0x20/0x20 [14986.488312] ret_from_fork_asm+0x11/0x20 [14986.492243] [15013.717661] watchdog: BUG: soft lockup - CPU#0 stuck for 2868s! [dbench:78683] [15013.724880] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15013.729661] watchdog: BUG: soft lockup - CPU#7 stuck for 11865s! [kworker/7:4:78808] [15013.780336] irq event stamp: 19259870 [15013.780337] hardirqs last enabled at (19259869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15013.788069] Modules linked in: [15013.791725] hardirqs last disabled at (19259870): [] sysvec_apic_timer_interrupt+0xe/0x90 [15013.801884] ceph [15013.804933] softirqs last enabled at (19259774): [] __do_softirq+0x2be/0x371 [15013.814658] libceph [15013.816582] softirqs last disabled at (19259769): [] irq_exit_rcu+0xb0/0xd0 [15013.825267] dns_resolver [15013.827450] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15013.835960] fscache netfs [15013.838580] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15013.847523] veth nft_chain_nat [15013.850228] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15013.857697] xt_MASQUERADE nf_nat [15013.860835] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15013.866312] nf_conntrack nf_defrag_ipv6 [15013.869640] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15013.888379] nf_defrag_ipv4 nft_compat [15013.892295] [15013.892296] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15013.897513] nf_tables [15013.901256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15013.902747] nfnetlink bridge [15013.909871] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15013.912229] stp llc [15013.919352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15013.922316] xfs libcrc32c [15013.929458] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15013.931644] sunrpc [15013.938767] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15013.941471] intel_rapl_msr intel_rapl_common [15013.948594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15013.950693] sb_edac x86_pkg_temp_thermal [15013.958770] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15013.963120] coretemp kvm_intel [15013.968857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15013.972863] kvm irqbypass [15013.979986] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15013.983123] i2c_i801 iTCO_wdt [15013.990248] Call Trace: [15013.990250] [15013.992951] iTCO_vendor_support i2c_smbus [15014.000077] ? watchdog_timer_fn+0x1b4/0x220 [15014.003126] crct10dif_pclmul [15014.005571] ? __hrtimer_run_queues+0x1d3/0x360 [15014.007583] joydev crc32_pclmul [15014.011675] ? hrtimer_interrupt+0x104/0x240 [15014.015935] ipmi_ssif ghash_clmulni_intel [15014.018901] ? __local_bh_enable+0x50/0x60 [15014.023424] mei_me mei [15014.026652] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15014.030912] lpc_ich [15014.035005] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15014.039094] sha512_ssse3 acpi_ipmi [15014.041539] [15014.046756] mfd_core wmi [15014.048940] [15014.048942] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15014.053898] ipmi_si ipmi_devintf [15014.057385] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15014.059477] ipmi_msghandler [15014.062098] do_raw_spin_lock+0xb5/0xc0 [15014.064192] ioatdma [15014.069497] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15014.072807] acpi_pad acpi_power_meter ixgbe [15014.077682] try_get_cap_refs+0x159/0x880 [ceph] [15014.080556] igb mdio [15014.084393] __ceph_get_caps+0x4a1/0x760 [ceph] [15014.086570] crc32c_intel nvme [15014.091358] ? __file_remove_privs+0x89/0x110 [15014.095620] ptp nvme_core [15014.100231] ? wake_bit_function+0x70/0x70 [15014.102500] i2c_algo_bit pps_core [15014.107029] ? ceph_write_iter+0x260/0x770 [ceph] [15014.110076] dca fuse [15014.114426] ceph_write_iter+0x260/0x770 [ceph] [15014.117129] [15014.117130] irq event stamp: 43297320 [15014.121224] ? lock_is_held_type+0xaa/0x120 [15014.124619] hardirqs last enabled at (43297319): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15014.129319] vfs_write+0x324/0x4f0 [15014.131586] hardirqs last disabled at (43297320): [] sysvec_apic_timer_interrupt+0xe/0x90 [15014.136115] ksys_pwrite64+0x65/0xa0 [15014.137601] softirqs last enabled at (43297318): [] __do_softirq+0x2be/0x371 [15014.141260] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15014.145435] softirqs last disabled at (43297313): [] irq_exit_rcu+0xb0/0xd0 [15014.155594] do_syscall_64+0x38/0x80 [15014.158991] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15014.168716] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15014.172287] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15014.180971] RIP: 0033:0x7f48f0eeb93b [15014.186102] Workqueue: events delayed_work [ceph] [15014.194613] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15014.198182] [15014.198183] RIP: 0010:igrab+0x15/0x50 [15014.207561] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15014.212604] Code: 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb <48> 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 [15014.220076] ORIG_RAX: 0000000000000012 [15014.223646] RSP: 0018:ffffc90002117da0 EFLAGS: 00000286 [15014.228344] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15014.247081] [15014.247082] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [15014.248570] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15014.252228] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [15014.257445] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15014.276185] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15014.280014] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15014.285232] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15014.292356] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15014.293848] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15014.300976] [15014.308095] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15014.368226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15014.373963] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15014.381088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15014.388212] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15014.395345] Call Trace: [15014.397789] [15014.399800] ? watchdog_timer_fn+0x1b4/0x220 [15014.404068] ? __hrtimer_run_queues+0x1d3/0x360 [15014.408598] ? hrtimer_interrupt+0x104/0x240 [15014.412860] ? __local_bh_enable+0x50/0x60 [15014.416953] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15014.422168] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15014.427126] [15014.429223] [15014.431320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15014.436628] ? igrab+0x15/0x50 [15014.439685] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15014.444834] delayed_work+0x1d8/0x280 [ceph] [15014.449117] ? process_scheduled_works+0x22a/0x510 [15014.453905] process_scheduled_works+0x2a4/0x510 [15014.458523] worker_thread+0x188/0x340 [15014.462269] ? rescuer_thread+0x310/0x310 [15014.466274] kthread+0xf5/0x130 [15014.469414] ? kthread_complete_and_exit+0x20/0x20 [15014.474204] ret_from_fork+0x31/0x40 [15014.477784] ? kthread_complete_and_exit+0x20/0x20 [15014.482567] ret_from_fork_asm+0x11/0x20 [15014.486492] [15041.718150] watchdog: BUG: soft lockup - CPU#0 stuck for 2894s! [dbench:78683] [15041.725368] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15041.730149] watchdog: BUG: soft lockup - CPU#7 stuck for 11891s! [kworker/7:4:78808] [15041.780824] irq event stamp: 19318018 [15041.780825] hardirqs last enabled at (19318017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15041.788556] Modules linked in: [15041.792212] hardirqs last disabled at (19318018): [] sysvec_apic_timer_interrupt+0xe/0x90 [15041.802371] ceph [15041.805422] softirqs last enabled at (19317902): [] __do_softirq+0x2be/0x371 [15041.815146] libceph [15041.817070] softirqs last disabled at (19317897): [] irq_exit_rcu+0xb0/0xd0 [15041.825752] dns_resolver fscache [15041.827938] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15041.836450] netfs veth [15041.839759] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15041.848703] nft_chain_nat [15041.851147] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15041.858617] xt_MASQUERADE nf_nat [15041.861323] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15041.866799] nf_conntrack nf_defrag_ipv6 [15041.870128] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15041.888865] nf_defrag_ipv4 nft_compat [15041.892782] [15041.898000] nf_tables nfnetlink [15041.901745] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15041.903234] bridge stp [15041.906459] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15041.913582] llc [15041.916026] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15041.923150] xfs libcrc32c [15041.924989] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15041.932113] sunrpc intel_rapl_msr [15041.934817] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15041.941960] intel_rapl_common [15041.945355] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15041.952481] sb_edac x86_pkg_temp_thermal [15041.955548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15041.963624] coretemp kvm_intel [15041.967629] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15041.973366] kvm irqbypass [15041.976504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15041.983627] i2c_i801 iTCO_wdt [15041.986332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15041.993457] iTCO_vendor_support [15041.996508] Call Trace: [15041.996511] [15042.003631] i2c_smbus crct10dif_pclmul [15042.006857] ? watchdog_timer_fn+0x1b4/0x220 [15042.009299] joydev crc32_pclmul [15042.011312] ? __hrtimer_run_queues+0x1d3/0x360 [15042.015140] ipmi_ssif ghash_clmulni_intel [15042.019408] ? hrtimer_interrupt+0x104/0x240 [15042.022630] mei_me mei [15042.027153] ? __local_bh_enable+0x50/0x60 [15042.031244] lpc_ich sha512_ssse3 [15042.035510] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15042.037952] acpi_ipmi mfd_core [15042.042042] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15042.045354] wmi [15042.050570] [15042.053709] ipmi_si ipmi_devintf [15042.058666] [15042.058668] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15042.060503] ipmi_msghandler [15042.062606] ? queued_spin_lock_slowpath+0x18/0x2e0 [15042.065911] ioatdma acpi_pad [15042.068012] do_raw_spin_lock+0xb5/0xc0 [15042.073312] acpi_power_meter ixgbe [15042.076192] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15042.081060] igb mdio [15042.084028] try_get_cap_refs+0x159/0x880 [ceph] [15042.087855] crc32c_intel [15042.091347] __ceph_get_caps+0x4a1/0x760 [ceph] [15042.096124] nvme ptp [15042.098398] ? __file_remove_privs+0x89/0x110 [15042.103004] nvme_core i2c_algo_bit [15042.105622] ? wake_bit_function+0x70/0x70 [15042.110147] pps_core dca [15042.112420] ? ceph_write_iter+0x260/0x770 [ceph] [15042.116769] fuse [15042.120252] ceph_write_iter+0x260/0x770 [ceph] [15042.124342] [15042.124343] irq event stamp: 43519534 [15042.126963] ? lock_is_held_type+0xaa/0x120 [15042.131657] hardirqs last enabled at (43519533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15042.133584] vfs_write+0x324/0x4f0 [15042.138107] hardirqs last disabled at (43519534): [] sysvec_apic_timer_interrupt+0xe/0x90 [15042.139602] ksys_pwrite64+0x65/0xa0 [15042.143253] softirqs last enabled at (43519532): [] __do_softirq+0x2be/0x371 [15042.147432] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15042.157589] softirqs last disabled at (43519527): [] irq_exit_rcu+0xb0/0xd0 [15042.160988] do_syscall_64+0x38/0x80 [15042.170712] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15042.174282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15042.182966] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15042.188097] RIP: 0033:0x7f48f0eeb93b [15042.196608] Workqueue: events delayed_work [ceph] [15042.200180] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15042.209555] [15042.209556] RIP: 0010:lock_acquire+0xf0/0x2b0 [15042.214600] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15042.222070] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15042.225641] ORIG_RAX: 0000000000000012 [15042.230337] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15042.249076] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15042.250566] [15042.250567] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15042.254917] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15042.260133] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15042.278873] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15042.282702] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15042.287920] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15042.295044] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15042.296552] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15042.303678] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15042.310807] [15042.317925] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15042.370941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15042.376677] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15042.383803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15042.390927] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15042.398050] Call Trace: [15042.400495] [15042.402507] ? watchdog_timer_fn+0x1b4/0x220 [15042.406770] ? __hrtimer_run_queues+0x1d3/0x360 [15042.411298] ? hrtimer_interrupt+0x104/0x240 [15042.415567] ? __local_bh_enable+0x50/0x60 [15042.419659] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15042.424884] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15042.429854] [15042.431954] [15042.434052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15042.439362] ? lock_acquire+0xf0/0x2b0 [15042.443110] ? igrab+0x43/0x50 [15042.446161] ? lock_release+0xc5/0x260 [15042.449907] _raw_spin_lock+0x30/0x40 [15042.453570] ? igrab+0x1d/0x50 [15042.456621] igrab+0x1d/0x50 [15042.459500] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15042.464649] delayed_work+0x1d8/0x280 [ceph] [15042.468932] ? process_scheduled_works+0x22a/0x510 [15042.473721] process_scheduled_works+0x2a4/0x510 [15042.478337] worker_thread+0x188/0x340 [15042.482086] ? rescuer_thread+0x310/0x310 [15042.486096] kthread+0xf5/0x130 [15042.489236] ? kthread_complete_and_exit+0x20/0x20 [15042.494027] ret_from_fork+0x31/0x40 [15042.497608] ? kthread_complete_and_exit+0x20/0x20 [15042.502390] ret_from_fork_asm+0x11/0x20 [15042.506314] [15069.718637] watchdog: BUG: soft lockup - CPU#0 stuck for 2920s! [dbench:78683] [15069.725852] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15069.730637] watchdog: BUG: soft lockup - CPU#7 stuck for 11917s! [kworker/7:4:78808] [15069.781312] irq event stamp: 19376184 [15069.781313] hardirqs last enabled at (19376183): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15069.789043] Modules linked in: [15069.792699] hardirqs last disabled at (19376184): [] sysvec_apic_timer_interrupt+0xe/0x90 [15069.802858] ceph [15069.805909] softirqs last enabled at (19376088): [] __do_softirq+0x2be/0x371 [15069.815633] libceph [15069.817557] softirqs last disabled at (19376083): [] irq_exit_rcu+0xb0/0xd0 [15069.826240] dns_resolver fscache [15069.828425] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15069.836935] netfs veth [15069.840246] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15069.849190] nft_chain_nat xt_MASQUERADE [15069.851635] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15069.859106] nf_nat nf_conntrack [15069.863032] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15069.868508] nf_defrag_ipv6 nf_defrag_ipv4 [15069.871734] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15069.890471] nft_compat [15069.894562] [15069.894563] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15069.899780] nf_tables nfnetlink [15069.902222] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15069.903713] bridge stp [15069.910838] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15069.914062] llc xfs [15069.921186] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15069.923631] libcrc32c sunrpc [15069.930755] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15069.932939] intel_rapl_msr intel_rapl_common [15069.940061] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15069.943036] sb_edac x86_pkg_temp_thermal [15069.950157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15069.954509] coretemp kvm_intel [15069.962588] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15069.966592] kvm irqbypass [15069.972327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15069.975465] i2c_i801 iTCO_wdt [15069.982591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15069.985295] iTCO_vendor_support [15069.992419] Call Trace: [15069.992420] [15069.995469] i2c_smbus crct10dif_pclmul [15070.002593] ? watchdog_timer_fn+0x1b4/0x220 [15070.005816] joydev crc32_pclmul [15070.008263] ? __hrtimer_run_queues+0x1d3/0x360 [15070.010272] ipmi_ssif [15070.014106] ? hrtimer_interrupt+0x104/0x240 [15070.018366] ghash_clmulni_intel mei_me [15070.021592] ? __local_bh_enable+0x50/0x60 [15070.026116] mei lpc_ich [15070.028474] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15070.032735] sha512_ssse3 acpi_ipmi [15070.036568] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15070.040657] mfd_core wmi [15070.043188] [15070.048406] ipmi_si ipmi_devintf [15070.051890] [15070.056846] ipmi_msghandler ioatdma [15070.059466] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15070.061561] acpi_pad acpi_power_meter [15070.064876] ? queued_spin_lock_slowpath+0x18/0x2e0 [15070.066971] ixgbe igb [15070.070543] do_raw_spin_lock+0xb5/0xc0 [15070.075844] mdio crc32c_intel nvme [15070.079591] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15070.084459] ptp nvme_core [15070.086820] try_get_cap_refs+0x159/0x880 [ceph] [15070.090648] i2c_algo_bit pps_core [15070.094138] __ceph_get_caps+0x4a1/0x760 [ceph] [15070.098915] dca [15070.101624] ? __file_remove_privs+0x89/0x110 [15070.106230] fuse [15070.109628] ? wake_bit_function+0x70/0x70 [15070.114152] irq event stamp: 43741300 [15070.115994] ? ceph_write_iter+0x260/0x770 [ceph] [15070.120341] hardirqs last enabled at (43741299): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15070.122265] ceph_write_iter+0x260/0x770 [ceph] [15070.126355] hardirqs last disabled at (43741300): [] sysvec_apic_timer_interrupt+0xe/0x90 [15070.130037] ? lock_is_held_type+0xaa/0x120 [15070.134727] softirqs last enabled at (43741298): [] __do_softirq+0x2be/0x371 [15070.144887] vfs_write+0x324/0x4f0 [15070.149409] softirqs last disabled at (43741293): [] irq_exit_rcu+0xb0/0xd0 [15070.159138] ksys_pwrite64+0x65/0xa0 [15070.163325] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15070.172012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15070.175428] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15070.183940] do_syscall_64+0x38/0x80 [15070.187510] Workqueue: events delayed_work [ceph] [15070.196887] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15070.202035] [15070.209504] RIP: 0033:0x7f48f0eeb93b [15070.213075] RIP: 0010:lock_acquire+0xf0/0x2b0 [15070.217775] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15070.222818] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15070.224308] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15070.227897] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15070.232247] ORIG_RAX: 0000000000000012 [15070.250985] [15070.250985] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15070.269721] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15070.274940] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15070.280157] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15070.283987] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15070.285478] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15070.292603] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15070.299726] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15070.306849] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15070.313975] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15070.321099] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15070.328228] [15070.335348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15070.372720] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15070.379842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15070.386968] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15070.394092] Call Trace: [15070.396535] [15070.398547] ? watchdog_timer_fn+0x1b4/0x220 [15070.402811] ? __hrtimer_run_queues+0x1d3/0x360 [15070.407338] ? hrtimer_interrupt+0x104/0x240 [15070.411606] ? __local_bh_enable+0x50/0x60 [15070.415700] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15070.420914] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15070.425874] [15070.427970] [15070.430069] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15070.435376] ? lock_acquire+0xf0/0x2b0 [15070.439125] ? igrab+0x43/0x50 [15070.442177] ? lock_release+0xc5/0x260 [15070.445921] _raw_spin_lock+0x30/0x40 [15070.449585] ? igrab+0x1d/0x50 [15070.452636] igrab+0x1d/0x50 [15070.455514] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15070.460663] delayed_work+0x1d8/0x280 [ceph] [15070.464946] ? process_scheduled_works+0x22a/0x510 [15070.469735] process_scheduled_works+0x2a4/0x510 [15070.474353] worker_thread+0x188/0x340 [15070.478101] ? rescuer_thread+0x310/0x310 [15070.482113] kthread+0xf5/0x130 [15070.485250] ? kthread_complete_and_exit+0x20/0x20 [15070.490046] ret_from_fork+0x31/0x40 [15070.493622] ? kthread_complete_and_exit+0x20/0x20 [15070.498407] ret_from_fork_asm+0x11/0x20 [15070.502330] [15097.719126] watchdog: BUG: soft lockup - CPU#0 stuck for 2946s! [dbench:78683] [15097.726342] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15097.731125] watchdog: BUG: soft lockup - CPU#7 stuck for 11943s! [kworker/7:4:78808] [15097.781806] irq event stamp: 19434290 [15097.781807] hardirqs last enabled at (19434289): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15097.789538] Modules linked in: [15097.793197] hardirqs last disabled at (19434290): [] sysvec_apic_timer_interrupt+0xe/0x90 [15097.803354] ceph [15097.806403] softirqs last enabled at (19434224): [] __do_softirq+0x2be/0x371 [15097.816129] libceph [15097.818052] softirqs last disabled at (19434219): [] irq_exit_rcu+0xb0/0xd0 [15097.826736] dns_resolver fscache [15097.828920] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15097.837432] netfs [15097.840742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15097.849685] veth nft_chain_nat [15097.851698] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15097.859167] xt_MASQUERADE nf_nat [15097.862306] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15097.867783] nf_conntrack nf_defrag_ipv6 [15097.871110] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15097.889849] nf_defrag_ipv4 nft_compat [15097.893764] [15097.898982] nf_tables nfnetlink [15097.902726] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15097.904217] bridge stp [15097.907442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15097.914567] llc xfs [15097.917010] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15097.924133] libcrc32c sunrpc [15097.926318] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15097.933441] intel_rapl_msr intel_rapl_common [15097.936405] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15097.943531] sb_edac x86_pkg_temp_thermal [15097.947881] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15097.955004] coretemp kvm_intel [15097.959009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15097.967087] kvm irqbypass [15097.970225] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15097.975962] i2c_i801 iTCO_wdt [15097.978665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15097.985790] iTCO_vendor_support i2c_smbus [15097.988840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15097.995965] crct10dif_pclmul joydev [15098.000055] Call Trace: [15098.000057] [15098.007178] crc32_pclmul [15098.010753] ? watchdog_timer_fn+0x1b4/0x220 [15098.013194] ipmi_ssif ghash_clmulni_intel [15098.015205] ? __hrtimer_run_queues+0x1d3/0x360 [15098.017821] mei_me mei [15098.022089] ? hrtimer_interrupt+0x104/0x240 [15098.026177] lpc_ich sha512_ssse3 [15098.030701] ? __local_bh_enable+0x50/0x60 [15098.033144] acpi_ipmi mfd_core [15098.037412] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15098.040719] wmi ipmi_si [15098.044812] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15098.047947] ipmi_devintf ipmi_msghandler [15098.053166] [15098.055695] ioatdma acpi_pad [15098.060654] [15098.060655] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15098.064659] acpi_power_meter ixgbe [15098.066760] ? queued_spin_lock_slowpath+0x16/0x2e0 [15098.069720] igb [15098.071819] do_raw_spin_lock+0xb5/0xc0 [15098.077121] mdio crc32c_intel [15098.080606] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15098.085476] nvme [15098.087316] try_get_cap_refs+0x159/0x880 [ceph] [15098.091143] ptp nvme_core [15098.094200] __ceph_get_caps+0x4a1/0x760 [ceph] [15098.098980] i2c_algo_bit pps_core [15098.100907] ? __file_remove_privs+0x89/0x110 [15098.105523] dca fuse [15098.108226] ? wake_bit_function+0x70/0x70 [15098.112750] [15098.112751] irq event stamp: 43963632 [15098.116150] ? ceph_write_iter+0x260/0x770 [ceph] [15098.120498] hardirqs last enabled at (43963631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15098.122804] ceph_write_iter+0x260/0x770 [ceph] [15098.126893] hardirqs last disabled at (43963632): [] sysvec_apic_timer_interrupt+0xe/0x90 [15098.128387] ? lock_is_held_type+0xaa/0x120 [15098.132041] softirqs last enabled at (43963630): [] __do_softirq+0x2be/0x371 [15098.136741] vfs_write+0x324/0x4f0 [15098.146897] softirqs last disabled at (43963625): [] irq_exit_rcu+0xb0/0xd0 [15098.151426] ksys_pwrite64+0x65/0xa0 [15098.161146] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15098.165323] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15098.174006] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15098.174007] Workqueue: events delayed_work [ceph] [15098.177407] do_syscall_64+0x38/0x80 [15098.185916] [15098.189486] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15098.198865] RIP: 0010:lock_acquire+0xf0/0x2b0 [15098.203994] RIP: 0033:0x7f48f0eeb93b [15098.211466] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15098.216163] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15098.219733] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15098.221225] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15098.226269] [15098.230619] ORIG_RAX: 0000000000000012 [15098.234189] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15098.252928] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15098.271666] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15098.276882] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15098.282099] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15098.283589] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15098.287422] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15098.294545] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15098.301670] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15098.308794] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15098.315934] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15098.323064] [15098.330182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15098.374678] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15098.381810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15098.388937] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15098.396060] Call Trace: [15098.398506] [15098.400534] ? watchdog_timer_fn+0x1b4/0x220 [15098.404797] ? __hrtimer_run_queues+0x1d3/0x360 [15098.409342] ? hrtimer_interrupt+0x104/0x240 [15098.413611] ? __local_bh_enable+0x50/0x60 [15098.417704] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15098.422928] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15098.427885] [15098.429982] [15098.432079] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15098.437387] ? lock_acquire+0xf0/0x2b0 [15098.441138] ? igrab+0x43/0x50 [15098.444189] ? lock_release+0xc5/0x260 [15098.447934] _raw_spin_lock+0x30/0x40 [15098.451598] ? igrab+0x1d/0x50 [15098.454648] igrab+0x1d/0x50 [15098.457533] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15098.462685] delayed_work+0x1d8/0x280 [ceph] [15098.466966] ? process_scheduled_works+0x22a/0x510 [15098.471755] process_scheduled_works+0x2a4/0x510 [15098.476372] worker_thread+0x188/0x340 [15098.480121] ? rescuer_thread+0x310/0x310 [15098.484133] kthread+0xf5/0x130 [15098.487272] ? kthread_complete_and_exit+0x20/0x20 [15098.492055] ret_from_fork+0x31/0x40 [15098.495635] ? kthread_complete_and_exit+0x20/0x20 [15098.500419] ret_from_fork_asm+0x11/0x20 [15098.504350] [15125.719613] watchdog: BUG: soft lockup - CPU#0 stuck for 2972s! [dbench:78683] [15125.726828] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15125.731613] watchdog: BUG: soft lockup - CPU#7 stuck for 11969s! [kworker/7:4:78808] [15125.782285] irq event stamp: 19492428 [15125.782286] hardirqs last enabled at (19492427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15125.790016] Modules linked in: [15125.793674] hardirqs last disabled at (19492428): [] sysvec_apic_timer_interrupt+0xe/0x90 [15125.803830] ceph [15125.806882] softirqs last enabled at (19492352): [] __do_softirq+0x2be/0x371 [15125.816605] libceph [15125.818531] softirqs last disabled at (19492347): [] irq_exit_rcu+0xb0/0xd0 [15125.827215] dns_resolver fscache [15125.829398] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15125.837909] netfs veth [15125.841221] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15125.841222] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15125.850165] nft_chain_nat xt_MASQUERADE [15125.852609] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15125.860078] nf_nat [15125.865557] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15125.869475] nf_conntrack nf_defrag_ipv6 [15125.888213] [15125.888214] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15125.890309] nf_defrag_ipv4 nft_compat [15125.895527] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15125.899443] nf_tables nfnetlink [15125.900934] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15125.908059] bridge stp [15125.911802] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15125.918925] llc xfs [15125.922150] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15125.929290] libcrc32c sunrpc [15125.931736] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15125.938859] intel_rapl_msr [15125.941043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15125.948168] intel_rapl_common sb_edac [15125.951131] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15125.959210] x86_pkg_temp_thermal [15125.962008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15125.967747] coretemp kvm_intel [15125.971492] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15125.978616] kvm irqbypass [15125.981926] Call Trace: [15125.981928] [15125.989050] i2c_i801 [15125.992189] ? watchdog_timer_fn+0x1b4/0x220 [15125.999313] iTCO_wdt iTCO_vendor_support [15126.002017] ? __hrtimer_run_queues+0x1d3/0x360 [15126.004460] i2c_smbus crct10dif_pclmul [15126.006473] ? hrtimer_interrupt+0x104/0x240 [15126.008741] joydev crc32_pclmul [15126.013013] ? __local_bh_enable+0x50/0x60 [15126.017017] ipmi_ssif ghash_clmulni_intel [15126.021546] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15126.025372] mei_me mei [15126.029638] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15126.032860] lpc_ich sha512_ssse3 [15126.036951] [15126.041042] acpi_ipmi [15126.046261] [15126.048703] mfd_core wmi [15126.053663] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15126.056971] ipmi_si ipmi_devintf [15126.059073] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15126.061426] ipmi_msghandler ioatdma [15126.063525] do_raw_spin_lock+0xb5/0xc0 [15126.066142] acpi_pad acpi_power_meter [15126.071447] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15126.074755] ixgbe igb [15126.079631] try_get_cap_refs+0x159/0x880 [ceph] [15126.083199] mdio crc32c_intel [15126.087035] __ceph_get_caps+0x4a1/0x760 [ceph] [15126.090772] nvme ptp [15126.095561] ? __file_remove_privs+0x89/0x110 [15126.097914] nvme_core [15126.102524] ? wake_bit_function+0x70/0x70 [15126.105576] i2c_algo_bit pps_core [15126.110103] ? ceph_write_iter+0x260/0x770 [ceph] [15126.112369] dca fuse [15126.116723] ceph_write_iter+0x260/0x770 [ceph] [15126.119077] [15126.119078] irq event stamp: 44185856 [15126.123172] ? lock_is_held_type+0xaa/0x120 [15126.126568] hardirqs last enabled at (44185855): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15126.131267] vfs_write+0x324/0x4f0 [15126.133536] hardirqs last disabled at (44185856): [] sysvec_apic_timer_interrupt+0xe/0x90 [15126.138063] ksys_pwrite64+0x65/0xa0 [15126.139548] softirqs last enabled at (44185854): [] __do_softirq+0x2be/0x371 [15126.143209] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15126.147385] softirqs last disabled at (44185849): [] irq_exit_rcu+0xb0/0xd0 [15126.157544] do_syscall_64+0x38/0x80 [15126.160939] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15126.170664] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15126.174234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15126.182920] RIP: 0033:0x7f48f0eeb93b [15126.188049] Workqueue: events delayed_work [ceph] [15126.196559] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15126.200131] [15126.200132] RIP: 0010:lock_acquire+0xf0/0x2b0 [15126.209508] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15126.214553] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15126.222022] ORIG_RAX: 0000000000000012 [15126.225595] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15126.230305] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15126.249045] [15126.249046] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15126.250536] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15126.254887] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15126.260106] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15126.278841] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15126.282673] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15126.287890] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15126.295016] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15126.296506] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15126.303636] [15126.310754] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15126.370893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15126.376632] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15126.383756] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15126.390878] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15126.398019] Call Trace: [15126.400465] [15126.402477] ? watchdog_timer_fn+0x1b4/0x220 [15126.406740] ? __hrtimer_run_queues+0x1d3/0x360 [15126.411267] ? hrtimer_interrupt+0x104/0x240 [15126.415536] ? __local_bh_enable+0x50/0x60 [15126.419630] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15126.424853] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15126.429811] [15126.431907] [15126.434018] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15126.439331] ? lock_acquire+0xf0/0x2b0 [15126.443082] ? igrab+0x43/0x50 [15126.446132] ? lock_release+0xc5/0x260 [15126.449876] _raw_spin_lock+0x30/0x40 [15126.453530] ? igrab+0x1d/0x50 [15126.456583] igrab+0x1d/0x50 [15126.459460] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15126.464611] delayed_work+0x1d8/0x280 [ceph] [15126.468892] ? process_scheduled_works+0x22a/0x510 [15126.473680] process_scheduled_works+0x2a4/0x510 [15126.478312] worker_thread+0x188/0x340 [15126.482063] ? rescuer_thread+0x310/0x310 [15126.486066] kthread+0xf5/0x130 [15126.489206] ? kthread_complete_and_exit+0x20/0x20 [15126.493989] ret_from_fork+0x31/0x40 [15126.497585] ? kthread_complete_and_exit+0x20/0x20 [15126.502370] ret_from_fork_asm+0x11/0x20 [15126.506316] [15128.482028] rcu: INFO: rcu_preempt self-detected stall on CPU [15128.487785] rcu: 7-....: (3103663 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/3102477 fqs=1537674 [15128.498211] rcu: (t=3191815 jiffies g=1309609 q=170866 ncpus=8) [15128.504209] Sending NMI from CPU 7 to CPUs 0: [15128.508559] NMI backtrace for cpu 0 [15128.508560] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15128.508562] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15128.508563] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15128.508566] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15128.508568] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15128.508569] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15128.508570] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15128.508571] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15128.508572] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15128.508572] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15128.508573] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15128.508575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15128.508575] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15128.508576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15128.508577] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15128.508578] Call Trace: [15128.508579] [15128.508580] ? nmi_cpu_backtrace+0xad/0x120 [15128.508583] ? nmi_cpu_backtrace_handler+0x11/0x20 [15128.508585] ? nmi_handle+0xd4/0x260 [15128.508589] ? default_do_nmi+0x49/0xf0 [15128.508591] ? exc_nmi+0x11e/0x150 [15128.508592] ? end_repeat_nmi+0x16/0x67 [15128.508596] ? queued_spin_lock_slowpath+0x18/0x2e0 [15128.508599] ? queued_spin_lock_slowpath+0x18/0x2e0 [15128.508601] ? queued_spin_lock_slowpath+0x18/0x2e0 [15128.508603] [15128.508604] [15128.508605] do_raw_spin_lock+0xb5/0xc0 [15128.508607] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15128.508621] try_get_cap_refs+0x159/0x880 [ceph] [15128.508636] __ceph_get_caps+0x4a1/0x760 [ceph] [15128.508649] ? __file_remove_privs+0x89/0x110 [15128.508651] ? wake_bit_function+0x70/0x70 [15128.508653] ? ceph_write_iter+0x260/0x770 [ceph] [15128.508664] ceph_write_iter+0x260/0x770 [ceph] [15128.508675] ? lock_is_held_type+0xaa/0x120 [15128.508678] vfs_write+0x324/0x4f0 [15128.508682] ksys_pwrite64+0x65/0xa0 [15128.508684] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15128.508687] do_syscall_64+0x38/0x80 [15128.508689] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15128.508691] RIP: 0033:0x7f48f0eeb93b [15128.508692] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15128.508693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [15128.508695] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15128.508696] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15128.508696] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15128.508697] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15128.508698] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15128.508701] [15128.509557] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15128.813378] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15128.820850] Workqueue: events delayed_work [ceph] [15128.825565] RIP: 0010:lock_release+0x14e/0x260 [15128.830001] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [15128.848739] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [15128.853957] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [15128.861080] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [15128.868205] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [15128.875330] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [15128.882452] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [15128.889577] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15128.897654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15128.903393] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15128.910516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15128.917641] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15128.924763] Call Trace: [15128.927209] [15128.929219] ? rcu_dump_cpu_stacks+0xd1/0x180 [15128.933580] ? rcu_sched_clock_irq+0x633/0x1390 [15128.938103] ? lock_release+0xc5/0x260 [15128.941846] ? lock_is_held_type+0xaa/0x120 [15128.946026] ? lock_is_held_type+0xaa/0x120 [15128.950203] ? tick_sched_do_timer+0xa0/0xa0 [15128.954475] ? rcu_read_lock_sched_held+0x56/0x90 [15128.959172] ? tick_sched_do_timer+0xa0/0xa0 [15128.963435] ? update_process_times+0x5f/0x90 [15128.967788] ? tick_sched_handle.isra.16+0x1e/0x50 [15128.972571] ? tick_sched_timer+0x6f/0x80 [15128.976577] ? __hrtimer_run_queues+0x1d3/0x360 [15128.981104] ? hrtimer_interrupt+0x104/0x240 [15128.985371] ? __local_bh_enable+0x50/0x60 [15128.989466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15128.994690] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15128.999646] [15129.001742] [15129.003841] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15129.009146] ? igrab+0x43/0x50 [15129.012198] ? lock_release+0x14e/0x260 [15129.016035] ? lock_release+0xc5/0x260 [15129.019782] _raw_spin_unlock+0x1b/0x40 [15129.023619] igrab+0x43/0x50 [15129.026497] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15129.031648] delayed_work+0x1d8/0x280 [ceph] [15129.035926] ? process_scheduled_works+0x22a/0x510 [15129.040710] process_scheduled_works+0x2a4/0x510 [15129.045324] worker_thread+0x188/0x340 [15129.049074] ? rescuer_thread+0x310/0x310 [15129.053078] kthread+0xf5/0x130 [15129.056215] ? kthread_complete_and_exit+0x20/0x20 [15129.061000] ret_from_fork+0x31/0x40 [15129.064578] ? kthread_complete_and_exit+0x20/0x20 [15129.069363] ret_from_fork_asm+0x11/0x20 [15129.073294] [15153.720102] watchdog: BUG: soft lockup - CPU#0 stuck for 2999s! [dbench:78683] [15153.727314] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15153.732101] watchdog: BUG: soft lockup - CPU#7 stuck for 11996s! [kworker/7:4:78808] [15153.782776] irq event stamp: 19550560 [15153.782778] hardirqs last enabled at (19550559): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15153.790512] Modules linked in: [15153.794169] hardirqs last disabled at (19550560): [] sysvec_apic_timer_interrupt+0xe/0x90 [15153.804328] ceph [15153.807378] softirqs last enabled at (19550480): [] __do_softirq+0x2be/0x371 [15153.817102] libceph [15153.819025] softirqs last disabled at (19550475): [] irq_exit_rcu+0xb0/0xd0 [15153.827711] dns_resolver [15153.829894] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15153.838406] fscache netfs [15153.841023] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15153.849967] veth nft_chain_nat [15153.852671] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15153.860142] xt_MASQUERADE nf_nat [15153.863279] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15153.868755] nf_conntrack [15153.872084] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15153.890821] nf_defrag_ipv6 nf_defrag_ipv4 [15153.893440] [15153.893441] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15153.898657] nft_compat nf_tables [15153.902748] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15153.904239] nfnetlink bridge [15153.911362] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15153.914673] stp llc [15153.921798] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15153.924777] xfs libcrc32c [15153.931902] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15153.934087] sunrpc intel_rapl_msr [15153.941212] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15153.943915] intel_rapl_common [15153.951039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15153.954438] sb_edac x86_pkg_temp_thermal [15153.962515] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15153.965565] coretemp kvm_intel [15153.971302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15153.975306] kvm irqbypass [15153.982430] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15153.985566] i2c_i801 iTCO_wdt [15153.992693] Call Trace: [15153.992695] [15153.995397] iTCO_vendor_support i2c_smbus [15154.002523] ? watchdog_timer_fn+0x1b4/0x220 [15154.005571] crct10dif_pclmul [15154.008017] ? __hrtimer_run_queues+0x1d3/0x360 [15154.010025] joydev crc32_pclmul [15154.014119] ? hrtimer_interrupt+0x104/0x240 [15154.018380] ipmi_ssif ghash_clmulni_intel [15154.021344] ? __local_bh_enable+0x50/0x60 [15154.025867] mei_me mei [15154.029096] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15154.033358] lpc_ich sha512_ssse3 [15154.037447] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15154.041537] acpi_ipmi [15154.043984] [15154.049200] mfd_core [15154.052510] [15154.052511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15154.057467] wmi ipmi_si [15154.059829] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15154.061921] ipmi_devintf [15154.064195] do_raw_spin_lock+0xb5/0xc0 [15154.066289] ipmi_msghandler [15154.071597] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15154.074126] ioatdma acpi_pad [15154.079000] try_get_cap_refs+0x159/0x880 [ceph] [15154.081613] acpi_power_meter ixgbe [15154.085448] __ceph_get_caps+0x4a1/0x760 [ceph] [15154.088321] igb mdio [15154.093110] ? __file_remove_privs+0x89/0x110 [15154.096070] crc32c_intel [15154.100681] ? wake_bit_function+0x70/0x70 [15154.104165] nvme [15154.108692] ? ceph_write_iter+0x260/0x770 [ceph] [15154.110960] ptp nvme_core [15154.115311] ceph_write_iter+0x260/0x770 [ceph] [15154.117926] i2c_algo_bit [15154.122022] ? lock_is_held_type+0xaa/0x120 [15154.123943] pps_core dca [15154.128642] vfs_write+0x324/0x4f0 [15154.131344] fuse [15154.135871] ksys_pwrite64+0x65/0xa0 [15154.138499] irq event stamp: 44402110 [15154.142682] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15154.145298] hardirqs last enabled at (44402109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15154.148695] do_syscall_64+0x38/0x80 [15154.150617] hardirqs last disabled at (44402110): [] sysvec_apic_timer_interrupt+0xe/0x90 [15154.154191] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15154.157846] softirqs last enabled at (44402108): [] __do_softirq+0x2be/0x371 [15154.162978] RIP: 0033:0x7f48f0eeb93b [15154.173135] softirqs last disabled at (44402103): [] irq_exit_rcu+0xb0/0xd0 [15154.176706] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15154.186429] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15154.191474] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15154.200176] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15154.203745] ORIG_RAX: 0000000000000012 [15154.212258] Workqueue: events delayed_work [ceph] [15154.230994] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15154.240372] [15154.245589] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15154.253061] RIP: 0010:lock_acquire+0xf0/0x2b0 [15154.256890] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15154.261590] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15154.268712] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15154.270203] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15154.277327] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15154.281678] [15154.281679] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15154.288809] [15154.307540] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15154.344939] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15154.352061] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15154.359186] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15154.366310] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15154.374388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15154.380127] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15154.387251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15154.394373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15154.401507] Call Trace: [15154.403953] [15154.405961] ? watchdog_timer_fn+0x1b4/0x220 [15154.410226] ? __hrtimer_run_queues+0x1d3/0x360 [15154.414753] ? hrtimer_interrupt+0x104/0x240 [15154.419022] ? __local_bh_enable+0x50/0x60 [15154.423115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15154.428339] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15154.433297] [15154.435393] [15154.437508] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15154.442816] ? lock_acquire+0xf0/0x2b0 [15154.446568] ? igrab+0x43/0x50 [15154.449616] ? lock_release+0xc5/0x260 [15154.453362] _raw_spin_lock+0x30/0x40 [15154.457027] ? igrab+0x1d/0x50 [15154.460078] igrab+0x1d/0x50 [15154.462956] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15154.468106] delayed_work+0x1d8/0x280 [ceph] [15154.472385] ? process_scheduled_works+0x22a/0x510 [15154.477176] process_scheduled_works+0x2a4/0x510 [15154.481797] worker_thread+0x188/0x340 [15154.485541] ? rescuer_thread+0x310/0x310 [15154.489543] kthread+0xf5/0x130 [15154.492682] ? kthread_complete_and_exit+0x20/0x20 [15154.497466] ret_from_fork+0x31/0x40 [15154.501047] ? kthread_complete_and_exit+0x20/0x20 [15154.505830] ret_from_fork_asm+0x11/0x20 [15154.509761] [15181.720589] watchdog: BUG: soft lockup - CPU#0 stuck for 3025s! [dbench:78683] [15181.727801] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15181.732589] watchdog: BUG: soft lockup - CPU#7 stuck for 12022s! [kworker/7:4:78808] [15181.783243] irq event stamp: 19608760 [15181.790991] Modules linked in: ceph libceph dns_resolver [15181.794648] hardirqs last enabled at (19608759): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15181.799952] fscache netfs veth [15181.810110] hardirqs last disabled at (19608760): [] sysvec_apic_timer_interrupt+0xe/0x90 [15181.813263] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [15181.822990] softirqs last enabled at (19608638): [] __do_softirq+0x2be/0x371 [15181.828640] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [15181.837322] softirqs last disabled at (19608633): [] irq_exit_rcu+0xb0/0xd0 [15181.843234] nfnetlink bridge stp [15181.851744] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15181.851746] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15181.855055] llc xfs [15181.864001] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15181.871471] libcrc32c sunrpc [15181.873656] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15181.879131] intel_rapl_msr intel_rapl_common [15181.882095] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15181.900833] sb_edac x86_pkg_temp_thermal [15181.905185] [15181.905186] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15181.910402] coretemp kvm_intel [15181.914406] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15181.915898] kvm irqbypass [15181.923021] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15181.926158] i2c_i801 iTCO_wdt [15181.933283] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15181.935987] iTCO_vendor_support [15181.943110] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15181.946161] i2c_smbus crct10dif_pclmul [15181.953285] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15181.956510] joydev crc32_pclmul [15181.963634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15181.967465] ipmi_ssif ghash_clmulni_intel [15181.975541] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15181.978765] mei_me mei [15181.984503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15181.988593] lpc_ich [15181.995717] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15181.998163] sha512_ssse3 acpi_ipmi [15182.005286] Call Trace: [15182.005289] [15182.007470] mfd_core wmi [15182.014595] ? watchdog_timer_fn+0x1b4/0x220 [15182.018077] ipmi_si ipmi_devintf [15182.020523] ? __hrtimer_run_queues+0x1d3/0x360 [15182.022534] ipmi_msghandler [15182.025155] ? hrtimer_interrupt+0x104/0x240 [15182.029416] ioatdma acpi_pad [15182.032725] ? __local_bh_enable+0x50/0x60 [15182.037250] acpi_power_meter [15182.040130] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15182.044392] ixgbe igb [15182.047356] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15182.051447] mdio crc32c_intel [15182.054410] [15182.059627] nvme ptp [15182.061988] [15182.066944] nvme_core i2c_algo_bit [15182.069995] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15182.072090] pps_core dca [15182.074366] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15182.076458] fuse [15182.079946] do_raw_spin_lock+0xb5/0xc0 [15182.085248] irq event stamp: 44623676 [15182.085265] hardirqs last enabled at (44623675): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15182.087882] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15182.092751] hardirqs last disabled at (44623676): [] sysvec_apic_timer_interrupt+0xe/0x90 [15182.094680] try_get_cap_refs+0x159/0x880 [ceph] [15182.098506] softirqs last enabled at (44623674): [] __do_softirq+0x2be/0x371 [15182.102170] __ceph_get_caps+0x4a1/0x760 [ceph] [15182.112321] softirqs last disabled at (44623669): [] irq_exit_rcu+0xb0/0xd0 [15182.117112] ? __file_remove_privs+0x89/0x110 [15182.126831] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15182.131442] ? wake_bit_function+0x70/0x70 [15182.140126] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15182.144653] ? ceph_write_iter+0x260/0x770 [ceph] [15182.153159] Workqueue: events delayed_work [ceph] [15182.157512] ceph_write_iter+0x260/0x770 [ceph] [15182.166887] [15182.166888] RIP: 0010:lock_acquire+0xf0/0x2b0 [15182.170993] ? lock_is_held_type+0xaa/0x120 [15182.178458] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15182.183158] vfs_write+0x324/0x4f0 [15182.187852] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15182.192382] ksys_pwrite64+0x65/0xa0 [15182.193869] [15182.193870] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15182.198220] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15182.202396] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15182.221135] do_syscall_64+0x38/0x80 [15182.224530] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15182.229748] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15182.233318] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15182.234811] RIP: 0033:0x7f48f0eeb93b [15182.241935] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15182.247066] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15182.254188] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15182.257760] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15182.264885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15182.269927] ORIG_RAX: 0000000000000012 [15182.277051] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15182.280622] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15182.287747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15182.306484] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15182.314562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15182.319779] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15182.325517] Call Trace: [15182.325519] [15182.329348] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15182.336473] ? watchdog_timer_fn+0x1b4/0x220 [15182.343596] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15182.350722] ? __hrtimer_run_queues+0x1d3/0x360 [15182.357851] [15182.364991] ? hrtimer_interrupt+0x104/0x240 [15182.406051] ? __local_bh_enable+0x50/0x60 [15182.410144] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15182.415368] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15182.420324] [15182.422420] [15182.424520] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15182.429826] ? lock_acquire+0xf0/0x2b0 [15182.433576] ? igrab+0x43/0x50 [15182.436628] ? lock_release+0xc5/0x260 [15182.440372] _raw_spin_lock+0x30/0x40 [15182.444027] ? igrab+0x1d/0x50 [15182.447078] igrab+0x1d/0x50 [15182.449957] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15182.455125] delayed_work+0x1d8/0x280 [ceph] [15182.459406] ? process_scheduled_works+0x22a/0x510 [15182.464197] process_scheduled_works+0x2a4/0x510 [15182.468810] worker_thread+0x188/0x340 [15182.472561] ? rescuer_thread+0x310/0x310 [15182.476572] kthread+0xf5/0x130 [15182.479710] ? kthread_complete_and_exit+0x20/0x20 [15182.484495] ret_from_fork+0x31/0x40 [15182.488073] ? kthread_complete_and_exit+0x20/0x20 [15182.492858] ret_from_fork_asm+0x11/0x20 [15182.496779] [15209.721078] watchdog: BUG: soft lockup - CPU#0 stuck for 3051s! [dbench:78683] [15209.728298] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15209.733077] watchdog: BUG: soft lockup - CPU#7 stuck for 12048s! [kworker/7:4:78808] [15209.783751] irq event stamp: 19666786 [15209.791486] Modules linked in: ceph libceph dns_resolver fscache netfs [15209.795144] hardirqs last enabled at (19666785): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15209.795149] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15209.811827] hardirqs last disabled at (19666786): [] sysvec_apic_timer_interrupt+0xe/0x90 [15209.820511] nft_compat nf_tables nfnetlink bridge stp llc [15209.830235] softirqs last enabled at (19666776): [] __do_softirq+0x2be/0x371 [15209.835713] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [15209.844396] softirqs last disabled at (19666771): [] irq_exit_rcu+0xb0/0xd0 [15209.851261] x86_pkg_temp_thermal coretemp kvm_intel kvm [15209.859774] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15209.865077] irqbypass i2c_i801 [15209.874020] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15209.874022] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15209.877157] iTCO_wdt iTCO_vendor_support [15209.884630] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15209.890107] i2c_smbus crct10dif_pclmul [15209.894110] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15209.912848] joydev crc32_pclmul [15209.916679] [15209.916679] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15209.921896] ipmi_ssif ghash_clmulni_intel [15209.925119] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15209.926610] mei_me [15209.933734] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15209.937843] mei lpc_ich [15209.944966] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15209.947065] sha512_ssse3 [15209.954189] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15209.956720] acpi_ipmi [15209.963843] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15209.966476] mfd_core [15209.973602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15209.975960] wmi ipmi_si [15209.984038] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15209.986309] ipmi_devintf ipmi_msghandler [15209.992045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15209.994576] ioatdma acpi_pad [15210.001701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15210.005704] acpi_power_meter ixgbe [15210.012829] Call Trace: [15210.015791] igb mdio [15210.022920] [15210.026401] crc32c_intel nvme [15210.028846] ? watchdog_timer_fn+0x1b4/0x220 [15210.031116] ptp nvme_core [15210.033127] ? __hrtimer_run_queues+0x1d3/0x360 [15210.036175] i2c_algo_bit [15210.040443] ? hrtimer_interrupt+0x104/0x240 [15210.043146] pps_core dca [15210.047670] ? __local_bh_enable+0x50/0x60 [15210.050287] fuse [15210.054553] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15210.057168] irq event stamp: 44844756 [15210.061260] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15210.063182] hardirqs last enabled at (44844755): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15210.068401] [15210.072057] hardirqs last disabled at (44844756): [] sysvec_apic_timer_interrupt+0xe/0x90 [15210.077014] [15210.087171] softirqs last enabled at (44844754): [] __do_softirq+0x2be/0x371 [15210.089271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15210.098993] softirqs last disabled at (44844749): [] irq_exit_rcu+0xb0/0xd0 [15210.101095] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15210.109776] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15210.115081] do_raw_spin_lock+0xb5/0xc0 [15210.123591] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15210.123593] Workqueue: events delayed_work [ceph] [15210.128479] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15210.137855] [15210.137856] RIP: 0010:lock_acquire+0xf0/0x2b0 [15210.141689] try_get_cap_refs+0x159/0x880 [ceph] [15210.149159] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15210.153860] __ceph_get_caps+0x4a1/0x760 [ceph] [15210.158639] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15210.160133] ? __file_remove_privs+0x89/0x110 [15210.164479] [15210.164480] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15210.169093] ? wake_bit_function+0x70/0x70 [15210.187829] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15210.192355] ? ceph_write_iter+0x260/0x770 [ceph] [15210.197569] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15210.201921] ceph_write_iter+0x260/0x770 [ceph] [15210.203411] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15210.210542] ? lock_is_held_type+0xaa/0x120 [15210.214626] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15210.221769] vfs_write+0x324/0x4f0 [15210.226480] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15210.233613] ksys_pwrite64+0x65/0xa0 [15210.238131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15210.245255] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15210.249432] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15210.256558] do_syscall_64+0x38/0x80 [15210.259954] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15210.268034] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15210.271602] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15210.277342] RIP: 0033:0x7f48f0eeb93b [15210.282479] Call Trace: [15210.289603] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15210.293176] [15210.300299] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15210.305342] ? watchdog_timer_fn+0x1b4/0x220 [15210.312481] ORIG_RAX: 0000000000000012 [15210.316055] ? __hrtimer_run_queues+0x1d3/0x360 [15210.318499] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15210.337239] ? hrtimer_interrupt+0x104/0x240 [15210.339246] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15210.344479] ? __local_bh_enable+0x50/0x60 [15210.348760] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15210.352597] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15210.357119] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15210.364244] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15210.368506] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15210.375631] [15210.379727] [15210.386844] [15210.386846] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15210.422956] ? lock_acquire+0xf0/0x2b0 [15210.426707] ? igrab+0x43/0x50 [15210.429764] ? lock_release+0xc5/0x260 [15210.433511] _raw_spin_lock+0x30/0x40 [15210.437176] ? igrab+0x1d/0x50 [15210.440224] igrab+0x1d/0x50 [15210.443105] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15210.448253] delayed_work+0x1d8/0x280 [ceph] [15210.452534] ? process_scheduled_works+0x22a/0x510 [15210.457323] process_scheduled_works+0x2a4/0x510 [15210.461940] worker_thread+0x188/0x340 [15210.465689] ? rescuer_thread+0x310/0x310 [15210.469693] kthread+0xf5/0x130 [15210.472838] ? kthread_complete_and_exit+0x20/0x20 [15210.477623] ret_from_fork+0x31/0x40 [15210.481202] ? kthread_complete_and_exit+0x20/0x20 [15210.485985] ret_from_fork_asm+0x11/0x20 [15210.489910] [15237.721565] watchdog: BUG: soft lockup - CPU#0 stuck for 3077s! [dbench:78683] [15237.728776] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15237.733565] watchdog: BUG: soft lockup - CPU#7 stuck for 12074s! [kworker/7:4:78808] [15237.784216] irq event stamp: 19724784 [15237.784217] hardirqs last enabled at (19724783): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15237.791965] Modules linked in: [15237.795621] hardirqs last disabled at (19724784): [] sysvec_apic_timer_interrupt+0xe/0x90 [15237.805780] ceph [15237.808831] softirqs last enabled at (19724716): [] __do_softirq+0x2be/0x371 [15237.818555] libceph [15237.820479] softirqs last disabled at (19724711): [] irq_exit_rcu+0xb0/0xd0 [15237.829163] dns_resolver fscache [15237.831346] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15237.839857] netfs [15237.843169] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15237.852111] veth nft_chain_nat [15237.854122] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15237.861595] xt_MASQUERADE [15237.864732] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15237.870208] nf_nat nf_conntrack [15237.872912] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15237.891650] nf_defrag_ipv6 nf_defrag_ipv4 [15237.894876] [15237.894876] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15237.900091] nft_compat nf_tables [15237.904182] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15237.905673] nfnetlink bridge [15237.912796] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15237.916107] stp llc [15237.923232] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15237.926215] xfs libcrc32c [15237.933338] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15237.935521] sunrpc intel_rapl_msr [15237.942645] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15237.945350] intel_rapl_common sb_edac [15237.952474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15237.955872] x86_pkg_temp_thermal coretemp [15237.963961] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15237.967701] kvm_intel kvm [15237.973440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15237.977529] irqbypass i2c_i801 [15237.984655] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15237.987357] iTCO_wdt iTCO_vendor_support [15237.994483] Call Trace: [15237.994484] [15237.997618] i2c_smbus [15238.004745] ? watchdog_timer_fn+0x1b4/0x220 [15238.008747] crct10dif_pclmul joydev [15238.011192] ? __hrtimer_run_queues+0x1d3/0x360 [15238.013202] crc32_pclmul ipmi_ssif [15238.015562] ? hrtimer_interrupt+0x104/0x240 [15238.019824] ghash_clmulni_intel mei_me [15238.023394] ? __local_bh_enable+0x50/0x60 [15238.027920] mei lpc_ich [15238.031406] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15238.035667] sha512_ssse3 acpi_ipmi [15238.039498] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15238.043588] mfd_core [15238.046120] [15238.051337] wmi ipmi_si [15238.054821] [15238.059778] ipmi_devintf ipmi_msghandler [15238.062049] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15238.064146] ioatdma acpi_pad [15238.066681] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15238.068773] acpi_power_meter [15238.072780] do_raw_spin_lock+0xb5/0xc0 [15238.078082] ixgbe igb [15238.081048] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15238.085917] mdio crc32c_intel [15238.088884] try_get_cap_refs+0x159/0x880 [ceph] [15238.092713] nvme ptp [15238.095074] __ceph_get_caps+0x4a1/0x760 [ceph] [15238.099853] nvme_core i2c_algo_bit [15238.102908] ? __file_remove_privs+0x89/0x110 [15238.107515] pps_core [15238.109785] ? wake_bit_function+0x70/0x70 [15238.114309] dca fuse [15238.117798] ? ceph_write_iter+0x260/0x770 [ceph] [15238.122144] [15238.122145] irq event stamp: 45066524 [15238.124417] ceph_write_iter+0x260/0x770 [ceph] [15238.128507] hardirqs last enabled at (45066523): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15238.130781] ? lock_is_held_type+0xaa/0x120 [15238.135473] hardirqs last disabled at (45066524): [] sysvec_apic_timer_interrupt+0xe/0x90 [15238.136968] vfs_write+0x324/0x4f0 [15238.140623] softirqs last enabled at (45066522): [] __do_softirq+0x2be/0x371 [15238.145151] ksys_pwrite64+0x65/0xa0 [15238.155303] softirqs last disabled at (45066517): [] irq_exit_rcu+0xb0/0xd0 [15238.159482] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15238.169206] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15238.172603] do_syscall_64+0x38/0x80 [15238.181288] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15238.184858] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15238.193368] Workqueue: events delayed_work [ceph] [15238.198498] RIP: 0033:0x7f48f0eeb93b [15238.207876] [15238.211449] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15238.218919] RIP: 0010:do_raw_spin_lock+0x5c/0xc0 [15238.223967] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15238.228660] Code: 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 89 43 10 5b e8 2e 22 45 00 85 c0 74 b1 48 c7 c6 29 e2 12 82 48 89 df e8 ab [15238.232247] ORIG_RAX: 0000000000000012 [15238.233737] RSP: 0018:ffffc90002117d98 EFLAGS: 00000246 [15238.252477] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15238.257087] [15238.257088] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [15238.262304] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15238.281043] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [15238.284874] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15238.290090] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15238.297215] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15238.298706] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15238.305829] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15238.312968] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15238.320100] [15238.327220] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15238.373101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15238.378840] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15238.385968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15238.393096] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15238.400221] Call Trace: [15238.402665] [15238.404676] ? watchdog_timer_fn+0x1b4/0x220 [15238.408939] ? __hrtimer_run_queues+0x1d3/0x360 [15238.413466] ? hrtimer_interrupt+0x104/0x240 [15238.417726] ? __local_bh_enable+0x50/0x60 [15238.421819] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15238.427036] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15238.431992] [15238.434090] [15238.436187] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15238.441497] ? do_raw_spin_lock+0x5c/0xc0 [15238.445505] igrab+0x1d/0x50 [15238.448382] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15238.453531] delayed_work+0x1d8/0x280 [ceph] [15238.457813] ? process_scheduled_works+0x22a/0x510 [15238.462604] process_scheduled_works+0x2a4/0x510 [15238.467219] worker_thread+0x188/0x340 [15238.470972] ? rescuer_thread+0x310/0x310 [15238.474980] kthread+0xf5/0x130 [15238.478118] ? kthread_complete_and_exit+0x20/0x20 [15238.482902] ret_from_fork+0x31/0x40 [15238.486473] ? kthread_complete_and_exit+0x20/0x20 [15238.491260] ret_from_fork_asm+0x11/0x20 [15238.495188] [15265.722054] watchdog: BUG: soft lockup - CPU#0 stuck for 3103s! [dbench:78683] [15265.729272] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15265.734053] watchdog: BUG: soft lockup - CPU#7 stuck for 12100s! [kworker/7:4:78808] [15265.784728] irq event stamp: 19782928 [15265.792459] Modules linked in: ceph libceph dns_resolver fscache netfs veth [15265.796118] hardirqs last enabled at (19782927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15265.803077] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15265.813236] hardirqs last disabled at (19782928): [] sysvec_apic_timer_interrupt+0xe/0x90 [15265.821494] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [15265.831219] softirqs last enabled at (19782860): [] __do_softirq+0x2be/0x371 [15265.837917] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15265.846602] softirqs last disabled at (19782855): [] irq_exit_rcu+0xb0/0xd0 [15265.854081] coretemp kvm_intel kvm irqbypass [15265.862594] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15265.866944] i2c_i801 iTCO_wdt [15265.875887] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15265.875889] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15265.878938] iTCO_vendor_support i2c_smbus [15265.886409] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15265.891886] crct10dif_pclmul joydev [15265.895977] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15265.914728] crc32_pclmul ipmi_ssif [15265.918302] [15265.918303] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15265.923520] ghash_clmulni_intel [15265.927005] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15265.928495] mei_me mei [15265.935619] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15265.938842] lpc_ich [15265.945967] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15265.948411] sha512_ssse3 acpi_ipmi [15265.955535] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15265.957731] mfd_core [15265.964852] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15265.968337] wmi ipmi_si [15265.975459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15265.977732] ipmi_devintf ipmi_msghandler [15265.985810] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15265.988339] ioatdma acpi_pad [15265.994076] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15265.998082] acpi_power_meter [15266.005205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15266.008169] ixgbe igb [15266.015294] Call Trace: [15266.018257] mdio crc32c_intel [15266.025383] [15266.027740] nvme [15266.030184] ? watchdog_timer_fn+0x1b4/0x220 [15266.033235] ptp nvme_core [15266.035245] ? __hrtimer_run_queues+0x1d3/0x360 [15266.037167] i2c_algo_bit pps_core [15266.041452] ? hrtimer_interrupt+0x104/0x240 [15266.044154] dca fuse [15266.048677] ? __local_bh_enable+0x50/0x60 [15266.052076] [15266.056342] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15266.058609] irq event stamp: 45287730 [15266.062701] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15266.064190] hardirqs last enabled at (45287729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15266.069409] [15266.073067] hardirqs last disabled at (45287730): [] sysvec_apic_timer_interrupt+0xe/0x90 [15266.078025] [15266.088181] softirqs last enabled at (45287728): [] __do_softirq+0x2be/0x371 [15266.090280] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15266.100004] softirqs last disabled at (45287723): [] irq_exit_rcu+0xb0/0xd0 [15266.102105] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15266.110785] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15266.116092] do_raw_spin_lock+0xb5/0xc0 [15266.124598] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15266.129473] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15266.138847] Workqueue: events delayed_work [ceph] [15266.142681] try_get_cap_refs+0x159/0x880 [ceph] [15266.150150] [15266.154939] __ceph_get_caps+0x4a1/0x760 [ceph] [15266.159631] RIP: 0010:lock_acquire+0x4a/0x2b0 [15266.164246] ? __file_remove_privs+0x89/0x110 [15266.165734] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [15266.170257] ? wake_bit_function+0x70/0x70 [15266.174606] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [15266.178960] ? ceph_write_iter+0x260/0x770 [ceph] [15266.197694] [15266.201785] ceph_write_iter+0x260/0x770 [ceph] [15266.207004] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [15266.211705] ? lock_is_held_type+0xaa/0x120 [15266.213192] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [15266.217735] vfs_write+0x324/0x4f0 [15266.224856] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [15266.229039] ksys_pwrite64+0x65/0xa0 [15266.236157] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [15266.239556] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15266.246679] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15266.250252] do_syscall_64+0x38/0x80 [15266.257374] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15266.262505] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15266.269630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15266.273203] RIP: 0033:0x7f48f0eeb93b [15266.281277] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15266.286324] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15266.292059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15266.295630] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15266.302754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15266.321491] ORIG_RAX: 0000000000000012 [15266.328617] Call Trace: [15266.328618] [15266.333834] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15266.340960] ? watchdog_timer_fn+0x1b4/0x220 [15266.344788] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15266.347233] ? __hrtimer_run_queues+0x1d3/0x360 [15266.349244] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15266.356372] ? hrtimer_interrupt+0x104/0x240 [15266.360632] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15266.367756] ? __local_bh_enable+0x50/0x60 [15266.372279] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15266.379407] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15266.383672] [15266.390793] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15266.414367] [15266.416462] [15266.418560] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15266.423869] ? lock_acquire+0x4a/0x2b0 [15266.427620] ? igrab+0x43/0x50 [15266.430670] ? lock_release+0xc5/0x260 [15266.434414] _raw_spin_lock+0x30/0x40 [15266.438079] ? igrab+0x1d/0x50 [15266.441131] igrab+0x1d/0x50 [15266.444007] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15266.449158] delayed_work+0x1d8/0x280 [ceph] [15266.453457] ? process_scheduled_works+0x22a/0x510 [15266.458247] process_scheduled_works+0x2a4/0x510 [15266.462863] worker_thread+0x188/0x340 [15266.466611] ? rescuer_thread+0x310/0x310 [15266.470613] kthread+0xf5/0x130 [15266.473754] ? kthread_complete_and_exit+0x20/0x20 [15266.478547] ret_from_fork+0x31/0x40 [15266.482125] ? kthread_complete_and_exit+0x20/0x20 [15266.486907] ret_from_fork_asm+0x11/0x20 [15266.490831] [15293.722541] watchdog: BUG: soft lockup - CPU#0 stuck for 3129s! [dbench:78683] [15293.729758] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15293.734541] watchdog: BUG: soft lockup - CPU#7 stuck for 12126s! [kworker/7:4:78808] [15293.785215] irq event stamp: 19840916 [15293.785217] hardirqs last enabled at (19840915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15293.792948] Modules linked in: [15293.796605] hardirqs last disabled at (19840916): [] sysvec_apic_timer_interrupt+0xe/0x90 [15293.806764] ceph [15293.809814] softirqs last enabled at (19840858): [] __do_softirq+0x2be/0x371 [15293.819537] libceph [15293.821460] softirqs last disabled at (19840853): [] irq_exit_rcu+0xb0/0xd0 [15293.830146] dns_resolver [15293.832330] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15293.840841] fscache netfs [15293.843458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15293.852403] veth [15293.855106] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15293.862577] nft_chain_nat xt_MASQUERADE [15293.864502] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15293.869979] nf_nat nf_conntrack [15293.873896] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15293.892635] nf_defrag_ipv6 nf_defrag_ipv4 [15293.895857] [15293.895858] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15293.901074] nft_compat nf_tables [15293.905167] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15293.906658] nfnetlink bridge [15293.913781] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15293.917091] stp llc [15293.924231] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15293.927197] xfs libcrc32c [15293.934320] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15293.936506] sunrpc intel_rapl_msr [15293.943629] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15293.946332] intel_rapl_common sb_edac [15293.953457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15293.956854] x86_pkg_temp_thermal coretemp [15293.964936] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15293.968675] kvm_intel kvm [15293.974414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15293.978504] irqbypass i2c_i801 [15293.985628] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15293.988331] iTCO_wdt iTCO_vendor_support [15293.995456] Call Trace: [15293.995457] [15293.998594] i2c_smbus crct10dif_pclmul [15294.005718] ? watchdog_timer_fn+0x1b4/0x220 [15294.009722] joydev crc32_pclmul [15294.012169] ? __hrtimer_run_queues+0x1d3/0x360 [15294.014177] ipmi_ssif ghash_clmulni_intel [15294.018012] ? hrtimer_interrupt+0x104/0x240 [15294.022273] mei_me mei [15294.025495] ? __local_bh_enable+0x50/0x60 [15294.030019] lpc_ich [15294.034114] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15294.038375] sha512_ssse3 acpi_ipmi [15294.040819] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15294.044909] mfd_core wmi [15294.047095] [15294.052312] ipmi_si ipmi_devintf [15294.055794] [15294.060752] ipmi_msghandler ioatdma [15294.063371] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15294.065468] acpi_pad [15294.068782] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15294.070875] acpi_power_meter [15294.074449] do_raw_spin_lock+0xb5/0xc0 [15294.079751] ixgbe igb mdio [15294.082023] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15294.086893] crc32c_intel nvme [15294.089859] try_get_cap_refs+0x159/0x880 [ceph] [15294.093687] ptp nvme_core [15294.096483] __ceph_get_caps+0x4a1/0x760 [ceph] [15294.101260] i2c_algo_bit pps_core [15294.104315] ? __file_remove_privs+0x89/0x110 [15294.108939] dca [15294.111644] ? wake_bit_function+0x70/0x70 [15294.116168] fuse [15294.119568] ? ceph_write_iter+0x260/0x770 [ceph] [15294.123917] irq event stamp: 45508828 [15294.125772] ceph_write_iter+0x260/0x770 [ceph] [15294.129863] hardirqs last enabled at (45508827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15294.131791] ? lock_is_held_type+0xaa/0x120 [15294.136484] hardirqs last disabled at (45508828): [] sysvec_apic_timer_interrupt+0xe/0x90 [15294.140143] vfs_write+0x324/0x4f0 [15294.144665] softirqs last enabled at (45508826): [] __do_softirq+0x2be/0x371 [15294.154826] ksys_pwrite64+0x65/0xa0 [15294.159000] softirqs last disabled at (45508821): [] irq_exit_rcu+0xb0/0xd0 [15294.168725] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15294.172121] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15294.180806] do_syscall_64+0x38/0x80 [15294.184375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15294.192889] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15294.198019] Workqueue: events delayed_work [ceph] [15294.207395] RIP: 0033:0x7f48f0eeb93b [15294.210966] [15294.218438] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15294.223480] RIP: 0010:lock_acquire+0xf0/0x2b0 [15294.228177] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15294.231750] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15294.233239] ORIG_RAX: 0000000000000012 [15294.251978] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15294.256329] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15294.261546] [15294.280284] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15294.284114] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15294.289331] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15294.296456] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15294.297946] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15294.305071] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15294.312194] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15294.319317] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15294.326449] [15294.333566] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15294.364256] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15294.372334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15294.378072] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15294.385196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15294.392320] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15294.399444] Call Trace: [15294.401889] [15294.403900] ? watchdog_timer_fn+0x1b4/0x220 [15294.408163] ? __hrtimer_run_queues+0x1d3/0x360 [15294.412691] ? hrtimer_interrupt+0x104/0x240 [15294.416960] ? __local_bh_enable+0x50/0x60 [15294.421052] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15294.426269] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15294.431226] [15294.433322] [15294.435419] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15294.440728] ? lock_acquire+0xf0/0x2b0 [15294.444478] ? igrab+0x43/0x50 [15294.447528] ? lock_release+0xc5/0x260 [15294.451272] _raw_spin_lock+0x30/0x40 [15294.454947] ? igrab+0x1d/0x50 [15294.457996] igrab+0x1d/0x50 [15294.460874] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15294.466024] delayed_work+0x1d8/0x280 [ceph] [15294.470306] ? process_scheduled_works+0x22a/0x510 [15294.475095] process_scheduled_works+0x2a4/0x510 [15294.479713] worker_thread+0x188/0x340 [15294.483462] ? rescuer_thread+0x310/0x310 [15294.487466] kthread+0xf5/0x130 [15294.490602] ? kthread_complete_and_exit+0x20/0x20 [15294.495388] ret_from_fork+0x31/0x40 [15294.498965] ? kthread_complete_and_exit+0x20/0x20 [15294.503748] ret_from_fork_asm+0x11/0x20 [15294.507673] [15321.723029] watchdog: BUG: soft lockup - CPU#0 stuck for 3155s! [dbench:78683] [15321.730246] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15321.735029] watchdog: BUG: soft lockup - CPU#7 stuck for 12152s! [kworker/7:4:78808] [15321.785686] irq event stamp: 19899088 [15321.785687] hardirqs last enabled at (19899087): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15321.793435] Modules linked in: [15321.797092] hardirqs last disabled at (19899088): [] sysvec_apic_timer_interrupt+0xe/0x90 [15321.807251] ceph [15321.810300] softirqs last enabled at (19899030): [] __do_softirq+0x2be/0x371 [15321.820025] libceph dns_resolver [15321.821949] softirqs last disabled at (19899025): [] irq_exit_rcu+0xb0/0xd0 [15321.830634] fscache [15321.833944] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15321.842454] netfs veth [15321.844639] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15321.853584] nft_chain_nat xt_MASQUERADE [15321.856029] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15321.863497] nf_nat nf_conntrack [15321.867425] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15321.872900] nf_defrag_ipv6 nf_defrag_ipv4 [15321.876126] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15321.894862] nft_compat [15321.898953] [15321.898953] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15321.904170] nf_tables nfnetlink [15321.906615] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15321.908106] bridge stp [15321.915231] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15321.918455] llc xfs [15321.925578] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15321.928023] libcrc32c sunrpc [15321.935145] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15321.937331] intel_rapl_msr [15321.944454] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15321.947425] intel_rapl_common sb_edac [15321.954552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15321.957343] x86_pkg_temp_thermal coretemp [15321.965424] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15321.969172] kvm_intel kvm [15321.974909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15321.979000] irqbypass i2c_i801 [15321.986126] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15321.988828] iTCO_wdt iTCO_vendor_support [15321.995953] Call Trace: [15321.995954] [15321.999091] i2c_smbus crct10dif_pclmul [15322.006215] ? watchdog_timer_fn+0x1b4/0x220 [15322.010218] joydev crc32_pclmul [15322.012665] ? __hrtimer_run_queues+0x1d3/0x360 [15322.014673] ipmi_ssif ghash_clmulni_intel [15322.018508] ? hrtimer_interrupt+0x104/0x240 [15322.022767] mei_me [15322.025993] ? __local_bh_enable+0x50/0x60 [15322.030517] mei lpc_ich [15322.034609] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15322.038871] sha512_ssse3 acpi_ipmi [15322.040969] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15322.045060] mfd_core [15322.047589] [15322.052807] wmi [15322.056291] [15322.061249] ipmi_si ipmi_devintf [15322.063521] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15322.065616] ipmi_msghandler ioatdma [15322.067459] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15322.069551] acpi_pad acpi_power_meter [15322.072865] do_raw_spin_lock+0xb5/0xc0 [15322.078167] ixgbe igb mdio [15322.081740] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15322.086608] crc32c_intel nvme [15322.090353] try_get_cap_refs+0x159/0x880 [ceph] [15322.094182] ptp nvme_core [15322.096980] __ceph_get_caps+0x4a1/0x760 [ceph] [15322.101758] i2c_algo_bit pps_core [15322.104812] ? __file_remove_privs+0x89/0x110 [15322.109430] dca fuse [15322.112133] ? wake_bit_function+0x70/0x70 [15322.116655] [15322.116656] irq event stamp: 45730526 [15322.120057] ? ceph_write_iter+0x260/0x770 [ceph] [15322.124405] hardirqs last enabled at (45730525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15322.126702] ceph_write_iter+0x260/0x770 [ceph] [15322.130791] hardirqs last disabled at (45730526): [] sysvec_apic_timer_interrupt+0xe/0x90 [15322.132286] ? lock_is_held_type+0xaa/0x120 [15322.135939] softirqs last enabled at (45730524): [] __do_softirq+0x2be/0x371 [15322.140640] vfs_write+0x324/0x4f0 [15322.150795] softirqs last disabled at (45730519): [] irq_exit_rcu+0xb0/0xd0 [15322.155325] ksys_pwrite64+0x65/0xa0 [15322.165043] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15322.169221] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15322.177904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15322.177905] Workqueue: events delayed_work [ceph] [15322.181302] do_syscall_64+0x38/0x80 [15322.189812] [15322.189812] RIP: 0010:lock_acquire+0xf0/0x2b0 [15322.193385] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15322.202762] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15322.207891] RIP: 0033:0x7f48f0eeb93b [15322.215363] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15322.220061] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15322.223631] [15322.223632] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15322.225122] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15322.229473] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15322.234515] ORIG_RAX: 0000000000000012 [15322.253253] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15322.256823] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15322.262043] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15322.280780] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15322.282271] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15322.289395] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15322.294611] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15322.301735] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15322.305565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15322.312690] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15322.319814] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15322.326943] [15322.334063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15322.392807] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15322.399932] Call Trace: [15322.402377] [15322.404388] ? watchdog_timer_fn+0x1b4/0x220 [15322.408650] ? __hrtimer_run_queues+0x1d3/0x360 [15322.413178] ? hrtimer_interrupt+0x104/0x240 [15322.417448] ? __local_bh_enable+0x50/0x60 [15322.421541] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15322.426764] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15322.431722] [15322.433818] [15322.435917] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15322.441225] ? lock_acquire+0xf0/0x2b0 [15322.444974] ? igrab+0x43/0x50 [15322.448024] ? lock_release+0xc5/0x260 [15322.451769] _raw_spin_lock+0x30/0x40 [15322.455435] ? igrab+0x1d/0x50 [15322.458485] igrab+0x1d/0x50 [15322.461363] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15322.466514] delayed_work+0x1d8/0x280 [ceph] [15322.470794] ? process_scheduled_works+0x22a/0x510 [15322.475585] process_scheduled_works+0x2a4/0x510 [15322.480199] worker_thread+0x188/0x340 [15322.483948] ? rescuer_thread+0x310/0x310 [15322.487951] kthread+0xf5/0x130 [15322.491089] ? kthread_complete_and_exit+0x20/0x20 [15322.495875] ret_from_fork+0x31/0x40 [15322.499453] ? kthread_complete_and_exit+0x20/0x20 [15322.504238] ret_from_fork_asm+0x11/0x20 [15322.508161] [15324.078437] rcu: INFO: rcu_preempt self-detected stall on CPU [15324.084201] rcu: 7-....: (3293292 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/3292103 fqs=1633562 [15324.094625] rcu: (t=3387408 jiffies g=1309609 q=176162 ncpus=8) [15324.100623] Sending NMI from CPU 7 to CPUs 0: [15324.104977] NMI backtrace for cpu 0 [15324.104979] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15324.104981] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15324.104981] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15324.104987] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15324.104988] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15324.104989] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15324.104990] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15324.104991] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15324.104992] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15324.104992] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15324.104993] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15324.104994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15324.104995] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15324.104996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15324.104997] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15324.104997] Call Trace: [15324.104999] [15324.105000] ? nmi_cpu_backtrace+0xad/0x120 [15324.105004] ? nmi_cpu_backtrace_handler+0x11/0x20 [15324.105005] ? nmi_handle+0xd4/0x260 [15324.105010] ? default_do_nmi+0x49/0xf0 [15324.105012] ? exc_nmi+0x11e/0x150 [15324.105013] ? end_repeat_nmi+0x16/0x67 [15324.105018] ? queued_spin_lock_slowpath+0x16/0x2e0 [15324.105020] ? queued_spin_lock_slowpath+0x16/0x2e0 [15324.105023] ? queued_spin_lock_slowpath+0x16/0x2e0 [15324.105025] [15324.105025] [15324.105026] do_raw_spin_lock+0xb5/0xc0 [15324.105029] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15324.105050] try_get_cap_refs+0x159/0x880 [ceph] [15324.105065] __ceph_get_caps+0x4a1/0x760 [ceph] [15324.105078] ? __file_remove_privs+0x89/0x110 [15324.105080] ? wake_bit_function+0x70/0x70 [15324.105083] ? ceph_write_iter+0x260/0x770 [ceph] [15324.105094] ceph_write_iter+0x260/0x770 [ceph] [15324.105105] ? lock_is_held_type+0xaa/0x120 [15324.105108] vfs_write+0x324/0x4f0 [15324.105113] ksys_pwrite64+0x65/0xa0 [15324.105115] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15324.105118] do_syscall_64+0x38/0x80 [15324.105120] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15324.105122] RIP: 0033:0x7f48f0eeb93b [15324.105123] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15324.105125] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [15324.105126] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15324.105127] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15324.105128] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15324.105128] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15324.105129] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15324.105132] [15324.105974] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15324.409786] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15324.417258] Workqueue: events delayed_work [ceph] [15324.421971] RIP: 0010:lock_acquire+0xf0/0x2b0 [15324.426321] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15324.445060] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15324.450278] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15324.457401] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15324.464525] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15324.471650] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15324.478774] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15324.485897] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15324.493974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15324.499712] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15324.506838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15324.513961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15324.521084] Call Trace: [15324.523529] [15324.525542] ? rcu_dump_cpu_stacks+0xd1/0x180 [15324.529899] ? rcu_sched_clock_irq+0x633/0x1390 [15324.534423] ? lock_release+0xc5/0x260 [15324.538168] ? lock_is_held_type+0xaa/0x120 [15324.542345] ? lock_is_held_type+0xaa/0x120 [15324.546524] ? tick_sched_do_timer+0xa0/0xa0 [15324.550795] ? rcu_read_lock_sched_held+0x56/0x90 [15324.555493] ? tick_sched_do_timer+0xa0/0xa0 [15324.559757] ? update_process_times+0x5f/0x90 [15324.564108] ? tick_sched_handle.isra.16+0x1e/0x50 [15324.568891] ? tick_sched_timer+0x6f/0x80 [15324.572897] ? __hrtimer_run_queues+0x1d3/0x360 [15324.577423] ? hrtimer_interrupt+0x104/0x240 [15324.581692] ? __local_bh_enable+0x50/0x60 [15324.585785] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15324.591009] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15324.595968] [15324.598064] [15324.600161] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15324.605470] ? lock_acquire+0xf0/0x2b0 [15324.609220] ? igrab+0x43/0x50 [15324.612270] ? lock_release+0xc5/0x260 [15324.616015] _raw_spin_lock+0x30/0x40 [15324.619678] ? igrab+0x1d/0x50 [15324.622729] igrab+0x1d/0x50 [15324.625610] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15324.630758] delayed_work+0x1d8/0x280 [ceph] [15324.635040] ? process_scheduled_works+0x22a/0x510 [15324.639829] process_scheduled_works+0x2a4/0x510 [15324.644445] worker_thread+0x188/0x340 [15324.648194] ? rescuer_thread+0x310/0x310 [15324.652196] kthread+0xf5/0x130 [15324.655336] ? kthread_complete_and_exit+0x20/0x20 [15324.660120] ret_from_fork+0x31/0x40 [15324.663698] ? kthread_complete_and_exit+0x20/0x20 [15324.668483] ret_from_fork_asm+0x11/0x20 [15324.672405] [15349.723518] watchdog: BUG: soft lockup - CPU#0 stuck for 3181s! [dbench:78683] [15349.730733] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15349.735518] watchdog: BUG: soft lockup - CPU#7 stuck for 12178s! [kworker/7:4:78808] [15349.786195] irq event stamp: 19957146 [15349.793932] Modules linked in: ceph [15349.797587] hardirqs last enabled at (19957145): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15349.797590] libceph dns_resolver fscache netfs [15349.811230] hardirqs last disabled at (19957146): [] sysvec_apic_timer_interrupt+0xe/0x90 [15349.815754] veth nft_chain_nat xt_MASQUERADE [15349.825479] softirqs last enabled at (19957104): [] __do_softirq+0x2be/0x371 [15349.825480] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [15349.838515] softirqs last disabled at (19957099): [] irq_exit_rcu+0xb0/0xd0 [15349.845292] nf_tables nfnetlink [15349.853803] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15349.857025] bridge stp [15349.865969] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15349.868415] llc xfs [15349.875886] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15349.878087] libcrc32c sunrpc [15349.883564] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15349.883565] intel_rapl_msr [15349.886529] intel_rapl_common [15349.905265] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15349.908056] sb_edac [15349.911107] [15349.916325] x86_pkg_temp_thermal [15349.918509] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15349.919999] coretemp kvm_intel [15349.923310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15349.930433] kvm [15349.933571] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15349.940696] irqbypass i2c_i801 [15349.942534] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15349.949656] iTCO_wdt iTCO_vendor_support [15349.952794] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15349.959919] i2c_smbus [15349.963923] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15349.971047] crct10dif_pclmul joydev [15349.973403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15349.981481] crc32_pclmul [15349.985052] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15349.990791] ipmi_ssif [15349.993406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15350.000530] ghash_clmulni_intel [15350.002889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15350.010030] mei_me mei [15350.013255] Call Trace: [15350.013256] [15350.020378] lpc_ich sha512_ssse3 [15350.022823] ? watchdog_timer_fn+0x1b4/0x220 [15350.025265] acpi_ipmi mfd_core [15350.027277] ? __hrtimer_run_queues+0x1d3/0x360 [15350.030587] wmi ipmi_si [15350.034854] ? hrtimer_interrupt+0x104/0x240 [15350.037988] ipmi_devintf ipmi_msghandler [15350.042514] ? __local_bh_enable+0x50/0x60 [15350.045045] ioatdma acpi_pad [15350.049310] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15350.053313] acpi_power_meter [15350.057404] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15350.060366] ixgbe igb [15350.065586] [15350.068547] mdio crc32c_intel [15350.073505] [15350.073506] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15350.075862] nvme [15350.077964] ? queued_spin_lock_slowpath+0x16/0x2e0 [15350.081012] ptp nvme_core [15350.083110] do_raw_spin_lock+0xb5/0xc0 [15350.088411] i2c_algo_bit pps_core [15350.090339] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15350.095207] dca fuse [15350.097921] try_get_cap_refs+0x159/0x880 [ceph] [15350.101750] [15350.101751] irq event stamp: 45947492 [15350.105152] __ceph_get_caps+0x4a1/0x760 [ceph] [15350.109933] hardirqs last enabled at (45947491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15350.112214] ? __file_remove_privs+0x89/0x110 [15350.116822] hardirqs last disabled at (45947492): [] sysvec_apic_timer_interrupt+0xe/0x90 [15350.118315] ? wake_bit_function+0x70/0x70 [15350.121971] softirqs last enabled at (45947490): [] __do_softirq+0x2be/0x371 [15350.126498] ? ceph_write_iter+0x260/0x770 [ceph] [15350.136651] softirqs last disabled at (45947485): [] irq_exit_rcu+0xb0/0xd0 [15350.141005] ceph_write_iter+0x260/0x770 [ceph] [15350.150729] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15350.154822] ? lock_is_held_type+0xaa/0x120 [15350.163503] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15350.168213] vfs_write+0x324/0x4f0 [15350.176719] Workqueue: events delayed_work [ceph] [15350.181248] ksys_pwrite64+0x65/0xa0 [15350.190620] [15350.190621] RIP: 0010:lock_acquire+0xf0/0x2b0 [15350.194799] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15350.202270] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15350.205669] do_syscall_64+0x38/0x80 [15350.210364] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15350.213935] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15350.215425] [15350.215426] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15350.219777] RIP: 0033:0x7f48f0eeb93b [15350.224918] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15350.243655] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15350.247224] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15350.252440] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15350.257484] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15350.258976] ORIG_RAX: 0000000000000012 [15350.266100] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15350.269670] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15350.276795] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15350.295532] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15350.302657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15350.307874] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15350.314998] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15350.318828] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15350.325952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15350.333076] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15350.341154] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15350.348285] [15350.354016] Call Trace: [15350.354018] [15350.403400] ? watchdog_timer_fn+0x1b4/0x220 [15350.407665] ? __hrtimer_run_queues+0x1d3/0x360 [15350.412208] ? hrtimer_interrupt+0x104/0x240 [15350.416469] ? __local_bh_enable+0x50/0x60 [15350.420563] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15350.425786] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15350.430743] [15350.432842] [15350.434939] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15350.440247] ? lock_acquire+0xf0/0x2b0 [15350.443998] ? igrab+0x43/0x50 [15350.447048] ? lock_release+0xc5/0x260 [15350.450793] _raw_spin_lock+0x30/0x40 [15350.454458] ? igrab+0x1d/0x50 [15350.457507] igrab+0x1d/0x50 [15350.460386] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15350.465536] delayed_work+0x1d8/0x280 [ceph] [15350.469817] ? process_scheduled_works+0x22a/0x510 [15350.474606] process_scheduled_works+0x2a4/0x510 [15350.479221] worker_thread+0x188/0x340 [15350.482972] ? rescuer_thread+0x310/0x310 [15350.486983] kthread+0xf5/0x130 [15350.490122] ? kthread_complete_and_exit+0x20/0x20 [15350.494925] ret_from_fork+0x31/0x40 [15350.498500] ? kthread_complete_and_exit+0x20/0x20 [15350.503286] ret_from_fork_asm+0x11/0x20 [15350.507211] [15377.724007] watchdog: BUG: soft lockup - CPU#0 stuck for 3207s! [dbench:78683] [15377.731222] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15377.736007] watchdog: BUG: soft lockup - CPU#7 stuck for 12204s! [kworker/7:4:78808] [15377.786677] irq event stamp: 20015236 [15377.794410] Modules linked in: ceph libceph dns_resolver [15377.798067] hardirqs last enabled at (20015235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15377.803371] fscache netfs veth [15377.813528] hardirqs last disabled at (20015236): [] sysvec_apic_timer_interrupt+0xe/0x90 [15377.816683] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [15377.826408] softirqs last enabled at (20015200): [] __do_softirq+0x2be/0x371 [15377.832059] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [15377.840742] softirqs last disabled at (20015195): [] irq_exit_rcu+0xb0/0xd0 [15377.846653] nfnetlink bridge [15377.855164] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15377.855166] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15377.858129] stp [15377.867072] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15377.874543] llc xfs [15377.876398] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15377.881876] libcrc32c sunrpc [15377.884059] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15377.902796] intel_rapl_msr [15377.905760] [15377.905761] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15377.910979] intel_rapl_common sb_edac [15377.913770] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15377.915259] x86_pkg_temp_thermal coretemp [15377.922401] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15377.926145] kvm_intel kvm [15377.933269] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15377.937359] irqbypass i2c_i801 [15377.944484] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15377.947188] iTCO_wdt iTCO_vendor_support [15377.954312] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15377.957449] i2c_smbus crct10dif_pclmul [15377.964574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15377.968578] joydev crc32_pclmul [15377.976655] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15377.980485] ipmi_ssif ghash_clmulni_intel [15377.986224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15377.989447] mei_me mei [15377.996571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15378.000664] lpc_ich sha512_ssse3 [15378.007787] Call Trace: [15378.007788] [15378.010231] acpi_ipmi [15378.017356] ? watchdog_timer_fn+0x1b4/0x220 [15378.020665] mfd_core wmi [15378.023112] ? __hrtimer_run_queues+0x1d3/0x360 [15378.025119] ipmi_si ipmi_devintf [15378.027482] ? hrtimer_interrupt+0x104/0x240 [15378.031743] ipmi_msghandler ioatdma [15378.034360] ? __local_bh_enable+0x50/0x60 [15378.038884] acpi_pad acpi_power_meter [15378.042196] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15378.046459] ixgbe [15378.050030] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15378.054119] igb mdio [15378.057864] [15378.063082] crc32c_intel nvme [15378.065092] [15378.070048] ptp nvme_core [15378.072320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15378.074417] i2c_algo_bit pps_core [15378.077473] ? queued_spin_lock_slowpath+0x18/0x2e0 [15378.079566] dca fuse [15378.082272] do_raw_spin_lock+0xb5/0xc0 [15378.087574] irq event stamp: 46168872 [15378.090973] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15378.095841] hardirqs last enabled at (46168871): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15378.098115] try_get_cap_refs+0x159/0x880 [ceph] [15378.101943] hardirqs last disabled at (46168872): [] sysvec_apic_timer_interrupt+0xe/0x90 [15378.105606] __ceph_get_caps+0x4a1/0x760 [ceph] [15378.110402] softirqs last enabled at (46168870): [] __do_softirq+0x2be/0x371 [15378.120565] ? __file_remove_privs+0x89/0x110 [15378.125171] softirqs last disabled at (46168865): [] irq_exit_rcu+0xb0/0xd0 [15378.134895] ? wake_bit_function+0x70/0x70 [15378.139418] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15378.148105] ? ceph_write_iter+0x260/0x770 [ceph] [15378.152453] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15378.160966] ceph_write_iter+0x260/0x770 [ceph] [15378.165055] Workqueue: events delayed_work [ceph] [15378.174436] ? lock_is_held_type+0xaa/0x120 [15378.179131] [15378.179132] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [15378.186603] vfs_write+0x324/0x4f0 [15378.191126] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [15378.195826] ksys_pwrite64+0x65/0xa0 [15378.200001] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [15378.201490] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15378.206101] [15378.206102] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [15378.209500] do_syscall_64+0x38/0x80 [15378.228235] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [15378.231808] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15378.237025] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15378.242155] RIP: 0033:0x7f48f0eeb93b [15378.243646] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15378.250770] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15378.254340] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15378.261465] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15378.266510] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15378.273634] ORIG_RAX: 0000000000000012 [15378.277203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15378.284329] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15378.303067] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15378.310191] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15378.315410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15378.323485] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15378.327315] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15378.333052] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15378.340178] Call Trace: [15378.347301] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15378.354425] [15378.361554] [15378.368690] ? watchdog_timer_fn+0x1b4/0x220 [15378.400967] ? __hrtimer_run_queues+0x1d3/0x360 [15378.405493] ? hrtimer_interrupt+0x104/0x240 [15378.409754] ? __local_bh_enable+0x50/0x60 [15378.413847] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15378.419065] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15378.424020] [15378.426119] [15378.428216] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15378.433523] ? do_raw_spin_lock+0x42/0xc0 [15378.437534] igrab+0x1d/0x50 [15378.440413] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15378.445561] delayed_work+0x1d8/0x280 [ceph] [15378.449841] ? process_scheduled_works+0x22a/0x510 [15378.454632] process_scheduled_works+0x2a4/0x510 [15378.459247] worker_thread+0x188/0x340 [15378.462997] ? rescuer_thread+0x310/0x310 [15378.466999] kthread+0xf5/0x130 [15378.470138] ? kthread_complete_and_exit+0x20/0x20 [15378.474921] ret_from_fork+0x31/0x40 [15378.478492] ? kthread_complete_and_exit+0x20/0x20 [15378.483277] ret_from_fork_asm+0x11/0x20 [15378.487199] [15405.724497] watchdog: BUG: soft lockup - CPU#0 stuck for 3233s! [dbench:78683] [15405.731718] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15405.736496] watchdog: BUG: soft lockup - CPU#7 stuck for 12230s! [kworker/7:4:78808] [15405.787174] irq event stamp: 20073268 [15405.794905] Modules linked in: ceph libceph dns_resolver fscache netfs veth [15405.798562] hardirqs last enabled at (20073267): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15405.798568] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [15405.815680] hardirqs last disabled at (20073268): [] sysvec_apic_timer_interrupt+0xe/0x90 [15405.824891] nf_tables nfnetlink bridge stp llc xfs libcrc32c [15405.834616] softirqs last enabled at (20073258): [] __do_softirq+0x2be/0x371 [15405.840353] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15405.849039] softirqs last disabled at (20073253): [] irq_exit_rcu+0xb0/0xd0 [15405.849043] coretemp kvm_intel kvm irqbypass i2c_i801 [15405.865030] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15405.870160] iTCO_wdt iTCO_vendor_support [15405.879122] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15405.879124] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15405.883125] i2c_smbus crct10dif_pclmul [15405.890596] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15405.896073] joydev crc32_pclmul [15405.899904] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15405.918642] ipmi_ssif ghash_clmulni_intel [15405.921866] [15405.921868] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15405.927100] mei_me mei [15405.931192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15405.932681] lpc_ich [15405.939806] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15405.942251] sha512_ssse3 acpi_ipmi [15405.949374] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15405.951559] mfd_core wmi [15405.958683] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15405.962182] ipmi_si [15405.969307] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15405.971925] ipmi_devintf ipmi_msghandler [15405.979049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15405.981235] ioatdma acpi_pad [15405.989312] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15405.993316] acpi_power_meter ixgbe [15405.999053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15406.002017] igb mdio [15406.009140] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15406.012624] crc32c_intel nvme [15406.019748] Call Trace: [15406.019750] [15406.022020] ptp nvme_core [15406.029146] ? watchdog_timer_fn+0x1b4/0x220 [15406.032195] i2c_algo_bit pps_core [15406.034640] ? __hrtimer_run_queues+0x1d3/0x360 [15406.036648] dca fuse [15406.039355] ? hrtimer_interrupt+0x104/0x240 [15406.043618] [15406.047014] ? __local_bh_enable+0x50/0x60 [15406.051539] irq event stamp: 46390700 [15406.053811] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15406.058073] hardirqs last enabled at (46390699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15406.059566] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15406.063655] hardirqs last disabled at (46390700): [] sysvec_apic_timer_interrupt+0xe/0x90 [15406.067312] [15406.072530] softirqs last enabled at (46390698): [] __do_softirq+0x2be/0x371 [15406.082688] [15406.082690] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15406.087646] softirqs last disabled at (46390693): [] irq_exit_rcu+0xb0/0xd0 [15406.097373] ? queued_spin_lock_slowpath+0x16/0x2e0 [15406.099466] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15406.108152] do_raw_spin_lock+0xb5/0xc0 [15406.110266] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15406.115571] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15406.124079] Workqueue: events delayed_work [ceph] [15406.128955] try_get_cap_refs+0x159/0x880 [ceph] [15406.138329] [15406.138330] RIP: 0010:lock_acquire+0x32/0x2b0 [15406.142181] __ceph_get_caps+0x4a1/0x760 [ceph] [15406.149647] Code: ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 <31> c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd [15406.154437] ? __file_remove_privs+0x89/0x110 [15406.159128] RSP: 0018:ffffc90002117d18 EFLAGS: 00000296 [15406.163740] ? wake_bit_function+0x70/0x70 [15406.165230] [15406.165230] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 0000000000000000 [15406.169583] ? ceph_write_iter+0x260/0x770 [ceph] [15406.174105] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [15406.192843] ceph_write_iter+0x260/0x770 [ceph] [15406.197194] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [15406.202416] ? lock_is_held_type+0xaa/0x120 [15406.206502] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [15406.207995] vfs_write+0x324/0x4f0 [15406.215116] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15406.219820] ksys_pwrite64+0x65/0xa0 [15406.226938] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15406.231462] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15406.238587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15406.242766] do_syscall_64+0x38/0x80 [15406.249899] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15406.253294] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15406.260417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15406.263991] RIP: 0033:0x7f48f0eeb93b [15406.272065] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15406.277199] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15406.282935] Call Trace: [15406.282936] [15406.286506] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15406.293629] ? watchdog_timer_fn+0x1b4/0x220 [15406.298674] ORIG_RAX: 0000000000000012 [15406.305798] ? __hrtimer_run_queues+0x1d3/0x360 [15406.309369] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15406.316495] ? hrtimer_interrupt+0x104/0x240 [15406.335230] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15406.337674] ? __local_bh_enable+0x50/0x60 [15406.339683] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15406.344905] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15406.349167] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15406.353016] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15406.357538] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15406.364663] [15406.368933] [15406.376052] [15406.418069] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15406.423376] ? lock_acquire+0x32/0x2b0 [15406.427126] ? igrab+0x43/0x50 [15406.430190] ? lock_release+0xc5/0x260 [15406.433938] _raw_spin_lock+0x30/0x40 [15406.437603] ? igrab+0x1d/0x50 [15406.440655] igrab+0x1d/0x50 [15406.443533] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15406.448681] delayed_work+0x1d8/0x280 [ceph] [15406.452965] ? process_scheduled_works+0x22a/0x510 [15406.457752] process_scheduled_works+0x2a4/0x510 [15406.462370] worker_thread+0x188/0x340 [15406.466118] ? rescuer_thread+0x310/0x310 [15406.470122] kthread+0xf5/0x130 [15406.473268] ? kthread_complete_and_exit+0x20/0x20 [15406.478053] ret_from_fork+0x31/0x40 [15406.481630] ? kthread_complete_and_exit+0x20/0x20 [15406.486414] ret_from_fork_asm+0x11/0x20 [15406.490338] [15433.724986] watchdog: BUG: soft lockup - CPU#0 stuck for 3259s! [dbench:78683] [15433.732202] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15433.736986] watchdog: BUG: soft lockup - CPU#7 stuck for 12256s! [kworker/7:4:78808] [15433.787644] irq event stamp: 20131334 [15433.787645] hardirqs last enabled at (20131333): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15433.795392] Modules linked in: [15433.799050] hardirqs last disabled at (20131334): [] sysvec_apic_timer_interrupt+0xe/0x90 [15433.809206] ceph [15433.812259] softirqs last enabled at (20131324): [] __do_softirq+0x2be/0x371 [15433.821983] libceph dns_resolver [15433.823906] softirqs last disabled at (20131319): [] irq_exit_rcu+0xb0/0xd0 [15433.832590] fscache netfs [15433.835900] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15433.844412] veth nft_chain_nat [15433.847116] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15433.856059] xt_MASQUERADE nf_nat [15433.859198] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15433.866676] nf_conntrack [15433.869988] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15433.875464] nf_defrag_ipv6 nf_defrag_ipv4 [15433.878082] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15433.896821] nft_compat nf_tables [15433.900910] [15433.900910] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15433.906128] nfnetlink [15433.909440] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15433.910929] bridge stp [15433.918054] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15433.920411] llc [15433.927534] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15433.929979] xfs libcrc32c [15433.937104] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15433.938940] sunrpc [15433.946064] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15433.948769] intel_rapl_msr intel_rapl_common [15433.955894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15433.957991] sb_edac [15433.966068] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15433.970419] x86_pkg_temp_thermal [15433.976156] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15433.978341] coretemp kvm_intel [15433.985464] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15433.988776] kvm irqbypass [15433.995898] Call Trace: [15433.995900] [15433.999038] i2c_i801 iTCO_wdt [15434.006163] ? watchdog_timer_fn+0x1b4/0x220 [15434.008865] iTCO_vendor_support i2c_smbus [15434.011310] ? __hrtimer_run_queues+0x1d3/0x360 [15434.013320] crct10dif_pclmul joydev [15434.016387] ? hrtimer_interrupt+0x104/0x240 [15434.020665] crc32_pclmul ipmi_ssif [15434.024760] ? __local_bh_enable+0x50/0x60 [15434.029284] ghash_clmulni_intel mei_me [15434.032857] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15434.037119] mei lpc_ich [15434.040603] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15434.044693] sha512_ssse3 acpi_ipmi [15434.048525] [15434.053743] mfd_core [15434.056272] [15434.056273] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15434.061230] wmi ipmi_si [15434.064719] ? queued_spin_lock_slowpath+0x18/0x2e0 [15434.066811] ipmi_devintf ipmi_msghandler [15434.069083] do_raw_spin_lock+0xb5/0xc0 [15434.071178] ioatdma acpi_pad [15434.076484] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15434.079014] acpi_power_meter [15434.083889] try_get_cap_refs+0x159/0x880 [ceph] [15434.087888] ixgbe igb [15434.091725] __ceph_get_caps+0x4a1/0x760 [ceph] [15434.094685] mdio crc32c_intel [15434.099473] ? __file_remove_privs+0x89/0x110 [15434.102432] nvme ptp [15434.107044] ? wake_bit_function+0x70/0x70 [15434.109401] nvme_core i2c_algo_bit [15434.113927] ? ceph_write_iter+0x260/0x770 [ceph] [15434.116975] pps_core dca [15434.121326] ceph_write_iter+0x260/0x770 [ceph] [15434.123597] fuse [15434.127690] ? lock_is_held_type+0xaa/0x120 [15434.131170] irq event stamp: 46612776 [15434.135870] vfs_write+0x324/0x4f0 [15434.138485] hardirqs last enabled at (46612775): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15434.143015] ksys_pwrite64+0x65/0xa0 [15434.144934] hardirqs last disabled at (46612776): [] sysvec_apic_timer_interrupt+0xe/0x90 [15434.149113] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15434.152768] softirqs last enabled at (46612774): [] __do_softirq+0x2be/0x371 [15434.156168] do_syscall_64+0x38/0x80 [15434.166324] softirqs last disabled at (46612769): [] irq_exit_rcu+0xb0/0xd0 [15434.169896] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15434.179620] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15434.184749] RIP: 0033:0x7f48f0eeb93b [15434.193433] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15434.197004] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15434.205514] Workqueue: events delayed_work [ceph] [15434.210559] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15434.219936] [15434.223508] ORIG_RAX: 0000000000000012 [15434.230977] RIP: 0010:lock_acquire+0xf0/0x2b0 [15434.249717] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15434.254413] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15434.259632] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15434.261122] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15434.264952] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15434.269303] [15434.269303] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15434.276426] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15434.295166] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15434.302289] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15434.307507] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15434.314635] [15434.316120] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15434.361068] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15434.368192] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15434.376270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15434.382007] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15434.389130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15434.396255] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15434.403393] Call Trace: [15434.405840] [15434.407853] ? watchdog_timer_fn+0x1b4/0x220 [15434.412117] ? __hrtimer_run_queues+0x1d3/0x360 [15434.416643] ? hrtimer_interrupt+0x104/0x240 [15434.420912] ? __local_bh_enable+0x50/0x60 [15434.425007] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15434.430228] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15434.435187] [15434.437283] [15434.439392] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15434.444706] ? lock_acquire+0xf0/0x2b0 [15434.448457] ? igrab+0x43/0x50 [15434.451508] ? lock_release+0xc5/0x260 [15434.455252] _raw_spin_lock+0x30/0x40 [15434.458917] ? igrab+0x1d/0x50 [15434.461967] igrab+0x1d/0x50 [15434.464846] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15434.469996] delayed_work+0x1d8/0x280 [ceph] [15434.474276] ? process_scheduled_works+0x22a/0x510 [15434.479067] process_scheduled_works+0x2a4/0x510 [15434.483682] worker_thread+0x188/0x340 [15434.487430] ? rescuer_thread+0x310/0x310 [15434.491434] kthread+0xf5/0x130 [15434.494572] ? kthread_complete_and_exit+0x20/0x20 [15434.499358] ret_from_fork+0x31/0x40 [15434.502935] ? kthread_complete_and_exit+0x20/0x20 [15434.507719] ret_from_fork_asm+0x11/0x20 [15434.511644] [15461.725476] watchdog: BUG: soft lockup - CPU#0 stuck for 3285s! [dbench:78683] [15461.732691] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15461.737475] watchdog: BUG: soft lockup - CPU#7 stuck for 12282s! [kworker/7:4:78808] [15461.788157] irq event stamp: 20189512 [15461.795889] Modules linked in: ceph libceph dns_resolver fscache netfs veth [15461.799547] hardirqs last enabled at (20189511): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15461.799552] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15461.816662] hardirqs last disabled at (20189512): [] sysvec_apic_timer_interrupt+0xe/0x90 [15461.816665] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [15461.834637] softirqs last enabled at (20189464): [] __do_softirq+0x2be/0x371 [15461.841327] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15461.850012] softirqs last disabled at (20189459): [] irq_exit_rcu+0xb0/0xd0 [15461.850016] coretemp kvm_intel kvm irqbypass i2c_i801 [15461.865996] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15461.871124] iTCO_wdt iTCO_vendor_support [15461.880087] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15461.880089] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15461.884089] i2c_smbus [15461.891562] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15461.897040] crct10dif_pclmul joydev [15461.899396] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15461.918135] crc32_pclmul ipmi_ssif [15461.921703] [15461.921704] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15461.926921] ghash_clmulni_intel [15461.930407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15461.931897] mei_me mei [15461.939020] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15461.942244] lpc_ich sha512_ssse3 [15461.949368] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15461.951812] acpi_ipmi mfd_core [15461.958936] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15461.962248] wmi ipmi_si [15461.969372] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15461.972508] ipmi_devintf [15461.979633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15461.982167] ipmi_msghandler ioatdma [15461.990243] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15461.992875] acpi_pad acpi_power_meter [15461.998614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15462.002184] ixgbe igb [15462.009308] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15462.013054] mdio crc32c_intel [15462.020176] Call Trace: [15462.020178] [15462.022533] nvme ptp [15462.029660] ? watchdog_timer_fn+0x1b4/0x220 [15462.032708] nvme_core [15462.035157] ? __hrtimer_run_queues+0x1d3/0x360 [15462.037172] i2c_algo_bit pps_core [15462.039446] ? hrtimer_interrupt+0x104/0x240 [15462.043706] dca fuse [15462.046066] ? __local_bh_enable+0x50/0x60 [15462.050588] [15462.050589] irq event stamp: 46835220 [15462.053988] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15462.058249] hardirqs last enabled at (46835219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15462.060522] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15462.064612] hardirqs last disabled at (46835220): [] sysvec_apic_timer_interrupt+0xe/0x90 [15462.066102] [15462.069760] softirqs last enabled at (46835218): [] __do_softirq+0x2be/0x371 [15462.074978] [15462.074979] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15462.085149] softirqs last disabled at (46835213): [] irq_exit_rcu+0xb0/0xd0 [15462.090115] ? queued_spin_lock_slowpath+0x16/0x2e0 [15462.099833] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15462.101933] do_raw_spin_lock+0xb5/0xc0 [15462.110615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15462.112715] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15462.118017] Workqueue: events delayed_work [ceph] [15462.126532] try_get_cap_refs+0x159/0x880 [ceph] [15462.131399] [15462.140781] __ceph_get_caps+0x4a1/0x760 [ceph] [15462.144608] RIP: 0010:rcu_is_watching+0x29/0x40 [15462.152083] ? __file_remove_privs+0x89/0x110 [15462.156876] Code: 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e <74> 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [15462.161576] ? wake_bit_function+0x70/0x70 [15462.166188] RSP: 0018:ffffc90002117d40 EFLAGS: 00000202 [15462.167681] ? ceph_write_iter+0x260/0x770 [ceph] [15462.172203] [15462.172204] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [15462.176725] ceph_write_iter+0x260/0x770 [ceph] [15462.181075] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [15462.199820] ? lock_is_held_type+0xaa/0x120 [15462.203904] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [15462.209125] vfs_write+0x324/0x4f0 [15462.213819] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [15462.215316] ksys_pwrite64+0x65/0xa0 [15462.222434] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15462.226961] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15462.234082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15462.238263] do_syscall_64+0x38/0x80 [15462.245386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15462.248782] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15462.255907] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15462.259478] RIP: 0033:0x7f48f0eeb93b [15462.266602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15462.271732] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15462.279808] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15462.283381] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15462.289117] Call Trace: [15462.289119] [15462.294168] ORIG_RAX: 0000000000000012 [15462.301296] ? watchdog_timer_fn+0x1b4/0x220 [15462.304877] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15462.311998] ? __hrtimer_run_queues+0x1d3/0x360 [15462.330735] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15462.337881] ? hrtimer_interrupt+0x104/0x240 [15462.343094] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15462.345538] ? __local_bh_enable+0x50/0x60 [15462.347548] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15462.351382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15462.355643] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15462.362767] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15462.367297] [15462.374415] [15462.418599] [15462.420699] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15462.426003] ? igrab+0x43/0x50 [15462.429054] ? igrab+0x43/0x50 [15462.432104] ? rcu_is_watching+0x29/0x40 [15462.436020] lock_release+0x1d7/0x260 [15462.439680] _raw_spin_unlock+0x1b/0x40 [15462.443519] igrab+0x43/0x50 [15462.446396] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15462.451544] delayed_work+0x1d8/0x280 [ceph] [15462.455827] ? process_scheduled_works+0x22a/0x510 [15462.460617] process_scheduled_works+0x2a4/0x510 [15462.465232] worker_thread+0x188/0x340 [15462.468982] ? rescuer_thread+0x310/0x310 [15462.472983] kthread+0xf5/0x130 [15462.476121] ? kthread_complete_and_exit+0x20/0x20 [15462.480907] ret_from_fork+0x31/0x40 [15462.484486] ? kthread_complete_and_exit+0x20/0x20 [15462.489270] ret_from_fork_asm+0x11/0x20 [15462.493202] [15489.725964] watchdog: BUG: soft lockup - CPU#0 stuck for 3311s! [dbench:78683] [15489.733178] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15489.737964] watchdog: BUG: soft lockup - CPU#7 stuck for 12308s! [kworker/7:4:78808] [15489.788620] irq event stamp: 20247570 [15489.788621] hardirqs last enabled at (20247569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15489.796367] Modules linked in: [15489.800025] hardirqs last disabled at (20247570): [] sysvec_apic_timer_interrupt+0xe/0x90 [15489.810183] ceph [15489.813234] softirqs last enabled at (20247560): [] __do_softirq+0x2be/0x371 [15489.822956] libceph [15489.824880] softirqs last disabled at (20247555): [] irq_exit_rcu+0xb0/0xd0 [15489.833564] dns_resolver fscache [15489.835749] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15489.844260] netfs veth [15489.847572] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15489.856515] nft_chain_nat [15489.858959] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [15489.866429] xt_MASQUERADE nf_nat [15489.869133] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [15489.874610] nf_conntrack nf_defrag_ipv6 [15489.877920] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15489.896659] nf_defrag_ipv4 nft_compat [15489.900577] [15489.900578] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15489.905795] nf_tables nfnetlink [15489.909537] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15489.911028] bridge stp [15489.918153] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15489.921376] llc xfs [15489.928502] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15489.930944] libcrc32c sunrpc [15489.938069] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15489.940252] intel_rapl_msr [15489.947379] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15489.950359] intel_rapl_common sb_edac [15489.957482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15489.960274] x86_pkg_temp_thermal coretemp [15489.968361] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15489.972104] kvm_intel kvm [15489.977841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15489.981931] irqbypass i2c_i801 [15489.989057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15489.991759] iTCO_wdt iTCO_vendor_support [15489.998883] Call Trace: [15489.998885] [15490.002021] i2c_smbus crct10dif_pclmul [15490.009148] ? watchdog_timer_fn+0x1b4/0x220 [15490.013150] joydev crc32_pclmul [15490.015596] ? __hrtimer_run_queues+0x1d3/0x360 [15490.017604] ipmi_ssif ghash_clmulni_intel [15490.021440] ? hrtimer_interrupt+0x104/0x240 [15490.025699] mei_me mei [15490.028924] ? __local_bh_enable+0x50/0x60 [15490.033448] lpc_ich sha512_ssse3 [15490.037542] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15490.041802] acpi_ipmi mfd_core [15490.044248] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15490.048336] wmi ipmi_si [15490.051665] [15490.056882] ipmi_devintf ipmi_msghandler [15490.060021] [15490.060022] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15490.064977] ioatdma acpi_pad [15490.067512] ? queued_spin_lock_slowpath+0x2b/0x2e0 [15490.069606] acpi_power_meter ixgbe [15490.073613] do_raw_spin_lock+0xb5/0xc0 [15490.075706] igb mdio [15490.081012] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15490.083974] crc32c_intel nvme [15490.088850] try_get_cap_refs+0x159/0x880 [ceph] [15490.092331] ptp nvme_core [15490.096165] __ceph_get_caps+0x4a1/0x760 [ceph] [15490.098431] i2c_algo_bit [15490.103219] ? __file_remove_privs+0x89/0x110 [15490.106268] pps_core dca [15490.110877] ? wake_bit_function+0x70/0x70 [15490.113582] fuse [15490.118109] ? ceph_write_iter+0x260/0x770 [ceph] [15490.120724] irq event stamp: 47057340 [15490.125075] ceph_write_iter+0x260/0x770 [ceph] [15490.127692] hardirqs last enabled at (47057339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15490.131787] ? lock_is_held_type+0xaa/0x120 [15490.133705] hardirqs last disabled at (47057340): [] sysvec_apic_timer_interrupt+0xe/0x90 [15490.138406] vfs_write+0x324/0x4f0 [15490.142060] softirqs last enabled at (47057338): [] __do_softirq+0x2be/0x371 [15490.146588] ksys_pwrite64+0x65/0xa0 [15490.156742] softirqs last disabled at (47057333): [] irq_exit_rcu+0xb0/0xd0 [15490.160921] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15490.170653] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15490.174050] do_syscall_64+0x38/0x80 [15490.182733] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15490.182734] Workqueue: events delayed_work [ceph] [15490.186305] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15490.194815] [15490.199945] RIP: 0033:0x7f48f0eeb93b [15490.209323] RIP: 0010:lock_acquire+0xf0/0x2b0 [15490.212896] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15490.220365] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15490.225063] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15490.230106] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15490.231598] ORIG_RAX: 0000000000000012 [15490.235167] [15490.235168] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15490.239519] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15490.258257] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15490.276994] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15490.282212] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15490.287430] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15490.291260] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15490.292751] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15490.299874] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15490.307000] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15490.314123] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15490.321252] [15490.328370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15490.379991] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15490.387116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15490.394240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15490.401369] Call Trace: [15490.403819] [15490.405827] ? watchdog_timer_fn+0x1b4/0x220 [15490.410093] ? __hrtimer_run_queues+0x1d3/0x360 [15490.414619] ? hrtimer_interrupt+0x104/0x240 [15490.418888] ? __local_bh_enable+0x50/0x60 [15490.422983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15490.428206] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15490.433162] [15490.435260] [15490.437369] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15490.442684] ? lock_acquire+0xf0/0x2b0 [15490.446432] ? igrab+0x43/0x50 [15490.449482] ? lock_release+0xc5/0x260 [15490.453227] _raw_spin_lock+0x30/0x40 [15490.456883] ? igrab+0x1d/0x50 [15490.459933] igrab+0x1d/0x50 [15490.462812] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15490.467961] delayed_work+0x1d8/0x280 [ceph] [15490.472244] ? process_scheduled_works+0x22a/0x510 [15490.477035] process_scheduled_works+0x2a4/0x510 [15490.481665] worker_thread+0x188/0x340 [15490.485415] ? rescuer_thread+0x310/0x310 [15490.489419] kthread+0xf5/0x130 [15490.492557] ? kthread_complete_and_exit+0x20/0x20 [15490.497342] ret_from_fork+0x31/0x40 [15490.500938] ? kthread_complete_and_exit+0x20/0x20 [15490.505720] ret_from_fork_asm+0x11/0x20 [15490.509657] [15517.726455] watchdog: BUG: soft lockup - CPU#0 stuck for 3338s! [dbench:78683] [15517.733673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15517.738454] watchdog: BUG: soft lockup - CPU#7 stuck for 12335s! [kworker/7:4:78808] [15517.789137] irq event stamp: 20305710 [15517.796871] Modules linked in: ceph libceph dns_resolver fscache netfs veth [15517.800530] hardirqs last enabled at (20305709): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15517.807489] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [15517.817646] hardirqs last disabled at (20305710): [] sysvec_apic_timer_interrupt+0xe/0x90 [15517.817649] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [15517.834322] softirqs last enabled at (20305662): [] __do_softirq+0x2be/0x371 [15517.841453] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15517.850139] softirqs last disabled at (20305657): [] irq_exit_rcu+0xb0/0xd0 [15517.850143] coretemp kvm_intel kvm irqbypass i2c_i801 [15517.866996] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15517.872139] iTCO_wdt iTCO_vendor_support [15517.881088] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15517.881090] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15517.885091] i2c_smbus crct10dif_pclmul [15517.892563] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15517.898039] joydev crc32_pclmul [15517.901871] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15517.920609] ipmi_ssif [15517.923848] [15517.923849] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15517.929067] ghash_clmulni_intel [15517.931423] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15517.932914] mei_me mei [15517.940038] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15517.943262] lpc_ich [15517.950388] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15517.952848] sha512_ssse3 acpi_ipmi [15517.959972] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15517.962156] mfd_core wmi [15517.969282] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15517.972765] ipmi_si ipmi_devintf [15517.979889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15517.982505] ipmi_msghandler [15517.990583] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15517.993895] ioatdma acpi_pad [15517.999633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15518.002509] acpi_power_meter [15518.009633] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15518.012598] ixgbe igb [15518.019721] Call Trace: [15518.019724] [15518.022686] mdio [15518.029812] ? watchdog_timer_fn+0x1b4/0x220 [15518.032167] crc32c_intel [15518.034611] ? __hrtimer_run_queues+0x1d3/0x360 [15518.036621] nvme ptp [15518.038550] ? hrtimer_interrupt+0x104/0x240 [15518.042812] nvme_core [15518.045429] ? __local_bh_enable+0x50/0x60 [15518.049951] i2c_algo_bit pps_core [15518.052224] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15518.056486] dca [15518.058851] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15518.062944] fuse [15518.066342] [15518.071558] irq event stamp: 47278886 [15518.073395] [15518.078353] hardirqs last enabled at (47278885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15518.080277] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15518.082373] hardirqs last disabled at (47278886): [] sysvec_apic_timer_interrupt+0xe/0x90 [15518.086037] ? queued_spin_lock_slowpath+0x16/0x2e0 [15518.088129] softirqs last enabled at (47278884): [] __do_softirq+0x2be/0x371 [15518.098307] do_raw_spin_lock+0xb5/0xc0 [15518.103608] softirqs last disabled at (47278879): [] irq_exit_rcu+0xb0/0xd0 [15518.113333] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15518.118203] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15518.126891] try_get_cap_refs+0x159/0x880 [ceph] [15518.130719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15518.139234] __ceph_get_caps+0x4a1/0x760 [ceph] [15518.144012] Workqueue: events delayed_work [ceph] [15518.153395] ? __file_remove_privs+0x89/0x110 [15518.158002] [15518.158002] RIP: 0010:lock_acquire+0xf0/0x2b0 [15518.165473] ? wake_bit_function+0x70/0x70 [15518.169997] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15518.174697] ? ceph_write_iter+0x260/0x770 [ceph] [15518.179044] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15518.180535] ceph_write_iter+0x260/0x770 [ceph] [15518.184887] [15518.184887] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15518.188981] ? lock_is_held_type+0xaa/0x120 [15518.207715] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15518.212415] vfs_write+0x324/0x4f0 [15518.217628] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15518.222157] ksys_pwrite64+0x65/0xa0 [15518.223645] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15518.230768] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15518.234945] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15518.242070] do_syscall_64+0x38/0x80 [15518.245467] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15518.252591] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15518.256161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15518.263286] RIP: 0033:0x7f48f0eeb93b [15518.268416] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15518.275541] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15518.279112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15518.287206] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15518.292250] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15518.297988] ORIG_RAX: 0000000000000012 [15518.301557] Call Trace: [15518.301559] [15518.308683] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15518.327420] ? watchdog_timer_fn+0x1b4/0x220 [15518.334544] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15518.339764] ? __hrtimer_run_queues+0x1d3/0x360 [15518.346885] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15518.350721] ? hrtimer_interrupt+0x104/0x240 [15518.353160] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15518.355172] ? __local_bh_enable+0x50/0x60 [15518.362296] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15518.366563] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15518.373689] [15518.378207] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15518.420296] [15518.422392] [15518.424489] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15518.429797] ? lock_acquire+0xf0/0x2b0 [15518.433547] ? igrab+0x43/0x50 [15518.436598] ? lock_release+0xc5/0x260 [15518.440341] _raw_spin_lock+0x30/0x40 [15518.443998] ? igrab+0x1d/0x50 [15518.447050] igrab+0x1d/0x50 [15518.449927] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15518.455077] delayed_work+0x1d8/0x280 [ceph] [15518.459358] ? process_scheduled_works+0x22a/0x510 [15518.464149] process_scheduled_works+0x2a4/0x510 [15518.468764] worker_thread+0x188/0x340 [15518.472513] ? rescuer_thread+0x310/0x310 [15518.476517] kthread+0xf5/0x130 [15518.479664] ? kthread_complete_and_exit+0x20/0x20 [15518.484448] ret_from_fork+0x31/0x40 [15518.488027] ? kthread_complete_and_exit+0x20/0x20 [15518.492811] ret_from_fork_asm+0x11/0x20 [15518.496744] [15519.679855] rcu: INFO: rcu_preempt self-detected stall on CPU [15519.685607] rcu: 7-....: (3482948 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/3481756 fqs=1729458 [15519.696027] rcu: (t=3583006 jiffies g=1309609 q=180415 ncpus=8) [15519.702024] Sending NMI from CPU 7 to CPUs 0: [15519.706375] NMI backtrace for cpu 0 [15519.706377] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15519.706379] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15519.706380] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15519.706383] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15519.706385] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15519.706386] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15519.706387] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15519.706388] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15519.706388] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15519.706389] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15519.706390] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15519.706391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15519.706392] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15519.706393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15519.706393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15519.706394] Call Trace: [15519.706395] [15519.706396] ? nmi_cpu_backtrace+0xad/0x120 [15519.706400] ? nmi_cpu_backtrace_handler+0x11/0x20 [15519.706401] ? nmi_handle+0xd4/0x260 [15519.706406] ? default_do_nmi+0x49/0xf0 [15519.706408] ? exc_nmi+0x11e/0x150 [15519.706409] ? end_repeat_nmi+0x16/0x67 [15519.706413] ? queued_spin_lock_slowpath+0x18/0x2e0 [15519.706416] ? queued_spin_lock_slowpath+0x18/0x2e0 [15519.706418] ? queued_spin_lock_slowpath+0x18/0x2e0 [15519.706420] [15519.706421] [15519.706422] do_raw_spin_lock+0xb5/0xc0 [15519.706424] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15519.706438] try_get_cap_refs+0x159/0x880 [ceph] [15519.706453] __ceph_get_caps+0x4a1/0x760 [ceph] [15519.706466] ? __file_remove_privs+0x89/0x110 [15519.706468] ? wake_bit_function+0x70/0x70 [15519.706470] ? ceph_write_iter+0x260/0x770 [ceph] [15519.706481] ceph_write_iter+0x260/0x770 [ceph] [15519.706493] ? lock_is_held_type+0xaa/0x120 [15519.706495] vfs_write+0x324/0x4f0 [15519.706499] ksys_pwrite64+0x65/0xa0 [15519.706501] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15519.706504] do_syscall_64+0x38/0x80 [15519.706506] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15519.706508] RIP: 0033:0x7f48f0eeb93b [15519.706510] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15519.706511] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [15519.706512] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15519.706513] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15519.706514] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15519.706514] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15519.706515] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15519.706518] [15519.707374] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15520.011204] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15520.018675] Workqueue: events delayed_work [ceph] [15520.023390] RIP: 0010:lock_release+0x14e/0x260 [15520.027826] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [15520.046564] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [15520.051781] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [15520.058905] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [15520.066028] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [15520.073153] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [15520.080278] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [15520.087402] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15520.095480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15520.101216] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15520.108342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15520.115466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15520.122589] Call Trace: [15520.125035] [15520.127045] ? rcu_dump_cpu_stacks+0xd1/0x180 [15520.131403] ? rcu_sched_clock_irq+0x633/0x1390 [15520.135928] ? lock_release+0xc5/0x260 [15520.139671] ? lock_is_held_type+0xaa/0x120 [15520.143850] ? lock_is_held_type+0xaa/0x120 [15520.148027] ? tick_sched_do_timer+0xa0/0xa0 [15520.152299] ? rcu_read_lock_sched_held+0x56/0x90 [15520.156999] ? tick_sched_do_timer+0xa0/0xa0 [15520.161269] ? update_process_times+0x5f/0x90 [15520.165621] ? tick_sched_handle.isra.16+0x1e/0x50 [15520.170404] ? tick_sched_timer+0x6f/0x80 [15520.174410] ? __hrtimer_run_queues+0x1d3/0x360 [15520.178935] ? hrtimer_interrupt+0x104/0x240 [15520.183205] ? __local_bh_enable+0x50/0x60 [15520.187298] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15520.192521] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15520.197480] [15520.199576] [15520.201673] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15520.206979] ? igrab+0x43/0x50 [15520.210032] ? lock_release+0x14e/0x260 [15520.213869] ? lock_release+0xc5/0x260 [15520.217614] _raw_spin_unlock+0x1b/0x40 [15520.221443] igrab+0x43/0x50 [15520.224322] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15520.229472] delayed_work+0x1d8/0x280 [ceph] [15520.233751] ? process_scheduled_works+0x22a/0x510 [15520.238542] process_scheduled_works+0x2a4/0x510 [15520.243158] worker_thread+0x188/0x340 [15520.246906] ? rescuer_thread+0x310/0x310 [15520.250911] kthread+0xf5/0x130 [15520.254048] ? kthread_complete_and_exit+0x20/0x20 [15520.258833] ret_from_fork+0x31/0x40 [15520.262403] ? kthread_complete_and_exit+0x20/0x20 [15520.267186] ret_from_fork_asm+0x11/0x20 [15520.271110] [15545.726943] watchdog: BUG: soft lockup - CPU#0 stuck for 3364s! [dbench:78683] [15545.734161] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15545.738943] watchdog: BUG: soft lockup - CPU#7 stuck for 12361s! [kworker/7:4:78808] [15545.789619] irq event stamp: 20363720 [15545.789620] hardirqs last enabled at (20363719): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15545.797350] Modules linked in: [15545.801008] hardirqs last disabled at (20363720): [] sysvec_apic_timer_interrupt+0xe/0x90 [15545.811165] ceph [15545.814217] softirqs last enabled at (20363676): [] __do_softirq+0x2be/0x371 [15545.823940] libceph [15545.825863] softirqs last disabled at (20363671): [] irq_exit_rcu+0xb0/0xd0 [15545.834550] dns_resolver fscache [15545.836732] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15545.845242] netfs [15545.848554] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15545.857499] veth nft_chain_nat [15545.859509] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15545.866981] xt_MASQUERADE nf_nat [15545.870117] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15545.875595] nf_conntrack [15545.878906] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15545.897644] nf_defrag_ipv6 [15545.900260] [15545.900261] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15545.905478] nf_defrag_ipv4 nft_compat [15545.908269] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15545.909758] nf_tables nfnetlink [15545.916882] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15545.920627] bridge [15545.927751] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15545.930975] stp llc [15545.938099] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15545.940196] xfs libcrc32c [15545.947338] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15545.949524] sunrpc intel_rapl_msr [15545.956646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15545.959350] intel_rapl_common [15545.967427] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15545.970825] sb_edac x86_pkg_temp_thermal [15545.976563] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15545.979620] coretemp kvm_intel [15545.986746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15545.990751] kvm irqbypass [15545.997876] Call Trace: [15545.997877] [15546.001013] i2c_i801 iTCO_wdt [15546.008136] ? watchdog_timer_fn+0x1b4/0x220 [15546.010839] iTCO_vendor_support i2c_smbus [15546.013286] ? __hrtimer_run_queues+0x1d3/0x360 [15546.015296] crct10dif_pclmul joydev [15546.018351] ? hrtimer_interrupt+0x104/0x240 [15546.022620] crc32_pclmul ipmi_ssif [15546.026709] ? __local_bh_enable+0x50/0x60 [15546.031233] ghash_clmulni_intel [15546.034808] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15546.039067] mei_me mei [15546.042554] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15546.046643] lpc_ich sha512_ssse3 [15546.049868] [15546.055084] acpi_ipmi [15546.057528] [15546.057530] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15546.062486] mfd_core wmi ipmi_si [15546.065801] ? queued_spin_lock_slowpath+0x16/0x2e0 [15546.067895] ipmi_devintf ipmi_msghandler [15546.070253] do_raw_spin_lock+0xb5/0xc0 [15546.072348] ioatdma acpi_pad [15546.077654] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15546.080964] acpi_power_meter [15546.085836] try_get_cap_refs+0x159/0x880 [ceph] [15546.089839] ixgbe igb [15546.093673] __ceph_get_caps+0x4a1/0x760 [ceph] [15546.096633] mdio crc32c_intel [15546.101421] ? __file_remove_privs+0x89/0x110 [15546.104381] nvme ptp [15546.108992] ? wake_bit_function+0x70/0x70 [15546.111350] nvme_core [15546.115877] ? ceph_write_iter+0x260/0x770 [ceph] [15546.118925] i2c_algo_bit pps_core [15546.123276] ceph_write_iter+0x260/0x770 [ceph] [15546.125545] dca fuse [15546.129641] ? lock_is_held_type+0xaa/0x120 [15546.131994] [15546.131995] irq event stamp: 47496138 [15546.136694] vfs_write+0x324/0x4f0 [15546.140089] hardirqs last enabled at (47496137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15546.144626] ksys_pwrite64+0x65/0xa0 [15546.146892] hardirqs last disabled at (47496138): [] sysvec_apic_timer_interrupt+0xe/0x90 [15546.151071] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15546.152560] softirqs last enabled at (47496136): [] __do_softirq+0x2be/0x371 [15546.156219] do_syscall_64+0x38/0x80 [15546.159623] softirqs last disabled at (47496131): [] irq_exit_rcu+0xb0/0xd0 [15546.169782] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15546.173352] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15546.183077] RIP: 0033:0x7f48f0eeb93b [15546.188206] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15546.196892] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15546.200463] Workqueue: events delayed_work [ceph] [15546.208973] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15546.214018] [15546.214019] RIP: 0010:lock_acquire+0xf0/0x2b0 [15546.223393] ORIG_RAX: 0000000000000012 [15546.226965] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15546.234437] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15546.253174] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15546.257870] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15546.263088] [15546.263089] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15546.264578] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15546.268929] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15546.272760] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15546.291497] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15546.298638] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15546.303857] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15546.310987] [15546.312472] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15546.371660] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15546.379736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15546.385473] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15546.392599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15546.399722] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15546.406845] Call Trace: [15546.409290] [15546.411300] ? watchdog_timer_fn+0x1b4/0x220 [15546.415565] ? __hrtimer_run_queues+0x1d3/0x360 [15546.420092] ? hrtimer_interrupt+0x104/0x240 [15546.424360] ? __local_bh_enable+0x50/0x60 [15546.428454] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15546.433679] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15546.438647] [15546.440742] [15546.442839] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15546.448146] ? lock_acquire+0xf0/0x2b0 [15546.451897] ? igrab+0x43/0x50 [15546.454948] ? lock_release+0xc5/0x260 [15546.458692] _raw_spin_lock+0x30/0x40 [15546.462349] ? igrab+0x1d/0x50 [15546.465398] igrab+0x1d/0x50 [15546.468277] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15546.473426] delayed_work+0x1d8/0x280 [ceph] [15546.477707] ? process_scheduled_works+0x22a/0x510 [15546.482490] process_scheduled_works+0x2a4/0x510 [15546.487105] worker_thread+0x188/0x340 [15546.490855] ? rescuer_thread+0x310/0x310 [15546.494858] kthread+0xf5/0x130 [15546.497995] ? kthread_complete_and_exit+0x20/0x20 [15546.502779] ret_from_fork+0x31/0x40 [15546.506351] ? kthread_complete_and_exit+0x20/0x20 [15546.511134] ret_from_fork_asm+0x11/0x20 [15546.515058] [15573.727433] watchdog: BUG: soft lockup - CPU#0 stuck for 3390s! [dbench:78683] [15573.734650] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15573.739432] watchdog: BUG: soft lockup - CPU#7 stuck for 12387s! [kworker/7:4:78808] [15573.790108] irq event stamp: 20421896 [15573.797838] Modules linked in: ceph libceph dns_resolver fscache netfs veth [15573.801496] hardirqs last enabled at (20421895): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15573.808453] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15573.818613] hardirqs last disabled at (20421896): [] sysvec_apic_timer_interrupt+0xe/0x90 [15573.826863] nft_compat nf_tables nfnetlink bridge stp llc xfs [15573.836587] softirqs last enabled at (20421858): [] __do_softirq+0x2be/0x371 [15573.842412] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15573.851107] softirqs last disabled at (20421853): [] irq_exit_rcu+0xb0/0xd0 [15573.859450] coretemp kvm_intel kvm irqbypass [15573.867961] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15573.867963] i2c_i801 [15573.867964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15573.872313] iTCO_wdt iTCO_vendor_support [15573.881256] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15573.883527] i2c_smbus crct10dif_pclmul [15573.890998] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15573.895002] joydev crc32_pclmul [15573.900479] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15573.904310] ipmi_ssif [15573.923046] [15573.926271] ghash_clmulni_intel [15573.931490] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15573.933847] mei_me mei [15573.935337] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15573.938562] lpc_ich sha512_ssse3 [15573.945685] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15573.948128] acpi_ipmi mfd_core [15573.955253] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15573.958563] wmi ipmi_si [15573.965687] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15573.968830] ipmi_devintf [15573.975950] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15573.978480] ipmi_msghandler [15573.985606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15573.988223] ioatdma acpi_pad [15573.996300] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15573.999176] acpi_power_meter ixgbe [15574.004915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15574.007879] igb mdio [15574.015003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15574.018487] crc32c_intel nvme [15574.025611] Call Trace: [15574.025614] [15574.027882] ptp nvme_core [15574.035007] ? watchdog_timer_fn+0x1b4/0x220 [15574.038057] i2c_algo_bit pps_core [15574.040501] ? __hrtimer_run_queues+0x1d3/0x360 [15574.042510] dca fuse [15574.045217] ? hrtimer_interrupt+0x104/0x240 [15574.049480] [15574.049481] irq event stamp: 47718416 [15574.052876] ? __local_bh_enable+0x50/0x60 [15574.057401] hardirqs last enabled at (47718415): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15574.059675] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15574.063935] hardirqs last disabled at (47718416): [] sysvec_apic_timer_interrupt+0xe/0x90 [15574.065427] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15574.069084] softirqs last enabled at (47718414): [] __do_softirq+0x2be/0x371 [15574.073174] [15574.083331] softirqs last disabled at (47718409): [] irq_exit_rcu+0xb0/0xd0 [15574.088549] [15574.088550] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15574.098273] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15574.103234] ? queued_spin_lock_slowpath+0x16/0x2e0 [15574.111916] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15574.114013] do_raw_spin_lock+0xb5/0xc0 [15574.122524] Workqueue: events delayed_work [ceph] [15574.124621] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15574.129925] [15574.139306] try_get_cap_refs+0x159/0x880 [ceph] [15574.144174] RIP: 0010:lock_acquire+0xf0/0x2b0 [15574.151649] __ceph_get_caps+0x4a1/0x760 [ceph] [15574.155475] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15574.160175] ? __file_remove_privs+0x89/0x110 [15574.164956] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15574.166446] ? wake_bit_function+0x70/0x70 [15574.171058] [15574.171059] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15574.175411] ? ceph_write_iter+0x260/0x770 [ceph] [15574.179931] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15574.198671] ceph_write_iter+0x260/0x770 [ceph] [15574.203022] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15574.208242] ? lock_is_held_type+0xaa/0x120 [15574.212327] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15574.213839] vfs_write+0x324/0x4f0 [15574.220961] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15574.225663] ksys_pwrite64+0x65/0xa0 [15574.232783] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15574.237308] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15574.244431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15574.248610] do_syscall_64+0x38/0x80 [15574.255733] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15574.259134] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15574.266254] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15574.269838] RIP: 0033:0x7f48f0eeb93b [15574.277910] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15574.283042] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15574.288779] Call Trace: [15574.288780] [15574.292349] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15574.299474] ? watchdog_timer_fn+0x1b4/0x220 [15574.304517] ORIG_RAX: 0000000000000012 [15574.311644] ? __hrtimer_run_queues+0x1d3/0x360 [15574.315213] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15574.322339] ? hrtimer_interrupt+0x104/0x240 [15574.341074] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15574.343518] ? __local_bh_enable+0x50/0x60 [15574.345529] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15574.350750] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15574.355011] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15574.358842] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15574.363366] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15574.370490] [15574.374758] [15574.381879] [15574.423930] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15574.429239] ? lock_acquire+0xf0/0x2b0 [15574.432989] ? igrab+0x43/0x50 [15574.436039] ? lock_release+0xc5/0x260 [15574.439783] _raw_spin_lock+0x30/0x40 [15574.443447] ? igrab+0x1d/0x50 [15574.446497] igrab+0x1d/0x50 [15574.449377] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15574.454528] delayed_work+0x1d8/0x280 [ceph] [15574.458808] ? process_scheduled_works+0x22a/0x510 [15574.463598] process_scheduled_works+0x2a4/0x510 [15574.468214] worker_thread+0x188/0x340 [15574.471962] ? rescuer_thread+0x310/0x310 [15574.475965] kthread+0xf5/0x130 [15574.479103] ? kthread_complete_and_exit+0x20/0x20 [15574.483887] ret_from_fork+0x31/0x40 [15574.487467] ? kthread_complete_and_exit+0x20/0x20 [15574.492251] ret_from_fork_asm+0x11/0x20 [15574.496182] [15601.727922] watchdog: BUG: soft lockup - CPU#0 stuck for 3416s! [dbench:78683] [15601.735136] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15601.739922] watchdog: BUG: soft lockup - CPU#7 stuck for 12413s! [kworker/7:4:78808] [15601.790593] irq event stamp: 20479916 [15601.790594] hardirqs last enabled at (20479915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15601.798326] Modules linked in: [15601.801982] hardirqs last disabled at (20479916): [] sysvec_apic_timer_interrupt+0xe/0x90 [15601.812138] ceph [15601.815189] softirqs last enabled at (20479878): [] __do_softirq+0x2be/0x371 [15601.824913] libceph [15601.826839] softirqs last disabled at (20479873): [] irq_exit_rcu+0xb0/0xd0 [15601.835522] dns_resolver fscache [15601.837707] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15601.846217] netfs [15601.849528] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15601.858473] veth nft_chain_nat [15601.860483] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15601.867954] xt_MASQUERADE nf_nat [15601.871092] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15601.876570] nf_conntrack nf_defrag_ipv6 [15601.879879] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15601.898616] nf_defrag_ipv4 [15601.902535] [15601.902536] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15601.907752] nft_compat nf_tables [15601.910542] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15601.912034] nfnetlink bridge [15601.919157] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15601.922468] stp llc [15601.929592] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15601.932574] xfs libcrc32c [15601.939698] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15601.941881] sunrpc intel_rapl_msr [15601.949006] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15601.951710] intel_rapl_common sb_edac [15601.958834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15601.962230] x86_pkg_temp_thermal coretemp [15601.970318] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15601.974062] kvm_intel kvm [15601.979800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15601.983889] irqbypass i2c_i801 [15601.991015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15601.993719] iTCO_wdt iTCO_vendor_support [15602.000843] Call Trace: [15602.000844] [15602.003980] i2c_smbus [15602.011105] ? watchdog_timer_fn+0x1b4/0x220 [15602.015109] crct10dif_pclmul joydev [15602.017552] ? __hrtimer_run_queues+0x1d3/0x360 [15602.019563] crc32_pclmul ipmi_ssif [15602.021923] ? hrtimer_interrupt+0x104/0x240 [15602.026183] ghash_clmulni_intel mei_me [15602.029756] ? __local_bh_enable+0x50/0x60 [15602.034278] mei [15602.037765] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15602.042027] lpc_ich sha512_ssse3 [15602.045859] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15602.049949] acpi_ipmi mfd_core [15602.051787] [15602.057004] wmi ipmi_si [15602.060320] [15602.065279] ipmi_devintf [15602.068420] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15602.070515] ipmi_msghandler [15602.073050] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15602.075143] ioatdma [15602.077761] do_raw_spin_lock+0xb5/0xc0 [15602.083064] acpi_pad acpi_power_meter ixgbe [15602.085944] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15602.090812] igb mdio [15602.093000] try_get_cap_refs+0x159/0x880 [ceph] [15602.096828] crc32c_intel nvme [15602.101096] __ceph_get_caps+0x4a1/0x760 [ceph] [15602.105875] ptp nvme_core [15602.108150] ? __file_remove_privs+0x89/0x110 [15602.112758] i2c_algo_bit pps_core [15602.115809] ? wake_bit_function+0x70/0x70 [15602.120332] dca fuse [15602.123037] ? ceph_write_iter+0x260/0x770 [ceph] [15602.127387] [15602.127388] irq event stamp: 47940780 [15602.130785] ceph_write_iter+0x260/0x770 [ceph] [15602.134873] hardirqs last enabled at (47940779): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15602.137150] ? lock_is_held_type+0xaa/0x120 [15602.141842] hardirqs last disabled at (47940780): [] sysvec_apic_timer_interrupt+0xe/0x90 [15602.143337] vfs_write+0x324/0x4f0 [15602.146991] softirqs last enabled at (47940778): [] __do_softirq+0x2be/0x371 [15602.151520] ksys_pwrite64+0x65/0xa0 [15602.161673] softirqs last disabled at (47940773): [] irq_exit_rcu+0xb0/0xd0 [15602.165851] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15602.175575] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15602.178974] do_syscall_64+0x38/0x80 [15602.187655] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15602.187657] Workqueue: events delayed_work [ceph] [15602.191227] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15602.199737] [15602.199738] RIP: 0010:__list_del_entry_valid_or_report+0x0/0xd0 [15602.204869] RIP: 0033:0x7f48f0eeb93b [15602.214247] Code: 1d 82 e8 83 0c b0 ff 0f 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 [15602.217817] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15602.225286] RSP: 0018:ffffc90002117db0 EFLAGS: 00000212 [15602.229985] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15602.235028] [15602.235028] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 0000000100e976c0 [15602.236521] ORIG_RAX: 0000000000000012 [15602.242429] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [15602.246000] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15602.264737] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [15602.283477] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15602.288694] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15602.293912] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15602.295402] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15602.302527] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15602.306356] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15602.313480] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15602.320605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15602.327733] [15602.334854] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15602.393609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15602.400729] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15602.407854] Call Trace: [15602.410297] [15602.412326] ? watchdog_timer_fn+0x1b4/0x220 [15602.416591] ? __hrtimer_run_queues+0x1d3/0x360 [15602.421135] ? hrtimer_interrupt+0x104/0x240 [15602.425403] ? __local_bh_enable+0x50/0x60 [15602.429499] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15602.434722] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15602.439679] [15602.441777] [15602.443874] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15602.449182] ? __list_add_valid_or_report+0xa0/0xa0 [15602.454059] ceph_check_delayed_caps+0x8f/0x270 [ceph] [15602.459207] delayed_work+0x1d8/0x280 [ceph] [15602.463489] ? process_scheduled_works+0x22a/0x510 [15602.468278] process_scheduled_works+0x2a4/0x510 [15602.472896] worker_thread+0x188/0x340 [15602.476643] ? rescuer_thread+0x310/0x310 [15602.480647] kthread+0xf5/0x130 [15602.483785] ? kthread_complete_and_exit+0x20/0x20 [15602.488570] ret_from_fork+0x31/0x40 [15602.492148] ? kthread_complete_and_exit+0x20/0x20 [15602.496933] ret_from_fork_asm+0x11/0x20 [15602.500855] [15629.728412] watchdog: BUG: soft lockup - CPU#0 stuck for 3442s! [dbench:78683] [15629.735630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15629.740411] watchdog: BUG: soft lockup - CPU#7 stuck for 12439s! [kworker/7:4:78808] [15629.791095] irq event stamp: 20538090 [15629.798828] Modules linked in: ceph libceph dns_resolver fscache netfs [15629.802485] hardirqs last enabled at (20538089): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15629.809004] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [15629.819162] hardirqs last disabled at (20538090): [] sysvec_apic_timer_interrupt+0xe/0x90 [15629.826554] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [15629.836277] softirqs last enabled at (20538052): [] __do_softirq+0x2be/0x371 [15629.843401] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15629.852094] softirqs last disabled at (20538047): [] irq_exit_rcu+0xb0/0xd0 [15629.860434] coretemp kvm_intel kvm irqbypass [15629.868945] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15629.868947] i2c_i801 [15629.868948] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15629.873295] iTCO_wdt iTCO_vendor_support [15629.882238] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15629.884508] i2c_smbus crct10dif_pclmul [15629.891981] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15629.895983] joydev crc32_pclmul [15629.901460] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15629.905293] ipmi_ssif ghash_clmulni_intel [15629.924031] [15629.924031] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15629.927254] mei_me mei [15629.932471] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15629.936563] lpc_ich [15629.938054] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15629.945177] sha512_ssse3 [15629.947620] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15629.954744] acpi_ipmi mfd_core [15629.956929] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15629.964052] wmi ipmi_si [15629.966671] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15629.973808] ipmi_devintf [15629.976950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15629.984088] ipmi_msghandler ioatdma [15629.986621] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15629.994700] acpi_pad [15629.997317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15630.003055] acpi_power_meter ixgbe [15630.006625] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15630.013748] igb mdio [15630.016019] Call Trace: [15630.016022] [15630.023143] crc32c_intel nvme [15630.026629] ? watchdog_timer_fn+0x1b4/0x220 [15630.033752] ptp nvme_core [15630.036022] ? __hrtimer_run_queues+0x1d3/0x360 [15630.038467] i2c_algo_bit pps_core [15630.040480] ? hrtimer_interrupt+0x104/0x240 [15630.043528] dca fuse [15630.047808] ? __local_bh_enable+0x50/0x60 [15630.050513] [15630.050514] irq event stamp: 48162726 [15630.055041] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15630.058435] hardirqs last enabled at (48162725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15630.062701] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15630.064968] hardirqs last disabled at (48162726): [] sysvec_apic_timer_interrupt+0xe/0x90 [15630.069060] [15630.070550] softirqs last enabled at (48162724): [] __do_softirq+0x2be/0x371 [15630.074208] [15630.074210] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15630.079427] softirqs last disabled at (48162719): [] irq_exit_rcu+0xb0/0xd0 [15630.089588] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15630.094540] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15630.104266] do_raw_spin_lock+0xb5/0xc0 [15630.106362] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15630.115048] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15630.117145] Workqueue: events delayed_work [ceph] [15630.122451] try_get_cap_refs+0x159/0x880 [ceph] [15630.130959] [15630.135835] __ceph_get_caps+0x4a1/0x760 [ceph] [15630.145207] RIP: 0010:_raw_spin_lock+0x0/0x40 [15630.149041] ? __file_remove_privs+0x89/0x110 [15630.156509] Code: 48 c7 c7 7c ca 15 82 e8 4e a8 57 ff e8 69 5d ff ff 48 89 ea 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 [15630.161294] ? wake_bit_function+0x70/0x70 [15630.165989] RSP: 0018:ffffc90002117d98 EFLAGS: 00000286 [15630.170604] ? ceph_write_iter+0x260/0x770 [ceph] [15630.172093] [15630.176615] ceph_write_iter+0x260/0x770 [ceph] [15630.180966] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [15630.185321] ? lock_is_held_type+0xaa/0x120 [15630.204055] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [15630.208149] vfs_write+0x324/0x4f0 [15630.213363] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15630.218065] ksys_pwrite64+0x65/0xa0 [15630.219550] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15630.224075] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15630.231198] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15630.235379] do_syscall_64+0x38/0x80 [15630.242501] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15630.245900] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15630.253023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15630.256593] RIP: 0033:0x7f48f0eeb93b [15630.263718] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15630.268849] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15630.275972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15630.279542] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15630.287619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15630.292664] ORIG_RAX: 0000000000000012 [15630.298401] Call Trace: [15630.298403] [15630.301972] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15630.309098] ? watchdog_timer_fn+0x1b4/0x220 [15630.327835] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15630.334959] ? __hrtimer_run_queues+0x1d3/0x360 [15630.340176] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15630.347303] ? hrtimer_interrupt+0x104/0x240 [15630.351132] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15630.353575] ? __local_bh_enable+0x50/0x60 [15630.355585] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15630.362712] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15630.366979] [15630.374098] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15630.420710] [15630.422819] [15630.424922] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15630.430238] ? __sched_text_end+0xa/0xa [15630.434074] igrab+0x1d/0x50 [15630.436968] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15630.442119] delayed_work+0x1d8/0x280 [ceph] [15630.446400] ? process_scheduled_works+0x22a/0x510 [15630.451190] process_scheduled_works+0x2a4/0x510 [15630.455823] worker_thread+0x188/0x340 [15630.459572] ? rescuer_thread+0x310/0x310 [15630.463576] kthread+0xf5/0x130 [15630.466722] ? kthread_complete_and_exit+0x20/0x20 [15630.471506] ret_from_fork+0x31/0x40 [15630.475085] ? kthread_complete_and_exit+0x20/0x20 [15630.479869] ret_from_fork_asm+0x11/0x20 [15630.483823] [15639.437063] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15650.188669] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15657.728901] watchdog: BUG: soft lockup - CPU#0 stuck for 3468s! [dbench:78683] [15657.736118] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15657.740901] watchdog: BUG: soft lockup - CPU#7 stuck for 12465s! [kworker/7:4:78808] [15657.791574] irq event stamp: 20596258 [15657.791576] hardirqs last enabled at (20596257): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15657.799306] Modules linked in: [15657.802965] hardirqs last disabled at (20596258): [] sysvec_apic_timer_interrupt+0xe/0x90 [15657.813123] ceph [15657.816173] softirqs last enabled at (20596238): [] __do_softirq+0x2be/0x371 [15657.825896] libceph dns_resolver [15657.827821] softirqs last disabled at (20596233): [] irq_exit_rcu+0xb0/0xd0 [15657.836505] fscache [15657.839815] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15657.848327] netfs veth [15657.850512] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15657.859454] nft_chain_nat xt_MASQUERADE [15657.861898] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15657.869369] nf_nat nf_conntrack [15657.873297] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15657.878773] nf_defrag_ipv6 nf_defrag_ipv4 [15657.881996] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15657.900735] nft_compat nf_tables [15657.904825] [15657.904826] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15657.910043] nfnetlink bridge [15657.913355] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15657.914843] stp llc [15657.921969] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15657.924932] xfs libcrc32c [15657.932056] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15657.934241] sunrpc [15657.941365] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15657.944069] intel_rapl_msr intel_rapl_common [15657.951194] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15657.953298] sb_edac [15657.960422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15657.964774] x86_pkg_temp_thermal coretemp [15657.972851] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15657.975036] kvm_intel kvm [15657.980772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15657.984863] irqbypass i2c_i801 [15657.991986] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15657.994693] iTCO_wdt iTCO_vendor_support [15658.001815] Call Trace: [15658.001816] [15658.004954] i2c_smbus crct10dif_pclmul [15658.012078] ? watchdog_timer_fn+0x1b4/0x220 [15658.016081] joydev crc32_pclmul [15658.018526] ? __hrtimer_run_queues+0x1d3/0x360 [15658.020535] ipmi_ssif ghash_clmulni_intel [15658.024371] ? hrtimer_interrupt+0x104/0x240 [15658.028632] mei_me mei [15658.031854] ? __local_bh_enable+0x50/0x60 [15658.036379] lpc_ich sha512_ssse3 [15658.040472] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15658.044735] acpi_ipmi mfd_core [15658.047179] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15658.051268] wmi ipmi_si [15658.054591] [15658.059805] ipmi_devintf ipmi_msghandler [15658.062943] [15658.067900] ioatdma acpi_pad [15658.070433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15658.072528] acpi_power_meter ixgbe [15658.076536] ? queued_spin_lock_slowpath+0x16/0x2e0 [15658.078631] igb mdio [15658.081597] do_raw_spin_lock+0xb5/0xc0 [15658.086897] crc32c_intel nvme ptp [15658.090385] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15658.095254] nvme_core i2c_algo_bit [15658.097525] try_get_cap_refs+0x159/0x880 [ceph] [15658.101353] pps_core dca [15658.104756] __ceph_get_caps+0x4a1/0x760 [ceph] [15658.109535] fuse [15658.113024] ? __file_remove_privs+0x89/0x110 [15658.117630] irq event stamp: 48383852 [15658.120248] ? wake_bit_function+0x70/0x70 [15658.124771] hardirqs last enabled at (48383851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15658.126700] ? ceph_write_iter+0x260/0x770 [ceph] [15658.131046] hardirqs last disabled at (48383852): [] sysvec_apic_timer_interrupt+0xe/0x90 [15658.134705] ceph_write_iter+0x260/0x770 [ceph] [15658.138796] softirqs last enabled at (48383850): [] __do_softirq+0x2be/0x371 [15658.148958] ? lock_is_held_type+0xaa/0x120 [15658.153649] softirqs last disabled at (48383845): [] irq_exit_rcu+0xb0/0xd0 [15658.163377] vfs_write+0x324/0x4f0 [15658.167899] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15658.176587] ksys_pwrite64+0x65/0xa0 [15658.180759] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15658.189272] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15658.192669] Workqueue: events delayed_work [ceph] [15658.202048] do_syscall_64+0x38/0x80 [15658.205617] [15658.205618] RIP: 0010:lock_acquire+0xf0/0x2b0 [15658.213087] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15658.218217] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15658.222915] RIP: 0033:0x7f48f0eeb93b [15658.226486] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15658.227978] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15658.232327] [15658.232327] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15658.237372] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15658.256110] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15658.259680] ORIG_RAX: 0000000000000012 [15658.264896] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15658.283636] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15658.285127] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15658.292251] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15658.297467] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15658.304593] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15658.308422] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15658.315546] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15658.322671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15658.329795] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15658.336918] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15658.344048] [15658.351168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15658.395664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15658.402787] Call Trace: [15658.405231] [15658.407242] ? watchdog_timer_fn+0x1b4/0x220 [15658.411508] ? __hrtimer_run_queues+0x1d3/0x360 [15658.416033] ? hrtimer_interrupt+0x104/0x240 [15658.420307] ? __local_bh_enable+0x50/0x60 [15658.424404] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15658.429621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15658.434593] [15658.436691] [15658.438790] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15658.444097] ? lock_acquire+0xf0/0x2b0 [15658.447848] ? igrab+0x43/0x50 [15658.450897] ? lock_release+0xc5/0x260 [15658.454643] _raw_spin_lock+0x30/0x40 [15658.458307] ? igrab+0x1d/0x50 [15658.461357] igrab+0x1d/0x50 [15658.464237] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15658.469386] delayed_work+0x1d8/0x280 [ceph] [15658.473667] ? process_scheduled_works+0x22a/0x510 [15658.478456] process_scheduled_works+0x2a4/0x510 [15658.483072] worker_thread+0x188/0x340 [15658.486822] ? rescuer_thread+0x310/0x310 [15658.490826] kthread+0xf5/0x130 [15658.493963] ? kthread_complete_and_exit+0x20/0x20 [15658.498747] ret_from_fork+0x31/0x40 [15658.502317] ? kthread_complete_and_exit+0x20/0x20 [15658.507101] ret_from_fork_asm+0x11/0x20 [15658.511025] [15660.428922] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15671.181204] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15681.421173] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15685.729392] watchdog: BUG: soft lockup - CPU#0 stuck for 3494s! [dbench:78683] [15685.736604] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15685.741391] watchdog: BUG: soft lockup - CPU#7 stuck for 12491s! [kworker/7:4:78808] [15685.792066] irq event stamp: 20654352 [15685.792067] hardirqs last enabled at (20654351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15685.799795] Modules linked in: [15685.803451] hardirqs last disabled at (20654352): [] sysvec_apic_timer_interrupt+0xe/0x90 [15685.813608] ceph [15685.816659] softirqs last enabled at (20654314): [] __do_softirq+0x2be/0x371 [15685.826384] libceph [15685.828307] softirqs last disabled at (20654309): [] irq_exit_rcu+0xb0/0xd0 [15685.836993] dns_resolver fscache [15685.839176] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15685.847686] netfs veth [15685.850997] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15685.859942] nft_chain_nat xt_MASQUERADE [15685.862386] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15685.869856] nf_nat nf_conntrack [15685.873787] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15685.879259] nf_defrag_ipv6 nf_defrag_ipv4 [15685.882484] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15685.901222] nft_compat nf_tables [15685.905313] [15685.905314] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15685.910531] nfnetlink bridge [15685.913842] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15685.915331] stp llc [15685.922455] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15685.925420] xfs libcrc32c [15685.932544] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15685.934728] sunrpc intel_rapl_msr [15685.941852] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15685.944556] intel_rapl_common [15685.951680] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15685.955078] sb_edac x86_pkg_temp_thermal [15685.962201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15685.965251] coretemp kvm_intel [15685.973329] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15685.977335] kvm irqbypass [15685.983071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15685.986209] i2c_i801 [15685.993332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15685.996037] iTCO_wdt iTCO_vendor_support [15686.003160] Call Trace: [15686.003163] [15686.005431] i2c_smbus crct10dif_pclmul [15686.012559] ? watchdog_timer_fn+0x1b4/0x220 [15686.016559] joydev crc32_pclmul [15686.019006] ? __hrtimer_run_queues+0x1d3/0x360 [15686.021014] ipmi_ssif ghash_clmulni_intel [15686.024850] ? hrtimer_interrupt+0x104/0x240 [15686.029110] mei_me mei [15686.032335] ? __local_bh_enable+0x50/0x60 [15686.036858] lpc_ich sha512_ssse3 [15686.040952] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15686.045214] acpi_ipmi [15686.047657] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15686.051748] mfd_core [15686.055071] [15686.060285] wmi ipmi_si [15686.062640] [15686.062642] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15686.067598] ipmi_devintf [15686.069873] ? queued_spin_lock_slowpath+0x16/0x2e0 [15686.071967] ipmi_msghandler ioatdma [15686.074500] do_raw_spin_lock+0xb5/0xc0 [15686.076596] acpi_pad acpi_power_meter [15686.081901] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15686.084518] ixgbe igb [15686.089390] try_get_cap_refs+0x159/0x880 [ceph] [15686.092957] mdio crc32c_intel [15686.096794] __ceph_get_caps+0x4a1/0x760 [ceph] [15686.100533] nvme ptp [15686.105320] ? __file_remove_privs+0x89/0x110 [15686.107675] nvme_core [15686.112287] ? wake_bit_function+0x70/0x70 [15686.115336] i2c_algo_bit pps_core [15686.119862] ? ceph_write_iter+0x260/0x770 [ceph] [15686.122129] dca fuse [15686.126482] ceph_write_iter+0x260/0x770 [ceph] [15686.128838] [15686.128839] irq event stamp: 48604768 [15686.132934] ? lock_is_held_type+0xaa/0x120 [15686.136326] hardirqs last enabled at (48604767): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15686.141027] vfs_write+0x324/0x4f0 [15686.143295] hardirqs last disabled at (48604768): [] sysvec_apic_timer_interrupt+0xe/0x90 [15686.147826] ksys_pwrite64+0x65/0xa0 [15686.149309] softirqs last enabled at (48604766): [] __do_softirq+0x2be/0x371 [15686.152968] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15686.157145] softirqs last disabled at (48604761): [] irq_exit_rcu+0xb0/0xd0 [15686.167304] do_syscall_64+0x38/0x80 [15686.170699] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15686.180423] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15686.183995] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15686.192678] RIP: 0033:0x7f48f0eeb93b [15686.197810] Workqueue: events delayed_work [ceph] [15686.206322] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15686.209890] [15686.209891] RIP: 0010:lock_acquire+0xf0/0x2b0 [15686.219268] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15686.224312] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15686.231792] ORIG_RAX: 0000000000000012 [15686.235363] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15686.240074] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15686.258816] [15686.258817] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15686.260305] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15686.264657] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15686.269875] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15686.288611] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15686.292442] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15686.297660] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15686.304796] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15686.306284] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15686.313412] [15686.320531] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15686.380671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15686.386407] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15686.393533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15686.400657] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15686.407795] Call Trace: [15686.410242] [15686.412254] ? watchdog_timer_fn+0x1b4/0x220 [15686.416518] ? __hrtimer_run_queues+0x1d3/0x360 [15686.421044] ? hrtimer_interrupt+0x104/0x240 [15686.425306] ? __local_bh_enable+0x50/0x60 [15686.429399] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15686.434621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15686.439580] [15686.441676] [15686.443775] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15686.449099] ? lock_acquire+0xf0/0x2b0 [15686.452850] ? igrab+0x43/0x50 [15686.455901] ? lock_release+0xc5/0x260 [15686.459646] _raw_spin_lock+0x30/0x40 [15686.463310] ? igrab+0x1d/0x50 [15686.466361] igrab+0x1d/0x50 [15686.469238] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15686.474390] delayed_work+0x1d8/0x280 [ceph] [15686.478671] ? process_scheduled_works+0x22a/0x510 [15686.483459] process_scheduled_works+0x2a4/0x510 [15686.488083] worker_thread+0x188/0x340 [15686.491832] ? rescuer_thread+0x310/0x310 [15686.495836] kthread+0xf5/0x130 [15686.498975] ? kthread_complete_and_exit+0x20/0x20 [15686.503767] ret_from_fork+0x31/0x40 [15686.507363] ? kthread_complete_and_exit+0x20/0x20 [15686.512147] ret_from_fork_asm+0x11/0x20 [15686.516096] [15692.173560] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15702.413547] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15713.165900] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15713.729881] watchdog: BUG: soft lockup - CPU#0 stuck for 3520s! [dbench:78683] [15713.737093] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15713.741880] watchdog: BUG: soft lockup - CPU#7 stuck for 12517s! [kworker/7:4:78808] [15713.792533] irq event stamp: 20712598 [15713.792551] hardirqs last enabled at (20712597): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15713.800282] Modules linked in: [15713.803937] hardirqs last disabled at (20712598): [] sysvec_apic_timer_interrupt+0xe/0x90 [15713.814096] ceph [15713.817148] softirqs last enabled at (20712560): [] __do_softirq+0x2be/0x371 [15713.826871] libceph [15713.828795] softirqs last disabled at (20712555): [] irq_exit_rcu+0xb0/0xd0 [15713.837480] dns_resolver fscache [15713.839664] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15713.848175] netfs veth [15713.851486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15713.860428] nft_chain_nat xt_MASQUERADE [15713.862874] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15713.870344] nf_nat nf_conntrack [15713.874277] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15713.879756] nf_defrag_ipv6 nf_defrag_ipv4 [15713.882980] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15713.901718] nft_compat [15713.905809] [15713.911026] nf_tables nfnetlink [15713.913470] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15713.914960] bridge stp [15713.918186] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15713.925308] llc xfs [15713.927752] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15713.934877] libcrc32c [15713.937062] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15713.944186] sunrpc intel_rapl_msr [15713.946543] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15713.953667] intel_rapl_common sb_edac [15713.957065] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15713.964189] x86_pkg_temp_thermal coretemp [15713.967932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15713.976011] kvm_intel kvm [15713.980101] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15713.985837] irqbypass i2c_i801 [15713.988558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15713.995683] iTCO_wdt iTCO_vendor_support [15713.998821] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15714.005945] i2c_smbus crct10dif_pclmul [15714.009949] Call Trace: [15714.009950] [15714.017074] joydev crc32_pclmul [15714.020905] ? watchdog_timer_fn+0x1b4/0x220 [15714.023349] ipmi_ssif ghash_clmulni_intel [15714.025359] ? __hrtimer_run_queues+0x1d3/0x360 [15714.028582] mei_me [15714.032849] ? hrtimer_interrupt+0x104/0x240 [15714.036937] mei lpc_ich [15714.041462] ? __local_bh_enable+0x50/0x60 [15714.043561] sha512_ssse3 acpi_ipmi [15714.047827] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15714.050354] mfd_core wmi [15714.054445] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15714.057929] ipmi_si ipmi_devintf [15714.063148] [15714.065763] ipmi_msghandler ioatdma [15714.070722] [15714.074033] acpi_pad acpi_power_meter [15714.076129] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15714.079701] ixgbe [15714.081802] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15714.085541] igb mdio crc32c_intel [15714.090849] do_raw_spin_lock+0xb5/0xc0 [15714.092857] nvme ptp [15714.097728] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15714.101124] nvme_core i2c_algo_bit [15714.104958] try_get_cap_refs+0x159/0x880 [ceph] [15714.107225] pps_core dca [15714.112015] __ceph_get_caps+0x4a1/0x760 [ceph] [15714.115495] fuse [15714.120110] ? __file_remove_privs+0x89/0x110 [15714.122723] irq event stamp: 48825590 [15714.127248] ? wake_bit_function+0x70/0x70 [15714.129170] hardirqs last enabled at (48825589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15714.133523] ? ceph_write_iter+0x260/0x770 [ceph] [15714.137179] hardirqs last disabled at (48825590): [] sysvec_apic_timer_interrupt+0xe/0x90 [15714.141282] ceph_write_iter+0x260/0x770 [ceph] [15714.151444] softirqs last enabled at (48825588): [] __do_softirq+0x2be/0x371 [15714.156145] ? lock_is_held_type+0xaa/0x120 [15714.165864] softirqs last disabled at (48825583): [] irq_exit_rcu+0xb0/0xd0 [15714.170394] vfs_write+0x324/0x4f0 [15714.179074] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15714.183257] ksys_pwrite64+0x65/0xa0 [15714.191762] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15714.191764] Workqueue: events delayed_work [ceph] [15714.195161] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15714.204537] [15714.208109] do_syscall_64+0x38/0x80 [15714.215579] RIP: 0010:lock_acquire+0xf0/0x2b0 [15714.220283] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15714.225414] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15714.226905] RIP: 0033:0x7f48f0eeb93b [15714.230476] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15714.234827] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15714.239871] [15714.239871] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15714.258610] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15714.262180] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15714.267398] ORIG_RAX: 0000000000000012 [15714.286136] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15714.287626] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15714.294750] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15714.299968] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15714.307090] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15714.310922] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15714.318045] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15714.325169] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15714.332294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15714.339418] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15714.346543] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15714.353673] [15714.361745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15714.398162] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15714.405288] Call Trace: [15714.407732] [15714.409742] ? watchdog_timer_fn+0x1b4/0x220 [15714.414007] ? __hrtimer_run_queues+0x1d3/0x360 [15714.418533] ? hrtimer_interrupt+0x104/0x240 [15714.422801] ? __local_bh_enable+0x50/0x60 [15714.426896] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15714.432119] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15714.437076] [15714.439173] [15714.441290] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15714.446598] ? lock_acquire+0xf0/0x2b0 [15714.450348] ? igrab+0x43/0x50 [15714.453398] ? lock_release+0xc5/0x260 [15714.457142] _raw_spin_lock+0x30/0x40 [15714.460807] ? igrab+0x1d/0x50 [15714.463857] igrab+0x1d/0x50 [15714.466736] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15714.471884] delayed_work+0x1d8/0x280 [ceph] [15714.476166] ? process_scheduled_works+0x22a/0x510 [15714.480947] process_scheduled_works+0x2a4/0x510 [15714.485574] worker_thread+0x188/0x340 [15714.489322] ? rescuer_thread+0x310/0x310 [15714.493324] kthread+0xf5/0x130 [15714.496462] ? kthread_complete_and_exit+0x20/0x20 [15714.501246] ret_from_fork+0x31/0x40 [15714.504818] ? kthread_complete_and_exit+0x20/0x20 [15714.509602] ret_from_fork_asm+0x11/0x20 [15714.513524] [15715.276275] rcu: INFO: rcu_preempt self-detected stall on CPU [15715.282032] rcu: 7-....: (3672587 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/3671392 fqs=1825340 [15715.292449] rcu: (t=3778599 jiffies g=1309609 q=184721 ncpus=8) [15715.298448] Sending NMI from CPU 7 to CPUs 0: [15715.302799] NMI backtrace for cpu 0 [15715.302801] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15715.302802] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15715.302803] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15715.302807] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15715.302808] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15715.302810] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15715.302811] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15715.302811] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15715.302812] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15715.302813] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15715.302814] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15715.302815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15715.302816] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15715.302817] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15715.302817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15715.302818] Call Trace: [15715.302819] [15715.302820] ? nmi_cpu_backtrace+0xad/0x120 [15715.302824] ? nmi_cpu_backtrace_handler+0x11/0x20 [15715.302825] ? nmi_handle+0xd4/0x260 [15715.302830] ? default_do_nmi+0x49/0xf0 [15715.302831] ? exc_nmi+0x11e/0x150 [15715.302833] ? end_repeat_nmi+0x16/0x67 [15715.302837] ? queued_spin_lock_slowpath+0x18/0x2e0 [15715.302840] ? queued_spin_lock_slowpath+0x18/0x2e0 [15715.302842] ? queued_spin_lock_slowpath+0x18/0x2e0 [15715.302845] [15715.302845] [15715.302846] do_raw_spin_lock+0xb5/0xc0 [15715.302848] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15715.302863] try_get_cap_refs+0x159/0x880 [ceph] [15715.302877] __ceph_get_caps+0x4a1/0x760 [ceph] [15715.302889] ? __file_remove_privs+0x89/0x110 [15715.302891] ? wake_bit_function+0x70/0x70 [15715.302894] ? ceph_write_iter+0x260/0x770 [ceph] [15715.302905] ceph_write_iter+0x260/0x770 [ceph] [15715.302916] ? lock_is_held_type+0xaa/0x120 [15715.302919] vfs_write+0x324/0x4f0 [15715.302923] ksys_pwrite64+0x65/0xa0 [15715.302925] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15715.302928] do_syscall_64+0x38/0x80 [15715.302930] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15715.302932] RIP: 0033:0x7f48f0eeb93b [15715.302933] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15715.302934] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [15715.302935] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15715.302936] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15715.302937] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15715.302937] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15715.302938] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15715.302941] [15715.303798] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15715.607619] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15715.615089] Workqueue: events delayed_work [ceph] [15715.619804] RIP: 0010:lock_acquire+0xf0/0x2b0 [15715.624154] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15715.642891] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15715.648109] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15715.655234] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15715.662357] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15715.669482] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15715.676607] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15715.683729] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15715.691808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15715.697546] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15715.704669] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15715.711794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15715.718916] Call Trace: [15715.721362] [15715.723374] ? rcu_dump_cpu_stacks+0xd1/0x180 [15715.727732] ? rcu_sched_clock_irq+0x633/0x1390 [15715.732257] ? lock_release+0xc5/0x260 [15715.736000] ? lock_is_held_type+0xaa/0x120 [15715.740179] ? lock_is_held_type+0xaa/0x120 [15715.744356] ? tick_sched_do_timer+0xa0/0xa0 [15715.748627] ? rcu_read_lock_sched_held+0x56/0x90 [15715.753327] ? tick_sched_do_timer+0xa0/0xa0 [15715.757598] ? update_process_times+0x5f/0x90 [15715.761949] ? tick_sched_handle.isra.16+0x1e/0x50 [15715.766732] ? tick_sched_timer+0x6f/0x80 [15715.770738] ? __hrtimer_run_queues+0x1d3/0x360 [15715.775266] ? hrtimer_interrupt+0x104/0x240 [15715.779532] ? __local_bh_enable+0x50/0x60 [15715.783628] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15715.788850] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15715.793807] [15715.795906] [15715.798003] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15715.803312] ? lock_acquire+0xf0/0x2b0 [15715.807061] ? igrab+0x43/0x50 [15715.810110] ? lock_release+0xc5/0x260 [15715.813855] _raw_spin_lock+0x30/0x40 [15715.817512] ? igrab+0x1d/0x50 [15715.820563] igrab+0x1d/0x50 [15715.823441] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15715.828589] delayed_work+0x1d8/0x280 [ceph] [15715.832870] ? process_scheduled_works+0x22a/0x510 [15715.837653] process_scheduled_works+0x2a4/0x510 [15715.842269] worker_thread+0x188/0x340 [15715.846017] ? rescuer_thread+0x310/0x310 [15715.850020] kthread+0xf5/0x130 [15715.853159] ? kthread_complete_and_exit+0x20/0x20 [15715.857943] ret_from_fork+0x31/0x40 [15715.861513] ? kthread_complete_and_exit+0x20/0x20 [15715.866296] ret_from_fork_asm+0x11/0x20 [15715.870221] [15723.405896] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15734.158064] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15741.730370] watchdog: BUG: soft lockup - CPU#0 stuck for 3546s! [dbench:78683] [15741.737587] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15741.742370] watchdog: BUG: soft lockup - CPU#7 stuck for 12543s! [kworker/7:4:78808] [15741.793043] irq event stamp: 20770736 [15741.793044] hardirqs last enabled at (20770735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15741.800778] Modules linked in: [15741.804434] hardirqs last disabled at (20770736): [] sysvec_apic_timer_interrupt+0xe/0x90 [15741.814591] ceph [15741.817643] softirqs last enabled at (20770734): [] __do_softirq+0x2be/0x371 [15741.827367] libceph [15741.829292] softirqs last disabled at (20770729): [] irq_exit_rcu+0xb0/0xd0 [15741.837975] dns_resolver fscache [15741.840159] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15741.848670] netfs [15741.851982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15741.860925] veth nft_chain_nat [15741.862936] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15741.870407] xt_MASQUERADE nf_nat [15741.873545] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15741.879022] nf_conntrack [15741.882350] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15741.901088] nf_defrag_ipv6 [15741.903705] [15741.903706] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15741.908923] nf_defrag_ipv4 nft_compat [15741.911713] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15741.913203] nf_tables [15741.920328] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15741.924072] nfnetlink bridge [15741.931195] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15741.933553] stp llc [15741.940677] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15741.943642] xfs [15741.950768] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15741.952949] libcrc32c sunrpc [15741.960073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15741.961911] intel_rapl_msr intel_rapl_common [15741.969989] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15741.972952] sb_edac [15741.978691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15741.983055] x86_pkg_temp_thermal [15741.990183] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15741.992366] coretemp kvm_intel [15741.999492] Call Trace: [15742.002801] kvm irqbypass [15742.009927] [15742.013061] i2c_i801 [15742.015508] ? watchdog_timer_fn+0x1b4/0x220 [15742.018211] iTCO_wdt iTCO_vendor_support [15742.020221] ? __hrtimer_run_queues+0x1d3/0x360 [15742.022491] i2c_smbus crct10dif_pclmul [15742.026770] ? hrtimer_interrupt+0x104/0x240 [15742.030768] joydev crc32_pclmul [15742.035294] ? __local_bh_enable+0x50/0x60 [15742.039123] ipmi_ssif ghash_clmulni_intel [15742.043391] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15742.046613] mei_me [15742.050702] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15742.054794] mei lpc_ich [15742.060012] [15742.062109] sha512_ssse3 acpi_ipmi [15742.067069] [15742.069598] mfd_core wmi [15742.071694] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15742.075177] ipmi_si ipmi_devintf [15742.077279] ? queued_spin_lock_slowpath+0x18/0x2e0 [15742.079892] ipmi_msghandler ioatdma [15742.085198] do_raw_spin_lock+0xb5/0xc0 [15742.088506] acpi_pad acpi_power_meter [15742.093380] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15742.096948] ixgbe igb [15742.100781] try_get_cap_refs+0x159/0x880 [ceph] [15742.104525] mdio crc32c_intel [15742.109312] __ceph_get_caps+0x4a1/0x760 [ceph] [15742.111664] nvme ptp [15742.116281] ? __file_remove_privs+0x89/0x110 [15742.119327] nvme_core i2c_algo_bit [15742.123851] ? wake_bit_function+0x70/0x70 [15742.126120] pps_core dca [15742.130476] ? ceph_write_iter+0x260/0x770 [ceph] [15742.133955] fuse [15742.138059] ceph_write_iter+0x260/0x770 [ceph] [15742.140672] irq event stamp: 49041668 [15742.145374] ? lock_is_held_type+0xaa/0x120 [15742.147294] hardirqs last enabled at (49041667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15742.151822] vfs_write+0x324/0x4f0 [15742.155477] hardirqs last disabled at (49041668): [] sysvec_apic_timer_interrupt+0xe/0x90 [15742.159659] ksys_pwrite64+0x65/0xa0 [15742.169810] softirqs last enabled at (49041666): [] __do_softirq+0x2be/0x371 [15742.173208] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15742.182931] softirqs last disabled at (49041661): [] irq_exit_rcu+0xb0/0xd0 [15742.186505] do_syscall_64+0x38/0x80 [15742.195187] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15742.200320] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15742.208830] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15742.212400] RIP: 0033:0x7f48f0eeb93b [15742.221776] Workqueue: events delayed_work [ceph] [15742.226823] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15742.234292] [15742.234293] RIP: 0010:lock_acquire+0xf0/0x2b0 [15742.237864] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15742.242560] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15742.261297] ORIG_RAX: 0000000000000012 [15742.262789] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15742.267141] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15742.272356] [15742.272357] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15742.291095] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15742.294924] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15742.300142] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15742.307266] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15742.308771] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15742.315900] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15742.323022] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15742.330146] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15742.337276] [15742.344395] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15742.383163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15742.388901] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15742.396025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15742.403149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15742.410272] Call Trace: [15742.412716] [15742.414728] ? watchdog_timer_fn+0x1b4/0x220 [15742.418992] ? __hrtimer_run_queues+0x1d3/0x360 [15742.423518] ? hrtimer_interrupt+0x104/0x240 [15742.427787] ? __local_bh_enable+0x50/0x60 [15742.431882] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15742.437104] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15742.442073] [15742.444167] [15742.446267] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15742.451584] ? lock_acquire+0xf0/0x2b0 [15742.455334] ? igrab+0x43/0x50 [15742.458383] ? lock_release+0xc5/0x260 [15742.462128] _raw_spin_lock+0x30/0x40 [15742.465791] ? igrab+0x1d/0x50 [15742.468844] igrab+0x1d/0x50 [15742.471721] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15742.476872] delayed_work+0x1d8/0x280 [ceph] [15742.481152] ? process_scheduled_works+0x22a/0x510 [15742.485943] process_scheduled_works+0x2a4/0x510 [15742.490558] worker_thread+0x188/0x340 [15742.494307] ? rescuer_thread+0x310/0x310 [15742.498311] kthread+0xf5/0x130 [15742.501447] ? kthread_complete_and_exit+0x20/0x20 [15742.506233] ret_from_fork+0x31/0x40 [15742.509801] ? kthread_complete_and_exit+0x20/0x20 [15742.514587] ret_from_fork_asm+0x11/0x20 [15742.518511] [15744.398278] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15755.150580] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15769.730860] watchdog: BUG: soft lockup - CPU#0 stuck for 3572s! [dbench:78683] [15769.738075] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15769.742860] watchdog: BUG: soft lockup - CPU#7 stuck for 12569s! [kworker/7:4:78808] [15769.793542] irq event stamp: 20828922 [15769.793543] hardirqs last enabled at (20828921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15769.801273] Modules linked in: [15769.804932] hardirqs last disabled at (20828922): [] sysvec_apic_timer_interrupt+0xe/0x90 [15769.815088] ceph [15769.818139] softirqs last enabled at (20828884): [] __do_softirq+0x2be/0x371 [15769.827864] libceph [15769.829789] softirqs last disabled at (20828879): [] irq_exit_rcu+0xb0/0xd0 [15769.838472] dns_resolver fscache [15769.840657] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15769.849166] netfs [15769.852477] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15769.861422] veth nft_chain_nat [15769.863433] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [15769.870904] xt_MASQUERADE nf_nat [15769.874041] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [15769.879532] nf_conntrack nf_defrag_ipv6 [15769.882847] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15769.901583] nf_defrag_ipv4 nft_compat [15769.905500] [15769.905501] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15769.910718] nf_tables nfnetlink [15769.914463] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15769.915952] bridge stp [15769.923079] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15769.926302] llc xfs [15769.933425] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15769.935870] libcrc32c sunrpc [15769.942993] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15769.945177] intel_rapl_msr [15769.952302] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15769.955266] intel_rapl_common sb_edac [15769.962390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15769.965180] x86_pkg_temp_thermal coretemp [15769.973258] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15769.977002] kvm_intel kvm [15769.982741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15769.986831] irqbypass i2c_i801 [15769.993956] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15769.996660] iTCO_wdt [15770.003782] Call Trace: [15770.003784] [15770.006920] iTCO_vendor_support [15770.014048] ? watchdog_timer_fn+0x1b4/0x220 [15770.016314] i2c_smbus crct10dif_pclmul [15770.018760] ? __hrtimer_run_queues+0x1d3/0x360 [15770.020770] joydev crc32_pclmul [15770.023997] ? hrtimer_interrupt+0x104/0x240 [15770.028259] ipmi_ssif [15770.032090] ? __local_bh_enable+0x50/0x60 [15770.036613] ghash_clmulni_intel [15770.039838] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15770.044100] mei_me mei [15770.046459] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15770.050549] lpc_ich sha512_ssse3 [15770.053774] [15770.058990] acpi_ipmi mfd_core [15770.061435] [15770.066393] wmi [15770.069704] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15770.071799] ipmi_si ipmi_devintf [15770.074942] ? queued_spin_lock_slowpath+0x18/0x2e0 [15770.077034] ipmi_msghandler [15770.078875] do_raw_spin_lock+0xb5/0xc0 [15770.084177] ioatdma acpi_pad [15770.087489] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15770.092357] acpi_power_meter ixgbe igb [15770.095256] try_get_cap_refs+0x159/0x880 [ceph] [15770.099083] mdio crc32c_intel [15770.102052] __ceph_get_caps+0x4a1/0x760 [ceph] [15770.106832] nvme ptp [15770.110666] ? __file_remove_privs+0x89/0x110 [15770.115272] nvme_core i2c_algo_bit [15770.118324] ? wake_bit_function+0x70/0x70 [15770.122848] pps_core dca [15770.125121] ? ceph_write_iter+0x260/0x770 [ceph] [15770.129469] fuse [15770.132953] ceph_write_iter+0x260/0x770 [ceph] [15770.137045] irq event stamp: 49263668 [15770.139667] ? lock_is_held_type+0xaa/0x120 [15770.144358] hardirqs last enabled at (49263667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15770.146286] vfs_write+0x324/0x4f0 [15770.150807] hardirqs last disabled at (49263668): [] sysvec_apic_timer_interrupt+0xe/0x90 [15770.154470] ksys_pwrite64+0x65/0xa0 [15770.158641] softirqs last enabled at (49263666): [] __do_softirq+0x2be/0x371 [15770.168801] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15770.172198] softirqs last disabled at (49263661): [] irq_exit_rcu+0xb0/0xd0 [15770.181921] do_syscall_64+0x38/0x80 [15770.185492] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15770.194176] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15770.199308] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15770.207819] RIP: 0033:0x7f48f0eeb93b [15770.211388] Workqueue: events delayed_work [ceph] [15770.220768] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15770.225810] [15770.225810] RIP: 0010:_raw_spin_unlock+0xf/0x40 [15770.233280] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15770.236851] Code: 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 <48> 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff [15770.241553] ORIG_RAX: 0000000000000012 [15770.260294] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [15770.261786] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15770.266310] [15770.266311] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [15770.271542] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15770.290283] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34500 [15770.294113] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15770.299332] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15770.306454] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15770.307946] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15770.315070] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15770.322193] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15770.329323] [15770.336440] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15770.382342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15770.388078] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15770.395204] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15770.402327] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15770.409452] Call Trace: [15770.411897] [15770.413907] ? watchdog_timer_fn+0x1b4/0x220 [15770.418170] ? __hrtimer_run_queues+0x1d3/0x360 [15770.422698] ? hrtimer_interrupt+0x104/0x240 [15770.426967] ? __local_bh_enable+0x50/0x60 [15770.431060] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15770.436284] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15770.441243] [15770.443356] [15770.445453] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15770.450760] ? igrab+0x43/0x50 [15770.453810] ? _raw_spin_unlock+0xf/0x40 [15770.457729] igrab+0x43/0x50 [15770.460606] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15770.465755] delayed_work+0x1d8/0x280 [ceph] [15770.470034] ? process_scheduled_works+0x22a/0x510 [15770.474825] process_scheduled_works+0x2a4/0x510 [15770.479440] worker_thread+0x188/0x340 [15770.483190] ? rescuer_thread+0x310/0x310 [15770.487194] kthread+0xf5/0x130 [15770.490341] ? kthread_complete_and_exit+0x20/0x20 [15770.495133] ret_from_fork+0x31/0x40 [15770.498712] ? kthread_complete_and_exit+0x20/0x20 [15770.503495] ret_from_fork_asm+0x11/0x20 [15770.507420] [15786.382562] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [15786.391086] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15796.623293] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15797.731350] watchdog: BUG: soft lockup - CPU#0 stuck for 3598s! [dbench:78683] [15797.738565] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15797.743350] watchdog: BUG: soft lockup - CPU#7 stuck for 12595s! [kworker/7:4:78808] [15797.794022] irq event stamp: 20887094 [15797.794024] hardirqs last enabled at (20887093): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15797.801752] Modules linked in: ceph [15797.805410] hardirqs last disabled at (20887094): [] sysvec_apic_timer_interrupt+0xe/0x90 [15797.815567] libceph [15797.819053] softirqs last enabled at (20887056): [] __do_softirq+0x2be/0x371 [15797.828776] dns_resolver [15797.830961] softirqs last disabled at (20887051): [] irq_exit_rcu+0xb0/0xd0 [15797.839644] fscache netfs [15797.842261] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15797.850772] veth nft_chain_nat [15797.853477] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15797.862421] xt_MASQUERADE nf_nat [15797.865559] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15797.873045] nf_conntrack nf_defrag_ipv6 [15797.876357] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15797.881833] nf_defrag_ipv4 nft_compat [15797.885751] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15797.904490] nf_tables nfnetlink [15797.908233] [15797.908234] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15797.913451] bridge [15797.916676] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15797.918166] stp llc [15797.925290] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15797.927387] xfs libcrc32c [15797.934512] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15797.936697] sunrpc intel_rapl_msr [15797.943819] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15797.946524] intel_rapl_common sb_edac [15797.953649] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15797.957044] x86_pkg_temp_thermal [15797.964169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15797.967913] coretemp kvm_intel [15797.975992] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15797.979301] kvm irqbypass [15797.985040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15797.988176] i2c_i801 [15797.995300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15797.998003] iTCO_wdt iTCO_vendor_support [15798.005130] Call Trace: [15798.005131] [15798.007398] i2c_smbus crct10dif_pclmul [15798.014523] ? watchdog_timer_fn+0x1b4/0x220 [15798.018528] joydev crc32_pclmul [15798.020974] ? __hrtimer_run_queues+0x1d3/0x360 [15798.022982] ipmi_ssif ghash_clmulni_intel [15798.026816] ? hrtimer_interrupt+0x104/0x240 [15798.031076] mei_me mei [15798.034301] ? __local_bh_enable+0x50/0x60 [15798.038825] lpc_ich sha512_ssse3 [15798.042918] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15798.047180] acpi_ipmi mfd_core [15798.049625] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15798.053714] wmi ipmi_si [15798.057031] [15798.062251] ipmi_devintf ipmi_msghandler [15798.065389] [15798.070346] ioatdma acpi_pad [15798.072879] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15798.074975] acpi_power_meter ixgbe [15798.078982] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15798.081076] igb mdio [15798.084043] do_raw_spin_lock+0xb5/0xc0 [15798.089344] crc32c_intel nvme [15798.092830] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15798.097700] ptp nvme_core [15798.099973] try_get_cap_refs+0x159/0x880 [ceph] [15798.103802] i2c_algo_bit pps_core [15798.106857] __ceph_get_caps+0x4a1/0x760 [ceph] [15798.111634] dca fuse [15798.114342] ? __file_remove_privs+0x89/0x110 [15798.118951] [15798.118951] irq event stamp: 49486272 [15798.122347] ? wake_bit_function+0x70/0x70 [15798.126873] hardirqs last enabled at (49486271): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15798.129145] ? ceph_write_iter+0x260/0x770 [ceph] [15798.133493] hardirqs last disabled at (49486272): [] sysvec_apic_timer_interrupt+0xe/0x90 [15798.134984] ceph_write_iter+0x260/0x770 [ceph] [15798.138642] softirqs last enabled at (49486270): [] __do_softirq+0x2be/0x371 [15798.142752] ? lock_is_held_type+0xaa/0x120 [15798.152905] softirqs last disabled at (49486265): [] irq_exit_rcu+0xb0/0xd0 [15798.157607] vfs_write+0x324/0x4f0 [15798.167328] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15798.171858] ksys_pwrite64+0x65/0xa0 [15798.180538] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15798.184715] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15798.193225] Workqueue: events delayed_work [ceph] [15798.196623] do_syscall_64+0x38/0x80 [15798.205999] [15798.209572] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15798.217042] RIP: 0010:lock_acquire+0xf0/0x2b0 [15798.222171] RIP: 0033:0x7f48f0eeb93b [15798.226870] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15798.230441] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15798.231931] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15798.236976] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15798.241325] [15798.241325] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15798.244898] ORIG_RAX: 0000000000000012 [15798.263633] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15798.282371] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15798.287590] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15798.292808] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15798.294298] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15798.301422] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15798.305252] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15798.312377] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15798.319500] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15798.326625] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15798.333752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15798.340878] [15798.347997] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15798.392493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15798.399616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15798.406753] Call Trace: [15798.409196] [15798.411205] ? watchdog_timer_fn+0x1b4/0x220 [15798.415471] ? __hrtimer_run_queues+0x1d3/0x360 [15798.419995] ? hrtimer_interrupt+0x104/0x240 [15798.424257] ? __local_bh_enable+0x50/0x60 [15798.428352] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15798.433575] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15798.438531] [15798.440629] [15798.442726] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15798.448057] ? lock_acquire+0xf0/0x2b0 [15798.451801] ? igrab+0x43/0x50 [15798.454852] ? lock_release+0xc5/0x260 [15798.458598] _raw_spin_lock+0x30/0x40 [15798.462261] ? igrab+0x1d/0x50 [15798.465312] igrab+0x1d/0x50 [15798.468190] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15798.473340] delayed_work+0x1d8/0x280 [ceph] [15798.477620] ? process_scheduled_works+0x22a/0x510 [15798.482401] process_scheduled_works+0x2a4/0x510 [15798.487033] worker_thread+0x188/0x340 [15798.490784] ? rescuer_thread+0x310/0x310 [15798.494787] kthread+0xf5/0x130 [15798.497925] ? kthread_complete_and_exit+0x20/0x20 [15798.502709] ret_from_fork+0x31/0x40 [15798.506281] ? kthread_complete_and_exit+0x20/0x20 [15798.511063] ret_from_fork_asm+0x11/0x20 [15798.514988] [15807.375500] libceph: auth protocol 'cephx' mauth authentication failed: -13 [15825.731839] watchdog: BUG: soft lockup - CPU#0 stuck for 3624s! [dbench:78683] [15825.739053] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15825.743839] watchdog: BUG: soft lockup - CPU#7 stuck for 12621s! [kworker/7:4:78808] [15825.794514] irq event stamp: 20945264 [15825.802250] Modules linked in: ceph libceph dns_resolver fscache netfs veth [15825.805907] hardirqs last enabled at (20945263): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15825.812866] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15825.823025] hardirqs last disabled at (20945264): [] sysvec_apic_timer_interrupt+0xe/0x90 [15825.831283] nft_compat nf_tables nfnetlink bridge stp llc xfs [15825.841007] softirqs last enabled at (20945226): [] __do_softirq+0x2be/0x371 [15825.846832] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [15825.855517] softirqs last disabled at (20945221): [] irq_exit_rcu+0xb0/0xd0 [15825.864642] kvm_intel kvm irqbypass i2c_i801 [15825.873152] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15825.873154] iTCO_wdt [15825.873155] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15825.877514] iTCO_vendor_support i2c_smbus [15825.886457] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [15825.888726] crct10dif_pclmul joydev [15825.896198] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [15825.900287] crc32_pclmul [15825.905766] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15825.909336] ipmi_ssif ghash_clmulni_intel [15825.928074] [15825.930692] mei_me mei [15825.935909] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15825.939999] lpc_ich sha512_ssse3 [15825.941492] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15825.943933] acpi_ipmi mfd_core [15825.951059] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15825.954371] wmi ipmi_si [15825.961495] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15825.964649] ipmi_devintf ipmi_msghandler [15825.971772] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15825.974303] ioatdma [15825.981427] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15825.985432] acpi_pad acpi_power_meter [15825.992556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15825.994738] ixgbe igb [15826.002816] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15826.006562] mdio crc32c_intel [15826.012298] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15826.014655] nvme ptp [15826.021781] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15826.024831] nvme_core i2c_algo_bit [15826.031955] Call Trace: [15826.034236] pps_core dca [15826.041361] [15826.044844] fuse [15826.047289] ? watchdog_timer_fn+0x1b4/0x220 [15826.049905] [15826.049906] irq event stamp: 49708648 [15826.051916] ? __hrtimer_run_queues+0x1d3/0x360 [15826.053840] hardirqs last enabled at (49708647): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15826.058106] ? hrtimer_interrupt+0x104/0x240 [15826.059593] hardirqs last disabled at (49708648): [] sysvec_apic_timer_interrupt+0xe/0x90 [15826.063251] ? __local_bh_enable+0x50/0x60 [15826.067774] softirqs last enabled at (49708646): [] __do_softirq+0x2be/0x371 [15826.077934] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15826.082196] softirqs last disabled at (49708641): [] irq_exit_rcu+0xb0/0xd0 [15826.091922] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15826.096012] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15826.104695] [15826.109912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15826.118423] [15826.118424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15826.123381] Workqueue: events delayed_work [ceph] [15826.132763] ? queued_spin_lock_slowpath+0x2b/0x2e0 [15826.134857] [15826.142328] do_raw_spin_lock+0xb5/0xc0 [15826.144424] RIP: 0010:lock_release+0x14e/0x260 [15826.149731] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15826.154425] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [15826.159299] try_get_cap_refs+0x159/0x880 [ceph] [15826.160787] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [15826.164623] __ceph_get_caps+0x4a1/0x760 [ceph] [15826.169055] [15826.169055] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [15826.173843] ? __file_remove_privs+0x89/0x110 [15826.192577] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [15826.197189] ? wake_bit_function+0x70/0x70 [15826.202405] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [15826.206931] ? ceph_write_iter+0x260/0x770 [ceph] [15826.208419] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [15826.215545] ceph_write_iter+0x260/0x770 [ceph] [15826.219896] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [15826.227023] ? lock_is_held_type+0xaa/0x120 [15826.231110] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15826.238245] vfs_write+0x324/0x4f0 [15826.242941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15826.250071] ksys_pwrite64+0x65/0xa0 [15826.254588] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15826.261714] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15826.265890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15826.273969] do_syscall_64+0x38/0x80 [15826.277366] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15826.283104] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15826.286673] Call Trace: [15826.286674] [15826.293800] RIP: 0033:0x7f48f0eeb93b [15826.298928] ? watchdog_timer_fn+0x1b4/0x220 [15826.306052] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15826.309623] ? __hrtimer_run_queues+0x1d3/0x360 [15826.316748] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15826.321793] ? hrtimer_interrupt+0x104/0x240 [15826.324244] ORIG_RAX: 0000000000000012 [15826.326253] ? __local_bh_enable+0x50/0x60 [15826.329826] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15826.334093] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15826.352827] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15826.357352] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15826.362568] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15826.366833] [15826.370662] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15826.374755] [15826.381879] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15826.387095] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15826.394243] [15826.399196] ? igrab+0x43/0x50 [15826.435304] ? lock_release+0x14e/0x260 [15826.439142] ? lock_release+0xc5/0x260 [15826.442888] _raw_spin_unlock+0x1b/0x40 [15826.446724] igrab+0x43/0x50 [15826.449602] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15826.454751] delayed_work+0x1d8/0x280 [ceph] [15826.459035] ? process_scheduled_works+0x22a/0x510 [15826.463824] process_scheduled_works+0x2a4/0x510 [15826.468440] worker_thread+0x188/0x340 [15826.472187] ? rescuer_thread+0x310/0x310 [15826.476190] kthread+0xf5/0x130 [15826.479330] ? kthread_complete_and_exit+0x20/0x20 [15826.484113] ret_from_fork+0x31/0x40 [15826.487693] ? kthread_complete_and_exit+0x20/0x20 [15826.492476] ret_from_fork_asm+0x11/0x20 [15826.496400] [15849.359661] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [15853.732329] watchdog: BUG: soft lockup - CPU#0 stuck for 3650s! [dbench:78683] [15853.739547] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15853.744329] watchdog: BUG: soft lockup - CPU#7 stuck for 12647s! [kworker/7:4:78808] [15853.795002] irq event stamp: 21003294 [15853.795003] hardirqs last enabled at (21003293): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15853.802738] Modules linked in: [15853.806393] hardirqs last disabled at (21003294): [] sysvec_apic_timer_interrupt+0xe/0x90 [15853.816553] ceph [15853.819601] softirqs last enabled at (21003256): [] __do_softirq+0x2be/0x371 [15853.829327] libceph dns_resolver [15853.831251] softirqs last disabled at (21003251): [] irq_exit_rcu+0xb0/0xd0 [15853.839936] fscache netfs [15853.843246] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15853.851755] veth nft_chain_nat [15853.854461] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15853.863403] xt_MASQUERADE nf_nat [15853.866542] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [15853.874021] nf_conntrack nf_defrag_ipv6 [15853.877331] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [15853.882809] nf_defrag_ipv4 nft_compat [15853.886727] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15853.905466] nf_tables nfnetlink [15853.909209] [15853.909210] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15853.914427] bridge [15853.917650] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15853.919141] stp llc [15853.926265] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15853.928362] xfs libcrc32c [15853.935486] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15853.937669] sunrpc intel_rapl_msr [15853.944794] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15853.947498] intel_rapl_common sb_edac [15853.954622] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15853.958020] x86_pkg_temp_thermal coretemp [15853.965146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15853.968888] kvm_intel kvm [15853.976965] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15853.981056] irqbypass i2c_i801 [15853.986793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15853.989498] iTCO_wdt iTCO_vendor_support [15853.996622] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15853.999759] i2c_smbus [15854.006883] Call Trace: [15854.006884] [15854.010887] crct10dif_pclmul joydev [15854.018012] ? watchdog_timer_fn+0x1b4/0x220 [15854.020370] crc32_pclmul [15854.022815] ? __hrtimer_run_queues+0x1d3/0x360 [15854.024825] ipmi_ssif ghash_clmulni_intel [15854.028399] ? hrtimer_interrupt+0x104/0x240 [15854.032658] mei_me mei [15854.035275] ? __local_bh_enable+0x50/0x60 [15854.039800] lpc_ich sha512_ssse3 [15854.043893] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15854.048154] acpi_ipmi mfd_core [15854.050600] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15854.054689] wmi ipmi_si [15854.058016] [15854.063235] ipmi_devintf ipmi_msghandler [15854.066374] [15854.071329] ioatdma acpi_pad [15854.073862] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15854.075959] acpi_power_meter [15854.079967] ? queued_spin_lock_slowpath+0x2b/0x2e0 [15854.082060] ixgbe igb [15854.085026] do_raw_spin_lock+0xb5/0xc0 [15854.090328] mdio crc32c_intel nvme [15854.093295] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15854.098164] ptp nvme_core [15854.100524] try_get_cap_refs+0x159/0x880 [ceph] [15854.104352] i2c_algo_bit pps_core [15854.107840] __ceph_get_caps+0x4a1/0x760 [ceph] [15854.112620] dca fuse [15854.115328] ? __file_remove_privs+0x89/0x110 [15854.119933] [15854.119934] irq event stamp: 49930200 [15854.123332] ? wake_bit_function+0x70/0x70 [15854.127855] hardirqs last enabled at (49930199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15854.130129] ? ceph_write_iter+0x260/0x770 [ceph] [15854.134477] hardirqs last disabled at (49930200): [] sysvec_apic_timer_interrupt+0xe/0x90 [15854.135969] ceph_write_iter+0x260/0x770 [ceph] [15854.139625] softirqs last enabled at (49930198): [] __do_softirq+0x2be/0x371 [15854.143733] ? lock_is_held_type+0xaa/0x120 [15854.153889] softirqs last disabled at (49930193): [] irq_exit_rcu+0xb0/0xd0 [15854.158589] vfs_write+0x324/0x4f0 [15854.168312] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15854.172841] ksys_pwrite64+0x65/0xa0 [15854.181521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15854.185698] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15854.194210] Workqueue: events delayed_work [ceph] [15854.197608] do_syscall_64+0x38/0x80 [15854.206982] [15854.210555] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15854.218026] RIP: 0010:lock_acquire+0xf0/0x2b0 [15854.223156] RIP: 0033:0x7f48f0eeb93b [15854.227854] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15854.231423] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15854.232914] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15854.237960] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15854.242308] [15854.242309] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15854.245881] ORIG_RAX: 0000000000000012 [15854.264618] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15854.283355] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15854.288573] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15854.293791] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15854.295282] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15854.302406] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15854.306236] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15854.313359] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15854.320484] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15854.327608] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15854.334732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15854.341862] [15854.348982] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15854.393476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15854.400601] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15854.407734] Call Trace: [15854.410178] [15854.412189] ? watchdog_timer_fn+0x1b4/0x220 [15854.416455] ? __hrtimer_run_queues+0x1d3/0x360 [15854.420981] ? hrtimer_interrupt+0x104/0x240 [15854.425250] ? __local_bh_enable+0x50/0x60 [15854.429342] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15854.434567] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15854.439524] [15854.441621] [15854.443735] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15854.449045] ? lock_acquire+0xf0/0x2b0 [15854.452793] ? igrab+0x43/0x50 [15854.455844] ? lock_release+0xc5/0x260 [15854.459589] _raw_spin_lock+0x30/0x40 [15854.463246] ? igrab+0x1d/0x50 [15854.466296] igrab+0x1d/0x50 [15854.469174] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15854.474322] delayed_work+0x1d8/0x280 [ceph] [15854.478606] ? process_scheduled_works+0x22a/0x510 [15854.483394] process_scheduled_works+0x2a4/0x510 [15854.488027] worker_thread+0x188/0x340 [15854.491778] ? rescuer_thread+0x310/0x310 [15854.495788] kthread+0xf5/0x130 [15854.498926] ? kthread_complete_and_exit+0x20/0x20 [15854.503713] ret_from_fork+0x31/0x40 [15854.507307] ? kthread_complete_and_exit+0x20/0x20 [15854.512090] ret_from_fork_asm+0x11/0x20 [15854.516024] [15881.732820] watchdog: BUG: soft lockup - CPU#0 stuck for 3677s! [dbench:78683] [15881.740034] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15881.744820] watchdog: BUG: soft lockup - CPU#7 stuck for 12674s! [kworker/7:4:78808] [15881.795496] irq event stamp: 21061410 [15881.803225] Modules linked in: ceph libceph dns_resolver fscache [15881.806880] hardirqs last enabled at (21061409): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15881.812880] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [15881.823035] hardirqs last disabled at (21061410): [] sysvec_apic_timer_interrupt+0xe/0x90 [15881.828514] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [15881.838238] softirqs last enabled at (21061372): [] __do_softirq+0x2be/0x371 [15881.838242] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [15881.854836] softirqs last disabled at (21061367): [] irq_exit_rcu+0xb0/0xd0 [15881.862314] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [15881.870827] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15881.876477] kvm irqbypass [15881.885421] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15881.888123] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [15881.895596] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15881.901244] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [15881.906724] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15881.912288] ghash_clmulni_intel [15881.931025] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15881.931026] mei_me [15881.931027] [15881.934250] mei lpc_ich [15881.939466] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15881.941562] sha512_ssse3 acpi_ipmi [15881.943054] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15881.945584] mfd_core wmi [15881.952710] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15881.956193] ipmi_si ipmi_devintf [15881.963334] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15881.965952] ipmi_msghandler [15881.973076] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15881.976388] ioatdma [15881.983512] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15881.986387] acpi_pad acpi_power_meter [15881.993512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15881.995696] ixgbe igb [15882.003774] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15882.007517] mdio crc32c_intel [15882.013256] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15882.015612] nvme ptp [15882.022736] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15882.025787] nvme_core i2c_algo_bit [15882.032913] Call Trace: [15882.032915] [15882.035184] pps_core dca [15882.042310] ? watchdog_timer_fn+0x1b4/0x220 [15882.045790] fuse [15882.048236] ? __hrtimer_run_queues+0x1d3/0x360 [15882.050247] irq event stamp: 50151676 [15882.052866] ? hrtimer_interrupt+0x104/0x240 [15882.057126] hardirqs last enabled at (50151675): [] asm_common_interrupt+0x26/0x40 [15882.059051] ? __local_bh_enable+0x50/0x60 [15882.063574] hardirqs last disabled at (50151676): [] sysvec_apic_timer_interrupt+0xe/0x90 [15882.067235] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15882.071499] softirqs last enabled at (50151674): [] __do_softirq+0x2be/0x371 [15882.080701] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15882.084791] softirqs last disabled at (50151669): [] irq_exit_rcu+0xb0/0xd0 [15882.094515] [15882.099733] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15882.108417] [15882.108419] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15882.113375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15882.121891] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15882.123983] Workqueue: events delayed_work [ceph] [15882.133362] do_raw_spin_lock+0xb5/0xc0 [15882.135457] [15882.140763] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15882.148234] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [15882.153108] try_get_cap_refs+0x159/0x880 [ceph] [15882.157800] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [15882.161638] __ceph_get_caps+0x4a1/0x760 [ceph] [15882.163123] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [15882.167911] ? __file_remove_privs+0x89/0x110 [15882.172517] [15882.172518] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [15882.177128] ? wake_bit_function+0x70/0x70 [15882.195866] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [15882.200392] ? ceph_write_iter+0x260/0x770 [ceph] [15882.205608] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15882.209958] ceph_write_iter+0x260/0x770 [ceph] [15882.211448] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15882.218576] ? lock_is_held_type+0xaa/0x120 [15882.222665] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15882.229789] vfs_write+0x324/0x4f0 [15882.234485] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15882.241615] ksys_pwrite64+0x65/0xa0 [15882.246133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15882.253257] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15882.257436] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15882.264561] do_syscall_64+0x38/0x80 [15882.267956] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15882.276035] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15882.279606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15882.285343] RIP: 0033:0x7f48f0eeb93b [15882.290474] Call Trace: [15882.290475] [15882.297597] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15882.301169] ? watchdog_timer_fn+0x1b4/0x220 [15882.308292] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15882.313337] ? __hrtimer_run_queues+0x1d3/0x360 [15882.320459] ORIG_RAX: 0000000000000012 [15882.324035] ? hrtimer_interrupt+0x104/0x240 [15882.326474] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15882.328486] ? __local_bh_enable+0x50/0x60 [15882.347239] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15882.351507] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15882.356722] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15882.361247] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15882.365077] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15882.369342] [15882.376464] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15882.380556] [15882.387685] [15882.392897] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15882.430933] ? do_raw_spin_lock+0x42/0xc0 [15882.434941] igrab+0x1d/0x50 [15882.437821] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15882.442970] delayed_work+0x1d8/0x280 [ceph] [15882.447250] ? process_scheduled_works+0x22a/0x510 [15882.452041] process_scheduled_works+0x2a4/0x510 [15882.456655] worker_thread+0x188/0x340 [15882.460405] ? rescuer_thread+0x310/0x310 [15882.464408] kthread+0xf5/0x130 [15882.467548] ? kthread_complete_and_exit+0x20/0x20 [15882.472340] ret_from_fork+0x31/0x40 [15882.475918] ? kthread_complete_and_exit+0x20/0x20 [15882.480702] ret_from_fork_asm+0x11/0x20 [15882.484625] [15909.733310] watchdog: BUG: soft lockup - CPU#0 stuck for 3703s! [dbench:78683] [15909.740521] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15909.745310] watchdog: BUG: soft lockup - CPU#7 stuck for 12700s! [kworker/7:4:78808] [15909.795980] irq event stamp: 21119432 [15909.795981] hardirqs last enabled at (21119431): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15909.803710] Modules linked in: [15909.807369] hardirqs last disabled at (21119432): [] sysvec_apic_timer_interrupt+0xe/0x90 [15909.817526] ceph [15909.820577] softirqs last enabled at (21119394): [] __do_softirq+0x2be/0x371 [15909.830299] libceph [15909.832226] softirqs last disabled at (21119389): [] irq_exit_rcu+0xb0/0xd0 [15909.840908] dns_resolver fscache [15909.843093] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15909.851604] netfs veth [15909.854915] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15909.863859] nft_chain_nat xt_MASQUERADE [15909.866302] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15909.873774] nf_nat nf_conntrack [15909.877706] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15909.883186] nf_defrag_ipv6 [15909.886410] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15909.905146] nf_defrag_ipv4 nft_compat [15909.907938] [15909.907939] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15909.913155] nf_tables nfnetlink [15909.916899] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15909.918391] bridge stp [15909.925514] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15909.928738] llc xfs [15909.935863] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15909.938307] libcrc32c sunrpc [15909.945431] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15909.947613] intel_rapl_msr [15909.954739] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15909.957706] intel_rapl_common sb_edac [15909.964827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15909.967617] x86_pkg_temp_thermal coretemp [15909.975706] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15909.979448] kvm_intel kvm [15909.985184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15909.989275] irqbypass i2c_i801 [15909.996401] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15909.999104] iTCO_wdt iTCO_vendor_support [15910.006227] Call Trace: [15910.009366] i2c_smbus crct10dif_pclmul [15910.016489] [15910.020494] joydev [15910.022940] ? watchdog_timer_fn+0x1b4/0x220 [15910.026769] crc32_pclmul ipmi_ssif [15910.028779] ? __hrtimer_run_queues+0x1d3/0x360 [15910.030878] ghash_clmulni_intel mei_me [15910.035145] ? hrtimer_interrupt+0x104/0x240 [15910.038625] mei lpc_ich [15910.043149] ? __local_bh_enable+0x50/0x60 [15910.046989] sha512_ssse3 [15910.051256] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15910.053783] acpi_ipmi mfd_core [15910.057875] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15910.060491] wmi ipmi_si [15910.065708] [15910.068845] ipmi_devintf ipmi_msghandler [15910.073802] [15910.073804] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15910.076333] ioatdma acpi_pad [15910.078434] ? queued_spin_lock_slowpath+0x16/0x2e0 [15910.082437] acpi_power_meter [15910.084534] do_raw_spin_lock+0xb5/0xc0 [15910.089838] ixgbe igb mdio [15910.092803] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15910.097673] crc32c_intel nvme [15910.100637] try_get_cap_refs+0x159/0x880 [ceph] [15910.104466] ptp nvme_core [15910.107263] __ceph_get_caps+0x4a1/0x760 [ceph] [15910.112041] i2c_algo_bit pps_core [15910.115097] ? __file_remove_privs+0x89/0x110 [15910.119712] dca fuse [15910.122416] ? wake_bit_function+0x70/0x70 [15910.126940] [15910.126941] irq event stamp: 50373218 [15910.130341] ? ceph_write_iter+0x260/0x770 [ceph] [15910.134689] hardirqs last enabled at (50373217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15910.136992] ceph_write_iter+0x260/0x770 [ceph] [15910.141085] hardirqs last disabled at (50373218): [] sysvec_apic_timer_interrupt+0xe/0x90 [15910.142579] ? lock_is_held_type+0xaa/0x120 [15910.146232] softirqs last enabled at (50373216): [] __do_softirq+0x2be/0x371 [15910.150932] vfs_write+0x324/0x4f0 [15910.161087] softirqs last disabled at (50373211): [] irq_exit_rcu+0xb0/0xd0 [15910.165616] ksys_pwrite64+0x65/0xa0 [15910.175335] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15910.179512] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15910.188197] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15910.191597] do_syscall_64+0x38/0x80 [15910.200106] Workqueue: events delayed_work [ceph] [15910.203677] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15910.213054] [15910.213055] RIP: 0010:lock_acquire+0xf0/0x2b0 [15910.218183] RIP: 0033:0x7f48f0eeb93b [15910.225654] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15910.229227] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15910.233923] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15910.238967] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15910.240476] [15910.240477] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15910.244826] ORIG_RAX: 0000000000000012 [15910.248396] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15910.267135] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15910.285872] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15910.291088] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15910.296306] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15910.297799] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15910.304921] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15910.308754] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15910.315877] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15910.323000] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15910.330126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15910.337254] [15910.344372] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15910.395994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15910.403117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15910.410241] Call Trace: [15910.412685] [15910.414714] ? watchdog_timer_fn+0x1b4/0x220 [15910.418979] ? __hrtimer_run_queues+0x1d3/0x360 [15910.423523] ? hrtimer_interrupt+0x104/0x240 [15910.427792] ? __local_bh_enable+0x50/0x60 [15910.431884] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15910.437109] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15910.442066] [15910.444163] [15910.446260] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15910.451569] ? lock_acquire+0xf0/0x2b0 [15910.455319] ? igrab+0x43/0x50 [15910.458370] ? lock_release+0xc5/0x260 [15910.462115] _raw_spin_lock+0x30/0x40 [15910.465780] ? igrab+0x1d/0x50 [15910.468829] igrab+0x1d/0x50 [15910.471715] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15910.476865] delayed_work+0x1d8/0x280 [ceph] [15910.481148] ? process_scheduled_works+0x22a/0x510 [15910.485937] process_scheduled_works+0x2a4/0x510 [15910.490553] worker_thread+0x188/0x340 [15910.494303] ? rescuer_thread+0x310/0x310 [15910.498304] kthread+0xf5/0x130 [15910.501442] ? kthread_complete_and_exit+0x20/0x20 [15910.506228] ret_from_fork+0x31/0x40 [15910.509807] ? kthread_complete_and_exit+0x20/0x20 [15910.514592] ret_from_fork_asm+0x11/0x20 [15910.518513] [15910.877697] rcu: INFO: rcu_preempt self-detected stall on CPU [15910.883450] rcu: 7-....: (3862231 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/3861033 fqs=1921223 [15910.893868] rcu: (t=3974197 jiffies g=1309609 q=189214 ncpus=8) [15910.899867] Sending NMI from CPU 7 to CPUs 0: [15910.904217] NMI backtrace for cpu 0 [15910.904219] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15910.904221] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15910.904222] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15910.904225] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15910.904227] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15910.904228] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15910.904229] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15910.904230] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15910.904231] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15910.904231] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15910.904232] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15910.904233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15910.904234] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15910.904235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15910.904236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15910.904237] Call Trace: [15910.904238] [15910.904239] ? nmi_cpu_backtrace+0xad/0x120 [15910.904242] ? nmi_cpu_backtrace_handler+0x11/0x20 [15910.904244] ? nmi_handle+0xd4/0x260 [15910.904248] ? default_do_nmi+0x49/0xf0 [15910.904250] ? exc_nmi+0x11e/0x150 [15910.904251] ? end_repeat_nmi+0x16/0x67 [15910.904255] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15910.904258] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15910.904260] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15910.904263] [15910.904263] [15910.904264] do_raw_spin_lock+0xb5/0xc0 [15910.904266] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15910.904281] try_get_cap_refs+0x159/0x880 [ceph] [15910.904295] __ceph_get_caps+0x4a1/0x760 [ceph] [15910.904308] ? __file_remove_privs+0x89/0x110 [15910.904310] ? wake_bit_function+0x70/0x70 [15910.904312] ? ceph_write_iter+0x260/0x770 [ceph] [15910.904323] ceph_write_iter+0x260/0x770 [ceph] [15910.904334] ? lock_is_held_type+0xaa/0x120 [15910.904337] vfs_write+0x324/0x4f0 [15910.904341] ksys_pwrite64+0x65/0xa0 [15910.904343] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15910.904346] do_syscall_64+0x38/0x80 [15910.904348] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15910.904350] RIP: 0033:0x7f48f0eeb93b [15910.904351] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15910.904353] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [15910.904354] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15910.904355] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15910.904355] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15910.904356] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15910.904357] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15910.904360] [15910.905216] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15911.209045] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15911.216516] Workqueue: events delayed_work [ceph] [15911.221231] RIP: 0010:lock_acquire+0xf0/0x2b0 [15911.225581] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15911.244319] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15911.249536] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15911.256662] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15911.263786] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15911.270908] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15911.278033] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15911.285157] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15911.293236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15911.298972] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15911.306097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15911.313220] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15911.320345] Call Trace: [15911.322788] [15911.324800] ? rcu_dump_cpu_stacks+0xd1/0x180 [15911.329152] ? rcu_sched_clock_irq+0x633/0x1390 [15911.333676] ? lock_release+0xc5/0x260 [15911.337419] ? lock_is_held_type+0xaa/0x120 [15911.341597] ? lock_is_held_type+0xaa/0x120 [15911.345775] ? tick_sched_do_timer+0xa0/0xa0 [15911.350046] ? rcu_read_lock_sched_held+0x56/0x90 [15911.354744] ? tick_sched_do_timer+0xa0/0xa0 [15911.359010] ? update_process_times+0x5f/0x90 [15911.363359] ? tick_sched_handle.isra.16+0x1e/0x50 [15911.368142] ? tick_sched_timer+0x6f/0x80 [15911.372148] ? __hrtimer_run_queues+0x1d3/0x360 [15911.376675] ? hrtimer_interrupt+0x104/0x240 [15911.380942] ? __local_bh_enable+0x50/0x60 [15911.385038] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15911.390259] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15911.395217] [15911.397315] [15911.399413] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15911.404720] ? lock_acquire+0xf0/0x2b0 [15911.408470] ? igrab+0x43/0x50 [15911.411519] ? lock_release+0xc5/0x260 [15911.415267] _raw_spin_lock+0x30/0x40 [15911.418929] ? igrab+0x1d/0x50 [15911.421980] igrab+0x1d/0x50 [15911.424859] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15911.430010] delayed_work+0x1d8/0x280 [ceph] [15911.434291] ? process_scheduled_works+0x22a/0x510 [15911.439080] process_scheduled_works+0x2a4/0x510 [15911.443697] worker_thread+0x188/0x340 [15911.447445] ? rescuer_thread+0x310/0x310 [15911.451448] kthread+0xf5/0x130 [15911.454587] ? kthread_complete_and_exit+0x20/0x20 [15911.459371] ret_from_fork+0x31/0x40 [15911.462949] ? kthread_complete_and_exit+0x20/0x20 [15911.467733] ret_from_fork_asm+0x11/0x20 [15911.471657] [15937.733801] watchdog: BUG: soft lockup - CPU#0 stuck for 3729s! [dbench:78683] [15937.741018] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15937.745800] watchdog: BUG: soft lockup - CPU#7 stuck for 12726s! [kworker/7:4:78808] [15937.796474] irq event stamp: 21177596 [15937.804206] Modules linked in: ceph libceph dns_resolver fscache netfs [15937.807864] hardirqs last enabled at (21177595): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15937.807869] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15937.824548] hardirqs last disabled at (21177596): [] sysvec_apic_timer_interrupt+0xe/0x90 [15937.833232] nft_compat nf_tables nfnetlink bridge stp llc xfs [15937.842956] softirqs last enabled at (21177564): [] __do_softirq+0x2be/0x371 [15937.848779] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [15937.857463] softirqs last disabled at (21177559): [] irq_exit_rcu+0xb0/0xd0 [15937.866598] kvm_intel kvm irqbypass i2c_i801 [15937.875110] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15937.875111] iTCO_wdt [15937.875112] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15937.879476] iTCO_vendor_support i2c_smbus [15937.888422] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15937.890691] crct10dif_pclmul joydev [15937.898164] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15937.902254] crc32_pclmul ipmi_ssif [15937.907731] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15937.911301] ghash_clmulni_intel mei_me [15937.930039] [15937.933523] mei lpc_ich [15937.938740] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15937.942573] sha512_ssse3 acpi_ipmi [15937.944063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15937.946593] mfd_core wmi [15937.953719] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15937.957201] ipmi_si ipmi_devintf [15937.964326] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15937.966944] ipmi_msghandler ioatdma [15937.974067] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15937.977379] acpi_pad acpi_power_meter [15937.984503] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15937.988072] ixgbe igb [15937.995197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15937.998942] mdio crc32c_intel [15938.007020] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15938.009375] nvme ptp [15938.015115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15938.018165] nvme_core i2c_algo_bit [15938.025289] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15938.027559] pps_core dca [15938.034684] Call Trace: [15938.034686] [15938.038168] fuse [15938.045293] ? watchdog_timer_fn+0x1b4/0x220 [15938.047907] irq event stamp: 50590248 [15938.050355] ? __hrtimer_run_queues+0x1d3/0x360 [15938.052363] hardirqs last enabled at (50590247): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15938.054291] ? hrtimer_interrupt+0x104/0x240 [15938.058553] hardirqs last disabled at (50590248): [] sysvec_apic_timer_interrupt+0xe/0x90 [15938.062208] ? __local_bh_enable+0x50/0x60 [15938.066733] softirqs last enabled at (50590246): [] __do_softirq+0x2be/0x371 [15938.076894] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15938.081154] softirqs last disabled at (50590237): [] irq_exit_rcu+0xb0/0xd0 [15938.090880] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15938.094970] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15938.103654] [15938.108870] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15938.117382] [15938.122340] Workqueue: events delayed_work [ceph] [15938.131719] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15938.133813] [15938.133814] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [15938.141290] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15938.143382] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [15938.148082] do_raw_spin_lock+0xb5/0xc0 [15938.153384] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [15938.154877] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15938.159485] [15938.159486] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [15938.164359] try_get_cap_refs+0x159/0x880 [ceph] [15938.183095] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [15938.186929] __ceph_get_caps+0x4a1/0x760 [ceph] [15938.192143] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [15938.196931] ? __file_remove_privs+0x89/0x110 [15938.198417] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [15938.205542] ? wake_bit_function+0x70/0x70 [15938.210151] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [15938.217280] ? ceph_write_iter+0x260/0x770 [ceph] [15938.221801] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15938.228925] ceph_write_iter+0x260/0x770 [ceph] [15938.233274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15938.240404] ? lock_is_held_type+0xaa/0x120 [15938.244491] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15938.251616] vfs_write+0x324/0x4f0 [15938.256312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15938.264393] ksys_pwrite64+0x65/0xa0 [15938.268913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15938.274650] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15938.278827] Call Trace: [15938.278828] [15938.285954] do_syscall_64+0x38/0x80 [15938.289349] ? watchdog_timer_fn+0x1b4/0x220 [15938.296483] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15938.300055] ? __hrtimer_run_queues+0x1d3/0x360 [15938.307178] RIP: 0033:0x7f48f0eeb93b [15938.312329] ? hrtimer_interrupt+0x104/0x240 [15938.314769] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15938.316779] ? __local_bh_enable+0x50/0x60 [15938.320352] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15938.324618] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15938.329658] ORIG_RAX: 0000000000000012 [15938.334182] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15938.337771] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15938.342035] [15938.342036] [15938.360773] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15938.364865] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15938.370081] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15938.375303] ? do_raw_spin_lock+0x42/0xc0 [15938.379129] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15938.384087] igrab+0x1d/0x50 [15938.391210] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15938.393308] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15938.395412] [15938.402533] delayed_work+0x1d8/0x280 [ceph] [15938.447670] ? process_scheduled_works+0x22a/0x510 [15938.452460] process_scheduled_works+0x2a4/0x510 [15938.457094] worker_thread+0x188/0x340 [15938.460842] ? rescuer_thread+0x310/0x310 [15938.464853] kthread+0xf5/0x130 [15938.467992] ? kthread_complete_and_exit+0x20/0x20 [15938.472774] ret_from_fork+0x31/0x40 [15938.476354] ? kthread_complete_and_exit+0x20/0x20 [15938.481137] ret_from_fork_asm+0x11/0x20 [15938.485061] [15965.734291] watchdog: BUG: soft lockup - CPU#0 stuck for 3755s! [dbench:78683] [15965.741504] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15965.746291] watchdog: BUG: soft lockup - CPU#7 stuck for 12752s! [kworker/7:4:78808] [15965.796961] irq event stamp: 21235640 [15965.804694] Modules linked in: ceph libceph [15965.808349] hardirqs last enabled at (21235639): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15965.812527] dns_resolver fscache netfs [15965.822686] hardirqs last disabled at (21235640): [] sysvec_apic_timer_interrupt+0xe/0x90 [15965.826517] veth nft_chain_nat xt_MASQUERADE [15965.836239] softirqs last enabled at (21235612): [] __do_softirq+0x2be/0x371 [15965.836241] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [15965.849276] softirqs last disabled at (21235607): [] irq_exit_rcu+0xb0/0xd0 [15965.849277] nft_compat [15965.849279] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15965.855101] nf_tables nfnetlink [15965.863611] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15965.866054] bridge stp [15965.875000] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [15965.878223] llc xfs [15965.885693] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [15965.888137] libcrc32c sunrpc [15965.893614] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15965.895799] intel_rapl_msr intel_rapl_common [15965.914536] [15965.914537] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15965.917501] sb_edac x86_pkg_temp_thermal [15965.922719] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15965.927068] coretemp kvm_intel [15965.928559] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15965.935687] kvm irqbypass [15965.939690] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15965.946811] i2c_i801 iTCO_wdt [15965.949949] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15965.957090] iTCO_vendor_support [15965.959794] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15965.966918] i2c_smbus crct10dif_pclmul [15965.969969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15965.977094] joydev crc32_pclmul [15965.980318] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15965.988397] ipmi_ssif ghash_clmulni_intel [15965.992226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15965.997967] mei_me [15966.001187] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15966.008311] mei lpc_ich [15966.012403] Call Trace: [15966.012404] [15966.019526] sha512_ssse3 [15966.021624] ? watchdog_timer_fn+0x1b4/0x220 [15966.028749] acpi_ipmi mfd_core [15966.031280] ? __hrtimer_run_queues+0x1d3/0x360 [15966.033722] wmi ipmi_si [15966.035737] ? hrtimer_interrupt+0x104/0x240 [15966.038352] ipmi_devintf ipmi_msghandler [15966.042615] ? __local_bh_enable+0x50/0x60 [15966.045752] ioatdma acpi_pad [15966.050279] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15966.052806] acpi_power_meter ixgbe [15966.057071] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15966.061076] igb mdio [15966.065167] [15966.068130] crc32c_intel nvme [15966.073348] [15966.073349] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15966.076831] ptp nvme_core [15966.081793] ? queued_spin_lock_slowpath+0x16/0x2e0 [15966.084059] i2c_algo_bit pps_core [15966.086160] do_raw_spin_lock+0xb5/0xc0 [15966.089208] dca fuse [15966.091308] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15966.096609] [15966.096610] irq event stamp: 50812374 [15966.099317] try_get_cap_refs+0x159/0x880 [ceph] [15966.104184] hardirqs last enabled at (50812373): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15966.107587] __ceph_get_caps+0x4a1/0x760 [ceph] [15966.111412] hardirqs last disabled at (50812374): [] sysvec_apic_timer_interrupt+0xe/0x90 [15966.113695] ? __file_remove_privs+0x89/0x110 [15966.118477] softirqs last enabled at (50812372): [] __do_softirq+0x2be/0x371 [15966.119970] ? wake_bit_function+0x70/0x70 [15966.123625] softirqs last disabled at (50812367): [] irq_exit_rcu+0xb0/0xd0 [15966.128237] ? ceph_write_iter+0x260/0x770 [ceph] [15966.138393] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15966.142916] ceph_write_iter+0x260/0x770 [ceph] [15966.152641] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15966.156995] ? lock_is_held_type+0xaa/0x120 [15966.165691] Workqueue: events delayed_work [ceph] [15966.169787] vfs_write+0x324/0x4f0 [15966.178293] [15966.182996] ksys_pwrite64+0x65/0xa0 [15966.192369] RIP: 0010:lock_acquire+0xc/0x2b0 [15966.196893] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15966.204364] Code: 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 <41> 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 [15966.208543] do_syscall_64+0x38/0x80 [15966.213238] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [15966.216636] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15966.218126] [15966.218127] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [15966.221698] RIP: 0033:0x7f48f0eeb93b [15966.225961] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [15966.231095] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15966.249829] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [15966.253401] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15966.258619] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [15966.263662] ORIG_RAX: 0000000000000012 [15966.265153] R13: ffff88817f867478 R14: 00000001001d239b R15: 0000000000000000 [15966.272277] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15966.275848] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15966.282982] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15966.301718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15966.308843] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15966.314060] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15966.321183] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15966.325015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15966.332140] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15966.339262] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15966.347346] [15966.354465] Call Trace: [15966.354466] [15966.409586] ? watchdog_timer_fn+0x1b4/0x220 [15966.413852] ? __hrtimer_run_queues+0x1d3/0x360 [15966.418378] ? hrtimer_interrupt+0x104/0x240 [15966.422647] ? __local_bh_enable+0x50/0x60 [15966.426741] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15966.431978] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15966.436940] [15966.439036] [15966.441134] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15966.446441] ? lock_acquire+0xc/0x2b0 [15966.450105] _raw_spin_lock+0x30/0x40 [15966.453771] ? igrab+0x1d/0x50 [15966.456822] igrab+0x1d/0x50 [15966.459700] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15966.464848] delayed_work+0x1d8/0x280 [ceph] [15966.469131] ? process_scheduled_works+0x22a/0x510 [15966.473921] process_scheduled_works+0x2a4/0x510 [15966.478535] worker_thread+0x188/0x340 [15966.482286] ? rescuer_thread+0x310/0x310 [15966.486297] kthread+0xf5/0x130 [15966.489434] ? kthread_complete_and_exit+0x20/0x20 [15966.494218] ret_from_fork+0x31/0x40 [15966.497788] ? kthread_complete_and_exit+0x20/0x20 [15966.502573] ret_from_fork_asm+0x11/0x20 [15966.506496] [15993.734782] watchdog: BUG: soft lockup - CPU#0 stuck for 3781s! [dbench:78683] [15993.742001] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [15993.746781] watchdog: BUG: soft lockup - CPU#7 stuck for 12778s! [kworker/7:4:78808] [15993.797464] irq event stamp: 21293758 [15993.805198] Modules linked in: ceph libceph dns_resolver fscache netfs [15993.808854] hardirqs last enabled at (21293757): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15993.815380] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [15993.825537] hardirqs last disabled at (21293758): [] sysvec_apic_timer_interrupt+0xe/0x90 [15993.832923] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [15993.842646] softirqs last enabled at (21293730): [] __do_softirq+0x2be/0x371 [15993.849425] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [15993.858109] softirqs last disabled at (21293725): [] irq_exit_rcu+0xb0/0xd0 [15993.866800] coretemp kvm_intel kvm irqbypass [15993.875313] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15993.875314] i2c_i801 [15993.875315] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15993.879663] iTCO_wdt iTCO_vendor_support [15993.888608] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [15993.890877] i2c_smbus crct10dif_pclmul [15993.898349] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [15993.902352] joydev crc32_pclmul [15993.907831] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [15993.911661] ipmi_ssif ghash_clmulni_intel [15993.930399] [15993.933622] mei_me mei [15993.938841] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [15993.942931] lpc_ich sha512_ssse3 [15993.944422] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [15993.946865] acpi_ipmi [15993.953990] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [15993.957300] mfd_core wmi [15993.964423] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [15993.966781] ipmi_si ipmi_devintf [15993.973907] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [15993.976523] ipmi_msghandler [15993.983649] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [15993.986957] ioatdma [15993.994081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15993.996960] acpi_pad acpi_power_meter [15994.005037] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [15994.007220] ixgbe [15994.012958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15994.016702] igb mdio [15994.023826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15994.025837] crc32c_intel nvme [15994.032963] Call Trace: [15994.032965] [15994.035231] ptp nvme_core [15994.042360] ? watchdog_timer_fn+0x1b4/0x220 [15994.045407] i2c_algo_bit pps_core [15994.047853] ? __hrtimer_run_queues+0x1d3/0x360 [15994.049861] dca fuse [15994.052570] ? hrtimer_interrupt+0x104/0x240 [15994.056829] [15994.056830] irq event stamp: 51034416 [15994.060227] ? __local_bh_enable+0x50/0x60 [15994.064750] hardirqs last enabled at (51034415): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15994.067025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15994.071286] hardirqs last disabled at (51034416): [] sysvec_apic_timer_interrupt+0xe/0x90 [15994.072779] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15994.076435] softirqs last enabled at (51034414): [] __do_softirq+0x2be/0x371 [15994.080525] [15994.090682] softirqs last disabled at (51034409): [] irq_exit_rcu+0xb0/0xd0 [15994.095900] [15994.105623] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [15994.110583] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15994.119265] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [15994.121368] ? queued_spin_lock_slowpath+0x2d/0x2e0 [15994.129873] Workqueue: events delayed_work [ceph] [15994.131972] do_raw_spin_lock+0xb5/0xc0 [15994.141348] [15994.141349] RIP: 0010:lock_acquire+0xf0/0x2b0 [15994.146655] __cap_delay_requeue+0xa7/0x1d0 [ceph] [15994.154123] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [15994.158998] try_get_cap_refs+0x159/0x880 [ceph] [15994.163692] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [15994.167528] __ceph_get_caps+0x4a1/0x760 [ceph] [15994.169014] [15994.169014] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [15994.173369] ? __file_remove_privs+0x89/0x110 [15994.178149] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [15994.196887] ? wake_bit_function+0x70/0x70 [15994.201497] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [15994.206716] ? ceph_write_iter+0x260/0x770 [ceph] [15994.211238] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [15994.212730] ceph_write_iter+0x260/0x770 [ceph] [15994.219853] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [15994.224208] ? lock_is_held_type+0xaa/0x120 [15994.231329] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [15994.235422] vfs_write+0x324/0x4f0 [15994.242544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15994.247247] ksys_pwrite64+0x65/0xa0 [15994.254364] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [15994.258889] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [15994.266012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15994.270190] do_syscall_64+0x38/0x80 [15994.278268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [15994.281666] entry_SYSCALL_64_after_hwframe+0x63/0xcd [15994.287402] Call Trace: [15994.287404] [15994.290975] RIP: 0033:0x7f48f0eeb93b [15994.298098] ? watchdog_timer_fn+0x1b4/0x220 [15994.303230] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [15994.310354] ? __hrtimer_run_queues+0x1d3/0x360 [15994.313922] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [15994.321051] ? hrtimer_interrupt+0x104/0x240 [15994.326091] ORIG_RAX: 0000000000000012 [15994.328535] ? __local_bh_enable+0x50/0x60 [15994.330545] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [15994.334119] ? __sysvec_apic_timer_interrupt+0x61/0x160 [15994.338381] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [15994.357119] ? sysvec_apic_timer_interrupt+0x6d/0x90 [15994.361642] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [15994.366859] [15994.371124] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [15994.374954] [15994.379044] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [15994.386187] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [15994.391411] [15994.398533] ? lock_acquire+0xf0/0x2b0 [15994.440287] ? igrab+0x43/0x50 [15994.443337] ? lock_release+0xc5/0x260 [15994.447082] _raw_spin_lock+0x30/0x40 [15994.450746] ? igrab+0x1d/0x50 [15994.453797] igrab+0x1d/0x50 [15994.456677] ceph_check_delayed_caps+0xb7/0x270 [ceph] [15994.461826] delayed_work+0x1d8/0x280 [ceph] [15994.466106] ? process_scheduled_works+0x22a/0x510 [15994.470897] process_scheduled_works+0x2a4/0x510 [15994.475514] worker_thread+0x188/0x340 [15994.479260] ? rescuer_thread+0x310/0x310 [15994.483263] kthread+0xf5/0x130 [15994.486404] ? kthread_complete_and_exit+0x20/0x20 [15994.491196] ret_from_fork+0x31/0x40 [15994.494774] ? kthread_complete_and_exit+0x20/0x20 [15994.499559] ret_from_fork_asm+0x11/0x20 [15994.503491] [16021.735272] watchdog: BUG: soft lockup - CPU#0 stuck for 3807s! [dbench:78683] [16021.742487] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16021.747272] watchdog: BUG: soft lockup - CPU#7 stuck for 12804s! [kworker/7:4:78808] [16021.797943] irq event stamp: 21351774 [16021.797944] hardirqs last enabled at (21351773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16021.805676] Modules linked in: [16021.809332] hardirqs last disabled at (21351774): [] sysvec_apic_timer_interrupt+0xe/0x90 [16021.819491] ceph [16021.822542] softirqs last enabled at (21351746): [] __do_softirq+0x2be/0x371 [16021.832264] libceph [16021.834190] softirqs last disabled at (21351741): [] irq_exit_rcu+0xb0/0xd0 [16021.842873] dns_resolver [16021.845058] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16021.853567] fscache [16021.856187] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16021.865129] netfs veth [16021.867315] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16021.874783] nft_chain_nat xt_MASQUERADE [16021.877229] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16021.882707] nf_nat nf_conntrack [16021.886623] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16021.905361] nf_defrag_ipv6 nf_defrag_ipv4 [16021.908584] [16021.913803] nft_compat nf_tables [16021.917894] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16021.919384] nfnetlink bridge [16021.922693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16021.929818] stp llc [16021.932782] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16021.939907] xfs libcrc32c [16021.942091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16021.949215] sunrpc intel_rapl_msr [16021.951919] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16021.959044] intel_rapl_common sb_edac [16021.962440] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16021.969564] x86_pkg_temp_thermal [16021.973310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16021.981387] coretemp kvm_intel [16021.984697] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16021.990436] kvm irqbypass [16021.993573] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16022.000695] i2c_i801 [16022.003401] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16022.010525] iTCO_wdt iTCO_vendor_support [16022.012795] Call Trace: [16022.012797] [16022.019919] i2c_smbus crct10dif_pclmul [16022.023925] ? watchdog_timer_fn+0x1b4/0x220 [16022.026368] joydev crc32_pclmul [16022.028378] ? __hrtimer_run_queues+0x1d3/0x360 [16022.032207] ipmi_ssif [16022.036477] ? hrtimer_interrupt+0x104/0x240 [16022.039696] ghash_clmulni_intel mei_me [16022.044220] ? __local_bh_enable+0x50/0x60 [16022.046579] mei lpc_ich [16022.050844] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16022.054673] sha512_ssse3 acpi_ipmi [16022.058765] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16022.061294] mfd_core wmi [16022.066511] [16022.069997] ipmi_si ipmi_devintf [16022.074953] [16022.077570] ipmi_msghandler [16022.079681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16022.082988] ioatdma [16022.085089] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16022.087962] acpi_pad [16022.093268] do_raw_spin_lock+0xb5/0xc0 [16022.095450] acpi_power_meter ixgbe [16022.100321] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16022.102593] igb mdio [16022.106425] try_get_cap_refs+0x159/0x880 [ceph] [16022.109906] crc32c_intel nvme [16022.114695] __ceph_get_caps+0x4a1/0x760 [ceph] [16022.116963] ptp [16022.121576] ? __file_remove_privs+0x89/0x110 [16022.124623] nvme_core i2c_algo_bit [16022.129147] ? wake_bit_function+0x70/0x70 [16022.130985] pps_core dca [16022.135338] ? ceph_write_iter+0x260/0x770 [ceph] [16022.138820] fuse [16022.142911] ceph_write_iter+0x260/0x770 [ceph] [16022.145526] irq event stamp: 51256792 [16022.150229] ? lock_is_held_type+0xaa/0x120 [16022.152149] hardirqs last enabled at (51256791): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16022.156675] vfs_write+0x324/0x4f0 [16022.160331] hardirqs last disabled at (51256792): [] sysvec_apic_timer_interrupt+0xe/0x90 [16022.164511] ksys_pwrite64+0x65/0xa0 [16022.174671] softirqs last enabled at (51256790): [] __do_softirq+0x2be/0x371 [16022.178072] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16022.187794] softirqs last disabled at (51256785): [] irq_exit_rcu+0xb0/0xd0 [16022.191368] do_syscall_64+0x38/0x80 [16022.200051] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16022.205181] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16022.213693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16022.217263] RIP: 0033:0x7f48f0eeb93b [16022.226640] Workqueue: events delayed_work [ceph] [16022.231684] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16022.239156] [16022.242727] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16022.247424] RIP: 0010:lock_acquire+0xf0/0x2b0 [16022.266161] ORIG_RAX: 0000000000000012 [16022.267669] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16022.272887] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16022.277237] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16022.281069] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16022.299806] [16022.299806] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16022.306930] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16022.312147] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16022.319272] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16022.320762] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16022.327887] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16022.335011] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16022.342139] [16022.349258] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16022.379956] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16022.388035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16022.393771] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16022.400895] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16022.408020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16022.415145] Call Trace: [16022.417588] [16022.419599] ? watchdog_timer_fn+0x1b4/0x220 [16022.423864] ? __hrtimer_run_queues+0x1d3/0x360 [16022.428390] ? hrtimer_interrupt+0x104/0x240 [16022.432652] ? __local_bh_enable+0x50/0x60 [16022.436761] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16022.441978] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16022.446935] [16022.449032] [16022.451130] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16022.456437] ? lock_acquire+0xf0/0x2b0 [16022.460188] ? igrab+0x43/0x50 [16022.463245] ? lock_release+0xc5/0x260 [16022.466991] _raw_spin_lock+0x30/0x40 [16022.470673] ? igrab+0x1d/0x50 [16022.473724] igrab+0x1d/0x50 [16022.476602] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16022.481752] delayed_work+0x1d8/0x280 [ceph] [16022.486032] ? process_scheduled_works+0x22a/0x510 [16022.490813] process_scheduled_works+0x2a4/0x510 [16022.495428] worker_thread+0x188/0x340 [16022.499178] ? rescuer_thread+0x310/0x310 [16022.503182] kthread+0xf5/0x130 [16022.506318] ? kthread_complete_and_exit+0x20/0x20 [16022.511106] ret_from_fork+0x31/0x40 [16022.514682] ? kthread_complete_and_exit+0x20/0x20 [16022.519468] ret_from_fork_asm+0x11/0x20 [16022.523390] [16049.735763] watchdog: BUG: soft lockup - CPU#0 stuck for 3833s! [dbench:78683] [16049.742983] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16049.747762] watchdog: BUG: soft lockup - CPU#7 stuck for 12830s! [kworker/7:4:78808] [16049.798438] irq event stamp: 21409918 [16049.806171] Modules linked in: ceph libceph dns_resolver fscache netfs veth [16049.809827] hardirqs last enabled at (21409917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16049.816787] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [16049.826944] hardirqs last disabled at (21409918): [] sysvec_apic_timer_interrupt+0xe/0x90 [16049.833896] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [16049.843621] softirqs last enabled at (21409890): [] __do_softirq+0x2be/0x371 [16049.849704] stp llc xfs libcrc32c sunrpc intel_rapl_msr [16049.858387] softirqs last disabled at (21409885): [] irq_exit_rcu+0xb0/0xd0 [16049.863693] intel_rapl_common sb_edac x86_pkg_temp_thermal [16049.872203] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16049.872205] coretemp [16049.872206] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16049.877768] kvm_intel kvm [16049.886712] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16049.888983] irqbypass i2c_i801 [16049.896455] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16049.899157] iTCO_wdt [16049.904634] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16049.907772] iTCO_vendor_support i2c_smbus [16049.926510] [16049.926510] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16049.928781] crct10dif_pclmul joydev [16049.933998] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16049.938087] crc32_pclmul [16049.939579] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16049.946704] ipmi_ssif ghash_clmulni_intel [16049.950273] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16049.957397] mei_me mei [16049.960015] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16049.967156] lpc_ich [16049.971248] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16049.978371] sha512_ssse3 acpi_ipmi [16049.980815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16049.987940] mfd_core [16049.990123] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16049.998201] wmi ipmi_si [16050.001687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16050.007422] ipmi_devintf ipmi_msghandler [16050.009694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16050.016817] ioatdma acpi_pad [16050.019348] Call Trace: [16050.026472] acpi_power_meter [16050.030479] [16050.037600] ixgbe igb [16050.040568] ? watchdog_timer_fn+0x1b4/0x220 [16050.043009] mdio crc32c_intel [16050.045975] ? __hrtimer_run_queues+0x1d3/0x360 [16050.047983] nvme [16050.050344] ? hrtimer_interrupt+0x104/0x240 [16050.054604] ptp nvme_core [16050.057656] ? __local_bh_enable+0x50/0x60 [16050.062180] i2c_algo_bit [16050.064106] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16050.068368] pps_core dca [16050.071072] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16050.075163] fuse [16050.077780] [16050.082997] irq event stamp: 51479130 [16050.085616] [16050.090571] hardirqs last enabled at (51479129): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16050.092497] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16050.094593] hardirqs last disabled at (51479130): [] sysvec_apic_timer_interrupt+0xe/0x90 [16050.098254] ? queued_spin_lock_slowpath+0x18/0x2e0 [16050.100349] softirqs last enabled at (51479128): [] __do_softirq+0x2be/0x371 [16050.110508] do_raw_spin_lock+0xb5/0xc0 [16050.115810] softirqs last disabled at (51479123): [] irq_exit_rcu+0xb0/0xd0 [16050.125536] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16050.130406] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16050.139092] try_get_cap_refs+0x159/0x880 [ceph] [16050.142919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16050.151447] __ceph_get_caps+0x4a1/0x760 [ceph] [16050.156224] Workqueue: events delayed_work [ceph] [16050.165605] ? __file_remove_privs+0x89/0x110 [16050.170212] [16050.170213] RIP: 0010:lock_acquire+0xf0/0x2b0 [16050.177685] ? wake_bit_function+0x70/0x70 [16050.182207] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16050.186906] ? ceph_write_iter+0x260/0x770 [ceph] [16050.191256] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16050.192747] ceph_write_iter+0x260/0x770 [ceph] [16050.197095] [16050.201190] ? lock_is_held_type+0xaa/0x120 [16050.219925] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16050.224625] vfs_write+0x324/0x4f0 [16050.229840] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16050.234370] ksys_pwrite64+0x65/0xa0 [16050.235856] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16050.240033] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16050.247163] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16050.250562] do_syscall_64+0x38/0x80 [16050.257687] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16050.261258] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16050.268381] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16050.273512] RIP: 0033:0x7f48f0eeb93b [16050.280637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16050.284207] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16050.291332] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16050.296376] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16050.304452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16050.308023] ORIG_RAX: 0000000000000012 [16050.313762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16050.332498] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16050.339621] Call Trace: [16050.339623] [16050.344839] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16050.351965] ? watchdog_timer_fn+0x1b4/0x220 [16050.355796] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16050.362919] ? __hrtimer_run_queues+0x1d3/0x360 [16050.370043] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16050.372490] ? hrtimer_interrupt+0x104/0x240 [16050.374497] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16050.381623] ? __local_bh_enable+0x50/0x60 [16050.385891] [16050.393012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16050.427541] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16050.432498] [16050.434593] [16050.436691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16050.442000] ? lock_acquire+0xf0/0x2b0 [16050.445750] ? igrab+0x43/0x50 [16050.448809] ? lock_release+0xc5/0x260 [16050.452552] _raw_spin_lock+0x30/0x40 [16050.456209] ? igrab+0x1d/0x50 [16050.459260] igrab+0x1d/0x50 [16050.462139] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16050.467305] delayed_work+0x1d8/0x280 [ceph] [16050.471588] ? process_scheduled_works+0x22a/0x510 [16050.476377] process_scheduled_works+0x2a4/0x510 [16050.480991] worker_thread+0x188/0x340 [16050.484742] ? rescuer_thread+0x310/0x310 [16050.488744] kthread+0xf5/0x130 [16050.491884] ? kthread_complete_and_exit+0x20/0x20 [16050.496675] ret_from_fork+0x31/0x40 [16050.500255] ? kthread_complete_and_exit+0x20/0x20 [16050.505038] ret_from_fork_asm+0x11/0x20 [16050.508971] [16077.736252] watchdog: BUG: soft lockup - CPU#0 stuck for 3859s! [dbench:78683] [16077.743469] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16077.748252] watchdog: BUG: soft lockup - CPU#7 stuck for 12856s! [kworker/7:4:78808] [16077.798925] irq event stamp: 21467954 [16077.798926] hardirqs last enabled at (21467953): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16077.806658] Modules linked in: ceph [16077.810315] hardirqs last disabled at (21467954): [] sysvec_apic_timer_interrupt+0xe/0x90 [16077.820472] libceph [16077.823956] softirqs last enabled at (21467926): [] __do_softirq+0x2be/0x371 [16077.833681] dns_resolver fscache [16077.835866] softirqs last disabled at (21467921): [] irq_exit_rcu+0xb0/0xd0 [16077.844548] netfs [16077.847861] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16077.856371] veth nft_chain_nat [16077.858381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16077.867324] xt_MASQUERADE nf_nat [16077.870462] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16077.877941] nf_conntrack [16077.881253] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16077.886730] nf_defrag_ipv6 nf_defrag_ipv4 [16077.889348] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16077.908086] nft_compat nf_tables [16077.912176] [16077.912177] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16077.917394] nfnetlink bridge [16077.920705] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16077.922196] stp llc [16077.929319] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16077.932282] xfs libcrc32c [16077.939408] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16077.941591] sunrpc intel_rapl_msr [16077.948716] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16077.951419] intel_rapl_common [16077.958544] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16077.961942] sb_edac x86_pkg_temp_thermal [16077.969064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16077.972115] coretemp kvm_intel [16077.980193] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16077.984197] kvm irqbypass [16077.989935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16077.993072] i2c_i801 [16078.000195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16078.002899] iTCO_wdt [16078.010024] Call Trace: [16078.010025] [16078.012294] iTCO_vendor_support i2c_smbus [16078.019422] ? watchdog_timer_fn+0x1b4/0x220 [16078.021690] crct10dif_pclmul joydev [16078.024135] ? __hrtimer_run_queues+0x1d3/0x360 [16078.026146] crc32_pclmul ipmi_ssif [16078.030238] ? hrtimer_interrupt+0x104/0x240 [16078.034499] ghash_clmulni_intel mei_me [16078.038072] ? __local_bh_enable+0x50/0x60 [16078.042594] mei [16078.046082] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16078.050343] lpc_ich sha512_ssse3 [16078.054174] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16078.058264] acpi_ipmi [16078.060102] [16078.065319] mfd_core wmi [16078.068648] [16078.073605] ipmi_si ipmi_devintf [16078.075963] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16078.078058] ipmi_msghandler [16078.080680] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16078.082774] ioatdma acpi_pad [16078.086088] do_raw_spin_lock+0xb5/0xc0 [16078.091388] acpi_power_meter ixgbe igb [16078.094267] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16078.099138] mdio crc32c_intel [16078.102102] try_get_cap_refs+0x159/0x880 [ceph] [16078.105931] nvme ptp [16078.109767] __ceph_get_caps+0x4a1/0x760 [ceph] [16078.114547] nvme_core [16078.117602] ? __file_remove_privs+0x89/0x110 [16078.122208] i2c_algo_bit [16078.124481] ? wake_bit_function+0x70/0x70 [16078.129004] pps_core dca [16078.131363] ? ceph_write_iter+0x260/0x770 [ceph] [16078.135712] fuse [16078.138329] ceph_write_iter+0x260/0x770 [ceph] [16078.142419] irq event stamp: 51701928 [16078.145040] ? lock_is_held_type+0xaa/0x120 [16078.149733] hardirqs last enabled at (51701927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16078.151661] vfs_write+0x324/0x4f0 [16078.156183] hardirqs last disabled at (51701928): [] sysvec_apic_timer_interrupt+0xe/0x90 [16078.159844] ksys_pwrite64+0x65/0xa0 [16078.164018] softirqs last enabled at (51701926): [] __do_softirq+0x2be/0x371 [16078.174175] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16078.177573] softirqs last disabled at (51701921): [] irq_exit_rcu+0xb0/0xd0 [16078.187299] do_syscall_64+0x38/0x80 [16078.190866] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16078.199552] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16078.204682] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16078.213192] RIP: 0033:0x7f48f0eeb93b [16078.216764] Workqueue: events delayed_work [ceph] [16078.226142] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16078.231185] [16078.231186] RIP: 0010:lock_acquire+0xf0/0x2b0 [16078.238656] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16078.242225] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16078.246941] ORIG_RAX: 0000000000000012 [16078.265680] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16078.267169] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16078.271521] [16078.271522] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16078.276737] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16078.295475] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16078.299307] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16078.304524] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16078.311654] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16078.313148] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16078.320271] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16078.327396] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16078.334526] [16078.341657] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16078.387543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16078.393281] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16078.400406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16078.407530] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16078.414658] Call Trace: [16078.417107] [16078.419117] ? watchdog_timer_fn+0x1b4/0x220 [16078.423381] ? __hrtimer_run_queues+0x1d3/0x360 [16078.427909] ? hrtimer_interrupt+0x104/0x240 [16078.432177] ? __local_bh_enable+0x50/0x60 [16078.436272] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16078.441494] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16078.446453] [16078.448550] [16078.450659] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16078.455973] ? lock_acquire+0xf0/0x2b0 [16078.459723] ? igrab+0x43/0x50 [16078.462772] ? lock_release+0xc5/0x260 [16078.466518] _raw_spin_lock+0x30/0x40 [16078.470174] ? igrab+0x1d/0x50 [16078.473225] igrab+0x1d/0x50 [16078.476101] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16078.481252] delayed_work+0x1d8/0x280 [ceph] [16078.485532] ? process_scheduled_works+0x22a/0x510 [16078.490323] process_scheduled_works+0x2a4/0x510 [16078.494945] worker_thread+0x188/0x340 [16078.498697] ? rescuer_thread+0x310/0x310 [16078.502700] kthread+0xf5/0x130 [16078.505838] ? kthread_complete_and_exit+0x20/0x20 [16078.510622] ret_from_fork+0x31/0x40 [16078.514201] ? kthread_complete_and_exit+0x20/0x20 [16078.518985] ret_from_fork_asm+0x11/0x20 [16078.522907] [16105.736744] watchdog: BUG: soft lockup - CPU#0 stuck for 3885s! [dbench:78683] [16105.743965] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16105.748743] watchdog: BUG: soft lockup - CPU#7 stuck for 12882s! [kworker/7:4:78808] [16105.799427] irq event stamp: 21526104 [16105.799428] hardirqs last enabled at (21526103): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16105.807162] Modules linked in: [16105.810820] hardirqs last disabled at (21526104): [] sysvec_apic_timer_interrupt+0xe/0x90 [16105.820976] ceph [16105.824028] softirqs last enabled at (21526076): [] __do_softirq+0x2be/0x371 [16105.833752] libceph [16105.835676] softirqs last disabled at (21526071): [] irq_exit_rcu+0xb0/0xd0 [16105.844360] dns_resolver [16105.846546] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16105.855055] fscache netfs [16105.857673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16105.866616] veth nft_chain_nat [16105.869321] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16105.876791] xt_MASQUERADE nf_nat [16105.879929] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16105.885419] nf_conntrack [16105.888725] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16105.907464] nf_defrag_ipv6 nf_defrag_ipv4 [16105.910081] [16105.910082] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16105.915299] nft_compat nf_tables [16105.919389] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16105.920881] nfnetlink bridge [16105.928003] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16105.931313] stp llc [16105.938438] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16105.941418] xfs libcrc32c [16105.948544] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16105.950728] sunrpc intel_rapl_msr [16105.957851] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16105.960557] intel_rapl_common [16105.967681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16105.971078] sb_edac x86_pkg_temp_thermal [16105.979157] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16105.982205] coretemp kvm_intel [16105.987944] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16105.991947] kvm irqbypass [16105.999072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16106.002209] i2c_i801 iTCO_wdt [16106.009333] Call Trace: [16106.009335] [16106.012038] iTCO_vendor_support i2c_smbus [16106.019162] ? watchdog_timer_fn+0x1b4/0x220 [16106.022212] crct10dif_pclmul joydev [16106.024656] ? __hrtimer_run_queues+0x1d3/0x360 [16106.026667] crc32_pclmul ipmi_ssif [16106.030760] ? hrtimer_interrupt+0x104/0x240 [16106.035023] ghash_clmulni_intel mei_me [16106.038592] ? __local_bh_enable+0x50/0x60 [16106.043116] mei [16106.046621] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16106.050882] lpc_ich sha512_ssse3 [16106.054714] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16106.058803] acpi_ipmi mfd_core [16106.060642] [16106.065858] wmi ipmi_si [16106.069169] [16106.074140] ipmi_devintf ipmi_msghandler [16106.077281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16106.079379] ioatdma acpi_pad [16106.081914] ? queued_spin_lock_slowpath+0x16/0x2e0 [16106.084006] acpi_power_meter ixgbe [16106.088014] do_raw_spin_lock+0xb5/0xc0 [16106.093314] igb [16106.096282] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16106.101149] mdio crc32c_intel [16106.104636] try_get_cap_refs+0x159/0x880 [ceph] [16106.108466] nvme ptp [16106.110307] __ceph_get_caps+0x4a1/0x760 [ceph] [16106.115087] nvme_core i2c_algo_bit [16106.118147] ? __file_remove_privs+0x89/0x110 [16106.122757] pps_core dca [16106.125026] ? wake_bit_function+0x70/0x70 [16106.129552] fuse [16106.133038] ? ceph_write_iter+0x260/0x770 [ceph] [16106.137384] irq event stamp: 51924478 [16106.140003] ceph_write_iter+0x260/0x770 [ceph] [16106.144095] hardirqs last enabled at (51924477): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16106.146023] ? lock_is_held_type+0xaa/0x120 [16106.150716] hardirqs last disabled at (51924478): [] sysvec_apic_timer_interrupt+0xe/0x90 [16106.154376] vfs_write+0x324/0x4f0 [16106.158897] softirqs last enabled at (51924476): [] __do_softirq+0x2be/0x371 [16106.169059] ksys_pwrite64+0x65/0xa0 [16106.173233] softirqs last disabled at (51924471): [] irq_exit_rcu+0xb0/0xd0 [16106.182957] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16106.186352] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16106.195038] do_syscall_64+0x38/0x80 [16106.198608] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16106.207121] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16106.212268] Workqueue: events delayed_work [ceph] [16106.221646] RIP: 0033:0x7f48f0eeb93b [16106.225214] [16106.225215] RIP: 0010:lock_release+0x1d2/0x260 [16106.232688] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16106.237731] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [16106.242432] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16106.245999] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [16106.247490] ORIG_RAX: 0000000000000012 [16106.251927] [16106.270663] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16106.289403] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [16106.289421] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [16106.294638] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16106.299855] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [16106.303684] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16106.305176] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [16106.312298] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16106.319427] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16106.326548] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16106.333672] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16106.340803] [16106.347920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16106.392438] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16106.399567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16106.406689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16106.413815] Call Trace: [16106.416259] [16106.418269] ? watchdog_timer_fn+0x1b4/0x220 [16106.422535] ? __hrtimer_run_queues+0x1d3/0x360 [16106.427061] ? hrtimer_interrupt+0x104/0x240 [16106.431320] ? __local_bh_enable+0x50/0x60 [16106.435415] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16106.440647] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16106.445605] [16106.447701] [16106.449801] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16106.455103] ? igrab+0x43/0x50 [16106.458157] ? igrab+0x43/0x50 [16106.461215] ? lock_release+0x1d2/0x260 [16106.465045] ? lock_release+0xc5/0x260 [16106.468791] _raw_spin_unlock+0x1b/0x40 [16106.472630] igrab+0x43/0x50 [16106.475506] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16106.480656] delayed_work+0x1d8/0x280 [ceph] [16106.483122] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [16106.484936] ? process_scheduled_works+0x22a/0x510 [16106.491021] rcu: 0-....: (4077275 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=2017129 [16106.495803] process_scheduled_works+0x2a4/0x510 [16106.506134] rcu: 7-....: (4051926 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/4050725 fqs=2017130 [16106.510751] worker_thread+0x188/0x340 [16106.521161] rcu: (detected by 4, t=4169819 jiffies, g=1309609, q=195020 ncpus=8) [16106.524907] ? rescuer_thread+0x310/0x310 [16106.532378] Sending NMI from CPU 4 to CPUs 0: [16106.536381] kthread+0xf5/0x130 [16106.540734] NMI backtrace for cpu 0 [16106.540735] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16106.540737] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16106.540738] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16106.540741] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16106.540743] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16106.540745] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16106.540746] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16106.540747] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16106.540749] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16106.540750] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16106.540751] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16106.540753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16106.540754] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16106.540755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16106.540756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16106.540758] Call Trace: [16106.540759] [16106.540760] ? nmi_cpu_backtrace+0xad/0x120 [16106.540764] ? nmi_cpu_backtrace_handler+0x11/0x20 [16106.540766] ? nmi_handle+0xd4/0x260 [16106.540772] ? default_do_nmi+0x49/0xf0 [16106.540775] ? exc_nmi+0x11e/0x150 [16106.540777] ? end_repeat_nmi+0x16/0x67 [16106.540784] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16106.540787] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16106.540791] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16106.540794] [16106.540795] [16106.540796] do_raw_spin_lock+0xb5/0xc0 [16106.540799] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16106.540818] try_get_cap_refs+0x159/0x880 [ceph] [16106.540839] __ceph_get_caps+0x4a1/0x760 [ceph] [16106.540859] ? __file_remove_privs+0x89/0x110 [16106.540861] ? wake_bit_function+0x70/0x70 [16106.540865] ? ceph_write_iter+0x260/0x770 [ceph] [16106.540881] ceph_write_iter+0x260/0x770 [ceph] [16106.540898] ? lock_is_held_type+0xaa/0x120 [16106.540903] vfs_write+0x324/0x4f0 [16106.540909] ksys_pwrite64+0x65/0xa0 [16106.540912] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16106.540916] do_syscall_64+0x38/0x80 [16106.540918] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16106.540921] RIP: 0033:0x7f48f0eeb93b [16106.540922] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16106.540924] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [16106.540926] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16106.540927] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16106.540928] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16106.540929] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16106.540930] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16106.540936] [16106.541731] Sending NMI from CPU 4 to CPUs 7: [16106.543869] ? kthread_complete_and_exit+0x20/0x20 [16106.547355] NMI backtrace for cpu 7 [16106.547356] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16106.547357] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16106.547358] Workqueue: events delayed_work [ceph] [16106.547371] RIP: 0010:console_unlock+0x16/0x120 [16106.547374] Code: e8 1f 74 a1 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 41 54 49 c7 c4 e0 bc 55 82 55 31 ed 53 <48> 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 8b 05 [16106.547375] RSP: 0018:ffffc90000264d10 EFLAGS: 00000046 [16106.547376] RAX: ffffffff81182541 RBX: 0000000000000000 RCX: 00000000de2c1856 [16106.547377] RDX: c31e29845e0f9ad2 RSI: 0000000000000000 RDI: 00000000000000c2 [16106.547377] RBP: 0000000000000000 R08: 00000000000000a0 R09: 0000000000000004 [16106.547378] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8255bce0 [16106.547379] R13: 0000000000000000 R14: ffffffff82123e9c R15: ffffc90000264d80 [16106.547379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16106.547380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16106.547381] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16106.547382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16106.547383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16106.547383] Call Trace: [16106.547384] [16106.547385] ? nmi_cpu_backtrace+0xad/0x120 [16106.547387] ? nmi_cpu_backtrace_handler+0x11/0x20 [16106.547388] ? nmi_handle+0xd4/0x260 [16106.547392] ? default_do_nmi+0x49/0xf0 [16106.547393] ? exc_nmi+0x11e/0x150 [16106.547394] ? end_repeat_nmi+0x16/0x67 [16106.547398] ? vprintk_emit+0x221/0x340 [16106.547400] ? console_unlock+0x16/0x120 [16106.547402] ? console_unlock+0x16/0x120 [16106.547403] ? console_unlock+0x16/0x120 [16106.547405] [16106.547405] [16106.547406] vprintk_emit+0x246/0x340 [16106.547408] ? kthread_complete_and_exit+0x20/0x20 [16106.547410] ? kthread_complete_and_exit+0x20/0x20 [16106.547412] _printk+0x5c/0x80 [16106.547415] ? kthread_complete_and_exit+0x20/0x20 [16106.547417] show_trace_log_lvl+0x1ac/0x3f0 [16106.547422] ? kthread_complete_and_exit+0x20/0x20 [16106.547423] ? ret_from_fork+0x31/0x40 [16106.547426] ? touch_softlockup_watchdog+0x20/0x20 [16106.547428] watchdog_timer_fn+0x1b4/0x220 [16106.547430] __hrtimer_run_queues+0x1d3/0x360 [16106.547432] hrtimer_interrupt+0x104/0x240 [16106.547434] ? __local_bh_enable+0x50/0x60 [16106.547437] __sysvec_apic_timer_interrupt+0x61/0x160 [16106.547439] sysvec_apic_timer_interrupt+0x6d/0x90 [16106.547441] [16106.547441] [16106.547442] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16106.547444] RIP: 0010:lock_release+0x1d2/0x260 [16106.547446] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [16106.547447] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [16106.547448] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [16106.547448] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [16106.547449] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [16106.547450] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [16106.547450] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16106.547451] ? igrab+0x43/0x50 [16106.547454] ? igrab+0x43/0x50 [16106.547455] ? lock_release+0xc5/0x260 [16106.547457] _raw_spin_unlock+0x1b/0x40 [16106.547460] igrab+0x43/0x50 [16106.547461] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16106.547475] delayed_work+0x1d8/0x280 [ceph] [16106.547488] ? process_scheduled_works+0x22a/0x510 [16106.547491] process_scheduled_works+0x2a4/0x510 [16106.547495] worker_thread+0x188/0x340 [16106.547497] ? rescuer_thread+0x310/0x310 [16106.547499] kthread+0xf5/0x130 [16106.547501] ? kthread_complete_and_exit+0x20/0x20 [16106.547504] ret_from_fork+0x31/0x40 [16106.547506] ? kthread_complete_and_exit+0x20/0x20 [16106.547508] ret_from_fork_asm+0x11/0x20 [16106.547513] [16107.224070] ret_from_fork+0x31/0x40 [16107.227646] ? kthread_complete_and_exit+0x20/0x20 [16107.232442] ret_from_fork_asm+0x11/0x20 [16107.236363] [16133.737235] watchdog: BUG: soft lockup - CPU#0 stuck for 3911s! [dbench:78683] [16133.744451] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16133.749235] watchdog: BUG: soft lockup - CPU#7 stuck for 12908s! [kworker/7:4:78808] [16133.799910] irq event stamp: 21584188 [16133.799911] hardirqs last enabled at (21584187): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16133.807641] Modules linked in: [16133.811299] hardirqs last disabled at (21584188): [] sysvec_apic_timer_interrupt+0xe/0x90 [16133.821456] ceph [16133.824506] softirqs last enabled at (21584160): [] __do_softirq+0x2be/0x371 [16133.834230] libceph [16133.836154] softirqs last disabled at (21584155): [] irq_exit_rcu+0xb0/0xd0 [16133.844839] dns_resolver [16133.847022] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16133.855535] fscache netfs [16133.858150] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16133.867095] veth [16133.869801] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16133.877271] nft_chain_nat [16133.879194] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16133.884671] xt_MASQUERADE [16133.887376] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16133.906113] nf_nat [16133.908817] [16133.908818] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16133.914036] nf_conntrack [16133.916132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16133.917633] nf_defrag_ipv6 [16133.924757] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16133.927374] nf_defrag_ipv4 nft_compat [16133.934496] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16133.937288] nf_tables nfnetlink [16133.944413] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16133.948156] bridge stp [16133.955281] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16133.958505] llc xfs [16133.965635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16133.968081] libcrc32c sunrpc [16133.976158] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16133.978342] intel_rapl_msr [16133.984080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16133.987045] intel_rapl_common [16133.994169] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16133.996958] sb_edac x86_pkg_temp_thermal [16134.004082] Call Trace: [16134.004084] [16134.007134] coretemp kvm_intel [16134.014260] ? watchdog_timer_fn+0x1b4/0x220 [16134.018261] kvm irqbypass [16134.020706] ? __hrtimer_run_queues+0x1d3/0x360 [16134.022718] i2c_i801 iTCO_wdt [16134.025857] ? hrtimer_interrupt+0x104/0x240 [16134.030119] iTCO_vendor_support i2c_smbus [16134.032822] ? __local_bh_enable+0x50/0x60 [16134.037346] crct10dif_pclmul [16134.040400] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16134.044661] joydev crc32_pclmul [16134.048752] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16134.052841] ipmi_ssif ghash_clmulni_intel [16134.055808] [16134.061025] mei_me [16134.064248] [16134.064249] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16134.069205] mei lpc_ich [16134.073301] ? queued_spin_lock_slowpath+0x18/0x2e0 [16134.075393] sha512_ssse3 acpi_ipmi [16134.077492] do_raw_spin_lock+0xb5/0xc0 [16134.079589] mfd_core wmi [16134.084893] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16134.087440] ipmi_si ipmi_devintf [16134.092314] try_get_cap_refs+0x159/0x880 [ceph] [16134.095794] ipmi_msghandler ioatdma [16134.099640] __ceph_get_caps+0x4a1/0x760 [ceph] [16134.102253] acpi_pad acpi_power_meter [16134.107041] ? __file_remove_privs+0x89/0x110 [16134.110347] ixgbe igb [16134.114957] ? wake_bit_function+0x70/0x70 [16134.118529] mdio crc32c_intel [16134.123056] ? ceph_write_iter+0x260/0x770 [ceph] [16134.126798] nvme ptp [16134.131148] ceph_write_iter+0x260/0x770 [ceph] [16134.133504] nvme_core i2c_algo_bit [16134.137598] ? lock_is_held_type+0xaa/0x120 [16134.140645] pps_core dca [16134.145346] vfs_write+0x324/0x4f0 [16134.147631] fuse [16134.152161] ksys_pwrite64+0x65/0xa0 [16134.155640] irq event stamp: 52140894 [16134.159817] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16134.162435] hardirqs last enabled at (52140893): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16134.165833] do_syscall_64+0x38/0x80 [16134.167756] hardirqs last disabled at (52140894): [] sysvec_apic_timer_interrupt+0xe/0x90 [16134.171327] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16134.174985] softirqs last enabled at (52140892): [] __do_softirq+0x2be/0x371 [16134.180115] RIP: 0033:0x7f48f0eeb93b [16134.190273] softirqs last disabled at (52140887): [] irq_exit_rcu+0xb0/0xd0 [16134.193842] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16134.203567] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16134.208611] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16134.217314] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16134.220884] ORIG_RAX: 0000000000000012 [16134.229394] Workqueue: events delayed_work [ceph] [16134.248132] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16134.257509] [16134.262727] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16134.270197] RIP: 0010:lock_acquire+0xf0/0x2b0 [16134.274029] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16134.278726] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16134.285849] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16134.287340] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16134.294466] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16134.298816] [16134.298817] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16134.305945] [16134.324678] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16134.362076] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16134.369199] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16134.376324] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16134.383449] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16134.391526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16134.397262] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16134.404388] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16134.411511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16134.418641] Call Trace: [16134.421088] [16134.423098] ? watchdog_timer_fn+0x1b4/0x220 [16134.427363] ? __hrtimer_run_queues+0x1d3/0x360 [16134.431890] ? hrtimer_interrupt+0x104/0x240 [16134.436158] ? __local_bh_enable+0x50/0x60 [16134.440253] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16134.445478] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16134.450434] [16134.452532] [16134.454641] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16134.459954] ? lock_acquire+0xf0/0x2b0 [16134.463703] ? igrab+0x43/0x50 [16134.466755] ? lock_release+0xc5/0x260 [16134.470499] _raw_spin_lock+0x30/0x40 [16134.474156] ? igrab+0x1d/0x50 [16134.477205] igrab+0x1d/0x50 [16134.480084] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16134.485233] delayed_work+0x1d8/0x280 [ceph] [16134.489513] ? process_scheduled_works+0x22a/0x510 [16134.494296] process_scheduled_works+0x2a4/0x510 [16134.498927] worker_thread+0x188/0x340 [16134.502679] ? rescuer_thread+0x310/0x310 [16134.506682] kthread+0xf5/0x130 [16134.509818] ? kthread_complete_and_exit+0x20/0x20 [16134.514606] ret_from_fork+0x31/0x40 [16134.518183] ? kthread_complete_and_exit+0x20/0x20 [16134.522966] ret_from_fork_asm+0x11/0x20 [16134.526890] [16161.737728] watchdog: BUG: soft lockup - CPU#0 stuck for 3937s! [dbench:78683] [16161.744939] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16161.749728] watchdog: BUG: soft lockup - CPU#7 stuck for 12934s! [kworker/7:4:78808] [16161.800381] irq event stamp: 21642378 [16161.800382] hardirqs last enabled at (21642377): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16161.808127] Modules linked in: [16161.811785] hardirqs last disabled at (21642378): [] sysvec_apic_timer_interrupt+0xe/0x90 [16161.821942] ceph [16161.824993] softirqs last enabled at (21642350): [] __do_softirq+0x2be/0x371 [16161.834717] libceph dns_resolver [16161.836642] softirqs last disabled at (21642345): [] irq_exit_rcu+0xb0/0xd0 [16161.845325] fscache [16161.848637] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16161.857148] netfs veth [16161.859333] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16161.859334] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16161.868276] nft_chain_nat xt_MASQUERADE [16161.870719] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16161.878191] nf_nat nf_conntrack [16161.883668] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16161.887586] nf_defrag_ipv6 nf_defrag_ipv4 [16161.906323] [16161.906324] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16161.909547] nft_compat nf_tables [16161.914764] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16161.918854] nfnetlink bridge [16161.920345] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16161.927471] stp llc [16161.930780] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16161.937906] xfs libcrc32c [16161.940870] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16161.947992] sunrpc [16161.950177] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16161.957300] intel_rapl_msr intel_rapl_common [16161.960005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16161.967128] sb_edac x86_pkg_temp_thermal [16161.969226] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16161.977303] coretemp kvm_intel [16161.981654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16161.987392] kvm irqbypass [16161.991416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16161.998538] i2c_i801 iTCO_wdt [16162.001675] Call Trace: [16162.001677] [16162.008801] iTCO_vendor_support i2c_smbus [16162.011505] ? watchdog_timer_fn+0x1b4/0x220 [16162.018629] crct10dif_pclmul joydev [16162.021679] ? __hrtimer_run_queues+0x1d3/0x360 [16162.024126] crc32_pclmul ipmi_ssif [16162.026135] ? hrtimer_interrupt+0x104/0x240 [16162.030225] ghash_clmulni_intel mei_me [16162.034488] ? __local_bh_enable+0x50/0x60 [16162.038060] mei lpc_ich [16162.042586] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16162.046067] sha512_ssse3 acpi_ipmi [16162.050332] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16162.054162] mfd_core wmi [16162.058252] [16162.060784] ipmi_si ipmi_devintf [16162.066001] [16162.066003] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16162.069486] ipmi_msghandler ioatdma [16162.074447] ? queued_spin_lock_slowpath+0x16/0x2e0 [16162.077060] acpi_pad acpi_power_meter [16162.079159] do_raw_spin_lock+0xb5/0xc0 [16162.082468] ixgbe igb [16162.084567] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16162.089870] mdio crc32c_intel [16162.093442] try_get_cap_refs+0x159/0x880 [ceph] [16162.098312] nvme ptp nvme_core [16162.102059] __ceph_get_caps+0x4a1/0x760 [ceph] [16162.105886] i2c_algo_bit pps_core [16162.108246] ? __file_remove_privs+0x89/0x110 [16162.113026] dca fuse [16162.116079] ? wake_bit_function+0x70/0x70 [16162.120688] [16162.120689] irq event stamp: 52362664 [16162.123829] ? ceph_write_iter+0x260/0x770 [ceph] [16162.128350] hardirqs last enabled at (52362663): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16162.131748] ceph_write_iter+0x260/0x770 [ceph] [16162.136098] hardirqs last disabled at (52362664): [] sysvec_apic_timer_interrupt+0xe/0x90 [16162.138371] ? lock_is_held_type+0xaa/0x120 [16162.142461] softirqs last enabled at (52362662): [] __do_softirq+0x2be/0x371 [16162.143953] vfs_write+0x324/0x4f0 [16162.147608] softirqs last disabled at (52362657): [] irq_exit_rcu+0xb0/0xd0 [16162.152310] ksys_pwrite64+0x65/0xa0 [16162.162464] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16162.166988] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16162.176710] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16162.176712] Workqueue: events delayed_work [ceph] [16162.180889] do_syscall_64+0x38/0x80 [16162.189573] [16162.189573] RIP: 0010:ceph_check_delayed_caps+0x85/0x270 [ceph] [16162.192970] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16162.201480] Code: 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 <78> 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 [16162.205052] RIP: 0033:0x7f48f0eeb93b [16162.214428] RSP: 0018:ffffc90002117db8 EFLAGS: 00000216 [16162.219559] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16162.227031] [16162.227032] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 0000000100f20240 [16162.231728] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16162.235299] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [16162.236788] ORIG_RAX: 0000000000000012 [16162.242700] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [16162.247744] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16162.266481] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [16162.270053] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16162.275270] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16162.294008] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16162.295499] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16162.302623] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16162.307840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16162.314965] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16162.318794] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16162.325924] [16162.333042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16162.406043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16162.413167] Call Trace: [16162.415613] [16162.417624] ? watchdog_timer_fn+0x1b4/0x220 [16162.421888] ? __hrtimer_run_queues+0x1d3/0x360 [16162.426425] ? hrtimer_interrupt+0x104/0x240 [16162.430692] ? __local_bh_enable+0x50/0x60 [16162.434787] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16162.440010] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16162.444968] [16162.447063] [16162.449162] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16162.454470] ? ceph_check_delayed_caps+0x85/0x270 [ceph] [16162.459787] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [16162.465101] delayed_work+0x1d8/0x280 [ceph] [16162.469400] ? process_scheduled_works+0x22a/0x510 [16162.474192] process_scheduled_works+0x2a4/0x510 [16162.478805] worker_thread+0x188/0x340 [16162.482555] ? rescuer_thread+0x310/0x310 [16162.486558] kthread+0xf5/0x130 [16162.489696] ? kthread_complete_and_exit+0x20/0x20 [16162.494480] ret_from_fork+0x31/0x40 [16162.498050] ? kthread_complete_and_exit+0x20/0x20 [16162.502834] ret_from_fork_asm+0x11/0x20 [16162.506758] [16189.738222] watchdog: BUG: soft lockup - CPU#0 stuck for 3963s! [dbench:78683] [16189.745442] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16189.750221] watchdog: BUG: soft lockup - CPU#7 stuck for 12960s! [kworker/7:4:78808] [16189.800895] irq event stamp: 21700484 [16189.800897] hardirqs last enabled at (21700483): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16189.808624] Modules linked in: [16189.812282] hardirqs last disabled at (21700484): [] sysvec_apic_timer_interrupt+0xe/0x90 [16189.822438] ceph [16189.825490] softirqs last enabled at (21700456): [] __do_softirq+0x2be/0x371 [16189.835212] libceph [16189.837138] softirqs last disabled at (21700451): [] irq_exit_rcu+0xb0/0xd0 [16189.845820] dns_resolver [16189.848005] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16189.856517] fscache netfs [16189.859133] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16189.868078] veth [16189.870783] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16189.878252] nft_chain_nat [16189.880177] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16189.885653] xt_MASQUERADE [16189.888358] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16189.907095] nf_nat nf_conntrack [16189.909801] [16189.909802] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16189.915017] nf_defrag_ipv6 [16189.918241] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16189.919733] nf_defrag_ipv4 nft_compat [16189.926856] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16189.929648] nf_tables [16189.936771] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16189.940516] nfnetlink bridge [16189.947639] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16189.949997] stp llc [16189.957120] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16189.960085] xfs libcrc32c [16189.967208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16189.969394] sunrpc [16189.977471] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16189.980173] intel_rapl_msr [16189.985913] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16189.988011] intel_rapl_common [16189.995132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16189.997924] sb_edac x86_pkg_temp_thermal [16190.005047] Call Trace: [16190.005050] [16190.008100] coretemp kvm_intel [16190.015225] ? watchdog_timer_fn+0x1b4/0x220 [16190.019226] kvm irqbypass [16190.021672] ? __hrtimer_run_queues+0x1d3/0x360 [16190.023681] i2c_i801 iTCO_wdt [16190.026823] ? hrtimer_interrupt+0x104/0x240 [16190.031082] iTCO_vendor_support [16190.033786] ? __local_bh_enable+0x50/0x60 [16190.038311] i2c_smbus crct10dif_pclmul [16190.041364] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16190.045627] joydev crc32_pclmul [16190.048850] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16190.052941] ipmi_ssif ghash_clmulni_intel [16190.056772] [16190.061988] mei_me mei [16190.065214] [16190.065216] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16190.070170] lpc_ich sha512_ssse3 [16190.074266] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16190.076358] acpi_ipmi mfd_core [16190.078805] do_raw_spin_lock+0xb5/0xc0 [16190.080900] wmi ipmi_si [16190.086207] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16190.089514] ipmi_devintf ipmi_msghandler [16190.094390] try_get_cap_refs+0x159/0x880 [ceph] [16190.097524] ioatdma acpi_pad [16190.101359] __ceph_get_caps+0x4a1/0x760 [ceph] [16190.103884] acpi_power_meter ixgbe [16190.108673] ? __file_remove_privs+0x89/0x110 [16190.112674] igb [16190.117283] ? wake_bit_function+0x70/0x70 [16190.120247] mdio crc32c_intel [16190.124775] ? ceph_write_iter+0x260/0x770 [ceph] [16190.128256] nvme ptp [16190.132621] ceph_write_iter+0x260/0x770 [ceph] [16190.134460] nvme_core [16190.138555] ? lock_is_held_type+0xaa/0x120 [16190.141621] i2c_algo_bit [16190.146321] vfs_write+0x324/0x4f0 [16190.148587] pps_core dca [16190.153119] ksys_pwrite64+0x65/0xa0 [16190.155469] fuse [16190.159648] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16190.162265] irq event stamp: 52584764 [16190.165662] do_syscall_64+0x38/0x80 [16190.168279] hardirqs last enabled at (52584763): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16190.171850] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16190.173774] hardirqs last disabled at (52584764): [] sysvec_apic_timer_interrupt+0xe/0x90 [16190.178915] RIP: 0033:0x7f48f0eeb93b [16190.182571] softirqs last enabled at (52584762): [] __do_softirq+0x2be/0x371 [16190.186142] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16190.196299] softirqs last disabled at (52584757): [] irq_exit_rcu+0xb0/0xd0 [16190.201343] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16190.211067] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16190.214639] ORIG_RAX: 0000000000000012 [16190.223322] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16190.242060] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16190.250571] Workqueue: events delayed_work [ceph] [16190.255789] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16190.265167] [16190.268995] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16190.276468] RIP: 0010:lock_acquire+0x0/0x2b0 [16190.283591] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16190.288289] Code: 0f c1 05 63 da ea 7e 83 f8 01 75 0e 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd [16190.295413] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16190.296905] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [16190.304033] [16190.308291] [16190.308292] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [16190.357293] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [16190.364418] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [16190.371543] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [16190.378667] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16190.385790] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16190.393868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16190.399622] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16190.406747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16190.413870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16190.420996] Call Trace: [16190.423440] [16190.425450] ? watchdog_timer_fn+0x1b4/0x220 [16190.429716] ? __hrtimer_run_queues+0x1d3/0x360 [16190.434243] ? hrtimer_interrupt+0x104/0x240 [16190.438512] ? __local_bh_enable+0x50/0x60 [16190.442623] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16190.447846] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16190.452804] [16190.454912] [16190.457008] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16190.462314] ? lock_sync+0xa0/0xa0 [16190.465718] _raw_spin_lock+0x30/0x40 [16190.469375] ? igrab+0x1d/0x50 [16190.472425] igrab+0x1d/0x50 [16190.475305] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16190.480452] delayed_work+0x1d8/0x280 [ceph] [16190.484735] ? process_scheduled_works+0x22a/0x510 [16190.489523] process_scheduled_works+0x2a4/0x510 [16190.494141] worker_thread+0x188/0x340 [16190.497905] ? rescuer_thread+0x310/0x310 [16190.501910] kthread+0xf5/0x130 [16190.505058] ? kthread_complete_and_exit+0x20/0x20 [16190.509849] ret_from_fork+0x31/0x40 [16190.513428] ? kthread_complete_and_exit+0x20/0x20 [16190.518211] ret_from_fork_asm+0x11/0x20 [16190.522144] [16217.738714] watchdog: BUG: soft lockup - CPU#0 stuck for 3989s! [dbench:78683] [16217.745928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16217.750714] watchdog: BUG: soft lockup - CPU#7 stuck for 12986s! [kworker/7:4:78808] [16217.801386] irq event stamp: 21758716 [16217.801387] hardirqs last enabled at (21758715): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16217.809119] Modules linked in: [16217.812776] hardirqs last disabled at (21758716): [] sysvec_apic_timer_interrupt+0xe/0x90 [16217.822932] ceph libceph [16217.825984] softirqs last enabled at (21758690): [] __do_softirq+0x2be/0x371 [16217.835709] dns_resolver [16217.838326] softirqs last disabled at (21758685): [] irq_exit_rcu+0xb0/0xd0 [16217.847010] fscache netfs [16217.849628] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16217.858137] veth nft_chain_nat [16217.860842] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16217.869787] xt_MASQUERADE nf_nat [16217.872923] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16217.880407] nf_conntrack [16217.883714] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16217.889190] nf_defrag_ipv6 nf_defrag_ipv4 [16217.891809] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16217.910546] nft_compat nf_tables [16217.914637] [16217.914637] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16217.919853] nfnetlink bridge [16217.923166] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16217.924655] stp llc [16217.931780] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16217.934743] xfs libcrc32c [16217.941868] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16217.944051] sunrpc [16217.951176] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16217.953881] intel_rapl_msr [16217.961005] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16217.963111] intel_rapl_common [16217.970235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16217.973026] sb_edac x86_pkg_temp_thermal [16217.981110] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16217.984163] coretemp [16217.989901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16217.993904] kvm_intel kvm [16218.001028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16218.003297] irqbypass i2c_i801 [16218.010423] Call Trace: [16218.010424] [16218.013126] iTCO_wdt iTCO_vendor_support [16218.020251] ? watchdog_timer_fn+0x1b4/0x220 [16218.023393] i2c_smbus crct10dif_pclmul [16218.025842] ? __hrtimer_run_queues+0x1d3/0x360 [16218.027852] joydev crc32_pclmul [16218.031860] ? hrtimer_interrupt+0x104/0x240 [16218.036120] ipmi_ssif [16218.039951] ? __local_bh_enable+0x50/0x60 [16218.044476] ghash_clmulni_intel mei_me [16218.047702] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16218.051963] mei lpc_ich [16218.054321] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16218.058412] sha512_ssse3 [16218.062243] [16218.067459] acpi_ipmi [16218.069991] [16218.069992] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16218.074948] mfd_core wmi [16218.077570] ? queued_spin_lock_slowpath+0x16/0x2e0 [16218.079662] ipmi_si [16218.082020] do_raw_spin_lock+0xb5/0xc0 [16218.084116] ipmi_devintf ipmi_msghandler [16218.089423] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16218.092039] ioatdma acpi_pad [16218.096911] try_get_cap_refs+0x159/0x880 [ceph] [16218.099109] acpi_power_meter [16218.102947] __ceph_get_caps+0x4a1/0x760 [ceph] [16218.106945] ixgbe igb [16218.111732] ? __file_remove_privs+0x89/0x110 [16218.114695] mdio crc32c_intel [16218.119305] ? wake_bit_function+0x70/0x70 [16218.122267] nvme ptp [16218.126794] ? ceph_write_iter+0x260/0x770 [ceph] [16218.129148] nvme_core i2c_algo_bit [16218.133501] ceph_write_iter+0x260/0x770 [ceph] [16218.136550] pps_core dca [16218.140645] ? lock_is_held_type+0xaa/0x120 [16218.142912] fuse [16218.147612] vfs_write+0x324/0x4f0 [16218.151111] irq event stamp: 52807530 [16218.155641] ksys_pwrite64+0x65/0xa0 [16218.158253] hardirqs last enabled at (52807529): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16218.162430] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16218.164355] hardirqs last disabled at (52807530): [] sysvec_apic_timer_interrupt+0xe/0x90 [16218.167754] do_syscall_64+0x38/0x80 [16218.171409] softirqs last enabled at (52807528): [] __do_softirq+0x2be/0x371 [16218.174980] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16218.185138] softirqs last disabled at (52807523): [] irq_exit_rcu+0xb0/0xd0 [16218.190270] RIP: 0033:0x7f48f0eeb93b [16218.199992] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16218.203564] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16218.212248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16218.217290] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16218.225803] Workqueue: events delayed_work [ceph] [16218.229372] ORIG_RAX: 0000000000000012 [16218.238768] [16218.238769] RIP: 0010:lock_acquire+0x86/0x2b0 [16218.257506] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16218.264976] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [16218.270194] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16218.274890] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [16218.278722] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16218.280212] [16218.284563] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16218.291688] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [16218.310425] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16218.317550] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [16218.322771] [16218.329892] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [16218.369203] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [16218.376329] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16218.383451] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16218.391529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16218.397266] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16218.404398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16218.411523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16218.418650] Call Trace: [16218.421093] [16218.423120] ? watchdog_timer_fn+0x1b4/0x220 [16218.427386] ? __hrtimer_run_queues+0x1d3/0x360 [16218.431930] ? hrtimer_interrupt+0x104/0x240 [16218.436197] ? __local_bh_enable+0x50/0x60 [16218.440291] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16218.445508] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16218.450466] [16218.452562] [16218.454659] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16218.459967] ? lock_acquire+0x86/0x2b0 [16218.463717] ? igrab+0x43/0x50 [16218.466766] ? lock_release+0xc5/0x260 [16218.470513] _raw_spin_lock+0x30/0x40 [16218.474175] ? igrab+0x1d/0x50 [16218.477228] igrab+0x1d/0x50 [16218.480119] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16218.485264] delayed_work+0x1d8/0x280 [ceph] [16218.489545] ? process_scheduled_works+0x22a/0x510 [16218.494336] process_scheduled_works+0x2a4/0x510 [16218.498951] worker_thread+0x188/0x340 [16218.502699] ? rescuer_thread+0x310/0x310 [16218.506702] kthread+0xf5/0x130 [16218.509840] ? kthread_complete_and_exit+0x20/0x20 [16218.514626] ret_from_fork+0x31/0x40 [16218.518194] ? kthread_complete_and_exit+0x20/0x20 [16218.522979] ret_from_fork_asm+0x11/0x20 [16218.526903] [16245.739207] watchdog: BUG: soft lockup - CPU#0 stuck for 4016s! [dbench:78683] [16245.746424] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16245.751206] watchdog: BUG: soft lockup - CPU#7 stuck for 13013s! [kworker/7:4:78808] [16245.801881] irq event stamp: 21816964 [16245.801882] hardirqs last enabled at (21816963): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16245.809614] Modules linked in: [16245.813272] hardirqs last disabled at (21816964): [] sysvec_apic_timer_interrupt+0xe/0x90 [16245.823429] ceph [16245.826479] softirqs last enabled at (21816936): [] __do_softirq+0x2be/0x371 [16245.836204] libceph [16245.838127] softirqs last disabled at (21816931): [] irq_exit_rcu+0xb0/0xd0 [16245.846813] dns_resolver fscache [16245.848996] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16245.857508] netfs veth [16245.860818] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16245.869761] nft_chain_nat [16245.872206] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16245.879675] xt_MASQUERADE nf_nat [16245.882382] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16245.887857] nf_conntrack [16245.891169] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16245.909907] nf_defrag_ipv6 nf_defrag_ipv4 [16245.912524] [16245.912525] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16245.917740] nft_compat nf_tables [16245.921831] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16245.923323] nfnetlink bridge [16245.930448] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16245.933756] stp llc [16245.940898] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16245.943862] xfs libcrc32c sunrpc [16245.951003] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16245.953187] intel_rapl_msr [16245.960314] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16245.963622] intel_rapl_common [16245.970749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16245.973538] sb_edac x86_pkg_temp_thermal [16245.981614] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16245.984665] coretemp kvm_intel [16245.990404] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16245.994407] kvm irqbypass [16246.001533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16246.004669] i2c_i801 iTCO_wdt [16246.011794] Call Trace: [16246.011796] [16246.014498] iTCO_vendor_support i2c_smbus [16246.021624] ? watchdog_timer_fn+0x1b4/0x220 [16246.024673] crct10dif_pclmul joydev [16246.027118] ? __hrtimer_run_queues+0x1d3/0x360 [16246.029127] crc32_pclmul ipmi_ssif [16246.033220] ? hrtimer_interrupt+0x104/0x240 [16246.037482] ghash_clmulni_intel [16246.041053] ? __local_bh_enable+0x50/0x60 [16246.045576] mei_me [16246.049065] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16246.053324] mei lpc_ich [16246.056549] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16246.060641] sha512_ssse3 acpi_ipmi [16246.062736] [16246.067954] mfd_core wmi [16246.070486] [16246.070488] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16246.075444] ipmi_si ipmi_devintf [16246.078932] ? queued_spin_lock_slowpath+0x18/0x2e0 [16246.081023] ipmi_msghandler ioatdma [16246.083645] do_raw_spin_lock+0xb5/0xc0 [16246.085740] acpi_pad acpi_power_meter [16246.091044] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16246.094353] ixgbe igb [16246.099227] try_get_cap_refs+0x159/0x880 [ceph] [16246.102796] mdio crc32c_intel [16246.106631] __ceph_get_caps+0x4a1/0x760 [ceph] [16246.110371] nvme ptp [16246.115158] ? __file_remove_privs+0x89/0x110 [16246.117510] nvme_core [16246.122123] ? wake_bit_function+0x70/0x70 [16246.125172] i2c_algo_bit pps_core [16246.129699] ? ceph_write_iter+0x260/0x770 [ceph] [16246.131968] dca fuse [16246.136320] ceph_write_iter+0x260/0x770 [ceph] [16246.138676] [16246.138677] irq event stamp: 53029854 [16246.142772] ? lock_is_held_type+0xaa/0x120 [16246.146164] hardirqs last enabled at (53029853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16246.150865] vfs_write+0x324/0x4f0 [16246.153132] hardirqs last disabled at (53029854): [] sysvec_apic_timer_interrupt+0xe/0x90 [16246.157661] ksys_pwrite64+0x65/0xa0 [16246.159147] softirqs last enabled at (53029852): [] __do_softirq+0x2be/0x371 [16246.162807] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16246.166981] softirqs last disabled at (53029847): [] irq_exit_rcu+0xb0/0xd0 [16246.177141] do_syscall_64+0x38/0x80 [16246.180536] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16246.190262] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16246.193832] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16246.202516] RIP: 0033:0x7f48f0eeb93b [16246.207646] Workqueue: events delayed_work [ceph] [16246.216159] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16246.219727] [16246.219728] RIP: 0010:lock_acquire+0xf0/0x2b0 [16246.229105] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16246.234149] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16246.241621] ORIG_RAX: 0000000000000012 [16246.245190] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16246.249889] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16246.268627] [16246.268628] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16246.270116] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16246.274467] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16246.279685] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16246.298422] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16246.302254] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16246.307470] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16246.314610] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16246.316104] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16246.323233] [16246.330351] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16246.390490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16246.396229] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16246.403352] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16246.410475] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16246.417611] Call Trace: [16246.420054] [16246.422065] ? watchdog_timer_fn+0x1b4/0x220 [16246.426330] ? __hrtimer_run_queues+0x1d3/0x360 [16246.430857] ? hrtimer_interrupt+0x104/0x240 [16246.435124] ? __local_bh_enable+0x50/0x60 [16246.439218] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16246.444442] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16246.449401] [16246.451497] [16246.453612] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16246.458920] ? lock_acquire+0xf0/0x2b0 [16246.462671] ? igrab+0x43/0x50 [16246.465720] ? lock_release+0xc5/0x260 [16246.469464] _raw_spin_lock+0x30/0x40 [16246.473121] ? igrab+0x1d/0x50 [16246.476171] igrab+0x1d/0x50 [16246.479051] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16246.484200] delayed_work+0x1d8/0x280 [ceph] [16246.488481] ? process_scheduled_works+0x22a/0x510 [16246.493271] process_scheduled_works+0x2a4/0x510 [16246.497897] worker_thread+0x188/0x340 [16246.501643] ? rescuer_thread+0x310/0x310 [16246.505646] kthread+0xf5/0x130 [16246.508787] ? kthread_complete_and_exit+0x20/0x20 [16246.513578] ret_from_fork+0x31/0x40 [16246.517148] ? kthread_complete_and_exit+0x20/0x20 [16246.521933] ret_from_fork_asm+0x11/0x20 [16246.525865] [16273.739699] watchdog: BUG: soft lockup - CPU#0 stuck for 4042s! [dbench:78683] [16273.746911] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16273.751699] watchdog: BUG: soft lockup - CPU#7 stuck for 13039s! [kworker/7:4:78808] [16273.802354] irq event stamp: 21875152 [16273.802355] hardirqs last enabled at (21875151): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16273.810102] Modules linked in: [16273.813757] hardirqs last disabled at (21875152): [] sysvec_apic_timer_interrupt+0xe/0x90 [16273.823915] ceph [16273.826967] softirqs last enabled at (21875124): [] __do_softirq+0x2be/0x371 [16273.836690] libceph [16273.838615] softirqs last disabled at (21875119): [] irq_exit_rcu+0xb0/0xd0 [16273.847299] dns_resolver [16273.849482] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16273.857992] fscache netfs [16273.860610] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16273.869555] veth nft_chain_nat [16273.872260] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16273.879729] xt_MASQUERADE nf_nat [16273.882867] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16273.888345] nf_conntrack nf_defrag_ipv6 [16273.891655] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16273.910393] nf_defrag_ipv4 [16273.914311] [16273.919529] nft_compat nf_tables [16273.922318] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16273.923810] nfnetlink bridge [16273.927119] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16273.934245] stp llc [16273.937209] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16273.944332] xfs libcrc32c [16273.946517] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16273.953641] sunrpc intel_rapl_msr [16273.956344] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16273.963470] intel_rapl_common sb_edac [16273.966867] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16273.973990] x86_pkg_temp_thermal coretemp [16273.977735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16273.985812] kvm_intel kvm [16273.989904] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16273.995639] irqbypass i2c_i801 [16273.998345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16274.005469] iTCO_wdt iTCO_vendor_support [16274.008605] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16274.015730] i2c_smbus crct10dif_pclmul [16274.019735] Call Trace: [16274.019737] [16274.026859] joydev crc32_pclmul [16274.030691] ? watchdog_timer_fn+0x1b4/0x220 [16274.033132] ipmi_ssif ghash_clmulni_intel [16274.035144] ? __hrtimer_run_queues+0x1d3/0x360 [16274.038377] mei_me mei [16274.042643] ? hrtimer_interrupt+0x104/0x240 [16274.046732] lpc_ich sha512_ssse3 [16274.051255] ? __local_bh_enable+0x50/0x60 [16274.053700] acpi_ipmi mfd_core [16274.057966] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16274.061275] wmi ipmi_si [16274.065381] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16274.068520] ipmi_devintf ipmi_msghandler [16274.073739] [16274.076268] ioatdma acpi_pad [16274.081224] [16274.085230] acpi_power_meter [16274.087328] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16274.090291] ixgbe igb [16274.092392] ? queued_spin_lock_slowpath+0x16/0x2e0 [16274.095353] mdio crc32c_intel [16274.100658] do_raw_spin_lock+0xb5/0xc0 [16274.103013] nvme ptp [16274.107886] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16274.110935] nvme_core [16274.114767] try_get_cap_refs+0x159/0x880 [ceph] [16274.117037] i2c_algo_bit pps_core [16274.121825] __ceph_get_caps+0x4a1/0x760 [ceph] [16274.124178] dca fuse [16274.128794] ? __file_remove_privs+0x89/0x110 [16274.132187] [16274.132187] irq event stamp: 53252582 [16274.136711] ? wake_bit_function+0x70/0x70 [16274.138981] hardirqs last enabled at (53252581): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16274.143335] ? ceph_write_iter+0x260/0x770 [ceph] [16274.144821] hardirqs last disabled at (53252582): [] sysvec_apic_timer_interrupt+0xe/0x90 [16274.148479] ceph_write_iter+0x260/0x770 [ceph] [16274.152569] softirqs last enabled at (53252580): [] __do_softirq+0x2be/0x371 [16274.162731] ? lock_is_held_type+0xaa/0x120 [16274.167425] softirqs last disabled at (53252575): [] irq_exit_rcu+0xb0/0xd0 [16274.177152] vfs_write+0x324/0x4f0 [16274.181675] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16274.190378] ksys_pwrite64+0x65/0xa0 [16274.194553] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16274.203064] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16274.206461] Workqueue: events delayed_work [ceph] [16274.215839] do_syscall_64+0x38/0x80 [16274.219410] [16274.219411] RIP: 0010:lock_acquire+0xf0/0x2b0 [16274.226880] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16274.232011] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16274.236707] RIP: 0033:0x7f48f0eeb93b [16274.240279] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16274.241771] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16274.246121] [16274.246122] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16274.251165] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16274.269903] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16274.273474] ORIG_RAX: 0000000000000012 [16274.278689] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16274.297429] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16274.298919] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16274.306043] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16274.311260] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16274.318391] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16274.322223] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16274.329347] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16274.336471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16274.343596] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16274.350721] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16274.357849] [16274.364970] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16274.409466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16274.416589] Call Trace: [16274.419033] [16274.421044] ? watchdog_timer_fn+0x1b4/0x220 [16274.425309] ? __hrtimer_run_queues+0x1d3/0x360 [16274.429836] ? hrtimer_interrupt+0x104/0x240 [16274.434105] ? __local_bh_enable+0x50/0x60 [16274.438197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16274.443422] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16274.448393] [16274.450491] [16274.452591] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16274.457898] ? lock_acquire+0xf0/0x2b0 [16274.461648] ? igrab+0x43/0x50 [16274.464699] ? lock_release+0xc5/0x260 [16274.468444] _raw_spin_lock+0x30/0x40 [16274.472105] ? igrab+0x1d/0x50 [16274.475159] igrab+0x1d/0x50 [16274.478037] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16274.483186] delayed_work+0x1d8/0x280 [ceph] [16274.487469] ? process_scheduled_works+0x22a/0x510 [16274.492259] process_scheduled_works+0x2a4/0x510 [16274.496874] worker_thread+0x188/0x340 [16274.500623] ? rescuer_thread+0x310/0x310 [16274.504627] kthread+0xf5/0x130 [16274.507764] ? kthread_complete_and_exit+0x20/0x20 [16274.512548] ret_from_fork+0x31/0x40 [16274.516117] ? kthread_complete_and_exit+0x20/0x20 [16274.520901] ret_from_fork_asm+0x11/0x20 [16274.524827] [16301.554556] rcu: INFO: rcu_preempt self-detected stall on CPU [16301.560316] rcu: 7-....: (4241603 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/4240397 fqs=2113032 [16301.570735] rcu: (t=4364867 jiffies g=1309609 q=199626 ncpus=8) [16301.576732] Sending NMI from CPU 7 to CPUs 0: [16301.581084] NMI backtrace for cpu 0 [16301.581086] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16301.581088] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16301.581089] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16301.581095] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16301.581096] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16301.581098] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16301.581099] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16301.581100] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16301.581101] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16301.581101] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16301.581102] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16301.581103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16301.581104] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16301.581105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16301.581105] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16301.581106] Call Trace: [16301.581107] [16301.581109] ? nmi_cpu_backtrace+0xad/0x120 [16301.581112] ? nmi_cpu_backtrace_handler+0x11/0x20 [16301.581114] ? nmi_handle+0xd4/0x260 [16301.581119] ? default_do_nmi+0x49/0xf0 [16301.581120] ? exc_nmi+0x11e/0x150 [16301.581122] ? end_repeat_nmi+0x16/0x67 [16301.581126] ? queued_spin_lock_slowpath+0x16/0x2e0 [16301.581129] ? queued_spin_lock_slowpath+0x16/0x2e0 [16301.581131] ? queued_spin_lock_slowpath+0x16/0x2e0 [16301.581134] [16301.581134] [16301.581135] do_raw_spin_lock+0xb5/0xc0 [16301.581139] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16301.581159] try_get_cap_refs+0x159/0x880 [ceph] [16301.581173] __ceph_get_caps+0x4a1/0x760 [ceph] [16301.581185] ? __file_remove_privs+0x89/0x110 [16301.581188] ? wake_bit_function+0x70/0x70 [16301.581191] ? ceph_write_iter+0x260/0x770 [ceph] [16301.581201] ceph_write_iter+0x260/0x770 [ceph] [16301.581212] ? lock_is_held_type+0xaa/0x120 [16301.581214] vfs_write+0x324/0x4f0 [16301.581219] ksys_pwrite64+0x65/0xa0 [16301.581221] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16301.581224] do_syscall_64+0x38/0x80 [16301.581226] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16301.581228] RIP: 0033:0x7f48f0eeb93b [16301.581230] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16301.581231] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [16301.581232] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16301.581233] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16301.581234] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16301.581234] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16301.581235] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16301.581238] [16301.582082] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16301.885965] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16301.893435] Workqueue: events delayed_work [ceph] [16301.898159] RIP: 0010:lock_acquire+0xf0/0x2b0 [16301.902518] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16301.921254] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16301.926472] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16301.933596] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16301.940720] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16301.947844] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16301.954986] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16301.962110] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16301.970188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16301.975925] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16301.983050] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16301.990173] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16301.997298] Call Trace: [16301.999742] [16302.001753] ? rcu_dump_cpu_stacks+0xd1/0x180 [16302.006112] ? rcu_sched_clock_irq+0x633/0x1390 [16302.010637] ? lock_release+0xc5/0x260 [16302.014380] ? lock_is_held_type+0xaa/0x120 [16302.018586] ? lock_is_held_type+0xaa/0x120 [16302.022773] ? tick_sched_do_timer+0xa0/0xa0 [16302.027044] ? rcu_read_lock_sched_held+0x56/0x90 [16302.031752] ? tick_sched_do_timer+0xa0/0xa0 [16302.036021] ? update_process_times+0x5f/0x90 [16302.040382] ? tick_sched_handle.isra.16+0x1e/0x50 [16302.045174] ? tick_sched_timer+0x6f/0x80 [16302.049180] ? __hrtimer_run_queues+0x1d3/0x360 [16302.053720] ? hrtimer_interrupt+0x104/0x240 [16302.057982] ? __local_bh_enable+0x50/0x60 [16302.062089] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16302.067318] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16302.072275] [16302.074372] [16302.076470] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16302.081793] ? lock_acquire+0xf0/0x2b0 [16302.085547] ? igrab+0x43/0x50 [16302.088605] ? lock_release+0xc5/0x260 [16302.092359] _raw_spin_lock+0x30/0x40 [16302.096023] ? igrab+0x1d/0x50 [16302.099080] igrab+0x1d/0x50 [16302.101960] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16302.107113] delayed_work+0x1d8/0x280 [ceph] [16302.111401] ? process_scheduled_works+0x22a/0x510 [16302.116189] process_scheduled_works+0x2a4/0x510 [16302.120812] worker_thread+0x188/0x340 [16302.124564] ? rescuer_thread+0x310/0x310 [16302.128591] kthread+0xf5/0x130 [16302.131739] ? kthread_complete_and_exit+0x20/0x20 [16302.136533] ret_from_fork+0x31/0x40 [16302.140110] ? kthread_complete_and_exit+0x20/0x20 [16302.144896] ret_from_fork_asm+0x11/0x20 [16302.148835] [16325.740614] watchdog: BUG: soft lockup - CPU#0 stuck for 4090s! [dbench:78683] [16325.747832] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16325.752614] watchdog: BUG: soft lockup - CPU#7 stuck for 13087s! [kworker/7:4:78808] [16325.803298] irq event stamp: 21984192 [16325.811030] Modules linked in: ceph libceph [16325.814688] hardirqs last enabled at (21984191): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16325.818864] dns_resolver fscache netfs [16325.829022] hardirqs last disabled at (21984192): [] sysvec_apic_timer_interrupt+0xe/0x90 [16325.832853] veth nft_chain_nat xt_MASQUERADE [16325.842578] softirqs last enabled at (21984190): [] __do_softirq+0x2be/0x371 [16325.842580] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [16325.855612] softirqs last disabled at (21984185): [] irq_exit_rcu+0xb0/0xd0 [16325.862390] nf_tables nfnetlink bridge [16325.870901] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16325.874731] stp llc [16325.883675] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16325.885859] xfs libcrc32c sunrpc [16325.893332] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16325.893333] intel_rapl_msr intel_rapl_common [16325.902118] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16325.902119] sb_edac [16325.906469] x86_pkg_temp_thermal [16325.925208] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16325.927392] coretemp [16325.930701] [16325.930702] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16325.935919] kvm_intel [16325.938191] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16325.939681] kvm irqbypass [16325.946805] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16325.949162] i2c_i801 iTCO_wdt [16325.956300] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16325.959012] iTCO_vendor_support [16325.966140] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16325.969191] i2c_smbus crct10dif_pclmul [16325.976314] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16325.979540] joydev [16325.986664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16325.990494] crc32_pclmul ipmi_ssif [16325.998571] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16326.000668] ghash_clmulni_intel mei_me [16326.006406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16326.009891] mei [16326.017016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16326.020846] lpc_ich sha512_ssse3 [16326.027971] Call Trace: [16326.027972] [16326.029806] acpi_ipmi mfd_core [16326.036931] ? watchdog_timer_fn+0x1b4/0x220 [16326.040242] wmi ipmi_si [16326.042688] ? __hrtimer_run_queues+0x1d3/0x360 [16326.044696] ipmi_devintf ipmi_msghandler [16326.047838] ? hrtimer_interrupt+0x104/0x240 [16326.052098] ioatdma [16326.054628] ? __local_bh_enable+0x50/0x60 [16326.059152] acpi_pad acpi_power_meter [16326.063161] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16326.067421] ixgbe igb [16326.069605] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16326.073695] mdio crc32c_intel [16326.077440] [16326.082656] nvme [16326.085015] [16326.089971] ptp nvme_core [16326.093022] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16326.095121] i2c_algo_bit pps_core [16326.097048] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16326.099143] dca fuse [16326.101848] do_raw_spin_lock+0xb5/0xc0 [16326.107151] [16326.107151] irq event stamp: 53665448 [16326.110548] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16326.115417] hardirqs last enabled at (53665447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16326.117690] try_get_cap_refs+0x159/0x880 [ceph] [16326.121519] hardirqs last disabled at (53665448): [] sysvec_apic_timer_interrupt+0xe/0x90 [16326.123019] __ceph_get_caps+0x4a1/0x760 [ceph] [16326.126666] softirqs last enabled at (53665446): [] __do_softirq+0x2be/0x371 [16326.131456] ? __file_remove_privs+0x89/0x110 [16326.141608] softirqs last disabled at (53665441): [] irq_exit_rcu+0xb0/0xd0 [16326.146220] ? wake_bit_function+0x70/0x70 [16326.155943] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16326.160471] ? ceph_write_iter+0x260/0x770 [ceph] [16326.169152] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16326.173505] ceph_write_iter+0x260/0x770 [ceph] [16326.182015] Workqueue: events delayed_work [ceph] [16326.186109] ? lock_is_held_type+0xaa/0x120 [16326.195481] [16326.200181] vfs_write+0x324/0x4f0 [16326.207652] RIP: 0010:lock_acquire+0xf0/0x2b0 [16326.212178] ksys_pwrite64+0x65/0xa0 [16326.216872] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16326.221050] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16326.222541] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16326.225939] do_syscall_64+0x38/0x80 [16326.230302] [16326.230303] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16326.233876] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16326.252615] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16326.257745] RIP: 0033:0x7f48f0eeb93b [16326.262963] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16326.266534] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16326.268024] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16326.275149] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16326.280191] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16326.287316] ORIG_RAX: 0000000000000012 [16326.290886] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16326.298020] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16326.316757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16326.323881] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16326.329099] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16326.336223] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16326.340053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16326.348130] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16326.355254] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16326.360993] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16326.368133] Call Trace: [16326.375263] [16326.382383] [16326.417519] ? watchdog_timer_fn+0x1b4/0x220 [16326.421783] ? __hrtimer_run_queues+0x1d3/0x360 [16326.426310] ? hrtimer_interrupt+0x104/0x240 [16326.430579] ? __local_bh_enable+0x50/0x60 [16326.434673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16326.439897] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16326.444852] [16326.446951] [16326.449048] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16326.454356] ? lock_acquire+0xf0/0x2b0 [16326.458106] ? igrab+0x43/0x50 [16326.461157] ? lock_release+0xc5/0x260 [16326.464902] _raw_spin_lock+0x30/0x40 [16326.468566] ? igrab+0x1d/0x50 [16326.471616] igrab+0x1d/0x50 [16326.474496] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16326.479645] delayed_work+0x1d8/0x280 [ceph] [16326.483926] ? process_scheduled_works+0x22a/0x510 [16326.488716] process_scheduled_works+0x2a4/0x510 [16326.493331] worker_thread+0x188/0x340 [16326.497080] ? rescuer_thread+0x310/0x310 [16326.501086] kthread+0xf5/0x130 [16326.504231] ? kthread_complete_and_exit+0x20/0x20 [16326.509023] ret_from_fork+0x31/0x40 [16326.512592] ? kthread_complete_and_exit+0x20/0x20 [16326.517378] ret_from_fork_asm+0x11/0x20 [16326.521305] [16353.741107] watchdog: BUG: soft lockup - CPU#0 stuck for 4116s! [dbench:78683] [16353.748328] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16353.753107] watchdog: BUG: soft lockup - CPU#7 stuck for 13113s! [kworker/7:4:78808] [16353.803792] irq event stamp: 22042298 [16353.803793] hardirqs last enabled at (22042297): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16353.811526] Modules linked in: [16353.815183] hardirqs last disabled at (22042298): [] sysvec_apic_timer_interrupt+0xe/0x90 [16353.825342] ceph [16353.828392] softirqs last enabled at (22042282): [] __do_softirq+0x2be/0x371 [16353.838117] libceph [16353.840040] softirqs last disabled at (22042277): [] irq_exit_rcu+0xb0/0xd0 [16353.848725] dns_resolver fscache [16353.850910] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16353.859420] netfs veth [16353.862729] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16353.871673] nft_chain_nat [16353.874119] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16353.881590] xt_MASQUERADE nf_nat [16353.884293] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16353.889769] nf_conntrack nf_defrag_ipv6 [16353.893097] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16353.911835] nf_defrag_ipv4 [16353.915754] [16353.920970] nft_compat [16353.923762] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16353.925268] nf_tables [16353.927713] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16353.934838] nfnetlink [16353.937194] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16353.944319] bridge stp [16353.946676] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16353.953801] llc [16353.956245] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16353.963367] xfs libcrc32c [16353.965205] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16353.972330] sunrpc [16353.975034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16353.983111] intel_rapl_msr [16353.985209] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16353.990949] intel_rapl_common [16353.993738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16354.000862] sb_edac x86_pkg_temp_thermal [16354.003913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16354.011035] coretemp kvm_intel [16354.015041] Call Trace: [16354.015044] [16354.022164] kvm irqbypass [16354.025306] ? watchdog_timer_fn+0x1b4/0x220 [16354.027746] i2c_i801 iTCO_wdt [16354.029758] ? __hrtimer_run_queues+0x1d3/0x360 [16354.032461] iTCO_vendor_support i2c_smbus [16354.036729] ? hrtimer_interrupt+0x104/0x240 [16354.039786] crct10dif_pclmul joydev [16354.044309] ? __local_bh_enable+0x50/0x60 [16354.048400] crc32_pclmul ipmi_ssif [16354.052666] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16354.056233] ghash_clmulni_intel [16354.060325] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16354.063808] mei_me [16354.069025] [16354.072250] mei lpc_ich [16354.077206] [16354.079306] sha512_ssse3 [16354.081403] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16354.083933] acpi_ipmi [16354.086035] ? queued_spin_lock_slowpath+0x16/0x2e0 [16354.088649] mfd_core [16354.093955] do_raw_spin_lock+0xb5/0xc0 [16354.096308] wmi ipmi_si [16354.101183] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16354.103451] ipmi_devintf ipmi_msghandler [16354.107285] try_get_cap_refs+0x159/0x880 [ceph] [16354.109813] ioatdma acpi_pad [16354.114603] __ceph_get_caps+0x4a1/0x760 [ceph] [16354.118601] acpi_power_meter ixgbe [16354.123217] ? __file_remove_privs+0x89/0x110 [16354.126175] igb mdio [16354.130700] ? wake_bit_function+0x70/0x70 [16354.134184] crc32c_intel nvme [16354.138537] ? ceph_write_iter+0x260/0x770 [ceph] [16354.140805] ptp nvme_core [16354.144895] ceph_write_iter+0x260/0x770 [ceph] [16354.147947] i2c_algo_bit pps_core [16354.152647] ? lock_is_held_type+0xaa/0x120 [16354.155347] dca fuse [16354.159873] vfs_write+0x324/0x4f0 [16354.163270] [16354.163271] irq event stamp: 53887438 [16354.167453] ksys_pwrite64+0x65/0xa0 [16354.169718] hardirqs last enabled at (53887437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16354.173114] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16354.174605] hardirqs last disabled at (53887438): [] sysvec_apic_timer_interrupt+0xe/0x90 [16354.178264] do_syscall_64+0x38/0x80 [16354.181834] softirqs last enabled at (53887436): [] __do_softirq+0x2be/0x371 [16354.191991] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16354.197122] softirqs last disabled at (53887431): [] irq_exit_rcu+0xb0/0xd0 [16354.206847] RIP: 0033:0x7f48f0eeb93b [16354.210416] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16354.219102] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16354.224146] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16354.232656] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16354.236227] Workqueue: events delayed_work [ceph] [16354.245604] ORIG_RAX: 0000000000000012 [16354.264342] [16354.264343] RIP: 0010:igrab+0x46/0x50 [16354.271812] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16354.277030] Code: 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 48 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 31 c0 5b <5d> c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 41 56 49 [16354.281728] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16354.285559] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [16354.287049] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16354.290706] [16354.290707] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [16354.297831] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16354.316569] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [16354.323692] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16354.328910] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [16354.336039] [16354.337524] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [16354.382470] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16354.389595] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16354.397671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16354.403409] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16354.410533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16354.417658] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16354.424793] Call Trace: [16354.427236] [16354.429247] ? watchdog_timer_fn+0x1b4/0x220 [16354.433512] ? __hrtimer_run_queues+0x1d3/0x360 [16354.438038] ? hrtimer_interrupt+0x104/0x240 [16354.442305] ? __local_bh_enable+0x50/0x60 [16354.446400] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16354.451624] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16354.456581] [16354.458677] [16354.460791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16354.466100] ? igrab+0x46/0x50 [16354.469157] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16354.474307] delayed_work+0x1d8/0x280 [ceph] [16354.478588] ? process_scheduled_works+0x22a/0x510 [16354.483379] process_scheduled_works+0x2a4/0x510 [16354.487996] worker_thread+0x188/0x340 [16354.491742] ? rescuer_thread+0x310/0x310 [16354.495747] kthread+0xf5/0x130 [16354.498894] ? kthread_complete_and_exit+0x20/0x20 [16354.503677] ret_from_fork+0x31/0x40 [16354.507258] ? kthread_complete_and_exit+0x20/0x20 [16354.512049] ret_from_fork_asm+0x11/0x20 [16354.515971] [16381.741602] watchdog: BUG: soft lockup - CPU#0 stuck for 4142s! [dbench:78683] [16381.748815] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16381.753602] watchdog: BUG: soft lockup - CPU#7 stuck for 13139s! [kworker/7:4:78808] [16381.804273] irq event stamp: 22100448 [16381.804274] hardirqs last enabled at (22100447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16381.812004] Modules linked in: [16381.815663] hardirqs last disabled at (22100448): [] sysvec_apic_timer_interrupt+0xe/0x90 [16381.825819] ceph [16381.828870] softirqs last enabled at (22100342): [] __do_softirq+0x2be/0x371 [16381.838594] libceph [16381.840519] softirqs last disabled at (22100337): [] irq_exit_rcu+0xb0/0xd0 [16381.849201] dns_resolver fscache [16381.851386] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16381.859898] netfs veth [16381.863208] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16381.872152] nft_chain_nat [16381.874596] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [16381.882068] xt_MASQUERADE nf_nat [16381.884772] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [16381.890247] nf_conntrack nf_defrag_ipv6 [16381.893560] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16381.912298] nf_defrag_ipv4 [16381.916214] [16381.921433] nft_compat nf_tables [16381.924223] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16381.925714] nfnetlink bridge [16381.929024] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16381.936149] stp llc [16381.939113] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16381.946236] xfs libcrc32c [16381.948420] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16381.955545] sunrpc intel_rapl_msr [16381.958248] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16381.965373] intel_rapl_common sb_edac [16381.968770] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16381.975893] x86_pkg_temp_thermal coretemp [16381.979638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16381.987715] kvm_intel kvm [16381.991805] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16381.997543] irqbypass i2c_i801 [16382.000249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16382.007372] iTCO_wdt iTCO_vendor_support [16382.010509] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16382.017634] i2c_smbus crct10dif_pclmul [16382.021639] Call Trace: [16382.021640] [16382.028761] joydev [16382.032594] ? watchdog_timer_fn+0x1b4/0x220 [16382.035035] crc32_pclmul ipmi_ssif [16382.037049] ? __hrtimer_run_queues+0x1d3/0x360 [16382.039145] ghash_clmulni_intel mei_me [16382.043411] ? hrtimer_interrupt+0x104/0x240 [16382.046894] mei lpc_ich [16382.051416] ? __local_bh_enable+0x50/0x60 [16382.055248] sha512_ssse3 acpi_ipmi [16382.059515] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16382.062042] mfd_core wmi [16382.066134] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16382.069618] ipmi_si [16382.074834] [16382.077453] ipmi_devintf ipmi_msghandler [16382.082410] [16382.084593] ioatdma acpi_pad [16382.086692] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16382.090694] acpi_power_meter ixgbe [16382.092796] ? queued_spin_lock_slowpath+0x2b/0x2e0 [16382.095757] igb mdio [16382.101062] do_raw_spin_lock+0xb5/0xc0 [16382.104544] crc32c_intel nvme [16382.109415] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16382.111687] ptp [16382.115518] try_get_cap_refs+0x159/0x880 [ceph] [16382.118567] nvme_core i2c_algo_bit [16382.123355] __ceph_get_caps+0x4a1/0x760 [ceph] [16382.125187] pps_core dca [16382.129803] ? __file_remove_privs+0x89/0x110 [16382.133295] fuse [16382.137818] ? wake_bit_function+0x70/0x70 [16382.140433] irq event stamp: 54110138 [16382.144786] ? ceph_write_iter+0x260/0x770 [ceph] [16382.146708] hardirqs last enabled at (54110137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16382.150800] ceph_write_iter+0x260/0x770 [ceph] [16382.154456] hardirqs last disabled at (54110138): [] sysvec_apic_timer_interrupt+0xe/0x90 [16382.159157] ? lock_is_held_type+0xaa/0x120 [16382.169310] softirqs last enabled at (54110136): [] __do_softirq+0x2be/0x371 [16382.173838] vfs_write+0x324/0x4f0 [16382.183560] softirqs last disabled at (54110131): [] irq_exit_rcu+0xb0/0xd0 [16382.187743] ksys_pwrite64+0x65/0xa0 [16382.196422] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16382.199819] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16382.208329] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16382.211903] do_syscall_64+0x38/0x80 [16382.221283] Workqueue: events delayed_work [ceph] [16382.226419] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16382.233889] [16382.233890] RIP: 0010:lock_acquire+0xf0/0x2b0 [16382.237459] RIP: 0033:0x7f48f0eeb93b [16382.242155] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16382.247201] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16382.248691] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16382.253043] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16382.256614] [16382.275351] ORIG_RAX: 0000000000000012 [16382.294089] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16382.299306] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16382.304523] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16382.306012] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16382.309845] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16382.316967] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16382.324092] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16382.331216] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16382.338340] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16382.345465] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16382.352589] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16382.359718] [16382.366838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16382.397085] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16382.404210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16382.411334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16382.418456] Call Trace: [16382.420904] [16382.422913] ? watchdog_timer_fn+0x1b4/0x220 [16382.427176] ? __hrtimer_run_queues+0x1d3/0x360 [16382.431702] ? hrtimer_interrupt+0x104/0x240 [16382.435965] ? __local_bh_enable+0x50/0x60 [16382.440058] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16382.445290] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16382.450248] [16382.452344] [16382.454443] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16382.459751] ? lock_acquire+0xf0/0x2b0 [16382.463499] ? igrab+0x43/0x50 [16382.466551] ? lock_release+0xc5/0x260 [16382.470295] _raw_spin_lock+0x30/0x40 [16382.473951] ? igrab+0x1d/0x50 [16382.477010] igrab+0x1d/0x50 [16382.479889] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16382.485038] delayed_work+0x1d8/0x280 [ceph] [16382.489320] ? process_scheduled_works+0x22a/0x510 [16382.494110] process_scheduled_works+0x2a4/0x510 [16382.498726] worker_thread+0x188/0x340 [16382.502475] ? rescuer_thread+0x310/0x310 [16382.506485] kthread+0xf5/0x130 [16382.509623] ? kthread_complete_and_exit+0x20/0x20 [16382.514408] ret_from_fork+0x31/0x40 [16382.517978] ? kthread_complete_and_exit+0x20/0x20 [16382.522764] ret_from_fork_asm+0x11/0x20 [16382.526694] [16409.742100] watchdog: BUG: soft lockup - CPU#0 stuck for 4168s! [dbench:78683] [16409.749320] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16409.754099] watchdog: BUG: soft lockup - CPU#7 stuck for 13165s! [kworker/7:4:78808] [16409.804774] irq event stamp: 22158582 [16409.804775] hardirqs last enabled at (22158581): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16409.812508] Modules linked in: [16409.816166] hardirqs last disabled at (22158582): [] sysvec_apic_timer_interrupt+0xe/0x90 [16409.826323] ceph [16409.829373] softirqs last enabled at (22158536): [] __do_softirq+0x2be/0x371 [16409.839100] libceph [16409.841023] softirqs last disabled at (22158531): [] irq_exit_rcu+0xb0/0xd0 [16409.849705] dns_resolver fscache [16409.851892] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16409.860403] netfs [16409.863712] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16409.872657] veth nft_chain_nat [16409.874669] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16409.882139] xt_MASQUERADE nf_nat [16409.885276] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16409.890752] nf_conntrack nf_defrag_ipv6 [16409.894080] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16409.912819] nf_defrag_ipv4 nft_compat [16409.916736] [16409.916737] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16409.921954] nf_tables nfnetlink [16409.925698] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16409.927189] bridge [16409.934311] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16409.937538] stp llc [16409.944661] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16409.946773] xfs libcrc32c [16409.953899] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16409.956083] sunrpc intel_rapl_msr [16409.963208] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16409.965913] intel_rapl_common [16409.973036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16409.976433] sb_edac x86_pkg_temp_thermal [16409.984510] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16409.987562] coretemp kvm_intel [16409.993299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16409.997302] kvm irqbypass [16410.004426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16410.007565] i2c_i801 [16410.014689] Call Trace: [16410.014691] [16410.017391] iTCO_wdt iTCO_vendor_support [16410.024520] ? watchdog_timer_fn+0x1b4/0x220 [16410.026788] i2c_smbus crct10dif_pclmul [16410.029232] ? __hrtimer_run_queues+0x1d3/0x360 [16410.031241] joydev crc32_pclmul [16410.035250] ? hrtimer_interrupt+0x104/0x240 [16410.039510] ipmi_ssif ghash_clmulni_intel [16410.043340] ? __local_bh_enable+0x50/0x60 [16410.047866] mei_me mei [16410.051092] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16410.055354] lpc_ich sha512_ssse3 [16410.059444] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16410.063533] acpi_ipmi mfd_core [16410.065978] [16410.071197] wmi ipmi_si [16410.074508] [16410.079463] ipmi_devintf ipmi_msghandler [16410.082602] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16410.084699] ioatdma acpi_pad [16410.087233] ? queued_spin_lock_slowpath+0x16/0x2e0 [16410.089328] acpi_power_meter [16410.093333] do_raw_spin_lock+0xb5/0xc0 [16410.098636] ixgbe igb mdio [16410.101602] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16410.106472] crc32c_intel [16410.109456] try_get_cap_refs+0x159/0x880 [ceph] [16410.113282] nvme ptp [16410.116078] __ceph_get_caps+0x4a1/0x760 [ceph] [16410.120858] nvme_core i2c_algo_bit [16410.123497] ? __file_remove_privs+0x89/0x110 [16410.128103] pps_core dca [16410.130373] ? wake_bit_function+0x70/0x70 [16410.134896] fuse [16410.138383] ? ceph_write_iter+0x260/0x770 [ceph] [16410.142732] irq event stamp: 54331976 [16410.142733] hardirqs last enabled at (54331975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16410.145350] ceph_write_iter+0x260/0x770 [ceph] [16410.149440] hardirqs last disabled at (54331976): [] sysvec_apic_timer_interrupt+0xe/0x90 [16410.151369] ? lock_is_held_type+0xaa/0x120 [16410.156061] softirqs last enabled at (54331974): [] __do_softirq+0x2be/0x371 [16410.159721] vfs_write+0x324/0x4f0 [16410.169877] softirqs last disabled at (54331969): [] irq_exit_rcu+0xb0/0xd0 [16410.174405] ksys_pwrite64+0x65/0xa0 [16410.184124] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16410.188302] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16410.196988] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16410.196989] Workqueue: events delayed_work [ceph] [16410.200386] do_syscall_64+0x38/0x80 [16410.208895] [16410.208895] RIP: 0010:lock_acquire+0xf0/0x2b0 [16410.212466] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16410.221843] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16410.226975] RIP: 0033:0x7f48f0eeb93b [16410.234445] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16410.239144] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16410.242712] [16410.242713] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16410.244203] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16410.248553] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16410.253599] ORIG_RAX: 0000000000000012 [16410.272337] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16410.275907] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16410.281125] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16410.299863] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16410.301351] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16410.308475] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16410.313710] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16410.320835] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16410.324665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16410.331793] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16410.338914] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16410.346043] [16410.353163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16410.411932] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16410.419058] Call Trace: [16410.421506] [16410.423521] ? watchdog_timer_fn+0x1b4/0x220 [16410.427785] ? __hrtimer_run_queues+0x1d3/0x360 [16410.432311] ? hrtimer_interrupt+0x104/0x240 [16410.436573] ? __local_bh_enable+0x50/0x60 [16410.440666] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16410.445891] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16410.450848] [16410.452945] [16410.455042] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16410.460352] ? lock_acquire+0xf0/0x2b0 [16410.464101] ? igrab+0x43/0x50 [16410.467149] ? lock_release+0xc5/0x260 [16410.470895] _raw_spin_lock+0x30/0x40 [16410.474560] ? igrab+0x1d/0x50 [16410.477611] igrab+0x1d/0x50 [16410.480506] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16410.485657] delayed_work+0x1d8/0x280 [ceph] [16410.489938] ? process_scheduled_works+0x22a/0x510 [16410.494726] process_scheduled_works+0x2a4/0x510 [16410.499342] worker_thread+0x188/0x340 [16410.503092] ? rescuer_thread+0x310/0x310 [16410.507095] kthread+0xf5/0x130 [16410.510234] ? kthread_complete_and_exit+0x20/0x20 [16410.515017] ret_from_fork+0x31/0x40 [16410.518596] ? kthread_complete_and_exit+0x20/0x20 [16410.523379] ret_from_fork_asm+0x11/0x20 [16410.527303] [16437.742596] watchdog: BUG: soft lockup - CPU#0 stuck for 4194s! [dbench:78683] [16437.749807] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16437.754596] watchdog: BUG: soft lockup - CPU#7 stuck for 13191s! [kworker/7:4:78808] [16437.805247] irq event stamp: 22216734 [16437.805248] hardirqs last enabled at (22216733): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16437.812991] Modules linked in: [16437.816644] hardirqs last disabled at (22216734): [] sysvec_apic_timer_interrupt+0xe/0x90 [16437.826801] ceph [16437.829852] softirqs last enabled at (22216658): [] __do_softirq+0x2be/0x371 [16437.839576] libceph [16437.841501] softirqs last disabled at (22216653): [] irq_exit_rcu+0xb0/0xd0 [16437.850184] dns_resolver [16437.852369] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16437.860880] fscache netfs [16437.863498] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16437.872441] veth nft_chain_nat [16437.875145] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16437.882616] xt_MASQUERADE [16437.885753] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16437.891231] nf_nat nf_conntrack [16437.893934] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16437.912673] nf_defrag_ipv6 [16437.915897] [16437.915898] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16437.921115] nf_defrag_ipv4 [16437.923904] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16437.925397] nft_compat nf_tables [16437.932520] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16437.935309] nfnetlink bridge [16437.942435] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16437.945747] stp [16437.952868] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16437.955835] llc xfs [16437.962957] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16437.964795] libcrc32c sunrpc [16437.971919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16437.974103] intel_rapl_msr intel_rapl_common [16437.982181] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16437.985144] sb_edac x86_pkg_temp_thermal [16437.990883] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16437.995234] coretemp kvm_intel [16438.002357] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16438.006362] kvm irqbypass [16438.013484] Call Trace: [16438.013486] [16438.016622] i2c_i801 iTCO_wdt [16438.023749] ? watchdog_timer_fn+0x1b4/0x220 [16438.026450] iTCO_vendor_support i2c_smbus [16438.028895] ? __hrtimer_run_queues+0x1d3/0x360 [16438.030905] crct10dif_pclmul [16438.033961] ? hrtimer_interrupt+0x104/0x240 [16438.038221] joydev crc32_pclmul [16438.042311] ? __local_bh_enable+0x50/0x60 [16438.046834] ipmi_ssif [16438.049801] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16438.054064] ghash_clmulni_intel mei_me [16438.057288] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16438.061379] mei lpc_ich [16438.063735] [16438.068953] sha512_ssse3 [16438.072783] [16438.077742] acpi_ipmi mfd_core [16438.080279] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16438.082377] wmi [16438.085001] ? queued_spin_lock_slowpath+0x18/0x2e0 [16438.087093] ipmi_si ipmi_devintf [16438.090232] do_raw_spin_lock+0xb5/0xc0 [16438.095535] ipmi_msghandler ioatdma [16438.097372] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16438.102242] acpi_pad acpi_power_meter [16438.105556] try_get_cap_refs+0x159/0x880 [ceph] [16438.109383] ixgbe igb [16438.112960] __ceph_get_caps+0x4a1/0x760 [ceph] [16438.117738] mdio crc32c_intel [16438.121486] ? __file_remove_privs+0x89/0x110 [16438.126093] nvme ptp [16438.128452] ? wake_bit_function+0x70/0x70 [16438.132974] nvme_core [16438.136046] ? ceph_write_iter+0x260/0x770 [ceph] [16438.140393] i2c_algo_bit pps_core [16438.142665] ceph_write_iter+0x260/0x770 [ceph] [16438.146755] dca fuse [16438.149117] ? lock_is_held_type+0xaa/0x120 [16438.153810] [16438.153810] irq event stamp: 54554894 [16438.157211] vfs_write+0x324/0x4f0 [16438.161730] hardirqs last enabled at (54554893): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16438.164007] ksys_pwrite64+0x65/0xa0 [16438.168180] hardirqs last disabled at (54554894): [] sysvec_apic_timer_interrupt+0xe/0x90 [16438.169672] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16438.173327] softirqs last enabled at (54554892): [] __do_softirq+0x2be/0x371 [16438.176725] do_syscall_64+0x38/0x80 [16438.186882] softirqs last disabled at (54554887): [] irq_exit_rcu+0xb0/0xd0 [16438.190453] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16438.200178] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16438.205310] RIP: 0033:0x7f48f0eeb93b [16438.213997] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16438.217573] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16438.226082] Workqueue: events delayed_work [ceph] [16438.231126] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16438.240503] [16438.240504] RIP: 0010:lock_acquire+0xf0/0x2b0 [16438.244076] ORIG_RAX: 0000000000000012 [16438.251546] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16438.270287] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16438.274998] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16438.280215] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16438.281705] [16438.281706] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16438.286058] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16438.289887] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16438.308624] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16438.315749] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16438.320967] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16438.328091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16438.329588] [16438.336707] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16438.388758] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16438.396838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16438.402575] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16438.409699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16438.416823] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16438.423947] Call Trace: [16438.426391] [16438.428401] ? watchdog_timer_fn+0x1b4/0x220 [16438.432666] ? __hrtimer_run_queues+0x1d3/0x360 [16438.437193] ? hrtimer_interrupt+0x104/0x240 [16438.441463] ? __local_bh_enable+0x50/0x60 [16438.445556] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16438.450780] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16438.455737] [16438.457834] [16438.459932] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16438.465240] ? lock_acquire+0xf0/0x2b0 [16438.469003] ? igrab+0x43/0x50 [16438.472058] ? lock_release+0xc5/0x260 [16438.475803] _raw_spin_lock+0x30/0x40 [16438.479467] ? igrab+0x1d/0x50 [16438.482516] igrab+0x1d/0x50 [16438.485394] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16438.490546] delayed_work+0x1d8/0x280 [ceph] [16438.494827] ? process_scheduled_works+0x22a/0x510 [16438.499617] process_scheduled_works+0x2a4/0x510 [16438.504231] worker_thread+0x188/0x340 [16438.507998] ? rescuer_thread+0x310/0x310 [16438.512001] kthread+0xf5/0x130 [16438.515139] ? kthread_complete_and_exit+0x20/0x20 [16438.519923] ret_from_fork+0x31/0x40 [16438.523502] ? kthread_complete_and_exit+0x20/0x20 [16438.528287] ret_from_fork_asm+0x11/0x20 [16438.532210] [16465.743094] watchdog: BUG: soft lockup - CPU#0 stuck for 4220s! [dbench:78683] [16465.750310] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16465.755093] watchdog: BUG: soft lockup - CPU#7 stuck for 13217s! [kworker/7:4:78808] [16465.805770] irq event stamp: 22274876 [16465.805771] hardirqs last enabled at (22274875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16465.813501] Modules linked in: [16465.817157] hardirqs last disabled at (22274876): [] sysvec_apic_timer_interrupt+0xe/0x90 [16465.827314] ceph [16465.830366] softirqs last enabled at (22274768): [] __do_softirq+0x2be/0x371 [16465.840089] libceph [16465.842013] softirqs last disabled at (22274763): [] irq_exit_rcu+0xb0/0xd0 [16465.850698] dns_resolver [16465.852883] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16465.861391] fscache netfs [16465.864009] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16465.872954] veth nft_chain_nat [16465.875660] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16465.883128] xt_MASQUERADE nf_nat [16465.886269] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16465.891745] nf_conntrack [16465.895056] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16465.913792] nf_defrag_ipv6 [16465.916409] [16465.916410] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16465.921627] nf_defrag_ipv4 nft_compat [16465.924417] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16465.925908] nf_tables nfnetlink [16465.933033] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16465.936777] bridge stp [16465.943900] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16465.947124] llc xfs [16465.954249] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16465.956694] libcrc32c sunrpc [16465.963819] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16465.966001] intel_rapl_msr intel_rapl_common [16465.973125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16465.976089] sb_edac x86_pkg_temp_thermal [16465.984166] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16465.988517] coretemp kvm_intel [16465.994256] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16465.998260] kvm irqbypass [16466.005384] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16466.008522] i2c_i801 iTCO_wdt [16466.015644] Call Trace: [16466.015646] [16466.018348] iTCO_vendor_support i2c_smbus [16466.025493] ? watchdog_timer_fn+0x1b4/0x220 [16466.028541] crct10dif_pclmul [16466.030987] ? __hrtimer_run_queues+0x1d3/0x360 [16466.032996] joydev crc32_pclmul [16466.037089] ? hrtimer_interrupt+0x104/0x240 [16466.041351] ipmi_ssif ghash_clmulni_intel [16466.044314] ? __local_bh_enable+0x50/0x60 [16466.048838] mei_me [16466.052067] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16466.056326] mei lpc_ich [16466.060417] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16466.064508] sha512_ssse3 acpi_ipmi [16466.066607] [16466.071823] mfd_core wmi [16466.074354] [16466.074355] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16466.079310] ipmi_si ipmi_devintf [16466.082800] ? queued_spin_lock_slowpath+0x16/0x2e0 [16466.084892] ipmi_msghandler ioatdma [16466.087513] do_raw_spin_lock+0xb5/0xc0 [16466.089607] acpi_pad acpi_power_meter [16466.094915] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16466.098222] ixgbe igb mdio [16466.103096] try_get_cap_refs+0x159/0x880 [ceph] [16466.106664] crc32c_intel nvme [16466.110501] __ceph_get_caps+0x4a1/0x760 [ceph] [16466.114239] ptp nvme_core [16466.119027] ? __file_remove_privs+0x89/0x110 [16466.121815] i2c_algo_bit pps_core [16466.126423] ? wake_bit_function+0x70/0x70 [16466.129491] dca fuse [16466.134019] ? ceph_write_iter+0x260/0x770 [ceph] [16466.136720] [16466.136721] irq event stamp: 54776950 [16466.141072] ceph_write_iter+0x260/0x770 [ceph] [16466.144467] hardirqs last enabled at (54776949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16466.148580] ? lock_is_held_type+0xaa/0x120 [16466.150847] hardirqs last disabled at (54776950): [] sysvec_apic_timer_interrupt+0xe/0x90 [16466.155547] vfs_write+0x324/0x4f0 [16466.157036] softirqs last enabled at (54776948): [] __do_softirq+0x2be/0x371 [16466.160699] ksys_pwrite64+0x65/0xa0 [16466.165216] softirqs last disabled at (54776943): [] irq_exit_rcu+0xb0/0xd0 [16466.175375] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16466.179553] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16466.189276] do_syscall_64+0x38/0x80 [16466.192673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16466.201358] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16466.204928] Workqueue: events delayed_work [ceph] [16466.213439] RIP: 0033:0x7f48f0eeb93b [16466.218570] [16466.227949] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16466.231519] RIP: 0010:lock_acquire+0xf0/0x2b0 [16466.238988] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16466.244033] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16466.248731] ORIG_RAX: 0000000000000012 [16466.252301] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16466.253791] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16466.272529] [16466.272530] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16466.276880] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16466.282098] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16466.300836] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16466.304666] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16466.309884] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16466.317008] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16466.318497] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16466.325622] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16466.332751] [16466.339871] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16466.392885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16466.398622] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16466.405748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16466.412871] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16466.419998] Call Trace: [16466.422440] [16466.424451] ? watchdog_timer_fn+0x1b4/0x220 [16466.428717] ? __hrtimer_run_queues+0x1d3/0x360 [16466.433250] ? hrtimer_interrupt+0x104/0x240 [16466.437511] ? __local_bh_enable+0x50/0x60 [16466.441606] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16466.446830] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16466.451788] [16466.453882] [16466.455982] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16466.461290] ? lock_acquire+0xf0/0x2b0 [16466.465039] ? igrab+0x43/0x50 [16466.468088] ? lock_release+0xc5/0x260 [16466.471834] _raw_spin_lock+0x30/0x40 [16466.475503] ? igrab+0x1d/0x50 [16466.478557] igrab+0x1d/0x50 [16466.481437] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16466.486586] delayed_work+0x1d8/0x280 [ceph] [16466.490868] ? process_scheduled_works+0x22a/0x510 [16466.495657] process_scheduled_works+0x2a4/0x510 [16466.500273] worker_thread+0x188/0x340 [16466.504023] ? rescuer_thread+0x310/0x310 [16466.508025] kthread+0xf5/0x130 [16466.511173] ? kthread_complete_and_exit+0x20/0x20 [16466.515955] ret_from_fork+0x31/0x40 [16466.519535] ? kthread_complete_and_exit+0x20/0x20 [16466.524318] ret_from_fork_asm+0x11/0x20 [16466.528243] [16493.743590] watchdog: BUG: soft lockup - CPU#0 stuck for 4246s! [dbench:78683] [16493.750806] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16493.755590] watchdog: BUG: soft lockup - CPU#7 stuck for 13244s! [kworker/7:4:78808] [16493.806246] irq event stamp: 22333058 [16493.806248] hardirqs last enabled at (22333057): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16493.813996] Modules linked in: [16493.817652] hardirqs last disabled at (22333058): [] sysvec_apic_timer_interrupt+0xe/0x90 [16493.827809] ceph [16493.830861] softirqs last enabled at (22332952): [] __do_softirq+0x2be/0x371 [16493.840586] libceph [16493.842509] softirqs last disabled at (22332947): [] irq_exit_rcu+0xb0/0xd0 [16493.851193] dns_resolver fscache [16493.853379] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16493.861888] netfs veth [16493.865200] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16493.874144] nft_chain_nat [16493.876589] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [16493.884059] xt_MASQUERADE nf_nat [16493.886762] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [16493.892240] nf_conntrack [16493.895568] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16493.914306] nf_defrag_ipv6 nf_defrag_ipv4 [16493.916924] [16493.922140] nft_compat nf_tables [16493.926232] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16493.927722] nfnetlink [16493.931031] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16493.938156] bridge [16493.940513] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16493.947637] stp [16493.949736] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16493.956860] llc xfs [16493.958696] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16493.965821] libcrc32c sunrpc [16493.968004] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16493.975130] intel_rapl_msr intel_rapl_common [16493.978093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16493.986170] sb_edac x86_pkg_temp_thermal [16493.990522] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16493.996267] coretemp kvm_intel [16494.000273] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16494.007396] kvm irqbypass [16494.010533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16494.017657] i2c_i801 iTCO_wdt [16494.020362] Call Trace: [16494.020363] [16494.027486] iTCO_vendor_support i2c_smbus [16494.030536] ? watchdog_timer_fn+0x1b4/0x220 [16494.032988] crct10dif_pclmul joydev [16494.035000] ? __hrtimer_run_queues+0x1d3/0x360 [16494.039090] crc32_pclmul ipmi_ssif [16494.043359] ? hrtimer_interrupt+0x104/0x240 [16494.046924] ghash_clmulni_intel [16494.051448] ? __local_bh_enable+0x50/0x60 [16494.054934] mei_me mei [16494.059200] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16494.062422] lpc_ich [16494.066514] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16494.068955] sha512_ssse3 acpi_ipmi [16494.074173] [16494.076357] mfd_core wmi [16494.081316] [16494.084800] ipmi_si ipmi_devintf [16494.086898] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16494.089514] ipmi_msghandler [16494.091615] ? queued_spin_lock_slowpath+0x2b/0x2e0 [16494.094921] ioatdma acpi_pad [16494.100229] do_raw_spin_lock+0xb5/0xc0 [16494.103104] acpi_power_meter ixgbe [16494.107991] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16494.110957] igb mdio [16494.114791] try_get_cap_refs+0x159/0x880 [ceph] [16494.118271] crc32c_intel nvme [16494.123059] __ceph_get_caps+0x4a1/0x760 [ceph] [16494.125324] ptp nvme_core [16494.129940] ? __file_remove_privs+0x89/0x110 [16494.132989] i2c_algo_bit pps_core [16494.137512] ? wake_bit_function+0x70/0x70 [16494.140216] dca fuse [16494.144570] ? ceph_write_iter+0x260/0x770 [ceph] [16494.147963] [16494.147964] irq event stamp: 54998388 [16494.152071] ceph_write_iter+0x260/0x770 [ceph] [16494.154341] hardirqs last enabled at (54998387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16494.159043] ? lock_is_held_type+0xaa/0x120 [16494.160531] hardirqs last disabled at (54998388): [] sysvec_apic_timer_interrupt+0xe/0x90 [16494.164191] vfs_write+0x324/0x4f0 [16494.168713] softirqs last enabled at (54998386): [] __do_softirq+0x2be/0x371 [16494.178873] ksys_pwrite64+0x65/0xa0 [16494.183047] softirqs last disabled at (54998381): [] irq_exit_rcu+0xb0/0xd0 [16494.192773] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16494.196167] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16494.204853] do_syscall_64+0x38/0x80 [16494.208423] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16494.216934] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16494.222065] Workqueue: events delayed_work [ceph] [16494.231442] RIP: 0033:0x7f48f0eeb93b [16494.235013] [16494.242485] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16494.247527] RIP: 0010:lock_acquire+0x17b/0x2b0 [16494.252225] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16494.255796] Code: c8 89 d1 89 f2 4c 89 e6 e8 62 5b ff ff 59 5e 4c 8b 14 24 65 ff 0d 5d e6 eb 7e 0f 85 c8 fe ff ff 0f 1f 44 00 00 e9 be fe ff ff <4c> 89 14 24 e8 9c 8a 02 00 4c 8b 14 24 84 c0 0f 85 c0 fe ff ff 0f [16494.257286] ORIG_RAX: 0000000000000012 [16494.276023] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [16494.280461] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16494.285678] [16494.285679] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [16494.304416] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16494.308246] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [16494.313466] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16494.320588] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [16494.322080] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16494.329204] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [16494.336329] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16494.343451] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16494.350582] [16494.357701] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16494.396459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16494.402195] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16494.409320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16494.416444] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16494.423568] Call Trace: [16494.426011] [16494.428024] ? watchdog_timer_fn+0x1b4/0x220 [16494.432289] ? __hrtimer_run_queues+0x1d3/0x360 [16494.436816] ? hrtimer_interrupt+0x104/0x240 [16494.441083] ? __local_bh_enable+0x50/0x60 [16494.445178] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16494.450403] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16494.455358] [16494.457455] [16494.459553] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16494.464861] ? lock_acquire+0x17b/0x2b0 [16494.468699] ? igrab+0x43/0x50 [16494.471749] ? lock_release+0xc5/0x260 [16494.475494] _raw_spin_lock+0x30/0x40 [16494.479158] ? igrab+0x1d/0x50 [16494.482209] igrab+0x1d/0x50 [16494.485087] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16494.490236] delayed_work+0x1d8/0x280 [ceph] [16494.494517] ? process_scheduled_works+0x22a/0x510 [16494.499308] process_scheduled_works+0x2a4/0x510 [16494.503923] worker_thread+0x188/0x340 [16494.507672] ? rescuer_thread+0x310/0x310 [16494.511676] kthread+0xf5/0x130 [16494.514814] ? kthread_complete_and_exit+0x20/0x20 [16494.519597] ret_from_fork+0x31/0x40 [16494.523168] ? kthread_complete_and_exit+0x20/0x20 [16494.527951] ret_from_fork_asm+0x11/0x20 [16494.531877] [16497.156018] rcu: INFO: rcu_preempt self-detected stall on CPU [16497.161774] rcu: 7-....: (4431192 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/4429984 fqs=2208905 [16497.172193] rcu: (t=4560465 jiffies g=1309609 q=204194 ncpus=8) [16497.178191] Sending NMI from CPU 7 to CPUs 0: [16497.182541] NMI backtrace for cpu 0 [16497.182543] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16497.182545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16497.182546] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16497.182549] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16497.182551] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16497.182552] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16497.182553] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16497.182554] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16497.182555] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16497.182556] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16497.182557] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16497.182558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16497.182559] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16497.182559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16497.182560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16497.182561] Call Trace: [16497.182562] [16497.182563] ? nmi_cpu_backtrace+0xad/0x120 [16497.182566] ? nmi_cpu_backtrace_handler+0x11/0x20 [16497.182568] ? nmi_handle+0xd4/0x260 [16497.182572] ? default_do_nmi+0x49/0xf0 [16497.182574] ? exc_nmi+0x11e/0x150 [16497.182576] ? end_repeat_nmi+0x16/0x67 [16497.182580] ? queued_spin_lock_slowpath+0x18/0x2e0 [16497.182582] ? queued_spin_lock_slowpath+0x18/0x2e0 [16497.182585] ? queued_spin_lock_slowpath+0x18/0x2e0 [16497.182587] [16497.182588] [16497.182588] do_raw_spin_lock+0xb5/0xc0 [16497.182591] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16497.182605] try_get_cap_refs+0x159/0x880 [ceph] [16497.182620] __ceph_get_caps+0x4a1/0x760 [ceph] [16497.182632] ? __file_remove_privs+0x89/0x110 [16497.182634] ? wake_bit_function+0x70/0x70 [16497.182637] ? ceph_write_iter+0x260/0x770 [ceph] [16497.182648] ceph_write_iter+0x260/0x770 [ceph] [16497.182659] ? lock_is_held_type+0xaa/0x120 [16497.182662] vfs_write+0x324/0x4f0 [16497.182666] ksys_pwrite64+0x65/0xa0 [16497.182668] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16497.182671] do_syscall_64+0x38/0x80 [16497.182673] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16497.182675] RIP: 0033:0x7f48f0eeb93b [16497.182676] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16497.182677] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [16497.182679] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16497.182679] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16497.182680] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16497.182681] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16497.182681] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16497.182684] [16497.183539] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16497.487361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16497.494832] Workqueue: events delayed_work [ceph] [16497.499545] RIP: 0010:lock_release+0x78/0x260 [16497.503897] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [16497.522636] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [16497.527853] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [16497.534976] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [16497.542101] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [16497.549226] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [16497.556348] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16497.563474] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16497.571550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16497.577288] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16497.584411] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16497.591536] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16497.598659] Call Trace: [16497.601104] [16497.603117] ? rcu_dump_cpu_stacks+0xd1/0x180 [16497.607475] ? rcu_sched_clock_irq+0x633/0x1390 [16497.612000] ? lock_release+0xc5/0x260 [16497.615743] ? lock_is_held_type+0xaa/0x120 [16497.619920] ? lock_is_held_type+0xaa/0x120 [16497.624100] ? tick_sched_do_timer+0xa0/0xa0 [16497.628371] ? rcu_read_lock_sched_held+0x56/0x90 [16497.633068] ? tick_sched_do_timer+0xa0/0xa0 [16497.637331] ? update_process_times+0x5f/0x90 [16497.641683] ? tick_sched_handle.isra.16+0x1e/0x50 [16497.646466] ? tick_sched_timer+0x6f/0x80 [16497.650472] ? __hrtimer_run_queues+0x1d3/0x360 [16497.654999] ? hrtimer_interrupt+0x104/0x240 [16497.659268] ? __local_bh_enable+0x50/0x60 [16497.663361] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16497.668584] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16497.673541] [16497.675640] [16497.677736] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16497.683043] ? igrab+0x43/0x50 [16497.686095] ? lock_release+0x78/0x260 [16497.689847] _raw_spin_unlock+0x1b/0x40 [16497.693684] igrab+0x43/0x50 [16497.696562] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16497.701711] delayed_work+0x1d8/0x280 [ceph] [16497.705992] ? process_scheduled_works+0x22a/0x510 [16497.710774] process_scheduled_works+0x2a4/0x510 [16497.715391] worker_thread+0x188/0x340 [16497.719138] ? rescuer_thread+0x310/0x310 [16497.723141] kthread+0xf5/0x130 [16497.726280] ? kthread_complete_and_exit+0x20/0x20 [16497.731064] ret_from_fork+0x31/0x40 [16497.734634] ? kthread_complete_and_exit+0x20/0x20 [16497.739420] ret_from_fork_asm+0x11/0x20 [16497.743341] [16521.744087] watchdog: BUG: soft lockup - CPU#0 stuck for 4273s! [dbench:78683] [16521.751302] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16521.756087] watchdog: BUG: soft lockup - CPU#7 stuck for 13270s! [kworker/7:4:78808] [16521.806759] irq event stamp: 22391228 [16521.806760] hardirqs last enabled at (22391227): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16521.814491] Modules linked in: [16521.818149] hardirqs last disabled at (22391228): [] sysvec_apic_timer_interrupt+0xe/0x90 [16521.828307] ceph [16521.831358] softirqs last enabled at (22391188): [] __do_softirq+0x2be/0x371 [16521.841081] libceph [16521.843005] softirqs last disabled at (22391183): [] irq_exit_rcu+0xb0/0xd0 [16521.851689] dns_resolver fscache [16521.853874] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16521.862384] netfs veth [16521.865694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16521.874638] nft_chain_nat [16521.877084] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16521.884555] xt_MASQUERADE [16521.887257] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16521.892735] nf_nat nf_conntrack [16521.895456] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16521.914212] nf_defrag_ipv6 nf_defrag_ipv4 [16521.917437] [16521.917438] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16521.922653] nft_compat [16521.926760] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16521.928251] nf_tables nfnetlink [16521.935376] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16521.937821] bridge stp [16521.944945] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16521.948168] llc xfs [16521.955293] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16521.957736] libcrc32c [16521.964861] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16521.967045] sunrpc [16521.974169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16521.976526] intel_rapl_msr intel_rapl_common [16521.984603] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16521.986702] sb_edac x86_pkg_temp_thermal [16521.992438] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16521.996788] coretemp [16522.003913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16522.007918] kvm_intel kvm [16522.015041] Call Trace: [16522.015042] [16522.017313] irqbypass i2c_i801 [16522.024438] ? watchdog_timer_fn+0x1b4/0x220 [16522.027140] iTCO_wdt iTCO_vendor_support [16522.029585] ? __hrtimer_run_queues+0x1d3/0x360 [16522.031594] i2c_smbus crct10dif_pclmul [16522.034735] ? hrtimer_interrupt+0x104/0x240 [16522.038998] joydev crc32_pclmul [16522.043002] ? __local_bh_enable+0x50/0x60 [16522.047525] ipmi_ssif ghash_clmulni_intel [16522.051359] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16522.055619] mei_me mei [16522.058845] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16522.062934] lpc_ich sha512_ssse3 [16522.067025] [16522.072242] acpi_ipmi mfd_core [16522.074688] [16522.074689] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16522.079643] wmi ipmi_si [16522.082958] ? queued_spin_lock_slowpath+0x18/0x2e0 [16522.085053] ipmi_devintf [16522.088193] do_raw_spin_lock+0xb5/0xc0 [16522.090287] ipmi_msghandler ioatdma [16522.095592] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16522.098121] acpi_pad [16522.102996] try_get_cap_refs+0x159/0x880 [ceph] [16522.105610] acpi_power_meter ixgbe [16522.109445] __ceph_get_caps+0x4a1/0x760 [ceph] [16522.113012] igb [16522.117799] ? __file_remove_privs+0x89/0x110 [16522.120065] mdio crc32c_intel [16522.124677] ? wake_bit_function+0x70/0x70 [16522.128162] nvme ptp [16522.132689] ? ceph_write_iter+0x260/0x770 [ceph] [16522.134523] nvme_core [16522.138874] ceph_write_iter+0x260/0x770 [ceph] [16522.141925] i2c_algo_bit [16522.146019] ? lock_is_held_type+0xaa/0x120 [16522.148286] pps_core dca [16522.152987] vfs_write+0x324/0x4f0 [16522.155339] fuse [16522.159870] ksys_pwrite64+0x65/0xa0 [16522.162485] irq event stamp: 55215086 [16522.166660] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16522.169276] hardirqs last enabled at (55215085): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16522.172676] do_syscall_64+0x38/0x80 [16522.174599] hardirqs last disabled at (55215086): [] sysvec_apic_timer_interrupt+0xe/0x90 [16522.178169] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16522.181825] softirqs last enabled at (55215084): [] __do_softirq+0x2be/0x371 [16522.186958] RIP: 0033:0x7f48f0eeb93b [16522.197115] softirqs last disabled at (55215079): [] irq_exit_rcu+0xb0/0xd0 [16522.200685] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16522.210409] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16522.215454] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16522.224139] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16522.227709] ORIG_RAX: 0000000000000012 [16522.236219] Workqueue: events delayed_work [ceph] [16522.254956] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16522.264335] [16522.269552] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16522.277022] RIP: 0010:lock_acquire+0xf0/0x2b0 [16522.280854] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16522.285550] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16522.292675] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16522.294165] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16522.301291] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16522.305641] [16522.305642] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16522.312778] [16522.331510] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16522.368917] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16522.376042] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16522.383165] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16522.390289] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16522.398367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16522.404106] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16522.411229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16522.418354] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16522.425490] Call Trace: [16522.427941] [16522.429949] ? watchdog_timer_fn+0x1b4/0x220 [16522.434215] ? __hrtimer_run_queues+0x1d3/0x360 [16522.438742] ? hrtimer_interrupt+0x104/0x240 [16522.443011] ? __local_bh_enable+0x50/0x60 [16522.447104] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16522.452328] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16522.457284] [16522.459383] [16522.461494] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16522.466804] ? lock_acquire+0xf0/0x2b0 [16522.470556] ? igrab+0x43/0x50 [16522.473604] ? lock_release+0xc5/0x260 [16522.477351] _raw_spin_lock+0x30/0x40 [16522.481014] ? igrab+0x1d/0x50 [16522.484064] igrab+0x1d/0x50 [16522.486943] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16522.492094] delayed_work+0x1d8/0x280 [ceph] [16522.496374] ? process_scheduled_works+0x22a/0x510 [16522.501163] process_scheduled_works+0x2a4/0x510 [16522.505794] worker_thread+0x188/0x340 [16522.509547] ? rescuer_thread+0x310/0x310 [16522.513558] kthread+0xf5/0x130 [16522.516696] ? kthread_complete_and_exit+0x20/0x20 [16522.521493] ret_from_fork+0x31/0x40 [16522.525068] ? kthread_complete_and_exit+0x20/0x20 [16522.529852] ret_from_fork_asm+0x11/0x20 [16522.533780] [16549.744585] watchdog: BUG: soft lockup - CPU#0 stuck for 4299s! [dbench:78683] [16549.751800] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16549.756584] watchdog: BUG: soft lockup - CPU#7 stuck for 13296s! [kworker/7:4:78808] [16549.807258] irq event stamp: 22449506 [16549.807259] hardirqs last enabled at (22449505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16549.814989] Modules linked in: [16549.818646] hardirqs last disabled at (22449506): [] sysvec_apic_timer_interrupt+0xe/0x90 [16549.828802] ceph [16549.831854] softirqs last enabled at (22449408): [] __do_softirq+0x2be/0x371 [16549.841577] libceph [16549.843501] softirqs last disabled at (22449403): [] irq_exit_rcu+0xb0/0xd0 [16549.852185] dns_resolver fscache [16549.854371] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16549.862880] netfs veth [16549.866193] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16549.875134] nft_chain_nat xt_MASQUERADE [16549.877580] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16549.885050] nf_nat nf_conntrack [16549.888980] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16549.894454] nf_defrag_ipv6 nf_defrag_ipv4 [16549.897677] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16549.916415] nft_compat [16549.920508] [16549.920508] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16549.925725] nf_tables [16549.928168] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16549.929659] nfnetlink bridge [16549.936782] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16549.939139] stp llc [16549.946278] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16549.949260] xfs libcrc32c [16549.956386] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16549.958571] sunrpc intel_rapl_msr [16549.965694] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16549.968399] intel_rapl_common [16549.975523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16549.978920] sb_edac x86_pkg_temp_thermal [16549.986998] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16549.990050] coretemp kvm_intel [16549.995786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16549.999789] kvm irqbypass [16550.006914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16550.010051] i2c_i801 iTCO_wdt [16550.017176] Call Trace: [16550.017178] [16550.019879] iTCO_vendor_support i2c_smbus [16550.027006] ? watchdog_timer_fn+0x1b4/0x220 [16550.030055] crct10dif_pclmul joydev [16550.032499] ? __hrtimer_run_queues+0x1d3/0x360 [16550.034510] crc32_pclmul ipmi_ssif [16550.038603] ? hrtimer_interrupt+0x104/0x240 [16550.042864] ghash_clmulni_intel mei_me [16550.046435] ? __local_bh_enable+0x50/0x60 [16550.050960] mei [16550.054464] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16550.058725] lpc_ich sha512_ssse3 [16550.062556] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16550.066646] acpi_ipmi [16550.068485] [16550.073701] mfd_core wmi [16550.077011] [16550.081985] ipmi_si ipmi_devintf [16550.084345] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16550.086441] ipmi_msghandler ioatdma [16550.089062] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16550.091157] acpi_pad acpi_power_meter [16550.094469] do_raw_spin_lock+0xb5/0xc0 [16550.099771] ixgbe igb mdio [16550.103345] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16550.108213] crc32c_intel nvme [16550.111960] try_get_cap_refs+0x159/0x880 [ceph] [16550.115804] ptp nvme_core [16550.118600] __ceph_get_caps+0x4a1/0x760 [ceph] [16550.123379] i2c_algo_bit [16550.126435] ? __file_remove_privs+0x89/0x110 [16550.131042] pps_core dca [16550.133744] ? wake_bit_function+0x70/0x70 [16550.138269] fuse [16550.140888] ? ceph_write_iter+0x260/0x770 [ceph] [16550.145237] irq event stamp: 55436552 [16550.147856] ceph_write_iter+0x260/0x770 [ceph] [16550.151946] hardirqs last enabled at (55436551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16550.153875] ? lock_is_held_type+0xaa/0x120 [16550.158568] hardirqs last disabled at (55436552): [] sysvec_apic_timer_interrupt+0xe/0x90 [16550.162228] vfs_write+0x324/0x4f0 [16550.166748] softirqs last enabled at (55436550): [] __do_softirq+0x2be/0x371 [16550.176910] ksys_pwrite64+0x65/0xa0 [16550.181084] softirqs last disabled at (55436545): [] irq_exit_rcu+0xb0/0xd0 [16550.190809] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16550.194206] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16550.202890] do_syscall_64+0x38/0x80 [16550.206460] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16550.206462] Workqueue: events delayed_work [ceph] [16550.214986] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16550.220117] [16550.220118] RIP: 0010:lock_acquire+0xf0/0x2b0 [16550.229497] RIP: 0033:0x7f48f0eeb93b [16550.233066] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16550.240538] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16550.245235] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16550.250279] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16550.251771] [16550.251771] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16550.256119] ORIG_RAX: 0000000000000012 [16550.259692] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16550.278429] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16550.297165] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16550.302385] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16550.307603] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16550.309093] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16550.316217] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16550.320047] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16550.327171] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16550.334296] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16550.341420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16550.348550] [16550.355667] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16550.407297] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16550.414421] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16550.421545] Call Trace: [16550.423991] [16550.426000] ? watchdog_timer_fn+0x1b4/0x220 [16550.430277] ? __hrtimer_run_queues+0x1d3/0x360 [16550.434808] ? hrtimer_interrupt+0x104/0x240 [16550.439069] ? __local_bh_enable+0x50/0x60 [16550.443162] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16550.448386] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16550.453344] [16550.455441] [16550.457538] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16550.462846] ? lock_acquire+0xf0/0x2b0 [16550.466597] ? igrab+0x43/0x50 [16550.469647] ? lock_release+0xc5/0x260 [16550.473392] _raw_spin_lock+0x30/0x40 [16550.477055] ? igrab+0x1d/0x50 [16550.480105] igrab+0x1d/0x50 [16550.482993] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16550.488144] delayed_work+0x1d8/0x280 [ceph] [16550.492424] ? process_scheduled_works+0x22a/0x510 [16550.497213] process_scheduled_works+0x2a4/0x510 [16550.501830] worker_thread+0x188/0x340 [16550.505579] ? rescuer_thread+0x310/0x310 [16550.509581] kthread+0xf5/0x130 [16550.512722] ? kthread_complete_and_exit+0x20/0x20 [16550.517513] ret_from_fork+0x31/0x40 [16550.521092] ? kthread_complete_and_exit+0x20/0x20 [16550.525876] ret_from_fork_asm+0x11/0x20 [16550.529801] [16577.745081] watchdog: BUG: soft lockup - CPU#0 stuck for 4325s! [dbench:78683] [16577.752295] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16577.757081] watchdog: BUG: soft lockup - CPU#7 stuck for 13322s! [kworker/7:4:78808] [16577.807737] irq event stamp: 22507720 [16577.807738] hardirqs last enabled at (22507719): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16577.815475] Modules linked in: [16577.819133] hardirqs last disabled at (22507720): [] sysvec_apic_timer_interrupt+0xe/0x90 [16577.829291] ceph [16577.832342] softirqs last enabled at (22507644): [] __do_softirq+0x2be/0x371 [16577.842065] libceph [16577.843989] softirqs last disabled at (22507639): [] irq_exit_rcu+0xb0/0xd0 [16577.852673] dns_resolver fscache [16577.854858] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16577.863369] netfs [16577.866680] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16577.875622] veth nft_chain_nat [16577.877633] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16577.885104] xt_MASQUERADE nf_nat [16577.888241] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16577.893719] nf_conntrack nf_defrag_ipv6 [16577.897029] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16577.915768] nf_defrag_ipv4 nft_compat [16577.919685] [16577.919686] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16577.924902] nf_tables nfnetlink [16577.928648] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16577.930137] bridge stp [16577.937263] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16577.940485] llc xfs [16577.947610] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16577.950054] libcrc32c sunrpc [16577.957179] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16577.959362] intel_rapl_msr intel_rapl_common [16577.966485] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16577.969450] sb_edac x86_pkg_temp_thermal [16577.976591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16577.980943] coretemp kvm_intel [16577.989020] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16577.993024] kvm irqbypass [16577.998762] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16578.001900] i2c_i801 iTCO_wdt [16578.009023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16578.011727] iTCO_vendor_support [16578.018852] Call Trace: [16578.021901] i2c_smbus crct10dif_pclmul [16578.029027] [16578.032249] joydev crc32_pclmul [16578.034695] ? watchdog_timer_fn+0x1b4/0x220 [16578.038524] ipmi_ssif ghash_clmulni_intel [16578.040537] ? __hrtimer_run_queues+0x1d3/0x360 [16578.043762] mei_me mei [16578.048026] ? hrtimer_interrupt+0x104/0x240 [16578.052114] lpc_ich sha512_ssse3 [16578.056639] ? __local_bh_enable+0x50/0x60 [16578.059083] acpi_ipmi mfd_core [16578.063349] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16578.066657] wmi ipmi_si [16578.070760] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16578.073894] ipmi_devintf ipmi_msghandler [16578.079113] [16578.081641] ioatdma acpi_pad [16578.086600] [16578.090604] acpi_power_meter ixgbe [16578.092702] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16578.095664] igb mdio [16578.097766] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16578.101248] crc32c_intel nvme ptp [16578.106552] do_raw_spin_lock+0xb5/0xc0 [16578.108822] nvme_core i2c_algo_bit [16578.113693] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16578.117091] pps_core dca [16578.120922] try_get_cap_refs+0x159/0x880 [ceph] [16578.124404] fuse [16578.129193] __ceph_get_caps+0x4a1/0x760 [ceph] [16578.131807] irq event stamp: 55657678 [16578.136420] ? __file_remove_privs+0x89/0x110 [16578.138340] hardirqs last enabled at (55657677): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16578.142864] ? wake_bit_function+0x70/0x70 [16578.146522] hardirqs last disabled at (55657678): [] sysvec_apic_timer_interrupt+0xe/0x90 [16578.150875] ? ceph_write_iter+0x260/0x770 [ceph] [16578.161031] softirqs last enabled at (55657676): [] __do_softirq+0x2be/0x371 [16578.165121] ceph_write_iter+0x260/0x770 [ceph] [16578.174846] softirqs last disabled at (55657671): [] irq_exit_rcu+0xb0/0xd0 [16578.179546] ? lock_is_held_type+0xaa/0x120 [16578.188228] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16578.192767] vfs_write+0x324/0x4f0 [16578.201279] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16578.201280] Workqueue: events delayed_work [ceph] [16578.205479] ksys_pwrite64+0x65/0xa0 [16578.214852] [16578.214853] RIP: 0010:lock_acquire+0x11/0x2b0 [16578.218248] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16578.225719] Code: fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 <41> 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b [16578.230418] do_syscall_64+0x38/0x80 [16578.233988] RSP: 0018:ffffc90002117d70 EFLAGS: 00000246 [16578.235483] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16578.239829] [16578.239830] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [16578.244960] RIP: 0033:0x7f48f0eeb93b [16578.263698] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [16578.267270] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16578.272485] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [16578.277529] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16578.279020] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [16578.286144] ORIG_RAX: 0000000000000012 [16578.289715] R13: ffff88817f867478 R14: 0000000000000000 R15: 0000000000000000 [16578.296839] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16578.315577] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16578.322701] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16578.327918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16578.335044] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16578.338873] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16578.345998] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16578.353123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16578.361199] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16578.368324] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16578.374067] [16578.381185] Call Trace: [16578.381186] [16578.423445] ? watchdog_timer_fn+0x1b4/0x220 [16578.427711] ? __hrtimer_run_queues+0x1d3/0x360 [16578.432235] ? hrtimer_interrupt+0x104/0x240 [16578.436497] ? __local_bh_enable+0x50/0x60 [16578.440590] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16578.445814] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16578.450777] [16578.452878] [16578.454976] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16578.460283] ? lock_acquire+0x11/0x2b0 [16578.464033] _raw_spin_lock+0x30/0x40 [16578.467688] ? igrab+0x1d/0x50 [16578.470740] igrab+0x1d/0x50 [16578.473635] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16578.478786] delayed_work+0x1d8/0x280 [ceph] [16578.483066] ? process_scheduled_works+0x22a/0x510 [16578.487849] process_scheduled_works+0x2a4/0x510 [16578.492482] worker_thread+0x188/0x340 [16578.496229] ? rescuer_thread+0x310/0x310 [16578.500233] kthread+0xf5/0x130 [16578.503370] ? kthread_complete_and_exit+0x20/0x20 [16578.508155] ret_from_fork+0x31/0x40 [16578.511726] ? kthread_complete_and_exit+0x20/0x20 [16578.516509] ret_from_fork_asm+0x11/0x20 [16578.520433] [16605.745578] watchdog: BUG: soft lockup - CPU#0 stuck for 4351s! [dbench:78683] [16605.752793] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16605.757578] watchdog: BUG: soft lockup - CPU#7 stuck for 13348s! [kworker/7:4:78808] [16605.808257] irq event stamp: 22565886 [16605.815989] Modules linked in: ceph libceph dns_resolver fscache netfs [16605.819647] hardirqs last enabled at (22565885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16605.826172] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [16605.836330] hardirqs last disabled at (22565886): [] sysvec_apic_timer_interrupt+0xe/0x90 [16605.842414] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [16605.852137] softirqs last enabled at (22565860): [] __do_softirq+0x2be/0x371 [16605.852141] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [16605.868206] softirqs last disabled at (22565855): [] irq_exit_rcu+0xb0/0xd0 [16605.875071] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [16605.883583] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16605.889232] kvm irqbypass [16605.898193] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16605.898195] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16605.900898] i2c_i801 iTCO_wdt [16605.908369] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16605.913848] iTCO_vendor_support [16605.916897] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16605.935635] i2c_smbus [16605.938860] [16605.944076] crct10dif_pclmul joydev [16605.946435] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16605.947925] crc32_pclmul ipmi_ssif [16605.951494] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16605.958620] ghash_clmulni_intel mei_me [16605.962103] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16605.969229] mei [16605.973076] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16605.980199] lpc_ich [16605.982038] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16605.989160] sha512_ssse3 acpi_ipmi [16605.991345] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16605.998471] mfd_core wmi [16606.001953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16606.010049] ipmi_si ipmi_devintf [16606.012666] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16606.018403] ipmi_msghandler ioatdma [16606.021713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16606.028837] acpi_pad acpi_power_meter [16606.032409] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16606.039532] ixgbe igb [16606.043276] Call Trace: [16606.043279] [16606.050400] mdio crc32c_intel [16606.052762] ? watchdog_timer_fn+0x1b4/0x220 [16606.055202] nvme ptp [16606.057216] ? __hrtimer_run_queues+0x1d3/0x360 [16606.060265] nvme_core i2c_algo_bit [16606.064530] ? hrtimer_interrupt+0x104/0x240 [16606.066798] pps_core dca [16606.071323] ? __local_bh_enable+0x50/0x60 [16606.074807] fuse [16606.079072] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16606.081687] irq event stamp: 55879156 [16606.085780] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16606.087702] hardirqs last enabled at (55879155): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16606.092922] [16606.096577] hardirqs last disabled at (55879156): [] sysvec_apic_timer_interrupt+0xe/0x90 [16606.101537] [16606.111693] softirqs last enabled at (55879154): [] __do_softirq+0x2be/0x371 [16606.113791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16606.123515] softirqs last disabled at (55879149): [] irq_exit_rcu+0xb0/0xd0 [16606.125615] ? queued_spin_lock_slowpath+0x18/0x2e0 [16606.134296] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16606.139602] do_raw_spin_lock+0xb5/0xc0 [16606.148112] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16606.152984] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16606.162360] Workqueue: events delayed_work [ceph] [16606.166193] try_get_cap_refs+0x159/0x880 [ceph] [16606.173660] [16606.173660] RIP: 0010:lock_acquire+0xf0/0x2b0 [16606.178450] __ceph_get_caps+0x4a1/0x760 [ceph] [16606.183142] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16606.187758] ? __file_remove_privs+0x89/0x110 [16606.189242] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16606.193613] ? wake_bit_function+0x70/0x70 [16606.198135] [16606.216876] ? ceph_write_iter+0x260/0x770 [ceph] [16606.221223] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16606.226441] ceph_write_iter+0x260/0x770 [ceph] [16606.230532] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16606.232026] ? lock_is_held_type+0xaa/0x120 [16606.236721] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16606.243846] vfs_write+0x324/0x4f0 [16606.248369] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16606.255499] ksys_pwrite64+0x65/0xa0 [16606.259671] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16606.266796] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16606.270192] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16606.277317] do_syscall_64+0x38/0x80 [16606.280886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16606.288012] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16606.293140] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16606.301221] RIP: 0033:0x7f48f0eeb93b [16606.304789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16606.310529] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16606.315571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16606.322697] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16606.326271] Call Trace: [16606.326272] [16606.333398] ORIG_RAX: 0000000000000012 [16606.352137] ? watchdog_timer_fn+0x1b4/0x220 [16606.359260] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16606.364478] ? __hrtimer_run_queues+0x1d3/0x360 [16606.366923] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16606.368938] ? hrtimer_interrupt+0x104/0x240 [16606.372764] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16606.377026] ? __local_bh_enable+0x50/0x60 [16606.384152] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16606.388678] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16606.395800] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16606.400064] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16606.407194] [16606.411278] [16606.439984] [16606.442083] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16606.447390] ? lock_acquire+0xf0/0x2b0 [16606.451139] ? igrab+0x43/0x50 [16606.454189] ? lock_release+0xc5/0x260 [16606.457934] _raw_spin_lock+0x30/0x40 [16606.461590] ? igrab+0x1d/0x50 [16606.464640] igrab+0x1d/0x50 [16606.467520] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16606.472670] delayed_work+0x1d8/0x280 [ceph] [16606.476951] ? process_scheduled_works+0x22a/0x510 [16606.481741] process_scheduled_works+0x2a4/0x510 [16606.486357] worker_thread+0x188/0x340 [16606.490106] ? rescuer_thread+0x310/0x310 [16606.494118] kthread+0xf5/0x130 [16606.497278] ? kthread_complete_and_exit+0x20/0x20 [16606.502064] ret_from_fork+0x31/0x40 [16606.505635] ? kthread_complete_and_exit+0x20/0x20 [16606.510419] ret_from_fork_asm+0x11/0x20 [16606.514343] [16623.518408] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16633.746075] watchdog: BUG: soft lockup - CPU#0 stuck for 4377s! [dbench:78683] [16633.753287] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16633.758075] watchdog: BUG: soft lockup - CPU#7 stuck for 13374s! [kworker/7:4:78808] [16633.808728] irq event stamp: 22624010 [16633.808746] hardirqs last enabled at (22624009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16633.816477] Modules linked in: [16633.820136] hardirqs last disabled at (22624010): [] sysvec_apic_timer_interrupt+0xe/0x90 [16633.830292] ceph [16633.833343] softirqs last enabled at (22623952): [] __do_softirq+0x2be/0x371 [16633.843066] libceph [16633.844991] softirqs last disabled at (22623947): [] irq_exit_rcu+0xb0/0xd0 [16633.853675] dns_resolver [16633.855858] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16633.864370] fscache netfs [16633.866988] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16633.875932] veth nft_chain_nat [16633.878634] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16633.886107] xt_MASQUERADE nf_nat [16633.889243] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16633.894722] nf_conntrack nf_defrag_ipv6 [16633.898032] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16633.916769] nf_defrag_ipv4 nft_compat [16633.920688] [16633.920689] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16633.925903] nf_tables [16633.929648] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16633.931138] nfnetlink [16633.938264] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16633.940620] bridge stp [16633.947754] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16633.950110] llc xfs [16633.957234] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16633.959678] libcrc32c [16633.966803] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16633.968986] sunrpc intel_rapl_msr [16633.976110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16633.978472] intel_rapl_common sb_edac [16633.986547] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16633.989943] x86_pkg_temp_thermal [16633.995681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16633.999424] coretemp [16634.006549] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16634.009861] kvm_intel [16634.016985] Call Trace: [16634.019254] kvm irqbypass [16634.026380] [16634.028736] i2c_i801 iTCO_wdt [16634.031198] ? watchdog_timer_fn+0x1b4/0x220 [16634.033901] iTCO_vendor_support i2c_smbus [16634.035913] ? __hrtimer_run_queues+0x1d3/0x360 [16634.038964] crct10dif_pclmul joydev [16634.043231] ? hrtimer_interrupt+0x104/0x240 [16634.047319] crc32_pclmul ipmi_ssif [16634.051842] ? __local_bh_enable+0x50/0x60 [16634.055412] ghash_clmulni_intel mei_me [16634.059678] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16634.063162] mei lpc_ich [16634.067252] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16634.071081] sha512_ssse3 acpi_ipmi [16634.076300] [16634.078829] mfd_core wmi [16634.083789] [16634.083791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16634.087272] ipmi_si ipmi_devintf [16634.089373] ? queued_spin_lock_slowpath+0x16/0x2e0 [16634.091987] ipmi_msghandler ioatdma [16634.094086] do_raw_spin_lock+0xb5/0xc0 [16634.099388] acpi_pad acpi_power_meter [16634.102701] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16634.107570] ixgbe igb [16634.111144] try_get_cap_refs+0x159/0x880 [ceph] [16634.114971] mdio crc32c_intel [16634.118721] __ceph_get_caps+0x4a1/0x760 [ceph] [16634.123498] nvme ptp [16634.125861] ? __file_remove_privs+0x89/0x110 [16634.130475] nvme_core [16634.133527] ? wake_bit_function+0x70/0x70 [16634.138051] i2c_algo_bit pps_core [16634.140325] ? ceph_write_iter+0x260/0x770 [ceph] [16634.144673] dca fuse [16634.147030] ceph_write_iter+0x260/0x770 [ceph] [16634.151122] [16634.154521] ? lock_is_held_type+0xaa/0x120 [16634.159216] irq event stamp: 56100156 [16634.161489] vfs_write+0x324/0x4f0 [16634.166010] hardirqs last enabled at (56100155): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16634.167504] ksys_pwrite64+0x65/0xa0 [16634.171677] hardirqs last disabled at (56100156): [] sysvec_apic_timer_interrupt+0xe/0x90 [16634.175336] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16634.178733] softirqs last enabled at (56100154): [] __do_softirq+0x2be/0x371 [16634.188910] do_syscall_64+0x38/0x80 [16634.192478] softirqs last disabled at (56100149): [] irq_exit_rcu+0xb0/0xd0 [16634.202205] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16634.207333] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16634.216017] RIP: 0033:0x7f48f0eeb93b [16634.219589] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16634.228100] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16634.233143] Workqueue: events delayed_work [ceph] [16634.242522] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16634.246091] [16634.246092] RIP: 0010:ceph_check_delayed_caps+0xc0/0x270 [ceph] [16634.253563] ORIG_RAX: 0000000000000012 [16634.272300] Code: 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 <48> 8b 85 78 04 00 00 49 39 c5 75 89 31 c0 48 8b 3c 24 48 89 44 24 [16634.273088] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16634.276996] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16634.282215] RSP: 0018:ffffc90002117db8 EFLAGS: 00000246 [16634.283706] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16634.289617] [16634.293446] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16634.312186] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [16634.319136] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16634.326260] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [16634.331478] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16634.338601] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [16634.340098] [16634.347216] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [16634.392163] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16634.399287] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16634.407364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16634.413102] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16634.420227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16634.427349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16634.434481] Call Trace: [16634.436927] [16634.438937] ? watchdog_timer_fn+0x1b4/0x220 [16634.443201] ? __hrtimer_run_queues+0x1d3/0x360 [16634.447729] ? hrtimer_interrupt+0x104/0x240 [16634.451998] ? __local_bh_enable+0x50/0x60 [16634.456091] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16634.461308] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16634.466265] [16634.468362] [16634.470477] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16634.475784] ? ceph_check_delayed_caps+0xc0/0x270 [ceph] [16634.481103] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [16634.486418] delayed_work+0x1d8/0x280 [ceph] [16634.490699] ? process_scheduled_works+0x22a/0x510 [16634.495489] process_scheduled_works+0x2a4/0x510 [16634.500104] worker_thread+0x188/0x340 [16634.503854] ? rescuer_thread+0x310/0x310 [16634.507856] kthread+0xf5/0x130 [16634.510994] ? kthread_complete_and_exit+0x20/0x20 [16634.515780] ret_from_fork+0x31/0x40 [16634.519357] ? kthread_complete_and_exit+0x20/0x20 [16634.524141] ret_from_fork_asm+0x11/0x20 [16634.528083] [16644.510258] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16655.262363] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16661.746574] watchdog: BUG: soft lockup - CPU#0 stuck for 4403s! [dbench:78683] [16661.753794] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16661.758574] watchdog: BUG: soft lockup - CPU#7 stuck for 13400s! [kworker/7:4:78808] [16661.809247] irq event stamp: 22682310 [16661.816982] Modules linked in: ceph libceph dns_resolver fscache netfs veth [16661.820639] hardirqs last enabled at (22682309): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16661.827598] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [16661.837756] hardirqs last disabled at (22682310): [] sysvec_apic_timer_interrupt+0xe/0x90 [16661.843407] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [16661.853132] softirqs last enabled at (22682252): [] __do_softirq+0x2be/0x371 [16661.860870] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [16661.869555] softirqs last disabled at (22682247): [] irq_exit_rcu+0xb0/0xd0 [16661.876775] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [16661.885286] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16661.891455] i2c_i801 iTCO_wdt [16661.900401] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16661.900403] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [16661.903450] iTCO_vendor_support i2c_smbus [16661.910924] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [16661.916400] crct10dif_pclmul [16661.920489] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16661.939228] joydev crc32_pclmul [16661.942209] [16661.942210] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16661.947427] ipmi_ssif ghash_clmulni_intel [16661.950650] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16661.952140] mei_me mei [16661.959264] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16661.963355] lpc_ich sha512_ssse3 [16661.970479] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16661.972923] acpi_ipmi mfd_core [16661.980049] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16661.983358] wmi [16661.990482] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16661.993621] ipmi_si [16662.000745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16662.002582] ipmi_devintf ipmi_msghandler [16662.010659] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16662.012844] ioatdma [16662.018582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16662.022585] acpi_pad acpi_power_meter [16662.029709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16662.031892] ixgbe igb [16662.039016] Call Trace: [16662.039018] [16662.042760] mdio crc32c_intel [16662.049887] ? watchdog_timer_fn+0x1b4/0x220 [16662.052252] nvme ptp [16662.054696] ? __hrtimer_run_queues+0x1d3/0x360 [16662.056707] nvme_core i2c_algo_bit [16662.059760] ? hrtimer_interrupt+0x104/0x240 [16662.064021] pps_core dca [16662.066291] ? __local_bh_enable+0x50/0x60 [16662.070815] fuse [16662.074302] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16662.078564] irq event stamp: 56321088 [16662.081183] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16662.085273] hardirqs last enabled at (56321087): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16662.087196] [16662.092413] hardirqs last disabled at (56321088): [] sysvec_apic_timer_interrupt+0xe/0x90 [16662.096072] [16662.101029] softirqs last enabled at (56321086): [] __do_softirq+0x2be/0x371 [16662.111186] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16662.113282] softirqs last disabled at (56321081): [] irq_exit_rcu+0xb0/0xd0 [16662.123012] ? queued_spin_lock_slowpath+0x2b/0x2e0 [16662.125104] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16662.133792] do_raw_spin_lock+0xb5/0xc0 [16662.139094] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16662.147606] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16662.152476] Workqueue: events delayed_work [ceph] [16662.161855] try_get_cap_refs+0x159/0x880 [ceph] [16662.165682] [16662.165683] RIP: 0010:lock_acquire+0xf0/0x2b0 [16662.173160] __ceph_get_caps+0x4a1/0x760 [ceph] [16662.177939] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16662.182639] ? __file_remove_privs+0x89/0x110 [16662.187254] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16662.188746] ? wake_bit_function+0x70/0x70 [16662.193095] [16662.193096] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16662.197623] ? ceph_write_iter+0x260/0x770 [ceph] [16662.216358] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16662.220709] ceph_write_iter+0x260/0x770 [ceph] [16662.225927] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16662.230020] ? lock_is_held_type+0xaa/0x120 [16662.231506] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16662.238635] vfs_write+0x324/0x4f0 [16662.243330] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16662.250459] ksys_pwrite64+0x65/0xa0 [16662.254978] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16662.262101] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16662.266280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16662.273405] do_syscall_64+0x38/0x80 [16662.276800] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16662.283924] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16662.287496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16662.295573] RIP: 0033:0x7f48f0eeb93b [16662.300702] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16662.306442] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16662.310011] Call Trace: [16662.310012] [16662.317135] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16662.322181] ? watchdog_timer_fn+0x1b4/0x220 [16662.329305] ORIG_RAX: 0000000000000012 [16662.332877] ? __hrtimer_run_queues+0x1d3/0x360 [16662.339998] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16662.358739] ? hrtimer_interrupt+0x104/0x240 [16662.361180] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16662.363190] ? __local_bh_enable+0x50/0x60 [16662.368409] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16662.372676] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16662.376503] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16662.381028] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16662.388152] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16662.392417] [16662.399545] [16662.403630] [16662.441567] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16662.446873] ? lock_acquire+0xf0/0x2b0 [16662.450623] ? igrab+0x43/0x50 [16662.453674] ? lock_release+0xc5/0x260 [16662.457418] _raw_spin_lock+0x30/0x40 [16662.461077] ? igrab+0x1d/0x50 [16662.464126] igrab+0x1d/0x50 [16662.467006] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16662.472154] delayed_work+0x1d8/0x280 [ceph] [16662.476436] ? process_scheduled_works+0x22a/0x510 [16662.481225] process_scheduled_works+0x2a4/0x510 [16662.485840] worker_thread+0x188/0x340 [16662.489589] ? rescuer_thread+0x310/0x310 [16662.493592] kthread+0xf5/0x130 [16662.496733] ? kthread_complete_and_exit+0x20/0x20 [16662.501524] ret_from_fork+0x31/0x40 [16662.505104] ? kthread_complete_and_exit+0x20/0x20 [16662.509886] ret_from_fork_asm+0x11/0x20 [16662.513820] [16665.502502] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16676.254856] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16686.494822] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16689.747072] watchdog: BUG: soft lockup - CPU#0 stuck for 4429s! [dbench:78683] [16689.754289] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16689.759072] watchdog: BUG: soft lockup - CPU#7 stuck for 13426s! [kworker/7:4:78808] [16689.809729] irq event stamp: 22740420 [16689.817476] Modules linked in: ceph libceph [16689.821135] hardirqs last enabled at (22740419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16689.825313] dns_resolver fscache netfs [16689.835468] hardirqs last disabled at (22740420): [] sysvec_apic_timer_interrupt+0xe/0x90 [16689.839301] veth nft_chain_nat xt_MASQUERADE nf_nat [16689.849024] softirqs last enabled at (22740362): [] __do_softirq+0x2be/0x371 [16689.853983] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [16689.862665] softirqs last disabled at (22740357): [] irq_exit_rcu+0xb0/0xd0 [16689.862666] nft_compat [16689.867883] nf_tables nfnetlink [16689.876395] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16689.878839] bridge stp [16689.882063] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16689.891006] llc xfs [16689.893469] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16689.900939] libcrc32c sunrpc [16689.903122] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16689.908601] intel_rapl_msr intel_rapl_common [16689.911564] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16689.930303] sb_edac x86_pkg_temp_thermal [16689.934652] [16689.939871] coretemp kvm_intel [16689.943874] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16689.945365] kvm irqbypass [16689.948503] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16689.955625] i2c_i801 iTCO_wdt [16689.958330] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16689.965469] iTCO_vendor_support i2c_smbus [16689.968522] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16689.975648] crct10dif_pclmul joydev [16689.979748] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16689.986870] crc32_pclmul ipmi_ssif [16689.990441] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16689.997583] ghash_clmulni_intel mei_me [16690.001066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16690.009144] mei lpc_ich [16690.012975] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16690.018713] sha512_ssse3 acpi_ipmi [16690.021242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16690.028367] mfd_core wmi [16690.031852] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16690.038975] ipmi_si ipmi_devintf [16690.041592] Call Trace: [16690.041593] [16690.048716] ipmi_msghandler ioatdma [16690.052029] ? watchdog_timer_fn+0x1b4/0x220 [16690.054471] acpi_pad acpi_power_meter [16690.056483] ? __hrtimer_run_queues+0x1d3/0x360 [16690.060052] ixgbe igb [16690.064321] ? hrtimer_interrupt+0x104/0x240 [16690.068061] mdio crc32c_intel [16690.072585] ? __local_bh_enable+0x50/0x60 [16690.074943] nvme ptp [16690.079210] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16690.082257] nvme_core i2c_algo_bit [16690.086348] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16690.088620] pps_core [16690.093837] [16690.097321] dca fuse [16690.102279] [16690.104548] [16690.104549] irq event stamp: 56542112 [16690.106647] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16690.108917] hardirqs last enabled at (56542111): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16690.111018] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16690.112505] hardirqs last disabled at (56542112): [] sysvec_apic_timer_interrupt+0xe/0x90 [16690.116165] do_raw_spin_lock+0xb5/0xc0 [16690.121472] softirqs last enabled at (56542110): [] __do_softirq+0x2be/0x371 [16690.131632] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16690.136504] softirqs last disabled at (56542105): [] irq_exit_rcu+0xb0/0xd0 [16690.146231] try_get_cap_refs+0x159/0x880 [ceph] [16690.150058] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16690.158748] __ceph_get_caps+0x4a1/0x760 [ceph] [16690.163545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16690.172058] ? __file_remove_privs+0x89/0x110 [16690.176666] Workqueue: events delayed_work [ceph] [16690.186044] ? wake_bit_function+0x70/0x70 [16690.190566] [16690.190567] RIP: 0010:lock_acquire+0xf0/0x2b0 [16690.198039] ? ceph_write_iter+0x260/0x770 [ceph] [16690.202388] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16690.207086] ceph_write_iter+0x260/0x770 [ceph] [16690.211178] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16690.212672] ? lock_is_held_type+0xaa/0x120 [16690.217019] [16690.221719] vfs_write+0x324/0x4f0 [16690.240470] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16690.244998] ksys_pwrite64+0x65/0xa0 [16690.250211] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16690.254390] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16690.255879] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16690.259280] do_syscall_64+0x38/0x80 [16690.266401] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16690.269973] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16690.277096] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16690.282228] RIP: 0033:0x7f48f0eeb93b [16690.289352] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16690.292923] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16690.300047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16690.305091] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16690.312213] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16690.315784] ORIG_RAX: 0000000000000012 [16690.323862] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16690.342600] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16690.348338] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16690.353554] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16690.360679] Call Trace: [16690.360681] [16690.364510] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16690.371636] ? watchdog_timer_fn+0x1b4/0x220 [16690.378774] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16690.385900] ? __hrtimer_run_queues+0x1d3/0x360 [16690.393025] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16690.395480] ? hrtimer_interrupt+0x104/0x240 [16690.397493] [16690.404613] ? __local_bh_enable+0x50/0x60 [16690.438191] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16690.443415] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16690.448371] [16690.450479] [16690.452575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16690.457883] ? lock_acquire+0xf0/0x2b0 [16690.461633] ? igrab+0x43/0x50 [16690.464683] ? lock_release+0xc5/0x260 [16690.468428] _raw_spin_lock+0x30/0x40 [16690.472092] ? igrab+0x1d/0x50 [16690.475144] igrab+0x1d/0x50 [16690.478021] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16690.483172] delayed_work+0x1d8/0x280 [ceph] [16690.487453] ? process_scheduled_works+0x22a/0x510 [16690.492260] process_scheduled_works+0x2a4/0x510 [16690.496876] worker_thread+0x188/0x340 [16690.500624] ? rescuer_thread+0x310/0x310 [16690.504626] kthread+0xf5/0x130 [16690.507765] ? kthread_complete_and_exit+0x20/0x20 [16690.512549] ret_from_fork+0x31/0x40 [16690.516120] ? kthread_complete_and_exit+0x20/0x20 [16690.520904] ret_from_fork_asm+0x11/0x20 [16690.524827] [16692.747492] rcu: INFO: rcu_preempt self-detected stall on CPU [16692.753254] rcu: 7-....: (4620820 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/4619610 fqs=2304792 [16692.763680] rcu: (t=4756053 jiffies g=1309609 q=208231 ncpus=8) [16692.769678] Sending NMI from CPU 7 to CPUs 0: [16692.774028] NMI backtrace for cpu 0 [16692.774030] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16692.774032] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16692.774032] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16692.774036] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16692.774037] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16692.774039] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16692.774039] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16692.774040] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16692.774041] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16692.774042] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16692.774043] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16692.774044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16692.774045] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16692.774046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16692.774046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16692.774047] Call Trace: [16692.774048] [16692.774049] ? nmi_cpu_backtrace+0xad/0x120 [16692.774053] ? nmi_cpu_backtrace_handler+0x11/0x20 [16692.774054] ? nmi_handle+0xd4/0x260 [16692.774059] ? default_do_nmi+0x49/0xf0 [16692.774060] ? exc_nmi+0x11e/0x150 [16692.774062] ? end_repeat_nmi+0x16/0x67 [16692.774066] ? queued_spin_lock_slowpath+0x16/0x2e0 [16692.774069] ? queued_spin_lock_slowpath+0x16/0x2e0 [16692.774072] ? queued_spin_lock_slowpath+0x16/0x2e0 [16692.774074] [16692.774074] [16692.774075] do_raw_spin_lock+0xb5/0xc0 [16692.774077] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16692.774093] try_get_cap_refs+0x159/0x880 [ceph] [16692.774110] __ceph_get_caps+0x4a1/0x760 [ceph] [16692.774127] ? __file_remove_privs+0x89/0x110 [16692.774129] ? wake_bit_function+0x70/0x70 [16692.774132] ? ceph_write_iter+0x260/0x770 [ceph] [16692.774144] ceph_write_iter+0x260/0x770 [ceph] [16692.774158] ? lock_is_held_type+0xaa/0x120 [16692.774161] vfs_write+0x324/0x4f0 [16692.774165] ksys_pwrite64+0x65/0xa0 [16692.774167] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16692.774170] do_syscall_64+0x38/0x80 [16692.774173] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16692.774175] RIP: 0033:0x7f48f0eeb93b [16692.774176] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16692.774177] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [16692.774179] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16692.774179] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16692.774180] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16692.774181] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16692.774181] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16692.774184] [16692.775026] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16693.078840] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16693.086310] Workqueue: events delayed_work [ceph] [16693.091025] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [16693.095635] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [16693.114372] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [16693.119589] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [16693.126715] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [16693.133838] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [16693.140962] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [16693.148087] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16693.155210] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16693.163289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16693.169025] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16693.176151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16693.183274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16693.190399] Call Trace: [16693.192842] [16693.194854] ? rcu_dump_cpu_stacks+0xd1/0x180 [16693.199213] ? rcu_sched_clock_irq+0x633/0x1390 [16693.203737] ? lock_release+0xc5/0x260 [16693.207481] ? lock_is_held_type+0xaa/0x120 [16693.211659] ? lock_is_held_type+0xaa/0x120 [16693.215837] ? tick_sched_do_timer+0xa0/0xa0 [16693.220108] ? rcu_read_lock_sched_held+0x56/0x90 [16693.224806] ? tick_sched_do_timer+0xa0/0xa0 [16693.229071] ? update_process_times+0x5f/0x90 [16693.233420] ? tick_sched_handle.isra.16+0x1e/0x50 [16693.238206] ? tick_sched_timer+0x6f/0x80 [16693.242211] ? __hrtimer_run_queues+0x1d3/0x360 [16693.246737] ? hrtimer_interrupt+0x104/0x240 [16693.251006] ? __local_bh_enable+0x50/0x60 [16693.255100] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16693.260324] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16693.265281] [16693.267377] [16693.269474] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16693.274785] ? do_raw_spin_lock+0x42/0xc0 [16693.278793] igrab+0x1d/0x50 [16693.281669] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16693.286820] delayed_work+0x1d8/0x280 [ceph] [16693.291102] ? process_scheduled_works+0x22a/0x510 [16693.295890] process_scheduled_works+0x2a4/0x510 [16693.300508] worker_thread+0x188/0x340 [16693.304255] ? rescuer_thread+0x310/0x310 [16693.308260] kthread+0xf5/0x130 [16693.311396] ? kthread_complete_and_exit+0x20/0x20 [16693.316180] ret_from_fork+0x31/0x40 [16693.319750] ? kthread_complete_and_exit+0x20/0x20 [16693.324536] ret_from_fork_asm+0x11/0x20 [16693.328460] [16697.247005] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16707.487303] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16717.747571] watchdog: BUG: soft lockup - CPU#0 stuck for 4455s! [dbench:78683] [16717.754784] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16717.759571] watchdog: BUG: soft lockup - CPU#7 stuck for 13452s! [kworker/7:4:78808] [16717.810247] irq event stamp: 22798566 [16717.817982] Modules linked in: ceph libceph dns_resolver fscache netfs [16717.821640] hardirqs last enabled at (22798565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16717.828165] veth nft_chain_nat xt_MASQUERADE nf_nat [16717.838323] hardirqs last disabled at (22798566): [] sysvec_apic_timer_interrupt+0xe/0x90 [16717.843280] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [16717.853005] softirqs last enabled at (22798524): [] __do_softirq+0x2be/0x371 [16717.853009] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [16717.869601] softirqs last disabled at (22798515): [] irq_exit_rcu+0xb0/0xd0 [16717.875512] intel_rapl_common sb_edac x86_pkg_temp_thermal [16717.884025] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16717.889587] coretemp kvm_intel [16717.898533] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16717.898534] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16717.901668] kvm [16717.909142] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16717.914617] irqbypass i2c_i801 [16717.916454] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16717.935192] iTCO_wdt iTCO_vendor_support [16717.938328] [16717.938329] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16717.943546] i2c_smbus crct10dif_pclmul [16717.947552] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16717.949041] joydev crc32_pclmul [16717.956165] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16717.959997] ipmi_ssif [16717.967120] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16717.970345] ghash_clmulni_intel mei_me [16717.977469] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16717.979827] mei [16717.986966] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16717.990799] lpc_ich sha512_ssse3 [16717.997922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16717.999759] acpi_ipmi mfd_core [16718.007838] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16718.011148] wmi ipmi_si [16718.016887] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16718.020023] ipmi_devintf [16718.027147] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16718.029678] ipmi_msghandler ioatdma [16718.036801] Call Trace: [16718.036803] [16718.039420] acpi_pad acpi_power_meter [16718.046546] ? watchdog_timer_fn+0x1b4/0x220 [16718.050114] ixgbe igb [16718.052559] ? __hrtimer_run_queues+0x1d3/0x360 [16718.054569] mdio crc32c_intel [16718.058317] ? hrtimer_interrupt+0x104/0x240 [16718.062578] nvme ptp [16718.064934] ? __local_bh_enable+0x50/0x60 [16718.069458] nvme_core i2c_algo_bit [16718.072512] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16718.076772] pps_core dca [16718.079044] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16718.083134] fuse [16718.086619] [16718.091836] irq event stamp: 56758336 [16718.094455] [16718.099410] hardirqs last enabled at (56758335): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16718.101337] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16718.103432] hardirqs last disabled at (56758336): [] sysvec_apic_timer_interrupt+0xe/0x90 [16718.107094] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16718.109188] softirqs last enabled at (56758334): [] __do_softirq+0x2be/0x371 [16718.119347] do_raw_spin_lock+0xb5/0xc0 [16718.124650] softirqs last disabled at (56758329): [] irq_exit_rcu+0xb0/0xd0 [16718.134375] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16718.139243] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16718.147948] try_get_cap_refs+0x159/0x880 [ceph] [16718.151792] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16718.160310] __ceph_get_caps+0x4a1/0x760 [ceph] [16718.165088] Workqueue: events delayed_work [ceph] [16718.174471] ? __file_remove_privs+0x89/0x110 [16718.179076] [16718.179077] RIP: 0010:lock_acquire+0xf0/0x2b0 [16718.186547] ? wake_bit_function+0x70/0x70 [16718.191070] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16718.195772] ? ceph_write_iter+0x260/0x770 [ceph] [16718.200119] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16718.201611] ceph_write_iter+0x260/0x770 [ceph] [16718.205970] [16718.205971] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16718.210065] ? lock_is_held_type+0xaa/0x120 [16718.228799] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16718.233498] vfs_write+0x324/0x4f0 [16718.238713] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16718.239480] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16718.243259] ksys_pwrite64+0x65/0xa0 [16718.244744] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16718.251869] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16718.256045] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16718.263173] do_syscall_64+0x38/0x80 [16718.266568] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16718.273693] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16718.280643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16718.284213] RIP: 0033:0x7f48f0eeb93b [16718.291339] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16718.296470] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16718.303593] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16718.307162] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16718.315256] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16718.320303] ORIG_RAX: 0000000000000012 [16718.326040] Call Trace: [16718.329612] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16718.336735] [16718.355472] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16718.362599] ? watchdog_timer_fn+0x1b4/0x220 [16718.367813] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16718.374940] ? __hrtimer_run_queues+0x1d3/0x360 [16718.378768] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16718.381217] ? hrtimer_interrupt+0x104/0x240 [16718.388336] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16718.390347] ? __local_bh_enable+0x50/0x60 [16718.397478] [16718.401740] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16718.443390] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16718.448347] [16718.450443] [16718.452543] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16718.457859] ? lock_acquire+0xf0/0x2b0 [16718.461609] ? igrab+0x43/0x50 [16718.464658] ? lock_release+0xc5/0x260 [16718.468404] _raw_spin_lock+0x30/0x40 [16718.472069] ? igrab+0x1d/0x50 [16718.475120] igrab+0x1d/0x50 [16718.477997] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16718.483146] delayed_work+0x1d8/0x280 [ceph] [16718.487427] ? process_scheduled_works+0x22a/0x510 [16718.492218] process_scheduled_works+0x2a4/0x510 [16718.496835] worker_thread+0x188/0x340 [16718.500582] ? rescuer_thread+0x310/0x310 [16718.504585] kthread+0xf5/0x130 [16718.507725] ? kthread_complete_and_exit+0x20/0x20 [16718.512508] ret_from_fork+0x31/0x40 [16718.516087] ? kthread_complete_and_exit+0x20/0x20 [16718.520871] ret_from_fork_asm+0x11/0x20 [16718.524795] [16728.479591] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16739.231757] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16745.748069] watchdog: BUG: soft lockup - CPU#0 stuck for 4481s! [dbench:78683] [16745.755281] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16745.760069] watchdog: BUG: soft lockup - CPU#7 stuck for 13478s! [kworker/7:4:78808] [16745.810721] irq event stamp: 22856700 [16745.810722] hardirqs last enabled at (22856699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16745.818469] Modules linked in: [16745.822126] hardirqs last disabled at (22856700): [] sysvec_apic_timer_interrupt+0xe/0x90 [16745.832285] ceph [16745.835334] softirqs last enabled at (22856652): [] __do_softirq+0x2be/0x371 [16745.845058] libceph dns_resolver [16745.846982] softirqs last disabled at (22856647): [] irq_exit_rcu+0xb0/0xd0 [16745.855667] fscache [16745.858977] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16745.867489] netfs veth [16745.869673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16745.878617] nft_chain_nat xt_MASQUERADE [16745.881061] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16745.888533] nf_nat nf_conntrack [16745.892464] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16745.897943] nf_defrag_ipv6 nf_defrag_ipv4 [16745.901169] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16745.919906] nft_compat nf_tables [16745.923996] [16745.929214] nfnetlink bridge [16745.932524] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16745.934014] stp llc [16745.936979] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16745.944102] xfs [16745.946287] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16745.953412] libcrc32c sunrpc [16745.955249] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16745.962373] intel_rapl_msr intel_rapl_common [16745.965336] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16745.972467] sb_edac x86_pkg_temp_thermal [16745.976822] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16745.983944] coretemp kvm_intel [16745.987950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16745.996027] kvm irqbypass [16745.999163] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16746.004902] i2c_i801 iTCO_wdt [16746.007605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16746.014730] iTCO_vendor_support i2c_smbus [16746.017781] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16746.024905] crct10dif_pclmul joydev [16746.028994] Call Trace: [16746.028995] [16746.036118] crc32_pclmul ipmi_ssif [16746.039691] ? watchdog_timer_fn+0x1b4/0x220 [16746.042134] ghash_clmulni_intel mei_me [16746.044145] ? __hrtimer_run_queues+0x1d3/0x360 [16746.047627] mei lpc_ich [16746.051895] ? hrtimer_interrupt+0x104/0x240 [16746.055724] sha512_ssse3 acpi_ipmi [16746.060264] ? __local_bh_enable+0x50/0x60 [16746.062796] mfd_core wmi [16746.067062] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16746.070543] ipmi_si ipmi_devintf [16746.074635] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16746.077252] ipmi_msghandler ioatdma [16746.082468] [16746.085780] acpi_pad acpi_power_meter [16746.090750] [16746.090751] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16746.094317] ixgbe igb [16746.096418] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16746.100159] mdio crc32c_intel [16746.102257] do_raw_spin_lock+0xb5/0xc0 [16746.107559] nvme ptp nvme_core [16746.109918] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16746.114788] i2c_algo_bit pps_core [16746.117840] try_get_cap_refs+0x159/0x880 [ceph] [16746.121668] dca fuse [16746.124812] __ceph_get_caps+0x4a1/0x760 [ceph] [16746.129590] [16746.129591] irq event stamp: 56979230 [16746.132991] ? __file_remove_privs+0x89/0x110 [16746.137600] hardirqs last enabled at (56979229): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16746.139870] ? wake_bit_function+0x70/0x70 [16746.144393] hardirqs last disabled at (56979230): [] sysvec_apic_timer_interrupt+0xe/0x90 [16746.145886] ? ceph_write_iter+0x260/0x770 [ceph] [16746.149542] softirqs last enabled at (56979228): [] __do_softirq+0x2be/0x371 [16746.153892] ceph_write_iter+0x260/0x770 [ceph] [16746.164049] softirqs last disabled at (56979223): [] irq_exit_rcu+0xb0/0xd0 [16746.168143] ? lock_is_held_type+0xaa/0x120 [16746.177865] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16746.182566] vfs_write+0x324/0x4f0 [16746.191248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16746.195775] ksys_pwrite64+0x65/0xa0 [16746.204282] Workqueue: events delayed_work [ceph] [16746.208473] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16746.217853] [16746.217854] RIP: 0010:lock_acquire+0xf0/0x2b0 [16746.221254] do_syscall_64+0x38/0x80 [16746.228722] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16746.232294] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16746.236990] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16746.242121] RIP: 0033:0x7f48f0eeb93b [16746.243613] [16746.243613] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16746.247962] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16746.251532] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16746.270270] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16746.275316] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16746.280533] ORIG_RAX: 0000000000000012 [16746.284104] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16746.285593] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16746.292718] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16746.311472] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16746.318597] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16746.323814] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16746.330939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16746.334773] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16746.341894] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16746.349017] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16746.356142] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16746.363270] [16746.371344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16746.422037] Call Trace: [16746.424480] [16746.426490] ? watchdog_timer_fn+0x1b4/0x220 [16746.430755] ? __hrtimer_run_queues+0x1d3/0x360 [16746.435283] ? hrtimer_interrupt+0x104/0x240 [16746.439551] ? __local_bh_enable+0x50/0x60 [16746.443646] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16746.448869] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16746.453825] [16746.455924] [16746.458021] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16746.463330] ? lock_acquire+0xf0/0x2b0 [16746.467079] ? igrab+0x43/0x50 [16746.470128] ? lock_release+0xc5/0x260 [16746.473876] _raw_spin_lock+0x30/0x40 [16746.477537] ? igrab+0x1d/0x50 [16746.480589] igrab+0x1d/0x50 [16746.483477] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16746.488626] delayed_work+0x1d8/0x280 [ceph] [16746.492906] ? process_scheduled_works+0x22a/0x510 [16746.497689] process_scheduled_works+0x2a4/0x510 [16746.502304] worker_thread+0x188/0x340 [16746.506054] ? rescuer_thread+0x310/0x310 [16746.510055] kthread+0xf5/0x130 [16746.513194] ? kthread_complete_and_exit+0x20/0x20 [16746.517978] ret_from_fork+0x31/0x40 [16746.521549] ? kthread_complete_and_exit+0x20/0x20 [16746.526332] ret_from_fork_asm+0x11/0x20 [16746.530255] [16749.472097] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16760.224159] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16770.464388] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16773.748568] watchdog: BUG: soft lockup - CPU#0 stuck for 4507s! [dbench:78683] [16773.755785] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16773.760568] watchdog: BUG: soft lockup - CPU#7 stuck for 13504s! [kworker/7:4:78808] [16773.811241] irq event stamp: 22914942 [16773.818974] Modules linked in: ceph libceph dns_resolver fscache [16773.822632] hardirqs last enabled at (22914941): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16773.822637] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [16773.838795] hardirqs last disabled at (22914942): [] sysvec_apic_timer_interrupt+0xe/0x90 [16773.845400] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [16773.855123] softirqs last enabled at (22914894): [] __do_softirq+0x2be/0x371 [16773.862862] llc xfs libcrc32c sunrpc intel_rapl_msr [16773.871548] softirqs last disabled at (22914889): [] irq_exit_rcu+0xb0/0xd0 [16773.876505] intel_rapl_common sb_edac x86_pkg_temp_thermal [16773.885015] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16773.890580] coretemp kvm_intel [16773.899525] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16773.899527] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16773.902661] kvm [16773.910132] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16773.915609] irqbypass i2c_i801 [16773.917447] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16773.936184] iTCO_wdt iTCO_vendor_support [16773.939323] [16773.944538] i2c_smbus crct10dif_pclmul [16773.948542] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16773.950034] joydev crc32_pclmul [16773.953863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16773.960989] ipmi_ssif ghash_clmulni_intel [16773.964211] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16773.971336] mei_me mei [16773.975428] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16773.982550] lpc_ich sha512_ssse3 [16773.984996] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16773.992120] acpi_ipmi mfd_core [16773.995430] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16774.002554] wmi ipmi_si [16774.005692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16774.013769] ipmi_devintf ipmi_msghandler [16774.016299] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16774.022038] ioatdma acpi_pad [16774.026043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16774.033165] acpi_power_meter ixgbe [16774.036129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16774.043254] igb mdio [16774.046737] Call Trace: [16774.046740] [16774.053863] crc32c_intel [16774.056135] ? watchdog_timer_fn+0x1b4/0x220 [16774.058577] nvme ptp [16774.060590] ? __hrtimer_run_queues+0x1d3/0x360 [16774.063205] nvme_core i2c_algo_bit [16774.067471] ? hrtimer_interrupt+0x104/0x240 [16774.069741] pps_core dca [16774.074264] ? __local_bh_enable+0x50/0x60 [16774.077749] fuse [16774.082016] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16774.084629] irq event stamp: 57200184 [16774.088722] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16774.090644] hardirqs last enabled at (57200183): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16774.095862] [16774.099519] hardirqs last disabled at (57200184): [] sysvec_apic_timer_interrupt+0xe/0x90 [16774.104477] [16774.114634] softirqs last enabled at (57200182): [] __do_softirq+0x2be/0x371 [16774.116732] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16774.126455] softirqs last disabled at (57200177): [] irq_exit_rcu+0xb0/0xd0 [16774.128556] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16774.137248] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16774.142553] do_raw_spin_lock+0xb5/0xc0 [16774.151059] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16774.155934] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16774.165309] Workqueue: events delayed_work [ceph] [16774.169143] try_get_cap_refs+0x159/0x880 [ceph] [16774.176611] [16774.181400] __ceph_get_caps+0x4a1/0x760 [ceph] [16774.186093] RIP: 0010:lock_release+0x14e/0x260 [16774.190708] ? __file_remove_privs+0x89/0x110 [16774.192194] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [16774.196719] ? wake_bit_function+0x70/0x70 [16774.201156] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [16774.205510] ? ceph_write_iter+0x260/0x770 [ceph] [16774.224256] [16774.228345] ceph_write_iter+0x260/0x770 [ceph] [16774.233561] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [16774.238262] ? lock_is_held_type+0xaa/0x120 [16774.239749] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [16774.244274] vfs_write+0x324/0x4f0 [16774.251396] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [16774.255580] ksys_pwrite64+0x65/0xa0 [16774.262699] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [16774.266097] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16774.273220] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [16774.276792] do_syscall_64+0x38/0x80 [16774.283916] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16774.289047] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16774.296169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16774.299741] RIP: 0033:0x7f48f0eeb93b [16774.307818] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16774.312863] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16774.318600] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16774.322171] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16774.329294] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16774.348032] ORIG_RAX: 0000000000000012 [16774.355155] Call Trace: [16774.360372] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16774.367500] [16774.371327] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16774.373773] ? watchdog_timer_fn+0x1b4/0x220 [16774.380896] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16774.382908] ? __hrtimer_run_queues+0x1d3/0x360 [16774.390031] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16774.394299] ? hrtimer_interrupt+0x104/0x240 [16774.401420] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16774.405944] ? __local_bh_enable+0x50/0x60 [16774.413091] [16774.417352] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16774.435973] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16774.440933] [16774.443028] [16774.445126] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16774.450430] ? igrab+0x43/0x50 [16774.453485] ? lock_release+0x14e/0x260 [16774.457323] ? lock_release+0xc5/0x260 [16774.461067] _raw_spin_unlock+0x1b/0x40 [16774.464906] igrab+0x43/0x50 [16774.467784] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16774.472934] delayed_work+0x1d8/0x280 [ceph] [16774.477249] ? process_scheduled_works+0x22a/0x510 [16774.482038] process_scheduled_works+0x2a4/0x510 [16774.486655] worker_thread+0x188/0x340 [16774.490404] ? rescuer_thread+0x310/0x310 [16774.494406] kthread+0xf5/0x130 [16774.497545] ? kthread_complete_and_exit+0x20/0x20 [16774.502328] ret_from_fork+0x31/0x40 [16774.505907] ? kthread_complete_and_exit+0x20/0x20 [16774.510693] ret_from_fork_asm+0x11/0x20 [16774.514623] [16781.216711] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16791.456756] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16801.749066] watchdog: BUG: soft lockup - CPU#0 stuck for 4533s! [dbench:78683] [16801.756281] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16801.761066] watchdog: BUG: soft lockup - CPU#7 stuck for 13530s! [kworker/7:4:78808] [16801.811738] irq event stamp: 22973082 [16801.819469] Modules linked in: ceph libceph dns_resolver [16801.823128] hardirqs last enabled at (22973081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16801.828432] fscache netfs veth [16801.838606] hardirqs last disabled at (22973082): [] sysvec_apic_timer_interrupt+0xe/0x90 [16801.841743] nft_chain_nat xt_MASQUERADE nf_nat [16801.851468] softirqs last enabled at (22973034): [] __do_softirq+0x2be/0x371 [16801.855992] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [16801.864677] softirqs last disabled at (22973029): [] irq_exit_rcu+0xb0/0xd0 [16801.871713] nfnetlink bridge [16801.880224] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16801.883189] stp llc [16801.892132] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16801.894317] xfs libcrc32c sunrpc [16801.901787] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16801.901789] intel_rapl_msr [16801.901790] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16801.905099] intel_rapl_common sb_edac [16801.910575] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16801.913366] x86_pkg_temp_thermal coretemp [16801.932104] [16801.932105] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16801.935848] kvm_intel kvm [16801.941066] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16801.945157] irqbypass i2c_i801 [16801.946648] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16801.953772] iTCO_wdt iTCO_vendor_support [16801.956475] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16801.963600] i2c_smbus crct10dif_pclmul [16801.966743] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16801.973870] joydev crc32_pclmul [16801.977874] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16801.984999] ipmi_ssif ghash_clmulni_intel [16801.988828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16801.995953] mei_me mei [16801.999176] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16802.007253] lpc_ich sha512_ssse3 [16802.011346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16802.017082] acpi_ipmi mfd_core [16802.019527] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16802.026649] wmi ipmi_si [16802.029962] Call Trace: [16802.037086] ipmi_devintf ipmi_msghandler [16802.040223] [16802.047348] ioatdma [16802.049878] ? watchdog_timer_fn+0x1b4/0x220 [16802.052321] acpi_pad acpi_power_meter [16802.056328] ? __hrtimer_run_queues+0x1d3/0x360 [16802.058338] ixgbe [16802.060524] ? hrtimer_interrupt+0x104/0x240 [16802.064784] igb mdio [16802.068529] ? __local_bh_enable+0x50/0x60 [16802.073052] crc32c_intel nvme [16802.075065] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16802.079326] ptp nvme_core [16802.081598] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16802.085688] i2c_algo_bit pps_core [16802.088744] [16802.093966] dca fuse [16802.096670] [16802.101626] [16802.101627] irq event stamp: 57421222 [16802.105027] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16802.107122] hardirqs last enabled at (57421221): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16802.109396] ? queued_spin_lock_slowpath+0x16/0x2e0 [16802.111490] hardirqs last disabled at (57421222): [] sysvec_apic_timer_interrupt+0xe/0x90 [16802.112984] do_raw_spin_lock+0xb5/0xc0 [16802.116638] softirqs last enabled at (57421220): [] __do_softirq+0x2be/0x371 [16802.121944] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16802.132100] softirqs last disabled at (57421215): [] irq_exit_rcu+0xb0/0xd0 [16802.136974] try_get_cap_refs+0x159/0x880 [ceph] [16802.146695] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16802.150530] __ceph_get_caps+0x4a1/0x760 [ceph] [16802.159209] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16802.163997] ? __file_remove_privs+0x89/0x110 [16802.172504] Workqueue: events delayed_work [ceph] [16802.177115] ? wake_bit_function+0x70/0x70 [16802.186494] [16802.191019] ? ceph_write_iter+0x260/0x770 [ceph] [16802.198487] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [16802.202840] ceph_write_iter+0x260/0x770 [ceph] [16802.207535] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [16802.211081] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16802.211631] ? lock_is_held_type+0xaa/0x120 [16802.213117] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [16802.217817] vfs_write+0x324/0x4f0 [16802.222427] [16802.222427] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [16802.226954] ksys_pwrite64+0x65/0xa0 [16802.245687] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [16802.252638] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16802.256815] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [16802.262035] do_syscall_64+0x38/0x80 [16802.265430] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [16802.266921] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16802.274046] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [16802.277616] RIP: 0033:0x7f48f0eeb93b [16802.284750] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16802.289880] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16802.297003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16802.300574] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16802.307698] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16802.312750] ORIG_RAX: 0000000000000012 [16802.319876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16802.323465] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16802.331541] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16802.350279] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16802.356017] Call Trace: [16802.356018] [16802.361232] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16802.368359] ? watchdog_timer_fn+0x1b4/0x220 [16802.372188] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16802.379313] ? __hrtimer_run_queues+0x1d3/0x360 [16802.386436] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16802.393565] ? hrtimer_interrupt+0x104/0x240 [16802.400691] [16802.403128] ? __local_bh_enable+0x50/0x60 [16802.445841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16802.451058] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16802.456014] [16802.458112] [16802.460208] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16802.465517] ? do_raw_spin_lock+0x42/0xc0 [16802.469527] igrab+0x1d/0x50 [16802.472404] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16802.477554] delayed_work+0x1d8/0x280 [ceph] [16802.481835] ? process_scheduled_works+0x22a/0x510 [16802.486625] process_scheduled_works+0x2a4/0x510 [16802.491242] worker_thread+0x188/0x340 [16802.494990] ? rescuer_thread+0x310/0x310 [16802.498994] kthread+0xf5/0x130 [16802.502132] ? kthread_complete_and_exit+0x20/0x20 [16802.506917] ret_from_fork+0x31/0x40 [16802.510494] ? kthread_complete_and_exit+0x20/0x20 [16802.515278] ret_from_fork_asm+0x11/0x20 [16802.519202] [16812.449082] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16823.201424] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16829.749565] watchdog: BUG: soft lockup - CPU#0 stuck for 4559s! [dbench:78683] [16829.756777] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16829.761565] watchdog: BUG: soft lockup - CPU#7 stuck for 13556s! [kworker/7:4:78808] [16829.812237] irq event stamp: 23031276 [16829.819966] Modules linked in: ceph libceph dns_resolver fscache netfs veth [16829.823625] hardirqs last enabled at (23031275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16829.830582] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [16829.840741] hardirqs last disabled at (23031276): [] sysvec_apic_timer_interrupt+0xe/0x90 [16829.847692] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [16829.857417] softirqs last enabled at (23031266): [] __do_softirq+0x2be/0x371 [16829.864194] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [16829.872879] softirqs last disabled at (23031261): [] irq_exit_rcu+0xb0/0xd0 [16829.879750] x86_pkg_temp_thermal coretemp kvm_intel kvm [16829.888263] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16829.893565] irqbypass i2c_i801 [16829.902510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16829.902512] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16829.905646] iTCO_wdt iTCO_vendor_support [16829.913119] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16829.918596] i2c_smbus crct10dif_pclmul [16829.922600] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16829.941338] joydev crc32_pclmul [16829.945166] [16829.950385] ipmi_ssif ghash_clmulni_intel [16829.953610] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16829.955100] mei_me mei [16829.959192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16829.966314] lpc_ich [16829.968760] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16829.975883] sha512_ssse3 [16829.978068] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16829.985190] acpi_ipmi mfd_core [16829.987809] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16829.994932] wmi ipmi_si [16829.998069] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16830.005194] ipmi_devintf [16830.007725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16830.015804] ipmi_msghandler ioatdma [16830.018421] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16830.024157] acpi_pad acpi_power_meter [16830.027727] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16830.034852] ixgbe igb [16830.038597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16830.045720] mdio crc32c_intel [16830.048078] Call Trace: [16830.048081] [16830.055201] nvme ptp [16830.058256] ? watchdog_timer_fn+0x1b4/0x220 [16830.060697] nvme_core i2c_algo_bit [16830.062709] ? __hrtimer_run_queues+0x1d3/0x360 [16830.064978] pps_core [16830.069248] ? hrtimer_interrupt+0x104/0x240 [16830.072726] dca fuse [16830.077251] ? __local_bh_enable+0x50/0x60 [16830.079521] [16830.079522] irq event stamp: 57642226 [16830.083789] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16830.086056] hardirqs last enabled at (57642225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16830.090147] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16830.091637] hardirqs last disabled at (57642226): [] sysvec_apic_timer_interrupt+0xe/0x90 [16830.095296] [16830.100513] softirqs last enabled at (57642224): [] __do_softirq+0x2be/0x371 [16830.110670] [16830.115626] softirqs last disabled at (57642219): [] irq_exit_rcu+0xb0/0xd0 [16830.125353] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16830.127448] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16830.136136] ? queued_spin_lock_slowpath+0x16/0x2e0 [16830.138243] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16830.146760] do_raw_spin_lock+0xb5/0xc0 [16830.152062] Workqueue: events delayed_work [ceph] [16830.161441] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16830.166311] [16830.173784] try_get_cap_refs+0x159/0x880 [ceph] [16830.177611] RIP: 0010:lock_release+0x14e/0x260 [16830.182314] __ceph_get_caps+0x4a1/0x760 [ceph] [16830.187092] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [16830.188588] ? __file_remove_privs+0x89/0x110 [16830.193195] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [16830.197632] ? wake_bit_function+0x70/0x70 [16830.202156] [16830.220898] ? ceph_write_iter+0x260/0x770 [ceph] [16830.225247] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [16830.230464] ceph_write_iter+0x260/0x770 [ceph] [16830.234554] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [16830.236048] ? lock_is_held_type+0xaa/0x120 [16830.240741] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [16830.247869] vfs_write+0x324/0x4f0 [16830.252389] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [16830.259518] ksys_pwrite64+0x65/0xa0 [16830.263690] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [16830.270816] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16830.274212] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16830.281337] do_syscall_64+0x38/0x80 [16830.284906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16830.292032] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16830.297163] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16830.305250] RIP: 0033:0x7f48f0eeb93b [16830.308819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16830.314558] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16830.319601] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16830.326724] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16830.330295] Call Trace: [16830.330296] [16830.337420] ORIG_RAX: 0000000000000012 [16830.356157] ? watchdog_timer_fn+0x1b4/0x220 [16830.363280] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16830.368501] ? __hrtimer_run_queues+0x1d3/0x360 [16830.370943] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16830.372976] ? hrtimer_interrupt+0x104/0x240 [16830.376802] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16830.381065] ? __local_bh_enable+0x50/0x60 [16830.388189] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16830.392717] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16830.399838] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16830.404103] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16830.411231] [16830.415318] [16830.444021] [16830.446120] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16830.451426] ? igrab+0x43/0x50 [16830.454487] ? lock_release+0x14e/0x260 [16830.458322] ? lock_release+0xc5/0x260 [16830.462068] _raw_spin_unlock+0x1b/0x40 [16830.465906] igrab+0x43/0x50 [16830.468784] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16830.473935] delayed_work+0x1d8/0x280 [ceph] [16830.478248] ? process_scheduled_works+0x22a/0x510 [16830.483042] process_scheduled_works+0x2a4/0x510 [16830.487656] worker_thread+0x188/0x340 [16830.491404] ? rescuer_thread+0x310/0x310 [16830.495407] kthread+0xf5/0x130 [16830.498547] ? kthread_complete_and_exit+0x20/0x20 [16830.503330] ret_from_fork+0x31/0x40 [16830.506910] ? kthread_complete_and_exit+0x20/0x20 [16830.511694] ret_from_fork_asm+0x11/0x20 [16830.515625] [16833.441683] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16844.193784] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16854.433947] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16857.750064] watchdog: BUG: soft lockup - CPU#0 stuck for 4586s! [dbench:78683] [16857.757273] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16857.762064] watchdog: BUG: soft lockup - CPU#7 stuck for 13583s! [kworker/7:4:78808] [16857.812716] irq event stamp: 23089394 [16857.812718] hardirqs last enabled at (23089393): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16857.820458] Modules linked in: [16857.824113] hardirqs last disabled at (23089394): [] sysvec_apic_timer_interrupt+0xe/0x90 [16857.834268] ceph [16857.837321] softirqs last enabled at (23089346): [] __do_softirq+0x2be/0x371 [16857.847044] libceph dns_resolver [16857.848969] softirqs last disabled at (23089341): [] irq_exit_rcu+0xb0/0xd0 [16857.857653] fscache netfs [16857.860964] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16857.869474] veth nft_chain_nat [16857.872179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16857.872180] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16857.881124] xt_MASQUERADE nf_nat [16857.884260] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16857.891731] nf_conntrack [16857.897225] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16857.900535] nf_defrag_ipv6 nf_defrag_ipv4 [16857.919273] [16857.921890] nft_compat nf_tables [16857.927108] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16857.931199] nfnetlink bridge [16857.932691] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16857.936000] stp llc [16857.943126] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16857.946088] xfs [16857.953214] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16857.955396] libcrc32c sunrpc [16857.962520] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16857.964360] intel_rapl_msr intel_rapl_common [16857.971483] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16857.974461] sb_edac x86_pkg_temp_thermal [16857.981589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16857.985938] coretemp [16857.994016] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16857.998020] kvm_intel [16858.003759] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16858.006028] kvm irqbypass [16858.013153] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16858.015509] i2c_i801 iTCO_wdt [16858.022633] Call Trace: [16858.022635] [16858.025338] iTCO_vendor_support [16858.032462] ? watchdog_timer_fn+0x1b4/0x220 [16858.035514] i2c_smbus [16858.037958] ? __hrtimer_run_queues+0x1d3/0x360 [16858.039969] crct10dif_pclmul joydev [16858.043195] ? hrtimer_interrupt+0x104/0x240 [16858.047461] crc32_pclmul [16858.049822] ? __local_bh_enable+0x50/0x60 [16858.054346] ipmi_ssif [16858.057918] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16858.062180] ghash_clmulni_intel [16858.064800] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16858.068890] mei_me mei [16858.071246] [16858.076463] lpc_ich [16858.079689] [16858.084645] sha512_ssse3 acpi_ipmi [16858.087090] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16858.089187] mfd_core [16858.091375] ? queued_spin_lock_slowpath+0x18/0x2e0 [16858.093469] wmi ipmi_si [16858.096955] do_raw_spin_lock+0xb5/0xc0 [16858.102255] ipmi_devintf ipmi_msghandler [16858.104528] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16858.109397] ioatdma acpi_pad [16858.111932] try_get_cap_refs+0x159/0x880 [ceph] [16858.115758] acpi_power_meter ixgbe [16858.119769] __ceph_get_caps+0x4a1/0x760 [ceph] [16858.124547] igb mdio [16858.127516] ? __file_remove_privs+0x89/0x110 [16858.132121] crc32c_intel nvme [16858.135608] ? wake_bit_function+0x70/0x70 [16858.140130] ptp [16858.142403] ? ceph_write_iter+0x260/0x770 [ceph] [16858.146756] nvme_core i2c_algo_bit [16858.149804] ceph_write_iter+0x260/0x770 [ceph] [16858.153894] pps_core dca [16858.155752] ? lock_is_held_type+0xaa/0x120 [16858.160462] fuse [16858.163949] vfs_write+0x324/0x4f0 [16858.168471] irq event stamp: 57863232 [16858.171093] ksys_pwrite64+0x65/0xa0 [16858.175265] hardirqs last enabled at (57863231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16858.177192] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16858.180587] hardirqs last disabled at (57863232): [] sysvec_apic_timer_interrupt+0xe/0x90 [16858.184245] do_syscall_64+0x38/0x80 [16858.187815] softirqs last enabled at (57863230): [] __do_softirq+0x2be/0x371 [16858.197974] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16858.203103] softirqs last disabled at (57863225): [] irq_exit_rcu+0xb0/0xd0 [16858.212828] RIP: 0033:0x7f48f0eeb93b [16858.216398] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16858.225084] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16858.230126] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16858.238638] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16858.242209] Workqueue: events delayed_work [ceph] [16858.251587] ORIG_RAX: 0000000000000012 [16858.270323] [16858.277794] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16858.283011] RIP: 0010:lock_acquire+0x86/0x2b0 [16858.287709] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16858.291541] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [16858.293031] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16858.300155] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [16858.304505] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16858.311630] [16858.311631] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [16858.330367] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16858.337492] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [16858.342715] [16858.349832] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [16858.382023] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [16858.389145] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16858.396271] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16858.404347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16858.410084] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16858.417209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16858.424333] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16858.431469] Call Trace: [16858.433921] [16858.435930] ? watchdog_timer_fn+0x1b4/0x220 [16858.440196] ? __hrtimer_run_queues+0x1d3/0x360 [16858.444721] ? hrtimer_interrupt+0x104/0x240 [16858.448989] ? __local_bh_enable+0x50/0x60 [16858.453084] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16858.458307] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16858.463266] [16858.465363] [16858.467470] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16858.472785] ? lock_acquire+0x86/0x2b0 [16858.476535] ? igrab+0x43/0x50 [16858.479585] ? lock_release+0xc5/0x260 [16858.483330] _raw_spin_lock+0x30/0x40 [16858.486987] ? igrab+0x1d/0x50 [16858.490038] igrab+0x1d/0x50 [16858.492914] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16858.498065] delayed_work+0x1d8/0x280 [ceph] [16858.502345] ? process_scheduled_works+0x22a/0x510 [16858.507137] process_scheduled_works+0x2a4/0x510 [16858.511767] worker_thread+0x188/0x340 [16858.515518] ? rescuer_thread+0x310/0x310 [16858.519522] kthread+0xf5/0x130 [16858.522659] ? kthread_complete_and_exit+0x20/0x20 [16858.527446] ret_from_fork+0x31/0x40 [16858.531041] ? kthread_complete_and_exit+0x20/0x20 [16858.535823] ret_from_fork_asm+0x11/0x20 [16858.539756] [16865.186242] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16875.426199] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16885.750564] watchdog: BUG: soft lockup - CPU#0 stuck for 4612s! [dbench:78683] [16885.757779] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16885.762562] watchdog: BUG: soft lockup - CPU#7 stuck for 13609s! [kworker/7:4:78808] [16885.813237] irq event stamp: 23147716 [16885.820970] Modules linked in: ceph libceph dns_resolver fscache netfs [16885.824627] hardirqs last enabled at (23147715): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16885.831152] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [16885.841309] hardirqs last disabled at (23147716): [] sysvec_apic_timer_interrupt+0xe/0x90 [16885.848695] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [16885.858419] softirqs last enabled at (23147668): [] __do_softirq+0x2be/0x371 [16885.858423] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [16885.873540] softirqs last disabled at (23147663): [] irq_exit_rcu+0xb0/0xd0 [16885.873544] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [16885.889263] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16885.895433] i2c_i801 iTCO_wdt [16885.904378] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16885.904379] iTCO_vendor_support [16885.907430] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16885.914900] i2c_smbus crct10dif_pclmul [16885.918124] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16885.923601] joydev crc32_pclmul [16885.927432] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16885.946169] ipmi_ssif ghash_clmulni_intel [16885.949393] [16885.954612] mei_me mei [16885.958701] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16885.960192] lpc_ich sha512_ssse3 [16885.962635] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16885.969761] acpi_ipmi mfd_core [16885.973072] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16885.980196] wmi ipmi_si [16885.983334] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16885.990456] ipmi_devintf ipmi_msghandler [16885.992986] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16886.000111] ioatdma acpi_pad [16886.004115] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16886.011249] acpi_power_meter ixgbe [16886.014212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16886.022290] igb mdio [16886.025774] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16886.031512] crc32c_intel [16886.033781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16886.040907] nvme ptp [16886.043524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16886.050648] nvme_core [16886.052918] Call Trace: [16886.052920] [16886.060043] i2c_algo_bit pps_core [16886.062400] ? watchdog_timer_fn+0x1b4/0x220 [16886.064843] dca fuse [16886.066857] ? __hrtimer_run_queues+0x1d3/0x360 [16886.070251] [16886.074519] ? hrtimer_interrupt+0x104/0x240 [16886.076788] irq event stamp: 58084642 [16886.081311] ? __local_bh_enable+0x50/0x60 [16886.082802] hardirqs last enabled at (58084641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16886.087068] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16886.090724] hardirqs last disabled at (58084642): [] sysvec_apic_timer_interrupt+0xe/0x90 [16886.094814] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16886.104971] softirqs last enabled at (58084640): [] __do_softirq+0x2be/0x371 [16886.110190] [16886.119912] softirqs last disabled at (58084635): [] irq_exit_rcu+0xb0/0xd0 [16886.124870] [16886.133555] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16886.135652] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16886.144162] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16886.146264] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16886.155639] Workqueue: events delayed_work [ceph] [16886.160963] do_raw_spin_lock+0xb5/0xc0 [16886.168430] [16886.168431] RIP: 0010:lock_acquire+0xf0/0x2b0 [16886.173302] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16886.177998] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16886.181832] try_get_cap_refs+0x159/0x880 [ceph] [16886.182417] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16886.183319] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16886.187676] __ceph_get_caps+0x4a1/0x760 [ceph] [16886.192455] [16886.192456] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16886.211197] ? __file_remove_privs+0x89/0x110 [16886.215802] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16886.222755] ? wake_bit_function+0x70/0x70 [16886.227970] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16886.232498] ? ceph_write_iter+0x260/0x770 [ceph] [16886.233985] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16886.241111] ceph_write_iter+0x260/0x770 [ceph] [16886.245460] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16886.252589] ? lock_is_held_type+0xaa/0x120 [16886.256675] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16886.263802] vfs_write+0x324/0x4f0 [16886.268496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16886.275626] ksys_pwrite64+0x65/0xa0 [16886.280146] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16886.287270] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16886.291446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16886.299526] do_syscall_64+0x38/0x80 [16886.302923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16886.308661] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16886.312245] Call Trace: [16886.312247] [16886.319372] RIP: 0033:0x7f48f0eeb93b [16886.324502] ? watchdog_timer_fn+0x1b4/0x220 [16886.331628] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16886.335198] ? __hrtimer_run_queues+0x1d3/0x360 [16886.342322] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16886.347368] ? hrtimer_interrupt+0x104/0x240 [16886.349808] ORIG_RAX: 0000000000000012 [16886.351820] ? __local_bh_enable+0x50/0x60 [16886.355391] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16886.359657] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16886.378392] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16886.382917] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16886.388133] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16886.392397] [16886.396230] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16886.400338] [16886.400340] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16886.407462] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16886.412683] ? lock_acquire+0xf0/0x2b0 [16886.419809] [16886.424761] ? igrab+0x43/0x50 [16886.464612] ? lock_release+0xc5/0x260 [16886.468356] _raw_spin_lock+0x30/0x40 [16886.472021] ? igrab+0x1d/0x50 [16886.475073] igrab+0x1d/0x50 [16886.477968] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16886.483117] delayed_work+0x1d8/0x280 [ceph] [16886.487398] ? process_scheduled_works+0x22a/0x510 [16886.492187] process_scheduled_works+0x2a4/0x510 [16886.496802] worker_thread+0x188/0x340 [16886.500552] ? rescuer_thread+0x310/0x310 [16886.504557] kthread+0xf5/0x130 [16886.507702] ? kthread_complete_and_exit+0x20/0x20 [16886.512487] ret_from_fork+0x31/0x40 [16886.516067] ? kthread_complete_and_exit+0x20/0x20 [16886.520851] ret_from_fork_asm+0x11/0x20 [16886.524783] [16888.333975] rcu: INFO: rcu_preempt self-detected stall on CPU [16888.339732] rcu: 0-....: (4842156 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=2400685 [16888.350070] rcu: (t=4951637 jiffies g=1309609 q=213469 ncpus=8) [16888.356068] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16888.365011] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16888.372481] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16888.377961] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16888.396698] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16888.401916] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16888.409039] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16888.416164] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16888.423288] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16888.430412] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16888.437536] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16888.445613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16888.451350] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16888.458474] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16888.465598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16888.472723] Call Trace: [16888.475166] [16888.477178] ? rcu_dump_cpu_stacks+0xd1/0x180 [16888.481537] ? rcu_sched_clock_irq+0x633/0x1390 [16888.486062] ? lock_is_held_type+0xaa/0x120 [16888.490241] ? lock_is_held_type+0xaa/0x120 [16888.494425] ? tick_sched_do_timer+0xa0/0xa0 [16888.498696] ? rcu_read_lock_sched_held+0x56/0x90 [16888.503396] ? tick_sched_do_timer+0xa0/0xa0 [16888.507659] ? update_process_times+0x5f/0x90 [16888.512009] ? tick_sched_handle.isra.16+0x1e/0x50 [16888.516794] ? tick_sched_timer+0x6f/0x80 [16888.520798] ? __hrtimer_run_queues+0x1d3/0x360 [16888.525325] ? hrtimer_interrupt+0x104/0x240 [16888.529595] ? __local_bh_enable+0x50/0x60 [16888.533688] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16888.538913] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16888.543868] [16888.545965] [16888.548063] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16888.553372] ? queued_spin_lock_slowpath+0x18/0x2e0 [16888.558250] do_raw_spin_lock+0xb5/0xc0 [16888.562088] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16888.566890] try_get_cap_refs+0x159/0x880 [ceph] [16888.571521] __ceph_get_caps+0x4a1/0x760 [ceph] [16888.576061] ? __file_remove_privs+0x89/0x110 [16888.580417] ? wake_bit_function+0x70/0x70 [16888.584510] ? ceph_write_iter+0x260/0x770 [ceph] [16888.589215] ceph_write_iter+0x260/0x770 [ceph] [16888.593749] ? lock_is_held_type+0xaa/0x120 [16888.597935] vfs_write+0x324/0x4f0 [16888.601343] ksys_pwrite64+0x65/0xa0 [16888.604919] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16888.610049] do_syscall_64+0x38/0x80 [16888.613620] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16888.618663] RIP: 0033:0x7f48f0eeb93b [16888.622235] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16888.640981] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [16888.648555] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16888.655680] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16888.662804] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16888.669928] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16888.677052] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16888.684181] [16888.686369] Sending NMI from CPU 0 to CPUs 7: [16888.690721] NMI backtrace for cpu 7 [16888.690723] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16888.690725] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16888.690726] Workqueue: events delayed_work [ceph] [16888.690747] RIP: 0010:__lock_acquire+0x339/0x1e20 [16888.690750] Code: f6 44 0f 47 f6 80 b8 ca 00 00 00 02 44 0f 44 f6 41 83 c5 01 44 3b ab 50 0a 00 00 7d 2d 49 63 c5 48 8d 04 80 41 0f b7 44 c4 20 <66> 25 ff 1f 0f b7 c0 48 0f a3 45 00 72 a8 8b 05 fb 6e 68 02 85 c0 [16888.690751] RSP: 0018:ffffc90002117c60 EFLAGS: 00000006 [16888.690753] RAX: 00000000000000d1 RBX: ffff88810debaa80 RCX: 00000000000006f5 [16888.690754] RDX: ffff88810debb550 RSI: 0000000000000006 RDI: 0000000000000003 [16888.690756] RBP: ffffffff82f75120 R08: 0000000000000078 R09: 0000000000000000 [16888.690757] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810debb4d8 [16888.690758] R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000002 [16888.690759] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16888.690760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16888.690762] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16888.690763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16888.690764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16888.690765] Call Trace: [16888.690766] [16888.690768] ? nmi_cpu_backtrace+0xad/0x120 [16888.690772] ? nmi_cpu_backtrace_handler+0x11/0x20 [16888.690774] ? nmi_handle+0xd4/0x260 [16888.690779] ? default_do_nmi+0x49/0xf0 [16888.690781] ? exc_nmi+0x11e/0x150 [16888.690784] ? end_repeat_nmi+0x16/0x67 [16888.690790] ? __lock_acquire+0x339/0x1e20 [16888.690792] ? __lock_acquire+0x339/0x1e20 [16888.690795] ? __lock_acquire+0x339/0x1e20 [16888.690797] [16888.690798] [16888.690801] lock_acquire+0xcb/0x2b0 [16888.690803] ? igrab+0x1d/0x50 [16888.690805] ? igrab+0x43/0x50 [16888.690807] ? lock_release+0xc5/0x260 [16888.690810] _raw_spin_lock+0x30/0x40 [16888.690813] ? igrab+0x1d/0x50 [16888.690814] igrab+0x1d/0x50 [16888.690816] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16888.690836] delayed_work+0x1d8/0x280 [ceph] [16888.690855] ? process_scheduled_works+0x22a/0x510 [16888.690858] process_scheduled_works+0x2a4/0x510 [16888.690863] worker_thread+0x188/0x340 [16888.690867] ? rescuer_thread+0x310/0x310 [16888.690869] kthread+0xf5/0x130 [16888.690872] ? kthread_complete_and_exit+0x20/0x20 [16888.690875] ret_from_fork+0x31/0x40 [16888.690878] ? kthread_complete_and_exit+0x20/0x20 [16888.690881] ret_from_fork_asm+0x11/0x20 [16888.690887] [16896.418733] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16907.170811] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16913.751066] watchdog: BUG: soft lockup - CPU#0 stuck for 4638s! [dbench:78683] [16913.758283] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16913.763066] watchdog: BUG: soft lockup - CPU#7 stuck for 13635s! [kworker/7:4:78808] [16913.813740] irq event stamp: 23205086 [16913.821474] Modules linked in: ceph libceph [16913.825131] hardirqs last enabled at (23205085): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16913.829307] dns_resolver fscache netfs [16913.839466] hardirqs last disabled at (23205086): [] sysvec_apic_timer_interrupt+0xe/0x90 [16913.843295] veth nft_chain_nat xt_MASQUERADE nf_nat [16913.853019] softirqs last enabled at (23205000): [] __do_softirq+0x2be/0x371 [16913.857977] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [16913.866661] softirqs last disabled at (23204995): [] irq_exit_rcu+0xb0/0xd0 [16913.866663] nft_compat [16913.866665] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16913.871880] nf_tables nfnetlink [16913.880390] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16913.882833] bridge stp [16913.891778] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16913.895001] llc xfs [16913.902474] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16913.904917] libcrc32c sunrpc [16913.910395] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16913.912578] intel_rapl_msr intel_rapl_common [16913.931318] [16913.931318] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16913.934279] sb_edac x86_pkg_temp_thermal [16913.939498] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16913.943849] coretemp [16913.945340] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16913.952464] kvm_intel [16913.956468] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16913.963592] kvm irqbypass [16913.965863] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16913.972987] i2c_i801 [16913.975344] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16913.982469] iTCO_wdt [16913.985171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16913.992297] iTCO_vendor_support i2c_smbus [16913.994568] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16914.002645] crct10dif_pclmul joydev [16914.004915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16914.010652] crc32_pclmul ipmi_ssif [16914.014752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16914.021875] ghash_clmulni_intel [16914.025464] Call Trace: [16914.025465] [16914.032588] mei_me mei [16914.036074] ? watchdog_timer_fn+0x1b4/0x220 [16914.043196] lpc_ich [16914.046422] ? __hrtimer_run_queues+0x1d3/0x360 [16914.048864] sha512_ssse3 acpi_ipmi [16914.050879] ? hrtimer_interrupt+0x104/0x240 [16914.053318] mfd_core wmi [16914.057582] ? __local_bh_enable+0x50/0x60 [16914.059768] ipmi_si ipmi_devintf [16914.064295] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16914.067775] ipmi_msghandler ioatdma [16914.072039] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16914.074656] acpi_pad acpi_power_meter [16914.078747] [16914.082059] ixgbe igb [16914.087276] [16914.087277] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16914.090846] mdio crc32c_intel [16914.095809] ? queued_spin_lock_slowpath+0x18/0x2e0 [16914.099548] nvme [16914.101648] do_raw_spin_lock+0xb5/0xc0 [16914.104003] ptp nvme_core [16914.106102] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16914.111405] i2c_algo_bit pps_core [16914.114468] try_get_cap_refs+0x159/0x880 [ceph] [16914.119333] dca fuse [16914.121264] __ceph_get_caps+0x4a1/0x760 [ceph] [16914.125090] [16914.125090] irq event stamp: 58304506 [16914.127797] ? __file_remove_privs+0x89/0x110 [16914.132577] hardirqs last enabled at (58304505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16914.135976] ? wake_bit_function+0x70/0x70 [16914.140585] hardirqs last disabled at (58304506): [] sysvec_apic_timer_interrupt+0xe/0x90 [16914.142859] ? ceph_write_iter+0x260/0x770 [ceph] [16914.147381] softirqs last enabled at (58304504): [] __do_softirq+0x2be/0x371 [16914.148871] ceph_write_iter+0x260/0x770 [ceph] [16914.152528] softirqs last disabled at (58304499): [] irq_exit_rcu+0xb0/0xd0 [16914.156884] ? lock_is_held_type+0xaa/0x120 [16914.167038] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16914.171131] vfs_write+0x324/0x4f0 [16914.180853] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16914.185554] ksys_pwrite64+0x65/0xa0 [16914.194235] Workqueue: events delayed_work [ceph] [16914.198757] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16914.207268] [16914.207269] RIP: 0010:lock_acquire+0xf0/0x2b0 [16914.211466] do_syscall_64+0x38/0x80 [16914.220841] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16914.224239] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16914.231709] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16914.235280] RIP: 0033:0x7f48f0eeb93b [16914.239977] [16914.239978] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16914.245109] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16914.246598] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16914.250949] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16914.254519] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16914.273257] ORIG_RAX: 0000000000000012 [16914.278302] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16914.283519] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16914.287089] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16914.288581] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16914.295703] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16914.314441] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16914.321583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16914.326801] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16914.333926] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16914.337764] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16914.344888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16914.352018] [16914.359137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16914.425022] Call Trace: [16914.427473] [16914.429488] ? watchdog_timer_fn+0x1b4/0x220 [16914.433759] ? __hrtimer_run_queues+0x1d3/0x360 [16914.438285] ? hrtimer_interrupt+0x104/0x240 [16914.442547] ? __local_bh_enable+0x50/0x60 [16914.446641] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16914.451865] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16914.456821] [16914.458919] [16914.461016] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16914.466324] ? lock_acquire+0xf0/0x2b0 [16914.470074] ? igrab+0x43/0x50 [16914.473124] ? lock_release+0xc5/0x260 [16914.476870] _raw_spin_lock+0x30/0x40 [16914.480525] ? igrab+0x1d/0x50 [16914.483576] igrab+0x1d/0x50 [16914.486470] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16914.491622] delayed_work+0x1d8/0x280 [ceph] [16914.495903] ? process_scheduled_works+0x22a/0x510 [16914.500693] process_scheduled_works+0x2a4/0x510 [16914.505308] worker_thread+0x188/0x340 [16914.509058] ? rescuer_thread+0x310/0x310 [16914.513061] kthread+0xf5/0x130 [16914.516197] ? kthread_complete_and_exit+0x20/0x20 [16914.520982] ret_from_fork+0x31/0x40 [16914.524551] ? kthread_complete_and_exit+0x20/0x20 [16914.529338] ret_from_fork_asm+0x11/0x20 [16914.533259] [16917.410980] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16941.751570] watchdog: BUG: soft lockup - CPU#0 stuck for 4664s! [dbench:78683] [16941.758789] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16941.763570] watchdog: BUG: soft lockup - CPU#7 stuck for 13661s! [kworker/7:4:78808] [16941.814243] irq event stamp: 23263296 [16941.814244] hardirqs last enabled at (23263295): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16941.821978] Modules linked in: ceph [16941.825634] hardirqs last disabled at (23263296): [] sysvec_apic_timer_interrupt+0xe/0x90 [16941.835793] libceph [16941.839277] softirqs last enabled at (23263210): [] __do_softirq+0x2be/0x371 [16941.849000] dns_resolver fscache [16941.851184] softirqs last disabled at (23263205): [] irq_exit_rcu+0xb0/0xd0 [16941.859869] netfs [16941.863181] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16941.871690] veth [16941.873702] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16941.882645] nft_chain_nat [16941.884568] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [16941.892040] xt_MASQUERADE nf_nat [16941.894745] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [16941.900221] nf_conntrack nf_defrag_ipv6 [16941.903532] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16941.922269] nf_defrag_ipv4 nft_compat [16941.926187] [16941.926188] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16941.931406] nf_tables nfnetlink [16941.935148] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16941.936639] bridge stp [16941.943763] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16941.946988] llc xfs [16941.954111] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16941.956556] libcrc32c [16941.963680] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16941.965865] sunrpc [16941.972988] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16941.975344] intel_rapl_msr [16941.982469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16941.984567] intel_rapl_common sb_edac [16941.992643] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16941.995435] x86_pkg_temp_thermal [16942.001172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16942.004917] coretemp kvm_intel [16942.012042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16942.015351] kvm irqbypass [16942.022476] Call Trace: [16942.022477] [16942.025614] i2c_i801 iTCO_wdt [16942.032737] ? watchdog_timer_fn+0x1b4/0x220 [16942.035441] iTCO_vendor_support i2c_smbus [16942.037886] ? __hrtimer_run_queues+0x1d3/0x360 [16942.039896] crct10dif_pclmul joydev [16942.042968] ? hrtimer_interrupt+0x104/0x240 [16942.047245] crc32_pclmul ipmi_ssif [16942.051336] ? __local_bh_enable+0x50/0x60 [16942.055861] ghash_clmulni_intel [16942.059434] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16942.063696] mei_me mei [16942.067181] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16942.071269] lpc_ich sha512_ssse3 [16942.074494] [16942.079712] acpi_ipmi mfd_core [16942.082157] [16942.082158] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16942.087113] wmi ipmi_si [16942.090427] ? queued_spin_lock_slowpath+0x18/0x2e0 [16942.092520] ipmi_devintf ipmi_msghandler [16942.095661] do_raw_spin_lock+0xb5/0xc0 [16942.097755] ioatdma acpi_pad [16942.103061] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16942.105591] acpi_power_meter ixgbe [16942.110465] try_get_cap_refs+0x159/0x880 [ceph] [16942.114465] igb mdio [16942.118302] __ceph_get_caps+0x4a1/0x760 [ceph] [16942.121259] crc32c_intel nvme [16942.126048] ? __file_remove_privs+0x89/0x110 [16942.129529] ptp nvme_core [16942.134140] ? wake_bit_function+0x70/0x70 [16942.136410] i2c_algo_bit pps_core [16942.140936] ? ceph_write_iter+0x260/0x770 [ceph] [16942.143984] dca fuse [16942.148337] ceph_write_iter+0x260/0x770 [ceph] [16942.151040] [16942.151041] irq event stamp: 58526186 [16942.155135] ? lock_is_held_type+0xaa/0x120 [16942.158528] hardirqs last enabled at (58526185): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16942.163228] vfs_write+0x324/0x4f0 [16942.165497] hardirqs last disabled at (58526186): [] sysvec_apic_timer_interrupt+0xe/0x90 [16942.170025] ksys_pwrite64+0x65/0xa0 [16942.171510] softirqs last enabled at (58526184): [] __do_softirq+0x2be/0x371 [16942.175169] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16942.179345] softirqs last disabled at (58526179): [] irq_exit_rcu+0xb0/0xd0 [16942.189504] do_syscall_64+0x38/0x80 [16942.192901] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16942.202624] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16942.206196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16942.214881] RIP: 0033:0x7f48f0eeb93b [16942.220009] Workqueue: events delayed_work [ceph] [16942.228522] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16942.232091] [16942.232092] RIP: 0010:lock_acquire+0x17b/0x2b0 [16942.241468] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16942.246514] Code: c8 89 d1 89 f2 4c 89 e6 e8 62 5b ff ff 59 5e 4c 8b 14 24 65 ff 0d 5d e6 eb 7e 0f 85 c8 fe ff ff 0f 1f 44 00 00 e9 be fe ff ff <4c> 89 14 24 e8 9c 8a 02 00 4c 8b 14 24 84 c0 0f 85 c0 fe ff ff 0f [16942.253985] ORIG_RAX: 0000000000000012 [16942.257556] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [16942.262253] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16942.280990] [16942.280991] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [16942.282482] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16942.286917] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [16942.292136] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16942.310872] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [16942.314704] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16942.319922] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [16942.327045] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16942.328537] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16942.335667] [16942.342784] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16942.402915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16942.408653] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16942.415776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16942.422902] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16942.430026] Call Trace: [16942.432470] [16942.434481] ? watchdog_timer_fn+0x1b4/0x220 [16942.438744] ? __hrtimer_run_queues+0x1d3/0x360 [16942.443272] ? hrtimer_interrupt+0x104/0x240 [16942.447541] ? __local_bh_enable+0x50/0x60 [16942.451634] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16942.456858] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16942.461814] [16942.463911] [16942.466010] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16942.471317] ? lock_acquire+0x17b/0x2b0 [16942.475155] ? igrab+0x43/0x50 [16942.478204] ? lock_release+0xc5/0x260 [16942.481953] _raw_spin_lock+0x30/0x40 [16942.485630] ? igrab+0x1d/0x50 [16942.488682] igrab+0x1d/0x50 [16942.491561] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16942.496709] delayed_work+0x1d8/0x280 [ceph] [16942.500991] ? process_scheduled_works+0x22a/0x510 [16942.505781] process_scheduled_works+0x2a4/0x510 [16942.510398] worker_thread+0x188/0x340 [16942.514146] ? rescuer_thread+0x310/0x310 [16942.518148] kthread+0xf5/0x130 [16942.521286] ? kthread_complete_and_exit+0x20/0x20 [16942.526073] ret_from_fork+0x31/0x40 [16942.529651] ? kthread_complete_and_exit+0x20/0x20 [16942.534434] ret_from_fork_asm+0x11/0x20 [16942.538356] [16948.643106] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [16948.651634] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16959.395976] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16969.636072] libceph: auth protocol 'cephx' mauth authentication failed: -13 [16969.752073] watchdog: BUG: soft lockup - CPU#0 stuck for 4690s! [dbench:78683] [16969.759294] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16969.764073] watchdog: BUG: soft lockup - CPU#7 stuck for 13687s! [kworker/7:4:78808] [16969.814749] irq event stamp: 23321750 [16969.822482] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [16969.826139] hardirqs last enabled at (23321749): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16969.834313] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [16969.844469] hardirqs last disabled at (23321750): [] sysvec_apic_timer_interrupt+0xe/0x90 [16969.851507] nft_compat nf_tables nfnetlink bridge stp llc [16969.861231] softirqs last enabled at (23321634): [] __do_softirq+0x2be/0x371 [16969.861235] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [16969.875392] softirqs last disabled at (23321629): [] irq_exit_rcu+0xb0/0xd0 [16969.884856] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [16969.893369] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16969.898499] iTCO_vendor_support i2c_smbus [16969.907444] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16969.907445] crct10dif_pclmul [16969.907446] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [16969.911551] joydev crc32_pclmul [16969.919023] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [16969.921986] ipmi_ssif ghash_clmulni_intel [16969.927472] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16969.930695] mei_me mei [16969.949435] [16969.953525] lpc_ich sha512_ssse3 [16969.958754] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16969.961196] acpi_ipmi mfd_core [16969.962686] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16969.965996] wmi [16969.973119] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16969.976257] ipmi_si [16969.983381] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16969.985219] ipmi_devintf [16969.992344] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16969.994526] ipmi_msghandler [16970.001650] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16970.004269] ioatdma acpi_pad [16970.011393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16970.014270] acpi_power_meter [16970.022348] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16970.025312] ixgbe igb [16970.031048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16970.034013] mdio crc32c_intel [16970.041137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16970.043494] nvme [16970.050619] Call Trace: [16970.050621] [16970.053669] ptp nvme_core [16970.060795] ? watchdog_timer_fn+0x1b4/0x220 [16970.062717] i2c_algo_bit pps_core [16970.065162] ? __hrtimer_run_queues+0x1d3/0x360 [16970.067173] dca fuse [16970.069879] ? hrtimer_interrupt+0x104/0x240 [16970.074142] [16970.074142] irq event stamp: 58747388 [16970.077536] ? __local_bh_enable+0x50/0x60 [16970.082061] hardirqs last enabled at (58747387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16970.084335] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16970.088598] hardirqs last disabled at (58747388): [] sysvec_apic_timer_interrupt+0xe/0x90 [16970.090087] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16970.093758] softirqs last enabled at (58747386): [] __do_softirq+0x2be/0x371 [16970.097854] [16970.108009] softirqs last disabled at (58747381): [] irq_exit_rcu+0xb0/0xd0 [16970.113227] [16970.113229] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16970.122953] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16970.127913] ? queued_spin_lock_slowpath+0x2d/0x2e0 [16970.136592] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16970.138693] do_raw_spin_lock+0xb5/0xc0 [16970.147202] Workqueue: events delayed_work [ceph] [16970.149299] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16970.154602] [16970.154603] RIP: 0010:lock_acquire+0xf0/0x2b0 [16970.163983] try_get_cap_refs+0x159/0x880 [ceph] [16970.168852] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16970.176328] __ceph_get_caps+0x4a1/0x760 [ceph] [16970.180153] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16970.184856] ? __file_remove_privs+0x89/0x110 [16970.189634] [16970.191126] ? wake_bit_function+0x70/0x70 [16970.195477] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16970.200091] ? ceph_write_iter+0x260/0x770 [ceph] [16970.218824] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16970.223350] ceph_write_iter+0x260/0x770 [ceph] [16970.228567] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16970.232922] ? lock_is_held_type+0xaa/0x120 [16970.234406] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16970.238500] vfs_write+0x324/0x4f0 [16970.245623] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16970.250325] ksys_pwrite64+0x65/0xa0 [16970.257444] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16970.261970] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16970.269091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16970.273271] do_syscall_64+0x38/0x80 [16970.280393] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16970.283791] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16970.290915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16970.294487] RIP: 0033:0x7f48f0eeb93b [16970.302563] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16970.307697] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16970.313432] Call Trace: [16970.313434] [16970.317003] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16970.324128] ? watchdog_timer_fn+0x1b4/0x220 [16970.329170] ORIG_RAX: 0000000000000012 [16970.336295] ? __hrtimer_run_queues+0x1d3/0x360 [16970.339866] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16970.346995] ? hrtimer_interrupt+0x104/0x240 [16970.365727] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16970.368170] ? __local_bh_enable+0x50/0x60 [16970.370181] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16970.375403] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16970.379664] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16970.383496] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16970.388017] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16970.395144] [16970.399412] [16970.406531] [16970.406532] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16970.453882] ? lock_acquire+0xf0/0x2b0 [16970.457632] ? igrab+0x43/0x50 [16970.460681] ? lock_release+0xc5/0x260 [16970.464427] _raw_spin_lock+0x30/0x40 [16970.468092] ? igrab+0x1d/0x50 [16970.471141] igrab+0x1d/0x50 [16970.474022] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16970.479170] delayed_work+0x1d8/0x280 [ceph] [16970.483450] ? process_scheduled_works+0x22a/0x510 [16970.488259] process_scheduled_works+0x2a4/0x510 [16970.492875] worker_thread+0x188/0x340 [16970.496624] ? rescuer_thread+0x310/0x310 [16970.500626] kthread+0xf5/0x130 [16970.503777] ? kthread_complete_and_exit+0x20/0x20 [16970.508566] ret_from_fork+0x31/0x40 [16970.512145] ? kthread_complete_and_exit+0x20/0x20 [16970.516929] ret_from_fork_asm+0x11/0x20 [16970.520861] [16997.752577] watchdog: BUG: soft lockup - CPU#0 stuck for 4716s! [dbench:78683] [16997.759789] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [16997.764577] watchdog: BUG: soft lockup - CPU#7 stuck for 13713s! [kworker/7:4:78808] [16997.815230] irq event stamp: 23379964 [16997.815231] hardirqs last enabled at (23379963): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16997.822977] Modules linked in: [16997.826635] hardirqs last disabled at (23379964): [] sysvec_apic_timer_interrupt+0xe/0x90 [16997.836794] ceph [16997.839842] softirqs last enabled at (23379876): [] __do_softirq+0x2be/0x371 [16997.849566] libceph dns_resolver [16997.851492] softirqs last disabled at (23379871): [] irq_exit_rcu+0xb0/0xd0 [16997.860175] fscache [16997.863486] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16997.871996] netfs veth [16997.874181] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16997.883124] nft_chain_nat [16997.885570] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [16997.893039] xt_MASQUERADE nf_nat [16997.895745] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [16997.901222] nf_conntrack [16997.904533] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [16997.923270] nf_defrag_ipv6 nf_defrag_ipv4 [16997.925888] [16997.931104] nft_compat nf_tables [16997.935195] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [16997.936686] nfnetlink bridge [16997.939996] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [16997.947122] stp [16997.950086] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [16997.957209] llc xfs [16997.959046] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [16997.966170] libcrc32c sunrpc [16997.968355] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [16997.975479] intel_rapl_msr intel_rapl_common [16997.978443] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [16997.985567] sb_edac x86_pkg_temp_thermal [16997.989917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16997.997996] coretemp kvm_intel [16998.001999] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [16998.007737] kvm irqbypass [16998.010873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16998.017998] i2c_i801 iTCO_wdt [16998.020702] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16998.027826] iTCO_vendor_support [16998.030878] Call Trace: [16998.030879] [16998.038000] i2c_smbus crct10dif_pclmul [16998.041227] ? watchdog_timer_fn+0x1b4/0x220 [16998.043670] joydev crc32_pclmul [16998.045683] ? __hrtimer_run_queues+0x1d3/0x360 [16998.049510] ipmi_ssif [16998.053780] ? hrtimer_interrupt+0x104/0x240 [16998.056999] ghash_clmulni_intel mei_me [16998.061522] ? __local_bh_enable+0x50/0x60 [16998.063880] mei lpc_ich [16998.068147] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16998.071977] sha512_ssse3 acpi_ipmi [16998.076066] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16998.078597] mfd_core wmi [16998.083815] [16998.087297] ipmi_si ipmi_devintf [16998.092258] [16998.094873] ipmi_msghandler ioatdma [16998.096977] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16998.100291] acpi_pad acpi_power_meter [16998.102390] ? queued_spin_lock_slowpath+0x16/0x2e0 [16998.105974] ixgbe igb [16998.111280] do_raw_spin_lock+0xb5/0xc0 [16998.115024] mdio crc32c_intel [16998.119894] __cap_delay_requeue+0xa7/0x1d0 [ceph] [16998.122256] nvme ptp [16998.126094] try_get_cap_refs+0x159/0x880 [ceph] [16998.129142] nvme_core i2c_algo_bit [16998.133931] __ceph_get_caps+0x4a1/0x760 [ceph] [16998.136198] pps_core dca [16998.140811] ? __file_remove_privs+0x89/0x110 [16998.144293] fuse [16998.148816] ? wake_bit_function+0x70/0x70 [16998.151433] irq event stamp: 58968864 [16998.155787] ? ceph_write_iter+0x260/0x770 [ceph] [16998.157708] hardirqs last enabled at (58968863): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16998.161798] ceph_write_iter+0x260/0x770 [ceph] [16998.165456] hardirqs last disabled at (58968864): [] sysvec_apic_timer_interrupt+0xe/0x90 [16998.170158] ? lock_is_held_type+0xaa/0x120 [16998.180312] softirqs last enabled at (58968862): [] __do_softirq+0x2be/0x371 [16998.184836] vfs_write+0x324/0x4f0 [16998.194559] softirqs last disabled at (58968857): [] irq_exit_rcu+0xb0/0xd0 [16998.198742] ksys_pwrite64+0x65/0xa0 [16998.207421] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [16998.210818] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [16998.219328] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [16998.222901] do_syscall_64+0x38/0x80 [16998.232277] Workqueue: events delayed_work [ceph] [16998.237407] entry_SYSCALL_64_after_hwframe+0x63/0xcd [16998.244880] [16998.244881] RIP: 0010:lock_acquire+0xf0/0x2b0 [16998.248450] RIP: 0033:0x7f48f0eeb93b [16998.253147] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [16998.258191] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [16998.259680] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [16998.264032] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [16998.267603] [16998.267604] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [16998.286341] ORIG_RAX: 0000000000000012 [16998.305079] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [16998.310295] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [16998.315512] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [16998.317005] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [16998.324128] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [16998.327978] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [16998.335101] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [16998.342225] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [16998.349349] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [16998.356474] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [16998.363598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16998.370725] [16998.377845] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [16998.415226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16998.422349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [16998.429473] Call Trace: [16998.431918] [16998.433928] ? watchdog_timer_fn+0x1b4/0x220 [16998.438193] ? __hrtimer_run_queues+0x1d3/0x360 [16998.442721] ? hrtimer_interrupt+0x104/0x240 [16998.446988] ? __local_bh_enable+0x50/0x60 [16998.451083] ? __sysvec_apic_timer_interrupt+0x61/0x160 [16998.456307] ? sysvec_apic_timer_interrupt+0x6d/0x90 [16998.461265] [16998.463362] [16998.465459] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [16998.470767] ? lock_acquire+0xf0/0x2b0 [16998.474517] ? igrab+0x43/0x50 [16998.477567] ? lock_release+0xc5/0x260 [16998.481313] _raw_spin_lock+0x30/0x40 [16998.484983] ? igrab+0x1d/0x50 [16998.488037] igrab+0x1d/0x50 [16998.490914] ceph_check_delayed_caps+0xb7/0x270 [ceph] [16998.496064] delayed_work+0x1d8/0x280 [ceph] [16998.500346] ? process_scheduled_works+0x22a/0x510 [16998.505136] process_scheduled_works+0x2a4/0x510 [16998.509751] worker_thread+0x188/0x340 [16998.513499] ? rescuer_thread+0x310/0x310 [16998.517502] kthread+0xf5/0x130 [16998.520640] ? kthread_complete_and_exit+0x20/0x20 [16998.525424] ret_from_fork+0x31/0x40 [16998.528995] ? kthread_complete_and_exit+0x20/0x20 [16998.533779] ret_from_fork_asm+0x11/0x20 [16998.537703] [17011.620235] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [17025.753080] watchdog: BUG: soft lockup - CPU#0 stuck for 4742s! [dbench:78683] [17025.760294] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17025.765081] watchdog: BUG: soft lockup - CPU#7 stuck for 13739s! [kworker/7:4:78808] [17025.815744] irq event stamp: 23438136 [17025.815745] hardirqs last enabled at (23438135): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17025.823491] Modules linked in: [17025.827148] hardirqs last disabled at (23438136): [] sysvec_apic_timer_interrupt+0xe/0x90 [17025.837307] ceph [17025.840357] softirqs last enabled at (23438090): [] __do_softirq+0x2be/0x371 [17025.850080] libceph [17025.852003] softirqs last disabled at (23438085): [] irq_exit_rcu+0xb0/0xd0 [17025.860690] dns_resolver [17025.862874] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17025.871383] fscache netfs [17025.874001] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17025.882945] veth nft_chain_nat [17025.885651] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17025.893119] xt_MASQUERADE [17025.896258] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17025.901734] nf_nat nf_conntrack [17025.904474] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17025.923211] nf_defrag_ipv6 [17025.926435] [17025.931654] nf_defrag_ipv4 nft_compat [17025.934443] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17025.935935] nf_tables [17025.939679] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17025.946803] nfnetlink [17025.949160] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17025.956284] bridge [17025.958642] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17025.965764] stp [17025.967862] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17025.974987] llc xfs [17025.976824] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17025.983948] libcrc32c [17025.986133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17025.994211] sunrpc [17025.996568] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17026.002304] intel_rapl_msr [17026.004401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17026.011527] intel_rapl_common sb_edac [17026.014317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17026.021442] x86_pkg_temp_thermal coretemp [17026.025186] Call Trace: [17026.025189] [17026.032308] kvm_intel kvm [17026.036403] ? watchdog_timer_fn+0x1b4/0x220 [17026.038844] irqbypass i2c_i801 [17026.040855] ? __hrtimer_run_queues+0x1d3/0x360 [17026.043560] iTCO_wdt iTCO_vendor_support [17026.047826] ? hrtimer_interrupt+0x104/0x240 [17026.050961] i2c_smbus crct10dif_pclmul [17026.055484] ? __local_bh_enable+0x50/0x60 [17026.059488] joydev [17026.063765] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17026.067592] crc32_pclmul ipmi_ssif [17026.071682] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17026.073781] ghash_clmulni_intel mei_me [17026.078997] [17026.082482] mei lpc_ich [17026.087439] [17026.091270] sha512_ssse3 acpi_ipmi [17026.093369] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17026.095898] mfd_core wmi [17026.097999] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17026.101481] ipmi_si [17026.106786] do_raw_spin_lock+0xb5/0xc0 [17026.109402] ipmi_devintf ipmi_msghandler ioatdma [17026.114274] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17026.116455] acpi_pad acpi_power_meter [17026.120306] try_get_cap_refs+0x159/0x880 [ceph] [17026.125001] ixgbe [17026.129791] __ceph_get_caps+0x4a1/0x760 [ceph] [17026.133530] igb mdio [17026.138145] ? __file_remove_privs+0x89/0x110 [17026.140151] crc32c_intel nvme [17026.144675] ? wake_bit_function+0x70/0x70 [17026.146945] ptp [17026.151300] ? ceph_write_iter+0x260/0x770 [ceph] [17026.154346] nvme_core [17026.158437] ceph_write_iter+0x260/0x770 [ceph] [17026.160276] i2c_algo_bit pps_core [17026.164976] ? lock_is_held_type+0xaa/0x120 [17026.167329] dca fuse [17026.171857] vfs_write+0x324/0x4f0 [17026.175251] [17026.179433] ksys_pwrite64+0x65/0xa0 [17026.181699] irq event stamp: 59190550 [17026.185097] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17026.186588] hardirqs last enabled at (59190549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17026.190159] do_syscall_64+0x38/0x80 [17026.193815] hardirqs last disabled at (59190550): [] sysvec_apic_timer_interrupt+0xe/0x90 [17026.198948] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17026.209103] softirqs last enabled at (59190548): [] __do_softirq+0x2be/0x371 [17026.212675] RIP: 0033:0x7f48f0eeb93b [17026.222399] softirqs last disabled at (59190543): [] irq_exit_rcu+0xb0/0xd0 [17026.227443] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17026.236128] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17026.239697] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17026.248207] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17026.266946] ORIG_RAX: 0000000000000012 [17026.276324] Workqueue: events delayed_work [ceph] [17026.281541] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17026.289011] [17026.289013] RIP: 0010:lock_acquire+0xf0/0x2b0 [17026.292844] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17026.297539] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17026.304665] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17026.306154] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17026.310505] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17026.317630] [17026.317631] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17026.336367] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17026.343491] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17026.348714] [17026.355833] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17026.388006] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17026.395128] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17026.402254] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17026.410330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17026.416068] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17026.423192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17026.430317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17026.437440] Call Trace: [17026.439886] [17026.441898] ? watchdog_timer_fn+0x1b4/0x220 [17026.446170] ? __hrtimer_run_queues+0x1d3/0x360 [17026.450697] ? hrtimer_interrupt+0x104/0x240 [17026.454963] ? __local_bh_enable+0x50/0x60 [17026.459058] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17026.464281] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17026.469239] [17026.471336] [17026.473435] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17026.478743] ? lock_acquire+0xf0/0x2b0 [17026.482510] ? igrab+0x43/0x50 [17026.485559] ? lock_release+0xc5/0x260 [17026.489306] _raw_spin_lock+0x30/0x40 [17026.492969] ? igrab+0x1d/0x50 [17026.496019] igrab+0x1d/0x50 [17026.498898] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17026.504048] delayed_work+0x1d8/0x280 [ceph] [17026.508329] ? process_scheduled_works+0x22a/0x510 [17026.513119] process_scheduled_works+0x2a4/0x510 [17026.517736] worker_thread+0x188/0x340 [17026.521487] ? rescuer_thread+0x310/0x310 [17026.525497] kthread+0xf5/0x130 [17026.528642] ? kthread_complete_and_exit+0x20/0x20 [17026.533426] ret_from_fork+0x31/0x40 [17026.537007] ? kthread_complete_and_exit+0x20/0x20 [17026.541798] ret_from_fork_asm+0x11/0x20 [17026.545730] [17053.753584] watchdog: BUG: soft lockup - CPU#0 stuck for 4768s! [dbench:78683] [17053.760799] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17053.765584] watchdog: BUG: soft lockup - CPU#7 stuck for 13765s! [kworker/7:4:78808] [17053.816249] irq event stamp: 23496474 [17053.823996] Modules linked in: ceph libceph dns_resolver fscache netfs [17053.827653] hardirqs last enabled at (23496473): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17053.834171] veth nft_chain_nat xt_MASQUERADE [17053.844329] hardirqs last disabled at (23496474): [] sysvec_apic_timer_interrupt+0xe/0x90 [17053.848678] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [17053.858403] softirqs last enabled at (23496386): [] __do_softirq+0x2be/0x371 [17053.866047] nfnetlink bridge stp llc [17053.874732] softirqs last disabled at (23496381): [] irq_exit_rcu+0xb0/0xd0 [17053.874735] xfs libcrc32c sunrpc intel_rapl_msr [17053.886910] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17053.886911] intel_rapl_common [17053.886912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17053.891519] sb_edac [17053.900463] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17053.903513] x86_pkg_temp_thermal coretemp [17053.910984] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17053.913169] kvm_intel kvm [17053.918647] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17053.922737] irqbypass i2c_i801 [17053.941475] [17053.944180] iTCO_wdt iTCO_vendor_support [17053.949397] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17053.952532] i2c_smbus crct10dif_pclmul [17053.954025] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17053.958027] joydev crc32_pclmul [17053.965151] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17053.968983] ipmi_ssif ghash_clmulni_intel [17053.976106] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17053.979331] mei_me mei [17053.986456] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17053.990545] lpc_ich sha512_ssse3 [17053.997671] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17054.000113] acpi_ipmi mfd_core [17054.007238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17054.010549] wmi ipmi_si [17054.018627] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17054.021764] ipmi_devintf ipmi_msghandler [17054.027501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17054.030032] ioatdma acpi_pad [17054.037156] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17054.041161] acpi_power_meter ixgbe [17054.048285] Call Trace: [17054.051249] igb mdio [17054.058392] [17054.061873] crc32c_intel nvme [17054.064319] ? watchdog_timer_fn+0x1b4/0x220 [17054.066588] ptp nvme_core [17054.068602] ? __hrtimer_run_queues+0x1d3/0x360 [17054.071649] i2c_algo_bit pps_core [17054.075918] ? hrtimer_interrupt+0x104/0x240 [17054.078618] dca fuse [17054.083141] ? __local_bh_enable+0x50/0x60 [17054.086539] [17054.086540] irq event stamp: 59412376 [17054.090807] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17054.093074] hardirqs last enabled at (59412375): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17054.097165] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17054.098656] hardirqs last disabled at (59412376): [] sysvec_apic_timer_interrupt+0xe/0x90 [17054.102313] [17054.107531] softirqs last enabled at (59412374): [] __do_softirq+0x2be/0x371 [17054.117689] [17054.122647] softirqs last disabled at (59412369): [] irq_exit_rcu+0xb0/0xd0 [17054.132369] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17054.134468] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17054.143156] ? queued_spin_lock_slowpath+0x18/0x2e0 [17054.145266] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17054.153778] do_raw_spin_lock+0xb5/0xc0 [17054.159081] Workqueue: events delayed_work [ceph] [17054.168461] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17054.173330] [17054.173331] RIP: 0010:lock_release+0x11/0x260 [17054.180803] try_get_cap_refs+0x159/0x880 [ceph] [17054.184632] Code: 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd <53> 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f [17054.189334] __ceph_get_caps+0x4a1/0x760 [ceph] [17054.194113] RSP: 0018:ffffc90002117d68 EFLAGS: 00000286 [17054.195606] ? __file_remove_privs+0x89/0x110 [17054.199954] [17054.199955] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [17054.204565] ? wake_bit_function+0x70/0x70 [17054.223302] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [17054.227828] ? ceph_write_iter+0x260/0x770 [ceph] [17054.233044] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [17054.237396] ceph_write_iter+0x260/0x770 [ceph] [17054.238886] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [17054.246013] ? lock_is_held_type+0xaa/0x120 [17054.250099] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17054.257226] vfs_write+0x324/0x4f0 [17054.261923] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17054.269051] ksys_pwrite64+0x65/0xa0 [17054.273571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17054.280696] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17054.284870] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17054.291996] do_syscall_64+0x38/0x80 [17054.295393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17054.303471] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17054.307040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17054.312779] RIP: 0033:0x7f48f0eeb93b [17054.317910] Call Trace: [17054.317911] [17054.325035] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17054.328604] ? watchdog_timer_fn+0x1b4/0x220 [17054.335729] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17054.340774] ? __hrtimer_run_queues+0x1d3/0x360 [17054.347897] ORIG_RAX: 0000000000000012 [17054.351470] ? hrtimer_interrupt+0x104/0x240 [17054.353911] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17054.355923] ? __local_bh_enable+0x50/0x60 [17054.374661] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17054.378927] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17054.384140] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17054.388667] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17054.392497] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17054.396761] [17054.403884] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17054.407987] [17054.415112] [17054.420325] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17054.458340] ? igrab+0x43/0x50 [17054.461391] ? igrab+0x43/0x50 [17054.464441] ? lock_release+0x11/0x260 [17054.468187] _raw_spin_unlock+0x1b/0x40 [17054.472024] igrab+0x43/0x50 [17054.474901] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17054.480052] delayed_work+0x1d8/0x280 [ceph] [17054.484332] ? process_scheduled_works+0x22a/0x510 [17054.489121] process_scheduled_works+0x2a4/0x510 [17054.493738] worker_thread+0x188/0x340 [17054.497489] ? rescuer_thread+0x310/0x310 [17054.501498] kthread+0xf5/0x130 [17054.504637] ? kthread_complete_and_exit+0x20/0x20 [17054.509421] ret_from_fork+0x31/0x40 [17054.513002] ? kthread_complete_and_exit+0x20/0x20 [17054.517794] ret_from_fork_asm+0x11/0x20 [17054.521726] [17081.754089] watchdog: BUG: soft lockup - CPU#0 stuck for 4794s! [dbench:78683] [17081.761304] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17081.766088] watchdog: BUG: soft lockup - CPU#7 stuck for 13791s! [kworker/7:4:78808] [17081.816761] irq event stamp: 23554594 [17081.824492] Modules linked in: ceph libceph dns_resolver fscache netfs veth [17081.828149] hardirqs last enabled at (23554593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17081.835109] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [17081.845267] hardirqs last disabled at (23554594): [] sysvec_apic_timer_interrupt+0xe/0x90 [17081.853517] nft_compat nf_tables nfnetlink bridge stp llc [17081.863240] softirqs last enabled at (23554536): [] __do_softirq+0x2be/0x371 [17081.863245] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [17081.877402] softirqs last disabled at (23554531): [] irq_exit_rcu+0xb0/0xd0 [17081.886868] kvm_intel kvm irqbypass [17081.895380] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17081.898948] i2c_i801 iTCO_wdt [17081.907894] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17081.907896] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17081.910944] iTCO_vendor_support i2c_smbus [17081.918414] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17081.923892] crct10dif_pclmul joydev [17081.927982] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17081.946719] crc32_pclmul [17081.950292] [17081.950293] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17081.955508] ipmi_ssif [17081.958127] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17081.959617] ghash_clmulni_intel mei_me [17081.966740] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17081.969116] mei lpc_ich [17081.976238] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17081.980071] sha512_ssse3 [17081.987195] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17081.989726] acpi_ipmi mfd_core [17081.996849] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17081.999484] wmi [17082.006607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17082.009761] ipmi_si ipmi_devintf [17082.017839] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17082.019678] ipmi_msghandler ioatdma [17082.025414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17082.028726] acpi_pad [17082.035849] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17082.039419] acpi_power_meter ixgbe [17082.046544] Call Trace: [17082.048816] igb [17082.055943] [17082.059424] mdio crc32c_intel [17082.061870] ? watchdog_timer_fn+0x1b4/0x220 [17082.063704] nvme ptp [17082.065716] ? __hrtimer_run_queues+0x1d3/0x360 [17082.068767] nvme_core [17082.073034] ? hrtimer_interrupt+0x104/0x240 [17082.075302] i2c_algo_bit [17082.079826] ? __local_bh_enable+0x50/0x60 [17082.082182] pps_core [17082.086449] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17082.089064] dca fuse [17082.093156] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17082.095425] [17082.095426] irq event stamp: 59634488 [17082.100644] [17082.102914] hardirqs last enabled at (59634487): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17082.107870] [17082.109361] hardirqs last disabled at (59634488): [] sysvec_apic_timer_interrupt+0xe/0x90 [17082.113020] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17082.115117] softirqs last enabled at (59634486): [] __do_softirq+0x2be/0x371 [17082.125279] ? queued_spin_lock_slowpath+0x16/0x2e0 [17082.127372] softirqs last disabled at (59634481): [] irq_exit_rcu+0xb0/0xd0 [17082.137098] do_raw_spin_lock+0xb5/0xc0 [17082.142400] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17082.151086] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17082.155956] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17082.164487] try_get_cap_refs+0x159/0x880 [ceph] [17082.168315] Workqueue: events delayed_work [ceph] [17082.177696] __ceph_get_caps+0x4a1/0x760 [ceph] [17082.182489] [17082.189970] ? __file_remove_privs+0x89/0x110 [17082.194573] RIP: 0010:lock_acquire+0xf0/0x2b0 [17082.199271] ? wake_bit_function+0x70/0x70 [17082.203795] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17082.205288] ? ceph_write_iter+0x260/0x770 [ceph] [17082.209636] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17082.213989] ceph_write_iter+0x260/0x770 [ceph] [17082.218078] [17082.218079] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17082.236820] ? lock_is_held_type+0xaa/0x120 [17082.241514] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17082.246733] vfs_write+0x324/0x4f0 [17082.251255] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17082.252768] ksys_pwrite64+0x65/0xa0 [17082.259888] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17082.264065] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17082.271188] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17082.274588] do_syscall_64+0x38/0x80 [17082.281711] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17082.285283] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17082.292405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17082.297538] RIP: 0033:0x7f48f0eeb93b [17082.304661] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17082.308233] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17082.316309] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17082.321353] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17082.327089] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17082.330661] ORIG_RAX: 0000000000000012 [17082.337785] Call Trace: [17082.337787] [17082.356522] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17082.363647] ? watchdog_timer_fn+0x1b4/0x220 [17082.368865] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17082.375988] ? __hrtimer_run_queues+0x1d3/0x360 [17082.379820] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17082.382267] ? hrtimer_interrupt+0x104/0x240 [17082.384273] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17082.391397] ? __local_bh_enable+0x50/0x60 [17082.395660] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17082.402789] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17082.407316] [17082.414434] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17082.449396] [17082.451494] [17082.453592] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17082.458899] ? lock_acquire+0xf0/0x2b0 [17082.462650] ? igrab+0x43/0x50 [17082.465710] ? lock_release+0xc5/0x260 [17082.469454] _raw_spin_lock+0x30/0x40 [17082.473119] ? igrab+0x1d/0x50 [17082.476167] igrab+0x1d/0x50 [17082.479047] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17082.484196] delayed_work+0x1d8/0x280 [ceph] [17082.488496] ? process_scheduled_works+0x22a/0x510 [17082.493286] process_scheduled_works+0x2a4/0x510 [17082.497903] worker_thread+0x188/0x340 [17082.501650] ? rescuer_thread+0x310/0x310 [17082.505652] kthread+0xf5/0x130 [17082.508792] ? kthread_complete_and_exit+0x20/0x20 [17082.513576] ret_from_fork+0x31/0x40 [17082.517156] ? kthread_complete_and_exit+0x20/0x20 [17082.521947] ret_from_fork_asm+0x11/0x20 [17082.525879] [17083.697489] rcu: INFO: rcu_preempt self-detected stall on CPU [17083.703228] rcu: 0-....: (5032719 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=2496501 [17083.713566] rcu: (t=5146997 jiffies g=1309609 q=218925 ncpus=8) [17083.719564] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17083.728519] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17083.735988] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17083.741472] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17083.760211] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17083.765428] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17083.772551] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17083.779677] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17083.786800] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17083.793924] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17083.801050] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17083.809125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17083.814863] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17083.821988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17083.829111] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17083.836237] Call Trace: [17083.838680] [17083.840692] ? rcu_dump_cpu_stacks+0xd1/0x180 [17083.845052] ? rcu_sched_clock_irq+0x633/0x1390 [17083.849576] ? lock_is_held_type+0xaa/0x120 [17083.853753] ? lock_is_held_type+0xaa/0x120 [17083.857932] ? tick_sched_do_timer+0xa0/0xa0 [17083.862202] ? rcu_read_lock_sched_held+0x56/0x90 [17083.866901] ? tick_sched_do_timer+0xa0/0xa0 [17083.871163] ? update_process_times+0x5f/0x90 [17083.875519] ? tick_sched_handle.isra.16+0x1e/0x50 [17083.880306] ? tick_sched_timer+0x6f/0x80 [17083.884312] ? __hrtimer_run_queues+0x1d3/0x360 [17083.888839] ? hrtimer_interrupt+0x104/0x240 [17083.893107] ? __local_bh_enable+0x50/0x60 [17083.897202] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17083.902426] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17083.907382] [17083.909479] [17083.911577] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17083.916884] ? queued_spin_lock_slowpath+0x18/0x2e0 [17083.921762] do_raw_spin_lock+0xb5/0xc0 [17083.925591] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17083.930394] try_get_cap_refs+0x159/0x880 [ceph] [17083.935024] __ceph_get_caps+0x4a1/0x760 [ceph] [17083.939567] ? __file_remove_privs+0x89/0x110 [17083.943921] ? wake_bit_function+0x70/0x70 [17083.948015] ? ceph_write_iter+0x260/0x770 [ceph] [17083.952728] ceph_write_iter+0x260/0x770 [ceph] [17083.957263] ? lock_is_held_type+0xaa/0x120 [17083.961448] vfs_write+0x324/0x4f0 [17083.964857] ksys_pwrite64+0x65/0xa0 [17083.968430] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17083.973563] do_syscall_64+0x38/0x80 [17083.977133] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17083.982176] RIP: 0033:0x7f48f0eeb93b [17083.985747] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17084.004484] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [17084.012042] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17084.019166] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17084.026290] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17084.033415] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17084.040540] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17084.047670] [17084.049857] Sending NMI from CPU 0 to CPUs 7: [17084.054208] NMI backtrace for cpu 7 [17084.054210] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17084.054213] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17084.054214] Workqueue: events delayed_work [ceph] [17084.054234] RIP: 0010:__lock_acquire+0x265/0x1e20 [17084.054237] Code: 84 04 02 00 00 0f b6 75 21 40 84 f6 0f 88 6c 06 00 00 44 0f b6 b8 c9 00 00 00 8b bb 50 0a 00 00 45 84 ff 45 0f 44 fd 41 89 fd <41> 83 ed 01 0f 88 34 09 00 00 49 63 c5 48 8b 14 24 48 8d 04 80 0f [17084.054239] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [17084.054240] RAX: ffffffff82fcfc30 RBX: ffff88810debaa80 RCX: 0000000000000004 [17084.054242] RDX: 00000000000014df RSI: 0000000000000006 RDI: 0000000000000003 [17084.054243] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [17084.054244] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [17084.054245] R13: 0000000000000003 R14: 0000000000000006 R15: 0000000000000002 [17084.054247] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17084.054248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17084.054250] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17084.054251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17084.054252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17084.054253] Call Trace: [17084.054254] [17084.054255] ? nmi_cpu_backtrace+0xad/0x120 [17084.054260] ? nmi_cpu_backtrace_handler+0x11/0x20 [17084.054262] ? nmi_handle+0xd4/0x260 [17084.054267] ? default_do_nmi+0x49/0xf0 [17084.054270] ? exc_nmi+0x11e/0x150 [17084.054272] ? end_repeat_nmi+0x16/0x67 [17084.054278] ? __lock_acquire+0x265/0x1e20 [17084.054280] ? __lock_acquire+0x265/0x1e20 [17084.054283] ? __lock_acquire+0x265/0x1e20 [17084.054285] [17084.054286] [17084.054289] lock_acquire+0xcb/0x2b0 [17084.054291] ? igrab+0x1d/0x50 [17084.054294] ? igrab+0x43/0x50 [17084.054296] ? lock_release+0xc5/0x260 [17084.054299] _raw_spin_lock+0x30/0x40 [17084.054301] ? igrab+0x1d/0x50 [17084.054303] igrab+0x1d/0x50 [17084.054305] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17084.054324] delayed_work+0x1d8/0x280 [ceph] [17084.054342] ? process_scheduled_works+0x22a/0x510 [17084.054345] process_scheduled_works+0x2a4/0x510 [17084.054351] worker_thread+0x188/0x340 [17084.054354] ? rescuer_thread+0x310/0x310 [17084.054357] kthread+0xf5/0x130 [17084.054360] ? kthread_complete_and_exit+0x20/0x20 [17084.054363] ret_from_fork+0x31/0x40 [17084.054366] ? kthread_complete_and_exit+0x20/0x20 [17084.054368] ret_from_fork_asm+0x11/0x20 [17084.054374] [17109.754591] watchdog: BUG: soft lockup - CPU#0 stuck for 4820s! [dbench:78683] [17109.761809] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17109.766591] watchdog: BUG: soft lockup - CPU#7 stuck for 13817s! [kworker/7:4:78808] [17109.817265] irq event stamp: 23611974 [17109.817266] hardirqs last enabled at (23611973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17109.824996] Modules linked in: [17109.828654] hardirqs last disabled at (23611974): [] sysvec_apic_timer_interrupt+0xe/0x90 [17109.838811] ceph [17109.841864] softirqs last enabled at (23611868): [] __do_softirq+0x2be/0x371 [17109.851588] libceph dns_resolver fscache [17109.853512] softirqs last disabled at (23611863): [] irq_exit_rcu+0xb0/0xd0 [17109.862196] netfs veth [17109.866198] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17109.874711] nft_chain_nat xt_MASQUERADE [17109.877153] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17109.886098] nf_nat nf_conntrack [17109.890016] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17109.897486] nf_defrag_ipv6 nf_defrag_ipv4 [17109.900710] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17109.906187] nft_compat nf_tables [17109.910280] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17109.929017] nfnetlink bridge [17109.932327] [17109.937545] stp llc [17109.940509] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17109.941999] xfs libcrc32c [17109.944183] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17109.951306] sunrpc intel_rapl_msr [17109.954012] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17109.961135] intel_rapl_common sb_edac [17109.964532] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17109.971658] x86_pkg_temp_thermal [17109.975401] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17109.982524] coretemp kvm_intel [17109.985837] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17109.992960] kvm irqbypass [17109.996097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17110.004175] i2c_i801 iTCO_wdt [17110.006880] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17110.012617] iTCO_vendor_support i2c_smbus [17110.015666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17110.022791] crct10dif_pclmul joydev [17110.026882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17110.034007] crc32_pclmul ipmi_ssif [17110.037577] Call Trace: [17110.037578] [17110.044702] ghash_clmulni_intel mei_me [17110.048187] ? watchdog_timer_fn+0x1b4/0x220 [17110.050629] mei [17110.052642] ? __hrtimer_run_queues+0x1d3/0x360 [17110.056470] lpc_ich sha512_ssse3 [17110.060739] ? hrtimer_interrupt+0x104/0x240 [17110.062573] acpi_ipmi mfd_core [17110.067096] ? __local_bh_enable+0x50/0x60 [17110.070408] wmi ipmi_si [17110.074674] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17110.077809] ipmi_devintf [17110.081900] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17110.084429] ipmi_msghandler ioatdma [17110.089649] [17110.092272] acpi_pad acpi_power_meter [17110.097230] [17110.097231] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17110.100802] ixgbe igb [17110.102903] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17110.106643] mdio crc32c_intel [17110.108743] do_raw_spin_lock+0xb5/0xc0 [17110.114044] nvme ptp [17110.116403] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17110.121274] nvme_core i2c_algo_bit [17110.124327] try_get_cap_refs+0x159/0x880 [ceph] [17110.128154] pps_core dca [17110.130431] __ceph_get_caps+0x4a1/0x760 [ceph] [17110.135209] fuse [17110.138696] ? __file_remove_privs+0x89/0x110 [17110.143304] irq event stamp: 59854194 [17110.145922] ? wake_bit_function+0x70/0x70 [17110.150445] hardirqs last enabled at (59854193): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17110.152372] ? ceph_write_iter+0x260/0x770 [ceph] [17110.156720] hardirqs last disabled at (59854194): [] sysvec_apic_timer_interrupt+0xe/0x90 [17110.160377] ceph_write_iter+0x260/0x770 [ceph] [17110.164468] softirqs last enabled at (59854192): [] __do_softirq+0x2be/0x371 [17110.174630] ? lock_is_held_type+0xaa/0x120 [17110.179323] softirqs last disabled at (59854187): [] irq_exit_rcu+0xb0/0xd0 [17110.189051] vfs_write+0x324/0x4f0 [17110.193572] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17110.202275] ksys_pwrite64+0x65/0xa0 [17110.206449] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17110.214962] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17110.218359] Workqueue: events delayed_work [ceph] [17110.227737] do_syscall_64+0x38/0x80 [17110.231306] [17110.238778] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17110.243908] RIP: 0010:ceph_check_delayed_caps+0xa1/0x270 [ceph] [17110.248605] RIP: 0033:0x7f48f0eeb93b [17110.252175] Code: e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 <48> 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 [17110.253669] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17110.258711] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [17110.264622] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17110.268191] [17110.268192] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [17110.286931] ORIG_RAX: 0000000000000012 [17110.305668] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [17110.310884] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17110.316102] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [17110.317592] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17110.324717] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17110.328549] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17110.335673] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17110.342798] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17110.349920] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17110.357044] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17110.364169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17110.371298] [17110.378416] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17110.415789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17110.422914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17110.430036] Call Trace: [17110.432481] [17110.434494] ? watchdog_timer_fn+0x1b4/0x220 [17110.438758] ? __hrtimer_run_queues+0x1d3/0x360 [17110.443284] ? hrtimer_interrupt+0x104/0x240 [17110.447554] ? __local_bh_enable+0x50/0x60 [17110.451646] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17110.456870] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17110.461828] [17110.463924] [17110.466023] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17110.471331] ? ceph_check_delayed_caps+0xa1/0x270 [ceph] [17110.476649] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [17110.481964] delayed_work+0x1d8/0x280 [ceph] [17110.486276] ? process_scheduled_works+0x22a/0x510 [17110.491059] process_scheduled_works+0x2a4/0x510 [17110.495677] worker_thread+0x188/0x340 [17110.499425] ? rescuer_thread+0x310/0x310 [17110.503427] kthread+0xf5/0x130 [17110.506566] ? kthread_complete_and_exit+0x20/0x20 [17110.511351] ret_from_fork+0x31/0x40 [17110.514919] ? kthread_complete_and_exit+0x20/0x20 [17110.519704] ret_from_fork_asm+0x11/0x20 [17110.523627] [17137.755095] watchdog: BUG: soft lockup - CPU#0 stuck for 4846s! [dbench:78683] [17137.762312] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17137.767095] watchdog: BUG: soft lockup - CPU#7 stuck for 13843s! [kworker/7:4:78808] [17137.817778] irq event stamp: 23670200 [17137.817779] hardirqs last enabled at (23670199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17137.825510] Modules linked in: [17137.829167] hardirqs last disabled at (23670200): [] sysvec_apic_timer_interrupt+0xe/0x90 [17137.839327] ceph [17137.842376] softirqs last enabled at (23670144): [] __do_softirq+0x2be/0x371 [17137.852100] libceph [17137.854024] softirqs last disabled at (23670139): [] irq_exit_rcu+0xb0/0xd0 [17137.862709] dns_resolver fscache [17137.864894] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17137.873404] netfs [17137.876714] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17137.885658] veth nft_chain_nat [17137.887671] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17137.895139] xt_MASQUERADE nf_nat [17137.898279] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17137.903755] nf_conntrack nf_defrag_ipv6 [17137.907084] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17137.925821] nf_defrag_ipv4 nft_compat [17137.929737] [17137.929738] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17137.934956] nf_tables [17137.938700] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17137.940190] nfnetlink bridge [17137.947314] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17137.949673] stp llc [17137.956797] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17137.959774] xfs libcrc32c [17137.966902] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17137.969085] sunrpc intel_rapl_msr [17137.976209] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17137.978914] intel_rapl_common [17137.986038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17137.989435] sb_edac x86_pkg_temp_thermal [17137.997514] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17138.000564] coretemp kvm_intel [17138.006300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17138.010304] kvm irqbypass [17138.017430] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17138.020566] i2c_i801 iTCO_wdt [17138.027690] Call Trace: [17138.030395] iTCO_vendor_support i2c_smbus [17138.037520] [17138.040569] crct10dif_pclmul joydev [17138.043015] ? watchdog_timer_fn+0x1b4/0x220 [17138.047105] crc32_pclmul [17138.049117] ? __hrtimer_run_queues+0x1d3/0x360 [17138.052687] ipmi_ssif ghash_clmulni_intel [17138.056952] ? hrtimer_interrupt+0x104/0x240 [17138.059568] mei_me [17138.064092] ? __local_bh_enable+0x50/0x60 [17138.068183] mei lpc_ich [17138.072447] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17138.074543] sha512_ssse3 [17138.078635] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17138.081166] acpi_ipmi [17138.086382] [17138.088999] mfd_core wmi [17138.093958] [17138.096315] ipmi_si ipmi_devintf [17138.098413] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17138.101029] ipmi_msghandler [17138.103131] ? queued_spin_lock_slowpath+0x18/0x2e0 [17138.106438] ioatdma acpi_pad [17138.111745] do_raw_spin_lock+0xb5/0xc0 [17138.114618] acpi_power_meter ixgbe [17138.119497] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17138.122454] igb mdio [17138.126286] try_get_cap_refs+0x159/0x880 [ceph] [17138.129776] crc32c_intel nvme [17138.134567] __ceph_get_caps+0x4a1/0x760 [ceph] [17138.136833] ptp [17138.141447] ? __file_remove_privs+0x89/0x110 [17138.144495] nvme_core [17138.149019] ? wake_bit_function+0x70/0x70 [17138.150855] i2c_algo_bit pps_core [17138.155209] ? ceph_write_iter+0x260/0x770 [ceph] [17138.157562] dca fuse [17138.161655] ceph_write_iter+0x260/0x770 [ceph] [17138.165051] [17138.169753] ? lock_is_held_type+0xaa/0x120 [17138.172020] irq event stamp: 60075742 [17138.176547] vfs_write+0x324/0x4f0 [17138.178035] hardirqs last enabled at (60075741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17138.182215] ksys_pwrite64+0x65/0xa0 [17138.185869] hardirqs last disabled at (60075742): [] sysvec_apic_timer_interrupt+0xe/0x90 [17138.189268] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17138.199423] softirqs last enabled at (60075740): [] __do_softirq+0x2be/0x371 [17138.202995] do_syscall_64+0x38/0x80 [17138.212717] softirqs last disabled at (60075735): [] irq_exit_rcu+0xb0/0xd0 [17138.217851] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17138.226535] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17138.230105] RIP: 0033:0x7f48f0eeb93b [17138.238615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17138.243661] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17138.253036] Workqueue: events delayed_work [ceph] [17138.256606] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17138.264078] [17138.264079] RIP: 0010:lock_release+0x78/0x260 [17138.282817] ORIG_RAX: 0000000000000012 [17138.287512] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [17138.292730] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17138.294221] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [17138.298573] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17138.302404] [17138.302405] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [17138.321141] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17138.328266] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [17138.333498] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17138.340624] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [17138.342114] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17138.349237] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [17138.356368] [17138.363500] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17138.401301] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17138.409378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17138.415116] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17138.422240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17138.429363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17138.436501] Call Trace: [17138.438952] [17138.440961] ? watchdog_timer_fn+0x1b4/0x220 [17138.445225] ? __hrtimer_run_queues+0x1d3/0x360 [17138.449752] ? hrtimer_interrupt+0x104/0x240 [17138.454022] ? __local_bh_enable+0x50/0x60 [17138.458115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17138.463338] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17138.468296] [17138.470393] [17138.472501] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17138.477812] ? igrab+0x43/0x50 [17138.480866] ? lock_release+0x78/0x260 [17138.484619] _raw_spin_unlock+0x1b/0x40 [17138.488456] igrab+0x43/0x50 [17138.491334] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17138.496503] delayed_work+0x1d8/0x280 [ceph] [17138.500792] ? process_scheduled_works+0x22a/0x510 [17138.505580] process_scheduled_works+0x2a4/0x510 [17138.510196] worker_thread+0x188/0x340 [17138.513944] ? rescuer_thread+0x310/0x310 [17138.517948] kthread+0xf5/0x130 [17138.521087] ? kthread_complete_and_exit+0x20/0x20 [17138.525870] ret_from_fork+0x31/0x40 [17138.529448] ? kthread_complete_and_exit+0x20/0x20 [17138.534233] ret_from_fork_asm+0x11/0x20 [17138.538155] [17165.755611] watchdog: BUG: soft lockup - CPU#0 stuck for 4872s! [dbench:78683] [17165.762826] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17165.767610] watchdog: BUG: soft lockup - CPU#7 stuck for 13869s! [kworker/7:4:78808] [17165.818284] irq event stamp: 23728408 [17165.826016] Modules linked in: ceph libceph dns_resolver [17165.829674] hardirqs last enabled at (23728407): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17165.829678] hardirqs last disabled at (23728408): [] sysvec_apic_timer_interrupt+0xe/0x90 [17165.834979] fscache netfs [17165.845153] softirqs last enabled at (23728322): [] __do_softirq+0x2be/0x371 [17165.854878] veth nft_chain_nat [17165.857581] softirqs last disabled at (23728317): [] irq_exit_rcu+0xb0/0xd0 [17165.866265] xt_MASQUERADE nf_nat [17165.869419] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17165.877931] nf_conntrack nf_defrag_ipv6 [17165.881242] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17165.890187] nf_defrag_ipv4 nft_compat [17165.894105] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17165.901574] nf_tables nfnetlink [17165.905319] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17165.910795] bridge stp [17165.914018] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17165.932757] llc xfs [17165.935200] [17165.940418] libcrc32c sunrpc [17165.942602] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17165.944092] intel_rapl_msr intel_rapl_common [17165.947057] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17165.954181] sb_edac x86_pkg_temp_thermal [17165.958533] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17165.965656] coretemp kvm_intel [17165.969660] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17165.976784] kvm irqbypass [17165.979921] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17165.987045] i2c_i801 iTCO_wdt [17165.989751] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17165.996874] iTCO_vendor_support i2c_smbus [17165.999924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17166.008007] crct10dif_pclmul joydev [17166.012102] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17166.017840] crc32_pclmul ipmi_ssif [17166.021410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17166.028533] ghash_clmulni_intel mei_me [17166.032019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17166.039142] mei lpc_ich [17166.042974] Call Trace: [17166.042975] [17166.050096] sha512_ssse3 [17166.052628] ? watchdog_timer_fn+0x1b4/0x220 [17166.055072] acpi_ipmi mfd_core [17166.057082] ? __hrtimer_run_queues+0x1d3/0x360 [17166.059699] wmi ipmi_si [17166.063968] ? hrtimer_interrupt+0x104/0x240 [17166.067101] ipmi_devintf ipmi_msghandler [17166.071626] ? __local_bh_enable+0x50/0x60 [17166.074156] ioatdma acpi_pad [17166.078423] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17166.082425] acpi_power_meter ixgbe [17166.086516] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17166.089478] igb mdio crc32c_intel [17166.094698] [17166.098181] nvme ptp [17166.103139] [17166.106535] nvme_core [17166.108633] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17166.110903] i2c_algo_bit pps_core [17166.113015] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17166.115368] dca fuse [17166.120673] do_raw_spin_lock+0xb5/0xc0 [17166.124068] [17166.124069] irq event stamp: 60296966 [17166.128941] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17166.131209] hardirqs last enabled at (60296965): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17166.135042] try_get_cap_refs+0x159/0x880 [ceph] [17166.136531] hardirqs last disabled at (60296966): [] sysvec_apic_timer_interrupt+0xe/0x90 [17166.140194] __ceph_get_caps+0x4a1/0x760 [ceph] [17166.144973] softirqs last enabled at (60296964): [] __do_softirq+0x2be/0x371 [17166.155133] ? __file_remove_privs+0x89/0x110 [17166.159741] softirqs last disabled at (60296959): [] irq_exit_rcu+0xb0/0xd0 [17166.169466] ? wake_bit_function+0x70/0x70 [17166.173989] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17166.182695] ? ceph_write_iter+0x260/0x770 [ceph] [17166.187042] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17166.195552] ceph_write_iter+0x260/0x770 [ceph] [17166.199643] Workqueue: events delayed_work [ceph] [17166.209026] ? lock_is_held_type+0xaa/0x120 [17166.213717] [17166.221192] vfs_write+0x324/0x4f0 [17166.225714] RIP: 0010:lock_acquire+0xf0/0x2b0 [17166.230417] ksys_pwrite64+0x65/0xa0 [17166.234589] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17166.236079] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17166.239476] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17166.243827] do_syscall_64+0x38/0x80 [17166.247397] [17166.247398] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17166.266135] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17166.271265] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17166.276485] RIP: 0033:0x7f48f0eeb93b [17166.280054] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17166.281546] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17166.288669] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17166.293712] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17166.300837] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17166.304408] ORIG_RAX: 0000000000000012 [17166.311532] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17166.330270] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17166.337411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17166.342629] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17166.349753] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17166.353583] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17166.361661] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17166.368784] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17166.374522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17166.381646] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17166.388771] Call Trace: [17166.395901] [17166.403021] [17166.431030] ? watchdog_timer_fn+0x1b4/0x220 [17166.435298] ? __hrtimer_run_queues+0x1d3/0x360 [17166.439830] ? hrtimer_interrupt+0x104/0x240 [17166.444101] ? __local_bh_enable+0x50/0x60 [17166.448195] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17166.453418] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17166.458376] [17166.460473] [17166.462570] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17166.467885] ? lock_acquire+0xf0/0x2b0 [17166.471636] ? igrab+0x43/0x50 [17166.474686] ? lock_release+0xc5/0x260 [17166.478430] _raw_spin_lock+0x30/0x40 [17166.482088] ? igrab+0x1d/0x50 [17166.485137] igrab+0x1d/0x50 [17166.488020] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17166.493165] delayed_work+0x1d8/0x280 [ceph] [17166.497447] ? process_scheduled_works+0x22a/0x510 [17166.502238] process_scheduled_works+0x2a4/0x510 [17166.506853] worker_thread+0x188/0x340 [17166.510601] ? rescuer_thread+0x310/0x310 [17166.514604] kthread+0xf5/0x130 [17166.517743] ? kthread_complete_and_exit+0x20/0x20 [17166.522527] ret_from_fork+0x31/0x40 [17166.526106] ? kthread_complete_and_exit+0x20/0x20 [17166.530890] ret_from_fork_asm+0x11/0x20 [17166.534813] [17193.756127] watchdog: BUG: soft lockup - CPU#0 stuck for 4898s! [dbench:78683] [17193.763342] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17193.768127] watchdog: BUG: soft lockup - CPU#7 stuck for 13895s! [kworker/7:4:78808] [17193.818800] irq event stamp: 23786654 [17193.818801] hardirqs last enabled at (23786653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17193.826530] Modules linked in: [17193.830189] hardirqs last disabled at (23786654): [] sysvec_apic_timer_interrupt+0xe/0x90 [17193.840344] ceph [17193.843396] softirqs last enabled at (23786648): [] __do_softirq+0x2be/0x371 [17193.853120] libceph [17193.855043] softirqs last disabled at (23786643): [] irq_exit_rcu+0xb0/0xd0 [17193.863727] dns_resolver [17193.865913] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17193.874423] fscache [17193.877041] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17193.885986] netfs [17193.888169] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17193.895641] veth nft_chain_nat [17193.897650] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17193.903129] xt_MASQUERADE nf_nat [17193.906264] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17193.925002] nf_conntrack [17193.928314] [17193.928315] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17193.933531] nf_defrag_ipv6 [17193.936147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17193.937639] nf_defrag_ipv4 nft_compat [17193.944763] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17193.947554] nf_tables nfnetlink [17193.954679] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17193.958421] bridge stp [17193.965545] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17193.968770] llc [17193.975895] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17193.978339] xfs [17193.985464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17193.987299] libcrc32c sunrpc [17193.995377] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17193.997216] intel_rapl_msr [17194.002952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17194.005916] intel_rapl_common [17194.013041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17194.015832] sb_edac x86_pkg_temp_thermal [17194.022956] Call Trace: [17194.022958] [17194.026007] coretemp kvm_intel [17194.033134] ? watchdog_timer_fn+0x1b4/0x220 [17194.037134] kvm irqbypass [17194.039580] ? __hrtimer_run_queues+0x1d3/0x360 [17194.041590] i2c_i801 iTCO_wdt [17194.044729] ? hrtimer_interrupt+0x104/0x240 [17194.048992] iTCO_vendor_support i2c_smbus [17194.051695] ? __local_bh_enable+0x50/0x60 [17194.056217] crct10dif_pclmul joydev [17194.059272] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17194.063532] crc32_pclmul ipmi_ssif [17194.067625] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17194.071715] ghash_clmulni_intel [17194.075287] [17194.080504] mei_me mei [17194.084004] [17194.084005] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17194.088962] lpc_ich [17194.092191] ? queued_spin_lock_slowpath+0x18/0x2e0 [17194.094284] sha512_ssse3 [17194.096729] do_raw_spin_lock+0xb5/0xc0 [17194.098826] acpi_ipmi mfd_core [17194.104132] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17194.106313] wmi ipmi_si [17194.111187] try_get_cap_refs+0x159/0x880 [ceph] [17194.113814] ipmi_devintf ipmi_msghandler [17194.117656] __ceph_get_caps+0x4a1/0x760 [ceph] [17194.120787] ioatdma acpi_pad [17194.125574] ? __file_remove_privs+0x89/0x110 [17194.128101] acpi_power_meter ixgbe [17194.132712] ? wake_bit_function+0x70/0x70 [17194.136716] igb mdio [17194.141242] ? ceph_write_iter+0x260/0x770 [ceph] [17194.144205] crc32c_intel nvme [17194.148555] ceph_write_iter+0x260/0x770 [ceph] [17194.152038] ptp nvme_core [17194.156133] ? lock_is_held_type+0xaa/0x120 [17194.158401] i2c_algo_bit pps_core [17194.163101] vfs_write+0x324/0x4f0 [17194.166149] dca fuse [17194.170678] ksys_pwrite64+0x65/0xa0 [17194.173375] [17194.173376] irq event stamp: 60518240 [17194.177556] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17194.180951] hardirqs last enabled at (60518239): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17194.184350] do_syscall_64+0x38/0x80 [17194.186618] hardirqs last disabled at (60518240): [] sysvec_apic_timer_interrupt+0xe/0x90 [17194.190192] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17194.191682] softirqs last enabled at (60518238): [] __do_softirq+0x2be/0x371 [17194.195340] RIP: 0033:0x7f48f0eeb93b [17194.200468] softirqs last disabled at (60518233): [] irq_exit_rcu+0xb0/0xd0 [17194.210627] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17194.214197] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17194.223922] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17194.228965] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17194.237650] ORIG_RAX: 0000000000000012 [17194.241220] Workqueue: events delayed_work [ceph] [17194.249732] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17194.268468] [17194.277847] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17194.283063] RIP: 0010:lock_acquire+0xf0/0x2b0 [17194.290534] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17194.294365] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17194.299064] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17194.306186] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17194.307678] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17194.314808] [17194.319166] [17194.326286] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17194.375289] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17194.382411] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17194.389536] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17194.396660] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17194.403786] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17194.411878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17194.417618] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17194.424741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17194.431866] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17194.438988] Call Trace: [17194.441435] [17194.443444] ? watchdog_timer_fn+0x1b4/0x220 [17194.447709] ? __hrtimer_run_queues+0x1d3/0x360 [17194.452235] ? hrtimer_interrupt+0x104/0x240 [17194.456505] ? __local_bh_enable+0x50/0x60 [17194.460615] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17194.465839] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17194.470813] [17194.472911] [17194.475009] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17194.480318] ? lock_acquire+0xf0/0x2b0 [17194.484067] ? igrab+0x43/0x50 [17194.487118] ? lock_release+0xc5/0x260 [17194.490862] _raw_spin_lock+0x30/0x40 [17194.494534] ? igrab+0x1d/0x50 [17194.497586] igrab+0x1d/0x50 [17194.500465] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17194.505613] delayed_work+0x1d8/0x280 [ceph] [17194.509897] ? process_scheduled_works+0x22a/0x510 [17194.514686] process_scheduled_works+0x2a4/0x510 [17194.519301] worker_thread+0x188/0x340 [17194.523051] ? rescuer_thread+0x310/0x310 [17194.527061] kthread+0xf5/0x130 [17194.530200] ? kthread_complete_and_exit+0x20/0x20 [17194.534984] ret_from_fork+0x31/0x40 [17194.538562] ? kthread_complete_and_exit+0x20/0x20 [17194.543348] ret_from_fork_asm+0x11/0x20 [17194.547280] [17208.232974] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17218.472674] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17221.756655] watchdog: BUG: soft lockup - CPU#0 stuck for 4925s! [dbench:78683] [17221.763871] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17221.768655] watchdog: BUG: soft lockup - CPU#7 stuck for 13922s! [kworker/7:4:78808] [17221.819314] irq event stamp: 23844964 [17221.819315] hardirqs last enabled at (23844963): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17221.827061] Modules linked in: [17221.830719] hardirqs last disabled at (23844964): [] sysvec_apic_timer_interrupt+0xe/0x90 [17221.840876] ceph [17221.843927] softirqs last enabled at (23844958): [] __do_softirq+0x2be/0x371 [17221.853652] libceph dns_resolver [17221.855576] softirqs last disabled at (23844953): [] irq_exit_rcu+0xb0/0xd0 [17221.864260] fscache netfs [17221.867571] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17221.876081] veth nft_chain_nat [17221.878784] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17221.878785] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17221.887729] xt_MASQUERADE [17221.890867] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17221.898347] nf_nat nf_conntrack [17221.903822] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17221.906528] nf_defrag_ipv6 nf_defrag_ipv4 [17221.925266] [17221.928490] nft_compat nf_tables [17221.933706] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17221.937799] nfnetlink bridge [17221.939289] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17221.942600] stp llc [17221.949723] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17221.952688] xfs libcrc32c [17221.959812] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17221.961995] sunrpc intel_rapl_msr [17221.969120] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17221.971823] intel_rapl_common [17221.978946] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17221.982345] sb_edac x86_pkg_temp_thermal [17221.989470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17221.992520] coretemp kvm_intel [17222.000597] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17222.004603] kvm irqbypass [17222.010339] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17222.013476] i2c_i801 iTCO_wdt [17222.020600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17222.023304] iTCO_vendor_support i2c_smbus [17222.030429] Call Trace: [17222.030430] [17222.033479] crct10dif_pclmul joydev [17222.040605] ? watchdog_timer_fn+0x1b4/0x220 [17222.044694] crc32_pclmul ipmi_ssif [17222.047140] ? __hrtimer_run_queues+0x1d3/0x360 [17222.049148] ghash_clmulni_intel mei_me [17222.052723] ? hrtimer_interrupt+0x104/0x240 [17222.056985] mei lpc_ich [17222.060469] ? __local_bh_enable+0x50/0x60 [17222.064991] sha512_ssse3 acpi_ipmi [17222.068824] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17222.073086] mfd_core wmi [17222.075619] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17222.079707] ipmi_si ipmi_devintf [17222.083193] [17222.088408] ipmi_msghandler ioatdma [17222.091027] [17222.095984] acpi_pad acpi_power_meter [17222.099297] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17222.101392] ixgbe igb [17222.104968] ? queued_spin_lock_slowpath+0x16/0x2e0 [17222.107059] mdio crc32c_intel [17222.110807] do_raw_spin_lock+0xb5/0xc0 [17222.116108] nvme [17222.118469] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17222.123354] ptp nvme_core [17222.126408] try_get_cap_refs+0x159/0x880 [ceph] [17222.130236] i2c_algo_bit pps_core [17222.132165] __ceph_get_caps+0x4a1/0x760 [ceph] [17222.136943] dca fuse [17222.139651] ? __file_remove_privs+0x89/0x110 [17222.144259] [17222.144260] irq event stamp: 60739058 [17222.147657] ? wake_bit_function+0x70/0x70 [17222.152180] hardirqs last enabled at (60739057): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17222.154453] ? ceph_write_iter+0x260/0x770 [ceph] [17222.158803] hardirqs last disabled at (60739058): [] sysvec_apic_timer_interrupt+0xe/0x90 [17222.160293] ceph_write_iter+0x260/0x770 [ceph] [17222.163951] softirqs last enabled at (60739056): [] __do_softirq+0x2be/0x371 [17222.168060] ? lock_is_held_type+0xaa/0x120 [17222.178214] softirqs last disabled at (60739051): [] irq_exit_rcu+0xb0/0xd0 [17222.182914] vfs_write+0x324/0x4f0 [17222.192637] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17222.197167] ksys_pwrite64+0x65/0xa0 [17222.205845] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17222.210022] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17222.218533] Workqueue: events delayed_work [ceph] [17222.221933] do_syscall_64+0x38/0x80 [17222.231308] [17222.234879] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17222.242350] RIP: 0010:lock_acquire+0xf0/0x2b0 [17222.247480] RIP: 0033:0x7f48f0eeb93b [17222.252179] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17222.255749] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17222.257240] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17222.262284] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17222.266635] [17222.266636] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17222.270205] ORIG_RAX: 0000000000000012 [17222.288943] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17222.307680] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17222.312896] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17222.318114] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17222.319605] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17222.326730] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17222.330560] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17222.337685] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17222.344809] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17222.351932] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17222.359061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17222.366186] [17222.373304] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17222.417803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17222.424925] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17222.432062] Call Trace: [17222.434511] [17222.436522] ? watchdog_timer_fn+0x1b4/0x220 [17222.440786] ? __hrtimer_run_queues+0x1d3/0x360 [17222.445313] ? hrtimer_interrupt+0x104/0x240 [17222.449582] ? __local_bh_enable+0x50/0x60 [17222.453676] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17222.458899] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17222.463857] [17222.465955] [17222.468062] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17222.473369] ? lock_acquire+0xf0/0x2b0 [17222.477120] ? igrab+0x43/0x50 [17222.480168] ? lock_release+0xc5/0x260 [17222.483914] _raw_spin_lock+0x30/0x40 [17222.487570] ? igrab+0x1d/0x50 [17222.490621] igrab+0x1d/0x50 [17222.493499] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17222.498647] delayed_work+0x1d8/0x280 [ceph] [17222.502929] ? process_scheduled_works+0x22a/0x510 [17222.507719] process_scheduled_works+0x2a4/0x510 [17222.512350] worker_thread+0x188/0x340 [17222.516093] ? rescuer_thread+0x310/0x310 [17222.520096] kthread+0xf5/0x130 [17222.523234] ? kthread_complete_and_exit+0x20/0x20 [17222.528019] ret_from_fork+0x31/0x40 [17222.531597] ? kthread_complete_and_exit+0x20/0x20 [17222.536381] ret_from_fork_asm+0x11/0x20 [17222.540304] [17229.224758] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17239.464768] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17249.757193] watchdog: BUG: soft lockup - CPU#0 stuck for 4951s! [dbench:78683] [17249.764412] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17249.769192] watchdog: BUG: soft lockup - CPU#7 stuck for 13948s! [kworker/7:4:78808] [17249.819878] irq event stamp: 23903250 [17249.827609] Modules linked in: ceph libceph dns_resolver fscache [17249.831266] hardirqs last enabled at (23903249): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17249.837266] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [17249.847421] hardirqs last disabled at (23903250): [] sysvec_apic_timer_interrupt+0xe/0x90 [17249.847424] nf_conntrack [17249.847425] softirqs last enabled at (23903184): [] __do_softirq+0x2be/0x371 [17249.852900] nf_defrag_ipv6 nf_defrag_ipv4 [17249.862624] softirqs last disabled at (23903179): [] irq_exit_rcu+0xb0/0xd0 [17249.865241] nft_compat nf_tables [17249.873927] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17249.878015] nfnetlink bridge [17249.886527] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17249.889837] stp [17249.898783] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17249.901746] llc xfs [17249.909217] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17249.911053] libcrc32c [17249.916532] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17249.918714] sunrpc intel_rapl_msr [17249.937454] [17249.939810] intel_rapl_common [17249.945028] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17249.948424] sb_edac x86_pkg_temp_thermal [17249.949915] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17249.952968] coretemp kvm_intel [17249.960090] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17249.964095] kvm irqbypass [17249.971218] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17249.974356] i2c_i801 [17249.981480] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17249.984183] iTCO_wdt iTCO_vendor_support [17249.991309] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17249.993591] i2c_smbus [17250.000711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17250.004716] crct10dif_pclmul joydev [17250.012793] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17250.015150] crc32_pclmul ipmi_ssif [17250.020889] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17250.024458] ghash_clmulni_intel mei_me [17250.031592] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17250.035075] mei lpc_ich [17250.042201] Call Trace: [17250.042204] [17250.046032] sha512_ssse3 acpi_ipmi [17250.053158] ? watchdog_timer_fn+0x1b4/0x220 [17250.055687] mfd_core wmi [17250.058130] ? __hrtimer_run_queues+0x1d3/0x360 [17250.060141] ipmi_si ipmi_devintf [17250.063627] ? hrtimer_interrupt+0x104/0x240 [17250.067889] ipmi_msghandler [17250.070505] ? __local_bh_enable+0x50/0x60 [17250.075029] ioatdma acpi_pad [17250.078343] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17250.082604] acpi_power_meter [17250.085483] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17250.089591] ixgbe [17250.092554] [17250.097773] igb mdio [17250.100735] [17250.100736] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17250.105693] crc32c_intel nvme [17250.107707] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17250.109802] ptp nvme_core [17250.112075] do_raw_spin_lock+0xb5/0xc0 [17250.114171] i2c_algo_bit pps_core [17250.119475] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17250.122523] dca fuse [17250.127398] try_get_cap_refs+0x159/0x880 [ceph] [17250.130100] [17250.130101] irq event stamp: 60959878 [17250.133936] __ceph_get_caps+0x4a1/0x760 [ceph] [17250.137327] hardirqs last enabled at (60959877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17250.142116] ? __file_remove_privs+0x89/0x110 [17250.144383] hardirqs last disabled at (60959878): [] sysvec_apic_timer_interrupt+0xe/0x90 [17250.148995] ? wake_bit_function+0x70/0x70 [17250.150485] softirqs last enabled at (60959876): [] __do_softirq+0x2be/0x371 [17250.154143] ? ceph_write_iter+0x260/0x770 [ceph] [17250.158664] softirqs last disabled at (60959871): [] irq_exit_rcu+0xb0/0xd0 [17250.168823] ceph_write_iter+0x260/0x770 [ceph] [17250.173173] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17250.182903] ? lock_is_held_type+0xaa/0x120 [17250.186989] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17250.195675] vfs_write+0x324/0x4f0 [17250.200369] Workqueue: events delayed_work [ceph] [17250.208885] ksys_pwrite64+0x65/0xa0 [17250.213405] [17250.213405] RIP: 0010:lock_release+0x11/0x260 [17250.219063] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17250.222783] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17250.226959] Code: 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd <53> 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f [17250.234433] do_syscall_64+0x38/0x80 [17250.237827] RSP: 0018:ffffc90002117d68 EFLAGS: 00000286 [17250.242525] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17250.246096] [17250.246097] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [17250.247597] RIP: 0033:0x7f48f0eeb93b [17250.251946] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [17250.258897] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17250.264027] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [17250.282764] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17250.286336] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [17250.291555] ORIG_RAX: 0000000000000012 [17250.296599] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17250.298088] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17250.305211] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17250.308784] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17250.315908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17250.334645] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17250.341769] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17250.346987] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17250.354111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17250.357941] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17250.365066] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17250.372194] [17250.380267] Call Trace: [17250.380268] [17250.442524] ? watchdog_timer_fn+0x1b4/0x220 [17250.446795] ? __hrtimer_run_queues+0x1d3/0x360 [17250.451321] ? hrtimer_interrupt+0x104/0x240 [17250.455599] ? __local_bh_enable+0x50/0x60 [17250.459693] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17250.464916] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17250.469881] [17250.471979] [17250.474078] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17250.479381] ? igrab+0x43/0x50 [17250.482435] ? igrab+0x43/0x50 [17250.485491] ? lock_release+0x11/0x260 [17250.489238] _raw_spin_unlock+0x1b/0x40 [17250.493076] igrab+0x43/0x50 [17250.495954] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17250.501105] delayed_work+0x1d8/0x280 [ceph] [17250.505384] ? process_scheduled_works+0x22a/0x510 [17250.510175] process_scheduled_works+0x2a4/0x510 [17250.514789] worker_thread+0x188/0x340 [17250.518538] ? rescuer_thread+0x310/0x310 [17250.522543] kthread+0xf5/0x130 [17250.525691] ? kthread_complete_and_exit+0x20/0x20 [17250.530482] ret_from_fork+0x31/0x40 [17250.534063] ? kthread_complete_and_exit+0x20/0x20 [17250.538853] ret_from_fork_asm+0x11/0x20 [17250.542786] [17260.457213] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17271.209435] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17277.757728] watchdog: BUG: soft lockup - CPU#0 stuck for 4977s! [dbench:78683] [17277.764943] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17277.769728] watchdog: BUG: soft lockup - CPU#7 stuck for 13974s! [kworker/7:4:78808] [17277.820400] irq event stamp: 23961490 [17277.820401] hardirqs last enabled at (23961489): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17277.828131] Modules linked in: [17277.831789] hardirqs last disabled at (23961490): [] sysvec_apic_timer_interrupt+0xe/0x90 [17277.841947] ceph [17277.844998] softirqs last enabled at (23961434): [] __do_softirq+0x2be/0x371 [17277.854721] libceph [17277.856644] softirqs last disabled at (23961429): [] irq_exit_rcu+0xb0/0xd0 [17277.865330] dns_resolver fscache [17277.867513] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17277.876023] netfs veth [17277.879334] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17277.879335] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17277.888279] nft_chain_nat xt_MASQUERADE [17277.890722] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17277.898194] nf_nat nf_conntrack [17277.903672] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17277.907588] nf_defrag_ipv6 nf_defrag_ipv4 [17277.926326] [17277.926327] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17277.929551] nft_compat nf_tables [17277.934769] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17277.938859] nfnetlink bridge [17277.940350] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17277.947475] stp llc [17277.950784] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17277.957909] xfs libcrc32c [17277.960873] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17277.967997] sunrpc [17277.970180] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17277.977304] intel_rapl_msr intel_rapl_common [17277.980010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17277.987132] sb_edac x86_pkg_temp_thermal [17277.989231] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17277.997309] coretemp kvm_intel [17278.001658] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17278.007406] kvm irqbypass [17278.011410] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17278.018532] i2c_i801 iTCO_wdt [17278.021671] Call Trace: [17278.021673] [17278.028795] iTCO_vendor_support i2c_smbus [17278.031500] ? watchdog_timer_fn+0x1b4/0x220 [17278.038624] crct10dif_pclmul joydev [17278.041673] ? __hrtimer_run_queues+0x1d3/0x360 [17278.044128] crc32_pclmul ipmi_ssif [17278.046140] ? hrtimer_interrupt+0x104/0x240 [17278.050228] ghash_clmulni_intel mei_me [17278.054492] ? __local_bh_enable+0x50/0x60 [17278.058062] mei lpc_ich [17278.062589] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17278.066070] sha512_ssse3 acpi_ipmi [17278.070336] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17278.074164] mfd_core wmi [17278.078255] [17278.080786] ipmi_si ipmi_devintf [17278.086003] [17278.086005] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17278.089489] ipmi_msghandler ioatdma [17278.094449] ? queued_spin_lock_slowpath+0x16/0x2e0 [17278.097064] acpi_pad acpi_power_meter [17278.099162] do_raw_spin_lock+0xb5/0xc0 [17278.102471] ixgbe igb [17278.104568] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17278.109873] mdio crc32c_intel [17278.113445] try_get_cap_refs+0x159/0x880 [ceph] [17278.118313] nvme ptp [17278.122064] __ceph_get_caps+0x4a1/0x760 [ceph] [17278.125889] nvme_core i2c_algo_bit [17278.128250] ? __file_remove_privs+0x89/0x110 [17278.133029] pps_core dca [17278.136081] ? wake_bit_function+0x70/0x70 [17278.140693] fuse [17278.142966] ? ceph_write_iter+0x260/0x770 [ceph] [17278.147486] irq event stamp: 61180742 [17278.150972] ceph_write_iter+0x260/0x770 [ceph] [17278.155323] hardirqs last enabled at (61180741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17278.157942] ? lock_is_held_type+0xaa/0x120 [17278.162030] hardirqs last disabled at (61180742): [] sysvec_apic_timer_interrupt+0xe/0x90 [17278.163955] vfs_write+0x324/0x4f0 [17278.168650] softirqs last enabled at (61180740): [] __do_softirq+0x2be/0x371 [17278.172313] ksys_pwrite64+0x65/0xa0 [17278.176832] softirqs last disabled at (61180735): [] irq_exit_rcu+0xb0/0xd0 [17278.186990] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17278.191168] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17278.200894] do_syscall_64+0x38/0x80 [17278.204290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17278.212974] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17278.216544] Workqueue: events delayed_work [ceph] [17278.225054] RIP: 0033:0x7f48f0eeb93b [17278.230185] [17278.230186] RIP: 0010:lock_release+0x14e/0x260 [17278.239565] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17278.243134] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [17278.250604] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17278.255647] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [17278.260345] ORIG_RAX: 0000000000000012 [17278.263916] [17278.263916] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [17278.265412] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17278.269844] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [17278.288581] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17278.307320] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [17278.312537] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17278.317756] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [17278.321586] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17278.323077] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [17278.330200] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17278.337324] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17278.344454] [17278.351572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17278.410333] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17278.417459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17278.424583] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17278.431708] Call Trace: [17278.434151] [17278.436162] ? watchdog_timer_fn+0x1b4/0x220 [17278.440428] ? __hrtimer_run_queues+0x1d3/0x360 [17278.444953] ? hrtimer_interrupt+0x104/0x240 [17278.449223] ? __local_bh_enable+0x50/0x60 [17278.453316] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17278.458540] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17278.463497] [17278.465594] [17278.467691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17278.472998] ? igrab+0x43/0x50 [17278.476050] ? lock_release+0x14e/0x260 [17278.479887] ? lock_release+0xc5/0x260 [17278.483633] _raw_spin_unlock+0x1b/0x40 [17278.487471] igrab+0x43/0x50 [17278.490347] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17278.495499] delayed_work+0x1d8/0x280 [ceph] [17278.499778] ? process_scheduled_works+0x22a/0x510 [17278.504570] process_scheduled_works+0x2a4/0x510 [17278.509184] worker_thread+0x188/0x340 [17278.512934] ? rescuer_thread+0x310/0x310 [17278.516936] kthread+0xf5/0x130 [17278.520075] ? kthread_complete_and_exit+0x20/0x20 [17278.524860] ret_from_fork+0x31/0x40 [17278.528437] ? kthread_complete_and_exit+0x20/0x20 [17278.533223] ret_from_fork_asm+0x11/0x20 [17278.537154] [17279.061120] rcu: INFO: rcu_preempt self-detected stall on CPU [17279.066884] rcu: 7-....: (5189937 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/5188713 fqs=2592302 [17279.077308] rcu: (t=5342356 jiffies g=1309609 q=222891 ncpus=8) [17279.083307] Sending NMI from CPU 7 to CPUs 0: [17279.087658] NMI backtrace for cpu 0 [17279.087660] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17279.087661] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17279.087662] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17279.087665] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17279.087666] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17279.087667] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17279.087668] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17279.087668] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17279.087669] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17279.087670] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17279.087670] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17279.087671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17279.087672] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17279.087673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17279.087674] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17279.087674] Call Trace: [17279.087675] [17279.087676] ? nmi_cpu_backtrace+0xad/0x120 [17279.087679] ? nmi_cpu_backtrace_handler+0x11/0x20 [17279.087681] ? nmi_handle+0xd4/0x260 [17279.087685] ? default_do_nmi+0x49/0xf0 [17279.087686] ? exc_nmi+0x11e/0x150 [17279.087688] ? end_repeat_nmi+0x16/0x67 [17279.087692] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17279.087694] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17279.087696] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17279.087699] [17279.087699] [17279.087700] do_raw_spin_lock+0xb5/0xc0 [17279.087702] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17279.087715] try_get_cap_refs+0x159/0x880 [ceph] [17279.087729] __ceph_get_caps+0x4a1/0x760 [ceph] [17279.087741] ? __file_remove_privs+0x89/0x110 [17279.087743] ? wake_bit_function+0x70/0x70 [17279.087745] ? ceph_write_iter+0x260/0x770 [ceph] [17279.087755] ceph_write_iter+0x260/0x770 [ceph] [17279.087766] ? lock_is_held_type+0xaa/0x120 [17279.087769] vfs_write+0x324/0x4f0 [17279.087773] ksys_pwrite64+0x65/0xa0 [17279.087775] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17279.087777] do_syscall_64+0x38/0x80 [17279.087779] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17279.087781] RIP: 0033:0x7f48f0eeb93b [17279.087782] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17279.087783] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [17279.087784] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17279.087785] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17279.087786] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17279.087786] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17279.087787] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17279.087790] [17279.088657] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17279.392529] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17279.400001] Workqueue: events delayed_work [ceph] [17279.404723] RIP: 0010:lock_acquire+0x184/0x2b0 [17279.409161] Code: 62 5b ff ff 59 5e 4c 8b 14 24 65 ff 0d 5d e6 eb 7e 0f 85 c8 fe ff ff 0f 1f 44 00 00 e9 be fe ff ff 4c 89 14 24 e8 9c 8a 02 00 <4c> 8b 14 24 84 c0 0f 85 c0 fe ff ff 0f 0b e9 b9 fe ff ff 65 8b 05 [17279.427898] RSP: 0018:ffffc90002117d18 EFLAGS: 00000202 [17279.433115] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 0000000000000000 [17279.440241] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [17279.447363] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [17279.454488] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [17279.461612] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17279.468736] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17279.476813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17279.482550] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17279.489676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17279.496800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17279.503924] Call Trace: [17279.506370] [17279.508379] ? rcu_dump_cpu_stacks+0xd1/0x180 [17279.512740] ? rcu_sched_clock_irq+0x633/0x1390 [17279.517271] ? lock_release+0xc5/0x260 [17279.521016] ? lock_is_held_type+0xaa/0x120 [17279.525204] ? lock_is_held_type+0xaa/0x120 [17279.529391] ? tick_sched_do_timer+0xa0/0xa0 [17279.533661] ? rcu_read_lock_sched_held+0x56/0x90 [17279.538369] ? tick_sched_do_timer+0xa0/0xa0 [17279.542640] ? update_process_times+0x5f/0x90 [17279.546991] ? tick_sched_handle.isra.16+0x1e/0x50 [17279.551783] ? tick_sched_timer+0x6f/0x80 [17279.555789] ? __hrtimer_run_queues+0x1d3/0x360 [17279.560326] ? hrtimer_interrupt+0x104/0x240 [17279.564591] ? __local_bh_enable+0x50/0x60 [17279.568688] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17279.573908] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17279.578866] [17279.580964] [17279.583062] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17279.588384] ? lock_acquire+0x184/0x2b0 [17279.592224] ? lock_acquire+0x184/0x2b0 [17279.596061] ? igrab+0x43/0x50 [17279.599114] ? lock_release+0xc5/0x260 [17279.602868] _raw_spin_lock+0x30/0x40 [17279.606531] ? igrab+0x1d/0x50 [17279.609582] igrab+0x1d/0x50 [17279.612459] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17279.617612] delayed_work+0x1d8/0x280 [ceph] [17279.621901] ? process_scheduled_works+0x22a/0x510 [17279.626688] process_scheduled_works+0x2a4/0x510 [17279.631311] worker_thread+0x188/0x340 [17279.635064] ? rescuer_thread+0x310/0x310 [17279.639074] kthread+0xf5/0x130 [17279.642211] ? kthread_complete_and_exit+0x20/0x20 [17279.646998] ret_from_fork+0x31/0x40 [17279.650575] ? kthread_complete_and_exit+0x20/0x20 [17279.655360] ret_from_fork_asm+0x11/0x20 [17279.659290] [17281.449634] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17292.201961] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17302.442014] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17305.758260] watchdog: BUG: soft lockup - CPU#0 stuck for 5003s! [dbench:78683] [17305.765472] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17305.770259] watchdog: BUG: soft lockup - CPU#7 stuck for 14000s! [kworker/7:4:78808] [17305.820938] irq event stamp: 24019802 [17305.828672] Modules linked in: ceph libceph dns_resolver fscache netfs veth [17305.832328] hardirqs last enabled at (24019801): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17305.839287] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [17305.849445] hardirqs last disabled at (24019802): [] sysvec_apic_timer_interrupt+0xe/0x90 [17305.849448] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [17305.864819] softirqs last enabled at (24019722): [] __do_softirq+0x2be/0x371 [17305.872204] stp llc xfs libcrc32c [17305.880889] softirqs last disabled at (24019717): [] irq_exit_rcu+0xb0/0xd0 [17305.884287] sunrpc intel_rapl_msr intel_rapl_common [17305.892798] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17305.897753] sb_edac x86_pkg_temp_thermal [17305.906699] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17305.906701] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17305.910702] coretemp kvm_intel [17305.918175] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17305.923656] kvm irqbypass [17305.926796] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17305.945535] i2c_i801 iTCO_wdt [17305.948237] [17305.948238] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17305.953455] iTCO_vendor_support i2c_smbus [17305.956507] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17305.957997] crct10dif_pclmul joydev [17305.965121] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17305.969212] crc32_pclmul ipmi_ssif [17305.976336] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17305.979908] ghash_clmulni_intel [17305.987031] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17305.990516] mei_me mei [17305.997655] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17306.000879] lpc_ich sha512_ssse3 [17306.008006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17306.010450] acpi_ipmi mfd_core [17306.018526] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17306.021836] wmi ipmi_si [17306.027574] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17306.030712] ipmi_devintf ipmi_msghandler [17306.037837] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17306.040366] ioatdma acpi_pad [17306.047490] Call Trace: [17306.047492] [17306.051494] acpi_power_meter [17306.058621] ? watchdog_timer_fn+0x1b4/0x220 [17306.061583] ixgbe igb [17306.064028] ? __hrtimer_run_queues+0x1d3/0x360 [17306.066039] mdio crc32c_intel [17306.069006] ? hrtimer_interrupt+0x104/0x240 [17306.073267] nvme [17306.075623] ? __local_bh_enable+0x50/0x60 [17306.080148] ptp nvme_core [17306.083201] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17306.087462] i2c_algo_bit [17306.089387] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17306.093477] pps_core dca [17306.096182] [17306.101398] fuse [17306.104015] [17306.104016] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17306.108972] irq event stamp: 61396732 [17306.111595] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17306.113688] hardirqs last enabled at (61396731): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17306.115615] do_raw_spin_lock+0xb5/0xc0 [17306.117708] hardirqs last disabled at (61396732): [] sysvec_apic_timer_interrupt+0xe/0x90 [17306.123016] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17306.126671] softirqs last enabled at (61396730): [] __do_softirq+0x2be/0x371 [17306.131544] try_get_cap_refs+0x159/0x880 [ceph] [17306.141698] softirqs last disabled at (61396725): [] irq_exit_rcu+0xb0/0xd0 [17306.145536] __ceph_get_caps+0x4a1/0x760 [ceph] [17306.155254] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17306.160042] ? __file_remove_privs+0x89/0x110 [17306.168724] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17306.173334] ? wake_bit_function+0x70/0x70 [17306.181844] Workqueue: events delayed_work [ceph] [17306.186371] ? ceph_write_iter+0x260/0x770 [ceph] [17306.195747] [17306.195747] RIP: 0010:lock_acquire+0xf0/0x2b0 [17306.200096] ceph_write_iter+0x260/0x770 [ceph] [17306.207566] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17306.211664] ? lock_is_held_type+0xaa/0x120 [17306.216355] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17306.221054] vfs_write+0x324/0x4f0 [17306.222542] [17306.222543] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17306.226899] ksys_pwrite64+0x65/0xa0 [17306.231419] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17306.250157] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17306.254333] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17306.259552] do_syscall_64+0x38/0x80 [17306.262949] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17306.264439] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17306.271562] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17306.275135] RIP: 0033:0x7f48f0eeb93b [17306.282258] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17306.287390] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17306.294513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17306.298083] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17306.305209] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17306.310250] ORIG_RAX: 0000000000000012 [17306.317376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17306.320947] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17306.329025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17306.347761] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17306.353500] Call Trace: [17306.353501] [17306.358715] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17306.365841] ? watchdog_timer_fn+0x1b4/0x220 [17306.369670] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17306.376798] ? __hrtimer_run_queues+0x1d3/0x360 [17306.383918] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17306.391065] ? hrtimer_interrupt+0x104/0x240 [17306.398190] [17306.400630] ? __local_bh_enable+0x50/0x60 [17306.443361] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17306.448584] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17306.453541] [17306.455639] [17306.457753] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17306.463062] ? lock_acquire+0xf0/0x2b0 [17306.466811] ? igrab+0x43/0x50 [17306.469861] ? lock_release+0xc5/0x260 [17306.473607] _raw_spin_lock+0x30/0x40 [17306.477270] ? igrab+0x1d/0x50 [17306.480322] igrab+0x1d/0x50 [17306.483201] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17306.488351] delayed_work+0x1d8/0x280 [ceph] [17306.492630] ? process_scheduled_works+0x22a/0x510 [17306.497421] process_scheduled_works+0x2a4/0x510 [17306.502035] worker_thread+0x188/0x340 [17306.505786] ? rescuer_thread+0x310/0x310 [17306.509788] kthread+0xf5/0x130 [17306.512943] ? kthread_complete_and_exit+0x20/0x20 [17306.517729] ret_from_fork+0x31/0x40 [17306.521308] ? kthread_complete_and_exit+0x20/0x20 [17306.526091] ret_from_fork_asm+0x11/0x20 [17306.530015] [17313.194400] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17323.434484] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17333.758789] watchdog: BUG: soft lockup - CPU#0 stuck for 5029s! [dbench:78683] [17333.766005] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17333.770789] watchdog: BUG: soft lockup - CPU#7 stuck for 14026s! [kworker/7:4:78808] [17333.821462] irq event stamp: 24078010 [17333.821463] hardirqs last enabled at (24078009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17333.829192] Modules linked in: [17333.832849] hardirqs last disabled at (24078010): [] sysvec_apic_timer_interrupt+0xe/0x90 [17333.843009] ceph [17333.846058] softirqs last enabled at (24077984): [] __do_softirq+0x2be/0x371 [17333.855782] libceph [17333.857707] softirqs last disabled at (24077979): [] irq_exit_rcu+0xb0/0xd0 [17333.866391] dns_resolver fscache [17333.868577] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17333.877086] netfs veth [17333.880397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17333.889341] nft_chain_nat [17333.891786] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17333.899255] xt_MASQUERADE [17333.901961] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17333.907436] nf_nat [17333.910141] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17333.928879] nf_conntrack nf_defrag_ipv6 [17333.930977] [17333.930978] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17333.936194] nf_defrag_ipv4 [17333.940111] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17333.941602] nft_compat nf_tables [17333.948725] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17333.951515] nfnetlink [17333.958641] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17333.961952] bridge stp [17333.969076] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17333.971433] llc xfs [17333.978558] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17333.981002] libcrc32c sunrpc [17333.988125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17333.990309] intel_rapl_msr [17333.998386] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17334.001351] intel_rapl_common sb_edac [17334.007087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17334.009878] x86_pkg_temp_thermal coretemp [17334.017002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17334.020747] kvm_intel kvm [17334.027872] Call Trace: [17334.027873] [17334.031963] irqbypass i2c_i801 [17334.039087] ? watchdog_timer_fn+0x1b4/0x220 [17334.041791] iTCO_wdt iTCO_vendor_support [17334.044236] ? __hrtimer_run_queues+0x1d3/0x360 [17334.046244] i2c_smbus crct10dif_pclmul [17334.049386] ? hrtimer_interrupt+0x104/0x240 [17334.053646] joydev crc32_pclmul [17334.057651] ? __local_bh_enable+0x50/0x60 [17334.062186] ipmi_ssif ghash_clmulni_intel [17334.066016] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17334.070279] mei_me mei [17334.073503] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17334.077592] lpc_ich [17334.081683] [17334.086901] sha512_ssse3 acpi_ipmi [17334.089344] [17334.089345] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17334.094303] mfd_core [17334.096490] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17334.098585] wmi ipmi_si [17334.102069] do_raw_spin_lock+0xb5/0xc0 [17334.104165] ipmi_devintf [17334.109488] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17334.111758] ipmi_msghandler ioatdma [17334.116631] try_get_cap_refs+0x159/0x880 [ceph] [17334.119159] acpi_pad acpi_power_meter [17334.122994] __ceph_get_caps+0x4a1/0x760 [ceph] [17334.125607] ixgbe igb [17334.130396] ? __file_remove_privs+0x89/0x110 [17334.133961] mdio crc32c_intel [17334.138573] ? wake_bit_function+0x70/0x70 [17334.142316] nvme ptp [17334.146844] ? ceph_write_iter+0x260/0x770 [ceph] [17334.149197] nvme_core i2c_algo_bit [17334.153549] ceph_write_iter+0x260/0x770 [ceph] [17334.156600] pps_core [17334.160694] ? lock_is_held_type+0xaa/0x120 [17334.162960] dca fuse [17334.167661] vfs_write+0x324/0x4f0 [17334.171141] [17334.171142] irq event stamp: 61617764 [17334.175670] ksys_pwrite64+0x65/0xa0 [17334.177937] hardirqs last enabled at (61617763): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17334.182115] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17334.184386] hardirqs last disabled at (61617764): [] sysvec_apic_timer_interrupt+0xe/0x90 [17334.187785] do_syscall_64+0x38/0x80 [17334.189274] softirqs last enabled at (61617762): [] __do_softirq+0x2be/0x371 [17334.190751] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17334.192932] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17334.196502] softirqs last disabled at (61617757): [] irq_exit_rcu+0xb0/0xd0 [17334.206660] RIP: 0033:0x7f48f0eeb93b [17334.211790] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17334.221515] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17334.225085] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17334.233769] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17334.240720] Workqueue: events delayed_work [ceph] [17334.245764] ORIG_RAX: 0000000000000012 [17334.254275] [17334.254276] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [17334.257846] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17334.267222] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [17334.285961] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17334.293432] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [17334.298650] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17334.303346] [17334.303346] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [17334.307193] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17334.308684] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [17334.313296] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17334.320420] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [17334.339162] [17334.346281] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17334.405043] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17334.412167] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17334.420262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17334.426000] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17334.433123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17334.440248] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17334.447371] Call Trace: [17334.449815] [17334.451828] ? watchdog_timer_fn+0x1b4/0x220 [17334.456093] ? __hrtimer_run_queues+0x1d3/0x360 [17334.460628] ? hrtimer_interrupt+0x104/0x240 [17334.464896] ? __local_bh_enable+0x50/0x60 [17334.468989] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17334.474213] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17334.479172] [17334.481285] [17334.483384] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17334.488691] ? do_raw_spin_lock+0x42/0xc0 [17334.492702] igrab+0x1d/0x50 [17334.495578] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17334.500728] delayed_work+0x1d8/0x280 [ceph] [17334.505009] ? process_scheduled_works+0x22a/0x510 [17334.509800] process_scheduled_works+0x2a4/0x510 [17334.514414] worker_thread+0x188/0x340 [17334.518165] ? rescuer_thread+0x310/0x310 [17334.522166] kthread+0xf5/0x130 [17334.525306] ? kthread_complete_and_exit+0x20/0x20 [17334.530098] ret_from_fork+0x31/0x40 [17334.533668] ? kthread_complete_and_exit+0x20/0x20 [17334.538454] ret_from_fork_asm+0x11/0x20 [17334.542392] [17344.426670] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17355.179231] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17361.759314] watchdog: BUG: soft lockup - CPU#0 stuck for 5055s! [dbench:78683] [17361.766527] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17361.771314] watchdog: BUG: soft lockup - CPU#7 stuck for 14052s! [kworker/7:4:78808] [17361.821984] irq event stamp: 24136324 [17361.829715] Modules linked in: ceph libceph [17361.833373] hardirqs last enabled at (24136323): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17361.837551] dns_resolver fscache netfs [17361.847709] hardirqs last disabled at (24136324): [] sysvec_apic_timer_interrupt+0xe/0x90 [17361.851539] veth nft_chain_nat xt_MASQUERADE [17361.861262] softirqs last enabled at (24136236): [] __do_softirq+0x2be/0x371 [17361.861263] nf_nat [17361.861265] softirqs last disabled at (24136231): [] irq_exit_rcu+0xb0/0xd0 [17361.865612] nf_conntrack [17361.874297] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17361.876396] nf_defrag_ipv6 [17361.884904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17361.887522] nf_defrag_ipv4 [17361.896469] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17361.899257] nft_compat nf_tables [17361.906728] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17361.909520] nfnetlink [17361.914996] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17361.918308] bridge stp [17361.937046] [17361.937047] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17361.939403] llc xfs [17361.944619] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17361.947065] libcrc32c sunrpc [17361.948554] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17361.955680] intel_rapl_msr intel_rapl_common [17361.957863] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17361.964993] sb_edac x86_pkg_temp_thermal [17361.967960] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17361.975084] coretemp kvm_intel [17361.979434] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17361.986558] kvm irqbypass [17361.990562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17361.997688] i2c_i801 [17362.000842] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17362.008919] iTCO_wdt iTCO_vendor_support [17362.011624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17362.017360] i2c_smbus crct10dif_pclmul [17362.019631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17362.026756] joydev [17362.030760] Call Trace: [17362.030761] [17362.037884] crc32_pclmul ipmi_ssif [17362.041717] ? watchdog_timer_fn+0x1b4/0x220 [17362.048838] ghash_clmulni_intel mei_me [17362.050937] ? __hrtimer_run_queues+0x1d3/0x360 [17362.053381] mei lpc_ich [17362.055393] ? hrtimer_interrupt+0x104/0x240 [17362.058875] sha512_ssse3 acpi_ipmi [17362.063139] ? __local_bh_enable+0x50/0x60 [17362.066971] mfd_core [17362.071513] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17362.074040] wmi ipmi_si [17362.078306] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17362.081789] ipmi_devintf ipmi_msghandler [17362.085881] [17362.088152] ioatdma acpi_pad [17362.093370] [17362.093371] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17362.095900] acpi_power_meter ixgbe [17362.100860] ? queued_spin_lock_slowpath+0x16/0x2e0 [17362.104861] igb mdio [17362.106961] do_raw_spin_lock+0xb5/0xc0 [17362.109922] crc32c_intel nvme [17362.112020] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17362.117323] ptp nvme_core [17362.120811] try_get_cap_refs+0x159/0x880 [ceph] [17362.125679] i2c_algo_bit pps_core [17362.127955] __ceph_get_caps+0x4a1/0x760 [ceph] [17362.131779] dca fuse [17362.134836] ? __file_remove_privs+0x89/0x110 [17362.139615] [17362.139615] irq event stamp: 61838626 [17362.142320] ? wake_bit_function+0x70/0x70 [17362.146930] hardirqs last enabled at (61838625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17362.150330] ? ceph_write_iter+0x260/0x770 [ceph] [17362.154852] hardirqs last disabled at (61838626): [] sysvec_apic_timer_interrupt+0xe/0x90 [17362.157122] ceph_write_iter+0x260/0x770 [ceph] [17362.161474] softirqs last enabled at (61838624): [] __do_softirq+0x2be/0x371 [17362.162966] ? lock_is_held_type+0xaa/0x120 [17362.166622] softirqs last disabled at (61838619): [] irq_exit_rcu+0xb0/0xd0 [17362.170718] vfs_write+0x324/0x4f0 [17362.180878] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17362.185579] ksys_pwrite64+0x65/0xa0 [17362.195300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17362.199824] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17362.208507] Workqueue: events delayed_work [ceph] [17362.212686] do_syscall_64+0x38/0x80 [17362.221197] [17362.221197] RIP: 0010:lock_acquire+0xf0/0x2b0 [17362.224593] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17362.233972] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17362.237559] RIP: 0033:0x7f48f0eeb93b [17362.245030] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17362.250161] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17362.254857] [17362.258427] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17362.259920] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17362.264270] ORIG_RAX: 0000000000000012 [17362.269315] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17362.288051] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17362.291622] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17362.296839] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17362.315578] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17362.317069] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17362.322285] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17362.329409] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17362.333242] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17362.340365] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17362.347488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17362.354617] [17362.361738] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17362.420482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17362.427604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17362.434728] Call Trace: [17362.437176] [17362.439186] ? watchdog_timer_fn+0x1b4/0x220 [17362.443448] ? __hrtimer_run_queues+0x1d3/0x360 [17362.447975] ? hrtimer_interrupt+0x104/0x240 [17362.452237] ? __local_bh_enable+0x50/0x60 [17362.456330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17362.461555] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17362.466510] [17362.468609] [17362.470721] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17362.476032] ? lock_acquire+0xf0/0x2b0 [17362.479780] ? igrab+0x43/0x50 [17362.482831] ? lock_release+0xc5/0x260 [17362.486577] _raw_spin_lock+0x30/0x40 [17362.490240] ? igrab+0x1d/0x50 [17362.493290] igrab+0x1d/0x50 [17362.496170] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17362.501319] delayed_work+0x1d8/0x280 [ceph] [17362.505601] ? process_scheduled_works+0x22a/0x510 [17362.510391] process_scheduled_works+0x2a4/0x510 [17362.515006] worker_thread+0x188/0x340 [17362.518754] ? rescuer_thread+0x310/0x310 [17362.522757] kthread+0xf5/0x130 [17362.525896] ? kthread_complete_and_exit+0x20/0x20 [17362.530681] ret_from_fork+0x31/0x40 [17362.534250] ? kthread_complete_and_exit+0x20/0x20 [17362.539034] ret_from_fork_asm+0x11/0x20 [17362.542958] [17365.419369] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17375.659597] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17386.411837] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17389.759838] watchdog: BUG: soft lockup - CPU#0 stuck for 5081s! [dbench:78683] [17389.767058] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17389.771838] watchdog: BUG: soft lockup - CPU#7 stuck for 14078s! [kworker/7:4:78808] [17389.822524] irq event stamp: 24194614 [17389.822525] hardirqs last enabled at (24194613): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17389.830254] Modules linked in: [17389.833913] hardirqs last disabled at (24194614): [] sysvec_apic_timer_interrupt+0xe/0x90 [17389.844070] ceph [17389.847119] softirqs last enabled at (24194526): [] __do_softirq+0x2be/0x371 [17389.856844] libceph [17389.858770] softirqs last disabled at (24194521): [] irq_exit_rcu+0xb0/0xd0 [17389.867453] dns_resolver fscache [17389.869638] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17389.878148] netfs veth [17389.881457] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17389.890401] nft_chain_nat xt_MASQUERADE [17389.892849] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17389.900318] nf_nat nf_conntrack [17389.904235] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17389.909712] nf_defrag_ipv6 nf_defrag_ipv4 [17389.912936] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17389.931674] nft_compat nf_tables [17389.935765] [17389.935765] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17389.940982] nfnetlink bridge [17389.944294] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17389.945785] stp llc [17389.952909] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17389.955873] xfs libcrc32c [17389.962997] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17389.965181] sunrpc intel_rapl_msr [17389.972305] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17389.975007] intel_rapl_common [17389.982133] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17389.985529] sb_edac [17389.992655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17389.995705] x86_pkg_temp_thermal coretemp [17390.003782] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17390.005967] kvm_intel kvm [17390.011703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17390.015794] irqbypass i2c_i801 [17390.022919] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17390.025621] iTCO_wdt iTCO_vendor_support [17390.032746] Call Trace: [17390.032748] [17390.035885] i2c_smbus crct10dif_pclmul [17390.043010] ? watchdog_timer_fn+0x1b4/0x220 [17390.047012] joydev crc32_pclmul [17390.049458] ? __hrtimer_run_queues+0x1d3/0x360 [17390.051466] ipmi_ssif [17390.055302] ? hrtimer_interrupt+0x104/0x240 [17390.059562] ghash_clmulni_intel [17390.062786] ? __local_bh_enable+0x50/0x60 [17390.067310] mei_me mei [17390.069671] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17390.073932] lpc_ich sha512_ssse3 [17390.077156] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17390.081245] acpi_ipmi mfd_core [17390.083691] [17390.088907] wmi ipmi_si [17390.092235] [17390.097192] ipmi_devintf [17390.100330] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17390.102427] ipmi_msghandler ioatdma [17390.104964] ? queued_spin_lock_slowpath+0x16/0x2e0 [17390.107055] acpi_pad acpi_power_meter [17390.109676] do_raw_spin_lock+0xb5/0xc0 [17390.114978] ixgbe igb [17390.118550] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17390.123417] mdio crc32c_intel [17390.127166] try_get_cap_refs+0x159/0x880 [ceph] [17390.130994] nvme [17390.133357] __ceph_get_caps+0x4a1/0x760 [ceph] [17390.138136] ptp nvme_core [17390.141190] ? __file_remove_privs+0x89/0x110 [17390.145797] i2c_algo_bit pps_core [17390.147721] ? wake_bit_function+0x70/0x70 [17390.152246] dca fuse [17390.154951] ? ceph_write_iter+0x260/0x770 [ceph] [17390.159300] [17390.159301] irq event stamp: 62059452 [17390.162697] ceph_write_iter+0x260/0x770 [ceph] [17390.166787] hardirqs last enabled at (62059451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17390.169061] ? lock_is_held_type+0xaa/0x120 [17390.173756] hardirqs last disabled at (62059452): [] sysvec_apic_timer_interrupt+0xe/0x90 [17390.175249] vfs_write+0x324/0x4f0 [17390.178904] softirqs last enabled at (62059450): [] __do_softirq+0x2be/0x371 [17390.183432] ksys_pwrite64+0x65/0xa0 [17390.193587] softirqs last disabled at (62059445): [] irq_exit_rcu+0xb0/0xd0 [17390.197764] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17390.207488] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17390.210886] do_syscall_64+0x38/0x80 [17390.219568] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17390.223141] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17390.231650] Workqueue: events delayed_work [ceph] [17390.236781] RIP: 0033:0x7f48f0eeb93b [17390.246158] [17390.246159] RIP: 0010:lock_acquire+0xf0/0x2b0 [17390.249731] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17390.257200] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17390.262244] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17390.266940] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17390.270529] ORIG_RAX: 0000000000000012 [17390.272020] [17390.272021] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17390.276370] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17390.295109] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17390.313845] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17390.319064] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17390.324281] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17390.328112] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17390.329602] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17390.336728] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17390.343851] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17390.350976] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17390.358106] [17390.365240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17390.416861] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17390.423984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17390.431109] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17390.438244] Call Trace: [17390.440687] [17390.442697] ? watchdog_timer_fn+0x1b4/0x220 [17390.446961] ? __hrtimer_run_queues+0x1d3/0x360 [17390.451488] ? hrtimer_interrupt+0x104/0x240 [17390.455757] ? __local_bh_enable+0x50/0x60 [17390.459851] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17390.465074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17390.470033] [17390.472129] [17390.474244] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17390.479553] ? lock_acquire+0xf0/0x2b0 [17390.483303] ? igrab+0x43/0x50 [17390.486352] ? lock_release+0xc5/0x260 [17390.490097] _raw_spin_lock+0x30/0x40 [17390.493762] ? igrab+0x1d/0x50 [17390.496811] igrab+0x1d/0x50 [17390.499692] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17390.504840] delayed_work+0x1d8/0x280 [ceph] [17390.509121] ? process_scheduled_works+0x22a/0x510 [17390.513911] process_scheduled_works+0x2a4/0x510 [17390.518532] worker_thread+0x188/0x340 [17390.522275] ? rescuer_thread+0x310/0x310 [17390.526280] kthread+0xf5/0x130 [17390.529418] ? kthread_complete_and_exit+0x20/0x20 [17390.534201] ret_from_fork+0x31/0x40 [17390.537781] ? kthread_complete_and_exit+0x20/0x20 [17390.542565] ret_from_fork_asm+0x11/0x20 [17390.546498] [17396.651876] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17407.404065] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17417.644145] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17417.760359] watchdog: BUG: soft lockup - CPU#0 stuck for 5107s! [dbench:78683] [17417.767572] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17417.772359] watchdog: BUG: soft lockup - CPU#7 stuck for 14104s! [kworker/7:4:78808] [17417.823012] irq event stamp: 24252892 [17417.830754] Modules linked in: ceph libceph [17417.834410] hardirqs last enabled at (24252891): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17417.838585] dns_resolver fscache [17417.848756] hardirqs last disabled at (24252892): [] sysvec_apic_timer_interrupt+0xe/0x90 [17417.852071] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [17417.861797] softirqs last enabled at (24252806): [] __do_softirq+0x2be/0x371 [17417.867273] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [17417.875958] softirqs last disabled at (24252801): [] irq_exit_rcu+0xb0/0xd0 [17417.881176] nft_compat nf_tables nfnetlink [17417.889687] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17417.893864] bridge stp [17417.902806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17417.902808] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17417.905252] llc xfs [17417.912722] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17417.918199] libcrc32c [17417.920384] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17417.939120] sunrpc [17417.941479] [17417.946695] intel_rapl_msr [17417.948794] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17417.950283] intel_rapl_common sb_edac [17417.953074] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17417.960198] x86_pkg_temp_thermal coretemp [17417.963943] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17417.971067] kvm_intel kvm [17417.975159] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17417.982281] irqbypass i2c_i801 [17417.984986] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17417.992111] iTCO_wdt iTCO_vendor_support [17417.995249] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17418.002371] i2c_smbus crct10dif_pclmul [17418.006375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17418.014453] joydev crc32_pclmul [17418.018284] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17418.024037] ipmi_ssif ghash_clmulni_intel [17418.027262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17418.034388] mei_me [17418.038478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17418.045602] mei lpc_ich [17418.047700] Call Trace: [17418.047702] [17418.054823] sha512_ssse3 acpi_ipmi [17418.057355] ? watchdog_timer_fn+0x1b4/0x220 [17418.059799] mfd_core wmi [17418.061810] ? __hrtimer_run_queues+0x1d3/0x360 [17418.065294] ipmi_si ipmi_devintf [17418.069560] ? hrtimer_interrupt+0x104/0x240 [17418.072175] ipmi_msghandler [17418.076697] ? __local_bh_enable+0x50/0x60 [17418.080008] ioatdma acpi_pad [17418.084276] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17418.087150] acpi_power_meter [17418.091242] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17418.094206] ixgbe igb [17418.099424] [17418.102388] mdio [17418.107345] [17418.109702] crc32c_intel [17418.111800] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17418.113723] nvme ptp [17418.115825] ? queued_spin_lock_slowpath+0x16/0x2e0 [17418.118439] nvme_core i2c_algo_bit [17418.123761] do_raw_spin_lock+0xb5/0xc0 [17418.126037] pps_core dca [17418.130910] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17418.134393] fuse [17418.138226] try_get_cap_refs+0x159/0x880 [ceph] [17418.140842] [17418.140843] irq event stamp: 62280406 [17418.145631] __ceph_get_caps+0x4a1/0x760 [ceph] [17418.147549] hardirqs last enabled at (62280405): [] asm_common_interrupt+0x26/0x40 [17418.152164] ? __file_remove_privs+0x89/0x110 [17418.153652] hardirqs last disabled at (62280406): [] sysvec_apic_timer_interrupt+0xe/0x90 [17418.157309] ? wake_bit_function+0x70/0x70 [17418.161834] softirqs last enabled at (62280404): [] __do_softirq+0x2be/0x371 [17418.171049] ? ceph_write_iter+0x260/0x770 [ceph] [17418.175396] softirqs last disabled at (62280397): [] irq_exit_rcu+0xb0/0xd0 [17418.185122] ceph_write_iter+0x260/0x770 [ceph] [17418.189213] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17418.197899] ? lock_is_held_type+0xaa/0x120 [17418.202592] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17418.211106] vfs_write+0x324/0x4f0 [17418.215629] Workqueue: events delayed_work [ceph] [17418.225010] ksys_pwrite64+0x65/0xa0 [17418.229183] [17418.229184] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [17418.236655] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17418.240052] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [17418.244764] do_syscall_64+0x38/0x80 [17418.248336] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [17418.249828] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17418.254438] [17418.254439] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [17418.259568] RIP: 0033:0x7f48f0eeb93b [17418.278307] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [17418.281877] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17418.287093] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [17418.292138] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17418.293628] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17418.300762] ORIG_RAX: 0000000000000012 [17418.304332] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17418.311457] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17418.330196] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17418.337319] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17418.342535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17418.349660] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17418.353491] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17418.360616] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17418.367740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17418.375834] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17418.382957] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17418.388702] [17418.395820] Call Trace: [17418.395821] [17418.438080] ? watchdog_timer_fn+0x1b4/0x220 [17418.442344] ? __hrtimer_run_queues+0x1d3/0x360 [17418.446870] ? hrtimer_interrupt+0x104/0x240 [17418.451141] ? __local_bh_enable+0x50/0x60 [17418.455234] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17418.460457] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17418.465415] [17418.467514] [17418.469610] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17418.474918] ? do_raw_spin_lock+0x42/0xc0 [17418.478928] igrab+0x1d/0x50 [17418.481806] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17418.486953] delayed_work+0x1d8/0x280 [ceph] [17418.491237] ? process_scheduled_works+0x22a/0x510 [17418.496042] process_scheduled_works+0x2a4/0x510 [17418.500660] worker_thread+0x188/0x340 [17418.504407] ? rescuer_thread+0x310/0x310 [17418.508411] kthread+0xf5/0x130 [17418.511548] ? kthread_complete_and_exit+0x20/0x20 [17418.516333] ret_from_fork+0x31/0x40 [17418.519902] ? kthread_complete_and_exit+0x20/0x20 [17418.524687] ret_from_fork_asm+0x11/0x20 [17418.528612] [17445.760879] watchdog: BUG: soft lockup - CPU#0 stuck for 5133s! [dbench:78683] [17445.768095] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17445.772879] watchdog: BUG: soft lockup - CPU#7 stuck for 14130s! [kworker/7:4:78808] [17445.823556] irq event stamp: 24311304 [17445.831291] Modules linked in: ceph libceph dns_resolver fscache netfs [17445.834949] hardirqs last enabled at (24311303): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17445.841474] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [17445.851633] hardirqs last disabled at (24311304): [] sysvec_apic_timer_interrupt+0xe/0x90 [17445.859017] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [17445.868741] softirqs last enabled at (24311278): [] __do_softirq+0x2be/0x371 [17445.875517] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [17445.884203] softirqs last disabled at (24311273): [] irq_exit_rcu+0xb0/0xd0 [17445.892895] coretemp kvm_intel kvm irqbypass [17445.901407] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17445.901408] i2c_i801 [17445.901409] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17445.905757] iTCO_wdt [17445.914702] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17445.916971] iTCO_vendor_support i2c_smbus [17445.924444] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17445.926714] crct10dif_pclmul joydev [17445.932191] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17445.936282] crc32_pclmul ipmi_ssif [17445.955020] [17445.955020] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17445.958591] ghash_clmulni_intel mei_me [17445.963808] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17445.967291] mei [17445.968782] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17445.975906] lpc_ich sha512_ssse3 [17445.979736] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17445.986860] acpi_ipmi mfd_core [17445.988698] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17445.995822] wmi ipmi_si [17445.999132] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17446.006275] ipmi_devintf [17446.009412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17446.016535] ipmi_msghandler ioatdma [17446.019084] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17446.027162] acpi_pad acpi_power_meter [17446.029779] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17446.035516] ixgbe igb [17446.039086] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17446.046211] mdio crc32c_intel [17446.049955] Call Trace: [17446.049957] [17446.057081] nvme [17446.059438] ? watchdog_timer_fn+0x1b4/0x220 [17446.066566] ptp nvme_core [17446.069622] ? __hrtimer_run_queues+0x1d3/0x360 [17446.072065] i2c_algo_bit [17446.074078] ? hrtimer_interrupt+0x104/0x240 [17446.076000] pps_core dca [17446.080277] ? __local_bh_enable+0x50/0x60 [17446.082985] fuse [17446.087510] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17446.090126] irq event stamp: 62502650 [17446.094390] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17446.097007] hardirqs last enabled at (62502649): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17446.101097] [17446.103022] hardirqs last disabled at (62502650): [] sysvec_apic_timer_interrupt+0xe/0x90 [17446.108240] [17446.108242] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17446.111897] softirqs last enabled at (62502648): [] __do_softirq+0x2be/0x371 [17446.116858] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17446.127013] softirqs last disabled at (62502643): [] irq_exit_rcu+0xb0/0xd0 [17446.129113] do_raw_spin_lock+0xb5/0xc0 [17446.138834] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17446.140933] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17446.146236] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17446.154923] try_get_cap_refs+0x159/0x880 [ceph] [17446.159790] Workqueue: events delayed_work [ceph] [17446.168305] __ceph_get_caps+0x4a1/0x760 [ceph] [17446.172132] [17446.181515] ? __file_remove_privs+0x89/0x110 [17446.186292] RIP: 0010:lock_acquire+0xf0/0x2b0 [17446.193766] ? wake_bit_function+0x70/0x70 [17446.198374] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17446.203076] ? ceph_write_iter+0x260/0x770 [ceph] [17446.207596] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17446.209089] ceph_write_iter+0x260/0x770 [ceph] [17446.213439] [17446.213440] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17446.217793] ? lock_is_held_type+0xaa/0x120 [17446.221880] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17446.240620] vfs_write+0x324/0x4f0 [17446.245315] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17446.250539] ksys_pwrite64+0x65/0xa0 [17446.255055] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17446.256562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17446.263688] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17446.267868] do_syscall_64+0x38/0x80 [17446.274989] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17446.278388] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17446.285511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17446.289082] RIP: 0033:0x7f48f0eeb93b [17446.296206] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17446.301338] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17446.308460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17446.312033] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17446.320110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17446.325152] ORIG_RAX: 0000000000000012 [17446.330892] Call Trace: [17446.330893] [17446.334461] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17446.341587] ? watchdog_timer_fn+0x1b4/0x220 [17446.360323] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17446.367450] ? __hrtimer_run_queues+0x1d3/0x360 [17446.372664] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17446.379793] ? hrtimer_interrupt+0x104/0x240 [17446.383618] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17446.386065] ? __local_bh_enable+0x50/0x60 [17446.388074] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17446.395201] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17446.399469] [17446.406587] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17446.453199] [17446.455294] [17446.457392] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17446.462700] ? lock_acquire+0xf0/0x2b0 [17446.466452] ? igrab+0x43/0x50 [17446.469499] ? lock_release+0xc5/0x260 [17446.473246] _raw_spin_lock+0x30/0x40 [17446.476909] ? igrab+0x1d/0x50 [17446.479962] igrab+0x1d/0x50 [17446.482841] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17446.487990] delayed_work+0x1d8/0x280 [ceph] [17446.492291] ? process_scheduled_works+0x22a/0x510 [17446.497077] process_scheduled_works+0x2a4/0x510 [17446.501694] worker_thread+0x188/0x340 [17446.505443] ? rescuer_thread+0x310/0x310 [17446.509445] kthread+0xf5/0x130 [17446.512592] ? kthread_complete_and_exit+0x20/0x20 [17446.517377] ret_from_fork+0x31/0x40 [17446.520956] ? kthread_complete_and_exit+0x20/0x20 [17446.525738] ret_from_fork_asm+0x11/0x20 [17446.529670] [17449.388358] libceph: mon1 (1)172.21.15.130:6789 session lost, hunting for new mon [17449.396986] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17459.650524] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17470.381168] libceph: auth protocol 'cephx' mauth authentication failed: -13 [17473.761398] watchdog: BUG: soft lockup - CPU#0 stuck for 5159s! [dbench:78683] [17473.768616] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17473.773398] watchdog: BUG: soft lockup - CPU#7 stuck for 14156s! [kworker/7:4:78808] [17473.824058] irq event stamp: 24369642 [17473.824060] hardirqs last enabled at (24369641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17473.831805] Modules linked in: [17473.835462] hardirqs last disabled at (24369642): [] sysvec_apic_timer_interrupt+0xe/0x90 [17473.845619] ceph [17473.848670] softirqs last enabled at (24369622): [] __do_softirq+0x2be/0x371 [17473.858395] libceph [17473.860318] softirqs last disabled at (24369617): [] irq_exit_rcu+0xb0/0xd0 [17473.869004] dns_resolver [17473.871189] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17473.879697] fscache netfs [17473.882316] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17473.891260] veth [17473.893965] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17473.901436] nft_chain_nat [17473.903358] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17473.908836] xt_MASQUERADE [17473.911540] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17473.930279] nf_nat nf_conntrack [17473.932981] [17473.932982] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17473.938199] nf_defrag_ipv6 [17473.941422] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17473.942915] nf_defrag_ipv4 nft_compat [17473.950038] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17473.952828] nf_tables [17473.959952] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17473.963697] nfnetlink bridge [17473.970820] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17473.973179] stp llc [17473.980302] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17473.983268] xfs libcrc32c [17473.990390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17473.992575] sunrpc intel_rapl_msr [17474.000652] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17474.003356] intel_rapl_common sb_edac [17474.009095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17474.012491] x86_pkg_temp_thermal coretemp [17474.019615] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17474.023361] kvm_intel kvm [17474.030483] Call Trace: [17474.034576] irqbypass i2c_i801 [17474.041698] [17474.044403] iTCO_wdt iTCO_vendor_support [17474.046848] ? watchdog_timer_fn+0x1b4/0x220 [17474.049983] i2c_smbus crct10dif_pclmul [17474.051997] ? __hrtimer_run_queues+0x1d3/0x360 [17474.055999] joydev [17474.060267] ? hrtimer_interrupt+0x104/0x240 [17474.064094] crc32_pclmul ipmi_ssif [17474.068618] ? __local_bh_enable+0x50/0x60 [17474.070714] ghash_clmulni_intel mei_me [17474.074982] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17474.078463] mei lpc_ich [17474.082554] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17474.086384] sha512_ssse3 acpi_ipmi [17474.091602] [17474.094132] mfd_core wmi [17474.099091] [17474.099092] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17474.102574] ipmi_si ipmi_devintf [17474.104675] ? queued_spin_lock_slowpath+0x18/0x2e0 [17474.107289] ipmi_msghandler [17474.109389] do_raw_spin_lock+0xb5/0xc0 [17474.114690] ioatdma [17474.118003] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17474.122872] acpi_pad acpi_power_meter [17474.125751] try_get_cap_refs+0x159/0x880 [ceph] [17474.129580] ixgbe [17474.131771] __ceph_get_caps+0x4a1/0x760 [ceph] [17474.136548] igb [17474.140297] ? __file_remove_privs+0x89/0x110 [17474.144903] mdio crc32c_intel [17474.146913] ? wake_bit_function+0x70/0x70 [17474.151438] nvme ptp [17474.153279] ? ceph_write_iter+0x260/0x770 [ceph] [17474.157627] nvme_core i2c_algo_bit [17474.160678] ceph_write_iter+0x260/0x770 [ceph] [17474.164768] pps_core dca [17474.167041] ? lock_is_held_type+0xaa/0x120 [17474.171736] fuse [17474.175222] vfs_write+0x324/0x4f0 [17474.179745] irq event stamp: 62725828 [17474.182366] ksys_pwrite64+0x65/0xa0 [17474.186539] hardirqs last enabled at (62725827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17474.188463] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17474.191860] hardirqs last disabled at (62725828): [] sysvec_apic_timer_interrupt+0xe/0x90 [17474.195518] do_syscall_64+0x38/0x80 [17474.199089] softirqs last enabled at (62725826): [] __do_softirq+0x2be/0x371 [17474.209245] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17474.214378] softirqs last disabled at (62725821): [] irq_exit_rcu+0xb0/0xd0 [17474.224100] RIP: 0033:0x7f48f0eeb93b [17474.227672] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17474.236356] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17474.241399] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17474.249909] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17474.253481] Workqueue: events delayed_work [ceph] [17474.262860] ORIG_RAX: 0000000000000012 [17474.281597] [17474.289082] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17474.294302] RIP: 0010:lock_acquire+0xf0/0x2b0 [17474.298999] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17474.302831] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17474.304319] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17474.311444] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17474.315800] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17474.322929] [17474.322930] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17474.341666] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17474.348801] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17474.354023] [17474.361140] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17474.393312] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17474.400437] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17474.407561] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17474.415639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17474.421375] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17474.428498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17474.435624] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17474.442748] Call Trace: [17474.445193] [17474.447204] ? watchdog_timer_fn+0x1b4/0x220 [17474.451468] ? __hrtimer_run_queues+0x1d3/0x360 [17474.455995] ? hrtimer_interrupt+0x104/0x240 [17474.460264] ? __local_bh_enable+0x50/0x60 [17474.464358] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17474.469581] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17474.474537] [17474.476635] [17474.478732] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17474.484041] ? lock_acquire+0xf0/0x2b0 [17474.487809] ? igrab+0x43/0x50 [17474.490866] ? lock_release+0xc5/0x260 [17474.494611] _raw_spin_lock+0x30/0x40 [17474.498267] ? igrab+0x1d/0x50 [17474.501317] igrab+0x1d/0x50 [17474.504198] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17474.509346] delayed_work+0x1d8/0x280 [ceph] [17474.513626] ? process_scheduled_works+0x22a/0x510 [17474.518408] process_scheduled_works+0x2a4/0x510 [17474.523024] worker_thread+0x188/0x340 [17474.526774] ? rescuer_thread+0x310/0x310 [17474.530804] kthread+0xf5/0x130 [17474.533940] ? kthread_complete_and_exit+0x20/0x20 [17474.538727] ret_from_fork+0x31/0x40 [17474.542302] ? kthread_complete_and_exit+0x20/0x20 [17474.547089] ret_from_fork_asm+0x11/0x20 [17474.551019] [17474.667780] rcu: INFO: rcu_preempt self-detected stall on CPU [17474.673538] rcu: 0-....: (5414637 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=2688189 [17474.683876] rcu: (t=5537960 jiffies g=1309609 q=227332 ncpus=8) [17474.689874] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17474.698818] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17474.706287] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17474.711766] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17474.730503] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17474.735721] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17474.742844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17474.749970] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17474.757093] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17474.764218] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17474.771340] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17474.779419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17474.785156] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17474.792281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17474.799404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17474.806529] Call Trace: [17474.808975] [17474.810986] ? rcu_dump_cpu_stacks+0xd1/0x180 [17474.815343] ? rcu_sched_clock_irq+0x633/0x1390 [17474.819869] ? lock_is_held_type+0xaa/0x120 [17474.824046] ? lock_is_held_type+0xaa/0x120 [17474.828224] ? tick_sched_do_timer+0xa0/0xa0 [17474.832495] ? rcu_read_lock_sched_held+0x56/0x90 [17474.837194] ? tick_sched_do_timer+0xa0/0xa0 [17474.841457] ? update_process_times+0x5f/0x90 [17474.845808] ? tick_sched_handle.isra.16+0x1e/0x50 [17474.850591] ? tick_sched_timer+0x6f/0x80 [17474.854596] ? __hrtimer_run_queues+0x1d3/0x360 [17474.859123] ? hrtimer_interrupt+0x104/0x240 [17474.863393] ? __local_bh_enable+0x50/0x60 [17474.867487] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17474.872711] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17474.877668] [17474.879765] [17474.881862] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17474.887170] ? queued_spin_lock_slowpath+0x18/0x2e0 [17474.892045] do_raw_spin_lock+0xb5/0xc0 [17474.895877] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17474.900679] try_get_cap_refs+0x159/0x880 [ceph] [17474.905311] __ceph_get_caps+0x4a1/0x760 [ceph] [17474.909852] ? __file_remove_privs+0x89/0x110 [17474.914208] ? wake_bit_function+0x70/0x70 [17474.918300] ? ceph_write_iter+0x260/0x770 [ceph] [17474.923011] ceph_write_iter+0x260/0x770 [ceph] [17474.927547] ? lock_is_held_type+0xaa/0x120 [17474.931732] vfs_write+0x324/0x4f0 [17474.935133] ksys_pwrite64+0x65/0xa0 [17474.938708] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17474.943838] do_syscall_64+0x38/0x80 [17474.947408] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17474.952453] RIP: 0033:0x7f48f0eeb93b [17474.956023] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17474.974761] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [17474.982318] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17474.989442] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17474.996565] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17475.003689] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17475.010816] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17475.017945] [17475.020131] Sending NMI from CPU 0 to CPUs 7: [17475.024482] NMI backtrace for cpu 7 [17475.024484] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17475.024486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17475.024487] Workqueue: events delayed_work [ceph] [17475.024502] RIP: 0010:__lock_acquire+0x64c/0x1e20 [17475.024504] Code: 8b 7c 24 38 8b 54 24 28 8b 7c 24 40 c1 e0 0d 66 0b 44 24 18 98 2b 44 24 30 41 31 c7 44 89 f8 44 29 fa 41 01 ff c1 c0 06 31 d0 <29> c7 89 c2 44 01 f8 c1 c2 08 41 89 fc 41 31 d4 44 89 e2 45 29 e7 [17475.024505] RSP: 0018:ffffc90002117c60 EFLAGS: 00000086 [17475.024506] RAX: 0000000096722896 RBX: ffff88810debaa80 RCX: 0000000000000003 [17475.024507] RDX: 00000000a6bedb5d RSI: 0000000000000000 RDI: 0000000089d8b44d [17475.024508] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [17475.024509] R10: 0000000000000000 R11: 0000000000000001 R12: b656a521d3820f2c [17475.024509] R13: 0000000000000003 R14: 0000000000000000 R15: 00000000b69be81c [17475.024510] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17475.024511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17475.024512] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17475.024513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17475.024513] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17475.024514] Call Trace: [17475.024515] [17475.024515] ? nmi_cpu_backtrace+0xad/0x120 [17475.024519] ? nmi_cpu_backtrace_handler+0x11/0x20 [17475.024521] ? nmi_handle+0xd4/0x260 [17475.024525] ? default_do_nmi+0x49/0xf0 [17475.024526] ? exc_nmi+0x11e/0x150 [17475.024527] ? end_repeat_nmi+0x16/0x67 [17475.024531] ? __lock_acquire+0x64c/0x1e20 [17475.024533] ? __lock_acquire+0x64c/0x1e20 [17475.024534] ? __lock_acquire+0x64c/0x1e20 [17475.024536] [17475.024536] [17475.024538] lock_acquire+0xcb/0x2b0 [17475.024539] ? igrab+0x1d/0x50 [17475.024541] ? igrab+0x43/0x50 [17475.024542] ? lock_release+0xc5/0x260 [17475.024544] _raw_spin_lock+0x30/0x40 [17475.024546] ? igrab+0x1d/0x50 [17475.024547] igrab+0x1d/0x50 [17475.024548] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17475.024561] delayed_work+0x1d8/0x280 [ceph] [17475.024575] ? process_scheduled_works+0x22a/0x510 [17475.024577] process_scheduled_works+0x2a4/0x510 [17475.024581] worker_thread+0x188/0x340 [17475.024583] ? rescuer_thread+0x310/0x310 [17475.024585] kthread+0xf5/0x130 [17475.024587] ? kthread_complete_and_exit+0x20/0x20 [17475.024590] ret_from_fork+0x31/0x40 [17475.024592] ? kthread_complete_and_exit+0x20/0x20 [17475.024594] ret_from_fork_asm+0x11/0x20 [17475.024598] [17501.613318] libceph: mon0 (1)172.21.15.50:6789 session lost, hunting for new mon [17501.761913] watchdog: BUG: soft lockup - CPU#0 stuck for 5185s! [dbench:78683] [17501.769130] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17501.773913] watchdog: BUG: soft lockup - CPU#7 stuck for 14182s! [kworker/7:4:78808] [17501.824590] irq event stamp: 24427216 [17501.832321] Modules linked in: ceph libceph dns_resolver fscache netfs [17501.835977] hardirqs last enabled at (24427215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17501.835982] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [17501.852661] hardirqs last disabled at (24427216): [] sysvec_apic_timer_interrupt+0xe/0x90 [17501.860054] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [17501.869778] softirqs last enabled at (24427178): [] __do_softirq+0x2be/0x371 [17501.869782] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [17501.884900] softirqs last disabled at (24427173): [] irq_exit_rcu+0xb0/0xd0 [17501.884903] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [17501.899931] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17501.906793] i2c_i801 iTCO_wdt [17501.915738] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17501.915740] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17501.918788] iTCO_vendor_support i2c_smbus [17501.926261] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17501.931737] crct10dif_pclmul [17501.935827] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17501.954565] joydev crc32_pclmul [17501.957548] [17501.957548] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17501.962765] ipmi_ssif ghash_clmulni_intel [17501.965988] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17501.967479] mei_me mei [17501.974602] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17501.978693] lpc_ich sha512_ssse3 [17501.985817] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17501.988261] acpi_ipmi mfd_core [17501.995387] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17501.998695] wmi [17502.005820] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17502.008958] ipmi_si ipmi_devintf [17502.016081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17502.017918] ipmi_msghandler ioatdma [17502.025996] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17502.029312] acpi_pad acpi_power_meter [17502.035044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17502.038615] ixgbe [17502.045740] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17502.049483] igb mdio [17502.056608] Call Trace: [17502.056611] [17502.058620] crc32c_intel nvme [17502.065745] ? watchdog_timer_fn+0x1b4/0x220 [17502.068013] ptp nvme_core [17502.070460] ? __hrtimer_run_queues+0x1d3/0x360 [17502.072468] i2c_algo_bit pps_core [17502.075522] ? hrtimer_interrupt+0x104/0x240 [17502.079783] dca fuse [17502.082487] ? __local_bh_enable+0x50/0x60 [17502.087012] [17502.087013] irq event stamp: 62946326 [17502.090411] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17502.094672] hardirqs last enabled at (62946325): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17502.096946] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17502.101035] hardirqs last disabled at (62946326): [] sysvec_apic_timer_interrupt+0xe/0x90 [17502.102526] [17502.106184] softirqs last enabled at (62946324): [] __do_softirq+0x2be/0x371 [17502.111399] [17502.111401] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17502.121558] softirqs last disabled at (62946319): [] irq_exit_rcu+0xb0/0xd0 [17502.126520] ? queued_spin_lock_slowpath+0x16/0x2e0 [17502.136240] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17502.138339] do_raw_spin_lock+0xb5/0xc0 [17502.147022] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17502.149119] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17502.154422] Workqueue: events delayed_work [ceph] [17502.162935] try_get_cap_refs+0x159/0x880 [ceph] [17502.167804] [17502.177185] __ceph_get_caps+0x4a1/0x760 [ceph] [17502.181013] RIP: 0010:rcu_is_watching+0x1f/0x40 [17502.188486] ? __file_remove_privs+0x89/0x110 [17502.193268] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [17502.197964] ? wake_bit_function+0x70/0x70 [17502.202592] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [17502.204085] ? ceph_write_iter+0x260/0x770 [ceph] [17502.208607] [17502.208608] RAX: 00000000007c1919 RBX: ffff8881a1d34500 RCX: 0000000000000000 [17502.213132] ceph_write_iter+0x260/0x770 [ceph] [17502.217483] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [17502.236224] ? lock_is_held_type+0xaa/0x120 [17502.240315] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [17502.245538] vfs_write+0x324/0x4f0 [17502.250233] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [17502.251730] ksys_pwrite64+0x65/0xa0 [17502.258848] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17502.263375] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17502.270497] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17502.274676] do_syscall_64+0x38/0x80 [17502.281797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17502.285195] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17502.292320] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17502.295893] RIP: 0033:0x7f48f0eeb93b [17502.303016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17502.308148] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17502.316224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17502.319794] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17502.325530] Call Trace: [17502.325532] [17502.330575] ORIG_RAX: 0000000000000012 [17502.337719] ? watchdog_timer_fn+0x1b4/0x220 [17502.341287] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17502.348413] ? __hrtimer_run_queues+0x1d3/0x360 [17502.367150] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17502.374278] ? hrtimer_interrupt+0x104/0x240 [17502.379491] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17502.381935] ? __local_bh_enable+0x50/0x60 [17502.383944] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17502.387780] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17502.392040] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17502.399165] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17502.403694] [17502.410812] [17502.454996] [17502.457094] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17502.462404] ? rcu_is_watching+0x1f/0x40 [17502.466326] lock_acquire+0x184/0x2b0 [17502.469991] ? igrab+0x43/0x50 [17502.473042] ? lock_release+0xc5/0x260 [17502.476787] _raw_spin_lock+0x30/0x40 [17502.480443] ? igrab+0x1d/0x50 [17502.483493] igrab+0x1d/0x50 [17502.486373] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17502.491522] delayed_work+0x1d8/0x280 [ceph] [17502.495804] ? process_scheduled_works+0x22a/0x510 [17502.500601] process_scheduled_works+0x2a4/0x510 [17502.505216] worker_thread+0x188/0x340 [17502.508966] ? rescuer_thread+0x310/0x310 [17502.512969] kthread+0xf5/0x130 [17502.516108] ? kthread_complete_and_exit+0x20/0x20 [17502.520892] ret_from_fork+0x31/0x40 [17502.524470] ? kthread_complete_and_exit+0x20/0x20 [17502.529255] ret_from_fork_asm+0x11/0x20 [17502.533186] [17529.762429] watchdog: BUG: soft lockup - CPU#0 stuck for 5211s! [dbench:78683] [17529.769645] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17529.774428] watchdog: BUG: soft lockup - CPU#7 stuck for 14208s! [kworker/7:4:78808] [17529.825086] irq event stamp: 24485412 [17529.832834] Modules linked in: ceph libceph dns_resolver [17529.836492] hardirqs last enabled at (24485411): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17529.841820] fscache netfs veth [17529.851978] hardirqs last disabled at (24485412): [] sysvec_apic_timer_interrupt+0xe/0x90 [17529.855117] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [17529.864841] softirqs last enabled at (24485402): [] __do_softirq+0x2be/0x371 [17529.870492] nf_defrag_ipv6 nf_defrag_ipv4 [17529.879175] softirqs last disabled at (24485397): [] irq_exit_rcu+0xb0/0xd0 [17529.879176] nft_compat nf_tables [17529.883266] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17529.891776] nfnetlink bridge [17529.895087] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17529.895089] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17529.904048] stp llc [17529.907012] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17529.914485] xfs libcrc32c [17529.919960] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17529.922146] sunrpc intel_rapl_msr [17529.940883] [17529.940883] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17529.943588] intel_rapl_common sb_edac [17529.948822] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17529.952220] x86_pkg_temp_thermal [17529.953711] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17529.960835] coretemp [17529.964579] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17529.971701] kvm_intel [17529.975013] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17529.982138] kvm irqbypass [17529.984408] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17529.991533] i2c_i801 iTCO_wdt [17529.993890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17530.001012] iTCO_vendor_support i2c_smbus [17530.003718] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17530.011795] crct10dif_pclmul joydev [17530.014845] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17530.020583] crc32_pclmul ipmi_ssif [17530.024675] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17530.031799] ghash_clmulni_intel [17530.035386] Call Trace: [17530.035388] [17530.042511] mei_me mei [17530.045995] ? watchdog_timer_fn+0x1b4/0x220 [17530.053117] lpc_ich sha512_ssse3 [17530.056344] ? __hrtimer_run_queues+0x1d3/0x360 [17530.058787] acpi_ipmi mfd_core [17530.060804] ? hrtimer_interrupt+0x104/0x240 [17530.063242] wmi ipmi_si [17530.067504] ? __local_bh_enable+0x50/0x60 [17530.070827] ipmi_devintf ipmi_msghandler [17530.075350] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17530.078486] ioatdma [17530.082751] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17530.085281] acpi_pad acpi_power_meter [17530.089371] [17530.093375] ixgbe igb [17530.098594] [17530.100777] mdio crc32c_intel [17530.105736] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17530.109478] nvme ptp [17530.111581] ? queued_spin_lock_slowpath+0x16/0x2e0 [17530.113934] nvme_core i2c_algo_bit [17530.116034] do_raw_spin_lock+0xb5/0xc0 [17530.119082] pps_core dca [17530.124386] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17530.126656] fuse [17530.131529] try_get_cap_refs+0x159/0x880 [ceph] [17530.135012] irq event stamp: 63168614 [17530.138848] __ceph_get_caps+0x4a1/0x760 [ceph] [17530.141458] hardirqs last enabled at (63168613): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17530.146247] ? __file_remove_privs+0x89/0x110 [17530.148168] hardirqs last disabled at (63168614): [] sysvec_apic_timer_interrupt+0xe/0x90 [17530.152779] ? wake_bit_function+0x70/0x70 [17530.156435] softirqs last enabled at (63168612): [] __do_softirq+0x2be/0x371 [17530.160963] ? ceph_write_iter+0x260/0x770 [ceph] [17530.171119] softirqs last disabled at (63168607): [] irq_exit_rcu+0xb0/0xd0 [17530.175469] ceph_write_iter+0x260/0x770 [ceph] [17530.185193] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17530.189287] ? lock_is_held_type+0xaa/0x120 [17530.197967] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17530.202666] vfs_write+0x324/0x4f0 [17530.211176] Workqueue: events delayed_work [ceph] [17530.215704] ksys_pwrite64+0x65/0xa0 [17530.225076] [17530.229255] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17530.236726] RIP: 0010:lock_acquire+0xf0/0x2b0 [17530.240125] do_syscall_64+0x38/0x80 [17530.244830] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17530.248400] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17530.249889] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17530.255022] RIP: 0033:0x7f48f0eeb93b [17530.259372] [17530.262943] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17530.281681] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17530.286725] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17530.291943] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17530.295511] ORIG_RAX: 0000000000000012 [17530.297003] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17530.315740] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17530.322864] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17530.328083] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17530.335206] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17530.339038] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17530.346161] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17530.353286] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17530.360408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17530.367532] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17530.374658] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17530.381787] [17530.389860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17530.426278] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17530.433403] Call Trace: [17530.435845] [17530.437856] ? watchdog_timer_fn+0x1b4/0x220 [17530.442121] ? __hrtimer_run_queues+0x1d3/0x360 [17530.446648] ? hrtimer_interrupt+0x104/0x240 [17530.450917] ? __local_bh_enable+0x50/0x60 [17530.455012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17530.460236] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17530.465191] [17530.467288] [17530.469388] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17530.474695] ? lock_acquire+0xf0/0x2b0 [17530.478444] ? igrab+0x43/0x50 [17530.481495] ? lock_release+0xc5/0x260 [17530.485240] _raw_spin_lock+0x30/0x40 [17530.488896] ? igrab+0x1d/0x50 [17530.491946] igrab+0x1d/0x50 [17530.494837] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17530.499983] delayed_work+0x1d8/0x280 [ceph] [17530.504263] ? process_scheduled_works+0x22a/0x510 [17530.509053] process_scheduled_works+0x2a4/0x510 [17530.513669] worker_thread+0x188/0x340 [17530.517418] ? rescuer_thread+0x310/0x310 [17530.521423] kthread+0xf5/0x130 [17530.524559] ? kthread_complete_and_exit+0x20/0x20 [17530.529345] ret_from_fork+0x31/0x40 [17530.532924] ? kthread_complete_and_exit+0x20/0x20 [17530.537707] ret_from_fork_asm+0x11/0x20 [17530.541630] [17557.762949] watchdog: BUG: soft lockup - CPU#0 stuck for 5237s! [dbench:78683] [17557.770167] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17557.774948] watchdog: BUG: soft lockup - CPU#7 stuck for 14234s! [kworker/7:4:78808] [17557.825622] irq event stamp: 24543578 [17557.825623] hardirqs last enabled at (24543577): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17557.833356] Modules linked in: ceph [17557.837014] hardirqs last disabled at (24543578): [] sysvec_apic_timer_interrupt+0xe/0x90 [17557.847172] libceph dns_resolver [17557.850657] softirqs last enabled at (24543532): [] __do_softirq+0x2be/0x371 [17557.860379] fscache netfs [17557.863690] softirqs last disabled at (24543527): [] irq_exit_rcu+0xb0/0xd0 [17557.872375] veth nft_chain_nat [17557.875081] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17557.883589] xt_MASQUERADE nf_nat [17557.886728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17557.895670] nf_conntrack [17557.898981] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17557.906452] nf_defrag_ipv6 nf_defrag_ipv4 [17557.909072] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17557.914548] nft_compat nf_tables [17557.918638] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17557.937376] nfnetlink bridge [17557.940686] [17557.945903] stp llc [17557.948868] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17557.950358] xfs libcrc32c [17557.952541] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17557.959668] sunrpc intel_rapl_msr [17557.962370] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17557.969496] intel_rapl_common [17557.972893] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17557.980015] sb_edac x86_pkg_temp_thermal [17557.983067] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17557.990191] coretemp kvm_intel [17557.994196] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17558.001319] kvm irqbypass [17558.004474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17558.012550] i2c_i801 iTCO_wdt [17558.015255] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17558.020993] iTCO_vendor_support i2c_smbus [17558.024044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17558.031167] crct10dif_pclmul joydev [17558.035259] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17558.042382] crc32_pclmul [17558.045953] Call Trace: [17558.045954] [17558.053077] ipmi_ssif ghash_clmulni_intel [17558.055697] ? watchdog_timer_fn+0x1b4/0x220 [17558.058138] mei_me mei [17558.060151] ? __hrtimer_run_queues+0x1d3/0x360 [17558.064241] lpc_ich sha512_ssse3 [17558.068507] ? hrtimer_interrupt+0x104/0x240 [17558.070949] acpi_ipmi mfd_core [17558.075472] ? __local_bh_enable+0x50/0x60 [17558.078783] wmi ipmi_si [17558.083051] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17558.086186] ipmi_devintf ipmi_msghandler [17558.090277] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17558.092806] ioatdma [17558.098024] [17558.102028] acpi_pad [17558.106984] [17558.109169] acpi_power_meter [17558.111267] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17558.113538] ixgbe igb [17558.115638] ? queued_spin_lock_slowpath+0x18/0x2e0 [17558.118598] mdio crc32c_intel [17558.123906] do_raw_spin_lock+0xb5/0xc0 [17558.126261] nvme [17558.131132] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17558.134181] ptp nvme_core [17558.138016] try_get_cap_refs+0x159/0x880 [ceph] [17558.139938] i2c_algo_bit pps_core dca [17558.144727] __ceph_get_caps+0x4a1/0x760 [ceph] [17558.147424] fuse [17558.152040] ? __file_remove_privs+0x89/0x110 [17558.155779] [17558.155779] irq event stamp: 63390224 [17558.160303] ? wake_bit_function+0x70/0x70 [17558.162227] hardirqs last enabled at (63390223): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17558.166582] ? ceph_write_iter+0x260/0x770 [ceph] [17558.168070] hardirqs last disabled at (63390224): [] sysvec_apic_timer_interrupt+0xe/0x90 [17558.171726] ceph_write_iter+0x260/0x770 [ceph] [17558.175816] softirqs last enabled at (63390222): [] __do_softirq+0x2be/0x371 [17558.185979] ? lock_is_held_type+0xaa/0x120 [17558.190671] softirqs last disabled at (63390217): [] irq_exit_rcu+0xb0/0xd0 [17558.200399] vfs_write+0x324/0x4f0 [17558.204922] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17558.213611] ksys_pwrite64+0x65/0xa0 [17558.217783] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17558.226293] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17558.229690] Workqueue: events delayed_work [ceph] [17558.239069] do_syscall_64+0x38/0x80 [17558.242639] [17558.242640] RIP: 0010:lock_acquire+0xf0/0x2b0 [17558.250110] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17558.255241] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17558.259938] RIP: 0033:0x7f48f0eeb93b [17558.263509] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17558.264999] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17558.269352] [17558.274394] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17558.293132] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17558.296702] ORIG_RAX: 0000000000000012 [17558.301920] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17558.320657] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17558.322148] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17558.327364] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17558.334490] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17558.338321] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17558.345445] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17558.352570] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17558.359692] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17558.366816] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17558.373942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17558.381070] [17558.388189] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17558.425561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17558.432686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17558.439809] Call Trace: [17558.442254] [17558.444264] ? watchdog_timer_fn+0x1b4/0x220 [17558.448529] ? __hrtimer_run_queues+0x1d3/0x360 [17558.453056] ? hrtimer_interrupt+0x104/0x240 [17558.457324] ? __local_bh_enable+0x50/0x60 [17558.461418] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17558.466643] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17558.471599] [17558.473698] [17558.475794] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17558.481104] ? lock_acquire+0xf0/0x2b0 [17558.484851] ? igrab+0x43/0x50 [17558.487901] ? lock_release+0xc5/0x260 [17558.491647] _raw_spin_lock+0x30/0x40 [17558.495312] ? igrab+0x1d/0x50 [17558.498362] igrab+0x1d/0x50 [17558.501243] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17558.506392] delayed_work+0x1d8/0x280 [ceph] [17558.510671] ? process_scheduled_works+0x22a/0x510 [17558.515462] process_scheduled_works+0x2a4/0x510 [17558.520077] worker_thread+0x188/0x340 [17558.523827] ? rescuer_thread+0x310/0x310 [17558.527829] kthread+0xf5/0x130 [17558.530977] ? kthread_complete_and_exit+0x20/0x20 [17558.535760] ret_from_fork+0x31/0x40 [17558.539358] ? kthread_complete_and_exit+0x20/0x20 [17558.544148] ret_from_fork_asm+0x11/0x20 [17558.548074] [17585.763468] watchdog: BUG: soft lockup - CPU#0 stuck for 5264s! [dbench:78683] [17585.770679] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17585.775468] watchdog: BUG: soft lockup - CPU#7 stuck for 14261s! [kworker/7:4:78808] [17585.826123] irq event stamp: 24601872 [17585.833864] Modules linked in: ceph libceph [17585.837519] hardirqs last enabled at (24601871): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17585.841697] dns_resolver fscache netfs [17585.851860] hardirqs last disabled at (24601872): [] sysvec_apic_timer_interrupt+0xe/0x90 [17585.855692] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [17585.865417] softirqs last enabled at (24601826): [] __do_softirq+0x2be/0x371 [17585.865420] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [17585.880184] softirqs last disabled at (24601821): [] irq_exit_rcu+0xb0/0xd0 [17585.886962] bridge stp [17585.895475] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17585.897917] llc xfs [17585.906861] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17585.909046] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [17585.916517] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17585.922340] sb_edac x86_pkg_temp_thermal [17585.927818] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17585.927818] coretemp kvm_intel [17585.931823] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17585.950561] kvm [17585.953696] [17585.953697] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17585.958914] irqbypass i2c_i801 [17585.960751] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17585.962244] iTCO_wdt [17585.969367] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17585.972505] iTCO_vendor_support i2c_smbus [17585.979628] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17585.981900] crct10dif_pclmul joydev [17585.989024] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17585.993113] crc32_pclmul ipmi_ssif [17586.000237] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17586.003809] ghash_clmulni_intel mei_me [17586.010934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17586.014418] mei lpc_ich [17586.022494] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17586.026325] sha512_ssse3 acpi_ipmi [17586.032063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17586.034594] mfd_core wmi [17586.041718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17586.045203] ipmi_si [17586.052326] Call Trace: [17586.052328] [17586.054944] ipmi_devintf ipmi_msghandler [17586.062069] ? watchdog_timer_fn+0x1b4/0x220 [17586.064252] ioatdma acpi_pad [17586.066696] ? __hrtimer_run_queues+0x1d3/0x360 [17586.068707] acpi_power_meter ixgbe [17586.072715] ? hrtimer_interrupt+0x104/0x240 [17586.076974] igb mdio [17586.079939] ? __local_bh_enable+0x50/0x60 [17586.084462] crc32c_intel [17586.087950] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17586.092209] nvme ptp [17586.094483] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17586.098573] nvme_core [17586.101190] [17586.106408] i2c_algo_bit [17586.108678] [17586.108680] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17586.113636] pps_core [17586.115997] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17586.118089] dca fuse [17586.120709] do_raw_spin_lock+0xb5/0xc0 [17586.122804] [17586.122805] irq event stamp: 63612056 [17586.128110] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17586.130379] hardirqs last enabled at (63612055): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17586.135253] try_get_cap_refs+0x159/0x880 [ceph] [17586.137521] hardirqs last disabled at (63612056): [] sysvec_apic_timer_interrupt+0xe/0x90 [17586.141357] __ceph_get_caps+0x4a1/0x760 [ceph] [17586.142843] softirqs last enabled at (63612054): [] __do_softirq+0x2be/0x371 [17586.146520] ? __file_remove_privs+0x89/0x110 [17586.151302] softirqs last disabled at (63612049): [] irq_exit_rcu+0xb0/0xd0 [17586.161458] ? wake_bit_function+0x70/0x70 [17586.166069] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17586.175796] ? ceph_write_iter+0x260/0x770 [ceph] [17586.180317] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17586.189002] ceph_write_iter+0x260/0x770 [ceph] [17586.193351] Workqueue: events delayed_work [ceph] [17586.201875] ? lock_is_held_type+0xaa/0x120 [17586.205963] [17586.215343] vfs_write+0x324/0x4f0 [17586.220037] RIP: 0010:lock_acquire+0xf0/0x2b0 [17586.227513] ksys_pwrite64+0x65/0xa0 [17586.232033] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17586.236730] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17586.240907] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17586.242398] do_syscall_64+0x38/0x80 [17586.245796] [17586.245797] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17586.250157] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17586.253726] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17586.272463] RIP: 0033:0x7f48f0eeb93b [17586.277594] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17586.282811] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17586.286382] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17586.287874] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17586.294998] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17586.300040] ORIG_RAX: 0000000000000012 [17586.307165] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17586.310735] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17586.317871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17586.336607] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17586.343729] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17586.348949] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17586.356072] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17586.359903] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17586.367980] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17586.375105] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17586.380841] Call Trace: [17586.380842] [17586.387971] [17586.395090] ? watchdog_timer_fn+0x1b4/0x220 [17586.441625] ? __hrtimer_run_queues+0x1d3/0x360 [17586.446164] ? hrtimer_interrupt+0x104/0x240 [17586.450427] ? __local_bh_enable+0x50/0x60 [17586.454522] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17586.459745] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17586.464702] [17586.466800] [17586.468899] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17586.474206] ? lock_acquire+0xf0/0x2b0 [17586.477955] ? igrab+0x43/0x50 [17586.481005] ? lock_release+0xc5/0x260 [17586.484752] _raw_spin_lock+0x30/0x40 [17586.488415] ? igrab+0x1d/0x50 [17586.491466] igrab+0x1d/0x50 [17586.494345] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17586.499494] delayed_work+0x1d8/0x280 [ceph] [17586.503775] ? process_scheduled_works+0x22a/0x510 [17586.508565] process_scheduled_works+0x2a4/0x510 [17586.513180] worker_thread+0x188/0x340 [17586.516930] ? rescuer_thread+0x310/0x310 [17586.520932] kthread+0xf5/0x130 [17586.524071] ? kthread_complete_and_exit+0x20/0x20 [17586.528872] ret_from_fork+0x31/0x40 [17586.532442] ? kthread_complete_and_exit+0x20/0x20 [17586.537227] ret_from_fork_asm+0x11/0x20 [17586.541162] [17613.763988] watchdog: BUG: soft lockup - CPU#0 stuck for 5290s! [dbench:78683] [17613.771203] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17613.775988] watchdog: BUG: soft lockup - CPU#7 stuck for 14287s! [kworker/7:4:78808] [17613.826664] irq event stamp: 24659994 [17613.834393] Modules linked in: ceph libceph dns_resolver fscache netfs veth [17613.838050] hardirqs last enabled at (24659993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17613.845009] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [17613.855166] hardirqs last disabled at (24659994): [] sysvec_apic_timer_interrupt+0xe/0x90 [17613.862117] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [17613.871842] softirqs last enabled at (24659958): [] __do_softirq+0x2be/0x371 [17613.871846] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [17613.887312] softirqs last disabled at (24659953): [] irq_exit_rcu+0xb0/0xd0 [17613.887315] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [17613.902688] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17613.909636] iTCO_wdt iTCO_vendor_support i2c_smbus [17613.918583] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17613.918585] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17613.923453] crct10dif_pclmul [17613.930924] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17613.936399] joydev [17613.939382] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17613.958119] crc32_pclmul ipmi_ssif [17613.960217] [17613.965434] ghash_clmulni_intel [17613.968918] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17613.970408] mei_me [17613.973634] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17613.980757] mei lpc_ich [17613.982855] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17613.989979] sha512_ssse3 acpi_ipmi [17613.992510] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17613.999634] mfd_core [17614.003117] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17614.010241] wmi ipmi_si [17614.012513] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17614.019636] ipmi_devintf ipmi_msghandler [17614.022168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17614.030245] ioatdma acpi_pad [17614.034249] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17614.039987] acpi_power_meter ixgbe [17614.042951] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17614.050074] igb mdio [17614.053559] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17614.060684] crc32c_intel [17614.062952] Call Trace: [17614.062955] [17614.070078] nvme ptp [17614.072696] ? watchdog_timer_fn+0x1b4/0x220 [17614.075138] nvme_core [17614.077150] ? __hrtimer_run_queues+0x1d3/0x360 [17614.079420] i2c_algo_bit pps_core [17614.083688] ? hrtimer_interrupt+0x104/0x240 [17614.086042] dca fuse [17614.090566] ? __local_bh_enable+0x50/0x60 [17614.093964] [17614.093965] irq event stamp: 63834700 [17614.098231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17614.100497] hardirqs last enabled at (63834699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17614.104588] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17614.106079] hardirqs last disabled at (63834700): [] sysvec_apic_timer_interrupt+0xe/0x90 [17614.109737] [17614.114955] softirqs last enabled at (63834698): [] __do_softirq+0x2be/0x371 [17614.125113] [17614.130070] softirqs last disabled at (63834693): [] irq_exit_rcu+0xb0/0xd0 [17614.139793] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17614.141892] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17614.150580] ? queued_spin_lock_slowpath+0x16/0x2e0 [17614.152673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17614.161185] do_raw_spin_lock+0xb5/0xc0 [17614.166488] Workqueue: events delayed_work [ceph] [17614.175867] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17614.180736] [17614.188209] try_get_cap_refs+0x159/0x880 [ceph] [17614.192036] RIP: 0010:lock_acquire+0xf0/0x2b0 [17614.196741] __ceph_get_caps+0x4a1/0x760 [ceph] [17614.201517] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17614.203013] ? __file_remove_privs+0x89/0x110 [17614.207620] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17614.211972] ? wake_bit_function+0x70/0x70 [17614.216495] [17614.235235] ? ceph_write_iter+0x260/0x770 [ceph] [17614.239582] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17614.244802] ceph_write_iter+0x260/0x770 [ceph] [17614.248891] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17614.250396] ? lock_is_held_type+0xaa/0x120 [17614.255088] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17614.262216] vfs_write+0x324/0x4f0 [17614.266737] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17614.273864] ksys_pwrite64+0x65/0xa0 [17614.278038] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17614.285162] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17614.288560] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17614.295684] do_syscall_64+0x38/0x80 [17614.299256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17614.306392] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17614.311518] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17614.319596] RIP: 0033:0x7f48f0eeb93b [17614.323167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17614.328904] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17614.333947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17614.341073] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17614.344643] Call Trace: [17614.344644] [17614.351766] ORIG_RAX: 0000000000000012 [17614.370506] ? watchdog_timer_fn+0x1b4/0x220 [17614.377628] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17614.382848] ? __hrtimer_run_queues+0x1d3/0x360 [17614.385290] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17614.387305] ? hrtimer_interrupt+0x104/0x240 [17614.391131] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17614.395395] ? __local_bh_enable+0x50/0x60 [17614.402521] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17614.407046] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17614.414168] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17614.418434] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17614.425562] [17614.429647] [17614.458353] [17614.460450] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17614.465758] ? lock_acquire+0xf0/0x2b0 [17614.469508] ? igrab+0x43/0x50 [17614.472557] ? lock_release+0xc5/0x260 [17614.476302] _raw_spin_lock+0x30/0x40 [17614.479958] ? igrab+0x1d/0x50 [17614.483010] igrab+0x1d/0x50 [17614.485888] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17614.491037] delayed_work+0x1d8/0x280 [ceph] [17614.495319] ? process_scheduled_works+0x22a/0x510 [17614.500109] process_scheduled_works+0x2a4/0x510 [17614.504725] worker_thread+0x188/0x340 [17614.508473] ? rescuer_thread+0x310/0x310 [17614.512478] kthread+0xf5/0x130 [17614.515624] ? kthread_complete_and_exit+0x20/0x20 [17614.520406] ret_from_fork+0x31/0x40 [17614.523987] ? kthread_complete_and_exit+0x20/0x20 [17614.528769] ret_from_fork_asm+0x11/0x20 [17614.532694] [17641.764508] watchdog: BUG: soft lockup - CPU#0 stuck for 5316s! [dbench:78683] [17641.771725] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17641.776508] watchdog: BUG: soft lockup - CPU#7 stuck for 14313s! [kworker/7:4:78808] [17641.827183] irq event stamp: 24718132 [17641.827184] hardirqs last enabled at (24718131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17641.834913] Modules linked in: [17641.838571] hardirqs last disabled at (24718132): [] sysvec_apic_timer_interrupt+0xe/0x90 [17641.848729] ceph [17641.851780] softirqs last enabled at (24718074): [] __do_softirq+0x2be/0x371 [17641.861504] libceph [17641.863427] softirqs last disabled at (24718069): [] irq_exit_rcu+0xb0/0xd0 [17641.872112] dns_resolver [17641.874297] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17641.882806] fscache netfs [17641.885423] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17641.894368] veth nft_chain_nat [17641.897071] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17641.904544] xt_MASQUERADE nf_nat [17641.907680] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17641.913157] nf_conntrack nf_defrag_ipv6 [17641.916467] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17641.935205] nf_defrag_ipv4 nft_compat [17641.939123] [17641.939124] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17641.944341] nf_tables nfnetlink [17641.948086] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17641.949575] bridge stp [17641.956701] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17641.959925] llc xfs [17641.967047] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17641.969493] libcrc32c [17641.976616] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17641.978799] sunrpc intel_rapl_msr [17641.985924] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17641.988282] intel_rapl_common sb_edac [17641.995406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17641.998804] x86_pkg_temp_thermal coretemp [17642.006880] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17642.010642] kvm_intel [17642.016379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17642.020471] kvm irqbypass [17642.027595] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17642.029952] i2c_i801 iTCO_wdt [17642.037076] Call Trace: [17642.037077] [17642.039781] iTCO_vendor_support i2c_smbus [17642.046905] ? watchdog_timer_fn+0x1b4/0x220 [17642.049955] crct10dif_pclmul joydev [17642.052401] ? __hrtimer_run_queues+0x1d3/0x360 [17642.054409] crc32_pclmul ipmi_ssif [17642.058505] ? hrtimer_interrupt+0x104/0x240 [17642.062764] ghash_clmulni_intel mei_me [17642.066334] ? __local_bh_enable+0x50/0x60 [17642.070859] mei lpc_ich [17642.074346] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17642.078608] sha512_ssse3 acpi_ipmi [17642.082439] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17642.086529] mfd_core wmi [17642.089059] [17642.094276] ipmi_si ipmi_devintf [17642.097760] [17642.097762] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17642.102718] ipmi_msghandler [17642.105339] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17642.107432] ioatdma acpi_pad [17642.110745] do_raw_spin_lock+0xb5/0xc0 [17642.112841] acpi_power_meter ixgbe [17642.118146] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17642.121022] igb mdio [17642.125910] try_get_cap_refs+0x159/0x880 [ceph] [17642.128876] crc32c_intel [17642.132712] __ceph_get_caps+0x4a1/0x760 [ceph] [17642.136206] nvme [17642.140994] ? __file_remove_privs+0x89/0x110 [17642.143263] ptp nvme_core [17642.147873] ? wake_bit_function+0x70/0x70 [17642.150489] i2c_algo_bit pps_core [17642.155017] ? ceph_write_iter+0x260/0x770 [ceph] [17642.156939] dca fuse [17642.161290] ceph_write_iter+0x260/0x770 [ceph] [17642.163992] [17642.163993] irq event stamp: 64057500 [17642.168088] ? lock_is_held_type+0xaa/0x120 [17642.171481] hardirqs last enabled at (64057499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17642.176182] vfs_write+0x324/0x4f0 [17642.178468] hardirqs last disabled at (64057500): [] sysvec_apic_timer_interrupt+0xe/0x90 [17642.182995] ksys_pwrite64+0x65/0xa0 [17642.184483] softirqs last enabled at (64057498): [] __do_softirq+0x2be/0x371 [17642.188140] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17642.192317] softirqs last disabled at (64057493): [] irq_exit_rcu+0xb0/0xd0 [17642.202475] do_syscall_64+0x38/0x80 [17642.205871] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17642.215598] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17642.219165] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17642.219182] Workqueue: events delayed_work [ceph] [17642.227868] RIP: 0033:0x7f48f0eeb93b [17642.232999] [17642.233000] RIP: 0010:lock_acquire+0xf0/0x2b0 [17642.241509] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17642.245079] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17642.254457] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17642.259503] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17642.266971] ORIG_RAX: 0000000000000012 [17642.271669] [17642.271669] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17642.275240] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17642.276731] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17642.281082] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17642.299821] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17642.318557] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17642.323774] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17642.328991] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17642.332824] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17642.334314] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17642.341437] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17642.348567] [17642.355685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17642.421562] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17642.428687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17642.435811] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17642.442934] Call Trace: [17642.445380] [17642.447391] ? watchdog_timer_fn+0x1b4/0x220 [17642.451656] ? __hrtimer_run_queues+0x1d3/0x360 [17642.456181] ? hrtimer_interrupt+0x104/0x240 [17642.460441] ? __local_bh_enable+0x50/0x60 [17642.464535] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17642.469751] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17642.474708] [17642.476804] [17642.478916] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17642.484228] ? lock_acquire+0xf0/0x2b0 [17642.487978] ? igrab+0x43/0x50 [17642.491028] ? lock_release+0xc5/0x260 [17642.494773] _raw_spin_lock+0x30/0x40 [17642.498429] ? igrab+0x1d/0x50 [17642.501479] igrab+0x1d/0x50 [17642.504357] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17642.509507] delayed_work+0x1d8/0x280 [ceph] [17642.513789] ? process_scheduled_works+0x22a/0x510 [17642.518579] process_scheduled_works+0x2a4/0x510 [17642.523208] worker_thread+0x188/0x340 [17642.526954] ? rescuer_thread+0x310/0x310 [17642.530965] kthread+0xf5/0x130 [17642.534102] ? kthread_complete_and_exit+0x20/0x20 [17642.538887] ret_from_fork+0x31/0x40 [17642.542482] ? kthread_complete_and_exit+0x20/0x20 [17642.547267] ret_from_fork_asm+0x11/0x20 [17642.551189] [17669.765029] watchdog: BUG: soft lockup - CPU#0 stuck for 5342s! [dbench:78683] [17669.772246] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17669.777029] watchdog: BUG: soft lockup - CPU#7 stuck for 14339s! [kworker/7:4:78808] [17669.827711] irq event stamp: 24776368 [17669.835445] Modules linked in: ceph libceph dns_resolver fscache netfs veth [17669.839101] hardirqs last enabled at (24776367): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17669.846061] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [17669.856218] hardirqs last disabled at (24776368): [] sysvec_apic_timer_interrupt+0xe/0x90 [17669.864470] nft_compat nf_tables nfnetlink bridge stp llc xfs [17669.874195] softirqs last enabled at (24776310): [] __do_softirq+0x2be/0x371 [17669.880025] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [17669.888710] softirqs last disabled at (24776305): [] irq_exit_rcu+0xb0/0xd0 [17669.897047] coretemp kvm_intel kvm [17669.905560] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17669.905561] irqbypass [17669.905562] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17669.909042] i2c_i801 iTCO_wdt [17669.917986] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17669.920344] iTCO_vendor_support [17669.927816] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17669.930866] i2c_smbus crct10dif_pclmul [17669.936342] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17669.939567] joydev crc32_pclmul [17669.958307] [17669.962135] ipmi_ssif ghash_clmulni_intel [17669.967354] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17669.970576] mei_me mei [17669.972068] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17669.976159] lpc_ich sha512_ssse3 [17669.983284] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17669.985726] acpi_ipmi mfd_core [17669.992850] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17669.996161] wmi ipmi_si [17670.003285] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17670.006428] ipmi_devintf [17670.013556] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17670.016088] ipmi_msghandler ioatdma [17670.023210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17670.025828] acpi_pad acpi_power_meter [17670.033906] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17670.036402] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [17670.036422] rcu: 0-....: (5605550 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=2784159 [17670.036427] rcu: 7-....: (5569824 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/5568597 fqs=2784159 [17670.036430] rcu: (detected by 4, t=5733309 jiffies, g=1309609, q=233008 ncpus=8) [17670.036432] Sending NMI from CPU 4 to CPUs 0: [17670.036436] NMI backtrace for cpu 0 [17670.036437] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17670.036439] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17670.036439] RIP: 0010:vprintk_emit+0x1ee/0x340 [17670.036443] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [17670.036445] RSP: 0018:ffffc90000003db0 EFLAGS: 00000002 [17670.036446] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [17670.036448] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [17670.036449] RBP: 0000000000000041 R08: 0000000000005ef4 R09: 0000000000000004 [17670.036450] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [17670.036451] R13: 0000000000000000 R14: ffffffff82188808 R15: ffffc90000003e00 [17670.036452] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17670.036454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17670.036455] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17670.036456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17670.036457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17670.036459] Call Trace: [17670.036460] [17670.036462] ? nmi_cpu_backtrace+0xad/0x120 [17670.036467] ? nmi_cpu_backtrace_handler+0x11/0x20 [17670.036469] ? nmi_handle+0xd4/0x260 [17670.036475] ? default_do_nmi+0x49/0xf0 [17670.036478] ? exc_nmi+0x11e/0x150 [17670.036480] ? end_repeat_nmi+0x16/0x67 [17670.036487] ? vprintk_emit+0x1ee/0x340 [17670.036489] ? vprintk_emit+0x1ee/0x340 [17670.036492] ? vprintk_emit+0x1ee/0x340 [17670.036495] [17670.036496] [17670.036498] _printk+0x5c/0x80 [17670.036503] __show_regs+0x1a1/0x2e0 [17670.036508] ? touch_softlockup_watchdog+0x20/0x20 [17670.036510] show_regs+0x38/0x60 [17670.036513] watchdog_timer_fn+0x1b4/0x220 [17670.036516] __hrtimer_run_queues+0x1d3/0x360 [17670.036522] hrtimer_interrupt+0x104/0x240 [17670.036523] ? __local_bh_enable+0x50/0x60 [17670.036528] __sysvec_apic_timer_interrupt+0x61/0x160 [17670.036532] sysvec_apic_timer_interrupt+0x6d/0x90 [17670.036535] [17670.036535] [17670.036536] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17670.036539] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17670.036542] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17670.036544] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17670.036545] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17670.036546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17670.036547] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17670.036548] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17670.036549] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17670.036556] do_raw_spin_lock+0xb5/0xc0 [17670.036560] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17670.036587] try_get_cap_refs+0x159/0x880 [ceph] [17670.036609] __ceph_get_caps+0x4a1/0x760 [ceph] [17670.036628] ? __file_remove_privs+0x89/0x110 [17670.036631] ? wake_bit_function+0x70/0x70 [17670.036636] ? ceph_write_iter+0x260/0x770 [ceph] [17670.036651] ceph_write_iter+0x260/0x770 [ceph] [17670.036669] ? lock_is_held_type+0xaa/0x120 [17670.036673] vfs_write+0x324/0x4f0 [17670.036680] ksys_pwrite64+0x65/0xa0 [17670.036683] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17670.036688] do_syscall_64+0x38/0x80 [17670.036692] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17670.036694] RIP: 0033:0x7f48f0eeb93b [17670.036696] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17670.036698] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [17670.036700] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17670.036701] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17670.036702] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17670.036703] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17670.036705] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17670.036710] [17670.037435] Sending NMI from CPU 4 to CPUs 7: [17670.037438] NMI backtrace for cpu 7 [17670.037439] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17670.037441] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17670.037442] Workqueue: events delayed_work [ceph] [17670.037459] RIP: 0010:delay_tsc+0x4d/0x90 [17670.037462] Code: c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e 65 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 48 89 c2 4c 29 c2 48 39 fa 72 c8 65 ff [17670.037463] RSP: 0018:ffffc90000264c80 EFLAGS: 00000046 [17670.037464] RAX: 000000001adb7f45 RBX: 0000000000000000 RCX: 0000000000000007 [17670.037465] RDX: 00000000000039d9 RSI: 0000000000000007 RDI: 0000000000000dac [17670.037466] RBP: 00000000000025b4 R08: 000039d91adb7659 R09: 0000000000000007 [17670.037466] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000060 [17670.037467] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [17670.037468] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17670.037469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17670.037470] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17670.037471] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17670.037471] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17670.037472] Call Trace: [17670.037473] [17670.037474] ? nmi_cpu_backtrace+0xad/0x120 [17670.037477] ? nmi_cpu_backtrace_handler+0x11/0x20 [17670.037478] ? nmi_handle+0xd4/0x260 [17670.037482] ? default_do_nmi+0x49/0xf0 [17670.037483] ? exc_nmi+0x11e/0x150 [17670.037485] ? end_repeat_nmi+0x16/0x67 [17670.037489] ? delay_tsc+0x4d/0x90 [17670.037491] ? delay_tsc+0x4d/0x90 [17670.037493] ? delay_tsc+0x4d/0x90 [17670.037494] [17670.037495] [17670.037495] wait_for_lsr+0x27/0x70 [17670.037499] wait_for_xmitr+0xf/0x60 [17670.037501] serial8250_console_write+0x22a/0x3f0 [17670.037504] ? lock_release+0xc5/0x260 [17670.037507] console_flush_all+0x227/0x5a0 [17670.037509] ? console_flush_all+0x1ff/0x5a0 [17670.037510] ? console_flush_all+0x1ff/0x5a0 [17670.037512] console_unlock+0x7b/0x120 [17670.037515] vprintk_emit+0x246/0x340 [17670.037516] ? irq_work_queue+0x2c/0x50 [17670.037521] ? touch_softlockup_watchdog+0x20/0x20 [17670.037522] _printk+0x5c/0x80 [17670.037525] print_modules+0x6e/0xe0 [17670.037527] watchdog_timer_fn+0x19f/0x220 [17670.037529] __hrtimer_run_queues+0x1d3/0x360 [17670.037532] hrtimer_interrupt+0x104/0x240 [17670.037533] ? __local_bh_enable+0x50/0x60 [17670.037537] __sysvec_apic_timer_interrupt+0x61/0x160 [17670.037539] sysvec_apic_timer_interrupt+0x6d/0x90 [17670.037541] [17670.037541] [17670.037542] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17670.037544] RIP: 0010:_raw_spin_lock+0x18/0x40 [17670.037547] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [17670.037548] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [17670.037549] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [17670.037550] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [17670.037550] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [17670.037551] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17670.037552] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17670.037555] ? igrab+0x1d/0x50 [17670.037557] igrab+0x1d/0x50 [17670.037558] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17670.037573] delayed_work+0x1d8/0x280 [ceph] [17670.037587] ? process_scheduled_works+0x22a/0x510 [17670.037590] process_scheduled_works+0x2a4/0x510 [17670.037594] worker_thread+0x188/0x340 [17670.037597] ? rescuer_thread+0x310/0x310 [17670.037599] kthread+0xf5/0x130 [17670.037601] ? kthread_complete_and_exit+0x20/0x20 [17670.037604] ret_from_fork+0x31/0x40 [17670.037606] ? kthread_complete_and_exit+0x20/0x20 [17670.037608] ret_from_fork_asm+0x11/0x20 [17670.037612] [17670.037619] ixgbe igb [17670.043361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17670.047105] mdio crc32c_intel [17670.054230] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17670.060313] nvme ptp [17670.070643] Call Trace: [17670.070644] [17670.081062] nvme_core i2c_algo_bit [17670.088533] ? watchdog_timer_fn+0x1b4/0x220 [17670.092884] pps_core dca [17670.096368] ? __hrtimer_run_queues+0x1d3/0x360 [17670.105312] fuse [17670.112786] ? hrtimer_interrupt+0x104/0x240 [17670.117221] [17670.117222] irq event stamp: 64279732 [17670.135959] ? __local_bh_enable+0x50/0x60 [17670.141174] hardirqs last enabled at (64279731): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17670.148301] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17670.155428] hardirqs last disabled at (64279732): [] sysvec_apic_timer_interrupt+0xe/0x90 [17670.162556] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17670.169680] softirqs last enabled at (64279730): [] __do_softirq+0x2be/0x371 [17670.176805] [17670.184883] softirqs last disabled at (64279725): [] irq_exit_rcu+0xb0/0xd0 [17670.190619] [17670.190621] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17670.197743] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17670.204872] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17670.211991] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17670.214438] do_raw_spin_lock+0xb5/0xc0 [17670.216447] Workqueue: events delayed_work [ceph] [17670.220626] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17670.225407] [17670.225407] RIP: 0010:_raw_spin_lock+0x18/0x40 [17670.228998] try_get_cap_refs+0x159/0x880 [ceph] [17670.232827] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [17670.236228] __ceph_get_caps+0x4a1/0x760 [ceph] [17670.240056] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [17670.243889] ? __file_remove_privs+0x89/0x110 [17670.247722] [17670.247723] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [17670.251555] ? wake_bit_function+0x70/0x70 [17670.253654] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [17670.255666] ? ceph_write_iter+0x260/0x770 [ceph] [17670.258713] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [17670.262286] ceph_write_iter+0x260/0x770 [ceph] [17670.267070] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17670.270296] ? lock_is_held_type+0xaa/0x120 [17670.274383] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17670.278738] vfs_write+0x324/0x4f0 [17670.282827] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17670.286921] ksys_pwrite64+0x65/0xa0 [17670.291960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17670.296746] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17670.298841] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17670.300941] do_syscall_64+0x38/0x80 [17670.306069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17670.311549] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17670.330286] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17670.335502] RIP: 0033:0x7f48f0eeb93b [17670.342626] Call Trace: [17670.342627] [17670.349751] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17670.356876] ? watchdog_timer_fn+0x1b4/0x220 [17670.364000] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17670.371125] ? __hrtimer_run_queues+0x1d3/0x360 [17670.374954] ORIG_RAX: 0000000000000012 [17670.379740] ? hrtimer_interrupt+0x104/0x240 [17670.384348] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17670.388872] ? __local_bh_enable+0x50/0x60 [17670.393224] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17670.397318] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17670.402011] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17670.406537] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17670.410723] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17670.414120] [17670.417691] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17670.422821] [17670.426396] [17670.431436] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17671.287896] ? _raw_spin_lock+0x18/0x40 [17671.291748] ? igrab+0x1d/0x50 [17671.294800] igrab+0x1d/0x50 [17671.297677] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17671.302827] delayed_work+0x1d8/0x280 [ceph] [17671.307107] ? process_scheduled_works+0x22a/0x510 [17671.311890] process_scheduled_works+0x2a4/0x510 [17671.316506] worker_thread+0x188/0x340 [17671.320255] ? rescuer_thread+0x310/0x310 [17671.324259] kthread+0xf5/0x130 [17671.327394] ? kthread_complete_and_exit+0x20/0x20 [17671.332180] ret_from_fork+0x31/0x40 [17671.335758] ? kthread_complete_and_exit+0x20/0x20 [17671.340543] ret_from_fork_asm+0x11/0x20 [17671.344465] [17697.765552] watchdog: BUG: soft lockup - CPU#0 stuck for 5368s! [dbench:78683] [17697.772768] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17697.777552] watchdog: BUG: soft lockup - CPU#7 stuck for 14365s! [kworker/7:4:78808] [17697.828225] irq event stamp: 24834518 [17697.828226] hardirqs last enabled at (24834517): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17697.835957] Modules linked in: [17697.839615] hardirqs last disabled at (24834518): [] sysvec_apic_timer_interrupt+0xe/0x90 [17697.849772] ceph [17697.852823] softirqs last enabled at (24834464): [] __do_softirq+0x2be/0x371 [17697.862548] libceph dns_resolver [17697.864471] softirqs last disabled at (24834459): [] irq_exit_rcu+0xb0/0xd0 [17697.873156] fscache netfs [17697.876466] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17697.884975] veth nft_chain_nat [17697.887681] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17697.896625] xt_MASQUERADE nf_nat [17697.899763] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17697.907233] nf_conntrack [17697.910544] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17697.916020] nf_defrag_ipv6 nf_defrag_ipv4 [17697.918638] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17697.937375] nft_compat [17697.941468] [17697.941469] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17697.946685] nf_tables nfnetlink [17697.949129] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17697.950618] bridge [17697.957744] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17697.960967] stp llc [17697.968091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17697.970189] xfs libcrc32c [17697.977314] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17697.979497] sunrpc intel_rapl_msr [17697.986621] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17697.989325] intel_rapl_common sb_edac [17697.996449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17697.999845] x86_pkg_temp_thermal [17698.007923] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17698.011686] coretemp kvm_intel [17698.017423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17698.020734] kvm irqbypass [17698.027856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17698.030994] i2c_i801 [17698.038119] Call Trace: [17698.038120] [17698.040824] iTCO_wdt iTCO_vendor_support [17698.047950] ? watchdog_timer_fn+0x1b4/0x220 [17698.050230] i2c_smbus crct10dif_pclmul [17698.052673] ? __hrtimer_run_queues+0x1d3/0x360 [17698.054681] joydev crc32_pclmul [17698.058689] ? hrtimer_interrupt+0x104/0x240 [17698.062951] ipmi_ssif ghash_clmulni_intel [17698.066781] ? __local_bh_enable+0x50/0x60 [17698.071304] mei_me mei [17698.074530] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17698.078792] lpc_ich sha512_ssse3 [17698.082884] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17698.086974] acpi_ipmi [17698.089418] [17698.094635] mfd_core wmi [17698.097953] [17698.102912] ipmi_si [17698.105271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17698.107367] ipmi_devintf ipmi_msghandler [17698.109990] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17698.112082] ioatdma acpi_pad [17698.114267] do_raw_spin_lock+0xb5/0xc0 [17698.119575] acpi_power_meter ixgbe [17698.123585] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17698.128454] igb mdio [17698.131420] try_get_cap_refs+0x159/0x880 [ceph] [17698.135247] crc32c_intel nvme [17698.138736] __ceph_get_caps+0x4a1/0x760 [ceph] [17698.143516] ptp nvme_core [17698.145792] ? __file_remove_privs+0x89/0x110 [17698.150397] i2c_algo_bit pps_core [17698.153448] ? wake_bit_function+0x70/0x70 [17698.157974] dca fuse [17698.160679] ? ceph_write_iter+0x260/0x770 [ceph] [17698.165027] [17698.165028] irq event stamp: 64496796 [17698.168425] ceph_write_iter+0x260/0x770 [ceph] [17698.172515] hardirqs last enabled at (64496795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17698.174790] ? lock_is_held_type+0xaa/0x120 [17698.179483] hardirqs last disabled at (64496796): [] sysvec_apic_timer_interrupt+0xe/0x90 [17698.180976] vfs_write+0x324/0x4f0 [17698.184633] softirqs last enabled at (64496794): [] __do_softirq+0x2be/0x371 [17698.189161] ksys_pwrite64+0x65/0xa0 [17698.199314] softirqs last disabled at (64496789): [] irq_exit_rcu+0xb0/0xd0 [17698.203492] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17698.213232] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17698.216632] do_syscall_64+0x38/0x80 [17698.225314] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17698.228885] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17698.237396] Workqueue: events delayed_work [ceph] [17698.242526] RIP: 0033:0x7f48f0eeb93b [17698.251903] [17698.251904] RIP: 0010:lock_acquire+0xf0/0x2b0 [17698.255476] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17698.262954] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17698.267998] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17698.272697] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17698.276265] ORIG_RAX: 0000000000000012 [17698.277758] [17698.277759] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17698.282109] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17698.300845] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17698.319584] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17698.324801] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17698.330017] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17698.333849] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17698.335339] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17698.342463] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17698.349587] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17698.356712] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17698.363840] [17698.370959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17698.422580] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17698.429705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17698.436827] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17698.443956] Call Trace: [17698.446407] [17698.448417] ? watchdog_timer_fn+0x1b4/0x220 [17698.452681] ? __hrtimer_run_queues+0x1d3/0x360 [17698.457206] ? hrtimer_interrupt+0x104/0x240 [17698.461469] ? __local_bh_enable+0x50/0x60 [17698.465562] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17698.470787] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17698.475743] [17698.477839] [17698.479956] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17698.485262] ? lock_acquire+0xf0/0x2b0 [17698.489012] ? igrab+0x43/0x50 [17698.492062] ? lock_release+0xc5/0x260 [17698.495809] _raw_spin_lock+0x30/0x40 [17698.499473] ? igrab+0x1d/0x50 [17698.502523] igrab+0x1d/0x50 [17698.505402] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17698.510550] delayed_work+0x1d8/0x280 [ceph] [17698.514833] ? process_scheduled_works+0x22a/0x510 [17698.519622] process_scheduled_works+0x2a4/0x510 [17698.524243] worker_thread+0x188/0x340 [17698.527986] ? rescuer_thread+0x310/0x310 [17698.531991] kthread+0xf5/0x130 [17698.535127] ? kthread_complete_and_exit+0x20/0x20 [17698.539911] ret_from_fork+0x31/0x40 [17698.543482] ? kthread_complete_and_exit+0x20/0x20 [17698.548267] ret_from_fork_asm+0x11/0x20 [17698.552189] [17725.766076] watchdog: BUG: soft lockup - CPU#0 stuck for 5394s! [dbench:78683] [17725.773289] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17725.778075] watchdog: BUG: soft lockup - CPU#7 stuck for 14391s! [kworker/7:4:78808] [17725.828752] irq event stamp: 24892722 [17725.836488] Modules linked in: ceph libceph dns_resolver fscache netfs [17725.840144] hardirqs last enabled at (24892721): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17725.840149] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [17725.856827] hardirqs last disabled at (24892722): [] sysvec_apic_timer_interrupt+0xe/0x90 [17725.865513] nft_compat nf_tables nfnetlink bridge stp llc xfs [17725.875235] softirqs last enabled at (24892664): [] __do_softirq+0x2be/0x371 [17725.881060] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [17725.889750] softirqs last disabled at (24892659): [] irq_exit_rcu+0xb0/0xd0 [17725.898870] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [17725.907382] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17725.912512] iTCO_vendor_support i2c_smbus crct10dif_pclmul [17725.921471] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17725.927038] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [17725.934509] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17725.940334] mei_me mei lpc_ich sha512_ssse3 [17725.945811] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17725.950074] acpi_ipmi mfd_core wmi [17725.968811] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17725.972296] ipmi_si ipmi_devintf [17725.977513] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17725.977514] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17725.980825] ipmi_msghandler ioatdma [17725.987948] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17725.995072] acpi_pad acpi_power_meter [17725.998642] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17726.005766] ixgbe igb [17726.009512] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17726.016635] mdio crc32c_intel [17726.018993] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17726.026117] nvme ptp [17726.029168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17726.037245] nvme_core i2c_algo_bit [17726.039516] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17726.045253] pps_core dca [17726.048737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17726.055880] fuse [17726.058497] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17726.065620] [17726.065621] irq event stamp: 64718794 [17726.067543] Call Trace: [17726.067546] [17726.074668] hardirqs last enabled at (64718793): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17726.076162] ? watchdog_timer_fn+0x1b4/0x220 [17726.079816] hardirqs last disabled at (64718794): [] sysvec_apic_timer_interrupt+0xe/0x90 [17726.082262] ? __hrtimer_run_queues+0x1d3/0x360 [17726.084272] softirqs last enabled at (64718792): [] __do_softirq+0x2be/0x371 [17726.094433] ? hrtimer_interrupt+0x104/0x240 [17726.098693] softirqs last disabled at (64718787): [] irq_exit_rcu+0xb0/0xd0 [17726.108418] ? __local_bh_enable+0x50/0x60 [17726.112941] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17726.121627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17726.125889] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17726.125890] Workqueue: events delayed_work [ceph] [17726.134401] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17726.138491] [17726.138492] RIP: 0010:lock_acquire+0xf0/0x2b0 [17726.147868] [17726.153087] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17726.160557] [17726.160558] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17726.165253] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17726.170214] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17726.171703] [17726.171704] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17726.176055] do_raw_spin_lock+0xb5/0xc0 [17726.178149] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17726.196889] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17726.198986] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17726.204294] try_get_cap_refs+0x159/0x880 [ceph] [17726.209506] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17726.214384] __ceph_get_caps+0x4a1/0x760 [ceph] [17726.215868] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17726.222997] ? __file_remove_privs+0x89/0x110 [17726.226823] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17726.233949] ? wake_bit_function+0x70/0x70 [17726.238731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17726.245859] ? ceph_write_iter+0x260/0x770 [ceph] [17726.250466] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17726.257591] ceph_write_iter+0x260/0x770 [ceph] [17726.262115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17726.269243] ? lock_is_held_type+0xaa/0x120 [17726.273589] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17726.281669] vfs_write+0x324/0x4f0 [17726.285766] Call Trace: [17726.285768] [17726.291509] ksys_pwrite64+0x65/0xa0 [17726.296202] ? watchdog_timer_fn+0x1b4/0x220 [17726.303326] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17726.307850] ? __hrtimer_run_queues+0x1d3/0x360 [17726.314975] do_syscall_64+0x38/0x80 [17726.319156] ? hrtimer_interrupt+0x104/0x240 [17726.326277] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17726.329673] ? __local_bh_enable+0x50/0x60 [17726.332118] RIP: 0033:0x7f48f0eeb93b [17726.334129] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17726.337700] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17726.341962] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17726.347093] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17726.351618] [17726.355187] ORIG_RAX: 0000000000000012 [17726.359452] [17726.359455] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17726.364496] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17726.368592] ? lock_acquire+0xf0/0x2b0 [17726.372158] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17726.377376] ? igrab+0x43/0x50 [17726.396114] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17726.401071] ? lock_release+0xc5/0x260 [17726.406288] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17726.408388] _raw_spin_lock+0x30/0x40 [17726.412215] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17726.414313] ? igrab+0x1d/0x50 [17726.419622] [17726.426756] igrab+0x1d/0x50 [17726.477567] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17726.482716] delayed_work+0x1d8/0x280 [ceph] [17726.486995] ? process_scheduled_works+0x22a/0x510 [17726.491786] process_scheduled_works+0x2a4/0x510 [17726.496402] worker_thread+0x188/0x340 [17726.500152] ? rescuer_thread+0x310/0x310 [17726.504163] kthread+0xf5/0x130 [17726.507310] ? kthread_complete_and_exit+0x20/0x20 [17726.512094] ret_from_fork+0x31/0x40 [17726.515674] ? kthread_complete_and_exit+0x20/0x20 [17726.520482] ret_from_fork_asm+0x11/0x20 [17726.524406] [17753.766598] watchdog: BUG: soft lockup - CPU#0 stuck for 5420s! [dbench:78683] [17753.773810] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17753.778598] watchdog: BUG: soft lockup - CPU#7 stuck for 14417s! [kworker/7:4:78808] [17753.829251] irq event stamp: 24950892 [17753.837000] Modules linked in: ceph libceph dns_resolver [17753.840657] hardirqs last enabled at (24950891): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17753.845960] fscache netfs veth [17753.856119] hardirqs last disabled at (24950892): [] sysvec_apic_timer_interrupt+0xe/0x90 [17753.859257] nft_chain_nat xt_MASQUERADE nf_nat [17753.868997] softirqs last enabled at (24950834): [] __do_softirq+0x2be/0x371 [17753.868998] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [17753.882206] softirqs last disabled at (24950829): [] irq_exit_rcu+0xb0/0xd0 [17753.882208] nf_tables nfnetlink bridge stp [17753.896887] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17753.901066] llc xfs [17753.910008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17753.912194] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [17753.919663] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [17753.925488] sb_edac x86_pkg_temp_thermal [17753.930965] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [17753.934992] coretemp kvm_intel [17753.953733] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17753.956872] kvm irqbypass [17753.962088] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17753.962090] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17753.964793] i2c_i801 iTCO_wdt [17753.971915] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17753.979041] iTCO_vendor_support [17753.982092] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17753.989214] i2c_smbus crct10dif_pclmul [17753.992440] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17753.999563] joydev [17754.003394] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17754.010517] crc32_pclmul ipmi_ssif [17754.012615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17754.020693] ghash_clmulni_intel [17754.024178] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17754.029915] mei_me mei [17754.033138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17754.040276] lpc_ich [17754.042724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17754.049849] sha512_ssse3 acpi_ipmi [17754.052033] Call Trace: [17754.052034] [17754.059156] mfd_core wmi [17754.062642] ? watchdog_timer_fn+0x1b4/0x220 [17754.065084] ipmi_si ipmi_devintf [17754.067095] ? __hrtimer_run_queues+0x1d3/0x360 [17754.069713] ipmi_msghandler ioatdma acpi_pad [17754.073998] ? hrtimer_interrupt+0x104/0x240 [17754.077304] acpi_power_meter ixgbe [17754.081829] ? __local_bh_enable+0x50/0x60 [17754.086179] igb [17754.090446] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17754.093928] mdio crc32c_intel [17754.098018] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17754.099855] nvme ptp nvme_core [17754.105073] [17754.108125] i2c_algo_bit pps_core [17754.113083] [17754.113084] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17754.116219] dca fuse [17754.118319] ? queued_spin_lock_slowpath+0x2d/0x2e0 [17754.121713] [17754.121714] irq event stamp: 64941248 [17754.123814] do_raw_spin_lock+0xb5/0xc0 [17754.129116] hardirqs last enabled at (64941247): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17754.131386] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17754.136257] hardirqs last disabled at (64941248): [] sysvec_apic_timer_interrupt+0xe/0x90 [17754.137750] try_get_cap_refs+0x159/0x880 [ceph] [17754.141404] softirqs last enabled at (64941246): [] __do_softirq+0x2be/0x371 [17754.145239] __ceph_get_caps+0x4a1/0x760 [ceph] [17754.155392] softirqs last disabled at (64941241): [] irq_exit_rcu+0xb0/0xd0 [17754.160183] ? __file_remove_privs+0x89/0x110 [17754.169900] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17754.174513] ? wake_bit_function+0x70/0x70 [17754.183197] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17754.187723] ? ceph_write_iter+0x260/0x770 [ceph] [17754.196231] Workqueue: events delayed_work [ceph] [17754.200583] ceph_write_iter+0x260/0x770 [ceph] [17754.209960] [17754.209961] RIP: 0010:lock_acquire+0xf0/0x2b0 [17754.214054] ? lock_is_held_type+0xaa/0x120 [17754.221522] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17754.226220] vfs_write+0x324/0x4f0 [17754.230915] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17754.235444] ksys_pwrite64+0x65/0xa0 [17754.236930] [17754.236931] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17754.241283] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17754.245459] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17754.264198] do_syscall_64+0x38/0x80 [17754.267594] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17754.272813] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17754.276381] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17754.277872] RIP: 0033:0x7f48f0eeb93b [17754.284999] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17754.290128] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17754.297252] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17754.300823] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17754.307948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17754.313000] ORIG_RAX: 0000000000000012 [17754.313001] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17754.320122] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17754.323695] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17754.330819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17754.349555] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17754.357634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17754.362850] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17754.368588] Call Trace: [17754.372418] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17754.379544] [17754.386674] [17754.393793] ? watchdog_timer_fn+0x1b4/0x220 [17754.440316] ? __hrtimer_run_queues+0x1d3/0x360 [17754.444842] ? hrtimer_interrupt+0x104/0x240 [17754.449104] ? __local_bh_enable+0x50/0x60 [17754.453198] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17754.458421] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17754.463377] [17754.465474] [17754.467573] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17754.472882] ? lock_acquire+0xf0/0x2b0 [17754.476631] ? igrab+0x43/0x50 [17754.479680] ? lock_release+0xc5/0x260 [17754.483426] _raw_spin_lock+0x30/0x40 [17754.487080] ? igrab+0x1d/0x50 [17754.490133] igrab+0x1d/0x50 [17754.493010] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17754.498161] delayed_work+0x1d8/0x280 [ceph] [17754.502442] ? process_scheduled_works+0x22a/0x510 [17754.507233] process_scheduled_works+0x2a4/0x510 [17754.511847] worker_thread+0x188/0x340 [17754.515595] ? rescuer_thread+0x310/0x310 [17754.519600] kthread+0xf5/0x130 [17754.522737] ? kthread_complete_and_exit+0x20/0x20 [17754.527521] ret_from_fork+0x31/0x40 [17754.531092] ? kthread_complete_and_exit+0x20/0x20 [17754.535876] ret_from_fork_asm+0x11/0x20 [17754.539798] [17781.767122] watchdog: BUG: soft lockup - CPU#0 stuck for 5446s! [dbench:78683] [17781.774332] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17781.779121] watchdog: BUG: soft lockup - CPU#7 stuck for 14443s! [kworker/7:4:78808] [17781.829772] irq event stamp: 25009158 [17781.837521] Modules linked in: ceph libceph [17781.841178] hardirqs last enabled at (25009157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17781.845357] dns_resolver fscache [17781.855516] hardirqs last disabled at (25009158): [] sysvec_apic_timer_interrupt+0xe/0x90 [17781.858825] netfs veth nft_chain_nat xt_MASQUERADE [17781.868550] softirqs last enabled at (25009112): [] __do_softirq+0x2be/0x371 [17781.873419] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [17781.882105] softirqs last disabled at (25009107): [] irq_exit_rcu+0xb0/0xd0 [17781.888881] nf_tables nfnetlink bridge [17781.897393] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17781.901222] stp llc [17781.910167] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17781.910169] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [17781.912351] xfs libcrc32c [17781.919823] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [17781.925298] sunrpc [17781.928002] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17781.946740] intel_rapl_msr intel_rapl_common [17781.948838] [17781.948839] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17781.954056] sb_edac x86_pkg_temp_thermal [17781.958407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17781.959897] coretemp kvm_intel [17781.967022] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17781.971024] kvm irqbypass [17781.978150] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17781.981286] i2c_i801 iTCO_wdt [17781.988411] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17781.991115] iTCO_vendor_support i2c_smbus [17781.998239] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17782.001290] crct10dif_pclmul joydev [17782.008415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17782.012519] crc32_pclmul [17782.020598] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17782.024171] ipmi_ssif ghash_clmulni_intel [17782.029908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17782.032526] mei_me [17782.039649] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17782.043739] mei lpc_ich [17782.050864] Call Trace: [17782.050865] [17782.052960] sha512_ssse3 acpi_ipmi [17782.060087] ? watchdog_timer_fn+0x1b4/0x220 [17782.062616] mfd_core wmi [17782.065062] ? __hrtimer_run_queues+0x1d3/0x360 [17782.067070] ipmi_si [17782.070559] ? hrtimer_interrupt+0x104/0x240 [17782.074818] ipmi_devintf [17782.077437] ? __local_bh_enable+0x50/0x60 [17782.081962] ipmi_msghandler ioatdma [17782.084147] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17782.088409] acpi_pad acpi_power_meter [17782.091028] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17782.095117] ixgbe igb [17782.098688] [17782.103904] mdio crc32c_intel [17782.107649] [17782.107650] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17782.112605] nvme ptp [17782.114969] ? queued_spin_lock_slowpath+0x2b/0x2e0 [17782.117062] nvme_core i2c_algo_bit [17782.120114] do_raw_spin_lock+0xb5/0xc0 [17782.122210] pps_core dca fuse [17782.127521] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17782.129802] irq event stamp: 65163456 [17782.134674] try_get_cap_refs+0x159/0x880 [ceph] [17782.138157] hardirqs last enabled at (65163455): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17782.141991] __ceph_get_caps+0x4a1/0x760 [ceph] [17782.145037] hardirqs last disabled at (65163456): [] sysvec_apic_timer_interrupt+0xe/0x90 [17782.149827] ? __file_remove_privs+0x89/0x110 [17782.153479] softirqs last enabled at (65163454): [] __do_softirq+0x2be/0x371 [17782.158090] ? wake_bit_function+0x70/0x70 [17782.168246] softirqs last disabled at (65163449): [] irq_exit_rcu+0xb0/0xd0 [17782.172775] ? ceph_write_iter+0x260/0x770 [ceph] [17782.182496] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17782.186865] ceph_write_iter+0x260/0x770 [ceph] [17782.195549] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17782.199642] ? lock_is_held_type+0xaa/0x120 [17782.208150] Workqueue: events delayed_work [ceph] [17782.212850] vfs_write+0x324/0x4f0 [17782.222225] [17782.222226] RIP: 0010:lock_acquire+0xf0/0x2b0 [17782.226754] ksys_pwrite64+0x65/0xa0 [17782.234219] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17782.238398] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17782.243095] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17782.246493] do_syscall_64+0x38/0x80 [17782.247981] [17782.247982] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17782.252334] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17782.255905] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17782.274642] RIP: 0033:0x7f48f0eeb93b [17782.279773] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17782.284990] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17782.288560] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17782.290052] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17782.297175] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17782.302219] ORIG_RAX: 0000000000000012 [17782.309345] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17782.312915] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17782.320038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17782.338776] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17782.345900] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17782.351117] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17782.358242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17782.362073] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17782.370151] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17782.377275] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17782.383012] Call Trace: [17782.383013] [17782.390140] [17782.397261] ? watchdog_timer_fn+0x1b4/0x220 [17782.443819] ? __hrtimer_run_queues+0x1d3/0x360 [17782.448345] ? hrtimer_interrupt+0x104/0x240 [17782.452607] ? __local_bh_enable+0x50/0x60 [17782.456699] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17782.461915] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17782.466872] [17782.468969] [17782.471068] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17782.476375] ? lock_acquire+0xf0/0x2b0 [17782.480126] ? igrab+0x43/0x50 [17782.483176] ? lock_release+0xc5/0x260 [17782.486919] _raw_spin_lock+0x30/0x40 [17782.490577] ? igrab+0x1d/0x50 [17782.493627] igrab+0x1d/0x50 [17782.496524] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17782.501673] delayed_work+0x1d8/0x280 [ceph] [17782.505954] ? process_scheduled_works+0x22a/0x510 [17782.510745] process_scheduled_works+0x2a4/0x510 [17782.515358] worker_thread+0x188/0x340 [17782.519109] ? rescuer_thread+0x310/0x310 [17782.523120] kthread+0xf5/0x130 [17782.526257] ? kthread_complete_and_exit+0x20/0x20 [17782.531042] ret_from_fork+0x31/0x40 [17782.534613] ? kthread_complete_and_exit+0x20/0x20 [17782.539397] ret_from_fork_asm+0x11/0x20 [17782.543320] [17809.767640] watchdog: BUG: soft lockup - CPU#0 stuck for 5472s! [dbench:78683] [17809.774855] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17809.779640] watchdog: BUG: soft lockup - CPU#7 stuck for 14469s! [kworker/7:4:78808] [17809.830314] irq event stamp: 25067372 [17809.830316] hardirqs last enabled at (25067371): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17809.838043] Modules linked in: [17809.841700] hardirqs last disabled at (25067372): [] sysvec_apic_timer_interrupt+0xe/0x90 [17809.851859] ceph [17809.854910] softirqs last enabled at (25067336): [] __do_softirq+0x2be/0x371 [17809.864633] libceph [17809.866558] softirqs last disabled at (25067331): [] irq_exit_rcu+0xb0/0xd0 [17809.875243] dns_resolver fscache [17809.877426] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17809.885936] netfs veth [17809.889249] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17809.898192] nft_chain_nat [17809.900634] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17809.908107] xt_MASQUERADE nf_nat [17809.910810] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17809.916288] nf_conntrack nf_defrag_ipv6 [17809.919598] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17809.938337] nf_defrag_ipv4 nft_compat [17809.942255] [17809.947470] nf_tables nfnetlink [17809.951216] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17809.952704] bridge [17809.955930] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17809.963053] stp llc [17809.965151] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17809.972276] xfs libcrc32c [17809.974458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17809.981583] sunrpc intel_rapl_msr [17809.984289] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17809.991411] intel_rapl_common sb_edac [17809.994809] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17810.001932] x86_pkg_temp_thermal coretemp [17810.005677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17810.013756] kvm_intel kvm [17810.017844] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17810.023582] irqbypass [17810.026287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17810.033410] i2c_i801 iTCO_wdt [17810.035768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17810.042894] iTCO_vendor_support i2c_smbus [17810.045943] Call Trace: [17810.045946] [17810.053067] crct10dif_pclmul joydev [17810.057160] ? watchdog_timer_fn+0x1b4/0x220 [17810.059604] crc32_pclmul ipmi_ssif [17810.061614] ? __hrtimer_run_queues+0x1d3/0x360 [17810.065185] ghash_clmulni_intel mei_me [17810.069452] ? hrtimer_interrupt+0x104/0x240 [17810.072931] mei lpc_ich [17810.077456] ? __local_bh_enable+0x50/0x60 [17810.081286] sha512_ssse3 [17810.085553] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17810.088083] acpi_ipmi mfd_core [17810.092173] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17810.094789] wmi ipmi_si [17810.100007] [17810.103145] ipmi_devintf ipmi_msghandler [17810.108101] [17810.110633] ioatdma [17810.112731] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17810.116735] acpi_pad acpi_power_meter [17810.118836] ? queued_spin_lock_slowpath+0x18/0x2e0 [17810.121017] ixgbe igb [17810.126338] do_raw_spin_lock+0xb5/0xc0 [17810.130081] mdio crc32c_intel [17810.134953] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17810.137321] nvme ptp [17810.141151] try_get_cap_refs+0x159/0x880 [ceph] [17810.144200] nvme_core i2c_algo_bit [17810.148989] __ceph_get_caps+0x4a1/0x760 [ceph] [17810.151253] pps_core dca [17810.155869] ? __file_remove_privs+0x89/0x110 [17810.159348] fuse [17810.163873] ? wake_bit_function+0x70/0x70 [17810.166490] [17810.166491] irq event stamp: 65386332 [17810.170843] ? ceph_write_iter+0x260/0x770 [ceph] [17810.172765] hardirqs last enabled at (65386331): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17810.176857] ceph_write_iter+0x260/0x770 [ceph] [17810.178347] hardirqs last disabled at (65386332): [] sysvec_apic_timer_interrupt+0xe/0x90 [17810.182010] ? lock_is_held_type+0xaa/0x120 [17810.186701] softirqs last enabled at (65386330): [] __do_softirq+0x2be/0x371 [17810.196861] vfs_write+0x324/0x4f0 [17810.201383] softirqs last disabled at (65386325): [] irq_exit_rcu+0xb0/0xd0 [17810.211112] ksys_pwrite64+0x65/0xa0 [17810.215284] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17810.223971] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17810.227367] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17810.235878] do_syscall_64+0x38/0x80 [17810.239447] Workqueue: events delayed_work [ceph] [17810.248825] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17810.253956] [17810.253957] RIP: 0010:_raw_spin_lock+0x0/0x40 [17810.261426] RIP: 0033:0x7f48f0eeb93b [17810.264996] Code: 48 c7 c7 7c ca 15 82 e8 4e a8 57 ff e8 69 5d ff ff 48 89 ea 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 [17810.269696] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17810.274739] RSP: 0018:ffffc90002117d98 EFLAGS: 00000286 [17810.276229] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17810.280581] [17810.280582] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [17810.284151] ORIG_RAX: 0000000000000012 [17810.302888] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [17810.321627] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17810.326845] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [17810.332062] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17810.333553] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17810.340676] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17810.344507] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17810.351630] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17810.358755] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17810.365879] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17810.373005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17810.380134] [17810.387251] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17810.431748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17810.438872] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17810.445996] Call Trace: [17810.448441] [17810.450451] ? watchdog_timer_fn+0x1b4/0x220 [17810.454715] ? __hrtimer_run_queues+0x1d3/0x360 [17810.459242] ? hrtimer_interrupt+0x104/0x240 [17810.463510] ? __local_bh_enable+0x50/0x60 [17810.467604] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17810.472828] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17810.477786] [17810.479883] [17810.481981] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17810.487288] ? __sched_text_end+0xa/0xa [17810.491126] igrab+0x1d/0x50 [17810.494011] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17810.499180] delayed_work+0x1d8/0x280 [ceph] [17810.503461] ? process_scheduled_works+0x22a/0x510 [17810.508249] process_scheduled_works+0x2a4/0x510 [17810.512866] worker_thread+0x188/0x340 [17810.516616] ? rescuer_thread+0x310/0x310 [17810.520626] kthread+0xf5/0x130 [17810.523763] ? kthread_complete_and_exit+0x20/0x20 [17810.528549] ret_from_fork+0x31/0x40 [17810.532127] ? kthread_complete_and_exit+0x20/0x20 [17810.536910] ret_from_fork_asm+0x11/0x20 [17810.540834] [17837.768157] watchdog: BUG: soft lockup - CPU#0 stuck for 5498s! [dbench:78683] [17837.775368] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17837.780157] watchdog: BUG: soft lockup - CPU#7 stuck for 14495s! [kworker/7:4:78808] [17837.830808] irq event stamp: 25125566 [17837.830809] hardirqs last enabled at (25125565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17837.838557] Modules linked in: [17837.842214] hardirqs last disabled at (25125566): [] sysvec_apic_timer_interrupt+0xe/0x90 [17837.852373] ceph [17837.855421] softirqs last enabled at (25125510): [] __do_softirq+0x2be/0x371 [17837.865147] libceph [17837.867070] softirqs last disabled at (25125505): [] irq_exit_rcu+0xb0/0xd0 [17837.875755] dns_resolver fscache [17837.877938] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17837.886451] netfs veth [17837.889762] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17837.898704] nft_chain_nat xt_MASQUERADE [17837.901148] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17837.908619] nf_nat nf_conntrack [17837.912552] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17837.918031] nf_defrag_ipv6 nf_defrag_ipv4 [17837.921256] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17837.939993] nft_compat nf_tables [17837.944085] [17837.944085] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17837.949302] nfnetlink [17837.952613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17837.954104] bridge stp [17837.961227] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17837.963586] llc xfs [17837.970710] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17837.973152] libcrc32c sunrpc [17837.980277] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17837.982461] intel_rapl_msr intel_rapl_common [17837.989585] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17837.992554] sb_edac x86_pkg_temp_thermal [17837.999682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17838.004034] coretemp kvm_intel [17838.012110] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17838.016114] kvm irqbypass [17838.021853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17838.024990] i2c_i801 [17838.032113] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17838.034818] iTCO_wdt iTCO_vendor_support [17838.041942] Call Trace: [17838.041943] [17838.044212] i2c_smbus crct10dif_pclmul [17838.051338] ? watchdog_timer_fn+0x1b4/0x220 [17838.055341] joydev crc32_pclmul [17838.057785] ? __hrtimer_run_queues+0x1d3/0x360 [17838.059796] ipmi_ssif ghash_clmulni_intel [17838.063630] ? hrtimer_interrupt+0x104/0x240 [17838.067890] mei_me mei [17838.071115] ? __local_bh_enable+0x50/0x60 [17838.075637] lpc_ich sha512_ssse3 [17838.079730] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17838.083992] acpi_ipmi mfd_core [17838.086437] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17838.090527] wmi [17838.093856] [17838.099073] ipmi_si ipmi_devintf [17838.102210] [17838.107168] ipmi_msghandler ioatdma [17838.109006] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17838.111102] acpi_pad acpi_power_meter [17838.114418] ? queued_spin_lock_slowpath+0x16/0x2e0 [17838.116512] ixgbe igb [17838.120085] do_raw_spin_lock+0xb5/0xc0 [17838.125385] mdio crc32c_intel [17838.129132] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17838.134000] nvme ptp [17838.136360] try_get_cap_refs+0x159/0x880 [ceph] [17838.140187] nvme_core i2c_algo_bit [17838.143244] __ceph_get_caps+0x4a1/0x760 [ceph] [17838.148025] pps_core dca [17838.150299] ? __file_remove_privs+0x89/0x110 [17838.154904] fuse [17838.158391] ? wake_bit_function+0x70/0x70 [17838.162913] irq event stamp: 65609192 [17838.165534] ? ceph_write_iter+0x260/0x770 [ceph] [17838.169882] hardirqs last enabled at (65609191): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17838.171805] ceph_write_iter+0x260/0x770 [ceph] [17838.175895] hardirqs last disabled at (65609192): [] sysvec_apic_timer_interrupt+0xe/0x90 [17838.179561] ? lock_is_held_type+0xaa/0x120 [17838.184250] softirqs last enabled at (65609190): [] __do_softirq+0x2be/0x371 [17838.194412] vfs_write+0x324/0x4f0 [17838.198932] softirqs last disabled at (65609185): [] irq_exit_rcu+0xb0/0xd0 [17838.208662] ksys_pwrite64+0x65/0xa0 [17838.212837] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17838.221517] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17838.224915] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17838.233427] do_syscall_64+0x38/0x80 [17838.236997] Workqueue: events delayed_work [ceph] [17838.246375] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17838.251506] [17838.251507] RIP: 0010:lock_release+0x78/0x260 [17838.258976] RIP: 0033:0x7f48f0eeb93b [17838.262557] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [17838.267253] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17838.272297] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [17838.273787] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17838.278139] [17838.278139] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [17838.281709] ORIG_RAX: 0000000000000012 [17838.300447] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [17838.319185] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17838.324402] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [17838.329620] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17838.331109] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [17838.338235] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17838.342064] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17838.349188] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17838.356313] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17838.363437] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17838.370561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17838.377691] [17838.384809] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17838.429305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17838.436429] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17838.443562] Call Trace: [17838.446007] [17838.448018] ? watchdog_timer_fn+0x1b4/0x220 [17838.452284] ? __hrtimer_run_queues+0x1d3/0x360 [17838.456810] ? hrtimer_interrupt+0x104/0x240 [17838.461078] ? __local_bh_enable+0x50/0x60 [17838.465172] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17838.470395] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17838.475354] [17838.477451] [17838.479564] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17838.484869] ? igrab+0x43/0x50 [17838.487922] ? lock_release+0x78/0x260 [17838.491666] _raw_spin_unlock+0x1b/0x40 [17838.495504] igrab+0x43/0x50 [17838.498380] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17838.503532] delayed_work+0x1d8/0x280 [ceph] [17838.507829] ? process_scheduled_works+0x22a/0x510 [17838.512629] process_scheduled_works+0x2a4/0x510 [17838.517243] worker_thread+0x188/0x340 [17838.520993] ? rescuer_thread+0x310/0x310 [17838.524996] kthread+0xf5/0x130 [17838.528135] ? kthread_complete_and_exit+0x20/0x20 [17838.532918] ret_from_fork+0x31/0x40 [17838.536489] ? kthread_complete_and_exit+0x20/0x20 [17838.541273] ret_from_fork_asm+0x11/0x20 [17838.545196] [17865.043026] rcu: INFO: rcu_preempt self-detected stall on CPU [17865.048787] rcu: 7-....: (5758936 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/5757705 fqs=2879889 [17865.059213] rcu: (t=5928327 jiffies g=1309609 q=237808 ncpus=8) [17865.065210] Sending NMI from CPU 7 to CPUs 0: [17865.069563] NMI backtrace for cpu 0 [17865.069565] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17865.069567] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17865.069568] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17865.069574] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17865.069575] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17865.069577] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17865.069578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17865.069579] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17865.069580] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17865.069580] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17865.069581] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17865.069582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17865.069583] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17865.069584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17865.069585] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17865.069585] Call Trace: [17865.069587] [17865.069589] ? nmi_cpu_backtrace+0xad/0x120 [17865.069592] ? nmi_cpu_backtrace_handler+0x11/0x20 [17865.069594] ? nmi_handle+0xd4/0x260 [17865.069598] ? default_do_nmi+0x49/0xf0 [17865.069600] ? exc_nmi+0x11e/0x150 [17865.069602] ? end_repeat_nmi+0x16/0x67 [17865.069606] ? queued_spin_lock_slowpath+0x18/0x2e0 [17865.069609] ? queued_spin_lock_slowpath+0x18/0x2e0 [17865.069611] ? queued_spin_lock_slowpath+0x18/0x2e0 [17865.069614] [17865.069614] [17865.069615] do_raw_spin_lock+0xb5/0xc0 [17865.069618] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17865.069640] try_get_cap_refs+0x159/0x880 [ceph] [17865.069654] __ceph_get_caps+0x4a1/0x760 [ceph] [17865.069667] ? __file_remove_privs+0x89/0x110 [17865.069670] ? wake_bit_function+0x70/0x70 [17865.069672] ? ceph_write_iter+0x260/0x770 [ceph] [17865.069683] ceph_write_iter+0x260/0x770 [ceph] [17865.069694] ? lock_is_held_type+0xaa/0x120 [17865.069697] vfs_write+0x324/0x4f0 [17865.069702] ksys_pwrite64+0x65/0xa0 [17865.069704] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17865.069707] do_syscall_64+0x38/0x80 [17865.069709] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17865.069711] RIP: 0033:0x7f48f0eeb93b [17865.069713] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17865.069714] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [17865.069716] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17865.069716] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17865.069717] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17865.069718] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17865.069718] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17865.069722] [17865.070561] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17865.374373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17865.381845] Workqueue: events delayed_work [ceph] [17865.386559] RIP: 0010:lock_acquire+0xf0/0x2b0 [17865.390919] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17865.409656] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17865.414874] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17865.421997] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17865.429122] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17865.436247] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17865.443369] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17865.450494] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17865.458572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17865.464308] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17865.471433] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17865.478558] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17865.485681] Call Trace: [17865.488128] [17865.490138] ? rcu_dump_cpu_stacks+0xd1/0x180 [17865.494497] ? rcu_sched_clock_irq+0x633/0x1390 [17865.499021] ? lock_release+0xc5/0x260 [17865.502764] ? lock_is_held_type+0xaa/0x120 [17865.506941] ? lock_is_held_type+0xaa/0x120 [17865.511119] ? tick_sched_do_timer+0xa0/0xa0 [17865.515392] ? rcu_read_lock_sched_held+0x56/0x90 [17865.520089] ? tick_sched_do_timer+0xa0/0xa0 [17865.524353] ? update_process_times+0x5f/0x90 [17865.528703] ? tick_sched_handle.isra.16+0x1e/0x50 [17865.533488] ? tick_sched_timer+0x6f/0x80 [17865.537491] ? __hrtimer_run_queues+0x1d3/0x360 [17865.542019] ? hrtimer_interrupt+0x104/0x240 [17865.546289] ? __local_bh_enable+0x50/0x60 [17865.550382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17865.555605] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17865.560562] [17865.562660] [17865.564758] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17865.570066] ? lock_acquire+0xf0/0x2b0 [17865.573816] ? igrab+0x43/0x50 [17865.576866] ? lock_release+0xc5/0x260 [17865.580610] _raw_spin_lock+0x30/0x40 [17865.584266] ? igrab+0x1d/0x50 [17865.587317] igrab+0x1d/0x50 [17865.590196] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17865.595345] delayed_work+0x1d8/0x280 [ceph] [17865.599627] ? process_scheduled_works+0x22a/0x510 [17865.604417] process_scheduled_works+0x2a4/0x510 [17865.609034] worker_thread+0x188/0x340 [17865.612781] ? rescuer_thread+0x310/0x310 [17865.616784] kthread+0xf5/0x130 [17865.619923] ? kthread_complete_and_exit+0x20/0x20 [17865.624707] ret_from_fork+0x31/0x40 [17865.628285] ? kthread_complete_and_exit+0x20/0x20 [17865.633068] ret_from_fork_asm+0x11/0x20 [17865.636993] [17889.769115] watchdog: BUG: soft lockup - CPU#0 stuck for 5547s! [dbench:78683] [17889.776335] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17889.781115] watchdog: BUG: soft lockup - CPU#7 stuck for 14544s! [kworker/7:4:78808] [17889.831790] irq event stamp: 25234672 [17889.839524] Modules linked in: ceph libceph dns_resolver [17889.843181] hardirqs last enabled at (25234671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17889.843183] fscache netfs veth nft_chain_nat xt_MASQUERADE [17889.858660] hardirqs last disabled at (25234672): [] sysvec_apic_timer_interrupt+0xe/0x90 [17889.864223] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [17889.873948] softirqs last enabled at (25234648): [] __do_softirq+0x2be/0x371 [17889.879787] nft_compat nf_tables nfnetlink bridge stp [17889.888473] softirqs last disabled at (25234643): [] irq_exit_rcu+0xb0/0xd0 [17889.888476] llc xfs libcrc32c sunrpc [17889.902114] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17889.902116] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17889.905771] intel_rapl_msr intel_rapl_common [17889.914716] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17889.922186] sb_edac x86_pkg_temp_thermal [17889.926537] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17889.932015] coretemp kvm_intel [17889.936019] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17889.954757] kvm irqbypass [17889.957893] [17889.957894] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17889.963110] i2c_i801 iTCO_wdt [17889.965815] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17889.967306] iTCO_vendor_support [17889.974431] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17889.977483] i2c_smbus crct10dif_pclmul [17889.984604] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17889.987828] joydev [17889.994953] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17889.998794] crc32_pclmul ipmi_ssif [17890.005917] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17890.008015] ghash_clmulni_intel [17890.015137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17890.018623] mei_me mei [17890.026700] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17890.029926] lpc_ich [17890.035663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17890.038107] sha512_ssse3 [17890.045230] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17890.047414] acpi_ipmi [17890.054538] Call Trace: [17890.054540] [17890.057154] mfd_core wmi [17890.064280] ? watchdog_timer_fn+0x1b4/0x220 [17890.066637] ipmi_si ipmi_devintf [17890.069081] ? __hrtimer_run_queues+0x1d3/0x360 [17890.071092] ipmi_msghandler ioatdma [17890.073714] ? hrtimer_interrupt+0x104/0x240 [17890.077973] acpi_pad acpi_power_meter [17890.081285] ? __local_bh_enable+0x50/0x60 [17890.085808] ixgbe igb [17890.089382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17890.093643] mdio crc32c_intel [17890.097386] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17890.101479] nvme ptp [17890.103837] [17890.109052] nvme_core i2c_algo_bit [17890.112103] [17890.117061] pps_core dca [17890.119333] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17890.121429] fuse [17890.124916] ? queued_spin_lock_slowpath+0x16/0x2e0 [17890.127010] irq event stamp: 66022292 [17890.129630] do_raw_spin_lock+0xb5/0xc0 [17890.134931] hardirqs last enabled at (66022291): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17890.136856] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17890.141727] hardirqs last disabled at (66022292): [] sysvec_apic_timer_interrupt+0xe/0x90 [17890.145385] try_get_cap_refs+0x159/0x880 [ceph] [17890.149213] softirqs last enabled at (66022290): [] __do_softirq+0x2be/0x371 [17890.159378] __ceph_get_caps+0x4a1/0x760 [ceph] [17890.164156] softirqs last disabled at (66022285): [] irq_exit_rcu+0xb0/0xd0 [17890.173884] ? __file_remove_privs+0x89/0x110 [17890.178492] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17890.187194] ? wake_bit_function+0x70/0x70 [17890.191717] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17890.200231] ? ceph_write_iter+0x260/0x770 [ceph] [17890.204578] Workqueue: events delayed_work [ceph] [17890.213958] ceph_write_iter+0x260/0x770 [ceph] [17890.218046] [17890.218047] RIP: 0010:lock_release+0x162/0x260 [17890.225524] ? lock_is_held_type+0xaa/0x120 [17890.230215] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [17890.234915] vfs_write+0x324/0x4f0 [17890.239437] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [17890.240930] ksys_pwrite64+0x65/0xa0 [17890.245365] [17890.249541] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17890.268279] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [17890.271679] do_syscall_64+0x38/0x80 [17890.276895] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [17890.280466] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17890.281956] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [17890.287087] RIP: 0033:0x7f48f0eeb93b [17890.294211] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [17890.297782] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17890.304922] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [17890.309967] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17890.317091] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17890.320661] ORIG_RAX: 0000000000000012 [17890.327797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17890.346532] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17890.353657] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17890.358873] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17890.366953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17890.370783] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17890.376520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17890.383643] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17890.390768] Call Trace: [17890.390769] [17890.397892] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17890.405016] ? watchdog_timer_fn+0x1b4/0x220 [17890.412147] [17890.419266] ? __hrtimer_run_queues+0x1d3/0x360 [17890.448943] ? hrtimer_interrupt+0x104/0x240 [17890.453211] ? __local_bh_enable+0x50/0x60 [17890.457306] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17890.462531] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17890.467486] [17890.469584] [17890.471681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17890.476986] ? igrab+0x43/0x50 [17890.480039] ? lock_release+0x162/0x260 [17890.483877] ? lock_release+0xc5/0x260 [17890.487623] _raw_spin_unlock+0x1b/0x40 [17890.491459] igrab+0x43/0x50 [17890.494338] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17890.499488] delayed_work+0x1d8/0x280 [ceph] [17890.503801] ? process_scheduled_works+0x22a/0x510 [17890.508585] process_scheduled_works+0x2a4/0x510 [17890.513200] worker_thread+0x188/0x340 [17890.516949] ? rescuer_thread+0x310/0x310 [17890.520953] kthread+0xf5/0x130 [17890.524090] ? kthread_complete_and_exit+0x20/0x20 [17890.528877] ret_from_fork+0x31/0x40 [17890.532454] ? kthread_complete_and_exit+0x20/0x20 [17890.537238] ret_from_fork_asm+0x11/0x20 [17890.541161] [17917.769630] watchdog: BUG: soft lockup - CPU#0 stuck for 5573s! [dbench:78683] [17917.776847] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17917.781630] watchdog: BUG: soft lockup - CPU#7 stuck for 14570s! [kworker/7:4:78808] [17917.832313] irq event stamp: 25292718 [17917.840045] Modules linked in: ceph libceph dns_resolver fscache netfs [17917.843701] hardirqs last enabled at (25292717): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17917.843706] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [17917.860376] hardirqs last disabled at (25292718): [] sysvec_apic_timer_interrupt+0xe/0x90 [17917.867762] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [17917.877485] softirqs last enabled at (25292630): [] __do_softirq+0x2be/0x371 [17917.884610] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [17917.893293] softirqs last disabled at (25292625): [] irq_exit_rcu+0xb0/0xd0 [17917.901648] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [17917.910162] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17917.916069] iTCO_vendor_support i2c_smbus [17917.925026] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17917.925027] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17917.929113] crct10dif_pclmul joydev [17917.936586] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17917.942061] crc32_pclmul ipmi_ssif [17917.945634] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17917.964371] ghash_clmulni_intel mei_me [17917.967856] [17917.973072] mei lpc_ich [17917.976902] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17917.978393] sha512_ssse3 acpi_ipmi [17917.980924] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17917.988047] mfd_core wmi [17917.991531] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17917.998656] ipmi_si ipmi_devintf [17918.001274] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17918.008397] ipmi_msghandler [17918.011708] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17918.018832] ioatdma acpi_pad [17918.021710] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17918.028833] acpi_power_meter ixgbe [17918.031797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17918.039875] igb [17918.043360] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17918.049096] mdio crc32c_intel [17918.050933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17918.058058] nvme ptp [17918.061110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17918.068233] nvme_core i2c_algo_bit [17918.070503] Call Trace: [17918.070505] [17918.077628] pps_core dca [17918.081113] ? watchdog_timer_fn+0x1b4/0x220 [17918.083557] fuse [17918.085569] ? __hrtimer_run_queues+0x1d3/0x360 [17918.088184] [17918.088184] irq event stamp: 66244586 [17918.092453] ? hrtimer_interrupt+0x104/0x240 [17918.094372] hardirqs last enabled at (66244585): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17918.098898] ? __local_bh_enable+0x50/0x60 [17918.100388] hardirqs last disabled at (66244586): [] sysvec_apic_timer_interrupt+0xe/0x90 [17918.104048] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17918.108308] softirqs last enabled at (66244584): [] __do_softirq+0x2be/0x371 [17918.118467] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17918.122556] softirqs last disabled at (66244579): [] irq_exit_rcu+0xb0/0xd0 [17918.132283] [17918.137500] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17918.146184] [17918.151141] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17918.159652] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17918.161750] Workqueue: events delayed_work [ceph] [17918.171131] ? queued_spin_lock_slowpath+0x16/0x2e0 [17918.173224] [17918.173224] RIP: 0010:__list_del_entry_valid_or_report+0x0/0xd0 [17918.180695] do_raw_spin_lock+0xb5/0xc0 [17918.185998] Code: 1d 82 e8 83 0c b0 ff 0f 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 [17918.190696] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17918.195566] RSP: 0018:ffffc90002117db0 EFLAGS: 00000212 [17918.197060] try_get_cap_refs+0x159/0x880 [ceph] [17918.202968] [17918.202969] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 00000001010ccda0 [17918.206804] __ceph_get_caps+0x4a1/0x760 [ceph] [17918.225537] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [17918.230326] ? __file_remove_privs+0x89/0x110 [17918.235536] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [17918.240149] ? wake_bit_function+0x70/0x70 [17918.241639] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17918.248766] ? ceph_write_iter+0x260/0x770 [ceph] [17918.253287] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17918.260431] ceph_write_iter+0x260/0x770 [ceph] [17918.264779] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17918.271907] ? lock_is_held_type+0xaa/0x120 [17918.275995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17918.283120] vfs_write+0x324/0x4f0 [17918.287816] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17918.294945] ksys_pwrite64+0x65/0xa0 [17918.299465] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17918.307542] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17918.311719] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17918.317457] do_syscall_64+0x38/0x80 [17918.320853] Call Trace: [17918.320856] [17918.327979] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17918.331549] ? watchdog_timer_fn+0x1b4/0x220 [17918.338674] RIP: 0033:0x7f48f0eeb93b [17918.343805] ? __hrtimer_run_queues+0x1d3/0x360 [17918.350930] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17918.354503] ? hrtimer_interrupt+0x104/0x240 [17918.356942] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17918.358954] ? __local_bh_enable+0x50/0x60 [17918.363997] ORIG_RAX: 0000000000000012 [17918.368265] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17918.371831] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17918.376356] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17918.395095] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17918.399359] [17918.404575] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17918.408667] [17918.408668] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17918.412496] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17918.417718] ? __list_add_valid_or_report+0xa0/0xa0 [17918.424839] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17918.429797] ceph_check_delayed_caps+0x8f/0x270 [ceph] [17918.436925] [17918.439038] delayed_work+0x1d8/0x280 [ceph] [17918.484261] ? process_scheduled_works+0x22a/0x510 [17918.489051] process_scheduled_works+0x2a4/0x510 [17918.493668] worker_thread+0x188/0x340 [17918.497418] ? rescuer_thread+0x310/0x310 [17918.501429] kthread+0xf5/0x130 [17918.504566] ? kthread_complete_and_exit+0x20/0x20 [17918.509351] ret_from_fork+0x31/0x40 [17918.512929] ? kthread_complete_and_exit+0x20/0x20 [17918.517714] ret_from_fork_asm+0x11/0x20 [17918.521644] [17945.770142] watchdog: BUG: soft lockup - CPU#0 stuck for 5599s! [dbench:78683] [17945.777361] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17945.782142] watchdog: BUG: soft lockup - CPU#7 stuck for 14596s! [kworker/7:4:78808] [17945.832817] irq event stamp: 25350708 [17945.832819] hardirqs last enabled at (25350707): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17945.840549] Modules linked in: [17945.844207] hardirqs last disabled at (25350708): [] sysvec_apic_timer_interrupt+0xe/0x90 [17945.854363] ceph [17945.857414] softirqs last enabled at (25350660): [] __do_softirq+0x2be/0x371 [17945.867138] libceph [17945.869062] softirqs last disabled at (25350655): [] irq_exit_rcu+0xb0/0xd0 [17945.877746] dns_resolver fscache [17945.879931] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17945.888441] netfs veth [17945.891753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17945.900698] nft_chain_nat xt_MASQUERADE [17945.903142] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [17945.910611] nf_nat nf_conntrack [17945.914537] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [17945.920015] nf_defrag_ipv6 nf_defrag_ipv4 [17945.923238] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17945.941977] nft_compat nf_tables [17945.946067] [17945.946068] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17945.951285] nfnetlink bridge [17945.954597] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17945.956086] stp llc [17945.963211] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17945.966174] xfs libcrc32c [17945.973299] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17945.975482] sunrpc intel_rapl_msr [17945.982606] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17945.985311] intel_rapl_common sb_edac [17945.992435] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17945.995833] x86_pkg_temp_thermal [17946.002956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17946.006701] coretemp kvm_intel [17946.014778] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17946.018088] kvm irqbypass [17946.023826] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17946.026965] i2c_i801 iTCO_wdt [17946.034087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17946.036793] iTCO_vendor_support [17946.043915] Call Trace: [17946.043916] [17946.046967] i2c_smbus crct10dif_pclmul [17946.054090] ? watchdog_timer_fn+0x1b4/0x220 [17946.057315] joydev crc32_pclmul [17946.059759] ? __hrtimer_run_queues+0x1d3/0x360 [17946.061770] ipmi_ssif ghash_clmulni_intel [17946.065604] ? hrtimer_interrupt+0x104/0x240 [17946.069863] mei_me mei [17946.073089] ? __local_bh_enable+0x50/0x60 [17946.077613] lpc_ich sha512_ssse3 [17946.081705] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17946.085968] acpi_ipmi [17946.088411] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17946.092503] mfd_core wmi [17946.095822] [17946.101038] ipmi_si ipmi_devintf [17946.103397] [17946.103398] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17946.108353] ipmi_msghandler ioatdma [17946.110976] ? queued_spin_lock_slowpath+0x16/0x2e0 [17946.113069] acpi_pad acpi_power_meter [17946.116382] do_raw_spin_lock+0xb5/0xc0 [17946.118476] ixgbe igb [17946.123783] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17946.127352] mdio crc32c_intel nvme [17946.132226] try_get_cap_refs+0x159/0x880 [ceph] [17946.135967] ptp nvme_core [17946.139819] __ceph_get_caps+0x4a1/0x760 [ceph] [17946.142172] i2c_algo_bit pps_core [17946.146960] ? __file_remove_privs+0x89/0x110 [17946.150439] dca [17946.155052] ? wake_bit_function+0x70/0x70 [17946.157755] fuse [17946.162282] ? ceph_write_iter+0x260/0x770 [ceph] [17946.165675] irq event stamp: 66467570 [17946.170028] ceph_write_iter+0x260/0x770 [ceph] [17946.171865] hardirqs last enabled at (66467569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17946.175959] ? lock_is_held_type+0xaa/0x120 [17946.177879] hardirqs last disabled at (66467570): [] sysvec_apic_timer_interrupt+0xe/0x90 [17946.182580] vfs_write+0x324/0x4f0 [17946.186233] softirqs last enabled at (66467568): [] __do_softirq+0x2be/0x371 [17946.190762] ksys_pwrite64+0x65/0xa0 [17946.200915] softirqs last disabled at (66467563): [] irq_exit_rcu+0xb0/0xd0 [17946.205092] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17946.214824] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17946.218224] do_syscall_64+0x38/0x80 [17946.226906] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17946.230479] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17946.238990] Workqueue: events delayed_work [ceph] [17946.244119] RIP: 0033:0x7f48f0eeb93b [17946.253498] [17946.253499] RIP: 0010:_raw_spin_unlock+0x2a/0x40 [17946.257067] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17946.264543] Code: f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 47 7e <74> 02 5b c3 0f 1f 44 00 00 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 [17946.269584] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17946.274281] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [17946.277851] ORIG_RAX: 0000000000000012 [17946.279342] [17946.279343] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [17946.283951] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17946.302691] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [17946.321429] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17946.326646] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [17946.331864] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17946.335694] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [17946.337185] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17946.344308] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [17946.351433] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17946.358557] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17946.365686] [17946.372804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17946.372822] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17946.431567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17946.438691] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17946.445826] Call Trace: [17946.448268] [17946.450278] ? watchdog_timer_fn+0x1b4/0x220 [17946.454550] ? __hrtimer_run_queues+0x1d3/0x360 [17946.459078] ? hrtimer_interrupt+0x104/0x240 [17946.463340] ? __local_bh_enable+0x50/0x60 [17946.467432] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17946.472647] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17946.477605] [17946.479701] [17946.481798] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17946.487108] ? _raw_spin_unlock+0x2a/0x40 [17946.491117] igrab+0x43/0x50 [17946.493995] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17946.499144] delayed_work+0x1d8/0x280 [ceph] [17946.503424] ? process_scheduled_works+0x22a/0x510 [17946.508208] process_scheduled_works+0x2a4/0x510 [17946.512832] worker_thread+0x188/0x340 [17946.516580] ? rescuer_thread+0x310/0x310 [17946.520584] kthread+0xf5/0x130 [17946.523720] ? kthread_complete_and_exit+0x20/0x20 [17946.528506] ret_from_fork+0x31/0x40 [17946.532076] ? kthread_complete_and_exit+0x20/0x20 [17946.536861] ret_from_fork_asm+0x11/0x20 [17946.540784] [17973.770654] watchdog: BUG: soft lockup - CPU#0 stuck for 5625s! [dbench:78683] [17973.777872] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [17973.782654] watchdog: BUG: soft lockup - CPU#7 stuck for 14622s! [kworker/7:4:78808] [17973.833327] irq event stamp: 25408840 [17973.833329] hardirqs last enabled at (25408839): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17973.841063] Modules linked in: [17973.844720] hardirqs last disabled at (25408840): [] sysvec_apic_timer_interrupt+0xe/0x90 [17973.854878] ceph [17973.857928] softirqs last enabled at (25408792): [] __do_softirq+0x2be/0x371 [17973.867653] libceph dns_resolver [17973.869576] softirqs last disabled at (25408787): [] irq_exit_rcu+0xb0/0xd0 [17973.878261] fscache netfs [17973.881572] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17973.890081] veth nft_chain_nat [17973.892788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17973.901729] xt_MASQUERADE nf_nat [17973.904867] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [17973.912347] nf_conntrack nf_defrag_ipv6 [17973.915657] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [17973.921135] nf_defrag_ipv4 nft_compat [17973.925053] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [17973.943789] nf_tables [17973.947535] [17973.952752] nfnetlink bridge [17973.955109] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [17973.956600] stp llc [17973.959563] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [17973.966687] xfs [17973.968871] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [17973.975996] libcrc32c sunrpc [17973.977834] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [17973.984958] intel_rapl_msr [17973.987921] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [17973.995053] intel_rapl_common [17973.997845] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [17974.004968] sb_edac [17974.008020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17974.016098] x86_pkg_temp_thermal coretemp [17974.018281] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [17974.024020] kvm_intel kvm [17974.028110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17974.035234] irqbypass [17974.037937] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17974.045061] i2c_i801 iTCO_wdt [17974.047418] Call Trace: [17974.047421] [17974.054543] iTCO_vendor_support i2c_smbus [17974.057594] ? watchdog_timer_fn+0x1b4/0x220 [17974.060053] crct10dif_pclmul joydev [17974.062067] ? __hrtimer_run_queues+0x1d3/0x360 [17974.066158] crc32_pclmul ipmi_ssif [17974.070423] ? hrtimer_interrupt+0x104/0x240 [17974.073992] ghash_clmulni_intel [17974.078515] ? __local_bh_enable+0x50/0x60 [17974.081999] mei_me mei [17974.086268] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17974.089488] lpc_ich [17974.093581] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17974.096023] sha512_ssse3 acpi_ipmi [17974.101239] [17974.103425] mfd_core wmi ipmi_si [17974.108381] [17974.111866] ipmi_devintf ipmi_msghandler [17974.113964] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17974.117274] ioatdma acpi_pad [17974.119375] ? queued_spin_lock_slowpath+0x18/0x2e0 [17974.123376] acpi_power_meter ixgbe [17974.128681] do_raw_spin_lock+0xb5/0xc0 [17974.131643] igb mdio [17974.136516] __cap_delay_requeue+0xa7/0x1d0 [ceph] [17974.139997] crc32c_intel nvme [17974.143832] try_get_cap_refs+0x159/0x880 [ceph] [17974.146099] ptp [17974.150890] __ceph_get_caps+0x4a1/0x760 [ceph] [17974.153934] nvme_core i2c_algo_bit [17974.158550] ? __file_remove_privs+0x89/0x110 [17974.160383] pps_core dca [17974.164907] ? wake_bit_function+0x70/0x70 [17974.168390] fuse [17974.172743] ? ceph_write_iter+0x260/0x770 [ceph] [17974.175359] irq event stamp: 66689602 [17974.179452] ceph_write_iter+0x260/0x770 [ceph] [17974.181373] hardirqs last enabled at (66689601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17974.186076] ? lock_is_held_type+0xaa/0x120 [17974.189729] hardirqs last disabled at (66689602): [] sysvec_apic_timer_interrupt+0xe/0x90 [17974.194254] vfs_write+0x324/0x4f0 [17974.204409] softirqs last enabled at (66689600): [] __do_softirq+0x2be/0x371 [17974.208591] ksys_pwrite64+0x65/0xa0 [17974.218311] softirqs last disabled at (66689595): [] irq_exit_rcu+0xb0/0xd0 [17974.221729] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [17974.230410] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [17974.233982] do_syscall_64+0x38/0x80 [17974.242492] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [17974.247625] entry_SYSCALL_64_after_hwframe+0x63/0xcd [17974.257002] Workqueue: events delayed_work [ceph] [17974.260572] RIP: 0033:0x7f48f0eeb93b [17974.268057] [17974.268058] RIP: 0010:lock_acquire+0xf0/0x2b0 [17974.273105] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [17974.277802] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [17974.281372] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [17974.282862] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [17974.287214] ORIG_RAX: 0000000000000012 [17974.305951] [17974.305952] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [17974.324690] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [17974.329907] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [17974.335122] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [17974.338954] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [17974.340445] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [17974.347568] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [17974.354694] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [17974.361816] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [17974.368942] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [17974.376065] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [17974.383196] [17974.390314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17974.427702] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [17974.434828] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17974.441952] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [17974.449075] Call Trace: [17974.451521] [17974.453531] ? watchdog_timer_fn+0x1b4/0x220 [17974.457797] ? __hrtimer_run_queues+0x1d3/0x360 [17974.462321] ? hrtimer_interrupt+0x104/0x240 [17974.466589] ? __local_bh_enable+0x50/0x60 [17974.470685] ? __sysvec_apic_timer_interrupt+0x61/0x160 [17974.475909] ? sysvec_apic_timer_interrupt+0x6d/0x90 [17974.480865] [17974.482962] [17974.485060] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [17974.490370] ? lock_acquire+0xf0/0x2b0 [17974.494118] ? igrab+0x43/0x50 [17974.497169] ? lock_release+0xc5/0x260 [17974.500921] _raw_spin_lock+0x30/0x40 [17974.504579] ? igrab+0x1d/0x50 [17974.507629] igrab+0x1d/0x50 [17974.510508] ceph_check_delayed_caps+0xb7/0x270 [ceph] [17974.515657] delayed_work+0x1d8/0x280 [ceph] [17974.519939] ? process_scheduled_works+0x22a/0x510 [17974.524728] process_scheduled_works+0x2a4/0x510 [17974.529348] worker_thread+0x188/0x340 [17974.533093] ? rescuer_thread+0x310/0x310 [17974.537105] kthread+0xf5/0x130 [17974.540251] ? kthread_complete_and_exit+0x20/0x20 [17974.545037] ret_from_fork+0x31/0x40 [17974.548631] ? kthread_complete_and_exit+0x20/0x20 [17974.553414] ret_from_fork_asm+0x11/0x20 [17974.557356] [18001.771167] watchdog: BUG: soft lockup - CPU#0 stuck for 5651s! [dbench:78683] [18001.778377] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18001.783167] watchdog: BUG: soft lockup - CPU#7 stuck for 14648s! [kworker/7:4:78808] [18001.833819] irq event stamp: 25466950 [18001.841566] Modules linked in: ceph libceph dns_resolver [18001.845225] hardirqs last enabled at (25466949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18001.850528] fscache netfs veth [18001.860685] hardirqs last disabled at (25466950): [] sysvec_apic_timer_interrupt+0xe/0x90 [18001.860688] nft_chain_nat [18001.863840] softirqs last enabled at (25466902): [] __do_softirq+0x2be/0x371 [18001.873564] xt_MASQUERADE nf_nat [18001.876269] softirqs last disabled at (25466897): [] irq_exit_rcu+0xb0/0xd0 [18001.884952] nf_conntrack nf_defrag_ipv6 [18001.888265] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18001.896775] nf_defrag_ipv4 nft_compat [18001.900693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18001.909636] nf_tables nfnetlink [18001.913379] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18001.920852] bridge stp [18001.924076] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18001.929563] llc xfs [18001.932006] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18001.950744] libcrc32c [18001.952928] [18001.952929] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18001.958143] sunrpc intel_rapl_msr [18001.960502] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18001.961991] intel_rapl_common sb_edac [18001.969116] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18001.972513] x86_pkg_temp_thermal coretemp [18001.979637] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18001.983382] kvm_intel kvm [18001.990507] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18001.994596] irqbypass i2c_i801 [18002.001722] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18002.004424] iTCO_wdt [18002.011565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18002.014703] iTCO_vendor_support [18002.022782] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18002.025053] i2c_smbus crct10dif_pclmul [18002.030789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18002.034013] joydev crc32_pclmul [18002.041137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18002.044968] ipmi_ssif ghash_clmulni_intel [18002.052092] Call Trace: [18002.052093] [18002.055316] mei_me [18002.062443] ? watchdog_timer_fn+0x1b4/0x220 [18002.066531] mei lpc_ich [18002.068977] ? __hrtimer_run_queues+0x1d3/0x360 [18002.070987] sha512_ssse3 acpi_ipmi [18002.073087] ? hrtimer_interrupt+0x104/0x240 [18002.077348] mfd_core wmi [18002.079877] ? __local_bh_enable+0x50/0x60 [18002.084403] ipmi_si ipmi_devintf [18002.087890] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18002.092151] ipmi_msghandler ioatdma [18002.094770] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18002.098859] acpi_pad acpi_power_meter [18002.102170] [18002.107388] ixgbe igb mdio [18002.110957] [18002.115915] crc32c_intel nvme [18002.119660] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18002.121757] ptp nvme_core [18002.124568] ? queued_spin_lock_slowpath+0x16/0x2e0 [18002.126663] i2c_algo_bit pps_core [18002.129715] do_raw_spin_lock+0xb5/0xc0 [18002.135018] dca fuse [18002.137721] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18002.142592] [18002.145991] try_get_cap_refs+0x159/0x880 [ceph] [18002.149819] irq event stamp: 66911018 [18002.152096] __ceph_get_caps+0x4a1/0x760 [ceph] [18002.156874] hardirqs last enabled at (66911017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18002.158369] ? __file_remove_privs+0x89/0x110 [18002.162975] hardirqs last disabled at (66911018): [] sysvec_apic_timer_interrupt+0xe/0x90 [18002.166634] ? wake_bit_function+0x70/0x70 [18002.171156] softirqs last enabled at (66911016): [] __do_softirq+0x2be/0x371 [18002.181319] ? ceph_write_iter+0x260/0x770 [ceph] [18002.185666] softirqs last disabled at (66911011): [] irq_exit_rcu+0xb0/0xd0 [18002.195390] ceph_write_iter+0x260/0x770 [ceph] [18002.199482] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18002.208169] ? lock_is_held_type+0xaa/0x120 [18002.212868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18002.212869] Workqueue: events delayed_work [ceph] [18002.221385] vfs_write+0x324/0x4f0 [18002.225906] [18002.225907] RIP: 0010:lock_acquire+0xf0/0x2b0 [18002.235287] ksys_pwrite64+0x65/0xa0 [18002.239461] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18002.246931] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18002.251630] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18002.255027] do_syscall_64+0x38/0x80 [18002.256517] [18002.260868] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18002.264440] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18002.283177] RIP: 0033:0x7f48f0eeb93b [18002.288307] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18002.293524] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18002.297095] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18002.298587] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18002.303630] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18002.310754] ORIG_RAX: 0000000000000012 [18002.314326] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18002.321449] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18002.340186] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18002.347311] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18002.352528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18002.359653] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18002.363483] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18002.370608] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18002.377732] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18002.385808] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18002.392933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18002.398675] [18002.405796] Call Trace: [18002.405797] [18002.448055] ? watchdog_timer_fn+0x1b4/0x220 [18002.452319] ? __hrtimer_run_queues+0x1d3/0x360 [18002.456858] ? hrtimer_interrupt+0x104/0x240 [18002.461123] ? __local_bh_enable+0x50/0x60 [18002.465219] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18002.470442] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18002.475397] [18002.477496] [18002.479595] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18002.484903] ? lock_acquire+0xf0/0x2b0 [18002.488652] ? igrab+0x43/0x50 [18002.491702] ? lock_release+0xc5/0x260 [18002.495446] _raw_spin_lock+0x30/0x40 [18002.499102] ? igrab+0x1d/0x50 [18002.502154] igrab+0x1d/0x50 [18002.505032] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18002.510180] delayed_work+0x1d8/0x280 [ceph] [18002.514463] ? process_scheduled_works+0x22a/0x510 [18002.519253] process_scheduled_works+0x2a4/0x510 [18002.523868] worker_thread+0x188/0x340 [18002.527617] ? rescuer_thread+0x310/0x310 [18002.531621] kthread+0xf5/0x130 [18002.534758] ? kthread_complete_and_exit+0x20/0x20 [18002.539543] ret_from_fork+0x31/0x40 [18002.543111] ? kthread_complete_and_exit+0x20/0x20 [18002.547897] ret_from_fork_asm+0x11/0x20 [18002.551820] [18029.771681] watchdog: BUG: soft lockup - CPU#0 stuck for 5677s! [dbench:78683] [18029.778901] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18029.783681] watchdog: BUG: soft lockup - CPU#7 stuck for 14674s! [kworker/7:4:78808] [18029.834355] irq event stamp: 25525132 [18029.842088] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [18029.845747] hardirqs last enabled at (25525131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18029.853919] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18029.864076] hardirqs last disabled at (25525132): [] sysvec_apic_timer_interrupt+0xe/0x90 [18029.864079] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [18029.880839] softirqs last enabled at (25525106): [] __do_softirq+0x2be/0x371 [18029.887538] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [18029.896221] softirqs last disabled at (25525101): [] irq_exit_rcu+0xb0/0xd0 [18029.905357] kvm irqbypass i2c_i801 iTCO_wdt [18029.913870] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18029.918131] iTCO_vendor_support i2c_smbus crct10dif_pclmul [18029.927078] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18029.927079] joydev [18029.932639] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18029.940111] crc32_pclmul [18029.942209] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18029.947686] ipmi_ssif ghash_clmulni_intel [18029.950302] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18029.969041] mei_me mei [18029.973132] [18029.978350] lpc_ich [18029.980809] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18029.982302] sha512_ssse3 acpi_ipmi [18029.984484] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18029.991609] mfd_core wmi [18029.995093] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18030.002216] ipmi_si ipmi_devintf [18030.004834] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18030.011957] ipmi_msghandler ioatdma [18030.015270] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18030.022394] acpi_pad acpi_power_meter [18030.025965] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18030.033088] ixgbe igb [18030.036833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18030.044909] mdio [18030.047267] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18030.053005] crc32c_intel nvme [18030.054930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18030.062053] ptp [18030.065121] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18030.072244] nvme_core i2c_algo_bit [18030.074082] Call Trace: [18030.074085] [18030.081206] pps_core dca [18030.084693] ? watchdog_timer_fn+0x1b4/0x220 [18030.087136] fuse [18030.089145] ? __hrtimer_run_queues+0x1d3/0x360 [18030.091762] [18030.091763] irq event stamp: 67133048 [18030.096029] ? hrtimer_interrupt+0x104/0x240 [18030.097951] hardirqs last enabled at (67133047): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18030.102475] ? __local_bh_enable+0x50/0x60 [18030.103965] hardirqs last disabled at (67133048): [] sysvec_apic_timer_interrupt+0xe/0x90 [18030.107626] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18030.111887] softirqs last enabled at (67133046): [] __do_softirq+0x2be/0x371 [18030.122046] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18030.126135] softirqs last disabled at (67133041): [] irq_exit_rcu+0xb0/0xd0 [18030.135859] [18030.141080] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18030.149760] [18030.154718] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18030.163230] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18030.165327] Workqueue: events delayed_work [ceph] [18030.174708] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18030.176800] [18030.176801] RIP: 0010:lock_acquire+0xf0/0x2b0 [18030.184273] do_raw_spin_lock+0xb5/0xc0 [18030.189577] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18030.194276] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18030.199145] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18030.200639] try_get_cap_refs+0x159/0x880 [ceph] [18030.204986] [18030.204987] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18030.208824] __ceph_get_caps+0x4a1/0x760 [ceph] [18030.227555] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18030.232344] ? __file_remove_privs+0x89/0x110 [18030.237556] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18030.242167] ? wake_bit_function+0x70/0x70 [18030.243658] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18030.250784] ? ceph_write_iter+0x260/0x770 [ceph] [18030.255307] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18030.255308] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18030.262430] ceph_write_iter+0x260/0x770 [ceph] [18030.266781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18030.273910] ? lock_is_held_type+0xaa/0x120 [18030.277994] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18030.285123] vfs_write+0x324/0x4f0 [18030.289818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18030.296946] ksys_pwrite64+0x65/0xa0 [18030.305019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18030.309545] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18030.315279] Call Trace: [18030.315281] [18030.319459] do_syscall_64+0x38/0x80 [18030.326582] ? watchdog_timer_fn+0x1b4/0x220 [18030.329981] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18030.337105] ? __hrtimer_run_queues+0x1d3/0x360 [18030.340674] RIP: 0033:0x7f48f0eeb93b [18030.347802] ? hrtimer_interrupt+0x104/0x240 [18030.352931] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18030.355373] ? __local_bh_enable+0x50/0x60 [18030.357384] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18030.360957] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18030.365217] ORIG_RAX: 0000000000000012 [18030.370263] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18030.374785] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18030.378370] [18030.382629] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18030.401367] [18030.405457] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18030.410676] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18030.415894] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18030.419730] ? lock_acquire+0xf0/0x2b0 [18030.424680] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18030.431806] ? igrab+0x43/0x50 [18030.433909] [18030.441027] ? lock_release+0xc5/0x260 [18030.482526] _raw_spin_lock+0x30/0x40 [18030.486183] ? igrab+0x1d/0x50 [18030.489233] igrab+0x1d/0x50 [18030.492111] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18030.497261] delayed_work+0x1d8/0x280 [ceph] [18030.501541] ? process_scheduled_works+0x22a/0x510 [18030.506331] process_scheduled_works+0x2a4/0x510 [18030.510947] worker_thread+0x188/0x340 [18030.514698] ? rescuer_thread+0x310/0x310 [18030.518709] kthread+0xf5/0x130 [18030.521856] ? kthread_complete_and_exit+0x20/0x20 [18030.526640] ret_from_fork+0x31/0x40 [18030.530219] ? kthread_complete_and_exit+0x20/0x20 [18030.535003] ret_from_fork_asm+0x11/0x20 [18030.538927] [18057.772193] watchdog: BUG: soft lockup - CPU#0 stuck for 5703s! [dbench:78683] [18057.779403] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18057.784193] watchdog: BUG: soft lockup - CPU#7 stuck for 14700s! [kworker/7:4:78808] [18057.834844] irq event stamp: 25583138 [18057.834845] hardirqs last enabled at (25583137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18057.842592] Modules linked in: ceph [18057.846250] hardirqs last disabled at (25583138): [] sysvec_apic_timer_interrupt+0xe/0x90 [18057.856407] libceph dns_resolver [18057.859892] softirqs last enabled at (25583090): [] __do_softirq+0x2be/0x371 [18057.869616] fscache netfs [18057.872927] softirqs last disabled at (25583085): [] irq_exit_rcu+0xb0/0xd0 [18057.881610] veth nft_chain_nat [18057.884314] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18057.892827] xt_MASQUERADE [18057.895963] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18057.904906] nf_nat nf_conntrack [18057.907611] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18057.915082] nf_defrag_ipv6 nf_defrag_ipv4 [18057.918306] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18057.923784] nft_compat nf_tables [18057.927875] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18057.946612] nfnetlink [18057.949922] [18057.955139] bridge stp [18057.957497] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18057.958989] llc xfs [18057.961431] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18057.968557] libcrc32c sunrpc [18057.970739] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18057.977869] intel_rapl_msr intel_rapl_common [18057.980836] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18057.987962] sb_edac x86_pkg_temp_thermal [18057.992313] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18057.999436] coretemp kvm_intel [18058.003442] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18058.010565] kvm [18058.013719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18058.021796] irqbypass i2c_i801 [18058.023634] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18058.029371] iTCO_wdt iTCO_vendor_support [18058.032509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18058.039633] i2c_smbus crct10dif_pclmul [18058.043637] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18058.050761] joydev crc32_pclmul [18058.054591] Call Trace: [18058.054593] [18058.061717] ipmi_ssif ghash_clmulni_intel [18058.064941] ? watchdog_timer_fn+0x1b4/0x220 [18058.067384] mei_me mei [18058.069397] ? __hrtimer_run_queues+0x1d3/0x360 [18058.073484] lpc_ich sha512_ssse3 [18058.077754] ? hrtimer_interrupt+0x104/0x240 [18058.080193] acpi_ipmi mfd_core [18058.084717] ? __local_bh_enable+0x50/0x60 [18058.088028] wmi ipmi_si [18058.092296] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18058.095429] ipmi_devintf ipmi_msghandler [18058.099521] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18058.102052] ioatdma acpi_pad [18058.107270] [18058.111274] acpi_power_meter ixgbe [18058.116230] [18058.119194] igb mdio [18058.121294] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18058.124776] crc32c_intel nvme [18058.126886] ? queued_spin_lock_slowpath+0x18/0x2e0 [18058.129154] ptp nvme_core [18058.134458] do_raw_spin_lock+0xb5/0xc0 [18058.137509] i2c_algo_bit pps_core [18058.142379] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18058.145082] dca fuse [18058.148916] try_get_cap_refs+0x159/0x880 [ceph] [18058.152309] [18058.152310] irq event stamp: 67354810 [18058.157100] __ceph_get_caps+0x4a1/0x760 [ceph] [18058.159365] hardirqs last enabled at (67354809): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18058.163979] ? __file_remove_privs+0x89/0x110 [18058.165466] hardirqs last disabled at (67354810): [] sysvec_apic_timer_interrupt+0xe/0x90 [18058.169126] ? wake_bit_function+0x70/0x70 [18058.173649] softirqs last enabled at (67354808): [] __do_softirq+0x2be/0x371 [18058.183809] ? ceph_write_iter+0x260/0x770 [ceph] [18058.188157] softirqs last disabled at (67354803): [] irq_exit_rcu+0xb0/0xd0 [18058.197880] ceph_write_iter+0x260/0x770 [ceph] [18058.201971] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18058.210658] ? lock_is_held_type+0xaa/0x120 [18058.215353] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18058.223883] vfs_write+0x324/0x4f0 [18058.228405] Workqueue: events delayed_work [ceph] [18058.237788] ksys_pwrite64+0x65/0xa0 [18058.241961] [18058.241962] RIP: 0010:lock_acquire+0xf0/0x2b0 [18058.249430] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18058.252828] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18058.257526] do_syscall_64+0x38/0x80 [18058.261096] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18058.262597] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18058.266947] [18058.266947] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18058.272078] RIP: 0033:0x7f48f0eeb93b [18058.290815] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18058.294387] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18058.299602] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18058.304646] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18058.306138] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18058.313262] ORIG_RAX: 0000000000000012 [18058.316832] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18058.323957] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18058.342695] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18058.349818] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18058.355035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18058.362161] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18058.365991] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18058.373115] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18058.380240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18058.388318] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18058.395440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18058.401185] [18058.408303] Call Trace: [18058.408304] [18058.450564] ? watchdog_timer_fn+0x1b4/0x220 [18058.454829] ? __hrtimer_run_queues+0x1d3/0x360 [18058.459353] ? hrtimer_interrupt+0x104/0x240 [18058.463615] ? __local_bh_enable+0x50/0x60 [18058.467708] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18058.472931] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18058.477889] [18058.479986] [18058.482085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18058.487391] ? lock_acquire+0xf0/0x2b0 [18058.491141] ? igrab+0x43/0x50 [18058.494193] ? lock_release+0xc5/0x260 [18058.497937] _raw_spin_lock+0x30/0x40 [18058.501599] ? igrab+0x1d/0x50 [18058.504652] igrab+0x1d/0x50 [18058.507532] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18058.512680] delayed_work+0x1d8/0x280 [ceph] [18058.516962] ? process_scheduled_works+0x22a/0x510 [18058.521751] process_scheduled_works+0x2a4/0x510 [18058.526368] worker_thread+0x188/0x340 [18058.530117] ? rescuer_thread+0x310/0x310 [18058.534120] kthread+0xf5/0x130 [18058.537257] ? kthread_complete_and_exit+0x20/0x20 [18058.542043] ret_from_fork+0x31/0x40 [18058.545611] ? kthread_complete_and_exit+0x20/0x20 [18058.550397] ret_from_fork_asm+0x11/0x20 [18058.554318] [18060.644612] rcu: INFO: rcu_preempt self-detected stall on CPU [18060.650369] rcu: 7-....: (5948603 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/5947370 fqs=2975787 [18060.660794] rcu: (t=6123925 jiffies g=1309609 q=242255 ncpus=8) [18060.666793] Sending NMI from CPU 7 to CPUs 0: [18060.671145] NMI backtrace for cpu 0 [18060.671147] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18060.671148] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18060.671149] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18060.671152] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18060.671154] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18060.671155] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18060.671156] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18060.671157] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18060.671158] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18060.671159] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18060.671160] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18060.671161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18060.671161] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18060.671162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18060.671163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18060.671164] Call Trace: [18060.671164] [18060.671166] ? nmi_cpu_backtrace+0xad/0x120 [18060.671169] ? nmi_cpu_backtrace_handler+0x11/0x20 [18060.671171] ? nmi_handle+0xd4/0x260 [18060.671175] ? default_do_nmi+0x49/0xf0 [18060.671177] ? exc_nmi+0x11e/0x150 [18060.671178] ? end_repeat_nmi+0x16/0x67 [18060.671182] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18060.671185] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18060.671187] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18060.671190] [18060.671190] [18060.671191] do_raw_spin_lock+0xb5/0xc0 [18060.671193] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18060.671208] try_get_cap_refs+0x159/0x880 [ceph] [18060.671222] __ceph_get_caps+0x4a1/0x760 [ceph] [18060.671235] ? __file_remove_privs+0x89/0x110 [18060.671237] ? wake_bit_function+0x70/0x70 [18060.671239] ? ceph_write_iter+0x260/0x770 [ceph] [18060.671250] ceph_write_iter+0x260/0x770 [ceph] [18060.671262] ? lock_is_held_type+0xaa/0x120 [18060.671264] vfs_write+0x324/0x4f0 [18060.671268] ksys_pwrite64+0x65/0xa0 [18060.671270] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18060.671273] do_syscall_64+0x38/0x80 [18060.671276] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18060.671277] RIP: 0033:0x7f48f0eeb93b [18060.671279] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18060.671280] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [18060.671281] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18060.671282] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18060.671283] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18060.671283] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18060.671284] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18060.671287] [18060.672143] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18060.975962] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18060.983432] Workqueue: events delayed_work [ceph] [18060.988147] RIP: 0010:lock_acquire+0xf0/0x2b0 [18060.992499] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18061.011236] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18061.016453] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18061.023578] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18061.030701] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18061.037826] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18061.044949] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18061.052075] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18061.060153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18061.065889] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18061.073014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18061.080139] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18061.087262] Call Trace: [18061.089707] [18061.091717] ? rcu_dump_cpu_stacks+0xd1/0x180 [18061.096077] ? rcu_sched_clock_irq+0x633/0x1390 [18061.100601] ? lock_release+0xc5/0x260 [18061.104344] ? lock_is_held_type+0xaa/0x120 [18061.108522] ? lock_is_held_type+0xaa/0x120 [18061.112702] ? tick_sched_do_timer+0xa0/0xa0 [18061.116973] ? rcu_read_lock_sched_held+0x56/0x90 [18061.121671] ? tick_sched_do_timer+0xa0/0xa0 [18061.125933] ? update_process_times+0x5f/0x90 [18061.130285] ? tick_sched_handle.isra.16+0x1e/0x50 [18061.135068] ? tick_sched_timer+0x6f/0x80 [18061.139074] ? __hrtimer_run_queues+0x1d3/0x360 [18061.143600] ? hrtimer_interrupt+0x104/0x240 [18061.147870] ? __local_bh_enable+0x50/0x60 [18061.151962] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18061.157186] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18061.162143] [18061.164241] [18061.166339] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18061.171648] ? lock_acquire+0xf0/0x2b0 [18061.175396] ? igrab+0x43/0x50 [18061.178448] ? lock_release+0xc5/0x260 [18061.182191] _raw_spin_lock+0x30/0x40 [18061.185847] ? igrab+0x1d/0x50 [18061.188899] igrab+0x1d/0x50 [18061.191776] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18061.196927] delayed_work+0x1d8/0x280 [ceph] [18061.201206] ? process_scheduled_works+0x22a/0x510 [18061.205989] process_scheduled_works+0x2a4/0x510 [18061.210605] worker_thread+0x188/0x340 [18061.214354] ? rescuer_thread+0x310/0x310 [18061.218357] kthread+0xf5/0x130 [18061.221494] ? kthread_complete_and_exit+0x20/0x20 [18061.226279] ret_from_fork+0x31/0x40 [18061.229849] ? kthread_complete_and_exit+0x20/0x20 [18061.234633] ret_from_fork_asm+0x11/0x20 [18061.238557] [18085.772704] watchdog: BUG: soft lockup - CPU#0 stuck for 5729s! [dbench:78683] [18085.779925] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18085.784703] watchdog: BUG: soft lockup - CPU#7 stuck for 14726s! [kworker/7:4:78808] [18085.835377] irq event stamp: 25641238 [18085.843105] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18085.846763] hardirqs last enabled at (25641237): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18085.853723] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [18085.863881] hardirqs last disabled at (25641238): [] sysvec_apic_timer_interrupt+0xe/0x90 [18085.870832] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [18085.880556] softirqs last enabled at (25641190): [] __do_softirq+0x2be/0x371 [18085.886993] llc xfs libcrc32c sunrpc intel_rapl_msr [18085.895679] softirqs last disabled at (25641185): [] irq_exit_rcu+0xb0/0xd0 [18085.900635] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [18085.909147] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18085.916358] kvm irqbypass i2c_i801 [18085.925320] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18085.928802] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [18085.936273] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18085.936275] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [18085.948100] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18085.953928] mei_me mei lpc_ich [18085.972667] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18085.972668] sha512_ssse3 [18085.972669] [18085.975804] acpi_ipmi mfd_core [18085.981020] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18085.983638] wmi ipmi_si [18085.985129] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18085.988268] ipmi_devintf [18085.995391] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18085.997921] ipmi_msghandler ioatdma [18086.005045] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18086.007663] acpi_pad acpi_power_meter [18086.014786] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18086.018357] ixgbe igb [18086.025500] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18086.029242] mdio crc32c_intel [18086.036379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18086.038733] nvme ptp [18086.046809] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18086.049860] nvme_core i2c_algo_bit [18086.055598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18086.057870] pps_core dca [18086.064995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18086.068478] fuse [18086.075602] Call Trace: [18086.075604] [18086.078220] [18086.078221] irq event stamp: 67571994 [18086.085345] ? watchdog_timer_fn+0x1b4/0x220 [18086.087267] hardirqs last enabled at (67571993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18086.089712] ? __hrtimer_run_queues+0x1d3/0x360 [18086.091723] hardirqs last disabled at (67571994): [] sysvec_apic_timer_interrupt+0xe/0x90 [18086.093217] ? hrtimer_interrupt+0x104/0x240 [18086.096870] softirqs last enabled at (67571992): [] __do_softirq+0x2be/0x371 [18086.101134] ? __local_bh_enable+0x50/0x60 [18086.111293] softirqs last disabled at (67571987): [] irq_exit_rcu+0xb0/0xd0 [18086.115820] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18086.125540] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18086.129805] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18086.138487] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18086.138489] Workqueue: events delayed_work [ceph] [18086.142579] [18086.151101] [18086.151102] RIP: 0010:ceph_check_delayed_caps+0x5f/0x270 [ceph] [18086.156315] [18086.156316] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18086.165693] Code: 1f 44 00 00 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 <48> 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff [18086.170655] ? queued_spin_lock_slowpath+0x16/0x2e0 [18086.178122] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [18086.182820] do_raw_spin_lock+0xb5/0xc0 [18086.184917] [18086.184918] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [18086.186410] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18086.192317] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [18086.194420] try_get_cap_refs+0x159/0x880 [ceph] [18086.199719] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [18086.218462] __ceph_get_caps+0x4a1/0x760 [ceph] [18086.223327] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18086.228551] ? __file_remove_privs+0x89/0x110 [18086.232392] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18086.233885] ? wake_bit_function+0x70/0x70 [18086.241008] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18086.245795] ? ceph_write_iter+0x260/0x770 [ceph] [18086.252918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18086.257528] ceph_write_iter+0x260/0x770 [ceph] [18086.264652] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18086.269180] ? lock_is_held_type+0xaa/0x120 [18086.276300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18086.280652] vfs_write+0x324/0x4f0 [18086.287775] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18086.291870] ksys_pwrite64+0x65/0xa0 [18086.299944] Call Trace: [18086.304641] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18086.310394] [18086.314921] do_syscall_64+0x38/0x80 [18086.322044] ? watchdog_timer_fn+0x1b4/0x220 [18086.326221] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18086.333347] ? __hrtimer_run_queues+0x1d3/0x360 [18086.336744] RIP: 0033:0x7f48f0eeb93b [18086.343869] ? hrtimer_interrupt+0x104/0x240 [18086.347437] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18086.349880] ? __local_bh_enable+0x50/0x60 [18086.355011] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18086.357025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18086.360593] ORIG_RAX: 0000000000000012 [18086.364858] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18086.369901] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18086.374426] [18086.377997] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18086.382260] [18086.382262] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18086.400998] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18086.405112] ? ceph_check_delayed_caps+0x5f/0x270 [ceph] [18086.410323] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18086.415542] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [18086.419371] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18086.424349] delayed_work+0x1d8/0x280 [ceph] [18086.431476] [18086.433571] ? process_scheduled_works+0x22a/0x510 [18086.491341] process_scheduled_works+0x2a4/0x510 [18086.495957] worker_thread+0x188/0x340 [18086.499707] ? rescuer_thread+0x310/0x310 [18086.503709] kthread+0xf5/0x130 [18086.506847] ? kthread_complete_and_exit+0x20/0x20 [18086.511631] ret_from_fork+0x31/0x40 [18086.515211] ? kthread_complete_and_exit+0x20/0x20 [18086.519993] ret_from_fork_asm+0x11/0x20 [18086.523918] [18113.773213] watchdog: BUG: soft lockup - CPU#0 stuck for 5755s! [dbench:78683] [18113.780429] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18113.785213] watchdog: BUG: soft lockup - CPU#7 stuck for 14752s! [kworker/7:4:78808] [18113.835887] irq event stamp: 25699278 [18113.835888] hardirqs last enabled at (25699277): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18113.843620] Modules linked in: [18113.847276] hardirqs last disabled at (25699278): [] sysvec_apic_timer_interrupt+0xe/0x90 [18113.857434] ceph [18113.860484] softirqs last enabled at (25699230): [] __do_softirq+0x2be/0x371 [18113.870209] libceph [18113.872132] softirqs last disabled at (25699225): [] irq_exit_rcu+0xb0/0xd0 [18113.880817] dns_resolver [18113.883002] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18113.891512] fscache netfs [18113.894128] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18113.903073] veth nft_chain_nat [18113.905776] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18113.913248] xt_MASQUERADE nf_nat [18113.916387] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18113.921863] nf_conntrack [18113.925191] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18113.943928] nf_defrag_ipv6 [18113.946546] [18113.946546] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18113.951762] nf_defrag_ipv4 nft_compat [18113.954553] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18113.956046] nf_tables nfnetlink [18113.963169] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18113.966912] bridge stp [18113.974038] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18113.977260] llc xfs [18113.984385] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18113.986830] libcrc32c sunrpc [18113.993953] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18113.996137] intel_rapl_msr intel_rapl_common [18114.003260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18114.006225] sb_edac x86_pkg_temp_thermal [18114.014303] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18114.018655] coretemp kvm_intel [18114.024392] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18114.028394] kvm irqbypass [18114.035520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18114.038656] i2c_i801 iTCO_wdt [18114.045782] Call Trace: [18114.045783] [18114.048484] iTCO_vendor_support i2c_smbus [18114.055613] ? watchdog_timer_fn+0x1b4/0x220 [18114.058660] crct10dif_pclmul joydev [18114.061105] ? __hrtimer_run_queues+0x1d3/0x360 [18114.063115] crc32_pclmul ipmi_ssif [18114.067209] ? hrtimer_interrupt+0x104/0x240 [18114.071469] ghash_clmulni_intel mei_me [18114.075040] ? __local_bh_enable+0x50/0x60 [18114.079563] mei lpc_ich [18114.083050] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18114.087312] sha512_ssse3 [18114.091145] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18114.095235] acpi_ipmi mfd_core [18114.097765] [18114.102981] wmi ipmi_si [18114.105599] [18114.110558] ipmi_devintf ipmi_msghandler [18114.113696] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18114.115792] ioatdma acpi_pad [18114.118326] ? queued_spin_lock_slowpath+0x18/0x2e0 [18114.120419] acpi_power_meter [18114.124426] do_raw_spin_lock+0xb5/0xc0 [18114.129729] ixgbe [18114.132693] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18114.137562] igb mdio [18114.140529] try_get_cap_refs+0x159/0x880 [ceph] [18114.144358] crc32c_intel nvme [18114.146374] __ceph_get_caps+0x4a1/0x760 [ceph] [18114.151153] ptp nvme_core [18114.153428] ? __file_remove_privs+0x89/0x110 [18114.158035] i2c_algo_bit pps_core [18114.161084] ? wake_bit_function+0x70/0x70 [18114.165614] dca [18114.168324] ? ceph_write_iter+0x260/0x770 [ceph] [18114.172673] fuse [18114.176069] ceph_write_iter+0x260/0x770 [ceph] [18114.180160] irq event stamp: 67793724 [18114.182001] ? lock_is_held_type+0xaa/0x120 [18114.186696] hardirqs last enabled at (67793723): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18114.188622] vfs_write+0x324/0x4f0 [18114.193143] hardirqs last disabled at (67793724): [] sysvec_apic_timer_interrupt+0xe/0x90 [18114.196804] ksys_pwrite64+0x65/0xa0 [18114.200978] softirqs last enabled at (67793722): [] __do_softirq+0x2be/0x371 [18114.211135] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18114.214534] softirqs last disabled at (67793717): [] irq_exit_rcu+0xb0/0xd0 [18114.224260] do_syscall_64+0x38/0x80 [18114.227828] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18114.236512] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18114.241643] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18114.250154] RIP: 0033:0x7f48f0eeb93b [18114.253724] Workqueue: events delayed_work [ceph] [18114.263101] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18114.268145] [18114.268146] RIP: 0010:lock_acquire+0x85/0x2b0 [18114.275617] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18114.279186] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [18114.283895] ORIG_RAX: 0000000000000012 [18114.302630] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [18114.304121] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18114.308472] [18114.313690] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18114.332428] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [18114.336259] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18114.341476] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [18114.348617] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18114.350108] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [18114.357231] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18114.364356] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [18114.371485] [18114.378619] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18114.416428] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18114.424503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18114.430241] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18114.437367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18114.444491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18114.451619] Call Trace: [18114.454068] [18114.456079] ? watchdog_timer_fn+0x1b4/0x220 [18114.460342] ? __hrtimer_run_queues+0x1d3/0x360 [18114.464870] ? hrtimer_interrupt+0x104/0x240 [18114.469139] ? __local_bh_enable+0x50/0x60 [18114.473232] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18114.478455] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18114.483414] [18114.485510] [18114.487619] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18114.492934] ? lock_acquire+0x85/0x2b0 [18114.496684] ? igrab+0x43/0x50 [18114.499734] ? lock_release+0xc5/0x260 [18114.503479] _raw_spin_lock+0x30/0x40 [18114.507141] ? igrab+0x1d/0x50 [18114.510193] igrab+0x1d/0x50 [18114.513074] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18114.518221] delayed_work+0x1d8/0x280 [ceph] [18114.522502] ? process_scheduled_works+0x22a/0x510 [18114.527284] process_scheduled_works+0x2a4/0x510 [18114.531904] worker_thread+0x188/0x340 [18114.535649] ? rescuer_thread+0x310/0x310 [18114.539651] kthread+0xf5/0x130 [18114.542789] ? kthread_complete_and_exit+0x20/0x20 [18114.547574] ret_from_fork+0x31/0x40 [18114.551144] ? kthread_complete_and_exit+0x20/0x20 [18114.555930] ret_from_fork_asm+0x11/0x20 [18114.559851] [18141.773723] watchdog: BUG: soft lockup - CPU#0 stuck for 5781s! [dbench:78683] [18141.780933] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18141.785722] watchdog: BUG: soft lockup - CPU#7 stuck for 14778s! [kworker/7:4:78808] [18141.836374] irq event stamp: 25757420 [18141.844123] Modules linked in: ceph libceph dns_resolver [18141.847780] hardirqs last enabled at (25757419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18141.853086] fscache netfs veth nft_chain_nat [18141.863242] hardirqs last disabled at (25757420): [] sysvec_apic_timer_interrupt+0xe/0x90 [18141.867593] xt_MASQUERADE nf_nat nf_conntrack [18141.877317] softirqs last enabled at (25757372): [] __do_softirq+0x2be/0x371 [18141.877318] nf_defrag_ipv6 [18141.877320] softirqs last disabled at (25757367): [] irq_exit_rcu+0xb0/0xd0 [18141.881756] nf_defrag_ipv4 nft_compat [18141.890438] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18141.893228] nf_tables [18141.901739] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18141.905485] nfnetlink bridge [18141.914428] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18141.916786] stp llc [18141.924256] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18141.927221] xfs libcrc32c [18141.932697] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18141.934882] sunrpc [18141.953621] [18141.956323] intel_rapl_msr intel_rapl_common [18141.961540] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18141.963638] sb_edac x86_pkg_temp_thermal [18141.965128] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18141.969480] coretemp kvm_intel [18141.976605] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18141.980609] kvm [18141.987732] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18141.990870] irqbypass i2c_i801 [18141.997994] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18141.999832] iTCO_wdt iTCO_vendor_support [18142.006954] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18142.010092] i2c_smbus crct10dif_pclmul [18142.017235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18142.021239] joydev crc32_pclmul [18142.029317] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18142.033146] ipmi_ssif ghash_clmulni_intel [18142.038883] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18142.042120] mei_me mei [18142.049240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18142.053333] lpc_ich sha512_ssse3 [18142.060456] Call Trace: [18142.060458] [18142.062900] acpi_ipmi mfd_core [18142.070025] ? watchdog_timer_fn+0x1b4/0x220 [18142.073334] wmi ipmi_si [18142.075780] ? __hrtimer_run_queues+0x1d3/0x360 [18142.077790] ipmi_devintf ipmi_msghandler [18142.080931] ? hrtimer_interrupt+0x104/0x240 [18142.085192] ioatdma acpi_pad [18142.087722] ? __local_bh_enable+0x50/0x60 [18142.092244] acpi_power_meter ixgbe [18142.096251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18142.100515] igb mdio [18142.103480] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18142.107568] crc32c_intel nvme [18142.111052] [18142.116269] ptp nvme_core [18142.118541] [18142.123499] i2c_algo_bit pps_core [18142.126551] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18142.128646] dca fuse [18142.131354] ? queued_spin_lock_slowpath+0x16/0x2e0 [18142.133448] [18142.133449] irq event stamp: 68015370 [18142.136847] do_raw_spin_lock+0xb5/0xc0 [18142.142150] hardirqs last enabled at (68015369): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18142.144420] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18142.149292] hardirqs last disabled at (68015370): [] sysvec_apic_timer_interrupt+0xe/0x90 [18142.150783] try_get_cap_refs+0x159/0x880 [ceph] [18142.154439] softirqs last enabled at (68015368): [] __do_softirq+0x2be/0x371 [18142.158274] __ceph_get_caps+0x4a1/0x760 [ceph] [18142.168428] softirqs last disabled at (68015363): [] irq_exit_rcu+0xb0/0xd0 [18142.173215] ? __file_remove_privs+0x89/0x110 [18142.182937] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18142.187548] ? wake_bit_function+0x70/0x70 [18142.196231] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18142.200756] ? ceph_write_iter+0x260/0x770 [ceph] [18142.209267] Workqueue: events delayed_work [ceph] [18142.213618] ceph_write_iter+0x260/0x770 [ceph] [18142.222993] [18142.222994] RIP: 0010:lock_release+0x14e/0x260 [18142.227087] ? lock_is_held_type+0xaa/0x120 [18142.234556] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [18142.239256] vfs_write+0x324/0x4f0 [18142.243949] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [18142.248480] ksys_pwrite64+0x65/0xa0 [18142.249966] [18142.249966] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [18142.254402] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18142.258579] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [18142.277320] do_syscall_64+0x38/0x80 [18142.280714] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [18142.285932] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18142.289503] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [18142.290994] RIP: 0033:0x7f48f0eeb93b [18142.298124] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [18142.303257] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18142.310380] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18142.313951] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18142.321077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18142.326124] ORIG_RAX: 0000000000000012 [18142.333244] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18142.336814] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18142.343940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18142.362677] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18142.370754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18142.375972] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18142.381709] Call Trace: [18142.381711] [18142.385541] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18142.392666] ? watchdog_timer_fn+0x1b4/0x220 [18142.399787] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18142.406913] ? __hrtimer_run_queues+0x1d3/0x360 [18142.414043] [18142.421165] ? hrtimer_interrupt+0x104/0x240 [18142.462223] ? __local_bh_enable+0x50/0x60 [18142.466319] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18142.471541] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18142.476499] [18142.478597] [18142.480694] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18142.485997] ? igrab+0x43/0x50 [18142.489051] ? lock_release+0x14e/0x260 [18142.492889] ? lock_release+0xc5/0x260 [18142.496634] _raw_spin_unlock+0x1b/0x40 [18142.500473] igrab+0x43/0x50 [18142.503350] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18142.508499] delayed_work+0x1d8/0x280 [ceph] [18142.512779] ? process_scheduled_works+0x22a/0x510 [18142.517562] process_scheduled_works+0x2a4/0x510 [18142.522176] worker_thread+0x188/0x340 [18142.525928] ? rescuer_thread+0x310/0x310 [18142.529938] kthread+0xf5/0x130 [18142.533075] ? kthread_complete_and_exit+0x20/0x20 [18142.537860] ret_from_fork+0x31/0x40 [18142.541430] ? kthread_complete_and_exit+0x20/0x20 [18142.546214] ret_from_fork_asm+0x11/0x20 [18142.550138] [18169.774232] watchdog: BUG: soft lockup - CPU#0 stuck for 5807s! [dbench:78683] [18169.781447] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18169.786232] watchdog: BUG: soft lockup - CPU#7 stuck for 14804s! [kworker/7:4:78808] [18169.836915] irq event stamp: 25815492 [18169.844646] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18169.848301] hardirqs last enabled at (25815491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18169.848307] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18169.865420] hardirqs last disabled at (25815492): [] sysvec_apic_timer_interrupt+0xe/0x90 [18169.865423] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [18169.883396] softirqs last enabled at (25815482): [] __do_softirq+0x2be/0x371 [18169.890094] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [18169.898779] softirqs last disabled at (25815477): [] irq_exit_rcu+0xb0/0xd0 [18169.907037] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [18169.915549] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18169.920681] iTCO_vendor_support i2c_smbus [18169.929630] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18169.929631] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18169.933722] crct10dif_pclmul [18169.941195] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18169.946671] joydev crc32_pclmul [18169.949636] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18169.968374] ipmi_ssif ghash_clmulni_intel [18169.971598] [18169.976813] mei_me mei [18169.980921] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18169.982412] lpc_ich sha512_ssse3 [18169.984858] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18169.991982] acpi_ipmi mfd_core [18169.995292] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18170.002416] wmi ipmi_si [18170.005554] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18170.012679] ipmi_devintf [18170.015208] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18170.022333] ipmi_msghandler ioatdma [18170.024950] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18170.032074] acpi_pad acpi_power_meter [18170.035644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18170.043723] ixgbe igb [18170.047466] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18170.053205] mdio [18170.055561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18170.062685] crc32c_intel nvme [18170.064627] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18170.071750] ptp nvme_core [18170.074802] Call Trace: [18170.074804] [18170.081927] i2c_algo_bit pps_core [18170.084632] ? watchdog_timer_fn+0x1b4/0x220 [18170.087073] dca fuse [18170.089086] ? __hrtimer_run_queues+0x1d3/0x360 [18170.092481] [18170.092482] irq event stamp: 68237732 [18170.096749] ? hrtimer_interrupt+0x104/0x240 [18170.099017] hardirqs last enabled at (68237731): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18170.103542] ? __local_bh_enable+0x50/0x60 [18170.105032] hardirqs last disabled at (68237732): [] sysvec_apic_timer_interrupt+0xe/0x90 [18170.108692] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18170.112953] softirqs last enabled at (68237730): [] __do_softirq+0x2be/0x371 [18170.123111] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18170.127200] softirqs last disabled at (68237725): [] irq_exit_rcu+0xb0/0xd0 [18170.136927] [18170.142144] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18170.150827] [18170.155783] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18170.164296] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18170.166392] Workqueue: events delayed_work [ceph] [18170.175775] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18170.177867] [18170.177868] RIP: 0010:__list_del_entry_valid_or_report+0x3f/0xd0 [18170.185340] do_raw_spin_lock+0xb5/0xc0 [18170.190642] Code: 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 <48> 39 f2 75 73 b8 01 00 00 00 c3 48 89 fe 48 c7 c7 98 99 1d 82 e8 [18170.195341] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18170.200211] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [18170.201705] try_get_cap_refs+0x159/0x880 [ceph] [18170.207700] [18170.207700] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [18170.211536] __ceph_get_caps+0x4a1/0x760 [ceph] [18170.230267] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [18170.235057] ? __file_remove_privs+0x89/0x110 [18170.240268] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [18170.244881] ? wake_bit_function+0x70/0x70 [18170.246371] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18170.253497] ? ceph_write_iter+0x260/0x770 [ceph] [18170.258018] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18170.265143] ceph_write_iter+0x260/0x770 [ceph] [18170.269494] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18170.276638] ? lock_is_held_type+0xaa/0x120 [18170.280724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18170.287852] vfs_write+0x324/0x4f0 [18170.292547] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18170.299675] ksys_pwrite64+0x65/0xa0 [18170.304195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18170.312273] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18170.316449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18170.322190] do_syscall_64+0x38/0x80 [18170.325585] Call Trace: [18170.332709] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18170.336279] [18170.343405] RIP: 0033:0x7f48f0eeb93b [18170.348534] ? watchdog_timer_fn+0x1b4/0x220 [18170.355660] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18170.359231] ? __hrtimer_run_queues+0x1d3/0x360 [18170.361674] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18170.366722] ? hrtimer_interrupt+0x104/0x240 [18170.368729] ORIG_RAX: 0000000000000012 [18170.372298] ? __local_bh_enable+0x50/0x60 [18170.376562] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18170.395305] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18170.399825] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18170.405044] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18170.409308] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18170.413137] [18170.417228] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18170.424352] [18170.424354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18170.429569] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18170.436698] ? __list_del_entry_valid_or_report+0x3f/0xd0 [18170.441656] [18170.448777] ceph_check_delayed_caps+0x8f/0x270 [ceph] [18170.485250] delayed_work+0x1d8/0x280 [ceph] [18170.489529] ? process_scheduled_works+0x22a/0x510 [18170.494320] process_scheduled_works+0x2a4/0x510 [18170.498935] worker_thread+0x188/0x340 [18170.502685] ? rescuer_thread+0x310/0x310 [18170.506687] kthread+0xf5/0x130 [18170.509824] ? kthread_complete_and_exit+0x20/0x20 [18170.514609] ret_from_fork+0x31/0x40 [18170.518207] ? kthread_complete_and_exit+0x20/0x20 [18170.522990] ret_from_fork_asm+0x11/0x20 [18170.526924] [18197.774743] watchdog: BUG: soft lockup - CPU#0 stuck for 5833s! [dbench:78683] [18197.781961] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18197.786743] watchdog: BUG: soft lockup - CPU#7 stuck for 14831s! [kworker/7:4:78808] [18197.837419] irq event stamp: 25873504 [18197.837420] hardirqs last enabled at (25873503): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18197.845150] Modules linked in: [18197.848808] hardirqs last disabled at (25873504): [] sysvec_apic_timer_interrupt+0xe/0x90 [18197.858965] ceph [18197.862016] softirqs last enabled at (25873456): [] __do_softirq+0x2be/0x371 [18197.871741] libceph [18197.873664] softirqs last disabled at (25873451): [] irq_exit_rcu+0xb0/0xd0 [18197.882348] dns_resolver fscache [18197.884531] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18197.893042] netfs veth [18197.896353] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18197.905297] nft_chain_nat xt_MASQUERADE [18197.907742] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18197.915212] nf_nat nf_conntrack [18197.919138] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18197.924616] nf_defrag_ipv6 nf_defrag_ipv4 [18197.927841] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18197.946577] nft_compat nf_tables [18197.950668] [18197.950669] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18197.955885] nfnetlink bridge [18197.959198] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18197.960687] stp llc [18197.967812] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18197.970776] xfs libcrc32c [18197.977899] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18197.980083] sunrpc [18197.987209] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18197.989912] intel_rapl_msr intel_rapl_common [18197.997037] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18197.999140] sb_edac x86_pkg_temp_thermal [18198.006266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18198.010618] coretemp kvm_intel [18198.018694] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18198.022699] kvm irqbypass [18198.028436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18198.031574] i2c_i801 iTCO_wdt [18198.038697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18198.041401] iTCO_vendor_support [18198.048526] Call Trace: [18198.048528] [18198.051578] i2c_smbus crct10dif_pclmul [18198.058703] ? watchdog_timer_fn+0x1b4/0x220 [18198.061925] joydev crc32_pclmul [18198.064371] ? __hrtimer_run_queues+0x1d3/0x360 [18198.066378] ipmi_ssif ghash_clmulni_intel [18198.070214] ? hrtimer_interrupt+0x104/0x240 [18198.074473] mei_me mei [18198.077699] ? __local_bh_enable+0x50/0x60 [18198.082223] lpc_ich sha512_ssse3 [18198.086317] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18198.090577] acpi_ipmi mfd_core [18198.093023] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18198.097111] wmi ipmi_si [18198.100434] [18198.105649] ipmi_devintf ipmi_msghandler [18198.108786] [18198.113745] ioatdma acpi_pad [18198.116275] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18198.118371] acpi_power_meter ixgbe [18198.122380] ? queued_spin_lock_slowpath+0x18/0x2e0 [18198.124473] igb mdio [18198.127439] do_raw_spin_lock+0xb5/0xc0 [18198.132741] crc32c_intel nvme [18198.136227] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18198.141097] ptp [18198.143370] try_get_cap_refs+0x159/0x880 [ceph] [18198.147197] nvme_core i2c_algo_bit [18198.150254] __ceph_get_caps+0x4a1/0x760 [ceph] [18198.155034] pps_core dca [18198.156875] ? __file_remove_privs+0x89/0x110 [18198.161482] fuse [18198.164965] ? wake_bit_function+0x70/0x70 [18198.169490] irq event stamp: 68460192 [18198.172109] ? ceph_write_iter+0x260/0x770 [ceph] [18198.176456] hardirqs last enabled at (68460191): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18198.178382] ceph_write_iter+0x260/0x770 [ceph] [18198.182473] hardirqs last disabled at (68460192): [] sysvec_apic_timer_interrupt+0xe/0x90 [18198.186146] ? lock_is_held_type+0xaa/0x120 [18198.190835] softirqs last enabled at (68460190): [] __do_softirq+0x2be/0x371 [18198.200996] vfs_write+0x324/0x4f0 [18198.205517] softirqs last disabled at (68460185): [] irq_exit_rcu+0xb0/0xd0 [18198.215245] ksys_pwrite64+0x65/0xa0 [18198.219426] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18198.228112] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18198.231508] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18198.240022] do_syscall_64+0x38/0x80 [18198.243590] Workqueue: events delayed_work [ceph] [18198.252970] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18198.258098] [18198.265569] RIP: 0033:0x7f48f0eeb93b [18198.269144] RIP: 0010:lock_acquire+0xf0/0x2b0 [18198.273838] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18198.278882] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18198.280372] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18198.283944] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18198.288294] ORIG_RAX: 0000000000000012 [18198.288296] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18198.307032] [18198.307033] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18198.325770] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18198.330986] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18198.336204] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18198.340034] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18198.347159] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18198.348649] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18198.355773] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18198.362897] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18198.370029] [18198.377149] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18198.423029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18198.428765] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18198.435890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18198.443014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18198.450150] Call Trace: [18198.452593] [18198.454603] ? watchdog_timer_fn+0x1b4/0x220 [18198.458866] ? __hrtimer_run_queues+0x1d3/0x360 [18198.463394] ? hrtimer_interrupt+0x104/0x240 [18198.467663] ? __local_bh_enable+0x50/0x60 [18198.471757] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18198.476979] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18198.481938] [18198.484034] [18198.486149] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18198.491458] ? lock_acquire+0xf0/0x2b0 [18198.495209] ? igrab+0x43/0x50 [18198.498256] ? lock_release+0xc5/0x260 [18198.502003] _raw_spin_lock+0x30/0x40 [18198.505667] ? igrab+0x1d/0x50 [18198.508719] igrab+0x1d/0x50 [18198.511598] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18198.516746] delayed_work+0x1d8/0x280 [ceph] [18198.521027] ? process_scheduled_works+0x22a/0x510 [18198.525818] process_scheduled_works+0x2a4/0x510 [18198.530434] worker_thread+0x188/0x340 [18198.534183] ? rescuer_thread+0x310/0x310 [18198.538186] kthread+0xf5/0x130 [18198.541322] ? kthread_complete_and_exit+0x20/0x20 [18198.546108] ret_from_fork+0x31/0x40 [18198.549687] ? kthread_complete_and_exit+0x20/0x20 [18198.554471] ret_from_fork_asm+0x11/0x20 [18198.558393] [18225.775254] watchdog: BUG: soft lockup - CPU#0 stuck for 5860s! [dbench:78683] [18225.782475] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18225.787254] watchdog: BUG: soft lockup - CPU#7 stuck for 14857s! [kworker/7:4:78808] [18225.837931] irq event stamp: 25931700 [18225.845664] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18225.849322] hardirqs last enabled at (25931699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18225.856282] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18225.866437] hardirqs last disabled at (25931700): [] sysvec_apic_timer_interrupt+0xe/0x90 [18225.874690] nft_compat nf_tables nfnetlink bridge stp llc xfs [18225.884413] softirqs last enabled at (25931652): [] __do_softirq+0x2be/0x371 [18225.884417] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [18225.898932] softirqs last disabled at (25931647): [] irq_exit_rcu+0xb0/0xd0 [18225.908931] kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support [18225.917445] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18225.923440] i2c_smbus crct10dif_pclmul joydev [18225.932386] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18225.932387] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18225.936821] crc32_pclmul ipmi_ssif [18225.944294] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18225.949769] ghash_clmulni_intel mei_me [18225.953253] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18225.971991] mei lpc_ich [18225.975824] [18225.981041] sha512_ssse3 acpi_ipmi [18225.983571] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18225.985062] mfd_core wmi [18225.988546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18225.995670] ipmi_si ipmi_devintf [18225.998286] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18226.005411] ipmi_msghandler ioatdma [18226.008721] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18226.015845] acpi_pad acpi_power_meter [18226.019417] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18226.026541] ixgbe igb [18226.030285] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18226.037408] mdio crc32c_intel [18226.039766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18226.047845] nvme ptp [18226.050894] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18226.056649] nvme_core [18226.058934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18226.066062] i2c_algo_bit pps_core [18226.068418] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18226.075544] dca fuse [18226.078943] Call Trace: [18226.078945] [18226.086065] [18226.086066] irq event stamp: 68682164 [18226.088337] ? watchdog_timer_fn+0x1b4/0x220 [18226.090778] hardirqs last enabled at (68682163): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18226.092792] ? __hrtimer_run_queues+0x1d3/0x360 [18226.094279] hardirqs last disabled at (68682164): [] sysvec_apic_timer_interrupt+0xe/0x90 [18226.097940] ? hrtimer_interrupt+0x104/0x240 [18226.102203] softirqs last enabled at (68682162): [] __do_softirq+0x2be/0x371 [18226.112361] ? __local_bh_enable+0x50/0x60 [18226.116882] softirqs last disabled at (68682157): [] irq_exit_rcu+0xb0/0xd0 [18226.126612] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18226.130872] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18226.139558] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18226.143654] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18226.152168] [18226.157386] Workqueue: events delayed_work [ceph] [18226.166762] [18226.166764] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18226.171719] [18226.171720] RIP: 0010:lock_acquire+0xf0/0x2b0 [18226.179193] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18226.181287] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18226.185987] do_raw_spin_lock+0xb5/0xc0 [18226.188081] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18226.193389] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18226.194876] [18226.194877] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18226.199231] try_get_cap_refs+0x159/0x880 [ceph] [18226.204098] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18226.222842] __ceph_get_caps+0x4a1/0x760 [ceph] [18226.226667] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18226.231888] ? __file_remove_privs+0x89/0x110 [18226.236668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18226.238159] ? wake_bit_function+0x70/0x70 [18226.245283] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18226.249897] ? ceph_write_iter+0x260/0x770 [ceph] [18226.257017] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18226.261542] ceph_write_iter+0x260/0x770 [ceph] [18226.268666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18226.273020] ? lock_is_held_type+0xaa/0x120 [18226.280140] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18226.284233] vfs_write+0x324/0x4f0 [18226.291356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18226.296057] ksys_pwrite64+0x65/0xa0 [18226.304131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18226.308659] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18226.314392] Call Trace: [18226.314394] [18226.318571] do_syscall_64+0x38/0x80 [18226.325694] ? watchdog_timer_fn+0x1b4/0x220 [18226.329093] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18226.336216] ? __hrtimer_run_queues+0x1d3/0x360 [18226.339788] RIP: 0033:0x7f48f0eeb93b [18226.346913] ? hrtimer_interrupt+0x104/0x240 [18226.352042] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18226.354485] ? __local_bh_enable+0x50/0x60 [18226.356495] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18226.360070] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18226.364329] ORIG_RAX: 0000000000000012 [18226.369376] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18226.373898] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18226.377470] [18226.381734] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18226.400470] [18226.400471] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18226.404562] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18226.409782] ? lock_acquire+0xf0/0x2b0 [18226.414997] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18226.418830] ? igrab+0x43/0x50 [18226.423783] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18226.430909] ? lock_release+0xc5/0x260 [18226.433011] [18226.440133] _raw_spin_lock+0x30/0x40 [18226.485285] ? igrab+0x1d/0x50 [18226.488335] igrab+0x1d/0x50 [18226.491216] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18226.496364] delayed_work+0x1d8/0x280 [ceph] [18226.500663] ? process_scheduled_works+0x22a/0x510 [18226.505454] process_scheduled_works+0x2a4/0x510 [18226.510068] worker_thread+0x188/0x340 [18226.513817] ? rescuer_thread+0x310/0x310 [18226.517821] kthread+0xf5/0x130 [18226.520967] ? kthread_complete_and_exit+0x20/0x20 [18226.525752] ret_from_fork+0x31/0x40 [18226.529330] ? kthread_complete_and_exit+0x20/0x20 [18226.534115] ret_from_fork_asm+0x11/0x20 [18226.538047] [18253.775766] watchdog: BUG: soft lockup - CPU#0 stuck for 5886s! [dbench:78683] [18253.782980] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18253.787766] watchdog: BUG: soft lockup - CPU#7 stuck for 14883s! [kworker/7:4:78808] [18253.838420] irq event stamp: 25989708 [18253.846162] Modules linked in: ceph libceph [18253.849818] hardirqs last enabled at (25989707): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18253.853994] dns_resolver fscache netfs [18253.864162] hardirqs last disabled at (25989708): [] sysvec_apic_timer_interrupt+0xe/0x90 [18253.867992] veth nft_chain_nat xt_MASQUERADE [18253.877716] softirqs last enabled at (25989660): [] __do_softirq+0x2be/0x371 [18253.877717] nf_nat [18253.877719] softirqs last disabled at (25989655): [] irq_exit_rcu+0xb0/0xd0 [18253.882066] nf_conntrack nf_defrag_ipv6 [18253.890750] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18253.892847] nf_defrag_ipv4 nft_compat [18253.901359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18253.905275] nf_tables nfnetlink [18253.914221] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18253.917964] bridge [18253.925451] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18253.928676] stp llc [18253.934162] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18253.936260] xfs libcrc32c [18253.954998] [18253.954999] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18253.957183] sunrpc intel_rapl_msr [18253.962398] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18253.965104] intel_rapl_common sb_edac [18253.966595] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18253.973718] x86_pkg_temp_thermal [18253.977117] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18253.984241] coretemp kvm_intel [18253.987984] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18253.995107] kvm irqbypass [18253.998420] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18254.005560] i2c_i801 iTCO_wdt [18254.008699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18254.015821] iTCO_vendor_support [18254.018524] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18254.026602] i2c_smbus crct10dif_pclmul [18254.029653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18254.035391] joydev [18254.038616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18254.045740] crc32_pclmul [18254.049570] Call Trace: [18254.049572] [18254.056694] ipmi_ssif ghash_clmulni_intel [18254.058793] ? watchdog_timer_fn+0x1b4/0x220 [18254.065915] mei_me mei [18254.068533] ? __hrtimer_run_queues+0x1d3/0x360 [18254.070977] lpc_ich sha512_ssse3 [18254.072992] ? hrtimer_interrupt+0x104/0x240 [18254.077080] acpi_ipmi mfd_core [18254.081343] ? __local_bh_enable+0x50/0x60 [18254.083787] wmi ipmi_si [18254.088314] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18254.091621] ipmi_devintf ipmi_msghandler [18254.095886] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18254.099024] ioatdma acpi_pad [18254.103114] [18254.105644] acpi_power_meter ixgbe [18254.110862] [18254.114866] igb mdio crc32c_intel [18254.119823] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18254.122787] nvme ptp [18254.124888] ? queued_spin_lock_slowpath+0x18/0x2e0 [18254.128369] nvme_core i2c_algo_bit [18254.130468] do_raw_spin_lock+0xb5/0xc0 [18254.133865] pps_core dca [18254.139168] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18254.141445] fuse [18254.146321] try_get_cap_refs+0x159/0x880 [ceph] [18254.149802] irq event stamp: 68904078 [18254.153637] __ceph_get_caps+0x4a1/0x760 [ceph] [18254.156251] hardirqs last enabled at (68904077): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18254.161039] ? __file_remove_privs+0x89/0x110 [18254.162957] hardirqs last disabled at (68904078): [] sysvec_apic_timer_interrupt+0xe/0x90 [18254.167569] ? wake_bit_function+0x70/0x70 [18254.171226] softirqs last enabled at (68904076): [] __do_softirq+0x2be/0x371 [18254.175753] ? ceph_write_iter+0x260/0x770 [ceph] [18254.185907] softirqs last disabled at (68904071): [] irq_exit_rcu+0xb0/0xd0 [18254.190258] ceph_write_iter+0x260/0x770 [ceph] [18254.199982] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18254.204077] ? lock_is_held_type+0xaa/0x120 [18254.212757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18254.212758] Workqueue: events delayed_work [ceph] [18254.217459] vfs_write+0x324/0x4f0 [18254.225966] [18254.225967] RIP: 0010:lock_acquire+0xf0/0x2b0 [18254.230494] ksys_pwrite64+0x65/0xa0 [18254.239867] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18254.244046] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18254.251515] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18254.256215] do_syscall_64+0x38/0x80 [18254.259612] [18254.259612] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18254.261102] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18254.265452] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18254.269024] RIP: 0033:0x7f48f0eeb93b [18254.287759] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18254.292890] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18254.298109] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18254.301680] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18254.303169] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18254.310295] ORIG_RAX: 0000000000000012 [18254.315339] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18254.322467] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18254.326032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18254.333169] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18254.351903] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18254.359028] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18254.364247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18254.371369] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18254.375199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18254.383278] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18254.390402] Call Trace: [18254.390404] [18254.396146] [18254.403266] ? watchdog_timer_fn+0x1b4/0x220 [18254.456913] ? __hrtimer_run_queues+0x1d3/0x360 [18254.461456] ? hrtimer_interrupt+0x104/0x240 [18254.465716] ? __local_bh_enable+0x50/0x60 [18254.469811] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18254.475033] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18254.479991] [18254.482087] [18254.484185] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18254.489494] ? lock_acquire+0xf0/0x2b0 [18254.493243] ? igrab+0x43/0x50 [18254.496295] ? lock_release+0xc5/0x260 [18254.500039] _raw_spin_lock+0x30/0x40 [18254.503695] ? igrab+0x1d/0x50 [18254.506745] igrab+0x1d/0x50 [18254.509624] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18254.514775] delayed_work+0x1d8/0x280 [ceph] [18254.519054] ? process_scheduled_works+0x22a/0x510 [18254.523846] process_scheduled_works+0x2a4/0x510 [18254.528468] worker_thread+0x188/0x340 [18254.532217] ? rescuer_thread+0x310/0x310 [18254.536221] kthread+0xf5/0x130 [18254.539360] ? kthread_complete_and_exit+0x20/0x20 [18254.544144] ret_from_fork+0x31/0x40 [18254.547732] ? kthread_complete_and_exit+0x20/0x20 [18254.552516] ret_from_fork_asm+0x11/0x20 [18254.556455] [18256.246178] rcu: INFO: rcu_preempt self-detected stall on CPU [18256.251932] rcu: 7-....: (6138286 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/6137051 fqs=3071687 [18256.262350] rcu: (t=6319523 jiffies g=1309609 q=246645 ncpus=8) [18256.268348] Sending NMI from CPU 7 to CPUs 0: [18256.272700] NMI backtrace for cpu 0 [18256.272701] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18256.272703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18256.272704] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18256.272707] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18256.272709] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18256.272710] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18256.272711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18256.272712] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18256.272713] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18256.272713] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18256.272714] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18256.272715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18256.272716] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18256.272717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18256.272718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18256.272718] Call Trace: [18256.272719] [18256.272720] ? nmi_cpu_backtrace+0xad/0x120 [18256.272724] ? nmi_cpu_backtrace_handler+0x11/0x20 [18256.272725] ? nmi_handle+0xd4/0x260 [18256.272730] ? default_do_nmi+0x49/0xf0 [18256.272731] ? exc_nmi+0x11e/0x150 [18256.272733] ? end_repeat_nmi+0x16/0x67 [18256.272737] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18256.272740] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18256.272742] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18256.272744] [18256.272745] [18256.272746] do_raw_spin_lock+0xb5/0xc0 [18256.272748] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18256.272763] try_get_cap_refs+0x159/0x880 [ceph] [18256.272777] __ceph_get_caps+0x4a1/0x760 [ceph] [18256.272790] ? __file_remove_privs+0x89/0x110 [18256.272792] ? wake_bit_function+0x70/0x70 [18256.272795] ? ceph_write_iter+0x260/0x770 [ceph] [18256.272806] ceph_write_iter+0x260/0x770 [ceph] [18256.272817] ? lock_is_held_type+0xaa/0x120 [18256.272820] vfs_write+0x324/0x4f0 [18256.272824] ksys_pwrite64+0x65/0xa0 [18256.272826] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18256.272828] do_syscall_64+0x38/0x80 [18256.272831] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18256.272833] RIP: 0033:0x7f48f0eeb93b [18256.272834] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18256.272836] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [18256.272837] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18256.272838] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18256.272839] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18256.272839] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18256.272840] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18256.272843] [18256.273698] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18256.577529] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18256.585000] Workqueue: events delayed_work [ceph] [18256.589714] RIP: 0010:lock_acquire+0xf5/0x2b0 [18256.594064] Code: b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 <65> 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 c4 38 5b 5d 41 [18256.612802] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18256.618019] RAX: c5e6ef7cc8fa9e00 RBX: 0000000000000200 RCX: 000000005553a634 [18256.625142] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18256.632267] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18256.639392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18256.646514] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18256.653640] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18256.661718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18256.667454] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18256.674579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18256.681702] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18256.688827] Call Trace: [18256.691273] [18256.693283] ? rcu_dump_cpu_stacks+0xd1/0x180 [18256.697642] ? rcu_sched_clock_irq+0x633/0x1390 [18256.702167] ? lock_release+0xc5/0x260 [18256.705911] ? lock_is_held_type+0xaa/0x120 [18256.710088] ? lock_is_held_type+0xaa/0x120 [18256.714266] ? tick_sched_do_timer+0xa0/0xa0 [18256.718536] ? rcu_read_lock_sched_held+0x56/0x90 [18256.723237] ? tick_sched_do_timer+0xa0/0xa0 [18256.727499] ? update_process_times+0x5f/0x90 [18256.731849] ? tick_sched_handle.isra.16+0x1e/0x50 [18256.736635] ? tick_sched_timer+0x6f/0x80 [18256.740638] ? __hrtimer_run_queues+0x1d3/0x360 [18256.745165] ? hrtimer_interrupt+0x104/0x240 [18256.749435] ? __local_bh_enable+0x50/0x60 [18256.753527] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18256.758751] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18256.763708] [18256.765807] [18256.767905] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18256.773212] ? lock_acquire+0xf5/0x2b0 [18256.776962] ? igrab+0x43/0x50 [18256.780012] ? lock_release+0xc5/0x260 [18256.783756] _raw_spin_lock+0x30/0x40 [18256.787422] ? igrab+0x1d/0x50 [18256.790471] igrab+0x1d/0x50 [18256.793349] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18256.798500] delayed_work+0x1d8/0x280 [ceph] [18256.802782] ? process_scheduled_works+0x22a/0x510 [18256.807570] process_scheduled_works+0x2a4/0x510 [18256.812186] worker_thread+0x188/0x340 [18256.815935] ? rescuer_thread+0x310/0x310 [18256.819938] kthread+0xf5/0x130 [18256.823078] ? kthread_complete_and_exit+0x20/0x20 [18256.827862] ret_from_fork+0x31/0x40 [18256.831440] ? kthread_complete_and_exit+0x20/0x20 [18256.836224] ret_from_fork_asm+0x11/0x20 [18256.840147] [18281.776278] watchdog: BUG: soft lockup - CPU#0 stuck for 5912s! [dbench:78683] [18281.783492] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18281.788278] watchdog: BUG: soft lockup - CPU#7 stuck for 14909s! [kworker/7:4:78808] [18281.838954] irq event stamp: 26047878 [18281.846682] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18281.850339] hardirqs last enabled at (26047877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18281.850345] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18281.867455] hardirqs last disabled at (26047878): [] sysvec_apic_timer_interrupt+0xe/0x90 [18281.867458] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [18281.885431] softirqs last enabled at (26047864): [] __do_softirq+0x2be/0x371 [18281.885435] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [18281.900814] softirqs last disabled at (26047859): [] irq_exit_rcu+0xb0/0xd0 [18281.910280] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [18281.918789] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18281.925307] crct10dif_pclmul joydev crc32_pclmul [18281.934253] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18281.934254] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18281.938961] ipmi_ssif ghash_clmulni_intel [18281.946429] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18281.951906] mei_me mei [18281.955996] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18281.974734] lpc_ich sha512_ssse3 [18281.977179] [18281.982395] acpi_ipmi mfd_core [18281.985705] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18281.987197] wmi [18281.990335] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18281.997458] ipmi_si [18281.999296] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18282.006419] ipmi_devintf [18282.008604] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18282.015727] ipmi_msghandler [18282.018345] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18282.025469] ioatdma acpi_pad [18282.028346] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18282.035471] acpi_power_meter ixgbe [18282.038435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18282.046512] igb [18282.049997] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18282.055734] mdio [18282.057572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18282.064695] crc32c_intel [18282.066620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18282.073742] nvme ptp [18282.076361] Call Trace: [18282.076363] [18282.083486] nvme_core i2c_algo_bit [18282.085757] ? watchdog_timer_fn+0x1b4/0x220 [18282.088199] pps_core [18282.090211] ? __hrtimer_run_queues+0x1d3/0x360 [18282.093694] dca fuse [18282.097975] ? hrtimer_interrupt+0x104/0x240 [18282.100239] [18282.100240] irq event stamp: 69120730 [18282.104762] ? __local_bh_enable+0x50/0x60 [18282.107034] hardirqs last enabled at (69120729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18282.111299] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18282.112789] hardirqs last disabled at (69120730): [] sysvec_apic_timer_interrupt+0xe/0x90 [18282.116445] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18282.120536] softirqs last enabled at (69120728): [] __do_softirq+0x2be/0x371 [18282.130694] [18282.135911] softirqs last disabled at (69120723): [] irq_exit_rcu+0xb0/0xd0 [18282.145635] [18282.145637] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18282.150593] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18282.159282] ? queued_spin_lock_slowpath+0x16/0x2e0 [18282.161374] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18282.169888] do_raw_spin_lock+0xb5/0xc0 [18282.171984] Workqueue: events delayed_work [ceph] [18282.177288] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18282.186678] [18282.191557] try_get_cap_refs+0x159/0x880 [ceph] [18282.199023] RIP: 0010:lock_acquire+0xf0/0x2b0 [18282.202858] __ceph_get_caps+0x4a1/0x760 [ceph] [18282.207551] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18282.212340] ? __file_remove_privs+0x89/0x110 [18282.213825] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18282.218438] ? wake_bit_function+0x70/0x70 [18282.222787] [18282.222788] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18282.227314] ? ceph_write_iter+0x260/0x770 [ceph] [18282.246049] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18282.250400] ceph_write_iter+0x260/0x770 [ceph] [18282.255617] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18282.259711] ? lock_is_held_type+0xaa/0x120 [18282.261198] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18282.268325] vfs_write+0x324/0x4f0 [18282.273020] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18282.280149] ksys_pwrite64+0x65/0xa0 [18282.284679] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18282.291801] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18282.295978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18282.303104] do_syscall_64+0x38/0x80 [18282.306499] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18282.313627] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18282.317194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18282.325273] RIP: 0033:0x7f48f0eeb93b [18282.330403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18282.336141] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18282.339710] Call Trace: [18282.339712] [18282.346836] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18282.351880] ? watchdog_timer_fn+0x1b4/0x220 [18282.359003] ORIG_RAX: 0000000000000012 [18282.362575] ? __hrtimer_run_queues+0x1d3/0x360 [18282.369697] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18282.388440] ? hrtimer_interrupt+0x104/0x240 [18282.390879] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18282.392890] ? __local_bh_enable+0x50/0x60 [18282.398109] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18282.402374] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18282.406204] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18282.410727] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18282.417852] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18282.422116] [18282.429244] [18282.433331] [18282.433333] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18282.476574] ? lock_acquire+0xf0/0x2b0 [18282.480323] ? igrab+0x43/0x50 [18282.483374] ? lock_release+0xc5/0x260 [18282.487119] _raw_spin_lock+0x30/0x40 [18282.490774] ? igrab+0x1d/0x50 [18282.493824] igrab+0x1d/0x50 [18282.496705] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18282.501855] delayed_work+0x1d8/0x280 [ceph] [18282.506134] ? process_scheduled_works+0x22a/0x510 [18282.510925] process_scheduled_works+0x2a4/0x510 [18282.515540] worker_thread+0x188/0x340 [18282.519289] ? rescuer_thread+0x310/0x310 [18282.523293] kthread+0xf5/0x130 [18282.526440] ? kthread_complete_and_exit+0x20/0x20 [18282.531223] ret_from_fork+0x31/0x40 [18282.534804] ? kthread_complete_and_exit+0x20/0x20 [18282.539586] ret_from_fork_asm+0x11/0x20 [18282.543510] [18309.776790] watchdog: BUG: soft lockup - CPU#0 stuck for 5938s! [dbench:78683] [18309.784006] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18309.788789] watchdog: BUG: soft lockup - CPU#7 stuck for 14935s! [kworker/7:4:78808] [18309.839463] irq event stamp: 26105964 [18309.839464] hardirqs last enabled at (26105963): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18309.847194] Modules linked in: [18309.850851] hardirqs last disabled at (26105964): [] sysvec_apic_timer_interrupt+0xe/0x90 [18309.861010] ceph [18309.864059] softirqs last enabled at (26105926): [] __do_softirq+0x2be/0x371 [18309.873785] libceph [18309.875708] softirqs last disabled at (26105921): [] irq_exit_rcu+0xb0/0xd0 [18309.884391] dns_resolver fscache [18309.886577] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18309.895086] netfs [18309.898397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18309.907341] veth nft_chain_nat [18309.909352] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18309.916823] xt_MASQUERADE [18309.919960] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18309.925439] nf_nat nf_conntrack [18309.928160] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18309.946914] nf_defrag_ipv6 nf_defrag_ipv4 [18309.950138] [18309.950139] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18309.955357] nft_compat nf_tables [18309.959464] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18309.960956] nfnetlink [18309.968079] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18309.971389] bridge stp [18309.978513] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18309.980870] llc xfs [18309.987994] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18309.990441] libcrc32c sunrpc [18309.997581] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18309.999764] intel_rapl_msr [18310.006889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18310.009852] intel_rapl_common sb_edac [18310.017932] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18310.020721] x86_pkg_temp_thermal coretemp [18310.026474] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18310.030221] kvm_intel kvm [18310.037346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18310.041435] irqbypass i2c_i801 [18310.048558] Call Trace: [18310.048560] [18310.051264] iTCO_wdt [18310.058388] ? watchdog_timer_fn+0x1b4/0x220 [18310.061526] iTCO_vendor_support i2c_smbus [18310.063970] ? __hrtimer_run_queues+0x1d3/0x360 [18310.065980] crct10dif_pclmul [18310.068254] ? hrtimer_interrupt+0x104/0x240 [18310.072515] joydev crc32_pclmul [18310.076605] ? __local_bh_enable+0x50/0x60 [18310.081129] ipmi_ssif ghash_clmulni_intel [18310.084096] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18310.088357] mei_me mei [18310.091581] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18310.095672] lpc_ich sha512_ssse3 [18310.099762] [18310.104979] acpi_ipmi mfd_core [18310.107425] [18310.107426] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18310.112382] wmi [18310.115697] ? queued_spin_lock_slowpath+0x18/0x2e0 [18310.117790] ipmi_si ipmi_devintf [18310.120928] do_raw_spin_lock+0xb5/0xc0 [18310.123025] ipmi_msghandler ioatdma [18310.128330] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18310.130167] acpi_pad acpi_power_meter [18310.135057] try_get_cap_refs+0x159/0x880 [ceph] [18310.138365] ixgbe igb [18310.142201] __ceph_get_caps+0x4a1/0x760 [ceph] [18310.145765] mdio crc32c_intel [18310.150554] ? __file_remove_privs+0x89/0x110 [18310.154293] nvme ptp [18310.158907] ? wake_bit_function+0x70/0x70 [18310.161261] nvme_core i2c_algo_bit [18310.165790] ? ceph_write_iter+0x260/0x770 [ceph] [18310.168836] pps_core dca [18310.173190] ceph_write_iter+0x260/0x770 [ceph] [18310.175471] fuse [18310.179563] ? lock_is_held_type+0xaa/0x120 [18310.183043] irq event stamp: 69342064 [18310.187743] vfs_write+0x324/0x4f0 [18310.190358] hardirqs last enabled at (69342063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18310.194886] ksys_pwrite64+0x65/0xa0 [18310.196806] hardirqs last disabled at (69342064): [] sysvec_apic_timer_interrupt+0xe/0x90 [18310.200983] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18310.204640] softirqs last enabled at (69342062): [] __do_softirq+0x2be/0x371 [18310.208038] do_syscall_64+0x38/0x80 [18310.218195] softirqs last disabled at (69342057): [] irq_exit_rcu+0xb0/0xd0 [18310.221766] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18310.231490] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18310.236621] RIP: 0033:0x7f48f0eeb93b [18310.245305] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18310.248876] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18310.257387] Workqueue: events delayed_work [ceph] [18310.262431] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18310.271809] [18310.271810] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [18310.275379] ORIG_RAX: 0000000000000012 [18310.282850] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [18310.301587] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18310.306285] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [18310.311502] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18310.312993] [18310.312994] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [18310.317604] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18310.321433] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [18310.340190] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18310.347313] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [18310.352532] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18310.359654] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18310.361151] [18310.368270] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18310.420324] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18310.428400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18310.434138] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18310.441261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18310.448386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18310.455511] Call Trace: [18310.457956] [18310.459966] ? watchdog_timer_fn+0x1b4/0x220 [18310.464231] ? __hrtimer_run_queues+0x1d3/0x360 [18310.468757] ? hrtimer_interrupt+0x104/0x240 [18310.473027] ? __local_bh_enable+0x50/0x60 [18310.477119] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18310.482342] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18310.487301] [18310.489398] [18310.491495] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18310.496804] ? do_raw_spin_lock+0x42/0xc0 [18310.500814] igrab+0x1d/0x50 [18310.503690] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18310.508840] delayed_work+0x1d8/0x280 [ceph] [18310.513120] ? process_scheduled_works+0x22a/0x510 [18310.517903] process_scheduled_works+0x2a4/0x510 [18310.522519] worker_thread+0x188/0x340 [18310.526268] ? rescuer_thread+0x310/0x310 [18310.530272] kthread+0xf5/0x130 [18310.533409] ? kthread_complete_and_exit+0x20/0x20 [18310.538197] ret_from_fork+0x31/0x40 [18310.541772] ? kthread_complete_and_exit+0x20/0x20 [18310.546557] ret_from_fork_asm+0x11/0x20 [18310.550485] [18337.777301] watchdog: BUG: soft lockup - CPU#0 stuck for 5964s! [dbench:78683] [18337.784520] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18337.789301] watchdog: BUG: soft lockup - CPU#7 stuck for 14961s! [kworker/7:4:78808] [18337.839977] irq event stamp: 26164260 [18337.847706] Modules linked in: ceph libceph dns_resolver fscache netfs [18337.851366] hardirqs last enabled at (26164259): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18337.851371] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [18337.868049] hardirqs last disabled at (26164260): [] sysvec_apic_timer_interrupt+0xe/0x90 [18337.868052] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [18337.885166] softirqs last enabled at (26164222): [] __do_softirq+0x2be/0x371 [18337.892290] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [18337.900978] softirqs last disabled at (26164217): [] irq_exit_rcu+0xb0/0xd0 [18337.909319] coretemp kvm_intel kvm irqbypass i2c_i801 [18337.917831] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18337.922977] iTCO_wdt iTCO_vendor_support i2c_smbus [18337.931923] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18337.931924] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18337.936792] crct10dif_pclmul [18337.944263] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18337.949741] joydev crc32_pclmul [18337.952707] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18337.971443] ipmi_ssif ghash_clmulni_intel [18337.974669] [18337.979885] mei_me mei [18337.983992] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18337.985484] lpc_ich [18337.987927] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18337.995052] sha512_ssse3 [18337.997237] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18338.004361] acpi_ipmi mfd_core [18338.006989] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18338.014109] wmi ipmi_si [18338.017248] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18338.024372] ipmi_devintf ipmi_msghandler [18338.026904] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18338.034026] ioatdma acpi_pad [18338.038031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18338.046108] acpi_power_meter ixgbe [18338.049073] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18338.054810] igb mdio [18338.058294] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18338.065419] crc32c_intel [18338.067699] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18338.074820] nvme ptp [18338.077439] Call Trace: [18338.077441] [18338.084563] nvme_core [18338.086836] ? watchdog_timer_fn+0x1b4/0x220 [18338.089278] i2c_algo_bit [18338.091289] ? __hrtimer_run_queues+0x1d3/0x360 [18338.093645] pps_core dca [18338.097913] ? hrtimer_interrupt+0x104/0x240 [18338.100527] fuse [18338.105052] ? __local_bh_enable+0x50/0x60 [18338.107670] [18338.107670] irq event stamp: 69563946 [18338.111935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18338.113857] hardirqs last enabled at (69563945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18338.117947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18338.119439] hardirqs last disabled at (69563946): [] sysvec_apic_timer_interrupt+0xe/0x90 [18338.123097] [18338.128314] softirqs last enabled at (69563944): [] __do_softirq+0x2be/0x371 [18338.138470] [18338.143429] softirqs last disabled at (69563939): [] irq_exit_rcu+0xb0/0xd0 [18338.153153] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18338.155248] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18338.163939] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18338.166032] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18338.174544] do_raw_spin_lock+0xb5/0xc0 [18338.179848] Workqueue: events delayed_work [ceph] [18338.189225] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18338.194093] [18338.194094] RIP: 0010:lock_acquire+0xf0/0x2b0 [18338.201567] try_get_cap_refs+0x159/0x880 [ceph] [18338.205395] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18338.210097] __ceph_get_caps+0x4a1/0x760 [ceph] [18338.214878] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18338.216372] ? __file_remove_privs+0x89/0x110 [18338.220720] [18338.220720] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18338.225330] ? wake_bit_function+0x70/0x70 [18338.244066] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18338.248594] ? ceph_write_iter+0x260/0x770 [ceph] [18338.253808] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18338.258159] ceph_write_iter+0x260/0x770 [ceph] [18338.259651] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18338.266778] ? lock_is_held_type+0xaa/0x120 [18338.270866] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18338.277997] vfs_write+0x324/0x4f0 [18338.282704] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18338.289832] ksys_pwrite64+0x65/0xa0 [18338.294352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18338.301478] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18338.305654] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18338.312780] do_syscall_64+0x38/0x80 [18338.316175] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18338.324255] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18338.327824] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18338.333563] RIP: 0033:0x7f48f0eeb93b [18338.338705] Call Trace: [18338.338707] [18338.345835] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18338.349405] ? watchdog_timer_fn+0x1b4/0x220 [18338.356529] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18338.361572] ? __hrtimer_run_queues+0x1d3/0x360 [18338.368703] ORIG_RAX: 0000000000000012 [18338.372279] ? hrtimer_interrupt+0x104/0x240 [18338.374720] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18338.376730] ? __local_bh_enable+0x50/0x60 [18338.395467] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18338.399736] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18338.404951] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18338.409475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18338.413304] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18338.417569] [18338.424706] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18338.428800] [18338.435931] [18338.441142] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18338.479159] ? lock_acquire+0xf0/0x2b0 [18338.482910] ? igrab+0x43/0x50 [18338.485959] ? lock_release+0xc5/0x260 [18338.489708] _raw_spin_lock+0x30/0x40 [18338.493369] ? igrab+0x1d/0x50 [18338.496422] igrab+0x1d/0x50 [18338.499299] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18338.504448] delayed_work+0x1d8/0x280 [ceph] [18338.508730] ? process_scheduled_works+0x22a/0x510 [18338.513518] process_scheduled_works+0x2a4/0x510 [18338.518136] worker_thread+0x188/0x340 [18338.521885] ? rescuer_thread+0x310/0x310 [18338.525897] kthread+0xf5/0x130 [18338.529043] ? kthread_complete_and_exit+0x20/0x20 [18338.533825] ret_from_fork+0x31/0x40 [18338.537397] ? kthread_complete_and_exit+0x20/0x20 [18338.542180] ret_from_fork_asm+0x11/0x20 [18338.546105] [18365.777813] watchdog: BUG: soft lockup - CPU#0 stuck for 5990s! [dbench:78683] [18365.785032] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18365.789813] watchdog: BUG: soft lockup - CPU#7 stuck for 14987s! [kworker/7:4:78808] [18365.840488] irq event stamp: 26222288 [18365.848221] Modules linked in: ceph libceph [18365.851878] hardirqs last enabled at (26222287): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18365.856056] dns_resolver fscache netfs [18365.866213] hardirqs last disabled at (26222288): [] sysvec_apic_timer_interrupt+0xe/0x90 [18365.870045] veth nft_chain_nat xt_MASQUERADE [18365.879768] softirqs last enabled at (26222250): [] __do_softirq+0x2be/0x371 [18365.879770] nf_nat [18365.879771] softirqs last disabled at (26222245): [] irq_exit_rcu+0xb0/0xd0 [18365.884119] nf_conntrack nf_defrag_ipv6 [18365.892803] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18365.894901] nf_defrag_ipv4 nft_compat [18365.903412] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18365.907327] nf_tables nfnetlink [18365.916273] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18365.920018] bridge stp [18365.927489] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18365.930712] llc xfs [18365.936205] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18365.938650] libcrc32c sunrpc [18365.957389] [18365.957390] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18365.959571] intel_rapl_msr intel_rapl_common [18365.964789] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18365.967753] sb_edac [18365.969244] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18365.976368] x86_pkg_temp_thermal coretemp [18365.980718] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18365.987843] kvm_intel kvm [18365.990026] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18365.997151] irqbypass i2c_i801 [18366.001242] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18366.008365] iTCO_wdt [18366.011070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18366.018209] iTCO_vendor_support i2c_smbus [18366.021350] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18366.029427] crct10dif_pclmul joydev [18366.031697] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18366.037436] crc32_pclmul ipmi_ssif [18366.041525] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18366.048650] ghash_clmulni_intel mei_me [18366.052219] Call Trace: [18366.052221] [18366.059345] mei lpc_ich [18366.062828] ? watchdog_timer_fn+0x1b4/0x220 [18366.069952] sha512_ssse3 acpi_ipmi [18366.073785] ? __hrtimer_run_queues+0x1d3/0x360 [18366.076227] mfd_core wmi [18366.078241] ? hrtimer_interrupt+0x104/0x240 [18366.080770] ipmi_si ipmi_devintf [18366.085032] ? __local_bh_enable+0x50/0x60 [18366.088518] ipmi_msghandler ioatdma [18366.093043] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18366.095659] acpi_pad acpi_power_meter [18366.099922] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18366.103233] ixgbe igb [18366.107324] [18366.110895] mdio crc32c_intel [18366.116112] [18366.116113] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18366.119857] nvme ptp [18366.124818] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18366.127172] nvme_core i2c_algo_bit [18366.129271] do_raw_spin_lock+0xb5/0xc0 [18366.132319] pps_core dca [18366.134416] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18366.139719] fuse [18366.141993] try_get_cap_refs+0x159/0x880 [ceph] [18366.146861] irq event stamp: 69785282 [18366.150350] __ceph_get_caps+0x4a1/0x760 [ceph] [18366.154177] hardirqs last enabled at (69785281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18366.156797] ? __file_remove_privs+0x89/0x110 [18366.161578] hardirqs last disabled at (69785282): [] sysvec_apic_timer_interrupt+0xe/0x90 [18366.163503] ? wake_bit_function+0x70/0x70 [18366.168114] softirqs last enabled at (69785280): [] __do_softirq+0x2be/0x371 [18366.171773] ? ceph_write_iter+0x260/0x770 [ceph] [18366.176295] softirqs last disabled at (69785275): [] irq_exit_rcu+0xb0/0xd0 [18366.186452] ceph_write_iter+0x260/0x770 [ceph] [18366.190803] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18366.200531] ? lock_is_held_type+0xaa/0x120 [18366.204618] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18366.213304] vfs_write+0x324/0x4f0 [18366.218000] Workqueue: events delayed_work [ceph] [18366.226514] ksys_pwrite64+0x65/0xa0 [18366.231034] [18366.231035] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [18366.240412] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18366.244590] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [18366.252062] do_syscall_64+0x38/0x80 [18366.255456] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [18366.260157] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18366.263725] [18366.263725] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [18366.265217] RIP: 0033:0x7f48f0eeb93b [18366.271216] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [18366.276344] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18366.295083] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [18366.298652] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18366.303871] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18366.308913] ORIG_RAX: 0000000000000012 [18366.310406] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18366.317528] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18366.321100] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18366.328223] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18366.346963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18366.354086] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18366.359304] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18366.366428] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18366.370258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18366.377382] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18366.384507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18366.392589] [18366.399708] Call Trace: [18366.399711] [18366.454831] ? watchdog_timer_fn+0x1b4/0x220 [18366.459094] ? __hrtimer_run_queues+0x1d3/0x360 [18366.463622] ? hrtimer_interrupt+0x104/0x240 [18366.467891] ? __local_bh_enable+0x50/0x60 [18366.471984] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18366.477221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18366.482182] [18366.484279] [18366.486376] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18366.491684] ? __list_del_entry_valid_or_report+0x10/0xd0 [18366.497081] ceph_check_delayed_caps+0x8f/0x270 [ceph] [18366.502231] delayed_work+0x1d8/0x280 [ceph] [18366.506512] ? process_scheduled_works+0x22a/0x510 [18366.511302] process_scheduled_works+0x2a4/0x510 [18366.515918] worker_thread+0x188/0x340 [18366.519667] ? rescuer_thread+0x310/0x310 [18366.523671] kthread+0xf5/0x130 [18366.526807] ? kthread_complete_and_exit+0x20/0x20 [18366.531593] ret_from_fork+0x31/0x40 [18366.535163] ? kthread_complete_and_exit+0x20/0x20 [18366.539948] ret_from_fork_asm+0x11/0x20 [18366.543878] [18393.778324] watchdog: BUG: soft lockup - CPU#0 stuck for 6016s! [dbench:78683] [18393.785544] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18393.790324] watchdog: BUG: soft lockup - CPU#7 stuck for 15013s! [kworker/7:4:78808] [18393.841000] irq event stamp: 26280430 [18393.841002] hardirqs last enabled at (26280429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18393.848734] Modules linked in: [18393.852393] hardirqs last disabled at (26280430): [] sysvec_apic_timer_interrupt+0xe/0x90 [18393.862549] ceph [18393.865600] softirqs last enabled at (26280392): [] __do_softirq+0x2be/0x371 [18393.875325] libceph [18393.877248] softirqs last disabled at (26280387): [] irq_exit_rcu+0xb0/0xd0 [18393.885933] dns_resolver fscache [18393.888118] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18393.896627] netfs veth [18393.899938] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18393.908881] nft_chain_nat [18393.911326] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18393.918796] xt_MASQUERADE nf_nat [18393.921502] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18393.926994] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18393.930305] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18393.949044] nft_compat nf_tables [18393.954262] [18393.959478] nfnetlink bridge [18393.962788] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18393.964279] stp llc [18393.967245] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18393.974369] xfs libcrc32c [18393.976551] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18393.983677] sunrpc intel_rapl_msr [18393.986381] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18393.993505] intel_rapl_common [18393.996902] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18394.004026] sb_edac x86_pkg_temp_thermal [18394.007076] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18394.014200] coretemp kvm_intel [18394.018206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18394.026281] kvm irqbypass [18394.029419] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18394.035158] i2c_i801 iTCO_wdt [18394.037863] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18394.044986] iTCO_vendor_support [18394.048035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18394.055160] i2c_smbus crct10dif_pclmul [18394.058385] Call Trace: [18394.065507] joydev crc32_pclmul [18394.069343] [18394.071783] ipmi_ssif ghash_clmulni_intel [18394.075011] ? watchdog_timer_fn+0x1b4/0x220 [18394.077018] mei_me mei [18394.081111] ? __hrtimer_run_queues+0x1d3/0x360 [18394.085374] lpc_ich sha512_ssse3 [18394.087821] ? hrtimer_interrupt+0x104/0x240 [18394.092342] acpi_ipmi mfd_core [18394.095652] ? __local_bh_enable+0x50/0x60 [18394.099917] wmi ipmi_si [18394.103055] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18394.107143] ipmi_devintf ipmi_msghandler [18394.109676] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18394.114891] ioatdma acpi_pad [18394.118896] [18394.123854] acpi_power_meter ixgbe [18394.126818] [18394.126820] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18394.128915] igb [18394.132404] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18394.134496] mdio crc32c_intel [18394.139803] do_raw_spin_lock+0xb5/0xc0 [18394.141638] nvme ptp [18394.146511] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18394.149559] nvme_core i2c_algo_bit [18394.153394] try_get_cap_refs+0x159/0x880 [ceph] [18394.155660] pps_core [18394.160451] __ceph_get_caps+0x4a1/0x760 [ceph] [18394.163929] dca fuse [18394.168545] ? __file_remove_privs+0x89/0x110 [18394.170810] [18394.170811] irq event stamp: 70007010 [18394.175335] ? wake_bit_function+0x70/0x70 [18394.177604] hardirqs last enabled at (70007009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18394.181959] ? ceph_write_iter+0x260/0x770 [ceph] [18394.183448] hardirqs last disabled at (70007010): [] sysvec_apic_timer_interrupt+0xe/0x90 [18394.187105] ceph_write_iter+0x260/0x770 [ceph] [18394.191194] softirqs last enabled at (70007008): [] __do_softirq+0x2be/0x371 [18394.201357] ? lock_is_held_type+0xaa/0x120 [18394.206050] softirqs last disabled at (70007003): [] irq_exit_rcu+0xb0/0xd0 [18394.215778] vfs_write+0x324/0x4f0 [18394.220297] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18394.229004] ksys_pwrite64+0x65/0xa0 [18394.233178] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18394.233179] Workqueue: events delayed_work [ceph] [18394.241687] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18394.245084] [18394.245085] RIP: 0010:lock_acquire+0xf0/0x2b0 [18394.254463] do_syscall_64+0x38/0x80 [18394.258032] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18394.265507] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18394.270203] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18394.275334] RIP: 0033:0x7f48f0eeb93b [18394.276823] [18394.276824] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18394.281175] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18394.284745] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18394.303483] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18394.308526] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18394.313744] ORIG_RAX: 0000000000000012 [18394.317314] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18394.318806] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18394.325930] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18394.344668] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18394.351790] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18394.357009] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18394.364133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18394.367963] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18394.375088] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18394.382213] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18394.389337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18394.396466] [18394.404539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18394.455205] Call Trace: [18394.457649] [18394.459659] ? watchdog_timer_fn+0x1b4/0x220 [18394.463924] ? __hrtimer_run_queues+0x1d3/0x360 [18394.468451] ? hrtimer_interrupt+0x104/0x240 [18394.472730] ? __local_bh_enable+0x50/0x60 [18394.476831] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18394.482054] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18394.487012] [18394.489108] [18394.491206] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18394.496515] ? lock_acquire+0xf0/0x2b0 [18394.500265] ? igrab+0x43/0x50 [18394.503314] ? lock_release+0xc5/0x260 [18394.507059] _raw_spin_lock+0x30/0x40 [18394.510732] ? igrab+0x1d/0x50 [18394.513784] igrab+0x1d/0x50 [18394.516663] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18394.521814] delayed_work+0x1d8/0x280 [ceph] [18394.526093] ? process_scheduled_works+0x22a/0x510 [18394.530882] process_scheduled_works+0x2a4/0x510 [18394.535500] worker_thread+0x188/0x340 [18394.539249] ? rescuer_thread+0x310/0x310 [18394.543258] kthread+0xf5/0x130 [18394.546398] ? kthread_complete_and_exit+0x20/0x20 [18394.551182] ret_from_fork+0x31/0x40 [18394.554761] ? kthread_complete_and_exit+0x20/0x20 [18394.559544] ret_from_fork_asm+0x11/0x20 [18394.563468] [18421.778834] watchdog: BUG: soft lockup - CPU#0 stuck for 6042s! [dbench:78683] [18421.786049] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18421.790834] watchdog: BUG: soft lockup - CPU#7 stuck for 15039s! [kworker/7:4:78808] [18421.841490] irq event stamp: 26338518 [18421.841492] hardirqs last enabled at (26338517): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18421.849238] Modules linked in: ceph [18421.852897] hardirqs last disabled at (26338518): [] sysvec_apic_timer_interrupt+0xe/0x90 [18421.863053] libceph [18421.866538] softirqs last enabled at (26338480): [] __do_softirq+0x2be/0x371 [18421.876261] dns_resolver [18421.878446] softirqs last disabled at (26338475): [] irq_exit_rcu+0xb0/0xd0 [18421.887130] fscache netfs [18421.889747] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18421.898259] veth nft_chain_nat [18421.900962] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18421.909906] xt_MASQUERADE nf_nat [18421.913045] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18421.920516] nf_conntrack nf_defrag_ipv6 [18421.923826] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18421.929303] nf_defrag_ipv4 nft_compat [18421.933231] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18421.951966] nf_tables nfnetlink [18421.955710] [18421.955711] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18421.960928] bridge stp [18421.964151] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18421.965642] llc xfs [18421.972768] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18421.975228] libcrc32c [18421.982354] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18421.984536] sunrpc intel_rapl_msr [18421.991661] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18421.994017] intel_rapl_common [18422.001142] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18422.004541] sb_edac x86_pkg_temp_thermal [18422.011663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18422.014714] coretemp kvm_intel [18422.022793] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18422.026797] kvm irqbypass [18422.032533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18422.035671] i2c_i801 iTCO_wdt [18422.042796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18422.045515] iTCO_vendor_support i2c_smbus [18422.052639] Call Trace: [18422.052641] [18422.055691] crct10dif_pclmul joydev [18422.062815] ? watchdog_timer_fn+0x1b4/0x220 [18422.066907] crc32_pclmul ipmi_ssif [18422.069353] ? __hrtimer_run_queues+0x1d3/0x360 [18422.071360] ghash_clmulni_intel mei_me [18422.074936] ? hrtimer_interrupt+0x104/0x240 [18422.079196] mei lpc_ich [18422.082680] ? __local_bh_enable+0x50/0x60 [18422.087203] sha512_ssse3 acpi_ipmi [18422.091038] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18422.095297] mfd_core [18422.097831] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18422.101920] wmi ipmi_si [18422.105406] [18422.110622] ipmi_devintf ipmi_msghandler [18422.112894] [18422.117851] ioatdma acpi_pad [18422.120382] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18422.122478] acpi_power_meter ixgbe [18422.126487] ? queued_spin_lock_slowpath+0x16/0x2e0 [18422.128578] igb mdio [18422.131547] do_raw_spin_lock+0xb5/0xc0 [18422.136847] crc32c_intel nvme [18422.140334] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18422.145202] ptp nvme_core [18422.147474] try_get_cap_refs+0x159/0x880 [ceph] [18422.151304] i2c_algo_bit pps_core [18422.154360] __ceph_get_caps+0x4a1/0x760 [ceph] [18422.159140] dca fuse [18422.161846] ? __file_remove_privs+0x89/0x110 [18422.166452] [18422.166453] irq event stamp: 70228582 [18422.169852] ? wake_bit_function+0x70/0x70 [18422.174375] hardirqs last enabled at (70228581): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18422.176647] ? ceph_write_iter+0x260/0x770 [ceph] [18422.180996] hardirqs last disabled at (70228582): [] sysvec_apic_timer_interrupt+0xe/0x90 [18422.182487] ceph_write_iter+0x260/0x770 [ceph] [18422.186144] softirqs last enabled at (70228580): [] __do_softirq+0x2be/0x371 [18422.190242] ? lock_is_held_type+0xaa/0x120 [18422.200391] softirqs last disabled at (70228575): [] irq_exit_rcu+0xb0/0xd0 [18422.205093] vfs_write+0x324/0x4f0 [18422.214813] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18422.219343] ksys_pwrite64+0x65/0xa0 [18422.228023] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18422.232201] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18422.240712] Workqueue: events delayed_work [ceph] [18422.244110] do_syscall_64+0x38/0x80 [18422.253487] [18422.257057] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18422.264528] RIP: 0010:lock_acquire+0xf0/0x2b0 [18422.269659] RIP: 0033:0x7f48f0eeb93b [18422.274357] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18422.277928] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18422.279417] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18422.284462] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18422.288813] [18422.292383] ORIG_RAX: 0000000000000012 [18422.311119] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18422.329857] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18422.335076] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18422.340293] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18422.341782] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18422.345613] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18422.352739] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18422.359863] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18422.366987] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18422.374110] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18422.381239] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18422.388374] [18422.395491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18422.432865] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18422.439987] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18422.447110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18422.454239] Call Trace: [18422.456680] [18422.458692] ? watchdog_timer_fn+0x1b4/0x220 [18422.462955] ? __hrtimer_run_queues+0x1d3/0x360 [18422.467481] ? hrtimer_interrupt+0x104/0x240 [18422.471742] ? __local_bh_enable+0x50/0x60 [18422.475836] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18422.481059] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18422.486018] [18422.488113] [18422.490214] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18422.495547] ? lock_acquire+0xf0/0x2b0 [18422.499297] ? igrab+0x43/0x50 [18422.502345] ? lock_release+0xc5/0x260 [18422.506091] _raw_spin_lock+0x30/0x40 [18422.509748] ? igrab+0x1d/0x50 [18422.512798] igrab+0x1d/0x50 [18422.515677] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18422.520824] delayed_work+0x1d8/0x280 [ceph] [18422.525105] ? process_scheduled_works+0x22a/0x510 [18422.529889] process_scheduled_works+0x2a4/0x510 [18422.534519] worker_thread+0x188/0x340 [18422.538272] ? rescuer_thread+0x310/0x310 [18422.542282] kthread+0xf5/0x130 [18422.545420] ? kthread_complete_and_exit+0x20/0x20 [18422.550205] ret_from_fork+0x31/0x40 [18422.553773] ? kthread_complete_and_exit+0x20/0x20 [18422.558560] ret_from_fork_asm+0x11/0x20 [18422.562481] [18449.779344] watchdog: BUG: soft lockup - CPU#0 stuck for 6068s! [dbench:78683] [18449.786564] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18449.791344] watchdog: BUG: soft lockup - CPU#7 stuck for 15065s! [kworker/7:4:78808] [18449.842029] irq event stamp: 26396598 [18449.849762] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18449.853420] hardirqs last enabled at (26396597): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18449.853425] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18449.870536] hardirqs last disabled at (26396598): [] sysvec_apic_timer_interrupt+0xe/0x90 [18449.870539] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [18449.888510] softirqs last enabled at (26396560): [] __do_softirq+0x2be/0x371 [18449.895203] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [18449.903885] softirqs last disabled at (26396555): [] irq_exit_rcu+0xb0/0xd0 [18449.903889] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support [18449.920650] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18449.927510] i2c_smbus crct10dif_pclmul [18449.936457] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18449.940287] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [18449.947759] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18449.947761] mei_me mei lpc_ich sha512_ssse3 acpi_ipmi [18449.959059] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18449.964190] mfd_core wmi ipmi_si [18449.982928] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18449.986237] ipmi_devintf ipmi_msghandler [18449.991456] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18449.991457] ioatdma [18449.991457] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18449.995460] acpi_pad acpi_power_meter [18450.002585] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18450.004766] ixgbe [18450.011892] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18450.015636] igb mdio [18450.022761] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18450.024770] crc32c_intel [18450.031896] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18450.034166] nvme ptp [18450.041288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18450.043908] nvme_core i2c_algo_bit [18450.051985] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18450.054254] pps_core [18450.059994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18450.063477] dca fuse [18450.070600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18450.072872] [18450.072873] irq event stamp: 70449866 [18450.079994] Call Trace: [18450.079997] [18450.082267] hardirqs last enabled at (70449865): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18450.089392] ? watchdog_timer_fn+0x1b4/0x220 [18450.090880] hardirqs last disabled at (70449866): [] sysvec_apic_timer_interrupt+0xe/0x90 [18450.094540] ? __hrtimer_run_queues+0x1d3/0x360 [18450.096983] softirqs last enabled at (70449864): [] __do_softirq+0x2be/0x371 [18450.098997] ? hrtimer_interrupt+0x104/0x240 [18450.109151] softirqs last disabled at (70449859): [] irq_exit_rcu+0xb0/0xd0 [18450.113414] ? __local_bh_enable+0x50/0x60 [18450.123140] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18450.127665] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18450.136346] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18450.140611] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18450.149122] Workqueue: events delayed_work [ceph] [18450.153213] [18450.162590] [18450.167809] [18450.167810] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18450.175279] RIP: 0010:lock_acquire+0xf0/0x2b0 [18450.180241] ? queued_spin_lock_slowpath+0x18/0x2e0 [18450.184933] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18450.187034] do_raw_spin_lock+0xb5/0xc0 [18450.188522] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18450.190622] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18450.195924] [18450.195925] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18450.200276] try_get_cap_refs+0x159/0x880 [ceph] [18450.205145] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18450.223889] __ceph_get_caps+0x4a1/0x760 [ceph] [18450.227713] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18450.232936] ? __file_remove_privs+0x89/0x110 [18450.237715] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18450.239206] ? wake_bit_function+0x70/0x70 [18450.246329] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18450.250943] ? ceph_write_iter+0x260/0x770 [ceph] [18450.258063] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18450.262588] ceph_write_iter+0x260/0x770 [ceph] [18450.269714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18450.274069] ? lock_is_held_type+0xaa/0x120 [18450.281186] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18450.285281] vfs_write+0x324/0x4f0 [18450.292401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18450.297103] ksys_pwrite64+0x65/0xa0 [18450.305177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18450.309701] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18450.315438] Call Trace: [18450.315440] [18450.319619] do_syscall_64+0x38/0x80 [18450.326747] ? watchdog_timer_fn+0x1b4/0x220 [18450.330148] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18450.337271] ? __hrtimer_run_queues+0x1d3/0x360 [18450.340841] RIP: 0033:0x7f48f0eeb93b [18450.347969] ? hrtimer_interrupt+0x104/0x240 [18450.353098] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18450.355539] ? __local_bh_enable+0x50/0x60 [18450.357552] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18450.361124] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18450.365386] ORIG_RAX: 0000000000000012 [18450.370430] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18450.374955] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18450.378526] [18450.382789] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18450.401527] [18450.401529] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18450.405617] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18450.410838] ? lock_acquire+0xf0/0x2b0 [18450.416053] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18450.419885] ? igrab+0x43/0x50 [18450.424840] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18450.431964] ? lock_release+0xc5/0x260 [18450.434067] [18450.441187] _raw_spin_lock+0x30/0x40 [18450.486340] ? igrab+0x1d/0x50 [18450.489392] igrab+0x1d/0x50 [18450.492272] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18450.497421] delayed_work+0x1d8/0x280 [ceph] [18450.501702] ? process_scheduled_works+0x22a/0x510 [18450.506490] process_scheduled_works+0x2a4/0x510 [18450.511107] worker_thread+0x188/0x340 [18450.514856] ? rescuer_thread+0x310/0x310 [18450.518857] kthread+0xf5/0x130 [18450.521996] ? kthread_complete_and_exit+0x20/0x20 [18450.526781] ret_from_fork+0x31/0x40 [18450.530360] ? kthread_complete_and_exit+0x20/0x20 [18450.535144] ret_from_fork_asm+0x11/0x20 [18450.539067] [18451.847747] rcu: INFO: rcu_preempt self-detected stall on CPU [18451.853507] rcu: 0-....: (6369635 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=3167587 [18451.863844] rcu: (t=6515122 jiffies g=1309609 q=251662 ncpus=8) [18451.869844] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18451.878788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18451.886257] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18451.891736] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18451.910473] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18451.915689] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18451.922813] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18451.929939] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18451.937063] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18451.944186] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18451.951311] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18451.959388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18451.965127] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18451.972249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18451.979374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18451.986498] Call Trace: [18451.988944] [18451.990954] ? rcu_dump_cpu_stacks+0xd1/0x180 [18451.995314] ? rcu_sched_clock_irq+0x633/0x1390 [18451.999846] ? lock_is_held_type+0xaa/0x120 [18452.004024] ? lock_is_held_type+0xaa/0x120 [18452.008202] ? tick_sched_do_timer+0xa0/0xa0 [18452.012473] ? rcu_read_lock_sched_held+0x56/0x90 [18452.017171] ? tick_sched_do_timer+0xa0/0xa0 [18452.021433] ? update_process_times+0x5f/0x90 [18452.025785] ? tick_sched_handle.isra.16+0x1e/0x50 [18452.030571] ? tick_sched_timer+0x6f/0x80 [18452.034575] ? __hrtimer_run_queues+0x1d3/0x360 [18452.039100] ? hrtimer_interrupt+0x104/0x240 [18452.043362] ? __local_bh_enable+0x50/0x60 [18452.047456] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18452.052680] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18452.057636] [18452.059733] [18452.061830] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18452.067138] ? queued_spin_lock_slowpath+0x18/0x2e0 [18452.072016] do_raw_spin_lock+0xb5/0xc0 [18452.075845] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18452.080649] try_get_cap_refs+0x159/0x880 [ceph] [18452.085279] __ceph_get_caps+0x4a1/0x760 [ceph] [18452.089819] ? __file_remove_privs+0x89/0x110 [18452.094175] ? wake_bit_function+0x70/0x70 [18452.098269] ? ceph_write_iter+0x260/0x770 [ceph] [18452.102980] ceph_write_iter+0x260/0x770 [ceph] [18452.107516] ? lock_is_held_type+0xaa/0x120 [18452.111701] vfs_write+0x324/0x4f0 [18452.115102] ksys_pwrite64+0x65/0xa0 [18452.118677] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18452.123809] do_syscall_64+0x38/0x80 [18452.127379] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18452.132421] RIP: 0033:0x7f48f0eeb93b [18452.135994] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18452.154729] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [18452.162288] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18452.169412] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18452.176537] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18452.183660] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18452.190783] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18452.197914] [18452.200102] Sending NMI from CPU 0 to CPUs 7: [18452.204454] NMI backtrace for cpu 7 [18452.204456] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18452.204458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18452.204458] Workqueue: events delayed_work [ceph] [18452.204474] RIP: 0010:__lock_acquire+0x6e8/0x1e20 [18452.204476] Code: 0a 00 00 4c 89 bb 48 0a 00 00 83 c0 01 89 83 50 0a 00 00 83 f8 2f 77 46 3b 05 98 71 e0 01 0f 87 b6 04 00 00 41 be 01 00 00 00 85 00 00 00 48 81 3f c0 50 d7 82 41 bb 00 00 00 00 45 0f 45 d8 [18452.204477] RSP: 0018:ffffc90002117c60 EFLAGS: 00000087 [18452.204478] RAX: 0000000000000004 RBX: ffff88810debaa80 RCX: 000000005553a634 [18452.204479] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18452.204480] RBP: ffff88810debb550 R08: 0000000000005c53 R09: 0000000000000003 [18452.204480] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8360d858 [18452.204481] R13: 0000000000000003 R14: 0000000000000001 R15: c5dbdfc36fd10127 [18452.204481] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18452.204482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18452.204483] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18452.204484] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18452.204485] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18452.204485] Call Trace: [18452.204486] [18452.204487] ? nmi_cpu_backtrace+0xad/0x120 [18452.204491] ? nmi_cpu_backtrace_handler+0x11/0x20 [18452.204492] ? nmi_handle+0xd4/0x260 [18452.204497] ? default_do_nmi+0x49/0xf0 [18452.204498] ? exc_nmi+0x11e/0x150 [18452.204499] ? end_repeat_nmi+0x16/0x67 [18452.204503] ? __lock_acquire+0x6e8/0x1e20 [18452.204505] ? __lock_acquire+0x6e8/0x1e20 [18452.204506] ? __lock_acquire+0x6e8/0x1e20 [18452.204508] [18452.204508] [18452.204510] lock_acquire+0xcb/0x2b0 [18452.204511] ? igrab+0x1d/0x50 [18452.204513] ? igrab+0x43/0x50 [18452.204514] ? lock_release+0xc5/0x260 [18452.204516] _raw_spin_lock+0x30/0x40 [18452.204518] ? igrab+0x1d/0x50 [18452.204519] igrab+0x1d/0x50 [18452.204520] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18452.204534] delayed_work+0x1d8/0x280 [ceph] [18452.204547] ? process_scheduled_works+0x22a/0x510 [18452.204549] process_scheduled_works+0x2a4/0x510 [18452.204553] worker_thread+0x188/0x340 [18452.204555] ? rescuer_thread+0x310/0x310 [18452.204558] kthread+0xf5/0x130 [18452.204560] ? kthread_complete_and_exit+0x20/0x20 [18452.204562] ret_from_fork+0x31/0x40 [18452.204564] ? kthread_complete_and_exit+0x20/0x20 [18452.204566] ret_from_fork_asm+0x11/0x20 [18452.204570] [18477.779852] watchdog: BUG: soft lockup - CPU#0 stuck for 6094s! [dbench:78683] [18477.787070] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18477.791852] watchdog: BUG: soft lockup - CPU#7 stuck for 15091s! [kworker/7:4:78808] [18477.842526] irq event stamp: 26453906 [18477.842528] hardirqs last enabled at (26453905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18477.850259] Modules linked in: [18477.853916] hardirqs last disabled at (26453906): [] sysvec_apic_timer_interrupt+0xe/0x90 [18477.864074] ceph [18477.867123] softirqs last enabled at (26453798): [] __do_softirq+0x2be/0x371 [18477.876847] libceph dns_resolver [18477.878771] softirqs last disabled at (26453793): [] irq_exit_rcu+0xb0/0xd0 [18477.887455] fscache netfs [18477.890768] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18477.899276] veth nft_chain_nat [18477.901982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18477.910926] xt_MASQUERADE nf_nat [18477.914064] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18477.921542] nf_conntrack nf_defrag_ipv6 [18477.924852] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18477.930331] nf_defrag_ipv4 nft_compat [18477.934248] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18477.952986] nf_tables nfnetlink [18477.956729] [18477.956730] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18477.961948] bridge [18477.965171] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18477.966663] stp llc [18477.973787] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18477.975883] xfs libcrc32c [18477.983009] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18477.985191] sunrpc intel_rapl_msr [18477.992316] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18477.995021] intel_rapl_common [18478.002144] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18478.005542] sb_edac x86_pkg_temp_thermal [18478.012665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18478.015715] coretemp kvm_intel [18478.023793] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18478.027797] kvm [18478.033535] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18478.036673] irqbypass i2c_i801 [18478.043796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18478.045635] iTCO_wdt iTCO_vendor_support [18478.052759] Call Trace: [18478.052761] [18478.055896] i2c_smbus crct10dif_pclmul [18478.063022] ? watchdog_timer_fn+0x1b4/0x220 [18478.067024] joydev crc32_pclmul [18478.069469] ? __hrtimer_run_queues+0x1d3/0x360 [18478.071479] ipmi_ssif [18478.075313] ? hrtimer_interrupt+0x104/0x240 [18478.079574] ghash_clmulni_intel mei_me [18478.082797] ? __local_bh_enable+0x50/0x60 [18478.087321] mei lpc_ich [18478.089682] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18478.093944] sha512_ssse3 acpi_ipmi [18478.097775] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18478.101864] mfd_core wmi [18478.104395] [18478.109612] ipmi_si ipmi_devintf [18478.113098] [18478.118054] ipmi_msghandler [18478.120672] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18478.122768] ioatdma acpi_pad [18478.126084] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18478.128176] acpi_power_meter ixgbe [18478.131057] do_raw_spin_lock+0xb5/0xc0 [18478.136359] igb mdio crc32c_intel [18478.139323] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18478.144193] nvme ptp [18478.147679] try_get_cap_refs+0x159/0x880 [ceph] [18478.151508] nvme_core [18478.154910] __ceph_get_caps+0x4a1/0x760 [ceph] [18478.159689] i2c_algo_bit pps_core [18478.161965] ? __file_remove_privs+0x89/0x110 [18478.166571] dca fuse [18478.168928] ? wake_bit_function+0x70/0x70 [18478.173452] [18478.173453] irq event stamp: 70669916 [18478.176853] ? ceph_write_iter+0x260/0x770 [ceph] [18478.181200] hardirqs last enabled at (70669915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18478.183472] ceph_write_iter+0x260/0x770 [ceph] [18478.187561] hardirqs last disabled at (70669916): [] sysvec_apic_timer_interrupt+0xe/0x90 [18478.189055] ? lock_is_held_type+0xaa/0x120 [18478.192709] softirqs last enabled at (70669914): [] __do_softirq+0x2be/0x371 [18478.197411] vfs_write+0x324/0x4f0 [18478.207566] softirqs last disabled at (70669909): [] irq_exit_rcu+0xb0/0xd0 [18478.212094] ksys_pwrite64+0x65/0xa0 [18478.221813] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18478.225992] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18478.234676] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18478.238073] do_syscall_64+0x38/0x80 [18478.246584] Workqueue: events delayed_work [ceph] [18478.250155] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18478.259532] [18478.264662] RIP: 0033:0x7f48f0eeb93b [18478.272132] RIP: 0010:igrab+0x12/0x50 [18478.275705] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18478.280402] Code: 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 <48> 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff [18478.285444] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18478.286936] RSP: 0018:ffffc90002117da0 EFLAGS: 00000286 [18478.290506] ORIG_RAX: 0000000000000012 [18478.294164] [18478.294165] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [18478.312901] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18478.331640] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [18478.336858] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18478.342074] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [18478.345905] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18478.347395] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18478.354534] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18478.361660] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18478.368784] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18478.375910] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18478.383040] [18478.390158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18478.434671] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18478.441796] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18478.448919] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18478.456044] Call Trace: [18478.458488] [18478.460498] ? watchdog_timer_fn+0x1b4/0x220 [18478.464763] ? __hrtimer_run_queues+0x1d3/0x360 [18478.469291] ? hrtimer_interrupt+0x104/0x240 [18478.473558] ? __local_bh_enable+0x50/0x60 [18478.477653] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18478.482877] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18478.487832] [18478.489931] [18478.492029] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18478.497338] ? igrab+0x12/0x50 [18478.500392] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18478.505547] delayed_work+0x1d8/0x280 [ceph] [18478.509822] ? process_scheduled_works+0x22a/0x510 [18478.514604] process_scheduled_works+0x2a4/0x510 [18478.519219] worker_thread+0x188/0x340 [18478.522970] ? rescuer_thread+0x310/0x310 [18478.526974] kthread+0xf5/0x130 [18478.530109] ? kthread_complete_and_exit+0x20/0x20 [18478.534895] ret_from_fork+0x31/0x40 [18478.538464] ? kthread_complete_and_exit+0x20/0x20 [18478.543260] ret_from_fork_asm+0x11/0x20 [18478.547183] [18505.780362] watchdog: BUG: soft lockup - CPU#0 stuck for 6120s! [dbench:78683] [18505.787584] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18505.792361] watchdog: BUG: soft lockup - CPU#7 stuck for 15117s! [kworker/7:4:78808] [18505.843043] irq event stamp: 26512074 [18505.850773] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18505.854428] hardirqs last enabled at (26512073): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18505.861388] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [18505.871545] hardirqs last disabled at (26512074): [] sysvec_apic_timer_interrupt+0xe/0x90 [18505.871549] nf_defrag_ipv6 [18505.871550] softirqs last enabled at (26511998): [] __do_softirq+0x2be/0x371 [18505.877198] nf_defrag_ipv4 nft_compat [18505.886920] softirqs last disabled at (26511993): [] irq_exit_rcu+0xb0/0xd0 [18505.889711] nf_tables nfnetlink [18505.898396] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18505.902139] bridge [18505.910652] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18505.913874] stp llc [18505.922819] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18505.924918] xfs libcrc32c [18505.932389] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18505.934572] sunrpc intel_rapl_msr [18505.940048] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18505.942757] intel_rapl_common sb_edac [18505.961499] [18505.964896] x86_pkg_temp_thermal [18505.970114] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18505.973857] coretemp kvm_intel [18505.975350] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18505.978659] kvm irqbypass [18505.985784] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18505.988921] i2c_i801 iTCO_wdt [18505.996045] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18505.998756] iTCO_vendor_support [18506.005883] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18506.008932] i2c_smbus crct10dif_pclmul [18506.016056] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18506.019280] joydev crc32_pclmul [18506.026404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18506.030235] ipmi_ssif ghash_clmulni_intel [18506.038312] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18506.041536] mei_me mei [18506.047276] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18506.051366] lpc_ich sha512_ssse3 [18506.058491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18506.060934] acpi_ipmi mfd_core [18506.068059] Call Trace: [18506.068061] [18506.071368] wmi [18506.078494] ? watchdog_timer_fn+0x1b4/0x220 [18506.081630] ipmi_si ipmi_devintf [18506.084074] ? __hrtimer_run_queues+0x1d3/0x360 [18506.086086] ipmi_msghandler ioatdma [18506.087927] ? hrtimer_interrupt+0x104/0x240 [18506.092185] acpi_pad acpi_power_meter [18506.095497] ? __local_bh_enable+0x50/0x60 [18506.100021] ixgbe igb [18506.103594] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18506.107856] mdio crc32c_intel [18506.111601] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18506.115690] nvme ptp [18506.118047] [18506.123265] nvme_core i2c_algo_bit [18506.126315] [18506.126317] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18506.131272] pps_core dca [18506.133548] ? queued_spin_lock_slowpath+0x18/0x2e0 [18506.135641] fuse [18506.139128] do_raw_spin_lock+0xb5/0xc0 [18506.141222] irq event stamp: 70891678 [18506.146531] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18506.149144] hardirqs last enabled at (70891677): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18506.154037] try_get_cap_refs+0x159/0x880 [ceph] [18506.155958] hardirqs last disabled at (70891678): [] sysvec_apic_timer_interrupt+0xe/0x90 [18506.159793] __ceph_get_caps+0x4a1/0x760 [ceph] [18506.163446] softirqs last enabled at (70891676): [] __do_softirq+0x2be/0x371 [18506.168232] ? __file_remove_privs+0x89/0x110 [18506.178387] softirqs last disabled at (70891671): [] irq_exit_rcu+0xb0/0xd0 [18506.182997] ? wake_bit_function+0x70/0x70 [18506.192722] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18506.197248] ? ceph_write_iter+0x260/0x770 [ceph] [18506.205929] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18506.210281] ceph_write_iter+0x260/0x770 [ceph] [18506.218794] Workqueue: events delayed_work [ceph] [18506.222886] ? lock_is_held_type+0xaa/0x120 [18506.232259] [18506.232260] RIP: 0010:lock_acquire+0xf0/0x2b0 [18506.236958] vfs_write+0x324/0x4f0 [18506.244427] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18506.248957] ksys_pwrite64+0x65/0xa0 [18506.253648] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18506.257828] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18506.259316] [18506.259317] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18506.263669] do_syscall_64+0x38/0x80 [18506.267065] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18506.285803] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18506.289374] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18506.294592] RIP: 0033:0x7f48f0eeb93b [18506.299722] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18506.301214] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18506.308336] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18506.311907] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18506.319043] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18506.324084] ORIG_RAX: 0000000000000012 [18506.331208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18506.334779] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18506.341905] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18506.360642] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18506.367765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18506.372982] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18506.381059] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18506.384890] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18506.390629] Call Trace: [18506.390631] [18506.397766] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18506.404896] ? watchdog_timer_fn+0x1b4/0x220 [18506.412025] [18506.419143] ? __hrtimer_run_queues+0x1d3/0x360 [18506.463070] ? hrtimer_interrupt+0x104/0x240 [18506.467340] ? __local_bh_enable+0x50/0x60 [18506.471433] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18506.476655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18506.481613] [18506.483709] [18506.485809] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18506.491116] ? lock_acquire+0xf0/0x2b0 [18506.494865] ? igrab+0x43/0x50 [18506.497916] ? lock_release+0xc5/0x260 [18506.501661] _raw_spin_lock+0x30/0x40 [18506.505317] ? igrab+0x1d/0x50 [18506.508368] igrab+0x1d/0x50 [18506.511246] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18506.516398] delayed_work+0x1d8/0x280 [ceph] [18506.520677] ? process_scheduled_works+0x22a/0x510 [18506.525468] process_scheduled_works+0x2a4/0x510 [18506.530084] worker_thread+0x188/0x340 [18506.533832] ? rescuer_thread+0x310/0x310 [18506.537834] kthread+0xf5/0x130 [18506.540973] ? kthread_complete_and_exit+0x20/0x20 [18506.545769] ret_from_fork+0x31/0x40 [18506.549344] ? kthread_complete_and_exit+0x20/0x20 [18506.554128] ret_from_fork_asm+0x11/0x20 [18506.558062] [18533.780868] watchdog: BUG: soft lockup - CPU#0 stuck for 6146s! [dbench:78683] [18533.788089] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18533.792869] watchdog: BUG: soft lockup - CPU#7 stuck for 15143s! [kworker/7:4:78808] [18533.843542] irq event stamp: 26570168 [18533.843543] hardirqs last enabled at (26570167): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18533.851276] Modules linked in: [18533.854935] hardirqs last disabled at (26570168): [] sysvec_apic_timer_interrupt+0xe/0x90 [18533.865090] ceph [18533.868143] softirqs last enabled at (26570142): [] __do_softirq+0x2be/0x371 [18533.877866] libceph [18533.879791] softirqs last disabled at (26570137): [] irq_exit_rcu+0xb0/0xd0 [18533.888474] dns_resolver fscache [18533.890658] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18533.899169] netfs veth [18533.902480] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18533.911423] nft_chain_nat xt_MASQUERADE [18533.913868] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18533.921339] nf_nat nf_conntrack [18533.925264] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18533.930741] nf_defrag_ipv6 nf_defrag_ipv4 [18533.933966] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18533.952703] nft_compat nf_tables [18533.956794] [18533.962011] nfnetlink bridge [18533.965322] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18533.966813] stp llc [18533.969778] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18533.976901] xfs libcrc32c [18533.979085] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18533.986211] sunrpc intel_rapl_msr [18533.988913] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18533.996037] intel_rapl_common [18533.999437] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18534.006560] sb_edac x86_pkg_temp_thermal [18534.009610] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18534.016736] coretemp kvm_intel [18534.020740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18534.028815] kvm irqbypass [18534.031955] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18534.037691] i2c_i801 iTCO_wdt [18534.040396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18534.047520] iTCO_vendor_support i2c_smbus [18534.050571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18534.057694] crct10dif_pclmul joydev crc32_pclmul [18534.061786] Call Trace: [18534.068908] ipmi_ssif ghash_clmulni_intel [18534.073607] [18534.076051] mei_me mei [18534.080143] ? watchdog_timer_fn+0x1b4/0x220 [18534.082151] lpc_ich sha512_ssse3 [18534.084598] ? __hrtimer_run_queues+0x1d3/0x360 [18534.088860] acpi_ipmi mfd_core [18534.092175] ? hrtimer_interrupt+0x104/0x240 [18534.096696] wmi ipmi_si [18534.099833] ? __local_bh_enable+0x50/0x60 [18534.104096] ipmi_devintf ipmi_msghandler [18534.106631] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18534.110718] ioatdma acpi_pad [18534.114723] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18534.119940] acpi_power_meter ixgbe [18534.122904] [18534.127860] igb mdio [18534.131346] [18534.131347] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18534.133442] crc32c_intel nvme [18534.135716] ? queued_spin_lock_slowpath+0x16/0x2e0 [18534.137811] ptp nvme_core [18534.143117] do_raw_spin_lock+0xb5/0xc0 [18534.146165] i2c_algo_bit pps_core [18534.151036] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18534.153740] dca fuse [18534.157574] try_get_cap_refs+0x159/0x880 [ceph] [18534.160967] [18534.160968] irq event stamp: 71112828 [18534.165757] __ceph_get_caps+0x4a1/0x760 [ceph] [18534.168023] hardirqs last enabled at (71112827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18534.172638] ? __file_remove_privs+0x89/0x110 [18534.174124] hardirqs last disabled at (71112828): [] sysvec_apic_timer_interrupt+0xe/0x90 [18534.177783] ? wake_bit_function+0x70/0x70 [18534.182305] softirqs last enabled at (71112826): [] __do_softirq+0x2be/0x371 [18534.192466] ? ceph_write_iter+0x260/0x770 [ceph] [18534.196814] softirqs last disabled at (71112821): [] irq_exit_rcu+0xb0/0xd0 [18534.206548] ceph_write_iter+0x260/0x770 [ceph] [18534.210638] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18534.219326] ? lock_is_held_type+0xaa/0x120 [18534.224019] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18534.232532] vfs_write+0x324/0x4f0 [18534.237056] Workqueue: events delayed_work [ceph] [18534.246436] ksys_pwrite64+0x65/0xa0 [18534.250609] [18534.250610] RIP: 0010:lock_acquire+0x86/0x2b0 [18534.258081] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18534.261477] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [18534.266176] do_syscall_64+0x38/0x80 [18534.269744] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [18534.271236] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18534.275587] [18534.280718] RIP: 0033:0x7f48f0eeb93b [18534.299455] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [18534.303025] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18534.308244] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [18534.313306] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18534.314796] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [18534.318366] ORIG_RAX: 0000000000000012 [18534.325492] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [18534.344229] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18534.351353] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18534.351355] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18534.356571] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18534.363693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18534.367526] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18534.374648] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18534.381772] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18534.388896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18534.396975] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18534.404099] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18534.409840] [18534.416960] Call Trace: [18534.416961] [18534.459221] ? watchdog_timer_fn+0x1b4/0x220 [18534.463485] ? __hrtimer_run_queues+0x1d3/0x360 [18534.468012] ? hrtimer_interrupt+0x104/0x240 [18534.472282] ? __local_bh_enable+0x50/0x60 [18534.476373] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18534.481589] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18534.486555] [18534.488652] [18534.490750] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18534.496060] ? lock_acquire+0x86/0x2b0 [18534.499808] ? igrab+0x43/0x50 [18534.502859] ? lock_release+0xc5/0x260 [18534.506604] _raw_spin_lock+0x30/0x40 [18534.510274] ? igrab+0x1d/0x50 [18534.513327] igrab+0x1d/0x50 [18534.516206] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18534.521356] delayed_work+0x1d8/0x280 [ceph] [18534.525635] ? process_scheduled_works+0x22a/0x510 [18534.530417] process_scheduled_works+0x2a4/0x510 [18534.535034] worker_thread+0x188/0x340 [18534.538783] ? rescuer_thread+0x310/0x310 [18534.542787] kthread+0xf5/0x130 [18534.545923] ? kthread_complete_and_exit+0x20/0x20 [18534.550707] ret_from_fork+0x31/0x40 [18534.554278] ? kthread_complete_and_exit+0x20/0x20 [18534.559062] ret_from_fork_asm+0x11/0x20 [18534.562986] [18561.781377] watchdog: BUG: soft lockup - CPU#0 stuck for 6173s! [dbench:78683] [18561.788592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18561.793377] watchdog: BUG: soft lockup - CPU#7 stuck for 15170s! [kworker/7:4:78808] [18561.844033] irq event stamp: 26628244 [18561.844034] hardirqs last enabled at (26628243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18561.851773] Modules linked in: [18561.855429] hardirqs last disabled at (26628244): [] sysvec_apic_timer_interrupt+0xe/0x90 [18561.865587] ceph [18561.868637] softirqs last enabled at (26628218): [] __do_softirq+0x2be/0x371 [18561.878363] libceph dns_resolver [18561.880287] softirqs last disabled at (26628213): [] irq_exit_rcu+0xb0/0xd0 [18561.888970] fscache netfs [18561.892281] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18561.900792] veth nft_chain_nat [18561.903496] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18561.912439] xt_MASQUERADE nf_nat [18561.915576] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [18561.923055] nf_conntrack nf_defrag_ipv6 [18561.926367] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [18561.931846] nf_defrag_ipv4 nft_compat [18561.935774] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18561.954510] nf_tables nfnetlink [18561.958254] [18561.963470] bridge stp [18561.966695] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18561.968185] llc xfs [18561.970630] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18561.977772] libcrc32c sunrpc [18561.979954] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18561.987078] intel_rapl_msr intel_rapl_common [18561.990055] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18561.997175] sb_edac x86_pkg_temp_thermal [18562.001526] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18562.008651] coretemp kvm_intel [18562.012656] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18562.019779] kvm irqbypass [18562.022916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18562.030992] i2c_i801 iTCO_wdt [18562.033697] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18562.039434] iTCO_vendor_support [18562.042486] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18562.049610] i2c_smbus crct10dif_pclmul [18562.052834] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18562.059959] joydev crc32_pclmul [18562.063788] Call Trace: [18562.063789] [18562.070914] ipmi_ssif ghash_clmulni_intel [18562.074137] ? watchdog_timer_fn+0x1b4/0x220 [18562.076582] mei_me mei [18562.078593] ? __hrtimer_run_queues+0x1d3/0x360 [18562.082683] lpc_ich sha512_ssse3 [18562.086950] ? hrtimer_interrupt+0x104/0x240 [18562.089390] acpi_ipmi mfd_core [18562.093914] ? __local_bh_enable+0x50/0x60 [18562.097225] wmi [18562.101492] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18562.104627] ipmi_si ipmi_devintf [18562.108719] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18562.110554] ipmi_msghandler ioatdma [18562.115775] [18562.119083] acpi_pad acpi_power_meter [18562.124056] [18562.124057] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18562.127630] ixgbe igb [18562.129732] ? queued_spin_lock_slowpath+0x2b/0x2e0 [18562.133471] mdio crc32c_intel [18562.135569] do_raw_spin_lock+0xb5/0xc0 [18562.140871] nvme ptp nvme_core [18562.143229] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18562.148099] i2c_algo_bit pps_core [18562.151153] try_get_cap_refs+0x159/0x880 [ceph] [18562.154982] dca fuse [18562.158123] __ceph_get_caps+0x4a1/0x760 [ceph] [18562.162903] [18562.162903] irq event stamp: 71334264 [18562.166304] ? __file_remove_privs+0x89/0x110 [18562.170911] hardirqs last enabled at (71334263): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18562.173181] ? wake_bit_function+0x70/0x70 [18562.177707] hardirqs last disabled at (71334264): [] sysvec_apic_timer_interrupt+0xe/0x90 [18562.179200] ? ceph_write_iter+0x260/0x770 [ceph] [18562.182853] softirqs last enabled at (71334262): [] __do_softirq+0x2be/0x371 [18562.187206] ceph_write_iter+0x260/0x770 [ceph] [18562.197362] softirqs last disabled at (71334257): [] irq_exit_rcu+0xb0/0xd0 [18562.201458] ? lock_is_held_type+0xaa/0x120 [18562.211176] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18562.215876] vfs_write+0x324/0x4f0 [18562.224558] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18562.229088] ksys_pwrite64+0x65/0xa0 [18562.237595] Workqueue: events delayed_work [ceph] [18562.241778] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18562.251156] [18562.251157] RIP: 0010:lock_acquire+0xf0/0x2b0 [18562.254555] do_syscall_64+0x38/0x80 [18562.262024] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18562.265596] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18562.270295] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18562.275425] RIP: 0033:0x7f48f0eeb93b [18562.276915] [18562.276916] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18562.281266] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18562.284836] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18562.303573] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18562.308619] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18562.313834] ORIG_RAX: 0000000000000012 [18562.317407] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18562.318898] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18562.326021] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18562.326022] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18562.344776] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18562.351900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18562.357118] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18562.364243] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18562.368073] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18562.375196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18562.382321] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18562.389445] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18562.397528] [18562.404647] Call Trace: [18562.404648] [18562.459794] ? watchdog_timer_fn+0x1b4/0x220 [18562.464069] ? __hrtimer_run_queues+0x1d3/0x360 [18562.468595] ? hrtimer_interrupt+0x104/0x240 [18562.472863] ? __local_bh_enable+0x50/0x60 [18562.476957] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18562.482182] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18562.487137] [18562.489237] [18562.491334] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18562.496642] ? lock_acquire+0xf0/0x2b0 [18562.500392] ? igrab+0x43/0x50 [18562.503440] ? lock_release+0xc5/0x260 [18562.507187] _raw_spin_lock+0x30/0x40 [18562.510852] ? igrab+0x1d/0x50 [18562.513900] igrab+0x1d/0x50 [18562.516784] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18562.521930] delayed_work+0x1d8/0x280 [ceph] [18562.526210] ? process_scheduled_works+0x22a/0x510 [18562.531000] process_scheduled_works+0x2a4/0x510 [18562.535615] worker_thread+0x188/0x340 [18562.539366] ? rescuer_thread+0x310/0x310 [18562.543370] kthread+0xf5/0x130 [18562.546506] ? kthread_complete_and_exit+0x20/0x20 [18562.551291] ret_from_fork+0x31/0x40 [18562.554861] ? kthread_complete_and_exit+0x20/0x20 [18562.559646] ret_from_fork_asm+0x11/0x20 [18562.563569] [18589.781885] watchdog: BUG: soft lockup - CPU#0 stuck for 6199s! [dbench:78683] [18589.789107] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18589.793884] watchdog: BUG: soft lockup - CPU#7 stuck for 15196s! [kworker/7:4:78808] [18589.844567] irq event stamp: 26686354 [18589.852296] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [18589.855953] hardirqs last enabled at (26686353): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18589.864125] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [18589.874281] hardirqs last disabled at (26686354): [] sysvec_apic_timer_interrupt+0xe/0x90 [18589.882277] nf_tables nfnetlink bridge stp llc xfs libcrc32c [18589.892007] softirqs last enabled at (26686298): [] __do_softirq+0x2be/0x371 [18589.892011] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [18589.906437] softirqs last disabled at (26686293): [] irq_exit_rcu+0xb0/0xd0 [18589.915908] irqbypass i2c_i801 iTCO_wdt [18589.924421] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18589.924422] iTCO_vendor_support [18589.924424] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18589.928337] i2c_smbus crct10dif_pclmul [18589.937281] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18589.940506] joydev crc32_pclmul [18589.947977] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18589.951807] ipmi_ssif ghash_clmulni_intel [18589.957284] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18589.960509] mei_me mei [18589.979248] [18589.983339] lpc_ich [18589.988572] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18589.991015] sha512_ssse3 acpi_ipmi [18589.992505] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18589.994691] mfd_core wmi [18590.001816] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18590.005297] ipmi_si ipmi_devintf [18590.012424] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18590.015041] ipmi_msghandler ioatdma [18590.022164] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18590.025474] acpi_pad acpi_power_meter [18590.032599] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18590.036169] ixgbe igb [18590.043294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18590.047039] mdio crc32c_intel [18590.055116] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18590.057472] nvme ptp [18590.063210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18590.066279] nvme_core i2c_algo_bit [18590.073403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18590.075673] pps_core [18590.082797] Call Trace: [18590.086284] dca fuse [18590.093409] [18590.095677] [18590.095678] irq event stamp: 71555872 [18590.098123] ? watchdog_timer_fn+0x1b4/0x220 [18590.100392] hardirqs last enabled at (71555871): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18590.102403] ? __hrtimer_run_queues+0x1d3/0x360 [18590.103893] hardirqs last disabled at (71555872): [] sysvec_apic_timer_interrupt+0xe/0x90 [18590.107568] ? hrtimer_interrupt+0x104/0x240 [18590.111831] softirqs last enabled at (71555870): [] __do_softirq+0x2be/0x371 [18590.121989] ? __local_bh_enable+0x50/0x60 [18590.126512] softirqs last disabled at (71555865): [] irq_exit_rcu+0xb0/0xd0 [18590.136240] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18590.140502] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18590.149186] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18590.153283] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18590.161796] [18590.167014] Workqueue: events delayed_work [ceph] [18590.176391] [18590.181349] [18590.181350] RIP: 0010:lock_acquire+0xf0/0x2b0 [18590.188820] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18590.190915] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18590.195616] ? queued_spin_lock_slowpath+0x16/0x2e0 [18590.197711] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18590.199204] do_raw_spin_lock+0xb5/0xc0 [18590.203567] [18590.203568] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18590.208875] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18590.227611] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18590.232485] try_get_cap_refs+0x159/0x880 [ceph] [18590.237700] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18590.241536] __ceph_get_caps+0x4a1/0x760 [ceph] [18590.243022] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18590.250150] ? __file_remove_privs+0x89/0x110 [18590.254930] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18590.262054] ? wake_bit_function+0x70/0x70 [18590.266664] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18590.273792] ? ceph_write_iter+0x260/0x770 [ceph] [18590.278313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18590.285438] ceph_write_iter+0x260/0x770 [ceph] [18590.289788] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18590.296916] ? lock_is_held_type+0xaa/0x120 [18590.301003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18590.309083] vfs_write+0x324/0x4f0 [18590.313777] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18590.319520] ksys_pwrite64+0x65/0xa0 [18590.324038] Call Trace: [18590.324040] [18590.331163] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18590.335341] ? watchdog_timer_fn+0x1b4/0x220 [18590.342467] do_syscall_64+0x38/0x80 [18590.345864] ? __hrtimer_run_queues+0x1d3/0x360 [18590.352988] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18590.356571] ? hrtimer_interrupt+0x104/0x240 [18590.359009] RIP: 0033:0x7f48f0eeb93b [18590.361019] ? __local_bh_enable+0x50/0x60 [18590.366152] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18590.370418] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18590.373985] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18590.378509] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18590.383569] ORIG_RAX: 0000000000000012 [18590.387835] [18590.391406] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18590.395496] [18590.414235] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18590.419453] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18590.424670] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18590.429631] ? lock_acquire+0xf0/0x2b0 [18590.433456] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18590.435570] ? igrab+0x43/0x50 [18590.442695] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18590.444795] ? lock_release+0xc5/0x260 [18590.451922] [18590.457223] _raw_spin_lock+0x30/0x40 [18590.494991] ? igrab+0x1d/0x50 [18590.498043] igrab+0x1d/0x50 [18590.500923] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18590.506072] delayed_work+0x1d8/0x280 [ceph] [18590.510354] ? process_scheduled_works+0x22a/0x510 [18590.515143] process_scheduled_works+0x2a4/0x510 [18590.519758] worker_thread+0x188/0x340 [18590.523508] ? rescuer_thread+0x310/0x310 [18590.527518] kthread+0xf5/0x130 [18590.530656] ? kthread_complete_and_exit+0x20/0x20 [18590.535441] ret_from_fork+0x31/0x40 [18590.539020] ? kthread_complete_and_exit+0x20/0x20 [18590.543804] ret_from_fork_asm+0x11/0x20 [18590.547737] [18617.782392] watchdog: BUG: soft lockup - CPU#0 stuck for 6225s! [dbench:78683] [18617.789611] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18617.794392] watchdog: BUG: soft lockup - CPU#7 stuck for 15222s! [kworker/7:4:78808] [18617.845066] irq event stamp: 26744326 [18617.845068] hardirqs last enabled at (26744325): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18617.852801] Modules linked in: ceph [18617.856456] hardirqs last disabled at (26744326): [] sysvec_apic_timer_interrupt+0xe/0x90 [18617.866615] libceph [18617.870099] softirqs last enabled at (26744288): [] __do_softirq+0x2be/0x371 [18617.879821] dns_resolver [18617.882006] softirqs last disabled at (26744283): [] irq_exit_rcu+0xb0/0xd0 [18617.890690] fscache netfs [18617.893307] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18617.901820] veth [18617.904523] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18617.913468] nft_chain_nat xt_MASQUERADE [18617.915391] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18617.922863] nf_nat [18617.926791] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18617.932265] nf_conntrack [18617.934363] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18617.953101] nf_defrag_ipv6 nf_defrag_ipv4 [18617.955719] [18617.955719] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18617.960936] nft_compat nf_tables [18617.965025] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18617.966516] nfnetlink bridge [18617.973641] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18617.976952] stp llc [18617.984083] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18617.987065] xfs libcrc32c [18617.994190] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18617.996373] sunrpc intel_rapl_msr [18618.003499] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18618.006203] intel_rapl_common sb_edac [18618.013326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18618.016724] x86_pkg_temp_thermal coretemp [18618.024801] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18618.028545] kvm_intel kvm [18618.034284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18618.038372] irqbypass i2c_i801 [18618.045498] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18618.048202] iTCO_wdt iTCO_vendor_support [18618.055326] Call Trace: [18618.055328] [18618.058462] i2c_smbus crct10dif_pclmul [18618.065589] ? watchdog_timer_fn+0x1b4/0x220 [18618.069592] joydev crc32_pclmul [18618.072038] ? __hrtimer_run_queues+0x1d3/0x360 [18618.074046] ipmi_ssif ghash_clmulni_intel [18618.077880] ? hrtimer_interrupt+0x104/0x240 [18618.082140] mei_me mei [18618.085366] ? __local_bh_enable+0x50/0x60 [18618.089888] lpc_ich [18618.093982] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18618.098244] sha512_ssse3 [18618.100688] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18618.104790] acpi_ipmi mfd_core [18618.106973] [18618.112189] wmi ipmi_si [18618.114807] [18618.119765] ipmi_devintf ipmi_msghandler [18618.122918] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18618.125017] ioatdma acpi_pad [18618.127551] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18618.129643] acpi_power_meter ixgbe [18618.133651] do_raw_spin_lock+0xb5/0xc0 [18618.138951] igb mdio [18618.141917] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18618.146791] crc32c_intel nvme [18618.150282] try_get_cap_refs+0x159/0x880 [ceph] [18618.154110] ptp nvme_core [18618.156387] __ceph_get_caps+0x4a1/0x760 [ceph] [18618.161164] i2c_algo_bit pps_core [18618.164220] ? __file_remove_privs+0x89/0x110 [18618.168827] dca fuse [18618.171531] ? wake_bit_function+0x70/0x70 [18618.176071] [18618.176072] irq event stamp: 71777542 [18618.179472] ? ceph_write_iter+0x260/0x770 [ceph] [18618.183820] hardirqs last enabled at (71777541): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18618.186091] ceph_write_iter+0x260/0x770 [ceph] [18618.190182] hardirqs last disabled at (71777542): [] sysvec_apic_timer_interrupt+0xe/0x90 [18618.191677] ? lock_is_held_type+0xaa/0x120 [18618.195330] softirqs last enabled at (71777540): [] __do_softirq+0x2be/0x371 [18618.200030] vfs_write+0x324/0x4f0 [18618.210184] softirqs last disabled at (71777535): [] irq_exit_rcu+0xb0/0xd0 [18618.214714] ksys_pwrite64+0x65/0xa0 [18618.224432] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18618.228611] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18618.237294] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18618.237295] Workqueue: events delayed_work [ceph] [18618.240694] do_syscall_64+0x38/0x80 [18618.249203] [18618.249204] RIP: 0010:lock_acquire+0xf0/0x2b0 [18618.252792] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18618.262168] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18618.267299] RIP: 0033:0x7f48f0eeb93b [18618.274771] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18618.279486] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18618.283071] [18618.283072] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18618.284562] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18618.288914] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18618.293958] ORIG_RAX: 0000000000000012 [18618.312697] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18618.316268] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18618.321485] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18618.340221] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18618.341711] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18618.348836] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18618.354054] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18618.361196] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18618.365025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18618.372150] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18618.379276] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18618.386405] [18618.393522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18618.452283] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18618.459407] Call Trace: [18618.461852] [18618.463865] ? watchdog_timer_fn+0x1b4/0x220 [18618.468129] ? __hrtimer_run_queues+0x1d3/0x360 [18618.472654] ? hrtimer_interrupt+0x104/0x240 [18618.476916] ? __local_bh_enable+0x50/0x60 [18618.481010] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18618.486232] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18618.491189] [18618.493288] [18618.495385] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18618.500693] ? lock_acquire+0xf0/0x2b0 [18618.504442] ? igrab+0x43/0x50 [18618.507494] ? lock_release+0xc5/0x260 [18618.511240] _raw_spin_lock+0x30/0x40 [18618.514903] ? igrab+0x1d/0x50 [18618.517952] igrab+0x1d/0x50 [18618.520831] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18618.525980] delayed_work+0x1d8/0x280 [ceph] [18618.530262] ? process_scheduled_works+0x22a/0x510 [18618.535053] process_scheduled_works+0x2a4/0x510 [18618.539668] worker_thread+0x188/0x340 [18618.543417] ? rescuer_thread+0x310/0x310 [18618.547420] kthread+0xf5/0x130 [18618.550558] ? kthread_complete_and_exit+0x20/0x20 [18618.555342] ret_from_fork+0x31/0x40 [18618.558913] ? kthread_complete_and_exit+0x20/0x20 [18618.563697] ret_from_fork_asm+0x11/0x20 [18618.567620] [18645.782900] watchdog: BUG: soft lockup - CPU#0 stuck for 6251s! [dbench:78683] [18645.790117] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18645.794900] watchdog: BUG: soft lockup - CPU#7 stuck for 15248s! [kworker/7:4:78808] [18645.845576] irq event stamp: 26802528 [18645.853304] Modules linked in: ceph libceph dns_resolver fscache netfs veth [18645.856961] hardirqs last enabled at (26802527): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18645.863914] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [18645.874069] hardirqs last disabled at (26802528): [] sysvec_apic_timer_interrupt+0xe/0x90 [18645.881020] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [18645.890745] softirqs last enabled at (26802490): [] __do_softirq+0x2be/0x371 [18645.897522] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [18645.906208] softirqs last disabled at (26802485): [] irq_exit_rcu+0xb0/0xd0 [18645.914901] coretemp kvm_intel kvm irqbypass i2c_i801 [18645.923412] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18645.928540] iTCO_wdt iTCO_vendor_support i2c_smbus [18645.937486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18645.937487] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18645.942355] crct10dif_pclmul joydev [18645.949827] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18645.955304] crc32_pclmul [18645.958876] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18645.977614] ipmi_ssif ghash_clmulni_intel [18645.980231] [18645.985447] mei_me mei [18645.989538] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18645.991029] lpc_ich [18645.993472] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18646.000598] sha512_ssse3 acpi_ipmi [18646.002780] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18646.009906] mfd_core wmi [18646.013390] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18646.020515] ipmi_si ipmi_devintf [18646.023130] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18646.030255] ipmi_msghandler [18646.033565] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18646.040707] ioatdma acpi_pad [18646.043584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18646.051662] acpi_power_meter ixgbe [18646.054627] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18646.060365] igb mdio [18646.063848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18646.070972] crc32c_intel nvme [18646.073242] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18646.080368] ptp nvme_core [18646.083418] Call Trace: [18646.090542] i2c_algo_bit pps_core [18646.093248] [18646.095690] dca fuse [18646.099090] ? watchdog_timer_fn+0x1b4/0x220 [18646.101097] [18646.103369] ? __hrtimer_run_queues+0x1d3/0x360 [18646.107631] irq event stamp: 71999098 [18646.109126] ? hrtimer_interrupt+0x104/0x240 [18646.113647] hardirqs last enabled at (71999097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18646.117305] ? __local_bh_enable+0x50/0x60 [18646.121579] hardirqs last disabled at (71999098): [] sysvec_apic_timer_interrupt+0xe/0x90 [18646.131738] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18646.135824] softirqs last enabled at (71999096): [] __do_softirq+0x2be/0x371 [18646.145552] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18646.150768] softirqs last disabled at (71999091): [] irq_exit_rcu+0xb0/0xd0 [18646.159452] [18646.164409] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18646.172920] [18646.172922] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18646.175016] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18646.184398] ? queued_spin_lock_slowpath+0x16/0x2e0 [18646.186493] Workqueue: events delayed_work [ceph] [18646.191799] do_raw_spin_lock+0xb5/0xc0 [18646.199268] [18646.199269] RIP: 0010:lock_acquire+0xf0/0x2b0 [18646.204140] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18646.208834] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18646.212670] try_get_cap_refs+0x159/0x880 [ceph] [18646.214155] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18646.218514] __ceph_get_caps+0x4a1/0x760 [ceph] [18646.223301] [18646.223302] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18646.242042] ? __file_remove_privs+0x89/0x110 [18646.246648] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18646.251866] ? wake_bit_function+0x70/0x70 [18646.256390] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18646.257882] ? ceph_write_iter+0x260/0x770 [ceph] [18646.265005] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18646.269357] ceph_write_iter+0x260/0x770 [ceph] [18646.276479] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18646.280587] ? lock_is_held_type+0xaa/0x120 [18646.287702] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18646.292404] vfs_write+0x324/0x4f0 [18646.299524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18646.304054] ksys_pwrite64+0x65/0xa0 [18646.311172] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18646.315350] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18646.323427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18646.326828] do_syscall_64+0x38/0x80 [18646.332563] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18646.336136] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18646.343257] Call Trace: [18646.343259] [18646.348388] RIP: 0033:0x7f48f0eeb93b [18646.355513] ? watchdog_timer_fn+0x1b4/0x220 [18646.359084] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18646.366210] ? __hrtimer_run_queues+0x1d3/0x360 [18646.371252] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18646.373699] ? hrtimer_interrupt+0x104/0x240 [18646.375706] ORIG_RAX: 0000000000000012 [18646.379277] ? __local_bh_enable+0x50/0x60 [18646.383542] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18646.402302] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18646.406819] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18646.412039] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18646.416303] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18646.420133] [18646.424223] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18646.431346] [18646.431348] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18646.436565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18646.443692] ? lock_acquire+0xf0/0x2b0 [18646.448651] [18646.455772] ? igrab+0x43/0x50 [18646.488497] ? lock_release+0xc5/0x260 [18646.492243] _raw_spin_lock+0x30/0x40 [18646.495899] ? igrab+0x1d/0x50 [18646.498949] igrab+0x1d/0x50 [18646.501829] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18646.506979] delayed_work+0x1d8/0x280 [ceph] [18646.511260] ? process_scheduled_works+0x22a/0x510 [18646.516049] process_scheduled_works+0x2a4/0x510 [18646.520666] worker_thread+0x188/0x340 [18646.524413] ? rescuer_thread+0x310/0x310 [18646.528416] kthread+0xf5/0x130 [18646.531554] ? kthread_complete_and_exit+0x20/0x20 [18646.536339] ret_from_fork+0x31/0x40 [18646.539918] ? kthread_complete_and_exit+0x20/0x20 [18646.544702] ret_from_fork_asm+0x11/0x20 [18646.548635] [18647.211293] rcu: INFO: rcu_preempt self-detected stall on CPU [18647.217049] rcu: 7-....: (6517729 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/6516488 fqs=3263411 [18647.227474] rcu: (t=6710481 jiffies g=1309609 q=255917 ncpus=8) [18647.233471] Sending NMI from CPU 7 to CPUs 0: [18647.237823] NMI backtrace for cpu 0 [18647.237825] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18647.237827] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18647.237828] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18647.237832] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18647.237834] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18647.237836] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18647.237837] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18647.237838] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18647.237840] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18647.237841] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18647.237842] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18647.237844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18647.237845] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18647.237847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18647.237848] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18647.237849] Call Trace: [18647.237850] [18647.237851] ? nmi_cpu_backtrace+0xad/0x120 [18647.237856] ? nmi_cpu_backtrace_handler+0x11/0x20 [18647.237858] ? nmi_handle+0xd4/0x260 [18647.237864] ? default_do_nmi+0x49/0xf0 [18647.237867] ? exc_nmi+0x11e/0x150 [18647.237869] ? end_repeat_nmi+0x16/0x67 [18647.237876] ? queued_spin_lock_slowpath+0x16/0x2e0 [18647.237879] ? queued_spin_lock_slowpath+0x16/0x2e0 [18647.237883] ? queued_spin_lock_slowpath+0x16/0x2e0 [18647.237887] [18647.237887] [18647.237889] do_raw_spin_lock+0xb5/0xc0 [18647.237892] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18647.237912] try_get_cap_refs+0x159/0x880 [ceph] [18647.237933] __ceph_get_caps+0x4a1/0x760 [ceph] [18647.237953] ? __file_remove_privs+0x89/0x110 [18647.237955] ? wake_bit_function+0x70/0x70 [18647.237960] ? ceph_write_iter+0x260/0x770 [ceph] [18647.237975] ceph_write_iter+0x260/0x770 [ceph] [18647.237993] ? lock_is_held_type+0xaa/0x120 [18647.237997] vfs_write+0x324/0x4f0 [18647.238004] ksys_pwrite64+0x65/0xa0 [18647.238007] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18647.238011] do_syscall_64+0x38/0x80 [18647.238014] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18647.238016] RIP: 0033:0x7f48f0eeb93b [18647.238018] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18647.238020] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [18647.238022] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18647.238023] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18647.238024] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18647.238026] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18647.238027] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18647.238033] [18647.238821] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18647.542703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18647.550175] Workqueue: events delayed_work [ceph] [18647.554889] RIP: 0010:lock_acquire+0xf0/0x2b0 [18647.559238] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18647.577977] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18647.583194] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18647.590326] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18647.597450] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18647.604576] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18647.611699] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18647.618825] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18647.626901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18647.632640] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18647.639763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18647.646886] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18647.654012] Call Trace: [18647.656457] [18647.658468] ? rcu_dump_cpu_stacks+0xd1/0x180 [18647.662826] ? rcu_sched_clock_irq+0x633/0x1390 [18647.667351] ? lock_release+0xc5/0x260 [18647.671094] ? lock_is_held_type+0xaa/0x120 [18647.675273] ? lock_is_held_type+0xaa/0x120 [18647.679449] ? tick_sched_do_timer+0xa0/0xa0 [18647.683722] ? rcu_read_lock_sched_held+0x56/0x90 [18647.688421] ? tick_sched_do_timer+0xa0/0xa0 [18647.692685] ? update_process_times+0x5f/0x90 [18647.697034] ? tick_sched_handle.isra.16+0x1e/0x50 [18647.701818] ? tick_sched_timer+0x6f/0x80 [18647.705823] ? __hrtimer_run_queues+0x1d3/0x360 [18647.710351] ? hrtimer_interrupt+0x104/0x240 [18647.714619] ? __local_bh_enable+0x50/0x60 [18647.718712] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18647.723936] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18647.728893] [18647.730990] [18647.733087] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18647.738396] ? lock_acquire+0xf0/0x2b0 [18647.742146] ? igrab+0x43/0x50 [18647.745196] ? lock_release+0xc5/0x260 [18647.748941] _raw_spin_lock+0x30/0x40 [18647.752613] ? igrab+0x1d/0x50 [18647.755664] igrab+0x1d/0x50 [18647.758544] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18647.763692] delayed_work+0x1d8/0x280 [ceph] [18647.767973] ? process_scheduled_works+0x22a/0x510 [18647.772755] process_scheduled_works+0x2a4/0x510 [18647.777371] worker_thread+0x188/0x340 [18647.781121] ? rescuer_thread+0x310/0x310 [18647.785123] kthread+0xf5/0x130 [18647.788261] ? kthread_complete_and_exit+0x20/0x20 [18647.793046] ret_from_fork+0x31/0x40 [18647.796624] ? kthread_complete_and_exit+0x20/0x20 [18647.801409] ret_from_fork_asm+0x11/0x20 [18647.805336] [18673.783408] watchdog: BUG: soft lockup - CPU#0 stuck for 6277s! [dbench:78683] [18673.790621] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18673.795408] watchdog: BUG: soft lockup - CPU#7 stuck for 15274s! [kworker/7:4:78808] [18673.846078] irq event stamp: 26860504 [18673.853811] Modules linked in: ceph libceph dns_resolver fscache [18673.857468] hardirqs last enabled at (26860503): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18673.863464] netfs veth [18673.873623] hardirqs last disabled at (26860504): [] sysvec_apic_timer_interrupt+0xe/0x90 [18673.873625] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [18673.885807] softirqs last enabled at (26860466): [] __do_softirq+0x2be/0x371 [18673.892758] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [18673.901444] softirqs last disabled at (26860461): [] irq_exit_rcu+0xb0/0xd0 [18673.906919] bridge stp llc [18673.915432] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18673.918222] xfs libcrc32c [18673.927165] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18673.927166] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [18673.929871] sunrpc [18673.937342] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [18673.942817] intel_rapl_msr [18673.944916] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18673.963653] intel_rapl_common sb_edac [18673.966444] [18673.966445] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18673.971661] x86_pkg_temp_thermal [18673.975405] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18673.976895] coretemp [18673.984020] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18673.987331] kvm_intel kvm [18673.994454] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18673.996725] irqbypass i2c_i801 [18674.003849] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18674.006555] iTCO_wdt iTCO_vendor_support [18674.013678] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18674.016815] i2c_smbus crct10dif_pclmul [18674.023940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18674.027943] joydev crc32_pclmul [18674.036021] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18674.039852] ipmi_ssif ghash_clmulni_intel [18674.045589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18674.048814] mei_me mei [18674.055938] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18674.060028] lpc_ich sha512_ssse3 [18674.067153] Call Trace: [18674.067155] [18674.069597] acpi_ipmi mfd_core [18674.076722] ? watchdog_timer_fn+0x1b4/0x220 [18674.080032] wmi ipmi_si [18674.082477] ? __hrtimer_run_queues+0x1d3/0x360 [18674.084487] ipmi_devintf [18674.087627] ? hrtimer_interrupt+0x104/0x240 [18674.091888] ipmi_msghandler [18674.094418] ? __local_bh_enable+0x50/0x60 [18674.098943] ioatdma acpi_pad [18674.101561] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18674.105823] acpi_power_meter [18674.108702] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18674.112807] ixgbe igb [18674.115773] [18674.120990] mdio crc32c_intel [18674.123955] [18674.128912] nvme ptp [18674.131271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18674.133368] nvme_core i2c_algo_bit [18674.136427] ? queued_spin_lock_slowpath+0x2b/0x2e0 [18674.138516] pps_core dca fuse [18674.140806] do_raw_spin_lock+0xb5/0xc0 [18674.146106] [18674.146107] irq event stamp: 72217562 [18674.149591] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18674.154461] hardirqs last enabled at (72217561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18674.157515] try_get_cap_refs+0x159/0x880 [ceph] [18674.161344] hardirqs last disabled at (72217562): [] sysvec_apic_timer_interrupt+0xe/0x90 [18674.162839] __ceph_get_caps+0x4a1/0x760 [ceph] [18674.166490] softirqs last enabled at (72217560): [] __do_softirq+0x2be/0x371 [18674.171279] ? __file_remove_privs+0x89/0x110 [18674.181433] softirqs last disabled at (72217555): [] irq_exit_rcu+0xb0/0xd0 [18674.186045] ? wake_bit_function+0x70/0x70 [18674.195767] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18674.200294] ? ceph_write_iter+0x260/0x770 [ceph] [18674.208975] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18674.208976] Workqueue: events delayed_work [ceph] [18674.213327] ceph_write_iter+0x260/0x770 [ceph] [18674.221838] [18674.225933] ? lock_is_held_type+0xaa/0x120 [18674.235307] RIP: 0010:lock_acquire+0xf0/0x2b0 [18674.240005] vfs_write+0x324/0x4f0 [18674.247475] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18674.252177] ksys_pwrite64+0x65/0xa0 [18674.256696] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18674.258187] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18674.262363] [18674.262364] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18674.266715] do_syscall_64+0x38/0x80 [18674.270111] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18674.288851] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18674.292420] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18674.297639] RIP: 0033:0x7f48f0eeb93b [18674.302768] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18674.304259] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18674.311383] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18674.314953] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18674.322078] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18674.327141] ORIG_RAX: 0000000000000012 [18674.334264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18674.337834] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18674.344958] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18674.363696] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18674.370821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18674.376038] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18674.384115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18674.387947] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18674.393683] Call Trace: [18674.393685] [18674.400813] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18674.407941] ? watchdog_timer_fn+0x1b4/0x220 [18674.415072] [18674.422191] ? __hrtimer_run_queues+0x1d3/0x360 [18674.466116] ? hrtimer_interrupt+0x104/0x240 [18674.470387] ? __local_bh_enable+0x50/0x60 [18674.474479] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18674.479704] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18674.484659] [18674.486756] [18674.488855] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18674.494162] ? lock_acquire+0xf0/0x2b0 [18674.497913] ? igrab+0x43/0x50 [18674.500963] ? lock_release+0xc5/0x260 [18674.504709] _raw_spin_lock+0x30/0x40 [18674.508372] ? igrab+0x1d/0x50 [18674.511423] igrab+0x1d/0x50 [18674.514302] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18674.519452] delayed_work+0x1d8/0x280 [ceph] [18674.523733] ? process_scheduled_works+0x22a/0x510 [18674.528523] process_scheduled_works+0x2a4/0x510 [18674.533138] worker_thread+0x188/0x340 [18674.536887] ? rescuer_thread+0x310/0x310 [18674.540891] kthread+0xf5/0x130 [18674.544036] ? kthread_complete_and_exit+0x20/0x20 [18674.548820] ret_from_fork+0x31/0x40 [18674.552391] ? kthread_complete_and_exit+0x20/0x20 [18674.557176] ret_from_fork_asm+0x11/0x20 [18674.561101] [18701.783915] watchdog: BUG: soft lockup - CPU#0 stuck for 6303s! [dbench:78683] [18701.791135] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18701.795915] watchdog: BUG: soft lockup - CPU#7 stuck for 15300s! [kworker/7:4:78808] [18701.846591] irq event stamp: 26918596 [18701.846592] hardirqs last enabled at (26918595): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18701.854323] Modules linked in: [18701.857982] hardirqs last disabled at (26918596): [] sysvec_apic_timer_interrupt+0xe/0x90 [18701.868139] ceph [18701.871190] softirqs last enabled at (26918570): [] __do_softirq+0x2be/0x371 [18701.880912] libceph [18701.882838] softirqs last disabled at (26918565): [] irq_exit_rcu+0xb0/0xd0 [18701.891522] dns_resolver fscache [18701.893708] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18701.902216] netfs veth [18701.905529] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18701.914471] nft_chain_nat xt_MASQUERADE [18701.916915] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18701.924386] nf_nat nf_conntrack [18701.928312] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18701.933790] nf_defrag_ipv6 nf_defrag_ipv4 [18701.937013] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18701.955752] nft_compat nf_tables [18701.959844] [18701.965059] nfnetlink bridge [18701.968370] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18701.969861] stp llc [18701.972826] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18701.979951] xfs libcrc32c [18701.982132] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18701.989258] sunrpc [18701.991961] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18701.999086] intel_rapl_msr intel_rapl_common [18702.001183] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18702.008313] sb_edac x86_pkg_temp_thermal [18702.012666] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18702.019792] coretemp kvm_intel [18702.023796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18702.031873] kvm irqbypass [18702.035009] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18702.040746] i2c_i801 iTCO_wdt [18702.043452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18702.050575] iTCO_vendor_support i2c_smbus [18702.053626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18702.060750] crct10dif_pclmul joydev [18702.064840] Call Trace: [18702.064842] [18702.071965] crc32_pclmul ipmi_ssif [18702.075537] ? watchdog_timer_fn+0x1b4/0x220 [18702.077979] ghash_clmulni_intel mei_me [18702.079993] ? __hrtimer_run_queues+0x1d3/0x360 [18702.083474] mei lpc_ich [18702.087741] ? hrtimer_interrupt+0x104/0x240 [18702.091571] sha512_ssse3 [18702.096111] ? __local_bh_enable+0x50/0x60 [18702.098642] acpi_ipmi mfd_core [18702.102909] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18702.105523] wmi ipmi_si [18702.109613] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18702.112752] ipmi_devintf [18702.117969] [18702.120499] ipmi_msghandler [18702.125455] [18702.125457] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18702.128074] ioatdma acpi_pad [18702.130175] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18702.133049] acpi_power_meter ixgbe [18702.135148] do_raw_spin_lock+0xb5/0xc0 [18702.140450] igb mdio [18702.143415] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18702.148286] crc32c_intel nvme [18702.151790] try_get_cap_refs+0x159/0x880 [ceph] [18702.155617] ptp nvme_core [18702.157894] __ceph_get_caps+0x4a1/0x760 [ceph] [18702.162672] i2c_algo_bit pps_core [18702.165727] ? __file_remove_privs+0x89/0x110 [18702.170335] dca fuse [18702.173039] ? wake_bit_function+0x70/0x70 [18702.177561] [18702.177562] irq event stamp: 72440728 [18702.180961] ? ceph_write_iter+0x260/0x770 [ceph] [18702.185316] hardirqs last enabled at (72440727): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18702.187596] ceph_write_iter+0x260/0x770 [ceph] [18702.191689] hardirqs last disabled at (72440728): [] sysvec_apic_timer_interrupt+0xe/0x90 [18702.193183] ? lock_is_held_type+0xaa/0x120 [18702.196836] softirqs last enabled at (72440726): [] __do_softirq+0x2be/0x371 [18702.201537] vfs_write+0x324/0x4f0 [18702.211693] softirqs last disabled at (72440721): [] irq_exit_rcu+0xb0/0xd0 [18702.216221] ksys_pwrite64+0x65/0xa0 [18702.225940] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18702.230118] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18702.238801] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18702.242201] do_syscall_64+0x38/0x80 [18702.250712] Workqueue: events delayed_work [ceph] [18702.254283] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18702.263659] [18702.263660] RIP: 0010:lock_release+0x14e/0x260 [18702.268790] RIP: 0033:0x7f48f0eeb93b [18702.276261] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [18702.279830] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18702.284527] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [18702.289572] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18702.291063] [18702.291063] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [18702.295500] ORIG_RAX: 0000000000000012 [18702.299070] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [18702.317810] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18702.336547] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [18702.341762] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18702.346982] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [18702.348471] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18702.355610] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [18702.359444] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18702.366567] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18702.373693] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18702.380817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18702.387947] [18702.395064] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18702.446694] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18702.453817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18702.460941] Call Trace: [18702.463388] [18702.465397] ? watchdog_timer_fn+0x1b4/0x220 [18702.469661] ? __hrtimer_run_queues+0x1d3/0x360 [18702.474187] ? hrtimer_interrupt+0x104/0x240 [18702.478457] ? __local_bh_enable+0x50/0x60 [18702.482550] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18702.487775] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18702.492732] [18702.494829] [18702.496927] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18702.502232] ? igrab+0x43/0x50 [18702.505285] ? lock_release+0x14e/0x260 [18702.509120] ? lock_release+0xc5/0x260 [18702.512867] _raw_spin_unlock+0x1b/0x40 [18702.516704] igrab+0x43/0x50 [18702.519598] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18702.524740] delayed_work+0x1d8/0x280 [ceph] [18702.529023] ? process_scheduled_works+0x22a/0x510 [18702.533812] process_scheduled_works+0x2a4/0x510 [18702.538430] worker_thread+0x188/0x340 [18702.542178] ? rescuer_thread+0x310/0x310 [18702.546190] kthread+0xf5/0x130 [18702.549335] ? kthread_complete_and_exit+0x20/0x20 [18702.554119] ret_from_fork+0x31/0x40 [18702.557700] ? kthread_complete_and_exit+0x20/0x20 [18702.562482] ret_from_fork_asm+0x11/0x20 [18702.566406] [18729.784422] watchdog: BUG: soft lockup - CPU#0 stuck for 6329s! [dbench:78683] [18729.791640] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18729.796422] watchdog: BUG: soft lockup - CPU#7 stuck for 15326s! [kworker/7:4:78808] [18729.847095] irq event stamp: 26976700 [18729.847096] hardirqs last enabled at (26976699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18729.854830] Modules linked in: [18729.858488] hardirqs last disabled at (26976700): [] sysvec_apic_timer_interrupt+0xe/0x90 [18729.868645] ceph [18729.871695] softirqs last enabled at (26976662): [] __do_softirq+0x2be/0x371 [18729.881418] libceph dns_resolver [18729.883342] softirqs last disabled at (26976657): [] irq_exit_rcu+0xb0/0xd0 [18729.892028] fscache netfs [18729.895339] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18729.903850] veth [18729.906552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18729.915496] nft_chain_nat xt_MASQUERADE [18729.917422] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18729.924893] nf_nat [18729.928817] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18729.934295] nf_conntrack nf_defrag_ipv6 [18729.936392] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18729.955130] nf_defrag_ipv4 nft_compat [18729.959049] [18729.959050] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18729.964264] nf_tables nfnetlink [18729.968009] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18729.969500] bridge stp [18729.976625] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18729.979848] llc xfs [18729.986972] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18729.989417] libcrc32c [18729.996541] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18729.998724] sunrpc intel_rapl_msr [18730.005850] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18730.008206] intel_rapl_common sb_edac [18730.015331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18730.018727] x86_pkg_temp_thermal coretemp [18730.026819] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18730.030567] kvm_intel kvm [18730.036303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18730.040396] irqbypass i2c_i801 [18730.047519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18730.050223] iTCO_wdt iTCO_vendor_support [18730.057348] Call Trace: [18730.057349] [18730.060485] i2c_smbus crct10dif_pclmul [18730.067608] ? watchdog_timer_fn+0x1b4/0x220 [18730.071613] joydev crc32_pclmul [18730.074058] ? __hrtimer_run_queues+0x1d3/0x360 [18730.076067] ipmi_ssif [18730.079901] ? hrtimer_interrupt+0x104/0x240 [18730.084163] ghash_clmulni_intel mei_me [18730.087386] ? __local_bh_enable+0x50/0x60 [18730.091909] mei lpc_ich [18730.094271] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18730.098532] sha512_ssse3 acpi_ipmi [18730.102363] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18730.106453] mfd_core wmi [18730.108983] [18730.114202] ipmi_si ipmi_devintf [18730.117686] [18730.117687] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18730.122642] ipmi_msghandler ioatdma [18730.125264] ? queued_spin_lock_slowpath+0x16/0x2e0 [18730.127356] acpi_pad acpi_power_meter [18730.130671] do_raw_spin_lock+0xb5/0xc0 [18730.132766] ixgbe igb [18730.138072] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18730.141639] mdio crc32c_intel [18730.146514] try_get_cap_refs+0x159/0x880 [ceph] [18730.150254] nvme ptp [18730.154105] __ceph_get_caps+0x4a1/0x760 [ceph] [18730.156462] nvme_core i2c_algo_bit [18730.161249] ? __file_remove_privs+0x89/0x110 [18730.164294] pps_core dca [18730.168908] ? wake_bit_function+0x70/0x70 [18730.171176] fuse [18730.175704] ? ceph_write_iter+0x260/0x770 [ceph] [18730.179184] irq event stamp: 72662418 [18730.183536] ceph_write_iter+0x260/0x770 [ceph] [18730.186152] hardirqs last enabled at (72662417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18730.190247] ? lock_is_held_type+0xaa/0x120 [18730.192167] hardirqs last disabled at (72662418): [] sysvec_apic_timer_interrupt+0xe/0x90 [18730.196868] vfs_write+0x324/0x4f0 [18730.200523] softirqs last enabled at (72662416): [] __do_softirq+0x2be/0x371 [18730.205052] ksys_pwrite64+0x65/0xa0 [18730.215206] softirqs last disabled at (72662411): [] irq_exit_rcu+0xb0/0xd0 [18730.219383] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18730.229112] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18730.232514] do_syscall_64+0x38/0x80 [18730.241197] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18730.244767] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18730.253278] Workqueue: events delayed_work [ceph] [18730.258409] RIP: 0033:0x7f48f0eeb93b [18730.267787] [18730.267788] RIP: 0010:lock_acquire+0xf0/0x2b0 [18730.271356] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18730.278828] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18730.283872] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18730.288570] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18730.292141] ORIG_RAX: 0000000000000012 [18730.293631] [18730.297982] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18730.316718] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18730.335457] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18730.340674] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18730.345891] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18730.349721] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18730.351213] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18730.358337] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18730.365461] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18730.372586] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18730.379714] [18730.386833] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18730.432715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18730.438454] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18730.445579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18730.452703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18730.459826] Call Trace: [18730.462271] [18730.464283] ? watchdog_timer_fn+0x1b4/0x220 [18730.468547] ? __hrtimer_run_queues+0x1d3/0x360 [18730.473073] ? hrtimer_interrupt+0x104/0x240 [18730.477342] ? __local_bh_enable+0x50/0x60 [18730.481434] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18730.486650] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18730.491607] [18730.493704] [18730.495803] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18730.501136] ? lock_acquire+0xf0/0x2b0 [18730.504886] ? igrab+0x43/0x50 [18730.507936] ? lock_release+0xc5/0x260 [18730.511681] _raw_spin_lock+0x30/0x40 [18730.515336] ? igrab+0x1d/0x50 [18730.518388] igrab+0x1d/0x50 [18730.521267] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18730.526417] delayed_work+0x1d8/0x280 [ceph] [18730.530698] ? process_scheduled_works+0x22a/0x510 [18730.535488] process_scheduled_works+0x2a4/0x510 [18730.540113] worker_thread+0x188/0x340 [18730.543860] ? rescuer_thread+0x310/0x310 [18730.547864] kthread+0xf5/0x130 [18730.551002] ? kthread_complete_and_exit+0x20/0x20 [18730.555786] ret_from_fork+0x31/0x40 [18730.559357] ? kthread_complete_and_exit+0x20/0x20 [18730.564143] ret_from_fork_asm+0x11/0x20 [18730.568073] [18757.784928] watchdog: BUG: soft lockup - CPU#0 stuck for 6355s! [dbench:78683] [18757.792144] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18757.796928] watchdog: BUG: soft lockup - CPU#7 stuck for 15352s! [kworker/7:4:78808] [18757.847601] irq event stamp: 27034946 [18757.855334] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [18757.858992] hardirqs last enabled at (27034945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18757.867164] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18757.877322] hardirqs last disabled at (27034946): [] sysvec_apic_timer_interrupt+0xe/0x90 [18757.877325] nft_compat nf_tables nfnetlink bridge stp llc xfs [18757.894083] softirqs last enabled at (27034908): [] __do_softirq+0x2be/0x371 [18757.899908] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [18757.908604] softirqs last disabled at (27034903): [] irq_exit_rcu+0xb0/0xd0 [18757.916947] coretemp kvm_intel kvm irqbypass i2c_i801 [18757.925459] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18757.930605] iTCO_wdt iTCO_vendor_support i2c_smbus [18757.939551] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18757.944421] crct10dif_pclmul joydev crc32_pclmul [18757.951892] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18757.951894] ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich [18757.962085] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18757.967820] sha512_ssse3 acpi_ipmi [18757.986559] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18757.990043] mfd_core [18757.990045] [18757.995262] wmi ipmi_si [18757.997530] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18757.999023] ipmi_devintf [18758.001551] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18758.008677] ipmi_msghandler [18758.011295] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18758.018419] ioatdma acpi_pad [18758.021297] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18758.028436] acpi_power_meter ixgbe [18758.031400] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18758.038525] igb mdio [18758.042008] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18758.049133] crc32c_intel nvme [18758.051404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18758.059481] ptp nvme_core [18758.062532] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18758.068270] i2c_algo_bit pps_core [18758.070974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18758.078099] dca fuse [18758.081496] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18758.088620] [18758.088620] irq event stamp: 72884180 [18758.090890] Call Trace: [18758.098014] hardirqs last enabled at (72884179): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18758.099506] [18758.103162] hardirqs last disabled at (72884180): [] sysvec_apic_timer_interrupt+0xe/0x90 [18758.105611] ? watchdog_timer_fn+0x1b4/0x220 [18758.115763] softirqs last enabled at (72884178): [] __do_softirq+0x2be/0x371 [18758.117775] ? __hrtimer_run_queues+0x1d3/0x360 [18758.127500] softirqs last disabled at (72884173): [] irq_exit_rcu+0xb0/0xd0 [18758.131765] ? hrtimer_interrupt+0x104/0x240 [18758.140449] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18758.144972] ? __local_bh_enable+0x50/0x60 [18758.153482] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18758.157749] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18758.167125] Workqueue: events delayed_work [ceph] [18758.171215] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18758.178685] [18758.178686] RIP: 0010:lock_acquire+0xf0/0x2b0 [18758.183902] [18758.188607] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18758.193566] [18758.195056] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18758.199409] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18758.201504] [18758.201505] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18758.220247] ? queued_spin_lock_slowpath+0x16/0x2e0 [18758.222339] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18758.227560] do_raw_spin_lock+0xb5/0xc0 [18758.232860] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18758.234355] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18758.241476] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18758.246352] try_get_cap_refs+0x159/0x880 [ceph] [18758.253472] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18758.257326] __ceph_get_caps+0x4a1/0x760 [ceph] [18758.264444] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18758.269233] ? __file_remove_privs+0x89/0x110 [18758.276351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18758.280964] ? wake_bit_function+0x70/0x70 [18758.288087] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18758.292614] ? ceph_write_iter+0x260/0x770 [ceph] [18758.300688] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18758.305040] ceph_write_iter+0x260/0x770 [ceph] [18758.310776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18758.314873] ? lock_is_held_type+0xaa/0x120 [18758.321991] Call Trace: [18758.321992] [18758.326692] vfs_write+0x324/0x4f0 [18758.333814] ? watchdog_timer_fn+0x1b4/0x220 [18758.338340] ksys_pwrite64+0x65/0xa0 [18758.345463] ? __hrtimer_run_queues+0x1d3/0x360 [18758.349640] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18758.352086] ? hrtimer_interrupt+0x104/0x240 [18758.354094] do_syscall_64+0x38/0x80 [18758.357490] ? __local_bh_enable+0x50/0x60 [18758.361755] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18758.365333] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18758.369852] RIP: 0033:0x7f48f0eeb93b [18758.374982] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18758.379244] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18758.382815] [18758.386905] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18758.391950] [18758.391952] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18758.397168] ORIG_RAX: 0000000000000012 [18758.400744] ? lock_acquire+0xf0/0x2b0 [18758.405695] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18758.424436] ? igrab+0x43/0x50 [18758.426530] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18758.431749] ? lock_release+0xc5/0x260 [18758.433847] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18758.439151] _raw_spin_lock+0x30/0x40 [18758.442981] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18758.446725] ? igrab+0x1d/0x50 [18758.453849] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18758.456901] igrab+0x1d/0x50 [18758.464030] [18758.467771] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18758.506045] delayed_work+0x1d8/0x280 [ceph] [18758.510338] ? process_scheduled_works+0x22a/0x510 [18758.515123] process_scheduled_works+0x2a4/0x510 [18758.519740] worker_thread+0x188/0x340 [18758.523489] ? rescuer_thread+0x310/0x310 [18758.527501] kthread+0xf5/0x130 [18758.530646] ? kthread_complete_and_exit+0x20/0x20 [18758.535431] ret_from_fork+0x31/0x40 [18758.539010] ? kthread_complete_and_exit+0x20/0x20 [18758.543792] ret_from_fork_asm+0x11/0x20 [18758.547717] [18785.785434] watchdog: BUG: soft lockup - CPU#0 stuck for 6381s! [dbench:78683] [18785.792652] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18785.797434] watchdog: BUG: soft lockup - CPU#7 stuck for 15378s! [kworker/7:4:78808] [18785.848106] irq event stamp: 27092922 [18785.855841] Modules linked in: ceph libceph [18785.859498] hardirqs last enabled at (27092921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18785.863675] dns_resolver fscache netfs [18785.873832] hardirqs last disabled at (27092922): [] sysvec_apic_timer_interrupt+0xe/0x90 [18785.877664] veth nft_chain_nat xt_MASQUERADE nf_nat [18785.887387] softirqs last enabled at (27092894): [] __do_softirq+0x2be/0x371 [18785.892344] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [18785.901028] softirqs last disabled at (27092889): [] irq_exit_rcu+0xb0/0xd0 [18785.901030] nft_compat nf_tables nfnetlink bridge [18785.914757] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18785.919540] stp llc [18785.928486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18785.930669] xfs libcrc32c sunrpc [18785.938141] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18785.941452] intel_rapl_msr intel_rapl_common [18785.946928] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18785.946929] sb_edac [18785.951278] x86_pkg_temp_thermal coretemp [18785.970018] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18785.972200] kvm_intel kvm [18785.976291] [18785.981510] irqbypass i2c_i801 [18785.984213] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18785.985704] iTCO_wdt iTCO_vendor_support [18785.988841] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18785.995966] i2c_smbus crct10dif_pclmul [18785.999970] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18786.007093] joydev crc32_pclmul [18786.010924] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18786.018047] ipmi_ssif [18786.021272] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18786.028396] ghash_clmulni_intel mei_me [18786.030755] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18786.037877] mei lpc_ich [18786.041708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18786.049787] sha512_ssse3 acpi_ipmi [18786.052318] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18786.058055] mfd_core wmi [18786.061537] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18786.068663] ipmi_si ipmi_devintf [18786.071279] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18786.078405] ipmi_msghandler ioatdma [18786.081715] Call Trace: [18786.081716] [18786.088838] acpi_pad acpi_power_meter [18786.092412] ? watchdog_timer_fn+0x1b4/0x220 [18786.094854] ixgbe igb [18786.096865] ? __hrtimer_run_queues+0x1d3/0x360 [18786.100609] mdio crc32c_intel [18786.104876] ? hrtimer_interrupt+0x104/0x240 [18786.107229] nvme ptp [18786.111754] ? __local_bh_enable+0x50/0x60 [18786.114805] nvme_core i2c_algo_bit [18786.119088] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18786.121374] pps_core dca [18786.125466] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18786.128950] fuse [18786.134167] [18786.136785] irq event stamp: 73106264 [18786.141740] [18786.143665] hardirqs last enabled at (73106263): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18786.145764] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18786.149420] hardirqs last disabled at (73106264): [] sysvec_apic_timer_interrupt+0xe/0x90 [18786.151521] ? queued_spin_lock_slowpath+0x16/0x2e0 [18786.161675] softirqs last enabled at (73106262): [] __do_softirq+0x2be/0x371 [18786.166981] do_raw_spin_lock+0xb5/0xc0 [18786.176704] softirqs last disabled at (73106257): [] irq_exit_rcu+0xb0/0xd0 [18786.181576] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18786.190260] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18786.194090] try_get_cap_refs+0x159/0x880 [ceph] [18786.202600] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18786.202601] Workqueue: events delayed_work [ceph] [18786.207388] __ceph_get_caps+0x4a1/0x760 [ceph] [18786.216762] [18786.216763] RIP: 0010:lock_acquire+0xf0/0x2b0 [18786.221376] ? __file_remove_privs+0x89/0x110 [18786.228844] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18786.233540] ? wake_bit_function+0x70/0x70 [18786.238065] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18786.239559] ? ceph_write_iter+0x260/0x770 [ceph] [18786.243906] [18786.243907] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18786.248256] ceph_write_iter+0x260/0x770 [ceph] [18786.266994] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18786.271088] ? lock_is_held_type+0xaa/0x120 [18786.276301] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18786.281003] vfs_write+0x324/0x4f0 [18786.282491] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18786.289618] ksys_pwrite64+0x65/0xa0 [18786.294140] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18786.301264] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18786.305441] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18786.312566] do_syscall_64+0x38/0x80 [18786.315962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18786.323087] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18786.326657] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18786.333782] RIP: 0033:0x7f48f0eeb93b [18786.338912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18786.346989] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18786.350561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18786.356298] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18786.361342] Call Trace: [18786.368467] ORIG_RAX: 0000000000000012 [18786.372036] [18786.379161] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18786.397900] ? watchdog_timer_fn+0x1b4/0x220 [18786.405023] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18786.410241] ? __hrtimer_run_queues+0x1d3/0x360 [18786.412684] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18786.416519] ? hrtimer_interrupt+0x104/0x240 [18786.418524] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18786.425650] ? __local_bh_enable+0x50/0x60 [18786.429914] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18786.437041] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18786.441567] [18786.448686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18786.483647] [18786.485745] [18786.487843] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18786.493151] ? lock_acquire+0xf0/0x2b0 [18786.496902] ? igrab+0x43/0x50 [18786.499950] ? lock_release+0xc5/0x260 [18786.503697] _raw_spin_lock+0x30/0x40 [18786.507359] ? igrab+0x1d/0x50 [18786.510410] igrab+0x1d/0x50 [18786.513291] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18786.518438] delayed_work+0x1d8/0x280 [ceph] [18786.522721] ? process_scheduled_works+0x22a/0x510 [18786.527511] process_scheduled_works+0x2a4/0x510 [18786.532128] worker_thread+0x188/0x340 [18786.535876] ? rescuer_thread+0x310/0x310 [18786.539878] kthread+0xf5/0x130 [18786.543015] ? kthread_complete_and_exit+0x20/0x20 [18786.547801] ret_from_fork+0x31/0x40 [18786.551372] ? kthread_complete_and_exit+0x20/0x20 [18786.556155] ret_from_fork_asm+0x11/0x20 [18786.560078] [18813.785940] watchdog: BUG: soft lockup - CPU#0 stuck for 6407s! [dbench:78683] [18813.793156] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18813.797940] watchdog: BUG: soft lockup - CPU#7 stuck for 15404s! [kworker/7:4:78808] [18813.848623] irq event stamp: 27151022 [18813.856355] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE [18813.860012] hardirqs last enabled at (27151021): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18813.869397] nf_nat nf_conntrack nf_defrag_ipv6 [18813.879556] hardirqs last disabled at (27151022): [] sysvec_apic_timer_interrupt+0xe/0x90 [18813.884079] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [18813.893803] softirqs last enabled at (27150994): [] __do_softirq+0x2be/0x371 [18813.900582] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [18813.909265] softirqs last disabled at (27150989): [] irq_exit_rcu+0xb0/0xd0 [18813.915435] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [18813.923947] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18813.929614] kvm irqbypass i2c_i801 [18813.938559] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18813.942042] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [18813.949514] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18813.956465] crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me [18813.961942] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18813.967766] mei lpc_ich sha512_ssse3 [18813.986503] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18813.986505] acpi_ipmi [18813.986506] [18813.990161] mfd_core wmi [18813.995378] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18813.997737] ipmi_si ipmi_devintf [18813.999226] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18814.001845] ipmi_msghandler [18814.008969] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18814.012279] ioatdma acpi_pad [18814.019404] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18814.022282] acpi_power_meter ixgbe [18814.029405] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18814.032369] igb mdio [18814.039493] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18814.042978] crc32c_intel [18814.050102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18814.052371] nvme ptp [18814.060451] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18814.063067] nvme_core i2c_algo_bit [18814.068804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18814.071076] pps_core [18814.078199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18814.081684] dca fuse [18814.088807] Call Trace: [18814.088810] [18814.091077] [18814.091078] irq event stamp: 73329002 [18814.098206] ? watchdog_timer_fn+0x1b4/0x220 [18814.100472] hardirqs last enabled at (73329001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18814.102919] ? __hrtimer_run_queues+0x1d3/0x360 [18814.104928] hardirqs last disabled at (73329002): [] sysvec_apic_timer_interrupt+0xe/0x90 [18814.106423] ? hrtimer_interrupt+0x104/0x240 [18814.110075] softirqs last enabled at (73329000): [] __do_softirq+0x2be/0x371 [18814.114341] ? __local_bh_enable+0x50/0x60 [18814.124498] softirqs last disabled at (73328995): [] irq_exit_rcu+0xb0/0xd0 [18814.129024] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18814.138746] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18814.143011] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18814.151695] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18814.155784] [18814.164298] Workqueue: events delayed_work [ceph] [18814.169512] [18814.169513] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18814.178891] [18814.178892] RIP: 0010:lock_acquire+0xf0/0x2b0 [18814.183851] ? queued_spin_lock_slowpath+0x16/0x2e0 [18814.191337] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18814.193436] do_raw_spin_lock+0xb5/0xc0 [18814.198131] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18814.200230] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18814.205532] [18814.205533] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18814.207027] try_get_cap_refs+0x159/0x880 [ceph] [18814.211374] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18814.216250] __ceph_get_caps+0x4a1/0x760 [ceph] [18814.234981] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18814.234983] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18814.238818] ? __file_remove_privs+0x89/0x110 [18814.244029] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18814.244031] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18814.248816] ? wake_bit_function+0x70/0x70 [18814.250304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18814.257432] ? ceph_write_iter+0x260/0x770 [ceph] [18814.262040] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18814.269164] ceph_write_iter+0x260/0x770 [ceph] [18814.273689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18814.280816] ? lock_is_held_type+0xaa/0x120 [18814.287937] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18814.292289] vfs_write+0x324/0x4f0 [18814.299410] Call Trace: [18814.299411] [18814.307494] ksys_pwrite64+0x65/0xa0 [18814.311580] ? watchdog_timer_fn+0x1b4/0x220 [18814.317319] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18814.322032] ? __hrtimer_run_queues+0x1d3/0x360 [18814.329157] do_syscall_64+0x38/0x80 [18814.333683] ? hrtimer_interrupt+0x104/0x240 [18814.340806] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18814.344981] ? __local_bh_enable+0x50/0x60 [18814.352106] RIP: 0033:0x7f48f0eeb93b [18814.355506] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18814.357950] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18814.359958] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18814.363528] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18814.367794] [18814.372923] ORIG_RAX: 0000000000000012 [18814.377449] [18814.377450] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18814.381017] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18814.385288] ? lock_acquire+0xf0/0x2b0 [18814.390344] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18814.394437] ? igrab+0x43/0x50 [18814.398004] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18814.403223] ? lock_release+0xc5/0x260 [18814.421960] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18814.426920] _raw_spin_lock+0x30/0x40 [18814.432134] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18814.434234] ? igrab+0x1d/0x50 [18814.438070] [18814.440162] igrab+0x1d/0x50 [18814.503404] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18814.508555] delayed_work+0x1d8/0x280 [ceph] [18814.512835] ? process_scheduled_works+0x22a/0x510 [18814.517628] process_scheduled_works+0x2a4/0x510 [18814.522249] worker_thread+0x188/0x340 [18814.525998] ? rescuer_thread+0x310/0x310 [18814.530002] kthread+0xf5/0x130 [18814.533141] ? kthread_complete_and_exit+0x20/0x20 [18814.537923] ret_from_fork+0x31/0x40 [18814.541493] ? kthread_complete_and_exit+0x20/0x20 [18814.546278] ret_from_fork_asm+0x11/0x20 [18814.550202] [18841.786447] watchdog: BUG: soft lockup - CPU#0 stuck for 6433s! [dbench:78683] [18841.793663] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18841.798447] watchdog: BUG: soft lockup - CPU#7 stuck for 15430s! [kworker/7:4:78808] [18841.849118] irq event stamp: 27208994 [18841.856853] Modules linked in: ceph libceph [18841.860508] hardirqs last enabled at (27208993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18841.864687] dns_resolver fscache netfs [18841.874842] hardirqs last disabled at (27208994): [] sysvec_apic_timer_interrupt+0xe/0x90 [18841.874844] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [18841.888398] softirqs last enabled at (27208966): [] __do_softirq+0x2be/0x371 [18841.894483] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [18841.903166] softirqs last disabled at (27208961): [] irq_exit_rcu+0xb0/0xd0 [18841.908212] nf_tables nfnetlink bridge [18841.916721] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18841.920551] stp llc [18841.929497] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18841.931681] xfs libcrc32c sunrpc intel_rapl_msr [18841.939152] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18841.943763] intel_rapl_common sb_edac x86_pkg_temp_thermal [18841.949240] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18841.954803] coretemp kvm_intel [18841.973541] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18841.973542] kvm [18841.973543] [18841.976680] irqbypass i2c_i801 [18841.981896] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18841.983735] iTCO_wdt iTCO_vendor_support [18841.985223] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18841.988362] i2c_smbus crct10dif_pclmul [18841.995486] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18841.999490] joydev [18842.006613] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18842.010445] crc32_pclmul ipmi_ssif [18842.017568] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18842.019667] ghash_clmulni_intel mei_me [18842.026790] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18842.030275] mei lpc_ich [18842.037398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18842.041230] sha512_ssse3 acpi_ipmi [18842.049307] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18842.051845] mfd_core [18842.057584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18842.061068] wmi ipmi_si [18842.068192] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18842.070462] ipmi_devintf ipmi_msghandler [18842.077586] Call Trace: [18842.077587] [18842.080126] ioatdma acpi_pad [18842.087250] ? watchdog_timer_fn+0x1b4/0x220 [18842.091254] acpi_power_meter ixgbe [18842.093699] ? __hrtimer_run_queues+0x1d3/0x360 [18842.095709] igb mdio [18842.098676] ? hrtimer_interrupt+0x104/0x240 [18842.102938] crc32c_intel nvme [18842.106421] ? __local_bh_enable+0x50/0x60 [18842.110946] ptp nvme_core [18842.113218] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18842.117480] i2c_algo_bit pps_core [18842.120532] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18842.124622] dca fuse [18842.127325] [18842.132543] [18842.132543] irq event stamp: 73552314 [18842.135941] [18842.140898] hardirqs last enabled at (73552313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18842.143170] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18842.145266] hardirqs last disabled at (73552314): [] sysvec_apic_timer_interrupt+0xe/0x90 [18842.146761] ? queued_spin_lock_slowpath+0x18/0x2e0 [18842.150414] softirqs last enabled at (73552312): [] __do_softirq+0x2be/0x371 [18842.152513] do_raw_spin_lock+0xb5/0xc0 [18842.162669] softirqs last disabled at (73552307): [] irq_exit_rcu+0xb0/0xd0 [18842.167974] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18842.177698] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18842.182569] try_get_cap_refs+0x159/0x880 [ceph] [18842.191252] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18842.195089] __ceph_get_caps+0x4a1/0x760 [ceph] [18842.203593] Workqueue: events delayed_work [ceph] [18842.208381] ? __file_remove_privs+0x89/0x110 [18842.217756] [18842.217757] RIP: 0010:lock_acquire+0xf0/0x2b0 [18842.222366] ? wake_bit_function+0x70/0x70 [18842.229848] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [18842.234373] ? ceph_write_iter+0x260/0x770 [ceph] [18842.239068] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [18842.243419] ceph_write_iter+0x260/0x770 [ceph] [18842.244909] [18842.244910] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [18842.249263] ? lock_is_held_type+0xaa/0x120 [18842.253349] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18842.272090] vfs_write+0x324/0x4f0 [18842.276785] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [18842.282007] ksys_pwrite64+0x65/0xa0 [18842.286526] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [18842.288017] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18842.295140] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [18842.299320] do_syscall_64+0x38/0x80 [18842.306444] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18842.309854] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18842.316973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18842.320545] RIP: 0033:0x7f48f0eeb93b [18842.327668] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18842.332798] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18842.339924] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18842.343493] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18842.351571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18842.356615] ORIG_RAX: 0000000000000012 [18842.362353] Call Trace: [18842.362354] [18842.365923] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18842.373049] ? watchdog_timer_fn+0x1b4/0x220 [18842.391785] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18842.398910] ? __hrtimer_run_queues+0x1d3/0x360 [18842.404131] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18842.411262] ? hrtimer_interrupt+0x104/0x240 [18842.415091] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18842.417534] ? __local_bh_enable+0x50/0x60 [18842.419545] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18842.426673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18842.430938] [18842.438059] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18842.484667] [18842.486766] [18842.488864] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18842.494180] ? lock_acquire+0xf0/0x2b0 [18842.497929] ? igrab+0x43/0x50 [18842.500981] ? lock_release+0xc5/0x260 [18842.504725] _raw_spin_lock+0x30/0x40 [18842.508381] ? igrab+0x1d/0x50 [18842.511431] igrab+0x1d/0x50 [18842.514310] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18842.519459] delayed_work+0x1d8/0x280 [ceph] [18842.523739] ? process_scheduled_works+0x22a/0x510 [18842.528522] process_scheduled_works+0x2a4/0x510 [18842.533145] worker_thread+0x188/0x340 [18842.536896] ? rescuer_thread+0x310/0x310 [18842.540897] kthread+0xf5/0x130 [18842.544035] ? kthread_complete_and_exit+0x20/0x20 [18842.548821] ret_from_fork+0x31/0x40 [18842.552391] ? kthread_complete_and_exit+0x20/0x20 [18842.557175] ret_from_fork_asm+0x11/0x20 [18842.561097] [18842.812831] rcu: INFO: rcu_preempt self-detected stall on CPU [18842.818594] rcu: 0-....: (6751189 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=3359303 [18842.828933] rcu: (t=6906080 jiffies g=1309609 q=260736 ncpus=8) [18842.834939] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18842.843881] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18842.851354] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18842.856856] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18842.875594] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18842.880812] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18842.887936] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18842.895060] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18842.902183] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18842.909307] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18842.916431] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18842.924509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18842.930248] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18842.937370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18842.944494] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18842.951619] Call Trace: [18842.954064] [18842.956075] ? rcu_dump_cpu_stacks+0xd1/0x180 [18842.960436] ? rcu_sched_clock_irq+0x633/0x1390 [18842.964968] ? lock_is_held_type+0xaa/0x120 [18842.969155] ? lock_is_held_type+0xaa/0x120 [18842.973341] ? tick_sched_do_timer+0xa0/0xa0 [18842.977612] ? rcu_read_lock_sched_held+0x56/0x90 [18842.982311] ? tick_sched_do_timer+0xa0/0xa0 [18842.986582] ? update_process_times+0x5f/0x90 [18842.990932] ? tick_sched_handle.isra.16+0x1e/0x50 [18842.995717] ? tick_sched_timer+0x6f/0x80 [18842.999722] ? __hrtimer_run_queues+0x1d3/0x360 [18843.004260] ? hrtimer_interrupt+0x104/0x240 [18843.008526] ? __local_bh_enable+0x50/0x60 [18843.012631] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18843.017867] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18843.022826] [18843.024923] [18843.027022] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18843.032343] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18843.037227] do_raw_spin_lock+0xb5/0xc0 [18843.041063] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18843.045875] try_get_cap_refs+0x159/0x880 [ceph] [18843.050519] __ceph_get_caps+0x4a1/0x760 [ceph] [18843.055067] ? __file_remove_privs+0x89/0x110 [18843.059418] ? wake_bit_function+0x70/0x70 [18843.063514] ? ceph_write_iter+0x260/0x770 [ceph] [18843.068233] ceph_write_iter+0x260/0x770 [ceph] [18843.072781] ? lock_is_held_type+0xaa/0x120 [18843.076965] vfs_write+0x324/0x4f0 [18843.080375] ksys_pwrite64+0x65/0xa0 [18843.083953] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18843.089087] do_syscall_64+0x38/0x80 [18843.092664] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18843.097708] RIP: 0033:0x7f48f0eeb93b [18843.101278] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18843.120024] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [18843.127582] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18843.134705] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18843.141829] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18843.148955] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18843.156078] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18843.163215] [18843.165403] Sending NMI from CPU 0 to CPUs 7: [18843.169757] NMI backtrace for cpu 7 [18843.169758] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18843.169760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18843.169760] Workqueue: events delayed_work [ceph] [18843.169775] RIP: 0010:__lock_acquire+0x6bb/0x1e20 [18843.169776] Code: 20 49 89 d7 25 00 80 04 00 3d 00 00 04 00 0f 84 4d 05 00 00 44 8b 15 dc 27 54 01 45 85 d2 0f 84 ba 00 00 00 f6 45 22 10 75 27 <8b> 83 50 0a 00 00 4c 89 bb 48 0a 00 00 83 c0 01 89 83 50 0a 00 00 [18843.169778] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [18843.169779] RAX: ffffffff82e139e8 RBX: ffff88810debaa80 RCX: 000000005553a634 [18843.169780] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [18843.169780] RBP: ffff88810debb550 R08: 0000000000005c53 R09: 0000000000000003 [18843.169781] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8360d858 [18843.169781] R13: 0000000000000003 R14: 0000000000000000 R15: c5dbdfc36fd10127 [18843.169782] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18843.169783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18843.169784] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18843.169784] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18843.169785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18843.169786] Call Trace: [18843.169786] [18843.169787] ? nmi_cpu_backtrace+0xad/0x120 [18843.169790] ? nmi_cpu_backtrace_handler+0x11/0x20 [18843.169792] ? nmi_handle+0xd4/0x260 [18843.169796] ? default_do_nmi+0x49/0xf0 [18843.169797] ? exc_nmi+0x11e/0x150 [18843.169799] ? end_repeat_nmi+0x16/0x67 [18843.169802] ? __lock_acquire+0x6bb/0x1e20 [18843.169804] ? __lock_acquire+0x6bb/0x1e20 [18843.169805] ? __lock_acquire+0x6bb/0x1e20 [18843.169806] [18843.169807] [18843.169808] lock_acquire+0xcb/0x2b0 [18843.169810] ? igrab+0x1d/0x50 [18843.169811] ? igrab+0x43/0x50 [18843.169812] ? lock_release+0xc5/0x260 [18843.169814] _raw_spin_lock+0x30/0x40 [18843.169816] ? igrab+0x1d/0x50 [18843.169816] igrab+0x1d/0x50 [18843.169818] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18843.169831] delayed_work+0x1d8/0x280 [ceph] [18843.169863] ? process_scheduled_works+0x22a/0x510 [18843.169865] process_scheduled_works+0x2a4/0x510 [18843.169869] worker_thread+0x188/0x340 [18843.169871] ? rescuer_thread+0x310/0x310 [18843.169874] kthread+0xf5/0x130 [18843.169876] ? kthread_complete_and_exit+0x20/0x20 [18843.169878] ret_from_fork+0x31/0x40 [18843.169880] ? kthread_complete_and_exit+0x20/0x20 [18843.169882] ret_from_fork_asm+0x11/0x20 [18843.169886] [18869.786954] watchdog: BUG: soft lockup - CPU#0 stuck for 6459s! [dbench:78683] [18869.794168] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18869.798954] watchdog: BUG: soft lockup - CPU#7 stuck for 15456s! [kworker/7:4:78808] [18869.849627] irq event stamp: 27266356 [18869.849628] hardirqs last enabled at (27266355): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18869.857358] Modules linked in: [18869.861015] hardirqs last disabled at (27266356): [] sysvec_apic_timer_interrupt+0xe/0x90 [18869.871171] ceph [18869.874222] softirqs last enabled at (27266332): [] __do_softirq+0x2be/0x371 [18869.883946] libceph dns_resolver [18869.885869] softirqs last disabled at (27266327): [] irq_exit_rcu+0xb0/0xd0 [18869.894556] fscache netfs [18869.897865] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18869.906376] veth [18869.909082] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18869.918025] nft_chain_nat [18869.919949] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18869.927419] xt_MASQUERADE nf_nat [18869.930124] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18869.935601] nf_conntrack nf_defrag_ipv6 [18869.938911] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18869.957648] nf_defrag_ipv4 [18869.961567] [18869.966784] nft_compat nf_tables [18869.969574] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18869.971065] nfnetlink bridge [18869.974376] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18869.981501] stp llc [18869.984463] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18869.991588] xfs [18869.993773] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18870.000896] libcrc32c sunrpc [18870.002735] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18870.009860] intel_rapl_msr [18870.012821] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18870.019946] intel_rapl_common [18870.022736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18870.030816] sb_edac [18870.033866] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18870.039604] x86_pkg_temp_thermal coretemp [18870.041788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18870.048911] kvm_intel kvm [18870.053002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18870.060125] irqbypass i2c_i801 [18870.062831] Call Trace: [18870.062833] [18870.069953] iTCO_wdt iTCO_vendor_support [18870.073094] ? watchdog_timer_fn+0x1b4/0x220 [18870.075535] i2c_smbus crct10dif_pclmul [18870.077546] ? __hrtimer_run_queues+0x1d3/0x360 [18870.081549] joydev crc32_pclmul [18870.085819] ? hrtimer_interrupt+0x104/0x240 [18870.089645] ipmi_ssif ghash_clmulni_intel [18870.094168] ? __local_bh_enable+0x50/0x60 [18870.097394] mei_me mei [18870.101659] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18870.105748] lpc_ich sha512_ssse3 [18870.109838] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18870.112282] acpi_ipmi [18870.117501] [18870.120811] mfd_core [18870.125770] [18870.125771] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18870.128127] wmi ipmi_si [18870.130226] ? queued_spin_lock_slowpath+0x16/0x2e0 [18870.132493] ipmi_devintf [18870.134592] do_raw_spin_lock+0xb5/0xc0 [18870.139894] ipmi_msghandler ioatdma [18870.142429] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18870.147297] acpi_pad [18870.149917] try_get_cap_refs+0x159/0x880 [ceph] [18870.153746] acpi_power_meter ixgbe [18870.157320] __ceph_get_caps+0x4a1/0x760 [ceph] [18870.162100] igb mdio [18870.164376] ? __file_remove_privs+0x89/0x110 [18870.168980] crc32c_intel nvme [18870.172465] ? wake_bit_function+0x70/0x70 [18870.176990] ptp nvme_core [18870.179262] ? ceph_write_iter+0x260/0x770 [ceph] [18870.183612] i2c_algo_bit pps_core [18870.186661] ceph_write_iter+0x260/0x770 [ceph] [18870.190751] dca fuse [18870.193461] ? lock_is_held_type+0xaa/0x120 [18870.198152] [18870.198153] irq event stamp: 73773040 [18870.201554] vfs_write+0x324/0x4f0 [18870.206076] hardirqs last enabled at (73773039): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18870.208359] ksys_pwrite64+0x65/0xa0 [18870.212532] hardirqs last disabled at (73773040): [] sysvec_apic_timer_interrupt+0xe/0x90 [18870.214023] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18870.217679] softirqs last enabled at (73773038): [] __do_softirq+0x2be/0x371 [18870.221079] do_syscall_64+0x38/0x80 [18870.231234] softirqs last disabled at (73773033): [] irq_exit_rcu+0xb0/0xd0 [18870.234806] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18870.244530] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18870.249660] RIP: 0033:0x7f48f0eeb93b [18870.258356] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18870.258357] Workqueue: events delayed_work [ceph] [18870.261925] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18870.270436] [18870.270437] RIP: 0010:_raw_spin_lock+0x18/0x40 [18870.275479] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18870.284856] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [18870.288426] ORIG_RAX: 0000000000000012 [18870.295899] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [18870.300594] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18870.319334] [18870.319335] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [18870.320840] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18870.325278] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [18870.330497] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18870.349234] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [18870.353063] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18870.358281] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18870.365407] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18870.366897] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18870.374025] [18870.381146] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18870.441276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18870.447013] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18870.454136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18870.461260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18870.468385] Call Trace: [18870.470829] [18870.472842] ? watchdog_timer_fn+0x1b4/0x220 [18870.477106] ? __hrtimer_run_queues+0x1d3/0x360 [18870.481645] ? hrtimer_interrupt+0x104/0x240 [18870.485909] ? __local_bh_enable+0x50/0x60 [18870.490003] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18870.495228] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18870.500185] [18870.502282] [18870.504378] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18870.509686] ? _raw_spin_lock+0x18/0x40 [18870.513523] ? igrab+0x1d/0x50 [18870.516573] igrab+0x1d/0x50 [18870.519453] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18870.524603] delayed_work+0x1d8/0x280 [ceph] [18870.528882] ? process_scheduled_works+0x22a/0x510 [18870.533674] process_scheduled_works+0x2a4/0x510 [18870.538288] worker_thread+0x188/0x340 [18870.542037] ? rescuer_thread+0x310/0x310 [18870.546042] kthread+0xf5/0x130 [18870.549187] ? kthread_complete_and_exit+0x20/0x20 [18870.553972] ret_from_fork+0x31/0x40 [18870.557552] ? kthread_complete_and_exit+0x20/0x20 [18870.562337] ret_from_fork_asm+0x11/0x20 [18870.566283] [18897.787462] watchdog: BUG: soft lockup - CPU#0 stuck for 6485s! [dbench:78683] [18897.794673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18897.799461] watchdog: BUG: soft lockup - CPU#7 stuck for 15482s! [kworker/7:4:78808] [18897.850116] irq event stamp: 27324446 [18897.857862] Modules linked in: ceph libceph [18897.861519] hardirqs last enabled at (27324445): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18897.865697] dns_resolver fscache netfs [18897.875855] hardirqs last disabled at (27324446): [] sysvec_apic_timer_interrupt+0xe/0x90 [18897.875857] veth [18897.875858] softirqs last enabled at (27324428): [] __do_softirq+0x2be/0x371 [18897.879684] nft_chain_nat xt_MASQUERADE [18897.889409] softirqs last disabled at (27324423): [] irq_exit_rcu+0xb0/0xd0 [18897.891332] nf_nat [18897.900017] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18897.903934] nf_conntrack nf_defrag_ipv6 [18897.912444] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18897.914543] nf_defrag_ipv4 nft_compat [18897.923487] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [18897.927405] nf_tables nfnetlink [18897.934874] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [18897.938618] bridge stp [18897.944096] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18897.947320] llc xfs [18897.966059] [18897.966060] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18897.968502] libcrc32c sunrpc [18897.973719] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18897.975904] intel_rapl_msr intel_rapl_common [18897.977395] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18897.984519] sb_edac [18897.987482] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18897.994606] x86_pkg_temp_thermal coretemp [18897.998958] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18898.006081] kvm_intel kvm [18898.008266] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18898.015389] irqbypass i2c_i801 [18898.019479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18898.026604] iTCO_wdt [18898.029308] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18898.037387] iTCO_vendor_support i2c_smbus [18898.040523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18898.046261] crct10dif_pclmul joydev [18898.048533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18898.055655] crc32_pclmul ipmi_ssif [18898.059746] Call Trace: [18898.059747] [18898.066871] ghash_clmulni_intel [18898.070443] ? watchdog_timer_fn+0x1b4/0x220 [18898.077566] mei_me mei [18898.081051] ? __hrtimer_run_queues+0x1d3/0x360 [18898.083494] lpc_ich sha512_ssse3 [18898.085508] ? hrtimer_interrupt+0x104/0x240 [18898.088729] acpi_ipmi mfd_core [18898.092993] ? __local_bh_enable+0x50/0x60 [18898.095438] wmi ipmi_si [18898.099964] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18898.103271] ipmi_devintf ipmi_msghandler [18898.107537] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18898.110674] ioatdma acpi_pad [18898.114763] [18898.117294] acpi_power_meter [18898.122512] [18898.126517] ixgbe igb [18898.131474] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18898.134439] mdio crc32c_intel [18898.136538] ? queued_spin_lock_slowpath+0x18/0x2e0 [18898.139498] nvme ptp [18898.141600] do_raw_spin_lock+0xb5/0xc0 [18898.143954] nvme_core [18898.149260] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18898.152309] i2c_algo_bit pps_core [18898.157180] try_get_cap_refs+0x159/0x880 [ceph] [18898.159449] dca fuse [18898.163285] __ceph_get_caps+0x4a1/0x760 [ceph] [18898.165637] [18898.165638] irq event stamp: 73995602 [18898.170426] ? __file_remove_privs+0x89/0x110 [18898.173820] hardirqs last enabled at (73995601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18898.178430] ? wake_bit_function+0x70/0x70 [18898.180701] hardirqs last disabled at (73995602): [] sysvec_apic_timer_interrupt+0xe/0x90 [18898.185228] ? ceph_write_iter+0x260/0x770 [ceph] [18898.186717] softirqs last enabled at (73995600): [] __do_softirq+0x2be/0x371 [18898.190373] ceph_write_iter+0x260/0x770 [ceph] [18898.194724] softirqs last disabled at (73995595): [] irq_exit_rcu+0xb0/0xd0 [18898.204886] ? lock_is_held_type+0xaa/0x120 [18898.208973] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18898.218698] vfs_write+0x324/0x4f0 [18898.223393] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18898.232083] ksys_pwrite64+0x65/0xa0 [18898.236603] Workqueue: events delayed_work [ceph] [18898.245112] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18898.249290] [18898.258669] do_syscall_64+0x38/0x80 [18898.262065] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [18898.269537] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18898.273106] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [18898.277803] RIP: 0033:0x7f48f0eeb93b [18898.282936] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [18898.284426] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18898.287995] [18898.287996] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [18898.293906] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18898.298952] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [18898.317690] ORIG_RAX: 0000000000000012 [18898.321260] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [18898.326476] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18898.345215] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18898.346706] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18898.353830] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18898.359047] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18898.366171] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18898.370002] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18898.377142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18898.384268] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18898.391392] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18898.398521] [18898.405639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18898.457295] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18898.464418] Call Trace: [18898.466869] [18898.468883] ? watchdog_timer_fn+0x1b4/0x220 [18898.473156] ? __hrtimer_run_queues+0x1d3/0x360 [18898.477682] ? hrtimer_interrupt+0x104/0x240 [18898.481952] ? __local_bh_enable+0x50/0x60 [18898.486046] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18898.491269] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18898.496226] [18898.498322] [18898.500420] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18898.505729] ? ceph_check_delayed_caps+0x9e/0x270 [ceph] [18898.511046] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [18898.516360] delayed_work+0x1d8/0x280 [ceph] [18898.520643] ? process_scheduled_works+0x22a/0x510 [18898.525432] process_scheduled_works+0x2a4/0x510 [18898.530049] worker_thread+0x188/0x340 [18898.533799] ? rescuer_thread+0x310/0x310 [18898.537801] kthread+0xf5/0x130 [18898.540939] ? kthread_complete_and_exit+0x20/0x20 [18898.545722] ret_from_fork+0x31/0x40 [18898.549300] ? kthread_complete_and_exit+0x20/0x20 [18898.554085] ret_from_fork_asm+0x11/0x20 [18898.558009] [18925.787970] watchdog: BUG: soft lockup - CPU#0 stuck for 6512s! [dbench:78683] [18925.795187] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18925.799969] watchdog: BUG: soft lockup - CPU#7 stuck for 15509s! [kworker/7:4:78808] [18925.850646] irq event stamp: 27382548 [18925.858376] Modules linked in: ceph libceph dns_resolver fscache netfs [18925.862032] hardirqs last enabled at (27382547): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18925.868558] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [18925.878718] hardirqs last disabled at (27382548): [] sysvec_apic_timer_interrupt+0xe/0x90 [18925.884800] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [18925.894525] softirqs last enabled at (27382530): [] __do_softirq+0x2be/0x371 [18925.901303] bridge stp llc xfs libcrc32c sunrpc [18925.909985] softirqs last disabled at (27382525): [] irq_exit_rcu+0xb0/0xd0 [18925.909989] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [18925.923108] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18925.931617] kvm irqbypass [18925.940580] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18925.943285] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [18925.950755] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [18925.950757] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [18925.961883] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [18925.969181] mei_me mei [18925.987919] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18925.990371] lpc_ich sha512_ssse3 [18925.995587] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18925.995590] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18925.998899] acpi_ipmi [18926.006023] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18926.013146] mfd_core wmi [18926.015505] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18926.022628] ipmi_si ipmi_devintf [18926.025246] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18926.032371] ipmi_msghandler ioatdma [18926.035682] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18926.042806] acpi_pad acpi_power_meter [18926.046376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18926.054454] ixgbe igb [18926.058198] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18926.063935] mdio crc32c_intel [18926.066291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18926.073417] nvme ptp [18926.076466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18926.083592] nvme_core i2c_algo_bit [18926.085861] Call Trace: [18926.085863] [18926.092988] pps_core [18926.096473] ? watchdog_timer_fn+0x1b4/0x220 [18926.098915] dca fuse [18926.100927] ? __hrtimer_run_queues+0x1d3/0x360 [18926.103195] [18926.103196] irq event stamp: 74217560 [18926.107464] ? hrtimer_interrupt+0x104/0x240 [18926.109730] hardirqs last enabled at (74217559): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18926.114255] ? __local_bh_enable+0x50/0x60 [18926.115746] hardirqs last disabled at (74217560): [] sysvec_apic_timer_interrupt+0xe/0x90 [18926.119404] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18926.123666] softirqs last enabled at (74217558): [] __do_softirq+0x2be/0x371 [18926.133825] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18926.137914] softirqs last disabled at (74217553): [] irq_exit_rcu+0xb0/0xd0 [18926.147648] [18926.147649] [18926.152865] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18926.161550] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18926.166506] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18926.175022] ? queued_spin_lock_slowpath+0x16/0x2e0 [18926.177117] Workqueue: events delayed_work [ceph] [18926.179214] do_raw_spin_lock+0xb5/0xc0 [18926.188591] [18926.193895] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18926.201370] RIP: 0010:_raw_spin_lock+0x27/0x40 [18926.206248] try_get_cap_refs+0x159/0x880 [ceph] [18926.210941] Code: cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 <31> d2 31 f6 e8 30 0c 5c ff 48 89 df 58 5b e9 76 33 5c ff 66 0f 1f [18926.214778] __ceph_get_caps+0x4a1/0x760 [ceph] [18926.216263] RSP: 0018:ffffc90002117d88 EFLAGS: 00000246 [18926.221053] ? __file_remove_privs+0x89/0x110 [18926.225483] [18926.225484] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [18926.230097] ? wake_bit_function+0x70/0x70 [18926.248833] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34518 [18926.253373] ? ceph_write_iter+0x260/0x770 [ceph] [18926.258593] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [18926.262942] ceph_write_iter+0x260/0x770 [ceph] [18926.264432] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18926.271562] ? lock_is_held_type+0xaa/0x120 [18926.275651] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18926.282776] vfs_write+0x324/0x4f0 [18926.287469] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18926.294599] ksys_pwrite64+0x65/0xa0 [18926.299119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18926.306243] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18926.310420] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18926.317546] do_syscall_64+0x38/0x80 [18926.320941] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18926.329021] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18926.332590] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18926.338328] RIP: 0033:0x7f48f0eeb93b [18926.343457] Call Trace: [18926.350583] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18926.354154] [18926.361278] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18926.366321] ? watchdog_timer_fn+0x1b4/0x220 [18926.373444] ORIG_RAX: 0000000000000012 [18926.377016] ? __hrtimer_run_queues+0x1d3/0x360 [18926.379459] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18926.398200] ? hrtimer_interrupt+0x104/0x240 [18926.400207] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18926.405425] ? __local_bh_enable+0x50/0x60 [18926.409689] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18926.413522] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18926.418045] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18926.425170] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18926.429432] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18926.436557] [18926.440666] [18926.447789] [18926.478593] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18926.483900] ? _raw_spin_lock+0x27/0x40 [18926.487737] ? igrab+0x1d/0x50 [18926.490786] igrab+0x1d/0x50 [18926.493668] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18926.498814] delayed_work+0x1d8/0x280 [ceph] [18926.503095] ? process_scheduled_works+0x22a/0x510 [18926.507885] process_scheduled_works+0x2a4/0x510 [18926.512501] worker_thread+0x188/0x340 [18926.516250] ? rescuer_thread+0x310/0x310 [18926.520253] kthread+0xf5/0x130 [18926.523401] ? kthread_complete_and_exit+0x20/0x20 [18926.528192] ret_from_fork+0x31/0x40 [18926.531772] ? kthread_complete_and_exit+0x20/0x20 [18926.536556] ret_from_fork_asm+0x11/0x20 [18926.540488] [18953.788476] watchdog: BUG: soft lockup - CPU#0 stuck for 6538s! [dbench:78683] [18953.795691] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18953.800477] watchdog: BUG: soft lockup - CPU#7 stuck for 15535s! [kworker/7:4:78808] [18953.851148] irq event stamp: 27440558 [18953.851149] hardirqs last enabled at (27440557): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18953.858880] Modules linked in: [18953.862537] hardirqs last disabled at (27440558): [] sysvec_apic_timer_interrupt+0xe/0x90 [18953.872695] ceph [18953.875746] softirqs last enabled at (27440540): [] __do_softirq+0x2be/0x371 [18953.885469] libceph [18953.887395] softirqs last disabled at (27440535): [] irq_exit_rcu+0xb0/0xd0 [18953.896077] dns_resolver [18953.898262] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18953.906774] fscache netfs [18953.909391] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18953.918335] veth nft_chain_nat [18953.921040] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18953.928510] xt_MASQUERADE [18953.931646] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18953.937124] nf_nat nf_conntrack [18953.939828] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18953.958566] nf_defrag_ipv6 [18953.961789] [18953.961790] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18953.967008] nf_defrag_ipv4 nft_compat [18953.969797] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18953.971288] nf_tables nfnetlink [18953.978414] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18953.982162] bridge stp [18953.989289] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18953.992513] llc xfs [18953.999638] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18954.002081] libcrc32c sunrpc [18954.009206] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18954.011390] intel_rapl_msr intel_rapl_common [18954.018514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18954.021477] sb_edac x86_pkg_temp_thermal [18954.029557] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18954.033907] coretemp kvm_intel [18954.039644] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18954.043648] kvm irqbypass [18954.050773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18954.053909] i2c_i801 [18954.061033] Call Trace: [18954.061034] [18954.063739] iTCO_wdt iTCO_vendor_support [18954.070875] ? watchdog_timer_fn+0x1b4/0x220 [18954.073157] i2c_smbus crct10dif_pclmul [18954.075604] ? __hrtimer_run_queues+0x1d3/0x360 [18954.077613] joydev crc32_pclmul [18954.081620] ? hrtimer_interrupt+0x104/0x240 [18954.085881] ipmi_ssif ghash_clmulni_intel [18954.089712] ? __local_bh_enable+0x50/0x60 [18954.094236] mei_me mei [18954.097464] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18954.101724] lpc_ich sha512_ssse3 [18954.105817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18954.109905] acpi_ipmi mfd_core [18954.112349] [18954.117567] wmi ipmi_si [18954.120878] [18954.120879] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18954.125836] ipmi_devintf [18954.128978] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18954.131070] ipmi_msghandler ioatdma [18954.133603] do_raw_spin_lock+0xb5/0xc0 [18954.135698] acpi_pad acpi_power_meter [18954.141004] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18954.143620] ixgbe igb [18954.148492] try_get_cap_refs+0x159/0x880 [ceph] [18954.152061] mdio crc32c_intel [18954.155898] __ceph_get_caps+0x4a1/0x760 [ceph] [18954.159637] nvme ptp [18954.164424] ? __file_remove_privs+0x89/0x110 [18954.166778] nvme_core i2c_algo_bit [18954.171388] ? wake_bit_function+0x70/0x70 [18954.174438] pps_core dca [18954.178965] ? ceph_write_iter+0x260/0x770 [ceph] [18954.181234] fuse [18954.185585] ceph_write_iter+0x260/0x770 [ceph] [18954.189068] irq event stamp: 74440060 [18954.193167] ? lock_is_held_type+0xaa/0x120 [18954.195776] hardirqs last enabled at (74440059): [] asm_common_interrupt+0x26/0x40 [18954.200477] vfs_write+0x324/0x4f0 [18954.202398] hardirqs last disabled at (74440060): [] sysvec_apic_timer_interrupt+0xe/0x90 [18954.206928] ksys_pwrite64+0x65/0xa0 [18954.210579] softirqs last enabled at (74440058): [] __do_softirq+0x2be/0x371 [18954.214756] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18954.223960] softirqs last disabled at (74440049): [] irq_exit_rcu+0xb0/0xd0 [18954.227361] do_syscall_64+0x38/0x80 [18954.237083] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18954.240654] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18954.249337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18954.254469] RIP: 0033:0x7f48f0eeb93b [18954.262980] Workqueue: events delayed_work [ceph] [18954.266551] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18954.275927] [18954.280971] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18954.288443] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [18954.292013] ORIG_RAX: 0000000000000012 [18954.296712] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [18954.315449] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18954.316939] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [18954.322158] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18954.326768] [18954.326769] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [18954.330598] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18954.349335] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [18954.356461] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18954.361677] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [18954.368801] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18954.370292] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18954.377423] [18954.384540] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18954.429480] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18954.437555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18954.443293] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18954.450417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18954.457541] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18954.464665] Call Trace: [18954.467109] [18954.469121] ? watchdog_timer_fn+0x1b4/0x220 [18954.473385] ? __hrtimer_run_queues+0x1d3/0x360 [18954.477913] ? hrtimer_interrupt+0x104/0x240 [18954.482181] ? __local_bh_enable+0x50/0x60 [18954.486276] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18954.491498] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18954.496457] [18954.498553] [18954.500651] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18954.505958] ? do_raw_spin_lock+0x42/0xc0 [18954.509967] igrab+0x1d/0x50 [18954.512845] ceph_check_delayed_caps+0xb7/0x270 [ceph] [18954.518011] delayed_work+0x1d8/0x280 [ceph] [18954.522295] ? process_scheduled_works+0x22a/0x510 [18954.527084] process_scheduled_works+0x2a4/0x510 [18954.531699] worker_thread+0x188/0x340 [18954.535449] ? rescuer_thread+0x310/0x310 [18954.539452] kthread+0xf5/0x130 [18954.542590] ? kthread_complete_and_exit+0x20/0x20 [18954.547382] ret_from_fork+0x31/0x40 [18954.550961] ? kthread_complete_and_exit+0x20/0x20 [18954.555746] ret_from_fork_asm+0x11/0x20 [18954.559669] [18981.788985] watchdog: BUG: soft lockup - CPU#0 stuck for 6564s! [dbench:78683] [18981.796195] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [18981.800985] watchdog: BUG: soft lockup - CPU#7 stuck for 15561s! [kworker/7:4:78808] [18981.851638] irq event stamp: 27498684 [18981.851639] hardirqs last enabled at (27498683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18981.859380] Modules linked in: [18981.863042] hardirqs last disabled at (27498684): [] sysvec_apic_timer_interrupt+0xe/0x90 [18981.873200] ceph [18981.876250] softirqs last enabled at (27498666): [] __do_softirq+0x2be/0x371 [18981.885975] libceph dns_resolver [18981.887898] softirqs last disabled at (27498661): [] irq_exit_rcu+0xb0/0xd0 [18981.896583] fscache netfs [18981.899893] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18981.908403] veth nft_chain_nat [18981.911109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18981.920053] xt_MASQUERADE [18981.923190] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [18981.930661] nf_nat nf_conntrack [18981.933380] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [18981.938859] nf_defrag_ipv6 nf_defrag_ipv4 [18981.942084] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [18981.960822] nft_compat [18981.964913] [18981.964914] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [18981.970128] nf_tables nfnetlink [18981.972574] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [18981.974065] bridge stp [18981.981187] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [18981.984412] llc xfs [18981.991536] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [18981.993980] libcrc32c [18982.001105] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [18982.003288] sunrpc intel_rapl_msr [18982.010414] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [18982.012770] intel_rapl_common sb_edac [18982.019895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18982.023291] x86_pkg_temp_thermal coretemp [18982.031382] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [18982.035131] kvm_intel kvm [18982.040868] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18982.044958] irqbypass i2c_i801 [18982.052084] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18982.054787] iTCO_wdt iTCO_vendor_support [18982.061910] Call Trace: [18982.061912] [18982.065048] i2c_smbus crct10dif_pclmul [18982.072174] ? watchdog_timer_fn+0x1b4/0x220 [18982.076177] joydev crc32_pclmul [18982.078623] ? __hrtimer_run_queues+0x1d3/0x360 [18982.080631] ipmi_ssif ghash_clmulni_intel [18982.084466] ? hrtimer_interrupt+0x104/0x240 [18982.088725] mei_me mei [18982.091950] ? __local_bh_enable+0x50/0x60 [18982.096473] lpc_ich [18982.100568] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18982.104830] sha512_ssse3 acpi_ipmi [18982.107273] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18982.111382] mfd_core wmi [18982.113567] [18982.118783] ipmi_si ipmi_devintf [18982.122266] [18982.122268] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18982.127223] ipmi_msghandler [18982.129846] ? queued_spin_lock_slowpath+0x2d/0x2e0 [18982.131940] ioatdma acpi_pad [18982.135252] do_raw_spin_lock+0xb5/0xc0 [18982.137346] acpi_power_meter ixgbe [18982.142652] __cap_delay_requeue+0xa7/0x1d0 [ceph] [18982.145545] igb mdio [18982.150418] try_get_cap_refs+0x159/0x880 [ceph] [18982.153384] crc32c_intel [18982.157215] __ceph_get_caps+0x4a1/0x760 [ceph] [18982.160695] nvme ptp [18982.165484] ? __file_remove_privs+0x89/0x110 [18982.167751] nvme_core i2c_algo_bit [18982.172362] ? wake_bit_function+0x70/0x70 [18982.174996] pps_core dca [18982.179523] ? ceph_write_iter+0x260/0x770 [ceph] [18982.181790] fuse [18982.186142] ceph_write_iter+0x260/0x770 [ceph] [18982.189625] irq event stamp: 74662418 [18982.193720] ? lock_is_held_type+0xaa/0x120 [18982.196333] hardirqs last enabled at (74662417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18982.201033] vfs_write+0x324/0x4f0 [18982.202955] hardirqs last disabled at (74662418): [] sysvec_apic_timer_interrupt+0xe/0x90 [18982.207483] ksys_pwrite64+0x65/0xa0 [18982.211137] softirqs last enabled at (74662416): [] __do_softirq+0x2be/0x371 [18982.215314] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [18982.225472] softirqs last disabled at (74662411): [] irq_exit_rcu+0xb0/0xd0 [18982.228869] do_syscall_64+0x38/0x80 [18982.238593] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [18982.242164] entry_SYSCALL_64_after_hwframe+0x63/0xcd [18982.250848] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [18982.255979] RIP: 0033:0x7f48f0eeb93b [18982.264488] Workqueue: events delayed_work [ceph] [18982.268059] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [18982.277437] [18982.277438] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [18982.282481] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [18982.289951] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [18982.293523] ORIG_RAX: 0000000000000012 [18982.298220] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [18982.316958] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [18982.318448] [18982.318449] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [18982.324446] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [18982.329676] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [18982.348411] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [18982.352240] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [18982.357459] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [18982.364584] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [18982.366073] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [18982.373196] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [18982.380327] [18982.387447] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [18982.440469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18982.446207] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [18982.453330] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18982.460456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [18982.467579] Call Trace: [18982.470025] [18982.472035] ? watchdog_timer_fn+0x1b4/0x220 [18982.476300] ? __hrtimer_run_queues+0x1d3/0x360 [18982.480826] ? hrtimer_interrupt+0x104/0x240 [18982.485094] ? __local_bh_enable+0x50/0x60 [18982.489189] ? __sysvec_apic_timer_interrupt+0x61/0x160 [18982.494412] ? sysvec_apic_timer_interrupt+0x6d/0x90 [18982.499387] [18982.501484] [18982.503582] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [18982.508890] ? __list_del_entry_valid_or_report+0x10/0xd0 [18982.514285] ceph_check_delayed_caps+0x8f/0x270 [ceph] [18982.519436] delayed_work+0x1d8/0x280 [ceph] [18982.523718] ? process_scheduled_works+0x22a/0x510 [18982.528506] process_scheduled_works+0x2a4/0x510 [18982.533122] worker_thread+0x188/0x340 [18982.536873] ? rescuer_thread+0x310/0x310 [18982.540875] kthread+0xf5/0x130 [18982.544013] ? kthread_complete_and_exit+0x20/0x20 [18982.548796] ret_from_fork+0x31/0x40 [18982.552367] ? kthread_complete_and_exit+0x20/0x20 [18982.557170] ret_from_fork_asm+0x11/0x20 [18982.561091] [19009.789493] watchdog: BUG: soft lockup - CPU#0 stuck for 6590s! [dbench:78683] [19009.796710] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19009.801493] watchdog: BUG: soft lockup - CPU#7 stuck for 15587s! [kworker/7:4:78808] [19009.852169] irq event stamp: 27556866 [19009.859899] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat [19009.863554] hardirqs last enabled at (27556865): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19009.873547] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [19009.883705] hardirqs last disabled at (27556866): [] sysvec_apic_timer_interrupt+0xe/0x90 [19009.890744] nfnetlink bridge stp llc xfs libcrc32c sunrpc [19009.900467] softirqs last enabled at (27556848): [] __do_softirq+0x2be/0x371 [19009.900471] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [19009.914639] softirqs last disabled at (27556843): [] irq_exit_rcu+0xb0/0xd0 [19009.924371] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [19009.932888] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19009.940014] joydev crc32_pclmul ipmi_ssif [19009.948960] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19009.948962] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19009.953049] ghash_clmulni_intel mei_me [19009.960521] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19009.965997] mei lpc_ich [19009.969827] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19009.988566] sha512_ssse3 acpi_ipmi [19009.991096] [19009.996313] mfd_core wmi [19009.999798] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19010.001288] ipmi_si ipmi_devintf [19010.003907] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19010.011031] ipmi_msghandler ioatdma [19010.014342] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19010.021464] acpi_pad acpi_power_meter [19010.025035] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19010.032173] ixgbe igb [19010.035922] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19010.043045] mdio crc32c_intel [19010.045404] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19010.052527] nvme ptp [19010.055577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19010.063655] nvme_core i2c_algo_bit [19010.065925] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19010.071665] pps_core [19010.075147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19010.082271] dca fuse [19010.084542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19010.091667] [19010.091668] irq event stamp: 74884636 [19010.093937] Call Trace: [19010.093940] [19010.101061] hardirqs last enabled at (74884635): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19010.102555] ? watchdog_timer_fn+0x1b4/0x220 [19010.106210] hardirqs last disabled at (74884636): [] sysvec_apic_timer_interrupt+0xe/0x90 [19010.108654] ? __hrtimer_run_queues+0x1d3/0x360 [19010.110664] softirqs last enabled at (74884634): [] __do_softirq+0x2be/0x371 [19010.120825] ? hrtimer_interrupt+0x104/0x240 [19010.125085] softirqs last disabled at (74884629): [] irq_exit_rcu+0xb0/0xd0 [19010.134811] ? __local_bh_enable+0x50/0x60 [19010.139335] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19010.148022] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19010.152283] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19010.152285] Workqueue: events delayed_work [ceph] [19010.160792] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19010.164892] [19010.174269] [19010.179487] RIP: 0010:lock_acquire+0xf0/0x2b0 [19010.186959] [19010.191654] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19010.196613] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19010.198103] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19010.200205] ? queued_spin_lock_slowpath+0x18/0x2e0 [19010.204553] [19010.204553] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19010.206652] do_raw_spin_lock+0xb5/0xc0 [19010.225386] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19010.230693] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19010.235907] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19010.240781] try_get_cap_refs+0x159/0x880 [ceph] [19010.242271] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19010.249401] __ceph_get_caps+0x4a1/0x760 [ceph] [19010.253225] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19010.260354] ? __file_remove_privs+0x89/0x110 [19010.265132] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19010.272257] ? wake_bit_function+0x70/0x70 [19010.276869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19010.284013] ? ceph_write_iter+0x260/0x770 [ceph] [19010.288533] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19010.295659] ceph_write_iter+0x260/0x770 [ceph] [19010.300008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19010.308089] ? lock_is_held_type+0xaa/0x120 [19010.312177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19010.317916] vfs_write+0x324/0x4f0 [19010.322610] Call Trace: [19010.322612] [19010.329741] ksys_pwrite64+0x65/0xa0 [19010.334261] ? watchdog_timer_fn+0x1b4/0x220 [19010.341384] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19010.345563] ? __hrtimer_run_queues+0x1d3/0x360 [19010.352686] do_syscall_64+0x38/0x80 [19010.356087] ? hrtimer_interrupt+0x104/0x240 [19010.358529] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19010.360536] ? __local_bh_enable+0x50/0x60 [19010.364109] RIP: 0033:0x7f48f0eeb93b [19010.368375] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19010.373504] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19010.378027] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19010.381598] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19010.385862] [19010.390907] ORIG_RAX: 0000000000000012 [19010.394996] [19010.398568] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19010.403785] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19010.422523] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19010.427483] ? lock_acquire+0xf0/0x2b0 [19010.432697] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19010.434795] ? igrab+0x43/0x50 [19010.438625] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19010.440723] ? lock_release+0xc5/0x260 [19010.447847] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19010.453153] _raw_spin_lock+0x30/0x40 [19010.460280] [19010.464021] ? igrab+0x1d/0x50 [19010.501080] igrab+0x1d/0x50 [19010.503958] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19010.509109] delayed_work+0x1d8/0x280 [ceph] [19010.513388] ? process_scheduled_works+0x22a/0x510 [19010.518178] process_scheduled_works+0x2a4/0x510 [19010.522795] worker_thread+0x188/0x340 [19010.526543] ? rescuer_thread+0x310/0x310 [19010.530548] kthread+0xf5/0x130 [19010.533693] ? kthread_complete_and_exit+0x20/0x20 [19010.538477] ret_from_fork+0x31/0x40 [19010.542057] ? kthread_complete_and_exit+0x20/0x20 [19010.546839] ret_from_fork_asm+0x11/0x20 [19010.550764] [19037.790001] watchdog: BUG: soft lockup - CPU#0 stuck for 6616s! [dbench:78683] [19037.797215] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19037.802001] watchdog: BUG: soft lockup - CPU#7 stuck for 15613s! [kworker/7:4:78808] [19037.852672] irq event stamp: 27614900 [19037.852673] hardirqs last enabled at (27614899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19037.860404] Modules linked in: [19037.864061] hardirqs last disabled at (27614900): [] sysvec_apic_timer_interrupt+0xe/0x90 [19037.874219] ceph [19037.877270] softirqs last enabled at (27614882): [] __do_softirq+0x2be/0x371 [19037.886993] libceph [19037.888916] softirqs last disabled at (27614877): [] irq_exit_rcu+0xb0/0xd0 [19037.897601] dns_resolver [19037.899786] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19037.908295] fscache netfs [19037.910915] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19037.919857] veth nft_chain_nat [19037.922563] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19037.930032] xt_MASQUERADE nf_nat [19037.933170] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19037.938648] nf_conntrack [19037.941958] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19037.960695] nf_defrag_ipv6 nf_defrag_ipv4 [19037.963312] [19037.968531] nft_compat nf_tables [19037.972620] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19037.974112] nfnetlink bridge [19037.977423] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19037.984547] stp llc [19037.987511] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19037.994634] xfs libcrc32c [19037.996819] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19038.003943] sunrpc intel_rapl_msr [19038.006647] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19038.013771] intel_rapl_common sb_edac [19038.017170] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19038.024292] x86_pkg_temp_thermal [19038.028037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19038.036114] coretemp kvm_intel [19038.039426] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19038.045163] kvm irqbypass [19038.048299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19038.055425] i2c_i801 iTCO_wdt [19038.058128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19038.065251] iTCO_vendor_support [19038.068302] Call Trace: [19038.068304] [19038.075427] i2c_smbus crct10dif_pclmul [19038.078653] ? watchdog_timer_fn+0x1b4/0x220 [19038.081096] joydev crc32_pclmul [19038.083108] ? __hrtimer_run_queues+0x1d3/0x360 [19038.086936] ipmi_ssif ghash_clmulni_intel [19038.091204] ? hrtimer_interrupt+0x104/0x240 [19038.094425] mei_me [19038.098949] ? __local_bh_enable+0x50/0x60 [19038.103040] mei lpc_ich [19038.107307] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19038.109401] sha512_ssse3 acpi_ipmi [19038.113492] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19038.116022] mfd_core [19038.121240] [19038.124725] wmi ipmi_si [19038.129682] [19038.131951] ipmi_devintf ipmi_msghandler [19038.134051] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19038.136581] ioatdma acpi_pad [19038.138682] ? queued_spin_lock_slowpath+0x18/0x2e0 [19038.142682] acpi_power_meter ixgbe [19038.147989] do_raw_spin_lock+0xb5/0xc0 [19038.150950] igb mdio [19038.155821] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19038.159305] crc32c_intel nvme [19038.163139] try_get_cap_refs+0x159/0x880 [ceph] [19038.165407] ptp nvme_core [19038.170195] __ceph_get_caps+0x4a1/0x760 [ceph] [19038.173241] i2c_algo_bit pps_core [19038.177857] ? __file_remove_privs+0x89/0x110 [19038.180556] dca fuse [19038.181376] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [19038.181378] rcu: 0-....: (6942030 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=3455253 [19038.181401] rcu: 7-....: (6897587 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/6896341 fqs=3455254 [19038.181404] rcu: (detected by 3, t=7101429 jiffies, g=1309609, q=265314 ncpus=8) [19038.181407] Sending NMI from CPU 3 to CPUs 0: [19038.181411] NMI backtrace for cpu 0 [19038.181412] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19038.181414] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19038.181415] RIP: 0010:io_serial_in+0x19/0x20 [19038.181418] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [19038.181420] RSP: 0018:ffffc90000003bc0 EFLAGS: 00000002 [19038.181421] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [19038.181422] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [19038.181422] RBP: 00000000000025e0 R08: 00003e34039ad8bb R09: 0000000000000000 [19038.181423] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [19038.181424] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [19038.181425] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19038.181426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19038.181427] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19038.181428] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19038.181428] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19038.181429] Call Trace: [19038.181430] [19038.181431] ? nmi_cpu_backtrace+0xad/0x120 [19038.181434] ? nmi_cpu_backtrace_handler+0x11/0x20 [19038.181436] ? nmi_handle+0xd4/0x260 [19038.181440] ? default_do_nmi+0x49/0xf0 [19038.181442] ? exc_nmi+0x11e/0x150 [19038.181444] ? end_repeat_nmi+0x16/0x67 [19038.181448] ? io_serial_in+0x19/0x20 [19038.181449] ? io_serial_in+0x19/0x20 [19038.181451] ? io_serial_in+0x19/0x20 [19038.181453] [19038.181453] [19038.181454] wait_for_lsr+0x45/0x70 [19038.181456] serial8250_console_write+0x341/0x3f0 [19038.181459] ? lock_release+0xc5/0x260 [19038.181462] console_flush_all+0x227/0x5a0 [19038.181464] ? console_flush_all+0x1ff/0x5a0 [19038.181465] ? console_flush_all+0x1ff/0x5a0 [19038.181468] console_unlock+0x7b/0x120 [19038.181470] vprintk_emit+0x246/0x340 [19038.181472] ? __file_remove_privs+0x89/0x110 [19038.181474] ? __file_remove_privs+0x89/0x110 [19038.181475] _printk+0x5c/0x80 [19038.181478] ? __file_remove_privs+0x89/0x110 [19038.181479] show_trace_log_lvl+0x1ac/0x3f0 [19038.181484] ? ceph_write_iter+0x260/0x770 [ceph] [19038.181496] ? touch_softlockup_watchdog+0x20/0x20 [19038.181498] watchdog_timer_fn+0x1b4/0x220 [19038.181500] __hrtimer_run_queues+0x1d3/0x360 [19038.181503] hrtimer_interrupt+0x104/0x240 [19038.181504] ? __local_bh_enable+0x50/0x60 [19038.181507] __sysvec_apic_timer_interrupt+0x61/0x160 [19038.181509] sysvec_apic_timer_interrupt+0x6d/0x90 [19038.181511] [19038.181512] [19038.181512] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19038.181514] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19038.181517] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19038.181518] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19038.181519] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19038.181520] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19038.181521] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19038.181522] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19038.181522] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19038.181526] do_raw_spin_lock+0xb5/0xc0 [19038.181528] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19038.181540] try_get_cap_refs+0x159/0x880 [ceph] [19038.181554] __ceph_get_caps+0x4a1/0x760 [ceph] [19038.181567] ? __file_remove_privs+0x89/0x110 [19038.181568] ? wake_bit_function+0x70/0x70 [19038.181571] ? ceph_write_iter+0x260/0x770 [ceph] [19038.181582] ceph_write_iter+0x260/0x770 [ceph] [19038.181593] ? lock_is_held_type+0xaa/0x120 [19038.181596] vfs_write+0x324/0x4f0 [19038.181600] ksys_pwrite64+0x65/0xa0 [19038.181602] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19038.181605] do_syscall_64+0x38/0x80 [19038.181607] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19038.181609] RIP: 0033:0x7f48f0eeb93b [19038.181610] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19038.181611] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [19038.181612] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19038.181613] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19038.181614] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19038.181615] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19038.181615] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19038.181618] [19038.182409] Sending NMI from CPU 3 to CPUs 7: [19038.182412] NMI backtrace for cpu 7 [19038.182413] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19038.182415] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19038.182416] Workqueue: events delayed_work [ceph] [19038.182437] RIP: 0010:vprintk_emit+0x1ee/0x340 [19038.182440] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [19038.182442] RSP: 0018:ffffc90000264e08 EFLAGS: 00000002 [19038.182444] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [19038.182445] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb578 [19038.182446] RBP: 0000000000000005 R08: 0000000000005ef4 R09: 0000000000000004 [19038.182447] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [19038.182448] R13: 0000000000000000 R14: ffffffff82130806 R15: ffffc90000264e58 [19038.182449] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19038.182451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19038.182452] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19038.182454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19038.182455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19038.182456] Call Trace: [19038.182457] [19038.182458] ? nmi_cpu_backtrace+0xad/0x120 [19038.182462] ? nmi_cpu_backtrace_handler+0x11/0x20 [19038.182464] ? nmi_handle+0xd4/0x260 [19038.182469] ? default_do_nmi+0x49/0xf0 [19038.182471] ? exc_nmi+0x11e/0x150 [19038.182474] ? end_repeat_nmi+0x16/0x67 [19038.182480] ? vprintk_emit+0x1ee/0x340 [19038.182483] ? vprintk_emit+0x1ee/0x340 [19038.182486] ? vprintk_emit+0x1ee/0x340 [19038.182489] [19038.182490] [19038.182491] ? irq_work_queue+0x2c/0x50 [19038.182496] ? touch_softlockup_watchdog+0x20/0x20 [19038.182498] _printk+0x5c/0x80 [19038.182502] print_modules+0x6e/0xe0 [19038.182506] watchdog_timer_fn+0x19f/0x220 [19038.182509] __hrtimer_run_queues+0x1d3/0x360 [19038.182513] hrtimer_interrupt+0x104/0x240 [19038.182515] ? __local_bh_enable+0x50/0x60 [19038.182520] __sysvec_apic_timer_interrupt+0x61/0x160 [19038.182523] sysvec_apic_timer_interrupt+0x6d/0x90 [19038.182525] [19038.182526] [19038.182527] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19038.182529] RIP: 0010:lock_acquire+0xf0/0x2b0 [19038.182531] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19038.182533] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19038.182535] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19038.182536] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19038.182537] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19038.182538] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19038.182539] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19038.182546] ? igrab+0x43/0x50 [19038.182548] ? lock_release+0xc5/0x260 [19038.182551] _raw_spin_lock+0x30/0x40 [19038.182554] ? igrab+0x1d/0x50 [19038.182555] igrab+0x1d/0x50 [19038.182557] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19038.182577] delayed_work+0x1d8/0x280 [ceph] [19038.182596] ? process_scheduled_works+0x22a/0x510 [19038.182599] process_scheduled_works+0x2a4/0x510 [19038.182606] worker_thread+0x188/0x340 [19038.182609] ? rescuer_thread+0x310/0x310 [19038.182612] kthread+0xf5/0x130 [19038.182615] ? kthread_complete_and_exit+0x20/0x20 [19038.182618] ret_from_fork+0x31/0x40 [19038.182621] ? kthread_complete_and_exit+0x20/0x20 [19038.182624] ret_from_fork_asm+0x11/0x20 [19038.182631] [19038.185080] ? wake_bit_function+0x70/0x70 [19038.188476] [19038.188477] irq event stamp: 75107500 [19038.192831] ? ceph_write_iter+0x260/0x770 [ceph] [19038.195098] hardirqs last enabled at (75107499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19038.201183] ceph_write_iter+0x260/0x770 [ceph] [19038.211513] hardirqs last disabled at (75107500): [] sysvec_apic_timer_interrupt+0xe/0x90 [19038.221935] ? lock_is_held_type+0xaa/0x120 [19038.229403] softirqs last enabled at (75107498): [] __do_softirq+0x2be/0x371 [19038.233755] vfs_write+0x324/0x4f0 [19038.237236] softirqs last disabled at (75107493): [] irq_exit_rcu+0xb0/0xd0 [19038.246185] ksys_pwrite64+0x65/0xa0 [19038.253653] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19038.257915] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19038.276653] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19038.276654] Workqueue: events delayed_work [ceph] [19038.281872] do_syscall_64+0x38/0x80 [19038.288996] [19038.288997] RIP: 0010:lock_acquire+0xf0/0x2b0 [19038.296121] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19038.303242] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19038.310367] RIP: 0033:0x7f48f0eeb93b [19038.317492] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19038.325570] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19038.331306] [19038.331307] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19038.338432] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19038.345554] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19038.352693] ORIG_RAX: 0000000000000012 [19038.355140] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19038.357150] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19038.361328] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19038.366112] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19038.369686] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19038.373514] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19038.376911] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19038.380742] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19038.384406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19038.388066] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19038.391722] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19038.393827] [19038.395832] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19039.305006] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19039.312131] Call Trace: [19039.314576] [19039.316588] ? watchdog_timer_fn+0x1b4/0x220 [19039.320853] ? __hrtimer_run_queues+0x1d3/0x360 [19039.325378] ? hrtimer_interrupt+0x104/0x240 [19039.329646] ? __local_bh_enable+0x50/0x60 [19039.333740] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19039.338957] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19039.343912] [19039.346010] [19039.348107] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19039.353428] ? lock_acquire+0xf0/0x2b0 [19039.357177] ? igrab+0x43/0x50 [19039.360233] ? lock_release+0xc5/0x260 [19039.363978] _raw_spin_lock+0x30/0x40 [19039.367634] ? igrab+0x1d/0x50 [19039.370700] igrab+0x1d/0x50 [19039.373580] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19039.378730] delayed_work+0x1d8/0x280 [ceph] [19039.383010] ? process_scheduled_works+0x22a/0x510 [19039.387794] process_scheduled_works+0x2a4/0x510 [19039.392425] worker_thread+0x188/0x340 [19039.396175] ? rescuer_thread+0x310/0x310 [19039.400179] kthread+0xf5/0x130 [19039.403316] ? kthread_complete_and_exit+0x20/0x20 [19039.408099] ret_from_fork+0x31/0x40 [19039.411669] ? kthread_complete_and_exit+0x20/0x20 [19039.416454] ret_from_fork_asm+0x11/0x20 [19039.420376] [19065.790509] watchdog: BUG: soft lockup - CPU#0 stuck for 6642s! [dbench:78683] [19065.797728] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19065.802509] watchdog: BUG: soft lockup - CPU#7 stuck for 15639s! [kworker/7:4:78808] [19065.853192] irq event stamp: 27673050 [19065.860925] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19065.864582] hardirqs last enabled at (27673049): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19065.864587] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [19065.881700] hardirqs last disabled at (27673050): [] sysvec_apic_timer_interrupt+0xe/0x90 [19065.881703] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [19065.899674] softirqs last enabled at (27673032): [] __do_softirq+0x2be/0x371 [19065.906365] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19065.915049] softirqs last disabled at (27673027): [] irq_exit_rcu+0xb0/0xd0 [19065.915053] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [19065.931033] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19065.936943] iTCO_vendor_support i2c_smbus [19065.945904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19065.945904] crct10dif_pclmul joydev [19065.949996] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19065.957465] crc32_pclmul ipmi_ssif [19065.961035] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19065.966513] ghash_clmulni_intel mei_me [19065.969998] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19065.988736] mei lpc_ich [19065.992566] [19065.997784] sha512_ssse3 acpi_ipmi [19066.000315] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19066.001806] mfd_core wmi [19066.005290] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19066.012413] ipmi_si ipmi_devintf [19066.015031] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19066.022154] ipmi_msghandler ioatdma [19066.025464] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19066.032588] acpi_pad acpi_power_meter [19066.036159] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19066.043284] ixgbe [19066.047029] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19066.054152] igb mdio [19066.056163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19066.064258] crc32c_intel nvme [19066.066530] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19066.072265] ptp nvme_core [19066.075318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19066.082441] i2c_algo_bit pps_core [19066.085145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19066.092269] dca fuse [19066.095668] Call Trace: [19066.095670] [19066.102791] [19066.105064] ? watchdog_timer_fn+0x1b4/0x220 [19066.107507] irq event stamp: 75322782 [19066.109518] ? __hrtimer_run_queues+0x1d3/0x360 [19066.111007] hardirqs last enabled at (75322781): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19066.115275] ? hrtimer_interrupt+0x104/0x240 [19066.118928] hardirqs last disabled at (75322782): [] sysvec_apic_timer_interrupt+0xe/0x90 [19066.123454] ? __local_bh_enable+0x50/0x60 [19066.133609] softirqs last enabled at (75322780): [] __do_softirq+0x2be/0x371 [19066.137878] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19066.147598] softirqs last disabled at (75322775): [] irq_exit_rcu+0xb0/0xd0 [19066.151689] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19066.160373] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19066.165590] [19066.174100] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19066.179058] [19066.188438] Workqueue: events delayed_work [ceph] [19066.190534] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19066.198003] [19066.198004] RIP: 0010:lock_acquire+0xf0/0x2b0 [19066.200105] ? queued_spin_lock_slowpath+0x18/0x2e0 [19066.204799] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19066.210105] do_raw_spin_lock+0xb5/0xc0 [19066.211595] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19066.215947] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19066.220816] [19066.220816] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19066.239557] try_get_cap_refs+0x159/0x880 [ceph] [19066.243383] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19066.248608] __ceph_get_caps+0x4a1/0x760 [ceph] [19066.253386] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19066.254879] ? __file_remove_privs+0x89/0x110 [19066.261999] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19066.266611] ? wake_bit_function+0x70/0x70 [19066.273736] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19066.278262] ? ceph_write_iter+0x260/0x770 [ceph] [19066.285383] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19066.289734] ceph_write_iter+0x260/0x770 [ceph] [19066.296859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19066.300954] ? lock_is_held_type+0xaa/0x120 [19066.308072] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19066.312772] vfs_write+0x324/0x4f0 [19066.320848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19066.325375] ksys_pwrite64+0x65/0xa0 [19066.331109] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19066.335288] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19066.342412] Call Trace: [19066.342413] [19066.345809] do_syscall_64+0x38/0x80 [19066.352934] ? watchdog_timer_fn+0x1b4/0x220 [19066.356504] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19066.363630] ? __hrtimer_run_queues+0x1d3/0x360 [19066.368758] RIP: 0033:0x7f48f0eeb93b [19066.371205] ? hrtimer_interrupt+0x104/0x240 [19066.373215] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19066.376784] ? __local_bh_enable+0x50/0x60 [19066.381048] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19066.386095] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19066.390615] ORIG_RAX: 0000000000000012 [19066.394195] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19066.398458] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19066.417200] [19066.421288] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19066.426505] [19066.426507] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19066.431723] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19066.435558] ? lock_acquire+0xf0/0x2b0 [19066.440511] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19066.447636] ? igrab+0x43/0x50 [19066.449732] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19066.456857] ? lock_release+0xc5/0x260 [19066.458960] [19066.464260] _raw_spin_lock+0x30/0x40 [19066.502012] ? igrab+0x1d/0x50 [19066.505063] igrab+0x1d/0x50 [19066.507940] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19066.513091] delayed_work+0x1d8/0x280 [ceph] [19066.517372] ? process_scheduled_works+0x22a/0x510 [19066.522160] process_scheduled_works+0x2a4/0x510 [19066.526778] worker_thread+0x188/0x340 [19066.530526] ? rescuer_thread+0x310/0x310 [19066.534530] kthread+0xf5/0x130 [19066.537676] ? kthread_complete_and_exit+0x20/0x20 [19066.542460] ret_from_fork+0x31/0x40 [19066.546038] ? kthread_complete_and_exit+0x20/0x20 [19066.550823] ret_from_fork_asm+0x11/0x20 [19066.554756] [19093.791019] watchdog: BUG: soft lockup - CPU#0 stuck for 6668s! [dbench:78683] [19093.798232] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19093.803019] watchdog: BUG: soft lockup - CPU#7 stuck for 15665s! [kworker/7:4:78808] [19093.853690] irq event stamp: 27730994 [19093.853691] hardirqs last enabled at (27730993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19093.861423] Modules linked in: [19093.865079] hardirqs last disabled at (27730994): [] sysvec_apic_timer_interrupt+0xe/0x90 [19093.875237] ceph [19093.878286] softirqs last enabled at (27730976): [] __do_softirq+0x2be/0x371 [19093.888011] libceph [19093.889935] softirqs last disabled at (27730971): [] irq_exit_rcu+0xb0/0xd0 [19093.898621] dns_resolver fscache [19093.900805] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19093.909315] netfs veth [19093.912625] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19093.921571] nft_chain_nat xt_MASQUERADE [19093.924014] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19093.931485] nf_nat [19093.935414] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19093.940887] nf_conntrack nf_defrag_ipv6 [19093.942985] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19093.961724] nf_defrag_ipv4 nft_compat [19093.965641] [19093.965642] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19093.970858] nf_tables nfnetlink [19093.974602] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19093.976094] bridge stp [19093.983217] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19093.986441] llc xfs [19093.993565] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19093.996010] libcrc32c [19094.003133] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19094.005318] sunrpc intel_rapl_msr [19094.012441] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19094.014798] intel_rapl_common [19094.021924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19094.025321] sb_edac x86_pkg_temp_thermal [19094.033415] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19094.036467] coretemp [19094.042204] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19094.046208] kvm_intel kvm [19094.053332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19094.055602] irqbypass i2c_i801 [19094.062726] Call Trace: [19094.062727] [19094.065430] iTCO_wdt iTCO_vendor_support [19094.072556] ? watchdog_timer_fn+0x1b4/0x220 [19094.075695] i2c_smbus crct10dif_pclmul [19094.078138] ? __hrtimer_run_queues+0x1d3/0x360 [19094.080147] joydev crc32_pclmul [19094.084154] ? hrtimer_interrupt+0x104/0x240 [19094.088417] ipmi_ssif [19094.092245] ? __local_bh_enable+0x50/0x60 [19094.096771] ghash_clmulni_intel [19094.099997] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19094.104259] mei_me mei [19094.106617] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19094.110705] lpc_ich [19094.113932] [19094.119146] sha512_ssse3 acpi_ipmi [19094.121590] [19094.126549] mfd_core wmi [19094.128734] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19094.130829] ipmi_si ipmi_devintf [19094.134317] ? queued_spin_lock_slowpath+0x16/0x2e0 [19094.136419] ipmi_msghandler [19094.139041] do_raw_spin_lock+0xb5/0xc0 [19094.144343] ioatdma [19094.147653] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19094.152524] acpi_pad acpi_power_meter [19094.155419] try_get_cap_refs+0x159/0x880 [ceph] [19094.159248] ixgbe igb [19094.161438] __ceph_get_caps+0x4a1/0x760 [ceph] [19094.166217] mdio crc32c_intel [19094.169966] ? __file_remove_privs+0x89/0x110 [19094.174571] nvme ptp [19094.176931] ? wake_bit_function+0x70/0x70 [19094.181453] nvme_core i2c_algo_bit [19094.184508] ? ceph_write_iter+0x260/0x770 [ceph] [19094.188855] pps_core dca [19094.191127] ceph_write_iter+0x260/0x770 [ceph] [19094.195217] fuse [19094.198704] ? lock_is_held_type+0xaa/0x120 [19094.203397] irq event stamp: 75544958 [19094.206034] vfs_write+0x324/0x4f0 [19094.210557] hardirqs last enabled at (75544957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19094.212486] ksys_pwrite64+0x65/0xa0 [19094.216657] hardirqs last disabled at (75544958): [] sysvec_apic_timer_interrupt+0xe/0x90 [19094.220316] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19094.223711] softirqs last enabled at (75544956): [] __do_softirq+0x2be/0x371 [19094.233873] do_syscall_64+0x38/0x80 [19094.237440] softirqs last disabled at (75544951): [] irq_exit_rcu+0xb0/0xd0 [19094.247166] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19094.252297] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19094.260981] RIP: 0033:0x7f48f0eeb93b [19094.264550] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19094.273062] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19094.278105] Workqueue: events delayed_work [ceph] [19094.287483] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19094.291053] [19094.298524] ORIG_RAX: 0000000000000012 [19094.317262] RIP: 0010:lock_acquire+0xf0/0x2b0 [19094.321960] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19094.327177] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19094.328667] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19094.332499] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19094.336849] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19094.343974] [19094.343974] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19094.362715] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19094.369836] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19094.375053] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19094.382177] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19094.383673] [19094.390793] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19094.428597] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19094.435722] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19094.443799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19094.449535] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19094.456660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19094.463784] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19094.470908] Call Trace: [19094.473352] [19094.475363] ? watchdog_timer_fn+0x1b4/0x220 [19094.479628] ? __hrtimer_run_queues+0x1d3/0x360 [19094.484155] ? hrtimer_interrupt+0x104/0x240 [19094.488424] ? __local_bh_enable+0x50/0x60 [19094.492517] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19094.497733] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19094.502689] [19094.504787] [19094.506884] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19094.512192] ? lock_acquire+0xf0/0x2b0 [19094.515942] ? igrab+0x43/0x50 [19094.518994] ? lock_release+0xc5/0x260 [19094.522738] _raw_spin_lock+0x30/0x40 [19094.526402] ? igrab+0x1d/0x50 [19094.529471] igrab+0x1d/0x50 [19094.532347] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19094.537497] delayed_work+0x1d8/0x280 [ceph] [19094.541779] ? process_scheduled_works+0x22a/0x510 [19094.546569] process_scheduled_works+0x2a4/0x510 [19094.551185] worker_thread+0x188/0x340 [19094.554934] ? rescuer_thread+0x310/0x310 [19094.558938] kthread+0xf5/0x130 [19094.562076] ? kthread_complete_and_exit+0x20/0x20 [19094.566859] ret_from_fork+0x31/0x40 [19094.570429] ? kthread_complete_and_exit+0x20/0x20 [19094.575215] ret_from_fork_asm+0x11/0x20 [19094.579136] [19121.791529] watchdog: BUG: soft lockup - CPU#0 stuck for 6694s! [dbench:78683] [19121.798749] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19121.803529] watchdog: BUG: soft lockup - CPU#7 stuck for 15691s! [kworker/7:4:78808] [19121.854206] irq event stamp: 27789136 [19121.861936] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19121.865595] hardirqs last enabled at (27789135): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19121.872553] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [19121.882711] hardirqs last disabled at (27789136): [] sysvec_apic_timer_interrupt+0xe/0x90 [19121.882714] nf_defrag_ipv6 [19121.888361] nf_defrag_ipv4 nft_compat [19121.898087] softirqs last enabled at (27789118): [] __do_softirq+0x2be/0x371 [19121.900875] nf_tables nfnetlink [19121.904619] softirqs last disabled at (27789113): [] irq_exit_rcu+0xb0/0xd0 [19121.913305] bridge stp [19121.916529] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19121.925039] llc xfs [19121.927484] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19121.936427] libcrc32c sunrpc [19121.938612] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19121.946083] intel_rapl_msr [19121.949047] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19121.954524] intel_rapl_common sb_edac [19121.957315] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19121.976053] x86_pkg_temp_thermal coretemp [19121.979797] [19121.985013] kvm_intel kvm [19121.989105] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19121.990596] irqbypass i2c_i801 [19121.993300] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19122.000424] iTCO_wdt iTCO_vendor_support [19122.003561] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19122.010685] i2c_smbus crct10dif_pclmul [19122.014689] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19122.021812] joydev crc32_pclmul [19122.025643] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19122.032769] ipmi_ssif [19122.035993] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19122.043117] ghash_clmulni_intel mei_me [19122.045474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19122.053550] mei lpc_ich [19122.057381] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19122.063119] sha512_ssse3 acpi_ipmi [19122.065650] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19122.072775] mfd_core wmi [19122.076257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19122.083381] ipmi_si ipmi_devintf [19122.086000] Call Trace: [19122.086002] [19122.093124] ipmi_msghandler [19122.096438] ? watchdog_timer_fn+0x1b4/0x220 [19122.098877] ioatdma acpi_pad [19122.100891] ? __hrtimer_run_queues+0x1d3/0x360 [19122.103767] acpi_power_meter ixgbe [19122.108034] ? hrtimer_interrupt+0x104/0x240 [19122.110994] igb mdio [19122.115519] ? __local_bh_enable+0x50/0x60 [19122.119004] crc32c_intel nvme [19122.123269] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19122.125539] ptp [19122.129629] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19122.132680] nvme_core i2c_algo_bit [19122.137898] [19122.139735] pps_core dca [19122.144691] [19122.148174] fuse [19122.150275] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19122.152890] irq event stamp: 75767018 [19122.154991] ? queued_spin_lock_slowpath+0x16/0x2e0 [19122.156928] hardirqs last enabled at (75767017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19122.162236] do_raw_spin_lock+0xb5/0xc0 [19122.165890] hardirqs last disabled at (75767018): [] sysvec_apic_timer_interrupt+0xe/0x90 [19122.170764] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19122.180931] softirqs last enabled at (75767016): [] __do_softirq+0x2be/0x371 [19122.184762] try_get_cap_refs+0x159/0x880 [ceph] [19122.194482] softirqs last disabled at (75767011): [] irq_exit_rcu+0xb0/0xd0 [19122.199272] __ceph_get_caps+0x4a1/0x760 [ceph] [19122.207951] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19122.212565] ? __file_remove_privs+0x89/0x110 [19122.221072] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19122.225598] ? wake_bit_function+0x70/0x70 [19122.234975] Workqueue: events delayed_work [ceph] [19122.239327] ? ceph_write_iter+0x260/0x770 [ceph] [19122.246796] [19122.250887] ceph_write_iter+0x260/0x770 [ceph] [19122.255583] RIP: 0010:lock_acquire+0xf0/0x2b0 [19122.260286] ? lock_is_held_type+0xaa/0x120 [19122.261770] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19122.266298] vfs_write+0x324/0x4f0 [19122.270647] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19122.274828] ksys_pwrite64+0x65/0xa0 [19122.293560] [19122.293561] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19122.296959] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19122.302177] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19122.305749] do_syscall_64+0x38/0x80 [19122.307237] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19122.314363] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19122.319492] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19122.326618] RIP: 0033:0x7f48f0eeb93b [19122.330188] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19122.337313] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19122.342355] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19122.349479] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19122.353050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19122.360175] ORIG_RAX: 0000000000000012 [19122.360176] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19122.378929] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19122.387007] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19122.392225] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19122.397961] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19122.401794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19122.408933] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19122.416058] Call Trace: [19122.416060] [19122.423182] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19122.430309] ? watchdog_timer_fn+0x1b4/0x220 [19122.437436] [19122.444556] ? __hrtimer_run_queues+0x1d3/0x360 [19122.474234] ? hrtimer_interrupt+0x104/0x240 [19122.478504] ? __local_bh_enable+0x50/0x60 [19122.482596] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19122.487821] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19122.492777] [19122.494874] [19122.496972] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19122.502281] ? lock_acquire+0xf0/0x2b0 [19122.506032] ? igrab+0x43/0x50 [19122.509088] ? lock_release+0xc5/0x260 [19122.512834] _raw_spin_lock+0x30/0x40 [19122.516490] ? igrab+0x1d/0x50 [19122.519542] igrab+0x1d/0x50 [19122.522419] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19122.527568] delayed_work+0x1d8/0x280 [ceph] [19122.531850] ? process_scheduled_works+0x22a/0x510 [19122.536641] process_scheduled_works+0x2a4/0x510 [19122.541256] worker_thread+0x188/0x340 [19122.545004] ? rescuer_thread+0x310/0x310 [19122.549008] kthread+0xf5/0x130 [19122.552146] ? kthread_complete_and_exit+0x20/0x20 [19122.556937] ret_from_fork+0x31/0x40 [19122.560509] ? kthread_complete_and_exit+0x20/0x20 [19122.565293] ret_from_fork_asm+0x11/0x20 [19122.569223] [19149.792040] watchdog: BUG: soft lockup - CPU#0 stuck for 6720s! [dbench:78683] [19149.799252] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19149.804040] watchdog: BUG: soft lockup - CPU#7 stuck for 15717s! [kworker/7:4:78808] [19149.854710] irq event stamp: 27847188 [19149.854711] hardirqs last enabled at (27847187): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19149.862442] Modules linked in: ceph [19149.866099] hardirqs last disabled at (27847188): [] sysvec_apic_timer_interrupt+0xe/0x90 [19149.876255] libceph dns_resolver [19149.879740] softirqs last enabled at (27847170): [] __do_softirq+0x2be/0x371 [19149.889464] fscache netfs [19149.892776] softirqs last disabled at (27847165): [] irq_exit_rcu+0xb0/0xd0 [19149.901459] veth nft_chain_nat [19149.904163] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19149.912674] xt_MASQUERADE nf_nat [19149.915813] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19149.924755] nf_conntrack nf_defrag_ipv6 [19149.928066] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [19149.935538] nf_defrag_ipv4 nft_compat [19149.939454] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [19149.944931] nf_tables nfnetlink [19149.948677] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19149.967414] bridge stp [19149.970655] [19149.975873] llc xfs [19149.978316] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19149.979808] libcrc32c sunrpc [19149.981991] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19149.989116] intel_rapl_msr intel_rapl_common [19149.992081] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19149.999205] sb_edac x86_pkg_temp_thermal [19150.003555] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19150.010679] coretemp kvm_intel [19150.014684] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19150.021806] kvm irqbypass [19150.024944] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19150.032069] i2c_i801 iTCO_wdt [19150.034773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19150.042851] iTCO_vendor_support i2c_smbus [19150.045900] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19150.051639] crct10dif_pclmul [19150.055730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19150.062854] joydev crc32_pclmul [19150.065816] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19150.072941] ipmi_ssif [19150.076165] Call Trace: [19150.076166] [19150.083290] ghash_clmulni_intel mei_me [19150.085648] ? watchdog_timer_fn+0x1b4/0x220 [19150.088091] mei lpc_ich [19150.090104] ? __hrtimer_run_queues+0x1d3/0x360 [19150.093933] sha512_ssse3 acpi_ipmi [19150.098200] ? hrtimer_interrupt+0x104/0x240 [19150.100728] mfd_core [19150.105251] ? __local_bh_enable+0x50/0x60 [19150.108734] wmi ipmi_si [19150.113001] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19150.115270] ipmi_devintf ipmi_msghandler [19150.119362] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19150.121892] ioatdma acpi_pad [19150.127110] [19150.127111] [19150.131113] acpi_power_meter [19150.136070] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19150.139034] ixgbe igb [19150.141136] ? queued_spin_lock_slowpath+0x2b/0x2e0 [19150.143228] mdio crc32c_intel [19150.146195] do_raw_spin_lock+0xb5/0xc0 [19150.151497] nvme ptp [19150.153855] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19150.158725] nvme_core i2c_algo_bit [19150.161780] try_get_cap_refs+0x159/0x880 [ceph] [19150.165607] pps_core dca [19150.167882] __ceph_get_caps+0x4a1/0x760 [ceph] [19150.172662] fuse [19150.176152] ? __file_remove_privs+0x89/0x110 [19150.180756] irq event stamp: 75989702 [19150.183375] ? wake_bit_function+0x70/0x70 [19150.187897] hardirqs last enabled at (75989701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19150.189825] ? ceph_write_iter+0x260/0x770 [ceph] [19150.194174] hardirqs last disabled at (75989702): [] sysvec_apic_timer_interrupt+0xe/0x90 [19150.197830] ceph_write_iter+0x260/0x770 [ceph] [19150.201922] softirqs last enabled at (75989700): [] __do_softirq+0x2be/0x371 [19150.212081] ? lock_is_held_type+0xaa/0x120 [19150.216777] softirqs last disabled at (75989695): [] irq_exit_rcu+0xb0/0xd0 [19150.226503] vfs_write+0x324/0x4f0 [19150.231024] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19150.239727] ksys_pwrite64+0x65/0xa0 [19150.243902] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19150.252414] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19150.255813] Workqueue: events delayed_work [ceph] [19150.265190] do_syscall_64+0x38/0x80 [19150.268761] [19150.268762] RIP: 0010:__list_del_entry_valid_or_report+0x0/0xd0 [19150.276232] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19150.281361] Code: 1d 82 e8 83 0c b0 ff 0f 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 [19150.286057] RIP: 0033:0x7f48f0eeb93b [19150.289630] RSP: 0018:ffffc90002117db0 EFLAGS: 00000216 [19150.291120] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19150.297032] [19150.297032] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 00000001011f9a20 [19150.302073] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19150.320812] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [19150.324384] ORIG_RAX: 0000000000000012 [19150.329602] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [19150.348338] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19150.349828] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19150.356953] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19150.362172] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19150.369295] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19150.373126] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19150.380249] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19150.387373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19150.394499] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19150.401623] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19150.408753] [19150.415870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19150.460367] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19150.467490] Call Trace: [19150.469936] [19150.471945] ? watchdog_timer_fn+0x1b4/0x220 [19150.476209] ? __hrtimer_run_queues+0x1d3/0x360 [19150.480738] ? hrtimer_interrupt+0x104/0x240 [19150.485006] ? __local_bh_enable+0x50/0x60 [19150.489100] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19150.494324] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19150.499281] [19150.501378] [19150.503476] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19150.508784] ? __list_add_valid_or_report+0xa0/0xa0 [19150.513659] ceph_check_delayed_caps+0x8f/0x270 [ceph] [19150.518808] delayed_work+0x1d8/0x280 [ceph] [19150.523089] ? process_scheduled_works+0x22a/0x510 [19150.527871] process_scheduled_works+0x2a4/0x510 [19150.532488] worker_thread+0x188/0x340 [19150.536237] ? rescuer_thread+0x310/0x310 [19150.540241] kthread+0xf5/0x130 [19150.543378] ? kthread_complete_and_exit+0x20/0x20 [19150.548162] ret_from_fork+0x31/0x40 [19150.551730] ? kthread_complete_and_exit+0x20/0x20 [19150.556516] ret_from_fork_asm+0x11/0x20 [19150.560453] [19177.792554] watchdog: BUG: soft lockup - CPU#0 stuck for 6746s! [dbench:78683] [19177.799775] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19177.804554] watchdog: BUG: soft lockup - CPU#7 stuck for 15743s! [kworker/7:4:78808] [19177.855237] irq event stamp: 27905240 [19177.862973] Modules linked in: ceph libceph dns_resolver fscache netfs [19177.866630] hardirqs last enabled at (27905239): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19177.866635] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [19177.883313] hardirqs last disabled at (27905240): [] sysvec_apic_timer_interrupt+0xe/0x90 [19177.891998] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [19177.901721] softirqs last enabled at (27905222): [] __do_softirq+0x2be/0x371 [19177.908420] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [19177.917104] softirqs last disabled at (27905217): [] irq_exit_rcu+0xb0/0xd0 [19177.925356] kvm_intel kvm irqbypass i2c_i801 [19177.933869] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19177.933870] iTCO_wdt [19177.933871] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19177.938232] iTCO_vendor_support i2c_smbus [19177.947180] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19177.949449] crct10dif_pclmul joydev [19177.956921] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19177.961010] crc32_pclmul ipmi_ssif [19177.966489] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19177.970060] ghash_clmulni_intel mei_me [19177.988798] [19177.992280] mei lpc_ich [19177.997498] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19178.001329] sha512_ssse3 [19178.002821] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19178.005352] acpi_ipmi [19178.012475] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19178.015093] mfd_core wmi [19178.022232] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19178.024590] ipmi_si ipmi_devintf [19178.031714] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19178.034332] ipmi_msghandler ioatdma [19178.041456] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19178.044768] acpi_pad acpi_power_meter [19178.051893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19178.055461] ixgbe igb [19178.063539] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19178.067284] mdio [19178.073021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19178.075379] crc32c_intel nvme [19178.082502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19178.084427] ptp nvme_core [19178.091552] Call Trace: [19178.091554] [19178.094601] i2c_algo_bit pps_core [19178.101727] ? watchdog_timer_fn+0x1b4/0x220 [19178.104430] dca fuse [19178.106874] ? __hrtimer_run_queues+0x1d3/0x360 [19178.108885] [19178.108886] irq event stamp: 76211438 [19178.112285] ? hrtimer_interrupt+0x104/0x240 [19178.116547] hardirqs last enabled at (76211437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19178.118818] ? __local_bh_enable+0x50/0x60 [19178.123340] hardirqs last disabled at (76211438): [] sysvec_apic_timer_interrupt+0xe/0x90 [19178.124834] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19178.128490] softirqs last enabled at (76211436): [] __do_softirq+0x2be/0x371 [19178.132754] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19178.142910] softirqs last disabled at (76211431): [] irq_exit_rcu+0xb0/0xd0 [19178.147000] [19178.156725] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19178.161956] [19178.170646] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19178.175603] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19178.184113] Workqueue: events delayed_work [ceph] [19178.186215] ? queued_spin_lock_slowpath+0x18/0x2e0 [19178.195588] [19178.195589] RIP: 0010:rcu_is_watching+0x1c/0x40 [19178.197688] do_raw_spin_lock+0xb5/0xc0 [19178.205155] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 [19178.210460] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19178.215156] RSP: 0018:ffffc90002117d10 EFLAGS: 00000286 [19178.220031] try_get_cap_refs+0x159/0x880 [ceph] [19178.221519] [19178.221519] RAX: 0000000001f06464 RBX: ffff8881a1d34500 RCX: 0000000000000000 [19178.226049] __ceph_get_caps+0x4a1/0x760 [ceph] [19178.229875] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [19178.248616] ? __file_remove_privs+0x89/0x110 [19178.253394] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [19178.258612] ? wake_bit_function+0x70/0x70 [19178.263235] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [19178.264726] ? ceph_write_iter+0x260/0x770 [ceph] [19178.271846] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19178.276371] ceph_write_iter+0x260/0x770 [ceph] [19178.283496] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19178.287849] ? lock_is_held_type+0xaa/0x120 [19178.294969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19178.299064] vfs_write+0x324/0x4f0 [19178.306184] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19178.310888] ksys_pwrite64+0x65/0xa0 [19178.318006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19178.322531] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19178.330608] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19178.334786] do_syscall_64+0x38/0x80 [19178.340523] Call Trace: [19178.340524] [19178.343923] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19178.351045] ? watchdog_timer_fn+0x1b4/0x220 [19178.354616] RIP: 0033:0x7f48f0eeb93b [19178.361742] ? __hrtimer_run_queues+0x1d3/0x360 [19178.366872] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19178.373998] ? hrtimer_interrupt+0x104/0x240 [19178.377564] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19178.380010] ? __local_bh_enable+0x50/0x60 [19178.382020] ORIG_RAX: 0000000000000012 [19178.387068] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19178.391328] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19178.394899] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19178.399423] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19178.418160] [19178.422423] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19178.427643] [19178.427644] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19178.431733] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19178.435568] ? rcu_is_watching+0x1c/0x40 [19178.440781] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19178.447906] lock_acquire+0x184/0x2b0 [19178.452868] [19178.459986] ? igrab+0x43/0x50 [19178.503667] ? lock_release+0xc5/0x260 [19178.507414] _raw_spin_lock+0x30/0x40 [19178.511070] ? igrab+0x1d/0x50 [19178.514121] igrab+0x1d/0x50 [19178.517000] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19178.522149] delayed_work+0x1d8/0x280 [ceph] [19178.526430] ? process_scheduled_works+0x22a/0x510 [19178.531220] process_scheduled_works+0x2a4/0x510 [19178.535853] worker_thread+0x188/0x340 [19178.539600] ? rescuer_thread+0x310/0x310 [19178.543604] kthread+0xf5/0x130 [19178.546742] ? kthread_complete_and_exit+0x20/0x20 [19178.551528] ret_from_fork+0x31/0x40 [19178.555106] ? kthread_complete_and_exit+0x20/0x20 [19178.559889] ret_from_fork_asm+0x11/0x20 [19178.563813] [19205.793068] watchdog: BUG: soft lockup - CPU#0 stuck for 6772s! [dbench:78683] [19205.800280] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19205.805068] watchdog: BUG: soft lockup - CPU#7 stuck for 15769s! [kworker/7:4:78808] [19205.855720] irq event stamp: 27963254 [19205.855721] hardirqs last enabled at (27963253): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19205.863467] Modules linked in: [19205.867126] hardirqs last disabled at (27963254): [] sysvec_apic_timer_interrupt+0xe/0x90 [19205.877283] ceph libceph [19205.880333] softirqs last enabled at (27963236): [] __do_softirq+0x2be/0x371 [19205.890058] dns_resolver fscache [19205.892676] softirqs last disabled at (27963231): [] irq_exit_rcu+0xb0/0xd0 [19205.901360] netfs veth [19205.904670] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19205.913181] nft_chain_nat [19205.915625] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19205.924570] xt_MASQUERADE nf_nat [19205.927272] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19205.934756] nf_conntrack nf_defrag_ipv6 [19205.938063] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19205.943541] nf_defrag_ipv4 [19205.947466] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19205.966205] nft_compat nf_tables [19205.968995] [19205.968996] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19205.974213] nfnetlink bridge [19205.977524] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19205.979015] stp llc [19205.986139] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19205.989103] xfs libcrc32c [19205.996227] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19205.998411] sunrpc [19206.005534] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19206.008240] intel_rapl_msr intel_rapl_common [19206.015362] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19206.017465] sb_edac x86_pkg_temp_thermal [19206.024592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19206.028944] coretemp kvm_intel [19206.037021] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19206.041025] kvm [19206.046764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19206.049900] irqbypass i2c_i801 [19206.057025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19206.058862] iTCO_wdt iTCO_vendor_support [19206.065986] Call Trace: [19206.069124] i2c_smbus crct10dif_pclmul [19206.076247] [19206.080252] joydev crc32_pclmul [19206.082697] ? watchdog_timer_fn+0x1b4/0x220 [19206.086525] ipmi_ssif ghash_clmulni_intel [19206.088537] ? __hrtimer_run_queues+0x1d3/0x360 [19206.091760] mei_me mei [19206.096027] ? hrtimer_interrupt+0x104/0x240 [19206.100114] lpc_ich sha512_ssse3 [19206.104640] ? __local_bh_enable+0x50/0x60 [19206.107084] acpi_ipmi mfd_core [19206.111351] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19206.114657] wmi ipmi_si [19206.118757] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19206.121895] ipmi_devintf ipmi_msghandler [19206.127113] [19206.129643] ioatdma acpi_pad [19206.134602] [19206.138606] acpi_power_meter [19206.140704] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19206.143667] ixgbe igb [19206.145767] ? queued_spin_lock_slowpath+0x18/0x2e0 [19206.148727] mdio crc32c_intel [19206.154034] do_raw_spin_lock+0xb5/0xc0 [19206.156389] nvme ptp [19206.161260] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19206.164310] nvme_core [19206.168144] try_get_cap_refs+0x159/0x880 [ceph] [19206.170413] i2c_algo_bit pps_core [19206.175200] __ceph_get_caps+0x4a1/0x760 [ceph] [19206.177554] dca fuse [19206.182169] ? __file_remove_privs+0x89/0x110 [19206.185562] [19206.185563] irq event stamp: 76433436 [19206.190087] ? wake_bit_function+0x70/0x70 [19206.192356] hardirqs last enabled at (76433435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19206.196709] ? ceph_write_iter+0x260/0x770 [ceph] [19206.198197] hardirqs last disabled at (76433436): [] sysvec_apic_timer_interrupt+0xe/0x90 [19206.201855] ceph_write_iter+0x260/0x770 [ceph] [19206.205946] softirqs last enabled at (76433434): [] __do_softirq+0x2be/0x371 [19206.216107] ? lock_is_held_type+0xaa/0x120 [19206.220802] softirqs last disabled at (76433429): [] irq_exit_rcu+0xb0/0xd0 [19206.230527] vfs_write+0x324/0x4f0 [19206.235050] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19206.243753] ksys_pwrite64+0x65/0xa0 [19206.247929] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19206.256440] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19206.259837] Workqueue: events delayed_work [ceph] [19206.269216] do_syscall_64+0x38/0x80 [19206.272786] [19206.272787] RIP: 0010:lock_acquire+0xf0/0x2b0 [19206.280256] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19206.285385] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19206.290083] RIP: 0033:0x7f48f0eeb93b [19206.293653] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19206.295145] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19206.299497] [19206.299497] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19206.304540] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19206.323277] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19206.326847] ORIG_RAX: 0000000000000012 [19206.332065] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19206.350803] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19206.352293] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19206.359419] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19206.364636] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19206.371761] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19206.375589] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19206.382715] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19206.389839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19206.396963] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19206.404088] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19206.411216] [19206.418335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19206.462831] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19206.469956] Call Trace: [19206.472400] [19206.474412] ? watchdog_timer_fn+0x1b4/0x220 [19206.478676] ? __hrtimer_run_queues+0x1d3/0x360 [19206.483203] ? hrtimer_interrupt+0x104/0x240 [19206.487475] ? __local_bh_enable+0x50/0x60 [19206.491573] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19206.496796] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19206.501755] [19206.503852] [19206.505948] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19206.511257] ? lock_acquire+0xf0/0x2b0 [19206.515008] ? igrab+0x43/0x50 [19206.518057] ? lock_release+0xc5/0x260 [19206.521803] _raw_spin_lock+0x30/0x40 [19206.525475] ? igrab+0x1d/0x50 [19206.528527] igrab+0x1d/0x50 [19206.531403] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19206.536555] delayed_work+0x1d8/0x280 [ceph] [19206.540837] ? process_scheduled_works+0x22a/0x510 [19206.545626] process_scheduled_works+0x2a4/0x510 [19206.550241] worker_thread+0x188/0x340 [19206.553989] ? rescuer_thread+0x310/0x310 [19206.557994] kthread+0xf5/0x130 [19206.561130] ? kthread_complete_and_exit+0x20/0x20 [19206.565916] ret_from_fork+0x31/0x40 [19206.569486] ? kthread_complete_and_exit+0x20/0x20 [19206.574269] ret_from_fork_asm+0x11/0x20 [19206.578192] [19233.187939] rcu: INFO: rcu_preempt self-detected stall on CPU [19233.193699] rcu: 7-....: (7086759 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/7085510 fqs=3550996 [19233.204123] rcu: (t=7296447 jiffies g=1309609 q=270667 ncpus=8) [19233.210124] Sending NMI from CPU 7 to CPUs 0: [19233.214476] NMI backtrace for cpu 0 [19233.214478] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19233.214480] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19233.214481] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19233.214487] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19233.214489] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19233.214492] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19233.214493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19233.214494] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19233.214496] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19233.214497] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19233.214498] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19233.214500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19233.214501] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19233.214503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19233.214504] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19233.214505] Call Trace: [19233.214507] [19233.214509] ? nmi_cpu_backtrace+0xad/0x120 [19233.214513] ? nmi_cpu_backtrace_handler+0x11/0x20 [19233.214516] ? nmi_handle+0xd4/0x260 [19233.214522] ? default_do_nmi+0x49/0xf0 [19233.214525] ? exc_nmi+0x11e/0x150 [19233.214527] ? end_repeat_nmi+0x16/0x67 [19233.214534] ? queued_spin_lock_slowpath+0x16/0x2e0 [19233.214538] ? queued_spin_lock_slowpath+0x16/0x2e0 [19233.214542] ? queued_spin_lock_slowpath+0x16/0x2e0 [19233.214545] [19233.214546] [19233.214548] do_raw_spin_lock+0xb5/0xc0 [19233.214552] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19233.214578] try_get_cap_refs+0x159/0x880 [ceph] [19233.214600] __ceph_get_caps+0x4a1/0x760 [ceph] [19233.214620] ? __file_remove_privs+0x89/0x110 [19233.214624] ? wake_bit_function+0x70/0x70 [19233.214628] ? ceph_write_iter+0x260/0x770 [ceph] [19233.214644] ceph_write_iter+0x260/0x770 [ceph] [19233.214662] ? lock_is_held_type+0xaa/0x120 [19233.214667] vfs_write+0x324/0x4f0 [19233.214675] ksys_pwrite64+0x65/0xa0 [19233.214677] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19233.214681] do_syscall_64+0x38/0x80 [19233.214684] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19233.214687] RIP: 0033:0x7f48f0eeb93b [19233.214689] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19233.214691] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [19233.214693] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19233.214694] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19233.214695] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19233.214697] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19233.214698] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19233.214704] [19233.215474] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19233.519346] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19233.526816] Workqueue: events delayed_work [ceph] [19233.531530] RIP: 0010:lock_acquire+0xf0/0x2b0 [19233.535882] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19233.554619] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19233.559837] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19233.566974] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19233.574102] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19233.581226] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19233.588351] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19233.595475] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19233.603551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19233.609290] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19233.616414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19233.623538] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19233.630661] Call Trace: [19233.633107] [19233.635118] ? rcu_dump_cpu_stacks+0xd1/0x180 [19233.639476] ? rcu_sched_clock_irq+0x633/0x1390 [19233.644000] ? lock_release+0xc5/0x260 [19233.647746] ? lock_is_held_type+0xaa/0x120 [19233.651924] ? lock_is_held_type+0xaa/0x120 [19233.656102] ? tick_sched_do_timer+0xa0/0xa0 [19233.660373] ? rcu_read_lock_sched_held+0x56/0x90 [19233.665071] ? tick_sched_do_timer+0xa0/0xa0 [19233.669334] ? update_process_times+0x5f/0x90 [19233.673686] ? tick_sched_handle.isra.16+0x1e/0x50 [19233.678469] ? tick_sched_timer+0x6f/0x80 [19233.682475] ? __hrtimer_run_queues+0x1d3/0x360 [19233.687008] ? hrtimer_interrupt+0x104/0x240 [19233.691277] ? __local_bh_enable+0x50/0x60 [19233.695373] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19233.700595] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19233.705552] [19233.707649] [19233.709746] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19233.715055] ? lock_acquire+0xf0/0x2b0 [19233.718806] ? igrab+0x43/0x50 [19233.721863] ? lock_release+0xc5/0x260 [19233.725608] _raw_spin_lock+0x30/0x40 [19233.729266] ? igrab+0x1d/0x50 [19233.732316] igrab+0x1d/0x50 [19233.735195] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19233.740345] delayed_work+0x1d8/0x280 [ceph] [19233.744625] ? process_scheduled_works+0x22a/0x510 [19233.749414] process_scheduled_works+0x2a4/0x510 [19233.754033] worker_thread+0x188/0x340 [19233.757780] ? rescuer_thread+0x310/0x310 [19233.761784] kthread+0xf5/0x130 [19233.764931] ? kthread_complete_and_exit+0x20/0x20 [19233.769713] ret_from_fork+0x31/0x40 [19233.773293] ? kthread_complete_and_exit+0x20/0x20 [19233.778077] ret_from_fork_asm+0x11/0x20 [19233.782001] [19257.794024] watchdog: BUG: soft lockup - CPU#0 stuck for 6821s! [dbench:78683] [19257.801236] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19257.806024] watchdog: BUG: soft lockup - CPU#7 stuck for 15818s! [kworker/7:4:78808] [19257.856697] irq event stamp: 28072218 [19257.864424] Modules linked in: ceph libceph dns_resolver [19257.868082] hardirqs last enabled at (28072217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19257.873387] fscache netfs veth [19257.883545] hardirqs last disabled at (28072218): [] sysvec_apic_timer_interrupt+0xe/0x90 [19257.883547] softirqs last enabled at (28072092): [] __do_softirq+0x2be/0x371 [19257.886698] nft_chain_nat xt_MASQUERADE [19257.896423] softirqs last disabled at (28072087): [] irq_exit_rcu+0xb0/0xd0 [19257.905108] nf_nat [19257.909025] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19257.917535] nf_conntrack nf_defrag_ipv6 [19257.919633] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19257.928577] nf_defrag_ipv4 nft_compat [19257.932494] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19257.939964] nf_tables nfnetlink [19257.943709] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19257.949186] bridge [19257.952418] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19257.971157] stp llc [19257.973255] [19257.973256] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19257.978471] xfs libcrc32c [19257.980657] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19257.982148] sunrpc intel_rapl_msr [19257.989272] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19257.991976] intel_rapl_common [19257.999100] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19258.002498] sb_edac [19258.009621] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19258.012671] x86_pkg_temp_thermal [19258.019795] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19258.021979] coretemp kvm_intel [19258.029103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19258.032420] kvm irqbypass [19258.040500] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19258.043637] i2c_i801 [19258.049376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19258.052079] iTCO_wdt iTCO_vendor_support [19258.059204] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19258.061475] i2c_smbus [19258.068598] Call Trace: [19258.072602] crct10dif_pclmul joydev [19258.079729] [19258.082085] crc32_pclmul ipmi_ssif [19258.084528] ? watchdog_timer_fn+0x1b4/0x220 [19258.088100] ghash_clmulni_intel [19258.090111] ? __hrtimer_run_queues+0x1d3/0x360 [19258.093593] mei_me mei [19258.097861] ? hrtimer_interrupt+0x104/0x240 [19258.101084] lpc_ich [19258.105605] ? __local_bh_enable+0x50/0x60 [19258.108049] sha512_ssse3 acpi_ipmi [19258.112317] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19258.114497] mfd_core wmi [19258.118588] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19258.122072] ipmi_si ipmi_devintf [19258.127291] [19258.129908] ipmi_msghandler ioatdma [19258.134865] [19258.138176] acpi_pad [19258.140273] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19258.143844] acpi_power_meter ixgbe [19258.145944] ? queued_spin_lock_slowpath+0x18/0x2e0 [19258.148211] igb [19258.153519] do_raw_spin_lock+0xb5/0xc0 [19258.157000] mdio crc32c_intel [19258.161872] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19258.163711] nvme ptp [19258.167542] try_get_cap_refs+0x159/0x880 [ceph] [19258.170590] nvme_core i2c_algo_bit [19258.175379] __ceph_get_caps+0x4a1/0x760 [ceph] [19258.177644] pps_core [19258.182260] ? __file_remove_privs+0x89/0x110 [19258.185739] dca fuse [19258.190263] ? wake_bit_function+0x70/0x70 [19258.192535] [19258.192536] irq event stamp: 76845442 [19258.196886] ? ceph_write_iter+0x260/0x770 [ceph] [19258.199155] hardirqs last enabled at (76845441): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19258.203245] ceph_write_iter+0x260/0x770 [ceph] [19258.204737] hardirqs last disabled at (76845442): [] sysvec_apic_timer_interrupt+0xe/0x90 [19258.208402] ? lock_is_held_type+0xaa/0x120 [19258.213091] softirqs last enabled at (76845440): [] __do_softirq+0x2be/0x371 [19258.223251] vfs_write+0x324/0x4f0 [19258.227773] softirqs last disabled at (76845435): [] irq_exit_rcu+0xb0/0xd0 [19258.237503] ksys_pwrite64+0x65/0xa0 [19258.241675] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19258.250359] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19258.253756] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19258.262269] do_syscall_64+0x38/0x80 [19258.265838] Workqueue: events delayed_work [ceph] [19258.275216] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19258.280347] [19258.280348] RIP: 0010:lock_release+0x11/0x260 [19258.287817] RIP: 0033:0x7f48f0eeb93b [19258.291389] Code: 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd <53> 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f [19258.296086] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19258.301129] RSP: 0018:ffffc90002117d68 EFLAGS: 00000286 [19258.302619] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19258.306971] [19258.306972] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [19258.310541] ORIG_RAX: 0000000000000012 [19258.329280] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [19258.348018] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19258.353235] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [19258.358451] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19258.359943] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [19258.367067] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19258.370896] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19258.378021] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19258.385145] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19258.392270] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19258.399393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19258.406524] [19258.413641] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19258.458138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19258.465261] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19258.472385] Call Trace: [19258.474831] [19258.476842] ? watchdog_timer_fn+0x1b4/0x220 [19258.481106] ? __hrtimer_run_queues+0x1d3/0x360 [19258.485632] ? hrtimer_interrupt+0x104/0x240 [19258.489892] ? __local_bh_enable+0x50/0x60 [19258.493987] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19258.499211] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19258.504169] [19258.506265] [19258.508363] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19258.513668] ? igrab+0x43/0x50 [19258.516718] ? igrab+0x43/0x50 [19258.519767] ? lock_release+0x11/0x260 [19258.523515] _raw_spin_unlock+0x1b/0x40 [19258.527352] igrab+0x43/0x50 [19258.530231] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19258.535381] delayed_work+0x1d8/0x280 [ceph] [19258.539660] ? process_scheduled_works+0x22a/0x510 [19258.544449] process_scheduled_works+0x2a4/0x510 [19258.549066] worker_thread+0x188/0x340 [19258.552816] ? rescuer_thread+0x310/0x310 [19258.556819] kthread+0xf5/0x130 [19258.559956] ? kthread_complete_and_exit+0x20/0x20 [19258.564741] ret_from_fork+0x31/0x40 [19258.568311] ? kthread_complete_and_exit+0x20/0x20 [19258.573094] ret_from_fork_asm+0x11/0x20 [19258.577017] [19285.794538] watchdog: BUG: soft lockup - CPU#0 stuck for 6847s! [dbench:78683] [19285.801759] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19285.806538] watchdog: BUG: soft lockup - CPU#7 stuck for 15844s! [kworker/7:4:78808] [19285.857213] irq event stamp: 28130352 [19285.857215] hardirqs last enabled at (28130351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19285.864947] Modules linked in: [19285.868605] hardirqs last disabled at (28130352): [] sysvec_apic_timer_interrupt+0xe/0x90 [19285.878764] ceph [19285.881814] softirqs last enabled at (28130296): [] __do_softirq+0x2be/0x371 [19285.891538] libceph [19285.893461] softirqs last disabled at (28130291): [] irq_exit_rcu+0xb0/0xd0 [19285.902145] dns_resolver [19285.904331] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19285.912840] fscache netfs [19285.915457] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19285.924402] veth [19285.927107] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19285.934578] nft_chain_nat [19285.936501] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19285.941978] xt_MASQUERADE nf_nat [19285.944683] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19285.963419] nf_conntrack nf_defrag_ipv6 [19285.966732] [19285.966732] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19285.971949] nf_defrag_ipv4 nft_compat [19285.975866] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19285.977355] nf_tables nfnetlink [19285.984481] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19285.988226] bridge [19285.995348] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19285.998573] stp llc [19286.005696] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19286.007795] xfs libcrc32c [19286.014935] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19286.017118] sunrpc intel_rapl_msr [19286.024244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19286.026949] intel_rapl_common [19286.035024] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19286.038424] sb_edac [19286.044161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19286.047218] x86_pkg_temp_thermal coretemp [19286.054343] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19286.056529] kvm_intel kvm [19286.063652] Call Trace: [19286.063654] [19286.067742] irqbypass i2c_i801 [19286.074869] ? watchdog_timer_fn+0x1b4/0x220 [19286.077570] iTCO_wdt [19286.080016] ? __hrtimer_run_queues+0x1d3/0x360 [19286.082026] iTCO_vendor_support i2c_smbus [19286.085167] ? hrtimer_interrupt+0x104/0x240 [19286.089426] crct10dif_pclmul joydev [19286.091697] ? __local_bh_enable+0x50/0x60 [19286.096234] crc32_pclmul ipmi_ssif [19286.100325] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19286.104584] ghash_clmulni_intel mei_me [19286.108158] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19286.112247] mei lpc_ich [19286.115730] [19286.120949] sha512_ssse3 acpi_ipmi [19286.124778] [19286.124780] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19286.129737] mfd_core [19286.132270] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19286.134365] wmi [19286.137851] do_raw_spin_lock+0xb5/0xc0 [19286.139946] ipmi_si ipmi_devintf [19286.145252] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19286.147521] ipmi_msghandler [19286.152395] try_get_cap_refs+0x159/0x880 [ceph] [19286.154230] ioatdma acpi_pad [19286.158065] __ceph_get_caps+0x4a1/0x760 [ceph] [19286.161371] acpi_power_meter [19286.166160] ? __file_remove_privs+0x89/0x110 [19286.169033] ixgbe igb [19286.173642] ? wake_bit_function+0x70/0x70 [19286.176607] mdio crc32c_intel [19286.181132] ? ceph_write_iter+0x260/0x770 [ceph] [19286.184094] nvme [19286.188446] ceph_write_iter+0x260/0x770 [ceph] [19286.190802] ptp nvme_core [19286.194898] ? lock_is_held_type+0xaa/0x120 [19286.197943] i2c_algo_bit pps_core [19286.202645] vfs_write+0x324/0x4f0 [19286.204565] dca fuse [19286.209094] ksys_pwrite64+0x65/0xa0 [19286.211794] [19286.211795] irq event stamp: 77066914 [19286.215972] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19286.219368] hardirqs last enabled at (77066913): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19286.222767] do_syscall_64+0x38/0x80 [19286.225038] hardirqs last disabled at (77066914): [] sysvec_apic_timer_interrupt+0xe/0x90 [19286.228608] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19286.230098] softirqs last enabled at (77066912): [] __do_softirq+0x2be/0x371 [19286.233758] RIP: 0033:0x7f48f0eeb93b [19286.238887] softirqs last disabled at (77066907): [] irq_exit_rcu+0xb0/0xd0 [19286.249046] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19286.252615] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19286.262340] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19286.267383] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19286.276068] ORIG_RAX: 0000000000000012 [19286.279640] Workqueue: events delayed_work [ceph] [19286.288149] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19286.306886] [19286.316264] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19286.321480] RIP: 0010:lock_acquire+0xf0/0x2b0 [19286.328952] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19286.332782] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19286.337479] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19286.344605] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19286.346095] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19286.353221] [19286.353221] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19286.357576] [19286.364695] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19286.420820] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19286.427944] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19286.435069] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19286.442193] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19286.450271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19286.456007] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19286.463131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19286.470255] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19286.477381] Call Trace: [19286.479825] [19286.481836] ? watchdog_timer_fn+0x1b4/0x220 [19286.486100] ? __hrtimer_run_queues+0x1d3/0x360 [19286.490626] ? hrtimer_interrupt+0x104/0x240 [19286.494888] ? __local_bh_enable+0x50/0x60 [19286.498980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19286.504222] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19286.509179] [19286.511275] [19286.513373] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19286.518682] ? lock_acquire+0xf0/0x2b0 [19286.522432] ? igrab+0x43/0x50 [19286.525483] ? lock_release+0xc5/0x260 [19286.529228] _raw_spin_lock+0x30/0x40 [19286.532891] ? igrab+0x1d/0x50 [19286.535944] igrab+0x1d/0x50 [19286.538821] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19286.543973] delayed_work+0x1d8/0x280 [ceph] [19286.548252] ? process_scheduled_works+0x22a/0x510 [19286.553042] process_scheduled_works+0x2a4/0x510 [19286.557657] worker_thread+0x188/0x340 [19286.561406] ? rescuer_thread+0x310/0x310 [19286.565409] kthread+0xf5/0x130 [19286.568549] ? kthread_complete_and_exit+0x20/0x20 [19286.573331] ret_from_fork+0x31/0x40 [19286.576912] ? kthread_complete_and_exit+0x20/0x20 [19286.581696] ret_from_fork_asm+0x11/0x20 [19286.585629] [19311.567533] libceph: auth protocol 'cephx' mauth authentication failed: -13 [19313.795053] watchdog: BUG: soft lockup - CPU#0 stuck for 6873s! [dbench:78683] [19313.802270] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19313.807053] watchdog: BUG: soft lockup - CPU#7 stuck for 15870s! [kworker/7:4:78808] [19313.857726] irq event stamp: 28188558 [19313.857727] hardirqs last enabled at (28188557): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19313.865460] Modules linked in: [19313.869117] hardirqs last disabled at (28188558): [] sysvec_apic_timer_interrupt+0xe/0x90 [19313.879276] ceph [19313.882328] softirqs last enabled at (28188552): [] __do_softirq+0x2be/0x371 [19313.892050] libceph dns_resolver [19313.893973] softirqs last disabled at (28188547): [] irq_exit_rcu+0xb0/0xd0 [19313.902657] fscache netfs [19313.905971] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19313.914481] veth nft_chain_nat [19313.917185] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19313.926128] xt_MASQUERADE [19313.929266] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19313.936745] nf_nat nf_conntrack [19313.939450] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19313.944928] nf_defrag_ipv6 nf_defrag_ipv4 [19313.948151] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19313.966888] nft_compat nf_tables [19313.970980] [19313.970981] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19313.976195] nfnetlink bridge [19313.979508] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19313.980999] stp llc [19313.988123] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19313.991087] xfs libcrc32c [19313.998211] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19314.000394] sunrpc intel_rapl_msr [19314.007519] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19314.010222] intel_rapl_common [19314.017347] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19314.020744] sb_edac x86_pkg_temp_thermal [19314.027869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19314.030920] coretemp kvm_intel [19314.038996] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19314.043001] kvm [19314.048739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19314.051875] irqbypass i2c_i801 [19314.058999] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19314.060837] iTCO_wdt iTCO_vendor_support [19314.067962] Call Trace: [19314.067963] [19314.071099] i2c_smbus crct10dif_pclmul [19314.078224] ? watchdog_timer_fn+0x1b4/0x220 [19314.082226] joydev crc32_pclmul [19314.084672] ? __hrtimer_run_queues+0x1d3/0x360 [19314.086683] ipmi_ssif ghash_clmulni_intel [19314.090514] ? hrtimer_interrupt+0x104/0x240 [19314.094776] mei_me mei [19314.098000] ? __local_bh_enable+0x50/0x60 [19314.102524] lpc_ich sha512_ssse3 [19314.106618] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19314.110879] acpi_ipmi mfd_core [19314.113323] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19314.117413] wmi ipmi_si [19314.120740] [19314.125960] ipmi_devintf ipmi_msghandler [19314.129096] [19314.134055] ioatdma acpi_pad [19314.136587] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19314.138683] acpi_power_meter ixgbe [19314.142691] ? queued_spin_lock_slowpath+0x16/0x2e0 [19314.144783] igb mdio [19314.147749] do_raw_spin_lock+0xb5/0xc0 [19314.153051] crc32c_intel nvme [19314.156536] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19314.161408] ptp nvme_core [19314.163681] try_get_cap_refs+0x159/0x880 [ceph] [19314.167508] i2c_algo_bit pps_core [19314.170564] __ceph_get_caps+0x4a1/0x760 [ceph] [19314.175342] dca fuse [19314.178052] ? __file_remove_privs+0x89/0x110 [19314.182658] [19314.182659] irq event stamp: 77287796 [19314.186056] ? wake_bit_function+0x70/0x70 [19314.190578] hardirqs last enabled at (77287795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19314.192853] ? ceph_write_iter+0x260/0x770 [ceph] [19314.197201] hardirqs last disabled at (77287796): [] sysvec_apic_timer_interrupt+0xe/0x90 [19314.198692] ceph_write_iter+0x260/0x770 [ceph] [19314.202350] softirqs last enabled at (77287794): [] __do_softirq+0x2be/0x371 [19314.206458] ? lock_is_held_type+0xaa/0x120 [19314.216613] softirqs last disabled at (77287789): [] irq_exit_rcu+0xb0/0xd0 [19314.221314] vfs_write+0x324/0x4f0 [19314.231036] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19314.235565] ksys_pwrite64+0x65/0xa0 [19314.244244] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19314.248423] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19314.256933] Workqueue: events delayed_work [ceph] [19314.260331] do_syscall_64+0x38/0x80 [19314.269707] [19314.269708] RIP: 0010:lock_acquire+0xf0/0x2b0 [19314.273280] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19314.280748] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19314.285880] RIP: 0033:0x7f48f0eeb93b [19314.290577] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19314.294147] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19314.295637] [19314.295638] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19314.299989] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19314.305033] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19314.323770] ORIG_RAX: 0000000000000012 [19314.327342] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19314.332558] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19314.351296] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19314.352787] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19314.359913] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19314.365128] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19314.372253] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19314.376085] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19314.383208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19314.390333] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19314.397459] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19314.404586] [19314.411706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19314.463324] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19314.470459] Call Trace: [19314.472902] [19314.474914] ? watchdog_timer_fn+0x1b4/0x220 [19314.479178] ? __hrtimer_run_queues+0x1d3/0x360 [19314.483704] ? hrtimer_interrupt+0x104/0x240 [19314.487973] ? __local_bh_enable+0x50/0x60 [19314.492067] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19314.497289] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19314.502247] [19314.504344] [19314.506460] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19314.511767] ? lock_acquire+0xf0/0x2b0 [19314.515518] ? igrab+0x43/0x50 [19314.518569] ? lock_release+0xc5/0x260 [19314.522314] _raw_spin_lock+0x30/0x40 [19314.525977] ? igrab+0x1d/0x50 [19314.529030] igrab+0x1d/0x50 [19314.531906] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19314.537057] delayed_work+0x1d8/0x280 [ceph] [19314.541336] ? process_scheduled_works+0x22a/0x510 [19314.546118] process_scheduled_works+0x2a4/0x510 [19314.550746] worker_thread+0x188/0x340 [19314.554492] ? rescuer_thread+0x310/0x310 [19314.558496] kthread+0xf5/0x130 [19314.561632] ? kthread_complete_and_exit+0x20/0x20 [19314.566416] ret_from_fork+0x31/0x40 [19314.569987] ? kthread_complete_and_exit+0x20/0x20 [19314.574773] ret_from_fork_asm+0x11/0x20 [19314.578695] [19322.319058] libceph: auth protocol 'cephx' mauth authentication failed: -13 [19341.795567] watchdog: BUG: soft lockup - CPU#0 stuck for 6899s! [dbench:78683] [19341.802785] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19341.807567] watchdog: BUG: soft lockup - CPU#7 stuck for 15896s! [kworker/7:4:78808] [19341.858227] irq event stamp: 28246734 [19341.858228] hardirqs last enabled at (28246733): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19341.865974] Modules linked in: [19341.869631] hardirqs last disabled at (28246734): [] sysvec_apic_timer_interrupt+0xe/0x90 [19341.879788] ceph [19341.882840] softirqs last enabled at (28246668): [] __do_softirq+0x2be/0x371 [19341.892564] libceph [19341.894489] softirqs last disabled at (28246663): [] irq_exit_rcu+0xb0/0xd0 [19341.903173] dns_resolver [19341.905356] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19341.913868] fscache netfs [19341.916484] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19341.925428] veth nft_chain_nat [19341.928133] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19341.935604] xt_MASQUERADE nf_nat [19341.938740] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19341.944217] nf_conntrack nf_defrag_ipv6 [19341.947529] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19341.966267] nf_defrag_ipv4 nft_compat [19341.970184] [19341.970185] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19341.975401] nf_tables [19341.979145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19341.980636] nfnetlink [19341.987760] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19341.990118] bridge stp [19341.997257] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19341.999616] llc xfs [19342.006739] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19342.009184] libcrc32c sunrpc [19342.016308] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19342.018492] intel_rapl_msr [19342.025618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19342.028582] intel_rapl_common sb_edac [19342.036658] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19342.039449] x86_pkg_temp_thermal coretemp [19342.045186] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19342.048932] kvm_intel kvm [19342.056055] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19342.060145] irqbypass i2c_i801 [19342.067270] Call Trace: [19342.067271] [19342.069973] iTCO_wdt [19342.077099] ? watchdog_timer_fn+0x1b4/0x220 [19342.080243] iTCO_vendor_support i2c_smbus [19342.082690] ? __hrtimer_run_queues+0x1d3/0x360 [19342.084700] crct10dif_pclmul joydev [19342.086973] ? hrtimer_interrupt+0x104/0x240 [19342.091245] crc32_pclmul ipmi_ssif [19342.095332] ? __local_bh_enable+0x50/0x60 [19342.099857] ghash_clmulni_intel mei_me [19342.103429] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19342.107693] mei lpc_ich [19342.111178] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19342.115266] sha512_ssse3 acpi_ipmi [19342.119098] [19342.124314] mfd_core wmi [19342.126846] [19342.126848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19342.131802] ipmi_si [19342.135292] ? queued_spin_lock_slowpath+0x18/0x2e0 [19342.137384] ipmi_devintf ipmi_msghandler [19342.140004] do_raw_spin_lock+0xb5/0xc0 [19342.142100] ioatdma acpi_pad [19342.147404] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19342.149588] acpi_power_meter [19342.154460] try_get_cap_refs+0x159/0x880 [ceph] [19342.158462] ixgbe igb [19342.162297] __ceph_get_caps+0x4a1/0x760 [ceph] [19342.165257] mdio crc32c_intel [19342.170044] ? __file_remove_privs+0x89/0x110 [19342.173004] nvme [19342.177615] ? wake_bit_function+0x70/0x70 [19342.179973] ptp nvme_core [19342.184499] ? ceph_write_iter+0x260/0x770 [ceph] [19342.187548] i2c_algo_bit pps_core [19342.191899] ceph_write_iter+0x260/0x770 [ceph] [19342.193822] dca fuse [19342.197916] ? lock_is_held_type+0xaa/0x120 [19342.200617] [19342.200618] irq event stamp: 77508420 [19342.205318] vfs_write+0x324/0x4f0 [19342.208712] hardirqs last enabled at (77508419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19342.213251] ksys_pwrite64+0x65/0xa0 [19342.215516] hardirqs last disabled at (77508420): [] sysvec_apic_timer_interrupt+0xe/0x90 [19342.219693] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19342.221185] softirqs last enabled at (77508418): [] __do_softirq+0x2be/0x371 [19342.224843] do_syscall_64+0x38/0x80 [19342.228255] softirqs last disabled at (77508413): [] irq_exit_rcu+0xb0/0xd0 [19342.238415] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19342.241984] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19342.251709] RIP: 0033:0x7f48f0eeb93b [19342.256840] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19342.265524] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19342.269095] Workqueue: events delayed_work [ceph] [19342.277606] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19342.282649] [19342.282650] RIP: 0010:lock_acquire+0xf0/0x2b0 [19342.292027] ORIG_RAX: 0000000000000012 [19342.295597] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19342.303068] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19342.321805] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19342.326503] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19342.331720] [19342.333211] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19342.337562] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19342.341393] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19342.360129] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19342.367254] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19342.372472] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19342.379601] [19342.381088] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19342.433141] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19342.440265] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19342.448342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19342.454079] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19342.461205] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19342.468328] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19342.475450] Call Trace: [19342.477895] [19342.479908] ? watchdog_timer_fn+0x1b4/0x220 [19342.484172] ? __hrtimer_run_queues+0x1d3/0x360 [19342.488699] ? hrtimer_interrupt+0x104/0x240 [19342.492975] ? __local_bh_enable+0x50/0x60 [19342.497069] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19342.502294] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19342.507266] [19342.509365] [19342.511462] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19342.516769] ? lock_acquire+0xf0/0x2b0 [19342.520522] ? igrab+0x43/0x50 [19342.523570] ? lock_release+0xc5/0x260 [19342.527317] _raw_spin_lock+0x30/0x40 [19342.530980] ? igrab+0x1d/0x50 [19342.534031] igrab+0x1d/0x50 [19342.536911] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19342.542058] delayed_work+0x1d8/0x280 [ceph] [19342.546341] ? process_scheduled_works+0x22a/0x510 [19342.551129] process_scheduled_works+0x2a4/0x510 [19342.555745] worker_thread+0x188/0x340 [19342.559496] ? rescuer_thread+0x310/0x310 [19342.563507] kthread+0xf5/0x130 [19342.566644] ? kthread_complete_and_exit+0x20/0x20 [19342.571429] ret_from_fork+0x31/0x40 [19342.575000] ? kthread_complete_and_exit+0x20/0x20 [19342.579783] ret_from_fork_asm+0x11/0x20 [19342.583706] [19364.303392] libceph: mon2 (1)172.21.15.130:6790 session lost, hunting for new mon [19369.796082] watchdog: BUG: soft lockup - CPU#0 stuck for 6925s! [dbench:78683] [19369.803298] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19369.808081] watchdog: BUG: soft lockup - CPU#7 stuck for 15922s! [kworker/7:4:78808] [19369.858759] irq event stamp: 28304876 [19369.866487] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19369.870145] hardirqs last enabled at (28304875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19369.877105] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [19369.887262] hardirqs last disabled at (28304876): [] sysvec_apic_timer_interrupt+0xe/0x90 [19369.894212] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [19369.903936] softirqs last enabled at (28304754): [] __do_softirq+0x2be/0x371 [19369.910713] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [19369.919399] softirqs last disabled at (28304749): [] irq_exit_rcu+0xb0/0xd0 [19369.926270] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [19369.934782] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19369.940952] i2c_i801 iTCO_wdt [19369.949897] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19369.952948] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [19369.960419] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19369.966590] crc32_pclmul ipmi_ssif ghash_clmulni_intel [19369.972067] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19369.977284] mei_me mei lpc_ich [19369.996022] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19369.996023] sha512_ssse3 [19369.996024] [19369.999158] acpi_ipmi mfd_core [19370.004377] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19370.006993] wmi [19370.008485] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19370.011623] ipmi_si ipmi_devintf [19370.018760] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19370.020602] ipmi_msghandler [19370.027725] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19370.031037] ioatdma acpi_pad [19370.038159] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19370.041038] acpi_power_meter [19370.048161] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19370.051127] ixgbe igb [19370.058250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19370.061214] mdio crc32c_intel [19370.069291] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19370.071648] nvme ptp [19370.077386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19370.080437] nvme_core i2c_algo_bit [19370.087560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19370.089832] pps_core [19370.096955] Call Trace: [19370.096958] [19370.100441] dca fuse [19370.107567] ? watchdog_timer_fn+0x1b4/0x220 [19370.109836] [19370.109837] irq event stamp: 77729162 [19370.112280] ? __hrtimer_run_queues+0x1d3/0x360 [19370.114289] hardirqs last enabled at (77729161): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19370.116565] ? hrtimer_interrupt+0x104/0x240 [19370.120824] hardirqs last disabled at (77729162): [] sysvec_apic_timer_interrupt+0xe/0x90 [19370.122315] ? __local_bh_enable+0x50/0x60 [19370.125971] softirqs last enabled at (77729160): [] __do_softirq+0x2be/0x371 [19370.130500] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19370.140653] softirqs last disabled at (77729155): [] irq_exit_rcu+0xb0/0xd0 [19370.144919] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19370.154641] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19370.158733] [19370.167417] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19370.172635] [19370.172637] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19370.181147] Workqueue: events delayed_work [ceph] [19370.186107] ? queued_spin_lock_slowpath+0x16/0x2e0 [19370.195483] [19370.197581] do_raw_spin_lock+0xb5/0xc0 [19370.205047] RIP: 0010:lock_acquire+0xf0/0x2b0 [19370.207147] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19370.212450] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19370.217151] try_get_cap_refs+0x159/0x880 [ceph] [19370.222017] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19370.223515] __ceph_get_caps+0x4a1/0x760 [ceph] [19370.227341] [19370.231696] ? __file_remove_privs+0x89/0x110 [19370.236482] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19370.255221] ? wake_bit_function+0x70/0x70 [19370.259833] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19370.265052] ? ceph_write_iter+0x260/0x770 [ceph] [19370.269573] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19370.271065] ceph_write_iter+0x260/0x770 [ceph] [19370.275413] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19370.282544] ? lock_is_held_type+0xaa/0x120 [19370.286628] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19370.293768] vfs_write+0x324/0x4f0 [19370.298459] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19370.305605] ksys_pwrite64+0x65/0xa0 [19370.310125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19370.317251] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19370.321426] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19370.328552] do_syscall_64+0x38/0x80 [19370.331948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19370.340028] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19370.343597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19370.349334] RIP: 0033:0x7f48f0eeb93b [19370.354483] Call Trace: [19370.354485] [19370.361607] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19370.365179] ? watchdog_timer_fn+0x1b4/0x220 [19370.372302] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19370.377347] ? __hrtimer_run_queues+0x1d3/0x360 [19370.384485] ORIG_RAX: 0000000000000012 [19370.388060] ? hrtimer_interrupt+0x104/0x240 [19370.390501] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19370.392512] ? __local_bh_enable+0x50/0x60 [19370.411250] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19370.415517] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19370.420731] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19370.425256] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19370.429087] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19370.433351] [19370.440486] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19370.444573] [19370.444575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19370.451703] [19370.456919] ? lock_acquire+0xf0/0x2b0 [19370.498692] ? igrab+0x43/0x50 [19370.501765] ? lock_release+0xc5/0x260 [19370.505511] _raw_spin_lock+0x30/0x40 [19370.509169] ? igrab+0x1d/0x50 [19370.512219] igrab+0x1d/0x50 [19370.515099] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19370.520248] delayed_work+0x1d8/0x280 [ceph] [19370.524529] ? process_scheduled_works+0x22a/0x510 [19370.529317] process_scheduled_works+0x2a4/0x510 [19370.533935] worker_thread+0x188/0x340 [19370.537684] ? rescuer_thread+0x310/0x310 [19370.541693] kthread+0xf5/0x130 [19370.544833] ? kthread_complete_and_exit+0x20/0x20 [19370.549617] ret_from_fork+0x31/0x40 [19370.553197] ? kthread_complete_and_exit+0x20/0x20 [19370.557980] ret_from_fork_asm+0x11/0x20 [19370.561912] [19397.796596] watchdog: BUG: soft lockup - CPU#0 stuck for 6951s! [dbench:78683] [19397.803812] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19397.808596] watchdog: BUG: soft lockup - CPU#7 stuck for 15948s! [kworker/7:4:78808] [19397.859253] irq event stamp: 28362938 [19397.859255] hardirqs last enabled at (28362937): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19397.867001] Modules linked in: [19397.870660] hardirqs last disabled at (28362938): [] sysvec_apic_timer_interrupt+0xe/0x90 [19397.880816] ceph [19397.883868] softirqs last enabled at (28362870): [] __do_softirq+0x2be/0x371 [19397.893590] libceph [19397.895515] softirqs last disabled at (28362865): [] irq_exit_rcu+0xb0/0xd0 [19397.904199] dns_resolver fscache [19397.906384] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19397.914895] netfs veth [19397.918204] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19397.927149] nft_chain_nat [19397.929593] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19397.937063] xt_MASQUERADE nf_nat [19397.939769] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19397.945244] nf_conntrack nf_defrag_ipv6 [19397.948556] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19397.967294] nf_defrag_ipv4 nft_compat [19397.971212] [19397.971213] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19397.976428] nf_tables nfnetlink [19397.980173] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19397.981664] bridge stp [19397.988788] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19397.992012] llc [19397.999137] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19398.001580] xfs libcrc32c [19398.008704] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19398.010541] sunrpc intel_rapl_msr [19398.017667] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19398.020368] intel_rapl_common [19398.027493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19398.030892] sb_edac [19398.038970] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19398.042037] x86_pkg_temp_thermal coretemp [19398.047773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19398.049959] kvm_intel kvm [19398.057081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19398.061173] irqbypass i2c_i801 [19398.068297] Call Trace: [19398.068298] [19398.071001] iTCO_wdt iTCO_vendor_support [19398.078127] ? watchdog_timer_fn+0x1b4/0x220 [19398.081276] i2c_smbus crct10dif_pclmul [19398.083725] ? __hrtimer_run_queues+0x1d3/0x360 [19398.085734] joydev crc32_pclmul [19398.089742] ? hrtimer_interrupt+0x104/0x240 [19398.094002] ipmi_ssif ghash_clmulni_intel [19398.097833] ? __local_bh_enable+0x50/0x60 [19398.102357] mei_me mei [19398.105584] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19398.109846] lpc_ich sha512_ssse3 [19398.113937] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19398.118029] acpi_ipmi [19398.120472] [19398.125688] mfd_core wmi [19398.129000] [19398.129001] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19398.133958] ipmi_si [19398.136317] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19398.138412] ipmi_devintf ipmi_msghandler [19398.141030] do_raw_spin_lock+0xb5/0xc0 [19398.143126] ioatdma acpi_pad [19398.148430] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19398.150615] acpi_power_meter [19398.155488] try_get_cap_refs+0x159/0x880 [ceph] [19398.159489] ixgbe igb [19398.163323] __ceph_get_caps+0x4a1/0x760 [ceph] [19398.166283] mdio crc32c_intel [19398.171072] ? __file_remove_privs+0x89/0x110 [19398.174032] nvme ptp [19398.178642] ? wake_bit_function+0x70/0x70 [19398.181000] nvme_core i2c_algo_bit [19398.185526] ? ceph_write_iter+0x260/0x770 [ceph] [19398.188576] pps_core dca [19398.192925] ceph_write_iter+0x260/0x770 [ceph] [19398.195197] fuse [19398.199290] ? lock_is_held_type+0xaa/0x120 [19398.202770] irq event stamp: 77950116 [19398.207471] vfs_write+0x324/0x4f0 [19398.210086] hardirqs last enabled at (77950115): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19398.214614] ksys_pwrite64+0x65/0xa0 [19398.216534] hardirqs last disabled at (77950116): [] sysvec_apic_timer_interrupt+0xe/0x90 [19398.220713] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19398.224370] softirqs last enabled at (77950114): [] __do_softirq+0x2be/0x371 [19398.227768] do_syscall_64+0x38/0x80 [19398.237924] softirqs last disabled at (77950109): [] irq_exit_rcu+0xb0/0xd0 [19398.241496] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19398.251220] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19398.256350] RIP: 0033:0x7f48f0eeb93b [19398.265034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19398.268604] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19398.277117] Workqueue: events delayed_work [ceph] [19398.282159] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19398.291538] [19398.291539] RIP: 0010:lock_acquire+0xf0/0x2b0 [19398.295107] ORIG_RAX: 0000000000000012 [19398.302578] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19398.321316] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19398.326014] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19398.331232] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19398.332721] [19398.337073] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19398.340902] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19398.359640] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19398.366766] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19398.371999] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19398.379124] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19398.380621] [19398.387738] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19398.432677] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19398.439800] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19398.447878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19398.453616] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19398.460739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19398.467865] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19398.475000] Call Trace: [19398.477449] [19398.479462] ? watchdog_timer_fn+0x1b4/0x220 [19398.483725] ? __hrtimer_run_queues+0x1d3/0x360 [19398.488252] ? hrtimer_interrupt+0x104/0x240 [19398.492522] ? __local_bh_enable+0x50/0x60 [19398.496615] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19398.501839] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19398.506795] [19398.508891] [19398.511003] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19398.516316] ? lock_acquire+0xf0/0x2b0 [19398.520067] ? igrab+0x43/0x50 [19398.523125] ? lock_release+0xc5/0x260 [19398.526868] _raw_spin_lock+0x30/0x40 [19398.530524] ? igrab+0x1d/0x50 [19398.533575] igrab+0x1d/0x50 [19398.536454] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19398.541605] delayed_work+0x1d8/0x280 [ceph] [19398.545886] ? process_scheduled_works+0x22a/0x510 [19398.550676] process_scheduled_works+0x2a4/0x510 [19398.555292] worker_thread+0x188/0x340 [19398.559040] ? rescuer_thread+0x310/0x310 [19398.563042] kthread+0xf5/0x130 [19398.566181] ? kthread_complete_and_exit+0x20/0x20 [19398.570965] ret_from_fork+0x31/0x40 [19398.574534] ? kthread_complete_and_exit+0x20/0x20 [19398.579321] ret_from_fork_asm+0x11/0x20 [19398.583244] [19425.797110] watchdog: BUG: soft lockup - CPU#0 stuck for 6977s! [dbench:78683] [19425.804327] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19425.809110] watchdog: BUG: soft lockup - CPU#7 stuck for 15974s! [kworker/7:4:78808] [19425.859786] irq event stamp: 28421136 [19425.867516] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [19425.871174] hardirqs last enabled at (28421135): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19425.879346] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [19425.889503] hardirqs last disabled at (28421136): [] sysvec_apic_timer_interrupt+0xe/0x90 [19425.897505] nf_tables nfnetlink bridge stp llc xfs libcrc32c [19425.907227] softirqs last enabled at (28421068): [] __do_softirq+0x2be/0x371 [19425.912973] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19425.921658] softirqs last disabled at (28421063): [] irq_exit_rcu+0xb0/0xd0 [19425.929137] coretemp kvm_intel kvm irqbypass [19425.937649] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19425.937650] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [19425.950942] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19425.956593] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [19425.964064] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19425.964066] ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 [19425.975105] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19425.981103] acpi_ipmi mfd_core wmi [19425.999840] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19425.999841] ipmi_si [19426.003324] ipmi_devintf ipmi_msghandler [19426.008543] [19426.010726] ioatdma acpi_pad [19426.014731] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19426.016222] acpi_power_meter [19426.019184] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19426.026308] ixgbe igb [19426.029274] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19426.036398] mdio crc32c_intel [19426.038755] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19426.045879] nvme ptp [19426.048928] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19426.056054] nvme_core i2c_algo_bit [19426.058323] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19426.065449] pps_core [19426.068933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19426.077008] dca fuse [19426.079280] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19426.085019] [19426.085020] irq event stamp: 78170794 [19426.087287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19426.094413] hardirqs last enabled at (78170793): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19426.095904] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19426.099562] hardirqs last disabled at (78170794): [] sysvec_apic_timer_interrupt+0xe/0x90 [19426.106685] Call Trace: [19426.106688] [19426.116841] softirqs last enabled at (78170792): [] __do_softirq+0x2be/0x371 [19426.123968] ? watchdog_timer_fn+0x1b4/0x220 [19426.133690] softirqs last disabled at (78170787): [] irq_exit_rcu+0xb0/0xd0 [19426.136136] ? __hrtimer_run_queues+0x1d3/0x360 [19426.138145] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19426.146832] ? hrtimer_interrupt+0x104/0x240 [19426.151093] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19426.159603] ? __local_bh_enable+0x50/0x60 [19426.164130] Workqueue: events delayed_work [ceph] [19426.173514] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19426.177785] [19426.177786] RIP: 0010:_raw_spin_unlock+0x0/0x40 [19426.185260] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19426.189348] Code: 18 bb 01 00 00 00 45 31 c9 41 b8 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 [19426.194047] [19426.199265] RSP: 0018:ffffc90002117d98 EFLAGS: 00000202 [19426.200756] [19426.200757] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19426.205277] [19426.205278] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [19426.210241] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19426.228973] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [19426.231073] do_raw_spin_lock+0xb5/0xc0 [19426.236289] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [19426.238388] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19426.243691] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19426.245184] try_get_cap_refs+0x159/0x880 [ceph] [19426.252305] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19426.257182] __ceph_get_caps+0x4a1/0x760 [ceph] [19426.264300] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19426.268135] ? __file_remove_privs+0x89/0x110 [19426.275255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19426.280039] ? wake_bit_function+0x70/0x70 [19426.287162] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19426.291793] ? ceph_write_iter+0x260/0x770 [ceph] [19426.298914] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19426.303439] ceph_write_iter+0x260/0x770 [ceph] [19426.311517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19426.315873] ? lock_is_held_type+0xaa/0x120 [19426.321605] Call Trace: [19426.321607] [19426.325698] vfs_write+0x324/0x4f0 [19426.332821] ? watchdog_timer_fn+0x1b4/0x220 [19426.337522] ksys_pwrite64+0x65/0xa0 [19426.344643] ? __hrtimer_run_queues+0x1d3/0x360 [19426.349167] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19426.356293] ? hrtimer_interrupt+0x104/0x240 [19426.360469] do_syscall_64+0x38/0x80 [19426.362911] ? __local_bh_enable+0x50/0x60 [19426.364924] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19426.368321] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19426.372584] RIP: 0033:0x7f48f0eeb93b [19426.376153] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19426.380680] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19426.385810] [19426.390072] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19426.393644] [19426.393646] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19426.397734] ORIG_RAX: 0000000000000012 [19426.402797] ? _raw_spin_trylock_bh+0x70/0x70 [19426.408014] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19426.411586] igrab+0x43/0x50 [19426.416542] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19426.435282] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19426.437377] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19426.442596] delayed_work+0x1d8/0x280 [ceph] [19426.444691] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19426.449998] ? process_scheduled_works+0x22a/0x510 [19426.453827] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19426.458177] process_scheduled_works+0x2a4/0x510 [19426.465307] [19426.468182] worker_thread+0x188/0x340 [19426.521395] ? rescuer_thread+0x310/0x310 [19426.525404] kthread+0xf5/0x130 [19426.528543] ? kthread_complete_and_exit+0x20/0x20 [19426.533328] ret_from_fork+0x31/0x40 [19426.536908] ? kthread_complete_and_exit+0x20/0x20 [19426.541691] ret_from_fork_asm+0x11/0x20 [19426.545615] [19428.789531] rcu: INFO: rcu_preempt self-detected stall on CPU [19428.795292] rcu: 0-....: (7324166 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=3646883 [19428.805632] rcu: (t=7492046 jiffies g=1309609 q=275018 ncpus=8) [19428.811628] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19428.820573] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19428.828042] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19428.833520] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19428.852258] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19428.857474] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19428.864598] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19428.871722] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19428.878847] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19428.885973] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19428.893097] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19428.901174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19428.906912] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19428.914034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19428.921160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19428.928285] Call Trace: [19428.930729] [19428.932740] ? rcu_dump_cpu_stacks+0xd1/0x180 [19428.937098] ? rcu_sched_clock_irq+0x633/0x1390 [19428.941633] ? lock_is_held_type+0xaa/0x120 [19428.945820] ? lock_is_held_type+0xaa/0x120 [19428.950005] ? tick_sched_do_timer+0xa0/0xa0 [19428.954276] ? rcu_read_lock_sched_held+0x56/0x90 [19428.958984] ? tick_sched_do_timer+0xa0/0xa0 [19428.963256] ? update_process_times+0x5f/0x90 [19428.967615] ? tick_sched_handle.isra.16+0x1e/0x50 [19428.972407] ? tick_sched_timer+0x6f/0x80 [19428.976421] ? __hrtimer_run_queues+0x1d3/0x360 [19428.980960] ? hrtimer_interrupt+0x104/0x240 [19428.985224] ? __local_bh_enable+0x50/0x60 [19428.989320] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19428.994559] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19428.999516] [19429.001614] [19429.003713] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19429.009032] ? queued_spin_lock_slowpath+0x18/0x2e0 [19429.013906] do_raw_spin_lock+0xb5/0xc0 [19429.017743] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19429.022565] try_get_cap_refs+0x159/0x880 [ceph] [19429.027209] __ceph_get_caps+0x4a1/0x760 [ceph] [19429.031757] ? __file_remove_privs+0x89/0x110 [19429.036117] ? wake_bit_function+0x70/0x70 [19429.040211] ? ceph_write_iter+0x260/0x770 [ceph] [19429.044921] ceph_write_iter+0x260/0x770 [ceph] [19429.049470] ? lock_is_held_type+0xaa/0x120 [19429.053655] vfs_write+0x324/0x4f0 [19429.057065] ksys_pwrite64+0x65/0xa0 [19429.060643] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19429.065777] do_syscall_64+0x38/0x80 [19429.069355] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19429.074406] RIP: 0033:0x7f48f0eeb93b [19429.077978] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19429.096724] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [19429.104290] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19429.111413] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19429.118538] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19429.125678] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19429.132803] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19429.139938] [19429.142121] Sending NMI from CPU 0 to CPUs 7: [19429.146473] NMI backtrace for cpu 7 [19429.146475] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19429.146477] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19429.146477] Workqueue: events delayed_work [ceph] [19429.146492] RIP: 0010:do_raw_spin_lock+0x24/0xc0 [19429.146495] Code: 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 <48> 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba [19429.146496] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [19429.146497] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [19429.146498] RDX: ffffffffffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [19429.146498] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [19429.146499] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19429.146500] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19429.146500] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19429.146501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19429.146502] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19429.146503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19429.146504] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19429.146504] Call Trace: [19429.146505] [19429.146506] ? nmi_cpu_backtrace+0xad/0x120 [19429.146510] ? nmi_cpu_backtrace_handler+0x11/0x20 [19429.146511] ? nmi_handle+0xd4/0x260 [19429.146515] ? default_do_nmi+0x49/0xf0 [19429.146517] ? exc_nmi+0x11e/0x150 [19429.146518] ? end_repeat_nmi+0x16/0x67 [19429.146522] ? do_raw_spin_lock+0x24/0xc0 [19429.146524] ? do_raw_spin_lock+0x24/0xc0 [19429.146526] ? do_raw_spin_lock+0x24/0xc0 [19429.146527] [19429.146528] [19429.146528] igrab+0x1d/0x50 [19429.146530] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19429.146563] delayed_work+0x1d8/0x280 [ceph] [19429.146576] ? process_scheduled_works+0x22a/0x510 [19429.146579] process_scheduled_works+0x2a4/0x510 [19429.146583] worker_thread+0x188/0x340 [19429.146585] ? rescuer_thread+0x310/0x310 [19429.146587] kthread+0xf5/0x130 [19429.146590] ? kthread_complete_and_exit+0x20/0x20 [19429.146592] ret_from_fork+0x31/0x40 [19429.146595] ? kthread_complete_and_exit+0x20/0x20 [19429.146597] ret_from_fork_asm+0x11/0x20 [19429.146601] [19453.797624] watchdog: BUG: soft lockup - CPU#0 stuck for 7003s! [dbench:78683] [19453.804841] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19453.809624] watchdog: BUG: soft lockup - CPU#7 stuck for 16000s! [kworker/7:4:78808] [19453.860280] irq event stamp: 28478440 [19453.868028] Modules linked in: ceph libceph [19453.871686] hardirqs last enabled at (28478439): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19453.875864] dns_resolver fscache netfs [19453.886021] hardirqs last disabled at (28478440): [] sysvec_apic_timer_interrupt+0xe/0x90 [19453.889853] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [19453.899575] softirqs last enabled at (28478416): [] __do_softirq+0x2be/0x371 [19453.905661] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [19453.914345] softirqs last disabled at (28478411): [] irq_exit_rcu+0xb0/0xd0 [19453.920256] nfnetlink bridge [19453.928767] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19453.931730] stp llc [19453.940673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19453.942859] xfs libcrc32c sunrpc intel_rapl_msr [19453.950328] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19453.954940] intel_rapl_common sb_edac x86_pkg_temp_thermal [19453.960417] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19453.965982] coretemp kvm_intel [19453.984720] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19453.987855] kvm irqbypass [19453.993075] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19453.993075] i2c_i801 iTCO_wdt [19453.995777] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19454.002903] iTCO_vendor_support i2c_smbus [19454.005952] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19454.013076] crct10dif_pclmul joydev [19454.017167] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19454.024304] crc32_pclmul [19454.027880] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19454.035022] ipmi_ssif ghash_clmulni_intel [19454.037639] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19454.044763] mei_me mei [19454.048854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19454.056932] lpc_ich sha512_ssse3 [19454.059376] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19454.065112] acpi_ipmi mfd_core [19454.068425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19454.075547] wmi [19454.078684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19454.085808] ipmi_si ipmi_devintf [19454.087647] Call Trace: [19454.087648] [19454.094770] ipmi_msghandler ioatdma [19454.098082] ? watchdog_timer_fn+0x1b4/0x220 [19454.100526] acpi_pad acpi_power_meter [19454.102538] ? __hrtimer_run_queues+0x1d3/0x360 [19454.106106] ixgbe igb [19454.110374] ? hrtimer_interrupt+0x104/0x240 [19454.114114] mdio crc32c_intel [19454.118638] ? __local_bh_enable+0x50/0x60 [19454.120995] nvme ptp [19454.125280] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19454.128328] nvme_core i2c_algo_bit [19454.132420] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19454.134691] pps_core [19454.139906] [19454.143392] dca fuse [19454.148350] [19454.150620] [19454.150620] irq event stamp: 78390070 [19454.152717] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19454.154987] hardirqs last enabled at (78390069): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19454.157088] ? queued_spin_lock_slowpath+0x18/0x2e0 [19454.158574] hardirqs last disabled at (78390070): [] sysvec_apic_timer_interrupt+0xe/0x90 [19454.162236] do_raw_spin_lock+0xb5/0xc0 [19454.167537] softirqs last enabled at (78390068): [] __do_softirq+0x2be/0x371 [19454.177695] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19454.182566] softirqs last disabled at (78390063): [] irq_exit_rcu+0xb0/0xd0 [19454.192291] try_get_cap_refs+0x159/0x880 [ceph] [19454.196138] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19454.204828] __ceph_get_caps+0x4a1/0x760 [ceph] [19454.209607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19454.209608] Workqueue: events delayed_work [ceph] [19454.218122] ? __file_remove_privs+0x89/0x110 [19454.222728] [19454.232104] ? wake_bit_function+0x70/0x70 [19454.236629] RIP: 0010:lock_acquire+0xf0/0x2b0 [19454.244102] ? ceph_write_iter+0x260/0x770 [ceph] [19454.248797] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19454.253147] ceph_write_iter+0x260/0x770 [ceph] [19454.254639] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19454.258732] ? lock_is_held_type+0xaa/0x120 [19454.263079] [19454.267780] vfs_write+0x324/0x4f0 [19454.286514] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19454.291044] ksys_pwrite64+0x65/0xa0 [19454.296256] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19454.300434] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19454.301924] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19454.305324] do_syscall_64+0x38/0x80 [19454.312447] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19454.316027] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19454.323149] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19454.328281] RIP: 0033:0x7f48f0eeb93b [19454.335422] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19454.338994] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19454.346116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19454.351162] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19454.358285] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19454.361874] ORIG_RAX: 0000000000000012 [19454.369950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19454.388688] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19454.394426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19454.399644] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19454.406767] Call Trace: [19454.406769] [19454.410599] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19454.417724] ? watchdog_timer_fn+0x1b4/0x220 [19454.424847] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19454.431971] ? __hrtimer_run_queues+0x1d3/0x360 [19454.439094] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19454.441542] ? hrtimer_interrupt+0x104/0x240 [19454.443555] [19454.450673] ? __local_bh_enable+0x50/0x60 [19454.484252] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19454.489475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19454.494434] [19454.496531] [19454.498628] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19454.503935] ? lock_acquire+0xf0/0x2b0 [19454.507687] ? igrab+0x43/0x50 [19454.510736] ? lock_release+0xc5/0x260 [19454.514481] _raw_spin_lock+0x30/0x40 [19454.518138] ? igrab+0x1d/0x50 [19454.521188] igrab+0x1d/0x50 [19454.524066] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19454.529214] delayed_work+0x1d8/0x280 [ceph] [19454.533496] ? process_scheduled_works+0x22a/0x510 [19454.538286] process_scheduled_works+0x2a4/0x510 [19454.542903] worker_thread+0x188/0x340 [19454.546652] ? rescuer_thread+0x310/0x310 [19454.550653] kthread+0xf5/0x130 [19454.553792] ? kthread_complete_and_exit+0x20/0x20 [19454.558577] ret_from_fork+0x31/0x40 [19454.562147] ? kthread_complete_and_exit+0x20/0x20 [19454.566931] ret_from_fork_asm+0x11/0x20 [19454.570855] [19481.798137] watchdog: BUG: soft lockup - CPU#0 stuck for 7029s! [dbench:78683] [19481.805355] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19481.810137] watchdog: BUG: soft lockup - CPU#7 stuck for 16026s! [kworker/7:4:78808] [19481.860821] irq event stamp: 28536494 [19481.868551] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [19481.872210] hardirqs last enabled at (28536493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19481.880381] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [19481.890538] hardirqs last disabled at (28536494): [] sysvec_apic_timer_interrupt+0xe/0x90 [19481.897575] nft_compat nf_tables nfnetlink bridge stp llc xfs [19481.907300] softirqs last enabled at (28536456): [] __do_softirq+0x2be/0x371 [19481.913124] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19481.921815] softirqs last disabled at (28536451): [] irq_exit_rcu+0xb0/0xd0 [19481.930155] coretemp kvm_intel kvm irqbypass i2c_i801 [19481.938667] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19481.943812] iTCO_wdt iTCO_vendor_support i2c_smbus [19481.952760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19481.952761] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19481.957629] crct10dif_pclmul [19481.965101] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19481.970577] joydev crc32_pclmul [19481.973540] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19481.992278] ipmi_ssif ghash_clmulni_intel [19481.995503] [19482.000721] mei_me mei [19482.004819] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19482.006309] lpc_ich [19482.008754] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19482.015880] sha512_ssse3 acpi_ipmi [19482.018062] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19482.025187] mfd_core wmi [19482.028670] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19482.035796] ipmi_si ipmi_devintf [19482.038431] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19482.045553] ipmi_msghandler [19482.048864] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19482.055988] ioatdma acpi_pad [19482.058867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19482.066945] acpi_power_meter ixgbe [19482.069907] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19482.075645] igb mdio [19482.079130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19482.086253] crc32c_intel nvme [19482.088537] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19482.095656] ptp nvme_core [19482.098708] Call Trace: [19482.098710] [19482.105831] i2c_algo_bit pps_core [19482.108539] ? watchdog_timer_fn+0x1b4/0x220 [19482.110980] dca fuse [19482.112992] ? __hrtimer_run_queues+0x1d3/0x360 [19482.116389] [19482.116389] irq event stamp: 78610960 [19482.120655] ? hrtimer_interrupt+0x104/0x240 [19482.122924] hardirqs last enabled at (78610959): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19482.127446] ? __local_bh_enable+0x50/0x60 [19482.128948] hardirqs last disabled at (78610960): [] sysvec_apic_timer_interrupt+0xe/0x90 [19482.132605] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19482.136868] softirqs last enabled at (78610958): [] __do_softirq+0x2be/0x371 [19482.147026] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19482.151116] softirqs last disabled at (78610953): [] irq_exit_rcu+0xb0/0xd0 [19482.160840] [19482.166057] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19482.174741] [19482.179699] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19482.188211] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19482.190307] Workqueue: events delayed_work [ceph] [19482.199690] ? queued_spin_lock_slowpath+0x16/0x2e0 [19482.201782] [19482.201783] RIP: 0010:lock_acquire+0xf0/0x2b0 [19482.209256] do_raw_spin_lock+0xb5/0xc0 [19482.214556] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19482.219256] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19482.224124] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19482.225618] try_get_cap_refs+0x159/0x880 [ceph] [19482.229967] [19482.233820] __ceph_get_caps+0x4a1/0x760 [ceph] [19482.252553] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19482.257342] ? __file_remove_privs+0x89/0x110 [19482.262554] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19482.267166] ? wake_bit_function+0x70/0x70 [19482.268655] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19482.273181] ? ceph_write_iter+0x260/0x770 [ceph] [19482.280302] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19482.284655] ceph_write_iter+0x260/0x770 [ceph] [19482.291779] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19482.291780] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19482.295873] ? lock_is_held_type+0xaa/0x120 [19482.302994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19482.307692] vfs_write+0x324/0x4f0 [19482.314825] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19482.319352] ksys_pwrite64+0x65/0xa0 [19482.326473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19482.334551] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19482.338727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19482.344466] do_syscall_64+0x38/0x80 [19482.347860] Call Trace: [19482.347862] [19482.354988] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19482.358556] ? watchdog_timer_fn+0x1b4/0x220 [19482.365680] RIP: 0033:0x7f48f0eeb93b [19482.370823] ? __hrtimer_run_queues+0x1d3/0x360 [19482.377945] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19482.381537] ? hrtimer_interrupt+0x104/0x240 [19482.383976] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19482.385986] ? __local_bh_enable+0x50/0x60 [19482.391030] ORIG_RAX: 0000000000000012 [19482.395297] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19482.398866] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19482.403391] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19482.422126] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19482.426391] [19482.431609] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19482.435699] [19482.435701] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19482.439540] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19482.444762] ? lock_acquire+0xf0/0x2b0 [19482.451882] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19482.456839] ? igrab+0x43/0x50 [19482.463969] [19482.466060] ? lock_release+0xc5/0x260 [19482.507567] _raw_spin_lock+0x30/0x40 [19482.511225] ? igrab+0x1d/0x50 [19482.514273] igrab+0x1d/0x50 [19482.517153] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19482.522303] delayed_work+0x1d8/0x280 [ceph] [19482.526582] ? process_scheduled_works+0x22a/0x510 [19482.531374] process_scheduled_works+0x2a4/0x510 [19482.535990] worker_thread+0x188/0x340 [19482.539739] ? rescuer_thread+0x310/0x310 [19482.543750] kthread+0xf5/0x130 [19482.546889] ? kthread_complete_and_exit+0x20/0x20 [19482.551681] ret_from_fork+0x31/0x40 [19482.555261] ? kthread_complete_and_exit+0x20/0x20 [19482.560044] ret_from_fork_asm+0x11/0x20 [19482.563968] [19509.798650] watchdog: BUG: soft lockup - CPU#0 stuck for 7055s! [dbench:78683] [19509.805868] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19509.810650] watchdog: BUG: soft lockup - CPU#7 stuck for 16052s! [kworker/7:4:78808] [19509.861323] irq event stamp: 28594648 [19509.861324] hardirqs last enabled at (28594647): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19509.869057] Modules linked in: [19509.872713] hardirqs last disabled at (28594648): [] sysvec_apic_timer_interrupt+0xe/0x90 [19509.882871] ceph [19509.885923] softirqs last enabled at (28594610): [] __do_softirq+0x2be/0x371 [19509.895646] libceph [19509.897570] softirqs last disabled at (28594605): [] irq_exit_rcu+0xb0/0xd0 [19509.906253] dns_resolver fscache [19509.908439] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19509.916949] netfs veth [19509.920260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19509.929203] nft_chain_nat xt_MASQUERADE [19509.931649] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19509.939119] nf_nat nf_conntrack [19509.943045] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19509.948523] nf_defrag_ipv6 nf_defrag_ipv4 [19509.951748] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19509.970484] nft_compat nf_tables [19509.974574] [19509.974575] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19509.979791] nfnetlink bridge [19509.983102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19509.984593] stp llc [19509.991717] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19509.994681] xfs libcrc32c [19510.001805] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19510.003989] sunrpc intel_rapl_msr [19510.011115] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19510.013817] intel_rapl_common sb_edac [19510.020942] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19510.024340] x86_pkg_temp_thermal coretemp [19510.031463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19510.035207] kvm_intel kvm [19510.043286] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19510.047376] irqbypass i2c_i801 [19510.053112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19510.055817] iTCO_wdt iTCO_vendor_support [19510.062943] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19510.066079] i2c_smbus crct10dif_pclmul [19510.073203] Call Trace: [19510.073204] [19510.077206] joydev [19510.084333] ? watchdog_timer_fn+0x1b4/0x220 [19510.088163] crc32_pclmul [19510.090607] ? __hrtimer_run_queues+0x1d3/0x360 [19510.092618] ipmi_ssif ghash_clmulni_intel [19510.094718] ? hrtimer_interrupt+0x104/0x240 [19510.098979] mei_me mei [19510.101596] ? __local_bh_enable+0x50/0x60 [19510.106120] lpc_ich sha512_ssse3 [19510.110214] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19510.114474] acpi_ipmi mfd_core [19510.116918] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19510.121010] wmi ipmi_si [19510.124336] [19510.129556] ipmi_devintf [19510.132693] [19510.137649] ipmi_msghandler ioatdma [19510.140181] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19510.142278] acpi_pad acpi_power_meter [19510.144899] ? queued_spin_lock_slowpath+0x16/0x2e0 [19510.146992] ixgbe igb [19510.150565] do_raw_spin_lock+0xb5/0xc0 [19510.155868] mdio [19510.159612] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19510.164481] crc32c_intel nvme [19510.166842] try_get_cap_refs+0x159/0x880 [ceph] [19510.170671] ptp nvme_core [19510.172598] __ceph_get_caps+0x4a1/0x760 [ceph] [19510.177378] i2c_algo_bit [19510.180432] ? __file_remove_privs+0x89/0x110 [19510.185052] pps_core dca [19510.187752] ? wake_bit_function+0x70/0x70 [19510.192275] fuse [19510.194896] ? ceph_write_iter+0x260/0x770 [ceph] [19510.199246] irq event stamp: 78831810 [19510.201864] ceph_write_iter+0x260/0x770 [ceph] [19510.205952] hardirqs last enabled at (78831809): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19510.207882] ? lock_is_held_type+0xaa/0x120 [19510.212573] hardirqs last disabled at (78831810): [] sysvec_apic_timer_interrupt+0xe/0x90 [19510.216233] vfs_write+0x324/0x4f0 [19510.220756] softirqs last enabled at (78831808): [] __do_softirq+0x2be/0x371 [19510.230918] ksys_pwrite64+0x65/0xa0 [19510.235090] softirqs last disabled at (78831803): [] irq_exit_rcu+0xb0/0xd0 [19510.244815] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19510.248213] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19510.256898] do_syscall_64+0x38/0x80 [19510.260467] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19510.268978] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19510.274108] Workqueue: events delayed_work [ceph] [19510.283486] RIP: 0033:0x7f48f0eeb93b [19510.287057] [19510.287058] RIP: 0010:lock_acquire+0xf0/0x2b0 [19510.294527] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19510.299571] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19510.304270] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19510.307841] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19510.309333] ORIG_RAX: 0000000000000012 [19510.313682] [19510.313682] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19510.332420] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19510.351158] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19510.356374] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19510.361590] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19510.365422] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19510.366913] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19510.374052] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19510.381179] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19510.388304] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19510.395428] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19510.402556] [19510.409676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19510.454179] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19510.461303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19510.468428] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19510.475551] Call Trace: [19510.477996] [19510.480008] ? watchdog_timer_fn+0x1b4/0x220 [19510.484272] ? __hrtimer_run_queues+0x1d3/0x360 [19510.488798] ? hrtimer_interrupt+0x104/0x240 [19510.493067] ? __local_bh_enable+0x50/0x60 [19510.497160] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19510.502378] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19510.507337] [19510.509431] [19510.511528] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19510.516837] ? lock_acquire+0xf0/0x2b0 [19510.520588] ? igrab+0x43/0x50 [19510.523644] ? lock_release+0xc5/0x260 [19510.527391] _raw_spin_lock+0x30/0x40 [19510.531057] ? igrab+0x1d/0x50 [19510.534104] igrab+0x1d/0x50 [19510.536985] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19510.542135] delayed_work+0x1d8/0x280 [ceph] [19510.546414] ? process_scheduled_works+0x22a/0x510 [19510.551206] process_scheduled_works+0x2a4/0x510 [19510.555820] worker_thread+0x188/0x340 [19510.559568] ? rescuer_thread+0x310/0x310 [19510.563572] kthread+0xf5/0x130 [19510.566711] ? kthread_complete_and_exit+0x20/0x20 [19510.571495] ret_from_fork+0x31/0x40 [19510.575073] ? kthread_complete_and_exit+0x20/0x20 [19510.579858] ret_from_fork_asm+0x11/0x20 [19510.583780] [19537.799163] watchdog: BUG: soft lockup - CPU#0 stuck for 7081s! [dbench:78683] [19537.806379] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19537.811163] watchdog: BUG: soft lockup - CPU#7 stuck for 16079s! [kworker/7:4:78808] [19537.861840] irq event stamp: 28652744 [19537.869569] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19537.873226] hardirqs last enabled at (28652743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19537.880186] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [19537.890344] hardirqs last disabled at (28652744): [] sysvec_apic_timer_interrupt+0xe/0x90 [19537.890347] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [19537.907019] softirqs last enabled at (28652706): [] __do_softirq+0x2be/0x371 [19537.907023] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [19537.922488] softirqs last disabled at (28652701): [] irq_exit_rcu+0xb0/0xd0 [19537.922491] x86_pkg_temp_thermal [19537.922493] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19537.929351] coretemp kvm_intel [19537.937866] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19537.941175] kvm irqbypass [19537.950119] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19537.953257] i2c_i801 iTCO_wdt [19537.960727] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19537.963430] iTCO_vendor_support [19537.968908] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19537.971960] i2c_smbus [19537.990698] [19537.993919] crct10dif_pclmul joydev [19537.999137] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19538.001495] crc32_pclmul ipmi_ssif [19538.002985] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19538.006561] ghash_clmulni_intel [19538.013690] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19538.017173] mei_me [19538.024297] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19538.027522] mei lpc_ich [19538.034647] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19538.036743] sha512_ssse3 acpi_ipmi [19538.043868] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19538.046397] mfd_core wmi [19538.053522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19538.057008] ipmi_si [19538.065084] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19538.067703] ipmi_devintf [19538.073438] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19538.075624] ipmi_msghandler [19538.082748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19538.085364] ioatdma acpi_pad [19538.092489] Call Trace: [19538.092492] [19538.095367] acpi_power_meter [19538.102493] ? watchdog_timer_fn+0x1b4/0x220 [19538.105455] ixgbe [19538.107899] ? __hrtimer_run_queues+0x1d3/0x360 [19538.109909] igb mdio [19538.112876] ? hrtimer_interrupt+0x104/0x240 [19538.117137] crc32c_intel nvme [19538.119147] ? __local_bh_enable+0x50/0x60 [19538.123672] ptp nvme_core [19538.125945] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19538.130207] i2c_algo_bit pps_core [19538.133259] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19538.137348] dca [19538.140052] [19538.145270] fuse [19538.148667] [19538.148669] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19538.153625] [19538.155466] ? queued_spin_lock_slowpath+0x16/0x2e0 [19538.157563] irq event stamp: 79052684 [19538.159486] do_raw_spin_lock+0xb5/0xc0 [19538.161579] hardirqs last enabled at (79052683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19538.166887] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19538.168375] hardirqs last disabled at (79052684): [] sysvec_apic_timer_interrupt+0xe/0x90 [19538.173249] try_get_cap_refs+0x159/0x880 [ceph] [19538.176903] softirqs last enabled at (79052682): [] __do_softirq+0x2be/0x371 [19538.180739] __ceph_get_caps+0x4a1/0x760 [ceph] [19538.190892] softirqs last disabled at (79052677): [] irq_exit_rcu+0xb0/0xd0 [19538.195680] ? __file_remove_privs+0x89/0x110 [19538.205401] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19538.210012] ? wake_bit_function+0x70/0x70 [19538.218695] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19538.223222] ? ceph_write_iter+0x260/0x770 [ceph] [19538.231731] Workqueue: events delayed_work [ceph] [19538.236080] ceph_write_iter+0x260/0x770 [ceph] [19538.245457] [19538.249570] ? lock_is_held_type+0xaa/0x120 [19538.257037] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [19538.261738] vfs_write+0x324/0x4f0 [19538.266431] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [19538.270960] ksys_pwrite64+0x65/0xa0 [19538.272446] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [19538.276626] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19538.281235] [19538.281236] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [19538.284633] do_syscall_64+0x38/0x80 [19538.303371] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [19538.306942] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19538.312159] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [19538.317291] RIP: 0033:0x7f48f0eeb93b [19538.318780] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19538.325904] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19538.329474] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19538.336599] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19538.341642] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19538.348767] ORIG_RAX: 0000000000000012 [19538.352336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19538.359461] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19538.378198] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19538.385322] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19538.390542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19538.398636] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19538.402466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19538.408203] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19538.415328] Call Trace: [19538.415329] [19538.422452] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19538.429578] ? watchdog_timer_fn+0x1b4/0x220 [19538.436706] [19538.443825] ? __hrtimer_run_queues+0x1d3/0x360 [19538.480662] ? hrtimer_interrupt+0x104/0x240 [19538.484932] ? __local_bh_enable+0x50/0x60 [19538.489024] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19538.494249] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19538.499206] [19538.501303] [19538.503401] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19538.508709] ? do_raw_spin_lock+0x42/0xc0 [19538.512718] igrab+0x1d/0x50 [19538.515598] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19538.520746] delayed_work+0x1d8/0x280 [ceph] [19538.525026] ? process_scheduled_works+0x22a/0x510 [19538.529817] process_scheduled_works+0x2a4/0x510 [19538.534433] worker_thread+0x188/0x340 [19538.538182] ? rescuer_thread+0x310/0x310 [19538.542184] kthread+0xf5/0x130 [19538.545322] ? kthread_complete_and_exit+0x20/0x20 [19538.550106] ret_from_fork+0x31/0x40 [19538.553677] ? kthread_complete_and_exit+0x20/0x20 [19538.558461] ret_from_fork_asm+0x11/0x20 [19538.562385] [19565.799676] watchdog: BUG: soft lockup - CPU#0 stuck for 7108s! [dbench:78683] [19565.806894] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19565.811676] watchdog: BUG: soft lockup - CPU#7 stuck for 16105s! [kworker/7:4:78808] [19565.862335] irq event stamp: 28710860 [19565.862336] hardirqs last enabled at (28710859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19565.870083] Modules linked in: [19565.873740] hardirqs last disabled at (28710860): [] sysvec_apic_timer_interrupt+0xe/0x90 [19565.883896] ceph [19565.886948] softirqs last enabled at (28710850): [] __do_softirq+0x2be/0x371 [19565.896672] libceph [19565.898595] softirqs last disabled at (28710845): [] irq_exit_rcu+0xb0/0xd0 [19565.907280] dns_resolver [19565.909464] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19565.917976] fscache netfs [19565.920591] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19565.929535] veth nft_chain_nat [19565.932242] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19565.939713] xt_MASQUERADE nf_nat [19565.942850] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19565.948327] nf_conntrack nf_defrag_ipv6 [19565.951638] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19565.970375] nf_defrag_ipv4 nft_compat [19565.974292] [19565.974293] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19565.979510] nf_tables nfnetlink [19565.983254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19565.984745] bridge [19565.991869] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19565.995092] stp llc [19566.002218] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19566.004315] xfs libcrc32c [19566.011439] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19566.013621] sunrpc intel_rapl_msr [19566.020746] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19566.023451] intel_rapl_common sb_edac [19566.030575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19566.033972] x86_pkg_temp_thermal [19566.042048] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19566.045810] coretemp kvm_intel [19566.051548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19566.054859] kvm irqbypass [19566.061984] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19566.065119] i2c_i801 iTCO_wdt [19566.072244] Call Trace: [19566.072245] [19566.074949] iTCO_vendor_support i2c_smbus [19566.082075] ? watchdog_timer_fn+0x1b4/0x220 [19566.085125] crct10dif_pclmul joydev [19566.087570] ? __hrtimer_run_queues+0x1d3/0x360 [19566.089579] crc32_pclmul ipmi_ssif [19566.093671] ? hrtimer_interrupt+0x104/0x240 [19566.097932] ghash_clmulni_intel mei_me [19566.101504] ? __local_bh_enable+0x50/0x60 [19566.106027] mei lpc_ich [19566.109515] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19566.113775] sha512_ssse3 acpi_ipmi [19566.117608] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19566.121698] mfd_core wmi [19566.124230] [19566.129446] ipmi_si ipmi_devintf [19566.132928] [19566.137886] ipmi_msghandler [19566.140506] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19566.142602] ioatdma acpi_pad [19566.145915] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19566.148009] acpi_power_meter ixgbe [19566.150890] do_raw_spin_lock+0xb5/0xc0 [19566.156190] igb mdio [19566.159156] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19566.164027] crc32c_intel nvme [19566.167513] try_get_cap_refs+0x159/0x880 [ceph] [19566.171340] ptp nvme_core [19566.173617] __ceph_get_caps+0x4a1/0x760 [ceph] [19566.178396] i2c_algo_bit pps_core [19566.181451] ? __file_remove_privs+0x89/0x110 [19566.186076] dca fuse [19566.188780] ? wake_bit_function+0x70/0x70 [19566.193302] irq event stamp: 79273880 [19566.196703] ? ceph_write_iter+0x260/0x770 [ceph] [19566.201052] hardirqs last enabled at (79273879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19566.203339] ceph_write_iter+0x260/0x770 [ceph] [19566.207448] hardirqs last disabled at (79273880): [] sysvec_apic_timer_interrupt+0xe/0x90 [19566.211108] ? lock_is_held_type+0xaa/0x120 [19566.215803] softirqs last enabled at (79273878): [] __do_softirq+0x2be/0x371 [19566.225962] vfs_write+0x324/0x4f0 [19566.230483] softirqs last disabled at (79273873): [] irq_exit_rcu+0xb0/0xd0 [19566.240213] ksys_pwrite64+0x65/0xa0 [19566.244386] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19566.253078] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19566.256474] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19566.264987] do_syscall_64+0x38/0x80 [19566.268557] Workqueue: events delayed_work [ceph] [19566.277934] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19566.283077] [19566.290546] RIP: 0033:0x7f48f0eeb93b [19566.294115] RIP: 0010:lock_release+0x3c/0x260 [19566.298813] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19566.303858] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [19566.305359] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19566.308928] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [19566.313278] ORIG_RAX: 0000000000000012 [19566.332014] [19566.332015] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [19566.350754] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19566.355970] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [19566.361187] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19566.365017] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [19566.366510] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19566.373634] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [19566.380759] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19566.387881] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19566.395005] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19566.402130] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19566.409259] [19566.416378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19566.453750] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19566.460873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19566.467998] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19566.475122] Call Trace: [19566.477568] [19566.479578] ? watchdog_timer_fn+0x1b4/0x220 [19566.483841] ? __hrtimer_run_queues+0x1d3/0x360 [19566.488370] ? hrtimer_interrupt+0x104/0x240 [19566.492639] ? __local_bh_enable+0x50/0x60 [19566.496732] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19566.501954] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19566.506912] [19566.509009] [19566.511108] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19566.516412] ? igrab+0x43/0x50 [19566.519465] ? igrab+0x43/0x50 [19566.522513] ? lock_release+0x3c/0x260 [19566.526260] ? lock_release+0xc5/0x260 [19566.530004] _raw_spin_unlock+0x1b/0x40 [19566.533842] igrab+0x43/0x50 [19566.536720] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19566.541868] delayed_work+0x1d8/0x280 [ceph] [19566.546150] ? process_scheduled_works+0x22a/0x510 [19566.550939] process_scheduled_works+0x2a4/0x510 [19566.555555] worker_thread+0x188/0x340 [19566.559304] ? rescuer_thread+0x310/0x310 [19566.563307] kthread+0xf5/0x130 [19566.566444] ? kthread_complete_and_exit+0x20/0x20 [19566.571231] ret_from_fork+0x31/0x40 [19566.574809] ? kthread_complete_and_exit+0x20/0x20 [19566.579592] ret_from_fork_asm+0x11/0x20 [19566.583514] [19593.800189] watchdog: BUG: soft lockup - CPU#0 stuck for 7134s! [dbench:78683] [19593.807407] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19593.812189] watchdog: BUG: soft lockup - CPU#7 stuck for 16131s! [kworker/7:4:78808] [19593.862872] irq event stamp: 28769044 [19593.862873] hardirqs last enabled at (28769043): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19593.870606] Modules linked in: [19593.874262] hardirqs last disabled at (28769044): [] sysvec_apic_timer_interrupt+0xe/0x90 [19593.884418] ceph [19593.887469] softirqs last enabled at (28769006): [] __do_softirq+0x2be/0x371 [19593.897194] libceph [19593.899118] softirqs last disabled at (28769001): [] irq_exit_rcu+0xb0/0xd0 [19593.907802] dns_resolver fscache [19593.909988] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19593.918497] netfs veth [19593.921810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19593.930751] nft_chain_nat xt_MASQUERADE [19593.933197] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19593.940666] nf_nat [19593.944585] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19593.950063] nf_conntrack nf_defrag_ipv6 [19593.952158] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19593.970897] nf_defrag_ipv4 nft_compat [19593.974816] [19593.980032] nf_tables nfnetlink [19593.983776] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19593.985265] bridge stp [19593.988491] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19593.995615] llc [19593.998059] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19594.005183] xfs libcrc32c [19594.007020] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19594.014145] sunrpc intel_rapl_msr [19594.016848] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19594.023973] intel_rapl_common [19594.027369] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19594.034495] sb_edac [19594.037545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19594.045622] x86_pkg_temp_thermal [19594.047807] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19594.053542] coretemp kvm_intel [19594.056854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19594.063978] kvm irqbypass [19594.067117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19594.074239] i2c_i801 [19594.076944] Call Trace: [19594.076946] [19594.084069] iTCO_wdt [19594.086341] ? watchdog_timer_fn+0x1b4/0x220 [19594.088782] iTCO_vendor_support i2c_smbus [19594.090796] ? __hrtimer_run_queues+0x1d3/0x360 [19594.093064] crct10dif_pclmul joydev [19594.097331] ? hrtimer_interrupt+0x104/0x240 [19594.101419] crc32_pclmul ipmi_ssif [19594.105942] ? __local_bh_enable+0x50/0x60 [19594.109513] ghash_clmulni_intel mei_me [19594.113782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19594.117262] mei lpc_ich [19594.121353] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19594.125184] sha512_ssse3 acpi_ipmi [19594.130400] [19594.132932] mfd_core [19594.137889] [19594.137891] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19594.141374] wmi ipmi_si [19594.143475] ? queued_spin_lock_slowpath+0x16/0x2e0 [19594.145742] ipmi_devintf [19594.147841] do_raw_spin_lock+0xb5/0xc0 [19594.153144] ipmi_msghandler [19594.155675] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19594.160545] ioatdma acpi_pad [19594.163166] try_get_cap_refs+0x159/0x880 [ceph] [19594.166991] acpi_power_meter ixgbe [19594.169883] __ceph_get_caps+0x4a1/0x760 [ceph] [19594.174663] igb mdio [19594.177632] ? __file_remove_privs+0x89/0x110 [19594.182238] crc32c_intel [19594.185721] ? wake_bit_function+0x70/0x70 [19594.190246] nvme [19594.192518] ? ceph_write_iter+0x260/0x770 [ceph] [19594.196883] ptp nvme_core [19594.199503] ceph_write_iter+0x260/0x770 [ceph] [19594.203592] i2c_algo_bit [19594.205521] ? lock_is_held_type+0xaa/0x120 [19594.210215] pps_core [19594.212921] vfs_write+0x324/0x4f0 [19594.217443] dca fuse [19594.220065] ksys_pwrite64+0x65/0xa0 [19594.224237] [19594.224237] irq event stamp: 79495144 [19594.226508] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19594.229904] hardirqs last enabled at (79495143): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19594.232176] do_syscall_64+0x38/0x80 [19594.235746] hardirqs last disabled at (79495144): [] sysvec_apic_timer_interrupt+0xe/0x90 [19594.237239] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19594.240895] softirqs last enabled at (79495142): [] __do_softirq+0x2be/0x371 [19594.246028] RIP: 0033:0x7f48f0eeb93b [19594.256183] softirqs last disabled at (79495137): [] irq_exit_rcu+0xb0/0xd0 [19594.259755] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19594.269477] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19594.274521] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19594.283206] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19594.286777] ORIG_RAX: 0000000000000012 [19594.295288] Workqueue: events delayed_work [ceph] [19594.314026] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19594.323403] [19594.328620] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19594.336091] RIP: 0010:lock_acquire+0xf0/0x2b0 [19594.339920] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19594.344618] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19594.351744] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19594.353233] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19594.360357] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19594.364709] [19594.364710] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19594.371837] [19594.390588] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19594.427977] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19594.435100] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19594.442226] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19594.449349] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19594.457426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19594.463165] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19594.470287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19594.477413] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19594.484538] Call Trace: [19594.486983] [19594.488993] ? watchdog_timer_fn+0x1b4/0x220 [19594.493257] ? __hrtimer_run_queues+0x1d3/0x360 [19594.497783] ? hrtimer_interrupt+0x104/0x240 [19594.502051] ? __local_bh_enable+0x50/0x60 [19594.506146] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19594.511371] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19594.516326] [19594.518424] [19594.520522] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19594.525831] ? lock_acquire+0xf0/0x2b0 [19594.529596] ? igrab+0x43/0x50 [19594.532647] ? lock_release+0xc5/0x260 [19594.536392] _raw_spin_lock+0x30/0x40 [19594.540057] ? igrab+0x1d/0x50 [19594.543106] igrab+0x1d/0x50 [19594.545987] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19594.551136] delayed_work+0x1d8/0x280 [ceph] [19594.555418] ? process_scheduled_works+0x22a/0x510 [19594.560206] process_scheduled_works+0x2a4/0x510 [19594.564822] worker_thread+0x188/0x340 [19594.568573] ? rescuer_thread+0x310/0x310 [19594.572601] kthread+0xf5/0x130 [19594.575747] ? kthread_complete_and_exit+0x20/0x20 [19594.580531] ret_from_fork+0x31/0x40 [19594.584112] ? kthread_complete_and_exit+0x20/0x20 [19594.588895] ret_from_fork_asm+0x11/0x20 [19594.592826] [19621.800701] watchdog: BUG: soft lockup - CPU#0 stuck for 7160s! [dbench:78683] [19621.807921] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19621.812702] watchdog: BUG: soft lockup - CPU#7 stuck for 16157s! [kworker/7:4:78808] [19621.863376] irq event stamp: 28827198 [19621.863377] hardirqs last enabled at (28827197): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19621.871109] Modules linked in: [19621.874767] hardirqs last disabled at (28827198): [] sysvec_apic_timer_interrupt+0xe/0x90 [19621.884924] ceph [19621.887975] softirqs last enabled at (28827160): [] __do_softirq+0x2be/0x371 [19621.897700] libceph [19621.899623] softirqs last disabled at (28827155): [] irq_exit_rcu+0xb0/0xd0 [19621.908308] dns_resolver [19621.910490] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19621.919002] fscache netfs [19621.921618] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19621.921620] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19621.930564] veth [19621.933268] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19621.940738] nft_chain_nat xt_MASQUERADE [19621.946217] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19621.948139] nf_nat nf_conntrack [19621.966878] [19621.966879] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19621.970796] nf_defrag_ipv6 nf_defrag_ipv4 [19621.976014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19621.979236] nft_compat nf_tables [19621.980727] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19621.987851] nfnetlink [19621.991943] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19621.999066] bridge [19622.002382] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19622.009509] stp [19622.011866] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19622.018990] llc xfs [19622.021099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19622.028221] libcrc32c sunrpc [19622.030058] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19622.038137] intel_rapl_msr [19622.040321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19622.046058] intel_rapl_common [19622.049021] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19622.056147] sb_edac x86_pkg_temp_thermal [19622.058936] Call Trace: [19622.066062] coretemp kvm_intel [19622.069113] [19622.076236] kvm irqbypass [19622.080242] ? watchdog_timer_fn+0x1b4/0x220 [19622.082684] i2c_i801 iTCO_wdt [19622.085821] ? __hrtimer_run_queues+0x1d3/0x360 [19622.087831] iTCO_vendor_support [19622.090539] ? hrtimer_interrupt+0x104/0x240 [19622.094800] i2c_smbus crct10dif_pclmul [19622.097850] ? __local_bh_enable+0x50/0x60 [19622.102382] joydev crc32_pclmul [19622.105610] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19622.109870] ipmi_ssif [19622.113702] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19622.117793] ghash_clmulni_intel mei_me [19622.121018] [19622.126233] mei lpc_ich [19622.128591] [19622.128593] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19622.133549] sha512_ssse3 [19622.137399] ? queued_spin_lock_slowpath+0x16/0x2e0 [19622.139494] acpi_ipmi [19622.142026] do_raw_spin_lock+0xb5/0xc0 [19622.144123] mfd_core [19622.149429] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19622.152044] wmi ipmi_si [19622.156919] try_get_cap_refs+0x159/0x880 [ceph] [19622.159273] ipmi_devintf [19622.163109] __ceph_get_caps+0x4a1/0x760 [ceph] [19622.165382] ipmi_msghandler ioatdma [19622.170170] ? __file_remove_privs+0x89/0x110 [19622.172697] acpi_pad acpi_power_meter [19622.177309] ? wake_bit_function+0x70/0x70 [19622.179927] ixgbe igb [19622.184453] ? ceph_write_iter+0x260/0x770 [ceph] [19622.188019] mdio crc32c_intel [19622.192371] ceph_write_iter+0x260/0x770 [ceph] [19622.196132] nvme [19622.200226] ? lock_is_held_type+0xaa/0x120 [19622.202580] ptp nvme_core [19622.207280] vfs_write+0x324/0x4f0 [19622.210329] i2c_algo_bit pps_core [19622.214857] ksys_pwrite64+0x65/0xa0 [19622.216777] dca fuse [19622.220954] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19622.223658] [19622.227058] do_syscall_64+0x38/0x80 [19622.230454] irq event stamp: 79715822 [19622.234026] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19622.236295] hardirqs last enabled at (79715821): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19622.241424] RIP: 0033:0x7f48f0eeb93b [19622.242915] hardirqs last disabled at (79715822): [] sysvec_apic_timer_interrupt+0xe/0x90 [19622.246487] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19622.250143] softirqs last enabled at (79715820): [] __do_softirq+0x2be/0x371 [19622.255187] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19622.265347] softirqs last disabled at (79715815): [] irq_exit_rcu+0xb0/0xd0 [19622.268918] ORIG_RAX: 0000000000000012 [19622.278642] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19622.297378] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19622.306081] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19622.311314] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19622.319825] Workqueue: events delayed_work [ceph] [19622.323657] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19622.333034] [19622.340157] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19622.347629] RIP: 0010:lock_release+0x78/0x260 [19622.354754] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19622.359450] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [19622.366579] [19622.368065] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [19622.412804] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [19622.419927] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [19622.427052] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [19622.434177] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [19622.441300] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19622.448424] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19622.456501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19622.462240] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19622.469362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19622.476505] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19622.483628] Call Trace: [19622.486073] [19622.488084] ? watchdog_timer_fn+0x1b4/0x220 [19622.492367] ? __hrtimer_run_queues+0x1d3/0x360 [19622.496910] ? hrtimer_interrupt+0x104/0x240 [19622.501180] ? __local_bh_enable+0x50/0x60 [19622.505272] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19622.510497] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19622.515455] [19622.517551] [19622.519648] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19622.524954] ? igrab+0x43/0x50 [19622.528007] ? lock_release+0x78/0x260 [19622.531759] _raw_spin_unlock+0x1b/0x40 [19622.535596] igrab+0x43/0x50 [19622.538472] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19622.543623] delayed_work+0x1d8/0x280 [ceph] [19622.547905] ? process_scheduled_works+0x22a/0x510 [19622.552696] process_scheduled_works+0x2a4/0x510 [19622.557309] worker_thread+0x188/0x340 [19622.561060] ? rescuer_thread+0x310/0x310 [19622.565064] kthread+0xf5/0x130 [19622.568200] ? kthread_complete_and_exit+0x20/0x20 [19622.572986] ret_from_fork+0x31/0x40 [19622.576563] ? kthread_complete_and_exit+0x20/0x20 [19622.581349] ret_from_fork_asm+0x11/0x20 [19622.585272] [19624.153111] rcu: INFO: rcu_preempt self-detected stall on CPU [19624.158869] rcu: 0-....: (7514778 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=3742704 [19624.169205] rcu: (t=7687406 jiffies g=1309609 q=279250 ncpus=8) [19624.175204] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19624.184147] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19624.191619] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19624.197095] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19624.215832] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19624.221049] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19624.228175] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19624.235297] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19624.242423] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19624.249546] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19624.256671] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19624.264748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19624.270485] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19624.277610] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19624.284735] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19624.291859] Call Trace: [19624.294302] [19624.296313] ? rcu_dump_cpu_stacks+0xd1/0x180 [19624.300674] ? rcu_sched_clock_irq+0x633/0x1390 [19624.305198] ? lock_is_held_type+0xaa/0x120 [19624.309375] ? lock_is_held_type+0xaa/0x120 [19624.313552] ? tick_sched_do_timer+0xa0/0xa0 [19624.317823] ? rcu_read_lock_sched_held+0x56/0x90 [19624.322523] ? tick_sched_do_timer+0xa0/0xa0 [19624.326787] ? update_process_times+0x5f/0x90 [19624.331142] ? tick_sched_handle.isra.16+0x1e/0x50 [19624.335929] ? tick_sched_timer+0x6f/0x80 [19624.339935] ? __hrtimer_run_queues+0x1d3/0x360 [19624.344461] ? hrtimer_interrupt+0x104/0x240 [19624.348731] ? __local_bh_enable+0x50/0x60 [19624.352824] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19624.358047] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19624.363004] [19624.365102] [19624.367199] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19624.372507] ? queued_spin_lock_slowpath+0x16/0x2e0 [19624.377384] do_raw_spin_lock+0xb5/0xc0 [19624.381215] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19624.386017] try_get_cap_refs+0x159/0x880 [ceph] [19624.390648] __ceph_get_caps+0x4a1/0x760 [ceph] [19624.395188] ? __file_remove_privs+0x89/0x110 [19624.399544] ? wake_bit_function+0x70/0x70 [19624.403638] ? ceph_write_iter+0x260/0x770 [ceph] [19624.408349] ceph_write_iter+0x260/0x770 [ceph] [19624.412886] ? lock_is_held_type+0xaa/0x120 [19624.417070] vfs_write+0x324/0x4f0 [19624.420479] ksys_pwrite64+0x65/0xa0 [19624.424054] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19624.429187] do_syscall_64+0x38/0x80 [19624.432765] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19624.437806] RIP: 0033:0x7f48f0eeb93b [19624.441379] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19624.460117] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [19624.467698] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19624.474824] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19624.481947] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19624.489071] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19624.496196] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19624.503325] [19624.505514] Sending NMI from CPU 0 to CPUs 7: [19624.509866] NMI backtrace for cpu 7 [19624.509868] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19624.509870] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19624.509871] Workqueue: events delayed_work [ceph] [19624.509891] RIP: 0010:lock_release+0x16/0x260 [19624.509894] Code: 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 <65> 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 [19624.509896] RSP: 0018:ffffc90002117d48 EFLAGS: 00000296 [19624.509897] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [19624.509899] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [19624.509900] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [19624.509901] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [19624.509902] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19624.509904] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19624.509905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19624.509907] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19624.509908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19624.509909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19624.509910] Call Trace: [19624.509911] [19624.509913] ? nmi_cpu_backtrace+0xad/0x120 [19624.509918] ? nmi_cpu_backtrace_handler+0x11/0x20 [19624.509920] ? nmi_handle+0xd4/0x260 [19624.509926] ? default_do_nmi+0x49/0xf0 [19624.509928] ? exc_nmi+0x11e/0x150 [19624.509931] ? end_repeat_nmi+0x16/0x67 [19624.509934] ? igrab+0x43/0x50 [19624.509938] ? igrab+0x43/0x50 [19624.509940] ? lock_release+0x16/0x260 [19624.509943] ? lock_release+0x16/0x260 [19624.509946] ? lock_release+0x16/0x260 [19624.509948] [19624.509949] [19624.509951] _raw_spin_unlock+0x1b/0x40 [19624.509954] igrab+0x43/0x50 [19624.509956] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19624.509976] delayed_work+0x1d8/0x280 [ceph] [19624.509995] ? process_scheduled_works+0x22a/0x510 [19624.509998] process_scheduled_works+0x2a4/0x510 [19624.510005] worker_thread+0x188/0x340 [19624.510009] ? rescuer_thread+0x310/0x310 [19624.510012] kthread+0xf5/0x130 [19624.510014] ? kthread_complete_and_exit+0x20/0x20 [19624.510018] ret_from_fork+0x31/0x40 [19624.510021] ? kthread_complete_and_exit+0x20/0x20 [19624.510024] ret_from_fork_asm+0x11/0x20 [19624.510031] [19649.801215] watchdog: BUG: soft lockup - CPU#0 stuck for 7186s! [dbench:78683] [19649.808434] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19649.813215] watchdog: BUG: soft lockup - CPU#7 stuck for 16183s! [kworker/7:4:78808] [19649.863899] irq event stamp: 28884700 [19649.871631] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19649.875289] hardirqs last enabled at (28884699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19649.882247] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [19649.892406] hardirqs last disabled at (28884700): [] sysvec_apic_timer_interrupt+0xe/0x90 [19649.900656] nft_compat nf_tables nfnetlink bridge stp llc xfs [19649.910380] softirqs last enabled at (28884668): [] __do_softirq+0x2be/0x371 [19649.916204] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19649.924893] softirqs last disabled at (28884663): [] irq_exit_rcu+0xb0/0xd0 [19649.933237] coretemp kvm_intel [19649.941745] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19649.944899] kvm [19649.953844] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19649.953846] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19649.955683] irqbypass i2c_i801 [19649.963154] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19649.968631] iTCO_wdt [19649.971769] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19649.990505] iTCO_vendor_support i2c_smbus [19649.992778] [19649.997993] crct10dif_pclmul joydev [19650.002084] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19650.003574] crc32_pclmul ipmi_ssif [19650.007145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19650.014270] ghash_clmulni_intel mei_me [19650.017754] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19650.024878] mei [19650.028726] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19650.035852] lpc_ich [19650.037688] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19650.044813] sha512_ssse3 acpi_ipmi [19650.046997] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19650.054121] mfd_core [19650.057615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19650.065691] wmi ipmi_si [19650.067960] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19650.073699] ipmi_devintf ipmi_msghandler [19650.076230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19650.083353] ioatdma acpi_pad [19650.087357] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19650.094483] acpi_power_meter ixgbe [19650.097444] Call Trace: [19650.097447] [19650.104570] igb mdio [19650.108055] ? watchdog_timer_fn+0x1b4/0x220 [19650.110497] crc32c_intel nvme [19650.112508] ? __hrtimer_run_queues+0x1d3/0x360 [19650.114779] ptp nvme_core [19650.119047] ? hrtimer_interrupt+0x104/0x240 [19650.122095] i2c_algo_bit pps_core [19650.126619] ? __local_bh_enable+0x50/0x60 [19650.129321] dca fuse [19650.133590] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19650.136984] [19650.136985] irq event stamp: 79934726 [19650.141076] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19650.143346] hardirqs last enabled at (79934725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19650.148562] [19650.150052] hardirqs last disabled at (79934726): [] sysvec_apic_timer_interrupt+0xe/0x90 [19650.153710] [19650.153711] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19650.158668] softirqs last enabled at (79934724): [] __do_softirq+0x2be/0x371 [19650.168830] ? queued_spin_lock_slowpath+0x18/0x2e0 [19650.170924] softirqs last disabled at (79934719): [] irq_exit_rcu+0xb0/0xd0 [19650.180649] do_raw_spin_lock+0xb5/0xc0 [19650.182745] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19650.188052] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19650.196733] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19650.196734] Workqueue: events delayed_work [ceph] [19650.201619] try_get_cap_refs+0x159/0x880 [ceph] [19650.210124] [19650.210125] RIP: 0010:lock_acquire+0xf0/0x2b0 [19650.213961] __ceph_get_caps+0x4a1/0x760 [ceph] [19650.223330] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19650.228120] ? __file_remove_privs+0x89/0x110 [19650.235586] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19650.240284] ? wake_bit_function+0x70/0x70 [19650.244897] [19650.244898] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19650.246389] ? ceph_write_iter+0x260/0x770 [ceph] [19650.250736] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19650.255260] ceph_write_iter+0x260/0x770 [ceph] [19650.273998] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19650.278353] ? lock_is_held_type+0xaa/0x120 [19650.283565] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19650.287658] vfs_write+0x324/0x4f0 [19650.289147] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19650.296275] ksys_pwrite64+0x65/0xa0 [19650.300969] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19650.308094] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19650.312617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19650.319742] do_syscall_64+0x38/0x80 [19650.323919] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19650.331042] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19650.334440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19650.341565] RIP: 0033:0x7f48f0eeb93b [19650.345135] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19650.353215] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19650.358343] Call Trace: [19650.364079] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19650.367652] [19650.374776] ORIG_RAX: 0000000000000012 [19650.379821] ? watchdog_timer_fn+0x1b4/0x220 [19650.386943] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19650.390515] ? __hrtimer_run_queues+0x1d3/0x360 [19650.397638] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19650.416379] ? hrtimer_interrupt+0x104/0x240 [19650.418821] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19650.424037] ? __local_bh_enable+0x50/0x60 [19650.426048] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19650.429898] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19650.434160] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19650.441284] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19650.445813] [19650.452934] [19650.497117] [19650.499214] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19650.504523] ? lock_acquire+0xf0/0x2b0 [19650.508272] ? igrab+0x43/0x50 [19650.511323] ? lock_release+0xc5/0x260 [19650.515067] _raw_spin_lock+0x30/0x40 [19650.518724] ? igrab+0x1d/0x50 [19650.521775] igrab+0x1d/0x50 [19650.524654] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19650.529802] delayed_work+0x1d8/0x280 [ceph] [19650.534084] ? process_scheduled_works+0x22a/0x510 [19650.538873] process_scheduled_works+0x2a4/0x510 [19650.543489] worker_thread+0x188/0x340 [19650.547237] ? rescuer_thread+0x310/0x310 [19650.551242] kthread+0xf5/0x130 [19650.554389] ? kthread_complete_and_exit+0x20/0x20 [19650.559180] ret_from_fork+0x31/0x40 [19650.562759] ? kthread_complete_and_exit+0x20/0x20 [19650.567545] ret_from_fork_asm+0x11/0x20 [19650.571475] [19677.801728] watchdog: BUG: soft lockup - CPU#0 stuck for 7212s! [dbench:78683] [19677.808946] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19677.813728] watchdog: BUG: soft lockup - CPU#7 stuck for 16209s! [kworker/7:4:78808] [19677.864402] irq event stamp: 28942846 [19677.864403] hardirqs last enabled at (28942845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19677.872136] Modules linked in: ceph [19677.875794] hardirqs last disabled at (28942846): [] sysvec_apic_timer_interrupt+0xe/0x90 [19677.885951] libceph [19677.889436] softirqs last enabled at (28942836): [] __do_softirq+0x2be/0x371 [19677.899158] dns_resolver fscache [19677.901342] softirqs last disabled at (28942831): [] irq_exit_rcu+0xb0/0xd0 [19677.910028] netfs [19677.913339] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19677.921848] veth nft_chain_nat [19677.923859] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19677.932803] xt_MASQUERADE [19677.935941] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19677.943415] nf_nat nf_conntrack [19677.946124] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19677.951603] nf_defrag_ipv6 nf_defrag_ipv4 [19677.954826] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19677.973563] nft_compat nf_tables [19677.977654] [19677.982871] nfnetlink bridge [19677.986182] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19677.987674] stp llc [19677.990638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19677.997761] xfs libcrc32c [19677.999945] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19678.007070] sunrpc intel_rapl_msr [19678.009774] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19678.016897] intel_rapl_common sb_edac [19678.020296] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19678.027419] x86_pkg_temp_thermal coretemp [19678.031164] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19678.038289] kvm_intel [19678.042378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19678.050456] kvm irqbypass [19678.052812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19678.058549] i2c_i801 iTCO_wdt [19678.061255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19678.068378] iTCO_vendor_support i2c_smbus [19678.071430] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19678.078555] crct10dif_pclmul [19678.082644] Call Trace: [19678.082645] [19678.089767] joydev crc32_pclmul [19678.092733] ? watchdog_timer_fn+0x1b4/0x220 [19678.095175] ipmi_ssif ghash_clmulni_intel [19678.097188] ? __hrtimer_run_queues+0x1d3/0x360 [19678.100410] mei_me mei [19678.104679] ? hrtimer_interrupt+0x104/0x240 [19678.108767] lpc_ich sha512_ssse3 [19678.113291] ? __local_bh_enable+0x50/0x60 [19678.115735] acpi_ipmi mfd_core [19678.120001] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19678.123310] wmi ipmi_si [19678.127408] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19678.130545] ipmi_devintf ipmi_msghandler [19678.135763] [19678.138293] ioatdma acpi_pad [19678.143251] [19678.147254] acpi_power_meter ixgbe [19678.149354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19678.152317] igb mdio [19678.154421] ? queued_spin_lock_slowpath+0x16/0x2e0 [19678.157898] crc32c_intel nvme [19678.163205] do_raw_spin_lock+0xb5/0xc0 [19678.165474] ptp nvme_core [19678.170344] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19678.173394] i2c_algo_bit [19678.177245] try_get_cap_refs+0x159/0x880 [ceph] [19678.179947] pps_core dca [19678.184735] __ceph_get_caps+0x4a1/0x760 [ceph] [19678.187349] fuse [19678.191963] ? __file_remove_privs+0x89/0x110 [19678.194577] irq event stamp: 80155430 [19678.199100] ? wake_bit_function+0x70/0x70 [19678.201025] hardirqs last enabled at (80155429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19678.205377] ? ceph_write_iter+0x260/0x770 [ceph] [19678.209032] hardirqs last disabled at (80155430): [] sysvec_apic_timer_interrupt+0xe/0x90 [19678.213129] ceph_write_iter+0x260/0x770 [ceph] [19678.223289] softirqs last enabled at (80155428): [] __do_softirq+0x2be/0x371 [19678.227989] ? lock_is_held_type+0xaa/0x120 [19678.237710] softirqs last disabled at (80155423): [] irq_exit_rcu+0xb0/0xd0 [19678.242239] vfs_write+0x324/0x4f0 [19678.250919] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19678.255101] ksys_pwrite64+0x65/0xa0 [19678.263607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19678.267004] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19678.276383] Workqueue: events delayed_work [ceph] [19678.279972] do_syscall_64+0x38/0x80 [19678.287442] [19678.287443] RIP: 0010:lock_release+0x14e/0x260 [19678.292573] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19678.297269] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [19678.300842] RIP: 0033:0x7f48f0eeb93b [19678.302331] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [19678.306769] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19678.311813] [19678.311813] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [19678.330551] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19678.334133] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [19678.339346] ORIG_RAX: 0000000000000012 [19678.358086] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [19678.359576] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19678.366699] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [19678.371917] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19678.379040] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [19678.382871] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19678.389996] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19678.397132] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19678.404254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19678.411378] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19678.418502] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19678.425631] [19678.433704] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19678.470137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19678.477263] Call Trace: [19678.479708] [19678.481719] ? watchdog_timer_fn+0x1b4/0x220 [19678.485983] ? __hrtimer_run_queues+0x1d3/0x360 [19678.490510] ? hrtimer_interrupt+0x104/0x240 [19678.494780] ? __local_bh_enable+0x50/0x60 [19678.498871] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19678.504095] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19678.509053] [19678.511150] [19678.513249] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19678.518553] ? igrab+0x43/0x50 [19678.521606] ? lock_release+0x14e/0x260 [19678.525443] ? lock_release+0xc5/0x260 [19678.529189] _raw_spin_unlock+0x1b/0x40 [19678.533027] igrab+0x43/0x50 [19678.535903] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19678.541054] delayed_work+0x1d8/0x280 [ceph] [19678.545336] ? process_scheduled_works+0x22a/0x510 [19678.550134] process_scheduled_works+0x2a4/0x510 [19678.554748] worker_thread+0x188/0x340 [19678.558498] ? rescuer_thread+0x310/0x310 [19678.562502] kthread+0xf5/0x130 [19678.565638] ? kthread_complete_and_exit+0x20/0x20 [19678.570423] ret_from_fork+0x31/0x40 [19678.574001] ? kthread_complete_and_exit+0x20/0x20 [19678.578787] ret_from_fork_asm+0x11/0x20 [19678.582710] [19705.802242] watchdog: BUG: soft lockup - CPU#0 stuck for 7238s! [dbench:78683] [19705.809460] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19705.814241] watchdog: BUG: soft lockup - CPU#7 stuck for 16235s! [kworker/7:4:78808] [19705.864917] irq event stamp: 29000920 [19705.872650] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19705.876306] hardirqs last enabled at (29000919): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19705.876311] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [19705.893424] hardirqs last disabled at (29000920): [] sysvec_apic_timer_interrupt+0xe/0x90 [19705.902636] nf_tables nfnetlink bridge stp llc xfs [19705.912359] softirqs last enabled at (29000892): [] __do_softirq+0x2be/0x371 [19705.912363] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19705.925916] softirqs last disabled at (29000887): [] irq_exit_rcu+0xb0/0xd0 [19705.925919] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [19705.942765] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19705.948675] iTCO_vendor_support i2c_smbus [19705.957636] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19705.961727] crct10dif_pclmul joydev crc32_pclmul [19705.969197] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19705.973894] ipmi_ssif ghash_clmulni_intel mei_me mei [19705.979374] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19705.984417] lpc_ich sha512_ssse3 acpi_ipmi [19706.003153] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19706.003155] mfd_core [19706.003155] [19706.007330] wmi ipmi_si [19706.012550] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19706.014820] ipmi_devintf [19706.016310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19706.018842] ipmi_msghandler [19706.025966] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19706.028584] ioatdma acpi_pad [19706.035708] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19706.038584] acpi_power_meter ixgbe [19706.045708] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19706.048672] igb mdio [19706.055797] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19706.059281] crc32c_intel nvme [19706.066405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19706.068676] ptp nvme_core [19706.076751] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19706.079804] i2c_algo_bit pps_core [19706.085540] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19706.088245] dca fuse [19706.095369] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19706.098767] [19706.098768] irq event stamp: 80376526 [19706.105890] Call Trace: [19706.105892] [19706.108160] hardirqs last enabled at (80376525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19706.115288] ? watchdog_timer_fn+0x1b4/0x220 [19706.116776] hardirqs last disabled at (80376526): [] sysvec_apic_timer_interrupt+0xe/0x90 [19706.120433] ? __hrtimer_run_queues+0x1d3/0x360 [19706.122878] softirqs last enabled at (80376524): [] __do_softirq+0x2be/0x371 [19706.124891] ? hrtimer_interrupt+0x104/0x240 [19706.135045] softirqs last disabled at (80376519): [] irq_exit_rcu+0xb0/0xd0 [19706.139309] ? __local_bh_enable+0x50/0x60 [19706.149034] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19706.153561] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19706.162241] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19706.162242] Workqueue: events delayed_work [ceph] [19706.166508] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19706.175018] [19706.175019] RIP: 0010:igrab+0x3b/0x50 [19706.179108] [19706.188487] Code: 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 48 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 <48> 89 ef e8 7d 60 7f 00 31 c0 5b 5d c3 0f 1f 84 00 00 00 00 00 f3 [19706.193703] [19706.201175] RSP: 0018:ffffc90002117da0 EFLAGS: 00000202 [19706.205872] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19706.210828] [19706.210829] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [19706.212322] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19706.215977] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [19706.218076] do_raw_spin_lock+0xb5/0xc0 [19706.236811] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [19706.238911] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19706.244125] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19706.249433] try_get_cap_refs+0x159/0x880 [ceph] [19706.250924] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19706.258051] __ceph_get_caps+0x4a1/0x760 [ceph] [19706.262921] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19706.270054] ? __file_remove_privs+0x89/0x110 [19706.273879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19706.281004] ? wake_bit_function+0x70/0x70 [19706.285788] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19706.292926] ? ceph_write_iter+0x260/0x770 [ceph] [19706.297532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19706.304655] ceph_write_iter+0x260/0x770 [ceph] [19706.309178] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19706.317262] ? lock_is_held_type+0xaa/0x120 [19706.321607] Call Trace: [19706.327347] vfs_write+0x324/0x4f0 [19706.331437] [19706.338565] ksys_pwrite64+0x65/0xa0 [19706.343260] ? watchdog_timer_fn+0x1b4/0x220 [19706.350384] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19706.354906] ? __hrtimer_run_queues+0x1d3/0x360 [19706.362032] do_syscall_64+0x38/0x80 [19706.366212] ? hrtimer_interrupt+0x104/0x240 [19706.368652] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19706.372049] ? __local_bh_enable+0x50/0x60 [19706.374061] RIP: 0033:0x7f48f0eeb93b [19706.377648] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19706.381926] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19706.387051] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19706.391574] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19706.395145] [19706.399411] ORIG_RAX: 0000000000000012 [19706.404455] [19706.408546] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19706.412117] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19706.417334] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19706.436075] ? igrab+0x3b/0x50 [19706.441029] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19706.446247] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19706.448343] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19706.452177] delayed_work+0x1d8/0x280 [ceph] [19706.454270] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19706.461398] ? process_scheduled_works+0x22a/0x510 [19706.466704] [19706.473822] process_scheduled_works+0x2a4/0x510 [19706.519234] worker_thread+0x188/0x340 [19706.522985] ? rescuer_thread+0x310/0x310 [19706.526996] kthread+0xf5/0x130 [19706.530141] ? kthread_complete_and_exit+0x20/0x20 [19706.534932] ret_from_fork+0x31/0x40 [19706.538504] ? kthread_complete_and_exit+0x20/0x20 [19706.543289] ret_from_fork_asm+0x11/0x20 [19706.547220] [19733.802754] watchdog: BUG: soft lockup - CPU#0 stuck for 7264s! [dbench:78683] [19733.809973] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19733.814754] watchdog: BUG: soft lockup - CPU#7 stuck for 16261s! [kworker/7:4:78808] [19733.865414] irq event stamp: 29058988 [19733.865415] hardirqs last enabled at (29058987): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19733.873164] Modules linked in: [19733.876819] hardirqs last disabled at (29058988): [] sysvec_apic_timer_interrupt+0xe/0x90 [19733.886977] ceph [19733.890027] softirqs last enabled at (29058960): [] __do_softirq+0x2be/0x371 [19733.899752] libceph [19733.901677] softirqs last disabled at (29058955): [] irq_exit_rcu+0xb0/0xd0 [19733.910359] dns_resolver fscache [19733.912544] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19733.921055] netfs veth [19733.924365] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19733.933311] nft_chain_nat xt_MASQUERADE [19733.935755] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19733.943226] nf_nat nf_conntrack [19733.947150] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19733.952628] nf_defrag_ipv6 nf_defrag_ipv4 [19733.955852] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19733.974590] nft_compat nf_tables [19733.978681] [19733.983898] nfnetlink bridge [19733.987209] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19733.988700] stp [19733.991664] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19733.998787] llc xfs [19734.000625] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19734.007750] libcrc32c [19734.009934] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19734.017059] sunrpc [19734.019416] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19734.026538] intel_rapl_msr [19734.028636] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19734.035762] intel_rapl_common sb_edac [19734.038551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19734.046630] x86_pkg_temp_thermal coretemp [19734.050373] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19734.056109] kvm_intel kvm [19734.060201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19734.067324] irqbypass i2c_i801 [19734.070031] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19734.077154] iTCO_wdt [19734.080290] Call Trace: [19734.080292] [19734.087414] iTCO_vendor_support i2c_smbus [19734.089687] ? watchdog_timer_fn+0x1b4/0x220 [19734.092130] crct10dif_pclmul joydev [19734.094159] ? __hrtimer_run_queues+0x1d3/0x360 [19734.098247] crc32_pclmul ipmi_ssif [19734.102516] ? hrtimer_interrupt+0x104/0x240 [19734.106083] ghash_clmulni_intel mei_me [19734.110607] ? __local_bh_enable+0x50/0x60 [19734.114091] mei lpc_ich [19734.118359] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19734.122185] sha512_ssse3 acpi_ipmi [19734.126276] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19734.128807] mfd_core wmi [19734.134026] [19734.137509] ipmi_si ipmi_devintf [19734.142468] [19734.142469] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19734.145085] ipmi_msghandler [19734.147184] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19734.150491] ioatdma [19734.152592] do_raw_spin_lock+0xb5/0xc0 [19734.157894] acpi_pad acpi_power_meter [19734.160771] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19734.165643] ixgbe [19734.167829] try_get_cap_refs+0x159/0x880 [ceph] [19734.171656] igb [19734.175406] __ceph_get_caps+0x4a1/0x760 [ceph] [19734.180184] mdio crc32c_intel [19734.182200] ? __file_remove_privs+0x89/0x110 [19734.186805] nvme ptp [19734.188644] ? wake_bit_function+0x70/0x70 [19734.193168] nvme_core i2c_algo_bit [19734.196220] ? ceph_write_iter+0x260/0x770 [ceph] [19734.200569] pps_core dca [19734.202841] ceph_write_iter+0x260/0x770 [ceph] [19734.206930] fuse [19734.210418] ? lock_is_held_type+0xaa/0x120 [19734.215112] irq event stamp: 80598168 [19734.217733] vfs_write+0x324/0x4f0 [19734.222253] hardirqs last enabled at (80598167): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19734.224182] ksys_pwrite64+0x65/0xa0 [19734.228355] hardirqs last disabled at (80598168): [] sysvec_apic_timer_interrupt+0xe/0x90 [19734.232012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19734.235410] softirqs last enabled at (80598166): [] __do_softirq+0x2be/0x371 [19734.245587] do_syscall_64+0x38/0x80 [19734.249157] softirqs last disabled at (80598161): [] irq_exit_rcu+0xb0/0xd0 [19734.258880] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19734.264009] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19734.272695] RIP: 0033:0x7f48f0eeb93b [19734.276264] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19734.284775] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19734.289819] Workqueue: events delayed_work [ceph] [19734.299198] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19734.302767] [19734.302768] RIP: 0010:lock_acquire+0xf0/0x2b0 [19734.310238] ORIG_RAX: 0000000000000012 [19734.328976] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19734.333675] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19734.338891] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19734.340381] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19734.344732] [19734.344732] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19734.348564] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19734.367301] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19734.374426] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19734.379659] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19734.386786] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19734.388276] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19734.395405] [19734.402523] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19734.447471] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19734.455547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19734.461284] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19734.468409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19734.475534] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19734.482658] Call Trace: [19734.485104] [19734.487112] ? watchdog_timer_fn+0x1b4/0x220 [19734.491378] ? __hrtimer_run_queues+0x1d3/0x360 [19734.495905] ? hrtimer_interrupt+0x104/0x240 [19734.500172] ? __local_bh_enable+0x50/0x60 [19734.504266] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19734.509490] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19734.514454] [19734.516553] [19734.518651] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19734.523959] ? lock_acquire+0xf0/0x2b0 [19734.527709] ? igrab+0x43/0x50 [19734.530759] ? lock_release+0xc5/0x260 [19734.534504] _raw_spin_lock+0x30/0x40 [19734.538162] ? igrab+0x1d/0x50 [19734.541211] igrab+0x1d/0x50 [19734.544089] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19734.549238] delayed_work+0x1d8/0x280 [ceph] [19734.553520] ? process_scheduled_works+0x22a/0x510 [19734.558311] process_scheduled_works+0x2a4/0x510 [19734.562926] worker_thread+0x188/0x340 [19734.566675] ? rescuer_thread+0x310/0x310 [19734.570687] kthread+0xf5/0x130 [19734.573825] ? kthread_complete_and_exit+0x20/0x20 [19734.578607] ret_from_fork+0x31/0x40 [19734.582186] ? kthread_complete_and_exit+0x20/0x20 [19734.586973] ret_from_fork_asm+0x11/0x20 [19734.590894] [19761.803268] watchdog: BUG: soft lockup - CPU#0 stuck for 7290s! [dbench:78683] [19761.810486] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19761.815268] watchdog: BUG: soft lockup - CPU#7 stuck for 16287s! [kworker/7:4:78808] [19761.865927] irq event stamp: 29117096 [19761.865928] hardirqs last enabled at (29117095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19761.873677] Modules linked in: [19761.877334] hardirqs last disabled at (29117096): [] sysvec_apic_timer_interrupt+0xe/0x90 [19761.887492] ceph [19761.890540] softirqs last enabled at (29117068): [] __do_softirq+0x2be/0x371 [19761.900265] libceph [19761.902190] softirqs last disabled at (29117063): [] irq_exit_rcu+0xb0/0xd0 [19761.910874] dns_resolver [19761.913059] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19761.921568] fscache netfs [19761.924185] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19761.933130] veth nft_chain_nat [19761.935835] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19761.943306] xt_MASQUERADE nf_nat [19761.946441] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19761.951920] nf_conntrack [19761.955249] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19761.973984] nf_defrag_ipv6 [19761.976602] [19761.976602] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19761.981821] nf_defrag_ipv4 nft_compat [19761.984610] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19761.986101] nf_tables nfnetlink [19761.993225] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19761.996970] bridge stp [19762.004095] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19762.007317] llc xfs [19762.014441] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19762.016885] libcrc32c sunrpc [19762.024009] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19762.026196] intel_rapl_msr intel_rapl_common [19762.033319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19762.036283] sb_edac x86_pkg_temp_thermal [19762.044360] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19762.048710] coretemp kvm_intel [19762.054449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19762.058451] kvm irqbypass [19762.065577] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19762.068715] i2c_i801 iTCO_wdt [19762.075837] Call Trace: [19762.075838] [19762.078542] iTCO_vendor_support i2c_smbus [19762.085667] ? watchdog_timer_fn+0x1b4/0x220 [19762.088717] crct10dif_pclmul joydev [19762.091162] ? __hrtimer_run_queues+0x1d3/0x360 [19762.093172] crc32_pclmul ipmi_ssif [19762.097265] ? hrtimer_interrupt+0x104/0x240 [19762.101525] ghash_clmulni_intel [19762.105096] ? __local_bh_enable+0x50/0x60 [19762.109622] mei_me mei [19762.113107] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19762.117369] lpc_ich sha512_ssse3 [19762.120595] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19762.124684] acpi_ipmi [19762.127130] [19762.132345] mfd_core [19762.135667] [19762.140623] wmi ipmi_si [19762.142981] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19762.145078] ipmi_devintf ipmi_msghandler [19762.147352] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19762.149445] ioatdma acpi_pad [19762.151979] do_raw_spin_lock+0xb5/0xc0 [19762.157279] acpi_power_meter ixgbe [19762.161286] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19762.166156] igb mdio [19762.169121] try_get_cap_refs+0x159/0x880 [ceph] [19762.172951] crc32c_intel nvme [19762.176439] __ceph_get_caps+0x4a1/0x760 [ceph] [19762.181217] ptp nvme_core [19762.183493] ? __file_remove_privs+0x89/0x110 [19762.188100] i2c_algo_bit pps_core [19762.191151] ? wake_bit_function+0x70/0x70 [19762.195674] dca fuse [19762.198382] ? ceph_write_iter+0x260/0x770 [ceph] [19762.202729] [19762.202730] irq event stamp: 80819060 [19762.206127] ceph_write_iter+0x260/0x770 [ceph] [19762.210216] hardirqs last enabled at (80819059): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19762.212491] ? lock_is_held_type+0xaa/0x120 [19762.217186] hardirqs last disabled at (80819060): [] sysvec_apic_timer_interrupt+0xe/0x90 [19762.218677] vfs_write+0x324/0x4f0 [19762.222333] softirqs last enabled at (80819058): [] __do_softirq+0x2be/0x371 [19762.226861] ksys_pwrite64+0x65/0xa0 [19762.237014] softirqs last disabled at (80819053): [] irq_exit_rcu+0xb0/0xd0 [19762.241192] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19762.250915] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19762.254316] do_syscall_64+0x38/0x80 [19762.262998] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19762.266568] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19762.275079] Workqueue: events delayed_work [ceph] [19762.280210] RIP: 0033:0x7f48f0eeb93b [19762.289589] [19762.289590] RIP: 0010:do_raw_spin_lock+0x37/0xc0 [19762.293159] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19762.300629] Code: 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a <31> c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 [19762.305675] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19762.310370] RSP: 0018:ffffc90002117d90 EFLAGS: 00000282 [19762.313951] ORIG_RAX: 0000000000000012 [19762.315441] [19762.315442] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [19762.320051] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19762.338789] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [19762.357526] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19762.362743] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [19762.367963] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19762.371793] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19762.373284] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19762.380408] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [19762.387531] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19762.394672] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19762.401802] [19762.408923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19762.460541] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19762.467675] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19762.474797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19762.481922] Call Trace: [19762.484367] [19762.486379] ? watchdog_timer_fn+0x1b4/0x220 [19762.490669] ? __hrtimer_run_queues+0x1d3/0x360 [19762.495196] ? hrtimer_interrupt+0x104/0x240 [19762.499463] ? __local_bh_enable+0x50/0x60 [19762.503557] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19762.508783] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19762.513739] [19762.515836] [19762.517934] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19762.523260] ? do_raw_spin_lock+0x37/0xc0 [19762.527268] igrab+0x1d/0x50 [19762.530146] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19762.535294] delayed_work+0x1d8/0x280 [ceph] [19762.539576] ? process_scheduled_works+0x22a/0x510 [19762.544367] process_scheduled_works+0x2a4/0x510 [19762.548984] worker_thread+0x188/0x340 [19762.552733] ? rescuer_thread+0x310/0x310 [19762.556743] kthread+0xf5/0x130 [19762.559881] ? kthread_complete_and_exit+0x20/0x20 [19762.564674] ret_from_fork+0x31/0x40 [19762.568245] ? kthread_complete_and_exit+0x20/0x20 [19762.573029] ret_from_fork_asm+0x11/0x20 [19762.576961] [19789.803781] watchdog: BUG: soft lockup - CPU#0 stuck for 7316s! [dbench:78683] [19789.810999] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19789.815781] watchdog: BUG: soft lockup - CPU#7 stuck for 16313s! [kworker/7:4:78808] [19789.866467] irq event stamp: 29175220 [19789.874198] Modules linked in: ceph libceph dns_resolver fscache [19789.877855] hardirqs last enabled at (29175219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19789.877859] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [19789.894019] hardirqs last disabled at (29175220): [] sysvec_apic_timer_interrupt+0xe/0x90 [19789.901922] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [19789.911648] softirqs last enabled at (29175192): [] __do_softirq+0x2be/0x371 [19789.911651] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19789.926771] softirqs last disabled at (29175187): [] irq_exit_rcu+0xb0/0xd0 [19789.935811] coretemp kvm_intel kvm irqbypass [19789.944321] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19789.948672] i2c_i801 iTCO_wdt iTCO_vendor_support [19789.957616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19789.962400] i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [19789.969871] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [19789.969873] ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 [19789.981779] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [19789.987776] acpi_ipmi mfd_core wmi [19790.006513] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19790.006514] ipmi_si [19790.006515] [19790.009999] ipmi_devintf [19790.015215] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19790.017398] ipmi_msghandler [19790.018889] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19790.021508] ioatdma [19790.028632] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19790.031508] acpi_pad acpi_power_meter [19790.038632] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19790.040817] ixgbe igb [19790.047942] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19790.051684] mdio crc32c_intel [19790.058809] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19790.061179] nvme ptp [19790.068299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19790.071350] nvme_core i2c_algo_bit [19790.079429] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19790.081700] pps_core [19790.087436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19790.090938] dca [19790.098061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19790.100334] fuse [19790.107461] Call Trace: [19790.107463] [19790.109302] [19790.109302] irq event stamp: 81040266 [19790.116429] ? watchdog_timer_fn+0x1b4/0x220 [19790.118350] hardirqs last enabled at (81040265): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19790.120795] ? __hrtimer_run_queues+0x1d3/0x360 [19790.122806] hardirqs last disabled at (81040266): [] sysvec_apic_timer_interrupt+0xe/0x90 [19790.124298] ? hrtimer_interrupt+0x104/0x240 [19790.127954] softirqs last enabled at (81040264): [] __do_softirq+0x2be/0x371 [19790.132217] ? __local_bh_enable+0x50/0x60 [19790.142374] softirqs last disabled at (81040259): [] irq_exit_rcu+0xb0/0xd0 [19790.146903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19790.156623] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19790.160887] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19790.169570] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19790.173662] [19790.182182] Workqueue: events delayed_work [ceph] [19790.187399] [19790.187401] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19790.196777] [19790.196778] RIP: 0010:lock_release+0x16a/0x260 [19790.201738] ? queued_spin_lock_slowpath+0x2b/0x2e0 [19790.209205] Code: f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 c4 18 5b 5d 41 5c <41> 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 48 0f a3 05 18 bf 53 01 [19790.211305] do_raw_spin_lock+0xb5/0xc0 [19790.216000] RSP: 0018:ffffc90002117d78 EFLAGS: 00000296 [19790.218099] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19790.223402] [19790.223403] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [19790.224894] try_get_cap_refs+0x159/0x880 [ceph] [19790.229330] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [19790.234206] __ceph_get_caps+0x4a1/0x760 [ceph] [19790.252937] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [19790.256774] ? __file_remove_privs+0x89/0x110 [19790.261985] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [19790.266770] ? wake_bit_function+0x70/0x70 [19790.268261] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [19790.275388] ? ceph_write_iter+0x260/0x770 [ceph] [19790.279995] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19790.287121] ceph_write_iter+0x260/0x770 [ceph] [19790.291645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19790.298773] ? lock_is_held_type+0xaa/0x120 [19790.303119] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19790.310246] vfs_write+0x324/0x4f0 [19790.314334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19790.321471] ksys_pwrite64+0x65/0xa0 [19790.326179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19790.334260] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19790.338783] Call Trace: [19790.338784] [19790.344521] do_syscall_64+0x38/0x80 [19790.348698] ? watchdog_timer_fn+0x1b4/0x220 [19790.355823] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19790.359220] ? __hrtimer_run_queues+0x1d3/0x360 [19790.366344] RIP: 0033:0x7f48f0eeb93b [19790.369918] ? hrtimer_interrupt+0x104/0x240 [19790.377040] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19790.382185] ? __local_bh_enable+0x50/0x60 [19790.384631] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19790.386644] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19790.390211] ORIG_RAX: 0000000000000012 [19790.394476] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19790.399520] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19790.404043] [19790.407615] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19790.411879] [19790.411881] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19790.430617] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19790.434712] ? lock_release+0x16a/0x260 [19790.439923] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19790.445145] _raw_spin_unlock+0x1b/0x40 [19790.448972] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19790.453931] igrab+0x43/0x50 [19790.461060] [19790.463154] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19790.516906] delayed_work+0x1d8/0x280 [ceph] [19790.521191] ? process_scheduled_works+0x22a/0x510 [19790.525977] process_scheduled_works+0x2a4/0x510 [19790.530595] worker_thread+0x188/0x340 [19790.534342] ? rescuer_thread+0x310/0x310 [19790.538346] kthread+0xf5/0x130 [19790.541483] ? kthread_complete_and_exit+0x20/0x20 [19790.546267] ret_from_fork+0x31/0x40 [19790.549848] ? kthread_complete_and_exit+0x20/0x20 [19790.554630] ret_from_fork_asm+0x11/0x20 [19790.558555] [19817.804299] watchdog: BUG: soft lockup - CPU#0 stuck for 7342s! [dbench:78683] [19817.811512] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19817.816298] watchdog: BUG: soft lockup - CPU#7 stuck for 16339s! [kworker/7:4:78808] [19817.866974] irq event stamp: 29233294 [19817.874702] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [19817.878359] hardirqs last enabled at (29233293): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19817.886531] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [19817.896692] hardirqs last disabled at (29233294): [] sysvec_apic_timer_interrupt+0xe/0x90 [19817.902437] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [19817.912159] softirqs last enabled at (29233266): [] __do_softirq+0x2be/0x371 [19817.912163] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [19817.927621] softirqs last disabled at (29233261): [] irq_exit_rcu+0xb0/0xd0 [19817.936308] coretemp kvm_intel kvm irqbypass [19817.944819] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19817.949168] i2c_i801 iTCO_wdt iTCO_vendor_support [19817.958113] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19817.958114] i2c_smbus [19817.958116] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19817.962896] crct10dif_pclmul joydev [19817.970369] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19817.972723] crc32_pclmul [19817.978202] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19817.981773] ipmi_ssif ghash_clmulni_intel [19818.000512] [19818.003127] mei_me mei [19818.008344] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19818.012435] lpc_ich sha512_ssse3 [19818.013926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19818.016369] acpi_ipmi mfd_core [19818.023493] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19818.026804] wmi [19818.033929] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19818.037067] ipmi_si ipmi_devintf [19818.044190] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19818.046027] ipmi_msghandler ioatdma [19818.053152] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19818.056462] acpi_pad acpi_power_meter [19818.063587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19818.067157] ixgbe igb [19818.075234] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19818.078979] mdio [19818.084716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19818.087073] crc32c_intel nvme [19818.094198] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19818.096123] ptp nvme_core [19818.103246] Call Trace: [19818.103248] [19818.106297] i2c_algo_bit pps_core [19818.113424] ? watchdog_timer_fn+0x1b4/0x220 [19818.116124] dca fuse [19818.118569] ? __hrtimer_run_queues+0x1d3/0x360 [19818.120580] [19818.120581] irq event stamp: 81264366 [19818.123981] ? hrtimer_interrupt+0x104/0x240 [19818.128242] hardirqs last enabled at (81264365): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19818.130511] ? __local_bh_enable+0x50/0x60 [19818.135037] hardirqs last disabled at (81264366): [] sysvec_apic_timer_interrupt+0xe/0x90 [19818.136530] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19818.140184] softirqs last enabled at (81264364): [] __do_softirq+0x2be/0x371 [19818.144449] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19818.154606] softirqs last disabled at (81264359): [] irq_exit_rcu+0xb0/0xd0 [19818.158699] [19818.168421] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19818.173638] [19818.182323] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19818.187281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19818.195792] Workqueue: events delayed_work [ceph] [19818.197892] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19818.207266] [19818.209365] do_raw_spin_lock+0xb5/0xc0 [19818.216834] RIP: 0010:lock_acquire+0xf0/0x2b0 [19818.222139] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19818.226836] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19818.231710] try_get_cap_refs+0x159/0x880 [ceph] [19818.233198] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19818.237035] __ceph_get_caps+0x4a1/0x760 [ceph] [19818.241379] [19818.241380] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19818.246166] ? __file_remove_privs+0x89/0x110 [19818.264899] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19818.269511] ? wake_bit_function+0x70/0x70 [19818.274729] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19818.279254] ? ceph_write_iter+0x260/0x770 [ceph] [19818.280744] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19818.287867] ceph_write_iter+0x260/0x770 [ceph] [19818.292218] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19818.299347] ? lock_is_held_type+0xaa/0x120 [19818.303434] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19818.310560] vfs_write+0x324/0x4f0 [19818.315253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19818.322382] ksys_pwrite64+0x65/0xa0 [19818.326902] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19818.334027] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19818.338203] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19818.346283] do_syscall_64+0x38/0x80 [19818.349680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19818.355436] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19818.359004] Call Trace: [19818.359005] [19818.366129] RIP: 0033:0x7f48f0eeb93b [19818.371260] ? watchdog_timer_fn+0x1b4/0x220 [19818.378384] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19818.381956] ? __hrtimer_run_queues+0x1d3/0x360 [19818.389078] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19818.394126] ? hrtimer_interrupt+0x104/0x240 [19818.396566] ORIG_RAX: 0000000000000012 [19818.398577] ? __local_bh_enable+0x50/0x60 [19818.402147] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19818.406415] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19818.425150] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19818.429676] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19818.434891] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19818.439155] [19818.442986] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19818.447077] [19818.454200] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19818.459419] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19818.466549] [19818.471505] ? lock_acquire+0xf0/0x2b0 [19818.508301] ? igrab+0x43/0x50 [19818.511350] ? lock_release+0xc5/0x260 [19818.515097] _raw_spin_lock+0x30/0x40 [19818.518761] ? igrab+0x1d/0x50 [19818.521811] igrab+0x1d/0x50 [19818.524707] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19818.529857] delayed_work+0x1d8/0x280 [ceph] [19818.534138] ? process_scheduled_works+0x22a/0x510 [19818.538929] process_scheduled_works+0x2a4/0x510 [19818.543543] worker_thread+0x188/0x340 [19818.547293] ? rescuer_thread+0x310/0x310 [19818.551296] kthread+0xf5/0x130 [19818.554443] ? kthread_complete_and_exit+0x20/0x20 [19818.559227] ret_from_fork+0x31/0x40 [19818.562807] ? kthread_complete_and_exit+0x20/0x20 [19818.567590] ret_from_fork_asm+0x11/0x20 [19818.571522] [19819.516697] rcu: INFO: rcu_preempt self-detected stall on CPU [19819.522457] rcu: 7-....: (7655985 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/7654728 fqs=3838542 [19819.532874] rcu: (t=7882765 jiffies g=1309609 q=286617 ncpus=8) [19819.538873] Sending NMI from CPU 7 to CPUs 0: [19819.543223] NMI backtrace for cpu 0 [19819.543225] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19819.543226] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19819.543227] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19819.543230] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19819.543232] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19819.543233] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19819.543234] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19819.543234] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19819.543235] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19819.543236] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19819.543237] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19819.543238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19819.543239] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19819.543240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19819.543240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19819.543241] Call Trace: [19819.543242] [19819.543243] ? nmi_cpu_backtrace+0xad/0x120 [19819.543246] ? nmi_cpu_backtrace_handler+0x11/0x20 [19819.543248] ? nmi_handle+0xd4/0x260 [19819.543253] ? default_do_nmi+0x49/0xf0 [19819.543254] ? exc_nmi+0x11e/0x150 [19819.543256] ? end_repeat_nmi+0x16/0x67 [19819.543260] ? queued_spin_lock_slowpath+0x16/0x2e0 [19819.543262] ? queued_spin_lock_slowpath+0x16/0x2e0 [19819.543265] ? queued_spin_lock_slowpath+0x16/0x2e0 [19819.543267] [19819.543267] [19819.543268] do_raw_spin_lock+0xb5/0xc0 [19819.543271] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19819.543285] try_get_cap_refs+0x159/0x880 [ceph] [19819.543299] __ceph_get_caps+0x4a1/0x760 [ceph] [19819.543311] ? __file_remove_privs+0x89/0x110 [19819.543313] ? wake_bit_function+0x70/0x70 [19819.543316] ? ceph_write_iter+0x260/0x770 [ceph] [19819.543326] ceph_write_iter+0x260/0x770 [ceph] [19819.543338] ? lock_is_held_type+0xaa/0x120 [19819.543340] vfs_write+0x324/0x4f0 [19819.543344] ksys_pwrite64+0x65/0xa0 [19819.543346] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19819.543349] do_syscall_64+0x38/0x80 [19819.543351] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19819.543353] RIP: 0033:0x7f48f0eeb93b [19819.543354] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19819.543356] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [19819.543357] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19819.543358] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19819.543358] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19819.543359] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19819.543360] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19819.543363] [19819.544222] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19819.848044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19819.855515] Workqueue: events delayed_work [ceph] [19819.860230] RIP: 0010:lock_acquire+0xf0/0x2b0 [19819.864580] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19819.883318] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19819.888533] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19819.895659] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19819.902784] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19819.909906] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19819.917032] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19819.924156] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19819.932232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19819.937971] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19819.945093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19819.952218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19819.959342] Call Trace: [19819.961787] [19819.963799] ? rcu_dump_cpu_stacks+0xd1/0x180 [19819.968158] ? rcu_sched_clock_irq+0x633/0x1390 [19819.972680] ? lock_release+0xc5/0x260 [19819.976426] ? lock_is_held_type+0xaa/0x120 [19819.980603] ? lock_is_held_type+0xaa/0x120 [19819.984781] ? tick_sched_do_timer+0xa0/0xa0 [19819.989053] ? rcu_read_lock_sched_held+0x56/0x90 [19819.993751] ? tick_sched_do_timer+0xa0/0xa0 [19819.998013] ? update_process_times+0x5f/0x90 [19820.002366] ? tick_sched_handle.isra.16+0x1e/0x50 [19820.007149] ? tick_sched_timer+0x6f/0x80 [19820.011155] ? __hrtimer_run_queues+0x1d3/0x360 [19820.015681] ? hrtimer_interrupt+0x104/0x240 [19820.019949] ? __local_bh_enable+0x50/0x60 [19820.024044] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19820.029268] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19820.034225] [19820.036321] [19820.038421] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19820.043728] ? lock_acquire+0xf0/0x2b0 [19820.047477] ? igrab+0x43/0x50 [19820.050528] ? lock_release+0xc5/0x260 [19820.054273] _raw_spin_lock+0x30/0x40 [19820.057937] ? igrab+0x1d/0x50 [19820.060989] igrab+0x1d/0x50 [19820.063866] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19820.069015] delayed_work+0x1d8/0x280 [ceph] [19820.073296] ? process_scheduled_works+0x22a/0x510 [19820.078087] process_scheduled_works+0x2a4/0x510 [19820.082704] worker_thread+0x188/0x340 [19820.086452] ? rescuer_thread+0x310/0x310 [19820.090455] kthread+0xf5/0x130 [19820.093592] ? kthread_complete_and_exit+0x20/0x20 [19820.098378] ret_from_fork+0x31/0x40 [19820.101956] ? kthread_complete_and_exit+0x20/0x20 [19820.106739] ret_from_fork_asm+0x11/0x20 [19820.110663] [19845.804816] watchdog: BUG: soft lockup - CPU#0 stuck for 7368s! [dbench:78683] [19845.812034] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19845.816816] watchdog: BUG: soft lockup - CPU#7 stuck for 16365s! [kworker/7:4:78808] [19845.867491] irq event stamp: 29291438 [19845.875225] Modules linked in: ceph libceph dns_resolver [19845.878882] hardirqs last enabled at (29291437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19845.884186] fscache netfs veth nft_chain_nat xt_MASQUERADE [19845.894359] hardirqs last disabled at (29291438): [] sysvec_apic_timer_interrupt+0xe/0x90 [19845.894362] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [19845.909650] softirqs last enabled at (29291412): [] __do_softirq+0x2be/0x371 [19845.918775] stp llc xfs libcrc32c sunrpc intel_rapl_msr [19845.927459] softirqs last disabled at (29291407): [] irq_exit_rcu+0xb0/0xd0 [19845.932762] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [19845.941276] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19845.947618] kvm_intel kvm [19845.956562] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19845.956563] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19845.959266] irqbypass i2c_i801 [19845.966736] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19845.972213] iTCO_wdt [19845.975351] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19845.994091] iTCO_vendor_support i2c_smbus [19845.996362] [19846.001578] crct10dif_pclmul joydev [19846.005668] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19846.007159] crc32_pclmul ipmi_ssif [19846.010729] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19846.017855] ghash_clmulni_intel mei_me [19846.021338] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19846.028461] mei lpc_ich [19846.032292] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19846.039418] sha512_ssse3 acpi_ipmi [19846.041947] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19846.049073] mfd_core wmi [19846.052556] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19846.059681] ipmi_si ipmi_devintf [19846.062297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19846.070374] ipmi_msghandler ioatdma [19846.073686] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19846.079424] acpi_pad acpi_power_meter [19846.082994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19846.090119] ixgbe igb [19846.093861] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19846.100985] mdio crc32c_intel [19846.103342] Call Trace: [19846.103345] [19846.110469] nvme ptp [19846.113539] ? watchdog_timer_fn+0x1b4/0x220 [19846.115979] nvme_core i2c_algo_bit [19846.117991] ? __hrtimer_run_queues+0x1d3/0x360 [19846.120261] pps_core dca [19846.124529] ? hrtimer_interrupt+0x104/0x240 [19846.128009] fuse [19846.132534] ? __local_bh_enable+0x50/0x60 [19846.135152] [19846.135153] irq event stamp: 81480852 [19846.139417] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19846.141339] hardirqs last enabled at (81480851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19846.145431] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19846.146921] hardirqs last disabled at (81480852): [] sysvec_apic_timer_interrupt+0xe/0x90 [19846.150577] [19846.155796] softirqs last enabled at (81480850): [] __do_softirq+0x2be/0x371 [19846.165952] [19846.170911] softirqs last disabled at (81480845): [] irq_exit_rcu+0xb0/0xd0 [19846.180635] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19846.182732] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19846.191420] ? queued_spin_lock_slowpath+0x16/0x2e0 [19846.193513] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19846.202027] do_raw_spin_lock+0xb5/0xc0 [19846.207330] Workqueue: events delayed_work [ceph] [19846.216707] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19846.221576] [19846.221577] RIP: 0010:lock_acquire+0xf0/0x2b0 [19846.229050] try_get_cap_refs+0x159/0x880 [ceph] [19846.232878] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19846.237582] __ceph_get_caps+0x4a1/0x760 [ceph] [19846.242359] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19846.243856] ? __file_remove_privs+0x89/0x110 [19846.248218] [19846.248219] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19846.252830] ? wake_bit_function+0x70/0x70 [19846.271567] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19846.276094] ? ceph_write_iter+0x260/0x770 [ceph] [19846.281309] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19846.285659] ceph_write_iter+0x260/0x770 [ceph] [19846.287149] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19846.294278] ? lock_is_held_type+0xaa/0x120 [19846.298365] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19846.305499] vfs_write+0x324/0x4f0 [19846.310195] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19846.317340] ksys_pwrite64+0x65/0xa0 [19846.321860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19846.328984] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19846.333161] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19846.340288] do_syscall_64+0x38/0x80 [19846.343684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19846.351761] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19846.355331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19846.361070] RIP: 0033:0x7f48f0eeb93b [19846.366217] Call Trace: [19846.366218] [19846.373342] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19846.376912] ? watchdog_timer_fn+0x1b4/0x220 [19846.384035] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19846.389083] ? __hrtimer_run_queues+0x1d3/0x360 [19846.396220] ORIG_RAX: 0000000000000012 [19846.399796] ? hrtimer_interrupt+0x104/0x240 [19846.402236] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19846.404248] ? __local_bh_enable+0x50/0x60 [19846.422986] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19846.427251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19846.432467] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19846.436992] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19846.440821] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19846.445086] [19846.452221] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19846.456308] [19846.456309] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19846.463438] [19846.468654] ? lock_acquire+0xf0/0x2b0 [19846.510409] ? igrab+0x43/0x50 [19846.513459] ? lock_release+0xc5/0x260 [19846.517204] _raw_spin_lock+0x30/0x40 [19846.520878] ? igrab+0x1d/0x50 [19846.523929] igrab+0x1d/0x50 [19846.526806] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19846.531956] delayed_work+0x1d8/0x280 [ceph] [19846.536239] ? process_scheduled_works+0x22a/0x510 [19846.541029] process_scheduled_works+0x2a4/0x510 [19846.545645] worker_thread+0x188/0x340 [19846.549393] ? rescuer_thread+0x310/0x310 [19846.553394] kthread+0xf5/0x130 [19846.556533] ? kthread_complete_and_exit+0x20/0x20 [19846.561318] ret_from_fork+0x31/0x40 [19846.564897] ? kthread_complete_and_exit+0x20/0x20 [19846.569682] ret_from_fork_asm+0x11/0x20 [19846.573612] [19873.805332] watchdog: BUG: soft lockup - CPU#0 stuck for 7394s! [dbench:78683] [19873.812548] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19873.817332] watchdog: BUG: soft lockup - CPU#7 stuck for 16391s! [kworker/7:4:78808] [19873.867990] irq event stamp: 29349504 [19873.875739] Modules linked in: ceph libceph dns_resolver [19873.879394] hardirqs last enabled at (29349503): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19873.884699] fscache netfs veth nft_chain_nat xt_MASQUERADE [19873.894874] hardirqs last disabled at (29349504): [] sysvec_apic_timer_interrupt+0xe/0x90 [19873.900439] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [19873.910164] softirqs last enabled at (29349486): [] __do_softirq+0x2be/0x371 [19873.915986] nft_compat nf_tables nfnetlink bridge stp [19873.924687] softirqs last disabled at (29349481): [] irq_exit_rcu+0xb0/0xd0 [19873.929819] llc xfs libcrc32c [19873.938329] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19873.941380] sunrpc intel_rapl_msr [19873.950325] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19873.953728] intel_rapl_common sb_edac [19873.961200] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [19873.961202] x86_pkg_temp_thermal coretemp kvm_intel kvm [19873.970424] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [19873.975727] irqbypass i2c_i801 [19873.994463] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19873.994464] iTCO_wdt iTCO_vendor_support [19873.997601] [19873.997602] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19874.002819] i2c_smbus crct10dif_pclmul [19874.006824] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19874.008314] joydev crc32_pclmul [19874.015438] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19874.019269] ipmi_ssif ghash_clmulni_intel [19874.026392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19874.029616] mei_me mei [19874.036740] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19874.040833] lpc_ich sha512_ssse3 [19874.047956] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19874.050401] acpi_ipmi mfd_core [19874.057523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19874.060834] wmi ipmi_si [19874.068912] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19874.072049] ipmi_devintf [19874.077789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19874.080318] ipmi_msghandler ioatdma [19874.087443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19874.090061] acpi_pad acpi_power_meter [19874.097184] Call Trace: [19874.097185] [19874.100754] ixgbe igb [19874.107880] ? watchdog_timer_fn+0x1b4/0x220 [19874.111622] mdio crc32c_intel [19874.114068] ? __hrtimer_run_queues+0x1d3/0x360 [19874.116076] nvme ptp [19874.118439] ? hrtimer_interrupt+0x104/0x240 [19874.122698] nvme_core i2c_algo_bit [19874.125750] ? __local_bh_enable+0x50/0x60 [19874.130273] pps_core [19874.132547] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19874.136809] dca fuse [19874.140293] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19874.144383] [19874.144384] irq event stamp: 81702202 [19874.146654] [19874.151871] hardirqs last enabled at (81702201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19874.154143] [19874.159099] hardirqs last disabled at (81702202): [] sysvec_apic_timer_interrupt+0xe/0x90 [19874.160590] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19874.164248] softirqs last enabled at (81702200): [] __do_softirq+0x2be/0x371 [19874.166348] ? queued_spin_lock_slowpath+0x18/0x2e0 [19874.176501] softirqs last disabled at (81702195): [] irq_exit_rcu+0xb0/0xd0 [19874.178603] do_raw_spin_lock+0xb5/0xc0 [19874.188324] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19874.193629] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19874.202313] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19874.207186] try_get_cap_refs+0x159/0x880 [ceph] [19874.215694] Workqueue: events delayed_work [ceph] [19874.219528] __ceph_get_caps+0x4a1/0x760 [ceph] [19874.228903] [19874.228904] RIP: 0010:lock_release+0x14e/0x260 [19874.233690] ? __file_remove_privs+0x89/0x110 [19874.241158] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [19874.245768] ? wake_bit_function+0x70/0x70 [19874.250466] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [19874.254992] ? ceph_write_iter+0x260/0x770 [ceph] [19874.256480] [19874.256481] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [19874.260918] ceph_write_iter+0x260/0x770 [ceph] [19874.265268] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [19874.284019] ? lock_is_held_type+0xaa/0x120 [19874.288106] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [19874.293326] vfs_write+0x324/0x4f0 [19874.298020] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [19874.299515] ksys_pwrite64+0x65/0xa0 [19874.306636] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [19874.311160] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19874.318282] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19874.322463] do_syscall_64+0x38/0x80 [19874.329584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19874.332982] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19874.340107] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19874.343676] RIP: 0033:0x7f48f0eeb93b [19874.350801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19874.355932] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19874.364025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19874.367596] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19874.373336] Call Trace: [19874.378380] ORIG_RAX: 0000000000000012 [19874.385503] [19874.389073] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19874.396199] ? watchdog_timer_fn+0x1b4/0x220 [19874.414935] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19874.422060] ? __hrtimer_run_queues+0x1d3/0x360 [19874.427278] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19874.429741] ? hrtimer_interrupt+0x104/0x240 [19874.433570] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19874.435581] ? __local_bh_enable+0x50/0x60 [19874.442705] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19874.446971] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19874.454097] [19874.458616] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19874.500736] [19874.502836] [19874.504933] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19874.510237] ? igrab+0x43/0x50 [19874.513291] ? lock_release+0x14e/0x260 [19874.517128] ? lock_release+0xc5/0x260 [19874.520874] _raw_spin_unlock+0x1b/0x40 [19874.524713] igrab+0x43/0x50 [19874.527605] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19874.532757] delayed_work+0x1d8/0x280 [ceph] [19874.537036] ? process_scheduled_works+0x22a/0x510 [19874.541827] process_scheduled_works+0x2a4/0x510 [19874.546441] worker_thread+0x188/0x340 [19874.550192] ? rescuer_thread+0x310/0x310 [19874.554194] kthread+0xf5/0x130 [19874.557332] ? kthread_complete_and_exit+0x20/0x20 [19874.562117] ret_from_fork+0x31/0x40 [19874.565696] ? kthread_complete_and_exit+0x20/0x20 [19874.570479] ret_from_fork_asm+0x11/0x20 [19874.574402] [19901.805848] watchdog: BUG: soft lockup - CPU#0 stuck for 7421s! [dbench:78683] [19901.813063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19901.817847] watchdog: BUG: soft lockup - CPU#7 stuck for 16418s! [kworker/7:4:78808] [19901.868530] irq event stamp: 29407526 [19901.876261] Modules linked in: ceph libceph dns_resolver fscache netfs [19901.879916] hardirqs last enabled at (29407525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19901.886435] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [19901.896592] hardirqs last disabled at (29407526): [] sysvec_apic_timer_interrupt+0xe/0x90 [19901.902677] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [19901.912401] softirqs last enabled at (29407508): [] __do_softirq+0x2be/0x371 [19901.912406] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [19901.928823] softirqs last disabled at (29407503): [] irq_exit_rcu+0xb0/0xd0 [19901.936035] x86_pkg_temp_thermal coretemp kvm_intel [19901.944548] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19901.949519] kvm irqbypass i2c_i801 [19901.958464] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19901.961948] iTCO_wdt iTCO_vendor_support i2c_smbus [19901.969420] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19901.969421] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [19901.979768] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19901.985331] ghash_clmulni_intel mei_me mei [19902.004069] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19902.008249] lpc_ich sha512_ssse3 [19902.013465] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19902.013467] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19902.016775] acpi_ipmi [19902.023898] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19902.031024] mfd_core wmi [19902.033381] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19902.040522] ipmi_si ipmi_devintf [19902.043140] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19902.050265] ipmi_msghandler ioatdma [19902.053575] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19902.060700] acpi_pad acpi_power_meter [19902.064269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19902.072348] ixgbe igb [19902.076090] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19902.081827] mdio [19902.084187] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19902.091311] crc32c_intel [19902.093246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19902.100367] nvme ptp [19902.102985] Call Trace: [19902.102988] [19902.110109] nvme_core [19902.112381] ? watchdog_timer_fn+0x1b4/0x220 [19902.114822] i2c_algo_bit pps_core [19902.116835] ? __hrtimer_run_queues+0x1d3/0x360 [19902.119191] dca fuse [19902.123459] ? hrtimer_interrupt+0x104/0x240 [19902.126852] [19902.126853] irq event stamp: 81923594 [19902.131376] ? __local_bh_enable+0x50/0x60 [19902.133648] hardirqs last enabled at (81923593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19902.137914] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19902.139402] hardirqs last disabled at (81923594): [] sysvec_apic_timer_interrupt+0xe/0x90 [19902.143059] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19902.147149] softirqs last enabled at (81923592): [] __do_softirq+0x2be/0x371 [19902.157309] [19902.162541] softirqs last disabled at (81923587): [] irq_exit_rcu+0xb0/0xd0 [19902.172265] [19902.177226] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19902.185927] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19902.188024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19902.196539] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19902.198631] Workqueue: events delayed_work [ceph] [19902.208012] do_raw_spin_lock+0xb5/0xc0 [19902.213312] [19902.213313] RIP: 0010:lock_acquire+0xf0/0x2b0 [19902.220785] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19902.225654] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19902.230356] try_get_cap_refs+0x159/0x880 [ceph] [19902.234184] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19902.235677] __ceph_get_caps+0x4a1/0x760 [ceph] [19902.240023] [19902.240024] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19902.244813] ? __file_remove_privs+0x89/0x110 [19902.263546] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19902.268158] ? wake_bit_function+0x70/0x70 [19902.273373] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19902.277900] ? ceph_write_iter+0x260/0x770 [ceph] [19902.279389] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19902.286514] ceph_write_iter+0x260/0x770 [ceph] [19902.290880] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19902.290882] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19902.298009] ? lock_is_held_type+0xaa/0x120 [19902.302095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19902.309222] vfs_write+0x324/0x4f0 [19902.313917] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19902.321045] ksys_pwrite64+0x65/0xa0 [19902.325567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19902.332692] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19902.340768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19902.344946] do_syscall_64+0x38/0x80 [19902.350681] Call Trace: [19902.350683] [19902.354080] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19902.361205] ? watchdog_timer_fn+0x1b4/0x220 [19902.364777] RIP: 0033:0x7f48f0eeb93b [19902.371900] ? __hrtimer_run_queues+0x1d3/0x360 [19902.377031] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19902.384157] ? hrtimer_interrupt+0x104/0x240 [19902.387724] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19902.390168] ? __local_bh_enable+0x50/0x60 [19902.392178] ORIG_RAX: 0000000000000012 [19902.397228] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19902.401504] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19902.405077] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19902.409599] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19902.428336] [19902.432600] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19902.437819] [19902.441909] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19902.445739] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19902.450957] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19902.458085] ? lock_acquire+0xf0/0x2b0 [19902.463044] [19902.470164] ? igrab+0x43/0x50 [19902.510014] ? lock_release+0xc5/0x260 [19902.513760] _raw_spin_lock+0x30/0x40 [19902.517425] ? igrab+0x1d/0x50 [19902.520475] igrab+0x1d/0x50 [19902.523353] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19902.528502] delayed_work+0x1d8/0x280 [ceph] [19902.532785] ? process_scheduled_works+0x22a/0x510 [19902.537575] process_scheduled_works+0x2a4/0x510 [19902.542191] worker_thread+0x188/0x340 [19902.545939] ? rescuer_thread+0x310/0x310 [19902.549941] kthread+0xf5/0x130 [19902.553079] ? kthread_complete_and_exit+0x20/0x20 [19902.557865] ret_from_fork+0x31/0x40 [19902.561443] ? kthread_complete_and_exit+0x20/0x20 [19902.566227] ret_from_fork_asm+0x11/0x20 [19902.570176] [19929.806363] watchdog: BUG: soft lockup - CPU#0 stuck for 7447s! [dbench:78683] [19929.813576] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19929.818363] watchdog: BUG: soft lockup - CPU#7 stuck for 16444s! [kworker/7:4:78808] [19929.869034] irq event stamp: 29465602 [19929.869035] hardirqs last enabled at (29465601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19929.876763] Modules linked in: [19929.880423] hardirqs last disabled at (29465602): [] sysvec_apic_timer_interrupt+0xe/0x90 [19929.890580] ceph [19929.893631] softirqs last enabled at (29465584): [] __do_softirq+0x2be/0x371 [19929.903355] libceph dns_resolver [19929.905278] softirqs last disabled at (29465579): [] irq_exit_rcu+0xb0/0xd0 [19929.913963] fscache [19929.917274] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19929.925783] netfs [19929.927969] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19929.936911] veth [19929.938924] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [19929.946394] nft_chain_nat [19929.948317] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [19929.953795] xt_MASQUERADE nf_nat [19929.956500] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19929.975237] nf_conntrack [19929.978548] [19929.978548] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19929.983765] nf_defrag_ipv6 nf_defrag_ipv4 [19929.986383] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19929.987873] nft_compat nf_tables [19929.994997] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19929.999088] nfnetlink bridge [19930.006213] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19930.009522] stp llc [19930.016646] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19930.019611] xfs libcrc32c [19930.026735] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19930.028936] sunrpc intel_rapl_msr [19930.036061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19930.038766] intel_rapl_common sb_edac [19930.046842] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19930.050238] x86_pkg_temp_thermal coretemp [19930.055977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19930.059720] kvm_intel kvm [19930.066845] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19930.070935] irqbypass [19930.078062] Call Trace: [19930.078064] [19930.080764] i2c_i801 iTCO_wdt [19930.087890] ? watchdog_timer_fn+0x1b4/0x220 [19930.090247] iTCO_vendor_support i2c_smbus [19930.092690] ? __hrtimer_run_queues+0x1d3/0x360 [19930.094701] crct10dif_pclmul joydev [19930.097767] ? hrtimer_interrupt+0x104/0x240 [19930.102024] crc32_pclmul ipmi_ssif [19930.106133] ? __local_bh_enable+0x50/0x60 [19930.110657] ghash_clmulni_intel [19930.114231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19930.118492] mei_me [19930.121975] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19930.126067] mei lpc_ich sha512_ssse3 [19930.129291] [19930.134508] acpi_ipmi mfd_core [19930.136603] [19930.141562] wmi ipmi_si [19930.145220] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19930.147317] ipmi_devintf ipmi_msghandler [19930.150458] ? queued_spin_lock_slowpath+0x2d/0x2e0 [19930.152552] ioatdma acpi_pad [19930.155084] do_raw_spin_lock+0xb5/0xc0 [19930.160386] acpi_power_meter [19930.164392] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19930.169261] ixgbe igb [19930.172226] try_get_cap_refs+0x159/0x880 [ceph] [19930.176055] mdio crc32c_intel [19930.179042] __ceph_get_caps+0x4a1/0x760 [ceph] [19930.183821] nvme ptp [19930.186182] ? __file_remove_privs+0x89/0x110 [19930.190788] nvme_core i2c_algo_bit [19930.193841] ? wake_bit_function+0x70/0x70 [19930.198365] pps_core dca [19930.200638] ? ceph_write_iter+0x260/0x770 [ceph] [19930.204986] fuse [19930.208470] ceph_write_iter+0x260/0x770 [ceph] [19930.212561] [19930.212562] irq event stamp: 82145000 [19930.215182] ? lock_is_held_type+0xaa/0x120 [19930.219876] hardirqs last enabled at (82144999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19930.221801] vfs_write+0x324/0x4f0 [19930.226323] hardirqs last disabled at (82145000): [] sysvec_apic_timer_interrupt+0xe/0x90 [19930.227818] ksys_pwrite64+0x65/0xa0 [19930.231472] softirqs last enabled at (82144998): [] __do_softirq+0x2be/0x371 [19930.235650] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19930.245805] softirqs last disabled at (82144993): [] irq_exit_rcu+0xb0/0xd0 [19930.249207] do_syscall_64+0x38/0x80 [19930.258929] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19930.262498] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19930.271182] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19930.276314] RIP: 0033:0x7f48f0eeb93b [19930.284825] Workqueue: events delayed_work [ceph] [19930.288396] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19930.297772] [19930.297773] RIP: 0010:lock_acquire+0xf0/0x2b0 [19930.302817] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19930.310289] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19930.313858] ORIG_RAX: 0000000000000012 [19930.318555] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19930.337294] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19930.338784] [19930.338785] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19930.343136] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19930.348351] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19930.367091] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19930.370921] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19930.376139] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19930.383262] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19930.384764] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19930.391886] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19930.399015] [19930.406134] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19930.459150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19930.464887] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19930.472010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19930.479136] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19930.486261] Call Trace: [19930.488703] [19930.490716] ? watchdog_timer_fn+0x1b4/0x220 [19930.494980] ? __hrtimer_run_queues+0x1d3/0x360 [19930.499506] ? hrtimer_interrupt+0x104/0x240 [19930.503774] ? __local_bh_enable+0x50/0x60 [19930.507870] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19930.513092] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19930.518060] [19930.520154] [19930.522254] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19930.527562] ? lock_acquire+0xf0/0x2b0 [19930.531312] ? igrab+0x43/0x50 [19930.534361] ? lock_release+0xc5/0x260 [19930.538106] _raw_spin_lock+0x30/0x40 [19930.541768] ? igrab+0x1d/0x50 [19930.544820] igrab+0x1d/0x50 [19930.547699] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19930.552849] delayed_work+0x1d8/0x280 [ceph] [19930.557130] ? process_scheduled_works+0x22a/0x510 [19930.561919] process_scheduled_works+0x2a4/0x510 [19930.566535] worker_thread+0x188/0x340 [19930.570286] ? rescuer_thread+0x310/0x310 [19930.574288] kthread+0xf5/0x130 [19930.577426] ? kthread_complete_and_exit+0x20/0x20 [19930.582209] ret_from_fork+0x31/0x40 [19930.585780] ? kthread_complete_and_exit+0x20/0x20 [19930.590566] ret_from_fork_asm+0x11/0x20 [19930.594487] [19957.806877] watchdog: BUG: soft lockup - CPU#0 stuck for 7473s! [dbench:78683] [19957.814098] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19957.818877] watchdog: BUG: soft lockup - CPU#7 stuck for 16470s! [kworker/7:4:78808] [19957.869553] irq event stamp: 29523738 [19957.877287] Modules linked in: ceph libceph dns_resolver fscache netfs veth [19957.880943] hardirqs last enabled at (29523737): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19957.880948] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [19957.898060] hardirqs last disabled at (29523738): [] sysvec_apic_timer_interrupt+0xe/0x90 [19957.906312] nft_compat nf_tables nfnetlink bridge stp [19957.916036] softirqs last enabled at (29523720): [] __do_softirq+0x2be/0x371 [19957.916041] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [19957.929859] softirqs last disabled at (29523715): [] irq_exit_rcu+0xb0/0xd0 [19957.937069] x86_pkg_temp_thermal coretemp kvm_intel kvm [19957.945582] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19957.945583] irqbypass [19957.945584] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19957.950884] i2c_i801 iTCO_wdt [19957.959829] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19957.962186] iTCO_vendor_support i2c_smbus [19957.969658] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19957.972708] crct10dif_pclmul joydev [19957.978185] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19957.982276] crc32_pclmul ipmi_ssif [19958.001016] [19958.004585] ghash_clmulni_intel mei_me [19958.009803] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19958.013285] mei lpc_ich [19958.014776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19958.018608] sha512_ssse3 acpi_ipmi [19958.025733] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19958.028276] mfd_core [19958.035404] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19958.038887] wmi ipmi_si [19958.046012] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19958.048282] ipmi_devintf ipmi_msghandler [19958.055407] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19958.057939] ioatdma acpi_pad [19958.065060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19958.069066] acpi_power_meter ixgbe [19958.077142] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19958.080107] igb mdio [19958.085846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19958.089327] crc32c_intel nvme [19958.096452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19958.098723] ptp nvme_core [19958.105848] Call Trace: [19958.105850] [19958.108898] i2c_algo_bit pps_core [19958.116024] ? watchdog_timer_fn+0x1b4/0x220 [19958.118726] dca fuse [19958.121172] ? __hrtimer_run_queues+0x1d3/0x360 [19958.123181] [19958.123181] irq event stamp: 82366114 [19958.126583] ? hrtimer_interrupt+0x104/0x240 [19958.130843] hardirqs last enabled at (82366113): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19958.133112] ? __local_bh_enable+0x50/0x60 [19958.137638] hardirqs last disabled at (82366114): [] sysvec_apic_timer_interrupt+0xe/0x90 [19958.139131] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19958.142785] softirqs last enabled at (82366112): [] __do_softirq+0x2be/0x371 [19958.147051] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19958.157207] softirqs last disabled at (82366107): [] irq_exit_rcu+0xb0/0xd0 [19958.161298] [19958.171023] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19958.176238] [19958.176240] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19958.184924] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19958.189884] ? queued_spin_lock_slowpath+0x16/0x2e0 [19958.198392] Workqueue: events delayed_work [ceph] [19958.200492] do_raw_spin_lock+0xb5/0xc0 [19958.209867] [19958.209868] RIP: 0010:lock_acquire+0xf0/0x2b0 [19958.211968] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19958.217279] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19958.224751] try_get_cap_refs+0x159/0x880 [ceph] [19958.229619] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19958.234321] __ceph_get_caps+0x4a1/0x760 [ceph] [19958.238146] [19958.238147] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19958.239643] ? __file_remove_privs+0x89/0x110 [19958.243987] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19958.248772] ? wake_bit_function+0x70/0x70 [19958.267510] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19958.272123] ? ceph_write_iter+0x260/0x770 [ceph] [19958.277338] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19958.281862] ceph_write_iter+0x260/0x770 [ceph] [19958.283352] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19958.290483] ? lock_is_held_type+0xaa/0x120 [19958.294827] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19958.301954] vfs_write+0x324/0x4f0 [19958.306042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19958.313171] ksys_pwrite64+0x65/0xa0 [19958.317864] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19958.324989] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19958.329512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19958.336639] do_syscall_64+0x38/0x80 [19958.340814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19958.348893] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19958.352289] Call Trace: [19958.352290] [19958.358028] RIP: 0033:0x7f48f0eeb93b [19958.361598] ? watchdog_timer_fn+0x1b4/0x220 [19958.368722] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19958.373853] ? __hrtimer_run_queues+0x1d3/0x360 [19958.380975] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19958.384565] ? hrtimer_interrupt+0x104/0x240 [19958.391688] ORIG_RAX: 0000000000000012 [19958.396731] ? __local_bh_enable+0x50/0x60 [19958.399177] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19958.401190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19958.404757] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19958.409023] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19958.427761] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19958.432285] [19958.437502] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19958.441764] [19958.441766] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19958.445597] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19958.449692] ? lock_acquire+0xf0/0x2b0 [19958.456816] [19958.462031] ? igrab+0x43/0x50 [19958.513969] ? lock_release+0xc5/0x260 [19958.517714] _raw_spin_lock+0x30/0x40 [19958.521371] ? igrab+0x1d/0x50 [19958.524421] igrab+0x1d/0x50 [19958.527301] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19958.532450] delayed_work+0x1d8/0x280 [ceph] [19958.536730] ? process_scheduled_works+0x22a/0x510 [19958.541520] process_scheduled_works+0x2a4/0x510 [19958.546138] worker_thread+0x188/0x340 [19958.549887] ? rescuer_thread+0x310/0x310 [19958.553897] kthread+0xf5/0x130 [19958.557037] ? kthread_complete_and_exit+0x20/0x20 [19958.561827] ret_from_fork+0x31/0x40 [19958.565397] ? kthread_complete_and_exit+0x20/0x20 [19958.570182] ret_from_fork_asm+0x11/0x20 [19958.574106] [19985.807392] watchdog: BUG: soft lockup - CPU#0 stuck for 7499s! [dbench:78683] [19985.814609] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [19985.819392] watchdog: BUG: soft lockup - CPU#7 stuck for 16496s! [kworker/7:4:78808] [19985.870067] irq event stamp: 29581796 [19985.870069] hardirqs last enabled at (29581795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19985.877800] Modules linked in: [19985.881457] hardirqs last disabled at (29581796): [] sysvec_apic_timer_interrupt+0xe/0x90 [19985.891614] ceph [19985.894665] softirqs last enabled at (29581778): [] __do_softirq+0x2be/0x371 [19985.904390] libceph [19985.906313] softirqs last disabled at (29581773): [] irq_exit_rcu+0xb0/0xd0 [19985.914997] dns_resolver fscache [19985.917182] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19985.925692] netfs veth [19985.929002] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19985.937946] nft_chain_nat xt_MASQUERADE [19985.940393] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [19985.947862] nf_nat nf_conntrack [19985.951787] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [19985.957264] nf_defrag_ipv6 nf_defrag_ipv4 [19985.960491] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [19985.979227] nft_compat nf_tables [19985.983319] [19985.983320] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [19985.988536] nfnetlink bridge [19985.991846] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [19985.993336] stp llc [19986.000462] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [19986.003425] xfs libcrc32c [19986.010549] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [19986.012734] sunrpc intel_rapl_msr [19986.019857] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [19986.022562] intel_rapl_common sb_edac [19986.029686] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [19986.033082] x86_pkg_temp_thermal coretemp [19986.040208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19986.043951] kvm_intel kvm [19986.052028] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [19986.056119] irqbypass i2c_i801 [19986.061856] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19986.064561] iTCO_wdt iTCO_vendor_support [19986.071686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19986.074823] i2c_smbus crct10dif_pclmul [19986.081946] Call Trace: [19986.081948] [19986.085949] joydev crc32_pclmul [19986.093075] ? watchdog_timer_fn+0x1b4/0x220 [19986.096904] ipmi_ssif ghash_clmulni_intel [19986.099351] ? __hrtimer_run_queues+0x1d3/0x360 [19986.101359] mei_me [19986.104587] ? hrtimer_interrupt+0x104/0x240 [19986.108847] mei lpc_ich [19986.112938] ? __local_bh_enable+0x50/0x60 [19986.117464] sha512_ssse3 acpi_ipmi [19986.119563] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19986.123824] mfd_core wmi [19986.126355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19986.130446] ipmi_si ipmi_devintf ipmi_msghandler [19986.133930] [19986.139148] ioatdma acpi_pad [19986.141766] [19986.146739] acpi_power_meter ixgbe [19986.151437] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19986.153535] igb mdio [19986.156502] ? queued_spin_lock_slowpath+0x16/0x2e0 [19986.158596] crc32c_intel [19986.162083] do_raw_spin_lock+0xb5/0xc0 [19986.167383] nvme ptp [19986.169655] __cap_delay_requeue+0xa7/0x1d0 [ceph] [19986.174527] nvme_core i2c_algo_bit [19986.177145] try_get_cap_refs+0x159/0x880 [ceph] [19986.180974] pps_core dca [19986.183248] __ceph_get_caps+0x4a1/0x760 [ceph] [19986.188027] fuse [19986.191516] ? __file_remove_privs+0x89/0x110 [19986.196124] irq event stamp: 82587384 [19986.198741] ? wake_bit_function+0x70/0x70 [19986.203265] hardirqs last enabled at (82587383): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19986.205191] ? ceph_write_iter+0x260/0x770 [ceph] [19986.209540] hardirqs last disabled at (82587384): [] sysvec_apic_timer_interrupt+0xe/0x90 [19986.213197] ceph_write_iter+0x260/0x770 [ceph] [19986.217287] softirqs last enabled at (82587382): [] __do_softirq+0x2be/0x371 [19986.227448] ? lock_is_held_type+0xaa/0x120 [19986.232143] softirqs last disabled at (82587377): [] irq_exit_rcu+0xb0/0xd0 [19986.241869] vfs_write+0x324/0x4f0 [19986.246391] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [19986.255084] ksys_pwrite64+0x65/0xa0 [19986.259261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [19986.267790] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [19986.271188] Workqueue: events delayed_work [ceph] [19986.280566] do_syscall_64+0x38/0x80 [19986.284134] [19986.291607] entry_SYSCALL_64_after_hwframe+0x63/0xcd [19986.296737] RIP: 0010:lock_acquire+0xf0/0x2b0 [19986.301433] RIP: 0033:0x7f48f0eeb93b [19986.305004] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [19986.306496] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [19986.311540] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [19986.315890] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [19986.319461] [19986.338200] ORIG_RAX: 0000000000000012 [19986.356937] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [19986.362153] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [19986.367371] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [19986.368861] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [19986.372692] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [19986.379817] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [19986.386941] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [19986.394064] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [19986.401190] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [19986.408313] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [19986.415437] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [19986.422567] [19986.429684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19986.459932] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [19986.467057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [19986.474181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [19986.481304] Call Trace: [19986.483751] [19986.485760] ? watchdog_timer_fn+0x1b4/0x220 [19986.490025] ? __hrtimer_run_queues+0x1d3/0x360 [19986.494552] ? hrtimer_interrupt+0x104/0x240 [19986.498820] ? __local_bh_enable+0x50/0x60 [19986.502915] ? __sysvec_apic_timer_interrupt+0x61/0x160 [19986.508139] ? sysvec_apic_timer_interrupt+0x6d/0x90 [19986.513096] [19986.515193] [19986.517290] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [19986.522600] ? lock_acquire+0xf0/0x2b0 [19986.526350] ? igrab+0x43/0x50 [19986.529399] ? lock_release+0xc5/0x260 [19986.533145] _raw_spin_lock+0x30/0x40 [19986.536808] ? igrab+0x1d/0x50 [19986.539858] igrab+0x1d/0x50 [19986.542737] ceph_check_delayed_caps+0xb7/0x270 [ceph] [19986.547887] delayed_work+0x1d8/0x280 [ceph] [19986.552168] ? process_scheduled_works+0x22a/0x510 [19986.556960] process_scheduled_works+0x2a4/0x510 [19986.561573] worker_thread+0x188/0x340 [19986.565322] ? rescuer_thread+0x310/0x310 [19986.569325] kthread+0xf5/0x130 [19986.572463] ? kthread_complete_and_exit+0x20/0x20 [19986.577247] ret_from_fork+0x31/0x40 [19986.580818] ? kthread_complete_and_exit+0x20/0x20 [19986.585603] ret_from_fork_asm+0x11/0x20 [19986.589527] [20013.807906] watchdog: BUG: soft lockup - CPU#0 stuck for 7525s! [dbench:78683] [20013.815124] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20013.819905] watchdog: BUG: soft lockup - CPU#7 stuck for 16522s! [kworker/7:4:78808] [20013.870582] irq event stamp: 29639880 [20013.878313] Modules linked in: ceph libceph dns_resolver fscache netfs [20013.881971] hardirqs last enabled at (29639879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20013.888495] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20013.898653] hardirqs last disabled at (29639880): [] sysvec_apic_timer_interrupt+0xe/0x90 [20013.898656] nf_defrag_ipv6 [20013.898657] softirqs last enabled at (29639862): [] __do_softirq+0x2be/0x371 [20013.904739] nf_defrag_ipv4 nft_compat [20013.914461] softirqs last disabled at (29639857): [] irq_exit_rcu+0xb0/0xd0 [20013.917253] nf_tables nfnetlink [20013.925938] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20013.929682] bridge stp [20013.938192] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20013.941416] llc xfs [20013.950360] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20013.952803] libcrc32c sunrpc [20013.960277] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20013.962475] intel_rapl_msr [20013.967954] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20013.970917] intel_rapl_common sb_edac [20013.989658] [20013.992446] x86_pkg_temp_thermal coretemp [20013.997663] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20014.001407] kvm_intel kvm [20014.002898] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20014.006990] irqbypass i2c_i801 [20014.014114] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20014.016818] iTCO_wdt iTCO_vendor_support [20014.023941] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20014.027079] i2c_smbus crct10dif_pclmul [20014.034203] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20014.038208] joydev crc32_pclmul [20014.045330] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20014.049163] ipmi_ssif ghash_clmulni_intel [20014.056304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20014.059528] mei_me mei [20014.067604] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20014.071695] lpc_ich [20014.077434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20014.079878] sha512_ssse3 acpi_ipmi [20014.087002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20014.089185] mfd_core wmi [20014.096309] Call Trace: [20014.096311] [20014.099792] ipmi_si ipmi_devintf [20014.106918] ? watchdog_timer_fn+0x1b4/0x220 [20014.109535] ipmi_msghandler [20014.111981] ? __hrtimer_run_queues+0x1d3/0x360 [20014.113990] ioatdma acpi_pad [20014.117307] ? hrtimer_interrupt+0x104/0x240 [20014.121565] acpi_power_meter [20014.124443] ? __local_bh_enable+0x50/0x60 [20014.128967] ixgbe igb [20014.131934] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20014.136194] mdio crc32c_intel [20014.139160] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20014.143249] nvme ptp [20014.145606] [20014.150824] nvme_core [20014.153874] [20014.153875] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20014.158832] i2c_algo_bit pps_core [20014.161107] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20014.163199] dca fuse [20014.165559] do_raw_spin_lock+0xb5/0xc0 [20014.167655] [20014.167655] irq event stamp: 82808442 [20014.172963] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20014.176355] hardirqs last enabled at (82808441): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20014.181230] try_get_cap_refs+0x159/0x880 [ceph] [20014.183497] hardirqs last disabled at (82808442): [] sysvec_apic_timer_interrupt+0xe/0x90 [20014.187335] __ceph_get_caps+0x4a1/0x760 [ceph] [20014.188820] softirqs last enabled at (82808440): [] __do_softirq+0x2be/0x371 [20014.192482] ? __file_remove_privs+0x89/0x110 [20014.197260] softirqs last disabled at (82808435): [] irq_exit_rcu+0xb0/0xd0 [20014.207418] ? wake_bit_function+0x70/0x70 [20014.212030] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20014.221756] ? ceph_write_iter+0x260/0x770 [ceph] [20014.226277] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20014.234963] ceph_write_iter+0x260/0x770 [ceph] [20014.239313] Workqueue: events delayed_work [ceph] [20014.247827] ? lock_is_held_type+0xaa/0x120 [20014.251912] [20014.251913] RIP: 0010:lock_acquire+0xf0/0x2b0 [20014.261309] vfs_write+0x324/0x4f0 [20014.266007] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20014.273483] ksys_pwrite64+0x65/0xa0 [20014.278001] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20014.282700] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20014.286875] [20014.286875] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20014.288367] do_syscall_64+0x38/0x80 [20014.292718] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20014.296117] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20014.314852] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20014.318424] RIP: 0033:0x7f48f0eeb93b [20014.323641] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20014.328771] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20014.330261] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20014.337386] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20014.340956] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20014.348081] ORIG_RAX: 0000000000000012 [20014.353124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20014.360250] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20014.363820] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20014.370945] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20014.389682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20014.396805] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20014.402024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20014.410101] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20014.413932] Call Trace: [20014.413933] [20014.419670] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20014.426795] ? watchdog_timer_fn+0x1b4/0x220 [20014.433923] [20014.441042] ? __hrtimer_run_queues+0x1d3/0x360 [20014.492118] ? hrtimer_interrupt+0x104/0x240 [20014.496389] ? __local_bh_enable+0x50/0x60 [20014.500483] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20014.505704] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20014.510662] [20014.512760] [20014.514856] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20014.520164] ? lock_acquire+0xf0/0x2b0 [20014.523917] ? igrab+0x43/0x50 [20014.526975] ? lock_release+0xc5/0x260 [20014.530719] _raw_spin_lock+0x30/0x40 [20014.534377] ? igrab+0x1d/0x50 [20014.537426] igrab+0x1d/0x50 [20014.540315] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20014.545464] delayed_work+0x1d8/0x280 [ceph] [20014.549743] ? process_scheduled_works+0x22a/0x510 [20014.554534] process_scheduled_works+0x2a4/0x510 [20014.559150] worker_thread+0x188/0x340 [20014.562899] ? rescuer_thread+0x310/0x310 [20014.566911] kthread+0xf5/0x130 [20014.570057] ? kthread_complete_and_exit+0x20/0x20 [20014.574841] ret_from_fork+0x31/0x40 [20014.578419] ? kthread_complete_and_exit+0x20/0x20 [20014.583204] ret_from_fork_asm+0x11/0x20 [20014.587137] [20015.118296] rcu: INFO: rcu_preempt self-detected stall on CPU [20015.124058] rcu: 0-....: (7896323 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=3934437 [20015.134397] rcu: (t=8078364 jiffies g=1309609 q=292050 ncpus=8) [20015.140394] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20015.149340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20015.156810] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20015.162296] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20015.181059] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20015.186275] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20015.193401] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20015.200525] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20015.207648] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20015.214772] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20015.221897] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20015.229973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20015.235712] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20015.242836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20015.249960] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20015.257084] Call Trace: [20015.259530] [20015.261541] ? rcu_dump_cpu_stacks+0xd1/0x180 [20015.265900] ? rcu_sched_clock_irq+0x633/0x1390 [20015.270434] ? lock_is_held_type+0xaa/0x120 [20015.274620] ? lock_is_held_type+0xaa/0x120 [20015.278806] ? tick_sched_do_timer+0xa0/0xa0 [20015.283077] ? rcu_read_lock_sched_held+0x56/0x90 [20015.287784] ? tick_sched_do_timer+0xa0/0xa0 [20015.292055] ? update_process_times+0x5f/0x90 [20015.296406] ? tick_sched_handle.isra.16+0x1e/0x50 [20015.301200] ? tick_sched_timer+0x6f/0x80 [20015.305214] ? __hrtimer_run_queues+0x1d3/0x360 [20015.309750] ? hrtimer_interrupt+0x104/0x240 [20015.314015] ? __local_bh_enable+0x50/0x60 [20015.318114] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20015.323334] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20015.328291] [20015.330388] [20015.332486] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20015.337808] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20015.342683] do_raw_spin_lock+0xb5/0xc0 [20015.346519] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20015.351333] try_get_cap_refs+0x159/0x880 [ceph] [20015.355974] __ceph_get_caps+0x4a1/0x760 [ceph] [20015.360524] ? __file_remove_privs+0x89/0x110 [20015.364883] ? wake_bit_function+0x70/0x70 [20015.368979] ? ceph_write_iter+0x260/0x770 [ceph] [20015.373697] ceph_write_iter+0x260/0x770 [ceph] [20015.378247] ? lock_is_held_type+0xaa/0x120 [20015.382429] vfs_write+0x324/0x4f0 [20015.385840] ksys_pwrite64+0x65/0xa0 [20015.389419] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20015.394560] do_syscall_64+0x38/0x80 [20015.398139] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20015.403189] RIP: 0033:0x7f48f0eeb93b [20015.406760] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20015.425498] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [20015.433065] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20015.440188] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20015.447329] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20015.454454] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20015.461577] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20015.468714] [20015.470903] Sending NMI from CPU 0 to CPUs 7: [20015.475257] NMI backtrace for cpu 7 [20015.475258] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20015.475260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20015.475260] Workqueue: events delayed_work [ceph] [20015.475274] RIP: 0010:lock_acquire+0x86/0x2b0 [20015.475276] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [20015.475277] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [20015.475278] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [20015.475279] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [20015.475280] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [20015.475280] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [20015.475281] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20015.475282] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20015.475283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20015.475283] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20015.475284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20015.475285] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20015.475285] Call Trace: [20015.475286] [20015.475287] ? nmi_cpu_backtrace+0xad/0x120 [20015.475290] ? nmi_cpu_backtrace_handler+0x11/0x20 [20015.475292] ? nmi_handle+0xd4/0x260 [20015.475296] ? default_do_nmi+0x49/0xf0 [20015.475297] ? exc_nmi+0x11e/0x150 [20015.475299] ? end_repeat_nmi+0x16/0x67 [20015.475305] ? lock_acquire+0x86/0x2b0 [20015.475307] ? lock_acquire+0x86/0x2b0 [20015.475308] ? lock_acquire+0x86/0x2b0 [20015.475327] [20015.475327] [20015.475328] ? igrab+0x43/0x50 [20015.475329] ? lock_release+0xc5/0x260 [20015.475331] _raw_spin_lock+0x30/0x40 [20015.475333] ? igrab+0x1d/0x50 [20015.475334] igrab+0x1d/0x50 [20015.475336] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20015.475350] delayed_work+0x1d8/0x280 [ceph] [20015.475363] ? process_scheduled_works+0x22a/0x510 [20015.475365] process_scheduled_works+0x2a4/0x510 [20015.475369] worker_thread+0x188/0x340 [20015.475372] ? rescuer_thread+0x310/0x310 [20015.475374] kthread+0xf5/0x130 [20015.475376] ? kthread_complete_and_exit+0x20/0x20 [20015.475379] ret_from_fork+0x31/0x40 [20015.475381] ? kthread_complete_and_exit+0x20/0x20 [20015.475383] ret_from_fork_asm+0x11/0x20 [20015.475387] [20041.808419] watchdog: BUG: soft lockup - CPU#0 stuck for 7551s! [dbench:78683] [20041.815637] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20041.820419] watchdog: BUG: soft lockup - CPU#7 stuck for 16548s! [kworker/7:4:78808] [20041.871079] irq event stamp: 29697200 [20041.878827] Modules linked in: ceph libceph dns_resolver [20041.882485] hardirqs last enabled at (29697199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20041.887813] fscache netfs [20041.897971] hardirqs last disabled at (29697200): [] sysvec_apic_timer_interrupt+0xe/0x90 [20041.900676] veth [20041.910400] softirqs last enabled at (29697088): [] __do_softirq+0x2be/0x371 [20041.910402] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [20041.921007] softirqs last disabled at (29697083): [] irq_exit_rcu+0xb0/0xd0 [20041.927959] nf_defrag_ipv4 nft_compat nf_tables [20041.936471] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20041.941094] nfnetlink bridge [20041.950041] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20041.953005] stp llc [20041.960478] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20041.960479] xfs libcrc32c sunrpc intel_rapl_msr [20041.968138] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20041.972748] intel_rapl_common [20041.991486] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20041.994537] sb_edac x86_pkg_temp_thermal [20041.999754] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20041.999756] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20042.003759] coretemp kvm_intel [20042.010883] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20042.018008] kvm irqbypass [20042.021144] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20042.028269] i2c_i801 iTCO_wdt [20042.030973] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20042.038104] iTCO_vendor_support i2c_smbus [20042.041156] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20042.048281] crct10dif_pclmul joydev [20042.052371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20042.060449] crc32_pclmul ipmi_ssif [20042.064019] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20042.069756] ghash_clmulni_intel [20042.073241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20042.080364] mei_me mei [20042.083589] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20042.090713] lpc_ich [20042.093158] Call Trace: [20042.093159] [20042.100281] sha512_ssse3 acpi_ipmi [20042.102465] ? watchdog_timer_fn+0x1b4/0x220 [20042.104910] mfd_core wmi [20042.106922] ? __hrtimer_run_queues+0x1d3/0x360 [20042.110404] ipmi_si ipmi_devintf [20042.114673] ? hrtimer_interrupt+0x104/0x240 [20042.117286] ipmi_msghandler ioatdma [20042.121817] ? __local_bh_enable+0x50/0x60 [20042.125129] acpi_pad acpi_power_meter [20042.129397] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20042.132963] ixgbe igb [20042.137056] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20042.140816] mdio crc32c_intel [20042.146033] [20042.148391] nvme ptp [20042.153348] [20042.156399] nvme_core i2c_algo_bit [20042.158498] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20042.160768] pps_core dca [20042.162868] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20042.166348] fuse [20042.171654] do_raw_spin_lock+0xb5/0xc0 [20042.174270] irq event stamp: 83027624 [20042.179143] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20042.181066] hardirqs last enabled at (83027623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20042.184897] try_get_cap_refs+0x159/0x880 [ceph] [20042.188553] hardirqs last disabled at (83027624): [] sysvec_apic_timer_interrupt+0xe/0x90 [20042.193341] __ceph_get_caps+0x4a1/0x760 [ceph] [20042.203493] softirqs last enabled at (83027622): [] __do_softirq+0x2be/0x371 [20042.208112] ? __file_remove_privs+0x89/0x110 [20042.217830] softirqs last disabled at (83027617): [] irq_exit_rcu+0xb0/0xd0 [20042.222354] ? wake_bit_function+0x70/0x70 [20042.231038] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20042.235390] ? ceph_write_iter+0x260/0x770 [ceph] [20042.243899] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20042.247990] ceph_write_iter+0x260/0x770 [ceph] [20042.257368] Workqueue: events delayed_work [ceph] [20042.262068] ? lock_is_held_type+0xaa/0x120 [20042.269535] [20042.269536] RIP: 0010:lock_acquire+0xf0/0x2b0 [20042.274062] vfs_write+0x324/0x4f0 [20042.278757] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20042.282940] ksys_pwrite64+0x65/0xa0 [20042.284424] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20042.288776] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20042.292173] [20042.310913] do_syscall_64+0x38/0x80 [20042.314481] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20042.319699] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20042.324830] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20042.326321] RIP: 0033:0x7f48f0eeb93b [20042.329891] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20042.337017] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20042.342059] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20042.349183] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20042.352756] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20042.359878] ORIG_RAX: 0000000000000012 [20042.378615] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20042.385742] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20042.390958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20042.398083] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20042.401912] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20042.409989] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20042.417116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20042.422852] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20042.429976] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20042.437104] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20042.444225] Call Trace: [20042.444226] [20042.451353] [20042.458473] ? watchdog_timer_fn+0x1b4/0x220 [20042.483626] ? __hrtimer_run_queues+0x1d3/0x360 [20042.488150] ? hrtimer_interrupt+0x104/0x240 [20042.492413] ? __local_bh_enable+0x50/0x60 [20042.496505] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20042.501730] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20042.506686] [20042.508785] [20042.510883] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20042.516188] ? lock_acquire+0xf0/0x2b0 [20042.519940] ? igrab+0x43/0x50 [20042.522989] ? lock_release+0xc5/0x260 [20042.526734] _raw_spin_lock+0x30/0x40 [20042.530389] ? igrab+0x1d/0x50 [20042.533441] igrab+0x1d/0x50 [20042.536320] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20042.541468] delayed_work+0x1d8/0x280 [ceph] [20042.545749] ? process_scheduled_works+0x22a/0x510 [20042.550531] process_scheduled_works+0x2a4/0x510 [20042.555148] worker_thread+0x188/0x340 [20042.558897] ? rescuer_thread+0x310/0x310 [20042.562901] kthread+0xf5/0x130 [20042.566036] ? kthread_complete_and_exit+0x20/0x20 [20042.570830] ret_from_fork+0x31/0x40 [20042.574400] ? kthread_complete_and_exit+0x20/0x20 [20042.579186] ret_from_fork_asm+0x11/0x20 [20042.583123] [20069.808932] watchdog: BUG: soft lockup - CPU#0 stuck for 7577s! [dbench:78683] [20069.816151] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20069.820932] watchdog: BUG: soft lockup - CPU#7 stuck for 16574s! [kworker/7:4:78808] [20069.871593] irq event stamp: 29755230 [20069.879340] Modules linked in: ceph libceph dns_resolver fscache netfs [20069.882998] hardirqs last enabled at (29755229): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20069.889524] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [20069.899681] hardirqs last disabled at (29755230): [] sysvec_apic_timer_interrupt+0xe/0x90 [20069.899684] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [20069.916790] softirqs last enabled at (29755142): [] __do_softirq+0x2be/0x371 [20069.923922] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [20069.932607] softirqs last disabled at (29755137): [] irq_exit_rcu+0xb0/0xd0 [20069.939123] x86_pkg_temp_thermal coretemp kvm_intel kvm [20069.947635] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20069.952938] irqbypass i2c_i801 [20069.961883] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20069.965021] iTCO_wdt iTCO_vendor_support i2c_smbus [20069.972492] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20069.977363] crct10dif_pclmul [20069.982841] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20069.982842] joydev [20069.985804] crc32_pclmul ipmi_ssif [20070.004541] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20070.006637] ghash_clmulni_intel mei_me [20070.010124] [20070.015339] mei lpc_ich [20070.019172] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20070.020662] sha512_ssse3 acpi_ipmi [20070.023194] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20070.030330] mfd_core wmi [20070.033817] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20070.040943] ipmi_si ipmi_devintf [20070.043560] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20070.050684] ipmi_msghandler ioatdma [20070.053995] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20070.061118] acpi_pad acpi_power_meter [20070.064689] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20070.071814] ixgbe igb [20070.075557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20070.083635] mdio [20070.085993] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20070.091731] crc32c_intel nvme [20070.093654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20070.100779] ptp nvme_core [20070.103828] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20070.110954] i2c_algo_bit pps_core [20070.113655] Call Trace: [20070.113658] [20070.120779] dca fuse [20070.124181] ? watchdog_timer_fn+0x1b4/0x220 [20070.126622] [20070.126623] irq event stamp: 83248676 [20070.128634] ? __hrtimer_run_queues+0x1d3/0x360 [20070.130903] hardirqs last enabled at (83248675): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20070.135171] ? hrtimer_interrupt+0x104/0x240 [20070.136658] hardirqs last disabled at (83248676): [] sysvec_apic_timer_interrupt+0xe/0x90 [20070.140315] ? __local_bh_enable+0x50/0x60 [20070.144839] softirqs last enabled at (83248674): [] __do_softirq+0x2be/0x371 [20070.155000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20070.159262] softirqs last disabled at (83248669): [] irq_exit_rcu+0xb0/0xd0 [20070.168986] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20070.173077] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20070.181762] [20070.186977] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20070.195489] [20070.195491] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20070.200447] Workqueue: events delayed_work [ceph] [20070.209829] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20070.211920] [20070.211921] RIP: 0010:do_raw_spin_lock+0x0/0xc0 [20070.219393] do_raw_spin_lock+0xb5/0xc0 [20070.221489] Code: c0 eb 7e 44 8b 80 28 05 00 00 49 89 e9 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 <66> 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 [20070.226797] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20070.231491] RSP: 0018:ffffc90002117d98 EFLAGS: 00000296 [20070.236364] try_get_cap_refs+0x159/0x880 [ceph] [20070.237853] [20070.237854] RAX: ffffffff813b940d RBX: ffff8881a1d34478 RCX: 000000005553a634 [20070.242382] __ceph_get_caps+0x4a1/0x760 [ceph] [20070.246208] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [20070.264951] ? __file_remove_privs+0x89/0x110 [20070.269729] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [20070.274946] ? wake_bit_function+0x70/0x70 [20070.279556] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20070.281050] ? ceph_write_iter+0x260/0x770 [ceph] [20070.288173] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20070.292696] ceph_write_iter+0x260/0x770 [ceph] [20070.299820] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20070.304176] ? lock_is_held_type+0xaa/0x120 [20070.311295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20070.315388] vfs_write+0x324/0x4f0 [20070.322509] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20070.327213] ksys_pwrite64+0x65/0xa0 [20070.334339] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20070.338866] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20070.346941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20070.351120] do_syscall_64+0x38/0x80 [20070.356856] Call Trace: [20070.356857] [20070.360255] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20070.367379] ? watchdog_timer_fn+0x1b4/0x220 [20070.370948] RIP: 0033:0x7f48f0eeb93b [20070.378075] ? __hrtimer_run_queues+0x1d3/0x360 [20070.383205] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20070.390342] ? hrtimer_interrupt+0x104/0x240 [20070.393907] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20070.396352] ? __local_bh_enable+0x50/0x60 [20070.398363] ORIG_RAX: 0000000000000012 [20070.403410] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20070.407671] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20070.411241] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20070.415765] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20070.434504] [20070.438767] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20070.443985] [20070.448075] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20070.451905] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20070.457123] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20070.464249] ? igrab+0x1d/0x50 [20070.469210] [20070.476338] ? rwlock_bug+0x50/0x50 [20070.515930] igrab+0x1d/0x50 [20070.518807] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20070.523957] delayed_work+0x1d8/0x280 [ceph] [20070.528238] ? process_scheduled_works+0x22a/0x510 [20070.533028] process_scheduled_works+0x2a4/0x510 [20070.537643] worker_thread+0x188/0x340 [20070.541394] ? rescuer_thread+0x310/0x310 [20070.545403] kthread+0xf5/0x130 [20070.548541] ? kthread_complete_and_exit+0x20/0x20 [20070.553339] ret_from_fork+0x31/0x40 [20070.556912] ? kthread_complete_and_exit+0x20/0x20 [20070.561697] ret_from_fork_asm+0x11/0x20 [20070.565636] [20097.809445] watchdog: BUG: soft lockup - CPU#0 stuck for 7603s! [dbench:78683] [20097.816657] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20097.821445] watchdog: BUG: soft lockup - CPU#7 stuck for 16600s! [kworker/7:4:78808] [20097.872098] irq event stamp: 29813254 [20097.872099] hardirqs last enabled at (29813253): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20097.879841] Modules linked in: [20097.883502] hardirqs last disabled at (29813254): [] sysvec_apic_timer_interrupt+0xe/0x90 [20097.893661] ceph [20097.896712] softirqs last enabled at (29813186): [] __do_softirq+0x2be/0x371 [20097.906435] libceph [20097.908359] softirqs last disabled at (29813181): [] irq_exit_rcu+0xb0/0xd0 [20097.917043] dns_resolver fscache [20097.919229] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20097.927739] netfs veth [20097.931048] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20097.939993] nft_chain_nat xt_MASQUERADE [20097.942437] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20097.949908] nf_nat nf_conntrack [20097.953840] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20097.959320] nf_defrag_ipv6 nf_defrag_ipv4 [20097.962545] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20097.981283] nft_compat [20097.985373] [20097.985374] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20097.990590] nf_tables [20097.993035] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20097.994524] nfnetlink bridge [20098.001650] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20098.004007] stp llc [20098.011137] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20098.014103] xfs libcrc32c sunrpc [20098.021245] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20098.023428] intel_rapl_msr intel_rapl_common [20098.030553] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20098.033862] sb_edac x86_pkg_temp_thermal [20098.040987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20098.045337] coretemp kvm_intel [20098.053417] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20098.057421] kvm irqbypass [20098.063159] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20098.066294] i2c_i801 [20098.073419] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20098.076139] iTCO_wdt iTCO_vendor_support [20098.083264] Call Trace: [20098.083265] [20098.085536] i2c_smbus [20098.092660] ? watchdog_timer_fn+0x1b4/0x220 [20098.096663] crct10dif_pclmul [20098.099109] ? __hrtimer_run_queues+0x1d3/0x360 [20098.101139] joydev crc32_pclmul [20098.103495] ? hrtimer_interrupt+0x104/0x240 [20098.107757] ipmi_ssif ghash_clmulni_intel [20098.110722] ? __local_bh_enable+0x50/0x60 [20098.115244] mei_me mei [20098.118471] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20098.122733] lpc_ich sha512_ssse3 [20098.126843] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20098.130933] acpi_ipmi mfd_core [20098.133376] [20098.138592] wmi ipmi_si [20098.141906] [20098.146861] ipmi_devintf [20098.149999] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20098.152097] ipmi_msghandler ioatdma [20098.154650] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20098.156741] acpi_pad acpi_power_meter [20098.159362] do_raw_spin_lock+0xb5/0xc0 [20098.164665] ixgbe igb [20098.168235] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20098.173104] mdio crc32c_intel [20098.176869] try_get_cap_refs+0x159/0x880 [ceph] [20098.180696] nvme [20098.183060] __ceph_get_caps+0x4a1/0x760 [ceph] [20098.187845] ptp nvme_core [20098.190903] ? __file_remove_privs+0x89/0x110 [20098.195509] i2c_algo_bit pps_core [20098.197433] ? wake_bit_function+0x70/0x70 [20098.201957] dca fuse [20098.204665] ? ceph_write_iter+0x260/0x770 [ceph] [20098.209011] [20098.212409] ceph_write_iter+0x260/0x770 [ceph] [20098.216499] irq event stamp: 83469818 [20098.218775] ? lock_is_held_type+0xaa/0x120 [20098.223469] hardirqs last enabled at (83469817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20098.224961] vfs_write+0x324/0x4f0 [20098.229482] hardirqs last disabled at (83469818): [] sysvec_apic_timer_interrupt+0xe/0x90 [20098.233145] ksys_pwrite64+0x65/0xa0 [20098.237318] softirqs last enabled at (83469816): [] __do_softirq+0x2be/0x371 [20098.247476] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20098.250873] softirqs last disabled at (83469811): [] irq_exit_rcu+0xb0/0xd0 [20098.260599] do_syscall_64+0x38/0x80 [20098.264168] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20098.272852] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20098.277982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20098.286493] RIP: 0033:0x7f48f0eeb93b [20098.290063] Workqueue: events delayed_work [ceph] [20098.299442] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20098.304485] [20098.304486] RIP: 0010:lock_release+0x14e/0x260 [20098.311956] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20098.315527] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [20098.320224] ORIG_RAX: 0000000000000012 [20098.338963] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [20098.340452] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20098.344890] [20098.344891] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [20098.350124] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20098.368861] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [20098.372694] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20098.377910] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [20098.385036] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20098.386526] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [20098.393649] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20098.400775] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [20098.400776] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20098.407905] [20098.415023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20098.466642] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20098.473766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20098.480890] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20098.488014] Call Trace: [20098.490460] [20098.492470] ? watchdog_timer_fn+0x1b4/0x220 [20098.496735] ? __hrtimer_run_queues+0x1d3/0x360 [20098.501261] ? hrtimer_interrupt+0x104/0x240 [20098.505530] ? __local_bh_enable+0x50/0x60 [20098.509623] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20098.514853] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20098.519814] [20098.521909] [20098.524008] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20098.529313] ? igrab+0x43/0x50 [20098.532365] ? lock_release+0x14e/0x260 [20098.536194] ? lock_release+0xc5/0x260 [20098.539939] _raw_spin_unlock+0x1b/0x40 [20098.543768] igrab+0x43/0x50 [20098.546647] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20098.551797] delayed_work+0x1d8/0x280 [ceph] [20098.556078] ? process_scheduled_works+0x22a/0x510 [20098.560868] process_scheduled_works+0x2a4/0x510 [20098.565483] worker_thread+0x188/0x340 [20098.569233] ? rescuer_thread+0x310/0x310 [20098.573236] kthread+0xf5/0x130 [20098.576375] ? kthread_complete_and_exit+0x20/0x20 [20098.581165] ret_from_fork+0x31/0x40 [20098.584735] ? kthread_complete_and_exit+0x20/0x20 [20098.589521] ret_from_fork_asm+0x11/0x20 [20098.593443] [20125.809960] watchdog: BUG: soft lockup - CPU#0 stuck for 7629s! [dbench:78683] [20125.817180] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20125.821959] watchdog: BUG: soft lockup - CPU#7 stuck for 16626s! [kworker/7:4:78808] [20125.872635] irq event stamp: 29871488 [20125.880368] Modules linked in: ceph libceph dns_resolver fscache netfs veth [20125.884024] hardirqs last enabled at (29871487): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20125.884030] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [20125.901143] hardirqs last disabled at (29871488): [] sysvec_apic_timer_interrupt+0xe/0x90 [20125.901145] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [20125.919118] softirqs last enabled at (29871420): [] __do_softirq+0x2be/0x371 [20125.925815] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [20125.934500] softirqs last disabled at (29871415): [] irq_exit_rcu+0xb0/0xd0 [20125.943620] kvm irqbypass i2c_i801 iTCO_wdt [20125.952148] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20125.952149] iTCO_vendor_support [20125.952150] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20125.956410] i2c_smbus [20125.965356] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20125.968579] crct10dif_pclmul joydev [20125.976049] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20125.978408] crc32_pclmul [20125.983885] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20125.987456] ipmi_ssif ghash_clmulni_intel [20126.006193] [20126.008810] mei_me [20126.014028] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20126.018118] mei lpc_ich [20126.019610] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20126.021707] sha512_ssse3 acpi_ipmi [20126.028830] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20126.031362] mfd_core wmi [20126.038484] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20126.041968] ipmi_si [20126.049093] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20126.051713] ipmi_devintf [20126.058836] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20126.061018] ipmi_msghandler ioatdma [20126.068143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20126.070760] acpi_pad acpi_power_meter [20126.078839] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20126.082409] ixgbe igb [20126.088148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20126.091891] mdio crc32c_intel [20126.099016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20126.101371] nvme ptp [20126.108496] Call Trace: [20126.111548] nvme_core [20126.118674] [20126.120941] i2c_algo_bit pps_core [20126.123389] ? watchdog_timer_fn+0x1b4/0x220 [20126.125742] dca fuse [20126.127756] ? __hrtimer_run_queues+0x1d3/0x360 [20126.131152] [20126.131153] irq event stamp: 83690776 [20126.135419] ? hrtimer_interrupt+0x104/0x240 [20126.137686] hardirqs last enabled at (83690775): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20126.142210] ? __local_bh_enable+0x50/0x60 [20126.143701] hardirqs last disabled at (83690776): [] sysvec_apic_timer_interrupt+0xe/0x90 [20126.147361] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20126.151623] softirqs last enabled at (83690774): [] __do_softirq+0x2be/0x371 [20126.161781] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20126.165871] softirqs last disabled at (83690769): [] irq_exit_rcu+0xb0/0xd0 [20126.175595] [20126.180812] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20126.189496] [20126.194455] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20126.202965] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20126.205063] Workqueue: events delayed_work [ceph] [20126.214444] ? queued_spin_lock_slowpath+0x16/0x2e0 [20126.216537] [20126.216537] RIP: 0010:lock_acquire+0xf0/0x2b0 [20126.224009] do_raw_spin_lock+0xb5/0xc0 [20126.229312] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20126.234012] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20126.238879] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20126.240374] try_get_cap_refs+0x159/0x880 [ceph] [20126.244721] [20126.244721] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20126.248557] __ceph_get_caps+0x4a1/0x760 [ceph] [20126.267289] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20126.272079] ? __file_remove_privs+0x89/0x110 [20126.277292] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20126.281902] ? wake_bit_function+0x70/0x70 [20126.283392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20126.290519] ? ceph_write_iter+0x260/0x770 [ceph] [20126.295040] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20126.302165] ceph_write_iter+0x260/0x770 [ceph] [20126.306516] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20126.313646] ? lock_is_held_type+0xaa/0x120 [20126.317732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20126.324857] vfs_write+0x324/0x4f0 [20126.329553] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20126.336681] ksys_pwrite64+0x65/0xa0 [20126.341201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20126.349279] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20126.353456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20126.359193] do_syscall_64+0x38/0x80 [20126.362590] Call Trace: [20126.362592] [20126.369716] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20126.373286] ? watchdog_timer_fn+0x1b4/0x220 [20126.380411] RIP: 0033:0x7f48f0eeb93b [20126.385542] ? __hrtimer_run_queues+0x1d3/0x360 [20126.392665] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20126.396238] ? hrtimer_interrupt+0x104/0x240 [20126.398679] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20126.400691] ? __local_bh_enable+0x50/0x60 [20126.405732] ORIG_RAX: 0000000000000012 [20126.410000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20126.413569] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20126.418094] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20126.436829] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20126.441094] [20126.446311] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20126.450402] [20126.454233] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20126.459450] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20126.466575] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20126.471538] ? lock_acquire+0xf0/0x2b0 [20126.478661] [20126.480755] ? igrab+0x43/0x50 [20126.518506] ? lock_release+0xc5/0x260 [20126.522252] _raw_spin_lock+0x30/0x40 [20126.525909] ? igrab+0x1d/0x50 [20126.528958] igrab+0x1d/0x50 [20126.531838] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20126.536989] delayed_work+0x1d8/0x280 [ceph] [20126.541268] ? process_scheduled_works+0x22a/0x510 [20126.546059] process_scheduled_works+0x2a4/0x510 [20126.550674] worker_thread+0x188/0x340 [20126.554423] ? rescuer_thread+0x310/0x310 [20126.558426] kthread+0xf5/0x130 [20126.561564] ? kthread_complete_and_exit+0x20/0x20 [20126.566365] ret_from_fork+0x31/0x40 [20126.569945] ? kthread_complete_and_exit+0x20/0x20 [20126.574728] ret_from_fork_asm+0x11/0x20 [20126.578663] [20153.810474] watchdog: BUG: soft lockup - CPU#0 stuck for 7655s! [dbench:78683] [20153.817692] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20153.822474] watchdog: BUG: soft lockup - CPU#7 stuck for 16652s! [kworker/7:4:78808] [20153.873148] irq event stamp: 29929534 [20153.873150] hardirqs last enabled at (29929533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20153.880880] Modules linked in: ceph [20153.884539] hardirqs last disabled at (29929534): [] sysvec_apic_timer_interrupt+0xe/0x90 [20153.894696] libceph [20153.898180] softirqs last enabled at (29929486): [] __do_softirq+0x2be/0x371 [20153.907903] dns_resolver fscache [20153.910088] softirqs last disabled at (29929481): [] irq_exit_rcu+0xb0/0xd0 [20153.918773] netfs veth [20153.922083] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20153.930593] nft_chain_nat xt_MASQUERADE [20153.933037] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20153.941983] nf_nat nf_conntrack [20153.945900] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20153.953370] nf_defrag_ipv6 [20153.956595] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20153.962071] nf_defrag_ipv4 nft_compat [20153.964869] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20153.983608] nf_tables nfnetlink [20153.987352] [20153.987353] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20153.992572] bridge stp [20153.995794] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20153.997285] llc [20154.004409] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20154.006870] xfs libcrc32c [20154.013996] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20154.015831] sunrpc intel_rapl_msr [20154.022957] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20154.025660] intel_rapl_common sb_edac [20154.032784] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20154.036182] x86_pkg_temp_thermal coretemp [20154.043305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20154.047051] kvm_intel kvm [20154.055128] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20154.059237] irqbypass i2c_i801 [20154.064974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20154.067678] iTCO_wdt iTCO_vendor_support [20154.074800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20154.077939] i2c_smbus crct10dif_pclmul [20154.085063] Call Trace: [20154.085064] [20154.089068] joydev crc32_pclmul [20154.096191] ? watchdog_timer_fn+0x1b4/0x220 [20154.100021] ipmi_ssif ghash_clmulni_intel [20154.102468] ? __hrtimer_run_queues+0x1d3/0x360 [20154.104476] mei_me [20154.107704] ? hrtimer_interrupt+0x104/0x240 [20154.111964] mei lpc_ich [20154.116054] ? __local_bh_enable+0x50/0x60 [20154.120578] sha512_ssse3 acpi_ipmi [20154.122681] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20154.126941] mfd_core [20154.129472] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20154.133562] wmi ipmi_si [20154.137046] [20154.142264] ipmi_devintf [20154.144535] [20154.144536] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20154.149493] ipmi_msghandler ioatdma [20154.152026] ? queued_spin_lock_slowpath+0x18/0x2e0 [20154.154121] acpi_pad acpi_power_meter [20154.156741] do_raw_spin_lock+0xb5/0xc0 [20154.158835] ixgbe [20154.164155] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20154.167726] igb mdio [20154.172601] try_get_cap_refs+0x159/0x880 [ceph] [20154.176342] crc32c_intel nvme [20154.180178] __ceph_get_caps+0x4a1/0x760 [ceph] [20154.182184] ptp nvme_core [20154.186971] ? __file_remove_privs+0x89/0x110 [20154.189238] i2c_algo_bit pps_core [20154.193849] ? wake_bit_function+0x70/0x70 [20154.196917] dca fuse [20154.201443] ? ceph_write_iter+0x260/0x770 [ceph] [20154.204161] [20154.204161] irq event stamp: 83911860 [20154.208513] ceph_write_iter+0x260/0x770 [ceph] [20154.211910] hardirqs last enabled at (83911859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20154.216005] ? lock_is_held_type+0xaa/0x120 [20154.218272] hardirqs last disabled at (83911860): [] sysvec_apic_timer_interrupt+0xe/0x90 [20154.222972] vfs_write+0x324/0x4f0 [20154.224461] softirqs last enabled at (83911858): [] __do_softirq+0x2be/0x371 [20154.228122] ksys_pwrite64+0x65/0xa0 [20154.232642] softirqs last disabled at (83911853): [] irq_exit_rcu+0xb0/0xd0 [20154.242799] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20154.246976] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20154.256702] do_syscall_64+0x38/0x80 [20154.260098] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20154.268782] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20154.272354] Workqueue: events delayed_work [ceph] [20154.280877] RIP: 0033:0x7f48f0eeb93b [20154.286012] [20154.286012] RIP: 0010:do_raw_spin_lock+0x0/0xc0 [20154.295390] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20154.298960] Code: c0 eb 7e 44 8b 80 28 05 00 00 49 89 e9 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 <66> 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 [20154.306431] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20154.311476] RSP: 0018:ffffc90002117d98 EFLAGS: 00000296 [20154.316171] ORIG_RAX: 0000000000000012 [20154.319744] [20154.319745] RAX: ffffffff813b940d RBX: ffff8881a1d34478 RCX: 000000005553a634 [20154.321235] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20154.325757] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [20154.344495] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20154.363234] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [20154.368451] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20154.373667] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20154.377498] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20154.378990] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20154.386114] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20154.393239] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20154.400367] [20154.407485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20154.466248] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20154.473373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20154.480495] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20154.487621] Call Trace: [20154.490065] [20154.492075] ? watchdog_timer_fn+0x1b4/0x220 [20154.496339] ? __hrtimer_run_queues+0x1d3/0x360 [20154.500882] ? hrtimer_interrupt+0x104/0x240 [20154.505145] ? __local_bh_enable+0x50/0x60 [20154.509254] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20154.514469] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20154.519427] [20154.521523] [20154.523623] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20154.528930] ? igrab+0x1d/0x50 [20154.531988] ? rwlock_bug+0x50/0x50 [20154.535472] igrab+0x1d/0x50 [20154.538349] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20154.543498] delayed_work+0x1d8/0x280 [ceph] [20154.547778] ? process_scheduled_works+0x22a/0x510 [20154.552561] process_scheduled_works+0x2a4/0x510 [20154.557182] worker_thread+0x188/0x340 [20154.560934] ? rescuer_thread+0x310/0x310 [20154.564936] kthread+0xf5/0x130 [20154.568074] ? kthread_complete_and_exit+0x20/0x20 [20154.572878] ret_from_fork+0x31/0x40 [20154.576455] ? kthread_complete_and_exit+0x20/0x20 [20154.581239] ret_from_fork_asm+0x11/0x20 [20154.585176] [20181.810990] watchdog: BUG: soft lockup - CPU#0 stuck for 7681s! [dbench:78683] [20181.818205] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20181.822990] watchdog: BUG: soft lockup - CPU#7 stuck for 16678s! [kworker/7:4:78808] [20181.873662] irq event stamp: 29987742 [20181.873664] hardirqs last enabled at (29987741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20181.881395] Modules linked in: [20181.885053] hardirqs last disabled at (29987742): [] sysvec_apic_timer_interrupt+0xe/0x90 [20181.895210] ceph [20181.898261] softirqs last enabled at (29987694): [] __do_softirq+0x2be/0x371 [20181.907984] libceph [20181.909909] softirqs last disabled at (29987689): [] irq_exit_rcu+0xb0/0xd0 [20181.918593] dns_resolver fscache [20181.920777] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20181.929288] netfs veth [20181.932599] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20181.941541] nft_chain_nat [20181.943986] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20181.951455] xt_MASQUERADE [20181.954161] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20181.959639] nf_nat nf_conntrack [20181.962341] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20181.981081] nf_defrag_ipv6 nf_defrag_ipv4 [20181.984304] [20181.984305] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20181.989520] nft_compat nf_tables [20181.993613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20181.995103] nfnetlink bridge [20182.002227] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20182.005537] stp llc [20182.012671] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20182.015634] xfs [20182.022760] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20182.024943] libcrc32c sunrpc [20182.032066] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20182.033904] intel_rapl_msr intel_rapl_common [20182.041029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20182.043994] sb_edac x86_pkg_temp_thermal [20182.052071] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20182.056420] coretemp kvm_intel [20182.062158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20182.066162] kvm irqbypass [20182.073287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20182.076424] i2c_i801 iTCO_wdt [20182.083548] Call Trace: [20182.083549] [20182.086253] iTCO_vendor_support [20182.093388] ? watchdog_timer_fn+0x1b4/0x220 [20182.096436] i2c_smbus crct10dif_pclmul [20182.098881] ? __hrtimer_run_queues+0x1d3/0x360 [20182.100889] joydev crc32_pclmul [20182.104117] ? hrtimer_interrupt+0x104/0x240 [20182.108387] ipmi_ssif ghash_clmulni_intel [20182.112217] ? __local_bh_enable+0x50/0x60 [20182.116741] mei_me mei [20182.119970] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20182.124229] lpc_ich sha512_ssse3 [20182.128322] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20182.132410] acpi_ipmi [20182.134857] [20182.140072] mfd_core wmi [20182.143392] [20182.148349] ipmi_si ipmi_devintf [20182.150709] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20182.152805] ipmi_msghandler ioatdma [20182.155427] ? queued_spin_lock_slowpath+0x18/0x2e0 [20182.157519] acpi_pad [20182.160833] do_raw_spin_lock+0xb5/0xc0 [20182.166134] acpi_power_meter ixgbe [20182.169705] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20182.174576] igb mdio [20182.176848] try_get_cap_refs+0x159/0x880 [ceph] [20182.180676] crc32c_intel nvme [20182.184165] __ceph_get_caps+0x4a1/0x760 [ceph] [20182.188945] ptp nvme_core [20182.191221] ? __file_remove_privs+0x89/0x110 [20182.195826] i2c_algo_bit pps_core [20182.198877] ? wake_bit_function+0x70/0x70 [20182.203401] dca fuse [20182.206108] ? ceph_write_iter+0x260/0x770 [ceph] [20182.210457] [20182.210458] irq event stamp: 84132784 [20182.213855] ceph_write_iter+0x260/0x770 [ceph] [20182.217945] hardirqs last enabled at (84132783): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20182.220218] ? lock_is_held_type+0xaa/0x120 [20182.224911] hardirqs last disabled at (84132784): [] sysvec_apic_timer_interrupt+0xe/0x90 [20182.226406] vfs_write+0x324/0x4f0 [20182.230060] softirqs last enabled at (84132782): [] __do_softirq+0x2be/0x371 [20182.234589] ksys_pwrite64+0x65/0xa0 [20182.244741] softirqs last disabled at (84132777): [] irq_exit_rcu+0xb0/0xd0 [20182.248921] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20182.258645] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20182.262059] do_syscall_64+0x38/0x80 [20182.270743] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20182.274315] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20182.282824] Workqueue: events delayed_work [ceph] [20182.287956] RIP: 0033:0x7f48f0eeb93b [20182.297333] [20182.300903] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20182.308392] RIP: 0010:lock_acquire+0xf0/0x2b0 [20182.313435] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20182.318133] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20182.321703] ORIG_RAX: 0000000000000012 [20182.323194] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20182.341933] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20182.346282] [20182.346282] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20182.351501] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20182.370237] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20182.374070] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20182.379285] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20182.386410] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20182.387901] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20182.395024] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20182.402150] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20182.409279] [20182.416399] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20182.462281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20182.468017] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20182.475143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20182.482264] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20182.489396] Call Trace: [20182.491845] [20182.493854] ? watchdog_timer_fn+0x1b4/0x220 [20182.498119] ? __hrtimer_run_queues+0x1d3/0x360 [20182.502644] ? hrtimer_interrupt+0x104/0x240 [20182.506915] ? __local_bh_enable+0x50/0x60 [20182.511007] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20182.516222] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20182.521179] [20182.523276] [20182.525374] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20182.530700] ? lock_acquire+0xf0/0x2b0 [20182.534450] ? igrab+0x43/0x50 [20182.537500] ? lock_release+0xc5/0x260 [20182.541246] _raw_spin_lock+0x30/0x40 [20182.544908] ? igrab+0x1d/0x50 [20182.547961] igrab+0x1d/0x50 [20182.550838] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20182.555987] delayed_work+0x1d8/0x280 [ceph] [20182.560269] ? process_scheduled_works+0x22a/0x510 [20182.565050] process_scheduled_works+0x2a4/0x510 [20182.569683] worker_thread+0x188/0x340 [20182.573434] ? rescuer_thread+0x310/0x310 [20182.577436] kthread+0xf5/0x130 [20182.580573] ? kthread_complete_and_exit+0x20/0x20 [20182.585360] ret_from_fork+0x31/0x40 [20182.588937] ? kthread_complete_and_exit+0x20/0x20 [20182.593722] ret_from_fork_asm+0x11/0x20 [20182.597645] [20209.811508] watchdog: BUG: soft lockup - CPU#0 stuck for 7707s! [dbench:78683] [20209.818729] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20209.823508] watchdog: BUG: soft lockup - CPU#7 stuck for 16704s! [kworker/7:4:78808] [20209.874182] irq event stamp: 30045840 [20209.874184] hardirqs last enabled at (30045839): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20209.881916] Modules linked in: [20209.885575] hardirqs last disabled at (30045840): [] sysvec_apic_timer_interrupt+0xe/0x90 [20209.895731] ceph [20209.898781] softirqs last enabled at (30045792): [] __do_softirq+0x2be/0x371 [20209.908507] libceph dns_resolver [20209.910430] softirqs last disabled at (30045787): [] irq_exit_rcu+0xb0/0xd0 [20209.919115] fscache netfs [20209.922426] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20209.930935] veth nft_chain_nat [20209.933640] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20209.942583] xt_MASQUERADE nf_nat [20209.945722] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20209.953201] nf_conntrack nf_defrag_ipv6 [20209.956511] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20209.961988] nf_defrag_ipv4 nft_compat [20209.965906] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20209.984643] nf_tables nfnetlink [20209.988389] [20209.993606] bridge stp [20209.996829] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20209.998322] llc [20210.000765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20210.007903] xfs libcrc32c [20210.009743] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20210.016868] sunrpc intel_rapl_msr [20210.019572] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20210.026696] intel_rapl_common sb_edac [20210.030093] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20210.037217] x86_pkg_temp_thermal [20210.040961] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20210.048085] coretemp [20210.051397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20210.059472] kvm_intel kvm [20210.061743] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20210.067480] irqbypass i2c_i801 [20210.070190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20210.077318] iTCO_wdt iTCO_vendor_support [20210.080456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20210.087581] i2c_smbus [20210.091584] Call Trace: [20210.098709] crct10dif_pclmul joydev [20210.101069] [20210.103509] crc32_pclmul ipmi_ssif [20210.107081] ? watchdog_timer_fn+0x1b4/0x220 [20210.109091] ghash_clmulni_intel mei_me [20210.112576] ? __hrtimer_run_queues+0x1d3/0x360 [20210.116840] mei lpc_ich [20210.120673] ? hrtimer_interrupt+0x104/0x240 [20210.125194] sha512_ssse3 acpi_ipmi [20210.127726] ? __local_bh_enable+0x50/0x60 [20210.131988] mfd_core wmi [20210.135476] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20210.139564] ipmi_si ipmi_devintf [20210.142188] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20210.147406] ipmi_msghandler [20210.150719] [20210.155674] ioatdma acpi_pad [20210.158554] [20210.160649] acpi_power_meter ixgbe [20210.163616] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20210.165710] igb mdio [20210.169199] ? queued_spin_lock_slowpath+0x16/0x2e0 [20210.174500] crc32c_intel nvme [20210.176772] do_raw_spin_lock+0xb5/0xc0 [20210.181642] ptp [20210.184694] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20210.188522] nvme_core i2c_algo_bit [20210.190363] try_get_cap_refs+0x159/0x880 [ceph] [20210.195145] pps_core [20210.198633] __ceph_get_caps+0x4a1/0x760 [ceph] [20210.203239] dca fuse [20210.205514] ? __file_remove_privs+0x89/0x110 [20210.210034] [20210.210035] irq event stamp: 84353552 [20210.212306] ? wake_bit_function+0x70/0x70 [20210.216656] hardirqs last enabled at (84353551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20210.218150] ? ceph_write_iter+0x260/0x770 [ceph] [20210.221803] hardirqs last disabled at (84353552): [] sysvec_apic_timer_interrupt+0xe/0x90 [20210.225912] ceph_write_iter+0x260/0x770 [ceph] [20210.236069] softirqs last enabled at (84353550): [] __do_softirq+0x2be/0x371 [20210.240770] ? lock_is_held_type+0xaa/0x120 [20210.250489] softirqs last disabled at (84353545): [] irq_exit_rcu+0xb0/0xd0 [20210.255016] vfs_write+0x324/0x4f0 [20210.263698] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20210.267880] ksys_pwrite64+0x65/0xa0 [20210.276386] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20210.279784] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20210.289162] Workqueue: events delayed_work [ceph] [20210.292734] do_syscall_64+0x38/0x80 [20210.300204] [20210.300205] RIP: 0010:lock_acquire+0xf0/0x2b0 [20210.305334] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20210.310032] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20210.313603] RIP: 0033:0x7f48f0eeb93b [20210.315093] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20210.319444] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20210.324488] [20210.324489] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20210.343224] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20210.346796] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20210.352013] ORIG_RAX: 0000000000000012 [20210.370750] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20210.372242] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20210.379365] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20210.384584] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20210.391708] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20210.395538] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20210.402664] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20210.409786] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20210.416912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20210.424034] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20210.431159] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20210.438289] [20210.446362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20210.482778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20210.482888] rcu: INFO: rcu_preempt self-detected stall on CPU [20210.489915] Call Trace: [20210.489917] [20210.495649] rcu: 0-....: (8086858 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=4030295 [20210.498093] ? watchdog_timer_fn+0x1b4/0x220 [20210.500104] rcu: (t=8273724 jiffies g=1309609 q=296409 ncpus=8) [20210.510435] ? __hrtimer_run_queues+0x1d3/0x360 [20210.514699] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20210.520700] ? hrtimer_interrupt+0x104/0x240 [20210.525219] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20210.534164] ? __local_bh_enable+0x50/0x60 [20210.538428] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20210.545901] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20210.549991] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20210.555469] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20210.560684] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20210.579423] [20210.584381] [20210.584382] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20210.589597] [20210.591693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20210.593186] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20210.600308] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20210.602411] ? lock_acquire+0xf0/0x2b0 [20210.609532] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20210.614837] ? igrab+0x43/0x50 [20210.621958] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20210.625703] ? lock_release+0xc5/0x260 [20210.632827] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20210.635879] _raw_spin_lock+0x30/0x40 [20210.643002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20210.646747] ? igrab+0x1d/0x50 [20210.654824] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20210.658482] igrab+0x1d/0x50 [20210.664217] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20210.667272] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20210.674392] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20210.677275] delayed_work+0x1d8/0x280 [ceph] [20210.684394] Call Trace: [20210.684395] [20210.689528] ? process_scheduled_works+0x22a/0x510 [20210.696651] ? rcu_dump_cpu_stacks+0xd1/0x180 [20210.700914] process_scheduled_works+0x2a4/0x510 [20210.703360] ? rcu_sched_clock_irq+0x633/0x1390 [20210.705373] worker_thread+0x188/0x340 [20210.710154] ? lock_release+0xc5/0x260 [20210.714506] ? rescuer_thread+0x310/0x310 [20210.719114] ? lock_is_held_type+0xaa/0x120 [20210.723638] kthread+0xf5/0x130 [20210.727384] ? lock_is_held_type+0xaa/0x120 [20210.731128] ? kthread_complete_and_exit+0x20/0x20 [20210.735131] ? tick_sched_do_timer+0xa0/0xa0 [20210.739310] ret_from_fork+0x31/0x40 [20210.742447] ? rcu_read_lock_sched_held+0x56/0x90 [20210.746624] ? kthread_complete_and_exit+0x20/0x20 [20210.751409] ? tick_sched_do_timer+0xa0/0xa0 [20210.755671] ret_from_fork_asm+0x11/0x20 [20210.759241] ? update_process_times+0x5f/0x90 [20210.763946] [20210.768723] ? tick_sched_handle.isra.16+0x1e/0x50 [20210.788224] ? tick_sched_timer+0x6f/0x80 [20210.792230] ? __hrtimer_run_queues+0x1d3/0x360 [20210.796755] ? hrtimer_interrupt+0x104/0x240 [20210.801015] ? __local_bh_enable+0x50/0x60 [20210.805110] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20210.810334] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20210.815291] [20210.817388] [20210.819485] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20210.824795] ? queued_spin_lock_slowpath+0x18/0x2e0 [20210.829670] do_raw_spin_lock+0xb5/0xc0 [20210.833499] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20210.838295] try_get_cap_refs+0x159/0x880 [ceph] [20210.842925] __ceph_get_caps+0x4a1/0x760 [ceph] [20210.847465] ? __file_remove_privs+0x89/0x110 [20210.851820] ? wake_bit_function+0x70/0x70 [20210.855915] ? ceph_write_iter+0x260/0x770 [ceph] [20210.860627] ceph_write_iter+0x260/0x770 [ceph] [20210.865163] ? lock_is_held_type+0xaa/0x120 [20210.869347] vfs_write+0x324/0x4f0 [20210.872756] ksys_pwrite64+0x65/0xa0 [20210.876331] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20210.881464] do_syscall_64+0x38/0x80 [20210.885042] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20210.890086] RIP: 0033:0x7f48f0eeb93b [20210.893655] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20210.912393] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [20210.919951] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20210.927074] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20210.934198] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20210.941322] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20210.948448] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20210.955579] [20210.957765] Sending NMI from CPU 0 to CPUs 7: [20210.962116] NMI backtrace for cpu 7 [20210.962117] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20210.962119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20210.962120] Workqueue: events delayed_work [ceph] [20210.962134] RIP: 0010:__lock_acquire+0x653/0x1e20 [20210.962136] Code: 28 8b 7c 24 40 c1 e0 0d 66 0b 44 24 18 98 2b 44 24 30 41 31 c7 44 89 f8 44 29 fa 41 01 ff c1 c0 06 31 d0 29 c7 89 c2 44 01 f8 c2 08 41 89 fc 41 31 d4 44 89 e2 45 29 e7 41 01 c4 c1 c2 10 44 [20210.962137] RSP: 0018:ffffc90002117c60 EFLAGS: 00000817 [20210.962138] RAX: 000000004d0e10b2 RBX: ffff88810debaa80 RCX: 0000000000000003 [20210.962139] RDX: 0000000096722896 RSI: 0000000000000000 RDI: 00000000f3668bb7 [20210.962139] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [20210.962140] R10: 0000000000000000 R11: 0000000000000001 R12: b656a521d3820f2c [20210.962141] R13: 0000000000000003 R14: 0000000000000000 R15: 00000000b69be81c [20210.962141] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20210.962142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20210.962143] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20210.962144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20210.962144] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20210.962145] Call Trace: [20210.962146] [20210.962147] ? nmi_cpu_backtrace+0xad/0x120 [20210.962150] ? nmi_cpu_backtrace_handler+0x11/0x20 [20210.962151] ? nmi_handle+0xd4/0x260 [20210.962155] ? default_do_nmi+0x49/0xf0 [20210.962157] ? exc_nmi+0x11e/0x150 [20210.962158] ? end_repeat_nmi+0x16/0x67 [20210.962162] ? __lock_acquire+0x653/0x1e20 [20210.962163] ? __lock_acquire+0x653/0x1e20 [20210.962165] ? __lock_acquire+0x653/0x1e20 [20210.962166] [20210.962166] [20210.962168] lock_acquire+0xcb/0x2b0 [20210.962170] ? igrab+0x1d/0x50 [20210.962171] ? igrab+0x43/0x50 [20210.962172] ? lock_release+0xc5/0x260 [20210.962174] _raw_spin_lock+0x30/0x40 [20210.962176] ? igrab+0x1d/0x50 [20210.962177] igrab+0x1d/0x50 [20210.962178] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20210.962192] delayed_work+0x1d8/0x280 [ceph] [20210.962205] ? process_scheduled_works+0x22a/0x510 [20210.962207] process_scheduled_works+0x2a4/0x510 [20210.962211] worker_thread+0x188/0x340 [20210.962213] ? rescuer_thread+0x310/0x310 [20210.962216] kthread+0xf5/0x130 [20210.962218] ? kthread_complete_and_exit+0x20/0x20 [20210.962220] ret_from_fork+0x31/0x40 [20210.962222] ? kthread_complete_and_exit+0x20/0x20 [20210.962224] ret_from_fork_asm+0x11/0x20 [20210.962228] [20237.812027] watchdog: BUG: soft lockup - CPU#0 stuck for 7733s! [dbench:78683] [20237.819239] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20237.824027] watchdog: BUG: soft lockup - CPU#7 stuck for 16730s! [kworker/7:4:78808] [20237.874682] irq event stamp: 30103016 [20237.874684] hardirqs last enabled at (30103015): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20237.882420] Modules linked in: [20237.886078] hardirqs last disabled at (30103016): [] sysvec_apic_timer_interrupt+0xe/0x90 [20237.896235] ceph [20237.899286] softirqs last enabled at (30103006): [] __do_softirq+0x2be/0x371 [20237.909012] libceph [20237.910936] softirqs last disabled at (30103001): [] irq_exit_rcu+0xb0/0xd0 [20237.919619] dns_resolver fscache [20237.921804] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20237.930314] netfs [20237.933623] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20237.942568] veth nft_chain_nat [20237.944578] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20237.952049] xt_MASQUERADE nf_nat [20237.955188] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20237.960666] nf_conntrack nf_defrag_ipv6 [20237.963976] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20237.982713] nf_defrag_ipv4 nft_compat [20237.986630] [20237.986631] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20237.991849] nf_tables nfnetlink [20237.995592] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20237.997084] bridge stp [20238.004207] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20238.007430] llc xfs [20238.014556] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20238.017000] libcrc32c sunrpc [20238.024123] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20238.026307] intel_rapl_msr intel_rapl_common [20238.033431] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20238.036395] sb_edac x86_pkg_temp_thermal [20238.043538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20238.047888] coretemp kvm_intel [20238.055964] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20238.059970] kvm irqbypass [20238.065708] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20238.068845] i2c_i801 iTCO_wdt [20238.075968] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20238.078673] iTCO_vendor_support [20238.085797] Call Trace: [20238.085798] [20238.088848] i2c_smbus [20238.095973] ? watchdog_timer_fn+0x1b4/0x220 [20238.099195] crct10dif_pclmul joydev [20238.101641] ? __hrtimer_run_queues+0x1d3/0x360 [20238.103650] crc32_pclmul ipmi_ssif [20238.106010] ? hrtimer_interrupt+0x104/0x240 [20238.110270] ghash_clmulni_intel mei_me [20238.113841] ? __local_bh_enable+0x50/0x60 [20238.118365] mei [20238.121853] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20238.126115] lpc_ich sha512_ssse3 [20238.129947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20238.134037] acpi_ipmi [20238.135873] [20238.141092] mfd_core wmi [20238.144402] [20238.149376] ipmi_si ipmi_devintf [20238.151733] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20238.153831] ipmi_msghandler ioatdma [20238.156451] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20238.158545] acpi_pad acpi_power_meter [20238.161858] do_raw_spin_lock+0xb5/0xc0 [20238.167162] ixgbe igb [20238.170731] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20238.175602] mdio crc32c_intel [20238.179349] try_get_cap_refs+0x159/0x880 [ceph] [20238.183177] nvme ptp [20238.185540] __ceph_get_caps+0x4a1/0x760 [ceph] [20238.190317] nvme_core i2c_algo_bit [20238.193373] ? __file_remove_privs+0x89/0x110 [20238.197979] pps_core dca [20238.200252] ? wake_bit_function+0x70/0x70 [20238.204774] fuse [20238.208261] ? ceph_write_iter+0x260/0x770 [ceph] [20238.212610] [20238.212611] irq event stamp: 84571054 [20238.215228] ceph_write_iter+0x260/0x770 [ceph] [20238.219317] hardirqs last enabled at (84571053): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20238.221244] ? lock_is_held_type+0xaa/0x120 [20238.225939] hardirqs last disabled at (84571054): [] sysvec_apic_timer_interrupt+0xe/0x90 [20238.227432] vfs_write+0x324/0x4f0 [20238.231087] softirqs last enabled at (84571052): [] __do_softirq+0x2be/0x371 [20238.235615] ksys_pwrite64+0x65/0xa0 [20238.245767] softirqs last disabled at (84571047): [] irq_exit_rcu+0xb0/0xd0 [20238.249945] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20238.259671] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20238.263068] do_syscall_64+0x38/0x80 [20238.271752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20238.275324] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20238.283834] Workqueue: events delayed_work [ceph] [20238.288965] RIP: 0033:0x7f48f0eeb93b [20238.298341] [20238.298342] RIP: 0010:lock_acquire+0xf0/0x2b0 [20238.301912] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20238.309383] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20238.314429] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20238.319126] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20238.322711] ORIG_RAX: 0000000000000012 [20238.324204] [20238.324205] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20238.328553] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20238.347291] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20238.366028] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20238.371247] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20238.376464] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20238.380295] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20238.381786] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20238.388909] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20238.396035] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20238.403159] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20238.410287] [20238.417407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20238.469043] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20238.476167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20238.483292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20238.490432] Call Trace: [20238.492878] [20238.494888] ? watchdog_timer_fn+0x1b4/0x220 [20238.499153] ? __hrtimer_run_queues+0x1d3/0x360 [20238.503680] ? hrtimer_interrupt+0x104/0x240 [20238.507950] ? __local_bh_enable+0x50/0x60 [20238.512043] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20238.517266] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20238.522225] [20238.524320] [20238.526433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20238.531743] ? lock_acquire+0xf0/0x2b0 [20238.535494] ? igrab+0x43/0x50 [20238.538544] ? lock_release+0xc5/0x260 [20238.542288] _raw_spin_lock+0x30/0x40 [20238.545944] ? igrab+0x1d/0x50 [20238.548997] igrab+0x1d/0x50 [20238.551875] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20238.557023] delayed_work+0x1d8/0x280 [ceph] [20238.561304] ? process_scheduled_works+0x22a/0x510 [20238.566086] process_scheduled_works+0x2a4/0x510 [20238.570718] worker_thread+0x188/0x340 [20238.574467] ? rescuer_thread+0x310/0x310 [20238.578472] kthread+0xf5/0x130 [20238.581608] ? kthread_complete_and_exit+0x20/0x20 [20238.586394] ret_from_fork+0x31/0x40 [20238.589971] ? kthread_complete_and_exit+0x20/0x20 [20238.594756] ret_from_fork_asm+0x11/0x20 [20238.598680] [20265.812549] watchdog: BUG: soft lockup - CPU#0 stuck for 7760s! [dbench:78683] [20265.819771] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20265.824549] watchdog: BUG: soft lockup - CPU#7 stuck for 16757s! [kworker/7:4:78808] [20265.875226] irq event stamp: 30161134 [20265.882961] Modules linked in: ceph libceph dns_resolver fscache netfs [20265.886617] hardirqs last enabled at (30161133): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20265.893136] veth nft_chain_nat xt_MASQUERADE nf_nat [20265.903293] hardirqs last disabled at (30161134): [] sysvec_apic_timer_interrupt+0xe/0x90 [20265.903295] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [20265.917973] softirqs last enabled at (30161048): [] __do_softirq+0x2be/0x371 [20265.926484] stp llc xfs libcrc32c sunrpc [20265.935168] softirqs last disabled at (30161043): [] irq_exit_rcu+0xb0/0xd0 [20265.939174] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [20265.947685] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20265.947685] coretemp [20265.947687] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20265.954549] kvm_intel [20265.963493] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20265.965762] kvm irqbypass [20265.973236] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20265.975591] i2c_i801 iTCO_wdt [20265.981070] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20265.983772] iTCO_vendor_support i2c_smbus [20266.002513] [20266.005560] crct10dif_pclmul [20266.010779] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20266.014869] joydev crc32_pclmul [20266.016359] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20266.019324] ipmi_ssif ghash_clmulni_intel [20266.026449] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20266.029673] mei_me mei [20266.036798] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20266.040886] lpc_ich sha512_ssse3 [20266.048010] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20266.050454] acpi_ipmi mfd_core [20266.057580] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20266.060890] wmi ipmi_si [20266.068014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20266.071152] ipmi_devintf ipmi_msghandler [20266.079246] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20266.081778] ioatdma acpi_pad [20266.087516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20266.091520] acpi_power_meter ixgbe [20266.098643] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20266.101608] igb mdio [20266.108731] Call Trace: [20266.108734] [20266.112226] crc32c_intel nvme [20266.119350] ? watchdog_timer_fn+0x1b4/0x220 [20266.121619] ptp [20266.124065] ? __hrtimer_run_queues+0x1d3/0x360 [20266.126074] nvme_core [20266.129128] ? hrtimer_interrupt+0x104/0x240 [20266.133389] i2c_algo_bit [20266.135240] ? __local_bh_enable+0x50/0x60 [20266.139767] pps_core [20266.142126] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20266.146388] dca fuse [20266.149007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20266.153096] [20266.153096] irq event stamp: 84792242 [20266.155368] [20266.160585] hardirqs last enabled at (84792241): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20266.162856] [20266.162857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20266.167812] hardirqs last disabled at (84792242): [] sysvec_apic_timer_interrupt+0xe/0x90 [20266.169307] ? queued_spin_lock_slowpath+0x16/0x2e0 [20266.172961] softirqs last enabled at (84792240): [] __do_softirq+0x2be/0x371 [20266.175061] do_raw_spin_lock+0xb5/0xc0 [20266.185231] softirqs last disabled at (84792235): [] irq_exit_rcu+0xb0/0xd0 [20266.187332] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20266.192635] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20266.202362] try_get_cap_refs+0x159/0x880 [ceph] [20266.207239] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20266.215948] __ceph_get_caps+0x4a1/0x760 [ceph] [20266.219770] Workqueue: events delayed_work [ceph] [20266.228286] ? __file_remove_privs+0x89/0x110 [20266.233064] [20266.233065] RIP: 0010:lock_acquire+0xf0/0x2b0 [20266.242443] ? wake_bit_function+0x70/0x70 [20266.247053] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20266.254526] ? ceph_write_iter+0x260/0x770 [ceph] [20266.259049] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20266.263745] ceph_write_iter+0x260/0x770 [ceph] [20266.268095] [20266.268096] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20266.269591] ? lock_is_held_type+0xaa/0x120 [20266.273951] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20266.278041] vfs_write+0x324/0x4f0 [20266.296776] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20266.301478] ksys_pwrite64+0x65/0xa0 [20266.306690] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20266.311215] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20266.312723] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20266.319849] do_syscall_64+0x38/0x80 [20266.324025] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20266.331150] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20266.334546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20266.341668] RIP: 0033:0x7f48f0eeb93b [20266.345241] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20266.352363] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20266.357496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20266.364618] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20266.368190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20266.376267] ORIG_RAX: 0000000000000012 [20266.381311] Call Trace: [20266.381312] [20266.387049] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20266.390620] ? watchdog_timer_fn+0x1b4/0x220 [20266.397743] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20266.416483] ? __hrtimer_run_queues+0x1d3/0x360 [20266.423606] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20266.428826] ? hrtimer_interrupt+0x104/0x240 [20266.435954] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20266.439787] ? __local_bh_enable+0x50/0x60 [20266.442235] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20266.444253] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20266.451380] [20266.455639] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20266.509373] [20266.511469] [20266.513569] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20266.518875] ? lock_acquire+0xf0/0x2b0 [20266.522627] ? igrab+0x43/0x50 [20266.525685] ? lock_release+0xc5/0x260 [20266.529430] _raw_spin_lock+0x30/0x40 [20266.533086] ? igrab+0x1d/0x50 [20266.536135] igrab+0x1d/0x50 [20266.539017] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20266.544165] delayed_work+0x1d8/0x280 [ceph] [20266.548446] ? process_scheduled_works+0x22a/0x510 [20266.553237] process_scheduled_works+0x2a4/0x510 [20266.557853] worker_thread+0x188/0x340 [20266.561601] ? rescuer_thread+0x310/0x310 [20266.565603] kthread+0xf5/0x130 [20266.568742] ? kthread_complete_and_exit+0x20/0x20 [20266.573525] ret_from_fork+0x31/0x40 [20266.577098] ? kthread_complete_and_exit+0x20/0x20 [20266.581880] ret_from_fork_asm+0x11/0x20 [20266.585804] [20293.813073] watchdog: BUG: soft lockup - CPU#0 stuck for 7786s! [dbench:78683] [20293.820287] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20293.825073] watchdog: BUG: soft lockup - CPU#7 stuck for 16783s! [kworker/7:4:78808] [20293.875727] irq event stamp: 30219256 [20293.883475] Modules linked in: ceph libceph [20293.887133] hardirqs last enabled at (30219255): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20293.891309] dns_resolver fscache netfs veth [20293.901468] hardirqs last disabled at (30219256): [] sysvec_apic_timer_interrupt+0xe/0x90 [20293.905745] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20293.915471] softirqs last enabled at (30219240): [] __do_softirq+0x2be/0x371 [20293.921123] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [20293.929806] softirqs last disabled at (30219235): [] irq_exit_rcu+0xb0/0xd0 [20293.935718] nfnetlink bridge stp [20293.944229] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20293.947538] llc xfs [20293.956483] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20293.958666] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [20293.966140] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20293.971963] sb_edac x86_pkg_temp_thermal [20293.977439] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20293.977440] coretemp [20293.977443] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20293.981444] kvm_intel kvm irqbypass [20294.000200] [20294.000201] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20294.002469] i2c_i801 iTCO_wdt [20294.007688] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20294.011258] iTCO_vendor_support i2c_smbus [20294.012751] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20294.019871] crct10dif_pclmul joydev [20294.022922] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20294.030047] crc32_pclmul ipmi_ssif [20294.034139] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20294.041261] ghash_clmulni_intel mei_me [20294.044832] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20294.051957] mei lpc_ich [20294.055443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20294.062566] sha512_ssse3 acpi_ipmi [20294.066395] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20294.074473] mfd_core wmi [20294.077003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20294.082750] ipmi_si ipmi_devintf [20294.086235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20294.093359] ipmi_msghandler [20294.095975] Call Trace: [20294.095977] [20294.103099] ioatdma [20294.106413] ? watchdog_timer_fn+0x1b4/0x220 [20294.113536] acpi_pad acpi_power_meter [20294.116412] ? __hrtimer_run_queues+0x1d3/0x360 [20294.118856] ixgbe igb [20294.120870] ? hrtimer_interrupt+0x104/0x240 [20294.123050] mdio crc32c_intel [20294.127316] ? __local_bh_enable+0x50/0x60 [20294.131060] nvme ptp [20294.135586] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20294.137941] nvme_core i2c_algo_bit [20294.142205] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20294.145255] pps_core dca [20294.149347] [20294.151616] fuse [20294.156835] [20294.160318] [20294.160318] irq event stamp: 85013714 [20294.165275] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20294.167893] hardirqs last enabled at (85013713): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20294.169994] ? queued_spin_lock_slowpath+0x16/0x2e0 [20294.171915] hardirqs last disabled at (85013714): [] sysvec_apic_timer_interrupt+0xe/0x90 [20294.174014] do_raw_spin_lock+0xb5/0xc0 [20294.175501] softirqs last enabled at (85013712): [] __do_softirq+0x2be/0x371 [20294.179162] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20294.184473] softirqs last disabled at (85013707): [] irq_exit_rcu+0xb0/0xd0 [20294.194634] try_get_cap_refs+0x159/0x880 [ceph] [20294.199500] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20294.209231] __ceph_get_caps+0x4a1/0x760 [ceph] [20294.213055] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20294.221758] ? __file_remove_privs+0x89/0x110 [20294.226542] Workqueue: events delayed_work [ceph] [20294.235053] ? wake_bit_function+0x70/0x70 [20294.239664] [20294.239665] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [20294.249043] ? ceph_write_iter+0x260/0x770 [ceph] [20294.253566] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [20294.261036] ceph_write_iter+0x260/0x770 [ceph] [20294.265387] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [20294.270089] ? lock_is_held_type+0xaa/0x120 [20294.274175] [20294.274176] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [20294.275668] vfs_write+0x324/0x4f0 [20294.280277] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [20294.284977] ksys_pwrite64+0x65/0xa0 [20294.303712] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [20294.308236] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20294.313454] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20294.317648] do_syscall_64+0x38/0x80 [20294.319137] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20294.326263] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20294.329659] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20294.336784] RIP: 0033:0x7f48f0eeb93b [20294.340355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20294.347479] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20294.352610] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20294.359734] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20294.363304] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20294.370428] ORIG_RAX: 0000000000000012 [20294.375476] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20294.383551] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20294.387122] Call Trace: [20294.387123] [20294.392859] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20294.411598] ? watchdog_timer_fn+0x1b4/0x220 [20294.418720] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20294.423938] ? __hrtimer_run_queues+0x1d3/0x360 [20294.431061] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20294.434895] ? hrtimer_interrupt+0x104/0x240 [20294.442015] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20294.449141] ? __local_bh_enable+0x50/0x60 [20294.451589] [20294.453598] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20294.506637] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20294.511594] [20294.513690] [20294.515789] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20294.521098] ? do_raw_spin_lock+0x42/0xc0 [20294.525107] igrab+0x1d/0x50 [20294.527983] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20294.533133] delayed_work+0x1d8/0x280 [ceph] [20294.537414] ? process_scheduled_works+0x22a/0x510 [20294.542197] process_scheduled_works+0x2a4/0x510 [20294.546812] worker_thread+0x188/0x340 [20294.550561] ? rescuer_thread+0x310/0x310 [20294.554563] kthread+0xf5/0x130 [20294.557702] ? kthread_complete_and_exit+0x20/0x20 [20294.562485] ret_from_fork+0x31/0x40 [20294.566056] ? kthread_complete_and_exit+0x20/0x20 [20294.570841] ret_from_fork_asm+0x11/0x20 [20294.574778] [20321.813596] watchdog: BUG: soft lockup - CPU#0 stuck for 7812s! [dbench:78683] [20321.820816] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20321.825596] watchdog: BUG: soft lockup - CPU#7 stuck for 16809s! [kworker/7:4:78808] [20321.876273] irq event stamp: 30277324 [20321.884006] Modules linked in: ceph libceph dns_resolver fscache netfs veth [20321.887663] hardirqs last enabled at (30277323): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20321.894622] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [20321.904779] hardirqs last disabled at (30277324): [] sysvec_apic_timer_interrupt+0xe/0x90 [20321.913030] nft_compat nf_tables nfnetlink bridge stp llc xfs [20321.922754] softirqs last enabled at (30277304): [] __do_softirq+0x2be/0x371 [20321.928579] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [20321.937272] softirqs last disabled at (30277299): [] irq_exit_rcu+0xb0/0xd0 [20321.945618] coretemp kvm_intel kvm irqbypass i2c_i801 [20321.954130] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20321.959271] iTCO_wdt iTCO_vendor_support [20321.968213] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20321.972216] i2c_smbus crct10dif_pclmul joydev crc32_pclmul [20321.979686] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20321.979688] ipmi_ssif ghash_clmulni_intel mei_me mei [20321.990729] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20321.995773] lpc_ich sha512_ssse3 acpi_ipmi mfd_core [20322.014510] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20322.019466] wmi ipmi_si ipmi_devintf [20322.024685] [20322.024686] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20322.028341] ipmi_msghandler ioatdma [20322.029832] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20322.036958] acpi_pad acpi_power_meter [20322.040528] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20322.047652] ixgbe [20322.051397] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20322.058519] igb mdio [20322.060532] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20322.067655] crc32c_intel nvme [20322.069926] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20322.077051] ptp [20322.080102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20322.088180] nvme_core [20322.090016] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20322.095754] i2c_algo_bit pps_core [20322.098110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20322.105233] dca fuse [20322.108632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20322.115755] [20322.115756] irq event stamp: 85234774 [20322.118026] Call Trace: [20322.118028] [20322.125150] hardirqs last enabled at (85234773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20322.126644] ? watchdog_timer_fn+0x1b4/0x220 [20322.130299] hardirqs last disabled at (85234774): [] sysvec_apic_timer_interrupt+0xe/0x90 [20322.132745] ? __hrtimer_run_queues+0x1d3/0x360 [20322.134753] softirqs last enabled at (85234772): [] __do_softirq+0x2be/0x371 [20322.144915] ? hrtimer_interrupt+0x104/0x240 [20322.149175] softirqs last disabled at (85234767): [] irq_exit_rcu+0xb0/0xd0 [20322.158900] ? __local_bh_enable+0x50/0x60 [20322.163424] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20322.172111] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20322.176373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20322.184883] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20322.188991] Workqueue: events delayed_work [ceph] [20322.198368] [20322.203586] [20322.203587] RIP: 0010:lock_acquire+0xf0/0x2b0 [20322.211057] [20322.216015] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20322.220711] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20322.222808] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20322.224303] ? queued_spin_lock_slowpath+0x16/0x2e0 [20322.228650] [20322.228651] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20322.230749] do_raw_spin_lock+0xb5/0xc0 [20322.249485] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20322.254792] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20322.260007] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20322.264881] try_get_cap_refs+0x159/0x880 [ceph] [20322.266368] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20322.273497] __ceph_get_caps+0x4a1/0x760 [ceph] [20322.277324] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20322.284451] ? __file_remove_privs+0x89/0x110 [20322.289231] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20322.296356] ? wake_bit_function+0x70/0x70 [20322.300965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20322.308092] ? ceph_write_iter+0x260/0x770 [ceph] [20322.312613] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20322.319739] ceph_write_iter+0x260/0x770 [ceph] [20322.324091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20322.332172] ? lock_is_held_type+0xaa/0x120 [20322.336256] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20322.342002] vfs_write+0x324/0x4f0 [20322.346701] Call Trace: [20322.346703] [20322.353828] ksys_pwrite64+0x65/0xa0 [20322.358351] ? watchdog_timer_fn+0x1b4/0x220 [20322.365473] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20322.369651] ? __hrtimer_run_queues+0x1d3/0x360 [20322.376776] do_syscall_64+0x38/0x80 [20322.380176] ? hrtimer_interrupt+0x104/0x240 [20322.382619] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20322.384627] ? __local_bh_enable+0x50/0x60 [20322.388200] RIP: 0033:0x7f48f0eeb93b [20322.392465] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20322.397592] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20322.402117] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20322.405688] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20322.409951] [20322.415001] ORIG_RAX: 0000000000000012 [20322.419096] [20322.419098] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20322.422665] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20322.427888] ? lock_acquire+0xf0/0x2b0 [20322.446622] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20322.451579] ? igrab+0x43/0x50 [20322.456795] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20322.458895] ? lock_release+0xc5/0x260 [20322.462724] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20322.464822] _raw_spin_lock+0x30/0x40 [20322.470126] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20322.477250] ? igrab+0x1d/0x50 [20322.481008] [20322.488126] igrab+0x1d/0x50 [20322.528065] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20322.533216] delayed_work+0x1d8/0x280 [ceph] [20322.537497] ? process_scheduled_works+0x22a/0x510 [20322.542286] process_scheduled_works+0x2a4/0x510 [20322.546900] worker_thread+0x188/0x340 [20322.550652] ? rescuer_thread+0x310/0x310 [20322.554663] kthread+0xf5/0x130 [20322.557809] ? kthread_complete_and_exit+0x20/0x20 [20322.562593] ret_from_fork+0x31/0x40 [20322.566173] ? kthread_complete_and_exit+0x20/0x20 [20322.570957] ret_from_fork_asm+0x11/0x20 [20322.574888] [20349.814119] watchdog: BUG: soft lockup - CPU#0 stuck for 7838s! [dbench:78683] [20349.821330] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20349.826119] watchdog: BUG: soft lockup - CPU#7 stuck for 16835s! [kworker/7:4:78808] [20349.876771] irq event stamp: 30335392 [20349.876772] hardirqs last enabled at (30335391): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20349.884518] Modules linked in: [20349.888176] hardirqs last disabled at (30335392): [] sysvec_apic_timer_interrupt+0xe/0x90 [20349.898333] ceph [20349.901385] softirqs last enabled at (30335344): [] __do_softirq+0x2be/0x371 [20349.911109] libceph [20349.913032] softirqs last disabled at (30335339): [] irq_exit_rcu+0xb0/0xd0 [20349.921716] dns_resolver fscache [20349.923902] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20349.932411] netfs veth [20349.935723] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20349.944667] nft_chain_nat [20349.947112] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20349.954581] xt_MASQUERADE nf_nat [20349.957286] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20349.962763] nf_conntrack nf_defrag_ipv6 [20349.966074] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20349.984812] nf_defrag_ipv4 nft_compat [20349.988730] [20349.988731] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20349.993947] nf_tables nfnetlink [20349.997689] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20349.999180] bridge stp [20350.006305] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20350.009529] llc xfs [20350.016652] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20350.019096] libcrc32c sunrpc [20350.026222] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20350.028406] intel_rapl_msr [20350.035530] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20350.038512] intel_rapl_common sb_edac [20350.045636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20350.048426] x86_pkg_temp_thermal coretemp [20350.056517] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20350.060264] kvm_intel kvm [20350.066001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20350.070092] irqbypass i2c_i801 [20350.077218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20350.079921] iTCO_wdt iTCO_vendor_support [20350.087046] Call Trace: [20350.087047] [20350.090183] i2c_smbus crct10dif_pclmul [20350.097308] ? watchdog_timer_fn+0x1b4/0x220 [20350.101310] joydev [20350.103756] ? __hrtimer_run_queues+0x1d3/0x360 [20350.105766] crc32_pclmul ipmi_ssif [20350.109599] ? hrtimer_interrupt+0x104/0x240 [20350.113861] ghash_clmulni_intel mei_me [20350.115959] ? __local_bh_enable+0x50/0x60 [20350.120482] mei [20350.123968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20350.128230] lpc_ich sha512_ssse3 [20350.132061] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20350.136151] acpi_ipmi mfd_core [20350.137989] [20350.143207] wmi ipmi_si [20350.146519] [20350.151476] ipmi_devintf [20350.154613] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20350.156708] ipmi_msghandler ioatdma [20350.159245] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20350.161336] acpi_pad acpi_power_meter [20350.163956] do_raw_spin_lock+0xb5/0xc0 [20350.169260] ixgbe igb [20350.172830] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20350.177701] mdio crc32c_intel [20350.181448] try_get_cap_refs+0x159/0x880 [ceph] [20350.185274] nvme ptp [20350.187636] __ceph_get_caps+0x4a1/0x760 [ceph] [20350.192417] nvme_core i2c_algo_bit [20350.195472] ? __file_remove_privs+0x89/0x110 [20350.200078] pps_core dca [20350.202351] ? wake_bit_function+0x70/0x70 [20350.206874] fuse [20350.210359] ? ceph_write_iter+0x260/0x770 [ceph] [20350.214707] irq event stamp: 85455754 [20350.217326] ceph_write_iter+0x260/0x770 [ceph] [20350.221415] hardirqs last enabled at (85455753): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20350.223342] ? lock_is_held_type+0xaa/0x120 [20350.228037] hardirqs last disabled at (85455754): [] sysvec_apic_timer_interrupt+0xe/0x90 [20350.231697] vfs_write+0x324/0x4f0 [20350.236218] softirqs last enabled at (85455752): [] __do_softirq+0x2be/0x371 [20350.246382] ksys_pwrite64+0x65/0xa0 [20350.250552] softirqs last disabled at (85455747): [] irq_exit_rcu+0xb0/0xd0 [20350.260277] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20350.263674] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20350.272361] do_syscall_64+0x38/0x80 [20350.275931] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20350.284444] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20350.289573] Workqueue: events delayed_work [ceph] [20350.298949] RIP: 0033:0x7f48f0eeb93b [20350.302520] [20350.309992] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20350.315034] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [20350.319732] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20350.323304] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [20350.324803] ORIG_RAX: 0000000000000012 [20350.343539] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [20350.349451] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20350.354668] [20350.373406] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20350.377237] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [20350.382455] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20350.389579] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [20350.391068] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20350.398194] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [20350.405317] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20350.412441] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20350.419570] [20350.426689] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20350.457370] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20350.465449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20350.471184] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20350.478308] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20350.485433] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20350.492557] Call Trace: [20350.495004] [20350.497014] ? watchdog_timer_fn+0x1b4/0x220 [20350.501278] ? __hrtimer_run_queues+0x1d3/0x360 [20350.505819] ? hrtimer_interrupt+0x104/0x240 [20350.510083] ? __local_bh_enable+0x50/0x60 [20350.514175] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20350.519401] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20350.524356] [20350.526454] [20350.528550] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20350.533859] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [20350.539177] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [20350.544491] delayed_work+0x1d8/0x280 [ceph] [20350.548808] ? process_scheduled_works+0x22a/0x510 [20350.553598] process_scheduled_works+0x2a4/0x510 [20350.558214] worker_thread+0x188/0x340 [20350.561964] ? rescuer_thread+0x310/0x310 [20350.565975] kthread+0xf5/0x130 [20350.569112] ? kthread_complete_and_exit+0x20/0x20 [20350.573897] ret_from_fork+0x31/0x40 [20350.577476] ? kthread_complete_and_exit+0x20/0x20 [20350.582261] ret_from_fork_asm+0x11/0x20 [20350.586192] [20377.814642] watchdog: BUG: soft lockup - CPU#0 stuck for 7864s! [dbench:78683] [20377.821861] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20377.826642] watchdog: BUG: soft lockup - CPU#7 stuck for 16861s! [kworker/7:4:78808] [20377.877316] irq event stamp: 30393452 [20377.885049] Modules linked in: ceph libceph dns_resolver fscache netfs [20377.888706] hardirqs last enabled at (30393451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20377.895225] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20377.905381] hardirqs last disabled at (30393452): [] sysvec_apic_timer_interrupt+0xe/0x90 [20377.911465] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [20377.921190] softirqs last enabled at (30393404): [] __do_softirq+0x2be/0x371 [20377.928921] llc xfs libcrc32c sunrpc [20377.937604] softirqs last disabled at (30393399): [] irq_exit_rcu+0xb0/0xd0 [20377.937607] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [20377.949775] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20377.958285] kvm irqbypass [20377.967229] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20377.969933] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [20377.977404] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20377.977406] crct10dif_pclmul joydev crc32_pclmul [20377.988531] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20377.988532] ipmi_ssif [20377.993230] ghash_clmulni_intel mei_me [20378.011968] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20378.014324] mei lpc_ich [20378.018155] [20378.023372] sha512_ssse3 acpi_ipmi [20378.025903] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20378.027393] mfd_core wmi [20378.030876] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20378.038002] ipmi_si ipmi_devintf [20378.040619] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20378.047744] ipmi_msghandler [20378.051054] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20378.058178] ioatdma [20378.061055] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20378.068179] acpi_pad acpi_power_meter [20378.070364] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20378.077487] ixgbe igb [20378.081231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20378.089308] mdio [20378.091685] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20378.097422] crc32c_intel nvme [20378.099346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20378.106470] ptp nvme_core [20378.109519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20378.116643] i2c_algo_bit [20378.119349] Call Trace: [20378.119352] [20378.126473] pps_core [20378.129092] ? watchdog_timer_fn+0x1b4/0x220 [20378.131535] dca fuse [20378.133545] ? __hrtimer_run_queues+0x1d3/0x360 [20378.135817] [20378.140082] ? hrtimer_interrupt+0x104/0x240 [20378.142351] irq event stamp: 85676750 [20378.146875] ? __local_bh_enable+0x50/0x60 [20378.148366] hardirqs last enabled at (85676749): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20378.152633] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20378.156288] hardirqs last disabled at (85676750): [] sysvec_apic_timer_interrupt+0xe/0x90 [20378.160378] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20378.170534] softirqs last enabled at (85676748): [] __do_softirq+0x2be/0x371 [20378.175753] [20378.185475] softirqs last disabled at (85676743): [] irq_exit_rcu+0xb0/0xd0 [20378.190435] [20378.190436] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20378.199119] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20378.201220] ? queued_spin_lock_slowpath+0x16/0x2e0 [20378.209726] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20378.211827] do_raw_spin_lock+0xb5/0xc0 [20378.217127] Workqueue: events delayed_work [ceph] [20378.226506] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20378.231376] [20378.231377] RIP: 0010:_raw_spin_lock+0x18/0x40 [20378.238851] try_get_cap_refs+0x159/0x880 [ceph] [20378.242677] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [20378.247379] __ceph_get_caps+0x4a1/0x760 [ceph] [20378.252158] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [20378.253654] ? __file_remove_privs+0x89/0x110 [20378.258088] [20378.258089] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [20378.262699] ? wake_bit_function+0x70/0x70 [20378.281435] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [20378.285963] ? ceph_write_iter+0x260/0x770 [ceph] [20378.291178] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [20378.295527] ceph_write_iter+0x260/0x770 [ceph] [20378.297018] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20378.304165] ? lock_is_held_type+0xaa/0x120 [20378.308250] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20378.315377] vfs_write+0x324/0x4f0 [20378.320072] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20378.327201] ksys_pwrite64+0x65/0xa0 [20378.331720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20378.338846] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20378.343022] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20378.350166] do_syscall_64+0x38/0x80 [20378.353560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20378.361640] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20378.365209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20378.370947] RIP: 0033:0x7f48f0eeb93b [20378.376078] Call Trace: [20378.376079] [20378.383202] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20378.386773] ? watchdog_timer_fn+0x1b4/0x220 [20378.393896] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20378.398943] ? __hrtimer_run_queues+0x1d3/0x360 [20378.406064] ORIG_RAX: 0000000000000012 [20378.409638] ? hrtimer_interrupt+0x104/0x240 [20378.412078] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20378.414090] ? __local_bh_enable+0x50/0x60 [20378.432827] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20378.437094] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20378.442308] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20378.446852] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20378.450681] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20378.454946] [20378.462070] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20378.466160] [20378.473290] [20378.478502] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20378.516536] ? _raw_spin_lock+0x18/0x40 [20378.520373] ? igrab+0x1d/0x50 [20378.523423] igrab+0x1d/0x50 [20378.526301] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20378.531450] delayed_work+0x1d8/0x280 [ceph] [20378.535731] ? process_scheduled_works+0x22a/0x510 [20378.540521] process_scheduled_works+0x2a4/0x510 [20378.545139] worker_thread+0x188/0x340 [20378.548886] ? rescuer_thread+0x310/0x310 [20378.552889] kthread+0xf5/0x130 [20378.556027] ? kthread_complete_and_exit+0x20/0x20 [20378.560828] ret_from_fork+0x31/0x40 [20378.564408] ? kthread_complete_and_exit+0x20/0x20 [20378.569193] ret_from_fork_asm+0x11/0x20 [20378.573124] [20405.815164] watchdog: BUG: soft lockup - CPU#0 stuck for 7890s! [dbench:78683] [20405.822382] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20405.827164] watchdog: BUG: soft lockup - CPU#7 stuck for 16887s! [kworker/7:4:78808] [20405.877837] irq event stamp: 30451480 [20405.877838] hardirqs last enabled at (30451479): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20405.885572] Modules linked in: [20405.889229] hardirqs last disabled at (30451480): [] sysvec_apic_timer_interrupt+0xe/0x90 [20405.899385] ceph [20405.902437] softirqs last enabled at (30451432): [] __do_softirq+0x2be/0x371 [20405.912160] libceph [20405.914086] softirqs last disabled at (30451427): [] irq_exit_rcu+0xb0/0xd0 [20405.922770] dns_resolver fscache [20405.924953] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20405.933464] netfs veth [20405.936776] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20405.945720] nft_chain_nat xt_MASQUERADE [20405.948164] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20405.955634] nf_nat nf_conntrack [20405.959560] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20405.965038] nf_defrag_ipv6 [20405.968261] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20405.971534] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [20405.971537] rcu: 0-....: (8277833 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=4126314 [20405.971559] rcu: 7-....: (8225794 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/8224533 fqs=4126314 [20405.971562] rcu: (detected by 3, t=8469194 jiffies, g=1309609, q=300841 ncpus=8) [20405.971564] Sending NMI from CPU 3 to CPUs 0: [20405.971567] NMI backtrace for cpu 0 [20405.971568] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20405.971570] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20405.971570] RIP: 0010:vprintk_emit+0x1ee/0x340 [20405.971573] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [20405.971574] RSP: 0018:ffffc90000003db8 EFLAGS: 00000002 [20405.971575] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [20405.971576] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [20405.971577] RBP: 000000000000002b R08: 0000000000005ef4 R09: 0000000000000004 [20405.971577] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [20405.971578] R13: 0000000000000000 R14: ffffffff82188e50 R15: ffffc90000003e08 [20405.971579] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20405.971580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20405.971581] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20405.971582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20405.971582] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20405.971583] Call Trace: [20405.971584] [20405.971585] ? nmi_cpu_backtrace+0xad/0x120 [20405.971588] ? nmi_cpu_backtrace_handler+0x11/0x20 [20405.971590] ? nmi_handle+0xd4/0x260 [20405.971594] ? default_do_nmi+0x49/0xf0 [20405.971596] ? exc_nmi+0x11e/0x150 [20405.971597] ? end_repeat_nmi+0x16/0x67 [20405.971601] ? vprintk_emit+0x1ee/0x340 [20405.971603] ? vprintk_emit+0x1ee/0x340 [20405.971605] ? vprintk_emit+0x1ee/0x340 [20405.971607] [20405.971607] [20405.971609] ? touch_softlockup_watchdog+0x20/0x20 [20405.971610] _printk+0x5c/0x80 [20405.971613] __show_regs+0x28/0x2e0 [20405.971616] ? touch_softlockup_watchdog+0x20/0x20 [20405.971617] show_regs+0x38/0x60 [20405.971620] watchdog_timer_fn+0x1b4/0x220 [20405.971621] __hrtimer_run_queues+0x1d3/0x360 [20405.971624] hrtimer_interrupt+0x104/0x240 [20405.971626] ? __local_bh_enable+0x50/0x60 [20405.971629] __sysvec_apic_timer_interrupt+0x61/0x160 [20405.971632] sysvec_apic_timer_interrupt+0x6d/0x90 [20405.971633] [20405.971634] [20405.971634] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20405.971636] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20405.971638] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20405.971640] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20405.971640] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20405.971641] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20405.971642] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20405.971643] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20405.971643] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20405.971647] do_raw_spin_lock+0xb5/0xc0 [20405.971649] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20405.971663] try_get_cap_refs+0x159/0x880 [ceph] [20405.971678] __ceph_get_caps+0x4a1/0x760 [ceph] [20405.971691] ? __file_remove_privs+0x89/0x110 [20405.971693] ? wake_bit_function+0x70/0x70 [20405.971695] ? ceph_write_iter+0x260/0x770 [ceph] [20405.971706] ceph_write_iter+0x260/0x770 [ceph] [20405.971717] ? lock_is_held_type+0xaa/0x120 [20405.971720] vfs_write+0x324/0x4f0 [20405.971724] ksys_pwrite64+0x65/0xa0 [20405.971727] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20405.971729] do_syscall_64+0x38/0x80 [20405.971731] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20405.971733] RIP: 0033:0x7f48f0eeb93b [20405.971734] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20405.971736] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [20405.971737] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20405.971738] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20405.971738] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20405.971739] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20405.971740] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20405.971743] [20405.972566] Sending NMI from CPU 3 to CPUs 7: [20405.972570] NMI backtrace for cpu 7 [20405.972571] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20405.972573] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20405.972574] Workqueue: events delayed_work [ceph] [20405.972595] RIP: 0010:io_serial_in+0x19/0x20 [20405.972599] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [20405.972600] RSP: 0018:ffffc90000264c98 EFLAGS: 00000002 [20405.972602] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [20405.972603] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [20405.972604] RBP: 00000000000026a6 R08: 0000428ea2c00ecb R09: 0000000000000007 [20405.972606] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [20405.972607] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [20405.972608] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20405.972609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20405.972611] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20405.972612] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20405.972613] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20405.972614] Call Trace: [20405.972615] [20405.972617] ? nmi_cpu_backtrace+0xad/0x120 [20405.972621] ? nmi_cpu_backtrace_handler+0x11/0x20 [20405.972623] ? nmi_handle+0xd4/0x260 [20405.972628] ? default_do_nmi+0x49/0xf0 [20405.972630] ? exc_nmi+0x11e/0x150 [20405.972633] ? end_repeat_nmi+0x16/0x67 [20405.972639] ? io_serial_in+0x19/0x20 [20405.972642] ? io_serial_in+0x19/0x20 [20405.972645] ? io_serial_in+0x19/0x20 [20405.972647] [20405.972648] [20405.972648] wait_for_lsr+0x45/0x70 [20405.972651] serial8250_console_write+0x341/0x3f0 [20405.972655] ? lock_release+0xc5/0x260 [20405.972660] console_flush_all+0x227/0x5a0 [20405.972662] ? console_flush_all+0x1ff/0x5a0 [20405.972664] ? console_flush_all+0x1ff/0x5a0 [20405.972668] console_unlock+0x7b/0x120 [20405.972672] vprintk_emit+0x246/0x340 [20405.972674] ? irq_work_queue+0x2c/0x50 [20405.972679] ? touch_softlockup_watchdog+0x20/0x20 [20405.972681] _printk+0x5c/0x80 [20405.972686] print_modules+0x6e/0xe0 [20405.972689] watchdog_timer_fn+0x19f/0x220 [20405.972692] __hrtimer_run_queues+0x1d3/0x360 [20405.972697] hrtimer_interrupt+0x104/0x240 [20405.972698] ? __local_bh_enable+0x50/0x60 [20405.972703] __sysvec_apic_timer_interrupt+0x61/0x160 [20405.972707] sysvec_apic_timer_interrupt+0x6d/0x90 [20405.972709] [20405.972710] [20405.972711] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20405.972713] RIP: 0010:lock_acquire+0xf0/0x2b0 [20405.972715] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20405.972717] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20405.972719] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20405.972720] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20405.972721] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20405.972722] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20405.972723] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20405.972729] ? igrab+0x43/0x50 [20405.972732] ? lock_release+0xc5/0x260 [20405.972735] _raw_spin_lock+0x30/0x40 [20405.972738] ? igrab+0x1d/0x50 [20405.972740] igrab+0x1d/0x50 [20405.972742] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20405.972762] delayed_work+0x1d8/0x280 [ceph] [20405.972781] ? process_scheduled_works+0x22a/0x510 [20405.972784] process_scheduled_works+0x2a4/0x510 [20405.972790] worker_thread+0x188/0x340 [20405.972794] ? rescuer_thread+0x310/0x310 [20405.972796] kthread+0xf5/0x130 [20405.972799] ? kthread_complete_and_exit+0x20/0x20 [20405.972803] ret_from_fork+0x31/0x40 [20405.972805] ? kthread_complete_and_exit+0x20/0x20 [20405.972808] ret_from_fork_asm+0x11/0x20 [20405.972816] [20405.987000] nf_defrag_ipv4 nft_compat [20405.989789] [20405.989790] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20405.995008] nf_tables nfnetlink [20406.001090] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20406.011421] bridge stp [20406.021857] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20406.029327] llc xfs [20406.033678] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20406.037163] libcrc32c sunrpc [20406.046106] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20406.053577] intel_rapl_msr [20406.058015] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20406.076752] intel_rapl_common [20406.081969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20406.089095] sb_edac [20406.096219] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20406.103343] x86_pkg_temp_thermal [20406.110467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20406.117591] coretemp [20406.125669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20406.131406] kvm_intel kvm [20406.138530] Call Trace: [20406.138531] [20406.145653] irqbypass i2c_i801 [20406.152778] ? watchdog_timer_fn+0x1b4/0x220 [20406.155223] iTCO_wdt iTCO_vendor_support [20406.157235] ? __hrtimer_run_queues+0x1d3/0x360 [20406.161410] i2c_smbus crct10dif_pclmul [20406.166197] ? hrtimer_interrupt+0x104/0x240 [20406.169765] joydev crc32_pclmul [20406.173595] ? __local_bh_enable+0x50/0x60 [20406.176994] ipmi_ssif ghash_clmulni_intel [20406.180842] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20406.184672] mei_me mei [20406.188504] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20406.192333] lpc_ich sha512_ssse3 [20406.194431] [20406.196441] acpi_ipmi mfd_core [20406.201226] [20406.204277] wmi ipmi_si [20406.207760] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20406.212563] ipmi_devintf [20406.215789] ? queued_spin_lock_slowpath+0x18/0x2e0 [20406.219878] ipmi_msghandler ioatdma [20406.224231] do_raw_spin_lock+0xb5/0xc0 [20406.228318] acpi_pad acpi_power_meter [20406.232409] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20406.237454] ixgbe igb [20406.242238] try_get_cap_refs+0x159/0x880 [ceph] [20406.244335] mdio crc32c_intel [20406.246436] __ceph_get_caps+0x4a1/0x760 [ceph] [20406.251564] nvme ptp nvme_core [20406.257045] ? __file_remove_privs+0x89/0x110 [20406.275778] i2c_algo_bit pps_core [20406.280996] ? wake_bit_function+0x70/0x70 [20406.288119] dca fuse [20406.295247] ? ceph_write_iter+0x260/0x770 [ceph] [20406.302368] [20406.302369] irq event stamp: 85897866 [20406.309492] ceph_write_iter+0x260/0x770 [ceph] [20406.316615] hardirqs last enabled at (85897865): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20406.320451] ? lock_is_held_type+0xaa/0x120 [20406.325231] hardirqs last disabled at (85897866): [] sysvec_apic_timer_interrupt+0xe/0x90 [20406.329851] vfs_write+0x324/0x4f0 [20406.334373] softirqs last enabled at (85897864): [] __do_softirq+0x2be/0x371 [20406.338730] ksys_pwrite64+0x65/0xa0 [20406.342815] softirqs last disabled at (85897859): [] irq_exit_rcu+0xb0/0xd0 [20406.347512] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20406.352036] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20406.356215] do_syscall_64+0x38/0x80 [20406.359613] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20406.363183] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20406.368314] Workqueue: events delayed_work [ceph] [20406.371884] RIP: 0033:0x7f48f0eeb93b [20406.376927] [20406.376928] RIP: 0010:lock_acquire+0xf0/0x2b0 [20406.380499] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20406.399236] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20406.406794] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20406.413918] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20406.421044] ORIG_RAX: 0000000000000012 [20406.428166] [20406.435290] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20406.442414] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20406.444600] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20406.448949] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20406.452433] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20406.461812] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20406.469281] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20406.473978] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20406.478244] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20406.496981] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20406.502203] [20406.509322] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20407.299800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20407.305537] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20407.312661] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20407.319785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20407.326908] Call Trace: [20407.329351] [20407.331365] ? watchdog_timer_fn+0x1b4/0x220 [20407.335627] ? __hrtimer_run_queues+0x1d3/0x360 [20407.340155] ? hrtimer_interrupt+0x104/0x240 [20407.344423] ? __local_bh_enable+0x50/0x60 [20407.348517] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20407.353732] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20407.358689] [20407.360787] [20407.362884] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20407.368191] ? lock_acquire+0xf0/0x2b0 [20407.371942] ? igrab+0x43/0x50 [20407.374992] ? lock_release+0xc5/0x260 [20407.378738] _raw_spin_lock+0x30/0x40 [20407.382401] ? igrab+0x1d/0x50 [20407.385453] igrab+0x1d/0x50 [20407.388332] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20407.393480] delayed_work+0x1d8/0x280 [ceph] [20407.397761] ? process_scheduled_works+0x22a/0x510 [20407.402543] process_scheduled_works+0x2a4/0x510 [20407.407159] worker_thread+0x188/0x340 [20407.410908] ? rescuer_thread+0x310/0x310 [20407.414911] kthread+0xf5/0x130 [20407.418050] ? kthread_complete_and_exit+0x20/0x20 [20407.422834] ret_from_fork+0x31/0x40 [20407.426402] ? kthread_complete_and_exit+0x20/0x20 [20407.431188] ret_from_fork_asm+0x11/0x20 [20407.435109] [20433.815686] watchdog: BUG: soft lockup - CPU#0 stuck for 7916s! [dbench:78683] [20433.822905] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20433.827686] watchdog: BUG: soft lockup - CPU#7 stuck for 16913s! [kworker/7:4:78808] [20433.878360] irq event stamp: 30509398 [20433.886092] Modules linked in: ceph libceph dns_resolver fscache netfs [20433.889751] hardirqs last enabled at (30509397): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20433.896276] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20433.906434] hardirqs last disabled at (30509398): [] sysvec_apic_timer_interrupt+0xe/0x90 [20433.906437] nf_defrag_ipv6 [20433.906438] softirqs last enabled at (30509390): [] __do_softirq+0x2be/0x371 [20433.912519] nf_defrag_ipv4 nft_compat [20433.922243] softirqs last disabled at (30509385): [] irq_exit_rcu+0xb0/0xd0 [20433.925034] nf_tables nfnetlink [20433.933717] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20433.937461] bridge stp [20433.945972] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20433.949196] llc [20433.958141] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20433.960585] xfs libcrc32c [20433.968057] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20433.969910] sunrpc intel_rapl_msr [20433.975387] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20433.978091] intel_rapl_common sb_edac [20433.996830] [20433.996831] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20434.000226] x86_pkg_temp_thermal [20434.005445] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20434.009188] coretemp kvm_intel [20434.010680] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20434.017802] kvm irqbypass [20434.021112] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20434.028236] i2c_i801 iTCO_wdt [20434.031374] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20434.038499] iTCO_vendor_support [20434.041202] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20434.048327] i2c_smbus crct10dif_pclmul [20434.051378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20434.058503] joydev crc32_pclmul [20434.061726] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20434.069804] ipmi_ssif [20434.073634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20434.079371] ghash_clmulni_intel [20434.082596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20434.089721] mei_me [20434.092085] Call Trace: [20434.092087] [20434.099209] mei lpc_ich [20434.102436] ? watchdog_timer_fn+0x1b4/0x220 [20434.109559] sha512_ssse3 acpi_ipmi [20434.111658] ? __hrtimer_run_queues+0x1d3/0x360 [20434.114099] mfd_core [20434.116113] ? hrtimer_interrupt+0x104/0x240 [20434.118641] wmi [20434.122904] ? __local_bh_enable+0x50/0x60 [20434.126389] ipmi_si ipmi_devintf [20434.130915] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20434.133186] ipmi_msghandler [20434.137450] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20434.139285] ioatdma acpi_pad [20434.143376] [20434.146687] acpi_power_meter [20434.151903] [20434.151905] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20434.154783] ixgbe igb [20434.159744] ? queued_spin_lock_slowpath+0x18/0x2e0 [20434.162704] mdio [20434.164803] do_raw_spin_lock+0xb5/0xc0 [20434.167764] crc32c_intel nvme [20434.169866] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20434.175166] ptp nvme_core [20434.177526] try_get_cap_refs+0x159/0x880 [ceph] [20434.182394] i2c_algo_bit pps_core [20434.184325] __ceph_get_caps+0x4a1/0x760 [ceph] [20434.188150] dca [20434.191206] ? __file_remove_privs+0x89/0x110 [20434.195985] fuse [20434.198690] ? wake_bit_function+0x70/0x70 [20434.203299] [20434.203299] irq event stamp: 86111992 [20434.206699] ? ceph_write_iter+0x260/0x770 [ceph] [20434.211221] hardirqs last enabled at (86111991): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20434.213057] ceph_write_iter+0x260/0x770 [ceph] [20434.217408] hardirqs last disabled at (86111992): [] sysvec_apic_timer_interrupt+0xe/0x90 [20434.219338] ? lock_is_held_type+0xaa/0x120 [20434.223422] softirqs last enabled at (86111990): [] __do_softirq+0x2be/0x371 [20434.224916] vfs_write+0x324/0x4f0 [20434.228571] softirqs last disabled at (86111985): [] irq_exit_rcu+0xb0/0xd0 [20434.233274] ksys_pwrite64+0x65/0xa0 [20434.243426] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20434.247951] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20434.257674] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20434.261854] do_syscall_64+0x38/0x80 [20434.270538] Workqueue: events delayed_work [ceph] [20434.273934] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20434.282444] [20434.282445] RIP: 0010:lock_acquire+0xf0/0x2b0 [20434.286017] RIP: 0033:0x7f48f0eeb93b [20434.295394] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20434.300525] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20434.307993] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20434.311564] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20434.316261] [20434.316262] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20434.321307] ORIG_RAX: 0000000000000012 [20434.322798] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20434.327147] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20434.330718] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20434.349456] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20434.368194] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20434.373410] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20434.378629] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20434.380119] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20434.387242] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20434.391090] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20434.398216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20434.405346] [20434.412464] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20434.478340] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20434.485465] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20434.492588] Call Trace: [20434.495036] [20434.497045] ? watchdog_timer_fn+0x1b4/0x220 [20434.501309] ? __hrtimer_run_queues+0x1d3/0x360 [20434.505837] ? hrtimer_interrupt+0x104/0x240 [20434.510105] ? __local_bh_enable+0x50/0x60 [20434.514198] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20434.519423] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20434.524381] [20434.526476] [20434.528575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20434.533884] ? lock_acquire+0xf0/0x2b0 [20434.537631] ? igrab+0x43/0x50 [20434.540681] ? lock_release+0xc5/0x260 [20434.544428] _raw_spin_lock+0x30/0x40 [20434.548094] ? igrab+0x1d/0x50 [20434.551143] igrab+0x1d/0x50 [20434.554022] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20434.559170] delayed_work+0x1d8/0x280 [ceph] [20434.563452] ? process_scheduled_works+0x22a/0x510 [20434.568242] process_scheduled_works+0x2a4/0x510 [20434.572859] worker_thread+0x188/0x340 [20434.576608] ? rescuer_thread+0x310/0x310 [20434.580609] kthread+0xf5/0x130 [20434.583748] ? kthread_complete_and_exit+0x20/0x20 [20434.588532] ret_from_fork+0x31/0x40 [20434.592111] ? kthread_complete_and_exit+0x20/0x20 [20434.596894] ret_from_fork_asm+0x11/0x20 [20434.600818] [20461.816204] watchdog: BUG: soft lockup - CPU#0 stuck for 7942s! [dbench:78683] [20461.823417] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20461.828204] watchdog: BUG: soft lockup - CPU#7 stuck for 16939s! [kworker/7:4:78808] [20461.878858] irq event stamp: 30567516 [20461.878860] hardirqs last enabled at (30567515): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20461.886600] Modules linked in: [20461.890255] hardirqs last disabled at (30567516): [] sysvec_apic_timer_interrupt+0xe/0x90 [20461.900413] ceph [20461.903462] softirqs last enabled at (30567508): [] __do_softirq+0x2be/0x371 [20461.913188] libceph [20461.915111] softirqs last disabled at (30567503): [] irq_exit_rcu+0xb0/0xd0 [20461.923796] dns_resolver fscache [20461.925981] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20461.934491] netfs veth [20461.937802] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20461.946745] nft_chain_nat xt_MASQUERADE [20461.949189] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20461.956660] nf_nat nf_conntrack [20461.960577] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20461.966073] nf_defrag_ipv6 nf_defrag_ipv4 [20461.969298] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20461.988034] nft_compat [20461.992126] [20461.992126] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20461.997343] nf_tables nfnetlink [20461.999785] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20462.001277] bridge stp [20462.008401] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20462.011625] llc xfs [20462.018749] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20462.021194] libcrc32c sunrpc [20462.028317] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20462.030501] intel_rapl_msr intel_rapl_common [20462.037627] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20462.040601] sb_edac x86_pkg_temp_thermal [20462.047722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20462.052072] coretemp kvm_intel [20462.060152] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20462.064155] kvm irqbypass [20462.069891] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20462.073029] i2c_i801 iTCO_wdt [20462.080155] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20462.082858] iTCO_vendor_support i2c_smbus [20462.089982] Call Trace: [20462.093033] crct10dif_pclmul [20462.100158] [20462.104248] joydev crc32_pclmul [20462.106692] ? watchdog_timer_fn+0x1b4/0x220 [20462.109655] ipmi_ssif ghash_clmulni_intel [20462.111668] ? __hrtimer_run_queues+0x1d3/0x360 [20462.114890] mei_me [20462.119157] ? hrtimer_interrupt+0x104/0x240 [20462.123246] mei lpc_ich [20462.127770] ? __local_bh_enable+0x50/0x60 [20462.129881] sha512_ssse3 acpi_ipmi [20462.134151] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20462.136679] mfd_core wmi [20462.140769] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20462.144254] ipmi_si ipmi_devintf [20462.149473] [20462.152089] ipmi_msghandler ioatdma [20462.157045] [20462.160356] acpi_pad acpi_power_meter [20462.162454] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20462.166025] ixgbe igb [20462.168127] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20462.171867] mdio crc32c_intel [20462.177172] do_raw_spin_lock+0xb5/0xc0 [20462.179527] nvme ptp [20462.184400] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20462.187449] nvme_core i2c_algo_bit [20462.191283] try_get_cap_refs+0x159/0x880 [ceph] [20462.193551] pps_core dca [20462.198340] __ceph_get_caps+0x4a1/0x760 [ceph] [20462.201818] fuse [20462.206434] ? __file_remove_privs+0x89/0x110 [20462.209046] irq event stamp: 86332976 [20462.209047] hardirqs last enabled at (86332975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20462.213571] ? wake_bit_function+0x70/0x70 [20462.215495] hardirqs last disabled at (86332976): [] sysvec_apic_timer_interrupt+0xe/0x90 [20462.219848] ? ceph_write_iter+0x260/0x770 [ceph] [20462.223504] softirqs last enabled at (86332974): [] __do_softirq+0x2be/0x371 [20462.233661] ceph_write_iter+0x260/0x770 [ceph] [20462.237750] softirqs last disabled at (86332969): [] irq_exit_rcu+0xb0/0xd0 [20462.247478] ? lock_is_held_type+0xaa/0x120 [20462.252172] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20462.260859] vfs_write+0x324/0x4f0 [20462.265381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20462.273896] ksys_pwrite64+0x65/0xa0 [20462.278069] Workqueue: events delayed_work [ceph] [20462.287447] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20462.290844] [20462.290845] RIP: 0010:lock_acquire+0xf0/0x2b0 [20462.298316] do_syscall_64+0x38/0x80 [20462.301894] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20462.306607] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20462.311740] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20462.313232] RIP: 0033:0x7f48f0eeb93b [20462.317583] [20462.317584] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20462.321170] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20462.339908] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20462.344952] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20462.350169] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20462.353738] ORIG_RAX: 0000000000000012 [20462.355229] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20462.362355] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20462.381091] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20462.388216] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20462.393434] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20462.400558] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20462.404388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20462.411512] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20462.418637] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20462.425761] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20462.432897] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20462.440977] [20462.448095] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20462.491637] Call Trace: [20462.494083] [20462.496093] ? watchdog_timer_fn+0x1b4/0x220 [20462.500359] ? __hrtimer_run_queues+0x1d3/0x360 [20462.504900] ? hrtimer_interrupt+0x104/0x240 [20462.509163] ? __local_bh_enable+0x50/0x60 [20462.513256] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20462.518479] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20462.523436] [20462.525533] [20462.527633] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20462.532940] ? lock_acquire+0xf0/0x2b0 [20462.536689] ? igrab+0x43/0x50 [20462.539739] ? lock_release+0xc5/0x260 [20462.543484] _raw_spin_lock+0x30/0x40 [20462.547140] ? igrab+0x1d/0x50 [20462.550192] igrab+0x1d/0x50 [20462.553068] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20462.558218] delayed_work+0x1d8/0x280 [ceph] [20462.562501] ? process_scheduled_works+0x22a/0x510 [20462.567290] process_scheduled_works+0x2a4/0x510 [20462.571905] worker_thread+0x188/0x340 [20462.575654] ? rescuer_thread+0x310/0x310 [20462.579659] kthread+0xf5/0x130 [20462.582796] ? kthread_complete_and_exit+0x20/0x20 [20462.587580] ret_from_fork+0x31/0x40 [20462.591150] ? kthread_complete_and_exit+0x20/0x20 [20462.595935] ret_from_fork_asm+0x11/0x20 [20462.599857] [20489.816722] watchdog: BUG: soft lockup - CPU#0 stuck for 7968s! [dbench:78683] [20489.823939] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20489.828722] watchdog: BUG: soft lockup - CPU#7 stuck for 16965s! [kworker/7:4:78808] [20489.879400] irq event stamp: 30625632 [20489.887129] Modules linked in: ceph libceph dns_resolver fscache netfs veth [20489.890786] hardirqs last enabled at (30625631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20489.897746] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20489.907904] hardirqs last disabled at (30625632): [] sysvec_apic_timer_interrupt+0xe/0x90 [20489.913554] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [20489.923277] softirqs last enabled at (30625624): [] __do_softirq+0x2be/0x371 [20489.931009] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [20489.939692] softirqs last disabled at (30625619): [] irq_exit_rcu+0xb0/0xd0 [20489.946210] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [20489.954724] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20489.960718] irqbypass i2c_i801 [20489.969663] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20489.972800] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [20489.980270] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20489.987221] crc32_pclmul ipmi_ssif [20489.992698] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20489.992699] ghash_clmulni_intel [20489.992700] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20489.996184] mei_me mei [20490.014920] [20490.018145] lpc_ich [20490.023363] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20490.025806] sha512_ssse3 acpi_ipmi [20490.027296] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20490.029482] mfd_core [20490.036606] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20490.040090] wmi ipmi_si [20490.047213] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20490.049485] ipmi_devintf [20490.056608] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20490.059139] ipmi_msghandler [20490.066264] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20490.068880] ioatdma acpi_pad [20490.076004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20490.078882] acpi_power_meter [20490.086959] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20490.089924] ixgbe [20490.095662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20490.098624] igb [20490.105750] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20490.107759] mdio [20490.114884] Call Trace: [20490.114887] [20490.116722] crc32c_intel [20490.123847] ? watchdog_timer_fn+0x1b4/0x220 [20490.125768] nvme ptp [20490.128216] ? __hrtimer_run_queues+0x1d3/0x360 [20490.130224] nvme_core [20490.132846] ? hrtimer_interrupt+0x104/0x240 [20490.137122] i2c_algo_bit [20490.139404] ? __local_bh_enable+0x50/0x60 [20490.143926] pps_core [20490.146286] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20490.150547] dca fuse [20490.153167] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20490.157257] [20490.157258] irq event stamp: 86553910 [20490.159528] [20490.164745] hardirqs last enabled at (86553909): [] asm_common_interrupt+0x26/0x40 [20490.167016] [20490.167017] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20490.171973] hardirqs last disabled at (86553910): [] sysvec_apic_timer_interrupt+0xe/0x90 [20490.173468] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20490.177122] softirqs last enabled at (86553908): [] __do_softirq+0x2be/0x371 [20490.179221] do_raw_spin_lock+0xb5/0xc0 [20490.188423] softirqs last disabled at (86553901): [] irq_exit_rcu+0xb0/0xd0 [20490.190523] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20490.195824] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20490.205549] try_get_cap_refs+0x159/0x880 [ceph] [20490.210418] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20490.219126] __ceph_get_caps+0x4a1/0x760 [ceph] [20490.222953] Workqueue: events delayed_work [ceph] [20490.231466] ? __file_remove_privs+0x89/0x110 [20490.236246] [20490.236247] RIP: 0010:lock_acquire+0xf0/0x2b0 [20490.245624] ? wake_bit_function+0x70/0x70 [20490.250233] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20490.257708] ? ceph_write_iter+0x260/0x770 [ceph] [20490.262228] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20490.266928] ceph_write_iter+0x260/0x770 [ceph] [20490.271277] [20490.271278] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20490.272773] ? lock_is_held_type+0xaa/0x120 [20490.277124] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20490.281221] vfs_write+0x324/0x4f0 [20490.299955] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20490.304656] ksys_pwrite64+0x65/0xa0 [20490.309870] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20490.314408] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20490.315892] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20490.323019] do_syscall_64+0x38/0x80 [20490.327196] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20490.334322] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20490.337715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20490.344841] RIP: 0033:0x7f48f0eeb93b [20490.348420] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20490.355544] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20490.360676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20490.367800] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20490.371369] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20490.379447] ORIG_RAX: 0000000000000012 [20490.384492] Call Trace: [20490.384493] [20490.390230] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20490.393799] ? watchdog_timer_fn+0x1b4/0x220 [20490.400925] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20490.419664] ? __hrtimer_run_queues+0x1d3/0x360 [20490.426787] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20490.432007] ? hrtimer_interrupt+0x104/0x240 [20490.439127] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20490.442958] ? __local_bh_enable+0x50/0x60 [20490.445406] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20490.447424] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20490.454551] [20490.458811] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20490.512553] [20490.514650] [20490.516747] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20490.522055] ? lock_acquire+0xf0/0x2b0 [20490.525805] ? igrab+0x43/0x50 [20490.528855] ? lock_release+0xc5/0x260 [20490.532601] _raw_spin_lock+0x30/0x40 [20490.536265] ? igrab+0x1d/0x50 [20490.539317] igrab+0x1d/0x50 [20490.542195] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20490.547345] delayed_work+0x1d8/0x280 [ceph] [20490.551626] ? process_scheduled_works+0x22a/0x510 [20490.556416] process_scheduled_works+0x2a4/0x510 [20490.561031] worker_thread+0x188/0x340 [20490.564781] ? rescuer_thread+0x310/0x310 [20490.568783] kthread+0xf5/0x130 [20490.571921] ? kthread_complete_and_exit+0x20/0x20 [20490.576705] ret_from_fork+0x31/0x40 [20490.580285] ? kthread_complete_and_exit+0x20/0x20 [20490.585069] ret_from_fork_asm+0x11/0x20 [20490.588994] [20517.817240] watchdog: BUG: soft lockup - CPU#0 stuck for 7994s! [dbench:78683] [20517.824453] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20517.829240] watchdog: BUG: soft lockup - CPU#7 stuck for 16991s! [kworker/7:4:78808] [20517.879912] irq event stamp: 30683690 [20517.887642] Modules linked in: ceph libceph [20517.891299] hardirqs last enabled at (30683689): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20517.895477] dns_resolver fscache netfs veth [20517.905634] hardirqs last disabled at (30683690): [] sysvec_apic_timer_interrupt+0xe/0x90 [20517.905636] nft_chain_nat [20517.905638] softirqs last enabled at (30683682): [] __do_softirq+0x2be/0x371 [20517.909899] xt_MASQUERADE nf_nat [20517.919641] softirqs last disabled at (30683677): [] irq_exit_rcu+0xb0/0xd0 [20517.922345] nf_conntrack nf_defrag_ipv6 [20517.931029] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20517.934339] nf_defrag_ipv4 [20517.942850] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20517.946766] nft_compat nf_tables [20517.955711] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20517.958501] nfnetlink bridge [20517.965973] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20517.969283] stp llc [20517.974762] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20517.977725] xfs libcrc32c [20517.996463] [20517.998648] sunrpc intel_rapl_msr [20518.003863] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20518.006569] intel_rapl_common sb_edac [20518.008058] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20518.011456] x86_pkg_temp_thermal coretemp [20518.018582] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20518.022325] kvm_intel kvm [20518.029449] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20518.033540] irqbypass i2c_i801 [20518.040664] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20518.043368] iTCO_wdt iTCO_vendor_support [20518.050492] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20518.053639] i2c_smbus [20518.060761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20518.064766] crct10dif_pclmul [20518.072843] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20518.075200] joydev crc32_pclmul [20518.080939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20518.083902] ipmi_ssif [20518.091027] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20518.094251] ghash_clmulni_intel [20518.101375] Call Trace: [20518.101376] [20518.103731] mei_me mei [20518.110858] ? watchdog_timer_fn+0x1b4/0x220 [20518.114081] lpc_ich sha512_ssse3 [20518.116524] ? __hrtimer_run_queues+0x1d3/0x360 [20518.118534] acpi_ipmi mfd_core [20518.120981] ? hrtimer_interrupt+0x104/0x240 [20518.125243] wmi ipmi_si [20518.128553] ? __local_bh_enable+0x50/0x60 [20518.133078] ipmi_devintf ipmi_msghandler [20518.136218] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20518.140480] ioatdma acpi_pad [20518.143010] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20518.147101] acpi_power_meter ixgbe [20518.151106] [20518.156323] igb mdio [20518.159287] [20518.159289] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20518.164244] crc32c_intel nvme [20518.167731] ? queued_spin_lock_slowpath+0x16/0x2e0 [20518.169826] ptp nvme_core [20518.172097] do_raw_spin_lock+0xb5/0xc0 [20518.174194] i2c_algo_bit pps_core [20518.179497] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20518.182547] dca fuse [20518.187420] try_get_cap_refs+0x159/0x880 [ceph] [20518.190123] [20518.190124] irq event stamp: 86774900 [20518.193957] __ceph_get_caps+0x4a1/0x760 [ceph] [20518.197351] hardirqs last enabled at (86774899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20518.202139] ? __file_remove_privs+0x89/0x110 [20518.204405] hardirqs last disabled at (86774900): [] sysvec_apic_timer_interrupt+0xe/0x90 [20518.209018] ? wake_bit_function+0x70/0x70 [20518.210507] softirqs last enabled at (86774898): [] __do_softirq+0x2be/0x371 [20518.214167] ? ceph_write_iter+0x260/0x770 [ceph] [20518.218688] softirqs last disabled at (86774893): [] irq_exit_rcu+0xb0/0xd0 [20518.228847] ceph_write_iter+0x260/0x770 [ceph] [20518.233198] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20518.242926] ? lock_is_held_type+0xaa/0x120 [20518.247012] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20518.255698] vfs_write+0x324/0x4f0 [20518.260394] Workqueue: events delayed_work [ceph] [20518.268909] ksys_pwrite64+0x65/0xa0 [20518.273428] [20518.273429] RIP: 0010:lock_acquire+0xf0/0x2b0 [20518.282806] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20518.286984] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20518.294456] do_syscall_64+0x38/0x80 [20518.297850] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20518.302550] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20518.306120] [20518.306120] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20518.307610] RIP: 0033:0x7f48f0eeb93b [20518.311961] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20518.317092] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20518.335830] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20518.339402] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20518.344619] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20518.349678] ORIG_RAX: 0000000000000012 [20518.351169] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20518.358293] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20518.361865] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20518.368989] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20518.387726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20518.394851] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20518.400069] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20518.407191] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20518.411023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20518.418148] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20518.425272] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20518.433353] [20518.440473] Call Trace: [20518.440474] [20518.495595] ? watchdog_timer_fn+0x1b4/0x220 [20518.499860] ? __hrtimer_run_queues+0x1d3/0x360 [20518.504387] ? hrtimer_interrupt+0x104/0x240 [20518.508654] ? __local_bh_enable+0x50/0x60 [20518.512749] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20518.517972] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20518.522936] [20518.525035] [20518.527133] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20518.532442] ? lock_acquire+0xf0/0x2b0 [20518.536192] ? igrab+0x43/0x50 [20518.539241] ? lock_release+0xc5/0x260 [20518.542987] _raw_spin_lock+0x30/0x40 [20518.546650] ? igrab+0x1d/0x50 [20518.549701] igrab+0x1d/0x50 [20518.552580] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20518.557730] delayed_work+0x1d8/0x280 [ceph] [20518.562010] ? process_scheduled_works+0x22a/0x510 [20518.566800] process_scheduled_works+0x2a4/0x510 [20518.571416] worker_thread+0x188/0x340 [20518.575166] ? rescuer_thread+0x310/0x310 [20518.579169] kthread+0xf5/0x130 [20518.582307] ? kthread_complete_and_exit+0x20/0x20 [20518.587093] ret_from_fork+0x31/0x40 [20518.590669] ? kthread_complete_and_exit+0x20/0x20 [20518.595454] ret_from_fork_asm+0x11/0x20 [20518.599378] [20545.817757] watchdog: BUG: soft lockup - CPU#0 stuck for 8020s! [dbench:78683] [20545.824976] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20545.829757] watchdog: BUG: soft lockup - CPU#7 stuck for 17017s! [kworker/7:4:78808] [20545.880442] irq event stamp: 30741854 [20545.880443] hardirqs last enabled at (30741853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20545.888173] Modules linked in: [20545.891831] hardirqs last disabled at (30741854): [] sysvec_apic_timer_interrupt+0xe/0x90 [20545.901987] ceph [20545.905038] softirqs last enabled at (30741846): [] __do_softirq+0x2be/0x371 [20545.914762] libceph [20545.916686] softirqs last disabled at (30741841): [] irq_exit_rcu+0xb0/0xd0 [20545.925372] dns_resolver fscache [20545.927555] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20545.936067] netfs veth [20545.939377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20545.948321] nft_chain_nat xt_MASQUERADE [20545.950764] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20545.958235] nf_nat nf_conntrack [20545.962154] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20545.967631] nf_defrag_ipv6 [20545.970853] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20545.989592] nf_defrag_ipv4 nft_compat [20545.992385] [20545.997601] nf_tables nfnetlink [20546.001345] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20546.002834] bridge stp [20546.006059] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20546.013185] llc [20546.015628] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20546.022752] xfs libcrc32c [20546.024588] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20546.031713] sunrpc [20546.034434] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20546.041560] intel_rapl_msr [20546.043656] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20546.050779] intel_rapl_common sb_edac [20546.053571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20546.061648] x86_pkg_temp_thermal coretemp [20546.065393] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20546.071129] kvm_intel kvm [20546.075237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20546.082361] irqbypass i2c_i801 [20546.085066] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20546.092191] iTCO_wdt [20546.095327] Call Trace: [20546.095329] [20546.102452] iTCO_vendor_support i2c_smbus [20546.104723] ? watchdog_timer_fn+0x1b4/0x220 [20546.107165] crct10dif_pclmul joydev [20546.109178] ? __hrtimer_run_queues+0x1d3/0x360 [20546.113267] crc32_pclmul ipmi_ssif [20546.117536] ? hrtimer_interrupt+0x104/0x240 [20546.121102] ghash_clmulni_intel [20546.125627] ? __local_bh_enable+0x50/0x60 [20546.129111] mei_me mei [20546.133377] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20546.136600] lpc_ich sha512_ssse3 [20546.140690] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20546.143133] acpi_ipmi mfd_core [20546.148368] [20546.151678] wmi ipmi_si [20546.156638] [20546.159775] ipmi_devintf [20546.161872] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20546.164402] ipmi_msghandler ioatdma [20546.166504] ? queued_spin_lock_slowpath+0x16/0x2e0 [20546.169117] acpi_pad acpi_power_meter [20546.174423] do_raw_spin_lock+0xb5/0xc0 [20546.177991] ixgbe [20546.182866] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20546.186608] igb mdio [20546.190440] try_get_cap_refs+0x159/0x880 [ceph] [20546.192449] crc32c_intel [20546.197237] __ceph_get_caps+0x4a1/0x760 [ceph] [20546.199504] nvme ptp [20546.204118] ? __file_remove_privs+0x89/0x110 [20546.206732] nvme_core [20546.211256] ? wake_bit_function+0x70/0x70 [20546.213525] i2c_algo_bit pps_core [20546.217880] ? ceph_write_iter+0x260/0x770 [ceph] [20546.220233] dca fuse [20546.224326] ceph_write_iter+0x260/0x770 [ceph] [20546.227722] [20546.227722] irq event stamp: 86995730 [20546.232425] ? lock_is_held_type+0xaa/0x120 [20546.234708] hardirqs last enabled at (86995729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20546.239233] vfs_write+0x324/0x4f0 [20546.240721] hardirqs last disabled at (86995730): [] sysvec_apic_timer_interrupt+0xe/0x90 [20546.244383] ksys_pwrite64+0x65/0xa0 [20546.248557] softirqs last enabled at (86995728): [] __do_softirq+0x2be/0x371 [20546.258715] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20546.262112] softirqs last disabled at (86995723): [] irq_exit_rcu+0xb0/0xd0 [20546.271837] do_syscall_64+0x38/0x80 [20546.275407] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20546.284095] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20546.289222] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20546.297735] RIP: 0033:0x7f48f0eeb93b [20546.301304] Workqueue: events delayed_work [ceph] [20546.310682] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20546.315725] [20546.315726] RIP: 0010:lock_acquire+0xf0/0x2b0 [20546.323197] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20546.326767] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20546.331465] ORIG_RAX: 0000000000000012 [20546.350201] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20546.351693] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20546.356043] [20546.356044] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20546.361261] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20546.379998] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20546.383828] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20546.389047] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20546.396169] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20546.397661] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20546.404784] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20546.411909] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20546.419040] [20546.426161] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20546.472040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20546.477779] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20546.484901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20546.492026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20546.499164] Call Trace: [20546.501613] [20546.503624] ? watchdog_timer_fn+0x1b4/0x220 [20546.507887] ? __hrtimer_run_queues+0x1d3/0x360 [20546.512413] ? hrtimer_interrupt+0x104/0x240 [20546.516673] ? __local_bh_enable+0x50/0x60 [20546.520766] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20546.525982] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20546.530940] [20546.533036] [20546.535135] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20546.540460] ? lock_acquire+0xf0/0x2b0 [20546.544212] ? igrab+0x43/0x50 [20546.547270] ? lock_release+0xc5/0x260 [20546.551013] _raw_spin_lock+0x30/0x40 [20546.554670] ? igrab+0x1d/0x50 [20546.557719] igrab+0x1d/0x50 [20546.560600] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20546.565749] delayed_work+0x1d8/0x280 [ceph] [20546.570028] ? process_scheduled_works+0x22a/0x510 [20546.574812] process_scheduled_works+0x2a4/0x510 [20546.579427] worker_thread+0x188/0x340 [20546.583178] ? rescuer_thread+0x310/0x310 [20546.587187] kthread+0xf5/0x130 [20546.590327] ? kthread_complete_and_exit+0x20/0x20 [20546.595118] ret_from_fork+0x31/0x40 [20546.598698] ? kthread_complete_and_exit+0x20/0x20 [20546.603482] ret_from_fork_asm+0x11/0x20 [20546.607413] [20573.818275] watchdog: BUG: soft lockup - CPU#0 stuck for 8046s! [dbench:78683] [20573.825490] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20573.830275] watchdog: BUG: soft lockup - CPU#7 stuck for 17043s! [kworker/7:4:78808] [20573.880946] irq event stamp: 30799968 [20573.880947] hardirqs last enabled at (30799967): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20573.888678] Modules linked in: [20573.892336] hardirqs last disabled at (30799968): [] sysvec_apic_timer_interrupt+0xe/0x90 [20573.902494] ceph [20573.905543] softirqs last enabled at (30799960): [] __do_softirq+0x2be/0x371 [20573.915267] libceph dns_resolver [20573.917191] softirqs last disabled at (30799955): [] irq_exit_rcu+0xb0/0xd0 [20573.925877] fscache netfs [20573.929188] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20573.937698] veth [20573.940401] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20573.949347] nft_chain_nat [20573.951270] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20573.958741] xt_MASQUERADE nf_nat [20573.961443] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20573.966921] nf_conntrack nf_defrag_ipv6 [20573.970232] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20573.988972] nf_defrag_ipv4 [20573.992887] [20573.992888] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20573.998105] nft_compat nf_tables [20574.000896] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20574.002386] nfnetlink bridge [20574.009511] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20574.012821] stp llc [20574.019946] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20574.022926] xfs libcrc32c [20574.030069] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20574.032252] sunrpc intel_rapl_msr [20574.039376] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20574.042081] intel_rapl_common sb_edac [20574.049206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20574.052602] x86_pkg_temp_thermal coretemp [20574.060679] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20574.064423] kvm_intel kvm [20574.070161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20574.074251] irqbypass i2c_i801 [20574.081375] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20574.084080] iTCO_wdt iTCO_vendor_support [20574.091205] Call Trace: [20574.091206] [20574.094342] i2c_smbus [20574.101466] ? watchdog_timer_fn+0x1b4/0x220 [20574.105469] crct10dif_pclmul joydev [20574.107915] ? __hrtimer_run_queues+0x1d3/0x360 [20574.109923] crc32_pclmul ipmi_ssif [20574.112286] ? hrtimer_interrupt+0x104/0x240 [20574.116547] ghash_clmulni_intel mei_me [20574.120116] ? __local_bh_enable+0x50/0x60 [20574.124641] mei [20574.128128] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20574.132388] lpc_ich sha512_ssse3 [20574.136220] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20574.140311] acpi_ipmi mfd_core [20574.142148] [20574.147366] wmi ipmi_si [20574.150675] [20574.150677] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20574.155633] ipmi_devintf ipmi_msghandler [20574.158775] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20574.160867] ioatdma acpi_pad [20574.163402] do_raw_spin_lock+0xb5/0xc0 [20574.165497] acpi_power_meter ixgbe [20574.170801] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20574.174804] igb mdio [20574.179676] try_get_cap_refs+0x159/0x880 [ceph] [20574.182639] crc32c_intel nvme [20574.186475] __ceph_get_caps+0x4a1/0x760 [ceph] [20574.189954] ptp nvme_core i2c_algo_bit [20574.194759] ? __file_remove_privs+0x89/0x110 [20574.197027] pps_core dca [20574.201638] ? wake_bit_function+0x70/0x70 [20574.204687] fuse [20574.209214] ? ceph_write_iter+0x260/0x770 [ceph] [20574.213042] irq event stamp: 87216484 [20574.217395] ceph_write_iter+0x260/0x770 [ceph] [20574.220009] hardirqs last enabled at (87216483): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20574.224106] ? lock_is_held_type+0xaa/0x120 [20574.226025] hardirqs last disabled at (87216484): [] sysvec_apic_timer_interrupt+0xe/0x90 [20574.230727] vfs_write+0x324/0x4f0 [20574.234381] softirqs last enabled at (87216482): [] __do_softirq+0x2be/0x371 [20574.238908] ksys_pwrite64+0x65/0xa0 [20574.249061] softirqs last disabled at (87216477): [] irq_exit_rcu+0xb0/0xd0 [20574.253240] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20574.262963] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20574.266363] do_syscall_64+0x38/0x80 [20574.275045] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20574.278616] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20574.287126] Workqueue: events delayed_work [ceph] [20574.292258] RIP: 0033:0x7f48f0eeb93b [20574.301635] [20574.301636] RIP: 0010:ceph_check_delayed_caps+0x5f/0x270 [ceph] [20574.305206] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20574.312676] Code: 1f 44 00 00 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 <48> 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff [20574.317721] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20574.322418] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [20574.325988] ORIG_RAX: 0000000000000012 [20574.327479] [20574.333391] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20574.352128] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [20574.370864] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20574.376082] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [20574.381301] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20574.385131] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [20574.386621] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20574.393744] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20574.400870] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20574.407995] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20574.415124] [20574.422243] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20574.468125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20574.473863] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20574.480986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20574.488110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20574.495235] Call Trace: [20574.497681] [20574.499690] ? watchdog_timer_fn+0x1b4/0x220 [20574.503962] ? __hrtimer_run_queues+0x1d3/0x360 [20574.508490] ? hrtimer_interrupt+0x104/0x240 [20574.512759] ? __local_bh_enable+0x50/0x60 [20574.516852] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20574.522078] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20574.527032] [20574.529132] [20574.531228] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20574.536536] ? ceph_check_delayed_caps+0x5f/0x270 [ceph] [20574.541854] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [20574.547177] delayed_work+0x1d8/0x280 [ceph] [20574.551460] ? process_scheduled_works+0x22a/0x510 [20574.556250] process_scheduled_works+0x2a4/0x510 [20574.560864] worker_thread+0x188/0x340 [20574.564613] ? rescuer_thread+0x310/0x310 [20574.568617] kthread+0xf5/0x130 [20574.571754] ? kthread_complete_and_exit+0x20/0x20 [20574.576539] ret_from_fork+0x31/0x40 [20574.580117] ? kthread_complete_and_exit+0x20/0x20 [20574.584901] ret_from_fork_asm+0x11/0x20 [20574.588825] [20600.978147] rcu: INFO: rcu_preempt self-detected stall on CPU [20600.983909] rcu: 7-....: (8414714 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/8413452 fqs=4221987 [20600.994333] rcu: (t=8664212 jiffies g=1309609 q=305057 ncpus=8) [20601.000330] Sending NMI from CPU 7 to CPUs 0: [20601.004682] NMI backtrace for cpu 0 [20601.004684] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20601.004686] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20601.004686] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20601.004690] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20601.004691] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20601.004693] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20601.004693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20601.004694] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20601.004695] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20601.004696] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20601.004696] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20601.004698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20601.004698] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20601.004699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20601.004700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20601.004700] Call Trace: [20601.004701] [20601.004702] ? nmi_cpu_backtrace+0xad/0x120 [20601.004706] ? nmi_cpu_backtrace_handler+0x11/0x20 [20601.004707] ? nmi_handle+0xd4/0x260 [20601.004712] ? default_do_nmi+0x49/0xf0 [20601.004713] ? exc_nmi+0x11e/0x150 [20601.004715] ? end_repeat_nmi+0x16/0x67 [20601.004719] ? queued_spin_lock_slowpath+0x18/0x2e0 [20601.004721] ? queued_spin_lock_slowpath+0x18/0x2e0 [20601.004724] ? queued_spin_lock_slowpath+0x18/0x2e0 [20601.004726] [20601.004727] [20601.004728] do_raw_spin_lock+0xb5/0xc0 [20601.004730] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20601.004745] try_get_cap_refs+0x159/0x880 [ceph] [20601.004760] __ceph_get_caps+0x4a1/0x760 [ceph] [20601.004773] ? __file_remove_privs+0x89/0x110 [20601.004775] ? wake_bit_function+0x70/0x70 [20601.004778] ? ceph_write_iter+0x260/0x770 [ceph] [20601.004788] ceph_write_iter+0x260/0x770 [ceph] [20601.004800] ? lock_is_held_type+0xaa/0x120 [20601.004803] vfs_write+0x324/0x4f0 [20601.004807] ksys_pwrite64+0x65/0xa0 [20601.004809] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20601.004812] do_syscall_64+0x38/0x80 [20601.004814] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20601.004816] RIP: 0033:0x7f48f0eeb93b [20601.004817] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20601.004818] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [20601.004819] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20601.004820] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20601.004821] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20601.004822] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20601.004822] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20601.004825] [20601.005681] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20601.309492] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20601.316963] Workqueue: events delayed_work [ceph] [20601.321679] RIP: 0010:lock_acquire+0xf0/0x2b0 [20601.326029] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20601.344767] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20601.349986] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20601.357109] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20601.364232] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20601.371358] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20601.378480] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20601.385605] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20601.393684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20601.399420] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20601.406546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20601.413669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20601.420793] Call Trace: [20601.423239] [20601.425249] ? rcu_dump_cpu_stacks+0xd1/0x180 [20601.429607] ? rcu_sched_clock_irq+0x633/0x1390 [20601.434132] ? lock_release+0xc5/0x260 [20601.437875] ? lock_is_held_type+0xaa/0x120 [20601.442055] ? lock_is_held_type+0xaa/0x120 [20601.446233] ? tick_sched_do_timer+0xa0/0xa0 [20601.450502] ? rcu_read_lock_sched_held+0x56/0x90 [20601.455202] ? tick_sched_do_timer+0xa0/0xa0 [20601.459473] ? update_process_times+0x5f/0x90 [20601.463823] ? tick_sched_handle.isra.16+0x1e/0x50 [20601.468609] ? tick_sched_timer+0x6f/0x80 [20601.472612] ? __hrtimer_run_queues+0x1d3/0x360 [20601.477139] ? hrtimer_interrupt+0x104/0x240 [20601.481408] ? __local_bh_enable+0x50/0x60 [20601.485503] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20601.490726] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20601.495682] [20601.497780] [20601.499877] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20601.505185] ? lock_acquire+0xf0/0x2b0 [20601.508935] ? igrab+0x43/0x50 [20601.511987] ? lock_release+0xc5/0x260 [20601.515732] _raw_spin_lock+0x30/0x40 [20601.519395] ? igrab+0x1d/0x50 [20601.522446] igrab+0x1d/0x50 [20601.525325] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20601.530474] delayed_work+0x1d8/0x280 [ceph] [20601.534756] ? process_scheduled_works+0x22a/0x510 [20601.539547] process_scheduled_works+0x2a4/0x510 [20601.544162] worker_thread+0x188/0x340 [20601.547910] ? rescuer_thread+0x310/0x310 [20601.551913] kthread+0xf5/0x130 [20601.555050] ? kthread_complete_and_exit+0x20/0x20 [20601.559836] ret_from_fork+0x31/0x40 [20601.563415] ? kthread_complete_and_exit+0x20/0x20 [20601.568198] ret_from_fork_asm+0x11/0x20 [20601.572122] [20625.819243] watchdog: BUG: soft lockup - CPU#0 stuck for 8095s! [dbench:78683] [20625.826462] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20625.831243] watchdog: BUG: soft lockup - CPU#7 stuck for 17092s! [kworker/7:4:78808] [20625.881917] irq event stamp: 30909078 [20625.881919] hardirqs last enabled at (30909077): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20625.889651] Modules linked in: [20625.893310] hardirqs last disabled at (30909078): [] sysvec_apic_timer_interrupt+0xe/0x90 [20625.903466] ceph [20625.906518] softirqs last enabled at (30909026): [] __do_softirq+0x2be/0x371 [20625.916241] libceph [20625.918167] softirqs last disabled at (30909021): [] irq_exit_rcu+0xb0/0xd0 [20625.926848] dns_resolver fscache [20625.929034] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20625.937543] netfs veth [20625.940855] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20625.949800] nft_chain_nat [20625.952243] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [20625.959714] xt_MASQUERADE [20625.962419] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [20625.967896] nf_nat nf_conntrack nf_defrag_ipv6 [20625.970634] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20625.989372] nf_defrag_ipv4 nft_compat [20625.993896] [20625.999130] nf_tables nfnetlink [20626.002875] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20626.004365] bridge [20626.007590] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20626.014714] stp llc [20626.016811] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20626.023937] xfs libcrc32c [20626.026121] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20626.033245] sunrpc [20626.035948] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20626.043071] intel_rapl_msr intel_rapl_common [20626.045170] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20626.052294] sb_edac x86_pkg_temp_thermal [20626.056643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20626.064722] coretemp kvm_intel [20626.068726] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20626.074463] kvm irqbypass [20626.077599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20626.084724] i2c_i801 [20626.087429] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20626.094553] iTCO_wdt iTCO_vendor_support [20626.096823] Call Trace: [20626.103947] i2c_smbus crct10dif_pclmul [20626.107955] [20626.110397] joydev crc32_pclmul [20626.114228] ? watchdog_timer_fn+0x1b4/0x220 [20626.116238] ipmi_ssif ghash_clmulni_intel [20626.119462] ? __hrtimer_run_queues+0x1d3/0x360 [20626.123724] mei_me mei [20626.127819] ? hrtimer_interrupt+0x104/0x240 [20626.132340] lpc_ich [20626.134785] ? __local_bh_enable+0x50/0x60 [20626.139049] sha512_ssse3 acpi_ipmi [20626.141236] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20626.145322] mfd_core [20626.148807] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20626.154024] wmi ipmi_si [20626.156295] [20626.161253] ipmi_devintf [20626.163783] [20626.163784] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20626.165880] ipmi_msghandler [20626.168501] ? queued_spin_lock_slowpath+0x2b/0x2e0 [20626.170596] ioatdma acpi_pad [20626.175903] do_raw_spin_lock+0xb5/0xc0 [20626.178777] acpi_power_meter [20626.183650] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20626.186611] ixgbe igb [20626.190445] try_get_cap_refs+0x159/0x880 [ceph] [20626.193407] mdio crc32c_intel [20626.198195] __ceph_get_caps+0x4a1/0x760 [ceph] [20626.200548] nvme [20626.205164] ? __file_remove_privs+0x89/0x110 [20626.208210] ptp nvme_core [20626.212734] ? wake_bit_function+0x70/0x70 [20626.214656] i2c_algo_bit [20626.219010] ? ceph_write_iter+0x260/0x770 [ceph] [20626.221712] pps_core [20626.225804] ceph_write_iter+0x260/0x770 [ceph] [20626.228422] dca fuse [20626.233122] ? lock_is_held_type+0xaa/0x120 [20626.235389] [20626.235390] irq event stamp: 87627306 [20626.239925] vfs_write+0x324/0x4f0 [20626.242193] hardirqs last enabled at (87627305): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20626.246375] ksys_pwrite64+0x65/0xa0 [20626.247861] hardirqs last disabled at (87627306): [] sysvec_apic_timer_interrupt+0xe/0x90 [20626.251518] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20626.254929] softirqs last enabled at (87627304): [] __do_softirq+0x2be/0x371 [20626.265092] do_syscall_64+0x38/0x80 [20626.268659] softirqs last disabled at (87627299): [] irq_exit_rcu+0xb0/0xd0 [20626.278387] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20626.283516] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20626.292201] RIP: 0033:0x7f48f0eeb93b [20626.295772] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20626.304280] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20626.309326] Workqueue: events delayed_work [ceph] [20626.318702] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20626.322273] [20626.322274] RIP: 0010:lock_acquire+0xf0/0x2b0 [20626.329745] ORIG_RAX: 0000000000000012 [20626.329746] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20626.348482] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20626.353179] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20626.358398] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20626.359889] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20626.364239] [20626.364240] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20626.368068] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20626.375192] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20626.393935] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20626.401063] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20626.406286] [20626.413405] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20626.459826] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20626.466949] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20626.475026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20626.480765] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20626.487888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20626.495011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20626.502136] Call Trace: [20626.504583] [20626.506591] ? watchdog_timer_fn+0x1b4/0x220 [20626.510857] ? __hrtimer_run_queues+0x1d3/0x360 [20626.515382] ? hrtimer_interrupt+0x104/0x240 [20626.519648] ? __local_bh_enable+0x50/0x60 [20626.523747] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20626.528969] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20626.533931] [20626.536033] [20626.538131] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20626.543437] ? lock_acquire+0xf0/0x2b0 [20626.547189] ? igrab+0x43/0x50 [20626.550247] ? lock_release+0xc5/0x260 [20626.553993] _raw_spin_lock+0x30/0x40 [20626.557658] ? igrab+0x1d/0x50 [20626.560708] igrab+0x1d/0x50 [20626.563585] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20626.568737] delayed_work+0x1d8/0x280 [ceph] [20626.573018] ? process_scheduled_works+0x22a/0x510 [20626.577805] process_scheduled_works+0x2a4/0x510 [20626.582422] worker_thread+0x188/0x340 [20626.586171] ? rescuer_thread+0x310/0x310 [20626.590173] kthread+0xf5/0x130 [20626.593313] ? kthread_complete_and_exit+0x20/0x20 [20626.598096] ret_from_fork+0x31/0x40 [20626.601667] ? kthread_complete_and_exit+0x20/0x20 [20626.606452] ret_from_fork_asm+0x11/0x20 [20626.610384] [20653.819765] watchdog: BUG: soft lockup - CPU#0 stuck for 8121s! [dbench:78683] [20653.826983] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20653.831765] watchdog: BUG: soft lockup - CPU#7 stuck for 17118s! [kworker/7:4:78808] [20653.882442] irq event stamp: 30967252 [20653.882443] hardirqs last enabled at (30967251): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20653.890172] Modules linked in: [20653.893830] hardirqs last disabled at (30967252): [] sysvec_apic_timer_interrupt+0xe/0x90 [20653.903989] ceph [20653.907039] softirqs last enabled at (30967130): [] __do_softirq+0x2be/0x371 [20653.916764] libceph [20653.918687] softirqs last disabled at (30967125): [] irq_exit_rcu+0xb0/0xd0 [20653.927371] dns_resolver fscache [20653.929554] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20653.938066] netfs veth [20653.941378] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20653.950321] nft_chain_nat [20653.952766] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20653.960235] xt_MASQUERADE nf_nat [20653.962939] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20653.968418] nf_conntrack nf_defrag_ipv6 [20653.971746] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20653.990484] nf_defrag_ipv4 nft_compat [20653.994400] [20653.994400] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20653.999617] nf_tables [20654.003363] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20654.004854] nfnetlink bridge [20654.011976] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20654.014334] stp llc [20654.021457] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20654.024439] xfs libcrc32c sunrpc [20654.031565] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20654.033748] intel_rapl_msr [20654.040871] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20654.044183] intel_rapl_common sb_edac [20654.051306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20654.054097] x86_pkg_temp_thermal [20654.062175] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20654.065918] coretemp kvm_intel [20654.071656] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20654.074967] kvm [20654.082092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20654.085228] irqbypass i2c_i801 [20654.092352] Call Trace: [20654.092354] [20654.094191] iTCO_wdt [20654.101316] ? watchdog_timer_fn+0x1b4/0x220 [20654.104453] iTCO_vendor_support i2c_smbus [20654.106896] ? __hrtimer_run_queues+0x1d3/0x360 [20654.108906] crct10dif_pclmul joydev [20654.111181] ? hrtimer_interrupt+0x104/0x240 [20654.115445] crc32_pclmul ipmi_ssif [20654.119533] ? __local_bh_enable+0x50/0x60 [20654.124056] ghash_clmulni_intel mei_me [20654.127631] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20654.131891] mei lpc_ich [20654.135375] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20654.139467] sha512_ssse3 acpi_ipmi [20654.143297] [20654.148513] mfd_core wmi [20654.151044] [20654.151045] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20654.156003] ipmi_si ipmi_devintf [20654.159489] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20654.161583] ipmi_msghandler [20654.164203] do_raw_spin_lock+0xb5/0xc0 [20654.166298] ioatdma acpi_pad [20654.171605] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20654.174913] acpi_power_meter [20654.179788] try_get_cap_refs+0x159/0x880 [ceph] [20654.182662] ixgbe igb [20654.186495] __ceph_get_caps+0x4a1/0x760 [ceph] [20654.189457] mdio crc32c_intel [20654.194246] ? __file_remove_privs+0x89/0x110 [20654.197204] nvme [20654.201816] ? wake_bit_function+0x70/0x70 [20654.204172] ptp nvme_core [20654.208698] ? ceph_write_iter+0x260/0x770 [ceph] [20654.211746] i2c_algo_bit [20654.216100] ceph_write_iter+0x260/0x770 [ceph] [20654.218022] pps_core [20654.222118] ? lock_is_held_type+0xaa/0x120 [20654.224818] dca fuse [20654.229516] vfs_write+0x324/0x4f0 [20654.232131] [20654.232132] irq event stamp: 87848084 [20654.236659] ksys_pwrite64+0x65/0xa0 [20654.238927] hardirqs last enabled at (87848083): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20654.243105] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20654.245373] hardirqs last disabled at (87848084): [] sysvec_apic_timer_interrupt+0xe/0x90 [20654.248772] do_syscall_64+0x38/0x80 [20654.250263] softirqs last enabled at (87848082): [] __do_softirq+0x2be/0x371 [20654.253923] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20654.257491] softirqs last disabled at (87848077): [] irq_exit_rcu+0xb0/0xd0 [20654.267647] RIP: 0033:0x7f48f0eeb93b [20654.272779] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20654.282504] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20654.286074] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20654.294759] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20654.299804] Workqueue: events delayed_work [ceph] [20654.308312] ORIG_RAX: 0000000000000012 [20654.308313] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20654.311883] [20654.311884] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [20654.321262] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20654.339999] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [20654.347470] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20654.352686] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [20654.357384] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20654.361216] [20654.361217] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [20654.368339] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20654.369830] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [20654.374455] [20654.381574] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [20654.451958] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20654.459083] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20654.466207] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20654.474284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20654.480022] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20654.487144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20654.494287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20654.501409] Call Trace: [20654.503854] [20654.505867] ? watchdog_timer_fn+0x1b4/0x220 [20654.510131] ? __hrtimer_run_queues+0x1d3/0x360 [20654.514657] ? hrtimer_interrupt+0x104/0x240 [20654.518925] ? __local_bh_enable+0x50/0x60 [20654.523019] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20654.528243] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20654.533201] [20654.535296] [20654.537394] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20654.542703] ? do_raw_spin_lock+0x42/0xc0 [20654.546712] igrab+0x1d/0x50 [20654.549591] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20654.554741] delayed_work+0x1d8/0x280 [ceph] [20654.559021] ? process_scheduled_works+0x22a/0x510 [20654.563812] process_scheduled_works+0x2a4/0x510 [20654.568428] worker_thread+0x188/0x340 [20654.572194] ? rescuer_thread+0x310/0x310 [20654.576205] kthread+0xf5/0x130 [20654.579343] ? kthread_complete_and_exit+0x20/0x20 [20654.584126] ret_from_fork+0x31/0x40 [20654.587698] ? kthread_complete_and_exit+0x20/0x20 [20654.592482] ret_from_fork_asm+0x11/0x20 [20654.596406] [20681.820288] watchdog: BUG: soft lockup - CPU#0 stuck for 8147s! [dbench:78683] [20681.827506] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20681.832288] watchdog: BUG: soft lockup - CPU#7 stuck for 17144s! [kworker/7:4:78808] [20681.882963] irq event stamp: 31025484 [20681.882965] hardirqs last enabled at (31025483): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20681.890696] Modules linked in: [20681.894352] hardirqs last disabled at (31025484): [] sysvec_apic_timer_interrupt+0xe/0x90 [20681.904509] ceph [20681.907562] softirqs last enabled at (31025458): [] __do_softirq+0x2be/0x371 [20681.917286] libceph [20681.919209] softirqs last disabled at (31025453): [] irq_exit_rcu+0xb0/0xd0 [20681.927894] dns_resolver fscache [20681.930078] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20681.938588] netfs [20681.941899] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20681.950842] veth nft_chain_nat [20681.952855] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20681.960324] xt_MASQUERADE nf_nat [20681.963463] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20681.968938] nf_conntrack nf_defrag_ipv6 [20681.972267] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20681.991006] nf_defrag_ipv4 nft_compat [20681.994923] [20682.000141] nf_tables nfnetlink [20682.003885] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20682.005374] bridge stp [20682.008598] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20682.015722] llc xfs [20682.018167] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20682.025291] libcrc32c sunrpc [20682.027475] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20682.034599] intel_rapl_msr intel_rapl_common [20682.037564] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20682.044687] sb_edac x86_pkg_temp_thermal [20682.049038] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20682.056162] coretemp kvm_intel [20682.060166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20682.068245] kvm [20682.071380] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20682.077119] irqbypass i2c_i801 [20682.078971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20682.086098] iTCO_wdt iTCO_vendor_support [20682.089234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20682.096360] i2c_smbus [20682.100364] Call Trace: [20682.100366] [20682.107487] crct10dif_pclmul [20682.109846] ? watchdog_timer_fn+0x1b4/0x220 [20682.112288] joydev crc32_pclmul [20682.114299] ? __hrtimer_run_queues+0x1d3/0x360 [20682.117264] ipmi_ssif ghash_clmulni_intel [20682.121532] ? hrtimer_interrupt+0x104/0x240 [20682.124752] mei_me mei [20682.129276] ? __local_bh_enable+0x50/0x60 [20682.133367] lpc_ich [20682.137632] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20682.140074] sha512_ssse3 [20682.144167] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20682.146350] acpi_ipmi mfd_core [20682.151566] [20682.154183] wmi ipmi_si [20682.159143] [20682.159144] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20682.162279] ipmi_devintf [20682.164380] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20682.166907] ipmi_msghandler [20682.169005] do_raw_spin_lock+0xb5/0xc0 [20682.174307] ioatdma acpi_pad [20682.176929] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20682.181797] acpi_power_meter [20682.184689] try_get_cap_refs+0x159/0x880 [ceph] [20682.188512] ixgbe igb [20682.191484] __ceph_get_caps+0x4a1/0x760 [ceph] [20682.196262] mdio crc32c_intel [20682.199230] ? __file_remove_privs+0x89/0x110 [20682.203837] nvme [20682.206194] ? wake_bit_function+0x70/0x70 [20682.210717] ptp [20682.213772] ? ceph_write_iter+0x260/0x770 [ceph] [20682.218120] nvme_core [20682.220044] ceph_write_iter+0x260/0x770 [ceph] [20682.224133] i2c_algo_bit pps_core [20682.225975] ? lock_is_held_type+0xaa/0x120 [20682.230687] dca [20682.233045] vfs_write+0x324/0x4f0 [20682.237568] fuse [20682.240974] ksys_pwrite64+0x65/0xa0 [20682.245152] [20682.245153] irq event stamp: 88068974 [20682.246989] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20682.250385] hardirqs last enabled at (88068973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20682.252310] do_syscall_64+0x38/0x80 [20682.255879] hardirqs last disabled at (88068974): [] sysvec_apic_timer_interrupt+0xe/0x90 [20682.257372] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20682.261029] softirqs last enabled at (88068972): [] __do_softirq+0x2be/0x371 [20682.266160] RIP: 0033:0x7f48f0eeb93b [20682.276316] softirqs last disabled at (88068967): [] irq_exit_rcu+0xb0/0xd0 [20682.279888] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20682.289612] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20682.294656] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20682.303341] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20682.306910] ORIG_RAX: 0000000000000012 [20682.315421] Workqueue: events delayed_work [ceph] [20682.334160] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20682.343536] [20682.343537] RIP: 0010:lock_acquire+0xf0/0x2b0 [20682.348754] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20682.356226] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20682.360056] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20682.364753] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20682.371878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20682.373368] [20682.373369] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20682.377718] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20682.384844] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20682.403586] [20682.410706] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20682.455217] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20682.462342] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20682.469466] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20682.477544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20682.483280] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20682.490405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20682.497529] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20682.504654] Call Trace: [20682.507098] [20682.509109] ? watchdog_timer_fn+0x1b4/0x220 [20682.513373] ? __hrtimer_run_queues+0x1d3/0x360 [20682.517900] ? hrtimer_interrupt+0x104/0x240 [20682.522168] ? __local_bh_enable+0x50/0x60 [20682.526261] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20682.531488] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20682.536443] [20682.538540] [20682.540638] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20682.545946] ? lock_acquire+0xf0/0x2b0 [20682.549697] ? igrab+0x43/0x50 [20682.552754] ? lock_release+0xc5/0x260 [20682.556500] _raw_spin_lock+0x30/0x40 [20682.560157] ? igrab+0x1d/0x50 [20682.563208] igrab+0x1d/0x50 [20682.566085] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20682.571234] delayed_work+0x1d8/0x280 [ceph] [20682.575516] ? process_scheduled_works+0x22a/0x510 [20682.580306] process_scheduled_works+0x2a4/0x510 [20682.584923] worker_thread+0x188/0x340 [20682.588672] ? rescuer_thread+0x310/0x310 [20682.592699] kthread+0xf5/0x130 [20682.595838] ? kthread_complete_and_exit+0x20/0x20 [20682.600623] ret_from_fork+0x31/0x40 [20682.604202] ? kthread_complete_and_exit+0x20/0x20 [20682.608986] ret_from_fork_asm+0x11/0x20 [20682.612917] [20709.820810] watchdog: BUG: soft lockup - CPU#0 stuck for 8173s! [dbench:78683] [20709.828028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20709.832810] watchdog: BUG: soft lockup - CPU#7 stuck for 17170s! [kworker/7:4:78808] [20709.883484] irq event stamp: 31083628 [20709.883486] hardirqs last enabled at (31083627): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20709.891217] Modules linked in: [20709.894875] hardirqs last disabled at (31083628): [] sysvec_apic_timer_interrupt+0xe/0x90 [20709.905032] ceph [20709.908084] softirqs last enabled at (31083506): [] __do_softirq+0x2be/0x371 [20709.917808] libceph [20709.919731] softirqs last disabled at (31083501): [] irq_exit_rcu+0xb0/0xd0 [20709.928415] dns_resolver fscache [20709.930600] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20709.939111] netfs veth [20709.942422] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20709.951365] nft_chain_nat xt_MASQUERADE [20709.953809] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20709.961280] nf_nat nf_conntrack [20709.965205] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20709.970683] nf_defrag_ipv6 nf_defrag_ipv4 [20709.973906] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20709.992644] nft_compat nf_tables [20709.996736] [20709.996736] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20710.001954] nfnetlink bridge [20710.005263] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20710.006755] stp llc [20710.013879] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20710.016843] xfs [20710.023966] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20710.026150] libcrc32c sunrpc [20710.033275] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20710.035112] intel_rapl_msr intel_rapl_common [20710.042238] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20710.045207] sb_edac x86_pkg_temp_thermal [20710.052333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20710.056685] coretemp [20710.064762] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20710.068766] kvm_intel kvm [20710.074503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20710.076773] irqbypass i2c_i801 [20710.083898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20710.086602] iTCO_wdt iTCO_vendor_support [20710.093725] Call Trace: [20710.093727] [20710.096865] i2c_smbus crct10dif_pclmul [20710.103988] ? watchdog_timer_fn+0x1b4/0x220 [20710.107991] joydev crc32_pclmul [20710.110438] ? __hrtimer_run_queues+0x1d3/0x360 [20710.112446] ipmi_ssif ghash_clmulni_intel [20710.116281] ? hrtimer_interrupt+0x104/0x240 [20710.120542] mei_me mei [20710.123764] ? __local_bh_enable+0x50/0x60 [20710.128290] lpc_ich [20710.132384] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20710.136644] sha512_ssse3 [20710.139090] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20710.143178] acpi_ipmi mfd_core [20710.145363] [20710.150581] wmi ipmi_si [20710.153209] [20710.158163] ipmi_devintf ipmi_msghandler [20710.161301] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20710.163400] ioatdma acpi_pad [20710.165934] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20710.168026] acpi_power_meter ixgbe [20710.172032] do_raw_spin_lock+0xb5/0xc0 [20710.177335] igb mdio [20710.180300] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20710.185170] crc32c_intel nvme [20710.188655] try_get_cap_refs+0x159/0x880 [ceph] [20710.192485] ptp nvme_core [20710.194778] __ceph_get_caps+0x4a1/0x760 [ceph] [20710.199557] i2c_algo_bit pps_core [20710.202611] ? __file_remove_privs+0x89/0x110 [20710.207219] dca fuse [20710.209923] ? wake_bit_function+0x70/0x70 [20710.214447] [20710.214448] irq event stamp: 88289850 [20710.217846] ? ceph_write_iter+0x260/0x770 [ceph] [20710.222213] hardirqs last enabled at (88289849): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20710.224493] ceph_write_iter+0x260/0x770 [ceph] [20710.228583] hardirqs last disabled at (88289850): [] sysvec_apic_timer_interrupt+0xe/0x90 [20710.230075] ? lock_is_held_type+0xaa/0x120 [20710.233731] softirqs last enabled at (88289848): [] __do_softirq+0x2be/0x371 [20710.238430] vfs_write+0x324/0x4f0 [20710.248586] softirqs last disabled at (88289843): [] irq_exit_rcu+0xb0/0xd0 [20710.253114] ksys_pwrite64+0x65/0xa0 [20710.262834] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20710.267012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20710.275695] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20710.279093] do_syscall_64+0x38/0x80 [20710.287604] Workqueue: events delayed_work [ceph] [20710.291174] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20710.300551] [20710.305683] RIP: 0033:0x7f48f0eeb93b [20710.313154] RIP: 0010:_raw_spin_unlock+0x23/0x40 [20710.316725] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20710.321421] Code: 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff <65> ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b c3 66 66 2e 0f [20710.326464] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20710.327957] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [20710.331527] ORIG_RAX: 0000000000000012 [20710.336137] [20710.336138] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [20710.354874] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20710.373614] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [20710.378829] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20710.384048] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [20710.387878] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20710.389368] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20710.396496] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20710.403618] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20710.410742] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20710.417865] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20710.424994] [20710.432113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20710.476619] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20710.483742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20710.490867] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20710.497991] Call Trace: [20710.500436] [20710.502447] ? watchdog_timer_fn+0x1b4/0x220 [20710.506712] ? __hrtimer_run_queues+0x1d3/0x360 [20710.511238] ? hrtimer_interrupt+0x104/0x240 [20710.515507] ? __local_bh_enable+0x50/0x60 [20710.519599] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20710.524824] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20710.529781] [20710.531877] [20710.533976] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20710.539284] ? _raw_spin_unlock+0x23/0x40 [20710.543294] igrab+0x43/0x50 [20710.546181] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20710.551346] delayed_work+0x1d8/0x280 [ceph] [20710.555627] ? process_scheduled_works+0x22a/0x510 [20710.560418] process_scheduled_works+0x2a4/0x510 [20710.565034] worker_thread+0x188/0x340 [20710.568784] ? rescuer_thread+0x310/0x310 [20710.572794] kthread+0xf5/0x130 [20710.575933] ? kthread_complete_and_exit+0x20/0x20 [20710.580717] ret_from_fork+0x31/0x40 [20710.584287] ? kthread_complete_and_exit+0x20/0x20 [20710.589070] ret_from_fork_asm+0x11/0x20 [20710.592994] [20737.821333] watchdog: BUG: soft lockup - CPU#0 stuck for 8199s! [dbench:78683] [20737.828552] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20737.833334] watchdog: BUG: soft lockup - CPU#7 stuck for 17196s! [kworker/7:4:78808] [20737.884009] irq event stamp: 31141794 [20737.891740] Modules linked in: ceph libceph dns_resolver fscache netfs veth [20737.895397] hardirqs last enabled at (31141793): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20737.902355] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20737.912513] hardirqs last disabled at (31141794): [] sysvec_apic_timer_interrupt+0xe/0x90 [20737.912516] nf_defrag_ipv6 [20737.912518] softirqs last enabled at (31141718): [] __do_softirq+0x2be/0x371 [20737.918164] nf_defrag_ipv4 nft_compat [20737.927889] softirqs last disabled at (31141713): [] irq_exit_rcu+0xb0/0xd0 [20737.930679] nf_tables [20737.939365] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20737.943107] nfnetlink bridge [20737.951620] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20737.953975] stp llc [20737.962921] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [20737.965884] xfs libcrc32c [20737.973355] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [20737.975538] sunrpc intel_rapl_msr [20737.981016] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20737.983729] intel_rapl_common [20738.002468] [20738.005863] sb_edac x86_pkg_temp_thermal [20738.011082] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20738.014133] coretemp kvm_intel [20738.015622] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20738.019627] kvm irqbypass [20738.026752] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20738.029888] i2c_i801 iTCO_wdt [20738.037019] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20738.039730] iTCO_vendor_support [20738.046857] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20738.049910] i2c_smbus crct10dif_pclmul [20738.057034] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20738.060256] joydev [20738.067380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20738.071212] crc32_pclmul [20738.079289] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20738.081386] ipmi_ssif [20738.087123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20738.089741] ghash_clmulni_intel [20738.096866] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20738.099222] mei_me [20738.106348] Call Trace: [20738.106351] [20738.109573] mei lpc_ich [20738.116697] ? watchdog_timer_fn+0x1b4/0x220 [20738.118794] sha512_ssse3 acpi_ipmi [20738.121238] ? __hrtimer_run_queues+0x1d3/0x360 [20738.123246] mfd_core [20738.125782] ? hrtimer_interrupt+0x104/0x240 [20738.130043] wmi ipmi_si [20738.133526] ? __local_bh_enable+0x50/0x60 [20738.138051] ipmi_devintf ipmi_msghandler [20738.140323] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20738.144586] ioatdma acpi_pad [20738.147117] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20738.151206] acpi_power_meter ixgbe [20738.155212] [20738.160429] igb [20738.163393] [20738.163395] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20738.168350] mdio [20738.171838] ? queued_spin_lock_slowpath+0x2b/0x2e0 [20738.173931] crc32c_intel [20738.175771] do_raw_spin_lock+0xb5/0xc0 [20738.177866] nvme ptp [20738.183173] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20738.185094] nvme_core [20738.189967] try_get_cap_refs+0x159/0x880 [ceph] [20738.192582] i2c_algo_bit pps_core [20738.196419] __ceph_get_caps+0x4a1/0x760 [ceph] [20738.198683] dca fuse [20738.203473] ? __file_remove_privs+0x89/0x110 [20738.205825] [20738.205826] irq event stamp: 88511132 [20738.210437] ? wake_bit_function+0x70/0x70 [20738.213834] hardirqs last enabled at (88511131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20738.218360] ? ceph_write_iter+0x260/0x770 [ceph] [20738.220628] hardirqs last disabled at (88511132): [] sysvec_apic_timer_interrupt+0xe/0x90 [20738.224978] ceph_write_iter+0x260/0x770 [ceph] [20738.226469] softirqs last enabled at (88511130): [] __do_softirq+0x2be/0x371 [20738.230133] ? lock_is_held_type+0xaa/0x120 [20738.234219] softirqs last disabled at (88511125): [] irq_exit_rcu+0xb0/0xd0 [20738.244379] vfs_write+0x324/0x4f0 [20738.249073] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20738.258802] ksys_pwrite64+0x65/0xa0 [20738.263321] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20738.272017] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20738.276191] Workqueue: events delayed_work [ceph] [20738.284703] do_syscall_64+0x38/0x80 [20738.288098] [20738.288099] RIP: 0010:lock_acquire+0xf0/0x2b0 [20738.297478] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20738.301046] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20738.308519] RIP: 0033:0x7f48f0eeb93b [20738.313650] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20738.318347] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20738.321917] [20738.323408] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20738.327759] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20738.332803] ORIG_RAX: 0000000000000012 [20738.351539] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20738.355112] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20738.360329] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20738.379066] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20738.380557] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20738.385774] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20738.392899] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20738.396737] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20738.403862] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20738.410987] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20738.418110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20738.425241] [20738.432360] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20738.483986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20738.491111] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20738.498235] Call Trace: [20738.500680] [20738.502692] ? watchdog_timer_fn+0x1b4/0x220 [20738.506956] ? __hrtimer_run_queues+0x1d3/0x360 [20738.511483] ? hrtimer_interrupt+0x104/0x240 [20738.515751] ? __local_bh_enable+0x50/0x60 [20738.519845] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20738.525068] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20738.530027] [20738.532123] [20738.534219] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20738.539530] ? lock_acquire+0xf0/0x2b0 [20738.543278] ? igrab+0x43/0x50 [20738.546330] ? lock_release+0xc5/0x260 [20738.550074] _raw_spin_lock+0x30/0x40 [20738.553743] ? igrab+0x1d/0x50 [20738.556797] igrab+0x1d/0x50 [20738.559678] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20738.564827] delayed_work+0x1d8/0x280 [ceph] [20738.569107] ? process_scheduled_works+0x22a/0x510 [20738.573896] process_scheduled_works+0x2a4/0x510 [20738.578513] worker_thread+0x188/0x340 [20738.582263] ? rescuer_thread+0x310/0x310 [20738.586273] kthread+0xf5/0x130 [20738.589411] ? kthread_complete_and_exit+0x20/0x20 [20738.594195] ret_from_fork+0x31/0x40 [20738.597775] ? kthread_complete_and_exit+0x20/0x20 [20738.602560] ret_from_fork_asm+0x11/0x20 [20738.606490] [20765.821857] watchdog: BUG: soft lockup - CPU#0 stuck for 8225s! [dbench:78683] [20765.829071] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20765.833857] watchdog: BUG: soft lockup - CPU#7 stuck for 17222s! [kworker/7:4:78808] [20765.884512] irq event stamp: 31199914 [20765.884514] hardirqs last enabled at (31199913): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20765.892253] Modules linked in: [20765.895910] hardirqs last disabled at (31199914): [] sysvec_apic_timer_interrupt+0xe/0x90 [20765.906068] ceph [20765.909119] softirqs last enabled at (31199838): [] __do_softirq+0x2be/0x371 [20765.918842] libceph [20765.920766] softirqs last disabled at (31199833): [] irq_exit_rcu+0xb0/0xd0 [20765.929452] dns_resolver fscache [20765.931634] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20765.940145] netfs veth [20765.943455] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20765.952400] nft_chain_nat [20765.954843] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20765.962314] xt_MASQUERADE [20765.965020] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20765.970498] nf_nat nf_conntrack [20765.973202] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20765.991939] nf_defrag_ipv6 nf_defrag_ipv4 [20765.995162] [20765.995163] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20766.000380] nft_compat [20766.004472] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20766.005962] nf_tables nfnetlink [20766.013086] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20766.015542] bridge stp [20766.022663] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20766.025885] llc xfs [20766.033009] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20766.035455] libcrc32c [20766.042578] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20766.044763] sunrpc intel_rapl_msr [20766.051886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20766.054255] intel_rapl_common [20766.062329] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20766.065729] sb_edac x86_pkg_temp_thermal [20766.071465] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20766.074532] coretemp kvm_intel [20766.081656] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20766.085661] kvm irqbypass [20766.092786] Call Trace: [20766.092787] [20766.095922] i2c_i801 iTCO_wdt [20766.103048] ? watchdog_timer_fn+0x1b4/0x220 [20766.105750] iTCO_vendor_support i2c_smbus [20766.108197] ? __hrtimer_run_queues+0x1d3/0x360 [20766.110205] crct10dif_pclmul [20766.113259] ? hrtimer_interrupt+0x104/0x240 [20766.117537] joydev crc32_pclmul [20766.121630] ? __local_bh_enable+0x50/0x60 [20766.126154] ipmi_ssif ghash_clmulni_intel [20766.129118] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20766.133380] mei_me mei [20766.136606] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20766.140695] lpc_ich sha512_ssse3 [20766.144787] [20766.150004] acpi_ipmi [20766.152448] [20766.152450] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20766.157405] mfd_core wmi [20766.160720] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20766.162812] ipmi_si ipmi_devintf [20766.165172] do_raw_spin_lock+0xb5/0xc0 [20766.167268] ipmi_msghandler [20766.172573] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20766.175190] ioatdma acpi_pad [20766.180063] try_get_cap_refs+0x159/0x880 [ceph] [20766.183370] acpi_power_meter ixgbe [20766.187208] __ceph_get_caps+0x4a1/0x760 [ceph] [20766.190079] igb [20766.194867] ? __file_remove_privs+0x89/0x110 [20766.197827] mdio crc32c_intel [20766.202438] ? wake_bit_function+0x70/0x70 [20766.205921] nvme ptp [20766.210450] ? ceph_write_iter+0x260/0x770 [ceph] [20766.212283] nvme_core [20766.216634] ceph_write_iter+0x260/0x770 [ceph] [20766.219686] i2c_algo_bit [20766.223779] ? lock_is_held_type+0xaa/0x120 [20766.226045] pps_core [20766.230747] vfs_write+0x324/0x4f0 [20766.233102] dca fuse [20766.237630] ksys_pwrite64+0x65/0xa0 [20766.240258] [20766.244437] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20766.246710] irq event stamp: 88732046 [20766.250108] do_syscall_64+0x38/0x80 [20766.252375] hardirqs last enabled at (88732045): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20766.255947] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20766.257438] hardirqs last disabled at (88732046): [] sysvec_apic_timer_interrupt+0xe/0x90 [20766.262569] RIP: 0033:0x7f48f0eeb93b [20766.266226] softirqs last enabled at (88732044): [] __do_softirq+0x2be/0x371 [20766.269798] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20766.279955] softirqs last disabled at (88732039): [] irq_exit_rcu+0xb0/0xd0 [20766.284998] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20766.294723] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20766.298294] ORIG_RAX: 0000000000000012 [20766.306979] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20766.325715] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20766.334227] Workqueue: events delayed_work [ceph] [20766.339443] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20766.348820] [20766.352652] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20766.360123] RIP: 0010:lock_acquire+0xf0/0x2b0 [20766.367260] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20766.371962] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20766.379085] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20766.380575] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20766.387707] [20766.392050] [20766.392051] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20766.441062] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20766.448186] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20766.455310] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20766.462435] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20766.469560] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20766.477635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20766.483374] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20766.490497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20766.497622] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20766.504746] Call Trace: [20766.507191] [20766.509201] ? watchdog_timer_fn+0x1b4/0x220 [20766.513466] ? __hrtimer_run_queues+0x1d3/0x360 [20766.517992] ? hrtimer_interrupt+0x104/0x240 [20766.522264] ? __local_bh_enable+0x50/0x60 [20766.526365] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20766.531588] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20766.536544] [20766.538643] [20766.540739] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20766.546049] ? lock_acquire+0xf0/0x2b0 [20766.549798] ? igrab+0x43/0x50 [20766.552848] ? lock_release+0xc5/0x260 [20766.556594] _raw_spin_lock+0x30/0x40 [20766.560265] ? igrab+0x1d/0x50 [20766.563316] igrab+0x1d/0x50 [20766.566195] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20766.571345] delayed_work+0x1d8/0x280 [ceph] [20766.575625] ? process_scheduled_works+0x22a/0x510 [20766.580416] process_scheduled_works+0x2a4/0x510 [20766.585032] worker_thread+0x188/0x340 [20766.588781] ? rescuer_thread+0x310/0x310 [20766.592784] kthread+0xf5/0x130 [20766.595921] ? kthread_complete_and_exit+0x20/0x20 [20766.600707] ret_from_fork+0x31/0x40 [20766.604275] ? kthread_complete_and_exit+0x20/0x20 [20766.609061] ret_from_fork_asm+0x11/0x20 [20766.612982] [20793.822378] watchdog: BUG: soft lockup - CPU#0 stuck for 8251s! [dbench:78683] [20793.829593] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20793.834378] watchdog: BUG: soft lockup - CPU#7 stuck for 17248s! [kworker/7:4:78808] [20793.885052] irq event stamp: 31258122 [20793.885053] hardirqs last enabled at (31258121): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20793.892783] Modules linked in: [20793.896440] hardirqs last disabled at (31258122): [] sysvec_apic_timer_interrupt+0xe/0x90 [20793.906598] ceph [20793.909649] softirqs last enabled at (31258096): [] __do_softirq+0x2be/0x371 [20793.919373] libceph [20793.921296] softirqs last disabled at (31258091): [] irq_exit_rcu+0xb0/0xd0 [20793.929980] dns_resolver [20793.932167] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20793.940675] fscache netfs [20793.943292] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20793.952236] veth [20793.954942] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20793.962411] nft_chain_nat [20793.964336] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20793.969814] xt_MASQUERADE [20793.972517] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20793.991255] nf_nat [20793.993961] [20793.999176] nf_conntrack [20794.001275] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20794.002775] nf_defrag_ipv6 [20794.005392] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20794.012514] nf_defrag_ipv4 nft_compat [20794.015305] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20794.022429] nf_tables [20794.026173] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20794.033298] nfnetlink [20794.035656] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20794.042779] bridge [20794.045138] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20794.052262] stp [20794.054359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20794.062436] llc xfs [20794.064273] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20794.070012] libcrc32c sunrpc [20794.072194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20794.079321] intel_rapl_msr [20794.082282] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20794.089407] intel_rapl_common [20794.092197] Call Trace: [20794.092199] [20794.099323] sb_edac x86_pkg_temp_thermal [20794.102374] ? watchdog_timer_fn+0x1b4/0x220 [20794.104817] coretemp kvm_intel [20794.106829] ? __hrtimer_run_queues+0x1d3/0x360 [20794.110832] kvm irqbypass [20794.115100] ? hrtimer_interrupt+0x104/0x240 [20794.118233] i2c_i801 iTCO_wdt [20794.122773] ? __local_bh_enable+0x50/0x60 [20794.125479] iTCO_vendor_support [20794.129744] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20794.132793] i2c_smbus crct10dif_pclmul [20794.136884] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20794.140108] joydev crc32_pclmul [20794.145326] [20794.149157] ipmi_ssif ghash_clmulni_intel [20794.154113] [20794.154115] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20794.157338] mei_me mei [20794.159440] ? queued_spin_lock_slowpath+0x18/0x2e0 [20794.163525] lpc_ich [20794.165625] do_raw_spin_lock+0xb5/0xc0 [20794.170927] sha512_ssse3 acpi_ipmi [20794.173374] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20794.178242] mfd_core wmi [20794.180430] try_get_cap_refs+0x159/0x880 [ceph] [20794.184258] ipmi_si ipmi_devintf [20794.187747] __ceph_get_caps+0x4a1/0x760 [ceph] [20794.192526] ipmi_msghandler [20794.195146] ? __file_remove_privs+0x89/0x110 [20794.199754] ioatdma acpi_pad [20794.203081] ? wake_bit_function+0x70/0x70 [20794.207604] acpi_power_meter ixgbe [20794.210484] ? ceph_write_iter+0x260/0x770 [ceph] [20794.214833] igb mdio [20794.217797] ceph_write_iter+0x260/0x770 [ceph] [20794.221888] crc32c_intel [20794.225377] ? lock_is_held_type+0xaa/0x120 [20794.230069] nvme ptp [20794.232344] vfs_write+0x324/0x4f0 [20794.236866] nvme_core [20794.239487] ksys_pwrite64+0x65/0xa0 [20794.243659] i2c_algo_bit pps_core [20794.245930] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20794.249326] dca fuse [20794.251687] do_syscall_64+0x38/0x80 [20794.255255] [20794.255256] irq event stamp: 88952832 [20794.258654] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20794.263783] hardirqs last enabled at (88952831): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20794.266070] RIP: 0033:0x7f48f0eeb93b [20794.269642] hardirqs last disabled at (88952832): [] sysvec_apic_timer_interrupt+0xe/0x90 [20794.271132] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20794.274790] softirqs last enabled at (88952830): [] __do_softirq+0x2be/0x371 [20794.279834] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20794.289993] softirqs last disabled at (88952825): [] irq_exit_rcu+0xb0/0xd0 [20794.293562] ORIG_RAX: 0000000000000012 [20794.303286] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20794.322026] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20794.330708] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20794.335926] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20794.344437] Workqueue: events delayed_work [ceph] [20794.348267] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20794.357646] [20794.357647] RIP: 0010:lock_acquire+0xf0/0x2b0 [20794.364783] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20794.372259] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20794.379382] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20794.384078] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20794.391209] [20794.392693] [20794.392694] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20794.446062] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20794.453188] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20794.460312] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20794.467438] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20794.474560] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20794.482637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20794.488375] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20794.495499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20794.502623] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20794.509747] Call Trace: [20794.512195] [20794.514205] ? watchdog_timer_fn+0x1b4/0x220 [20794.518469] ? __hrtimer_run_queues+0x1d3/0x360 [20794.522994] ? hrtimer_interrupt+0x104/0x240 [20794.527254] ? __local_bh_enable+0x50/0x60 [20794.531349] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20794.536572] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20794.541529] [20794.543628] [20794.545726] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20794.551033] ? lock_acquire+0xf0/0x2b0 [20794.554787] ? igrab+0x43/0x50 [20794.557841] ? lock_release+0xc5/0x260 [20794.561586] _raw_spin_lock+0x30/0x40 [20794.565242] ? igrab+0x1d/0x50 [20794.568293] igrab+0x1d/0x50 [20794.571173] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20794.576321] delayed_work+0x1d8/0x280 [ceph] [20794.580603] ? process_scheduled_works+0x22a/0x510 [20794.585391] process_scheduled_works+0x2a4/0x510 [20794.590008] worker_thread+0x188/0x340 [20794.593759] ? rescuer_thread+0x310/0x310 [20794.597787] kthread+0xf5/0x130 [20794.600933] ? kthread_complete_and_exit+0x20/0x20 [20794.605717] ret_from_fork+0x31/0x40 [20794.609296] ? kthread_complete_and_exit+0x20/0x20 [20794.614079] ret_from_fork_asm+0x11/0x20 [20794.618004] [20796.579797] rcu: INFO: rcu_preempt self-detected stall on CPU [20796.585559] rcu: 7-....: (8604289 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/8603025 fqs=4317857 [20796.595982] rcu: (t=8859810 jiffies g=1309609 q=312036 ncpus=8) [20796.601983] Sending NMI from CPU 7 to CPUs 0: [20796.606333] NMI backtrace for cpu 0 [20796.606335] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20796.606337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20796.606337] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20796.606341] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20796.606342] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20796.606344] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20796.606345] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20796.606345] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20796.606346] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20796.606346] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20796.606347] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20796.606348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20796.606349] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20796.606350] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20796.606351] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20796.606351] Call Trace: [20796.606352] [20796.606353] ? nmi_cpu_backtrace+0xad/0x120 [20796.606357] ? nmi_cpu_backtrace_handler+0x11/0x20 [20796.606359] ? nmi_handle+0xd4/0x260 [20796.606363] ? default_do_nmi+0x49/0xf0 [20796.606364] ? exc_nmi+0x11e/0x150 [20796.606366] ? end_repeat_nmi+0x16/0x67 [20796.606370] ? queued_spin_lock_slowpath+0x16/0x2e0 [20796.606373] ? queued_spin_lock_slowpath+0x16/0x2e0 [20796.606375] ? queued_spin_lock_slowpath+0x16/0x2e0 [20796.606377] [20796.606378] [20796.606379] do_raw_spin_lock+0xb5/0xc0 [20796.606381] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20796.606395] try_get_cap_refs+0x159/0x880 [ceph] [20796.606410] __ceph_get_caps+0x4a1/0x760 [ceph] [20796.606423] ? __file_remove_privs+0x89/0x110 [20796.606424] ? wake_bit_function+0x70/0x70 [20796.606427] ? ceph_write_iter+0x260/0x770 [ceph] [20796.606438] ceph_write_iter+0x260/0x770 [ceph] [20796.606449] ? lock_is_held_type+0xaa/0x120 [20796.606452] vfs_write+0x324/0x4f0 [20796.606456] ksys_pwrite64+0x65/0xa0 [20796.606458] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20796.606461] do_syscall_64+0x38/0x80 [20796.606463] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20796.606465] RIP: 0033:0x7f48f0eeb93b [20796.606466] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20796.606467] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [20796.606469] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20796.606469] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20796.606470] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20796.606471] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20796.606471] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20796.606474] [20796.607332] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20796.911144] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20796.918615] Workqueue: events delayed_work [ceph] [20796.923330] RIP: 0010:lock_acquire+0xf0/0x2b0 [20796.927680] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20796.946417] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20796.951634] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20796.958760] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20796.965883] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20796.973007] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20796.980132] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20796.987257] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20796.995333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20797.001072] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20797.008195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20797.015318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20797.022443] Call Trace: [20797.024888] [20797.026898] ? rcu_dump_cpu_stacks+0xd1/0x180 [20797.031259] ? rcu_sched_clock_irq+0x633/0x1390 [20797.035783] ? lock_release+0xc5/0x260 [20797.039527] ? lock_is_held_type+0xaa/0x120 [20797.043704] ? lock_is_held_type+0xaa/0x120 [20797.047881] ? tick_sched_do_timer+0xa0/0xa0 [20797.052152] ? rcu_read_lock_sched_held+0x56/0x90 [20797.056852] ? tick_sched_do_timer+0xa0/0xa0 [20797.061115] ? update_process_times+0x5f/0x90 [20797.065465] ? tick_sched_handle.isra.16+0x1e/0x50 [20797.070250] ? tick_sched_timer+0x6f/0x80 [20797.074255] ? __hrtimer_run_queues+0x1d3/0x360 [20797.078781] ? hrtimer_interrupt+0x104/0x240 [20797.083051] ? __local_bh_enable+0x50/0x60 [20797.087145] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20797.092368] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20797.097326] [20797.099422] [20797.101519] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20797.106829] ? lock_acquire+0xf0/0x2b0 [20797.110578] ? igrab+0x43/0x50 [20797.113628] ? lock_release+0xc5/0x260 [20797.117373] _raw_spin_lock+0x30/0x40 [20797.121029] ? igrab+0x1d/0x50 [20797.124079] igrab+0x1d/0x50 [20797.126957] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20797.132107] delayed_work+0x1d8/0x280 [ceph] [20797.136389] ? process_scheduled_works+0x22a/0x510 [20797.141180] process_scheduled_works+0x2a4/0x510 [20797.145794] worker_thread+0x188/0x340 [20797.149545] ? rescuer_thread+0x310/0x310 [20797.153555] kthread+0xf5/0x130 [20797.156692] ? kthread_complete_and_exit+0x20/0x20 [20797.161477] ret_from_fork+0x31/0x40 [20797.165047] ? kthread_complete_and_exit+0x20/0x20 [20797.169833] ret_from_fork_asm+0x11/0x20 [20797.173755] [20821.822900] watchdog: BUG: soft lockup - CPU#0 stuck for 8277s! [dbench:78683] [20821.830116] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20821.834900] watchdog: BUG: soft lockup - CPU#7 stuck for 17274s! [kworker/7:4:78808] [20821.885556] irq event stamp: 31316334 [20821.893304] Modules linked in: ceph [20821.896961] hardirqs last enabled at (31316333): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20821.896963] libceph [20821.900447] dns_resolver fscache [20821.910604] hardirqs last disabled at (31316334): [] sysvec_apic_timer_interrupt+0xe/0x90 [20821.912788] netfs veth [20821.916098] softirqs last enabled at (31316330): [] __do_softirq+0x2be/0x371 [20821.925823] nft_chain_nat [20821.928266] softirqs last disabled at (31316325): [] irq_exit_rcu+0xb0/0xd0 [20821.936951] xt_MASQUERADE [20821.939655] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20821.948164] nf_nat [20821.950870] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20821.959813] nf_conntrack nf_defrag_ipv6 [20821.961910] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20821.969381] nf_defrag_ipv4 nft_compat [20821.973300] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20821.978777] nf_tables [20821.982520] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20822.001258] nfnetlink bridge [20822.003617] [20822.003617] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20822.008832] stp llc [20822.011796] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20822.013296] xfs libcrc32c [20822.020420] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20822.022605] sunrpc [20822.029728] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20822.032434] intel_rapl_msr intel_rapl_common [20822.039558] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20822.041653] sb_edac x86_pkg_temp_thermal [20822.048778] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20822.053128] coretemp kvm_intel [20822.060253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20822.064257] kvm irqbypass [20822.072334] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20822.075471] i2c_i801 iTCO_wdt [20822.081209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20822.083913] iTCO_vendor_support i2c_smbus [20822.091037] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20822.094089] crct10dif_pclmul joydev [20822.101212] Call Trace: [20822.101213] [20822.105304] crc32_pclmul ipmi_ssif [20822.112428] ? watchdog_timer_fn+0x1b4/0x220 [20822.115998] ghash_clmulni_intel mei_me [20822.118444] ? __hrtimer_run_queues+0x1d3/0x360 [20822.120453] mei lpc_ich [20822.123942] ? hrtimer_interrupt+0x104/0x240 [20822.128202] sha512_ssse3 acpi_ipmi [20822.132031] ? __local_bh_enable+0x50/0x60 [20822.136555] mfd_core wmi [20822.139090] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20822.143352] ipmi_si ipmi_devintf [20822.146836] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20822.150925] ipmi_msghandler ioatdma [20822.153542] [20822.158762] acpi_pad acpi_power_meter [20822.162071] [20822.162072] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20822.167028] ixgbe igb [20822.170603] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20822.172698] mdio crc32c_intel [20822.176444] do_raw_spin_lock+0xb5/0xc0 [20822.178538] nvme ptp [20822.183843] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20822.186199] nvme_core [20822.191071] try_get_cap_refs+0x159/0x880 [ceph] [20822.194120] i2c_algo_bit pps_core [20822.197957] __ceph_get_caps+0x4a1/0x760 [ceph] [20822.200223] dca fuse [20822.205010] ? __file_remove_privs+0x89/0x110 [20822.207364] [20822.207365] irq event stamp: 89168768 [20822.211977] ? wake_bit_function+0x70/0x70 [20822.215371] hardirqs last enabled at (89168767): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20822.219900] ? ceph_write_iter+0x260/0x770 [ceph] [20822.222167] hardirqs last disabled at (89168768): [] sysvec_apic_timer_interrupt+0xe/0x90 [20822.226519] ceph_write_iter+0x260/0x770 [ceph] [20822.228007] softirqs last enabled at (89168766): [] __do_softirq+0x2be/0x371 [20822.231669] ? lock_is_held_type+0xaa/0x120 [20822.235756] softirqs last disabled at (89168761): [] irq_exit_rcu+0xb0/0xd0 [20822.245916] vfs_write+0x324/0x4f0 [20822.250612] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20822.260339] ksys_pwrite64+0x65/0xa0 [20822.264860] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20822.273543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20822.277722] Workqueue: events delayed_work [ceph] [20822.286233] do_syscall_64+0x38/0x80 [20822.289631] [20822.289632] RIP: 0010:lock_acquire+0xf0/0x2b0 [20822.299008] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20822.302589] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20822.310057] RIP: 0033:0x7f48f0eeb93b [20822.315187] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20822.319886] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20822.323455] [20822.323456] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20822.324947] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20822.329301] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20822.334343] ORIG_RAX: 0000000000000012 [20822.353078] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20822.356650] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20822.361866] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20822.380604] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20822.382095] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20822.389219] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20822.394437] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20822.401561] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20822.405391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20822.412516] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20822.419640] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20822.426771] [20822.433888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20822.492633] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20822.499758] Call Trace: [20822.502201] [20822.504212] ? watchdog_timer_fn+0x1b4/0x220 [20822.508476] ? __hrtimer_run_queues+0x1d3/0x360 [20822.513004] ? hrtimer_interrupt+0x104/0x240 [20822.517272] ? __local_bh_enable+0x50/0x60 [20822.521367] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20822.526590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20822.531548] [20822.533645] [20822.535742] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20822.541051] ? lock_acquire+0xf0/0x2b0 [20822.544800] ? igrab+0x43/0x50 [20822.547850] ? lock_release+0xc5/0x260 [20822.551595] _raw_spin_lock+0x30/0x40 [20822.555252] ? igrab+0x1d/0x50 [20822.558307] igrab+0x1d/0x50 [20822.561189] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20822.566338] delayed_work+0x1d8/0x280 [ceph] [20822.570620] ? process_scheduled_works+0x22a/0x510 [20822.575410] process_scheduled_works+0x2a4/0x510 [20822.580024] worker_thread+0x188/0x340 [20822.583774] ? rescuer_thread+0x310/0x310 [20822.587779] kthread+0xf5/0x130 [20822.590916] ? kthread_complete_and_exit+0x20/0x20 [20822.595701] ret_from_fork+0x31/0x40 [20822.599270] ? kthread_complete_and_exit+0x20/0x20 [20822.604053] ret_from_fork_asm+0x11/0x20 [20822.607977] [20849.823421] watchdog: BUG: soft lockup - CPU#0 stuck for 8303s! [dbench:78683] [20849.830638] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20849.835421] watchdog: BUG: soft lockup - CPU#7 stuck for 17300s! [kworker/7:4:78808] [20849.886096] irq event stamp: 31374468 [20849.893828] Modules linked in: ceph libceph dns_resolver fscache netfs veth [20849.897483] hardirqs last enabled at (31374467): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20849.904443] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [20849.914601] hardirqs last disabled at (31374468): [] sysvec_apic_timer_interrupt+0xe/0x90 [20849.920251] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [20849.929975] softirqs last enabled at (31374392): [] __do_softirq+0x2be/0x371 [20849.937359] stp llc xfs libcrc32c sunrpc intel_rapl_msr [20849.946044] softirqs last disabled at (31374387): [] irq_exit_rcu+0xb0/0xd0 [20849.951349] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [20849.959860] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20849.967070] kvm irqbypass [20849.976014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20849.978718] i2c_i801 iTCO_wdt iTCO_vendor_support [20849.986188] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20849.990973] i2c_smbus [20849.996451] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20849.996452] crct10dif_pclmul [20849.998818] joydev crc32_pclmul [20850.017554] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20850.020519] ipmi_ssif [20850.023743] [20850.028958] ghash_clmulni_intel [20850.031317] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20850.032817] mei_me [20850.036039] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20850.043165] mei lpc_ich [20850.045262] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20850.052387] sha512_ssse3 [20850.054916] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20850.062040] acpi_ipmi mfd_core [20850.064659] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20850.071783] wmi ipmi_si [20850.074918] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20850.082045] ipmi_devintf [20850.084573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20850.092653] ipmi_msghandler [20850.095270] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20850.101008] ioatdma [20850.103885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20850.111008] acpi_pad acpi_power_meter [20850.113191] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20850.120316] ixgbe igb [20850.124060] Call Trace: [20850.124062] [20850.131184] mdio [20850.133544] ? watchdog_timer_fn+0x1b4/0x220 [20850.135987] crc32c_intel [20850.137997] ? __hrtimer_run_queues+0x1d3/0x360 [20850.139922] nvme ptp [20850.144189] ? hrtimer_interrupt+0x104/0x240 [20850.146820] nvme_core [20850.151345] ? __local_bh_enable+0x50/0x60 [20850.153614] i2c_algo_bit pps_core [20850.157882] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20850.160236] dca fuse [20850.164327] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20850.167724] [20850.167725] irq event stamp: 89389640 [20850.172941] [20850.175213] hardirqs last enabled at (89389639): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20850.180171] [20850.181659] hardirqs last disabled at (89389640): [] sysvec_apic_timer_interrupt+0xe/0x90 [20850.185319] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20850.187416] softirqs last enabled at (89389638): [] __do_softirq+0x2be/0x371 [20850.197576] ? queued_spin_lock_slowpath+0x16/0x2e0 [20850.199671] softirqs last disabled at (89389633): [] irq_exit_rcu+0xb0/0xd0 [20850.209396] do_raw_spin_lock+0xb5/0xc0 [20850.214698] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20850.223384] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20850.228252] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20850.236767] try_get_cap_refs+0x159/0x880 [ceph] [20850.240596] Workqueue: events delayed_work [ceph] [20850.249977] __ceph_get_caps+0x4a1/0x760 [ceph] [20850.254757] [20850.254758] RIP: 0010:igrab+0x9/0x50 [20850.262232] ? __file_remove_privs+0x89/0x110 [20850.266838] Code: 0f 1f 44 00 00 c3 b8 00 04 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 <55> 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 [20850.271535] ? wake_bit_function+0x70/0x70 [20850.276058] RSP: 0018:ffffc90002117db0 EFLAGS: 00000286 [20850.277552] ? ceph_write_iter+0x260/0x770 [ceph] [20850.281119] [20850.281120] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [20850.285471] ceph_write_iter+0x260/0x770 [ceph] [20850.304210] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [20850.308304] ? lock_is_held_type+0xaa/0x120 [20850.313517] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [20850.318217] vfs_write+0x324/0x4f0 [20850.319704] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [20850.326833] ksys_pwrite64+0x65/0xa0 [20850.331353] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20850.338480] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20850.342656] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20850.349780] do_syscall_64+0x38/0x80 [20850.353176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20850.360303] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20850.363870] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20850.370996] RIP: 0033:0x7f48f0eeb93b [20850.376125] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20850.384205] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20850.387774] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20850.393511] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20850.398556] Call Trace: [20850.398557] [20850.405681] ORIG_RAX: 0000000000000012 [20850.409252] ? watchdog_timer_fn+0x1b4/0x220 [20850.416375] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20850.435117] ? __hrtimer_run_queues+0x1d3/0x360 [20850.442237] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20850.447459] ? hrtimer_interrupt+0x104/0x240 [20850.449897] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20850.451910] ? __local_bh_enable+0x50/0x60 [20850.455740] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20850.460007] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20850.467128] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20850.471652] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20850.478783] [20850.483041] [20850.522959] [20850.525057] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20850.530365] ? igrab+0x9/0x50 [20850.533335] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20850.538485] delayed_work+0x1d8/0x280 [ceph] [20850.542766] ? process_scheduled_works+0x22a/0x510 [20850.547557] process_scheduled_works+0x2a4/0x510 [20850.552173] worker_thread+0x188/0x340 [20850.555923] ? rescuer_thread+0x310/0x310 [20850.559934] kthread+0xf5/0x130 [20850.563070] ? kthread_complete_and_exit+0x20/0x20 [20850.567855] ret_from_fork+0x31/0x40 [20850.571436] ? kthread_complete_and_exit+0x20/0x20 [20850.576219] ret_from_fork_asm+0x11/0x20 [20850.580151] [20877.823941] watchdog: BUG: soft lockup - CPU#0 stuck for 8329s! [dbench:78683] [20877.831158] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20877.835941] watchdog: BUG: soft lockup - CPU#7 stuck for 17326s! [kworker/7:4:78808] [20877.886600] irq event stamp: 31432524 [20877.886601] hardirqs last enabled at (31432523): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20877.894349] Modules linked in: [20877.898005] hardirqs last disabled at (31432524): [] sysvec_apic_timer_interrupt+0xe/0x90 [20877.908163] ceph [20877.911214] softirqs last enabled at (31432456): [] __do_softirq+0x2be/0x371 [20877.920938] libceph [20877.922863] softirqs last disabled at (31432451): [] irq_exit_rcu+0xb0/0xd0 [20877.931545] dns_resolver fscache [20877.933732] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20877.942240] netfs veth [20877.945552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20877.954495] nft_chain_nat xt_MASQUERADE [20877.956940] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [20877.964410] nf_nat nf_conntrack [20877.968338] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [20877.973814] nf_defrag_ipv6 nf_defrag_ipv4 [20877.977039] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20877.995777] nft_compat [20877.999866] [20877.999867] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20878.005084] nf_tables nfnetlink [20878.007529] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20878.009019] bridge [20878.016143] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20878.019367] stp llc [20878.026492] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20878.028588] xfs libcrc32c [20878.035712] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20878.037896] sunrpc intel_rapl_msr [20878.045021] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20878.047724] intel_rapl_common [20878.054849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20878.058246] sb_edac x86_pkg_temp_thermal [20878.066339] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20878.069392] coretemp kvm_intel [20878.075130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20878.079134] kvm irqbypass [20878.086256] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20878.089394] i2c_i801 [20878.096519] Call Trace: [20878.096520] [20878.099224] iTCO_wdt [20878.106349] ? watchdog_timer_fn+0x1b4/0x220 [20878.108620] iTCO_vendor_support i2c_smbus [20878.111065] ? __hrtimer_run_queues+0x1d3/0x360 [20878.113073] crct10dif_pclmul [20878.115348] ? hrtimer_interrupt+0x104/0x240 [20878.119619] joydev crc32_pclmul [20878.123706] ? __local_bh_enable+0x50/0x60 [20878.128231] ipmi_ssif ghash_clmulni_intel [20878.131197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20878.135458] mei_me mei [20878.138685] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20878.142775] lpc_ich sha512_ssse3 [20878.146865] [20878.152083] acpi_ipmi mfd_core [20878.154526] [20878.154528] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20878.159482] wmi [20878.162799] ? queued_spin_lock_slowpath+0x18/0x2e0 [20878.164891] ipmi_si [20878.168032] do_raw_spin_lock+0xb5/0xc0 [20878.170127] ipmi_devintf ipmi_msghandler [20878.175432] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20878.177268] ioatdma acpi_pad [20878.182142] try_get_cap_refs+0x159/0x880 [ceph] [20878.184340] acpi_power_meter [20878.188177] __ceph_get_caps+0x4a1/0x760 [ceph] [20878.192175] ixgbe [20878.196964] ? __file_remove_privs+0x89/0x110 [20878.199924] igb [20878.204533] ? wake_bit_function+0x70/0x70 [20878.207499] mdio [20878.212024] ? ceph_write_iter+0x260/0x770 [ceph] [20878.214033] crc32c_intel nvme [20878.218383] ceph_write_iter+0x260/0x770 [ceph] [20878.220222] ptp nvme_core [20878.224316] ? lock_is_held_type+0xaa/0x120 [20878.226235] i2c_algo_bit [20878.230935] vfs_write+0x324/0x4f0 [20878.233984] pps_core dca [20878.238513] ksys_pwrite64+0x65/0xa0 [20878.241212] fuse [20878.245388] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20878.248007] irq event stamp: 89610618 [20878.251406] do_syscall_64+0x38/0x80 [20878.254022] hardirqs last enabled at (89610617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20878.257594] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20878.259515] hardirqs last disabled at (89610618): [] sysvec_apic_timer_interrupt+0xe/0x90 [20878.264647] RIP: 0033:0x7f48f0eeb93b [20878.268303] softirqs last enabled at (89610616): [] __do_softirq+0x2be/0x371 [20878.271874] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20878.282033] softirqs last disabled at (89610611): [] irq_exit_rcu+0xb0/0xd0 [20878.287078] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20878.296802] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20878.300371] ORIG_RAX: 0000000000000012 [20878.309055] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20878.327792] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20878.336304] Workqueue: events delayed_work [ceph] [20878.341523] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20878.350900] [20878.350901] RIP: 0010:lock_acquire+0xf0/0x2b0 [20878.354729] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20878.362202] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20878.369341] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20878.374039] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20878.381164] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20878.382655] [20878.382656] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20878.387012] [20878.394130] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20878.450264] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20878.457388] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20878.464514] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20878.471638] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20878.479715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20878.485451] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20878.492576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20878.499702] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20878.506825] Call Trace: [20878.509268] [20878.511280] ? watchdog_timer_fn+0x1b4/0x220 [20878.515544] ? __hrtimer_run_queues+0x1d3/0x360 [20878.520071] ? hrtimer_interrupt+0x104/0x240 [20878.524348] ? __local_bh_enable+0x50/0x60 [20878.528443] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20878.533667] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20878.538627] [20878.540729] [20878.542827] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20878.548134] ? lock_acquire+0xf0/0x2b0 [20878.551885] ? igrab+0x43/0x50 [20878.554936] ? lock_release+0xc5/0x260 [20878.558680] _raw_spin_lock+0x30/0x40 [20878.562350] ? igrab+0x1d/0x50 [20878.565403] igrab+0x1d/0x50 [20878.568281] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20878.573431] delayed_work+0x1d8/0x280 [ceph] [20878.577713] ? process_scheduled_works+0x22a/0x510 [20878.582502] process_scheduled_works+0x2a4/0x510 [20878.587117] worker_thread+0x188/0x340 [20878.590869] ? rescuer_thread+0x310/0x310 [20878.594878] kthread+0xf5/0x130 [20878.598018] ? kthread_complete_and_exit+0x20/0x20 [20878.602802] ret_from_fork+0x31/0x40 [20878.606380] ? kthread_complete_and_exit+0x20/0x20 [20878.611164] ret_from_fork_asm+0x11/0x20 [20878.615088] [20905.824460] watchdog: BUG: soft lockup - CPU#0 stuck for 8356s! [dbench:78683] [20905.831682] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20905.836458] watchdog: BUG: soft lockup - CPU#7 stuck for 17353s! [kworker/7:4:78808] [20905.887141] irq event stamp: 31490702 [20905.894871] Modules linked in: ceph libceph dns_resolver fscache netfs veth [20905.898527] hardirqs last enabled at (31490701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20905.898533] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [20905.915644] hardirqs last disabled at (31490702): [] sysvec_apic_timer_interrupt+0xe/0x90 [20905.915647] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [20905.933619] softirqs last enabled at (31490696): [] __do_softirq+0x2be/0x371 [20905.940311] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [20905.948994] softirqs last disabled at (31490691): [] irq_exit_rcu+0xb0/0xd0 [20905.956466] coretemp kvm_intel kvm irqbypass i2c_i801 [20905.964977] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20905.964978] iTCO_wdt iTCO_vendor_support [20905.970108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20905.979050] i2c_smbus [20905.983055] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20905.990527] crct10dif_pclmul [20905.992882] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20905.998361] joydev crc32_pclmul [20906.001325] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20906.020062] ipmi_ssif ghash_clmulni_intel [20906.023288] [20906.028504] mei_me mei [20906.032595] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20906.034087] lpc_ich [20906.036529] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20906.043653] sha512_ssse3 [20906.045855] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20906.052978] acpi_ipmi mfd_core [20906.055596] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20906.062722] wmi ipmi_si [20906.065857] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20906.072983] ipmi_devintf ipmi_msghandler [20906.075512] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20906.082638] ioatdma acpi_pad [20906.086641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20906.094720] acpi_power_meter ixgbe [20906.097683] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20906.103421] igb [20906.106905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20906.114027] mdio crc32c_intel [20906.115866] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20906.122989] nvme ptp [20906.126041] Call Trace: [20906.126043] [20906.133165] nvme_core [20906.135437] ? watchdog_timer_fn+0x1b4/0x220 [20906.137879] i2c_algo_bit pps_core [20906.139891] ? __hrtimer_run_queues+0x1d3/0x360 [20906.142249] dca fuse [20906.146516] ? hrtimer_interrupt+0x104/0x240 [20906.149910] [20906.149910] irq event stamp: 89831576 [20906.154434] ? __local_bh_enable+0x50/0x60 [20906.156704] hardirqs last enabled at (89831575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20906.160969] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20906.162458] hardirqs last disabled at (89831576): [] sysvec_apic_timer_interrupt+0xe/0x90 [20906.166116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20906.170207] softirqs last enabled at (89831574): [] __do_softirq+0x2be/0x371 [20906.180364] [20906.185582] softirqs last disabled at (89831569): [] irq_exit_rcu+0xb0/0xd0 [20906.195305] [20906.200264] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20906.208948] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20906.211044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20906.219560] ? queued_spin_lock_slowpath+0x16/0x2e0 [20906.221653] Workqueue: events delayed_work [ceph] [20906.231033] do_raw_spin_lock+0xb5/0xc0 [20906.236336] [20906.236337] RIP: 0010:lock_acquire+0xf0/0x2b0 [20906.243807] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20906.248676] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20906.253375] try_get_cap_refs+0x159/0x880 [ceph] [20906.257205] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20906.258702] __ceph_get_caps+0x4a1/0x760 [ceph] [20906.263046] [20906.263047] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20906.267836] ? __file_remove_privs+0x89/0x110 [20906.286568] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20906.291179] ? wake_bit_function+0x70/0x70 [20906.296396] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20906.300923] ? ceph_write_iter+0x260/0x770 [ceph] [20906.302411] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20906.309535] ceph_write_iter+0x260/0x770 [ceph] [20906.313885] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20906.321013] ? lock_is_held_type+0xaa/0x120 [20906.325099] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20906.332227] vfs_write+0x324/0x4f0 [20906.336921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20906.344050] ksys_pwrite64+0x65/0xa0 [20906.348569] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20906.355694] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20906.359873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20906.367951] do_syscall_64+0x38/0x80 [20906.371347] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20906.377086] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20906.380655] Call Trace: [20906.380656] [20906.387781] RIP: 0033:0x7f48f0eeb93b [20906.392910] ? watchdog_timer_fn+0x1b4/0x220 [20906.400033] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20906.403605] ? __hrtimer_run_queues+0x1d3/0x360 [20906.410728] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20906.415776] ? hrtimer_interrupt+0x104/0x240 [20906.418217] ORIG_RAX: 0000000000000012 [20906.420226] ? __local_bh_enable+0x50/0x60 [20906.423797] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20906.428065] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20906.446800] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20906.451325] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20906.456542] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20906.460805] [20906.464636] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20906.468728] [20906.475863] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20906.481077] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20906.488207] [20906.493163] ? lock_acquire+0xf0/0x2b0 [20906.529960] ? igrab+0x43/0x50 [20906.533010] ? lock_release+0xc5/0x260 [20906.536755] _raw_spin_lock+0x30/0x40 [20906.540411] ? igrab+0x1d/0x50 [20906.543462] igrab+0x1d/0x50 [20906.546342] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20906.551488] delayed_work+0x1d8/0x280 [ceph] [20906.555771] ? process_scheduled_works+0x22a/0x510 [20906.560561] process_scheduled_works+0x2a4/0x510 [20906.565178] worker_thread+0x188/0x340 [20906.568927] ? rescuer_thread+0x310/0x310 [20906.572937] kthread+0xf5/0x130 [20906.576075] ? kthread_complete_and_exit+0x20/0x20 [20906.580867] ret_from_fork+0x31/0x40 [20906.584447] ? kthread_complete_and_exit+0x20/0x20 [20906.589230] ret_from_fork_asm+0x11/0x20 [20906.593163] [20933.824969] watchdog: BUG: soft lockup - CPU#0 stuck for 8382s! [dbench:78683] [20933.832187] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20933.836970] watchdog: BUG: soft lockup - CPU#7 stuck for 17379s! [kworker/7:4:78808] [20933.887644] irq event stamp: 31548702 [20933.887646] hardirqs last enabled at (31548701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20933.895375] Modules linked in: [20933.899033] hardirqs last disabled at (31548702): [] sysvec_apic_timer_interrupt+0xe/0x90 [20933.909190] ceph [20933.912240] softirqs last enabled at (31548644): [] __do_softirq+0x2be/0x371 [20933.921965] libceph [20933.923890] softirqs last disabled at (31548639): [] irq_exit_rcu+0xb0/0xd0 [20933.932572] dns_resolver [20933.934756] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20933.943268] fscache netfs [20933.945885] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20933.954829] veth nft_chain_nat [20933.957535] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20933.965005] xt_MASQUERADE nf_nat [20933.968143] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20933.973618] nf_conntrack nf_defrag_ipv6 [20933.976929] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20933.995668] nf_defrag_ipv4 nft_compat [20933.999586] [20933.999587] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20934.004802] nf_tables nfnetlink [20934.008546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20934.010037] bridge stp [20934.017162] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20934.020386] llc xfs [20934.027510] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20934.029953] libcrc32c sunrpc [20934.037077] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20934.039263] intel_rapl_msr intel_rapl_common [20934.046387] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20934.049366] sb_edac x86_pkg_temp_thermal [20934.056491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20934.060844] coretemp kvm_intel [20934.068921] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20934.072923] kvm irqbypass [20934.078662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20934.081799] i2c_i801 iTCO_wdt [20934.088923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20934.091628] iTCO_vendor_support i2c_smbus [20934.098751] Call Trace: [20934.098752] [20934.101802] crct10dif_pclmul joydev [20934.108927] ? watchdog_timer_fn+0x1b4/0x220 [20934.113017] crc32_pclmul ipmi_ssif [20934.115462] ? __hrtimer_run_queues+0x1d3/0x360 [20934.117472] ghash_clmulni_intel [20934.121044] ? hrtimer_interrupt+0x104/0x240 [20934.125305] mei_me mei [20934.128789] ? __local_bh_enable+0x50/0x60 [20934.133313] lpc_ich sha512_ssse3 [20934.136540] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20934.140802] acpi_ipmi mfd_core [20934.143247] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20934.147336] wmi ipmi_si [20934.150657] [20934.155873] ipmi_devintf ipmi_msghandler [20934.159011] [20934.159012] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20934.163969] ioatdma [20934.166503] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20934.168598] acpi_pad acpi_power_meter [20934.172604] do_raw_spin_lock+0xb5/0xc0 [20934.174699] ixgbe igb [20934.180004] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20934.182186] mdio crc32c_intel [20934.187058] try_get_cap_refs+0x159/0x880 [ceph] [20934.190802] nvme ptp [20934.194653] __ceph_get_caps+0x4a1/0x760 [ceph] [20934.197007] nvme_core i2c_algo_bit [20934.201795] ? __file_remove_privs+0x89/0x110 [20934.204842] pps_core dca [20934.209452] ? wake_bit_function+0x70/0x70 [20934.211722] fuse [20934.216250] ? ceph_write_iter+0x260/0x770 [ceph] [20934.219731] irq event stamp: 90052414 [20934.224084] ceph_write_iter+0x260/0x770 [ceph] [20934.226699] hardirqs last enabled at (90052413): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20934.230794] ? lock_is_held_type+0xaa/0x120 [20934.232714] hardirqs last disabled at (90052414): [] sysvec_apic_timer_interrupt+0xe/0x90 [20934.237414] vfs_write+0x324/0x4f0 [20934.241068] softirqs last enabled at (90052412): [] __do_softirq+0x2be/0x371 [20934.245599] ksys_pwrite64+0x65/0xa0 [20934.255750] softirqs last disabled at (90052407): [] irq_exit_rcu+0xb0/0xd0 [20934.259929] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20934.269653] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20934.273050] do_syscall_64+0x38/0x80 [20934.281734] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20934.285306] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20934.293815] Workqueue: events delayed_work [ceph] [20934.298946] RIP: 0033:0x7f48f0eeb93b [20934.308323] [20934.311895] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20934.319373] RIP: 0010:lock_acquire+0xf0/0x2b0 [20934.324418] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20934.329115] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20934.332685] ORIG_RAX: 0000000000000012 [20934.334176] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20934.352915] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20934.357264] [20934.357265] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20934.362484] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20934.381221] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20934.385052] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20934.390269] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20934.397393] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20934.398883] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20934.406007] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20934.413133] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20934.420261] [20934.427379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20934.473263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20934.479001] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20934.486123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20934.493249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20934.500376] Call Trace: [20934.502825] [20934.504836] ? watchdog_timer_fn+0x1b4/0x220 [20934.509100] ? __hrtimer_run_queues+0x1d3/0x360 [20934.513627] ? hrtimer_interrupt+0x104/0x240 [20934.517896] ? __local_bh_enable+0x50/0x60 [20934.521990] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20934.527214] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20934.532172] [20934.534269] [20934.536379] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20934.541690] ? lock_acquire+0xf0/0x2b0 [20934.545442] ? igrab+0x43/0x50 [20934.548492] ? lock_release+0xc5/0x260 [20934.552235] _raw_spin_lock+0x30/0x40 [20934.555891] ? igrab+0x1d/0x50 [20934.558942] igrab+0x1d/0x50 [20934.561822] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20934.566972] delayed_work+0x1d8/0x280 [ceph] [20934.571251] ? process_scheduled_works+0x22a/0x510 [20934.576034] process_scheduled_works+0x2a4/0x510 [20934.580661] worker_thread+0x188/0x340 [20934.584407] ? rescuer_thread+0x310/0x310 [20934.588411] kthread+0xf5/0x130 [20934.591548] ? kthread_complete_and_exit+0x20/0x20 [20934.596333] ret_from_fork+0x31/0x40 [20934.599912] ? kthread_complete_and_exit+0x20/0x20 [20934.604695] ret_from_fork_asm+0x11/0x20 [20934.608619] [20961.825479] watchdog: BUG: soft lockup - CPU#0 stuck for 8408s! [dbench:78683] [20961.832690] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20961.837479] watchdog: BUG: soft lockup - CPU#7 stuck for 17405s! [kworker/7:4:78808] [20961.888150] irq event stamp: 31606894 [20961.895881] Modules linked in: ceph libceph dns_resolver [20961.899538] hardirqs last enabled at (31606893): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20961.904843] fscache netfs veth [20961.914999] hardirqs last disabled at (31606894): [] sysvec_apic_timer_interrupt+0xe/0x90 [20961.918151] nft_chain_nat xt_MASQUERADE nf_nat [20961.927878] softirqs last enabled at (31606836): [] __do_softirq+0x2be/0x371 [20961.927879] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [20961.941087] softirqs last disabled at (31606831): [] irq_exit_rcu+0xb0/0xd0 [20961.948123] nfnetlink bridge stp [20961.956635] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20961.959946] llc [20961.968890] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20961.968891] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [20961.970727] xfs libcrc32c [20961.978197] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [20961.983674] sunrpc [20961.986380] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20962.005118] intel_rapl_msr intel_rapl_common [20962.007215] [20962.012431] sb_edac x86_pkg_temp_thermal [20962.016782] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20962.018272] coretemp kvm_intel [20962.022278] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20962.029402] kvm irqbypass [20962.032539] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20962.039662] i2c_i801 [20962.042366] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20962.049490] iTCO_wdt iTCO_vendor_support [20962.051762] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20962.058886] i2c_smbus [20962.062889] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20962.070015] crct10dif_pclmul joydev [20962.072372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20962.080450] crc32_pclmul ipmi_ssif [20962.084020] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20962.089757] ghash_clmulni_intel mei_me [20962.093241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20962.100366] mei [20962.104196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20962.111321] lpc_ich [20962.113158] Call Trace: [20962.113159] [20962.120281] sha512_ssse3 acpi_ipmi [20962.122466] ? watchdog_timer_fn+0x1b4/0x220 [20962.124909] mfd_core [20962.126923] ? __hrtimer_run_queues+0x1d3/0x360 [20962.130406] wmi ipmi_si [20962.134673] ? hrtimer_interrupt+0x104/0x240 [20962.136939] ipmi_devintf ipmi_msghandler [20962.141464] ? __local_bh_enable+0x50/0x60 [20962.143995] ioatdma [20962.148261] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20962.152262] acpi_pad acpi_power_meter [20962.156353] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20962.158538] ixgbe igb [20962.163754] [20962.167499] mdio crc32c_intel [20962.172457] [20962.172459] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20962.174813] nvme ptp [20962.176916] ? queued_spin_lock_slowpath+0x16/0x2e0 [20962.179961] nvme_core [20962.182061] do_raw_spin_lock+0xb5/0xc0 [20962.187364] i2c_algo_bit pps_core [20962.189634] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20962.194504] dca [20962.196880] try_get_cap_refs+0x159/0x880 [ceph] [20962.200709] fuse [20962.204113] __ceph_get_caps+0x4a1/0x760 [ceph] [20962.208893] [20962.210734] ? __file_remove_privs+0x89/0x110 [20962.215340] irq event stamp: 90273194 [20962.217265] ? wake_bit_function+0x70/0x70 [20962.221788] hardirqs last enabled at (90273193): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20962.223280] ? ceph_write_iter+0x260/0x770 [ceph] [20962.227630] hardirqs last disabled at (90273194): [] sysvec_apic_timer_interrupt+0xe/0x90 [20962.231289] ceph_write_iter+0x260/0x770 [ceph] [20962.235377] softirqs last enabled at (90273192): [] __do_softirq+0x2be/0x371 [20962.245538] ? lock_is_held_type+0xaa/0x120 [20962.250232] softirqs last disabled at (90273187): [] irq_exit_rcu+0xb0/0xd0 [20962.259960] vfs_write+0x324/0x4f0 [20962.264481] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20962.273169] ksys_pwrite64+0x65/0xa0 [20962.277343] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20962.285854] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20962.289267] Workqueue: events delayed_work [ceph] [20962.298646] do_syscall_64+0x38/0x80 [20962.302217] [20962.309688] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20962.314817] RIP: 0010:lock_acquire+0xf0/0x2b0 [20962.319516] RIP: 0033:0x7f48f0eeb93b [20962.323086] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [20962.324576] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20962.329621] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [20962.333970] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20962.337543] [20962.337544] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [20962.356279] ORIG_RAX: 0000000000000012 [20962.375017] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [20962.380233] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20962.385453] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [20962.386942] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20962.394068] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [20962.397898] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20962.405022] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [20962.412145] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20962.419288] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20962.426412] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20962.433535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20962.440666] [20962.447785] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20962.485165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20962.492288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20962.499412] Call Trace: [20962.501857] [20962.503885] ? watchdog_timer_fn+0x1b4/0x220 [20962.508151] ? __hrtimer_run_queues+0x1d3/0x360 [20962.512693] ? hrtimer_interrupt+0x104/0x240 [20962.516954] ? __local_bh_enable+0x50/0x60 [20962.521047] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20962.526271] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20962.531229] [20962.533326] [20962.535424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20962.540733] ? lock_acquire+0xf0/0x2b0 [20962.544482] ? igrab+0x43/0x50 [20962.547532] ? lock_release+0xc5/0x260 [20962.551278] _raw_spin_lock+0x30/0x40 [20962.554940] ? igrab+0x1d/0x50 [20962.557991] igrab+0x1d/0x50 [20962.560886] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20962.566038] delayed_work+0x1d8/0x280 [ceph] [20962.570319] ? process_scheduled_works+0x22a/0x510 [20962.575108] process_scheduled_works+0x2a4/0x510 [20962.579725] worker_thread+0x188/0x340 [20962.583475] ? rescuer_thread+0x310/0x310 [20962.587486] kthread+0xf5/0x130 [20962.590624] ? kthread_complete_and_exit+0x20/0x20 [20962.595408] ret_from_fork+0x31/0x40 [20962.598977] ? kthread_complete_and_exit+0x20/0x20 [20962.603763] ret_from_fork_asm+0x11/0x20 [20962.607683] [20989.825978] watchdog: BUG: soft lockup - CPU#0 stuck for 8434s! [dbench:78683] [20989.833197] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [20989.837978] watchdog: BUG: soft lockup - CPU#7 stuck for 17431s! [kworker/7:4:78808] [20989.888663] irq event stamp: 31664982 [20989.888665] hardirqs last enabled at (31664981): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20989.896395] Modules linked in: [20989.900052] hardirqs last disabled at (31664982): [] sysvec_apic_timer_interrupt+0xe/0x90 [20989.910209] ceph [20989.913261] softirqs last enabled at (31664926): [] __do_softirq+0x2be/0x371 [20989.922983] libceph dns_resolver [20989.924908] softirqs last disabled at (31664921): [] irq_exit_rcu+0xb0/0xd0 [20989.933591] fscache netfs [20989.936905] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20989.945413] veth [20989.948118] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20989.957063] nft_chain_nat xt_MASQUERADE [20989.958986] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20989.966456] nf_nat [20989.970375] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20989.975850] nf_conntrack nf_defrag_ipv6 [20989.977950] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20989.996688] nf_defrag_ipv4 nft_compat [20990.000605] [20990.005823] nf_tables nfnetlink [20990.009565] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20990.011057] bridge stp [20990.014281] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20990.021405] llc xfs [20990.023849] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20990.030974] libcrc32c sunrpc [20990.033156] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20990.040280] intel_rapl_msr intel_rapl_common [20990.043244] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20990.050376] sb_edac x86_pkg_temp_thermal [20990.054728] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20990.061854] coretemp kvm_intel [20990.065857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20990.073933] kvm irqbypass [20990.077072] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20990.082808] i2c_i801 [20990.085513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20990.092638] iTCO_wdt iTCO_vendor_support [20990.094907] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20990.102032] i2c_smbus crct10dif_pclmul [20990.106035] Call Trace: [20990.113161] joydev crc32_pclmul [20990.116992] [20990.119436] ipmi_ssif ghash_clmulni_intel [20990.122675] ? watchdog_timer_fn+0x1b4/0x220 [20990.124688] mei_me mei [20990.128779] ? __hrtimer_run_queues+0x1d3/0x360 [20990.133042] lpc_ich sha512_ssse3 [20990.135490] ? hrtimer_interrupt+0x104/0x240 [20990.140009] acpi_ipmi [20990.143321] ? __local_bh_enable+0x50/0x60 [20990.147584] mfd_core wmi [20990.149945] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20990.154033] ipmi_si ipmi_devintf [20990.156660] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20990.161877] ipmi_msghandler [20990.165187] [20990.170145] ioatdma acpi_pad [20990.173023] [20990.173024] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20990.175119] acpi_power_meter ixgbe [20990.178088] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20990.180182] igb [20990.185486] do_raw_spin_lock+0xb5/0xc0 [20990.188970] mdio [20990.193841] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20990.195677] crc32c_intel nvme [20990.199511] try_get_cap_refs+0x159/0x880 [ceph] [20990.201432] ptp nvme_core [20990.206222] __ceph_get_caps+0x4a1/0x760 [ceph] [20990.209266] i2c_algo_bit pps_core [20990.213881] ? __file_remove_privs+0x89/0x110 [20990.216580] dca [20990.221106] ? wake_bit_function+0x70/0x70 [20990.224503] fuse [20990.228857] ? ceph_write_iter+0x260/0x770 [ceph] [20990.230690] [20990.230691] irq event stamp: 90493902 [20990.234782] ceph_write_iter+0x260/0x770 [ceph] [20990.236706] hardirqs last enabled at (90493901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20990.241409] ? lock_is_held_type+0xaa/0x120 [20990.242895] hardirqs last disabled at (90493902): [] sysvec_apic_timer_interrupt+0xe/0x90 [20990.246553] vfs_write+0x324/0x4f0 [20990.251076] softirqs last enabled at (90493900): [] __do_softirq+0x2be/0x371 [20990.261238] ksys_pwrite64+0x65/0xa0 [20990.265412] softirqs last disabled at (90493895): [] irq_exit_rcu+0xb0/0xd0 [20990.275136] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20990.278532] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20990.287216] do_syscall_64+0x38/0x80 [20990.290786] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20990.299299] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20990.304429] Workqueue: events delayed_work [ceph] [20990.313808] RIP: 0033:0x7f48f0eeb93b [20990.317382] [20990.317383] RIP: 0010:lock_release+0x162/0x260 [20990.324857] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20990.329900] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [20990.334597] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [20990.338169] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [20990.339662] ORIG_RAX: 0000000000000012 [20990.344097] [20990.344098] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [20990.362835] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20990.381572] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [20990.386790] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20990.392006] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [20990.395837] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20990.397327] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [20990.404453] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20990.411577] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [20990.418700] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20990.425824] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20990.432954] [20990.440074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20990.484585] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20990.491710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20990.498833] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20990.505959] Call Trace: [20990.508404] [20990.510415] ? watchdog_timer_fn+0x1b4/0x220 [20990.514678] ? __hrtimer_run_queues+0x1d3/0x360 [20990.519204] ? hrtimer_interrupt+0x104/0x240 [20990.523473] ? __local_bh_enable+0x50/0x60 [20990.527569] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20990.532791] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20990.537750] [20990.539847] [20990.541944] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20990.547250] ? igrab+0x43/0x50 [20990.550302] ? lock_release+0x162/0x260 [20990.554138] ? lock_release+0xc5/0x260 [20990.557884] _raw_spin_unlock+0x1b/0x40 [20990.561723] igrab+0x43/0x50 [20990.564600] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20990.569748] delayed_work+0x1d8/0x280 [ceph] [20990.574030] ? process_scheduled_works+0x22a/0x510 [20990.578821] process_scheduled_works+0x2a4/0x510 [20990.583438] worker_thread+0x188/0x340 [20990.587187] ? rescuer_thread+0x310/0x310 [20990.591197] kthread+0xf5/0x130 [20990.594343] ? kthread_complete_and_exit+0x20/0x20 [20990.599127] ret_from_fork+0x31/0x40 [20990.602708] ? kthread_complete_and_exit+0x20/0x20 [20990.607500] ret_from_fork_asm+0x11/0x20 [20990.611432] [20992.181385] rcu: INFO: rcu_preempt self-detected stall on CPU [20992.187141] rcu: 0-....: (8851185 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=4413745 [20992.197471] rcu: (t=9055409 jiffies g=1309609 q=316134 ncpus=8) [20992.203470] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20992.212413] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20992.219885] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [20992.225371] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [20992.244108] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [20992.249326] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [20992.256449] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [20992.263573] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [20992.270696] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [20992.277821] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [20992.284945] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [20992.293024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20992.298762] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [20992.305885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20992.313010] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20992.320133] Call Trace: [20992.322579] [20992.324588] ? rcu_dump_cpu_stacks+0xd1/0x180 [20992.328948] ? rcu_sched_clock_irq+0x633/0x1390 [20992.333472] ? lock_is_held_type+0xaa/0x120 [20992.337651] ? lock_is_held_type+0xaa/0x120 [20992.341829] ? tick_sched_do_timer+0xa0/0xa0 [20992.346100] ? rcu_read_lock_sched_held+0x56/0x90 [20992.350798] ? tick_sched_do_timer+0xa0/0xa0 [20992.355060] ? update_process_times+0x5f/0x90 [20992.359411] ? tick_sched_handle.isra.16+0x1e/0x50 [20992.364197] ? tick_sched_timer+0x6f/0x80 [20992.368201] ? __hrtimer_run_queues+0x1d3/0x360 [20992.372728] ? hrtimer_interrupt+0x104/0x240 [20992.376995] ? __local_bh_enable+0x50/0x60 [20992.381089] ? __sysvec_apic_timer_interrupt+0x61/0x160 [20992.386306] ? sysvec_apic_timer_interrupt+0x6d/0x90 [20992.391262] [20992.393358] [20992.395458] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [20992.400765] ? queued_spin_lock_slowpath+0x2d/0x2e0 [20992.405642] do_raw_spin_lock+0xb5/0xc0 [20992.409471] __cap_delay_requeue+0xa7/0x1d0 [ceph] [20992.414274] try_get_cap_refs+0x159/0x880 [ceph] [20992.418907] __ceph_get_caps+0x4a1/0x760 [ceph] [20992.423446] ? __file_remove_privs+0x89/0x110 [20992.427802] ? wake_bit_function+0x70/0x70 [20992.431894] ? ceph_write_iter+0x260/0x770 [ceph] [20992.436608] ceph_write_iter+0x260/0x770 [ceph] [20992.441145] ? lock_is_held_type+0xaa/0x120 [20992.445329] vfs_write+0x324/0x4f0 [20992.448735] ksys_pwrite64+0x65/0xa0 [20992.452310] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [20992.457444] do_syscall_64+0x38/0x80 [20992.461022] entry_SYSCALL_64_after_hwframe+0x63/0xcd [20992.466066] RIP: 0033:0x7f48f0eeb93b [20992.469636] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [20992.488375] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [20992.495932] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [20992.503056] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [20992.510180] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [20992.517303] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [20992.524426] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [20992.531557] [20992.533745] Sending NMI from CPU 0 to CPUs 7: [20992.538097] NMI backtrace for cpu 7 [20992.538099] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [20992.538101] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [20992.538101] Workqueue: events delayed_work [ceph] [20992.538117] RIP: 0010:rcu_is_watching+0xb/0x40 [20992.538120] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [20992.538122] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [20992.538123] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [20992.538124] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [20992.538125] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [20992.538125] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [20992.538126] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [20992.538127] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [20992.538128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20992.538128] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [20992.538129] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [20992.538130] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [20992.538131] Call Trace: [20992.538131] [20992.538133] ? nmi_cpu_backtrace+0xad/0x120 [20992.538136] ? nmi_cpu_backtrace_handler+0x11/0x20 [20992.538137] ? nmi_handle+0xd4/0x260 [20992.538141] ? default_do_nmi+0x49/0xf0 [20992.538143] ? exc_nmi+0x11e/0x150 [20992.538144] ? end_repeat_nmi+0x16/0x67 [20992.538147] ? igrab+0x43/0x50 [20992.538149] ? igrab+0x43/0x50 [20992.538150] ? rcu_is_watching+0xb/0x40 [20992.538152] ? rcu_is_watching+0xb/0x40 [20992.538154] ? rcu_is_watching+0xb/0x40 [20992.538156] [20992.538157] [20992.538157] lock_release+0x1d7/0x260 [20992.538159] _raw_spin_unlock+0x1b/0x40 [20992.538162] igrab+0x43/0x50 [20992.538163] ceph_check_delayed_caps+0xb7/0x270 [ceph] [20992.538178] delayed_work+0x1d8/0x280 [ceph] [20992.538191] ? process_scheduled_works+0x22a/0x510 [20992.538194] process_scheduled_works+0x2a4/0x510 [20992.538198] worker_thread+0x188/0x340 [20992.538200] ? rescuer_thread+0x310/0x310 [20992.538203] kthread+0xf5/0x130 [20992.538205] ? kthread_complete_and_exit+0x20/0x20 [20992.538207] ret_from_fork+0x31/0x40 [20992.538209] ? kthread_complete_and_exit+0x20/0x20 [20992.538211] ret_from_fork_asm+0x11/0x20 [20992.538216] [21017.826476] watchdog: BUG: soft lockup - CPU#0 stuck for 8460s! [dbench:78683] [21017.833692] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21017.838476] watchdog: BUG: soft lockup - CPU#7 stuck for 17457s! [kworker/7:4:78808] [21017.889150] irq event stamp: 31722454 [21017.889151] hardirqs last enabled at (31722453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21017.896882] Modules linked in: [21017.900540] hardirqs last disabled at (31722454): [] sysvec_apic_timer_interrupt+0xe/0x90 [21017.910696] ceph [21017.913747] softirqs last enabled at (31722416): [] __do_softirq+0x2be/0x371 [21017.923471] libceph [21017.925396] softirqs last disabled at (31722411): [] irq_exit_rcu+0xb0/0xd0 [21017.934081] dns_resolver [21017.936266] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21017.944776] fscache [21017.947393] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21017.956336] netfs veth [21017.958520] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21017.965990] nft_chain_nat xt_MASQUERADE [21017.968435] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21017.973914] nf_nat [21017.977831] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21017.996567] nf_conntrack [21017.998666] [21017.998667] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21018.003883] nf_defrag_ipv6 [21018.006501] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21018.007991] nf_defrag_ipv4 nft_compat [21018.015116] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21018.017907] nf_tables nfnetlink [21018.025029] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21018.028774] bridge stp [21018.035898] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21018.039121] llc xfs [21018.046247] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21018.048690] libcrc32c sunrpc [21018.055814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21018.057999] intel_rapl_msr intel_rapl_common [21018.066075] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21018.069040] sb_edac x86_pkg_temp_thermal [21018.074776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21018.079128] coretemp kvm_intel [21018.086253] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21018.090257] kvm irqbypass [21018.097381] Call Trace: [21018.097382] [21018.100519] i2c_i801 [21018.107642] ? watchdog_timer_fn+0x1b4/0x220 [21018.110346] iTCO_wdt iTCO_vendor_support [21018.112790] ? __hrtimer_run_queues+0x1d3/0x360 [21018.114801] i2c_smbus [21018.117075] ? hrtimer_interrupt+0x104/0x240 [21018.121335] crct10dif_pclmul [21018.125339] ? __local_bh_enable+0x50/0x60 [21018.129872] joydev crc32_pclmul [21018.132234] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21018.136494] ipmi_ssif [21018.139459] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21018.143549] ghash_clmulni_intel [21018.146772] [21018.151990] mei_me mei [21018.154347] [21018.154348] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21018.159306] lpc_ich sha512_ssse3 [21018.162532] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21018.164626] acpi_ipmi mfd_core [21018.167071] do_raw_spin_lock+0xb5/0xc0 [21018.169169] wmi ipmi_si [21018.174474] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21018.177782] ipmi_devintf [21018.182657] try_get_cap_refs+0x159/0x880 [ceph] [21018.185792] ipmi_msghandler ioatdma [21018.189626] __ceph_get_caps+0x4a1/0x760 [ceph] [21018.192157] acpi_pad [21018.196949] ? __file_remove_privs+0x89/0x110 [21018.199562] acpi_power_meter [21018.204174] ? wake_bit_function+0x70/0x70 [21018.207744] ixgbe [21018.212270] ? ceph_write_iter+0x260/0x770 [ceph] [21018.214538] igb mdio [21018.218889] ceph_write_iter+0x260/0x770 [ceph] [21018.221853] crc32c_intel nvme [21018.225965] ? lock_is_held_type+0xaa/0x120 [21018.227972] ptp nvme_core [21018.232672] vfs_write+0x324/0x4f0 [21018.234941] i2c_algo_bit pps_core [21018.239470] ksys_pwrite64+0x65/0xa0 [21018.242514] dca fuse [21018.246693] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21018.249397] [21018.252795] do_syscall_64+0x38/0x80 [21018.256191] irq event stamp: 90712958 [21018.259763] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21018.262034] hardirqs last enabled at (90712957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21018.267167] RIP: 0033:0x7f48f0eeb93b [21018.268655] hardirqs last disabled at (90712958): [] sysvec_apic_timer_interrupt+0xe/0x90 [21018.272225] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21018.275881] softirqs last enabled at (90712956): [] __do_softirq+0x2be/0x371 [21018.280926] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21018.291084] softirqs last disabled at (90712951): [] irq_exit_rcu+0xb0/0xd0 [21018.294656] ORIG_RAX: 0000000000000012 [21018.304378] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21018.323115] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21018.331802] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21018.337019] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21018.345529] Workqueue: events delayed_work [ceph] [21018.349360] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21018.358738] [21018.365878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21018.373349] RIP: 0010:lock_release+0x14e/0x260 [21018.380472] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21018.385171] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [21018.392302] [21018.393787] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [21018.438610] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [21018.445736] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [21018.452877] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [21018.460000] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [21018.467124] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [21018.474249] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21018.482325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21018.488064] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21018.495189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21018.502313] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21018.509438] Call Trace: [21018.511883] [21018.513893] ? watchdog_timer_fn+0x1b4/0x220 [21018.518171] ? __hrtimer_run_queues+0x1d3/0x360 [21018.522700] ? hrtimer_interrupt+0x104/0x240 [21018.526970] ? __local_bh_enable+0x50/0x60 [21018.531065] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21018.536288] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21018.541244] [21018.543341] [21018.545439] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21018.550743] ? igrab+0x43/0x50 [21018.553796] ? lock_release+0x14e/0x260 [21018.557625] ? lock_release+0xc5/0x260 [21018.561372] _raw_spin_unlock+0x1b/0x40 [21018.565207] igrab+0x43/0x50 [21018.568086] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21018.573236] delayed_work+0x1d8/0x280 [ceph] [21018.577515] ? process_scheduled_works+0x22a/0x510 [21018.582299] process_scheduled_works+0x2a4/0x510 [21018.586915] worker_thread+0x188/0x340 [21018.590663] ? rescuer_thread+0x310/0x310 [21018.594667] kthread+0xf5/0x130 [21018.597804] ? kthread_complete_and_exit+0x20/0x20 [21018.602587] ret_from_fork+0x31/0x40 [21018.606163] ? kthread_complete_and_exit+0x20/0x20 [21018.610952] ret_from_fork_asm+0x11/0x20 [21018.614890] [21045.826974] watchdog: BUG: soft lockup - CPU#0 stuck for 8486s! [dbench:78683] [21045.834190] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21045.838974] watchdog: BUG: soft lockup - CPU#7 stuck for 17483s! [kworker/7:4:78808] [21045.889656] irq event stamp: 31780744 [21045.897389] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21045.901043] hardirqs last enabled at (31780743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21045.901049] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [21045.918163] hardirqs last disabled at (31780744): [] sysvec_apic_timer_interrupt+0xe/0x90 [21045.918166] nft_compat [21045.918167] softirqs last enabled at (31780708): [] __do_softirq+0x2be/0x371 [21045.926421] nf_tables nfnetlink [21045.936145] softirqs last disabled at (31780703): [] irq_exit_rcu+0xb0/0xd0 [21045.938588] bridge stp [21045.947276] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21045.950497] llc xfs [21045.959009] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21045.961452] libcrc32c sunrpc [21045.970397] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21045.972579] intel_rapl_msr intel_rapl_common [21045.980052] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21045.983016] sb_edac x86_pkg_temp_thermal [21045.988493] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21045.992843] coretemp kvm_intel [21046.011580] [21046.011581] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21046.015584] kvm irqbypass [21046.020803] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21046.023941] i2c_i801 iTCO_wdt [21046.025431] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21046.032555] iTCO_vendor_support i2c_smbus [21046.035260] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21046.042382] crct10dif_pclmul joydev [21046.045435] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21046.052558] crc32_pclmul ipmi_ssif [21046.056650] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21046.063772] ghash_clmulni_intel mei_me [21046.067345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21046.074467] mei lpc_ich [21046.077953] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21046.086030] sha512_ssse3 [21046.089861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21046.095597] acpi_ipmi [21046.098128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21046.098129] Call Trace: [21046.105252] mfd_core [21046.107871] [21046.114995] wmi [21046.117371] ? watchdog_timer_fn+0x1b4/0x220 [21046.124492] ipmi_si ipmi_devintf [21046.126938] ? __hrtimer_run_queues+0x1d3/0x360 [21046.129207] ipmi_msghandler ioatdma [21046.131223] ? hrtimer_interrupt+0x104/0x240 [21046.133057] acpi_pad acpi_power_meter [21046.137319] ? __local_bh_enable+0x50/0x60 [21046.140630] ixgbe igb [21046.145176] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21046.148742] mdio crc32c_intel [21046.153006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21046.156750] nvme ptp [21046.160843] [21046.163199] nvme_core [21046.168417] [21046.171467] i2c_algo_bit pps_core [21046.176425] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21046.178695] dca fuse [21046.180797] ? queued_spin_lock_slowpath+0x16/0x2e0 [21046.183149] [21046.183150] irq event stamp: 90934028 [21046.185251] do_raw_spin_lock+0xb5/0xc0 [21046.188644] hardirqs last enabled at (90934027): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21046.193968] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21046.196238] hardirqs last disabled at (90934028): [] sysvec_apic_timer_interrupt+0xe/0x90 [21046.201112] try_get_cap_refs+0x159/0x880 [ceph] [21046.202597] softirqs last enabled at (90934026): [] __do_softirq+0x2be/0x371 [21046.206262] __ceph_get_caps+0x4a1/0x760 [ceph] [21046.210085] softirqs last disabled at (90934021): [] irq_exit_rcu+0xb0/0xd0 [21046.220248] ? __file_remove_privs+0x89/0x110 [21046.225029] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21046.234753] ? wake_bit_function+0x70/0x70 [21046.239375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21046.239377] Workqueue: events delayed_work [ceph] [21046.248060] ? ceph_write_iter+0x260/0x770 [ceph] [21046.252580] [21046.261092] ceph_write_iter+0x260/0x770 [ceph] [21046.265441] RIP: 0010:lock_acquire+0xf0/0x2b0 [21046.274823] ? lock_is_held_type+0xaa/0x120 [21046.278908] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21046.286382] vfs_write+0x324/0x4f0 [21046.291079] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21046.295780] ksys_pwrite64+0x65/0xa0 [21046.297267] [21046.301790] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21046.306141] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21046.310320] do_syscall_64+0x38/0x80 [21046.329055] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21046.332454] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21046.337671] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21046.341242] RIP: 0033:0x7f48f0eeb93b [21046.342732] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21046.347863] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21046.354986] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21046.358558] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21046.365682] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21046.370726] ORIG_RAX: 0000000000000012 [21046.377850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21046.381420] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21046.388544] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21046.407282] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21046.414407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21046.419625] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21046.427701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21046.431533] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21046.437269] Call Trace: [21046.444394] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21046.451518] [21046.458648] [21046.465767] ? watchdog_timer_fn+0x1b4/0x220 [21046.505167] ? __hrtimer_run_queues+0x1d3/0x360 [21046.509694] ? hrtimer_interrupt+0x104/0x240 [21046.513962] ? __local_bh_enable+0x50/0x60 [21046.518057] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21046.523282] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21046.528247] [21046.530343] [21046.532440] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21046.537750] ? lock_acquire+0xf0/0x2b0 [21046.541500] ? igrab+0x43/0x50 [21046.544557] ? lock_release+0xc5/0x260 [21046.548303] _raw_spin_lock+0x30/0x40 [21046.551959] ? igrab+0x1d/0x50 [21046.555010] igrab+0x1d/0x50 [21046.557888] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21046.563038] delayed_work+0x1d8/0x280 [ceph] [21046.567318] ? process_scheduled_works+0x22a/0x510 [21046.572109] process_scheduled_works+0x2a4/0x510 [21046.576725] worker_thread+0x188/0x340 [21046.580473] ? rescuer_thread+0x310/0x310 [21046.584477] kthread+0xf5/0x130 [21046.587616] ? kthread_complete_and_exit+0x20/0x20 [21046.592407] ret_from_fork+0x31/0x40 [21046.595988] ? kthread_complete_and_exit+0x20/0x20 [21046.600771] ret_from_fork_asm+0x11/0x20 [21046.604703] [21073.827471] watchdog: BUG: soft lockup - CPU#0 stuck for 8512s! [dbench:78683] [21073.834687] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21073.839471] watchdog: BUG: soft lockup - CPU#7 stuck for 17509s! [kworker/7:4:78808] [21073.890145] irq event stamp: 31838908 [21073.890146] hardirqs last enabled at (31838907): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21073.897875] Modules linked in: [21073.901534] hardirqs last disabled at (31838908): [] sysvec_apic_timer_interrupt+0xe/0x90 [21073.911690] ceph [21073.914741] softirqs last enabled at (31838882): [] __do_softirq+0x2be/0x371 [21073.924466] libceph dns_resolver [21073.926389] softirqs last disabled at (31838877): [] irq_exit_rcu+0xb0/0xd0 [21073.935073] fscache [21073.938385] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21073.946894] netfs veth [21073.949078] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21073.958024] nft_chain_nat [21073.960467] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21073.967937] xt_MASQUERADE nf_nat [21073.970643] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21073.976119] nf_conntrack [21073.979431] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21073.998168] nf_defrag_ipv6 nf_defrag_ipv4 [21074.000784] [21074.000785] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21074.006002] nft_compat [21074.010093] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21074.011584] nf_tables nfnetlink [21074.018708] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21074.021153] bridge stp [21074.028277] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21074.031500] llc xfs [21074.038625] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21074.041069] libcrc32c sunrpc [21074.048192] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21074.050377] intel_rapl_msr intel_rapl_common [21074.057500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21074.060464] sb_edac x86_pkg_temp_thermal [21074.068542] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21074.072893] coretemp kvm_intel [21074.078631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21074.082636] kvm irqbypass [21074.089759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21074.092896] i2c_i801 iTCO_wdt [21074.100019] Call Trace: [21074.100021] [21074.102725] iTCO_vendor_support i2c_smbus [21074.109867] ? watchdog_timer_fn+0x1b4/0x220 [21074.112916] crct10dif_pclmul joydev [21074.115361] ? __hrtimer_run_queues+0x1d3/0x360 [21074.117371] crc32_pclmul ipmi_ssif [21074.121465] ? hrtimer_interrupt+0x104/0x240 [21074.125725] ghash_clmulni_intel mei_me [21074.129298] ? __local_bh_enable+0x50/0x60 [21074.133822] mei lpc_ich [21074.137309] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21074.141568] sha512_ssse3 acpi_ipmi [21074.145399] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21074.149490] mfd_core wmi [21074.152021] [21074.157239] ipmi_si [21074.160722] [21074.160724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21074.165680] ipmi_devintf [21074.168301] ? queued_spin_lock_slowpath+0x16/0x2e0 [21074.170395] ipmi_msghandler ioatdma [21074.172581] do_raw_spin_lock+0xb5/0xc0 [21074.174676] acpi_pad acpi_power_meter [21074.179981] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21074.182599] ixgbe igb [21074.187471] try_get_cap_refs+0x159/0x880 [ceph] [21074.191040] mdio crc32c_intel [21074.194876] __ceph_get_caps+0x4a1/0x760 [ceph] [21074.198613] nvme ptp [21074.203403] ? __file_remove_privs+0x89/0x110 [21074.205755] nvme_core [21074.210367] ? wake_bit_function+0x70/0x70 [21074.213418] i2c_algo_bit pps_core [21074.217944] ? ceph_write_iter+0x260/0x770 [ceph] [21074.220212] dca fuse [21074.224563] ceph_write_iter+0x260/0x770 [ceph] [21074.226919] [21074.226920] irq event stamp: 91154948 [21074.231015] ? lock_is_held_type+0xaa/0x120 [21074.234408] hardirqs last enabled at (91154947): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21074.239108] vfs_write+0x324/0x4f0 [21074.241376] hardirqs last disabled at (91154948): [] sysvec_apic_timer_interrupt+0xe/0x90 [21074.245905] ksys_pwrite64+0x65/0xa0 [21074.247392] softirqs last enabled at (91154946): [] __do_softirq+0x2be/0x371 [21074.251048] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21074.255225] softirqs last disabled at (91154941): [] irq_exit_rcu+0xb0/0xd0 [21074.265386] do_syscall_64+0x38/0x80 [21074.268780] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21074.278506] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21074.282077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21074.290759] RIP: 0033:0x7f48f0eeb93b [21074.295890] Workqueue: events delayed_work [ceph] [21074.304401] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21074.307973] [21074.307974] RIP: 0010:lock_acquire+0xf0/0x2b0 [21074.317350] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21074.322393] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21074.329874] ORIG_RAX: 0000000000000012 [21074.333443] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21074.338155] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21074.356898] [21074.356898] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21074.358387] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21074.362739] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21074.367957] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21074.386692] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21074.390525] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21074.395742] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21074.402876] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21074.404365] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21074.411494] [21074.418613] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21074.478753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21074.484489] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21074.491615] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21074.498737] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21074.505877] Call Trace: [21074.508325] [21074.510335] ? watchdog_timer_fn+0x1b4/0x220 [21074.514599] ? __hrtimer_run_queues+0x1d3/0x360 [21074.519127] ? hrtimer_interrupt+0x104/0x240 [21074.523395] ? __local_bh_enable+0x50/0x60 [21074.527489] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21074.532711] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21074.537671] [21074.539767] [21074.541878] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21074.547192] ? lock_acquire+0xf0/0x2b0 [21074.550941] ? igrab+0x43/0x50 [21074.553990] ? lock_release+0xc5/0x260 [21074.557735] _raw_spin_lock+0x30/0x40 [21074.561392] ? igrab+0x1d/0x50 [21074.564440] igrab+0x1d/0x50 [21074.567320] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21074.572470] delayed_work+0x1d8/0x280 [ceph] [21074.576750] ? process_scheduled_works+0x22a/0x510 [21074.581532] process_scheduled_works+0x2a4/0x510 [21074.586164] worker_thread+0x188/0x340 [21074.589915] ? rescuer_thread+0x310/0x310 [21074.593918] kthread+0xf5/0x130 [21074.597054] ? kthread_complete_and_exit+0x20/0x20 [21074.601839] ret_from_fork+0x31/0x40 [21074.605410] ? kthread_complete_and_exit+0x20/0x20 [21074.610195] ret_from_fork_asm+0x11/0x20 [21074.614116] [21101.827968] watchdog: BUG: soft lockup - CPU#0 stuck for 8538s! [dbench:78683] [21101.835183] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21101.839968] watchdog: BUG: soft lockup - CPU#7 stuck for 17535s! [kworker/7:4:78808] [21101.890649] irq event stamp: 31897142 [21101.890650] hardirqs last enabled at (31897141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21101.898380] Modules linked in: [21101.902038] hardirqs last disabled at (31897142): [] sysvec_apic_timer_interrupt+0xe/0x90 [21101.912197] ceph [21101.915246] softirqs last enabled at (31897094): [] __do_softirq+0x2be/0x371 [21101.924971] libceph [21101.926894] softirqs last disabled at (31897089): [] irq_exit_rcu+0xb0/0xd0 [21101.935579] dns_resolver fscache [21101.937766] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21101.946274] netfs veth [21101.949584] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21101.958528] nft_chain_nat [21101.960972] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21101.968444] xt_MASQUERADE [21101.971147] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21101.976639] nf_nat nf_conntrack [21101.979363] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21101.998102] nf_defrag_ipv6 nf_defrag_ipv4 [21102.001326] [21102.006544] nft_compat [21102.010645] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21102.012133] nf_tables nfnetlink [21102.014578] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21102.021700] bridge stp [21102.024926] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21102.032050] llc [21102.034492] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21102.041618] xfs libcrc32c [21102.043456] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21102.050578] sunrpc intel_rapl_msr [21102.053283] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21102.060408] intel_rapl_common [21102.063805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21102.071882] sb_edac [21102.074934] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21102.080671] x86_pkg_temp_thermal [21102.082855] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21102.089979] coretemp kvm_intel [21102.093290] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21102.100414] kvm [21102.103550] Call Trace: [21102.103552] [21102.110674] irqbypass [21102.112515] ? watchdog_timer_fn+0x1b4/0x220 [21102.114957] i2c_i801 iTCO_wdt [21102.116968] ? __hrtimer_run_queues+0x1d3/0x360 [21102.119324] iTCO_vendor_support i2c_smbus [21102.123591] ? hrtimer_interrupt+0x104/0x240 [21102.126645] crct10dif_pclmul joydev [21102.131173] ? __local_bh_enable+0x50/0x60 [21102.135264] crc32_pclmul ipmi_ssif [21102.139529] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21102.143096] ghash_clmulni_intel [21102.147187] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21102.150673] mei_me mei [21102.155890] [21102.159112] lpc_ich sha512_ssse3 [21102.164071] [21102.164073] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21102.166515] acpi_ipmi [21102.168617] ? queued_spin_lock_slowpath+0x16/0x2e0 [21102.171923] mfd_core [21102.174023] do_raw_spin_lock+0xb5/0xc0 [21102.179324] wmi ipmi_si [21102.181685] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21102.186552] ipmi_devintf ipmi_msghandler [21102.188827] try_get_cap_refs+0x159/0x880 [ceph] [21102.192655] ioatdma acpi_pad [21102.195190] __ceph_get_caps+0x4a1/0x760 [ceph] [21102.199969] acpi_power_meter [21102.203979] ? __file_remove_privs+0x89/0x110 [21102.208585] ixgbe igb [21102.211549] ? wake_bit_function+0x70/0x70 [21102.216071] mdio [21102.219038] ? ceph_write_iter+0x260/0x770 [ceph] [21102.223386] crc32c_intel nvme [21102.225746] ceph_write_iter+0x260/0x770 [ceph] [21102.229834] ptp [21102.231763] ? lock_is_held_type+0xaa/0x120 [21102.236458] nvme_core [21102.239509] vfs_write+0x324/0x4f0 [21102.244030] i2c_algo_bit pps_core [21102.245874] ksys_pwrite64+0x65/0xa0 [21102.250045] dca fuse [21102.252405] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21102.255802] [21102.255803] irq event stamp: 91375914 [21102.259198] do_syscall_64+0x38/0x80 [21102.262768] hardirqs last enabled at (91375913): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21102.265041] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21102.270170] hardirqs last disabled at (91375914): [] sysvec_apic_timer_interrupt+0xe/0x90 [21102.271663] RIP: 0033:0x7f48f0eeb93b [21102.275319] softirqs last enabled at (91375912): [] __do_softirq+0x2be/0x371 [21102.278888] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21102.289046] softirqs last disabled at (91375907): [] irq_exit_rcu+0xb0/0xd0 [21102.294092] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21102.303814] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21102.307385] ORIG_RAX: 0000000000000012 [21102.316070] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21102.334808] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21102.343320] Workqueue: events delayed_work [ceph] [21102.348535] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21102.357914] [21102.361744] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21102.369215] RIP: 0010:lock_acquire+0xf0/0x2b0 [21102.376339] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21102.381037] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21102.388160] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21102.389663] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21102.396791] [21102.401134] [21102.401135] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21102.450139] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21102.457261] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21102.464387] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21102.471508] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21102.478635] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21102.486729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21102.492465] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21102.499590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21102.506714] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21102.513838] Call Trace: [21102.516283] [21102.518295] ? watchdog_timer_fn+0x1b4/0x220 [21102.522559] ? __hrtimer_run_queues+0x1d3/0x360 [21102.527084] ? hrtimer_interrupt+0x104/0x240 [21102.531347] ? __local_bh_enable+0x50/0x60 [21102.535464] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21102.540690] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21102.545653] [21102.547754] [21102.549851] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21102.555159] ? lock_acquire+0xf0/0x2b0 [21102.558907] ? igrab+0x43/0x50 [21102.561957] ? lock_release+0xc5/0x260 [21102.565702] _raw_spin_lock+0x30/0x40 [21102.569375] ? igrab+0x1d/0x50 [21102.572426] igrab+0x1d/0x50 [21102.575306] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21102.580455] delayed_work+0x1d8/0x280 [ceph] [21102.584736] ? process_scheduled_works+0x22a/0x510 [21102.589527] process_scheduled_works+0x2a4/0x510 [21102.594141] worker_thread+0x188/0x340 [21102.597891] ? rescuer_thread+0x310/0x310 [21102.601895] kthread+0xf5/0x130 [21102.605039] ? kthread_complete_and_exit+0x20/0x20 [21102.609824] ret_from_fork+0x31/0x40 [21102.613405] ? kthread_complete_and_exit+0x20/0x20 [21102.618197] ret_from_fork_asm+0x11/0x20 [21102.622129] [21129.828463] watchdog: BUG: soft lockup - CPU#0 stuck for 8564s! [dbench:78683] [21129.835679] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21129.840463] watchdog: BUG: soft lockup - CPU#7 stuck for 17561s! [kworker/7:4:78808] [21129.891138] irq event stamp: 31955412 [21129.898869] Modules linked in: ceph [21129.902527] hardirqs last enabled at (31955411): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21129.902529] libceph dns_resolver fscache netfs [21129.916176] hardirqs last disabled at (31955412): [] sysvec_apic_timer_interrupt+0xe/0x90 [21129.920700] veth nft_chain_nat xt_MASQUERADE [21129.930424] softirqs last enabled at (31955386): [] __do_softirq+0x2be/0x371 [21129.930426] nf_nat [21129.934776] nf_conntrack nf_defrag_ipv6 [21129.943460] softirqs last disabled at (31955381): [] irq_exit_rcu+0xb0/0xd0 [21129.945556] nf_defrag_ipv4 nft_compat [21129.949475] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21129.957984] nf_tables nfnetlink [21129.961729] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21129.970673] bridge stp [21129.973897] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21129.981368] llc xfs [21129.983811] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21129.989291] libcrc32c sunrpc [21129.991475] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21130.010212] intel_rapl_msr intel_rapl_common [21130.013176] [21130.013176] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21130.018393] sb_edac x86_pkg_temp_thermal [21130.022743] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21130.024234] coretemp [21130.031360] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21130.035364] kvm_intel kvm [21130.042487] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21130.044757] irqbypass i2c_i801 [21130.051883] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21130.054587] iTCO_wdt iTCO_vendor_support [21130.061710] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21130.064859] i2c_smbus [21130.071979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21130.075984] crct10dif_pclmul joydev [21130.084061] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21130.086420] crc32_pclmul ipmi_ssif [21130.092157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21130.095727] ghash_clmulni_intel mei_me [21130.102861] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21130.102862] Call Trace: [21130.106344] mei [21130.113468] [21130.117299] lpc_ich [21130.124424] ? watchdog_timer_fn+0x1b4/0x220 [21130.126867] sha512_ssse3 acpi_ipmi [21130.128705] ? __hrtimer_run_queues+0x1d3/0x360 [21130.130715] mfd_core [21130.132901] ? hrtimer_interrupt+0x104/0x240 [21130.137163] wmi ipmi_si [21130.140647] ? __local_bh_enable+0x50/0x60 [21130.145171] ipmi_devintf [21130.147445] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21130.151707] ipmi_msghandler [21130.154238] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21130.158327] ioatdma acpi_pad [21130.160944] [21130.166164] acpi_power_meter ixgbe [21130.169041] [21130.169042] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21130.173996] igb mdio [21130.176966] ? queued_spin_lock_slowpath+0x16/0x2e0 [21130.179060] crc32c_intel nvme [21130.182544] do_raw_spin_lock+0xb5/0xc0 [21130.184641] ptp [21130.189945] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21130.192216] nvme_core i2c_algo_bit [21130.197087] try_get_cap_refs+0x159/0x880 [ceph] [21130.200151] pps_core dca [21130.203990] __ceph_get_caps+0x4a1/0x760 [ceph] [21130.205822] fuse [21130.210611] ? __file_remove_privs+0x89/0x110 [21130.214090] [21130.214091] irq event stamp: 91596694 [21130.218701] ? wake_bit_function+0x70/0x70 [21130.221318] hardirqs last enabled at (91596693): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21130.225862] ? ceph_write_iter+0x260/0x770 [ceph] [21130.227782] hardirqs last disabled at (91596694): [] sysvec_apic_timer_interrupt+0xe/0x90 [21130.232144] ceph_write_iter+0x260/0x770 [ceph] [21130.233634] softirqs last enabled at (91596692): [] __do_softirq+0x2be/0x371 [21130.237296] ? lock_is_held_type+0xaa/0x120 [21130.241382] softirqs last disabled at (91596683): [] irq_exit_rcu+0xb0/0xd0 [21130.251542] vfs_write+0x324/0x4f0 [21130.256236] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21130.265965] ksys_pwrite64+0x65/0xa0 [21130.270485] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21130.279170] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21130.283347] Workqueue: events delayed_work [ceph] [21130.291866] do_syscall_64+0x38/0x80 [21130.295264] [21130.295265] RIP: 0010:lock_acquire+0xf0/0x2b0 [21130.304641] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21130.308212] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21130.315683] RIP: 0033:0x7f48f0eeb93b [21130.320812] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21130.325510] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21130.329083] [21130.330571] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21130.334923] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21130.339967] ORIG_RAX: 0000000000000012 [21130.358705] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21130.362276] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21130.367494] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21130.386230] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21130.387722] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21130.392939] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21130.400062] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21130.403894] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21130.411017] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21130.418154] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21130.425274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21130.432405] [21130.439522] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21130.491148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21130.498274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21130.505400] Call Trace: [21130.507844] [21130.509873] ? watchdog_timer_fn+0x1b4/0x220 [21130.514150] ? __hrtimer_run_queues+0x1d3/0x360 [21130.518682] ? hrtimer_interrupt+0x104/0x240 [21130.522950] ? __local_bh_enable+0x50/0x60 [21130.527044] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21130.532268] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21130.537223] [21130.539322] [21130.541420] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21130.546727] ? lock_acquire+0xf0/0x2b0 [21130.550477] ? igrab+0x43/0x50 [21130.553528] ? lock_release+0xc5/0x260 [21130.557272] _raw_spin_lock+0x30/0x40 [21130.560928] ? igrab+0x1d/0x50 [21130.563980] igrab+0x1d/0x50 [21130.566872] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21130.572016] delayed_work+0x1d8/0x280 [ceph] [21130.576298] ? process_scheduled_works+0x22a/0x510 [21130.581087] process_scheduled_works+0x2a4/0x510 [21130.585702] worker_thread+0x188/0x340 [21130.589451] ? rescuer_thread+0x310/0x310 [21130.593454] kthread+0xf5/0x130 [21130.596592] ? kthread_complete_and_exit+0x20/0x20 [21130.601378] ret_from_fork+0x31/0x40 [21130.604955] ? kthread_complete_and_exit+0x20/0x20 [21130.609741] ret_from_fork_asm+0x11/0x20 [21130.613663] [21157.828958] watchdog: BUG: soft lockup - CPU#0 stuck for 8590s! [dbench:78683] [21157.836179] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21157.840958] watchdog: BUG: soft lockup - CPU#7 stuck for 17587s! [kworker/7:4:78808] [21157.891633] irq event stamp: 32013618 [21157.891634] hardirqs last enabled at (32013617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21157.899365] Modules linked in: [21157.903024] hardirqs last disabled at (32013618): [] sysvec_apic_timer_interrupt+0xe/0x90 [21157.913181] ceph [21157.916230] softirqs last enabled at (32013570): [] __do_softirq+0x2be/0x371 [21157.925957] libceph [21157.927881] softirqs last disabled at (32013565): [] irq_exit_rcu+0xb0/0xd0 [21157.936562] dns_resolver [21157.938748] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21157.947259] fscache netfs [21157.949878] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21157.958820] veth [21157.961525] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21157.968995] nft_chain_nat [21157.970921] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21157.976397] xt_MASQUERADE [21157.979100] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21157.997839] nf_nat nf_conntrack [21158.000542] [21158.005759] nf_defrag_ipv6 [21158.008984] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21158.010475] nf_defrag_ipv4 nft_compat [21158.013264] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21158.020389] nf_tables nfnetlink [21158.024134] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21158.031258] bridge stp [21158.034483] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21158.041605] llc xfs [21158.044049] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21158.051174] libcrc32c [21158.053358] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21158.060483] sunrpc intel_rapl_msr [21158.062840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21158.070918] intel_rapl_common [21158.074314] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21158.080053] sb_edac [21158.083104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21158.090228] x86_pkg_temp_thermal [21158.092411] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21158.099533] coretemp kvm_intel [21158.102846] Call Trace: [21158.109968] kvm [21158.113109] [21158.115550] irqbypass i2c_i801 [21158.117389] ? watchdog_timer_fn+0x1b4/0x220 [21158.119400] iTCO_wdt iTCO_vendor_support [21158.122537] ? __hrtimer_run_queues+0x1d3/0x360 [21158.126800] i2c_smbus crct10dif_pclmul [21158.130809] ? hrtimer_interrupt+0x104/0x240 [21158.135329] joydev crc32_pclmul [21158.139176] ? __local_bh_enable+0x50/0x60 [21158.143442] ipmi_ssif [21158.146667] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21158.150757] ghash_clmulni_intel mei_me [21158.153115] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21158.158330] mei lpc_ich [21158.162179] [21158.167135] sha512_ssse3 acpi_ipmi [21158.169667] [21158.169668] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21158.171763] mfd_core [21158.175253] ? queued_spin_lock_slowpath+0x18/0x2e0 [21158.177357] wmi ipmi_si [21158.182660] do_raw_spin_lock+0xb5/0xc0 [21158.184929] ipmi_devintf ipmi_msghandler [21158.189802] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21158.192331] ioatdma acpi_pad [21158.196182] try_get_cap_refs+0x159/0x880 [ceph] [21158.200182] acpi_power_meter ixgbe [21158.204972] __ceph_get_caps+0x4a1/0x760 [ceph] [21158.207930] igb mdio [21158.212546] ? __file_remove_privs+0x89/0x110 [21158.216025] crc32c_intel nvme [21158.220550] ? wake_bit_function+0x70/0x70 [21158.222820] ptp nvme_core [21158.227173] ? ceph_write_iter+0x260/0x770 [ceph] [21158.230221] i2c_algo_bit [21158.234314] ceph_write_iter+0x260/0x770 [ceph] [21158.237017] pps_core [21158.241719] ? lock_is_held_type+0xaa/0x120 [21158.244332] dca fuse [21158.248875] vfs_write+0x324/0x4f0 [21158.251143] [21158.255324] ksys_pwrite64+0x65/0xa0 [21158.257593] irq event stamp: 91817644 [21158.260990] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21158.262480] hardirqs last enabled at (91817643): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21158.266051] do_syscall_64+0x38/0x80 [21158.269708] hardirqs last disabled at (91817644): [] sysvec_apic_timer_interrupt+0xe/0x90 [21158.274840] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21158.284995] softirqs last enabled at (91817642): [] __do_softirq+0x2be/0x371 [21158.288568] RIP: 0033:0x7f48f0eeb93b [21158.298291] softirqs last disabled at (91817637): [] irq_exit_rcu+0xb0/0xd0 [21158.303336] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21158.312038] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21158.315608] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21158.324117] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21158.342857] ORIG_RAX: 0000000000000012 [21158.352234] Workqueue: events delayed_work [ceph] [21158.357451] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21158.364921] [21158.368751] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21158.373450] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [21158.380574] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21158.382064] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [21158.389189] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21158.393799] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [21158.400924] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21158.419663] [21158.426791] [21158.432002] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [21158.449927] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [21158.457050] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [21158.464174] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [21158.471300] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [21158.478422] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21158.486501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21158.492238] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21158.499363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21158.506486] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21158.513610] Call Trace: [21158.516054] [21158.518065] ? watchdog_timer_fn+0x1b4/0x220 [21158.522329] ? __hrtimer_run_queues+0x1d3/0x360 [21158.526856] ? hrtimer_interrupt+0x104/0x240 [21158.531125] ? __local_bh_enable+0x50/0x60 [21158.535219] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21158.540443] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21158.545399] [21158.547498] [21158.549595] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21158.554904] ? do_raw_spin_lock+0x42/0xc0 [21158.558913] igrab+0x1d/0x50 [21158.561791] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21158.566941] delayed_work+0x1d8/0x280 [ceph] [21158.571221] ? process_scheduled_works+0x22a/0x510 [21158.576012] process_scheduled_works+0x2a4/0x510 [21158.580629] worker_thread+0x188/0x340 [21158.584394] ? rescuer_thread+0x310/0x310 [21158.588406] kthread+0xf5/0x130 [21158.591553] ? kthread_complete_and_exit+0x20/0x20 [21158.596335] ret_from_fork+0x31/0x40 [21158.599932] ? kthread_complete_and_exit+0x20/0x20 [21158.604716] ret_from_fork_asm+0x11/0x20 [21158.608651] [21185.829453] watchdog: BUG: soft lockup - CPU#0 stuck for 8616s! [dbench:78683] [21185.836666] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21185.841453] watchdog: BUG: soft lockup - CPU#7 stuck for 17613s! [kworker/7:4:78808] [21185.892123] irq event stamp: 32071830 [21185.892124] hardirqs last enabled at (32071829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21185.899854] Modules linked in: [21185.903511] hardirqs last disabled at (32071830): [] sysvec_apic_timer_interrupt+0xe/0x90 [21185.913668] ceph [21185.916720] softirqs last enabled at (32071782): [] __do_softirq+0x2be/0x371 [21185.926443] libceph dns_resolver [21185.928369] softirqs last disabled at (32071777): [] irq_exit_rcu+0xb0/0xd0 [21185.937051] fscache [21185.940364] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21185.948872] netfs veth [21185.951057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21185.951058] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21185.960003] nft_chain_nat xt_MASQUERADE [21185.962445] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21185.969916] nf_nat nf_conntrack [21185.975394] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21185.979310] nf_defrag_ipv6 nf_defrag_ipv4 [21185.998048] [21185.998049] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21186.001272] nft_compat nf_tables [21186.006490] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21186.010582] nfnetlink bridge [21186.012073] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21186.019195] stp llc [21186.022506] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21186.029631] xfs libcrc32c [21186.032594] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21186.039719] sunrpc intel_rapl_msr [21186.041903] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21186.049026] intel_rapl_common [21186.051732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21186.058855] sb_edac [21186.062253] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21186.070331] x86_pkg_temp_thermal coretemp [21186.073382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21186.079135] kvm_intel [21186.081319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21186.088444] kvm irqbypass [21186.092534] Call Trace: [21186.092535] [21186.099659] i2c_i801 iTCO_wdt [21186.102017] ? watchdog_timer_fn+0x1b4/0x220 [21186.109140] iTCO_vendor_support i2c_smbus [21186.111852] ? __hrtimer_run_queues+0x1d3/0x360 [21186.114296] crct10dif_pclmul joydev [21186.116310] ? hrtimer_interrupt+0x104/0x240 [21186.119358] crc32_pclmul ipmi_ssif [21186.123622] ? __local_bh_enable+0x50/0x60 [21186.127713] ghash_clmulni_intel [21186.132240] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21186.135809] mei_me mei [21186.140074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21186.143557] lpc_ich sha512_ssse3 [21186.147647] [21186.150872] acpi_ipmi mfd_core [21186.156088] [21186.158533] wmi ipmi_si [21186.163492] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21186.166802] ipmi_devintf ipmi_msghandler [21186.168902] ? queued_spin_lock_slowpath+0x16/0x2e0 [21186.172037] ioatdma acpi_pad [21186.174134] do_raw_spin_lock+0xb5/0xc0 [21186.176664] acpi_power_meter ixgbe [21186.181969] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21186.185973] igb mdio [21186.190854] try_get_cap_refs+0x159/0x880 [ceph] [21186.193815] crc32c_intel nvme [21186.197651] __ceph_get_caps+0x4a1/0x760 [ceph] [21186.201145] ptp nvme_core [21186.205935] ? __file_remove_privs+0x89/0x110 [21186.208202] i2c_algo_bit pps_core [21186.212814] ? wake_bit_function+0x70/0x70 [21186.215863] dca fuse [21186.220391] ? ceph_write_iter+0x260/0x770 [ceph] [21186.223091] [21186.223092] irq event stamp: 92038572 [21186.227443] ceph_write_iter+0x260/0x770 [ceph] [21186.230852] hardirqs last enabled at (92038571): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21186.234942] ? lock_is_held_type+0xaa/0x120 [21186.237209] hardirqs last disabled at (92038572): [] sysvec_apic_timer_interrupt+0xe/0x90 [21186.241910] vfs_write+0x324/0x4f0 [21186.243398] softirqs last enabled at (92038570): [] __do_softirq+0x2be/0x371 [21186.247059] ksys_pwrite64+0x65/0xa0 [21186.251580] softirqs last disabled at (92038565): [] irq_exit_rcu+0xb0/0xd0 [21186.261737] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21186.265915] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21186.275642] do_syscall_64+0x38/0x80 [21186.279036] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21186.287722] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21186.291306] Workqueue: events delayed_work [ceph] [21186.299820] RIP: 0033:0x7f48f0eeb93b [21186.304949] [21186.304950] RIP: 0010:lock_acquire+0xf0/0x2b0 [21186.314328] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21186.317898] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21186.325368] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21186.330412] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21186.335111] ORIG_RAX: 0000000000000012 [21186.338699] [21186.338699] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21186.340188] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21186.344541] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21186.363278] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21186.382016] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21186.387232] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21186.392451] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21186.396280] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21186.397771] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21186.404896] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21186.412019] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21186.419148] [21186.426268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21186.485013] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21186.492138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21186.499261] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21186.506384] Call Trace: [21186.508829] [21186.510858] ? watchdog_timer_fn+0x1b4/0x220 [21186.515123] ? __hrtimer_run_queues+0x1d3/0x360 [21186.519666] ? hrtimer_interrupt+0x104/0x240 [21186.523935] ? __local_bh_enable+0x50/0x60 [21186.528028] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21186.533253] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21186.538208] [21186.540308] [21186.542404] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21186.547712] ? lock_acquire+0xf0/0x2b0 [21186.551463] ? igrab+0x43/0x50 [21186.554514] ? lock_release+0xc5/0x260 [21186.558259] _raw_spin_lock+0x30/0x40 [21186.561922] ? igrab+0x1d/0x50 [21186.564974] igrab+0x1d/0x50 [21186.567860] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21186.573010] delayed_work+0x1d8/0x280 [ceph] [21186.577291] ? process_scheduled_works+0x22a/0x510 [21186.582080] process_scheduled_works+0x2a4/0x510 [21186.586696] worker_thread+0x188/0x340 [21186.590446] ? rescuer_thread+0x310/0x310 [21186.594449] kthread+0xf5/0x130 [21186.597587] ? kthread_complete_and_exit+0x20/0x20 [21186.602371] ret_from_fork+0x31/0x40 [21186.605940] ? kthread_complete_and_exit+0x20/0x20 [21186.610724] ret_from_fork_asm+0x11/0x20 [21186.614647] [21187.544850] rcu: INFO: rcu_preempt self-detected stall on CPU [21187.550610] rcu: 7-....: (8983619 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/8982353 fqs=4509548 [21187.561033] rcu: (t=9250768 jiffies g=1309609 q=320359 ncpus=8) [21187.567033] Sending NMI from CPU 7 to CPUs 0: [21187.571382] NMI backtrace for cpu 0 [21187.571384] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21187.571386] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21187.571386] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21187.571389] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21187.571391] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21187.571392] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21187.571393] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21187.571394] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21187.571394] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21187.571395] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21187.571396] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21187.571397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21187.571398] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21187.571399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21187.571399] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21187.571400] Call Trace: [21187.571401] [21187.571402] ? nmi_cpu_backtrace+0xad/0x120 [21187.571405] ? nmi_cpu_backtrace_handler+0x11/0x20 [21187.571407] ? nmi_handle+0xd4/0x260 [21187.571412] ? default_do_nmi+0x49/0xf0 [21187.571413] ? exc_nmi+0x11e/0x150 [21187.571415] ? end_repeat_nmi+0x16/0x67 [21187.571419] ? queued_spin_lock_slowpath+0x18/0x2e0 [21187.571422] ? queued_spin_lock_slowpath+0x18/0x2e0 [21187.571424] ? queued_spin_lock_slowpath+0x18/0x2e0 [21187.571427] [21187.571427] [21187.571428] do_raw_spin_lock+0xb5/0xc0 [21187.571430] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21187.571444] try_get_cap_refs+0x159/0x880 [ceph] [21187.571459] __ceph_get_caps+0x4a1/0x760 [ceph] [21187.571472] ? __file_remove_privs+0x89/0x110 [21187.571473] ? wake_bit_function+0x70/0x70 [21187.571476] ? ceph_write_iter+0x260/0x770 [ceph] [21187.571487] ceph_write_iter+0x260/0x770 [ceph] [21187.571498] ? lock_is_held_type+0xaa/0x120 [21187.571501] vfs_write+0x324/0x4f0 [21187.571505] ksys_pwrite64+0x65/0xa0 [21187.571507] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21187.571510] do_syscall_64+0x38/0x80 [21187.571512] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21187.571514] RIP: 0033:0x7f48f0eeb93b [21187.571515] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21187.571516] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [21187.571518] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21187.571518] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21187.571519] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21187.571520] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21187.571520] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21187.571523] [21187.572381] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21187.876194] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21187.883665] Workqueue: events delayed_work [ceph] [21187.888380] RIP: 0010:lock_acquire+0xf0/0x2b0 [21187.892730] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21187.911468] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21187.916686] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21187.923811] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21187.930933] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21187.938058] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21187.945183] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21187.952305] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21187.960384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21187.966122] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21187.973245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21187.980370] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21187.987493] Call Trace: [21187.989937] [21187.991950] ? rcu_dump_cpu_stacks+0xd1/0x180 [21187.996309] ? rcu_sched_clock_irq+0x633/0x1390 [21188.000831] ? lock_release+0xc5/0x260 [21188.004577] ? lock_is_held_type+0xaa/0x120 [21188.008754] ? lock_is_held_type+0xaa/0x120 [21188.012932] ? tick_sched_do_timer+0xa0/0xa0 [21188.017204] ? rcu_read_lock_sched_held+0x56/0x90 [21188.021901] ? tick_sched_do_timer+0xa0/0xa0 [21188.026166] ? update_process_times+0x5f/0x90 [21188.030516] ? tick_sched_handle.isra.16+0x1e/0x50 [21188.035299] ? tick_sched_timer+0x6f/0x80 [21188.039305] ? __hrtimer_run_queues+0x1d3/0x360 [21188.043832] ? hrtimer_interrupt+0x104/0x240 [21188.048100] ? __local_bh_enable+0x50/0x60 [21188.052193] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21188.057417] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21188.062374] [21188.064473] [21188.066570] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21188.071878] ? lock_acquire+0xf0/0x2b0 [21188.075629] ? igrab+0x43/0x50 [21188.078687] ? lock_release+0xc5/0x260 [21188.082432] _raw_spin_lock+0x30/0x40 [21188.086096] ? igrab+0x1d/0x50 [21188.089148] igrab+0x1d/0x50 [21188.092024] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21188.097175] delayed_work+0x1d8/0x280 [ceph] [21188.101458] ? process_scheduled_works+0x22a/0x510 [21188.106245] process_scheduled_works+0x2a4/0x510 [21188.110864] worker_thread+0x188/0x340 [21188.114611] ? rescuer_thread+0x310/0x310 [21188.118623] kthread+0xf5/0x130 [21188.121760] ? kthread_complete_and_exit+0x20/0x20 [21188.126545] ret_from_fork+0x31/0x40 [21188.130115] ? kthread_complete_and_exit+0x20/0x20 [21188.134900] ret_from_fork_asm+0x11/0x20 [21188.138822] [21213.829948] watchdog: BUG: soft lockup - CPU#0 stuck for 8642s! [dbench:78683] [21213.837162] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21213.841948] watchdog: BUG: soft lockup - CPU#7 stuck for 17639s! [kworker/7:4:78808] [21213.892627] irq event stamp: 32130102 [21213.900359] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21213.904016] hardirqs last enabled at (32130101): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21213.910975] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [21213.921135] hardirqs last disabled at (32130102): [] sysvec_apic_timer_interrupt+0xe/0x90 [21213.928086] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [21213.937810] softirqs last enabled at (32130062): [] __do_softirq+0x2be/0x371 [21213.943285] bridge stp llc xfs libcrc32c sunrpc [21213.951969] softirqs last disabled at (32130057): [] irq_exit_rcu+0xb0/0xd0 [21213.956580] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [21213.965092] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21213.972737] kvm_intel kvm irqbypass [21213.981680] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21213.985251] i2c_i801 iTCO_wdt iTCO_vendor_support [21213.992722] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21213.997505] i2c_smbus crct10dif_pclmul joydev [21214.002982] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21214.007421] crc32_pclmul ipmi_ssif ghash_clmulni_intel [21214.026158] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21214.031375] mei_me mei [21214.036593] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21214.036594] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21214.039038] lpc_ich sha512_ssse3 [21214.046161] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21214.053285] acpi_ipmi mfd_core [21214.056597] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21214.063721] wmi ipmi_si [21214.066857] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21214.073981] ipmi_devintf [21214.076511] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21214.083636] ipmi_msghandler [21214.086254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21214.094346] ioatdma acpi_pad [21214.097226] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21214.102964] acpi_power_meter [21214.105927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21214.113051] ixgbe [21214.116015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21214.123139] igb mdio [21214.125151] Call Trace: [21214.125154] [21214.132274] crc32c_intel nvme [21214.134549] ? watchdog_timer_fn+0x1b4/0x220 [21214.136991] ptp nvme_core [21214.139003] ? __hrtimer_run_queues+0x1d3/0x360 [21214.142051] i2c_algo_bit [21214.146318] ? hrtimer_interrupt+0x104/0x240 [21214.149018] pps_core dca [21214.153543] ? __local_bh_enable+0x50/0x60 [21214.156161] fuse [21214.160427] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21214.163043] irq event stamp: 92254612 [21214.167134] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21214.169058] hardirqs last enabled at (92254611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21214.174274] [21214.177932] hardirqs last disabled at (92254612): [] sysvec_apic_timer_interrupt+0xe/0x90 [21214.182890] [21214.182892] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21214.193048] softirqs last enabled at (92254610): [] __do_softirq+0x2be/0x371 [21214.195147] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21214.204869] softirqs last disabled at (92254605): [] irq_exit_rcu+0xb0/0xd0 [21214.206969] do_raw_spin_lock+0xb5/0xc0 [21214.212271] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21214.220955] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21214.225825] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21214.234353] try_get_cap_refs+0x159/0x880 [ceph] [21214.238184] Workqueue: events delayed_work [ceph] [21214.247565] __ceph_get_caps+0x4a1/0x760 [ceph] [21214.252349] [21214.259829] ? __file_remove_privs+0x89/0x110 [21214.264435] RIP: 0010:lock_release+0x14e/0x260 [21214.269132] ? wake_bit_function+0x70/0x70 [21214.273656] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [21214.275150] ? ceph_write_iter+0x260/0x770 [ceph] [21214.279497] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [21214.283935] ceph_write_iter+0x260/0x770 [ceph] [21214.288026] [21214.306768] ? lock_is_held_type+0xaa/0x120 [21214.311460] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [21214.316682] vfs_write+0x324/0x4f0 [21214.321204] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [21214.322699] ksys_pwrite64+0x65/0xa0 [21214.326870] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [21214.333995] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21214.337392] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [21214.344518] do_syscall_64+0x38/0x80 [21214.348086] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [21214.355213] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21214.360350] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21214.367476] RIP: 0033:0x7f48f0eeb93b [21214.371045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21214.378170] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21214.383215] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21214.391291] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21214.394863] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21214.400599] ORIG_RAX: 0000000000000012 [21214.419353] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21214.426478] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21214.431696] Call Trace: [21214.431697] [21214.438820] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21214.442651] ? watchdog_timer_fn+0x1b4/0x220 [21214.449774] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21214.456900] ? __hrtimer_run_queues+0x1d3/0x360 [21214.459351] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21214.461367] ? hrtimer_interrupt+0x104/0x240 [21214.468486] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21214.472751] ? __local_bh_enable+0x50/0x60 [21214.479881] [21214.484402] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21214.514404] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21214.519362] [21214.521459] [21214.523557] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21214.528862] ? igrab+0x43/0x50 [21214.531914] ? lock_release+0x14e/0x260 [21214.535752] ? lock_release+0xc5/0x260 [21214.539496] _raw_spin_unlock+0x1b/0x40 [21214.543327] igrab+0x43/0x50 [21214.546221] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21214.551370] delayed_work+0x1d8/0x280 [ceph] [21214.555652] ? process_scheduled_works+0x22a/0x510 [21214.560443] process_scheduled_works+0x2a4/0x510 [21214.565058] worker_thread+0x188/0x340 [21214.568807] ? rescuer_thread+0x310/0x310 [21214.572810] kthread+0xf5/0x130 [21214.575948] ? kthread_complete_and_exit+0x20/0x20 [21214.580734] ret_from_fork+0x31/0x40 [21214.584310] ? kthread_complete_and_exit+0x20/0x20 [21214.589096] ret_from_fork_asm+0x11/0x20 [21214.593028] [21241.830446] watchdog: BUG: soft lockup - CPU#0 stuck for 8668s! [dbench:78683] [21241.837658] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21241.842446] watchdog: BUG: soft lockup - CPU#7 stuck for 17666s! [kworker/7:4:78808] [21241.893098] irq event stamp: 32188168 [21241.893099] hardirqs last enabled at (32188167): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21241.900838] Modules linked in: [21241.904496] hardirqs last disabled at (32188168): [] sysvec_apic_timer_interrupt+0xe/0x90 [21241.914653] ceph [21241.917704] softirqs last enabled at (32188130): [] __do_softirq+0x2be/0x371 [21241.927429] libceph dns_resolver [21241.929351] softirqs last disabled at (32188125): [] irq_exit_rcu+0xb0/0xd0 [21241.938037] fscache netfs [21241.941346] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21241.949859] veth nft_chain_nat [21241.952561] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21241.961506] xt_MASQUERADE nf_nat [21241.964644] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21241.972129] nf_conntrack nf_defrag_ipv6 [21241.975441] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21241.980919] nf_defrag_ipv4 nft_compat [21241.984841] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21242.003576] nf_tables nfnetlink [21242.007318] [21242.007318] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21242.012537] bridge stp [21242.015761] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21242.017250] llc xfs [21242.024376] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21242.026820] libcrc32c [21242.033960] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21242.036143] sunrpc [21242.043269] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21242.045626] intel_rapl_msr intel_rapl_common [21242.052751] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21242.054848] sb_edac x86_pkg_temp_thermal coretemp [21242.061973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21242.066321] kvm_intel kvm [21242.074399] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21242.079185] irqbypass i2c_i801 [21242.084922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21242.087627] iTCO_wdt iTCO_vendor_support [21242.094751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21242.097886] i2c_smbus [21242.105012] Call Trace: [21242.105013] [21242.109016] crct10dif_pclmul [21242.116141] ? watchdog_timer_fn+0x1b4/0x220 [21242.118497] joydev crc32_pclmul [21242.120941] ? __hrtimer_run_queues+0x1d3/0x360 [21242.122952] ipmi_ssif ghash_clmulni_intel [21242.125918] ? hrtimer_interrupt+0x104/0x240 [21242.130179] mei_me mei [21242.133404] ? __local_bh_enable+0x50/0x60 [21242.137927] lpc_ich sha512_ssse3 [21242.142020] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21242.146284] acpi_ipmi mfd_core [21242.148727] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21242.152818] wmi ipmi_si [21242.156146] [21242.161363] ipmi_devintf ipmi_msghandler [21242.164499] [21242.164501] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21242.169458] ioatdma [21242.171994] ? queued_spin_lock_slowpath+0x16/0x2e0 [21242.174086] acpi_pad acpi_power_meter [21242.178093] do_raw_spin_lock+0xb5/0xc0 [21242.180188] ixgbe igb [21242.185492] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21242.187675] mdio [21242.192549] try_get_cap_refs+0x159/0x880 [ceph] [21242.196290] crc32c_intel nvme [21242.200138] __ceph_get_caps+0x4a1/0x760 [ceph] [21242.202487] ptp [21242.207275] ? __file_remove_privs+0x89/0x110 [21242.209196] nvme_core i2c_algo_bit [21242.213807] ? wake_bit_function+0x70/0x70 [21242.216855] pps_core dca [21242.221383] ? ceph_write_iter+0x260/0x770 [ceph] [21242.223218] fuse [21242.227570] ceph_write_iter+0x260/0x770 [ceph] [21242.231052] irq event stamp: 92475634 [21242.235147] ? lock_is_held_type+0xaa/0x120 [21242.237761] hardirqs last enabled at (92475633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21242.242460] vfs_write+0x324/0x4f0 [21242.244381] hardirqs last disabled at (92475634): [] sysvec_apic_timer_interrupt+0xe/0x90 [21242.248911] ksys_pwrite64+0x65/0xa0 [21242.252563] softirqs last enabled at (92475632): [] __do_softirq+0x2be/0x371 [21242.256742] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21242.266899] softirqs last disabled at (92475627): [] irq_exit_rcu+0xb0/0xd0 [21242.270298] do_syscall_64+0x38/0x80 [21242.280021] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21242.283592] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21242.292275] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21242.297407] RIP: 0033:0x7f48f0eeb93b [21242.305916] Workqueue: events delayed_work [ceph] [21242.309489] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21242.318864] [21242.323909] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21242.331380] RIP: 0010:lock_acquire+0xf0/0x2b0 [21242.334949] ORIG_RAX: 0000000000000012 [21242.339648] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21242.358387] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21242.359876] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21242.365093] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21242.369444] [21242.373275] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21242.392014] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21242.399138] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21242.404353] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21242.411478] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21242.412970] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21242.420100] [21242.427217] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21242.465024] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21242.472147] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21242.480225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21242.485963] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21242.493087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21242.500210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21242.507333] Call Trace: [21242.509781] [21242.511790] ? watchdog_timer_fn+0x1b4/0x220 [21242.516055] ? __hrtimer_run_queues+0x1d3/0x360 [21242.520582] ? hrtimer_interrupt+0x104/0x240 [21242.524851] ? __local_bh_enable+0x50/0x60 [21242.528944] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21242.534168] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21242.539132] [21242.541229] [21242.543329] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21242.548637] ? lock_acquire+0xf0/0x2b0 [21242.552387] ? igrab+0x43/0x50 [21242.555436] ? lock_release+0xc5/0x260 [21242.559181] _raw_spin_lock+0x30/0x40 [21242.562853] ? igrab+0x1d/0x50 [21242.565906] igrab+0x1d/0x50 [21242.568783] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21242.573934] delayed_work+0x1d8/0x280 [ceph] [21242.578213] ? process_scheduled_works+0x22a/0x510 [21242.582996] process_scheduled_works+0x2a4/0x510 [21242.587611] worker_thread+0x188/0x340 [21242.591360] ? rescuer_thread+0x310/0x310 [21242.595364] kthread+0xf5/0x130 [21242.598502] ? kthread_complete_and_exit+0x20/0x20 [21242.603286] ret_from_fork+0x31/0x40 [21242.606855] ? kthread_complete_and_exit+0x20/0x20 [21242.611641] ret_from_fork_asm+0x11/0x20 [21242.615563] [21269.830944] watchdog: BUG: soft lockup - CPU#0 stuck for 8695s! [dbench:78683] [21269.838164] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21269.842944] watchdog: BUG: soft lockup - CPU#7 stuck for 17692s! [kworker/7:4:78808] [21269.893621] irq event stamp: 32246436 [21269.901352] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21269.905008] hardirqs last enabled at (32246435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21269.911960] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [21269.922116] hardirqs last disabled at (32246436): [] sysvec_apic_timer_interrupt+0xe/0x90 [21269.922119] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [21269.938791] softirqs last enabled at (32246398): [] __do_softirq+0x2be/0x371 [21269.938795] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [21269.954254] softirqs last disabled at (32246393): [] irq_exit_rcu+0xb0/0xd0 [21269.962946] coretemp kvm_intel kvm irqbypass [21269.971459] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21269.975808] i2c_i801 iTCO_wdt iTCO_vendor_support [21269.984753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21269.984755] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21269.989535] i2c_smbus crct10dif_pclmul [21269.997007] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21270.002484] joydev crc32_pclmul [21270.006316] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21270.025069] ipmi_ssif ghash_clmulni_intel [21270.028296] [21270.028297] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21270.033513] mei_me mei [21270.037602] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21270.039093] lpc_ich sha512_ssse3 [21270.046218] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21270.048662] acpi_ipmi mfd_core [21270.055785] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21270.059096] wmi ipmi_si [21270.066222] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21270.069359] ipmi_devintf [21270.076482] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21270.079013] ipmi_msghandler [21270.086137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21270.088754] ioatdma acpi_pad [21270.096831] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21270.099710] acpi_power_meter [21270.105447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21270.105448] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21270.108411] ixgbe [21270.115535] Call Trace: [21270.118498] igb mdio [21270.125627] [21270.132747] crc32c_intel [21270.134760] ? watchdog_timer_fn+0x1b4/0x220 [21270.137201] nvme ptp [21270.139473] ? __hrtimer_run_queues+0x1d3/0x360 [21270.141482] nvme_core i2c_algo_bit [21270.144105] ? hrtimer_interrupt+0x104/0x240 [21270.148366] pps_core [21270.150635] ? __local_bh_enable+0x50/0x60 [21270.155160] dca fuse [21270.158645] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21270.162907] [21270.162907] irq event stamp: 92696524 [21270.165178] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21270.169270] hardirqs last enabled at (92696523): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21270.171541] [21270.176757] hardirqs last disabled at (92696524): [] sysvec_apic_timer_interrupt+0xe/0x90 [21270.178249] [21270.178250] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21270.181905] softirqs last enabled at (92696522): [] __do_softirq+0x2be/0x371 [21270.186867] ? queued_spin_lock_slowpath+0x18/0x2e0 [21270.197021] softirqs last disabled at (92696517): [] irq_exit_rcu+0xb0/0xd0 [21270.199121] do_raw_spin_lock+0xb5/0xc0 [21270.208841] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21270.210943] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21270.216244] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21270.224931] try_get_cap_refs+0x159/0x880 [ceph] [21270.229798] Workqueue: events delayed_work [ceph] [21270.238314] __ceph_get_caps+0x4a1/0x760 [ceph] [21270.242140] [21270.251524] ? __file_remove_privs+0x89/0x110 [21270.256302] RIP: 0010:lock_acquire+0xf0/0x2b0 [21270.263774] ? wake_bit_function+0x70/0x70 [21270.268384] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21270.273084] ? ceph_write_iter+0x260/0x770 [ceph] [21270.277605] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21270.279097] ceph_write_iter+0x260/0x770 [ceph] [21270.283446] [21270.287800] ? lock_is_held_type+0xaa/0x120 [21270.291888] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21270.310629] vfs_write+0x324/0x4f0 [21270.315323] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21270.320563] ksys_pwrite64+0x65/0xa0 [21270.325081] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21270.326572] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21270.330749] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21270.337875] do_syscall_64+0x38/0x80 [21270.341271] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21270.348398] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21270.351966] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21270.359091] RIP: 0033:0x7f48f0eeb93b [21270.364221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21270.371349] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21270.374916] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21270.382041] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21270.387083] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21270.395162] ORIG_RAX: 0000000000000012 [21270.398733] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21270.404469] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21270.423207] Call Trace: [21270.423208] [21270.430346] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21270.435567] ? watchdog_timer_fn+0x1b4/0x220 [21270.442689] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21270.446522] ? __hrtimer_run_queues+0x1d3/0x360 [21270.453646] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21270.460773] ? hrtimer_interrupt+0x104/0x240 [21270.463215] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21270.465224] ? __local_bh_enable+0x50/0x60 [21270.472354] [21270.476631] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21270.518284] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21270.523241] [21270.525349] [21270.527444] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21270.532752] ? lock_acquire+0xf0/0x2b0 [21270.536503] ? igrab+0x43/0x50 [21270.539560] ? lock_release+0xc5/0x260 [21270.543307] _raw_spin_lock+0x30/0x40 [21270.546970] ? igrab+0x1d/0x50 [21270.550021] igrab+0x1d/0x50 [21270.552898] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21270.558048] delayed_work+0x1d8/0x280 [ceph] [21270.562350] ? process_scheduled_works+0x22a/0x510 [21270.567138] process_scheduled_works+0x2a4/0x510 [21270.571754] worker_thread+0x188/0x340 [21270.575502] ? rescuer_thread+0x310/0x310 [21270.579505] kthread+0xf5/0x130 [21270.582647] ? kthread_complete_and_exit+0x20/0x20 [21270.587436] ret_from_fork+0x31/0x40 [21270.591016] ? kthread_complete_and_exit+0x20/0x20 [21270.595800] ret_from_fork_asm+0x11/0x20 [21270.599730] [21297.831442] watchdog: BUG: soft lockup - CPU#0 stuck for 8721s! [dbench:78683] [21297.838659] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21297.843442] watchdog: BUG: soft lockup - CPU#7 stuck for 17718s! [kworker/7:4:78808] [21297.894115] irq event stamp: 32304544 [21297.894116] hardirqs last enabled at (32304543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21297.901848] Modules linked in: [21297.905506] hardirqs last disabled at (32304544): [] sysvec_apic_timer_interrupt+0xe/0x90 [21297.915662] ceph [21297.918713] softirqs last enabled at (32304534): [] __do_softirq+0x2be/0x371 [21297.928438] libceph [21297.930362] softirqs last disabled at (32304529): [] irq_exit_rcu+0xb0/0xd0 [21297.939046] dns_resolver fscache [21297.941230] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21297.949740] netfs veth [21297.953052] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21297.961996] nft_chain_nat [21297.964439] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21297.971911] xt_MASQUERADE nf_nat [21297.974614] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21297.980091] nf_conntrack nf_defrag_ipv6 [21297.983403] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21298.002140] nf_defrag_ipv4 nft_compat [21298.006058] [21298.006059] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21298.011275] nf_tables nfnetlink [21298.015018] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21298.016509] bridge stp [21298.023635] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21298.026859] llc xfs [21298.033982] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21298.036426] libcrc32c sunrpc [21298.043551] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21298.045735] intel_rapl_msr [21298.052859] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21298.055838] intel_rapl_common [21298.062964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21298.065756] sb_edac x86_pkg_temp_thermal [21298.073842] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21298.076891] coretemp kvm_intel [21298.082631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21298.086634] kvm irqbypass [21298.093757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21298.096895] i2c_i801 [21298.104020] Call Trace: [21298.104021] [21298.106722] iTCO_wdt [21298.113848] ? watchdog_timer_fn+0x1b4/0x220 [21298.116128] iTCO_vendor_support [21298.118573] ? __hrtimer_run_queues+0x1d3/0x360 [21298.120582] i2c_smbus [21298.122856] ? hrtimer_interrupt+0x104/0x240 [21298.127119] crct10dif_pclmul [21298.130339] ? __local_bh_enable+0x50/0x60 [21298.134864] joydev [21298.137224] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21298.141485] crc32_pclmul ipmi_ssif [21298.144451] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21298.148541] ghash_clmulni_intel mei_me [21298.150638] [21298.155856] mei lpc_ich [21298.159338] [21298.164297] sha512_ssse3 acpi_ipmi [21298.168145] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21298.170242] mfd_core [21298.172778] ? queued_spin_lock_slowpath+0x16/0x2e0 [21298.174871] wmi ipmi_si [21298.178356] do_raw_spin_lock+0xb5/0xc0 [21298.183659] ipmi_devintf [21298.185930] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21298.190800] ipmi_msghandler [21298.193334] try_get_cap_refs+0x159/0x880 [ceph] [21298.197161] ioatdma acpi_pad [21298.199784] __ceph_get_caps+0x4a1/0x760 [ceph] [21298.204564] acpi_power_meter ixgbe [21298.207443] ? __file_remove_privs+0x89/0x110 [21298.212051] igb mdio [21298.215016] ? wake_bit_function+0x70/0x70 [21298.219540] crc32c_intel nvme [21298.223025] ? ceph_write_iter+0x260/0x770 [ceph] [21298.227373] ptp nvme_core [21298.229646] ceph_write_iter+0x260/0x770 [ceph] [21298.233735] i2c_algo_bit pps_core [21298.236789] ? lock_is_held_type+0xaa/0x120 [21298.241484] dca fuse [21298.244191] vfs_write+0x324/0x4f0 [21298.248711] [21298.248712] irq event stamp: 92917546 [21298.252127] ksys_pwrite64+0x65/0xa0 [21298.256303] hardirqs last enabled at (92917545): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21298.258574] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21298.261972] hardirqs last disabled at (92917546): [] sysvec_apic_timer_interrupt+0xe/0x90 [21298.263464] do_syscall_64+0x38/0x80 [21298.267120] softirqs last enabled at (92917544): [] __do_softirq+0x2be/0x371 [21298.270693] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21298.280847] softirqs last disabled at (92917539): [] irq_exit_rcu+0xb0/0xd0 [21298.285980] RIP: 0033:0x7f48f0eeb93b [21298.295704] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21298.299275] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21298.307959] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21298.313001] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21298.321514] Workqueue: events delayed_work [ceph] [21298.325083] ORIG_RAX: 0000000000000012 [21298.334462] [21298.334463] RIP: 0010:lock_release+0x0/0x260 [21298.353200] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21298.360670] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [21298.365886] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21298.370585] RSP: 0018:ffffc90002117d88 EFLAGS: 00000286 [21298.374416] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21298.375905] [21298.375906] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [21298.380170] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21298.387295] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [21298.406033] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21298.413156] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [21298.418380] [21298.425497] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [21298.471935] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [21298.479059] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21298.487137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21298.492875] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21298.499998] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21298.507127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21298.514254] Call Trace: [21298.516699] [21298.518711] ? watchdog_timer_fn+0x1b4/0x220 [21298.522974] ? __hrtimer_run_queues+0x1d3/0x360 [21298.527501] ? hrtimer_interrupt+0x104/0x240 [21298.531770] ? __local_bh_enable+0x50/0x60 [21298.535863] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21298.541080] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21298.546037] [21298.548144] [21298.550240] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21298.555547] ? igrab+0x43/0x50 [21298.558605] ? lock_downgrade+0x250/0x250 [21298.562608] ? _raw_spin_lock+0x30/0x40 [21298.566439] _raw_spin_unlock+0x1b/0x40 [21298.570269] igrab+0x43/0x50 [21298.573146] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21298.578298] delayed_work+0x1d8/0x280 [ceph] [21298.582580] ? process_scheduled_works+0x22a/0x510 [21298.587368] process_scheduled_works+0x2a4/0x510 [21298.591984] worker_thread+0x188/0x340 [21298.595732] ? rescuer_thread+0x310/0x310 [21298.599735] kthread+0xf5/0x130 [21298.602875] ? kthread_complete_and_exit+0x20/0x20 [21298.607659] ret_from_fork+0x31/0x40 [21298.611237] ? kthread_complete_and_exit+0x20/0x20 [21298.616020] ret_from_fork_asm+0x11/0x20 [21298.619945] [21325.831940] watchdog: BUG: soft lockup - CPU#0 stuck for 8747s! [dbench:78683] [21325.839156] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21325.843940] watchdog: BUG: soft lockup - CPU#7 stuck for 17744s! [kworker/7:4:78808] [21325.894616] irq event stamp: 32362808 [21325.894617] hardirqs last enabled at (32362807): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21325.902345] Modules linked in: [21325.906002] hardirqs last disabled at (32362808): [] sysvec_apic_timer_interrupt+0xe/0x90 [21325.916160] ceph [21325.919211] softirqs last enabled at (32362770): [] __do_softirq+0x2be/0x371 [21325.928935] libceph [21325.930859] softirqs last disabled at (32362765): [] irq_exit_rcu+0xb0/0xd0 [21325.939544] dns_resolver [21325.941728] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21325.950236] fscache netfs [21325.952855] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21325.961798] veth [21325.964504] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21325.971974] nft_chain_nat xt_MASQUERADE [21325.973899] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21325.979374] nf_nat [21325.983294] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21326.002030] nf_conntrack [21326.004126] [21326.004127] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21326.009345] nf_defrag_ipv6 [21326.011961] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21326.013454] nf_defrag_ipv4 nft_compat [21326.020577] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21326.023368] nf_tables nfnetlink [21326.030493] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21326.034235] bridge [21326.041360] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21326.044585] stp llc [21326.051708] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21326.053807] xfs libcrc32c [21326.060931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21326.063114] sunrpc intel_rapl_msr [21326.071191] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21326.073896] intel_rapl_common [21326.079634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21326.083030] sb_edac [21326.090153] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21326.093204] x86_pkg_temp_thermal [21326.100340] Call Trace: [21326.100342] [21326.102521] coretemp kvm_intel [21326.109647] ? watchdog_timer_fn+0x1b4/0x220 [21326.112958] kvm irqbypass [21326.115402] ? __hrtimer_run_queues+0x1d3/0x360 [21326.117412] i2c_i801 [21326.120552] ? hrtimer_interrupt+0x104/0x240 [21326.124813] iTCO_wdt iTCO_vendor_support [21326.127517] ? __local_bh_enable+0x50/0x60 [21326.132042] i2c_smbus crct10dif_pclmul [21326.134316] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21326.138593] joydev crc32_pclmul [21326.142615] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21326.146705] ipmi_ssif [21326.150535] [21326.155753] ghash_clmulni_intel [21326.158978] [21326.158980] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21326.163935] mei_me [21326.166298] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21326.168390] mei lpc_ich [21326.171622] do_raw_spin_lock+0xb5/0xc0 [21326.173719] sha512_ssse3 acpi_ipmi [21326.179028] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21326.181122] mfd_core wmi [21326.185997] try_get_cap_refs+0x159/0x880 [ceph] [21326.188522] ipmi_si [21326.192361] __ceph_get_caps+0x4a1/0x760 [ceph] [21326.195838] ipmi_devintf [21326.200626] ? __file_remove_privs+0x89/0x110 [21326.203239] ipmi_msghandler ioatdma [21326.207851] ? wake_bit_function+0x70/0x70 [21326.210035] acpi_pad acpi_power_meter [21326.214561] ? ceph_write_iter+0x260/0x770 [ceph] [21326.217176] ixgbe igb [21326.221527] ceph_write_iter+0x260/0x770 [ceph] [21326.225096] mdio [21326.229192] ? lock_is_held_type+0xaa/0x120 [21326.232932] crc32c_intel nvme [21326.237631] vfs_write+0x324/0x4f0 [21326.239986] ptp nvme_core [21326.244516] ksys_pwrite64+0x65/0xa0 [21326.246433] i2c_algo_bit [21326.250630] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21326.253679] pps_core [21326.257078] do_syscall_64+0x38/0x80 [21326.259782] dca fuse [21326.263353] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21326.265970] [21326.265971] irq event stamp: 93138208 [21326.271102] RIP: 0033:0x7f48f0eeb93b [21326.273370] hardirqs last enabled at (93138207): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21326.276942] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21326.279212] hardirqs last disabled at (93138208): [] sysvec_apic_timer_interrupt+0xe/0x90 [21326.284256] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21326.285747] softirqs last enabled at (93138206): [] __do_softirq+0x2be/0x371 [21326.289405] ORIG_RAX: 0000000000000012 [21326.292976] softirqs last disabled at (93138201): [] irq_exit_rcu+0xb0/0xd0 [21326.303134] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21326.321872] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21326.331596] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21326.336830] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21326.345515] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21326.349345] Workqueue: events delayed_work [ceph] [21326.357856] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21326.364978] [21326.364979] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [21326.374356] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21326.381480] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [21326.388956] [21326.396077] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [21326.447261] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [21326.454387] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [21326.461510] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [21326.468643] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [21326.475768] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [21326.482893] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21326.490970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21326.496708] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21326.503831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21326.510955] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21326.518078] Call Trace: [21326.520524] [21326.522535] ? watchdog_timer_fn+0x1b4/0x220 [21326.526799] ? __hrtimer_run_queues+0x1d3/0x360 [21326.531344] ? hrtimer_interrupt+0x104/0x240 [21326.535627] ? __local_bh_enable+0x50/0x60 [21326.539722] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21326.544947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21326.549904] [21326.552001] [21326.554100] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21326.559415] ? do_raw_spin_lock+0x42/0xc0 [21326.563426] igrab+0x1d/0x50 [21326.566312] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21326.571478] delayed_work+0x1d8/0x280 [ceph] [21326.575761] ? process_scheduled_works+0x22a/0x510 [21326.580551] process_scheduled_works+0x2a4/0x510 [21326.585168] worker_thread+0x188/0x340 [21326.588916] ? rescuer_thread+0x310/0x310 [21326.592925] kthread+0xf5/0x130 [21326.596065] ? kthread_complete_and_exit+0x20/0x20 [21326.600848] ret_from_fork+0x31/0x40 [21326.604426] ? kthread_complete_and_exit+0x20/0x20 [21326.609212] ret_from_fork_asm+0x11/0x20 [21326.613145] [21353.832437] watchdog: BUG: soft lockup - CPU#0 stuck for 8773s! [dbench:78683] [21353.839651] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21353.844437] watchdog: BUG: soft lockup - CPU#7 stuck for 17770s! [kworker/7:4:78808] [21353.895093] irq event stamp: 32421104 [21353.902841] Modules linked in: ceph libceph dns_resolver [21353.906499] hardirqs last enabled at (32421103): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21353.911801] fscache netfs veth nft_chain_nat [21353.921961] hardirqs last disabled at (32421104): [] sysvec_apic_timer_interrupt+0xe/0x90 [21353.926309] xt_MASQUERADE nf_nat nf_conntrack [21353.936035] softirqs last enabled at (32421066): [] __do_softirq+0x2be/0x371 [21353.936037] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [21353.949155] softirqs last disabled at (32421061): [] irq_exit_rcu+0xb0/0xd0 [21353.956540] stp llc xfs [21353.965051] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21353.967582] libcrc32c sunrpc [21353.976527] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21353.979491] intel_rapl_msr intel_rapl_common [21353.986961] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21353.986963] sb_edac x86_pkg_temp_thermal coretemp [21353.996788] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21353.996789] kvm_intel [21353.996791] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21354.001573] kvm irqbypass [21354.020310] [21354.020310] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21354.022667] i2c_i801 [21354.027887] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21354.030590] iTCO_wdt iTCO_vendor_support [21354.032079] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21354.039204] i2c_smbus crct10dif_pclmul [21354.041476] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21354.048599] joydev [21354.052602] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21354.059728] crc32_pclmul ipmi_ssif [21354.063558] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21354.070682] ghash_clmulni_intel [21354.072780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21354.079905] mei_me mei [21354.083388] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21354.091466] lpc_ich sha512_ssse3 [21354.094690] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21354.100427] acpi_ipmi mfd_core [21354.102870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21354.109996] wmi ipmi_si [21354.113305] Call Trace: [21354.113306] [21354.120431] ipmi_devintf ipmi_msghandler [21354.123568] ? watchdog_timer_fn+0x1b4/0x220 [21354.130692] ioatdma [21354.133224] ? __hrtimer_run_queues+0x1d3/0x360 [21354.135666] acpi_pad [21354.137680] ? hrtimer_interrupt+0x104/0x240 [21354.141682] acpi_power_meter ixgbe [21354.145945] ? __local_bh_enable+0x50/0x60 [21354.148130] igb mdio [21354.152656] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21354.154925] crc32c_intel nvme [21354.159189] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21354.162672] ptp nvme_core [21354.166762] [21354.169033] i2c_algo_bit pps_core [21354.174251] [21354.174253] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21354.177303] dca fuse [21354.182264] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21354.184963] [21354.187062] do_raw_spin_lock+0xb5/0xc0 [21354.190459] irq event stamp: 93359068 [21354.192557] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21354.197860] hardirqs last enabled at (93359067): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21354.200133] try_get_cap_refs+0x159/0x880 [ceph] [21354.205000] hardirqs last disabled at (93359068): [] sysvec_apic_timer_interrupt+0xe/0x90 [21354.206495] __ceph_get_caps+0x4a1/0x760 [ceph] [21354.210323] softirqs last enabled at (93359066): [] __do_softirq+0x2be/0x371 [21354.213984] ? __file_remove_privs+0x89/0x110 [21354.218764] softirqs last disabled at (93359061): [] irq_exit_rcu+0xb0/0xd0 [21354.228923] ? wake_bit_function+0x70/0x70 [21354.233532] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21354.243259] ? ceph_write_iter+0x260/0x770 [ceph] [21354.247781] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21354.256466] ceph_write_iter+0x260/0x770 [ceph] [21354.260814] Workqueue: events delayed_work [ceph] [21354.269348] ? lock_is_held_type+0xaa/0x120 [21354.273434] [21354.273435] RIP: 0010:lock_acquire+0xf0/0x2b0 [21354.282834] vfs_write+0x324/0x4f0 [21354.287528] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21354.295003] ksys_pwrite64+0x65/0xa0 [21354.299522] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21354.304220] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21354.308397] [21354.308398] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21354.309889] do_syscall_64+0x38/0x80 [21354.314237] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21354.317636] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21354.336373] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21354.339944] RIP: 0033:0x7f48f0eeb93b [21354.345162] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21354.350292] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21354.351782] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21354.358907] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21354.362478] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21354.369601] ORIG_RAX: 0000000000000012 [21354.374645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21354.381771] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21354.385340] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21354.392465] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21354.411201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21354.418326] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21354.423543] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21354.431621] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21354.435452] Call Trace: [21354.435454] [21354.441189] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21354.448315] ? watchdog_timer_fn+0x1b4/0x220 [21354.455444] [21354.462563] ? __hrtimer_run_queues+0x1d3/0x360 [21354.513614] ? hrtimer_interrupt+0x104/0x240 [21354.517883] ? __local_bh_enable+0x50/0x60 [21354.521976] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21354.527202] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21354.532158] [21354.534255] [21354.536352] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21354.541659] ? lock_acquire+0xf0/0x2b0 [21354.545411] ? igrab+0x43/0x50 [21354.548461] ? lock_release+0xc5/0x260 [21354.552204] _raw_spin_lock+0x30/0x40 [21354.555861] ? igrab+0x1d/0x50 [21354.558911] igrab+0x1d/0x50 [21354.561790] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21354.566940] delayed_work+0x1d8/0x280 [ceph] [21354.571220] ? process_scheduled_works+0x22a/0x510 [21354.576010] process_scheduled_works+0x2a4/0x510 [21354.580627] worker_thread+0x188/0x340 [21354.584377] ? rescuer_thread+0x310/0x310 [21354.588389] kthread+0xf5/0x130 [21354.591534] ? kthread_complete_and_exit+0x20/0x20 [21354.596320] ret_from_fork+0x31/0x40 [21354.599897] ? kthread_complete_and_exit+0x20/0x20 [21354.604680] ret_from_fork_asm+0x11/0x20 [21354.608605] [21381.832932] watchdog: BUG: soft lockup - CPU#0 stuck for 8799s! [dbench:78683] [21381.840148] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21381.844933] watchdog: BUG: soft lockup - CPU#7 stuck for 17796s! [kworker/7:4:78808] [21381.895605] irq event stamp: 32479242 [21381.895606] hardirqs last enabled at (32479241): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21381.903336] Modules linked in: [21381.906994] hardirqs last disabled at (32479242): [] sysvec_apic_timer_interrupt+0xe/0x90 [21381.917153] ceph [21381.920204] softirqs last enabled at (32479206): [] __do_softirq+0x2be/0x371 [21381.929926] libceph [21381.931852] softirqs last disabled at (32479201): [] irq_exit_rcu+0xb0/0xd0 [21381.940535] dns_resolver fscache [21381.942719] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21381.951231] netfs veth [21381.954540] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21381.963485] nft_chain_nat xt_MASQUERADE [21381.965930] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21381.973400] nf_nat nf_conntrack [21381.977327] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21381.982804] nf_defrag_ipv6 nf_defrag_ipv4 [21381.986027] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21382.004766] nft_compat nf_tables [21382.008856] [21382.008857] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21382.014074] nfnetlink bridge [21382.017384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21382.018875] stp [21382.026000] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21382.028962] llc xfs [21382.036087] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21382.037924] libcrc32c sunrpc [21382.045048] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21382.047233] intel_rapl_msr intel_rapl_common [21382.054357] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21382.057331] sb_edac x86_pkg_temp_thermal [21382.064453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21382.068803] coretemp kvm_intel [21382.076882] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21382.080886] kvm irqbypass [21382.086622] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21382.089761] i2c_i801 iTCO_wdt [21382.096885] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21382.099589] iTCO_vendor_support [21382.106712] Call Trace: [21382.106714] [21382.109764] i2c_smbus crct10dif_pclmul [21382.116888] ? watchdog_timer_fn+0x1b4/0x220 [21382.120111] joydev crc32_pclmul [21382.122557] ? __hrtimer_run_queues+0x1d3/0x360 [21382.124567] ipmi_ssif ghash_clmulni_intel [21382.128400] ? hrtimer_interrupt+0x104/0x240 [21382.132662] mei_me mei [21382.135886] ? __local_bh_enable+0x50/0x60 [21382.140410] lpc_ich sha512_ssse3 [21382.144503] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21382.148764] acpi_ipmi mfd_core [21382.151209] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21382.155299] wmi ipmi_si [21382.158612] [21382.163828] ipmi_devintf [21382.166964] [21382.166965] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21382.171923] ipmi_msghandler ioatdma [21382.174457] ? queued_spin_lock_slowpath+0x16/0x2e0 [21382.176549] acpi_pad acpi_power_meter [21382.179169] do_raw_spin_lock+0xb5/0xc0 [21382.181266] ixgbe [21382.186569] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21382.190141] igb mdio [21382.195014] try_get_cap_refs+0x159/0x880 [ceph] [21382.198755] crc32c_intel [21382.202590] __ceph_get_caps+0x4a1/0x760 [ceph] [21382.204597] nvme ptp [21382.209384] ? __file_remove_privs+0x89/0x110 [21382.211651] nvme_core i2c_algo_bit [21382.216262] ? wake_bit_function+0x70/0x70 [21382.218879] pps_core dca [21382.223405] ? ceph_write_iter+0x260/0x770 [ceph] [21382.225675] fuse [21382.230025] ceph_write_iter+0x260/0x770 [ceph] [21382.233508] irq event stamp: 93580050 [21382.237617] ? lock_is_held_type+0xaa/0x120 [21382.240233] hardirqs last enabled at (93580049): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21382.244935] vfs_write+0x324/0x4f0 [21382.246854] hardirqs last disabled at (93580050): [] sysvec_apic_timer_interrupt+0xe/0x90 [21382.251386] ksys_pwrite64+0x65/0xa0 [21382.255038] softirqs last enabled at (93580048): [] __do_softirq+0x2be/0x371 [21382.259215] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21382.269372] softirqs last disabled at (93580043): [] irq_exit_rcu+0xb0/0xd0 [21382.272771] do_syscall_64+0x38/0x80 [21382.282495] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21382.286066] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21382.294750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21382.299879] RIP: 0033:0x7f48f0eeb93b [21382.308390] Workqueue: events delayed_work [ceph] [21382.311962] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21382.321338] [21382.326383] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21382.333853] RIP: 0010:lock_acquire+0xf0/0x2b0 [21382.337425] ORIG_RAX: 0000000000000012 [21382.342121] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21382.360860] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21382.362350] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21382.367567] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21382.371918] [21382.371918] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21382.375749] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21382.394487] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21382.401615] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21382.406837] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21382.413961] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21382.415451] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21382.422582] [21382.429701] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21382.474646] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21382.482725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21382.488461] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21382.495584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21382.502708] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21382.509834] Call Trace: [21382.512279] [21382.514290] ? watchdog_timer_fn+0x1b4/0x220 [21382.518552] ? __hrtimer_run_queues+0x1d3/0x360 [21382.523079] ? hrtimer_interrupt+0x104/0x240 [21382.527341] ? __local_bh_enable+0x50/0x60 [21382.531434] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21382.536658] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21382.541618] [21382.543713] [21382.545811] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21382.551118] ? lock_acquire+0xf0/0x2b0 [21382.554869] ? igrab+0x43/0x50 [21382.557918] ? lock_release+0xc5/0x260 [21382.561662] _raw_spin_lock+0x30/0x40 [21382.565338] ? igrab+0x1d/0x50 [21382.568387] igrab+0x1d/0x50 [21382.571265] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21382.576414] delayed_work+0x1d8/0x280 [ceph] [21382.580697] ? process_scheduled_works+0x22a/0x510 [21382.585485] process_scheduled_works+0x2a4/0x510 [21382.590101] worker_thread+0x188/0x340 [21382.593852] ? rescuer_thread+0x310/0x310 [21382.597854] kthread+0xf5/0x130 [21382.600991] ? kthread_complete_and_exit+0x20/0x20 [21382.605777] ret_from_fork+0x31/0x40 [21382.609346] ? kthread_complete_and_exit+0x20/0x20 [21382.614132] ret_from_fork_asm+0x11/0x20 [21382.618054] [21383.146322] rcu: INFO: rcu_preempt self-detected stall on CPU [21383.152081] rcu: 0-....: (9233243 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=4605437 [21383.162420] rcu: (t=9446367 jiffies g=1309609 q=324286 ncpus=8) [21383.168419] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21383.177361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21383.184832] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [21383.190311] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [21383.209048] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21383.214265] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21383.221387] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21383.228514] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21383.235637] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21383.242761] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21383.249884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21383.257961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21383.263701] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21383.270825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21383.277949] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21383.285073] Call Trace: [21383.287517] [21383.289528] ? rcu_dump_cpu_stacks+0xd1/0x180 [21383.293888] ? rcu_sched_clock_irq+0x633/0x1390 [21383.298420] ? lock_is_held_type+0xaa/0x120 [21383.302599] ? lock_is_held_type+0xaa/0x120 [21383.306776] ? tick_sched_do_timer+0xa0/0xa0 [21383.311046] ? rcu_read_lock_sched_held+0x56/0x90 [21383.315745] ? tick_sched_do_timer+0xa0/0xa0 [21383.320008] ? update_process_times+0x5f/0x90 [21383.324360] ? tick_sched_handle.isra.16+0x1e/0x50 [21383.329144] ? tick_sched_timer+0x6f/0x80 [21383.333149] ? __hrtimer_run_queues+0x1d3/0x360 [21383.337674] ? hrtimer_interrupt+0x104/0x240 [21383.341936] ? __local_bh_enable+0x50/0x60 [21383.346030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21383.351253] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21383.356209] [21383.358307] [21383.360404] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21383.365713] ? queued_spin_lock_slowpath+0x2b/0x2e0 [21383.370591] do_raw_spin_lock+0xb5/0xc0 [21383.374427] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21383.379232] try_get_cap_refs+0x159/0x880 [ceph] [21383.383861] __ceph_get_caps+0x4a1/0x760 [ceph] [21383.388403] ? __file_remove_privs+0x89/0x110 [21383.392759] ? wake_bit_function+0x70/0x70 [21383.396851] ? ceph_write_iter+0x260/0x770 [ceph] [21383.401562] ceph_write_iter+0x260/0x770 [ceph] [21383.406100] ? lock_is_held_type+0xaa/0x120 [21383.410285] vfs_write+0x324/0x4f0 [21383.413693] ksys_pwrite64+0x65/0xa0 [21383.417268] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21383.422399] do_syscall_64+0x38/0x80 [21383.425970] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21383.431014] RIP: 0033:0x7f48f0eeb93b [21383.434584] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21383.453322] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [21383.460880] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21383.468002] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21383.475126] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21383.482251] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21383.489375] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21383.496504] [21383.498692] Sending NMI from CPU 0 to CPUs 7: [21383.503046] NMI backtrace for cpu 7 [21383.503048] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21383.503050] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21383.503051] Workqueue: events delayed_work [ceph] [21383.503071] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [21383.503089] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [21383.503090] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [21383.503092] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [21383.503094] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [21383.503095] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [21383.503096] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [21383.503097] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [21383.503099] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21383.503100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21383.503102] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21383.503103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21383.503104] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21383.503105] Call Trace: [21383.503106] [21383.503108] ? nmi_cpu_backtrace+0xad/0x120 [21383.503112] ? nmi_cpu_backtrace_handler+0x11/0x20 [21383.503115] ? nmi_handle+0xd4/0x260 [21383.503121] ? default_do_nmi+0x49/0xf0 [21383.503123] ? exc_nmi+0x11e/0x150 [21383.503126] ? end_repeat_nmi+0x16/0x67 [21383.503132] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [21383.503149] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [21383.503166] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [21383.503181] [21383.503182] [21383.503184] delayed_work+0x1d8/0x280 [ceph] [21383.503204] ? process_scheduled_works+0x22a/0x510 [21383.503207] process_scheduled_works+0x2a4/0x510 [21383.503214] worker_thread+0x188/0x340 [21383.503218] ? rescuer_thread+0x310/0x310 [21383.503220] kthread+0xf5/0x130 [21383.503223] ? kthread_complete_and_exit+0x20/0x20 [21383.503227] ret_from_fork+0x31/0x40 [21383.503229] ? kthread_complete_and_exit+0x20/0x20 [21383.503232] ret_from_fork_asm+0x11/0x20 [21383.503240] [21409.833428] watchdog: BUG: soft lockup - CPU#0 stuck for 8825s! [dbench:78683] [21409.840645] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21409.845428] watchdog: BUG: soft lockup - CPU#7 stuck for 17822s! [kworker/7:4:78808] [21409.896103] irq event stamp: 32536740 [21409.903834] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21409.907492] hardirqs last enabled at (32536739): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21409.914451] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [21409.924607] hardirqs last disabled at (32536740): [] sysvec_apic_timer_interrupt+0xe/0x90 [21409.924610] nf_defrag_ipv4 [21409.924612] softirqs last enabled at (32536634): [] __do_softirq+0x2be/0x371 [21409.931560] nft_compat [21409.941283] softirqs last disabled at (32536629): [] irq_exit_rcu+0xb0/0xd0 [21409.944074] nf_tables nfnetlink [21409.952759] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21409.955203] bridge stp [21409.963713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21409.966938] llc xfs [21409.975881] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21409.978324] libcrc32c [21409.985795] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21409.987979] sunrpc intel_rapl_msr [21409.993457] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21409.995822] intel_rapl_common [21410.014562] [21410.017959] sb_edac x86_pkg_temp_thermal [21410.023175] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21410.026226] coretemp kvm_intel [21410.027718] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21410.031722] kvm irqbypass [21410.038846] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21410.041984] i2c_i801 iTCO_wdt [21410.049112] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21410.051824] iTCO_vendor_support [21410.058943] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21410.061996] i2c_smbus crct10dif_pclmul [21410.069120] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21410.072344] joydev crc32_pclmul [21410.079468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21410.083299] ipmi_ssif [21410.091374] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21410.094599] ghash_clmulni_intel mei_me [21410.100338] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21410.102696] mei lpc_ich [21410.109826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21410.113658] sha512_ssse3 acpi_ipmi [21410.120782] Call Trace: [21410.120784] [21410.123312] mfd_core [21410.130440] ? watchdog_timer_fn+0x1b4/0x220 [21410.133920] wmi ipmi_si [21410.136367] ? __hrtimer_run_queues+0x1d3/0x360 [21410.138374] ipmi_devintf ipmi_msghandler [21410.140649] ? hrtimer_interrupt+0x104/0x240 [21410.144910] ioatdma acpi_pad [21410.147442] ? __local_bh_enable+0x50/0x60 [21410.151965] acpi_power_meter ixgbe [21410.155971] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21410.160234] igb mdio [21410.163199] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21410.167289] crc32c_intel [21410.170773] [21410.175989] nvme ptp [21410.178259] [21410.183218] nvme_core [21410.185836] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21410.187931] i2c_algo_bit pps_core [21410.190207] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21410.192301] dca fuse [21410.194660] do_raw_spin_lock+0xb5/0xc0 [21410.199962] [21410.199963] irq event stamp: 93799234 [21410.203360] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21410.208231] hardirqs last enabled at (93799233): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21410.210505] try_get_cap_refs+0x159/0x880 [ceph] [21410.214331] hardirqs last disabled at (93799234): [] sysvec_apic_timer_interrupt+0xe/0x90 [21410.215833] __ceph_get_caps+0x4a1/0x760 [ceph] [21410.219488] softirqs last enabled at (93799232): [] __do_softirq+0x2be/0x371 [21410.224278] ? __file_remove_privs+0x89/0x110 [21410.234431] softirqs last disabled at (93799227): [] irq_exit_rcu+0xb0/0xd0 [21410.239041] ? wake_bit_function+0x70/0x70 [21410.248764] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21410.253291] ? ceph_write_iter+0x260/0x770 [ceph] [21410.261974] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21410.266323] ceph_write_iter+0x260/0x770 [ceph] [21410.274835] Workqueue: events delayed_work [ceph] [21410.278930] ? lock_is_held_type+0xaa/0x120 [21410.288302] [21410.293004] vfs_write+0x324/0x4f0 [21410.300470] RIP: 0010:lock_acquire+0xf0/0x2b0 [21410.304999] ksys_pwrite64+0x65/0xa0 [21410.309692] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21410.313872] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21410.315360] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21410.318760] do_syscall_64+0x38/0x80 [21410.323109] [21410.326683] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21410.345418] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21410.350550] RIP: 0033:0x7f48f0eeb93b [21410.355765] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21410.359337] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21410.360830] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21410.365870] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21410.372996] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21410.376567] ORIG_RAX: 0000000000000012 [21410.383690] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21410.402428] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21410.409551] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21410.414770] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21410.421894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21410.425724] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21410.432848] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21410.439973] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21410.448051] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21410.455174] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21410.460911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21410.468041] [21410.475161] Call Trace: [21410.475163] [21410.510298] ? watchdog_timer_fn+0x1b4/0x220 [21410.514561] ? __hrtimer_run_queues+0x1d3/0x360 [21410.519086] ? hrtimer_interrupt+0x104/0x240 [21410.523347] ? __local_bh_enable+0x50/0x60 [21410.527442] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21410.532666] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21410.537622] [21410.539719] [21410.541834] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21410.547143] ? lock_acquire+0xf0/0x2b0 [21410.550893] ? igrab+0x43/0x50 [21410.553942] ? lock_release+0xc5/0x260 [21410.557687] _raw_spin_lock+0x30/0x40 [21410.561345] ? igrab+0x1d/0x50 [21410.564394] igrab+0x1d/0x50 [21410.567274] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21410.572423] delayed_work+0x1d8/0x280 [ceph] [21410.576702] ? process_scheduled_works+0x22a/0x510 [21410.581484] process_scheduled_works+0x2a4/0x510 [21410.586117] worker_thread+0x188/0x340 [21410.589867] ? rescuer_thread+0x310/0x310 [21410.593869] kthread+0xf5/0x130 [21410.597008] ? kthread_complete_and_exit+0x20/0x20 [21410.601792] ret_from_fork+0x31/0x40 [21410.605371] ? kthread_complete_and_exit+0x20/0x20 [21410.610155] ret_from_fork_asm+0x11/0x20 [21410.614081] [21437.833923] watchdog: BUG: soft lockup - CPU#0 stuck for 8851s! [dbench:78683] [21437.841140] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21437.845923] watchdog: BUG: soft lockup - CPU#7 stuck for 17848s! [kworker/7:4:78808] [21437.896597] irq event stamp: 32594876 [21437.904330] Modules linked in: ceph libceph [21437.907989] hardirqs last enabled at (32594875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21437.912165] dns_resolver fscache netfs [21437.922323] hardirqs last disabled at (32594876): [] sysvec_apic_timer_interrupt+0xe/0x90 [21437.926154] veth nft_chain_nat xt_MASQUERADE nf_nat [21437.935877] softirqs last enabled at (32594860): [] __do_softirq+0x2be/0x371 [21437.940836] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [21437.949520] softirqs last disabled at (32594855): [] irq_exit_rcu+0xb0/0xd0 [21437.955690] nf_tables nfnetlink bridge [21437.964202] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21437.968032] stp llc [21437.976976] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21437.976978] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [21437.979160] xfs [21437.986631] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [21437.992109] libcrc32c sunrpc [21437.993945] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21438.012682] intel_rapl_msr intel_rapl_common [21438.015649] [21438.015649] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21438.020865] sb_edac x86_pkg_temp_thermal [21438.025215] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21438.026708] coretemp kvm_intel [21438.033830] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21438.037835] kvm irqbypass [21438.044958] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21438.048097] i2c_i801 iTCO_wdt [21438.055220] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21438.057925] iTCO_vendor_support i2c_smbus [21438.065049] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21438.068100] crct10dif_pclmul joydev [21438.075222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21438.079320] crc32_pclmul [21438.087399] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21438.090970] ipmi_ssif ghash_clmulni_intel [21438.096707] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21438.099326] mei_me mei [21438.106449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21438.110540] lpc_ich sha512_ssse3 [21438.117666] Call Trace: [21438.120109] acpi_ipmi mfd_core [21438.127232] [21438.130543] wmi ipmi_si [21438.132989] ? watchdog_timer_fn+0x1b4/0x220 [21438.136126] ipmi_devintf ipmi_msghandler [21438.138137] ? __hrtimer_run_queues+0x1d3/0x360 [21438.140666] ioatdma acpi_pad [21438.144933] ? hrtimer_interrupt+0x104/0x240 [21438.148935] acpi_power_meter ixgbe [21438.153458] ? __local_bh_enable+0x50/0x60 [21438.156423] igb mdio [21438.160690] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21438.164170] crc32c_intel nvme [21438.168261] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21438.170532] ptp nvme_core [21438.175751] [21438.178799] i2c_algo_bit pps_core [21438.183758] [21438.183759] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21438.186462] dca fuse [21438.188563] ? queued_spin_lock_slowpath+0x2b/0x2e0 [21438.191957] [21438.191958] irq event stamp: 94020158 [21438.194055] do_raw_spin_lock+0xb5/0xc0 [21438.199358] hardirqs last enabled at (94020157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21438.201631] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21438.206501] hardirqs last disabled at (94020158): [] sysvec_apic_timer_interrupt+0xe/0x90 [21438.207993] try_get_cap_refs+0x159/0x880 [ceph] [21438.211647] softirqs last enabled at (94020156): [] __do_softirq+0x2be/0x371 [21438.215483] __ceph_get_caps+0x4a1/0x760 [ceph] [21438.225636] softirqs last disabled at (94020151): [] irq_exit_rcu+0xb0/0xd0 [21438.230425] ? __file_remove_privs+0x89/0x110 [21438.240144] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21438.244756] ? wake_bit_function+0x70/0x70 [21438.253438] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21438.253440] Workqueue: events delayed_work [ceph] [21438.257967] ? ceph_write_iter+0x260/0x770 [ceph] [21438.266475] [21438.266476] RIP: 0010:rcu_is_watching+0x22/0x40 [21438.270825] ceph_write_iter+0x260/0x770 [ceph] [21438.280202] Code: 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 01 <65> ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 [21438.284300] ? lock_is_held_type+0xaa/0x120 [21438.291763] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [21438.296465] vfs_write+0x324/0x4f0 [21438.301158] [21438.301159] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 0000000000000000 [21438.302654] ksys_pwrite64+0x65/0xa0 [21438.307174] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [21438.311699] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21438.330435] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [21438.334617] do_syscall_64+0x38/0x80 [21438.339831] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [21438.343228] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21438.344719] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21438.351843] RIP: 0033:0x7f48f0eeb93b [21438.355413] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21438.362538] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21438.367668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21438.374792] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21438.378362] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21438.385486] ORIG_RAX: 0000000000000012 [21438.390531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21438.397656] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21438.401225] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21438.409302] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21438.428058] Call Trace: [21438.428059] [21438.433795] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21438.439013] ? watchdog_timer_fn+0x1b4/0x220 [21438.446139] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21438.449970] ? __hrtimer_run_queues+0x1d3/0x360 [21438.457092] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21438.464221] ? hrtimer_interrupt+0x104/0x240 [21438.471345] [21438.478465] ? __local_bh_enable+0x50/0x60 [21438.523640] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21438.528862] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21438.533820] [21438.535917] [21438.538015] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21438.543334] ? rcu_is_watching+0x22/0x40 [21438.547256] lock_acquire+0x184/0x2b0 [21438.550912] ? igrab+0x43/0x50 [21438.553961] ? lock_release+0xc5/0x260 [21438.557708] _raw_spin_lock+0x30/0x40 [21438.561373] ? igrab+0x1d/0x50 [21438.564422] igrab+0x1d/0x50 [21438.567301] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21438.572469] delayed_work+0x1d8/0x280 [ceph] [21438.576749] ? process_scheduled_works+0x22a/0x510 [21438.581538] process_scheduled_works+0x2a4/0x510 [21438.586155] worker_thread+0x188/0x340 [21438.589905] ? rescuer_thread+0x310/0x310 [21438.593906] kthread+0xf5/0x130 [21438.597045] ? kthread_complete_and_exit+0x20/0x20 [21438.601831] ret_from_fork+0x31/0x40 [21438.605407] ? kthread_complete_and_exit+0x20/0x20 [21438.610192] ret_from_fork_asm+0x11/0x20 [21438.614115] [21465.834418] watchdog: BUG: soft lockup - CPU#0 stuck for 8877s! [dbench:78683] [21465.841640] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21465.846418] watchdog: BUG: soft lockup - CPU#7 stuck for 17874s! [kworker/7:4:78808] [21465.897091] irq event stamp: 32653046 [21465.904818] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21465.908476] hardirqs last enabled at (32653045): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21465.908482] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [21465.925592] hardirqs last disabled at (32653046): [] sysvec_apic_timer_interrupt+0xe/0x90 [21465.932543] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [21465.942268] softirqs last enabled at (32653030): [] __do_softirq+0x2be/0x371 [21465.942272] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [21465.957046] softirqs last disabled at (32653025): [] irq_exit_rcu+0xb0/0xd0 [21465.963918] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [21465.972431] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21465.978426] irqbypass i2c_i801 iTCO_wdt [21465.987370] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21465.987371] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21465.991288] iTCO_vendor_support [21465.998758] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21466.004235] i2c_smbus crct10dif_pclmul [21466.007459] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21466.026197] joydev crc32_pclmul [21466.030030] [21466.035247] ipmi_ssif ghash_clmulni_intel [21466.038470] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21466.039961] mei_me mei [21466.044052] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21466.051175] lpc_ich sha512_ssse3 [21466.053619] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21466.060744] acpi_ipmi mfd_core [21466.064055] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21466.071178] wmi ipmi_si [21466.074316] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21466.081439] ipmi_devintf ipmi_msghandler [21466.083971] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21466.091110] ioatdma acpi_pad [21466.095116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21466.103193] acpi_power_meter [21466.106159] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21466.111895] ixgbe [21466.114859] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21466.121983] igb mdio [21466.123993] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21466.131117] crc32c_intel nvme [21466.133389] Call Trace: [21466.133392] [21466.140512] ptp nvme_core [21466.143566] ? watchdog_timer_fn+0x1b4/0x220 [21466.146008] i2c_algo_bit pps_core [21466.148020] ? __hrtimer_run_queues+0x1d3/0x360 [21466.150721] dca fuse [21466.154990] ? hrtimer_interrupt+0x104/0x240 [21466.158384] [21466.158385] irq event stamp: 94241030 [21466.162907] ? __local_bh_enable+0x50/0x60 [21466.165178] hardirqs last enabled at (94241029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21466.169446] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21466.170933] hardirqs last disabled at (94241030): [] sysvec_apic_timer_interrupt+0xe/0x90 [21466.174592] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21466.178682] softirqs last enabled at (94241028): [] __do_softirq+0x2be/0x371 [21466.188839] [21466.194057] softirqs last disabled at (94241023): [] irq_exit_rcu+0xb0/0xd0 [21466.203780] [21466.208738] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21466.217422] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21466.219520] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21466.228034] ? queued_spin_lock_slowpath+0x16/0x2e0 [21466.230128] Workqueue: events delayed_work [ceph] [21466.239508] do_raw_spin_lock+0xb5/0xc0 [21466.244817] [21466.244818] RIP: 0010:lock_acquire+0xf0/0x2b0 [21466.252290] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21466.257159] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21466.261859] try_get_cap_refs+0x159/0x880 [ceph] [21466.265687] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21466.267184] __ceph_get_caps+0x4a1/0x760 [ceph] [21466.271530] [21466.271530] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21466.276319] ? __file_remove_privs+0x89/0x110 [21466.295051] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21466.299662] ? wake_bit_function+0x70/0x70 [21466.304880] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21466.309405] ? ceph_write_iter+0x260/0x770 [ceph] [21466.310893] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21466.318019] ceph_write_iter+0x260/0x770 [ceph] [21466.322368] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21466.329497] ? lock_is_held_type+0xaa/0x120 [21466.333583] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21466.340709] vfs_write+0x324/0x4f0 [21466.345406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21466.352533] ksys_pwrite64+0x65/0xa0 [21466.357053] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21466.364178] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21466.368354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21466.376434] do_syscall_64+0x38/0x80 [21466.379829] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21466.385568] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21466.389137] Call Trace: [21466.389139] [21466.396263] RIP: 0033:0x7f48f0eeb93b [21466.401393] ? watchdog_timer_fn+0x1b4/0x220 [21466.408517] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21466.412101] ? __hrtimer_run_queues+0x1d3/0x360 [21466.419221] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21466.424269] ? hrtimer_interrupt+0x104/0x240 [21466.426708] ORIG_RAX: 0000000000000012 [21466.428719] ? __local_bh_enable+0x50/0x60 [21466.432291] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21466.436557] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21466.455291] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21466.459825] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21466.465041] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21466.469305] [21466.473138] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21466.477228] [21466.477230] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21466.484353] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21466.489575] ? lock_acquire+0xf0/0x2b0 [21466.496698] [21466.501652] ? igrab+0x43/0x50 [21466.541510] ? lock_release+0xc5/0x260 [21466.545257] _raw_spin_lock+0x30/0x40 [21466.548920] ? igrab+0x1d/0x50 [21466.551970] igrab+0x1d/0x50 [21466.554850] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21466.560000] delayed_work+0x1d8/0x280 [ceph] [21466.564280] ? process_scheduled_works+0x22a/0x510 [21466.569071] process_scheduled_works+0x2a4/0x510 [21466.573686] worker_thread+0x188/0x340 [21466.577437] ? rescuer_thread+0x310/0x310 [21466.581446] kthread+0xf5/0x130 [21466.584585] ? kthread_complete_and_exit+0x20/0x20 [21466.589370] ret_from_fork+0x31/0x40 [21466.592949] ? kthread_complete_and_exit+0x20/0x20 [21466.597733] ret_from_fork_asm+0x11/0x20 [21466.601663] [21493.834915] watchdog: BUG: soft lockup - CPU#0 stuck for 8903s! [dbench:78683] [21493.842125] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21493.846915] watchdog: BUG: soft lockup - CPU#7 stuck for 17900s! [kworker/7:4:78808] [21493.897566] irq event stamp: 32711118 [21493.897567] hardirqs last enabled at (32711117): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21493.905315] Modules linked in: [21493.908971] hardirqs last disabled at (32711118): [] sysvec_apic_timer_interrupt+0xe/0x90 [21493.919129] ceph [21493.922180] softirqs last enabled at (32711060): [] __do_softirq+0x2be/0x371 [21493.931904] libceph [21493.933830] softirqs last disabled at (32711055): [] irq_exit_rcu+0xb0/0xd0 [21493.942512] dns_resolver fscache [21493.944697] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21493.953208] netfs veth [21493.956517] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21493.965461] nft_chain_nat [21493.967905] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21493.975376] xt_MASQUERADE nf_nat [21493.978082] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21493.983557] nf_conntrack nf_defrag_ipv6 [21493.986869] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21494.005606] nf_defrag_ipv4 nft_compat [21494.009524] [21494.014742] nf_tables nfnetlink [21494.018486] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21494.019977] bridge stp [21494.023201] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21494.030324] llc xfs [21494.032768] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21494.039893] libcrc32c [21494.042076] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21494.049202] sunrpc intel_rapl_msr [21494.051559] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21494.058683] intel_rapl_common sb_edac [21494.062079] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21494.069205] x86_pkg_temp_thermal coretemp [21494.072948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21494.081026] kvm_intel kvm [21494.085116] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21494.090854] irqbypass i2c_i801 [21494.093557] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21494.100682] iTCO_wdt iTCO_vendor_support [21494.103820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21494.110943] i2c_smbus crct10dif_pclmul [21494.114947] Call Trace: [21494.114948] [21494.122072] joydev [21494.125904] ? watchdog_timer_fn+0x1b4/0x220 [21494.128348] crc32_pclmul [21494.130360] ? __hrtimer_run_queues+0x1d3/0x360 [21494.132454] ipmi_ssif ghash_clmulni_intel [21494.136721] ? hrtimer_interrupt+0x104/0x240 [21494.139335] mei_me mei [21494.143861] ? __local_bh_enable+0x50/0x60 [21494.147952] lpc_ich [21494.152217] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21494.154658] sha512_ssse3 [21494.158751] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21494.160933] acpi_ipmi mfd_core [21494.166152] [21494.168770] wmi ipmi_si [21494.173726] [21494.176863] ipmi_devintf [21494.178961] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21494.181490] ipmi_msghandler [21494.183599] ? queued_spin_lock_slowpath+0x18/0x2e0 [21494.186216] ioatdma acpi_pad [21494.191521] do_raw_spin_lock+0xb5/0xc0 [21494.194395] acpi_power_meter ixgbe [21494.199268] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21494.202231] igb [21494.206064] try_get_cap_refs+0x159/0x880 [ceph] [21494.209547] mdio crc32c_intel [21494.214335] __ceph_get_caps+0x4a1/0x760 [ceph] [21494.216169] nvme ptp [21494.220782] ? __file_remove_privs+0x89/0x110 [21494.223829] nvme_core i2c_algo_bit [21494.228354] ? wake_bit_function+0x70/0x70 [21494.230625] pps_core dca [21494.234977] ? ceph_write_iter+0x260/0x770 [ceph] [21494.238458] fuse [21494.242549] ceph_write_iter+0x260/0x770 [ceph] [21494.245168] irq event stamp: 94462096 [21494.249868] ? lock_is_held_type+0xaa/0x120 [21494.251788] hardirqs last enabled at (94462095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21494.256319] vfs_write+0x324/0x4f0 [21494.259978] hardirqs last disabled at (94462096): [] sysvec_apic_timer_interrupt+0xe/0x90 [21494.264159] ksys_pwrite64+0x65/0xa0 [21494.274315] softirqs last enabled at (94462094): [] __do_softirq+0x2be/0x371 [21494.277710] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21494.287435] softirqs last disabled at (94462089): [] irq_exit_rcu+0xb0/0xd0 [21494.291007] do_syscall_64+0x38/0x80 [21494.299691] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21494.304823] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21494.313330] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21494.316902] RIP: 0033:0x7f48f0eeb93b [21494.326280] Workqueue: events delayed_work [ceph] [21494.331323] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21494.338794] [21494.342364] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21494.347062] RIP: 0010:lock_release+0x77/0x260 [21494.365800] ORIG_RAX: 0000000000000012 [21494.367292] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [21494.372526] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21494.376876] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [21494.380708] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21494.399446] [21494.399447] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [21494.406570] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21494.411786] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [21494.418912] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21494.420401] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [21494.427526] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21494.434650] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [21494.441780] [21494.448899] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [21494.479579] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21494.487673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21494.493412] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21494.500534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21494.507660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21494.514784] Call Trace: [21494.517229] [21494.519240] ? watchdog_timer_fn+0x1b4/0x220 [21494.523503] ? __hrtimer_run_queues+0x1d3/0x360 [21494.528029] ? hrtimer_interrupt+0x104/0x240 [21494.532316] ? __local_bh_enable+0x50/0x60 [21494.536410] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21494.541633] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21494.546607] [21494.548707] [21494.550805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21494.556109] ? igrab+0x43/0x50 [21494.559160] ? lock_release+0x77/0x260 [21494.562906] _raw_spin_unlock+0x1b/0x40 [21494.566742] igrab+0x43/0x50 [21494.569620] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21494.574770] delayed_work+0x1d8/0x280 [ceph] [21494.579051] ? process_scheduled_works+0x22a/0x510 [21494.583841] process_scheduled_works+0x2a4/0x510 [21494.588456] worker_thread+0x188/0x340 [21494.592204] ? rescuer_thread+0x310/0x310 [21494.596209] kthread+0xf5/0x130 [21494.599345] ? kthread_complete_and_exit+0x20/0x20 [21494.604130] ret_from_fork+0x31/0x40 [21494.607702] ? kthread_complete_and_exit+0x20/0x20 [21494.612486] ret_from_fork_asm+0x11/0x20 [21494.616408] [21521.835413] watchdog: BUG: soft lockup - CPU#0 stuck for 8929s! [dbench:78683] [21521.842631] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21521.847413] watchdog: BUG: soft lockup - CPU#7 stuck for 17926s! [kworker/7:4:78808] [21521.898095] irq event stamp: 32769346 [21521.898096] hardirqs last enabled at (32769345): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21521.905828] Modules linked in: [21521.909485] hardirqs last disabled at (32769346): [] sysvec_apic_timer_interrupt+0xe/0x90 [21521.919642] ceph [21521.922694] softirqs last enabled at (32769288): [] __do_softirq+0x2be/0x371 [21521.932419] libceph [21521.934342] softirqs last disabled at (32769283): [] irq_exit_rcu+0xb0/0xd0 [21521.943027] dns_resolver fscache [21521.945212] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21521.953721] netfs veth [21521.957033] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21521.965976] nft_chain_nat xt_MASQUERADE [21521.968421] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21521.975891] nf_nat nf_conntrack [21521.979810] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21521.985286] nf_defrag_ipv6 nf_defrag_ipv4 [21521.988510] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21522.007246] nft_compat nf_tables [21522.011338] [21522.016555] nfnetlink [21522.019865] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21522.021356] bridge stp [21522.023714] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21522.030838] llc xfs [21522.033282] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21522.040406] libcrc32c sunrpc [21522.042589] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21522.049715] intel_rapl_msr [21522.052679] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21522.059811] intel_rapl_common sb_edac [21522.062603] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21522.069726] x86_pkg_temp_thermal [21522.073469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21522.081548] coretemp kvm_intel [21522.084859] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21522.090595] kvm irqbypass [21522.093734] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21522.100858] i2c_i801 iTCO_wdt [21522.103562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21522.110685] iTCO_vendor_support i2c_smbus crct10dif_pclmul [21522.113736] Call Trace: [21522.120861] joydev crc32_pclmul [21522.126427] [21522.128868] ipmi_ssif ghash_clmulni_intel [21522.132108] ? watchdog_timer_fn+0x1b4/0x220 [21522.134120] mei_me mei [21522.138213] ? __hrtimer_run_queues+0x1d3/0x360 [21522.142475] lpc_ich [21522.144923] ? hrtimer_interrupt+0x104/0x240 [21522.149442] sha512_ssse3 acpi_ipmi [21522.151626] ? __local_bh_enable+0x50/0x60 [21522.155892] mfd_core [21522.159379] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21522.163467] wmi ipmi_si [21522.165739] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21522.170955] ipmi_devintf [21522.173486] [21522.178441] ipmi_msghandler ioatdma [21522.181060] [21522.181061] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21522.183157] acpi_pad acpi_power_meter [21522.186732] ? queued_spin_lock_slowpath+0x18/0x2e0 [21522.188827] ixgbe igb [21522.194133] do_raw_spin_lock+0xb5/0xc0 [21522.197875] mdio crc32c_intel [21522.202747] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21522.205102] nvme ptp [21522.208936] try_get_cap_refs+0x159/0x880 [ceph] [21522.211982] nvme_core i2c_algo_bit [21522.216772] __ceph_get_caps+0x4a1/0x760 [ceph] [21522.219038] pps_core [21522.223654] ? __file_remove_privs+0x89/0x110 [21522.227133] dca fuse [21522.231658] ? wake_bit_function+0x70/0x70 [21522.233929] [21522.233929] irq event stamp: 94683040 [21522.238281] ? ceph_write_iter+0x260/0x770 [ceph] [21522.240549] hardirqs last enabled at (94683039): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21522.244640] ceph_write_iter+0x260/0x770 [ceph] [21522.246131] hardirqs last disabled at (94683040): [] sysvec_apic_timer_interrupt+0xe/0x90 [21522.249811] ? lock_is_held_type+0xaa/0x120 [21522.254503] softirqs last enabled at (94683038): [] __do_softirq+0x2be/0x371 [21522.264664] vfs_write+0x324/0x4f0 [21522.269185] softirqs last disabled at (94683033): [] irq_exit_rcu+0xb0/0xd0 [21522.278914] ksys_pwrite64+0x65/0xa0 [21522.283094] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21522.291781] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21522.295175] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21522.295177] Workqueue: events delayed_work [ceph] [21522.303689] do_syscall_64+0x38/0x80 [21522.307257] [21522.307258] RIP: 0010:lock_acquire+0xf0/0x2b0 [21522.316638] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21522.321766] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21522.329237] RIP: 0033:0x7f48f0eeb93b [21522.333934] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21522.337506] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21522.338996] [21522.338996] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21522.343346] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21522.348390] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21522.367127] ORIG_RAX: 0000000000000012 [21522.370700] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21522.375916] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21522.394654] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21522.396144] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21522.403270] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21522.408485] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21522.415610] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21522.419442] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21522.426566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21522.433690] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21522.440819] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21522.447952] [21522.455070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21522.506708] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21522.513831] Call Trace: [21522.516278] [21522.518287] ? watchdog_timer_fn+0x1b4/0x220 [21522.522553] ? __hrtimer_run_queues+0x1d3/0x360 [21522.527079] ? hrtimer_interrupt+0x104/0x240 [21522.531348] ? __local_bh_enable+0x50/0x60 [21522.535441] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21522.540664] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21522.545621] [21522.547720] [21522.549821] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21522.555135] ? lock_acquire+0xf0/0x2b0 [21522.558883] ? igrab+0x43/0x50 [21522.561932] ? lock_release+0xc5/0x260 [21522.565677] _raw_spin_lock+0x30/0x40 [21522.569334] ? igrab+0x1d/0x50 [21522.572386] igrab+0x1d/0x50 [21522.575264] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21522.580414] delayed_work+0x1d8/0x280 [ceph] [21522.584693] ? process_scheduled_works+0x22a/0x510 [21522.589476] process_scheduled_works+0x2a4/0x510 [21522.594109] worker_thread+0x188/0x340 [21522.597859] ? rescuer_thread+0x310/0x310 [21522.601869] kthread+0xf5/0x130 [21522.605007] ? kthread_complete_and_exit+0x20/0x20 [21522.609791] ret_from_fork+0x31/0x40 [21522.613390] ? kthread_complete_and_exit+0x20/0x20 [21522.618171] ret_from_fork_asm+0x11/0x20 [21522.622095] [21549.835909] watchdog: BUG: soft lockup - CPU#0 stuck for 8955s! [dbench:78683] [21549.843125] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21549.847909] watchdog: BUG: soft lockup - CPU#7 stuck for 17952s! [kworker/7:4:78808] [21549.898584] irq event stamp: 32827510 [21549.898585] hardirqs last enabled at (32827509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21549.906314] Modules linked in: [21549.909972] hardirqs last disabled at (32827510): [] sysvec_apic_timer_interrupt+0xe/0x90 [21549.920130] ceph [21549.923180] softirqs last enabled at (32827454): [] __do_softirq+0x2be/0x371 [21549.932904] libceph [21549.934830] softirqs last disabled at (32827449): [] irq_exit_rcu+0xb0/0xd0 [21549.943513] dns_resolver fscache [21549.945697] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21549.954207] netfs veth [21549.957518] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21549.966464] nft_chain_nat xt_MASQUERADE [21549.968907] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21549.976378] nf_nat nf_conntrack [21549.980303] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21549.985780] nf_defrag_ipv6 nf_defrag_ipv4 [21549.989005] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21550.007742] nft_compat [21550.011833] [21550.011834] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21550.017051] nf_tables nfnetlink [21550.019496] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21550.020984] bridge stp [21550.028110] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21550.031334] llc xfs [21550.038457] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21550.040902] libcrc32c sunrpc [21550.048027] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21550.050211] intel_rapl_msr intel_rapl_common [21550.057334] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21550.060305] sb_edac x86_pkg_temp_thermal [21550.067431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21550.071782] coretemp kvm_intel [21550.079859] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21550.083864] kvm irqbypass [21550.089601] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21550.092737] i2c_i801 iTCO_wdt [21550.099862] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21550.102567] iTCO_vendor_support i2c_smbus [21550.109691] Call Trace: [21550.109693] [21550.112740] crct10dif_pclmul [21550.119865] ? watchdog_timer_fn+0x1b4/0x220 [21550.123957] joydev crc32_pclmul [21550.126402] ? __hrtimer_run_queues+0x1d3/0x360 [21550.128410] ipmi_ssif ghash_clmulni_intel [21550.131378] ? hrtimer_interrupt+0x104/0x240 [21550.135640] mei_me mei [21550.138862] ? __local_bh_enable+0x50/0x60 [21550.143388] lpc_ich sha512_ssse3 [21550.147482] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21550.151741] acpi_ipmi mfd_core [21550.154186] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21550.158276] wmi ipmi_si [21550.161588] [21550.166805] ipmi_devintf [21550.169942] [21550.174900] ipmi_msghandler ioatdma [21550.177431] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21550.179527] acpi_pad acpi_power_meter [21550.182148] ? queued_spin_lock_slowpath+0x16/0x2e0 [21550.184241] ixgbe igb [21550.187816] do_raw_spin_lock+0xb5/0xc0 [21550.193117] mdio crc32c_intel [21550.196863] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21550.201733] nvme ptp [21550.204092] try_get_cap_refs+0x159/0x880 [ceph] [21550.207920] nvme_core [21550.210975] __ceph_get_caps+0x4a1/0x760 [ceph] [21550.215755] i2c_algo_bit pps_core [21550.218031] ? __file_remove_privs+0x89/0x110 [21550.222636] dca fuse [21550.224995] ? wake_bit_function+0x70/0x70 [21550.229519] [21550.232918] ? ceph_write_iter+0x260/0x770 [ceph] [21550.237267] irq event stamp: 94904032 [21550.239538] ceph_write_iter+0x260/0x770 [ceph] [21550.243628] hardirqs last enabled at (94904031): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21550.245122] ? lock_is_held_type+0xaa/0x120 [21550.249816] hardirqs last disabled at (94904032): [] sysvec_apic_timer_interrupt+0xe/0x90 [21550.253476] vfs_write+0x324/0x4f0 [21550.257996] softirqs last enabled at (94904030): [] __do_softirq+0x2be/0x371 [21550.268159] ksys_pwrite64+0x65/0xa0 [21550.272332] softirqs last disabled at (94904025): [] irq_exit_rcu+0xb0/0xd0 [21550.282058] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21550.285453] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21550.294140] do_syscall_64+0x38/0x80 [21550.297710] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21550.306221] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21550.311351] Workqueue: events delayed_work [ceph] [21550.320727] RIP: 0033:0x7f48f0eeb93b [21550.324312] [21550.324313] RIP: 0010:lock_acquire+0xf0/0x2b0 [21550.331786] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21550.336830] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21550.341527] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21550.345099] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21550.346592] ORIG_RAX: 0000000000000012 [21550.350941] [21550.350942] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21550.369678] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21550.388416] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21550.393633] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21550.398850] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21550.402681] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21550.404173] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21550.411310] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21550.418438] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21550.425563] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21550.432685] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21550.439816] [21550.446935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21550.491440] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21550.498563] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21550.505687] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21550.512811] Call Trace: [21550.515256] [21550.517267] ? watchdog_timer_fn+0x1b4/0x220 [21550.521532] ? __hrtimer_run_queues+0x1d3/0x360 [21550.526058] ? hrtimer_interrupt+0x104/0x240 [21550.530325] ? __local_bh_enable+0x50/0x60 [21550.534421] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21550.539645] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21550.544601] [21550.546697] [21550.548797] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21550.554104] ? lock_acquire+0xf0/0x2b0 [21550.557854] ? igrab+0x43/0x50 [21550.560905] ? lock_release+0xc5/0x260 [21550.564649] _raw_spin_lock+0x30/0x40 [21550.568317] ? igrab+0x1d/0x50 [21550.571374] igrab+0x1d/0x50 [21550.574252] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21550.579402] delayed_work+0x1d8/0x280 [ceph] [21550.583682] ? process_scheduled_works+0x22a/0x510 [21550.588472] process_scheduled_works+0x2a4/0x510 [21550.593088] worker_thread+0x188/0x340 [21550.596837] ? rescuer_thread+0x310/0x310 [21550.600840] kthread+0xf5/0x130 [21550.603979] ? kthread_complete_and_exit+0x20/0x20 [21550.608762] ret_from_fork+0x31/0x40 [21550.612333] ? kthread_complete_and_exit+0x20/0x20 [21550.617117] ret_from_fork_asm+0x11/0x20 [21550.621040] [21577.836402] watchdog: BUG: soft lockup - CPU#0 stuck for 8981s! [dbench:78683] [21577.843623] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21577.848402] watchdog: BUG: soft lockup - CPU#7 stuck for 17978s! [kworker/7:4:78808] [21577.899077] irq event stamp: 32885710 [21577.899079] hardirqs last enabled at (32885709): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21577.906812] Modules linked in: [21577.910469] hardirqs last disabled at (32885710): [] sysvec_apic_timer_interrupt+0xe/0x90 [21577.920626] ceph [21577.923677] softirqs last enabled at (32885652): [] __do_softirq+0x2be/0x371 [21577.933401] libceph [21577.935325] softirqs last disabled at (32885647): [] irq_exit_rcu+0xb0/0xd0 [21577.944009] dns_resolver fscache [21577.946195] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21577.954704] netfs veth [21577.958015] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21577.966959] nft_chain_nat xt_MASQUERADE [21577.969403] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21577.976874] nf_nat [21577.980798] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21577.986278] nf_conntrack nf_defrag_ipv6 [21577.988374] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21578.007111] nf_defrag_ipv4 [21578.011030] [21578.016247] nft_compat nf_tables [21578.019038] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21578.020529] nfnetlink [21578.023838] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21578.030964] bridge stp [21578.033321] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21578.040445] llc xfs [21578.042888] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21578.050012] libcrc32c sunrpc [21578.052197] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21578.059320] intel_rapl_msr intel_rapl_common [21578.062286] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21578.069410] sb_edac x86_pkg_temp_thermal [21578.073760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21578.081837] coretemp [21578.085842] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21578.091579] kvm_intel kvm [21578.093849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21578.100973] irqbypass [21578.103677] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21578.110803] i2c_i801 iTCO_wdt [21578.113160] Call Trace: [21578.120283] iTCO_vendor_support i2c_smbus [21578.123336] [21578.125778] crct10dif_pclmul [21578.129888] ? watchdog_timer_fn+0x1b4/0x220 [21578.131897] joydev crc32_pclmul [21578.134862] ? __hrtimer_run_queues+0x1d3/0x360 [21578.139125] ipmi_ssif ghash_clmulni_intel [21578.142353] ? hrtimer_interrupt+0x104/0x240 [21578.146874] mei_me mei [21578.150963] ? __local_bh_enable+0x50/0x60 [21578.155229] lpc_ich sha512_ssse3 [21578.157674] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21578.161762] acpi_ipmi mfd_core [21578.165082] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21578.170299] wmi ipmi_si [21578.173436] [21578.178396] ipmi_devintf ipmi_msghandler [21578.180925] [21578.180926] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21578.183022] ioatdma acpi_pad [21578.187032] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21578.189124] acpi_power_meter [21578.194430] do_raw_spin_lock+0xb5/0xc0 [21578.197393] ixgbe igb [21578.202265] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21578.205226] mdio crc32c_intel [21578.209061] try_get_cap_refs+0x159/0x880 [ceph] [21578.211415] nvme ptp [21578.216205] __ceph_get_caps+0x4a1/0x760 [ceph] [21578.219249] nvme_core i2c_algo_bit [21578.223866] ? __file_remove_privs+0x89/0x110 [21578.226132] pps_core dca [21578.230655] ? wake_bit_function+0x70/0x70 [21578.234141] fuse [21578.238493] ? ceph_write_iter+0x260/0x770 [ceph] [21578.241107] [21578.241108] irq event stamp: 95124928 [21578.245198] ceph_write_iter+0x260/0x770 [ceph] [21578.247123] hardirqs last enabled at (95124927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21578.251825] ? lock_is_held_type+0xaa/0x120 [21578.253312] hardirqs last disabled at (95124928): [] sysvec_apic_timer_interrupt+0xe/0x90 [21578.256970] vfs_write+0x324/0x4f0 [21578.261492] softirqs last enabled at (95124926): [] __do_softirq+0x2be/0x371 [21578.271655] ksys_pwrite64+0x65/0xa0 [21578.275826] softirqs last disabled at (95124921): [] irq_exit_rcu+0xb0/0xd0 [21578.285552] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21578.288947] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21578.297634] do_syscall_64+0x38/0x80 [21578.301204] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21578.309715] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21578.314846] Workqueue: events delayed_work [ceph] [21578.324224] RIP: 0033:0x7f48f0eeb93b [21578.327805] [21578.327805] RIP: 0010:lock_acquire+0xf0/0x2b0 [21578.335274] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21578.340317] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21578.345013] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21578.348585] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21578.350091] ORIG_RAX: 0000000000000012 [21578.354443] [21578.354444] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21578.373181] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21578.391920] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21578.397135] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21578.402353] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21578.406185] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21578.407675] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21578.414806] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21578.421933] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21578.429056] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21578.436181] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21578.443310] [21578.450429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21578.494934] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21578.502057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21578.509181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21578.512781] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [21578.516306] Call Trace: [21578.516308] [21578.522391] rcu: 0-....: (9423809 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=4701291 [21578.524836] ? watchdog_timer_fn+0x1b4/0x220 [21578.526845] rcu: 7-....: (9363114 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/9361847 fqs=4701292 [21578.537176] ? __hrtimer_run_queues+0x1d3/0x360 [21578.541439] rcu: (detected by 3, t=9641739 jiffies, g=1309609, q=329698 ncpus=8) [21578.551862] ? hrtimer_interrupt+0x104/0x240 [21578.556382] Sending NMI from CPU 3 to CPUs 0: [21578.563852] ? __local_bh_enable+0x50/0x60 [21578.568119] NMI backtrace for cpu 0 [21578.568120] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21578.568122] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21578.568122] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21578.568125] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21578.568126] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21578.568128] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21578.568128] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21578.568129] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21578.568130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21578.568131] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21578.568131] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21578.568132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21578.568133] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21578.568134] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21578.568135] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21578.568135] Call Trace: [21578.568136] [21578.568137] ? nmi_cpu_backtrace+0xad/0x120 [21578.568140] ? nmi_cpu_backtrace_handler+0x11/0x20 [21578.568142] ? nmi_handle+0xd4/0x260 [21578.568146] ? default_do_nmi+0x49/0xf0 [21578.568148] ? exc_nmi+0x11e/0x150 [21578.568150] ? end_repeat_nmi+0x16/0x67 [21578.568154] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21578.568156] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21578.568159] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21578.568161] [21578.568161] [21578.568162] do_raw_spin_lock+0xb5/0xc0 [21578.568164] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21578.568179] try_get_cap_refs+0x159/0x880 [ceph] [21578.568193] __ceph_get_caps+0x4a1/0x760 [ceph] [21578.568206] ? __file_remove_privs+0x89/0x110 [21578.568208] ? wake_bit_function+0x70/0x70 [21578.568211] ? ceph_write_iter+0x260/0x770 [ceph] [21578.568221] ceph_write_iter+0x260/0x770 [ceph] [21578.568233] ? lock_is_held_type+0xaa/0x120 [21578.568235] vfs_write+0x324/0x4f0 [21578.568239] ksys_pwrite64+0x65/0xa0 [21578.568242] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21578.568245] do_syscall_64+0x38/0x80 [21578.568247] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21578.568249] RIP: 0033:0x7f48f0eeb93b [21578.568250] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21578.568251] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [21578.568253] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21578.568253] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21578.568254] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21578.568255] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21578.568255] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21578.568258] [21578.569117] Sending NMI from CPU 3 to CPUs 7: [21578.572471] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21578.576557] NMI backtrace for cpu 7 [21578.576558] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21578.576560] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21578.576561] Workqueue: events delayed_work [ceph] [21578.576580] RIP: 0010:mark_lock+0x5d/0x4c0 [21578.576582] Code: f5 89 d3 83 fa 08 74 6c 41 be 01 00 00 00 89 d1 41 d3 e6 4d 63 f6 41 0f b7 45 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 d3 7e e0 01 <73> 79 48 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 41 bc [21578.576584] RSP: 0018:ffffc90000264b50 EFLAGS: 00000003 [21578.576586] RAX: 0000000000000002 RBX: 0000000000000008 RCX: 000000000000000c [21578.576587] RDX: 0000000000000008 RSI: ffff88810debb578 RDI: ffff88810debaa80 [21578.576588] RBP: ffffc90000264bf0 R08: 00000000000000a0 R09: 0000000000000008 [21578.576589] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000002 [21578.576590] R13: ffff88810debb578 R14: 0000000000000100 R15: ffff88810debaa80 [21578.576592] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21578.576593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21578.576594] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21578.576596] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21578.576597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21578.576598] Call Trace: [21578.576599] [21578.576600] ? nmi_cpu_backtrace+0xad/0x120 [21578.576603] ? nmi_cpu_backtrace_handler+0x11/0x20 [21578.576605] ? nmi_handle+0xd4/0x260 [21578.576609] ? default_do_nmi+0x49/0xf0 [21578.576611] ? exc_nmi+0x11e/0x150 [21578.576613] ? end_repeat_nmi+0x16/0x67 [21578.576618] ? mark_lock+0x5d/0x4c0 [21578.576621] ? mark_lock+0x5d/0x4c0 [21578.576623] ? mark_lock+0x5d/0x4c0 [21578.576625] [21578.576626] [21578.576626] ? proc_sched_show_task+0xa63/0x1b80 [21578.576633] __lock_acquire+0x4d6/0x1e20 [21578.576637] lock_acquire+0xcb/0x2b0 [21578.576639] ? vprintk_emit+0x221/0x340 [21578.576642] ? vprintk_emit+0x200/0x340 [21578.576644] ? lock_release+0xc5/0x260 [21578.576647] vprintk_emit+0x240/0x340 [21578.576649] ? vprintk_emit+0x221/0x340 [21578.576652] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21578.576654] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21578.576657] _printk+0x5c/0x80 [21578.576660] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21578.576663] show_trace_log_lvl+0x1ac/0x3f0 [21578.576669] ? _raw_spin_lock+0x30/0x40 [21578.576673] ? touch_softlockup_watchdog+0x20/0x20 [21578.576675] watchdog_timer_fn+0x1b4/0x220 [21578.576678] __hrtimer_run_queues+0x1d3/0x360 [21578.576682] hrtimer_interrupt+0x104/0x240 [21578.576684] ? __local_bh_enable+0x50/0x60 [21578.576688] __sysvec_apic_timer_interrupt+0x61/0x160 [21578.576691] sysvec_apic_timer_interrupt+0x6d/0x90 [21578.576693] [21578.576694] [21578.576695] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21578.576698] RIP: 0010:lock_acquire+0xf0/0x2b0 [21578.576700] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21578.576701] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21578.576703] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21578.576704] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21578.576705] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21578.576706] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21578.576707] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21578.576712] ? igrab+0x43/0x50 [21578.576714] ? lock_release+0xc5/0x260 [21578.576717] _raw_spin_lock+0x30/0x40 [21578.576719] ? igrab+0x1d/0x50 [21578.576721] igrab+0x1d/0x50 [21578.576723] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21578.576743] delayed_work+0x1d8/0x280 [ceph] [21578.576762] ? process_scheduled_works+0x22a/0x510 [21578.576766] process_scheduled_works+0x2a4/0x510 [21578.576772] worker_thread+0x188/0x340 [21578.576775] ? rescuer_thread+0x310/0x310 [21578.576778] kthread+0xf5/0x130 [21578.576781] ? kthread_complete_and_exit+0x20/0x20 [21578.576785] ret_from_fork+0x31/0x40 [21578.576788] ? kthread_complete_and_exit+0x20/0x20 [21578.576790] ret_from_fork_asm+0x11/0x20 [21578.576797] [21579.275398] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21579.280357] [21579.282453] [21579.284551] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21579.289859] ? lock_acquire+0xf0/0x2b0 [21579.293610] ? igrab+0x43/0x50 [21579.296660] ? lock_release+0xc5/0x260 [21579.300403] _raw_spin_lock+0x30/0x40 [21579.304061] ? igrab+0x1d/0x50 [21579.307111] igrab+0x1d/0x50 [21579.309989] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21579.315138] delayed_work+0x1d8/0x280 [ceph] [21579.319418] ? process_scheduled_works+0x22a/0x510 [21579.324200] process_scheduled_works+0x2a4/0x510 [21579.328827] worker_thread+0x188/0x340 [21579.332574] ? rescuer_thread+0x310/0x310 [21579.336579] kthread+0xf5/0x130 [21579.339716] ? kthread_complete_and_exit+0x20/0x20 [21579.344501] ret_from_fork+0x31/0x40 [21579.348080] ? kthread_complete_and_exit+0x20/0x20 [21579.352864] ret_from_fork_asm+0x11/0x20 [21579.356785] [21605.836894] watchdog: BUG: soft lockup - CPU#0 stuck for 9008s! [dbench:78683] [21605.844108] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21605.848894] watchdog: BUG: soft lockup - CPU#7 stuck for 18005s! [kworker/7:4:78808] [21605.899567] irq event stamp: 32943946 [21605.907297] Modules linked in: ceph libceph dns_resolver [21605.910956] hardirqs last enabled at (32943945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21605.916260] fscache netfs veth [21605.926418] hardirqs last disabled at (32943946): [] sysvec_apic_timer_interrupt+0xe/0x90 [21605.926420] softirqs last enabled at (32943914): [] __do_softirq+0x2be/0x371 [21605.929568] nft_chain_nat xt_MASQUERADE [21605.939296] softirqs last disabled at (32943909): [] irq_exit_rcu+0xb0/0xd0 [21605.947979] nf_nat nf_conntrack [21605.951899] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21605.960408] nf_defrag_ipv6 nf_defrag_ipv4 [21605.963632] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21605.972576] nft_compat nf_tables [21605.976666] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21605.984139] nfnetlink bridge [21605.987449] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21605.992926] stp llc [21605.995890] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21606.014629] xfs [21606.016812] [21606.016813] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21606.022029] libcrc32c sunrpc [21606.023866] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21606.025356] intel_rapl_msr intel_rapl_common [21606.032482] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21606.035446] sb_edac [21606.042570] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21606.046922] x86_pkg_temp_thermal [21606.054044] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21606.056230] coretemp [21606.063353] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21606.066665] kvm_intel [21606.073787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21606.076058] kvm irqbypass [21606.084135] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21606.086493] i2c_i801 iTCO_wdt [21606.092232] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21606.094935] iTCO_vendor_support i2c_smbus [21606.102060] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21606.105110] crct10dif_pclmul [21606.112233] Call Trace: [21606.116324] joydev crc32_pclmul [21606.123449] [21606.126413] ipmi_ssif ghash_clmulni_intel [21606.128857] ? watchdog_timer_fn+0x1b4/0x220 [21606.132080] mei_me [21606.134092] ? __hrtimer_run_queues+0x1d3/0x360 [21606.138183] mei lpc_ich [21606.142449] ? hrtimer_interrupt+0x104/0x240 [21606.144545] sha512_ssse3 acpi_ipmi [21606.149067] ? __local_bh_enable+0x50/0x60 [21606.151599] mfd_core wmi [21606.155866] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21606.159348] ipmi_si ipmi_devintf [21606.163439] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21606.166054] ipmi_msghandler ioatdma [21606.171272] [21606.174601] acpi_pad acpi_power_meter [21606.179573] [21606.179575] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21606.183145] ixgbe igb [21606.185247] ? queued_spin_lock_slowpath+0x16/0x2e0 [21606.188988] mdio crc32c_intel nvme [21606.191086] do_raw_spin_lock+0xb5/0xc0 [21606.196388] ptp nvme_core [21606.198746] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21606.203617] i2c_algo_bit pps_core [21606.207103] try_get_cap_refs+0x159/0x880 [ceph] [21606.210931] dca fuse [21606.213640] __ceph_get_caps+0x4a1/0x760 [ceph] [21606.218421] irq event stamp: 95339788 [21606.221823] ? __file_remove_privs+0x89/0x110 [21606.226427] hardirqs last enabled at (95339787): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21606.228698] ? wake_bit_function+0x70/0x70 [21606.233223] hardirqs last disabled at (95339788): [] sysvec_apic_timer_interrupt+0xe/0x90 [21606.236882] ? ceph_write_iter+0x260/0x770 [ceph] [21606.241230] softirqs last enabled at (95339786): [] __do_softirq+0x2be/0x371 [21606.251388] ceph_write_iter+0x260/0x770 [ceph] [21606.255480] softirqs last disabled at (95339781): [] irq_exit_rcu+0xb0/0xd0 [21606.265207] ? lock_is_held_type+0xaa/0x120 [21606.269902] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21606.278588] vfs_write+0x324/0x4f0 [21606.283108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21606.291624] ksys_pwrite64+0x65/0xa0 [21606.295798] Workqueue: events delayed_work [ceph] [21606.305176] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21606.308580] [21606.316053] do_syscall_64+0x38/0x80 [21606.319622] RIP: 0010:lock_acquire+0xf0/0x2b0 [21606.324320] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21606.329450] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21606.330941] RIP: 0033:0x7f48f0eeb93b [21606.334511] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21606.338863] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21606.343907] [21606.343908] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21606.362643] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21606.366215] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21606.371432] ORIG_RAX: 0000000000000012 [21606.371433] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21606.390171] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21606.391661] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21606.398784] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21606.404004] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21606.411128] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21606.414957] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21606.422080] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21606.429205] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21606.436331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21606.443460] [21606.450578] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21606.495075] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21606.502199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21606.509323] Call Trace: [21606.511766] [21606.513778] ? watchdog_timer_fn+0x1b4/0x220 [21606.518043] ? __hrtimer_run_queues+0x1d3/0x360 [21606.522584] ? hrtimer_interrupt+0x104/0x240 [21606.526846] ? __local_bh_enable+0x50/0x60 [21606.530939] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21606.536155] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21606.541112] [21606.543208] [21606.545306] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21606.550616] ? lock_acquire+0xf0/0x2b0 [21606.554365] ? igrab+0x43/0x50 [21606.557415] ? lock_release+0xc5/0x260 [21606.561159] _raw_spin_lock+0x30/0x40 [21606.564815] ? igrab+0x1d/0x50 [21606.567867] igrab+0x1d/0x50 [21606.570746] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21606.575896] delayed_work+0x1d8/0x280 [ceph] [21606.580176] ? process_scheduled_works+0x22a/0x510 [21606.584966] process_scheduled_works+0x2a4/0x510 [21606.589586] worker_thread+0x188/0x340 [21606.593332] ? rescuer_thread+0x310/0x310 [21606.597344] kthread+0xf5/0x130 [21606.600479] ? kthread_complete_and_exit+0x20/0x20 [21606.605265] ret_from_fork+0x31/0x40 [21606.608833] ? kthread_complete_and_exit+0x20/0x20 [21606.613618] ret_from_fork_asm+0x11/0x20 [21606.617541] [21633.837382] watchdog: BUG: soft lockup - CPU#0 stuck for 9034s! [dbench:78683] [21633.844595] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21633.849382] watchdog: BUG: soft lockup - CPU#7 stuck for 18031s! [kworker/7:4:78808] [21633.900063] irq event stamp: 33002218 [21633.907795] Modules linked in: ceph libceph dns_resolver fscache netfs [21633.911451] hardirqs last enabled at (33002217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21633.917978] veth nft_chain_nat xt_MASQUERADE nf_nat [21633.928134] hardirqs last disabled at (33002218): [] sysvec_apic_timer_interrupt+0xe/0x90 [21633.933092] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [21633.942816] softirqs last enabled at (33002160): [] __do_softirq+0x2be/0x371 [21633.950720] bridge stp llc xfs libcrc32c sunrpc [21633.959405] softirqs last disabled at (33002155): [] irq_exit_rcu+0xb0/0xd0 [21633.964016] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [21633.972528] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21633.979390] coretemp kvm_intel [21633.988335] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21633.988336] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21633.991471] kvm [21633.998943] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21634.004421] irqbypass i2c_i801 [21634.006257] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21634.024996] iTCO_wdt iTCO_vendor_support [21634.028133] [21634.033350] i2c_smbus crct10dif_pclmul [21634.037354] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21634.038844] joydev crc32_pclmul [21634.042675] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21634.049799] ipmi_ssif ghash_clmulni_intel [21634.053024] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21634.060149] mei_me [21634.064240] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21634.071362] mei lpc_ich [21634.073459] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21634.080585] sha512_ssse3 acpi_ipmi [21634.083116] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21634.090240] mfd_core [21634.093723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21634.101802] wmi [21634.104074] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21634.109809] ipmi_si ipmi_devintf [21634.111647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21634.118780] ipmi_msghandler [21634.122090] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21634.129214] ioatdma acpi_pad [21634.132091] Call Trace: [21634.132093] [21634.139214] acpi_power_meter [21634.142183] ? watchdog_timer_fn+0x1b4/0x220 [21634.144624] ixgbe igb [21634.146635] ? __hrtimer_run_queues+0x1d3/0x360 [21634.149598] mdio crc32c_intel [21634.153865] ? hrtimer_interrupt+0x104/0x240 [21634.156218] nvme ptp [21634.160743] ? __local_bh_enable+0x50/0x60 [21634.163794] nvme_core [21634.168075] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21634.170345] i2c_algo_bit pps_core [21634.174438] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21634.176795] dca fuse [21634.182014] [21634.185409] [21634.185410] irq event stamp: 95560880 [21634.190367] [21634.192638] hardirqs last enabled at (95560879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21634.194736] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21634.196226] hardirqs last disabled at (95560880): [] sysvec_apic_timer_interrupt+0xe/0x90 [21634.199887] ? queued_spin_lock_slowpath+0x16/0x2e0 [21634.201979] softirqs last enabled at (95560878): [] __do_softirq+0x2be/0x371 [21634.212140] do_raw_spin_lock+0xb5/0xc0 [21634.217442] softirqs last disabled at (95560873): [] irq_exit_rcu+0xb0/0xd0 [21634.227167] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21634.232037] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21634.240741] try_get_cap_refs+0x159/0x880 [ceph] [21634.244568] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21634.253085] __ceph_get_caps+0x4a1/0x760 [ceph] [21634.257866] Workqueue: events delayed_work [ceph] [21634.267246] ? __file_remove_privs+0x89/0x110 [21634.271852] [21634.279325] ? wake_bit_function+0x70/0x70 [21634.283847] RIP: 0010:lock_acquire+0xf0/0x2b0 [21634.288548] ? ceph_write_iter+0x260/0x770 [ceph] [21634.292896] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21634.294387] ceph_write_iter+0x260/0x770 [ceph] [21634.298478] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21634.302834] ? lock_is_held_type+0xaa/0x120 [21634.307525] [21634.307525] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21634.326265] vfs_write+0x324/0x4f0 [21634.330786] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21634.336008] ksys_pwrite64+0x65/0xa0 [21634.340182] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21634.341672] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21634.348796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21634.352196] do_syscall_64+0x38/0x80 [21634.359319] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21634.362891] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21634.370014] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21634.375146] RIP: 0033:0x7f48f0eeb93b [21634.382267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21634.385839] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21634.392962] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21634.398007] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21634.406084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21634.409655] ORIG_RAX: 0000000000000012 [21634.415393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21634.434129] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21634.441255] Call Trace: [21634.441257] [21634.446472] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21634.453597] ? watchdog_timer_fn+0x1b4/0x220 [21634.457426] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21634.464551] ? __hrtimer_run_queues+0x1d3/0x360 [21634.471675] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21634.474122] ? hrtimer_interrupt+0x104/0x240 [21634.476130] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21634.483255] ? __local_bh_enable+0x50/0x60 [21634.487524] [21634.494645] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21634.529188] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21634.534146] [21634.536244] [21634.538342] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21634.543650] ? lock_acquire+0xf0/0x2b0 [21634.547399] ? igrab+0x43/0x50 [21634.550449] ? lock_release+0xc5/0x260 [21634.554193] _raw_spin_lock+0x30/0x40 [21634.557850] ? igrab+0x1d/0x50 [21634.560899] igrab+0x1d/0x50 [21634.563791] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21634.568937] delayed_work+0x1d8/0x280 [ceph] [21634.573218] ? process_scheduled_works+0x22a/0x510 [21634.578009] process_scheduled_works+0x2a4/0x510 [21634.582623] worker_thread+0x188/0x340 [21634.586374] ? rescuer_thread+0x310/0x310 [21634.590384] kthread+0xf5/0x130 [21634.593522] ? kthread_complete_and_exit+0x20/0x20 [21634.598308] ret_from_fork+0x31/0x40 [21634.601886] ? kthread_complete_and_exit+0x20/0x20 [21634.606670] ret_from_fork_asm+0x11/0x20 [21634.610594] [21661.837870] watchdog: BUG: soft lockup - CPU#0 stuck for 9060s! [dbench:78683] [21661.845084] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21661.849870] watchdog: BUG: soft lockup - CPU#7 stuck for 18057s! [kworker/7:4:78808] [21661.900524] irq event stamp: 33060328 [21661.900542] hardirqs last enabled at (33060327): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21661.908272] Modules linked in: [21661.911928] hardirqs last disabled at (33060328): [] sysvec_apic_timer_interrupt+0xe/0x90 [21661.922087] ceph [21661.925138] softirqs last enabled at (33060270): [] __do_softirq+0x2be/0x371 [21661.934862] libceph [21661.936787] softirqs last disabled at (33060265): [] irq_exit_rcu+0xb0/0xd0 [21661.945469] dns_resolver fscache [21661.947653] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21661.956166] netfs [21661.959476] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21661.968420] veth nft_chain_nat [21661.970430] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [21661.977900] xt_MASQUERADE [21661.981038] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [21661.986517] nf_nat nf_conntrack [21661.989219] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21662.007958] nf_defrag_ipv6 [21662.011181] [21662.011182] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21662.016398] nf_defrag_ipv4 nft_compat [21662.019189] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21662.020680] nf_tables [21662.027805] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21662.031548] nfnetlink bridge [21662.038674] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21662.041031] stp llc [21662.048155] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21662.051119] xfs libcrc32c [21662.058244] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21662.060443] sunrpc intel_rapl_msr [21662.067569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21662.070273] intel_rapl_common [21662.078349] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21662.081748] sb_edac [21662.087484] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21662.090548] x86_pkg_temp_thermal [21662.097677] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21662.099862] coretemp [21662.106984] Call Trace: [21662.106986] [21662.110295] kvm_intel kvm [21662.117420] ? watchdog_timer_fn+0x1b4/0x220 [21662.119691] irqbypass i2c_i801 [21662.122137] ? __hrtimer_run_queues+0x1d3/0x360 [21662.124145] iTCO_wdt iTCO_vendor_support [21662.126851] ? hrtimer_interrupt+0x104/0x240 [21662.131113] i2c_smbus crct10dif_pclmul [21662.134267] ? __local_bh_enable+0x50/0x60 [21662.138793] joydev [21662.142799] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21662.147061] crc32_pclmul [21662.150891] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21662.154981] ipmi_ssif ghash_clmulni_intel [21662.157080] [21662.162297] mei_me mei [21662.164914] [21662.164916] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21662.169871] lpc_ich [21662.173967] ? queued_spin_lock_slowpath+0x2b/0x2e0 [21662.176058] sha512_ssse3 [21662.178505] do_raw_spin_lock+0xb5/0xc0 [21662.180600] acpi_ipmi mfd_core [21662.185907] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21662.188090] wmi ipmi_si [21662.192963] try_get_cap_refs+0x159/0x880 [ceph] [21662.195578] ipmi_devintf ipmi_msghandler [21662.199413] __ceph_get_caps+0x4a1/0x760 [ceph] [21662.202550] ioatdma acpi_pad [21662.207344] ? __file_remove_privs+0x89/0x110 [21662.209868] acpi_power_meter ixgbe [21662.214479] ? wake_bit_function+0x70/0x70 [21662.218483] igb mdio [21662.223010] ? ceph_write_iter+0x260/0x770 [ceph] [21662.225971] crc32c_intel nvme [21662.230322] ceph_write_iter+0x260/0x770 [ceph] [21662.233807] ptp nvme_core [21662.237902] ? lock_is_held_type+0xaa/0x120 [21662.240168] i2c_algo_bit pps_core [21662.244869] vfs_write+0x324/0x4f0 [21662.247915] dca fuse [21662.252446] ksys_pwrite64+0x65/0xa0 [21662.255144] [21662.255145] irq event stamp: 95781908 [21662.259321] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21662.262718] hardirqs last enabled at (95781907): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21662.266117] do_syscall_64+0x38/0x80 [21662.268388] hardirqs last disabled at (95781908): [] sysvec_apic_timer_interrupt+0xe/0x90 [21662.271959] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21662.273448] softirqs last enabled at (95781906): [] __do_softirq+0x2be/0x371 [21662.277106] RIP: 0033:0x7f48f0eeb93b [21662.282236] softirqs last disabled at (95781901): [] irq_exit_rcu+0xb0/0xd0 [21662.292395] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21662.295966] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21662.305689] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21662.310732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21662.319418] ORIG_RAX: 0000000000000012 [21662.322989] Workqueue: events delayed_work [ceph] [21662.331499] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21662.350236] [21662.359613] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21662.364832] RIP: 0010:lock_acquire+0x85/0x2b0 [21662.372303] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21662.376133] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [21662.380831] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21662.387954] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [21662.389444] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21662.396569] [21662.396570] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [21662.400926] [21662.408044] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [21662.464189] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [21662.471313] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [21662.478438] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21662.485562] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21662.493637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21662.499376] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21662.506499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21662.513623] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21662.520747] Call Trace: [21662.523192] [21662.525204] ? watchdog_timer_fn+0x1b4/0x220 [21662.529469] ? __hrtimer_run_queues+0x1d3/0x360 [21662.533993] ? hrtimer_interrupt+0x104/0x240 [21662.538271] ? __local_bh_enable+0x50/0x60 [21662.542366] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21662.547590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21662.552554] [21662.554653] [21662.556750] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21662.562058] ? lock_acquire+0x85/0x2b0 [21662.565809] ? igrab+0x43/0x50 [21662.568858] ? lock_release+0xc5/0x260 [21662.572603] _raw_spin_lock+0x30/0x40 [21662.576276] ? igrab+0x1d/0x50 [21662.579326] igrab+0x1d/0x50 [21662.582207] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21662.587355] delayed_work+0x1d8/0x280 [ceph] [21662.591638] ? process_scheduled_works+0x22a/0x510 [21662.596427] process_scheduled_works+0x2a4/0x510 [21662.601043] worker_thread+0x188/0x340 [21662.604793] ? rescuer_thread+0x310/0x310 [21662.608804] kthread+0xf5/0x130 [21662.611941] ? kthread_complete_and_exit+0x20/0x20 [21662.616725] ret_from_fork+0x31/0x40 [21662.620304] ? kthread_complete_and_exit+0x20/0x20 [21662.625089] ret_from_fork_asm+0x11/0x20 [21662.629011] [21689.838359] watchdog: BUG: soft lockup - CPU#0 stuck for 9086s! [dbench:78683] [21689.845577] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21689.850359] watchdog: BUG: soft lockup - CPU#7 stuck for 18083s! [kworker/7:4:78808] [21689.901036] irq event stamp: 33118618 [21689.908767] Modules linked in: ceph libceph dns_resolver fscache netfs [21689.912426] hardirqs last enabled at (33118617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21689.918952] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [21689.929108] hardirqs last disabled at (33118618): [] sysvec_apic_timer_interrupt+0xe/0x90 [21689.936492] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [21689.946218] softirqs last enabled at (33118612): [] __do_softirq+0x2be/0x371 [21689.946220] stp llc xfs libcrc32c sunrpc intel_rapl_msr [21689.960985] softirqs last disabled at (33118607): [] irq_exit_rcu+0xb0/0xd0 [21689.960989] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [21689.974810] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21689.982368] irqbypass i2c_i801 [21689.991312] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21689.994449] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [21690.001919] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21690.008264] joydev crc32_pclmul [21690.013755] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21690.013756] ipmi_ssif [21690.016982] ghash_clmulni_intel [21690.035718] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21690.038075] mei_me [21690.041302] [21690.041302] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21690.046518] mei [21690.048616] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21690.050107] lpc_ich [21690.057230] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21690.059066] sha512_ssse3 acpi_ipmi [21690.066192] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21690.068376] mfd_core [21690.075501] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21690.078985] wmi ipmi_si [21690.086107] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21690.088379] ipmi_devintf [21690.095503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21690.098043] ipmi_msghandler ioatdma [21690.106121] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21690.108755] acpi_pad acpi_power_meter [21690.114492] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21690.118064] ixgbe igb [21690.125188] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21690.128932] mdio crc32c_intel [21690.136056] Call Trace: [21690.136058] [21690.138412] nvme ptp [21690.145539] ? watchdog_timer_fn+0x1b4/0x220 [21690.148588] nvme_core i2c_algo_bit [21690.151039] ? __hrtimer_run_queues+0x1d3/0x360 [21690.153052] pps_core dca [21690.155324] ? hrtimer_interrupt+0x104/0x240 [21690.159585] fuse [21690.163070] ? __local_bh_enable+0x50/0x60 [21690.167593] [21690.167594] irq event stamp: 96002634 [21690.170215] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21690.174474] hardirqs last enabled at (96002633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21690.176400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21690.180490] hardirqs last disabled at (96002634): [] sysvec_apic_timer_interrupt+0xe/0x90 [21690.181981] [21690.185638] softirqs last enabled at (96002632): [] __do_softirq+0x2be/0x371 [21690.190855] [21690.190857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21690.201014] softirqs last disabled at (96002627): [] irq_exit_rcu+0xb0/0xd0 [21690.205993] ? queued_spin_lock_slowpath+0x18/0x2e0 [21690.215711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21690.217812] do_raw_spin_lock+0xb5/0xc0 [21690.226495] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21690.228592] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21690.233895] Workqueue: events delayed_work [ceph] [21690.242409] try_get_cap_refs+0x159/0x880 [ceph] [21690.247278] [21690.247278] RIP: 0010:lock_acquire+0xf0/0x2b0 [21690.256660] __ceph_get_caps+0x4a1/0x760 [ceph] [21690.260485] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21690.267960] ? __file_remove_privs+0x89/0x110 [21690.272756] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21690.277455] ? wake_bit_function+0x70/0x70 [21690.282066] [21690.282066] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21690.283559] ? ceph_write_iter+0x260/0x770 [ceph] [21690.287906] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21690.292432] ceph_write_iter+0x260/0x770 [ceph] [21690.311168] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21690.315522] ? lock_is_held_type+0xaa/0x120 [21690.320737] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21690.324846] vfs_write+0x324/0x4f0 [21690.326336] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21690.333463] ksys_pwrite64+0x65/0xa0 [21690.338156] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21690.345283] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21690.349805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21690.356931] do_syscall_64+0x38/0x80 [21690.361105] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21690.368232] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21690.371629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21690.378762] RIP: 0033:0x7f48f0eeb93b [21690.382331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21690.390411] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21690.395540] Call Trace: [21690.401278] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21690.404849] [21690.411973] ORIG_RAX: 0000000000000012 [21690.417017] ? watchdog_timer_fn+0x1b4/0x220 [21690.424140] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21690.427713] ? __hrtimer_run_queues+0x1d3/0x360 [21690.434836] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21690.453576] ? hrtimer_interrupt+0x104/0x240 [21690.456016] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21690.461234] ? __local_bh_enable+0x50/0x60 [21690.463244] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21690.467080] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21690.471340] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21690.478466] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21690.482994] [21690.490113] [21690.534296] [21690.536393] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21690.541703] ? lock_acquire+0xf0/0x2b0 [21690.545452] ? igrab+0x43/0x50 [21690.548501] ? lock_release+0xc5/0x260 [21690.552247] _raw_spin_lock+0x30/0x40 [21690.555903] ? igrab+0x1d/0x50 [21690.558952] igrab+0x1d/0x50 [21690.561832] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21690.566982] delayed_work+0x1d8/0x280 [ceph] [21690.571262] ? process_scheduled_works+0x22a/0x510 [21690.576052] process_scheduled_works+0x2a4/0x510 [21690.580668] worker_thread+0x188/0x340 [21690.584417] ? rescuer_thread+0x310/0x310 [21690.588422] kthread+0xf5/0x130 [21690.591567] ? kthread_complete_and_exit+0x20/0x20 [21690.596352] ret_from_fork+0x31/0x40 [21690.599930] ? kthread_complete_and_exit+0x20/0x20 [21690.604715] ret_from_fork_asm+0x11/0x20 [21690.608637] [21717.838847] watchdog: BUG: soft lockup - CPU#0 stuck for 9112s! [dbench:78683] [21717.846065] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21717.850847] watchdog: BUG: soft lockup - CPU#7 stuck for 18109s! [kworker/7:4:78808] [21717.901523] irq event stamp: 33176762 [21717.901524] hardirqs last enabled at (33176761): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21717.909254] Modules linked in: [21717.912911] hardirqs last disabled at (33176762): [] sysvec_apic_timer_interrupt+0xe/0x90 [21717.923069] ceph [21717.926120] softirqs last enabled at (33176752): [] __do_softirq+0x2be/0x371 [21717.935845] libceph [21717.937769] softirqs last disabled at (33176747): [] irq_exit_rcu+0xb0/0xd0 [21717.946453] dns_resolver fscache [21717.948637] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21717.957146] netfs [21717.960458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21717.969401] veth nft_chain_nat [21717.971415] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21717.978884] xt_MASQUERADE nf_nat [21717.982022] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21717.987500] nf_conntrack nf_defrag_ipv6 [21717.990826] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21718.009564] nf_defrag_ipv4 nft_compat [21718.013482] [21718.018701] nf_tables nfnetlink [21718.022444] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21718.023935] bridge stp [21718.027159] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21718.034281] llc xfs [21718.036726] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21718.043850] libcrc32c sunrpc [21718.046035] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21718.053158] intel_rapl_msr intel_rapl_common [21718.056122] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21718.063247] sb_edac x86_pkg_temp_thermal [21718.067598] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21718.074722] coretemp kvm_intel [21718.078725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21718.086802] kvm irqbypass [21718.089941] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21718.095679] i2c_i801 iTCO_wdt [21718.098383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21718.105507] iTCO_vendor_support i2c_smbus [21718.108575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21718.115700] crct10dif_pclmul [21718.119789] Call Trace: [21718.119791] [21718.126914] joydev crc32_pclmul [21718.129880] ? watchdog_timer_fn+0x1b4/0x220 [21718.132321] ipmi_ssif ghash_clmulni_intel [21718.134333] ? __hrtimer_run_queues+0x1d3/0x360 [21718.137555] mei_me mei [21718.141823] ? hrtimer_interrupt+0x104/0x240 [21718.145911] lpc_ich sha512_ssse3 [21718.150434] ? __local_bh_enable+0x50/0x60 [21718.152879] acpi_ipmi mfd_core [21718.157146] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21718.160454] wmi ipmi_si [21718.164547] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21718.167682] ipmi_devintf ipmi_msghandler [21718.172899] [21718.175431] ioatdma acpi_pad [21718.180387] [21718.180388] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21718.184392] acpi_power_meter [21718.186494] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21718.189453] ixgbe igb [21718.191554] do_raw_spin_lock+0xb5/0xc0 [21718.196854] mdio crc32c_intel [21718.199820] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21718.204690] nvme ptp [21718.207049] try_get_cap_refs+0x159/0x880 [ceph] [21718.210877] nvme_core i2c_algo_bit [21718.213933] __ceph_get_caps+0x4a1/0x760 [ceph] [21718.218712] pps_core dca [21718.220987] ? __file_remove_privs+0x89/0x110 [21718.225595] fuse [21718.229079] ? wake_bit_function+0x70/0x70 [21718.233602] irq event stamp: 96223652 [21718.236243] ? ceph_write_iter+0x260/0x770 [ceph] [21718.240587] hardirqs last enabled at (96223651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21718.242528] ceph_write_iter+0x260/0x770 [ceph] [21718.246619] hardirqs last disabled at (96223652): [] sysvec_apic_timer_interrupt+0xe/0x90 [21718.250282] ? lock_is_held_type+0xaa/0x120 [21718.254973] softirqs last enabled at (96223650): [] __do_softirq+0x2be/0x371 [21718.265133] vfs_write+0x324/0x4f0 [21718.269656] softirqs last disabled at (96223645): [] irq_exit_rcu+0xb0/0xd0 [21718.279384] ksys_pwrite64+0x65/0xa0 [21718.283557] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21718.292248] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21718.295648] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21718.304160] do_syscall_64+0x38/0x80 [21718.307729] Workqueue: events delayed_work [ceph] [21718.317108] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21718.322249] [21718.329716] RIP: 0033:0x7f48f0eeb93b [21718.333288] RIP: 0010:lock_acquire+0xf0/0x2b0 [21718.337985] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21718.343030] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21718.344528] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21718.348101] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21718.352449] ORIG_RAX: 0000000000000012 [21718.371188] [21718.371188] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21718.389925] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21718.395143] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21718.400360] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21718.404190] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21718.405683] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21718.412805] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21718.419930] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21718.427054] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21718.434180] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21718.441302] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21718.448432] [21718.455550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21718.492924] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21718.500047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21718.507171] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21718.514295] Call Trace: [21718.516740] [21718.518750] ? watchdog_timer_fn+0x1b4/0x220 [21718.523015] ? __hrtimer_run_queues+0x1d3/0x360 [21718.527540] ? hrtimer_interrupt+0x104/0x240 [21718.531811] ? __local_bh_enable+0x50/0x60 [21718.535903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21718.541118] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21718.546077] [21718.548174] [21718.550271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21718.555578] ? lock_acquire+0xf0/0x2b0 [21718.559330] ? igrab+0x43/0x50 [21718.562379] ? lock_release+0xc5/0x260 [21718.566125] _raw_spin_lock+0x30/0x40 [21718.569788] ? igrab+0x1d/0x50 [21718.572839] igrab+0x1d/0x50 [21718.575716] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21718.580868] delayed_work+0x1d8/0x280 [ceph] [21718.585149] ? process_scheduled_works+0x22a/0x510 [21718.589939] process_scheduled_works+0x2a4/0x510 [21718.594556] worker_thread+0x188/0x340 [21718.598305] ? rescuer_thread+0x310/0x310 [21718.602316] kthread+0xf5/0x130 [21718.605462] ? kthread_complete_and_exit+0x20/0x20 [21718.610257] ret_from_fork+0x31/0x40 [21718.613835] ? kthread_complete_and_exit+0x20/0x20 [21718.618619] ret_from_fork_asm+0x11/0x20 [21718.622549] [21745.839337] watchdog: BUG: soft lockup - CPU#0 stuck for 9138s! [dbench:78683] [21745.846553] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21745.851338] watchdog: BUG: soft lockup - CPU#7 stuck for 18135s! [kworker/7:4:78808] [21745.901996] irq event stamp: 33234982 [21745.909742] Modules linked in: ceph libceph dns_resolver fscache netfs [21745.913399] hardirqs last enabled at (33234981): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21745.919917] veth nft_chain_nat xt_MASQUERADE nf_nat [21745.930075] hardirqs last disabled at (33234982): [] sysvec_apic_timer_interrupt+0xe/0x90 [21745.935033] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [21745.944756] softirqs last enabled at (33234924): [] __do_softirq+0x2be/0x371 [21745.951795] nfnetlink bridge stp llc xfs [21745.960477] softirqs last disabled at (33234919): [] irq_exit_rcu+0xb0/0xd0 [21745.964481] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [21745.973010] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21745.978834] sb_edac x86_pkg_temp_thermal [21745.987779] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21745.987780] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21745.991783] coretemp [21745.999254] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21746.004735] kvm_intel kvm [21746.007011] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21746.025746] irqbypass i2c_i801 [21746.028451] [21746.033670] iTCO_wdt iTCO_vendor_support [21746.036807] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21746.038297] i2c_smbus crct10dif_pclmul [21746.042302] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21746.049427] joydev [21746.053255] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21746.060380] crc32_pclmul [21746.062477] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21746.069602] ipmi_ssif ghash_clmulni_intel [21746.072220] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21746.079343] mei_me mei [21746.083434] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21746.090558] lpc_ich [21746.093015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21746.101089] sha512_ssse3 acpi_ipmi [21746.103273] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21746.109013] mfd_core wmi [21746.112493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21746.119619] ipmi_si ipmi_devintf [21746.122235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21746.129359] ipmi_msghandler ioatdma [21746.132670] Call Trace: [21746.132672] [21746.139795] acpi_pad acpi_power_meter [21746.143366] ? watchdog_timer_fn+0x1b4/0x220 [21746.145810] ixgbe igb [21746.147821] ? __hrtimer_run_queues+0x1d3/0x360 [21746.151565] mdio crc32c_intel [21746.155832] ? hrtimer_interrupt+0x104/0x240 [21746.158185] nvme ptp [21746.162708] ? __local_bh_enable+0x50/0x60 [21746.165777] nvme_core i2c_algo_bit [21746.170043] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21746.172313] pps_core dca [21746.176403] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21746.179887] fuse [21746.185106] [21746.187734] irq event stamp: 96444550 [21746.192688] [21746.194613] hardirqs last enabled at (96444549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21746.196711] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21746.200385] hardirqs last disabled at (96444550): [] sysvec_apic_timer_interrupt+0xe/0x90 [21746.202486] ? queued_spin_lock_slowpath+0x16/0x2e0 [21746.212640] softirqs last enabled at (96444548): [] __do_softirq+0x2be/0x371 [21746.217944] do_raw_spin_lock+0xb5/0xc0 [21746.227667] softirqs last disabled at (96444543): [] irq_exit_rcu+0xb0/0xd0 [21746.232540] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21746.241222] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21746.245057] try_get_cap_refs+0x159/0x880 [ceph] [21746.253565] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21746.258354] __ceph_get_caps+0x4a1/0x760 [ceph] [21746.267740] Workqueue: events delayed_work [ceph] [21746.272359] ? __file_remove_privs+0x89/0x110 [21746.279824] [21746.279825] RIP: 0010:lock_release+0x14e/0x260 [21746.284350] ? wake_bit_function+0x70/0x70 [21746.289046] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [21746.293400] ? ceph_write_iter+0x260/0x770 [ceph] [21746.294886] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [21746.299325] ceph_write_iter+0x260/0x770 [ceph] [21746.303416] [21746.303417] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [21746.322159] ? lock_is_held_type+0xaa/0x120 [21746.326851] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [21746.332071] vfs_write+0x324/0x4f0 [21746.336591] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [21746.338088] ksys_pwrite64+0x65/0xa0 [21746.345207] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [21746.349385] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21746.356508] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [21746.359908] do_syscall_64+0x38/0x80 [21746.367030] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21746.370603] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21746.377738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21746.382874] RIP: 0033:0x7f48f0eeb93b [21746.389997] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21746.393570] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21746.401646] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21746.406689] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21746.412427] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21746.415997] ORIG_RAX: 0000000000000012 [21746.423120] Call Trace: [21746.423122] [21746.441860] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21746.448983] ? watchdog_timer_fn+0x1b4/0x220 [21746.454202] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21746.461326] ? __hrtimer_run_queues+0x1d3/0x360 [21746.465156] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21746.467604] ? hrtimer_interrupt+0x104/0x240 [21746.469610] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21746.476739] ? __local_bh_enable+0x50/0x60 [21746.481006] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21746.488135] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21746.492661] [21746.499779] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21746.534743] [21746.536838] [21746.538938] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21746.544240] ? igrab+0x43/0x50 [21746.547295] ? lock_release+0x14e/0x260 [21746.551124] ? lock_release+0xc5/0x260 [21746.554869] _raw_spin_unlock+0x1b/0x40 [21746.558708] igrab+0x43/0x50 [21746.561585] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21746.566744] delayed_work+0x1d8/0x280 [ceph] [21746.571038] ? process_scheduled_works+0x22a/0x510 [21746.575823] process_scheduled_works+0x2a4/0x510 [21746.580439] worker_thread+0x188/0x340 [21746.584188] ? rescuer_thread+0x310/0x310 [21746.588189] kthread+0xf5/0x130 [21746.591329] ? kthread_complete_and_exit+0x20/0x20 [21746.596112] ret_from_fork+0x31/0x40 [21746.599693] ? kthread_complete_and_exit+0x20/0x20 [21746.604475] ret_from_fork_asm+0x11/0x20 [21746.608401] [21773.579190] rcu: INFO: rcu_preempt self-detected stall on CPU [21773.584948] rcu: 7-....: (9552728 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/9551460 fqs=4797176 [21773.595373] rcu: (t=9836792 jiffies g=1309609 q=334126 ncpus=8) [21773.601371] Sending NMI from CPU 7 to CPUs 0: [21773.605724] NMI backtrace for cpu 0 [21773.605726] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21773.605728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21773.605728] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21773.605732] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21773.605733] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21773.605735] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21773.605735] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21773.605736] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21773.605737] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21773.605738] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21773.605739] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21773.605740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21773.605740] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21773.605741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21773.605742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21773.605742] Call Trace: [21773.605743] [21773.605744] ? nmi_cpu_backtrace+0xad/0x120 [21773.605748] ? nmi_cpu_backtrace_handler+0x11/0x20 [21773.605749] ? nmi_handle+0xd4/0x260 [21773.605754] ? default_do_nmi+0x49/0xf0 [21773.605755] ? exc_nmi+0x11e/0x150 [21773.605757] ? end_repeat_nmi+0x16/0x67 [21773.605761] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21773.605763] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21773.605765] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21773.605768] [21773.605768] [21773.605769] do_raw_spin_lock+0xb5/0xc0 [21773.605771] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21773.605788] try_get_cap_refs+0x159/0x880 [ceph] [21773.605802] __ceph_get_caps+0x4a1/0x760 [ceph] [21773.605814] ? __file_remove_privs+0x89/0x110 [21773.605816] ? wake_bit_function+0x70/0x70 [21773.605819] ? ceph_write_iter+0x260/0x770 [ceph] [21773.605829] ceph_write_iter+0x260/0x770 [ceph] [21773.605840] ? lock_is_held_type+0xaa/0x120 [21773.605842] vfs_write+0x324/0x4f0 [21773.605847] ksys_pwrite64+0x65/0xa0 [21773.605849] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21773.605852] do_syscall_64+0x38/0x80 [21773.605854] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21773.605856] RIP: 0033:0x7f48f0eeb93b [21773.605857] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21773.605859] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [21773.605860] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21773.605861] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21773.605862] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21773.605862] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21773.605863] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21773.605866] [21773.606723] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21773.910603] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21773.918075] Workqueue: events delayed_work [ceph] [21773.922798] RIP: 0010:lock_acquire+0x85/0x2b0 [21773.927157] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [21773.945893] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [21773.951112] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [21773.958236] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [21773.965360] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [21773.972500] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [21773.979626] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21773.986750] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21773.994827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21774.000564] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21774.007688] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21774.014812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21774.021936] Call Trace: [21774.024383] [21774.026392] ? rcu_dump_cpu_stacks+0xd1/0x180 [21774.030753] ? rcu_sched_clock_irq+0x633/0x1390 [21774.035286] ? lock_release+0xc5/0x260 [21774.039037] ? lock_is_held_type+0xaa/0x120 [21774.043227] ? lock_is_held_type+0xaa/0x120 [21774.047412] ? tick_sched_do_timer+0xa0/0xa0 [21774.051683] ? rcu_read_lock_sched_held+0x56/0x90 [21774.056390] ? tick_sched_do_timer+0xa0/0xa0 [21774.060662] ? update_process_times+0x5f/0x90 [21774.065020] ? tick_sched_handle.isra.16+0x1e/0x50 [21774.069806] ? tick_sched_timer+0x6f/0x80 [21774.073819] ? __hrtimer_run_queues+0x1d3/0x360 [21774.078359] ? hrtimer_interrupt+0x104/0x240 [21774.082623] ? __local_bh_enable+0x50/0x60 [21774.086727] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21774.091949] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21774.096914] [21774.099011] [21774.101111] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21774.106432] ? lock_acquire+0x85/0x2b0 [21774.110187] ? igrab+0x43/0x50 [21774.113243] ? lock_release+0xc5/0x260 [21774.116992] _raw_spin_lock+0x30/0x40 [21774.120654] ? igrab+0x1d/0x50 [21774.123702] igrab+0x1d/0x50 [21774.126582] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21774.131734] delayed_work+0x1d8/0x280 [ceph] [21774.136025] ? process_scheduled_works+0x22a/0x510 [21774.140812] process_scheduled_works+0x2a4/0x510 [21774.145443] worker_thread+0x188/0x340 [21774.149195] ? rescuer_thread+0x310/0x310 [21774.153223] kthread+0xf5/0x130 [21774.156370] ? kthread_complete_and_exit+0x20/0x20 [21774.161163] ret_from_fork+0x31/0x40 [21774.164741] ? kthread_complete_and_exit+0x20/0x20 [21774.169525] ret_from_fork_asm+0x11/0x20 [21774.173466] [21797.840248] watchdog: BUG: soft lockup - CPU#0 stuck for 9186s! [dbench:78683] [21797.847466] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21797.852249] watchdog: BUG: soft lockup - CPU#7 stuck for 18183s! [kworker/7:4:78808] [21797.902925] irq event stamp: 33344096 [21797.902926] hardirqs last enabled at (33344095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21797.910656] Modules linked in: [21797.914314] hardirqs last disabled at (33344096): [] sysvec_apic_timer_interrupt+0xe/0x90 [21797.924473] ceph [21797.927523] softirqs last enabled at (33344020): [] __do_softirq+0x2be/0x371 [21797.937247] libceph [21797.939170] softirqs last disabled at (33344015): [] irq_exit_rcu+0xb0/0xd0 [21797.947854] dns_resolver fscache [21797.950040] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21797.958549] netfs veth [21797.961861] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21797.970805] nft_chain_nat xt_MASQUERADE [21797.973248] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21797.980719] nf_nat nf_conntrack [21797.984644] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21797.990124] nf_defrag_ipv6 nf_defrag_ipv4 [21797.993347] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21798.012085] nft_compat nf_tables [21798.016175] [21798.016176] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21798.021392] nfnetlink bridge [21798.024703] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21798.026195] stp [21798.033317] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21798.036283] llc xfs [21798.043405] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21798.045242] libcrc32c sunrpc [21798.052367] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21798.054551] intel_rapl_msr [21798.061676] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21798.064647] intel_rapl_common sb_edac [21798.071772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21798.074563] x86_pkg_temp_thermal coretemp [21798.082646] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21798.086393] kvm_intel kvm [21798.092132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21798.096221] irqbypass i2c_i801 [21798.103346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21798.106050] iTCO_wdt iTCO_vendor_support [21798.113174] Call Trace: [21798.113175] [21798.116311] i2c_smbus crct10dif_pclmul [21798.123438] ? watchdog_timer_fn+0x1b4/0x220 [21798.127441] joydev [21798.129884] ? __hrtimer_run_queues+0x1d3/0x360 [21798.131895] crc32_pclmul ipmi_ssif [21798.135729] ? hrtimer_interrupt+0x104/0x240 [21798.139988] ghash_clmulni_intel mei_me [21798.142087] ? __local_bh_enable+0x50/0x60 [21798.146612] mei lpc_ich [21798.150097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21798.154359] sha512_ssse3 acpi_ipmi [21798.158189] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21798.162279] mfd_core wmi [21798.164812] [21798.170028] ipmi_si ipmi_devintf [21798.173512] [21798.173513] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21798.178470] ipmi_msghandler [21798.181090] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21798.183185] ioatdma [21798.186497] do_raw_spin_lock+0xb5/0xc0 [21798.188592] acpi_pad [21798.193897] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21798.196774] acpi_power_meter ixgbe [21798.201651] try_get_cap_refs+0x159/0x880 [ceph] [21798.203830] igb mdio [21798.207664] __ceph_get_caps+0x4a1/0x760 [ceph] [21798.209936] crc32c_intel nvme [21798.214728] ? __file_remove_privs+0x89/0x110 [21798.218208] ptp nvme_core [21798.222818] ? wake_bit_function+0x70/0x70 [21798.225090] i2c_algo_bit pps_core [21798.229617] ? ceph_write_iter+0x260/0x770 [ceph] [21798.232664] dca fuse [21798.237015] ceph_write_iter+0x260/0x770 [ceph] [21798.239718] [21798.239719] irq event stamp: 96855566 [21798.243814] ? lock_is_held_type+0xaa/0x120 [21798.247207] hardirqs last enabled at (96855565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21798.251907] vfs_write+0x324/0x4f0 [21798.254176] hardirqs last disabled at (96855566): [] sysvec_apic_timer_interrupt+0xe/0x90 [21798.258704] ksys_pwrite64+0x65/0xa0 [21798.260190] softirqs last enabled at (96855564): [] __do_softirq+0x2be/0x371 [21798.263846] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21798.268024] softirqs last disabled at (96855559): [] irq_exit_rcu+0xb0/0xd0 [21798.278182] do_syscall_64+0x38/0x80 [21798.281578] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21798.291304] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21798.294874] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21798.303558] RIP: 0033:0x7f48f0eeb93b [21798.308690] Workqueue: events delayed_work [ceph] [21798.317201] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21798.320771] [21798.330149] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21798.335191] RIP: 0010:lock_acquire+0xf0/0x2b0 [21798.342664] ORIG_RAX: 0000000000000012 [21798.346233] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21798.350931] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21798.369669] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21798.371160] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21798.376377] [21798.376378] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21798.380726] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21798.384557] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21798.403295] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21798.410420] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21798.415650] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21798.422770] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21798.424267] [21798.431386] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21798.483455] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21798.491534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21798.497272] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21798.504394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21798.511519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21798.518655] Call Trace: [21798.521098] [21798.523107] ? watchdog_timer_fn+0x1b4/0x220 [21798.527373] ? __hrtimer_run_queues+0x1d3/0x360 [21798.531898] ? hrtimer_interrupt+0x104/0x240 [21798.536167] ? __local_bh_enable+0x50/0x60 [21798.540261] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21798.545485] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21798.550443] [21798.552540] [21798.554654] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21798.559963] ? lock_acquire+0xf0/0x2b0 [21798.563711] ? igrab+0x43/0x50 [21798.566762] ? lock_release+0xc5/0x260 [21798.570508] _raw_spin_lock+0x30/0x40 [21798.574171] ? igrab+0x1d/0x50 [21798.577223] igrab+0x1d/0x50 [21798.580100] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21798.585249] delayed_work+0x1d8/0x280 [ceph] [21798.589531] ? process_scheduled_works+0x22a/0x510 [21798.594314] process_scheduled_works+0x2a4/0x510 [21798.598941] worker_thread+0x188/0x340 [21798.602686] ? rescuer_thread+0x310/0x310 [21798.606690] kthread+0xf5/0x130 [21798.609827] ? kthread_complete_and_exit+0x20/0x20 [21798.614611] ret_from_fork+0x31/0x40 [21798.618182] ? kthread_complete_and_exit+0x20/0x20 [21798.622966] ret_from_fork_asm+0x11/0x20 [21798.626889] [21825.840740] watchdog: BUG: soft lockup - CPU#0 stuck for 9212s! [dbench:78683] [21825.847954] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21825.852740] watchdog: BUG: soft lockup - CPU#7 stuck for 18209s! [kworker/7:4:78808] [21825.903413] irq event stamp: 33402318 [21825.911143] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21825.914801] hardirqs last enabled at (33402317): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21825.921761] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [21825.931918] hardirqs last disabled at (33402318): [] sysvec_apic_timer_interrupt+0xe/0x90 [21825.940168] nft_compat nf_tables nfnetlink bridge stp llc [21825.949893] softirqs last enabled at (33402202): [] __do_softirq+0x2be/0x371 [21825.949896] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [21825.964054] softirqs last disabled at (33402197): [] irq_exit_rcu+0xb0/0xd0 [21825.972738] coretemp kvm_intel kvm irqbypass i2c_i801 [21825.981250] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21825.986380] iTCO_wdt iTCO_vendor_support i2c_smbus [21825.995326] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21826.000196] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [21826.007667] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21826.007669] ghash_clmulni_intel [21826.007670] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21826.013231] mei_me mei [21826.018707] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21826.021932] lpc_ich [21826.040671] [21826.043132] sha512_ssse3 acpi_ipmi [21826.048348] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21826.050531] mfd_core [21826.052024] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21826.055508] wmi ipmi_si [21826.062632] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21826.064903] ipmi_devintf [21826.072027] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21826.074557] ipmi_msghandler ioatdma [21826.081681] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21826.084299] acpi_pad acpi_power_meter [21826.091422] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21826.094994] ixgbe [21826.102135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21826.105880] igb mdio [21826.113957] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21826.115968] crc32c_intel nvme [21826.121706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21826.123976] ptp nvme_core [21826.131100] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21826.134149] i2c_algo_bit pps_core [21826.141275] Call Trace: [21826.141277] [21826.143978] dca fuse [21826.151105] ? watchdog_timer_fn+0x1b4/0x220 [21826.154500] [21826.154500] irq event stamp: 97076494 [21826.156944] ? __hrtimer_run_queues+0x1d3/0x360 [21826.158954] hardirqs last enabled at (97076493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21826.161229] ? hrtimer_interrupt+0x104/0x240 [21826.165488] hardirqs last disabled at (97076494): [] sysvec_apic_timer_interrupt+0xe/0x90 [21826.166979] ? __local_bh_enable+0x50/0x60 [21826.170637] softirqs last enabled at (97076492): [] __do_softirq+0x2be/0x371 [21826.175164] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21826.185318] softirqs last disabled at (97076487): [] irq_exit_rcu+0xb0/0xd0 [21826.189585] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21826.199307] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21826.203414] [21826.212100] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21826.212102] Workqueue: events delayed_work [ceph] [21826.217318] [21826.225828] [21826.225829] RIP: 0010:lock_acquire+0xf0/0x2b0 [21826.230787] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21826.240161] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21826.242266] ? queued_spin_lock_slowpath+0x18/0x2e0 [21826.249731] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21826.254430] do_raw_spin_lock+0xb5/0xc0 [21826.256526] [21826.256527] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21826.258019] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21826.262367] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21826.267674] try_get_cap_refs+0x159/0x880 [ceph] [21826.286420] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21826.291293] __ceph_get_caps+0x4a1/0x760 [ceph] [21826.296506] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21826.300341] ? __file_remove_privs+0x89/0x110 [21826.301827] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21826.308952] ? wake_bit_function+0x70/0x70 [21826.313736] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21826.320862] ? ceph_write_iter+0x260/0x770 [ceph] [21826.325471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21826.332595] ceph_write_iter+0x260/0x770 [ceph] [21826.337137] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21826.344264] ? lock_is_held_type+0xaa/0x120 [21826.348611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21826.355738] vfs_write+0x324/0x4f0 [21826.359825] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21826.367908] ksys_pwrite64+0x65/0xa0 [21826.372600] Call Trace: [21826.372603] [21826.378340] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21826.382863] ? watchdog_timer_fn+0x1b4/0x220 [21826.389988] do_syscall_64+0x38/0x80 [21826.394164] ? __hrtimer_run_queues+0x1d3/0x360 [21826.401290] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21826.404690] ? hrtimer_interrupt+0x104/0x240 [21826.411811] RIP: 0033:0x7f48f0eeb93b [21826.415379] ? __local_bh_enable+0x50/0x60 [21826.417825] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21826.419839] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21826.424965] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21826.429231] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21826.432801] ORIG_RAX: 0000000000000012 [21826.437325] [21826.442369] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21826.446631] [21826.450204] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21826.454293] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21826.473031] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21826.478255] ? lock_acquire+0xf0/0x2b0 [21826.483466] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21826.488425] ? igrab+0x43/0x50 [21826.492255] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21826.494354] ? lock_release+0xc5/0x260 [21826.501481] [21826.503577] _raw_spin_lock+0x30/0x40 [21826.553773] ? igrab+0x1d/0x50 [21826.556825] igrab+0x1d/0x50 [21826.559702] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21826.564851] delayed_work+0x1d8/0x280 [ceph] [21826.569150] ? process_scheduled_works+0x22a/0x510 [21826.573939] process_scheduled_works+0x2a4/0x510 [21826.578555] worker_thread+0x188/0x340 [21826.582304] ? rescuer_thread+0x310/0x310 [21826.586308] kthread+0xf5/0x130 [21826.589455] ? kthread_complete_and_exit+0x20/0x20 [21826.594240] ret_from_fork+0x31/0x40 [21826.597818] ? kthread_complete_and_exit+0x20/0x20 [21826.602602] ret_from_fork_asm+0x11/0x20 [21826.606525] [21853.841231] watchdog: BUG: soft lockup - CPU#0 stuck for 9238s! [dbench:78683] [21853.848441] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21853.853231] watchdog: BUG: soft lockup - CPU#7 stuck for 18235s! [kworker/7:4:78808] [21853.903883] irq event stamp: 33460408 [21853.903884] hardirqs last enabled at (33460407): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21853.911631] Modules linked in: [21853.915289] hardirqs last disabled at (33460408): [] sysvec_apic_timer_interrupt+0xe/0x90 [21853.925446] ceph [21853.928497] softirqs last enabled at (33460340): [] __do_softirq+0x2be/0x371 [21853.938220] libceph dns_resolver [21853.940145] softirqs last disabled at (33460335): [] irq_exit_rcu+0xb0/0xd0 [21853.948828] fscache [21853.952139] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21853.960650] netfs veth [21853.962834] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21853.962835] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21853.971779] nft_chain_nat xt_MASQUERADE [21853.974223] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21853.981694] nf_nat nf_conntrack [21853.987171] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21853.991089] nf_defrag_ipv6 nf_defrag_ipv4 [21854.009825] [21854.009826] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21854.013049] nft_compat nf_tables [21854.018267] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21854.022359] nfnetlink [21854.023849] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21854.030974] bridge stp [21854.034285] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21854.041408] llc xfs [21854.043766] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21854.050905] libcrc32c sunrpc [21854.053352] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21854.060476] intel_rapl_msr [21854.062660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21854.069783] intel_rapl_common sb_edac [21854.072747] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21854.080824] x86_pkg_temp_thermal coretemp [21854.083628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21854.089361] kvm_intel kvm [21854.093105] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21854.100229] irqbypass i2c_i801 [21854.104320] Call Trace: [21854.104321] [21854.111444] iTCO_wdt iTCO_vendor_support [21854.114150] ? watchdog_timer_fn+0x1b4/0x220 [21854.121274] i2c_smbus crct10dif_pclmul [21854.124411] ? __hrtimer_run_queues+0x1d3/0x360 [21854.126854] joydev crc32_pclmul [21854.128868] ? hrtimer_interrupt+0x104/0x240 [21854.132868] ipmi_ssif ghash_clmulni_intel [21854.137133] ? __local_bh_enable+0x50/0x60 [21854.140963] mei_me mei [21854.145490] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21854.148713] lpc_ich sha512_ssse3 [21854.152978] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21854.157066] acpi_ipmi mfd_core [21854.161158] [21854.163602] wmi ipmi_si [21854.168819] [21854.172129] ipmi_devintf ipmi_msghandler [21854.177088] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21854.180224] ioatdma acpi_pad [21854.182327] ? queued_spin_lock_slowpath+0x16/0x2e0 [21854.184853] acpi_power_meter ixgbe [21854.186953] do_raw_spin_lock+0xb5/0xc0 [21854.190954] igb mdio [21854.196259] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21854.199223] crc32c_intel nvme [21854.204094] try_get_cap_refs+0x159/0x880 [ceph] [21854.207577] ptp nvme_core [21854.211413] __ceph_get_caps+0x4a1/0x760 [ceph] [21854.213679] i2c_algo_bit pps_core [21854.218468] ? __file_remove_privs+0x89/0x110 [21854.221514] dca fuse [21854.226124] ? wake_bit_function+0x70/0x70 [21854.228827] [21854.228828] irq event stamp: 97297562 [21854.233356] ? ceph_write_iter+0x260/0x770 [ceph] [21854.236751] hardirqs last enabled at (97297561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21854.241100] ceph_write_iter+0x260/0x770 [ceph] [21854.243370] hardirqs last disabled at (97297562): [] sysvec_apic_timer_interrupt+0xe/0x90 [21854.247467] ? lock_is_held_type+0xaa/0x120 [21854.248953] softirqs last enabled at (97297560): [] __do_softirq+0x2be/0x371 [21854.252630] vfs_write+0x324/0x4f0 [21854.257325] softirqs last disabled at (97297555): [] irq_exit_rcu+0xb0/0xd0 [21854.267487] ksys_pwrite64+0x65/0xa0 [21854.272006] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21854.281731] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21854.285919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21854.294603] do_syscall_64+0x38/0x80 [21854.297999] Workqueue: events delayed_work [ceph] [21854.306511] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21854.310081] [21854.310082] RIP: 0010:lock_acquire+0xf0/0x2b0 [21854.319456] RIP: 0033:0x7f48f0eeb93b [21854.324588] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21854.332058] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21854.335632] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21854.340326] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21854.345371] [21854.345372] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21854.346861] ORIG_RAX: 0000000000000012 [21854.351213] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21854.354783] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21854.373520] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21854.392257] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21854.397475] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21854.402693] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21854.404185] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21854.411307] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21854.415140] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21854.422263] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21854.429387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21854.436518] [21854.443635] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21854.502379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21854.509503] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21854.516637] Call Trace: [21854.519081] [21854.521092] ? watchdog_timer_fn+0x1b4/0x220 [21854.525356] ? __hrtimer_run_queues+0x1d3/0x360 [21854.529884] ? hrtimer_interrupt+0x104/0x240 [21854.534153] ? __local_bh_enable+0x50/0x60 [21854.538246] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21854.543469] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21854.548426] [21854.550525] [21854.552637] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21854.557946] ? lock_acquire+0xf0/0x2b0 [21854.561698] ? igrab+0x43/0x50 [21854.564746] ? lock_release+0xc5/0x260 [21854.568493] _raw_spin_lock+0x30/0x40 [21854.572158] ? igrab+0x1d/0x50 [21854.575207] igrab+0x1d/0x50 [21854.578085] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21854.583236] delayed_work+0x1d8/0x280 [ceph] [21854.587516] ? process_scheduled_works+0x22a/0x510 [21854.592306] process_scheduled_works+0x2a4/0x510 [21854.596923] worker_thread+0x188/0x340 [21854.600671] ? rescuer_thread+0x310/0x310 [21854.604674] kthread+0xf5/0x130 [21854.607812] ? kthread_complete_and_exit+0x20/0x20 [21854.612597] ret_from_fork+0x31/0x40 [21854.616174] ? kthread_complete_and_exit+0x20/0x20 [21854.620959] ret_from_fork_asm+0x11/0x20 [21854.624882] [21881.841718] watchdog: BUG: soft lockup - CPU#0 stuck for 9265s! [dbench:78683] [21881.848938] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21881.853718] watchdog: BUG: soft lockup - CPU#7 stuck for 18262s! [kworker/7:4:78808] [21881.904393] irq event stamp: 33518682 [21881.912128] Modules linked in: ceph libceph dns_resolver fscache netfs veth [21881.915785] hardirqs last enabled at (33518681): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21881.922743] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [21881.932900] hardirqs last disabled at (33518682): [] sysvec_apic_timer_interrupt+0xe/0x90 [21881.932903] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [21881.948275] softirqs last enabled at (33518614): [] __do_softirq+0x2be/0x371 [21881.956353] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [21881.965037] softirqs last disabled at (33518609): [] irq_exit_rcu+0xb0/0xd0 [21881.971903] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [21881.980415] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21881.980416] i2c_i801 [21881.980417] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21881.986584] iTCO_wdt iTCO_vendor_support [21881.995528] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21881.997800] i2c_smbus crct10dif_pclmul [21882.005270] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21882.009274] joydev crc32_pclmul [21882.014750] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21882.018580] ipmi_ssif ghash_clmulni_intel [21882.037321] [21882.040544] mei_me [21882.045760] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21882.049853] mei lpc_ich [21882.051343] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21882.053439] sha512_ssse3 acpi_ipmi [21882.060565] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21882.063111] mfd_core wmi [21882.070236] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21882.073719] ipmi_si [21882.080843] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21882.083462] ipmi_devintf [21882.090586] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21882.092771] ipmi_msghandler [21882.099894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21882.102512] ioatdma acpi_pad [21882.110588] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21882.113468] acpi_power_meter [21882.119204] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21882.122169] ixgbe igb [21882.129291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21882.132256] mdio crc32c_intel [21882.139397] Call Trace: [21882.139399] [21882.141754] nvme ptp [21882.148880] ? watchdog_timer_fn+0x1b4/0x220 [21882.151929] nvme_core i2c_algo_bit [21882.154375] ? __hrtimer_run_queues+0x1d3/0x360 [21882.156402] pps_core dca [21882.158676] ? hrtimer_interrupt+0x104/0x240 [21882.162937] fuse [21882.166420] ? __local_bh_enable+0x50/0x60 [21882.170945] irq event stamp: 97518460 [21882.173565] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21882.177827] hardirqs last enabled at (97518459): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21882.179751] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21882.183840] hardirqs last disabled at (97518460): [] sysvec_apic_timer_interrupt+0xe/0x90 [21882.187499] [21882.192714] softirqs last enabled at (97518458): [] __do_softirq+0x2be/0x371 [21882.202874] [21882.202875] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21882.207830] softirqs last disabled at (97518453): [] irq_exit_rcu+0xb0/0xd0 [21882.217558] ? queued_spin_lock_slowpath+0x18/0x2e0 [21882.219651] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21882.228338] do_raw_spin_lock+0xb5/0xc0 [21882.230433] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21882.235740] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21882.244249] Workqueue: events delayed_work [ceph] [21882.249122] try_get_cap_refs+0x159/0x880 [ceph] [21882.258498] [21882.258499] RIP: 0010:lock_acquire+0xf0/0x2b0 [21882.262334] __ceph_get_caps+0x4a1/0x760 [ceph] [21882.269799] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21882.274586] ? __file_remove_privs+0x89/0x110 [21882.279279] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21882.283892] ? wake_bit_function+0x70/0x70 [21882.285397] [21882.285398] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21882.289751] ? ceph_write_iter+0x260/0x770 [ceph] [21882.294273] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21882.313011] ceph_write_iter+0x260/0x770 [ceph] [21882.317362] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21882.322583] ? lock_is_held_type+0xaa/0x120 [21882.326669] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21882.328163] vfs_write+0x324/0x4f0 [21882.335284] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21882.339988] ksys_pwrite64+0x65/0xa0 [21882.347120] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21882.351648] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21882.358772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21882.362952] do_syscall_64+0x38/0x80 [21882.370073] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21882.373474] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21882.380594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21882.384167] RIP: 0033:0x7f48f0eeb93b [21882.392243] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21882.397376] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21882.403119] Call Trace: [21882.403121] [21882.406690] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21882.413815] ? watchdog_timer_fn+0x1b4/0x220 [21882.418860] ORIG_RAX: 0000000000000012 [21882.425984] ? __hrtimer_run_queues+0x1d3/0x360 [21882.429555] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21882.436682] ? hrtimer_interrupt+0x104/0x240 [21882.455417] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21882.457859] ? __local_bh_enable+0x50/0x60 [21882.459869] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21882.465091] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21882.469353] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21882.473183] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21882.477706] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21882.484831] [21882.489121] [21882.496235] [21882.496237] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21882.543563] ? lock_acquire+0xf0/0x2b0 [21882.547314] ? igrab+0x43/0x50 [21882.550370] ? lock_release+0xc5/0x260 [21882.554127] _raw_spin_lock+0x30/0x40 [21882.557790] ? igrab+0x1d/0x50 [21882.560841] igrab+0x1d/0x50 [21882.563719] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21882.568867] delayed_work+0x1d8/0x280 [ceph] [21882.573151] ? process_scheduled_works+0x22a/0x510 [21882.577940] process_scheduled_works+0x2a4/0x510 [21882.582555] worker_thread+0x188/0x340 [21882.586304] ? rescuer_thread+0x310/0x310 [21882.590307] kthread+0xf5/0x130 [21882.593445] ? kthread_complete_and_exit+0x20/0x20 [21882.598229] ret_from_fork+0x31/0x40 [21882.601808] ? kthread_complete_and_exit+0x20/0x20 [21882.606592] ret_from_fork_asm+0x11/0x20 [21882.610524] [21909.842205] watchdog: BUG: soft lockup - CPU#0 stuck for 9291s! [dbench:78683] [21909.849418] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21909.854204] watchdog: BUG: soft lockup - CPU#7 stuck for 18288s! [kworker/7:4:78808] [21909.904859] irq event stamp: 33576802 [21909.912607] Modules linked in: ceph libceph [21909.916264] hardirqs last enabled at (33576801): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21909.920440] dns_resolver fscache [21909.930600] hardirqs last disabled at (33576802): [] sysvec_apic_timer_interrupt+0xe/0x90 [21909.933910] netfs veth nft_chain_nat [21909.943632] softirqs last enabled at (33576734): [] __do_softirq+0x2be/0x371 [21909.943634] xt_MASQUERADE [21909.943636] softirqs last disabled at (33576729): [] irq_exit_rcu+0xb0/0xd0 [21909.947290] nf_nat nf_conntrack [21909.955975] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21909.958679] nf_defrag_ipv6 nf_defrag_ipv4 [21909.967189] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21909.970414] nft_compat nf_tables [21909.979358] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21909.983448] nfnetlink bridge [21909.990919] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21909.994230] stp llc [21909.999706] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21910.002671] xfs libcrc32c [21910.021409] [21910.023601] sunrpc intel_rapl_msr [21910.028819] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21910.031524] intel_rapl_common sb_edac [21910.033014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21910.036411] x86_pkg_temp_thermal coretemp [21910.043534] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21910.047279] kvm_intel kvm [21910.054403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21910.058495] irqbypass i2c_i801 [21910.065617] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21910.068323] iTCO_wdt [21910.075448] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21910.078600] iTCO_vendor_support i2c_smbus [21910.085725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21910.087997] crct10dif_pclmul joydev [21910.096074] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21910.100166] crc32_pclmul ipmi_ssif [21910.105902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21910.109474] ghash_clmulni_intel mei_me [21910.116602] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21910.120089] mei [21910.127215] Call Trace: [21910.127216] [21910.131044] lpc_ich [21910.138171] ? watchdog_timer_fn+0x1b4/0x220 [21910.140005] sha512_ssse3 acpi_ipmi [21910.142452] ? __hrtimer_run_queues+0x1d3/0x360 [21910.144460] mfd_core wmi [21910.146649] ? hrtimer_interrupt+0x104/0x240 [21910.150908] ipmi_si ipmi_devintf [21910.154392] ? __local_bh_enable+0x50/0x60 [21910.158918] ipmi_msghandler ioatdma [21910.161537] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21910.165799] acpi_pad acpi_power_meter [21910.169110] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21910.173201] ixgbe igb [21910.176772] [21910.181987] mdio crc32c_intel [21910.185733] [21910.185734] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21910.190688] nvme ptp [21910.193053] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21910.195145] nvme_core i2c_algo_bit [21910.198196] do_raw_spin_lock+0xb5/0xc0 [21910.200292] pps_core dca [21910.205606] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21910.207883] fuse [21910.212757] try_get_cap_refs+0x159/0x880 [ceph] [21910.216238] irq event stamp: 97739444 [21910.220075] __ceph_get_caps+0x4a1/0x760 [ceph] [21910.222688] hardirqs last enabled at (97739443): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21910.227475] ? __file_remove_privs+0x89/0x110 [21910.229396] hardirqs last disabled at (97739444): [] sysvec_apic_timer_interrupt+0xe/0x90 [21910.234006] ? wake_bit_function+0x70/0x70 [21910.237665] softirqs last enabled at (97739442): [] __do_softirq+0x2be/0x371 [21910.242190] ? ceph_write_iter+0x260/0x770 [ceph] [21910.252345] softirqs last disabled at (97739437): [] irq_exit_rcu+0xb0/0xd0 [21910.256696] ceph_write_iter+0x260/0x770 [ceph] [21910.266420] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21910.270514] ? lock_is_held_type+0xaa/0x120 [21910.279196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21910.283902] vfs_write+0x324/0x4f0 [21910.292412] Workqueue: events delayed_work [ceph] [21910.296940] ksys_pwrite64+0x65/0xa0 [21910.306314] [21910.306315] RIP: 0010:lock_release+0x5f/0x260 [21910.310493] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21910.317962] Code: 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 <65> 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 [21910.321361] do_syscall_64+0x38/0x80 [21910.326058] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [21910.329629] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21910.331119] [21910.335470] RIP: 0033:0x7f48f0eeb93b [21910.340606] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [21910.359347] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21910.362916] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: 0000000000000001 [21910.368134] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21910.373179] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [21910.374668] ORIG_RAX: 0000000000000012 [21910.378239] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [21910.385363] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21910.404101] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [21910.411226] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21910.416444] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21910.423567] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21910.427397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21910.434523] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21910.441646] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21910.448771] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21910.455894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21910.463977] [21910.471095] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21910.514639] Call Trace: [21910.517083] [21910.519093] ? watchdog_timer_fn+0x1b4/0x220 [21910.523359] ? __hrtimer_run_queues+0x1d3/0x360 [21910.527901] ? hrtimer_interrupt+0x104/0x240 [21910.532162] ? __local_bh_enable+0x50/0x60 [21910.536257] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21910.541481] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21910.546438] [21910.548534] [21910.550633] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21910.555946] ? igrab+0x43/0x50 [21910.558998] ? igrab+0x43/0x50 [21910.562047] ? lock_release+0x5f/0x260 [21910.565790] ? lock_release+0x1d7/0x260 [21910.569622] _raw_spin_unlock+0x1b/0x40 [21910.573453] igrab+0x43/0x50 [21910.576332] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21910.581479] delayed_work+0x1d8/0x280 [ceph] [21910.585761] ? process_scheduled_works+0x22a/0x510 [21910.590550] process_scheduled_works+0x2a4/0x510 [21910.595168] worker_thread+0x188/0x340 [21910.598915] ? rescuer_thread+0x310/0x310 [21910.602919] kthread+0xf5/0x130 [21910.606059] ? kthread_complete_and_exit+0x20/0x20 [21910.610842] ret_from_fork+0x31/0x40 [21910.614420] ? kthread_complete_and_exit+0x20/0x20 [21910.619205] ret_from_fork_asm+0x11/0x20 [21910.623128] [21937.842689] watchdog: BUG: soft lockup - CPU#0 stuck for 9317s! [dbench:78683] [21937.849904] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21937.854688] watchdog: BUG: soft lockup - CPU#7 stuck for 18314s! [kworker/7:4:78808] [21937.905364] irq event stamp: 33634994 [21937.913093] Modules linked in: ceph libceph dns_resolver fscache netfs [21937.916751] hardirqs last enabled at (33634993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21937.923277] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [21937.933436] hardirqs last disabled at (33634994): [] sysvec_apic_timer_interrupt+0xe/0x90 [21937.939519] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [21937.949242] softirqs last enabled at (33634948): [] __do_softirq+0x2be/0x371 [21937.956019] bridge stp llc xfs libcrc32c sunrpc [21937.964703] softirqs last disabled at (33634943): [] irq_exit_rcu+0xb0/0xd0 [21937.969314] intel_rapl_msr intel_rapl_common sb_edac [21937.977827] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21937.977828] x86_pkg_temp_thermal [21937.977829] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21937.982871] coretemp [21937.991814] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21937.995126] kvm_intel [21938.002598] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21938.004866] kvm irqbypass [21938.010342] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21938.012718] i2c_i801 iTCO_wdt [21938.031458] [21938.031458] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21938.034160] iTCO_vendor_support [21938.039378] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21938.042429] i2c_smbus crct10dif_pclmul [21938.043919] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21938.051045] joydev [21938.054268] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21938.061391] crc32_pclmul [21938.065223] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21938.072362] ipmi_ssif ghash_clmulni_intel [21938.074462] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21938.081584] mei_me mei [21938.084202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21938.091328] lpc_ich sha512_ssse3 [21938.095418] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21938.103494] acpi_ipmi mfd_core [21938.105940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21938.111676] wmi [21938.114986] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21938.122112] ipmi_si ipmi_devintf [21938.125249] Call Trace: [21938.132372] ipmi_msghandler [21938.134211] [21938.141334] ioatdma [21938.144646] ? watchdog_timer_fn+0x1b4/0x220 [21938.147088] acpi_pad acpi_power_meter [21938.149968] ? __hrtimer_run_queues+0x1d3/0x360 [21938.151978] ixgbe igb [21938.154164] ? hrtimer_interrupt+0x104/0x240 [21938.158424] mdio crc32c_intel [21938.162170] ? __local_bh_enable+0x50/0x60 [21938.166695] nvme [21938.169054] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21938.173314] ptp nvme_core [21938.176368] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21938.180456] i2c_algo_bit pps_core [21938.182382] [21938.187599] dca fuse [21938.190302] [21938.195260] [21938.195260] irq event stamp: 97960490 [21938.198658] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21938.200754] hardirqs last enabled at (97960489): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21938.203028] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21938.205123] hardirqs last disabled at (97960490): [] sysvec_apic_timer_interrupt+0xe/0x90 [21938.206614] do_raw_spin_lock+0xb5/0xc0 [21938.210271] softirqs last enabled at (97960488): [] __do_softirq+0x2be/0x371 [21938.215576] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21938.225733] softirqs last disabled at (97960483): [] irq_exit_rcu+0xb0/0xd0 [21938.230606] try_get_cap_refs+0x159/0x880 [ceph] [21938.240327] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21938.244164] __ceph_get_caps+0x4a1/0x760 [ceph] [21938.252842] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21938.257630] ? __file_remove_privs+0x89/0x110 [21938.266136] Workqueue: events delayed_work [ceph] [21938.270748] ? wake_bit_function+0x70/0x70 [21938.280124] [21938.284651] ? ceph_write_iter+0x260/0x770 [ceph] [21938.292120] RIP: 0010:lock_release+0x14e/0x260 [21938.296470] ceph_write_iter+0x260/0x770 [ceph] [21938.301168] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [21938.305262] ? lock_is_held_type+0xaa/0x120 [21938.306750] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [21938.311450] vfs_write+0x324/0x4f0 [21938.315883] [21938.315884] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [21938.320413] ksys_pwrite64+0x65/0xa0 [21938.339145] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [21938.343324] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21938.348541] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [21938.351939] do_syscall_64+0x38/0x80 [21938.353429] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [21938.360553] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21938.364123] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [21938.371248] RIP: 0033:0x7f48f0eeb93b [21938.376379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21938.383503] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21938.387090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21938.394215] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21938.399260] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21938.406383] ORIG_RAX: 0000000000000012 [21938.409953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21938.418032] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21938.436768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21938.442506] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21938.447723] Call Trace: [21938.447724] [21938.454848] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21938.458679] ? watchdog_timer_fn+0x1b4/0x220 [21938.465804] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21938.472928] ? __hrtimer_run_queues+0x1d3/0x360 [21938.480051] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21938.487180] ? hrtimer_interrupt+0x104/0x240 [21938.489625] [21938.491630] ? __local_bh_enable+0x50/0x60 [21938.532332] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21938.537557] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21938.542514] [21938.544610] [21938.546709] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21938.552013] ? igrab+0x43/0x50 [21938.555069] ? lock_release+0x14e/0x260 [21938.558923] ? lock_release+0xc5/0x260 [21938.562665] _raw_spin_unlock+0x1b/0x40 [21938.566495] igrab+0x43/0x50 [21938.569384] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21938.574532] delayed_work+0x1d8/0x280 [ceph] [21938.578814] ? process_scheduled_works+0x22a/0x510 [21938.583604] process_scheduled_works+0x2a4/0x510 [21938.588220] worker_thread+0x188/0x340 [21938.591968] ? rescuer_thread+0x310/0x310 [21938.595971] kthread+0xf5/0x130 [21938.599119] ? kthread_complete_and_exit+0x20/0x20 [21938.603910] ret_from_fork+0x31/0x40 [21938.607490] ? kthread_complete_and_exit+0x20/0x20 [21938.612273] ret_from_fork_asm+0x11/0x20 [21938.616205] [21965.843170] watchdog: BUG: soft lockup - CPU#0 stuck for 9343s! [dbench:78683] [21965.850384] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21965.855170] watchdog: BUG: soft lockup - CPU#7 stuck for 18340s! [kworker/7:4:78808] [21965.905823] irq event stamp: 33693124 [21965.905824] hardirqs last enabled at (33693123): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21965.913563] Modules linked in: [21965.917220] hardirqs last disabled at (33693124): [] sysvec_apic_timer_interrupt+0xe/0x90 [21965.927378] ceph [21965.930429] softirqs last enabled at (33693114): [] __do_softirq+0x2be/0x371 [21965.940154] libceph [21965.942077] softirqs last disabled at (33693109): [] irq_exit_rcu+0xb0/0xd0 [21965.950762] dns_resolver fscache [21965.952947] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21965.961456] netfs veth [21965.964767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21965.973713] nft_chain_nat xt_MASQUERADE [21965.976156] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [21965.983627] nf_nat [21965.987544] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [21965.993037] nf_conntrack [21965.995137] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21966.013874] nf_defrag_ipv6 nf_defrag_ipv4 [21966.016490] [21966.016491] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21966.021709] nft_compat nf_tables [21966.025800] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21966.027291] nfnetlink bridge [21966.034413] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21966.037724] stp llc [21966.044858] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21966.047820] xfs libcrc32c [21966.054963] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21966.057146] sunrpc intel_rapl_msr [21966.064272] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21966.066974] intel_rapl_common [21966.074099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21966.077496] sb_edac x86_pkg_temp_thermal [21966.085574] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21966.088625] coretemp kvm_intel [21966.094362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21966.098365] kvm irqbypass [21966.105492] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21966.108627] i2c_i801 iTCO_wdt [21966.115753] Call Trace: [21966.118458] iTCO_vendor_support i2c_smbus [21966.125581] [21966.128632] crct10dif_pclmul joydev [21966.131076] ? watchdog_timer_fn+0x1b4/0x220 [21966.135166] crc32_pclmul [21966.137178] ? __hrtimer_run_queues+0x1d3/0x360 [21966.140748] ipmi_ssif ghash_clmulni_intel [21966.145014] ? hrtimer_interrupt+0x104/0x240 [21966.147630] mei_me [21966.152153] ? __local_bh_enable+0x50/0x60 [21966.156244] mei [21966.160510] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21966.162605] lpc_ich [21966.166695] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21966.168533] sha512_ssse3 acpi_ipmi [21966.173750] [21966.175933] mfd_core wmi [21966.180893] [21966.180894] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21966.184375] ipmi_si ipmi_devintf [21966.186479] ? queued_spin_lock_slowpath+0x16/0x2e0 [21966.189092] ipmi_msghandler ioatdma [21966.191191] do_raw_spin_lock+0xb5/0xc0 [21966.196493] acpi_pad acpi_power_meter [21966.199805] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21966.204673] ixgbe igb [21966.208246] try_get_cap_refs+0x159/0x880 [ceph] [21966.212076] mdio crc32c_intel [21966.215825] __ceph_get_caps+0x4a1/0x760 [ceph] [21966.220602] nvme ptp [21966.222964] ? __file_remove_privs+0x89/0x110 [21966.227571] nvme_core i2c_algo_bit [21966.230623] ? wake_bit_function+0x70/0x70 [21966.235147] pps_core dca [21966.237419] ? ceph_write_iter+0x260/0x770 [ceph] [21966.241767] fuse [21966.245253] ceph_write_iter+0x260/0x770 [ceph] [21966.249344] irq event stamp: 98181254 [21966.251964] ? lock_is_held_type+0xaa/0x120 [21966.256656] hardirqs last enabled at (98181253): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21966.258583] vfs_write+0x324/0x4f0 [21966.263107] hardirqs last disabled at (98181254): [] sysvec_apic_timer_interrupt+0xe/0x90 [21966.266767] ksys_pwrite64+0x65/0xa0 [21966.270940] softirqs last enabled at (98181252): [] __do_softirq+0x2be/0x371 [21966.281100] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21966.284496] softirqs last disabled at (98181247): [] irq_exit_rcu+0xb0/0xd0 [21966.294221] do_syscall_64+0x38/0x80 [21966.297790] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21966.306474] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21966.311604] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21966.320116] RIP: 0033:0x7f48f0eeb93b [21966.323686] Workqueue: events delayed_work [ceph] [21966.333065] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21966.338107] [21966.338108] RIP: 0010:lock_acquire+0xf0/0x2b0 [21966.345579] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21966.349150] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21966.353861] ORIG_RAX: 0000000000000012 [21966.372603] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21966.374094] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21966.378443] [21966.378444] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21966.383661] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21966.402399] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21966.406229] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21966.411446] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21966.418574] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21966.420063] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21966.427185] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21966.434310] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21966.441439] [21966.448575] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21966.494457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21966.500195] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21966.507320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21966.514444] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21966.521576] Call Trace: [21966.524021] [21966.526032] ? watchdog_timer_fn+0x1b4/0x220 [21966.530296] ? __hrtimer_run_queues+0x1d3/0x360 [21966.534822] ? hrtimer_interrupt+0x104/0x240 [21966.539085] ? __local_bh_enable+0x50/0x60 [21966.543178] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21966.548400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21966.553358] [21966.555454] [21966.557553] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21966.562879] ? lock_acquire+0xf0/0x2b0 [21966.566629] ? igrab+0x43/0x50 [21966.569679] ? lock_release+0xc5/0x260 [21966.573424] _raw_spin_lock+0x30/0x40 [21966.577087] ? igrab+0x1d/0x50 [21966.580139] igrab+0x1d/0x50 [21966.583017] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21966.588168] delayed_work+0x1d8/0x280 [ceph] [21966.592448] ? process_scheduled_works+0x22a/0x510 [21966.597237] process_scheduled_works+0x2a4/0x510 [21966.601869] worker_thread+0x188/0x340 [21966.605620] ? rescuer_thread+0x310/0x310 [21966.609622] kthread+0xf5/0x130 [21966.612761] ? kthread_complete_and_exit+0x20/0x20 [21966.617545] ret_from_fork+0x31/0x40 [21966.621115] ? kthread_complete_and_exit+0x20/0x20 [21966.625900] ret_from_fork_asm+0x11/0x20 [21966.629822] [21969.180594] rcu: INFO: rcu_preempt self-detected stall on CPU [21969.186358] rcu: 7-....: (9742360 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/9741091 fqs=4893064 [21969.196784] rcu: (t=10032390 jiffies g=1309609 q=338671 ncpus=8) [21969.202869] Sending NMI from CPU 7 to CPUs 0: [21969.207220] NMI backtrace for cpu 0 [21969.207222] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21969.207224] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21969.207224] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [21969.207227] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [21969.207229] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21969.207230] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21969.207231] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21969.207232] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21969.207232] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21969.207233] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21969.207234] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21969.207235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21969.207236] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21969.207236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21969.207237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21969.207238] Call Trace: [21969.207238] [21969.207239] ? nmi_cpu_backtrace+0xad/0x120 [21969.207243] ? nmi_cpu_backtrace_handler+0x11/0x20 [21969.207244] ? nmi_handle+0xd4/0x260 [21969.207249] ? default_do_nmi+0x49/0xf0 [21969.207250] ? exc_nmi+0x11e/0x150 [21969.207252] ? end_repeat_nmi+0x16/0x67 [21969.207255] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21969.207258] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21969.207260] ? queued_spin_lock_slowpath+0x2d/0x2e0 [21969.207263] [21969.207263] [21969.207264] do_raw_spin_lock+0xb5/0xc0 [21969.207266] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21969.207279] try_get_cap_refs+0x159/0x880 [ceph] [21969.207293] __ceph_get_caps+0x4a1/0x760 [ceph] [21969.207306] ? __file_remove_privs+0x89/0x110 [21969.207307] ? wake_bit_function+0x70/0x70 [21969.207310] ? ceph_write_iter+0x260/0x770 [ceph] [21969.207320] ceph_write_iter+0x260/0x770 [ceph] [21969.207331] ? lock_is_held_type+0xaa/0x120 [21969.207333] vfs_write+0x324/0x4f0 [21969.207337] ksys_pwrite64+0x65/0xa0 [21969.207339] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21969.207342] do_syscall_64+0x38/0x80 [21969.207344] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21969.207346] RIP: 0033:0x7f48f0eeb93b [21969.207347] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21969.207348] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [21969.207349] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21969.207350] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21969.207350] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21969.207351] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21969.207352] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21969.207355] [21969.208218] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21969.512006] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21969.519475] Workqueue: events delayed_work [ceph] [21969.524200] RIP: 0010:lock_acquire+0xf0/0x2b0 [21969.528557] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [21969.547296] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [21969.552523] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [21969.559647] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [21969.566769] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [21969.573909] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [21969.581036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21969.588161] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21969.596238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21969.601974] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21969.609100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21969.616224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21969.623347] Call Trace: [21969.625792] [21969.627803] ? rcu_dump_cpu_stacks+0xd1/0x180 [21969.632164] ? rcu_sched_clock_irq+0x633/0x1390 [21969.636695] ? lock_release+0xc5/0x260 [21969.640438] ? lock_is_held_type+0xaa/0x120 [21969.644632] ? lock_is_held_type+0xaa/0x120 [21969.648813] ? tick_sched_do_timer+0xa0/0xa0 [21969.653084] ? rcu_read_lock_sched_held+0x56/0x90 [21969.657794] ? tick_sched_do_timer+0xa0/0xa0 [21969.662064] ? update_process_times+0x5f/0x90 [21969.666422] ? tick_sched_handle.isra.16+0x1e/0x50 [21969.671215] ? tick_sched_timer+0x6f/0x80 [21969.675221] ? __hrtimer_run_queues+0x1d3/0x360 [21969.679760] ? hrtimer_interrupt+0x104/0x240 [21969.684023] ? __local_bh_enable+0x50/0x60 [21969.688120] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21969.693342] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21969.698309] [21969.700404] [21969.702502] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21969.707826] ? lock_acquire+0xf0/0x2b0 [21969.711578] ? igrab+0x43/0x50 [21969.714638] ? lock_release+0xc5/0x260 [21969.718391] _raw_spin_lock+0x30/0x40 [21969.722054] ? igrab+0x1d/0x50 [21969.725106] igrab+0x1d/0x50 [21969.727983] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21969.733137] delayed_work+0x1d8/0x280 [ceph] [21969.737425] ? process_scheduled_works+0x22a/0x510 [21969.742213] process_scheduled_works+0x2a4/0x510 [21969.746835] worker_thread+0x188/0x340 [21969.750588] ? rescuer_thread+0x310/0x310 [21969.754598] kthread+0xf5/0x130 [21969.757736] ? kthread_complete_and_exit+0x20/0x20 [21969.762521] ret_from_fork+0x31/0x40 [21969.766100] ? kthread_complete_and_exit+0x20/0x20 [21969.770884] ret_from_fork_asm+0x11/0x20 [21969.774824] [21993.843652] watchdog: BUG: soft lockup - CPU#0 stuck for 9369s! [dbench:78683] [21993.850871] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [21993.855652] watchdog: BUG: soft lockup - CPU#7 stuck for 18366s! [kworker/7:4:78808] [21993.906313] irq event stamp: 33751360 [21993.906314] hardirqs last enabled at (33751359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21993.914061] Modules linked in: [21993.917716] hardirqs last disabled at (33751360): [] sysvec_apic_timer_interrupt+0xe/0x90 [21993.927874] ceph [21993.930927] softirqs last enabled at (33751306): [] __do_softirq+0x2be/0x371 [21993.940650] libceph [21993.942573] softirqs last disabled at (33751301): [] irq_exit_rcu+0xb0/0xd0 [21993.951259] dns_resolver fscache [21993.953442] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21993.961952] netfs [21993.965263] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21993.974209] veth nft_chain_nat [21993.976219] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [21993.983689] xt_MASQUERADE [21993.986827] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [21993.992305] nf_nat nf_conntrack nf_defrag_ipv6 [21993.995044] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [21994.013781] nf_defrag_ipv4 nft_compat [21994.018306] [21994.018306] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [21994.023539] nf_tables nfnetlink [21994.027284] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [21994.028775] bridge [21994.035899] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [21994.039121] stp [21994.046247] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [21994.048343] llc xfs [21994.055469] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [21994.057305] libcrc32c sunrpc [21994.064448] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [21994.066631] intel_rapl_msr [21994.073756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21994.076719] intel_rapl_common [21994.084796] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [21994.087587] sb_edac [21994.093338] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21994.096393] x86_pkg_temp_thermal [21994.103516] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21994.105701] coretemp kvm_intel [21994.112825] Call Trace: [21994.112827] [21994.116136] kvm [21994.123262] ? watchdog_timer_fn+0x1b4/0x220 [21994.126398] irqbypass i2c_i801 [21994.128844] ? __hrtimer_run_queues+0x1d3/0x360 [21994.130853] iTCO_wdt [21994.132693] ? hrtimer_interrupt+0x104/0x240 [21994.136953] iTCO_vendor_support [21994.140091] ? __local_bh_enable+0x50/0x60 [21994.144614] i2c_smbus [21994.146888] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21994.151148] crct10dif_pclmul joydev [21994.154375] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21994.158464] crc32_pclmul ipmi_ssif [21994.160822] [21994.166050] ghash_clmulni_intel [21994.169618] [21994.174577] mei_me [21994.178059] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21994.180157] mei lpc_ich [21994.183387] ? queued_spin_lock_slowpath+0x18/0x2e0 [21994.185480] sha512_ssse3 acpi_ipmi [21994.187578] do_raw_spin_lock+0xb5/0xc0 [21994.192881] mfd_core [21994.195414] __cap_delay_requeue+0xa7/0x1d0 [ceph] [21994.200283] wmi [21994.203770] try_get_cap_refs+0x159/0x880 [ceph] [21994.207598] ipmi_si ipmi_devintf [21994.209873] __ceph_get_caps+0x4a1/0x760 [ceph] [21994.214652] ipmi_msghandler [21994.216492] ? __file_remove_privs+0x89/0x110 [21994.221099] ioatdma acpi_pad [21994.224412] ? wake_bit_function+0x70/0x70 [21994.228935] acpi_power_meter ixgbe [21994.231813] ? ceph_write_iter+0x260/0x770 [ceph] [21994.236162] igb mdio [21994.239127] ceph_write_iter+0x260/0x770 [ceph] [21994.243218] crc32c_intel nvme [21994.246704] ? lock_is_held_type+0xaa/0x120 [21994.251398] ptp nvme_core [21994.253671] vfs_write+0x324/0x4f0 [21994.258193] i2c_algo_bit pps_core [21994.261250] ksys_pwrite64+0x65/0xa0 [21994.265422] dca [21994.268128] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [21994.271523] fuse [21994.274921] do_syscall_64+0x38/0x80 [21994.278491] irq event stamp: 98397232 [21994.280345] entry_SYSCALL_64_after_hwframe+0x63/0xcd [21994.285477] hardirqs last enabled at (98397231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21994.287401] RIP: 0033:0x7f48f0eeb93b [21994.290973] hardirqs last disabled at (98397232): [] sysvec_apic_timer_interrupt+0xe/0x90 [21994.294630] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [21994.299673] softirqs last enabled at (98397230): [] __do_softirq+0x2be/0x371 [21994.309831] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [21994.313401] softirqs last disabled at (98397225): [] irq_exit_rcu+0xb0/0xd0 [21994.323126] ORIG_RAX: 0000000000000012 [21994.341863] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [21994.350548] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [21994.355766] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [21994.364276] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [21994.368108] Workqueue: events delayed_work [ceph] [21994.377483] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [21994.384609] [21994.392080] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [21994.399202] RIP: 0010:lock_acquire+0x85/0x2b0 [21994.403901] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [21994.411024] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [21994.412520] [21994.419640] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [21994.457272] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [21994.464394] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [21994.471520] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [21994.478644] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [21994.485768] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [21994.492892] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [21994.500968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21994.506707] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [21994.513831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21994.520955] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [21994.528079] Call Trace: [21994.530523] [21994.532535] ? watchdog_timer_fn+0x1b4/0x220 [21994.536799] ? __hrtimer_run_queues+0x1d3/0x360 [21994.541339] ? hrtimer_interrupt+0x104/0x240 [21994.545604] ? __local_bh_enable+0x50/0x60 [21994.549697] ? __sysvec_apic_timer_interrupt+0x61/0x160 [21994.554920] ? sysvec_apic_timer_interrupt+0x6d/0x90 [21994.559879] [21994.561974] [21994.564072] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [21994.569380] ? lock_acquire+0x85/0x2b0 [21994.573131] ? igrab+0x43/0x50 [21994.576180] ? lock_release+0xc5/0x260 [21994.579925] _raw_spin_lock+0x30/0x40 [21994.583583] ? igrab+0x1d/0x50 [21994.586631] igrab+0x1d/0x50 [21994.589512] ceph_check_delayed_caps+0xb7/0x270 [ceph] [21994.594660] delayed_work+0x1d8/0x280 [ceph] [21994.598943] ? process_scheduled_works+0x22a/0x510 [21994.603732] process_scheduled_works+0x2a4/0x510 [21994.608348] worker_thread+0x188/0x340 [21994.612096] ? rescuer_thread+0x310/0x310 [21994.616098] kthread+0xf5/0x130 [21994.619239] ? kthread_complete_and_exit+0x20/0x20 [21994.624031] ret_from_fork+0x31/0x40 [21994.627627] ? kthread_complete_and_exit+0x20/0x20 [21994.632411] ret_from_fork_asm+0x11/0x20 [21994.636343] [22021.844135] watchdog: BUG: soft lockup - CPU#0 stuck for 9395s! [dbench:78683] [22021.851350] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22021.856136] watchdog: BUG: soft lockup - CPU#7 stuck for 18392s! [kworker/7:4:78808] [22021.906806] irq event stamp: 33809640 [22021.906807] hardirqs last enabled at (33809639): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22021.914540] Modules linked in: [22021.918198] hardirqs last disabled at (33809640): [] sysvec_apic_timer_interrupt+0xe/0x90 [22021.928354] ceph [22021.931406] softirqs last enabled at (33809614): [] __do_softirq+0x2be/0x371 [22021.941129] libceph [22021.943053] softirqs last disabled at (33809609): [] irq_exit_rcu+0xb0/0xd0 [22021.951737] dns_resolver fscache [22021.953921] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22021.962431] netfs veth [22021.965742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22021.974687] nft_chain_nat xt_MASQUERADE [22021.977131] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22021.984601] nf_nat nf_conntrack [22021.988531] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22021.994005] nf_defrag_ipv6 nf_defrag_ipv4 [22021.997230] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22022.015966] nft_compat nf_tables [22022.020058] [22022.020059] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22022.025274] nfnetlink bridge [22022.028586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22022.030076] stp llc [22022.037200] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22022.040164] xfs [22022.047290] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22022.049474] libcrc32c sunrpc [22022.056597] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22022.058433] intel_rapl_msr intel_rapl_common [22022.065558] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22022.068534] sb_edac [22022.075656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22022.080006] x86_pkg_temp_thermal coretemp [22022.088084] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22022.090268] kvm_intel kvm [22022.096005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22022.100095] irqbypass [22022.107219] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22022.109924] i2c_i801 iTCO_wdt [22022.117049] Call Trace: [22022.119404] iTCO_vendor_support i2c_smbus [22022.126540] [22022.129588] crct10dif_pclmul joydev [22022.132034] ? watchdog_timer_fn+0x1b4/0x220 [22022.136123] crc32_pclmul ipmi_ssif [22022.138135] ? __hrtimer_run_queues+0x1d3/0x360 [22022.141705] ghash_clmulni_intel mei_me [22022.145973] ? hrtimer_interrupt+0x104/0x240 [22022.149454] mei [22022.153978] ? __local_bh_enable+0x50/0x60 [22022.157816] lpc_ich sha512_ssse3 [22022.162084] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22022.163919] acpi_ipmi mfd_core [22022.168009] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22022.171321] wmi ipmi_si [22022.176537] [22022.179674] ipmi_devintf [22022.184633] [22022.184634] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22022.187161] ipmi_msghandler ioatdma [22022.189265] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22022.191878] acpi_pad acpi_power_meter [22022.193977] do_raw_spin_lock+0xb5/0xc0 [22022.199279] ixgbe igb mdio [22022.202851] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22022.207720] crc32c_intel nvme [22022.211468] try_get_cap_refs+0x159/0x880 [ceph] [22022.215296] ptp nvme_core [22022.218091] __ceph_get_caps+0x4a1/0x760 [ceph] [22022.222871] i2c_algo_bit pps_core [22022.225924] ? __file_remove_privs+0x89/0x110 [22022.230534] dca [22022.233235] ? wake_bit_function+0x70/0x70 [22022.237761] fuse [22022.241159] ? ceph_write_iter+0x260/0x770 [ceph] [22022.245509] irq event stamp: 98618140 [22022.247364] ceph_write_iter+0x260/0x770 [ceph] [22022.251454] hardirqs last enabled at (98618139): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22022.253381] ? lock_is_held_type+0xaa/0x120 [22022.258074] hardirqs last disabled at (98618140): [] sysvec_apic_timer_interrupt+0xe/0x90 [22022.261735] vfs_write+0x324/0x4f0 [22022.266257] softirqs last enabled at (98618138): [] __do_softirq+0x2be/0x371 [22022.276417] ksys_pwrite64+0x65/0xa0 [22022.280592] softirqs last disabled at (98618133): [] irq_exit_rcu+0xb0/0xd0 [22022.290315] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22022.293713] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22022.302398] do_syscall_64+0x38/0x80 [22022.305967] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22022.314479] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22022.319609] Workqueue: events delayed_work [ceph] [22022.328986] RIP: 0033:0x7f48f0eeb93b [22022.332559] [22022.340028] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22022.345072] RIP: 0010:lock_acquire+0xf0/0x2b0 [22022.349769] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22022.353340] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22022.354830] ORIG_RAX: 0000000000000012 [22022.373568] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22022.377919] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22022.383136] [22022.383137] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22022.401874] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22022.405704] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22022.410923] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22022.418046] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22022.419538] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22022.426661] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22022.433787] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22022.440910] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22022.448041] [22022.455159] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22022.493926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22022.499663] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22022.506787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22022.513911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22022.521036] Call Trace: [22022.523481] [22022.525490] ? watchdog_timer_fn+0x1b4/0x220 [22022.529754] ? __hrtimer_run_queues+0x1d3/0x360 [22022.534281] ? hrtimer_interrupt+0x104/0x240 [22022.538551] ? __local_bh_enable+0x50/0x60 [22022.542643] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22022.547868] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22022.552827] [22022.554923] [22022.557020] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22022.562328] ? lock_acquire+0xf0/0x2b0 [22022.566079] ? igrab+0x43/0x50 [22022.569137] ? lock_release+0xc5/0x260 [22022.572882] _raw_spin_lock+0x30/0x40 [22022.576543] ? igrab+0x1d/0x50 [22022.579597] igrab+0x1d/0x50 [22022.582476] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22022.587625] delayed_work+0x1d8/0x280 [ceph] [22022.591907] ? process_scheduled_works+0x22a/0x510 [22022.596696] process_scheduled_works+0x2a4/0x510 [22022.601312] worker_thread+0x188/0x340 [22022.605061] ? rescuer_thread+0x310/0x310 [22022.609065] kthread+0xf5/0x130 [22022.612202] ? kthread_complete_and_exit+0x20/0x20 [22022.616987] ret_from_fork+0x31/0x40 [22022.620558] ? kthread_complete_and_exit+0x20/0x20 [22022.625340] ret_from_fork_asm+0x11/0x20 [22022.629263] [22049.844620] watchdog: BUG: soft lockup - CPU#0 stuck for 9421s! [dbench:78683] [22049.851839] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22049.856619] watchdog: BUG: soft lockup - CPU#7 stuck for 18418s! [kworker/7:4:78808] [22049.907301] irq event stamp: 33867880 [22049.907302] hardirqs last enabled at (33867879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22049.915037] Modules linked in: [22049.918692] hardirqs last disabled at (33867880): [] sysvec_apic_timer_interrupt+0xe/0x90 [22049.928849] ceph [22049.931902] softirqs last enabled at (33867822): [] __do_softirq+0x2be/0x371 [22049.941626] libceph [22049.943548] softirqs last disabled at (33867817): [] irq_exit_rcu+0xb0/0xd0 [22049.952233] dns_resolver fscache [22049.954419] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22049.962930] netfs [22049.966238] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22049.975183] veth nft_chain_nat [22049.977195] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22049.984665] xt_MASQUERADE nf_nat [22049.987802] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22049.993296] nf_conntrack nf_defrag_ipv6 [22049.996608] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22050.015346] nf_defrag_ipv4 [22050.019261] [22050.024481] nft_compat nf_tables [22050.027271] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22050.028778] nfnetlink bridge [22050.032089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22050.039213] stp [22050.042177] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22050.049301] llc xfs [22050.051139] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22050.058265] libcrc32c sunrpc [22050.060448] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22050.067573] intel_rapl_msr [22050.070535] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22050.077660] intel_rapl_common [22050.080451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22050.088529] sb_edac x86_pkg_temp_thermal [22050.091578] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22050.097317] coretemp kvm_intel [22050.101320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22050.108445] kvm [22050.111582] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22050.118705] irqbypass i2c_i801 [22050.120543] Call Trace: [22050.120546] [22050.127667] iTCO_wdt iTCO_vendor_support [22050.130808] ? watchdog_timer_fn+0x1b4/0x220 [22050.133248] i2c_smbus [22050.135260] ? __hrtimer_run_queues+0x1d3/0x360 [22050.139264] crct10dif_pclmul joydev [22050.143532] ? hrtimer_interrupt+0x104/0x240 [22050.145886] crc32_pclmul [22050.150410] ? __local_bh_enable+0x50/0x60 [22050.153980] ipmi_ssif ghash_clmulni_intel [22050.158246] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22050.160861] mei_me [22050.164952] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22050.169042] mei lpc_ich [22050.174259] [22050.176358] sha512_ssse3 acpi_ipmi [22050.181314] [22050.183845] mfd_core wmi [22050.185943] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22050.189426] ipmi_si ipmi_devintf [22050.191530] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22050.194143] ipmi_msghandler ioatdma [22050.199448] do_raw_spin_lock+0xb5/0xc0 [22050.202756] acpi_pad acpi_power_meter [22050.207628] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22050.211197] ixgbe igb [22050.215032] try_get_cap_refs+0x159/0x880 [ceph] [22050.218773] mdio crc32c_intel [22050.223562] __ceph_get_caps+0x4a1/0x760 [ceph] [22050.225914] nvme ptp [22050.230530] ? __file_remove_privs+0x89/0x110 [22050.233577] nvme_core i2c_algo_bit [22050.238101] ? wake_bit_function+0x70/0x70 [22050.240370] pps_core dca [22050.244725] ? ceph_write_iter+0x260/0x770 [ceph] [22050.248207] fuse [22050.252300] ceph_write_iter+0x260/0x770 [ceph] [22050.254915] [22050.254916] irq event stamp: 98839010 [22050.259614] ? lock_is_held_type+0xaa/0x120 [22050.261534] hardirqs last enabled at (98839009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22050.266061] vfs_write+0x324/0x4f0 [22050.267550] hardirqs last disabled at (98839010): [] sysvec_apic_timer_interrupt+0xe/0x90 [22050.271214] ksys_pwrite64+0x65/0xa0 [22050.275385] softirqs last enabled at (98839008): [] __do_softirq+0x2be/0x371 [22050.285543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22050.288940] softirqs last disabled at (98839003): [] irq_exit_rcu+0xb0/0xd0 [22050.298665] do_syscall_64+0x38/0x80 [22050.302235] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22050.310920] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22050.316048] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22050.324561] RIP: 0033:0x7f48f0eeb93b [22050.328130] Workqueue: events delayed_work [ceph] [22050.337509] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22050.342552] [22050.342552] RIP: 0010:lock_release+0x11/0x260 [22050.350024] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22050.353593] Code: 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd <53> 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f [22050.358304] ORIG_RAX: 0000000000000012 [22050.377046] RSP: 0018:ffffc90002117d68 EFLAGS: 00000286 [22050.378537] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22050.382888] [22050.382888] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [22050.388104] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22050.406844] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [22050.410675] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22050.415890] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [22050.423023] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22050.424514] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [22050.431638] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22050.438764] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22050.445894] [22050.453022] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22050.498902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22050.504641] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22050.511765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22050.518888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22050.526025] Call Trace: [22050.528465] [22050.530478] ? watchdog_timer_fn+0x1b4/0x220 [22050.534741] ? __hrtimer_run_queues+0x1d3/0x360 [22050.539268] ? hrtimer_interrupt+0x104/0x240 [22050.543537] ? __local_bh_enable+0x50/0x60 [22050.547630] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22050.552853] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22050.557811] [22050.559907] [22050.562022] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22050.567327] ? igrab+0x43/0x50 [22050.570380] ? igrab+0x43/0x50 [22050.573430] ? lock_release+0x11/0x260 [22050.577177] _raw_spin_unlock+0x1b/0x40 [22050.581026] igrab+0x43/0x50 [22050.583908] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22050.589059] delayed_work+0x1d8/0x280 [ceph] [22050.593339] ? process_scheduled_works+0x22a/0x510 [22050.598128] process_scheduled_works+0x2a4/0x510 [22050.602746] worker_thread+0x188/0x340 [22050.606493] ? rescuer_thread+0x310/0x310 [22050.610498] kthread+0xf5/0x130 [22050.613635] ? kthread_complete_and_exit+0x20/0x20 [22050.618420] ret_from_fork+0x31/0x40 [22050.621998] ? kthread_complete_and_exit+0x20/0x20 [22050.626798] ret_from_fork_asm+0x11/0x20 [22050.630722] [22077.845104] watchdog: BUG: soft lockup - CPU#0 stuck for 9447s! [dbench:78683] [22077.852316] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22077.857104] watchdog: BUG: soft lockup - CPU#7 stuck for 18444s! [kworker/7:4:78808] [22077.907757] irq event stamp: 33926066 [22077.907759] hardirqs last enabled at (33926065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22077.915500] Modules linked in: [22077.919154] hardirqs last disabled at (33926066): [] sysvec_apic_timer_interrupt+0xe/0x90 [22077.929311] ceph [22077.932362] softirqs last enabled at (33926008): [] __do_softirq+0x2be/0x371 [22077.942086] libceph [22077.944012] softirqs last disabled at (33926003): [] irq_exit_rcu+0xb0/0xd0 [22077.952694] dns_resolver [22077.954878] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22077.963389] fscache netfs [22077.966008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22077.974952] veth nft_chain_nat [22077.977656] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22077.985126] xt_MASQUERADE nf_nat [22077.988265] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22077.993741] nf_conntrack nf_defrag_ipv6 [22077.997051] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22078.015790] nf_defrag_ipv4 [22078.019706] [22078.019706] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22078.024926] nft_compat [22078.027716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22078.029207] nf_tables [22078.036329] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22078.038787] nfnetlink bridge [22078.045914] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22078.048272] stp [22078.055397] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22078.058361] llc xfs [22078.065501] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22078.067339] libcrc32c [22078.074465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22078.076649] sunrpc [22078.084726] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22078.087083] intel_rapl_msr [22078.092821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22078.094917] intel_rapl_common sb_edac [22078.102043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22078.104833] x86_pkg_temp_thermal coretemp [22078.111958] Call Trace: [22078.111959] [22078.115701] kvm_intel kvm [22078.122825] ? watchdog_timer_fn+0x1b4/0x220 [22078.126917] irqbypass i2c_i801 [22078.129361] ? __hrtimer_run_queues+0x1d3/0x360 [22078.131370] iTCO_wdt iTCO_vendor_support [22078.134079] ? hrtimer_interrupt+0x104/0x240 [22078.138340] i2c_smbus crct10dif_pclmul [22078.141477] ? __local_bh_enable+0x50/0x60 [22078.146017] joydev crc32_pclmul [22078.150025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22078.154285] ipmi_ssif ghash_clmulni_intel [22078.158116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22078.162207] mei_me mei [22078.165432] [22078.170649] lpc_ich sha512_ssse3 [22078.174739] [22078.174741] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22078.179697] acpi_ipmi mfd_core [22078.182145] ? queued_spin_lock_slowpath+0x18/0x2e0 [22078.184239] wmi ipmi_si [22078.187552] do_raw_spin_lock+0xb5/0xc0 [22078.189647] ipmi_devintf ipmi_msghandler [22078.194951] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22078.198088] ioatdma acpi_pad [22078.202962] try_get_cap_refs+0x159/0x880 [ceph] [22078.205504] acpi_power_meter [22078.209341] __ceph_get_caps+0x4a1/0x760 [ceph] [22078.213340] ixgbe igb [22078.218130] ? __file_remove_privs+0x89/0x110 [22078.221089] mdio crc32c_intel [22078.225700] ? wake_bit_function+0x70/0x70 [22078.228665] nvme [22078.233190] ? ceph_write_iter+0x260/0x770 [ceph] [22078.235545] ptp nvme_core [22078.239896] ceph_write_iter+0x260/0x770 [ceph] [22078.242946] i2c_algo_bit [22078.247041] ? lock_is_held_type+0xaa/0x120 [22078.248961] pps_core dca [22078.253662] vfs_write+0x324/0x4f0 [22078.256363] fuse [22078.260892] ksys_pwrite64+0x65/0xa0 [22078.263505] irq event stamp: 99059876 [22078.267682] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22078.270299] hardirqs last enabled at (99059875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22078.273699] do_syscall_64+0x38/0x80 [22078.275620] hardirqs last disabled at (99059876): [] sysvec_apic_timer_interrupt+0xe/0x90 [22078.279193] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22078.282850] softirqs last enabled at (99059874): [] __do_softirq+0x2be/0x371 [22078.287980] RIP: 0033:0x7f48f0eeb93b [22078.298136] softirqs last disabled at (99059869): [] irq_exit_rcu+0xb0/0xd0 [22078.301708] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22078.311432] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22078.316477] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22078.325161] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22078.328731] ORIG_RAX: 0000000000000012 [22078.337243] Workqueue: events delayed_work [ceph] [22078.355980] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22078.365357] [22078.370576] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22078.378045] RIP: 0010:lock_acquire+0x4a/0x2b0 [22078.381877] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22078.386575] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [22078.393697] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22078.395188] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [22078.402312] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22078.406663] [22078.406664] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [22078.413794] [22078.432525] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [22078.469932] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [22078.477055] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [22078.484180] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22078.491303] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22078.499382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22078.505120] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22078.512242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22078.519368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22078.526508] Call Trace: [22078.528953] [22078.530965] ? watchdog_timer_fn+0x1b4/0x220 [22078.535229] ? __hrtimer_run_queues+0x1d3/0x360 [22078.539754] ? hrtimer_interrupt+0x104/0x240 [22078.544024] ? __local_bh_enable+0x50/0x60 [22078.548118] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22078.553342] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22078.558299] [22078.560395] [22078.562510] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22078.567819] ? lock_acquire+0x4a/0x2b0 [22078.571570] ? igrab+0x43/0x50 [22078.574628] ? lock_release+0xc5/0x260 [22078.578372] _raw_spin_lock+0x30/0x40 [22078.582029] ? igrab+0x1d/0x50 [22078.585079] igrab+0x1d/0x50 [22078.587959] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22078.593108] delayed_work+0x1d8/0x280 [ceph] [22078.597390] ? process_scheduled_works+0x22a/0x510 [22078.602179] process_scheduled_works+0x2a4/0x510 [22078.606806] worker_thread+0x188/0x340 [22078.610552] ? rescuer_thread+0x310/0x310 [22078.614555] kthread+0xf5/0x130 [22078.617694] ? kthread_complete_and_exit+0x20/0x20 [22078.622477] ret_from_fork+0x31/0x40 [22078.626056] ? kthread_complete_and_exit+0x20/0x20 [22078.630842] ret_from_fork_asm+0x11/0x20 [22078.634772] [22105.845590] watchdog: BUG: soft lockup - CPU#0 stuck for 9473s! [dbench:78683] [22105.852803] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22105.857590] watchdog: BUG: soft lockup - CPU#7 stuck for 18470s! [kworker/7:4:78808] [22105.908244] irq event stamp: 33984320 [22105.915992] Modules linked in: ceph libceph dns_resolver fscache netfs [22105.919649] hardirqs last enabled at (33984319): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22105.926177] veth nft_chain_nat xt_MASQUERADE [22105.936333] hardirqs last disabled at (33984320): [] sysvec_apic_timer_interrupt+0xe/0x90 [22105.940684] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [22105.950408] softirqs last enabled at (33984262): [] __do_softirq+0x2be/0x371 [22105.958054] nfnetlink bridge stp llc xfs [22105.966736] softirqs last disabled at (33984257): [] irq_exit_rcu+0xb0/0xd0 [22105.970741] libcrc32c sunrpc intel_rapl_msr [22105.979266] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22105.983533] intel_rapl_common sb_edac [22105.992478] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22105.992479] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22105.996222] x86_pkg_temp_thermal coretemp [22106.003694] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22106.009170] kvm_intel [22106.013267] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22106.032006] kvm irqbypass [22106.034366] [22106.034366] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22106.039581] i2c_i801 iTCO_wdt [22106.042285] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22106.043775] iTCO_vendor_support i2c_smbus [22106.050899] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22106.053952] crct10dif_pclmul joydev [22106.061075] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22106.065166] crc32_pclmul ipmi_ssif [22106.072289] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22106.075862] ghash_clmulni_intel [22106.082987] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22106.086469] mei_me [22106.093593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22106.096817] mei lpc_ich [22106.104896] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22106.106992] sha512_ssse3 acpi_ipmi [22106.112732] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22106.115269] mfd_core [22106.122393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22106.125877] wmi [22106.133003] Call Trace: [22106.135277] ipmi_si ipmi_devintf [22106.142408] [22106.144241] ipmi_msghandler ioatdma [22106.146689] ? watchdog_timer_fn+0x1b4/0x220 [22106.149997] acpi_pad acpi_power_meter [22106.152008] ? __hrtimer_run_queues+0x1d3/0x360 [22106.155580] ixgbe igb [22106.159845] ? hrtimer_interrupt+0x104/0x240 [22106.163586] mdio crc32c_intel [22106.168110] ? __local_bh_enable+0x50/0x60 [22106.170469] nvme [22106.174734] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22106.177783] ptp nvme_core [22106.181874] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22106.183798] i2c_algo_bit [22106.189015] [22106.191720] pps_core [22106.196677] [22106.199293] dca fuse [22106.201393] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22106.203662] [22106.203663] irq event stamp: 99280720 [22106.205764] ? queued_spin_lock_slowpath+0x16/0x2e0 [22106.208030] hardirqs last enabled at (99280719): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22106.213336] do_raw_spin_lock+0xb5/0xc0 [22106.214824] hardirqs last disabled at (99280720): [] sysvec_apic_timer_interrupt+0xe/0x90 [22106.218484] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22106.223354] softirqs last enabled at (99280718): [] __do_softirq+0x2be/0x371 [22106.233515] try_get_cap_refs+0x159/0x880 [ceph] [22106.237342] softirqs last disabled at (99280713): [] irq_exit_rcu+0xb0/0xd0 [22106.247072] __ceph_get_caps+0x4a1/0x760 [ceph] [22106.251849] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22106.260537] ? __file_remove_privs+0x89/0x110 [22106.265145] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22106.273656] ? wake_bit_function+0x70/0x70 [22106.278180] Workqueue: events delayed_work [ceph] [22106.287559] ? ceph_write_iter+0x260/0x770 [ceph] [22106.291908] [22106.299379] ceph_write_iter+0x260/0x770 [ceph] [22106.303470] RIP: 0010:lock_acquire+0xf0/0x2b0 [22106.308172] ? lock_is_held_type+0xaa/0x120 [22106.312864] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22106.314358] vfs_write+0x324/0x4f0 [22106.318878] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22106.323236] ksys_pwrite64+0x65/0xa0 [22106.327408] [22106.327408] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22106.346145] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22106.349542] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22106.354761] do_syscall_64+0x38/0x80 [22106.358330] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22106.359822] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22106.366944] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22106.372076] RIP: 0033:0x7f48f0eeb93b [22106.379199] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22106.382772] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22106.389895] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22106.394939] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22106.402063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22106.405635] ORIG_RAX: 0000000000000012 [22106.412758] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22106.431497] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22106.439572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22106.444791] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22106.450529] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22106.454358] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22106.461484] Call Trace: [22106.461485] [22106.468606] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22106.475732] ? watchdog_timer_fn+0x1b4/0x220 [22106.482856] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22106.489996] ? __hrtimer_run_queues+0x1d3/0x360 [22106.497126] [22106.499569] ? hrtimer_interrupt+0x104/0x240 [22106.531061] ? __local_bh_enable+0x50/0x60 [22106.535153] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22106.540377] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22106.545336] [22106.547432] [22106.549531] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22106.554839] ? lock_acquire+0xf0/0x2b0 [22106.558587] ? igrab+0x43/0x50 [22106.561637] ? lock_release+0xc5/0x260 [22106.565383] _raw_spin_lock+0x30/0x40 [22106.569048] ? igrab+0x1d/0x50 [22106.572097] igrab+0x1d/0x50 [22106.574995] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22106.580145] delayed_work+0x1d8/0x280 [ceph] [22106.584424] ? process_scheduled_works+0x22a/0x510 [22106.589215] process_scheduled_works+0x2a4/0x510 [22106.593830] worker_thread+0x188/0x340 [22106.597580] ? rescuer_thread+0x310/0x310 [22106.601583] kthread+0xf5/0x130 [22106.604723] ? kthread_complete_and_exit+0x20/0x20 [22106.609512] ret_from_fork+0x31/0x40 [22106.613092] ? kthread_complete_and_exit+0x20/0x20 [22106.617876] ret_from_fork_asm+0x11/0x20 [22106.621799] [22133.846077] watchdog: BUG: soft lockup - CPU#0 stuck for 9499s! [dbench:78683] [22133.853289] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22133.858077] watchdog: BUG: soft lockup - CPU#7 stuck for 18496s! [kworker/7:4:78808] [22133.908748] irq event stamp: 34042476 [22133.908749] hardirqs last enabled at (34042475): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22133.916479] Modules linked in: [22133.920136] hardirqs last disabled at (34042476): [] sysvec_apic_timer_interrupt+0xe/0x90 [22133.930293] ceph [22133.933346] softirqs last enabled at (34042466): [] __do_softirq+0x2be/0x371 [22133.943069] libceph [22133.944992] softirqs last disabled at (34042461): [] irq_exit_rcu+0xb0/0xd0 [22133.953677] dns_resolver fscache [22133.955863] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22133.964373] netfs [22133.967683] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22133.976627] veth nft_chain_nat [22133.978639] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22133.986108] xt_MASQUERADE nf_nat [22133.989246] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22133.994722] nf_conntrack nf_defrag_ipv6 [22133.998035] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22134.016771] nf_defrag_ipv4 nft_compat [22134.020690] [22134.020691] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22134.025906] nf_tables nfnetlink [22134.029650] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22134.031142] bridge stp [22134.038265] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22134.041489] llc [22134.048615] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22134.051059] xfs libcrc32c [22134.058182] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22134.060020] sunrpc [22134.067145] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22134.069847] intel_rapl_msr intel_rapl_common [22134.076973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22134.079069] sb_edac x86_pkg_temp_thermal [22134.087146] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22134.091498] coretemp kvm_intel [22134.097236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22134.101239] kvm irqbypass [22134.108363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22134.111502] i2c_i801 iTCO_wdt [22134.118626] Call Trace: [22134.121330] iTCO_vendor_support i2c_smbus [22134.128471] [22134.131521] crct10dif_pclmul joydev [22134.133967] ? watchdog_timer_fn+0x1b4/0x220 [22134.138057] crc32_pclmul ipmi_ssif [22134.140068] ? __hrtimer_run_queues+0x1d3/0x360 [22134.143639] ghash_clmulni_intel [22134.147905] ? hrtimer_interrupt+0x104/0x240 [22134.151384] mei_me mei [22134.155908] ? __local_bh_enable+0x50/0x60 [22134.159132] lpc_ich sha512_ssse3 [22134.163399] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22134.165842] acpi_ipmi mfd_core [22134.169933] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22134.173244] wmi [22134.178475] [22134.181614] ipmi_si ipmi_devintf [22134.186571] [22134.188410] ipmi_msghandler [22134.190508] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22134.193818] ioatdma acpi_pad [22134.195919] ? queued_spin_lock_slowpath+0x16/0x2e0 [22134.198792] acpi_power_meter ixgbe igb [22134.204099] do_raw_spin_lock+0xb5/0xc0 [22134.207062] mdio crc32c_intel [22134.211934] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22134.215762] nvme ptp [22134.219597] try_get_cap_refs+0x159/0x880 [ceph] [22134.222643] nvme_core i2c_algo_bit [22134.227434] __ceph_get_caps+0x4a1/0x760 [ceph] [22134.229698] pps_core dca [22134.234315] ? __file_remove_privs+0x89/0x110 [22134.237793] fuse [22134.242318] ? wake_bit_function+0x70/0x70 [22134.244935] irq event stamp: 99501726 [22134.249287] ? ceph_write_iter+0x260/0x770 [ceph] [22134.251209] hardirqs last enabled at (99501725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22134.255300] ceph_write_iter+0x260/0x770 [ceph] [22134.258958] hardirqs last disabled at (99501726): [] sysvec_apic_timer_interrupt+0xe/0x90 [22134.263659] ? lock_is_held_type+0xaa/0x120 [22134.273813] softirqs last enabled at (99501724): [] __do_softirq+0x2be/0x371 [22134.278340] vfs_write+0x324/0x4f0 [22134.288060] softirqs last disabled at (99501719): [] irq_exit_rcu+0xb0/0xd0 [22134.292243] ksys_pwrite64+0x65/0xa0 [22134.300923] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22134.304321] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22134.312831] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22134.316403] do_syscall_64+0x38/0x80 [22134.325778] Workqueue: events delayed_work [ceph] [22134.330910] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22134.338382] [22134.338383] RIP: 0010:ceph_check_delayed_caps+0x93/0x270 [ceph] [22134.341953] RIP: 0033:0x7f48f0eeb93b [22134.346648] Code: 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e <48> 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 [22134.351693] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22134.353183] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [22134.359093] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22134.362664] [22134.362665] RAX: 0000000000000001 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [22134.381402] ORIG_RAX: 0000000000000012 [22134.400141] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [22134.405357] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22134.410576] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [22134.412066] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22134.419190] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [22134.423022] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22134.430145] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22134.437269] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22134.444394] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22134.451517] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22134.458642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22134.465772] [22134.472890] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22134.510261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22134.517387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22134.524509] Call Trace: [22134.526955] [22134.528965] ? watchdog_timer_fn+0x1b4/0x220 [22134.533230] ? __hrtimer_run_queues+0x1d3/0x360 [22134.537756] ? hrtimer_interrupt+0x104/0x240 [22134.542033] ? __local_bh_enable+0x50/0x60 [22134.546127] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22134.551342] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22134.556301] [22134.558398] [22134.560495] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22134.565804] ? ceph_check_delayed_caps+0x93/0x270 [ceph] [22134.571121] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [22134.576434] delayed_work+0x1d8/0x280 [ceph] [22134.580717] ? process_scheduled_works+0x22a/0x510 [22134.585506] process_scheduled_works+0x2a4/0x510 [22134.590122] worker_thread+0x188/0x340 [22134.593873] ? rescuer_thread+0x310/0x310 [22134.597876] kthread+0xf5/0x130 [22134.601022] ? kthread_complete_and_exit+0x20/0x20 [22134.605805] ret_from_fork+0x31/0x40 [22134.609383] ? kthread_complete_and_exit+0x20/0x20 [22134.614168] ret_from_fork_asm+0x11/0x20 [22134.618091] [22161.846567] watchdog: BUG: soft lockup - CPU#0 stuck for 9525s! [dbench:78683] [22161.853786] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22161.858567] watchdog: BUG: soft lockup - CPU#7 stuck for 18522s! [kworker/7:4:78808] [22161.909242] irq event stamp: 34100688 [22161.909243] hardirqs last enabled at (34100687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22161.916976] Modules linked in: [22161.920632] hardirqs last disabled at (34100688): [] sysvec_apic_timer_interrupt+0xe/0x90 [22161.930791] ceph [22161.933842] softirqs last enabled at (34100642): [] __do_softirq+0x2be/0x371 [22161.943565] libceph [22161.945489] softirqs last disabled at (34100637): [] irq_exit_rcu+0xb0/0xd0 [22161.954174] dns_resolver fscache [22161.956358] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22161.964867] netfs veth [22161.968180] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22161.977122] nft_chain_nat xt_MASQUERADE [22161.979567] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22161.987038] nf_nat nf_conntrack nf_defrag_ipv6 [22161.990963] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22161.996442] nf_defrag_ipv4 nft_compat [22162.000965] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22162.019703] nf_tables nfnetlink [22162.023448] [22162.028664] bridge stp [22162.031889] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22162.033380] llc xfs [22162.035824] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22162.042965] libcrc32c [22162.045148] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22162.052272] sunrpc [22162.054631] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22162.061755] intel_rapl_msr [22162.063853] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22162.070975] intel_rapl_common [22162.073768] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22162.080890] sb_edac x86_pkg_temp_thermal [22162.083941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22162.092036] coretemp kvm_intel [22162.096041] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22162.101777] kvm irqbypass [22162.104916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22162.112038] i2c_i801 iTCO_wdt [22162.114744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22162.121868] iTCO_vendor_support [22162.124919] Call Trace: [22162.132042] i2c_smbus crct10dif_pclmul [22162.135267] [22162.137711] joydev crc32_pclmul [22162.141542] ? watchdog_timer_fn+0x1b4/0x220 [22162.143551] ipmi_ssif ghash_clmulni_intel [22162.146777] ? __hrtimer_run_queues+0x1d3/0x360 [22162.151039] mei_me mei [22162.155135] ? hrtimer_interrupt+0x104/0x240 [22162.159656] lpc_ich sha512_ssse3 [22162.162098] ? __local_bh_enable+0x50/0x60 [22162.166363] acpi_ipmi mfd_core [22162.169677] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22162.173765] wmi ipmi_si [22162.176903] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22162.182119] ipmi_devintf ipmi_msghandler [22162.184650] [22162.189607] ioatdma acpi_pad [22162.193612] [22162.193614] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22162.195708] acpi_power_meter [22162.198677] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22162.200770] ixgbe igb [22162.206077] do_raw_spin_lock+0xb5/0xc0 [22162.209038] mdio crc32c_intel [22162.213911] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22162.216267] nvme ptp [22162.220099] try_get_cap_refs+0x159/0x880 [ceph] [22162.223147] nvme_core i2c_algo_bit [22162.227938] __ceph_get_caps+0x4a1/0x760 [ceph] [22162.230203] pps_core dca [22162.234819] ? __file_remove_privs+0x89/0x110 [22162.238299] fuse [22162.242824] ? wake_bit_function+0x70/0x70 [22162.245440] irq event stamp: 99723174 [22162.249792] ? ceph_write_iter+0x260/0x770 [ceph] [22162.251714] hardirqs last enabled at (99723173): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22162.255804] ceph_write_iter+0x260/0x770 [ceph] [22162.259463] hardirqs last disabled at (99723174): [] sysvec_apic_timer_interrupt+0xe/0x90 [22162.264163] ? lock_is_held_type+0xaa/0x120 [22162.274316] softirqs last enabled at (99723172): [] __do_softirq+0x2be/0x371 [22162.278843] vfs_write+0x324/0x4f0 [22162.288565] softirqs last disabled at (99723167): [] irq_exit_rcu+0xb0/0xd0 [22162.292746] ksys_pwrite64+0x65/0xa0 [22162.301426] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22162.304826] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22162.313336] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22162.316906] do_syscall_64+0x38/0x80 [22162.326284] Workqueue: events delayed_work [ceph] [22162.331416] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22162.338884] [22162.338885] RIP: 0010:__list_del_entry_valid_or_report+0x2e/0xd0 [22162.342456] RIP: 0033:0x7f48f0eeb93b [22162.347153] Code: 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de <48> 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 48 39 f2 75 73 [22162.352197] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22162.353687] RSP: 0018:ffffc90002117db0 EFLAGS: 00000202 [22162.359686] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22162.363267] [22162.363268] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [22162.382003] ORIG_RAX: 0000000000000012 [22162.400740] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [22162.405970] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22162.411183] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [22162.412674] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22162.419799] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [22162.423630] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22162.430754] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22162.437877] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22162.445002] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22162.452126] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22162.459250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22162.466380] [22162.473497] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22162.510869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22162.517995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22162.525119] Call Trace: [22162.527563] [22162.529574] ? watchdog_timer_fn+0x1b4/0x220 [22162.533839] ? __hrtimer_run_queues+0x1d3/0x360 [22162.538373] ? hrtimer_interrupt+0x104/0x240 [22162.542643] ? __local_bh_enable+0x50/0x60 [22162.546737] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22162.551975] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22162.556935] [22162.559031] [22162.561129] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22162.566438] ? __list_del_entry_valid_or_report+0x2e/0xd0 [22162.571833] ceph_check_delayed_caps+0x8f/0x270 [ceph] [22162.576983] delayed_work+0x1d8/0x280 [ceph] [22162.581264] ? process_scheduled_works+0x22a/0x510 [22162.586055] process_scheduled_works+0x2a4/0x510 [22162.590669] worker_thread+0x188/0x340 [22162.594419] ? rescuer_thread+0x310/0x310 [22162.598422] kthread+0xf5/0x130 [22162.601560] ? kthread_complete_and_exit+0x20/0x20 [22162.606343] ret_from_fork+0x31/0x40 [22162.609914] ? kthread_complete_and_exit+0x20/0x20 [22162.614698] ret_from_fork_asm+0x11/0x20 [22162.618621] [22164.781986] rcu: INFO: rcu_preempt self-detected stall on CPU [22164.787742] rcu: 7-....: (9931965 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/9930695 fqs=4988944 [22164.798167] rcu: (t=10227988 jiffies g=1309609 q=343015 ncpus=8) [22164.804253] Sending NMI from CPU 7 to CPUs 0: [22164.808602] NMI backtrace for cpu 0 [22164.808604] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22164.808605] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22164.808606] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22164.808609] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22164.808611] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22164.808612] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22164.808613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22164.808614] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22164.808614] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22164.808615] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22164.808616] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22164.808617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22164.808618] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22164.808619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22164.808619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22164.808620] Call Trace: [22164.808621] [22164.808622] ? nmi_cpu_backtrace+0xad/0x120 [22164.808626] ? nmi_cpu_backtrace_handler+0x11/0x20 [22164.808627] ? nmi_handle+0xd4/0x260 [22164.808632] ? default_do_nmi+0x49/0xf0 [22164.808633] ? exc_nmi+0x11e/0x150 [22164.808635] ? end_repeat_nmi+0x16/0x67 [22164.808639] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22164.808642] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22164.808644] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22164.808647] [22164.808647] [22164.808648] do_raw_spin_lock+0xb5/0xc0 [22164.808650] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22164.808665] try_get_cap_refs+0x159/0x880 [ceph] [22164.808679] __ceph_get_caps+0x4a1/0x760 [ceph] [22164.808692] ? __file_remove_privs+0x89/0x110 [22164.808694] ? wake_bit_function+0x70/0x70 [22164.808696] ? ceph_write_iter+0x260/0x770 [ceph] [22164.808707] ceph_write_iter+0x260/0x770 [ceph] [22164.808719] ? lock_is_held_type+0xaa/0x120 [22164.808721] vfs_write+0x324/0x4f0 [22164.808725] ksys_pwrite64+0x65/0xa0 [22164.808728] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22164.808730] do_syscall_64+0x38/0x80 [22164.808733] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22164.808734] RIP: 0033:0x7f48f0eeb93b [22164.808736] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22164.808737] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [22164.808738] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22164.808739] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22164.808740] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22164.808740] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22164.808741] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22164.808744] [22164.809600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22165.113336] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22165.120807] Workqueue: events delayed_work [ceph] [22165.125520] RIP: 0010:lock_acquire+0xf0/0x2b0 [22165.129873] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22165.148611] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22165.153826] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22165.160952] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22165.168074] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22165.175199] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22165.182325] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22165.189447] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22165.197524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22165.203262] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22165.210388] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22165.217512] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22165.224634] Call Trace: [22165.227080] [22165.229090] ? rcu_dump_cpu_stacks+0xd1/0x180 [22165.233451] ? rcu_sched_clock_irq+0x633/0x1390 [22165.237983] ? lock_release+0xc5/0x260 [22165.241726] ? lock_is_held_type+0xaa/0x120 [22165.245904] ? lock_is_held_type+0xaa/0x120 [22165.250082] ? tick_sched_do_timer+0xa0/0xa0 [22165.254354] ? rcu_read_lock_sched_held+0x56/0x90 [22165.259052] ? tick_sched_do_timer+0xa0/0xa0 [22165.263315] ? update_process_times+0x5f/0x90 [22165.267667] ? tick_sched_handle.isra.16+0x1e/0x50 [22165.272450] ? tick_sched_timer+0x6f/0x80 [22165.276456] ? __hrtimer_run_queues+0x1d3/0x360 [22165.280981] ? hrtimer_interrupt+0x104/0x240 [22165.285243] ? __local_bh_enable+0x50/0x60 [22165.289336] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22165.294559] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22165.299516] [22165.301615] [22165.303713] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22165.309020] ? lock_acquire+0xf0/0x2b0 [22165.312770] ? igrab+0x43/0x50 [22165.315819] ? lock_release+0xc5/0x260 [22165.319566] _raw_spin_lock+0x30/0x40 [22165.323229] ? igrab+0x1d/0x50 [22165.326280] igrab+0x1d/0x50 [22165.329160] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22165.334308] delayed_work+0x1d8/0x280 [ceph] [22165.338589] ? process_scheduled_works+0x22a/0x510 [22165.343380] process_scheduled_works+0x2a4/0x510 [22165.347998] worker_thread+0x188/0x340 [22165.351743] ? rescuer_thread+0x310/0x310 [22165.355746] kthread+0xf5/0x130 [22165.358884] ? kthread_complete_and_exit+0x20/0x20 [22165.363670] ret_from_fork+0x31/0x40 [22165.367240] ? kthread_complete_and_exit+0x20/0x20 [22165.372023] ret_from_fork_asm+0x11/0x20 [22165.375946] [22189.847059] watchdog: BUG: soft lockup - CPU#0 stuck for 9551s! [dbench:78683] [22189.854274] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22189.859059] watchdog: BUG: soft lockup - CPU#7 stuck for 18548s! [kworker/7:4:78808] [22189.909734] irq event stamp: 34158974 [22189.917462] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [22189.921120] hardirqs last enabled at (34158973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22189.929293] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [22189.939454] hardirqs last disabled at (34158974): [] sysvec_apic_timer_interrupt+0xe/0x90 [22189.946496] nft_compat nf_tables nfnetlink bridge stp [22189.956219] softirqs last enabled at (34158968): [] __do_softirq+0x2be/0x371 [22189.956223] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [22189.970035] softirqs last disabled at (34158963): [] irq_exit_rcu+0xb0/0xd0 [22189.976552] sb_edac x86_pkg_temp_thermal coretemp [22189.985064] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22189.985065] kvm_intel kvm irqbypass [22189.998792] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22189.998793] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22190.002361] i2c_i801 iTCO_wdt [22190.009834] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22190.015310] iTCO_vendor_support i2c_smbus [22190.018362] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22190.037098] crct10dif_pclmul joydev [22190.041190] [22190.041191] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22190.046406] crc32_pclmul [22190.049979] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22190.051469] ipmi_ssif ghash_clmulni_intel [22190.058592] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22190.061210] mei_me mei [22190.068335] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22190.072424] lpc_ich sha512_ssse3 [22190.079549] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22190.081994] acpi_ipmi mfd_core [22190.089118] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22190.092429] wmi ipmi_si [22190.099570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22190.102706] ipmi_devintf ipmi_msghandler [22190.110784] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22190.113315] ioatdma acpi_pad [22190.119054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22190.123057] acpi_power_meter [22190.130180] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22190.133145] ixgbe igb [22190.140270] Call Trace: [22190.140272] [22190.143233] mdio crc32c_intel [22190.150360] ? watchdog_timer_fn+0x1b4/0x220 [22190.152714] nvme ptp [22190.155176] ? __hrtimer_run_queues+0x1d3/0x360 [22190.157186] nvme_core i2c_algo_bit [22190.160241] ? hrtimer_interrupt+0x104/0x240 [22190.164500] pps_core [22190.166772] ? __local_bh_enable+0x50/0x60 [22190.171297] dca [22190.174782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22190.179045] fuse [22190.181317] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22190.185406] [22190.185407] irq event stamp: 99939506 [22190.187244] [22190.192461] hardirqs last enabled at (99939505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22190.194386] [22190.194387] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22190.199341] hardirqs last disabled at (99939506): [] sysvec_apic_timer_interrupt+0xe/0x90 [22190.200838] ? queued_spin_lock_slowpath+0x16/0x2e0 [22190.204490] softirqs last enabled at (99939504): [] __do_softirq+0x2be/0x371 [22190.206590] do_raw_spin_lock+0xb5/0xc0 [22190.216746] softirqs last disabled at (99939495): [] irq_exit_rcu+0xb0/0xd0 [22190.218845] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22190.224146] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22190.233873] try_get_cap_refs+0x159/0x880 [ceph] [22190.238754] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22190.247464] __ceph_get_caps+0x4a1/0x760 [ceph] [22190.251291] Workqueue: events delayed_work [ceph] [22190.259806] ? __file_remove_privs+0x89/0x110 [22190.264586] [22190.264587] RIP: 0010:lock_acquire+0xf0/0x2b0 [22190.273964] ? wake_bit_function+0x70/0x70 [22190.278575] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22190.286047] ? ceph_write_iter+0x260/0x770 [ceph] [22190.290570] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22190.295267] ceph_write_iter+0x260/0x770 [ceph] [22190.299616] [22190.301113] ? lock_is_held_type+0xaa/0x120 [22190.305460] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22190.309552] vfs_write+0x324/0x4f0 [22190.328287] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22190.332989] ksys_pwrite64+0x65/0xa0 [22190.338201] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22190.342742] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22190.344234] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22190.348413] do_syscall_64+0x38/0x80 [22190.355536] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22190.358933] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22190.366057] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22190.369629] RIP: 0033:0x7f48f0eeb93b [22190.376753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22190.381886] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22190.389006] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22190.392579] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22190.399703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22190.404746] ORIG_RAX: 0000000000000012 [22190.412824] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22190.416395] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22190.422132] Call Trace: [22190.422133] [22190.440870] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22190.447995] ? watchdog_timer_fn+0x1b4/0x220 [22190.453211] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22190.460336] ? __hrtimer_run_queues+0x1d3/0x360 [22190.464166] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22190.471292] ? hrtimer_interrupt+0x104/0x240 [22190.478415] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22190.480859] ? __local_bh_enable+0x50/0x60 [22190.482875] [22190.489995] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22190.535911] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22190.540867] [22190.542964] [22190.545062] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22190.550371] ? lock_acquire+0xf0/0x2b0 [22190.554121] ? igrab+0x43/0x50 [22190.557171] ? lock_release+0xc5/0x260 [22190.560915] _raw_spin_lock+0x30/0x40 [22190.564573] ? igrab+0x1d/0x50 [22190.567622] igrab+0x1d/0x50 [22190.570502] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22190.575650] delayed_work+0x1d8/0x280 [ceph] [22190.579932] ? process_scheduled_works+0x22a/0x510 [22190.584721] process_scheduled_works+0x2a4/0x510 [22190.589356] worker_thread+0x188/0x340 [22190.593103] ? rescuer_thread+0x310/0x310 [22190.597105] kthread+0xf5/0x130 [22190.600245] ? kthread_complete_and_exit+0x20/0x20 [22190.605029] ret_from_fork+0x31/0x40 [22190.608608] ? kthread_complete_and_exit+0x20/0x20 [22190.613391] ret_from_fork_asm+0x11/0x20 [22190.617315] [22217.847559] watchdog: BUG: soft lockup - CPU#0 stuck for 9577s! [dbench:78683] [22217.854778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22217.859559] watchdog: BUG: soft lockup - CPU#7 stuck for 18574s! [kworker/7:4:78808] [22217.910232] irq event stamp: 34217172 [22217.910233] hardirqs last enabled at (34217171): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22217.917967] Modules linked in: [22217.921624] hardirqs last disabled at (34217172): [] sysvec_apic_timer_interrupt+0xe/0x90 [22217.931783] ceph [22217.934832] softirqs last enabled at (34217152): [] __do_softirq+0x2be/0x371 [22217.944557] libceph [22217.946480] softirqs last disabled at (34217147): [] irq_exit_rcu+0xb0/0xd0 [22217.955165] dns_resolver fscache [22217.957349] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22217.965859] netfs veth [22217.969170] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22217.978115] nft_chain_nat xt_MASQUERADE [22217.980560] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22217.988028] nf_nat nf_conntrack [22217.991952] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22217.997433] nf_defrag_ipv6 nf_defrag_ipv4 [22218.000657] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22218.019396] nft_compat nf_tables [22218.023484] [22218.028703] nfnetlink bridge [22218.032015] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22218.033504] stp llc [22218.036469] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22218.043593] xfs libcrc32c [22218.045777] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22218.052900] sunrpc intel_rapl_msr [22218.055606] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22218.062728] intel_rapl_common sb_edac [22218.066126] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22218.073250] x86_pkg_temp_thermal coretemp [22218.076995] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22218.084118] kvm_intel kvm [22218.088209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22218.096286] irqbypass i2c_i801 [22218.098989] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22218.104728] iTCO_wdt iTCO_vendor_support [22218.107867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22218.114989] i2c_smbus crct10dif_pclmul [22218.118995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22218.126117] joydev crc32_pclmul [22218.129957] Call Trace: [22218.129958] [22218.137080] ipmi_ssif [22218.140307] ? watchdog_timer_fn+0x1b4/0x220 [22218.142748] ghash_clmulni_intel mei_me [22218.144762] ? __hrtimer_run_queues+0x1d3/0x360 [22218.147118] mei lpc_ich [22218.151385] ? hrtimer_interrupt+0x104/0x240 [22218.155212] sha512_ssse3 acpi_ipmi [22218.159753] ? __local_bh_enable+0x50/0x60 [22218.162284] mfd_core [22218.166551] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22218.170033] wmi ipmi_si [22218.174125] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22218.176393] ipmi_devintf ipmi_msghandler [22218.181612] [22218.184142] ioatdma acpi_pad [22218.189101] [22218.189102] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22218.193105] acpi_power_meter ixgbe [22218.195206] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22218.198166] igb mdio [22218.200266] do_raw_spin_lock+0xb5/0xc0 [22218.205568] crc32c_intel nvme [22218.209053] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22218.213922] ptp nvme_core [22218.216195] try_get_cap_refs+0x159/0x880 [ceph] [22218.220022] i2c_algo_bit pps_core [22218.223079] __ceph_get_caps+0x4a1/0x760 [ceph] [22218.227858] dca fuse [22218.230566] ? __file_remove_privs+0x89/0x110 [22218.235173] [22218.235174] irq event stamp: 100161166 [22218.238572] ? wake_bit_function+0x70/0x70 [22218.243094] hardirqs last enabled at (100161165): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22218.245368] ? ceph_write_iter+0x260/0x770 [ceph] [22218.249715] hardirqs last disabled at (100161166): [] sysvec_apic_timer_interrupt+0xe/0x90 [22218.251208] ceph_write_iter+0x260/0x770 [ceph] [22218.254957] softirqs last enabled at (100161164): [] __do_softirq+0x2be/0x371 [22218.259055] ? lock_is_held_type+0xaa/0x120 [22218.269294] softirqs last disabled at (100161159): [] irq_exit_rcu+0xb0/0xd0 [22218.273993] vfs_write+0x324/0x4f0 [22218.283801] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22218.288330] ksys_pwrite64+0x65/0xa0 [22218.297098] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22218.301275] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22218.309871] Workqueue: events delayed_work [ceph] [22218.313271] do_syscall_64+0x38/0x80 [22218.322648] [22218.322649] RIP: 0010:lock_acquire+0xf0/0x2b0 [22218.326236] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22218.333707] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22218.338837] RIP: 0033:0x7f48f0eeb93b [22218.343533] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22218.347107] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22218.348594] [22218.348595] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22218.352960] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22218.358007] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22218.376745] ORIG_RAX: 0000000000000012 [22218.380315] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22218.385534] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22218.404271] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22218.405762] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22218.412887] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22218.418104] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22218.425240] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22218.429066] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22218.436191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22218.443316] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22218.450438] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22218.457568] [22218.464688] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22218.516309] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22218.523432] Call Trace: [22218.525877] [22218.527888] ? watchdog_timer_fn+0x1b4/0x220 [22218.532150] ? __hrtimer_run_queues+0x1d3/0x360 [22218.536677] ? hrtimer_interrupt+0x104/0x240 [22218.540939] ? __local_bh_enable+0x50/0x60 [22218.545050] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22218.550274] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22218.555243] [22218.557345] [22218.559444] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22218.564750] ? lock_acquire+0xf0/0x2b0 [22218.568500] ? igrab+0x43/0x50 [22218.571552] ? lock_release+0xc5/0x260 [22218.575296] _raw_spin_lock+0x30/0x40 [22218.578963] ? igrab+0x1d/0x50 [22218.582019] igrab+0x1d/0x50 [22218.584897] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22218.590047] delayed_work+0x1d8/0x280 [ceph] [22218.594328] ? process_scheduled_works+0x22a/0x510 [22218.599109] process_scheduled_works+0x2a4/0x510 [22218.603725] worker_thread+0x188/0x340 [22218.607476] ? rescuer_thread+0x310/0x310 [22218.611486] kthread+0xf5/0x130 [22218.614625] ? kthread_complete_and_exit+0x20/0x20 [22218.619409] ret_from_fork+0x31/0x40 [22218.622978] ? kthread_complete_and_exit+0x20/0x20 [22218.627763] ret_from_fork_asm+0x11/0x20 [22218.631687] [22245.848059] watchdog: BUG: soft lockup - CPU#0 stuck for 9604s! [dbench:78683] [22245.855274] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22245.860058] watchdog: BUG: soft lockup - CPU#7 stuck for 18601s! [kworker/7:4:78808] [22245.910735] irq event stamp: 34275416 [22245.910737] hardirqs last enabled at (34275415): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22245.918472] Modules linked in: [22245.922129] hardirqs last disabled at (34275416): [] sysvec_apic_timer_interrupt+0xe/0x90 [22245.932287] ceph [22245.935337] softirqs last enabled at (34275368): [] __do_softirq+0x2be/0x371 [22245.945060] libceph [22245.946986] softirqs last disabled at (34275363): [] irq_exit_rcu+0xb0/0xd0 [22245.955669] dns_resolver [22245.957854] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22245.966364] fscache [22245.968981] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22245.977927] netfs [22245.980111] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22245.987582] veth nft_chain_nat [22245.989593] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22245.995068] xt_MASQUERADE [22245.998206] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22246.016944] nf_nat nf_conntrack [22246.019648] [22246.019649] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22246.024865] nf_defrag_ipv6 nf_defrag_ipv4 [22246.028088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22246.029579] nft_compat nf_tables [22246.036703] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22246.040795] nfnetlink bridge [22246.047920] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22246.051230] stp llc [22246.058353] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22246.061317] xfs libcrc32c [22246.068456] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22246.070643] sunrpc intel_rapl_msr [22246.077766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22246.080472] intel_rapl_common [22246.088550] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22246.091946] sb_edac x86_pkg_temp_thermal [22246.097684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22246.100746] coretemp [22246.107868] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22246.111873] kvm_intel [22246.118996] Call Trace: [22246.121267] kvm irqbypass [22246.128392] [22246.130748] i2c_i801 iTCO_wdt [22246.133196] ? watchdog_timer_fn+0x1b4/0x220 [22246.135897] iTCO_vendor_support i2c_smbus [22246.137908] ? __hrtimer_run_queues+0x1d3/0x360 [22246.140959] crct10dif_pclmul joydev [22246.145226] ? hrtimer_interrupt+0x104/0x240 [22246.149312] crc32_pclmul ipmi_ssif [22246.153837] ? __local_bh_enable+0x50/0x60 [22246.157406] ghash_clmulni_intel mei_me [22246.161673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22246.165156] mei lpc_ich [22246.169247] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22246.173076] sha512_ssse3 [22246.178294] [22246.180824] acpi_ipmi [22246.185783] [22246.185785] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22246.188401] mfd_core [22246.190502] ? queued_spin_lock_slowpath+0x18/0x2e0 [22246.192854] wmi ipmi_si [22246.194954] do_raw_spin_lock+0xb5/0xc0 [22246.200256] ipmi_devintf ipmi_msghandler [22246.202528] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22246.207397] ioatdma [22246.209930] try_get_cap_refs+0x159/0x880 [ceph] [22246.213758] acpi_pad acpi_power_meter [22246.217769] __ceph_get_caps+0x4a1/0x760 [ceph] [22246.222546] ixgbe igb [22246.224747] ? __file_remove_privs+0x89/0x110 [22246.229349] mdio [22246.233095] ? wake_bit_function+0x70/0x70 [22246.237619] crc32c_intel [22246.239978] ? ceph_write_iter+0x260/0x770 [ceph] [22246.244327] nvme [22246.246250] ceph_write_iter+0x260/0x770 [ceph] [22246.250341] ptp nvme_core [22246.252963] ? lock_is_held_type+0xaa/0x120 [22246.257657] i2c_algo_bit pps_core [22246.259584] vfs_write+0x324/0x4f0 [22246.264104] dca fuse [22246.266812] ksys_pwrite64+0x65/0xa0 [22246.270986] [22246.270987] irq event stamp: 100381968 [22246.274383] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22246.277781] hardirqs last enabled at (100381967): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22246.280053] do_syscall_64+0x38/0x80 [22246.283621] hardirqs last disabled at (100381968): [] sysvec_apic_timer_interrupt+0xe/0x90 [22246.285115] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22246.288856] softirqs last enabled at (100381966): [] __do_softirq+0x2be/0x371 [22246.293990] RIP: 0033:0x7f48f0eeb93b [22246.304233] softirqs last disabled at (100381961): [] irq_exit_rcu+0xb0/0xd0 [22246.307805] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22246.317613] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22246.322656] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22246.331429] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22246.334998] ORIG_RAX: 0000000000000012 [22246.343596] Workqueue: events delayed_work [ceph] [22246.362335] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22246.371713] [22246.371714] RIP: 0010:lock_acquire+0xf0/0x2b0 [22246.376930] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22246.384401] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22246.388229] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22246.392928] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22246.400052] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22246.401543] [22246.401543] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22246.405894] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22246.413017] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22246.431762] [22246.438879] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22246.483392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22246.490516] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22246.497641] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22246.505718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22246.511472] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22246.518598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22246.525721] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22246.532864] Call Trace: [22246.535309] [22246.537319] ? watchdog_timer_fn+0x1b4/0x220 [22246.541582] ? __hrtimer_run_queues+0x1d3/0x360 [22246.546109] ? hrtimer_interrupt+0x104/0x240 [22246.550379] ? __local_bh_enable+0x50/0x60 [22246.554472] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22246.559697] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22246.564653] [22246.566750] [22246.568848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22246.574157] ? lock_acquire+0xf0/0x2b0 [22246.577907] ? igrab+0x43/0x50 [22246.580957] ? lock_release+0xc5/0x260 [22246.584702] _raw_spin_lock+0x30/0x40 [22246.588367] ? igrab+0x1d/0x50 [22246.591415] igrab+0x1d/0x50 [22246.594295] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22246.599464] delayed_work+0x1d8/0x280 [ceph] [22246.603748] ? process_scheduled_works+0x22a/0x510 [22246.608533] process_scheduled_works+0x2a4/0x510 [22246.613148] worker_thread+0x188/0x340 [22246.616897] ? rescuer_thread+0x310/0x310 [22246.620901] kthread+0xf5/0x130 [22246.624039] ? kthread_complete_and_exit+0x20/0x20 [22246.628822] ret_from_fork+0x31/0x40 [22246.632402] ? kthread_complete_and_exit+0x20/0x20 [22246.637187] ret_from_fork_asm+0x11/0x20 [22246.641118] [22273.848560] watchdog: BUG: soft lockup - CPU#0 stuck for 9630s! [dbench:78683] [22273.855778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22273.860560] watchdog: BUG: soft lockup - CPU#7 stuck for 18627s! [kworker/7:4:78808] [22273.911234] irq event stamp: 34333626 [22273.911235] hardirqs last enabled at (34333625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22273.918968] Modules linked in: [22273.922624] hardirqs last disabled at (34333626): [] sysvec_apic_timer_interrupt+0xe/0x90 [22273.932782] ceph [22273.935832] softirqs last enabled at (34333578): [] __do_softirq+0x2be/0x371 [22273.945556] libceph [22273.947482] softirqs last disabled at (34333573): [] irq_exit_rcu+0xb0/0xd0 [22273.956165] dns_resolver fscache [22273.958348] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22273.966860] netfs veth [22273.970169] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22273.979114] nft_chain_nat xt_MASQUERADE [22273.981559] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22273.989028] nf_nat nf_conntrack [22273.992956] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22273.998432] nf_defrag_ipv6 nf_defrag_ipv4 [22274.001658] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22274.020394] nft_compat nf_tables [22274.024486] [22274.024487] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22274.029704] nfnetlink [22274.033014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22274.034505] bridge [22274.041628] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22274.043984] stp llc [22274.051110] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22274.053208] xfs [22274.060330] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22274.062515] libcrc32c sunrpc [22274.069640] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22274.071477] intel_rapl_msr [22274.078601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22274.081565] intel_rapl_common [22274.089643] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22274.092434] sb_edac x86_pkg_temp_thermal [22274.098169] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22274.101236] coretemp kvm_intel [22274.108364] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22274.112366] kvm [22274.119491] Call Trace: [22274.119492] [22274.122629] irqbypass i2c_i801 [22274.129753] ? watchdog_timer_fn+0x1b4/0x220 [22274.131591] iTCO_wdt iTCO_vendor_support [22274.134035] ? __hrtimer_run_queues+0x1d3/0x360 [22274.136045] i2c_smbus crct10dif_pclmul [22274.139185] ? hrtimer_interrupt+0x104/0x240 [22274.143445] joydev crc32_pclmul [22274.147451] ? __local_bh_enable+0x50/0x60 [22274.151973] ipmi_ssif [22274.155806] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22274.160070] ghash_clmulni_intel mei_me [22274.163293] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22274.167383] mei lpc_ich [22274.169742] [22274.174959] sha512_ssse3 acpi_ipmi [22274.178789] [22274.178790] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22274.183748] mfd_core wmi [22274.186281] ? queued_spin_lock_slowpath+0x16/0x2e0 [22274.188375] ipmi_si [22274.191862] do_raw_spin_lock+0xb5/0xc0 [22274.193961] ipmi_devintf [22274.199271] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22274.201887] ipmi_msghandler ioatdma [22274.206759] try_get_cap_refs+0x159/0x880 [ceph] [22274.208960] acpi_pad acpi_power_meter [22274.212795] __ceph_get_caps+0x4a1/0x760 [ceph] [22274.215407] ixgbe igb [22274.220194] ? __file_remove_privs+0x89/0x110 [22274.223762] mdio [22274.228374] ? wake_bit_function+0x70/0x70 [22274.232117] crc32c_intel nvme [22274.236644] ? ceph_write_iter+0x260/0x770 [ceph] [22274.238997] ptp nvme_core [22274.243349] ceph_write_iter+0x260/0x770 [ceph] [22274.245272] i2c_algo_bit pps_core [22274.249366] ? lock_is_held_type+0xaa/0x120 [22274.252415] dca fuse [22274.257114] vfs_write+0x324/0x4f0 [22274.259816] [22274.259817] irq event stamp: 100602934 [22274.264345] ksys_pwrite64+0x65/0xa0 [22274.267738] hardirqs last enabled at (100602933): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22274.271915] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22274.274185] hardirqs last disabled at (100602934): [] sysvec_apic_timer_interrupt+0xe/0x90 [22274.277584] do_syscall_64+0x38/0x80 [22274.279072] softirqs last enabled at (100602932): [] __do_softirq+0x2be/0x371 [22274.282819] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22274.286389] softirqs last disabled at (100602927): [] irq_exit_rcu+0xb0/0xd0 [22274.296634] RIP: 0033:0x7f48f0eeb93b [22274.301764] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22274.311576] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22274.315144] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22274.323916] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22274.328961] Workqueue: events delayed_work [ceph] [22274.337556] ORIG_RAX: 0000000000000012 [22274.341127] [22274.350507] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22274.369243] RIP: 0010:_raw_spin_unlock+0x2a/0x40 [22274.376713] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22274.381931] Code: f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 47 7e <74> 02 5b c3 0f 1f 44 00 00 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 [22274.386629] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22274.390459] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [22274.391962] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22274.399083] [22274.399084] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [22274.403694] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22274.410816] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [22274.429560] [22274.436679] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [22274.481193] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [22274.488316] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22274.495442] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22274.503518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22274.509255] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22274.516380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22274.523503] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22274.530629] Call Trace: [22274.533072] [22274.535083] ? watchdog_timer_fn+0x1b4/0x220 [22274.539349] ? __hrtimer_run_queues+0x1d3/0x360 [22274.543876] ? hrtimer_interrupt+0x104/0x240 [22274.548144] ? __local_bh_enable+0x50/0x60 [22274.552250] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22274.557471] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22274.562429] [22274.564525] [22274.566622] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22274.571929] ? _raw_spin_unlock+0x2a/0x40 [22274.575942] igrab+0x43/0x50 [22274.578842] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22274.583991] delayed_work+0x1d8/0x280 [ceph] [22274.588275] ? process_scheduled_works+0x22a/0x510 [22274.593065] process_scheduled_works+0x2a4/0x510 [22274.597680] worker_thread+0x188/0x340 [22274.601427] ? rescuer_thread+0x310/0x310 [22274.605431] kthread+0xf5/0x130 [22274.608570] ? kthread_complete_and_exit+0x20/0x20 [22274.613355] ret_from_fork+0x31/0x40 [22274.616933] ? kthread_complete_and_exit+0x20/0x20 [22274.621717] ret_from_fork_asm+0x11/0x20 [22274.625640] [22301.849063] watchdog: BUG: soft lockup - CPU#0 stuck for 9656s! [dbench:78683] [22301.856281] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22301.861062] watchdog: BUG: soft lockup - CPU#7 stuck for 18653s! [kworker/7:4:78808] [22301.911735] irq event stamp: 34391922 [22301.919471] Modules linked in: ceph libceph dns_resolver fscache netfs veth [22301.923127] hardirqs last enabled at (34391921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22301.930087] nft_chain_nat xt_MASQUERADE [22301.940245] hardirqs last disabled at (34391922): [] sysvec_apic_timer_interrupt+0xe/0x90 [22301.944162] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [22301.953888] softirqs last enabled at (34391874): [] __do_softirq+0x2be/0x371 [22301.960672] nf_tables nfnetlink bridge stp [22301.969357] softirqs last disabled at (34391869): [] irq_exit_rcu+0xb0/0xd0 [22301.969359] llc [22301.969361] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22301.973534] xfs libcrc32c [22301.982044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22301.983882] sunrpc intel_rapl_msr [22301.992827] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22301.995529] intel_rapl_common sb_edac [22302.003002] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22302.006399] x86_pkg_temp_thermal [22302.011875] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22302.015621] coretemp kvm_intel [22302.034358] [22302.034359] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22302.037670] kvm irqbypass [22302.042886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22302.046024] i2c_i801 iTCO_wdt [22302.047514] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22302.054637] iTCO_vendor_support [22302.057343] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22302.064467] i2c_smbus [22302.067518] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22302.074641] crct10dif_pclmul joydev [22302.077866] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22302.084990] crc32_pclmul ipmi_ssif [22302.087346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22302.094472] ghash_clmulni_intel mei_me [22302.098043] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22302.106120] mei lpc_ich [22302.109605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22302.115341] sha512_ssse3 [22302.119172] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22302.126296] acpi_ipmi mfd_core [22302.128826] Call Trace: [22302.128828] [22302.135951] wmi ipmi_si [22302.138570] ? watchdog_timer_fn+0x1b4/0x220 [22302.145692] ipmi_devintf ipmi_msghandler [22302.148831] ? __hrtimer_run_queues+0x1d3/0x360 [22302.151273] ioatdma acpi_pad [22302.153289] ? hrtimer_interrupt+0x104/0x240 [22302.155814] acpi_power_meter ixgbe [22302.160079] ? __local_bh_enable+0x50/0x60 [22302.164084] igb mdio [22302.168610] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22302.171571] crc32c_intel nvme [22302.175837] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22302.179319] ptp nvme_core [22302.183410] [22302.185680] i2c_algo_bit pps_core [22302.190899] [22302.190900] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22302.193948] dca fuse [22302.198910] ? queued_spin_lock_slowpath+0x18/0x2e0 [22302.201610] [22302.201611] irq event stamp: 100823898 [22302.203711] do_raw_spin_lock+0xb5/0xc0 [22302.207106] hardirqs last enabled at (100823897): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22302.209205] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22302.214508] hardirqs last disabled at (100823898): [] sysvec_apic_timer_interrupt+0xe/0x90 [22302.216780] try_get_cap_refs+0x159/0x880 [ceph] [22302.221648] softirqs last enabled at (100823896): [] __do_softirq+0x2be/0x371 [22302.223145] __ceph_get_caps+0x4a1/0x760 [ceph] [22302.226883] softirqs last disabled at (100823891): [] irq_exit_rcu+0xb0/0xd0 [22302.230718] ? __file_remove_privs+0x89/0x110 [22302.240959] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22302.245759] ? wake_bit_function+0x70/0x70 [22302.255571] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22302.260183] ? ceph_write_iter+0x260/0x770 [ceph] [22302.268952] Workqueue: events delayed_work [ceph] [22302.273475] ceph_write_iter+0x260/0x770 [ceph] [22302.282074] [22302.282075] RIP: 0010:lock_acquire+0xf0/0x2b0 [22302.286429] ? lock_is_held_type+0xaa/0x120 [22302.295803] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22302.299894] vfs_write+0x324/0x4f0 [22302.307363] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22302.312065] ksys_pwrite64+0x65/0xa0 [22302.316758] [22302.316759] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22302.321284] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22302.322773] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22302.327124] do_syscall_64+0x38/0x80 [22302.331300] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22302.350039] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22302.353436] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22302.358654] RIP: 0033:0x7f48f0eeb93b [22302.362223] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22302.362225] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22302.363715] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22302.370839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22302.375968] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22302.383095] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22302.386664] ORIG_RAX: 0000000000000012 [22302.393788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22302.398833] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22302.405956] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22302.409528] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22302.416652] Call Trace: [22302.416654] [22302.424745] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22302.443485] ? watchdog_timer_fn+0x1b4/0x220 [22302.449221] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22302.454443] ? __hrtimer_run_queues+0x1d3/0x360 [22302.461581] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22302.465414] ? hrtimer_interrupt+0x104/0x240 [22302.472542] [22302.479658] ? __local_bh_enable+0x50/0x60 [22302.539065] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22302.544288] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22302.549246] [22302.551343] [22302.553443] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22302.558776] ? lock_acquire+0xf0/0x2b0 [22302.562526] ? igrab+0x43/0x50 [22302.565584] ? lock_release+0xc5/0x260 [22302.569330] _raw_spin_lock+0x30/0x40 [22302.572995] ? igrab+0x1d/0x50 [22302.576044] igrab+0x1d/0x50 [22302.578922] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22302.584074] delayed_work+0x1d8/0x280 [ceph] [22302.588353] ? process_scheduled_works+0x22a/0x510 [22302.593144] process_scheduled_works+0x2a4/0x510 [22302.597759] worker_thread+0x188/0x340 [22302.601509] ? rescuer_thread+0x310/0x310 [22302.605520] kthread+0xf5/0x130 [22302.608659] ? kthread_complete_and_exit+0x20/0x20 [22302.613444] ret_from_fork+0x31/0x40 [22302.617040] ? kthread_complete_and_exit+0x20/0x20 [22302.621822] ret_from_fork_asm+0x11/0x20 [22302.625745] [22329.849564] watchdog: BUG: soft lockup - CPU#0 stuck for 9682s! [dbench:78683] [22329.856777] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22329.861564] watchdog: BUG: soft lockup - CPU#7 stuck for 18679s! [kworker/7:4:78808] [22329.912234] irq event stamp: 34450076 [22329.912235] hardirqs last enabled at (34450075): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22329.919965] Modules linked in: [22329.923624] hardirqs last disabled at (34450076): [] sysvec_apic_timer_interrupt+0xe/0x90 [22329.933782] ceph [22329.936833] softirqs last enabled at (34450040): [] __do_softirq+0x2be/0x371 [22329.946556] libceph [22329.948480] softirqs last disabled at (34450035): [] irq_exit_rcu+0xb0/0xd0 [22329.957164] dns_resolver [22329.959350] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22329.967858] fscache netfs [22329.970476] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22329.979421] veth nft_chain_nat [22329.982125] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [22329.989595] xt_MASQUERADE [22329.992734] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [22329.998211] nf_nat nf_conntrack [22330.000914] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22330.019653] nf_defrag_ipv6 nf_defrag_ipv4 [22330.022876] [22330.022877] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22330.028093] nft_compat nf_tables [22330.032185] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22330.033674] nfnetlink bridge [22330.040800] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22330.044110] stp llc [22330.051233] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22330.054216] xfs libcrc32c [22330.061340] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22330.063523] sunrpc intel_rapl_msr [22330.070648] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22330.073352] intel_rapl_common sb_edac [22330.080475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22330.083874] x86_pkg_temp_thermal coretemp [22330.091961] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22330.095703] kvm_intel kvm [22330.101440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22330.105530] irqbypass i2c_i801 [22330.112656] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22330.115359] iTCO_wdt iTCO_vendor_support [22330.122484] Call Trace: [22330.125621] i2c_smbus crct10dif_pclmul [22330.132746] [22330.136750] joydev crc32_pclmul [22330.139195] ? watchdog_timer_fn+0x1b4/0x220 [22330.143025] ipmi_ssif ghash_clmulni_intel [22330.145037] ? __hrtimer_run_queues+0x1d3/0x360 [22330.148259] mei_me mei [22330.152526] ? hrtimer_interrupt+0x104/0x240 [22330.156614] lpc_ich sha512_ssse3 [22330.161137] ? __local_bh_enable+0x50/0x60 [22330.163583] acpi_ipmi mfd_core [22330.167849] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22330.171157] wmi ipmi_si [22330.175248] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22330.178385] ipmi_devintf ipmi_msghandler [22330.183603] [22330.186132] ioatdma acpi_pad [22330.191091] [22330.191093] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22330.195094] acpi_power_meter ixgbe [22330.197197] ? queued_spin_lock_slowpath+0x2b/0x2e0 [22330.200157] igb [22330.202257] do_raw_spin_lock+0xb5/0xc0 [22330.207556] mdio [22330.211043] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22330.215911] crc32c_intel nvme [22330.217754] try_get_cap_refs+0x159/0x880 [ceph] [22330.221581] ptp nvme_core [22330.223509] __ceph_get_caps+0x4a1/0x760 [ceph] [22330.228288] i2c_algo_bit [22330.231344] ? __file_remove_privs+0x89/0x110 [22330.235964] pps_core dca [22330.238671] ? wake_bit_function+0x70/0x70 [22330.243196] fuse [22330.245814] ? ceph_write_iter+0x260/0x770 [ceph] [22330.250164] irq event stamp: 101044656 [22330.252782] ceph_write_iter+0x260/0x770 [ceph] [22330.256872] hardirqs last enabled at (101044655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22330.258799] ? lock_is_held_type+0xaa/0x120 [22330.263493] hardirqs last disabled at (101044656): [] sysvec_apic_timer_interrupt+0xe/0x90 [22330.267248] vfs_write+0x324/0x4f0 [22330.271770] softirqs last enabled at (101044654): [] __do_softirq+0x2be/0x371 [22330.282020] ksys_pwrite64+0x65/0xa0 [22330.286193] softirqs last disabled at (101044649): [] irq_exit_rcu+0xb0/0xd0 [22330.296002] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22330.299401] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22330.308172] do_syscall_64+0x38/0x80 [22330.311741] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22330.320340] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22330.325469] Workqueue: events delayed_work [ceph] [22330.334849] RIP: 0033:0x7f48f0eeb93b [22330.338418] [22330.345888] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22330.350934] RIP: 0010:lock_acquire+0xf0/0x2b0 [22330.355630] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22330.359200] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22330.360692] ORIG_RAX: 0000000000000012 [22330.379430] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22330.383780] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22330.388997] [22330.388998] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22330.407736] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22330.411567] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22330.416783] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22330.423907] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22330.425399] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22330.432522] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22330.439647] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22330.446770] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22330.453901] [22330.461018] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22330.499778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22330.505515] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22330.512639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22330.519764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22330.526886] Call Trace: [22330.529333] [22330.531343] ? watchdog_timer_fn+0x1b4/0x220 [22330.535607] ? __hrtimer_run_queues+0x1d3/0x360 [22330.540134] ? hrtimer_interrupt+0x104/0x240 [22330.544402] ? __local_bh_enable+0x50/0x60 [22330.548497] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22330.553720] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22330.558677] [22330.560773] [22330.562872] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22330.568180] ? lock_acquire+0xf0/0x2b0 [22330.571930] ? igrab+0x43/0x50 [22330.574981] ? lock_release+0xc5/0x260 [22330.578724] _raw_spin_lock+0x30/0x40 [22330.582380] ? igrab+0x1d/0x50 [22330.585431] igrab+0x1d/0x50 [22330.588311] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22330.593460] delayed_work+0x1d8/0x280 [ceph] [22330.597741] ? process_scheduled_works+0x22a/0x510 [22330.602531] process_scheduled_works+0x2a4/0x510 [22330.607147] worker_thread+0x188/0x340 [22330.610896] ? rescuer_thread+0x310/0x310 [22330.614899] kthread+0xf5/0x130 [22330.618037] ? kthread_complete_and_exit+0x20/0x20 [22330.622821] ret_from_fork+0x31/0x40 [22330.626398] ? kthread_complete_and_exit+0x20/0x20 [22330.631185] ret_from_fork_asm+0x11/0x20 [22330.635108] [22357.850066] watchdog: BUG: soft lockup - CPU#0 stuck for 9708s! [dbench:78683] [22357.857283] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22357.862065] watchdog: BUG: soft lockup - CPU#7 stuck for 18705s! [kworker/7:4:78808] [22357.912743] irq event stamp: 34508236 [22357.920472] Modules linked in: ceph libceph dns_resolver fscache [22357.924129] hardirqs last enabled at (34508235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22357.930134] netfs veth [22357.940293] hardirqs last disabled at (34508236): [] sysvec_apic_timer_interrupt+0xe/0x90 [22357.940294] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [22357.952460] softirqs last enabled at (34508188): [] __do_softirq+0x2be/0x371 [22357.961664] nf_tables nfnetlink bridge stp llc [22357.970348] softirqs last disabled at (34508183): [] irq_exit_rcu+0xb0/0xd0 [22357.974873] xfs libcrc32c sunrpc [22357.983384] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22357.983385] intel_rapl_msr intel_rapl_common sb_edac [22357.995638] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22358.000681] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [22358.008155] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22358.014323] i2c_i801 iTCO_wdt iTCO_vendor_support [22358.019801] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22358.019802] i2c_smbus [22358.024584] crct10dif_pclmul [22358.043323] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22358.045679] joydev crc32_pclmul [22358.048645] [22358.048646] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22358.053861] ipmi_ssif ghash_clmulni_intel [22358.057087] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22358.058578] mei_me [22358.065700] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22358.069791] mei lpc_ich [22358.076916] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22358.079014] sha512_ssse3 acpi_ipmi [22358.086138] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22358.088668] mfd_core wmi [22358.095793] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22358.099276] ipmi_si ipmi_devintf [22358.106400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22358.109016] ipmi_msghandler [22358.117095] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22358.120406] ioatdma acpi_pad [22358.126142] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22358.129020] acpi_power_meter [22358.136143] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22358.139110] ixgbe igb [22358.146234] Call Trace: [22358.146236] [22358.149198] mdio crc32c_intel [22358.156323] ? watchdog_timer_fn+0x1b4/0x220 [22358.158680] nvme [22358.161124] ? __hrtimer_run_queues+0x1d3/0x360 [22358.163132] ptp nvme_core [22358.166186] ? hrtimer_interrupt+0x104/0x240 [22358.170465] i2c_algo_bit [22358.172389] ? __local_bh_enable+0x50/0x60 [22358.176913] pps_core dca [22358.179619] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22358.183881] fuse [22358.186501] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22358.190589] irq event stamp: 101265670 [22358.193206] [22358.198425] hardirqs last enabled at (101265669): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22358.200349] [22358.200351] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22358.205307] hardirqs last disabled at (101265670): [] sysvec_apic_timer_interrupt+0xe/0x90 [22358.209055] ? queued_spin_lock_slowpath+0x18/0x2e0 [22358.211147] softirqs last enabled at (101265668): [] __do_softirq+0x2be/0x371 [22358.221393] do_raw_spin_lock+0xb5/0xc0 [22358.223490] softirqs last disabled at (101265663): [] irq_exit_rcu+0xb0/0xd0 [22358.228797] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22358.238604] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22358.243476] try_get_cap_refs+0x159/0x880 [ceph] [22358.252245] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22358.256082] __ceph_get_caps+0x4a1/0x760 [ceph] [22358.264674] Workqueue: events delayed_work [ceph] [22358.269472] ? __file_remove_privs+0x89/0x110 [22358.278844] [22358.278845] RIP: 0010:lock_acquire+0xf0/0x2b0 [22358.283465] ? wake_bit_function+0x70/0x70 [22358.290935] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22358.295470] ? ceph_write_iter+0x260/0x770 [ceph] [22358.300165] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22358.304515] ceph_write_iter+0x260/0x770 [ceph] [22358.306006] [22358.306007] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22358.310361] ? lock_is_held_type+0xaa/0x120 [22358.314463] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22358.333204] vfs_write+0x324/0x4f0 [22358.337899] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22358.343122] ksys_pwrite64+0x65/0xa0 [22358.347641] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22358.349133] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22358.356255] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22358.360434] do_syscall_64+0x38/0x80 [22358.367559] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22358.370955] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22358.378079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22358.381650] RIP: 0033:0x7f48f0eeb93b [22358.388774] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22358.393907] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22358.401029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22358.404599] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22358.412676] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22358.417722] ORIG_RAX: 0000000000000012 [22358.423468] Call Trace: [22358.423470] [22358.427039] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22358.434162] ? watchdog_timer_fn+0x1b4/0x220 [22358.452901] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22358.460026] ? __hrtimer_run_queues+0x1d3/0x360 [22358.465241] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22358.472368] ? hrtimer_interrupt+0x104/0x240 [22358.476197] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22358.478640] ? __local_bh_enable+0x50/0x60 [22358.480651] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22358.487778] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22358.492045] [22358.499165] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22358.545775] [22358.547872] [22358.549969] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22358.555278] ? lock_acquire+0xf0/0x2b0 [22358.559026] ? igrab+0x43/0x50 [22358.562077] ? lock_release+0xc5/0x260 [22358.565823] _raw_spin_lock+0x30/0x40 [22358.569488] ? igrab+0x1d/0x50 [22358.572538] igrab+0x1d/0x50 [22358.575417] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22358.580565] delayed_work+0x1d8/0x280 [ceph] [22358.584848] ? process_scheduled_works+0x22a/0x510 [22358.589636] process_scheduled_works+0x2a4/0x510 [22358.594254] worker_thread+0x188/0x340 [22358.598000] ? rescuer_thread+0x310/0x310 [22358.602004] kthread+0xf5/0x130 [22358.605144] ? kthread_complete_and_exit+0x20/0x20 [22358.609935] ret_from_fork+0x31/0x40 [22358.613516] ? kthread_complete_and_exit+0x20/0x20 [22358.618299] ret_from_fork_asm+0x11/0x20 [22358.622232] [22360.383477] rcu: INFO: rcu_preempt self-detected stall on CPU [22360.389236] rcu: 7-....: (10121585 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/10120313 fqs=5084829 [22360.399835] rcu: (t=10423586 jiffies g=1309609 q=348043 ncpus=8) [22360.405918] Sending NMI from CPU 7 to CPUs 0: [22360.410271] NMI backtrace for cpu 0 [22360.410272] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22360.410274] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22360.410275] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22360.410278] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22360.410279] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22360.410281] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22360.410281] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22360.410282] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22360.410283] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22360.410284] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22360.410284] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22360.410286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22360.410286] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22360.410287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22360.410288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22360.410289] Call Trace: [22360.410290] [22360.410291] ? nmi_cpu_backtrace+0xad/0x120 [22360.410294] ? nmi_cpu_backtrace_handler+0x11/0x20 [22360.410296] ? nmi_handle+0xd4/0x260 [22360.410300] ? default_do_nmi+0x49/0xf0 [22360.410302] ? exc_nmi+0x11e/0x150 [22360.410303] ? end_repeat_nmi+0x16/0x67 [22360.410308] ? queued_spin_lock_slowpath+0x18/0x2e0 [22360.410310] ? queued_spin_lock_slowpath+0x18/0x2e0 [22360.410313] ? queued_spin_lock_slowpath+0x18/0x2e0 [22360.410315] [22360.410315] [22360.410316] do_raw_spin_lock+0xb5/0xc0 [22360.410319] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22360.410333] try_get_cap_refs+0x159/0x880 [ceph] [22360.410348] __ceph_get_caps+0x4a1/0x760 [ceph] [22360.410360] ? __file_remove_privs+0x89/0x110 [22360.410362] ? wake_bit_function+0x70/0x70 [22360.410365] ? ceph_write_iter+0x260/0x770 [ceph] [22360.410375] ceph_write_iter+0x260/0x770 [ceph] [22360.410386] ? lock_is_held_type+0xaa/0x120 [22360.410389] vfs_write+0x324/0x4f0 [22360.410393] ksys_pwrite64+0x65/0xa0 [22360.410395] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22360.410398] do_syscall_64+0x38/0x80 [22360.410400] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22360.410402] RIP: 0033:0x7f48f0eeb93b [22360.410403] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22360.410404] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [22360.410406] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22360.410407] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22360.410407] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22360.410408] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22360.410409] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22360.410412] [22360.411269] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22360.714821] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22360.722292] Workqueue: events delayed_work [ceph] [22360.727006] RIP: 0010:lock_acquire+0xf0/0x2b0 [22360.731356] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22360.750094] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22360.755311] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22360.762436] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22360.769559] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22360.776685] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22360.783808] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22360.790931] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22360.799010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22360.804748] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22360.811871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22360.818997] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22360.826119] Call Trace: [22360.828564] [22360.830575] ? rcu_dump_cpu_stacks+0xd1/0x180 [22360.834927] ? rcu_sched_clock_irq+0x633/0x1390 [22360.839451] ? lock_release+0xc5/0x260 [22360.843194] ? lock_is_held_type+0xaa/0x120 [22360.847371] ? lock_is_held_type+0xaa/0x120 [22360.851549] ? tick_sched_do_timer+0xa0/0xa0 [22360.855822] ? rcu_read_lock_sched_held+0x56/0x90 [22360.860519] ? tick_sched_do_timer+0xa0/0xa0 [22360.864784] ? update_process_times+0x5f/0x90 [22360.869133] ? tick_sched_handle.isra.16+0x1e/0x50 [22360.873919] ? tick_sched_timer+0x6f/0x80 [22360.877922] ? __hrtimer_run_queues+0x1d3/0x360 [22360.882449] ? hrtimer_interrupt+0x104/0x240 [22360.886718] ? __local_bh_enable+0x50/0x60 [22360.890811] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22360.896035] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22360.900994] [22360.903090] [22360.905187] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22360.910497] ? lock_acquire+0xf0/0x2b0 [22360.914247] ? igrab+0x43/0x50 [22360.917305] ? lock_release+0xc5/0x260 [22360.921050] _raw_spin_lock+0x30/0x40 [22360.924714] ? igrab+0x1d/0x50 [22360.927765] igrab+0x1d/0x50 [22360.930643] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22360.935794] delayed_work+0x1d8/0x280 [ceph] [22360.940073] ? process_scheduled_works+0x22a/0x510 [22360.944856] process_scheduled_works+0x2a4/0x510 [22360.949472] worker_thread+0x188/0x340 [22360.953219] ? rescuer_thread+0x310/0x310 [22360.957222] kthread+0xf5/0x130 [22360.960361] ? kthread_complete_and_exit+0x20/0x20 [22360.965147] ret_from_fork+0x31/0x40 [22360.968725] ? kthread_complete_and_exit+0x20/0x20 [22360.973517] ret_from_fork_asm+0x11/0x20 [22360.977441] [22385.850566] watchdog: BUG: soft lockup - CPU#0 stuck for 9734s! [dbench:78683] [22385.857778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22385.862565] watchdog: BUG: soft lockup - CPU#7 stuck for 18731s! [kworker/7:4:78808] [22385.913234] irq event stamp: 34566366 [22385.913235] hardirqs last enabled at (34566365): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22385.920967] Modules linked in: [22385.924623] hardirqs last disabled at (34566366): [] sysvec_apic_timer_interrupt+0xe/0x90 [22385.934780] ceph [22385.937832] softirqs last enabled at (34566322): [] __do_softirq+0x2be/0x371 [22385.947557] libceph dns_resolver [22385.949481] softirqs last disabled at (34566317): [] irq_exit_rcu+0xb0/0xd0 [22385.958164] fscache [22385.961477] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22385.969985] netfs veth [22385.972169] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22385.981115] nft_chain_nat [22385.983560] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22385.991028] xt_MASQUERADE nf_nat [22385.993733] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22385.999211] nf_conntrack nf_defrag_ipv6 [22386.002521] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22386.021260] nf_defrag_ipv4 nft_compat [22386.025175] [22386.025176] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22386.030395] nf_tables [22386.034138] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22386.035629] nfnetlink [22386.042752] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22386.045110] bridge stp [22386.052235] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22386.054608] llc xfs [22386.061734] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22386.064178] libcrc32c sunrpc [22386.071301] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22386.073484] intel_rapl_msr intel_rapl_common [22386.080610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22386.083573] sb_edac x86_pkg_temp_thermal [22386.091651] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22386.096002] coretemp [22386.101741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22386.105745] kvm_intel [22386.112867] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22386.115137] kvm irqbypass [22386.122262] Call Trace: [22386.124620] i2c_i801 [22386.131746] [22386.134448] iTCO_wdt iTCO_vendor_support [22386.136892] ? watchdog_timer_fn+0x1b4/0x220 [22386.139163] i2c_smbus crct10dif_pclmul [22386.141175] ? __hrtimer_run_queues+0x1d3/0x360 [22386.145176] joydev crc32_pclmul [22386.149444] ? hrtimer_interrupt+0x104/0x240 [22386.153271] ipmi_ssif ghash_clmulni_intel [22386.157795] ? __local_bh_enable+0x50/0x60 [22386.161019] mei_me mei [22386.165286] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22386.169375] lpc_ich [22386.173465] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22386.175909] sha512_ssse3 acpi_ipmi [22386.181127] [22386.183311] mfd_core wmi [22386.188268] [22386.191753] ipmi_si ipmi_devintf [22386.193852] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22386.196468] ipmi_msghandler ioatdma [22386.198569] ? queued_spin_lock_slowpath+0x16/0x2e0 [22386.201876] acpi_pad acpi_power_meter [22386.207182] do_raw_spin_lock+0xb5/0xc0 [22386.210750] ixgbe igb [22386.215622] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22386.219365] mdio crc32c_intel [22386.223199] try_get_cap_refs+0x159/0x880 [ceph] [22386.225554] nvme ptp [22386.230343] __ceph_get_caps+0x4a1/0x760 [ceph] [22386.233389] nvme_core i2c_algo_bit [22386.238003] ? __file_remove_privs+0x89/0x110 [22386.240269] pps_core dca [22386.244793] ? wake_bit_function+0x70/0x70 [22386.248277] fuse [22386.252633] ? ceph_write_iter+0x260/0x770 [ceph] [22386.255252] irq event stamp: 101482010 [22386.259345] ceph_write_iter+0x260/0x770 [ceph] [22386.261268] hardirqs last enabled at (101482009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22386.265970] ? lock_is_held_type+0xaa/0x120 [22386.269711] hardirqs last disabled at (101482010): [] sysvec_apic_timer_interrupt+0xe/0x90 [22386.274248] vfs_write+0x324/0x4f0 [22386.284488] softirqs last enabled at (101482008): [] __do_softirq+0x2be/0x371 [22386.288671] ksys_pwrite64+0x65/0xa0 [22386.298477] softirqs last disabled at (101482003): [] irq_exit_rcu+0xb0/0xd0 [22386.301874] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22386.310644] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22386.314218] do_syscall_64+0x38/0x80 [22386.322813] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22386.327963] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22386.337338] Workqueue: events delayed_work [ceph] [22386.340910] RIP: 0033:0x7f48f0eeb93b [22386.348380] [22386.348381] RIP: 0010:lock_acquire+0xf0/0x2b0 [22386.353425] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22386.358120] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22386.361691] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22386.363182] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22386.367534] ORIG_RAX: 0000000000000012 [22386.367535] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22386.386272] [22386.386272] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22386.405009] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22386.410227] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22386.415460] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22386.419291] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22386.426415] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22386.427907] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22386.435030] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22386.442155] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22386.449283] [22386.456404] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22386.502286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22386.508023] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22386.515146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22386.522272] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22386.529397] Call Trace: [22386.531839] [22386.533852] ? watchdog_timer_fn+0x1b4/0x220 [22386.538116] ? __hrtimer_run_queues+0x1d3/0x360 [22386.542643] ? hrtimer_interrupt+0x104/0x240 [22386.546910] ? __local_bh_enable+0x50/0x60 [22386.551006] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22386.556229] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22386.561203] [22386.563300] [22386.565397] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22386.570707] ? lock_acquire+0xf0/0x2b0 [22386.574456] ? igrab+0x43/0x50 [22386.577507] ? lock_release+0xc5/0x260 [22386.581253] _raw_spin_lock+0x30/0x40 [22386.584915] ? igrab+0x1d/0x50 [22386.587971] igrab+0x1d/0x50 [22386.590853] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22386.596003] delayed_work+0x1d8/0x280 [ceph] [22386.600283] ? process_scheduled_works+0x22a/0x510 [22386.605065] process_scheduled_works+0x2a4/0x510 [22386.609681] worker_thread+0x188/0x340 [22386.613431] ? rescuer_thread+0x310/0x310 [22386.617433] kthread+0xf5/0x130 [22386.620572] ? kthread_complete_and_exit+0x20/0x20 [22386.625357] ret_from_fork+0x31/0x40 [22386.628935] ? kthread_complete_and_exit+0x20/0x20 [22386.633718] ret_from_fork_asm+0x11/0x20 [22386.637643] [22413.851059] watchdog: BUG: soft lockup - CPU#0 stuck for 9760s! [dbench:78683] [22413.858273] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22413.863058] watchdog: BUG: soft lockup - CPU#7 stuck for 18757s! [kworker/7:4:78808] [22413.913732] irq event stamp: 34624576 [22413.913733] hardirqs last enabled at (34624575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22413.921462] Modules linked in: [22413.925119] hardirqs last disabled at (34624576): [] sysvec_apic_timer_interrupt+0xe/0x90 [22413.935279] ceph [22413.938327] softirqs last enabled at (34624538): [] __do_softirq+0x2be/0x371 [22413.948052] libceph [22413.949977] softirqs last disabled at (34624533): [] irq_exit_rcu+0xb0/0xd0 [22413.958661] dns_resolver fscache [22413.960847] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22413.969357] netfs [22413.972667] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22413.981611] veth nft_chain_nat [22413.983622] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22413.991093] xt_MASQUERADE nf_nat [22413.994229] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22413.999705] nf_conntrack nf_defrag_ipv6 [22414.003017] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22414.021756] nf_defrag_ipv4 nft_compat [22414.025672] [22414.025673] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22414.030890] nf_tables nfnetlink [22414.034634] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22414.036125] bridge stp [22414.043250] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22414.046473] llc xfs [22414.053596] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22414.056042] libcrc32c [22414.063166] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22414.065350] sunrpc intel_rapl_msr [22414.072474] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22414.074830] intel_rapl_common sb_edac [22414.081955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22414.085353] x86_pkg_temp_thermal coretemp [22414.093431] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22414.097192] kvm_intel [22414.102928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22414.107018] kvm irqbypass [22414.114143] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22414.116502] i2c_i801 iTCO_wdt [22414.123626] Call Trace: [22414.123628] [22414.126328] iTCO_vendor_support i2c_smbus [22414.133459] ? watchdog_timer_fn+0x1b4/0x220 [22414.136504] crct10dif_pclmul joydev [22414.138949] ? __hrtimer_run_queues+0x1d3/0x360 [22414.140960] crc32_pclmul ipmi_ssif [22414.145053] ? hrtimer_interrupt+0x104/0x240 [22414.149313] ghash_clmulni_intel [22414.152884] ? __local_bh_enable+0x50/0x60 [22414.157407] mei_me mei [22414.160896] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22414.165155] lpc_ich sha512_ssse3 [22414.168381] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22414.172472] acpi_ipmi mfd_core [22414.174916] [22414.180133] wmi ipmi_si [22414.183457] [22414.188417] ipmi_devintf ipmi_msghandler [22414.191555] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22414.193652] ioatdma acpi_pad [22414.196187] ? queued_spin_lock_slowpath+0x18/0x2e0 [22414.198282] acpi_power_meter ixgbe [22414.202288] do_raw_spin_lock+0xb5/0xc0 [22414.207590] igb mdio [22414.210554] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22414.215424] crc32c_intel nvme [22414.218911] try_get_cap_refs+0x159/0x880 [ceph] [22414.222754] ptp nvme_core [22414.225031] __ceph_get_caps+0x4a1/0x760 [ceph] [22414.229812] i2c_algo_bit [22414.232867] ? __file_remove_privs+0x89/0x110 [22414.237473] pps_core [22414.240176] ? wake_bit_function+0x70/0x70 [22414.244700] dca fuse [22414.247320] ? ceph_write_iter+0x260/0x770 [ceph] [22414.251669] [22414.251670] irq event stamp: 101703006 [22414.253939] ceph_write_iter+0x260/0x770 [ceph] [22414.258030] hardirqs last enabled at (101703005): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22414.260304] ? lock_is_held_type+0xaa/0x120 [22414.264999] hardirqs last disabled at (101703006): [] sysvec_apic_timer_interrupt+0xe/0x90 [22414.266493] vfs_write+0x324/0x4f0 [22414.270234] softirqs last enabled at (101703004): [] __do_softirq+0x2be/0x371 [22414.274761] ksys_pwrite64+0x65/0xa0 [22414.285000] softirqs last disabled at (101702999): [] irq_exit_rcu+0xb0/0xd0 [22414.289181] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22414.298991] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22414.302389] do_syscall_64+0x38/0x80 [22414.311159] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22414.314739] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22414.323335] Workqueue: events delayed_work [ceph] [22414.328465] RIP: 0033:0x7f48f0eeb93b [22414.337842] [22414.337843] RIP: 0010:lock_acquire+0xf0/0x2b0 [22414.341416] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22414.348885] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22414.353928] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22414.358626] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22414.362197] ORIG_RAX: 0000000000000012 [22414.363687] [22414.363688] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22414.368038] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22414.386777] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22414.405514] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22414.410741] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22414.415958] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22414.419787] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22414.421278] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22414.428404] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22414.435526] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22414.442651] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22414.449780] [22414.456900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22414.508519] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22414.515643] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22414.522768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22414.529892] Call Trace: [22414.532338] [22414.534348] ? watchdog_timer_fn+0x1b4/0x220 [22414.538611] ? __hrtimer_run_queues+0x1d3/0x360 [22414.543138] ? hrtimer_interrupt+0x104/0x240 [22414.547407] ? __local_bh_enable+0x50/0x60 [22414.551500] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22414.556740] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22414.561698] [22414.563796] [22414.565893] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22414.571203] ? lock_acquire+0xf0/0x2b0 [22414.574951] ? igrab+0x43/0x50 [22414.578002] ? lock_release+0xc5/0x260 [22414.581748] _raw_spin_lock+0x30/0x40 [22414.585411] ? igrab+0x1d/0x50 [22414.588464] igrab+0x1d/0x50 [22414.591343] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22414.596489] delayed_work+0x1d8/0x280 [ceph] [22414.600771] ? process_scheduled_works+0x22a/0x510 [22414.605561] process_scheduled_works+0x2a4/0x510 [22414.610178] worker_thread+0x188/0x340 [22414.613926] ? rescuer_thread+0x310/0x310 [22414.617929] kthread+0xf5/0x130 [22414.621068] ? kthread_complete_and_exit+0x20/0x20 [22414.625852] ret_from_fork+0x31/0x40 [22414.629430] ? kthread_complete_and_exit+0x20/0x20 [22414.634216] ret_from_fork_asm+0x11/0x20 [22414.638146] [22441.851551] watchdog: BUG: soft lockup - CPU#0 stuck for 9786s! [dbench:78683] [22441.858770] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22441.863551] watchdog: BUG: soft lockup - CPU#7 stuck for 18783s! [kworker/7:4:78808] [22441.914228] irq event stamp: 34682830 [22441.914229] hardirqs last enabled at (34682829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22441.921958] Modules linked in: [22441.925617] hardirqs last disabled at (34682830): [] sysvec_apic_timer_interrupt+0xe/0x90 [22441.935775] ceph [22441.938824] softirqs last enabled at (34682792): [] __do_softirq+0x2be/0x371 [22441.948549] libceph [22441.950473] softirqs last disabled at (34682787): [] irq_exit_rcu+0xb0/0xd0 [22441.959157] dns_resolver [22441.961341] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22441.969851] fscache netfs [22441.972468] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22441.981412] veth nft_chain_nat [22441.984118] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22441.991588] xt_MASQUERADE [22441.994726] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22442.000202] nf_nat nf_conntrack [22442.002907] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22442.021646] nf_defrag_ipv6 [22442.024870] [22442.024870] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22442.030086] nf_defrag_ipv4 nft_compat [22442.032878] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22442.034367] nf_tables nfnetlink [22442.041492] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22442.045236] bridge stp [22442.052359] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22442.055584] llc [22442.062708] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22442.065152] xfs libcrc32c [22442.072277] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22442.074115] sunrpc intel_rapl_msr [22442.081238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22442.083947] intel_rapl_common [22442.092028] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22442.095425] sb_edac [22442.101162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22442.104227] x86_pkg_temp_thermal [22442.111356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22442.113540] coretemp kvm_intel [22442.120662] Call Trace: [22442.120664] [22442.123974] kvm irqbypass [22442.131098] ? watchdog_timer_fn+0x1b4/0x220 [22442.134236] i2c_i801 [22442.136679] ? __hrtimer_run_queues+0x1d3/0x360 [22442.138691] iTCO_wdt [22442.141398] ? hrtimer_interrupt+0x104/0x240 [22442.145657] iTCO_vendor_support [22442.147946] ? __local_bh_enable+0x50/0x60 [22442.152472] i2c_smbus [22442.154743] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22442.159006] crct10dif_pclmul joydev [22442.162231] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22442.166321] crc32_pclmul [22442.168678] [22442.173895] ipmi_ssif [22442.177466] [22442.177467] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22442.182422] ghash_clmulni_intel [22442.185044] ? queued_spin_lock_slowpath+0x18/0x2e0 [22442.187138] mei_me mei [22442.189497] do_raw_spin_lock+0xb5/0xc0 [22442.191591] lpc_ich sha512_ssse3 [22442.196897] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22442.200119] acpi_ipmi mfd_core [22442.204995] try_get_cap_refs+0x159/0x880 [ceph] [22442.207435] wmi ipmi_si [22442.211272] __ceph_get_caps+0x4a1/0x760 [ceph] [22442.214578] ipmi_devintf [22442.219365] ? __file_remove_privs+0x89/0x110 [22442.222499] ipmi_msghandler ioatdma [22442.227110] ? wake_bit_function+0x70/0x70 [22442.229639] acpi_pad acpi_power_meter [22442.234168] ? ceph_write_iter+0x260/0x770 [ceph] [22442.236781] ixgbe igb [22442.241132] ceph_write_iter+0x260/0x770 [ceph] [22442.244702] mdio [22442.248797] ? lock_is_held_type+0xaa/0x120 [22442.252538] crc32c_intel nvme [22442.257237] vfs_write+0x324/0x4f0 [22442.259593] ptp nvme_core [22442.264122] ksys_pwrite64+0x65/0xa0 [22442.266040] i2c_algo_bit pps_core [22442.270233] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22442.273286] dca fuse [22442.276684] do_syscall_64+0x38/0x80 [22442.279387] [22442.279388] irq event stamp: 101923820 [22442.282959] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22442.286356] hardirqs last enabled at (101923819): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22442.291486] RIP: 0033:0x7f48f0eeb93b [22442.293756] hardirqs last disabled at (101923820): [] sysvec_apic_timer_interrupt+0xe/0x90 [22442.297329] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22442.298819] softirqs last enabled at (101923818): [] __do_softirq+0x2be/0x371 [22442.302564] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22442.307607] softirqs last disabled at (101923813): [] irq_exit_rcu+0xb0/0xd0 [22442.317851] ORIG_RAX: 0000000000000012 [22442.321423] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22442.331232] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22442.349969] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22442.358741] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22442.363957] Workqueue: events delayed_work [ceph] [22442.372556] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22442.376385] [22442.376386] RIP: 0010:lock_release+0x42/0x260 [22442.385763] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22442.392888] Code: 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 <8b> 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 [22442.400358] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22442.407484] RSP: 0018:ffffc90002117d48 EFLAGS: 00000202 [22442.412185] [22442.419305] [22442.419306] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [22442.474150] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [22442.481273] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [22442.488397] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [22442.495522] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22442.502644] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22442.510725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22442.516461] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22442.523584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22442.530708] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22442.537833] Call Trace: [22442.540279] [22442.542289] ? watchdog_timer_fn+0x1b4/0x220 [22442.546551] ? __hrtimer_run_queues+0x1d3/0x360 [22442.551078] ? hrtimer_interrupt+0x104/0x240 [22442.555340] ? __local_bh_enable+0x50/0x60 [22442.559433] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22442.564658] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22442.569615] [22442.571712] [22442.573808] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22442.579114] ? igrab+0x43/0x50 [22442.582166] ? igrab+0x43/0x50 [22442.585215] ? lock_release+0x42/0x260 [22442.588976] ? lock_release+0x1d7/0x260 [22442.592809] _raw_spin_unlock+0x1b/0x40 [22442.596647] igrab+0x43/0x50 [22442.599525] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22442.604674] delayed_work+0x1d8/0x280 [ceph] [22442.608961] ? process_scheduled_works+0x22a/0x510 [22442.613747] process_scheduled_works+0x2a4/0x510 [22442.618361] worker_thread+0x188/0x340 [22442.622111] ? rescuer_thread+0x310/0x310 [22442.626113] kthread+0xf5/0x130 [22442.629252] ? kthread_complete_and_exit+0x20/0x20 [22442.634036] ret_from_fork+0x31/0x40 [22442.637614] ? kthread_complete_and_exit+0x20/0x20 [22442.642399] ret_from_fork_asm+0x11/0x20 [22442.646320] [22469.852040] watchdog: BUG: soft lockup - CPU#0 stuck for 9812s! [dbench:78683] [22469.859257] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22469.864039] watchdog: BUG: soft lockup - CPU#7 stuck for 18809s! [kworker/7:4:78808] [22469.914713] irq event stamp: 34741144 [22469.922446] Modules linked in: ceph libceph dns_resolver fscache netfs [22469.926104] hardirqs last enabled at (34741143): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22469.932629] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [22469.942787] hardirqs last disabled at (34741144): [] sysvec_apic_timer_interrupt+0xe/0x90 [22469.942790] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [22469.958596] softirqs last enabled at (34741106): [] __do_softirq+0x2be/0x371 [22469.958600] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [22469.975012] softirqs last disabled at (34741101): [] irq_exit_rcu+0xb0/0xd0 [22469.975015] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [22469.990049] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22469.996044] irqbypass i2c_i801 [22470.004989] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22470.004991] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22470.008125] iTCO_wdt iTCO_vendor_support [22470.015600] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22470.021075] i2c_smbus [22470.025080] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22470.043818] crct10dif_pclmul joydev [22470.046174] [22470.046175] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22470.051392] crc32_pclmul ipmi_ssif [22470.054961] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22470.056454] ghash_clmulni_intel mei_me [22470.063578] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22470.067061] mei lpc_ich [22470.074185] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22470.078016] sha512_ssse3 acpi_ipmi [22470.085141] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22470.087672] mfd_core wmi [22470.094795] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22470.098279] ipmi_si ipmi_devintf [22470.105403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22470.108020] ipmi_msghandler ioatdma [22470.116098] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22470.119408] acpi_pad acpi_power_meter [22470.125165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22470.128734] ixgbe igb [22470.135858] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22470.139601] mdio crc32c_intel [22470.146726] Call Trace: [22470.146729] [22470.149084] nvme ptp [22470.156209] ? watchdog_timer_fn+0x1b4/0x220 [22470.159259] nvme_core [22470.161718] ? __hrtimer_run_queues+0x1d3/0x360 [22470.163732] i2c_algo_bit pps_core [22470.166004] ? hrtimer_interrupt+0x104/0x240 [22470.170267] dca fuse [22470.172624] ? __local_bh_enable+0x50/0x60 [22470.177148] [22470.177149] irq event stamp: 102144656 [22470.180546] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22470.184808] hardirqs last enabled at (102144655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22470.187080] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22470.191171] hardirqs last disabled at (102144656): [] sysvec_apic_timer_interrupt+0xe/0x90 [22470.192662] [22470.196404] softirqs last enabled at (102144654): [] __do_softirq+0x2be/0x371 [22470.201622] [22470.211866] softirqs last disabled at (102144649): [] irq_exit_rcu+0xb0/0xd0 [22470.216825] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22470.226635] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22470.228744] ? queued_spin_lock_slowpath+0x16/0x2e0 [22470.237511] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22470.239612] do_raw_spin_lock+0xb5/0xc0 [22470.248208] Workqueue: events delayed_work [ceph] [22470.253514] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22470.262888] [22470.262889] RIP: 0010:lock_acquire+0xf0/0x2b0 [22470.267761] try_get_cap_refs+0x159/0x880 [ceph] [22470.275228] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22470.279065] __ceph_get_caps+0x4a1/0x760 [ceph] [22470.283758] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22470.288546] ? __file_remove_privs+0x89/0x110 [22470.290033] [22470.290033] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22470.294384] ? wake_bit_function+0x70/0x70 [22470.298994] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22470.317734] ? ceph_write_iter+0x260/0x770 [ceph] [22470.322255] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22470.327474] ceph_write_iter+0x260/0x770 [ceph] [22470.331824] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22470.333318] ? lock_is_held_type+0xaa/0x120 [22470.340439] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22470.344532] vfs_write+0x324/0x4f0 [22470.351654] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22470.356357] ksys_pwrite64+0x65/0xa0 [22470.363476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22470.368002] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22470.375124] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22470.379301] do_syscall_64+0x38/0x80 [22470.386440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22470.389841] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22470.397917] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22470.401490] RIP: 0033:0x7f48f0eeb93b [22470.407226] Call Trace: [22470.407228] [22470.412356] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22470.419482] ? watchdog_timer_fn+0x1b4/0x220 [22470.423052] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22470.430176] ? __hrtimer_run_queues+0x1d3/0x360 [22470.435220] ORIG_RAX: 0000000000000012 [22470.442348] ? hrtimer_interrupt+0x104/0x240 [22470.445915] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22470.448358] ? __local_bh_enable+0x50/0x60 [22470.450368] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22470.469110] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22470.473371] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22470.478589] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22470.483112] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22470.486943] [22470.491206] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22470.498331] [22470.498333] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22470.502444] [22470.509569] ? lock_acquire+0xf0/0x2b0 [22470.556540] ? igrab+0x43/0x50 [22470.559589] ? lock_release+0xc5/0x260 [22470.563335] _raw_spin_lock+0x30/0x40 [22470.566999] ? igrab+0x1d/0x50 [22470.570050] igrab+0x1d/0x50 [22470.572931] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22470.578077] delayed_work+0x1d8/0x280 [ceph] [22470.582360] ? process_scheduled_works+0x22a/0x510 [22470.587151] process_scheduled_works+0x2a4/0x510 [22470.591765] worker_thread+0x188/0x340 [22470.595514] ? rescuer_thread+0x310/0x310 [22470.599516] kthread+0xf5/0x130 [22470.602657] ? kthread_complete_and_exit+0x20/0x20 [22470.607449] ret_from_fork+0x31/0x40 [22470.611028] ? kthread_complete_and_exit+0x20/0x20 [22470.615820] ret_from_fork_asm+0x11/0x20 [22470.619744] [22497.852526] watchdog: BUG: soft lockup - CPU#0 stuck for 9838s! [dbench:78683] [22497.859737] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22497.864526] watchdog: BUG: soft lockup - CPU#7 stuck for 18835s! [kworker/7:4:78808] [22497.915198] irq event stamp: 34799272 [22497.915199] hardirqs last enabled at (34799271): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22497.922926] Modules linked in: [22497.926584] hardirqs last disabled at (34799272): [] sysvec_apic_timer_interrupt+0xe/0x90 [22497.936741] ceph [22497.939791] softirqs last enabled at (34799234): [] __do_softirq+0x2be/0x371 [22497.949515] libceph [22497.951440] softirqs last disabled at (34799229): [] irq_exit_rcu+0xb0/0xd0 [22497.960123] dns_resolver [22497.962308] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22497.970817] fscache netfs [22497.973435] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22497.982380] veth [22497.985085] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22497.992555] nft_chain_nat xt_MASQUERADE [22497.994479] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22497.999957] nf_nat nf_conntrack [22498.003874] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22498.022610] nf_defrag_ipv6 [22498.025836] [22498.025837] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22498.031054] nf_defrag_ipv4 nft_compat [22498.033844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22498.035334] nf_tables nfnetlink [22498.042459] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22498.046202] bridge [22498.053326] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22498.056551] stp [22498.063674] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22498.065773] llc xfs [22498.072897] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22498.074751] libcrc32c sunrpc [22498.081875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22498.084060] intel_rapl_msr intel_rapl_common [22498.092138] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22498.095100] sb_edac x86_pkg_temp_thermal [22498.100840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22498.105205] coretemp kvm_intel [22498.112331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22498.116334] kvm irqbypass [22498.123459] Call Trace: [22498.123460] [22498.126595] i2c_i801 iTCO_wdt [22498.133722] ? watchdog_timer_fn+0x1b4/0x220 [22498.136425] iTCO_vendor_support [22498.138868] ? __hrtimer_run_queues+0x1d3/0x360 [22498.140879] i2c_smbus crct10dif_pclmul [22498.143933] ? hrtimer_interrupt+0x104/0x240 [22498.148205] joydev crc32_pclmul [22498.151427] ? __local_bh_enable+0x50/0x60 [22498.155951] ipmi_ssif [22498.159784] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22498.164045] ghash_clmulni_intel [22498.167270] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22498.171359] mei_me [22498.173718] [22498.178935] mei [22498.182160] [22498.182162] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22498.187115] lpc_ich [22498.189220] ? queued_spin_lock_slowpath+0x16/0x2e0 [22498.191312] sha512_ssse3 acpi_ipmi [22498.193151] do_raw_spin_lock+0xb5/0xc0 [22498.195246] mfd_core wmi [22498.200552] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22498.202734] ipmi_si ipmi_devintf [22498.207608] try_get_cap_refs+0x159/0x880 [ceph] [22498.211090] ipmi_msghandler ioatdma [22498.214931] __ceph_get_caps+0x4a1/0x760 [ceph] [22498.217546] acpi_pad acpi_power_meter [22498.222335] ? __file_remove_privs+0x89/0x110 [22498.225641] ixgbe [22498.230252] ? wake_bit_function+0x70/0x70 [22498.233822] igb mdio [22498.238350] ? ceph_write_iter+0x260/0x770 [ceph] [22498.242089] crc32c_intel nvme [22498.246442] ceph_write_iter+0x260/0x770 [ceph] [22498.248451] ptp nvme_core [22498.252545] ? lock_is_held_type+0xaa/0x120 [22498.254812] i2c_algo_bit pps_core [22498.259513] vfs_write+0x324/0x4f0 [22498.262561] dca fuse [22498.267090] ksys_pwrite64+0x65/0xa0 [22498.269790] [22498.269791] irq event stamp: 102365740 [22498.273967] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22498.277365] hardirqs last enabled at (102365739): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22498.280764] do_syscall_64+0x38/0x80 [22498.283033] hardirqs last disabled at (102365740): [] sysvec_apic_timer_interrupt+0xe/0x90 [22498.286604] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22498.288093] softirqs last enabled at (102365738): [] __do_softirq+0x2be/0x371 [22498.291840] RIP: 0033:0x7f48f0eeb93b [22498.296968] softirqs last disabled at (102365733): [] irq_exit_rcu+0xb0/0xd0 [22498.307213] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22498.310783] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22498.320594] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22498.325638] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22498.334410] ORIG_RAX: 0000000000000012 [22498.337979] Workqueue: events delayed_work [ceph] [22498.346578] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22498.365315] [22498.374691] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22498.379927] RIP: 0010:lock_acquire+0xf0/0x2b0 [22498.387398] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22498.391228] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22498.395927] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22498.403051] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22498.404541] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22498.411665] [22498.411665] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22498.416022] [22498.423139] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22498.479267] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22498.486392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22498.493516] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22498.500638] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22498.508717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22498.514455] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22498.521577] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22498.528701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22498.535826] Call Trace: [22498.538272] [22498.540283] ? watchdog_timer_fn+0x1b4/0x220 [22498.544547] ? __hrtimer_run_queues+0x1d3/0x360 [22498.549075] ? hrtimer_interrupt+0x104/0x240 [22498.553341] ? __local_bh_enable+0x50/0x60 [22498.557435] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22498.562659] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22498.567616] [22498.569713] [22498.571813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22498.577120] ? lock_acquire+0xf0/0x2b0 [22498.580870] ? igrab+0x43/0x50 [22498.583933] ? lock_release+0xc5/0x260 [22498.587683] _raw_spin_lock+0x30/0x40 [22498.591347] ? igrab+0x1d/0x50 [22498.594398] igrab+0x1d/0x50 [22498.597276] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22498.602426] delayed_work+0x1d8/0x280 [ceph] [22498.606707] ? process_scheduled_works+0x22a/0x510 [22498.611496] process_scheduled_works+0x2a4/0x510 [22498.616113] worker_thread+0x188/0x340 [22498.619862] ? rescuer_thread+0x310/0x310 [22498.623865] kthread+0xf5/0x130 [22498.627003] ? kthread_complete_and_exit+0x20/0x20 [22498.631787] ret_from_fork+0x31/0x40 [22498.635364] ? kthread_complete_and_exit+0x20/0x20 [22498.640150] ret_from_fork_asm+0x11/0x20 [22498.644083] [22525.853012] watchdog: BUG: soft lockup - CPU#0 stuck for 9864s! [dbench:78683] [22525.860224] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22525.865012] watchdog: BUG: soft lockup - CPU#7 stuck for 18861s! [kworker/7:4:78808] [22525.915682] irq event stamp: 34857538 [22525.915683] hardirqs last enabled at (34857537): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22525.923413] Modules linked in: [22525.927070] hardirqs last disabled at (34857538): [] sysvec_apic_timer_interrupt+0xe/0x90 [22525.937227] ceph [22525.940279] softirqs last enabled at (34857532): [] __do_softirq+0x2be/0x371 [22525.950003] libceph [22525.951926] softirqs last disabled at (34857527): [] irq_exit_rcu+0xb0/0xd0 [22525.960611] dns_resolver [22525.962796] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22525.971306] fscache netfs [22525.973923] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22525.973924] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22525.982867] veth nft_chain_nat [22525.985572] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22525.993043] xt_MASQUERADE nf_nat [22525.998519] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22526.001658] nf_conntrack [22526.020394] [22526.020395] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22526.023706] nf_defrag_ipv6 [22526.028924] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22526.028925] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22526.031540] nf_defrag_ipv4 [22526.033031] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22526.033032] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22526.040154] nft_compat [22526.042945] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22526.050070] nf_tables [22526.057195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22526.059986] nfnetlink [22526.067109] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22526.074234] bridge [22526.076678] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22526.084772] stp [22526.087130] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22526.092867] llc xfs [22526.095225] Call Trace: [22526.102349] libcrc32c [22526.104446] [22526.111572] sunrpc [22526.113413] ? watchdog_timer_fn+0x1b4/0x220 [22526.120532] intel_rapl_msr [22526.122716] ? __hrtimer_run_queues+0x1d3/0x360 [22526.125160] intel_rapl_common sb_edac [22526.127522] ? hrtimer_interrupt+0x104/0x240 [22526.129528] x86_pkg_temp_thermal [22526.131626] ? __local_bh_enable+0x50/0x60 [22526.135889] coretemp kvm_intel [22526.138693] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22526.143213] kvm irqbypass [22526.146956] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22526.151221] i2c_i801 [22526.154532] [22526.158621] iTCO_wdt iTCO_vendor_support [22526.161760] [22526.166978] i2c_smbus [22526.169699] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22526.174656] crct10dif_pclmul [22526.176931] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22526.179025] joydev [22526.183029] do_raw_spin_lock+0xb5/0xc0 [22526.185126] crc32_pclmul [22526.187485] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22526.192789] ipmi_ssif [22526.195754] try_get_cap_refs+0x159/0x880 [ceph] [22526.200622] ghash_clmulni_intel [22526.202724] __ceph_get_caps+0x4a1/0x760 [ceph] [22526.206549] mei_me [22526.209170] ? __file_remove_privs+0x89/0x110 [22526.213951] mei [22526.216309] ? wake_bit_function+0x70/0x70 [22526.220920] lpc_ich [22526.224147] ? ceph_write_iter+0x260/0x770 [ceph] [22526.228666] sha512_ssse3 acpi_ipmi [22526.230782] ceph_write_iter+0x260/0x770 [ceph] [22526.235133] mfd_core [22526.236975] ? lock_is_held_type+0xaa/0x120 [22526.241062] wmi [22526.243247] vfs_write+0x324/0x4f0 [22526.247943] ipmi_si ipmi_devintf [22526.251432] ksys_pwrite64+0x65/0xa0 [22526.255951] ipmi_msghandler ioatdma [22526.258223] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22526.262410] acpi_pad [22526.264247] do_syscall_64+0x38/0x80 [22526.267642] acpi_power_meter [22526.270954] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22526.274525] ixgbe [22526.278095] RIP: 0033:0x7f48f0eeb93b [22526.283225] igb mdio [22526.285496] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22526.289067] crc32c_intel [22526.292030] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22526.297076] nvme [22526.299087] ORIG_RAX: 0000000000000012 [22526.302656] ptp [22526.304928] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22526.323666] nvme_core [22526.326283] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22526.331499] i2c_algo_bit [22526.333424] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22526.337255] pps_core [22526.339091] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22526.346216] dca fuse [22526.348573] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22526.355698] [22526.355700] irq event stamp: 102586814 [22526.358321] [22526.365438] hardirqs last enabled at (102586813): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22526.401891] hardirqs last disabled at (102586814): [] sysvec_apic_timer_interrupt+0xe/0x90 [22526.411704] softirqs last enabled at (102586812): [] __do_softirq+0x2be/0x371 [22526.420473] softirqs last disabled at (102586807): [] irq_exit_rcu+0xb0/0xd0 [22526.429072] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22526.438448] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22526.445920] Workqueue: events delayed_work [ceph] [22526.450634] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [22526.455246] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [22526.473982] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [22526.479199] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [22526.486324] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [22526.493447] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [22526.500571] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [22526.507704] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22526.514828] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22526.522907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22526.528644] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22526.535767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22526.542892] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22526.550015] Call Trace: [22526.552462] [22526.554472] ? watchdog_timer_fn+0x1b4/0x220 [22526.558736] ? __hrtimer_run_queues+0x1d3/0x360 [22526.563263] ? hrtimer_interrupt+0x104/0x240 [22526.567531] ? __local_bh_enable+0x50/0x60 [22526.571627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22526.576850] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22526.581807] [22526.583902] [22526.586001] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22526.591311] ? do_raw_spin_lock+0x42/0xc0 [22526.595320] igrab+0x1d/0x50 [22526.598197] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22526.603347] delayed_work+0x1d8/0x280 [ceph] [22526.607628] ? process_scheduled_works+0x22a/0x510 [22526.612420] process_scheduled_works+0x2a4/0x510 [22526.617041] worker_thread+0x188/0x340 [22526.620790] ? rescuer_thread+0x310/0x310 [22526.624795] kthread+0xf5/0x130 [22526.627932] ? kthread_complete_and_exit+0x20/0x20 [22526.632716] ret_from_fork+0x31/0x40 [22526.636295] ? kthread_complete_and_exit+0x20/0x20 [22526.641080] ret_from_fork_asm+0x11/0x20 [22526.645012] [22553.853497] watchdog: BUG: soft lockup - CPU#0 stuck for 9890s! [dbench:78683] [22553.860711] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22553.865497] watchdog: BUG: soft lockup - CPU#7 stuck for 18887s! [kworker/7:4:78808] [22553.916169] irq event stamp: 34915984 [22553.916170] hardirqs last enabled at (34915983): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22553.923901] Modules linked in: [22553.927557] hardirqs last disabled at (34915984): [] sysvec_apic_timer_interrupt+0xe/0x90 [22553.937716] ceph [22553.940767] softirqs last enabled at (34915958): [] __do_softirq+0x2be/0x371 [22553.950489] libceph [22553.952415] softirqs last disabled at (34915953): [] irq_exit_rcu+0xb0/0xd0 [22553.961098] dns_resolver fscache [22553.963281] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22553.971792] netfs veth [22553.975104] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22553.984047] nft_chain_nat [22553.986492] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22553.993962] xt_MASQUERADE [22553.996668] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22554.002144] nf_nat nf_conntrack [22554.004849] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22554.023587] nf_defrag_ipv6 [22554.026810] [22554.026811] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22554.032028] nf_defrag_ipv4 nft_compat [22554.034817] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22554.036308] nf_tables [22554.043433] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22554.047183] nfnetlink bridge [22554.054310] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22554.056668] stp llc [22554.063791] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22554.066755] xfs libcrc32c [22554.073893] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22554.076082] sunrpc intel_rapl_msr [22554.083205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22554.085911] intel_rapl_common [22554.093986] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22554.097384] sb_edac x86_pkg_temp_thermal [22554.103121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22554.106171] coretemp kvm_intel kvm [22554.113314] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22554.117317] irqbypass [22554.124443] Call Trace: [22554.127926] i2c_i801 iTCO_wdt [22554.135050] [22554.137408] iTCO_vendor_support i2c_smbus [22554.139852] ? watchdog_timer_fn+0x1b4/0x220 [22554.142903] crct10dif_pclmul joydev [22554.144916] ? __hrtimer_run_queues+0x1d3/0x360 [22554.149005] crc32_pclmul ipmi_ssif [22554.153271] ? hrtimer_interrupt+0x104/0x240 [22554.156838] ghash_clmulni_intel mei_me [22554.161362] ? __local_bh_enable+0x50/0x60 [22554.164847] mei lpc_ich [22554.169113] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22554.172941] sha512_ssse3 acpi_ipmi [22554.177033] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22554.179563] mfd_core wmi [22554.184781] [22554.188265] ipmi_si [22554.193222] [22554.193223] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22554.195841] ipmi_devintf ipmi_msghandler [22554.197942] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22554.200120] ioatdma acpi_pad [22554.202220] do_raw_spin_lock+0xb5/0xc0 [22554.207523] acpi_power_meter ixgbe igb [22554.211527] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22554.216397] mdio crc32c_intel [22554.219365] try_get_cap_refs+0x159/0x880 [ceph] [22554.223191] nvme ptp [22554.227027] __ceph_get_caps+0x4a1/0x760 [ceph] [22554.231806] nvme_core i2c_algo_bit [22554.234862] ? __file_remove_privs+0x89/0x110 [22554.239468] pps_core dca [22554.241740] ? wake_bit_function+0x70/0x70 [22554.246262] fuse [22554.249751] ? ceph_write_iter+0x260/0x770 [ceph] [22554.254099] irq event stamp: 102807664 [22554.256717] ceph_write_iter+0x260/0x770 [ceph] [22554.260805] hardirqs last enabled at (102807663): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22554.262735] ? lock_is_held_type+0xaa/0x120 [22554.267428] hardirqs last disabled at (102807664): [] sysvec_apic_timer_interrupt+0xe/0x90 [22554.271191] vfs_write+0x324/0x4f0 [22554.275712] softirqs last enabled at (102807662): [] __do_softirq+0x2be/0x371 [22554.285961] ksys_pwrite64+0x65/0xa0 [22554.290135] softirqs last disabled at (102807657): [] irq_exit_rcu+0xb0/0xd0 [22554.299945] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22554.303344] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22554.312114] do_syscall_64+0x38/0x80 [22554.315684] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22554.315685] Workqueue: events delayed_work [ceph] [22554.324283] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22554.329413] [22554.329413] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [22554.338791] RIP: 0033:0x7f48f0eeb93b [22554.342360] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [22554.349831] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22554.354530] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [22554.359574] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22554.361064] [22554.361064] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [22554.365674] ORIG_RAX: 0000000000000012 [22554.369246] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [22554.387982] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22554.406719] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [22554.411939] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22554.417155] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [22554.418646] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22554.425771] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22554.429600] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22554.436726] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22554.443850] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22554.450972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22554.458104] [22554.465222] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22554.516841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22554.523966] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22554.531090] Call Trace: [22554.533533] [22554.535545] ? watchdog_timer_fn+0x1b4/0x220 [22554.539811] ? __hrtimer_run_queues+0x1d3/0x360 [22554.544335] ? hrtimer_interrupt+0x104/0x240 [22554.548597] ? __local_bh_enable+0x50/0x60 [22554.552691] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22554.557914] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22554.562871] [22554.564969] [22554.567067] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22554.572374] ? do_raw_spin_lock+0x42/0xc0 [22554.576384] igrab+0x1d/0x50 [22554.579260] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22554.584412] delayed_work+0x1d8/0x280 [ceph] [22554.588691] ? process_scheduled_works+0x22a/0x510 [22554.593472] process_scheduled_works+0x2a4/0x510 [22554.598088] worker_thread+0x188/0x340 [22554.601838] ? rescuer_thread+0x310/0x310 [22554.605841] kthread+0xf5/0x130 [22554.608980] ? kthread_complete_and_exit+0x20/0x20 [22554.613765] ret_from_fork+0x31/0x40 [22554.617341] ? kthread_complete_and_exit+0x20/0x20 [22554.622127] ret_from_fork_asm+0x11/0x20 [22554.626051] [22555.984901] rcu: INFO: rcu_preempt self-detected stall on CPU [22555.990663] rcu: 7-....: (10311171 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/10309898 fqs=5180702 [22556.001262] rcu: (t=10619184 jiffies g=1309609 q=351942 ncpus=8) [22556.007347] Sending NMI from CPU 7 to CPUs 0: [22556.011698] NMI backtrace for cpu 0 [22556.011699] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22556.011701] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22556.011701] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22556.011705] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22556.011706] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22556.011707] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22556.011708] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22556.011709] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22556.011710] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22556.011711] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22556.011712] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22556.011713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22556.011714] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22556.011715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22556.011715] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22556.011716] Call Trace: [22556.011717] [22556.011718] ? nmi_cpu_backtrace+0xad/0x120 [22556.011722] ? nmi_cpu_backtrace_handler+0x11/0x20 [22556.011723] ? nmi_handle+0xd4/0x260 [22556.011727] ? default_do_nmi+0x49/0xf0 [22556.011729] ? exc_nmi+0x11e/0x150 [22556.011730] ? end_repeat_nmi+0x16/0x67 [22556.011735] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22556.011737] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22556.011740] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22556.011742] [22556.011742] [22556.011743] do_raw_spin_lock+0xb5/0xc0 [22556.011745] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22556.011760] try_get_cap_refs+0x159/0x880 [ceph] [22556.011774] __ceph_get_caps+0x4a1/0x760 [ceph] [22556.011787] ? __file_remove_privs+0x89/0x110 [22556.011789] ? wake_bit_function+0x70/0x70 [22556.011791] ? ceph_write_iter+0x260/0x770 [ceph] [22556.011802] ceph_write_iter+0x260/0x770 [ceph] [22556.011813] ? lock_is_held_type+0xaa/0x120 [22556.011816] vfs_write+0x324/0x4f0 [22556.011820] ksys_pwrite64+0x65/0xa0 [22556.011822] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22556.011825] do_syscall_64+0x38/0x80 [22556.011827] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22556.011829] RIP: 0033:0x7f48f0eeb93b [22556.011830] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22556.011831] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [22556.011833] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22556.011834] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22556.011834] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22556.011835] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22556.011836] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22556.011839] [22556.012696] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22556.316248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22556.323719] Workqueue: events delayed_work [ceph] [22556.328434] RIP: 0010:lock_acquire+0xf0/0x2b0 [22556.332783] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22556.351523] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22556.356741] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22556.363863] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22556.370989] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22556.378111] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22556.385236] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22556.392359] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22556.400437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22556.406175] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22556.413300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22556.420424] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22556.427547] Call Trace: [22556.429992] [22556.432002] ? rcu_dump_cpu_stacks+0xd1/0x180 [22556.436355] ? rcu_sched_clock_irq+0x633/0x1390 [22556.440879] ? lock_release+0xc5/0x260 [22556.444622] ? lock_is_held_type+0xaa/0x120 [22556.448799] ? lock_is_held_type+0xaa/0x120 [22556.452976] ? tick_sched_do_timer+0xa0/0xa0 [22556.457249] ? rcu_read_lock_sched_held+0x56/0x90 [22556.461947] ? tick_sched_do_timer+0xa0/0xa0 [22556.466210] ? update_process_times+0x5f/0x90 [22556.470562] ? tick_sched_handle.isra.16+0x1e/0x50 [22556.475345] ? tick_sched_timer+0x6f/0x80 [22556.479351] ? __hrtimer_run_queues+0x1d3/0x360 [22556.483876] ? hrtimer_interrupt+0x104/0x240 [22556.488138] ? __local_bh_enable+0x50/0x60 [22556.492231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22556.497454] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22556.502411] [22556.504509] [22556.506606] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22556.511917] ? lock_acquire+0xf0/0x2b0 [22556.515665] ? igrab+0x43/0x50 [22556.518714] ? lock_release+0xc5/0x260 [22556.522460] _raw_spin_lock+0x30/0x40 [22556.526124] ? igrab+0x1d/0x50 [22556.529176] igrab+0x1d/0x50 [22556.532052] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22556.537202] delayed_work+0x1d8/0x280 [ceph] [22556.541483] ? process_scheduled_works+0x22a/0x510 [22556.546274] process_scheduled_works+0x2a4/0x510 [22556.550890] worker_thread+0x188/0x340 [22556.554640] ? rescuer_thread+0x310/0x310 [22556.558651] kthread+0xf5/0x130 [22556.561789] ? kthread_complete_and_exit+0x20/0x20 [22556.566573] ret_from_fork+0x31/0x40 [22556.570143] ? kthread_complete_and_exit+0x20/0x20 [22556.574928] ret_from_fork_asm+0x11/0x20 [22556.578849] [22581.853983] watchdog: BUG: soft lockup - CPU#0 stuck for 9916s! [dbench:78683] [22581.861197] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22581.865983] watchdog: BUG: soft lockup - CPU#7 stuck for 18914s! [kworker/7:4:78808] [22581.916656] irq event stamp: 34974192 [22581.916657] hardirqs last enabled at (34974191): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22581.924387] Modules linked in: [22581.928045] hardirqs last disabled at (34974192): [] sysvec_apic_timer_interrupt+0xe/0x90 [22581.938203] ceph [22581.941253] softirqs last enabled at (34974160): [] __do_softirq+0x2be/0x371 [22581.950977] libceph dns_resolver [22581.952901] softirqs last disabled at (34974155): [] irq_exit_rcu+0xb0/0xd0 [22581.961585] fscache netfs [22581.964896] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22581.973407] veth nft_chain_nat [22581.976110] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22581.985055] xt_MASQUERADE nf_nat [22581.988194] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22581.995678] nf_conntrack nf_defrag_ipv6 [22581.998993] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22582.004468] nf_defrag_ipv4 nft_compat [22582.008387] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22582.027125] nf_tables nfnetlink [22582.030869] [22582.030870] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22582.036087] bridge stp [22582.039309] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22582.040800] llc xfs [22582.047923] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22582.050380] libcrc32c [22582.057501] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22582.059685] sunrpc intel_rapl_msr [22582.066809] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22582.069166] intel_rapl_common sb_edac [22582.076291] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22582.079688] x86_pkg_temp_thermal [22582.086812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22582.090558] coretemp kvm_intel [22582.098634] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22582.101945] kvm irqbypass [22582.107682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22582.110818] i2c_i801 iTCO_wdt [22582.117944] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22582.120648] iTCO_vendor_support i2c_smbus [22582.127772] Call Trace: [22582.130822] crct10dif_pclmul joydev [22582.137947] [22582.142037] crc32_pclmul ipmi_ssif [22582.144482] ? watchdog_timer_fn+0x1b4/0x220 [22582.148052] ghash_clmulni_intel mei_me [22582.150064] ? __hrtimer_run_queues+0x1d3/0x360 [22582.153547] mei lpc_ich [22582.157815] ? hrtimer_interrupt+0x104/0x240 [22582.161657] sha512_ssse3 acpi_ipmi [22582.166182] ? __local_bh_enable+0x50/0x60 [22582.168715] mfd_core wmi [22582.172980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22582.176461] ipmi_si [22582.180552] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22582.183169] ipmi_devintf ipmi_msghandler [22582.188387] [22582.190570] ioatdma acpi_pad [22582.195528] [22582.195529] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22582.199534] acpi_power_meter ixgbe [22582.201635] ? queued_spin_lock_slowpath+0x16/0x2e0 [22582.204595] igb mdio [22582.206693] do_raw_spin_lock+0xb5/0xc0 [22582.211996] crc32c_intel nvme [22582.215482] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22582.220351] ptp nvme_core [22582.222624] try_get_cap_refs+0x159/0x880 [ceph] [22582.226451] i2c_algo_bit pps_core [22582.229509] __ceph_get_caps+0x4a1/0x760 [ceph] [22582.234288] dca fuse [22582.236995] ? __file_remove_privs+0x89/0x110 [22582.241601] [22582.241602] irq event stamp: 103023846 [22582.245001] ? wake_bit_function+0x70/0x70 [22582.249524] hardirqs last enabled at (103023845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22582.251797] ? ceph_write_iter+0x260/0x770 [ceph] [22582.256143] hardirqs last disabled at (103023846): [] sysvec_apic_timer_interrupt+0xe/0x90 [22582.257637] ceph_write_iter+0x260/0x770 [ceph] [22582.261396] softirqs last enabled at (103023844): [] __do_softirq+0x2be/0x371 [22582.265490] ? lock_is_held_type+0xaa/0x120 [22582.275731] softirqs last disabled at (103023839): [] irq_exit_rcu+0xb0/0xd0 [22582.280432] vfs_write+0x324/0x4f0 [22582.290242] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22582.294768] ksys_pwrite64+0x65/0xa0 [22582.303536] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22582.307711] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22582.316309] Workqueue: events delayed_work [ceph] [22582.319709] do_syscall_64+0x38/0x80 [22582.329084] [22582.329085] RIP: 0010:lock_acquire+0xf0/0x2b0 [22582.332671] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22582.340143] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22582.345274] RIP: 0033:0x7f48f0eeb93b [22582.349972] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22582.353542] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22582.355034] [22582.355034] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22582.359384] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22582.364428] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22582.383165] ORIG_RAX: 0000000000000012 [22582.386736] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22582.391953] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22582.410691] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22582.412182] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22582.419305] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22582.424524] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22582.431663] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22582.435495] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22582.442620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22582.449745] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22582.456869] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22582.463997] [22582.471117] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22582.522737] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22582.529862] Call Trace: [22582.532304] [22582.534316] ? watchdog_timer_fn+0x1b4/0x220 [22582.538580] ? __hrtimer_run_queues+0x1d3/0x360 [22582.543107] ? hrtimer_interrupt+0x104/0x240 [22582.547385] ? __local_bh_enable+0x50/0x60 [22582.551477] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22582.556695] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22582.561666] [22582.563766] [22582.565862] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22582.571172] ? lock_acquire+0xf0/0x2b0 [22582.574921] ? igrab+0x43/0x50 [22582.577973] ? lock_release+0xc5/0x260 [22582.581717] _raw_spin_lock+0x30/0x40 [22582.585388] ? igrab+0x1d/0x50 [22582.588441] igrab+0x1d/0x50 [22582.591318] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22582.596468] delayed_work+0x1d8/0x280 [ceph] [22582.600748] ? process_scheduled_works+0x22a/0x510 [22582.605530] process_scheduled_works+0x2a4/0x510 [22582.610147] worker_thread+0x188/0x340 [22582.613895] ? rescuer_thread+0x310/0x310 [22582.617899] kthread+0xf5/0x130 [22582.621037] ? kthread_complete_and_exit+0x20/0x20 [22582.625821] ret_from_fork+0x31/0x40 [22582.629391] ? kthread_complete_and_exit+0x20/0x20 [22582.634175] ret_from_fork_asm+0x11/0x20 [22582.638099] [22609.854471] watchdog: BUG: soft lockup - CPU#0 stuck for 9943s! [dbench:78683] [22609.861685] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22609.866470] watchdog: BUG: soft lockup - CPU#7 stuck for 18940s! [kworker/7:4:78808] [22609.917126] irq event stamp: 35032418 [22609.924875] Modules linked in: ceph libceph dns_resolver fscache netfs [22609.928533] hardirqs last enabled at (35032417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22609.935057] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [22609.945215] hardirqs last disabled at (35032418): [] sysvec_apic_timer_interrupt+0xe/0x90 [22609.945218] nf_defrag_ipv6 [22609.945220] softirqs last enabled at (35032380): [] __do_softirq+0x2be/0x371 [22609.951299] nf_defrag_ipv4 [22609.961022] softirqs last disabled at (35032375): [] irq_exit_rcu+0xb0/0xd0 [22609.963815] nft_compat nf_tables [22609.972500] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22609.975288] nfnetlink bridge [22609.983799] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22609.987111] stp llc [22609.996055] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22609.999018] xfs libcrc32c [22610.006491] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22610.008675] sunrpc [22610.014152] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22610.016868] intel_rapl_msr [22610.035601] [22610.035602] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22610.037698] intel_rapl_common sb_edac [22610.042916] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22610.045708] x86_pkg_temp_thermal [22610.047198] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22610.054321] coretemp kvm_intel [22610.058065] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22610.065190] kvm irqbypass [22610.068502] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22610.075624] i2c_i801 [22610.078763] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22610.085887] iTCO_wdt iTCO_vendor_support [22610.088592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22610.095715] i2c_smbus crct10dif_pclmul [22610.097985] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22610.106063] joydev crc32_pclmul [22610.110067] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22610.115806] ipmi_ssif [22610.119634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22610.126759] ghash_clmulni_intel mei_me [22610.129984] Call Trace: [22610.129986] [22610.137107] mei lpc_ich [22610.139468] ? watchdog_timer_fn+0x1b4/0x220 [22610.146588] sha512_ssse3 [22610.150422] ? __hrtimer_run_queues+0x1d3/0x360 [22610.152868] acpi_ipmi [22610.154887] ? hrtimer_interrupt+0x104/0x240 [22610.157412] mfd_core wmi [22610.161679] ? __local_bh_enable+0x50/0x60 [22610.164295] ipmi_si ipmi_devintf [22610.168822] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22610.171177] ipmi_msghandler [22610.175440] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22610.178058] ioatdma acpi_pad [22610.182157] [22610.185467] acpi_power_meter [22610.190687] [22610.190688] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22610.193562] ixgbe igb [22610.198524] ? queued_spin_lock_slowpath+0x16/0x2e0 [22610.201484] mdio crc32c_intel [22610.203583] do_raw_spin_lock+0xb5/0xc0 [22610.206546] nvme [22610.208646] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22610.213948] ptp nvme_core [22610.216306] try_get_cap_refs+0x159/0x880 [ceph] [22610.221177] i2c_algo_bit pps_core [22610.224233] __ceph_get_caps+0x4a1/0x760 [ceph] [22610.228056] dca fuse [22610.229986] ? __file_remove_privs+0x89/0x110 [22610.234766] [22610.234767] irq event stamp: 103244686 [22610.237468] ? wake_bit_function+0x70/0x70 [22610.242079] hardirqs last enabled at (103244685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22610.245479] ? ceph_write_iter+0x260/0x770 [ceph] [22610.250001] hardirqs last disabled at (103244686): [] sysvec_apic_timer_interrupt+0xe/0x90 [22610.252271] ceph_write_iter+0x260/0x770 [ceph] [22610.256622] softirqs last enabled at (103244684): [] __do_softirq+0x2be/0x371 [22610.258118] ? lock_is_held_type+0xaa/0x120 [22610.261871] softirqs last disabled at (103244679): [] irq_exit_rcu+0xb0/0xd0 [22610.265969] vfs_write+0x324/0x4f0 [22610.276211] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22610.280912] ksys_pwrite64+0x65/0xa0 [22610.290719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22610.295242] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22610.304014] Workqueue: events delayed_work [ceph] [22610.308191] do_syscall_64+0x38/0x80 [22610.316787] [22610.316788] RIP: 0010:lock_release+0x78/0x260 [22610.320187] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22610.329563] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [22610.333150] RIP: 0033:0x7f48f0eeb93b [22610.340621] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [22610.345754] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22610.350449] [22610.350450] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [22610.354021] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22610.355512] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [22610.359871] ORIG_RAX: 0000000000000012 [22610.364915] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [22610.383653] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22610.387223] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [22610.392440] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22610.411178] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22610.412669] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22610.419793] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22610.425012] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22610.432135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22610.435964] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22610.443090] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22610.450218] [22610.457339] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22610.523205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22610.530329] Call Trace: [22610.532775] [22610.534786] ? watchdog_timer_fn+0x1b4/0x220 [22610.539051] ? __hrtimer_run_queues+0x1d3/0x360 [22610.543576] ? hrtimer_interrupt+0x104/0x240 [22610.547846] ? __local_bh_enable+0x50/0x60 [22610.551955] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22610.557180] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22610.562153] [22610.564252] [22610.566351] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22610.571655] ? igrab+0x43/0x50 [22610.574708] ? lock_release+0x78/0x260 [22610.578462] _raw_spin_unlock+0x1b/0x40 [22610.582299] igrab+0x43/0x50 [22610.585177] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22610.590324] delayed_work+0x1d8/0x280 [ceph] [22610.594606] ? process_scheduled_works+0x22a/0x510 [22610.599398] process_scheduled_works+0x2a4/0x510 [22610.604014] worker_thread+0x188/0x340 [22610.607761] ? rescuer_thread+0x310/0x310 [22610.611764] kthread+0xf5/0x130 [22610.614903] ? kthread_complete_and_exit+0x20/0x20 [22610.619686] ret_from_fork+0x31/0x40 [22610.623265] ? kthread_complete_and_exit+0x20/0x20 [22610.628050] ret_from_fork_asm+0x11/0x20 [22610.631981] [22637.854957] watchdog: BUG: soft lockup - CPU#0 stuck for 9969s! [dbench:78683] [22637.862171] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22637.866957] watchdog: BUG: soft lockup - CPU#7 stuck for 18966s! [kworker/7:4:78808] [22637.917614] irq event stamp: 35090664 [22637.917615] hardirqs last enabled at (35090663): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22637.925362] Modules linked in: [22637.929017] hardirqs last disabled at (35090664): [] sysvec_apic_timer_interrupt+0xe/0x90 [22637.939176] ceph [22637.942227] softirqs last enabled at (35090626): [] __do_softirq+0x2be/0x371 [22637.951950] libceph dns_resolver [22637.953876] softirqs last disabled at (35090621): [] irq_exit_rcu+0xb0/0xd0 [22637.962558] fscache netfs [22637.965870] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22637.974380] veth nft_chain_nat [22637.977083] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22637.986029] xt_MASQUERADE [22637.989168] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22637.996645] nf_nat nf_conntrack [22637.999353] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22638.004828] nf_defrag_ipv6 nf_defrag_ipv4 [22638.008052] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22638.026787] nft_compat nf_tables [22638.030878] [22638.030879] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22638.036098] nfnetlink bridge [22638.039407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22638.040898] stp llc [22638.048022] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22638.050987] xfs libcrc32c [22638.058110] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22638.060296] sunrpc intel_rapl_msr [22638.067419] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22638.070124] intel_rapl_common [22638.077248] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22638.080645] sb_edac [22638.087768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22638.090818] x86_pkg_temp_thermal coretemp [22638.098896] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22638.101081] kvm_intel kvm [22638.106819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22638.110908] irqbypass i2c_i801 [22638.118034] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22638.120737] iTCO_wdt iTCO_vendor_support [22638.127862] Call Trace: [22638.127864] [22638.130998] i2c_smbus crct10dif_pclmul [22638.138124] ? watchdog_timer_fn+0x1b4/0x220 [22638.142127] joydev [22638.144572] ? __hrtimer_run_queues+0x1d3/0x360 [22638.146582] crc32_pclmul ipmi_ssif [22638.150415] ? hrtimer_interrupt+0x104/0x240 [22638.154676] ghash_clmulni_intel [22638.156773] ? __local_bh_enable+0x50/0x60 [22638.161297] mei_me mei [22638.164786] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22638.169045] lpc_ich sha512_ssse3 [22638.172270] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22638.176361] acpi_ipmi mfd_core [22638.178805] [22638.184022] wmi ipmi_si [22638.187332] [22638.192308] ipmi_devintf [22638.195445] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22638.197541] ipmi_msghandler [22638.200077] ? queued_spin_lock_slowpath+0x18/0x2e0 [22638.202171] ioatdma acpi_pad [22638.204791] do_raw_spin_lock+0xb5/0xc0 [22638.210091] acpi_power_meter ixgbe [22638.212970] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22638.217840] igb mdio [22638.220806] try_get_cap_refs+0x159/0x880 [ceph] [22638.224636] crc32c_intel [22638.228123] __ceph_get_caps+0x4a1/0x760 [ceph] [22638.232904] nvme ptp [22638.235178] ? __file_remove_privs+0x89/0x110 [22638.239785] nvme_core i2c_algo_bit [22638.242402] ? wake_bit_function+0x70/0x70 [22638.246926] pps_core dca [22638.249200] ? ceph_write_iter+0x260/0x770 [ceph] [22638.253548] fuse [22638.257032] ceph_write_iter+0x260/0x770 [ceph] [22638.261122] irq event stamp: 103465744 [22638.263744] ? lock_is_held_type+0xaa/0x120 [22638.268436] hardirqs last enabled at (103465743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22638.270364] vfs_write+0x324/0x4f0 [22638.274885] hardirqs last disabled at (103465744): [] sysvec_apic_timer_interrupt+0xe/0x90 [22638.278642] ksys_pwrite64+0x65/0xa0 [22638.282816] softirqs last enabled at (103465742): [] __do_softirq+0x2be/0x371 [22638.293059] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22638.296458] softirqs last disabled at (103465737): [] irq_exit_rcu+0xb0/0xd0 [22638.306269] do_syscall_64+0x38/0x80 [22638.309838] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22638.318610] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22638.323740] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22638.332357] RIP: 0033:0x7f48f0eeb93b [22638.335926] Workqueue: events delayed_work [ceph] [22638.345304] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22638.350358] [22638.350359] RIP: 0010:lock_acquire+0xf0/0x2b0 [22638.357827] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22638.361398] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22638.366095] ORIG_RAX: 0000000000000012 [22638.384831] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22638.386322] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22638.390675] [22638.390676] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22638.395893] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22638.414639] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22638.418469] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22638.423685] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22638.430809] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22638.432300] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22638.439424] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22638.446549] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22638.453677] [22638.460796] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22638.506681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22638.512418] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22638.519542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22638.526667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22638.533790] Call Trace: [22638.536236] [22638.538245] ? watchdog_timer_fn+0x1b4/0x220 [22638.542510] ? __hrtimer_run_queues+0x1d3/0x360 [22638.547036] ? hrtimer_interrupt+0x104/0x240 [22638.551306] ? __local_bh_enable+0x50/0x60 [22638.555399] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22638.560622] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22638.565580] [22638.567676] [22638.569775] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22638.575082] ? lock_acquire+0xf0/0x2b0 [22638.578832] ? igrab+0x43/0x50 [22638.581883] ? lock_release+0xc5/0x260 [22638.585645] _raw_spin_lock+0x30/0x40 [22638.589302] ? igrab+0x1d/0x50 [22638.592363] igrab+0x1d/0x50 [22638.595238] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22638.600387] delayed_work+0x1d8/0x280 [ceph] [22638.604670] ? process_scheduled_works+0x22a/0x510 [22638.609459] process_scheduled_works+0x2a4/0x510 [22638.614075] worker_thread+0x188/0x340 [22638.617824] ? rescuer_thread+0x310/0x310 [22638.621829] kthread+0xf5/0x130 [22638.624964] ? kthread_complete_and_exit+0x20/0x20 [22638.629751] ret_from_fork+0x31/0x40 [22638.633329] ? kthread_complete_and_exit+0x20/0x20 [22638.638112] ret_from_fork_asm+0x11/0x20 [22638.642035] [22665.855444] watchdog: BUG: soft lockup - CPU#0 stuck for 9995s! [dbench:78683] [22665.862659] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22665.867443] watchdog: BUG: soft lockup - CPU#7 stuck for 18992s! [kworker/7:4:78808] [22665.918118] irq event stamp: 35148916 [22665.918119] hardirqs last enabled at (35148915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22665.925848] Modules linked in: [22665.929506] hardirqs last disabled at (35148916): [] sysvec_apic_timer_interrupt+0xe/0x90 [22665.939662] ceph [22665.942714] softirqs last enabled at (35148878): [] __do_softirq+0x2be/0x371 [22665.952437] libceph [22665.954362] softirqs last disabled at (35148873): [] irq_exit_rcu+0xb0/0xd0 [22665.963045] dns_resolver fscache [22665.965232] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22665.973741] netfs veth [22665.977052] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22665.985996] nft_chain_nat xt_MASQUERADE [22665.988439] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22665.995910] nf_nat nf_conntrack [22665.999840] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22666.005315] nf_defrag_ipv6 [22666.008538] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22666.027276] nf_defrag_ipv4 [22666.030068] [22666.030069] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22666.035284] nft_compat nf_tables [22666.038073] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22666.039565] nfnetlink bridge [22666.046688] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22666.049999] stp llc [22666.057139] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22666.060105] xfs libcrc32c [22666.067247] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22666.069431] sunrpc [22666.076556] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22666.079260] intel_rapl_msr intel_rapl_common [22666.086384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22666.088481] sb_edac [22666.096558] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22666.100909] x86_pkg_temp_thermal coretemp [22666.106646] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22666.108842] kvm_intel kvm [22666.115963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22666.120054] irqbypass [22666.127179] Call Trace: [22666.127180] [22666.129883] i2c_i801 iTCO_wdt [22666.137009] ? watchdog_timer_fn+0x1b4/0x220 [22666.139364] iTCO_vendor_support i2c_smbus [22666.141809] ? __hrtimer_run_queues+0x1d3/0x360 [22666.143817] crct10dif_pclmul joydev [22666.146891] ? hrtimer_interrupt+0x104/0x240 [22666.151149] crc32_pclmul ipmi_ssif [22666.155242] ? __local_bh_enable+0x50/0x60 [22666.159764] ghash_clmulni_intel [22666.163339] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22666.167600] mei_me [22666.171084] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22666.175176] mei lpc_ich [22666.178400] [22666.183617] sha512_ssse3 acpi_ipmi [22666.185714] [22666.190671] mfd_core wmi [22666.193203] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22666.195299] ipmi_si ipmi_devintf [22666.198787] ? queued_spin_lock_slowpath+0x18/0x2e0 [22666.200881] ipmi_msghandler [22666.203501] do_raw_spin_lock+0xb5/0xc0 [22666.208801] ioatdma acpi_pad acpi_power_meter [22666.212116] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22666.217001] ixgbe igb [22666.219881] try_get_cap_refs+0x159/0x880 [ceph] [22666.223709] mdio crc32c_intel [22666.228152] __ceph_get_caps+0x4a1/0x760 [ceph] [22666.232931] nvme ptp [22666.235294] ? __file_remove_privs+0x89/0x110 [22666.239898] nvme_core i2c_algo_bit [22666.242949] ? wake_bit_function+0x70/0x70 [22666.247475] pps_core dca [22666.249748] ? ceph_write_iter+0x260/0x770 [ceph] [22666.254097] fuse [22666.257579] ceph_write_iter+0x260/0x770 [ceph] [22666.261670] [22666.261671] irq event stamp: 103686480 [22666.264291] ? lock_is_held_type+0xaa/0x120 [22666.268984] hardirqs last enabled at (103686479): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22666.270910] vfs_write+0x324/0x4f0 [22666.275433] hardirqs last disabled at (103686480): [] sysvec_apic_timer_interrupt+0xe/0x90 [22666.276928] ksys_pwrite64+0x65/0xa0 [22666.280668] softirqs last enabled at (103686478): [] __do_softirq+0x2be/0x371 [22666.284847] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22666.295089] softirqs last disabled at (103686473): [] irq_exit_rcu+0xb0/0xd0 [22666.298488] do_syscall_64+0x38/0x80 [22666.308297] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22666.311870] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22666.320639] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22666.325771] RIP: 0033:0x7f48f0eeb93b [22666.334367] Workqueue: events delayed_work [ceph] [22666.337939] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22666.347315] [22666.352359] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22666.359844] RIP: 0010:lock_acquire+0xf0/0x2b0 [22666.363418] ORIG_RAX: 0000000000000012 [22666.368125] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22666.386863] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22666.388352] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22666.393571] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22666.397921] [22666.397922] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22666.401752] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22666.420489] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22666.427614] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22666.432845] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22666.439973] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22666.441462] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22666.448593] [22666.455712] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22666.500650] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22666.508726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22666.514463] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22666.521588] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22666.528712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22666.535849] Call Trace: [22666.538290] [22666.540301] ? watchdog_timer_fn+0x1b4/0x220 [22666.544566] ? __hrtimer_run_queues+0x1d3/0x360 [22666.549093] ? hrtimer_interrupt+0x104/0x240 [22666.553360] ? __local_bh_enable+0x50/0x60 [22666.557454] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22666.562679] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22666.567636] [22666.569734] [22666.571847] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22666.577156] ? lock_acquire+0xf0/0x2b0 [22666.580905] ? igrab+0x43/0x50 [22666.583955] ? lock_release+0xc5/0x260 [22666.587702] _raw_spin_lock+0x30/0x40 [22666.591366] ? igrab+0x1d/0x50 [22666.594417] igrab+0x1d/0x50 [22666.597296] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22666.602445] delayed_work+0x1d8/0x280 [ceph] [22666.606725] ? process_scheduled_works+0x22a/0x510 [22666.611515] process_scheduled_works+0x2a4/0x510 [22666.616145] worker_thread+0x188/0x340 [22666.619888] ? rescuer_thread+0x310/0x310 [22666.623892] kthread+0xf5/0x130 [22666.627031] ? kthread_complete_and_exit+0x20/0x20 [22666.631823] ret_from_fork+0x31/0x40 [22666.635419] ? kthread_complete_and_exit+0x20/0x20 [22666.640203] ret_from_fork_asm+0x11/0x20 [22666.644138] [22693.855930] watchdog: BUG: soft lockup - CPU#0 stuck for 10021s! [dbench:78683] [22693.863233] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22693.867930] watchdog: BUG: soft lockup - CPU#7 stuck for 19018s! [kworker/7:4:78808] [22693.918672] irq event stamp: 35207152 [22693.918674] hardirqs last enabled at (35207151): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22693.926404] Modules linked in: [22693.930061] hardirqs last disabled at (35207152): [] sysvec_apic_timer_interrupt+0xe/0x90 [22693.940219] ceph [22693.943270] softirqs last enabled at (35207114): [] __do_softirq+0x2be/0x371 [22693.952995] libceph [22693.954917] softirqs last disabled at (35207109): [] irq_exit_rcu+0xb0/0xd0 [22693.963606] dns_resolver [22693.965788] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22693.974298] fscache netfs [22693.976932] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22693.985876] veth nft_chain_nat [22693.988582] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22693.996067] xt_MASQUERADE nf_nat [22693.999205] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22694.004684] nf_conntrack nf_defrag_ipv6 [22694.007993] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22694.026730] nf_defrag_ipv4 nft_compat [22694.030649] [22694.030650] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22694.035866] nf_tables nfnetlink [22694.039611] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22694.041101] bridge stp [22694.048224] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22694.051450] llc [22694.058574] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22694.061017] xfs libcrc32c [22694.068142] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22694.069980] sunrpc intel_rapl_msr [22694.077102] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22694.079806] intel_rapl_common sb_edac [22694.086932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22694.090327] x86_pkg_temp_thermal [22694.098405] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22694.102150] coretemp kvm_intel [22694.107888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22694.111198] kvm irqbypass [22694.118327] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22694.121468] i2c_i801 iTCO_wdt [22694.128609] Call Trace: [22694.128610] [22694.131328] iTCO_vendor_support i2c_smbus [22694.138457] ? watchdog_timer_fn+0x1b4/0x220 [22694.141505] crct10dif_pclmul joydev [22694.143950] ? __hrtimer_run_queues+0x1d3/0x360 [22694.145962] crc32_pclmul [22694.150054] ? hrtimer_interrupt+0x104/0x240 [22694.154330] ipmi_ssif [22694.157903] ? __local_bh_enable+0x50/0x60 [22694.162428] ghash_clmulni_intel mei_me [22694.165048] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22694.169326] mei [22694.171684] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22694.175776] lpc_ich sha512_ssse3 [22694.179621] [22694.184840] acpi_ipmi [22694.186679] [22694.186680] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22694.191635] mfd_core wmi [22694.194949] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22694.197044] ipmi_si [22694.199403] do_raw_spin_lock+0xb5/0xc0 [22694.201497] ipmi_devintf ipmi_msghandler [22694.206804] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22694.209418] ioatdma acpi_pad acpi_power_meter [22694.214293] try_get_cap_refs+0x159/0x880 [ceph] [22694.216475] ixgbe igb [22694.220329] __ceph_get_caps+0x4a1/0x760 [ceph] [22694.224328] mdio crc32c_intel [22694.229115] ? __file_remove_privs+0x89/0x110 [22694.233549] nvme ptp [22694.238159] ? wake_bit_function+0x70/0x70 [22694.240515] nvme_core i2c_algo_bit [22694.245043] ? ceph_write_iter+0x260/0x770 [ceph] [22694.248091] pps_core dca [22694.252443] ceph_write_iter+0x260/0x770 [ceph] [22694.254712] fuse [22694.258807] ? lock_is_held_type+0xaa/0x120 [22694.262288] irq event stamp: 103907340 [22694.266988] vfs_write+0x324/0x4f0 [22694.269611] hardirqs last enabled at (103907339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22694.274140] ksys_pwrite64+0x65/0xa0 [22694.276058] hardirqs last disabled at (103907340): [] sysvec_apic_timer_interrupt+0xe/0x90 [22694.280237] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22694.283980] softirqs last enabled at (103907338): [] __do_softirq+0x2be/0x371 [22694.287380] do_syscall_64+0x38/0x80 [22694.297627] softirqs last disabled at (103907333): [] irq_exit_rcu+0xb0/0xd0 [22694.301201] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22694.311013] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22694.316143] RIP: 0033:0x7f48f0eeb93b [22694.324913] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22694.328486] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22694.337081] Workqueue: events delayed_work [ceph] [22694.342125] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22694.351502] [22694.351503] RIP: 0010:__list_del_entry_valid_or_report+0x0/0xd0 [22694.355074] ORIG_RAX: 0000000000000012 [22694.362546] Code: 1d 82 e8 83 0c b0 ff 0f 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 [22694.381282] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22694.385980] RSP: 0018:ffffc90002117db0 EFLAGS: 00000216 [22694.391197] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22694.392688] [22694.392689] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 000000010155ade0 [22694.398599] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22694.402447] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [22694.421184] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22694.428308] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [22694.433544] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22694.440667] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [22694.442164] [22694.449282] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [22694.501335] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22694.509414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22694.515151] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22694.522274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22694.529399] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22694.536522] Call Trace: [22694.538968] [22694.540979] ? watchdog_timer_fn+0x1b4/0x220 [22694.545244] ? __hrtimer_run_queues+0x1d3/0x360 [22694.549770] ? hrtimer_interrupt+0x104/0x240 [22694.554039] ? __local_bh_enable+0x50/0x60 [22694.558133] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22694.563355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22694.568313] [22694.570427] [22694.572527] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22694.577843] ? __list_add_valid_or_report+0xa0/0xa0 [22694.582720] ceph_check_delayed_caps+0x8f/0x270 [ceph] [22694.587867] delayed_work+0x1d8/0x280 [ceph] [22694.592149] ? process_scheduled_works+0x22a/0x510 [22694.596940] process_scheduled_works+0x2a4/0x510 [22694.601556] worker_thread+0x188/0x340 [22694.605305] ? rescuer_thread+0x310/0x310 [22694.609334] kthread+0xf5/0x130 [22694.612473] ? kthread_complete_and_exit+0x20/0x20 [22694.617264] ret_from_fork+0x31/0x40 [22694.620834] ? kthread_complete_and_exit+0x20/0x20 [22694.625619] ret_from_fork_asm+0x11/0x20 [22694.629542] [22721.856419] watchdog: BUG: soft lockup - CPU#0 stuck for 10047s! [dbench:78683] [22721.863720] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22721.868418] watchdog: BUG: soft lockup - CPU#7 stuck for 19044s! [kworker/7:4:78808] [22721.919170] irq event stamp: 35265376 [22721.919171] hardirqs last enabled at (35265375): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22721.926899] Modules linked in: [22721.930557] hardirqs last disabled at (35265376): [] sysvec_apic_timer_interrupt+0xe/0x90 [22721.940716] ceph [22721.943766] softirqs last enabled at (35265338): [] __do_softirq+0x2be/0x371 [22721.953491] libceph [22721.955415] softirqs last disabled at (35265333): [] irq_exit_rcu+0xb0/0xd0 [22721.964097] dns_resolver fscache [22721.966284] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22721.974810] netfs veth [22721.978122] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22721.987066] nft_chain_nat [22721.989510] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22721.996981] xt_MASQUERADE nf_nat [22721.999684] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22722.005161] nf_conntrack nf_defrag_ipv6 [22722.008473] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22722.027209] nf_defrag_ipv4 nft_compat [22722.031126] [22722.031127] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22722.036345] nf_tables nfnetlink [22722.040095] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22722.041589] bridge stp [22722.048711] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22722.051937] llc [22722.059062] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22722.061505] xfs libcrc32c [22722.068627] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22722.070466] sunrpc intel_rapl_msr [22722.077591] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22722.080294] intel_rapl_common sb_edac [22722.087418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22722.090817] x86_pkg_temp_thermal [22722.098892] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22722.102638] coretemp kvm_intel [22722.108375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22722.111684] kvm irqbypass [22722.118815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22722.121956] i2c_i801 iTCO_wdt [22722.129096] Call Trace: [22722.131817] iTCO_vendor_support [22722.138945] [22722.141993] i2c_smbus crct10dif_pclmul [22722.144441] ? watchdog_timer_fn+0x1b4/0x220 [22722.147663] joydev crc32_pclmul [22722.149672] ? __hrtimer_run_queues+0x1d3/0x360 [22722.153503] ipmi_ssif [22722.157771] ? hrtimer_interrupt+0x104/0x240 [22722.160992] ghash_clmulni_intel mei_me [22722.165514] ? __local_bh_enable+0x50/0x60 [22722.167873] mei lpc_ich [22722.172140] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22722.175968] sha512_ssse3 [22722.180059] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22722.182589] acpi_ipmi mfd_core [22722.187818] [22722.190431] wmi ipmi_si [22722.195389] [22722.195391] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22722.198526] ipmi_devintf [22722.200630] ? queued_spin_lock_slowpath+0x16/0x2e0 [22722.203155] ipmi_msghandler ioatdma [22722.205255] do_raw_spin_lock+0xb5/0xc0 [22722.210556] acpi_pad [22722.213174] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22722.218045] acpi_power_meter [22722.221618] try_get_cap_refs+0x159/0x880 [ceph] [22722.225447] ixgbe [22722.227723] __ceph_get_caps+0x4a1/0x760 [ceph] [22722.232501] igb mdio [22722.235470] ? __file_remove_privs+0x89/0x110 [22722.240075] crc32c_intel nvme [22722.242108] ? wake_bit_function+0x70/0x70 [22722.246628] ptp [22722.248902] ? ceph_write_iter+0x260/0x770 [ceph] [22722.253248] nvme_core [22722.256300] ceph_write_iter+0x260/0x770 [ceph] [22722.260391] i2c_algo_bit pps_core [22722.262231] ? lock_is_held_type+0xaa/0x120 [22722.266926] dca fuse [22722.269285] vfs_write+0x324/0x4f0 [22722.273818] [22722.273819] irq event stamp: 104128364 [22722.277219] ksys_pwrite64+0x65/0xa0 [22722.281391] hardirqs last enabled at (104128363): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22722.283663] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22722.287058] hardirqs last disabled at (104128364): [] sysvec_apic_timer_interrupt+0xe/0x90 [22722.288550] do_syscall_64+0x38/0x80 [22722.292293] softirqs last enabled at (104128362): [] __do_softirq+0x2be/0x371 [22722.295865] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22722.306107] softirqs last disabled at (104128357): [] irq_exit_rcu+0xb0/0xd0 [22722.311239] RIP: 0033:0x7f48f0eeb93b [22722.321050] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22722.324623] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22722.333391] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22722.338435] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22722.347032] Workqueue: events delayed_work [ceph] [22722.350604] ORIG_RAX: 0000000000000012 [22722.359980] [22722.359981] RIP: 0010:lock_acquire+0xf0/0x2b0 [22722.378719] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22722.386191] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22722.391408] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22722.396105] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22722.399935] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22722.401427] [22722.401428] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22722.405777] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22722.412900] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22722.431639] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22722.438763] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22722.443986] [22722.451105] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22722.497541] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22722.504667] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22722.512742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22722.518482] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22722.525605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22722.532730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22722.539852] Call Trace: [22722.542298] [22722.544307] ? watchdog_timer_fn+0x1b4/0x220 [22722.548572] ? __hrtimer_run_queues+0x1d3/0x360 [22722.553100] ? hrtimer_interrupt+0x104/0x240 [22722.557378] ? __local_bh_enable+0x50/0x60 [22722.561470] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22722.566695] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22722.571651] [22722.573750] [22722.575848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22722.581154] ? lock_acquire+0xf0/0x2b0 [22722.584906] ? igrab+0x43/0x50 [22722.587954] ? lock_release+0xc5/0x260 [22722.591700] _raw_spin_lock+0x30/0x40 [22722.595366] ? igrab+0x1d/0x50 [22722.598415] igrab+0x1d/0x50 [22722.601295] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22722.606444] delayed_work+0x1d8/0x280 [ceph] [22722.610724] ? process_scheduled_works+0x22a/0x510 [22722.615513] process_scheduled_works+0x2a4/0x510 [22722.620129] worker_thread+0x188/0x340 [22722.623880] ? rescuer_thread+0x310/0x310 [22722.627892] kthread+0xf5/0x130 [22722.631038] ? kthread_complete_and_exit+0x20/0x20 [22722.635825] ret_from_fork+0x31/0x40 [22722.639401] ? kthread_complete_and_exit+0x20/0x20 [22722.644184] ret_from_fork_asm+0x11/0x20 [22722.648124] [22749.856907] watchdog: BUG: soft lockup - CPU#0 stuck for 10073s! [dbench:78683] [22749.864207] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22749.868907] watchdog: BUG: soft lockup - CPU#7 stuck for 19070s! [kworker/7:4:78808] [22749.919647] irq event stamp: 35323666 [22749.927379] Modules linked in: ceph libceph [22749.931036] hardirqs last enabled at (35323665): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22749.935212] dns_resolver fscache netfs [22749.945369] hardirqs last disabled at (35323666): [] sysvec_apic_timer_interrupt+0xe/0x90 [22749.949202] veth nft_chain_nat xt_MASQUERADE nf_nat [22749.958924] softirqs last enabled at (35323628): [] __do_softirq+0x2be/0x371 [22749.963884] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [22749.972583] softirqs last disabled at (35323623): [] irq_exit_rcu+0xb0/0xd0 [22749.972584] nft_compat [22749.977801] nf_tables [22749.986313] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22749.988756] nfnetlink [22749.991113] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22750.000058] bridge stp [22750.002416] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22750.009887] llc xfs [22750.012331] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22750.017807] libcrc32c sunrpc [22750.019992] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22750.038730] intel_rapl_msr intel_rapl_common [22750.041694] [22750.041695] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22750.046912] sb_edac x86_pkg_temp_thermal [22750.051260] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22750.052751] coretemp kvm_intel [22750.059875] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22750.063881] kvm irqbypass [22750.071006] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22750.074141] i2c_i801 iTCO_wdt [22750.081267] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22750.083971] iTCO_vendor_support i2c_smbus [22750.091094] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22750.094145] crct10dif_pclmul [22750.101269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22750.105361] joydev [22750.113437] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22750.116401] crc32_pclmul ipmi_ssif [22750.122139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22750.124236] ghash_clmulni_intel mei_me [22750.131361] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22750.134845] mei [22750.141968] Call Trace: [22750.141970] [22750.145799] lpc_ich sha512_ssse3 [22750.152924] ? watchdog_timer_fn+0x1b4/0x220 [22750.154762] acpi_ipmi [22750.157207] ? __hrtimer_run_queues+0x1d3/0x360 [22750.159215] mfd_core wmi [22750.162529] ? hrtimer_interrupt+0x104/0x240 [22750.166790] ipmi_si [22750.169148] ? __local_bh_enable+0x50/0x60 [22750.173671] ipmi_devintf ipmi_msghandler [22750.176306] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22750.180586] ioatdma acpi_pad [22750.182772] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22750.186862] acpi_power_meter ixgbe [22750.190868] [22750.196083] igb [22750.199048] [22750.199050] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22750.204007] mdio crc32c_intel [22750.207494] ? queued_spin_lock_slowpath+0x18/0x2e0 [22750.209587] nvme ptp nvme_core [22750.211428] do_raw_spin_lock+0xb5/0xc0 [22750.213521] i2c_algo_bit [22750.218826] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22750.221876] pps_core dca [22750.226752] try_get_cap_refs+0x159/0x880 [ceph] [22750.229885] fuse [22750.233721] __ceph_get_caps+0x4a1/0x760 [ceph] [22750.236332] irq event stamp: 104349276 [22750.241121] ? __file_remove_privs+0x89/0x110 [22750.243735] hardirqs last enabled at (104349275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22750.248347] ? wake_bit_function+0x70/0x70 [22750.250270] hardirqs last disabled at (104349276): [] sysvec_apic_timer_interrupt+0xe/0x90 [22750.254796] ? ceph_write_iter+0x260/0x770 [ceph] [22750.258537] softirqs last enabled at (104349274): [] __do_softirq+0x2be/0x371 [22750.262889] ceph_write_iter+0x260/0x770 [ceph] [22750.273133] softirqs last disabled at (104349269): [] irq_exit_rcu+0xb0/0xd0 [22750.277227] ? lock_is_held_type+0xaa/0x120 [22750.287035] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22750.291734] vfs_write+0x324/0x4f0 [22750.300503] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22750.305031] ksys_pwrite64+0x65/0xa0 [22750.313624] Workqueue: events delayed_work [ceph] [22750.317802] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22750.327179] [22750.330591] do_syscall_64+0x38/0x80 [22750.338066] RIP: 0010:lock_acquire+0xf0/0x2b0 [22750.341637] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22750.346332] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22750.351465] RIP: 0033:0x7f48f0eeb93b [22750.352954] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22750.356524] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22750.360874] [22750.360875] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22750.365920] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22750.384656] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22750.388229] ORIG_RAX: 0000000000000012 [22750.393446] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22750.412182] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22750.413674] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22750.420798] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22750.426016] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22750.433138] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22750.436971] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22750.444093] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22750.451217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22750.458342] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22750.465466] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22750.472600] [22750.479716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22750.524212] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22750.531334] Call Trace: [22750.533778] [22750.535790] ? watchdog_timer_fn+0x1b4/0x220 [22750.540055] ? __hrtimer_run_queues+0x1d3/0x360 [22750.544597] ? hrtimer_interrupt+0x104/0x240 [22750.548860] ? __local_bh_enable+0x50/0x60 [22750.552952] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22750.558167] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22750.563125] [22750.565222] [22750.567320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22750.572629] ? lock_acquire+0xf0/0x2b0 [22750.576379] ? igrab+0x43/0x50 [22750.579427] ? lock_release+0xc5/0x260 [22750.583172] _raw_spin_lock+0x30/0x40 [22750.586829] ? igrab+0x1d/0x50 [22750.589881] igrab+0x1d/0x50 [22750.592757] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22750.597907] delayed_work+0x1d8/0x280 [ceph] [22750.602187] ? process_scheduled_works+0x22a/0x510 [22750.606970] process_scheduled_works+0x2a4/0x510 [22750.611601] worker_thread+0x188/0x340 [22750.615352] ? rescuer_thread+0x310/0x310 [22750.619356] kthread+0xf5/0x130 [22750.622493] ? kthread_complete_and_exit+0x20/0x20 [22750.627277] ret_from_fork+0x31/0x40 [22750.630856] ? kthread_complete_and_exit+0x20/0x20 [22750.635640] ret_from_fork_asm+0x11/0x20 [22750.639564] [22751.586304] rcu: INFO: rcu_preempt self-detected stall on CPU [22751.592061] rcu: 7-....: (10500764 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/10499490 fqs=5276579 [22751.602661] rcu: (t=10814782 jiffies g=1309609 q=355911 ncpus=8) [22751.608753] Sending NMI from CPU 7 to CPUs 0: [22751.613106] NMI backtrace for cpu 0 [22751.613107] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22751.613109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22751.613110] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22751.613113] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22751.613114] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22751.613115] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22751.613116] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22751.613117] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22751.613118] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22751.613118] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22751.613119] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22751.613120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22751.613121] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22751.613122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22751.613123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22751.613123] Call Trace: [22751.613124] [22751.613125] ? nmi_cpu_backtrace+0xad/0x120 [22751.613129] ? nmi_cpu_backtrace_handler+0x11/0x20 [22751.613130] ? nmi_handle+0xd4/0x260 [22751.613134] ? default_do_nmi+0x49/0xf0 [22751.613136] ? exc_nmi+0x11e/0x150 [22751.613137] ? end_repeat_nmi+0x16/0x67 [22751.613141] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22751.613143] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22751.613146] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22751.613148] [22751.613148] [22751.613149] do_raw_spin_lock+0xb5/0xc0 [22751.613151] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22751.613165] try_get_cap_refs+0x159/0x880 [ceph] [22751.613179] __ceph_get_caps+0x4a1/0x760 [ceph] [22751.613191] ? __file_remove_privs+0x89/0x110 [22751.613193] ? wake_bit_function+0x70/0x70 [22751.613195] ? ceph_write_iter+0x260/0x770 [ceph] [22751.613205] ceph_write_iter+0x260/0x770 [ceph] [22751.613216] ? lock_is_held_type+0xaa/0x120 [22751.613218] vfs_write+0x324/0x4f0 [22751.613222] ksys_pwrite64+0x65/0xa0 [22751.613224] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22751.613227] do_syscall_64+0x38/0x80 [22751.613229] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22751.613231] RIP: 0033:0x7f48f0eeb93b [22751.613232] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22751.613233] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [22751.613234] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22751.613235] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22751.613236] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22751.613236] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22751.613237] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22751.613240] [22751.614104] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22751.917716] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22751.925186] Workqueue: events delayed_work [ceph] [22751.929910] RIP: 0010:lock_acquire+0xf0/0x2b0 [22751.934260] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22751.952998] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22751.958216] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22751.965340] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22751.972464] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22751.979587] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22751.986711] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22751.993837] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22752.001914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22752.007650] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22752.014776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22752.021899] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22752.029023] Call Trace: [22752.031468] [22752.033479] ? rcu_dump_cpu_stacks+0xd1/0x180 [22752.037840] ? rcu_sched_clock_irq+0x633/0x1390 [22752.042371] ? lock_release+0xc5/0x260 [22752.046115] ? lock_is_held_type+0xaa/0x120 [22752.050294] ? lock_is_held_type+0xaa/0x120 [22752.054481] ? tick_sched_do_timer+0xa0/0xa0 [22752.058751] ? rcu_read_lock_sched_held+0x56/0x90 [22752.063460] ? tick_sched_do_timer+0xa0/0xa0 [22752.067730] ? update_process_times+0x5f/0x90 [22752.072082] ? tick_sched_handle.isra.16+0x1e/0x50 [22752.076875] ? tick_sched_timer+0x6f/0x80 [22752.080888] ? __hrtimer_run_queues+0x1d3/0x360 [22752.085428] ? hrtimer_interrupt+0x104/0x240 [22752.089691] ? __local_bh_enable+0x50/0x60 [22752.093788] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22752.099008] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22752.103966] [22752.106064] [22752.108160] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22752.113474] ? lock_acquire+0xf0/0x2b0 [22752.117220] ? igrab+0x43/0x50 [22752.120279] ? lock_release+0xc5/0x260 [22752.124035] _raw_spin_lock+0x30/0x40 [22752.127695] ? igrab+0x1d/0x50 [22752.130746] igrab+0x1d/0x50 [22752.133624] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22752.138778] delayed_work+0x1d8/0x280 [ceph] [22752.143067] ? process_scheduled_works+0x22a/0x510 [22752.147853] process_scheduled_works+0x2a4/0x510 [22752.152476] worker_thread+0x188/0x340 [22752.156228] ? rescuer_thread+0x310/0x310 [22752.160240] kthread+0xf5/0x130 [22752.163387] ? kthread_complete_and_exit+0x20/0x20 [22752.168180] ret_from_fork+0x31/0x40 [22752.171758] ? kthread_complete_and_exit+0x20/0x20 [22752.176542] ret_from_fork_asm+0x11/0x20 [22752.180473] [22777.857396] watchdog: BUG: soft lockup - CPU#0 stuck for 10099s! [dbench:78683] [22777.864703] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22777.869395] watchdog: BUG: soft lockup - CPU#7 stuck for 19096s! [kworker/7:4:78808] [22777.920152] irq event stamp: 35381800 [22777.920154] hardirqs last enabled at (35381799): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22777.927883] Modules linked in: [22777.931539] hardirqs last disabled at (35381800): [] sysvec_apic_timer_interrupt+0xe/0x90 [22777.941697] ceph [22777.944748] softirqs last enabled at (35381776): [] __do_softirq+0x2be/0x371 [22777.954473] libceph [22777.956397] softirqs last disabled at (35381771): [] irq_exit_rcu+0xb0/0xd0 [22777.965080] dns_resolver [22777.967265] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22777.975789] fscache netfs [22777.978409] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22777.987355] veth nft_chain_nat [22777.990072] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22777.997545] xt_MASQUERADE nf_nat [22778.000685] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22778.006160] nf_conntrack [22778.009470] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22778.028210] nf_defrag_ipv6 nf_defrag_ipv4 [22778.030825] [22778.036043] nft_compat [22778.040136] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22778.041625] nf_tables nfnetlink [22778.044085] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22778.051211] bridge stp [22778.054436] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22778.061558] llc xfs [22778.064004] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22778.071126] libcrc32c sunrpc [22778.073311] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22778.080436] intel_rapl_msr intel_rapl_common [22778.083399] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22778.090523] sb_edac x86_pkg_temp_thermal [22778.094873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22778.102953] coretemp kvm_intel [22778.106955] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22778.112693] kvm irqbypass [22778.115831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22778.122956] i2c_i801 iTCO_wdt [22778.125660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22778.132795] iTCO_vendor_support i2c_smbus [22778.135842] Call Trace: [22778.142967] crct10dif_pclmul joydev [22778.147075] [22778.149518] crc32_pclmul ipmi_ssif [22778.153091] ? watchdog_timer_fn+0x1b4/0x220 [22778.155101] ghash_clmulni_intel [22778.158586] ? __hrtimer_run_queues+0x1d3/0x360 [22778.162848] mei_me mei [22778.166076] ? hrtimer_interrupt+0x104/0x240 [22778.170596] lpc_ich sha512_ssse3 [22778.173041] ? __local_bh_enable+0x50/0x60 [22778.177304] acpi_ipmi [22778.180619] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22778.184706] mfd_core wmi [22778.187075] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22778.192290] ipmi_si [22778.194907] [22778.199866] ipmi_devintf ipmi_msghandler [22778.202048] [22778.204147] ioatdma acpi_pad [22778.208150] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22778.210249] acpi_power_meter ixgbe [22778.213215] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22778.218516] igb [22778.222003] do_raw_spin_lock+0xb5/0xc0 [22778.226871] mdio crc32c_intel [22778.228711] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22778.232539] nvme ptp [22778.235593] try_get_cap_refs+0x159/0x880 [ceph] [22778.240373] nvme_core i2c_algo_bit [22778.242650] __ceph_get_caps+0x4a1/0x760 [ceph] [22778.247254] pps_core dca [22778.250743] ? __file_remove_privs+0x89/0x110 [22778.255263] fuse [22778.257880] ? wake_bit_function+0x70/0x70 [22778.262231] irq event stamp: 104565544 [22778.264157] ? ceph_write_iter+0x260/0x770 [ceph] [22778.268246] hardirqs last enabled at (104565543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22778.271990] ceph_write_iter+0x260/0x770 [ceph] [22778.276687] hardirqs last disabled at (104565544): [] sysvec_apic_timer_interrupt+0xe/0x90 [22778.286935] ? lock_is_held_type+0xaa/0x120 [22778.291456] softirqs last enabled at (104565542): [] __do_softirq+0x2be/0x371 [22778.301269] vfs_write+0x324/0x4f0 [22778.305444] softirqs last disabled at (104565537): [] irq_exit_rcu+0xb0/0xd0 [22778.314220] ksys_pwrite64+0x65/0xa0 [22778.317612] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22778.326211] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22778.329795] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22778.339177] do_syscall_64+0x38/0x80 [22778.344307] Workqueue: events delayed_work [ceph] [22778.351796] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22778.355366] [22778.355367] RIP: 0010:lock_acquire+0xf0/0x2b0 [22778.360063] RIP: 0033:0x7f48f0eeb93b [22778.365124] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22778.366616] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22778.370964] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22778.374535] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22778.393274] [22778.393275] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22778.412011] ORIG_RAX: 0000000000000012 [22778.417227] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22778.422447] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22778.423937] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22778.431077] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22778.434910] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22778.442032] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22778.449157] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22778.456282] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22778.463406] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22778.470529] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22778.477653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22778.484801] [22778.491920] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22778.522168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22778.529292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22778.536415] Call Trace: [22778.538861] [22778.540871] ? watchdog_timer_fn+0x1b4/0x220 [22778.545136] ? __hrtimer_run_queues+0x1d3/0x360 [22778.549660] ? hrtimer_interrupt+0x104/0x240 [22778.553922] ? __local_bh_enable+0x50/0x60 [22778.558014] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22778.563232] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22778.568188] [22778.570285] [22778.572382] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22778.577691] ? lock_acquire+0xf0/0x2b0 [22778.581441] ? igrab+0x43/0x50 [22778.584489] ? lock_release+0xc5/0x260 [22778.588235] _raw_spin_lock+0x30/0x40 [22778.591891] ? igrab+0x1d/0x50 [22778.594941] igrab+0x1d/0x50 [22778.597821] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22778.602972] delayed_work+0x1d8/0x280 [ceph] [22778.607252] ? process_scheduled_works+0x22a/0x510 [22778.612041] process_scheduled_works+0x2a4/0x510 [22778.616657] worker_thread+0x188/0x340 [22778.620407] ? rescuer_thread+0x310/0x310 [22778.624410] kthread+0xf5/0x130 [22778.627549] ? kthread_complete_and_exit+0x20/0x20 [22778.632331] ret_from_fork+0x31/0x40 [22778.635911] ? kthread_complete_and_exit+0x20/0x20 [22778.640697] ret_from_fork_asm+0x11/0x20 [22778.644627] [22805.857882] watchdog: BUG: soft lockup - CPU#0 stuck for 10125s! [dbench:78683] [22805.865181] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22805.869882] watchdog: BUG: soft lockup - CPU#7 stuck for 19122s! [kworker/7:4:78808] [22805.920621] irq event stamp: 35439962 [22805.920622] hardirqs last enabled at (35439961): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22805.928353] Modules linked in: [22805.932009] hardirqs last disabled at (35439962): [] sysvec_apic_timer_interrupt+0xe/0x90 [22805.942167] ceph [22805.945218] softirqs last enabled at (35439936): [] __do_softirq+0x2be/0x371 [22805.954941] libceph [22805.956866] softirqs last disabled at (35439931): [] irq_exit_rcu+0xb0/0xd0 [22805.965558] dns_resolver fscache [22805.967743] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22805.976253] netfs veth [22805.979580] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22805.988526] nft_chain_nat xt_MASQUERADE [22805.990970] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22805.998439] nf_nat nf_conntrack [22806.002358] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22806.007834] nf_defrag_ipv6 nf_defrag_ipv4 [22806.011060] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22806.029796] nft_compat nf_tables [22806.033888] [22806.039106] nfnetlink bridge [22806.042416] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22806.043907] stp llc [22806.046870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22806.053995] xfs libcrc32c [22806.056178] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22806.063302] sunrpc intel_rapl_msr [22806.066007] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22806.073131] intel_rapl_common [22806.076529] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22806.083652] sb_edac x86_pkg_temp_thermal [22806.086702] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22806.093827] coretemp [22806.097831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22806.105909] kvm_intel kvm [22806.108178] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22806.113916] irqbypass i2c_i801 [22806.116621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22806.123745] iTCO_wdt iTCO_vendor_support [22806.126883] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22806.134006] i2c_smbus crct10dif_pclmul [22806.138012] Call Trace: [22806.138014] [22806.145136] joydev [22806.148966] ? watchdog_timer_fn+0x1b4/0x220 [22806.151409] crc32_pclmul [22806.153421] ? __hrtimer_run_queues+0x1d3/0x360 [22806.155518] ipmi_ssif ghash_clmulni_intel [22806.159786] ? hrtimer_interrupt+0x104/0x240 [22806.162399] mei_me mei [22806.166923] ? __local_bh_enable+0x50/0x60 [22806.171015] lpc_ich sha512_ssse3 [22806.175282] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22806.177722] acpi_ipmi mfd_core [22806.181813] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22806.185125] wmi ipmi_si [22806.190342] [22806.193477] ipmi_devintf ipmi_msghandler [22806.198435] [22806.200966] ioatdma acpi_pad [22806.203066] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22806.207069] acpi_power_meter [22806.209170] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22806.212130] ixgbe igb [22806.217436] do_raw_spin_lock+0xb5/0xc0 [22806.220399] mdio crc32c_intel [22806.225281] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22806.227636] nvme [22806.231467] try_get_cap_refs+0x159/0x880 [ceph] [22806.234515] ptp nvme_core [22806.239305] __ceph_get_caps+0x4a1/0x760 [ceph] [22806.241225] i2c_algo_bit pps_core [22806.245839] ? __file_remove_privs+0x89/0x110 [22806.248538] dca fuse [22806.253081] ? wake_bit_function+0x70/0x70 [22806.256477] [22806.256478] irq event stamp: 104786394 [22806.260831] ? ceph_write_iter+0x260/0x770 [ceph] [22806.263099] hardirqs last enabled at (104786393): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22806.267190] ceph_write_iter+0x260/0x770 [ceph] [22806.268679] hardirqs last disabled at (104786394): [] sysvec_apic_timer_interrupt+0xe/0x90 [22806.272429] ? lock_is_held_type+0xaa/0x120 [22806.277123] softirqs last enabled at (104786392): [] __do_softirq+0x2be/0x371 [22806.287369] vfs_write+0x324/0x4f0 [22806.291890] softirqs last disabled at (104786387): [] irq_exit_rcu+0xb0/0xd0 [22806.301705] ksys_pwrite64+0x65/0xa0 [22806.305878] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22806.314651] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22806.318048] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22806.326645] do_syscall_64+0x38/0x80 [22806.330216] Workqueue: events delayed_work [ceph] [22806.339592] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22806.344722] [22806.344723] RIP: 0010:lock_acquire+0xf0/0x2b0 [22806.352194] RIP: 0033:0x7f48f0eeb93b [22806.355765] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22806.360463] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22806.365507] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22806.366997] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22806.371348] [22806.371349] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22806.374918] ORIG_RAX: 0000000000000012 [22806.393656] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22806.412395] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22806.417612] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22806.422830] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22806.424319] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22806.431443] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22806.435285] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22806.442407] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22806.449530] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22806.456655] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22806.463781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22806.470909] [22806.478027] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22806.522534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22806.529655] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22806.536781] Call Trace: [22806.539224] [22806.541237] ? watchdog_timer_fn+0x1b4/0x220 [22806.545502] ? __hrtimer_run_queues+0x1d3/0x360 [22806.550036] ? hrtimer_interrupt+0x104/0x240 [22806.554305] ? __local_bh_enable+0x50/0x60 [22806.558398] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22806.563621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22806.568579] [22806.570677] [22806.572775] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22806.578082] ? lock_acquire+0xf0/0x2b0 [22806.581832] ? igrab+0x43/0x50 [22806.584882] ? lock_release+0xc5/0x260 [22806.588628] _raw_spin_lock+0x30/0x40 [22806.592292] ? igrab+0x1d/0x50 [22806.595343] igrab+0x1d/0x50 [22806.598221] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22806.603372] delayed_work+0x1d8/0x280 [ceph] [22806.607652] ? process_scheduled_works+0x22a/0x510 [22806.612442] process_scheduled_works+0x2a4/0x510 [22806.617056] worker_thread+0x188/0x340 [22806.620807] ? rescuer_thread+0x310/0x310 [22806.624810] kthread+0xf5/0x130 [22806.627949] ? kthread_complete_and_exit+0x20/0x20 [22806.632731] ret_from_fork+0x31/0x40 [22806.636302] ? kthread_complete_and_exit+0x20/0x20 [22806.641085] ret_from_fork_asm+0x11/0x20 [22806.645009] [22833.858369] watchdog: BUG: soft lockup - CPU#0 stuck for 10151s! [dbench:78683] [22833.865675] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22833.870369] watchdog: BUG: soft lockup - CPU#7 stuck for 19148s! [kworker/7:4:78808] [22833.921126] irq event stamp: 35498172 [22833.921127] hardirqs last enabled at (35498171): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22833.928856] Modules linked in: [22833.932514] hardirqs last disabled at (35498172): [] sysvec_apic_timer_interrupt+0xe/0x90 [22833.942669] ceph [22833.945722] softirqs last enabled at (35498144): [] __do_softirq+0x2be/0x371 [22833.955446] libceph [22833.957370] softirqs last disabled at (35498139): [] irq_exit_rcu+0xb0/0xd0 [22833.966054] dns_resolver fscache [22833.968239] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22833.976765] netfs veth [22833.980075] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22833.989021] nft_chain_nat [22833.991465] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22833.998936] xt_MASQUERADE nf_nat [22834.001642] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22834.007117] nf_conntrack nf_defrag_ipv6 [22834.010428] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22834.029166] nf_defrag_ipv4 nft_compat [22834.033082] [22834.033083] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22834.038300] nf_tables nfnetlink [22834.042045] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22834.043536] bridge stp [22834.050660] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22834.053884] llc xfs [22834.061007] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22834.063450] libcrc32c sunrpc [22834.070577] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22834.072765] intel_rapl_msr [22834.079894] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22834.082857] intel_rapl_common [22834.089981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22834.092772] sb_edac x86_pkg_temp_thermal [22834.100849] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22834.103899] coretemp kvm_intel [22834.109637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22834.113641] kvm irqbypass [22834.120766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22834.123902] i2c_i801 iTCO_wdt [22834.131043] Call Trace: [22834.131046] [22834.133766] iTCO_vendor_support [22834.140892] ? watchdog_timer_fn+0x1b4/0x220 [22834.143940] i2c_smbus [22834.146386] ? __hrtimer_run_queues+0x1d3/0x360 [22834.148395] crct10dif_pclmul [22834.151622] ? hrtimer_interrupt+0x104/0x240 [22834.155883] joydev crc32_pclmul [22834.158241] ? __local_bh_enable+0x50/0x60 [22834.162768] ipmi_ssif [22834.165733] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22834.169993] ghash_clmulni_intel [22834.173218] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22834.177309] mei_me [22834.179665] [22834.184881] mei lpc_ich [22834.188105] [22834.193064] sha512_ssse3 [22834.195161] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22834.197258] acpi_ipmi [22834.199794] ? queued_spin_lock_slowpath+0x18/0x2e0 [22834.201887] mfd_core [22834.204506] do_raw_spin_lock+0xb5/0xc0 [22834.209809] wmi ipmi_si ipmi_devintf [22834.212167] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22834.217049] ipmi_msghandler ioatdma [22834.219319] try_get_cap_refs+0x159/0x880 [ceph] [22834.223146] acpi_pad acpi_power_meter [22834.226809] __ceph_get_caps+0x4a1/0x760 [ceph] [22834.231589] ixgbe [22834.235162] ? __file_remove_privs+0x89/0x110 [22834.239769] igb [22834.243514] ? wake_bit_function+0x70/0x70 [22834.248048] mdio crc32c_intel [22834.250063] ? ceph_write_iter+0x260/0x770 [ceph] [22834.254408] nvme ptp [22834.256246] ceph_write_iter+0x260/0x770 [ceph] [22834.260336] nvme_core i2c_algo_bit [22834.263389] ? lock_is_held_type+0xaa/0x120 [22834.268084] pps_core dca [22834.270358] vfs_write+0x324/0x4f0 [22834.274880] fuse [22834.278367] ksys_pwrite64+0x65/0xa0 [22834.282539] irq event stamp: 105007338 [22834.285158] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22834.288556] hardirqs last enabled at (105007337): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22834.290480] do_syscall_64+0x38/0x80 [22834.294051] hardirqs last disabled at (105007338): [] sysvec_apic_timer_interrupt+0xe/0x90 [22834.297795] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22834.302925] softirqs last enabled at (105007336): [] __do_softirq+0x2be/0x371 [22834.313170] RIP: 0033:0x7f48f0eeb93b [22834.316740] softirqs last disabled at (105007331): [] irq_exit_rcu+0xb0/0xd0 [22834.326551] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22834.331593] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22834.340365] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22834.343935] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22834.352533] ORIG_RAX: 0000000000000012 [22834.371272] Workqueue: events delayed_work [ceph] [22834.380649] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22834.385865] [22834.385866] RIP: 0010:lock_acquire+0xf0/0x2b0 [22834.393337] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22834.397169] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22834.401866] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22834.408990] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22834.410481] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22834.414829] [22834.414830] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22834.421955] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22834.440693] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22834.447822] [22834.453035] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22834.492364] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22834.499488] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22834.506612] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22834.514690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22834.520428] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22834.527552] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22834.534676] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22834.541801] Call Trace: [22834.544246] [22834.546257] ? watchdog_timer_fn+0x1b4/0x220 [22834.550520] ? __hrtimer_run_queues+0x1d3/0x360 [22834.555064] ? hrtimer_interrupt+0x104/0x240 [22834.559334] ? __local_bh_enable+0x50/0x60 [22834.563426] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22834.568649] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22834.573607] [22834.575706] [22834.577801] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22834.583112] ? lock_acquire+0xf0/0x2b0 [22834.586859] ? igrab+0x43/0x50 [22834.589911] ? lock_release+0xc5/0x260 [22834.593655] _raw_spin_lock+0x30/0x40 [22834.597321] ? igrab+0x1d/0x50 [22834.600371] igrab+0x1d/0x50 [22834.603250] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22834.608399] delayed_work+0x1d8/0x280 [ceph] [22834.612680] ? process_scheduled_works+0x22a/0x510 [22834.617470] process_scheduled_works+0x2a4/0x510 [22834.622087] worker_thread+0x188/0x340 [22834.625836] ? rescuer_thread+0x310/0x310 [22834.629847] kthread+0xf5/0x130 [22834.632994] ? kthread_complete_and_exit+0x20/0x20 [22834.637776] ret_from_fork+0x31/0x40 [22834.641358] ? kthread_complete_and_exit+0x20/0x20 [22834.646150] ret_from_fork_asm+0x11/0x20 [22834.650082] [22861.858858] watchdog: BUG: soft lockup - CPU#0 stuck for 10177s! [dbench:78683] [22861.866164] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22861.870858] watchdog: BUG: soft lockup - CPU#7 stuck for 19174s! [kworker/7:4:78808] [22861.921604] irq event stamp: 35556400 [22861.929333] Modules linked in: ceph libceph dns_resolver [22861.932992] hardirqs last enabled at (35556399): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22861.938295] fscache netfs veth [22861.948452] hardirqs last disabled at (35556400): [] sysvec_apic_timer_interrupt+0xe/0x90 [22861.951590] nft_chain_nat xt_MASQUERADE [22861.961315] softirqs last enabled at (35556372): [] __do_softirq+0x2be/0x371 [22861.961316] nf_nat nf_conntrack nf_defrag_ipv6 [22861.973934] softirqs last disabled at (35556367): [] irq_exit_rcu+0xb0/0xd0 [22861.978457] nf_defrag_ipv4 nft_compat [22861.986969] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22861.990712] nf_tables nfnetlink [22861.999655] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22862.002880] bridge stp [22862.010351] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22862.010353] llc xfs libcrc32c sunrpc [22862.018274] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22862.021931] intel_rapl_msr intel_rapl_common [22862.040668] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22862.045019] sb_edac x86_pkg_temp_thermal [22862.050253] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22862.050256] coretemp [22862.054258] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22862.061381] kvm_intel kvm [22862.063651] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22862.070777] irqbypass i2c_i801 [22862.073480] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22862.080606] iTCO_wdt iTCO_vendor_support [22862.083741] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22862.090867] i2c_smbus crct10dif_pclmul [22862.094870] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22862.101995] joydev [22862.105826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22862.113901] crc32_pclmul ipmi_ssif [22862.116001] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22862.121738] ghash_clmulni_intel mei_me [22862.125220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22862.132347] mei lpc_ich [22862.136177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22862.143300] sha512_ssse3 [22862.145832] Call Trace: [22862.152955] acpi_ipmi [22862.155574] [22862.158016] mfd_core [22862.160374] ? watchdog_timer_fn+0x1b4/0x220 [22862.162384] wmi ipmi_si [22862.164658] ? __hrtimer_run_queues+0x1d3/0x360 [22862.168919] ipmi_devintf [22862.171455] ? hrtimer_interrupt+0x104/0x240 [22862.175974] ipmi_msghandler ioatdma [22862.178592] ? __local_bh_enable+0x50/0x60 [22862.182855] acpi_pad [22862.186429] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22862.190518] acpi_power_meter [22862.192788] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22862.198004] ixgbe igb [22862.200970] [22862.205926] mdio crc32c_intel [22862.208285] [22862.210381] nvme ptp nvme_core [22862.213435] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22862.215546] i2c_algo_bit [22862.218690] ? queued_spin_lock_slowpath+0x16/0x2e0 [22862.223987] pps_core dca [22862.226609] do_raw_spin_lock+0xb5/0xc0 [22862.231476] fuse [22862.234096] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22862.237925] irq event stamp: 105228418 [22862.239852] try_get_cap_refs+0x159/0x880 [ceph] [22862.244634] hardirqs last enabled at (105228417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22862.248383] __ceph_get_caps+0x4a1/0x760 [ceph] [22862.252987] hardirqs last disabled at (105228418): [] sysvec_apic_timer_interrupt+0xe/0x90 [22862.263255] ? __file_remove_privs+0x89/0x110 [22862.267774] softirqs last enabled at (105228416): [] __do_softirq+0x2be/0x371 [22862.277584] ? wake_bit_function+0x70/0x70 [22862.281934] softirqs last disabled at (105228411): [] irq_exit_rcu+0xb0/0xd0 [22862.290711] ? ceph_write_iter+0x260/0x770 [ceph] [22862.294797] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22862.303396] ceph_write_iter+0x260/0x770 [ceph] [22862.308091] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22862.317474] ? lock_is_held_type+0xaa/0x120 [22862.321994] Workqueue: events delayed_work [ceph] [22862.329465] vfs_write+0x324/0x4f0 [22862.333642] [22862.333643] RIP: 0010:lock_acquire+0xf0/0x2b0 [22862.338343] ksys_pwrite64+0x65/0xa0 [22862.341737] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22862.343228] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22862.347578] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22862.351149] do_syscall_64+0x38/0x80 [22862.369886] [22862.369887] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22862.375017] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22862.380233] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22862.383823] RIP: 0033:0x7f48f0eeb93b [22862.385312] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22862.392436] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22862.397482] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22862.404604] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22862.408175] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22862.415299] ORIG_RAX: 0000000000000012 [22862.434037] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22862.441161] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22862.446378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22862.453504] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22862.457335] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22862.465410] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22862.472543] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22862.478282] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22862.485407] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22862.492542] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22862.499663] Call Trace: [22862.499664] [22862.506791] [22862.513913] ? watchdog_timer_fn+0x1b4/0x220 [22862.539065] ? __hrtimer_run_queues+0x1d3/0x360 [22862.543589] ? hrtimer_interrupt+0x104/0x240 [22862.547850] ? __local_bh_enable+0x50/0x60 [22862.551944] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22862.557168] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22862.562125] [22862.564222] [22862.566320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22862.571629] ? lock_acquire+0xf0/0x2b0 [22862.575379] ? igrab+0x43/0x50 [22862.578438] ? lock_release+0xc5/0x260 [22862.582183] _raw_spin_lock+0x30/0x40 [22862.585847] ? igrab+0x1d/0x50 [22862.588896] igrab+0x1d/0x50 [22862.591774] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22862.596926] delayed_work+0x1d8/0x280 [ceph] [22862.601205] ? process_scheduled_works+0x22a/0x510 [22862.605989] process_scheduled_works+0x2a4/0x510 [22862.610602] worker_thread+0x188/0x340 [22862.614353] ? rescuer_thread+0x310/0x310 [22862.618356] kthread+0xf5/0x130 [22862.621492] ? kthread_complete_and_exit+0x20/0x20 [22862.626278] ret_from_fork+0x31/0x40 [22862.629856] ? kthread_complete_and_exit+0x20/0x20 [22862.634640] ret_from_fork_asm+0x11/0x20 [22862.638565] [22889.859348] watchdog: BUG: soft lockup - CPU#0 stuck for 10203s! [dbench:78683] [22889.866649] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22889.871347] watchdog: BUG: soft lockup - CPU#7 stuck for 19200s! [kworker/7:4:78808] [22889.922099] irq event stamp: 35614516 [22889.922100] hardirqs last enabled at (35614515): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22889.929830] Modules linked in: [22889.933487] hardirqs last disabled at (35614516): [] sysvec_apic_timer_interrupt+0xe/0x90 [22889.943645] ceph [22889.946695] softirqs last enabled at (35614488): [] __do_softirq+0x2be/0x371 [22889.956420] libceph [22889.958343] softirqs last disabled at (35614483): [] irq_exit_rcu+0xb0/0xd0 [22889.967027] dns_resolver fscache [22889.969211] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22889.977739] netfs [22889.981050] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22889.989994] veth nft_chain_nat [22889.992005] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22889.999476] xt_MASQUERADE [22890.002615] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22890.008091] nf_nat nf_conntrack [22890.010795] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22890.029532] nf_defrag_ipv6 nf_defrag_ipv4 [22890.032756] [22890.037975] nft_compat nf_tables [22890.042064] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22890.043555] nfnetlink bridge [22890.046867] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22890.053989] stp llc [22890.056955] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22890.064078] xfs [22890.066262] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22890.073386] libcrc32c sunrpc [22890.075225] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22890.082349] intel_rapl_msr intel_rapl_common [22890.085313] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22890.092436] sb_edac x86_pkg_temp_thermal [22890.096786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22890.104864] coretemp kvm_intel [22890.108867] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22890.114605] kvm irqbypass [22890.117746] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22890.124868] i2c_i801 iTCO_wdt [22890.127572] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22890.134696] iTCO_vendor_support [22890.137746] Call Trace: [22890.144870] i2c_smbus crct10dif_pclmul [22890.148098] [22890.150537] joydev crc32_pclmul [22890.154372] ? watchdog_timer_fn+0x1b4/0x220 [22890.156380] ipmi_ssif [22890.159605] ? __hrtimer_run_queues+0x1d3/0x360 [22890.163869] ghash_clmulni_intel [22890.166228] ? hrtimer_interrupt+0x104/0x240 [22890.170750] mei_me [22890.173974] ? __local_bh_enable+0x50/0x60 [22890.178238] mei lpc_ich [22890.180338] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22890.184427] sha512_ssse3 [22890.186956] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22890.192174] acpi_ipmi mfd_core [22890.194792] [22890.199748] wmi ipmi_si [22890.202886] [22890.204985] ipmi_devintf ipmi_msghandler [22890.207516] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22890.209612] ioatdma acpi_pad [22890.213619] ? queued_spin_lock_slowpath+0x16/0x2e0 [22890.218921] acpi_power_meter [22890.221887] do_raw_spin_lock+0xb5/0xc0 [22890.226754] ixgbe [22890.229723] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22890.233568] igb mdio [22890.235581] try_get_cap_refs+0x159/0x880 [ceph] [22890.240361] crc32c_intel [22890.242638] __ceph_get_caps+0x4a1/0x760 [ceph] [22890.247243] nvme ptp [22890.249866] ? __file_remove_privs+0x89/0x110 [22890.254385] nvme_core i2c_algo_bit [22890.256656] ? wake_bit_function+0x70/0x70 [22890.261007] pps_core [22890.264492] ? ceph_write_iter+0x260/0x770 [ceph] [22890.268580] dca fuse [22890.270853] ceph_write_iter+0x260/0x770 [ceph] [22890.275548] [22890.275549] irq event stamp: 105449460 [22890.277823] ? lock_is_held_type+0xaa/0x120 [22890.282344] hardirqs last enabled at (105449459): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22890.283837] vfs_write+0x324/0x4f0 [22890.287579] hardirqs last disabled at (105449460): [] sysvec_apic_timer_interrupt+0xe/0x90 [22890.291761] ksys_pwrite64+0x65/0xa0 [22890.302000] softirqs last enabled at (105449458): [] __do_softirq+0x2be/0x371 [22890.305399] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22890.315209] softirqs last disabled at (105449453): [] irq_exit_rcu+0xb0/0xd0 [22890.318782] do_syscall_64+0x38/0x80 [22890.327549] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22890.332682] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22890.341278] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22890.344849] RIP: 0033:0x7f48f0eeb93b [22890.354227] Workqueue: events delayed_work [ceph] [22890.359272] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22890.366749] [22890.366750] RIP: 0010:rcu_is_watching+0x0/0x40 [22890.370320] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22890.375033] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [22890.393774] ORIG_RAX: 0000000000000012 [22890.395265] RSP: 0018:ffffc90002117d10 EFLAGS: 00000247 [22890.399702] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22890.404919] [22890.404919] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [22890.423656] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22890.427488] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [22890.432706] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22890.439828] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [22890.441320] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22890.448443] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [22890.455567] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22890.462693] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22890.469821] [22890.476939] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22890.515699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22890.521435] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22890.528560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22890.535683] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22890.542808] Call Trace: [22890.545253] [22890.547263] ? watchdog_timer_fn+0x1b4/0x220 [22890.551528] ? __hrtimer_run_queues+0x1d3/0x360 [22890.556054] ? hrtimer_interrupt+0x104/0x240 [22890.560323] ? __local_bh_enable+0x50/0x60 [22890.564417] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22890.569641] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22890.574598] [22890.576696] [22890.578792] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22890.584100] ? rcutorture_get_gp_data+0x30/0x30 [22890.588630] lock_acquire+0x184/0x2b0 [22890.592289] ? igrab+0x43/0x50 [22890.595339] ? lock_release+0xc5/0x260 [22890.599083] _raw_spin_lock+0x30/0x40 [22890.602752] ? igrab+0x1d/0x50 [22890.605809] igrab+0x1d/0x50 [22890.608686] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22890.613835] delayed_work+0x1d8/0x280 [ceph] [22890.618118] ? process_scheduled_works+0x22a/0x510 [22890.622906] process_scheduled_works+0x2a4/0x510 [22890.627523] worker_thread+0x188/0x340 [22890.631273] ? rescuer_thread+0x310/0x310 [22890.635274] kthread+0xf5/0x130 [22890.638413] ? kthread_complete_and_exit+0x20/0x20 [22890.643198] ret_from_fork+0x31/0x40 [22890.646777] ? kthread_complete_and_exit+0x20/0x20 [22890.651559] ret_from_fork_asm+0x11/0x20 [22890.655493] [22917.859832] watchdog: BUG: soft lockup - CPU#0 stuck for 10229s! [dbench:78683] [22917.867137] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22917.871832] watchdog: BUG: soft lockup - CPU#7 stuck for 19226s! [kworker/7:4:78808] [22917.922577] irq event stamp: 35672730 [22917.922578] hardirqs last enabled at (35672729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22917.930307] Modules linked in: ceph [22917.933966] hardirqs last disabled at (35672730): [] sysvec_apic_timer_interrupt+0xe/0x90 [22917.944121] libceph [22917.947607] softirqs last enabled at (35672702): [] __do_softirq+0x2be/0x371 [22917.957331] dns_resolver fscache [22917.959516] softirqs last disabled at (35672697): [] irq_exit_rcu+0xb0/0xd0 [22917.968200] netfs veth [22917.971526] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22917.980037] nft_chain_nat [22917.982482] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22917.991444] xt_MASQUERADE nf_nat [22917.994149] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [22918.001619] nf_conntrack nf_defrag_ipv6 [22918.004928] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [22918.010407] nf_defrag_ipv4 nft_compat [22918.014324] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22918.033061] nf_tables nfnetlink [22918.036805] [22918.036806] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22918.042023] bridge [22918.045247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22918.046737] stp llc [22918.053862] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22918.055960] xfs libcrc32c [22918.063082] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22918.065267] sunrpc [22918.072391] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22918.075096] intel_rapl_msr intel_rapl_common [22918.082231] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22918.084327] sb_edac [22918.091451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22918.095801] x86_pkg_temp_thermal [22918.103877] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22918.106063] coretemp kvm_intel [22918.111799] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22918.115111] kvm [22918.122235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22918.125371] irqbypass i2c_i801 [22918.132512] Call Trace: [22918.132513] [22918.134350] iTCO_wdt iTCO_vendor_support [22918.141476] ? watchdog_timer_fn+0x1b4/0x220 [22918.144612] i2c_smbus crct10dif_pclmul [22918.147059] ? __hrtimer_run_queues+0x1d3/0x360 [22918.149068] joydev crc32_pclmul [22918.153075] ? hrtimer_interrupt+0x104/0x240 [22918.157336] ipmi_ssif [22918.161166] ? __local_bh_enable+0x50/0x60 [22918.165691] ghash_clmulni_intel [22918.168917] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22918.173179] mei_me [22918.175536] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22918.179627] mei lpc_ich [22918.182850] [22918.188068] sha512_ssse3 acpi_ipmi [22918.190165] [22918.195124] mfd_core wmi [22918.197654] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22918.199752] ipmi_si ipmi_devintf [22918.203238] ? queued_spin_lock_slowpath+0x16/0x2e0 [22918.205332] ipmi_msghandler ioatdma [22918.207951] do_raw_spin_lock+0xb5/0xc0 [22918.213252] acpi_pad acpi_power_meter ixgbe [22918.216566] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22918.221435] igb mdio [22918.225008] try_get_cap_refs+0x159/0x880 [ceph] [22918.228837] crc32c_intel nvme [22918.233106] __ceph_get_caps+0x4a1/0x760 [ceph] [22918.237884] ptp nvme_core [22918.240160] ? __file_remove_privs+0x89/0x110 [22918.244766] i2c_algo_bit pps_core [22918.247818] ? wake_bit_function+0x70/0x70 [22918.252341] dca fuse [22918.255047] ? ceph_write_iter+0x260/0x770 [ceph] [22918.259396] [22918.259397] irq event stamp: 105670368 [22918.262794] ceph_write_iter+0x260/0x770 [ceph] [22918.266885] hardirqs last enabled at (105670367): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22918.269157] ? lock_is_held_type+0xaa/0x120 [22918.273852] hardirqs last disabled at (105670368): [] sysvec_apic_timer_interrupt+0xe/0x90 [22918.275346] vfs_write+0x324/0x4f0 [22918.279088] softirqs last enabled at (105670366): [] __do_softirq+0x2be/0x371 [22918.283614] ksys_pwrite64+0x65/0xa0 [22918.293855] softirqs last disabled at (105670361): [] irq_exit_rcu+0xb0/0xd0 [22918.298033] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22918.307843] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22918.311242] do_syscall_64+0x38/0x80 [22918.320012] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22918.323582] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22918.332180] Workqueue: events delayed_work [ceph] [22918.337311] RIP: 0033:0x7f48f0eeb93b [22918.346687] [22918.350260] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22918.357728] RIP: 0010:lock_acquire+0xf0/0x2b0 [22918.362774] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22918.367470] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22918.371042] ORIG_RAX: 0000000000000012 [22918.372532] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22918.391269] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22918.395621] [22918.400839] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22918.419576] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22918.423408] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22918.428624] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22918.435748] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22918.437240] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22918.444363] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22918.451486] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22918.458617] [22918.465735] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22918.503550] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22918.511626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22918.517365] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22918.524489] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22918.531629] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22918.538753] Call Trace: [22918.541199] [22918.543211] ? watchdog_timer_fn+0x1b4/0x220 [22918.547492] ? __hrtimer_run_queues+0x1d3/0x360 [22918.552036] ? hrtimer_interrupt+0x104/0x240 [22918.556304] ? __local_bh_enable+0x50/0x60 [22918.560399] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22918.565622] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22918.570578] [22918.572675] [22918.574774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22918.580081] ? lock_acquire+0xf0/0x2b0 [22918.583832] ? igrab+0x43/0x50 [22918.586881] ? lock_release+0xc5/0x260 [22918.590627] _raw_spin_lock+0x30/0x40 [22918.594282] ? igrab+0x1d/0x50 [22918.597332] igrab+0x1d/0x50 [22918.600213] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22918.605378] delayed_work+0x1d8/0x280 [ceph] [22918.609659] ? process_scheduled_works+0x22a/0x510 [22918.614441] process_scheduled_works+0x2a4/0x510 [22918.619056] worker_thread+0x188/0x340 [22918.622805] ? rescuer_thread+0x310/0x310 [22918.626810] kthread+0xf5/0x130 [22918.629947] ? kthread_complete_and_exit+0x20/0x20 [22918.634732] ret_from_fork+0x31/0x40 [22918.638300] ? kthread_complete_and_exit+0x20/0x20 [22918.643085] ret_from_fork_asm+0x11/0x20 [22918.647008] [22945.860316] watchdog: BUG: soft lockup - CPU#0 stuck for 10255s! [dbench:78683] [22945.867614] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22945.872316] watchdog: BUG: soft lockup - CPU#7 stuck for 19253s! [kworker/7:4:78808] [22945.923055] irq event stamp: 35730940 [22945.923056] hardirqs last enabled at (35730939): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22945.930786] Modules linked in: [22945.934444] hardirqs last disabled at (35730940): [] sysvec_apic_timer_interrupt+0xe/0x90 [22945.944600] ceph [22945.947651] softirqs last enabled at (35730912): [] __do_softirq+0x2be/0x371 [22945.957375] libceph [22945.959299] softirqs last disabled at (35730907): [] irq_exit_rcu+0xb0/0xd0 [22945.967991] dns_resolver [22945.970176] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22945.978688] fscache [22945.981323] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22945.981324] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [22945.990266] netfs veth [22945.992450] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [22945.999922] nft_chain_nat xt_MASQUERADE [22946.005400] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22946.007844] nf_nat nf_conntrack [22946.026582] [22946.026583] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22946.030498] nf_defrag_ipv6 nf_defrag_ipv4 [22946.035715] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22946.038940] nft_compat [22946.040431] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22946.047556] nf_tables nfnetlink [22946.051646] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22946.058768] bridge stp [22946.061215] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22946.068338] llc xfs [22946.071563] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22946.078686] libcrc32c sunrpc [22946.081148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22946.088272] intel_rapl_msr [22946.090456] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22946.098532] intel_rapl_common sb_edac [22946.101498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22946.107234] x86_pkg_temp_thermal [22946.110024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22946.117149] coretemp [22946.120894] Call Trace: [22946.120895] [22946.128017] kvm_intel [22946.131329] ? watchdog_timer_fn+0x1b4/0x220 [22946.138453] kvm irqbypass [22946.140725] ? __hrtimer_run_queues+0x1d3/0x360 [22946.143168] i2c_i801 iTCO_wdt [22946.145181] ? hrtimer_interrupt+0x104/0x240 [22946.147535] iTCO_vendor_support [22946.151800] ? __local_bh_enable+0x50/0x60 [22946.154504] i2c_smbus crct10dif_pclmul [22946.159030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22946.162078] joydev [22946.166343] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22946.169567] crc32_pclmul ipmi_ssif [22946.173658] [22946.177487] ghash_clmulni_intel [22946.182716] [22946.182718] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22946.184812] mei_me [22946.189774] ? queued_spin_lock_slowpath+0x2d/0x2e0 [22946.193253] mei lpc_ich [22946.195353] do_raw_spin_lock+0xb5/0xc0 [22946.198573] sha512_ssse3 acpi_ipmi [22946.200671] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22946.205977] mfd_core [22946.208075] try_get_cap_refs+0x159/0x880 [ceph] [22946.212945] wmi [22946.215480] __ceph_get_caps+0x4a1/0x760 [ceph] [22946.219304] ipmi_si ipmi_devintf [22946.222792] ? __file_remove_privs+0x89/0x110 [22946.227574] ipmi_msghandler [22946.229844] ? wake_bit_function+0x70/0x70 [22946.234455] ioatdma [22946.236294] ? ceph_write_iter+0x260/0x770 [ceph] [22946.240817] acpi_pad [22946.244128] ceph_write_iter+0x260/0x770 [ceph] [22946.248478] acpi_power_meter ixgbe [22946.251359] ? lock_is_held_type+0xaa/0x120 [22946.255447] igb [22946.257633] vfs_write+0x324/0x4f0 [22946.262328] mdio crc32c_intel [22946.264602] ksys_pwrite64+0x65/0xa0 [22946.269123] nvme [22946.272607] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22946.276785] ptp nvme_core [22946.278622] do_syscall_64+0x38/0x80 [22946.282019] i2c_algo_bit [22946.285070] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22946.288639] pps_core [22946.290565] RIP: 0033:0x7f48f0eeb93b [22946.295694] dca fuse [22946.298416] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22946.301987] [22946.301988] irq event stamp: 105891282 [22946.304604] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22946.309648] hardirqs last enabled at (105891281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22946.311918] ORIG_RAX: 0000000000000012 [22946.315489] hardirqs last disabled at (105891282): [] sysvec_apic_timer_interrupt+0xe/0x90 [22946.317759] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22946.336497] softirqs last enabled at (105891280): [] __do_softirq+0x2be/0x371 [22946.338005] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22946.341749] softirqs last disabled at (105891275): [] irq_exit_rcu+0xb0/0xd0 [22946.346967] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22946.357214] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22946.361043] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22946.370852] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22946.377978] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22946.386767] Workqueue: events delayed_work [ceph] [22946.393896] [22946.402487] [22946.402488] RIP: 0010:lock_acquire+0xf0/0x2b0 [22946.453448] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22946.472185] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22946.477402] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22946.484528] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22946.491652] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22946.498775] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22946.505900] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22946.513024] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22946.521102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22946.526838] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22946.533963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22946.541088] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22946.548211] Call Trace: [22946.550656] [22946.552667] ? watchdog_timer_fn+0x1b4/0x220 [22946.556933] ? __hrtimer_run_queues+0x1d3/0x360 [22946.561466] ? hrtimer_interrupt+0x104/0x240 [22946.565729] ? __local_bh_enable+0x50/0x60 [22946.569821] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22946.575045] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22946.580010] [22946.582107] [22946.584205] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22946.589512] ? lock_acquire+0xf0/0x2b0 [22946.593263] ? igrab+0x43/0x50 [22946.596314] ? lock_release+0xc5/0x260 [22946.600057] _raw_spin_lock+0x30/0x40 [22946.603725] ? igrab+0x1d/0x50 [22946.606773] igrab+0x1d/0x50 [22946.609653] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22946.614802] delayed_work+0x1d8/0x280 [ceph] [22946.619084] ? process_scheduled_works+0x22a/0x510 [22946.623873] process_scheduled_works+0x2a4/0x510 [22946.628490] worker_thread+0x188/0x340 [22946.632238] ? rescuer_thread+0x310/0x310 [22946.636241] kthread+0xf5/0x130 [22946.639380] ? kthread_complete_and_exit+0x20/0x20 [22946.644171] ret_from_fork+0x31/0x40 [22946.647742] ? kthread_complete_and_exit+0x20/0x20 [22946.652525] ret_from_fork_asm+0x11/0x20 [22946.656449] [22947.187705] rcu: INFO: rcu_preempt self-detected stall on CPU [22947.193466] rcu: 0-....: (10762740 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=5372445 [22947.203885] rcu: (t=11010381 jiffies g=1309609 q=361110 ncpus=8) [22947.209968] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22947.218910] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22947.226382] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22947.231860] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22947.250596] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22947.255815] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22947.262938] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22947.270062] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22947.277186] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22947.284310] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22947.291436] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22947.299512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22947.305251] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22947.312376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22947.319500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22947.326623] Call Trace: [22947.329066] [22947.331079] ? rcu_dump_cpu_stacks+0xd1/0x180 [22947.335438] ? rcu_sched_clock_irq+0x633/0x1390 [22947.339961] ? lock_is_held_type+0xaa/0x120 [22947.344138] ? lock_is_held_type+0xaa/0x120 [22947.348317] ? tick_sched_do_timer+0xa0/0xa0 [22947.352589] ? rcu_read_lock_sched_held+0x56/0x90 [22947.357287] ? tick_sched_do_timer+0xa0/0xa0 [22947.361550] ? update_process_times+0x5f/0x90 [22947.365900] ? tick_sched_handle.isra.16+0x1e/0x50 [22947.370686] ? tick_sched_timer+0x6f/0x80 [22947.374689] ? __hrtimer_run_queues+0x1d3/0x360 [22947.379217] ? hrtimer_interrupt+0x104/0x240 [22947.383486] ? __local_bh_enable+0x50/0x60 [22947.387579] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22947.392804] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22947.397760] [22947.399856] [22947.401956] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22947.407263] ? queued_spin_lock_slowpath+0x18/0x2e0 [22947.412140] do_raw_spin_lock+0xb5/0xc0 [22947.415978] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22947.420781] try_get_cap_refs+0x159/0x880 [ceph] [22947.425414] __ceph_get_caps+0x4a1/0x760 [ceph] [22947.429952] ? __file_remove_privs+0x89/0x110 [22947.434308] ? wake_bit_function+0x70/0x70 [22947.438402] ? ceph_write_iter+0x260/0x770 [ceph] [22947.443113] ceph_write_iter+0x260/0x770 [ceph] [22947.447649] ? lock_is_held_type+0xaa/0x120 [22947.451836] vfs_write+0x324/0x4f0 [22947.455242] ksys_pwrite64+0x65/0xa0 [22947.458817] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22947.463951] do_syscall_64+0x38/0x80 [22947.467529] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22947.472572] RIP: 0033:0x7f48f0eeb93b [22947.476142] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22947.494879] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [22947.502437] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22947.509562] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22947.516687] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22947.523810] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22947.530933] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22947.538063] [22947.540252] Sending NMI from CPU 0 to CPUs 7: [22947.544605] NMI backtrace for cpu 7 [22947.544606] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22947.544608] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22947.544609] Workqueue: events delayed_work [ceph] [22947.544624] RIP: 0010:lock_acquire+0x85/0x2b0 [22947.544626] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [22947.544628] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [22947.544629] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [22947.544629] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [22947.544630] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [22947.544631] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [22947.544631] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22947.544632] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22947.544633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22947.544634] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22947.544635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22947.544635] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22947.544636] Call Trace: [22947.544637] [22947.544638] ? nmi_cpu_backtrace+0xad/0x120 [22947.544641] ? nmi_cpu_backtrace_handler+0x11/0x20 [22947.544643] ? nmi_handle+0xd4/0x260 [22947.544647] ? default_do_nmi+0x49/0xf0 [22947.544648] ? exc_nmi+0x11e/0x150 [22947.544650] ? end_repeat_nmi+0x16/0x67 [22947.544654] ? lock_acquire+0x85/0x2b0 [22947.544656] ? lock_acquire+0x85/0x2b0 [22947.544657] ? lock_acquire+0x85/0x2b0 [22947.544659] [22947.544659] [22947.544660] ? igrab+0x43/0x50 [22947.544661] ? lock_release+0xc5/0x260 [22947.544663] _raw_spin_lock+0x30/0x40 [22947.544665] ? igrab+0x1d/0x50 [22947.544666] igrab+0x1d/0x50 [22947.544668] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22947.544682] delayed_work+0x1d8/0x280 [ceph] [22947.544694] ? process_scheduled_works+0x22a/0x510 [22947.544697] process_scheduled_works+0x2a4/0x510 [22947.544701] worker_thread+0x188/0x340 [22947.544703] ? rescuer_thread+0x310/0x310 [22947.544705] kthread+0xf5/0x130 [22947.544708] ? kthread_complete_and_exit+0x20/0x20 [22947.544710] ret_from_fork+0x31/0x40 [22947.544713] ? kthread_complete_and_exit+0x20/0x20 [22947.544716] ret_from_fork_asm+0x11/0x20 [22947.544720] [22973.860799] watchdog: BUG: soft lockup - CPU#0 stuck for 10282s! [dbench:78683] [22973.868102] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [22973.872799] watchdog: BUG: soft lockup - CPU#7 stuck for 19279s! [kworker/7:4:78808] [22973.923551] irq event stamp: 35788554 [22973.923552] hardirqs last enabled at (35788553): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22973.931282] Modules linked in: ceph [22973.934939] hardirqs last disabled at (35788554): [] sysvec_apic_timer_interrupt+0xe/0x90 [22973.945097] libceph [22973.948581] softirqs last enabled at (35788476): [] __do_softirq+0x2be/0x371 [22973.958307] dns_resolver [22973.960489] softirqs last disabled at (35788471): [] irq_exit_rcu+0xb0/0xd0 [22973.969192] fscache [22973.971810] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22973.980319] netfs veth [22973.982504] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22973.991449] nft_chain_nat [22973.993893] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [22974.001363] xt_MASQUERADE nf_nat [22974.004067] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [22974.009544] nf_conntrack nf_defrag_ipv6 [22974.012855] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [22974.031594] nf_defrag_ipv4 nft_compat [22974.035511] [22974.035512] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [22974.040727] nf_tables nfnetlink [22974.044474] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [22974.045962] bridge stp [22974.053086] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [22974.056311] llc xfs [22974.063435] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [22974.065879] libcrc32c sunrpc [22974.073002] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [22974.075195] intel_rapl_msr [22974.082320] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [22974.085282] intel_rapl_common [22974.092406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22974.095199] sb_edac [22974.103277] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [22974.106325] x86_pkg_temp_thermal coretemp [22974.112065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22974.114249] kvm_intel kvm [22974.121371] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22974.125477] irqbypass i2c_i801 [22974.132603] Call Trace: [22974.135309] iTCO_wdt iTCO_vendor_support [22974.142434] [22974.145570] i2c_smbus crct10dif_pclmul [22974.148016] ? watchdog_timer_fn+0x1b4/0x220 [22974.152017] joydev crc32_pclmul [22974.154031] ? __hrtimer_run_queues+0x1d3/0x360 [22974.157859] ipmi_ssif ghash_clmulni_intel [22974.162127] ? hrtimer_interrupt+0x104/0x240 [22974.165347] mei_me [22974.169871] ? __local_bh_enable+0x50/0x60 [22974.173963] mei lpc_ich [22974.178229] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22974.180324] sha512_ssse3 acpi_ipmi [22974.184414] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22974.186944] mfd_core [22974.192164] [22974.195646] wmi ipmi_si [22974.200603] [22974.200605] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22974.202875] ipmi_devintf ipmi_msghandler [22974.204976] ? queued_spin_lock_slowpath+0x18/0x2e0 [22974.207504] ioatdma acpi_pad [22974.209602] do_raw_spin_lock+0xb5/0xc0 [22974.214904] acpi_power_meter ixgbe igb [22974.218909] __cap_delay_requeue+0xa7/0x1d0 [ceph] [22974.223779] mdio crc32c_intel [22974.226746] try_get_cap_refs+0x159/0x880 [ceph] [22974.230573] nvme ptp [22974.234409] __ceph_get_caps+0x4a1/0x760 [ceph] [22974.239199] nvme_core i2c_algo_bit [22974.242252] ? __file_remove_privs+0x89/0x110 [22974.246858] pps_core dca [22974.249130] ? wake_bit_function+0x70/0x70 [22974.253653] fuse [22974.257140] ? ceph_write_iter+0x260/0x770 [ceph] [22974.261488] irq event stamp: 106110368 [22974.264108] ceph_write_iter+0x260/0x770 [ceph] [22974.268199] hardirqs last enabled at (106110367): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22974.270124] ? lock_is_held_type+0xaa/0x120 [22974.274819] hardirqs last disabled at (106110368): [] sysvec_apic_timer_interrupt+0xe/0x90 [22974.278564] vfs_write+0x324/0x4f0 [22974.283087] softirqs last enabled at (106110366): [] __do_softirq+0x2be/0x371 [22974.293334] ksys_pwrite64+0x65/0xa0 [22974.297508] softirqs last disabled at (106110361): [] irq_exit_rcu+0xb0/0xd0 [22974.307318] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [22974.310717] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [22974.319489] do_syscall_64+0x38/0x80 [22974.323058] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [22974.331657] entry_SYSCALL_64_after_hwframe+0x63/0xcd [22974.336786] Workqueue: events delayed_work [ceph] [22974.346165] RIP: 0033:0x7f48f0eeb93b [22974.349734] [22974.357204] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [22974.362250] RIP: 0010:lock_acquire+0xf0/0x2b0 [22974.366945] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [22974.370517] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [22974.372008] ORIG_RAX: 0000000000000012 [22974.390746] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [22974.395097] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [22974.400315] [22974.400316] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [22974.419050] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [22974.422882] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [22974.428099] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [22974.435225] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [22974.436715] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [22974.443839] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [22974.450963] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [22974.458088] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [22974.465217] [22974.472335] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [22974.511112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22974.516849] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [22974.523972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [22974.531095] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [22974.538219] Call Trace: [22974.540664] [22974.542676] ? watchdog_timer_fn+0x1b4/0x220 [22974.546940] ? __hrtimer_run_queues+0x1d3/0x360 [22974.551467] ? hrtimer_interrupt+0x104/0x240 [22974.555746] ? __local_bh_enable+0x50/0x60 [22974.559837] ? __sysvec_apic_timer_interrupt+0x61/0x160 [22974.565052] ? sysvec_apic_timer_interrupt+0x6d/0x90 [22974.570010] [22974.572108] [22974.574206] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [22974.579522] ? lock_acquire+0xf0/0x2b0 [22974.583274] ? igrab+0x43/0x50 [22974.586321] ? lock_release+0xc5/0x260 [22974.590068] _raw_spin_lock+0x30/0x40 [22974.593724] ? igrab+0x1d/0x50 [22974.596774] igrab+0x1d/0x50 [22974.599653] ceph_check_delayed_caps+0xb7/0x270 [ceph] [22974.604802] delayed_work+0x1d8/0x280 [ceph] [22974.609083] ? process_scheduled_works+0x22a/0x510 [22974.613874] process_scheduled_works+0x2a4/0x510 [22974.618490] worker_thread+0x188/0x340 [22974.622239] ? rescuer_thread+0x310/0x310 [22974.626250] kthread+0xf5/0x130 [22974.629389] ? kthread_complete_and_exit+0x20/0x20 [22974.634173] ret_from_fork+0x31/0x40 [22974.637743] ? kthread_complete_and_exit+0x20/0x20 [22974.642528] ret_from_fork_asm+0x11/0x20 [22974.646449] [23001.861280] watchdog: BUG: soft lockup - CPU#0 stuck for 10308s! [dbench:78683] [23001.868581] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23001.873280] watchdog: BUG: soft lockup - CPU#7 stuck for 19305s! [kworker/7:4:78808] [23001.924023] irq event stamp: 35846732 [23001.924024] hardirqs last enabled at (35846731): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23001.931752] Modules linked in: [23001.935410] hardirqs last disabled at (35846732): [] sysvec_apic_timer_interrupt+0xe/0x90 [23001.945568] ceph [23001.948619] softirqs last enabled at (35846656): [] __do_softirq+0x2be/0x371 [23001.958342] libceph [23001.960265] softirqs last disabled at (35846651): [] irq_exit_rcu+0xb0/0xd0 [23001.968955] dns_resolver fscache [23001.971142] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23001.979655] netfs veth [23001.982981] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23001.982982] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23001.991926] nft_chain_nat xt_MASQUERADE [23001.994371] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23002.001840] nf_nat nf_conntrack [23002.007320] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23002.011236] nf_defrag_ipv6 nf_defrag_ipv4 [23002.029975] [23002.029975] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23002.033199] nft_compat nf_tables [23002.038415] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23002.042506] nfnetlink bridge [23002.043996] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23002.051122] stp llc [23002.054431] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23002.061556] xfs libcrc32c [23002.064520] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23002.071645] sunrpc [23002.073829] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23002.080960] intel_rapl_msr [23002.083678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23002.090797] intel_rapl_common [23002.092894] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23002.100973] sb_edac x86_pkg_temp_thermal [23002.103764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23002.109500] coretemp kvm_intel [23002.112552] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23002.119681] kvm irqbypass [23002.123687] Call Trace: [23002.123688] [23002.130812] i2c_i801 iTCO_wdt [23002.133961] ? watchdog_timer_fn+0x1b4/0x220 [23002.141084] iTCO_vendor_support [23002.143787] ? __hrtimer_run_queues+0x1d3/0x360 [23002.146230] i2c_smbus [23002.148245] ? hrtimer_interrupt+0x104/0x240 [23002.151292] crct10dif_pclmul joydev [23002.155556] ? __local_bh_enable+0x50/0x60 [23002.158780] crc32_pclmul ipmi_ssif [23002.163308] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23002.165676] ghash_clmulni_intel [23002.169960] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23002.173532] mei_me mei [23002.177623] [23002.181106] lpc_ich sha512_ssse3 [23002.186325] [23002.186326] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23002.189548] acpi_ipmi mfd_core [23002.194509] ? queued_spin_lock_slowpath+0x16/0x2e0 [23002.196964] wmi ipmi_si [23002.199066] do_raw_spin_lock+0xb5/0xc0 [23002.202375] ipmi_devintf ipmi_msghandler [23002.204473] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23002.209776] ioatdma acpi_pad [23002.212916] try_get_cap_refs+0x159/0x880 [ceph] [23002.217783] acpi_power_meter ixgbe [23002.220320] __ceph_get_caps+0x4a1/0x760 [ceph] [23002.224144] igb mdio [23002.228154] ? __file_remove_privs+0x89/0x110 [23002.232935] crc32c_intel nvme [23002.235899] ? wake_bit_function+0x70/0x70 [23002.240508] ptp nvme_core [23002.243996] ? ceph_write_iter+0x260/0x770 [ceph] [23002.248518] i2c_algo_bit pps_core [23002.250788] ceph_write_iter+0x260/0x770 [ceph] [23002.255139] dca fuse [23002.258192] ? lock_is_held_type+0xaa/0x120 [23002.262279] [23002.264987] vfs_write+0x324/0x4f0 [23002.269681] irq event stamp: 106331234 [23002.273084] ksys_pwrite64+0x65/0xa0 [23002.277604] hardirqs last enabled at (106331233): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23002.279874] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23002.284050] hardirqs last disabled at (106331234): [] sysvec_apic_timer_interrupt+0xe/0x90 [23002.285541] do_syscall_64+0x38/0x80 [23002.288938] softirqs last enabled at (106331232): [] __do_softirq+0x2be/0x371 [23002.292700] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23002.296270] softirqs last disabled at (106331227): [] irq_exit_rcu+0xb0/0xd0 [23002.306516] RIP: 0033:0x7f48f0eeb93b [23002.311647] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23002.321457] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23002.325027] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23002.325028] Workqueue: events delayed_work [ceph] [23002.333798] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23002.338842] [23002.338843] RIP: 0010:lock_acquire+0xf0/0x2b0 [23002.347440] ORIG_RAX: 0000000000000012 [23002.351011] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23002.360388] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23002.379126] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23002.386597] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23002.391294] [23002.391295] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23002.396512] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23002.398001] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23002.402353] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23002.406182] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23002.424922] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23002.432046] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23002.437267] [23002.444386] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23002.505056] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23002.513133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23002.518871] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23002.525994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23002.533117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23002.540241] Call Trace: [23002.542688] [23002.544697] ? watchdog_timer_fn+0x1b4/0x220 [23002.548968] ? __hrtimer_run_queues+0x1d3/0x360 [23002.553498] ? hrtimer_interrupt+0x104/0x240 [23002.557765] ? __local_bh_enable+0x50/0x60 [23002.561860] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23002.567083] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23002.572042] [23002.574139] [23002.576236] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23002.581544] ? lock_acquire+0xf0/0x2b0 [23002.585295] ? igrab+0x43/0x50 [23002.588344] ? lock_release+0xc5/0x260 [23002.592089] _raw_spin_lock+0x30/0x40 [23002.595745] ? igrab+0x1d/0x50 [23002.598795] igrab+0x1d/0x50 [23002.601687] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23002.606833] delayed_work+0x1d8/0x280 [ceph] [23002.611113] ? process_scheduled_works+0x22a/0x510 [23002.615904] process_scheduled_works+0x2a4/0x510 [23002.620518] worker_thread+0x188/0x340 [23002.624269] ? rescuer_thread+0x310/0x310 [23002.628272] kthread+0xf5/0x130 [23002.631410] ? kthread_complete_and_exit+0x20/0x20 [23002.636193] ret_from_fork+0x31/0x40 [23002.639764] ? kthread_complete_and_exit+0x20/0x20 [23002.644549] ret_from_fork_asm+0x11/0x20 [23002.648481] [23029.861762] watchdog: BUG: soft lockup - CPU#0 stuck for 10334s! [dbench:78683] [23029.869069] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23029.873761] watchdog: BUG: soft lockup - CPU#7 stuck for 19331s! [kworker/7:4:78808] [23029.924519] irq event stamp: 35905048 [23029.924520] hardirqs last enabled at (35905047): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23029.932250] Modules linked in: [23029.935906] hardirqs last disabled at (35905048): [] sysvec_apic_timer_interrupt+0xe/0x90 [23029.946065] ceph [23029.949114] softirqs last enabled at (35905032): [] __do_softirq+0x2be/0x371 [23029.958840] libceph [23029.960763] softirqs last disabled at (35905027): [] irq_exit_rcu+0xb0/0xd0 [23029.969448] dns_resolver [23029.971633] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23029.980156] fscache netfs [23029.982776] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23029.991721] veth nft_chain_nat [23029.994439] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23030.001912] xt_MASQUERADE [23030.005052] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23030.010529] nf_nat nf_conntrack [23030.013232] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23030.031969] nf_defrag_ipv6 nf_defrag_ipv4 [23030.035193] [23030.035194] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23030.040410] nft_compat nf_tables [23030.044501] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23030.045992] nfnetlink bridge [23030.053117] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23030.056443] stp llc [23030.063569] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23030.066533] xfs libcrc32c [23030.073658] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23030.075842] sunrpc intel_rapl_msr [23030.082965] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23030.085669] intel_rapl_common sb_edac [23030.092793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23030.096191] x86_pkg_temp_thermal coretemp [23030.104267] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23030.108014] kvm_intel [23030.113750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23030.117840] kvm irqbypass [23030.124964] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23030.127323] i2c_i801 iTCO_wdt [23030.134446] Call Trace: [23030.137162] iTCO_vendor_support i2c_smbus [23030.144287] [23030.147334] crct10dif_pclmul joydev [23030.149781] ? watchdog_timer_fn+0x1b4/0x220 [23030.153867] crc32_pclmul ipmi_ssif [23030.155881] ? __hrtimer_run_queues+0x1d3/0x360 [23030.159451] ghash_clmulni_intel mei_me [23030.163718] ? hrtimer_interrupt+0x104/0x240 [23030.167197] mei lpc_ich [23030.171723] ? __local_bh_enable+0x50/0x60 [23030.175553] sha512_ssse3 acpi_ipmi [23030.179820] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23030.182348] mfd_core wmi [23030.186441] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23030.189922] ipmi_si ipmi_devintf [23030.195157] [23030.197775] ipmi_msghandler ioatdma [23030.202733] [23030.206044] acpi_pad acpi_power_meter [23030.208160] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23030.211727] ixgbe igb [23030.213830] ? queued_spin_lock_slowpath+0x16/0x2e0 [23030.217569] mdio crc32c_intel [23030.222876] do_raw_spin_lock+0xb5/0xc0 [23030.225232] nvme ptp [23030.230105] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23030.233160] nvme_core i2c_algo_bit [23030.236995] try_get_cap_refs+0x159/0x880 [ceph] [23030.239264] pps_core dca [23030.244051] __ceph_get_caps+0x4a1/0x760 [ceph] [23030.247532] fuse [23030.252163] ? __file_remove_privs+0x89/0x110 [23030.254775] [23030.254776] irq event stamp: 106552282 [23030.259302] ? wake_bit_function+0x70/0x70 [23030.261224] hardirqs last enabled at (106552281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23030.265579] ? ceph_write_iter+0x260/0x770 [ceph] [23030.267066] hardirqs last disabled at (106552282): [] sysvec_apic_timer_interrupt+0xe/0x90 [23030.270811] ceph_write_iter+0x260/0x770 [ceph] [23030.274901] softirqs last enabled at (106552280): [] __do_softirq+0x2be/0x371 [23030.285164] ? lock_is_held_type+0xaa/0x120 [23030.289859] softirqs last disabled at (106552275): [] irq_exit_rcu+0xb0/0xd0 [23030.299674] vfs_write+0x324/0x4f0 [23030.304195] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23030.312976] ksys_pwrite64+0x65/0xa0 [23030.317159] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23030.325759] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23030.329164] Workqueue: events delayed_work [ceph] [23030.338542] do_syscall_64+0x38/0x80 [23030.342113] [23030.349583] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23030.354714] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [23030.359412] RIP: 0033:0x7f48f0eeb93b [23030.362982] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [23030.364472] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23030.369517] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [23030.375443] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23030.379015] [23030.379016] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [23030.397753] ORIG_RAX: 0000000000000012 [23030.416490] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [23030.421708] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23030.426925] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [23030.428415] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23030.435540] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23030.439371] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23030.446494] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23030.453620] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23030.460744] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23030.467869] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23030.474993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23030.482121] [23030.489239] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23030.526612] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23030.533735] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23030.540861] Call Trace: [23030.543304] [23030.545315] ? watchdog_timer_fn+0x1b4/0x220 [23030.549580] ? __hrtimer_run_queues+0x1d3/0x360 [23030.554106] ? hrtimer_interrupt+0x104/0x240 [23030.558367] ? __local_bh_enable+0x50/0x60 [23030.562460] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23030.567685] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23030.572642] [23030.574738] [23030.576837] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23030.582164] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [23030.587481] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [23030.592795] delayed_work+0x1d8/0x280 [ceph] [23030.597076] ? process_scheduled_works+0x22a/0x510 [23030.601865] process_scheduled_works+0x2a4/0x510 [23030.606481] worker_thread+0x188/0x340 [23030.610232] ? rescuer_thread+0x310/0x310 [23030.614243] kthread+0xf5/0x130 [23030.617381] ? kthread_complete_and_exit+0x20/0x20 [23030.622169] ret_from_fork+0x31/0x40 [23030.625743] ? kthread_complete_and_exit+0x20/0x20 [23030.630528] ret_from_fork_asm+0x11/0x20 [23030.634460] [23057.862241] watchdog: BUG: soft lockup - CPU#0 stuck for 10360s! [dbench:78683] [23057.869539] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23057.874241] watchdog: BUG: soft lockup - CPU#7 stuck for 19357s! [kworker/7:4:78808] [23057.924980] irq event stamp: 35963198 [23057.924981] hardirqs last enabled at (35963197): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23057.932710] Modules linked in: ceph [23057.936369] hardirqs last disabled at (35963198): [] sysvec_apic_timer_interrupt+0xe/0x90 [23057.946526] libceph [23057.950009] softirqs last enabled at (35963152): [] __do_softirq+0x2be/0x371 [23057.959734] dns_resolver [23057.961918] softirqs last disabled at (35963147): [] irq_exit_rcu+0xb0/0xd0 [23057.970601] fscache netfs [23057.973221] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23057.981730] veth nft_chain_nat [23057.984434] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23057.993378] xt_MASQUERADE nf_nat [23057.996516] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23058.003986] nf_conntrack nf_defrag_ipv6 [23058.007299] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23058.012776] nf_defrag_ipv4 nft_compat [23058.016691] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23058.035429] nf_tables nfnetlink [23058.039175] [23058.039176] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23058.044392] bridge [23058.047615] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23058.049105] stp llc [23058.056232] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23058.058328] xfs libcrc32c [23058.065452] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23058.067637] sunrpc intel_rapl_msr [23058.074759] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23058.077464] intel_rapl_common sb_edac [23058.084588] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23058.087987] x86_pkg_temp_thermal coretemp [23058.095109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23058.098853] kvm_intel kvm [23058.106931] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23058.111022] irqbypass i2c_i801 [23058.116759] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23058.119464] iTCO_wdt iTCO_vendor_support [23058.126586] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23058.129725] i2c_smbus [23058.136849] Call Trace: [23058.140854] crct10dif_pclmul joydev [23058.147979] [23058.150335] crc32_pclmul ipmi_ssif [23058.152780] ? watchdog_timer_fn+0x1b4/0x220 [23058.156351] ghash_clmulni_intel mei_me [23058.158362] ? __hrtimer_run_queues+0x1d3/0x360 [23058.161845] mei [23058.166112] ? hrtimer_interrupt+0x104/0x240 [23058.169938] lpc_ich sha512_ssse3 [23058.174462] ? __local_bh_enable+0x50/0x60 [23058.176300] acpi_ipmi mfd_core [23058.180568] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23058.183876] wmi ipmi_si [23058.187967] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23058.191103] ipmi_devintf ipmi_msghandler [23058.196321] [23058.198852] ioatdma acpi_pad [23058.203810] [23058.203811] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23058.207813] acpi_power_meter ixgbe [23058.209925] ? queued_spin_lock_slowpath+0x16/0x2e0 [23058.212883] igb mdio [23058.214983] do_raw_spin_lock+0xb5/0xc0 [23058.220284] crc32c_intel nvme [23058.223770] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23058.228641] ptp nvme_core [23058.230925] try_get_cap_refs+0x159/0x880 [ceph] [23058.234749] i2c_algo_bit pps_core [23058.237806] __ceph_get_caps+0x4a1/0x760 [ceph] [23058.242584] dca fuse [23058.245292] ? __file_remove_privs+0x89/0x110 [23058.249900] [23058.249917] irq event stamp: 106773226 [23058.253314] ? wake_bit_function+0x70/0x70 [23058.257838] hardirqs last enabled at (106773225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23058.260112] ? ceph_write_iter+0x260/0x770 [ceph] [23058.264460] hardirqs last disabled at (106773226): [] sysvec_apic_timer_interrupt+0xe/0x90 [23058.265951] ceph_write_iter+0x260/0x770 [ceph] [23058.269693] softirqs last enabled at (106773224): [] __do_softirq+0x2be/0x371 [23058.273788] ? lock_is_held_type+0xaa/0x120 [23058.284028] softirqs last disabled at (106773219): [] irq_exit_rcu+0xb0/0xd0 [23058.288728] vfs_write+0x324/0x4f0 [23058.298538] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23058.303066] ksys_pwrite64+0x65/0xa0 [23058.311833] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23058.316011] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23058.324607] Workqueue: events delayed_work [ceph] [23058.328007] do_syscall_64+0x38/0x80 [23058.337381] [23058.337382] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [23058.340954] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23058.348424] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [23058.353555] RIP: 0033:0x7f48f0eeb93b [23058.358253] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [23058.361823] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23058.363314] [23058.363315] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [23058.367925] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23058.372969] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [23058.391706] ORIG_RAX: 0000000000000012 [23058.391707] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23058.395278] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [23058.400495] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23058.419232] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23058.420722] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23058.427847] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23058.433064] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23058.440188] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23058.444018] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23058.451142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23058.458272] [23058.465392] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23058.524144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23058.531268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23058.538393] Call Trace: [23058.540836] [23058.542848] ? watchdog_timer_fn+0x1b4/0x220 [23058.547113] ? __hrtimer_run_queues+0x1d3/0x360 [23058.551652] ? hrtimer_interrupt+0x104/0x240 [23058.555925] ? __local_bh_enable+0x50/0x60 [23058.560019] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23058.565244] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23058.570199] [23058.572296] [23058.574395] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23058.579704] ? do_raw_spin_lock+0x42/0xc0 [23058.583712] igrab+0x1d/0x50 [23058.586590] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23058.591740] delayed_work+0x1d8/0x280 [ceph] [23058.596021] ? process_scheduled_works+0x22a/0x510 [23058.600810] process_scheduled_works+0x2a4/0x510 [23058.605425] worker_thread+0x188/0x340 [23058.609177] ? rescuer_thread+0x310/0x310 [23058.613186] kthread+0xf5/0x130 [23058.616326] ? kthread_complete_and_exit+0x20/0x20 [23058.621110] ret_from_fork+0x31/0x40 [23058.624688] ? kthread_complete_and_exit+0x20/0x20 [23058.629474] ret_from_fork_asm+0x11/0x20 [23058.633404] [23085.862723] watchdog: BUG: soft lockup - CPU#0 stuck for 10386s! [dbench:78683] [23085.870028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23085.874722] watchdog: BUG: soft lockup - CPU#7 stuck for 19383s! [kworker/7:4:78808] [23085.925477] irq event stamp: 36021500 [23085.933209] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [23085.936865] hardirqs last enabled at (36021499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23085.945036] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [23085.955194] hardirqs last disabled at (36021500): [] sysvec_apic_timer_interrupt+0xe/0x90 [23085.962233] nft_compat nf_tables nfnetlink bridge stp [23085.971956] softirqs last enabled at (36021404): [] __do_softirq+0x2be/0x371 [23085.977088] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [23085.985790] softirqs last disabled at (36021399): [] irq_exit_rcu+0xb0/0xd0 [23085.992314] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [23086.000828] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23086.006823] irqbypass i2c_i801 [23086.015769] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23086.015770] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23086.018904] iTCO_wdt iTCO_vendor_support [23086.026375] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23086.031854] i2c_smbus crct10dif_pclmul [23086.035856] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23086.054595] joydev crc32_pclmul [23086.058425] [23086.058426] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23086.063642] ipmi_ssif [23086.066868] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23086.068358] ghash_clmulni_intel mei_me [23086.075482] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23086.077839] mei lpc_ich [23086.084964] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23086.088794] sha512_ssse3 acpi_ipmi [23086.095918] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23086.098448] mfd_core [23086.105573] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23086.109056] wmi ipmi_si [23086.116181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23086.118452] ipmi_devintf [23086.126530] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23086.129060] ipmi_msghandler ioatdma [23086.134797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23086.137416] acpi_pad [23086.144539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23086.148120] acpi_power_meter [23086.155242] Call Trace: [23086.155244] [23086.157513] ixgbe igb [23086.164640] ? watchdog_timer_fn+0x1b4/0x220 [23086.167601] mdio crc32c_intel [23086.170046] ? __hrtimer_run_queues+0x1d3/0x360 [23086.172056] nvme ptp [23086.174416] ? hrtimer_interrupt+0x104/0x240 [23086.178679] nvme_core [23086.181729] ? __local_bh_enable+0x50/0x60 [23086.186254] i2c_algo_bit pps_core [23086.188526] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23086.192788] dca fuse [23086.195145] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23086.199236] [23086.199237] irq event stamp: 106994204 [23086.202633] [23086.207851] hardirqs last enabled at (106994203): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23086.210122] [23086.215078] hardirqs last disabled at (106994204): [] sysvec_apic_timer_interrupt+0xe/0x90 [23086.216570] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23086.220312] softirqs last enabled at (106994202): [] __do_softirq+0x2be/0x371 [23086.222416] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23086.232656] softirqs last disabled at (106994197): [] irq_exit_rcu+0xb0/0xd0 [23086.234753] do_raw_spin_lock+0xb5/0xc0 [23086.244563] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23086.249870] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23086.258638] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23086.263511] try_get_cap_refs+0x159/0x880 [ceph] [23086.272122] Workqueue: events delayed_work [ceph] [23086.275959] __ceph_get_caps+0x4a1/0x760 [ceph] [23086.285333] [23086.285334] RIP: 0010:lock_acquire+0xf0/0x2b0 [23086.290125] ? __file_remove_privs+0x89/0x110 [23086.297595] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23086.302206] ? wake_bit_function+0x70/0x70 [23086.306904] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23086.311429] ? ceph_write_iter+0x260/0x770 [ceph] [23086.312918] [23086.312918] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23086.317271] ceph_write_iter+0x260/0x770 [ceph] [23086.321619] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23086.340362] ? lock_is_held_type+0xaa/0x120 [23086.344449] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23086.349667] vfs_write+0x324/0x4f0 [23086.354364] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23086.355860] ksys_pwrite64+0x65/0xa0 [23086.362978] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23086.367503] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23086.374625] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23086.378805] do_syscall_64+0x38/0x80 [23086.385927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23086.389327] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23086.396449] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23086.400020] RIP: 0033:0x7f48f0eeb93b [23086.407144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23086.412275] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23086.420352] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23086.423922] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23086.429661] Call Trace: [23086.429664] [23086.434705] ORIG_RAX: 0000000000000012 [23086.441828] ? watchdog_timer_fn+0x1b4/0x220 [23086.445406] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23086.452534] ? __hrtimer_run_queues+0x1d3/0x360 [23086.471269] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23086.478410] ? hrtimer_interrupt+0x104/0x240 [23086.483619] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23086.486065] ? __local_bh_enable+0x50/0x60 [23086.488075] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23086.491909] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23086.496169] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23086.503295] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23086.507824] [23086.514944] [23086.559129] [23086.561233] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23086.566540] ? lock_acquire+0xf0/0x2b0 [23086.570291] ? igrab+0x43/0x50 [23086.573339] ? lock_release+0xc5/0x260 [23086.577084] _raw_spin_lock+0x30/0x40 [23086.580742] ? igrab+0x1d/0x50 [23086.583793] igrab+0x1d/0x50 [23086.586672] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23086.591821] delayed_work+0x1d8/0x280 [ceph] [23086.596103] ? process_scheduled_works+0x22a/0x510 [23086.600908] process_scheduled_works+0x2a4/0x510 [23086.605524] worker_thread+0x188/0x340 [23086.609274] ? rescuer_thread+0x310/0x310 [23086.613276] kthread+0xf5/0x130 [23086.616415] ? kthread_complete_and_exit+0x20/0x20 [23086.621199] ret_from_fork+0x31/0x40 [23086.624778] ? kthread_complete_and_exit+0x20/0x20 [23086.629562] ret_from_fork_asm+0x11/0x20 [23086.633493] [23113.863204] watchdog: BUG: soft lockup - CPU#0 stuck for 10412s! [dbench:78683] [23113.870505] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23113.875203] watchdog: BUG: soft lockup - CPU#7 stuck for 19409s! [kworker/7:4:78808] [23113.925948] irq event stamp: 36079714 [23113.933677] Modules linked in: ceph libceph dns_resolver fscache netfs [23113.937336] hardirqs last enabled at (36079713): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23113.943853] veth nft_chain_nat xt_MASQUERADE [23113.954026] hardirqs last disabled at (36079714): [] sysvec_apic_timer_interrupt+0xe/0x90 [23113.958378] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [23113.968103] softirqs last enabled at (36079626): [] __do_softirq+0x2be/0x371 [23113.968105] nft_compat nf_tables nfnetlink bridge stp [23113.982611] softirqs last disabled at (36079621): [] irq_exit_rcu+0xb0/0xd0 [23113.987740] llc xfs libcrc32c sunrpc [23113.996254] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23113.996255] intel_rapl_msr [23113.996256] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23113.999909] intel_rapl_common sb_edac [23114.008855] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23114.011661] x86_pkg_temp_thermal coretemp [23114.019132] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23114.022877] kvm_intel [23114.028355] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23114.032444] kvm irqbypass [23114.051182] [23114.051183] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23114.053539] i2c_i801 iTCO_wdt [23114.058758] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23114.061462] iTCO_vendor_support i2c_smbus [23114.062953] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23114.070076] crct10dif_pclmul joydev [23114.073126] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23114.080251] crc32_pclmul ipmi_ssif [23114.084341] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23114.091466] ghash_clmulni_intel [23114.095038] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23114.102160] mei_me mei [23114.105646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23114.112769] lpc_ich [23114.115993] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23114.124071] sha512_ssse3 acpi_ipmi [23114.126514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23114.132253] mfd_core [23114.134438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23114.141559] wmi ipmi_si [23114.145044] Call Trace: [23114.145046] [23114.152169] ipmi_devintf ipmi_msghandler ioatdma [23114.154440] ? watchdog_timer_fn+0x1b4/0x220 [23114.161562] acpi_pad acpi_power_meter [23114.164095] ? __hrtimer_run_queues+0x1d3/0x360 [23114.166537] ixgbe igb [23114.168551] ? hrtimer_interrupt+0x104/0x240 [23114.173245] mdio crc32c_intel [23114.177511] ? __local_bh_enable+0x50/0x60 [23114.181253] nvme ptp [23114.185782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23114.188136] nvme_core i2c_algo_bit [23114.192401] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23114.195452] pps_core dca [23114.199542] [23114.201813] fuse [23114.207030] [23114.210513] irq event stamp: 107215186 [23114.215472] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23114.218090] hardirqs last enabled at (107215185): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23114.220190] ? queued_spin_lock_slowpath+0x16/0x2e0 [23114.222110] hardirqs last disabled at (107215186): [] sysvec_apic_timer_interrupt+0xe/0x90 [23114.224210] do_raw_spin_lock+0xb5/0xc0 [23114.227950] softirqs last enabled at (107215184): [] __do_softirq+0x2be/0x371 [23114.233258] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23114.243499] softirqs last disabled at (107215179): [] irq_exit_rcu+0xb0/0xd0 [23114.248372] try_get_cap_refs+0x159/0x880 [ceph] [23114.258181] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23114.262017] __ceph_get_caps+0x4a1/0x760 [ceph] [23114.270782] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23114.270784] Workqueue: events delayed_work [ceph] [23114.275571] ? __file_remove_privs+0x89/0x110 [23114.284163] [23114.284164] RIP: 0010:lock_acquire+0xf0/0x2b0 [23114.288776] ? wake_bit_function+0x70/0x70 [23114.298153] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23114.302679] ? ceph_write_iter+0x260/0x770 [ceph] [23114.310146] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23114.314846] ceph_write_iter+0x260/0x770 [ceph] [23114.319197] [23114.319198] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23114.320690] ? lock_is_held_type+0xaa/0x120 [23114.325054] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23114.329147] vfs_write+0x324/0x4f0 [23114.347885] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23114.352604] ksys_pwrite64+0x65/0xa0 [23114.357816] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23114.357817] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23114.362338] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23114.363831] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23114.370955] do_syscall_64+0x38/0x80 [23114.375131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23114.382257] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23114.385653] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23114.392779] RIP: 0033:0x7f48f0eeb93b [23114.396348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23114.403472] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23114.410606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23114.415735] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23114.423813] Call Trace: [23114.423814] [23114.427384] ORIG_RAX: 0000000000000012 [23114.433123] ? watchdog_timer_fn+0x1b4/0x220 [23114.438165] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23114.445291] ? __hrtimer_run_queues+0x1d3/0x360 [23114.448860] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23114.455988] ? hrtimer_interrupt+0x104/0x240 [23114.474721] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23114.481845] ? __local_bh_enable+0x50/0x60 [23114.487064] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23114.489511] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23114.491519] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23114.495350] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23114.499619] [23114.506738] [23114.562605] [23114.564702] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23114.570008] ? lock_acquire+0xf0/0x2b0 [23114.573759] ? igrab+0x43/0x50 [23114.576809] ? lock_release+0xc5/0x260 [23114.580555] _raw_spin_lock+0x30/0x40 [23114.584219] ? igrab+0x1d/0x50 [23114.587270] igrab+0x1d/0x50 [23114.590148] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23114.595298] delayed_work+0x1d8/0x280 [ceph] [23114.599581] ? process_scheduled_works+0x22a/0x510 [23114.604386] process_scheduled_works+0x2a4/0x510 [23114.609002] worker_thread+0x188/0x340 [23114.612751] ? rescuer_thread+0x310/0x310 [23114.616753] kthread+0xf5/0x130 [23114.619906] ? kthread_complete_and_exit+0x20/0x20 [23114.624695] ret_from_fork+0x31/0x40 [23114.628273] ? kthread_complete_and_exit+0x20/0x20 [23114.633056] ret_from_fork_asm+0x11/0x20 [23114.636981] [23141.863686] watchdog: BUG: soft lockup - CPU#0 stuck for 10438s! [dbench:78683] [23141.870995] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23141.875685] watchdog: BUG: soft lockup - CPU#7 stuck for 19435s! [kworker/7:4:78808] [23141.926442] irq event stamp: 36137914 [23141.926444] hardirqs last enabled at (36137913): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23141.934175] Modules linked in: [23141.937830] hardirqs last disabled at (36137914): [] sysvec_apic_timer_interrupt+0xe/0x90 [23141.947990] ceph [23141.951041] softirqs last enabled at (36137898): [] __do_softirq+0x2be/0x371 [23141.960765] libceph [23141.962689] softirqs last disabled at (36137893): [] irq_exit_rcu+0xb0/0xd0 [23141.971373] dns_resolver [23141.973558] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23141.982080] fscache netfs [23141.984693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23141.993637] veth nft_chain_nat [23141.996343] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23142.003812] xt_MASQUERADE [23142.006950] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23142.012427] nf_nat nf_conntrack [23142.015130] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23142.033869] nf_defrag_ipv6 [23142.037092] [23142.042311] nf_defrag_ipv4 [23142.045101] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23142.046590] nft_compat [23142.049381] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23142.056507] nf_tables [23142.058949] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23142.066079] nfnetlink [23142.068440] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23142.075566] bridge [23142.077922] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23142.085047] stp llc [23142.087144] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23142.094268] xfs libcrc32c [23142.096453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23142.104530] sunrpc [23142.107233] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23142.112971] intel_rapl_msr [23142.115084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23142.122208] intel_rapl_common [23142.124999] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23142.132123] sb_edac [23142.135174] Call Trace: [23142.135177] [23142.142299] x86_pkg_temp_thermal [23142.144486] ? watchdog_timer_fn+0x1b4/0x220 [23142.146927] coretemp kvm_intel [23142.148939] ? __hrtimer_run_queues+0x1d3/0x360 [23142.152250] kvm irqbypass [23142.156517] ? hrtimer_interrupt+0x104/0x240 [23142.159650] i2c_i801 iTCO_wdt [23142.164173] ? __local_bh_enable+0x50/0x60 [23142.166879] iTCO_vendor_support [23142.171145] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23142.174193] i2c_smbus crct10dif_pclmul [23142.178285] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23142.181508] joydev crc32_pclmul [23142.186725] [23142.190557] ipmi_ssif [23142.195514] [23142.195515] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23142.198739] ghash_clmulni_intel [23142.200838] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23142.203192] mei_me mei [23142.205291] do_raw_spin_lock+0xb5/0xc0 [23142.210594] lpc_ich [23142.213819] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23142.218688] sha512_ssse3 [23142.221136] try_get_cap_refs+0x159/0x880 [ceph] [23142.224964] acpi_ipmi mfd_core [23142.227154] __ceph_get_caps+0x4a1/0x760 [ceph] [23142.231932] wmi [23142.234552] ? __file_remove_privs+0x89/0x110 [23142.239160] ipmi_si ipmi_devintf [23142.242298] ? wake_bit_function+0x70/0x70 [23142.246821] ipmi_msghandler [23142.248661] ? ceph_write_iter+0x260/0x770 [ceph] [23142.253010] ioatdma acpi_pad [23142.256320] ceph_write_iter+0x260/0x770 [ceph] [23142.260410] acpi_power_meter [23142.263292] ? lock_is_held_type+0xaa/0x120 [23142.267986] ixgbe [23142.270952] vfs_write+0x324/0x4f0 [23142.275474] igb [23142.278443] ksys_pwrite64+0x65/0xa0 [23142.282616] mdio [23142.284627] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23142.288025] crc32c_intel nvme [23142.289863] do_syscall_64+0x38/0x80 [23142.293432] ptp nvme_core [23142.295356] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23142.300486] i2c_algo_bit [23142.303538] RIP: 0033:0x7f48f0eeb93b [23142.307108] pps_core [23142.309812] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23142.314855] dca fuse [23142.317472] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23142.321043] [23142.321044] irq event stamp: 107436348 [23142.323313] ORIG_RAX: 0000000000000012 [23142.342053] hardirqs last enabled at (107436347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23142.344324] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23142.349541] hardirqs last disabled at (107436348): [] sysvec_apic_timer_interrupt+0xe/0x90 [23142.351031] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23142.354775] softirqs last enabled at (107436346): [] __do_softirq+0x2be/0x371 [23142.358607] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23142.368851] softirqs last disabled at (107436341): [] irq_exit_rcu+0xb0/0xd0 [23142.375974] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23142.385784] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23142.392910] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23142.401680] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23142.408810] [23142.417401] Workqueue: events delayed_work [ceph] [23142.455382] RIP: 0010:__list_del_entry_valid_or_report+0x3f/0xd0 [23142.461385] Code: 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 <48> 39 f2 75 73 b8 01 00 00 00 c3 48 89 fe 48 c7 c7 98 99 1d 82 e8 [23142.480124] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [23142.485342] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [23142.492465] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [23142.499590] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [23142.506713] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23142.513839] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23142.520963] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23142.529041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23142.534777] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23142.541901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23142.549025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23142.555063] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [23142.556149] Call Trace: [23142.556151] [23142.562237] rcu: 0-....: (10953447 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=5468296 [23142.564678] ? watchdog_timer_fn+0x1b4/0x220 [23142.566689] rcu: 7-....: (10880174 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/10878898 fqs=5468297 [23142.577108] ? __hrtimer_run_queues+0x1d3/0x360 [23142.581379] rcu: (detected by 4, t=11205754 jiffies, g=1309609, q=365224 ncpus=8) [23142.591973] ? hrtimer_interrupt+0x104/0x240 [23142.596493] Sending NMI from CPU 4 to CPUs 0: [23142.604050] ? __local_bh_enable+0x50/0x60 [23142.608318] NMI backtrace for cpu 0 [23142.608319] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23142.608321] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23142.608322] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23142.608325] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23142.608327] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23142.608329] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23142.608330] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23142.608332] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23142.608333] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23142.608334] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23142.608335] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23142.608337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23142.608338] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23142.608339] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23142.608341] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23142.608342] Call Trace: [23142.608343] [23142.608344] ? nmi_cpu_backtrace+0xad/0x120 [23142.608349] ? nmi_cpu_backtrace_handler+0x11/0x20 [23142.608368] ? nmi_handle+0xd4/0x260 [23142.608374] ? default_do_nmi+0x49/0xf0 [23142.608376] ? exc_nmi+0x11e/0x150 [23142.608379] ? end_repeat_nmi+0x16/0x67 [23142.608385] ? queued_spin_lock_slowpath+0x18/0x2e0 [23142.608388] ? queued_spin_lock_slowpath+0x18/0x2e0 [23142.608392] ? queued_spin_lock_slowpath+0x18/0x2e0 [23142.608395] [23142.608395] [23142.608397] do_raw_spin_lock+0xb5/0xc0 [23142.608400] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23142.608418] try_get_cap_refs+0x159/0x880 [ceph] [23142.608438] __ceph_get_caps+0x4a1/0x760 [ceph] [23142.608457] ? __file_remove_privs+0x89/0x110 [23142.608459] ? wake_bit_function+0x70/0x70 [23142.608463] ? ceph_write_iter+0x260/0x770 [ceph] [23142.608478] ceph_write_iter+0x260/0x770 [ceph] [23142.608495] ? lock_is_held_type+0xaa/0x120 [23142.608499] vfs_write+0x324/0x4f0 [23142.608505] ksys_pwrite64+0x65/0xa0 [23142.608507] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23142.608511] do_syscall_64+0x38/0x80 [23142.608514] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23142.608516] RIP: 0033:0x7f48f0eeb93b [23142.608518] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23142.608519] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23142.608521] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23142.608522] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23142.608523] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23142.608524] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23142.608525] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23142.608531] [23142.609316] Sending NMI from CPU 4 to CPUs 7: [23142.612671] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23142.616758] NMI backtrace for cpu 7 [23142.616759] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23142.616761] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23142.616762] Workqueue: events delayed_work [ceph] [23142.616781] RIP: 0010:console_unlock+0x30/0x120 [23142.616784] Code: 44 00 00 41 54 49 c7 c4 e0 bc 55 82 55 31 ed 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 8b 05 24 4d 6b 02 <85> c0 40 0f 95 c5 eb 28 84 db 0f 84 8d 00 00 00 48 8b 74 24 08 48 [23142.616786] RSP: 0018:ffffc90000264cf8 EFLAGS: 00000046 [23142.616787] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000de2c1856 [23142.616789] RDX: c31e29845e0f9ad2 RSI: 0000000000000000 RDI: 00000000000000c2 [23142.616790] RBP: 0000000000000000 R08: 0000000000000078 R09: 0000000000000003 [23142.616791] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8255bce0 [23142.616792] R13: 0000000000000000 R14: ffffffff82123e9c R15: ffffc90000264d80 [23142.616793] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23142.616795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23142.616796] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23142.616798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23142.616799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23142.616800] Call Trace: [23142.616801] [23142.616802] ? nmi_cpu_backtrace+0xad/0x120 [23142.616805] ? nmi_cpu_backtrace_handler+0x11/0x20 [23142.616807] ? nmi_handle+0xd4/0x260 [23142.616812] ? default_do_nmi+0x49/0xf0 [23142.616815] ? exc_nmi+0x11e/0x150 [23142.616817] ? end_repeat_nmi+0x16/0x67 [23142.616824] ? console_unlock+0x30/0x120 [23142.616827] ? console_unlock+0x30/0x120 [23142.616830] ? console_unlock+0x30/0x120 [23142.616833] [23142.616833] [23142.616835] vprintk_emit+0x246/0x340 [23142.616838] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23142.616841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23142.616844] _printk+0x5c/0x80 [23142.616848] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23142.616851] show_trace_log_lvl+0x1ac/0x3f0 [23142.616859] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [23142.616878] ? touch_softlockup_watchdog+0x20/0x20 [23142.616880] watchdog_timer_fn+0x1b4/0x220 [23142.616883] __hrtimer_run_queues+0x1d3/0x360 [23142.616888] hrtimer_interrupt+0x104/0x240 [23142.616890] ? __local_bh_enable+0x50/0x60 [23142.616895] __sysvec_apic_timer_interrupt+0x61/0x160 [23142.616898] sysvec_apic_timer_interrupt+0x6d/0x90 [23142.616901] [23142.616901] [23142.616903] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23142.616905] RIP: 0010:__list_del_entry_valid_or_report+0x3f/0xd0 [23142.616908] Code: 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 <48> 39 f2 75 73 b8 01 00 00 00 c3 48 89 fe 48 c7 c7 98 99 1d 82 e8 [23142.616910] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [23142.616911] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [23142.616912] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [23142.616913] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [23142.616915] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23142.616916] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23142.616922] ceph_check_delayed_caps+0x8f/0x270 [ceph] [23142.616940] delayed_work+0x1d8/0x280 [ceph] [23142.616960] ? process_scheduled_works+0x22a/0x510 [23142.616963] process_scheduled_works+0x2a4/0x510 [23142.616970] worker_thread+0x188/0x340 [23142.616974] ? rescuer_thread+0x310/0x310 [23142.616977] kthread+0xf5/0x130 [23142.616980] ? kthread_complete_and_exit+0x20/0x20 [23142.616984] ret_from_fork+0x31/0x40 [23142.616987] ? kthread_complete_and_exit+0x20/0x20 [23142.616989] ret_from_fork_asm+0x11/0x20 [23142.616998] [23143.276719] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23143.281676] [23143.283773] [23143.285870] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23143.291180] ? __list_del_entry_valid_or_report+0x3f/0xd0 [23143.296575] ceph_check_delayed_caps+0x8f/0x270 [ceph] [23143.301723] delayed_work+0x1d8/0x280 [ceph] [23143.306005] ? process_scheduled_works+0x22a/0x510 [23143.310788] process_scheduled_works+0x2a4/0x510 [23143.315403] worker_thread+0x188/0x340 [23143.319151] ? rescuer_thread+0x310/0x310 [23143.323154] kthread+0xf5/0x130 [23143.326293] ? kthread_complete_and_exit+0x20/0x20 [23143.331078] ret_from_fork+0x31/0x40 [23143.334672] ? kthread_complete_and_exit+0x20/0x20 [23143.339457] ret_from_fork_asm+0x11/0x20 [23143.343389] [23169.864164] watchdog: BUG: soft lockup - CPU#0 stuck for 10464s! [dbench:78683] [23169.871464] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23169.876164] watchdog: BUG: soft lockup - CPU#7 stuck for 19461s! [kworker/7:4:78808] [23169.926904] irq event stamp: 36196236 [23169.926905] hardirqs last enabled at (36196235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23169.934636] Modules linked in: ceph [23169.938292] hardirqs last disabled at (36196236): [] sysvec_apic_timer_interrupt+0xe/0x90 [23169.948451] libceph [23169.951934] softirqs last enabled at (36196214): [] __do_softirq+0x2be/0x371 [23169.961658] dns_resolver [23169.963849] softirqs last disabled at (36196209): [] irq_exit_rcu+0xb0/0xd0 [23169.972535] fscache [23169.975171] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23169.983682] netfs veth [23169.985864] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23169.994810] nft_chain_nat [23169.997254] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23170.004724] xt_MASQUERADE nf_nat [23170.007429] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23170.012905] nf_conntrack nf_defrag_ipv6 [23170.016215] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23170.034954] nf_defrag_ipv4 nft_compat [23170.038872] [23170.044088] nf_tables nfnetlink [23170.047839] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23170.049333] bridge stp [23170.052560] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23170.059679] llc xfs [23170.062124] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23170.069248] libcrc32c sunrpc [23170.071432] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23170.078561] intel_rapl_msr [23170.081529] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23170.088653] intel_rapl_common [23170.091445] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23170.098568] sb_edac x86_pkg_temp_thermal [23170.101618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23170.109697] coretemp kvm_intel [23170.113700] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23170.119438] kvm irqbypass [23170.122576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23170.129699] i2c_i801 iTCO_wdt [23170.132404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23170.139529] iTCO_vendor_support [23170.142580] Call Trace: [23170.142581] [23170.149702] i2c_smbus crct10dif_pclmul [23170.152927] ? watchdog_timer_fn+0x1b4/0x220 [23170.155370] joydev [23170.157382] ? __hrtimer_run_queues+0x1d3/0x360 [23170.161211] crc32_pclmul ipmi_ssif [23170.165481] ? hrtimer_interrupt+0x104/0x240 [23170.167573] ghash_clmulni_intel [23170.172099] ? __local_bh_enable+0x50/0x60 [23170.175581] mei_me mei [23170.179848] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23170.183070] lpc_ich [23170.187161] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23170.189605] sha512_ssse3 acpi_ipmi [23170.194822] [23170.197023] mfd_core wmi [23170.201981] [23170.201982] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23170.205466] ipmi_si ipmi_devintf [23170.207566] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23170.210180] ipmi_msghandler ioatdma [23170.212280] do_raw_spin_lock+0xb5/0xc0 [23170.217581] acpi_pad acpi_power_meter [23170.220893] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23170.225764] ixgbe igb [23170.229335] try_get_cap_refs+0x159/0x880 [ceph] [23170.233165] mdio crc32c_intel [23170.236913] __ceph_get_caps+0x4a1/0x760 [ceph] [23170.241694] nvme ptp [23170.244054] ? __file_remove_privs+0x89/0x110 [23170.248661] nvme_core i2c_algo_bit [23170.251711] ? wake_bit_function+0x70/0x70 [23170.256235] pps_core dca [23170.258509] ? ceph_write_iter+0x260/0x770 [ceph] [23170.262858] fuse [23170.266341] ceph_write_iter+0x260/0x770 [ceph] [23170.270433] irq event stamp: 107651926 [23170.273053] ? lock_is_held_type+0xaa/0x120 [23170.277747] hardirqs last enabled at (107651925): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23170.279672] vfs_write+0x324/0x4f0 [23170.284195] hardirqs last disabled at (107651926): [] sysvec_apic_timer_interrupt+0xe/0x90 [23170.287942] ksys_pwrite64+0x65/0xa0 [23170.292115] softirqs last enabled at (107651924): [] __do_softirq+0x2be/0x371 [23170.302360] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23170.305758] softirqs last disabled at (107651919): [] irq_exit_rcu+0xb0/0xd0 [23170.315569] do_syscall_64+0x38/0x80 [23170.319139] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23170.327912] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23170.333042] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23170.341638] RIP: 0033:0x7f48f0eeb93b [23170.345208] Workqueue: events delayed_work [ceph] [23170.354586] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23170.359631] [23170.367100] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23170.370671] RIP: 0010:lock_acquire+0xf0/0x2b0 [23170.375371] ORIG_RAX: 0000000000000012 [23170.375372] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23170.394108] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23170.395598] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23170.400816] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23170.405166] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23170.408997] [23170.408997] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23170.416121] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23170.434859] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23170.441984] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23170.447199] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23170.454329] [23170.455814] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23170.500745] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23170.507868] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23170.515947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23170.521683] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23170.528806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23170.535932] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23170.543058] Call Trace: [23170.545501] [23170.547512] ? watchdog_timer_fn+0x1b4/0x220 [23170.551777] ? __hrtimer_run_queues+0x1d3/0x360 [23170.556303] ? hrtimer_interrupt+0x104/0x240 [23170.560572] ? __local_bh_enable+0x50/0x60 [23170.564665] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23170.569888] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23170.574861] [23170.576961] [23170.579060] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23170.584367] ? lock_acquire+0xf0/0x2b0 [23170.588115] ? igrab+0x43/0x50 [23170.591165] ? lock_release+0xc5/0x260 [23170.594912] _raw_spin_lock+0x30/0x40 [23170.598575] ? igrab+0x1d/0x50 [23170.601625] igrab+0x1d/0x50 [23170.604507] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23170.609653] delayed_work+0x1d8/0x280 [ceph] [23170.613935] ? process_scheduled_works+0x22a/0x510 [23170.618727] process_scheduled_works+0x2a4/0x510 [23170.623341] worker_thread+0x188/0x340 [23170.627091] ? rescuer_thread+0x310/0x310 [23170.631103] kthread+0xf5/0x130 [23170.634242] ? kthread_complete_and_exit+0x20/0x20 [23170.639034] ret_from_fork+0x31/0x40 [23170.642603] ? kthread_complete_and_exit+0x20/0x20 [23170.647388] ret_from_fork_asm+0x11/0x20 [23170.651311] [23197.864640] watchdog: BUG: soft lockup - CPU#0 stuck for 10490s! [dbench:78683] [23197.871943] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23197.876639] watchdog: BUG: soft lockup - CPU#7 stuck for 19487s! [kworker/7:4:78808] [23197.927391] irq event stamp: 36254422 [23197.935123] Modules linked in: ceph libceph dns_resolver fscache netfs [23197.938779] hardirqs last enabled at (36254421): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23197.945312] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23197.955472] hardirqs last disabled at (36254422): [] sysvec_apic_timer_interrupt+0xe/0x90 [23197.962858] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [23197.972580] softirqs last enabled at (36254334): [] __do_softirq+0x2be/0x371 [23197.979357] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [23197.988044] softirqs last disabled at (36254329): [] irq_exit_rcu+0xb0/0xd0 [23197.988047] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [23198.003428] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23198.003430] i2c_i801 [23198.003431] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23198.009596] iTCO_wdt iTCO_vendor_support [23198.018542] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23198.020811] i2c_smbus crct10dif_pclmul [23198.028283] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23198.032286] joydev crc32_pclmul [23198.037765] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23198.041594] ipmi_ssif ghash_clmulni_intel [23198.060333] [23198.060334] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23198.063557] mei_me [23198.068773] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23198.072863] mei lpc_ich [23198.074354] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23198.081478] sha512_ssse3 [23198.083576] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23198.090701] acpi_ipmi [23198.093233] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23198.100355] mfd_core wmi [23198.102972] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23198.110096] ipmi_si [23198.112455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23198.119578] ipmi_devintf ipmi_msghandler [23198.122195] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23198.130274] ioatdma acpi_pad [23198.132458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23198.138196] acpi_power_meter ixgbe [23198.142199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23198.149327] igb [23198.152288] Call Trace: [23198.152291] [23198.159413] mdio crc32c_intel [23198.162899] ? watchdog_timer_fn+0x1b4/0x220 [23198.170035] nvme ptp [23198.171877] ? __hrtimer_run_queues+0x1d3/0x360 [23198.174320] nvme_core i2c_algo_bit [23198.176333] ? hrtimer_interrupt+0x104/0x240 [23198.179380] pps_core dca [23198.183645] ? __local_bh_enable+0x50/0x60 [23198.185914] fuse [23198.190442] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23198.193923] irq event stamp: 107872826 [23198.198187] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23198.200803] hardirqs last enabled at (107872825): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23198.204894] [23198.206818] hardirqs last disabled at (107872826): [] sysvec_apic_timer_interrupt+0xe/0x90 [23198.212038] [23198.215780] softirqs last enabled at (107872824): [] __do_softirq+0x2be/0x371 [23198.220739] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23198.230981] softirqs last disabled at (107872819): [] irq_exit_rcu+0xb0/0xd0 [23198.233084] ? queued_spin_lock_slowpath+0x18/0x2e0 [23198.242891] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23198.244991] do_raw_spin_lock+0xb5/0xc0 [23198.253760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23198.259065] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23198.267662] Workqueue: events delayed_work [ceph] [23198.272535] try_get_cap_refs+0x159/0x880 [ceph] [23198.281909] [23198.285743] __ceph_get_caps+0x4a1/0x760 [ceph] [23198.293210] RIP: 0010:_raw_spin_unlock+0xf/0x40 [23198.297998] ? __file_remove_privs+0x89/0x110 [23198.302692] Code: 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 <48> 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff [23198.307318] ? wake_bit_function+0x70/0x70 [23198.308809] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [23198.313338] ? ceph_write_iter+0x260/0x770 [ceph] [23198.317858] [23198.317859] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [23198.322211] ceph_write_iter+0x260/0x770 [ceph] [23198.340948] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34500 [23198.340949] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [23198.345046] ? lock_is_held_type+0xaa/0x120 [23198.350256] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23198.354955] vfs_write+0x324/0x4f0 [23198.356442] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23198.363573] ksys_pwrite64+0x65/0xa0 [23198.368092] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23198.375215] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23198.382338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23198.386518] do_syscall_64+0x38/0x80 [23198.393640] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23198.397042] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23198.404163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23198.407736] RIP: 0033:0x7f48f0eeb93b [23198.415810] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23198.420942] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23198.426678] Call Trace: [23198.430250] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23198.437376] [23198.442419] ORIG_RAX: 0000000000000012 [23198.449543] ? watchdog_timer_fn+0x1b4/0x220 [23198.453114] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23198.460237] ? __hrtimer_run_queues+0x1d3/0x360 [23198.478974] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23198.481422] ? hrtimer_interrupt+0x104/0x240 [23198.486635] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23198.488647] ? __local_bh_enable+0x50/0x60 [23198.492478] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23198.496744] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23198.503867] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23198.508391] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23198.515518] [23198.519779] [23198.559697] [23198.561797] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23198.567104] ? igrab+0x43/0x50 [23198.570159] ? _raw_spin_unlock+0xf/0x40 [23198.574077] igrab+0x43/0x50 [23198.576956] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23198.582106] delayed_work+0x1d8/0x280 [ceph] [23198.586387] ? process_scheduled_works+0x22a/0x510 [23198.591175] process_scheduled_works+0x2a4/0x510 [23198.595790] worker_thread+0x188/0x340 [23198.599541] ? rescuer_thread+0x310/0x310 [23198.603553] kthread+0xf5/0x130 [23198.606700] ? kthread_complete_and_exit+0x20/0x20 [23198.611483] ret_from_fork+0x31/0x40 [23198.615062] ? kthread_complete_and_exit+0x20/0x20 [23198.619845] ret_from_fork_asm+0x11/0x20 [23198.623770] [23225.865115] watchdog: BUG: soft lockup - CPU#0 stuck for 10516s! [dbench:78683] [23225.872413] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23225.877115] watchdog: BUG: soft lockup - CPU#7 stuck for 19513s! [kworker/7:4:78808] [23225.927855] irq event stamp: 36312520 [23225.927856] hardirqs last enabled at (36312519): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23225.935584] Modules linked in: ceph [23225.939241] hardirqs last disabled at (36312520): [] sysvec_apic_timer_interrupt+0xe/0x90 [23225.949400] libceph [23225.952884] softirqs last enabled at (36312442): [] __do_softirq+0x2be/0x371 [23225.962608] dns_resolver fscache [23225.964793] softirqs last disabled at (36312437): [] irq_exit_rcu+0xb0/0xd0 [23225.973477] netfs veth [23225.976786] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23225.985314] nft_chain_nat xt_MASQUERADE [23225.987758] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23225.996704] nf_nat nf_conntrack [23226.000620] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23226.008091] nf_defrag_ipv6 nf_defrag_ipv4 [23226.011314] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23226.016807] nft_compat nf_tables [23226.020899] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23226.039637] nfnetlink bridge [23226.042949] [23226.042950] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23226.048167] stp llc [23226.051129] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23226.052620] xfs libcrc32c [23226.059744] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23226.061930] sunrpc [23226.069052] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23226.071757] intel_rapl_msr intel_rapl_common [23226.078882] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23226.080978] sb_edac x86_pkg_temp_thermal [23226.088103] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23226.092454] coretemp kvm_intel [23226.099579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23226.103583] kvm [23226.111659] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23226.114797] irqbypass i2c_i801 [23226.120535] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23226.122371] iTCO_wdt iTCO_vendor_support [23226.129513] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23226.132650] i2c_smbus crct10dif_pclmul [23226.139774] Call Trace: [23226.139776] [23226.143779] joydev [23226.150921] ? watchdog_timer_fn+0x1b4/0x220 [23226.154751] crc32_pclmul ipmi_ssif [23226.157197] ? __hrtimer_run_queues+0x1d3/0x360 [23226.159206] ghash_clmulni_intel mei_me [23226.161307] ? hrtimer_interrupt+0x104/0x240 [23226.165566] mei lpc_ich [23226.169052] ? __local_bh_enable+0x50/0x60 [23226.173576] sha512_ssse3 acpi_ipmi [23226.177407] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23226.181671] mfd_core wmi [23226.184202] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23226.188292] ipmi_si ipmi_devintf [23226.191777] [23226.196993] ipmi_msghandler ioatdma [23226.199611] [23226.199613] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23226.204568] acpi_pad acpi_power_meter [23226.207883] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23226.209975] ixgbe igb [23226.213549] do_raw_spin_lock+0xb5/0xc0 [23226.215643] mdio crc32c_intel [23226.220949] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23226.224691] nvme ptp [23226.229566] try_get_cap_refs+0x159/0x880 [ceph] [23226.231921] nvme_core i2c_algo_bit [23226.235755] __ceph_get_caps+0x4a1/0x760 [ceph] [23226.238804] pps_core dca [23226.243591] ? __file_remove_privs+0x89/0x110 [23226.245857] fuse [23226.250468] ? wake_bit_function+0x70/0x70 [23226.253952] irq event stamp: 108093928 [23226.258477] ? ceph_write_iter+0x260/0x770 [ceph] [23226.261092] hardirqs last enabled at (108093927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23226.265443] ceph_write_iter+0x260/0x770 [ceph] [23226.267367] hardirqs last disabled at (108093928): [] sysvec_apic_timer_interrupt+0xe/0x90 [23226.271463] ? lock_is_held_type+0xaa/0x120 [23226.275203] softirqs last enabled at (108093926): [] __do_softirq+0x2be/0x371 [23226.279901] vfs_write+0x324/0x4f0 [23226.290145] softirqs last disabled at (108093921): [] irq_exit_rcu+0xb0/0xd0 [23226.294672] ksys_pwrite64+0x65/0xa0 [23226.304479] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23226.308655] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23226.317427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23226.320826] do_syscall_64+0x38/0x80 [23226.329422] Workqueue: events delayed_work [ceph] [23226.332993] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23226.342371] [23226.342372] RIP: 0010:lock_acquire+0xf0/0x2b0 [23226.347517] RIP: 0033:0x7f48f0eeb93b [23226.354988] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23226.358560] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23226.363258] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23226.368301] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23226.369796] [23226.374144] ORIG_RAX: 0000000000000012 [23226.377714] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23226.396452] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23226.415189] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23226.420407] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23226.425625] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23226.427113] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23226.430944] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23226.438069] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23226.445193] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23226.452319] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23226.459442] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23226.466571] [23226.473689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23226.518186] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23226.525311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23226.532435] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23226.539558] Call Trace: [23226.542004] [23226.544014] ? watchdog_timer_fn+0x1b4/0x220 [23226.548277] ? __hrtimer_run_queues+0x1d3/0x360 [23226.552814] ? hrtimer_interrupt+0x104/0x240 [23226.557082] ? __local_bh_enable+0x50/0x60 [23226.561176] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23226.566402] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23226.571365] [23226.573463] [23226.575562] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23226.580869] ? lock_acquire+0xf0/0x2b0 [23226.584619] ? igrab+0x43/0x50 [23226.587669] ? lock_release+0xc5/0x260 [23226.591413] _raw_spin_lock+0x30/0x40 [23226.595071] ? igrab+0x1d/0x50 [23226.598121] igrab+0x1d/0x50 [23226.600998] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23226.606150] delayed_work+0x1d8/0x280 [ceph] [23226.610430] ? process_scheduled_works+0x22a/0x510 [23226.615219] process_scheduled_works+0x2a4/0x510 [23226.619836] worker_thread+0x188/0x340 [23226.623585] ? rescuer_thread+0x310/0x310 [23226.627588] kthread+0xf5/0x130 [23226.630726] ? kthread_complete_and_exit+0x20/0x20 [23226.635522] ret_from_fork+0x31/0x40 [23226.639098] ? kthread_complete_and_exit+0x20/0x20 [23226.643882] ret_from_fork_asm+0x11/0x20 [23226.647807] [23253.865591] watchdog: BUG: soft lockup - CPU#0 stuck for 10542s! [dbench:78683] [23253.872902] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23253.877591] watchdog: BUG: soft lockup - CPU#7 stuck for 19539s! [kworker/7:4:78808] [23253.928349] irq event stamp: 36370704 [23253.936080] Modules linked in: ceph libceph dns_resolver fscache netfs veth [23253.939737] hardirqs last enabled at (36370703): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23253.946696] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23253.956855] hardirqs last disabled at (36370704): [] sysvec_apic_timer_interrupt+0xe/0x90 [23253.963806] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [23253.973529] softirqs last enabled at (36370626): [] __do_softirq+0x2be/0x371 [23253.980306] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [23253.988991] softirqs last disabled at (36370621): [] irq_exit_rcu+0xb0/0xd0 [23253.995856] x86_pkg_temp_thermal coretemp kvm_intel kvm [23254.004366] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23254.004368] irqbypass [23254.004369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23254.009670] i2c_i801 [23254.018615] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23254.020987] iTCO_wdt iTCO_vendor_support [23254.028461] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23254.030730] i2c_smbus crct10dif_pclmul [23254.036208] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23254.040213] joydev crc32_pclmul [23254.058950] [23254.062780] ipmi_ssif ghash_clmulni_intel [23254.067998] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23254.071222] mei_me [23254.072712] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23254.076803] mei lpc_ich [23254.083928] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23254.086025] sha512_ssse3 acpi_ipmi [23254.093149] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23254.095679] mfd_core [23254.102804] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23254.106288] wmi ipmi_si [23254.113412] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23254.115684] ipmi_devintf ipmi_msghandler [23254.122807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23254.125338] ioatdma acpi_pad [23254.133416] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23254.137420] acpi_power_meter ixgbe [23254.143158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23254.146122] igb mdio [23254.153246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23254.156747] crc32c_intel nvme [23254.163870] Call Trace: [23254.163871] [23254.166141] ptp nvme_core [23254.173271] ? watchdog_timer_fn+0x1b4/0x220 [23254.176317] i2c_algo_bit pps_core [23254.178762] ? __hrtimer_run_queues+0x1d3/0x360 [23254.180770] dca [23254.183477] ? hrtimer_interrupt+0x104/0x240 [23254.187739] fuse [23254.191135] ? __local_bh_enable+0x50/0x60 [23254.195660] irq event stamp: 108314854 [23254.197500] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23254.201762] hardirqs last enabled at (108314853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23254.203686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23254.207776] hardirqs last disabled at (108314854): [] sysvec_apic_timer_interrupt+0xe/0x90 [23254.211522] [23254.216739] softirqs last enabled at (108314852): [] __do_softirq+0x2be/0x371 [23254.226991] [23254.231948] softirqs last disabled at (108314847): [] irq_exit_rcu+0xb0/0xd0 [23254.241761] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23254.243856] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23254.252631] ? queued_spin_lock_slowpath+0x18/0x2e0 [23254.254725] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23254.263325] do_raw_spin_lock+0xb5/0xc0 [23254.268627] Workqueue: events delayed_work [ceph] [23254.278006] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23254.282874] [23254.282875] RIP: 0010:lock_acquire+0xf0/0x2b0 [23254.290347] try_get_cap_refs+0x159/0x880 [ceph] [23254.294175] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23254.298879] __ceph_get_caps+0x4a1/0x760 [ceph] [23254.303658] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23254.305153] ? __file_remove_privs+0x89/0x110 [23254.309498] [23254.309499] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23254.314110] ? wake_bit_function+0x70/0x70 [23254.332848] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23254.337375] ? ceph_write_iter+0x260/0x770 [ceph] [23254.342590] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23254.346941] ceph_write_iter+0x260/0x770 [ceph] [23254.348430] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23254.355558] ? lock_is_held_type+0xaa/0x120 [23254.359646] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23254.366772] vfs_write+0x324/0x4f0 [23254.371468] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23254.378598] ksys_pwrite64+0x65/0xa0 [23254.383115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23254.390239] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23254.394416] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23254.401543] do_syscall_64+0x38/0x80 [23254.404939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23254.413019] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23254.416587] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23254.422326] RIP: 0033:0x7f48f0eeb93b [23254.427455] Call Trace: [23254.434581] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23254.438150] [23254.445284] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23254.450328] ? watchdog_timer_fn+0x1b4/0x220 [23254.457451] ORIG_RAX: 0000000000000012 [23254.461023] ? __hrtimer_run_queues+0x1d3/0x360 [23254.463466] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23254.482207] ? hrtimer_interrupt+0x104/0x240 [23254.484213] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23254.489432] ? __local_bh_enable+0x50/0x60 [23254.493696] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23254.497530] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23254.502049] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23254.509174] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23254.513439] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23254.520562] [23254.524657] [23254.531776] [23254.562581] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23254.567889] ? lock_acquire+0xf0/0x2b0 [23254.571638] ? igrab+0x43/0x50 [23254.574689] ? lock_release+0xc5/0x260 [23254.578433] _raw_spin_lock+0x30/0x40 [23254.582098] ? igrab+0x1d/0x50 [23254.585147] igrab+0x1d/0x50 [23254.588027] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23254.593177] delayed_work+0x1d8/0x280 [ceph] [23254.597457] ? process_scheduled_works+0x22a/0x510 [23254.602248] process_scheduled_works+0x2a4/0x510 [23254.606864] worker_thread+0x188/0x340 [23254.610613] ? rescuer_thread+0x310/0x310 [23254.614623] kthread+0xf5/0x130 [23254.617763] ? kthread_complete_and_exit+0x20/0x20 [23254.622545] ret_from_fork+0x31/0x40 [23254.626126] ? kthread_complete_and_exit+0x20/0x20 [23254.630908] ret_from_fork_asm+0x11/0x20 [23254.634833] [23281.866066] watchdog: BUG: soft lockup - CPU#0 stuck for 10568s! [dbench:78683] [23281.873369] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23281.878067] watchdog: BUG: soft lockup - CPU#7 stuck for 19565s! [kworker/7:4:78808] [23281.928810] irq event stamp: 36428812 [23281.928811] hardirqs last enabled at (36428811): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23281.936543] Modules linked in: [23281.940198] hardirqs last disabled at (36428812): [] sysvec_apic_timer_interrupt+0xe/0x90 [23281.950357] ceph [23281.953408] softirqs last enabled at (36428754): [] __do_softirq+0x2be/0x371 [23281.963132] libceph [23281.965055] softirqs last disabled at (36428749): [] irq_exit_rcu+0xb0/0xd0 [23281.973743] dns_resolver fscache [23281.975925] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23281.984434] netfs veth [23281.987763] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23281.996707] nft_chain_nat xt_MASQUERADE [23281.999151] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23282.006622] nf_nat nf_conntrack [23282.010540] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23282.016017] nf_defrag_ipv6 nf_defrag_ipv4 [23282.019241] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23282.037979] nft_compat nf_tables [23282.042069] [23282.042070] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23282.047286] nfnetlink bridge [23282.050597] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23282.052087] stp llc [23282.059212] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23282.062176] xfs libcrc32c [23282.069300] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23282.071483] sunrpc intel_rapl_msr [23282.078608] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23282.081313] intel_rapl_common [23282.088437] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23282.091850] sb_edac x86_pkg_temp_thermal [23282.098977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23282.102025] coretemp kvm_intel [23282.110103] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23282.114109] kvm irqbypass [23282.119845] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23282.122984] i2c_i801 [23282.130107] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23282.132811] iTCO_wdt iTCO_vendor_support [23282.139935] Call Trace: [23282.142205] i2c_smbus crct10dif_pclmul [23282.149331] [23282.153334] joydev crc32_pclmul [23282.155780] ? watchdog_timer_fn+0x1b4/0x220 [23282.159610] ipmi_ssif ghash_clmulni_intel [23282.161622] ? __hrtimer_run_queues+0x1d3/0x360 [23282.164844] mei_me [23282.169111] ? hrtimer_interrupt+0x104/0x240 [23282.173197] mei lpc_ich [23282.177722] ? __local_bh_enable+0x50/0x60 [23282.179820] sha512_ssse3 acpi_ipmi [23282.184086] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23282.186615] mfd_core wmi [23282.190706] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23282.194189] ipmi_si [23282.199406] [23282.202024] ipmi_devintf ipmi_msghandler [23282.206982] [23282.209166] ioatdma acpi_pad [23282.211263] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23282.215266] acpi_power_meter ixgbe [23282.217368] ? queued_spin_lock_slowpath+0x16/0x2e0 [23282.220328] igb [23282.225636] do_raw_spin_lock+0xb5/0xc0 [23282.229117] mdio crc32c_intel [23282.233988] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23282.235826] nvme ptp [23282.239659] try_get_cap_refs+0x159/0x880 [ceph] [23282.242705] nvme_core i2c_algo_bit [23282.247496] __ceph_get_caps+0x4a1/0x760 [ceph] [23282.249761] pps_core dca [23282.254377] ? __file_remove_privs+0x89/0x110 [23282.257855] fuse [23282.262381] ? wake_bit_function+0x70/0x70 [23282.264997] irq event stamp: 108535986 [23282.269351] ? ceph_write_iter+0x260/0x770 [ceph] [23282.271273] hardirqs last enabled at (108535985): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23282.275363] ceph_write_iter+0x260/0x770 [ceph] [23282.279107] hardirqs last disabled at (108535986): [] sysvec_apic_timer_interrupt+0xe/0x90 [23282.283808] ? lock_is_held_type+0xaa/0x120 [23282.294048] softirqs last enabled at (108535984): [] __do_softirq+0x2be/0x371 [23282.298575] vfs_write+0x324/0x4f0 [23282.308383] softirqs last disabled at (108535979): [] irq_exit_rcu+0xb0/0xd0 [23282.312567] ksys_pwrite64+0x65/0xa0 [23282.321333] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23282.324745] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23282.333344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23282.336917] do_syscall_64+0x38/0x80 [23282.346291] Workqueue: events delayed_work [ceph] [23282.351424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23282.358894] [23282.358895] RIP: 0010:lock_acquire+0xf0/0x2b0 [23282.362469] RIP: 0033:0x7f48f0eeb93b [23282.367161] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23282.372207] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23282.373697] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23282.378047] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23282.381617] [23282.381618] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23282.400355] ORIG_RAX: 0000000000000012 [23282.419094] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23282.424310] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23282.429528] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23282.431018] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23282.438144] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23282.441973] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23282.449098] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23282.456223] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23282.463347] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23282.470469] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23282.477595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23282.484725] [23282.491842] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23282.529213] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23282.536338] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23282.543469] Call Trace: [23282.545915] [23282.547926] ? watchdog_timer_fn+0x1b4/0x220 [23282.552190] ? __hrtimer_run_queues+0x1d3/0x360 [23282.556717] ? hrtimer_interrupt+0x104/0x240 [23282.560980] ? __local_bh_enable+0x50/0x60 [23282.565072] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23282.570296] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23282.575253] [23282.577351] [23282.579448] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23282.584773] ? lock_acquire+0xf0/0x2b0 [23282.588523] ? igrab+0x43/0x50 [23282.591574] ? lock_release+0xc5/0x260 [23282.595319] _raw_spin_lock+0x30/0x40 [23282.598983] ? igrab+0x1d/0x50 [23282.602034] igrab+0x1d/0x50 [23282.604910] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23282.610054] delayed_work+0x1d8/0x280 [ceph] [23282.614333] ? process_scheduled_works+0x22a/0x510 [23282.619114] process_scheduled_works+0x2a4/0x510 [23282.623732] worker_thread+0x188/0x340 [23282.627480] ? rescuer_thread+0x310/0x310 [23282.631482] kthread+0xf5/0x130 [23282.634621] ? kthread_complete_and_exit+0x20/0x20 [23282.639406] ret_from_fork+0x31/0x40 [23282.642974] ? kthread_complete_and_exit+0x20/0x20 [23282.647771] ret_from_fork_asm+0x11/0x20 [23282.651699] [23309.866543] watchdog: BUG: soft lockup - CPU#0 stuck for 10595s! [dbench:78683] [23309.873850] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23309.878542] watchdog: BUG: soft lockup - CPU#7 stuck for 19592s! [kworker/7:4:78808] [23309.929299] irq event stamp: 36486986 [23309.929300] hardirqs last enabled at (36486985): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23309.937031] Modules linked in: ceph [23309.940686] hardirqs last disabled at (36486986): [] sysvec_apic_timer_interrupt+0xe/0x90 [23309.950844] libceph [23309.954329] softirqs last enabled at (36486940): [] __do_softirq+0x2be/0x371 [23309.964053] dns_resolver fscache [23309.966236] softirqs last disabled at (36486935): [] irq_exit_rcu+0xb0/0xd0 [23309.974936] netfs veth [23309.978249] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23309.986760] nft_chain_nat xt_MASQUERADE [23309.989219] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23309.998164] nf_nat nf_conntrack [23310.002083] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23310.009554] nf_defrag_ipv6 [23310.012779] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23310.018256] nf_defrag_ipv4 nft_compat [23310.021047] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23310.039783] nf_tables nfnetlink [23310.043527] [23310.043528] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23310.048744] bridge [23310.051970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23310.053461] stp llc [23310.060584] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23310.062680] xfs libcrc32c [23310.069806] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23310.071988] sunrpc [23310.079113] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23310.081819] intel_rapl_msr intel_rapl_common [23310.088941] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23310.091039] sb_edac x86_pkg_temp_thermal [23310.098164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23310.102515] coretemp kvm_intel [23310.110590] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23310.114596] kvm [23310.120333] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23310.123471] irqbypass i2c_i801 [23310.130594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23310.132432] iTCO_wdt iTCO_vendor_support [23310.139556] Call Trace: [23310.142693] i2c_smbus crct10dif_pclmul [23310.149820] [23310.153822] joydev [23310.156267] ? watchdog_timer_fn+0x1b4/0x220 [23310.160098] crc32_pclmul [23310.162108] ? __hrtimer_run_queues+0x1d3/0x360 [23310.164220] ipmi_ssif ghash_clmulni_intel [23310.168488] ? hrtimer_interrupt+0x104/0x240 [23310.171102] mei_me mei [23310.175627] ? __local_bh_enable+0x50/0x60 [23310.179717] lpc_ich sha512_ssse3 [23310.183986] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23310.186425] acpi_ipmi mfd_core [23310.190517] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23310.193828] wmi ipmi_si [23310.199046] [23310.202182] ipmi_devintf ipmi_msghandler [23310.207140] [23310.209671] ioatdma acpi_pad [23310.211769] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23310.215772] acpi_power_meter [23310.217873] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23310.220832] ixgbe igb [23310.226140] do_raw_spin_lock+0xb5/0xc0 [23310.229101] mdio crc32c_intel [23310.233974] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23310.236329] nvme [23310.240164] try_get_cap_refs+0x159/0x880 [ceph] [23310.243212] ptp nvme_core [23310.248000] __ceph_get_caps+0x4a1/0x760 [ceph] [23310.249918] i2c_algo_bit pps_core [23310.254552] ? __file_remove_privs+0x89/0x110 [23310.257251] dca fuse [23310.261777] ? wake_bit_function+0x70/0x70 [23310.265173] [23310.265174] irq event stamp: 108756886 [23310.269527] ? ceph_write_iter+0x260/0x770 [ceph] [23310.271795] hardirqs last enabled at (108756885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23310.275886] ceph_write_iter+0x260/0x770 [ceph] [23310.277376] hardirqs last disabled at (108756886): [] sysvec_apic_timer_interrupt+0xe/0x90 [23310.281125] ? lock_is_held_type+0xaa/0x120 [23310.285817] softirqs last enabled at (108756884): [] __do_softirq+0x2be/0x371 [23310.296065] vfs_write+0x324/0x4f0 [23310.300584] softirqs last disabled at (108756879): [] irq_exit_rcu+0xb0/0xd0 [23310.310401] ksys_pwrite64+0x65/0xa0 [23310.314573] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23310.323346] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23310.326742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23310.335341] do_syscall_64+0x38/0x80 [23310.338910] Workqueue: events delayed_work [ceph] [23310.348288] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23310.353419] [23310.353420] RIP: 0010:ceph_check_delayed_caps+0x66/0x270 [ceph] [23310.360890] RIP: 0033:0x7f48f0eeb93b [23310.364459] Code: 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 <4c> 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 [23310.369158] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23310.374218] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [23310.375709] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23310.381621] [23310.381622] RAX: 0000000000023e20 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [23310.385192] ORIG_RAX: 0000000000000012 [23310.403945] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [23310.403946] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [23310.422683] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23310.427901] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23310.433118] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23310.434609] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23310.441733] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23310.445565] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23310.452689] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23310.459813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23310.466946] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23310.474070] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23310.481200] [23310.488318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23310.539948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23310.547071] Call Trace: [23310.549514] [23310.551526] ? watchdog_timer_fn+0x1b4/0x220 [23310.555789] ? __hrtimer_run_queues+0x1d3/0x360 [23310.560318] ? hrtimer_interrupt+0x104/0x240 [23310.564585] ? __local_bh_enable+0x50/0x60 [23310.568681] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23310.573902] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23310.578861] [23310.580959] [23310.583055] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23310.588364] ? ceph_check_delayed_caps+0x66/0x270 [ceph] [23310.593683] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [23310.599004] delayed_work+0x1d8/0x280 [ceph] [23310.603288] ? process_scheduled_works+0x22a/0x510 [23310.608075] process_scheduled_works+0x2a4/0x510 [23310.612692] worker_thread+0x188/0x340 [23310.616441] ? rescuer_thread+0x310/0x310 [23310.620444] kthread+0xf5/0x130 [23310.623592] ? kthread_complete_and_exit+0x20/0x20 [23310.628384] ret_from_fork+0x31/0x40 [23310.631963] ? kthread_complete_and_exit+0x20/0x20 [23310.636747] ret_from_fork_asm+0x11/0x20 [23310.640679] [23337.621378] rcu: INFO: rcu_preempt self-detected stall on CPU [23337.627141] rcu: 0-....: (11144698 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=5564202 [23337.637560] rcu: (t=11400808 jiffies g=1309609 q=369387 ncpus=8) [23337.643644] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23337.652588] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23337.660059] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23337.665544] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23337.684282] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23337.689498] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23337.696623] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23337.703748] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23337.710871] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23337.717996] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23337.725119] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23337.733196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23337.738935] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23337.746059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23337.753184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23337.760306] Call Trace: [23337.762753] [23337.764761] ? rcu_dump_cpu_stacks+0xd1/0x180 [23337.769121] ? rcu_sched_clock_irq+0x633/0x1390 [23337.773645] ? lock_is_held_type+0xaa/0x120 [23337.777825] ? lock_is_held_type+0xaa/0x120 [23337.782002] ? tick_sched_do_timer+0xa0/0xa0 [23337.786273] ? rcu_read_lock_sched_held+0x56/0x90 [23337.790971] ? tick_sched_do_timer+0xa0/0xa0 [23337.795233] ? update_process_times+0x5f/0x90 [23337.799586] ? tick_sched_handle.isra.16+0x1e/0x50 [23337.804370] ? tick_sched_timer+0x6f/0x80 [23337.808375] ? __hrtimer_run_queues+0x1d3/0x360 [23337.812901] ? hrtimer_interrupt+0x104/0x240 [23337.817171] ? __local_bh_enable+0x50/0x60 [23337.821262] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23337.826478] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23337.831437] [23337.833534] [23337.835631] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23337.840939] ? queued_spin_lock_slowpath+0x16/0x2e0 [23337.845815] do_raw_spin_lock+0xb5/0xc0 [23337.849646] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23337.854448] try_get_cap_refs+0x159/0x880 [ceph] [23337.859079] __ceph_get_caps+0x4a1/0x760 [ceph] [23337.863619] ? __file_remove_privs+0x89/0x110 [23337.867976] ? wake_bit_function+0x70/0x70 [23337.872067] ? ceph_write_iter+0x260/0x770 [ceph] [23337.876773] ceph_write_iter+0x260/0x770 [ceph] [23337.879017] watchdog: BUG: soft lockup - CPU#7 stuck for 19618s! [kworker/7:4:78808] [23337.881308] ? lock_is_held_type+0xaa/0x120 [23337.889036] Modules linked in: ceph libceph [23337.893215] vfs_write+0x324/0x4f0 [23337.893217] dns_resolver [23337.893223] ksys_pwrite64+0x65/0xa0 [23337.897410] fscache [23337.900804] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23337.903421] netfs [23337.906993] do_syscall_64+0x38/0x80 [23337.909177] veth [23337.914309] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23337.916319] nft_chain_nat [23337.919888] RIP: 0033:0x7f48f0eeb93b [23337.921812] xt_MASQUERADE [23337.926857] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23337.929561] nf_nat [23337.933132] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23337.935837] nf_conntrack [23337.954573] ORIG_RAX: 0000000000000012 [23337.954574] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23337.956670] nf_defrag_ipv6 [23337.961888] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23337.964505] nf_defrag_ipv4 [23337.968337] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23337.968339] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23337.975460] nft_compat [23337.978251] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23337.985376] nf_tables [23337.988171] [23337.995289] nfnetlink [23338.002414] Sending NMI from CPU 0 to CPUs 7: [23338.004860] bridge [23338.011985] NMI backtrace for cpu 7 [23338.011987] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23338.011989] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23338.011990] Workqueue: events delayed_work [ceph] [23338.012010] RIP: 0010:__lock_acquire+0x63b/0x1e20 [23338.012013] Code: 48 83 78 40 00 0f 84 6c 06 00 00 0f b7 44 24 20 44 8b 7c 24 38 8b 54 24 28 8b 7c 24 40 c1 e0 0d 66 0b 44 24 18 98 2b 44 24 30 <41> 31 c7 44 89 f8 44 29 fa 41 01 ff c1 c0 06 31 d0 29 c7 89 c2 44 [23338.012015] RSP: 0018:ffffc90000264cd8 EFLAGS: 00000017 [23338.012016] RAX: 0000000000000003 RBX: ffff88810debaa80 RCX: 0000000000000003 [23338.012018] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: 00000000fffffffe [23338.012019] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [23338.012020] R10: 0000000000000000 R11: 0000000000000001 R12: b656a521d3820f2c [23338.012022] R13: 0000000000000000 R14: 0000000000000001 R15: 00000000ffffffff [23338.012023] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23338.012024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23338.012026] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23338.012027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23338.012028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23338.012030] Call Trace: [23338.012031] [23338.012032] ? nmi_cpu_backtrace+0xad/0x120 [23338.012036] ? nmi_cpu_backtrace_handler+0x11/0x20 [23338.012038] ? nmi_handle+0xd4/0x260 [23338.012043] ? default_do_nmi+0x49/0xf0 [23338.012045] ? exc_nmi+0x11e/0x150 [23338.012048] ? end_repeat_nmi+0x16/0x67 [23338.012053] ? __lock_acquire+0x63b/0x1e20 [23338.012056] ? __lock_acquire+0x63b/0x1e20 [23338.012058] ? __lock_acquire+0x63b/0x1e20 [23338.012061] [23338.012061] [23338.012065] lock_acquire+0xcb/0x2b0 [23338.012067] ? vprintk_emit+0x221/0x340 [23338.012070] ? vprintk_emit+0x200/0x340 [23338.012073] ? lock_release+0xc5/0x260 [23338.012076] vprintk_emit+0x240/0x340 [23338.012078] ? vprintk_emit+0x221/0x340 [23338.012080] ? irq_work_queue+0x2c/0x50 [23338.012085] ? touch_softlockup_watchdog+0x20/0x20 [23338.012087] _printk+0x5c/0x80 [23338.012092] print_modules+0x6e/0xe0 [23338.012095] watchdog_timer_fn+0x19f/0x220 [23338.012098] __hrtimer_run_queues+0x1d3/0x360 [23338.012102] hrtimer_interrupt+0x104/0x240 [23338.012104] ? __local_bh_enable+0x50/0x60 [23338.012109] __sysvec_apic_timer_interrupt+0x61/0x160 [23338.012112] sysvec_apic_timer_interrupt+0x6d/0x90 [23338.012115] [23338.012116] [23338.012117] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23338.012119] RIP: 0010:lock_release+0x14e/0x260 [23338.012121] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [23338.012123] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [23338.012124] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [23338.012126] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [23338.012127] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [23338.012128] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [23338.012129] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [23338.012131] ? igrab+0x43/0x50 [23338.012136] ? lock_release+0xc5/0x260 [23338.012139] _raw_spin_unlock+0x1b/0x40 [23338.012142] igrab+0x43/0x50 [23338.012145] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23338.012164] delayed_work+0x1d8/0x280 [ceph] [23338.012184] ? process_scheduled_works+0x22a/0x510 [23338.012187] process_scheduled_works+0x2a4/0x510 [23338.012192] worker_thread+0x188/0x340 [23338.012196] ? rescuer_thread+0x310/0x310 [23338.012199] kthread+0xf5/0x130 [23338.012201] ? kthread_complete_and_exit+0x20/0x20 [23338.012205] ret_from_fork+0x31/0x40 [23338.012207] ? kthread_complete_and_exit+0x20/0x20 [23338.012210] ret_from_fork_asm+0x11/0x20 [23338.012216] [23338.394433] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23338.434351] irq event stamp: 108978462 [23338.438096] hardirqs last enabled at (108978461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23338.448341] hardirqs last disabled at (108978462): [] sysvec_apic_timer_interrupt+0xe/0x90 [23338.458150] softirqs last enabled at (108978460): [] __do_softirq+0x2be/0x371 [23338.466921] softirqs last disabled at (108978455): [] irq_exit_rcu+0xb0/0xd0 [23338.475519] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23338.484897] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23338.492367] Workqueue: events delayed_work [ceph] [23338.497082] RIP: 0010:lock_release+0x14e/0x260 [23338.501520] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [23338.520259] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [23338.525474] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [23338.532600] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [23338.539723] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [23338.546848] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [23338.553971] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [23338.561096] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23338.569174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23338.574912] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23338.582034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23338.589158] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23338.596282] Call Trace: [23338.598727] [23338.600738] ? watchdog_timer_fn+0x1b4/0x220 [23338.605002] ? __hrtimer_run_queues+0x1d3/0x360 [23338.609531] ? hrtimer_interrupt+0x104/0x240 [23338.613799] ? __local_bh_enable+0x50/0x60 [23338.617892] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23338.623116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23338.628072] [23338.630171] [23338.632269] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23338.637573] ? igrab+0x43/0x50 [23338.640627] ? lock_release+0x14e/0x260 [23338.644464] ? lock_release+0xc5/0x260 [23338.648207] _raw_spin_unlock+0x1b/0x40 [23338.652037] igrab+0x43/0x50 [23338.654914] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23338.660064] delayed_work+0x1d8/0x280 [ceph] [23338.664345] ? process_scheduled_works+0x22a/0x510 [23338.669127] process_scheduled_works+0x2a4/0x510 [23338.673744] worker_thread+0x188/0x340 [23338.677491] ? rescuer_thread+0x310/0x310 [23338.681497] kthread+0xf5/0x130 [23338.684643] ? kthread_complete_and_exit+0x20/0x20 [23338.689428] ret_from_fork+0x31/0x40 [23338.693005] ? kthread_complete_and_exit+0x20/0x20 [23338.697788] ret_from_fork_asm+0x11/0x20 [23338.701711] [23361.867424] watchdog: BUG: soft lockup - CPU#0 stuck for 10643s! [dbench:78683] [23361.874727] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23361.930169] irq event stamp: 36595426 [23361.933828] hardirqs last enabled at (36595425): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23361.943992] hardirqs last disabled at (36595426): [] sysvec_apic_timer_interrupt+0xe/0x90 [23361.953717] softirqs last enabled at (36595376): [] __do_softirq+0x2be/0x371 [23361.962401] softirqs last disabled at (36595371): [] irq_exit_rcu+0xb0/0xd0 [23361.970912] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23361.979858] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23361.987327] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23361.992820] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23362.011559] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23362.016776] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23362.023902] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23362.031024] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23362.038150] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23362.045274] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23362.052398] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23362.060476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23362.066213] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23362.073336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23362.080462] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23362.087584] Call Trace: [23362.090029] [23362.092039] ? watchdog_timer_fn+0x1b4/0x220 [23362.096304] ? __hrtimer_run_queues+0x1d3/0x360 [23362.100830] ? hrtimer_interrupt+0x104/0x240 [23362.105093] ? __local_bh_enable+0x50/0x60 [23362.109202] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23362.114427] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23362.119383] [23362.121481] [23362.123578] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23362.128886] ? queued_spin_lock_slowpath+0x16/0x2e0 [23362.133765] do_raw_spin_lock+0xb5/0xc0 [23362.137602] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23362.142404] try_get_cap_refs+0x159/0x880 [ceph] [23362.147036] __ceph_get_caps+0x4a1/0x760 [ceph] [23362.151577] ? __file_remove_privs+0x89/0x110 [23362.155933] ? wake_bit_function+0x70/0x70 [23362.160025] ? ceph_write_iter+0x260/0x770 [ceph] [23362.164738] ceph_write_iter+0x260/0x770 [ceph] [23362.169274] ? lock_is_held_type+0xaa/0x120 [23362.173458] vfs_write+0x324/0x4f0 [23362.176866] ksys_pwrite64+0x65/0xa0 [23362.180442] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23362.185574] do_syscall_64+0x38/0x80 [23362.189151] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23362.194195] RIP: 0033:0x7f48f0eeb93b [23362.197768] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23362.216504] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23362.224062] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23362.231186] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23362.238311] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23362.245435] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23362.252559] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23362.259688] [23365.879492] watchdog: BUG: soft lockup - CPU#7 stuck for 19644s! [kworker/7:4:78808] [23365.887245] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23365.942684] irq event stamp: 109198870 [23365.946428] hardirqs last enabled at (109198869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23365.956674] hardirqs last disabled at (109198870): [] sysvec_apic_timer_interrupt+0xe/0x90 [23365.966483] softirqs last enabled at (109198868): [] __do_softirq+0x2be/0x371 [23365.975255] softirqs last disabled at (109198863): [] irq_exit_rcu+0xb0/0xd0 [23365.983851] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23365.993229] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23366.000702] Workqueue: events delayed_work [ceph] [23366.005414] RIP: 0010:lock_release+0x3c/0x260 [23366.009765] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [23366.028504] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [23366.033720] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [23366.040846] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [23366.047968] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [23366.055093] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [23366.062219] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23366.069342] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23366.077420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23366.083167] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23366.090290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23366.097414] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23366.104537] Call Trace: [23366.106982] [23366.108994] ? watchdog_timer_fn+0x1b4/0x220 [23366.113259] ? __hrtimer_run_queues+0x1d3/0x360 [23366.117785] ? hrtimer_interrupt+0x104/0x240 [23366.122055] ? __local_bh_enable+0x50/0x60 [23366.126148] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23366.131389] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23366.136346] [23366.138441] [23366.140541] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23366.145847] ? igrab+0x43/0x50 [23366.148898] ? igrab+0x43/0x50 [23366.151954] ? lock_release+0x3c/0x260 [23366.155700] ? lock_release+0xc5/0x260 [23366.159444] _raw_spin_unlock+0x1b/0x40 [23366.163272] igrab+0x43/0x50 [23366.166150] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23366.171309] delayed_work+0x1d8/0x280 [ceph] [23366.175591] ? process_scheduled_works+0x22a/0x510 [23366.180380] process_scheduled_works+0x2a4/0x510 [23366.184997] worker_thread+0x188/0x340 [23366.188747] ? rescuer_thread+0x310/0x310 [23366.192748] kthread+0xf5/0x130 [23366.195893] ? kthread_complete_and_exit+0x20/0x20 [23366.200680] ret_from_fork+0x31/0x40 [23366.204257] ? kthread_complete_and_exit+0x20/0x20 [23366.209043] ret_from_fork_asm+0x11/0x20 [23366.212967] [23389.867901] watchdog: BUG: soft lockup - CPU#0 stuck for 10669s! [dbench:78683] [23389.875208] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23389.930656] irq event stamp: 36654096 [23389.934314] hardirqs last enabled at (36654095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23389.944481] hardirqs last disabled at (36654096): [] sysvec_apic_timer_interrupt+0xe/0x90 [23389.954205] softirqs last enabled at (36654094): [] __do_softirq+0x2be/0x371 [23389.962898] softirqs last disabled at (36654089): [] irq_exit_rcu+0xb0/0xd0 [23389.971416] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23389.980361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23389.987833] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23389.993310] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23390.012047] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23390.017265] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23390.024389] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23390.031513] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23390.038636] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23390.045762] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23390.052884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23390.060961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23390.066699] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23390.073825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23390.080947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23390.088073] Call Trace: [23390.090517] [23390.092528] ? watchdog_timer_fn+0x1b4/0x220 [23390.096799] ? __hrtimer_run_queues+0x1d3/0x360 [23390.101328] ? hrtimer_interrupt+0x104/0x240 [23390.105597] ? __local_bh_enable+0x50/0x60 [23390.109691] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23390.114913] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23390.119871] [23390.121968] [23390.124066] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23390.129372] ? queued_spin_lock_slowpath+0x18/0x2e0 [23390.134250] do_raw_spin_lock+0xb5/0xc0 [23390.138090] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23390.142902] try_get_cap_refs+0x159/0x880 [ceph] [23390.147532] __ceph_get_caps+0x4a1/0x760 [ceph] [23390.152071] ? __file_remove_privs+0x89/0x110 [23390.156427] ? wake_bit_function+0x70/0x70 [23390.160519] ? ceph_write_iter+0x260/0x770 [ceph] [23390.165224] ceph_write_iter+0x260/0x770 [ceph] [23390.169760] ? lock_is_held_type+0xaa/0x120 [23390.173946] vfs_write+0x324/0x4f0 [23390.177355] ksys_pwrite64+0x65/0xa0 [23390.180930] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23390.186068] do_syscall_64+0x38/0x80 [23390.189638] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23390.194683] RIP: 0033:0x7f48f0eeb93b [23390.198253] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23390.216992] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23390.224550] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23390.231674] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23390.238796] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23390.245922] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23390.253044] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23390.260175] [23393.879968] watchdog: BUG: soft lockup - CPU#7 stuck for 19670s! [kworker/7:4:78808] [23393.887722] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23393.943164] irq event stamp: 109423556 [23393.946906] hardirqs last enabled at (109423555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23393.957151] hardirqs last disabled at (109423556): [] sysvec_apic_timer_interrupt+0xe/0x90 [23393.966963] softirqs last enabled at (109423554): [] __do_softirq+0x2be/0x371 [23393.975734] softirqs last disabled at (109423549): [] irq_exit_rcu+0xb0/0xd0 [23393.984331] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23393.993707] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23394.001179] Workqueue: events delayed_work [ceph] [23394.005895] RIP: 0010:rcu_is_watching+0x0/0x40 [23394.010340] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [23394.029094] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [23394.034312] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [23394.041435] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [23394.048559] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [23394.055685] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [23394.062807] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23394.069932] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23394.078011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23394.083746] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23394.090872] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23394.097995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23394.105120] Call Trace: [23394.107565] [23394.109576] ? watchdog_timer_fn+0x1b4/0x220 [23394.113840] ? __hrtimer_run_queues+0x1d3/0x360 [23394.118369] ? hrtimer_interrupt+0x104/0x240 [23394.122647] ? __local_bh_enable+0x50/0x60 [23394.126748] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23394.131971] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23394.136928] [23394.139024] [23394.141123] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23394.146434] ? igrab+0x43/0x50 [23394.149487] ? igrab+0x43/0x50 [23394.152536] ? rcutorture_get_gp_data+0x30/0x30 [23394.157061] lock_release+0x1d7/0x260 [23394.160720] _raw_spin_unlock+0x1b/0x40 [23394.164558] igrab+0x43/0x50 [23394.167436] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23394.172587] delayed_work+0x1d8/0x280 [ceph] [23394.176866] ? process_scheduled_works+0x22a/0x510 [23394.181656] process_scheduled_works+0x2a4/0x510 [23394.186272] worker_thread+0x188/0x340 [23394.190022] ? rescuer_thread+0x310/0x310 [23394.194023] kthread+0xf5/0x130 [23394.197164] ? kthread_complete_and_exit+0x20/0x20 [23394.201955] ret_from_fork+0x31/0x40 [23394.205534] ? kthread_complete_and_exit+0x20/0x20 [23394.210319] ret_from_fork_asm+0x11/0x20 [23394.214249] [23417.868376] watchdog: BUG: soft lockup - CPU#0 stuck for 10695s! [dbench:78683] [23417.875677] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23417.931119] irq event stamp: 36712844 [23417.934776] hardirqs last enabled at (36712843): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23417.944940] hardirqs last disabled at (36712844): [] sysvec_apic_timer_interrupt+0xe/0x90 [23417.954666] softirqs last enabled at (36712842): [] __do_softirq+0x2be/0x371 [23417.963349] softirqs last disabled at (36712837): [] irq_exit_rcu+0xb0/0xd0 [23417.971861] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23417.980803] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23417.988275] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23417.993771] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23418.012507] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23418.017725] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23418.024848] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23418.031974] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23418.039098] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23418.046221] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23418.053346] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23418.061423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23418.067162] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23418.074285] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23418.081408] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23418.088533] Call Trace: [23418.090979] [23418.092989] ? watchdog_timer_fn+0x1b4/0x220 [23418.097255] ? __hrtimer_run_queues+0x1d3/0x360 [23418.101780] ? hrtimer_interrupt+0x104/0x240 [23418.106065] ? __local_bh_enable+0x50/0x60 [23418.110159] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23418.115384] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23418.120340] [23418.122437] [23418.124537] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23418.129842] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23418.134720] do_raw_spin_lock+0xb5/0xc0 [23418.138552] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23418.143354] try_get_cap_refs+0x159/0x880 [ceph] [23418.147985] __ceph_get_caps+0x4a1/0x760 [ceph] [23418.152524] ? __file_remove_privs+0x89/0x110 [23418.156879] ? wake_bit_function+0x70/0x70 [23418.160973] ? ceph_write_iter+0x260/0x770 [ceph] [23418.165685] ceph_write_iter+0x260/0x770 [ceph] [23418.170220] ? lock_is_held_type+0xaa/0x120 [23418.174397] vfs_write+0x324/0x4f0 [23418.177796] ksys_pwrite64+0x65/0xa0 [23418.181373] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23418.186513] do_syscall_64+0x38/0x80 [23418.190083] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23418.195125] RIP: 0033:0x7f48f0eeb93b [23418.198698] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23418.217435] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23418.224991] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23418.232115] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23418.239240] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23418.246365] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23418.253490] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23418.260619] [23421.880444] watchdog: BUG: soft lockup - CPU#7 stuck for 19696s! [kworker/7:4:78808] [23421.888201] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23421.943640] irq event stamp: 109648280 [23421.947386] hardirqs last enabled at (109648279): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23421.957631] hardirqs last disabled at (109648280): [] sysvec_apic_timer_interrupt+0xe/0x90 [23421.967440] softirqs last enabled at (109648278): [] __do_softirq+0x2be/0x371 [23421.976211] softirqs last disabled at (109648273): [] irq_exit_rcu+0xb0/0xd0 [23421.984808] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23421.994186] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23422.001658] Workqueue: events delayed_work [ceph] [23422.006372] RIP: 0010:lock_acquire+0xf0/0x2b0 [23422.010722] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23422.029460] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23422.034678] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23422.041801] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23422.048925] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23422.056050] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23422.063174] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23422.070297] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23422.078375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23422.084123] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23422.091246] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23422.098370] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23422.105495] Call Trace: [23422.107940] [23422.109951] ? watchdog_timer_fn+0x1b4/0x220 [23422.114215] ? __hrtimer_run_queues+0x1d3/0x360 [23422.118740] ? hrtimer_interrupt+0x104/0x240 [23422.123002] ? __local_bh_enable+0x50/0x60 [23422.127094] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23422.132327] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23422.137285] [23422.139381] [23422.141479] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23422.146787] ? lock_acquire+0xf0/0x2b0 [23422.150536] ? igrab+0x43/0x50 [23422.153587] ? lock_release+0xc5/0x260 [23422.157332] _raw_spin_lock+0x30/0x40 [23422.160998] ? igrab+0x1d/0x50 [23422.164046] igrab+0x1d/0x50 [23422.166927] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23422.172075] delayed_work+0x1d8/0x280 [ceph] [23422.176358] ? process_scheduled_works+0x22a/0x510 [23422.181148] process_scheduled_works+0x2a4/0x510 [23422.185763] worker_thread+0x188/0x340 [23422.189513] ? rescuer_thread+0x310/0x310 [23422.193522] kthread+0xf5/0x130 [23422.196662] ? kthread_complete_and_exit+0x20/0x20 [23422.201447] ret_from_fork+0x31/0x40 [23422.205024] ? kthread_complete_and_exit+0x20/0x20 [23422.209809] ret_from_fork_asm+0x11/0x20 [23422.213733] [23445.868853] watchdog: BUG: soft lockup - CPU#0 stuck for 10721s! [dbench:78683] [23445.876155] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23445.931604] irq event stamp: 36771642 [23445.935262] hardirqs last enabled at (36771641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23445.945428] hardirqs last disabled at (36771642): [] sysvec_apic_timer_interrupt+0xe/0x90 [23445.955152] softirqs last enabled at (36771640): [] __do_softirq+0x2be/0x371 [23445.963845] softirqs last disabled at (36771635): [] irq_exit_rcu+0xb0/0xd0 [23445.972357] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23445.981300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23445.988773] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23445.994258] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23446.012995] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23446.018212] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23446.025337] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23446.032460] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23446.039585] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23446.046710] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23446.053833] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23446.061909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23446.067648] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23446.074773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23446.081897] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23446.089019] Call Trace: [23446.091467] [23446.093476] ? watchdog_timer_fn+0x1b4/0x220 [23446.097740] ? __hrtimer_run_queues+0x1d3/0x360 [23446.102274] ? hrtimer_interrupt+0x104/0x240 [23446.106542] ? __local_bh_enable+0x50/0x60 [23446.110639] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23446.115861] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23446.120820] [23446.122917] [23446.125013] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23446.130322] ? queued_spin_lock_slowpath+0x18/0x2e0 [23446.135198] do_raw_spin_lock+0xb5/0xc0 [23446.139027] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23446.143831] try_get_cap_refs+0x159/0x880 [ceph] [23446.148463] __ceph_get_caps+0x4a1/0x760 [ceph] [23446.153004] ? __file_remove_privs+0x89/0x110 [23446.157360] ? wake_bit_function+0x70/0x70 [23446.161461] ? ceph_write_iter+0x260/0x770 [ceph] [23446.166173] ceph_write_iter+0x260/0x770 [ceph] [23446.170710] ? lock_is_held_type+0xaa/0x120 [23446.174893] vfs_write+0x324/0x4f0 [23446.178303] ksys_pwrite64+0x65/0xa0 [23446.181877] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23446.187019] do_syscall_64+0x38/0x80 [23446.190595] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23446.195641] RIP: 0033:0x7f48f0eeb93b [23446.199210] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23446.217948] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23446.225506] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23446.232647] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23446.239770] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23446.246894] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23446.254020] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23446.261148] [23449.880920] watchdog: BUG: soft lockup - CPU#7 stuck for 19722s! [kworker/7:4:78808] [23449.888669] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23449.944111] irq event stamp: 109872678 [23449.947856] hardirqs last enabled at (109872677): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23449.958109] hardirqs last disabled at (109872678): [] sysvec_apic_timer_interrupt+0xe/0x90 [23449.967919] softirqs last enabled at (109872676): [] __do_softirq+0x2be/0x371 [23449.976689] softirqs last disabled at (109872671): [] irq_exit_rcu+0xb0/0xd0 [23449.985288] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23449.994682] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23450.002153] Workqueue: events delayed_work [ceph] [23450.006869] RIP: 0010:lock_acquire+0xf0/0x2b0 [23450.011218] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23450.029955] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23450.035175] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23450.042315] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23450.049438] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23450.056562] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23450.063686] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23450.070811] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23450.078889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23450.084626] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23450.091750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23450.098876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23450.105999] Call Trace: [23450.108444] [23450.110454] ? watchdog_timer_fn+0x1b4/0x220 [23450.114719] ? __hrtimer_run_queues+0x1d3/0x360 [23450.119246] ? hrtimer_interrupt+0x104/0x240 [23450.123515] ? __local_bh_enable+0x50/0x60 [23450.127608] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23450.132833] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23450.137790] [23450.139885] [23450.141983] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23450.147294] ? lock_acquire+0xf0/0x2b0 [23450.151061] ? igrab+0x43/0x50 [23450.154119] ? lock_release+0xc5/0x260 [23450.157862] _raw_spin_lock+0x30/0x40 [23450.161527] ? igrab+0x1d/0x50 [23450.164578] igrab+0x1d/0x50 [23450.167458] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23450.172606] delayed_work+0x1d8/0x280 [ceph] [23450.176888] ? process_scheduled_works+0x22a/0x510 [23450.181678] process_scheduled_works+0x2a4/0x510 [23450.186297] worker_thread+0x188/0x340 [23450.190059] ? rescuer_thread+0x310/0x310 [23450.194063] kthread+0xf5/0x130 [23450.197202] ? kthread_complete_and_exit+0x20/0x20 [23450.201994] ret_from_fork+0x31/0x40 [23450.205574] ? kthread_complete_and_exit+0x20/0x20 [23450.210355] ret_from_fork_asm+0x11/0x20 [23450.214279] [23473.869327] watchdog: BUG: soft lockup - CPU#0 stuck for 10747s! [dbench:78683] [23473.876624] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23473.932066] irq event stamp: 36830362 [23473.935722] hardirqs last enabled at (36830361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23473.945880] hardirqs last disabled at (36830362): [] sysvec_apic_timer_interrupt+0xe/0x90 [23473.955606] softirqs last enabled at (36830360): [] __do_softirq+0x2be/0x371 [23473.964288] softirqs last disabled at (36830355): [] irq_exit_rcu+0xb0/0xd0 [23473.972802] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23473.981745] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23473.989215] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23473.994692] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23474.013430] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23474.018647] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23474.025771] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23474.032896] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23474.040021] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23474.047144] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23474.054269] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23474.062346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23474.068084] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23474.075206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23474.082331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23474.089455] Call Trace: [23474.091899] [23474.093911] ? watchdog_timer_fn+0x1b4/0x220 [23474.098175] ? __hrtimer_run_queues+0x1d3/0x360 [23474.102721] ? hrtimer_interrupt+0x104/0x240 [23474.106988] ? __local_bh_enable+0x50/0x60 [23474.111099] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23474.116324] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23474.121281] [23474.123379] [23474.125476] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23474.130784] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23474.135660] do_raw_spin_lock+0xb5/0xc0 [23474.139491] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23474.144293] try_get_cap_refs+0x159/0x880 [ceph] [23474.148922] __ceph_get_caps+0x4a1/0x760 [ceph] [23474.153465] ? __file_remove_privs+0x89/0x110 [23474.157819] ? wake_bit_function+0x70/0x70 [23474.161914] ? ceph_write_iter+0x260/0x770 [ceph] [23474.166626] ceph_write_iter+0x260/0x770 [ceph] [23474.171161] ? lock_is_held_type+0xaa/0x120 [23474.175347] vfs_write+0x324/0x4f0 [23474.178754] ksys_pwrite64+0x65/0xa0 [23474.182329] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23474.187461] do_syscall_64+0x38/0x80 [23474.191031] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23474.196076] RIP: 0033:0x7f48f0eeb93b [23474.199645] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23474.218382] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23474.225941] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23474.233065] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23474.240188] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23474.247312] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23474.254438] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23474.261567] [23477.881395] watchdog: BUG: soft lockup - CPU#7 stuck for 19748s! [kworker/7:4:78808] [23477.889150] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23477.944591] irq event stamp: 110097052 [23477.948335] hardirqs last enabled at (110097051): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23477.958588] hardirqs last disabled at (110097052): [] sysvec_apic_timer_interrupt+0xe/0x90 [23477.968398] softirqs last enabled at (110097050): [] __do_softirq+0x2be/0x371 [23477.977169] softirqs last disabled at (110097045): [] irq_exit_rcu+0xb0/0xd0 [23477.985767] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23477.995161] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23478.002631] Workqueue: events delayed_work [ceph] [23478.007346] RIP: 0010:ceph_check_delayed_caps+0x93/0x270 [ceph] [23478.013266] Code: 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e <48> 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 [23478.032003] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [23478.037221] RAX: 0000000000000001 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [23478.044346] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [23478.051469] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [23478.058594] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23478.065717] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23478.072842] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23478.080919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23478.086655] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23478.093791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23478.100914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23478.108036] Call Trace: [23478.110482] [23478.112492] ? watchdog_timer_fn+0x1b4/0x220 [23478.116757] ? __hrtimer_run_queues+0x1d3/0x360 [23478.121284] ? hrtimer_interrupt+0x104/0x240 [23478.125553] ? __local_bh_enable+0x50/0x60 [23478.129647] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23478.134873] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23478.139827] [23478.141925] [23478.144023] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23478.149332] ? ceph_check_delayed_caps+0x93/0x270 [ceph] [23478.154648] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [23478.159963] delayed_work+0x1d8/0x280 [ceph] [23478.164244] ? process_scheduled_works+0x22a/0x510 [23478.169034] process_scheduled_works+0x2a4/0x510 [23478.173650] worker_thread+0x188/0x340 [23478.177399] ? rescuer_thread+0x310/0x310 [23478.181402] kthread+0xf5/0x130 [23478.184540] ? kthread_complete_and_exit+0x20/0x20 [23478.189325] ret_from_fork+0x31/0x40 [23478.192894] ? kthread_complete_and_exit+0x20/0x20 [23478.197680] ret_from_fork_asm+0x11/0x20 [23478.201603] [23501.869803] watchdog: BUG: soft lockup - CPU#0 stuck for 10773s! [dbench:78683] [23501.877104] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23501.932553] irq event stamp: 36889034 [23501.936211] hardirqs last enabled at (36889033): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23501.946377] hardirqs last disabled at (36889034): [] sysvec_apic_timer_interrupt+0xe/0x90 [23501.956101] softirqs last enabled at (36889032): [] __do_softirq+0x2be/0x371 [23501.964785] softirqs last disabled at (36889027): [] irq_exit_rcu+0xb0/0xd0 [23501.973306] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23501.982248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23501.989721] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23501.995199] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23502.013936] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23502.019152] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23502.026276] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23502.033400] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23502.040524] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23502.047648] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23502.054773] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23502.062852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23502.068587] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23502.075713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23502.082835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23502.089961] Call Trace: [23502.092406] [23502.094417] ? watchdog_timer_fn+0x1b4/0x220 [23502.098689] ? __hrtimer_run_queues+0x1d3/0x360 [23502.103225] ? hrtimer_interrupt+0x104/0x240 [23502.107492] ? __local_bh_enable+0x50/0x60 [23502.111587] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23502.116812] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23502.121768] [23502.123867] [23502.125964] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23502.131271] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23502.136149] do_raw_spin_lock+0xb5/0xc0 [23502.139987] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23502.144799] try_get_cap_refs+0x159/0x880 [ceph] [23502.149429] __ceph_get_caps+0x4a1/0x760 [ceph] [23502.153969] ? __file_remove_privs+0x89/0x110 [23502.158325] ? wake_bit_function+0x70/0x70 [23502.162426] ? ceph_write_iter+0x260/0x770 [ceph] [23502.167140] ceph_write_iter+0x260/0x770 [ceph] [23502.171675] ? lock_is_held_type+0xaa/0x120 [23502.175859] vfs_write+0x324/0x4f0 [23502.179269] ksys_pwrite64+0x65/0xa0 [23502.182843] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23502.187982] do_syscall_64+0x38/0x80 [23502.191555] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23502.196598] RIP: 0033:0x7f48f0eeb93b [23502.200169] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23502.218905] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23502.226464] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23502.233586] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23502.240711] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23502.247834] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23502.254961] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23502.262090] [23505.881870] watchdog: BUG: soft lockup - CPU#7 stuck for 19774s! [kworker/7:4:78808] [23505.889603] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23505.945051] irq event stamp: 110321534 [23505.948795] hardirqs last enabled at (110321533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23505.959039] hardirqs last disabled at (110321534): [] sysvec_apic_timer_interrupt+0xe/0x90 [23505.968851] softirqs last enabled at (110321532): [] __do_softirq+0x2be/0x371 [23505.977631] softirqs last disabled at (110321527): [] irq_exit_rcu+0xb0/0xd0 [23505.986229] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23505.995604] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23506.003077] Workqueue: events delayed_work [ceph] [23506.007791] RIP: 0010:lock_acquire+0xf0/0x2b0 [23506.012141] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23506.030878] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23506.036097] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23506.043220] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23506.050344] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23506.057469] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23506.064594] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23506.071718] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23506.079796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23506.085548] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23506.092674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23506.099796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23506.106921] Call Trace: [23506.109366] [23506.111377] ? watchdog_timer_fn+0x1b4/0x220 [23506.115640] ? __hrtimer_run_queues+0x1d3/0x360 [23506.120168] ? hrtimer_interrupt+0x104/0x240 [23506.124428] ? __local_bh_enable+0x50/0x60 [23506.128522] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23506.133746] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23506.138704] [23506.140801] [23506.142897] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23506.148207] ? lock_acquire+0xf0/0x2b0 [23506.151957] ? igrab+0x43/0x50 [23506.155007] ? lock_release+0xc5/0x260 [23506.158752] _raw_spin_lock+0x30/0x40 [23506.162416] ? igrab+0x1d/0x50 [23506.165465] igrab+0x1d/0x50 [23506.168345] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23506.173496] delayed_work+0x1d8/0x280 [ceph] [23506.177774] ? process_scheduled_works+0x22a/0x510 [23506.182564] process_scheduled_works+0x2a4/0x510 [23506.187182] worker_thread+0x188/0x340 [23506.190931] ? rescuer_thread+0x310/0x310 [23506.194933] kthread+0xf5/0x130 [23506.198073] ? kthread_complete_and_exit+0x20/0x20 [23506.202864] ret_from_fork+0x31/0x40 [23506.206445] ? kthread_complete_and_exit+0x20/0x20 [23506.211237] ret_from_fork_asm+0x11/0x20 [23506.215187] [23529.870277] watchdog: BUG: soft lockup - CPU#0 stuck for 10799s! [dbench:78683] [23529.877574] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23529.933016] irq event stamp: 36947756 [23529.936672] hardirqs last enabled at (36947755): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23529.946831] hardirqs last disabled at (36947756): [] sysvec_apic_timer_interrupt+0xe/0x90 [23529.956554] softirqs last enabled at (36947754): [] __do_softirq+0x2be/0x371 [23529.965240] softirqs last disabled at (36947749): [] irq_exit_rcu+0xb0/0xd0 [23529.973748] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23529.982692] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23529.990165] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23529.995642] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23530.014380] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23530.019596] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23530.026721] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23530.033845] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23530.040970] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23530.048093] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23530.055218] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23530.063295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23530.069032] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23530.076157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23530.083281] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23530.090405] Call Trace: [23530.092849] [23530.094860] ? watchdog_timer_fn+0x1b4/0x220 [23530.099125] ? __hrtimer_run_queues+0x1d3/0x360 [23530.103670] ? hrtimer_interrupt+0x104/0x240 [23530.107938] ? __local_bh_enable+0x50/0x60 [23530.112049] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23530.117272] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23530.122230] [23530.124325] [23530.126424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23530.131733] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23530.136609] do_raw_spin_lock+0xb5/0xc0 [23530.140440] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23530.145242] try_get_cap_refs+0x159/0x880 [ceph] [23530.149873] __ceph_get_caps+0x4a1/0x760 [ceph] [23530.154412] ? __file_remove_privs+0x89/0x110 [23530.158770] ? wake_bit_function+0x70/0x70 [23530.162861] ? ceph_write_iter+0x260/0x770 [ceph] [23530.167576] ceph_write_iter+0x260/0x770 [ceph] [23530.172110] ? lock_is_held_type+0xaa/0x120 [23530.176296] vfs_write+0x324/0x4f0 [23530.179705] ksys_pwrite64+0x65/0xa0 [23530.183278] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23530.188412] do_syscall_64+0x38/0x80 [23530.191989] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23530.197032] RIP: 0033:0x7f48f0eeb93b [23530.200602] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23530.219341] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23530.226899] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23530.234023] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23530.241147] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23530.248271] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23530.255396] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23530.262524] [23533.014696] rcu: INFO: rcu_preempt self-detected stall on CPU [23533.020452] rcu: 0-....: (11336940 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=5660898 [23533.030878] rcu: (t=11596198 jiffies g=1309609 q=373369 ncpus=8) [23533.036962] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23533.045905] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23533.053376] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23533.058854] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23533.077592] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23533.082810] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23533.089934] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23533.097057] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23533.104180] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23533.111306] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23533.118430] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23533.126506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23533.132245] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23533.139368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23533.146491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23533.153616] Call Trace: [23533.156060] [23533.158071] ? rcu_dump_cpu_stacks+0xd1/0x180 [23533.162432] ? rcu_sched_clock_irq+0x633/0x1390 [23533.166957] ? lock_is_held_type+0xaa/0x120 [23533.171132] ? lock_is_held_type+0xaa/0x120 [23533.175311] ? tick_sched_do_timer+0xa0/0xa0 [23533.179582] ? rcu_read_lock_sched_held+0x56/0x90 [23533.184281] ? tick_sched_do_timer+0xa0/0xa0 [23533.188544] ? update_process_times+0x5f/0x90 [23533.192895] ? tick_sched_handle.isra.16+0x1e/0x50 [23533.197679] ? tick_sched_timer+0x6f/0x80 [23533.201685] ? __hrtimer_run_queues+0x1d3/0x360 [23533.206211] ? hrtimer_interrupt+0x104/0x240 [23533.210480] ? __local_bh_enable+0x50/0x60 [23533.214572] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23533.219787] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23533.224745] [23533.226843] [23533.228941] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23533.234257] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23533.239133] do_raw_spin_lock+0xb5/0xc0 [23533.242964] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23533.247767] try_get_cap_refs+0x159/0x880 [ceph] [23533.252397] __ceph_get_caps+0x4a1/0x760 [ceph] [23533.256939] ? __file_remove_privs+0x89/0x110 [23533.261294] ? wake_bit_function+0x70/0x70 [23533.265387] ? ceph_write_iter+0x260/0x770 [ceph] [23533.270099] ceph_write_iter+0x260/0x770 [ceph] [23533.274634] ? lock_is_held_type+0xaa/0x120 [23533.278819] vfs_write+0x324/0x4f0 [23533.282219] ksys_pwrite64+0x65/0xa0 [23533.285794] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23533.290925] do_syscall_64+0x38/0x80 [23533.294497] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23533.299539] RIP: 0033:0x7f48f0eeb93b [23533.303110] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23533.321848] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23533.329405] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23533.336529] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23533.343655] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23533.350778] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23533.357903] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23533.365033] [23533.367218] Sending NMI from CPU 0 to CPUs 7: [23533.371572] NMI backtrace for cpu 7 [23533.371575] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23533.371577] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23533.371579] Workqueue: events delayed_work [ceph] [23533.371599] RIP: 0010:__lock_acquire+0x308/0x1e20 [23533.371602] Code: 89 e1 4c 8b 24 24 48 89 d0 48 89 ea 48 89 c5 eb 3d 48 8d 34 40 48 8d 04 b0 48 c1 e0 04 48 05 20 55 f7 82 0f b6 b0 c8 00 00 00 <40> 84 f6 74 12 41 38 f6 44 0f 47 f6 80 b8 ca 00 00 00 02 44 0f 44 [23533.371604] RSP: 0018:ffffc90002117c60 EFLAGS: 00000086 [23533.371605] RAX: ffffffff82f7fef0 RBX: ffff88810debaa80 RCX: 00000000000006f5 [23533.371607] RDX: ffff88810debb550 RSI: 0000000000000000 RDI: 0000000000000003 [23533.371608] RBP: ffffffff82f75120 R08: 0000000000000078 R09: 0000000000000000 [23533.371609] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810debb4d8 [23533.371610] R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000002 [23533.371611] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23533.371613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23533.371614] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23533.371616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23533.371617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23533.371618] Call Trace: [23533.371619] [23533.371620] ? nmi_cpu_backtrace+0xad/0x120 [23533.371625] ? nmi_cpu_backtrace_handler+0x11/0x20 [23533.371627] ? nmi_handle+0xd4/0x260 [23533.371632] ? default_do_nmi+0x49/0xf0 [23533.371634] ? exc_nmi+0x11e/0x150 [23533.371637] ? end_repeat_nmi+0x16/0x67 [23533.371642] ? __lock_acquire+0x308/0x1e20 [23533.371645] ? __lock_acquire+0x308/0x1e20 [23533.371648] ? __lock_acquire+0x308/0x1e20 [23533.371650] [23533.371650] [23533.371654] lock_acquire+0xcb/0x2b0 [23533.371656] ? igrab+0x1d/0x50 [23533.371658] ? igrab+0x43/0x50 [23533.371660] ? lock_release+0xc5/0x260 [23533.371663] _raw_spin_lock+0x30/0x40 [23533.371666] ? igrab+0x1d/0x50 [23533.371667] igrab+0x1d/0x50 [23533.371669] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23533.371689] delayed_work+0x1d8/0x280 [ceph] [23533.371725] ? process_scheduled_works+0x22a/0x510 [23533.371728] process_scheduled_works+0x2a4/0x510 [23533.371734] worker_thread+0x188/0x340 [23533.371738] ? rescuer_thread+0x310/0x310 [23533.371740] kthread+0xf5/0x130 [23533.371743] ? kthread_complete_and_exit+0x20/0x20 [23533.371747] ret_from_fork+0x31/0x40 [23533.371749] ? kthread_complete_and_exit+0x20/0x20 [23533.371752] ret_from_fork_asm+0x11/0x20 [23533.371759] [23557.870754] watchdog: BUG: soft lockup - CPU#0 stuck for 10825s! [dbench:78683] [23557.878063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23557.882753] watchdog: BUG: soft lockup - CPU#7 stuck for 19822s! [kworker/7:4:78808] [23557.933512] irq event stamp: 37005674 [23557.933513] hardirqs last enabled at (37005673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23557.941244] Modules linked in: [23557.944899] hardirqs last disabled at (37005674): [] sysvec_apic_timer_interrupt+0xe/0x90 [23557.955056] ceph [23557.958108] softirqs last enabled at (37005576): [] __do_softirq+0x2be/0x371 [23557.967831] libceph [23557.969755] softirqs last disabled at (37005571): [] irq_exit_rcu+0xb0/0xd0 [23557.978439] dns_resolver [23557.980624] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23557.989148] fscache [23557.991770] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23558.000713] netfs veth [23558.002898] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23558.010368] nft_chain_nat xt_MASQUERADE [23558.012813] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23558.018289] nf_nat [23558.022206] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23558.040946] nf_conntrack [23558.043043] [23558.043044] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23558.048261] nf_defrag_ipv6 [23558.050877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23558.052367] nf_defrag_ipv4 nft_compat [23558.059491] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23558.062283] nf_tables nfnetlink [23558.069406] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23558.073168] bridge stp [23558.080293] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23558.083517] llc xfs [23558.090640] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23558.093085] libcrc32c [23558.100209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23558.102394] sunrpc intel_rapl_msr [23558.110472] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23558.112829] intel_rapl_common [23558.118566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23558.121963] sb_edac x86_pkg_temp_thermal [23558.129088] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23558.132153] coretemp [23558.139280] Call Trace: [23558.139282] [23558.143283] kvm_intel kvm [23558.150409] ? watchdog_timer_fn+0x1b4/0x220 [23558.152696] irqbypass i2c_i801 [23558.155154] ? __hrtimer_run_queues+0x1d3/0x360 [23558.157168] iTCO_wdt [23558.159875] ? hrtimer_interrupt+0x104/0x240 [23558.164153] iTCO_vendor_support i2c_smbus [23558.167291] ? __local_bh_enable+0x50/0x60 [23558.171813] crct10dif_pclmul joydev [23558.174087] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23558.178348] crc32_pclmul ipmi_ssif [23558.182439] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23558.186530] ghash_clmulni_intel mei_me [23558.190100] [23558.195318] mei lpc_ich [23558.198803] [23558.203761] sha512_ssse3 acpi_ipmi [23558.207591] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23558.209688] mfd_core wmi [23558.212222] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23558.214315] ipmi_si ipmi_devintf [23558.217803] do_raw_spin_lock+0xb5/0xc0 [23558.223105] ipmi_msghandler [23558.225723] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23558.230593] ioatdma acpi_pad [23558.233905] try_get_cap_refs+0x159/0x880 [ceph] [23558.237735] acpi_power_meter ixgbe [23558.240616] __ceph_get_caps+0x4a1/0x760 [ceph] [23558.245397] igb [23558.248363] ? __file_remove_privs+0x89/0x110 [23558.252971] mdio crc32c_intel [23558.256456] ? wake_bit_function+0x70/0x70 [23558.260978] nvme ptp [23558.262817] ? ceph_write_iter+0x260/0x770 [ceph] [23558.267167] nvme_core [23558.270217] ceph_write_iter+0x260/0x770 [ceph] [23558.274308] i2c_algo_bit [23558.276583] ? lock_is_held_type+0xaa/0x120 [23558.281278] pps_core [23558.283636] vfs_write+0x324/0x4f0 [23558.288156] dca [23558.290780] ksys_pwrite64+0x65/0xa0 [23558.294953] fuse [23558.297225] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23558.300620] [23558.302459] do_syscall_64+0x38/0x80 [23558.306028] irq event stamp: 110738620 [23558.307956] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23558.313084] hardirqs last enabled at (110738619): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23558.314577] RIP: 0033:0x7f48f0eeb93b [23558.318154] hardirqs last disabled at (110738620): [] sysvec_apic_timer_interrupt+0xe/0x90 [23558.321899] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23558.326943] softirqs last enabled at (110738618): [] __do_softirq+0x2be/0x371 [23558.337185] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23558.340757] softirqs last disabled at (110738613): [] irq_exit_rcu+0xb0/0xd0 [23558.350566] ORIG_RAX: 0000000000000012 [23558.350567] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23558.369304] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23558.378077] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23558.383294] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23558.391892] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23558.395723] Workqueue: events delayed_work [ceph] [23558.402846] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23558.402848] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23558.412223] [23558.412224] RIP: 0010:lock_acquire+0xf0/0x2b0 [23558.419352] [23558.426818] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23558.479651] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23558.484869] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23558.491994] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23558.499117] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23558.506242] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23558.513365] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23558.520490] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23558.528568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23558.534305] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23558.541444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23558.548569] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23558.555695] Call Trace: [23558.558156] [23558.560167] ? watchdog_timer_fn+0x1b4/0x220 [23558.564439] ? __hrtimer_run_queues+0x1d3/0x360 [23558.568968] ? hrtimer_interrupt+0x104/0x240 [23558.573236] ? __local_bh_enable+0x50/0x60 [23558.577329] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23558.582553] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23558.587511] [23558.589609] [23558.591706] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23558.597013] ? lock_acquire+0xf0/0x2b0 [23558.600765] ? igrab+0x43/0x50 [23558.603822] ? lock_release+0xc5/0x260 [23558.607568] _raw_spin_lock+0x30/0x40 [23558.611232] ? igrab+0x1d/0x50 [23558.614282] igrab+0x1d/0x50 [23558.617164] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23558.622310] delayed_work+0x1d8/0x280 [ceph] [23558.626592] ? process_scheduled_works+0x22a/0x510 [23558.631383] process_scheduled_works+0x2a4/0x510 [23558.635999] worker_thread+0x188/0x340 [23558.639746] ? rescuer_thread+0x310/0x310 [23558.643750] kthread+0xf5/0x130 [23558.646897] ? kthread_complete_and_exit+0x20/0x20 [23558.651680] ret_from_fork+0x31/0x40 [23558.655260] ? kthread_complete_and_exit+0x20/0x20 [23558.660044] ret_from_fork_asm+0x11/0x20 [23558.663975] [23585.871233] watchdog: BUG: soft lockup - CPU#0 stuck for 10852s! [dbench:78683] [23585.878531] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23585.883233] watchdog: BUG: soft lockup - CPU#7 stuck for 19849s! [kworker/7:4:78808] [23585.933973] irq event stamp: 37064000 [23585.933974] hardirqs last enabled at (37063999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23585.941704] Modules linked in: [23585.945360] hardirqs last disabled at (37064000): [] sysvec_apic_timer_interrupt+0xe/0x90 [23585.955520] ceph [23585.958569] softirqs last enabled at (37063944): [] __do_softirq+0x2be/0x371 [23585.968293] libceph [23585.970217] softirqs last disabled at (37063939): [] irq_exit_rcu+0xb0/0xd0 [23585.978916] dns_resolver fscache [23585.981102] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23585.989627] netfs [23585.992941] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23585.992942] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23586.001903] veth [23586.003915] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23586.011386] nft_chain_nat xt_MASQUERADE [23586.016862] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23586.018787] nf_nat nf_conntrack [23586.037525] [23586.037526] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23586.041440] nf_defrag_ipv6 nf_defrag_ipv4 [23586.046660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23586.049883] nft_compat nf_tables [23586.051375] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23586.058497] nfnetlink bridge [23586.062589] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23586.069714] stp llc [23586.073023] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23586.080148] xfs libcrc32c [23586.083112] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23586.090235] sunrpc intel_rapl_msr [23586.092419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23586.099544] intel_rapl_common [23586.102248] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23586.110325] sb_edac x86_pkg_temp_thermal [23586.113723] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23586.119461] coretemp kvm_intel [23586.122510] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23586.129636] kvm irqbypass [23586.133640] Call Trace: [23586.140764] i2c_i801 [23586.143910] [23586.151033] iTCO_wdt iTCO_vendor_support [23586.153739] ? watchdog_timer_fn+0x1b4/0x220 [23586.156181] i2c_smbus crct10dif_pclmul [23586.158454] ? __hrtimer_run_queues+0x1d3/0x360 [23586.160462] joydev crc32_pclmul [23586.164471] ? hrtimer_interrupt+0x104/0x240 [23586.168731] ipmi_ssif ghash_clmulni_intel [23586.172561] ? __local_bh_enable+0x50/0x60 [23586.177086] mei_me [23586.180313] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23586.184575] mei lpc_ich [23586.188667] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23586.192757] sha512_ssse3 acpi_ipmi [23586.194852] [23586.200070] mfd_core wmi [23586.202601] [23586.202602] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23586.207558] ipmi_si ipmi_devintf ipmi_msghandler [23586.211046] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23586.213140] ioatdma acpi_pad [23586.215759] do_raw_spin_lock+0xb5/0xc0 [23586.217854] acpi_power_meter ixgbe [23586.223160] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23586.227857] igb mdio crc32c_intel [23586.232731] try_get_cap_refs+0x159/0x880 [ceph] [23586.235692] nvme [23586.239526] __ceph_get_caps+0x4a1/0x760 [ceph] [23586.243006] ptp nvme_core [23586.247794] ? __file_remove_privs+0x89/0x110 [23586.251186] i2c_algo_bit pps_core [23586.255799] ? wake_bit_function+0x70/0x70 [23586.257721] dca fuse [23586.262249] ? ceph_write_iter+0x260/0x770 [ceph] [23586.264950] [23586.269301] ceph_write_iter+0x260/0x770 [ceph] [23586.272698] irq event stamp: 110959406 [23586.276794] ? lock_is_held_type+0xaa/0x120 [23586.279060] hardirqs last enabled at (110959405): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23586.283760] vfs_write+0x324/0x4f0 [23586.285248] hardirqs last disabled at (110959406): [] sysvec_apic_timer_interrupt+0xe/0x90 [23586.289776] ksys_pwrite64+0x65/0xa0 [23586.293517] softirqs last enabled at (110959404): [] __do_softirq+0x2be/0x371 [23586.297694] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23586.307937] softirqs last disabled at (110959395): [] irq_exit_rcu+0xb0/0xd0 [23586.311336] do_syscall_64+0x38/0x80 [23586.321147] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23586.324718] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23586.333488] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23586.333489] Workqueue: events delayed_work [ceph] [23586.338634] RIP: 0033:0x7f48f0eeb93b [23586.347233] [23586.347234] RIP: 0010:igrab+0x12/0x50 [23586.350803] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23586.360181] Code: 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 <48> 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff [23586.365226] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23586.372696] RSP: 0018:ffffc90002117da0 EFLAGS: 00000286 [23586.377393] ORIG_RAX: 0000000000000012 [23586.380964] [23586.380965] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [23586.382455] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23586.386113] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [23586.404851] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23586.423587] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [23586.428806] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23586.434022] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [23586.437854] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23586.439345] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [23586.446469] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23586.453593] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23586.460723] [23586.467840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23586.526586] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23586.533709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23586.540833] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23586.547956] Call Trace: [23586.550402] [23586.552413] ? watchdog_timer_fn+0x1b4/0x220 [23586.556677] ? __hrtimer_run_queues+0x1d3/0x360 [23586.561204] ? hrtimer_interrupt+0x104/0x240 [23586.565472] ? __local_bh_enable+0x50/0x60 [23586.569566] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23586.574791] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23586.579748] [23586.581845] [23586.583943] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23586.589250] ? igrab+0x12/0x50 [23586.592307] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23586.597457] delayed_work+0x1d8/0x280 [ceph] [23586.601737] ? process_scheduled_works+0x22a/0x510 [23586.606526] process_scheduled_works+0x2a4/0x510 [23586.611143] worker_thread+0x188/0x340 [23586.614893] ? rescuer_thread+0x310/0x310 [23586.618918] kthread+0xf5/0x130 [23586.622058] ? kthread_complete_and_exit+0x20/0x20 [23586.626843] ret_from_fork+0x31/0x40 [23586.630414] ? kthread_complete_and_exit+0x20/0x20 [23586.635198] ret_from_fork_asm+0x11/0x20 [23586.639120] [23613.871713] watchdog: BUG: soft lockup - CPU#0 stuck for 10878s! [dbench:78683] [23613.879021] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23613.883712] watchdog: BUG: soft lockup - CPU#7 stuck for 19875s! [kworker/7:4:78808] [23613.934469] irq event stamp: 37122216 [23613.942201] Modules linked in: ceph libceph dns_resolver fscache netfs veth [23613.945858] hardirqs last enabled at (37122215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23613.952817] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [23613.962975] hardirqs last disabled at (37122216): [] sysvec_apic_timer_interrupt+0xe/0x90 [23613.962978] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [23613.978359] softirqs last enabled at (37122110): [] __do_softirq+0x2be/0x371 [23613.986445] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [23613.995128] softirqs last disabled at (37122105): [] irq_exit_rcu+0xb0/0xd0 [23614.003822] coretemp kvm_intel kvm [23614.012333] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23614.015816] irqbypass i2c_i801 [23614.024761] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23614.024763] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23614.027897] iTCO_wdt iTCO_vendor_support [23614.035385] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23614.040864] i2c_smbus crct10dif_pclmul [23614.044867] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23614.063606] joydev crc32_pclmul [23614.067437] [23614.067438] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23614.072652] ipmi_ssif ghash_clmulni_intel [23614.075877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23614.077368] mei_me mei [23614.084508] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23614.088600] lpc_ich sha512_ssse3 [23614.095725] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23614.098168] acpi_ipmi mfd_core [23614.105293] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23614.108604] wmi ipmi_si [23614.115728] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23614.118865] ipmi_devintf ipmi_msghandler [23614.125989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23614.128519] ioatdma [23614.136598] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23614.140603] acpi_pad [23614.146340] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23614.148522] acpi_power_meter [23614.155647] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23614.157917] ixgbe igb [23614.165042] Call Trace: [23614.165045] [23614.168007] mdio [23614.175133] ? watchdog_timer_fn+0x1b4/0x220 [23614.177487] crc32c_intel nvme [23614.179933] ? __hrtimer_run_queues+0x1d3/0x360 [23614.181942] ptp nvme_core [23614.183869] ? hrtimer_interrupt+0x104/0x240 [23614.188130] i2c_algo_bit [23614.191180] ? __local_bh_enable+0x50/0x60 [23614.195706] pps_core dca [23614.198414] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23614.202673] fuse [23614.205292] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23614.209391] irq event stamp: 111180384 [23614.212008] [23614.217226] hardirqs last enabled at (111180383): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23614.219149] [23614.224110] hardirqs last disabled at (111180384): [] sysvec_apic_timer_interrupt+0xe/0x90 [23614.227852] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23614.229948] softirqs last enabled at (111180382): [] __do_softirq+0x2be/0x371 [23614.240195] ? queued_spin_lock_slowpath+0x18/0x2e0 [23614.242290] softirqs last disabled at (111180377): [] irq_exit_rcu+0xb0/0xd0 [23614.252113] do_raw_spin_lock+0xb5/0xc0 [23614.257413] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23614.266187] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23614.271053] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23614.279654] try_get_cap_refs+0x159/0x880 [ceph] [23614.283483] Workqueue: events delayed_work [ceph] [23614.292866] __ceph_get_caps+0x4a1/0x760 [ceph] [23614.297644] [23614.297645] RIP: 0010:lock_acquire+0xf0/0x2b0 [23614.305120] ? __file_remove_privs+0x89/0x110 [23614.309725] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23614.314424] ? wake_bit_function+0x70/0x70 [23614.318947] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23614.320440] ? ceph_write_iter+0x260/0x770 [ceph] [23614.324789] [23614.324789] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23614.329139] ceph_write_iter+0x260/0x770 [ceph] [23614.347877] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23614.351973] ? lock_is_held_type+0xaa/0x120 [23614.357185] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23614.361886] vfs_write+0x324/0x4f0 [23614.363374] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23614.370504] ksys_pwrite64+0x65/0xa0 [23614.375022] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23614.382146] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23614.386323] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23614.393448] do_syscall_64+0x38/0x80 [23614.396846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23614.403971] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23614.407540] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23614.414664] RIP: 0033:0x7f48f0eeb93b [23614.419795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23614.427873] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23614.431443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23614.437180] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23614.442224] Call Trace: [23614.442225] [23614.449348] ORIG_RAX: 0000000000000012 [23614.452919] ? watchdog_timer_fn+0x1b4/0x220 [23614.460043] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23614.478783] ? __hrtimer_run_queues+0x1d3/0x360 [23614.485904] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23614.491127] ? hrtimer_interrupt+0x104/0x240 [23614.493568] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23614.495576] ? __local_bh_enable+0x50/0x60 [23614.499408] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23614.503675] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23614.510795] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23614.515320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23614.522450] [23614.526708] [23614.566647] [23614.568745] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23614.574052] ? lock_acquire+0xf0/0x2b0 [23614.577802] ? igrab+0x43/0x50 [23614.580860] ? lock_release+0xc5/0x260 [23614.584606] _raw_spin_lock+0x30/0x40 [23614.588271] ? igrab+0x1d/0x50 [23614.591321] igrab+0x1d/0x50 [23614.594199] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23614.599350] delayed_work+0x1d8/0x280 [ceph] [23614.603629] ? process_scheduled_works+0x22a/0x510 [23614.608420] process_scheduled_works+0x2a4/0x510 [23614.613036] worker_thread+0x188/0x340 [23614.616785] ? rescuer_thread+0x310/0x310 [23614.620796] kthread+0xf5/0x130 [23614.623936] ? kthread_complete_and_exit+0x20/0x20 [23614.628727] ret_from_fork+0x31/0x40 [23614.632306] ? kthread_complete_and_exit+0x20/0x20 [23614.637089] ret_from_fork_asm+0x11/0x20 [23614.641039] [23641.872191] watchdog: BUG: soft lockup - CPU#0 stuck for 10904s! [dbench:78683] [23641.879490] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23641.884191] watchdog: BUG: soft lockup - CPU#7 stuck for 19901s! [kworker/7:4:78808] [23641.934930] irq event stamp: 37180338 [23641.934931] hardirqs last enabled at (37180337): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23641.942662] Modules linked in: [23641.946318] hardirqs last disabled at (37180338): [] sysvec_apic_timer_interrupt+0xe/0x90 [23641.956477] ceph [23641.959528] softirqs last enabled at (37180260): [] __do_softirq+0x2be/0x371 [23641.969251] libceph [23641.971177] softirqs last disabled at (37180255): [] irq_exit_rcu+0xb0/0xd0 [23641.979865] dns_resolver fscache [23641.982053] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23641.990564] netfs veth [23641.993893] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23642.002836] nft_chain_nat [23642.005279] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23642.012750] xt_MASQUERADE nf_nat [23642.015455] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23642.020932] nf_conntrack nf_defrag_ipv6 [23642.024242] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23642.042982] nf_defrag_ipv4 nft_compat [23642.046897] [23642.052116] nf_tables [23642.055876] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23642.057368] nfnetlink bridge [23642.059725] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23642.066865] stp llc [23642.069830] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23642.076954] xfs libcrc32c [23642.079138] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23642.086264] sunrpc [23642.088966] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23642.096091] intel_rapl_msr intel_rapl_common [23642.098189] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23642.105313] sb_edac x86_pkg_temp_thermal [23642.109663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23642.117742] coretemp kvm_intel [23642.121744] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23642.127482] kvm irqbypass [23642.130619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23642.137744] i2c_i801 [23642.140447] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23642.147589] iTCO_wdt [23642.149867] Call Trace: [23642.149868] [23642.156992] iTCO_vendor_support i2c_smbus [23642.159263] ? watchdog_timer_fn+0x1b4/0x220 [23642.161706] crct10dif_pclmul [23642.163720] ? __hrtimer_run_queues+0x1d3/0x360 [23642.167808] joydev crc32_pclmul [23642.172076] ? hrtimer_interrupt+0x104/0x240 [23642.175036] ipmi_ssif ghash_clmulni_intel [23642.179560] ? __local_bh_enable+0x50/0x60 [23642.182784] mei_me mei [23642.187052] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23642.191139] lpc_ich sha512_ssse3 [23642.195231] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23642.197675] acpi_ipmi mfd_core [23642.202893] [23642.206202] wmi ipmi_si [23642.211159] [23642.211161] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23642.214296] ipmi_devintf [23642.216400] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23642.218926] ipmi_msghandler ioatdma [23642.221026] do_raw_spin_lock+0xb5/0xc0 [23642.226328] acpi_pad acpi_power_meter [23642.228946] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23642.233816] ixgbe igb [23642.237389] try_get_cap_refs+0x159/0x880 [ceph] [23642.241216] mdio crc32c_intel [23642.244966] __ceph_get_caps+0x4a1/0x760 [ceph] [23642.249745] nvme ptp [23642.252106] ? __file_remove_privs+0x89/0x110 [23642.256712] nvme_core i2c_algo_bit [23642.259765] ? wake_bit_function+0x70/0x70 [23642.264289] pps_core dca [23642.266560] ? ceph_write_iter+0x260/0x770 [ceph] [23642.270910] fuse [23642.274394] ceph_write_iter+0x260/0x770 [ceph] [23642.278484] irq event stamp: 111401170 [23642.281104] ? lock_is_held_type+0xaa/0x120 [23642.285799] hardirqs last enabled at (111401169): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23642.287725] vfs_write+0x324/0x4f0 [23642.292247] hardirqs last disabled at (111401170): [] sysvec_apic_timer_interrupt+0xe/0x90 [23642.295996] ksys_pwrite64+0x65/0xa0 [23642.300168] softirqs last enabled at (111401168): [] __do_softirq+0x2be/0x371 [23642.310414] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23642.313811] softirqs last disabled at (111401163): [] irq_exit_rcu+0xb0/0xd0 [23642.323621] do_syscall_64+0x38/0x80 [23642.327192] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23642.335962] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23642.341093] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23642.349691] RIP: 0033:0x7f48f0eeb93b [23642.353262] Workqueue: events delayed_work [ceph] [23642.362640] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23642.367683] [23642.367684] RIP: 0010:lock_acquire+0xf0/0x2b0 [23642.375153] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23642.378726] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23642.383422] ORIG_RAX: 0000000000000012 [23642.402159] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23642.403651] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23642.408001] [23642.408002] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23642.413219] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23642.431956] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23642.435786] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23642.441004] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23642.448128] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23642.449618] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23642.456742] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23642.463884] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23642.471015] [23642.478133] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23642.524015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23642.529752] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23642.536883] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23642.544009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23642.551136] Call Trace: [23642.553595] [23642.555606] ? watchdog_timer_fn+0x1b4/0x220 [23642.559876] ? __hrtimer_run_queues+0x1d3/0x360 [23642.564406] ? hrtimer_interrupt+0x104/0x240 [23642.568677] ? __local_bh_enable+0x50/0x60 [23642.572770] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23642.577993] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23642.582950] [23642.585047] [23642.587145] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23642.592453] ? lock_acquire+0xf0/0x2b0 [23642.596203] ? igrab+0x43/0x50 [23642.599253] ? lock_release+0xc5/0x260 [23642.602997] _raw_spin_lock+0x30/0x40 [23642.606655] ? igrab+0x1d/0x50 [23642.609704] igrab+0x1d/0x50 [23642.612597] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23642.617742] delayed_work+0x1d8/0x280 [ceph] [23642.622022] ? process_scheduled_works+0x22a/0x510 [23642.626814] process_scheduled_works+0x2a4/0x510 [23642.631428] worker_thread+0x188/0x340 [23642.635178] ? rescuer_thread+0x310/0x310 [23642.639180] kthread+0xf5/0x130 [23642.642319] ? kthread_complete_and_exit+0x20/0x20 [23642.647102] ret_from_fork+0x31/0x40 [23642.650673] ? kthread_complete_and_exit+0x20/0x20 [23642.655458] ret_from_fork_asm+0x11/0x20 [23642.659381] [23669.872670] watchdog: BUG: soft lockup - CPU#0 stuck for 10930s! [dbench:78683] [23669.879979] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23669.884669] watchdog: BUG: soft lockup - CPU#7 stuck for 19927s! [kworker/7:4:78808] [23669.935428] irq event stamp: 37238612 [23669.943159] Modules linked in: ceph libceph dns_resolver fscache netfs [23669.946814] hardirqs last enabled at (37238611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23669.953342] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23669.963500] hardirqs last disabled at (37238612): [] sysvec_apic_timer_interrupt+0xe/0x90 [23669.970882] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [23669.980609] softirqs last enabled at (37238534): [] __do_softirq+0x2be/0x371 [23669.980612] stp llc xfs libcrc32c sunrpc intel_rapl_msr [23669.995385] softirqs last disabled at (37238529): [] irq_exit_rcu+0xb0/0xd0 [23669.995388] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [23670.009208] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23670.016418] kvm irqbypass [23670.025364] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23670.025365] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23670.028066] i2c_i801 iTCO_wdt [23670.035537] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23670.041015] iTCO_vendor_support i2c_smbus [23670.044066] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23670.062803] crct10dif_pclmul [23670.066894] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23670.072112] joydev crc32_pclmul [23670.075076] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23670.082200] ipmi_ssif ghash_clmulni_intel [23670.085423] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23670.092548] mei_me [23670.096640] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23670.103764] mei lpc_ich [23670.105861] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23670.112985] sha512_ssse3 acpi_ipmi [23670.115515] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23670.122640] mfd_core wmi [23670.126124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23670.134201] ipmi_si ipmi_devintf [23670.136819] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23670.142555] ipmi_msghandler [23670.145865] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23670.152991] ioatdma acpi_pad [23670.155868] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23670.162992] acpi_power_meter [23670.165956] Call Trace: [23670.165958] [23670.173080] ixgbe igb [23670.176065] ? watchdog_timer_fn+0x1b4/0x220 [23670.178506] mdio crc32c_intel [23670.180517] ? __hrtimer_run_queues+0x1d3/0x360 [23670.182872] nvme ptp [23670.187142] ? hrtimer_interrupt+0x104/0x240 [23670.190189] nvme_core [23670.194713] ? __local_bh_enable+0x50/0x60 [23670.196983] i2c_algo_bit pps_core [23670.201250] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23670.203604] dca fuse [23670.207695] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23670.211093] [23670.216309] [23670.218580] irq event stamp: 111621910 [23670.223538] [23670.225029] hardirqs last enabled at (111621909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23670.227126] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23670.230870] hardirqs last disabled at (111621910): [] sysvec_apic_timer_interrupt+0xe/0x90 [23670.232973] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23670.243212] softirqs last enabled at (111621908): [] __do_softirq+0x2be/0x371 [23670.248519] do_raw_spin_lock+0xb5/0xc0 [23670.258326] softirqs last disabled at (111621903): [] irq_exit_rcu+0xb0/0xd0 [23670.263216] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23670.271986] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23670.275820] try_get_cap_refs+0x159/0x880 [ceph] [23670.284414] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23670.289205] __ceph_get_caps+0x4a1/0x760 [ceph] [23670.298576] Workqueue: events delayed_work [ceph] [23670.303191] ? __file_remove_privs+0x89/0x110 [23670.310658] [23670.310659] RIP: 0010:lock_release+0x162/0x260 [23670.315182] ? wake_bit_function+0x70/0x70 [23670.319880] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [23670.324232] ? ceph_write_iter+0x260/0x770 [ceph] [23670.325720] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [23670.330158] ceph_write_iter+0x260/0x770 [ceph] [23670.334247] [23670.334248] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [23670.352991] ? lock_is_held_type+0xaa/0x120 [23670.357684] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [23670.362902] vfs_write+0x324/0x4f0 [23670.367424] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [23670.368922] ksys_pwrite64+0x65/0xa0 [23670.376039] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [23670.380218] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23670.387351] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [23670.390748] do_syscall_64+0x38/0x80 [23670.397872] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23670.401445] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23670.408567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23670.413698] RIP: 0033:0x7f48f0eeb93b [23670.420823] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23670.424394] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23670.432469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23670.437513] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23670.443252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23670.446821] ORIG_RAX: 0000000000000012 [23670.453946] Call Trace: [23670.453948] [23670.472683] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23670.479809] ? watchdog_timer_fn+0x1b4/0x220 [23670.485025] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23670.492151] ? __hrtimer_run_queues+0x1d3/0x360 [23670.495980] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23670.498428] ? hrtimer_interrupt+0x104/0x240 [23670.500435] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23670.507559] ? __local_bh_enable+0x50/0x60 [23670.511824] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23670.518950] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23670.523476] [23670.530595] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23670.565558] [23670.567654] [23670.569754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23670.575074] ? igrab+0x43/0x50 [23670.578129] ? lock_release+0x162/0x260 [23670.581966] ? lock_release+0xc5/0x260 [23670.585712] _raw_spin_unlock+0x1b/0x40 [23670.589548] igrab+0x43/0x50 [23670.592425] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23670.597577] delayed_work+0x1d8/0x280 [ceph] [23670.601857] ? process_scheduled_works+0x22a/0x510 [23670.606648] process_scheduled_works+0x2a4/0x510 [23670.611264] worker_thread+0x188/0x340 [23670.615012] ? rescuer_thread+0x310/0x310 [23670.619014] kthread+0xf5/0x130 [23670.622153] ? kthread_complete_and_exit+0x20/0x20 [23670.626937] ret_from_fork+0x31/0x40 [23670.630518] ? kthread_complete_and_exit+0x20/0x20 [23670.635301] ret_from_fork_asm+0x11/0x20 [23670.639231] [23697.873151] watchdog: BUG: soft lockup - CPU#0 stuck for 10956s! [dbench:78683] [23697.880456] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23697.885151] watchdog: BUG: soft lockup - CPU#7 stuck for 19953s! [kworker/7:4:78808] [23697.935897] irq event stamp: 37296692 [23697.935898] hardirqs last enabled at (37296691): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23697.943628] Modules linked in: [23697.947286] hardirqs last disabled at (37296692): [] sysvec_apic_timer_interrupt+0xe/0x90 [23697.957442] ceph [23697.960494] softirqs last enabled at (37296614): [] __do_softirq+0x2be/0x371 [23697.970218] libceph [23697.972141] softirqs last disabled at (37296609): [] irq_exit_rcu+0xb0/0xd0 [23697.980827] dns_resolver [23697.983011] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23697.991522] fscache netfs [23697.994156] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23698.003100] veth nft_chain_nat [23698.005803] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23698.013273] xt_MASQUERADE nf_nat [23698.016412] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23698.021888] nf_conntrack [23698.025199] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23698.043938] nf_defrag_ipv6 nf_defrag_ipv4 [23698.046554] [23698.046555] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23698.051773] nft_compat nf_tables [23698.055863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23698.057354] nfnetlink [23698.064479] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23698.067789] bridge stp [23698.074911] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23698.077270] llc xfs [23698.084393] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23698.086837] libcrc32c sunrpc [23698.093961] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23698.096146] intel_rapl_msr intel_rapl_common [23698.103271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23698.106236] sb_edac x86_pkg_temp_thermal [23698.114311] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23698.118662] coretemp kvm_intel [23698.124399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23698.128404] kvm irqbypass [23698.135547] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23698.138683] i2c_i801 iTCO_wdt [23698.145807] Call Trace: [23698.145808] [23698.148528] iTCO_vendor_support i2c_smbus [23698.155671] ? watchdog_timer_fn+0x1b4/0x220 [23698.158721] crct10dif_pclmul joydev [23698.161166] ? __hrtimer_run_queues+0x1d3/0x360 [23698.163176] crc32_pclmul ipmi_ssif [23698.167271] ? hrtimer_interrupt+0x104/0x240 [23698.171547] ghash_clmulni_intel mei_me [23698.175120] ? __local_bh_enable+0x50/0x60 [23698.179643] mei lpc_ich [23698.183130] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23698.187392] sha512_ssse3 acpi_ipmi [23698.191221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23698.195311] mfd_core wmi [23698.197842] [23698.203060] ipmi_si [23698.206548] [23698.206549] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23698.211503] ipmi_devintf ipmi_msghandler [23698.214124] ? queued_spin_lock_slowpath+0x16/0x2e0 [23698.216216] ioatdma acpi_pad [23698.218402] do_raw_spin_lock+0xb5/0xc0 [23698.220499] acpi_power_meter ixgbe [23698.225803] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23698.229806] igb mdio [23698.234681] try_get_cap_refs+0x159/0x880 [ceph] [23698.237642] crc32c_intel nvme [23698.241477] __ceph_get_caps+0x4a1/0x760 [ceph] [23698.244955] ptp nvme_core [23698.249744] ? __file_remove_privs+0x89/0x110 [23698.252009] i2c_algo_bit pps_core [23698.256623] ? wake_bit_function+0x70/0x70 [23698.259671] dca fuse [23698.264198] ? ceph_write_iter+0x260/0x770 [ceph] [23698.266901] [23698.266902] irq event stamp: 111842846 [23698.271252] ceph_write_iter+0x260/0x770 [ceph] [23698.274649] hardirqs last enabled at (111842845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23698.278743] ? lock_is_held_type+0xaa/0x120 [23698.281009] hardirqs last disabled at (111842846): [] sysvec_apic_timer_interrupt+0xe/0x90 [23698.285709] vfs_write+0x324/0x4f0 [23698.287197] softirqs last enabled at (111842844): [] __do_softirq+0x2be/0x371 [23698.290946] ksys_pwrite64+0x65/0xa0 [23698.295466] softirqs last disabled at (111842839): [] irq_exit_rcu+0xb0/0xd0 [23698.305710] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23698.309889] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23698.319699] do_syscall_64+0x38/0x80 [23698.323095] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23698.331866] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23698.335438] Workqueue: events delayed_work [ceph] [23698.344035] RIP: 0033:0x7f48f0eeb93b [23698.349166] [23698.349166] RIP: 0010:lock_acquire+0xf0/0x2b0 [23698.358552] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23698.362122] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23698.369594] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23698.374637] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23698.379334] ORIG_RAX: 0000000000000012 [23698.382906] [23698.382907] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23698.384397] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23698.388746] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23698.407486] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23698.426223] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23698.431441] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23698.436657] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23698.440488] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23698.441980] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23698.449102] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23698.456228] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23698.463356] [23698.470475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23698.529218] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23698.536344] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23698.543468] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23698.550592] Call Trace: [23698.553038] [23698.555048] ? watchdog_timer_fn+0x1b4/0x220 [23698.559311] ? __hrtimer_run_queues+0x1d3/0x360 [23698.563840] ? hrtimer_interrupt+0x104/0x240 [23698.568108] ? __local_bh_enable+0x50/0x60 [23698.572200] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23698.577417] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23698.582373] [23698.584470] [23698.586569] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23698.591877] ? lock_acquire+0xf0/0x2b0 [23698.595627] ? igrab+0x43/0x50 [23698.598675] ? lock_release+0xc5/0x260 [23698.602422] _raw_spin_lock+0x30/0x40 [23698.606087] ? igrab+0x1d/0x50 [23698.609137] igrab+0x1d/0x50 [23698.612016] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23698.617164] delayed_work+0x1d8/0x280 [ceph] [23698.621446] ? process_scheduled_works+0x22a/0x510 [23698.626237] process_scheduled_works+0x2a4/0x510 [23698.630851] worker_thread+0x188/0x340 [23698.634601] ? rescuer_thread+0x310/0x310 [23698.638603] kthread+0xf5/0x130 [23698.641742] ? kthread_complete_and_exit+0x20/0x20 [23698.646527] ret_from_fork+0x31/0x40 [23698.650104] ? kthread_complete_and_exit+0x20/0x20 [23698.654888] ret_from_fork_asm+0x11/0x20 [23698.658811] [23725.873634] watchdog: BUG: soft lockup - CPU#0 stuck for 10982s! [dbench:78683] [23725.880934] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23725.885634] watchdog: BUG: soft lockup - CPU#7 stuck for 19979s! [kworker/7:4:78808] [23725.936376] irq event stamp: 37354946 [23725.936377] hardirqs last enabled at (37354945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23725.944107] Modules linked in: [23725.947763] hardirqs last disabled at (37354946): [] sysvec_apic_timer_interrupt+0xe/0x90 [23725.957922] ceph [23725.960972] softirqs last enabled at (37354868): [] __do_softirq+0x2be/0x371 [23725.970696] libceph [23725.972619] softirqs last disabled at (37354863): [] irq_exit_rcu+0xb0/0xd0 [23725.981317] dns_resolver fscache [23725.983507] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23725.992027] netfs veth [23725.995337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23726.004280] nft_chain_nat [23726.006724] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23726.014196] xt_MASQUERADE nf_nat [23726.016898] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23726.022376] nf_conntrack nf_defrag_ipv6 [23726.025688] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23726.044426] nf_defrag_ipv4 nft_compat [23726.048341] [23726.048342] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23726.053560] nf_tables nfnetlink [23726.057310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23726.058802] bridge [23726.065928] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23726.069152] stp llc [23726.076276] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23726.078373] xfs libcrc32c [23726.085496] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23726.087680] sunrpc intel_rapl_msr [23726.094804] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23726.097508] intel_rapl_common sb_edac [23726.104633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23726.108032] x86_pkg_temp_thermal [23726.116107] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23726.119851] coretemp kvm_intel [23726.125589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23726.128899] kvm irqbypass [23726.136030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23726.139170] i2c_i801 iTCO_wdt [23726.146310] Call Trace: [23726.146311] [23726.149032] iTCO_vendor_support i2c_smbus [23726.156159] ? watchdog_timer_fn+0x1b4/0x220 [23726.159208] crct10dif_pclmul joydev [23726.161653] ? __hrtimer_run_queues+0x1d3/0x360 [23726.163663] crc32_pclmul ipmi_ssif [23726.167757] ? hrtimer_interrupt+0x104/0x240 [23726.172033] ghash_clmulni_intel [23726.175605] ? __local_bh_enable+0x50/0x60 [23726.180129] mei_me mei [23726.183616] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23726.187877] lpc_ich [23726.191103] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23726.195193] sha512_ssse3 acpi_ipmi [23726.197638] [23726.202853] mfd_core wmi [23726.205037] [23726.205039] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23726.209995] ipmi_si ipmi_devintf [23726.213485] ? queued_spin_lock_slowpath+0x18/0x2e0 [23726.215576] ipmi_msghandler ioatdma [23726.218197] do_raw_spin_lock+0xb5/0xc0 [23726.220291] acpi_pad acpi_power_meter [23726.225598] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23726.228906] ixgbe igb [23726.233780] try_get_cap_refs+0x159/0x880 [ceph] [23726.237348] mdio crc32c_intel [23726.241184] __ceph_get_caps+0x4a1/0x760 [ceph] [23726.244924] nvme ptp [23726.249712] ? __file_remove_privs+0x89/0x110 [23726.252064] nvme_core i2c_algo_bit [23726.256677] ? wake_bit_function+0x70/0x70 [23726.259725] pps_core dca [23726.264253] ? ceph_write_iter+0x260/0x770 [ceph] [23726.266521] fuse [23726.270873] ceph_write_iter+0x260/0x770 [ceph] [23726.274355] irq event stamp: 112063586 [23726.278451] ? lock_is_held_type+0xaa/0x120 [23726.281064] hardirqs last enabled at (112063585): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23726.285763] vfs_write+0x324/0x4f0 [23726.287685] hardirqs last disabled at (112063586): [] sysvec_apic_timer_interrupt+0xe/0x90 [23726.292213] ksys_pwrite64+0x65/0xa0 [23726.295954] softirqs last enabled at (112063584): [] __do_softirq+0x2be/0x371 [23726.300130] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23726.310376] softirqs last disabled at (112063579): [] irq_exit_rcu+0xb0/0xd0 [23726.313774] do_syscall_64+0x38/0x80 [23726.323583] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23726.327156] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23726.335926] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23726.341056] RIP: 0033:0x7f48f0eeb93b [23726.349652] Workqueue: events delayed_work [ceph] [23726.353224] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23726.362601] [23726.367646] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23726.375116] RIP: 0010:lock_acquire+0xf0/0x2b0 [23726.378687] ORIG_RAX: 0000000000000012 [23726.383383] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23726.402122] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23726.403613] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23726.408831] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23726.413182] [23726.413183] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23726.417011] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23726.435768] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23726.442891] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23726.448108] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23726.455232] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23726.456724] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23726.463853] [23726.470970] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23726.515900] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23726.523978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23726.529715] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23726.536840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23726.543963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23726.551087] Call Trace: [23726.553533] [23726.555542] ? watchdog_timer_fn+0x1b4/0x220 [23726.559808] ? __hrtimer_run_queues+0x1d3/0x360 [23726.564334] ? hrtimer_interrupt+0x104/0x240 [23726.568602] ? __local_bh_enable+0x50/0x60 [23726.572698] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23726.577923] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23726.582887] [23726.584983] [23726.587080] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23726.592389] ? lock_acquire+0xf0/0x2b0 [23726.596140] ? igrab+0x43/0x50 [23726.599199] ? lock_release+0xc5/0x260 [23726.602951] _raw_spin_lock+0x30/0x40 [23726.606617] ? igrab+0x1d/0x50 [23726.609667] igrab+0x1d/0x50 [23726.612547] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23726.617696] delayed_work+0x1d8/0x280 [ceph] [23726.621977] ? process_scheduled_works+0x22a/0x510 [23726.626768] process_scheduled_works+0x2a4/0x510 [23726.631382] worker_thread+0x188/0x340 [23726.635130] ? rescuer_thread+0x310/0x310 [23726.639133] kthread+0xf5/0x130 [23726.642273] ? kthread_complete_and_exit+0x20/0x20 [23726.647057] ret_from_fork+0x31/0x40 [23726.650636] ? kthread_complete_and_exit+0x20/0x20 [23726.655420] ret_from_fork_asm+0x11/0x20 [23726.659342] [23728.378043] rcu: INFO: rcu_preempt self-detected stall on CPU [23728.383802] rcu: 0-....: (11527637 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=5756715 [23728.394228] rcu: (t=11791558 jiffies g=1309609 q=378536 ncpus=8) [23728.400314] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23728.409263] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23728.416735] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23728.422212] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23728.440951] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23728.446168] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23728.453292] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23728.460415] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23728.467540] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23728.474664] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23728.481788] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23728.489866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23728.495604] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23728.502728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23728.509852] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23728.516975] Call Trace: [23728.519419] [23728.521432] ? rcu_dump_cpu_stacks+0xd1/0x180 [23728.525791] ? rcu_sched_clock_irq+0x633/0x1390 [23728.530323] ? lock_is_held_type+0xaa/0x120 [23728.534509] ? lock_is_held_type+0xaa/0x120 [23728.538688] ? tick_sched_do_timer+0xa0/0xa0 [23728.542959] ? rcu_read_lock_sched_held+0x56/0x90 [23728.547657] ? tick_sched_do_timer+0xa0/0xa0 [23728.551919] ? update_process_times+0x5f/0x90 [23728.556271] ? tick_sched_handle.isra.16+0x1e/0x50 [23728.561072] ? tick_sched_timer+0x6f/0x80 [23728.565078] ? __hrtimer_run_queues+0x1d3/0x360 [23728.569612] ? hrtimer_interrupt+0x104/0x240 [23728.573874] ? __local_bh_enable+0x50/0x60 [23728.577974] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23728.583200] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23728.588157] [23728.590254] [23728.592367] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23728.597675] ? queued_spin_lock_slowpath+0x18/0x2e0 [23728.602553] do_raw_spin_lock+0xb5/0xc0 [23728.606384] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23728.611188] try_get_cap_refs+0x159/0x880 [ceph] [23728.615817] __ceph_get_caps+0x4a1/0x760 [ceph] [23728.620375] ? __file_remove_privs+0x89/0x110 [23728.624730] ? wake_bit_function+0x70/0x70 [23728.628823] ? ceph_write_iter+0x260/0x770 [ceph] [23728.633528] ceph_write_iter+0x260/0x770 [ceph] [23728.638079] ? lock_is_held_type+0xaa/0x120 [23728.642257] vfs_write+0x324/0x4f0 [23728.645666] ksys_pwrite64+0x65/0xa0 [23728.649239] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23728.654371] do_syscall_64+0x38/0x80 [23728.657941] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23728.662986] RIP: 0033:0x7f48f0eeb93b [23728.666556] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23728.685293] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23728.692851] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23728.699974] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23728.707100] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23728.714223] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23728.721357] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23728.728487] [23728.730673] Sending NMI from CPU 0 to CPUs 7: [23728.735029] NMI backtrace for cpu 7 [23728.735030] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23728.735033] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23728.735034] Workqueue: events delayed_work [ceph] [23728.735071] RIP: 0010:mark_lock+0x6/0x4c0 [23728.735074] Code: c7 c6 74 49 14 82 48 c7 c7 8a 3f 12 82 e8 42 bf f7 ff 0f 0b 31 c0 e9 8b fe ff ff 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 <41> 56 41 55 41 54 53 48 83 e4 f0 48 83 ec 70 65 48 8b 04 25 28 00 [23728.735076] RSP: 0018:ffffc90002117c48 EFLAGS: 00000002 [23728.735078] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: 0000000000000000 [23728.735079] RDX: 0000000000000006 RSI: ffff88810debb550 RDI: ffff88810debaa80 [23728.735081] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [23728.735082] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [23728.735083] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [23728.735084] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23728.735086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23728.735088] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23728.735089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23728.735090] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23728.735091] Call Trace: [23728.735093] [23728.735094] ? nmi_cpu_backtrace+0xad/0x120 [23728.735098] ? nmi_cpu_backtrace_handler+0x11/0x20 [23728.735100] ? nmi_handle+0xd4/0x260 [23728.735106] ? default_do_nmi+0x49/0xf0 [23728.735109] ? exc_nmi+0x11e/0x150 [23728.735111] ? end_repeat_nmi+0x16/0x67 [23728.735117] ? mark_lock+0x6/0x4c0 [23728.735120] ? mark_lock+0x6/0x4c0 [23728.735122] ? mark_lock+0x6/0x4c0 [23728.735125] [23728.735125] [23728.735126] __lock_acquire+0x4be/0x1e20 [23728.735132] lock_acquire+0xcb/0x2b0 [23728.735134] ? igrab+0x1d/0x50 [23728.735136] ? igrab+0x43/0x50 [23728.735138] ? lock_release+0xc5/0x260 [23728.735141] _raw_spin_lock+0x30/0x40 [23728.735144] ? igrab+0x1d/0x50 [23728.735146] igrab+0x1d/0x50 [23728.735148] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23728.735168] delayed_work+0x1d8/0x280 [ceph] [23728.735188] ? process_scheduled_works+0x22a/0x510 [23728.735191] process_scheduled_works+0x2a4/0x510 [23728.735197] worker_thread+0x188/0x340 [23728.735201] ? rescuer_thread+0x310/0x310 [23728.735203] kthread+0xf5/0x130 [23728.735207] ? kthread_complete_and_exit+0x20/0x20 [23728.735210] ret_from_fork+0x31/0x40 [23728.735213] ? kthread_complete_and_exit+0x20/0x20 [23728.735216] ret_from_fork_asm+0x11/0x20 [23728.735222] [23753.874119] watchdog: BUG: soft lockup - CPU#0 stuck for 11008s! [dbench:78683] [23753.881421] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23753.886118] watchdog: BUG: soft lockup - CPU#7 stuck for 20005s! [kworker/7:4:78808] [23753.936872] irq event stamp: 37412420 [23753.944601] Modules linked in: ceph [23753.948259] hardirqs last enabled at (37412419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23753.948261] libceph dns_resolver fscache netfs [23753.961902] hardirqs last disabled at (37412420): [] sysvec_apic_timer_interrupt+0xe/0x90 [23753.961904] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [23753.976149] softirqs last enabled at (37412302): [] __do_softirq+0x2be/0x371 [23753.982234] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [23753.990917] softirqs last disabled at (37412297): [] irq_exit_rcu+0xb0/0xd0 [23753.990920] nf_tables nfnetlink bridge stp [23754.004472] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23754.008649] llc xfs [23754.017595] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23754.019793] libcrc32c sunrpc intel_rapl_msr [23754.027265] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23754.027267] intel_rapl_common sb_edac x86_pkg_temp_thermal [23754.037009] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23754.037010] coretemp [23754.042572] kvm_intel [23754.061309] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23754.063581] kvm irqbypass [23754.065936] [23754.065937] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23754.071156] i2c_i801 [23754.073858] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23754.075350] iTCO_wdt iTCO_vendor_support [23754.082474] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23754.084743] i2c_smbus crct10dif_pclmul [23754.091870] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23754.095872] joydev crc32_pclmul [23754.102998] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23754.106828] ipmi_ssif ghash_clmulni_intel [23754.113951] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23754.117177] mei_me mei [23754.124300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23754.128392] lpc_ich sha512_ssse3 [23754.136467] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23754.138911] acpi_ipmi mfd_core [23754.144650] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23754.147962] wmi ipmi_si [23754.155085] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23754.158221] ipmi_devintf [23754.165346] Call Trace: [23754.165347] [23754.167877] ipmi_msghandler [23754.175003] ? watchdog_timer_fn+0x1b4/0x220 [23754.177618] ioatdma acpi_pad [23754.180063] ? __hrtimer_run_queues+0x1d3/0x360 [23754.182073] acpi_power_meter [23754.184954] ? hrtimer_interrupt+0x104/0x240 [23754.189215] ixgbe [23754.192178] ? __local_bh_enable+0x50/0x60 [23754.196703] igb mdio [23754.199670] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23754.203930] crc32c_intel nvme [23754.205943] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23754.210032] ptp nvme_core [23754.212303] [23754.217520] i2c_algo_bit [23754.220571] [23754.225528] pps_core dca [23754.228235] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23754.230330] fuse [23754.232951] ? queued_spin_lock_slowpath+0x18/0x2e0 [23754.235045] irq event stamp: 112282466 [23754.237664] do_raw_spin_lock+0xb5/0xc0 [23754.242967] hardirqs last enabled at (112282465): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23754.244890] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23754.249761] hardirqs last disabled at (112282466): [] sysvec_apic_timer_interrupt+0xe/0x90 [23754.253521] try_get_cap_refs+0x159/0x880 [ceph] [23754.257352] softirqs last enabled at (112282464): [] __do_softirq+0x2be/0x371 [23754.267603] __ceph_get_caps+0x4a1/0x760 [ceph] [23754.272382] softirqs last disabled at (112282459): [] irq_exit_rcu+0xb0/0xd0 [23754.282197] ? __file_remove_privs+0x89/0x110 [23754.286802] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23754.295573] ? wake_bit_function+0x70/0x70 [23754.300099] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23754.308698] ? ceph_write_iter+0x260/0x770 [ceph] [23754.313047] Workqueue: events delayed_work [ceph] [23754.322423] ceph_write_iter+0x260/0x770 [ceph] [23754.326522] [23754.333998] ? lock_is_held_type+0xaa/0x120 [23754.338692] RIP: 0010:lock_acquire+0xf0/0x2b0 [23754.343392] vfs_write+0x324/0x4f0 [23754.347912] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23754.349407] ksys_pwrite64+0x65/0xa0 [23754.353580] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23754.357932] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23754.361329] [23754.361330] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23754.380067] do_syscall_64+0x38/0x80 [23754.383637] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23754.388856] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23754.393987] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23754.395478] RIP: 0033:0x7f48f0eeb93b [23754.402599] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23754.406171] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23754.413296] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23754.418338] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23754.425463] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23754.429035] ORIG_RAX: 0000000000000012 [23754.436159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23754.454897] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23754.462021] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23754.467237] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23754.475316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23754.479145] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23754.484883] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23754.492008] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23754.499131] Call Trace: [23754.506256] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23754.513380] [23754.520526] [23754.527645] ? watchdog_timer_fn+0x1b4/0x220 [23754.552814] ? __hrtimer_run_queues+0x1d3/0x360 [23754.557341] ? hrtimer_interrupt+0x104/0x240 [23754.561603] ? __local_bh_enable+0x50/0x60 [23754.565696] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23754.570919] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23754.575878] [23754.577974] [23754.580073] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23754.585380] ? lock_acquire+0xf0/0x2b0 [23754.589129] ? igrab+0x43/0x50 [23754.592180] ? lock_release+0xc5/0x260 [23754.595925] _raw_spin_lock+0x30/0x40 [23754.599580] ? igrab+0x1d/0x50 [23754.602633] igrab+0x1d/0x50 [23754.605527] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23754.610676] delayed_work+0x1d8/0x280 [ceph] [23754.614957] ? process_scheduled_works+0x22a/0x510 [23754.619748] process_scheduled_works+0x2a4/0x510 [23754.624363] worker_thread+0x188/0x340 [23754.628113] ? rescuer_thread+0x310/0x310 [23754.632117] kthread+0xf5/0x130 [23754.635253] ? kthread_complete_and_exit+0x20/0x20 [23754.640039] ret_from_fork+0x31/0x40 [23754.643607] ? kthread_complete_and_exit+0x20/0x20 [23754.648393] ret_from_fork_asm+0x11/0x20 [23754.652316] [23781.874604] watchdog: BUG: soft lockup - CPU#0 stuck for 11034s! [dbench:78683] [23781.881908] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23781.886604] watchdog: BUG: soft lockup - CPU#7 stuck for 20031s! [kworker/7:4:78808] [23781.937351] irq event stamp: 37470642 [23781.937352] hardirqs last enabled at (37470641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23781.945082] Modules linked in: [23781.948738] hardirqs last disabled at (37470642): [] sysvec_apic_timer_interrupt+0xe/0x90 [23781.958894] ceph [23781.961948] softirqs last enabled at (37470554): [] __do_softirq+0x2be/0x371 [23781.971672] libceph [23781.973596] softirqs last disabled at (37470549): [] irq_exit_rcu+0xb0/0xd0 [23781.982280] dns_resolver [23781.984462] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23781.992974] fscache netfs [23781.995607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23781.995608] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23782.004551] veth nft_chain_nat [23782.007257] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23782.014728] xt_MASQUERADE nf_nat [23782.020205] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23782.023342] nf_conntrack nf_defrag_ipv6 [23782.042080] [23782.042081] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23782.045390] nf_defrag_ipv4 nft_compat [23782.050607] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23782.054525] nf_tables nfnetlink [23782.056017] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23782.063140] bridge stp [23782.066883] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23782.074008] llc xfs [23782.077234] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23782.084357] libcrc32c sunrpc [23782.086800] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23782.093924] intel_rapl_msr [23782.096109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23782.103233] intel_rapl_common [23782.106196] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23782.114289] sb_edac x86_pkg_temp_thermal [23782.117084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23782.122819] coretemp kvm_intel [23782.125872] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23782.133002] kvm [23782.137008] Call Trace: [23782.137009] [23782.144130] irqbypass [23782.147282] ? watchdog_timer_fn+0x1b4/0x220 [23782.154403] i2c_i801 iTCO_wdt [23782.156242] ? __hrtimer_run_queues+0x1d3/0x360 [23782.158682] iTCO_vendor_support i2c_smbus [23782.160698] ? hrtimer_interrupt+0x104/0x240 [23782.163051] crct10dif_pclmul joydev [23782.167316] ? __local_bh_enable+0x50/0x60 [23782.170367] crc32_pclmul ipmi_ssif [23782.174893] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23782.178981] ghash_clmulni_intel [23782.183264] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23782.186850] mei_me mei [23782.190940] [23782.194425] lpc_ich sha512_ssse3 [23782.199642] [23782.199643] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23782.202867] acpi_ipmi mfd_core [23782.207828] ? queued_spin_lock_slowpath+0x18/0x2e0 [23782.210283] wmi ipmi_si [23782.212385] do_raw_spin_lock+0xb5/0xc0 [23782.215693] ipmi_devintf ipmi_msghandler [23782.217791] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23782.223096] ioatdma acpi_pad [23782.226235] try_get_cap_refs+0x159/0x880 [ceph] [23782.231104] acpi_power_meter ixgbe [23782.233637] __ceph_get_caps+0x4a1/0x760 [ceph] [23782.237465] igb mdio [23782.241472] ? __file_remove_privs+0x89/0x110 [23782.246253] crc32c_intel nvme [23782.249217] ? wake_bit_function+0x70/0x70 [23782.253829] ptp nvme_core [23782.257314] ? ceph_write_iter+0x260/0x770 [ceph] [23782.261836] i2c_algo_bit pps_core [23782.264107] ceph_write_iter+0x260/0x770 [ceph] [23782.268458] dca fuse [23782.271512] ? lock_is_held_type+0xaa/0x120 [23782.275598] [23782.275598] irq event stamp: 112503246 [23782.278305] vfs_write+0x324/0x4f0 [23782.283004] hardirqs last enabled at (112503245): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23782.286403] ksys_pwrite64+0x65/0xa0 [23782.290923] hardirqs last disabled at (112503246): [] sysvec_apic_timer_interrupt+0xe/0x90 [23782.293193] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23782.297369] softirqs last enabled at (112503244): [] __do_softirq+0x2be/0x371 [23782.298861] do_syscall_64+0x38/0x80 [23782.302606] softirqs last disabled at (112503239): [] irq_exit_rcu+0xb0/0xd0 [23782.306006] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23782.316247] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23782.319816] RIP: 0033:0x7f48f0eeb93b [23782.329627] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23782.334760] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23782.343531] Workqueue: events delayed_work [ceph] [23782.347099] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23782.355699] [23782.360741] ORIG_RAX: 0000000000000012 [23782.370120] RIP: 0010:lock_acquire+0xf0/0x2b0 [23782.373691] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23782.381161] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23782.399898] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23782.404597] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23782.409812] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23782.411305] [23782.411306] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23782.415133] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23782.419485] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23782.426610] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23782.445348] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23782.452477] [23782.457689] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23782.511233] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23782.518356] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23782.526434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23782.532173] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23782.539299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23782.546430] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23782.553552] Call Trace: [23782.556010] [23782.558025] ? watchdog_timer_fn+0x1b4/0x220 [23782.562302] ? __hrtimer_run_queues+0x1d3/0x360 [23782.566835] ? hrtimer_interrupt+0x104/0x240 [23782.571104] ? __local_bh_enable+0x50/0x60 [23782.575197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23782.580420] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23782.585377] [23782.587474] [23782.589573] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23782.594881] ? lock_acquire+0xf0/0x2b0 [23782.598630] ? igrab+0x43/0x50 [23782.601681] ? lock_release+0xc5/0x260 [23782.605425] _raw_spin_lock+0x30/0x40 [23782.609081] ? igrab+0x1d/0x50 [23782.612133] igrab+0x1d/0x50 [23782.615012] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23782.620161] delayed_work+0x1d8/0x280 [ceph] [23782.624441] ? process_scheduled_works+0x22a/0x510 [23782.629231] process_scheduled_works+0x2a4/0x510 [23782.633847] worker_thread+0x188/0x340 [23782.637597] ? rescuer_thread+0x310/0x310 [23782.641609] kthread+0xf5/0x130 [23782.644747] ? kthread_complete_and_exit+0x20/0x20 [23782.649531] ret_from_fork+0x31/0x40 [23782.653100] ? kthread_complete_and_exit+0x20/0x20 [23782.657884] ret_from_fork_asm+0x11/0x20 [23782.661808] [23809.875091] watchdog: BUG: soft lockup - CPU#0 stuck for 11060s! [dbench:78683] [23809.882396] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23809.887090] watchdog: BUG: soft lockup - CPU#7 stuck for 20057s! [kworker/7:4:78808] [23809.937844] irq event stamp: 37528906 [23809.945576] Modules linked in: ceph libceph dns_resolver fscache netfs [23809.949232] hardirqs last enabled at (37528905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23809.955764] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23809.965917] hardirqs last disabled at (37528906): [] sysvec_apic_timer_interrupt+0xe/0x90 [23809.973300] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [23809.983024] softirqs last enabled at (37528900): [] __do_softirq+0x2be/0x371 [23809.989456] llc xfs libcrc32c sunrpc intel_rapl_msr [23809.998140] softirqs last disabled at (37528895): [] irq_exit_rcu+0xb0/0xd0 [23810.003098] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [23810.011609] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23810.017952] kvm_intel kvm [23810.026897] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23810.026899] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23810.029600] irqbypass i2c_i801 [23810.037074] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23810.042548] iTCO_wdt [23810.045686] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23810.064423] iTCO_vendor_support [23810.066695] [23810.071912] i2c_smbus crct10dif_pclmul [23810.075136] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23810.076627] joydev crc32_pclmul [23810.080457] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23810.087581] ipmi_ssif ghash_clmulni_intel [23810.090806] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23810.097930] mei_me [23810.102021] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23810.109145] mei lpc_ich [23810.111243] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23810.118367] sha512_ssse3 acpi_ipmi [23810.120897] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23810.128022] mfd_core wmi [23810.131506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23810.139583] ipmi_si ipmi_devintf [23810.142200] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23810.147939] ipmi_msghandler [23810.151249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23810.158373] ioatdma acpi_pad [23810.161249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23810.168374] acpi_power_meter ixgbe [23810.171337] Call Trace: [23810.171339] [23810.178465] igb [23810.181965] ? watchdog_timer_fn+0x1b4/0x220 [23810.184409] mdio crc32c_intel [23810.186420] ? __hrtimer_run_queues+0x1d3/0x360 [23810.188258] nvme [23810.192523] ? hrtimer_interrupt+0x104/0x240 [23810.195571] ptp nvme_core [23810.200095] ? __local_bh_enable+0x50/0x60 [23810.202020] i2c_algo_bit pps_core [23810.206286] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23810.208986] dca fuse [23810.213078] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23810.216487] [23810.216488] irq event stamp: 112724030 [23810.221701] [23810.223971] hardirqs last enabled at (112724029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23810.228930] [23810.230421] hardirqs last disabled at (112724030): [] sysvec_apic_timer_interrupt+0xe/0x90 [23810.234164] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23810.236263] softirqs last enabled at (112724028): [] __do_softirq+0x2be/0x371 [23810.246509] ? queued_spin_lock_slowpath+0x18/0x2e0 [23810.248603] softirqs last disabled at (112724023): [] irq_exit_rcu+0xb0/0xd0 [23810.258416] do_raw_spin_lock+0xb5/0xc0 [23810.263719] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23810.272492] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23810.277359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23810.277360] Workqueue: events delayed_work [ceph] [23810.285960] try_get_cap_refs+0x159/0x880 [ceph] [23810.289787] [23810.299171] __ceph_get_caps+0x4a1/0x760 [ceph] [23810.303950] RIP: 0010:lock_acquire+0xf0/0x2b0 [23810.311425] ? __file_remove_privs+0x89/0x110 [23810.316117] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23810.320730] ? wake_bit_function+0x70/0x70 [23810.322218] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23810.326746] ? ceph_write_iter+0x260/0x770 [ceph] [23810.331093] [23810.331093] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23810.335445] ceph_write_iter+0x260/0x770 [ceph] [23810.354182] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23810.358276] ? lock_is_held_type+0xaa/0x120 [23810.363491] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23810.368189] vfs_write+0x324/0x4f0 [23810.369678] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23810.376808] ksys_pwrite64+0x65/0xa0 [23810.381326] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23810.388452] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23810.392627] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23810.399754] do_syscall_64+0x38/0x80 [23810.403151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23810.410275] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23810.413846] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23810.420971] RIP: 0033:0x7f48f0eeb93b [23810.426099] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23810.434179] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23810.437748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23810.443502] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23810.448546] Call Trace: [23810.448547] [23810.455670] ORIG_RAX: 0000000000000012 [23810.459243] ? watchdog_timer_fn+0x1b4/0x220 [23810.466366] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23810.485105] ? __hrtimer_run_queues+0x1d3/0x360 [23810.492227] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23810.497450] ? hrtimer_interrupt+0x104/0x240 [23810.499888] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23810.501899] ? __local_bh_enable+0x50/0x60 [23810.505732] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23810.509998] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23810.517119] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23810.521643] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23810.528778] [23810.533039] [23810.572959] [23810.575058] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23810.580366] ? lock_acquire+0xf0/0x2b0 [23810.584115] ? igrab+0x43/0x50 [23810.587166] ? lock_release+0xc5/0x260 [23810.590910] _raw_spin_lock+0x30/0x40 [23810.594567] ? igrab+0x1d/0x50 [23810.597618] igrab+0x1d/0x50 [23810.600498] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23810.605645] delayed_work+0x1d8/0x280 [ceph] [23810.609927] ? process_scheduled_works+0x22a/0x510 [23810.614716] process_scheduled_works+0x2a4/0x510 [23810.619332] worker_thread+0x188/0x340 [23810.623080] ? rescuer_thread+0x310/0x310 [23810.627085] kthread+0xf5/0x130 [23810.630232] ? kthread_complete_and_exit+0x20/0x20 [23810.635015] ret_from_fork+0x31/0x40 [23810.638595] ? kthread_complete_and_exit+0x20/0x20 [23810.643378] ret_from_fork_asm+0x11/0x20 [23810.647309] [23837.875576] watchdog: BUG: soft lockup - CPU#0 stuck for 11086s! [dbench:78683] [23837.882875] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23837.887576] watchdog: BUG: soft lockup - CPU#7 stuck for 20083s! [kworker/7:4:78808] [23837.938315] irq event stamp: 37587018 [23837.938317] hardirqs last enabled at (37587017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23837.946045] Modules linked in: [23837.949704] hardirqs last disabled at (37587018): [] sysvec_apic_timer_interrupt+0xe/0x90 [23837.959861] ceph [23837.962911] softirqs last enabled at (37586940): [] __do_softirq+0x2be/0x371 [23837.972635] libceph [23837.974560] softirqs last disabled at (37586935): [] irq_exit_rcu+0xb0/0xd0 [23837.983260] dns_resolver fscache [23837.985447] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23837.993973] netfs [23837.997285] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23838.006229] veth nft_chain_nat [23838.008256] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23838.015728] xt_MASQUERADE nf_nat [23838.018863] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23838.024341] nf_conntrack [23838.027651] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23838.046391] nf_defrag_ipv6 nf_defrag_ipv4 [23838.049008] [23838.049008] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23838.054224] nft_compat nf_tables [23838.058315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23838.059806] nfnetlink bridge [23838.066931] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23838.070253] stp llc [23838.077373] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23838.080338] xfs libcrc32c [23838.087462] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23838.089646] sunrpc intel_rapl_msr [23838.096770] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23838.099476] intel_rapl_common [23838.106599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23838.109995] sb_edac x86_pkg_temp_thermal [23838.118072] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23838.121125] coretemp kvm_intel [23838.126863] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23838.130867] kvm irqbypass [23838.137990] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23838.141126] i2c_i801 iTCO_wdt [23838.148253] Call Trace: [23838.148254] [23838.150973] iTCO_vendor_support [23838.158099] ? watchdog_timer_fn+0x1b4/0x220 [23838.161148] i2c_smbus crct10dif_pclmul [23838.163593] ? __hrtimer_run_queues+0x1d3/0x360 [23838.165602] joydev crc32_pclmul [23838.168829] ? hrtimer_interrupt+0x104/0x240 [23838.173092] ipmi_ssif ghash_clmulni_intel [23838.176920] ? __local_bh_enable+0x50/0x60 [23838.181446] mei_me mei [23838.184673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23838.188933] lpc_ich sha512_ssse3 [23838.193026] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23838.197115] acpi_ipmi mfd_core [23838.199560] [23838.204777] wmi ipmi_si [23838.208088] [23838.213043] ipmi_devintf ipmi_msghandler [23838.216181] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23838.218279] ioatdma [23838.220814] ? queued_spin_lock_slowpath+0x16/0x2e0 [23838.222908] acpi_pad acpi_power_meter [23838.226913] do_raw_spin_lock+0xb5/0xc0 [23838.232216] ixgbe [23838.234401] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23838.239270] igb mdio [23838.243017] try_get_cap_refs+0x159/0x880 [ceph] [23838.246846] crc32c_intel nvme [23838.248862] __ceph_get_caps+0x4a1/0x760 [ceph] [23838.253640] ptp nvme_core [23838.255915] ? __file_remove_privs+0x89/0x110 [23838.260522] i2c_algo_bit pps_core [23838.263573] ? wake_bit_function+0x70/0x70 [23838.268096] dca fuse [23838.270803] ? ceph_write_iter+0x260/0x770 [ceph] [23838.275151] [23838.275152] irq event stamp: 112944870 [23838.278549] ceph_write_iter+0x260/0x770 [ceph] [23838.282638] hardirqs last enabled at (112944869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23838.284912] ? lock_is_held_type+0xaa/0x120 [23838.289608] hardirqs last disabled at (112944870): [] sysvec_apic_timer_interrupt+0xe/0x90 [23838.291102] vfs_write+0x324/0x4f0 [23838.294843] softirqs last enabled at (112944868): [] __do_softirq+0x2be/0x371 [23838.299371] ksys_pwrite64+0x65/0xa0 [23838.309611] softirqs last disabled at (112944863): [] irq_exit_rcu+0xb0/0xd0 [23838.313787] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23838.323600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23838.326997] do_syscall_64+0x38/0x80 [23838.335767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23838.339337] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23838.347935] Workqueue: events delayed_work [ceph] [23838.353066] RIP: 0033:0x7f48f0eeb93b [23838.362443] [23838.366016] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23838.373485] RIP: 0010:lock_acquire+0xf0/0x2b0 [23838.378529] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23838.383225] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23838.386797] ORIG_RAX: 0000000000000012 [23838.388288] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23838.407026] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23838.411377] [23838.411377] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23838.416593] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23838.435331] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23838.439163] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23838.444379] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23838.451504] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23838.452994] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23838.460119] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23838.467242] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23838.474372] [23838.481491] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23838.527374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23838.533110] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23838.540235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23838.547360] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23838.554483] Call Trace: [23838.556928] [23838.558939] ? watchdog_timer_fn+0x1b4/0x220 [23838.563205] ? __hrtimer_run_queues+0x1d3/0x360 [23838.567739] ? hrtimer_interrupt+0x104/0x240 [23838.572007] ? __local_bh_enable+0x50/0x60 [23838.576100] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23838.581316] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23838.586273] [23838.588372] [23838.590469] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23838.595775] ? lock_acquire+0xf0/0x2b0 [23838.599526] ? igrab+0x43/0x50 [23838.602576] ? lock_release+0xc5/0x260 [23838.606322] _raw_spin_lock+0x30/0x40 [23838.609981] ? igrab+0x1d/0x50 [23838.613036] igrab+0x1d/0x50 [23838.615914] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23838.621066] delayed_work+0x1d8/0x280 [ceph] [23838.625345] ? process_scheduled_works+0x22a/0x510 [23838.630135] process_scheduled_works+0x2a4/0x510 [23838.634751] worker_thread+0x188/0x340 [23838.638500] ? rescuer_thread+0x310/0x310 [23838.642505] kthread+0xf5/0x130 [23838.645641] ? kthread_complete_and_exit+0x20/0x20 [23838.650426] ret_from_fork+0x31/0x40 [23838.653995] ? kthread_complete_and_exit+0x20/0x20 [23838.658781] ret_from_fork_asm+0x11/0x20 [23838.662703] [23865.876062] watchdog: BUG: soft lockup - CPU#0 stuck for 11112s! [dbench:78683] [23865.883372] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23865.888062] watchdog: BUG: soft lockup - CPU#7 stuck for 20109s! [kworker/7:4:78808] [23865.938820] irq event stamp: 37645236 [23865.946550] Modules linked in: ceph libceph dns_resolver fscache netfs veth [23865.950207] hardirqs last enabled at (37645235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23865.950212] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23865.967325] hardirqs last disabled at (37645236): [] sysvec_apic_timer_interrupt+0xe/0x90 [23865.974275] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [23865.983999] softirqs last enabled at (37645158): [] __do_softirq+0x2be/0x371 [23865.990776] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [23865.999461] softirqs last disabled at (37645153): [] irq_exit_rcu+0xb0/0xd0 [23865.999465] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [23866.014143] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23866.020140] irqbypass [23866.029085] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23866.029087] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23866.031458] i2c_i801 iTCO_wdt [23866.038931] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23866.044408] iTCO_vendor_support i2c_smbus [23866.047459] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23866.066197] crct10dif_pclmul joydev [23866.070287] [23866.075504] crc32_pclmul [23866.079075] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23866.080565] ipmi_ssif ghash_clmulni_intel [23866.083183] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23866.090308] mei_me mei [23866.094396] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23866.101522] lpc_ich [23866.103965] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23866.111089] sha512_ssse3 acpi_ipmi [23866.113273] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23866.120398] mfd_core wmi [23866.123883] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23866.131007] ipmi_si [23866.133624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23866.141702] ipmi_devintf ipmi_msghandler [23866.143886] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23866.149624] ioatdma acpi_pad [23866.153627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23866.160752] acpi_power_meter ixgbe [23866.163715] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23866.170856] igb [23866.174340] Call Trace: [23866.174343] [23866.181466] mdio crc32c_intel [23866.183306] ? watchdog_timer_fn+0x1b4/0x220 [23866.185746] nvme ptp [23866.187758] ? __hrtimer_run_queues+0x1d3/0x360 [23866.190808] nvme_core i2c_algo_bit [23866.195074] ? hrtimer_interrupt+0x104/0x240 [23866.197342] pps_core dca [23866.201865] ? __local_bh_enable+0x50/0x60 [23866.205350] fuse [23866.209617] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23866.212232] [23866.212232] irq event stamp: 113165624 [23866.216323] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23866.218246] hardirqs last enabled at (113165623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23866.223464] [23866.224953] hardirqs last disabled at (113165624): [] sysvec_apic_timer_interrupt+0xe/0x90 [23866.228699] [23866.228701] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23866.233655] softirqs last enabled at (113165622): [] __do_softirq+0x2be/0x371 [23866.243906] ? queued_spin_lock_slowpath+0x16/0x2e0 [23866.245998] softirqs last disabled at (113165617): [] irq_exit_rcu+0xb0/0xd0 [23866.255811] do_raw_spin_lock+0xb5/0xc0 [23866.257905] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23866.263212] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23866.271982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23866.276856] try_get_cap_refs+0x159/0x880 [ceph] [23866.285462] Workqueue: events delayed_work [ceph] [23866.289295] __ceph_get_caps+0x4a1/0x760 [ceph] [23866.298666] [23866.298667] RIP: 0010:lock_acquire+0xf0/0x2b0 [23866.303467] ? __file_remove_privs+0x89/0x110 [23866.310930] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23866.315541] ? wake_bit_function+0x70/0x70 [23866.320238] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23866.324764] ? ceph_write_iter+0x260/0x770 [ceph] [23866.326252] [23866.326252] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23866.330603] ceph_write_iter+0x260/0x770 [ceph] [23866.334955] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23866.353696] ? lock_is_held_type+0xaa/0x120 [23866.357782] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23866.363003] vfs_write+0x324/0x4f0 [23866.367698] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23866.369194] ksys_pwrite64+0x65/0xa0 [23866.376313] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23866.380836] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23866.387960] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23866.392139] do_syscall_64+0x38/0x80 [23866.399261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23866.402659] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23866.409784] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23866.413356] RIP: 0033:0x7f48f0eeb93b [23866.420478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23866.425608] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23866.433686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23866.437257] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23866.442995] Call Trace: [23866.448039] ORIG_RAX: 0000000000000012 [23866.455165] [23866.458744] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23866.465868] ? watchdog_timer_fn+0x1b4/0x220 [23866.484603] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23866.491746] ? __hrtimer_run_queues+0x1d3/0x360 [23866.496964] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23866.499410] ? hrtimer_interrupt+0x104/0x240 [23866.503237] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23866.505248] ? __local_bh_enable+0x50/0x60 [23866.512373] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23866.516639] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23866.523767] [23866.528286] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23866.570371] [23866.572470] [23866.574567] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23866.579873] ? lock_acquire+0xf0/0x2b0 [23866.583626] ? igrab+0x43/0x50 [23866.586682] ? lock_release+0xc5/0x260 [23866.590429] _raw_spin_lock+0x30/0x40 [23866.594094] ? igrab+0x1d/0x50 [23866.597143] igrab+0x1d/0x50 [23866.600023] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23866.605173] delayed_work+0x1d8/0x280 [ceph] [23866.609470] ? process_scheduled_works+0x22a/0x510 [23866.614260] process_scheduled_works+0x2a4/0x510 [23866.618876] worker_thread+0x188/0x340 [23866.622625] ? rescuer_thread+0x310/0x310 [23866.626628] kthread+0xf5/0x130 [23866.629774] ? kthread_complete_and_exit+0x20/0x20 [23866.634559] ret_from_fork+0x31/0x40 [23866.638139] ? kthread_complete_and_exit+0x20/0x20 [23866.642922] ret_from_fork_asm+0x11/0x20 [23866.646855] [23893.876551] watchdog: BUG: soft lockup - CPU#0 stuck for 11138s! [dbench:78683] [23893.883848] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23893.888551] watchdog: BUG: soft lockup - CPU#7 stuck for 20135s! [kworker/7:4:78808] [23893.939289] irq event stamp: 37703352 [23893.939290] hardirqs last enabled at (37703351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23893.947020] Modules linked in: [23893.950678] hardirqs last disabled at (37703352): [] sysvec_apic_timer_interrupt+0xe/0x90 [23893.960834] ceph [23893.963885] softirqs last enabled at (37703274): [] __do_softirq+0x2be/0x371 [23893.973610] libceph [23893.975535] softirqs last disabled at (37703269): [] irq_exit_rcu+0xb0/0xd0 [23893.984233] dns_resolver fscache [23893.986420] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23893.994945] netfs veth [23893.998257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23894.007202] nft_chain_nat xt_MASQUERADE [23894.009666] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [23894.017133] nf_nat nf_conntrack [23894.021053] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [23894.026529] nf_defrag_ipv6 nf_defrag_ipv4 [23894.029752] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23894.048492] nft_compat [23894.052582] [23894.052583] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23894.057800] nf_tables nfnetlink [23894.060245] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23894.061734] bridge stp [23894.068858] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23894.072083] llc xfs [23894.079205] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23894.081651] libcrc32c sunrpc [23894.088775] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23894.090958] intel_rapl_msr [23894.098082] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23894.101046] intel_rapl_common sb_edac [23894.108171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23894.110962] x86_pkg_temp_thermal [23894.119039] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23894.122782] coretemp kvm_intel [23894.128520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23894.131831] kvm [23894.138956] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23894.142092] irqbypass [23894.149234] Call Trace: [23894.149235] [23894.151071] i2c_i801 iTCO_wdt [23894.158197] ? watchdog_timer_fn+0x1b4/0x220 [23894.160554] iTCO_vendor_support i2c_smbus [23894.162997] ? __hrtimer_run_queues+0x1d3/0x360 [23894.165008] crct10dif_pclmul [23894.168062] ? hrtimer_interrupt+0x104/0x240 [23894.172323] joydev crc32_pclmul [23894.176414] ? __local_bh_enable+0x50/0x60 [23894.180951] ipmi_ssif ghash_clmulni_intel [23894.183922] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23894.188183] mei_me [23894.191407] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23894.195498] mei lpc_ich [23894.199589] [23894.204806] sha512_ssse3 acpi_ipmi [23894.206904] [23894.206905] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23894.211862] mfd_core [23894.214395] ? queued_spin_lock_slowpath+0x18/0x2e0 [23894.216488] wmi ipmi_si [23894.219974] do_raw_spin_lock+0xb5/0xc0 [23894.222070] ipmi_devintf ipmi_msghandler [23894.227376] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23894.229646] ioatdma acpi_pad [23894.234518] try_get_cap_refs+0x159/0x880 [ceph] [23894.237045] acpi_power_meter ixgbe [23894.240882] __ceph_get_caps+0x4a1/0x760 [ceph] [23894.244881] igb mdio [23894.249671] ? __file_remove_privs+0x89/0x110 [23894.252629] crc32c_intel nvme [23894.257240] ? wake_bit_function+0x70/0x70 [23894.260724] ptp nvme_core [23894.265250] ? ceph_write_iter+0x260/0x770 [ceph] [23894.267519] i2c_algo_bit pps_core [23894.271872] ceph_write_iter+0x260/0x770 [ceph] [23894.274922] dca fuse [23894.279015] ? lock_is_held_type+0xaa/0x120 [23894.281715] [23894.281716] irq event stamp: 113386550 [23894.286414] vfs_write+0x324/0x4f0 [23894.289811] hardirqs last enabled at (113386549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23894.294340] ksys_pwrite64+0x65/0xa0 [23894.296604] hardirqs last disabled at (113386550): [] sysvec_apic_timer_interrupt+0xe/0x90 [23894.300782] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23894.302272] softirqs last enabled at (113386548): [] __do_softirq+0x2be/0x371 [23894.306017] do_syscall_64+0x38/0x80 [23894.309415] softirqs last disabled at (113386543): [] irq_exit_rcu+0xb0/0xd0 [23894.319660] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23894.323238] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23894.333049] RIP: 0033:0x7f48f0eeb93b [23894.338179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23894.346951] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23894.350521] Workqueue: events delayed_work [ceph] [23894.359118] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23894.364163] [23894.364164] RIP: 0010:lock_acquire+0x104/0x2b0 [23894.373539] ORIG_RAX: 0000000000000012 [23894.377112] Code: ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 <48> 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 b2 e6 eb 7e [23894.384581] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23894.403320] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [23894.408017] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23894.413249] [23894.413250] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 000000005553a634 [23894.414743] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23894.419181] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23894.423010] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23894.441747] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23894.448873] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23894.454091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23894.461235] [23894.462723] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23894.521917] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23894.529994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23894.535732] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23894.542855] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23894.549981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23894.557104] Call Trace: [23894.559550] [23894.561560] ? watchdog_timer_fn+0x1b4/0x220 [23894.565825] ? __hrtimer_run_queues+0x1d3/0x360 [23894.570351] ? hrtimer_interrupt+0x104/0x240 [23894.574620] ? __local_bh_enable+0x50/0x60 [23894.578713] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23894.583929] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23894.588902] [23894.590999] [23894.593099] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23894.598415] ? lock_acquire+0x104/0x2b0 [23894.602252] ? igrab+0x43/0x50 [23894.605302] ? lock_release+0xc5/0x260 [23894.609046] _raw_spin_lock+0x30/0x40 [23894.612701] ? igrab+0x1d/0x50 [23894.615752] igrab+0x1d/0x50 [23894.618632] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23894.623781] delayed_work+0x1d8/0x280 [ceph] [23894.628062] ? process_scheduled_works+0x22a/0x510 [23894.632853] process_scheduled_works+0x2a4/0x510 [23894.637469] worker_thread+0x188/0x340 [23894.641216] ? rescuer_thread+0x310/0x310 [23894.645248] kthread+0xf5/0x130 [23894.648393] ? kthread_complete_and_exit+0x20/0x20 [23894.653176] ret_from_fork+0x31/0x40 [23894.656746] ? kthread_complete_and_exit+0x20/0x20 [23894.661530] ret_from_fork_asm+0x11/0x20 [23894.665454] [23921.877042] watchdog: BUG: soft lockup - CPU#0 stuck for 11164s! [dbench:78683] [23921.884344] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23921.889041] watchdog: BUG: soft lockup - CPU#7 stuck for 20161s! [kworker/7:4:78808] [23921.939794] irq event stamp: 37761646 [23921.947525] Modules linked in: ceph libceph dns_resolver fscache netfs veth [23921.951183] hardirqs last enabled at (37761645): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23921.958141] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23921.968298] hardirqs last disabled at (37761646): [] sysvec_apic_timer_interrupt+0xe/0x90 [23921.975250] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [23921.984973] softirqs last enabled at (37761568): [] __do_softirq+0x2be/0x371 [23921.991751] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [23922.000436] softirqs last disabled at (37761563): [] irq_exit_rcu+0xb0/0xd0 [23922.007299] x86_pkg_temp_thermal coretemp kvm_intel [23922.015811] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23922.015812] kvm [23922.015814] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23922.020768] irqbypass i2c_i801 [23922.029717] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23922.031549] iTCO_wdt iTCO_vendor_support [23922.039023] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23922.042158] i2c_smbus [23922.047636] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23922.051640] crct10dif_pclmul joydev [23922.070378] [23922.072733] crc32_pclmul ipmi_ssif [23922.077951] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23922.081523] ghash_clmulni_intel mei_me [23922.083012] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23922.086497] mei lpc_ich [23922.093622] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23922.097451] sha512_ssse3 [23922.104576] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23922.107107] acpi_ipmi mfd_core [23922.114230] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23922.116847] wmi ipmi_si [23922.123974] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23922.127109] ipmi_devintf [23922.134233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23922.136766] ipmi_msghandler ioatdma [23922.144842] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23922.147459] acpi_pad [23922.153197] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23922.156768] acpi_power_meter ixgbe [23922.163892] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23922.166164] igb [23922.173287] Call Trace: [23922.176770] mdio crc32c_intel [23922.183898] [23922.185733] nvme [23922.188179] ? watchdog_timer_fn+0x1b4/0x220 [23922.191228] ptp nvme_core [23922.193239] ? __hrtimer_run_queues+0x1d3/0x360 [23922.195163] i2c_algo_bit [23922.199444] ? hrtimer_interrupt+0x104/0x240 [23922.202147] pps_core dca [23922.206672] ? __local_bh_enable+0x50/0x60 [23922.209290] fuse [23922.213555] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23922.216170] irq event stamp: 113607314 [23922.220260] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23922.222184] hardirqs last enabled at (113607313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23922.227402] [23922.231146] hardirqs last disabled at (113607314): [] sysvec_apic_timer_interrupt+0xe/0x90 [23922.236103] [23922.246348] softirqs last enabled at (113607312): [] __do_softirq+0x2be/0x371 [23922.248446] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23922.258256] softirqs last disabled at (113607307): [] irq_exit_rcu+0xb0/0xd0 [23922.260358] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23922.269126] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23922.274444] do_raw_spin_lock+0xb5/0xc0 [23922.283044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23922.287917] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23922.297293] Workqueue: events delayed_work [ceph] [23922.301126] try_get_cap_refs+0x159/0x880 [ceph] [23922.308593] [23922.308594] RIP: 0010:lock_acquire+0xf0/0x2b0 [23922.313382] __ceph_get_caps+0x4a1/0x760 [ceph] [23922.318075] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23922.322690] ? __file_remove_privs+0x89/0x110 [23922.324176] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23922.328527] ? wake_bit_function+0x70/0x70 [23922.333052] [23922.333053] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23922.351790] ? ceph_write_iter+0x260/0x770 [ceph] [23922.356141] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23922.361358] ceph_write_iter+0x260/0x770 [ceph] [23922.365447] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23922.366943] ? lock_is_held_type+0xaa/0x120 [23922.374063] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23922.378763] vfs_write+0x324/0x4f0 [23922.385884] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23922.390414] ksys_pwrite64+0x65/0xa0 [23922.397532] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23922.401726] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23922.408851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23922.412250] do_syscall_64+0x38/0x80 [23922.419374] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23922.422945] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23922.431022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23922.436152] RIP: 0033:0x7f48f0eeb93b [23922.441888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23922.445461] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23922.452585] Call Trace: [23922.452587] [23922.457627] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23922.464753] ? watchdog_timer_fn+0x1b4/0x220 [23922.468323] ORIG_RAX: 0000000000000012 [23922.475447] ? __hrtimer_run_queues+0x1d3/0x360 [23922.494185] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23922.496633] ? hrtimer_interrupt+0x104/0x240 [23922.498639] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23922.503856] ? __local_bh_enable+0x50/0x60 [23922.508121] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23922.511953] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23922.516477] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23922.523601] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23922.527865] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23922.534990] [23922.539084] [23922.546203] [23922.577023] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23922.582332] ? lock_acquire+0xf0/0x2b0 [23922.586082] ? igrab+0x43/0x50 [23922.589132] ? lock_release+0xc5/0x260 [23922.592876] _raw_spin_lock+0x30/0x40 [23922.596540] ? igrab+0x1d/0x50 [23922.599590] igrab+0x1d/0x50 [23922.602470] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23922.607620] delayed_work+0x1d8/0x280 [ceph] [23922.611901] ? process_scheduled_works+0x22a/0x510 [23922.616689] process_scheduled_works+0x2a4/0x510 [23922.621307] worker_thread+0x188/0x340 [23922.625054] ? rescuer_thread+0x310/0x310 [23922.629057] kthread+0xf5/0x130 [23922.632196] ? kthread_complete_and_exit+0x20/0x20 [23922.636980] ret_from_fork+0x31/0x40 [23922.640560] ? kthread_complete_and_exit+0x20/0x20 [23922.645343] ret_from_fork_asm+0x11/0x20 [23922.649268] [23923.741441] rcu: INFO: rcu_preempt self-detected stall on CPU [23923.747200] rcu: 7-....: (11641898 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/11640617 fqs=5852528 [23923.757793] rcu: (t=11986917 jiffies g=1309609 q=382895 ncpus=8) [23923.763886] Sending NMI from CPU 7 to CPUs 0: [23923.768237] NMI backtrace for cpu 0 [23923.768238] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23923.768240] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23923.768240] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23923.768243] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23923.768245] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23923.768246] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23923.768247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23923.768247] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23923.768248] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23923.768249] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23923.768249] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23923.768251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23923.768251] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23923.768252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23923.768253] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23923.768253] Call Trace: [23923.768254] [23923.768255] ? nmi_cpu_backtrace+0xad/0x120 [23923.768258] ? nmi_cpu_backtrace_handler+0x11/0x20 [23923.768259] ? nmi_handle+0xd4/0x260 [23923.768264] ? default_do_nmi+0x49/0xf0 [23923.768265] ? exc_nmi+0x11e/0x150 [23923.768267] ? end_repeat_nmi+0x16/0x67 [23923.768271] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23923.768273] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23923.768276] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23923.768278] [23923.768278] [23923.768279] do_raw_spin_lock+0xb5/0xc0 [23923.768281] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23923.768295] try_get_cap_refs+0x159/0x880 [ceph] [23923.768308] __ceph_get_caps+0x4a1/0x760 [ceph] [23923.768320] ? __file_remove_privs+0x89/0x110 [23923.768322] ? wake_bit_function+0x70/0x70 [23923.768324] ? ceph_write_iter+0x260/0x770 [ceph] [23923.768334] ceph_write_iter+0x260/0x770 [ceph] [23923.768345] ? lock_is_held_type+0xaa/0x120 [23923.768348] vfs_write+0x324/0x4f0 [23923.768352] ksys_pwrite64+0x65/0xa0 [23923.768354] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23923.768356] do_syscall_64+0x38/0x80 [23923.768358] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23923.768360] RIP: 0033:0x7f48f0eeb93b [23923.768361] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23923.768362] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [23923.768363] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23923.768364] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23923.768365] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23923.768365] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23923.768366] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23923.768369] [23923.769235] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23924.072847] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23924.080319] Workqueue: events delayed_work [ceph] [23924.085041] RIP: 0010:lock_acquire+0xf0/0x2b0 [23924.089394] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23924.108132] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23924.113357] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23924.120480] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23924.127604] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23924.134728] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23924.141870] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23924.148994] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23924.157071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23924.162809] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23924.169933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23924.177058] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23924.184181] Call Trace: [23924.186627] [23924.188639] ? rcu_dump_cpu_stacks+0xd1/0x180 [23924.192997] ? rcu_sched_clock_irq+0x633/0x1390 [23924.197530] ? lock_release+0xc5/0x260 [23924.201283] ? lock_is_held_type+0xaa/0x120 [23924.205482] ? lock_is_held_type+0xaa/0x120 [23924.209665] ? tick_sched_do_timer+0xa0/0xa0 [23924.213937] ? rcu_read_lock_sched_held+0x56/0x90 [23924.218643] ? tick_sched_do_timer+0xa0/0xa0 [23924.222914] ? update_process_times+0x5f/0x90 [23924.227266] ? tick_sched_handle.isra.16+0x1e/0x50 [23924.232057] ? tick_sched_timer+0x6f/0x80 [23924.236064] ? __hrtimer_run_queues+0x1d3/0x360 [23924.240604] ? hrtimer_interrupt+0x104/0x240 [23924.244867] ? __local_bh_enable+0x50/0x60 [23924.248963] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23924.254185] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23924.259142] [23924.261238] [23924.263338] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23924.268658] ? lock_acquire+0xf0/0x2b0 [23924.272405] ? igrab+0x43/0x50 [23924.275478] ? lock_release+0xc5/0x260 [23924.279226] _raw_spin_lock+0x30/0x40 [23924.282888] ? igrab+0x1d/0x50 [23924.285940] igrab+0x1d/0x50 [23924.288819] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23924.293969] delayed_work+0x1d8/0x280 [ceph] [23924.298259] ? process_scheduled_works+0x22a/0x510 [23924.303047] process_scheduled_works+0x2a4/0x510 [23924.307668] worker_thread+0x188/0x340 [23924.311422] ? rescuer_thread+0x310/0x310 [23924.315432] kthread+0xf5/0x130 [23924.318570] ? kthread_complete_and_exit+0x20/0x20 [23924.323364] ret_from_fork+0x31/0x40 [23924.326942] ? kthread_complete_and_exit+0x20/0x20 [23924.331727] ret_from_fork_asm+0x11/0x20 [23924.335657] [23949.877530] watchdog: BUG: soft lockup - CPU#0 stuck for 11191s! [dbench:78683] [23949.884831] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23949.889530] watchdog: BUG: soft lockup - CPU#7 stuck for 20188s! [kworker/7:4:78808] [23949.940272] irq event stamp: 37819754 [23949.948003] Modules linked in: ceph [23949.951661] hardirqs last enabled at (37819753): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23949.951664] libceph dns_resolver fscache netfs [23949.965303] hardirqs last disabled at (37819754): [] sysvec_apic_timer_interrupt+0xe/0x90 [23949.969827] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [23949.979551] softirqs last enabled at (37819702): [] __do_softirq+0x2be/0x371 [23949.985634] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [23949.994318] softirqs last disabled at (37819697): [] irq_exit_rcu+0xb0/0xd0 [23950.001095] bridge stp llc [23950.009608] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23950.012397] xfs libcrc32c [23950.021342] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23950.024046] sunrpc intel_rapl_msr intel_rapl_common sb_edac [23950.031519] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [23950.031520] x86_pkg_temp_thermal coretemp kvm_intel [23950.042646] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [23950.042646] kvm [23950.047603] irqbypass i2c_i801 [23950.066340] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23950.068178] iTCO_wdt iTCO_vendor_support [23950.071332] [23950.071333] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23950.076550] i2c_smbus crct10dif_pclmul [23950.080554] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23950.082044] joydev crc32_pclmul [23950.089168] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23950.093000] ipmi_ssif ghash_clmulni_intel [23950.100123] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23950.103348] mei_me mei [23950.110472] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23950.114562] lpc_ich sha512_ssse3 [23950.121687] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23950.124131] acpi_ipmi [23950.131256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23950.134564] mfd_core wmi [23950.142642] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23950.145001] ipmi_si ipmi_devintf [23950.150738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23950.153354] ipmi_msghandler [23950.160480] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23950.163789] ioatdma acpi_pad [23950.170928] Call Trace: [23950.170929] [23950.173809] acpi_power_meter [23950.180933] ? watchdog_timer_fn+0x1b4/0x220 [23950.183896] ixgbe igb [23950.186342] ? __hrtimer_run_queues+0x1d3/0x360 [23950.188352] mdio crc32c_intel [23950.191319] ? hrtimer_interrupt+0x104/0x240 [23950.195579] nvme ptp [23950.197937] ? __local_bh_enable+0x50/0x60 [23950.202460] nvme_core i2c_algo_bit [23950.205514] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23950.209776] pps_core dca [23950.212047] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23950.216139] fuse [23950.219621] [23950.224838] irq event stamp: 113823372 [23950.227456] [23950.232414] hardirqs last enabled at (113823371): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23950.234338] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23950.236434] hardirqs last disabled at (113823372): [] sysvec_apic_timer_interrupt+0xe/0x90 [23950.240182] ? queued_spin_lock_slowpath+0x2d/0x2e0 [23950.242275] softirqs last enabled at (113823370): [] __do_softirq+0x2be/0x371 [23950.252523] do_raw_spin_lock+0xb5/0xc0 [23950.257824] softirqs last disabled at (113823365): [] irq_exit_rcu+0xb0/0xd0 [23950.267637] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23950.272508] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23950.281278] try_get_cap_refs+0x159/0x880 [ceph] [23950.285107] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23950.293711] __ceph_get_caps+0x4a1/0x760 [ceph] [23950.298490] Workqueue: events delayed_work [ceph] [23950.307870] ? __file_remove_privs+0x89/0x110 [23950.312478] [23950.319950] ? wake_bit_function+0x70/0x70 [23950.324474] RIP: 0010:lock_acquire+0xf0/0x2b0 [23950.329173] ? ceph_write_iter+0x260/0x770 [ceph] [23950.333522] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [23950.335011] ceph_write_iter+0x260/0x770 [ceph] [23950.339103] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [23950.343457] ? lock_is_held_type+0xaa/0x120 [23950.348149] [23950.348150] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [23950.366890] vfs_write+0x324/0x4f0 [23950.371413] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [23950.376635] ksys_pwrite64+0x65/0xa0 [23950.380806] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [23950.382299] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23950.389423] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [23950.392820] do_syscall_64+0x38/0x80 [23950.399943] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23950.403515] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23950.410639] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23950.415769] RIP: 0033:0x7f48f0eeb93b [23950.422892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23950.426464] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23950.433588] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23950.438633] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23950.446710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23950.450280] ORIG_RAX: 0000000000000012 [23950.456017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23950.474757] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23950.481881] Call Trace: [23950.487096] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23950.494224] [23950.498052] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23950.505178] ? watchdog_timer_fn+0x1b4/0x220 [23950.512301] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23950.514746] ? __hrtimer_run_queues+0x1d3/0x360 [23950.521869] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23950.523882] ? hrtimer_interrupt+0x104/0x240 [23950.531008] [23950.535266] ? __local_bh_enable+0x50/0x60 [23950.564599] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23950.569823] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23950.574780] [23950.576877] [23950.578974] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23950.584283] ? lock_acquire+0xf0/0x2b0 [23950.588032] ? igrab+0x43/0x50 [23950.591083] ? lock_release+0xc5/0x260 [23950.594827] _raw_spin_lock+0x30/0x40 [23950.598484] ? igrab+0x1d/0x50 [23950.601533] igrab+0x1d/0x50 [23950.604413] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23950.609562] delayed_work+0x1d8/0x280 [ceph] [23950.613844] ? process_scheduled_works+0x22a/0x510 [23950.618634] process_scheduled_works+0x2a4/0x510 [23950.623250] worker_thread+0x188/0x340 [23950.626998] ? rescuer_thread+0x310/0x310 [23950.631003] kthread+0xf5/0x130 [23950.634138] ? kthread_complete_and_exit+0x20/0x20 [23950.638939] ret_from_fork+0x31/0x40 [23950.642511] ? kthread_complete_and_exit+0x20/0x20 [23950.647295] ret_from_fork_asm+0x11/0x20 [23950.651225] [23977.878017] watchdog: BUG: soft lockup - CPU#0 stuck for 11217s! [dbench:78683] [23977.885318] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [23977.890016] watchdog: BUG: soft lockup - CPU#7 stuck for 20214s! [kworker/7:4:78808] [23977.940768] irq event stamp: 37877972 [23977.948498] Modules linked in: ceph libceph dns_resolver fscache netfs [23977.952157] hardirqs last enabled at (37877971): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23977.958697] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [23977.968857] hardirqs last disabled at (37877972): [] sysvec_apic_timer_interrupt+0xe/0x90 [23977.976240] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [23977.985965] softirqs last enabled at (37877924): [] __do_softirq+0x2be/0x371 [23977.985968] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [23978.000734] softirqs last disabled at (37877919): [] irq_exit_rcu+0xb0/0xd0 [23978.008300] x86_pkg_temp_thermal coretemp kvm_intel [23978.016811] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23978.021768] kvm [23978.030713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23978.030715] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [23978.032549] irqbypass [23978.040021] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [23978.045498] i2c_i801 iTCO_wdt [23978.047855] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [23978.066594] iTCO_vendor_support i2c_smbus [23978.069644] [23978.069645] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [23978.074861] crct10dif_pclmul [23978.078952] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [23978.080444] joydev [23978.087565] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [23978.090530] crc32_pclmul ipmi_ssif [23978.097655] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [23978.097656] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [23978.099753] ghash_clmulni_intel [23978.106875] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [23978.110360] mei_me mei [23978.117485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23978.124609] lpc_ich [23978.127832] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [23978.135910] sha512_ssse3 [23978.138356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23978.144093] acpi_ipmi [23978.146275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23978.153412] mfd_core [23978.156027] Call Trace: [23978.156029] [23978.163149] wmi [23978.165510] ? watchdog_timer_fn+0x1b4/0x220 [23978.172631] ipmi_si ipmi_devintf [23978.174904] ? __hrtimer_run_queues+0x1d3/0x360 [23978.177346] ipmi_msghandler [23978.179360] ? hrtimer_interrupt+0x104/0x240 [23978.181195] ioatdma acpi_pad [23978.185458] ? __local_bh_enable+0x50/0x60 [23978.188770] acpi_power_meter ixgbe [23978.193296] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23978.196171] igb [23978.200434] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23978.203416] mdio crc32c_intel [23978.207506] [23978.210991] nvme [23978.216209] [23978.218047] ptp nvme_core [23978.223005] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23978.226054] i2c_algo_bit pps_core [23978.228157] ? queued_spin_lock_slowpath+0x16/0x2e0 [23978.230076] dca fuse [23978.232176] do_raw_spin_lock+0xb5/0xc0 [23978.234875] [23978.234876] irq event stamp: 114044200 [23978.240183] __cap_delay_requeue+0xa7/0x1d0 [ceph] [23978.243579] hardirqs last enabled at (114044199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23978.248453] try_get_cap_refs+0x159/0x880 [ceph] [23978.250720] hardirqs last disabled at (114044200): [] sysvec_apic_timer_interrupt+0xe/0x90 [23978.254557] __ceph_get_caps+0x4a1/0x760 [ceph] [23978.256041] softirqs last enabled at (114044198): [] __do_softirq+0x2be/0x371 [23978.259790] ? __file_remove_privs+0x89/0x110 [23978.264569] softirqs last disabled at (114044193): [] irq_exit_rcu+0xb0/0xd0 [23978.274815] ? wake_bit_function+0x70/0x70 [23978.279424] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [23978.289238] ? ceph_write_iter+0x260/0x770 [ceph] [23978.293760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [23978.302531] ceph_write_iter+0x260/0x770 [ceph] [23978.306882] Workqueue: events delayed_work [ceph] [23978.315482] ? lock_is_held_type+0xaa/0x120 [23978.319569] [23978.319570] RIP: 0010:lock_acquire+0x1f/0x2b0 [23978.328948] vfs_write+0x324/0x4f0 [23978.333645] Code: 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 <53> 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f [23978.341121] ksys_pwrite64+0x65/0xa0 [23978.345640] RSP: 0018:ffffc90002117d58 EFLAGS: 00000246 [23978.350338] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [23978.354513] [23978.354514] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [23978.356006] do_syscall_64+0x38/0x80 [23978.360354] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [23978.363754] entry_SYSCALL_64_after_hwframe+0x63/0xcd [23978.382489] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [23978.386062] RIP: 0033:0x7f48f0eeb93b [23978.391278] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [23978.396420] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [23978.397908] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [23978.405032] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [23978.408602] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [23978.415727] ORIG_RAX: 0000000000000012 [23978.420771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [23978.427895] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [23978.431466] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [23978.438589] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [23978.457328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [23978.464453] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [23978.469670] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [23978.477746] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [23978.481579] Call Trace: [23978.481580] [23978.487316] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [23978.494439] ? watchdog_timer_fn+0x1b4/0x220 [23978.501568] [23978.508701] ? __hrtimer_run_queues+0x1d3/0x360 [23978.559749] ? hrtimer_interrupt+0x104/0x240 [23978.564017] ? __local_bh_enable+0x50/0x60 [23978.568112] ? __sysvec_apic_timer_interrupt+0x61/0x160 [23978.573335] ? sysvec_apic_timer_interrupt+0x6d/0x90 [23978.578292] [23978.580390] [23978.582487] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [23978.587796] ? lock_acquire+0x1f/0x2b0 [23978.591545] _raw_spin_lock+0x30/0x40 [23978.595203] ? igrab+0x1d/0x50 [23978.598251] igrab+0x1d/0x50 [23978.601130] ceph_check_delayed_caps+0xb7/0x270 [ceph] [23978.606281] delayed_work+0x1d8/0x280 [ceph] [23978.610563] ? process_scheduled_works+0x22a/0x510 [23978.615350] process_scheduled_works+0x2a4/0x510 [23978.619967] worker_thread+0x188/0x340 [23978.623714] ? rescuer_thread+0x310/0x310 [23978.627718] kthread+0xf5/0x130 [23978.630857] ? kthread_complete_and_exit+0x20/0x20 [23978.635643] ret_from_fork+0x31/0x40 [23978.639220] ? kthread_complete_and_exit+0x20/0x20 [23978.644004] ret_from_fork_asm+0x11/0x20 [23978.647935] [24005.878503] watchdog: BUG: soft lockup - CPU#0 stuck for 11243s! [dbench:78683] [24005.885806] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24005.890503] watchdog: BUG: soft lockup - CPU#7 stuck for 20240s! [kworker/7:4:78808] [24005.941246] irq event stamp: 37936216 [24005.941247] hardirqs last enabled at (37936215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24005.948977] Modules linked in: [24005.952633] hardirqs last disabled at (37936216): [] sysvec_apic_timer_interrupt+0xe/0x90 [24005.962793] ceph [24005.965842] softirqs last enabled at (37936180): [] __do_softirq+0x2be/0x371 [24005.975567] libceph dns_resolver [24005.977490] softirqs last disabled at (37936175): [] irq_exit_rcu+0xb0/0xd0 [24005.986180] fscache netfs [24005.989494] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24005.998004] veth nft_chain_nat [24006.000709] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24006.009654] xt_MASQUERADE nf_nat [24006.012793] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24006.020261] nf_conntrack nf_defrag_ipv6 [24006.023571] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24006.029049] nf_defrag_ipv4 nft_compat [24006.032966] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24006.051705] nf_tables nfnetlink [24006.055448] [24006.055448] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24006.060666] bridge stp [24006.063900] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24006.065391] llc xfs [24006.072513] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24006.074959] libcrc32c [24006.082081] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24006.084267] sunrpc intel_rapl_msr [24006.091389] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24006.093749] intel_rapl_common [24006.100873] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24006.104285] sb_edac x86_pkg_temp_thermal [24006.111411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24006.114462] coretemp kvm_intel [24006.122540] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24006.126542] kvm irqbypass [24006.132282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24006.135416] i2c_i801 iTCO_wdt [24006.142541] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24006.145246] iTCO_vendor_support i2c_smbus [24006.152370] Call Trace: [24006.155421] crct10dif_pclmul joydev [24006.162546] [24006.166636] crc32_pclmul ipmi_ssif [24006.169079] ? watchdog_timer_fn+0x1b4/0x220 [24006.172651] ghash_clmulni_intel [24006.174661] ? __hrtimer_run_queues+0x1d3/0x360 [24006.178146] mei_me mei [24006.182411] ? hrtimer_interrupt+0x104/0x240 [24006.185634] lpc_ich [24006.190158] ? __local_bh_enable+0x50/0x60 [24006.192601] sha512_ssse3 acpi_ipmi [24006.196869] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24006.199050] mfd_core wmi [24006.203141] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24006.206623] ipmi_si ipmi_devintf [24006.211843] [24006.214458] ipmi_msghandler ioatdma [24006.219418] [24006.222727] acpi_pad acpi_power_meter [24006.224825] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24006.228396] ixgbe igb [24006.230498] ? queued_spin_lock_slowpath+0x16/0x2e0 [24006.234236] mdio crc32c_intel [24006.239542] do_raw_spin_lock+0xb5/0xc0 [24006.241900] nvme [24006.246770] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24006.249819] ptp nvme_core [24006.253653] try_get_cap_refs+0x159/0x880 [ceph] [24006.255575] i2c_algo_bit pps_core [24006.260364] __ceph_get_caps+0x4a1/0x760 [ceph] [24006.263063] dca fuse [24006.267676] ? __file_remove_privs+0x89/0x110 [24006.271070] [24006.271071] irq event stamp: 114265740 [24006.275596] ? wake_bit_function+0x70/0x70 [24006.277866] hardirqs last enabled at (114265739): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24006.282219] ? ceph_write_iter+0x260/0x770 [ceph] [24006.283707] hardirqs last disabled at (114265740): [] sysvec_apic_timer_interrupt+0xe/0x90 [24006.287451] ceph_write_iter+0x260/0x770 [ceph] [24006.291543] softirqs last enabled at (114265738): [] __do_softirq+0x2be/0x371 [24006.301791] ? lock_is_held_type+0xaa/0x120 [24006.306484] softirqs last disabled at (114265733): [] irq_exit_rcu+0xb0/0xd0 [24006.316298] vfs_write+0x324/0x4f0 [24006.320818] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24006.329595] ksys_pwrite64+0x65/0xa0 [24006.333767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24006.342365] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24006.345763] Workqueue: events delayed_work [ceph] [24006.355141] do_syscall_64+0x38/0x80 [24006.358711] [24006.358712] RIP: 0010:lock_release+0x14e/0x260 [24006.366191] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24006.371319] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [24006.376018] RIP: 0033:0x7f48f0eeb93b [24006.379589] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [24006.381078] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24006.385516] [24006.385517] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [24006.390561] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24006.409297] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [24006.412868] ORIG_RAX: 0000000000000012 [24006.418085] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24006.436824] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24006.438314] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24006.445440] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24006.450657] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [24006.457780] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24006.461610] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24006.468735] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24006.475858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24006.482983] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24006.490108] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24006.497236] [24006.504357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24006.548853] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24006.555975] Call Trace: [24006.558422] [24006.560432] ? watchdog_timer_fn+0x1b4/0x220 [24006.564696] ? __hrtimer_run_queues+0x1d3/0x360 [24006.569223] ? hrtimer_interrupt+0x104/0x240 [24006.573491] ? __local_bh_enable+0x50/0x60 [24006.577586] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24006.582810] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24006.587765] [24006.589862] [24006.591962] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24006.597266] ? igrab+0x43/0x50 [24006.600317] ? lock_release+0x14e/0x260 [24006.604149] ? lock_release+0xc5/0x260 [24006.607910] _raw_spin_unlock+0x1b/0x40 [24006.611749] igrab+0x43/0x50 [24006.614627] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24006.619775] delayed_work+0x1d8/0x280 [ceph] [24006.624056] ? process_scheduled_works+0x22a/0x510 [24006.628846] process_scheduled_works+0x2a4/0x510 [24006.633462] worker_thread+0x188/0x340 [24006.637212] ? rescuer_thread+0x310/0x310 [24006.641222] kthread+0xf5/0x130 [24006.644360] ? kthread_complete_and_exit+0x20/0x20 [24006.649147] ret_from_fork+0x31/0x40 [24006.652725] ? kthread_complete_and_exit+0x20/0x20 [24006.657508] ret_from_fork_asm+0x11/0x20 [24006.661432] [24033.878990] watchdog: BUG: soft lockup - CPU#0 stuck for 11269s! [dbench:78683] [24033.886293] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24033.890990] watchdog: BUG: soft lockup - CPU#7 stuck for 20266s! [kworker/7:4:78808] [24033.941741] irq event stamp: 37994472 [24033.941742] hardirqs last enabled at (37994471): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24033.949473] Modules linked in: [24033.953130] hardirqs last disabled at (37994472): [] sysvec_apic_timer_interrupt+0xe/0x90 [24033.963287] ceph [24033.966338] softirqs last enabled at (37994426): [] __do_softirq+0x2be/0x371 [24033.976063] libceph [24033.977986] softirqs last disabled at (37994421): [] irq_exit_rcu+0xb0/0xd0 [24033.986670] dns_resolver [24033.988856] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24033.997382] fscache [24033.999999] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24034.008946] netfs [24034.011130] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24034.018599] veth [24034.020609] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24034.026089] nft_chain_nat [24034.028011] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24034.046748] xt_MASQUERADE nf_nat [24034.049454] [24034.049455] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24034.054670] nf_conntrack nf_defrag_ipv6 [24034.057981] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24034.059471] nf_defrag_ipv4 nft_compat [24034.066596] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24034.070514] nf_tables nfnetlink [24034.077638] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24034.081386] bridge [24034.088515] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24034.091738] stp [24034.098862] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24034.100960] llc xfs [24034.108085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24034.109921] libcrc32c [24034.117999] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24034.120184] sunrpc [24034.125922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24034.128277] intel_rapl_msr [24034.135403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24034.137498] intel_rapl_common sb_edac [24034.144624] Call Trace: [24034.144626] [24034.147414] x86_pkg_temp_thermal [24034.154542] ? watchdog_timer_fn+0x1b4/0x220 [24034.158283] coretemp kvm_intel [24034.160727] ? __hrtimer_run_queues+0x1d3/0x360 [24034.162736] kvm irqbypass [24034.166052] ? hrtimer_interrupt+0x104/0x240 [24034.170311] i2c_i801 [24034.173448] ? __local_bh_enable+0x50/0x60 [24034.177974] iTCO_wdt iTCO_vendor_support [24034.180680] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24034.184941] i2c_smbus crct10dif_pclmul [24034.187213] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24034.191303] joydev [24034.195308] [24034.200524] crc32_pclmul [24034.204354] [24034.209313] ipmi_ssif ghash_clmulni_intel [24034.211409] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24034.213508] mei_me mei [24034.216130] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24034.218222] lpc_ich [24034.222315] do_raw_spin_lock+0xb5/0xc0 [24034.227617] sha512_ssse3 acpi_ipmi [24034.230062] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24034.234932] mfd_core [24034.237118] try_get_cap_refs+0x159/0x880 [ceph] [24034.240946] wmi [24034.244435] __ceph_get_caps+0x4a1/0x760 [ceph] [24034.249216] ipmi_si [24034.251490] ? __file_remove_privs+0x89/0x110 [24034.256096] ipmi_devintf [24034.257935] ? wake_bit_function+0x70/0x70 [24034.262459] ipmi_msghandler [24034.264645] ? ceph_write_iter+0x260/0x770 [ceph] [24034.269010] ioatdma [24034.271629] ceph_write_iter+0x260/0x770 [ceph] [24034.275719] acpi_pad acpi_power_meter [24034.278599] ? lock_is_held_type+0xaa/0x120 [24034.283292] ixgbe [24034.285480] vfs_write+0x324/0x4f0 [24034.290000] igb mdio [24034.293749] ksys_pwrite64+0x65/0xa0 [24034.297924] crc32c_intel nvme [24034.299934] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24034.303332] ptp [24034.305603] do_syscall_64+0x38/0x80 [24034.309172] nvme_core i2c_algo_bit [24034.312224] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24034.317354] pps_core [24034.319191] RIP: 0033:0x7f48f0eeb93b [24034.322763] dca [24034.326246] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24034.331290] fuse [24034.333560] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24034.337132] [24034.337132] irq event stamp: 114486588 [24034.338969] ORIG_RAX: 0000000000000012 [24034.357706] hardirqs last enabled at (114486587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24034.359631] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24034.364848] hardirqs last disabled at (114486588): [] sysvec_apic_timer_interrupt+0xe/0x90 [24034.366339] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24034.370082] softirqs last enabled at (114486586): [] __do_softirq+0x2be/0x371 [24034.373914] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24034.384158] softirqs last disabled at (114486581): [] irq_exit_rcu+0xb0/0xd0 [24034.391281] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24034.401093] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24034.408217] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24034.416987] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24034.424117] [24034.432708] Workqueue: events delayed_work [ceph] [24034.470686] RIP: 0010:lock_release+0x78/0x260 [24034.475039] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [24034.493774] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [24034.498993] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [24034.506118] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [24034.513242] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24034.520367] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24034.527508] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24034.534632] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24034.542707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24034.548446] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24034.555571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24034.562695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24034.569817] Call Trace: [24034.572263] [24034.574275] ? watchdog_timer_fn+0x1b4/0x220 [24034.578539] ? __hrtimer_run_queues+0x1d3/0x360 [24034.583073] ? hrtimer_interrupt+0x104/0x240 [24034.587342] ? __local_bh_enable+0x50/0x60 [24034.591436] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24034.596676] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24034.601634] [24034.603733] [24034.605830] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24034.611134] ? igrab+0x43/0x50 [24034.614187] ? lock_release+0x78/0x260 [24034.617933] _raw_spin_unlock+0x1b/0x40 [24034.621769] igrab+0x43/0x50 [24034.624648] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24034.629813] delayed_work+0x1d8/0x280 [ceph] [24034.634095] ? process_scheduled_works+0x22a/0x510 [24034.638885] process_scheduled_works+0x2a4/0x510 [24034.643500] worker_thread+0x188/0x340 [24034.647248] ? rescuer_thread+0x310/0x310 [24034.651252] kthread+0xf5/0x130 [24034.654400] ? kthread_complete_and_exit+0x20/0x20 [24034.659184] ret_from_fork+0x31/0x40 [24034.662764] ? kthread_complete_and_exit+0x20/0x20 [24034.667554] ret_from_fork_asm+0x11/0x20 [24034.671478] [24061.879476] watchdog: BUG: soft lockup - CPU#0 stuck for 11295s! [dbench:78683] [24061.886779] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24061.891477] watchdog: BUG: soft lockup - CPU#7 stuck for 20292s! [kworker/7:4:78808] [24061.942221] irq event stamp: 38052896 [24061.942223] hardirqs last enabled at (38052895): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24061.949950] Modules linked in: [24061.953607] hardirqs last disabled at (38052896): [] sysvec_apic_timer_interrupt+0xe/0x90 [24061.963767] ceph [24061.966817] softirqs last enabled at (38052848): [] __do_softirq+0x2be/0x371 [24061.976541] libceph [24061.978464] softirqs last disabled at (38052843): [] irq_exit_rcu+0xb0/0xd0 [24061.987159] dns_resolver fscache [24061.989342] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24061.997870] netfs [24062.001180] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24062.010124] veth nft_chain_nat [24062.012134] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24062.019623] xt_MASQUERADE [24062.022761] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24062.028238] nf_nat nf_conntrack [24062.030943] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24062.049681] nf_defrag_ipv6 nf_defrag_ipv4 [24062.052905] [24062.052906] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24062.058121] nft_compat nf_tables [24062.062213] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24062.063704] nfnetlink bridge [24062.070828] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24062.074151] stp llc [24062.081279] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24062.084243] xfs [24062.091368] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24062.093550] libcrc32c sunrpc [24062.100677] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24062.102512] intel_rapl_msr intel_rapl_common [24062.109636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24062.112602] sb_edac x86_pkg_temp_thermal [24062.120680] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24062.125029] coretemp kvm_intel [24062.130768] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24062.134772] kvm irqbypass [24062.141895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24062.145033] i2c_i801 [24062.152156] Call Trace: [24062.152157] [24062.154876] iTCO_wdt [24062.162003] ? watchdog_timer_fn+0x1b4/0x220 [24062.164272] iTCO_vendor_support i2c_smbus [24062.166718] ? __hrtimer_run_queues+0x1d3/0x360 [24062.168728] crct10dif_pclmul [24062.171002] ? hrtimer_interrupt+0x104/0x240 [24062.175263] joydev crc32_pclmul [24062.179354] ? __local_bh_enable+0x50/0x60 [24062.183877] ipmi_ssif ghash_clmulni_intel [24062.186845] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24062.191105] mei_me mei [24062.194330] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24062.198419] lpc_ich sha512_ssse3 [24062.202512] [24062.207728] acpi_ipmi mfd_core [24062.210172] [24062.215131] wmi ipmi_si [24062.218441] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24062.220539] ipmi_devintf [24062.223681] ? queued_spin_lock_slowpath+0x16/0x2e0 [24062.225773] ipmi_msghandler [24062.228306] do_raw_spin_lock+0xb5/0xc0 [24062.233607] ioatdma acpi_pad [24062.236227] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24062.241096] acpi_power_meter [24062.243976] try_get_cap_refs+0x159/0x880 [ceph] [24062.247804] ixgbe igb [24062.250773] __ceph_get_caps+0x4a1/0x760 [ceph] [24062.255552] mdio [24062.258520] ? __file_remove_privs+0x89/0x110 [24062.263127] crc32c_intel nvme [24062.265484] ? wake_bit_function+0x70/0x70 [24062.270009] ptp nvme_core [24062.271934] ? ceph_write_iter+0x260/0x770 [ceph] [24062.276283] i2c_algo_bit pps_core [24062.279335] ceph_write_iter+0x260/0x770 [ceph] [24062.283426] dca fuse [24062.286133] ? lock_is_held_type+0xaa/0x120 [24062.290826] [24062.290826] irq event stamp: 114708260 [24062.294227] vfs_write+0x324/0x4f0 [24062.298746] hardirqs last enabled at (114708259): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24062.301024] ksys_pwrite64+0x65/0xa0 [24062.305197] hardirqs last disabled at (114708260): [] sysvec_apic_timer_interrupt+0xe/0x90 [24062.306685] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24062.310431] softirqs last enabled at (114708258): [] __do_softirq+0x2be/0x371 [24062.313830] do_syscall_64+0x38/0x80 [24062.324073] softirqs last disabled at (114708253): [] irq_exit_rcu+0xb0/0xd0 [24062.327643] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24062.337455] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24062.342584] RIP: 0033:0x7f48f0eeb93b [24062.351356] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24062.354927] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24062.363524] Workqueue: events delayed_work [ceph] [24062.368568] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24062.377946] [24062.381517] ORIG_RAX: 0000000000000012 [24062.388988] RIP: 0010:lock_acquire+0xf0/0x2b0 [24062.407723] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24062.412420] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24062.417638] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24062.419129] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24062.422960] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24062.427310] [24062.427311] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24062.434435] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24062.453174] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24062.460297] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24062.465515] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24062.472645] [24062.474130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24062.474131] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24062.526182] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24062.534261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24062.539998] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24062.547121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24062.554246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24062.561371] Call Trace: [24062.563816] [24062.565826] ? watchdog_timer_fn+0x1b4/0x220 [24062.570089] ? __hrtimer_run_queues+0x1d3/0x360 [24062.574618] ? hrtimer_interrupt+0x104/0x240 [24062.578885] ? __local_bh_enable+0x50/0x60 [24062.582979] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24062.588204] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24062.593163] [24062.595267] [24062.597364] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24062.602671] ? lock_acquire+0xf0/0x2b0 [24062.606423] ? igrab+0x43/0x50 [24062.609471] ? lock_release+0xc5/0x260 [24062.613217] _raw_spin_lock+0x30/0x40 [24062.616885] ? igrab+0x1d/0x50 [24062.619940] igrab+0x1d/0x50 [24062.622819] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24062.627969] delayed_work+0x1d8/0x280 [ceph] [24062.632249] ? process_scheduled_works+0x22a/0x510 [24062.637040] process_scheduled_works+0x2a4/0x510 [24062.641656] worker_thread+0x188/0x340 [24062.645406] ? rescuer_thread+0x310/0x310 [24062.649418] kthread+0xf5/0x130 [24062.652554] ? kthread_complete_and_exit+0x20/0x20 [24062.657340] ret_from_fork+0x31/0x40 [24062.660916] ? kthread_complete_and_exit+0x20/0x20 [24062.665701] ret_from_fork_asm+0x11/0x20 [24062.669623] [24089.879964] watchdog: BUG: soft lockup - CPU#0 stuck for 11321s! [dbench:78683] [24089.887267] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24089.891962] watchdog: BUG: soft lockup - CPU#7 stuck for 20318s! [kworker/7:4:78808] [24089.942716] irq event stamp: 38111142 [24089.942718] hardirqs last enabled at (38111141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24089.950447] Modules linked in: [24089.954104] hardirqs last disabled at (38111142): [] sysvec_apic_timer_interrupt+0xe/0x90 [24089.964262] ceph [24089.967313] softirqs last enabled at (38111094): [] __do_softirq+0x2be/0x371 [24089.977037] libceph [24089.978961] softirqs last disabled at (38111089): [] irq_exit_rcu+0xb0/0xd0 [24089.987644] dns_resolver [24089.989830] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24089.998357] fscache netfs [24090.000975] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24090.009918] veth [24090.012625] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24090.020112] nft_chain_nat [24090.022035] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24090.027512] xt_MASQUERADE nf_nat [24090.030217] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24090.048955] nf_conntrack nf_defrag_ipv6 [24090.052265] [24090.052266] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24090.057483] nf_defrag_ipv4 [24090.061400] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24090.062891] nft_compat nf_tables [24090.070014] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24090.072805] nfnetlink [24090.079929] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24090.083241] bridge [24090.090364] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24090.092721] stp llc [24090.099844] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24090.101944] xfs libcrc32c [24090.109068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24090.111250] sunrpc intel_rapl_msr [24090.119328] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24090.122032] intel_rapl_common sb_edac [24090.127769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24090.131168] x86_pkg_temp_thermal coretemp [24090.138291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24090.142037] kvm_intel [24090.149161] Call Trace: [24090.149163] [24090.153251] kvm [24090.160377] ? watchdog_timer_fn+0x1b4/0x220 [24090.162732] irqbypass i2c_i801 [24090.165176] ? __hrtimer_run_queues+0x1d3/0x360 [24090.167187] iTCO_wdt [24090.169028] ? hrtimer_interrupt+0x104/0x240 [24090.173289] iTCO_vendor_support [24090.176424] ? __local_bh_enable+0x50/0x60 [24090.180951] i2c_smbus [24090.183222] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24090.187483] crct10dif_pclmul [24090.190710] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24090.194798] joydev crc32_pclmul [24090.197157] [24090.202375] ipmi_ssif [24090.205338] [24090.210314] ghash_clmulni_intel [24090.213539] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24090.215641] mei_me mei [24090.218005] ? queued_spin_lock_slowpath+0x16/0x2e0 [24090.220098] lpc_ich sha512_ssse3 [24090.223323] do_raw_spin_lock+0xb5/0xc0 [24090.228640] acpi_ipmi mfd_core [24090.231090] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24090.235959] wmi ipmi_si [24090.239271] try_get_cap_refs+0x159/0x880 [ceph] [24090.243101] ipmi_devintf [24090.246242] __ceph_get_caps+0x4a1/0x760 [ceph] [24090.251021] ipmi_msghandler [24090.253557] ? __file_remove_privs+0x89/0x110 [24090.258161] ioatdma acpi_pad [24090.260781] ? wake_bit_function+0x70/0x70 [24090.265303] acpi_power_meter ixgbe [24090.268183] ? ceph_write_iter+0x260/0x770 [ceph] [24090.272531] igb [24090.275496] ceph_write_iter+0x260/0x770 [ceph] [24090.279586] mdio crc32c_intel [24090.283075] ? lock_is_held_type+0xaa/0x120 [24090.287769] nvme ptp [24090.289609] vfs_write+0x324/0x4f0 [24090.294129] nvme_core i2c_algo_bit [24090.297186] ksys_pwrite64+0x65/0xa0 [24090.301365] pps_core [24090.303645] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24090.307042] dca fuse [24090.310528] do_syscall_64+0x38/0x80 [24090.314099] [24090.316370] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24090.321499] irq event stamp: 114929058 [24090.323772] RIP: 0033:0x7f48f0eeb93b [24090.327341] hardirqs last enabled at (114929057): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24090.328849] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24090.333893] hardirqs last disabled at (114929058): [] sysvec_apic_timer_interrupt+0xe/0x90 [24090.337645] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24090.341217] softirqs last enabled at (114929056): [] __do_softirq+0x2be/0x371 [24090.351460] ORIG_RAX: 0000000000000012 [24090.351461] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24090.370197] softirqs last disabled at (114929051): [] irq_exit_rcu+0xb0/0xd0 [24090.380010] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24090.385227] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24090.393997] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24090.397829] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24090.404954] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24090.413549] Workqueue: events delayed_work [ceph] [24090.420675] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24090.430053] [24090.437182] [24090.444647] RIP: 0010:lock_release+0x78/0x260 [24090.471619] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [24090.490367] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [24090.495591] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [24090.502715] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [24090.509839] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24090.516964] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24090.524087] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24090.531212] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24090.539290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24090.545025] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24090.552149] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24090.559275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24090.566399] Call Trace: [24090.568845] [24090.570856] ? watchdog_timer_fn+0x1b4/0x220 [24090.575128] ? __hrtimer_run_queues+0x1d3/0x360 [24090.579653] ? hrtimer_interrupt+0x104/0x240 [24090.583924] ? __local_bh_enable+0x50/0x60 [24090.588015] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24090.593241] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24090.598198] [24090.600295] [24090.602393] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24090.607696] ? igrab+0x43/0x50 [24090.610749] ? lock_release+0x78/0x260 [24090.614493] _raw_spin_unlock+0x1b/0x40 [24090.618333] igrab+0x43/0x50 [24090.621209] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24090.626374] delayed_work+0x1d8/0x280 [ceph] [24090.630658] ? process_scheduled_works+0x22a/0x510 [24090.635446] process_scheduled_works+0x2a4/0x510 [24090.640063] worker_thread+0x188/0x340 [24090.643811] ? rescuer_thread+0x310/0x310 [24090.647814] kthread+0xf5/0x130 [24090.650953] ? kthread_complete_and_exit+0x20/0x20 [24090.655737] ret_from_fork+0x31/0x40 [24090.659317] ? kthread_complete_and_exit+0x20/0x20 [24090.664100] ret_from_fork_asm+0x11/0x20 [24090.668022] [24117.880448] watchdog: BUG: soft lockup - CPU#0 stuck for 11347s! [dbench:78683] [24117.887753] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24117.892448] watchdog: BUG: soft lockup - CPU#7 stuck for 20344s! [kworker/7:4:78808] [24117.943193] irq event stamp: 38169462 [24117.943194] hardirqs last enabled at (38169461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24117.950926] Modules linked in: ceph [24117.954583] hardirqs last disabled at (38169462): [] sysvec_apic_timer_interrupt+0xe/0x90 [24117.964740] libceph [24117.968224] softirqs last enabled at (38169426): [] __do_softirq+0x2be/0x371 [24117.977947] dns_resolver [24117.980131] softirqs last disabled at (38169421): [] irq_exit_rcu+0xb0/0xd0 [24117.988816] fscache [24117.991450] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24117.999963] netfs veth [24118.002147] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24118.011089] nft_chain_nat [24118.013534] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24118.021005] xt_MASQUERADE nf_nat [24118.023710] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24118.029187] nf_conntrack nf_defrag_ipv6 [24118.032497] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24118.051234] nf_defrag_ipv4 nft_compat [24118.055151] [24118.055152] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24118.060369] nf_tables nfnetlink [24118.064124] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24118.065612] bridge stp [24118.072737] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24118.075961] llc xfs [24118.083086] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24118.085528] libcrc32c sunrpc [24118.092654] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24118.094847] intel_rapl_msr intel_rapl_common [24118.101970] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24118.104934] sb_edac x86_pkg_temp_thermal [24118.112058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24118.116408] coretemp kvm_intel [24118.124487] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24118.128492] kvm irqbypass [24118.134229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24118.137366] i2c_i801 iTCO_wdt [24118.144491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24118.147194] iTCO_vendor_support i2c_smbus [24118.154317] Call Trace: [24118.154319] [24118.157370] crct10dif_pclmul [24118.164492] ? watchdog_timer_fn+0x1b4/0x220 [24118.168583] joydev [24118.171030] ? __hrtimer_run_queues+0x1d3/0x360 [24118.173039] crc32_pclmul ipmi_ssif [24118.176007] ? hrtimer_interrupt+0x104/0x240 [24118.180267] ghash_clmulni_intel [24118.182363] ? __local_bh_enable+0x50/0x60 [24118.186888] mei_me [24118.190374] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24118.194637] mei [24118.197861] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24118.201951] lpc_ich sha512_ssse3 [24118.204048] [24118.209265] acpi_ipmi mfd_core [24118.211103] [24118.211105] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24118.216078] wmi ipmi_si [24118.219391] ? queued_spin_lock_slowpath+0x18/0x2e0 [24118.221485] ipmi_devintf ipmi_msghandler [24118.224625] do_raw_spin_lock+0xb5/0xc0 [24118.226721] ioatdma acpi_pad [24118.232027] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24118.234556] acpi_power_meter ixgbe [24118.239429] try_get_cap_refs+0x159/0x880 [ceph] [24118.243430] igb [24118.247265] __ceph_get_caps+0x4a1/0x760 [ceph] [24118.250226] mdio [24118.255013] ? __file_remove_privs+0x89/0x110 [24118.258492] crc32c_intel nvme [24118.263104] ? wake_bit_function+0x70/0x70 [24118.264958] ptp nvme_core [24118.269486] ? ceph_write_iter+0x260/0x770 [ceph] [24118.271406] i2c_algo_bit pps_core [24118.275759] ceph_write_iter+0x260/0x770 [ceph] [24118.278808] dca fuse [24118.282902] ? lock_is_held_type+0xaa/0x120 [24118.285604] [24118.285605] irq event stamp: 115149820 [24118.290303] vfs_write+0x324/0x4f0 [24118.293698] hardirqs last enabled at (115149819): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24118.298228] ksys_pwrite64+0x65/0xa0 [24118.300493] hardirqs last disabled at (115149820): [] sysvec_apic_timer_interrupt+0xe/0x90 [24118.304672] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24118.306162] softirqs last enabled at (115149818): [] __do_softirq+0x2be/0x371 [24118.309907] do_syscall_64+0x38/0x80 [24118.313301] softirqs last disabled at (115149813): [] irq_exit_rcu+0xb0/0xd0 [24118.323548] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24118.327118] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24118.336945] RIP: 0033:0x7f48f0eeb93b [24118.342077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24118.350852] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24118.354418] Workqueue: events delayed_work [ceph] [24118.363016] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24118.368059] [24118.368060] RIP: 0010:lock_acquire+0xf0/0x2b0 [24118.377436] ORIG_RAX: 0000000000000012 [24118.381008] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24118.388479] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24118.407217] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24118.411915] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24118.417147] [24118.417148] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24118.418638] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24118.422988] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24118.426820] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24118.445558] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24118.452683] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24118.457899] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24118.465030] [24118.466513] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24118.525699] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24118.533778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24118.539516] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24118.546639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24118.553764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24118.560888] Call Trace: [24118.563333] [24118.565343] ? watchdog_timer_fn+0x1b4/0x220 [24118.569608] ? __hrtimer_run_queues+0x1d3/0x360 [24118.574139] ? hrtimer_interrupt+0x104/0x240 [24118.578402] ? __local_bh_enable+0x50/0x60 [24118.582497] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24118.587722] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24118.592678] [24118.594775] [24118.596874] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24118.602182] ? lock_acquire+0xf0/0x2b0 [24118.605931] ? igrab+0x43/0x50 [24118.608981] ? lock_release+0xc5/0x260 [24118.612726] _raw_spin_lock+0x30/0x40 [24118.616390] ? igrab+0x1d/0x50 [24118.619442] igrab+0x1d/0x50 [24118.622320] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24118.627469] delayed_work+0x1d8/0x280 [ceph] [24118.631750] ? process_scheduled_works+0x22a/0x510 [24118.636540] process_scheduled_works+0x2a4/0x510 [24118.641156] worker_thread+0x188/0x340 [24118.644905] ? rescuer_thread+0x310/0x310 [24118.648909] kthread+0xf5/0x130 [24118.652046] ? kthread_complete_and_exit+0x20/0x20 [24118.656850] ret_from_fork+0x31/0x40 [24118.660426] ? kthread_complete_and_exit+0x20/0x20 [24118.665210] ret_from_fork_asm+0x11/0x20 [24118.669142] [24119.342839] rcu: INFO: rcu_preempt self-detected stall on CPU [24119.348599] rcu: 0-....: (11909530 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=5948404 [24119.359026] rcu: (t=12182516 jiffies g=1309609 q=387086 ncpus=8) [24119.365110] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24119.374053] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24119.381523] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24119.387002] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24119.405739] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24119.410955] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24119.418081] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24119.425204] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24119.432327] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24119.439451] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24119.446578] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24119.454655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24119.460391] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24119.467516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24119.474640] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24119.481763] Call Trace: [24119.484207] [24119.486220] ? rcu_dump_cpu_stacks+0xd1/0x180 [24119.490579] ? rcu_sched_clock_irq+0x633/0x1390 [24119.495103] ? lock_is_held_type+0xaa/0x120 [24119.499280] ? lock_is_held_type+0xaa/0x120 [24119.503457] ? tick_sched_do_timer+0xa0/0xa0 [24119.507731] ? rcu_read_lock_sched_held+0x56/0x90 [24119.512428] ? tick_sched_do_timer+0xa0/0xa0 [24119.516690] ? update_process_times+0x5f/0x90 [24119.521042] ? tick_sched_handle.isra.16+0x1e/0x50 [24119.525827] ? tick_sched_timer+0x6f/0x80 [24119.529830] ? __hrtimer_run_queues+0x1d3/0x360 [24119.534358] ? hrtimer_interrupt+0x104/0x240 [24119.538628] ? __local_bh_enable+0x50/0x60 [24119.542720] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24119.547944] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24119.552901] [24119.554999] [24119.557097] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24119.562404] ? queued_spin_lock_slowpath+0x16/0x2e0 [24119.567283] do_raw_spin_lock+0xb5/0xc0 [24119.571119] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24119.575939] try_get_cap_refs+0x159/0x880 [ceph] [24119.580571] __ceph_get_caps+0x4a1/0x760 [ceph] [24119.585112] ? __file_remove_privs+0x89/0x110 [24119.589467] ? wake_bit_function+0x70/0x70 [24119.593560] ? ceph_write_iter+0x260/0x770 [ceph] [24119.598272] ceph_write_iter+0x260/0x770 [ceph] [24119.602810] ? lock_is_held_type+0xaa/0x120 [24119.606993] vfs_write+0x324/0x4f0 [24119.610402] ksys_pwrite64+0x65/0xa0 [24119.613977] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24119.619109] do_syscall_64+0x38/0x80 [24119.622678] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24119.627721] RIP: 0033:0x7f48f0eeb93b [24119.631294] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24119.650030] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [24119.657587] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24119.664712] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24119.671835] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24119.678960] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24119.686084] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24119.693214] [24119.695402] Sending NMI from CPU 0 to CPUs 7: [24119.699753] NMI backtrace for cpu 7 [24119.699756] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24119.699758] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24119.699759] Workqueue: events delayed_work [ceph] [24119.699779] RIP: 0010:__lock_acquire+0x67a/0x1e20 [24119.699782] Code: 44 01 f8 c1 c2 08 41 89 fc 41 31 d4 44 89 e2 45 29 e7 41 01 c4 c1 c2 10 44 89 e1 41 31 d7 44 89 fa 44 29 f8 45 01 e7 c1 ca 0d <31> d0 89 c2 29 c1 44 01 f8 c1 c2 04 31 ca 48 c1 e2 20 48 09 c2 8b [24119.699783] RSP: 0018:ffffc90002117c60 EFLAGS: 00000086 [24119.699785] RAX: 0000000024a1c4fd RBX: ffff88810debaa80 RCX: 00000000ce5c2dd3 [24119.699786] RDX: 000000005da94362 RSI: 0000000000000000 RDI: 00000000f3668bb7 [24119.699788] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [24119.699789] R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ce5c2dd3 [24119.699790] R13: 0000000000000003 R14: 0000000000000000 R15: 00000000f6c87988 [24119.699791] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24119.699792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24119.699794] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24119.699795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24119.699796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24119.699797] Call Trace: [24119.699798] [24119.699799] ? nmi_cpu_backtrace+0xad/0x120 [24119.699803] ? nmi_cpu_backtrace_handler+0x11/0x20 [24119.699805] ? nmi_handle+0xd4/0x260 [24119.699810] ? default_do_nmi+0x49/0xf0 [24119.699813] ? exc_nmi+0x11e/0x150 [24119.699815] ? end_repeat_nmi+0x16/0x67 [24119.699821] ? __lock_acquire+0x67a/0x1e20 [24119.699823] ? __lock_acquire+0x67a/0x1e20 [24119.699826] ? __lock_acquire+0x67a/0x1e20 [24119.699828] [24119.699829] [24119.699832] lock_acquire+0xcb/0x2b0 [24119.699835] ? igrab+0x1d/0x50 [24119.699837] ? igrab+0x43/0x50 [24119.699839] ? lock_release+0xc5/0x260 [24119.699842] _raw_spin_lock+0x30/0x40 [24119.699844] ? igrab+0x1d/0x50 [24119.699846] igrab+0x1d/0x50 [24119.699848] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24119.699885] delayed_work+0x1d8/0x280 [ceph] [24119.699904] ? process_scheduled_works+0x22a/0x510 [24119.699907] process_scheduled_works+0x2a4/0x510 [24119.699913] worker_thread+0x188/0x340 [24119.699917] ? rescuer_thread+0x310/0x310 [24119.699920] kthread+0xf5/0x130 [24119.699923] ? kthread_complete_and_exit+0x20/0x20 [24119.699926] ret_from_fork+0x31/0x40 [24119.699929] ? kthread_complete_and_exit+0x20/0x20 [24119.699932] ret_from_fork_asm+0x11/0x20 [24119.699938] [24145.880936] watchdog: BUG: soft lockup - CPU#0 stuck for 11373s! [dbench:78683] [24145.888240] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24145.892936] watchdog: BUG: soft lockup - CPU#7 stuck for 20370s! [kworker/7:4:78808] [24145.943681] irq event stamp: 38226958 [24145.943682] hardirqs last enabled at (38226957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24145.951413] Modules linked in: [24145.955069] hardirqs last disabled at (38226958): [] sysvec_apic_timer_interrupt+0xe/0x90 [24145.965227] ceph [24145.968277] softirqs last enabled at (38226910): [] __do_softirq+0x2be/0x371 [24145.978002] libceph [24145.979926] softirqs last disabled at (38226905): [] irq_exit_rcu+0xb0/0xd0 [24145.988610] dns_resolver [24145.990794] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24145.999304] fscache netfs [24146.001939] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24146.001941] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24146.010884] veth nft_chain_nat [24146.013587] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24146.021059] xt_MASQUERADE nf_nat [24146.026537] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24146.029673] nf_conntrack nf_defrag_ipv6 [24146.048410] [24146.051720] nf_defrag_ipv4 [24146.056939] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24146.060856] nft_compat nf_tables [24146.062347] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24146.065137] nfnetlink bridge [24146.072263] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24146.075573] stp llc [24146.082696] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24146.085661] xfs libcrc32c [24146.092785] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24146.094971] sunrpc [24146.102094] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24146.104798] intel_rapl_msr intel_rapl_common [24146.111921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24146.114018] sb_edac x86_pkg_temp_thermal [24146.122097] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24146.126447] coretemp kvm_intel [24146.132185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24146.136189] kvm irqbypass [24146.143330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24146.146468] i2c_i801 iTCO_wdt [24146.153591] Call Trace: [24146.153592] [24146.156295] iTCO_vendor_support i2c_smbus [24146.163421] ? watchdog_timer_fn+0x1b4/0x220 [24146.166470] crct10dif_pclmul joydev [24146.168915] ? __hrtimer_run_queues+0x1d3/0x360 [24146.170925] crc32_pclmul [24146.175020] ? hrtimer_interrupt+0x104/0x240 [24146.179281] ipmi_ssif ghash_clmulni_intel [24146.182851] ? __local_bh_enable+0x50/0x60 [24146.187375] mei_me mei [24146.189995] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24146.194257] lpc_ich sha512_ssse3 [24146.198348] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24146.202437] acpi_ipmi [24146.204883] [24146.210100] mfd_core wmi [24146.213410] [24146.218368] ipmi_si ipmi_devintf [24146.220725] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24146.222822] ipmi_msghandler ioatdma [24146.225443] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24146.227537] acpi_pad [24146.230849] do_raw_spin_lock+0xb5/0xc0 [24146.236153] acpi_power_meter ixgbe [24146.239723] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24146.244593] igb mdio [24146.246866] try_get_cap_refs+0x159/0x880 [ceph] [24146.250694] crc32c_intel [24146.254185] __ceph_get_caps+0x4a1/0x760 [ceph] [24146.258962] nvme ptp [24146.261239] ? __file_remove_privs+0x89/0x110 [24146.265845] nvme_core i2c_algo_bit [24146.268463] ? wake_bit_function+0x70/0x70 [24146.272987] pps_core dca [24146.275258] ? ceph_write_iter+0x260/0x770 [ceph] [24146.279614] fuse [24146.283101] ceph_write_iter+0x260/0x770 [ceph] [24146.287190] irq event stamp: 115368808 [24146.289812] ? lock_is_held_type+0xaa/0x120 [24146.294506] hardirqs last enabled at (115368807): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24146.296431] vfs_write+0x324/0x4f0 [24146.300953] hardirqs last disabled at (115368808): [] sysvec_apic_timer_interrupt+0xe/0x90 [24146.304701] ksys_pwrite64+0x65/0xa0 [24146.308874] softirqs last enabled at (115368806): [] __do_softirq+0x2be/0x371 [24146.319119] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24146.322518] softirqs last disabled at (115368801): [] irq_exit_rcu+0xb0/0xd0 [24146.332339] do_syscall_64+0x38/0x80 [24146.335908] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24146.344679] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24146.349808] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24146.358405] RIP: 0033:0x7f48f0eeb93b [24146.361976] Workqueue: events delayed_work [ceph] [24146.371355] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24146.376397] [24146.383868] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24146.387439] RIP: 0010:lock_acquire+0xf0/0x2b0 [24146.392137] ORIG_RAX: 0000000000000012 [24146.410875] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24146.412365] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24146.417583] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24146.421935] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24146.425764] [24146.444502] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24146.451629] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24146.456845] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24146.463967] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24146.465460] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24146.472584] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24146.479712] [24146.486831] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24146.517511] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24146.524635] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24146.532713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24146.538451] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24146.545576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24146.552700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24146.559823] Call Trace: [24146.562267] [24146.564278] ? watchdog_timer_fn+0x1b4/0x220 [24146.568544] ? __hrtimer_run_queues+0x1d3/0x360 [24146.573069] ? hrtimer_interrupt+0x104/0x240 [24146.577342] ? __local_bh_enable+0x50/0x60 [24146.581440] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24146.586655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24146.591629] [24146.593728] [24146.595825] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24146.601135] ? lock_acquire+0xf0/0x2b0 [24146.604885] ? igrab+0x43/0x50 [24146.607935] ? lock_release+0xc5/0x260 [24146.611679] _raw_spin_lock+0x30/0x40 [24146.615344] ? igrab+0x1d/0x50 [24146.618395] igrab+0x1d/0x50 [24146.621273] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24146.626421] delayed_work+0x1d8/0x280 [ceph] [24146.630704] ? process_scheduled_works+0x22a/0x510 [24146.635494] process_scheduled_works+0x2a4/0x510 [24146.640110] worker_thread+0x188/0x340 [24146.643857] ? rescuer_thread+0x310/0x310 [24146.647861] kthread+0xf5/0x130 [24146.650999] ? kthread_complete_and_exit+0x20/0x20 [24146.655784] ret_from_fork+0x31/0x40 [24146.659353] ? kthread_complete_and_exit+0x20/0x20 [24146.664137] ret_from_fork_asm+0x11/0x20 [24146.668061] [24173.881427] watchdog: BUG: soft lockup - CPU#0 stuck for 11399s! [dbench:78683] [24173.888728] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24173.893426] watchdog: BUG: soft lockup - CPU#7 stuck for 20396s! [kworker/7:4:78808] [24173.944178] irq event stamp: 38285148 [24173.951909] Modules linked in: ceph libceph dns_resolver fscache [24173.955565] hardirqs last enabled at (38285147): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24173.955569] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [24173.971727] hardirqs last disabled at (38285148): [] sysvec_apic_timer_interrupt+0xe/0x90 [24173.978334] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [24173.988056] softirqs last enabled at (38285100): [] __do_softirq+0x2be/0x371 [24173.995787] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [24174.004471] softirqs last disabled at (38285095): [] irq_exit_rcu+0xb0/0xd0 [24174.010989] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [24174.019502] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24174.019503] kvm [24174.019504] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24174.025150] irqbypass i2c_i801 [24174.034105] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24174.035942] iTCO_wdt iTCO_vendor_support [24174.043414] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24174.046550] i2c_smbus crct10dif_pclmul [24174.052027] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24174.056032] joydev crc32_pclmul [24174.074768] [24174.078599] ipmi_ssif ghash_clmulni_intel [24174.083821] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24174.087040] mei_me mei [24174.088531] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24174.092622] lpc_ich [24174.099745] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24174.102189] sha512_ssse3 acpi_ipmi [24174.109314] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24174.111499] mfd_core [24174.118623] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24174.122106] wmi ipmi_si [24174.129230] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24174.131500] ipmi_devintf [24174.138626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24174.141157] ipmi_msghandler ioatdma [24174.149233] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24174.151850] acpi_pad acpi_power_meter [24174.157589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24174.161161] ixgbe igb [24174.168285] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24174.172027] mdio [24174.179151] Call Trace: [24174.181508] crc32c_intel nvme [24174.188635] [24174.190557] ptp [24174.193003] ? watchdog_timer_fn+0x1b4/0x220 [24174.196051] nvme_core i2c_algo_bit [24174.198064] ? __hrtimer_run_queues+0x1d3/0x360 [24174.199900] pps_core [24174.204168] ? hrtimer_interrupt+0x104/0x240 [24174.207649] dca fuse [24174.212172] ? __local_bh_enable+0x50/0x60 [24174.214444] [24174.214445] irq event stamp: 115589592 [24174.218709] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24174.220979] hardirqs last enabled at (115589591): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24174.225069] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24174.226560] hardirqs last disabled at (115589592): [] sysvec_apic_timer_interrupt+0xe/0x90 [24174.230303] [24174.235521] softirqs last enabled at (115589590): [] __do_softirq+0x2be/0x371 [24174.245765] [24174.250722] softirqs last disabled at (115589585): [] irq_exit_rcu+0xb0/0xd0 [24174.260535] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24174.262630] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24174.271405] ? queued_spin_lock_slowpath+0x18/0x2e0 [24174.273500] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24174.282114] do_raw_spin_lock+0xb5/0xc0 [24174.287418] Workqueue: events delayed_work [ceph] [24174.296797] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24174.301666] [24174.301667] RIP: 0010:lock_acquire+0xf0/0x2b0 [24174.309140] try_get_cap_refs+0x159/0x880 [ceph] [24174.312967] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24174.317670] __ceph_get_caps+0x4a1/0x760 [ceph] [24174.322450] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24174.323945] ? __file_remove_privs+0x89/0x110 [24174.328292] [24174.328292] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24174.332902] ? wake_bit_function+0x70/0x70 [24174.351640] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24174.356165] ? ceph_write_iter+0x260/0x770 [ceph] [24174.361380] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24174.365732] ceph_write_iter+0x260/0x770 [ceph] [24174.367223] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24174.374351] ? lock_is_held_type+0xaa/0x120 [24174.378437] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24174.385564] vfs_write+0x324/0x4f0 [24174.390258] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24174.397388] ksys_pwrite64+0x65/0xa0 [24174.401907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24174.409032] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24174.413209] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24174.420334] do_syscall_64+0x38/0x80 [24174.423730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24174.431826] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24174.435396] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24174.441134] RIP: 0033:0x7f48f0eeb93b [24174.446263] Call Trace: [24174.446265] [24174.453388] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24174.456960] ? watchdog_timer_fn+0x1b4/0x220 [24174.464084] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24174.469146] ? __hrtimer_run_queues+0x1d3/0x360 [24174.476268] ORIG_RAX: 0000000000000012 [24174.479843] ? hrtimer_interrupt+0x104/0x240 [24174.482282] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24174.484293] ? __local_bh_enable+0x50/0x60 [24174.503032] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24174.507299] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24174.512512] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24174.517037] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24174.520867] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24174.525131] [24174.532256] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24174.536345] [24174.536347] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24174.543477] [24174.548691] ? lock_acquire+0xf0/0x2b0 [24174.590465] ? igrab+0x43/0x50 [24174.593513] ? lock_release+0xc5/0x260 [24174.597259] _raw_spin_lock+0x30/0x40 [24174.600916] ? igrab+0x1d/0x50 [24174.603966] igrab+0x1d/0x50 [24174.606845] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24174.611995] delayed_work+0x1d8/0x280 [ceph] [24174.616275] ? process_scheduled_works+0x22a/0x510 [24174.621064] process_scheduled_works+0x2a4/0x510 [24174.625680] worker_thread+0x188/0x340 [24174.629430] ? rescuer_thread+0x310/0x310 [24174.633432] kthread+0xf5/0x130 [24174.636571] ? kthread_complete_and_exit+0x20/0x20 [24174.641356] ret_from_fork+0x31/0x40 [24174.644935] ? kthread_complete_and_exit+0x20/0x20 [24174.649718] ret_from_fork_asm+0x11/0x20 [24174.653650] [24201.881916] watchdog: BUG: soft lockup - CPU#0 stuck for 11425s! [dbench:78683] [24201.889215] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24201.893916] watchdog: BUG: soft lockup - CPU#7 stuck for 20422s! [kworker/7:4:78808] [24201.944655] irq event stamp: 38343238 [24201.944656] hardirqs last enabled at (38343237): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24201.952385] Modules linked in: [24201.956044] hardirqs last disabled at (38343238): [] sysvec_apic_timer_interrupt+0xe/0x90 [24201.966202] ceph [24201.969251] softirqs last enabled at (38343190): [] __do_softirq+0x2be/0x371 [24201.978976] libceph [24201.980901] softirqs last disabled at (38343185): [] irq_exit_rcu+0xb0/0xd0 [24201.989599] dns_resolver fscache [24201.991786] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24202.000309] netfs veth [24202.003616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24202.012560] nft_chain_nat [24202.015004] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24202.022476] xt_MASQUERADE [24202.025178] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24202.030657] nf_nat nf_conntrack [24202.033361] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24202.052097] nf_defrag_ipv6 nf_defrag_ipv4 [24202.055322] [24202.060539] nft_compat nf_tables [24202.064629] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24202.066121] nfnetlink bridge [24202.069432] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24202.076557] stp llc [24202.079521] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24202.086643] xfs libcrc32c [24202.088827] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24202.095951] sunrpc intel_rapl_msr [24202.098656] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24202.105780] intel_rapl_common sb_edac [24202.109177] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24202.116312] x86_pkg_temp_thermal coretemp [24202.120054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24202.128133] kvm_intel kvm [24202.132222] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24202.137959] irqbypass i2c_i801 [24202.140663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24202.147789] iTCO_wdt [24202.150925] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24202.158049] iTCO_vendor_support [24202.160321] Call Trace: [24202.160323] [24202.167444] i2c_smbus crct10dif_pclmul [24202.170670] ? watchdog_timer_fn+0x1b4/0x220 [24202.173113] joydev crc32_pclmul [24202.175124] ? __hrtimer_run_queues+0x1d3/0x360 [24202.178953] ipmi_ssif ghash_clmulni_intel [24202.183222] ? hrtimer_interrupt+0x104/0x240 [24202.186442] mei_me mei [24202.190966] ? __local_bh_enable+0x50/0x60 [24202.195057] lpc_ich sha512_ssse3 [24202.199323] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24202.201767] acpi_ipmi mfd_core [24202.205858] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24202.209167] wmi ipmi_si [24202.214385] [24202.217522] ipmi_devintf ipmi_msghandler [24202.222478] [24202.222480] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24202.225009] ioatdma acpi_pad [24202.227111] ? queued_spin_lock_slowpath+0x16/0x2e0 [24202.231111] acpi_power_meter ixgbe [24202.233211] do_raw_spin_lock+0xb5/0xc0 [24202.238513] igb mdio [24202.241479] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24202.246347] crc32c_intel nvme [24202.249835] try_get_cap_refs+0x159/0x880 [ceph] [24202.253662] ptp nvme_core [24202.255938] __ceph_get_caps+0x4a1/0x760 [ceph] [24202.260718] i2c_algo_bit [24202.263773] ? __file_remove_privs+0x89/0x110 [24202.268379] pps_core [24202.271084] ? wake_bit_function+0x70/0x70 [24202.275607] dca fuse [24202.278227] ? ceph_write_iter+0x260/0x770 [ceph] [24202.282576] [24202.282577] irq event stamp: 115810688 [24202.284846] ceph_write_iter+0x260/0x770 [ceph] [24202.288937] hardirqs last enabled at (115810687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24202.291212] ? lock_is_held_type+0xaa/0x120 [24202.295903] hardirqs last disabled at (115810688): [] sysvec_apic_timer_interrupt+0xe/0x90 [24202.297399] vfs_write+0x324/0x4f0 [24202.301139] softirqs last enabled at (115810686): [] __do_softirq+0x2be/0x371 [24202.305668] ksys_pwrite64+0x65/0xa0 [24202.315909] softirqs last disabled at (115810681): [] irq_exit_rcu+0xb0/0xd0 [24202.320084] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24202.329896] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24202.333313] do_syscall_64+0x38/0x80 [24202.342083] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24202.345653] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24202.354249] Workqueue: events delayed_work [ceph] [24202.359381] RIP: 0033:0x7f48f0eeb93b [24202.368759] [24202.368759] RIP: 0010:lock_acquire+0xf0/0x2b0 [24202.372329] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24202.379799] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24202.384843] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24202.389541] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24202.393113] ORIG_RAX: 0000000000000012 [24202.394601] [24202.394602] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24202.398954] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24202.417690] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24202.436428] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24202.441647] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24202.446863] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24202.450694] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24202.452185] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24202.459319] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24202.466441] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24202.473566] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24202.480696] [24202.487814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24202.539444] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24202.546567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24202.553690] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24202.560816] Call Trace: [24202.563261] [24202.565271] ? watchdog_timer_fn+0x1b4/0x220 [24202.569535] ? __hrtimer_run_queues+0x1d3/0x360 [24202.574061] ? hrtimer_interrupt+0x104/0x240 [24202.578332] ? __local_bh_enable+0x50/0x60 [24202.582425] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24202.587649] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24202.592611] [24202.594713] [24202.596808] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24202.602117] ? lock_acquire+0xf0/0x2b0 [24202.605866] ? igrab+0x43/0x50 [24202.608917] ? lock_release+0xc5/0x260 [24202.612663] _raw_spin_lock+0x30/0x40 [24202.616327] ? igrab+0x1d/0x50 [24202.619377] igrab+0x1d/0x50 [24202.622256] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24202.627405] delayed_work+0x1d8/0x280 [ceph] [24202.631686] ? process_scheduled_works+0x22a/0x510 [24202.636478] process_scheduled_works+0x2a4/0x510 [24202.641092] worker_thread+0x188/0x340 [24202.644840] ? rescuer_thread+0x310/0x310 [24202.648845] kthread+0xf5/0x130 [24202.651982] ? kthread_complete_and_exit+0x20/0x20 [24202.656768] ret_from_fork+0x31/0x40 [24202.660345] ? kthread_complete_and_exit+0x20/0x20 [24202.665130] ret_from_fork_asm+0x11/0x20 [24202.669051] [24229.882406] watchdog: BUG: soft lockup - CPU#0 stuck for 11451s! [dbench:78683] [24229.889711] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24229.894406] watchdog: BUG: soft lockup - CPU#7 stuck for 20448s! [kworker/7:4:78808] [24229.945161] irq event stamp: 38401454 [24229.945162] hardirqs last enabled at (38401453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24229.952890] Modules linked in: [24229.956549] hardirqs last disabled at (38401454): [] sysvec_apic_timer_interrupt+0xe/0x90 [24229.966705] ceph [24229.969757] softirqs last enabled at (38401406): [] __do_softirq+0x2be/0x371 [24229.979481] libceph [24229.981403] softirqs last disabled at (38401401): [] irq_exit_rcu+0xb0/0xd0 [24229.990090] dns_resolver fscache [24229.992275] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24230.000801] netfs veth [24230.004112] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24230.013055] nft_chain_nat [24230.015519] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24230.022988] xt_MASQUERADE nf_nat [24230.025694] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24230.031170] nf_conntrack nf_defrag_ipv6 [24230.034480] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24230.053218] nf_defrag_ipv4 [24230.057135] [24230.057136] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24230.062352] nft_compat nf_tables [24230.065143] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24230.066633] nfnetlink bridge [24230.073757] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24230.077069] stp [24230.084211] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24230.087173] llc xfs [24230.094299] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24230.096135] libcrc32c sunrpc [24230.103260] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24230.105444] intel_rapl_msr intel_rapl_common [24230.112567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24230.115532] sb_edac x86_pkg_temp_thermal [24230.123611] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24230.127962] coretemp kvm_intel [24230.133698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24230.137703] kvm irqbypass [24230.144827] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24230.147962] i2c_i801 iTCO_wdt [24230.155087] Call Trace: [24230.155089] [24230.157804] iTCO_vendor_support [24230.164927] ? watchdog_timer_fn+0x1b4/0x220 [24230.167976] i2c_smbus crct10dif_pclmul [24230.170421] ? __hrtimer_run_queues+0x1d3/0x360 [24230.172431] joydev crc32_pclmul [24230.175658] ? hrtimer_interrupt+0x104/0x240 [24230.179918] ipmi_ssif ghash_clmulni_intel [24230.183749] ? __local_bh_enable+0x50/0x60 [24230.188274] mei_me mei [24230.191499] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24230.195761] lpc_ich sha512_ssse3 [24230.199852] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24230.203943] acpi_ipmi mfd_core [24230.206386] [24230.211605] wmi ipmi_si [24230.214916] [24230.214917] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24230.219872] ipmi_devintf ipmi_msghandler [24230.223014] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24230.225107] ioatdma acpi_pad [24230.227641] do_raw_spin_lock+0xb5/0xc0 [24230.229735] acpi_power_meter ixgbe [24230.235040] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24230.239045] igb [24230.243918] try_get_cap_refs+0x159/0x880 [ceph] [24230.246879] mdio crc32c_intel [24230.250714] __ceph_get_caps+0x4a1/0x760 [ceph] [24230.254192] nvme ptp [24230.258982] ? __file_remove_privs+0x89/0x110 [24230.260815] nvme_core [24230.265425] ? wake_bit_function+0x70/0x70 [24230.268476] i2c_algo_bit [24230.273003] ? ceph_write_iter+0x260/0x770 [ceph] [24230.275270] pps_core [24230.279623] ceph_write_iter+0x260/0x770 [ceph] [24230.281978] dca fuse [24230.286090] ? lock_is_held_type+0xaa/0x120 [24230.288704] [24230.288705] irq event stamp: 116031464 [24230.293405] vfs_write+0x324/0x4f0 [24230.295673] hardirqs last enabled at (116031463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24230.300202] ksys_pwrite64+0x65/0xa0 [24230.302467] hardirqs last disabled at (116031464): [] sysvec_apic_timer_interrupt+0xe/0x90 [24230.306645] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24230.308136] softirqs last enabled at (116031462): [] __do_softirq+0x2be/0x371 [24230.311880] do_syscall_64+0x38/0x80 [24230.315278] softirqs last disabled at (116031457): [] irq_exit_rcu+0xb0/0xd0 [24230.325522] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24230.329104] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24230.338912] RIP: 0033:0x7f48f0eeb93b [24230.344042] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24230.352815] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24230.356384] Workqueue: events delayed_work [ceph] [24230.364982] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24230.370025] [24230.370026] RIP: 0010:lock_acquire+0xf0/0x2b0 [24230.379403] ORIG_RAX: 0000000000000012 [24230.382973] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24230.390443] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24230.409181] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24230.413879] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24230.419105] [24230.419106] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24230.420596] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24230.424946] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24230.428778] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24230.447516] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24230.454640] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24230.459856] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24230.466987] [24230.468472] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24230.527666] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24230.535743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24230.541481] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24230.548606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24230.555729] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24230.562855] Call Trace: [24230.565298] [24230.567310] ? watchdog_timer_fn+0x1b4/0x220 [24230.571573] ? __hrtimer_run_queues+0x1d3/0x360 [24230.576099] ? hrtimer_interrupt+0x104/0x240 [24230.580361] ? __local_bh_enable+0x50/0x60 [24230.584453] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24230.589670] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24230.594627] [24230.596723] [24230.598822] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24230.604130] ? lock_acquire+0xf0/0x2b0 [24230.607880] ? igrab+0x43/0x50 [24230.610928] ? lock_release+0xc5/0x260 [24230.614675] _raw_spin_lock+0x30/0x40 [24230.618340] ? igrab+0x1d/0x50 [24230.621390] igrab+0x1d/0x50 [24230.624270] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24230.629419] delayed_work+0x1d8/0x280 [ceph] [24230.633698] ? process_scheduled_works+0x22a/0x510 [24230.638490] process_scheduled_works+0x2a4/0x510 [24230.643105] worker_thread+0x188/0x340 [24230.646855] ? rescuer_thread+0x310/0x310 [24230.650857] kthread+0xf5/0x130 [24230.653996] ? kthread_complete_and_exit+0x20/0x20 [24230.658780] ret_from_fork+0x31/0x40 [24230.662376] ? kthread_complete_and_exit+0x20/0x20 [24230.667161] ret_from_fork_asm+0x11/0x20 [24230.671099] [24257.882896] watchdog: BUG: soft lockup - CPU#0 stuck for 11477s! [dbench:78683] [24257.890197] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24257.894896] watchdog: BUG: soft lockup - CPU#7 stuck for 20474s! [kworker/7:4:78808] [24257.945639] irq event stamp: 38459728 [24257.953370] Modules linked in: ceph libceph dns_resolver [24257.957027] hardirqs last enabled at (38459727): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24257.962340] fscache netfs veth [24257.972498] hardirqs last disabled at (38459728): [] sysvec_apic_timer_interrupt+0xe/0x90 [24257.975634] nft_chain_nat xt_MASQUERADE [24257.985359] softirqs last enabled at (38459680): [] __do_softirq+0x2be/0x371 [24257.989293] nf_nat nf_conntrack nf_defrag_ipv6 [24257.997978] softirqs last disabled at (38459675): [] irq_exit_rcu+0xb0/0xd0 [24257.997980] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [24258.011013] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24258.017097] stp llc [24258.026040] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24258.026041] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24258.028224] xfs libcrc32c [24258.035696] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24258.041172] sunrpc intel_rapl_msr [24258.043877] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24258.062614] intel_rapl_common [24258.066011] [24258.066012] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24258.071230] sb_edac [24258.074292] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24258.074293] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24258.075779] x86_pkg_temp_thermal [24258.082904] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24258.085087] coretemp [24258.092212] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24258.099337] kvm_intel [24258.102646] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24258.109770] kvm irqbypass [24258.112042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24258.119166] i2c_i801 iTCO_wdt [24258.121524] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24258.129601] iTCO_vendor_support [24258.132304] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24258.138041] i2c_smbus crct10dif_pclmul [24258.141092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24258.148216] joydev crc32_pclmul [24258.151440] Call Trace: [24258.151442] [24258.158573] ipmi_ssif ghash_clmulni_intel [24258.162407] ? watchdog_timer_fn+0x1b4/0x220 [24258.169527] mei_me [24258.172754] ? __hrtimer_run_queues+0x1d3/0x360 [24258.175196] mei [24258.177210] ? hrtimer_interrupt+0x104/0x240 [24258.181297] lpc_ich sha512_ssse3 [24258.185575] ? __local_bh_enable+0x50/0x60 [24258.187668] acpi_ipmi mfd_core [24258.192195] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24258.194030] wmi [24258.198297] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24258.201603] ipmi_si ipmi_devintf [24258.205696] [24258.208833] ipmi_msghandler ioatdma [24258.214050] [24258.214052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24258.215887] acpi_pad acpi_power_meter [24258.220849] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24258.224154] ixgbe igb [24258.226255] do_raw_spin_lock+0xb5/0xc0 [24258.229822] mdio crc32c_intel [24258.231922] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24258.237224] nvme ptp [24258.240972] try_get_cap_refs+0x159/0x880 [ceph] [24258.245840] nvme_core i2c_algo_bit [24258.248203] __ceph_get_caps+0x4a1/0x760 [ceph] [24258.252029] pps_core dca [24258.255082] ? __file_remove_privs+0x89/0x110 [24258.259863] fuse [24258.262134] ? wake_bit_function+0x70/0x70 [24258.266744] irq event stamp: 116252374 [24258.270231] ? ceph_write_iter+0x260/0x770 [ceph] [24258.274753] hardirqs last enabled at (116252373): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24258.277370] ceph_write_iter+0x260/0x770 [ceph] [24258.281721] hardirqs last disabled at (116252374): [] sysvec_apic_timer_interrupt+0xe/0x90 [24258.283648] ? lock_is_held_type+0xaa/0x120 [24258.287734] softirqs last enabled at (116252372): [] __do_softirq+0x2be/0x371 [24258.291483] vfs_write+0x324/0x4f0 [24258.296177] softirqs last disabled at (116252367): [] irq_exit_rcu+0xb0/0xd0 [24258.306425] ksys_pwrite64+0x65/0xa0 [24258.310944] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24258.320755] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24258.324933] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24258.333706] do_syscall_64+0x38/0x80 [24258.337102] Workqueue: events delayed_work [ceph] [24258.345699] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24258.349271] [24258.349272] RIP: 0010:lock_acquire+0xf0/0x2b0 [24258.358666] RIP: 0033:0x7f48f0eeb93b [24258.363796] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24258.371266] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24258.374838] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24258.379533] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24258.384590] [24258.384590] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24258.386078] ORIG_RAX: 0000000000000012 [24258.390427] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24258.393999] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24258.412736] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24258.431473] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24258.436691] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24258.441910] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24258.443400] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24258.450524] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24258.454354] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24258.461479] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24258.468604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24258.475732] [24258.482851] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24258.541603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24258.548727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24258.555853] Call Trace: [24258.558302] [24258.560317] ? watchdog_timer_fn+0x1b4/0x220 [24258.564596] ? __hrtimer_run_queues+0x1d3/0x360 [24258.569126] ? hrtimer_interrupt+0x104/0x240 [24258.573395] ? __local_bh_enable+0x50/0x60 [24258.577488] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24258.582712] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24258.587669] [24258.589765] [24258.591863] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24258.597173] ? lock_acquire+0xf0/0x2b0 [24258.600922] ? igrab+0x43/0x50 [24258.603971] ? lock_release+0xc5/0x260 [24258.607716] _raw_spin_lock+0x30/0x40 [24258.611383] ? igrab+0x1d/0x50 [24258.614433] igrab+0x1d/0x50 [24258.617309] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24258.622460] delayed_work+0x1d8/0x280 [ceph] [24258.626740] ? process_scheduled_works+0x22a/0x510 [24258.631531] process_scheduled_works+0x2a4/0x510 [24258.636146] worker_thread+0x188/0x340 [24258.639896] ? rescuer_thread+0x310/0x310 [24258.643906] kthread+0xf5/0x130 [24258.647046] ? kthread_complete_and_exit+0x20/0x20 [24258.651830] ret_from_fork+0x31/0x40 [24258.655409] ? kthread_complete_and_exit+0x20/0x20 [24258.660193] ret_from_fork_asm+0x11/0x20 [24258.664116] [24285.883385] watchdog: BUG: soft lockup - CPU#0 stuck for 11503s! [dbench:78683] [24285.890693] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24285.895384] watchdog: BUG: soft lockup - CPU#7 stuck for 20501s! [kworker/7:4:78808] [24285.946143] irq event stamp: 38517976 [24285.953873] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [24285.957530] hardirqs last enabled at (38517975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24285.957536] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [24285.975860] hardirqs last disabled at (38517976): [] sysvec_apic_timer_interrupt+0xe/0x90 [24285.975863] nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc [24285.993577] softirqs last enabled at (38517928): [] __do_softirq+0x2be/0x371 [24285.999928] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [24286.008612] softirqs last disabled at (38517923): [] irq_exit_rcu+0xb0/0xd0 [24286.017471] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support [24286.025983] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24286.031632] i2c_smbus crct10dif_pclmul [24286.040576] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24286.040578] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24286.044406] joydev crc32_pclmul [24286.051878] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24286.057356] ipmi_ssif ghash_clmulni_intel [24286.060579] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24286.079316] mei_me mei [24286.083408] [24286.088624] lpc_ich sha512_ssse3 [24286.091070] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24286.092560] acpi_ipmi mfd_core [24286.095870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24286.102995] wmi ipmi_si [24286.106131] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24286.113257] ipmi_devintf [24286.115786] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24286.122911] ipmi_msghandler [24286.125529] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24286.132652] ioatdma acpi_pad [24286.135529] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24286.142653] acpi_power_meter [24286.145617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24286.153695] ixgbe [24286.156659] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24286.162397] igb [24286.164408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24286.171531] mdio crc32c_intel [24286.173370] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24286.180494] nvme [24286.183544] Call Trace: [24286.183546] [24286.190669] ptp nvme_core [24286.192595] ? watchdog_timer_fn+0x1b4/0x220 [24286.195036] i2c_algo_bit pps_core [24286.197064] ? __hrtimer_run_queues+0x1d3/0x360 [24286.199785] dca [24286.204067] ? hrtimer_interrupt+0x104/0x240 [24286.207465] fuse [24286.211988] ? __local_bh_enable+0x50/0x60 [24286.213825] irq event stamp: 116473214 [24286.218093] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24286.220013] hardirqs last enabled at (116473213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24286.224106] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24286.227850] hardirqs last disabled at (116473214): [] sysvec_apic_timer_interrupt+0xe/0x90 [24286.233066] [24286.243311] softirqs last enabled at (116473212): [] __do_softirq+0x2be/0x371 [24286.248268] [24286.258079] softirqs last disabled at (116473207): [] irq_exit_rcu+0xb0/0xd0 [24286.260178] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24286.268948] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24286.271066] ? queued_spin_lock_slowpath+0x18/0x2e0 [24286.279658] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24286.284966] do_raw_spin_lock+0xb5/0xc0 [24286.294343] Workqueue: events delayed_work [ceph] [24286.299214] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24286.306682] [24286.306683] RIP: 0010:lock_acquire+0xf0/0x2b0 [24286.310516] try_get_cap_refs+0x159/0x880 [ceph] [24286.315211] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24286.320000] __ceph_get_caps+0x4a1/0x760 [ceph] [24286.321485] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24286.325842] ? __file_remove_privs+0x89/0x110 [24286.330447] [24286.349185] ? wake_bit_function+0x70/0x70 [24286.353708] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24286.358928] ? ceph_write_iter+0x260/0x770 [ceph] [24286.363276] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24286.364784] ceph_write_iter+0x260/0x770 [ceph] [24286.368876] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24286.376003] ? lock_is_held_type+0xaa/0x120 [24286.380698] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24286.387824] vfs_write+0x324/0x4f0 [24286.392345] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24286.399474] ksys_pwrite64+0x65/0xa0 [24286.403648] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24286.410788] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24286.414185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24286.421311] do_syscall_64+0x38/0x80 [24286.424881] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24286.432961] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24286.438089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24286.443828] RIP: 0033:0x7f48f0eeb93b [24286.447396] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24286.454524] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24286.459565] Call Trace: [24286.459566] [24286.466690] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24286.470260] ? watchdog_timer_fn+0x1b4/0x220 [24286.477384] ORIG_RAX: 0000000000000012 [24286.496122] ? __hrtimer_run_queues+0x1d3/0x360 [24286.498566] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24286.500580] ? hrtimer_interrupt+0x104/0x240 [24286.505793] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24286.510057] ? __local_bh_enable+0x50/0x60 [24286.513889] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24286.518416] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24286.525536] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24286.529802] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24286.536926] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24286.541016] [24286.548145] [24286.553357] [24286.578943] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24286.584251] ? lock_acquire+0xf0/0x2b0 [24286.588000] ? igrab+0x43/0x50 [24286.591050] ? lock_release+0xc5/0x260 [24286.594814] _raw_spin_lock+0x30/0x40 [24286.598478] ? igrab+0x1d/0x50 [24286.601528] igrab+0x1d/0x50 [24286.604407] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24286.609557] delayed_work+0x1d8/0x280 [ceph] [24286.613838] ? process_scheduled_works+0x22a/0x510 [24286.618629] process_scheduled_works+0x2a4/0x510 [24286.623243] worker_thread+0x188/0x340 [24286.626992] ? rescuer_thread+0x310/0x310 [24286.630997] kthread+0xf5/0x130 [24286.634144] ? kthread_complete_and_exit+0x20/0x20 [24286.638934] ret_from_fork+0x31/0x40 [24286.642514] ? kthread_complete_and_exit+0x20/0x20 [24286.647299] ret_from_fork_asm+0x11/0x20 [24286.651229] [24313.883873] watchdog: BUG: soft lockup - CPU#0 stuck for 11530s! [dbench:78683] [24313.891172] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24313.895873] watchdog: BUG: soft lockup - CPU#7 stuck for 20527s! [kworker/7:4:78808] [24313.946613] irq event stamp: 38576090 [24313.954344] Modules linked in: ceph libceph [24313.958001] hardirqs last enabled at (38576089): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24313.962176] dns_resolver fscache [24313.972334] hardirqs last disabled at (38576090): [] sysvec_apic_timer_interrupt+0xe/0x90 [24313.975647] netfs veth nft_chain_nat [24313.985370] softirqs last enabled at (38576042): [] __do_softirq+0x2be/0x371 [24313.985371] xt_MASQUERADE nf_nat [24313.989029] softirqs last disabled at (38576037): [] irq_exit_rcu+0xb0/0xd0 [24313.997712] nf_conntrack [24314.001024] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24314.009548] nf_defrag_ipv6 nf_defrag_ipv4 [24314.012168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24314.021113] nft_compat nf_tables [24314.025203] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24314.032673] nfnetlink bridge [24314.035985] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24314.041462] stp llc [24314.044426] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24314.063165] xfs libcrc32c [24314.065348] [24314.065349] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24314.070566] sunrpc intel_rapl_msr [24314.073270] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24314.074759] intel_rapl_common [24314.081883] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24314.085281] sb_edac [24314.092404] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24314.095456] x86_pkg_temp_thermal [24314.102582] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24314.104764] coretemp kvm_intel [24314.111889] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24314.115198] kvm irqbypass [24314.122324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24314.125460] i2c_i801 iTCO_wdt [24314.133539] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24314.136270] iTCO_vendor_support [24314.142005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24314.145056] i2c_smbus crct10dif_pclmul [24314.152181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24314.155405] joydev [24314.162545] Call Trace: [24314.166377] crc32_pclmul ipmi_ssif [24314.173501] [24314.175598] ghash_clmulni_intel mei_me [24314.178044] ? watchdog_timer_fn+0x1b4/0x220 [24314.181526] mei lpc_ich [24314.183553] ? __hrtimer_run_queues+0x1d3/0x360 [24314.187386] sha512_ssse3 acpi_ipmi [24314.191652] ? hrtimer_interrupt+0x104/0x240 [24314.194180] mfd_core wmi [24314.198705] ? __local_bh_enable+0x50/0x60 [24314.202189] ipmi_si ipmi_devintf [24314.206454] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24314.209069] ipmi_msghandler ioatdma [24314.213161] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24314.216471] acpi_pad acpi_power_meter [24314.221690] [24314.225272] ixgbe [24314.230226] [24314.230228] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24314.233969] igb mdio [24314.236073] ? queued_spin_lock_slowpath+0x18/0x2e0 [24314.238078] crc32c_intel nvme [24314.240177] do_raw_spin_lock+0xb5/0xc0 [24314.245478] ptp nvme_core [24314.247751] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24314.252620] i2c_algo_bit pps_core [24314.255674] try_get_cap_refs+0x159/0x880 [ceph] [24314.259502] dca fuse [24314.262212] __ceph_get_caps+0x4a1/0x760 [ceph] [24314.266990] [24314.266991] irq event stamp: 116694146 [24314.270390] ? __file_remove_privs+0x89/0x110 [24314.274998] hardirqs last enabled at (116694145): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24314.277275] ? wake_bit_function+0x70/0x70 [24314.281800] hardirqs last disabled at (116694146): [] sysvec_apic_timer_interrupt+0xe/0x90 [24314.283296] ? ceph_write_iter+0x260/0x770 [ceph] [24314.287037] softirqs last enabled at (116694144): [] __do_softirq+0x2be/0x371 [24314.291388] ceph_write_iter+0x260/0x770 [ceph] [24314.301630] softirqs last disabled at (116694139): [] irq_exit_rcu+0xb0/0xd0 [24314.305725] ? lock_is_held_type+0xaa/0x120 [24314.315533] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24314.320234] vfs_write+0x324/0x4f0 [24314.329001] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24314.333529] ksys_pwrite64+0x65/0xa0 [24314.342124] Workqueue: events delayed_work [ceph] [24314.346301] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24314.355677] [24314.355678] RIP: 0010:do_raw_spin_unlock+0x9/0xc0 [24314.359076] do_syscall_64+0x38/0x80 [24314.366561] Code: eb 7e 89 47 08 65 48 8b 04 25 c0 e5 02 00 48 89 47 10 b8 01 00 00 00 c3 31 c0 c3 66 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 <53> 81 7f 04 ad 4e ad de 48 89 fb 75 34 8b 03 85 c0 74 48 65 48 8b [24314.370134] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24314.374832] RSP: 0018:ffffc90002117d88 EFLAGS: 00000296 [24314.379962] RIP: 0033:0x7f48f0eeb93b [24314.381452] [24314.381453] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [24314.386151] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24314.389720] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [24314.408458] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24314.413503] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [24314.418721] ORIG_RAX: 0000000000000012 [24314.422290] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [24314.423782] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24314.430906] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24314.449644] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24314.456769] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24314.461984] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24314.469109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24314.472939] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24314.480064] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24314.487188] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24314.494313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24314.501441] [24314.509513] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24314.560181] Call Trace: [24314.562624] [24314.564637] ? watchdog_timer_fn+0x1b4/0x220 [24314.568900] ? __hrtimer_run_queues+0x1d3/0x360 [24314.573426] ? hrtimer_interrupt+0x104/0x240 [24314.577688] ? __local_bh_enable+0x50/0x60 [24314.581781] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24314.587006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24314.591963] [24314.594059] [24314.596158] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24314.601465] ? do_raw_spin_unlock+0x9/0xc0 [24314.605576] _raw_spin_unlock+0x23/0x40 [24314.609410] igrab+0x43/0x50 [24314.612287] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24314.617437] delayed_work+0x1d8/0x280 [ceph] [24314.621716] ? process_scheduled_works+0x22a/0x510 [24314.626498] process_scheduled_works+0x2a4/0x510 [24314.631113] worker_thread+0x188/0x340 [24314.634863] ? rescuer_thread+0x310/0x310 [24314.638867] kthread+0xf5/0x130 [24314.642005] ? kthread_complete_and_exit+0x20/0x20 [24314.646789] ret_from_fork+0x31/0x40 [24314.650367] ? kthread_complete_and_exit+0x20/0x20 [24314.655152] ret_from_fork_asm+0x11/0x20 [24314.659075] [24314.706254] rcu: INFO: rcu_preempt self-detected stall on CPU [24314.712006] rcu: 7-....: (12021226 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/12019943 fqs=6044221 [24314.722597] rcu: (t=12377875 jiffies g=1309609 q=392790 ncpus=8) [24314.728681] Sending NMI from CPU 7 to CPUs 0: [24314.733035] NMI backtrace for cpu 0 [24314.733036] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24314.733038] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24314.733038] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24314.733041] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24314.733043] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24314.733044] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24314.733045] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24314.733045] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24314.733046] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24314.733047] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24314.733048] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24314.733049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24314.733050] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24314.733051] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24314.733052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24314.733052] Call Trace: [24314.733053] [24314.733054] ? nmi_cpu_backtrace+0xad/0x120 [24314.733058] ? nmi_cpu_backtrace_handler+0x11/0x20 [24314.733059] ? nmi_handle+0xd4/0x260 [24314.733064] ? default_do_nmi+0x49/0xf0 [24314.733065] ? exc_nmi+0x11e/0x150 [24314.733067] ? end_repeat_nmi+0x16/0x67 [24314.733071] ? queued_spin_lock_slowpath+0x18/0x2e0 [24314.733074] ? queued_spin_lock_slowpath+0x18/0x2e0 [24314.733076] ? queued_spin_lock_slowpath+0x18/0x2e0 [24314.733078] [24314.733079] [24314.733080] do_raw_spin_lock+0xb5/0xc0 [24314.733082] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24314.733096] try_get_cap_refs+0x159/0x880 [ceph] [24314.733110] __ceph_get_caps+0x4a1/0x760 [ceph] [24314.733123] ? __file_remove_privs+0x89/0x110 [24314.733125] ? wake_bit_function+0x70/0x70 [24314.733127] ? ceph_write_iter+0x260/0x770 [ceph] [24314.733138] ceph_write_iter+0x260/0x770 [ceph] [24314.733150] ? lock_is_held_type+0xaa/0x120 [24314.733152] vfs_write+0x324/0x4f0 [24314.733156] ksys_pwrite64+0x65/0xa0 [24314.733159] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24314.733161] do_syscall_64+0x38/0x80 [24314.733163] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24314.733165] RIP: 0033:0x7f48f0eeb93b [24314.733166] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24314.733167] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [24314.733169] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24314.733169] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24314.733170] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24314.733171] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24314.733171] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24314.733174] [24314.734032] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24315.037602] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24315.045073] Workqueue: events delayed_work [ceph] [24315.049787] RIP: 0010:lock_acquire+0xf0/0x2b0 [24315.054137] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24315.072874] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24315.078092] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24315.085217] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24315.092340] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24315.099464] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24315.106589] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24315.113714] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24315.121790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24315.127528] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24315.134652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24315.141777] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24315.148900] Call Trace: [24315.151345] [24315.153355] ? rcu_dump_cpu_stacks+0xd1/0x180 [24315.157708] ? rcu_sched_clock_irq+0x633/0x1390 [24315.162232] ? lock_release+0xc5/0x260 [24315.165976] ? lock_is_held_type+0xaa/0x120 [24315.170153] ? lock_is_held_type+0xaa/0x120 [24315.174331] ? tick_sched_do_timer+0xa0/0xa0 [24315.178602] ? rcu_read_lock_sched_held+0x56/0x90 [24315.183302] ? tick_sched_do_timer+0xa0/0xa0 [24315.187564] ? update_process_times+0x5f/0x90 [24315.191914] ? tick_sched_handle.isra.16+0x1e/0x50 [24315.196699] ? tick_sched_timer+0x6f/0x80 [24315.200703] ? __hrtimer_run_queues+0x1d3/0x360 [24315.205230] ? hrtimer_interrupt+0x104/0x240 [24315.209498] ? __local_bh_enable+0x50/0x60 [24315.213593] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24315.218816] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24315.223774] [24315.225870] [24315.227970] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24315.233277] ? lock_acquire+0xf0/0x2b0 [24315.237027] ? igrab+0x43/0x50 [24315.240076] ? lock_release+0xc5/0x260 [24315.243821] _raw_spin_lock+0x30/0x40 [24315.247486] ? igrab+0x1d/0x50 [24315.250536] igrab+0x1d/0x50 [24315.253415] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24315.258565] delayed_work+0x1d8/0x280 [ceph] [24315.262847] ? process_scheduled_works+0x22a/0x510 [24315.267635] process_scheduled_works+0x2a4/0x510 [24315.272251] worker_thread+0x188/0x340 [24315.276002] ? rescuer_thread+0x310/0x310 [24315.280012] kthread+0xf5/0x130 [24315.283149] ? kthread_complete_and_exit+0x20/0x20 [24315.287935] ret_from_fork+0x31/0x40 [24315.291505] ? kthread_complete_and_exit+0x20/0x20 [24315.296289] ret_from_fork_asm+0x11/0x20 [24315.300213] [24341.884363] watchdog: BUG: soft lockup - CPU#0 stuck for 11556s! [dbench:78683] [24341.891668] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24341.896362] watchdog: BUG: soft lockup - CPU#7 stuck for 20553s! [kworker/7:4:78808] [24341.947117] irq event stamp: 38634298 [24341.947118] hardirqs last enabled at (38634297): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24341.954847] Modules linked in: [24341.958504] hardirqs last disabled at (38634298): [] sysvec_apic_timer_interrupt+0xe/0x90 [24341.968664] ceph [24341.971712] softirqs last enabled at (38634294): [] __do_softirq+0x2be/0x371 [24341.981437] libceph [24341.983362] softirqs last disabled at (38634289): [] irq_exit_rcu+0xb0/0xd0 [24341.992046] dns_resolver [24341.994230] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24342.002756] fscache netfs [24342.005375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24342.014320] veth nft_chain_nat [24342.017039] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24342.024511] xt_MASQUERADE nf_nat [24342.027649] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24342.033126] nf_conntrack nf_defrag_ipv6 [24342.036436] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24342.055175] nf_defrag_ipv4 [24342.059092] [24342.064309] nft_compat nf_tables [24342.067101] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24342.068592] nfnetlink bridge [24342.071902] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24342.079039] stp [24342.081999] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24342.089124] llc xfs [24342.090961] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24342.098084] libcrc32c sunrpc [24342.100267] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24342.107391] intel_rapl_msr [24342.110356] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24342.117481] intel_rapl_common sb_edac [24342.120271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24342.128350] x86_pkg_temp_thermal coretemp [24342.132094] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24342.137831] kvm_intel [24342.141922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24342.149045] kvm irqbypass [24342.151402] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24342.158526] i2c_i801 [24342.161230] Call Trace: [24342.161232] [24342.168355] iTCO_wdt [24342.170626] ? watchdog_timer_fn+0x1b4/0x220 [24342.173068] iTCO_vendor_support [24342.175081] ? __hrtimer_run_queues+0x1d3/0x360 [24342.177351] i2c_smbus crct10dif_pclmul [24342.181617] ? hrtimer_interrupt+0x104/0x240 [24342.184840] joydev crc32_pclmul [24342.189362] ? __local_bh_enable+0x50/0x60 [24342.193194] ipmi_ssif [24342.197461] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24342.200683] ghash_clmulni_intel [24342.204772] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24342.207129] mei_me mei [24342.212349] [24342.215571] lpc_ich sha512_ssse3 [24342.220530] [24342.220531] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24342.222973] acpi_ipmi mfd_core [24342.225074] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24342.228381] wmi ipmi_si [24342.230480] do_raw_spin_lock+0xb5/0xc0 [24342.235783] ipmi_devintf ipmi_msghandler [24342.238923] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24342.243792] ioatdma acpi_pad [24342.246326] try_get_cap_refs+0x159/0x880 [ceph] [24342.250152] acpi_power_meter [24342.254162] __ceph_get_caps+0x4a1/0x760 [ceph] [24342.258941] ixgbe igb [24342.261909] ? __file_remove_privs+0x89/0x110 [24342.266514] mdio crc32c_intel [24342.269479] ? wake_bit_function+0x70/0x70 [24342.274005] nvme [24342.276362] ? ceph_write_iter+0x260/0x770 [ceph] [24342.280712] ptp nvme_core [24342.283765] ceph_write_iter+0x260/0x770 [ceph] [24342.287854] i2c_algo_bit pps_core [24342.289782] ? lock_is_held_type+0xaa/0x120 [24342.294475] dca fuse [24342.297181] vfs_write+0x324/0x4f0 [24342.301703] [24342.301703] irq event stamp: 116910142 [24342.305105] ksys_pwrite64+0x65/0xa0 [24342.309277] hardirqs last enabled at (116910141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24342.311549] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24342.314946] hardirqs last disabled at (116910142): [] sysvec_apic_timer_interrupt+0xe/0x90 [24342.316438] do_syscall_64+0x38/0x80 [24342.320179] softirqs last enabled at (116910140): [] __do_softirq+0x2be/0x371 [24342.323762] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24342.334004] softirqs last disabled at (116910135): [] irq_exit_rcu+0xb0/0xd0 [24342.339135] RIP: 0033:0x7f48f0eeb93b [24342.348945] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24342.352518] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24342.361287] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24342.366332] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24342.374928] Workqueue: events delayed_work [ceph] [24342.378499] ORIG_RAX: 0000000000000012 [24342.387876] [24342.406614] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24342.414086] RIP: 0010:lock_release+0x0/0x260 [24342.419304] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24342.423999] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [24342.427830] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24342.429322] RSP: 0018:ffffc90002117d88 EFLAGS: 00000286 [24342.436446] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24342.440710] [24342.447835] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24342.466571] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [24342.473700] [24342.478914] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [24342.511103] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [24342.518226] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [24342.525350] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24342.532473] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24342.540551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24342.546289] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24342.553413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24342.560537] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24342.567663] Call Trace: [24342.570107] [24342.572119] ? watchdog_timer_fn+0x1b4/0x220 [24342.576392] ? __hrtimer_run_queues+0x1d3/0x360 [24342.580918] ? hrtimer_interrupt+0x104/0x240 [24342.585186] ? __local_bh_enable+0x50/0x60 [24342.589280] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24342.594504] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24342.599461] [24342.601558] [24342.603654] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24342.608962] ? igrab+0x43/0x50 [24342.612020] ? lock_downgrade+0x250/0x250 [24342.616025] ? _raw_spin_lock+0x30/0x40 [24342.619862] _raw_spin_unlock+0x1b/0x40 [24342.623695] igrab+0x43/0x50 [24342.626572] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24342.631721] delayed_work+0x1d8/0x280 [ceph] [24342.636004] ? process_scheduled_works+0x22a/0x510 [24342.640793] process_scheduled_works+0x2a4/0x510 [24342.645408] worker_thread+0x188/0x340 [24342.649157] ? rescuer_thread+0x310/0x310 [24342.653161] kthread+0xf5/0x130 [24342.656308] ? kthread_complete_and_exit+0x20/0x20 [24342.661101] ret_from_fork+0x31/0x40 [24342.664679] ? kthread_complete_and_exit+0x20/0x20 [24342.669463] ret_from_fork_asm+0x11/0x20 [24342.673395] [24369.884852] watchdog: BUG: soft lockup - CPU#0 stuck for 11582s! [dbench:78683] [24369.892153] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24369.896852] watchdog: BUG: soft lockup - CPU#7 stuck for 20579s! [kworker/7:4:78808] [24369.947596] irq event stamp: 38692572 [24369.947597] hardirqs last enabled at (38692571): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24369.955327] Modules linked in: [24369.958983] hardirqs last disabled at (38692572): [] sysvec_apic_timer_interrupt+0xe/0x90 [24369.969141] ceph [24369.972192] softirqs last enabled at (38692534): [] __do_softirq+0x2be/0x371 [24369.981916] libceph dns_resolver [24369.983840] softirqs last disabled at (38692529): [] irq_exit_rcu+0xb0/0xd0 [24369.992528] fscache netfs [24369.995843] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24370.004353] veth nft_chain_nat [24370.007058] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24370.016003] xt_MASQUERADE [24370.019141] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24370.026610] nf_nat nf_conntrack [24370.029314] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24370.034793] nf_defrag_ipv6 nf_defrag_ipv4 [24370.038017] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24370.056755] nft_compat nf_tables [24370.060844] [24370.060845] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24370.066062] nfnetlink [24370.069371] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24370.070863] bridge stp [24370.077988] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24370.080344] llc xfs [24370.087468] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24370.089913] libcrc32c sunrpc [24370.097038] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24370.099222] intel_rapl_msr [24370.106362] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24370.109327] intel_rapl_common sb_edac [24370.116450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24370.119250] x86_pkg_temp_thermal [24370.127329] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24370.131072] coretemp kvm_intel [24370.136810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24370.140120] kvm irqbypass [24370.147251] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24370.150391] i2c_i801 iTCO_wdt [24370.157530] Call Trace: [24370.157531] [24370.160252] iTCO_vendor_support [24370.167379] ? watchdog_timer_fn+0x1b4/0x220 [24370.170429] i2c_smbus crct10dif_pclmul [24370.172874] ? __hrtimer_run_queues+0x1d3/0x360 [24370.174882] joydev crc32_pclmul [24370.178110] ? hrtimer_interrupt+0x104/0x240 [24370.182371] ipmi_ssif ghash_clmulni_intel [24370.186203] ? __local_bh_enable+0x50/0x60 [24370.190725] mei_me mei [24370.193953] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24370.198215] lpc_ich sha512_ssse3 [24370.202305] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24370.206395] acpi_ipmi mfd_core [24370.208840] [24370.214056] wmi ipmi_si [24370.217368] [24370.217370] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24370.222325] ipmi_devintf [24370.225467] ? queued_spin_lock_slowpath+0x18/0x2e0 [24370.227559] ipmi_msghandler ioatdma [24370.230093] do_raw_spin_lock+0xb5/0xc0 [24370.232187] acpi_pad acpi_power_meter [24370.237493] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24370.240110] ixgbe igb [24370.244983] try_get_cap_refs+0x159/0x880 [ceph] [24370.248550] mdio crc32c_intel [24370.252386] __ceph_get_caps+0x4a1/0x760 [ceph] [24370.256126] nvme ptp [24370.260914] ? __file_remove_privs+0x89/0x110 [24370.263267] nvme_core i2c_algo_bit [24370.267877] ? wake_bit_function+0x70/0x70 [24370.270929] pps_core dca [24370.275455] ? ceph_write_iter+0x260/0x770 [ceph] [24370.277724] fuse [24370.282075] ceph_write_iter+0x260/0x770 [ceph] [24370.285559] irq event stamp: 117130892 [24370.289653] ? lock_is_held_type+0xaa/0x120 [24370.292266] hardirqs last enabled at (117130891): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24370.296967] vfs_write+0x324/0x4f0 [24370.298887] hardirqs last disabled at (117130892): [] sysvec_apic_timer_interrupt+0xe/0x90 [24370.303416] ksys_pwrite64+0x65/0xa0 [24370.307155] softirqs last enabled at (117130890): [] __do_softirq+0x2be/0x371 [24370.311334] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24370.321576] softirqs last disabled at (117130885): [] irq_exit_rcu+0xb0/0xd0 [24370.324975] do_syscall_64+0x38/0x80 [24370.334787] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24370.338358] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24370.347128] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24370.352259] RIP: 0033:0x7f48f0eeb93b [24370.360855] Workqueue: events delayed_work [ceph] [24370.364428] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24370.373804] [24370.373805] RIP: 0010:lock_acquire+0xf0/0x2b0 [24370.378849] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24370.386319] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24370.389890] ORIG_RAX: 0000000000000012 [24370.394586] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24370.413325] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24370.414814] [24370.419166] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24370.424384] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24370.443121] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24370.446952] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24370.452170] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24370.459292] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24370.460784] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24370.467907] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24370.475036] [24370.482155] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24370.527103] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24370.535180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24370.540916] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24370.548042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24370.555167] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24370.562291] Call Trace: [24370.564734] [24370.566745] ? watchdog_timer_fn+0x1b4/0x220 [24370.571009] ? __hrtimer_run_queues+0x1d3/0x360 [24370.575542] ? hrtimer_interrupt+0x104/0x240 [24370.579804] ? __local_bh_enable+0x50/0x60 [24370.583900] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24370.589122] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24370.594080] [24370.596177] [24370.598276] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24370.603583] ? lock_acquire+0xf0/0x2b0 [24370.607333] ? igrab+0x43/0x50 [24370.610383] ? lock_release+0xc5/0x260 [24370.614129] _raw_spin_lock+0x30/0x40 [24370.617792] ? igrab+0x1d/0x50 [24370.620845] igrab+0x1d/0x50 [24370.623722] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24370.628872] delayed_work+0x1d8/0x280 [ceph] [24370.633153] ? process_scheduled_works+0x22a/0x510 [24370.637942] process_scheduled_works+0x2a4/0x510 [24370.642558] worker_thread+0x188/0x340 [24370.646307] ? rescuer_thread+0x310/0x310 [24370.650311] kthread+0xf5/0x130 [24370.653449] ? kthread_complete_and_exit+0x20/0x20 [24370.658235] ret_from_fork+0x31/0x40 [24370.661829] ? kthread_complete_and_exit+0x20/0x20 [24370.666613] ret_from_fork_asm+0x11/0x20 [24370.670548] [24397.885343] watchdog: BUG: soft lockup - CPU#0 stuck for 11608s! [dbench:78683] [24397.892651] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24397.897342] watchdog: BUG: soft lockup - CPU#7 stuck for 20605s! [kworker/7:4:78808] [24397.948101] irq event stamp: 38750850 [24397.955832] Modules linked in: ceph libceph dns_resolver fscache netfs veth [24397.959487] hardirqs last enabled at (38750849): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24397.966440] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [24397.976595] hardirqs last disabled at (38750850): [] sysvec_apic_timer_interrupt+0xe/0x90 [24397.976598] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [24397.991971] softirqs last enabled at (38750812): [] __do_softirq+0x2be/0x371 [24397.991974] stp llc xfs libcrc32c sunrpc intel_rapl_msr [24398.008039] softirqs last disabled at (38750807): [] irq_exit_rcu+0xb0/0xd0 [24398.013344] intel_rapl_common sb_edac x86_pkg_temp_thermal [24398.021856] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24398.027418] coretemp kvm_intel [24398.036364] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24398.036365] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24398.039500] kvm irqbypass [24398.046970] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24398.052448] i2c_i801 [24398.055154] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24398.073890] iTCO_wdt [24398.076162] [24398.081377] iTCO_vendor_support [24398.083650] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24398.085140] i2c_smbus crct10dif_pclmul [24398.088363] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24398.095489] joydev crc32_pclmul [24398.099318] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24398.106443] ipmi_ssif ghash_clmulni_intel [24398.109668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24398.116790] mei_me [24398.120882] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24398.128007] mei lpc_ich [24398.130121] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24398.137244] sha512_ssse3 acpi_ipmi [24398.139776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24398.147852] mfd_core [24398.151336] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24398.157075] wmi ipmi_si [24398.159344] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24398.166470] ipmi_devintf ipmi_msghandler [24398.169001] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24398.176124] ioatdma [24398.180128] Call Trace: [24398.180131] [24398.187252] acpi_pad acpi_power_meter [24398.189439] ? watchdog_timer_fn+0x1b4/0x220 [24398.191879] ixgbe igb [24398.193893] ? __hrtimer_run_queues+0x1d3/0x360 [24398.197634] mdio crc32c_intel [24398.201902] ? hrtimer_interrupt+0x104/0x240 [24398.204256] nvme ptp [24398.208782] ? __local_bh_enable+0x50/0x60 [24398.211832] nvme_core i2c_algo_bit [24398.216097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24398.218367] pps_core [24398.222457] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24398.225942] dca fuse [24398.231157] [24398.233429] [24398.233430] irq event stamp: 117351688 [24398.238387] [24398.240656] hardirqs last enabled at (117351687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24398.242756] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24398.244246] hardirqs last disabled at (117351688): [] sysvec_apic_timer_interrupt+0xe/0x90 [24398.247994] ? queued_spin_lock_slowpath+0x16/0x2e0 [24398.250088] softirqs last enabled at (117351686): [] __do_softirq+0x2be/0x371 [24398.260335] do_raw_spin_lock+0xb5/0xc0 [24398.265635] softirqs last disabled at (117351681): [] irq_exit_rcu+0xb0/0xd0 [24398.275447] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24398.280316] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24398.289091] try_get_cap_refs+0x159/0x880 [ceph] [24398.292917] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24398.301521] __ceph_get_caps+0x4a1/0x760 [ceph] [24398.306301] Workqueue: events delayed_work [ceph] [24398.315682] ? __file_remove_privs+0x89/0x110 [24398.320287] [24398.320288] RIP: 0010:lock_release+0x14e/0x260 [24398.327759] ? wake_bit_function+0x70/0x70 [24398.332284] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [24398.336984] ? ceph_write_iter+0x260/0x770 [ceph] [24398.341331] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [24398.342822] ceph_write_iter+0x260/0x770 [ceph] [24398.347260] [24398.347260] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [24398.351355] ? lock_is_held_type+0xaa/0x120 [24398.370087] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [24398.374789] vfs_write+0x324/0x4f0 [24398.380002] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24398.384550] ksys_pwrite64+0x65/0xa0 [24398.386034] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24398.393160] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24398.397335] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [24398.404463] do_syscall_64+0x38/0x80 [24398.407857] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24398.414982] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24398.418554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24398.425677] RIP: 0033:0x7f48f0eeb93b [24398.430806] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24398.437933] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24398.441502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24398.449580] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24398.454624] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24398.460361] ORIG_RAX: 0000000000000012 [24398.463933] Call Trace: [24398.463934] [24398.471057] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24398.489796] ? watchdog_timer_fn+0x1b4/0x220 [24398.496918] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24398.502136] ? __hrtimer_run_queues+0x1d3/0x360 [24398.509259] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24398.513094] ? hrtimer_interrupt+0x104/0x240 [24398.515535] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24398.517544] ? __local_bh_enable+0x50/0x60 [24398.524670] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24398.528936] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24398.536062] [24398.540583] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24398.582669] [24398.584766] [24398.586863] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24398.592167] ? igrab+0x43/0x50 [24398.595221] ? lock_release+0x14e/0x260 [24398.599057] ? lock_release+0xc5/0x260 [24398.602803] _raw_spin_unlock+0x1b/0x40 [24398.606633] igrab+0x43/0x50 [24398.609511] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24398.614660] delayed_work+0x1d8/0x280 [ceph] [24398.618941] ? process_scheduled_works+0x22a/0x510 [24398.623748] process_scheduled_works+0x2a4/0x510 [24398.628365] worker_thread+0x188/0x340 [24398.632113] ? rescuer_thread+0x310/0x310 [24398.636117] kthread+0xf5/0x130 [24398.639264] ? kthread_complete_and_exit+0x20/0x20 [24398.644048] ret_from_fork+0x31/0x40 [24398.647625] ? kthread_complete_and_exit+0x20/0x20 [24398.652411] ret_from_fork_asm+0x11/0x20 [24398.656341] [24425.885833] watchdog: BUG: soft lockup - CPU#0 stuck for 11634s! [dbench:78683] [24425.893138] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24425.897833] watchdog: BUG: soft lockup - CPU#7 stuck for 20631s! [kworker/7:4:78808] [24425.948580] irq event stamp: 38808940 [24425.948581] hardirqs last enabled at (38808939): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24425.956310] Modules linked in: [24425.959968] hardirqs last disabled at (38808940): [] sysvec_apic_timer_interrupt+0xe/0x90 [24425.970126] ceph [24425.973174] softirqs last enabled at (38808902): [] __do_softirq+0x2be/0x371 [24425.982898] libceph [24425.984824] softirqs last disabled at (38808897): [] irq_exit_rcu+0xb0/0xd0 [24425.993518] dns_resolver fscache [24425.995701] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24426.004226] netfs veth [24426.007540] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24426.016482] nft_chain_nat xt_MASQUERADE [24426.018946] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24426.026416] nf_nat nf_conntrack [24426.030333] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24426.035809] nf_defrag_ipv6 nf_defrag_ipv4 [24426.039036] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24426.057771] nft_compat nf_tables [24426.061863] [24426.061864] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24426.067082] nfnetlink [24426.070391] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24426.071883] bridge stp [24426.079006] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24426.081364] llc xfs [24426.088487] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24426.090949] libcrc32c sunrpc [24426.098073] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24426.100256] intel_rapl_msr intel_rapl_common [24426.107380] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24426.110346] sb_edac [24426.117469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24426.121821] x86_pkg_temp_thermal coretemp [24426.129899] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24426.132083] kvm_intel kvm [24426.137818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24426.141908] irqbypass i2c_i801 [24426.149034] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24426.151738] iTCO_wdt iTCO_vendor_support [24426.158862] Call Trace: [24426.158863] [24426.161999] i2c_smbus crct10dif_pclmul [24426.169124] ? watchdog_timer_fn+0x1b4/0x220 [24426.173128] joydev [24426.175572] ? __hrtimer_run_queues+0x1d3/0x360 [24426.177583] crc32_pclmul [24426.181415] ? hrtimer_interrupt+0x104/0x240 [24426.185677] ipmi_ssif [24426.187775] ? __local_bh_enable+0x50/0x60 [24426.192299] ghash_clmulni_intel [24426.194920] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24426.199180] mei_me mei [24426.201537] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24426.205628] lpc_ich sha512_ssse3 [24426.208851] [24426.214069] acpi_ipmi mfd_core [24426.216520] [24426.221479] wmi ipmi_si [24426.224791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24426.226888] ipmi_devintf ipmi_msghandler [24426.230030] ? queued_spin_lock_slowpath+0x18/0x2e0 [24426.232122] ioatdma acpi_pad [24426.234657] do_raw_spin_lock+0xb5/0xc0 [24426.239957] acpi_power_meter [24426.243964] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24426.248833] ixgbe igb [24426.251798] try_get_cap_refs+0x159/0x880 [ceph] [24426.255628] mdio crc32c_intel [24426.258597] __ceph_get_caps+0x4a1/0x760 [ceph] [24426.263374] nvme ptp [24426.265738] ? __file_remove_privs+0x89/0x110 [24426.270344] nvme_core i2c_algo_bit [24426.273395] ? wake_bit_function+0x70/0x70 [24426.277918] pps_core dca [24426.280192] ? ceph_write_iter+0x260/0x770 [ceph] [24426.284540] fuse [24426.288025] ceph_write_iter+0x260/0x770 [ceph] [24426.292115] irq event stamp: 117572602 [24426.294736] ? lock_is_held_type+0xaa/0x120 [24426.299429] hardirqs last enabled at (117572601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24426.301355] vfs_write+0x324/0x4f0 [24426.305879] hardirqs last disabled at (117572602): [] sysvec_apic_timer_interrupt+0xe/0x90 [24426.309627] ksys_pwrite64+0x65/0xa0 [24426.313800] softirqs last enabled at (117572600): [] __do_softirq+0x2be/0x371 [24426.324045] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24426.327439] softirqs last disabled at (117572595): [] irq_exit_rcu+0xb0/0xd0 [24426.337254] do_syscall_64+0x38/0x80 [24426.340823] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24426.349594] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24426.354723] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24426.363322] RIP: 0033:0x7f48f0eeb93b [24426.366892] Workqueue: events delayed_work [ceph] [24426.376272] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24426.381313] [24426.388783] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24426.392356] RIP: 0010:lock_acquire+0xf0/0x2b0 [24426.397053] ORIG_RAX: 0000000000000012 [24426.415791] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24426.417281] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24426.422498] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24426.426848] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24426.430680] [24426.430681] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24426.449417] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24426.456542] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24426.461759] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24426.468883] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24426.470373] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24426.477498] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24426.484628] [24426.491747] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24426.529552] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24426.537628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24426.543365] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24426.550490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24426.557615] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24426.564738] Call Trace: [24426.567182] [24426.569193] ? watchdog_timer_fn+0x1b4/0x220 [24426.573459] ? __hrtimer_run_queues+0x1d3/0x360 [24426.577985] ? hrtimer_interrupt+0x104/0x240 [24426.582254] ? __local_bh_enable+0x50/0x60 [24426.586347] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24426.591571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24426.596530] [24426.598627] [24426.600724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24426.606031] ? lock_acquire+0xf0/0x2b0 [24426.609783] ? igrab+0x43/0x50 [24426.612841] ? lock_release+0xc5/0x260 [24426.616584] _raw_spin_lock+0x30/0x40 [24426.620241] ? igrab+0x1d/0x50 [24426.623293] igrab+0x1d/0x50 [24426.626171] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24426.631319] delayed_work+0x1d8/0x280 [ceph] [24426.635601] ? process_scheduled_works+0x22a/0x510 [24426.640390] process_scheduled_works+0x2a4/0x510 [24426.645006] worker_thread+0x188/0x340 [24426.648754] ? rescuer_thread+0x310/0x310 [24426.652759] kthread+0xf5/0x130 [24426.655896] ? kthread_complete_and_exit+0x20/0x20 [24426.660681] ret_from_fork+0x31/0x40 [24426.664259] ? kthread_complete_and_exit+0x20/0x20 [24426.669045] ret_from_fork_asm+0x11/0x20 [24426.672967] [24453.886324] watchdog: BUG: soft lockup - CPU#0 stuck for 11660s! [dbench:78683] [24453.893625] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24453.898323] watchdog: BUG: soft lockup - CPU#7 stuck for 20657s! [kworker/7:4:78808] [24453.949067] irq event stamp: 38867178 [24453.956796] Modules linked in: ceph libceph dns_resolver fscache [24453.960455] hardirqs last enabled at (38867177): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24453.966461] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [24453.976617] hardirqs last disabled at (38867178): [] sysvec_apic_timer_interrupt+0xe/0x90 [24453.982096] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [24453.991820] softirqs last enabled at (38867140): [] __do_softirq+0x2be/0x371 [24453.998858] nfnetlink bridge stp llc xfs [24454.007543] softirqs last disabled at (38867135): [] irq_exit_rcu+0xb0/0xd0 [24454.011554] libcrc32c sunrpc intel_rapl_msr [24454.020066] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24454.020068] intel_rapl_common [24454.020069] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24454.024330] sb_edac [24454.033274] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24454.036324] x86_pkg_temp_thermal [24454.043796] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24454.045978] coretemp kvm_intel [24454.051474] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24454.054784] kvm irqbypass [24454.073523] [24454.073524] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24454.076660] i2c_i801 [24454.081877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24454.084581] iTCO_wdt [24454.086071] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24454.093196] iTCO_vendor_support [24454.095466] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24454.102591] i2c_smbus [24454.104861] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24454.111986] crct10dif_pclmul [24454.115227] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24454.122350] joydev [24454.124721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24454.131840] crc32_pclmul ipmi_ssif [24454.134804] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24454.142882] ghash_clmulni_intel mei_me [24454.144980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24454.150721] mei [24454.154202] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24454.161325] lpc_ich sha512_ssse3 [24454.165155] Call Trace: [24454.165158] [24454.172281] acpi_ipmi mfd_core [24454.174119] ? watchdog_timer_fn+0x1b4/0x220 [24454.181242] wmi ipmi_si [24454.184554] ? __hrtimer_run_queues+0x1d3/0x360 [24454.187002] ipmi_devintf [24454.189020] ? hrtimer_interrupt+0x104/0x240 [24454.192152] ipmi_msghandler [24454.196418] ? __local_bh_enable+0x50/0x60 [24454.198948] ioatdma acpi_pad [24454.203475] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24454.206089] acpi_power_meter [24454.210353] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24454.213232] ixgbe igb [24454.217322] [24454.220287] mdio [24454.225503] [24454.228469] crc32c_intel [24454.233427] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24454.235783] nvme [24454.237884] ? queued_spin_lock_slowpath+0x18/0x2e0 [24454.239803] ptp [24454.241903] do_raw_spin_lock+0xb5/0xc0 [24454.244519] nvme_core i2c_algo_bit [24454.249823] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24454.251746] pps_core [24454.256621] try_get_cap_refs+0x159/0x880 [ceph] [24454.258454] dca fuse [24454.262291] __ceph_get_caps+0x4a1/0x760 [ceph] [24454.265770] [24454.265771] irq event stamp: 117793400 [24454.270559] ? __file_remove_privs+0x89/0x110 [24454.272825] hardirqs last enabled at (117793399): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24454.277435] ? wake_bit_function+0x70/0x70 [24454.279723] hardirqs last disabled at (117793400): [] sysvec_apic_timer_interrupt+0xe/0x90 [24454.284250] ? ceph_write_iter+0x260/0x770 [ceph] [24454.285737] softirqs last enabled at (117793398): [] __do_softirq+0x2be/0x371 [24454.289482] ceph_write_iter+0x260/0x770 [ceph] [24454.293833] softirqs last disabled at (117793393): [] irq_exit_rcu+0xb0/0xd0 [24454.304081] ? lock_is_held_type+0xaa/0x120 [24454.308168] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24454.317980] vfs_write+0x324/0x4f0 [24454.322676] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24454.331451] ksys_pwrite64+0x65/0xa0 [24454.335973] Workqueue: events delayed_work [ceph] [24454.344569] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24454.348745] [24454.358125] do_syscall_64+0x38/0x80 [24454.361522] RIP: 0010:lock_acquire+0xf0/0x2b0 [24454.369005] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24454.372571] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24454.377269] RIP: 0033:0x7f48f0eeb93b [24454.382399] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24454.383890] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24454.387461] [24454.387462] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24454.391811] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24454.396856] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24454.415593] ORIG_RAX: 0000000000000012 [24454.419163] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24454.424381] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24454.443118] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24454.444609] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24454.451732] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24454.456950] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24454.464075] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24454.467905] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24454.475029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24454.482154] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24454.489278] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24454.496407] [24454.503526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24454.555164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24454.562288] Call Trace: [24454.564732] [24454.566743] ? watchdog_timer_fn+0x1b4/0x220 [24454.571011] ? __hrtimer_run_queues+0x1d3/0x360 [24454.575542] ? hrtimer_interrupt+0x104/0x240 [24454.579812] ? __local_bh_enable+0x50/0x60 [24454.583906] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24454.589129] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24454.594086] [24454.596183] [24454.598283] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24454.603590] ? lock_acquire+0xf0/0x2b0 [24454.607339] ? igrab+0x43/0x50 [24454.610391] ? lock_release+0xc5/0x260 [24454.614135] _raw_spin_lock+0x30/0x40 [24454.617800] ? igrab+0x1d/0x50 [24454.620849] igrab+0x1d/0x50 [24454.623732] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24454.628878] delayed_work+0x1d8/0x280 [ceph] [24454.633158] ? process_scheduled_works+0x22a/0x510 [24454.637941] process_scheduled_works+0x2a4/0x510 [24454.642557] worker_thread+0x188/0x340 [24454.646306] ? rescuer_thread+0x310/0x310 [24454.650318] kthread+0xf5/0x130 [24454.653465] ? kthread_complete_and_exit+0x20/0x20 [24454.658247] ret_from_fork+0x31/0x40 [24454.661829] ? kthread_complete_and_exit+0x20/0x20 [24454.666612] ret_from_fork_asm+0x11/0x20 [24454.670544] [24481.886812] watchdog: BUG: soft lockup - CPU#0 stuck for 11686s! [dbench:78683] [24481.894113] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24481.898812] watchdog: BUG: soft lockup - CPU#7 stuck for 20683s! [kworker/7:4:78808] [24481.949554] irq event stamp: 38925338 [24481.949555] hardirqs last enabled at (38925337): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24481.957286] Modules linked in: [24481.960943] hardirqs last disabled at (38925338): [] sysvec_apic_timer_interrupt+0xe/0x90 [24481.971099] ceph [24481.974151] softirqs last enabled at (38925300): [] __do_softirq+0x2be/0x371 [24481.983876] libceph [24481.985799] softirqs last disabled at (38925295): [] irq_exit_rcu+0xb0/0xd0 [24481.994492] dns_resolver [24481.996676] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24482.005185] fscache netfs [24482.007821] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24482.007822] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24482.016766] veth nft_chain_nat [24482.019486] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24482.026956] xt_MASQUERADE [24482.032436] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24482.035572] nf_nat nf_conntrack [24482.054309] [24482.057013] nf_defrag_ipv6 nf_defrag_ipv4 [24482.062230] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24482.065456] nft_compat [24482.066947] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24482.071038] nf_tables nfnetlink [24482.078161] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24482.080604] bridge stp [24482.087729] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24482.090952] llc xfs [24482.098078] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24482.100522] libcrc32c [24482.107647] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24482.109828] sunrpc intel_rapl_msr [24482.116954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24482.119311] intel_rapl_common sb_edac [24482.127389] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24482.130786] x86_pkg_temp_thermal coretemp [24482.136524] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24482.140268] kvm_intel kvm [24482.147392] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24482.151491] irqbypass i2c_i801 [24482.158616] Call Trace: [24482.158618] [24482.161319] iTCO_wdt [24482.168444] ? watchdog_timer_fn+0x1b4/0x220 [24482.171580] iTCO_vendor_support i2c_smbus [24482.174025] ? __hrtimer_run_queues+0x1d3/0x360 [24482.176036] crct10dif_pclmul [24482.178309] ? hrtimer_interrupt+0x104/0x240 [24482.182570] joydev crc32_pclmul [24482.186662] ? __local_bh_enable+0x50/0x60 [24482.191186] ipmi_ssif ghash_clmulni_intel [24482.194170] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24482.198430] mei_me mei [24482.201655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24482.205746] lpc_ich sha512_ssse3 [24482.209836] [24482.215053] acpi_ipmi mfd_core [24482.217499] [24482.217500] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24482.222455] wmi ipmi_si [24482.225769] ? queued_spin_lock_slowpath+0x18/0x2e0 [24482.227864] ipmi_devintf ipmi_msghandler [24482.231001] do_raw_spin_lock+0xb5/0xc0 [24482.233098] ioatdma acpi_pad [24482.238403] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24482.240933] acpi_power_meter ixgbe [24482.245807] try_get_cap_refs+0x159/0x880 [ceph] [24482.249807] igb [24482.253642] __ceph_get_caps+0x4a1/0x760 [ceph] [24482.256602] mdio [24482.261392] ? __file_remove_privs+0x89/0x110 [24482.264870] crc32c_intel nvme [24482.269498] ? wake_bit_function+0x70/0x70 [24482.271335] ptp nvme_core [24482.275862] ? ceph_write_iter+0x260/0x770 [ceph] [24482.277784] i2c_algo_bit pps_core [24482.282135] ceph_write_iter+0x260/0x770 [ceph] [24482.285185] dca [24482.289299] ? lock_is_held_type+0xaa/0x120 [24482.291997] fuse [24482.296697] vfs_write+0x324/0x4f0 [24482.300092] irq event stamp: 118014236 [24482.304620] ksys_pwrite64+0x65/0xa0 [24482.306454] hardirqs last enabled at (118014235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24482.310631] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24482.312555] hardirqs last disabled at (118014236): [] sysvec_apic_timer_interrupt+0xe/0x90 [24482.315955] do_syscall_64+0x38/0x80 [24482.319696] softirqs last enabled at (118014234): [] __do_softirq+0x2be/0x371 [24482.323267] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24482.333513] softirqs last disabled at (118014229): [] irq_exit_rcu+0xb0/0xd0 [24482.338643] RIP: 0033:0x7f48f0eeb93b [24482.348453] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24482.352025] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24482.360795] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24482.365838] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24482.374436] Workqueue: events delayed_work [ceph] [24482.378008] ORIG_RAX: 0000000000000012 [24482.387384] [24482.387385] RIP: 0010:lock_acquire+0xf0/0x2b0 [24482.406122] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24482.413593] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24482.418809] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24482.423508] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24482.427339] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24482.428831] [24482.428831] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24482.433179] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24482.440304] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24482.459043] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24482.466165] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24482.471390] [24482.478508] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24482.524927] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24482.532051] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24482.540129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24482.545866] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24482.552991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24482.560116] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24482.567239] Call Trace: [24482.569683] [24482.571694] ? watchdog_timer_fn+0x1b4/0x220 [24482.575958] ? __hrtimer_run_queues+0x1d3/0x360 [24482.580501] ? hrtimer_interrupt+0x104/0x240 [24482.584763] ? __local_bh_enable+0x50/0x60 [24482.588857] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24482.594081] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24482.599037] [24482.601134] [24482.603232] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24482.608541] ? lock_acquire+0xf0/0x2b0 [24482.612290] ? igrab+0x43/0x50 [24482.615340] ? lock_release+0xc5/0x260 [24482.619087] _raw_spin_lock+0x30/0x40 [24482.622751] ? igrab+0x1d/0x50 [24482.625800] igrab+0x1d/0x50 [24482.628679] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24482.633829] delayed_work+0x1d8/0x280 [ceph] [24482.638111] ? process_scheduled_works+0x22a/0x510 [24482.642900] process_scheduled_works+0x2a4/0x510 [24482.647516] worker_thread+0x188/0x340 [24482.651265] ? rescuer_thread+0x310/0x310 [24482.655278] kthread+0xf5/0x130 [24482.658415] ? kthread_complete_and_exit+0x20/0x20 [24482.663219] ret_from_fork+0x31/0x40 [24482.666796] ? kthread_complete_and_exit+0x20/0x20 [24482.671580] ret_from_fork_asm+0x11/0x20 [24482.675514] [24509.887301] watchdog: BUG: soft lockup - CPU#0 stuck for 11712s! [dbench:78683] [24509.894600] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24509.899300] watchdog: BUG: soft lockup - CPU#7 stuck for 20709s! [kworker/7:4:78808] [24509.950051] irq event stamp: 38983600 [24509.950053] hardirqs last enabled at (38983599): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24509.957781] Modules linked in: [24509.961438] hardirqs last disabled at (38983600): [] sysvec_apic_timer_interrupt+0xe/0x90 [24509.971595] ceph [24509.974646] softirqs last enabled at (38983562): [] __do_softirq+0x2be/0x371 [24509.984371] libceph [24509.986294] softirqs last disabled at (38983557): [] irq_exit_rcu+0xb0/0xd0 [24509.994978] dns_resolver fscache [24509.997163] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24510.005674] netfs veth [24510.009003] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24510.017946] nft_chain_nat xt_MASQUERADE [24510.020392] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24510.027859] nf_nat nf_conntrack [24510.031780] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24510.037256] nf_defrag_ipv6 nf_defrag_ipv4 [24510.040481] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24510.059217] nft_compat nf_tables [24510.063308] [24510.063309] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24510.068525] nfnetlink bridge [24510.071837] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24510.073328] stp llc [24510.080450] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24510.083415] xfs libcrc32c [24510.090540] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24510.092725] sunrpc [24510.099847] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24510.102552] intel_rapl_msr intel_rapl_common [24510.109693] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24510.111789] sb_edac [24510.118914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24510.123265] x86_pkg_temp_thermal coretemp [24510.131343] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24510.133527] kvm_intel [24510.139263] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24510.143354] kvm irqbypass [24510.150479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24510.152837] i2c_i801 [24510.159977] Call Trace: [24510.162697] iTCO_wdt [24510.169824] [24510.172095] iTCO_vendor_support [24510.174540] ? watchdog_timer_fn+0x1b4/0x220 [24510.176809] i2c_smbus crct10dif_pclmul [24510.178821] ? __hrtimer_run_queues+0x1d3/0x360 [24510.182042] joydev [24510.186312] ? hrtimer_interrupt+0x104/0x240 [24510.190138] crc32_pclmul ipmi_ssif [24510.194663] ? __local_bh_enable+0x50/0x60 [24510.196760] ghash_clmulni_intel mei_me [24510.201026] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24510.204509] mei lpc_ich [24510.208599] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24510.212428] sha512_ssse3 [24510.217648] [24510.220177] acpi_ipmi mfd_core [24510.225135] [24510.227753] wmi ipmi_si [24510.229850] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24510.232989] ipmi_devintf [24510.235088] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24510.237615] ipmi_msghandler ioatdma [24510.242921] do_raw_spin_lock+0xb5/0xc0 [24510.245536] acpi_pad acpi_power_meter [24510.250408] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24510.253994] ixgbe [24510.257830] try_get_cap_refs+0x159/0x880 [ceph] [24510.261571] igb mdio [24510.266360] __ceph_get_caps+0x4a1/0x760 [ceph] [24510.268366] crc32c_intel nvme [24510.272986] ? __file_remove_privs+0x89/0x110 [24510.275256] ptp [24510.279780] ? wake_bit_function+0x70/0x70 [24510.282830] nvme_core i2c_algo_bit [24510.287183] ? ceph_write_iter+0x260/0x770 [ceph] [24510.289019] pps_core dca [24510.293110] ceph_write_iter+0x260/0x770 [ceph] [24510.296592] fuse [24510.301295] ? lock_is_held_type+0xaa/0x120 [24510.303908] irq event stamp: 118235002 [24510.308433] vfs_write+0x324/0x4f0 [24510.310355] hardirqs last enabled at (118235001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24510.313674] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [24510.313677] rcu: 0-....: (12291612 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=6140204 [24510.313699] rcu: 7-....: (12211179 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/12209896 fqs=6140204 [24510.313702] rcu: (detected by 3, t=12573464 jiffies, g=1309609, q=397912 ncpus=8) [24510.313705] Sending NMI from CPU 3 to CPUs 0: [24510.313710] NMI backtrace for cpu 0 [24510.313711] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24510.313713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24510.313713] RIP: 0010:io_serial_in+0x19/0x20 [24510.313716] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [24510.313718] RSP: 0018:ffffc90000003ba8 EFLAGS: 00000002 [24510.313719] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [24510.313720] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [24510.313721] RBP: 0000000000002607 R08: 00004f9f4a0e5bb6 R09: 0000000000000000 [24510.313722] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000060 [24510.313723] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [24510.313723] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24510.313724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24510.313725] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24510.313726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24510.313727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24510.313728] Call Trace: [24510.313728] [24510.313729] ? nmi_cpu_backtrace+0xad/0x120 [24510.313732] ? nmi_cpu_backtrace_handler+0x11/0x20 [24510.313734] ? nmi_handle+0xd4/0x260 [24510.313739] ? default_do_nmi+0x49/0xf0 [24510.313740] ? exc_nmi+0x11e/0x150 [24510.313742] ? end_repeat_nmi+0x16/0x67 [24510.313746] ? io_serial_in+0x19/0x20 [24510.313748] ? io_serial_in+0x19/0x20 [24510.313750] ? io_serial_in+0x19/0x20 [24510.313751] [24510.313752] [24510.313752] wait_for_lsr+0x45/0x70 [24510.313754] wait_for_xmitr+0xf/0x60 [24510.313756] serial8250_console_write+0x22a/0x3f0 [24510.313759] ? lock_release+0xc5/0x260 [24510.313762] console_flush_all+0x227/0x5a0 [24510.313764] ? console_flush_all+0x1ff/0x5a0 [24510.313765] ? console_flush_all+0x1ff/0x5a0 [24510.313768] console_unlock+0x7b/0x120 [24510.313770] vprintk_emit+0x246/0x340 [24510.313772] ? vfs_write+0x324/0x4f0 [24510.313774] ? vfs_write+0x324/0x4f0 [24510.313776] _printk+0x5c/0x80 [24510.313779] ? vfs_write+0x324/0x4f0 [24510.313781] show_trace_log_lvl+0x264/0x3f0 [24510.313786] ? vfs_write+0x324/0x4f0 [24510.313789] ? touch_softlockup_watchdog+0x20/0x20 [24510.313790] watchdog_timer_fn+0x1b4/0x220 [24510.313792] __hrtimer_run_queues+0x1d3/0x360 [24510.313795] hrtimer_interrupt+0x104/0x240 [24510.313796] ? __local_bh_enable+0x50/0x60 [24510.313799] __sysvec_apic_timer_interrupt+0x61/0x160 [24510.313802] sysvec_apic_timer_interrupt+0x6d/0x90 [24510.313804] [24510.313804] [24510.313805] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24510.313807] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24510.313809] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24510.313811] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24510.313811] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24510.313812] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24510.313813] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24510.313813] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24510.313814] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24510.313817] do_raw_spin_lock+0xb5/0xc0 [24510.313819] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24510.313833] try_get_cap_refs+0x159/0x880 [ceph] [24510.313847] __ceph_get_caps+0x4a1/0x760 [ceph] [24510.313859] ? __file_remove_privs+0x89/0x110 [24510.313861] ? wake_bit_function+0x70/0x70 [24510.313863] ? ceph_write_iter+0x260/0x770 [ceph] [24510.313874] ceph_write_iter+0x260/0x770 [ceph] [24510.313885] ? lock_is_held_type+0xaa/0x120 [24510.313888] vfs_write+0x324/0x4f0 [24510.313892] ksys_pwrite64+0x65/0xa0 [24510.313894] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24510.313897] do_syscall_64+0x38/0x80 [24510.313900] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24510.313902] RIP: 0033:0x7f48f0eeb93b [24510.313903] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24510.313904] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [24510.313905] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24510.313906] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24510.313907] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24510.313908] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24510.313908] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24510.313911] [24510.314538] ksys_pwrite64+0x65/0xa0 [24510.314707] Sending NMI from CPU 3 to CPUs 7: [24510.314710] NMI backtrace for cpu 7 [24510.314711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24510.314714] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24510.314715] Workqueue: events delayed_work [ceph] [24510.314737] RIP: 0010:delay_tsc+0x3b/0x90 [24510.314740] Code: 44 8b 0d 54 f6 49 7e 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e <65> 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 66 90 48 c1 [24510.314742] RSP: 0018:ffffc90000264cb8 EFLAGS: 00000003 [24510.314743] RAX: 00004f9f4a43f4b7 RBX: 0000000000000000 RCX: 0000000000000007 [24510.314745] RDX: 0000000000000cd7 RSI: 0000000000000007 RDI: 0000000000000dac [24510.314746] RBP: 00000000000026d9 R08: 00004f9f4a43e7e0 R09: 0000000000000007 [24510.314747] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [24510.314748] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [24510.314749] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24510.314751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24510.314752] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24510.314754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24510.314755] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24510.314756] Call Trace: [24510.314757] [24510.314759] ? nmi_cpu_backtrace+0xad/0x120 [24510.314763] ? nmi_cpu_backtrace_handler+0x11/0x20 [24510.314764] ? nmi_handle+0xd4/0x260 [24510.314769] ? default_do_nmi+0x49/0xf0 [24510.314772] ? exc_nmi+0x11e/0x150 [24510.314774] ? end_repeat_nmi+0x16/0x67 [24510.314780] ? delay_tsc+0x3b/0x90 [24510.314783] ? delay_tsc+0x3b/0x90 [24510.314786] ? delay_tsc+0x3b/0x90 [24510.314789] [24510.314790] [24510.314790] wait_for_lsr+0x27/0x70 [24510.314794] serial8250_console_write+0x341/0x3f0 [24510.314798] ? lock_release+0xc5/0x260 [24510.314802] console_flush_all+0x227/0x5a0 [24510.314805] ? console_flush_all+0x1ff/0x5a0 [24510.314807] ? console_flush_all+0x1ff/0x5a0 [24510.314811] console_unlock+0x7b/0x120 [24510.314815] vprintk_emit+0x246/0x340 [24510.314818] ? touch_softlockup_watchdog+0x20/0x20 [24510.314820] _printk+0x5c/0x80 [24510.314824] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24510.314826] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24510.314829] print_irqtrace_events+0x36/0x90 [24510.314831] watchdog_timer_fn+0x1a7/0x220 [24510.314834] __hrtimer_run_queues+0x1d3/0x360 [24510.314839] hrtimer_interrupt+0x104/0x240 [24510.314840] ? __local_bh_enable+0x50/0x60 [24510.314845] __sysvec_apic_timer_interrupt+0x61/0x160 [24510.314849] sysvec_apic_timer_interrupt+0x6d/0x90 [24510.314851] [24510.314851] [24510.314852] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24510.314855] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [24510.314874] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [24510.314875] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [24510.314877] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [24510.314878] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [24510.314879] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [24510.314881] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [24510.314882] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24510.314889] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [24510.314906] delayed_work+0x1d8/0x280 [ceph] [24510.314926] ? process_scheduled_works+0x22a/0x510 [24510.314929] process_scheduled_works+0x2a4/0x510 [24510.314936] worker_thread+0x188/0x340 [24510.314940] ? rescuer_thread+0x310/0x310 [24510.314942] kthread+0xf5/0x130 [24510.314946] ? kthread_complete_and_exit+0x20/0x20 [24510.314949] ret_from_fork+0x31/0x40 [24510.314953] ? kthread_complete_and_exit+0x20/0x20 [24510.314955] ret_from_fork_asm+0x11/0x20 [24510.314963] [24510.318278] hardirqs last disabled at (118235002): [] sysvec_apic_timer_interrupt+0xe/0x90 [24510.321676] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24510.331937] softirqs last enabled at (118235000): [] __do_softirq+0x2be/0x371 [24510.338022] do_syscall_64+0x38/0x80 [24510.348439] softirqs last disabled at (118234995): [] irq_exit_rcu+0xb0/0xd0 [24510.359030] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24510.366586] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24510.370938] RIP: 0033:0x7f48f0eeb93b [24510.374420] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24510.383365] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24510.390836] Workqueue: events delayed_work [ceph] [24510.395100] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24510.413839] [24510.413840] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [24510.419054] ORIG_RAX: 0000000000000012 [24510.426180] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [24510.433303] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24510.440427] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [24510.447551] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24510.454676] [24510.454677] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [24510.462772] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24510.468507] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [24510.475632] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24510.482756] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [24510.489881] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24510.492325] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [24510.494341] [24510.498514] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24511.402854] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24511.410930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24511.416670] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24511.423793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24511.430915] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24511.438039] Call Trace: [24511.440485] [24511.442495] ? watchdog_timer_fn+0x1b4/0x220 [24511.446760] ? __hrtimer_run_queues+0x1d3/0x360 [24511.451287] ? hrtimer_interrupt+0x104/0x240 [24511.455556] ? __local_bh_enable+0x50/0x60 [24511.459649] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24511.464873] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24511.469831] [24511.471927] [24511.474024] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24511.479333] ? ceph_check_delayed_caps+0x9e/0x270 [ceph] [24511.484651] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [24511.489967] delayed_work+0x1d8/0x280 [ceph] [24511.494265] ? process_scheduled_works+0x22a/0x510 [24511.499055] process_scheduled_works+0x2a4/0x510 [24511.503671] worker_thread+0x188/0x340 [24511.507419] ? rescuer_thread+0x310/0x310 [24511.511424] kthread+0xf5/0x130 [24511.514560] ? kthread_complete_and_exit+0x20/0x20 [24511.519346] ret_from_fork+0x31/0x40 [24511.522923] ? kthread_complete_and_exit+0x20/0x20 [24511.527722] ret_from_fork_asm+0x11/0x20 [24511.531648] [24537.887789] watchdog: BUG: soft lockup - CPU#0 stuck for 11738s! [dbench:78683] [24537.895088] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24537.899789] watchdog: BUG: soft lockup - CPU#7 stuck for 20735s! [kworker/7:4:78808] [24537.950529] irq event stamp: 39041856 [24537.950530] hardirqs last enabled at (39041855): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24537.958260] Modules linked in: ceph [24537.961918] hardirqs last disabled at (39041856): [] sysvec_apic_timer_interrupt+0xe/0x90 [24537.972074] libceph [24537.975559] softirqs last enabled at (39041822): [] __do_softirq+0x2be/0x371 [24537.985282] dns_resolver [24537.987474] softirqs last disabled at (39041817): [] irq_exit_rcu+0xb0/0xd0 [24537.996161] fscache netfs [24537.998795] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24538.007306] veth nft_chain_nat [24538.010008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24538.018952] xt_MASQUERADE nf_nat [24538.022090] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24538.029562] nf_conntrack [24538.032873] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24538.038349] nf_defrag_ipv6 nf_defrag_ipv4 [24538.040966] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24538.059705] nft_compat [24538.063794] [24538.063795] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24538.069012] nf_tables nfnetlink [24538.071465] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24538.072956] bridge stp [24538.080079] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24538.083305] llc xfs [24538.090429] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24538.092873] libcrc32c sunrpc [24538.099996] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24538.102187] intel_rapl_msr intel_rapl_common [24538.109313] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24538.112278] sb_edac x86_pkg_temp_thermal [24538.119402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24538.123752] coretemp kvm_intel [24538.131829] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24538.135834] kvm irqbypass [24538.141572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24538.144709] i2c_i801 iTCO_wdt [24538.151834] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24538.154538] iTCO_vendor_support [24538.161662] Call Trace: [24538.161664] [24538.164711] i2c_smbus [24538.171836] ? watchdog_timer_fn+0x1b4/0x220 [24538.175060] crct10dif_pclmul [24538.177505] ? __hrtimer_run_queues+0x1d3/0x360 [24538.179515] joydev crc32_pclmul [24538.181875] ? hrtimer_interrupt+0x104/0x240 [24538.186136] ipmi_ssif ghash_clmulni_intel [24538.189100] ? __local_bh_enable+0x50/0x60 [24538.193623] mei_me [24538.196850] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24538.201112] mei lpc_ich [24538.205204] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24538.209294] sha512_ssse3 acpi_ipmi [24538.211391] [24538.216608] mfd_core wmi [24538.219138] [24538.219140] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24538.224098] ipmi_si ipmi_devintf [24538.227585] ? queued_spin_lock_slowpath+0x16/0x2e0 [24538.229678] ipmi_msghandler [24538.232297] do_raw_spin_lock+0xb5/0xc0 [24538.234392] ioatdma acpi_pad [24538.239698] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24538.243009] acpi_power_meter ixgbe [24538.247882] try_get_cap_refs+0x159/0x880 [ceph] [24538.250757] igb mdio [24538.254592] __ceph_get_caps+0x4a1/0x760 [ceph] [24538.257551] crc32c_intel nvme ptp [24538.262339] ? __file_remove_privs+0x89/0x110 [24538.265818] nvme_core i2c_algo_bit [24538.270432] ? wake_bit_function+0x70/0x70 [24538.272701] pps_core dca [24538.277228] ? ceph_write_iter+0x260/0x770 [ceph] [24538.280621] fuse [24538.284974] ceph_write_iter+0x260/0x770 [ceph] [24538.288469] irq event stamp: 118448870 [24538.292560] ? lock_is_held_type+0xaa/0x120 [24538.295189] hardirqs last enabled at (118448869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24538.299891] vfs_write+0x324/0x4f0 [24538.301811] hardirqs last disabled at (118448870): [] sysvec_apic_timer_interrupt+0xe/0x90 [24538.306341] ksys_pwrite64+0x65/0xa0 [24538.310081] softirqs last enabled at (118448868): [] __do_softirq+0x2be/0x371 [24538.314258] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24538.324501] softirqs last disabled at (118448863): [] irq_exit_rcu+0xb0/0xd0 [24538.327900] do_syscall_64+0x38/0x80 [24538.337711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24538.341282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24538.350052] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24538.355192] RIP: 0033:0x7f48f0eeb93b [24538.363789] Workqueue: events delayed_work [ceph] [24538.367360] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24538.376737] [24538.381781] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24538.389253] RIP: 0010:lock_acquire+0xf0/0x2b0 [24538.392824] ORIG_RAX: 0000000000000012 [24538.397520] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24538.416257] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24538.417748] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24538.422965] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24538.427316] [24538.427317] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24538.431148] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24538.449884] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24538.457008] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24538.462227] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24538.469350] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24538.470841] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24538.477971] [24538.485090] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24538.530036] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24538.538113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24538.543852] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24538.550976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24538.558099] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24538.565222] Call Trace: [24538.567670] [24538.569678] ? watchdog_timer_fn+0x1b4/0x220 [24538.573944] ? __hrtimer_run_queues+0x1d3/0x360 [24538.578479] ? hrtimer_interrupt+0x104/0x240 [24538.582748] ? __local_bh_enable+0x50/0x60 [24538.586843] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24538.592065] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24538.597023] [24538.599119] [24538.601218] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24538.606534] ? lock_acquire+0xf0/0x2b0 [24538.610285] ? igrab+0x43/0x50 [24538.613334] ? lock_release+0xc5/0x260 [24538.617079] _raw_spin_lock+0x30/0x40 [24538.620745] ? igrab+0x1d/0x50 [24538.623795] igrab+0x1d/0x50 [24538.626673] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24538.631821] delayed_work+0x1d8/0x280 [ceph] [24538.636103] ? process_scheduled_works+0x22a/0x510 [24538.640893] process_scheduled_works+0x2a4/0x510 [24538.645509] worker_thread+0x188/0x340 [24538.649258] ? rescuer_thread+0x310/0x310 [24538.653260] kthread+0xf5/0x130 [24538.656398] ? kthread_complete_and_exit+0x20/0x20 [24538.661195] ret_from_fork+0x31/0x40 [24538.664770] ? kthread_complete_and_exit+0x20/0x20 [24538.669554] ret_from_fork_asm+0x11/0x20 [24538.673483] [24565.888278] watchdog: BUG: soft lockup - CPU#0 stuck for 11764s! [dbench:78683] [24565.895584] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24565.900278] watchdog: BUG: soft lockup - CPU#7 stuck for 20761s! [kworker/7:4:78808] [24565.951024] irq event stamp: 39100088 [24565.951025] hardirqs last enabled at (39100087): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24565.958755] Modules linked in: [24565.962414] hardirqs last disabled at (39100088): [] sysvec_apic_timer_interrupt+0xe/0x90 [24565.972570] ceph [24565.975620] softirqs last enabled at (39100050): [] __do_softirq+0x2be/0x371 [24565.985344] libceph [24565.987270] softirqs last disabled at (39100045): [] irq_exit_rcu+0xb0/0xd0 [24565.995952] dns_resolver fscache [24565.998137] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24566.006649] netfs [24566.009977] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24566.018921] veth nft_chain_nat [24566.020930] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24566.028403] xt_MASQUERADE nf_nat [24566.031539] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24566.037017] nf_conntrack nf_defrag_ipv6 [24566.040328] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24566.059065] nf_defrag_ipv4 [24566.062983] [24566.062984] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24566.068200] nft_compat [24566.070990] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24566.072480] nf_tables nfnetlink [24566.079606] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24566.082049] bridge stp [24566.089174] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24566.092397] llc xfs [24566.099522] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24566.101966] libcrc32c sunrpc [24566.109090] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24566.111275] intel_rapl_msr intel_rapl_common [24566.118399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24566.121363] sb_edac x86_pkg_temp_thermal [24566.129440] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24566.133790] coretemp kvm_intel [24566.139528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24566.143532] kvm irqbypass [24566.150672] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24566.153811] i2c_i801 iTCO_wdt [24566.160934] Call Trace: [24566.163640] iTCO_vendor_support i2c_smbus [24566.170764] [24566.173814] crct10dif_pclmul joydev [24566.176259] ? watchdog_timer_fn+0x1b4/0x220 [24566.180348] crc32_pclmul ipmi_ssif [24566.182359] ? __hrtimer_run_queues+0x1d3/0x360 [24566.185930] ghash_clmulni_intel mei_me [24566.190197] ? hrtimer_interrupt+0x104/0x240 [24566.193678] mei lpc_ich [24566.198203] ? __local_bh_enable+0x50/0x60 [24566.202032] sha512_ssse3 acpi_ipmi [24566.206299] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24566.208828] mfd_core wmi [24566.212918] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24566.216402] ipmi_si ipmi_devintf [24566.221619] [24566.224238] ipmi_msghandler ioatdma [24566.229196] [24566.229197] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24566.232506] acpi_pad acpi_power_meter [24566.234607] ? queued_spin_lock_slowpath+0x18/0x2e0 [24566.238173] ixgbe igb [24566.240273] do_raw_spin_lock+0xb5/0xc0 [24566.245575] mdio crc32c_intel [24566.249321] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24566.254191] nvme ptp [24566.256549] try_get_cap_refs+0x159/0x880 [ceph] [24566.260378] nvme_core i2c_algo_bit [24566.263434] __ceph_get_caps+0x4a1/0x760 [ceph] [24566.268214] pps_core dca [24566.270488] ? __file_remove_privs+0x89/0x110 [24566.275094] fuse [24566.278579] ? wake_bit_function+0x70/0x70 [24566.283102] irq event stamp: 118669658 [24566.285722] ? ceph_write_iter+0x260/0x770 [ceph] [24566.290071] hardirqs last enabled at (118669657): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24566.291996] ceph_write_iter+0x260/0x770 [ceph] [24566.296085] hardirqs last disabled at (118669658): [] sysvec_apic_timer_interrupt+0xe/0x90 [24566.299833] ? lock_is_held_type+0xaa/0x120 [24566.304526] softirqs last enabled at (118669656): [] __do_softirq+0x2be/0x371 [24566.314774] vfs_write+0x324/0x4f0 [24566.319295] softirqs last disabled at (118669651): [] irq_exit_rcu+0xb0/0xd0 [24566.329110] ksys_pwrite64+0x65/0xa0 [24566.333284] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24566.342054] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24566.345452] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24566.345453] Workqueue: events delayed_work [ceph] [24566.354052] do_syscall_64+0x38/0x80 [24566.357621] [24566.357621] RIP: 0010:lock_acquire+0xf0/0x2b0 [24566.366997] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24566.372129] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24566.379598] RIP: 0033:0x7f48f0eeb93b [24566.384297] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24566.387868] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24566.389357] [24566.389357] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24566.393709] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24566.398754] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24566.417490] ORIG_RAX: 0000000000000012 [24566.421061] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24566.426279] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24566.445016] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24566.446507] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24566.453632] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24566.458848] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24566.465974] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24566.469803] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24566.476928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24566.484052] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24566.491177] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24566.498306] [24566.505423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24566.557045] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24566.564167] Call Trace: [24566.566613] [24566.568624] ? watchdog_timer_fn+0x1b4/0x220 [24566.572890] ? __hrtimer_run_queues+0x1d3/0x360 [24566.577424] ? hrtimer_interrupt+0x104/0x240 [24566.581693] ? __local_bh_enable+0x50/0x60 [24566.585787] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24566.591011] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24566.595975] [24566.598071] [24566.600172] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24566.605478] ? lock_acquire+0xf0/0x2b0 [24566.609227] ? igrab+0x43/0x50 [24566.612279] ? lock_release+0xc5/0x260 [24566.616024] _raw_spin_lock+0x30/0x40 [24566.619688] ? igrab+0x1d/0x50 [24566.622738] igrab+0x1d/0x50 [24566.625616] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24566.630767] delayed_work+0x1d8/0x280 [ceph] [24566.635047] ? process_scheduled_works+0x22a/0x510 [24566.639839] process_scheduled_works+0x2a4/0x510 [24566.644454] worker_thread+0x188/0x340 [24566.648202] ? rescuer_thread+0x310/0x310 [24566.652205] kthread+0xf5/0x130 [24566.655344] ? kthread_complete_and_exit+0x20/0x20 [24566.660129] ret_from_fork+0x31/0x40 [24566.663697] ? kthread_complete_and_exit+0x20/0x20 [24566.668482] ret_from_fork_asm+0x11/0x20 [24566.672404] [24593.888763] watchdog: BUG: soft lockup - CPU#0 stuck for 11790s! [dbench:78683] [24593.896071] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24593.900763] watchdog: BUG: soft lockup - CPU#7 stuck for 20787s! [kworker/7:4:78808] [24593.951522] irq event stamp: 39158266 [24593.959250] Modules linked in: ceph libceph dns_resolver fscache netfs [24593.962909] hardirqs last enabled at (39158265): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24593.969437] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [24593.979592] hardirqs last disabled at (39158266): [] sysvec_apic_timer_interrupt+0xe/0x90 [24593.986975] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [24593.996700] softirqs last enabled at (39158228): [] __do_softirq+0x2be/0x371 [24594.002785] stp llc xfs libcrc32c sunrpc intel_rapl_msr [24594.011469] softirqs last disabled at (39158223): [] irq_exit_rcu+0xb0/0xd0 [24594.016772] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [24594.025286] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24594.031630] kvm_intel [24594.040572] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24594.040573] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24594.042929] kvm irqbypass [24594.050401] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24594.055877] i2c_i801 iTCO_wdt [24594.058583] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24594.077318] iTCO_vendor_support i2c_smbus [24594.080370] [24594.085588] crct10dif_pclmul [24594.089678] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24594.091168] joydev [24594.094134] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24594.101274] crc32_pclmul [24594.103372] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24594.110495] ipmi_ssif ghash_clmulni_intel [24594.113114] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24594.120238] mei_me [24594.124328] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24594.131454] mei lpc_ich [24594.133549] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24594.140674] sha512_ssse3 acpi_ipmi [24594.143205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24594.151282] mfd_core [24594.154766] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24594.160503] wmi ipmi_si [24594.162775] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24594.169898] ipmi_devintf [24594.172428] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24594.179571] ipmi_msghandler [24594.182187] Call Trace: [24594.189313] ioatdma acpi_pad [24594.192191] [24594.194634] acpi_power_meter ixgbe [24594.197600] ? watchdog_timer_fn+0x1b4/0x220 [24594.199609] igb mdio [24594.203094] ? __hrtimer_run_queues+0x1d3/0x360 [24594.207358] crc32c_intel nvme [24594.209630] ? hrtimer_interrupt+0x104/0x240 [24594.214162] ptp nvme_core [24594.217211] ? __local_bh_enable+0x50/0x60 [24594.221475] i2c_algo_bit [24594.224181] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24594.228271] pps_core dca [24594.230889] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24594.236104] fuse [24594.238721] [24594.243680] [24594.243680] irq event stamp: 118890540 [24594.245604] [24594.245605] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24594.247701] hardirqs last enabled at (118890539): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24594.249196] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24594.252935] hardirqs last disabled at (118890540): [] sysvec_apic_timer_interrupt+0xe/0x90 [24594.255035] do_raw_spin_lock+0xb5/0xc0 [24594.260336] softirqs last enabled at (118890538): [] __do_softirq+0x2be/0x371 [24594.270582] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24594.275454] softirqs last disabled at (118890533): [] irq_exit_rcu+0xb0/0xd0 [24594.285266] try_get_cap_refs+0x159/0x880 [ceph] [24594.289092] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24594.297870] __ceph_get_caps+0x4a1/0x760 [ceph] [24594.302647] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24594.311251] ? __file_remove_privs+0x89/0x110 [24594.315859] Workqueue: events delayed_work [ceph] [24594.325235] ? wake_bit_function+0x70/0x70 [24594.329759] [24594.329760] RIP: 0010:lock_release+0x42/0x260 [24594.337233] ? ceph_write_iter+0x260/0x770 [ceph] [24594.341581] Code: 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 <8b> 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 [24594.346277] ceph_write_iter+0x260/0x770 [ceph] [24594.350368] RSP: 0018:ffffc90002117d48 EFLAGS: 00000202 [24594.351861] ? lock_is_held_type+0xaa/0x120 [24594.356209] [24594.356209] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [24594.360908] vfs_write+0x324/0x4f0 [24594.379644] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [24594.384174] ksys_pwrite64+0x65/0xa0 [24594.389387] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24594.393564] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24594.395054] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24594.402181] do_syscall_64+0x38/0x80 [24594.405577] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24594.412699] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24594.416270] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24594.423396] RIP: 0033:0x7f48f0eeb93b [24594.428526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24594.435652] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24594.439219] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24594.446344] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24594.451390] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24594.459465] ORIG_RAX: 0000000000000012 [24594.463038] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24594.468773] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24594.487511] Call Trace: [24594.494636] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24594.499855] [24594.506976] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24594.510808] ? watchdog_timer_fn+0x1b4/0x220 [24594.517933] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24594.525056] ? __hrtimer_run_queues+0x1d3/0x360 [24594.527501] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24594.534627] ? hrtimer_interrupt+0x104/0x240 [24594.536641] [24594.543760] ? __local_bh_enable+0x50/0x60 [24594.577356] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24594.582578] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24594.587536] [24594.589632] [24594.591732] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24594.597037] ? igrab+0x43/0x50 [24594.600089] ? igrab+0x43/0x50 [24594.603136] ? lock_release+0x42/0x260 [24594.606883] ? lock_release+0x1d7/0x260 [24594.610713] _raw_spin_unlock+0x1b/0x40 [24594.614553] igrab+0x43/0x50 [24594.617430] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24594.622598] delayed_work+0x1d8/0x280 [ceph] [24594.626878] ? process_scheduled_works+0x22a/0x510 [24594.631666] process_scheduled_works+0x2a4/0x510 [24594.636283] worker_thread+0x188/0x340 [24594.640034] ? rescuer_thread+0x310/0x310 [24594.644045] kthread+0xf5/0x130 [24594.647192] ? kthread_complete_and_exit+0x20/0x20 [24594.651983] ret_from_fork+0x31/0x40 [24594.655563] ? kthread_complete_and_exit+0x20/0x20 [24594.660347] ret_from_fork_asm+0x11/0x20 [24594.664277] [24621.889243] watchdog: BUG: soft lockup - CPU#0 stuck for 11816s! [dbench:78683] [24621.896541] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24621.901243] watchdog: BUG: soft lockup - CPU#7 stuck for 20813s! [kworker/7:4:78808] [24621.951982] irq event stamp: 39216360 [24621.951983] hardirqs last enabled at (39216359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24621.959712] Modules linked in: [24621.963370] hardirqs last disabled at (39216360): [] sysvec_apic_timer_interrupt+0xe/0x90 [24621.973527] ceph [24621.976578] softirqs last enabled at (39216322): [] __do_softirq+0x2be/0x371 [24621.986301] libceph [24621.988227] softirqs last disabled at (39216317): [] irq_exit_rcu+0xb0/0xd0 [24621.996926] dns_resolver fscache [24621.999112] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24622.007640] netfs veth [24622.010951] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24622.019895] nft_chain_nat xt_MASQUERADE [24622.022339] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24622.029811] nf_nat [24622.033727] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24622.039204] nf_conntrack nf_defrag_ipv6 [24622.041302] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24622.060040] nf_defrag_ipv4 [24622.063957] [24622.063958] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24622.069174] nft_compat nf_tables [24622.071966] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24622.073457] nfnetlink bridge [24622.080579] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24622.083890] stp llc [24622.091016] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24622.093979] xfs libcrc32c [24622.101103] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24622.103288] sunrpc intel_rapl_msr [24622.110412] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24622.113114] intel_rapl_common sb_edac [24622.120240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24622.123641] x86_pkg_temp_thermal coretemp [24622.131714] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24622.135458] kvm_intel kvm [24622.141197] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24622.145288] irqbypass i2c_i801 [24622.152411] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24622.155115] iTCO_wdt iTCO_vendor_support [24622.162240] Call Trace: [24622.165377] i2c_smbus crct10dif_pclmul [24622.172501] [24622.176505] joydev crc32_pclmul [24622.178951] ? watchdog_timer_fn+0x1b4/0x220 [24622.182780] ipmi_ssif ghash_clmulni_intel [24622.184791] ? __hrtimer_run_queues+0x1d3/0x360 [24622.188015] mei_me mei [24622.192282] ? hrtimer_interrupt+0x104/0x240 [24622.196368] lpc_ich sha512_ssse3 [24622.200893] ? __local_bh_enable+0x50/0x60 [24622.203338] acpi_ipmi mfd_core [24622.207605] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24622.210912] wmi ipmi_si [24622.215020] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24622.218157] ipmi_devintf ipmi_msghandler [24622.223375] [24622.225907] ioatdma acpi_pad [24622.230862] [24622.230863] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24622.234867] acpi_power_meter [24622.236969] ? queued_spin_lock_slowpath+0x16/0x2e0 [24622.239929] ixgbe [24622.242029] do_raw_spin_lock+0xb5/0xc0 [24622.247329] igb mdio [24622.250294] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24622.255166] crc32c_intel nvme [24622.257178] try_get_cap_refs+0x159/0x880 [ceph] [24622.261006] ptp nvme_core [24622.263281] __ceph_get_caps+0x4a1/0x760 [ceph] [24622.268061] i2c_algo_bit pps_core [24622.271116] ? __file_remove_privs+0x89/0x110 [24622.275721] dca fuse [24622.278427] ? wake_bit_function+0x70/0x70 [24622.282951] [24622.282952] irq event stamp: 119111338 [24622.286352] ? ceph_write_iter+0x260/0x770 [ceph] [24622.290698] hardirqs last enabled at (119111337): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24622.292970] ceph_write_iter+0x260/0x770 [ceph] [24622.297061] hardirqs last disabled at (119111338): [] sysvec_apic_timer_interrupt+0xe/0x90 [24622.298554] ? lock_is_held_type+0xaa/0x120 [24622.302294] softirqs last enabled at (119111336): [] __do_softirq+0x2be/0x371 [24622.306995] vfs_write+0x324/0x4f0 [24622.317235] softirqs last disabled at (119111331): [] irq_exit_rcu+0xb0/0xd0 [24622.321764] ksys_pwrite64+0x65/0xa0 [24622.331572] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24622.335750] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24622.344519] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24622.344520] Workqueue: events delayed_work [ceph] [24622.347933] do_syscall_64+0x38/0x80 [24622.356531] [24622.356532] RIP: 0010:lock_acquire+0xf0/0x2b0 [24622.360104] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24622.369479] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24622.374612] RIP: 0033:0x7f48f0eeb93b [24622.382082] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24622.386780] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24622.390350] [24622.390350] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24622.391841] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24622.396191] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24622.401236] ORIG_RAX: 0000000000000012 [24622.419974] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24622.423544] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24622.428760] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24622.447500] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24622.448990] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24622.456115] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24622.461332] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24622.468457] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24622.472286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24622.479410] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24622.486533] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24622.493664] [24622.500784] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24622.559545] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24622.566667] Call Trace: [24622.569114] [24622.571123] ? watchdog_timer_fn+0x1b4/0x220 [24622.575389] ? __hrtimer_run_queues+0x1d3/0x360 [24622.579915] ? hrtimer_interrupt+0x104/0x240 [24622.584192] ? __local_bh_enable+0x50/0x60 [24622.588286] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24622.593509] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24622.598468] [24622.600564] [24622.602662] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24622.607969] ? lock_acquire+0xf0/0x2b0 [24622.611719] ? igrab+0x43/0x50 [24622.614769] ? lock_release+0xc5/0x260 [24622.618516] _raw_spin_lock+0x30/0x40 [24622.622180] ? igrab+0x1d/0x50 [24622.625230] igrab+0x1d/0x50 [24622.628108] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24622.633259] delayed_work+0x1d8/0x280 [ceph] [24622.637540] ? process_scheduled_works+0x22a/0x510 [24622.642330] process_scheduled_works+0x2a4/0x510 [24622.646944] worker_thread+0x188/0x340 [24622.650693] ? rescuer_thread+0x310/0x310 [24622.654697] kthread+0xf5/0x130 [24622.657835] ? kthread_complete_and_exit+0x20/0x20 [24622.662618] ret_from_fork+0x31/0x40 [24622.666206] ? kthread_complete_and_exit+0x20/0x20 [24622.670991] ret_from_fork_asm+0x11/0x20 [24622.674931] [24649.889724] watchdog: BUG: soft lockup - CPU#0 stuck for 11842s! [dbench:78683] [24649.897029] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24649.901723] watchdog: BUG: soft lockup - CPU#7 stuck for 20840s! [kworker/7:4:78808] [24649.952478] irq event stamp: 39274646 [24649.960210] Modules linked in: ceph libceph dns_resolver fscache netfs veth [24649.963866] hardirqs last enabled at (39274645): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24649.970825] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [24649.980984] hardirqs last disabled at (39274646): [] sysvec_apic_timer_interrupt+0xe/0x90 [24649.986633] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [24649.996359] softirqs last enabled at (39274608): [] __do_softirq+0x2be/0x371 [24650.003751] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [24650.012435] softirqs last disabled at (39274603): [] irq_exit_rcu+0xb0/0xd0 [24650.019300] sb_edac x86_pkg_temp_thermal coretemp [24650.027810] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24650.032594] kvm_intel kvm [24650.041537] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24650.041539] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24650.044241] irqbypass i2c_i801 [24650.051715] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24650.057191] iTCO_wdt [24650.060329] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24650.079065] iTCO_vendor_support i2c_smbus [24650.081335] [24650.081336] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24650.086553] crct10dif_pclmul [24650.090645] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24650.092135] joydev crc32_pclmul [24650.099259] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24650.102224] ipmi_ssif ghash_clmulni_intel [24650.109348] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24650.112572] mei_me mei [24650.119695] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24650.123785] lpc_ich [24650.130910] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24650.133356] sha512_ssse3 acpi_ipmi [24650.140478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24650.142663] mfd_core wmi [24650.150741] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24650.154224] ipmi_si ipmi_devintf [24650.159962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24650.162580] ipmi_msghandler ioatdma [24650.169703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24650.173014] acpi_pad acpi_power_meter [24650.180139] Call Trace: [24650.180142] [24650.183710] ixgbe [24650.190835] ? watchdog_timer_fn+0x1b4/0x220 [24650.194577] igb mdio [24650.197022] ? __hrtimer_run_queues+0x1d3/0x360 [24650.199033] crc32c_intel nvme [24650.201046] ? hrtimer_interrupt+0x104/0x240 [24650.205307] ptp nvme_core [24650.207577] ? __local_bh_enable+0x50/0x60 [24650.212119] i2c_algo_bit [24650.215171] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24650.219433] pps_core dca [24650.222137] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24650.226228] fuse [24650.228846] [24650.234064] irq event stamp: 119332206 [24650.236680] [24650.241638] hardirqs last enabled at (119332205): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24650.243562] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24650.245660] hardirqs last disabled at (119332206): [] sysvec_apic_timer_interrupt+0xe/0x90 [24650.249416] ? queued_spin_lock_slowpath+0x16/0x2e0 [24650.251511] softirqs last enabled at (119332204): [] __do_softirq+0x2be/0x371 [24650.261756] do_raw_spin_lock+0xb5/0xc0 [24650.267057] softirqs last disabled at (119332199): [] irq_exit_rcu+0xb0/0xd0 [24650.276871] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24650.281739] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24650.290513] try_get_cap_refs+0x159/0x880 [ceph] [24650.294340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24650.302944] __ceph_get_caps+0x4a1/0x760 [ceph] [24650.307724] Workqueue: events delayed_work [ceph] [24650.317124] ? __file_remove_privs+0x89/0x110 [24650.321728] [24650.321730] RIP: 0010:lock_release+0x77/0x260 [24650.329200] ? wake_bit_function+0x70/0x70 [24650.333722] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [24650.338422] ? ceph_write_iter+0x260/0x770 [ceph] [24650.342771] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [24650.344262] ceph_write_iter+0x260/0x770 [ceph] [24650.348613] [24650.348614] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [24650.352707] ? lock_is_held_type+0xaa/0x120 [24650.371441] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [24650.376142] vfs_write+0x324/0x4f0 [24650.381356] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24650.385884] ksys_pwrite64+0x65/0xa0 [24650.387370] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24650.394495] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24650.398673] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24650.405797] do_syscall_64+0x38/0x80 [24650.409192] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24650.416319] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24650.419888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24650.427014] RIP: 0033:0x7f48f0eeb93b [24650.432142] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24650.439268] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24650.442838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24650.450915] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24650.455959] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24650.461699] ORIG_RAX: 0000000000000012 [24650.465267] Call Trace: [24650.465270] [24650.472405] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24650.491139] ? watchdog_timer_fn+0x1b4/0x220 [24650.498264] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24650.503481] ? __hrtimer_run_queues+0x1d3/0x360 [24650.510605] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24650.514438] ? hrtimer_interrupt+0x104/0x240 [24650.516878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24650.518891] ? __local_bh_enable+0x50/0x60 [24650.526014] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24650.530280] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24650.537422] [24650.541944] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24650.584031] [24650.586128] [24650.588226] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24650.593539] ? igrab+0x43/0x50 [24650.596591] ? lock_release+0x77/0x260 [24650.600337] _raw_spin_unlock+0x1b/0x40 [24650.604174] igrab+0x43/0x50 [24650.607050] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24650.612201] delayed_work+0x1d8/0x280 [ceph] [24650.616481] ? process_scheduled_works+0x22a/0x510 [24650.621270] process_scheduled_works+0x2a4/0x510 [24650.625888] worker_thread+0x188/0x340 [24650.629635] ? rescuer_thread+0x310/0x310 [24650.633639] kthread+0xf5/0x130 [24650.636779] ? kthread_complete_and_exit+0x20/0x20 [24650.641571] ret_from_fork+0x31/0x40 [24650.645151] ? kthread_complete_and_exit+0x20/0x20 [24650.649942] ret_from_fork_asm+0x11/0x20 [24650.653875] [24677.890202] watchdog: BUG: soft lockup - CPU#0 stuck for 11869s! [dbench:78683] [24677.897507] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24677.902202] watchdog: BUG: soft lockup - CPU#7 stuck for 20866s! [kworker/7:4:78808] [24677.952949] irq event stamp: 39332796 [24677.952950] hardirqs last enabled at (39332795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24677.960680] Modules linked in: [24677.964336] hardirqs last disabled at (39332796): [] sysvec_apic_timer_interrupt+0xe/0x90 [24677.974494] ceph [24677.977545] softirqs last enabled at (39332758): [] __do_softirq+0x2be/0x371 [24677.987270] libceph [24677.989193] softirqs last disabled at (39332753): [] irq_exit_rcu+0xb0/0xd0 [24677.997880] dns_resolver fscache [24678.000062] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24678.008573] netfs [24678.011900] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24678.020844] veth nft_chain_nat [24678.022855] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24678.030325] xt_MASQUERADE nf_nat [24678.033463] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24678.038939] nf_conntrack nf_defrag_ipv6 [24678.042250] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24678.060990] nf_defrag_ipv4 nft_compat [24678.064907] [24678.064907] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24678.070124] nf_tables nfnetlink [24678.073885] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24678.075375] bridge stp [24678.082499] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24678.085724] llc xfs [24678.092848] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24678.095292] libcrc32c sunrpc [24678.102417] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24678.104599] intel_rapl_msr intel_rapl_common [24678.111724] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24678.114688] sb_edac x86_pkg_temp_thermal [24678.121813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24678.126164] coretemp kvm_intel [24678.134240] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24678.138244] kvm irqbypass [24678.143981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24678.147119] i2c_i801 iTCO_wdt [24678.154245] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24678.156947] iTCO_vendor_support i2c_smbus [24678.164071] Call Trace: [24678.164072] [24678.167123] crct10dif_pclmul [24678.174249] ? watchdog_timer_fn+0x1b4/0x220 [24678.178337] joydev crc32_pclmul [24678.180783] ? __hrtimer_run_queues+0x1d3/0x360 [24678.182791] ipmi_ssif ghash_clmulni_intel [24678.185759] ? hrtimer_interrupt+0x104/0x240 [24678.190021] mei_me mei [24678.193246] ? __local_bh_enable+0x50/0x60 [24678.197770] lpc_ich [24678.201862] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24678.206123] sha512_ssse3 [24678.208569] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24678.212659] acpi_ipmi mfd_core [24678.214844] [24678.220061] wmi ipmi_si [24678.222677] [24678.222678] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24678.227634] ipmi_devintf [24678.230777] ? queued_spin_lock_slowpath+0x18/0x2e0 [24678.232880] ipmi_msghandler [24678.235411] do_raw_spin_lock+0xb5/0xc0 [24678.237506] ioatdma acpi_pad [24678.242811] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24678.245426] acpi_power_meter ixgbe [24678.250301] try_get_cap_refs+0x159/0x880 [ceph] [24678.253175] igb mdio [24678.257012] __ceph_get_caps+0x4a1/0x760 [ceph] [24678.259969] crc32c_intel nvme [24678.264759] ? __file_remove_privs+0x89/0x110 [24678.268238] ptp nvme_core [24678.272850] ? wake_bit_function+0x70/0x70 [24678.275120] i2c_algo_bit pps_core [24678.279646] ? ceph_write_iter+0x260/0x770 [ceph] [24678.282695] dca fuse [24678.287047] ceph_write_iter+0x260/0x770 [ceph] [24678.289749] [24678.289750] irq event stamp: 119553042 [24678.293844] ? lock_is_held_type+0xaa/0x120 [24678.297238] hardirqs last enabled at (119553041): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24678.301939] vfs_write+0x324/0x4f0 [24678.304205] hardirqs last disabled at (119553042): [] sysvec_apic_timer_interrupt+0xe/0x90 [24678.308735] ksys_pwrite64+0x65/0xa0 [24678.310221] softirqs last enabled at (119553040): [] __do_softirq+0x2be/0x371 [24678.313965] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24678.318143] softirqs last disabled at (119553035): [] irq_exit_rcu+0xb0/0xd0 [24678.328387] do_syscall_64+0x38/0x80 [24678.331784] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24678.341603] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24678.345175] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24678.353945] RIP: 0033:0x7f48f0eeb93b [24678.359075] Workqueue: events delayed_work [ceph] [24678.367673] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24678.371244] [24678.380620] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24678.385666] RIP: 0010:lock_acquire+0xf0/0x2b0 [24678.393135] ORIG_RAX: 0000000000000012 [24678.396708] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24678.401404] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24678.420143] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24678.421634] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24678.426849] [24678.426850] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24678.431202] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24678.435032] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24678.453771] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24678.460895] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24678.466110] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24678.473235] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24678.474731] [24678.481849] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24678.533913] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24678.541990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24678.547727] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24678.554850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24678.561975] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24678.569098] Call Trace: [24678.571545] [24678.573555] ? watchdog_timer_fn+0x1b4/0x220 [24678.577819] ? __hrtimer_run_queues+0x1d3/0x360 [24678.582347] ? hrtimer_interrupt+0x104/0x240 [24678.586614] ? __local_bh_enable+0x50/0x60 [24678.590708] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24678.595931] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24678.600889] [24678.602985] [24678.605085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24678.610392] ? lock_acquire+0xf0/0x2b0 [24678.614141] ? igrab+0x43/0x50 [24678.617193] ? lock_release+0xc5/0x260 [24678.620938] _raw_spin_lock+0x30/0x40 [24678.624610] ? igrab+0x1d/0x50 [24678.627661] igrab+0x1d/0x50 [24678.630540] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24678.635690] delayed_work+0x1d8/0x280 [ceph] [24678.639969] ? process_scheduled_works+0x22a/0x510 [24678.644751] process_scheduled_works+0x2a4/0x510 [24678.649368] worker_thread+0x188/0x340 [24678.653117] ? rescuer_thread+0x310/0x310 [24678.657121] kthread+0xf5/0x130 [24678.660258] ? kthread_complete_and_exit+0x20/0x20 [24678.665041] ret_from_fork+0x31/0x40 [24678.668612] ? kthread_complete_and_exit+0x20/0x20 [24678.673397] ret_from_fork_asm+0x11/0x20 [24678.677320] [24705.320038] rcu: INFO: rcu_preempt self-detected stall on CPU [24705.325804] rcu: 0-....: (12482668 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=6235952 [24705.336229] rcu: (t=12768483 jiffies g=1309609 q=402237 ncpus=8) [24705.342315] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24705.351266] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24705.358737] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24705.364223] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24705.382960] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24705.388178] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24705.395302] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24705.402426] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24705.409550] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24705.416675] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24705.423798] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24705.431875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24705.437612] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24705.444739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24705.451861] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24705.458985] Call Trace: [24705.461431] [24705.463442] ? rcu_dump_cpu_stacks+0xd1/0x180 [24705.467801] ? rcu_sched_clock_irq+0x633/0x1390 [24705.472333] ? lock_is_held_type+0xaa/0x120 [24705.476520] ? lock_is_held_type+0xaa/0x120 [24705.480698] ? tick_sched_do_timer+0xa0/0xa0 [24705.484968] ? rcu_read_lock_sched_held+0x56/0x90 [24705.489666] ? tick_sched_do_timer+0xa0/0xa0 [24705.493929] ? update_process_times+0x5f/0x90 [24705.498280] ? tick_sched_handle.isra.16+0x1e/0x50 [24705.503064] ? tick_sched_timer+0x6f/0x80 [24705.507071] ? __hrtimer_run_queues+0x1d3/0x360 [24705.511595] ? hrtimer_interrupt+0x104/0x240 [24705.515857] ? __local_bh_enable+0x50/0x60 [24705.519960] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24705.525184] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24705.530141] [24705.532237] [24705.534334] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24705.539644] ? queued_spin_lock_slowpath+0x16/0x2e0 [24705.544521] do_raw_spin_lock+0xb5/0xc0 [24705.548361] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24705.553170] try_get_cap_refs+0x159/0x880 [ceph] [24705.557800] __ceph_get_caps+0x4a1/0x760 [ceph] [24705.562341] ? __file_remove_privs+0x89/0x110 [24705.566697] ? wake_bit_function+0x70/0x70 [24705.570791] ? ceph_write_iter+0x260/0x770 [ceph] [24705.575502] ceph_write_iter+0x260/0x770 [ceph] [24705.580038] ? lock_is_held_type+0xaa/0x120 [24705.584224] vfs_write+0x324/0x4f0 [24705.587632] ksys_pwrite64+0x65/0xa0 [24705.591208] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24705.596347] do_syscall_64+0x38/0x80 [24705.599927] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24705.604970] RIP: 0033:0x7f48f0eeb93b [24705.608541] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24705.627277] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [24705.634835] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24705.641959] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24705.649083] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24705.656206] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24705.663333] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24705.670461] [24705.672648] Sending NMI from CPU 0 to CPUs 7: [24705.677002] NMI backtrace for cpu 7 [24705.677004] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24705.677006] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24705.677007] Workqueue: events delayed_work [ceph] [24705.677024] RIP: 0010:_raw_spin_unlock+0x0/0x40 [24705.677027] Code: 18 bb 01 00 00 00 45 31 c9 41 b8 01 00 00 00 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 [24705.677028] RSP: 0018:ffffc90002117d98 EFLAGS: 00000202 [24705.677029] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [24705.677030] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [24705.677031] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [24705.677032] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [24705.677032] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24705.677033] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24705.677034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24705.677035] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24705.677036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24705.677036] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24705.677037] Call Trace: [24705.677038] [24705.677039] ? nmi_cpu_backtrace+0xad/0x120 [24705.677043] ? nmi_cpu_backtrace_handler+0x11/0x20 [24705.677045] ? nmi_handle+0xd4/0x260 [24705.677050] ? default_do_nmi+0x49/0xf0 [24705.677052] ? exc_nmi+0x11e/0x150 [24705.677053] ? end_repeat_nmi+0x16/0x67 [24705.677058] ? _raw_spin_trylock_bh+0x70/0x70 [24705.677060] ? _raw_spin_trylock_bh+0x70/0x70 [24705.677062] ? _raw_spin_trylock_bh+0x70/0x70 [24705.677064] [24705.677065] [24705.677065] igrab+0x43/0x50 [24705.677067] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24705.677081] delayed_work+0x1d8/0x280 [ceph] [24705.677095] ? process_scheduled_works+0x22a/0x510 [24705.677098] process_scheduled_works+0x2a4/0x510 [24705.677102] worker_thread+0x188/0x340 [24705.677105] ? rescuer_thread+0x310/0x310 [24705.677107] kthread+0xf5/0x130 [24705.677110] ? kthread_complete_and_exit+0x20/0x20 [24705.677112] ret_from_fork+0x31/0x40 [24705.677115] ? kthread_complete_and_exit+0x20/0x20 [24705.677117] ret_from_fork_asm+0x11/0x20 [24705.677121] [24729.891092] watchdog: BUG: soft lockup - CPU#0 stuck for 11917s! [dbench:78683] [24729.898395] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24729.903091] watchdog: BUG: soft lockup - CPU#7 stuck for 20914s! [kworker/7:4:78808] [24729.953847] irq event stamp: 39441208 [24729.961578] Modules linked in: ceph libceph [24729.965233] hardirqs last enabled at (39441207): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24729.969412] dns_resolver fscache netfs [24729.979569] hardirqs last disabled at (39441208): [] sysvec_apic_timer_interrupt+0xe/0x90 [24729.983398] veth nft_chain_nat xt_MASQUERADE nf_nat [24729.993124] softirqs last enabled at (39441176): [] __do_softirq+0x2be/0x371 [24729.993126] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [24730.006768] softirqs last disabled at (39441171): [] irq_exit_rcu+0xb0/0xd0 [24730.014679] bridge stp llc [24730.023189] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24730.025980] xfs libcrc32c [24730.034923] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24730.034925] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24730.037629] sunrpc [24730.045098] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24730.050576] intel_rapl_msr intel_rapl_common [24730.052673] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24730.071410] sb_edac [24730.075778] [24730.075779] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24730.080995] x86_pkg_temp_thermal [24730.083180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24730.084672] coretemp kvm_intel [24730.091796] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24730.095107] kvm [24730.102230] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24730.105368] irqbypass [24730.112490] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24730.112492] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24730.114330] i2c_i801 iTCO_wdt [24730.121453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24730.123809] iTCO_vendor_support i2c_smbus [24730.130934] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24730.139011] crct10dif_pclmul joydev [24730.142063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24730.147801] crc32_pclmul ipmi_ssif [24730.151890] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24730.159015] ghash_clmulni_intel mei_me [24730.162587] Call Trace: [24730.169710] mei lpc_ich [24730.173194] [24730.180318] sha512_ssse3 [24730.184149] ? watchdog_timer_fn+0x1b4/0x220 [24730.186594] acpi_ipmi mfd_core [24730.189125] ? __hrtimer_run_queues+0x1d3/0x360 [24730.191134] wmi ipmi_si [24730.193771] ? hrtimer_interrupt+0x104/0x240 [24730.198034] ipmi_devintf [24730.201171] ? __local_bh_enable+0x50/0x60 [24730.205695] ipmi_msghandler ioatdma [24730.208228] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24730.212489] acpi_pad [24730.215107] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24730.219199] acpi_power_meter [24730.222774] [24730.227994] ixgbe igb [24730.230266] [24730.230267] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24730.235223] mdio [24730.238191] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24730.240284] crc32c_intel [24730.242644] do_raw_spin_lock+0xb5/0xc0 [24730.244737] nvme ptp [24730.250044] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24730.251966] nvme_core i2c_algo_bit [24730.256841] try_get_cap_refs+0x159/0x880 [ceph] [24730.259454] pps_core dca [24730.263292] __ceph_get_caps+0x4a1/0x760 [ceph] [24730.265556] fuse [24730.270345] ? __file_remove_privs+0x89/0x110 [24730.273824] irq event stamp: 119966632 [24730.278435] ? wake_bit_function+0x70/0x70 [24730.281052] hardirqs last enabled at (119966631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24730.285581] ? ceph_write_iter+0x260/0x770 [ceph] [24730.287501] hardirqs last disabled at (119966632): [] sysvec_apic_timer_interrupt+0xe/0x90 [24730.291852] ceph_write_iter+0x260/0x770 [ceph] [24730.295597] softirqs last enabled at (119966630): [] __do_softirq+0x2be/0x371 [24730.299689] ? lock_is_held_type+0xaa/0x120 [24730.309931] softirqs last disabled at (119966625): [] irq_exit_rcu+0xb0/0xd0 [24730.314632] vfs_write+0x324/0x4f0 [24730.324439] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24730.328969] ksys_pwrite64+0x65/0xa0 [24730.337733] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24730.341912] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24730.350508] Workqueue: events delayed_work [ceph] [24730.353906] do_syscall_64+0x38/0x80 [24730.363284] [24730.366855] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24730.374325] RIP: 0010:lock_acquire+0xf0/0x2b0 [24730.379455] RIP: 0033:0x7f48f0eeb93b [24730.384153] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24730.387724] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24730.389215] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24730.394258] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24730.398609] [24730.398610] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24730.402180] ORIG_RAX: 0000000000000012 [24730.420918] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24730.439656] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24730.444872] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24730.450091] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24730.451581] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24730.458705] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24730.462535] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24730.469661] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24730.476785] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24730.483908] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24730.491033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24730.498162] [24730.505281] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24730.549785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24730.556910] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24730.564035] Call Trace: [24730.566497] [24730.568505] ? watchdog_timer_fn+0x1b4/0x220 [24730.572780] ? __hrtimer_run_queues+0x1d3/0x360 [24730.577307] ? hrtimer_interrupt+0x104/0x240 [24730.581576] ? __local_bh_enable+0x50/0x60 [24730.585670] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24730.590893] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24730.595849] [24730.597947] [24730.600046] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24730.605352] ? lock_acquire+0xf0/0x2b0 [24730.609103] ? igrab+0x43/0x50 [24730.612153] ? lock_release+0xc5/0x260 [24730.615898] _raw_spin_lock+0x30/0x40 [24730.619562] ? igrab+0x1d/0x50 [24730.622612] igrab+0x1d/0x50 [24730.625498] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24730.630650] delayed_work+0x1d8/0x280 [ceph] [24730.634930] ? process_scheduled_works+0x22a/0x510 [24730.639721] process_scheduled_works+0x2a4/0x510 [24730.644337] worker_thread+0x188/0x340 [24730.648086] ? rescuer_thread+0x310/0x310 [24730.652089] kthread+0xf5/0x130 [24730.655227] ? kthread_complete_and_exit+0x20/0x20 [24730.660010] ret_from_fork+0x31/0x40 [24730.663581] ? kthread_complete_and_exit+0x20/0x20 [24730.668364] ret_from_fork_asm+0x11/0x20 [24730.672289] [24757.891571] watchdog: BUG: soft lockup - CPU#0 stuck for 11943s! [dbench:78683] [24757.898876] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24757.903570] watchdog: BUG: soft lockup - CPU#7 stuck for 20940s! [kworker/7:4:78808] [24757.954324] irq event stamp: 39499434 [24757.962054] Modules linked in: ceph libceph dns_resolver fscache netfs [24757.965713] hardirqs last enabled at (39499433): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24757.972246] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [24757.982405] hardirqs last disabled at (39499434): [] sysvec_apic_timer_interrupt+0xe/0x90 [24757.982408] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [24757.998213] softirqs last enabled at (39499358): [] __do_softirq+0x2be/0x371 [24758.006298] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [24758.014984] softirqs last disabled at (39499353): [] irq_exit_rcu+0xb0/0xd0 [24758.021163] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [24758.029674] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24758.035325] kvm irqbypass [24758.044268] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24758.044269] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24758.046972] i2c_i801 [24758.054443] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24758.059921] iTCO_wdt [24758.062191] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24758.080930] iTCO_vendor_support [24758.083202] [24758.088417] i2c_smbus crct10dif_pclmul [24758.091643] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24758.093133] joydev crc32_pclmul [24758.096965] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24758.104086] ipmi_ssif ghash_clmulni_intel [24758.107312] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24758.114435] mei_me [24758.118525] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24758.125650] mei lpc_ich [24758.127748] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24758.134871] sha512_ssse3 [24758.137403] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24758.144525] acpi_ipmi mfd_core [24758.147144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24758.155220] wmi [24758.158359] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24758.164097] ipmi_si ipmi_devintf [24758.165933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24758.173058] ipmi_msghandler [24758.176367] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24758.183493] ioatdma acpi_pad [24758.186370] Call Trace: [24758.186372] [24758.193493] acpi_power_meter ixgbe [24758.196459] ? watchdog_timer_fn+0x1b4/0x220 [24758.198901] igb mdio [24758.200913] ? __hrtimer_run_queues+0x1d3/0x360 [24758.204396] crc32c_intel nvme [24758.208665] ? hrtimer_interrupt+0x104/0x240 [24758.210932] ptp nvme_core [24758.215455] ? __local_bh_enable+0x50/0x60 [24758.218505] i2c_algo_bit [24758.222774] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24758.225475] pps_core [24758.229565] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24758.232183] dca fuse [24758.237401] [24758.239672] [24758.239673] irq event stamp: 120187470 [24758.244628] [24758.246898] hardirqs last enabled at (120187469): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24758.248998] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24758.250486] hardirqs last disabled at (120187470): [] sysvec_apic_timer_interrupt+0xe/0x90 [24758.254251] ? queued_spin_lock_slowpath+0x18/0x2e0 [24758.256345] softirqs last enabled at (120187468): [] __do_softirq+0x2be/0x371 [24758.266592] do_raw_spin_lock+0xb5/0xc0 [24758.271894] softirqs last disabled at (120187463): [] irq_exit_rcu+0xb0/0xd0 [24758.281706] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24758.286575] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24758.295350] try_get_cap_refs+0x159/0x880 [ceph] [24758.299177] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24758.307780] __ceph_get_caps+0x4a1/0x760 [ceph] [24758.312559] Workqueue: events delayed_work [ceph] [24758.321940] ? __file_remove_privs+0x89/0x110 [24758.326548] [24758.334020] ? wake_bit_function+0x70/0x70 [24758.338542] RIP: 0010:lock_release+0x14e/0x260 [24758.343242] ? ceph_write_iter+0x260/0x770 [ceph] [24758.347589] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [24758.349081] ceph_write_iter+0x260/0x770 [ceph] [24758.353173] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [24758.357612] ? lock_is_held_type+0xaa/0x120 [24758.362306] [24758.362307] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [24758.381046] vfs_write+0x324/0x4f0 [24758.385569] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [24758.390792] ksys_pwrite64+0x65/0xa0 [24758.394973] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24758.396462] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24758.403586] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24758.406985] do_syscall_64+0x38/0x80 [24758.414109] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [24758.417681] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24758.424802] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24758.429934] RIP: 0033:0x7f48f0eeb93b [24758.437057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24758.440630] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24758.447753] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24758.452797] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24758.460874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24758.464445] ORIG_RAX: 0000000000000012 [24758.470181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24758.488921] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24758.496044] Call Trace: [24758.501261] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24758.508388] [24758.512216] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24758.519343] ? watchdog_timer_fn+0x1b4/0x220 [24758.526464] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24758.528910] ? __hrtimer_run_queues+0x1d3/0x360 [24758.536032] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24758.538048] ? hrtimer_interrupt+0x104/0x240 [24758.545172] [24758.549433] ? __local_bh_enable+0x50/0x60 [24758.578747] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24758.583975] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24758.588935] [24758.591034] [24758.593131] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24758.598435] ? igrab+0x43/0x50 [24758.601487] ? lock_release+0x14e/0x260 [24758.605317] ? lock_release+0xc5/0x260 [24758.609062] _raw_spin_unlock+0x1b/0x40 [24758.612901] igrab+0x43/0x50 [24758.615779] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24758.620927] delayed_work+0x1d8/0x280 [ceph] [24758.625209] ? process_scheduled_works+0x22a/0x510 [24758.629999] process_scheduled_works+0x2a4/0x510 [24758.634616] worker_thread+0x188/0x340 [24758.638362] ? rescuer_thread+0x310/0x310 [24758.642366] kthread+0xf5/0x130 [24758.645505] ? kthread_complete_and_exit+0x20/0x20 [24758.650289] ret_from_fork+0x31/0x40 [24758.653870] ? kthread_complete_and_exit+0x20/0x20 [24758.658652] ret_from_fork_asm+0x11/0x20 [24758.662584] [24785.892050] watchdog: BUG: soft lockup - CPU#0 stuck for 11969s! [dbench:78683] [24785.899353] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24785.904050] watchdog: BUG: soft lockup - CPU#7 stuck for 20966s! [kworker/7:4:78808] [24785.954793] irq event stamp: 39557566 [24785.954794] hardirqs last enabled at (39557565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24785.962525] Modules linked in: [24785.966183] hardirqs last disabled at (39557566): [] sysvec_apic_timer_interrupt+0xe/0x90 [24785.976339] ceph [24785.979391] softirqs last enabled at (39557488): [] __do_softirq+0x2be/0x371 [24785.989114] libceph [24785.991038] softirqs last disabled at (39557483): [] irq_exit_rcu+0xb0/0xd0 [24785.999725] dns_resolver [24786.001907] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24786.010417] fscache netfs [24786.013035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24786.021980] veth nft_chain_nat [24786.024683] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24786.032155] xt_MASQUERADE nf_nat [24786.035293] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24786.040770] nf_conntrack nf_defrag_ipv6 [24786.044080] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24786.062818] nf_defrag_ipv4 nft_compat [24786.066735] [24786.066735] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24786.071953] nf_tables [24786.075697] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24786.077186] nfnetlink [24786.084312] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24786.086669] bridge stp [24786.093792] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24786.096150] llc xfs [24786.103274] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24786.105727] libcrc32c sunrpc [24786.112852] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24786.115036] intel_rapl_msr intel_rapl_common [24786.122160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24786.125124] sb_edac x86_pkg_temp_thermal [24786.133200] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24786.137550] coretemp kvm_intel [24786.143288] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24786.147293] kvm [24786.154418] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24786.157553] irqbypass i2c_i801 [24786.164679] Call Trace: [24786.164680] [24786.166517] iTCO_wdt iTCO_vendor_support [24786.173641] ? watchdog_timer_fn+0x1b4/0x220 [24786.176778] i2c_smbus crct10dif_pclmul [24786.179223] ? __hrtimer_run_queues+0x1d3/0x360 [24786.181233] joydev [24786.185240] ? hrtimer_interrupt+0x104/0x240 [24786.189500] crc32_pclmul [24786.193332] ? __local_bh_enable+0x50/0x60 [24786.197854] ipmi_ssif [24786.199956] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24786.204215] ghash_clmulni_intel [24786.206834] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24786.210924] mei_me mei [24786.213283] [24786.218499] lpc_ich [24786.221738] [24786.221739] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24786.226697] sha512_ssse3 acpi_ipmi [24786.229145] ? queued_spin_lock_slowpath+0x16/0x2e0 [24786.231240] mfd_core wmi [24786.233445] do_raw_spin_lock+0xb5/0xc0 [24786.235538] ipmi_si [24786.240843] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24786.244328] ipmi_devintf ipmi_msghandler ioatdma [24786.249200] try_get_cap_refs+0x159/0x880 [ceph] [24786.251814] acpi_pad acpi_power_meter [24786.255652] __ceph_get_caps+0x4a1/0x760 [ceph] [24786.257830] ixgbe igb [24786.262619] ? __file_remove_privs+0x89/0x110 [24786.267311] mdio [24786.271923] ? wake_bit_function+0x70/0x70 [24786.275666] crc32c_intel [24786.280193] ? ceph_write_iter+0x260/0x770 [ceph] [24786.282547] nvme ptp [24786.286898] ceph_write_iter+0x260/0x770 [ceph] [24786.288823] nvme_core i2c_algo_bit [24786.292916] ? lock_is_held_type+0xaa/0x120 [24786.295530] pps_core dca [24786.300232] vfs_write+0x324/0x4f0 [24786.302498] fuse [24786.307026] ksys_pwrite64+0x65/0xa0 [24786.310507] irq event stamp: 120408436 [24786.314684] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24786.317303] hardirqs last enabled at (120408435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24786.320701] do_syscall_64+0x38/0x80 [24786.322623] hardirqs last disabled at (120408436): [] sysvec_apic_timer_interrupt+0xe/0x90 [24786.326195] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24786.329938] softirqs last enabled at (120408434): [] __do_softirq+0x2be/0x371 [24786.335069] RIP: 0033:0x7f48f0eeb93b [24786.345313] softirqs last disabled at (120408429): [] irq_exit_rcu+0xb0/0xd0 [24786.348884] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24786.358695] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24786.363746] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24786.372517] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24786.376088] ORIG_RAX: 0000000000000012 [24786.384687] Workqueue: events delayed_work [ceph] [24786.403425] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24786.412818] [24786.412819] RIP: 0010:lock_acquire+0x85/0x2b0 [24786.418035] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24786.425507] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [24786.429338] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24786.434036] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [24786.441160] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24786.442650] [24786.442651] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [24786.447001] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24786.454125] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [24786.472869] [24786.479986] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [24786.524500] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [24786.531624] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24786.538753] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24786.546833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24786.552571] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24786.559696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24786.566819] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24786.573943] Call Trace: [24786.576390] [24786.578400] ? watchdog_timer_fn+0x1b4/0x220 [24786.582664] ? __hrtimer_run_queues+0x1d3/0x360 [24786.587189] ? hrtimer_interrupt+0x104/0x240 [24786.591456] ? __local_bh_enable+0x50/0x60 [24786.595553] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24786.600776] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24786.605736] [24786.607830] [24786.609929] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24786.615236] ? lock_acquire+0x85/0x2b0 [24786.618988] ? igrab+0x43/0x50 [24786.622045] ? lock_release+0xc5/0x260 [24786.625790] _raw_spin_lock+0x30/0x40 [24786.629458] ? igrab+0x1d/0x50 [24786.632515] igrab+0x1d/0x50 [24786.635393] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24786.640543] delayed_work+0x1d8/0x280 [ceph] [24786.644824] ? process_scheduled_works+0x22a/0x510 [24786.649614] process_scheduled_works+0x2a4/0x510 [24786.654229] worker_thread+0x188/0x340 [24786.657979] ? rescuer_thread+0x310/0x310 [24786.661982] kthread+0xf5/0x130 [24786.665119] ? kthread_complete_and_exit+0x20/0x20 [24786.669905] ret_from_fork+0x31/0x40 [24786.673483] ? kthread_complete_and_exit+0x20/0x20 [24786.678267] ret_from_fork_asm+0x11/0x20 [24786.682190] [24813.892532] watchdog: BUG: soft lockup - CPU#0 stuck for 11995s! [dbench:78683] [24813.899832] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24813.904531] watchdog: BUG: soft lockup - CPU#7 stuck for 20992s! [kworker/7:4:78808] [24813.955282] irq event stamp: 39615866 [24813.963012] Modules linked in: ceph libceph dns_resolver fscache netfs [24813.966669] hardirqs last enabled at (39615865): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24813.973187] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [24813.983344] hardirqs last disabled at (39615866): [] sysvec_apic_timer_interrupt+0xe/0x90 [24813.990729] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [24814.000454] softirqs last enabled at (39615788): [] __do_softirq+0x2be/0x371 [24814.007238] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [24814.015925] softirqs last disabled at (39615783): [] irq_exit_rcu+0xb0/0xd0 [24814.015929] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [24814.031306] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24814.037479] i2c_i801 iTCO_wdt [24814.046421] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24814.046423] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24814.049472] iTCO_vendor_support i2c_smbus [24814.056945] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24814.062420] crct10dif_pclmul joydev [24814.066511] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24814.085249] crc32_pclmul ipmi_ssif [24814.088820] [24814.088821] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24814.094038] ghash_clmulni_intel [24814.097521] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24814.099013] mei_me [24814.106135] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24814.109360] mei lpc_ich [24814.116483] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24814.118582] sha512_ssse3 acpi_ipmi [24814.125705] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24814.128237] mfd_core wmi [24814.135361] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24814.138845] ipmi_si [24814.145970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24814.148587] ipmi_devintf ipmi_msghandler [24814.156663] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24814.158848] ioatdma acpi_pad [24814.164585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24814.168588] acpi_power_meter [24814.175712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24814.178678] ixgbe igb [24814.185803] Call Trace: [24814.188767] mdio crc32c_intel [24814.195893] [24814.198246] nvme ptp [24814.200695] ? watchdog_timer_fn+0x1b4/0x220 [24814.203742] nvme_core i2c_algo_bit [24814.205753] ? __hrtimer_run_queues+0x1d3/0x360 [24814.208024] pps_core [24814.212291] ? hrtimer_interrupt+0x104/0x240 [24814.215771] dca fuse [24814.220295] ? __local_bh_enable+0x50/0x60 [24814.222567] [24814.222568] irq event stamp: 120629188 [24814.226833] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24814.229101] hardirqs last enabled at (120629187): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24814.233208] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24814.234698] hardirqs last disabled at (120629188): [] sysvec_apic_timer_interrupt+0xe/0x90 [24814.238444] [24814.243661] softirqs last enabled at (120629186): [] __do_softirq+0x2be/0x371 [24814.253906] [24814.258880] softirqs last disabled at (120629181): [] irq_exit_rcu+0xb0/0xd0 [24814.268693] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24814.270789] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24814.279563] ? queued_spin_lock_slowpath+0x16/0x2e0 [24814.281657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24814.290257] do_raw_spin_lock+0xb5/0xc0 [24814.295559] Workqueue: events delayed_work [ceph] [24814.304939] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24814.309805] [24814.309806] RIP: 0010:lock_acquire+0xf0/0x2b0 [24814.317279] try_get_cap_refs+0x159/0x880 [ceph] [24814.321107] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24814.325810] __ceph_get_caps+0x4a1/0x760 [ceph] [24814.330590] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24814.332085] ? __file_remove_privs+0x89/0x110 [24814.336432] [24814.336432] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24814.341042] ? wake_bit_function+0x70/0x70 [24814.359780] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24814.364306] ? ceph_write_iter+0x260/0x770 [ceph] [24814.369522] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24814.373871] ceph_write_iter+0x260/0x770 [ceph] [24814.375361] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24814.382489] ? lock_is_held_type+0xaa/0x120 [24814.386577] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24814.393703] vfs_write+0x324/0x4f0 [24814.398399] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24814.405528] ksys_pwrite64+0x65/0xa0 [24814.410047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24814.417173] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24814.421349] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24814.428474] do_syscall_64+0x38/0x80 [24814.431869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24814.439949] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24814.443518] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24814.449256] RIP: 0033:0x7f48f0eeb93b [24814.454385] Call Trace: [24814.461511] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24814.465082] [24814.472221] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24814.477268] ? watchdog_timer_fn+0x1b4/0x220 [24814.484390] ORIG_RAX: 0000000000000012 [24814.487962] ? __hrtimer_run_queues+0x1d3/0x360 [24814.490407] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24814.509148] ? hrtimer_interrupt+0x104/0x240 [24814.511155] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24814.516373] ? __local_bh_enable+0x50/0x60 [24814.520634] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24814.524469] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24814.528991] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24814.536116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24814.540379] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24814.547504] [24814.551599] [24814.558717] [24814.558719] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24814.594829] ? lock_acquire+0xf0/0x2b0 [24814.598579] ? igrab+0x43/0x50 [24814.601628] ? lock_release+0xc5/0x260 [24814.605374] _raw_spin_lock+0x30/0x40 [24814.609040] ? igrab+0x1d/0x50 [24814.612089] igrab+0x1d/0x50 [24814.614969] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24814.620116] delayed_work+0x1d8/0x280 [ceph] [24814.624397] ? process_scheduled_works+0x22a/0x510 [24814.629189] process_scheduled_works+0x2a4/0x510 [24814.633804] worker_thread+0x188/0x340 [24814.637552] ? rescuer_thread+0x310/0x310 [24814.641555] kthread+0xf5/0x130 [24814.644692] ? kthread_complete_and_exit+0x20/0x20 [24814.649479] ret_from_fork+0x31/0x40 [24814.653057] ? kthread_complete_and_exit+0x20/0x20 [24814.657842] ret_from_fork_asm+0x11/0x20 [24814.661772] [24841.893011] watchdog: BUG: soft lockup - CPU#0 stuck for 12021s! [dbench:78683] [24841.900311] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24841.905011] watchdog: BUG: soft lockup - CPU#7 stuck for 21018s! [kworker/7:4:78808] [24841.955752] irq event stamp: 39673932 [24841.955753] hardirqs last enabled at (39673931): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24841.963481] Modules linked in: [24841.967139] hardirqs last disabled at (39673932): [] sysvec_apic_timer_interrupt+0xe/0x90 [24841.977297] ceph [24841.980348] softirqs last enabled at (39673874): [] __do_softirq+0x2be/0x371 [24841.990073] libceph [24841.991996] softirqs last disabled at (39673869): [] irq_exit_rcu+0xb0/0xd0 [24842.000685] dns_resolver fscache [24842.002873] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24842.011402] netfs veth [24842.014713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24842.023657] nft_chain_nat [24842.026102] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24842.033571] xt_MASQUERADE nf_nat [24842.036275] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24842.041752] nf_conntrack nf_defrag_ipv6 [24842.045064] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24842.063801] nf_defrag_ipv4 nft_compat [24842.067719] [24842.067720] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24842.072936] nf_tables nfnetlink [24842.076685] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24842.078178] bridge [24842.085302] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24842.088528] stp llc [24842.095652] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24842.097748] xfs libcrc32c [24842.104874] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24842.107058] sunrpc intel_rapl_msr [24842.114180] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24842.116884] intel_rapl_common sb_edac [24842.124010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24842.127409] x86_pkg_temp_thermal coretemp [24842.135483] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24842.139228] kvm_intel kvm [24842.144966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24842.149057] irqbypass i2c_i801 [24842.156179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24842.158885] iTCO_wdt iTCO_vendor_support [24842.166008] Call Trace: [24842.169145] i2c_smbus crct10dif_pclmul [24842.176270] [24842.180274] joydev crc32_pclmul [24842.182719] ? watchdog_timer_fn+0x1b4/0x220 [24842.186549] ipmi_ssif ghash_clmulni_intel [24842.188560] ? __hrtimer_run_queues+0x1d3/0x360 [24842.191784] mei_me [24842.196052] ? hrtimer_interrupt+0x104/0x240 [24842.200139] mei lpc_ich [24842.204661] ? __local_bh_enable+0x50/0x60 [24842.206760] sha512_ssse3 acpi_ipmi [24842.211026] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24842.213555] mfd_core wmi [24842.217645] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24842.221129] ipmi_si ipmi_devintf [24842.226348] [24842.228965] ipmi_msghandler ioatdma [24842.233921] [24842.233922] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24842.237233] acpi_pad acpi_power_meter [24842.239333] ? queued_spin_lock_slowpath+0x16/0x2e0 [24842.242899] ixgbe igb [24842.244999] do_raw_spin_lock+0xb5/0xc0 [24842.250303] mdio crc32c_intel [24842.254046] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24842.258918] nvme ptp [24842.261276] try_get_cap_refs+0x159/0x880 [ceph] [24842.265104] nvme_core i2c_algo_bit [24842.268161] __ceph_get_caps+0x4a1/0x760 [ceph] [24842.272940] pps_core dca [24842.275216] ? __file_remove_privs+0x89/0x110 [24842.279821] fuse [24842.283305] ? wake_bit_function+0x70/0x70 [24842.287830] irq event stamp: 120850150 [24842.290448] ? ceph_write_iter+0x260/0x770 [ceph] [24842.294797] hardirqs last enabled at (120850149): [] asm_common_interrupt+0x26/0x40 [24842.296721] ceph_write_iter+0x260/0x770 [ceph] [24842.300813] hardirqs last disabled at (120850150): [] sysvec_apic_timer_interrupt+0xe/0x90 [24842.304560] ? lock_is_held_type+0xaa/0x120 [24842.309254] softirqs last enabled at (120850148): [] __do_softirq+0x2be/0x371 [24842.318546] vfs_write+0x324/0x4f0 [24842.323068] softirqs last disabled at (120850143): [] irq_exit_rcu+0xb0/0xd0 [24842.332885] ksys_pwrite64+0x65/0xa0 [24842.337058] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24842.345827] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24842.349225] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24842.349226] Workqueue: events delayed_work [ceph] [24842.357825] do_syscall_64+0x38/0x80 [24842.361411] [24842.370788] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24842.375919] RIP: 0010:ceph_check_delayed_caps+0x6f/0x270 [ceph] [24842.383389] RIP: 0033:0x7f48f0eeb93b [24842.388104] Code: 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 <48> 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 [24842.391692] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24842.393183] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [24842.398226] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24842.404138] [24842.404139] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [24842.407708] ORIG_RAX: 0000000000000012 [24842.426446] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [24842.445184] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24842.450415] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [24842.455637] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24842.457128] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [24842.464252] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24842.468083] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24842.475205] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24842.482329] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24842.489453] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24842.496578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24842.503708] [24842.510828] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24842.555324] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24842.562448] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24842.569570] Call Trace: [24842.572017] [24842.574027] ? watchdog_timer_fn+0x1b4/0x220 [24842.578292] ? __hrtimer_run_queues+0x1d3/0x360 [24842.582826] ? hrtimer_interrupt+0x104/0x240 [24842.587096] ? __local_bh_enable+0x50/0x60 [24842.591190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24842.596417] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24842.601379] [24842.603475] [24842.605573] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24842.610881] ? ceph_check_delayed_caps+0x6f/0x270 [ceph] [24842.616199] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [24842.621514] delayed_work+0x1d8/0x280 [ceph] [24842.625795] ? process_scheduled_works+0x22a/0x510 [24842.630585] process_scheduled_works+0x2a4/0x510 [24842.635202] worker_thread+0x188/0x340 [24842.638951] ? rescuer_thread+0x310/0x310 [24842.642961] kthread+0xf5/0x130 [24842.646100] ? kthread_complete_and_exit+0x20/0x20 [24842.650884] ret_from_fork+0x31/0x40 [24842.654453] ? kthread_complete_and_exit+0x20/0x20 [24842.659237] ret_from_fork_asm+0x11/0x20 [24842.663161] [24869.893488] watchdog: BUG: soft lockup - CPU#0 stuck for 12047s! [dbench:78683] [24869.900790] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24869.905488] watchdog: BUG: soft lockup - CPU#7 stuck for 21044s! [kworker/7:4:78808] [24869.956239] irq event stamp: 39732158 [24869.963971] Modules linked in: ceph libceph dns_resolver fscache netfs [24869.967628] hardirqs last enabled at (39732157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24869.974168] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [24869.984328] hardirqs last disabled at (39732158): [] sysvec_apic_timer_interrupt+0xe/0x90 [24869.991712] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [24870.001436] softirqs last enabled at (39732100): [] __do_softirq+0x2be/0x371 [24870.001440] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [24870.016569] softirqs last disabled at (39732095): [] irq_exit_rcu+0xb0/0xd0 [24870.023085] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [24870.031598] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24870.037248] kvm irqbypass [24870.046191] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24870.046193] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24870.048896] i2c_i801 [24870.056368] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24870.061844] iTCO_wdt [24870.064115] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24870.082853] iTCO_vendor_support i2c_smbus [24870.085123] [24870.090342] crct10dif_pclmul joydev [24870.094431] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24870.095922] crc32_pclmul ipmi_ssif [24870.099493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24870.106616] ghash_clmulni_intel [24870.110101] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24870.117227] mei_me [24870.120449] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24870.127572] mei lpc_ich [24870.129671] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24870.136794] sha512_ssse3 acpi_ipmi [24870.139326] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24870.146449] mfd_core wmi [24870.149934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24870.158012] ipmi_si ipmi_devintf [24870.160630] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24870.166365] ipmi_msghandler ioatdma [24870.169676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24870.176800] acpi_pad acpi_power_meter [24870.180371] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24870.187496] ixgbe igb [24870.191241] Call Trace: [24870.191243] [24870.198365] mdio crc32c_intel [24870.200723] ? watchdog_timer_fn+0x1b4/0x220 [24870.203168] nvme ptp [24870.205178] ? __hrtimer_run_queues+0x1d3/0x360 [24870.208225] nvme_core i2c_algo_bit [24870.212495] ? hrtimer_interrupt+0x104/0x240 [24870.214761] pps_core dca [24870.219285] ? __local_bh_enable+0x50/0x60 [24870.222770] fuse [24870.227037] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24870.229651] irq event stamp: 121071112 [24870.233742] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24870.235667] hardirqs last enabled at (121071111): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24870.240886] [24870.244627] hardirqs last disabled at (121071112): [] sysvec_apic_timer_interrupt+0xe/0x90 [24870.249585] [24870.259829] softirqs last enabled at (121071110): [] __do_softirq+0x2be/0x371 [24870.261926] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24870.271738] softirqs last disabled at (121071105): [] irq_exit_rcu+0xb0/0xd0 [24870.273837] ? queued_spin_lock_slowpath+0x18/0x2e0 [24870.282605] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24870.287911] do_raw_spin_lock+0xb5/0xc0 [24870.296506] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24870.301379] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24870.310756] Workqueue: events delayed_work [ceph] [24870.314588] try_get_cap_refs+0x159/0x880 [ceph] [24870.322057] [24870.326846] __ceph_get_caps+0x4a1/0x760 [ceph] [24870.331539] RIP: 0010:lock_acquire+0xf0/0x2b0 [24870.336170] ? __file_remove_privs+0x89/0x110 [24870.337657] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24870.342182] ? wake_bit_function+0x70/0x70 [24870.346531] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24870.350892] ? ceph_write_iter+0x260/0x770 [ceph] [24870.369630] [24870.369631] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24870.373721] ceph_write_iter+0x260/0x770 [ceph] [24870.378938] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24870.383638] ? lock_is_held_type+0xaa/0x120 [24870.385126] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24870.392253] vfs_write+0x324/0x4f0 [24870.396773] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24870.403903] ksys_pwrite64+0x65/0xa0 [24870.408076] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24870.415200] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24870.418597] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24870.425722] do_syscall_64+0x38/0x80 [24870.429290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24870.436416] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24870.441545] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24870.449625] RIP: 0033:0x7f48f0eeb93b [24870.453194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24870.458933] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24870.463977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24870.471101] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24870.474670] Call Trace: [24870.474672] [24870.481795] ORIG_RAX: 0000000000000012 [24870.500533] ? watchdog_timer_fn+0x1b4/0x220 [24870.507658] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24870.512892] ? __hrtimer_run_queues+0x1d3/0x360 [24870.515336] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24870.517349] ? hrtimer_interrupt+0x104/0x240 [24870.521182] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24870.525450] ? __local_bh_enable+0x50/0x60 [24870.532573] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24870.537101] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24870.544223] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24870.548486] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24870.555617] [24870.559703] [24870.588424] [24870.590521] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24870.595829] ? lock_acquire+0xf0/0x2b0 [24870.599578] ? igrab+0x43/0x50 [24870.602629] ? lock_release+0xc5/0x260 [24870.606375] _raw_spin_lock+0x30/0x40 [24870.610038] ? igrab+0x1d/0x50 [24870.613091] igrab+0x1d/0x50 [24870.615968] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24870.621119] delayed_work+0x1d8/0x280 [ceph] [24870.625399] ? process_scheduled_works+0x22a/0x510 [24870.630190] process_scheduled_works+0x2a4/0x510 [24870.634805] worker_thread+0x188/0x340 [24870.638553] ? rescuer_thread+0x310/0x310 [24870.642557] kthread+0xf5/0x130 [24870.645694] ? kthread_complete_and_exit+0x20/0x20 [24870.650480] ret_from_fork+0x31/0x40 [24870.654059] ? kthread_complete_and_exit+0x20/0x20 [24870.658843] ret_from_fork_asm+0x11/0x20 [24870.662773] [24897.893965] watchdog: BUG: soft lockup - CPU#0 stuck for 12073s! [dbench:78683] [24897.901267] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24897.905965] watchdog: BUG: soft lockup - CPU#7 stuck for 21070s! [kworker/7:4:78808] [24897.956710] irq event stamp: 39790292 [24897.956712] hardirqs last enabled at (39790291): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24897.964440] Modules linked in: ceph [24897.968098] hardirqs last disabled at (39790292): [] sysvec_apic_timer_interrupt+0xe/0x90 [24897.978254] libceph [24897.981739] softirqs last enabled at (39790234): [] __do_softirq+0x2be/0x371 [24897.991461] dns_resolver fscache [24897.993648] softirqs last disabled at (39790229): [] irq_exit_rcu+0xb0/0xd0 [24898.002331] netfs veth [24898.005644] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24898.014151] nft_chain_nat xt_MASQUERADE [24898.016596] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24898.025540] nf_nat nf_conntrack [24898.029459] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [24898.036929] nf_defrag_ipv6 nf_defrag_ipv4 [24898.040152] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [24898.045643] nft_compat nf_tables [24898.049739] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24898.068477] nfnetlink bridge [24898.071787] [24898.071788] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24898.077004] stp llc [24898.079968] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24898.081458] xfs libcrc32c [24898.088584] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24898.090768] sunrpc intel_rapl_msr [24898.097892] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24898.100594] intel_rapl_common sb_edac [24898.107720] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24898.111117] x86_pkg_temp_thermal coretemp [24898.118241] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24898.121986] kvm_intel kvm [24898.129109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24898.133199] irqbypass i2c_i801 [24898.141278] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24898.143983] iTCO_wdt iTCO_vendor_support [24898.149718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24898.152855] i2c_smbus crct10dif_pclmul [24898.159982] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24898.163983] joydev crc32_pclmul [24898.171108] Call Trace: [24898.174939] ipmi_ssif ghash_clmulni_intel [24898.182064] [24898.185288] mei_me mei [24898.187735] ? watchdog_timer_fn+0x1b4/0x220 [24898.191824] lpc_ich sha512_ssse3 [24898.193833] ? __hrtimer_run_queues+0x1d3/0x360 [24898.196276] acpi_ipmi mfd_core [24898.200544] ? hrtimer_interrupt+0x104/0x240 [24898.203852] wmi ipmi_si [24898.208377] ? __local_bh_enable+0x50/0x60 [24898.211513] ipmi_devintf ipmi_msghandler [24898.215779] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24898.218309] ioatdma acpi_pad [24898.222400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24898.226404] acpi_power_meter [24898.231620] [24898.234584] ixgbe igb [24898.239542] [24898.242505] mdio crc32c_intel [24898.244604] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24898.246962] nvme ptp [24898.249064] ? queued_spin_lock_slowpath+0x18/0x2e0 [24898.252110] nvme_core [24898.257415] do_raw_spin_lock+0xb5/0xc0 [24898.259683] i2c_algo_bit pps_core [24898.264555] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24898.266912] dca fuse [24898.270744] try_get_cap_refs+0x159/0x880 [ceph] [24898.274140] [24898.274141] irq event stamp: 121292012 [24898.278930] __ceph_get_caps+0x4a1/0x760 [ceph] [24898.281194] hardirqs last enabled at (121292011): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24898.285811] ? __file_remove_privs+0x89/0x110 [24898.287296] hardirqs last disabled at (121292012): [] sysvec_apic_timer_interrupt+0xe/0x90 [24898.291040] ? wake_bit_function+0x70/0x70 [24898.295565] softirqs last enabled at (121292010): [] __do_softirq+0x2be/0x371 [24898.305811] ? ceph_write_iter+0x260/0x770 [ceph] [24898.310159] softirqs last disabled at (121292005): [] irq_exit_rcu+0xb0/0xd0 [24898.319972] ceph_write_iter+0x260/0x770 [ceph] [24898.324062] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24898.332835] ? lock_is_held_type+0xaa/0x120 [24898.337528] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24898.346129] vfs_write+0x324/0x4f0 [24898.350661] Workqueue: events delayed_work [ceph] [24898.360043] ksys_pwrite64+0x65/0xa0 [24898.364216] [24898.364216] RIP: 0010:lock_release+0x1f/0x260 [24898.371686] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24898.375082] Code: a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 <48> 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 [24898.379782] do_syscall_64+0x38/0x80 [24898.383364] RSP: 0018:ffffc90002117d48 EFLAGS: 00000296 [24898.384859] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24898.389210] [24898.389211] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 000000005553a634 [24898.394342] RIP: 0033:0x7f48f0eeb93b [24898.413094] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [24898.416665] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24898.421883] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [24898.426927] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24898.428419] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [24898.435542] ORIG_RAX: 0000000000000012 [24898.439114] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [24898.446236] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24898.464975] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24898.472098] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24898.477315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24898.484440] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24898.488270] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24898.495396] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24898.502520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24898.510598] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24898.517720] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24898.523465] [24898.530582] Call Trace: [24898.530584] [24898.572843] ? watchdog_timer_fn+0x1b4/0x220 [24898.577107] ? __hrtimer_run_queues+0x1d3/0x360 [24898.581650] ? hrtimer_interrupt+0x104/0x240 [24898.585911] ? __local_bh_enable+0x50/0x60 [24898.590006] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24898.595230] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24898.600187] [24898.602284] [24898.604381] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24898.609685] ? igrab+0x43/0x50 [24898.612738] ? igrab+0x43/0x50 [24898.615786] ? lock_release+0x1f/0x260 [24898.619533] ? lock_release+0xc5/0x260 [24898.623277] _raw_spin_unlock+0x1b/0x40 [24898.627116] igrab+0x43/0x50 [24898.629992] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24898.635142] delayed_work+0x1d8/0x280 [ceph] [24898.639422] ? process_scheduled_works+0x22a/0x510 [24898.644206] process_scheduled_works+0x2a4/0x510 [24898.648821] worker_thread+0x188/0x340 [24898.652569] ? rescuer_thread+0x310/0x310 [24898.656572] kthread+0xf5/0x130 [24898.659709] ? kthread_complete_and_exit+0x20/0x20 [24898.664495] ret_from_fork+0x31/0x40 [24898.668074] ? kthread_complete_and_exit+0x20/0x20 [24898.672859] ret_from_fork_asm+0x11/0x20 [24898.676789] [24900.683379] rcu: INFO: rcu_preempt self-detected stall on CPU [24900.689135] rcu: 7-....: (12590127 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/12588842 fqs=6331780 [24900.699735] rcu: (t=12963842 jiffies g=1309609 q=406474 ncpus=8) [24900.705820] Sending NMI from CPU 7 to CPUs 0: [24900.710171] NMI backtrace for cpu 0 [24900.710173] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24900.710175] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24900.710176] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24900.710180] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24900.710182] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24900.710184] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24900.710185] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24900.710186] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24900.710188] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24900.710189] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24900.710190] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24900.710192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24900.710193] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24900.710195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24900.710196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24900.710197] Call Trace: [24900.710198] [24900.710199] ? nmi_cpu_backtrace+0xad/0x120 [24900.710204] ? nmi_cpu_backtrace_handler+0x11/0x20 [24900.710206] ? nmi_handle+0xd4/0x260 [24900.710212] ? default_do_nmi+0x49/0xf0 [24900.710214] ? exc_nmi+0x11e/0x150 [24900.710217] ? end_repeat_nmi+0x16/0x67 [24900.710223] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24900.710227] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24900.710231] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24900.710234] [24900.710235] [24900.710236] do_raw_spin_lock+0xb5/0xc0 [24900.710239] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24900.710259] try_get_cap_refs+0x159/0x880 [ceph] [24900.710280] __ceph_get_caps+0x4a1/0x760 [ceph] [24900.710300] ? __file_remove_privs+0x89/0x110 [24900.710302] ? wake_bit_function+0x70/0x70 [24900.710307] ? ceph_write_iter+0x260/0x770 [ceph] [24900.710322] ceph_write_iter+0x260/0x770 [ceph] [24900.710340] ? lock_is_held_type+0xaa/0x120 [24900.710344] vfs_write+0x324/0x4f0 [24900.710351] ksys_pwrite64+0x65/0xa0 [24900.710354] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24900.710358] do_syscall_64+0x38/0x80 [24900.710361] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24900.710363] RIP: 0033:0x7f48f0eeb93b [24900.710365] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24900.710366] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [24900.710368] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24900.710370] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24900.710371] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24900.710372] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24900.710373] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24900.710380] [24900.711169] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24901.014789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24901.022261] Workqueue: events delayed_work [ceph] [24901.026975] RIP: 0010:lock_acquire+0xf0/0x2b0 [24901.031325] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24901.050065] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24901.055281] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24901.062412] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24901.069538] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24901.076664] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24901.083787] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24901.090910] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24901.098989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24901.104724] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24901.111849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24901.118973] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24901.126097] Call Trace: [24901.128543] [24901.130554] ? rcu_dump_cpu_stacks+0xd1/0x180 [24901.134914] ? rcu_sched_clock_irq+0x633/0x1390 [24901.139438] ? lock_release+0xc5/0x260 [24901.143181] ? lock_is_held_type+0xaa/0x120 [24901.147359] ? lock_is_held_type+0xaa/0x120 [24901.151538] ? tick_sched_do_timer+0xa0/0xa0 [24901.155808] ? rcu_read_lock_sched_held+0x56/0x90 [24901.160508] ? tick_sched_do_timer+0xa0/0xa0 [24901.164769] ? update_process_times+0x5f/0x90 [24901.169120] ? tick_sched_handle.isra.16+0x1e/0x50 [24901.173905] ? tick_sched_timer+0x6f/0x80 [24901.177911] ? __hrtimer_run_queues+0x1d3/0x360 [24901.182436] ? hrtimer_interrupt+0x104/0x240 [24901.186704] ? __local_bh_enable+0x50/0x60 [24901.190798] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24901.196024] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24901.200980] [24901.203076] [24901.205175] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24901.210483] ? lock_acquire+0xf0/0x2b0 [24901.214233] ? igrab+0x43/0x50 [24901.217284] ? lock_release+0xc5/0x260 [24901.221029] _raw_spin_lock+0x30/0x40 [24901.224695] ? igrab+0x1d/0x50 [24901.227743] igrab+0x1d/0x50 [24901.230621] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24901.235771] delayed_work+0x1d8/0x280 [ceph] [24901.240051] ? process_scheduled_works+0x22a/0x510 [24901.244833] process_scheduled_works+0x2a4/0x510 [24901.249450] worker_thread+0x188/0x340 [24901.253197] ? rescuer_thread+0x310/0x310 [24901.257202] kthread+0xf5/0x130 [24901.260340] ? kthread_complete_and_exit+0x20/0x20 [24901.265123] ret_from_fork+0x31/0x40 [24901.268708] ? kthread_complete_and_exit+0x20/0x20 [24901.273495] ret_from_fork_asm+0x11/0x20 [24901.277422] [24925.894442] watchdog: BUG: soft lockup - CPU#0 stuck for 12100s! [dbench:78683] [24925.901747] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24925.906442] watchdog: BUG: soft lockup - CPU#7 stuck for 21097s! [kworker/7:4:78808] [24925.957189] irq event stamp: 39848464 [24925.957190] hardirqs last enabled at (39848463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24925.964918] Modules linked in: [24925.968577] hardirqs last disabled at (39848464): [] sysvec_apic_timer_interrupt+0xe/0x90 [24925.978734] ceph [24925.981783] softirqs last enabled at (39848406): [] __do_softirq+0x2be/0x371 [24925.991508] libceph dns_resolver [24925.993433] softirqs last disabled at (39848401): [] irq_exit_rcu+0xb0/0xd0 [24926.002130] fscache netfs [24926.005444] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24926.013954] veth nft_chain_nat [24926.016660] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24926.025604] xt_MASQUERADE nf_nat [24926.028740] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24926.036213] nf_conntrack nf_defrag_ipv6 [24926.039523] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24926.044999] nf_defrag_ipv4 nft_compat [24926.048918] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24926.067655] nf_tables nfnetlink [24926.071399] [24926.071400] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24926.076617] bridge stp [24926.079840] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24926.081332] llc [24926.088454] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24926.090899] xfs libcrc32c [24926.098022] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24926.099860] sunrpc intel_rapl_msr [24926.106984] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24926.109688] intel_rapl_common sb_edac [24926.116814] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24926.120228] x86_pkg_temp_thermal [24926.127351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24926.131095] coretemp kvm_intel [24926.139174] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24926.142483] kvm irqbypass [24926.148221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24926.151358] i2c_i801 iTCO_wdt [24926.158484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24926.161188] iTCO_vendor_support i2c_smbus [24926.168311] Call Trace: [24926.168313] [24926.171362] crct10dif_pclmul joydev [24926.178487] ? watchdog_timer_fn+0x1b4/0x220 [24926.182576] crc32_pclmul ipmi_ssif [24926.185022] ? __hrtimer_run_queues+0x1d3/0x360 [24926.187031] ghash_clmulni_intel mei_me [24926.190605] ? hrtimer_interrupt+0x104/0x240 [24926.194866] mei lpc_ich [24926.198352] ? __local_bh_enable+0x50/0x60 [24926.202874] sha512_ssse3 [24926.206709] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24926.210970] acpi_ipmi mfd_core [24926.213500] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24926.217591] wmi ipmi_si [24926.220209] [24926.225425] ipmi_devintf ipmi_msghandler [24926.228563] [24926.233521] ioatdma acpi_pad [24926.236051] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24926.238150] acpi_power_meter ixgbe [24926.242156] ? queued_spin_lock_slowpath+0x16/0x2e0 [24926.244251] igb mdio [24926.247217] do_raw_spin_lock+0xb5/0xc0 [24926.252518] crc32c_intel nvme [24926.256003] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24926.260874] ptp nvme_core [24926.263146] try_get_cap_refs+0x159/0x880 [ceph] [24926.266973] i2c_algo_bit pps_core [24926.270030] __ceph_get_caps+0x4a1/0x760 [ceph] [24926.274808] dca fuse [24926.277518] ? __file_remove_privs+0x89/0x110 [24926.282123] [24926.282124] irq event stamp: 121507996 [24926.285522] ? wake_bit_function+0x70/0x70 [24926.290046] hardirqs last enabled at (121507995): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24926.292319] ? ceph_write_iter+0x260/0x770 [ceph] [24926.296668] hardirqs last disabled at (121507996): [] sysvec_apic_timer_interrupt+0xe/0x90 [24926.298157] ceph_write_iter+0x260/0x770 [ceph] [24926.301903] softirqs last enabled at (121507994): [] __do_softirq+0x2be/0x371 [24926.305998] ? lock_is_held_type+0xaa/0x120 [24926.316238] softirqs last disabled at (121507989): [] irq_exit_rcu+0xb0/0xd0 [24926.320936] vfs_write+0x324/0x4f0 [24926.330745] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24926.335273] ksys_pwrite64+0x65/0xa0 [24926.344041] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24926.348218] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24926.356814] Workqueue: events delayed_work [ceph] [24926.360213] do_syscall_64+0x38/0x80 [24926.369590] [24926.373163] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24926.380633] RIP: 0010:lock_acquire+0xf0/0x2b0 [24926.385761] RIP: 0033:0x7f48f0eeb93b [24926.390459] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24926.394030] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24926.395520] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24926.400566] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24926.404916] [24926.404917] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24926.408487] ORIG_RAX: 0000000000000012 [24926.427225] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24926.445963] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24926.451179] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24926.456397] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24926.457886] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24926.465011] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24926.468845] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24926.475966] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24926.483091] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24926.490216] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24926.497338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24926.504468] [24926.511588] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24926.556091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24926.563216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24926.570340] Call Trace: [24926.572784] [24926.574796] ? watchdog_timer_fn+0x1b4/0x220 [24926.579059] ? __hrtimer_run_queues+0x1d3/0x360 [24926.583587] ? hrtimer_interrupt+0x104/0x240 [24926.587854] ? __local_bh_enable+0x50/0x60 [24926.591948] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24926.597165] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24926.602135] [24926.604236] [24926.606333] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24926.611642] ? lock_acquire+0xf0/0x2b0 [24926.615393] ? igrab+0x43/0x50 [24926.618440] ? lock_release+0xc5/0x260 [24926.622186] _raw_spin_lock+0x30/0x40 [24926.625849] ? igrab+0x1d/0x50 [24926.628902] igrab+0x1d/0x50 [24926.631779] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24926.636930] delayed_work+0x1d8/0x280 [ceph] [24926.641211] ? process_scheduled_works+0x22a/0x510 [24926.646001] process_scheduled_works+0x2a4/0x510 [24926.650617] worker_thread+0x188/0x340 [24926.654366] ? rescuer_thread+0x310/0x310 [24926.658369] kthread+0xf5/0x130 [24926.661507] ? kthread_complete_and_exit+0x20/0x20 [24926.666292] ret_from_fork+0x31/0x40 [24926.669861] ? kthread_complete_and_exit+0x20/0x20 [24926.674647] ret_from_fork_asm+0x11/0x20 [24926.678569] [24953.894919] watchdog: BUG: soft lockup - CPU#0 stuck for 12126s! [dbench:78683] [24953.902225] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24953.906918] watchdog: BUG: soft lockup - CPU#7 stuck for 21123s! [kworker/7:4:78808] [24953.957677] irq event stamp: 39906690 [24953.957678] hardirqs last enabled at (39906689): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24953.965406] Modules linked in: [24953.969063] hardirqs last disabled at (39906690): [] sysvec_apic_timer_interrupt+0xe/0x90 [24953.979221] ceph [24953.982272] softirqs last enabled at (39906632): [] __do_softirq+0x2be/0x371 [24953.991995] libceph dns_resolver [24953.993920] softirqs last disabled at (39906627): [] irq_exit_rcu+0xb0/0xd0 [24954.002603] fscache netfs [24954.005916] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24954.014424] veth nft_chain_nat [24954.017129] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24954.026074] xt_MASQUERADE [24954.029212] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [24954.036682] nf_nat [24954.039386] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [24954.044864] nf_conntrack [24954.046961] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24954.065698] nf_defrag_ipv6 nf_defrag_ipv4 [24954.068316] [24954.068317] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24954.073534] nft_compat nf_tables nfnetlink [24954.077623] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24954.079114] bridge stp [24954.086240] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24954.090416] llc xfs [24954.097542] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24954.099985] libcrc32c sunrpc [24954.107108] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24954.109292] intel_rapl_msr intel_rapl_common [24954.116434] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24954.119399] sb_edac x86_pkg_temp_thermal [24954.126523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24954.130874] coretemp kvm_intel [24954.138950] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24954.142953] kvm irqbypass [24954.148693] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24954.151830] i2c_i801 iTCO_wdt [24954.158954] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24954.161658] iTCO_vendor_support i2c_smbus [24954.168783] Call Trace: [24954.171832] crct10dif_pclmul joydev [24954.178959] [24954.183047] crc32_pclmul ipmi_ssif [24954.185494] ? watchdog_timer_fn+0x1b4/0x220 [24954.189063] ghash_clmulni_intel mei_me [24954.191075] ? __hrtimer_run_queues+0x1d3/0x360 [24954.194557] mei lpc_ich [24954.198824] ? hrtimer_interrupt+0x104/0x240 [24954.202651] sha512_ssse3 acpi_ipmi [24954.207176] ? __local_bh_enable+0x50/0x60 [24954.209707] mfd_core wmi [24954.213974] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24954.217454] ipmi_si ipmi_devintf [24954.221547] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24954.224162] ipmi_msghandler [24954.229380] [24954.232690] ioatdma acpi_pad [24954.237647] [24954.237649] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24954.240526] acpi_power_meter [24954.242626] ? queued_spin_lock_slowpath+0x2d/0x2e0 [24954.245586] ixgbe igb mdio [24954.247704] do_raw_spin_lock+0xb5/0xc0 [24954.253005] crc32c_intel [24954.255972] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24954.260841] nvme ptp [24954.263635] try_get_cap_refs+0x159/0x880 [ceph] [24954.267462] nvme_core i2c_algo_bit [24954.270084] __ceph_get_caps+0x4a1/0x760 [ceph] [24954.274864] pps_core dca fuse [24954.277139] ? __file_remove_privs+0x89/0x110 [24954.281745] [24954.281746] irq event stamp: 121728738 [24954.285230] ? wake_bit_function+0x70/0x70 [24954.289754] hardirqs last enabled at (121728737): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24954.292807] ? ceph_write_iter+0x260/0x770 [ceph] [24954.297155] hardirqs last disabled at (121728738): [] sysvec_apic_timer_interrupt+0xe/0x90 [24954.298646] ceph_write_iter+0x260/0x770 [ceph] [24954.302389] softirqs last enabled at (121728736): [] __do_softirq+0x2be/0x371 [24954.306485] ? lock_is_held_type+0xaa/0x120 [24954.316725] softirqs last disabled at (121728731): [] irq_exit_rcu+0xb0/0xd0 [24954.321424] vfs_write+0x324/0x4f0 [24954.331232] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24954.335762] ksys_pwrite64+0x65/0xa0 [24954.344526] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24954.348706] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24954.357319] Workqueue: events delayed_work [ceph] [24954.360718] do_syscall_64+0x38/0x80 [24954.370095] [24954.370096] RIP: 0010:lock_acquire+0x32/0x2b0 [24954.373666] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24954.381136] Code: ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 <31> c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd [24954.386267] RIP: 0033:0x7f48f0eeb93b [24954.390965] RSP: 0018:ffffc90002117d18 EFLAGS: 00000296 [24954.394536] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24954.396026] [24954.400378] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24954.405422] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 0000000000000000 [24954.424157] ORIG_RAX: 0000000000000012 [24954.427730] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [24954.432946] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24954.451683] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [24954.453174] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24954.458392] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [24954.465515] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24954.469348] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24954.476472] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24954.483603] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24954.490728] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24954.497852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24954.504983] [24954.512100] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24954.556598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24954.563737] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24954.570862] Call Trace: [24954.573322] [24954.575334] ? watchdog_timer_fn+0x1b4/0x220 [24954.579604] ? __hrtimer_run_queues+0x1d3/0x360 [24954.584134] ? hrtimer_interrupt+0x104/0x240 [24954.588395] ? __local_bh_enable+0x50/0x60 [24954.592487] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24954.597703] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24954.602661] [24954.604758] [24954.606857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24954.612165] ? lock_acquire+0x32/0x2b0 [24954.615913] ? igrab+0x43/0x50 [24954.618964] ? lock_release+0xc5/0x260 [24954.622709] _raw_spin_lock+0x30/0x40 [24954.626374] ? igrab+0x1d/0x50 [24954.629424] igrab+0x1d/0x50 [24954.632320] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24954.637471] delayed_work+0x1d8/0x280 [ceph] [24954.641750] ? process_scheduled_works+0x22a/0x510 [24954.646541] process_scheduled_works+0x2a4/0x510 [24954.651157] worker_thread+0x188/0x340 [24954.654906] ? rescuer_thread+0x310/0x310 [24954.658910] kthread+0xf5/0x130 [24954.662056] ? kthread_complete_and_exit+0x20/0x20 [24954.666839] ret_from_fork+0x31/0x40 [24954.670411] ? kthread_complete_and_exit+0x20/0x20 [24954.675195] ret_from_fork_asm+0x11/0x20 [24954.679126] [24981.895396] watchdog: BUG: soft lockup - CPU#0 stuck for 12152s! [dbench:78683] [24981.902695] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [24981.907396] watchdog: BUG: soft lockup - CPU#7 stuck for 21149s! [kworker/7:4:78808] [24981.958137] irq event stamp: 39964910 [24981.958139] hardirqs last enabled at (39964909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24981.965867] Modules linked in: [24981.969524] hardirqs last disabled at (39964910): [] sysvec_apic_timer_interrupt+0xe/0x90 [24981.979684] ceph [24981.982733] softirqs last enabled at (39964852): [] __do_softirq+0x2be/0x371 [24981.992456] libceph [24981.994383] softirqs last disabled at (39964847): [] irq_exit_rcu+0xb0/0xd0 [24982.003070] dns_resolver [24982.005259] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24982.013770] fscache [24982.016405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24982.025347] netfs veth [24982.027533] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [24982.035004] nft_chain_nat [24982.037447] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [24982.042924] xt_MASQUERADE nf_nat [24982.045629] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [24982.064366] nf_conntrack nf_defrag_ipv6 [24982.067677] [24982.067677] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [24982.072893] nf_defrag_ipv4 nft_compat [24982.076811] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [24982.078302] nf_tables nfnetlink [24982.085425] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [24982.089171] bridge stp [24982.096295] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [24982.099520] llc xfs [24982.106643] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [24982.109087] libcrc32c sunrpc [24982.116210] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [24982.118394] intel_rapl_msr intel_rapl_common [24982.125518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24982.128483] sb_edac [24982.136560] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [24982.140912] x86_pkg_temp_thermal coretemp [24982.146649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24982.148833] kvm_intel [24982.155958] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24982.160048] kvm irqbypass [24982.167172] Call Trace: [24982.167173] [24982.169529] i2c_i801 iTCO_wdt [24982.176655] ? watchdog_timer_fn+0x1b4/0x220 [24982.179358] iTCO_vendor_support i2c_smbus [24982.181803] ? __hrtimer_run_queues+0x1d3/0x360 [24982.183811] crct10dif_pclmul joydev [24982.186867] ? hrtimer_interrupt+0x104/0x240 [24982.191127] crc32_pclmul ipmi_ssif [24982.195218] ? __local_bh_enable+0x50/0x60 [24982.199743] ghash_clmulni_intel mei_me [24982.203316] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24982.207577] mei lpc_ich [24982.211076] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24982.215168] sha512_ssse3 acpi_ipmi [24982.219001] [24982.224217] mfd_core wmi [24982.226747] [24982.226749] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24982.231705] ipmi_si ipmi_devintf [24982.235192] ? queued_spin_lock_slowpath+0x16/0x2e0 [24982.237288] ipmi_msghandler ioatdma [24982.239907] do_raw_spin_lock+0xb5/0xc0 [24982.242002] acpi_pad acpi_power_meter [24982.247308] __cap_delay_requeue+0xa7/0x1d0 [ceph] [24982.250617] ixgbe igb [24982.255488] try_get_cap_refs+0x159/0x880 [ceph] [24982.259059] mdio crc32c_intel [24982.262910] __ceph_get_caps+0x4a1/0x760 [ceph] [24982.266649] nvme [24982.271437] ? __file_remove_privs+0x89/0x110 [24982.273795] ptp [24982.278402] ? wake_bit_function+0x70/0x70 [24982.281452] nvme_core i2c_algo_bit [24982.285980] ? ceph_write_iter+0x260/0x770 [ceph] [24982.287900] pps_core dca [24982.292252] ceph_write_iter+0x260/0x770 [ceph] [24982.294088] fuse [24982.298182] ? lock_is_held_type+0xaa/0x120 [24982.301663] irq event stamp: 121949448 [24982.306364] vfs_write+0x324/0x4f0 [24982.308977] hardirqs last enabled at (121949447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24982.313508] ksys_pwrite64+0x65/0xa0 [24982.315426] hardirqs last disabled at (121949448): [] sysvec_apic_timer_interrupt+0xe/0x90 [24982.319605] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [24982.323347] softirqs last enabled at (121949446): [] __do_softirq+0x2be/0x371 [24982.326746] do_syscall_64+0x38/0x80 [24982.336990] softirqs last disabled at (121949441): [] irq_exit_rcu+0xb0/0xd0 [24982.340562] entry_SYSCALL_64_after_hwframe+0x63/0xcd [24982.350371] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [24982.355501] RIP: 0033:0x7f48f0eeb93b [24982.364273] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [24982.367845] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [24982.376442] Workqueue: events delayed_work [ceph] [24982.381486] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [24982.390862] [24982.394434] ORIG_RAX: 0000000000000012 [24982.401904] RIP: 0010:lock_acquire+0xf0/0x2b0 [24982.420643] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [24982.425339] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [24982.430556] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [24982.432048] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [24982.435878] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [24982.440228] [24982.440229] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [24982.447352] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [24982.466091] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [24982.473215] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [24982.478433] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [24982.485563] [24982.487048] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [24982.531975] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [24982.539100] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [24982.547178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24982.552915] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [24982.560040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [24982.567163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [24982.574288] Call Trace: [24982.576733] [24982.578744] ? watchdog_timer_fn+0x1b4/0x220 [24982.583009] ? __hrtimer_run_queues+0x1d3/0x360 [24982.587542] ? hrtimer_interrupt+0x104/0x240 [24982.591803] ? __local_bh_enable+0x50/0x60 [24982.595896] ? __sysvec_apic_timer_interrupt+0x61/0x160 [24982.601111] ? sysvec_apic_timer_interrupt+0x6d/0x90 [24982.606081] [24982.608175] [24982.610273] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [24982.615581] ? lock_acquire+0xf0/0x2b0 [24982.619332] ? igrab+0x43/0x50 [24982.622382] ? lock_release+0xc5/0x260 [24982.626125] _raw_spin_lock+0x30/0x40 [24982.629799] ? igrab+0x1d/0x50 [24982.632849] igrab+0x1d/0x50 [24982.635729] ceph_check_delayed_caps+0xb7/0x270 [ceph] [24982.640877] delayed_work+0x1d8/0x280 [ceph] [24982.645158] ? process_scheduled_works+0x22a/0x510 [24982.649941] process_scheduled_works+0x2a4/0x510 [24982.654556] worker_thread+0x188/0x340 [24982.658306] ? rescuer_thread+0x310/0x310 [24982.662317] kthread+0xf5/0x130 [24982.665453] ? kthread_complete_and_exit+0x20/0x20 [24982.670240] ret_from_fork+0x31/0x40 [24982.673819] ? kthread_complete_and_exit+0x20/0x20 [24982.678603] ret_from_fork_asm+0x11/0x20 [24982.682534] [25009.895875] watchdog: BUG: soft lockup - CPU#0 stuck for 12178s! [dbench:78683] [25009.903175] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25009.907874] watchdog: BUG: soft lockup - CPU#7 stuck for 21175s! [kworker/7:4:78808] [25009.958623] irq event stamp: 40023204 [25009.966355] Modules linked in: ceph libceph dns_resolver fscache netfs [25009.970013] hardirqs last enabled at (40023203): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25009.970018] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [25009.986704] hardirqs last disabled at (40023204): [] sysvec_apic_timer_interrupt+0xe/0x90 [25009.994090] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [25010.003812] softirqs last enabled at (40023146): [] __do_softirq+0x2be/0x371 [25010.010245] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25010.018946] softirqs last disabled at (40023141): [] irq_exit_rcu+0xb0/0xd0 [25010.025462] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [25010.033973] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25010.039625] kvm irqbypass [25010.048570] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25010.048572] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25010.051273] i2c_i801 iTCO_wdt [25010.058745] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25010.064221] iTCO_vendor_support [25010.067272] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25010.086010] i2c_smbus crct10dif_pclmul [25010.089233] [25010.089234] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25010.094450] joydev crc32_pclmul [25010.098281] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25010.099773] ipmi_ssif [25010.106896] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25010.110121] ghash_clmulni_intel [25010.117245] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25010.119620] mei_me mei [25010.126743] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25010.129969] lpc_ich sha512_ssse3 [25010.137091] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25010.139549] acpi_ipmi mfd_core [25010.146678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25010.149988] wmi ipmi_si [25010.158065] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25010.161202] ipmi_devintf [25010.166939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25010.169472] ipmi_msghandler ioatdma [25010.176594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25010.179214] acpi_pad acpi_power_meter [25010.186338] Call Trace: [25010.186340] [25010.189908] ixgbe igb [25010.197035] ? watchdog_timer_fn+0x1b4/0x220 [25010.200776] mdio crc32c_intel [25010.203221] ? __hrtimer_run_queues+0x1d3/0x360 [25010.205232] nvme [25010.207591] ? hrtimer_interrupt+0x104/0x240 [25010.211851] ptp nvme_core [25010.214902] ? __local_bh_enable+0x50/0x60 [25010.219427] i2c_algo_bit [25010.221354] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25010.225614] pps_core [25010.228318] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25010.232409] dca fuse [25010.235028] [25010.240245] [25010.240247] irq event stamp: 122170266 [25010.242533] [25010.247489] hardirqs last enabled at (122170265): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25010.249762] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25010.251857] hardirqs last disabled at (122170266): [] sysvec_apic_timer_interrupt+0xe/0x90 [25010.253353] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25010.257093] softirqs last enabled at (122170264): [] __do_softirq+0x2be/0x371 [25010.259193] do_raw_spin_lock+0xb5/0xc0 [25010.269435] softirqs last disabled at (122170259): [] irq_exit_rcu+0xb0/0xd0 [25010.274739] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25010.284554] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25010.289432] try_get_cap_refs+0x159/0x880 [ceph] [25010.298200] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25010.302035] __ceph_get_caps+0x4a1/0x760 [ceph] [25010.310627] Workqueue: events delayed_work [ceph] [25010.315418] ? __file_remove_privs+0x89/0x110 [25010.324790] [25010.329400] ? wake_bit_function+0x70/0x70 [25010.336872] RIP: 0010:lock_release+0x3c/0x260 [25010.341397] ? ceph_write_iter+0x260/0x770 [ceph] [25010.346092] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [25010.350443] ceph_write_iter+0x260/0x770 [ceph] [25010.351933] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [25010.356029] ? lock_is_held_type+0xaa/0x120 [25010.360376] [25010.360376] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [25010.365075] vfs_write+0x324/0x4f0 [25010.383810] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [25010.388340] ksys_pwrite64+0x65/0xa0 [25010.393555] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [25010.397732] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25010.399219] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [25010.406345] do_syscall_64+0x38/0x80 [25010.409741] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25010.416867] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25010.420435] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25010.427573] RIP: 0033:0x7f48f0eeb93b [25010.432700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25010.439826] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25010.443395] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25010.450518] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25010.455563] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25010.463641] ORIG_RAX: 0000000000000012 [25010.467212] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25010.472948] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25010.491685] Call Trace: [25010.498810] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25010.504028] [25010.511152] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25010.514983] ? watchdog_timer_fn+0x1b4/0x220 [25010.522108] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25010.529232] ? __hrtimer_run_queues+0x1d3/0x360 [25010.531674] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25010.538804] ? hrtimer_interrupt+0x104/0x240 [25010.540816] [25010.547933] ? __local_bh_enable+0x50/0x60 [25010.581512] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25010.586737] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25010.591694] [25010.593790] [25010.595888] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25010.601192] ? igrab+0x43/0x50 [25010.604245] ? igrab+0x43/0x50 [25010.607294] ? lock_release+0x3c/0x260 [25010.611040] ? lock_release+0xc5/0x260 [25010.614783] _raw_spin_unlock+0x1b/0x40 [25010.618615] igrab+0x43/0x50 [25010.621492] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25010.626641] delayed_work+0x1d8/0x280 [ceph] [25010.630921] ? process_scheduled_works+0x22a/0x510 [25010.635713] process_scheduled_works+0x2a4/0x510 [25010.640326] worker_thread+0x188/0x340 [25010.644076] ? rescuer_thread+0x310/0x310 [25010.648080] kthread+0xf5/0x130 [25010.651217] ? kthread_complete_and_exit+0x20/0x20 [25010.656003] ret_from_fork+0x31/0x40 [25010.659581] ? kthread_complete_and_exit+0x20/0x20 [25010.664364] ret_from_fork_asm+0x11/0x20 [25010.668297] [25037.896353] watchdog: BUG: soft lockup - CPU#0 stuck for 12204s! [dbench:78683] [25037.903654] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25037.908352] watchdog: BUG: soft lockup - CPU#7 stuck for 21201s! [kworker/7:4:78808] [25037.959096] irq event stamp: 40081322 [25037.966827] Modules linked in: ceph libceph [25037.970483] hardirqs last enabled at (40081321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25037.974660] dns_resolver fscache netfs veth [25037.984818] hardirqs last disabled at (40081322): [] sysvec_apic_timer_interrupt+0xe/0x90 [25037.989082] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [25037.998807] softirqs last enabled at (40081312): [] __do_softirq+0x2be/0x371 [25038.004458] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [25038.013142] softirqs last disabled at (40081307): [] irq_exit_rcu+0xb0/0xd0 [25038.019051] nfnetlink bridge stp [25038.027564] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25038.030873] llc xfs [25038.039818] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25038.039819] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25038.042002] libcrc32c [25038.049472] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25038.054951] sunrpc intel_rapl_msr [25038.057307] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25038.076044] intel_rapl_common sb_edac [25038.079443] [25038.079444] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25038.084660] x86_pkg_temp_thermal coretemp [25038.088405] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25038.089894] kvm_intel kvm [25038.097034] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25038.101127] irqbypass [25038.108251] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25038.110955] i2c_i801 iTCO_wdt [25038.118080] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25038.120435] iTCO_vendor_support i2c_smbus [25038.127561] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25038.130611] crct10dif_pclmul joydev [25038.137751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25038.141842] crc32_pclmul [25038.149922] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25038.153493] ipmi_ssif ghash_clmulni_intel [25038.159229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25038.161846] mei_me mei [25038.168970] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25038.173062] lpc_ich [25038.180185] Call Trace: [25038.180186] [25038.182628] sha512_ssse3 acpi_ipmi [25038.189754] ? watchdog_timer_fn+0x1b4/0x220 [25038.191939] mfd_core wmi [25038.194383] ? __hrtimer_run_queues+0x1d3/0x360 [25038.196394] ipmi_si ipmi_devintf [25038.199880] ? hrtimer_interrupt+0x104/0x240 [25038.204142] ipmi_msghandler ioatdma [25038.206758] ? __local_bh_enable+0x50/0x60 [25038.211282] acpi_pad acpi_power_meter [25038.214596] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25038.218857] ixgbe igb [25038.222428] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25038.226518] mdio crc32c_intel [25038.230262] [25038.235481] nvme ptp [25038.237836] [25038.237838] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25038.242794] nvme_core i2c_algo_bit [25038.245850] ? queued_spin_lock_slowpath+0x18/0x2e0 [25038.247942] pps_core dca [25038.250217] do_raw_spin_lock+0xb5/0xc0 [25038.252311] fuse [25038.257617] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25038.261099] irq event stamp: 122391188 [25038.265972] try_get_cap_refs+0x159/0x880 [ceph] [25038.268588] hardirqs last enabled at (122391187): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25038.272422] __ceph_get_caps+0x4a1/0x760 [ceph] [25038.274343] hardirqs last disabled at (122391188): [] sysvec_apic_timer_interrupt+0xe/0x90 [25038.279130] ? __file_remove_privs+0x89/0x110 [25038.282871] softirqs last enabled at (122391186): [] __do_softirq+0x2be/0x371 [25038.287481] ? wake_bit_function+0x70/0x70 [25038.297725] softirqs last disabled at (122391181): [] irq_exit_rcu+0xb0/0xd0 [25038.302252] ? ceph_write_iter+0x260/0x770 [ceph] [25038.312059] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25038.316412] ceph_write_iter+0x260/0x770 [ceph] [25038.325181] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25038.329276] ? lock_is_held_type+0xaa/0x120 [25038.337871] Workqueue: events delayed_work [ceph] [25038.342570] vfs_write+0x324/0x4f0 [25038.351945] [25038.351945] RIP: 0010:lock_acquire+0xf0/0x2b0 [25038.356472] ksys_pwrite64+0x65/0xa0 [25038.363940] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25038.368117] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25038.372815] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25038.376215] do_syscall_64+0x38/0x80 [25038.377703] [25038.377703] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25038.382053] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25038.385625] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25038.404361] RIP: 0033:0x7f48f0eeb93b [25038.409492] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25038.414711] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25038.418282] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25038.419771] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25038.426895] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25038.431939] ORIG_RAX: 0000000000000012 [25038.439063] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25038.442634] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25038.449757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25038.468497] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25038.475620] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25038.480837] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25038.487962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25038.491792] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25038.499870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25038.506994] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25038.512731] Call Trace: [25038.512733] [25038.519880] [25038.526998] ? watchdog_timer_fn+0x1b4/0x220 [25038.573525] ? __hrtimer_run_queues+0x1d3/0x360 [25038.578057] ? hrtimer_interrupt+0x104/0x240 [25038.582317] ? __local_bh_enable+0x50/0x60 [25038.586412] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25038.591636] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25038.596591] [25038.598690] [25038.600786] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25038.606096] ? lock_acquire+0xf0/0x2b0 [25038.609846] ? igrab+0x43/0x50 [25038.612895] ? lock_release+0xc5/0x260 [25038.616640] _raw_spin_lock+0x30/0x40 [25038.620297] ? igrab+0x1d/0x50 [25038.623346] igrab+0x1d/0x50 [25038.626226] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25038.631376] delayed_work+0x1d8/0x280 [ceph] [25038.635655] ? process_scheduled_works+0x22a/0x510 [25038.640439] process_scheduled_works+0x2a4/0x510 [25038.645053] worker_thread+0x188/0x340 [25038.648802] ? rescuer_thread+0x310/0x310 [25038.652806] kthread+0xf5/0x130 [25038.655944] ? kthread_complete_and_exit+0x20/0x20 [25038.660728] ret_from_fork+0x31/0x40 [25038.664323] ? kthread_complete_and_exit+0x20/0x20 [25038.669109] ret_from_fork_asm+0x11/0x20 [25038.673031] [25065.896831] watchdog: BUG: soft lockup - CPU#0 stuck for 12230s! [dbench:78683] [25065.904135] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25065.908830] watchdog: BUG: soft lockup - CPU#7 stuck for 21227s! [kworker/7:4:78808] [25065.959583] irq event stamp: 40139488 [25065.967315] Modules linked in: ceph libceph dns_resolver fscache netfs veth [25065.970971] hardirqs last enabled at (40139487): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25065.977930] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [25065.988087] hardirqs last disabled at (40139488): [] sysvec_apic_timer_interrupt+0xe/0x90 [25065.988090] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [25066.006063] softirqs last enabled at (40139452): [] __do_softirq+0x2be/0x371 [25066.012754] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [25066.021439] softirqs last disabled at (40139447): [] irq_exit_rcu+0xb0/0xd0 [25066.030565] kvm irqbypass i2c_i801 iTCO_wdt [25066.039074] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25066.039076] iTCO_vendor_support [25066.039077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25066.043339] i2c_smbus [25066.052285] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25066.055506] crct10dif_pclmul joydev [25066.062979] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25066.065335] crc32_pclmul ipmi_ssif [25066.070812] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25066.074383] ghash_clmulni_intel [25066.093122] [25066.096605] mei_me mei [25066.101823] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25066.105048] lpc_ich sha512_ssse3 [25066.106538] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25066.108982] acpi_ipmi mfd_core [25066.116104] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25066.119416] wmi [25066.126540] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25066.129678] ipmi_si ipmi_devintf [25066.136803] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25066.138639] ipmi_msghandler ioatdma acpi_pad [25066.145763] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25066.149073] acpi_power_meter ixgbe [25066.156197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25066.160548] igb mdio [25066.168627] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25066.172110] crc32c_intel nvme [25066.177848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25066.180119] ptp nvme_core [25066.187243] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25066.190294] i2c_algo_bit pps_core [25066.197417] Call Trace: [25066.197420] [25066.200122] dca [25066.207248] ? watchdog_timer_fn+0x1b4/0x220 [25066.210644] fuse [25066.213088] ? __hrtimer_run_queues+0x1d3/0x360 [25066.215099] irq event stamp: 122611984 [25066.216940] ? hrtimer_interrupt+0x104/0x240 [25066.221199] hardirqs last enabled at (122611983): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25066.223124] ? __local_bh_enable+0x50/0x60 [25066.227647] hardirqs last disabled at (122611984): [] sysvec_apic_timer_interrupt+0xe/0x90 [25066.231395] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25066.235656] softirqs last enabled at (122611982): [] __do_softirq+0x2be/0x371 [25066.245902] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25066.249991] softirqs last disabled at (122611977): [] irq_exit_rcu+0xb0/0xd0 [25066.259801] [25066.265020] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25066.273789] [25066.278748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25066.278749] Workqueue: events delayed_work [ceph] [25066.287344] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25066.289442] [25066.289443] RIP: 0010:lock_acquire+0xf0/0x2b0 [25066.298825] ? queued_spin_lock_slowpath+0x16/0x2e0 [25066.300916] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25066.308389] do_raw_spin_lock+0xb5/0xc0 [25066.313085] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25066.318392] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25066.319881] [25066.319882] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25066.324234] try_get_cap_refs+0x159/0x880 [ceph] [25066.329102] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25066.347846] __ceph_get_caps+0x4a1/0x760 [ceph] [25066.351669] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25066.356894] ? __file_remove_privs+0x89/0x110 [25066.361670] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25066.363162] ? wake_bit_function+0x70/0x70 [25066.370287] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25066.374899] ? ceph_write_iter+0x260/0x770 [ceph] [25066.382021] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25066.386546] ceph_write_iter+0x260/0x770 [ceph] [25066.393669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25066.398025] ? lock_is_held_type+0xaa/0x120 [25066.405144] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25066.409237] vfs_write+0x324/0x4f0 [25066.416358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25066.421063] ksys_pwrite64+0x65/0xa0 [25066.429133] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25066.433659] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25066.439395] Call Trace: [25066.443573] do_syscall_64+0x38/0x80 [25066.450698] [25066.454094] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25066.461232] ? watchdog_timer_fn+0x1b4/0x220 [25066.464800] RIP: 0033:0x7f48f0eeb93b [25066.471923] ? __hrtimer_run_queues+0x1d3/0x360 [25066.477054] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25066.479516] ? hrtimer_interrupt+0x104/0x240 [25066.483086] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25066.485095] ? __local_bh_enable+0x50/0x60 [25066.490140] ORIG_RAX: 0000000000000012 [25066.494407] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25066.497974] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25066.502516] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25066.521253] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25066.525517] [25066.530734] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25066.534825] [25066.538657] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25066.543874] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25066.550999] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25066.555961] ? lock_acquire+0xf0/0x2b0 [25066.563085] [25066.565180] ? igrab+0x43/0x50 [25066.602932] ? lock_release+0xc5/0x260 [25066.606675] _raw_spin_lock+0x30/0x40 [25066.610333] ? igrab+0x1d/0x50 [25066.613382] igrab+0x1d/0x50 [25066.616263] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25066.621412] delayed_work+0x1d8/0x280 [ceph] [25066.625690] ? process_scheduled_works+0x22a/0x510 [25066.630473] process_scheduled_works+0x2a4/0x510 [25066.635090] worker_thread+0x188/0x340 [25066.638837] ? rescuer_thread+0x310/0x310 [25066.642841] kthread+0xf5/0x130 [25066.645981] ? kthread_complete_and_exit+0x20/0x20 [25066.650773] ret_from_fork+0x31/0x40 [25066.654351] ? kthread_complete_and_exit+0x20/0x20 [25066.659135] ret_from_fork_asm+0x11/0x20 [25066.663068] [25093.897308] watchdog: BUG: soft lockup - CPU#0 stuck for 12256s! [dbench:78683] [25093.904612] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25093.909308] watchdog: BUG: soft lockup - CPU#7 stuck for 21253s! [kworker/7:4:78808] [25093.960054] irq event stamp: 40197646 [25093.960055] hardirqs last enabled at (40197645): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25093.967784] Modules linked in: [25093.971441] hardirqs last disabled at (40197646): [] sysvec_apic_timer_interrupt+0xe/0x90 [25093.981599] ceph [25093.984650] softirqs last enabled at (40197588): [] __do_softirq+0x2be/0x371 [25093.994373] libceph dns_resolver [25093.996297] softirqs last disabled at (40197583): [] irq_exit_rcu+0xb0/0xd0 [25094.004984] fscache netfs [25094.008292] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25094.016803] veth nft_chain_nat [25094.019508] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25094.028451] xt_MASQUERADE [25094.031588] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25094.039060] nf_nat [25094.041765] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25094.047240] nf_conntrack [25094.049340] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25094.068077] nf_defrag_ipv6 nf_defrag_ipv4 [25094.070705] [25094.075919] nft_compat [25094.080011] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25094.081501] nf_tables [25094.083945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25094.091069] nfnetlink bridge [25094.093428] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25094.100550] stp llc [25094.103515] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25094.110639] xfs libcrc32c [25094.112823] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25094.119946] sunrpc intel_rapl_msr [25094.122652] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25094.129776] intel_rapl_common sb_edac [25094.133174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25094.141249] x86_pkg_temp_thermal coretemp [25094.144995] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25094.150731] kvm_intel kvm [25094.154824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25094.161946] irqbypass i2c_i801 [25094.164651] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25094.171775] iTCO_wdt iTCO_vendor_support [25094.174913] Call Trace: [25094.174915] [25094.182037] i2c_smbus crct10dif_pclmul [25094.186042] ? watchdog_timer_fn+0x1b4/0x220 [25094.188484] joydev crc32_pclmul [25094.190495] ? __hrtimer_run_queues+0x1d3/0x360 [25094.194327] ipmi_ssif ghash_clmulni_intel [25094.198593] ? hrtimer_interrupt+0x104/0x240 [25094.201813] mei_me mei [25094.206338] ? __local_bh_enable+0x50/0x60 [25094.210430] lpc_ich sha512_ssse3 [25094.214709] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25094.217153] acpi_ipmi mfd_core [25094.221247] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25094.224556] wmi ipmi_si [25094.229774] [25094.232911] ipmi_devintf ipmi_msghandler [25094.237869] [25094.237870] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25094.240400] ioatdma [25094.242500] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25094.246500] acpi_pad [25094.248600] do_raw_spin_lock+0xb5/0xc0 [25094.253902] acpi_power_meter ixgbe [25094.256088] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25094.260958] igb mdio [25094.263230] try_get_cap_refs+0x159/0x880 [ceph] [25094.267059] crc32c_intel [25094.270547] __ceph_get_caps+0x4a1/0x760 [ceph] [25094.275326] nvme ptp [25094.277602] ? __file_remove_privs+0x89/0x110 [25094.282208] nvme_core i2c_algo_bit [25094.284824] ? wake_bit_function+0x70/0x70 [25094.289349] pps_core [25094.291623] ? ceph_write_iter+0x260/0x770 [ceph] [25094.295970] dca fuse [25094.299455] ceph_write_iter+0x260/0x770 [ceph] [25094.303546] [25094.305819] ? lock_is_held_type+0xaa/0x120 [25094.310514] irq event stamp: 122832962 [25094.312787] vfs_write+0x324/0x4f0 [25094.317309] hardirqs last enabled at (122832961): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25094.318802] ksys_pwrite64+0x65/0xa0 [25094.322988] hardirqs last disabled at (122832962): [] sysvec_apic_timer_interrupt+0xe/0x90 [25094.326730] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25094.330126] softirqs last enabled at (122832960): [] __do_softirq+0x2be/0x371 [25094.340372] do_syscall_64+0x38/0x80 [25094.343942] softirqs last disabled at (122832955): [] irq_exit_rcu+0xb0/0xd0 [25094.353754] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25094.358883] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25094.367653] RIP: 0033:0x7f48f0eeb93b [25094.371225] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25094.379824] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25094.384867] Workqueue: events delayed_work [ceph] [25094.394245] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25094.397814] [25094.397815] RIP: 0010:igrab+0x9/0x50 [25094.405285] ORIG_RAX: 0000000000000012 [25094.405286] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25094.424024] Code: 0f 1f 44 00 00 c3 b8 00 04 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 <55> 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 [25094.428720] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25094.433937] RSP: 0018:ffffc90002117db0 EFLAGS: 00000286 [25094.435428] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25094.439006] [25094.439007] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [25094.442840] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25094.449964] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [25094.468717] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25094.475843] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [25094.481064] [25094.488184] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25094.534605] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25094.541728] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25094.549806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25094.555541] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25094.562667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25094.569791] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25094.576913] Call Trace: [25094.579359] [25094.581370] ? watchdog_timer_fn+0x1b4/0x220 [25094.585634] ? __hrtimer_run_queues+0x1d3/0x360 [25094.590160] ? hrtimer_interrupt+0x104/0x240 [25094.594423] ? __local_bh_enable+0x50/0x60 [25094.598515] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25094.603738] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25094.608712] [25094.610811] [25094.612909] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25094.618218] ? igrab+0x9/0x50 [25094.621187] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25094.626336] delayed_work+0x1d8/0x280 [ceph] [25094.630618] ? process_scheduled_works+0x22a/0x510 [25094.635406] process_scheduled_works+0x2a4/0x510 [25094.640022] worker_thread+0x188/0x340 [25094.643773] ? rescuer_thread+0x310/0x310 [25094.647784] kthread+0xf5/0x130 [25094.650923] ? kthread_complete_and_exit+0x20/0x20 [25094.655713] ret_from_fork+0x31/0x40 [25094.659284] ? kthread_complete_and_exit+0x20/0x20 [25094.664068] ret_from_fork_asm+0x11/0x20 [25094.668001] [25096.284716] rcu: INFO: rcu_preempt self-detected stall on CPU [25096.290477] rcu: 7-....: (12779763 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/12778476 fqs=6427668 [25096.301077] rcu: (t=13159440 jiffies g=1309609 q=410674 ncpus=8) [25096.307162] Sending NMI from CPU 7 to CPUs 0: [25096.311513] NMI backtrace for cpu 0 [25096.311515] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25096.311517] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25096.311518] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25096.311522] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25096.311524] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25096.311526] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25096.311527] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25096.311529] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25096.311530] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25096.311531] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25096.311533] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25096.311535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25096.311536] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25096.311538] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25096.311539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25096.311540] Call Trace: [25096.311541] [25096.311542] ? nmi_cpu_backtrace+0xad/0x120 [25096.311547] ? nmi_cpu_backtrace_handler+0x11/0x20 [25096.311549] ? nmi_handle+0xd4/0x260 [25096.311555] ? default_do_nmi+0x49/0xf0 [25096.311557] ? exc_nmi+0x11e/0x150 [25096.311560] ? end_repeat_nmi+0x16/0x67 [25096.311567] ? queued_spin_lock_slowpath+0x16/0x2e0 [25096.311570] ? queued_spin_lock_slowpath+0x16/0x2e0 [25096.311574] ? queued_spin_lock_slowpath+0x16/0x2e0 [25096.311577] [25096.311578] [25096.311580] do_raw_spin_lock+0xb5/0xc0 [25096.311583] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25096.311602] try_get_cap_refs+0x159/0x880 [ceph] [25096.311624] __ceph_get_caps+0x4a1/0x760 [ceph] [25096.311643] ? __file_remove_privs+0x89/0x110 [25096.311646] ? wake_bit_function+0x70/0x70 [25096.311651] ? ceph_write_iter+0x260/0x770 [ceph] [25096.311666] ceph_write_iter+0x260/0x770 [ceph] [25096.311684] ? lock_is_held_type+0xaa/0x120 [25096.311688] vfs_write+0x324/0x4f0 [25096.311695] ksys_pwrite64+0x65/0xa0 [25096.311697] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25096.311701] do_syscall_64+0x38/0x80 [25096.311704] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25096.311707] RIP: 0033:0x7f48f0eeb93b [25096.311708] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25096.311710] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [25096.311712] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25096.311713] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25096.311714] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25096.311716] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25096.311717] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25096.311740] [25096.312512] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25096.616124] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25096.623595] Workqueue: events delayed_work [ceph] [25096.628310] RIP: 0010:lock_acquire+0xf0/0x2b0 [25096.632659] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25096.651397] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25096.656614] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25096.663747] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25096.670873] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25096.677995] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25096.685120] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25096.692244] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25096.700321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25096.706060] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25096.713184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25096.720309] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25096.727433] Call Trace: [25096.729877] [25096.731886] ? rcu_dump_cpu_stacks+0xd1/0x180 [25096.736247] ? rcu_sched_clock_irq+0x633/0x1390 [25096.740770] ? lock_release+0xc5/0x260 [25096.744514] ? lock_is_held_type+0xaa/0x120 [25096.748695] ? lock_is_held_type+0xaa/0x120 [25096.752879] ? tick_sched_do_timer+0xa0/0xa0 [25096.757150] ? rcu_read_lock_sched_held+0x56/0x90 [25096.761850] ? tick_sched_do_timer+0xa0/0xa0 [25096.766112] ? update_process_times+0x5f/0x90 [25096.770462] ? tick_sched_handle.isra.16+0x1e/0x50 [25096.775247] ? tick_sched_timer+0x6f/0x80 [25096.779252] ? __hrtimer_run_queues+0x1d3/0x360 [25096.783779] ? hrtimer_interrupt+0x104/0x240 [25096.788049] ? __local_bh_enable+0x50/0x60 [25096.792140] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25096.797355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25096.802313] [25096.804411] [25096.806509] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25096.811816] ? lock_acquire+0xf0/0x2b0 [25096.815567] ? igrab+0x43/0x50 [25096.818616] ? lock_release+0xc5/0x260 [25096.822361] _raw_spin_lock+0x30/0x40 [25096.826033] ? igrab+0x1d/0x50 [25096.829085] igrab+0x1d/0x50 [25096.831963] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25096.837114] delayed_work+0x1d8/0x280 [ceph] [25096.841394] ? process_scheduled_works+0x22a/0x510 [25096.846178] process_scheduled_works+0x2a4/0x510 [25096.850791] worker_thread+0x188/0x340 [25096.854541] ? rescuer_thread+0x310/0x310 [25096.858543] kthread+0xf5/0x130 [25096.861680] ? kthread_complete_and_exit+0x20/0x20 [25096.866466] ret_from_fork+0x31/0x40 [25096.870037] ? kthread_complete_and_exit+0x20/0x20 [25096.874820] ret_from_fork_asm+0x11/0x20 [25096.878758] [25121.897795] watchdog: BUG: soft lockup - CPU#0 stuck for 12282s! [dbench:78683] [25121.905101] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25121.909794] watchdog: BUG: soft lockup - CPU#7 stuck for 21279s! [kworker/7:4:78808] [25121.960549] irq event stamp: 40255958 [25121.968281] Modules linked in: ceph libceph dns_resolver fscache netfs veth [25121.971937] hardirqs last enabled at (40255957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25121.978899] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [25121.989053] hardirqs last disabled at (40255958): [] sysvec_apic_timer_interrupt+0xe/0x90 [25121.994706] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [25122.004428] softirqs last enabled at (40255902): [] __do_softirq+0x2be/0x371 [25122.011814] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25122.020498] softirqs last disabled at (40255897): [] irq_exit_rcu+0xb0/0xd0 [25122.027361] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [25122.035874] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25122.041523] kvm irqbypass [25122.050470] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25122.050472] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25122.053188] i2c_i801 [25122.060661] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25122.066138] iTCO_wdt iTCO_vendor_support [25122.068409] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25122.087148] i2c_smbus [25122.091151] [25122.096366] crct10dif_pclmul [25122.098726] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25122.100216] joydev crc32_pclmul [25122.103192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25122.110314] ipmi_ssif [25122.113537] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25122.120660] ghash_clmulni_intel mei_me [25122.123017] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25122.130143] mei lpc_ich [25122.133974] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25122.141098] sha512_ssse3 acpi_ipmi [25122.143627] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25122.150752] mfd_core wmi [25122.154237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25122.162313] ipmi_si [25122.164930] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25122.170667] ipmi_devintf [25122.172852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25122.179977] ipmi_msghandler ioatdma [25122.182594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25122.189719] acpi_pad acpi_power_meter [25122.193287] Call Trace: [25122.193290] [25122.200413] ixgbe igb [25122.204158] ? watchdog_timer_fn+0x1b4/0x220 [25122.206600] mdio crc32c_intel [25122.208614] ? __hrtimer_run_queues+0x1d3/0x360 [25122.210970] nvme [25122.215235] ? hrtimer_interrupt+0x104/0x240 [25122.218284] ptp nvme_core [25122.222809] ? __local_bh_enable+0x50/0x60 [25122.224733] i2c_algo_bit [25122.228998] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25122.231700] pps_core dca [25122.235792] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25122.238409] fuse [25122.243625] [25122.246243] irq event stamp: 123049152 [25122.251199] [25122.253123] hardirqs last enabled at (123049151): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25122.255223] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25122.258965] hardirqs last disabled at (123049152): [] sysvec_apic_timer_interrupt+0xe/0x90 [25122.261067] ? queued_spin_lock_slowpath+0x18/0x2e0 [25122.271308] softirqs last enabled at (123049150): [] __do_softirq+0x2be/0x371 [25122.276614] do_raw_spin_lock+0xb5/0xc0 [25122.286422] softirqs last disabled at (123049145): [] irq_exit_rcu+0xb0/0xd0 [25122.291296] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25122.300064] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25122.303897] try_get_cap_refs+0x159/0x880 [ceph] [25122.312491] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25122.317282] __ceph_get_caps+0x4a1/0x760 [ceph] [25122.326655] Workqueue: events delayed_work [ceph] [25122.331270] ? __file_remove_privs+0x89/0x110 [25122.338734] [25122.338736] RIP: 0010:lock_acquire+0xf0/0x2b0 [25122.343259] ? wake_bit_function+0x70/0x70 [25122.347957] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25122.352309] ? ceph_write_iter+0x260/0x770 [ceph] [25122.353799] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25122.358149] ceph_write_iter+0x260/0x770 [ceph] [25122.362241] [25122.362242] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25122.380981] ? lock_is_held_type+0xaa/0x120 [25122.385675] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25122.390894] vfs_write+0x324/0x4f0 [25122.395416] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25122.396912] ksys_pwrite64+0x65/0xa0 [25122.404031] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25122.408210] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25122.415333] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25122.418730] do_syscall_64+0x38/0x80 [25122.425855] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25122.429426] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25122.436548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25122.441681] RIP: 0033:0x7f48f0eeb93b [25122.448804] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25122.452376] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25122.460451] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25122.465514] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25122.471252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25122.474820] ORIG_RAX: 0000000000000012 [25122.481946] Call Trace: [25122.481947] [25122.500682] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25122.507809] ? watchdog_timer_fn+0x1b4/0x220 [25122.513026] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25122.520151] ? __hrtimer_run_queues+0x1d3/0x360 [25122.523979] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25122.526429] ? hrtimer_interrupt+0x104/0x240 [25122.528435] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25122.535560] ? __local_bh_enable+0x50/0x60 [25122.539822] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25122.546949] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25122.551491] [25122.558612] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25122.593576] [25122.595674] [25122.597769] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25122.603078] ? lock_acquire+0xf0/0x2b0 [25122.606829] ? igrab+0x43/0x50 [25122.609878] ? lock_release+0xc5/0x260 [25122.613622] _raw_spin_lock+0x30/0x40 [25122.617281] ? igrab+0x1d/0x50 [25122.620330] igrab+0x1d/0x50 [25122.623211] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25122.628359] delayed_work+0x1d8/0x280 [ceph] [25122.632641] ? process_scheduled_works+0x22a/0x510 [25122.637430] process_scheduled_works+0x2a4/0x510 [25122.642045] worker_thread+0x188/0x340 [25122.645793] ? rescuer_thread+0x310/0x310 [25122.649797] kthread+0xf5/0x130 [25122.652945] ? kthread_complete_and_exit+0x20/0x20 [25122.657737] ret_from_fork+0x31/0x40 [25122.661315] ? kthread_complete_and_exit+0x20/0x20 [25122.666100] ret_from_fork_asm+0x11/0x20 [25122.670031] [25149.898281] watchdog: BUG: soft lockup - CPU#0 stuck for 12308s! [dbench:78683] [25149.905578] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25149.910281] watchdog: BUG: soft lockup - CPU#7 stuck for 21305s! [kworker/7:4:78808] [25149.961020] irq event stamp: 40314146 [25149.961022] hardirqs last enabled at (40314145): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25149.968752] Modules linked in: [25149.972409] hardirqs last disabled at (40314146): [] sysvec_apic_timer_interrupt+0xe/0x90 [25149.982565] ceph libceph [25149.985616] softirqs last enabled at (40314098): [] __do_softirq+0x2be/0x371 [25149.995340] dns_resolver fscache [25149.997965] softirqs last disabled at (40314093): [] irq_exit_rcu+0xb0/0xd0 [25150.006652] netfs veth [25150.009979] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25150.018491] nft_chain_nat xt_MASQUERADE [25150.020934] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25150.029896] nf_nat nf_conntrack [25150.033812] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25150.041283] nf_defrag_ipv6 nf_defrag_ipv4 [25150.044507] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25150.049984] nft_compat nf_tables [25150.054075] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25150.072814] nfnetlink bridge [25150.076124] [25150.076125] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25150.081343] stp llc [25150.084305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25150.085795] xfs libcrc32c [25150.092921] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25150.095105] sunrpc [25150.102229] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25150.104934] intel_rapl_msr intel_rapl_common [25150.112057] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25150.114155] sb_edac x86_pkg_temp_thermal [25150.121277] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25150.125629] coretemp kvm_intel [25150.132753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25150.136756] kvm [25150.144834] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25150.147971] irqbypass i2c_i801 [25150.153709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25150.155547] iTCO_wdt iTCO_vendor_support [25150.162678] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25150.165817] i2c_smbus crct10dif_pclmul [25150.172956] Call Trace: [25150.172957] [25150.176966] joydev [25150.184087] ? watchdog_timer_fn+0x1b4/0x220 [25150.187917] crc32_pclmul ipmi_ssif [25150.190362] ? __hrtimer_run_queues+0x1d3/0x360 [25150.192373] ghash_clmulni_intel mei_me [25150.194474] ? hrtimer_interrupt+0x104/0x240 [25150.198733] mei lpc_ich [25150.202217] ? __local_bh_enable+0x50/0x60 [25150.206743] sha512_ssse3 acpi_ipmi [25150.210575] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25150.214836] mfd_core wmi [25150.217369] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25150.221457] ipmi_si ipmi_devintf [25150.224942] [25150.230177] ipmi_msghandler ioatdma [25150.232794] [25150.237751] acpi_pad acpi_power_meter [25150.241062] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25150.243160] ixgbe igb [25150.246736] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25150.248827] mdio crc32c_intel [25150.252573] do_raw_spin_lock+0xb5/0xc0 [25150.257876] nvme ptp [25150.260235] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25150.265103] nvme_core i2c_algo_bit [25150.268156] try_get_cap_refs+0x159/0x880 [ceph] [25150.271986] pps_core dca [25150.274261] __ceph_get_caps+0x4a1/0x760 [ceph] [25150.279041] fuse [25150.282528] ? __file_remove_privs+0x89/0x110 [25150.287135] irq event stamp: 123270618 [25150.289754] ? wake_bit_function+0x70/0x70 [25150.294276] hardirqs last enabled at (123270617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25150.296202] ? ceph_write_iter+0x260/0x770 [ceph] [25150.300553] hardirqs last disabled at (123270618): [] sysvec_apic_timer_interrupt+0xe/0x90 [25150.304295] ceph_write_iter+0x260/0x770 [ceph] [25150.308385] softirqs last enabled at (123270616): [] __do_softirq+0x2be/0x371 [25150.318633] ? lock_is_held_type+0xaa/0x120 [25150.323327] softirqs last disabled at (123270611): [] irq_exit_rcu+0xb0/0xd0 [25150.333141] vfs_write+0x324/0x4f0 [25150.337680] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25150.346455] ksys_pwrite64+0x65/0xa0 [25150.350628] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25150.359227] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25150.362623] Workqueue: events delayed_work [ceph] [25150.372002] do_syscall_64+0x38/0x80 [25150.375572] [25150.375573] RIP: 0010:lock_acquire+0xf0/0x2b0 [25150.383044] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25150.388172] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25150.392871] RIP: 0033:0x7f48f0eeb93b [25150.396440] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25150.397931] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25150.402283] [25150.402284] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25150.407326] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25150.426066] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25150.429636] ORIG_RAX: 0000000000000012 [25150.434854] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25150.453590] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25150.455082] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25150.462205] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25150.467422] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25150.474548] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25150.478379] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25150.485501] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25150.492627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25150.499749] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25150.506873] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25150.514004] [25150.521122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25150.565619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25150.572743] Call Trace: [25150.575188] [25150.577198] ? watchdog_timer_fn+0x1b4/0x220 [25150.581463] ? __hrtimer_run_queues+0x1d3/0x360 [25150.585989] ? hrtimer_interrupt+0x104/0x240 [25150.590257] ? __local_bh_enable+0x50/0x60 [25150.594351] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25150.599567] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25150.604525] [25150.606620] [25150.608720] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25150.614028] ? lock_acquire+0xf0/0x2b0 [25150.617778] ? igrab+0x43/0x50 [25150.620827] ? lock_release+0xc5/0x260 [25150.624572] _raw_spin_lock+0x30/0x40 [25150.628236] ? igrab+0x1d/0x50 [25150.631286] igrab+0x1d/0x50 [25150.634166] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25150.639314] delayed_work+0x1d8/0x280 [ceph] [25150.643596] ? process_scheduled_works+0x22a/0x510 [25150.648385] process_scheduled_works+0x2a4/0x510 [25150.653001] worker_thread+0x188/0x340 [25150.656751] ? rescuer_thread+0x310/0x310 [25150.660755] kthread+0xf5/0x130 [25150.663891] ? kthread_complete_and_exit+0x20/0x20 [25150.668689] ret_from_fork+0x31/0x40 [25150.672264] ? kthread_complete_and_exit+0x20/0x20 [25150.677049] ret_from_fork_asm+0x11/0x20 [25150.680980] [25177.898769] watchdog: BUG: soft lockup - CPU#0 stuck for 12334s! [dbench:78683] [25177.906076] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25177.910769] watchdog: BUG: soft lockup - CPU#7 stuck for 21331s! [kworker/7:4:78808] [25177.961525] irq event stamp: 40372298 [25177.961527] hardirqs last enabled at (40372297): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25177.969256] Modules linked in: [25177.972913] hardirqs last disabled at (40372298): [] sysvec_apic_timer_interrupt+0xe/0x90 [25177.983071] ceph [25177.986121] softirqs last enabled at (40372250): [] __do_softirq+0x2be/0x371 [25177.995847] libceph [25177.997770] softirqs last disabled at (40372245): [] irq_exit_rcu+0xb0/0xd0 [25178.006453] dns_resolver [25178.008638] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25178.017164] fscache netfs [25178.019784] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25178.028728] veth nft_chain_nat [25178.031447] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25178.038919] xt_MASQUERADE nf_nat [25178.042058] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25178.047535] nf_conntrack [25178.050845] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25178.069583] nf_defrag_ipv6 nf_defrag_ipv4 [25178.072201] [25178.077419] nft_compat nf_tables [25178.081509] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25178.083000] nfnetlink bridge [25178.086310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25178.093445] stp llc [25178.096406] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25178.103531] xfs libcrc32c [25178.105716] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25178.112840] sunrpc [25178.115543] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25178.122666] intel_rapl_msr [25178.124764] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25178.131889] intel_rapl_common sb_edac [25178.134679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25178.142759] x86_pkg_temp_thermal [25178.146500] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25178.152238] coretemp kvm_intel [25178.155549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25178.162673] kvm irqbypass [25178.165811] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25178.172934] i2c_i801 [25178.175639] Call Trace: [25178.175642] [25178.182764] iTCO_wdt iTCO_vendor_support [25178.185035] ? watchdog_timer_fn+0x1b4/0x220 [25178.187479] i2c_smbus crct10dif_pclmul [25178.189490] ? __hrtimer_run_queues+0x1d3/0x360 [25178.193493] joydev [25178.197760] ? hrtimer_interrupt+0x104/0x240 [25178.201587] crc32_pclmul ipmi_ssif [25178.206115] ? __local_bh_enable+0x50/0x60 [25178.208209] ghash_clmulni_intel mei_me [25178.212475] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25178.215957] mei lpc_ich [25178.220049] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25178.223878] sha512_ssse3 [25178.229097] [25178.231628] acpi_ipmi mfd_core [25178.236585] [25178.236587] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25178.239202] wmi ipmi_si [25178.241304] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25178.244436] ipmi_devintf ipmi_msghandler [25178.246553] do_raw_spin_lock+0xb5/0xc0 [25178.251855] ioatdma acpi_pad [25178.254388] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25178.259258] acpi_power_meter ixgbe [25178.263265] try_get_cap_refs+0x159/0x880 [ceph] [25178.267092] igb mdio [25178.270060] __ceph_get_caps+0x4a1/0x760 [ceph] [25178.274839] crc32c_intel nvme [25178.278328] ? __file_remove_privs+0x89/0x110 [25178.282934] ptp nvme_core [25178.285205] ? wake_bit_function+0x70/0x70 [25178.289730] i2c_algo_bit [25178.292783] ? ceph_write_iter+0x260/0x770 [ceph] [25178.297130] pps_core dca [25178.299836] ceph_write_iter+0x260/0x770 [ceph] [25178.303925] fuse [25178.306547] ? lock_is_held_type+0xaa/0x120 [25178.311240] irq event stamp: 123491452 [25178.313860] vfs_write+0x324/0x4f0 [25178.318383] hardirqs last enabled at (123491451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25178.320310] ksys_pwrite64+0x65/0xa0 [25178.324482] hardirqs last disabled at (123491452): [] sysvec_apic_timer_interrupt+0xe/0x90 [25178.328228] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25178.331625] softirqs last enabled at (123491450): [] __do_softirq+0x2be/0x371 [25178.341870] do_syscall_64+0x38/0x80 [25178.345439] softirqs last disabled at (123491445): [] irq_exit_rcu+0xb0/0xd0 [25178.355269] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25178.360399] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25178.369170] RIP: 0033:0x7f48f0eeb93b [25178.372739] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25178.381340] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25178.386381] Workqueue: events delayed_work [ceph] [25178.395760] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25178.399331] [25178.399332] RIP: 0010:lock_acquire+0xf0/0x2b0 [25178.406802] ORIG_RAX: 0000000000000012 [25178.425539] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25178.430235] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25178.435462] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25178.436953] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25178.441302] [25178.445133] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25178.463871] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25178.470995] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25178.476214] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25178.483337] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25178.484829] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25178.491957] [25178.499075] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25178.536898] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25178.544022] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25178.552100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25178.557838] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25178.564961] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25178.572086] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25178.579211] Call Trace: [25178.581656] [25178.583665] ? watchdog_timer_fn+0x1b4/0x220 [25178.587930] ? __hrtimer_run_queues+0x1d3/0x360 [25178.592458] ? hrtimer_interrupt+0x104/0x240 [25178.596725] ? __local_bh_enable+0x50/0x60 [25178.600820] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25178.606044] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25178.611000] [25178.613099] [25178.615195] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25178.620502] ? lock_acquire+0xf0/0x2b0 [25178.624254] ? igrab+0x43/0x50 [25178.627313] ? lock_release+0xc5/0x260 [25178.631057] _raw_spin_lock+0x30/0x40 [25178.634713] ? igrab+0x1d/0x50 [25178.637763] igrab+0x1d/0x50 [25178.640644] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25178.645791] delayed_work+0x1d8/0x280 [ceph] [25178.650074] ? process_scheduled_works+0x22a/0x510 [25178.654862] process_scheduled_works+0x2a4/0x510 [25178.659478] worker_thread+0x188/0x340 [25178.663227] ? rescuer_thread+0x310/0x310 [25178.667231] kthread+0xf5/0x130 [25178.670379] ? kthread_complete_and_exit+0x20/0x20 [25178.675177] ret_from_fork+0x31/0x40 [25178.678749] ? kthread_complete_and_exit+0x20/0x20 [25178.683534] ret_from_fork_asm+0x11/0x20 [25178.687464] [25205.899256] watchdog: BUG: soft lockup - CPU#0 stuck for 12360s! [dbench:78683] [25205.906555] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25205.911256] watchdog: BUG: soft lockup - CPU#7 stuck for 21357s! [kworker/7:4:78808] [25205.961995] irq event stamp: 40430544 [25205.961997] hardirqs last enabled at (40430543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25205.969727] Modules linked in: [25205.973385] hardirqs last disabled at (40430544): [] sysvec_apic_timer_interrupt+0xe/0x90 [25205.983541] ceph [25205.986592] softirqs last enabled at (40430496): [] __do_softirq+0x2be/0x371 [25205.996318] libceph [25205.998241] softirqs last disabled at (40430491): [] irq_exit_rcu+0xb0/0xd0 [25206.006931] dns_resolver fscache [25206.009117] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25206.017628] netfs [25206.020955] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25206.029901] veth nft_chain_nat [25206.031927] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25206.039398] xt_MASQUERADE nf_nat [25206.042536] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25206.048014] nf_conntrack [25206.051325] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25206.070062] nf_defrag_ipv6 nf_defrag_ipv4 [25206.072680] [25206.072681] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25206.077896] nft_compat [25206.081988] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25206.083480] nf_tables nfnetlink [25206.090602] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25206.093048] bridge stp [25206.100171] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25206.103395] llc xfs [25206.110520] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25206.112963] libcrc32c sunrpc [25206.120088] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25206.122272] intel_rapl_msr intel_rapl_common [25206.129397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25206.132360] sb_edac [25206.140438] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25206.144788] x86_pkg_temp_thermal [25206.150525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25206.152709] coretemp kvm_intel [25206.159835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25206.163144] kvm irqbypass [25206.170269] Call Trace: [25206.170270] [25206.173405] i2c_i801 [25206.180531] ? watchdog_timer_fn+0x1b4/0x220 [25206.183235] iTCO_wdt [25206.185678] ? __hrtimer_run_queues+0x1d3/0x360 [25206.187688] iTCO_vendor_support [25206.189962] ? hrtimer_interrupt+0x104/0x240 [25206.194223] i2c_smbus [25206.196495] ? __local_bh_enable+0x50/0x60 [25206.201018] crct10dif_pclmul joydev [25206.204244] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25206.208505] crc32_pclmul ipmi_ssif [25206.210863] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25206.214955] ghash_clmulni_intel mei_me [25206.218525] [25206.223742] mei lpc_ich [25206.227226] [25206.232184] sha512_ssse3 [25206.236015] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25206.238113] acpi_ipmi mfd_core [25206.240657] ? queued_spin_lock_slowpath+0x18/0x2e0 [25206.242750] wmi ipmi_si [25206.245369] do_raw_spin_lock+0xb5/0xc0 [25206.250669] ipmi_devintf [25206.253808] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25206.258679] ipmi_msghandler ioatdma [25206.261212] try_get_cap_refs+0x159/0x880 [ceph] [25206.265040] acpi_pad [25206.267662] __ceph_get_caps+0x4a1/0x760 [ceph] [25206.272441] acpi_power_meter [25206.276016] ? __file_remove_privs+0x89/0x110 [25206.280623] ixgbe igb [25206.282894] ? wake_bit_function+0x70/0x70 [25206.287419] mdio crc32c_intel [25206.290385] ? ceph_write_iter+0x260/0x770 [ceph] [25206.294733] nvme ptp [25206.297089] ceph_write_iter+0x260/0x770 [ceph] [25206.301181] nvme_core i2c_algo_bit [25206.304235] ? lock_is_held_type+0xaa/0x120 [25206.308928] pps_core dca [25206.311220] vfs_write+0x324/0x4f0 [25206.315741] fuse [25206.319229] ksys_pwrite64+0x65/0xa0 [25206.323402] irq event stamp: 123712216 [25206.326020] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25206.329418] hardirqs last enabled at (123712215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25206.331343] do_syscall_64+0x38/0x80 [25206.334911] hardirqs last disabled at (123712216): [] sysvec_apic_timer_interrupt+0xe/0x90 [25206.338659] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25206.343787] softirqs last enabled at (123712214): [] __do_softirq+0x2be/0x371 [25206.354030] RIP: 0033:0x7f48f0eeb93b [25206.357602] softirqs last disabled at (123712209): [] irq_exit_rcu+0xb0/0xd0 [25206.367412] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25206.372456] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25206.381229] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25206.384798] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25206.393396] ORIG_RAX: 0000000000000012 [25206.412132] Workqueue: events delayed_work [ceph] [25206.421511] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25206.426727] [25206.426728] RIP: 0010:lock_acquire+0xf0/0x2b0 [25206.434199] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25206.438031] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25206.442726] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25206.449851] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25206.451342] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25206.455693] [25206.462816] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25206.481556] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25206.488685] [25206.493896] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25206.526086] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25206.533209] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25206.540333] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25206.547457] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25206.555534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25206.561273] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25206.568397] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25206.575519] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25206.582660] Call Trace: [25206.585107] [25206.587118] ? watchdog_timer_fn+0x1b4/0x220 [25206.591382] ? __hrtimer_run_queues+0x1d3/0x360 [25206.595908] ? hrtimer_interrupt+0x104/0x240 [25206.600168] ? __local_bh_enable+0x50/0x60 [25206.604263] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25206.609485] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25206.614444] [25206.616542] [25206.618640] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25206.623965] ? lock_acquire+0xf0/0x2b0 [25206.627715] ? igrab+0x43/0x50 [25206.630763] ? lock_release+0xc5/0x260 [25206.634510] _raw_spin_lock+0x30/0x40 [25206.638173] ? igrab+0x1d/0x50 [25206.641224] igrab+0x1d/0x50 [25206.644103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25206.649254] delayed_work+0x1d8/0x280 [ceph] [25206.653534] ? process_scheduled_works+0x22a/0x510 [25206.658322] process_scheduled_works+0x2a4/0x510 [25206.662948] worker_thread+0x188/0x340 [25206.666696] ? rescuer_thread+0x310/0x310 [25206.670699] kthread+0xf5/0x130 [25206.673838] ? kthread_complete_and_exit+0x20/0x20 [25206.678623] ret_from_fork+0x31/0x40 [25206.682201] ? kthread_complete_and_exit+0x20/0x20 [25206.686986] ret_from_fork_asm+0x11/0x20 [25206.690917] [25233.899743] watchdog: BUG: soft lockup - CPU#0 stuck for 12386s! [dbench:78683] [25233.907045] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25233.911742] watchdog: BUG: soft lockup - CPU#7 stuck for 21383s! [kworker/7:4:78808] [25233.962492] irq event stamp: 40488808 [25233.970224] Modules linked in: ceph libceph dns_resolver fscache netfs [25233.973880] hardirqs last enabled at (40488807): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25233.980417] veth nft_chain_nat xt_MASQUERADE nf_nat [25233.990572] hardirqs last disabled at (40488808): [] sysvec_apic_timer_interrupt+0xe/0x90 [25233.995531] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [25234.005256] softirqs last enabled at (40488760): [] __do_softirq+0x2be/0x371 [25234.011431] nf_tables nfnetlink bridge stp llc xfs [25234.020135] softirqs last disabled at (40488755): [] irq_exit_rcu+0xb0/0xd0 [25234.025007] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25234.033517] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25234.039342] sb_edac x86_pkg_temp_thermal [25234.048286] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25234.048287] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25234.052289] coretemp kvm_intel [25234.059763] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25234.065237] kvm irqbypass [25234.068376] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25234.087113] i2c_i801 [25234.089834] [25234.095052] iTCO_wdt iTCO_vendor_support [25234.097323] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25234.098814] i2c_smbus crct10dif_pclmul [25234.102816] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25234.109941] joydev crc32_pclmul [25234.113773] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25234.120897] ipmi_ssif [25234.124136] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25234.131262] ghash_clmulni_intel mei_me [25234.133618] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25234.140742] mei [25234.144573] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25234.151699] lpc_ich [25234.153535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25234.161613] sha512_ssse3 acpi_ipmi [25234.163798] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25234.169533] mfd_core wmi [25234.173017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25234.180141] ipmi_si ipmi_devintf [25234.182759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25234.189885] ipmi_msghandler [25234.193195] Call Trace: [25234.200318] ioatdma acpi_pad [25234.203199] [25234.205641] acpi_power_meter ixgbe [25234.208608] ? watchdog_timer_fn+0x1b4/0x220 [25234.210614] igb mdio [25234.214101] ? __hrtimer_run_queues+0x1d3/0x360 [25234.218363] crc32c_intel nvme [25234.220638] ? hrtimer_interrupt+0x104/0x240 [25234.225158] ptp nvme_core [25234.228209] ? __local_bh_enable+0x50/0x60 [25234.232473] i2c_algo_bit [25234.235178] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25234.239268] pps_core dca [25234.241887] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25234.247101] fuse [25234.249720] [25234.254677] irq event stamp: 123932974 [25234.256601] [25234.258698] hardirqs last enabled at (123932973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25234.262443] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25234.264540] hardirqs last disabled at (123932974): [] sysvec_apic_timer_interrupt+0xe/0x90 [25234.274787] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25234.280087] softirqs last enabled at (123932972): [] __do_softirq+0x2be/0x371 [25234.289901] do_raw_spin_lock+0xb5/0xc0 [25234.294771] softirqs last disabled at (123932967): [] irq_exit_rcu+0xb0/0xd0 [25234.303541] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25234.307372] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25234.315970] try_get_cap_refs+0x159/0x880 [ceph] [25234.320752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25234.330149] __ceph_get_caps+0x4a1/0x760 [ceph] [25234.334752] Workqueue: events delayed_work [ceph] [25234.342227] ? __file_remove_privs+0x89/0x110 [25234.346745] [25234.346746] RIP: 0010:lock_release+0x14e/0x260 [25234.351442] ? wake_bit_function+0x70/0x70 [25234.355792] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [25234.357285] ? ceph_write_iter+0x260/0x770 [ceph] [25234.361720] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [25234.365811] ceph_write_iter+0x260/0x770 [ceph] [25234.384549] [25234.384550] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [25234.389250] ? lock_is_held_type+0xaa/0x120 [25234.394464] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [25234.398990] vfs_write+0x324/0x4f0 [25234.400480] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [25234.407608] ksys_pwrite64+0x65/0xa0 [25234.411780] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [25234.418907] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25234.422302] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [25234.429427] do_syscall_64+0x38/0x80 [25234.432996] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25234.440122] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25234.445268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25234.452394] RIP: 0033:0x7f48f0eeb93b [25234.455964] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25234.464042] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25234.469084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25234.474824] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25234.478394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25234.485519] ORIG_RAX: 0000000000000012 [25234.504257] Call Trace: [25234.504258] [25234.511381] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25234.516598] ? watchdog_timer_fn+0x1b4/0x220 [25234.523721] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25234.527554] ? __hrtimer_run_queues+0x1d3/0x360 [25234.529995] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25234.532011] ? hrtimer_interrupt+0x104/0x240 [25234.539146] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25234.543428] ? __local_bh_enable+0x50/0x60 [25234.550553] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25234.555080] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25234.562208] [25234.566465] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25234.597166] [25234.599262] [25234.601360] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25234.606663] ? igrab+0x43/0x50 [25234.609718] ? lock_release+0x14e/0x260 [25234.613555] ? lock_release+0xc5/0x260 [25234.617299] _raw_spin_unlock+0x1b/0x40 [25234.621146] igrab+0x43/0x50 [25234.624026] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25234.629173] delayed_work+0x1d8/0x280 [ceph] [25234.633455] ? process_scheduled_works+0x22a/0x510 [25234.638245] process_scheduled_works+0x2a4/0x510 [25234.642861] worker_thread+0x188/0x340 [25234.646612] ? rescuer_thread+0x310/0x310 [25234.650621] kthread+0xf5/0x130 [25234.653758] ? kthread_complete_and_exit+0x20/0x20 [25234.658543] ret_from_fork+0x31/0x40 [25234.662124] ? kthread_complete_and_exit+0x20/0x20 [25234.666923] ret_from_fork_asm+0x11/0x20 [25234.670848] [25261.900226] watchdog: BUG: soft lockup - CPU#0 stuck for 12412s! [dbench:78683] [25261.907532] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25261.912226] watchdog: BUG: soft lockup - CPU#7 stuck for 21409s! [kworker/7:4:78808] [25261.962971] irq event stamp: 40546920 [25261.962972] hardirqs last enabled at (40546919): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25261.970704] Modules linked in: [25261.974360] hardirqs last disabled at (40546920): [] sysvec_apic_timer_interrupt+0xe/0x90 [25261.984517] ceph [25261.987568] softirqs last enabled at (40546872): [] __do_softirq+0x2be/0x371 [25261.997291] libceph [25261.999215] softirqs last disabled at (40546867): [] irq_exit_rcu+0xb0/0xd0 [25262.007907] dns_resolver fscache [25262.010093] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25262.018621] netfs veth [25262.021932] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25262.021933] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25262.030876] nft_chain_nat xt_MASQUERADE [25262.033338] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25262.040809] nf_nat nf_conntrack [25262.046286] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25262.050203] nf_defrag_ipv6 nf_defrag_ipv4 [25262.068941] [25262.068942] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25262.072165] nft_compat [25262.077383] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25262.081472] nf_tables nfnetlink [25262.082963] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25262.090087] bridge stp [25262.092531] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25262.099655] llc xfs [25262.102880] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25262.110022] libcrc32c sunrpc [25262.112465] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25262.119589] intel_rapl_msr intel_rapl_common [25262.121773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25262.128911] sb_edac x86_pkg_temp_thermal [25262.131879] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25262.139956] coretemp kvm_intel [25262.144307] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25262.150045] kvm irqbypass [25262.154048] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25262.161174] i2c_i801 iTCO_wdt [25262.164310] Call Trace: [25262.164312] [25262.171434] iTCO_vendor_support i2c_smbus [25262.174139] ? watchdog_timer_fn+0x1b4/0x220 [25262.181263] crct10dif_pclmul joydev [25262.184315] ? __hrtimer_run_queues+0x1d3/0x360 [25262.186756] crc32_pclmul ipmi_ssif [25262.188773] ? hrtimer_interrupt+0x104/0x240 [25262.192860] ghash_clmulni_intel mei_me [25262.197123] ? __local_bh_enable+0x50/0x60 [25262.200695] mei lpc_ich [25262.205221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25262.208703] sha512_ssse3 acpi_ipmi [25262.212967] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25262.216796] mfd_core wmi [25262.220905] [25262.223436] ipmi_si ipmi_devintf [25262.228652] [25262.232136] ipmi_msghandler ioatdma [25262.237094] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25262.239712] acpi_pad acpi_power_meter [25262.241814] ? queued_spin_lock_slowpath+0x18/0x2e0 [25262.245120] ixgbe igb [25262.247218] do_raw_spin_lock+0xb5/0xc0 [25262.250788] mdio crc32c_intel [25262.256093] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25262.259836] nvme ptp [25262.264708] try_get_cap_refs+0x159/0x880 [ceph] [25262.267064] nvme_core i2c_algo_bit [25262.270911] __ceph_get_caps+0x4a1/0x760 [ceph] [25262.273954] pps_core dca [25262.278742] ? __file_remove_privs+0x89/0x110 [25262.281010] fuse [25262.285626] ? wake_bit_function+0x70/0x70 [25262.289114] [25262.289114] irq event stamp: 124153840 [25262.293640] ? ceph_write_iter+0x260/0x770 [ceph] [25262.296254] hardirqs last enabled at (124153839): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25262.300624] ceph_write_iter+0x260/0x770 [ceph] [25262.302548] hardirqs last disabled at (124153840): [] sysvec_apic_timer_interrupt+0xe/0x90 [25262.306641] ? lock_is_held_type+0xaa/0x120 [25262.308129] softirqs last enabled at (124153838): [] __do_softirq+0x2be/0x371 [25262.311875] vfs_write+0x324/0x4f0 [25262.316570] softirqs last disabled at (124153833): [] irq_exit_rcu+0xb0/0xd0 [25262.326819] ksys_pwrite64+0x65/0xa0 [25262.331338] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25262.341149] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25262.345327] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25262.354098] do_syscall_64+0x38/0x80 [25262.357494] Workqueue: events delayed_work [ceph] [25262.366091] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25262.369661] [25262.369662] RIP: 0010:do_raw_spin_lock+0xd/0xc0 [25262.379041] RIP: 0033:0x7f48f0eeb93b [25262.384171] Code: 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 <48> 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 [25262.391641] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25262.395212] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [25262.399909] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25262.404955] [25262.404956] RAX: 00000000dead4ead RBX: ffff8881a1d34478 RCX: 000000005553a634 [25262.406445] ORIG_RAX: 0000000000000012 [25262.410969] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [25262.414539] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25262.433275] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [25262.452015] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25262.457232] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25262.462449] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25262.463939] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25262.471065] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25262.474910] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25262.482035] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25262.489160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25262.496290] [25262.503409] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25262.562171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25262.569293] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25262.576419] Call Trace: [25262.578863] [25262.580874] ? watchdog_timer_fn+0x1b4/0x220 [25262.585140] ? __hrtimer_run_queues+0x1d3/0x360 [25262.589665] ? hrtimer_interrupt+0x104/0x240 [25262.593933] ? __local_bh_enable+0x50/0x60 [25262.598027] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25262.603242] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25262.608199] [25262.610296] [25262.612395] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25262.617703] ? do_raw_spin_lock+0xd/0xc0 [25262.621633] igrab+0x1d/0x50 [25262.624513] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25262.629661] delayed_work+0x1d8/0x280 [ceph] [25262.633943] ? process_scheduled_works+0x22a/0x510 [25262.638733] process_scheduled_works+0x2a4/0x510 [25262.643349] worker_thread+0x188/0x340 [25262.647098] ? rescuer_thread+0x310/0x310 [25262.651102] kthread+0xf5/0x130 [25262.654237] ? kthread_complete_and_exit+0x20/0x20 [25262.659023] ret_from_fork+0x31/0x40 [25262.662593] ? kthread_complete_and_exit+0x20/0x20 [25262.667376] ret_from_fork_asm+0x11/0x20 [25262.671300] [25289.900711] watchdog: BUG: soft lockup - CPU#0 stuck for 12439s! [dbench:78683] [25289.908020] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25289.912709] watchdog: BUG: soft lockup - CPU#7 stuck for 21436s! [kworker/7:4:78808] [25289.963467] irq event stamp: 40605202 [25289.971200] Modules linked in: ceph libceph dns_resolver fscache netfs [25289.974857] hardirqs last enabled at (40605201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25289.981384] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [25289.991541] hardirqs last disabled at (40605202): [] sysvec_apic_timer_interrupt+0xe/0x90 [25289.997623] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [25290.007348] softirqs last enabled at (40605154): [] __do_softirq+0x2be/0x371 [25290.014731] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25290.023417] softirqs last disabled at (40605149): [] irq_exit_rcu+0xb0/0xd0 [25290.030289] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [25290.038802] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25290.044796] irqbypass i2c_i801 [25290.053741] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25290.056880] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [25290.064351] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25290.070694] joydev crc32_pclmul ipmi_ssif [25290.076171] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25290.076172] ghash_clmulni_intel [25290.080263] mei_me mei [25290.098999] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25290.102224] lpc_ich sha512_ssse3 [25290.104669] [25290.104669] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25290.109885] acpi_ipmi mfd_core [25290.113195] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25290.114686] wmi ipmi_si [25290.121810] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25290.124948] ipmi_devintf ipmi_msghandler [25290.132072] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25290.134603] ioatdma acpi_pad [25290.141727] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25290.145731] acpi_power_meter [25290.152854] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25290.155818] ixgbe igb [25290.162943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25290.165907] mdio crc32c_intel [25290.173986] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25290.176343] nvme ptp [25290.182081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25290.185147] nvme_core [25290.192273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25290.194544] i2c_algo_bit [25290.201666] Call Trace: [25290.201669] [25290.204024] pps_core [25290.211152] ? watchdog_timer_fn+0x1b4/0x220 [25290.213765] dca fuse [25290.216211] ? __hrtimer_run_queues+0x1d3/0x360 [25290.218220] [25290.218221] irq event stamp: 124375202 [25290.220495] ? hrtimer_interrupt+0x104/0x240 [25290.224756] hardirqs last enabled at (124375201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25290.227025] ? __local_bh_enable+0x50/0x60 [25290.231550] hardirqs last disabled at (124375202): [] sysvec_apic_timer_interrupt+0xe/0x90 [25290.233043] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25290.236784] softirqs last enabled at (124375200): [] __do_softirq+0x2be/0x371 [25290.241049] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25290.251292] softirqs last disabled at (124375195): [] irq_exit_rcu+0xb0/0xd0 [25290.255396] [25290.265203] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25290.270421] [25290.279192] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25290.284151] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25290.292748] Workqueue: events delayed_work [ceph] [25290.294848] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25290.304222] [25290.304223] RIP: 0010:lock_release+0x0/0x260 [25290.306320] do_raw_spin_lock+0xb5/0xc0 [25290.313791] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [25290.319111] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25290.323809] RSP: 0018:ffffc90002117d88 EFLAGS: 00000286 [25290.328682] try_get_cap_refs+0x159/0x880 [ceph] [25290.330169] [25290.330170] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [25290.334438] __ceph_get_caps+0x4a1/0x760 [ceph] [25290.338264] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [25290.357007] ? __file_remove_privs+0x89/0x110 [25290.361787] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [25290.367004] ? wake_bit_function+0x70/0x70 [25290.371615] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25290.373113] ? ceph_write_iter+0x260/0x770 [ceph] [25290.380237] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25290.384762] ceph_write_iter+0x260/0x770 [ceph] [25290.391885] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25290.396242] ? lock_is_held_type+0xaa/0x120 [25290.403362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25290.407455] vfs_write+0x324/0x4f0 [25290.414576] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25290.419279] ksys_pwrite64+0x65/0xa0 [25290.426398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25290.430922] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25290.438998] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25290.443178] do_syscall_64+0x38/0x80 [25290.448915] Call Trace: [25290.448916] [25290.452312] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25290.459437] ? watchdog_timer_fn+0x1b4/0x220 [25290.463008] RIP: 0033:0x7f48f0eeb93b [25290.470131] ? __hrtimer_run_queues+0x1d3/0x360 [25290.475262] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25290.482405] ? hrtimer_interrupt+0x104/0x240 [25290.485974] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25290.488417] ? __local_bh_enable+0x50/0x60 [25290.490427] ORIG_RAX: 0000000000000012 [25290.495476] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25290.499737] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25290.503309] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25290.507832] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25290.526570] [25290.530833] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25290.536051] [25290.536052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25290.540140] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25290.543975] ? igrab+0x43/0x50 [25290.549190] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25290.556315] ? lock_downgrade+0x250/0x250 [25290.561277] [25290.568396] ? _raw_spin_lock+0x30/0x40 [25290.612337] _raw_spin_unlock+0x1b/0x40 [25290.616168] igrab+0x43/0x50 [25290.619048] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25290.624197] delayed_work+0x1d8/0x280 [ceph] [25290.628477] ? process_scheduled_works+0x22a/0x510 [25290.633266] process_scheduled_works+0x2a4/0x510 [25290.637881] worker_thread+0x188/0x340 [25290.641631] ? rescuer_thread+0x310/0x310 [25290.645634] kthread+0xf5/0x130 [25290.648773] ? kthread_complete_and_exit+0x20/0x20 [25290.653558] ret_from_fork+0x31/0x40 [25290.657136] ? kthread_complete_and_exit+0x20/0x20 [25290.661920] ret_from_fork_asm+0x11/0x20 [25290.665853] [25291.886111] rcu: INFO: rcu_preempt self-detected stall on CPU [25291.891874] rcu: 7-....: (12969399 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/12968111 fqs=6523551 [25291.902472] rcu: (t=13355038 jiffies g=1309609 q=415878 ncpus=8) [25291.908557] Sending NMI from CPU 7 to CPUs 0: [25291.912911] NMI backtrace for cpu 0 [25291.912912] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25291.912914] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25291.912914] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25291.912917] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25291.912918] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25291.912920] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25291.912921] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25291.912921] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25291.912922] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25291.912923] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25291.912923] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25291.912925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25291.912925] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25291.912926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25291.912927] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25291.912928] Call Trace: [25291.912929] [25291.912930] ? nmi_cpu_backtrace+0xad/0x120 [25291.912933] ? nmi_cpu_backtrace_handler+0x11/0x20 [25291.912935] ? nmi_handle+0xd4/0x260 [25291.912939] ? default_do_nmi+0x49/0xf0 [25291.912941] ? exc_nmi+0x11e/0x150 [25291.912942] ? end_repeat_nmi+0x16/0x67 [25291.912946] ? queued_spin_lock_slowpath+0x16/0x2e0 [25291.912949] ? queued_spin_lock_slowpath+0x16/0x2e0 [25291.912951] ? queued_spin_lock_slowpath+0x16/0x2e0 [25291.912954] [25291.912954] [25291.912955] do_raw_spin_lock+0xb5/0xc0 [25291.912957] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25291.912971] try_get_cap_refs+0x159/0x880 [ceph] [25291.912987] __ceph_get_caps+0x4a1/0x760 [ceph] [25291.913004] ? __file_remove_privs+0x89/0x110 [25291.913006] ? wake_bit_function+0x70/0x70 [25291.913008] ? ceph_write_iter+0x260/0x770 [ceph] [25291.913021] ceph_write_iter+0x260/0x770 [ceph] [25291.913035] ? lock_is_held_type+0xaa/0x120 [25291.913038] vfs_write+0x324/0x4f0 [25291.913042] ksys_pwrite64+0x65/0xa0 [25291.913045] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25291.913047] do_syscall_64+0x38/0x80 [25291.913050] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25291.913052] RIP: 0033:0x7f48f0eeb93b [25291.913053] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25291.913054] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [25291.913055] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25291.913056] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25291.913057] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25291.913058] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25291.913058] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25291.913061] [25291.913907] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25292.217457] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25292.224929] Workqueue: events delayed_work [ceph] [25292.229644] RIP: 0010:lock_acquire+0xf0/0x2b0 [25292.233994] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25292.252733] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25292.257950] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25292.265074] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25292.272199] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25292.279322] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25292.286447] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25292.293570] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25292.301648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25292.307386] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25292.314509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25292.321634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25292.328757] Call Trace: [25292.331204] [25292.333214] ? rcu_dump_cpu_stacks+0xd1/0x180 [25292.337574] ? rcu_sched_clock_irq+0x633/0x1390 [25292.342098] ? lock_release+0xc5/0x260 [25292.345841] ? lock_is_held_type+0xaa/0x120 [25292.350018] ? lock_is_held_type+0xaa/0x120 [25292.354195] ? tick_sched_do_timer+0xa0/0xa0 [25292.358469] ? rcu_read_lock_sched_held+0x56/0x90 [25292.363166] ? tick_sched_do_timer+0xa0/0xa0 [25292.367429] ? update_process_times+0x5f/0x90 [25292.371782] ? tick_sched_handle.isra.16+0x1e/0x50 [25292.376573] ? tick_sched_timer+0x6f/0x80 [25292.380579] ? __hrtimer_run_queues+0x1d3/0x360 [25292.385106] ? hrtimer_interrupt+0x104/0x240 [25292.389373] ? __local_bh_enable+0x50/0x60 [25292.393468] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25292.398692] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25292.403649] [25292.405746] [25292.407843] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25292.413152] ? lock_acquire+0xf0/0x2b0 [25292.416902] ? igrab+0x43/0x50 [25292.419952] ? lock_release+0xc5/0x260 [25292.423697] _raw_spin_lock+0x30/0x40 [25292.427362] ? igrab+0x1d/0x50 [25292.430411] igrab+0x1d/0x50 [25292.433289] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25292.438438] delayed_work+0x1d8/0x280 [ceph] [25292.442721] ? process_scheduled_works+0x22a/0x510 [25292.447511] process_scheduled_works+0x2a4/0x510 [25292.452127] worker_thread+0x188/0x340 [25292.455876] ? rescuer_thread+0x310/0x310 [25292.459878] kthread+0xf5/0x130 [25292.463015] ? kthread_complete_and_exit+0x20/0x20 [25292.467801] ret_from_fork+0x31/0x40 [25292.471371] ? kthread_complete_and_exit+0x20/0x20 [25292.476155] ret_from_fork_asm+0x11/0x20 [25292.480087] [25317.901192] watchdog: BUG: soft lockup - CPU#0 stuck for 12465s! [dbench:78683] [25317.908496] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25317.913192] watchdog: BUG: soft lockup - CPU#7 stuck for 21462s! [kworker/7:4:78808] [25317.963937] irq event stamp: 40663302 [25317.971668] Modules linked in: ceph [25317.975327] hardirqs last enabled at (40663301): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25317.975329] libceph [25317.978809] dns_resolver fscache [25317.988968] hardirqs last disabled at (40663302): [] sysvec_apic_timer_interrupt+0xe/0x90 [25317.991151] netfs veth [25317.994462] softirqs last enabled at (40663262): [] __do_softirq+0x2be/0x371 [25318.004188] nft_chain_nat [25318.006632] softirqs last disabled at (40663257): [] irq_exit_rcu+0xb0/0xd0 [25318.015315] xt_MASQUERADE nf_nat [25318.018020] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25318.026531] nf_conntrack nf_defrag_ipv6 [25318.029840] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25318.038784] nf_defrag_ipv4 nft_compat [25318.042704] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25318.050174] nf_tables nfnetlink [25318.053916] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25318.059395] bridge [25318.062619] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25318.081356] stp llc [25318.083453] [25318.083453] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25318.088672] xfs libcrc32c [25318.090867] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25318.092355] sunrpc [25318.099479] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25318.102184] intel_rapl_msr intel_rapl_common [25318.109307] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25318.111405] sb_edac x86_pkg_temp_thermal [25318.118528] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25318.122881] coretemp kvm_intel [25318.130005] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25318.134006] kvm irqbypass [25318.141131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25318.144268] i2c_i801 iTCO_wdt [25318.152345] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25318.155052] iTCO_vendor_support i2c_smbus [25318.160789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25318.163839] crct10dif_pclmul joydev [25318.170963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25318.175052] crc32_pclmul [25318.182177] Call Trace: [25318.185748] ipmi_ssif ghash_clmulni_intel [25318.192874] [25318.195489] mei_me mei [25318.197934] ? watchdog_timer_fn+0x1b4/0x220 [25318.202025] lpc_ich sha512_ssse3 [25318.204037] ? __hrtimer_run_queues+0x1d3/0x360 [25318.206479] acpi_ipmi [25318.210748] ? hrtimer_interrupt+0x104/0x240 [25318.214054] mfd_core wmi [25318.218591] ? __local_bh_enable+0x50/0x60 [25318.220945] ipmi_si [25318.225210] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25318.227826] ipmi_devintf ipmi_msghandler [25318.231918] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25318.234100] ioatdma [25318.239319] [25318.243322] acpi_pad [25318.248280] [25318.248281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25318.250463] acpi_power_meter [25318.252568] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25318.254831] ixgbe igb [25318.256932] do_raw_spin_lock+0xb5/0xc0 [25318.262234] mdio crc32c_intel [25318.265198] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25318.270067] nvme ptp [25318.272428] try_get_cap_refs+0x159/0x880 [ceph] [25318.276257] nvme_core i2c_algo_bit [25318.279312] __ceph_get_caps+0x4a1/0x760 [ceph] [25318.284091] pps_core dca [25318.286366] ? __file_remove_privs+0x89/0x110 [25318.290972] fuse [25318.294457] ? wake_bit_function+0x70/0x70 [25318.298982] irq event stamp: 124591504 [25318.301601] ? ceph_write_iter+0x260/0x770 [ceph] [25318.305949] hardirqs last enabled at (124591503): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25318.307878] ceph_write_iter+0x260/0x770 [ceph] [25318.311973] hardirqs last disabled at (124591504): [] sysvec_apic_timer_interrupt+0xe/0x90 [25318.315719] ? lock_is_held_type+0xaa/0x120 [25318.320413] softirqs last enabled at (124591502): [] __do_softirq+0x2be/0x371 [25318.330662] vfs_write+0x324/0x4f0 [25318.335181] softirqs last disabled at (124591497): [] irq_exit_rcu+0xb0/0xd0 [25318.344998] ksys_pwrite64+0x65/0xa0 [25318.349171] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25318.357941] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25318.361340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25318.369938] do_syscall_64+0x38/0x80 [25318.373508] Workqueue: events delayed_work [ceph] [25318.382884] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25318.388016] [25318.388017] RIP: 0010:lock_acquire+0xf0/0x2b0 [25318.395486] RIP: 0033:0x7f48f0eeb93b [25318.399057] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25318.403755] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25318.408798] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25318.410287] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25318.414639] [25318.414640] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25318.418209] ORIG_RAX: 0000000000000012 [25318.436947] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25318.455684] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25318.460902] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25318.466120] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25318.467610] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25318.474735] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25318.478566] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25318.485688] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25318.492815] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25318.499937] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25318.507063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25318.514191] [25318.521309] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25318.565807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25318.572929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25318.580055] Call Trace: [25318.582498] [25318.584511] ? watchdog_timer_fn+0x1b4/0x220 [25318.588775] ? __hrtimer_run_queues+0x1d3/0x360 [25318.593301] ? hrtimer_interrupt+0x104/0x240 [25318.597569] ? __local_bh_enable+0x50/0x60 [25318.601682] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25318.606904] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25318.611878] [25318.613976] [25318.616073] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25318.621383] ? lock_acquire+0xf0/0x2b0 [25318.625132] ? igrab+0x43/0x50 [25318.628181] ? lock_release+0xc5/0x260 [25318.631928] _raw_spin_lock+0x30/0x40 [25318.635600] ? igrab+0x1d/0x50 [25318.638652] igrab+0x1d/0x50 [25318.641529] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25318.646680] delayed_work+0x1d8/0x280 [ceph] [25318.650959] ? process_scheduled_works+0x22a/0x510 [25318.655741] process_scheduled_works+0x2a4/0x510 [25318.660357] worker_thread+0x188/0x340 [25318.664106] ? rescuer_thread+0x310/0x310 [25318.668109] kthread+0xf5/0x130 [25318.671246] ? kthread_complete_and_exit+0x20/0x20 [25318.676032] ret_from_fork+0x31/0x40 [25318.679602] ? kthread_complete_and_exit+0x20/0x20 [25318.684387] ret_from_fork_asm+0x11/0x20 [25318.688320] [25345.901675] watchdog: BUG: soft lockup - CPU#0 stuck for 12491s! [dbench:78683] [25345.908976] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25345.913675] watchdog: BUG: soft lockup - CPU#7 stuck for 21488s! [kworker/7:4:78808] [25345.964418] irq event stamp: 40721478 [25345.964420] hardirqs last enabled at (40721477): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25345.972147] Modules linked in: ceph [25345.975804] hardirqs last disabled at (40721478): [] sysvec_apic_timer_interrupt+0xe/0x90 [25345.985964] libceph [25345.989448] softirqs last enabled at (40721440): [] __do_softirq+0x2be/0x371 [25345.999171] dns_resolver fscache [25346.001362] softirqs last disabled at (40721435): [] irq_exit_rcu+0xb0/0xd0 [25346.010048] netfs [25346.013376] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25346.021888] veth nft_chain_nat [25346.023898] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25346.032842] xt_MASQUERADE nf_nat [25346.035980] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25346.043450] nf_conntrack nf_defrag_ipv6 [25346.046761] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25346.052238] nf_defrag_ipv4 nft_compat [25346.056156] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25346.074892] nf_tables nfnetlink [25346.078636] [25346.078637] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25346.083854] bridge [25346.087079] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25346.088570] stp llc [25346.095694] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25346.097792] xfs libcrc32c [25346.104916] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25346.107098] sunrpc intel_rapl_msr [25346.114224] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25346.116927] intel_rapl_common [25346.124070] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25346.127465] sb_edac x86_pkg_temp_thermal [25346.134591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25346.137640] coretemp kvm_intel [25346.145718] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25346.149721] kvm irqbypass [25346.155459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25346.158597] i2c_i801 iTCO_wdt [25346.165723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25346.168426] iTCO_vendor_support i2c_smbus [25346.175550] Call Trace: [25346.175551] [25346.178601] crct10dif_pclmul [25346.185726] ? watchdog_timer_fn+0x1b4/0x220 [25346.189816] joydev crc32_pclmul [25346.192261] ? __hrtimer_run_queues+0x1d3/0x360 [25346.194271] ipmi_ssif [25346.197237] ? hrtimer_interrupt+0x104/0x240 [25346.201498] ghash_clmulni_intel mei_me [25346.204723] ? __local_bh_enable+0x50/0x60 [25346.209245] mei lpc_ich [25346.211608] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25346.215868] sha512_ssse3 [25346.219700] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25346.223791] acpi_ipmi mfd_core [25346.226320] [25346.231537] wmi ipmi_si [25346.234155] [25346.234156] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25346.239112] ipmi_devintf [25346.242255] ? queued_spin_lock_slowpath+0x16/0x2e0 [25346.244355] ipmi_msghandler [25346.246888] do_raw_spin_lock+0xb5/0xc0 [25346.248983] ioatdma acpi_pad [25346.254289] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25346.256906] acpi_power_meter ixgbe [25346.261777] try_get_cap_refs+0x159/0x880 [ceph] [25346.264653] igb mdio [25346.268490] __ceph_get_caps+0x4a1/0x760 [ceph] [25346.271449] crc32c_intel nvme [25346.276236] ? __file_remove_privs+0x89/0x110 [25346.279717] ptp nvme_core [25346.284327] ? wake_bit_function+0x70/0x70 [25346.286597] i2c_algo_bit [25346.291124] ? ceph_write_iter+0x260/0x770 [ceph] [25346.294172] pps_core dca [25346.298523] ceph_write_iter+0x260/0x770 [ceph] [25346.301229] fuse [25346.305321] ? lock_is_held_type+0xaa/0x120 [25346.307935] irq event stamp: 124812412 [25346.312636] vfs_write+0x324/0x4f0 [25346.315251] hardirqs last enabled at (124812411): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25346.319778] ksys_pwrite64+0x65/0xa0 [25346.321699] hardirqs last disabled at (124812412): [] sysvec_apic_timer_interrupt+0xe/0x90 [25346.325875] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25346.329620] softirqs last enabled at (124812410): [] __do_softirq+0x2be/0x371 [25346.333019] do_syscall_64+0x38/0x80 [25346.343261] softirqs last disabled at (124812405): [] irq_exit_rcu+0xb0/0xd0 [25346.346833] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25346.356642] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25346.361773] RIP: 0033:0x7f48f0eeb93b [25346.370544] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25346.374116] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25346.382714] Workqueue: events delayed_work [ceph] [25346.387756] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25346.397134] [25346.397135] RIP: 0010:lock_acquire+0xf0/0x2b0 [25346.400705] ORIG_RAX: 0000000000000012 [25346.408175] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25346.426915] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25346.431612] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25346.436828] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25346.438319] [25346.438319] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25346.442671] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25346.446500] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25346.465239] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25346.472362] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25346.477581] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25346.484704] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25346.486201] [25346.493320] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25346.545381] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25346.553458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25346.559197] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25346.566320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25346.573443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25346.580569] Call Trace: [25346.583014] [25346.585023] ? watchdog_timer_fn+0x1b4/0x220 [25346.589288] ? __hrtimer_run_queues+0x1d3/0x360 [25346.593814] ? hrtimer_interrupt+0x104/0x240 [25346.598082] ? __local_bh_enable+0x50/0x60 [25346.602178] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25346.607401] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25346.612371] [25346.614464] [25346.616563] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25346.621870] ? lock_acquire+0xf0/0x2b0 [25346.625620] ? igrab+0x43/0x50 [25346.628670] ? lock_release+0xc5/0x260 [25346.632416] _raw_spin_lock+0x30/0x40 [25346.636083] ? igrab+0x1d/0x50 [25346.639139] igrab+0x1d/0x50 [25346.642018] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25346.647166] delayed_work+0x1d8/0x280 [ceph] [25346.651447] ? process_scheduled_works+0x22a/0x510 [25346.656230] process_scheduled_works+0x2a4/0x510 [25346.660845] worker_thread+0x188/0x340 [25346.664595] ? rescuer_thread+0x310/0x310 [25346.668597] kthread+0xf5/0x130 [25346.671736] ? kthread_complete_and_exit+0x20/0x20 [25346.676519] ret_from_fork+0x31/0x40 [25346.680089] ? kthread_complete_and_exit+0x20/0x20 [25346.684875] ret_from_fork_asm+0x11/0x20 [25346.688797] [25373.902159] watchdog: BUG: soft lockup - CPU#0 stuck for 12517s! [dbench:78683] [25373.909466] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25373.914159] watchdog: BUG: soft lockup - CPU#7 stuck for 21514s! [kworker/7:4:78808] [25373.964914] irq event stamp: 40779694 [25373.972645] Modules linked in: ceph libceph dns_resolver fscache netfs [25373.976302] hardirqs last enabled at (40779693): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25373.982833] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [25373.992994] hardirqs last disabled at (40779694): [] sysvec_apic_timer_interrupt+0xe/0x90 [25374.000378] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [25374.010102] softirqs last enabled at (40779656): [] __do_softirq+0x2be/0x371 [25374.016549] llc xfs libcrc32c sunrpc intel_rapl_msr [25374.025236] softirqs last disabled at (40779651): [] irq_exit_rcu+0xb0/0xd0 [25374.030192] intel_rapl_common sb_edac x86_pkg_temp_thermal [25374.038704] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25374.044266] coretemp kvm_intel [25374.053212] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25374.053214] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25374.056348] kvm irqbypass [25374.063822] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25374.069296] i2c_i801 iTCO_wdt [25374.072000] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25374.090740] iTCO_vendor_support i2c_smbus [25374.093789] [25374.093790] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25374.099008] crct10dif_pclmul [25374.103097] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25374.104588] joydev [25374.111713] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25374.114675] crc32_pclmul ipmi_ssif [25374.121802] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25374.123898] ghash_clmulni_intel mei_me [25374.131023] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25374.134507] mei [25374.141631] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25374.145463] lpc_ich [25374.152585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25374.154424] sha512_ssse3 acpi_ipmi [25374.162499] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25374.164685] mfd_core wmi [25374.170423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25374.173906] ipmi_si ipmi_devintf [25374.181031] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25374.183647] ipmi_msghandler ioatdma [25374.190770] Call Trace: [25374.190772] [25374.194081] acpi_pad [25374.201209] ? watchdog_timer_fn+0x1b4/0x220 [25374.204777] acpi_power_meter [25374.207222] ? __hrtimer_run_queues+0x1d3/0x360 [25374.209233] ixgbe [25374.211507] ? hrtimer_interrupt+0x104/0x240 [25374.215768] igb [25374.218730] ? __local_bh_enable+0x50/0x60 [25374.223256] mdio [25374.225268] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25374.229529] crc32c_intel [25374.231368] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25374.235457] nvme [25374.237381] [25374.242599] ptp [25374.245215] [25374.245216] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25374.250175] nvme_core [25374.252102] ? queued_spin_lock_slowpath+0x16/0x2e0 [25374.254195] i2c_algo_bit [25374.256034] do_raw_spin_lock+0xb5/0xc0 [25374.258130] pps_core dca [25374.263435] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25374.265792] fuse [25374.270666] try_get_cap_refs+0x159/0x880 [ceph] [25374.273280] irq event stamp: 125033312 [25374.277115] __ceph_get_caps+0x4a1/0x760 [ceph] [25374.279728] hardirqs last enabled at (125033311): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25374.284516] ? __file_remove_privs+0x89/0x110 [25374.286435] hardirqs last disabled at (125033312): [] sysvec_apic_timer_interrupt+0xe/0x90 [25374.291048] ? wake_bit_function+0x70/0x70 [25374.294791] softirqs last enabled at (125033310): [] __do_softirq+0x2be/0x371 [25374.299317] ? ceph_write_iter+0x260/0x770 [ceph] [25374.309558] softirqs last disabled at (125033305): [] irq_exit_rcu+0xb0/0xd0 [25374.313909] ceph_write_iter+0x260/0x770 [ceph] [25374.323722] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25374.327816] ? lock_is_held_type+0xaa/0x120 [25374.336581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25374.341281] vfs_write+0x324/0x4f0 [25374.349877] Workqueue: events delayed_work [ceph] [25374.354406] ksys_pwrite64+0x65/0xa0 [25374.363780] [25374.363781] RIP: 0010:lock_acquire+0xf0/0x2b0 [25374.367956] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25374.375426] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25374.378826] do_syscall_64+0x38/0x80 [25374.383521] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25374.387095] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25374.388583] [25374.388584] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25374.392936] RIP: 0033:0x7f48f0eeb93b [25374.398065] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25374.416804] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25374.420374] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25374.425590] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25374.430635] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25374.432126] ORIG_RAX: 0000000000000012 [25374.439249] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25374.442820] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25374.449962] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25374.468698] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25374.475823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25374.481040] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25374.488164] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25374.491995] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25374.499120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25374.506245] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25374.514321] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25374.521452] [25374.527183] Call Trace: [25374.527185] [25374.576576] ? watchdog_timer_fn+0x1b4/0x220 [25374.580843] ? __hrtimer_run_queues+0x1d3/0x360 [25374.585376] ? hrtimer_interrupt+0x104/0x240 [25374.589645] ? __local_bh_enable+0x50/0x60 [25374.593740] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25374.598963] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25374.603922] [25374.606016] [25374.608115] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25374.613422] ? lock_acquire+0xf0/0x2b0 [25374.617173] ? igrab+0x43/0x50 [25374.620222] ? lock_release+0xc5/0x260 [25374.623967] _raw_spin_lock+0x30/0x40 [25374.627623] ? igrab+0x1d/0x50 [25374.630675] igrab+0x1d/0x50 [25374.633566] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25374.638721] delayed_work+0x1d8/0x280 [ceph] [25374.643002] ? process_scheduled_works+0x22a/0x510 [25374.647791] process_scheduled_works+0x2a4/0x510 [25374.652408] worker_thread+0x188/0x340 [25374.656157] ? rescuer_thread+0x310/0x310 [25374.660167] kthread+0xf5/0x130 [25374.663304] ? kthread_complete_and_exit+0x20/0x20 [25374.668091] ret_from_fork+0x31/0x40 [25374.671669] ? kthread_complete_and_exit+0x20/0x20 [25374.676452] ret_from_fork_asm+0x11/0x20 [25374.680386] [25401.902643] watchdog: BUG: soft lockup - CPU#0 stuck for 12543s! [dbench:78683] [25401.909943] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25401.914643] watchdog: BUG: soft lockup - CPU#7 stuck for 21540s! [kworker/7:4:78808] [25401.965385] irq event stamp: 40837846 [25401.965386] hardirqs last enabled at (40837845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25401.973116] Modules linked in: [25401.976772] hardirqs last disabled at (40837846): [] sysvec_apic_timer_interrupt+0xe/0x90 [25401.986931] ceph [25401.989980] softirqs last enabled at (40837808): [] __do_softirq+0x2be/0x371 [25401.999706] libceph [25402.001629] softirqs last disabled at (40837803): [] irq_exit_rcu+0xb0/0xd0 [25402.010317] dns_resolver [25402.012505] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25402.021035] fscache netfs [25402.023652] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25402.023653] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25402.032596] veth nft_chain_nat [25402.035316] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25402.042787] xt_MASQUERADE nf_nat [25402.048265] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25402.051402] nf_conntrack nf_defrag_ipv6 [25402.070140] [25402.070141] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25402.073450] nf_defrag_ipv4 nft_compat [25402.078668] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25402.082586] nf_tables [25402.084076] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25402.091200] nfnetlink [25402.094946] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25402.102068] bridge stp [25402.104426] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25402.111551] llc xfs [25402.113908] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25402.121041] libcrc32c [25402.123485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25402.130610] sunrpc [25402.132793] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25402.140869] intel_rapl_msr [25402.143227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25402.148965] intel_rapl_common sb_edac [25402.151063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25402.158187] x86_pkg_temp_thermal [25402.160977] Call Trace: [25402.160979] [25402.168102] coretemp kvm_intel [25402.171845] ? watchdog_timer_fn+0x1b4/0x220 [25402.178969] kvm irqbypass [25402.182281] ? __hrtimer_run_queues+0x1d3/0x360 [25402.184724] i2c_i801 iTCO_wdt [25402.186737] ? hrtimer_interrupt+0x104/0x240 [25402.189872] iTCO_vendor_support i2c_smbus [25402.194135] ? __local_bh_enable+0x50/0x60 [25402.196841] crct10dif_pclmul joydev [25402.201367] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25402.204416] crc32_pclmul ipmi_ssif [25402.208680] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25402.212771] ghash_clmulni_intel mei_me [25402.216861] [25402.220431] mei lpc_ich [25402.225650] [25402.225651] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25402.229132] sha512_ssse3 acpi_ipmi [25402.234095] ? queued_spin_lock_slowpath+0x18/0x2e0 [25402.237922] mfd_core wmi [25402.240039] do_raw_spin_lock+0xb5/0xc0 [25402.242567] ipmi_si ipmi_devintf [25402.244665] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25402.249969] ipmi_msghandler ioatdma [25402.253454] try_get_cap_refs+0x159/0x880 [ceph] [25402.258322] acpi_pad acpi_power_meter [25402.260944] __ceph_get_caps+0x4a1/0x760 [ceph] [25402.264772] ixgbe igb [25402.268087] ? __file_remove_privs+0x89/0x110 [25402.272866] mdio crc32c_intel [25402.276438] ? wake_bit_function+0x70/0x70 [25402.281047] nvme [25402.284794] ? ceph_write_iter+0x260/0x770 [ceph] [25402.289314] ptp nvme_core [25402.291692] ceph_write_iter+0x260/0x770 [ceph] [25402.296044] i2c_algo_bit pps_core [25402.299096] ? lock_is_held_type+0xaa/0x120 [25402.303183] dca [25402.305110] vfs_write+0x324/0x4f0 [25402.309803] fuse [25402.312512] ksys_pwrite64+0x65/0xa0 [25402.317044] [25402.317044] irq event stamp: 125254628 [25402.320438] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25402.324616] hardirqs last enabled at (125254627): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25402.326455] do_syscall_64+0x38/0x80 [25402.329850] hardirqs last disabled at (125254628): [] sysvec_apic_timer_interrupt+0xe/0x90 [25402.331776] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25402.335344] softirqs last enabled at (125254626): [] __do_softirq+0x2be/0x371 [25402.336835] RIP: 0033:0x7f48f0eeb93b [25402.340580] softirqs last disabled at (125254621): [] irq_exit_rcu+0xb0/0xd0 [25402.345711] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25402.355954] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25402.359527] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25402.369340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25402.374381] ORIG_RAX: 0000000000000012 [25402.383151] Workqueue: events delayed_work [ceph] [25402.386722] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25402.395322] [25402.395323] RIP: 0010:lock_acquire+0xf0/0x2b0 [25402.414058] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25402.423434] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25402.428652] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25402.436122] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25402.439952] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25402.444651] [25402.444651] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25402.451776] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25402.453267] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25402.457621] [25402.464739] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25402.535117] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25402.542240] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25402.549365] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25402.557440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25402.563179] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25402.570304] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25402.577444] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25402.584568] Call Trace: [25402.587015] [25402.589024] ? watchdog_timer_fn+0x1b4/0x220 [25402.593305] ? __hrtimer_run_queues+0x1d3/0x360 [25402.597848] ? hrtimer_interrupt+0x104/0x240 [25402.602110] ? __local_bh_enable+0x50/0x60 [25402.606203] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25402.611418] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25402.616375] [25402.618475] [25402.620571] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25402.625880] ? lock_acquire+0xf0/0x2b0 [25402.629629] ? igrab+0x43/0x50 [25402.632680] ? lock_release+0xc5/0x260 [25402.636425] _raw_spin_lock+0x30/0x40 [25402.640089] ? igrab+0x1d/0x50 [25402.643139] igrab+0x1d/0x50 [25402.646020] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25402.651184] delayed_work+0x1d8/0x280 [ceph] [25402.655466] ? process_scheduled_works+0x22a/0x510 [25402.660257] process_scheduled_works+0x2a4/0x510 [25402.664871] worker_thread+0x188/0x340 [25402.668621] ? rescuer_thread+0x310/0x310 [25402.672624] kthread+0xf5/0x130 [25402.675761] ? kthread_complete_and_exit+0x20/0x20 [25402.680545] ret_from_fork+0x31/0x40 [25402.684116] ? kthread_complete_and_exit+0x20/0x20 [25402.688900] ret_from_fork_asm+0x11/0x20 [25402.692823] [25429.903129] watchdog: BUG: soft lockup - CPU#0 stuck for 12569s! [dbench:78683] [25429.910432] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25429.915128] watchdog: BUG: soft lockup - CPU#7 stuck for 21566s! [kworker/7:4:78808] [25429.965881] irq event stamp: 40896140 [25429.973611] Modules linked in: ceph libceph dns_resolver fscache netfs [25429.977269] hardirqs last enabled at (40896139): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25429.983809] veth nft_chain_nat xt_MASQUERADE nf_nat [25429.993970] hardirqs last disabled at (40896140): [] sysvec_apic_timer_interrupt+0xe/0x90 [25429.998928] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [25430.008651] softirqs last enabled at (40896114): [] __do_softirq+0x2be/0x371 [25430.016554] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [25430.025238] softirqs last disabled at (40896109): [] irq_exit_rcu+0xb0/0xd0 [25430.031149] intel_rapl_common sb_edac x86_pkg_temp_thermal [25430.039661] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25430.039662] coretemp [25430.039663] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25430.045225] kvm_intel [25430.054169] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25430.056441] kvm irqbypass [25430.063911] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25430.066267] i2c_i801 iTCO_wdt [25430.071745] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25430.074449] iTCO_vendor_support i2c_smbus [25430.093186] [25430.093187] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25430.096238] crct10dif_pclmul [25430.101455] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25430.105546] joydev [25430.107035] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25430.114160] crc32_pclmul ipmi_ssif [25430.117125] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25430.124250] ghash_clmulni_intel mei_me [25430.126347] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25430.133473] mei [25430.136955] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25430.144080] lpc_ich [25430.147909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25430.155034] sha512_ssse3 acpi_ipmi [25430.156871] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25430.164948] mfd_core wmi [25430.167132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25430.172870] ipmi_si ipmi_devintf [25430.176353] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25430.183477] ipmi_msghandler ioatdma [25430.186095] Call Trace: [25430.193219] acpi_pad acpi_power_meter [25430.196532] [25430.203655] ixgbe igb [25430.207229] ? watchdog_timer_fn+0x1b4/0x220 [25430.209670] mdio crc32c_intel [25430.213413] ? __hrtimer_run_queues+0x1d3/0x360 [25430.215425] nvme [25430.217803] ? hrtimer_interrupt+0x104/0x240 [25430.222063] ptp nvme_core [25430.225112] ? __local_bh_enable+0x50/0x60 [25430.229638] i2c_algo_bit pps_core [25430.231566] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25430.235825] dca fuse [25430.238529] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25430.242622] [25430.242622] irq event stamp: 125475552 [25430.246017] [25430.251234] hardirqs last enabled at (125475551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25430.253524] [25430.258482] hardirqs last disabled at (125475552): [] sysvec_apic_timer_interrupt+0xe/0x90 [25430.259972] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25430.263716] softirqs last enabled at (125475550): [] __do_softirq+0x2be/0x371 [25430.265816] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25430.276058] softirqs last disabled at (125475545): [] irq_exit_rcu+0xb0/0xd0 [25430.278157] do_raw_spin_lock+0xb5/0xc0 [25430.287964] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25430.293273] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25430.302039] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25430.306915] try_get_cap_refs+0x159/0x880 [ceph] [25430.315526] Workqueue: events delayed_work [ceph] [25430.319363] __ceph_get_caps+0x4a1/0x760 [ceph] [25430.328735] [25430.328736] RIP: 0010:lock_acquire+0xf0/0x2b0 [25430.333532] ? __file_remove_privs+0x89/0x110 [25430.340997] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25430.345610] ? wake_bit_function+0x70/0x70 [25430.350306] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25430.354833] ? ceph_write_iter+0x260/0x770 [ceph] [25430.356321] [25430.356322] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25430.360672] ceph_write_iter+0x260/0x770 [ceph] [25430.365022] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25430.383765] ? lock_is_held_type+0xaa/0x120 [25430.387849] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25430.393071] vfs_write+0x324/0x4f0 [25430.397766] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25430.399262] ksys_pwrite64+0x65/0xa0 [25430.406380] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25430.410904] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25430.418029] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25430.422208] do_syscall_64+0x38/0x80 [25430.429330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25430.432728] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25430.439850] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25430.443424] RIP: 0033:0x7f48f0eeb93b [25430.450546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25430.455678] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25430.463754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25430.467325] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25430.473062] Call Trace: [25430.473063] [25430.478106] ORIG_RAX: 0000000000000012 [25430.485232] ? watchdog_timer_fn+0x1b4/0x220 [25430.488811] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25430.495935] ? __hrtimer_run_queues+0x1d3/0x360 [25430.514671] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25430.521814] ? hrtimer_interrupt+0x104/0x240 [25430.527031] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25430.529474] ? __local_bh_enable+0x50/0x60 [25430.531486] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25430.535320] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25430.539581] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25430.546706] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25430.551233] [25430.558354] [25430.602538] [25430.604634] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25430.609943] ? lock_acquire+0xf0/0x2b0 [25430.613692] ? igrab+0x43/0x50 [25430.616743] ? lock_release+0xc5/0x260 [25430.620486] _raw_spin_lock+0x30/0x40 [25430.624144] ? igrab+0x1d/0x50 [25430.627193] igrab+0x1d/0x50 [25430.630074] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25430.635221] delayed_work+0x1d8/0x280 [ceph] [25430.639503] ? process_scheduled_works+0x22a/0x510 [25430.644294] process_scheduled_works+0x2a4/0x510 [25430.648910] worker_thread+0x188/0x340 [25430.652659] ? rescuer_thread+0x310/0x310 [25430.656662] kthread+0xf5/0x130 [25430.659799] ? kthread_complete_and_exit+0x20/0x20 [25430.664584] ret_from_fork+0x31/0x40 [25430.668163] ? kthread_complete_and_exit+0x20/0x20 [25430.672946] ret_from_fork_asm+0x11/0x20 [25430.676871] [25457.903616] watchdog: BUG: soft lockup - CPU#0 stuck for 12595s! [dbench:78683] [25457.910918] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25457.915616] watchdog: BUG: soft lockup - CPU#7 stuck for 21592s! [kworker/7:4:78808] [25457.966359] irq event stamp: 40954228 [25457.966360] hardirqs last enabled at (40954227): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25457.974089] Modules linked in: [25457.977746] hardirqs last disabled at (40954228): [] sysvec_apic_timer_interrupt+0xe/0x90 [25457.987903] ceph [25457.990956] softirqs last enabled at (40954190): [] __do_softirq+0x2be/0x371 [25458.000679] libceph [25458.002604] softirqs last disabled at (40954185): [] irq_exit_rcu+0xb0/0xd0 [25458.011300] dns_resolver fscache [25458.013481] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25458.022007] netfs [25458.025320] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25458.025321] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25458.034262] veth [25458.036273] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25458.043744] nft_chain_nat xt_MASQUERADE [25458.049222] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25458.051146] nf_nat [25458.069885] [25458.069886] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25458.073803] nf_conntrack nf_defrag_ipv6 [25458.079018] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25458.081115] nf_defrag_ipv4 [25458.082607] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25458.089730] nft_compat nf_tables [25458.093649] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25458.100772] nfnetlink bridge [25458.103562] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25458.110687] stp llc [25458.114015] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25458.121139] xfs libcrc32c [25458.124103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25458.131227] sunrpc [25458.133412] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25458.141488] intel_rapl_msr intel_rapl_common [25458.144193] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25458.149932] sb_edac [25458.152028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25458.159153] x86_pkg_temp_thermal coretemp [25458.163503] Call Trace: [25458.163504] [25458.170627] kvm_intel [25458.172812] ? watchdog_timer_fn+0x1b4/0x220 [25458.179934] kvm [25458.184027] ? __hrtimer_run_queues+0x1d3/0x360 [25458.186469] irqbypass [25458.188484] ? hrtimer_interrupt+0x104/0x240 [25458.190839] i2c_i801 iTCO_wdt [25458.195101] ? __local_bh_enable+0x50/0x60 [25458.196939] iTCO_vendor_support i2c_smbus [25458.201466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25458.203820] crct10dif_pclmul [25458.208085] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25458.211136] joydev [25458.215227] [25458.219318] crc32_pclmul ipmi_ssif [25458.224535] [25458.224536] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25458.227498] ghash_clmulni_intel [25458.232461] ? queued_spin_lock_slowpath+0x16/0x2e0 [25458.234553] mei_me [25458.236652] do_raw_spin_lock+0xb5/0xc0 [25458.240135] mei lpc_ich [25458.242234] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25458.247538] sha512_ssse3 acpi_ipmi [25458.250762] try_get_cap_refs+0x159/0x880 [ceph] [25458.255631] mfd_core wmi [25458.257734] __ceph_get_caps+0x4a1/0x760 [ceph] [25458.261560] ipmi_si ipmi_devintf [25458.264095] ? __file_remove_privs+0x89/0x110 [25458.268874] ipmi_msghandler ioatdma [25458.272360] ? wake_bit_function+0x70/0x70 [25458.276969] acpi_pad acpi_power_meter [25458.279588] ? ceph_write_iter+0x260/0x770 [ceph] [25458.284111] ixgbe igb [25458.287422] ceph_write_iter+0x260/0x770 [ceph] [25458.291773] mdio crc32c_intel [25458.295346] ? lock_is_held_type+0xaa/0x120 [25458.299433] nvme ptp [25458.303181] vfs_write+0x324/0x4f0 [25458.307876] nvme_core [25458.310236] ksys_pwrite64+0x65/0xa0 [25458.314757] i2c_algo_bit pps_core [25458.317806] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25458.321984] dca fuse [25458.324256] do_syscall_64+0x38/0x80 [25458.327654] [25458.327655] irq event stamp: 125696826 [25458.330017] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25458.333589] hardirqs last enabled at (125696825): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25458.336987] RIP: 0033:0x7f48f0eeb93b [25458.342117] hardirqs last disabled at (125696826): [] sysvec_apic_timer_interrupt+0xe/0x90 [25458.344388] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25458.347958] softirqs last enabled at (125696824): [] __do_softirq+0x2be/0x371 [25458.349451] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25458.353193] softirqs last disabled at (125696819): [] irq_exit_rcu+0xb0/0xd0 [25458.358238] ORIG_RAX: 0000000000000012 [25458.368483] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25458.372053] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25458.381863] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25458.400601] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25458.409371] Workqueue: events delayed_work [ceph] [25458.414589] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25458.423187] [25458.423189] RIP: 0010:lock_acquire+0xf0/0x2b0 [25458.427019] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25458.436395] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25458.443519] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25458.450992] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25458.458138] [25458.462829] [25458.462830] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25458.524814] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25458.531939] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25458.539062] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25458.546186] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25458.553311] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25458.561389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25458.567127] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25458.574251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25458.581375] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25458.588498] Call Trace: [25458.590945] [25458.592954] ? watchdog_timer_fn+0x1b4/0x220 [25458.597218] ? __hrtimer_run_queues+0x1d3/0x360 [25458.601744] ? hrtimer_interrupt+0x104/0x240 [25458.606020] ? __local_bh_enable+0x50/0x60 [25458.610117] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25458.615341] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25458.620307] [25458.622403] [25458.624502] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25458.629809] ? lock_acquire+0xf0/0x2b0 [25458.633558] ? igrab+0x43/0x50 [25458.636609] ? lock_release+0xc5/0x260 [25458.640355] _raw_spin_lock+0x30/0x40 [25458.644022] ? igrab+0x1d/0x50 [25458.647077] igrab+0x1d/0x50 [25458.649957] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25458.655106] delayed_work+0x1d8/0x280 [ceph] [25458.659386] ? process_scheduled_works+0x22a/0x510 [25458.664177] process_scheduled_works+0x2a4/0x510 [25458.668791] worker_thread+0x188/0x340 [25458.672541] ? rescuer_thread+0x310/0x310 [25458.676545] kthread+0xf5/0x130 [25458.679684] ? kthread_complete_and_exit+0x20/0x20 [25458.684467] ret_from_fork+0x31/0x40 [25458.688045] ? kthread_complete_and_exit+0x20/0x20 [25458.692830] ret_from_fork_asm+0x11/0x20 [25458.696752] [25485.904105] watchdog: BUG: soft lockup - CPU#0 stuck for 12621s! [dbench:78683] [25485.911406] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25485.916104] watchdog: BUG: soft lockup - CPU#7 stuck for 21618s! [kworker/7:4:78808] [25485.966856] irq event stamp: 41012562 [25485.974585] Modules linked in: ceph libceph dns_resolver fscache netfs [25485.978243] hardirqs last enabled at (41012561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25485.978248] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [25485.994934] hardirqs last disabled at (41012562): [] sysvec_apic_timer_interrupt+0xe/0x90 [25486.003618] nft_compat nf_tables nfnetlink bridge stp llc xfs [25486.013345] softirqs last enabled at (41012524): [] __do_softirq+0x2be/0x371 [25486.019177] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [25486.027860] softirqs last disabled at (41012519): [] irq_exit_rcu+0xb0/0xd0 [25486.036980] kvm_intel kvm irqbypass [25486.045500] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25486.049069] i2c_i801 iTCO_wdt [25486.058013] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25486.058014] iTCO_vendor_support i2c_smbus [25486.061064] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25486.068533] crct10dif_pclmul joydev [25486.072625] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25486.078102] crc32_pclmul ipmi_ssif [25486.081673] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25486.100409] ghash_clmulni_intel mei_me [25486.103895] [25486.109112] mei [25486.112943] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25486.114433] lpc_ich sha512_ssse3 [25486.116272] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25486.123394] acpi_ipmi mfd_core [25486.126705] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25486.133829] wmi ipmi_si [25486.136967] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25486.144090] ipmi_devintf ipmi_msghandler [25486.146622] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25486.153745] ioatdma [25486.157749] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25486.164875] acpi_pad acpi_power_meter [25486.167058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25486.175137] ixgbe [25486.178881] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25486.184618] igb mdio [25486.186628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25486.193752] crc32c_intel nvme [25486.196023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25486.203147] ptp nvme_core [25486.206198] Call Trace: [25486.206200] [25486.213322] i2c_algo_bit [25486.216029] ? watchdog_timer_fn+0x1b4/0x220 [25486.218470] pps_core dca [25486.220482] ? __hrtimer_run_queues+0x1d3/0x360 [25486.223116] fuse [25486.227384] ? hrtimer_interrupt+0x104/0x240 [25486.229996] irq event stamp: 125917788 [25486.234521] ? __local_bh_enable+0x50/0x60 [25486.236446] hardirqs last enabled at (125917787): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25486.240711] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25486.244452] hardirqs last disabled at (125917788): [] sysvec_apic_timer_interrupt+0xe/0x90 [25486.248544] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25486.258788] softirqs last enabled at (125917786): [] __do_softirq+0x2be/0x371 [25486.264006] [25486.273817] softirqs last disabled at (125917781): [] irq_exit_rcu+0xb0/0xd0 [25486.278783] [25486.278784] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25486.287554] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25486.289654] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25486.298248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25486.300347] do_raw_spin_lock+0xb5/0xc0 [25486.305651] Workqueue: events delayed_work [ceph] [25486.315030] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25486.319898] [25486.319899] RIP: 0010:lock_release+0x14e/0x260 [25486.327371] try_get_cap_refs+0x159/0x880 [ceph] [25486.331198] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [25486.335903] __ceph_get_caps+0x4a1/0x760 [ceph] [25486.340682] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [25486.342178] ? __file_remove_privs+0x89/0x110 [25486.346608] [25486.346608] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [25486.351221] ? wake_bit_function+0x70/0x70 [25486.369956] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [25486.374483] ? ceph_write_iter+0x260/0x770 [ceph] [25486.379716] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [25486.384069] ceph_write_iter+0x260/0x770 [ceph] [25486.385557] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [25486.392687] ? lock_is_held_type+0xaa/0x120 [25486.396784] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [25486.403909] vfs_write+0x324/0x4f0 [25486.408602] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25486.415733] ksys_pwrite64+0x65/0xa0 [25486.420250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25486.427376] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25486.431552] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25486.438677] do_syscall_64+0x38/0x80 [25486.442075] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25486.450151] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25486.453722] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25486.459462] RIP: 0033:0x7f48f0eeb93b [25486.464591] Call Trace: [25486.471716] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25486.475286] [25486.482411] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25486.487456] ? watchdog_timer_fn+0x1b4/0x220 [25486.494578] ORIG_RAX: 0000000000000012 [25486.498150] ? __hrtimer_run_queues+0x1d3/0x360 [25486.500593] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25486.519333] ? hrtimer_interrupt+0x104/0x240 [25486.521341] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25486.526559] ? __local_bh_enable+0x50/0x60 [25486.530823] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25486.534656] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25486.539178] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25486.546303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25486.550565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25486.557691] [25486.561793] [25486.568914] [25486.568916] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25486.605021] ? igrab+0x43/0x50 [25486.608074] ? lock_release+0x14e/0x260 [25486.611911] ? lock_release+0xc5/0x260 [25486.615656] _raw_spin_unlock+0x1b/0x40 [25486.619509] igrab+0x43/0x50 [25486.622388] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25486.627538] delayed_work+0x1d8/0x280 [ceph] [25486.631820] ? process_scheduled_works+0x22a/0x510 [25486.636610] process_scheduled_works+0x2a4/0x510 [25486.641226] worker_thread+0x188/0x340 [25486.644974] ? rescuer_thread+0x310/0x310 [25486.648977] kthread+0xf5/0x130 [25486.652115] ? kthread_complete_and_exit+0x20/0x20 [25486.656901] ret_from_fork+0x31/0x40 [25486.660479] ? kthread_complete_and_exit+0x20/0x20 [25486.665264] ret_from_fork_asm+0x11/0x20 [25486.669195] [25487.487497] rcu: INFO: rcu_preempt self-detected stall on CPU [25487.493256] rcu: 0-....: (13246791 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=6619433 [25487.503676] rcu: (t=13550637 jiffies g=1309609 q=419907 ncpus=8) [25487.509760] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25487.518703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25487.526174] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25487.531650] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25487.550388] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25487.555607] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25487.562730] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25487.569853] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25487.576979] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25487.584102] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25487.591225] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25487.599304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25487.605040] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25487.612167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25487.619288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25487.626413] Call Trace: [25487.628858] [25487.630868] ? rcu_dump_cpu_stacks+0xd1/0x180 [25487.635221] ? rcu_sched_clock_irq+0x633/0x1390 [25487.639743] ? lock_is_held_type+0xaa/0x120 [25487.643923] ? lock_is_held_type+0xaa/0x120 [25487.648100] ? tick_sched_do_timer+0xa0/0xa0 [25487.652370] ? rcu_read_lock_sched_held+0x56/0x90 [25487.657070] ? tick_sched_do_timer+0xa0/0xa0 [25487.661333] ? update_process_times+0x5f/0x90 [25487.665683] ? tick_sched_handle.isra.16+0x1e/0x50 [25487.670467] ? tick_sched_timer+0x6f/0x80 [25487.674473] ? __hrtimer_run_queues+0x1d3/0x360 [25487.678998] ? hrtimer_interrupt+0x104/0x240 [25487.683268] ? __local_bh_enable+0x50/0x60 [25487.687360] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25487.692585] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25487.697542] [25487.699641] [25487.701738] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25487.707045] ? queued_spin_lock_slowpath+0x16/0x2e0 [25487.711923] do_raw_spin_lock+0xb5/0xc0 [25487.715760] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25487.720562] try_get_cap_refs+0x159/0x880 [ceph] [25487.725194] __ceph_get_caps+0x4a1/0x760 [ceph] [25487.729735] ? __file_remove_privs+0x89/0x110 [25487.734090] ? wake_bit_function+0x70/0x70 [25487.738185] ? ceph_write_iter+0x260/0x770 [ceph] [25487.742895] ceph_write_iter+0x260/0x770 [ceph] [25487.747432] ? lock_is_held_type+0xaa/0x120 [25487.751616] vfs_write+0x324/0x4f0 [25487.755016] ksys_pwrite64+0x65/0xa0 [25487.758592] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25487.763724] do_syscall_64+0x38/0x80 [25487.767295] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25487.772337] RIP: 0033:0x7f48f0eeb93b [25487.775907] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25487.794646] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [25487.802203] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25487.809327] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25487.816452] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25487.823574] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25487.830700] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25487.837828] [25487.840017] Sending NMI from CPU 0 to CPUs 7: [25487.844369] NMI backtrace for cpu 7 [25487.844371] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25487.844372] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25487.844373] Workqueue: events delayed_work [ceph] [25487.844388] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [25487.844390] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [25487.844391] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [25487.844392] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [25487.844393] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [25487.844394] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [25487.844395] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25487.844395] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25487.844396] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25487.844397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25487.844398] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25487.844399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25487.844399] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25487.844400] Call Trace: [25487.844401] [25487.844402] ? nmi_cpu_backtrace+0xad/0x120 [25487.844405] ? nmi_cpu_backtrace_handler+0x11/0x20 [25487.844407] ? nmi_handle+0xd4/0x260 [25487.844411] ? default_do_nmi+0x49/0xf0 [25487.844413] ? exc_nmi+0x11e/0x150 [25487.844414] ? end_repeat_nmi+0x16/0x67 [25487.844418] ? do_raw_spin_lock+0x42/0xc0 [25487.844420] ? do_raw_spin_lock+0x42/0xc0 [25487.844422] ? do_raw_spin_lock+0x42/0xc0 [25487.844424] [25487.844424] [25487.844425] igrab+0x1d/0x50 [25487.844426] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25487.844440] delayed_work+0x1d8/0x280 [ceph] [25487.844454] ? process_scheduled_works+0x22a/0x510 [25487.844456] process_scheduled_works+0x2a4/0x510 [25487.844460] worker_thread+0x188/0x340 [25487.844463] ? rescuer_thread+0x310/0x310 [25487.844465] kthread+0xf5/0x130 [25487.844467] ? kthread_complete_and_exit+0x20/0x20 [25487.844469] ret_from_fork+0x31/0x40 [25487.844472] ? kthread_complete_and_exit+0x20/0x20 [25487.844474] ret_from_fork_asm+0x11/0x20 [25487.844478] [25513.904589] watchdog: BUG: soft lockup - CPU#0 stuck for 12647s! [dbench:78683] [25513.911891] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25513.916589] watchdog: BUG: soft lockup - CPU#7 stuck for 21644s! [kworker/7:4:78808] [25513.967332] irq event stamp: 41069918 [25513.967333] hardirqs last enabled at (41069917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25513.975064] Modules linked in: [25513.978722] hardirqs last disabled at (41069918): [] sysvec_apic_timer_interrupt+0xe/0x90 [25513.988880] ceph [25513.991930] softirqs last enabled at (41069888): [] __do_softirq+0x2be/0x371 [25514.001654] libceph [25514.003579] softirqs last disabled at (41069883): [] irq_exit_rcu+0xb0/0xd0 [25514.012263] dns_resolver [25514.014447] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25514.022958] fscache netfs [25514.025592] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25514.025593] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25514.034536] veth [25514.037239] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25514.044728] nft_chain_nat xt_MASQUERADE [25514.050205] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25514.052130] nf_nat nf_conntrack [25514.070867] [25514.070868] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25514.074784] nf_defrag_ipv6 nf_defrag_ipv4 [25514.080002] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25514.083225] nft_compat [25514.084717] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25514.091842] nf_tables [25514.095933] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25514.103057] nfnetlink bridge [25514.105500] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25514.112624] stp llc [25514.114987] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25514.122115] xfs libcrc32c [25514.125079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25514.132203] sunrpc intel_rapl_msr [25514.134385] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25514.142464] intel_rapl_common [25514.145168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25514.150904] sb_edac x86_pkg_temp_thermal [25514.154304] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25514.161427] coretemp [25514.164479] Call Trace: [25514.164480] [25514.171603] kvm_intel [25514.175608] ? watchdog_timer_fn+0x1b4/0x220 [25514.182729] kvm [25514.185001] ? __hrtimer_run_queues+0x1d3/0x360 [25514.187445] irqbypass i2c_i801 [25514.189459] ? hrtimer_interrupt+0x104/0x240 [25514.191813] iTCO_wdt iTCO_vendor_support [25514.196078] ? __local_bh_enable+0x50/0x60 [25514.197914] i2c_smbus crct10dif_pclmul [25514.202442] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25514.205576] joydev [25514.209841] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25514.213845] crc32_pclmul [25514.217934] [25514.221767] ipmi_ssif ghash_clmulni_intel [25514.226987] [25514.226989] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25514.229089] mei_me mei [25514.234050] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25514.236665] lpc_ich sha512_ssse3 [25514.238763] do_raw_spin_lock+0xb5/0xc0 [25514.242853] acpi_ipmi [25514.244950] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25514.250268] mfd_core [25514.252716] try_get_cap_refs+0x159/0x880 [ceph] [25514.257585] wmi ipmi_si [25514.260900] __ceph_get_caps+0x4a1/0x760 [ceph] [25514.264727] ipmi_devintf ipmi_msghandler [25514.267089] ? __file_remove_privs+0x89/0x110 [25514.271868] ioatdma [25514.274139] ? wake_bit_function+0x70/0x70 [25514.278750] acpi_pad [25514.281284] ? ceph_write_iter+0x260/0x770 [ceph] [25514.285805] acpi_power_meter ixgbe [25514.289808] ceph_write_iter+0x260/0x770 [ceph] [25514.294159] igb mdio [25514.296347] ? lock_is_held_type+0xaa/0x120 [25514.300434] crc32c_intel [25514.302707] vfs_write+0x324/0x4f0 [25514.307402] nvme ptp [25514.310891] ksys_pwrite64+0x65/0xa0 [25514.315410] nvme_core i2c_algo_bit [25514.317682] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25514.321859] pps_core dca [25514.324476] do_syscall_64+0x38/0x80 [25514.327874] fuse [25514.330146] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25514.333714] irq event stamp: 126137282 [25514.337199] RIP: 0033:0x7f48f0eeb93b [25514.342329] hardirqs last enabled at (126137281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25514.344948] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25514.348519] hardirqs last disabled at (126137282): [] sysvec_apic_timer_interrupt+0xe/0x90 [25514.350442] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25514.355485] softirqs last enabled at (126137280): [] __do_softirq+0x2be/0x371 [25514.359231] ORIG_RAX: 0000000000000012 [25514.362801] softirqs last disabled at (126137275): [] irq_exit_rcu+0xb0/0xd0 [25514.373046] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25514.391782] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25514.401593] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25514.406811] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25514.415581] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25514.419413] Workqueue: events delayed_work [ceph] [25514.428009] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25514.435133] [25514.444512] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25514.451637] RIP: 0010:do_raw_spin_unlock+0x42/0xc0 [25514.459113] [25514.466230] Code: 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 8b 05 4a bf eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 10 <89> 43 08 c6 03 00 5b c3 e8 41 21 45 00 85 c0 74 c3 48 c7 c6 29 e2 [25514.512373] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [25514.517590] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [25514.524715] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [25514.531839] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [25514.538962] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25514.546087] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25514.553211] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25514.561290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25514.567026] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25514.574150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25514.581278] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25514.588408] Call Trace: [25514.590852] [25514.592862] ? watchdog_timer_fn+0x1b4/0x220 [25514.597128] ? __hrtimer_run_queues+0x1d3/0x360 [25514.601654] ? hrtimer_interrupt+0x104/0x240 [25514.605923] ? __local_bh_enable+0x50/0x60 [25514.610015] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25514.615233] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25514.620197] [25514.622294] [25514.624394] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25514.629710] ? do_raw_spin_unlock+0x42/0xc0 [25514.633893] _raw_spin_unlock+0x23/0x40 [25514.637733] igrab+0x43/0x50 [25514.640609] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25514.645759] delayed_work+0x1d8/0x280 [ceph] [25514.650040] ? process_scheduled_works+0x22a/0x510 [25514.654830] process_scheduled_works+0x2a4/0x510 [25514.659445] worker_thread+0x188/0x340 [25514.663197] ? rescuer_thread+0x310/0x310 [25514.667206] kthread+0xf5/0x130 [25514.670344] ? kthread_complete_and_exit+0x20/0x20 [25514.675129] ret_from_fork+0x31/0x40 [25514.678699] ? kthread_complete_and_exit+0x20/0x20 [25514.683483] ret_from_fork_asm+0x11/0x20 [25514.687406] [25541.905075] watchdog: BUG: soft lockup - CPU#0 stuck for 12673s! [dbench:78683] [25541.912379] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25541.917074] watchdog: BUG: soft lockup - CPU#7 stuck for 21670s! [kworker/7:4:78808] [25541.967820] irq event stamp: 41128220 [25541.967821] hardirqs last enabled at (41128219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25541.975551] Modules linked in: [25541.979209] hardirqs last disabled at (41128220): [] sysvec_apic_timer_interrupt+0xe/0x90 [25541.989367] ceph [25541.992418] softirqs last enabled at (41128192): [] __do_softirq+0x2be/0x371 [25542.002141] libceph [25542.004064] softirqs last disabled at (41128187): [] irq_exit_rcu+0xb0/0xd0 [25542.012758] dns_resolver fscache [25542.014943] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25542.023471] netfs [25542.026780] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25542.035726] veth nft_chain_nat [25542.037754] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25542.045225] xt_MASQUERADE [25542.048363] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25542.053838] nf_nat nf_conntrack [25542.056543] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25542.075280] nf_defrag_ipv6 [25542.078506] [25542.078507] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25542.083723] nf_defrag_ipv4 nft_compat [25542.086512] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25542.088004] nf_tables nfnetlink [25542.095129] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25542.098872] bridge stp [25542.105997] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25542.109222] llc [25542.116346] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25542.118789] xfs libcrc32c [25542.125914] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25542.127752] sunrpc [25542.134874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25542.137577] intel_rapl_msr intel_rapl_common [25542.145654] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25542.147754] sb_edac x86_pkg_temp_thermal [25542.153490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25542.157842] coretemp kvm_intel [25542.164965] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25542.168970] kvm irqbypass [25542.176094] Call Trace: [25542.176097] [25542.179230] i2c_i801 iTCO_wdt [25542.186357] ? watchdog_timer_fn+0x1b4/0x220 [25542.189059] iTCO_vendor_support i2c_smbus [25542.191505] ? __hrtimer_run_queues+0x1d3/0x360 [25542.193514] crct10dif_pclmul [25542.196567] ? hrtimer_interrupt+0x104/0x240 [25542.200828] joydev crc32_pclmul [25542.204919] ? __local_bh_enable+0x50/0x60 [25542.209443] ipmi_ssif ghash_clmulni_intel [25542.212410] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25542.216671] mei_me [25542.219897] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25542.223988] mei lpc_ich [25542.228077] [25542.233295] sha512_ssse3 acpi_ipmi [25542.235392] [25542.240351] mfd_core wmi [25542.242881] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25542.244979] ipmi_si [25542.248479] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25542.250567] ipmi_devintf ipmi_msghandler [25542.253188] do_raw_spin_lock+0xb5/0xc0 [25542.258488] ioatdma acpi_pad [25542.260673] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25542.265543] acpi_power_meter ixgbe [25542.269552] try_get_cap_refs+0x159/0x880 [ceph] [25542.273379] igb mdio [25542.276347] __ceph_get_caps+0x4a1/0x760 [ceph] [25542.281126] crc32c_intel [25542.284614] ? __file_remove_privs+0x89/0x110 [25542.289223] nvme [25542.291493] ? wake_bit_function+0x70/0x70 [25542.296017] ptp nvme_core [25542.298636] ? ceph_write_iter+0x260/0x770 [ceph] [25542.302984] i2c_algo_bit pps_core [25542.304909] ceph_write_iter+0x260/0x770 [ceph] [25542.309000] dca [25542.311708] ? lock_is_held_type+0xaa/0x120 [25542.316399] fuse [25542.319802] vfs_write+0x324/0x4f0 [25542.324322] irq event stamp: 126358216 [25542.326163] ksys_pwrite64+0x65/0xa0 [25542.330337] hardirqs last enabled at (126358215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25542.332262] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25542.335658] hardirqs last disabled at (126358216): [] sysvec_apic_timer_interrupt+0xe/0x90 [25542.339404] do_syscall_64+0x38/0x80 [25542.342973] softirqs last enabled at (126358214): [] __do_softirq+0x2be/0x371 [25542.353219] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25542.358349] softirqs last disabled at (126358209): [] irq_exit_rcu+0xb0/0xd0 [25542.368160] RIP: 0033:0x7f48f0eeb93b [25542.371731] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25542.380501] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25542.385545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25542.394142] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25542.397714] Workqueue: events delayed_work [ceph] [25542.407091] ORIG_RAX: 0000000000000012 [25542.425829] [25542.433298] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25542.438516] RIP: 0010:lock_acquire+0xf0/0x2b0 [25542.443213] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25542.447046] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25542.448536] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25542.455660] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25542.460009] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25542.467134] [25542.485871] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25542.492996] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25542.498220] [25542.505338] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25542.530386] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25542.537508] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25542.544632] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25542.551772] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25542.559852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25542.565589] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25542.572713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25542.579839] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25542.586963] Call Trace: [25542.589407] [25542.591417] ? watchdog_timer_fn+0x1b4/0x220 [25542.595683] ? __hrtimer_run_queues+0x1d3/0x360 [25542.600218] ? hrtimer_interrupt+0x104/0x240 [25542.604486] ? __local_bh_enable+0x50/0x60 [25542.608579] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25542.613797] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25542.618767] [25542.620866] [25542.622965] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25542.628274] ? lock_acquire+0xf0/0x2b0 [25542.632024] ? igrab+0x43/0x50 [25542.635074] ? lock_release+0xc5/0x260 [25542.638817] _raw_spin_lock+0x30/0x40 [25542.642483] ? igrab+0x1d/0x50 [25542.645532] igrab+0x1d/0x50 [25542.648411] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25542.653562] delayed_work+0x1d8/0x280 [ceph] [25542.657843] ? process_scheduled_works+0x22a/0x510 [25542.662632] process_scheduled_works+0x2a4/0x510 [25542.667248] worker_thread+0x188/0x340 [25542.670997] ? rescuer_thread+0x310/0x310 [25542.675001] kthread+0xf5/0x130 [25542.678147] ? kthread_complete_and_exit+0x20/0x20 [25542.682932] ret_from_fork+0x31/0x40 [25542.686509] ? kthread_complete_and_exit+0x20/0x20 [25542.691295] ret_from_fork_asm+0x11/0x20 [25542.695224] [25569.905560] watchdog: BUG: soft lockup - CPU#0 stuck for 12699s! [dbench:78683] [25569.912858] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25569.917560] watchdog: BUG: soft lockup - CPU#7 stuck for 21696s! [kworker/7:4:78808] [25569.968300] irq event stamp: 41186514 [25569.968301] hardirqs last enabled at (41186513): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25569.976032] Modules linked in: ceph [25569.979689] hardirqs last disabled at (41186514): [] sysvec_apic_timer_interrupt+0xe/0x90 [25569.989845] libceph [25569.993329] softirqs last enabled at (41186486): [] __do_softirq+0x2be/0x371 [25570.003053] dns_resolver [25570.005245] softirqs last disabled at (41186481): [] irq_exit_rcu+0xb0/0xd0 [25570.013930] fscache [25570.016566] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25570.025077] netfs veth [25570.027261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25570.036204] nft_chain_nat [25570.038647] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25570.046119] xt_MASQUERADE nf_nat [25570.048823] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25570.054301] nf_conntrack nf_defrag_ipv6 [25570.057611] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25570.076348] nf_defrag_ipv4 nft_compat [25570.080267] [25570.080268] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25570.085483] nf_tables nfnetlink [25570.089236] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25570.090726] bridge [25570.097852] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25570.101076] stp llc [25570.108198] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25570.110298] xfs [25570.117421] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25570.119606] libcrc32c sunrpc [25570.126730] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25570.128566] intel_rapl_msr [25570.135691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25570.138655] intel_rapl_common sb_edac [25570.146731] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25570.149524] x86_pkg_temp_thermal [25570.155262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25570.159006] coretemp kvm_intel [25570.166129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25570.169440] kvm irqbypass [25570.176563] Call Trace: [25570.176564] [25570.179702] i2c_i801 iTCO_wdt [25570.186826] ? watchdog_timer_fn+0x1b4/0x220 [25570.189530] iTCO_vendor_support i2c_smbus [25570.191974] ? __hrtimer_run_queues+0x1d3/0x360 [25570.193985] crct10dif_pclmul [25570.197039] ? hrtimer_interrupt+0x104/0x240 [25570.201299] joydev crc32_pclmul [25570.205389] ? __local_bh_enable+0x50/0x60 [25570.209913] ipmi_ssif ghash_clmulni_intel [25570.212880] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25570.217142] mei_me mei [25570.220366] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25570.224457] lpc_ich sha512_ssse3 [25570.228547] [25570.233765] acpi_ipmi [25570.236209] [25570.236211] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25570.241166] mfd_core wmi [25570.244480] ? queued_spin_lock_slowpath+0x16/0x2e0 [25570.246575] ipmi_si ipmi_devintf [25570.248934] do_raw_spin_lock+0xb5/0xc0 [25570.251047] ipmi_msghandler [25570.256351] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25570.258969] ioatdma acpi_pad [25570.263841] try_get_cap_refs+0x159/0x880 [ceph] [25570.267150] acpi_power_meter ixgbe [25570.270985] __ceph_get_caps+0x4a1/0x760 [ceph] [25570.273858] igb mdio [25570.278646] ? __file_remove_privs+0x89/0x110 [25570.281604] crc32c_intel [25570.286216] ? wake_bit_function+0x70/0x70 [25570.289717] nvme [25570.294245] ? ceph_write_iter+0x260/0x770 [ceph] [25570.296513] ptp nvme_core [25570.300863] ceph_write_iter+0x260/0x770 [ceph] [25570.303481] i2c_algo_bit [25570.307575] ? lock_is_held_type+0xaa/0x120 [25570.309495] pps_core dca [25570.314197] vfs_write+0x324/0x4f0 [25570.316898] fuse [25570.321426] ksys_pwrite64+0x65/0xa0 [25570.324039] irq event stamp: 126579132 [25570.328217] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25570.330832] hardirqs last enabled at (126579131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25570.334242] do_syscall_64+0x38/0x80 [25570.336164] hardirqs last disabled at (126579132): [] sysvec_apic_timer_interrupt+0xe/0x90 [25570.339734] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25570.343478] softirqs last enabled at (126579130): [] __do_softirq+0x2be/0x371 [25570.348609] RIP: 0033:0x7f48f0eeb93b [25570.358852] softirqs last disabled at (126579125): [] irq_exit_rcu+0xb0/0xd0 [25570.362425] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25570.372246] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25570.377286] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25570.386057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25570.389629] ORIG_RAX: 0000000000000012 [25570.389630] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25570.398243] Workqueue: events delayed_work [ceph] [25570.416982] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25570.426358] [25570.431576] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25570.439048] RIP: 0010:lock_acquire+0xf0/0x2b0 [25570.442878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25570.450002] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25570.454698] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25570.461825] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25570.463320] [25570.470439] [25570.470440] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25570.523793] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25570.530916] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25570.538041] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25570.545165] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25570.552288] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25570.560365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25570.566103] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25570.573226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25570.580368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25570.587493] Call Trace: [25570.589938] [25570.591967] ? watchdog_timer_fn+0x1b4/0x220 [25570.596254] ? __hrtimer_run_queues+0x1d3/0x360 [25570.600784] ? hrtimer_interrupt+0x104/0x240 [25570.605051] ? __local_bh_enable+0x50/0x60 [25570.609145] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25570.614371] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25570.619327] [25570.621423] [25570.623521] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25570.628830] ? lock_acquire+0xf0/0x2b0 [25570.632579] ? igrab+0x43/0x50 [25570.635630] ? lock_release+0xc5/0x260 [25570.639373] _raw_spin_lock+0x30/0x40 [25570.643031] ? igrab+0x1d/0x50 [25570.646082] igrab+0x1d/0x50 [25570.648970] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25570.654118] delayed_work+0x1d8/0x280 [ceph] [25570.658399] ? process_scheduled_works+0x22a/0x510 [25570.663189] process_scheduled_works+0x2a4/0x510 [25570.667804] worker_thread+0x188/0x340 [25570.671553] ? rescuer_thread+0x310/0x310 [25570.675557] kthread+0xf5/0x130 [25570.678696] ? kthread_complete_and_exit+0x20/0x20 [25570.683487] ret_from_fork+0x31/0x40 [25570.687066] ? kthread_complete_and_exit+0x20/0x20 [25570.691852] ret_from_fork_asm+0x11/0x20 [25570.695781] [25597.906048] watchdog: BUG: soft lockup - CPU#0 stuck for 12725s! [dbench:78683] [25597.913356] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25597.918047] watchdog: BUG: soft lockup - CPU#7 stuck for 21722s! [kworker/7:4:78808] [25597.968806] irq event stamp: 41244824 [25597.968807] hardirqs last enabled at (41244823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25597.976535] Modules linked in: [25597.980192] hardirqs last disabled at (41244824): [] sysvec_apic_timer_interrupt+0xe/0x90 [25597.990350] ceph [25597.993401] softirqs last enabled at (41244796): [] __do_softirq+0x2be/0x371 [25598.003125] libceph [25598.005049] softirqs last disabled at (41244791): [] irq_exit_rcu+0xb0/0xd0 [25598.013734] dns_resolver fscache [25598.015917] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25598.024441] netfs veth [25598.027757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25598.036701] nft_chain_nat xt_MASQUERADE [25598.039164] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25598.046632] nf_nat nf_conntrack [25598.050551] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25598.056026] nf_defrag_ipv6 [25598.059252] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25598.077988] nf_defrag_ipv4 nft_compat [25598.080780] [25598.080781] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25598.085999] nf_tables [25598.089741] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25598.091232] nfnetlink bridge [25598.098356] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25598.100722] stp llc [25598.107847] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25598.110810] xfs libcrc32c [25598.117934] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25598.120120] sunrpc [25598.127243] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25598.129948] intel_rapl_msr intel_rapl_common [25598.137071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25598.139167] sb_edac x86_pkg_temp_thermal [25598.147246] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25598.151597] coretemp kvm_intel [25598.157335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25598.161338] kvm irqbypass [25598.168461] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25598.171599] i2c_i801 iTCO_wdt [25598.178735] Call Trace: [25598.181444] iTCO_vendor_support [25598.188572] [25598.191619] i2c_smbus crct10dif_pclmul [25598.194066] ? watchdog_timer_fn+0x1b4/0x220 [25598.197288] joydev crc32_pclmul [25598.199300] ? __hrtimer_run_queues+0x1d3/0x360 [25598.203130] ipmi_ssif [25598.207398] ? hrtimer_interrupt+0x104/0x240 [25598.210619] ghash_clmulni_intel mei_me [25598.215143] ? __local_bh_enable+0x50/0x60 [25598.217500] mei lpc_ich [25598.221766] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25598.225594] sha512_ssse3 acpi_ipmi [25598.229686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25598.232214] mfd_core wmi [25598.237446] [25598.240935] ipmi_si [25598.245891] [25598.245893] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25598.248509] ipmi_devintf [25598.250610] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25598.252790] ipmi_msghandler [25598.254890] do_raw_spin_lock+0xb5/0xc0 [25598.260191] ioatdma acpi_pad [25598.262810] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25598.267681] acpi_power_meter ixgbe [25598.270560] try_get_cap_refs+0x159/0x880 [ceph] [25598.274388] igb mdio [25598.277357] __ceph_get_caps+0x4a1/0x760 [ceph] [25598.282136] crc32c_intel nvme [25598.285624] ? __file_remove_privs+0x89/0x110 [25598.290231] ptp [25598.292502] ? wake_bit_function+0x70/0x70 [25598.297025] nvme_core i2c_algo_bit [25598.300079] ? ceph_write_iter+0x260/0x770 [ceph] [25598.304444] pps_core dca [25598.306284] ceph_write_iter+0x260/0x770 [ceph] [25598.310372] fuse [25598.313860] ? lock_is_held_type+0xaa/0x120 [25598.318554] irq event stamp: 126799872 [25598.321173] vfs_write+0x324/0x4f0 [25598.325695] hardirqs last enabled at (126799871): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25598.327624] ksys_pwrite64+0x65/0xa0 [25598.331797] hardirqs last disabled at (126799872): [] sysvec_apic_timer_interrupt+0xe/0x90 [25598.335543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25598.338940] softirqs last enabled at (126799870): [] __do_softirq+0x2be/0x371 [25598.349185] do_syscall_64+0x38/0x80 [25598.352752] softirqs last disabled at (126799865): [] irq_exit_rcu+0xb0/0xd0 [25598.362565] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25598.367695] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25598.376468] RIP: 0033:0x7f48f0eeb93b [25598.380038] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25598.388635] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25598.393679] Workqueue: events delayed_work [ceph] [25598.403055] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25598.406627] [25598.414098] ORIG_RAX: 0000000000000012 [25598.432836] RIP: 0010:_raw_spin_lock+0x18/0x40 [25598.437533] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25598.442749] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [25598.444240] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25598.448071] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [25598.452509] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25598.459634] [25598.459635] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [25598.478369] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25598.485494] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [25598.490712] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25598.497853] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [25598.499349] [25598.506467] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25598.544290] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25598.551415] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25598.559494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25598.565230] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25598.572353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25598.579478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25598.586601] Call Trace: [25598.589048] [25598.591059] ? watchdog_timer_fn+0x1b4/0x220 [25598.595323] ? __hrtimer_run_queues+0x1d3/0x360 [25598.599850] ? hrtimer_interrupt+0x104/0x240 [25598.604117] ? __local_bh_enable+0x50/0x60 [25598.608210] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25598.613428] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25598.618401] [25598.620499] [25598.622596] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25598.627904] ? _raw_spin_lock+0x18/0x40 [25598.631740] ? igrab+0x1d/0x50 [25598.634790] igrab+0x1d/0x50 [25598.637670] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25598.642818] delayed_work+0x1d8/0x280 [ceph] [25598.647100] ? process_scheduled_works+0x22a/0x510 [25598.651890] process_scheduled_works+0x2a4/0x510 [25598.656506] worker_thread+0x188/0x340 [25598.660254] ? rescuer_thread+0x310/0x310 [25598.664257] kthread+0xf5/0x130 [25598.667395] ? kthread_complete_and_exit+0x20/0x20 [25598.672179] ret_from_fork+0x31/0x40 [25598.675759] ? kthread_complete_and_exit+0x20/0x20 [25598.680544] ret_from_fork_asm+0x11/0x20 [25598.684474] [25625.906534] watchdog: BUG: soft lockup - CPU#0 stuck for 12751s! [dbench:78683] [25625.913833] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25625.918534] watchdog: BUG: soft lockup - CPU#7 stuck for 21748s! [kworker/7:4:78808] [25625.969275] irq event stamp: 41303050 [25625.969276] hardirqs last enabled at (41303049): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25625.977005] Modules linked in: [25625.980662] hardirqs last disabled at (41303050): [] sysvec_apic_timer_interrupt+0xe/0x90 [25625.990821] ceph [25625.993870] softirqs last enabled at (41303022): [] __do_softirq+0x2be/0x371 [25626.003595] libceph [25626.005519] softirqs last disabled at (41303017): [] irq_exit_rcu+0xb0/0xd0 [25626.014210] dns_resolver fscache [25626.016396] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25626.024908] netfs veth [25626.028234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25626.028236] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25626.037179] nft_chain_nat xt_MASQUERADE [25626.039622] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25626.047095] nf_nat nf_conntrack [25626.052572] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25626.056490] nf_defrag_ipv6 nf_defrag_ipv4 [25626.075227] [25626.075228] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25626.078452] nft_compat [25626.083668] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25626.087757] nf_tables [25626.089250] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25626.096373] nfnetlink bridge [25626.098817] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25626.105941] stp llc [25626.108299] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25626.115423] xfs libcrc32c [25626.118387] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25626.125511] sunrpc intel_rapl_msr [25626.127694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25626.134818] intel_rapl_common [25626.137522] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25626.145601] sb_edac x86_pkg_temp_thermal [25626.148999] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25626.154735] coretemp kvm_intel [25626.157786] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25626.164910] kvm [25626.168933] Call Trace: [25626.176057] irqbypass i2c_i801 [25626.179209] [25626.186336] iTCO_wdt iTCO_vendor_support [25626.188172] ? watchdog_timer_fn+0x1b4/0x220 [25626.190617] i2c_smbus crct10dif_pclmul [25626.193756] ? __hrtimer_run_queues+0x1d3/0x360 [25626.195764] joydev crc32_pclmul [25626.199771] ? hrtimer_interrupt+0x104/0x240 [25626.204034] ipmi_ssif ghash_clmulni_intel [25626.207864] ? __local_bh_enable+0x50/0x60 [25626.212386] mei_me mei [25626.215614] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25626.219874] lpc_ich sha512_ssse3 [25626.223967] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25626.228058] acpi_ipmi mfd_core [25626.230501] [25626.235718] wmi ipmi_si [25626.239028] [25626.239030] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25626.243987] ipmi_devintf ipmi_msghandler [25626.247129] ? queued_spin_lock_slowpath+0x16/0x2e0 [25626.249221] ioatdma [25626.251754] do_raw_spin_lock+0xb5/0xc0 [25626.253849] acpi_pad acpi_power_meter [25626.259155] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25626.263157] ixgbe igb [25626.268030] try_get_cap_refs+0x159/0x880 [ceph] [25626.270215] mdio crc32c_intel [25626.274049] __ceph_get_caps+0x4a1/0x760 [ceph] [25626.277787] nvme ptp [25626.282575] ? __file_remove_privs+0x89/0x110 [25626.284932] nvme_core i2c_algo_bit [25626.289541] ? wake_bit_function+0x70/0x70 [25626.292591] pps_core dca [25626.297118] ? ceph_write_iter+0x260/0x770 [ceph] [25626.299384] fuse [25626.303736] ceph_write_iter+0x260/0x770 [ceph] [25626.307219] irq event stamp: 127021018 [25626.311315] ? lock_is_held_type+0xaa/0x120 [25626.313936] hardirqs last enabled at (127021017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25626.318637] vfs_write+0x324/0x4f0 [25626.320557] hardirqs last disabled at (127021018): [] sysvec_apic_timer_interrupt+0xe/0x90 [25626.325087] ksys_pwrite64+0x65/0xa0 [25626.328827] softirqs last enabled at (127021016): [] __do_softirq+0x2be/0x371 [25626.333004] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25626.343247] softirqs last disabled at (127021011): [] irq_exit_rcu+0xb0/0xd0 [25626.346647] do_syscall_64+0x38/0x80 [25626.356457] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25626.360027] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25626.368797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25626.373936] RIP: 0033:0x7f48f0eeb93b [25626.382536] Workqueue: events delayed_work [ceph] [25626.386105] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25626.395483] [25626.400526] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25626.407997] RIP: 0010:lock_acquire+0xf0/0x2b0 [25626.411569] ORIG_RAX: 0000000000000012 [25626.416265] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25626.435002] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25626.436494] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25626.441713] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25626.446062] [25626.446063] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25626.449893] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25626.468632] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25626.475754] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25626.480972] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25626.488097] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25626.489588] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25626.496716] [25626.503835] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25626.548773] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25626.556852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25626.562588] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25626.569712] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25626.576836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25626.583961] Call Trace: [25626.586407] [25626.588415] ? watchdog_timer_fn+0x1b4/0x220 [25626.592681] ? __hrtimer_run_queues+0x1d3/0x360 [25626.597223] ? hrtimer_interrupt+0x104/0x240 [25626.601486] ? __local_bh_enable+0x50/0x60 [25626.605577] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25626.610804] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25626.615760] [25626.617857] [25626.619956] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25626.625271] ? lock_acquire+0xf0/0x2b0 [25626.629021] ? igrab+0x43/0x50 [25626.632071] ? lock_release+0xc5/0x260 [25626.635817] _raw_spin_lock+0x30/0x40 [25626.639480] ? igrab+0x1d/0x50 [25626.642531] igrab+0x1d/0x50 [25626.645410] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25626.650560] delayed_work+0x1d8/0x280 [ceph] [25626.654840] ? process_scheduled_works+0x22a/0x510 [25626.659622] process_scheduled_works+0x2a4/0x510 [25626.664238] worker_thread+0x188/0x340 [25626.667988] ? rescuer_thread+0x310/0x310 [25626.671999] kthread+0xf5/0x130 [25626.675135] ? kthread_complete_and_exit+0x20/0x20 [25626.679940] ret_from_fork+0x31/0x40 [25626.683516] ? kthread_complete_and_exit+0x20/0x20 [25626.688300] ret_from_fork_asm+0x11/0x20 [25626.692226] [25653.907024] watchdog: BUG: soft lockup - CPU#0 stuck for 12778s! [dbench:78683] [25653.914331] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25653.919024] watchdog: BUG: soft lockup - CPU#7 stuck for 21775s! [kworker/7:4:78808] [25653.969778] irq event stamp: 41361298 [25653.977510] Modules linked in: ceph libceph dns_resolver fscache netfs [25653.981167] hardirqs last enabled at (41361297): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25653.987699] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [25653.997859] hardirqs last disabled at (41361298): [] sysvec_apic_timer_interrupt+0xe/0x90 [25654.005243] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [25654.014969] softirqs last enabled at (41361270): [] __do_softirq+0x2be/0x371 [25654.021418] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25654.030099] softirqs last disabled at (41361265): [] irq_exit_rcu+0xb0/0xd0 [25654.030104] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [25654.045129] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25654.051127] irqbypass i2c_i801 [25654.060071] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25654.060072] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25654.063207] iTCO_wdt [25654.070678] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25654.076172] iTCO_vendor_support i2c_smbus [25654.078444] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25654.097182] crct10dif_pclmul [25654.101271] [25654.101272] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25654.106489] joydev crc32_pclmul [25654.109454] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25654.110944] ipmi_ssif [25654.118068] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25654.121293] ghash_clmulni_intel [25654.128419] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25654.130773] mei_me mei [25654.137898] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25654.141122] lpc_ich sha512_ssse3 [25654.148248] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25654.150699] acpi_ipmi mfd_core [25654.157824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25654.161135] wmi ipmi_si [25654.169210] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25654.172349] ipmi_devintf [25654.178085] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25654.180618] ipmi_msghandler ioatdma [25654.187741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25654.190359] acpi_pad [25654.197482] Call Trace: [25654.201052] acpi_power_meter ixgbe [25654.208179] [25654.210448] igb mdio [25654.212892] ? watchdog_timer_fn+0x1b4/0x220 [25654.216376] crc32c_intel nvme [25654.218388] ? __hrtimer_run_queues+0x1d3/0x360 [25654.220659] ptp nvme_core [25654.224924] ? hrtimer_interrupt+0x104/0x240 [25654.227973] i2c_algo_bit [25654.232496] ? __local_bh_enable+0x50/0x60 [25654.235202] pps_core [25654.239467] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25654.242082] dca fuse [25654.246172] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25654.248443] [25654.248444] irq event stamp: 127241876 [25654.253662] [25654.255931] hardirqs last enabled at (127241875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25654.260889] [25654.262380] hardirqs last disabled at (127241876): [] sysvec_apic_timer_interrupt+0xe/0x90 [25654.266123] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25654.268222] softirqs last enabled at (127241874): [] __do_softirq+0x2be/0x371 [25654.278469] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25654.280562] softirqs last disabled at (127241869): [] irq_exit_rcu+0xb0/0xd0 [25654.290376] do_raw_spin_lock+0xb5/0xc0 [25654.295678] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25654.304451] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25654.309319] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25654.317921] try_get_cap_refs+0x159/0x880 [ceph] [25654.321747] Workqueue: events delayed_work [ceph] [25654.331131] __ceph_get_caps+0x4a1/0x760 [ceph] [25654.335909] [25654.335910] RIP: 0010:lock_release+0x14e/0x260 [25654.343385] ? __file_remove_privs+0x89/0x110 [25654.347990] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [25654.352689] ? wake_bit_function+0x70/0x70 [25654.357230] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [25654.358724] ? ceph_write_iter+0x260/0x770 [ceph] [25654.363158] [25654.363159] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [25654.367510] ceph_write_iter+0x260/0x770 [ceph] [25654.386246] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [25654.390341] ? lock_is_held_type+0xaa/0x120 [25654.395554] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [25654.400253] vfs_write+0x324/0x4f0 [25654.401743] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [25654.408871] ksys_pwrite64+0x65/0xa0 [25654.413391] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [25654.420515] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25654.424704] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25654.431827] do_syscall_64+0x38/0x80 [25654.435222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25654.442346] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25654.445917] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25654.453043] RIP: 0033:0x7f48f0eeb93b [25654.458171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25654.466250] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25654.469821] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25654.475557] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25654.480603] Call Trace: [25654.480605] [25654.487726] ORIG_RAX: 0000000000000012 [25654.491297] ? watchdog_timer_fn+0x1b4/0x220 [25654.498427] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25654.517169] ? __hrtimer_run_queues+0x1d3/0x360 [25654.524290] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25654.529514] ? hrtimer_interrupt+0x104/0x240 [25654.531953] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25654.533964] ? __local_bh_enable+0x50/0x60 [25654.537793] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25654.542062] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25654.549182] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25654.553724] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25654.560852] [25654.565111] [25654.605031] [25654.607128] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25654.612434] ? igrab+0x43/0x50 [25654.615487] ? lock_release+0x14e/0x260 [25654.619323] ? lock_release+0xc5/0x260 [25654.623071] _raw_spin_unlock+0x1b/0x40 [25654.626908] igrab+0x43/0x50 [25654.629786] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25654.634936] delayed_work+0x1d8/0x280 [ceph] [25654.639217] ? process_scheduled_works+0x22a/0x510 [25654.644005] process_scheduled_works+0x2a4/0x510 [25654.648624] worker_thread+0x188/0x340 [25654.652370] ? rescuer_thread+0x310/0x310 [25654.656374] kthread+0xf5/0x130 [25654.659514] ? kthread_complete_and_exit+0x20/0x20 [25654.664304] ret_from_fork+0x31/0x40 [25654.667885] ? kthread_complete_and_exit+0x20/0x20 [25654.672667] ret_from_fork_asm+0x11/0x20 [25654.676590] [25681.907513] watchdog: BUG: soft lockup - CPU#0 stuck for 12804s! [dbench:78683] [25681.914816] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25681.919513] watchdog: BUG: soft lockup - CPU#7 stuck for 21801s! [kworker/7:4:78808] [25681.970258] irq event stamp: 41419380 [25681.970259] hardirqs last enabled at (41419379): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25681.977988] Modules linked in: ceph [25681.981647] hardirqs last disabled at (41419380): [] sysvec_apic_timer_interrupt+0xe/0x90 [25681.991804] libceph [25681.995289] softirqs last enabled at (41419352): [] __do_softirq+0x2be/0x371 [25682.005013] dns_resolver fscache [25682.007198] softirqs last disabled at (41419347): [] irq_exit_rcu+0xb0/0xd0 [25682.015880] netfs veth [25682.019207] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25682.027718] nft_chain_nat xt_MASQUERADE [25682.030163] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25682.039125] nf_nat nf_conntrack [25682.043041] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25682.050512] nf_defrag_ipv6 nf_defrag_ipv4 [25682.053738] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25682.059213] nft_compat nf_tables [25682.063304] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25682.082042] nfnetlink bridge [25682.085353] [25682.085354] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25682.090570] stp llc [25682.093534] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25682.095025] xfs [25682.102150] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25682.104332] libcrc32c sunrpc [25682.111457] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25682.113296] intel_rapl_msr intel_rapl_common [25682.120419] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25682.123383] sb_edac x86_pkg_temp_thermal [25682.130507] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25682.134859] coretemp [25682.141982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25682.145986] kvm_intel kvm [25682.154063] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25682.156333] irqbypass i2c_i801 [25682.162073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25682.164777] iTCO_wdt iTCO_vendor_support [25682.171909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25682.175047] i2c_smbus crct10dif_pclmul [25682.182186] Call Trace: [25682.182187] [25682.186192] joydev crc32_pclmul [25682.193317] ? watchdog_timer_fn+0x1b4/0x220 [25682.197146] ipmi_ssif ghash_clmulni_intel [25682.199592] ? __hrtimer_run_queues+0x1d3/0x360 [25682.201603] mei_me mei [25682.204829] ? hrtimer_interrupt+0x104/0x240 [25682.209088] lpc_ich [25682.213196] ? __local_bh_enable+0x50/0x60 [25682.217720] sha512_ssse3 acpi_ipmi [25682.220167] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25682.224431] mfd_core wmi [25682.226615] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25682.230704] ipmi_si ipmi_devintf [25682.234205] [25682.239424] ipmi_msghandler [25682.242041] [25682.242042] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25682.246997] ioatdma acpi_pad [25682.250313] ? queued_spin_lock_slowpath+0x18/0x2e0 [25682.252405] acpi_power_meter ixgbe [25682.255287] do_raw_spin_lock+0xb5/0xc0 [25682.257380] igb [25682.262686] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25682.265649] mdio crc32c_intel [25682.270522] try_get_cap_refs+0x159/0x880 [ceph] [25682.274005] nvme ptp [25682.277840] __ceph_get_caps+0x4a1/0x760 [ceph] [25682.279671] nvme_core i2c_algo_bit [25682.284459] ? __file_remove_privs+0x89/0x110 [25682.287507] pps_core dca [25682.292118] ? wake_bit_function+0x70/0x70 [25682.294388] fuse [25682.298915] ? ceph_write_iter+0x260/0x770 [ceph] [25682.302395] irq event stamp: 127462878 [25682.306749] ceph_write_iter+0x260/0x770 [ceph] [25682.309365] hardirqs last enabled at (127462877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25682.313458] ? lock_is_held_type+0xaa/0x120 [25682.315380] hardirqs last disabled at (127462878): [] sysvec_apic_timer_interrupt+0xe/0x90 [25682.320080] vfs_write+0x324/0x4f0 [25682.323820] softirqs last enabled at (127462876): [] __do_softirq+0x2be/0x371 [25682.328350] ksys_pwrite64+0x65/0xa0 [25682.338590] softirqs last disabled at (127462871): [] irq_exit_rcu+0xb0/0xd0 [25682.342767] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25682.352578] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25682.355975] do_syscall_64+0x38/0x80 [25682.364745] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25682.368316] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25682.376915] Workqueue: events delayed_work [ceph] [25682.382044] RIP: 0033:0x7f48f0eeb93b [25682.391423] [25682.391423] RIP: 0010:lock_acquire+0x0/0x2b0 [25682.394994] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25682.402463] Code: 0f c1 05 63 da ea 7e 83 f8 01 75 0e 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd [25682.407508] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25682.412206] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [25682.415775] ORIG_RAX: 0000000000000012 [25682.417267] [25682.421531] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25682.440267] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [25682.459006] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25682.464223] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [25682.469441] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25682.473273] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [25682.474762] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25682.481887] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25682.489010] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25682.496134] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25682.503264] [25682.510384] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25682.556283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25682.562021] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25682.569146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25682.576269] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25682.583394] Call Trace: [25682.585838] [25682.587848] ? watchdog_timer_fn+0x1b4/0x220 [25682.592113] ? __hrtimer_run_queues+0x1d3/0x360 [25682.596638] ? hrtimer_interrupt+0x104/0x240 [25682.600917] ? __local_bh_enable+0x50/0x60 [25682.605010] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25682.610225] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25682.615182] [25682.617297] [25682.619394] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25682.624702] ? lock_sync+0xa0/0xa0 [25682.628105] _raw_spin_lock+0x30/0x40 [25682.631763] ? igrab+0x1d/0x50 [25682.634814] igrab+0x1d/0x50 [25682.637692] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25682.642842] delayed_work+0x1d8/0x280 [ceph] [25682.647122] ? process_scheduled_works+0x22a/0x510 [25682.651921] process_scheduled_works+0x2a4/0x510 [25682.656537] worker_thread+0x188/0x340 [25682.660286] ? rescuer_thread+0x310/0x310 [25682.664289] kthread+0xf5/0x130 [25682.667426] ? kthread_complete_and_exit+0x20/0x20 [25682.672211] ret_from_fork+0x31/0x40 [25682.675780] ? kthread_complete_and_exit+0x20/0x20 [25682.680565] ret_from_fork_asm+0x11/0x20 [25682.684488] [25682.850895] rcu: INFO: rcu_preempt self-detected stall on CPU [25682.856651] rcu: 0-....: (13437577 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=6715251 [25682.867077] rcu: (t=13745997 jiffies g=1309609 q=423822 ncpus=8) [25682.873161] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25682.882104] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25682.889573] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25682.895051] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25682.913788] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25682.919007] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25682.926130] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25682.933256] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25682.940380] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25682.947504] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25682.954627] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25682.962704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25682.968442] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25682.975566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25682.982690] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25682.989816] Call Trace: [25682.992259] [25682.994271] ? rcu_dump_cpu_stacks+0xd1/0x180 [25682.998629] ? rcu_sched_clock_irq+0x633/0x1390 [25683.003155] ? lock_is_held_type+0xaa/0x120 [25683.007332] ? lock_is_held_type+0xaa/0x120 [25683.011510] ? tick_sched_do_timer+0xa0/0xa0 [25683.015780] ? rcu_read_lock_sched_held+0x56/0x90 [25683.020480] ? tick_sched_do_timer+0xa0/0xa0 [25683.024743] ? update_process_times+0x5f/0x90 [25683.029093] ? tick_sched_handle.isra.16+0x1e/0x50 [25683.033879] ? tick_sched_timer+0x6f/0x80 [25683.037882] ? __hrtimer_run_queues+0x1d3/0x360 [25683.042409] ? hrtimer_interrupt+0x104/0x240 [25683.046679] ? __local_bh_enable+0x50/0x60 [25683.050772] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25683.055996] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25683.060952] [25683.063048] [25683.065148] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25683.070456] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25683.075333] do_raw_spin_lock+0xb5/0xc0 [25683.079170] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25683.083974] try_get_cap_refs+0x159/0x880 [ceph] [25683.088603] __ceph_get_caps+0x4a1/0x760 [ceph] [25683.093144] ? __file_remove_privs+0x89/0x110 [25683.097502] ? wake_bit_function+0x70/0x70 [25683.101594] ? ceph_write_iter+0x260/0x770 [ceph] [25683.106307] ceph_write_iter+0x260/0x770 [ceph] [25683.110843] ? lock_is_held_type+0xaa/0x120 [25683.115027] vfs_write+0x324/0x4f0 [25683.118436] ksys_pwrite64+0x65/0xa0 [25683.122009] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25683.127141] do_syscall_64+0x38/0x80 [25683.130711] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25683.135755] RIP: 0033:0x7f48f0eeb93b [25683.139325] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25683.158063] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [25683.165622] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25683.172745] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25683.179868] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25683.186992] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25683.194119] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25683.201248] [25683.203434] Sending NMI from CPU 0 to CPUs 7: [25683.207786] NMI backtrace for cpu 7 [25683.207788] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25683.207789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25683.207790] Workqueue: events delayed_work [ceph] [25683.207805] RIP: 0010:__lock_acquire+0x1c6/0x1e20 [25683.207807] Code: 03 c1 e2 02 41 09 f6 83 e1 f8 44 88 75 21 09 c1 09 d1 83 e1 e7 44 09 e9 44 09 c9 88 4d 22 0f b7 94 24 a0 00 00 00 0f b7 45 22 e2 05 83 e0 1f 09 d0 66 89 45 22 8b 84 24 a8 00 00 00 89 45 24 [25683.207808] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [25683.207809] RAX: 0000000000000004 RBX: ffff88810debaa80 RCX: 0000000000000004 [25683.207810] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [25683.207811] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [25683.207811] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [25683.207812] R13: 0000000000000000 R14: 0000000000000006 R15: ffff8881a1d34518 [25683.207813] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25683.207814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25683.207814] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25683.207815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25683.207816] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25683.207817] Call Trace: [25683.207817] [25683.207818] ? nmi_cpu_backtrace+0xad/0x120 [25683.207821] ? nmi_cpu_backtrace_handler+0x11/0x20 [25683.207822] ? nmi_handle+0xd4/0x260 [25683.207826] ? default_do_nmi+0x49/0xf0 [25683.207828] ? exc_nmi+0x11e/0x150 [25683.207829] ? end_repeat_nmi+0x16/0x67 [25683.207833] ? __lock_acquire+0x1c6/0x1e20 [25683.207834] ? __lock_acquire+0x1c6/0x1e20 [25683.207836] ? __lock_acquire+0x1c6/0x1e20 [25683.207837] [25683.207837] [25683.207839] lock_acquire+0xcb/0x2b0 [25683.207841] ? igrab+0x1d/0x50 [25683.207842] ? igrab+0x43/0x50 [25683.207843] ? lock_release+0xc5/0x260 [25683.207845] _raw_spin_lock+0x30/0x40 [25683.207847] ? igrab+0x1d/0x50 [25683.207848] igrab+0x1d/0x50 [25683.207849] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25683.207863] delayed_work+0x1d8/0x280 [ceph] [25683.207876] ? process_scheduled_works+0x22a/0x510 [25683.207878] process_scheduled_works+0x2a4/0x510 [25683.207882] worker_thread+0x188/0x340 [25683.207884] ? rescuer_thread+0x310/0x310 [25683.207887] kthread+0xf5/0x130 [25683.207889] ? kthread_complete_and_exit+0x20/0x20 [25683.207891] ret_from_fork+0x31/0x40 [25683.207893] ? kthread_complete_and_exit+0x20/0x20 [25683.207895] ret_from_fork_asm+0x11/0x20 [25683.207899] [25709.908002] watchdog: BUG: soft lockup - CPU#0 stuck for 12830s! [dbench:78683] [25709.915305] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25709.920001] watchdog: BUG: soft lockup - CPU#7 stuck for 21827s! [kworker/7:4:78808] [25709.970753] irq event stamp: 41476808 [25709.978484] Modules linked in: ceph libceph dns_resolver fscache netfs [25709.982143] hardirqs last enabled at (41476807): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25709.988674] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [25709.998834] hardirqs last disabled at (41476808): [] sysvec_apic_timer_interrupt+0xe/0x90 [25709.998838] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [25710.014643] softirqs last enabled at (41476788): [] __do_softirq+0x2be/0x371 [25710.022395] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25710.031084] softirqs last disabled at (41476783): [] irq_exit_rcu+0xb0/0xd0 [25710.037611] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [25710.046121] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25710.046122] kvm [25710.046123] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25710.051772] irqbypass i2c_i801 [25710.060717] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25710.062552] iTCO_wdt iTCO_vendor_support [25710.070025] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25710.073162] i2c_smbus [25710.078637] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25710.082644] crct10dif_pclmul joydev [25710.101396] [25710.101397] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25710.103754] crc32_pclmul ipmi_ssif [25710.108971] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25710.112542] ghash_clmulni_intel mei_me [25710.114034] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25710.121158] mei [25710.124642] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25710.131765] lpc_ich [25710.135597] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25710.142720] sha512_ssse3 acpi_ipmi [25710.144559] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25710.151687] mfd_core [25710.153866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25710.160991] wmi [25710.164475] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25710.172553] ipmi_si [25710.174824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25710.180561] ipmi_devintf [25710.182399] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25710.189521] ipmi_msghandler [25710.191707] Call Trace: [25710.198829] ioatdma acpi_pad [25710.201450] [25710.208572] acpi_power_meter [25710.211451] ? watchdog_timer_fn+0x1b4/0x220 [25710.213894] ixgbe igb [25710.216858] ? __hrtimer_run_queues+0x1d3/0x360 [25710.218867] mdio crc32c_intel [25710.221836] ? hrtimer_interrupt+0x104/0x240 [25710.226097] nvme ptp [25710.228452] ? __local_bh_enable+0x50/0x60 [25710.232976] nvme_core i2c_algo_bit [25710.236031] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25710.240293] pps_core [25710.242563] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25710.246653] dca fuse [25710.250155] [25710.255372] [25710.255373] irq event stamp: 127682038 [25710.257677] [25710.262635] hardirqs last enabled at (127682037): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25710.264906] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25710.267004] hardirqs last disabled at (127682038): [] sysvec_apic_timer_interrupt+0xe/0x90 [25710.268498] ? queued_spin_lock_slowpath+0x16/0x2e0 [25710.272239] softirqs last enabled at (127682036): [] __do_softirq+0x2be/0x371 [25710.274338] do_raw_spin_lock+0xb5/0xc0 [25710.284579] softirqs last disabled at (127682031): [] irq_exit_rcu+0xb0/0xd0 [25710.289885] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25710.299694] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25710.304569] try_get_cap_refs+0x159/0x880 [ceph] [25710.313337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25710.317171] __ceph_get_caps+0x4a1/0x760 [ceph] [25710.325763] Workqueue: events delayed_work [ceph] [25710.330554] ? __file_remove_privs+0x89/0x110 [25710.339926] [25710.344538] ? wake_bit_function+0x70/0x70 [25710.352008] RIP: 0010:lock_acquire+0xf0/0x2b0 [25710.356535] ? ceph_write_iter+0x260/0x770 [ceph] [25710.361228] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25710.365579] ceph_write_iter+0x260/0x770 [ceph] [25710.367070] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25710.371164] ? lock_is_held_type+0xaa/0x120 [25710.375513] [25710.375513] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25710.380212] vfs_write+0x324/0x4f0 [25710.398946] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25710.403476] ksys_pwrite64+0x65/0xa0 [25710.408690] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25710.412866] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25710.414356] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25710.421483] do_syscall_64+0x38/0x80 [25710.424877] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25710.432003] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25710.435573] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25710.442699] RIP: 0033:0x7f48f0eeb93b [25710.447827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25710.454953] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25710.458524] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25710.465646] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25710.470692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25710.478769] ORIG_RAX: 0000000000000012 [25710.482340] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25710.488078] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25710.506814] Call Trace: [25710.506815] [25710.513939] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25710.519156] ? watchdog_timer_fn+0x1b4/0x220 [25710.526279] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25710.530112] ? __hrtimer_run_queues+0x1d3/0x360 [25710.537236] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25710.544362] ? hrtimer_interrupt+0x104/0x240 [25710.546802] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25710.548815] ? __local_bh_enable+0x50/0x60 [25710.555942] [25710.560204] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25710.601855] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25710.606813] [25710.608909] [25710.611008] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25710.616316] ? lock_acquire+0xf0/0x2b0 [25710.620067] ? igrab+0x43/0x50 [25710.623124] ? lock_release+0xc5/0x260 [25710.626870] _raw_spin_lock+0x30/0x40 [25710.630534] ? igrab+0x1d/0x50 [25710.633584] igrab+0x1d/0x50 [25710.636465] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25710.641613] delayed_work+0x1d8/0x280 [ceph] [25710.645894] ? process_scheduled_works+0x22a/0x510 [25710.650687] process_scheduled_works+0x2a4/0x510 [25710.655309] worker_thread+0x188/0x340 [25710.659058] ? rescuer_thread+0x310/0x310 [25710.663060] kthread+0xf5/0x130 [25710.666198] ? kthread_complete_and_exit+0x20/0x20 [25710.670983] ret_from_fork+0x31/0x40 [25710.674561] ? kthread_complete_and_exit+0x20/0x20 [25710.679346] ret_from_fork_asm+0x11/0x20 [25710.683278] [25737.908488] watchdog: BUG: soft lockup - CPU#0 stuck for 12856s! [dbench:78683] [25737.915792] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25737.920488] watchdog: BUG: soft lockup - CPU#7 stuck for 21853s! [kworker/7:4:78808] [25737.971232] irq event stamp: 41534948 [25737.971233] hardirqs last enabled at (41534947): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25737.978964] Modules linked in: [25737.982620] hardirqs last disabled at (41534948): [] sysvec_apic_timer_interrupt+0xe/0x90 [25737.992778] ceph [25737.995828] softirqs last enabled at (41534930): [] __do_softirq+0x2be/0x371 [25738.005553] libceph [25738.007476] softirqs last disabled at (41534925): [] irq_exit_rcu+0xb0/0xd0 [25738.016165] dns_resolver [25738.018346] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25738.026855] fscache netfs [25738.029473] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25738.038417] veth nft_chain_nat [25738.041121] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25738.048592] xt_MASQUERADE nf_nat [25738.051729] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25738.057208] nf_conntrack nf_defrag_ipv6 [25738.060518] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25738.079257] nf_defrag_ipv4 nft_compat [25738.083173] [25738.083174] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25738.088389] nf_tables [25738.092134] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25738.093625] nfnetlink bridge [25738.100749] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25738.103107] stp llc [25738.110232] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25738.113194] xfs libcrc32c [25738.120318] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25738.122502] sunrpc intel_rapl_msr [25738.129628] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25738.132330] intel_rapl_common sb_edac [25738.139456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25738.142852] x86_pkg_temp_thermal coretemp [25738.150931] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25738.154675] kvm_intel kvm [25738.160410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25738.164502] irqbypass i2c_i801 [25738.171627] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25738.174330] iTCO_wdt iTCO_vendor_support [25738.181454] Call Trace: [25738.181456] [25738.184592] i2c_smbus [25738.191716] ? watchdog_timer_fn+0x1b4/0x220 [25738.195720] crct10dif_pclmul joydev [25738.198168] ? __hrtimer_run_queues+0x1d3/0x360 [25738.200181] crc32_pclmul ipmi_ssif [25738.202545] ? hrtimer_interrupt+0x104/0x240 [25738.206804] ghash_clmulni_intel [25738.210376] ? __local_bh_enable+0x50/0x60 [25738.214901] mei_me mei [25738.218387] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25738.222649] lpc_ich sha512_ssse3 [25738.225887] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25738.229980] acpi_ipmi mfd_core [25738.232424] [25738.237642] wmi [25738.240951] [25738.245911] ipmi_si ipmi_devintf [25738.249048] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25738.251145] ipmi_msghandler [25738.252986] ? queued_spin_lock_slowpath+0x16/0x2e0 [25738.255078] ioatdma acpi_pad [25738.258392] do_raw_spin_lock+0xb5/0xc0 [25738.263694] acpi_power_meter ixgbe [25738.266573] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25738.271443] igb mdio [25738.274408] try_get_cap_refs+0x159/0x880 [ceph] [25738.278237] crc32c_intel nvme [25738.281727] __ceph_get_caps+0x4a1/0x760 [ceph] [25738.286506] ptp nvme_core [25738.288780] ? __file_remove_privs+0x89/0x110 [25738.293387] i2c_algo_bit pps_core [25738.296438] ? wake_bit_function+0x70/0x70 [25738.300960] dca fuse [25738.303668] ? ceph_write_iter+0x260/0x770 [ceph] [25738.308016] [25738.308017] irq event stamp: 127902998 [25738.311415] ceph_write_iter+0x260/0x770 [ceph] [25738.315505] hardirqs last enabled at (127902997): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25738.317778] ? lock_is_held_type+0xaa/0x120 [25738.322473] hardirqs last disabled at (127902998): [] sysvec_apic_timer_interrupt+0xe/0x90 [25738.323965] vfs_write+0x324/0x4f0 [25738.327707] softirqs last enabled at (127902996): [] __do_softirq+0x2be/0x371 [25738.332237] ksys_pwrite64+0x65/0xa0 [25738.342477] softirqs last disabled at (127902991): [] irq_exit_rcu+0xb0/0xd0 [25738.346653] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25738.356463] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25738.359864] do_syscall_64+0x38/0x80 [25738.368650] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25738.372221] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25738.380818] Workqueue: events delayed_work [ceph] [25738.385948] RIP: 0033:0x7f48f0eeb93b [25738.395326] [25738.395327] RIP: 0010:lock_release+0x14e/0x260 [25738.398896] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25738.406367] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [25738.411412] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25738.416109] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [25738.419680] ORIG_RAX: 0000000000000012 [25738.421171] [25738.421172] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [25738.425607] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25738.444346] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [25738.463082] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25738.468302] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [25738.473518] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25738.477349] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [25738.478839] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25738.485963] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [25738.493088] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25738.500212] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25738.507342] [25738.514461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25738.566098] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25738.573221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25738.580346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25738.587470] Call Trace: [25738.589914] [25738.591925] ? watchdog_timer_fn+0x1b4/0x220 [25738.596190] ? __hrtimer_run_queues+0x1d3/0x360 [25738.600717] ? hrtimer_interrupt+0x104/0x240 [25738.604986] ? __local_bh_enable+0x50/0x60 [25738.609079] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25738.614303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25738.619259] [25738.621358] [25738.623455] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25738.628759] ? igrab+0x43/0x50 [25738.631814] ? lock_release+0x14e/0x260 [25738.635649] ? lock_release+0xc5/0x260 [25738.639394] _raw_spin_unlock+0x1b/0x40 [25738.643224] igrab+0x43/0x50 [25738.646103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25738.651253] delayed_work+0x1d8/0x280 [ceph] [25738.655533] ? process_scheduled_works+0x22a/0x510 [25738.660323] process_scheduled_works+0x2a4/0x510 [25738.664939] worker_thread+0x188/0x340 [25738.668688] ? rescuer_thread+0x310/0x310 [25738.672690] kthread+0xf5/0x130 [25738.675828] ? kthread_complete_and_exit+0x20/0x20 [25738.680613] ret_from_fork+0x31/0x40 [25738.684182] ? kthread_complete_and_exit+0x20/0x20 [25738.688968] ret_from_fork_asm+0x11/0x20 [25738.692900] [25765.908977] watchdog: BUG: soft lockup - CPU#0 stuck for 12882s! [dbench:78683] [25765.916279] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25765.920978] watchdog: BUG: soft lockup - CPU#7 stuck for 21879s! [kworker/7:4:78808] [25765.971719] irq event stamp: 41593160 [25765.971720] hardirqs last enabled at (41593159): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25765.979450] Modules linked in: [25765.983107] hardirqs last disabled at (41593160): [] sysvec_apic_timer_interrupt+0xe/0x90 [25765.993265] ceph [25765.996315] softirqs last enabled at (41593142): [] __do_softirq+0x2be/0x371 [25766.006041] libceph [25766.007964] softirqs last disabled at (41593137): [] irq_exit_rcu+0xb0/0xd0 [25766.016650] dns_resolver fscache [25766.018832] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25766.027342] netfs veth [25766.030671] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25766.030672] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25766.039615] nft_chain_nat xt_MASQUERADE [25766.042060] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25766.049530] nf_nat nf_conntrack [25766.055008] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25766.058924] nf_defrag_ipv6 nf_defrag_ipv4 [25766.077662] [25766.080886] nft_compat nf_tables [25766.086103] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25766.090194] nfnetlink [25766.091684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25766.094997] bridge stp [25766.102121] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25766.104479] llc xfs [25766.111601] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25766.114045] libcrc32c sunrpc [25766.121169] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25766.123354] intel_rapl_msr [25766.130496] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25766.133460] intel_rapl_common [25766.140584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25766.143374] sb_edac x86_pkg_temp_thermal [25766.151452] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25766.154504] coretemp kvm_intel [25766.160240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25766.164245] kvm [25766.171373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25766.174514] irqbypass i2c_i801 [25766.181637] Call Trace: [25766.181638] [25766.183492] iTCO_wdt [25766.190619] ? watchdog_timer_fn+0x1b4/0x220 [25766.193754] iTCO_vendor_support i2c_smbus [25766.196199] ? __hrtimer_run_queues+0x1d3/0x360 [25766.198209] crct10dif_pclmul joydev [25766.200484] ? hrtimer_interrupt+0x104/0x240 [25766.204745] crc32_pclmul [25766.208836] ? __local_bh_enable+0x50/0x60 [25766.213377] ipmi_ssif [25766.216950] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25766.221211] ghash_clmulni_intel mei_me [25766.223830] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25766.227919] mei lpc_ich [25766.230276] [25766.235494] sha512_ssse3 acpi_ipmi [25766.239326] [25766.244281] mfd_core wmi [25766.246815] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25766.248909] ipmi_si ipmi_devintf [25766.252400] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25766.254491] ipmi_msghandler [25766.257111] do_raw_spin_lock+0xb5/0xc0 [25766.262413] ioatdma acpi_pad [25766.265725] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25766.270596] acpi_power_meter [25766.273473] try_get_cap_refs+0x159/0x880 [ceph] [25766.277303] ixgbe igb [25766.280271] __ceph_get_caps+0x4a1/0x760 [ceph] [25766.285052] mdio crc32c_intel [25766.288018] ? __file_remove_privs+0x89/0x110 [25766.292626] nvme [25766.294985] ? wake_bit_function+0x70/0x70 [25766.299508] ptp nvme_core [25766.302560] ? ceph_write_iter+0x260/0x770 [ceph] [25766.306910] i2c_algo_bit [25766.308833] ceph_write_iter+0x260/0x770 [ceph] [25766.312924] pps_core dca [25766.315630] ? lock_is_held_type+0xaa/0x120 [25766.320324] fuse [25766.322944] vfs_write+0x324/0x4f0 [25766.327466] irq event stamp: 128124068 [25766.330088] ksys_pwrite64+0x65/0xa0 [25766.334260] hardirqs last enabled at (128124067): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25766.336184] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25766.339583] hardirqs last disabled at (128124068): [] sysvec_apic_timer_interrupt+0xe/0x90 [25766.343328] do_syscall_64+0x38/0x80 [25766.346896] softirqs last enabled at (128124066): [] __do_softirq+0x2be/0x371 [25766.357143] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25766.362272] softirqs last disabled at (128124061): [] irq_exit_rcu+0xb0/0xd0 [25766.372084] RIP: 0033:0x7f48f0eeb93b [25766.375654] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25766.384442] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25766.389486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25766.398084] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25766.401665] Workqueue: events delayed_work [ceph] [25766.411042] ORIG_RAX: 0000000000000012 [25766.411043] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25766.429779] [25766.429780] RIP: 0010:ceph_check_delayed_caps+0xc0/0x270 [ceph] [25766.437250] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25766.442467] Code: 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 <48> 8b 85 78 04 00 00 49 39 c5 75 89 31 c0 48 8b 3c 24 48 89 44 24 [25766.447164] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25766.450994] RSP: 0018:ffffc90002117db8 EFLAGS: 00000246 [25766.458118] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25766.459609] [25766.459610] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [25766.465521] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25766.465528] [25766.472660] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [25766.535929] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [25766.543054] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [25766.550179] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25766.557302] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25766.565383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25766.571118] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25766.578242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25766.585381] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25766.592508] Call Trace: [25766.594953] [25766.596963] ? watchdog_timer_fn+0x1b4/0x220 [25766.601226] ? __hrtimer_run_queues+0x1d3/0x360 [25766.605754] ? hrtimer_interrupt+0x104/0x240 [25766.610023] ? __local_bh_enable+0x50/0x60 [25766.614117] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25766.619340] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25766.624298] [25766.626393] [25766.628493] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25766.633801] ? ceph_check_delayed_caps+0xc0/0x270 [ceph] [25766.639118] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [25766.644433] delayed_work+0x1d8/0x280 [ceph] [25766.648714] ? process_scheduled_works+0x22a/0x510 [25766.653504] process_scheduled_works+0x2a4/0x510 [25766.658118] worker_thread+0x188/0x340 [25766.661870] ? rescuer_thread+0x310/0x310 [25766.665872] kthread+0xf5/0x130 [25766.669011] ? kthread_complete_and_exit+0x20/0x20 [25766.673795] ret_from_fork+0x31/0x40 [25766.677384] ? kthread_complete_and_exit+0x20/0x20 [25766.682175] ret_from_fork_asm+0x11/0x20 [25766.686097] [25793.909467] watchdog: BUG: soft lockup - CPU#0 stuck for 12908s! [dbench:78683] [25793.916774] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25793.921467] watchdog: BUG: soft lockup - CPU#7 stuck for 21905s! [kworker/7:4:78808] [25793.972215] irq event stamp: 41651450 [25793.972216] hardirqs last enabled at (41651449): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25793.979946] Modules linked in: [25793.983604] hardirqs last disabled at (41651450): [] sysvec_apic_timer_interrupt+0xe/0x90 [25793.993761] ceph [25793.996813] softirqs last enabled at (41651434): [] __do_softirq+0x2be/0x371 [25794.006537] libceph [25794.008459] softirqs last disabled at (41651429): [] irq_exit_rcu+0xb0/0xd0 [25794.017144] dns_resolver fscache [25794.019328] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25794.027838] netfs veth [25794.031166] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25794.040110] nft_chain_nat xt_MASQUERADE [25794.042558] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25794.050025] nf_nat nf_conntrack [25794.053942] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25794.059420] nf_defrag_ipv6 [25794.062644] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25794.081381] nf_defrag_ipv4 nft_compat [25794.084175] [25794.089390] nf_tables nfnetlink [25794.093151] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25794.094642] bridge stp [25794.097868] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25794.104991] llc xfs [25794.107434] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25794.114560] libcrc32c sunrpc [25794.116744] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25794.123868] intel_rapl_msr intel_rapl_common [25794.126831] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25794.133956] sb_edac x86_pkg_temp_thermal [25794.138306] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25794.145431] coretemp kvm_intel [25794.149435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25794.157513] kvm irqbypass [25794.160649] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25794.166386] i2c_i801 iTCO_wdt [25794.169092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25794.176216] iTCO_vendor_support i2c_smbus [25794.179266] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25794.186389] crct10dif_pclmul joydev [25794.190481] Call Trace: [25794.197605] crc32_pclmul ipmi_ssif [25794.201179] [25794.203618] ghash_clmulni_intel mei_me [25794.207107] ? watchdog_timer_fn+0x1b4/0x220 [25794.209115] mei lpc_ich [25794.212947] ? __hrtimer_run_queues+0x1d3/0x360 [25794.217208] sha512_ssse3 acpi_ipmi [25794.219743] ? hrtimer_interrupt+0x104/0x240 [25794.224265] mfd_core wmi [25794.227747] ? __local_bh_enable+0x50/0x60 [25794.232012] ipmi_si [25794.234632] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25794.238720] ipmi_devintf ipmi_msghandler [25794.240904] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25794.246123] ioatdma acpi_pad [25794.250126] [25794.255100] acpi_power_meter ixgbe [25794.258064] [25794.258066] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25794.260163] igb mdio [25794.263649] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25794.265743] crc32c_intel [25794.271050] do_raw_spin_lock+0xb5/0xc0 [25794.273319] nvme ptp [25794.278192] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25794.280805] nvme_core [25794.284641] try_get_cap_refs+0x159/0x880 [ceph] [25794.286908] i2c_algo_bit [25794.291699] __ceph_get_caps+0x4a1/0x760 [ceph] [25794.294049] pps_core [25794.298664] ? __file_remove_privs+0x89/0x110 [25794.301276] dca fuse [25794.305801] ? wake_bit_function+0x70/0x70 [25794.308071] [25794.308072] irq event stamp: 128344986 [25794.312426] ? ceph_write_iter+0x260/0x770 [ceph] [25794.314695] hardirqs last enabled at (128344985): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25794.318784] ceph_write_iter+0x260/0x770 [ceph] [25794.320275] hardirqs last disabled at (128344986): [] sysvec_apic_timer_interrupt+0xe/0x90 [25794.324024] ? lock_is_held_type+0xaa/0x120 [25794.328716] softirqs last enabled at (128344984): [] __do_softirq+0x2be/0x371 [25794.338962] vfs_write+0x324/0x4f0 [25794.343484] softirqs last disabled at (128344979): [] irq_exit_rcu+0xb0/0xd0 [25794.353302] ksys_pwrite64+0x65/0xa0 [25794.357473] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25794.366245] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25794.369641] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25794.378241] do_syscall_64+0x38/0x80 [25794.381810] Workqueue: events delayed_work [ceph] [25794.391187] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25794.396318] [25794.396319] RIP: 0010:lock_acquire+0xf0/0x2b0 [25794.403791] RIP: 0033:0x7f48f0eeb93b [25794.407359] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25794.412058] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25794.417100] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25794.418592] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25794.422943] [25794.422944] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25794.426514] ORIG_RAX: 0000000000000012 [25794.426515] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25794.445251] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25794.463988] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25794.469205] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25794.474422] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25794.475915] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25794.483038] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25794.486868] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25794.493994] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25794.501118] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25794.508247] [25794.515365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25794.566995] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25794.574118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25794.581241] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25794.588365] Call Trace: [25794.590811] [25794.592822] ? watchdog_timer_fn+0x1b4/0x220 [25794.597085] ? __hrtimer_run_queues+0x1d3/0x360 [25794.601612] ? hrtimer_interrupt+0x104/0x240 [25794.605873] ? __local_bh_enable+0x50/0x60 [25794.609966] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25794.615181] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25794.620152] [25794.622253] [25794.624350] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25794.629659] ? lock_acquire+0xf0/0x2b0 [25794.633410] ? igrab+0x43/0x50 [25794.636459] ? lock_release+0xc5/0x260 [25794.640205] _raw_spin_lock+0x30/0x40 [25794.643878] ? igrab+0x1d/0x50 [25794.646928] igrab+0x1d/0x50 [25794.649809] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25794.654958] delayed_work+0x1d8/0x280 [ceph] [25794.659237] ? process_scheduled_works+0x22a/0x510 [25794.664026] process_scheduled_works+0x2a4/0x510 [25794.668643] worker_thread+0x188/0x340 [25794.672393] ? rescuer_thread+0x310/0x310 [25794.676396] kthread+0xf5/0x130 [25794.679544] ? kthread_complete_and_exit+0x20/0x20 [25794.684335] ret_from_fork+0x31/0x40 [25794.687915] ? kthread_complete_and_exit+0x20/0x20 [25794.692697] ret_from_fork_asm+0x11/0x20 [25794.696621] [25821.909957] watchdog: BUG: soft lockup - CPU#0 stuck for 12934s! [dbench:78683] [25821.917263] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25821.921956] watchdog: BUG: soft lockup - CPU#7 stuck for 21931s! [kworker/7:4:78808] [25821.972704] irq event stamp: 41709654 [25821.972705] hardirqs last enabled at (41709653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25821.980433] Modules linked in: ceph [25821.984091] hardirqs last disabled at (41709654): [] sysvec_apic_timer_interrupt+0xe/0x90 [25821.994249] libceph [25821.997732] softirqs last enabled at (41709636): [] __do_softirq+0x2be/0x371 [25822.007456] dns_resolver [25822.009639] softirqs last disabled at (41709631): [] irq_exit_rcu+0xb0/0xd0 [25822.018325] fscache netfs [25822.020959] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25822.029469] veth nft_chain_nat [25822.032175] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25822.032176] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25822.041120] xt_MASQUERADE nf_nat [25822.044256] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25822.051726] nf_conntrack nf_defrag_ipv6 [25822.057204] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25822.060515] nf_defrag_ipv4 nft_compat [25822.079253] [25822.079254] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25822.083170] nf_tables nfnetlink [25822.088388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25822.092130] bridge [25822.093631] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25822.100755] stp llc [25822.103978] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25822.111103] xfs libcrc32c [25822.113200] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25822.120324] sunrpc intel_rapl_msr [25822.122508] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25822.129643] intel_rapl_common sb_edac [25822.132354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25822.139477] x86_pkg_temp_thermal coretemp [25822.142874] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25822.150952] kvm_intel kvm [25822.154698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25822.160434] irqbypass i2c_i801 [25822.164524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25822.171651] iTCO_wdt [25822.174354] Call Trace: [25822.174355] [25822.181479] iTCO_vendor_support i2c_smbus [25822.184616] ? watchdog_timer_fn+0x1b4/0x220 [25822.191740] crct10dif_pclmul [25822.194010] ? __hrtimer_run_queues+0x1d3/0x360 [25822.196454] joydev crc32_pclmul [25822.198469] ? hrtimer_interrupt+0x104/0x240 [25822.202556] ipmi_ssif ghash_clmulni_intel [25822.206821] ? __local_bh_enable+0x50/0x60 [25822.209783] mei_me mei [25822.214311] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25822.217532] lpc_ich sha512_ssse3 [25822.221795] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25822.225887] acpi_ipmi mfd_core [25822.229977] [25822.232422] wmi ipmi_si [25822.237638] [25822.240949] ipmi_devintf [25822.245909] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25822.249045] ipmi_msghandler ioatdma [25822.251146] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25822.253673] acpi_pad acpi_power_meter [25822.255773] do_raw_spin_lock+0xb5/0xc0 [25822.258387] ixgbe igb [25822.263692] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25822.267261] mdio crc32c_intel [25822.272136] try_get_cap_refs+0x159/0x880 [ceph] [25822.275877] nvme ptp [25822.279713] __ceph_get_caps+0x4a1/0x760 [ceph] [25822.282066] nvme_core i2c_algo_bit [25822.286854] ? __file_remove_privs+0x89/0x110 [25822.289901] pps_core dca [25822.294510] ? wake_bit_function+0x70/0x70 [25822.296780] fuse [25822.301309] ? ceph_write_iter+0x260/0x770 [ceph] [25822.304790] irq event stamp: 128565946 [25822.309140] ceph_write_iter+0x260/0x770 [ceph] [25822.311758] hardirqs last enabled at (128565945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25822.315852] ? lock_is_held_type+0xaa/0x120 [25822.317773] hardirqs last disabled at (128565946): [] sysvec_apic_timer_interrupt+0xe/0x90 [25822.322473] vfs_write+0x324/0x4f0 [25822.326213] softirqs last enabled at (128565944): [] __do_softirq+0x2be/0x371 [25822.330743] ksys_pwrite64+0x65/0xa0 [25822.340982] softirqs last disabled at (128565939): [] irq_exit_rcu+0xb0/0xd0 [25822.345160] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25822.354970] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25822.358368] do_syscall_64+0x38/0x80 [25822.367137] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25822.370709] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25822.379308] Workqueue: events delayed_work [ceph] [25822.384439] RIP: 0033:0x7f48f0eeb93b [25822.393814] [25822.393815] RIP: 0010:lock_release+0x42/0x260 [25822.397386] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25822.404857] Code: 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 <8b> 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 [25822.409900] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25822.414598] RSP: 0018:ffffc90002117d48 EFLAGS: 00000202 [25822.418169] ORIG_RAX: 0000000000000012 [25822.419659] [25822.419660] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [25822.424010] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25822.442747] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [25822.461485] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25822.466702] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [25822.471921] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25822.475750] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [25822.477241] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25822.484367] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [25822.491489] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25822.498613] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25822.505761] [25822.512881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25822.564499] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25822.571623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25822.578749] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25822.585872] Call Trace: [25822.588318] [25822.590329] ? watchdog_timer_fn+0x1b4/0x220 [25822.594593] ? __hrtimer_run_queues+0x1d3/0x360 [25822.599117] ? hrtimer_interrupt+0x104/0x240 [25822.603379] ? __local_bh_enable+0x50/0x60 [25822.607473] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25822.612697] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25822.617652] [25822.619751] [25822.621848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25822.627154] ? igrab+0x43/0x50 [25822.630205] ? igrab+0x43/0x50 [25822.633254] ? lock_release+0x42/0x260 [25822.637000] ? lock_release+0x1d7/0x260 [25822.640831] _raw_spin_unlock+0x1b/0x40 [25822.644662] igrab+0x43/0x50 [25822.647539] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25822.652687] delayed_work+0x1d8/0x280 [ceph] [25822.656968] ? process_scheduled_works+0x22a/0x510 [25822.661759] process_scheduled_works+0x2a4/0x510 [25822.666375] worker_thread+0x188/0x340 [25822.670124] ? rescuer_thread+0x310/0x310 [25822.674127] kthread+0xf5/0x130 [25822.677265] ? kthread_complete_and_exit+0x20/0x20 [25822.682049] ret_from_fork+0x31/0x40 [25822.685643] ? kthread_complete_and_exit+0x20/0x20 [25822.690429] ret_from_fork_asm+0x11/0x20 [25822.694369] [25849.910446] watchdog: BUG: soft lockup - CPU#0 stuck for 12960s! [dbench:78683] [25849.917750] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25849.922445] watchdog: BUG: soft lockup - CPU#7 stuck for 21957s! [kworker/7:4:78808] [25849.973200] irq event stamp: 41767854 [25849.973201] hardirqs last enabled at (41767853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25849.980929] Modules linked in: [25849.984587] hardirqs last disabled at (41767854): [] sysvec_apic_timer_interrupt+0xe/0x90 [25849.994745] ceph [25849.997795] softirqs last enabled at (41767836): [] __do_softirq+0x2be/0x371 [25850.007518] libceph [25850.009444] softirqs last disabled at (41767831): [] irq_exit_rcu+0xb0/0xd0 [25850.018128] dns_resolver fscache [25850.020311] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25850.028840] netfs [25850.032151] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25850.041096] veth nft_chain_nat [25850.043123] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25850.050594] xt_MASQUERADE nf_nat [25850.053731] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25850.059209] nf_conntrack nf_defrag_ipv6 [25850.062520] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25850.081256] nf_defrag_ipv4 [25850.085174] [25850.085175] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25850.090391] nft_compat [25850.093182] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25850.094673] nf_tables nfnetlink [25850.101797] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25850.104241] bridge [25850.111366] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25850.114590] stp [25850.121713] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25850.123810] llc [25850.130934] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25850.132771] xfs libcrc32c [25850.139896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25850.141735] sunrpc [25850.149812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25850.152514] intel_rapl_msr [25850.158253] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25850.160349] intel_rapl_common [25850.167475] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25850.170265] sb_edac x86_pkg_temp_thermal [25850.177390] Call Trace: [25850.180441] coretemp kvm_intel [25850.187568] [25850.191568] kvm irqbypass [25850.194014] ? watchdog_timer_fn+0x1b4/0x220 [25850.197149] i2c_i801 iTCO_wdt [25850.199160] ? __hrtimer_run_queues+0x1d3/0x360 [25850.201863] iTCO_vendor_support [25850.206135] ? hrtimer_interrupt+0x104/0x240 [25850.209179] i2c_smbus [25850.213702] ? __local_bh_enable+0x50/0x60 [25850.216926] crct10dif_pclmul joydev [25850.221193] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25850.223548] crc32_pclmul ipmi_ssif [25850.227641] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25850.231209] ghash_clmulni_intel mei_me [25850.236429] [25850.239913] mei [25850.244870] [25850.244871] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25850.248699] lpc_ich [25850.250802] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25850.252633] sha512_ssse3 [25850.254733] do_raw_spin_lock+0xb5/0xc0 [25850.260035] acpi_ipmi mfd_core [25850.262222] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25850.267091] wmi ipmi_si [25850.269712] try_get_cap_refs+0x159/0x880 [ceph] [25850.273540] ipmi_devintf ipmi_msghandler [25850.276682] __ceph_get_caps+0x4a1/0x760 [ceph] [25850.281461] ioatdma acpi_pad [25850.283995] ? __file_remove_privs+0x89/0x110 [25850.288601] acpi_power_meter [25850.292606] ? wake_bit_function+0x70/0x70 [25850.297138] ixgbe igb [25850.300105] ? ceph_write_iter+0x260/0x770 [ceph] [25850.304471] mdio crc32c_intel [25850.307436] ceph_write_iter+0x260/0x770 [ceph] [25850.311527] nvme [25850.313887] ? lock_is_held_type+0xaa/0x120 [25850.318580] ptp nvme_core [25850.321635] vfs_write+0x324/0x4f0 [25850.326155] i2c_algo_bit [25850.328085] ksys_pwrite64+0x65/0xa0 [25850.332256] pps_core [25850.334962] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25850.338359] dca [25850.340977] do_syscall_64+0x38/0x80 [25850.344547] fuse [25850.346819] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25850.351964] irq event stamp: 128786750 [25850.353804] RIP: 0033:0x7f48f0eeb93b [25850.357372] hardirqs last enabled at (128786749): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25850.359298] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25850.364342] hardirqs last disabled at (128786750): [] sysvec_apic_timer_interrupt+0xe/0x90 [25850.368084] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25850.371657] softirqs last enabled at (128786748): [] __do_softirq+0x2be/0x371 [25850.381899] ORIG_RAX: 0000000000000012 [25850.400637] softirqs last disabled at (128786743): [] irq_exit_rcu+0xb0/0xd0 [25850.410448] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25850.415666] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25850.424436] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25850.428268] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25850.436864] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25850.443988] Workqueue: events delayed_work [ceph] [25850.453368] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25850.460491] [25850.467961] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25850.475087] RIP: 0010:lock_acquire+0xf0/0x2b0 [25850.479790] [25850.486907] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25850.520794] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25850.526012] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25850.533140] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25850.540262] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25850.547385] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25850.554508] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25850.561634] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25850.569710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25850.575447] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25850.582572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25850.589696] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25850.596819] Call Trace: [25850.599266] [25850.601277] ? watchdog_timer_fn+0x1b4/0x220 [25850.605541] ? __hrtimer_run_queues+0x1d3/0x360 [25850.610066] ? hrtimer_interrupt+0x104/0x240 [25850.614328] ? __local_bh_enable+0x50/0x60 [25850.618420] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25850.623645] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25850.628603] [25850.630701] [25850.632797] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25850.638105] ? lock_acquire+0xf0/0x2b0 [25850.641863] ? igrab+0x43/0x50 [25850.644914] ? lock_release+0xc5/0x260 [25850.648659] _raw_spin_lock+0x30/0x40 [25850.652325] ? igrab+0x1d/0x50 [25850.655374] igrab+0x1d/0x50 [25850.658252] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25850.663402] delayed_work+0x1d8/0x280 [ceph] [25850.667685] ? process_scheduled_works+0x22a/0x510 [25850.672474] process_scheduled_works+0x2a4/0x510 [25850.677090] worker_thread+0x188/0x340 [25850.680855] ? rescuer_thread+0x310/0x310 [25850.684859] kthread+0xf5/0x130 [25850.688006] ? kthread_complete_and_exit+0x20/0x20 [25850.692798] ret_from_fork+0x31/0x40 [25850.696378] ? kthread_complete_and_exit+0x20/0x20 [25850.701161] ret_from_fork_asm+0x11/0x20 [25850.705094] [25877.910934] watchdog: BUG: soft lockup - CPU#0 stuck for 12986s! [dbench:78683] [25877.918237] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25877.922934] watchdog: BUG: soft lockup - CPU#7 stuck for 21983s! [kworker/7:4:78808] [25877.973678] irq event stamp: 41826196 [25877.973679] hardirqs last enabled at (41826195): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25877.981408] Modules linked in: [25877.985065] hardirqs last disabled at (41826196): [] sysvec_apic_timer_interrupt+0xe/0x90 [25877.995224] ceph [25877.998275] softirqs last enabled at (41826178): [] __do_softirq+0x2be/0x371 [25878.007998] libceph [25878.009921] softirqs last disabled at (41826173): [] irq_exit_rcu+0xb0/0xd0 [25878.018607] dns_resolver fscache [25878.020791] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25878.029300] netfs veth [25878.032626] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25878.041573] nft_chain_nat [25878.044018] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25878.051488] xt_MASQUERADE nf_nat [25878.054191] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25878.059670] nf_conntrack nf_defrag_ipv6 [25878.062979] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25878.081717] nf_defrag_ipv4 nft_compat [25878.085636] [25878.085637] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25878.090854] nf_tables [25878.094597] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25878.096104] nfnetlink bridge [25878.103230] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25878.105586] stp [25878.112727] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25878.115691] llc xfs [25878.122816] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25878.124655] libcrc32c sunrpc [25878.131777] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25878.133963] intel_rapl_msr intel_rapl_common [25878.141085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25878.144049] sb_edac x86_pkg_temp_thermal [25878.152126] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25878.156478] coretemp kvm_intel [25878.162216] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25878.166221] kvm irqbypass [25878.173343] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25878.176482] i2c_i801 iTCO_wdt [25878.183612] Call Trace: [25878.183613] [25878.186331] iTCO_vendor_support [25878.193460] ? watchdog_timer_fn+0x1b4/0x220 [25878.196509] i2c_smbus crct10dif_pclmul [25878.198956] ? __hrtimer_run_queues+0x1d3/0x360 [25878.200965] joydev crc32_pclmul [25878.204193] ? hrtimer_interrupt+0x104/0x240 [25878.208453] ipmi_ssif ghash_clmulni_intel [25878.212284] ? __local_bh_enable+0x50/0x60 [25878.216809] mei_me [25878.217307] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [25878.217309] rcu: 0-....: (13628627 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=6811200 [25878.217333] rcu: 7-....: (13538936 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/13537644 fqs=6811200 [25878.217336] rcu: (detected by 3, t=13941344 jiffies, g=1309609, q=427910 ncpus=8) [25878.217339] Sending NMI from CPU 3 to CPUs 0: [25878.217342] NMI backtrace for cpu 0 [25878.217343] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25878.217344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25878.217345] RIP: 0010:delay_tsc+0x3b/0x90 [25878.217347] Code: 44 8b 0d 54 f6 49 7e 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e <65> 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 66 90 48 c1 [25878.217348] RSP: 0018:ffffc90000003bc0 EFLAGS: 00000007 [25878.217349] RAX: 000053fa00aa74a5 RBX: 0000000000000000 RCX: 0000000000000000 [25878.217350] RDX: 0000000000000106 RSI: 0000000000000000 RDI: 0000000000000dac [25878.217351] RBP: 0000000000002655 R08: 000053fa00aa739f R09: 0000000000000000 [25878.217351] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [25878.217352] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [25878.217353] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25878.217354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25878.217354] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25878.217355] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25878.217356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25878.217356] Call Trace: [25878.217357] [25878.217358] ? nmi_cpu_backtrace+0xad/0x120 [25878.217361] ? nmi_cpu_backtrace_handler+0x11/0x20 [25878.217363] ? nmi_handle+0xd4/0x260 [25878.217367] ? default_do_nmi+0x49/0xf0 [25878.217369] ? exc_nmi+0x11e/0x150 [25878.217370] ? end_repeat_nmi+0x16/0x67 [25878.217374] ? delay_tsc+0x3b/0x90 [25878.217376] ? delay_tsc+0x3b/0x90 [25878.217378] ? delay_tsc+0x3b/0x90 [25878.217380] [25878.217380] [25878.217380] wait_for_lsr+0x27/0x70 [25878.217384] serial8250_console_write+0x341/0x3f0 [25878.217387] ? lock_release+0xc5/0x260 [25878.217390] console_flush_all+0x227/0x5a0 [25878.217392] ? console_flush_all+0x1ff/0x5a0 [25878.217393] ? console_flush_all+0x1ff/0x5a0 [25878.217396] console_unlock+0x7b/0x120 [25878.217398] vprintk_emit+0x246/0x340 [25878.217400] ? __local_bh_enable+0x50/0x60 [25878.217403] ? __local_bh_enable+0x50/0x60 [25878.217405] _printk+0x5c/0x80 [25878.217407] ? __local_bh_enable+0x50/0x60 [25878.217410] show_trace_log_lvl+0x1ac/0x3f0 [25878.217415] ? do_raw_spin_lock+0xb5/0xc0 [25878.217417] ? touch_softlockup_watchdog+0x20/0x20 [25878.217418] watchdog_timer_fn+0x1b4/0x220 [25878.217420] __hrtimer_run_queues+0x1d3/0x360 [25878.217423] hrtimer_interrupt+0x104/0x240 [25878.217424] ? __local_bh_enable+0x50/0x60 [25878.217427] __sysvec_apic_timer_interrupt+0x61/0x160 [25878.217430] sysvec_apic_timer_interrupt+0x6d/0x90 [25878.217432] [25878.217432] [25878.217433] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25878.217435] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [25878.217437] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [25878.217439] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25878.217439] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25878.217440] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25878.217441] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25878.217441] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25878.217442] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25878.217445] do_raw_spin_lock+0xb5/0xc0 [25878.217447] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25878.217462] try_get_cap_refs+0x159/0x880 [ceph] [25878.217477] __ceph_get_caps+0x4a1/0x760 [ceph] [25878.217490] ? __file_remove_privs+0x89/0x110 [25878.217491] ? wake_bit_function+0x70/0x70 [25878.217494] ? ceph_write_iter+0x260/0x770 [ceph] [25878.217505] ceph_write_iter+0x260/0x770 [ceph] [25878.217516] ? lock_is_held_type+0xaa/0x120 [25878.217519] vfs_write+0x324/0x4f0 [25878.217523] ksys_pwrite64+0x65/0xa0 [25878.217525] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25878.217528] do_syscall_64+0x38/0x80 [25878.217530] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25878.217532] RIP: 0033:0x7f48f0eeb93b [25878.217533] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25878.217534] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [25878.217535] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25878.217536] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25878.217536] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25878.217537] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25878.217538] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25878.217541] [25878.218341] Sending NMI from CPU 3 to CPUs 7: [25878.218344] NMI backtrace for cpu 7 [25878.218345] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25878.218347] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25878.218348] Workqueue: events delayed_work [ceph] [25878.218369] RIP: 0010:vprintk_emit+0x1eb/0x340 [25878.218372] Code: 50 41 b8 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 <0f> b6 11 84 d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 [25878.218374] RSP: 0018:ffffc90000264e08 EFLAGS: 00000002 [25878.218376] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [25878.218377] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb578 [25878.218378] RBP: 0000000000000007 R08: 0000000000005ef4 R09: 0000000000000004 [25878.218379] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [25878.218380] R13: 0000000000000000 R14: ffffffff82130806 R15: ffffc90000264e58 [25878.218382] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25878.218383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25878.218385] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25878.218386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25878.218387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25878.218388] Call Trace: [25878.218389] [25878.218391] ? nmi_cpu_backtrace+0xad/0x120 [25878.218395] ? nmi_cpu_backtrace_handler+0x11/0x20 [25878.218396] ? nmi_handle+0xd4/0x260 [25878.218401] ? default_do_nmi+0x49/0xf0 [25878.218404] ? exc_nmi+0x11e/0x150 [25878.218406] ? end_repeat_nmi+0x16/0x67 [25878.218413] ? vprintk_emit+0x1eb/0x340 [25878.218415] ? vprintk_emit+0x1eb/0x340 [25878.218419] ? vprintk_emit+0x1eb/0x340 [25878.218421] [25878.218422] [25878.218423] ? irq_work_queue+0x2c/0x50 [25878.218429] ? touch_softlockup_watchdog+0x20/0x20 [25878.218431] _printk+0x5c/0x80 [25878.218435] print_modules+0x6e/0xe0 [25878.218438] watchdog_timer_fn+0x19f/0x220 [25878.218441] __hrtimer_run_queues+0x1d3/0x360 [25878.218446] hrtimer_interrupt+0x104/0x240 [25878.218447] ? __local_bh_enable+0x50/0x60 [25878.218452] __sysvec_apic_timer_interrupt+0x61/0x160 [25878.218455] sysvec_apic_timer_interrupt+0x6d/0x90 [25878.218458] [25878.218458] [25878.218459] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25878.218462] RIP: 0010:lock_acquire+0xf0/0x2b0 [25878.218464] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25878.218466] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25878.218467] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25878.218468] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25878.218469] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25878.218470] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25878.218471] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25878.218478] ? igrab+0x43/0x50 [25878.218480] ? lock_release+0xc5/0x260 [25878.218483] _raw_spin_lock+0x30/0x40 [25878.218486] ? igrab+0x1d/0x50 [25878.218487] igrab+0x1d/0x50 [25878.218490] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25878.218510] delayed_work+0x1d8/0x280 [ceph] [25878.218529] ? process_scheduled_works+0x22a/0x510 [25878.218532] process_scheduled_works+0x2a4/0x510 [25878.218539] worker_thread+0x188/0x340 [25878.218542] ? rescuer_thread+0x310/0x310 [25878.218545] kthread+0xf5/0x130 [25878.218548] ? kthread_complete_and_exit+0x20/0x20 [25878.218551] ret_from_fork+0x31/0x40 [25878.218554] ? kthread_complete_and_exit+0x20/0x20 [25878.218557] ret_from_fork_asm+0x11/0x20 [25878.218564] [25878.220035] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25878.224295] mei lpc_ich [25878.228387] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25878.232477] sha512_ssse3 [25878.234575] [25878.240658] acpi_ipmi mfd_core [25878.251077] [25878.251078] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25878.261667] wmi ipmi_si [25878.269229] ? queued_spin_lock_slowpath+0x16/0x2e0 [25878.273576] ipmi_devintf ipmi_msghandler [25878.277062] do_raw_spin_lock+0xb5/0xc0 [25878.286005] ioatdma acpi_pad [25878.293475] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25878.297478] acpi_power_meter ixgbe [25878.316219] try_get_cap_refs+0x159/0x880 [ceph] [25878.321434] igb [25878.328562] __ceph_get_caps+0x4a1/0x760 [ceph] [25878.335681] mdio crc32c_intel [25878.342809] ? __file_remove_privs+0x89/0x110 [25878.349932] nvme ptp [25878.357056] ? wake_bit_function+0x70/0x70 [25878.365131] nvme_core i2c_algo_bit [25878.370873] ? ceph_write_iter+0x260/0x770 [ceph] [25878.377994] pps_core dca [25878.385117] ceph_write_iter+0x260/0x770 [ceph] [25878.392241] fuse [25878.394689] ? lock_is_held_type+0xaa/0x120 [25878.396695] irq event stamp: 129007676 [25878.400876] vfs_write+0x324/0x4f0 [25878.405658] hardirqs last enabled at (129007675): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25878.409233] ksys_pwrite64+0x65/0xa0 [25878.413059] hardirqs last disabled at (129007676): [] sysvec_apic_timer_interrupt+0xe/0x90 [25878.416458] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25878.420287] softirqs last enabled at (129007674): [] __do_softirq+0x2be/0x371 [25878.423686] do_syscall_64+0x38/0x80 [25878.427083] softirqs last disabled at (129007669): [] irq_exit_rcu+0xb0/0xd0 [25878.430480] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25878.432578] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25878.434587] RIP: 0033:0x7f48f0eeb93b [25878.438072] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25878.442771] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25878.446513] Workqueue: events delayed_work [ceph] [25878.450617] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25878.454886] [25878.454886] RIP: 0010:lock_acquire+0xf0/0x2b0 [25878.459150] ORIG_RAX: 0000000000000012 [25878.462894] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25878.466551] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25878.470642] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25878.474731] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25878.477783] [25878.477783] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25878.481873] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25878.486051] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25878.490055] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25878.494840] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25878.498929] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25878.503282] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25878.507377] [25878.511463] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25879.396285] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25879.404362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25879.410101] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25879.417223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25879.424353] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25879.431481] Call Trace: [25879.433927] [25879.435937] ? watchdog_timer_fn+0x1b4/0x220 [25879.440202] ? __hrtimer_run_queues+0x1d3/0x360 [25879.444728] ? hrtimer_interrupt+0x104/0x240 [25879.448996] ? __local_bh_enable+0x50/0x60 [25879.453089] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25879.458304] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25879.463262] [25879.465360] [25879.467459] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25879.472765] ? lock_acquire+0xf0/0x2b0 [25879.476516] ? igrab+0x43/0x50 [25879.479566] ? lock_release+0xc5/0x260 [25879.483311] _raw_spin_lock+0x30/0x40 [25879.486974] ? igrab+0x1d/0x50 [25879.490026] igrab+0x1d/0x50 [25879.492904] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25879.498054] delayed_work+0x1d8/0x280 [ceph] [25879.502352] ? process_scheduled_works+0x22a/0x510 [25879.507142] process_scheduled_works+0x2a4/0x510 [25879.511758] worker_thread+0x188/0x340 [25879.515508] ? rescuer_thread+0x310/0x310 [25879.519519] kthread+0xf5/0x130 [25879.522657] ? kthread_complete_and_exit+0x20/0x20 [25879.527441] ret_from_fork+0x31/0x40 [25879.531011] ? kthread_complete_and_exit+0x20/0x20 [25879.535796] ret_from_fork_asm+0x11/0x20 [25879.539720] [25905.911423] watchdog: BUG: soft lockup - CPU#0 stuck for 13012s! [dbench:78683] [25905.918725] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25905.923422] watchdog: BUG: soft lockup - CPU#7 stuck for 22009s! [kworker/7:4:78808] [25905.974175] irq event stamp: 41884404 [25905.974176] hardirqs last enabled at (41884403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25905.981904] Modules linked in: [25905.985563] hardirqs last disabled at (41884404): [] sysvec_apic_timer_interrupt+0xe/0x90 [25905.995720] ceph [25905.998770] softirqs last enabled at (41884386): [] __do_softirq+0x2be/0x371 [25906.008495] libceph dns_resolver [25906.010418] softirqs last disabled at (41884381): [] irq_exit_rcu+0xb0/0xd0 [25906.019104] fscache netfs [25906.022413] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25906.030923] veth nft_chain_nat [25906.033627] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25906.042573] xt_MASQUERADE nf_nat [25906.045710] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [25906.053180] nf_conntrack [25906.056491] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [25906.061969] nf_defrag_ipv6 nf_defrag_ipv4 [25906.064587] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25906.083324] nft_compat nf_tables [25906.087414] [25906.087415] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25906.092632] nfnetlink [25906.095941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25906.097432] bridge stp [25906.104556] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25906.106915] llc xfs [25906.114039] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25906.116483] libcrc32c [25906.123607] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25906.125791] sunrpc [25906.132916] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25906.135272] intel_rapl_msr [25906.142396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25906.144495] intel_rapl_common sb_edac [25906.152570] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25906.155361] x86_pkg_temp_thermal coretemp [25906.161101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25906.164845] kvm_intel [25906.171967] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25906.176058] kvm irqbypass [25906.183182] Call Trace: [25906.183184] [25906.185540] i2c_i801 iTCO_wdt [25906.192665] ? watchdog_timer_fn+0x1b4/0x220 [25906.195368] iTCO_vendor_support [25906.197823] ? __hrtimer_run_queues+0x1d3/0x360 [25906.199833] i2c_smbus [25906.202886] ? hrtimer_interrupt+0x104/0x240 [25906.207145] crct10dif_pclmul joydev [25906.210371] ? __local_bh_enable+0x50/0x60 [25906.214894] crc32_pclmul [25906.217254] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25906.221516] ipmi_ssif [25906.225105] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25906.229194] ghash_clmulni_intel [25906.231824] [25906.237039] mei_me [25906.239396] [25906.239398] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25906.244353] mei [25906.247582] ? queued_spin_lock_slowpath+0x2d/0x2e0 [25906.249674] lpc_ich sha512_ssse3 [25906.251773] do_raw_spin_lock+0xb5/0xc0 [25906.253869] acpi_ipmi mfd_core [25906.259174] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25906.261010] wmi ipmi_si [25906.265885] try_get_cap_refs+0x159/0x880 [ceph] [25906.269191] ipmi_devintf ipmi_msghandler [25906.273028] __ceph_get_caps+0x4a1/0x760 [ceph] [25906.276160] ioatdma acpi_pad [25906.280950] ? __file_remove_privs+0x89/0x110 [25906.283474] acpi_power_meter [25906.288086] ? wake_bit_function+0x70/0x70 [25906.292103] ixgbe igb [25906.296624] ? ceph_write_iter+0x260/0x770 [ceph] [25906.299586] mdio crc32c_intel [25906.303938] ceph_write_iter+0x260/0x770 [ceph] [25906.306901] nvme [25906.310996] ? lock_is_held_type+0xaa/0x120 [25906.313349] ptp [25906.318050] vfs_write+0x324/0x4f0 [25906.321100] nvme_core i2c_algo_bit [25906.325627] ksys_pwrite64+0x65/0xa0 [25906.327545] pps_core dca [25906.331724] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25906.333561] fuse [25906.336959] do_syscall_64+0x38/0x80 [25906.340441] [25906.344014] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25906.346631] irq event stamp: 129221770 [25906.351763] RIP: 0033:0x7f48f0eeb93b [25906.353685] hardirqs last enabled at (129221769): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25906.357257] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25906.358745] hardirqs last disabled at (129221770): [] sysvec_apic_timer_interrupt+0xe/0x90 [25906.363791] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25906.367534] softirqs last enabled at (129221768): [] __do_softirq+0x2be/0x371 [25906.371107] ORIG_RAX: 0000000000000012 [25906.381350] softirqs last disabled at (129221763): [] irq_exit_rcu+0xb0/0xd0 [25906.400102] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25906.409916] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25906.415134] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25906.423904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25906.427735] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25906.436332] Workqueue: events delayed_work [ceph] [25906.443455] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25906.452835] [25906.459957] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25906.467427] RIP: 0010:lock_acquire+0xf0/0x2b0 [25906.474557] [25906.479251] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25906.520261] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25906.525479] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25906.532604] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25906.539728] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25906.546852] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25906.553976] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25906.561108] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25906.569185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25906.574925] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25906.582048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25906.589172] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25906.596295] Call Trace: [25906.598740] [25906.600752] ? watchdog_timer_fn+0x1b4/0x220 [25906.605016] ? __hrtimer_run_queues+0x1d3/0x360 [25906.609543] ? hrtimer_interrupt+0x104/0x240 [25906.613827] ? __local_bh_enable+0x50/0x60 [25906.617923] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25906.623146] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25906.628103] [25906.630218] [25906.632315] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25906.637624] ? lock_acquire+0xf0/0x2b0 [25906.641374] ? igrab+0x43/0x50 [25906.644423] ? lock_release+0xc5/0x260 [25906.648169] _raw_spin_lock+0x30/0x40 [25906.651834] ? igrab+0x1d/0x50 [25906.654884] igrab+0x1d/0x50 [25906.657763] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25906.662913] delayed_work+0x1d8/0x280 [ceph] [25906.667194] ? process_scheduled_works+0x22a/0x510 [25906.671983] process_scheduled_works+0x2a4/0x510 [25906.676599] worker_thread+0x188/0x340 [25906.680349] ? rescuer_thread+0x310/0x310 [25906.684351] kthread+0xf5/0x130 [25906.687499] ? kthread_complete_and_exit+0x20/0x20 [25906.692291] ret_from_fork+0x31/0x40 [25906.695869] ? kthread_complete_and_exit+0x20/0x20 [25906.700652] ret_from_fork_asm+0x11/0x20 [25906.704577] [25933.911910] watchdog: BUG: soft lockup - CPU#0 stuck for 13038s! [dbench:78683] [25933.919212] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25933.923910] watchdog: BUG: soft lockup - CPU#7 stuck for 22035s! [kworker/7:4:78808] [25933.974651] irq event stamp: 41942726 [25933.974652] hardirqs last enabled at (41942725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25933.982383] Modules linked in: [25933.986040] hardirqs last disabled at (41942726): [] sysvec_apic_timer_interrupt+0xe/0x90 [25933.996198] ceph [25933.999248] softirqs last enabled at (41942708): [] __do_softirq+0x2be/0x371 [25934.008971] libceph [25934.010898] softirqs last disabled at (41942703): [] irq_exit_rcu+0xb0/0xd0 [25934.019594] dns_resolver fscache [25934.021781] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25934.030307] netfs veth [25934.033621] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25934.033622] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25934.042566] nft_chain_nat [25934.045009] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25934.052480] xt_MASQUERADE nf_nat [25934.057957] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25934.060663] nf_conntrack [25934.079399] [25934.079400] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25934.082709] nf_defrag_ipv6 nf_defrag_ipv4 [25934.087926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25934.090545] nft_compat [25934.092037] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25934.099158] nf_tables nfnetlink [25934.103251] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25934.110375] bridge stp [25934.112819] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25934.119942] llc xfs [25934.123168] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25934.130307] libcrc32c [25934.132752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25934.139875] sunrpc intel_rapl_msr [25934.142060] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25934.150138] intel_rapl_common [25934.152496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25934.158233] sb_edac x86_pkg_temp_thermal [25934.161630] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25934.168754] coretemp kvm_intel [25934.171804] Call Trace: [25934.171805] [25934.178930] kvm irqbypass [25934.182934] ? watchdog_timer_fn+0x1b4/0x220 [25934.190058] i2c_i801 [25934.193195] ? __hrtimer_run_queues+0x1d3/0x360 [25934.195638] iTCO_wdt [25934.197652] ? hrtimer_interrupt+0x104/0x240 [25934.200354] iTCO_vendor_support i2c_smbus [25934.204618] ? __local_bh_enable+0x50/0x60 [25934.206887] crct10dif_pclmul joydev [25934.211416] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25934.213683] crc32_pclmul ipmi_ssif [25934.217948] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25934.222038] ghash_clmulni_intel [25934.226129] [25934.229700] mei_me mei [25934.234917] [25934.238400] lpc_ich sha512_ssse3 [25934.243359] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25934.246594] acpi_ipmi mfd_core [25934.248693] ? queued_spin_lock_slowpath+0x18/0x2e0 [25934.251133] wmi [25934.253232] do_raw_spin_lock+0xb5/0xc0 [25934.256541] ipmi_si ipmi_devintf [25934.261845] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25934.264981] ipmi_msghandler ioatdma [25934.269855] try_get_cap_refs+0x159/0x880 [ceph] [25934.271691] acpi_pad acpi_power_meter [25934.275527] __ceph_get_caps+0x4a1/0x760 [ceph] [25934.278833] ixgbe igb [25934.283619] ? __file_remove_privs+0x89/0x110 [25934.287187] mdio [25934.291797] ? wake_bit_function+0x70/0x70 [25934.295541] crc32c_intel nvme [25934.300069] ? ceph_write_iter+0x260/0x770 [ceph] [25934.302424] ptp nvme_core [25934.306774] ceph_write_iter+0x260/0x770 [ceph] [25934.308698] i2c_algo_bit pps_core [25934.312793] ? lock_is_held_type+0xaa/0x120 [25934.315838] dca fuse [25934.320539] vfs_write+0x324/0x4f0 [25934.323241] [25934.323242] irq event stamp: 129442594 [25934.327768] ksys_pwrite64+0x65/0xa0 [25934.331162] hardirqs last enabled at (129442593): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25934.335338] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25934.337609] hardirqs last disabled at (129442594): [] sysvec_apic_timer_interrupt+0xe/0x90 [25934.341007] do_syscall_64+0x38/0x80 [25934.342498] softirqs last enabled at (129442592): [] __do_softirq+0x2be/0x371 [25934.346244] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25934.349813] softirqs last disabled at (129442587): [] irq_exit_rcu+0xb0/0xd0 [25934.360056] RIP: 0033:0x7f48f0eeb93b [25934.365189] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25934.374999] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25934.378569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25934.387339] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25934.392385] Workqueue: events delayed_work [ceph] [25934.400982] ORIG_RAX: 0000000000000012 [25934.400983] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25934.404552] [25934.413930] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25934.432668] RIP: 0010:lock_acquire+0xf0/0x2b0 [25934.440140] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25934.445356] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25934.450053] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25934.453884] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25934.461009] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25934.462499] [25934.462500] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25934.469628] [25934.473973] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25934.537233] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25934.544357] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25934.551482] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25934.558610] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25934.566691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25934.572430] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25934.579552] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25934.586677] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25934.593801] Call Trace: [25934.596247] [25934.598257] ? watchdog_timer_fn+0x1b4/0x220 [25934.602521] ? __hrtimer_run_queues+0x1d3/0x360 [25934.607048] ? hrtimer_interrupt+0x104/0x240 [25934.611316] ? __local_bh_enable+0x50/0x60 [25934.615412] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25934.620635] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25934.625595] [25934.627689] [25934.629786] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25934.635094] ? lock_acquire+0xf0/0x2b0 [25934.638845] ? igrab+0x43/0x50 [25934.641895] ? lock_release+0xc5/0x260 [25934.645641] _raw_spin_lock+0x30/0x40 [25934.649318] ? igrab+0x1d/0x50 [25934.652373] igrab+0x1d/0x50 [25934.655249] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25934.660401] delayed_work+0x1d8/0x280 [ceph] [25934.664682] ? process_scheduled_works+0x22a/0x510 [25934.669472] process_scheduled_works+0x2a4/0x510 [25934.674088] worker_thread+0x188/0x340 [25934.677838] ? rescuer_thread+0x310/0x310 [25934.681847] kthread+0xf5/0x130 [25934.684986] ? kthread_complete_and_exit+0x20/0x20 [25934.689770] ret_from_fork+0x31/0x40 [25934.693341] ? kthread_complete_and_exit+0x20/0x20 [25934.698124] ret_from_fork_asm+0x11/0x20 [25934.702046] [25961.912398] watchdog: BUG: soft lockup - CPU#0 stuck for 13064s! [dbench:78683] [25961.919700] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25961.924397] watchdog: BUG: soft lockup - CPU#7 stuck for 22061s! [kworker/7:4:78808] [25961.975147] irq event stamp: 42000974 [25961.982879] Modules linked in: ceph libceph dns_resolver fscache netfs [25961.986537] hardirqs last enabled at (42000973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25961.986542] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [25962.003238] hardirqs last disabled at (42000974): [] sysvec_apic_timer_interrupt+0xe/0x90 [25962.011928] nft_compat nf_tables nfnetlink bridge stp [25962.021653] softirqs last enabled at (42000956): [] __do_softirq+0x2be/0x371 [25962.021657] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [25962.035477] softirqs last disabled at (42000951): [] irq_exit_rcu+0xb0/0xd0 [25962.041995] sb_edac x86_pkg_temp_thermal coretemp [25962.050507] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25962.050508] kvm_intel [25962.050510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25962.055289] kvm irqbypass [25962.064235] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [25962.066591] i2c_i801 iTCO_wdt [25962.074062] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [25962.076765] iTCO_vendor_support i2c_smbus [25962.082261] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25962.085312] crct10dif_pclmul joydev [25962.104050] [25962.104051] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25962.108140] crc32_pclmul [25962.113358] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25962.116930] ipmi_ssif [25962.118419] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25962.125543] ghash_clmulni_intel [25962.128161] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25962.135285] mei_me [25962.137641] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25962.144766] mei lpc_ich [25962.147989] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25962.155113] sha512_ssse3 acpi_ipmi [25962.157213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25962.164337] mfd_core wmi [25962.166865] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25962.174945] ipmi_si [25962.178429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25962.184178] ipmi_devintf ipmi_msghandler [25962.186795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25962.193915] ioatdma acpi_pad [25962.196100] Call Trace: [25962.203224] acpi_power_meter [25962.207229] [25962.214351] ixgbe igb [25962.217320] ? watchdog_timer_fn+0x1b4/0x220 [25962.219761] mdio crc32c_intel [25962.222726] ? __hrtimer_run_queues+0x1d3/0x360 [25962.224735] nvme ptp [25962.227095] ? hrtimer_interrupt+0x104/0x240 [25962.231356] nvme_core i2c_algo_bit [25962.234407] ? __local_bh_enable+0x50/0x60 [25962.238931] pps_core [25962.241205] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25962.245465] dca fuse [25962.248950] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25962.253041] [25962.253042] irq event stamp: 129663420 [25962.255313] [25962.260530] hardirqs last enabled at (129663419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25962.262800] [25962.267756] hardirqs last disabled at (129663420): [] sysvec_apic_timer_interrupt+0xe/0x90 [25962.269248] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25962.272993] softirqs last enabled at (129663418): [] __do_softirq+0x2be/0x371 [25962.275092] ? queued_spin_lock_slowpath+0x2b/0x2e0 [25962.285333] softirqs last disabled at (129663413): [] irq_exit_rcu+0xb0/0xd0 [25962.287432] do_raw_spin_lock+0xb5/0xc0 [25962.297241] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25962.302547] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25962.311317] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25962.316191] try_get_cap_refs+0x159/0x880 [ceph] [25962.324796] Workqueue: events delayed_work [ceph] [25962.328630] __ceph_get_caps+0x4a1/0x760 [ceph] [25962.338001] [25962.338002] RIP: 0010:lock_acquire+0xf0/0x2b0 [25962.342803] ? __file_remove_privs+0x89/0x110 [25962.350267] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25962.354875] ? wake_bit_function+0x70/0x70 [25962.359575] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25962.364099] ? ceph_write_iter+0x260/0x770 [ceph] [25962.365588] [25962.365588] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25962.369939] ceph_write_iter+0x260/0x770 [ceph] [25962.374290] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25962.393032] ? lock_is_held_type+0xaa/0x120 [25962.397119] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25962.402338] vfs_write+0x324/0x4f0 [25962.407033] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25962.408530] ksys_pwrite64+0x65/0xa0 [25962.415647] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25962.420174] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25962.427296] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25962.431475] do_syscall_64+0x38/0x80 [25962.438599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25962.441995] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25962.449120] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25962.452691] RIP: 0033:0x7f48f0eeb93b [25962.459813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25962.464946] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25962.473022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25962.476592] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25962.482331] Call Trace: [25962.482332] [25962.487374] ORIG_RAX: 0000000000000012 [25962.494500] ? watchdog_timer_fn+0x1b4/0x220 [25962.498081] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25962.505204] ? __hrtimer_run_queues+0x1d3/0x360 [25962.523939] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25962.531084] ? hrtimer_interrupt+0x104/0x240 [25962.536300] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25962.538743] ? __local_bh_enable+0x50/0x60 [25962.540754] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25962.544587] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25962.548848] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25962.555974] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25962.560503] [25962.567621] [25962.611805] [25962.613903] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25962.619210] ? lock_acquire+0xf0/0x2b0 [25962.622961] ? igrab+0x43/0x50 [25962.626011] ? lock_release+0xc5/0x260 [25962.629756] _raw_spin_lock+0x30/0x40 [25962.633420] ? igrab+0x1d/0x50 [25962.636471] igrab+0x1d/0x50 [25962.639349] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25962.644499] delayed_work+0x1d8/0x280 [ceph] [25962.648781] ? process_scheduled_works+0x22a/0x510 [25962.653588] process_scheduled_works+0x2a4/0x510 [25962.658201] worker_thread+0x188/0x340 [25962.661951] ? rescuer_thread+0x310/0x310 [25962.665954] kthread+0xf5/0x130 [25962.669102] ? kthread_complete_and_exit+0x20/0x20 [25962.673885] ret_from_fork+0x31/0x40 [25962.677467] ? kthread_complete_and_exit+0x20/0x20 [25962.682258] ret_from_fork_asm+0x11/0x20 [25962.686191] [25989.912883] watchdog: BUG: soft lockup - CPU#0 stuck for 13090s! [dbench:78683] [25989.920185] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [25989.924883] watchdog: BUG: soft lockup - CPU#7 stuck for 22088s! [kworker/7:4:78808] [25989.975627] irq event stamp: 42059088 [25989.975628] hardirqs last enabled at (42059087): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25989.983356] Modules linked in: [25989.987014] hardirqs last disabled at (42059088): [] sysvec_apic_timer_interrupt+0xe/0x90 [25989.997173] ceph [25990.000223] softirqs last enabled at (42059070): [] __do_softirq+0x2be/0x371 [25990.009947] libceph [25990.011870] softirqs last disabled at (42059065): [] irq_exit_rcu+0xb0/0xd0 [25990.020559] dns_resolver fscache [25990.022740] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25990.031251] netfs veth [25990.034577] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25990.043522] nft_chain_nat xt_MASQUERADE [25990.045966] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [25990.053436] nf_nat nf_conntrack [25990.057353] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [25990.062831] nf_defrag_ipv6 nf_defrag_ipv4 [25990.066055] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [25990.084794] nft_compat [25990.088885] [25990.088886] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [25990.094102] nf_tables nfnetlink [25990.096560] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [25990.098053] bridge stp [25990.105177] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [25990.108401] llc xfs [25990.115526] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [25990.117970] libcrc32c [25990.125094] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [25990.127281] sunrpc intel_rapl_msr [25990.134403] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [25990.136759] intel_rapl_common sb_edac [25990.143884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25990.147281] x86_pkg_temp_thermal coretemp [25990.155358] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [25990.159103] kvm_intel kvm [25990.164840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25990.168931] irqbypass i2c_i801 [25990.176054] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25990.178759] iTCO_wdt iTCO_vendor_support [25990.185882] Call Trace: [25990.185883] [25990.189021] i2c_smbus [25990.196146] ? watchdog_timer_fn+0x1b4/0x220 [25990.200148] crct10dif_pclmul joydev [25990.202594] ? __hrtimer_run_queues+0x1d3/0x360 [25990.204603] crc32_pclmul ipmi_ssif [25990.206965] ? hrtimer_interrupt+0x104/0x240 [25990.211226] ghash_clmulni_intel [25990.214797] ? __local_bh_enable+0x50/0x60 [25990.219321] mei_me [25990.222808] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25990.227069] mei lpc_ich [25990.230294] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25990.234383] sha512_ssse3 acpi_ipmi [25990.236481] [25990.241697] mfd_core wmi [25990.244230] [25990.244231] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25990.249185] ipmi_si ipmi_devintf [25990.252673] ? queued_spin_lock_slowpath+0x18/0x2e0 [25990.254767] ipmi_msghandler [25990.257387] do_raw_spin_lock+0xb5/0xc0 [25990.259482] ioatdma acpi_pad [25990.264801] __cap_delay_requeue+0xa7/0x1d0 [ceph] [25990.268115] acpi_power_meter ixgbe [25990.272987] try_get_cap_refs+0x159/0x880 [ceph] [25990.275862] igb mdio [25990.279699] __ceph_get_caps+0x4a1/0x760 [ceph] [25990.282658] crc32c_intel nvme [25990.287446] ? __file_remove_privs+0x89/0x110 [25990.290924] ptp nvme_core [25990.295535] ? wake_bit_function+0x70/0x70 [25990.297808] i2c_algo_bit pps_core [25990.302335] ? ceph_write_iter+0x260/0x770 [ceph] [25990.305381] dca fuse [25990.309732] ceph_write_iter+0x260/0x770 [ceph] [25990.312437] [25990.312438] irq event stamp: 129884346 [25990.316531] ? lock_is_held_type+0xaa/0x120 [25990.319924] hardirqs last enabled at (129884345): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25990.324624] vfs_write+0x324/0x4f0 [25990.326894] hardirqs last disabled at (129884346): [] sysvec_apic_timer_interrupt+0xe/0x90 [25990.331421] ksys_pwrite64+0x65/0xa0 [25990.332907] softirqs last enabled at (129884344): [] __do_softirq+0x2be/0x371 [25990.336653] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [25990.340828] softirqs last disabled at (129884339): [] irq_exit_rcu+0xb0/0xd0 [25990.351074] do_syscall_64+0x38/0x80 [25990.354470] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [25990.364285] entry_SYSCALL_64_after_hwframe+0x63/0xcd [25990.367852] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [25990.376622] RIP: 0033:0x7f48f0eeb93b [25990.381753] Workqueue: events delayed_work [ceph] [25990.390352] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [25990.393923] [25990.403300] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [25990.408343] RIP: 0010:lock_acquire+0xf0/0x2b0 [25990.415815] ORIG_RAX: 0000000000000012 [25990.419385] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [25990.424081] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [25990.442820] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [25990.444310] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [25990.449529] [25990.449530] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [25990.453878] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [25990.457709] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [25990.476447] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [25990.483572] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [25990.488790] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [25990.495912] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [25990.497410] [25990.504529] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [25990.556599] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [25990.564676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25990.570414] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [25990.577536] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25990.584662] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [25990.591787] Call Trace: [25990.594229] [25990.596241] ? watchdog_timer_fn+0x1b4/0x220 [25990.600505] ? __hrtimer_run_queues+0x1d3/0x360 [25990.605032] ? hrtimer_interrupt+0x104/0x240 [25990.609302] ? __local_bh_enable+0x50/0x60 [25990.613395] ? __sysvec_apic_timer_interrupt+0x61/0x160 [25990.618618] ? sysvec_apic_timer_interrupt+0x6d/0x90 [25990.623579] [25990.625673] [25990.627770] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [25990.633080] ? lock_acquire+0xf0/0x2b0 [25990.636830] ? igrab+0x43/0x50 [25990.639878] ? lock_release+0xc5/0x260 [25990.643623] _raw_spin_lock+0x30/0x40 [25990.647290] ? igrab+0x1d/0x50 [25990.650340] igrab+0x1d/0x50 [25990.653217] ceph_check_delayed_caps+0xb7/0x270 [ceph] [25990.658366] delayed_work+0x1d8/0x280 [ceph] [25990.662647] ? process_scheduled_works+0x22a/0x510 [25990.667430] process_scheduled_works+0x2a4/0x510 [25990.672044] worker_thread+0x188/0x340 [25990.675795] ? rescuer_thread+0x310/0x310 [25990.679797] kthread+0xf5/0x130 [25990.682935] ? kthread_complete_and_exit+0x20/0x20 [25990.687719] ret_from_fork+0x31/0x40 [25990.691291] ? kthread_complete_and_exit+0x20/0x20 [25990.696075] ret_from_fork_asm+0x11/0x20 [25990.699997] [26017.913369] watchdog: BUG: soft lockup - CPU#0 stuck for 13117s! [dbench:78683] [26017.920672] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26017.925369] watchdog: BUG: soft lockup - CPU#7 stuck for 22114s! [kworker/7:4:78808] [26017.976123] irq event stamp: 42117288 [26017.983854] Modules linked in: ceph libceph dns_resolver fscache netfs veth [26017.987511] hardirqs last enabled at (42117287): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26017.994471] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [26018.004626] hardirqs last disabled at (42117288): [] sysvec_apic_timer_interrupt+0xe/0x90 [26018.004629] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [26018.021301] softirqs last enabled at (42117270): [] __do_softirq+0x2be/0x371 [26018.021305] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [26018.036765] softirqs last disabled at (42117265): [] irq_exit_rcu+0xb0/0xd0 [26018.045455] coretemp kvm_intel kvm [26018.053969] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26018.057451] irqbypass i2c_i801 [26018.066395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26018.066397] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26018.069532] iTCO_wdt iTCO_vendor_support [26018.077006] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26018.082482] i2c_smbus crct10dif_pclmul [26018.086486] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26018.105223] joydev [26018.109055] [26018.109055] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26018.114271] crc32_pclmul [26018.116368] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26018.117861] ipmi_ssif [26018.124984] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26018.127600] ghash_clmulni_intel [26018.134725] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26018.137082] mei_me mei [26018.144206] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26018.147431] lpc_ich sha512_ssse3 [26018.154554] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26018.157000] acpi_ipmi mfd_core [26018.164122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26018.167432] wmi ipmi_si [26018.175512] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26018.178650] ipmi_devintf ipmi_msghandler [26018.184386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26018.186918] ioatdma [26018.194057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26018.198063] acpi_pad [26018.205186] Call Trace: [26018.205189] [26018.207370] acpi_power_meter [26018.214498] ? watchdog_timer_fn+0x1b4/0x220 [26018.216765] ixgbe igb [26018.219210] ? __hrtimer_run_queues+0x1d3/0x360 [26018.221219] mdio crc32c_intel [26018.224186] ? hrtimer_interrupt+0x104/0x240 [26018.228448] nvme [26018.230805] ? __local_bh_enable+0x50/0x60 [26018.235329] ptp nvme_core [26018.238383] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26018.242645] i2c_algo_bit [26018.244568] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26018.248659] pps_core [26018.251362] [26018.256580] dca fuse [26018.259198] [26018.259200] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26018.264156] [26018.264157] irq event stamp: 130105254 [26018.266429] ? queued_spin_lock_slowpath+0x16/0x2e0 [26018.268523] hardirqs last enabled at (130105253): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26018.270796] do_raw_spin_lock+0xb5/0xc0 [26018.272890] hardirqs last disabled at (130105254): [] sysvec_apic_timer_interrupt+0xe/0x90 [26018.278197] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26018.279687] softirqs last enabled at (130105252): [] __do_softirq+0x2be/0x371 [26018.283435] try_get_cap_refs+0x159/0x880 [ceph] [26018.288302] softirqs last disabled at (130105247): [] irq_exit_rcu+0xb0/0xd0 [26018.298549] __ceph_get_caps+0x4a1/0x760 [ceph] [26018.302376] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26018.312192] ? __file_remove_privs+0x89/0x110 [26018.316970] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26018.325742] ? wake_bit_function+0x70/0x70 [26018.330369] Workqueue: events delayed_work [ceph] [26018.338970] ? ceph_write_iter+0x260/0x770 [ceph] [26018.343492] [26018.343493] RIP: 0010:lock_acquire+0xf0/0x2b0 [26018.352869] ceph_write_iter+0x260/0x770 [ceph] [26018.357219] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26018.364696] ? lock_is_held_type+0xaa/0x120 [26018.368781] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26018.373480] vfs_write+0x324/0x4f0 [26018.378177] [26018.378178] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26018.379673] ksys_pwrite64+0x65/0xa0 [26018.384018] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26018.388543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26018.407278] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26018.411459] do_syscall_64+0x38/0x80 [26018.416674] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26018.420074] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26018.421562] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26018.428687] RIP: 0033:0x7f48f0eeb93b [26018.432257] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26018.439383] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26018.444512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26018.451637] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26018.455206] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26018.462332] ORIG_RAX: 0000000000000012 [26018.467375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26018.474499] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26018.478071] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26018.486148] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26018.504885] Call Trace: [26018.504887] [26018.510622] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26018.515841] ? watchdog_timer_fn+0x1b4/0x220 [26018.522964] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26018.526795] ? __hrtimer_run_queues+0x1d3/0x360 [26018.533919] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26018.541056] ? hrtimer_interrupt+0x104/0x240 [26018.548182] [26018.555299] ? __local_bh_enable+0x50/0x60 [26018.600475] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26018.605698] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26018.610657] [26018.612770] [26018.614867] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26018.620176] ? lock_acquire+0xf0/0x2b0 [26018.623927] ? igrab+0x43/0x50 [26018.626976] ? lock_release+0xc5/0x260 [26018.630720] _raw_spin_lock+0x30/0x40 [26018.634378] ? igrab+0x1d/0x50 [26018.637426] igrab+0x1d/0x50 [26018.640306] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26018.645456] delayed_work+0x1d8/0x280 [ceph] [26018.649737] ? process_scheduled_works+0x22a/0x510 [26018.654526] process_scheduled_works+0x2a4/0x510 [26018.659144] worker_thread+0x188/0x340 [26018.662890] ? rescuer_thread+0x310/0x310 [26018.666894] kthread+0xf5/0x130 [26018.670033] ? kthread_complete_and_exit+0x20/0x20 [26018.674817] ret_from_fork+0x31/0x40 [26018.678398] ? kthread_complete_and_exit+0x20/0x20 [26018.683189] ret_from_fork_asm+0x11/0x20 [26018.687120] [26045.913853] watchdog: BUG: soft lockup - CPU#0 stuck for 13143s! [dbench:78683] [26045.921150] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26045.925853] watchdog: BUG: soft lockup - CPU#7 stuck for 22140s! [kworker/7:4:78808] [26045.976591] irq event stamp: 42175420 [26045.976592] hardirqs last enabled at (42175419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26045.984322] Modules linked in: [26045.987981] hardirqs last disabled at (42175420): [] sysvec_apic_timer_interrupt+0xe/0x90 [26045.998139] ceph [26046.001188] softirqs last enabled at (42175402): [] __do_softirq+0x2be/0x371 [26046.010913] libceph [26046.012836] softirqs last disabled at (42175397): [] irq_exit_rcu+0xb0/0xd0 [26046.021528] dns_resolver fscache [26046.023715] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26046.032225] netfs [26046.035551] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26046.035552] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [26046.044497] veth nft_chain_nat [26046.046524] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [26046.053995] xt_MASQUERADE nf_nat [26046.059473] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26046.062611] nf_conntrack nf_defrag_ipv6 [26046.081347] [26046.081347] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26046.084659] nf_defrag_ipv4 [26046.089875] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26046.093793] nft_compat nf_tables [26046.095285] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26046.102409] nfnetlink bridge [26046.105199] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26046.112324] stp llc [26046.115633] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26046.122758] xfs libcrc32c [26046.125722] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26046.132847] sunrpc intel_rapl_msr [26046.135031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26046.142155] intel_rapl_common sb_edac [26046.144860] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26046.152935] x86_pkg_temp_thermal [26046.156334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26046.162070] coretemp kvm_intel [26046.165814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26046.172939] kvm irqbypass [26046.176250] Call Trace: [26046.176252] [26046.183374] i2c_i801 iTCO_wdt [26046.186529] ? watchdog_timer_fn+0x1b4/0x220 [26046.193652] iTCO_vendor_support i2c_smbus [26046.196359] ? __hrtimer_run_queues+0x1d3/0x360 [26046.198801] crct10dif_pclmul joydev [26046.200815] ? hrtimer_interrupt+0x104/0x240 [26046.203862] crc32_pclmul ipmi_ssif [26046.208126] ? __local_bh_enable+0x50/0x60 [26046.212217] ghash_clmulni_intel [26046.216743] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26046.220312] mei_me mei [26046.224576] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26046.228061] lpc_ich sha512_ssse3 [26046.232151] [26046.235374] acpi_ipmi [26046.240591] [26046.243035] mfd_core [26046.247995] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26046.251303] wmi ipmi_si [26046.253406] ? queued_spin_lock_slowpath+0x2b/0x2e0 [26046.255758] ipmi_devintf ipmi_msghandler [26046.257858] do_raw_spin_lock+0xb5/0xc0 [26046.260126] ioatdma [26046.265433] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26046.267962] acpi_pad acpi_power_meter [26046.272836] try_get_cap_refs+0x159/0x880 [ceph] [26046.276837] ixgbe igb [26046.280671] __ceph_get_caps+0x4a1/0x760 [ceph] [26046.282851] mdio crc32c_intel [26046.287639] ? __file_remove_privs+0x89/0x110 [26046.291379] nvme ptp [26046.295992] ? wake_bit_function+0x70/0x70 [26046.298348] nvme_core i2c_algo_bit [26046.302876] ? ceph_write_iter+0x260/0x770 [ceph] [26046.305922] pps_core dca [26046.310273] ceph_write_iter+0x260/0x770 [ceph] [26046.312545] fuse [26046.316639] ? lock_is_held_type+0xaa/0x120 [26046.320119] irq event stamp: 130326192 [26046.324818] vfs_write+0x324/0x4f0 [26046.327433] hardirqs last enabled at (130326191): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26046.331962] ksys_pwrite64+0x65/0xa0 [26046.333882] hardirqs last disabled at (130326192): [] sysvec_apic_timer_interrupt+0xe/0x90 [26046.338059] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26046.341804] softirqs last enabled at (130326190): [] __do_softirq+0x2be/0x371 [26046.345201] do_syscall_64+0x38/0x80 [26046.355444] softirqs last disabled at (130326185): [] irq_exit_rcu+0xb0/0xd0 [26046.359017] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26046.368827] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26046.373958] RIP: 0033:0x7f48f0eeb93b [26046.382727] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26046.386300] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26046.394896] Workqueue: events delayed_work [ceph] [26046.399942] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26046.409318] [26046.409319] RIP: 0010:do_raw_spin_lock+0x37/0xc0 [26046.412888] ORIG_RAX: 0000000000000012 [26046.420359] Code: 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a <31> c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 [26046.439098] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26046.443794] RSP: 0018:ffffc90002117d90 EFLAGS: 00000282 [26046.449011] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26046.450503] [26046.450503] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [26046.455113] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26046.458944] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [26046.477681] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26046.484806] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [26046.490023] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26046.497146] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26046.498643] [26046.505762] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26046.557817] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26046.565892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26046.571632] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26046.578754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26046.585879] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26046.593002] Call Trace: [26046.595447] [26046.597460] ? watchdog_timer_fn+0x1b4/0x220 [26046.601723] ? __hrtimer_run_queues+0x1d3/0x360 [26046.606261] ? hrtimer_interrupt+0x104/0x240 [26046.610526] ? __local_bh_enable+0x50/0x60 [26046.614622] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26046.619845] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26046.624802] [26046.626899] [26046.628996] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26046.634304] ? do_raw_spin_lock+0x37/0xc0 [26046.638314] igrab+0x1d/0x50 [26046.641191] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26046.646342] delayed_work+0x1d8/0x280 [ceph] [26046.650623] ? process_scheduled_works+0x22a/0x510 [26046.655412] process_scheduled_works+0x2a4/0x510 [26046.660028] worker_thread+0x188/0x340 [26046.663778] ? rescuer_thread+0x310/0x310 [26046.667780] kthread+0xf5/0x130 [26046.670918] ? kthread_complete_and_exit+0x20/0x20 [26046.675704] ret_from_fork+0x31/0x40 [26046.679282] ? kthread_complete_and_exit+0x20/0x20 [26046.684067] ret_from_fork_asm+0x11/0x20 [26046.687989] [26073.223692] rcu: INFO: rcu_preempt self-detected stall on CPU [26073.229454] rcu: 7-....: (13727990 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/13726697 fqs=6906911 [26073.240045] rcu: (t=14136362 jiffies g=1309609 q=433235 ncpus=8) [26073.246130] Sending NMI from CPU 7 to CPUs 0: [26073.250480] NMI backtrace for cpu 0 [26073.250482] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26073.250484] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26073.250485] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26073.250491] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26073.250492] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26073.250494] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26073.250495] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26073.250496] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26073.250497] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26073.250497] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26073.250498] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26073.250499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26073.250500] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26073.250501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26073.250502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26073.250502] Call Trace: [26073.250504] [26073.250505] ? nmi_cpu_backtrace+0xad/0x120 [26073.250509] ? nmi_cpu_backtrace_handler+0x11/0x20 [26073.250510] ? nmi_handle+0xd4/0x260 [26073.250515] ? default_do_nmi+0x49/0xf0 [26073.250517] ? exc_nmi+0x11e/0x150 [26073.250518] ? end_repeat_nmi+0x16/0x67 [26073.250523] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26073.250525] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26073.250528] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26073.250530] [26073.250531] [26073.250532] do_raw_spin_lock+0xb5/0xc0 [26073.250535] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26073.250556] try_get_cap_refs+0x159/0x880 [ceph] [26073.250571] __ceph_get_caps+0x4a1/0x760 [ceph] [26073.250583] ? __file_remove_privs+0x89/0x110 [26073.250586] ? wake_bit_function+0x70/0x70 [26073.250589] ? ceph_write_iter+0x260/0x770 [ceph] [26073.250599] ceph_write_iter+0x260/0x770 [ceph] [26073.250611] ? lock_is_held_type+0xaa/0x120 [26073.250614] vfs_write+0x324/0x4f0 [26073.250619] ksys_pwrite64+0x65/0xa0 [26073.250621] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26073.250625] do_syscall_64+0x38/0x80 [26073.250628] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26073.250630] RIP: 0033:0x7f48f0eeb93b [26073.250632] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26073.250633] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [26073.250635] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26073.250635] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26073.250636] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26073.250637] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26073.250638] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26073.250641] [26073.251479] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26073.555041] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26073.562512] Workqueue: events delayed_work [ceph] [26073.567226] RIP: 0010:lock_acquire+0xf0/0x2b0 [26073.571576] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26073.590314] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26073.595531] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26073.602654] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26073.609779] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26073.616904] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26073.624029] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26073.631151] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26073.639230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26073.644965] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26073.652091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26073.659216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26073.666339] Call Trace: [26073.668784] [26073.670795] ? rcu_dump_cpu_stacks+0xd1/0x180 [26073.675155] ? rcu_sched_clock_irq+0x633/0x1390 [26073.679677] ? lock_release+0xc5/0x260 [26073.683420] ? lock_is_held_type+0xaa/0x120 [26073.687600] ? lock_is_held_type+0xaa/0x120 [26073.691778] ? tick_sched_do_timer+0xa0/0xa0 [26073.696049] ? rcu_read_lock_sched_held+0x56/0x90 [26073.700749] ? tick_sched_do_timer+0xa0/0xa0 [26073.705011] ? update_process_times+0x5f/0x90 [26073.709363] ? tick_sched_handle.isra.16+0x1e/0x50 [26073.714146] ? tick_sched_timer+0x6f/0x80 [26073.718152] ? __hrtimer_run_queues+0x1d3/0x360 [26073.722687] ? hrtimer_interrupt+0x104/0x240 [26073.726954] ? __local_bh_enable+0x50/0x60 [26073.731048] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26073.736273] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26073.741229] [26073.743326] [26073.745425] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26073.750733] ? lock_acquire+0xf0/0x2b0 [26073.754484] ? igrab+0x43/0x50 [26073.757533] ? lock_release+0xc5/0x260 [26073.761278] _raw_spin_lock+0x30/0x40 [26073.764943] ? igrab+0x1d/0x50 [26073.767992] igrab+0x1d/0x50 [26073.770872] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26073.776020] delayed_work+0x1d8/0x280 [ceph] [26073.780303] ? process_scheduled_works+0x22a/0x510 [26073.785092] process_scheduled_works+0x2a4/0x510 [26073.789710] worker_thread+0x188/0x340 [26073.793457] ? rescuer_thread+0x310/0x310 [26073.797460] kthread+0xf5/0x130 [26073.800598] ? kthread_complete_and_exit+0x20/0x20 [26073.805381] ret_from_fork+0x31/0x40 [26073.808962] ? kthread_complete_and_exit+0x20/0x20 [26073.813745] ret_from_fork_asm+0x11/0x20 [26073.817678] [26097.914750] watchdog: BUG: soft lockup - CPU#0 stuck for 13191s! [dbench:78683] [26097.922056] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26097.926750] watchdog: BUG: soft lockup - CPU#7 stuck for 22188s! [kworker/7:4:78808] [26097.977505] irq event stamp: 42284586 [26097.985236] Modules linked in: ceph libceph [26097.988895] hardirqs last enabled at (42284585): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26097.993070] dns_resolver fscache [26098.003229] hardirqs last disabled at (42284586): [] sysvec_apic_timer_interrupt+0xe/0x90 [26098.003231] netfs [26098.003232] softirqs last enabled at (42284516): [] __do_softirq+0x2be/0x371 [26098.006539] veth nft_chain_nat [26098.016263] softirqs last disabled at (42284511): [] irq_exit_rcu+0xb0/0xd0 [26098.018275] xt_MASQUERADE nf_nat [26098.026959] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26098.030096] nf_conntrack nf_defrag_ipv6 [26098.038607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26098.041918] nf_defrag_ipv4 nft_compat [26098.050860] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26098.054779] nf_tables [26098.062250] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26098.065993] nfnetlink [26098.071470] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26098.073827] bridge stp [26098.092567] [26098.092568] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26098.094923] llc xfs [26098.100147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26098.102594] libcrc32c sunrpc [26098.104085] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26098.111208] intel_rapl_msr [26098.113392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26098.120518] intel_rapl_common [26098.123481] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26098.130606] sb_edac [26098.133396] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26098.140519] x86_pkg_temp_thermal coretemp [26098.143569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26098.150695] kvm_intel [26098.152879] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26098.160957] kvm irqbypass [26098.165046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26098.170784] i2c_i801 iTCO_wdt [26098.173149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26098.180274] iTCO_vendor_support [26098.182978] Call Trace: [26098.182980] [26098.190103] i2c_smbus [26098.193153] ? watchdog_timer_fn+0x1b4/0x220 [26098.200277] crct10dif_pclmul joydev [26098.203503] ? __hrtimer_run_queues+0x1d3/0x360 [26098.205944] crc32_pclmul ipmi_ssif [26098.207960] ? hrtimer_interrupt+0x104/0x240 [26098.210313] ghash_clmulni_intel [26098.214579] ? __local_bh_enable+0x50/0x60 [26098.218151] mei_me mei [26098.222675] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26098.226157] lpc_ich sha512_ssse3 [26098.230429] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26098.233654] acpi_ipmi mfd_core [26098.237743] [26098.240189] wmi [26098.245406] [26098.245424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26098.248733] ipmi_si ipmi_devintf [26098.253695] ? queued_spin_lock_slowpath+0x16/0x2e0 [26098.256828] ipmi_msghandler ioatdma [26098.258928] do_raw_spin_lock+0xb5/0xc0 [26098.260763] acpi_pad [26098.262860] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26098.268164] acpi_power_meter [26098.271479] try_get_cap_refs+0x159/0x880 [ceph] [26098.276345] ixgbe igb [26098.279922] __ceph_get_caps+0x4a1/0x760 [ceph] [26098.283748] mdio crc32c_intel [26098.286021] ? __file_remove_privs+0x89/0x110 [26098.290802] nvme [26098.293766] ? wake_bit_function+0x70/0x70 [26098.298378] ptp nvme_core [26098.300737] ? ceph_write_iter+0x260/0x770 [ceph] [26098.305259] i2c_algo_bit pps_core [26098.308309] ceph_write_iter+0x260/0x770 [ceph] [26098.312660] dca fuse [26098.314589] ? lock_is_held_type+0xaa/0x120 [26098.318674] [26098.318675] irq event stamp: 130736982 [26098.321382] vfs_write+0x324/0x4f0 [26098.326077] hardirqs last enabled at (130736981): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26098.329477] ksys_pwrite64+0x65/0xa0 [26098.333998] hardirqs last disabled at (130736982): [] sysvec_apic_timer_interrupt+0xe/0x90 [26098.336268] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26098.340447] softirqs last enabled at (130736980): [] __do_softirq+0x2be/0x371 [26098.341939] do_syscall_64+0x38/0x80 [26098.345680] softirqs last disabled at (130736975): [] irq_exit_rcu+0xb0/0xd0 [26098.349079] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26098.359322] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26098.362892] RIP: 0033:0x7f48f0eeb93b [26098.372705] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26098.377835] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26098.386606] Workqueue: events delayed_work [ceph] [26098.390176] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26098.398773] [26098.398774] RIP: 0010:lock_acquire+0xf0/0x2b0 [26098.403817] ORIG_RAX: 0000000000000012 [26098.413195] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26098.416765] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26098.424238] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26098.442976] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26098.447672] [26098.447673] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26098.452890] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26098.454380] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26098.458730] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26098.462561] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26098.481299] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26098.488432] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26098.493654] [26098.500773] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26098.561458] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26098.569536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26098.575274] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26098.582399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26098.589523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26098.596646] Call Trace: [26098.599092] [26098.601101] ? watchdog_timer_fn+0x1b4/0x220 [26098.605367] ? __hrtimer_run_queues+0x1d3/0x360 [26098.609892] ? hrtimer_interrupt+0x104/0x240 [26098.614156] ? __local_bh_enable+0x50/0x60 [26098.618255] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26098.623480] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26098.628443] [26098.630542] [26098.632641] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26098.637947] ? lock_acquire+0xf0/0x2b0 [26098.641698] ? igrab+0x43/0x50 [26098.644749] ? lock_release+0xc5/0x260 [26098.648492] _raw_spin_lock+0x30/0x40 [26098.652158] ? igrab+0x1d/0x50 [26098.655209] igrab+0x1d/0x50 [26098.658086] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26098.663236] delayed_work+0x1d8/0x280 [ceph] [26098.667519] ? process_scheduled_works+0x22a/0x510 [26098.672308] process_scheduled_works+0x2a4/0x510 [26098.676922] worker_thread+0x188/0x340 [26098.680672] ? rescuer_thread+0x310/0x310 [26098.684675] kthread+0xf5/0x130 [26098.687813] ? kthread_complete_and_exit+0x20/0x20 [26098.692599] ret_from_fork+0x31/0x40 [26098.696178] ? kthread_complete_and_exit+0x20/0x20 [26098.700961] ret_from_fork_asm+0x11/0x20 [26098.704885] [26125.915232] watchdog: BUG: soft lockup - CPU#0 stuck for 13217s! [dbench:78683] [26125.922533] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26125.927232] watchdog: BUG: soft lockup - CPU#7 stuck for 22214s! [kworker/7:4:78808] [26125.977974] irq event stamp: 42342858 [26125.977976] hardirqs last enabled at (42342857): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26125.985706] Modules linked in: ceph [26125.989363] hardirqs last disabled at (42342858): [] sysvec_apic_timer_interrupt+0xe/0x90 [26125.999521] libceph [26126.003005] softirqs last enabled at (42342852): [] __do_softirq+0x2be/0x371 [26126.012729] dns_resolver fscache [26126.014914] softirqs last disabled at (42342847): [] irq_exit_rcu+0xb0/0xd0 [26126.023596] netfs veth [26126.026907] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26126.035420] nft_chain_nat [26126.037863] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26126.037864] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26126.046807] xt_MASQUERADE nf_nat [26126.049512] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26126.056981] nf_conntrack nf_defrag_ipv6 [26126.062459] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26126.065770] nf_defrag_ipv4 nft_compat [26126.084509] [26126.084510] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26126.088424] nf_tables nfnetlink [26126.093644] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26126.097388] bridge stp [26126.098878] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26126.106002] llc xfs [26126.109226] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26126.116350] libcrc32c [26126.118794] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26126.125917] sunrpc intel_rapl_msr [26126.128102] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26126.135227] intel_rapl_common [26126.137584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26126.144708] sb_edac x86_pkg_temp_thermal [26126.148104] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26126.156182] coretemp [26126.159233] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26126.164972] kvm_intel kvm [26126.168974] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26126.176098] irqbypass i2c_i801 [26126.178370] Call Trace: [26126.178371] [26126.185494] iTCO_wdt [26126.188198] ? watchdog_timer_fn+0x1b4/0x220 [26126.195321] iTCO_vendor_support [26126.198461] ? __hrtimer_run_queues+0x1d3/0x360 [26126.200909] i2c_smbus crct10dif_pclmul [26126.202926] ? hrtimer_interrupt+0x104/0x240 [26126.205193] joydev crc32_pclmul [26126.209457] ? __local_bh_enable+0x50/0x60 [26126.212681] ipmi_ssif ghash_clmulni_intel [26126.217208] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26126.221037] mei_me mei [26126.225301] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26126.228525] lpc_ich [26126.232630] [26126.236722] sha512_ssse3 acpi_ipmi [26126.241942] [26126.241943] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26126.244384] mfd_core [26126.249345] ? queued_spin_lock_slowpath+0x16/0x2e0 [26126.251526] wmi ipmi_si [26126.253634] do_raw_spin_lock+0xb5/0xc0 [26126.257117] ipmi_devintf ipmi_msghandler [26126.259216] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26126.264518] ioatdma acpi_pad [26126.266792] try_get_cap_refs+0x159/0x880 [ceph] [26126.271659] acpi_power_meter ixgbe [26126.274195] __ceph_get_caps+0x4a1/0x760 [ceph] [26126.278020] igb mdio [26126.282028] ? __file_remove_privs+0x89/0x110 [26126.286809] crc32c_intel nvme [26126.289774] ? wake_bit_function+0x70/0x70 [26126.294384] ptp nvme_core [26126.297870] ? ceph_write_iter+0x260/0x770 [ceph] [26126.302392] i2c_algo_bit pps_core [26126.304662] ceph_write_iter+0x260/0x770 [ceph] [26126.309013] dca [26126.312068] ? lock_is_held_type+0xaa/0x120 [26126.316155] fuse [26126.318861] vfs_write+0x324/0x4f0 [26126.323555] irq event stamp: 130957860 [26126.326958] ksys_pwrite64+0x65/0xa0 [26126.331478] hardirqs last enabled at (130957859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26126.333314] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26126.337493] hardirqs last disabled at (130957860): [] sysvec_apic_timer_interrupt+0xe/0x90 [26126.339419] do_syscall_64+0x38/0x80 [26126.342814] softirqs last enabled at (130957858): [] __do_softirq+0x2be/0x371 [26126.346558] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26126.350128] softirqs last disabled at (130957853): [] irq_exit_rcu+0xb0/0xd0 [26126.360373] RIP: 0033:0x7f48f0eeb93b [26126.365504] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26126.375315] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26126.378884] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26126.387655] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26126.392701] Workqueue: events delayed_work [ceph] [26126.401297] ORIG_RAX: 0000000000000012 [26126.404869] [26126.404870] RIP: 0010:lock_release+0x3c/0x260 [26126.414246] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26126.432983] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [26126.440454] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26126.445671] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [26126.450369] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26126.454199] [26126.454200] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [26126.455691] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26126.460041] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [26126.467165] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26126.485902] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26126.485920] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26126.493051] [26126.498262] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26126.558948] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26126.567025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26126.572763] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26126.579886] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26126.587012] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26126.594135] Call Trace: [26126.596581] [26126.598592] ? watchdog_timer_fn+0x1b4/0x220 [26126.602855] ? __hrtimer_run_queues+0x1d3/0x360 [26126.607382] ? hrtimer_interrupt+0x104/0x240 [26126.611651] ? __local_bh_enable+0x50/0x60 [26126.615743] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26126.620958] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26126.625920] [26126.628013] [26126.630111] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26126.635415] ? igrab+0x43/0x50 [26126.638467] ? igrab+0x43/0x50 [26126.641518] ? lock_release+0x3c/0x260 [26126.645261] ? lock_release+0xc5/0x260 [26126.649008] _raw_spin_unlock+0x1b/0x40 [26126.652846] igrab+0x43/0x50 [26126.655722] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26126.660873] delayed_work+0x1d8/0x280 [ceph] [26126.665153] ? process_scheduled_works+0x22a/0x510 [26126.669944] process_scheduled_works+0x2a4/0x510 [26126.674559] worker_thread+0x188/0x340 [26126.678307] ? rescuer_thread+0x310/0x310 [26126.682311] kthread+0xf5/0x130 [26126.685449] ? kthread_complete_and_exit+0x20/0x20 [26126.690235] ret_from_fork+0x31/0x40 [26126.693811] ? kthread_complete_and_exit+0x20/0x20 [26126.698596] ret_from_fork_asm+0x11/0x20 [26126.702521] [26153.915714] watchdog: BUG: soft lockup - CPU#0 stuck for 13243s! [dbench:78683] [26153.923022] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26153.927714] watchdog: BUG: soft lockup - CPU#7 stuck for 22240s! [kworker/7:4:78808] [26153.978471] irq event stamp: 42401110 [26153.986201] Modules linked in: ceph libceph dns_resolver fscache netfs veth [26153.989861] hardirqs last enabled at (42401109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26153.996819] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [26154.006977] hardirqs last disabled at (42401110): [] sysvec_apic_timer_interrupt+0xe/0x90 [26154.006980] nf_defrag_ipv4 [26154.006982] softirqs last enabled at (42401104): [] __do_softirq+0x2be/0x371 [26154.013935] nft_compat nf_tables [26154.023660] softirqs last disabled at (42401099): [] irq_exit_rcu+0xb0/0xd0 [26154.026452] nfnetlink bridge [26154.035137] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26154.038445] stp llc [26154.046957] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26154.049922] xfs libcrc32c [26154.058868] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26154.061050] sunrpc [26154.068521] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26154.071224] intel_rapl_msr intel_rapl_common [26154.076701] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26154.078798] sb_edac x86_pkg_temp_thermal [26154.097536] [26154.097537] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26154.101887] coretemp kvm_intel [26154.107108] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26154.107109] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26154.111116] kvm [26154.112607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26154.119731] irqbypass i2c_i801 [26154.122870] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26154.129995] iTCO_wdt [26154.137118] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26154.138955] iTCO_vendor_support [26154.146079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26154.149217] i2c_smbus crct10dif_pclmul [26154.156341] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26154.158612] joydev [26154.166689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26154.169913] crc32_pclmul [26154.175650] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26154.179482] ipmi_ssif [26154.186607] Call Trace: [26154.188703] ghash_clmulni_intel [26154.195828] [26154.198443] mei_me mei [26154.205570] ? watchdog_timer_fn+0x1b4/0x220 [26154.207924] lpc_ich sha512_ssse3 [26154.210388] ? __hrtimer_run_queues+0x1d3/0x360 [26154.213611] acpi_ipmi [26154.215624] ? hrtimer_interrupt+0x104/0x240 [26154.218066] mfd_core wmi [26154.222331] ? __local_bh_enable+0x50/0x60 [26154.225642] ipmi_si [26154.230168] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26154.232522] ipmi_devintf [26154.236786] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26154.239403] ipmi_msghandler ioatdma [26154.243494] [26154.245680] acpi_pad [26154.250895] [26154.250897] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26154.253512] acpi_power_meter [26154.258474] ? queued_spin_lock_slowpath+0x18/0x2e0 [26154.262041] ixgbe igb [26154.264142] do_raw_spin_lock+0xb5/0xc0 [26154.266411] mdio crc32c_intel [26154.268508] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26154.273810] nvme ptp [26154.276777] try_get_cap_refs+0x159/0x880 [ceph] [26154.281645] nvme_core i2c_algo_bit [26154.284009] __ceph_get_caps+0x4a1/0x760 [ceph] [26154.287833] pps_core dca [26154.290889] ? __file_remove_privs+0x89/0x110 [26154.295669] fuse [26154.297941] ? wake_bit_function+0x70/0x70 [26154.302551] irq event stamp: 131178812 [26154.306036] ? ceph_write_iter+0x260/0x770 [ceph] [26154.310559] hardirqs last enabled at (131178811): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26154.313176] ceph_write_iter+0x260/0x770 [ceph] [26154.317527] hardirqs last disabled at (131178812): [] sysvec_apic_timer_interrupt+0xe/0x90 [26154.319455] ? lock_is_held_type+0xaa/0x120 [26154.323540] softirqs last enabled at (131178810): [] __do_softirq+0x2be/0x371 [26154.327288] vfs_write+0x324/0x4f0 [26154.331982] softirqs last disabled at (131178805): [] irq_exit_rcu+0xb0/0xd0 [26154.342231] ksys_pwrite64+0x65/0xa0 [26154.346750] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26154.356563] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26154.360739] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26154.369511] do_syscall_64+0x38/0x80 [26154.372907] Workqueue: events delayed_work [ceph] [26154.381506] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26154.385075] [26154.394455] RIP: 0033:0x7f48f0eeb93b [26154.399584] RIP: 0010:lock_acquire+0xf0/0x2b0 [26154.407056] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26154.410624] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26154.415322] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26154.420367] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26154.421859] ORIG_RAX: 0000000000000012 [26154.425428] [26154.425429] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26154.429779] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26154.448517] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26154.467255] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26154.472472] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26154.477689] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26154.481520] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26154.483010] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26154.490135] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26154.497258] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26154.504398] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26154.511530] [26154.518648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26154.570268] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26154.577409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26154.584534] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26154.591659] Call Trace: [26154.594119] [26154.596131] ? watchdog_timer_fn+0x1b4/0x220 [26154.600398] ? __hrtimer_run_queues+0x1d3/0x360 [26154.604931] ? hrtimer_interrupt+0x104/0x240 [26154.609200] ? __local_bh_enable+0x50/0x60 [26154.613295] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26154.618517] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26154.623474] [26154.625572] [26154.627669] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26154.632978] ? lock_acquire+0xf0/0x2b0 [26154.636727] ? igrab+0x43/0x50 [26154.639778] ? lock_release+0xc5/0x260 [26154.643521] _raw_spin_lock+0x30/0x40 [26154.647179] ? igrab+0x1d/0x50 [26154.650230] igrab+0x1d/0x50 [26154.653120] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26154.658266] delayed_work+0x1d8/0x280 [ceph] [26154.662547] ? process_scheduled_works+0x22a/0x510 [26154.667338] process_scheduled_works+0x2a4/0x510 [26154.671954] worker_thread+0x188/0x340 [26154.675701] ? rescuer_thread+0x310/0x310 [26154.679704] kthread+0xf5/0x130 [26154.682844] ? kthread_complete_and_exit+0x20/0x20 [26154.687628] ret_from_fork+0x31/0x40 [26154.691207] ? kthread_complete_and_exit+0x20/0x20 [26154.695991] ret_from_fork_asm+0x11/0x20 [26154.699923] [26181.916196] watchdog: BUG: soft lockup - CPU#0 stuck for 13269s! [dbench:78683] [26181.923501] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26181.928196] watchdog: BUG: soft lockup - CPU#7 stuck for 22266s! [kworker/7:4:78808] [26181.978941] irq event stamp: 42459348 [26181.978943] hardirqs last enabled at (42459347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26181.986672] Modules linked in: [26181.990329] hardirqs last disabled at (42459348): [] sysvec_apic_timer_interrupt+0xe/0x90 [26182.000486] ceph [26182.003537] softirqs last enabled at (42459252): [] __do_softirq+0x2be/0x371 [26182.013263] libceph [26182.015185] softirqs last disabled at (42459247): [] irq_exit_rcu+0xb0/0xd0 [26182.023870] dns_resolver fscache [26182.026055] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26182.034565] netfs veth [26182.037892] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26182.046837] nft_chain_nat xt_MASQUERADE [26182.049283] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26182.056753] nf_nat nf_conntrack [26182.060669] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26182.066146] nf_defrag_ipv6 nf_defrag_ipv4 [26182.069371] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26182.088109] nft_compat [26182.092200] [26182.092201] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26182.097418] nf_tables nfnetlink [26182.099874] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26182.101362] bridge stp [26182.108485] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26182.111709] llc xfs [26182.118833] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26182.121277] libcrc32c sunrpc [26182.128400] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26182.130591] intel_rapl_msr intel_rapl_common [26182.137717] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26182.140681] sb_edac x86_pkg_temp_thermal [26182.147805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26182.152156] coretemp kvm_intel [26182.160234] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26182.164239] kvm irqbypass [26182.169976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26182.173113] i2c_i801 iTCO_wdt [26182.180237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26182.182942] iTCO_vendor_support [26182.190066] Call Trace: [26182.190067] [26182.193116] i2c_smbus [26182.200240] ? watchdog_timer_fn+0x1b4/0x220 [26182.203464] crct10dif_pclmul joydev [26182.205910] ? __hrtimer_run_queues+0x1d3/0x360 [26182.207918] crc32_pclmul ipmi_ssif [26182.210280] ? hrtimer_interrupt+0x104/0x240 [26182.214540] ghash_clmulni_intel mei_me [26182.218112] ? __local_bh_enable+0x50/0x60 [26182.222635] mei [26182.226122] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26182.230384] lpc_ich sha512_ssse3 [26182.234216] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26182.238305] acpi_ipmi mfd_core [26182.240142] [26182.245361] wmi ipmi_si [26182.248671] [26182.248673] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26182.253628] ipmi_devintf [26182.256768] ? queued_spin_lock_slowpath+0x16/0x2e0 [26182.258873] ipmi_msghandler [26182.261404] do_raw_spin_lock+0xb5/0xc0 [26182.263501] ioatdma acpi_pad [26182.268803] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26182.271421] acpi_power_meter ixgbe [26182.276293] try_get_cap_refs+0x159/0x880 [ceph] [26182.279168] igb mdio [26182.283004] __ceph_get_caps+0x4a1/0x760 [ceph] [26182.285964] crc32c_intel nvme [26182.290752] ? __file_remove_privs+0x89/0x110 [26182.294231] ptp nvme_core [26182.298843] ? wake_bit_function+0x70/0x70 [26182.301112] i2c_algo_bit pps_core [26182.305639] ? ceph_write_iter+0x260/0x770 [ceph] [26182.308687] dca fuse [26182.313038] ceph_write_iter+0x260/0x770 [ceph] [26182.315744] [26182.315745] irq event stamp: 131399702 [26182.319837] ? lock_is_held_type+0xaa/0x120 [26182.323231] hardirqs last enabled at (131399701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26182.327931] vfs_write+0x324/0x4f0 [26182.330200] hardirqs last disabled at (131399702): [] sysvec_apic_timer_interrupt+0xe/0x90 [26182.334728] ksys_pwrite64+0x65/0xa0 [26182.336214] softirqs last enabled at (131399700): [] __do_softirq+0x2be/0x371 [26182.339959] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26182.344135] softirqs last disabled at (131399695): [] irq_exit_rcu+0xb0/0xd0 [26182.354382] do_syscall_64+0x38/0x80 [26182.357778] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26182.367598] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26182.371168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26182.379938] RIP: 0033:0x7f48f0eeb93b [26182.385069] Workqueue: events delayed_work [ceph] [26182.393667] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26182.397236] [26182.397237] RIP: 0010:lock_acquire+0xf0/0x2b0 [26182.406615] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26182.411658] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26182.419130] ORIG_RAX: 0000000000000012 [26182.422701] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26182.427397] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26182.446134] [26182.446135] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26182.447627] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26182.451978] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26182.457195] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26182.475931] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26182.479761] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26182.484981] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26182.492105] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26182.493599] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26182.500726] [26182.507842] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26182.567983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26182.573721] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26182.580844] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26182.587969] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26182.595092] Call Trace: [26182.597537] [26182.599548] ? watchdog_timer_fn+0x1b4/0x220 [26182.603812] ? __hrtimer_run_queues+0x1d3/0x360 [26182.608339] ? hrtimer_interrupt+0x104/0x240 [26182.612610] ? __local_bh_enable+0x50/0x60 [26182.616702] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26182.621927] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26182.626893] [26182.628990] [26182.631086] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26182.636393] ? lock_acquire+0xf0/0x2b0 [26182.640144] ? igrab+0x43/0x50 [26182.643196] ? lock_release+0xc5/0x260 [26182.646940] _raw_spin_lock+0x30/0x40 [26182.650605] ? igrab+0x1d/0x50 [26182.653654] igrab+0x1d/0x50 [26182.656532] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26182.661682] delayed_work+0x1d8/0x280 [ceph] [26182.665963] ? process_scheduled_works+0x22a/0x510 [26182.670755] process_scheduled_works+0x2a4/0x510 [26182.675370] worker_thread+0x188/0x340 [26182.679118] ? rescuer_thread+0x310/0x310 [26182.683122] kthread+0xf5/0x130 [26182.686259] ? kthread_complete_and_exit+0x20/0x20 [26182.691045] ret_from_fork+0x31/0x40 [26182.694624] ? kthread_complete_and_exit+0x20/0x20 [26182.699407] ret_from_fork_asm+0x11/0x20 [26182.703329] [26209.916678] watchdog: BUG: soft lockup - CPU#0 stuck for 13295s! [dbench:78683] [26209.923980] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26209.928677] watchdog: BUG: soft lockup - CPU#7 stuck for 22292s! [kworker/7:4:78808] [26209.979430] irq event stamp: 42517554 [26209.979432] hardirqs last enabled at (42517553): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26209.987160] Modules linked in: [26209.990816] hardirqs last disabled at (42517554): [] sysvec_apic_timer_interrupt+0xe/0x90 [26210.000975] ceph [26210.004025] softirqs last enabled at (42517438): [] __do_softirq+0x2be/0x371 [26210.013748] libceph [26210.015673] softirqs last disabled at (42517433): [] irq_exit_rcu+0xb0/0xd0 [26210.024358] dns_resolver fscache [26210.026543] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26210.035069] netfs veth [26210.038382] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26210.047325] nft_chain_nat xt_MASQUERADE [26210.049770] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26210.057239] nf_nat nf_conntrack [26210.061157] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26210.066636] nf_defrag_ipv6 nf_defrag_ipv4 [26210.069858] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26210.088596] nft_compat nf_tables [26210.092687] [26210.092688] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26210.097904] nfnetlink [26210.101215] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26210.102705] bridge stp [26210.109831] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26210.112187] llc xfs [26210.119313] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26210.121756] libcrc32c sunrpc [26210.128879] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26210.131074] intel_rapl_msr [26210.138196] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26210.141160] intel_rapl_common sb_edac [26210.148286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26210.151075] x86_pkg_temp_thermal [26210.159153] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26210.162897] coretemp kvm_intel [26210.168635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26210.171946] kvm irqbypass [26210.179073] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26210.182207] i2c_i801 iTCO_wdt [26210.189333] Call Trace: [26210.192053] iTCO_vendor_support [26210.199194] [26210.202244] i2c_smbus [26210.204692] ? watchdog_timer_fn+0x1b4/0x220 [26210.207912] crct10dif_pclmul [26210.209924] ? __hrtimer_run_queues+0x1d3/0x360 [26210.212281] joydev crc32_pclmul [26210.216547] ? hrtimer_interrupt+0x104/0x240 [26210.219509] ipmi_ssif [26210.224033] ? __local_bh_enable+0x50/0x60 [26210.227257] ghash_clmulni_intel mei_me [26210.231523] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26210.233880] mei [26210.237969] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26210.241801] lpc_ich sha512_ssse3 [26210.247017] [26210.248854] acpi_ipmi mfd_core [26210.253813] [26210.253815] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26210.257124] wmi ipmi_si [26210.259223] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26210.262363] ipmi_devintf [26210.264465] do_raw_spin_lock+0xb5/0xc0 [26210.269767] ipmi_msghandler ioatdma [26210.272301] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26210.277168] acpi_pad acpi_power_meter [26210.279788] try_get_cap_refs+0x159/0x880 [ceph] [26210.283617] ixgbe igb [26210.287193] __ceph_get_caps+0x4a1/0x760 [ceph] [26210.291973] mdio crc32c_intel [26210.295720] ? __file_remove_privs+0x89/0x110 [26210.300328] nvme [26210.302684] ? wake_bit_function+0x70/0x70 [26210.307209] ptp nvme_core [26210.310261] ? ceph_write_iter+0x260/0x770 [ceph] [26210.314609] i2c_algo_bit pps_core [26210.316533] ceph_write_iter+0x260/0x770 [ceph] [26210.320624] dca [26210.323334] ? lock_is_held_type+0xaa/0x120 [26210.328026] fuse [26210.331425] vfs_write+0x324/0x4f0 [26210.335948] [26210.335949] irq event stamp: 131620976 [26210.337790] ksys_pwrite64+0x65/0xa0 [26210.341963] hardirqs last enabled at (131620975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26210.343887] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26210.347283] hardirqs last disabled at (131620976): [] sysvec_apic_timer_interrupt+0xe/0x90 [26210.348777] do_syscall_64+0x38/0x80 [26210.352519] softirqs last enabled at (131620974): [] __do_softirq+0x2be/0x371 [26210.356090] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26210.366334] softirqs last disabled at (131620969): [] irq_exit_rcu+0xb0/0xd0 [26210.371466] RIP: 0033:0x7f48f0eeb93b [26210.381275] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26210.384847] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26210.393616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26210.398662] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26210.407258] Workqueue: events delayed_work [ceph] [26210.410830] ORIG_RAX: 0000000000000012 [26210.420205] [26210.420206] RIP: 0010:lock_release+0x14e/0x260 [26210.438943] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26210.446414] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [26210.451632] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26210.456331] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [26210.460160] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26210.461651] [26210.461652] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [26210.466088] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26210.473212] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [26210.491951] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26210.499082] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26210.504307] [26210.511424] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26210.557846] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [26210.564968] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26210.573046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26210.578785] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26210.585909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26210.593033] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26210.600155] Call Trace: [26210.602601] [26210.604612] ? watchdog_timer_fn+0x1b4/0x220 [26210.608876] ? __hrtimer_run_queues+0x1d3/0x360 [26210.613402] ? hrtimer_interrupt+0x104/0x240 [26210.617662] ? __local_bh_enable+0x50/0x60 [26210.621755] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26210.626973] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26210.631930] [26210.634027] [26210.636124] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26210.641428] ? igrab+0x43/0x50 [26210.644481] ? lock_release+0x14e/0x260 [26210.648310] ? lock_release+0xc5/0x260 [26210.652056] _raw_spin_unlock+0x1b/0x40 [26210.655912] igrab+0x43/0x50 [26210.658790] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26210.663939] delayed_work+0x1d8/0x280 [ceph] [26210.668219] ? process_scheduled_works+0x22a/0x510 [26210.673009] process_scheduled_works+0x2a4/0x510 [26210.677625] worker_thread+0x188/0x340 [26210.681374] ? rescuer_thread+0x310/0x310 [26210.685377] kthread+0xf5/0x130 [26210.688516] ? kthread_complete_and_exit+0x20/0x20 [26210.693299] ret_from_fork+0x31/0x40 [26210.696878] ? kthread_complete_and_exit+0x20/0x20 [26210.701663] ret_from_fork_asm+0x11/0x20 [26210.705594] [26237.917159] watchdog: BUG: soft lockup - CPU#0 stuck for 13321s! [dbench:78683] [26237.924459] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26237.929159] watchdog: BUG: soft lockup - CPU#7 stuck for 22318s! [kworker/7:4:78808] [26237.979900] irq event stamp: 42575940 [26237.979901] hardirqs last enabled at (42575939): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26237.987631] Modules linked in: [26237.991288] hardirqs last disabled at (42575940): [] sysvec_apic_timer_interrupt+0xe/0x90 [26238.001446] ceph [26238.004496] softirqs last enabled at (42575818): [] __do_softirq+0x2be/0x371 [26238.014219] libceph dns_resolver [26238.016145] softirqs last disabled at (42575813): [] irq_exit_rcu+0xb0/0xd0 [26238.024834] fscache netfs [26238.028148] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26238.036658] veth [26238.039363] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26238.048307] nft_chain_nat xt_MASQUERADE [26238.050230] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26238.057702] nf_nat nf_conntrack [26238.061618] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26238.067095] nf_defrag_ipv6 nf_defrag_ipv4 [26238.070321] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26238.089058] nft_compat nf_tables [26238.093148] [26238.093149] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26238.098366] nfnetlink bridge [26238.101677] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26238.103168] stp llc [26238.110292] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26238.113257] xfs libcrc32c [26238.120379] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26238.122565] sunrpc [26238.129687] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26238.132393] intel_rapl_msr [26238.139517] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26238.141613] intel_rapl_common [26238.148737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26238.151529] sb_edac x86_pkg_temp_thermal [26238.159623] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26238.162673] coretemp kvm_intel [26238.168411] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26238.172415] kvm irqbypass [26238.179557] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26238.182694] i2c_i801 iTCO_wdt [26238.189836] Call Trace: [26238.189837] [26238.192557] iTCO_vendor_support [26238.199681] ? watchdog_timer_fn+0x1b4/0x220 [26238.202731] i2c_smbus crct10dif_pclmul [26238.205177] ? __hrtimer_run_queues+0x1d3/0x360 [26238.207187] joydev crc32_pclmul [26238.210413] ? hrtimer_interrupt+0x104/0x240 [26238.214674] ipmi_ssif ghash_clmulni_intel [26238.218506] ? __local_bh_enable+0x50/0x60 [26238.223029] mei_me [26238.226256] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26238.230517] mei lpc_ich [26238.234609] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26238.238700] sha512_ssse3 [26238.240796] [26238.246014] acpi_ipmi mfd_core [26238.248559] [26238.253519] wmi ipmi_si [26238.256138] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26238.258234] ipmi_devintf ipmi_msghandler [26238.261377] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26238.263469] ioatdma [26238.266001] do_raw_spin_lock+0xb5/0xc0 [26238.271304] acpi_pad acpi_power_meter [26238.275309] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26238.280178] ixgbe igb [26238.282365] try_get_cap_refs+0x159/0x880 [ceph] [26238.286194] mdio crc32c_intel [26238.289944] __ceph_get_caps+0x4a1/0x760 [ceph] [26238.294722] nvme ptp [26238.297082] ? __file_remove_privs+0x89/0x110 [26238.301689] nvme_core i2c_algo_bit [26238.304742] ? wake_bit_function+0x70/0x70 [26238.309266] pps_core [26238.311555] ? ceph_write_iter+0x260/0x770 [ceph] [26238.315903] dca fuse [26238.319388] ceph_write_iter+0x260/0x770 [ceph] [26238.323478] [26238.323479] irq event stamp: 131841886 [26238.325754] ? lock_is_held_type+0xaa/0x120 [26238.330445] hardirqs last enabled at (131841885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26238.332720] vfs_write+0x324/0x4f0 [26238.337241] hardirqs last disabled at (131841886): [] sysvec_apic_timer_interrupt+0xe/0x90 [26238.338737] ksys_pwrite64+0x65/0xa0 [26238.342476] softirqs last enabled at (131841884): [] __do_softirq+0x2be/0x371 [26238.346654] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26238.356897] softirqs last disabled at (131841879): [] irq_exit_rcu+0xb0/0xd0 [26238.360297] do_syscall_64+0x38/0x80 [26238.370107] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26238.373678] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26238.382447] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26238.387578] RIP: 0033:0x7f48f0eeb93b [26238.396176] Workqueue: events delayed_work [ceph] [26238.399745] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26238.409123] [26238.409124] RIP: 0010:lock_release+0x14e/0x260 [26238.414167] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26238.421638] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [26238.425209] ORIG_RAX: 0000000000000012 [26238.429907] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [26238.448645] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26238.450136] [26238.454573] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26238.459790] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [26238.478528] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26238.482357] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [26238.487575] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26238.494700] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26238.496191] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26238.503315] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26238.510445] [26238.517562] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [26238.562492] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26238.570569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26238.576308] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26238.583431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26238.590563] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26238.597689] Call Trace: [26238.600134] [26238.602145] ? watchdog_timer_fn+0x1b4/0x220 [26238.606408] ? __hrtimer_run_queues+0x1d3/0x360 [26238.610936] ? hrtimer_interrupt+0x104/0x240 [26238.615204] ? __local_bh_enable+0x50/0x60 [26238.619296] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26238.624521] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26238.629477] [26238.631575] [26238.633674] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26238.638978] ? igrab+0x43/0x50 [26238.642031] ? lock_release+0x14e/0x260 [26238.645859] ? lock_release+0xc5/0x260 [26238.649604] _raw_spin_unlock+0x1b/0x40 [26238.653435] igrab+0x43/0x50 [26238.656313] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26238.661461] delayed_work+0x1d8/0x280 [ceph] [26238.665743] ? process_scheduled_works+0x22a/0x510 [26238.670532] process_scheduled_works+0x2a4/0x510 [26238.675148] worker_thread+0x188/0x340 [26238.678898] ? rescuer_thread+0x310/0x310 [26238.682902] kthread+0xf5/0x130 [26238.686038] ? kthread_complete_and_exit+0x20/0x20 [26238.690824] ret_from_fork+0x31/0x40 [26238.694409] ? kthread_complete_and_exit+0x20/0x20 [26238.699195] ret_from_fork_asm+0x11/0x20 [26238.703118] [26265.917642] watchdog: BUG: soft lockup - CPU#0 stuck for 13348s! [dbench:78683] [26265.924946] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26265.929642] watchdog: BUG: soft lockup - CPU#7 stuck for 22345s! [kworker/7:4:78808] [26265.980396] irq event stamp: 42634158 [26265.988128] Modules linked in: ceph libceph dns_resolver fscache netfs [26265.991785] hardirqs last enabled at (42634157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26265.998317] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [26266.008476] hardirqs last disabled at (42634158): [] sysvec_apic_timer_interrupt+0xe/0x90 [26266.015860] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [26266.025585] softirqs last enabled at (42634062): [] __do_softirq+0x2be/0x371 [26266.032371] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [26266.041054] softirqs last disabled at (42634057): [] irq_exit_rcu+0xb0/0xd0 [26266.047919] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [26266.056430] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26266.056431] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [26266.071545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26266.077196] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [26266.084667] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26266.090231] ghash_clmulni_intel mei_me mei [26266.095707] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26266.099886] lpc_ich sha512_ssse3 [26266.118623] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26266.118625] acpi_ipmi mfd_core [26266.121936] [26266.127152] wmi ipmi_si [26266.130287] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26266.131779] ipmi_devintf [26266.134310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26266.141450] ipmi_msghandler ioatdma [26266.144069] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26266.151193] acpi_pad acpi_power_meter [26266.154765] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26266.161889] ixgbe igb [26266.165632] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26266.172756] mdio crc32c_intel [26266.175113] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26266.182238] nvme ptp [26266.185289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26266.193366] nvme_core [26266.195636] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26266.201375] i2c_algo_bit [26266.203730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26266.210856] pps_core [26266.213473] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26266.220598] dca fuse [26266.222868] Call Trace: [26266.229992] [26266.232266] [26266.234705] irq event stamp: 132062902 [26266.236199] ? watchdog_timer_fn+0x1b4/0x220 [26266.238206] hardirqs last enabled at (132062901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26266.241954] ? __hrtimer_run_queues+0x1d3/0x360 [26266.246216] hardirqs last disabled at (132062902): [] sysvec_apic_timer_interrupt+0xe/0x90 [26266.256463] ? hrtimer_interrupt+0x104/0x240 [26266.260983] softirqs last enabled at (132062900): [] __do_softirq+0x2be/0x371 [26266.270796] ? __local_bh_enable+0x50/0x60 [26266.275058] softirqs last disabled at (132062895): [] irq_exit_rcu+0xb0/0xd0 [26266.283833] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26266.287922] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26266.296518] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26266.301735] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26266.311114] [26266.316072] Workqueue: events delayed_work [ceph] [26266.323542] [26266.323544] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26266.325639] [26266.325640] RIP: 0010:lock_release+0x14e/0x260 [26266.330340] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26266.332434] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [26266.337740] do_raw_spin_lock+0xb5/0xc0 [26266.339229] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [26266.343668] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26266.348537] [26266.348538] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [26266.367277] try_get_cap_refs+0x159/0x880 [ceph] [26266.371104] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [26266.376328] __ceph_get_caps+0x4a1/0x760 [ceph] [26266.381106] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26266.382602] ? __file_remove_privs+0x89/0x110 [26266.389720] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26266.394341] ? wake_bit_function+0x70/0x70 [26266.401465] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [26266.405991] ? ceph_write_iter+0x260/0x770 [ceph] [26266.413114] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26266.417463] ceph_write_iter+0x260/0x770 [ceph] [26266.424588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26266.428684] ? lock_is_held_type+0xaa/0x120 [26266.435803] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26266.440504] vfs_write+0x324/0x4f0 [26266.448577] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26266.453107] ksys_pwrite64+0x65/0xa0 [26266.458840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26266.463019] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26266.470159] Call Trace: [26266.473557] do_syscall_64+0x38/0x80 [26266.480682] [26266.484251] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26266.491376] ? watchdog_timer_fn+0x1b4/0x220 [26266.496507] RIP: 0033:0x7f48f0eeb93b [26266.498952] ? __hrtimer_run_queues+0x1d3/0x360 [26266.502519] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26266.504534] ? hrtimer_interrupt+0x104/0x240 [26266.509574] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26266.513839] ? __local_bh_enable+0x50/0x60 [26266.517410] ORIG_RAX: 0000000000000012 [26266.521938] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26266.540672] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26266.544936] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26266.550153] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26266.554244] [26266.558073] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26266.563291] [26266.563293] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26266.570416] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26266.575373] ? igrab+0x43/0x50 [26266.582497] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26266.584599] ? lock_release+0x14e/0x260 [26266.591725] [26266.593817] ? lock_release+0xc5/0x260 [26266.626182] _raw_spin_unlock+0x1b/0x40 [26266.630020] igrab+0x43/0x50 [26266.632915] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26266.638064] delayed_work+0x1d8/0x280 [ceph] [26266.642346] ? process_scheduled_works+0x22a/0x510 [26266.647137] process_scheduled_works+0x2a4/0x510 [26266.651751] worker_thread+0x188/0x340 [26266.655500] ? rescuer_thread+0x310/0x310 [26266.659502] kthread+0xf5/0x130 [26266.662641] ? kthread_complete_and_exit+0x20/0x20 [26266.667426] ret_from_fork+0x31/0x40 [26266.671005] ? kthread_complete_and_exit+0x20/0x20 [26266.675788] ret_from_fork_asm+0x11/0x20 [26266.679721] [26268.825058] rcu: INFO: rcu_preempt self-detected stall on CPU [26268.830820] rcu: 0-....: (14011021 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7002789 [26268.841237] rcu: (t=14331961 jiffies g=1309609 q=437057 ncpus=8) [26268.847322] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26268.856266] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26268.863736] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26268.869214] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26268.887953] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26268.893169] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26268.900293] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26268.907416] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26268.914542] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26268.921666] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26268.928788] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26268.936868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26268.942604] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26268.949728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26268.956852] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26268.963976] Call Trace: [26268.966422] [26268.968432] ? rcu_dump_cpu_stacks+0xd1/0x180 [26268.972791] ? rcu_sched_clock_irq+0x633/0x1390 [26268.977316] ? lock_is_held_type+0xaa/0x120 [26268.981494] ? lock_is_held_type+0xaa/0x120 [26268.985672] ? tick_sched_do_timer+0xa0/0xa0 [26268.989943] ? rcu_read_lock_sched_held+0x56/0x90 [26268.994640] ? tick_sched_do_timer+0xa0/0xa0 [26268.998903] ? update_process_times+0x5f/0x90 [26269.003254] ? tick_sched_handle.isra.16+0x1e/0x50 [26269.008040] ? tick_sched_timer+0x6f/0x80 [26269.012044] ? __hrtimer_run_queues+0x1d3/0x360 [26269.016572] ? hrtimer_interrupt+0x104/0x240 [26269.020839] ? __local_bh_enable+0x50/0x60 [26269.024934] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26269.030156] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26269.035114] [26269.037213] [26269.039310] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26269.044617] ? queued_spin_lock_slowpath+0x16/0x2e0 [26269.049495] do_raw_spin_lock+0xb5/0xc0 [26269.053332] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26269.058135] try_get_cap_refs+0x159/0x880 [ceph] [26269.062767] __ceph_get_caps+0x4a1/0x760 [ceph] [26269.067307] ? __file_remove_privs+0x89/0x110 [26269.071663] ? wake_bit_function+0x70/0x70 [26269.075757] ? ceph_write_iter+0x260/0x770 [ceph] [26269.080467] ceph_write_iter+0x260/0x770 [ceph] [26269.085003] ? lock_is_held_type+0xaa/0x120 [26269.089188] vfs_write+0x324/0x4f0 [26269.092589] ksys_pwrite64+0x65/0xa0 [26269.096163] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26269.101296] do_syscall_64+0x38/0x80 [26269.104866] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26269.109910] RIP: 0033:0x7f48f0eeb93b [26269.113480] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26269.132218] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [26269.139776] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26269.146898] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26269.154022] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26269.161146] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26269.168272] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26269.175401] [26269.177589] Sending NMI from CPU 0 to CPUs 7: [26269.181941] NMI backtrace for cpu 7 [26269.181943] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26269.181944] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26269.181945] Workqueue: events delayed_work [ceph] [26269.181961] RIP: 0010:__lock_acquire+0x66e/0x1e20 [26269.181963] Code: 41 01 ff c1 c0 06 31 d0 29 c7 89 c2 44 01 f8 c1 c2 08 41 89 fc 41 31 d4 44 89 e2 45 29 e7 41 01 c4 c1 c2 10 44 89 e1 41 31 d7 <44> 89 fa 44 29 f8 45 01 e7 c1 ca 0d 31 d0 89 c2 29 c1 44 01 f8 c1 [26269.181964] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [26269.181966] RAX: 000000004d0e10b2 RBX: ffff88810debaa80 RCX: 00000000ce5c2dd3 [26269.181966] RDX: 000000001d21814e RSI: 0000000000000000 RDI: 00000000f3668bb7 [26269.181967] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [26269.181968] R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ce5c2dd3 [26269.181968] R13: 0000000000000003 R14: 0000000000000000 R15: 00000000286c4bb5 [26269.181969] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26269.181970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26269.181971] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26269.181972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26269.181972] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26269.181973] Call Trace: [26269.181974] [26269.181975] ? nmi_cpu_backtrace+0xad/0x120 [26269.181978] ? nmi_cpu_backtrace_handler+0x11/0x20 [26269.181980] ? nmi_handle+0xd4/0x260 [26269.181984] ? default_do_nmi+0x49/0xf0 [26269.181985] ? exc_nmi+0x11e/0x150 [26269.181987] ? end_repeat_nmi+0x16/0x67 [26269.181990] ? __lock_acquire+0x66e/0x1e20 [26269.181992] ? __lock_acquire+0x66e/0x1e20 [26269.181994] ? __lock_acquire+0x66e/0x1e20 [26269.181995] [26269.181995] [26269.181997] lock_acquire+0xcb/0x2b0 [26269.181999] ? igrab+0x1d/0x50 [26269.182000] ? igrab+0x43/0x50 [26269.182001] ? lock_release+0xc5/0x260 [26269.182003] _raw_spin_lock+0x30/0x40 [26269.182005] ? igrab+0x1d/0x50 [26269.182006] igrab+0x1d/0x50 [26269.182007] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26269.182022] delayed_work+0x1d8/0x280 [ceph] [26269.182035] ? process_scheduled_works+0x22a/0x510 [26269.182037] process_scheduled_works+0x2a4/0x510 [26269.182041] worker_thread+0x188/0x340 [26269.182043] ? rescuer_thread+0x310/0x310 [26269.182045] kthread+0xf5/0x130 [26269.182047] ? kthread_complete_and_exit+0x20/0x20 [26269.182050] ret_from_fork+0x31/0x40 [26269.182052] ? kthread_complete_and_exit+0x20/0x20 [26269.182054] ret_from_fork_asm+0x11/0x20 [26269.182058] [26293.918129] watchdog: BUG: soft lockup - CPU#0 stuck for 13374s! [dbench:78683] [26293.925434] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26293.930129] watchdog: BUG: soft lockup - CPU#7 stuck for 22371s! [kworker/7:4:78808] [26293.980874] irq event stamp: 42691504 [26293.988606] Modules linked in: ceph libceph dns_resolver [26293.992262] hardirqs last enabled at (42691503): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26293.997566] fscache netfs veth [26294.007724] hardirqs last disabled at (42691504): [] sysvec_apic_timer_interrupt+0xe/0x90 [26294.010863] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [26294.020585] softirqs last enabled at (42691472): [] __do_softirq+0x2be/0x371 [26294.026236] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [26294.034920] softirqs last disabled at (42691467): [] irq_exit_rcu+0xb0/0xd0 [26294.040832] nfnetlink bridge stp [26294.049343] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26294.052654] llc xfs [26294.061598] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26294.061599] libcrc32c [26294.061600] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26294.063780] sunrpc intel_rapl_msr [26294.071270] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26294.073626] intel_rapl_common sb_edac [26294.079104] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26294.082502] x86_pkg_temp_thermal coretemp [26294.101258] [26294.105002] kvm_intel kvm [26294.110218] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26294.114310] irqbypass i2c_i801 [26294.115803] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26294.118522] iTCO_wdt iTCO_vendor_support [26294.125645] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26294.128783] i2c_smbus crct10dif_pclmul [26294.135925] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26294.139930] joydev crc32_pclmul [26294.147051] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26294.150884] ipmi_ssif ghash_clmulni_intel [26294.158008] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26294.161232] mei_me mei [26294.168354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26294.172446] lpc_ich sha512_ssse3 [26294.180528] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26294.182977] acpi_ipmi mfd_core [26294.188713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26294.192025] wmi [26294.199148] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26294.202286] ipmi_si ipmi_devintf [26294.209410] Call Trace: [26294.209411] [26294.211248] ipmi_msghandler ioatdma [26294.218373] ? watchdog_timer_fn+0x1b4/0x220 [26294.221682] acpi_pad acpi_power_meter [26294.224129] ? __hrtimer_run_queues+0x1d3/0x360 [26294.226137] ixgbe igb [26294.229711] ? hrtimer_interrupt+0x104/0x240 [26294.233972] mdio crc32c_intel [26294.237716] ? __local_bh_enable+0x50/0x60 [26294.242239] nvme ptp [26294.244600] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26294.248862] nvme_core i2c_algo_bit [26294.251912] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26294.256003] pps_core [26294.258273] [26294.263490] dca fuse [26294.266974] [26294.266976] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26294.271933] [26294.271933] irq event stamp: 132281922 [26294.274207] ? queued_spin_lock_slowpath+0x18/0x2e0 [26294.276300] hardirqs last enabled at (132281921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26294.278572] do_raw_spin_lock+0xb5/0xc0 [26294.280668] hardirqs last disabled at (132281922): [] sysvec_apic_timer_interrupt+0xe/0x90 [26294.285975] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26294.287464] softirqs last enabled at (132281920): [] __do_softirq+0x2be/0x371 [26294.291210] try_get_cap_refs+0x159/0x880 [ceph] [26294.296080] softirqs last disabled at (132281911): [] irq_exit_rcu+0xb0/0xd0 [26294.306329] __ceph_get_caps+0x4a1/0x760 [ceph] [26294.310154] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26294.319970] ? __file_remove_privs+0x89/0x110 [26294.324748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26294.333529] ? wake_bit_function+0x70/0x70 [26294.338139] Workqueue: events delayed_work [ceph] [26294.346738] ? ceph_write_iter+0x260/0x770 [ceph] [26294.351261] [26294.360638] ceph_write_iter+0x260/0x770 [ceph] [26294.364987] RIP: 0010:lock_release+0x14e/0x260 [26294.372463] ? lock_is_held_type+0xaa/0x120 [26294.376549] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [26294.381251] vfs_write+0x324/0x4f0 [26294.385944] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [26294.387439] ksys_pwrite64+0x65/0xa0 [26294.391959] [26294.391959] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [26294.396398] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26294.400574] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [26294.419314] do_syscall_64+0x38/0x80 [26294.422709] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26294.427928] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26294.431498] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26294.432989] RIP: 0033:0x7f48f0eeb93b [26294.440111] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [26294.445243] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26294.452368] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26294.455939] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26294.463061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26294.468107] ORIG_RAX: 0000000000000012 [26294.475230] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26294.478811] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26294.485935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26294.504672] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26294.512748] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26294.517966] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26294.523704] Call Trace: [26294.523705] [26294.527535] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26294.534660] ? watchdog_timer_fn+0x1b4/0x220 [26294.541783] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26294.548910] ? __hrtimer_run_queues+0x1d3/0x360 [26294.556036] [26294.563158] ? hrtimer_interrupt+0x104/0x240 [26294.604219] ? __local_bh_enable+0x50/0x60 [26294.608312] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26294.613535] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26294.618493] [26294.620591] [26294.622688] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26294.627994] ? igrab+0x43/0x50 [26294.631046] ? lock_release+0x14e/0x260 [26294.634876] ? lock_release+0xc5/0x260 [26294.638621] _raw_spin_unlock+0x1b/0x40 [26294.642458] igrab+0x43/0x50 [26294.645337] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26294.650485] delayed_work+0x1d8/0x280 [ceph] [26294.654766] ? process_scheduled_works+0x22a/0x510 [26294.659548] process_scheduled_works+0x2a4/0x510 [26294.664164] worker_thread+0x188/0x340 [26294.667912] ? rescuer_thread+0x310/0x310 [26294.671916] kthread+0xf5/0x130 [26294.675052] ? kthread_complete_and_exit+0x20/0x20 [26294.679838] ret_from_fork+0x31/0x40 [26294.683407] ? kthread_complete_and_exit+0x20/0x20 [26294.688192] ret_from_fork_asm+0x11/0x20 [26294.692117] [26321.918616] watchdog: BUG: soft lockup - CPU#0 stuck for 13400s! [dbench:78683] [26321.925922] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26321.930616] watchdog: BUG: soft lockup - CPU#7 stuck for 22397s! [kworker/7:4:78808] [26321.981371] irq event stamp: 42749634 [26321.981372] hardirqs last enabled at (42749633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26321.989103] Modules linked in: [26321.992759] hardirqs last disabled at (42749634): [] sysvec_apic_timer_interrupt+0xe/0x90 [26322.002916] ceph [26322.005967] softirqs last enabled at (42749624): [] __do_softirq+0x2be/0x371 [26322.015691] libceph [26322.017616] softirqs last disabled at (42749619): [] irq_exit_rcu+0xb0/0xd0 [26322.026300] dns_resolver [26322.028486] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26322.037011] fscache netfs [26322.039631] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26322.048575] veth [26322.051278] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26322.058766] nft_chain_nat [26322.060689] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26322.066166] xt_MASQUERADE nf_nat [26322.068871] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26322.087609] nf_conntrack nf_defrag_ipv6 [26322.090922] [26322.096137] nf_defrag_ipv4 nft_compat [26322.100056] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26322.101545] nf_tables [26322.105306] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26322.112430] nfnetlink [26322.114789] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26322.121913] bridge [26322.124269] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26322.131410] stp [26322.133510] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26322.140634] llc [26322.142471] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26322.149594] xfs libcrc32c [26322.151431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26322.159508] sunrpc intel_rapl_msr [26322.162214] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26322.167950] intel_rapl_common [26322.171347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26322.178473] sb_edac x86_pkg_temp_thermal [26322.181523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26322.188647] coretemp [26322.192653] Call Trace: [26322.199775] kvm_intel [26322.202048] [26322.204490] kvm irqbypass [26322.206851] ? watchdog_timer_fn+0x1b4/0x220 [26322.208858] i2c_i801 iTCO_wdt [26322.211563] ? __hrtimer_run_queues+0x1d3/0x360 [26322.215825] iTCO_vendor_support [26322.218879] ? hrtimer_interrupt+0x104/0x240 [26322.223402] i2c_smbus [26322.226625] ? __local_bh_enable+0x50/0x60 [26322.230888] crct10dif_pclmul joydev [26322.233248] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26322.237338] crc32_pclmul ipmi_ssif [26322.240908] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26322.246124] ghash_clmulni_intel mei_me [26322.249610] [26322.254567] mei lpc_ich [26322.258398] [26322.260494] sha512_ssse3 acpi_ipmi [26322.263026] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26322.265124] mfd_core wmi [26322.268610] ? queued_spin_lock_slowpath+0x16/0x2e0 [26322.273910] ipmi_si ipmi_devintf [26322.276532] do_raw_spin_lock+0xb5/0xc0 [26322.281400] ipmi_msghandler ioatdma [26322.284712] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26322.288541] acpi_pad acpi_power_meter [26322.292114] try_get_cap_refs+0x159/0x880 [ceph] [26322.296896] ixgbe igb [26322.300646] __ceph_get_caps+0x4a1/0x760 [ceph] [26322.305252] mdio crc32c_intel [26322.307613] ? __file_remove_privs+0x89/0x110 [26322.312132] nvme [26322.315182] ? wake_bit_function+0x70/0x70 [26322.319533] ptp nvme_core [26322.321460] ? ceph_write_iter+0x260/0x770 [ceph] [26322.325549] i2c_algo_bit [26322.328253] ceph_write_iter+0x260/0x770 [ceph] [26322.332951] pps_core [26322.335572] ? lock_is_held_type+0xaa/0x120 [26322.340092] dca fuse [26322.342364] vfs_write+0x324/0x4f0 [26322.346539] [26322.346540] irq event stamp: 132502894 [26322.348814] ksys_pwrite64+0x65/0xa0 [26322.352208] hardirqs last enabled at (132502893): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26322.353700] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26322.357441] hardirqs last disabled at (132502894): [] sysvec_apic_timer_interrupt+0xe/0x90 [26322.361018] do_syscall_64+0x38/0x80 [26322.371258] softirqs last enabled at (132502892): [] __do_softirq+0x2be/0x371 [26322.376387] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26322.386199] softirqs last disabled at (132502887): [] irq_exit_rcu+0xb0/0xd0 [26322.389771] RIP: 0033:0x7f48f0eeb93b [26322.398541] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26322.403585] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26322.412182] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26322.415752] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26322.425132] Workqueue: events delayed_work [ceph] [26322.443868] ORIG_RAX: 0000000000000012 [26322.451339] [26322.451340] RIP: 0010:lock_acquire+0xf0/0x2b0 [26322.456556] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26322.461254] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26322.465085] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26322.466576] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26322.470925] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26322.478051] [26322.478052] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26322.496788] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26322.503911] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26322.509128] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26322.516253] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26322.517750] [26322.524868] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26322.562690] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26322.569814] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26322.577891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26322.583630] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26322.590754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26322.597877] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26322.605020] Call Trace: [26322.607464] [26322.609475] ? watchdog_timer_fn+0x1b4/0x220 [26322.613738] ? __hrtimer_run_queues+0x1d3/0x360 [26322.618265] ? hrtimer_interrupt+0x104/0x240 [26322.622535] ? __local_bh_enable+0x50/0x60 [26322.626629] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26322.631851] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26322.636811] [26322.638907] [26322.641021] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26322.646330] ? lock_acquire+0xf0/0x2b0 [26322.650080] ? igrab+0x43/0x50 [26322.653129] ? lock_release+0xc5/0x260 [26322.656875] _raw_spin_lock+0x30/0x40 [26322.660540] ? igrab+0x1d/0x50 [26322.663589] igrab+0x1d/0x50 [26322.666470] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26322.671620] delayed_work+0x1d8/0x280 [ceph] [26322.675899] ? process_scheduled_works+0x22a/0x510 [26322.680690] process_scheduled_works+0x2a4/0x510 [26322.685317] worker_thread+0x188/0x340 [26322.689062] ? rescuer_thread+0x310/0x310 [26322.693065] kthread+0xf5/0x130 [26322.696205] ? kthread_complete_and_exit+0x20/0x20 [26322.700997] ret_from_fork+0x31/0x40 [26322.704593] ? kthread_complete_and_exit+0x20/0x20 [26322.709377] ret_from_fork_asm+0x11/0x20 [26322.713316] [26349.919104] watchdog: BUG: soft lockup - CPU#0 stuck for 13426s! [dbench:78683] [26349.926401] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26349.931104] watchdog: BUG: soft lockup - CPU#7 stuck for 22423s! [kworker/7:4:78808] [26349.981843] irq event stamp: 42807814 [26349.981844] hardirqs last enabled at (42807813): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26349.989572] Modules linked in: [26349.993230] hardirqs last disabled at (42807814): [] sysvec_apic_timer_interrupt+0xe/0x90 [26350.003388] ceph [26350.006437] softirqs last enabled at (42807786): [] __do_softirq+0x2be/0x371 [26350.016162] libceph [26350.018087] softirqs last disabled at (42807781): [] irq_exit_rcu+0xb0/0xd0 [26350.026780] dns_resolver fscache [26350.028964] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26350.037474] netfs veth [26350.040801] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26350.049746] nft_chain_nat [26350.052192] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26350.059661] xt_MASQUERADE nf_nat [26350.062366] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26350.067844] nf_conntrack nf_defrag_ipv6 [26350.071153] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26350.089892] nf_defrag_ipv4 nft_compat [26350.093807] [26350.093808] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26350.099026] nf_tables nfnetlink [26350.102782] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26350.104270] bridge stp [26350.111395] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26350.114617] llc xfs [26350.121742] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26350.124185] libcrc32c sunrpc [26350.131310] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26350.133499] intel_rapl_msr [26350.140626] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26350.143591] intel_rapl_common [26350.150715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26350.153504] sb_edac x86_pkg_temp_thermal [26350.161582] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26350.164633] coretemp kvm_intel [26350.170370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26350.174375] kvm irqbypass [26350.181503] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26350.184636] i2c_i801 [26350.191760] Call Trace: [26350.191761] [26350.194498] iTCO_wdt [26350.201625] ? watchdog_timer_fn+0x1b4/0x220 [26350.203895] iTCO_vendor_support [26350.206341] ? __hrtimer_run_queues+0x1d3/0x360 [26350.208351] i2c_smbus crct10dif_pclmul [26350.210624] ? hrtimer_interrupt+0x104/0x240 [26350.214884] joydev crc32_pclmul [26350.218108] ? __local_bh_enable+0x50/0x60 [26350.222632] ipmi_ssif ghash_clmulni_intel [26350.226465] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26350.230726] mei_me mei [26350.233950] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26350.238043] lpc_ich sha512_ssse3 [26350.242133] [26350.247350] acpi_ipmi [26350.249793] [26350.249795] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26350.254751] mfd_core wmi [26350.258066] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26350.260159] ipmi_si ipmi_devintf [26350.262519] do_raw_spin_lock+0xb5/0xc0 [26350.264613] ipmi_msghandler ioatdma [26350.269919] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26350.272537] acpi_pad acpi_power_meter [26350.277410] try_get_cap_refs+0x159/0x880 [ceph] [26350.280717] ixgbe igb [26350.284554] __ceph_get_caps+0x4a1/0x760 [ceph] [26350.288118] mdio crc32c_intel [26350.292907] ? __file_remove_privs+0x89/0x110 [26350.296646] nvme ptp [26350.301259] ? wake_bit_function+0x70/0x70 [26350.303616] nvme_core i2c_algo_bit [26350.308143] ? ceph_write_iter+0x260/0x770 [ceph] [26350.311189] pps_core dca [26350.315540] ceph_write_iter+0x260/0x770 [ceph] [26350.317811] fuse [26350.321907] ? lock_is_held_type+0xaa/0x120 [26350.325386] irq event stamp: 132723754 [26350.330087] vfs_write+0x324/0x4f0 [26350.332702] hardirqs last enabled at (132723753): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26350.337230] ksys_pwrite64+0x65/0xa0 [26350.339149] hardirqs last disabled at (132723754): [] sysvec_apic_timer_interrupt+0xe/0x90 [26350.343327] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26350.347070] softirqs last enabled at (132723752): [] __do_softirq+0x2be/0x371 [26350.350468] do_syscall_64+0x38/0x80 [26350.360713] softirqs last disabled at (132723747): [] irq_exit_rcu+0xb0/0xd0 [26350.364284] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26350.374094] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26350.379224] RIP: 0033:0x7f48f0eeb93b [26350.387995] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26350.391565] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26350.400165] Workqueue: events delayed_work [ceph] [26350.405207] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26350.414584] [26350.414585] RIP: 0010:lock_acquire+0xf0/0x2b0 [26350.418156] ORIG_RAX: 0000000000000012 [26350.425627] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26350.444365] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26350.449061] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26350.454280] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26350.455771] [26350.455772] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26350.460121] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26350.463950] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26350.482690] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26350.489813] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26350.495031] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26350.502155] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26350.503651] [26350.510770] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26350.562822] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26350.570901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26350.576637] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26350.583761] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26350.590885] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26350.598010] Call Trace: [26350.600454] [26350.602467] ? watchdog_timer_fn+0x1b4/0x220 [26350.606730] ? __hrtimer_run_queues+0x1d3/0x360 [26350.611258] ? hrtimer_interrupt+0x104/0x240 [26350.615526] ? __local_bh_enable+0x50/0x60 [26350.619619] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26350.624842] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26350.629802] [26350.631896] [26350.633996] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26350.639304] ? lock_acquire+0xf0/0x2b0 [26350.643053] ? igrab+0x43/0x50 [26350.646104] ? lock_release+0xc5/0x260 [26350.649849] _raw_spin_lock+0x30/0x40 [26350.653514] ? igrab+0x1d/0x50 [26350.656563] igrab+0x1d/0x50 [26350.659442] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26350.664591] delayed_work+0x1d8/0x280 [ceph] [26350.668873] ? process_scheduled_works+0x22a/0x510 [26350.673663] process_scheduled_works+0x2a4/0x510 [26350.678277] worker_thread+0x188/0x340 [26350.682029] ? rescuer_thread+0x310/0x310 [26350.686040] kthread+0xf5/0x130 [26350.689177] ? kthread_complete_and_exit+0x20/0x20 [26350.693963] ret_from_fork+0x31/0x40 [26350.697541] ? kthread_complete_and_exit+0x20/0x20 [26350.702325] ret_from_fork_asm+0x11/0x20 [26350.706248] [26377.919591] watchdog: BUG: soft lockup - CPU#0 stuck for 13452s! [dbench:78683] [26377.926898] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26377.931591] watchdog: BUG: soft lockup - CPU#7 stuck for 22449s! [kworker/7:4:78808] [26377.982347] irq event stamp: 42866066 [26377.982348] hardirqs last enabled at (42866065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26377.990079] Modules linked in: [26377.993736] hardirqs last disabled at (42866066): [] sysvec_apic_timer_interrupt+0xe/0x90 [26378.003893] ceph [26378.006943] softirqs last enabled at (42866038): [] __do_softirq+0x2be/0x371 [26378.016668] libceph [26378.018591] softirqs last disabled at (42866033): [] irq_exit_rcu+0xb0/0xd0 [26378.027275] dns_resolver fscache [26378.029462] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26378.037986] netfs veth [26378.041300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26378.050242] nft_chain_nat xt_MASQUERADE [26378.052705] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26378.060175] nf_nat nf_conntrack [26378.064092] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26378.069569] nf_defrag_ipv6 [26378.072793] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26378.091532] nf_defrag_ipv4 nft_compat [26378.094323] [26378.099541] nf_tables [26378.103285] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26378.104773] nfnetlink bridge [26378.107132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26378.114268] stp [26378.117228] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26378.124353] llc xfs [26378.126190] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26378.133315] libcrc32c sunrpc [26378.135499] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26378.142624] intel_rapl_msr [26378.145588] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26378.152710] intel_rapl_common [26378.155501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26378.163580] sb_edac x86_pkg_temp_thermal [26378.166629] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26378.172368] coretemp kvm_intel [26378.176371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26378.183496] kvm irqbypass [26378.186632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26378.193757] i2c_i801 [26378.196461] Call Trace: [26378.196463] [26378.203585] iTCO_wdt iTCO_vendor_support [26378.205859] ? watchdog_timer_fn+0x1b4/0x220 [26378.208299] i2c_smbus crct10dif_pclmul [26378.210312] ? __hrtimer_run_queues+0x1d3/0x360 [26378.214316] joydev [26378.218582] ? hrtimer_interrupt+0x104/0x240 [26378.222411] crc32_pclmul ipmi_ssif [26378.226934] ? __local_bh_enable+0x50/0x60 [26378.229032] ghash_clmulni_intel mei_me [26378.233298] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26378.236779] mei lpc_ich [26378.240871] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26378.244700] sha512_ssse3 acpi_ipmi [26378.249920] [26378.252449] mfd_core wmi [26378.257406] [26378.257407] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26378.260891] ipmi_si ipmi_devintf [26378.262995] ? queued_spin_lock_slowpath+0x16/0x2e0 [26378.265606] ipmi_msghandler ioatdma [26378.267705] do_raw_spin_lock+0xb5/0xc0 [26378.273007] acpi_pad [26378.276318] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26378.281188] acpi_power_meter [26378.284761] try_get_cap_refs+0x159/0x880 [ceph] [26378.288589] ixgbe [26378.290866] __ceph_get_caps+0x4a1/0x760 [ceph] [26378.295644] igb mdio [26378.298612] ? __file_remove_privs+0x89/0x110 [26378.303219] crc32c_intel [26378.305231] ? wake_bit_function+0x70/0x70 [26378.309755] nvme ptp [26378.312028] ? ceph_write_iter+0x260/0x770 [ceph] [26378.316376] nvme_core [26378.318994] ceph_write_iter+0x260/0x770 [ceph] [26378.323083] i2c_algo_bit pps_core [26378.325359] ? lock_is_held_type+0xaa/0x120 [26378.330050] dca fuse [26378.332410] vfs_write+0x324/0x4f0 [26378.336934] [26378.336935] irq event stamp: 132944634 [26378.340333] ksys_pwrite64+0x65/0xa0 [26378.344506] hardirqs last enabled at (132944633): [] asm_common_interrupt+0x26/0x40 [26378.346779] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26378.350175] hardirqs last disabled at (132944634): [] sysvec_apic_timer_interrupt+0xe/0x90 [26378.351668] do_syscall_64+0x38/0x80 [26378.355412] softirqs last enabled at (132944632): [] __do_softirq+0x2be/0x371 [26378.358995] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26378.368284] softirqs last disabled at (132944625): [] irq_exit_rcu+0xb0/0xd0 [26378.373412] RIP: 0033:0x7f48f0eeb93b [26378.383221] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26378.386792] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26378.395564] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26378.400609] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26378.409207] Workqueue: events delayed_work [ceph] [26378.412775] ORIG_RAX: 0000000000000012 [26378.422154] [26378.422155] RIP: 0010:lock_acquire+0x4a/0x2b0 [26378.440892] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26378.448363] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [26378.453579] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26378.458280] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [26378.462108] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26378.463599] [26378.463600] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [26378.467949] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26378.475074] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [26378.493812] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26378.500936] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [26378.506157] [26378.513276] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [26378.559705] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26378.566830] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26378.574907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26378.580643] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26378.587768] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26378.594894] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26378.602016] Call Trace: [26378.604463] [26378.606471] ? watchdog_timer_fn+0x1b4/0x220 [26378.610737] ? __hrtimer_run_queues+0x1d3/0x360 [26378.615278] ? hrtimer_interrupt+0x104/0x240 [26378.619541] ? __local_bh_enable+0x50/0x60 [26378.623633] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26378.628849] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26378.633807] [26378.635903] [26378.638002] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26378.643309] ? lock_acquire+0x4a/0x2b0 [26378.647059] ? igrab+0x43/0x50 [26378.650109] ? lock_release+0xc5/0x260 [26378.653856] _raw_spin_lock+0x30/0x40 [26378.657520] ? igrab+0x1d/0x50 [26378.660570] igrab+0x1d/0x50 [26378.663448] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26378.668599] delayed_work+0x1d8/0x280 [ceph] [26378.672879] ? process_scheduled_works+0x22a/0x510 [26378.677669] process_scheduled_works+0x2a4/0x510 [26378.682286] worker_thread+0x188/0x340 [26378.686034] ? rescuer_thread+0x310/0x310 [26378.690038] kthread+0xf5/0x130 [26378.693185] ? kthread_complete_and_exit+0x20/0x20 [26378.697967] ret_from_fork+0x31/0x40 [26378.701564] ? kthread_complete_and_exit+0x20/0x20 [26378.706349] ret_from_fork_asm+0x11/0x20 [26378.710293] [26405.920078] watchdog: BUG: soft lockup - CPU#0 stuck for 13478s! [dbench:78683] [26405.927375] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26405.932078] watchdog: BUG: soft lockup - CPU#7 stuck for 22475s! [kworker/7:4:78808] [26405.982816] irq event stamp: 42924334 [26405.982817] hardirqs last enabled at (42924333): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26405.990547] Modules linked in: [26405.994204] hardirqs last disabled at (42924334): [] sysvec_apic_timer_interrupt+0xe/0x90 [26406.004363] ceph [26406.007413] softirqs last enabled at (42924306): [] __do_softirq+0x2be/0x371 [26406.017138] libceph dns_resolver [26406.019062] softirqs last disabled at (42924301): [] irq_exit_rcu+0xb0/0xd0 [26406.027753] fscache netfs [26406.031065] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26406.039575] veth nft_chain_nat [26406.042281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26406.051224] xt_MASQUERADE nf_nat [26406.054361] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26406.061832] nf_conntrack nf_defrag_ipv6 [26406.065142] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26406.070620] nf_defrag_ipv4 nft_compat [26406.074538] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26406.093275] nf_tables nfnetlink [26406.097019] [26406.097020] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26406.102238] bridge stp [26406.105475] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26406.106969] llc xfs [26406.114094] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26406.116537] libcrc32c sunrpc [26406.123662] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26406.125846] intel_rapl_msr intel_rapl_common [26406.132970] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26406.135933] sb_edac x86_pkg_temp_thermal [26406.143059] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26406.147409] coretemp kvm_intel [26406.154532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26406.158537] kvm [26406.166614] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26406.169755] irqbypass i2c_i801 [26406.175490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26406.177326] iTCO_wdt iTCO_vendor_support [26406.184450] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26406.187605] i2c_smbus crct10dif_pclmul [26406.194728] Call Trace: [26406.194729] [26406.198733] joydev [26406.205875] ? watchdog_timer_fn+0x1b4/0x220 [26406.209704] crc32_pclmul ipmi_ssif [26406.212151] ? __hrtimer_run_queues+0x1d3/0x360 [26406.214160] ghash_clmulni_intel mei_me [26406.216261] ? hrtimer_interrupt+0x104/0x240 [26406.220522] mei [26406.224006] ? __local_bh_enable+0x50/0x60 [26406.228531] lpc_ich sha512_ssse3 [26406.232363] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26406.236625] acpi_ipmi mfd_core [26406.238478] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26406.242570] wmi ipmi_si [26406.245881] [26406.251099] ipmi_devintf [26406.254235] [26406.254236] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26406.259194] ipmi_msghandler ioatdma [26406.261729] ? queued_spin_lock_slowpath+0x16/0x2e0 [26406.263821] acpi_pad acpi_power_meter [26406.266440] do_raw_spin_lock+0xb5/0xc0 [26406.268535] ixgbe igb [26406.273841] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26406.277412] mdio crc32c_intel [26406.282282] try_get_cap_refs+0x159/0x880 [ceph] [26406.286026] nvme ptp [26406.289861] __ceph_get_caps+0x4a1/0x760 [ceph] [26406.292213] nvme_core i2c_algo_bit [26406.297003] ? __file_remove_privs+0x89/0x110 [26406.300049] pps_core dca [26406.304661] ? wake_bit_function+0x70/0x70 [26406.306929] fuse [26406.311477] ? ceph_write_iter+0x260/0x770 [ceph] [26406.314955] irq event stamp: 133165736 [26406.319306] ceph_write_iter+0x260/0x770 [ceph] [26406.321925] hardirqs last enabled at (133165735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26406.326018] ? lock_is_held_type+0xaa/0x120 [26406.327939] hardirqs last disabled at (133165736): [] sysvec_apic_timer_interrupt+0xe/0x90 [26406.332640] vfs_write+0x324/0x4f0 [26406.336379] softirqs last enabled at (133165734): [] __do_softirq+0x2be/0x371 [26406.340908] ksys_pwrite64+0x65/0xa0 [26406.351149] softirqs last disabled at (133165729): [] irq_exit_rcu+0xb0/0xd0 [26406.355325] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26406.365136] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26406.368535] do_syscall_64+0x38/0x80 [26406.377304] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26406.380876] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26406.389480] Workqueue: events delayed_work [ceph] [26406.394612] RIP: 0033:0x7f48f0eeb93b [26406.403990] [26406.407562] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26406.415030] RIP: 0010:lock_acquire+0xf0/0x2b0 [26406.420075] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26406.424773] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26406.428343] ORIG_RAX: 0000000000000012 [26406.429833] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26406.448571] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26406.452924] [26406.452924] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26406.458139] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26406.476879] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26406.480710] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26406.485926] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26406.493050] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26406.494542] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26406.501665] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26406.508789] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26406.515919] [26406.523037] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26406.568921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26406.574658] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26406.581780] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26406.588906] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26406.596029] Call Trace: [26406.598484] [26406.600494] ? watchdog_timer_fn+0x1b4/0x220 [26406.604765] ? __hrtimer_run_queues+0x1d3/0x360 [26406.609293] ? hrtimer_interrupt+0x104/0x240 [26406.613562] ? __local_bh_enable+0x50/0x60 [26406.617656] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26406.622881] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26406.627836] [26406.629934] [26406.632031] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26406.637341] ? lock_acquire+0xf0/0x2b0 [26406.641091] ? igrab+0x43/0x50 [26406.644141] ? lock_release+0xc5/0x260 [26406.647885] _raw_spin_lock+0x30/0x40 [26406.651540] ? igrab+0x1d/0x50 [26406.654592] igrab+0x1d/0x50 [26406.657487] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26406.662638] delayed_work+0x1d8/0x280 [ceph] [26406.666917] ? process_scheduled_works+0x22a/0x510 [26406.671701] process_scheduled_works+0x2a4/0x510 [26406.676314] worker_thread+0x188/0x340 [26406.680065] ? rescuer_thread+0x310/0x310 [26406.684076] kthread+0xf5/0x130 [26406.687214] ? kthread_complete_and_exit+0x20/0x20 [26406.692000] ret_from_fork+0x31/0x40 [26406.695577] ? kthread_complete_and_exit+0x20/0x20 [26406.700362] ret_from_fork_asm+0x11/0x20 [26406.704283] [26433.920565] watchdog: BUG: soft lockup - CPU#0 stuck for 13504s! [dbench:78683] [26433.927872] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26433.932565] watchdog: BUG: soft lockup - CPU#7 stuck for 22501s! [kworker/7:4:78808] [26433.983321] irq event stamp: 42982612 [26433.991052] Modules linked in: ceph libceph dns_resolver fscache netfs [26433.994710] hardirqs last enabled at (42982611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26434.001239] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [26434.011393] hardirqs last disabled at (42982612): [] sysvec_apic_timer_interrupt+0xe/0x90 [26434.017479] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [26434.027202] softirqs last enabled at (42982584): [] __do_softirq+0x2be/0x371 [26434.034595] stp llc xfs libcrc32c sunrpc [26434.043279] softirqs last disabled at (42982579): [] irq_exit_rcu+0xb0/0xd0 [26434.047291] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [26434.055804] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26434.055805] coretemp [26434.055806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26434.062667] kvm_intel [26434.071610] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26434.073880] kvm irqbypass [26434.081353] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26434.083710] i2c_i801 [26434.089187] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26434.091890] iTCO_wdt iTCO_vendor_support [26434.110630] [26434.112899] i2c_smbus [26434.118116] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26434.122120] crct10dif_pclmul joydev [26434.123611] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26434.125969] crc32_pclmul ipmi_ssif [26434.133093] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26434.136664] ghash_clmulni_intel mei_me [26434.143788] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26434.147271] mei lpc_ich [26434.154397] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26434.158242] sha512_ssse3 acpi_ipmi [26434.165367] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26434.167899] mfd_core wmi [26434.175023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26434.178508] ipmi_si ipmi_devintf [26434.186586] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26434.189202] ipmi_msghandler [26434.194940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26434.198268] ioatdma [26434.205391] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26434.208268] acpi_pad acpi_power_meter [26434.215393] Call Trace: [26434.217578] ixgbe igb [26434.224702] [26434.228444] mdio [26434.230891] ? watchdog_timer_fn+0x1b4/0x220 [26434.233248] crc32c_intel [26434.235258] ? __hrtimer_run_queues+0x1d3/0x360 [26434.237181] nvme ptp [26434.241450] ? hrtimer_interrupt+0x104/0x240 [26434.244064] nvme_core [26434.248587] ? __local_bh_enable+0x50/0x60 [26434.250856] i2c_algo_bit pps_core [26434.255126] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26434.257479] dca [26434.261571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26434.264969] fuse [26434.270184] [26434.272021] [26434.276979] [26434.278904] irq event stamp: 133386700 [26434.281003] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26434.282493] hardirqs last enabled at (133386699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26434.284592] ? queued_spin_lock_slowpath+0x16/0x2e0 [26434.288332] hardirqs last disabled at (133386700): [] sysvec_apic_timer_interrupt+0xe/0x90 [26434.293640] do_raw_spin_lock+0xb5/0xc0 [26434.303882] softirqs last enabled at (133386698): [] __do_softirq+0x2be/0x371 [26434.308754] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26434.318564] softirqs last disabled at (133386693): [] irq_exit_rcu+0xb0/0xd0 [26434.322398] try_get_cap_refs+0x159/0x880 [ceph] [26434.331164] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26434.335971] __ceph_get_caps+0x4a1/0x760 [ceph] [26434.344563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26434.349179] ? __file_remove_privs+0x89/0x110 [26434.358551] Workqueue: events delayed_work [ceph] [26434.363075] ? wake_bit_function+0x70/0x70 [26434.370546] [26434.370548] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [26434.374901] ? ceph_write_iter+0x260/0x770 [ceph] [26434.379595] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [26434.383685] ceph_write_iter+0x260/0x770 [ceph] [26434.385175] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [26434.391091] ? lock_is_held_type+0xaa/0x120 [26434.395785] [26434.395786] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [26434.414525] vfs_write+0x324/0x4f0 [26434.419046] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [26434.424267] ksys_pwrite64+0x65/0xa0 [26434.428440] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [26434.429933] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26434.437055] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26434.440454] do_syscall_64+0x38/0x80 [26434.447577] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26434.451149] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26434.458272] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26434.463405] RIP: 0033:0x7f48f0eeb93b [26434.470528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26434.474098] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26434.481222] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26434.486284] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26434.494362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26434.497932] ORIG_RAX: 0000000000000012 [26434.503668] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26434.522406] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26434.529530] Call Trace: [26434.529532] [26434.534749] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26434.541873] ? watchdog_timer_fn+0x1b4/0x220 [26434.545702] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26434.552828] ? __hrtimer_run_queues+0x1d3/0x360 [26434.559968] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26434.562417] ? hrtimer_interrupt+0x104/0x240 [26434.564423] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26434.571548] ? __local_bh_enable+0x50/0x60 [26434.575818] [26434.582938] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26434.617475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26434.622432] [26434.624529] [26434.626627] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26434.631934] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [26434.637279] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [26434.642602] delayed_work+0x1d8/0x280 [ceph] [26434.646882] ? process_scheduled_works+0x22a/0x510 [26434.651673] process_scheduled_works+0x2a4/0x510 [26434.656289] worker_thread+0x188/0x340 [26434.660038] ? rescuer_thread+0x310/0x310 [26434.664040] kthread+0xf5/0x130 [26434.667179] ? kthread_complete_and_exit+0x20/0x20 [26434.671974] ret_from_fork+0x31/0x40 [26434.675550] ? kthread_complete_and_exit+0x20/0x20 [26434.680334] ret_from_fork_asm+0x11/0x20 [26434.684264] [26461.921052] watchdog: BUG: soft lockup - CPU#0 stuck for 13530s! [dbench:78683] [26461.928351] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26461.933052] watchdog: BUG: soft lockup - CPU#7 stuck for 22527s! [kworker/7:4:78808] [26461.983792] irq event stamp: 43040736 [26461.983793] hardirqs last enabled at (43040735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26461.991523] Modules linked in: [26461.995179] hardirqs last disabled at (43040736): [] sysvec_apic_timer_interrupt+0xe/0x90 [26462.005336] ceph [26462.008387] softirqs last enabled at (43040708): [] __do_softirq+0x2be/0x371 [26462.018111] libceph [26462.020036] softirqs last disabled at (43040703): [] irq_exit_rcu+0xb0/0xd0 [26462.028726] dns_resolver fscache [26462.030914] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26462.039423] netfs veth [26462.042751] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26462.051696] nft_chain_nat xt_MASQUERADE [26462.054139] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26462.061610] nf_nat nf_conntrack [26462.065528] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26462.071005] nf_defrag_ipv6 nf_defrag_ipv4 [26462.074230] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26462.092967] nft_compat nf_tables [26462.097058] [26462.097059] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26462.102276] nfnetlink [26462.105587] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26462.107078] bridge stp [26462.114202] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26462.116557] llc xfs [26462.123682] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26462.126125] libcrc32c sunrpc [26462.133252] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26462.135448] intel_rapl_msr [26462.142577] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26462.145541] intel_rapl_common sb_edac [26462.152664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26462.155456] x86_pkg_temp_thermal [26462.163533] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26462.167277] coretemp kvm_intel [26462.173014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26462.176324] kvm irqbypass [26462.183449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26462.186588] i2c_i801 iTCO_wdt [26462.193727] Call Trace: [26462.193728] [26462.196448] iTCO_vendor_support [26462.203575] ? watchdog_timer_fn+0x1b4/0x220 [26462.206625] i2c_smbus [26462.209068] ? __hrtimer_run_queues+0x1d3/0x360 [26462.211080] crct10dif_pclmul joydev [26462.214305] ? hrtimer_interrupt+0x104/0x240 [26462.218566] crc32_pclmul ipmi_ssif [26462.220925] ? __local_bh_enable+0x50/0x60 [26462.225450] ghash_clmulni_intel mei_me [26462.229022] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26462.233283] mei [26462.236767] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26462.240858] lpc_ich sha512_ssse3 [26462.244689] [26462.249905] acpi_ipmi mfd_core [26462.251744] [26462.251745] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26462.256700] wmi ipmi_si [26462.260015] ? queued_spin_lock_slowpath+0x18/0x2e0 [26462.262110] ipmi_devintf ipmi_msghandler [26462.265248] do_raw_spin_lock+0xb5/0xc0 [26462.267344] ioatdma acpi_pad [26462.272648] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26462.275178] acpi_power_meter ixgbe [26462.280052] try_get_cap_refs+0x159/0x880 [ceph] [26462.284053] igb mdio [26462.287888] __ceph_get_caps+0x4a1/0x760 [ceph] [26462.290849] crc32c_intel [26462.295638] ? __file_remove_privs+0x89/0x110 [26462.299117] nvme ptp [26462.303733] ? wake_bit_function+0x70/0x70 [26462.306007] nvme_core i2c_algo_bit [26462.310533] ? ceph_write_iter+0x260/0x770 [ceph] [26462.313150] pps_core [26462.317498] ceph_write_iter+0x260/0x770 [ceph] [26462.319769] dca fuse [26462.323864] ? lock_is_held_type+0xaa/0x120 [26462.327343] [26462.327344] irq event stamp: 133607886 [26462.332043] vfs_write+0x324/0x4f0 [26462.334312] hardirqs last enabled at (133607885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26462.338841] ksys_pwrite64+0x65/0xa0 [26462.341108] hardirqs last disabled at (133607886): [] sysvec_apic_timer_interrupt+0xe/0x90 [26462.345285] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26462.346775] softirqs last enabled at (133607884): [] __do_softirq+0x2be/0x371 [26462.350522] do_syscall_64+0x38/0x80 [26462.353918] softirqs last disabled at (133607879): [] irq_exit_rcu+0xb0/0xd0 [26462.364161] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26462.367746] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26462.377559] RIP: 0033:0x7f48f0eeb93b [26462.382691] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26462.391461] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26462.395031] Workqueue: events delayed_work [ceph] [26462.403630] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26462.408674] [26462.408675] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [26462.418052] ORIG_RAX: 0000000000000012 [26462.421621] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [26462.429092] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26462.447831] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [26462.452528] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26462.457751] [26462.457752] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [26462.459246] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26462.463857] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [26462.467686] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26462.486424] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [26462.493547] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26462.498765] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26462.505896] [26462.507381] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26462.566574] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26462.574652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26462.580390] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26462.587514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26462.594638] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26462.601763] Call Trace: [26462.604207] [26462.606217] ? watchdog_timer_fn+0x1b4/0x220 [26462.610483] ? __hrtimer_run_queues+0x1d3/0x360 [26462.615008] ? hrtimer_interrupt+0x104/0x240 [26462.619269] ? __local_bh_enable+0x50/0x60 [26462.623363] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26462.628587] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26462.633543] [26462.635641] [26462.637754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26462.643063] ? do_raw_spin_lock+0x42/0xc0 [26462.647075] igrab+0x1d/0x50 [26462.649951] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26462.655100] delayed_work+0x1d8/0x280 [ceph] [26462.659382] ? process_scheduled_works+0x22a/0x510 [26462.664172] process_scheduled_works+0x2a4/0x510 [26462.668788] worker_thread+0x188/0x340 [26462.672537] ? rescuer_thread+0x310/0x310 [26462.676550] kthread+0xf5/0x130 [26462.679686] ? kthread_complete_and_exit+0x20/0x20 [26462.684472] ret_from_fork+0x31/0x40 [26462.688049] ? kthread_complete_and_exit+0x20/0x20 [26462.692834] ret_from_fork_asm+0x11/0x20 [26462.696756] [26464.188457] rcu: INFO: rcu_preempt self-detected stall on CPU [26464.194215] rcu: 0-....: (14201701 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7098601 [26464.204631] rcu: (t=14527321 jiffies g=1309609 q=441246 ncpus=8) [26464.210717] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26464.219660] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26464.227130] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26464.232608] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26464.251344] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26464.256563] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26464.263686] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26464.270812] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26464.277936] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26464.285060] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26464.292185] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26464.300261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26464.306000] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26464.313123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26464.320247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26464.327370] Call Trace: [26464.329816] [26464.331826] ? rcu_dump_cpu_stacks+0xd1/0x180 [26464.336177] ? rcu_sched_clock_irq+0x633/0x1390 [26464.340702] ? lock_is_held_type+0xaa/0x120 [26464.344888] ? lock_is_held_type+0xaa/0x120 [26464.349066] ? tick_sched_do_timer+0xa0/0xa0 [26464.353337] ? rcu_read_lock_sched_held+0x56/0x90 [26464.358035] ? tick_sched_do_timer+0xa0/0xa0 [26464.362298] ? update_process_times+0x5f/0x90 [26464.366648] ? tick_sched_handle.isra.16+0x1e/0x50 [26464.371434] ? tick_sched_timer+0x6f/0x80 [26464.375438] ? __hrtimer_run_queues+0x1d3/0x360 [26464.379965] ? hrtimer_interrupt+0x104/0x240 [26464.384235] ? __local_bh_enable+0x50/0x60 [26464.388327] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26464.393552] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26464.398509] [26464.400605] [26464.402703] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26464.408010] ? queued_spin_lock_slowpath+0x18/0x2e0 [26464.412888] do_raw_spin_lock+0xb5/0xc0 [26464.416719] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26464.421521] try_get_cap_refs+0x159/0x880 [ceph] [26464.426153] __ceph_get_caps+0x4a1/0x760 [ceph] [26464.430693] ? __file_remove_privs+0x89/0x110 [26464.435049] ? wake_bit_function+0x70/0x70 [26464.439141] ? ceph_write_iter+0x260/0x770 [ceph] [26464.443854] ceph_write_iter+0x260/0x770 [ceph] [26464.448390] ? lock_is_held_type+0xaa/0x120 [26464.452574] vfs_write+0x324/0x4f0 [26464.455982] ksys_pwrite64+0x65/0xa0 [26464.459558] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26464.464690] do_syscall_64+0x38/0x80 [26464.468269] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26464.473313] RIP: 0033:0x7f48f0eeb93b [26464.476882] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26464.495621] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [26464.503179] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26464.510303] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26464.517427] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26464.524550] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26464.531674] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26464.538804] [26464.540991] Sending NMI from CPU 0 to CPUs 7: [26464.545343] NMI backtrace for cpu 7 [26464.545345] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26464.545346] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26464.545347] Workqueue: events delayed_work [ceph] [26464.545362] RIP: 0010:ceph_check_delayed_caps+0x85/0x270 [ceph] [26464.545375] Code: 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 <78> 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 [26464.545377] RSP: 0018:ffffc90002117db8 EFLAGS: 00000216 [26464.545378] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 00000001018f36d4 [26464.545379] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [26464.545379] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [26464.545380] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26464.545381] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26464.545382] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26464.545383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26464.545383] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26464.545384] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26464.545385] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26464.545385] Call Trace: [26464.545386] [26464.545387] ? nmi_cpu_backtrace+0xad/0x120 [26464.545390] ? nmi_cpu_backtrace_handler+0x11/0x20 [26464.545392] ? nmi_handle+0xd4/0x260 [26464.545396] ? default_do_nmi+0x49/0xf0 [26464.545398] ? exc_nmi+0x11e/0x150 [26464.545400] ? end_repeat_nmi+0x16/0x67 [26464.545404] ? ceph_check_delayed_caps+0x85/0x270 [ceph] [26464.545415] ? ceph_check_delayed_caps+0x85/0x270 [ceph] [26464.545426] ? ceph_check_delayed_caps+0x85/0x270 [ceph] [26464.545437] [26464.545437] [26464.545438] delayed_work+0x1d8/0x280 [ceph] [26464.545452] ? process_scheduled_works+0x22a/0x510 [26464.545454] process_scheduled_works+0x2a4/0x510 [26464.545458] worker_thread+0x188/0x340 [26464.545461] ? rescuer_thread+0x310/0x310 [26464.545464] kthread+0xf5/0x130 [26464.545467] ? kthread_complete_and_exit+0x20/0x20 [26464.545469] ret_from_fork+0x31/0x40 [26464.545472] ? kthread_complete_and_exit+0x20/0x20 [26464.545474] ret_from_fork_asm+0x11/0x20 [26464.545478] [26489.921539] watchdog: BUG: soft lockup - CPU#0 stuck for 13556s! [dbench:78683] [26489.928840] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26489.933538] watchdog: BUG: soft lockup - CPU#7 stuck for 22553s! [kworker/7:4:78808] [26489.984288] irq event stamp: 43098248 [26489.992019] Modules linked in: ceph libceph dns_resolver fscache [26489.995676] hardirqs last enabled at (43098247): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26490.001682] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [26490.011840] hardirqs last disabled at (43098248): [] sysvec_apic_timer_interrupt+0xe/0x90 [26490.017315] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [26490.027040] softirqs last enabled at (43098226): [] __do_softirq+0x2be/0x371 [26490.034945] bridge stp llc xfs libcrc32c [26490.043630] softirqs last disabled at (43098221): [] irq_exit_rcu+0xb0/0xd0 [26490.047633] sunrpc intel_rapl_msr intel_rapl_common sb_edac [26490.056145] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26490.061794] x86_pkg_temp_thermal [26490.070739] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26490.074048] coretemp kvm_intel kvm irqbypass [26490.081521] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26490.085871] i2c_i801 iTCO_wdt [26490.091349] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26490.091350] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [26490.113136] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26490.119308] crc32_pclmul ipmi_ssif [26490.124525] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26490.124526] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26490.128009] ghash_clmulni_intel [26490.135134] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26490.142258] mei_me [26490.145481] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26490.152606] mei lpc_ich [26490.154703] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26490.161827] sha512_ssse3 acpi_ipmi [26490.164359] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26490.171481] mfd_core wmi [26490.174966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26490.183044] ipmi_si [26490.185660] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26490.191398] ipmi_devintf [26490.193584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26490.200706] ipmi_msghandler [26490.203325] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26490.210448] ioatdma [26490.213324] Call Trace: [26490.213326] [26490.220449] acpi_pad acpi_power_meter [26490.222635] ? watchdog_timer_fn+0x1b4/0x220 [26490.225078] ixgbe igb [26490.227090] ? __hrtimer_run_queues+0x1d3/0x360 [26490.230833] mdio crc32c_intel [26490.235100] ? hrtimer_interrupt+0x104/0x240 [26490.237454] nvme ptp [26490.241978] ? __local_bh_enable+0x50/0x60 [26490.245031] nvme_core [26490.249297] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26490.251564] i2c_algo_bit pps_core [26490.255655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26490.258011] dca fuse [26490.263228] [26490.266626] [26490.266627] irq event stamp: 133827142 [26490.271584] [26490.273854] hardirqs last enabled at (133827141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26490.275952] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26490.277443] hardirqs last disabled at (133827142): [] sysvec_apic_timer_interrupt+0xe/0x90 [26490.281192] ? queued_spin_lock_slowpath+0x16/0x2e0 [26490.283284] softirqs last enabled at (133827140): [] __do_softirq+0x2be/0x371 [26490.293531] do_raw_spin_lock+0xb5/0xc0 [26490.298833] softirqs last disabled at (133827135): [] irq_exit_rcu+0xb0/0xd0 [26490.308645] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26490.313515] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26490.322287] try_get_cap_refs+0x159/0x880 [ceph] [26490.326117] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26490.334718] __ceph_get_caps+0x4a1/0x760 [ceph] [26490.339497] Workqueue: events delayed_work [ceph] [26490.348879] ? __file_remove_privs+0x89/0x110 [26490.353486] [26490.353487] RIP: 0010:lock_acquire+0xf0/0x2b0 [26490.360956] ? wake_bit_function+0x70/0x70 [26490.365482] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26490.370180] ? ceph_write_iter+0x260/0x770 [ceph] [26490.374528] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26490.376020] ceph_write_iter+0x260/0x770 [ceph] [26490.380370] [26490.380371] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26490.384464] ? lock_is_held_type+0xaa/0x120 [26490.403199] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26490.407899] vfs_write+0x324/0x4f0 [26490.413113] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26490.417642] ksys_pwrite64+0x65/0xa0 [26490.419128] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26490.426254] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26490.430431] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26490.437554] do_syscall_64+0x38/0x80 [26490.440952] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26490.448075] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26490.451645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26490.458772] RIP: 0033:0x7f48f0eeb93b [26490.463902] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26490.471026] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26490.474596] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26490.482672] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26490.487717] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26490.493455] ORIG_RAX: 0000000000000012 [26490.497026] Call Trace: [26490.497028] [26490.504151] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26490.522888] ? watchdog_timer_fn+0x1b4/0x220 [26490.530011] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26490.535229] ? __hrtimer_run_queues+0x1d3/0x360 [26490.542353] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26490.546188] ? hrtimer_interrupt+0x104/0x240 [26490.548627] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26490.550638] ? __local_bh_enable+0x50/0x60 [26490.557763] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26490.562029] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26490.569158] [26490.573677] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26490.615761] [26490.617857] [26490.619956] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26490.625266] ? lock_acquire+0xf0/0x2b0 [26490.629015] ? igrab+0x43/0x50 [26490.632072] ? lock_release+0xc5/0x260 [26490.635819] _raw_spin_lock+0x30/0x40 [26490.639484] ? igrab+0x1d/0x50 [26490.642534] igrab+0x1d/0x50 [26490.645414] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26490.650561] delayed_work+0x1d8/0x280 [ceph] [26490.654842] ? process_scheduled_works+0x22a/0x510 [26490.659632] process_scheduled_works+0x2a4/0x510 [26490.664248] worker_thread+0x188/0x340 [26490.667997] ? rescuer_thread+0x310/0x310 [26490.672001] kthread+0xf5/0x130 [26490.675139] ? kthread_complete_and_exit+0x20/0x20 [26490.679941] ret_from_fork+0x31/0x40 [26490.683519] ? kthread_complete_and_exit+0x20/0x20 [26490.688303] ret_from_fork_asm+0x11/0x20 [26490.692233] [26517.922024] watchdog: BUG: soft lockup - CPU#0 stuck for 13582s! [dbench:78683] [26517.929324] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26517.934024] watchdog: BUG: soft lockup - CPU#7 stuck for 22579s! [kworker/7:4:78808] [26517.984767] irq event stamp: 43156388 [26517.992496] Modules linked in: ceph libceph dns_resolver [26517.996154] hardirqs last enabled at (43156387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26517.996158] fscache netfs veth nft_chain_nat [26518.011617] hardirqs last disabled at (43156388): [] sysvec_apic_timer_interrupt+0xe/0x90 [26518.015965] xt_MASQUERADE nf_nat [26518.025707] softirqs last enabled at (43156370): [] __do_softirq+0x2be/0x371 [26518.025709] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [26518.037708] softirqs last disabled at (43156365): [] irq_exit_rcu+0xb0/0xd0 [26518.045616] bridge stp llc xfs [26518.054128] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26518.057263] libcrc32c sunrpc [26518.066208] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26518.069172] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [26518.076642] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26518.083506] coretemp kvm_intel [26518.088984] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26518.088986] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26518.092121] kvm irqbypass [26518.110859] [26518.110861] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26518.116076] i2c_i801 iTCO_wdt [26518.118781] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26518.120272] iTCO_vendor_support i2c_smbus [26518.127395] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26518.130465] crct10dif_pclmul joydev [26518.137588] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26518.141679] crc32_pclmul ipmi_ssif [26518.148821] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26518.152392] ghash_clmulni_intel mei_me [26518.159517] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26518.163000] mei lpc_ich [26518.170125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26518.173954] sha512_ssse3 acpi_ipmi [26518.182031] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26518.184562] mfd_core wmi [26518.190300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26518.193784] ipmi_si [26518.200909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26518.203526] ipmi_devintf ipmi_msghandler [26518.210650] Call Trace: [26518.212834] ioatdma acpi_pad [26518.219959] [26518.223961] acpi_power_meter ixgbe [26518.226422] ? watchdog_timer_fn+0x1b4/0x220 [26518.229387] igb mdio [26518.231418] ? __hrtimer_run_queues+0x1d3/0x360 [26518.234898] crc32c_intel nvme [26518.239166] ? hrtimer_interrupt+0x104/0x240 [26518.241434] ptp nvme_core [26518.245958] ? __local_bh_enable+0x50/0x60 [26518.249010] i2c_algo_bit pps_core [26518.253275] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26518.255976] dca fuse [26518.260069] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26518.263465] [26518.263466] irq event stamp: 134048242 [26518.268684] [26518.270954] hardirqs last enabled at (134048241): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26518.275911] [26518.277402] hardirqs last disabled at (134048242): [] sysvec_apic_timer_interrupt+0xe/0x90 [26518.281164] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26518.283259] softirqs last enabled at (134048240): [] __do_softirq+0x2be/0x371 [26518.293509] ? queued_spin_lock_slowpath+0x16/0x2e0 [26518.295601] softirqs last disabled at (134048235): [] irq_exit_rcu+0xb0/0xd0 [26518.305427] do_raw_spin_lock+0xb5/0xc0 [26518.310725] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26518.319499] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26518.324367] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26518.332966] try_get_cap_refs+0x159/0x880 [ceph] [26518.336795] Workqueue: events delayed_work [ceph] [26518.346178] __ceph_get_caps+0x4a1/0x760 [ceph] [26518.350958] [26518.358432] ? __file_remove_privs+0x89/0x110 [26518.363039] RIP: 0010:lock_release+0x14e/0x260 [26518.367736] ? wake_bit_function+0x70/0x70 [26518.372260] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [26518.373754] ? ceph_write_iter+0x260/0x770 [ceph] [26518.378101] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [26518.382539] ceph_write_iter+0x260/0x770 [ceph] [26518.386628] [26518.405371] ? lock_is_held_type+0xaa/0x120 [26518.410064] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [26518.415284] vfs_write+0x324/0x4f0 [26518.419807] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [26518.421300] ksys_pwrite64+0x65/0xa0 [26518.425475] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26518.432599] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26518.435995] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26518.443121] do_syscall_64+0x38/0x80 [26518.446707] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [26518.453834] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26518.458964] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26518.466088] RIP: 0033:0x7f48f0eeb93b [26518.469658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26518.476783] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26518.481826] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26518.489904] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26518.493473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26518.499212] ORIG_RAX: 0000000000000012 [26518.517950] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26518.525074] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26518.530291] Call Trace: [26518.530292] [26518.537430] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26518.541263] ? watchdog_timer_fn+0x1b4/0x220 [26518.548387] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26518.555513] ? __hrtimer_run_queues+0x1d3/0x360 [26518.557955] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26518.559971] ? hrtimer_interrupt+0x104/0x240 [26518.567089] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26518.571354] ? __local_bh_enable+0x50/0x60 [26518.578483] [26518.583005] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26518.613008] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26518.617966] [26518.620061] [26518.622159] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26518.627464] ? igrab+0x43/0x50 [26518.630516] ? lock_release+0x14e/0x260 [26518.634348] ? lock_release+0xc5/0x260 [26518.638091] _raw_spin_unlock+0x1b/0x40 [26518.641920] igrab+0x43/0x50 [26518.644798] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26518.649949] delayed_work+0x1d8/0x280 [ceph] [26518.654231] ? process_scheduled_works+0x22a/0x510 [26518.659019] process_scheduled_works+0x2a4/0x510 [26518.663636] worker_thread+0x188/0x340 [26518.667383] ? rescuer_thread+0x310/0x310 [26518.671387] kthread+0xf5/0x130 [26518.674526] ? kthread_complete_and_exit+0x20/0x20 [26518.679309] ret_from_fork+0x31/0x40 [26518.682880] ? kthread_complete_and_exit+0x20/0x20 [26518.687663] ret_from_fork_asm+0x11/0x20 [26518.691587] [26545.922510] watchdog: BUG: soft lockup - CPU#0 stuck for 13608s! [dbench:78683] [26545.929813] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26545.934510] watchdog: BUG: soft lockup - CPU#7 stuck for 22605s! [kworker/7:4:78808] [26545.985253] irq event stamp: 43214474 [26545.985254] hardirqs last enabled at (43214473): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26545.992983] Modules linked in: [26545.996641] hardirqs last disabled at (43214474): [] sysvec_apic_timer_interrupt+0xe/0x90 [26546.006798] ceph [26546.009849] softirqs last enabled at (43214456): [] __do_softirq+0x2be/0x371 [26546.019575] libceph [26546.021498] softirqs last disabled at (43214451): [] irq_exit_rcu+0xb0/0xd0 [26546.030184] dns_resolver fscache [26546.032366] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26546.040878] netfs veth [26546.044204] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26546.053150] nft_chain_nat xt_MASQUERADE [26546.055594] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26546.063064] nf_nat [26546.066980] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26546.072459] nf_conntrack nf_defrag_ipv6 [26546.074557] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26546.093295] nf_defrag_ipv4 nft_compat [26546.097211] [26546.097212] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26546.102429] nf_tables nfnetlink [26546.106188] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26546.107681] bridge stp [26546.114806] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26546.118030] llc xfs [26546.125154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26546.127599] libcrc32c sunrpc [26546.134722] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26546.136905] intel_rapl_msr [26546.144030] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26546.146994] intel_rapl_common sb_edac [26546.154117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26546.156908] x86_pkg_temp_thermal coretemp [26546.164986] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26546.168730] kvm_intel kvm [26546.174469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26546.178558] irqbypass i2c_i801 [26546.185683] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26546.188387] iTCO_wdt iTCO_vendor_support [26546.195512] Call Trace: [26546.195513] [26546.198648] i2c_smbus [26546.205773] ? watchdog_timer_fn+0x1b4/0x220 [26546.209777] crct10dif_pclmul [26546.212220] ? __hrtimer_run_queues+0x1d3/0x360 [26546.214230] joydev crc32_pclmul [26546.216591] ? hrtimer_interrupt+0x104/0x240 [26546.220852] ipmi_ssif ghash_clmulni_intel [26546.223816] ? __local_bh_enable+0x50/0x60 [26546.228341] mei_me [26546.231567] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26546.235828] mei lpc_ich [26546.239920] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26546.244010] sha512_ssse3 acpi_ipmi [26546.246109] [26546.251325] mfd_core wmi [26546.253856] [26546.258814] ipmi_si [26546.262297] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26546.264394] ipmi_devintf [26546.267015] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26546.269110] ipmi_msghandler [26546.271295] do_raw_spin_lock+0xb5/0xc0 [26546.276597] ioatdma acpi_pad [26546.279215] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26546.284085] acpi_power_meter ixgbe [26546.286964] try_get_cap_refs+0x159/0x880 [ceph] [26546.290794] igb mdio [26546.293763] __ceph_get_caps+0x4a1/0x760 [ceph] [26546.298542] crc32c_intel [26546.302029] ? __file_remove_privs+0x89/0x110 [26546.306635] nvme ptp [26546.308910] ? wake_bit_function+0x70/0x70 [26546.313432] nvme_core i2c_algo_bit [26546.316051] ? ceph_write_iter+0x260/0x770 [ceph] [26546.320398] pps_core dca [26546.322671] ceph_write_iter+0x260/0x770 [ceph] [26546.326762] fuse [26546.330249] ? lock_is_held_type+0xaa/0x120 [26546.334943] [26546.334944] irq event stamp: 134269350 [26546.337563] vfs_write+0x324/0x4f0 [26546.342084] hardirqs last enabled at (134269349): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26546.344011] ksys_pwrite64+0x65/0xa0 [26546.348199] hardirqs last disabled at (134269350): [] sysvec_apic_timer_interrupt+0xe/0x90 [26546.349692] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26546.353438] softirqs last enabled at (134269348): [] __do_softirq+0x2be/0x371 [26546.356838] do_syscall_64+0x38/0x80 [26546.367079] softirqs last disabled at (134269343): [] irq_exit_rcu+0xb0/0xd0 [26546.370650] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26546.380461] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26546.385591] RIP: 0033:0x7f48f0eeb93b [26546.394361] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26546.397934] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26546.406531] Workqueue: events delayed_work [ceph] [26546.411576] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26546.420953] [26546.424524] ORIG_RAX: 0000000000000012 [26546.424525] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26546.431995] RIP: 0010:lock_acquire+0xf0/0x2b0 [26546.450732] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26546.455429] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26546.460645] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26546.462136] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26546.465968] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26546.473093] [26546.473094] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26546.477442] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26546.484567] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26546.503310] [26546.510428] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26546.554941] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26546.562066] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26546.569202] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26546.577285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26546.583023] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26546.590145] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26546.597270] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26546.604395] Call Trace: [26546.606839] [26546.608851] ? watchdog_timer_fn+0x1b4/0x220 [26546.613115] ? __hrtimer_run_queues+0x1d3/0x360 [26546.617640] ? hrtimer_interrupt+0x104/0x240 [26546.621917] ? __local_bh_enable+0x50/0x60 [26546.626012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26546.631237] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26546.636196] [26546.638290] [26546.640388] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26546.645698] ? lock_acquire+0xf0/0x2b0 [26546.649446] ? igrab+0x43/0x50 [26546.652497] ? lock_release+0xc5/0x260 [26546.656242] _raw_spin_lock+0x30/0x40 [26546.659917] ? igrab+0x1d/0x50 [26546.662966] igrab+0x1d/0x50 [26546.665844] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26546.670993] delayed_work+0x1d8/0x280 [ceph] [26546.675275] ? process_scheduled_works+0x22a/0x510 [26546.680064] process_scheduled_works+0x2a4/0x510 [26546.684679] worker_thread+0x188/0x340 [26546.688429] ? rescuer_thread+0x310/0x310 [26546.692432] kthread+0xf5/0x130 [26546.695570] ? kthread_complete_and_exit+0x20/0x20 [26546.700354] ret_from_fork+0x31/0x40 [26546.703925] ? kthread_complete_and_exit+0x20/0x20 [26546.708709] ret_from_fork_asm+0x11/0x20 [26546.712632] [26573.922997] watchdog: BUG: soft lockup - CPU#0 stuck for 13634s! [dbench:78683] [26573.930301] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26573.934997] watchdog: BUG: soft lockup - CPU#7 stuck for 22631s! [kworker/7:4:78808] [26573.985750] irq event stamp: 43272768 [26573.985751] hardirqs last enabled at (43272767): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26573.993481] Modules linked in: [26573.997137] hardirqs last disabled at (43272768): [] sysvec_apic_timer_interrupt+0xe/0x90 [26574.007296] ceph [26574.010345] softirqs last enabled at (43272750): [] __do_softirq+0x2be/0x371 [26574.020071] libceph [26574.021995] softirqs last disabled at (43272745): [] irq_exit_rcu+0xb0/0xd0 [26574.030678] dns_resolver fscache [26574.032863] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26574.041391] netfs veth [26574.044702] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26574.053645] nft_chain_nat xt_MASQUERADE [26574.056091] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26574.063561] nf_nat nf_conntrack [26574.067478] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26574.072955] nf_defrag_ipv6 [26574.076179] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26574.094918] nf_defrag_ipv4 nft_compat [26574.097707] [26574.102926] nf_tables [26574.106674] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26574.108168] nfnetlink bridge [26574.110525] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26574.117649] stp llc [26574.120632] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26574.127757] xfs libcrc32c [26574.129938] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26574.137064] sunrpc intel_rapl_msr [26574.139768] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26574.146892] intel_rapl_common [26574.150289] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26574.157413] sb_edac [26574.160464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26574.168541] x86_pkg_temp_thermal [26574.170726] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26574.176463] coretemp kvm_intel [26574.179773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26574.186897] kvm irqbypass [26574.190035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26574.197159] i2c_i801 [26574.199863] Call Trace: [26574.199865] [26574.206987] iTCO_wdt iTCO_vendor_support [26574.209261] ? watchdog_timer_fn+0x1b4/0x220 [26574.211702] i2c_smbus [26574.213715] ? __hrtimer_run_queues+0x1d3/0x360 [26574.217716] crct10dif_pclmul joydev [26574.221986] ? hrtimer_interrupt+0x104/0x240 [26574.224338] crc32_pclmul ipmi_ssif [26574.228864] ? __local_bh_enable+0x50/0x60 [26574.232433] ghash_clmulni_intel mei_me [26574.236699] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26574.240180] mei lpc_ich [26574.244274] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26574.248103] sha512_ssse3 acpi_ipmi [26574.253321] [26574.255852] mfd_core wmi [26574.260809] [26574.260810] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26574.264293] ipmi_si [26574.266401] ? queued_spin_lock_slowpath+0x16/0x2e0 [26574.269017] ipmi_devintf ipmi_msghandler [26574.271116] do_raw_spin_lock+0xb5/0xc0 [26574.276417] ioatdma acpi_pad [26574.278604] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26574.283473] acpi_power_meter [26574.287478] try_get_cap_refs+0x159/0x880 [ceph] [26574.291306] ixgbe igb [26574.294276] __ceph_get_caps+0x4a1/0x760 [ceph] [26574.299055] mdio crc32c_intel [26574.302025] ? __file_remove_privs+0x89/0x110 [26574.306630] nvme ptp [26574.308987] ? wake_bit_function+0x70/0x70 [26574.313512] nvme_core [26574.316566] ? ceph_write_iter+0x260/0x770 [ceph] [26574.320913] i2c_algo_bit [26574.323183] ceph_write_iter+0x260/0x770 [ceph] [26574.327275] pps_core [26574.329636] ? lock_is_held_type+0xaa/0x120 [26574.334330] dca fuse [26574.336949] vfs_write+0x324/0x4f0 [26574.341471] [26574.341472] irq event stamp: 134490196 [26574.343746] ksys_pwrite64+0x65/0xa0 [26574.347918] hardirqs last enabled at (134490195): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26574.350190] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26574.353588] hardirqs last disabled at (134490196): [] sysvec_apic_timer_interrupt+0xe/0x90 [26574.355079] do_syscall_64+0x38/0x80 [26574.358822] softirqs last enabled at (134490194): [] __do_softirq+0x2be/0x371 [26574.362398] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26574.372646] softirqs last disabled at (134490189): [] irq_exit_rcu+0xb0/0xd0 [26574.377776] RIP: 0033:0x7f48f0eeb93b [26574.387586] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26574.391157] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26574.399929] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26574.404973] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26574.413572] Workqueue: events delayed_work [ceph] [26574.417140] ORIG_RAX: 0000000000000012 [26574.426519] [26574.426520] RIP: 0010:lock_acquire+0xf0/0x2b0 [26574.445257] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26574.452728] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26574.457945] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26574.462642] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26574.466472] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26574.467964] [26574.467964] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26574.472315] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26574.479438] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26574.498176] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26574.505299] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26574.510523] [26574.517641] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26574.564078] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26574.571202] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26574.579280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26574.585016] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26574.592143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26574.599265] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26574.606400] Call Trace: [26574.608843] [26574.610855] ? watchdog_timer_fn+0x1b4/0x220 [26574.615118] ? __hrtimer_run_queues+0x1d3/0x360 [26574.619645] ? hrtimer_interrupt+0x104/0x240 [26574.623914] ? __local_bh_enable+0x50/0x60 [26574.628008] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26574.633231] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26574.638188] [26574.640285] [26574.642400] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26574.647710] ? lock_acquire+0xf0/0x2b0 [26574.651460] ? igrab+0x43/0x50 [26574.654516] ? lock_release+0xc5/0x260 [26574.658263] _raw_spin_lock+0x30/0x40 [26574.661928] ? igrab+0x1d/0x50 [26574.664978] igrab+0x1d/0x50 [26574.667857] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26574.673006] delayed_work+0x1d8/0x280 [ceph] [26574.677288] ? process_scheduled_works+0x22a/0x510 [26574.682077] process_scheduled_works+0x2a4/0x510 [26574.686692] worker_thread+0x188/0x340 [26574.690443] ? rescuer_thread+0x310/0x310 [26574.694445] kthread+0xf5/0x130 [26574.697584] ? kthread_complete_and_exit+0x20/0x20 [26574.702368] ret_from_fork+0x31/0x40 [26574.705947] ? kthread_complete_and_exit+0x20/0x20 [26574.710730] ret_from_fork_asm+0x11/0x20 [26574.714655] [26601.923482] watchdog: BUG: soft lockup - CPU#0 stuck for 13660s! [dbench:78683] [26601.930779] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26601.935482] watchdog: BUG: soft lockup - CPU#7 stuck for 22657s! [kworker/7:4:78808] [26601.986221] irq event stamp: 43330996 [26601.986222] hardirqs last enabled at (43330995): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26601.993951] Modules linked in: [26601.997609] hardirqs last disabled at (43330996): [] sysvec_apic_timer_interrupt+0xe/0x90 [26602.007766] ceph [26602.010817] softirqs last enabled at (43330978): [] __do_softirq+0x2be/0x371 [26602.020540] libceph [26602.022465] softirqs last disabled at (43330973): [] irq_exit_rcu+0xb0/0xd0 [26602.031165] dns_resolver fscache [26602.033349] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26602.041877] netfs veth [26602.045188] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26602.054133] nft_chain_nat xt_MASQUERADE [26602.056595] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26602.064065] nf_nat nf_conntrack [26602.067983] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26602.073460] nf_defrag_ipv6 nf_defrag_ipv4 [26602.076684] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26602.095421] nft_compat nf_tables [26602.099513] [26602.099514] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26602.104730] nfnetlink bridge [26602.108039] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26602.109531] stp llc [26602.116656] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26602.119619] xfs libcrc32c [26602.126744] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26602.128927] sunrpc [26602.136053] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26602.138755] intel_rapl_msr intel_rapl_common [26602.145879] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26602.147976] sb_edac [26602.155100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26602.159451] x86_pkg_temp_thermal [26602.167529] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26602.169715] coretemp kvm_intel [26602.175452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26602.178761] kvm irqbypass [26602.185886] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26602.189024] i2c_i801 [26602.196159] Call Trace: [26602.196160] [26602.198878] iTCO_wdt [26602.206002] ? watchdog_timer_fn+0x1b4/0x220 [26602.208272] iTCO_vendor_support i2c_smbus [26602.210717] ? __hrtimer_run_queues+0x1d3/0x360 [26602.212727] crct10dif_pclmul [26602.215002] ? hrtimer_interrupt+0x104/0x240 [26602.219262] joydev crc32_pclmul [26602.223353] ? __local_bh_enable+0x50/0x60 [26602.227881] ipmi_ssif [26602.230843] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26602.235104] ghash_clmulni_intel mei_me [26602.238331] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26602.242420] mei [26602.244777] [26602.249994] lpc_ich sha512_ssse3 [26602.253824] [26602.253826] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26602.258782] acpi_ipmi mfd_core [26602.260625] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26602.262717] wmi [26602.266031] do_raw_spin_lock+0xb5/0xc0 [26602.268125] ipmi_si ipmi_devintf [26602.273431] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26602.276568] ipmi_msghandler ioatdma [26602.281439] try_get_cap_refs+0x159/0x880 [ceph] [26602.283274] acpi_pad acpi_power_meter [26602.287111] __ceph_get_caps+0x4a1/0x760 [ceph] [26602.290416] ixgbe igb [26602.295205] ? __file_remove_privs+0x89/0x110 [26602.298771] mdio crc32c_intel [26602.303383] ? wake_bit_function+0x70/0x70 [26602.307127] nvme [26602.311653] ? ceph_write_iter+0x260/0x770 [ceph] [26602.314009] ptp nvme_core [26602.318359] ceph_write_iter+0x260/0x770 [ceph] [26602.321410] i2c_algo_bit pps_core [26602.325503] ? lock_is_held_type+0xaa/0x120 [26602.327425] dca fuse [26602.332124] vfs_write+0x324/0x4f0 [26602.334825] [26602.339353] ksys_pwrite64+0x65/0xa0 [26602.342746] irq event stamp: 134711030 [26602.346923] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26602.349194] hardirqs last enabled at (134711029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26602.352594] do_syscall_64+0x38/0x80 [26602.354083] hardirqs last disabled at (134711030): [] sysvec_apic_timer_interrupt+0xe/0x90 [26602.357655] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26602.361397] softirqs last enabled at (134711028): [] __do_softirq+0x2be/0x371 [26602.366530] RIP: 0033:0x7f48f0eeb93b [26602.376774] softirqs last disabled at (134711023): [] irq_exit_rcu+0xb0/0xd0 [26602.380344] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26602.390156] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26602.395217] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26602.403987] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26602.407557] ORIG_RAX: 0000000000000012 [26602.416164] Workqueue: events delayed_work [ceph] [26602.434902] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26602.444278] [26602.444279] RIP: 0010:do_raw_spin_unlock+0x16/0xc0 [26602.449496] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26602.456967] Code: 00 48 89 47 10 b8 01 00 00 00 c3 31 c0 c3 66 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 53 81 7f 04 ad 4e ad de 48 89 fb 75 34 <8b> 03 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 [26602.460797] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26602.465495] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [26602.472620] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26602.474110] [26602.474110] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [26602.478895] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26602.486018] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [26602.504760] [26602.511883] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [26602.556393] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26602.563518] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26602.570641] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26602.578720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26602.584457] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26602.591582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26602.598703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26602.605828] Call Trace: [26602.608273] [26602.610284] ? watchdog_timer_fn+0x1b4/0x220 [26602.614548] ? __hrtimer_run_queues+0x1d3/0x360 [26602.619074] ? hrtimer_interrupt+0x104/0x240 [26602.623337] ? __local_bh_enable+0x50/0x60 [26602.627430] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26602.632653] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26602.637610] [26602.639707] [26602.641805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26602.647114] ? do_raw_spin_unlock+0x16/0xc0 [26602.651297] _raw_spin_unlock+0x23/0x40 [26602.655135] igrab+0x43/0x50 [26602.658014] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26602.663179] delayed_work+0x1d8/0x280 [ceph] [26602.667460] ? process_scheduled_works+0x22a/0x510 [26602.672242] process_scheduled_works+0x2a4/0x510 [26602.676859] worker_thread+0x188/0x340 [26602.680626] ? rescuer_thread+0x310/0x310 [26602.684635] kthread+0xf5/0x130 [26602.687773] ? kthread_complete_and_exit+0x20/0x20 [26602.692558] ret_from_fork+0x31/0x40 [26602.696128] ? kthread_complete_and_exit+0x20/0x20 [26602.700914] ret_from_fork_asm+0x11/0x20 [26602.704834] [26629.923966] watchdog: BUG: soft lockup - CPU#0 stuck for 13687s! [dbench:78683] [26629.931267] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26629.935966] watchdog: BUG: soft lockup - CPU#7 stuck for 22684s! [kworker/7:4:78808] [26629.986717] irq event stamp: 43389292 [26629.986718] hardirqs last enabled at (43389291): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26629.994447] Modules linked in: [26629.998103] hardirqs last disabled at (43389292): [] sysvec_apic_timer_interrupt+0xe/0x90 [26630.008261] ceph [26630.011312] softirqs last enabled at (43389274): [] __do_softirq+0x2be/0x371 [26630.021038] libceph dns_resolver [26630.022960] softirqs last disabled at (43389269): [] irq_exit_rcu+0xb0/0xd0 [26630.031651] fscache netfs [26630.034964] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26630.043475] veth nft_chain_nat [26630.046179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26630.055123] xt_MASQUERADE nf_nat [26630.058262] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26630.065730] nf_conntrack nf_defrag_ipv6 [26630.069042] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26630.074518] nf_defrag_ipv4 nft_compat [26630.078438] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26630.097175] nf_tables [26630.100920] [26630.106135] nfnetlink bridge [26630.108493] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26630.109984] stp llc [26630.112947] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26630.120073] xfs libcrc32c [26630.122257] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26630.129379] sunrpc [26630.132085] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26630.139209] intel_rapl_msr [26630.141306] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26630.148429] intel_rapl_common sb_edac [26630.151221] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26630.158362] x86_pkg_temp_thermal coretemp [26630.162107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26630.170184] kvm_intel kvm [26630.174273] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26630.180013] irqbypass [26630.182715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26630.189839] i2c_i801 iTCO_wdt [26630.192198] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26630.199323] iTCO_vendor_support [26630.202372] Call Trace: [26630.202374] [26630.209496] i2c_smbus crct10dif_pclmul [26630.212724] ? watchdog_timer_fn+0x1b4/0x220 [26630.215164] joydev crc32_pclmul [26630.217175] ? __hrtimer_run_queues+0x1d3/0x360 [26630.221006] ipmi_ssif ghash_clmulni_intel [26630.225274] ? hrtimer_interrupt+0x104/0x240 [26630.228495] mei_me [26630.233018] ? __local_bh_enable+0x50/0x60 [26630.237110] mei lpc_ich [26630.241376] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26630.243470] sha512_ssse3 [26630.247562] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26630.250091] acpi_ipmi mfd_core [26630.255309] [26630.257927] wmi ipmi_si [26630.262884] [26630.262885] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26630.266022] ipmi_devintf [26630.268124] ? queued_spin_lock_slowpath+0x16/0x2e0 [26630.270662] ipmi_msghandler [26630.272757] do_raw_spin_lock+0xb5/0xc0 [26630.278061] ioatdma acpi_pad [26630.280680] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26630.285550] acpi_power_meter [26630.288429] try_get_cap_refs+0x159/0x880 [ceph] [26630.292257] ixgbe igb [26630.295225] __ceph_get_caps+0x4a1/0x760 [ceph] [26630.300004] mdio crc32c_intel [26630.302972] ? __file_remove_privs+0x89/0x110 [26630.307580] nvme ptp [26630.309936] ? wake_bit_function+0x70/0x70 [26630.314461] nvme_core i2c_algo_bit [26630.317514] ? ceph_write_iter+0x260/0x770 [ceph] [26630.321864] pps_core [26630.324133] ceph_write_iter+0x260/0x770 [ceph] [26630.328222] dca fuse [26630.331712] ? lock_is_held_type+0xaa/0x120 [26630.336406] [26630.336407] irq event stamp: 134931894 [26630.338677] vfs_write+0x324/0x4f0 [26630.343201] hardirqs last enabled at (134931893): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26630.345475] ksys_pwrite64+0x65/0xa0 [26630.349648] hardirqs last disabled at (134931894): [] sysvec_apic_timer_interrupt+0xe/0x90 [26630.351140] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26630.354883] softirqs last enabled at (134931892): [] __do_softirq+0x2be/0x371 [26630.358280] do_syscall_64+0x38/0x80 [26630.368525] softirqs last disabled at (134931887): [] irq_exit_rcu+0xb0/0xd0 [26630.372096] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26630.381906] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26630.387039] RIP: 0033:0x7f48f0eeb93b [26630.395806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26630.399378] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26630.407978] Workqueue: events delayed_work [ceph] [26630.413018] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26630.422397] [26630.425968] ORIG_RAX: 0000000000000012 [26630.433439] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [26630.452178] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26630.456874] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [26630.462090] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26630.463581] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [26630.467414] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26630.473323] [26630.473324] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [26630.480448] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26630.499186] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [26630.506310] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26630.511527] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [26630.518663] [26630.520151] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26630.565097] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26630.572221] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26630.580299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26630.586035] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26630.593160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26630.600284] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26630.607408] Call Trace: [26630.609855] [26630.611863] ? watchdog_timer_fn+0x1b4/0x220 [26630.616127] ? __hrtimer_run_queues+0x1d3/0x360 [26630.620672] ? hrtimer_interrupt+0x104/0x240 [26630.624942] ? __local_bh_enable+0x50/0x60 [26630.629034] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26630.634257] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26630.639217] [26630.641313] [26630.643410] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26630.648719] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [26630.654036] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [26630.659350] delayed_work+0x1d8/0x280 [ceph] [26630.663661] ? process_scheduled_works+0x22a/0x510 [26630.668449] process_scheduled_works+0x2a4/0x510 [26630.673064] worker_thread+0x188/0x340 [26630.676813] ? rescuer_thread+0x310/0x310 [26630.680816] kthread+0xf5/0x130 [26630.683954] ? kthread_complete_and_exit+0x20/0x20 [26630.688739] ret_from_fork+0x31/0x40 [26630.692318] ? kthread_complete_and_exit+0x20/0x20 [26630.697101] ret_from_fork_asm+0x11/0x20 [26630.701025] [26657.924451] watchdog: BUG: soft lockup - CPU#0 stuck for 13713s! [dbench:78683] [26657.931753] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26657.936451] watchdog: BUG: soft lockup - CPU#7 stuck for 22710s! [kworker/7:4:78808] [26657.987195] irq event stamp: 43447492 [26657.987196] hardirqs last enabled at (43447491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26657.994924] Modules linked in: [26657.998582] hardirqs last disabled at (43447492): [] sysvec_apic_timer_interrupt+0xe/0x90 [26658.008741] ceph [26658.011792] softirqs last enabled at (43447474): [] __do_softirq+0x2be/0x371 [26658.021515] libceph [26658.023440] softirqs last disabled at (43447469): [] irq_exit_rcu+0xb0/0xd0 [26658.032126] dns_resolver fscache [26658.034308] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26658.042818] netfs veth [26658.046146] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26658.055090] nft_chain_nat xt_MASQUERADE [26658.057534] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26658.065005] nf_nat nf_conntrack [26658.068922] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26658.074400] nf_defrag_ipv6 nf_defrag_ipv4 [26658.077624] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26658.096363] nft_compat nf_tables [26658.100453] [26658.100453] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26658.105671] nfnetlink bridge [26658.108981] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26658.110470] stp llc [26658.117596] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26658.120560] xfs libcrc32c [26658.127684] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26658.129869] sunrpc intel_rapl_msr [26658.136992] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26658.139696] intel_rapl_common sb_edac [26658.146821] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26658.150234] x86_pkg_temp_thermal coretemp [26658.157359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26658.161103] kvm_intel [26658.169180] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26658.173272] kvm irqbypass [26658.179010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26658.181365] i2c_i801 iTCO_wdt [26658.188489] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26658.191193] iTCO_vendor_support i2c_smbus [26658.198319] Call Trace: [26658.198320] [26658.201369] crct10dif_pclmul [26658.208495] ? watchdog_timer_fn+0x1b4/0x220 [26658.212583] joydev crc32_pclmul [26658.215029] ? __hrtimer_run_queues+0x1d3/0x360 [26658.217039] ipmi_ssif ghash_clmulni_intel [26658.220007] ? hrtimer_interrupt+0x104/0x240 [26658.224267] mei_me mei [26658.227492] ? __local_bh_enable+0x50/0x60 [26658.232015] lpc_ich [26658.236108] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26658.240370] sha512_ssse3 acpi_ipmi [26658.242814] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26658.246904] mfd_core wmi [26658.249089] [26658.254306] ipmi_si ipmi_devintf [26658.257790] [26658.262749] ipmi_msghandler ioatdma [26658.265367] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26658.267462] acpi_pad acpi_power_meter [26658.270777] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26658.272871] ixgbe [26658.276442] do_raw_spin_lock+0xb5/0xc0 [26658.281744] igb mdio [26658.285490] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26658.290360] crc32c_intel nvme [26658.292373] try_get_cap_refs+0x159/0x880 [ceph] [26658.296201] ptp nvme_core [26658.298478] __ceph_get_caps+0x4a1/0x760 [ceph] [26658.303256] i2c_algo_bit pps_core [26658.306311] ? __file_remove_privs+0x89/0x110 [26658.310917] dca fuse [26658.313622] ? wake_bit_function+0x70/0x70 [26658.318146] [26658.318147] irq event stamp: 135152864 [26658.321546] ? ceph_write_iter+0x260/0x770 [ceph] [26658.325894] hardirqs last enabled at (135152863): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26658.328166] ceph_write_iter+0x260/0x770 [ceph] [26658.332255] hardirqs last disabled at (135152864): [] sysvec_apic_timer_interrupt+0xe/0x90 [26658.333750] ? lock_is_held_type+0xaa/0x120 [26658.337492] softirqs last enabled at (135152862): [] __do_softirq+0x2be/0x371 [26658.342189] vfs_write+0x324/0x4f0 [26658.352432] softirqs last disabled at (135152857): [] irq_exit_rcu+0xb0/0xd0 [26658.356960] ksys_pwrite64+0x65/0xa0 [26658.366767] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26658.370945] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26658.379715] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26658.383130] do_syscall_64+0x38/0x80 [26658.391727] Workqueue: events delayed_work [ceph] [26658.395299] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26658.404676] [26658.404677] RIP: 0010:lock_acquire+0xf0/0x2b0 [26658.409806] RIP: 0033:0x7f48f0eeb93b [26658.417276] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26658.420853] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26658.425554] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26658.430597] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26658.432088] [26658.436439] ORIG_RAX: 0000000000000012 [26658.440009] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26658.458747] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26658.477485] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26658.482702] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26658.487920] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26658.489411] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26658.493241] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26658.500365] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26658.507491] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26658.514613] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26658.521740] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26658.528869] [26658.535987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26658.580482] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26658.587606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26658.594730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26658.601858] Call Trace: [26658.604299] [26658.606310] ? watchdog_timer_fn+0x1b4/0x220 [26658.610574] ? __hrtimer_run_queues+0x1d3/0x360 [26658.615101] ? hrtimer_interrupt+0x104/0x240 [26658.619370] ? __local_bh_enable+0x50/0x60 [26658.623465] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26658.628687] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26658.633646] [26658.635743] [26658.637855] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26658.643166] ? lock_acquire+0xf0/0x2b0 [26658.646915] ? igrab+0x43/0x50 [26658.649966] ? lock_release+0xc5/0x260 [26658.653710] _raw_spin_lock+0x30/0x40 [26658.657367] ? igrab+0x1d/0x50 [26658.660417] igrab+0x1d/0x50 [26658.663296] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26658.668446] delayed_work+0x1d8/0x280 [ceph] [26658.672726] ? process_scheduled_works+0x22a/0x510 [26658.677515] process_scheduled_works+0x2a4/0x510 [26658.682141] worker_thread+0x188/0x340 [26658.685889] ? rescuer_thread+0x310/0x310 [26658.689892] kthread+0xf5/0x130 [26658.693031] ? kthread_complete_and_exit+0x20/0x20 [26658.697814] ret_from_fork+0x31/0x40 [26658.701385] ? kthread_complete_and_exit+0x20/0x20 [26658.706170] ret_from_fork_asm+0x11/0x20 [26658.710091] [26659.551846] rcu: INFO: rcu_preempt self-detected stall on CPU [26659.557607] rcu: 7-....: (14297062 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/14295766 fqs=7194421 [26659.568198] rcu: (t=14722680 jiffies g=1309609 q=445321 ncpus=8) [26659.574282] Sending NMI from CPU 7 to CPUs 0: [26659.578635] NMI backtrace for cpu 0 [26659.578637] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26659.578638] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26659.578639] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26659.578642] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26659.578643] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26659.578644] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26659.578645] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26659.578646] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26659.578647] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26659.578648] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26659.578649] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26659.578650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26659.578651] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26659.578651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26659.578652] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26659.578653] Call Trace: [26659.578653] [26659.578654] ? nmi_cpu_backtrace+0xad/0x120 [26659.578658] ? nmi_cpu_backtrace_handler+0x11/0x20 [26659.578659] ? nmi_handle+0xd4/0x260 [26659.578664] ? default_do_nmi+0x49/0xf0 [26659.578665] ? exc_nmi+0x11e/0x150 [26659.578667] ? end_repeat_nmi+0x16/0x67 [26659.578671] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26659.578673] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26659.578676] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26659.578678] [26659.578679] [26659.578680] do_raw_spin_lock+0xb5/0xc0 [26659.578682] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26659.578697] try_get_cap_refs+0x159/0x880 [ceph] [26659.578714] __ceph_get_caps+0x4a1/0x760 [ceph] [26659.578731] ? __file_remove_privs+0x89/0x110 [26659.578733] ? wake_bit_function+0x70/0x70 [26659.578735] ? ceph_write_iter+0x260/0x770 [ceph] [26659.578748] ceph_write_iter+0x260/0x770 [ceph] [26659.578762] ? lock_is_held_type+0xaa/0x120 [26659.578765] vfs_write+0x324/0x4f0 [26659.578769] ksys_pwrite64+0x65/0xa0 [26659.578771] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26659.578774] do_syscall_64+0x38/0x80 [26659.578776] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26659.578778] RIP: 0033:0x7f48f0eeb93b [26659.578779] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26659.578781] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [26659.578782] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26659.578783] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26659.578783] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26659.578784] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26659.578785] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26659.578788] [26659.579634] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26659.883194] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26659.890665] Workqueue: events delayed_work [ceph] [26659.895380] RIP: 0010:lock_acquire+0xf0/0x2b0 [26659.899730] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26659.918468] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26659.923686] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26659.930809] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26659.937932] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26659.945057] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26659.952180] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26659.959306] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26659.967382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26659.973120] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26659.980245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26659.987368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26659.994493] Call Trace: [26659.996938] [26659.998948] ? rcu_dump_cpu_stacks+0xd1/0x180 [26660.003306] ? rcu_sched_clock_irq+0x633/0x1390 [26660.007830] ? lock_release+0xc5/0x260 [26660.011576] ? lock_is_held_type+0xaa/0x120 [26660.015753] ? lock_is_held_type+0xaa/0x120 [26660.019931] ? tick_sched_do_timer+0xa0/0xa0 [26660.024204] ? rcu_read_lock_sched_held+0x56/0x90 [26660.028901] ? tick_sched_do_timer+0xa0/0xa0 [26660.033164] ? update_process_times+0x5f/0x90 [26660.037514] ? tick_sched_handle.isra.16+0x1e/0x50 [26660.042300] ? tick_sched_timer+0x6f/0x80 [26660.046303] ? __hrtimer_run_queues+0x1d3/0x360 [26660.050830] ? hrtimer_interrupt+0x104/0x240 [26660.055100] ? __local_bh_enable+0x50/0x60 [26660.059193] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26660.064417] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26660.069374] [26660.071472] [26660.073569] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26660.078876] ? lock_acquire+0xf0/0x2b0 [26660.082628] ? igrab+0x43/0x50 [26660.085678] ? lock_release+0xc5/0x260 [26660.089424] _raw_spin_lock+0x30/0x40 [26660.093086] ? igrab+0x1d/0x50 [26660.096138] igrab+0x1d/0x50 [26660.099016] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26660.104164] delayed_work+0x1d8/0x280 [ceph] [26660.108445] ? process_scheduled_works+0x22a/0x510 [26660.113228] process_scheduled_works+0x2a4/0x510 [26660.117843] worker_thread+0x188/0x340 [26660.121593] ? rescuer_thread+0x310/0x310 [26660.125596] kthread+0xf5/0x130 [26660.128733] ? kthread_complete_and_exit+0x20/0x20 [26660.133517] ret_from_fork+0x31/0x40 [26660.137087] ? kthread_complete_and_exit+0x20/0x20 [26660.141874] ret_from_fork_asm+0x11/0x20 [26660.145795] [26685.924935] watchdog: BUG: soft lockup - CPU#0 stuck for 13739s! [dbench:78683] [26685.932242] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26685.936935] watchdog: BUG: soft lockup - CPU#7 stuck for 22736s! [kworker/7:4:78808] [26685.987692] irq event stamp: 43505682 [26685.995423] Modules linked in: ceph libceph dns_resolver fscache netfs [26685.999079] hardirqs last enabled at (43505681): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26685.999084] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [26686.015772] hardirqs last disabled at (43505682): [] sysvec_apic_timer_interrupt+0xe/0x90 [26686.015775] nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c [26686.034188] softirqs last enabled at (43505666): [] __do_softirq+0x2be/0x371 [26686.040889] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [26686.049571] softirqs last disabled at (43505661): [] irq_exit_rcu+0xb0/0xd0 [26686.049575] kvm_intel kvm irqbypass i2c_i801 [26686.066335] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26686.066337] iTCO_wdt [26686.066338] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26686.070693] iTCO_vendor_support i2c_smbus [26686.079638] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26686.081907] crct10dif_pclmul [26686.089377] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26686.093468] joydev crc32_pclmul [26686.098946] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26686.101911] ipmi_ssif ghash_clmulni_intel [26686.120650] [26686.123872] mei_me [26686.129090] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26686.133181] mei lpc_ich [26686.134671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26686.136770] sha512_ssse3 acpi_ipmi [26686.143892] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26686.146423] mfd_core [26686.153549] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26686.157031] wmi ipmi_si [26686.164157] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26686.166428] ipmi_devintf [26686.173550] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26686.176082] ipmi_msghandler [26686.183206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26686.185822] ioatdma acpi_pad [26686.193901] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26686.196779] acpi_power_meter [26686.202514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26686.205480] ixgbe [26686.212619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26686.215584] igb mdio [26686.222711] Call Trace: [26686.224720] crc32c_intel [26686.231845] [26686.234114] nvme [26686.236562] ? watchdog_timer_fn+0x1b4/0x220 [26686.239175] ptp nvme_core [26686.241189] ? __hrtimer_run_queues+0x1d3/0x360 [26686.243111] i2c_algo_bit pps_core [26686.247378] ? hrtimer_interrupt+0x104/0x240 [26686.250079] dca fuse [26686.254619] ? __local_bh_enable+0x50/0x60 [26686.258017] [26686.258018] irq event stamp: 135368918 [26686.262284] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26686.264553] hardirqs last enabled at (135368917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26686.268643] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26686.270134] hardirqs last disabled at (135368918): [] sysvec_apic_timer_interrupt+0xe/0x90 [26686.273878] [26686.279096] softirqs last enabled at (135368916): [] __do_softirq+0x2be/0x371 [26686.289340] [26686.294298] softirqs last disabled at (135368911): [] irq_exit_rcu+0xb0/0xd0 [26686.304109] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26686.306206] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26686.314979] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26686.317074] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26686.325673] do_raw_spin_lock+0xb5/0xc0 [26686.330975] Workqueue: events delayed_work [ceph] [26686.340353] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26686.345223] [26686.352697] try_get_cap_refs+0x159/0x880 [ceph] [26686.356525] RIP: 0010:lock_acquire+0xf0/0x2b0 [26686.361228] __ceph_get_caps+0x4a1/0x760 [ceph] [26686.366007] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26686.367501] ? __file_remove_privs+0x89/0x110 [26686.372108] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26686.376460] ? wake_bit_function+0x70/0x70 [26686.380983] [26686.399724] ? ceph_write_iter+0x260/0x770 [ceph] [26686.404071] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26686.409288] ceph_write_iter+0x260/0x770 [ceph] [26686.413380] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26686.414873] ? lock_is_held_type+0xaa/0x120 [26686.419568] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26686.426694] vfs_write+0x324/0x4f0 [26686.431215] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26686.438344] ksys_pwrite64+0x65/0xa0 [26686.442518] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26686.449641] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26686.453039] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26686.460163] do_syscall_64+0x38/0x80 [26686.463733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26686.470858] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26686.475989] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26686.484066] RIP: 0033:0x7f48f0eeb93b [26686.487638] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26686.493373] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26686.498418] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26686.505541] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26686.509112] Call Trace: [26686.509114] [26686.516237] ORIG_RAX: 0000000000000012 [26686.534977] ? watchdog_timer_fn+0x1b4/0x220 [26686.542099] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26686.547336] ? __hrtimer_run_queues+0x1d3/0x360 [26686.549777] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26686.551793] ? hrtimer_interrupt+0x104/0x240 [26686.555620] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26686.559884] ? __local_bh_enable+0x50/0x60 [26686.567008] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26686.571535] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26686.578654] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26686.582919] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26686.590048] [26686.594134] [26686.622856] [26686.624955] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26686.630263] ? lock_acquire+0xf0/0x2b0 [26686.634013] ? igrab+0x43/0x50 [26686.637070] ? lock_release+0xc5/0x260 [26686.640817] _raw_spin_lock+0x30/0x40 [26686.644482] ? igrab+0x1d/0x50 [26686.647533] igrab+0x1d/0x50 [26686.650410] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26686.655561] delayed_work+0x1d8/0x280 [ceph] [26686.659840] ? process_scheduled_works+0x22a/0x510 [26686.664631] process_scheduled_works+0x2a4/0x510 [26686.669248] worker_thread+0x188/0x340 [26686.672997] ? rescuer_thread+0x310/0x310 [26686.677007] kthread+0xf5/0x130 [26686.680144] ? kthread_complete_and_exit+0x20/0x20 [26686.684928] ret_from_fork+0x31/0x40 [26686.688510] ? kthread_complete_and_exit+0x20/0x20 [26686.693301] ret_from_fork_asm+0x11/0x20 [26686.697224] [26713.925419] watchdog: BUG: soft lockup - CPU#0 stuck for 13765s! [dbench:78683] [26713.932719] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26713.937419] watchdog: BUG: soft lockup - CPU#7 stuck for 22762s! [kworker/7:4:78808] [26713.988162] irq event stamp: 43563802 [26713.988163] hardirqs last enabled at (43563801): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26713.995892] Modules linked in: [26713.999548] hardirqs last disabled at (43563802): [] sysvec_apic_timer_interrupt+0xe/0x90 [26714.009707] ceph [26714.012758] softirqs last enabled at (43563794): [] __do_softirq+0x2be/0x371 [26714.022481] libceph dns_resolver [26714.024405] softirqs last disabled at (43563789): [] irq_exit_rcu+0xb0/0xd0 [26714.033093] fscache netfs [26714.036400] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26714.044912] veth nft_chain_nat [26714.047615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26714.056560] xt_MASQUERADE [26714.059696] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26714.067168] nf_nat nf_conntrack [26714.069872] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26714.075349] nf_defrag_ipv6 nf_defrag_ipv4 [26714.078572] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26714.097310] nft_compat nf_tables [26714.101402] [26714.101402] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26714.106618] nfnetlink bridge [26714.109930] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26714.111421] stp llc [26714.118544] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26714.121509] xfs libcrc32c [26714.128632] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26714.130815] sunrpc [26714.137940] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26714.140646] intel_rapl_msr intel_rapl_common [26714.147769] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26714.149867] sb_edac x86_pkg_temp_thermal [26714.156990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26714.161341] coretemp kvm_intel [26714.169418] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26714.173422] kvm [26714.179161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26714.182299] irqbypass i2c_i801 [26714.189423] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26714.191258] iTCO_wdt iTCO_vendor_support [26714.198384] Call Trace: [26714.198385] [26714.201520] i2c_smbus [26714.208646] ? watchdog_timer_fn+0x1b4/0x220 [26714.212649] crct10dif_pclmul joydev [26714.215096] ? __hrtimer_run_queues+0x1d3/0x360 [26714.217115] crc32_pclmul ipmi_ssif [26714.219472] ? hrtimer_interrupt+0x104/0x240 [26714.223733] ghash_clmulni_intel mei_me [26714.227303] ? __local_bh_enable+0x50/0x60 [26714.231829] mei lpc_ich [26714.235316] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26714.239577] sha512_ssse3 acpi_ipmi [26714.243407] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26714.247499] mfd_core wmi [26714.250030] [26714.255247] ipmi_si ipmi_devintf [26714.258730] [26714.258731] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26714.263688] ipmi_msghandler ioatdma [26714.266308] ? queued_spin_lock_slowpath+0x16/0x2e0 [26714.268404] acpi_pad acpi_power_meter [26714.271714] do_raw_spin_lock+0xb5/0xc0 [26714.273818] ixgbe igb [26714.279125] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26714.282693] mdio crc32c_intel [26714.287567] try_get_cap_refs+0x159/0x880 [ceph] [26714.291308] nvme [26714.295145] __ceph_get_caps+0x4a1/0x760 [ceph] [26714.297498] ptp nvme_core [26714.302285] ? __file_remove_privs+0x89/0x110 [26714.305331] i2c_algo_bit pps_core [26714.309944] ? wake_bit_function+0x70/0x70 [26714.311867] dca fuse [26714.316392] ? ceph_write_iter+0x260/0x770 [ceph] [26714.319107] [26714.319108] irq event stamp: 135589948 [26714.323455] ceph_write_iter+0x260/0x770 [ceph] [26714.326850] hardirqs last enabled at (135589947): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26714.330945] ? lock_is_held_type+0xaa/0x120 [26714.333214] hardirqs last disabled at (135589948): [] sysvec_apic_timer_interrupt+0xe/0x90 [26714.337912] vfs_write+0x324/0x4f0 [26714.339400] softirqs last enabled at (135589946): [] __do_softirq+0x2be/0x371 [26714.343151] ksys_pwrite64+0x65/0xa0 [26714.347668] softirqs last disabled at (135589941): [] irq_exit_rcu+0xb0/0xd0 [26714.357912] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26714.362091] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26714.371921] do_syscall_64+0x38/0x80 [26714.375317] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26714.384101] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26714.387677] Workqueue: events delayed_work [ceph] [26714.396273] RIP: 0033:0x7f48f0eeb93b [26714.401404] [26714.401404] RIP: 0010:lock_acquire+0xf0/0x2b0 [26714.410782] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26714.414353] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26714.421822] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26714.426867] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26714.431564] ORIG_RAX: 0000000000000012 [26714.435134] [26714.435134] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26714.436625] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26714.440976] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26714.459715] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26714.478453] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26714.483669] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26714.488885] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26714.492718] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26714.494209] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26714.501333] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26714.508457] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26714.515585] [26714.522704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26714.581448] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26714.588574] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26714.595697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26714.602830] Call Trace: [26714.605274] [26714.607286] ? watchdog_timer_fn+0x1b4/0x220 [26714.611550] ? __hrtimer_run_queues+0x1d3/0x360 [26714.616075] ? hrtimer_interrupt+0x104/0x240 [26714.620336] ? __local_bh_enable+0x50/0x60 [26714.624431] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26714.629655] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26714.634611] [26714.636709] [26714.638824] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26714.644132] ? lock_acquire+0xf0/0x2b0 [26714.647881] ? igrab+0x43/0x50 [26714.650932] ? lock_release+0xc5/0x260 [26714.654677] _raw_spin_lock+0x30/0x40 [26714.658342] ? igrab+0x1d/0x50 [26714.661393] igrab+0x1d/0x50 [26714.664270] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26714.669420] delayed_work+0x1d8/0x280 [ceph] [26714.673699] ? process_scheduled_works+0x22a/0x510 [26714.678482] process_scheduled_works+0x2a4/0x510 [26714.683109] worker_thread+0x188/0x340 [26714.686857] ? rescuer_thread+0x310/0x310 [26714.690868] kthread+0xf5/0x130 [26714.694006] ? kthread_complete_and_exit+0x20/0x20 [26714.698789] ret_from_fork+0x31/0x40 [26714.702361] ? kthread_complete_and_exit+0x20/0x20 [26714.707143] ret_from_fork_asm+0x11/0x20 [26714.711067] [26741.925905] watchdog: BUG: soft lockup - CPU#0 stuck for 13791s! [dbench:78683] [26741.933208] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26741.937905] watchdog: BUG: soft lockup - CPU#7 stuck for 22788s! [kworker/7:4:78808] [26741.988658] irq event stamp: 43622084 [26741.996389] Modules linked in: ceph libceph dns_resolver fscache netfs veth [26742.000046] hardirqs last enabled at (43622083): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26742.007004] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [26742.017163] hardirqs last disabled at (43622084): [] sysvec_apic_timer_interrupt+0xe/0x90 [26742.024113] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [26742.033838] softirqs last enabled at (43622076): [] __do_softirq+0x2be/0x371 [26742.033842] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [26742.048977] softirqs last disabled at (43622071): [] irq_exit_rcu+0xb0/0xd0 [26742.058009] coretemp kvm_intel kvm [26742.066519] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26742.066520] irqbypass [26742.066522] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26742.070004] i2c_i801 iTCO_wdt [26742.078947] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26742.081305] iTCO_vendor_support [26742.088777] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26742.091827] i2c_smbus crct10dif_pclmul [26742.097305] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26742.100529] joydev crc32_pclmul [26742.119266] [26742.123097] ipmi_ssif ghash_clmulni_intel [26742.128315] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26742.131538] mei_me mei [26742.133030] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26742.137119] lpc_ich [26742.144245] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26742.146688] sha512_ssse3 acpi_ipmi [26742.153813] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26742.155996] mfd_core wmi [26742.163120] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26742.166605] ipmi_si ipmi_devintf [26742.173729] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26742.176345] ipmi_msghandler ioatdma [26742.183469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26742.186780] acpi_pad acpi_power_meter [26742.194859] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26742.198430] ixgbe [26742.204165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26742.207911] igb mdio [26742.215034] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26742.217046] crc32c_intel [26742.224169] Call Trace: [26742.224172] [26742.226441] nvme ptp [26742.233567] ? watchdog_timer_fn+0x1b4/0x220 [26742.236183] nvme_core [26742.238628] ? __hrtimer_run_queues+0x1d3/0x360 [26742.240635] i2c_algo_bit pps_core [26742.242910] ? hrtimer_interrupt+0x104/0x240 [26742.247170] dca fuse [26742.249527] ? __local_bh_enable+0x50/0x60 [26742.254052] [26742.254052] irq event stamp: 135811122 [26742.257453] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26742.261715] hardirqs last enabled at (135811121): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26742.263985] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26742.268075] hardirqs last disabled at (135811122): [] sysvec_apic_timer_interrupt+0xe/0x90 [26742.269582] [26742.273329] softirqs last enabled at (135811120): [] __do_softirq+0x2be/0x371 [26742.278544] [26742.278546] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26742.288788] softirqs last disabled at (135811115): [] irq_exit_rcu+0xb0/0xd0 [26742.293751] ? queued_spin_lock_slowpath+0x18/0x2e0 [26742.303557] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26742.305656] do_raw_spin_lock+0xb5/0xc0 [26742.314425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26742.316525] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26742.321827] Workqueue: events delayed_work [ceph] [26742.330427] try_get_cap_refs+0x159/0x880 [ceph] [26742.335306] [26742.335307] RIP: 0010:lock_acquire+0xf0/0x2b0 [26742.344688] __ceph_get_caps+0x4a1/0x760 [ceph] [26742.348512] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26742.355988] ? __file_remove_privs+0x89/0x110 [26742.360766] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26742.365465] ? wake_bit_function+0x70/0x70 [26742.370074] [26742.370075] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26742.371584] ? ceph_write_iter+0x260/0x770 [ceph] [26742.375934] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26742.380458] ceph_write_iter+0x260/0x770 [ceph] [26742.399195] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26742.403552] ? lock_is_held_type+0xaa/0x120 [26742.408764] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26742.412856] vfs_write+0x324/0x4f0 [26742.414346] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26742.421476] ksys_pwrite64+0x65/0xa0 [26742.426166] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26742.433308] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26742.437833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26742.444958] do_syscall_64+0x38/0x80 [26742.449135] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26742.456260] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26742.459656] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26742.466782] RIP: 0033:0x7f48f0eeb93b [26742.470350] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26742.478428] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26742.483558] Call Trace: [26742.483560] [26742.489308] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26742.492878] ? watchdog_timer_fn+0x1b4/0x220 [26742.499999] ORIG_RAX: 0000000000000012 [26742.505046] ? __hrtimer_run_queues+0x1d3/0x360 [26742.512167] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26742.515743] ? hrtimer_interrupt+0x104/0x240 [26742.522863] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26742.541599] ? __local_bh_enable+0x50/0x60 [26742.544044] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26742.546057] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26742.551271] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26742.555538] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26742.559368] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26742.563893] [26742.571021] [26742.575279] [26742.575281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26742.629731] ? lock_acquire+0xf0/0x2b0 [26742.633480] ? igrab+0x43/0x50 [26742.636537] ? lock_release+0xc5/0x260 [26742.640283] _raw_spin_lock+0x30/0x40 [26742.643958] ? igrab+0x1d/0x50 [26742.647007] igrab+0x1d/0x50 [26742.649886] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26742.655035] delayed_work+0x1d8/0x280 [ceph] [26742.659316] ? process_scheduled_works+0x22a/0x510 [26742.664107] process_scheduled_works+0x2a4/0x510 [26742.668723] worker_thread+0x188/0x340 [26742.672471] ? rescuer_thread+0x310/0x310 [26742.676475] kthread+0xf5/0x130 [26742.679622] ? kthread_complete_and_exit+0x20/0x20 [26742.684404] ret_from_fork+0x31/0x40 [26742.687984] ? kthread_complete_and_exit+0x20/0x20 [26742.692768] ret_from_fork_asm+0x11/0x20 [26742.696691] [26769.926392] watchdog: BUG: soft lockup - CPU#0 stuck for 13817s! [dbench:78683] [26769.933696] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26769.938392] watchdog: BUG: soft lockup - CPU#7 stuck for 22814s! [kworker/7:4:78808] [26769.989137] irq event stamp: 43680308 [26769.989138] hardirqs last enabled at (43680307): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26769.996868] Modules linked in: [26770.000523] hardirqs last disabled at (43680308): [] sysvec_apic_timer_interrupt+0xe/0x90 [26770.010681] ceph [26770.013732] softirqs last enabled at (43680300): [] __do_softirq+0x2be/0x371 [26770.023456] libceph [26770.025379] softirqs last disabled at (43680295): [] irq_exit_rcu+0xb0/0xd0 [26770.034077] dns_resolver fscache [26770.036256] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26770.044784] netfs veth [26770.048096] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26770.057040] nft_chain_nat xt_MASQUERADE [26770.059485] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26770.066956] nf_nat nf_conntrack [26770.070872] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26770.076349] nf_defrag_ipv6 nf_defrag_ipv4 [26770.079575] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26770.098311] nft_compat nf_tables [26770.102403] [26770.102404] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26770.107619] nfnetlink bridge [26770.110931] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26770.112423] stp llc [26770.119545] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26770.122510] xfs libcrc32c [26770.129635] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26770.131818] sunrpc intel_rapl_msr [26770.138941] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26770.141646] intel_rapl_common [26770.148788] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26770.152184] sb_edac x86_pkg_temp_thermal [26770.159309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26770.162360] coretemp kvm_intel [26770.170437] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26770.174442] kvm irqbypass [26770.180178] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26770.183316] i2c_i801 [26770.190439] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26770.193144] iTCO_wdt iTCO_vendor_support [26770.200268] Call Trace: [26770.200269] [26770.202538] i2c_smbus crct10dif_pclmul [26770.209665] ? watchdog_timer_fn+0x1b4/0x220 [26770.213666] joydev crc32_pclmul [26770.216113] ? __hrtimer_run_queues+0x1d3/0x360 [26770.218122] ipmi_ssif ghash_clmulni_intel [26770.221956] ? hrtimer_interrupt+0x104/0x240 [26770.226216] mei_me mei [26770.229442] ? __local_bh_enable+0x50/0x60 [26770.233964] lpc_ich sha512_ssse3 [26770.238057] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26770.242321] acpi_ipmi mfd_core [26770.244766] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26770.248873] wmi ipmi_si [26770.252182] [26770.257399] ipmi_devintf ipmi_msghandler [26770.260537] [26770.265494] ioatdma acpi_pad [26770.268026] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26770.270123] acpi_power_meter ixgbe [26770.274131] ? queued_spin_lock_slowpath+0x16/0x2e0 [26770.276223] igb mdio [26770.279190] do_raw_spin_lock+0xb5/0xc0 [26770.284492] crc32c_intel nvme [26770.287978] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26770.292848] ptp nvme_core [26770.295120] try_get_cap_refs+0x159/0x880 [ceph] [26770.298950] i2c_algo_bit pps_core [26770.302005] __ceph_get_caps+0x4a1/0x760 [ceph] [26770.306794] dca fuse [26770.309500] ? __file_remove_privs+0x89/0x110 [26770.314108] [26770.317504] ? wake_bit_function+0x70/0x70 [26770.322027] irq event stamp: 136032222 [26770.324302] ? ceph_write_iter+0x260/0x770 [ceph] [26770.328650] hardirqs last enabled at (136032221): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26770.330141] ceph_write_iter+0x260/0x770 [ceph] [26770.334232] hardirqs last disabled at (136032222): [] sysvec_apic_timer_interrupt+0xe/0x90 [26770.337980] ? lock_is_held_type+0xaa/0x120 [26770.342673] softirqs last enabled at (136032220): [] __do_softirq+0x2be/0x371 [26770.352920] vfs_write+0x324/0x4f0 [26770.357441] softirqs last disabled at (136032215): [] irq_exit_rcu+0xb0/0xd0 [26770.367257] ksys_pwrite64+0x65/0xa0 [26770.371429] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26770.380200] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26770.383597] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26770.392197] do_syscall_64+0x38/0x80 [26770.395767] Workqueue: events delayed_work [ceph] [26770.405161] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26770.410291] [26770.410292] RIP: 0010:lock_acquire+0xf0/0x2b0 [26770.417763] RIP: 0033:0x7f48f0eeb93b [26770.421334] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26770.426031] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26770.431080] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26770.432574] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26770.436924] [26770.436925] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26770.440495] ORIG_RAX: 0000000000000012 [26770.459232] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26770.477971] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26770.483188] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26770.488405] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26770.489895] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26770.497020] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26770.500851] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26770.507976] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26770.515101] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26770.522223] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26770.529348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26770.536477] [26770.543596] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26770.588100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26770.595226] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26770.602350] Call Trace: [26770.604798] [26770.606812] ? watchdog_timer_fn+0x1b4/0x220 [26770.611088] ? __hrtimer_run_queues+0x1d3/0x360 [26770.615612] ? hrtimer_interrupt+0x104/0x240 [26770.619872] ? __local_bh_enable+0x50/0x60 [26770.623968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26770.629190] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26770.634149] [26770.636244] [26770.638343] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26770.643651] ? lock_acquire+0xf0/0x2b0 [26770.647400] ? igrab+0x43/0x50 [26770.650450] ? lock_release+0xc5/0x260 [26770.654196] _raw_spin_lock+0x30/0x40 [26770.657859] ? igrab+0x1d/0x50 [26770.660911] igrab+0x1d/0x50 [26770.663800] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26770.668946] delayed_work+0x1d8/0x280 [ceph] [26770.673227] ? process_scheduled_works+0x22a/0x510 [26770.678010] process_scheduled_works+0x2a4/0x510 [26770.682625] worker_thread+0x188/0x340 [26770.686375] ? rescuer_thread+0x310/0x310 [26770.690386] kthread+0xf5/0x130 [26770.693523] ? kthread_complete_and_exit+0x20/0x20 [26770.698309] ret_from_fork+0x31/0x40 [26770.701880] ? kthread_complete_and_exit+0x20/0x20 [26770.706664] ret_from_fork_asm+0x11/0x20 [26770.710586] [26797.926881] watchdog: BUG: soft lockup - CPU#0 stuck for 13843s! [dbench:78683] [26797.934184] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26797.938881] watchdog: BUG: soft lockup - CPU#7 stuck for 22840s! [kworker/7:4:78808] [26797.989633] irq event stamp: 43738562 [26797.997364] Modules linked in: ceph libceph dns_resolver fscache [26798.001020] hardirqs last enabled at (43738561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26798.007027] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [26798.017182] hardirqs last disabled at (43738562): [] sysvec_apic_timer_interrupt+0xe/0x90 [26798.022660] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [26798.032386] softirqs last enabled at (43738554): [] __do_softirq+0x2be/0x371 [26798.032391] stp llc xfs libcrc32c sunrpc intel_rapl_msr [26798.049588] softirqs last disabled at (43738549): [] irq_exit_rcu+0xb0/0xd0 [26798.054892] intel_rapl_common sb_edac x86_pkg_temp_thermal [26798.063405] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26798.068967] coretemp kvm_intel [26798.077913] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26798.081048] kvm irqbypass i2c_i801 iTCO_wdt [26798.088521] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26798.092784] iTCO_vendor_support i2c_smbus crct10dif_pclmul [26798.098279] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26798.098281] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26798.103844] joydev crc32_pclmul [26798.122582] [26798.127798] ipmi_ssif [26798.131022] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26798.132513] ghash_clmulni_intel [26798.134869] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26798.141994] mei_me [26798.145218] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26798.152342] mei lpc_ich [26798.154439] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26798.161566] sha512_ssse3 acpi_ipmi [26798.164096] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26798.164097] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26798.171218] mfd_core [26798.174703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26798.181827] wmi ipmi_si [26798.189904] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26798.192176] ipmi_devintf [26798.197912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26798.200446] ipmi_msghandler [26798.207567] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26798.210184] ioatdma acpi_pad [26798.217309] Call Trace: [26798.220187] acpi_power_meter [26798.227314] [26798.230279] ixgbe [26798.232721] ? watchdog_timer_fn+0x1b4/0x220 [26798.235683] igb mdio [26798.237694] ? __hrtimer_run_queues+0x1d3/0x360 [26798.239705] crc32c_intel [26798.243972] ? hrtimer_interrupt+0x104/0x240 [26798.246240] nvme ptp [26798.250764] ? __local_bh_enable+0x50/0x60 [26798.253382] nvme_core [26798.257647] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26798.259915] i2c_algo_bit pps_core [26798.264007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26798.266364] dca fuse [26798.271582] [26798.274979] [26798.274980] irq event stamp: 136253236 [26798.279936] [26798.282206] hardirqs last enabled at (136253235): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26798.284304] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26798.285795] hardirqs last disabled at (136253236): [] sysvec_apic_timer_interrupt+0xe/0x90 [26798.289542] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26798.291636] softirqs last enabled at (136253234): [] __do_softirq+0x2be/0x371 [26798.301883] do_raw_spin_lock+0xb5/0xc0 [26798.307185] softirqs last disabled at (136253229): [] irq_exit_rcu+0xb0/0xd0 [26798.316998] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26798.321867] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26798.330639] try_get_cap_refs+0x159/0x880 [ceph] [26798.334468] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26798.343072] __ceph_get_caps+0x4a1/0x760 [ceph] [26798.347849] Workqueue: events delayed_work [ceph] [26798.357231] ? __file_remove_privs+0x89/0x110 [26798.361838] [26798.361839] RIP: 0010:ceph_check_delayed_caps+0x63/0x270 [ceph] [26798.369309] ? wake_bit_function+0x70/0x70 [26798.373833] Code: 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 <48> 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 [26798.378532] ? ceph_write_iter+0x260/0x770 [ceph] [26798.382881] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [26798.384371] ceph_write_iter+0x260/0x770 [ceph] [26798.390281] [26798.390282] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [26798.394376] ? lock_is_held_type+0xaa/0x120 [26798.413110] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [26798.417811] vfs_write+0x324/0x4f0 [26798.423025] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [26798.427568] ksys_pwrite64+0x65/0xa0 [26798.429057] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [26798.436183] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26798.440358] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [26798.447485] do_syscall_64+0x38/0x80 [26798.450881] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26798.458005] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26798.461575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26798.468698] RIP: 0033:0x7f48f0eeb93b [26798.473831] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26798.480955] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26798.484526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26798.492603] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26798.497646] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26798.503384] ORIG_RAX: 0000000000000012 [26798.506953] Call Trace: [26798.506955] [26798.514080] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26798.532818] ? watchdog_timer_fn+0x1b4/0x220 [26798.539940] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26798.545160] ? __hrtimer_run_queues+0x1d3/0x360 [26798.552284] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26798.556116] ? hrtimer_interrupt+0x104/0x240 [26798.558572] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26798.560584] ? __local_bh_enable+0x50/0x60 [26798.567709] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26798.571975] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26798.579102] [26798.583621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26798.625726] [26798.627824] [26798.629921] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26798.635229] ? ceph_check_delayed_caps+0x63/0x270 [ceph] [26798.640546] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [26798.645878] delayed_work+0x1d8/0x280 [ceph] [26798.650160] ? process_scheduled_works+0x22a/0x510 [26798.654949] process_scheduled_works+0x2a4/0x510 [26798.659582] worker_thread+0x188/0x340 [26798.663332] ? rescuer_thread+0x310/0x310 [26798.667335] kthread+0xf5/0x130 [26798.670483] ? kthread_complete_and_exit+0x20/0x20 [26798.675288] ret_from_fork+0x31/0x40 [26798.678861] ? kthread_complete_and_exit+0x20/0x20 [26798.683647] ret_from_fork_asm+0x11/0x20 [26798.687577] [26825.927370] watchdog: BUG: soft lockup - CPU#0 stuck for 13869s! [dbench:78683] [26825.934668] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26825.939370] watchdog: BUG: soft lockup - CPU#7 stuck for 22866s! [kworker/7:4:78808] [26825.990111] irq event stamp: 43796708 [26825.997841] Modules linked in: ceph libceph [26826.001500] hardirqs last enabled at (43796707): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26826.005675] dns_resolver fscache netfs [26826.015835] hardirqs last disabled at (43796708): [] sysvec_apic_timer_interrupt+0xe/0x90 [26826.019664] veth nft_chain_nat xt_MASQUERADE nf_nat [26826.029390] softirqs last enabled at (43796700): [] __do_softirq+0x2be/0x371 [26826.034346] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [26826.043046] softirqs last disabled at (43796695): [] irq_exit_rcu+0xb0/0xd0 [26826.049219] nf_tables nfnetlink [26826.057729] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26826.060954] bridge stp [26826.069896] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26826.072343] llc xfs libcrc32c sunrpc [26826.079813] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26826.083471] intel_rapl_msr intel_rapl_common [26826.088947] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26826.088948] sb_edac [26826.093298] x86_pkg_temp_thermal coretemp [26826.112049] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26826.114238] kvm_intel kvm [26826.118328] [26826.118329] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26826.123545] irqbypass [26826.126250] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26826.127741] i2c_i801 iTCO_wdt [26826.134882] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26826.137237] iTCO_vendor_support [26826.144364] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26826.147413] i2c_smbus crct10dif_pclmul [26826.154538] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26826.157768] joydev [26826.164894] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26826.168725] crc32_pclmul ipmi_ssif [26826.175849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26826.177946] ghash_clmulni_intel mei_me [26826.186023] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26826.189525] mei lpc_ich [26826.195264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26826.199094] sha512_ssse3 [26826.206218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26826.208765] acpi_ipmi mfd_core [26826.215890] Call Trace: [26826.215892] [26826.218508] wmi ipmi_si [26826.225633] ? watchdog_timer_fn+0x1b4/0x220 [26826.228770] ipmi_devintf [26826.231215] ? __hrtimer_run_queues+0x1d3/0x360 [26826.233223] ipmi_msghandler ioatdma [26826.235772] ? hrtimer_interrupt+0x104/0x240 [26826.240049] acpi_pad acpi_power_meter [26826.242671] ? __local_bh_enable+0x50/0x60 [26826.247195] ixgbe igb [26826.250772] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26826.255029] mdio crc32c_intel [26826.258792] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26826.262881] nvme ptp [26826.265238] [26826.270456] nvme_core i2c_algo_bit [26826.273507] [26826.273509] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26826.278465] pps_core dca [26826.280738] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26826.282832] fuse [26826.286319] do_raw_spin_lock+0xb5/0xc0 [26826.288413] irq event stamp: 136474506 [26826.293719] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26826.296337] hardirqs last enabled at (136474505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26826.301209] try_get_cap_refs+0x159/0x880 [ceph] [26826.303129] hardirqs last disabled at (136474506): [] sysvec_apic_timer_interrupt+0xe/0x90 [26826.306966] __ceph_get_caps+0x4a1/0x760 [ceph] [26826.310705] softirqs last enabled at (136474504): [] __do_softirq+0x2be/0x371 [26826.315493] ? __file_remove_privs+0x89/0x110 [26826.325735] softirqs last disabled at (136474499): [] irq_exit_rcu+0xb0/0xd0 [26826.330345] ? wake_bit_function+0x70/0x70 [26826.340155] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26826.344682] ? ceph_write_iter+0x260/0x770 [ceph] [26826.353449] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26826.357802] ceph_write_iter+0x260/0x770 [ceph] [26826.366397] Workqueue: events delayed_work [ceph] [26826.370491] ? lock_is_held_type+0xaa/0x120 [26826.379867] [26826.379868] RIP: 0010:lock_acquire+0xf0/0x2b0 [26826.384567] vfs_write+0x324/0x4f0 [26826.392049] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26826.396580] ksys_pwrite64+0x65/0xa0 [26826.401273] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26826.405451] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26826.406941] [26826.406942] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26826.411294] do_syscall_64+0x38/0x80 [26826.414688] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26826.433428] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26826.436998] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26826.442216] RIP: 0033:0x7f48f0eeb93b [26826.447347] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26826.448838] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26826.455961] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26826.459532] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26826.466657] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26826.471699] ORIG_RAX: 0000000000000012 [26826.478825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26826.482394] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26826.489518] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26826.508257] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26826.515380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26826.520598] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26826.528675] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26826.532506] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26826.538244] Call Trace: [26826.538246] [26826.545368] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26826.552494] ? watchdog_timer_fn+0x1b4/0x220 [26826.559622] [26826.566742] ? __hrtimer_run_queues+0x1d3/0x360 [26826.610668] ? hrtimer_interrupt+0x104/0x240 [26826.614936] ? __local_bh_enable+0x50/0x60 [26826.619031] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26826.624253] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26826.629213] [26826.631307] [26826.633406] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26826.638716] ? lock_acquire+0xf0/0x2b0 [26826.642466] ? igrab+0x43/0x50 [26826.645523] ? lock_release+0xc5/0x260 [26826.649267] _raw_spin_lock+0x30/0x40 [26826.652924] ? igrab+0x1d/0x50 [26826.655976] igrab+0x1d/0x50 [26826.658854] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26826.664004] delayed_work+0x1d8/0x280 [ceph] [26826.668284] ? process_scheduled_works+0x22a/0x510 [26826.673075] process_scheduled_works+0x2a4/0x510 [26826.677689] worker_thread+0x188/0x340 [26826.681439] ? rescuer_thread+0x310/0x310 [26826.685441] kthread+0xf5/0x130 [26826.688579] ? kthread_complete_and_exit+0x20/0x20 [26826.693364] ret_from_fork+0x31/0x40 [26826.696934] ? kthread_complete_and_exit+0x20/0x20 [26826.701718] ret_from_fork_asm+0x11/0x20 [26826.705641] [26853.927861] watchdog: BUG: soft lockup - CPU#0 stuck for 13895s! [dbench:78683] [26853.935166] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26853.939860] watchdog: BUG: soft lockup - CPU#7 stuck for 22892s! [kworker/7:4:78808] [26853.990616] irq event stamp: 43854946 [26853.998348] Modules linked in: ceph libceph dns_resolver fscache [26854.002004] hardirqs last enabled at (43854945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26854.008009] netfs veth nft_chain_nat xt_MASQUERADE [26854.018166] hardirqs last disabled at (43854946): [] sysvec_apic_timer_interrupt+0xe/0x90 [26854.018169] nf_nat [26854.018170] softirqs last enabled at (43854938): [] __do_softirq+0x2be/0x371 [26854.023038] nf_conntrack nf_defrag_ipv6 [26854.032762] softirqs last disabled at (43854933): [] irq_exit_rcu+0xb0/0xd0 [26854.034861] nf_defrag_ipv4 [26854.043546] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26854.047462] nft_compat nf_tables [26854.055973] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26854.058761] nfnetlink bridge [26854.067708] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26854.071016] stp llc [26854.078489] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26854.081453] xfs libcrc32c [26854.086929] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26854.089113] sunrpc intel_rapl_msr [26854.107854] [26854.110554] intel_rapl_common sb_edac [26854.115773] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26854.119171] x86_pkg_temp_thermal coretemp [26854.120660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26854.124404] kvm_intel kvm [26854.131537] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26854.135629] irqbypass [26854.142754] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26854.145457] i2c_i801 [26854.152582] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26854.154939] iTCO_wdt [26854.162063] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26854.164332] iTCO_vendor_support i2c_smbus [26854.171458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26854.173728] crct10dif_pclmul joydev [26854.181806] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26854.185897] crc32_pclmul ipmi_ssif [26854.191634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26854.195205] ghash_clmulni_intel [26854.202328] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26854.205812] mei_me mei [26854.212937] Call Trace: [26854.212940] [26854.216160] lpc_ich [26854.223288] ? watchdog_timer_fn+0x1b4/0x220 [26854.225730] sha512_ssse3 acpi_ipmi [26854.228174] ? __hrtimer_run_queues+0x1d3/0x360 [26854.230185] mfd_core wmi [26854.232372] ? hrtimer_interrupt+0x104/0x240 [26854.236632] ipmi_si [26854.240117] ? __local_bh_enable+0x50/0x60 [26854.244640] ipmi_devintf ipmi_msghandler [26854.247262] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26854.251537] ioatdma acpi_pad [26854.253724] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26854.257814] acpi_power_meter [26854.261818] [26854.267036] ixgbe [26854.269998] [26854.270000] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26854.274957] igb [26854.277925] ? queued_spin_lock_slowpath+0x18/0x2e0 [26854.280019] mdio [26854.282030] do_raw_spin_lock+0xb5/0xc0 [26854.284127] crc32c_intel [26854.289433] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26854.291268] nvme ptp [26854.296140] try_get_cap_refs+0x159/0x880 [ceph] [26854.298063] nvme_core [26854.301899] __ceph_get_caps+0x4a1/0x760 [ceph] [26854.304510] i2c_algo_bit [26854.309299] ? __file_remove_privs+0x89/0x110 [26854.311567] pps_core [26854.316175] ? wake_bit_function+0x70/0x70 [26854.318550] dca fuse [26854.323077] ? ceph_write_iter+0x260/0x770 [ceph] [26854.325691] [26854.325692] irq event stamp: 136695836 [26854.330042] ceph_write_iter+0x260/0x770 [ceph] [26854.332313] hardirqs last enabled at (136695835): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26854.336410] ? lock_is_held_type+0xaa/0x120 [26854.338676] hardirqs last disabled at (136695836): [] sysvec_apic_timer_interrupt+0xe/0x90 [26854.343374] vfs_write+0x324/0x4f0 [26854.344863] softirqs last enabled at (136695834): [] __do_softirq+0x2be/0x371 [26854.348612] ksys_pwrite64+0x65/0xa0 [26854.353130] softirqs last disabled at (136695829): [] irq_exit_rcu+0xb0/0xd0 [26854.363377] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26854.367552] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26854.377365] do_syscall_64+0x38/0x80 [26854.380760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26854.389546] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26854.393112] Workqueue: events delayed_work [ceph] [26854.401711] RIP: 0033:0x7f48f0eeb93b [26854.406841] [26854.406842] RIP: 0010:lock_acquire+0xf0/0x2b0 [26854.416218] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26854.419789] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26854.427261] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26854.432303] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26854.436999] ORIG_RAX: 0000000000000012 [26854.440572] [26854.440573] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26854.442061] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26854.446413] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26854.465149] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26854.483887] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26854.489106] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26854.494324] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26854.498153] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26854.499645] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26854.506769] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26854.513892] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26854.521021] [26854.528141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26854.586883] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26854.594008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26854.601134] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26854.608267] Call Trace: [26854.610711] [26854.612722] ? watchdog_timer_fn+0x1b4/0x220 [26854.616985] ? __hrtimer_run_queues+0x1d3/0x360 [26854.621513] ? hrtimer_interrupt+0x104/0x240 [26854.625781] ? __local_bh_enable+0x50/0x60 [26854.629875] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26854.635098] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26854.640056] [26854.642152] [26854.644268] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26854.649576] ? lock_acquire+0xf0/0x2b0 [26854.653327] ? igrab+0x43/0x50 [26854.656375] ? lock_release+0xc5/0x260 [26854.660122] _raw_spin_lock+0x30/0x40 [26854.663786] ? igrab+0x1d/0x50 [26854.666835] igrab+0x1d/0x50 [26854.669715] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26854.674865] delayed_work+0x1d8/0x280 [ceph] [26854.679147] ? process_scheduled_works+0x22a/0x510 [26854.683935] process_scheduled_works+0x2a4/0x510 [26854.688553] worker_thread+0x188/0x340 [26854.692300] ? rescuer_thread+0x310/0x310 [26854.696303] kthread+0xf5/0x130 [26854.699441] ? kthread_complete_and_exit+0x20/0x20 [26854.704226] ret_from_fork+0x31/0x40 [26854.707805] ? kthread_complete_and_exit+0x20/0x20 [26854.712588] ret_from_fork_asm+0x11/0x20 [26854.716512] [26855.153248] rcu: INFO: rcu_preempt self-detected stall on CPU [26855.159011] rcu: 7-....: (14486698 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/14485401 fqs=7290310 [26855.169609] rcu: (t=14918278 jiffies g=1309609 q=450601 ncpus=8) [26855.175692] Sending NMI from CPU 7 to CPUs 0: [26855.180044] NMI backtrace for cpu 0 [26855.180045] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26855.180046] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26855.180047] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26855.180050] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26855.180051] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26855.180052] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26855.180053] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26855.180054] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26855.180055] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26855.180056] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26855.180056] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26855.180058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26855.180059] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26855.180059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26855.180060] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26855.180061] Call Trace: [26855.180061] [26855.180062] ? nmi_cpu_backtrace+0xad/0x120 [26855.180065] ? nmi_cpu_backtrace_handler+0x11/0x20 [26855.180067] ? nmi_handle+0xd4/0x260 [26855.180071] ? default_do_nmi+0x49/0xf0 [26855.180073] ? exc_nmi+0x11e/0x150 [26855.180074] ? end_repeat_nmi+0x16/0x67 [26855.180078] ? queued_spin_lock_slowpath+0x16/0x2e0 [26855.180081] ? queued_spin_lock_slowpath+0x16/0x2e0 [26855.180083] ? queued_spin_lock_slowpath+0x16/0x2e0 [26855.180085] [26855.180086] [26855.180087] do_raw_spin_lock+0xb5/0xc0 [26855.180089] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26855.180102] try_get_cap_refs+0x159/0x880 [ceph] [26855.180116] __ceph_get_caps+0x4a1/0x760 [ceph] [26855.180128] ? __file_remove_privs+0x89/0x110 [26855.180130] ? wake_bit_function+0x70/0x70 [26855.180132] ? ceph_write_iter+0x260/0x770 [ceph] [26855.180142] ceph_write_iter+0x260/0x770 [ceph] [26855.180153] ? lock_is_held_type+0xaa/0x120 [26855.180155] vfs_write+0x324/0x4f0 [26855.180159] ksys_pwrite64+0x65/0xa0 [26855.180161] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26855.180164] do_syscall_64+0x38/0x80 [26855.180166] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26855.180168] RIP: 0033:0x7f48f0eeb93b [26855.180169] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26855.180170] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [26855.180171] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26855.180172] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26855.180172] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26855.180173] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26855.180174] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26855.180176] [26855.181042] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26855.484657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26855.492127] Workqueue: events delayed_work [ceph] [26855.496849] RIP: 0010:lock_release+0x14e/0x260 [26855.501288] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [26855.520025] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [26855.525242] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [26855.532367] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [26855.539489] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [26855.546614] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [26855.553738] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [26855.560864] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26855.568939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26855.574677] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26855.581803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26855.588927] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26855.596051] Call Trace: [26855.598495] [26855.600507] ? rcu_dump_cpu_stacks+0xd1/0x180 [26855.604867] ? rcu_sched_clock_irq+0x633/0x1390 [26855.609398] ? lock_release+0xc5/0x260 [26855.613143] ? lock_is_held_type+0xaa/0x120 [26855.617331] ? lock_is_held_type+0xaa/0x120 [26855.621516] ? tick_sched_do_timer+0xa0/0xa0 [26855.625786] ? rcu_read_lock_sched_held+0x56/0x90 [26855.630486] ? tick_sched_do_timer+0xa0/0xa0 [26855.634757] ? update_process_times+0x5f/0x90 [26855.639108] ? tick_sched_handle.isra.16+0x1e/0x50 [26855.643892] ? tick_sched_timer+0x6f/0x80 [26855.647897] ? __hrtimer_run_queues+0x1d3/0x360 [26855.652436] ? hrtimer_interrupt+0x104/0x240 [26855.656702] ? __local_bh_enable+0x50/0x60 [26855.660806] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26855.666027] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26855.670983] [26855.673082] [26855.675178] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26855.680484] ? igrab+0x43/0x50 [26855.683567] ? lock_release+0x14e/0x260 [26855.687401] ? lock_release+0xc5/0x260 [26855.691158] _raw_spin_unlock+0x1b/0x40 [26855.694993] igrab+0x43/0x50 [26855.697869] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26855.703021] delayed_work+0x1d8/0x280 [ceph] [26855.707313] ? process_scheduled_works+0x22a/0x510 [26855.712100] process_scheduled_works+0x2a4/0x510 [26855.716720] worker_thread+0x188/0x340 [26855.720475] ? rescuer_thread+0x310/0x310 [26855.724486] kthread+0xf5/0x130 [26855.727631] ? kthread_complete_and_exit+0x20/0x20 [26855.732426] ret_from_fork+0x31/0x40 [26855.736003] ? kthread_complete_and_exit+0x20/0x20 [26855.740789] ret_from_fork_asm+0x11/0x20 [26855.744727] [26881.928350] watchdog: BUG: soft lockup - CPU#0 stuck for 13921s! [dbench:78683] [26881.935653] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26881.940350] watchdog: BUG: soft lockup - CPU#7 stuck for 22918s! [kworker/7:4:78808] [26881.991095] irq event stamp: 43913252 [26881.991096] hardirqs last enabled at (43913251): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26881.998827] Modules linked in: [26882.002482] hardirqs last disabled at (43913252): [] sysvec_apic_timer_interrupt+0xe/0x90 [26882.012639] ceph [26882.015691] softirqs last enabled at (43913188): [] __do_softirq+0x2be/0x371 [26882.025414] libceph [26882.027338] softirqs last disabled at (43913183): [] irq_exit_rcu+0xb0/0xd0 [26882.036033] dns_resolver [26882.038217] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26882.046744] fscache netfs [26882.049362] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26882.058305] veth [26882.061024] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [26882.068496] nft_chain_nat [26882.070422] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [26882.075900] xt_MASQUERADE [26882.078603] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26882.097340] nf_nat [26882.100045] [26882.100046] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26882.105263] nf_conntrack [26882.107360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26882.108851] nf_defrag_ipv6 nf_defrag_ipv4 [26882.115976] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26882.118593] nft_compat nf_tables [26882.125717] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26882.129806] nfnetlink bridge [26882.136930] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26882.140243] stp llc [26882.147367] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26882.150330] xfs libcrc32c [26882.157453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26882.159639] sunrpc intel_rapl_msr [26882.167714] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26882.170420] intel_rapl_common sb_edac [26882.176157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26882.179554] x86_pkg_temp_thermal coretemp [26882.186679] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26882.190422] kvm_intel kvm [26882.197548] Call Trace: [26882.197549] [26882.201639] irqbypass [26882.208762] ? watchdog_timer_fn+0x1b4/0x220 [26882.211466] i2c_i801 iTCO_wdt [26882.213911] ? __hrtimer_run_queues+0x1d3/0x360 [26882.215920] iTCO_vendor_support [26882.218280] ? hrtimer_interrupt+0x104/0x240 [26882.222542] i2c_smbus [26882.225593] ? __local_bh_enable+0x50/0x60 [26882.230117] crct10dif_pclmul [26882.233344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26882.237605] joydev crc32_pclmul [26882.239964] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26882.244054] ipmi_ssif ghash_clmulni_intel [26882.247028] [26882.252244] mei_me mei [26882.255467] [26882.255469] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26882.260425] lpc_ich sha512_ssse3 [26882.264520] ? queued_spin_lock_slowpath+0x2d/0x2e0 [26882.266613] acpi_ipmi mfd_core [26882.269059] do_raw_spin_lock+0xb5/0xc0 [26882.271153] wmi ipmi_si [26882.276459] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26882.279769] ipmi_devintf ipmi_msghandler [26882.284642] try_get_cap_refs+0x159/0x880 [ceph] [26882.287777] ioatdma acpi_pad [26882.291613] __ceph_get_caps+0x4a1/0x760 [ceph] [26882.294138] acpi_power_meter ixgbe [26882.298927] ? __file_remove_privs+0x89/0x110 [26882.302928] igb mdio [26882.307537] ? wake_bit_function+0x70/0x70 [26882.310503] crc32c_intel [26882.315046] ? ceph_write_iter+0x260/0x770 [ceph] [26882.318526] nvme ptp [26882.322878] ceph_write_iter+0x260/0x770 [ceph] [26882.325149] nvme_core [26882.329242] ? lock_is_held_type+0xaa/0x120 [26882.331856] i2c_algo_bit pps_core [26882.336557] vfs_write+0x324/0x4f0 [26882.338824] dca fuse [26882.343352] ksys_pwrite64+0x65/0xa0 [26882.345705] [26882.345706] irq event stamp: 136911976 [26882.349884] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26882.353281] hardirqs last enabled at (136911975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26882.356680] do_syscall_64+0x38/0x80 [26882.358949] hardirqs last disabled at (136911976): [] sysvec_apic_timer_interrupt+0xe/0x90 [26882.362520] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26882.364028] softirqs last enabled at (136911974): [] __do_softirq+0x2be/0x371 [26882.367772] RIP: 0033:0x7f48f0eeb93b [26882.372904] softirqs last disabled at (136911969): [] irq_exit_rcu+0xb0/0xd0 [26882.383148] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26882.386717] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26882.396528] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26882.401572] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26882.410343] ORIG_RAX: 0000000000000012 [26882.413915] Workqueue: events delayed_work [ceph] [26882.422511] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26882.441251] [26882.441252] RIP: 0010:lock_acquire+0x86/0x2b0 [26882.450628] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26882.455844] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [26882.463314] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26882.467146] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [26882.471844] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26882.478966] [26882.478967] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [26882.480458] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26882.484809] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [26882.491937] [26882.510670] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [26882.562343] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [26882.569467] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26882.576591] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26882.584669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26882.590406] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26882.597530] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26882.604654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26882.611780] Call Trace: [26882.614223] [26882.616233] ? watchdog_timer_fn+0x1b4/0x220 [26882.620498] ? __hrtimer_run_queues+0x1d3/0x360 [26882.625025] ? hrtimer_interrupt+0x104/0x240 [26882.629312] ? __local_bh_enable+0x50/0x60 [26882.633403] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26882.638619] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26882.643576] [26882.645673] [26882.647771] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26882.653080] ? lock_acquire+0x86/0x2b0 [26882.656829] ? igrab+0x43/0x50 [26882.659881] ? lock_release+0xc5/0x260 [26882.663625] _raw_spin_lock+0x30/0x40 [26882.667281] ? igrab+0x1d/0x50 [26882.670331] igrab+0x1d/0x50 [26882.673211] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26882.678359] delayed_work+0x1d8/0x280 [ceph] [26882.682641] ? process_scheduled_works+0x22a/0x510 [26882.687431] process_scheduled_works+0x2a4/0x510 [26882.692047] worker_thread+0x188/0x340 [26882.695797] ? rescuer_thread+0x310/0x310 [26882.699809] kthread+0xf5/0x130 [26882.702947] ? kthread_complete_and_exit+0x20/0x20 [26882.707755] ret_from_fork+0x31/0x40 [26882.711326] ? kthread_complete_and_exit+0x20/0x20 [26882.716109] ret_from_fork_asm+0x11/0x20 [26882.720048] [26909.928842] watchdog: BUG: soft lockup - CPU#0 stuck for 13947s! [dbench:78683] [26909.936151] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26909.940840] watchdog: BUG: soft lockup - CPU#7 stuck for 22944s! [kworker/7:4:78808] [26909.991601] irq event stamp: 43971522 [26909.999330] Modules linked in: ceph libceph dns_resolver fscache netfs [26910.002989] hardirqs last enabled at (43971521): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26910.009525] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [26910.019679] hardirqs last disabled at (43971522): [] sysvec_apic_timer_interrupt+0xe/0x90 [26910.019682] nf_defrag_ipv6 [26910.019683] softirqs last enabled at (43971516): [] __do_softirq+0x2be/0x371 [26910.025763] nf_defrag_ipv4 [26910.035489] softirqs last disabled at (43971511): [] irq_exit_rcu+0xb0/0xd0 [26910.038279] nft_compat [26910.046964] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26910.049753] nf_tables nfnetlink [26910.058264] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26910.060707] bridge stp [26910.069652] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26910.072878] llc [26910.080347] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26910.082793] xfs libcrc32c [26910.088268] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26910.090108] sunrpc intel_rapl_msr [26910.108845] [26910.111547] intel_rapl_common [26910.116764] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26910.120162] sb_edac x86_pkg_temp_thermal [26910.121652] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26910.124704] coretemp kvm_intel [26910.131829] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26910.135832] kvm irqbypass [26910.142957] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26910.146093] i2c_i801 iTCO_wdt [26910.153236] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26910.155940] iTCO_vendor_support [26910.163063] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26910.166114] i2c_smbus crct10dif_pclmul [26910.173237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26910.176461] joydev crc32_pclmul [26910.184540] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26910.188370] ipmi_ssif ghash_clmulni_intel [26910.194108] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26910.197333] mei_me mei [26910.204456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26910.208547] lpc_ich [26910.215671] Call Trace: [26910.218115] sha512_ssse3 acpi_ipmi [26910.225241] [26910.227424] mfd_core wmi [26910.229870] ? watchdog_timer_fn+0x1b4/0x220 [26910.233352] ipmi_si ipmi_devintf [26910.235364] ? __hrtimer_run_queues+0x1d3/0x360 [26910.237980] ipmi_msghandler ioatdma [26910.242246] ? hrtimer_interrupt+0x104/0x240 [26910.245554] acpi_pad acpi_power_meter [26910.250079] ? __local_bh_enable+0x50/0x60 [26910.253648] ixgbe igb [26910.257917] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26910.261658] mdio crc32c_intel [26910.265748] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26910.268105] nvme ptp [26910.273323] [26910.276376] nvme_core [26910.281330] [26910.283602] i2c_algo_bit pps_core [26910.285700] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26910.288056] dca fuse [26910.290159] ? queued_spin_lock_slowpath+0x16/0x2e0 [26910.293551] [26910.293552] irq event stamp: 137132804 [26910.298858] do_raw_spin_lock+0xb5/0xc0 [26910.301126] hardirqs last enabled at (137132803): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26910.305999] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26910.307487] hardirqs last disabled at (137132804): [] sysvec_apic_timer_interrupt+0xe/0x90 [26910.311243] try_get_cap_refs+0x159/0x880 [ceph] [26910.315071] softirqs last enabled at (137132802): [] __do_softirq+0x2be/0x371 [26910.325321] __ceph_get_caps+0x4a1/0x760 [ceph] [26910.330100] softirqs last disabled at (137132797): [] irq_exit_rcu+0xb0/0xd0 [26910.339914] ? __file_remove_privs+0x89/0x110 [26910.344527] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26910.353300] ? wake_bit_function+0x70/0x70 [26910.357824] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26910.366425] ? ceph_write_iter+0x260/0x770 [ceph] [26910.370773] Workqueue: events delayed_work [ceph] [26910.380150] ceph_write_iter+0x260/0x770 [ceph] [26910.384243] [26910.384243] RIP: 0010:lock_acquire+0xf0/0x2b0 [26910.391717] ? lock_is_held_type+0xaa/0x120 [26910.396410] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26910.401108] vfs_write+0x324/0x4f0 [26910.405631] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26910.407125] ksys_pwrite64+0x65/0xa0 [26910.411472] [26910.415651] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26910.434388] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26910.437787] do_syscall_64+0x38/0x80 [26910.443003] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26910.446574] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26910.448064] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26910.453195] RIP: 0033:0x7f48f0eeb93b [26910.460318] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26910.463890] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26910.471013] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26910.476058] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26910.483182] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26910.486752] ORIG_RAX: 0000000000000012 [26910.493876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26910.512615] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26910.519737] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26910.524955] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26910.533032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26910.536863] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26910.542601] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26910.549726] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26910.556850] Call Trace: [26910.563973] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26910.571099] [26910.578246] [26910.585364] ? watchdog_timer_fn+0x1b4/0x220 [26910.610516] ? __hrtimer_run_queues+0x1d3/0x360 [26910.615061] ? hrtimer_interrupt+0x104/0x240 [26910.619330] ? __local_bh_enable+0x50/0x60 [26910.623423] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26910.628645] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26910.633603] [26910.635702] [26910.637798] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26910.643106] ? lock_acquire+0xf0/0x2b0 [26910.646855] ? igrab+0x43/0x50 [26910.649907] ? lock_release+0xc5/0x260 [26910.653651] _raw_spin_lock+0x30/0x40 [26910.657307] ? igrab+0x1d/0x50 [26910.660357] igrab+0x1d/0x50 [26910.663250] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26910.668395] delayed_work+0x1d8/0x280 [ceph] [26910.672675] ? process_scheduled_works+0x22a/0x510 [26910.677467] process_scheduled_works+0x2a4/0x510 [26910.682083] worker_thread+0x188/0x340 [26910.685832] ? rescuer_thread+0x310/0x310 [26910.689835] kthread+0xf5/0x130 [26910.692972] ? kthread_complete_and_exit+0x20/0x20 [26910.697756] ret_from_fork+0x31/0x40 [26910.701335] ? kthread_complete_and_exit+0x20/0x20 [26910.706120] ret_from_fork_asm+0x11/0x20 [26910.710050] [26937.929331] watchdog: BUG: soft lockup - CPU#0 stuck for 13973s! [dbench:78683] [26937.936627] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26937.941331] watchdog: BUG: soft lockup - CPU#7 stuck for 22970s! [kworker/7:4:78808] [26937.992068] irq event stamp: 44029604 [26937.992069] hardirqs last enabled at (44029603): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26937.999799] Modules linked in: [26938.003457] hardirqs last disabled at (44029604): [] sysvec_apic_timer_interrupt+0xe/0x90 [26938.013615] ceph [26938.016666] softirqs last enabled at (44029526): [] __do_softirq+0x2be/0x371 [26938.026390] libceph [26938.028313] softirqs last disabled at (44029521): [] irq_exit_rcu+0xb0/0xd0 [26938.037013] dns_resolver [26938.039199] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26938.047726] fscache [26938.050344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26938.059289] netfs veth [26938.061474] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [26938.068945] nft_chain_nat xt_MASQUERADE [26938.071389] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [26938.076866] nf_nat nf_conntrack [26938.080782] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26938.099519] nf_defrag_ipv6 nf_defrag_ipv4 [26938.102745] [26938.102746] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26938.107961] nft_compat nf_tables [26938.112052] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26938.113543] nfnetlink [26938.120667] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26938.123978] bridge stp [26938.131102] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26938.133459] llc xfs [26938.140584] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26938.143028] libcrc32c [26938.150152] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26938.152336] sunrpc [26938.159460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26938.161817] intel_rapl_msr [26938.169895] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26938.172008] intel_rapl_common sb_edac [26938.177747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26938.180539] x86_pkg_temp_thermal coretemp [26938.187662] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26938.191407] kvm_intel [26938.198532] Call Trace: [26938.198533] [26938.202620] kvm [26938.209746] ? watchdog_timer_fn+0x1b4/0x220 [26938.212104] irqbypass i2c_i801 [26938.214548] ? __hrtimer_run_queues+0x1d3/0x360 [26938.216556] iTCO_wdt iTCO_vendor_support [26938.218398] ? hrtimer_interrupt+0x104/0x240 [26938.222659] i2c_smbus [26938.225797] ? __local_bh_enable+0x50/0x60 [26938.230322] crct10dif_pclmul joydev [26938.234327] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26938.238589] crc32_pclmul [26938.240947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26938.245037] ipmi_ssif ghash_clmulni_intel [26938.248607] [26938.253825] mei_me mei [26938.256442] [26938.256443] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26938.261401] lpc_ich sha512_ssse3 [26938.265495] ? queued_spin_lock_slowpath+0x18/0x2e0 [26938.267587] acpi_ipmi mfd_core [26938.270035] do_raw_spin_lock+0xb5/0xc0 [26938.272130] wmi ipmi_si [26938.277436] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26938.280744] ipmi_devintf [26938.285618] try_get_cap_refs+0x159/0x880 [ceph] [26938.288752] ipmi_msghandler [26938.292588] __ceph_get_caps+0x4a1/0x760 [ceph] [26938.295113] ioatdma acpi_pad [26938.299903] ? __file_remove_privs+0x89/0x110 [26938.302516] acpi_power_meter ixgbe [26938.307127] ? wake_bit_function+0x70/0x70 [26938.310010] igb mdio [26938.314538] ? ceph_write_iter+0x260/0x770 [ceph] [26938.317499] crc32c_intel nvme [26938.321850] ceph_write_iter+0x260/0x770 [ceph] [26938.325336] ptp nvme_core [26938.329430] ? lock_is_held_type+0xaa/0x120 [26938.331696] i2c_algo_bit [26938.336395] vfs_write+0x324/0x4f0 [26938.339444] pps_core dca [26938.343972] ksys_pwrite64+0x65/0xa0 [26938.346673] fuse [26938.350850] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26938.353468] irq event stamp: 137353798 [26938.356866] do_syscall_64+0x38/0x80 [26938.359481] hardirqs last enabled at (137353797): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26938.363053] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26938.364977] hardirqs last disabled at (137353798): [] sysvec_apic_timer_interrupt+0xe/0x90 [26938.370108] RIP: 0033:0x7f48f0eeb93b [26938.373851] softirqs last enabled at (137353796): [] __do_softirq+0x2be/0x371 [26938.377423] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26938.387667] softirqs last disabled at (137353791): [] irq_exit_rcu+0xb0/0xd0 [26938.392728] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26938.402539] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26938.406111] ORIG_RAX: 0000000000000012 [26938.414880] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26938.433618] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26938.442215] Workqueue: events delayed_work [ceph] [26938.447433] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26938.456812] [26938.460642] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26938.468112] RIP: 0010:lock_acquire+0xf0/0x2b0 [26938.475237] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26938.479933] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26938.487057] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26938.488549] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26938.495678] [26938.500023] [26938.500024] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26938.549034] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26938.556159] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26938.563282] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26938.570406] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26938.577531] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26938.585607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26938.591347] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26938.598471] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26938.605594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26938.612736] Call Trace: [26938.615179] [26938.617193] ? watchdog_timer_fn+0x1b4/0x220 [26938.621455] ? __hrtimer_run_queues+0x1d3/0x360 [26938.625981] ? hrtimer_interrupt+0x104/0x240 [26938.630253] ? __local_bh_enable+0x50/0x60 [26938.634344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26938.639571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26938.644527] [26938.646622] [26938.648737] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26938.654047] ? lock_acquire+0xf0/0x2b0 [26938.657795] ? igrab+0x43/0x50 [26938.660847] ? lock_release+0xc5/0x260 [26938.664590] _raw_spin_lock+0x30/0x40 [26938.668247] ? igrab+0x1d/0x50 [26938.671298] igrab+0x1d/0x50 [26938.674177] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26938.679325] delayed_work+0x1d8/0x280 [ceph] [26938.683607] ? process_scheduled_works+0x22a/0x510 [26938.688396] process_scheduled_works+0x2a4/0x510 [26938.693021] worker_thread+0x188/0x340 [26938.696770] ? rescuer_thread+0x310/0x310 [26938.700773] kthread+0xf5/0x130 [26938.703912] ? kthread_complete_and_exit+0x20/0x20 [26938.708696] ret_from_fork+0x31/0x40 [26938.712274] ? kthread_complete_and_exit+0x20/0x20 [26938.717060] ret_from_fork_asm+0x11/0x20 [26938.720983] [26965.929822] watchdog: BUG: soft lockup - CPU#0 stuck for 13999s! [dbench:78683] [26965.937124] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26965.941822] watchdog: BUG: soft lockup - CPU#7 stuck for 22996s! [kworker/7:4:78808] [26965.992567] irq event stamp: 44087942 [26965.992568] hardirqs last enabled at (44087941): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26966.000298] Modules linked in: [26966.003953] hardirqs last disabled at (44087942): [] sysvec_apic_timer_interrupt+0xe/0x90 [26966.014113] ceph [26966.017163] softirqs last enabled at (44087916): [] __do_softirq+0x2be/0x371 [26966.026887] libceph [26966.028811] softirqs last disabled at (44087911): [] irq_exit_rcu+0xb0/0xd0 [26966.037495] dns_resolver [26966.039679] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26966.048190] fscache netfs [26966.050823] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26966.059769] veth nft_chain_nat [26966.062473] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26966.069961] xt_MASQUERADE [26966.073097] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26966.078576] nf_nat nf_conntrack [26966.081279] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26966.100016] nf_defrag_ipv6 [26966.103242] [26966.103243] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26966.108459] nf_defrag_ipv4 nft_compat [26966.111250] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26966.112739] nf_tables nfnetlink [26966.119864] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26966.123608] bridge stp [26966.130731] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26966.133955] llc xfs [26966.141081] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26966.143524] libcrc32c sunrpc [26966.150649] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26966.152833] intel_rapl_msr [26966.159957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26966.162922] intel_rapl_common sb_edac [26966.170998] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26966.173789] x86_pkg_temp_thermal [26966.179527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26966.183270] coretemp kvm_intel [26966.190394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26966.193707] kvm [26966.200830] Call Trace: [26966.200831] [26966.203966] irqbypass [26966.211092] ? watchdog_timer_fn+0x1b4/0x220 [26966.212929] i2c_i801 iTCO_wdt [26966.215372] ? __hrtimer_run_queues+0x1d3/0x360 [26966.217383] iTCO_vendor_support i2c_smbus [26966.219745] ? hrtimer_interrupt+0x104/0x240 [26966.224005] crct10dif_pclmul joydev [26966.227056] ? __local_bh_enable+0x50/0x60 [26966.231579] crc32_pclmul [26966.235672] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26966.239933] ipmi_ssif ghash_clmulni_intel [26966.243505] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26966.247596] mei_me [26966.250220] [26966.255439] mei lpc_ich [26966.259529] [26966.259531] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26966.264499] sha512_ssse3 [26966.266598] ? queued_spin_lock_slowpath+0x16/0x2e0 [26966.268690] acpi_ipmi mfd_core [26966.271224] do_raw_spin_lock+0xb5/0xc0 [26966.273319] wmi [26966.278622] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26966.281239] ipmi_si ipmi_devintf [26966.286113] try_get_cap_refs+0x159/0x880 [ceph] [26966.289248] ipmi_msghandler ioatdma [26966.293085] __ceph_get_caps+0x4a1/0x760 [ceph] [26966.294916] acpi_pad acpi_power_meter [26966.299705] ? __file_remove_privs+0x89/0x110 [26966.303010] ixgbe igb [26966.307623] ? wake_bit_function+0x70/0x70 [26966.311192] mdio [26966.315719] ? ceph_write_iter+0x260/0x770 [ceph] [26966.319462] crc32c_intel nvme [26966.323812] ceph_write_iter+0x260/0x770 [ceph] [26966.326170] ptp nvme_core [26966.330263] ? lock_is_held_type+0xaa/0x120 [26966.332183] i2c_algo_bit [26966.336884] vfs_write+0x324/0x4f0 [26966.339932] pps_core dca [26966.344461] ksys_pwrite64+0x65/0xa0 [26966.347159] fuse [26966.351339] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26966.353956] irq event stamp: 137574634 [26966.357353] do_syscall_64+0x38/0x80 [26966.359969] hardirqs last enabled at (137574633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26966.363541] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26966.365464] hardirqs last disabled at (137574634): [] sysvec_apic_timer_interrupt+0xe/0x90 [26966.370595] RIP: 0033:0x7f48f0eeb93b [26966.374340] softirqs last enabled at (137574632): [] __do_softirq+0x2be/0x371 [26966.377909] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26966.388154] softirqs last disabled at (137574627): [] irq_exit_rcu+0xb0/0xd0 [26966.393198] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26966.403027] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26966.406596] ORIG_RAX: 0000000000000012 [26966.415368] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26966.415369] Workqueue: events delayed_work [ceph] [26966.434107] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26966.442702] [26966.442703] RIP: 0010:lock_acquire+0xf0/0x2b0 [26966.447920] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26966.457299] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26966.461128] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26966.468599] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26966.473298] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26966.480420] [26966.480421] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26966.481911] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26966.486263] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26966.493393] [26966.512124] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26966.563779] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26966.570902] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26966.578027] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26966.586104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26966.591841] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26966.598965] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26966.606091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26966.613230] Call Trace: [26966.615676] [26966.617687] ? watchdog_timer_fn+0x1b4/0x220 [26966.621951] ? __hrtimer_run_queues+0x1d3/0x360 [26966.626478] ? hrtimer_interrupt+0x104/0x240 [26966.630749] ? __local_bh_enable+0x50/0x60 [26966.634842] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26966.640066] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26966.645022] [26966.647120] [26966.649226] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26966.654534] ? lock_acquire+0xf0/0x2b0 [26966.658283] ? igrab+0x43/0x50 [26966.661333] ? lock_release+0xc5/0x260 [26966.665079] _raw_spin_lock+0x30/0x40 [26966.668736] ? igrab+0x1d/0x50 [26966.671785] igrab+0x1d/0x50 [26966.674665] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26966.679814] delayed_work+0x1d8/0x280 [ceph] [26966.684095] ? process_scheduled_works+0x22a/0x510 [26966.688886] process_scheduled_works+0x2a4/0x510 [26966.693517] worker_thread+0x188/0x340 [26966.697268] ? rescuer_thread+0x310/0x310 [26966.701278] kthread+0xf5/0x130 [26966.704415] ? kthread_complete_and_exit+0x20/0x20 [26966.709202] ret_from_fork+0x31/0x40 [26966.712797] ? kthread_complete_and_exit+0x20/0x20 [26966.717582] ret_from_fork_asm+0x11/0x20 [26966.721504] [26993.930311] watchdog: BUG: soft lockup - CPU#0 stuck for 14026s! [dbench:78683] [26993.937613] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [26993.942311] watchdog: BUG: soft lockup - CPU#7 stuck for 23023s! [kworker/7:4:78808] [26993.993064] irq event stamp: 44146276 [26994.000794] Modules linked in: ceph libceph dns_resolver fscache netfs veth [26994.004449] hardirqs last enabled at (44146275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26994.011410] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [26994.021568] hardirqs last disabled at (44146276): [] sysvec_apic_timer_interrupt+0xe/0x90 [26994.027218] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [26994.036943] softirqs last enabled at (44146270): [] __do_softirq+0x2be/0x371 [26994.036947] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [26994.052413] softirqs last disabled at (44146265): [] irq_exit_rcu+0xb0/0xd0 [26994.059891] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [26994.068403] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26994.074400] irqbypass i2c_i801 iTCO_wdt [26994.083344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26994.083346] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [26994.087262] iTCO_vendor_support [26994.094733] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [26994.100209] i2c_smbus [26994.103435] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [26994.122171] crct10dif_pclmul [26994.124531] [26994.129747] joydev [26994.132710] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [26994.134202] crc32_pclmul [26994.136299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [26994.143422] ipmi_ssif [26994.146040] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [26994.153163] ghash_clmulni_intel [26994.155521] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [26994.162646] mei_me [26994.165871] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [26994.172993] mei lpc_ich [26994.175091] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [26994.182216] sha512_ssse3 [26994.184747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26994.192823] acpi_ipmi mfd_core [26994.195441] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [26994.201179] wmi ipmi_si [26994.204317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26994.211439] ipmi_devintf [26994.213988] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26994.221111] ipmi_msghandler [26994.223729] Call Trace: [26994.223732] [26994.230854] ioatdma acpi_pad [26994.233734] ? watchdog_timer_fn+0x1b4/0x220 [26994.236177] acpi_power_meter [26994.238188] ? __hrtimer_run_queues+0x1d3/0x360 [26994.241149] ixgbe igb [26994.245417] ? hrtimer_interrupt+0x104/0x240 [26994.248378] mdio [26994.252903] ? __local_bh_enable+0x50/0x60 [26994.255260] crc32c_intel [26994.259526] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26994.261448] nvme [26994.265540] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26994.268156] ptp nvme_core [26994.273373] [26994.275297] i2c_algo_bit pps_core [26994.280255] [26994.282957] dca fuse [26994.285057] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26994.288452] [26994.288453] irq event stamp: 137795422 [26994.290554] ? queued_spin_lock_slowpath+0x16/0x2e0 [26994.292822] hardirqs last enabled at (137795421): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26994.298127] do_raw_spin_lock+0xb5/0xc0 [26994.299615] hardirqs last disabled at (137795422): [] sysvec_apic_timer_interrupt+0xe/0x90 [26994.303361] __cap_delay_requeue+0xa7/0x1d0 [ceph] [26994.308232] softirqs last enabled at (137795420): [] __do_softirq+0x2be/0x371 [26994.318477] try_get_cap_refs+0x159/0x880 [ceph] [26994.322305] softirqs last disabled at (137795415): [] irq_exit_rcu+0xb0/0xd0 [26994.332121] __ceph_get_caps+0x4a1/0x760 [ceph] [26994.336902] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [26994.345676] ? __file_remove_privs+0x89/0x110 [26994.350283] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [26994.358881] ? wake_bit_function+0x70/0x70 [26994.363406] Workqueue: events delayed_work [ceph] [26994.372785] ? ceph_write_iter+0x260/0x770 [ceph] [26994.377131] [26994.377132] RIP: 0010:lock_acquire+0xf0/0x2b0 [26994.384603] ceph_write_iter+0x260/0x770 [ceph] [26994.388710] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [26994.393414] ? lock_is_held_type+0xaa/0x120 [26994.398106] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [26994.399600] vfs_write+0x324/0x4f0 [26994.403947] [26994.403947] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [26994.408475] ksys_pwrite64+0x65/0xa0 [26994.427209] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [26994.431387] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [26994.436605] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [26994.440002] do_syscall_64+0x38/0x80 [26994.441491] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [26994.448616] entry_SYSCALL_64_after_hwframe+0x63/0xcd [26994.452187] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [26994.459312] RIP: 0033:0x7f48f0eeb93b [26994.464442] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [26994.471566] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [26994.475136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26994.482261] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [26994.487304] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [26994.494429] ORIG_RAX: 0000000000000012 [26994.498001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26994.506078] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [26994.524815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26994.530553] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [26994.535769] Call Trace: [26994.542893] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [26994.546726] [26994.553848] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [26994.560973] ? watchdog_timer_fn+0x1b4/0x220 [26994.568114] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [26994.575241] ? __hrtimer_run_queues+0x1d3/0x360 [26994.577687] [26994.584811] ? hrtimer_interrupt+0x104/0x240 [26994.616303] ? __local_bh_enable+0x50/0x60 [26994.620397] ? __sysvec_apic_timer_interrupt+0x61/0x160 [26994.625620] ? sysvec_apic_timer_interrupt+0x6d/0x90 [26994.630578] [26994.632675] [26994.634772] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [26994.640080] ? lock_acquire+0xf0/0x2b0 [26994.643831] ? igrab+0x43/0x50 [26994.646887] ? lock_release+0xc5/0x260 [26994.650634] _raw_spin_lock+0x30/0x40 [26994.654299] ? igrab+0x1d/0x50 [26994.657349] igrab+0x1d/0x50 [26994.660229] ceph_check_delayed_caps+0xb7/0x270 [ceph] [26994.665378] delayed_work+0x1d8/0x280 [ceph] [26994.669659] ? process_scheduled_works+0x22a/0x510 [26994.674449] process_scheduled_works+0x2a4/0x510 [26994.679063] worker_thread+0x188/0x340 [26994.682815] ? rescuer_thread+0x310/0x310 [26994.686824] kthread+0xf5/0x130 [26994.689962] ? kthread_complete_and_exit+0x20/0x20 [26994.694749] ret_from_fork+0x31/0x40 [26994.698325] ? kthread_complete_and_exit+0x20/0x20 [26994.703111] ret_from_fork_asm+0x11/0x20 [26994.707043] [27021.930801] watchdog: BUG: soft lockup - CPU#0 stuck for 14052s! [dbench:78683] [27021.938100] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27021.942801] watchdog: BUG: soft lockup - CPU#7 stuck for 23049s! [kworker/7:4:78808] [27021.993542] irq event stamp: 44204480 [27021.993543] hardirqs last enabled at (44204479): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27022.001271] Modules linked in: [27022.004929] hardirqs last disabled at (44204480): [] sysvec_apic_timer_interrupt+0xe/0x90 [27022.015086] ceph [27022.018137] softirqs last enabled at (44204402): [] __do_softirq+0x2be/0x371 [27022.027861] libceph [27022.029784] softirqs last disabled at (44204397): [] irq_exit_rcu+0xb0/0xd0 [27022.038485] dns_resolver fscache [27022.040672] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27022.049197] netfs veth [27022.052510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27022.061453] nft_chain_nat xt_MASQUERADE [27022.063897] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27022.071368] nf_nat nf_conntrack [27022.075287] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27022.080764] nf_defrag_ipv6 nf_defrag_ipv4 [27022.083987] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27022.102725] nft_compat nf_tables [27022.106816] [27022.106817] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27022.112035] nfnetlink bridge [27022.115345] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27022.116834] stp llc [27022.123959] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27022.126923] xfs [27022.134048] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27022.136231] libcrc32c [27022.143355] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27022.145197] sunrpc [27022.152327] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27022.154684] intel_rapl_msr intel_rapl_common [27022.161808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27022.163904] sb_edac [27022.171983] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27022.176333] x86_pkg_temp_thermal [27022.182070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27022.184255] coretemp kvm_intel [27022.191377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27022.194690] kvm irqbypass [27022.201813] Call Trace: [27022.201814] [27022.204950] i2c_i801 [27022.212076] ? watchdog_timer_fn+0x1b4/0x220 [27022.214780] iTCO_wdt iTCO_vendor_support [27022.217225] ? __hrtimer_run_queues+0x1d3/0x360 [27022.219233] i2c_smbus crct10dif_pclmul [27022.221508] ? hrtimer_interrupt+0x104/0x240 [27022.225768] joydev crc32_pclmul [27022.229771] ? __local_bh_enable+0x50/0x60 [27022.234296] ipmi_ssif ghash_clmulni_intel [27022.238131] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27022.242392] mei_me mei [27022.245615] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27022.249705] lpc_ich [27022.253797] [27022.259013] sha512_ssse3 [27022.261458] [27022.266432] acpi_ipmi mfd_core [27022.268618] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27022.270713] wmi ipmi_si [27022.273335] ? queued_spin_lock_slowpath+0x16/0x2e0 [27022.275429] ipmi_devintf ipmi_msghandler [27022.278569] do_raw_spin_lock+0xb5/0xc0 [27022.283870] ioatdma acpi_pad [27022.286401] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27022.291272] acpi_power_meter ixgbe [27022.295279] try_get_cap_refs+0x159/0x880 [ceph] [27022.299108] igb mdio [27022.302076] __ceph_get_caps+0x4a1/0x760 [ceph] [27022.306855] crc32c_intel nvme [27022.310343] ? __file_remove_privs+0x89/0x110 [27022.314949] ptp nvme_core [27022.317220] ? wake_bit_function+0x70/0x70 [27022.321745] i2c_algo_bit pps_core [27022.324799] ? ceph_write_iter+0x260/0x770 [ceph] [27022.329146] dca fuse [27022.331851] ceph_write_iter+0x260/0x770 [ceph] [27022.335942] [27022.335943] irq event stamp: 138016538 [27022.339342] ? lock_is_held_type+0xaa/0x120 [27022.344035] hardirqs last enabled at (138016537): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27022.346310] vfs_write+0x324/0x4f0 [27022.350830] hardirqs last disabled at (138016538): [] sysvec_apic_timer_interrupt+0xe/0x90 [27022.352325] ksys_pwrite64+0x65/0xa0 [27022.356064] softirqs last enabled at (138016536): [] __do_softirq+0x2be/0x371 [27022.360244] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27022.370491] softirqs last disabled at (138016531): [] irq_exit_rcu+0xb0/0xd0 [27022.373885] do_syscall_64+0x38/0x80 [27022.383695] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27022.387268] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27022.396037] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27022.401167] RIP: 0033:0x7f48f0eeb93b [27022.409765] Workqueue: events delayed_work [ceph] [27022.413335] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27022.422713] [27022.422714] RIP: 0010:lock_acquire+0xf0/0x2b0 [27022.427758] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27022.435229] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27022.438799] ORIG_RAX: 0000000000000012 [27022.443497] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27022.462235] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27022.463726] [27022.463727] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27022.468075] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27022.473294] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27022.492032] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27022.495861] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27022.501078] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27022.508204] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27022.509693] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27022.516819] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27022.523948] [27022.531066] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27022.584089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27022.589827] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27022.596953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27022.604076] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27022.611206] Call Trace: [27022.613652] [27022.615665] ? watchdog_timer_fn+0x1b4/0x220 [27022.619928] ? __hrtimer_run_queues+0x1d3/0x360 [27022.624456] ? hrtimer_interrupt+0x104/0x240 [27022.628724] ? __local_bh_enable+0x50/0x60 [27022.632818] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27022.638043] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27022.642999] [27022.645095] [27022.647206] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27022.652519] ? lock_acquire+0xf0/0x2b0 [27022.656269] ? igrab+0x43/0x50 [27022.659319] ? lock_release+0xc5/0x260 [27022.663063] _raw_spin_lock+0x30/0x40 [27022.666719] ? igrab+0x1d/0x50 [27022.669769] igrab+0x1d/0x50 [27022.672648] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27022.677799] delayed_work+0x1d8/0x280 [ceph] [27022.682078] ? process_scheduled_works+0x22a/0x510 [27022.686861] process_scheduled_works+0x2a4/0x510 [27022.691491] worker_thread+0x188/0x340 [27022.695235] ? rescuer_thread+0x310/0x310 [27022.699237] kthread+0xf5/0x130 [27022.702375] ? kthread_complete_and_exit+0x20/0x20 [27022.707160] ret_from_fork+0x31/0x40 [27022.710730] ? kthread_complete_and_exit+0x20/0x20 [27022.715514] ret_from_fork_asm+0x11/0x20 [27022.719437] [27049.931291] watchdog: BUG: soft lockup - CPU#0 stuck for 14078s! [dbench:78683] [27049.938597] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27049.943291] watchdog: BUG: soft lockup - CPU#7 stuck for 23075s! [kworker/7:4:78808] [27049.994045] irq event stamp: 44262706 [27049.994046] hardirqs last enabled at (44262705): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27050.001777] Modules linked in: [27050.005433] hardirqs last disabled at (44262706): [] sysvec_apic_timer_interrupt+0xe/0x90 [27050.015590] ceph [27050.018643] softirqs last enabled at (44262628): [] __do_softirq+0x2be/0x371 [27050.028368] libceph [27050.030291] softirqs last disabled at (44262623): [] irq_exit_rcu+0xb0/0xd0 [27050.038974] dns_resolver fscache [27050.041161] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27050.049686] netfs veth [27050.052998] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27050.061941] nft_chain_nat [27050.064403] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27050.071875] xt_MASQUERADE nf_nat [27050.074580] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27050.080055] nf_conntrack nf_defrag_ipv6 [27050.083366] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27050.102104] nf_defrag_ipv4 nft_compat [27050.106023] [27050.111239] nf_tables [27050.114982] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27050.116473] nfnetlink [27050.118830] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27050.125965] bridge [27050.128321] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27050.135445] stp llc [27050.137541] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27050.144665] xfs [27050.146869] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27050.153992] libcrc32c [27050.155828] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27050.162954] sunrpc intel_rapl_msr [27050.165328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27050.173406] intel_rapl_common sb_edac [27050.176802] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27050.182541] x86_pkg_temp_thermal [27050.186284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27050.193410] coretemp kvm_intel [27050.196719] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27050.203843] kvm [27050.206981] Call Trace: [27050.214105] irqbypass [27050.215945] [27050.218386] i2c_i801 [27050.220746] ? watchdog_timer_fn+0x1b4/0x220 [27050.222754] iTCO_wdt iTCO_vendor_support [27050.225027] ? __hrtimer_run_queues+0x1d3/0x360 [27050.229288] i2c_smbus crct10dif_pclmul [27050.233297] ? hrtimer_interrupt+0x104/0x240 [27050.237818] joydev crc32_pclmul [27050.241648] ? __local_bh_enable+0x50/0x60 [27050.245913] ipmi_ssif ghash_clmulni_intel [27050.249139] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27050.253227] mei_me mei [27050.257320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27050.262537] lpc_ich [27050.264979] [27050.269938] sha512_ssse3 acpi_ipmi [27050.272120] [27050.274217] mfd_core wmi [27050.277703] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27050.279801] ipmi_si ipmi_devintf [27050.282422] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27050.287722] ipmi_msghandler ioatdma [27050.291033] do_raw_spin_lock+0xb5/0xc0 [27050.295901] acpi_pad acpi_power_meter [27050.299475] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27050.303304] ixgbe [27050.307051] try_get_cap_refs+0x159/0x880 [ceph] [27050.311833] igb mdio [27050.313850] __ceph_get_caps+0x4a1/0x760 [ceph] [27050.318456] crc32c_intel [27050.320730] ? __file_remove_privs+0x89/0x110 [27050.325250] nvme [27050.327867] ? wake_bit_function+0x70/0x70 [27050.332215] ptp nvme_core [27050.334144] ? ceph_write_iter+0x260/0x770 [ceph] [27050.338231] i2c_algo_bit pps_core [27050.340935] ceph_write_iter+0x260/0x770 [ceph] [27050.345632] dca fuse [27050.349035] ? lock_is_held_type+0xaa/0x120 [27050.353555] [27050.353556] irq event stamp: 138237462 [27050.355828] vfs_write+0x324/0x4f0 [27050.360003] hardirqs last enabled at (138237461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27050.361497] ksys_pwrite64+0x65/0xa0 [27050.365237] hardirqs last disabled at (138237462): [] sysvec_apic_timer_interrupt+0xe/0x90 [27050.368636] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27050.378878] softirqs last enabled at (138237460): [] __do_softirq+0x2be/0x371 [27050.382450] do_syscall_64+0x38/0x80 [27050.392260] softirqs last disabled at (138237455): [] irq_exit_rcu+0xb0/0xd0 [27050.397393] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27050.406163] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27050.409734] RIP: 0033:0x7f48f0eeb93b [27050.418331] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27050.423374] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27050.432752] Workqueue: events delayed_work [ceph] [27050.436322] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27050.443793] [27050.443794] RIP: 0010:lock_release+0x77/0x260 [27050.462531] ORIG_RAX: 0000000000000012 [27050.467228] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [27050.472445] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27050.473937] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [27050.478286] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27050.482117] [27050.482118] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [27050.500856] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27050.507986] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [27050.513207] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27050.520329] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [27050.521820] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27050.528945] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [27050.536074] [27050.543193] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27050.581007] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27050.589084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27050.594820] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27050.601946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27050.609070] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27050.616193] Call Trace: [27050.618639] [27050.620651] ? watchdog_timer_fn+0x1b4/0x220 [27050.624915] ? __hrtimer_run_queues+0x1d3/0x360 [27050.629439] ? hrtimer_interrupt+0x104/0x240 [27050.633702] ? __local_bh_enable+0x50/0x60 [27050.637794] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27050.643018] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27050.647991] [27050.650091] [27050.652189] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27050.657492] ? igrab+0x43/0x50 [27050.660545] ? lock_release+0x77/0x260 [27050.664289] _raw_spin_unlock+0x1b/0x40 [27050.668126] igrab+0x43/0x50 [27050.671004] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27050.676155] delayed_work+0x1d8/0x280 [ceph] [27050.680434] ? process_scheduled_works+0x22a/0x510 [27050.685227] process_scheduled_works+0x2a4/0x510 [27050.689842] worker_thread+0x188/0x340 [27050.693591] ? rescuer_thread+0x310/0x310 [27050.697592] kthread+0xf5/0x130 [27050.700732] ? kthread_complete_and_exit+0x20/0x20 [27050.705516] ret_from_fork+0x31/0x40 [27050.709095] ? kthread_complete_and_exit+0x20/0x20 [27050.713880] ret_from_fork_asm+0x11/0x20 [27050.717810] [27050.749671] rcu: INFO: rcu_preempt self-detected stall on CPU [27050.755428] rcu: 0-....: (14774794 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7386179 [27050.765854] rcu: (t=15113872 jiffies g=1309609 q=454675 ncpus=8) [27050.771939] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27050.780882] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27050.788353] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27050.793831] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27050.812568] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27050.817784] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27050.824910] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27050.832033] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27050.839158] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27050.846282] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27050.853406] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27050.861481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27050.867220] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27050.874344] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27050.881469] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27050.888593] Call Trace: [27050.891039] [27050.893048] ? rcu_dump_cpu_stacks+0xd1/0x180 [27050.897407] ? rcu_sched_clock_irq+0x633/0x1390 [27050.901932] ? lock_is_held_type+0xaa/0x120 [27050.906111] ? lock_is_held_type+0xaa/0x120 [27050.910287] ? tick_sched_do_timer+0xa0/0xa0 [27050.914559] ? rcu_read_lock_sched_held+0x56/0x90 [27050.919258] ? tick_sched_do_timer+0xa0/0xa0 [27050.923528] ? update_process_times+0x5f/0x90 [27050.927879] ? tick_sched_handle.isra.16+0x1e/0x50 [27050.932663] ? tick_sched_timer+0x6f/0x80 [27050.936669] ? __hrtimer_run_queues+0x1d3/0x360 [27050.941194] ? hrtimer_interrupt+0x104/0x240 [27050.945456] ? __local_bh_enable+0x50/0x60 [27050.949550] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27050.954774] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27050.959730] [27050.961827] [27050.963926] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27050.969233] ? queued_spin_lock_slowpath+0x16/0x2e0 [27050.974111] do_raw_spin_lock+0xb5/0xc0 [27050.977948] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27050.982751] try_get_cap_refs+0x159/0x880 [ceph] [27050.987382] __ceph_get_caps+0x4a1/0x760 [ceph] [27050.991924] ? __file_remove_privs+0x89/0x110 [27050.996280] ? wake_bit_function+0x70/0x70 [27051.000370] ? ceph_write_iter+0x260/0x770 [ceph] [27051.005075] ceph_write_iter+0x260/0x770 [ceph] [27051.009610] ? lock_is_held_type+0xaa/0x120 [27051.013797] vfs_write+0x324/0x4f0 [27051.017203] ksys_pwrite64+0x65/0xa0 [27051.020779] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27051.025912] do_syscall_64+0x38/0x80 [27051.029491] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27051.034533] RIP: 0033:0x7f48f0eeb93b [27051.038103] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27051.056842] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [27051.064399] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27051.071524] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27051.078647] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27051.085772] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27051.092895] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27051.100026] [27051.102213] Sending NMI from CPU 0 to CPUs 7: [27051.106566] NMI backtrace for cpu 7 [27051.106567] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27051.106569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27051.106570] Workqueue: events delayed_work [ceph] [27051.106589] RIP: 0010:__lock_acquire+0x6/0x1e20 [27051.106592] Code: 0d a3 2e 54 01 85 c9 0f 85 74 ff ff ff 48 c7 c6 e1 de 12 82 48 c7 c7 8a 3f 12 82 e8 94 b8 f7 ff 0f 0b 5b c3 41 57 41 56 41 55 <41> 54 55 53 65 48 8b 1c 25 c0 e5 02 00 48 83 ec 58 44 8b 15 6a 2e [27051.106594] RSP: 0018:ffffc90002117cd0 EFLAGS: 00000006 [27051.106595] RAX: 0000000000000246 RBX: 0000000000000200 RCX: 0000000000000000 [27051.106597] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [27051.106598] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [27051.106599] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [27051.106600] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27051.106601] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27051.106603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27051.106604] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27051.106605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27051.106606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27051.106607] Call Trace: [27051.106608] [27051.106610] ? nmi_cpu_backtrace+0xad/0x120 [27051.106614] ? nmi_cpu_backtrace_handler+0x11/0x20 [27051.106616] ? nmi_handle+0xd4/0x260 [27051.106621] ? default_do_nmi+0x49/0xf0 [27051.106623] ? exc_nmi+0x11e/0x150 [27051.106625] ? end_repeat_nmi+0x16/0x67 [27051.106631] ? __lock_acquire+0x6/0x1e20 [27051.106634] ? __lock_acquire+0x6/0x1e20 [27051.106636] ? __lock_acquire+0x6/0x1e20 [27051.106639] [27051.106639] [27051.106640] lock_acquire+0xcb/0x2b0 [27051.106643] ? igrab+0x1d/0x50 [27051.106645] ? igrab+0x43/0x50 [27051.106647] ? lock_release+0xc5/0x260 [27051.106650] _raw_spin_lock+0x30/0x40 [27051.106652] ? igrab+0x1d/0x50 [27051.106654] igrab+0x1d/0x50 [27051.106656] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27051.106675] delayed_work+0x1d8/0x280 [ceph] [27051.106711] ? process_scheduled_works+0x22a/0x510 [27051.106715] process_scheduled_works+0x2a4/0x510 [27051.106721] worker_thread+0x188/0x340 [27051.106724] ? rescuer_thread+0x310/0x310 [27051.106727] kthread+0xf5/0x130 [27051.106730] ? kthread_complete_and_exit+0x20/0x20 [27051.106733] ret_from_fork+0x31/0x40 [27051.106736] ? kthread_complete_and_exit+0x20/0x20 [27051.106739] ret_from_fork_asm+0x11/0x20 [27051.106746] [27077.931781] watchdog: BUG: soft lockup - CPU#0 stuck for 14104s! [dbench:78683] [27077.939084] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27077.943782] watchdog: BUG: soft lockup - CPU#7 stuck for 23101s! [kworker/7:4:78808] [27077.994525] irq event stamp: 44320206 [27077.994526] hardirqs last enabled at (44320205): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27078.002257] Modules linked in: [27078.005913] hardirqs last disabled at (44320206): [] sysvec_apic_timer_interrupt+0xe/0x90 [27078.016071] ceph [27078.019120] softirqs last enabled at (44320136): [] __do_softirq+0x2be/0x371 [27078.028846] libceph [27078.030769] softirqs last disabled at (44320131): [] irq_exit_rcu+0xb0/0xd0 [27078.039456] dns_resolver fscache [27078.041639] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27078.050148] netfs veth [27078.053473] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27078.062420] nft_chain_nat [27078.064865] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27078.072336] xt_MASQUERADE nf_nat [27078.075041] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27078.080517] nf_conntrack nf_defrag_ipv6 [27078.083826] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27078.102566] nf_defrag_ipv4 nft_compat [27078.106482] [27078.106483] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27078.111699] nf_tables nfnetlink [27078.115460] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27078.116952] bridge stp [27078.124076] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27078.127301] llc xfs [27078.134424] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27078.136868] libcrc32c [27078.143992] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27078.146179] sunrpc [27078.153302] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27078.155657] intel_rapl_msr intel_rapl_common [27078.162782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27078.164881] sb_edac x86_pkg_temp_thermal [27078.172956] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27078.177308] coretemp kvm_intel [27078.183044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27078.187049] kvm irqbypass [27078.194179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27078.197320] i2c_i801 iTCO_wdt [27078.204461] Call Trace: [27078.204462] [27078.207180] iTCO_vendor_support [27078.214307] ? watchdog_timer_fn+0x1b4/0x220 [27078.217357] i2c_smbus crct10dif_pclmul joydev [27078.219802] ? __hrtimer_run_queues+0x1d3/0x360 [27078.221813] crc32_pclmul ipmi_ssif [27078.225040] ? hrtimer_interrupt+0x104/0x240 [27078.229301] ghash_clmulni_intel mei_me [27078.233737] ? __local_bh_enable+0x50/0x60 [27078.238263] mei lpc_ich [27078.241748] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27078.246010] sha512_ssse3 acpi_ipmi [27078.249842] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27078.253932] mfd_core [27078.256469] [27078.261688] wmi ipmi_si [27078.265179] [27078.265181] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27078.270138] ipmi_devintf [27078.272413] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27078.274506] ipmi_msghandler [27078.277040] do_raw_spin_lock+0xb5/0xc0 [27078.279134] ioatdma acpi_pad [27078.284440] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27078.287073] acpi_power_meter ixgbe [27078.291946] try_get_cap_refs+0x159/0x880 [ceph] [27078.294821] igb mdio [27078.298657] __ceph_get_caps+0x4a1/0x760 [ceph] [27078.301616] crc32c_intel nvme [27078.306404] ? __file_remove_privs+0x89/0x110 [27078.309884] ptp nvme_core [27078.314496] ? wake_bit_function+0x70/0x70 [27078.316765] i2c_algo_bit pps_core [27078.321293] ? ceph_write_iter+0x260/0x770 [ceph] [27078.324340] dca fuse [27078.328692] ceph_write_iter+0x260/0x770 [ceph] [27078.331394] [27078.331395] irq event stamp: 138456402 [27078.335490] ? lock_is_held_type+0xaa/0x120 [27078.338883] hardirqs last enabled at (138456401): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27078.343585] vfs_write+0x324/0x4f0 [27078.345852] hardirqs last disabled at (138456402): [] sysvec_apic_timer_interrupt+0xe/0x90 [27078.350380] ksys_pwrite64+0x65/0xa0 [27078.351868] softirqs last enabled at (138456400): [] __do_softirq+0x2be/0x371 [27078.355611] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27078.359787] softirqs last disabled at (138456395): [] irq_exit_rcu+0xb0/0xd0 [27078.370034] do_syscall_64+0x38/0x80 [27078.373430] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27078.383240] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27078.386812] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27078.395582] RIP: 0033:0x7f48f0eeb93b [27078.400712] Workqueue: events delayed_work [ceph] [27078.409310] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27078.412880] [27078.422258] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27078.427303] RIP: 0010:lock_acquire+0xf0/0x2b0 [27078.434772] ORIG_RAX: 0000000000000012 [27078.438343] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27078.443040] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27078.461778] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27078.463270] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27078.468488] [27078.468488] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27078.472839] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27078.476670] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27078.495406] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27078.502532] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27078.507749] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27078.514871] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27078.516369] [27078.523486] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27078.575542] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27078.583618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27078.589356] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27078.596479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27078.603604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27078.610728] Call Trace: [27078.613187] [27078.615200] ? watchdog_timer_fn+0x1b4/0x220 [27078.619470] ? __hrtimer_run_queues+0x1d3/0x360 [27078.623999] ? hrtimer_interrupt+0x104/0x240 [27078.628270] ? __local_bh_enable+0x50/0x60 [27078.632362] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27078.637578] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27078.642536] [27078.644633] [27078.646731] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27078.652038] ? lock_acquire+0xf0/0x2b0 [27078.655787] ? igrab+0x43/0x50 [27078.658838] ? lock_release+0xc5/0x260 [27078.662583] _raw_spin_lock+0x30/0x40 [27078.666240] ? igrab+0x1d/0x50 [27078.669290] igrab+0x1d/0x50 [27078.672186] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27078.677335] delayed_work+0x1d8/0x280 [ceph] [27078.681616] ? process_scheduled_works+0x22a/0x510 [27078.686406] process_scheduled_works+0x2a4/0x510 [27078.691022] worker_thread+0x188/0x340 [27078.694770] ? rescuer_thread+0x310/0x310 [27078.698775] kthread+0xf5/0x130 [27078.701913] ? kthread_complete_and_exit+0x20/0x20 [27078.706697] ret_from_fork+0x31/0x40 [27078.710267] ? kthread_complete_and_exit+0x20/0x20 [27078.715051] ret_from_fork_asm+0x11/0x20 [27078.718973] [27105.932272] watchdog: BUG: soft lockup - CPU#0 stuck for 14130s! [dbench:78683] [27105.939581] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27105.944272] watchdog: BUG: soft lockup - CPU#7 stuck for 23127s! [kworker/7:4:78808] [27105.995031] irq event stamp: 44378462 [27106.002761] Modules linked in: ceph libceph dns_resolver fscache netfs veth [27106.006417] hardirqs last enabled at (44378461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27106.013370] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [27106.023527] hardirqs last disabled at (44378462): [] sysvec_apic_timer_interrupt+0xe/0x90 [27106.029177] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [27106.038900] softirqs last enabled at (44378394): [] __do_softirq+0x2be/0x371 [27106.046633] llc xfs libcrc32c sunrpc intel_rapl_msr [27106.055316] softirqs last disabled at (44378389): [] irq_exit_rcu+0xb0/0xd0 [27106.055320] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [27106.068794] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27106.068796] irqbypass i2c_i801 [27106.076352] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27106.085294] iTCO_wdt [27106.088432] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27106.095903] iTCO_vendor_support [27106.098175] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27106.103668] i2c_smbus crct10dif_pclmul [27106.106893] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27106.125630] joydev crc32_pclmul [27106.129463] [27106.134678] ipmi_ssif [27106.137902] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27106.139392] ghash_clmulni_intel mei_me [27106.141750] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27106.148875] mei [27106.152706] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27106.159829] lpc_ich [27106.161670] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27106.168790] sha512_ssse3 acpi_ipmi [27106.170975] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27106.178100] mfd_core wmi [27106.181584] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27106.188706] ipmi_si [27106.191325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27106.199403] ipmi_devintf [27106.201586] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27106.207324] ipmi_msghandler ioatdma [27106.209950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27106.217074] acpi_pad acpi_power_meter [27106.220643] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27106.227785] ixgbe igb [27106.231530] Call Trace: [27106.238670] mdio crc32c_intel [27106.241031] [27106.243473] nvme [27106.246527] ? watchdog_timer_fn+0x1b4/0x220 [27106.248534] ptp [27106.250461] ? __hrtimer_run_queues+0x1d3/0x360 [27106.254723] nvme_core [27106.256564] ? hrtimer_interrupt+0x104/0x240 [27106.261084] i2c_algo_bit [27106.263440] ? __local_bh_enable+0x50/0x60 [27106.267705] pps_core [27106.270325] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27106.274413] dca [27106.276684] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27106.281902] fuse [27106.283739] [27106.288697] [27106.290619] [27106.292718] irq event stamp: 138677262 [27106.294209] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27106.296307] hardirqs last enabled at (138677261): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27106.300053] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27106.305353] hardirqs last disabled at (138677262): [] sysvec_apic_timer_interrupt+0xe/0x90 [27106.315601] do_raw_spin_lock+0xb5/0xc0 [27106.320470] softirqs last enabled at (138677260): [] __do_softirq+0x2be/0x371 [27106.330282] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27106.334110] softirqs last disabled at (138677255): [] irq_exit_rcu+0xb0/0xd0 [27106.342884] try_get_cap_refs+0x159/0x880 [ceph] [27106.347672] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27106.356275] __ceph_get_caps+0x4a1/0x760 [ceph] [27106.360882] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27106.360884] Workqueue: events delayed_work [ceph] [27106.370263] ? __file_remove_privs+0x89/0x110 [27106.374784] [27106.382257] ? wake_bit_function+0x70/0x70 [27106.386955] RIP: 0010:lock_acquire+0xf0/0x2b0 [27106.391304] ? ceph_write_iter+0x260/0x770 [ceph] [27106.392794] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27106.396886] ceph_write_iter+0x260/0x770 [ceph] [27106.401236] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27106.405953] ? lock_is_held_type+0xaa/0x120 [27106.424689] [27106.424689] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27106.429214] vfs_write+0x324/0x4f0 [27106.434430] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27106.438612] ksys_pwrite64+0x65/0xa0 [27106.440098] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27106.447223] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27106.450618] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27106.457745] do_syscall_64+0x38/0x80 [27106.461313] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27106.468439] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27106.473568] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27106.480693] RIP: 0033:0x7f48f0eeb93b [27106.484264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27106.491388] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27106.496432] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27106.504509] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27106.508080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27106.513818] ORIG_RAX: 0000000000000012 [27106.532555] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27106.539680] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27106.544897] Call Trace: [27106.544899] [27106.552020] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27106.555851] ? watchdog_timer_fn+0x1b4/0x220 [27106.562975] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27106.570100] ? __hrtimer_run_queues+0x1d3/0x360 [27106.572542] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27106.574557] ? hrtimer_interrupt+0x104/0x240 [27106.581679] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27106.585959] ? __local_bh_enable+0x50/0x60 [27106.593090] [27106.597611] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27106.627613] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27106.632570] [27106.634677] [27106.636774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27106.642083] ? lock_acquire+0xf0/0x2b0 [27106.645833] ? igrab+0x43/0x50 [27106.648883] ? lock_release+0xc5/0x260 [27106.652626] _raw_spin_lock+0x30/0x40 [27106.656282] ? igrab+0x1d/0x50 [27106.659335] igrab+0x1d/0x50 [27106.662213] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27106.667364] delayed_work+0x1d8/0x280 [ceph] [27106.671642] ? process_scheduled_works+0x22a/0x510 [27106.676434] process_scheduled_works+0x2a4/0x510 [27106.681050] worker_thread+0x188/0x340 [27106.684797] ? rescuer_thread+0x310/0x310 [27106.688802] kthread+0xf5/0x130 [27106.691962] ? kthread_complete_and_exit+0x20/0x20 [27106.696750] ret_from_fork+0x31/0x40 [27106.700329] ? kthread_complete_and_exit+0x20/0x20 [27106.705112] ret_from_fork_asm+0x11/0x20 [27106.709035] [27133.932763] watchdog: BUG: soft lockup - CPU#0 stuck for 14156s! [dbench:78683] [27133.940068] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27133.944763] watchdog: BUG: soft lockup - CPU#7 stuck for 23153s! [kworker/7:4:78808] [27133.995508] irq event stamp: 44436532 [27133.995509] hardirqs last enabled at (44436531): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27134.003240] Modules linked in: [27134.006896] hardirqs last disabled at (44436532): [] sysvec_apic_timer_interrupt+0xe/0x90 [27134.017053] ceph [27134.020105] softirqs last enabled at (44436474): [] __do_softirq+0x2be/0x371 [27134.029830] libceph dns_resolver [27134.031753] softirqs last disabled at (44436469): [] irq_exit_rcu+0xb0/0xd0 [27134.040440] fscache netfs [27134.043749] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27134.052258] veth nft_chain_nat [27134.054964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27134.063907] xt_MASQUERADE nf_nat [27134.067043] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [27134.074515] nf_conntrack nf_defrag_ipv6 [27134.077826] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [27134.083304] nf_defrag_ipv4 nft_compat [27134.087221] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27134.105958] nf_tables nfnetlink [27134.109703] [27134.109703] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27134.114920] bridge stp [27134.118159] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27134.119653] llc xfs [27134.126776] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27134.129219] libcrc32c sunrpc [27134.136344] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27134.138529] intel_rapl_msr [27134.145652] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27134.148616] intel_rapl_common sb_edac [27134.155740] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27134.158532] x86_pkg_temp_thermal coretemp [27134.165655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27134.169401] kvm_intel kvm [27134.177477] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27134.181567] irqbypass i2c_i801 [27134.187305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27134.190009] iTCO_wdt iTCO_vendor_support [27134.197135] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27134.200289] i2c_smbus crct10dif_pclmul [27134.207412] Call Trace: [27134.207413] [27134.211418] joydev [27134.218540] ? watchdog_timer_fn+0x1b4/0x220 [27134.222372] crc32_pclmul ipmi_ssif [27134.224816] ? __hrtimer_run_queues+0x1d3/0x360 [27134.226827] ghash_clmulni_intel mei_me [27134.228928] ? hrtimer_interrupt+0x104/0x240 [27134.233187] mei lpc_ich [27134.236672] ? __local_bh_enable+0x50/0x60 [27134.241197] sha512_ssse3 acpi_ipmi [27134.245030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27134.249290] mfd_core wmi [27134.251821] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27134.255912] ipmi_si ipmi_devintf [27134.259396] [27134.264614] ipmi_msghandler [27134.267230] [27134.267232] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27134.272188] ioatdma [27134.275502] ? queued_spin_lock_slowpath+0x2b/0x2e0 [27134.277597] acpi_pad acpi_power_meter [27134.280477] do_raw_spin_lock+0xb5/0xc0 [27134.282571] ixgbe igb [27134.287877] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27134.290059] mdio crc32c_intel [27134.294932] try_get_cap_refs+0x159/0x880 [ceph] [27134.298674] nvme ptp [27134.302510] __ceph_get_caps+0x4a1/0x760 [ceph] [27134.304862] nvme_core i2c_algo_bit [27134.309652] ? __file_remove_privs+0x89/0x110 [27134.312698] pps_core dca [27134.317308] ? wake_bit_function+0x70/0x70 [27134.319580] fuse [27134.324106] ? ceph_write_iter+0x260/0x770 [ceph] [27134.327586] irq event stamp: 138898332 [27134.331939] ceph_write_iter+0x260/0x770 [ceph] [27134.334555] hardirqs last enabled at (138898331): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27134.338650] ? lock_is_held_type+0xaa/0x120 [27134.340571] hardirqs last disabled at (138898332): [] sysvec_apic_timer_interrupt+0xe/0x90 [27134.345270] vfs_write+0x324/0x4f0 [27134.349011] softirqs last enabled at (138898330): [] __do_softirq+0x2be/0x371 [27134.353540] ksys_pwrite64+0x65/0xa0 [27134.363780] softirqs last disabled at (138898325): [] irq_exit_rcu+0xb0/0xd0 [27134.367957] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27134.377768] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27134.381165] do_syscall_64+0x38/0x80 [27134.389935] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27134.393508] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27134.402104] Workqueue: events delayed_work [ceph] [27134.407235] RIP: 0033:0x7f48f0eeb93b [27134.416613] [27134.416614] RIP: 0010:lock_release+0x1f/0x260 [27134.420183] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27134.427654] Code: a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 <48> 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 [27134.432698] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27134.437395] RSP: 0018:ffffc90002117d48 EFLAGS: 00000296 [27134.440965] ORIG_RAX: 0000000000000012 [27134.442456] [27134.442456] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 000000005553a634 [27134.446808] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27134.465546] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [27134.484282] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27134.489500] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [27134.494718] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27134.498548] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [27134.500039] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27134.507165] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27134.514288] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27134.521413] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27134.528542] [27134.535660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27134.587289] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27134.594414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27134.601536] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27134.608660] Call Trace: [27134.611104] [27134.613115] ? watchdog_timer_fn+0x1b4/0x220 [27134.617382] ? __hrtimer_run_queues+0x1d3/0x360 [27134.621908] ? hrtimer_interrupt+0x104/0x240 [27134.626177] ? __local_bh_enable+0x50/0x60 [27134.630271] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27134.635494] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27134.640450] [27134.642547] [27134.644647] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27134.649950] ? igrab+0x43/0x50 [27134.653004] ? igrab+0x43/0x50 [27134.656052] ? lock_release+0x1f/0x260 [27134.659797] ? lock_release+0xc5/0x260 [27134.663540] _raw_spin_unlock+0x1b/0x40 [27134.667372] igrab+0x43/0x50 [27134.670249] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27134.675397] delayed_work+0x1d8/0x280 [ceph] [27134.679680] ? process_scheduled_works+0x22a/0x510 [27134.684468] process_scheduled_works+0x2a4/0x510 [27134.689084] worker_thread+0x188/0x340 [27134.692835] ? rescuer_thread+0x310/0x310 [27134.696837] kthread+0xf5/0x130 [27134.699975] ? kthread_complete_and_exit+0x20/0x20 [27134.704758] ret_from_fork+0x31/0x40 [27134.708330] ? kthread_complete_and_exit+0x20/0x20 [27134.713113] ret_from_fork_asm+0x11/0x20 [27134.717036] [27161.933254] watchdog: BUG: soft lockup - CPU#0 stuck for 14182s! [dbench:78683] [27161.940557] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27161.945254] watchdog: BUG: soft lockup - CPU#7 stuck for 23179s! [kworker/7:4:78808] [27161.996006] irq event stamp: 44494766 [27162.003735] Modules linked in: ceph libceph dns_resolver fscache netfs [27162.007393] hardirqs last enabled at (44494765): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27162.007398] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [27162.024086] hardirqs last disabled at (44494766): [] sysvec_apic_timer_interrupt+0xe/0x90 [27162.032768] nft_compat nf_tables nfnetlink bridge stp llc xfs [27162.042493] softirqs last enabled at (44494708): [] __do_softirq+0x2be/0x371 [27162.048318] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [27162.057001] softirqs last disabled at (44494703): [] irq_exit_rcu+0xb0/0xd0 [27162.057005] coretemp kvm_intel kvm irqbypass [27162.073850] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27162.073851] i2c_i801 [27162.073852] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27162.078202] iTCO_wdt iTCO_vendor_support [27162.087145] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27162.089415] i2c_smbus crct10dif_pclmul [27162.096886] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27162.100892] joydev [27162.106369] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27162.110200] crc32_pclmul ipmi_ssif [27162.128938] [27162.131033] ghash_clmulni_intel mei_me [27162.136252] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27162.139735] mei lpc_ich [27162.141226] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27162.145056] sha512_ssse3 acpi_ipmi [27162.152180] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27162.154713] mfd_core wmi [27162.161835] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27162.165320] ipmi_si ipmi_devintf [27162.172444] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27162.175062] ipmi_msghandler ioatdma [27162.182185] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27162.185497] acpi_pad acpi_power_meter [27162.192621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27162.196192] ixgbe [27162.204267] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27162.208013] igb [27162.213750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27162.215762] mdio crc32c_intel [27162.222886] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27162.224721] nvme ptp [27162.231845] Call Trace: [27162.231848] [27162.234898] nvme_core [27162.242023] ? watchdog_timer_fn+0x1b4/0x220 [27162.244291] i2c_algo_bit [27162.246736] ? __hrtimer_run_queues+0x1d3/0x360 [27162.248747] pps_core dca [27162.251107] ? hrtimer_interrupt+0x104/0x240 [27162.255370] fuse [27162.257987] ? __local_bh_enable+0x50/0x60 [27162.262510] [27162.262510] irq event stamp: 139119246 [27162.265129] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27162.269392] hardirqs last enabled at (139119245): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27162.271316] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27162.275405] hardirqs last disabled at (139119246): [] sysvec_apic_timer_interrupt+0xe/0x90 [27162.276897] [27162.280652] softirqs last enabled at (139119244): [] __do_softirq+0x2be/0x371 [27162.285866] [27162.285867] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27162.296112] softirqs last disabled at (139119239): [] irq_exit_rcu+0xb0/0xd0 [27162.301072] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27162.310880] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27162.312979] do_raw_spin_lock+0xb5/0xc0 [27162.321747] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27162.323848] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27162.329149] Workqueue: events delayed_work [ceph] [27162.337749] try_get_cap_refs+0x159/0x880 [ceph] [27162.342617] [27162.352002] __ceph_get_caps+0x4a1/0x760 [ceph] [27162.355826] RIP: 0010:lock_acquire+0xf0/0x2b0 [27162.363302] ? __file_remove_privs+0x89/0x110 [27162.368080] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27162.372777] ? wake_bit_function+0x70/0x70 [27162.377390] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27162.378882] ? ceph_write_iter+0x260/0x770 [ceph] [27162.383402] [27162.387753] ceph_write_iter+0x260/0x770 [ceph] [27162.392105] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27162.410846] ? lock_is_held_type+0xaa/0x120 [27162.414949] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27162.420171] vfs_write+0x324/0x4f0 [27162.424866] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27162.426361] ksys_pwrite64+0x65/0xa0 [27162.430881] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27162.438003] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27162.442182] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27162.449307] do_syscall_64+0x38/0x80 [27162.452704] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27162.459828] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27162.463397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27162.470523] RIP: 0033:0x7f48f0eeb93b [27162.475657] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27162.482776] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27162.486347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27162.494425] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27162.499468] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27162.505206] ORIG_RAX: 0000000000000012 [27162.508779] Call Trace: [27162.508781] [27162.515902] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27162.534659] ? watchdog_timer_fn+0x1b4/0x220 [27162.541780] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27162.547000] ? __hrtimer_run_queues+0x1d3/0x360 [27162.554122] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27162.557957] ? hrtimer_interrupt+0x104/0x240 [27162.560398] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27162.562408] ? __local_bh_enable+0x50/0x60 [27162.569532] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27162.573799] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27162.580941] [27162.585463] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27162.627549] [27162.629660] [27162.631760] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27162.637068] ? lock_acquire+0xf0/0x2b0 [27162.640819] ? igrab+0x43/0x50 [27162.643869] ? lock_release+0xc5/0x260 [27162.647614] _raw_spin_lock+0x30/0x40 [27162.651269] ? igrab+0x1d/0x50 [27162.654321] igrab+0x1d/0x50 [27162.657199] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27162.662348] delayed_work+0x1d8/0x280 [ceph] [27162.666630] ? process_scheduled_works+0x22a/0x510 [27162.671436] process_scheduled_works+0x2a4/0x510 [27162.676052] worker_thread+0x188/0x340 [27162.679803] ? rescuer_thread+0x310/0x310 [27162.683814] kthread+0xf5/0x130 [27162.686960] ? kthread_complete_and_exit+0x20/0x20 [27162.691745] ret_from_fork+0x31/0x40 [27162.695323] ? kthread_complete_and_exit+0x20/0x20 [27162.700108] ret_from_fork_asm+0x11/0x20 [27162.704041] [27189.933740] watchdog: BUG: soft lockup - CPU#0 stuck for 14208s! [dbench:78683] [27189.941043] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27189.945740] watchdog: BUG: soft lockup - CPU#7 stuck for 23205s! [kworker/7:4:78808] [27189.996483] irq event stamp: 44552892 [27189.996484] hardirqs last enabled at (44552891): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27190.004216] Modules linked in: [27190.007873] hardirqs last disabled at (44552892): [] sysvec_apic_timer_interrupt+0xe/0x90 [27190.018029] ceph [27190.021079] softirqs last enabled at (44552834): [] __do_softirq+0x2be/0x371 [27190.030804] libceph [27190.032727] softirqs last disabled at (44552829): [] irq_exit_rcu+0xb0/0xd0 [27190.041414] dns_resolver [27190.043596] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27190.052108] fscache netfs [27190.054742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27190.063687] veth [27190.066390] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27190.073862] nft_chain_nat xt_MASQUERADE [27190.075787] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27190.081262] nf_nat nf_conntrack [27190.085181] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27190.103917] nf_defrag_ipv6 nf_defrag_ipv4 [27190.107141] [27190.112359] nft_compat [27190.116451] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27190.117939] nf_tables nfnetlink [27190.120385] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27190.127509] bridge stp [27190.130734] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27190.137857] llc xfs [27190.140300] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27190.147426] libcrc32c [27190.149608] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27190.156733] sunrpc intel_rapl_msr [27190.159092] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27190.166216] intel_rapl_common [27190.169613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27190.177689] sb_edac x86_pkg_temp_thermal [27190.180741] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27190.186479] coretemp kvm_intel [27190.190482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27190.197607] kvm irqbypass [27190.200744] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27190.207868] i2c_i801 [27190.210572] Call Trace: [27190.210573] [27190.217695] iTCO_wdt iTCO_vendor_support [27190.219969] ? watchdog_timer_fn+0x1b4/0x220 [27190.222418] i2c_smbus crct10dif_pclmul [27190.224430] ? __hrtimer_run_queues+0x1d3/0x360 [27190.228433] joydev crc32_pclmul [27190.232702] ? hrtimer_interrupt+0x104/0x240 [27190.236528] ipmi_ssif ghash_clmulni_intel [27190.241053] ? __local_bh_enable+0x50/0x60 [27190.244277] mei_me mei [27190.248543] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27190.252632] lpc_ich sha512_ssse3 [27190.256724] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27190.259166] acpi_ipmi mfd_core [27190.264385] [27190.267695] wmi ipmi_si [27190.272652] [27190.275790] ipmi_devintf [27190.277887] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27190.280433] ipmi_msghandler ioatdma acpi_pad [27190.282537] ? queued_spin_lock_slowpath+0x18/0x2e0 [27190.285149] acpi_power_meter ixgbe [27190.290456] do_raw_spin_lock+0xb5/0xc0 [27190.294806] igb mdio [27190.299676] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27190.303160] crc32c_intel nvme [27190.306994] try_get_cap_refs+0x159/0x880 [ceph] [27190.309261] ptp nvme_core [27190.314050] __ceph_get_caps+0x4a1/0x760 [ceph] [27190.317096] i2c_algo_bit pps_core [27190.321712] ? __file_remove_privs+0x89/0x110 [27190.324420] dca fuse [27190.328944] ? wake_bit_function+0x70/0x70 [27190.332341] [27190.332342] irq event stamp: 139340326 [27190.336694] ? ceph_write_iter+0x260/0x770 [ceph] [27190.338961] hardirqs last enabled at (139340325): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27190.343052] ceph_write_iter+0x260/0x770 [ceph] [27190.344545] hardirqs last disabled at (139340326): [] sysvec_apic_timer_interrupt+0xe/0x90 [27190.348292] ? lock_is_held_type+0xaa/0x120 [27190.352986] softirqs last enabled at (139340324): [] __do_softirq+0x2be/0x371 [27190.363231] vfs_write+0x324/0x4f0 [27190.367753] softirqs last disabled at (139340319): [] irq_exit_rcu+0xb0/0xd0 [27190.377568] ksys_pwrite64+0x65/0xa0 [27190.381741] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27190.390513] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27190.393909] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27190.402510] do_syscall_64+0x38/0x80 [27190.406078] Workqueue: events delayed_work [ceph] [27190.415455] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27190.420586] [27190.428058] RIP: 0033:0x7f48f0eeb93b [27190.431629] RIP: 0010:lock_acquire+0x13/0x2b0 [27190.436324] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27190.441370] Code: 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 <41> 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 [27190.442861] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27190.446431] RSP: 0018:ffffc90002117d68 EFLAGS: 00000246 [27190.450780] ORIG_RAX: 0000000000000012 [27190.469518] [27190.469519] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [27190.488258] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27190.493475] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [27190.498693] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27190.502522] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [27190.504012] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27190.511143] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [27190.518271] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27190.525394] R13: ffff88817f867478 R14: 0000000000000000 R15: 0000000000000000 [27190.532517] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27190.539643] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27190.546773] [27190.553892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27190.591271] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27190.598396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27190.605520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27190.612644] Call Trace: [27190.615088] [27190.617098] ? watchdog_timer_fn+0x1b4/0x220 [27190.621363] ? __hrtimer_run_queues+0x1d3/0x360 [27190.625890] ? hrtimer_interrupt+0x104/0x240 [27190.630159] ? __local_bh_enable+0x50/0x60 [27190.634252] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27190.639476] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27190.644434] [27190.646531] [27190.648629] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27190.653937] ? lock_acquire+0x13/0x2b0 [27190.657686] _raw_spin_lock+0x30/0x40 [27190.661341] ? igrab+0x1d/0x50 [27190.664394] igrab+0x1d/0x50 [27190.667271] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27190.672431] delayed_work+0x1d8/0x280 [ceph] [27190.676711] ? process_scheduled_works+0x22a/0x510 [27190.681502] process_scheduled_works+0x2a4/0x510 [27190.686116] worker_thread+0x188/0x340 [27190.689867] ? rescuer_thread+0x310/0x310 [27190.693877] kthread+0xf5/0x130 [27190.697014] ? kthread_complete_and_exit+0x20/0x20 [27190.701800] ret_from_fork+0x31/0x40 [27190.705371] ? kthread_complete_and_exit+0x20/0x20 [27190.710153] ret_from_fork_asm+0x11/0x20 [27190.714077] [27217.934227] watchdog: BUG: soft lockup - CPU#0 stuck for 14234s! [dbench:78683] [27217.941530] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27217.946227] watchdog: BUG: soft lockup - CPU#7 stuck for 23231s! [kworker/7:4:78808] [27217.996980] irq event stamp: 44611100 [27217.996981] hardirqs last enabled at (44611099): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27218.004712] Modules linked in: [27218.008368] hardirqs last disabled at (44611100): [] sysvec_apic_timer_interrupt+0xe/0x90 [27218.018526] ceph [27218.021576] softirqs last enabled at (44611044): [] __do_softirq+0x2be/0x371 [27218.031301] libceph [27218.033226] softirqs last disabled at (44611039): [] irq_exit_rcu+0xb0/0xd0 [27218.041910] dns_resolver [27218.044094] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27218.052620] fscache netfs [27218.055240] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27218.064183] veth nft_chain_nat [27218.066887] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27218.074375] xt_MASQUERADE nf_nat [27218.077513] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27218.082990] nf_conntrack [27218.086299] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27218.105039] nf_defrag_ipv6 [27218.107655] [27218.112873] nf_defrag_ipv4 [27218.115663] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27218.117154] nft_compat [27218.119945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27218.127070] nf_tables [27218.129513] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27218.136637] nfnetlink bridge [27218.138995] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27218.146119] stp llc [27218.149083] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27218.156207] xfs libcrc32c [27218.158391] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27218.165515] sunrpc [27218.168220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27218.176297] intel_rapl_msr intel_rapl_common [27218.178393] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27218.184131] sb_edac x86_pkg_temp_thermal [27218.188483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27218.195624] coretemp kvm_intel [27218.199627] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27218.206751] kvm irqbypass [27218.209906] Call Trace: [27218.217030] i2c_i801 iTCO_wdt [27218.219738] [27218.222178] iTCO_vendor_support i2c_smbus [27218.225231] ? watchdog_timer_fn+0x1b4/0x220 [27218.227242] crct10dif_pclmul [27218.231333] ? __hrtimer_run_queues+0x1d3/0x360 [27218.235596] joydev crc32_pclmul [27218.238563] ? hrtimer_interrupt+0x104/0x240 [27218.243084] ipmi_ssif [27218.246306] ? __local_bh_enable+0x50/0x60 [27218.250572] ghash_clmulni_intel mei_me [27218.252931] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27218.257021] mei lpc_ich [27218.260850] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27218.266068] sha512_ssse3 [27218.268599] [27218.273573] acpi_ipmi [27218.276190] [27218.276191] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27218.278289] mfd_core [27218.280651] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27218.282743] wmi ipmi_si [27218.288048] do_raw_spin_lock+0xb5/0xc0 [27218.290318] ipmi_devintf [27218.295190] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27218.297718] ipmi_msghandler ioatdma [27218.301552] try_get_cap_refs+0x159/0x880 [ceph] [27218.304166] acpi_pad acpi_power_meter [27218.308956] __ceph_get_caps+0x4a1/0x760 [ceph] [27218.312522] ixgbe igb [27218.317137] ? __file_remove_privs+0x89/0x110 [27218.320876] mdio crc32c_intel [27218.325401] ? wake_bit_function+0x70/0x70 [27218.327757] nvme [27218.332112] ? ceph_write_iter+0x260/0x770 [ceph] [27218.335159] ptp [27218.339251] ceph_write_iter+0x260/0x770 [ceph] [27218.341175] nvme_core [27218.345875] ? lock_is_held_type+0xaa/0x120 [27218.347708] i2c_algo_bit pps_core [27218.352237] vfs_write+0x324/0x4f0 [27218.354591] dca fuse [27218.358773] ksys_pwrite64+0x65/0xa0 [27218.362165] [27218.362166] irq event stamp: 139561250 [27218.365563] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27218.367833] hardirqs last enabled at (139561249): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27218.371406] do_syscall_64+0x38/0x80 [27218.372907] hardirqs last disabled at (139561250): [] sysvec_apic_timer_interrupt+0xe/0x90 [27218.376650] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27218.381778] softirqs last enabled at (139561248): [] __do_softirq+0x2be/0x371 [27218.392025] RIP: 0033:0x7f48f0eeb93b [27218.395594] softirqs last disabled at (139561243): [] irq_exit_rcu+0xb0/0xd0 [27218.405404] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27218.410449] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27218.419220] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27218.422789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27218.431387] ORIG_RAX: 0000000000000012 [27218.450126] Workqueue: events delayed_work [ceph] [27218.459502] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27218.464719] [27218.464720] RIP: 0010:lock_acquire+0xf0/0x2b0 [27218.472191] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27218.476023] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27218.480719] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27218.487844] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27218.489333] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27218.493684] [27218.493684] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27218.500808] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27218.519547] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27218.526675] [27218.531886] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27218.571219] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27218.578343] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27218.585466] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27218.593543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27218.599282] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27218.606405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27218.613530] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27218.620654] Call Trace: [27218.623100] [27218.625110] ? watchdog_timer_fn+0x1b4/0x220 [27218.629373] ? __hrtimer_run_queues+0x1d3/0x360 [27218.633916] ? hrtimer_interrupt+0x104/0x240 [27218.638177] ? __local_bh_enable+0x50/0x60 [27218.642273] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27218.647497] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27218.652452] [27218.654549] [27218.656647] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27218.661955] ? lock_acquire+0xf0/0x2b0 [27218.665706] ? igrab+0x43/0x50 [27218.668755] ? lock_release+0xc5/0x260 [27218.672500] _raw_spin_lock+0x30/0x40 [27218.676155] ? igrab+0x1d/0x50 [27218.679206] igrab+0x1d/0x50 [27218.682086] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27218.687237] delayed_work+0x1d8/0x280 [ceph] [27218.691518] ? process_scheduled_works+0x22a/0x510 [27218.696307] process_scheduled_works+0x2a4/0x510 [27218.700937] worker_thread+0x188/0x340 [27218.704690] ? rescuer_thread+0x310/0x310 [27218.708699] kthread+0xf5/0x130 [27218.711840] ? kthread_complete_and_exit+0x20/0x20 [27218.716636] ret_from_fork+0x31/0x40 [27218.720211] ? kthread_complete_and_exit+0x20/0x20 [27218.724994] ret_from_fork_asm+0x11/0x20 [27218.728928] [27245.934713] watchdog: BUG: soft lockup - CPU#0 stuck for 14260s! [dbench:78683] [27245.942017] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27245.946713] watchdog: BUG: soft lockup - CPU#7 stuck for 23257s! [kworker/7:4:78808] [27245.997458] irq event stamp: 44669340 [27245.997459] hardirqs last enabled at (44669339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27246.005190] Modules linked in: [27246.008847] hardirqs last disabled at (44669340): [] sysvec_apic_timer_interrupt+0xe/0x90 [27246.019003] ceph [27246.022054] softirqs last enabled at (44669282): [] __do_softirq+0x2be/0x371 [27246.031781] libceph dns_resolver [27246.033702] softirqs last disabled at (44669277): [] irq_exit_rcu+0xb0/0xd0 [27246.042397] fscache netfs [27246.045707] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27246.054219] veth nft_chain_nat [27246.056921] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27246.056922] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27246.065867] xt_MASQUERADE [27246.069003] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27246.076475] nf_nat nf_conntrack [27246.081952] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27246.084656] nf_defrag_ipv6 nf_defrag_ipv4 [27246.103394] [27246.103394] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27246.106617] nft_compat nf_tables [27246.111836] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27246.115926] nfnetlink bridge [27246.116082] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [27246.116085] rcu: 0-....: (14965852 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7482171 [27246.116107] rcu: 7-....: (14866588 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/14865290 fqs=7482171 [27246.116110] rcu: (detected by 3, t=15309219 jiffies, g=1309609, q=458989 ncpus=8) [27246.116113] Sending NMI from CPU 3 to CPUs 0: [27246.116117] NMI backtrace for cpu 0 [27246.116118] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27246.116119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27246.116119] RIP: 0010:io_serial_in+0x19/0x20 [27246.116122] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [27246.116124] RSP: 0018:ffffc90000003c48 EFLAGS: 00000002 [27246.116125] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [27246.116126] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [27246.116127] RBP: 00000000000026da R08: 00005854b642d4f3 R09: 0000000000000000 [27246.116127] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [27246.116128] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000001 [27246.116129] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27246.116130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27246.116131] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27246.116132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27246.116133] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27246.116133] Call Trace: [27246.116134] [27246.116135] ? nmi_cpu_backtrace+0xad/0x120 [27246.116139] ? nmi_cpu_backtrace_handler+0x11/0x20 [27246.116140] ? nmi_handle+0xd4/0x260 [27246.116144] ? default_do_nmi+0x49/0xf0 [27246.116146] ? exc_nmi+0x11e/0x150 [27246.116147] ? end_repeat_nmi+0x16/0x67 [27246.116151] ? io_serial_in+0x19/0x20 [27246.116153] ? io_serial_in+0x19/0x20 [27246.116155] ? io_serial_in+0x19/0x20 [27246.116157] [27246.116157] [27246.116157] wait_for_lsr+0x45/0x70 [27246.116160] serial8250_console_write+0x341/0x3f0 [27246.116163] ? lock_release+0xc5/0x260 [27246.116165] console_flush_all+0x227/0x5a0 [27246.116168] ? console_flush_all+0x1ff/0x5a0 [27246.116169] ? console_flush_all+0x1ff/0x5a0 [27246.116172] console_unlock+0x7b/0x120 [27246.116174] vprintk_emit+0x246/0x340 [27246.116176] ? touch_softlockup_watchdog+0x20/0x20 [27246.116178] _printk+0x5c/0x80 [27246.116181] __show_regs+0x78/0x2e0 [27246.116184] ? touch_softlockup_watchdog+0x20/0x20 [27246.116185] show_regs+0x38/0x60 [27246.116187] watchdog_timer_fn+0x1b4/0x220 [27246.116189] __hrtimer_run_queues+0x1d3/0x360 [27246.116192] hrtimer_interrupt+0x104/0x240 [27246.116193] ? __local_bh_enable+0x50/0x60 [27246.116196] __sysvec_apic_timer_interrupt+0x61/0x160 [27246.116199] sysvec_apic_timer_interrupt+0x6d/0x90 [27246.116201] [27246.116202] [27246.116202] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27246.116204] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27246.116207] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27246.116208] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27246.116209] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27246.116210] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27246.116210] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27246.116211] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27246.116212] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27246.116215] do_raw_spin_lock+0xb5/0xc0 [27246.116218] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27246.116233] try_get_cap_refs+0x159/0x880 [ceph] [27246.116248] __ceph_get_caps+0x4a1/0x760 [ceph] [27246.116261] ? __file_remove_privs+0x89/0x110 [27246.116263] ? wake_bit_function+0x70/0x70 [27246.116265] ? ceph_write_iter+0x260/0x770 [ceph] [27246.116276] ceph_write_iter+0x260/0x770 [ceph] [27246.116288] ? lock_is_held_type+0xaa/0x120 [27246.116290] vfs_write+0x324/0x4f0 [27246.116294] ksys_pwrite64+0x65/0xa0 [27246.116296] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27246.116299] do_syscall_64+0x38/0x80 [27246.116301] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27246.116303] RIP: 0033:0x7f48f0eeb93b [27246.116304] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27246.116306] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [27246.116307] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27246.116307] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27246.116308] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27246.116309] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27246.116310] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27246.116313] [27246.117115] Sending NMI from CPU 3 to CPUs 7: [27246.117118] NMI backtrace for cpu 7 [27246.117119] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27246.117121] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27246.117122] Workqueue: events delayed_work [ceph] [27246.117143] RIP: 0010:vprintk_emit+0x1ee/0x340 [27246.117146] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [27246.117148] RSP: 0018:ffffc90000264e08 EFLAGS: 00000002 [27246.117149] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [27246.117150] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb550 [27246.117152] RBP: 0000000000000007 R08: 0000000000005ef4 R09: 0000000000000003 [27246.117153] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [27246.117154] R13: 0000000000000000 R14: ffffffff82130806 R15: ffffc90000264e58 [27246.117155] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27246.117157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27246.117158] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27246.117160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27246.117161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27246.117162] Call Trace: [27246.117163] [27246.117164] ? nmi_cpu_backtrace+0xad/0x120 [27246.117168] ? nmi_cpu_backtrace_handler+0x11/0x20 [27246.117170] ? nmi_handle+0xd4/0x260 [27246.117175] ? default_do_nmi+0x49/0xf0 [27246.117177] ? exc_nmi+0x11e/0x150 [27246.117180] ? end_repeat_nmi+0x16/0x67 [27246.117186] ? vprintk_emit+0x1ee/0x340 [27246.117189] ? vprintk_emit+0x1ee/0x340 [27246.117192] ? vprintk_emit+0x1ee/0x340 [27246.117195] [27246.117195] [27246.117196] ? irq_work_queue+0x2c/0x50 [27246.117201] ? touch_softlockup_watchdog+0x20/0x20 [27246.117203] _printk+0x5c/0x80 [27246.117208] print_modules+0x6e/0xe0 [27246.117211] watchdog_timer_fn+0x19f/0x220 [27246.117214] __hrtimer_run_queues+0x1d3/0x360 [27246.117219] hrtimer_interrupt+0x104/0x240 [27246.117221] ? __local_bh_enable+0x50/0x60 [27246.117225] __sysvec_apic_timer_interrupt+0x61/0x160 [27246.117229] sysvec_apic_timer_interrupt+0x6d/0x90 [27246.117231] [27246.117232] [27246.117233] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27246.117235] RIP: 0010:rcu_is_watching+0x0/0x40 [27246.117239] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [27246.117241] RSP: 0018:ffffc90002117d10 EFLAGS: 00000247 [27246.117243] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [27246.117244] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [27246.117245] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [27246.117246] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [27246.117247] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27246.117253] lock_acquire+0x184/0x2b0 [27246.117255] ? igrab+0x43/0x50 [27246.117258] ? lock_release+0xc5/0x260 [27246.117261] _raw_spin_lock+0x30/0x40 [27246.117263] ? igrab+0x1d/0x50 [27246.117265] igrab+0x1d/0x50 [27246.117267] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27246.117287] delayed_work+0x1d8/0x280 [ceph] [27246.117307] ? process_scheduled_works+0x22a/0x510 [27246.117310] process_scheduled_works+0x2a4/0x510 [27246.117316] worker_thread+0x188/0x340 [27246.117320] ? rescuer_thread+0x310/0x310 [27246.117322] kthread+0xf5/0x130 [27246.117325] ? kthread_complete_and_exit+0x20/0x20 [27246.117329] ret_from_fork+0x31/0x40 [27246.117331] ? kthread_complete_and_exit+0x20/0x20 [27246.117334] ret_from_fork_asm+0x11/0x20 [27246.117341] [27246.117417] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27246.117418] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27246.124542] stp llc [27246.127852] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27246.134976] xfs libcrc32c [27246.137939] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27246.144023] sunrpc intel_rapl_msr [27246.154441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27246.165033] intel_rapl_common sb_edac [27246.172590] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27246.176940] x86_pkg_temp_thermal coretemp [27246.180423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27246.189367] kvm_intel kvm [27246.196856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27246.201121] irqbypass i2c_i801 [27246.219859] Call Trace: [27246.225075] iTCO_wdt iTCO_vendor_support [27246.232201] [27246.239323] i2c_smbus crct10dif_pclmul [27246.246449] ? watchdog_timer_fn+0x1b4/0x220 [27246.253573] joydev crc32_pclmul [27246.260696] ? __hrtimer_run_queues+0x1d3/0x360 [27246.268773] ipmi_ssif ghash_clmulni_intel [27246.274514] ? hrtimer_interrupt+0x104/0x240 [27246.281636] mei_me mei [27246.288760] ? __local_bh_enable+0x50/0x60 [27246.295882] lpc_ich sha512_ssse3 [27246.298331] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27246.300339] acpi_ipmi mfd_core [27246.304516] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27246.309300] wmi ipmi_si [27246.312871] [27246.316702] ipmi_devintf ipmi_msghandler [27246.320112] [27246.323948] ioatdma acpi_pad [27246.327605] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27246.331262] acpi_power_meter ixgbe [27246.334923] ? queued_spin_lock_slowpath+0x18/0x2e0 [27246.337017] igb mdio [27246.339030] do_raw_spin_lock+0xb5/0xc0 [27246.342510] crc32c_intel nvme [27246.347211] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27246.350952] ptp nvme_core [27246.355046] try_get_cap_refs+0x159/0x880 [ceph] [27246.359308] i2c_algo_bit pps_core [27246.363576] __ceph_get_caps+0x4a1/0x760 [ceph] [27246.367316] dca fuse [27246.370978] ? __file_remove_privs+0x89/0x110 [27246.375757] [27246.375758] irq event stamp: 139782108 [27246.378809] ? wake_bit_function+0x70/0x70 [27246.382291] hardirqs last enabled at (139782107): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27246.387079] ? ceph_write_iter+0x260/0x770 [ceph] [27246.390300] hardirqs last disabled at (139782108): [] sysvec_apic_timer_interrupt+0xe/0x90 [27246.394393] ceph_write_iter+0x260/0x770 [ceph] [27246.398741] softirqs last enabled at (139782106): [] __do_softirq+0x2be/0x371 [27246.402835] ? lock_is_held_type+0xaa/0x120 [27246.406923] softirqs last disabled at (139782101): [] irq_exit_rcu+0xb0/0xd0 [27246.411971] vfs_write+0x324/0x4f0 [27246.416751] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27246.418853] ksys_pwrite64+0x65/0xa0 [27246.420945] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27246.426077] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27246.431555] Workqueue: events delayed_work [ceph] [27246.450292] do_syscall_64+0x38/0x80 [27246.455509] [27246.455510] RIP: 0010:rcu_is_watching+0x0/0x40 [27246.462634] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27246.469757] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [27246.476882] RIP: 0033:0x7f48f0eeb93b [27246.484006] RSP: 0018:ffffc90002117d10 EFLAGS: 00000247 [27246.491131] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27246.494961] [27246.494962] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [27246.499745] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27246.504355] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [27246.508880] ORIG_RAX: 0000000000000012 [27246.513229] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [27246.517320] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27246.522018] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [27246.526543] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27246.530720] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27246.534117] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27246.537688] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27246.542819] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27246.546406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27246.551452] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27246.555022] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27246.573765] [27246.581318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27247.439480] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27247.446604] Call Trace: [27247.449049] [27247.451059] ? watchdog_timer_fn+0x1b4/0x220 [27247.455325] ? __hrtimer_run_queues+0x1d3/0x360 [27247.459850] ? hrtimer_interrupt+0x104/0x240 [27247.464135] ? __local_bh_enable+0x50/0x60 [27247.468232] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27247.473454] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27247.478413] [27247.480510] [27247.482607] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27247.487915] ? rcutorture_get_gp_data+0x30/0x30 [27247.492444] lock_acquire+0x184/0x2b0 [27247.496101] ? igrab+0x43/0x50 [27247.499151] ? lock_release+0xc5/0x260 [27247.502897] _raw_spin_lock+0x30/0x40 [27247.506553] ? igrab+0x1d/0x50 [27247.509603] igrab+0x1d/0x50 [27247.512483] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27247.517631] delayed_work+0x1d8/0x280 [ceph] [27247.521911] ? process_scheduled_works+0x22a/0x510 [27247.526694] process_scheduled_works+0x2a4/0x510 [27247.531311] worker_thread+0x188/0x340 [27247.535059] ? rescuer_thread+0x310/0x310 [27247.539064] kthread+0xf5/0x130 [27247.542200] ? kthread_complete_and_exit+0x20/0x20 [27247.546985] ret_from_fork+0x31/0x40 [27247.550563] ? kthread_complete_and_exit+0x20/0x20 [27247.555347] ret_from_fork_asm+0x11/0x20 [27247.559271] [27273.935200] watchdog: BUG: soft lockup - CPU#0 stuck for 14286s! [dbench:78683] [27273.942504] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27273.947199] watchdog: BUG: soft lockup - CPU#7 stuck for 23283s! [kworker/7:4:78808] [27273.997945] irq event stamp: 44727482 [27273.997946] hardirqs last enabled at (44727481): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27274.005676] Modules linked in: [27274.009335] hardirqs last disabled at (44727482): [] sysvec_apic_timer_interrupt+0xe/0x90 [27274.019492] ceph [27274.022542] softirqs last enabled at (44727424): [] __do_softirq+0x2be/0x371 [27274.032266] libceph [27274.034191] softirqs last disabled at (44727419): [] irq_exit_rcu+0xb0/0xd0 [27274.042876] dns_resolver [27274.045060] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27274.053569] fscache [27274.056204] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27274.065149] netfs veth [27274.067332] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27274.074805] nft_chain_nat xt_MASQUERADE [27274.077248] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27274.082725] nf_nat nf_conntrack [27274.086643] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27274.105381] nf_defrag_ipv6 [27274.108605] [27274.113822] nf_defrag_ipv4 nft_compat [27274.116613] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27274.118104] nf_tables [27274.121847] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27274.128971] nfnetlink bridge [27274.131328] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27274.138453] stp llc [27274.141418] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27274.148542] xfs libcrc32c [27274.150726] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27274.157849] sunrpc intel_rapl_msr [27274.160554] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27274.167677] intel_rapl_common sb_edac [27274.171075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27274.179153] x86_pkg_temp_thermal coretemp [27274.182897] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27274.188634] kvm_intel [27274.192723] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27274.199848] kvm irqbypass [27274.202207] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27274.209330] i2c_i801 iTCO_wdt [27274.212034] Call Trace: [27274.212036] [27274.219158] iTCO_vendor_support [27274.222212] ? watchdog_timer_fn+0x1b4/0x220 [27274.224652] i2c_smbus crct10dif_pclmul [27274.226664] ? __hrtimer_run_queues+0x1d3/0x360 [27274.229887] joydev [27274.234154] ? hrtimer_interrupt+0x104/0x240 [27274.237983] crc32_pclmul ipmi_ssif [27274.242507] ? __local_bh_enable+0x50/0x60 [27274.244605] ghash_clmulni_intel [27274.248881] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27274.252361] mei_me mei [27274.256452] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27274.259677] lpc_ich [27274.264893] [27274.267338] sha512_ssse3 acpi_ipmi [27274.272296] [27274.274478] mfd_core wmi [27274.276594] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27274.280077] ipmi_si ipmi_devintf [27274.282178] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27274.284792] ipmi_msghandler ioatdma [27274.290097] do_raw_spin_lock+0xb5/0xc0 [27274.293407] acpi_pad acpi_power_meter [27274.298280] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27274.301848] ixgbe igb [27274.305682] try_get_cap_refs+0x159/0x880 [ceph] [27274.309422] mdio crc32c_intel [27274.314213] __ceph_get_caps+0x4a1/0x760 [ceph] [27274.316566] nvme ptp [27274.321180] ? __file_remove_privs+0x89/0x110 [27274.324227] nvme_core [27274.328749] ? wake_bit_function+0x70/0x70 [27274.331021] i2c_algo_bit pps_core [27274.335373] ? ceph_write_iter+0x260/0x770 [ceph] [27274.337729] dca fuse [27274.341819] ceph_write_iter+0x260/0x770 [ceph] [27274.345218] [27274.345219] irq event stamp: 139996270 [27274.349919] ? lock_is_held_type+0xaa/0x120 [27274.352186] hardirqs last enabled at (139996269): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27274.356711] vfs_write+0x324/0x4f0 [27274.358199] hardirqs last disabled at (139996270): [] sysvec_apic_timer_interrupt+0xe/0x90 [27274.361948] ksys_pwrite64+0x65/0xa0 [27274.366121] softirqs last enabled at (139996268): [] __do_softirq+0x2be/0x371 [27274.376366] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27274.379762] softirqs last disabled at (139996263): [] irq_exit_rcu+0xb0/0xd0 [27274.389576] do_syscall_64+0x38/0x80 [27274.393162] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27274.401934] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27274.407065] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27274.415663] RIP: 0033:0x7f48f0eeb93b [27274.419232] Workqueue: events delayed_work [ceph] [27274.428608] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27274.433652] [27274.433653] RIP: 0010:lock_acquire+0xf0/0x2b0 [27274.441124] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27274.444695] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27274.449393] ORIG_RAX: 0000000000000012 [27274.468131] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27274.469621] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27274.473971] [27274.473972] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27274.479190] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27274.497926] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27274.501757] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27274.506974] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27274.514098] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27274.515602] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27274.522723] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27274.529845] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27274.536977] [27274.544095] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27274.589985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27274.595723] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27274.602847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27274.609971] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27274.617098] Call Trace: [27274.619541] [27274.621552] ? watchdog_timer_fn+0x1b4/0x220 [27274.625815] ? __hrtimer_run_queues+0x1d3/0x360 [27274.630342] ? hrtimer_interrupt+0x104/0x240 [27274.634612] ? __local_bh_enable+0x50/0x60 [27274.638704] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27274.643930] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27274.648887] [27274.650983] [27274.653081] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27274.658389] ? lock_acquire+0xf0/0x2b0 [27274.662138] ? igrab+0x43/0x50 [27274.665188] ? lock_release+0xc5/0x260 [27274.668935] _raw_spin_lock+0x30/0x40 [27274.672604] ? igrab+0x1d/0x50 [27274.675659] igrab+0x1d/0x50 [27274.678538] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27274.683687] delayed_work+0x1d8/0x280 [ceph] [27274.687969] ? process_scheduled_works+0x22a/0x510 [27274.692756] process_scheduled_works+0x2a4/0x510 [27274.697373] worker_thread+0x188/0x340 [27274.701123] ? rescuer_thread+0x310/0x310 [27274.705134] kthread+0xf5/0x130 [27274.708281] ? kthread_complete_and_exit+0x20/0x20 [27274.713065] ret_from_fork+0x31/0x40 [27274.716644] ? kthread_complete_and_exit+0x20/0x20 [27274.721427] ret_from_fork_asm+0x11/0x20 [27274.725350] [27301.935685] watchdog: BUG: soft lockup - CPU#0 stuck for 14312s! [dbench:78683] [27301.942984] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27301.947685] watchdog: BUG: soft lockup - CPU#7 stuck for 23309s! [kworker/7:4:78808] [27301.998426] irq event stamp: 44785674 [27301.998427] hardirqs last enabled at (44785673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27302.006155] Modules linked in: ceph [27302.009814] hardirqs last disabled at (44785674): [] sysvec_apic_timer_interrupt+0xe/0x90 [27302.019970] libceph [27302.023456] softirqs last enabled at (44785616): [] __do_softirq+0x2be/0x371 [27302.033178] dns_resolver fscache [27302.035363] softirqs last disabled at (44785611): [] irq_exit_rcu+0xb0/0xd0 [27302.044046] netfs [27302.047374] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27302.055886] veth nft_chain_nat [27302.057897] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27302.057898] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27302.066841] xt_MASQUERADE nf_nat [27302.069979] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27302.077450] nf_conntrack nf_defrag_ipv6 [27302.082927] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27302.086237] nf_defrag_ipv4 nft_compat [27302.104974] [27302.104975] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27302.108893] nf_tables nfnetlink [27302.114110] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27302.117854] bridge stp [27302.119344] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27302.126469] llc xfs [27302.129692] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27302.136817] libcrc32c sunrpc [27302.139260] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27302.146385] intel_rapl_msr [27302.148569] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27302.155692] intel_rapl_common [27302.158656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27302.165780] sb_edac x86_pkg_temp_thermal [27302.168571] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27302.176650] coretemp kvm_intel [27302.179701] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27302.185438] kvm irqbypass [27302.189441] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27302.196565] i2c_i801 iTCO_wdt [27302.199703] Call Trace: [27302.199705] [27302.206827] iTCO_vendor_support i2c_smbus [27302.209532] ? watchdog_timer_fn+0x1b4/0x220 [27302.216656] crct10dif_pclmul joydev [27302.219707] ? __hrtimer_run_queues+0x1d3/0x360 [27302.222150] crc32_pclmul ipmi_ssif [27302.224164] ? hrtimer_interrupt+0x104/0x240 [27302.228251] ghash_clmulni_intel mei_me [27302.232516] ? __local_bh_enable+0x50/0x60 [27302.236085] mei lpc_ich [27302.240614] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27302.244094] sha512_ssse3 acpi_ipmi [27302.248365] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27302.252199] mfd_core wmi [27302.256290] [27302.258819] ipmi_si ipmi_devintf [27302.264037] [27302.267521] ipmi_msghandler ioatdma [27302.272478] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27302.275095] acpi_pad [27302.277198] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27302.280503] acpi_power_meter ixgbe [27302.282602] do_raw_spin_lock+0xb5/0xc0 [27302.286171] igb mdio [27302.291479] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27302.293747] crc32c_intel nvme [27302.298621] try_get_cap_refs+0x159/0x880 [ceph] [27302.302101] ptp nvme_core [27302.305938] __ceph_get_caps+0x4a1/0x760 [ceph] [27302.308204] i2c_algo_bit pps_core [27302.312992] ? __file_remove_privs+0x89/0x110 [27302.316038] dca fuse [27302.320648] ? wake_bit_function+0x70/0x70 [27302.323353] [27302.323354] irq event stamp: 140217188 [27302.327898] ? ceph_write_iter+0x260/0x770 [ceph] [27302.331291] hardirqs last enabled at (140217187): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27302.335643] ceph_write_iter+0x260/0x770 [ceph] [27302.337913] hardirqs last disabled at (140217188): [] sysvec_apic_timer_interrupt+0xe/0x90 [27302.342008] ? lock_is_held_type+0xaa/0x120 [27302.343494] softirqs last enabled at (140217186): [] __do_softirq+0x2be/0x371 [27302.347241] vfs_write+0x324/0x4f0 [27302.351935] softirqs last disabled at (140217181): [] irq_exit_rcu+0xb0/0xd0 [27302.362184] ksys_pwrite64+0x65/0xa0 [27302.366704] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27302.376516] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27302.380694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27302.389465] do_syscall_64+0x38/0x80 [27302.392860] Workqueue: events delayed_work [ceph] [27302.401460] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27302.405029] [27302.405030] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [27302.414407] RIP: 0033:0x7f48f0eeb93b [27302.419538] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [27302.427008] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27302.430578] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [27302.435277] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27302.440321] [27302.440322] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [27302.441810] ORIG_RAX: 0000000000000012 [27302.446423] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [27302.449993] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27302.468731] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [27302.487467] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27302.492684] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27302.497904] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27302.499392] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27302.506517] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27302.510347] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27302.517491] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27302.524615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27302.531743] [27302.538862] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27302.597608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27302.604731] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27302.611854] Call Trace: [27302.614299] [27302.616309] ? watchdog_timer_fn+0x1b4/0x220 [27302.620573] ? __hrtimer_run_queues+0x1d3/0x360 [27302.625102] ? hrtimer_interrupt+0x104/0x240 [27302.629372] ? __local_bh_enable+0x50/0x60 [27302.633473] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27302.638696] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27302.643652] [27302.645749] [27302.647848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27302.653155] ? do_raw_spin_lock+0x42/0xc0 [27302.657167] igrab+0x1d/0x50 [27302.660052] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27302.665220] delayed_work+0x1d8/0x280 [ceph] [27302.669500] ? process_scheduled_works+0x22a/0x510 [27302.674290] process_scheduled_works+0x2a4/0x510 [27302.678906] worker_thread+0x188/0x340 [27302.682655] ? rescuer_thread+0x310/0x310 [27302.686658] kthread+0xf5/0x130 [27302.689796] ? kthread_complete_and_exit+0x20/0x20 [27302.694582] ret_from_fork+0x31/0x40 [27302.698158] ? kthread_complete_and_exit+0x20/0x20 [27302.702944] ret_from_fork_asm+0x11/0x20 [27302.706865] [27329.936174] watchdog: BUG: soft lockup - CPU#0 stuck for 14338s! [dbench:78683] [27329.943482] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27329.948173] watchdog: BUG: soft lockup - CPU#7 stuck for 23335s! [kworker/7:4:78808] [27329.998930] irq event stamp: 44843890 [27330.006660] Modules linked in: ceph libceph dns_resolver fscache netfs [27330.010317] hardirqs last enabled at (44843889): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27330.016834] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [27330.027011] hardirqs last disabled at (44843890): [] sysvec_apic_timer_interrupt+0xe/0x90 [27330.034402] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [27330.044127] softirqs last enabled at (44843832): [] __do_softirq+0x2be/0x371 [27330.050570] llc xfs libcrc32c sunrpc intel_rapl_msr [27330.059250] softirqs last disabled at (44843827): [] irq_exit_rcu+0xb0/0xd0 [27330.059253] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [27330.072720] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27330.080275] irqbypass i2c_i801 iTCO_wdt [27330.089222] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27330.089223] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27330.093138] iTCO_vendor_support i2c_smbus [27330.100608] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27330.106087] crct10dif_pclmul joydev [27330.110178] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27330.128914] crc32_pclmul [27330.132485] [27330.132486] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27330.137702] ipmi_ssif [27330.140319] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27330.141812] ghash_clmulni_intel mei_me [27330.148935] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27330.151293] mei lpc_ich [27330.158415] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27330.162247] sha512_ssse3 [27330.169370] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27330.171901] acpi_ipmi mfd_core [27330.179026] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27330.181643] wmi ipmi_si [27330.188768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27330.191906] ipmi_devintf [27330.199983] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27330.202515] ipmi_msghandler [27330.208250] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27330.210869] ioatdma acpi_pad [27330.217991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27330.220870] acpi_power_meter [27330.227993] Call Trace: [27330.227996] [27330.230957] ixgbe [27330.238084] ? watchdog_timer_fn+0x1b4/0x220 [27330.241047] igb mdio [27330.243491] ? __hrtimer_run_queues+0x1d3/0x360 [27330.245503] crc32c_intel [27330.247515] ? hrtimer_interrupt+0x104/0x240 [27330.251776] nvme ptp [27330.254045] ? __local_bh_enable+0x50/0x60 [27330.258573] nvme_core [27330.261191] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27330.265451] i2c_algo_bit [27330.267722] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27330.271813] pps_core [27330.274171] [27330.279387] dca fuse [27330.282006] [27330.286962] [27330.286963] irq event stamp: 140438206 [27330.289234] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27330.291332] hardirqs last enabled at (140438205): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27330.293607] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27330.295699] hardirqs last disabled at (140438206): [] sysvec_apic_timer_interrupt+0xe/0x90 [27330.297193] do_raw_spin_lock+0xb5/0xc0 [27330.300933] softirqs last enabled at (140438204): [] __do_softirq+0x2be/0x371 [27330.306238] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27330.316481] softirqs last disabled at (140438199): [] irq_exit_rcu+0xb0/0xd0 [27330.321355] try_get_cap_refs+0x159/0x880 [ceph] [27330.331164] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27330.335002] __ceph_get_caps+0x4a1/0x760 [ceph] [27330.343765] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27330.348571] ? __file_remove_privs+0x89/0x110 [27330.357165] Workqueue: events delayed_work [ceph] [27330.361777] ? wake_bit_function+0x70/0x70 [27330.371153] [27330.371154] RIP: 0010:lock_acquire+0xf0/0x2b0 [27330.375680] ? ceph_write_iter+0x260/0x770 [ceph] [27330.383147] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27330.387497] ceph_write_iter+0x260/0x770 [ceph] [27330.392194] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27330.396290] ? lock_is_held_type+0xaa/0x120 [27330.397777] [27330.397778] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27330.402130] vfs_write+0x324/0x4f0 [27330.406826] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27330.425579] ksys_pwrite64+0x65/0xa0 [27330.430095] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27330.435315] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27330.439489] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27330.440983] do_syscall_64+0x38/0x80 [27330.448105] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27330.451504] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27330.458628] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27330.462200] RIP: 0033:0x7f48f0eeb93b [27330.469321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27330.474452] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27330.481576] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27330.485147] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27330.492273] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27330.497316] ORIG_RAX: 0000000000000012 [27330.505393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27330.508964] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27330.514700] Call Trace: [27330.514702] [27330.533439] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27330.540576] ? watchdog_timer_fn+0x1b4/0x220 [27330.545789] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27330.552915] ? __hrtimer_run_queues+0x1d3/0x360 [27330.556744] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27330.563871] ? hrtimer_interrupt+0x104/0x240 [27330.570992] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27330.573435] ? __local_bh_enable+0x50/0x60 [27330.575451] [27330.582582] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27330.628515] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27330.633471] [27330.635580] [27330.637676] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27330.642984] ? lock_acquire+0xf0/0x2b0 [27330.646734] ? igrab+0x43/0x50 [27330.649792] ? lock_release+0xc5/0x260 [27330.653536] _raw_spin_lock+0x30/0x40 [27330.657193] ? igrab+0x1d/0x50 [27330.660244] igrab+0x1d/0x50 [27330.663124] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27330.668272] delayed_work+0x1d8/0x280 [ceph] [27330.672555] ? process_scheduled_works+0x22a/0x510 [27330.677361] process_scheduled_works+0x2a4/0x510 [27330.681978] worker_thread+0x188/0x340 [27330.685726] ? rescuer_thread+0x310/0x310 [27330.689727] kthread+0xf5/0x130 [27330.692875] ? kthread_complete_and_exit+0x20/0x20 [27330.697660] ret_from_fork+0x31/0x40 [27330.701239] ? kthread_complete_and_exit+0x20/0x20 [27330.706023] ret_from_fork_asm+0x11/0x20 [27330.709954] [27357.936661] watchdog: BUG: soft lockup - CPU#0 stuck for 14365s! [dbench:78683] [27357.943960] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27357.948660] watchdog: BUG: soft lockup - CPU#7 stuck for 23362s! [kworker/7:4:78808] [27357.999400] irq event stamp: 44902044 [27357.999401] hardirqs last enabled at (44902043): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27358.007132] Modules linked in: [27358.010788] hardirqs last disabled at (44902044): [] sysvec_apic_timer_interrupt+0xe/0x90 [27358.020946] ceph [27358.023997] softirqs last enabled at (44902028): [] __do_softirq+0x2be/0x371 [27358.033721] libceph [27358.035644] softirqs last disabled at (44902023): [] irq_exit_rcu+0xb0/0xd0 [27358.044335] dns_resolver fscache [27358.046522] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27358.055032] netfs veth [27358.058359] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27358.067306] nft_chain_nat xt_MASQUERADE [27358.069748] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27358.077219] nf_nat nf_conntrack [27358.081136] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27358.086613] nf_defrag_ipv6 nf_defrag_ipv4 [27358.089837] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27358.108577] nft_compat nf_tables [27358.112666] [27358.112667] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27358.117883] nfnetlink [27358.121195] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27358.122687] bridge stp [27358.129810] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27358.132168] llc xfs [27358.139290] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27358.141735] libcrc32c sunrpc [27358.148860] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27358.151057] intel_rapl_msr intel_rapl_common [27358.158185] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27358.161148] sb_edac x86_pkg_temp_thermal [27358.168273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27358.172625] coretemp kvm_intel [27358.180700] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27358.184705] kvm [27358.190442] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27358.193580] irqbypass i2c_i801 [27358.200703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27358.202541] iTCO_wdt [27358.209666] Call Trace: [27358.212803] iTCO_vendor_support [27358.219928] [27358.222199] i2c_smbus crct10dif_pclmul [27358.224643] ? watchdog_timer_fn+0x1b4/0x220 [27358.227866] joydev [27358.229878] ? __hrtimer_run_queues+0x1d3/0x360 [27358.233708] crc32_pclmul ipmi_ssif [27358.237974] ? hrtimer_interrupt+0x104/0x240 [27358.240069] ghash_clmulni_intel mei_me [27358.244594] ? __local_bh_enable+0x50/0x60 [27358.248076] mei lpc_ich [27358.252343] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27358.256171] sha512_ssse3 [27358.260264] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27358.262794] acpi_ipmi mfd_core [27358.268010] [27358.270629] wmi ipmi_si [27358.275586] [27358.275587] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27358.278723] ipmi_devintf ipmi_msghandler [27358.280825] ? queued_spin_lock_slowpath+0x18/0x2e0 [27358.283353] ioatdma acpi_pad [27358.285450] do_raw_spin_lock+0xb5/0xc0 [27358.290753] acpi_power_meter ixgbe [27358.294758] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27358.299628] igb mdio [27358.302593] try_get_cap_refs+0x159/0x880 [ceph] [27358.306423] crc32c_intel nvme [27358.309911] __ceph_get_caps+0x4a1/0x760 [ceph] [27358.314691] ptp nvme_core [27358.316965] ? __file_remove_privs+0x89/0x110 [27358.321571] i2c_algo_bit pps_core [27358.324623] ? wake_bit_function+0x70/0x70 [27358.329147] dca fuse [27358.331854] ? ceph_write_iter+0x260/0x770 [ceph] [27358.336202] [27358.336203] irq event stamp: 140659066 [27358.339600] ceph_write_iter+0x260/0x770 [ceph] [27358.343689] hardirqs last enabled at (140659065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27358.345964] ? lock_is_held_type+0xaa/0x120 [27358.350658] hardirqs last disabled at (140659066): [] sysvec_apic_timer_interrupt+0xe/0x90 [27358.352150] vfs_write+0x324/0x4f0 [27358.355892] softirqs last enabled at (140659064): [] __do_softirq+0x2be/0x371 [27358.360420] ksys_pwrite64+0x65/0xa0 [27358.370659] softirqs last disabled at (140659059): [] irq_exit_rcu+0xb0/0xd0 [27358.374839] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27358.384649] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27358.388064] do_syscall_64+0x38/0x80 [27358.396834] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27358.400405] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27358.409002] Workqueue: events delayed_work [ceph] [27358.414134] RIP: 0033:0x7f48f0eeb93b [27358.423511] [27358.427081] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27358.434554] RIP: 0010:lock_acquire+0xf0/0x2b0 [27358.439598] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27358.444294] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27358.447866] ORIG_RAX: 0000000000000012 [27358.449356] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27358.468093] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27358.472444] [27358.477661] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27358.496400] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27358.500230] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27358.505446] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27358.512571] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27358.514062] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27358.521186] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27358.528310] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27358.535440] [27358.542560] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27358.580373] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27358.588449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27358.594188] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27358.601311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27358.608434] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27358.615561] Call Trace: [27358.618005] [27358.620016] ? watchdog_timer_fn+0x1b4/0x220 [27358.624280] ? __hrtimer_run_queues+0x1d3/0x360 [27358.628805] ? hrtimer_interrupt+0x104/0x240 [27358.633074] ? __local_bh_enable+0x50/0x60 [27358.637168] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27358.642384] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27358.647346] [27358.649447] [27358.651546] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27358.656854] ? lock_acquire+0xf0/0x2b0 [27358.660602] ? igrab+0x43/0x50 [27358.663652] ? lock_release+0xc5/0x260 [27358.667397] _raw_spin_lock+0x30/0x40 [27358.671068] ? igrab+0x1d/0x50 [27358.674121] igrab+0x1d/0x50 [27358.677001] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27358.682148] delayed_work+0x1d8/0x280 [ceph] [27358.686430] ? process_scheduled_works+0x22a/0x510 [27358.691222] process_scheduled_works+0x2a4/0x510 [27358.695836] worker_thread+0x188/0x340 [27358.699585] ? rescuer_thread+0x310/0x310 [27358.703590] kthread+0xf5/0x130 [27358.706727] ? kthread_complete_and_exit+0x20/0x20 [27358.711512] ret_from_fork+0x31/0x40 [27358.715088] ? kthread_complete_and_exit+0x20/0x20 [27358.719875] ret_from_fork_asm+0x11/0x20 [27358.723805] [27385.937149] watchdog: BUG: soft lockup - CPU#0 stuck for 14391s! [dbench:78683] [27385.944456] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27385.949148] watchdog: BUG: soft lockup - CPU#7 stuck for 23388s! [kworker/7:4:78808] [27385.999907] irq event stamp: 44960280 [27386.007636] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [27386.011292] hardirqs last enabled at (44960279): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27386.019465] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [27386.029623] hardirqs last disabled at (44960280): [] sysvec_apic_timer_interrupt+0xe/0x90 [27386.029627] nf_tables nfnetlink bridge stp llc xfs [27386.047347] softirqs last enabled at (44960222): [] __do_softirq+0x2be/0x371 [27386.052218] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [27386.060900] softirqs last disabled at (44960217): [] irq_exit_rcu+0xb0/0xd0 [27386.070019] kvm_intel kvm irqbypass [27386.078544] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27386.078546] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27386.082118] i2c_i801 iTCO_wdt [27386.091064] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27386.098545] iTCO_vendor_support [27386.101594] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27386.107069] i2c_smbus crct10dif_pclmul [27386.110294] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27386.129031] joydev crc32_pclmul [27386.132864] [27386.138079] ipmi_ssif [27386.141303] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27386.142796] ghash_clmulni_intel mei_me [27386.145152] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27386.152275] mei lpc_ich [27386.156106] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27386.163230] sha512_ssse3 [27386.165761] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27386.172887] acpi_ipmi mfd_core [27386.175502] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27386.182628] wmi ipmi_si [27386.185764] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27386.192889] ipmi_devintf [27386.195420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27386.203497] ipmi_msghandler [27386.206113] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27386.211852] ioatdma [27386.214730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27386.221854] acpi_pad acpi_power_meter [27386.224039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27386.231162] ixgbe [27386.234905] Call Trace: [27386.234907] [27386.242031] igb mdio [27386.244043] ? watchdog_timer_fn+0x1b4/0x220 [27386.246484] crc32c_intel [27386.248498] ? __hrtimer_run_queues+0x1d3/0x360 [27386.250766] nvme ptp [27386.255033] ? hrtimer_interrupt+0x104/0x240 [27386.257647] nvme_core i2c_algo_bit [27386.262172] ? __local_bh_enable+0x50/0x60 [27386.264443] pps_core [27386.268710] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27386.272191] dca fuse [27386.276282] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27386.278551] [27386.278551] irq event stamp: 140883130 [27386.283770] [27386.286040] hardirqs last enabled at (140883129): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27386.290996] [27386.290998] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27386.292487] hardirqs last disabled at (140883130): [] sysvec_apic_timer_interrupt+0xe/0x90 [27386.296235] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27386.298329] softirqs last enabled at (140883128): [] __do_softirq+0x2be/0x371 [27386.308577] do_raw_spin_lock+0xb5/0xc0 [27386.310671] softirqs last disabled at (140883123): [] irq_exit_rcu+0xb0/0xd0 [27386.315978] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27386.325787] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27386.330658] try_get_cap_refs+0x159/0x880 [ceph] [27386.339427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27386.343264] __ceph_get_caps+0x4a1/0x760 [ceph] [27386.351858] Workqueue: events delayed_work [ceph] [27386.356643] ? __file_remove_privs+0x89/0x110 [27386.366017] [27386.366018] RIP: 0010:lock_acquire+0xf0/0x2b0 [27386.370630] ? wake_bit_function+0x70/0x70 [27386.378098] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27386.382626] ? ceph_write_iter+0x260/0x770 [ceph] [27386.387321] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27386.391673] ceph_write_iter+0x260/0x770 [ceph] [27386.393161] [27386.397516] ? lock_is_held_type+0xaa/0x120 [27386.401604] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27386.420344] vfs_write+0x324/0x4f0 [27386.425039] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27386.430262] ksys_pwrite64+0x65/0xa0 [27386.434780] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27386.436270] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27386.440447] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27386.447573] do_syscall_64+0x38/0x80 [27386.450971] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27386.458095] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27386.461664] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27386.468788] RIP: 0033:0x7f48f0eeb93b [27386.473921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27386.481045] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27386.484613] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27386.491739] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27386.496784] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27386.504860] ORIG_RAX: 0000000000000012 [27386.508430] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27386.514168] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27386.532905] Call Trace: [27386.532906] [27386.540031] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27386.545248] ? watchdog_timer_fn+0x1b4/0x220 [27386.552372] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27386.556204] ? __hrtimer_run_queues+0x1d3/0x360 [27386.563326] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27386.570455] ? hrtimer_interrupt+0x104/0x240 [27386.572896] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27386.574905] ? __local_bh_enable+0x50/0x60 [27386.582035] [27386.586296] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27386.627957] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27386.632912] [27386.635011] [27386.637108] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27386.642416] ? lock_acquire+0xf0/0x2b0 [27386.646165] ? igrab+0x43/0x50 [27386.649217] ? lock_release+0xc5/0x260 [27386.652961] _raw_spin_lock+0x30/0x40 [27386.656616] ? igrab+0x1d/0x50 [27386.659667] igrab+0x1d/0x50 [27386.662554] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27386.667705] delayed_work+0x1d8/0x280 [ceph] [27386.671985] ? process_scheduled_works+0x22a/0x510 [27386.676776] process_scheduled_works+0x2a4/0x510 [27386.681391] worker_thread+0x188/0x340 [27386.685140] ? rescuer_thread+0x310/0x310 [27386.689144] kthread+0xf5/0x130 [27386.692283] ? kthread_complete_and_exit+0x20/0x20 [27386.697076] ret_from_fork+0x31/0x40 [27386.700653] ? kthread_complete_and_exit+0x20/0x20 [27386.705438] ret_from_fork_asm+0x11/0x20 [27386.709368] [27413.937637] watchdog: BUG: soft lockup - CPU#0 stuck for 14417s! [dbench:78683] [27413.944943] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27413.949636] watchdog: BUG: soft lockup - CPU#7 stuck for 23414s! [kworker/7:4:78808] [27414.000391] irq event stamp: 45018448 [27414.000392] hardirqs last enabled at (45018447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27414.008123] Modules linked in: [27414.011780] hardirqs last disabled at (45018448): [] sysvec_apic_timer_interrupt+0xe/0x90 [27414.021939] ceph [27414.024988] softirqs last enabled at (45018438): [] __do_softirq+0x2be/0x371 [27414.034713] libceph [27414.036637] softirqs last disabled at (45018433): [] irq_exit_rcu+0xb0/0xd0 [27414.045322] dns_resolver [27414.047507] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27414.056032] fscache [27414.058651] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27414.067596] netfs veth [27414.069780] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27414.077250] nft_chain_nat [27414.079694] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27414.085171] xt_MASQUERADE nf_nat [27414.087875] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27414.106614] nf_conntrack [27414.109924] [27414.109925] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27414.115140] nf_defrag_ipv6 nf_defrag_ipv4 [27414.117759] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27414.119250] nft_compat nf_tables [27414.126372] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27414.130465] nfnetlink bridge [27414.137588] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27414.140898] stp llc [27414.148035] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27414.150996] xfs [27414.158119] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27414.160314] libcrc32c [27414.167438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27414.169274] sunrpc [27414.177350] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27414.179709] intel_rapl_msr intel_rapl_common [27414.185447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27414.187543] sb_edac [27414.194668] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27414.199035] x86_pkg_temp_thermal coretemp [27414.206161] Call Trace: [27414.206163] [27414.208345] kvm_intel kvm [27414.215470] ? watchdog_timer_fn+0x1b4/0x220 [27414.219560] irqbypass i2c_i801 [27414.222003] ? __hrtimer_run_queues+0x1d3/0x360 [27414.224032] iTCO_wdt [27414.226739] ? hrtimer_interrupt+0x104/0x240 [27414.230998] iTCO_vendor_support i2c_smbus [27414.234137] ? __local_bh_enable+0x50/0x60 [27414.238661] crct10dif_pclmul joydev [27414.240935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27414.245195] crc32_pclmul ipmi_ssif [27414.249286] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27414.253376] ghash_clmulni_intel [27414.256947] [27414.262165] mei_me mei [27414.265649] [27414.265650] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27414.270607] lpc_ich [27414.273834] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27414.275929] sha512_ssse3 [27414.278375] do_raw_spin_lock+0xb5/0xc0 [27414.280469] acpi_ipmi mfd_core [27414.285776] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27414.287958] wmi ipmi_si [27414.292832] try_get_cap_refs+0x159/0x880 [ceph] [27414.295448] ipmi_devintf [27414.299282] __ceph_get_caps+0x4a1/0x760 [ceph] [27414.302416] ipmi_msghandler [27414.307201] ? __file_remove_privs+0x89/0x110 [27414.309728] ioatdma [27414.314339] ? wake_bit_function+0x70/0x70 [27414.316956] acpi_pad acpi_power_meter [27414.321483] ? ceph_write_iter+0x260/0x770 [ceph] [27414.324359] ixgbe igb [27414.328710] ceph_write_iter+0x260/0x770 [ceph] [27414.330893] mdio [27414.334988] ? lock_is_held_type+0xaa/0x120 [27414.338727] crc32c_intel nvme [27414.343427] vfs_write+0x324/0x4f0 [27414.345783] ptp nvme_core [27414.350320] ksys_pwrite64+0x65/0xa0 [27414.352239] i2c_algo_bit pps_core [27414.356419] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27414.359467] dca [27414.362867] do_syscall_64+0x38/0x80 [27414.365569] fuse [27414.369141] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27414.372538] [27414.372539] irq event stamp: 141104084 [27414.377669] RIP: 0033:0x7f48f0eeb93b [27414.379505] hardirqs last enabled at (141104083): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27414.383075] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27414.385001] hardirqs last disabled at (141104084): [] sysvec_apic_timer_interrupt+0xe/0x90 [27414.390044] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27414.391536] softirqs last enabled at (141104082): [] __do_softirq+0x2be/0x371 [27414.395279] ORIG_RAX: 0000000000000012 [27414.398848] softirqs last disabled at (141104077): [] irq_exit_rcu+0xb0/0xd0 [27414.409094] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27414.427831] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27414.437643] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27414.442860] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27414.451630] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27414.455461] Workqueue: events delayed_work [ceph] [27414.464059] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27414.471183] [27414.471183] RIP: 0010:lock_acquire+0xf0/0x2b0 [27414.480560] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27414.487684] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27414.495160] [27414.502278] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27414.553206] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27414.560331] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27414.567454] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27414.574580] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27414.581702] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27414.588827] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27414.596904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27414.602640] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27414.609766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27414.616890] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27414.624014] Call Trace: [27414.626460] [27414.628470] ? watchdog_timer_fn+0x1b4/0x220 [27414.632735] ? __hrtimer_run_queues+0x1d3/0x360 [27414.637261] ? hrtimer_interrupt+0x104/0x240 [27414.641530] ? __local_bh_enable+0x50/0x60 [27414.645622] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27414.650837] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27414.655796] [27414.657892] [27414.659992] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27414.665300] ? lock_acquire+0xf0/0x2b0 [27414.669050] ? igrab+0x43/0x50 [27414.672107] ? lock_release+0xc5/0x260 [27414.675851] _raw_spin_lock+0x30/0x40 [27414.679508] ? igrab+0x1d/0x50 [27414.682558] igrab+0x1d/0x50 [27414.685438] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27414.690587] delayed_work+0x1d8/0x280 [ceph] [27414.694867] ? process_scheduled_works+0x22a/0x510 [27414.699658] process_scheduled_works+0x2a4/0x510 [27414.704273] worker_thread+0x188/0x340 [27414.708024] ? rescuer_thread+0x310/0x310 [27414.712050] kthread+0xf5/0x130 [27414.715191] ? kthread_complete_and_exit+0x20/0x20 [27414.719984] ret_from_fork+0x31/0x40 [27414.723562] ? kthread_complete_and_exit+0x20/0x20 [27414.728347] ret_from_fork_asm+0x11/0x20 [27414.732278] [27441.122476] rcu: INFO: rcu_preempt self-detected stall on CPU [27441.128236] rcu: 0-....: (15156675 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7577848 [27441.138661] rcu: (t=15504238 jiffies g=1309609 q=475366 ncpus=8) [27441.144745] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27441.153688] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27441.161160] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27441.166636] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27441.185374] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27441.190593] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27441.197716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27441.204840] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27441.211966] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27441.219089] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27441.226214] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27441.234292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27441.240028] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27441.247151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27441.254275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27441.261399] Call Trace: [27441.263846] [27441.265856] ? rcu_dump_cpu_stacks+0xd1/0x180 [27441.270214] ? rcu_sched_clock_irq+0x633/0x1390 [27441.274741] ? lock_is_held_type+0xaa/0x120 [27441.278916] ? lock_is_held_type+0xaa/0x120 [27441.283096] ? tick_sched_do_timer+0xa0/0xa0 [27441.287368] ? rcu_read_lock_sched_held+0x56/0x90 [27441.292066] ? tick_sched_do_timer+0xa0/0xa0 [27441.296337] ? update_process_times+0x5f/0x90 [27441.300688] ? tick_sched_handle.isra.16+0x1e/0x50 [27441.305472] ? tick_sched_timer+0x6f/0x80 [27441.309475] ? __hrtimer_run_queues+0x1d3/0x360 [27441.314003] ? hrtimer_interrupt+0x104/0x240 [27441.318271] ? __local_bh_enable+0x50/0x60 [27441.322366] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27441.327588] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27441.332548] [27441.334644] [27441.336741] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27441.342050] ? queued_spin_lock_slowpath+0x16/0x2e0 [27441.346928] do_raw_spin_lock+0xb5/0xc0 [27441.350766] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27441.355567] try_get_cap_refs+0x159/0x880 [ceph] [27441.360198] __ceph_get_caps+0x4a1/0x760 [ceph] [27441.364738] ? __file_remove_privs+0x89/0x110 [27441.369094] ? wake_bit_function+0x70/0x70 [27441.373187] ? ceph_write_iter+0x260/0x770 [ceph] [27441.377901] ceph_write_iter+0x260/0x770 [ceph] [27441.382438] ? lock_is_held_type+0xaa/0x120 [27441.386622] vfs_write+0x324/0x4f0 [27441.390029] ksys_pwrite64+0x65/0xa0 [27441.393605] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27441.398744] do_syscall_64+0x38/0x80 [27441.402316] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27441.407358] RIP: 0033:0x7f48f0eeb93b [27441.410928] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27441.429666] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [27441.437223] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27441.444350] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27441.451473] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27441.458596] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27441.465720] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27441.472852] [27441.475040] Sending NMI from CPU 0 to CPUs 7: [27441.479391] NMI backtrace for cpu 7 [27441.479393] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27441.479395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27441.479396] Workqueue: events delayed_work [ceph] [27441.479413] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [27441.479416] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [27441.479418] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [27441.479419] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [27441.479420] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [27441.479420] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [27441.479421] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27441.479422] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27441.479423] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27441.479424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27441.479425] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27441.479425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27441.479426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27441.479427] Call Trace: [27441.479428] [27441.479429] ? nmi_cpu_backtrace+0xad/0x120 [27441.479433] ? nmi_cpu_backtrace_handler+0x11/0x20 [27441.479434] ? nmi_handle+0xd4/0x260 [27441.479438] ? default_do_nmi+0x49/0xf0 [27441.479440] ? exc_nmi+0x11e/0x150 [27441.479441] ? end_repeat_nmi+0x16/0x67 [27441.479445] ? __list_del_entry_valid_or_report+0x10/0xd0 [27441.479448] ? __list_del_entry_valid_or_report+0x10/0xd0 [27441.479450] ? __list_del_entry_valid_or_report+0x10/0xd0 [27441.479453] [27441.479453] [27441.479454] ceph_check_delayed_caps+0x8f/0x270 [ceph] [27441.479468] delayed_work+0x1d8/0x280 [ceph] [27441.479483] ? process_scheduled_works+0x22a/0x510 [27441.479486] process_scheduled_works+0x2a4/0x510 [27441.479490] worker_thread+0x188/0x340 [27441.479493] ? rescuer_thread+0x310/0x310 [27441.479495] kthread+0xf5/0x130 [27441.479497] ? kthread_complete_and_exit+0x20/0x20 [27441.479500] ret_from_fork+0x31/0x40 [27441.479503] ? kthread_complete_and_exit+0x20/0x20 [27441.479505] ret_from_fork_asm+0x11/0x20 [27441.479509] [27465.938542] watchdog: BUG: soft lockup - CPU#0 stuck for 14465s! [dbench:78683] [27465.945848] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27465.950542] watchdog: BUG: soft lockup - CPU#7 stuck for 23462s! [kworker/7:4:78808] [27466.001297] irq event stamp: 45130360 [27466.009030] Modules linked in: ceph libceph dns_resolver fscache [27466.012686] hardirqs last enabled at (45130359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27466.012691] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [27466.028850] hardirqs last disabled at (45130360): [] sysvec_apic_timer_interrupt+0xe/0x90 [27466.035454] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [27466.045179] softirqs last enabled at (45130342): [] __do_softirq+0x2be/0x371 [27466.045183] bridge stp llc xfs libcrc32c sunrpc [27466.060648] softirqs last disabled at (45130337): [] irq_exit_rcu+0xb0/0xd0 [27466.065260] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [27466.073772] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27466.081415] kvm_intel [27466.090358] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27466.092715] kvm irqbypass i2c_i801 iTCO_wdt [27466.100185] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27466.104451] iTCO_vendor_support i2c_smbus [27466.109940] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27466.109942] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27466.114036] crct10dif_pclmul [27466.132775] [27466.137992] joydev crc32_pclmul [27466.140954] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27466.142446] ipmi_ssif [27466.145669] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27466.152794] ghash_clmulni_intel [27466.155152] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27466.162276] mei_me [27466.165500] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27466.172624] mei lpc_ich [27466.174721] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27466.181844] sha512_ssse3 [27466.184376] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27466.191499] acpi_ipmi mfd_core [27466.194117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27466.202196] wmi ipmi_si [27466.205349] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27466.211087] ipmi_devintf [27466.213617] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27466.220742] ipmi_msghandler [27466.223360] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27466.230484] ioatdma acpi_pad [27466.233360] Call Trace: [27466.233363] [27466.240485] acpi_power_meter [27466.243450] ? watchdog_timer_fn+0x1b4/0x220 [27466.245893] ixgbe igb [27466.247905] ? __hrtimer_run_queues+0x1d3/0x360 [27466.250869] mdio crc32c_intel [27466.255136] ? hrtimer_interrupt+0x104/0x240 [27466.257489] nvme [27466.262012] ? __local_bh_enable+0x50/0x60 [27466.265064] ptp nvme_core [27466.269330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27466.271253] i2c_algo_bit pps_core [27466.275344] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27466.278046] dca [27466.283265] [27466.286662] fuse [27466.291619] [27466.291621] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27466.293455] [27466.293456] irq event stamp: 141517842 [27466.295557] ? queued_spin_lock_slowpath+0x16/0x2e0 [27466.297479] hardirqs last enabled at (141517841): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27466.299577] do_raw_spin_lock+0xb5/0xc0 [27466.304878] hardirqs last disabled at (141517842): [] sysvec_apic_timer_interrupt+0xe/0x90 [27466.306373] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27466.310114] softirqs last enabled at (141517840): [] __do_softirq+0x2be/0x371 [27466.314986] try_get_cap_refs+0x159/0x880 [ceph] [27466.325230] softirqs last disabled at (141517835): [] irq_exit_rcu+0xb0/0xd0 [27466.329064] __ceph_get_caps+0x4a1/0x760 [ceph] [27466.338870] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27466.343660] ? __file_remove_privs+0x89/0x110 [27466.352425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27466.357037] ? wake_bit_function+0x70/0x70 [27466.365636] Workqueue: events delayed_work [ceph] [27466.370161] ? ceph_write_iter+0x260/0x770 [ceph] [27466.379535] [27466.383887] ceph_write_iter+0x260/0x770 [ceph] [27466.391357] RIP: 0010:lock_acquire+0xf0/0x2b0 [27466.395451] ? lock_is_held_type+0xaa/0x120 [27466.400145] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27466.404845] vfs_write+0x324/0x4f0 [27466.406334] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27466.410863] ksys_pwrite64+0x65/0xa0 [27466.415207] [27466.415208] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27466.419387] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27466.438123] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27466.441521] do_syscall_64+0x38/0x80 [27466.446739] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27466.450310] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27466.451799] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27466.458941] RIP: 0033:0x7f48f0eeb93b [27466.464070] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27466.471196] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27466.474766] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27466.481892] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27466.486945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27466.494067] ORIG_RAX: 0000000000000012 [27466.497637] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27466.504761] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27466.523500] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27466.531577] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27466.536795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27466.542533] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27466.546364] Call Trace: [27466.553487] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27466.560615] [27466.567736] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27466.574861] ? watchdog_timer_fn+0x1b4/0x220 [27466.581989] [27466.589110] ? __hrtimer_run_queues+0x1d3/0x360 [27466.618788] ? hrtimer_interrupt+0x104/0x240 [27466.623056] ? __local_bh_enable+0x50/0x60 [27466.627149] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27466.632374] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27466.637332] [27466.639428] [27466.641526] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27466.646835] ? lock_acquire+0xf0/0x2b0 [27466.650584] ? igrab+0x43/0x50 [27466.653642] ? lock_release+0xc5/0x260 [27466.657387] _raw_spin_lock+0x30/0x40 [27466.661044] ? igrab+0x1d/0x50 [27466.664093] igrab+0x1d/0x50 [27466.666972] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27466.672123] delayed_work+0x1d8/0x280 [ceph] [27466.676404] ? process_scheduled_works+0x22a/0x510 [27466.681192] process_scheduled_works+0x2a4/0x510 [27466.685810] worker_thread+0x188/0x340 [27466.689559] ? rescuer_thread+0x310/0x310 [27466.693570] kthread+0xf5/0x130 [27466.696715] ? kthread_complete_and_exit+0x20/0x20 [27466.701502] ret_from_fork+0x31/0x40 [27466.705079] ? kthread_complete_and_exit+0x20/0x20 [27466.709863] ret_from_fork_asm+0x11/0x20 [27466.713786] [27493.939030] watchdog: BUG: soft lockup - CPU#0 stuck for 14491s! [dbench:78683] [27493.946327] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27493.951030] watchdog: BUG: soft lockup - CPU#7 stuck for 23488s! [kworker/7:4:78808] [27494.001769] irq event stamp: 45188492 [27494.001770] hardirqs last enabled at (45188491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27494.009499] Modules linked in: [27494.013157] hardirqs last disabled at (45188492): [] sysvec_apic_timer_interrupt+0xe/0x90 [27494.023315] ceph [27494.026365] softirqs last enabled at (45188482): [] __do_softirq+0x2be/0x371 [27494.036090] libceph [27494.038014] softirqs last disabled at (45188477): [] irq_exit_rcu+0xb0/0xd0 [27494.046705] dns_resolver fscache [27494.048889] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27494.057400] netfs [27494.060728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27494.069674] veth nft_chain_nat [27494.071684] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27494.079172] xt_MASQUERADE [27494.082308] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27494.087787] nf_nat nf_conntrack [27494.090491] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27494.109228] nf_defrag_ipv6 [27494.112452] [27494.112453] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27494.117671] nf_defrag_ipv4 [27494.120461] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27494.121952] nft_compat nf_tables [27494.129074] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27494.131865] nfnetlink bridge [27494.138989] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27494.142302] stp llc [27494.149426] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27494.152390] xfs [27494.159513] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27494.161708] libcrc32c sunrpc [27494.168831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27494.170667] intel_rapl_msr intel_rapl_common [27494.178744] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27494.181714] sb_edac x86_pkg_temp_thermal [27494.187455] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27494.191806] coretemp [27494.198930] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27494.202935] kvm_intel kvm [27494.210058] Call Trace: [27494.210060] [27494.212328] irqbypass i2c_i801 [27494.219454] ? watchdog_timer_fn+0x1b4/0x220 [27494.222156] iTCO_wdt [27494.224602] ? __hrtimer_run_queues+0x1d3/0x360 [27494.226611] iTCO_vendor_support [27494.229752] ? hrtimer_interrupt+0x104/0x240 [27494.234012] i2c_smbus crct10dif_pclmul [27494.236284] ? __local_bh_enable+0x50/0x60 [27494.240808] joydev crc32_pclmul [27494.244034] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27494.248297] ipmi_ssif ghash_clmulni_intel [27494.252126] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27494.256218] mei_me [27494.259443] [27494.264660] mei [27494.268750] [27494.268751] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27494.273708] lpc_ich [27494.275809] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27494.277901] sha512_ssse3 acpi_ipmi [27494.279741] do_raw_spin_lock+0xb5/0xc0 [27494.281837] mfd_core wmi [27494.287142] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27494.289324] ipmi_si ipmi_devintf [27494.294198] try_get_cap_refs+0x159/0x880 [ceph] [27494.297680] ipmi_msghandler [27494.301515] __ceph_get_caps+0x4a1/0x760 [ceph] [27494.304126] ioatdma acpi_pad [27494.308915] ? __file_remove_privs+0x89/0x110 [27494.312223] acpi_power_meter [27494.316833] ? wake_bit_function+0x70/0x70 [27494.319710] ixgbe igb [27494.324237] ? ceph_write_iter+0x260/0x770 [ceph] [27494.327198] mdio crc32c_intel [27494.331550] ceph_write_iter+0x260/0x770 [ceph] [27494.334514] nvme [27494.338609] ? lock_is_held_type+0xaa/0x120 [27494.340962] ptp nvme_core [27494.345662] vfs_write+0x324/0x4f0 [27494.348714] i2c_algo_bit pps_core [27494.353238] ksys_pwrite64+0x65/0xa0 [27494.355158] dca fuse [27494.359337] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27494.362039] [27494.362040] irq event stamp: 141738862 [27494.365438] do_syscall_64+0x38/0x80 [27494.368834] hardirqs last enabled at (141738861): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27494.372407] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27494.374694] hardirqs last disabled at (141738862): [] sysvec_apic_timer_interrupt+0xe/0x90 [27494.379841] RIP: 0033:0x7f48f0eeb93b [27494.381332] softirqs last enabled at (141738860): [] __do_softirq+0x2be/0x371 [27494.385076] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27494.388646] softirqs last disabled at (141738855): [] irq_exit_rcu+0xb0/0xd0 [27494.398890] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27494.403935] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27494.413744] ORIG_RAX: 0000000000000012 [27494.417316] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27494.426087] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27494.444827] Workqueue: events delayed_work [ceph] [27494.453431] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27494.458650] [27494.468027] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27494.471856] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [27494.479327] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27494.486452] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [27494.491148] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27494.498274] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [27494.499769] [27494.506889] [27494.506889] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [27494.560519] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [27494.567644] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [27494.574767] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27494.581891] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27494.589015] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27494.597094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27494.602829] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27494.609955] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27494.617077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27494.624201] Call Trace: [27494.626647] [27494.628658] ? watchdog_timer_fn+0x1b4/0x220 [27494.632923] ? __hrtimer_run_queues+0x1d3/0x360 [27494.637449] ? hrtimer_interrupt+0x104/0x240 [27494.641717] ? __local_bh_enable+0x50/0x60 [27494.645813] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27494.651035] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27494.655993] [27494.658091] [27494.660187] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27494.665496] ? do_raw_spin_lock+0x42/0xc0 [27494.669507] igrab+0x1d/0x50 [27494.672393] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27494.677541] delayed_work+0x1d8/0x280 [ceph] [27494.681823] ? process_scheduled_works+0x22a/0x510 [27494.686613] process_scheduled_works+0x2a4/0x510 [27494.691228] worker_thread+0x188/0x340 [27494.694977] ? rescuer_thread+0x310/0x310 [27494.698980] kthread+0xf5/0x130 [27494.702119] ? kthread_complete_and_exit+0x20/0x20 [27494.706902] ret_from_fork+0x31/0x40 [27494.710480] ? kthread_complete_and_exit+0x20/0x20 [27494.715265] ret_from_fork_asm+0x11/0x20 [27494.719189] [27521.939518] watchdog: BUG: soft lockup - CPU#0 stuck for 14517s! [dbench:78683] [27521.946825] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27521.951518] watchdog: BUG: soft lockup - CPU#7 stuck for 23514s! [kworker/7:4:78808] [27522.002273] irq event stamp: 45246766 [27522.010004] Modules linked in: ceph libceph dns_resolver fscache netfs [27522.013662] hardirqs last enabled at (45246765): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27522.020191] veth nft_chain_nat xt_MASQUERADE nf_nat [27522.030346] hardirqs last disabled at (45246766): [] sysvec_apic_timer_interrupt+0xe/0x90 [27522.035303] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [27522.045026] softirqs last enabled at (45246730): [] __do_softirq+0x2be/0x371 [27522.045030] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [27522.061614] softirqs last disabled at (45246725): [] irq_exit_rcu+0xb0/0xd0 [27522.067525] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [27522.076037] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27522.083248] kvm irqbypass i2c_i801 [27522.092195] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27522.095674] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [27522.103147] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27522.109489] joydev crc32_pclmul ipmi_ssif [27522.114970] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27522.119059] ghash_clmulni_intel mei_me [27522.137797] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27522.141628] mei lpc_ich [27522.146844] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27522.146846] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27522.149376] sha512_ssse3 acpi_ipmi [27522.156499] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27522.163624] mfd_core wmi [27522.167108] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27522.174232] ipmi_si [27522.176849] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27522.183974] ipmi_devintf [27522.186158] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27522.193282] ipmi_msghandler [27522.195916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27522.203994] ioatdma [27522.206872] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27522.212609] acpi_pad acpi_power_meter [27522.214793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27522.221917] ixgbe igb [27522.225660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27522.232785] mdio crc32c_intel [27522.235142] Call Trace: [27522.235145] [27522.242267] nvme ptp [27522.245320] ? watchdog_timer_fn+0x1b4/0x220 [27522.247760] nvme_core i2c_algo_bit [27522.249774] ? __hrtimer_run_queues+0x1d3/0x360 [27522.252042] pps_core dca [27522.256309] ? hrtimer_interrupt+0x104/0x240 [27522.259791] fuse [27522.264314] ? __local_bh_enable+0x50/0x60 [27522.266932] [27522.266933] irq event stamp: 141959800 [27522.271211] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27522.273129] hardirqs last enabled at (141959799): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27522.277221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27522.278711] hardirqs last disabled at (141959800): [] sysvec_apic_timer_interrupt+0xe/0x90 [27522.282455] [27522.287672] softirqs last enabled at (141959798): [] __do_softirq+0x2be/0x371 [27522.297917] [27522.302873] softirqs last disabled at (141959793): [] irq_exit_rcu+0xb0/0xd0 [27522.312684] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27522.314781] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27522.323555] ? queued_spin_lock_slowpath+0x16/0x2e0 [27522.325650] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27522.334250] do_raw_spin_lock+0xb5/0xc0 [27522.339551] Workqueue: events delayed_work [ceph] [27522.348929] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27522.353799] [27522.353800] RIP: 0010:lock_release+0x0/0x260 [27522.361272] try_get_cap_refs+0x159/0x880 [ceph] [27522.365100] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [27522.369803] __ceph_get_caps+0x4a1/0x760 [ceph] [27522.374581] RSP: 0018:ffffc90002117d88 EFLAGS: 00000286 [27522.376078] ? __file_remove_privs+0x89/0x110 [27522.380337] [27522.380338] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [27522.384948] ? wake_bit_function+0x70/0x70 [27522.403685] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [27522.408213] ? ceph_write_iter+0x260/0x770 [ceph] [27522.413427] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [27522.417778] ceph_write_iter+0x260/0x770 [ceph] [27522.419268] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27522.426396] ? lock_is_held_type+0xaa/0x120 [27522.430484] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27522.437609] vfs_write+0x324/0x4f0 [27522.442305] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27522.449433] ksys_pwrite64+0x65/0xa0 [27522.453954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27522.461079] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27522.465256] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27522.472380] do_syscall_64+0x38/0x80 [27522.475777] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27522.483855] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27522.487425] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27522.493161] RIP: 0033:0x7f48f0eeb93b [27522.498292] Call Trace: [27522.498293] [27522.505417] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27522.508987] ? watchdog_timer_fn+0x1b4/0x220 [27522.516111] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27522.521158] ? __hrtimer_run_queues+0x1d3/0x360 [27522.528280] ORIG_RAX: 0000000000000012 [27522.531853] ? hrtimer_interrupt+0x104/0x240 [27522.534294] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27522.536304] ? __local_bh_enable+0x50/0x60 [27522.555043] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27522.559309] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27522.564523] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27522.569051] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27522.572878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27522.577143] [27522.584266] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27522.588360] [27522.588361] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27522.595487] [27522.600704] ? igrab+0x43/0x50 [27522.641781] ? lock_downgrade+0x250/0x250 [27522.645786] ? _raw_spin_lock+0x30/0x40 [27522.649616] _raw_spin_unlock+0x1b/0x40 [27522.653447] igrab+0x43/0x50 [27522.656326] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27522.661476] delayed_work+0x1d8/0x280 [ceph] [27522.665757] ? process_scheduled_works+0x22a/0x510 [27522.670547] process_scheduled_works+0x2a4/0x510 [27522.675163] worker_thread+0x188/0x340 [27522.678926] ? rescuer_thread+0x310/0x310 [27522.682932] kthread+0xf5/0x130 [27522.686080] ? kthread_complete_and_exit+0x20/0x20 [27522.690872] ret_from_fork+0x31/0x40 [27522.694450] ? kthread_complete_and_exit+0x20/0x20 [27522.699233] ret_from_fork_asm+0x11/0x20 [27522.703157] [27549.940006] watchdog: BUG: soft lockup - CPU#0 stuck for 14543s! [dbench:78683] [27549.947310] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27549.952006] watchdog: BUG: soft lockup - CPU#7 stuck for 23540s! [kworker/7:4:78808] [27550.002752] irq event stamp: 45304886 [27550.002753] hardirqs last enabled at (45304885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27550.010482] Modules linked in: [27550.014141] hardirqs last disabled at (45304886): [] sysvec_apic_timer_interrupt+0xe/0x90 [27550.024297] ceph [27550.027348] softirqs last enabled at (45304818): [] __do_softirq+0x2be/0x371 [27550.037072] libceph [27550.038996] softirqs last disabled at (45304813): [] irq_exit_rcu+0xb0/0xd0 [27550.047681] dns_resolver [27550.049865] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27550.058375] fscache [27550.060994] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27550.069936] netfs veth [27550.072123] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27550.079591] nft_chain_nat [27550.082037] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27550.087514] xt_MASQUERADE nf_nat [27550.090217] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27550.108956] nf_conntrack [27550.112265] [27550.112266] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27550.117483] nf_defrag_ipv6 nf_defrag_ipv4 [27550.120100] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27550.121591] nft_compat nf_tables [27550.128716] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27550.132806] nfnetlink bridge [27550.139931] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27550.143242] stp llc [27550.150365] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27550.153329] xfs libcrc32c [27550.160454] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27550.162638] sunrpc intel_rapl_msr [27550.169761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27550.172466] intel_rapl_common [27550.180544] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27550.183940] sb_edac x86_pkg_temp_thermal [27550.189683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27550.192737] coretemp kvm_intel [27550.199862] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27550.203866] kvm irqbypass [27550.210990] Call Trace: [27550.214128] i2c_i801 iTCO_wdt [27550.221253] [27550.223955] iTCO_vendor_support i2c_smbus [27550.226404] ? watchdog_timer_fn+0x1b4/0x220 [27550.229451] crct10dif_pclmul joydev [27550.231461] ? __hrtimer_run_queues+0x1d3/0x360 [27550.235551] crc32_pclmul ipmi_ssif [27550.239818] ? hrtimer_interrupt+0x104/0x240 [27550.243403] ghash_clmulni_intel [27550.247927] ? __local_bh_enable+0x50/0x60 [27550.251413] mei_me mei [27550.255689] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27550.258908] lpc_ich sha512_ssse3 [27550.262999] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27550.265443] acpi_ipmi mfd_core [27550.270662] [27550.273989] wmi ipmi_si [27550.278946] [27550.282084] ipmi_devintf ipmi_msghandler [27550.284182] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27550.286713] ioatdma acpi_pad [27550.288814] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27550.292813] acpi_power_meter ixgbe [27550.298119] do_raw_spin_lock+0xb5/0xc0 [27550.301081] igb mdio [27550.305953] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27550.309436] crc32c_intel nvme [27550.313268] try_get_cap_refs+0x159/0x880 [ceph] [27550.315539] ptp nvme_core [27550.320327] __ceph_get_caps+0x4a1/0x760 [ceph] [27550.323372] i2c_algo_bit pps_core [27550.327988] ? __file_remove_privs+0x89/0x110 [27550.330688] dca fuse [27550.335213] ? wake_bit_function+0x70/0x70 [27550.338608] [27550.338609] irq event stamp: 142180858 [27550.342961] ? ceph_write_iter+0x260/0x770 [ceph] [27550.345231] hardirqs last enabled at (142180857): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27550.349323] ceph_write_iter+0x260/0x770 [ceph] [27550.350811] hardirqs last disabled at (142180858): [] sysvec_apic_timer_interrupt+0xe/0x90 [27550.354561] ? lock_is_held_type+0xaa/0x120 [27550.359252] softirqs last enabled at (142180856): [] __do_softirq+0x2be/0x371 [27550.369499] vfs_write+0x324/0x4f0 [27550.374021] softirqs last disabled at (142180851): [] irq_exit_rcu+0xb0/0xd0 [27550.383836] ksys_pwrite64+0x65/0xa0 [27550.388011] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27550.396780] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27550.400179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27550.408776] do_syscall_64+0x38/0x80 [27550.412346] Workqueue: events delayed_work [ceph] [27550.421725] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27550.426854] [27550.426855] RIP: 0010:lock_acquire+0xf0/0x2b0 [27550.434325] RIP: 0033:0x7f48f0eeb93b [27550.437896] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27550.442594] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27550.447636] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27550.449129] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27550.453478] [27550.453479] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27550.457049] ORIG_RAX: 0000000000000012 [27550.475788] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27550.494524] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27550.499743] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27550.504959] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27550.506450] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27550.513576] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27550.517410] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27550.524538] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27550.531663] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27550.538787] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27550.545911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27550.553039] [27550.560158] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27550.604654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27550.611779] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27550.618902] Call Trace: [27550.621349] [27550.623359] ? watchdog_timer_fn+0x1b4/0x220 [27550.627623] ? __hrtimer_run_queues+0x1d3/0x360 [27550.632148] ? hrtimer_interrupt+0x104/0x240 [27550.636411] ? __local_bh_enable+0x50/0x60 [27550.640503] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27550.645727] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27550.650699] [27550.652799] [27550.654896] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27550.660206] ? lock_acquire+0xf0/0x2b0 [27550.663954] ? igrab+0x43/0x50 [27550.667004] ? lock_release+0xc5/0x260 [27550.670751] _raw_spin_lock+0x30/0x40 [27550.674414] ? igrab+0x1d/0x50 [27550.677466] igrab+0x1d/0x50 [27550.680343] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27550.685493] delayed_work+0x1d8/0x280 [ceph] [27550.689774] ? process_scheduled_works+0x22a/0x510 [27550.694565] process_scheduled_works+0x2a4/0x510 [27550.699180] worker_thread+0x188/0x340 [27550.702929] ? rescuer_thread+0x310/0x310 [27550.706932] kthread+0xf5/0x130 [27550.710070] ? kthread_complete_and_exit+0x20/0x20 [27550.714856] ret_from_fork+0x31/0x40 [27550.718432] ? kthread_complete_and_exit+0x20/0x20 [27550.723216] ret_from_fork_asm+0x11/0x20 [27550.727139] [27577.940492] watchdog: BUG: soft lockup - CPU#0 stuck for 14569s! [dbench:78683] [27577.947799] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27577.952492] watchdog: BUG: soft lockup - CPU#7 stuck for 23566s! [kworker/7:4:78808] [27578.003248] irq event stamp: 45363040 [27578.010979] Modules linked in: ceph libceph dns_resolver fscache netfs [27578.014636] hardirqs last enabled at (45363039): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27578.021170] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [27578.031328] hardirqs last disabled at (45363040): [] sysvec_apic_timer_interrupt+0xe/0x90 [27578.038712] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [27578.048437] softirqs last enabled at (45362984): [] __do_softirq+0x2be/0x371 [27578.055216] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [27578.063898] softirqs last disabled at (45362979): [] irq_exit_rcu+0xb0/0xd0 [27578.070072] sb_edac x86_pkg_temp_thermal [27578.078581] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27578.078582] coretemp [27578.078583] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27578.082585] kvm_intel kvm [27578.091530] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27578.093798] irqbypass i2c_i801 [27578.101271] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27578.103975] iTCO_wdt iTCO_vendor_support [27578.109452] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27578.112589] i2c_smbus crct10dif_pclmul [27578.131328] [27578.135332] joydev crc32_pclmul [27578.140548] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27578.144378] ipmi_ssif ghash_clmulni_intel [27578.145886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27578.149110] mei_me mei [27578.156235] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27578.160326] lpc_ich sha512_ssse3 [27578.167450] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27578.169894] acpi_ipmi [27578.177019] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27578.180330] mfd_core [27578.187453] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27578.189810] wmi ipmi_si [27578.196934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27578.199206] ipmi_devintf [27578.207282] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27578.209813] ipmi_msghandler [27578.215552] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27578.218171] ioatdma acpi_pad [27578.225291] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27578.228180] acpi_power_meter [27578.235302] Call Trace: [27578.235304] [27578.238266] ixgbe [27578.245393] ? watchdog_timer_fn+0x1b4/0x220 [27578.248355] igb mdio [27578.250801] ? __hrtimer_run_queues+0x1d3/0x360 [27578.252811] crc32c_intel [27578.254824] ? hrtimer_interrupt+0x104/0x240 [27578.259085] nvme ptp [27578.261355] ? __local_bh_enable+0x50/0x60 [27578.265891] nvme_core [27578.268507] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27578.272770] i2c_algo_bit pps_core [27578.275041] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27578.279132] dca fuse [27578.281488] [27578.286706] [27578.286707] irq event stamp: 142401700 [27578.290102] [27578.290104] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27578.295062] hardirqs last enabled at (142401699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27578.297335] ? queued_spin_lock_slowpath+0x18/0x2e0 [27578.299427] hardirqs last disabled at (142401700): [] sysvec_apic_timer_interrupt+0xe/0x90 [27578.300922] do_raw_spin_lock+0xb5/0xc0 [27578.304664] softirqs last enabled at (142401698): [] __do_softirq+0x2be/0x371 [27578.306764] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27578.312065] softirqs last disabled at (142401693): [] irq_exit_rcu+0xb0/0xd0 [27578.322314] try_get_cap_refs+0x159/0x880 [ceph] [27578.327188] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27578.337005] __ceph_get_caps+0x4a1/0x760 [ceph] [27578.340829] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27578.349605] ? __file_remove_privs+0x89/0x110 [27578.354386] Workqueue: events delayed_work [ceph] [27578.362982] ? wake_bit_function+0x70/0x70 [27578.367592] [27578.367593] RIP: 0010:rcu_is_watching+0x0/0x40 [27578.376973] ? ceph_write_iter+0x260/0x770 [ceph] [27578.381494] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [27578.388965] ceph_write_iter+0x260/0x770 [ceph] [27578.393317] RSP: 0018:ffffc90002117d10 EFLAGS: 00000247 [27578.398018] ? lock_is_held_type+0xaa/0x120 [27578.402106] [27578.402106] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [27578.403599] vfs_write+0x324/0x4f0 [27578.408033] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [27578.412735] ksys_pwrite64+0x65/0xa0 [27578.431469] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [27578.435993] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27578.441209] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [27578.445389] do_syscall_64+0x38/0x80 [27578.446892] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27578.454020] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27578.457416] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27578.464543] RIP: 0033:0x7f48f0eeb93b [27578.468110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27578.475237] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27578.480365] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27578.487491] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27578.491060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27578.498185] ORIG_RAX: 0000000000000012 [27578.503230] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27578.511307] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27578.514895] Call Trace: [27578.514896] [27578.520631] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27578.539371] ? watchdog_timer_fn+0x1b4/0x220 [27578.546493] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27578.551712] ? __hrtimer_run_queues+0x1d3/0x360 [27578.558836] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27578.562670] ? hrtimer_interrupt+0x104/0x240 [27578.569790] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27578.576913] ? __local_bh_enable+0x50/0x60 [27578.579364] [27578.581371] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27578.634429] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27578.639386] [27578.641483] [27578.643580] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27578.648903] ? rcutorture_get_gp_data+0x30/0x30 [27578.653426] lock_acquire+0x184/0x2b0 [27578.657084] ? igrab+0x43/0x50 [27578.660134] ? lock_release+0xc5/0x260 [27578.663898] _raw_spin_lock+0x30/0x40 [27578.667562] ? igrab+0x1d/0x50 [27578.670612] igrab+0x1d/0x50 [27578.673492] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27578.678641] delayed_work+0x1d8/0x280 [ceph] [27578.682921] ? process_scheduled_works+0x22a/0x510 [27578.687712] process_scheduled_works+0x2a4/0x510 [27578.692327] worker_thread+0x188/0x340 [27578.696076] ? rescuer_thread+0x310/0x310 [27578.700079] kthread+0xf5/0x130 [27578.703217] ? kthread_complete_and_exit+0x20/0x20 [27578.708002] ret_from_fork+0x31/0x40 [27578.711581] ? kthread_complete_and_exit+0x20/0x20 [27578.716364] ret_from_fork_asm+0x11/0x20 [27578.720288] [27605.940979] watchdog: BUG: soft lockup - CPU#0 stuck for 14596s! [dbench:78683] [27605.948278] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27605.952979] watchdog: BUG: soft lockup - CPU#7 stuck for 23593s! [kworker/7:4:78808] [27606.003719] irq event stamp: 45421226 [27606.011451] Modules linked in: ceph libceph dns_resolver [27606.015107] hardirqs last enabled at (45421225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27606.020410] fscache netfs veth [27606.030568] hardirqs last disabled at (45421226): [] sysvec_apic_timer_interrupt+0xe/0x90 [27606.033706] nft_chain_nat xt_MASQUERADE nf_nat [27606.043429] softirqs last enabled at (45421210): [] __do_softirq+0x2be/0x371 [27606.043431] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [27606.056655] softirqs last disabled at (45421205): [] irq_exit_rcu+0xb0/0xd0 [27606.063692] nfnetlink bridge stp llc [27606.072204] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27606.075861] xfs libcrc32c [27606.084806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27606.087511] sunrpc intel_rapl_msr intel_rapl_common [27606.094980] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27606.099938] sb_edac [27606.105415] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27606.105416] x86_pkg_temp_thermal [27606.105417] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27606.107598] coretemp kvm_intel [27606.126338] [27606.126338] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27606.129664] kvm [27606.134883] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27606.138021] irqbypass [27606.139510] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27606.146651] i2c_i801 iTCO_wdt [27606.148489] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27606.155614] iTCO_vendor_support i2c_smbus [27606.157972] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27606.165095] crct10dif_pclmul joydev [27606.168145] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27606.175270] crc32_pclmul ipmi_ssif [27606.179375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27606.186502] ghash_clmulni_intel mei_me [27606.190074] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27606.198151] mei [27606.201634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27606.207389] lpc_ich [27606.211219] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27606.218345] sha512_ssse3 acpi_ipmi [27606.220181] Call Trace: [27606.220182] [27606.227306] mfd_core wmi [27606.229489] ? watchdog_timer_fn+0x1b4/0x220 [27606.236613] ipmi_si [27606.240099] ? __hrtimer_run_queues+0x1d3/0x360 [27606.242541] ipmi_devintf ipmi_msghandler [27606.244556] ? hrtimer_interrupt+0x104/0x240 [27606.247171] ioatdma acpi_pad [27606.251434] ? __local_bh_enable+0x50/0x60 [27606.253618] acpi_power_meter ixgbe [27606.258145] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27606.262146] igb mdio [27606.266410] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27606.269379] crc32c_intel nvme [27606.273473] [27606.276958] ptp nvme_core [27606.282176] [27606.284446] i2c_algo_bit pps_core [27606.289403] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27606.292453] dca fuse [27606.294555] ? queued_spin_lock_slowpath+0x16/0x2e0 [27606.297254] [27606.299374] do_raw_spin_lock+0xb5/0xc0 [27606.302768] irq event stamp: 142622562 [27606.308073] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27606.310344] hardirqs last enabled at (142622561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27606.315215] try_get_cap_refs+0x159/0x880 [ceph] [27606.316705] hardirqs last disabled at (142622562): [] sysvec_apic_timer_interrupt+0xe/0x90 [27606.320539] __ceph_get_caps+0x4a1/0x760 [ceph] [27606.324279] softirqs last enabled at (142622560): [] __do_softirq+0x2be/0x371 [27606.329066] ? __file_remove_privs+0x89/0x110 [27606.339306] softirqs last disabled at (142622555): [] irq_exit_rcu+0xb0/0xd0 [27606.343919] ? wake_bit_function+0x70/0x70 [27606.353728] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27606.358255] ? ceph_write_iter+0x260/0x770 [ceph] [27606.367024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27606.371379] ceph_write_iter+0x260/0x770 [ceph] [27606.379973] Workqueue: events delayed_work [ceph] [27606.384067] ? lock_is_held_type+0xaa/0x120 [27606.393440] [27606.393441] RIP: 0010:lock_release+0x14e/0x260 [27606.398140] vfs_write+0x324/0x4f0 [27606.405609] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [27606.410137] ksys_pwrite64+0x65/0xa0 [27606.414830] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [27606.419006] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27606.420497] [27606.420497] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [27606.424937] do_syscall_64+0x38/0x80 [27606.428333] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [27606.447071] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27606.450656] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [27606.455876] RIP: 0033:0x7f48f0eeb93b [27606.461007] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [27606.462497] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27606.469621] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [27606.473192] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27606.480315] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27606.485378] ORIG_RAX: 0000000000000012 [27606.492502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27606.496072] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27606.503195] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27606.521935] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27606.529058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27606.534276] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27606.542352] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27606.546185] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27606.551922] Call Trace: [27606.551924] [27606.559046] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27606.566171] ? watchdog_timer_fn+0x1b4/0x220 [27606.573299] [27606.580418] ? __hrtimer_run_queues+0x1d3/0x360 [27606.624355] ? hrtimer_interrupt+0x104/0x240 [27606.628640] ? __local_bh_enable+0x50/0x60 [27606.632750] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27606.637966] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27606.642925] [27606.645021] [27606.647119] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27606.652423] ? igrab+0x43/0x50 [27606.655476] ? lock_release+0x14e/0x260 [27606.659313] ? lock_release+0xc5/0x260 [27606.663059] _raw_spin_unlock+0x1b/0x40 [27606.666897] igrab+0x43/0x50 [27606.669776] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27606.674926] delayed_work+0x1d8/0x280 [ceph] [27606.679205] ? process_scheduled_works+0x22a/0x510 [27606.683994] process_scheduled_works+0x2a4/0x510 [27606.688612] worker_thread+0x188/0x340 [27606.692361] ? rescuer_thread+0x310/0x310 [27606.696387] kthread+0xf5/0x130 [27606.699526] ? kthread_complete_and_exit+0x20/0x20 [27606.704311] ret_from_fork+0x31/0x40 [27606.707889] ? kthread_complete_and_exit+0x20/0x20 [27606.712678] ret_from_fork_asm+0x11/0x20 [27606.716607] [27633.941465] watchdog: BUG: soft lockup - CPU#0 stuck for 14622s! [dbench:78683] [27633.948767] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27633.953465] watchdog: BUG: soft lockup - CPU#7 stuck for 23619s! [kworker/7:4:78808] [27634.004215] irq event stamp: 45479346 [27634.004216] hardirqs last enabled at (45479345): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27634.011945] Modules linked in: [27634.015603] hardirqs last disabled at (45479346): [] sysvec_apic_timer_interrupt+0xe/0x90 [27634.025760] ceph [27634.028811] softirqs last enabled at (45479330): [] __do_softirq+0x2be/0x371 [27634.038537] libceph [27634.040460] softirqs last disabled at (45479325): [] irq_exit_rcu+0xb0/0xd0 [27634.049144] dns_resolver fscache [27634.051329] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27634.059855] netfs veth [27634.063167] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27634.072110] nft_chain_nat [27634.074555] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27634.082025] xt_MASQUERADE [27634.084730] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27634.090208] nf_nat nf_conntrack [27634.092911] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27634.111649] nf_defrag_ipv6 [27634.114874] [27634.120089] nf_defrag_ipv4 nft_compat [27634.122880] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27634.124371] nf_tables nfnetlink [27634.128117] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27634.135241] bridge stp [27634.138463] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27634.145588] llc [27634.148031] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27634.155156] xfs libcrc32c [27634.156994] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27634.164117] sunrpc [27634.166838] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27634.173980] intel_rapl_msr intel_rapl_common [27634.176077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27634.184156] sb_edac [27634.188505] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27634.194242] x86_pkg_temp_thermal [27634.196427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27634.203551] coretemp [27634.206864] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27634.213987] kvm_intel [27634.216258] Call Trace: [27634.216260] [27634.223381] kvm [27634.225742] ? watchdog_timer_fn+0x1b4/0x220 [27634.228184] irqbypass i2c_i801 [27634.230195] ? __hrtimer_run_queues+0x1d3/0x360 [27634.232032] iTCO_wdt iTCO_vendor_support [27634.236299] ? hrtimer_interrupt+0x104/0x240 [27634.239434] i2c_smbus [27634.243956] ? __local_bh_enable+0x50/0x60 [27634.247960] crct10dif_pclmul joydev [27634.252228] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27634.254581] crc32_pclmul [27634.258674] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27634.262245] ipmi_ssif ghash_clmulni_intel [27634.267462] [27634.270077] mei_me [27634.275036] [27634.279142] mei lpc_ich [27634.281242] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27634.283338] sha512_ssse3 acpi_ipmi [27634.285439] ? queued_spin_lock_slowpath+0x16/0x2e0 [27634.287967] mfd_core wmi [27634.293274] do_raw_spin_lock+0xb5/0xc0 [27634.296754] ipmi_si ipmi_devintf [27634.301628] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27634.304244] ipmi_msghandler ioatdma [27634.308075] try_get_cap_refs+0x159/0x880 [ceph] [27634.311384] acpi_pad acpi_power_meter [27634.316175] __ceph_get_caps+0x4a1/0x760 [ceph] [27634.319739] ixgbe igb [27634.324355] ? __file_remove_privs+0x89/0x110 [27634.328095] mdio [27634.332618] ? wake_bit_function+0x70/0x70 [27634.334977] crc32c_intel [27634.339330] ? ceph_write_iter+0x260/0x770 [ceph] [27634.341249] nvme ptp [27634.345341] ceph_write_iter+0x260/0x770 [ceph] [27634.347957] nvme_core i2c_algo_bit [27634.352660] ? lock_is_held_type+0xaa/0x120 [27634.354928] pps_core [27634.359452] vfs_write+0x324/0x4f0 [27634.362934] dca fuse [27634.367116] ksys_pwrite64+0x65/0xa0 [27634.369384] [27634.369384] irq event stamp: 142843502 [27634.372782] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27634.375052] hardirqs last enabled at (142843501): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27634.378622] do_syscall_64+0x38/0x80 [27634.380112] hardirqs last disabled at (142843502): [] sysvec_apic_timer_interrupt+0xe/0x90 [27634.383868] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27634.388997] softirqs last enabled at (142843500): [] __do_softirq+0x2be/0x371 [27634.399240] RIP: 0033:0x7f48f0eeb93b [27634.402811] softirqs last disabled at (142843495): [] irq_exit_rcu+0xb0/0xd0 [27634.412622] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27634.417667] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27634.426436] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27634.430008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27634.438604] ORIG_RAX: 0000000000000012 [27634.457341] Workqueue: events delayed_work [ceph] [27634.466719] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27634.471938] [27634.471940] RIP: 0010:lock_acquire+0xf0/0x2b0 [27634.479409] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27634.483238] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27634.487935] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27634.495061] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27634.496551] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27634.500903] [27634.500904] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27634.508027] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27634.526763] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27634.533894] [27634.539106] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27634.578418] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27634.585543] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27634.592667] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27634.600745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27634.606480] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27634.613604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27634.620730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27634.627870] Call Trace: [27634.630317] [27634.632327] ? watchdog_timer_fn+0x1b4/0x220 [27634.636592] ? __hrtimer_run_queues+0x1d3/0x360 [27634.641118] ? hrtimer_interrupt+0x104/0x240 [27634.645387] ? __local_bh_enable+0x50/0x60 [27634.649481] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27634.654704] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27634.659662] [27634.661759] [27634.663871] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27634.669182] ? lock_acquire+0xf0/0x2b0 [27634.672932] ? igrab+0x43/0x50 [27634.675981] ? lock_release+0xc5/0x260 [27634.679728] _raw_spin_lock+0x30/0x40 [27634.683392] ? igrab+0x1d/0x50 [27634.686442] igrab+0x1d/0x50 [27634.689320] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27634.694470] delayed_work+0x1d8/0x280 [ceph] [27634.698753] ? process_scheduled_works+0x22a/0x510 [27634.703541] process_scheduled_works+0x2a4/0x510 [27634.708172] worker_thread+0x188/0x340 [27634.711923] ? rescuer_thread+0x310/0x310 [27634.715927] kthread+0xf5/0x130 [27634.719074] ? kthread_complete_and_exit+0x20/0x20 [27634.723874] ret_from_fork+0x31/0x40 [27634.727454] ? kthread_complete_and_exit+0x20/0x20 [27634.732237] ret_from_fork_asm+0x11/0x20 [27634.736162] [27636.485875] rcu: INFO: rcu_preempt self-detected stall on CPU [27636.491639] rcu: 0-....: (15347265 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7673670 [27636.502064] rcu: (t=15699598 jiffies g=1309609 q=480927 ncpus=8) [27636.508150] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27636.517092] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27636.524564] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27636.530041] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27636.548777] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27636.553995] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27636.561119] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27636.568244] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27636.575367] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27636.582491] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27636.589615] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27636.597694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27636.603430] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27636.610556] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27636.617679] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27636.624803] Call Trace: [27636.627248] [27636.629258] ? rcu_dump_cpu_stacks+0xd1/0x180 [27636.633617] ? rcu_sched_clock_irq+0x633/0x1390 [27636.638141] ? lock_is_held_type+0xaa/0x120 [27636.642321] ? lock_is_held_type+0xaa/0x120 [27636.646499] ? tick_sched_do_timer+0xa0/0xa0 [27636.650768] ? rcu_read_lock_sched_held+0x56/0x90 [27636.655469] ? tick_sched_do_timer+0xa0/0xa0 [27636.659731] ? update_process_times+0x5f/0x90 [27636.664081] ? tick_sched_handle.isra.16+0x1e/0x50 [27636.668866] ? tick_sched_timer+0x6f/0x80 [27636.672870] ? __hrtimer_run_queues+0x1d3/0x360 [27636.677398] ? hrtimer_interrupt+0x104/0x240 [27636.681667] ? __local_bh_enable+0x50/0x60 [27636.685761] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27636.690983] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27636.695941] [27636.698038] [27636.700137] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27636.705445] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27636.710321] do_raw_spin_lock+0xb5/0xc0 [27636.714150] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27636.718953] try_get_cap_refs+0x159/0x880 [ceph] [27636.723583] __ceph_get_caps+0x4a1/0x760 [ceph] [27636.728126] ? __file_remove_privs+0x89/0x110 [27636.732481] ? wake_bit_function+0x70/0x70 [27636.736573] ? ceph_write_iter+0x260/0x770 [ceph] [27636.741277] ceph_write_iter+0x260/0x770 [ceph] [27636.745813] ? lock_is_held_type+0xaa/0x120 [27636.749998] vfs_write+0x324/0x4f0 [27636.753407] ksys_pwrite64+0x65/0xa0 [27636.756980] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27636.762112] do_syscall_64+0x38/0x80 [27636.765683] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27636.770728] RIP: 0033:0x7f48f0eeb93b [27636.774298] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27636.793035] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [27636.800592] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27636.807718] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27636.814842] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27636.821965] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27636.829088] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27636.836219] [27636.838406] Sending NMI from CPU 0 to CPUs 7: [27636.842759] NMI backtrace for cpu 7 [27636.842761] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27636.842763] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27636.842764] Workqueue: events delayed_work [ceph] [27636.842785] RIP: 0010:__lock_acquire+0xc1f/0x1e20 [27636.842787] Code: e8 16 ac ff ff 48 b8 eb 83 b5 80 46 86 c8 61 49 0f af c7 48 c1 e8 31 49 89 c0 4c 8d 24 c5 c0 f5 5d 83 48 8b 04 c5 c0 f5 5d 83 <48> 85 c0 74 1b 48 83 f8 08 74 15 48 8b 50 10 49 39 d7 0f 84 6e fa [27636.842789] RSP: 0018:ffffc90002117c60 EFLAGS: 00000806 [27636.842791] RAX: ffffffff82e139e8 RBX: ffff88810debaa80 RCX: 000000005553a634 [27636.842792] RDX: 00000000000014df RSI: 0000000000000000 RDI: ffff88810debb550 [27636.842794] RBP: ffff88810debb550 R08: 0000000000005c53 R09: 0000000000000003 [27636.842795] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8360d858 [27636.842796] R13: 0000000000000003 R14: 0000000000000000 R15: c5dbdfc36fd10127 [27636.842797] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27636.842799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27636.842800] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27636.842801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27636.842802] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27636.842804] Call Trace: [27636.842805] [27636.842806] ? nmi_cpu_backtrace+0xad/0x120 [27636.842810] ? nmi_cpu_backtrace_handler+0x11/0x20 [27636.842812] ? nmi_handle+0xd4/0x260 [27636.842817] ? default_do_nmi+0x49/0xf0 [27636.842819] ? exc_nmi+0x11e/0x150 [27636.842822] ? end_repeat_nmi+0x16/0x67 [27636.842827] ? __lock_acquire+0xc1f/0x1e20 [27636.842830] ? __lock_acquire+0xc1f/0x1e20 [27636.842832] ? __lock_acquire+0xc1f/0x1e20 [27636.842835] [27636.842835] [27636.842839] lock_acquire+0xcb/0x2b0 [27636.842841] ? igrab+0x1d/0x50 [27636.842843] ? igrab+0x43/0x50 [27636.842845] ? lock_release+0xc5/0x260 [27636.842848] _raw_spin_lock+0x30/0x40 [27636.842851] ? igrab+0x1d/0x50 [27636.842852] igrab+0x1d/0x50 [27636.842854] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27636.842874] delayed_work+0x1d8/0x280 [ceph] [27636.842909] ? process_scheduled_works+0x22a/0x510 [27636.842913] process_scheduled_works+0x2a4/0x510 [27636.842919] worker_thread+0x188/0x340 [27636.842922] ? rescuer_thread+0x310/0x310 [27636.842925] kthread+0xf5/0x130 [27636.842928] ? kthread_complete_and_exit+0x20/0x20 [27636.842931] ret_from_fork+0x31/0x40 [27636.842934] ? kthread_complete_and_exit+0x20/0x20 [27636.842937] ret_from_fork_asm+0x11/0x20 [27636.842944] [27661.941952] watchdog: BUG: soft lockup - CPU#0 stuck for 14648s! [dbench:78683] [27661.949253] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27661.953951] watchdog: BUG: soft lockup - CPU#7 stuck for 23645s! [kworker/7:4:78808] [27662.004692] irq event stamp: 45536832 [27662.004694] hardirqs last enabled at (45536831): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27662.012424] Modules linked in: ceph [27662.016083] hardirqs last disabled at (45536832): [] sysvec_apic_timer_interrupt+0xe/0x90 [27662.026240] libceph [27662.029722] softirqs last enabled at (45536794): [] __do_softirq+0x2be/0x371 [27662.039448] dns_resolver fscache [27662.041632] softirqs last disabled at (45536789): [] irq_exit_rcu+0xb0/0xd0 [27662.050317] netfs [27662.053627] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27662.062137] veth nft_chain_nat [27662.064148] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27662.073092] xt_MASQUERADE [27662.076230] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27662.083699] nf_nat nf_conntrack [27662.086404] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27662.091882] nf_defrag_ipv6 nf_defrag_ipv4 [27662.095105] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27662.113844] nft_compat nf_tables [27662.117936] [27662.117936] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27662.123152] nfnetlink bridge [27662.126463] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27662.127953] stp llc [27662.135076] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27662.138041] xfs libcrc32c [27662.145167] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27662.147349] sunrpc [27662.154474] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27662.157176] intel_rapl_msr intel_rapl_common [27662.164302] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27662.166400] sb_edac [27662.173523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27662.177875] x86_pkg_temp_thermal coretemp [27662.185951] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27662.188136] kvm_intel kvm [27662.193874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27662.197963] irqbypass i2c_i801 [27662.205089] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27662.207793] iTCO_wdt [27662.214915] Call Trace: [27662.214917] [27662.218054] iTCO_vendor_support i2c_smbus [27662.225178] ? watchdog_timer_fn+0x1b4/0x220 [27662.227449] crct10dif_pclmul joydev [27662.229893] ? __hrtimer_run_queues+0x1d3/0x360 [27662.231903] crc32_pclmul ipmi_ssif [27662.235997] ? hrtimer_interrupt+0x104/0x240 [27662.240257] ghash_clmulni_intel [27662.243829] ? __local_bh_enable+0x50/0x60 [27662.248353] mei_me mei [27662.251840] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27662.256102] lpc_ich sha512_ssse3 [27662.259326] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27662.263432] acpi_ipmi mfd_core [27662.265876] [27662.271094] wmi ipmi_si [27662.274406] [27662.274408] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27662.279363] ipmi_devintf [27662.282505] ? queued_spin_lock_slowpath+0x18/0x2e0 [27662.284597] ipmi_msghandler ioatdma [27662.287129] do_raw_spin_lock+0xb5/0xc0 [27662.289226] acpi_pad acpi_power_meter [27662.294531] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27662.297146] ixgbe igb [27662.302021] try_get_cap_refs+0x159/0x880 [ceph] [27662.305588] mdio crc32c_intel [27662.309424] __ceph_get_caps+0x4a1/0x760 [ceph] [27662.313164] nvme ptp [27662.317950] ? __file_remove_privs+0x89/0x110 [27662.320306] nvme_core i2c_algo_bit [27662.324916] ? wake_bit_function+0x70/0x70 [27662.327966] pps_core dca [27662.332494] ? ceph_write_iter+0x260/0x770 [ceph] [27662.334760] fuse [27662.339111] ceph_write_iter+0x260/0x770 [ceph] [27662.342597] irq event stamp: 143062282 [27662.346691] ? lock_is_held_type+0xaa/0x120 [27662.349303] hardirqs last enabled at (143062281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27662.354003] vfs_write+0x324/0x4f0 [27662.355925] hardirqs last disabled at (143062282): [] sysvec_apic_timer_interrupt+0xe/0x90 [27662.360453] ksys_pwrite64+0x65/0xa0 [27662.364192] softirqs last enabled at (143062280): [] __do_softirq+0x2be/0x371 [27662.368370] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27662.378614] softirqs last disabled at (143062275): [] irq_exit_rcu+0xb0/0xd0 [27662.382015] do_syscall_64+0x38/0x80 [27662.391823] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27662.395394] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27662.404164] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27662.409296] RIP: 0033:0x7f48f0eeb93b [27662.417892] Workqueue: events delayed_work [ceph] [27662.421464] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27662.430842] [27662.435884] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27662.443357] RIP: 0010:lock_acquire+0xf0/0x2b0 [27662.446927] ORIG_RAX: 0000000000000012 [27662.451631] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27662.470372] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27662.471861] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27662.477080] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27662.481430] [27662.481431] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27662.485262] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27662.503999] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27662.511122] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27662.516354] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27662.523481] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27662.524972] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27662.532103] [27662.539220] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27662.584157] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27662.592237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27662.597972] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27662.605097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27662.612222] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27662.619357] Call Trace: [27662.621799] [27662.623808] ? watchdog_timer_fn+0x1b4/0x220 [27662.628074] ? __hrtimer_run_queues+0x1d3/0x360 [27662.632600] ? hrtimer_interrupt+0x104/0x240 [27662.636870] ? __local_bh_enable+0x50/0x60 [27662.640962] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27662.646178] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27662.651135] [27662.653234] [27662.655331] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27662.660657] ? lock_acquire+0xf0/0x2b0 [27662.664405] ? igrab+0x43/0x50 [27662.667455] ? lock_release+0xc5/0x260 [27662.671200] _raw_spin_lock+0x30/0x40 [27662.674856] ? igrab+0x1d/0x50 [27662.677908] igrab+0x1d/0x50 [27662.680785] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27662.685935] delayed_work+0x1d8/0x280 [ceph] [27662.690216] ? process_scheduled_works+0x22a/0x510 [27662.695006] process_scheduled_works+0x2a4/0x510 [27662.699622] worker_thread+0x188/0x340 [27662.703389] ? rescuer_thread+0x310/0x310 [27662.707391] kthread+0xf5/0x130 [27662.710529] ? kthread_complete_and_exit+0x20/0x20 [27662.715314] ret_from_fork+0x31/0x40 [27662.718883] ? kthread_complete_and_exit+0x20/0x20 [27662.723669] ret_from_fork_asm+0x11/0x20 [27662.727590] [27689.942439] watchdog: BUG: soft lockup - CPU#0 stuck for 14674s! [dbench:78683] [27689.949742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27689.954438] watchdog: BUG: soft lockup - CPU#7 stuck for 23671s! [kworker/7:4:78808] [27690.005189] irq event stamp: 45595046 [27690.005190] hardirqs last enabled at (45595045): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27690.012921] Modules linked in: [27690.016577] hardirqs last disabled at (45595046): [] sysvec_apic_timer_interrupt+0xe/0x90 [27690.026735] ceph [27690.029786] softirqs last enabled at (45594940): [] __do_softirq+0x2be/0x371 [27690.039510] libceph [27690.041434] softirqs last disabled at (45594935): [] irq_exit_rcu+0xb0/0xd0 [27690.050119] dns_resolver [27690.052305] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27690.060830] fscache [27690.063448] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27690.072394] netfs [27690.074576] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27690.082048] veth nft_chain_nat [27690.084058] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27690.089534] xt_MASQUERADE nf_nat [27690.092674] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27690.111410] nf_conntrack [27690.114723] [27690.119939] nf_defrag_ipv6 [27690.122555] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27690.124047] nf_defrag_ipv4 nft_compat [27690.126836] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27690.133962] nf_tables [27690.137705] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27690.144834] nfnetlink [27690.147187] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27690.154310] bridge stp [27690.156669] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27690.163794] llc [27690.166237] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27690.173363] xfs [27690.175198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27690.183276] libcrc32c sunrpc [27690.185113] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27690.190852] intel_rapl_msr [27690.193814] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27690.200956] intel_rapl_common [27690.203746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27690.210871] sb_edac [27690.213922] Call Trace: [27690.213923] [27690.221046] x86_pkg_temp_thermal [27690.223232] ? watchdog_timer_fn+0x1b4/0x220 [27690.225673] coretemp kvm_intel [27690.227685] ? __hrtimer_run_queues+0x1d3/0x360 [27690.230995] kvm [27690.235263] ? hrtimer_interrupt+0x104/0x240 [27690.238397] irqbypass i2c_i801 [27690.242920] ? __local_bh_enable+0x50/0x60 [27690.244759] iTCO_wdt iTCO_vendor_support [27690.249025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27690.252159] i2c_smbus crct10dif_pclmul [27690.256250] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27690.260255] joydev crc32_pclmul [27690.265472] [27690.269304] ipmi_ssif ghash_clmulni_intel [27690.274260] [27690.277484] mei_me mei [27690.279583] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27690.283672] lpc_ich sha512_ssse3 [27690.285774] ? queued_spin_lock_slowpath+0x16/0x2e0 [27690.288213] acpi_ipmi mfd_core [27690.293519] do_raw_spin_lock+0xb5/0xc0 [27690.296840] wmi ipmi_si [27690.301711] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27690.304845] ipmi_devintf [27690.308680] try_get_cap_refs+0x159/0x880 [ceph] [27690.311208] ipmi_msghandler ioatdma [27690.315996] __ceph_get_caps+0x4a1/0x760 [ceph] [27690.318608] acpi_pad acpi_power_meter [27690.323225] ? __file_remove_privs+0x89/0x110 [27690.326790] ixgbe igb [27690.331315] ? wake_bit_function+0x70/0x70 [27690.335057] mdio crc32c_intel [27690.339411] ? ceph_write_iter+0x260/0x770 [ceph] [27690.341766] nvme [27690.345858] ceph_write_iter+0x260/0x770 [ceph] [27690.348907] ptp nvme_core [27690.353610] ? lock_is_held_type+0xaa/0x120 [27690.355529] i2c_algo_bit pps_core [27690.360054] vfs_write+0x324/0x4f0 [27690.362758] dca fuse [27690.366939] ksys_pwrite64+0x65/0xa0 [27690.370333] [27690.370334] irq event stamp: 143283178 [27690.373729] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27690.376001] hardirqs last enabled at (143283177): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27690.379572] do_syscall_64+0x38/0x80 [27690.381062] hardirqs last disabled at (143283178): [] sysvec_apic_timer_interrupt+0xe/0x90 [27690.384807] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27690.389935] softirqs last enabled at (143283176): [] __do_softirq+0x2be/0x371 [27690.400181] RIP: 0033:0x7f48f0eeb93b [27690.403752] softirqs last disabled at (143283171): [] irq_exit_rcu+0xb0/0xd0 [27690.413563] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27690.418606] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27690.427378] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27690.430946] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27690.439545] ORIG_RAX: 0000000000000012 [27690.458283] Workqueue: events delayed_work [ceph] [27690.467661] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27690.472878] [27690.472879] RIP: 0010:lock_acquire+0xf0/0x2b0 [27690.480349] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27690.484180] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27690.488877] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27690.496001] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27690.497491] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27690.501843] [27690.501843] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27690.508966] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27690.527704] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27690.534847] [27690.540064] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27690.579360] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27690.586482] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27690.593607] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27690.601685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27690.607422] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27690.614547] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27690.621670] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27690.628793] Call Trace: [27690.631238] [27690.633250] ? watchdog_timer_fn+0x1b4/0x220 [27690.637514] ? __hrtimer_run_queues+0x1d3/0x360 [27690.642041] ? hrtimer_interrupt+0x104/0x240 [27690.646310] ? __local_bh_enable+0x50/0x60 [27690.650403] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27690.655629] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27690.660585] [27690.662682] [27690.664780] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27690.670088] ? lock_acquire+0xf0/0x2b0 [27690.673849] ? igrab+0x43/0x50 [27690.676903] ? lock_release+0xc5/0x260 [27690.680649] _raw_spin_lock+0x30/0x40 [27690.684307] ? igrab+0x1d/0x50 [27690.687358] igrab+0x1d/0x50 [27690.690237] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27690.695386] delayed_work+0x1d8/0x280 [ceph] [27690.699666] ? process_scheduled_works+0x22a/0x510 [27690.704455] process_scheduled_works+0x2a4/0x510 [27690.709072] worker_thread+0x188/0x340 [27690.712822] ? rescuer_thread+0x310/0x310 [27690.716849] kthread+0xf5/0x130 [27690.719988] ? kthread_complete_and_exit+0x20/0x20 [27690.724772] ret_from_fork+0x31/0x40 [27690.728351] ? kthread_complete_and_exit+0x20/0x20 [27690.733134] ret_from_fork_asm+0x11/0x20 [27690.737060] [27717.942924] watchdog: BUG: soft lockup - CPU#0 stuck for 14700s! [dbench:78683] [27717.950228] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27717.954924] watchdog: BUG: soft lockup - CPU#7 stuck for 23697s! [kworker/7:4:78808] [27718.005669] irq event stamp: 45653296 [27718.005671] hardirqs last enabled at (45653295): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27718.013399] Modules linked in: [27718.017057] hardirqs last disabled at (45653296): [] sysvec_apic_timer_interrupt+0xe/0x90 [27718.027215] ceph [27718.030265] softirqs last enabled at (45653194): [] __do_softirq+0x2be/0x371 [27718.039990] libceph [27718.041914] softirqs last disabled at (45653189): [] irq_exit_rcu+0xb0/0xd0 [27718.050608] dns_resolver fscache [27718.052790] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27718.061319] netfs [27718.064629] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27718.073573] veth nft_chain_nat [27718.075583] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27718.083054] xt_MASQUERADE nf_nat [27718.086193] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27718.091669] nf_conntrack nf_defrag_ipv6 [27718.094980] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27718.113718] nf_defrag_ipv4 nft_compat [27718.117636] [27718.117637] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27718.122853] nf_tables nfnetlink [27718.126611] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27718.128105] bridge stp [27718.135229] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27718.138452] llc xfs [27718.145577] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27718.148038] libcrc32c [27718.155163] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27718.157347] sunrpc intel_rapl_msr [27718.164470] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27718.166828] intel_rapl_common sb_edac [27718.173951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27718.177349] x86_pkg_temp_thermal coretemp [27718.185428] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27718.189172] kvm_intel kvm [27718.194910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27718.199000] irqbypass i2c_i801 [27718.206125] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27718.208827] iTCO_wdt iTCO_vendor_support [27718.215952] Call Trace: [27718.215953] [27718.219088] i2c_smbus crct10dif_pclmul [27718.226215] ? watchdog_timer_fn+0x1b4/0x220 [27718.230218] joydev crc32_pclmul [27718.232662] ? __hrtimer_run_queues+0x1d3/0x360 [27718.234672] ipmi_ssif ghash_clmulni_intel [27718.238506] ? hrtimer_interrupt+0x104/0x240 [27718.242767] mei_me mei [27718.245992] ? __local_bh_enable+0x50/0x60 [27718.250515] lpc_ich sha512_ssse3 [27718.254621] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27718.258879] acpi_ipmi mfd_core [27718.261326] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27718.265412] wmi ipmi_si [27718.268725] [27718.273941] ipmi_devintf ipmi_msghandler [27718.277078] [27718.277080] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27718.282035] ioatdma acpi_pad [27718.284571] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27718.286664] acpi_power_meter ixgbe [27718.290671] do_raw_spin_lock+0xb5/0xc0 [27718.292765] igb mdio [27718.298073] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27718.301033] crc32c_intel [27718.305909] try_get_cap_refs+0x159/0x880 [ceph] [27718.309389] nvme ptp [27718.313223] __ceph_get_caps+0x4a1/0x760 [ceph] [27718.315489] nvme_core i2c_algo_bit [27718.320278] ? __file_remove_privs+0x89/0x110 [27718.322893] pps_core dca [27718.327504] ? wake_bit_function+0x70/0x70 [27718.329775] fuse [27718.334303] ? ceph_write_iter+0x260/0x770 [ceph] [27718.337799] [27718.337799] irq event stamp: 143504042 [27718.342149] ceph_write_iter+0x260/0x770 [ceph] [27718.344767] hardirqs last enabled at (143504041): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27718.348861] ? lock_is_held_type+0xaa/0x120 [27718.350782] hardirqs last disabled at (143504042): [] sysvec_apic_timer_interrupt+0xe/0x90 [27718.355483] vfs_write+0x324/0x4f0 [27718.356969] softirqs last enabled at (143504040): [] __do_softirq+0x2be/0x371 [27718.360719] ksys_pwrite64+0x65/0xa0 [27718.365239] softirqs last disabled at (143504035): [] irq_exit_rcu+0xb0/0xd0 [27718.375483] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27718.379659] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27718.389473] do_syscall_64+0x38/0x80 [27718.392867] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27718.401640] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27718.405211] Workqueue: events delayed_work [ceph] [27718.413807] RIP: 0033:0x7f48f0eeb93b [27718.418938] [27718.428325] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27718.431896] RIP: 0010:lock_acquire+0xf0/0x2b0 [27718.439366] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27718.444411] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27718.449107] ORIG_RAX: 0000000000000012 [27718.452679] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27718.454167] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27718.472906] [27718.472907] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27718.477256] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27718.482475] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27718.501212] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27718.505043] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27718.510261] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27718.517385] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27718.518874] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27718.526000] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27718.533129] [27718.540248] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27718.593254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27718.598991] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27718.606116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27718.613239] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27718.620365] Call Trace: [27718.622809] [27718.624819] ? watchdog_timer_fn+0x1b4/0x220 [27718.629084] ? __hrtimer_run_queues+0x1d3/0x360 [27718.633623] ? hrtimer_interrupt+0x104/0x240 [27718.637889] ? __local_bh_enable+0x50/0x60 [27718.641981] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27718.647197] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27718.652153] [27718.654250] [27718.656350] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27718.661657] ? lock_acquire+0xf0/0x2b0 [27718.665406] ? igrab+0x43/0x50 [27718.668458] ? lock_release+0xc5/0x260 [27718.672201] _raw_spin_lock+0x30/0x40 [27718.675858] ? igrab+0x1d/0x50 [27718.678910] igrab+0x1d/0x50 [27718.681786] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27718.686936] delayed_work+0x1d8/0x280 [ceph] [27718.691217] ? process_scheduled_works+0x22a/0x510 [27718.696007] process_scheduled_works+0x2a4/0x510 [27718.700624] worker_thread+0x188/0x340 [27718.704373] ? rescuer_thread+0x310/0x310 [27718.708374] kthread+0xf5/0x130 [27718.711512] ? kthread_complete_and_exit+0x20/0x20 [27718.716298] ret_from_fork+0x31/0x40 [27718.719867] ? kthread_complete_and_exit+0x20/0x20 [27718.724652] ret_from_fork_asm+0x11/0x20 [27718.728575] [27745.943411] watchdog: BUG: soft lockup - CPU#0 stuck for 14726s! [dbench:78683] [27745.950715] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27745.955411] watchdog: BUG: soft lockup - CPU#7 stuck for 23723s! [kworker/7:4:78808] [27746.006156] irq event stamp: 45711520 [27746.006157] hardirqs last enabled at (45711519): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27746.013888] Modules linked in: [27746.017544] hardirqs last disabled at (45711520): [] sysvec_apic_timer_interrupt+0xe/0x90 [27746.027701] ceph [27746.030753] softirqs last enabled at (45711414): [] __do_softirq+0x2be/0x371 [27746.040478] libceph [27746.042402] softirqs last disabled at (45711409): [] irq_exit_rcu+0xb0/0xd0 [27746.051086] dns_resolver fscache [27746.053270] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27746.061781] netfs [27746.065108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27746.074053] veth nft_chain_nat [27746.076064] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27746.083534] xt_MASQUERADE [27746.086670] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27746.092149] nf_nat nf_conntrack [27746.094853] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27746.113591] nf_defrag_ipv6 nf_defrag_ipv4 [27746.116816] [27746.116817] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27746.122033] nft_compat nf_tables [27746.126123] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27746.127613] nfnetlink [27746.134739] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27746.138049] bridge stp [27746.145171] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27746.147529] llc xfs [27746.154655] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27746.157099] libcrc32c sunrpc [27746.164222] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27746.166405] intel_rapl_msr [27746.173530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27746.176494] intel_rapl_common [27746.184572] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27746.187363] sb_edac x86_pkg_temp_thermal [27746.193100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27746.196150] coretemp [27746.203275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27746.207280] kvm_intel kvm [27746.214402] Call Trace: [27746.214404] [27746.216673] irqbypass [27746.223808] ? watchdog_timer_fn+0x1b4/0x220 [27746.226510] i2c_i801 iTCO_wdt [27746.228956] ? __hrtimer_run_queues+0x1d3/0x360 [27746.230964] iTCO_vendor_support i2c_smbus [27746.233326] ? hrtimer_interrupt+0x104/0x240 [27746.237587] crct10dif_pclmul [27746.240638] ? __local_bh_enable+0x50/0x60 [27746.245162] joydev [27746.249254] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27746.253515] crc32_pclmul [27746.256480] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27746.260570] ipmi_ssif ghash_clmulni_intel [27746.262668] [27746.267886] mei_me mei [27746.270504] [27746.270505] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27746.275461] lpc_ich sha512_ssse3 [27746.279556] ? queued_spin_lock_slowpath+0x16/0x2e0 [27746.281648] acpi_ipmi [27746.284095] do_raw_spin_lock+0xb5/0xc0 [27746.286189] mfd_core wmi [27746.291496] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27746.294812] ipmi_si [27746.299688] try_get_cap_refs+0x159/0x880 [ceph] [27746.302043] ipmi_devintf [27746.305878] __ceph_get_caps+0x4a1/0x760 [ceph] [27746.308490] ipmi_msghandler ioatdma [27746.313277] ? __file_remove_privs+0x89/0x110 [27746.315457] acpi_pad [27746.320070] ? wake_bit_function+0x70/0x70 [27746.322686] acpi_power_meter ixgbe [27746.327214] ? ceph_write_iter+0x260/0x770 [ceph] [27746.330780] igb mdio [27746.335133] ceph_write_iter+0x260/0x770 [ceph] [27746.337402] crc32c_intel nvme [27746.341499] ? lock_is_held_type+0xaa/0x120 [27746.344977] ptp nvme_core [27746.349679] vfs_write+0x324/0x4f0 [27746.351946] i2c_algo_bit pps_core [27746.356475] ksys_pwrite64+0x65/0xa0 [27746.359520] dca fuse [27746.363699] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27746.366402] [27746.369813] do_syscall_64+0x38/0x80 [27746.373206] irq event stamp: 143724912 [27746.376777] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27746.379047] hardirqs last enabled at (143724911): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27746.384177] RIP: 0033:0x7f48f0eeb93b [27746.385668] hardirqs last disabled at (143724912): [] sysvec_apic_timer_interrupt+0xe/0x90 [27746.389238] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27746.392982] softirqs last enabled at (143724910): [] __do_softirq+0x2be/0x371 [27746.398027] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27746.408272] softirqs last disabled at (143724905): [] irq_exit_rcu+0xb0/0xd0 [27746.411841] ORIG_RAX: 0000000000000012 [27746.421654] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27746.440391] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27746.449162] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27746.454380] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27746.462975] Workqueue: events delayed_work [ceph] [27746.466813] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27746.476193] [27746.476195] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [27746.483318] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27746.490788] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [27746.497930] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27746.502628] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [27746.509756] [27746.511243] [27746.511243] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [27746.564856] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [27746.571979] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [27746.579104] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27746.586226] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27746.593352] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27746.601430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27746.607166] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27746.614290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27746.621416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27746.628538] Call Trace: [27746.630983] [27746.632994] ? watchdog_timer_fn+0x1b4/0x220 [27746.637260] ? __hrtimer_run_queues+0x1d3/0x360 [27746.641786] ? hrtimer_interrupt+0x104/0x240 [27746.646054] ? __local_bh_enable+0x50/0x60 [27746.650147] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27746.655363] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27746.660321] [27746.662418] [27746.664515] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27746.669824] ? do_raw_spin_lock+0x42/0xc0 [27746.673833] igrab+0x1d/0x50 [27746.676711] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27746.681859] delayed_work+0x1d8/0x280 [ceph] [27746.686142] ? process_scheduled_works+0x22a/0x510 [27746.690932] process_scheduled_works+0x2a4/0x510 [27746.695546] worker_thread+0x188/0x340 [27746.699296] ? rescuer_thread+0x310/0x310 [27746.703298] kthread+0xf5/0x130 [27746.706437] ? kthread_complete_and_exit+0x20/0x20 [27746.711222] ret_from_fork+0x31/0x40 [27746.714790] ? kthread_complete_and_exit+0x20/0x20 [27746.719593] ret_from_fork_asm+0x11/0x20 [27746.723516] [27773.943897] watchdog: BUG: soft lockup - CPU#0 stuck for 14752s! [dbench:78683] [27773.951202] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27773.955897] watchdog: BUG: soft lockup - CPU#7 stuck for 23749s! [kworker/7:4:78808] [27774.006644] irq event stamp: 45769854 [27774.006645] hardirqs last enabled at (45769853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27774.014376] Modules linked in: [27774.018032] hardirqs last disabled at (45769854): [] sysvec_apic_timer_interrupt+0xe/0x90 [27774.028191] ceph [27774.031242] softirqs last enabled at (45769818): [] __do_softirq+0x2be/0x371 [27774.040965] libceph [27774.042890] softirqs last disabled at (45769813): [] irq_exit_rcu+0xb0/0xd0 [27774.051572] dns_resolver [27774.053759] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27774.062269] fscache netfs [27774.064904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27774.073846] veth [27774.076550] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27774.084022] nft_chain_nat xt_MASQUERADE [27774.085946] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27774.091423] nf_nat [27774.095341] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27774.114077] nf_conntrack nf_defrag_ipv6 [27774.116177] [27774.121393] nf_defrag_ipv4 nft_compat [27774.125311] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27774.126800] nf_tables nfnetlink [27774.130545] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27774.137669] bridge [27774.140892] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27774.148019] stp [27774.150116] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27774.157240] llc [27774.159077] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27774.166201] xfs libcrc32c [27774.168038] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27774.175163] sunrpc intel_rapl_msr [27774.177867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27774.185944] intel_rapl_common sb_edac [27774.189340] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27774.195077] x86_pkg_temp_thermal coretemp [27774.198824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27774.205946] kvm_intel kvm [27774.210038] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27774.217161] irqbypass [27774.219867] Call Trace: [27774.219869] [27774.226989] i2c_i801 [27774.229349] ? watchdog_timer_fn+0x1b4/0x220 [27774.231791] iTCO_wdt iTCO_vendor_support [27774.233802] ? __hrtimer_run_queues+0x1d3/0x360 [27774.236072] i2c_smbus crct10dif_pclmul [27774.240339] ? hrtimer_interrupt+0x104/0x240 [27774.244341] joydev crc32_pclmul [27774.248863] ? __local_bh_enable+0x50/0x60 [27774.252697] ipmi_ssif ghash_clmulni_intel [27774.256963] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27774.260182] mei_me mei [27774.264293] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27774.268384] lpc_ich [27774.273600] [27774.276043] sha512_ssse3 acpi_ipmi [27774.281000] [27774.283185] mfd_core wmi [27774.285296] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27774.288784] ipmi_si [27774.290886] ? queued_spin_lock_slowpath+0x16/0x2e0 [27774.293499] ipmi_devintf [27774.298806] do_raw_spin_lock+0xb5/0xc0 [27774.300987] ipmi_msghandler ioatdma [27774.305860] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27774.308476] acpi_pad acpi_power_meter [27774.312309] try_get_cap_refs+0x159/0x880 [ceph] [27774.315877] ixgbe igb [27774.320666] __ceph_get_caps+0x4a1/0x760 [ceph] [27774.324405] mdio crc32c_intel [27774.329020] ? __file_remove_privs+0x89/0x110 [27774.331373] nvme ptp [27774.335898] ? wake_bit_function+0x70/0x70 [27774.338949] nvme_core [27774.343302] ? ceph_write_iter+0x260/0x770 [ceph] [27774.345570] i2c_algo_bit pps_core [27774.349677] ceph_write_iter+0x260/0x770 [ceph] [27774.352033] dca fuse [27774.356736] ? lock_is_held_type+0xaa/0x120 [27774.360129] [27774.360130] irq event stamp: 143946002 [27774.364656] vfs_write+0x324/0x4f0 [27774.366924] hardirqs last enabled at (143946001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27774.371105] ksys_pwrite64+0x65/0xa0 [27774.372593] hardirqs last disabled at (143946002): [] sysvec_apic_timer_interrupt+0xe/0x90 [27774.376336] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27774.379733] softirqs last enabled at (143946000): [] __do_softirq+0x2be/0x371 [27774.389980] do_syscall_64+0x38/0x80 [27774.393548] softirqs last disabled at (143945995): [] irq_exit_rcu+0xb0/0xd0 [27774.403360] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27774.408491] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27774.417260] RIP: 0033:0x7f48f0eeb93b [27774.420832] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27774.429429] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27774.434474] Workqueue: events delayed_work [ceph] [27774.443851] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27774.447422] [27774.447423] RIP: 0010:lock_acquire+0xf0/0x2b0 [27774.454893] ORIG_RAX: 0000000000000012 [27774.454894] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27774.473630] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27774.478326] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27774.483544] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27774.485035] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27774.489385] [27774.489386] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27774.493218] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27774.500342] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27774.519078] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27774.526204] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27774.531426] [27774.538543] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27774.584982] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27774.592107] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27774.600183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27774.605920] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27774.613044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27774.620169] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27774.627303] Call Trace: [27774.629746] [27774.631757] ? watchdog_timer_fn+0x1b4/0x220 [27774.636022] ? __hrtimer_run_queues+0x1d3/0x360 [27774.640548] ? hrtimer_interrupt+0x104/0x240 [27774.644817] ? __local_bh_enable+0x50/0x60 [27774.648912] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27774.654135] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27774.659092] [27774.661189] [27774.663304] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27774.668612] ? lock_acquire+0xf0/0x2b0 [27774.672364] ? igrab+0x43/0x50 [27774.675420] ? lock_release+0xc5/0x260 [27774.679167] _raw_spin_lock+0x30/0x40 [27774.682831] ? igrab+0x1d/0x50 [27774.685882] igrab+0x1d/0x50 [27774.688761] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27774.693910] delayed_work+0x1d8/0x280 [ceph] [27774.698191] ? process_scheduled_works+0x22a/0x510 [27774.702980] process_scheduled_works+0x2a4/0x510 [27774.707596] worker_thread+0x188/0x340 [27774.711346] ? rescuer_thread+0x310/0x310 [27774.715349] kthread+0xf5/0x130 [27774.718495] ? kthread_complete_and_exit+0x20/0x20 [27774.723281] ret_from_fork+0x31/0x40 [27774.726877] ? kthread_complete_and_exit+0x20/0x20 [27774.731660] ret_from_fork_asm+0x11/0x20 [27774.735600] [27801.944384] watchdog: BUG: soft lockup - CPU#0 stuck for 14778s! [dbench:78683] [27801.951681] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27801.956384] watchdog: BUG: soft lockup - CPU#7 stuck for 23775s! [kworker/7:4:78808] [27802.007125] irq event stamp: 45828128 [27802.014853] Modules linked in: ceph libceph [27802.018511] hardirqs last enabled at (45828127): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27802.022689] dns_resolver fscache netfs veth [27802.032846] hardirqs last disabled at (45828128): [] sysvec_apic_timer_interrupt+0xe/0x90 [27802.032848] nft_chain_nat [27802.032849] softirqs last enabled at (45828006): [] __do_softirq+0x2be/0x371 [27802.037111] xt_MASQUERADE nf_nat [27802.046834] softirqs last disabled at (45828001): [] irq_exit_rcu+0xb0/0xd0 [27802.049539] nf_conntrack nf_defrag_ipv6 [27802.058224] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27802.061533] nf_defrag_ipv4 nft_compat [27802.070061] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27802.073980] nf_tables nfnetlink [27802.082923] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27802.086666] bridge stp [27802.094139] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27802.097363] llc xfs [27802.102839] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27802.105284] libcrc32c sunrpc [27802.124020] [27802.124021] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27802.126206] intel_rapl_msr intel_rapl_common [27802.131422] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27802.134387] sb_edac x86_pkg_temp_thermal [27802.135877] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27802.143001] coretemp [27802.147352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27802.154477] kvm_intel kvm [27802.158481] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27802.165605] irqbypass [27802.167875] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27802.174999] i2c_i801 [27802.177704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27802.184829] iTCO_wdt [27802.187185] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27802.195263] iTCO_vendor_support [27802.197533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27802.203272] i2c_smbus crct10dif_pclmul [27802.205541] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27802.212667] joydev [27802.215889] Call Trace: [27802.215890] [27802.223013] crc32_pclmul ipmi_ssif [27802.226845] ? watchdog_timer_fn+0x1b4/0x220 [27802.233969] ghash_clmulni_intel mei_me [27802.236079] ? __hrtimer_run_queues+0x1d3/0x360 [27802.238519] mei lpc_ich [27802.240534] ? hrtimer_interrupt+0x104/0x240 [27802.244012] sha512_ssse3 acpi_ipmi [27802.248278] ? __local_bh_enable+0x50/0x60 [27802.252109] mfd_core wmi [27802.256634] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27802.259164] ipmi_si ipmi_devintf [27802.263427] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27802.266912] ipmi_msghandler ioatdma [27802.271002] [27802.273618] acpi_pad acpi_power_meter [27802.278836] [27802.282146] ixgbe igb [27802.287105] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27802.290676] mdio crc32c_intel [27802.292788] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27802.296525] nvme ptp [27802.298624] do_raw_spin_lock+0xb5/0xc0 [27802.300980] nvme_core i2c_algo_bit [27802.306286] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27802.309335] pps_core dca [27802.314208] try_get_cap_refs+0x159/0x880 [ceph] [27802.316477] fuse [27802.320313] __ceph_get_caps+0x4a1/0x760 [ceph] [27802.323793] [27802.323794] irq event stamp: 144166798 [27802.328582] ? __file_remove_privs+0x89/0x110 [27802.331193] hardirqs last enabled at (144166797): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27802.335804] ? wake_bit_function+0x70/0x70 [27802.337727] hardirqs last disabled at (144166798): [] sysvec_apic_timer_interrupt+0xe/0x90 [27802.342253] ? ceph_write_iter+0x260/0x770 [ceph] [27802.343742] softirqs last enabled at (144166796): [] __do_softirq+0x2be/0x371 [27802.347487] ceph_write_iter+0x260/0x770 [ceph] [27802.351837] softirqs last disabled at (144166791): [] irq_exit_rcu+0xb0/0xd0 [27802.362086] ? lock_is_held_type+0xaa/0x120 [27802.366171] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27802.375986] vfs_write+0x324/0x4f0 [27802.380680] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27802.389457] ksys_pwrite64+0x65/0xa0 [27802.393977] Workqueue: events delayed_work [ceph] [27802.402574] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27802.406750] [27802.416128] do_syscall_64+0x38/0x80 [27802.419526] RIP: 0010:ceph_check_delayed_caps+0xc7/0x270 [ceph] [27802.426999] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27802.430568] Code: 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 48 8b 85 78 04 00 00 <49> 39 c5 75 89 31 c0 48 8b 3c 24 48 89 44 24 08 e8 84 5b 07 e1 48 [27802.435265] RIP: 0033:0x7f48f0eeb93b [27802.440396] RSP: 0018:ffffc90002117db8 EFLAGS: 00000246 [27802.441887] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27802.445457] [27802.445458] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [27802.451367] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27802.456411] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [27802.475148] ORIG_RAX: 0000000000000012 [27802.478721] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [27802.483938] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27802.502676] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27802.504166] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27802.511289] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27802.516506] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27802.523631] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27802.527462] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27802.534585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27802.541709] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27802.548833] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27802.555965] [27802.563082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27802.614703] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27802.621826] Call Trace: [27802.624272] [27802.626282] ? watchdog_timer_fn+0x1b4/0x220 [27802.630547] ? __hrtimer_run_queues+0x1d3/0x360 [27802.635074] ? hrtimer_interrupt+0x104/0x240 [27802.639343] ? __local_bh_enable+0x50/0x60 [27802.643435] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27802.648661] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27802.653618] [27802.655714] [27802.657811] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27802.663119] ? ceph_check_delayed_caps+0xc7/0x270 [ceph] [27802.668437] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [27802.673752] delayed_work+0x1d8/0x280 [ceph] [27802.678067] ? process_scheduled_works+0x22a/0x510 [27802.682851] process_scheduled_works+0x2a4/0x510 [27802.687466] worker_thread+0x188/0x340 [27802.691216] ? rescuer_thread+0x310/0x310 [27802.695218] kthread+0xf5/0x130 [27802.698356] ? kthread_complete_and_exit+0x20/0x20 [27802.703139] ret_from_fork+0x31/0x40 [27802.706709] ? kthread_complete_and_exit+0x20/0x20 [27802.711493] ret_from_fork_asm+0x11/0x20 [27802.715418] [27829.944872] watchdog: BUG: soft lockup - CPU#0 stuck for 14804s! [dbench:78683] [27829.952179] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27829.956872] watchdog: BUG: soft lockup - CPU#7 stuck for 23801s! [kworker/7:4:78808] [27830.007629] irq event stamp: 45886300 [27830.015360] Modules linked in: ceph libceph dns_resolver fscache [27830.019018] hardirqs last enabled at (45886299): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27830.025023] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [27830.035180] hardirqs last disabled at (45886300): [] sysvec_apic_timer_interrupt+0xe/0x90 [27830.040658] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [27830.050381] softirqs last enabled at (45886234): [] __do_softirq+0x2be/0x371 [27830.050385] nfnetlink bridge stp llc xfs libcrc32c sunrpc [27830.066104] softirqs last disabled at (45886229): [] irq_exit_rcu+0xb0/0xd0 [27830.071589] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [27830.080101] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27830.088610] kvm irqbypass i2c_i801 [27830.097556] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27830.101038] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [27830.108510] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [27830.114855] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [27830.120333] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [27830.126155] mei_me mei lpc_ich [27830.144894] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27830.148030] sha512_ssse3 acpi_ipmi [27830.153266] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27830.153266] mfd_core [27830.156751] wmi ipmi_si [27830.163874] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27830.166144] ipmi_devintf ipmi_msghandler [27830.168674] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27830.175799] ioatdma acpi_pad [27830.179803] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27830.186929] acpi_power_meter [27830.189891] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27830.197015] ixgbe [27830.199979] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27830.207104] igb mdio [27830.209114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27830.217192] crc32c_intel nvme [27830.219464] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27830.225200] ptp nvme_core [27830.228267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27830.235391] i2c_algo_bit pps_core [27830.238096] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27830.245221] dca fuse [27830.248618] Call Trace: [27830.248621] [27830.255741] [27830.255742] irq event stamp: 144387818 [27830.258015] ? watchdog_timer_fn+0x1b4/0x220 [27830.260456] hardirqs last enabled at (144387817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27830.262468] ? __hrtimer_run_queues+0x1d3/0x360 [27830.263957] hardirqs last disabled at (144387818): [] sysvec_apic_timer_interrupt+0xe/0x90 [27830.267706] ? hrtimer_interrupt+0x104/0x240 [27830.271967] softirqs last enabled at (144387816): [] __do_softirq+0x2be/0x371 [27830.282212] ? __local_bh_enable+0x50/0x60 [27830.286736] softirqs last disabled at (144387811): [] irq_exit_rcu+0xb0/0xd0 [27830.296564] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27830.300826] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27830.309598] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27830.313688] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27830.322285] [27830.327505] Workqueue: events delayed_work [ceph] [27830.336882] [27830.336884] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27830.341839] [27830.349312] ? queued_spin_lock_slowpath+0x16/0x2e0 [27830.351407] RIP: 0010:lock_acquire+0xf0/0x2b0 [27830.356105] do_raw_spin_lock+0xb5/0xc0 [27830.358201] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27830.363506] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27830.364997] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27830.369870] try_get_cap_refs+0x159/0x880 [ceph] [27830.374216] [27830.374217] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27830.378055] __ceph_get_caps+0x4a1/0x760 [ceph] [27830.396786] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27830.401575] ? __file_remove_privs+0x89/0x110 [27830.406788] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27830.411399] ? wake_bit_function+0x70/0x70 [27830.412889] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27830.420016] ? ceph_write_iter+0x260/0x770 [ceph] [27830.424551] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27830.431679] ceph_write_iter+0x260/0x770 [ceph] [27830.436029] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27830.443157] ? lock_is_held_type+0xaa/0x120 [27830.447244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27830.454388] vfs_write+0x324/0x4f0 [27830.459082] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27830.466210] ksys_pwrite64+0x65/0xa0 [27830.470730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27830.478810] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27830.482986] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27830.488725] do_syscall_64+0x38/0x80 [27830.492121] Call Trace: [27830.492123] [27830.499247] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27830.502816] ? watchdog_timer_fn+0x1b4/0x220 [27830.509941] RIP: 0033:0x7f48f0eeb93b [27830.515072] ? __hrtimer_run_queues+0x1d3/0x360 [27830.522195] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27830.525769] ? hrtimer_interrupt+0x104/0x240 [27830.528211] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27830.530221] ? __local_bh_enable+0x50/0x60 [27830.535276] ORIG_RAX: 0000000000000012 [27830.539556] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27830.543124] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27830.547650] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27830.566387] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27830.570652] [27830.575868] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27830.579959] [27830.583790] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27830.589008] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27830.596130] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27830.601094] ? lock_acquire+0xf0/0x2b0 [27830.608217] [27830.610312] ? igrab+0x43/0x50 [27830.648083] ? lock_release+0xc5/0x260 [27830.651827] _raw_spin_lock+0x30/0x40 [27830.655492] ? igrab+0x1d/0x50 [27830.658543] igrab+0x1d/0x50 [27830.661439] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27830.666588] delayed_work+0x1d8/0x280 [ceph] [27830.670868] ? process_scheduled_works+0x22a/0x510 [27830.675660] process_scheduled_works+0x2a4/0x510 [27830.680281] worker_thread+0x188/0x340 [27830.684034] ? rescuer_thread+0x310/0x310 [27830.688043] kthread+0xf5/0x130 [27830.691183] ? kthread_complete_and_exit+0x20/0x20 [27830.695965] ret_from_fork+0x31/0x40 [27830.699559] ? kthread_complete_and_exit+0x20/0x20 [27830.704346] ret_from_fork_asm+0x11/0x20 [27830.708284] [27831.849272] rcu: INFO: rcu_preempt self-detected stall on CPU [27831.855032] rcu: 7-....: (15435003 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/15433701 fqs=7769481 [27831.865624] rcu: (t=15894957 jiffies g=1309609 q=485144 ncpus=8) [27831.871708] Sending NMI from CPU 7 to CPUs 0: [27831.876058] NMI backtrace for cpu 0 [27831.876059] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27831.876061] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27831.876062] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [27831.876065] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [27831.876066] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27831.876067] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27831.876068] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27831.876069] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27831.876070] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27831.876071] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27831.876071] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27831.876072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27831.876073] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27831.876074] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27831.876075] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27831.876075] Call Trace: [27831.876076] [27831.876077] ? nmi_cpu_backtrace+0xad/0x120 [27831.876080] ? nmi_cpu_backtrace_handler+0x11/0x20 [27831.876082] ? nmi_handle+0xd4/0x260 [27831.876087] ? default_do_nmi+0x49/0xf0 [27831.876088] ? exc_nmi+0x11e/0x150 [27831.876090] ? end_repeat_nmi+0x16/0x67 [27831.876094] ? queued_spin_lock_slowpath+0x2b/0x2e0 [27831.876096] ? queued_spin_lock_slowpath+0x2b/0x2e0 [27831.876099] ? queued_spin_lock_slowpath+0x2b/0x2e0 [27831.876101] [27831.876102] [27831.876102] do_raw_spin_lock+0xb5/0xc0 [27831.876105] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27831.876119] try_get_cap_refs+0x159/0x880 [ceph] [27831.876133] __ceph_get_caps+0x4a1/0x760 [ceph] [27831.876146] ? __file_remove_privs+0x89/0x110 [27831.876148] ? wake_bit_function+0x70/0x70 [27831.876150] ? ceph_write_iter+0x260/0x770 [ceph] [27831.876161] ceph_write_iter+0x260/0x770 [ceph] [27831.876172] ? lock_is_held_type+0xaa/0x120 [27831.876175] vfs_write+0x324/0x4f0 [27831.876179] ksys_pwrite64+0x65/0xa0 [27831.876181] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27831.876184] do_syscall_64+0x38/0x80 [27831.876186] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27831.876188] RIP: 0033:0x7f48f0eeb93b [27831.876189] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27831.876190] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [27831.876192] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27831.876192] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27831.876193] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27831.876194] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27831.876195] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27831.876197] [27831.877057] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27832.180618] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27832.188089] Workqueue: events delayed_work [ceph] [27832.192803] RIP: 0010:lock_acquire+0xf0/0x2b0 [27832.197155] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27832.215894] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27832.221109] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27832.228234] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27832.235357] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27832.242483] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27832.249607] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27832.256730] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27832.264809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27832.270544] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27832.277670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27832.284794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27832.291918] Call Trace: [27832.294362] [27832.296374] ? rcu_dump_cpu_stacks+0xd1/0x180 [27832.300733] ? rcu_sched_clock_irq+0x633/0x1390 [27832.305257] ? lock_release+0xc5/0x260 [27832.309000] ? lock_is_held_type+0xaa/0x120 [27832.313178] ? lock_is_held_type+0xaa/0x120 [27832.317358] ? tick_sched_do_timer+0xa0/0xa0 [27832.321629] ? rcu_read_lock_sched_held+0x56/0x90 [27832.326326] ? tick_sched_do_timer+0xa0/0xa0 [27832.330589] ? update_process_times+0x5f/0x90 [27832.334941] ? tick_sched_handle.isra.16+0x1e/0x50 [27832.339724] ? tick_sched_timer+0x6f/0x80 [27832.343730] ? __hrtimer_run_queues+0x1d3/0x360 [27832.348257] ? hrtimer_interrupt+0x104/0x240 [27832.352524] ? __local_bh_enable+0x50/0x60 [27832.356620] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27832.361841] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27832.366800] [27832.368896] [27832.370995] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27832.376311] ? lock_acquire+0xf0/0x2b0 [27832.380062] ? igrab+0x43/0x50 [27832.383111] ? lock_release+0xc5/0x260 [27832.386857] _raw_spin_lock+0x30/0x40 [27832.390521] ? igrab+0x1d/0x50 [27832.393572] igrab+0x1d/0x50 [27832.396451] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27832.401598] delayed_work+0x1d8/0x280 [ceph] [27832.405880] ? process_scheduled_works+0x22a/0x510 [27832.410671] process_scheduled_works+0x2a4/0x510 [27832.415288] worker_thread+0x188/0x340 [27832.419035] ? rescuer_thread+0x310/0x310 [27832.423038] kthread+0xf5/0x130 [27832.426175] ? kthread_complete_and_exit+0x20/0x20 [27832.430961] ret_from_fork+0x31/0x40 [27832.434531] ? kthread_complete_and_exit+0x20/0x20 [27832.439314] ret_from_fork_asm+0x11/0x20 [27832.443237] [27857.945360] watchdog: BUG: soft lockup - CPU#0 stuck for 14830s! [dbench:78683] [27857.952666] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27857.957360] watchdog: BUG: soft lockup - CPU#7 stuck for 23827s! [kworker/7:4:78808] [27858.008108] irq event stamp: 45944432 [27858.008109] hardirqs last enabled at (45944431): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27858.015838] Modules linked in: [27858.019495] hardirqs last disabled at (45944432): [] sysvec_apic_timer_interrupt+0xe/0x90 [27858.029652] ceph [27858.032704] softirqs last enabled at (45944380): [] __do_softirq+0x2be/0x371 [27858.042427] libceph [27858.044353] softirqs last disabled at (45944375): [] irq_exit_rcu+0xb0/0xd0 [27858.053038] dns_resolver fscache [27858.055221] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27858.063730] netfs veth [27858.067060] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27858.076004] nft_chain_nat [27858.078448] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27858.085917] xt_MASQUERADE nf_nat [27858.088622] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27858.094099] nf_conntrack nf_defrag_ipv6 [27858.097411] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27858.116148] nf_defrag_ipv4 nft_compat [27858.120065] [27858.120066] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27858.125283] nf_tables nfnetlink [27858.129042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27858.130534] bridge stp [27858.137659] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27858.140883] llc xfs [27858.148006] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27858.150451] libcrc32c [27858.157574] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27858.159759] sunrpc intel_rapl_msr [27858.166884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27858.169240] intel_rapl_common sb_edac [27858.176366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27858.179762] x86_pkg_temp_thermal [27858.187841] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27858.191585] coretemp [27858.197321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27858.200633] kvm_intel kvm [27858.207757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27858.210039] irqbypass i2c_i801 [27858.217159] Call Trace: [27858.217160] [27858.219865] iTCO_wdt [27858.226991] ? watchdog_timer_fn+0x1b4/0x220 [27858.230126] iTCO_vendor_support i2c_smbus [27858.232570] ? __hrtimer_run_queues+0x1d3/0x360 [27858.234580] crct10dif_pclmul joydev [27858.236855] ? hrtimer_interrupt+0x104/0x240 [27858.241115] crc32_pclmul ipmi_ssif [27858.245205] ? __local_bh_enable+0x50/0x60 [27858.249731] ghash_clmulni_intel [27858.253303] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27858.257564] mei_me mei [27858.261050] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27858.265141] lpc_ich [27858.268365] [27858.273581] sha512_ssse3 [27858.276026] [27858.280999] acpi_ipmi [27858.283183] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27858.285282] mfd_core [27858.287902] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27858.289996] wmi ipmi_si [27858.292354] do_raw_spin_lock+0xb5/0xc0 [27858.297657] ipmi_devintf [27858.299931] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27858.304797] ipmi_msghandler ioatdma [27858.307333] try_get_cap_refs+0x159/0x880 [ceph] [27858.311159] acpi_pad acpi_power_meter [27858.313783] __ceph_get_caps+0x4a1/0x760 [ceph] [27858.318561] ixgbe igb [27858.322137] ? __file_remove_privs+0x89/0x110 [27858.326758] mdio [27858.330506] ? wake_bit_function+0x70/0x70 [27858.335042] crc32c_intel [27858.337398] ? ceph_write_iter+0x260/0x770 [ceph] [27858.341760] nvme ptp [27858.343687] ceph_write_iter+0x260/0x770 [ceph] [27858.347777] nvme_core [27858.350399] ? lock_is_held_type+0xaa/0x120 [27858.355092] i2c_algo_bit pps_core [27858.357366] vfs_write+0x324/0x4f0 [27858.361887] dca fuse [27858.364250] ksys_pwrite64+0x65/0xa0 [27858.368422] [27858.368423] irq event stamp: 144604066 [27858.371818] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27858.375217] hardirqs last enabled at (144604065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27858.377490] do_syscall_64+0x38/0x80 [27858.381058] hardirqs last disabled at (144604066): [] sysvec_apic_timer_interrupt+0xe/0x90 [27858.382550] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27858.386291] softirqs last enabled at (144604064): [] __do_softirq+0x2be/0x371 [27858.391424] RIP: 0033:0x7f48f0eeb93b [27858.401668] softirqs last disabled at (144604059): [] irq_exit_rcu+0xb0/0xd0 [27858.405239] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27858.415049] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27858.420093] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27858.428864] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27858.432434] ORIG_RAX: 0000000000000012 [27858.441041] Workqueue: events delayed_work [ceph] [27858.459778] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27858.469157] [27858.469158] RIP: 0010:lock_acquire+0x4a/0x2b0 [27858.474374] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27858.481845] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [27858.485676] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27858.490374] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [27858.497497] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27858.498987] [27858.498988] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [27858.503337] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27858.510461] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [27858.529205] [27858.536323] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [27858.580838] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [27858.587960] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27858.595085] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27858.603163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27858.608901] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27858.616024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27858.623150] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27858.630274] Call Trace: [27858.632719] [27858.634729] ? watchdog_timer_fn+0x1b4/0x220 [27858.638992] ? __hrtimer_run_queues+0x1d3/0x360 [27858.643519] ? hrtimer_interrupt+0x104/0x240 [27858.647781] ? __local_bh_enable+0x50/0x60 [27858.651873] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27858.657088] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27858.662045] [27858.664142] [27858.666241] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27858.671548] ? lock_acquire+0x4a/0x2b0 [27858.675298] ? igrab+0x43/0x50 [27858.678348] ? lock_release+0xc5/0x260 [27858.682093] _raw_spin_lock+0x30/0x40 [27858.685765] ? igrab+0x1d/0x50 [27858.688818] igrab+0x1d/0x50 [27858.691696] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27858.696845] delayed_work+0x1d8/0x280 [ceph] [27858.701126] ? process_scheduled_works+0x22a/0x510 [27858.705909] process_scheduled_works+0x2a4/0x510 [27858.710523] worker_thread+0x188/0x340 [27858.714274] ? rescuer_thread+0x310/0x310 [27858.718286] kthread+0xf5/0x130 [27858.721432] ? kthread_complete_and_exit+0x20/0x20 [27858.726215] ret_from_fork+0x31/0x40 [27858.729793] ? kthread_complete_and_exit+0x20/0x20 [27858.734579] ret_from_fork_asm+0x11/0x20 [27858.738501] [27885.945850] watchdog: BUG: soft lockup - CPU#0 stuck for 14856s! [dbench:78683] [27885.953153] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27885.957849] watchdog: BUG: soft lockup - CPU#7 stuck for 23853s! [kworker/7:4:78808] [27886.008604] irq event stamp: 46002684 [27886.016335] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [27886.019992] hardirqs last enabled at (46002683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27886.028165] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [27886.038321] hardirqs last disabled at (46002684): [] sysvec_apic_timer_interrupt+0xe/0x90 [27886.045359] nft_compat nf_tables nfnetlink bridge stp llc xfs [27886.055083] softirqs last enabled at (46002636): [] __do_softirq+0x2be/0x371 [27886.060907] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [27886.069593] softirqs last disabled at (46002631): [] irq_exit_rcu+0xb0/0xd0 [27886.069597] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [27886.084630] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27886.084631] iTCO_wdt [27886.084633] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27886.091579] iTCO_vendor_support [27886.100527] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27886.102796] i2c_smbus crct10dif_pclmul [27886.110265] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27886.113489] joydev crc32_pclmul [27886.118966] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27886.122796] ipmi_ssif ghash_clmulni_intel [27886.141537] [27886.144759] mei_me [27886.149976] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27886.154067] mei lpc_ich [27886.155558] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27886.157657] sha512_ssse3 acpi_ipmi [27886.164779] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27886.167311] mfd_core wmi [27886.174435] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27886.177918] ipmi_si ipmi_devintf [27886.185042] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27886.187659] ipmi_msghandler [27886.194783] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27886.198094] ioatdma acpi_pad [27886.205220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27886.208113] acpi_power_meter ixgbe [27886.216191] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27886.219155] igb [27886.224892] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27886.228377] mdio crc32c_intel [27886.235501] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27886.237339] nvme [27886.244463] Call Trace: [27886.244465] [27886.247529] ptp nvme_core [27886.254658] ? watchdog_timer_fn+0x1b4/0x220 [27886.256578] i2c_algo_bit pps_core [27886.259024] ? __hrtimer_run_queues+0x1d3/0x360 [27886.261033] dca fuse [27886.263740] ? hrtimer_interrupt+0x104/0x240 [27886.268002] [27886.268003] irq event stamp: 144824950 [27886.271398] ? __local_bh_enable+0x50/0x60 [27886.275924] hardirqs last enabled at (144824949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27886.278197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27886.282458] hardirqs last disabled at (144824950): [] sysvec_apic_timer_interrupt+0xe/0x90 [27886.283950] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27886.287692] softirqs last enabled at (144824948): [] __do_softirq+0x2be/0x371 [27886.291783] [27886.302026] softirqs last disabled at (144824943): [] irq_exit_rcu+0xb0/0xd0 [27886.307248] [27886.307250] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27886.317057] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27886.322019] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27886.330785] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27886.332884] do_raw_spin_lock+0xb5/0xc0 [27886.341480] Workqueue: events delayed_work [ceph] [27886.343579] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27886.348881] [27886.348882] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [27886.358262] try_get_cap_refs+0x159/0x880 [ceph] [27886.363129] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [27886.370604] __ceph_get_caps+0x4a1/0x760 [ceph] [27886.374429] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [27886.379132] ? __file_remove_privs+0x89/0x110 [27886.383911] [27886.383912] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [27886.385404] ? wake_bit_function+0x70/0x70 [27886.390013] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [27886.394628] ? ceph_write_iter+0x260/0x770 [ceph] [27886.413361] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [27886.417885] ceph_write_iter+0x260/0x770 [ceph] [27886.423102] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [27886.427458] ? lock_is_held_type+0xaa/0x120 [27886.428945] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [27886.436070] vfs_write+0x324/0x4f0 [27886.440159] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27886.447287] ksys_pwrite64+0x65/0xa0 [27886.451981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27886.459106] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27886.463630] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27886.470754] do_syscall_64+0x38/0x80 [27886.474931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27886.482058] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27886.485452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27886.493542] RIP: 0033:0x7f48f0eeb93b [27886.497110] Call Trace: [27886.502848] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27886.507979] [27886.515102] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27886.518673] ? watchdog_timer_fn+0x1b4/0x220 [27886.525796] ORIG_RAX: 0000000000000012 [27886.530842] ? __hrtimer_run_queues+0x1d3/0x360 [27886.537965] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27886.541540] ? hrtimer_interrupt+0x104/0x240 [27886.543978] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27886.562717] ? __local_bh_enable+0x50/0x60 [27886.564727] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27886.569948] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27886.574209] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27886.578042] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27886.582563] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27886.589687] [27886.593957] [27886.601076] [27886.643093] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27886.648402] ? do_raw_spin_lock+0x42/0xc0 [27886.652412] igrab+0x1d/0x50 [27886.655299] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27886.660449] delayed_work+0x1d8/0x280 [ceph] [27886.664730] ? process_scheduled_works+0x22a/0x510 [27886.669517] process_scheduled_works+0x2a4/0x510 [27886.674135] worker_thread+0x188/0x340 [27886.677884] ? rescuer_thread+0x310/0x310 [27886.681885] kthread+0xf5/0x130 [27886.685024] ? kthread_complete_and_exit+0x20/0x20 [27886.689809] ret_from_fork+0x31/0x40 [27886.693387] ? kthread_complete_and_exit+0x20/0x20 [27886.698171] ret_from_fork_asm+0x11/0x20 [27886.702096] [27913.946338] watchdog: BUG: soft lockup - CPU#0 stuck for 14882s! [dbench:78683] [27913.953641] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27913.958338] watchdog: BUG: soft lockup - CPU#7 stuck for 23879s! [kworker/7:4:78808] [27914.009084] irq event stamp: 46060834 [27914.009085] hardirqs last enabled at (46060833): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27914.016814] Modules linked in: ceph [27914.020470] hardirqs last disabled at (46060834): [] sysvec_apic_timer_interrupt+0xe/0x90 [27914.030628] libceph [27914.034111] softirqs last enabled at (46060786): [] __do_softirq+0x2be/0x371 [27914.043836] dns_resolver fscache [27914.046022] softirqs last disabled at (46060781): [] irq_exit_rcu+0xb0/0xd0 [27914.054705] netfs [27914.058034] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27914.066543] veth [27914.068555] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27914.077499] nft_chain_nat [27914.079422] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27914.086894] xt_MASQUERADE nf_nat [27914.089597] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27914.095076] nf_conntrack [27914.098387] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27914.117122] nf_defrag_ipv6 [27914.119741] [27914.119742] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27914.124957] nf_defrag_ipv4 nft_compat [27914.127749] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27914.129240] nf_tables nfnetlink [27914.136365] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27914.140107] bridge stp [27914.147232] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27914.150455] llc xfs [27914.157580] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27914.160024] libcrc32c sunrpc [27914.167148] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27914.169332] intel_rapl_msr intel_rapl_common [27914.176457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27914.179421] sb_edac x86_pkg_temp_thermal [27914.187499] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27914.191849] coretemp kvm_intel [27914.197587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27914.201590] kvm irqbypass [27914.208732] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27914.211868] i2c_i801 iTCO_wdt [27914.218994] Call Trace: [27914.221731] iTCO_vendor_support [27914.228857] [27914.231907] i2c_smbus crct10dif_pclmul [27914.234352] ? watchdog_timer_fn+0x1b4/0x220 [27914.237575] joydev crc32_pclmul [27914.239587] ? __hrtimer_run_queues+0x1d3/0x360 [27914.243416] ipmi_ssif ghash_clmulni_intel [27914.247685] ? hrtimer_interrupt+0x104/0x240 [27914.250905] mei_me mei [27914.255427] ? __local_bh_enable+0x50/0x60 [27914.259518] lpc_ich sha512_ssse3 [27914.263787] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27914.266226] acpi_ipmi [27914.270319] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27914.273628] mfd_core wmi [27914.278846] [27914.281204] ipmi_si ipmi_devintf [27914.286161] [27914.288780] ipmi_msghandler ioatdma [27914.290877] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27914.294187] acpi_pad acpi_power_meter [27914.296288] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27914.299855] ixgbe igb [27914.305162] do_raw_spin_lock+0xb5/0xc0 [27914.308902] mdio crc32c_intel [27914.313776] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27914.316132] nvme ptp [27914.319964] try_get_cap_refs+0x159/0x880 [ceph] [27914.323014] nvme_core i2c_algo_bit [27914.327819] __ceph_get_caps+0x4a1/0x760 [ceph] [27914.330084] pps_core dca [27914.334701] ? __file_remove_privs+0x89/0x110 [27914.338178] fuse [27914.342703] ? wake_bit_function+0x70/0x70 [27914.345322] irq event stamp: 145046116 [27914.349674] ? ceph_write_iter+0x260/0x770 [ceph] [27914.351597] hardirqs last enabled at (145046115): [] asm_common_interrupt+0x26/0x40 [27914.355686] ceph_write_iter+0x260/0x770 [ceph] [27914.359431] hardirqs last disabled at (145046116): [] sysvec_apic_timer_interrupt+0xe/0x90 [27914.364132] ? lock_is_held_type+0xaa/0x120 [27914.373420] softirqs last enabled at (145046114): [] __do_softirq+0x2be/0x371 [27914.377945] vfs_write+0x324/0x4f0 [27914.387754] softirqs last disabled at (145046107): [] irq_exit_rcu+0xb0/0xd0 [27914.391935] ksys_pwrite64+0x65/0xa0 [27914.400702] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27914.404098] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27914.412697] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27914.416270] do_syscall_64+0x38/0x80 [27914.425645] Workqueue: events delayed_work [ceph] [27914.430779] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27914.438246] [27914.438247] RIP: 0010:lock_acquire+0xf0/0x2b0 [27914.441819] RIP: 0033:0x7f48f0eeb93b [27914.446515] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27914.451560] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27914.453050] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27914.457401] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27914.460971] [27914.460972] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27914.479709] ORIG_RAX: 0000000000000012 [27914.498447] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27914.503665] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27914.508881] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27914.510372] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27914.517495] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27914.521326] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27914.528450] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27914.535575] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27914.542698] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27914.549824] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27914.556947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27914.564077] [27914.571197] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27914.608568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27914.615690] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27914.622815] Call Trace: [27914.625261] [27914.627271] ? watchdog_timer_fn+0x1b4/0x220 [27914.631535] ? __hrtimer_run_queues+0x1d3/0x360 [27914.636063] ? hrtimer_interrupt+0x104/0x240 [27914.640332] ? __local_bh_enable+0x50/0x60 [27914.644426] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27914.649648] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27914.654605] [27914.656702] [27914.658802] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27914.664108] ? lock_acquire+0xf0/0x2b0 [27914.667859] ? igrab+0x43/0x50 [27914.670909] ? lock_release+0xc5/0x260 [27914.674654] _raw_spin_lock+0x30/0x40 [27914.678309] ? igrab+0x1d/0x50 [27914.681359] igrab+0x1d/0x50 [27914.684239] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27914.689388] delayed_work+0x1d8/0x280 [ceph] [27914.693670] ? process_scheduled_works+0x22a/0x510 [27914.698460] process_scheduled_works+0x2a4/0x510 [27914.703074] worker_thread+0x188/0x340 [27914.706825] ? rescuer_thread+0x310/0x310 [27914.710827] kthread+0xf5/0x130 [27914.713964] ? kthread_complete_and_exit+0x20/0x20 [27914.718751] ret_from_fork+0x31/0x40 [27914.722329] ? kthread_complete_and_exit+0x20/0x20 [27914.727112] ret_from_fork_asm+0x11/0x20 [27914.731034] [27941.946827] watchdog: BUG: soft lockup - CPU#0 stuck for 14908s! [dbench:78683] [27941.954132] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27941.958827] watchdog: BUG: soft lockup - CPU#7 stuck for 23905s! [kworker/7:4:78808] [27942.009580] irq event stamp: 46119062 [27942.017311] Modules linked in: ceph libceph dns_resolver fscache netfs [27942.020966] hardirqs last enabled at (46119061): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27942.020971] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [27942.037660] hardirqs last disabled at (46119062): [] sysvec_apic_timer_interrupt+0xe/0x90 [27942.045044] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [27942.054768] softirqs last enabled at (46119016): [] __do_softirq+0x2be/0x371 [27942.060851] stp llc xfs libcrc32c sunrpc intel_rapl_msr [27942.069537] softirqs last disabled at (46119011): [] irq_exit_rcu+0xb0/0xd0 [27942.074849] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [27942.083362] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27942.090570] kvm irqbypass i2c_i801 [27942.099515] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27942.102999] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [27942.110470] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [27942.116814] joydev crc32_pclmul [27942.122292] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [27942.122292] ipmi_ssif [27942.125516] ghash_clmulni_intel mei_me [27942.144254] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27942.146611] mei [27942.150442] [27942.155660] lpc_ich [27942.157505] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27942.158996] sha512_ssse3 acpi_ipmi [27942.161178] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27942.168304] mfd_core wmi [27942.171788] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27942.178911] ipmi_si [27942.181530] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27942.188653] ipmi_devintf [27942.190836] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27942.197962] ipmi_msghandler [27942.200578] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27942.207702] ioatdma acpi_pad [27942.210581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27942.218657] acpi_power_meter ixgbe [27942.221622] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27942.227359] igb mdio [27942.230843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27942.237968] crc32c_intel [27942.240238] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27942.247363] nvme [27942.249979] Call Trace: [27942.249981] [27942.257104] ptp nvme_core [27942.259029] ? watchdog_timer_fn+0x1b4/0x220 [27942.261472] i2c_algo_bit pps_core [27942.263484] ? __hrtimer_run_queues+0x1d3/0x360 [27942.266187] dca fuse [27942.270453] ? hrtimer_interrupt+0x104/0x240 [27942.273847] [27942.273848] irq event stamp: 145266916 [27942.278373] ? __local_bh_enable+0x50/0x60 [27942.280643] hardirqs last enabled at (145266915): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27942.284910] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27942.286398] hardirqs last disabled at (145266916): [] sysvec_apic_timer_interrupt+0xe/0x90 [27942.290143] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27942.294232] softirqs last enabled at (145266914): [] __do_softirq+0x2be/0x371 [27942.304478] [27942.309694] softirqs last disabled at (145266909): [] irq_exit_rcu+0xb0/0xd0 [27942.319516] [27942.319517] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27942.324471] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27942.333245] ? queued_spin_lock_slowpath+0x18/0x2e0 [27942.335339] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27942.343938] do_raw_spin_lock+0xb5/0xc0 [27942.346034] Workqueue: events delayed_work [ceph] [27942.351339] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27942.360716] [27942.360717] RIP: 0010:lock_acquire+0xf0/0x2b0 [27942.365589] try_get_cap_refs+0x159/0x880 [ceph] [27942.373058] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27942.376894] __ceph_get_caps+0x4a1/0x760 [ceph] [27942.381585] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27942.386373] ? __file_remove_privs+0x89/0x110 [27942.387860] [27942.387861] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27942.392228] ? wake_bit_function+0x70/0x70 [27942.396838] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27942.415580] ? ceph_write_iter+0x260/0x770 [ceph] [27942.420101] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27942.425318] ceph_write_iter+0x260/0x770 [ceph] [27942.429668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27942.431165] ? lock_is_held_type+0xaa/0x120 [27942.438285] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27942.442377] vfs_write+0x324/0x4f0 [27942.449515] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27942.454232] ksys_pwrite64+0x65/0xa0 [27942.461356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27942.465879] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27942.473002] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27942.477183] do_syscall_64+0x38/0x80 [27942.484304] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27942.487703] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27942.495780] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27942.499353] RIP: 0033:0x7f48f0eeb93b [27942.505087] Call Trace: [27942.505089] [27942.510231] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27942.517353] ? watchdog_timer_fn+0x1b4/0x220 [27942.520922] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27942.528047] ? __hrtimer_run_queues+0x1d3/0x360 [27942.533089] ORIG_RAX: 0000000000000012 [27942.540235] ? hrtimer_interrupt+0x104/0x240 [27942.543802] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27942.546247] ? __local_bh_enable+0x50/0x60 [27942.548257] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27942.566997] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27942.571259] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27942.576478] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27942.581000] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27942.584833] [27942.589096] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27942.596231] [27942.600325] [27942.607445] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27942.650679] ? lock_acquire+0xf0/0x2b0 [27942.654429] ? igrab+0x43/0x50 [27942.657486] ? lock_release+0xc5/0x260 [27942.661249] _raw_spin_lock+0x30/0x40 [27942.664905] ? igrab+0x1d/0x50 [27942.667956] igrab+0x1d/0x50 [27942.670834] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27942.675985] delayed_work+0x1d8/0x280 [ceph] [27942.680265] ? process_scheduled_works+0x22a/0x510 [27942.685056] process_scheduled_works+0x2a4/0x510 [27942.689672] worker_thread+0x188/0x340 [27942.693421] ? rescuer_thread+0x310/0x310 [27942.697431] kthread+0xf5/0x130 [27942.700568] ? kthread_complete_and_exit+0x20/0x20 [27942.705355] ret_from_fork+0x31/0x40 [27942.708932] ? kthread_complete_and_exit+0x20/0x20 [27942.713717] ret_from_fork_asm+0x11/0x20 [27942.717649] [27969.947316] watchdog: BUG: soft lockup - CPU#0 stuck for 14935s! [dbench:78683] [27969.954619] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27969.959316] watchdog: BUG: soft lockup - CPU#7 stuck for 23932s! [kworker/7:4:78808] [27970.010058] irq event stamp: 46177178 [27970.010060] hardirqs last enabled at (46177177): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27970.017791] Modules linked in: ceph [27970.021447] hardirqs last disabled at (46177178): [] sysvec_apic_timer_interrupt+0xe/0x90 [27970.031606] libceph [27970.035089] softirqs last enabled at (46177130): [] __do_softirq+0x2be/0x371 [27970.044812] dns_resolver fscache [27970.046997] softirqs last disabled at (46177125): [] irq_exit_rcu+0xb0/0xd0 [27970.055681] netfs veth [27970.059007] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27970.067519] nft_chain_nat [27970.069964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27970.078908] xt_MASQUERADE nf_nat [27970.081611] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27970.089082] nf_conntrack nf_defrag_ipv6 [27970.092392] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27970.097872] nf_defrag_ipv4 nft_compat [27970.101789] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27970.120527] nf_tables [27970.124270] [27970.129486] nfnetlink bridge [27970.131844] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27970.133336] stp llc [27970.136299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27970.143424] xfs libcrc32c [27970.145607] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27970.152731] sunrpc intel_rapl_msr [27970.155436] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27970.162559] intel_rapl_common sb_edac [27970.165958] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27970.173080] x86_pkg_temp_thermal coretemp [27970.176827] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27970.183950] kvm_intel kvm [27970.188041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27970.196118] irqbypass i2c_i801 [27970.198821] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27970.204559] iTCO_wdt iTCO_vendor_support [27970.207712] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27970.214838] i2c_smbus crct10dif_pclmul [27970.218842] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27970.225965] joydev crc32_pclmul [27970.229797] Call Trace: [27970.229798] [27970.236921] ipmi_ssif ghash_clmulni_intel [27970.240146] ? watchdog_timer_fn+0x1b4/0x220 [27970.242590] mei_me mei [27970.244602] ? __hrtimer_run_queues+0x1d3/0x360 [27970.248691] lpc_ich sha512_ssse3 [27970.252992] ? hrtimer_interrupt+0x104/0x240 [27970.255433] acpi_ipmi mfd_core [27970.259958] ? __local_bh_enable+0x50/0x60 [27970.263268] wmi ipmi_si [27970.267536] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27970.270669] ipmi_devintf ipmi_msghandler [27970.274761] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27970.277291] ioatdma acpi_pad [27970.282509] [27970.286514] acpi_power_meter [27970.291471] [27970.294435] ixgbe [27970.296533] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27970.299496] igb mdio [27970.301596] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27970.303605] crc32c_intel nvme [27970.308911] do_raw_spin_lock+0xb5/0xc0 [27970.311180] ptp nvme_core [27970.316051] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27970.319100] i2c_algo_bit pps_core [27970.322934] try_get_cap_refs+0x159/0x880 [ceph] [27970.325635] dca fuse [27970.330424] __ceph_get_caps+0x4a1/0x760 [ceph] [27970.333816] [27970.333817] irq event stamp: 145487914 [27970.338430] ? __file_remove_privs+0x89/0x110 [27970.340716] hardirqs last enabled at (145487913): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27970.345239] ? wake_bit_function+0x70/0x70 [27970.346730] hardirqs last disabled at (145487914): [] sysvec_apic_timer_interrupt+0xe/0x90 [27970.350476] ? ceph_write_iter+0x260/0x770 [ceph] [27970.354824] softirqs last enabled at (145487912): [] __do_softirq+0x2be/0x371 [27970.365069] ceph_write_iter+0x260/0x770 [ceph] [27970.369159] softirqs last disabled at (145487907): [] irq_exit_rcu+0xb0/0xd0 [27970.378976] ? lock_is_held_type+0xaa/0x120 [27970.383669] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27970.392442] vfs_write+0x324/0x4f0 [27970.396963] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27970.405565] ksys_pwrite64+0x65/0xa0 [27970.409739] Workqueue: events delayed_work [ceph] [27970.419115] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27970.422512] [27970.422513] RIP: 0010:lock_acquire+0xf0/0x2b0 [27970.430000] do_syscall_64+0x38/0x80 [27970.433572] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27970.438270] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27970.443400] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27970.444891] RIP: 0033:0x7f48f0eeb93b [27970.449241] [27970.449242] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27970.452811] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27970.471549] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27970.476593] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27970.481811] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27970.485382] ORIG_RAX: 0000000000000012 [27970.486873] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27970.494013] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27970.512752] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27970.519876] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27970.525093] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27970.532217] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27970.536047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27970.543172] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27970.550296] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27970.557420] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27970.564544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27970.572627] [27970.579747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27970.623288] Call Trace: [27970.625734] [27970.627745] ? watchdog_timer_fn+0x1b4/0x220 [27970.632009] ? __hrtimer_run_queues+0x1d3/0x360 [27970.636536] ? hrtimer_interrupt+0x104/0x240 [27970.640804] ? __local_bh_enable+0x50/0x60 [27970.644897] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27970.650122] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27970.655079] [27970.657176] [27970.659273] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27970.664581] ? lock_acquire+0xf0/0x2b0 [27970.668332] ? igrab+0x43/0x50 [27970.671382] ? lock_release+0xc5/0x260 [27970.675126] _raw_spin_lock+0x30/0x40 [27970.678783] ? igrab+0x1d/0x50 [27970.681833] igrab+0x1d/0x50 [27970.684722] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27970.689869] delayed_work+0x1d8/0x280 [ceph] [27970.694151] ? process_scheduled_works+0x22a/0x510 [27970.698941] process_scheduled_works+0x2a4/0x510 [27970.703557] worker_thread+0x188/0x340 [27970.707306] ? rescuer_thread+0x310/0x310 [27970.711310] kthread+0xf5/0x130 [27970.714447] ? kthread_complete_and_exit+0x20/0x20 [27970.719231] ret_from_fork+0x31/0x40 [27970.722801] ? kthread_complete_and_exit+0x20/0x20 [27970.727586] ret_from_fork_asm+0x11/0x20 [27970.731509] [27997.947805] watchdog: BUG: soft lockup - CPU#0 stuck for 14961s! [dbench:78683] [27997.955107] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [27997.959805] watchdog: BUG: soft lockup - CPU#7 stuck for 23958s! [kworker/7:4:78808] [27998.010556] irq event stamp: 46235340 [27998.018287] Modules linked in: ceph libceph dns_resolver fscache netfs [27998.021944] hardirqs last enabled at (46235339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27998.028478] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [27998.038635] hardirqs last disabled at (46235340): [] sysvec_apic_timer_interrupt+0xe/0x90 [27998.038638] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [27998.054444] softirqs last enabled at (46235304): [] __do_softirq+0x2be/0x371 [27998.062521] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [27998.071207] softirqs last disabled at (46235299): [] irq_exit_rcu+0xb0/0xd0 [27998.077385] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [27998.085897] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27998.091894] irqbypass i2c_i801 iTCO_wdt [27998.100839] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27998.104754] iTCO_vendor_support i2c_smbus crct10dif_pclmul [27998.112227] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [27998.112229] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [27998.123269] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [27998.129091] mei_me mei [27998.147830] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [27998.150274] lpc_ich [27998.155490] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [27998.155492] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [27998.157674] sha512_ssse3 acpi_ipmi [27998.164799] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [27998.171922] mfd_core wmi [27998.175407] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [27998.182530] ipmi_si ipmi_devintf [27998.185147] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [27998.192274] ipmi_msghandler ioatdma [27998.195584] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [27998.202708] acpi_pad acpi_power_meter [27998.206279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27998.214356] ixgbe igb [27998.218098] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [27998.223836] mdio crc32c_intel [27998.226201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27998.233327] nvme [27998.236378] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27998.243503] ptp nvme_core [27998.245427] Call Trace: [27998.252550] i2c_algo_bit pps_core [27998.255256] [27998.257699] dca fuse [27998.261098] ? watchdog_timer_fn+0x1b4/0x220 [27998.263107] [27998.263108] irq event stamp: 145708674 [27998.265378] ? __hrtimer_run_queues+0x1d3/0x360 [27998.269642] hardirqs last enabled at (145708673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27998.271137] ? hrtimer_interrupt+0x104/0x240 [27998.274877] hardirqs last disabled at (145708674): [] sysvec_apic_timer_interrupt+0xe/0x90 [27998.279399] ? __local_bh_enable+0x50/0x60 [27998.289644] softirqs last enabled at (145708672): [] __do_softirq+0x2be/0x371 [27998.293911] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27998.303719] softirqs last disabled at (145708667): [] irq_exit_rcu+0xb0/0xd0 [27998.307812] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27998.316582] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [27998.321800] [27998.330397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [27998.335353] [27998.335354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27998.344732] Workqueue: events delayed_work [ceph] [27998.346833] ? queued_spin_lock_slowpath+0x2d/0x2e0 [27998.354300] [27998.354301] RIP: 0010:lock_acquire+0xf0/0x2b0 [27998.356399] do_raw_spin_lock+0xb5/0xc0 [27998.361701] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [27998.366400] __cap_delay_requeue+0xa7/0x1d0 [ceph] [27998.371268] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [27998.372763] try_get_cap_refs+0x159/0x880 [ceph] [27998.377110] [27998.377111] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [27998.380947] __ceph_get_caps+0x4a1/0x760 [ceph] [27998.399678] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [27998.404468] ? __file_remove_privs+0x89/0x110 [27998.409679] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [27998.414291] ? wake_bit_function+0x70/0x70 [27998.415782] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [27998.422907] ? ceph_write_iter+0x260/0x770 [ceph] [27998.427430] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [27998.434555] ceph_write_iter+0x260/0x770 [ceph] [27998.438905] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [27998.446032] ? lock_is_held_type+0xaa/0x120 [27998.450120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [27998.457247] vfs_write+0x324/0x4f0 [27998.461940] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [27998.469070] ksys_pwrite64+0x65/0xa0 [27998.473590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27998.481667] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [27998.485843] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [27998.491582] do_syscall_64+0x38/0x80 [27998.494978] Call Trace: [27998.494980] [27998.502105] entry_SYSCALL_64_after_hwframe+0x63/0xcd [27998.505675] ? watchdog_timer_fn+0x1b4/0x220 [27998.512799] RIP: 0033:0x7f48f0eeb93b [27998.517931] ? __hrtimer_run_queues+0x1d3/0x360 [27998.525053] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [27998.528628] ? hrtimer_interrupt+0x104/0x240 [27998.531069] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [27998.533078] ? __local_bh_enable+0x50/0x60 [27998.538123] ORIG_RAX: 0000000000000012 [27998.542388] ? __sysvec_apic_timer_interrupt+0x61/0x160 [27998.545957] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [27998.550496] ? sysvec_apic_timer_interrupt+0x6d/0x90 [27998.569236] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [27998.573501] [27998.578717] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [27998.582809] [27998.586640] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [27998.591857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [27998.598980] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [27998.603942] ? lock_acquire+0xf0/0x2b0 [27998.611068] [27998.613161] ? igrab+0x43/0x50 [27998.650913] ? lock_release+0xc5/0x260 [27998.654659] _raw_spin_lock+0x30/0x40 [27998.658324] ? igrab+0x1d/0x50 [27998.661374] igrab+0x1d/0x50 [27998.664253] ceph_check_delayed_caps+0xb7/0x270 [ceph] [27998.669401] delayed_work+0x1d8/0x280 [ceph] [27998.673684] ? process_scheduled_works+0x22a/0x510 [27998.678472] process_scheduled_works+0x2a4/0x510 [27998.683106] worker_thread+0x188/0x340 [27998.686856] ? rescuer_thread+0x310/0x310 [27998.690866] kthread+0xf5/0x130 [27998.694005] ? kthread_complete_and_exit+0x20/0x20 [27998.698790] ret_from_fork+0x31/0x40 [27998.702369] ? kthread_complete_and_exit+0x20/0x20 [27998.707152] ret_from_fork_asm+0x11/0x20 [27998.711075] [28025.948294] watchdog: BUG: soft lockup - CPU#0 stuck for 14987s! [dbench:78683] [28025.955594] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28025.960294] watchdog: BUG: soft lockup - CPU#7 stuck for 23984s! [kworker/7:4:78808] [28026.011035] irq event stamp: 46293430 [28026.011036] hardirqs last enabled at (46293429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28026.018766] Modules linked in: [28026.022423] hardirqs last disabled at (46293430): [] sysvec_apic_timer_interrupt+0xe/0x90 [28026.032580] ceph [28026.035632] softirqs last enabled at (46293382): [] __do_softirq+0x2be/0x371 [28026.045356] libceph [28026.047280] softirqs last disabled at (46293377): [] irq_exit_rcu+0xb0/0xd0 [28026.055970] dns_resolver [28026.058156] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28026.066668] fscache netfs [28026.069302] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28026.078245] veth [28026.080967] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28026.088439] nft_chain_nat [28026.090363] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28026.095839] xt_MASQUERADE nf_nat [28026.098544] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28026.117280] nf_conntrack [28026.120591] [28026.120592] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28026.125809] nf_defrag_ipv6 nf_defrag_ipv4 [28026.128427] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28026.129918] nft_compat nf_tables [28026.137041] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28026.141132] nfnetlink bridge [28026.148256] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28026.151568] stp llc [28026.158692] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28026.161657] xfs libcrc32c [28026.168780] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28026.170970] sunrpc intel_rapl_msr [28026.178097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28026.180800] intel_rapl_common sb_edac [28026.188878] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28026.192274] x86_pkg_temp_thermal coretemp [28026.198012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28026.201758] kvm_intel kvm [28026.208882] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28026.212974] irqbypass i2c_i801 [28026.220095] Call Trace: [28026.222800] iTCO_wdt iTCO_vendor_support [28026.229924] [28026.233061] i2c_smbus crct10dif_pclmul [28026.235506] ? watchdog_timer_fn+0x1b4/0x220 [28026.239509] joydev crc32_pclmul [28026.241521] ? __hrtimer_run_queues+0x1d3/0x360 [28026.245350] ipmi_ssif ghash_clmulni_intel [28026.249618] ? hrtimer_interrupt+0x104/0x240 [28026.252838] mei_me mei [28026.257364] ? __local_bh_enable+0x50/0x60 [28026.261454] lpc_ich sha512_ssse3 [28026.265720] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28026.268163] acpi_ipmi mfd_core [28026.272254] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28026.275564] wmi ipmi_si [28026.280782] [28026.283918] ipmi_devintf [28026.288875] [28026.288877] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28026.291407] ipmi_msghandler ioatdma [28026.293508] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28026.296122] acpi_pad acpi_power_meter [28026.298221] do_raw_spin_lock+0xb5/0xc0 [28026.303522] ixgbe igb mdio [28026.307095] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28026.311980] crc32c_intel [28026.315727] try_get_cap_refs+0x159/0x880 [ceph] [28026.319557] nvme ptp [28026.322354] __ceph_get_caps+0x4a1/0x760 [ceph] [28026.327132] nvme_core i2c_algo_bit [28026.329752] ? __file_remove_privs+0x89/0x110 [28026.334360] pps_core dca [28026.336630] ? wake_bit_function+0x70/0x70 [28026.341153] fuse [28026.344641] ? ceph_write_iter+0x260/0x770 [ceph] [28026.348989] irq event stamp: 145929908 [28026.351608] ceph_write_iter+0x260/0x770 [ceph] [28026.355697] hardirqs last enabled at (145929907): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28026.357626] ? lock_is_held_type+0xaa/0x120 [28026.362319] hardirqs last disabled at (145929908): [] sysvec_apic_timer_interrupt+0xe/0x90 [28026.366066] vfs_write+0x324/0x4f0 [28026.370586] softirqs last enabled at (145929906): [] __do_softirq+0x2be/0x371 [28026.380834] ksys_pwrite64+0x65/0xa0 [28026.385008] softirqs last disabled at (145929901): [] irq_exit_rcu+0xb0/0xd0 [28026.394820] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28026.398216] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28026.406990] do_syscall_64+0x38/0x80 [28026.410557] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28026.410559] Workqueue: events delayed_work [ceph] [28026.419158] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28026.424287] [28026.433664] RIP: 0033:0x7f48f0eeb93b [28026.437234] RIP: 0010:lock_acquire+0xf0/0x2b0 [28026.444705] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28026.449403] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28026.454446] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28026.455937] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28026.459509] ORIG_RAX: 0000000000000012 [28026.463858] [28026.463859] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28026.482596] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28026.501334] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28026.506551] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28026.511769] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28026.515600] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28026.517091] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28026.524214] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28026.531340] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28026.538463] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28026.545587] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28026.552717] [28026.559836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28026.604348] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28026.611473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28026.618595] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28026.625721] Call Trace: [28026.628166] [28026.630175] ? watchdog_timer_fn+0x1b4/0x220 [28026.634442] ? __hrtimer_run_queues+0x1d3/0x360 [28026.638982] ? hrtimer_interrupt+0x104/0x240 [28026.643246] ? __local_bh_enable+0x50/0x60 [28026.647338] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28026.652553] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28026.657512] [28026.659609] [28026.661707] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28026.667013] ? lock_acquire+0xf0/0x2b0 [28026.670764] ? igrab+0x43/0x50 [28026.673814] ? lock_release+0xc5/0x260 [28026.677561] _raw_spin_lock+0x30/0x40 [28026.681225] ? igrab+0x1d/0x50 [28026.684274] igrab+0x1d/0x50 [28026.687151] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28026.692302] delayed_work+0x1d8/0x280 [ceph] [28026.696583] ? process_scheduled_works+0x22a/0x510 [28026.701365] process_scheduled_works+0x2a4/0x510 [28026.705988] worker_thread+0x188/0x340 [28026.709740] ? rescuer_thread+0x310/0x310 [28026.713749] kthread+0xf5/0x130 [28026.716888] ? kthread_complete_and_exit+0x20/0x20 [28026.721674] ret_from_fork+0x31/0x40 [28026.725269] ? kthread_complete_and_exit+0x20/0x20 [28026.730052] ret_from_fork_asm+0x11/0x20 [28026.733986] [28027.450687] rcu: INFO: rcu_preempt self-detected stall on CPU [28027.456445] rcu: 7-....: (15624642 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/15623339 fqs=7865371 [28027.467035] rcu: (t=16090555 jiffies g=1309609 q=490704 ncpus=8) [28027.473122] Sending NMI from CPU 7 to CPUs 0: [28027.477473] NMI backtrace for cpu 0 [28027.477475] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28027.477476] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28027.477477] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28027.477480] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28027.477481] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28027.477483] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28027.477484] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28027.477485] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28027.477486] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28027.477486] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28027.477487] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28027.477488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28027.477489] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28027.477490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28027.477490] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28027.477491] Call Trace: [28027.477492] [28027.477493] ? nmi_cpu_backtrace+0xad/0x120 [28027.477497] ? nmi_cpu_backtrace_handler+0x11/0x20 [28027.477498] ? nmi_handle+0xd4/0x260 [28027.477503] ? default_do_nmi+0x49/0xf0 [28027.477504] ? exc_nmi+0x11e/0x150 [28027.477506] ? end_repeat_nmi+0x16/0x67 [28027.477510] ? queued_spin_lock_slowpath+0x16/0x2e0 [28027.477512] ? queued_spin_lock_slowpath+0x16/0x2e0 [28027.477515] ? queued_spin_lock_slowpath+0x16/0x2e0 [28027.477517] [28027.477518] [28027.477519] do_raw_spin_lock+0xb5/0xc0 [28027.477521] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28027.477535] try_get_cap_refs+0x159/0x880 [ceph] [28027.477550] __ceph_get_caps+0x4a1/0x760 [ceph] [28027.477562] ? __file_remove_privs+0x89/0x110 [28027.477564] ? wake_bit_function+0x70/0x70 [28027.477566] ? ceph_write_iter+0x260/0x770 [ceph] [28027.477577] ceph_write_iter+0x260/0x770 [ceph] [28027.477589] ? lock_is_held_type+0xaa/0x120 [28027.477592] vfs_write+0x324/0x4f0 [28027.477596] ksys_pwrite64+0x65/0xa0 [28027.477598] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28027.477601] do_syscall_64+0x38/0x80 [28027.477603] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28027.477605] RIP: 0033:0x7f48f0eeb93b [28027.477606] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28027.477607] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [28027.477608] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28027.477609] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28027.477610] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28027.477610] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28027.477611] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28027.477614] [28027.478471] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28027.782031] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28027.789502] Workqueue: events delayed_work [ceph] [28027.794216] RIP: 0010:lock_acquire+0xf0/0x2b0 [28027.798569] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28027.817306] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28027.822524] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28027.829646] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28027.836772] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28027.843894] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28027.851020] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28027.858145] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28027.866220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28027.871959] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28027.879082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28027.886208] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28027.893331] Call Trace: [28027.895775] [28027.897785] ? rcu_dump_cpu_stacks+0xd1/0x180 [28027.902147] ? rcu_sched_clock_irq+0x633/0x1390 [28027.906671] ? lock_release+0xc5/0x260 [28027.910414] ? lock_is_held_type+0xaa/0x120 [28027.914592] ? lock_is_held_type+0xaa/0x120 [28027.918769] ? tick_sched_do_timer+0xa0/0xa0 [28027.923042] ? rcu_read_lock_sched_held+0x56/0x90 [28027.927740] ? tick_sched_do_timer+0xa0/0xa0 [28027.932003] ? update_process_times+0x5f/0x90 [28027.936355] ? tick_sched_handle.isra.16+0x1e/0x50 [28027.941138] ? tick_sched_timer+0x6f/0x80 [28027.945142] ? __hrtimer_run_queues+0x1d3/0x360 [28027.949669] ? hrtimer_interrupt+0x104/0x240 [28027.953939] ? __local_bh_enable+0x50/0x60 [28027.958031] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28027.963255] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28027.968211] [28027.970310] [28027.972407] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28027.977717] ? lock_acquire+0xf0/0x2b0 [28027.981467] ? igrab+0x43/0x50 [28027.984516] ? lock_release+0xc5/0x260 [28027.988261] _raw_spin_lock+0x30/0x40 [28027.991925] ? igrab+0x1d/0x50 [28027.994976] igrab+0x1d/0x50 [28027.997855] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28028.003005] delayed_work+0x1d8/0x280 [ceph] [28028.007286] ? process_scheduled_works+0x22a/0x510 [28028.012074] process_scheduled_works+0x2a4/0x510 [28028.016690] worker_thread+0x188/0x340 [28028.020441] ? rescuer_thread+0x310/0x310 [28028.024444] kthread+0xf5/0x130 [28028.027581] ? kthread_complete_and_exit+0x20/0x20 [28028.032365] ret_from_fork+0x31/0x40 [28028.035934] ? kthread_complete_and_exit+0x20/0x20 [28028.040720] ret_from_fork_asm+0x11/0x20 [28028.044644] [28053.948784] watchdog: BUG: soft lockup - CPU#0 stuck for 15013s! [dbench:78683] [28053.956091] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28053.960783] watchdog: BUG: soft lockup - CPU#7 stuck for 24010s! [kworker/7:4:78808] [28054.011540] irq event stamp: 46351654 [28054.019272] Modules linked in: ceph libceph dns_resolver fscache netfs [28054.022928] hardirqs last enabled at (46351653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28054.029461] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [28054.039621] hardirqs last disabled at (46351654): [] sysvec_apic_timer_interrupt+0xe/0x90 [28054.047005] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [28054.056730] softirqs last enabled at (46351614): [] __do_softirq+0x2be/0x371 [28054.063515] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [28054.072200] softirqs last disabled at (46351609): [] irq_exit_rcu+0xb0/0xd0 [28054.078379] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [28054.086891] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28054.093753] i2c_i801 iTCO_wdt iTCO_vendor_support [28054.102697] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28054.102699] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28054.107482] i2c_smbus crct10dif_pclmul [28054.114953] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28054.120431] joydev crc32_pclmul [28054.124261] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28054.143000] ipmi_ssif ghash_clmulni_intel [28054.146224] [28054.151456] mei_me mei [28054.155549] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28054.157041] lpc_ich [28054.159483] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28054.166606] sha512_ssse3 [28054.168791] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28054.175915] acpi_ipmi mfd_core [28054.178532] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28054.185656] wmi ipmi_si [28054.188795] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28054.195919] ipmi_devintf [28054.198461] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28054.205581] ipmi_msghandler [28054.208199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28054.216276] ioatdma acpi_pad [28054.219154] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28054.224908] acpi_power_meter ixgbe [28054.227873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28054.234996] igb [28054.238480] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28054.245605] mdio crc32c_intel [28054.247458] Call Trace: [28054.247460] [28054.254583] nvme [28054.257635] ? watchdog_timer_fn+0x1b4/0x220 [28054.260078] ptp nvme_core [28054.262091] ? __hrtimer_run_queues+0x1d3/0x360 [28054.264014] i2c_algo_bit pps_core [28054.268281] ? hrtimer_interrupt+0x104/0x240 [28054.270983] dca fuse [28054.275504] ? __local_bh_enable+0x50/0x60 [28054.278904] [28054.278905] irq event stamp: 146145968 [28054.283184] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28054.285454] hardirqs last enabled at (146145967): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28054.289564] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28054.291053] hardirqs last disabled at (146145968): [] sysvec_apic_timer_interrupt+0xe/0x90 [28054.294798] [28054.300016] softirqs last enabled at (146145966): [] __do_softirq+0x2be/0x371 [28054.310258] [28054.315217] softirqs last disabled at (146145961): [] irq_exit_rcu+0xb0/0xd0 [28054.325029] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28054.327125] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28054.335899] ? queued_spin_lock_slowpath+0x16/0x2e0 [28054.337994] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28054.346593] do_raw_spin_lock+0xb5/0xc0 [28054.351896] Workqueue: events delayed_work [ceph] [28054.361275] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28054.366143] [28054.366144] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [28054.373618] try_get_cap_refs+0x159/0x880 [ceph] [28054.377446] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [28054.382147] __ceph_get_caps+0x4a1/0x760 [ceph] [28054.386927] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [28054.388421] ? __file_remove_privs+0x89/0x110 [28054.393027] [28054.393028] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [28054.397639] ? wake_bit_function+0x70/0x70 [28054.416377] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [28054.420903] ? ceph_write_iter+0x260/0x770 [ceph] [28054.426118] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [28054.430478] ceph_write_iter+0x260/0x770 [ceph] [28054.431967] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28054.439096] ? lock_is_held_type+0xaa/0x120 [28054.443185] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28054.450311] vfs_write+0x324/0x4f0 [28054.455004] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28054.462133] ksys_pwrite64+0x65/0xa0 [28054.466652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28054.473777] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28054.477953] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28054.485079] do_syscall_64+0x38/0x80 [28054.488475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28054.496554] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28054.500124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28054.505863] RIP: 0033:0x7f48f0eeb93b [28054.510992] Call Trace: [28054.510994] [28054.518117] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28054.521688] ? watchdog_timer_fn+0x1b4/0x220 [28054.528810] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28054.533855] ? __hrtimer_run_queues+0x1d3/0x360 [28054.540979] ORIG_RAX: 0000000000000012 [28054.544554] ? hrtimer_interrupt+0x104/0x240 [28054.546993] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28054.549004] ? __local_bh_enable+0x50/0x60 [28054.567743] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28054.572010] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28054.577224] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28054.581748] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28054.585578] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28054.589843] [28054.596968] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28054.601058] [28054.601059] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28054.608194] [28054.613412] ? do_raw_spin_lock+0x42/0xc0 [28054.655428] igrab+0x1d/0x50 [28054.658312] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28054.663477] delayed_work+0x1d8/0x280 [ceph] [28054.667760] ? process_scheduled_works+0x22a/0x510 [28054.672551] process_scheduled_works+0x2a4/0x510 [28054.677185] worker_thread+0x188/0x340 [28054.680934] ? rescuer_thread+0x310/0x310 [28054.684945] kthread+0xf5/0x130 [28054.688090] ? kthread_complete_and_exit+0x20/0x20 [28054.692875] ret_from_fork+0x31/0x40 [28054.696469] ? kthread_complete_and_exit+0x20/0x20 [28054.701254] ret_from_fork_asm+0x11/0x20 [28054.705196] [28081.949275] watchdog: BUG: soft lockup - CPU#0 stuck for 15039s! [dbench:78683] [28081.956578] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28081.961275] watchdog: BUG: soft lockup - CPU#7 stuck for 24036s! [kworker/7:4:78808] [28082.012020] irq event stamp: 46409748 [28082.012022] hardirqs last enabled at (46409747): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28082.019752] Modules linked in: ceph [28082.023407] hardirqs last disabled at (46409748): [] sysvec_apic_timer_interrupt+0xe/0x90 [28082.033564] libceph [28082.037049] softirqs last enabled at (46409710): [] __do_softirq+0x2be/0x371 [28082.046773] dns_resolver fscache [28082.048957] softirqs last disabled at (46409705): [] irq_exit_rcu+0xb0/0xd0 [28082.057642] netfs veth [28082.060970] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28082.069479] nft_chain_nat [28082.071925] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28082.080887] xt_MASQUERADE nf_nat [28082.083590] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28082.091061] nf_conntrack nf_defrag_ipv6 [28082.094372] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28082.099849] nf_defrag_ipv4 nft_compat [28082.103767] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28082.122503] nf_tables nfnetlink [28082.126249] [28082.126250] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28082.131465] bridge [28082.134690] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28082.136181] stp llc [28082.143303] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28082.145402] xfs libcrc32c [28082.152525] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28082.154710] sunrpc intel_rapl_msr [28082.161835] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28082.164538] intel_rapl_common sb_edac [28082.171670] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28082.175069] x86_pkg_temp_thermal [28082.182191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28082.185937] coretemp kvm_intel [28082.194014] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28082.197326] kvm irqbypass [28082.203063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28082.206200] i2c_i801 iTCO_wdt [28082.213323] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28082.216028] iTCO_vendor_support [28082.223151] Call Trace: [28082.226202] i2c_smbus crct10dif_pclmul [28082.233327] [28082.236550] joydev crc32_pclmul [28082.238996] ? watchdog_timer_fn+0x1b4/0x220 [28082.242825] ipmi_ssif ghash_clmulni_intel [28082.244837] ? __hrtimer_run_queues+0x1d3/0x360 [28082.248060] mei_me mei [28082.252329] ? hrtimer_interrupt+0x104/0x240 [28082.256414] lpc_ich sha512_ssse3 [28082.260940] ? __local_bh_enable+0x50/0x60 [28082.263384] acpi_ipmi mfd_core [28082.267669] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28082.270974] wmi ipmi_si [28082.275068] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28082.278205] ipmi_devintf [28082.283420] [28082.285951] ipmi_msghandler ioatdma [28082.290927] [28082.293544] acpi_pad acpi_power_meter [28082.295641] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28082.299213] ixgbe igb [28082.301314] ? queued_spin_lock_slowpath+0x18/0x2e0 [28082.305052] mdio crc32c_intel [28082.310361] do_raw_spin_lock+0xb5/0xc0 [28082.312714] nvme ptp [28082.317587] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28082.320637] nvme_core i2c_algo_bit [28082.324470] try_get_cap_refs+0x159/0x880 [ceph] [28082.326737] pps_core dca [28082.331526] __ceph_get_caps+0x4a1/0x760 [ceph] [28082.335007] fuse [28082.339621] ? __file_remove_privs+0x89/0x110 [28082.342233] irq event stamp: 146367096 [28082.346758] ? wake_bit_function+0x70/0x70 [28082.348682] hardirqs last enabled at (146367095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28082.353035] ? ceph_write_iter+0x260/0x770 [ceph] [28082.356777] hardirqs last disabled at (146367096): [] sysvec_apic_timer_interrupt+0xe/0x90 [28082.360867] ceph_write_iter+0x260/0x770 [ceph] [28082.371112] softirqs last enabled at (146367094): [] __do_softirq+0x2be/0x371 [28082.375815] ? lock_is_held_type+0xaa/0x120 [28082.385621] softirqs last disabled at (146367089): [] irq_exit_rcu+0xb0/0xd0 [28082.390146] vfs_write+0x324/0x4f0 [28082.398917] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28082.403098] ksys_pwrite64+0x65/0xa0 [28082.411689] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28082.415088] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28082.424466] Workqueue: events delayed_work [ceph] [28082.428038] do_syscall_64+0x38/0x80 [28082.435506] [28082.435507] RIP: 0010:lock_acquire+0xf0/0x2b0 [28082.440638] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28082.445335] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28082.448904] RIP: 0033:0x7f48f0eeb93b [28082.450397] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28082.454748] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28082.459792] [28082.478530] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28082.482100] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28082.487316] ORIG_RAX: 0000000000000012 [28082.506055] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28082.507544] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28082.512762] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28082.519886] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28082.523716] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28082.530842] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28082.537965] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28082.545089] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28082.552215] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28082.559337] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28082.566461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28082.573592] [28082.580711] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28082.610973] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28082.618100] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28082.625222] Call Trace: [28082.627679] [28082.629687] ? watchdog_timer_fn+0x1b4/0x220 [28082.633967] ? __hrtimer_run_queues+0x1d3/0x360 [28082.638497] ? hrtimer_interrupt+0x104/0x240 [28082.642764] ? __local_bh_enable+0x50/0x60 [28082.646859] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28082.652082] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28082.657041] [28082.659136] [28082.661234] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28082.666543] ? lock_acquire+0xf0/0x2b0 [28082.670292] ? igrab+0x43/0x50 [28082.673343] ? lock_release+0xc5/0x260 [28082.677086] _raw_spin_lock+0x30/0x40 [28082.680742] ? igrab+0x1d/0x50 [28082.683795] igrab+0x1d/0x50 [28082.686683] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28082.691830] delayed_work+0x1d8/0x280 [ceph] [28082.696111] ? process_scheduled_works+0x22a/0x510 [28082.700894] process_scheduled_works+0x2a4/0x510 [28082.705509] worker_thread+0x188/0x340 [28082.709258] ? rescuer_thread+0x310/0x310 [28082.713263] kthread+0xf5/0x130 [28082.716398] ? kthread_complete_and_exit+0x20/0x20 [28082.721185] ret_from_fork+0x31/0x40 [28082.724761] ? kthread_complete_and_exit+0x20/0x20 [28082.729547] ret_from_fork_asm+0x11/0x20 [28082.733470] [28109.949773] watchdog: BUG: soft lockup - CPU#0 stuck for 15065s! [dbench:78683] [28109.957074] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28109.961772] watchdog: BUG: soft lockup - CPU#7 stuck for 24062s! [kworker/7:4:78808] [28110.012524] irq event stamp: 46467924 [28110.012525] hardirqs last enabled at (46467923): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28110.020255] Modules linked in: [28110.023914] hardirqs last disabled at (46467924): [] sysvec_apic_timer_interrupt+0xe/0x90 [28110.034069] ceph [28110.037121] softirqs last enabled at (46467888): [] __do_softirq+0x2be/0x371 [28110.046846] libceph [28110.048769] softirqs last disabled at (46467883): [] irq_exit_rcu+0xb0/0xd0 [28110.057453] dns_resolver [28110.059640] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28110.068165] fscache netfs [28110.070784] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28110.079726] veth nft_chain_nat [28110.082448] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28110.089920] xt_MASQUERADE nf_nat [28110.093057] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28110.098534] nf_conntrack nf_defrag_ipv6 [28110.101844] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28110.120583] nf_defrag_ipv4 nft_compat [28110.124500] [28110.129717] nf_tables [28110.133460] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28110.134951] nfnetlink bridge [28110.137311] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28110.144448] stp [28110.147416] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28110.154540] llc xfs [28110.156377] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28110.163501] libcrc32c [28110.165684] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28110.172809] sunrpc [28110.175169] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28110.182290] intel_rapl_msr [28110.184387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28110.192465] intel_rapl_common [28110.195257] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28110.200993] sb_edac x86_pkg_temp_thermal [28110.204044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28110.211169] coretemp [28110.215172] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28110.222297] kvm_intel kvm [28110.224568] Call Trace: [28110.231693] irqbypass [28110.234397] [28110.236838] i2c_i801 iTCO_wdt [28110.239200] ? watchdog_timer_fn+0x1b4/0x220 [28110.241207] iTCO_vendor_support [28110.244260] ? __hrtimer_run_queues+0x1d3/0x360 [28110.248523] i2c_smbus crct10dif_pclmul [28110.251749] ? hrtimer_interrupt+0x104/0x240 [28110.256271] joydev crc32_pclmul [28110.260102] ? __local_bh_enable+0x50/0x60 [28110.264364] ipmi_ssif ghash_clmulni_intel [28110.267593] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28110.271681] mei_me mei [28110.275771] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28110.280988] lpc_ich [28110.283449] [28110.288406] sha512_ssse3 acpi_ipmi [28110.290591] [28110.292689] mfd_core [28110.296172] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28110.298270] wmi ipmi_si [28110.300544] ? queued_spin_lock_slowpath+0x18/0x2e0 [28110.305844] ipmi_devintf ipmi_msghandler [28110.308378] do_raw_spin_lock+0xb5/0xc0 [28110.313245] ioatdma acpi_pad [28110.317252] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28110.321079] acpi_power_meter [28110.324049] try_get_cap_refs+0x159/0x880 [ceph] [28110.328829] ixgbe [28110.331798] __ceph_get_caps+0x4a1/0x760 [ceph] [28110.336404] igb mdio [28110.338420] ? __file_remove_privs+0x89/0x110 [28110.342939] crc32c_intel nvme [28110.345209] ? wake_bit_function+0x70/0x70 [28110.349559] ptp [28110.352613] ? ceph_write_iter+0x260/0x770 [ceph] [28110.356701] nvme_core i2c_algo_bit [28110.358540] ceph_write_iter+0x260/0x770 [ceph] [28110.363236] pps_core [28110.366725] ? lock_is_held_type+0xaa/0x120 [28110.371243] dca fuse [28110.373517] vfs_write+0x324/0x4f0 [28110.377692] [28110.377693] irq event stamp: 146587996 [28110.379969] ksys_pwrite64+0x65/0xa0 [28110.383361] hardirqs last enabled at (146587995): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28110.384851] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28110.388594] hardirqs last disabled at (146587996): [] sysvec_apic_timer_interrupt+0xe/0x90 [28110.392175] do_syscall_64+0x38/0x80 [28110.402419] softirqs last enabled at (146587994): [] __do_softirq+0x2be/0x371 [28110.407551] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28110.417360] softirqs last disabled at (146587989): [] irq_exit_rcu+0xb0/0xd0 [28110.420933] RIP: 0033:0x7f48f0eeb93b [28110.429704] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28110.434747] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28110.443345] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28110.446915] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28110.456292] Workqueue: events delayed_work [ceph] [28110.475031] ORIG_RAX: 0000000000000012 [28110.482500] [28110.482501] RIP: 0010:lock_acquire+0xf0/0x2b0 [28110.487718] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28110.492415] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28110.496245] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28110.497737] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28110.502087] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28110.509212] [28110.509213] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28110.527949] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28110.535072] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28110.540292] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28110.547414] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28110.548911] [28110.556029] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28110.593853] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28110.600977] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28110.609052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28110.614791] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28110.621916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28110.629040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28110.636177] Call Trace: [28110.638625] [28110.640636] ? watchdog_timer_fn+0x1b4/0x220 [28110.644901] ? __hrtimer_run_queues+0x1d3/0x360 [28110.649428] ? hrtimer_interrupt+0x104/0x240 [28110.653697] ? __local_bh_enable+0x50/0x60 [28110.657789] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28110.663013] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28110.667972] [28110.670069] [28110.672179] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28110.677492] ? lock_acquire+0xf0/0x2b0 [28110.681242] ? igrab+0x43/0x50 [28110.684300] ? lock_release+0xc5/0x260 [28110.688045] _raw_spin_lock+0x30/0x40 [28110.691702] ? igrab+0x1d/0x50 [28110.694752] igrab+0x1d/0x50 [28110.697631] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28110.702781] delayed_work+0x1d8/0x280 [ceph] [28110.707061] ? process_scheduled_works+0x22a/0x510 [28110.711851] process_scheduled_works+0x2a4/0x510 [28110.716476] worker_thread+0x188/0x340 [28110.720225] ? rescuer_thread+0x310/0x310 [28110.724227] kthread+0xf5/0x130 [28110.727367] ? kthread_complete_and_exit+0x20/0x20 [28110.732176] ret_from_fork+0x31/0x40 [28110.735754] ? kthread_complete_and_exit+0x20/0x20 [28110.740538] ret_from_fork_asm+0x11/0x20 [28110.744467] [28137.950268] watchdog: BUG: soft lockup - CPU#0 stuck for 15091s! [dbench:78683] [28137.957570] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28137.962268] watchdog: BUG: soft lockup - CPU#7 stuck for 24088s! [kworker/7:4:78808] [28138.013013] irq event stamp: 46526132 [28138.013014] hardirqs last enabled at (46526131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28138.020743] Modules linked in: [28138.024399] hardirqs last disabled at (46526132): [] sysvec_apic_timer_interrupt+0xe/0x90 [28138.034557] ceph [28138.037609] softirqs last enabled at (46526094): [] __do_softirq+0x2be/0x371 [28138.047332] libceph [28138.049258] softirqs last disabled at (46526089): [] irq_exit_rcu+0xb0/0xd0 [28138.057943] dns_resolver fscache [28138.060125] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28138.068637] netfs veth [28138.071964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28138.080909] nft_chain_nat [28138.083351] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28138.090823] xt_MASQUERADE nf_nat [28138.093526] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28138.099005] nf_conntrack nf_defrag_ipv6 [28138.102315] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28138.121054] nf_defrag_ipv4 [28138.124970] [28138.124971] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28138.130188] nft_compat nf_tables [28138.132979] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28138.134470] nfnetlink bridge [28138.141592] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28138.144903] stp llc [28138.152028] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28138.154992] xfs libcrc32c [28138.162116] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28138.164301] sunrpc [28138.171425] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28138.174128] intel_rapl_msr [28138.181253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28138.183350] intel_rapl_common sb_edac [28138.191427] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28138.194218] x86_pkg_temp_thermal coretemp [28138.199955] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28138.203700] kvm_intel [28138.210824] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28138.214914] kvm irqbypass [28138.222039] Call Trace: [28138.224396] i2c_i801 iTCO_wdt [28138.231520] [28138.234224] iTCO_vendor_support i2c_smbus [28138.236670] ? watchdog_timer_fn+0x1b4/0x220 [28138.239718] crct10dif_pclmul joydev [28138.241729] ? __hrtimer_run_queues+0x1d3/0x360 [28138.245819] crc32_pclmul ipmi_ssif [28138.250089] ? hrtimer_interrupt+0x104/0x240 [28138.253666] ghash_clmulni_intel mei_me [28138.258188] ? __local_bh_enable+0x50/0x60 [28138.261671] mei [28138.265949] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28138.269775] lpc_ich [28138.273866] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28138.275704] sha512_ssse3 acpi_ipmi [28138.280921] [28138.283122] mfd_core wmi [28138.288080] [28138.288082] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28138.291564] ipmi_si ipmi_devintf [28138.293673] ? queued_spin_lock_slowpath+0x18/0x2e0 [28138.296288] ipmi_msghandler [28138.298387] do_raw_spin_lock+0xb5/0xc0 [28138.303688] ioatdma acpi_pad acpi_power_meter [28138.307000] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28138.311870] ixgbe igb [28138.314749] try_get_cap_refs+0x159/0x880 [ceph] [28138.318579] mdio crc32c_intel [28138.323020] __ceph_get_caps+0x4a1/0x760 [ceph] [28138.327799] nvme ptp [28138.330163] ? __file_remove_privs+0x89/0x110 [28138.334767] nvme_core i2c_algo_bit [28138.337820] ? wake_bit_function+0x70/0x70 [28138.342344] pps_core dca [28138.344615] ? ceph_write_iter+0x260/0x770 [ceph] [28138.348963] fuse [28138.352450] ceph_write_iter+0x260/0x770 [ceph] [28138.356539] irq event stamp: 146808814 [28138.359161] ? lock_is_held_type+0xaa/0x120 [28138.363853] hardirqs last enabled at (146808813): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28138.365781] vfs_write+0x324/0x4f0 [28138.370303] hardirqs last disabled at (146808814): [] sysvec_apic_timer_interrupt+0xe/0x90 [28138.374051] ksys_pwrite64+0x65/0xa0 [28138.378222] softirqs last enabled at (146808812): [] __do_softirq+0x2be/0x371 [28138.388467] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28138.391865] softirqs last disabled at (146808807): [] irq_exit_rcu+0xb0/0xd0 [28138.401678] do_syscall_64+0x38/0x80 [28138.405247] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28138.414018] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28138.419147] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28138.427747] RIP: 0033:0x7f48f0eeb93b [28138.431317] Workqueue: events delayed_work [ceph] [28138.440695] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28138.445737] [28138.453209] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28138.456778] RIP: 0010:lock_acquire+0xf0/0x2b0 [28138.461477] ORIG_RAX: 0000000000000012 [28138.480214] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28138.481705] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28138.486924] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28138.491274] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28138.495105] [28138.495106] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28138.513841] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28138.520965] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28138.526182] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28138.533308] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28138.534799] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28138.541923] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28138.549052] [28138.556171] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28138.593986] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28138.602062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28138.607800] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28138.614923] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28138.622049] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28138.629172] Call Trace: [28138.631617] [28138.633628] ? watchdog_timer_fn+0x1b4/0x220 [28138.637892] ? __hrtimer_run_queues+0x1d3/0x360 [28138.642418] ? hrtimer_interrupt+0x104/0x240 [28138.646688] ? __local_bh_enable+0x50/0x60 [28138.650780] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28138.656006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28138.660963] [28138.663058] [28138.665157] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28138.670465] ? lock_acquire+0xf0/0x2b0 [28138.674214] ? igrab+0x43/0x50 [28138.677264] ? lock_release+0xc5/0x260 [28138.681009] _raw_spin_lock+0x30/0x40 [28138.684675] ? igrab+0x1d/0x50 [28138.687724] igrab+0x1d/0x50 [28138.690604] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28138.695752] delayed_work+0x1d8/0x280 [ceph] [28138.700034] ? process_scheduled_works+0x22a/0x510 [28138.704824] process_scheduled_works+0x2a4/0x510 [28138.709439] worker_thread+0x188/0x340 [28138.713188] ? rescuer_thread+0x310/0x310 [28138.717193] kthread+0xf5/0x130 [28138.720329] ? kthread_complete_and_exit+0x20/0x20 [28138.725115] ret_from_fork+0x31/0x40 [28138.728692] ? kthread_complete_and_exit+0x20/0x20 [28138.733478] ret_from_fork_asm+0x11/0x20 [28138.737399] [28165.950765] watchdog: BUG: soft lockup - CPU#0 stuck for 15117s! [dbench:78683] [28165.958067] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28165.962765] watchdog: BUG: soft lockup - CPU#7 stuck for 24114s! [kworker/7:4:78808] [28166.013510] irq event stamp: 46584362 [28166.021239] Modules linked in: ceph libceph [28166.024896] hardirqs last enabled at (46584361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28166.029073] dns_resolver fscache netfs [28166.039231] hardirqs last disabled at (46584362): [] sysvec_apic_timer_interrupt+0xe/0x90 [28166.043062] veth nft_chain_nat xt_MASQUERADE [28166.052787] softirqs last enabled at (46584324): [] __do_softirq+0x2be/0x371 [28166.052788] nf_nat [28166.057136] nf_conntrack nf_defrag_ipv6 [28166.065838] softirqs last disabled at (46584319): [] irq_exit_rcu+0xb0/0xd0 [28166.067936] nf_defrag_ipv4 nft_compat [28166.071855] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28166.080365] nf_tables nfnetlink [28166.084108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28166.093053] bridge [28166.096276] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28166.103748] stp [28166.105845] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28166.111322] llc xfs [28166.113160] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28166.131896] libcrc32c sunrpc [28166.134080] [28166.134081] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28166.139299] intel_rapl_msr intel_rapl_common [28166.142262] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28166.143753] sb_edac x86_pkg_temp_thermal [28166.150877] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28166.155228] coretemp kvm_intel [28166.162353] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28166.166357] kvm irqbypass [28166.173479] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28166.176619] i2c_i801 iTCO_wdt [28166.183742] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28166.186446] iTCO_vendor_support i2c_smbus [28166.193569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28166.196621] crct10dif_pclmul joydev [28166.204697] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28166.208789] crc32_pclmul ipmi_ssif [28166.214526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28166.218098] ghash_clmulni_intel [28166.225222] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28166.228706] mei_me mei [28166.235829] Call Trace: [28166.235830] [28166.239053] lpc_ich sha512_ssse3 [28166.246181] ? watchdog_timer_fn+0x1b4/0x220 [28166.248622] acpi_ipmi mfd_core [28166.251066] ? __hrtimer_run_queues+0x1d3/0x360 [28166.253077] wmi ipmi_si [28166.256390] ? hrtimer_interrupt+0x104/0x240 [28166.260651] ipmi_devintf ipmi_msghandler [28166.263789] ? __local_bh_enable+0x50/0x60 [28166.268313] ioatdma acpi_pad [28166.270846] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28166.275107] acpi_power_meter [28166.279112] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28166.283202] ixgbe [28166.286166] [28166.291385] igb mdio [28166.294347] [28166.294349] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28166.299305] crc32c_intel nvme [28166.301320] ? queued_spin_lock_slowpath+0x16/0x2e0 [28166.303414] ptp nvme_core i2c_algo_bit [28166.305686] do_raw_spin_lock+0xb5/0xc0 [28166.307782] pps_core dca [28166.313087] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28166.316136] fuse [28166.321028] try_get_cap_refs+0x159/0x880 [ceph] [28166.324857] irq event stamp: 147029712 [28166.328691] __ceph_get_caps+0x4a1/0x760 [ceph] [28166.331303] hardirqs last enabled at (147029711): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28166.336094] ? __file_remove_privs+0x89/0x110 [28166.338012] hardirqs last disabled at (147029712): [] sysvec_apic_timer_interrupt+0xe/0x90 [28166.342623] ? wake_bit_function+0x70/0x70 [28166.346368] softirqs last enabled at (147029710): [] __do_softirq+0x2be/0x371 [28166.350894] ? ceph_write_iter+0x260/0x770 [ceph] [28166.361135] softirqs last disabled at (147029705): [] irq_exit_rcu+0xb0/0xd0 [28166.365487] ceph_write_iter+0x260/0x770 [ceph] [28166.375297] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28166.379392] ? lock_is_held_type+0xaa/0x120 [28166.388165] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28166.392865] vfs_write+0x324/0x4f0 [28166.401463] Workqueue: events delayed_work [ceph] [28166.405990] ksys_pwrite64+0x65/0xa0 [28166.415362] [28166.415363] RIP: 0010:lock_acquire+0x34/0x2b0 [28166.419541] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28166.427011] Code: 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 <0f> 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 [28166.430411] do_syscall_64+0x38/0x80 [28166.435105] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [28166.438678] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28166.440167] [28166.440168] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [28166.444518] RIP: 0033:0x7f48f0eeb93b [28166.449650] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [28166.468387] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28166.471957] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [28166.477174] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28166.482220] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [28166.483708] ORIG_RAX: 0000000000000012 [28166.490833] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28166.494403] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28166.501529] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28166.520267] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28166.527391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28166.532609] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28166.539733] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28166.543562] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28166.550686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28166.557811] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28166.565888] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28166.573019] [28166.578750] Call Trace: [28166.578751] [28166.628135] ? watchdog_timer_fn+0x1b4/0x220 [28166.632400] ? __hrtimer_run_queues+0x1d3/0x360 [28166.636926] ? hrtimer_interrupt+0x104/0x240 [28166.641194] ? __local_bh_enable+0x50/0x60 [28166.645287] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28166.650503] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28166.655460] [28166.657557] [28166.659657] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28166.664964] ? lock_acquire+0x34/0x2b0 [28166.668715] ? igrab+0x43/0x50 [28166.671763] ? lock_release+0xc5/0x260 [28166.675509] _raw_spin_lock+0x30/0x40 [28166.679169] ? igrab+0x1d/0x50 [28166.682224] igrab+0x1d/0x50 [28166.685102] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28166.690253] delayed_work+0x1d8/0x280 [ceph] [28166.694532] ? process_scheduled_works+0x22a/0x510 [28166.699315] process_scheduled_works+0x2a4/0x510 [28166.703931] worker_thread+0x188/0x340 [28166.707679] ? rescuer_thread+0x310/0x310 [28166.711683] kthread+0xf5/0x130 [28166.714820] ? kthread_complete_and_exit+0x20/0x20 [28166.719603] ret_from_fork+0x31/0x40 [28166.723175] ? kthread_complete_and_exit+0x20/0x20 [28166.727959] ret_from_fork_asm+0x11/0x20 [28166.731881] [28193.951261] watchdog: BUG: soft lockup - CPU#0 stuck for 15143s! [dbench:78683] [28193.958564] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28193.963261] watchdog: BUG: soft lockup - CPU#7 stuck for 24140s! [kworker/7:4:78808] [28194.014015] irq event stamp: 46642550 [28194.021744] Modules linked in: ceph libceph dns_resolver fscache netfs veth [28194.025402] hardirqs last enabled at (46642549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28194.032362] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28194.042518] hardirqs last disabled at (46642550): [] sysvec_apic_timer_interrupt+0xe/0x90 [28194.042521] nf_defrag_ipv6 [28194.042522] softirqs last enabled at (46642512): [] __do_softirq+0x2be/0x371 [28194.048169] nf_defrag_ipv4 [28194.057894] softirqs last disabled at (46642507): [] irq_exit_rcu+0xb0/0xd0 [28194.060685] nft_compat nf_tables [28194.069369] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28194.072159] nfnetlink bridge [28194.080672] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28194.083980] stp llc [28194.092925] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28194.095889] xfs [28194.103359] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28194.105544] libcrc32c sunrpc [28194.111022] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28194.112858] intel_rapl_msr intel_rapl_common [28194.131598] [28194.134560] sb_edac x86_pkg_temp_thermal [28194.139778] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28194.144130] coretemp kvm_intel [28194.145618] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28194.149623] kvm irqbypass [28194.156748] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28194.159884] i2c_i801 [28194.167008] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28194.169713] iTCO_wdt iTCO_vendor_support [28194.176838] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28194.179107] i2c_smbus crct10dif_pclmul [28194.186233] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28194.190235] joydev crc32_pclmul [28194.197360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28194.201192] ipmi_ssif ghash_clmulni_intel [28194.209269] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28194.212493] mei_me mei [28194.218231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28194.222322] lpc_ich [28194.229446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28194.231889] sha512_ssse3 acpi_ipmi [28194.239014] Call Trace: [28194.239016] [28194.241198] mfd_core wmi [28194.248324] ? watchdog_timer_fn+0x1b4/0x220 [28194.251804] ipmi_si ipmi_devintf [28194.254250] ? __hrtimer_run_queues+0x1d3/0x360 [28194.256260] ipmi_msghandler ioatdma [28194.258880] ? hrtimer_interrupt+0x104/0x240 [28194.263140] acpi_pad acpi_power_meter [28194.266453] ? __local_bh_enable+0x50/0x60 [28194.270975] ixgbe igb [28194.274550] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28194.278810] mdio [28194.282555] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28194.286661] crc32c_intel [28194.289020] [28194.294237] nvme ptp [28194.296161] [28194.296163] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28194.301120] nvme_core i2c_algo_bit [28194.303743] ? queued_spin_lock_slowpath+0x18/0x2e0 [28194.305834] pps_core dca [28194.308106] do_raw_spin_lock+0xb5/0xc0 [28194.310204] fuse [28194.315509] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28194.318991] [28194.318992] irq event stamp: 147250498 [28194.323866] try_get_cap_refs+0x159/0x880 [ceph] [28194.326478] hardirqs last enabled at (147250497): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28194.330316] __ceph_get_caps+0x4a1/0x760 [ceph] [28194.332233] hardirqs last disabled at (147250498): [] sysvec_apic_timer_interrupt+0xe/0x90 [28194.337022] ? __file_remove_privs+0x89/0x110 [28194.338509] softirqs last enabled at (147250496): [] __do_softirq+0x2be/0x371 [28194.342253] ? wake_bit_function+0x70/0x70 [28194.346863] softirqs last disabled at (147250491): [] irq_exit_rcu+0xb0/0xd0 [28194.357110] ? ceph_write_iter+0x260/0x770 [ceph] [28194.361632] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28194.371442] ceph_write_iter+0x260/0x770 [ceph] [28194.375794] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28194.384567] ? lock_is_held_type+0xaa/0x120 [28194.388663] Workqueue: events delayed_work [ceph] [28194.397263] vfs_write+0x324/0x4f0 [28194.401959] [28194.401960] RIP: 0010:lock_release+0x14e/0x260 [28194.411340] ksys_pwrite64+0x65/0xa0 [28194.415860] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [28194.423331] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28194.427508] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [28194.432208] do_syscall_64+0x38/0x80 [28194.435602] [28194.435603] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [28194.437093] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28194.441530] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [28194.445103] RIP: 0033:0x7f48f0eeb93b [28194.463838] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [28194.468972] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28194.474188] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [28194.477759] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28194.479248] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [28194.479249] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28194.486373] ORIG_RAX: 0000000000000012 [28194.491418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28194.498541] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28194.502112] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28194.509237] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28194.527974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28194.535096] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28194.540316] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28194.547440] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28194.555516] Call Trace: [28194.555518] [28194.559347] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28194.565085] ? watchdog_timer_fn+0x1b4/0x220 [28194.572214] [28194.579335] ? __hrtimer_run_queues+0x1d3/0x360 [28194.637510] ? hrtimer_interrupt+0x104/0x240 [28194.641779] ? __local_bh_enable+0x50/0x60 [28194.645871] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28194.651096] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28194.656052] [28194.658150] [28194.660247] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28194.665552] ? igrab+0x43/0x50 [28194.668605] ? lock_release+0x14e/0x260 [28194.672434] ? lock_release+0xc5/0x260 [28194.676178] _raw_spin_unlock+0x1b/0x40 [28194.680009] igrab+0x43/0x50 [28194.682886] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28194.688036] delayed_work+0x1d8/0x280 [ceph] [28194.692317] ? process_scheduled_works+0x22a/0x510 [28194.697108] process_scheduled_works+0x2a4/0x510 [28194.701722] worker_thread+0x188/0x340 [28194.705472] ? rescuer_thread+0x310/0x310 [28194.709475] kthread+0xf5/0x130 [28194.712614] ? kthread_complete_and_exit+0x20/0x20 [28194.717397] ret_from_fork+0x31/0x40 [28194.720977] ? kthread_complete_and_exit+0x20/0x20 [28194.725760] ret_from_fork_asm+0x11/0x20 [28194.729683] [28221.951758] watchdog: BUG: soft lockup - CPU#0 stuck for 15169s! [dbench:78683] [28221.959061] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28221.963758] watchdog: BUG: soft lockup - CPU#7 stuck for 24166s! [kworker/7:4:78808] [28222.014503] irq event stamp: 46700744 [28222.022232] Modules linked in: ceph libceph [28222.025888] hardirqs last enabled at (46700743): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28222.030066] dns_resolver fscache netfs [28222.040224] hardirqs last disabled at (46700744): [] sysvec_apic_timer_interrupt+0xe/0x90 [28222.040227] veth [28222.044054] softirqs last enabled at (46700706): [] __do_softirq+0x2be/0x371 [28222.053779] nft_chain_nat xt_MASQUERADE [28222.055703] softirqs last disabled at (46700701): [] irq_exit_rcu+0xb0/0xd0 [28222.064387] nf_nat nf_conntrack [28222.068306] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28222.076816] nf_defrag_ipv6 nf_defrag_ipv4 [28222.080039] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28222.088985] nft_compat nf_tables [28222.093074] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28222.100546] nfnetlink bridge [28222.103857] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28222.109334] stp llc [28222.112297] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28222.131035] xfs libcrc32c [28222.133218] [28222.138453] sunrpc intel_rapl_msr [28222.141157] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28222.142648] intel_rapl_common [28222.146047] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28222.153170] sb_edac [28222.156220] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28222.163346] x86_pkg_temp_thermal coretemp [28222.165528] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28222.172653] kvm_intel kvm [28222.176743] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28222.183868] irqbypass i2c_i801 [28222.186571] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28222.193695] iTCO_wdt iTCO_vendor_support [28222.196833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28222.204912] i2c_smbus crct10dif_pclmul [28222.208915] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28222.214652] joydev crc32_pclmul [28222.218485] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28222.225609] ipmi_ssif ghash_clmulni_intel [28222.228831] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28222.235957] mei_me mei [28222.240045] Call Trace: [28222.240047] [28222.247172] lpc_ich sha512_ssse3 [28222.249616] ? watchdog_timer_fn+0x1b4/0x220 [28222.252059] acpi_ipmi mfd_core [28222.254071] ? __hrtimer_run_queues+0x1d3/0x360 [28222.257380] wmi ipmi_si [28222.261647] ? hrtimer_interrupt+0x104/0x240 [28222.264782] ipmi_devintf [28222.269306] ? __local_bh_enable+0x50/0x60 [28222.271837] ipmi_msghandler ioatdma [28222.276102] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28222.278717] acpi_pad acpi_power_meter [28222.282810] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28222.286378] ixgbe [28222.291597] [28222.295342] igb mdio [28222.300299] [28222.300301] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28222.302309] crc32c_intel nvme [28222.304411] ? queued_spin_lock_slowpath+0x18/0x2e0 [28222.306677] ptp nvme_core i2c_algo_bit [28222.308777] do_raw_spin_lock+0xb5/0xc0 [28222.314078] pps_core dca [28222.317131] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28222.322001] fuse [28222.325834] try_get_cap_refs+0x159/0x880 [ceph] [28222.329662] irq event stamp: 147471530 [28222.332285] __ceph_get_caps+0x4a1/0x760 [ceph] [28222.337063] hardirqs last enabled at (147471529): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28222.338992] ? __file_remove_privs+0x89/0x110 [28222.343597] hardirqs last disabled at (147471530): [] sysvec_apic_timer_interrupt+0xe/0x90 [28222.347342] ? wake_bit_function+0x70/0x70 [28222.351867] softirqs last enabled at (147471528): [] __do_softirq+0x2be/0x371 [28222.362114] ? ceph_write_iter+0x260/0x770 [ceph] [28222.366462] softirqs last disabled at (147471523): [] irq_exit_rcu+0xb0/0xd0 [28222.376273] ceph_write_iter+0x260/0x770 [ceph] [28222.380363] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28222.389156] ? lock_is_held_type+0xaa/0x120 [28222.393848] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28222.402447] vfs_write+0x324/0x4f0 [28222.406970] Workqueue: events delayed_work [ceph] [28222.416353] ksys_pwrite64+0x65/0xa0 [28222.420524] [28222.427997] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28222.431392] RIP: 0010:lock_acquire+0xf0/0x2b0 [28222.436093] do_syscall_64+0x38/0x80 [28222.439662] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28222.441160] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28222.446292] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28222.450641] RIP: 0033:0x7f48f0eeb93b [28222.454213] [28222.454214] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28222.472951] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28222.477993] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28222.483212] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28222.486782] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28222.488273] ORIG_RAX: 0000000000000012 [28222.495397] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28222.514135] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28222.521277] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28222.526493] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28222.533618] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28222.537450] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28222.544574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28222.551697] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28222.558822] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28222.565946] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28222.574024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28222.581169] [28222.586900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28222.624717] Call Trace: [28222.627169] [28222.629181] ? watchdog_timer_fn+0x1b4/0x220 [28222.633457] ? __hrtimer_run_queues+0x1d3/0x360 [28222.637987] ? hrtimer_interrupt+0x104/0x240 [28222.642249] ? __local_bh_enable+0x50/0x60 [28222.646341] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28222.651565] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28222.656522] [28222.658620] [28222.660717] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28222.666026] ? lock_acquire+0xf0/0x2b0 [28222.669775] ? igrab+0x43/0x50 [28222.672827] ? lock_release+0xc5/0x260 [28222.676571] _raw_spin_lock+0x30/0x40 [28222.680228] ? igrab+0x1d/0x50 [28222.683277] igrab+0x1d/0x50 [28222.686165] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28222.691314] delayed_work+0x1d8/0x280 [ceph] [28222.695594] ? process_scheduled_works+0x22a/0x510 [28222.700376] process_scheduled_works+0x2a4/0x510 [28222.704992] worker_thread+0x188/0x340 [28222.708740] ? rescuer_thread+0x310/0x310 [28222.712743] kthread+0xf5/0x130 [28222.715883] ? kthread_complete_and_exit+0x20/0x20 [28222.720667] ret_from_fork+0x31/0x40 [28222.724238] ? kthread_complete_and_exit+0x20/0x20 [28222.729022] ret_from_fork_asm+0x11/0x20 [28222.732945] [28223.052143] rcu: INFO: rcu_preempt self-detected stall on CPU [28223.057902] rcu: 0-....: (15920265 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=7961256 [28223.068328] rcu: (t=16286154 jiffies g=1309609 q=494920 ncpus=8) [28223.074412] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28223.083356] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28223.090825] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28223.096303] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28223.115041] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28223.120258] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28223.127382] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28223.134505] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28223.141631] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28223.148754] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28223.155880] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28223.163957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28223.169694] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28223.176818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28223.183943] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28223.191066] Call Trace: [28223.193510] [28223.195521] ? rcu_dump_cpu_stacks+0xd1/0x180 [28223.199880] ? rcu_sched_clock_irq+0x633/0x1390 [28223.204405] ? lock_is_held_type+0xaa/0x120 [28223.208584] ? lock_is_held_type+0xaa/0x120 [28223.212761] ? tick_sched_do_timer+0xa0/0xa0 [28223.217032] ? rcu_read_lock_sched_held+0x56/0x90 [28223.221731] ? tick_sched_do_timer+0xa0/0xa0 [28223.225993] ? update_process_times+0x5f/0x90 [28223.230346] ? tick_sched_handle.isra.16+0x1e/0x50 [28223.235137] ? tick_sched_timer+0x6f/0x80 [28223.239143] ? __hrtimer_run_queues+0x1d3/0x360 [28223.243668] ? hrtimer_interrupt+0x104/0x240 [28223.247928] ? __local_bh_enable+0x50/0x60 [28223.252024] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28223.257246] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28223.262205] [28223.264300] [28223.266398] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28223.271708] ? queued_spin_lock_slowpath+0x18/0x2e0 [28223.276584] do_raw_spin_lock+0xb5/0xc0 [28223.280414] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28223.285217] try_get_cap_refs+0x159/0x880 [ceph] [28223.289846] __ceph_get_caps+0x4a1/0x760 [ceph] [28223.294388] ? __file_remove_privs+0x89/0x110 [28223.298744] ? wake_bit_function+0x70/0x70 [28223.302836] ? ceph_write_iter+0x260/0x770 [ceph] [28223.307549] ceph_write_iter+0x260/0x770 [ceph] [28223.312086] ? lock_is_held_type+0xaa/0x120 [28223.316269] vfs_write+0x324/0x4f0 [28223.319668] ksys_pwrite64+0x65/0xa0 [28223.323245] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28223.328375] do_syscall_64+0x38/0x80 [28223.331947] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28223.336990] RIP: 0033:0x7f48f0eeb93b [28223.340561] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28223.359297] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [28223.366856] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28223.373980] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28223.381103] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28223.388227] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28223.395351] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28223.402482] [28223.404669] Sending NMI from CPU 0 to CPUs 7: [28223.409022] NMI backtrace for cpu 7 [28223.409024] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28223.409026] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28223.409027] Workqueue: events delayed_work [ceph] [28223.409047] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [28223.409050] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [28223.409051] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [28223.409053] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [28223.409054] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [28223.409055] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [28223.409057] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28223.409058] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28223.409059] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28223.409061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28223.409062] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28223.409063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28223.409064] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28223.409066] Call Trace: [28223.409067] [28223.409068] ? nmi_cpu_backtrace+0xad/0x120 [28223.409072] ? nmi_cpu_backtrace_handler+0x11/0x20 [28223.409074] ? nmi_handle+0xd4/0x260 [28223.409080] ? default_do_nmi+0x49/0xf0 [28223.409083] ? exc_nmi+0x11e/0x150 [28223.409085] ? end_repeat_nmi+0x16/0x67 [28223.409092] ? do_raw_spin_lock+0x42/0xc0 [28223.409095] ? do_raw_spin_lock+0x42/0xc0 [28223.409098] ? do_raw_spin_lock+0x42/0xc0 [28223.409101] [28223.409102] [28223.409102] igrab+0x1d/0x50 [28223.409105] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28223.409124] delayed_work+0x1d8/0x280 [ceph] [28223.409143] ? process_scheduled_works+0x22a/0x510 [28223.409146] process_scheduled_works+0x2a4/0x510 [28223.409153] worker_thread+0x188/0x340 [28223.409174] ? rescuer_thread+0x310/0x310 [28223.409177] kthread+0xf5/0x130 [28223.409180] ? kthread_complete_and_exit+0x20/0x20 [28223.409183] ret_from_fork+0x31/0x40 [28223.409186] ? kthread_complete_and_exit+0x20/0x20 [28223.409189] ret_from_fork_asm+0x11/0x20 [28223.409197] [28249.952255] watchdog: BUG: soft lockup - CPU#0 stuck for 15195s! [dbench:78683] [28249.959557] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28249.964255] watchdog: BUG: soft lockup - CPU#7 stuck for 24192s! [kworker/7:4:78808] [28250.015006] irq event stamp: 46758146 [28250.022736] Modules linked in: ceph libceph dns_resolver fscache [28250.026394] hardirqs last enabled at (46758145): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28250.032392] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [28250.042549] hardirqs last disabled at (46758146): [] sysvec_apic_timer_interrupt+0xe/0x90 [28250.048026] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [28250.057750] softirqs last enabled at (46758118): [] __do_softirq+0x2be/0x371 [28250.057754] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [28250.074340] softirqs last disabled at (46758113): [] irq_exit_rcu+0xb0/0xd0 [28250.074343] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [28250.088770] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28250.096326] irqbypass i2c_i801 iTCO_wdt [28250.105271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28250.109187] iTCO_vendor_support i2c_smbus crct10dif_pclmul [28250.116660] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28250.122224] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [28250.127701] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28250.133524] mei_me mei [28250.152262] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28250.152263] lpc_ich sha512_ssse3 [28250.154708] [28250.159930] acpi_ipmi mfd_core [28250.163243] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28250.164733] wmi ipmi_si [28250.167872] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28250.174996] ipmi_devintf ipmi_msghandler [28250.177526] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28250.184650] ioatdma acpi_pad [28250.188656] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28250.195779] acpi_power_meter ixgbe [28250.198742] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28250.205868] igb [28250.209352] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28250.216475] mdio crc32c_intel [28250.218311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28250.226391] nvme ptp [28250.229442] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28250.235179] nvme_core i2c_algo_bit [28250.237448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28250.244574] pps_core dca [28250.248057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28250.255183] fuse [28250.257798] Call Trace: [28250.257800] [28250.264932] [28250.266857] ? watchdog_timer_fn+0x1b4/0x220 [28250.269299] irq event stamp: 147690620 [28250.271311] ? __hrtimer_run_queues+0x1d3/0x360 [28250.272800] hardirqs last enabled at (147690619): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28250.277069] ? hrtimer_interrupt+0x104/0x240 [28250.280809] hardirqs last disabled at (147690620): [] sysvec_apic_timer_interrupt+0xe/0x90 [28250.285333] ? __local_bh_enable+0x50/0x60 [28250.295576] softirqs last enabled at (147690618): [] __do_softirq+0x2be/0x371 [28250.299843] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28250.309654] softirqs last disabled at (147690609): [] irq_exit_rcu+0xb0/0xd0 [28250.313742] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28250.322513] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28250.327732] [28250.336328] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28250.341287] [28250.341288] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28250.350664] Workqueue: events delayed_work [ceph] [28250.352765] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28250.360232] [28250.360233] RIP: 0010:lock_acquire+0xf0/0x2b0 [28250.362331] do_raw_spin_lock+0xb5/0xc0 [28250.367632] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28250.372351] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28250.377220] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28250.378713] try_get_cap_refs+0x159/0x880 [ceph] [28250.383059] [28250.383060] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28250.386897] __ceph_get_caps+0x4a1/0x760 [ceph] [28250.405628] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28250.410436] ? __file_remove_privs+0x89/0x110 [28250.415657] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28250.420266] ? wake_bit_function+0x70/0x70 [28250.421758] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28250.428885] ? ceph_write_iter+0x260/0x770 [ceph] [28250.433407] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28250.440529] ceph_write_iter+0x260/0x770 [ceph] [28250.444881] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28250.452009] ? lock_is_held_type+0xaa/0x120 [28250.456096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28250.463223] vfs_write+0x324/0x4f0 [28250.467933] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28250.475064] ksys_pwrite64+0x65/0xa0 [28250.479583] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28250.487663] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28250.491838] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28250.497576] do_syscall_64+0x38/0x80 [28250.500971] Call Trace: [28250.500972] [28250.508098] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28250.511667] ? watchdog_timer_fn+0x1b4/0x220 [28250.518791] RIP: 0033:0x7f48f0eeb93b [28250.523925] ? __hrtimer_run_queues+0x1d3/0x360 [28250.531049] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28250.534621] ? hrtimer_interrupt+0x104/0x240 [28250.537061] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28250.539073] ? __local_bh_enable+0x50/0x60 [28250.544115] ORIG_RAX: 0000000000000012 [28250.548382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28250.551950] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28250.556476] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28250.575212] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28250.579476] [28250.584693] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28250.588784] [28250.592614] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28250.597834] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28250.604957] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28250.609937] ? lock_acquire+0xf0/0x2b0 [28250.617061] [28250.619154] ? igrab+0x43/0x50 [28250.656941] ? lock_release+0xc5/0x260 [28250.660687] _raw_spin_lock+0x30/0x40 [28250.664344] ? igrab+0x1d/0x50 [28250.667393] igrab+0x1d/0x50 [28250.670272] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28250.675422] delayed_work+0x1d8/0x280 [ceph] [28250.679704] ? process_scheduled_works+0x22a/0x510 [28250.684494] process_scheduled_works+0x2a4/0x510 [28250.689108] worker_thread+0x188/0x340 [28250.692857] ? rescuer_thread+0x310/0x310 [28250.696860] kthread+0xf5/0x130 [28250.699999] ? kthread_complete_and_exit+0x20/0x20 [28250.704782] ret_from_fork+0x31/0x40 [28250.708361] ? kthread_complete_and_exit+0x20/0x20 [28250.713146] ret_from_fork_asm+0x11/0x20 [28250.717078] [28277.952751] watchdog: BUG: soft lockup - CPU#0 stuck for 15221s! [dbench:78683] [28277.960052] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28277.964751] watchdog: BUG: soft lockup - CPU#7 stuck for 24218s! [kworker/7:4:78808] [28278.015493] irq event stamp: 46816228 [28278.015494] hardirqs last enabled at (46816227): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28278.023226] Modules linked in: [28278.026882] hardirqs last disabled at (46816228): [] sysvec_apic_timer_interrupt+0xe/0x90 [28278.037039] ceph [28278.040090] softirqs last enabled at (46816170): [] __do_softirq+0x2be/0x371 [28278.049815] libceph [28278.051738] softirqs last disabled at (46816165): [] irq_exit_rcu+0xb0/0xd0 [28278.060435] dns_resolver fscache [28278.062625] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28278.071146] netfs [28278.074454] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28278.083398] veth nft_chain_nat [28278.085424] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28278.092898] xt_MASQUERADE [28278.096034] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28278.101511] nf_nat nf_conntrack [28278.104216] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28278.122953] nf_defrag_ipv6 [28278.126177] [28278.126178] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28278.131395] nf_defrag_ipv4 [28278.134187] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28278.135675] nft_compat nf_tables [28278.142801] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28278.145592] nfnetlink bridge [28278.152716] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28278.156026] stp llc [28278.163150] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28278.166114] xfs libcrc32c [28278.173238] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28278.175436] sunrpc [28278.182565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28278.185269] intel_rapl_msr intel_rapl_common [28278.193345] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28278.195443] sb_edac x86_pkg_temp_thermal [28278.201182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28278.205531] coretemp kvm_intel [28278.212655] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28278.216659] kvm irqbypass [28278.223785] Call Trace: [28278.223786] [28278.226922] i2c_i801 [28278.234046] ? watchdog_timer_fn+0x1b4/0x220 [28278.236749] iTCO_wdt iTCO_vendor_support [28278.239195] ? __hrtimer_run_queues+0x1d3/0x360 [28278.241205] i2c_smbus [28278.243477] ? hrtimer_interrupt+0x104/0x240 [28278.247738] crct10dif_pclmul joydev [28278.251743] ? __local_bh_enable+0x50/0x60 [28278.256266] crc32_pclmul ipmi_ssif [28278.258627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28278.262889] ghash_clmulni_intel mei_me [28278.266458] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28278.270550] mei [28278.274034] [28278.279251] lpc_ich sha512_ssse3 [28278.283082] [28278.283084] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28278.288040] acpi_ipmi mfd_core [28278.289880] ? queued_spin_lock_slowpath+0x18/0x2e0 [28278.291974] wmi ipmi_si [28278.295287] do_raw_spin_lock+0xb5/0xc0 [28278.297383] ipmi_devintf ipmi_msghandler [28278.302687] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28278.305822] ioatdma acpi_pad [28278.310696] try_get_cap_refs+0x159/0x880 [ceph] [28278.313225] acpi_power_meter ixgbe [28278.317061] __ceph_get_caps+0x4a1/0x760 [ceph] [28278.321060] igb mdio [28278.325847] ? __file_remove_privs+0x89/0x110 [28278.328808] crc32c_intel [28278.333437] ? wake_bit_function+0x70/0x70 [28278.336921] nvme ptp [28278.341447] ? ceph_write_iter+0x260/0x770 [ceph] [28278.343715] nvme_core i2c_algo_bit [28278.348067] ceph_write_iter+0x260/0x770 [ceph] [28278.350684] pps_core [28278.354778] ? lock_is_held_type+0xaa/0x120 [28278.357044] dca fuse [28278.361744] vfs_write+0x324/0x4f0 [28278.365225] [28278.365226] irq event stamp: 147911726 [28278.369755] ksys_pwrite64+0x65/0xa0 [28278.372022] hardirqs last enabled at (147911725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28278.376197] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28278.378468] hardirqs last disabled at (147911726): [] sysvec_apic_timer_interrupt+0xe/0x90 [28278.381869] do_syscall_64+0x38/0x80 [28278.383358] softirqs last enabled at (147911724): [] __do_softirq+0x2be/0x371 [28278.387101] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28278.390671] softirqs last disabled at (147911719): [] irq_exit_rcu+0xb0/0xd0 [28278.400917] RIP: 0033:0x7f48f0eeb93b [28278.406046] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28278.415859] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28278.419439] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28278.428208] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28278.433252] Workqueue: events delayed_work [ceph] [28278.441849] ORIG_RAX: 0000000000000012 [28278.441850] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28278.445435] [28278.445436] RIP: 0010:lock_acquire+0xf0/0x2b0 [28278.454815] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28278.473552] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28278.481024] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28278.486242] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28278.490939] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28278.494769] [28278.494769] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28278.501893] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28278.503383] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28278.507740] [28278.514858] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28278.585233] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28278.592359] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28278.599481] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28278.607558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28278.613297] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28278.620420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28278.627562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28278.634687] Call Trace: [28278.637132] [28278.639161] ? watchdog_timer_fn+0x1b4/0x220 [28278.643440] ? __hrtimer_run_queues+0x1d3/0x360 [28278.647969] ? hrtimer_interrupt+0x104/0x240 [28278.652237] ? __local_bh_enable+0x50/0x60 [28278.656331] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28278.661554] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28278.666511] [28278.668609] [28278.670706] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28278.676015] ? lock_acquire+0xf0/0x2b0 [28278.679764] ? igrab+0x43/0x50 [28278.682815] ? lock_release+0xc5/0x260 [28278.686560] _raw_spin_lock+0x30/0x40 [28278.690225] ? igrab+0x1d/0x50 [28278.693274] igrab+0x1d/0x50 [28278.696160] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28278.701311] delayed_work+0x1d8/0x280 [ceph] [28278.705594] ? process_scheduled_works+0x22a/0x510 [28278.710384] process_scheduled_works+0x2a4/0x510 [28278.714999] worker_thread+0x188/0x340 [28278.718749] ? rescuer_thread+0x310/0x310 [28278.722759] kthread+0xf5/0x130 [28278.725898] ? kthread_complete_and_exit+0x20/0x20 [28278.730682] ret_from_fork+0x31/0x40 [28278.734261] ? kthread_complete_and_exit+0x20/0x20 [28278.739045] ret_from_fork_asm+0x11/0x20 [28278.742966] [28305.953248] watchdog: BUG: soft lockup - CPU#0 stuck for 15247s! [dbench:78683] [28305.960551] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28305.965248] watchdog: BUG: soft lockup - CPU#7 stuck for 24244s! [kworker/7:4:78808] [28306.016001] irq event stamp: 46874512 [28306.023730] Modules linked in: ceph libceph dns_resolver fscache netfs [28306.027387] hardirqs last enabled at (46874511): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28306.027392] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [28306.044088] hardirqs last disabled at (46874512): [] sysvec_apic_timer_interrupt+0xe/0x90 [28306.053726] nf_tables nfnetlink bridge stp llc xfs libcrc32c [28306.063449] softirqs last enabled at (46874486): [] __do_softirq+0x2be/0x371 [28306.069186] sunrpc intel_rapl_msr intel_rapl_common sb_edac [28306.077871] softirqs last disabled at (46874481): [] irq_exit_rcu+0xb0/0xd0 [28306.083523] x86_pkg_temp_thermal coretemp kvm_intel kvm [28306.092035] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28306.097337] irqbypass i2c_i801 [28306.106282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28306.106284] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28306.109418] iTCO_wdt iTCO_vendor_support [28306.116889] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28306.122368] i2c_smbus [28306.126372] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28306.145109] crct10dif_pclmul [28306.147466] [28306.152684] joydev [28306.155647] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28306.157138] crc32_pclmul [28306.159236] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28306.166359] ipmi_ssif [28306.168976] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28306.176100] ghash_clmulni_intel [28306.178458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28306.185584] mei_me [28306.188807] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28306.195932] mei lpc_ich [28306.198027] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28306.205153] sha512_ssse3 [28306.207684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28306.215761] acpi_ipmi mfd_core [28306.218377] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28306.224115] wmi ipmi_si [28306.227252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28306.234378] ipmi_devintf [28306.236922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28306.244050] ipmi_msghandler [28306.246667] Call Trace: [28306.246670] [28306.253790] ioatdma acpi_pad [28306.256669] ? watchdog_timer_fn+0x1b4/0x220 [28306.259113] acpi_power_meter ixgbe [28306.261125] ? __hrtimer_run_queues+0x1d3/0x360 [28306.264086] igb mdio [28306.268353] ? hrtimer_interrupt+0x104/0x240 [28306.271835] crc32c_intel nvme [28306.276358] ? __local_bh_enable+0x50/0x60 [28306.278646] ptp nvme_core [28306.282914] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28306.285979] i2c_algo_bit pps_core [28306.290070] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28306.292773] dca fuse [28306.297993] [28306.301388] [28306.301389] irq event stamp: 148132562 [28306.306346] [28306.306347] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28306.308616] hardirqs last enabled at (148132561): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28306.310720] ? queued_spin_lock_slowpath+0x18/0x2e0 [28306.312205] hardirqs last disabled at (148132562): [] sysvec_apic_timer_interrupt+0xe/0x90 [28306.315952] do_raw_spin_lock+0xb5/0xc0 [28306.318047] softirqs last enabled at (148132560): [] __do_softirq+0x2be/0x371 [28306.323353] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28306.333595] softirqs last disabled at (148132555): [] irq_exit_rcu+0xb0/0xd0 [28306.338469] try_get_cap_refs+0x159/0x880 [ceph] [28306.348275] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28306.352112] __ceph_get_caps+0x4a1/0x760 [ceph] [28306.360876] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28306.365666] ? __file_remove_privs+0x89/0x110 [28306.374260] Workqueue: events delayed_work [ceph] [28306.378871] ? wake_bit_function+0x70/0x70 [28306.388247] [28306.392774] ? ceph_write_iter+0x260/0x770 [ceph] [28306.400242] RIP: 0010:lock_acquire+0xf0/0x2b0 [28306.404595] ceph_write_iter+0x260/0x770 [ceph] [28306.409290] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28306.413386] ? lock_is_held_type+0xaa/0x120 [28306.414872] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28306.419572] vfs_write+0x324/0x4f0 [28306.423930] [28306.423931] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28306.428458] ksys_pwrite64+0x65/0xa0 [28306.447190] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28306.451370] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28306.456585] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28306.459983] do_syscall_64+0x38/0x80 [28306.461474] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28306.468597] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28306.472169] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28306.479294] RIP: 0033:0x7f48f0eeb93b [28306.484423] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28306.491549] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28306.495118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28306.502243] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28306.507288] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28306.514410] ORIG_RAX: 0000000000000012 [28306.517981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28306.526059] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28306.544796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28306.550533] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28306.555752] Call Trace: [28306.555754] [28306.562876] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28306.566707] ? watchdog_timer_fn+0x1b4/0x220 [28306.573831] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28306.580957] ? __hrtimer_run_queues+0x1d3/0x360 [28306.588079] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28306.595206] ? hrtimer_interrupt+0x104/0x240 [28306.597659] [28306.599666] ? __local_bh_enable+0x50/0x60 [28306.640368] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28306.645592] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28306.650551] [28306.652654] [28306.654754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28306.660070] ? lock_acquire+0xf0/0x2b0 [28306.663820] ? igrab+0x43/0x50 [28306.666870] ? lock_release+0xc5/0x260 [28306.670615] _raw_spin_lock+0x30/0x40 [28306.674270] ? igrab+0x1d/0x50 [28306.677323] igrab+0x1d/0x50 [28306.680201] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28306.685350] delayed_work+0x1d8/0x280 [ceph] [28306.689652] ? process_scheduled_works+0x22a/0x510 [28306.694439] process_scheduled_works+0x2a4/0x510 [28306.699056] worker_thread+0x188/0x340 [28306.702803] ? rescuer_thread+0x310/0x310 [28306.706807] kthread+0xf5/0x130 [28306.709949] ? kthread_complete_and_exit+0x20/0x20 [28306.714737] ret_from_fork+0x31/0x40 [28306.718317] ? kthread_complete_and_exit+0x20/0x20 [28306.723100] ret_from_fork_asm+0x11/0x20 [28306.727023] [28333.953746] watchdog: BUG: soft lockup - CPU#0 stuck for 15274s! [dbench:78683] [28333.961045] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28333.965746] watchdog: BUG: soft lockup - CPU#7 stuck for 24271s! [kworker/7:4:78808] [28334.016486] irq event stamp: 46932656 [28334.016487] hardirqs last enabled at (46932655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28334.024218] Modules linked in: [28334.027874] hardirqs last disabled at (46932656): [] sysvec_apic_timer_interrupt+0xe/0x90 [28334.038033] ceph [28334.041083] softirqs last enabled at (46932646): [] __do_softirq+0x2be/0x371 [28334.050808] libceph dns_resolver [28334.052730] softirqs last disabled at (46932641): [] irq_exit_rcu+0xb0/0xd0 [28334.061419] fscache netfs [28334.064726] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28334.073237] veth nft_chain_nat [28334.075941] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28334.084886] xt_MASQUERADE [28334.088022] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28334.095494] nf_nat nf_conntrack [28334.098197] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28334.103676] nf_defrag_ipv6 nf_defrag_ipv4 [28334.106898] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28334.125637] nft_compat nf_tables [28334.129728] [28334.129729] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28334.134944] nfnetlink bridge [28334.138254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28334.139746] stp llc [28334.146870] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28334.149834] xfs libcrc32c [28334.156957] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28334.159142] sunrpc [28334.166267] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28334.168972] intel_rapl_msr intel_rapl_common [28334.176096] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28334.178192] sb_edac x86_pkg_temp_thermal [28334.185316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28334.189668] coretemp kvm_intel [28334.197745] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28334.201748] kvm irqbypass [28334.207486] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28334.210623] i2c_i801 iTCO_wdt [28334.217747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28334.220450] iTCO_vendor_support i2c_smbus [28334.227577] Call Trace: [28334.230627] crct10dif_pclmul joydev [28334.237752] [28334.241840] crc32_pclmul ipmi_ssif [28334.244285] ? watchdog_timer_fn+0x1b4/0x220 [28334.247857] ghash_clmulni_intel mei_me [28334.249867] ? __hrtimer_run_queues+0x1d3/0x360 [28334.253351] mei lpc_ich [28334.257619] ? hrtimer_interrupt+0x104/0x240 [28334.261447] sha512_ssse3 acpi_ipmi [28334.265970] ? __local_bh_enable+0x50/0x60 [28334.268501] mfd_core [28334.272766] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28334.276248] wmi ipmi_si [28334.280340] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28334.282611] ipmi_devintf [28334.287827] [28334.290357] ipmi_msghandler ioatdma [28334.295316] [28334.295318] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28334.297934] acpi_pad acpi_power_meter [28334.300034] ? queued_spin_lock_slowpath+0x18/0x2e0 [28334.303601] ixgbe igb [28334.305701] do_raw_spin_lock+0xb5/0xc0 [28334.311002] mdio crc32c_intel [28334.314747] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28334.319617] nvme ptp [28334.321976] try_get_cap_refs+0x159/0x880 [ceph] [28334.325804] nvme_core i2c_algo_bit [28334.328861] __ceph_get_caps+0x4a1/0x760 [ceph] [28334.333640] pps_core dca [28334.335914] ? __file_remove_privs+0x89/0x110 [28334.340522] fuse [28334.344006] ? wake_bit_function+0x70/0x70 [28334.348531] irq event stamp: 148353380 [28334.351149] ? ceph_write_iter+0x260/0x770 [ceph] [28334.355499] hardirqs last enabled at (148353379): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28334.357436] ceph_write_iter+0x260/0x770 [ceph] [28334.361529] hardirqs last disabled at (148353380): [] sysvec_apic_timer_interrupt+0xe/0x90 [28334.365278] ? lock_is_held_type+0xaa/0x120 [28334.369972] softirqs last enabled at (148353378): [] __do_softirq+0x2be/0x371 [28334.380218] vfs_write+0x324/0x4f0 [28334.384739] softirqs last disabled at (148353373): [] irq_exit_rcu+0xb0/0xd0 [28334.394554] ksys_pwrite64+0x65/0xa0 [28334.398728] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28334.407499] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28334.410896] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28334.419495] do_syscall_64+0x38/0x80 [28334.423066] Workqueue: events delayed_work [ceph] [28334.432444] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28334.437573] [28334.445045] RIP: 0033:0x7f48f0eeb93b [28334.448614] RIP: 0010:lock_acquire+0xf0/0x2b0 [28334.453312] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28334.458355] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28334.459847] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28334.463417] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28334.467785] ORIG_RAX: 0000000000000012 [28334.467786] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28334.486522] [28334.505261] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28334.510478] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28334.515695] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28334.519526] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28334.526650] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28334.528151] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28334.535273] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28334.542399] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28334.549528] [28334.556646] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28334.594460] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28334.602538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28334.608274] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28334.615401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28334.622523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28334.629649] Call Trace: [28334.632091] [28334.634103] ? watchdog_timer_fn+0x1b4/0x220 [28334.638367] ? __hrtimer_run_queues+0x1d3/0x360 [28334.642894] ? hrtimer_interrupt+0x104/0x240 [28334.647164] ? __local_bh_enable+0x50/0x60 [28334.651256] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28334.656472] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28334.661431] [28334.663526] [28334.665624] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28334.670931] ? lock_acquire+0xf0/0x2b0 [28334.674682] ? igrab+0x43/0x50 [28334.677731] ? lock_release+0xc5/0x260 [28334.681478] _raw_spin_lock+0x30/0x40 [28334.685153] ? igrab+0x1d/0x50 [28334.688210] igrab+0x1d/0x50 [28334.691088] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28334.696237] delayed_work+0x1d8/0x280 [ceph] [28334.700518] ? process_scheduled_works+0x22a/0x510 [28334.705308] process_scheduled_works+0x2a4/0x510 [28334.709924] worker_thread+0x188/0x340 [28334.713675] ? rescuer_thread+0x310/0x310 [28334.717686] kthread+0xf5/0x130 [28334.720823] ? kthread_complete_and_exit+0x20/0x20 [28334.725608] ret_from_fork+0x31/0x40 [28334.729178] ? kthread_complete_and_exit+0x20/0x20 [28334.733961] ret_from_fork_asm+0x11/0x20 [28334.737885] [28361.954259] watchdog: BUG: soft lockup - CPU#0 stuck for 15300s! [dbench:78683] [28361.961560] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28361.966259] watchdog: BUG: soft lockup - CPU#7 stuck for 24297s! [kworker/7:4:78808] [28362.017011] irq event stamp: 46990882 [28362.024740] Modules linked in: ceph libceph dns_resolver fscache netfs veth [28362.028397] hardirqs last enabled at (46990881): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28362.028403] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [28362.045515] hardirqs last disabled at (46990882): [] sysvec_apic_timer_interrupt+0xe/0x90 [28362.053766] nft_compat nf_tables nfnetlink bridge stp llc xfs [28362.063489] softirqs last enabled at (46990824): [] __do_softirq+0x2be/0x371 [28362.063493] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [28362.078006] softirqs last disabled at (46990819): [] irq_exit_rcu+0xb0/0xd0 [28362.088000] kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support [28362.096511] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28362.102507] i2c_smbus crct10dif_pclmul [28362.111452] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28362.115282] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [28362.122751] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28362.128578] mei_me [28362.134054] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28362.134055] mei lpc_ich sha512_ssse3 acpi_ipmi [28362.154891] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28362.159413] mfd_core wmi [28362.164630] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28362.164631] ipmi_si [28362.164632] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28362.167265] ipmi_devintf ipmi_msghandler [28362.174390] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28362.176574] ioatdma acpi_pad [28362.183697] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28362.187703] acpi_power_meter [28362.194826] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28362.197790] ixgbe igb [28362.204915] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28362.207895] mdio crc32c_intel [28362.215020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28362.217377] nvme ptp [28362.225455] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28362.228504] nvme_core i2c_algo_bit [28362.234242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28362.236512] pps_core dca [28362.243655] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28362.247140] fuse [28362.254262] Call Trace: [28362.256881] [28362.256882] irq event stamp: 148574314 [28362.264008] [28362.265928] hardirqs last enabled at (148574313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28362.268376] ? watchdog_timer_fn+0x1b4/0x220 [28362.269863] hardirqs last disabled at (148574314): [] sysvec_apic_timer_interrupt+0xe/0x90 [28362.273609] ? __hrtimer_run_queues+0x1d3/0x360 [28362.275618] softirqs last enabled at (148574312): [] __do_softirq+0x2be/0x371 [28362.285866] ? hrtimer_interrupt+0x104/0x240 [28362.290128] softirqs last disabled at (148574307): [] irq_exit_rcu+0xb0/0xd0 [28362.299946] ? __local_bh_enable+0x50/0x60 [28362.304471] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28362.313244] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28362.317505] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28362.326103] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28362.330195] Workqueue: events delayed_work [ceph] [28362.339572] [28362.344787] [28362.344788] RIP: 0010:lock_acquire+0xf0/0x2b0 [28362.352258] [28362.352260] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28362.357217] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28362.361918] ? queued_spin_lock_slowpath+0x16/0x2e0 [28362.364030] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28362.365522] do_raw_spin_lock+0xb5/0xc0 [28362.369870] [28362.369871] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28362.371970] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28362.377270] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28362.396012] try_get_cap_refs+0x159/0x880 [ceph] [28362.400879] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28362.406105] __ceph_get_caps+0x4a1/0x760 [ceph] [28362.409939] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28362.409941] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28362.411434] ? __file_remove_privs+0x89/0x110 [28362.418551] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28362.423336] ? wake_bit_function+0x70/0x70 [28362.430459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28362.435074] ? ceph_write_iter+0x260/0x770 [ceph] [28362.442194] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28362.446720] ceph_write_iter+0x260/0x770 [ceph] [28362.453842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28362.460970] ? lock_is_held_type+0xaa/0x120 [28362.465318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28362.473397] vfs_write+0x324/0x4f0 [28362.477487] Call Trace: [28362.477489] [28362.483227] ksys_pwrite64+0x65/0xa0 [28362.487921] ? watchdog_timer_fn+0x1b4/0x220 [28362.495047] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28362.499570] ? __hrtimer_run_queues+0x1d3/0x360 [28362.506694] do_syscall_64+0x38/0x80 [28362.510874] ? hrtimer_interrupt+0x104/0x240 [28362.517997] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28362.521391] ? __local_bh_enable+0x50/0x60 [28362.523836] RIP: 0033:0x7f48f0eeb93b [28362.525851] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28362.529417] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28362.533683] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28362.538812] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28362.543337] [28362.546907] ORIG_RAX: 0000000000000012 [28362.551170] [28362.556216] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28362.560306] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28362.563876] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28362.569099] ? lock_acquire+0xf0/0x2b0 [28362.587831] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28362.592791] ? igrab+0x43/0x50 [28362.598008] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28362.600105] ? lock_release+0xc5/0x260 [28362.603946] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28362.606043] _raw_spin_lock+0x30/0x40 [28362.613170] [28362.618470] ? igrab+0x1d/0x50 [28362.666415] igrab+0x1d/0x50 [28362.669294] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28362.674443] delayed_work+0x1d8/0x280 [ceph] [28362.678724] ? process_scheduled_works+0x22a/0x510 [28362.683513] process_scheduled_works+0x2a4/0x510 [28362.688130] worker_thread+0x188/0x340 [28362.691881] ? rescuer_thread+0x310/0x310 [28362.695890] kthread+0xf5/0x130 [28362.699027] ? kthread_complete_and_exit+0x20/0x20 [28362.703812] ret_from_fork+0x31/0x40 [28362.707393] ? kthread_complete_and_exit+0x20/0x20 [28362.712175] ret_from_fork_asm+0x11/0x20 [28362.716099] [28389.954771] watchdog: BUG: soft lockup - CPU#0 stuck for 15326s! [dbench:78683] [28389.962073] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28389.966771] watchdog: BUG: soft lockup - CPU#7 stuck for 24323s! [kworker/7:4:78808] [28390.017515] irq event stamp: 47048944 [28390.017517] hardirqs last enabled at (47048943): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28390.025246] Modules linked in: [28390.028903] hardirqs last disabled at (47048944): [] sysvec_apic_timer_interrupt+0xe/0x90 [28390.039060] ceph [28390.042111] softirqs last enabled at (47048886): [] __do_softirq+0x2be/0x371 [28390.051836] libceph dns_resolver [28390.053759] softirqs last disabled at (47048881): [] irq_exit_rcu+0xb0/0xd0 [28390.062447] fscache netfs [28390.065763] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28390.074273] veth nft_chain_nat [28390.076977] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28390.085922] xt_MASQUERADE [28390.089059] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28390.096530] nf_nat nf_conntrack [28390.099234] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28390.104711] nf_defrag_ipv6 nf_defrag_ipv4 [28390.107935] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28390.126673] nft_compat nf_tables [28390.130764] [28390.130764] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28390.135981] nfnetlink bridge [28390.139290] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28390.140782] stp llc [28390.147906] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28390.150870] xfs libcrc32c [28390.157994] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28390.160177] sunrpc intel_rapl_msr [28390.167304] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28390.170007] intel_rapl_common sb_edac [28390.177131] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28390.180528] x86_pkg_temp_thermal coretemp [28390.187653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28390.191396] kvm_intel kvm [28390.199473] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28390.203565] irqbypass i2c_i801 [28390.209301] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28390.212007] iTCO_wdt iTCO_vendor_support [28390.219130] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28390.222268] i2c_smbus crct10dif_pclmul [28390.229392] Call Trace: [28390.229393] [28390.233395] joydev crc32_pclmul [28390.240522] ? watchdog_timer_fn+0x1b4/0x220 [28390.244352] ipmi_ssif [28390.246797] ? __hrtimer_run_queues+0x1d3/0x360 [28390.248804] ghash_clmulni_intel [28390.252032] ? hrtimer_interrupt+0x104/0x240 [28390.256294] mei_me mei [28390.258652] ? __local_bh_enable+0x50/0x60 [28390.263175] lpc_ich sha512_ssse3 [28390.266401] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28390.270663] acpi_ipmi [28390.273109] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28390.277199] mfd_core wmi [28390.280508] [28390.285727] ipmi_si ipmi_devintf [28390.288083] [28390.288084] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28390.293040] ipmi_msghandler ioatdma [28390.295662] ? queued_spin_lock_slowpath+0x16/0x2e0 [28390.297755] acpi_pad [28390.301067] do_raw_spin_lock+0xb5/0xc0 [28390.303172] acpi_power_meter ixgbe [28390.308476] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28390.312046] igb mdio [28390.316921] try_get_cap_refs+0x159/0x880 [ceph] [28390.319190] crc32c_intel [28390.323023] __ceph_get_caps+0x4a1/0x760 [ceph] [28390.326504] nvme ptp [28390.331292] ? __file_remove_privs+0x89/0x110 [28390.333558] nvme_core [28390.338173] ? wake_bit_function+0x70/0x70 [28390.340787] i2c_algo_bit pps_core [28390.345313] ? ceph_write_iter+0x260/0x770 [ceph] [28390.347582] dca fuse [28390.351932] ceph_write_iter+0x260/0x770 [ceph] [28390.354289] [28390.358383] ? lock_is_held_type+0xaa/0x120 [28390.361777] irq event stamp: 148795496 [28390.366477] vfs_write+0x324/0x4f0 [28390.368747] hardirqs last enabled at (148795495): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28390.373273] ksys_pwrite64+0x65/0xa0 [28390.374760] hardirqs last disabled at (148795496): [] sysvec_apic_timer_interrupt+0xe/0x90 [28390.378939] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28390.382683] softirqs last enabled at (148795494): [] __do_softirq+0x2be/0x371 [28390.386080] do_syscall_64+0x38/0x80 [28390.396323] softirqs last disabled at (148795489): [] irq_exit_rcu+0xb0/0xd0 [28390.399895] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28390.409706] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28390.414837] RIP: 0033:0x7f48f0eeb93b [28390.423606] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28390.427178] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28390.435776] Workqueue: events delayed_work [ceph] [28390.440819] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28390.450197] [28390.453768] ORIG_RAX: 0000000000000012 [28390.461239] RIP: 0010:__list_del_entry_valid_or_report+0x3f/0xd0 [28390.479977] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28390.484674] Code: 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 <48> 39 f2 75 73 b8 01 00 00 00 c3 48 89 fe 48 c7 c7 98 99 1d 82 e8 [28390.489889] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28390.491382] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [28390.495213] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28390.501210] [28390.501210] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [28390.508334] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28390.527070] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [28390.534195] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28390.539413] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [28390.546542] [28390.548027] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28390.592974] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28390.600099] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28390.608176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28390.613913] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28390.621037] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28390.628179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28390.635303] Call Trace: [28390.637748] [28390.639759] ? watchdog_timer_fn+0x1b4/0x220 [28390.644022] ? __hrtimer_run_queues+0x1d3/0x360 [28390.648548] ? hrtimer_interrupt+0x104/0x240 [28390.652810] ? __local_bh_enable+0x50/0x60 [28390.656904] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28390.662128] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28390.667083] [28390.669181] [28390.671280] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28390.676587] ? __list_del_entry_valid_or_report+0x3f/0xd0 [28390.681983] ceph_check_delayed_caps+0x8f/0x270 [ceph] [28390.687133] delayed_work+0x1d8/0x280 [ceph] [28390.691414] ? process_scheduled_works+0x22a/0x510 [28390.696205] process_scheduled_works+0x2a4/0x510 [28390.700821] worker_thread+0x188/0x340 [28390.704569] ? rescuer_thread+0x310/0x310 [28390.708572] kthread+0xf5/0x130 [28390.711710] ? kthread_complete_and_exit+0x20/0x20 [28390.716494] ret_from_fork+0x31/0x40 [28390.720064] ? kthread_complete_and_exit+0x20/0x20 [28390.724849] ret_from_fork_asm+0x11/0x20 [28390.728773] [28417.955284] watchdog: BUG: soft lockup - CPU#0 stuck for 15352s! [dbench:78683] [28417.962589] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28417.967284] watchdog: BUG: soft lockup - CPU#7 stuck for 24349s! [kworker/7:4:78808] [28418.018037] irq event stamp: 47107140 [28418.018038] hardirqs last enabled at (47107139): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28418.025768] Modules linked in: [28418.029425] hardirqs last disabled at (47107140): [] sysvec_apic_timer_interrupt+0xe/0x90 [28418.039582] ceph [28418.042632] softirqs last enabled at (47107082): [] __do_softirq+0x2be/0x371 [28418.052357] libceph [28418.054280] softirqs last disabled at (47107077): [] irq_exit_rcu+0xb0/0xd0 [28418.062965] dns_resolver [28418.065152] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28418.073676] fscache [28418.076295] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28418.085239] netfs [28418.087423] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28418.094893] veth nft_chain_nat [28418.096905] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28418.102382] xt_MASQUERADE nf_nat [28418.105521] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28418.124256] nf_conntrack nf_defrag_ipv6 [28418.127568] [28418.132785] nf_defrag_ipv4 [28418.136704] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28418.138195] nft_compat nf_tables [28418.140984] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28418.148108] nfnetlink bridge [28418.151419] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28418.158542] stp llc [28418.161508] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28418.168632] xfs libcrc32c [28418.170815] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28418.177941] sunrpc intel_rapl_msr [28418.180660] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28418.187802] intel_rapl_common sb_edac [28418.191201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28418.199278] x86_pkg_temp_thermal coretemp [28418.203021] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28418.208758] kvm_intel kvm [28418.212849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28418.219973] irqbypass i2c_i801 [28418.222684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28418.229810] iTCO_wdt iTCO_vendor_support [28418.232961] Call Trace: [28418.240090] i2c_smbus crct10dif_pclmul [28418.244096] [28418.246537] joydev [28418.250371] ? watchdog_timer_fn+0x1b4/0x220 [28418.252379] crc32_pclmul [28418.254476] ? __hrtimer_run_queues+0x1d3/0x360 [28418.258740] ipmi_ssif ghash_clmulni_intel [28418.261361] ? hrtimer_interrupt+0x104/0x240 [28418.265882] mei_me mei [28418.269972] ? __local_bh_enable+0x50/0x60 [28418.274236] lpc_ich sha512_ssse3 [28418.276687] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28418.280772] acpi_ipmi [28418.284081] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28418.289300] mfd_core wmi [28418.291657] [28418.296632] ipmi_si ipmi_devintf [28418.299248] [28418.299249] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28418.301347] ipmi_msghandler ioatdma [28418.304681] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28418.306772] acpi_pad acpi_power_meter [28418.312078] do_raw_spin_lock+0xb5/0xc0 [28418.315646] ixgbe igb [28418.320519] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28418.324263] mdio crc32c_intel [28418.328097] try_get_cap_refs+0x159/0x880 [ceph] [28418.330448] nvme ptp [28418.335240] __ceph_get_caps+0x4a1/0x760 [ceph] [28418.338284] nvme_core i2c_algo_bit [28418.342901] ? __file_remove_privs+0x89/0x110 [28418.345166] pps_core [28418.349691] ? wake_bit_function+0x70/0x70 [28418.353173] dca fuse [28418.357527] ? ceph_write_iter+0x260/0x770 [ceph] [28418.359795] [28418.359796] irq event stamp: 149016568 [28418.363887] ceph_write_iter+0x260/0x770 [ceph] [28418.366156] hardirqs last enabled at (149016567): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28418.370858] ? lock_is_held_type+0xaa/0x120 [28418.372345] hardirqs last disabled at (149016568): [] sysvec_apic_timer_interrupt+0xe/0x90 [28418.376093] vfs_write+0x324/0x4f0 [28418.380614] softirqs last enabled at (149016566): [] __do_softirq+0x2be/0x371 [28418.390864] ksys_pwrite64+0x65/0xa0 [28418.395034] softirqs last disabled at (149016561): [] irq_exit_rcu+0xb0/0xd0 [28418.404847] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28418.408242] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28418.417016] do_syscall_64+0x38/0x80 [28418.420585] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28418.421659] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [28418.421662] rcu: 0-....: (16111023 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=8057180 [28418.421685] rcu: 7-....: (16004365 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/16003060 fqs=8057180 [28418.421688] rcu: (detected by 3, t=16481504 jiffies, g=1309609, q=499986 ncpus=8) [28418.421691] Sending NMI from CPU 3 to CPUs 0: [28418.421694] NMI backtrace for cpu 0 [28418.421695] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28418.421697] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28418.421697] RIP: 0010:delay_tsc+0x4d/0x90 [28418.421700] Code: c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e 65 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 48 89 c2 4c 29 c2 48 39 fa 72 c8 65 ff [28418.421701] RSP: 0018:ffffc90000003bc0 EFLAGS: 00000046 [28418.421702] RAX: 0000000007fbb583 RBX: 0000000000000000 RCX: 0000000000000000 [28418.421703] RDX: 0000000000005c10 RSI: 0000000000000000 RDI: 0000000000000dac [28418.421704] RBP: 0000000000002584 R08: 00005c1007fbaa27 R09: 0000000000000000 [28418.421705] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [28418.421705] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [28418.421706] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28418.421707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28418.421708] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28418.421709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28418.421710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28418.421710] Call Trace: [28418.421711] [28418.421712] ? nmi_cpu_backtrace+0xad/0x120 [28418.421715] ? nmi_cpu_backtrace_handler+0x11/0x20 [28418.421717] ? nmi_handle+0xd4/0x260 [28418.421721] ? default_do_nmi+0x49/0xf0 [28418.421723] ? exc_nmi+0x11e/0x150 [28418.421724] ? end_repeat_nmi+0x16/0x67 [28418.421728] ? delay_tsc+0x4d/0x90 [28418.421730] ? delay_tsc+0x4d/0x90 [28418.421732] ? delay_tsc+0x4d/0x90 [28418.421734] [28418.421735] [28418.421735] wait_for_lsr+0x27/0x70 [28418.421738] serial8250_console_write+0x341/0x3f0 [28418.421742] ? lock_release+0xc5/0x260 [28418.421744] console_flush_all+0x227/0x5a0 [28418.421747] ? console_flush_all+0x1ff/0x5a0 [28418.421748] ? console_flush_all+0x1ff/0x5a0 [28418.421751] console_unlock+0x7b/0x120 [28418.421753] vprintk_emit+0x246/0x340 [28418.421755] ? do_syscall_64+0x38/0x80 [28418.421757] ? do_syscall_64+0x38/0x80 [28418.421759] _printk+0x5c/0x80 [28418.421762] ? do_syscall_64+0x38/0x80 [28418.421764] show_trace_log_lvl+0x264/0x3f0 [28418.421769] ? do_syscall_64+0x38/0x80 [28418.421771] ? touch_softlockup_watchdog+0x20/0x20 [28418.421773] watchdog_timer_fn+0x1b4/0x220 [28418.421775] __hrtimer_run_queues+0x1d3/0x360 [28418.421777] hrtimer_interrupt+0x104/0x240 [28418.421779] ? __local_bh_enable+0x50/0x60 [28418.421782] __sysvec_apic_timer_interrupt+0x61/0x160 [28418.421784] sysvec_apic_timer_interrupt+0x6d/0x90 [28418.421786] [28418.421787] [28418.421787] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28418.421789] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28418.421792] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28418.421793] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28418.421794] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28418.421795] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28418.421795] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28418.421796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28418.421796] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28418.421800] do_raw_spin_lock+0xb5/0xc0 [28418.421802] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28418.421816] try_get_cap_refs+0x159/0x880 [ceph] [28418.421830] __ceph_get_caps+0x4a1/0x760 [ceph] [28418.421842] ? __file_remove_privs+0x89/0x110 [28418.421844] ? wake_bit_function+0x70/0x70 [28418.421846] ? ceph_write_iter+0x260/0x770 [ceph] [28418.421857] ceph_write_iter+0x260/0x770 [ceph] [28418.421869] ? lock_is_held_type+0xaa/0x120 [28418.421871] vfs_write+0x324/0x4f0 [28418.421875] ksys_pwrite64+0x65/0xa0 [28418.421877] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28418.421880] do_syscall_64+0x38/0x80 [28418.421882] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28418.421884] RIP: 0033:0x7f48f0eeb93b [28418.421885] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28418.421886] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [28418.421888] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28418.421888] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28418.421889] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28418.421890] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28418.421890] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28418.421893] [28418.422693] Sending NMI from CPU 3 to CPUs 7: [28418.422696] NMI backtrace for cpu 7 [28418.422696] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28418.422698] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28418.422700] Workqueue: events delayed_work [ceph] [28418.422722] RIP: 0010:vprintk_emit+0x1ee/0x340 [28418.422725] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [28418.422727] RSP: 0018:ffffc90000264e08 EFLAGS: 00000002 [28418.422728] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [28418.422730] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb578 [28418.422731] RBP: 0000000000000045 R08: 0000000000005ef4 R09: 0000000000000004 [28418.422732] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [28418.422733] R13: 0000000000000000 R14: ffffffff82184b8e R15: ffffc90000264e58 [28418.422735] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28418.422736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28418.422737] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28418.422739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28418.422740] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28418.422741] Call Trace: [28418.422742] [28418.422743] ? nmi_cpu_backtrace+0xad/0x120 [28418.422747] ? nmi_cpu_backtrace_handler+0x11/0x20 [28418.422749] ? nmi_handle+0xd4/0x260 [28418.422754] ? default_do_nmi+0x49/0xf0 [28418.422756] ? exc_nmi+0x11e/0x150 [28418.422758] ? end_repeat_nmi+0x16/0x67 [28418.422765] ? vprintk_emit+0x1ee/0x340 [28418.422767] ? vprintk_emit+0x1ee/0x340 [28418.422771] ? vprintk_emit+0x1ee/0x340 [28418.422773] [28418.422774] [28418.422775] ? touch_softlockup_watchdog+0x20/0x20 [28418.422778] ? touch_softlockup_watchdog+0x20/0x20 [28418.422779] _printk+0x5c/0x80 [28418.422784] dump_stack_print_info+0xc2/0xd0 [28418.422788] show_regs+0x19/0x60 [28418.422791] watchdog_timer_fn+0x1b4/0x220 [28418.422794] __hrtimer_run_queues+0x1d3/0x360 [28418.422799] hrtimer_interrupt+0x104/0x240 [28418.422800] ? __local_bh_enable+0x50/0x60 [28418.422805] __sysvec_apic_timer_interrupt+0x61/0x160 [28418.422809] sysvec_apic_timer_interrupt+0x6d/0x90 [28418.422811] [28418.422812] [28418.422813] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28418.422815] RIP: 0010:lock_acquire+0xf0/0x2b0 [28418.422818] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28418.422819] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28418.422821] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28418.422822] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28418.422823] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28418.422824] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28418.422825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28418.422832] ? igrab+0x43/0x50 [28418.422834] ? lock_release+0xc5/0x260 [28418.422837] _raw_spin_lock+0x30/0x40 [28418.422840] ? igrab+0x1d/0x50 [28418.422842] igrab+0x1d/0x50 [28418.422844] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28418.422865] delayed_work+0x1d8/0x280 [ceph] [28418.422884] ? process_scheduled_works+0x22a/0x510 [28418.422888] process_scheduled_works+0x2a4/0x510 [28418.422895] worker_thread+0x188/0x340 [28418.422898] ? rescuer_thread+0x310/0x310 [28418.422901] kthread+0xf5/0x130 [28418.422904] ? kthread_complete_and_exit+0x20/0x20 [28418.422907] ret_from_fork+0x31/0x40 [28418.422910] ? kthread_complete_and_exit+0x20/0x20 [28418.422913] ret_from_fork_asm+0x11/0x20 [28418.422921] [28418.429184] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28418.434313] Workqueue: events delayed_work [ceph] [28418.443691] RIP: 0033:0x7f48f0eeb93b [28418.447262] [28418.447263] RIP: 0010:lock_acquire+0xf0/0x2b0 [28418.454731] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28418.460816] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28418.471233] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28418.481825] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28418.489381] ORIG_RAX: 0000000000000012 [28418.493733] [28418.493734] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28418.497217] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28418.506162] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28418.513632] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28418.517636] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28418.536372] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28418.541591] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28418.548716] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28418.555840] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28418.562968] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28418.570096] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28418.577225] [28418.585297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28419.459573] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28419.466705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28419.473830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28419.480953] Call Trace: [28419.483399] [28419.485410] ? watchdog_timer_fn+0x1b4/0x220 [28419.489674] ? __hrtimer_run_queues+0x1d3/0x360 [28419.494200] ? hrtimer_interrupt+0x104/0x240 [28419.498469] ? __local_bh_enable+0x50/0x60 [28419.502564] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28419.507787] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28419.512744] [28419.514840] [28419.516938] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28419.522248] ? lock_acquire+0xf0/0x2b0 [28419.525998] ? igrab+0x43/0x50 [28419.529055] ? lock_release+0xc5/0x260 [28419.532800] _raw_spin_lock+0x30/0x40 [28419.536455] ? igrab+0x1d/0x50 [28419.539507] igrab+0x1d/0x50 [28419.542385] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28419.547534] delayed_work+0x1d8/0x280 [ceph] [28419.551817] ? process_scheduled_works+0x22a/0x510 [28419.556606] process_scheduled_works+0x2a4/0x510 [28419.561221] worker_thread+0x188/0x340 [28419.564987] ? rescuer_thread+0x310/0x310 [28419.568990] kthread+0xf5/0x130 [28419.572130] ? kthread_complete_and_exit+0x20/0x20 [28419.576914] ret_from_fork+0x31/0x40 [28419.580483] ? kthread_complete_and_exit+0x20/0x20 [28419.585268] ret_from_fork_asm+0x11/0x20 [28419.589190] [28445.955797] watchdog: BUG: soft lockup - CPU#0 stuck for 15378s! [dbench:78683] [28445.963101] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28445.967797] watchdog: BUG: soft lockup - CPU#7 stuck for 24375s! [kworker/7:4:78808] [28446.018541] irq event stamp: 47165322 [28446.018542] hardirqs last enabled at (47165321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28446.026272] Modules linked in: [28446.029929] hardirqs last disabled at (47165322): [] sysvec_apic_timer_interrupt+0xe/0x90 [28446.040088] ceph [28446.043139] softirqs last enabled at (47165264): [] __do_softirq+0x2be/0x371 [28446.052863] libceph [28446.054786] softirqs last disabled at (47165259): [] irq_exit_rcu+0xb0/0xd0 [28446.063473] dns_resolver [28446.065654] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28446.074167] fscache [28446.076800] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28446.085744] netfs veth [28446.087929] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28446.095398] nft_chain_nat xt_MASQUERADE [28446.097842] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28446.103320] nf_nat [28446.107239] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28446.125976] nf_conntrack nf_defrag_ipv6 [28446.128073] [28446.128073] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28446.133291] nf_defrag_ipv4 nft_compat [28446.137206] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28446.138700] nf_tables nfnetlink [28446.145822] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28446.149565] bridge [28446.156690] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28446.159914] stp llc [28446.167040] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28446.169136] xfs libcrc32c [28446.176262] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28446.178444] sunrpc intel_rapl_msr [28446.185568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28446.188272] intel_rapl_common sb_edac [28446.196351] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28446.199748] x86_pkg_temp_thermal coretemp [28446.205486] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28446.209229] kvm_intel kvm [28446.216354] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28446.220443] irqbypass [28446.227569] Call Trace: [28446.230273] i2c_i801 iTCO_wdt [28446.237399] [28446.239753] iTCO_vendor_support i2c_smbus [28446.242200] ? watchdog_timer_fn+0x1b4/0x220 [28446.245250] crct10dif_pclmul joydev [28446.247260] ? __hrtimer_run_queues+0x1d3/0x360 [28446.251350] crc32_pclmul [28446.255616] ? hrtimer_interrupt+0x104/0x240 [28446.259195] ipmi_ssif ghash_clmulni_intel [28446.263717] ? __local_bh_enable+0x50/0x60 [28446.266335] mei_me mei [28446.270601] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28446.274689] lpc_ich sha512_ssse3 [28446.278782] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28446.281225] acpi_ipmi mfd_core [28446.286441] [28446.289752] wmi ipmi_si [28446.294711] [28446.294713] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28446.297847] ipmi_devintf ipmi_msghandler [28446.299949] ? queued_spin_lock_slowpath+0x18/0x2e0 [28446.302491] ioatdma acpi_pad [28446.304592] do_raw_spin_lock+0xb5/0xc0 [28446.309895] acpi_power_meter [28446.313900] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28446.318769] ixgbe igb [28446.321736] try_get_cap_refs+0x159/0x880 [ceph] [28446.325563] mdio crc32c_intel [28446.328533] __ceph_get_caps+0x4a1/0x760 [ceph] [28446.333313] nvme ptp [28446.335673] ? __file_remove_privs+0x89/0x110 [28446.340281] nvme_core i2c_algo_bit [28446.343331] ? wake_bit_function+0x70/0x70 [28446.347854] pps_core dca [28446.350129] ? ceph_write_iter+0x260/0x770 [ceph] [28446.354478] fuse [28446.357960] ceph_write_iter+0x260/0x770 [ceph] [28446.362052] irq event stamp: 149230678 [28446.364671] ? lock_is_held_type+0xaa/0x120 [28446.369367] hardirqs last enabled at (149230677): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28446.371293] vfs_write+0x324/0x4f0 [28446.375814] hardirqs last disabled at (149230678): [] sysvec_apic_timer_interrupt+0xe/0x90 [28446.379563] ksys_pwrite64+0x65/0xa0 [28446.383736] softirqs last enabled at (149230676): [] __do_softirq+0x2be/0x371 [28446.393981] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28446.397376] softirqs last disabled at (149230671): [] irq_exit_rcu+0xb0/0xd0 [28446.407200] do_syscall_64+0x38/0x80 [28446.410766] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28446.419540] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28446.424670] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28446.433268] RIP: 0033:0x7f48f0eeb93b [28446.436838] Workqueue: events delayed_work [ceph] [28446.446216] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28446.451258] [28446.458731] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28446.462301] RIP: 0010:lock_acquire+0xf0/0x2b0 [28446.466999] ORIG_RAX: 0000000000000012 [28446.485735] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28446.487227] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28446.492443] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28446.496795] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28446.500626] [28446.500626] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28446.519362] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28446.526488] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28446.531704] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28446.538827] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28446.540319] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28446.547444] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28446.554574] [28446.561692] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28446.599506] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28446.607582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28446.613320] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28446.620445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28446.627570] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28446.634692] Call Trace: [28446.637136] [28446.639147] ? watchdog_timer_fn+0x1b4/0x220 [28446.643413] ? __hrtimer_run_queues+0x1d3/0x360 [28446.647938] ? hrtimer_interrupt+0x104/0x240 [28446.652201] ? __local_bh_enable+0x50/0x60 [28446.656301] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28446.661525] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28446.666485] [28446.668581] [28446.670677] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28446.675985] ? lock_acquire+0xf0/0x2b0 [28446.679736] ? igrab+0x43/0x50 [28446.682786] ? lock_release+0xc5/0x260 [28446.686530] _raw_spin_lock+0x30/0x40 [28446.690204] ? igrab+0x1d/0x50 [28446.693256] igrab+0x1d/0x50 [28446.696132] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28446.701283] delayed_work+0x1d8/0x280 [ceph] [28446.705563] ? process_scheduled_works+0x22a/0x510 [28446.710354] process_scheduled_works+0x2a4/0x510 [28446.714969] worker_thread+0x188/0x340 [28446.718720] ? rescuer_thread+0x310/0x310 [28446.722721] kthread+0xf5/0x130 [28446.725859] ? kthread_complete_and_exit+0x20/0x20 [28446.730645] ret_from_fork+0x31/0x40 [28446.734223] ? kthread_complete_and_exit+0x20/0x20 [28446.739008] ret_from_fork_asm+0x11/0x20 [28446.742931] [28473.956309] watchdog: BUG: soft lockup - CPU#0 stuck for 15404s! [dbench:78683] [28473.963615] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28473.968309] watchdog: BUG: soft lockup - CPU#7 stuck for 24401s! [kworker/7:4:78808] [28474.019066] irq event stamp: 47223486 [28474.026795] Modules linked in: ceph libceph dns_resolver fscache netfs [28474.030453] hardirqs last enabled at (47223485): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28474.036981] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [28474.047135] hardirqs last disabled at (47223486): [] sysvec_apic_timer_interrupt+0xe/0x90 [28474.054520] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [28474.064243] softirqs last enabled at (47223428): [] __do_softirq+0x2be/0x371 [28474.071021] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [28474.079705] softirqs last disabled at (47223423): [] irq_exit_rcu+0xb0/0xd0 [28474.088390] coretemp kvm_intel kvm irqbypass i2c_i801 [28474.096902] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28474.096904] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [28474.110987] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28474.117328] joydev crc32_pclmul ipmi_ssif [28474.124799] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28474.128889] ghash_clmulni_intel mei_me mei [28474.134368] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28474.138544] lpc_ich sha512_ssse3 [28474.157282] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28474.157283] acpi_ipmi [28474.157284] [28474.160594] mfd_core [28474.165811] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28474.168169] wmi ipmi_si [28474.169660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28474.171929] ipmi_devintf [28474.179055] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28474.181584] ipmi_msghandler ioatdma [28474.188709] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28474.191325] acpi_pad acpi_power_meter [28474.198450] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28474.202020] ixgbe igb [28474.209145] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28474.212889] mdio crc32c_intel [28474.220014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28474.222371] nvme ptp [28474.230448] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28474.233498] nvme_core i2c_algo_bit [28474.239237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28474.241507] pps_core [28474.248630] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28474.252116] dca fuse [28474.259239] Call Trace: [28474.261509] [28474.261510] irq event stamp: 149451638 [28474.268637] [28474.270905] hardirqs last enabled at (149451637): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28474.273350] ? watchdog_timer_fn+0x1b4/0x220 [28474.274838] hardirqs last disabled at (149451638): [] sysvec_apic_timer_interrupt+0xe/0x90 [28474.278585] ? __hrtimer_run_queues+0x1d3/0x360 [28474.280594] softirqs last enabled at (149451636): [] __do_softirq+0x2be/0x371 [28474.290842] ? hrtimer_interrupt+0x104/0x240 [28474.295101] softirqs last disabled at (149451631): [] irq_exit_rcu+0xb0/0xd0 [28474.304914] ? __local_bh_enable+0x50/0x60 [28474.309436] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28474.318210] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28474.322472] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28474.331070] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28474.335161] Workqueue: events delayed_work [ceph] [28474.344537] [28474.349756] [28474.349757] RIP: 0010:__list_del_entry_valid_or_report+0x3f/0xd0 [28474.357225] [28474.362185] Code: 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 <48> 39 f2 75 73 b8 01 00 00 00 c3 48 89 fe 48 c7 c7 98 99 1d 82 e8 [28474.366881] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28474.368992] RSP: 0018:ffffc90002117db0 EFLAGS: 00000246 [28474.370490] ? queued_spin_lock_slowpath+0x18/0x2e0 [28474.376484] [28474.376485] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [28474.378583] do_raw_spin_lock+0xb5/0xc0 [28474.397319] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [28474.402625] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28474.407840] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [28474.412715] try_get_cap_refs+0x159/0x880 [ceph] [28474.414202] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28474.421331] __ceph_get_caps+0x4a1/0x760 [ceph] [28474.425157] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28474.432286] ? __file_remove_privs+0x89/0x110 [28474.437065] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28474.444190] ? wake_bit_function+0x70/0x70 [28474.448799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28474.455927] ? ceph_write_iter+0x260/0x770 [ceph] [28474.460449] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28474.467572] ceph_write_iter+0x260/0x770 [ceph] [28474.471922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28474.480005] ? lock_is_held_type+0xaa/0x120 [28474.484091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28474.489832] vfs_write+0x324/0x4f0 [28474.494526] Call Trace: [28474.494527] [28474.501655] ksys_pwrite64+0x65/0xa0 [28474.506175] ? watchdog_timer_fn+0x1b4/0x220 [28474.513300] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28474.517478] ? __hrtimer_run_queues+0x1d3/0x360 [28474.524602] do_syscall_64+0x38/0x80 [28474.528013] ? hrtimer_interrupt+0x104/0x240 [28474.530451] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28474.532461] ? __local_bh_enable+0x50/0x60 [28474.536033] RIP: 0033:0x7f48f0eeb93b [28474.540298] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28474.545426] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28474.549950] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28474.553521] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28474.557785] [28474.562830] ORIG_RAX: 0000000000000012 [28474.566919] [28474.566920] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28474.570490] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28474.575718] ? __list_del_entry_valid_or_report+0x3f/0xd0 [28474.594456] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28474.599413] ceph_check_delayed_caps+0x8f/0x270 [ceph] [28474.604630] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28474.606731] delayed_work+0x1d8/0x280 [ceph] [28474.610558] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28474.612658] ? process_scheduled_works+0x22a/0x510 [28474.617959] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28474.625084] process_scheduled_works+0x2a4/0x510 [28474.630479] [28474.637604] worker_thread+0x188/0x340 [28474.683692] ? rescuer_thread+0x310/0x310 [28474.687720] kthread+0xf5/0x130 [28474.690867] ? kthread_complete_and_exit+0x20/0x20 [28474.695650] ret_from_fork+0x31/0x40 [28474.699228] ? kthread_complete_and_exit+0x20/0x20 [28474.704013] ret_from_fork_asm+0x11/0x20 [28474.707946] [28501.956822] watchdog: BUG: soft lockup - CPU#0 stuck for 15430s! [dbench:78683] [28501.964120] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28501.968822] watchdog: BUG: soft lockup - CPU#7 stuck for 24427s! [kworker/7:4:78808] [28502.019559] irq event stamp: 47281560 [28502.019560] hardirqs last enabled at (47281559): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28502.027292] Modules linked in: [28502.030947] hardirqs last disabled at (47281560): [] sysvec_apic_timer_interrupt+0xe/0x90 [28502.041106] ceph [28502.044157] softirqs last enabled at (47281532): [] __do_softirq+0x2be/0x371 [28502.053880] libceph dns_resolver [28502.055803] softirqs last disabled at (47281527): [] irq_exit_rcu+0xb0/0xd0 [28502.064503] fscache netfs [28502.067818] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28502.076327] veth nft_chain_nat [28502.079032] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28502.079033] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28502.087976] xt_MASQUERADE nf_nat [28502.091112] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28502.098583] nf_conntrack nf_defrag_ipv6 [28502.104061] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28502.107373] nf_defrag_ipv4 nft_compat [28502.126111] [28502.130028] nf_tables nfnetlink [28502.135245] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28502.138988] bridge stp [28502.140478] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28502.143703] llc xfs [28502.150828] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28502.153271] libcrc32c [28502.160397] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28502.162580] sunrpc intel_rapl_msr [28502.169705] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28502.172060] intel_rapl_common [28502.179186] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28502.182584] sb_edac x86_pkg_temp_thermal [28502.189708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28502.192757] coretemp kvm_intel [28502.200835] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28502.204840] kvm irqbypass [28502.210578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28502.213714] i2c_i801 iTCO_wdt [28502.220839] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28502.223541] iTCO_vendor_support i2c_smbus [28502.230666] Call Trace: [28502.233716] crct10dif_pclmul joydev [28502.240843] [28502.244933] crc32_pclmul ipmi_ssif [28502.247377] ? watchdog_timer_fn+0x1b4/0x220 [28502.250948] ghash_clmulni_intel mei_me [28502.252958] ? __hrtimer_run_queues+0x1d3/0x360 [28502.256441] mei lpc_ich [28502.260708] ? hrtimer_interrupt+0x104/0x240 [28502.264535] sha512_ssse3 [28502.269061] ? __local_bh_enable+0x50/0x60 [28502.271591] acpi_ipmi mfd_core [28502.275858] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28502.278474] wmi ipmi_si [28502.282565] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28502.285701] ipmi_devintf ipmi_msghandler [28502.290919] [28502.293448] ioatdma [28502.298406] [28502.302411] acpi_pad acpi_power_meter [28502.304509] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28502.306691] ixgbe [28502.308793] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28502.312534] igb mdio crc32c_intel [28502.317839] do_raw_spin_lock+0xb5/0xc0 [28502.319848] nvme ptp [28502.324719] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28502.328117] nvme_core [28502.331949] try_get_cap_refs+0x159/0x880 [ceph] [28502.334223] i2c_algo_bit pps_core [28502.339015] __ceph_get_caps+0x4a1/0x760 [ceph] [28502.341369] dca fuse [28502.345983] ? __file_remove_privs+0x89/0x110 [28502.349375] [28502.349376] irq event stamp: 149672824 [28502.353901] ? wake_bit_function+0x70/0x70 [28502.356172] hardirqs last enabled at (149672823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28502.360524] ? ceph_write_iter+0x260/0x770 [ceph] [28502.362011] hardirqs last disabled at (149672824): [] sysvec_apic_timer_interrupt+0xe/0x90 [28502.365757] ceph_write_iter+0x260/0x770 [ceph] [28502.369846] softirqs last enabled at (149672822): [] __do_softirq+0x2be/0x371 [28502.380096] ? lock_is_held_type+0xaa/0x120 [28502.384788] softirqs last disabled at (149672817): [] irq_exit_rcu+0xb0/0xd0 [28502.394601] vfs_write+0x324/0x4f0 [28502.399123] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28502.407899] ksys_pwrite64+0x65/0xa0 [28502.412071] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28502.420670] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28502.424066] Workqueue: events delayed_work [ceph] [28502.433447] do_syscall_64+0x38/0x80 [28502.437015] [28502.437016] RIP: 0010:lock_release+0x14e/0x260 [28502.444486] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28502.449616] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [28502.454313] RIP: 0033:0x7f48f0eeb93b [28502.457884] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [28502.459376] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28502.463813] [28502.463814] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [28502.468857] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28502.487596] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [28502.491165] ORIG_RAX: 0000000000000012 [28502.496382] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [28502.515119] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28502.516612] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [28502.523734] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28502.528953] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [28502.536078] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28502.539907] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28502.547031] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28502.554155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28502.561281] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28502.568405] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28502.575535] [28502.582653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28502.627148] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28502.634273] Call Trace: [28502.636718] [28502.638727] ? watchdog_timer_fn+0x1b4/0x220 [28502.642993] ? __hrtimer_run_queues+0x1d3/0x360 [28502.647518] ? hrtimer_interrupt+0x104/0x240 [28502.651779] ? __local_bh_enable+0x50/0x60 [28502.655873] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28502.661095] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28502.666053] [28502.668150] [28502.670248] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28502.675554] ? igrab+0x43/0x50 [28502.678605] ? lock_release+0x14e/0x260 [28502.682444] ? lock_release+0xc5/0x260 [28502.686190] _raw_spin_unlock+0x1b/0x40 [28502.690029] igrab+0x43/0x50 [28502.692906] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28502.698055] delayed_work+0x1d8/0x280 [ceph] [28502.702337] ? process_scheduled_works+0x22a/0x510 [28502.707125] process_scheduled_works+0x2a4/0x510 [28502.711741] worker_thread+0x188/0x340 [28502.715491] ? rescuer_thread+0x310/0x310 [28502.719518] kthread+0xf5/0x130 [28502.722657] ? kthread_complete_and_exit+0x20/0x20 [28502.727443] ret_from_fork+0x31/0x40 [28502.731021] ? kthread_complete_and_exit+0x20/0x20 [28502.735804] ret_from_fork_asm+0x11/0x20 [28502.739728] [28529.957336] watchdog: BUG: soft lockup - CPU#0 stuck for 15456s! [dbench:78683] [28529.964642] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28529.969335] watchdog: BUG: soft lockup - CPU#7 stuck for 24453s! [kworker/7:4:78808] [28530.020091] irq event stamp: 47339746 [28530.027821] Modules linked in: ceph libceph dns_resolver fscache netfs veth [28530.031478] hardirqs last enabled at (47339745): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28530.038439] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28530.048595] hardirqs last disabled at (47339746): [] sysvec_apic_timer_interrupt+0xe/0x90 [28530.054246] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [28530.063972] softirqs last enabled at (47339718): [] __do_softirq+0x2be/0x371 [28530.071364] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [28530.080047] softirqs last disabled at (47339713): [] irq_exit_rcu+0xb0/0xd0 [28530.086912] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [28530.095423] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28530.101419] irqbypass i2c_i801 [28530.110366] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28530.113502] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [28530.120973] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28530.127316] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [28530.132795] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28530.138619] mei_me mei [28530.157355] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28530.159802] lpc_ich sha512_ssse3 [28530.165016] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28530.165017] acpi_ipmi [28530.165019] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28530.168328] mfd_core wmi [28530.175453] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28530.177810] ipmi_si ipmi_devintf [28530.184933] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28530.187552] ipmi_msghandler [28530.194675] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28530.197987] ioatdma acpi_pad [28530.205128] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28530.208014] acpi_power_meter ixgbe [28530.215138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28530.218102] igb mdio [28530.226178] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28530.229663] crc32c_intel nvme [28530.235402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28530.237671] ptp nvme_core [28530.244795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28530.247846] i2c_algo_bit pps_core [28530.254971] Call Trace: [28530.254973] [28530.257675] dca fuse [28530.264801] ? watchdog_timer_fn+0x1b4/0x220 [28530.268197] [28530.268198] irq event stamp: 149893886 [28530.270641] ? __hrtimer_run_queues+0x1d3/0x360 [28530.272651] hardirqs last enabled at (149893885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28530.274924] ? hrtimer_interrupt+0x104/0x240 [28530.279185] hardirqs last disabled at (149893886): [] sysvec_apic_timer_interrupt+0xe/0x90 [28530.280676] ? __local_bh_enable+0x50/0x60 [28530.284419] softirqs last enabled at (149893884): [] __do_softirq+0x2be/0x371 [28530.288946] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28530.299188] softirqs last disabled at (149893879): [] irq_exit_rcu+0xb0/0xd0 [28530.303454] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28530.313263] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28530.317355] [28530.326124] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28530.331344] [28530.339940] Workqueue: events delayed_work [ceph] [28530.344897] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28530.354274] [28530.354275] RIP: 0010:lock_release+0x16a/0x260 [28530.356375] ? queued_spin_lock_slowpath+0x18/0x2e0 [28530.363844] Code: f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 c4 18 5b 5d 41 5c <41> 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 48 0f a3 05 18 bf 53 01 [28530.365943] do_raw_spin_lock+0xb5/0xc0 [28530.370637] RSP: 0018:ffffc90002117d78 EFLAGS: 00000296 [28530.375945] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28530.377433] [28530.377434] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [28530.381873] try_get_cap_refs+0x159/0x880 [ceph] [28530.386742] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [28530.405486] __ceph_get_caps+0x4a1/0x760 [ceph] [28530.409308] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [28530.414531] ? __file_remove_privs+0x89/0x110 [28530.419311] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28530.420801] ? wake_bit_function+0x70/0x70 [28530.427927] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [28530.432539] ? ceph_write_iter+0x260/0x770 [ceph] [28530.439659] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28530.444184] ceph_write_iter+0x260/0x770 [ceph] [28530.451310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28530.455664] ? lock_is_held_type+0xaa/0x120 [28530.462784] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28530.466877] vfs_write+0x324/0x4f0 [28530.473998] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28530.478702] ksys_pwrite64+0x65/0xa0 [28530.486772] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28530.491298] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28530.497036] Call Trace: [28530.501215] do_syscall_64+0x38/0x80 [28530.508337] [28530.511738] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28530.518858] ? watchdog_timer_fn+0x1b4/0x220 [28530.522431] RIP: 0033:0x7f48f0eeb93b [28530.529554] ? __hrtimer_run_queues+0x1d3/0x360 [28530.534683] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28530.537132] ? hrtimer_interrupt+0x104/0x240 [28530.540698] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28530.542708] ? __local_bh_enable+0x50/0x60 [28530.547753] ORIG_RAX: 0000000000000012 [28530.552021] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28530.555588] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28530.560113] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28530.578851] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28530.583114] [28530.588332] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28530.592423] [28530.592425] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28530.596253] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28530.601474] ? lock_release+0x16a/0x260 [28530.608595] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28530.613555] _raw_spin_unlock+0x1b/0x40 [28530.620680] [28530.622773] igrab+0x43/0x50 [28530.664273] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28530.669420] delayed_work+0x1d8/0x280 [ceph] [28530.673702] ? process_scheduled_works+0x22a/0x510 [28530.678494] process_scheduled_works+0x2a4/0x510 [28530.683109] worker_thread+0x188/0x340 [28530.686857] ? rescuer_thread+0x310/0x310 [28530.690860] kthread+0xf5/0x130 [28530.694008] ? kthread_complete_and_exit+0x20/0x20 [28530.698800] ret_from_fork+0x31/0x40 [28530.702379] ? kthread_complete_and_exit+0x20/0x20 [28530.707163] ret_from_fork_asm+0x11/0x20 [28530.711095] [28557.957847] watchdog: BUG: soft lockup - CPU#0 stuck for 15482s! [dbench:78683] [28557.965147] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28557.969847] watchdog: BUG: soft lockup - CPU#7 stuck for 24479s! [kworker/7:4:78808] [28558.020587] irq event stamp: 47397846 [28558.028318] Modules linked in: ceph libceph dns_resolver [28558.031976] hardirqs last enabled at (47397845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28558.037279] fscache netfs [28558.047437] hardirqs last disabled at (47397846): [] sysvec_apic_timer_interrupt+0xe/0x90 [28558.050141] veth nft_chain_nat xt_MASQUERADE [28558.059866] softirqs last enabled at (47397818): [] __do_softirq+0x2be/0x371 [28558.059867] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [28558.072917] softirqs last disabled at (47397813): [] irq_exit_rcu+0xb0/0xd0 [28558.072918] nft_compat [28558.072920] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28558.078742] nf_tables nfnetlink [28558.087252] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28558.089696] bridge stp [28558.098641] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28558.101863] llc xfs [28558.109335] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28558.111780] libcrc32c sunrpc [28558.117256] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28558.119440] intel_rapl_msr intel_rapl_common [28558.138177] [28558.141142] sb_edac x86_pkg_temp_thermal [28558.146359] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28558.150710] coretemp kvm_intel [28558.152200] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28558.156205] kvm [28558.163328] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28558.166466] irqbypass i2c_i801 [28558.173591] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28558.175429] iTCO_wdt iTCO_vendor_support [28558.182553] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28558.185689] i2c_smbus crct10dif_pclmul [28558.192814] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28558.196819] joydev crc32_pclmul [28558.203942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28558.207773] ipmi_ssif ghash_clmulni_intel [28558.215849] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28558.219076] mei_me mei [28558.224813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28558.228903] lpc_ich sha512_ssse3 [28558.236026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28558.238472] acpi_ipmi mfd_core [28558.245594] Call Trace: [28558.245595] [28558.248907] wmi ipmi_si [28558.256031] ? watchdog_timer_fn+0x1b4/0x220 [28558.259168] ipmi_devintf ipmi_msghandler [28558.261612] ? __hrtimer_run_queues+0x1d3/0x360 [28558.263622] ioatdma acpi_pad [28558.266155] ? hrtimer_interrupt+0x104/0x240 [28558.270417] acpi_power_meter ixgbe [28558.274421] ? __local_bh_enable+0x50/0x60 [28558.278944] igb mdio [28558.281912] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28558.286172] crc32c_intel nvme [28558.289658] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28558.293747] ptp nvme_core [28558.296018] [28558.301246] i2c_algo_bit [28558.304296] [28558.309254] pps_core [28558.311958] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28558.314055] dca fuse [28558.316675] ? queued_spin_lock_slowpath+0x18/0x2e0 [28558.318770] [28558.318771] irq event stamp: 150115098 [28558.321041] do_raw_spin_lock+0xb5/0xc0 [28558.326343] hardirqs last enabled at (150115097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28558.328617] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28558.333486] hardirqs last disabled at (150115098): [] sysvec_apic_timer_interrupt+0xe/0x90 [28558.334979] try_get_cap_refs+0x159/0x880 [ceph] [28558.338721] softirqs last enabled at (150115096): [] __do_softirq+0x2be/0x371 [28558.342556] __ceph_get_caps+0x4a1/0x760 [ceph] [28558.352796] softirqs last disabled at (150115091): [] irq_exit_rcu+0xb0/0xd0 [28558.357584] ? __file_remove_privs+0x89/0x110 [28558.367390] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28558.372002] ? wake_bit_function+0x70/0x70 [28558.380771] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28558.385299] ? ceph_write_iter+0x260/0x770 [ceph] [28558.393894] Workqueue: events delayed_work [ceph] [28558.398249] ceph_write_iter+0x260/0x770 [ceph] [28558.407629] [28558.407630] RIP: 0010:lock_acquire+0xf0/0x2b0 [28558.411725] ? lock_is_held_type+0xaa/0x120 [28558.419192] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28558.423891] vfs_write+0x324/0x4f0 [28558.428587] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28558.433115] ksys_pwrite64+0x65/0xa0 [28558.434601] [28558.434602] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28558.438952] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28558.443129] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28558.461868] do_syscall_64+0x38/0x80 [28558.465264] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28558.470482] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28558.474053] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28558.475543] RIP: 0033:0x7f48f0eeb93b [28558.482668] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28558.487799] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28558.494921] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28558.498494] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28558.505616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28558.510662] ORIG_RAX: 0000000000000012 [28558.517784] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28558.521355] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28558.528482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28558.547219] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28558.555296] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28558.560530] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28558.566268] Call Trace: [28558.570099] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28558.577224] [28558.584346] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28558.591471] ? watchdog_timer_fn+0x1b4/0x220 [28558.598600] [28558.605720] ? __hrtimer_run_queues+0x1d3/0x360 [28558.642536] ? hrtimer_interrupt+0x104/0x240 [28558.646801] ? __local_bh_enable+0x50/0x60 [28558.650894] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28558.656116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28558.661076] [28558.663173] [28558.665270] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28558.670577] ? lock_acquire+0xf0/0x2b0 [28558.674329] ? igrab+0x43/0x50 [28558.677377] ? lock_release+0xc5/0x260 [28558.681122] _raw_spin_lock+0x30/0x40 [28558.684778] ? igrab+0x1d/0x50 [28558.687829] igrab+0x1d/0x50 [28558.690708] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28558.695858] delayed_work+0x1d8/0x280 [ceph] [28558.700139] ? process_scheduled_works+0x22a/0x510 [28558.704929] process_scheduled_works+0x2a4/0x510 [28558.709549] worker_thread+0x188/0x340 [28558.713293] ? rescuer_thread+0x310/0x310 [28558.717296] kthread+0xf5/0x130 [28558.720434] ? kthread_complete_and_exit+0x20/0x20 [28558.725219] ret_from_fork+0x31/0x40 [28558.728788] ? kthread_complete_and_exit+0x20/0x20 [28558.733573] ret_from_fork_asm+0x11/0x20 [28558.737496] [28585.958360] watchdog: BUG: soft lockup - CPU#0 stuck for 15508s! [dbench:78683] [28585.965660] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28585.970360] watchdog: BUG: soft lockup - CPU#7 stuck for 24505s! [kworker/7:4:78808] [28586.021100] irq event stamp: 47455968 [28586.021102] hardirqs last enabled at (47455967): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28586.028831] Modules linked in: [28586.032489] hardirqs last disabled at (47455968): [] sysvec_apic_timer_interrupt+0xe/0x90 [28586.042646] ceph [28586.045696] softirqs last enabled at (47455962): [] __do_softirq+0x2be/0x371 [28586.055421] libceph [28586.057344] softirqs last disabled at (47455957): [] irq_exit_rcu+0xb0/0xd0 [28586.066036] dns_resolver fscache [28586.068223] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28586.076732] netfs veth [28586.080061] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28586.089004] nft_chain_nat [28586.091448] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28586.098919] xt_MASQUERADE nf_nat [28586.101624] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28586.107101] nf_conntrack nf_defrag_ipv6 [28586.110413] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28586.129150] nf_defrag_ipv4 nft_compat [28586.133067] [28586.133068] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28586.138285] nf_tables [28586.142037] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28586.143528] nfnetlink bridge [28586.150651] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28586.153008] stp llc [28586.160135] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28586.163097] xfs libcrc32c [28586.170223] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28586.172406] sunrpc intel_rapl_msr [28586.179530] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28586.182234] intel_rapl_common [28586.189357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28586.192756] sb_edac x86_pkg_temp_thermal [28586.200833] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28586.203883] coretemp kvm_intel [28586.209620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28586.213626] kvm irqbypass [28586.220758] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28586.223894] i2c_i801 iTCO_wdt [28586.231019] Call Trace: [28586.231020] [28586.233757] iTCO_vendor_support i2c_smbus [28586.240884] ? watchdog_timer_fn+0x1b4/0x220 [28586.243932] crct10dif_pclmul [28586.246378] ? __hrtimer_run_queues+0x1d3/0x360 [28586.248388] joydev crc32_pclmul [28586.252481] ? hrtimer_interrupt+0x104/0x240 [28586.256757] ipmi_ssif [28586.259724] ? __local_bh_enable+0x50/0x60 [28586.264249] ghash_clmulni_intel mei_me [28586.267474] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28586.271736] mei lpc_ich [28586.274113] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28586.278202] sha512_ssse3 acpi_ipmi [28586.282049] [28586.287268] mfd_core wmi [28586.289798] [28586.289800] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28586.294759] ipmi_si ipmi_devintf [28586.298243] ? queued_spin_lock_slowpath+0x18/0x2e0 [28586.300338] ipmi_msghandler ioatdma [28586.302955] do_raw_spin_lock+0xb5/0xc0 [28586.305055] acpi_pad acpi_power_meter [28586.310356] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28586.313666] ixgbe igb [28586.318539] try_get_cap_refs+0x159/0x880 [ceph] [28586.322109] mdio crc32c_intel [28586.325944] __ceph_get_caps+0x4a1/0x760 [ceph] [28586.329683] nvme ptp [28586.334470] ? __file_remove_privs+0x89/0x110 [28586.336825] nvme_core [28586.341435] ? wake_bit_function+0x70/0x70 [28586.344487] i2c_algo_bit pps_core [28586.349012] ? ceph_write_iter+0x260/0x770 [ceph] [28586.351281] dca fuse [28586.355632] ceph_write_iter+0x260/0x770 [ceph] [28586.357988] [28586.357989] irq event stamp: 150336032 [28586.362082] ? lock_is_held_type+0xaa/0x120 [28586.365477] hardirqs last enabled at (150336031): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28586.370176] vfs_write+0x324/0x4f0 [28586.372446] hardirqs last disabled at (150336032): [] sysvec_apic_timer_interrupt+0xe/0x90 [28586.376974] ksys_pwrite64+0x65/0xa0 [28586.378459] softirqs last enabled at (150336030): [] __do_softirq+0x2be/0x371 [28586.382204] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28586.386381] softirqs last disabled at (150336025): [] irq_exit_rcu+0xb0/0xd0 [28586.396626] do_syscall_64+0x38/0x80 [28586.400023] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28586.409833] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28586.413405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28586.422174] RIP: 0033:0x7f48f0eeb93b [28586.427305] Workqueue: events delayed_work [ceph] [28586.435902] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28586.439474] [28586.448850] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28586.453895] RIP: 0010:ceph_check_delayed_caps+0x93/0x270 [ceph] [28586.461367] ORIG_RAX: 0000000000000012 [28586.464938] Code: 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e <48> 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 [28586.469634] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28586.488371] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [28586.489863] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28586.495079] [28586.495080] RAX: 0000000000000001 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [28586.500991] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28586.504821] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [28586.523559] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28586.530684] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [28586.535901] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28586.543041] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28586.544538] [28586.551657] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28586.603719] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28586.611796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28586.617534] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28586.624657] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28586.631784] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28586.638906] Call Trace: [28586.641352] [28586.643362] ? watchdog_timer_fn+0x1b4/0x220 [28586.647627] ? __hrtimer_run_queues+0x1d3/0x360 [28586.652152] ? hrtimer_interrupt+0x104/0x240 [28586.656423] ? __local_bh_enable+0x50/0x60 [28586.660514] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28586.665730] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28586.670689] [28586.672785] [28586.674883] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28586.680191] ? ceph_check_delayed_caps+0x93/0x270 [ceph] [28586.685510] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [28586.690823] delayed_work+0x1d8/0x280 [ceph] [28586.695105] ? process_scheduled_works+0x22a/0x510 [28586.699896] process_scheduled_works+0x2a4/0x510 [28586.704511] worker_thread+0x188/0x340 [28586.708260] ? rescuer_thread+0x310/0x310 [28586.712263] kthread+0xf5/0x130 [28586.715400] ? kthread_complete_and_exit+0x20/0x20 [28586.720184] ret_from_fork+0x31/0x40 [28586.723767] ? kthread_complete_and_exit+0x20/0x20 [28586.728556] ret_from_fork_asm+0x11/0x20 [28586.732480] [28613.428225] rcu: INFO: rcu_preempt self-detected stall on CPU [28613.433989] rcu: 0-....: (16302001 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=8152959 [28613.444412] rcu: (t=16676523 jiffies g=1309609 q=504279 ncpus=8) [28613.450497] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28613.459441] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28613.466911] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28613.472397] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28613.491135] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28613.496351] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28613.503477] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28613.510600] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28613.517726] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28613.524848] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28613.531974] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28613.540051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28613.545789] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28613.552912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28613.560037] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28613.567161] Call Trace: [28613.569607] [28613.571618] ? rcu_dump_cpu_stacks+0xd1/0x180 [28613.575975] ? rcu_sched_clock_irq+0x633/0x1390 [28613.580499] ? lock_is_held_type+0xaa/0x120 [28613.584677] ? lock_is_held_type+0xaa/0x120 [28613.588855] ? tick_sched_do_timer+0xa0/0xa0 [28613.593126] ? rcu_read_lock_sched_held+0x56/0x90 [28613.597825] ? tick_sched_do_timer+0xa0/0xa0 [28613.602088] ? update_process_times+0x5f/0x90 [28613.606440] ? tick_sched_handle.isra.16+0x1e/0x50 [28613.611224] ? tick_sched_timer+0x6f/0x80 [28613.615229] ? __hrtimer_run_queues+0x1d3/0x360 [28613.619764] ? hrtimer_interrupt+0x104/0x240 [28613.624032] ? __local_bh_enable+0x50/0x60 [28613.628126] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28613.633351] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28613.638308] [28613.640405] [28613.642503] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28613.647819] ? queued_spin_lock_slowpath+0x18/0x2e0 [28613.652695] do_raw_spin_lock+0xb5/0xc0 [28613.656536] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28613.661346] try_get_cap_refs+0x159/0x880 [ceph] [28613.665976] __ceph_get_caps+0x4a1/0x760 [ceph] [28613.670518] ? __file_remove_privs+0x89/0x110 [28613.674873] ? wake_bit_function+0x70/0x70 [28613.678975] ? ceph_write_iter+0x260/0x770 [ceph] [28613.683686] ceph_write_iter+0x260/0x770 [ceph] [28613.688222] ? lock_is_held_type+0xaa/0x120 [28613.692407] vfs_write+0x324/0x4f0 [28613.695815] ksys_pwrite64+0x65/0xa0 [28613.699392] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28613.704531] do_syscall_64+0x38/0x80 [28613.708112] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28613.713162] RIP: 0033:0x7f48f0eeb93b [28613.716732] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28613.735470] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [28613.743028] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28613.750151] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28613.757275] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28613.764401] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28613.771523] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28613.778654] [28613.780841] Sending NMI from CPU 0 to CPUs 7: [28613.785192] NMI backtrace for cpu 7 [28613.785194] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28613.785196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28613.785197] Workqueue: events delayed_work [ceph] [28613.785215] RIP: 0010:mark_lock+0xae/0x4c0 [28613.785217] Code: 24 68 65 48 33 0c 25 28 00 00 00 44 89 e0 0f 85 1e 04 00 00 48 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f b6 46 22 83 e0 03 <3c> 01 4d 19 f6 45 30 f6 49 81 c6 00 02 00 00 84 c0 b8 09 00 00 00 [28613.785218] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000046 [28613.785219] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 0000000000000000 [28613.785220] RDX: 0000000000000008 RSI: ffff88810debb550 RDI: ffff88810debaa80 [28613.785221] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [28613.785222] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [28613.785222] R13: ffff88810debb550 R14: 0000000000000000 R15: ffff88810debaa80 [28613.785223] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28613.785224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28613.785225] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28613.785226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28613.785226] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28613.785227] Call Trace: [28613.785228] [28613.785229] ? nmi_cpu_backtrace+0xad/0x120 [28613.785234] ? nmi_cpu_backtrace_handler+0x11/0x20 [28613.785236] ? nmi_handle+0xd4/0x260 [28613.785240] ? default_do_nmi+0x49/0xf0 [28613.785242] ? exc_nmi+0x11e/0x150 [28613.785243] ? end_repeat_nmi+0x16/0x67 [28613.785247] ? mark_lock+0xae/0x4c0 [28613.785248] ? mark_lock+0xae/0x4c0 [28613.785250] ? mark_lock+0xae/0x4c0 [28613.785251] [28613.785251] [28613.785252] ? _raw_spin_unlock+0x23/0x40 [28613.785254] ? ceph_check_caps+0x275/0xae0 [ceph] [28613.785268] ? mark_held_locks+0x48/0x70 [28613.785270] __lock_acquire+0x4d6/0x1e20 [28613.785273] lock_acquire+0xcb/0x2b0 [28613.785274] ? igrab+0x1d/0x50 [28613.785276] ? igrab+0x43/0x50 [28613.785277] ? lock_release+0xc5/0x260 [28613.785279] _raw_spin_lock+0x30/0x40 [28613.785281] ? igrab+0x1d/0x50 [28613.785282] igrab+0x1d/0x50 [28613.785283] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28613.785296] delayed_work+0x1d8/0x280 [ceph] [28613.785309] ? process_scheduled_works+0x22a/0x510 [28613.785312] process_scheduled_works+0x2a4/0x510 [28613.785316] worker_thread+0x188/0x340 [28613.785318] ? rescuer_thread+0x310/0x310 [28613.785320] kthread+0xf5/0x130 [28613.785323] ? kthread_complete_and_exit+0x20/0x20 [28613.785326] ret_from_fork+0x31/0x40 [28613.785328] ? kthread_complete_and_exit+0x20/0x20 [28613.785330] ret_from_fork_asm+0x11/0x20 [28613.785335] [28637.959303] watchdog: BUG: soft lockup - CPU#0 stuck for 15557s! [dbench:78683] [28637.966607] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28637.971303] watchdog: BUG: soft lockup - CPU#7 stuck for 24554s! [kworker/7:4:78808] [28638.022058] irq event stamp: 47564452 [28638.029787] Modules linked in: ceph libceph dns_resolver fscache [28638.033445] hardirqs last enabled at (47564451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28638.039444] netfs veth nft_chain_nat [28638.049602] hardirqs last disabled at (47564452): [] sysvec_apic_timer_interrupt+0xe/0x90 [28638.053257] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [28638.062987] softirqs last enabled at (47564374): [] __do_softirq+0x2be/0x371 [28638.068729] nf_defrag_ipv4 nft_compat nf_tables [28638.077412] softirqs last disabled at (47564369): [] irq_exit_rcu+0xb0/0xd0 [28638.082023] nfnetlink bridge stp [28638.090535] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28638.093845] llc xfs [28638.102789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28638.104981] libcrc32c sunrpc [28638.112453] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28638.115417] intel_rapl_msr intel_rapl_common [28638.120895] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28638.125246] sb_edac x86_pkg_temp_thermal [28638.143982] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28638.147986] coretemp kvm_intel [28638.153203] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28638.153204] kvm irqbypass [28638.156342] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28638.163466] i2c_i801 iTCO_wdt [28638.166169] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28638.173294] iTCO_vendor_support [28638.176345] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28638.183469] i2c_smbus crct10dif_pclmul [28638.186702] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28638.193826] joydev crc32_pclmul [28638.197657] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28638.204780] ipmi_ssif ghash_clmulni_intel [28638.208005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28638.216082] mei_me mei [28638.220172] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28638.225910] lpc_ich sha512_ssse3 [28638.228353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28638.235479] acpi_ipmi mfd_core [28638.238788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28638.245912] wmi ipmi_si [28638.249051] Call Trace: [28638.249053] [28638.256177] ipmi_devintf [28638.258706] ? watchdog_timer_fn+0x1b4/0x220 [28638.261150] ipmi_msghandler ioatdma [28638.263161] ? __hrtimer_run_queues+0x1d3/0x360 [28638.265777] acpi_pad acpi_power_meter [28638.270044] ? hrtimer_interrupt+0x104/0x240 [28638.273611] ixgbe igb [28638.278136] ? __local_bh_enable+0x50/0x60 [28638.281882] mdio crc32c_intel [28638.286146] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28638.288501] nvme [28638.292593] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28638.295644] ptp nvme_core [28638.300860] [28638.302785] i2c_algo_bit [28638.307743] [28638.307744] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28638.310446] pps_core dca [28638.312549] ? queued_spin_lock_slowpath+0x16/0x2e0 [28638.315161] fuse [28638.317261] do_raw_spin_lock+0xb5/0xc0 [28638.322562] irq event stamp: 150749546 [28638.325180] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28638.330050] hardirqs last enabled at (150749545): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28638.331986] try_get_cap_refs+0x159/0x880 [ceph] [28638.335815] hardirqs last disabled at (150749546): [] sysvec_apic_timer_interrupt+0xe/0x90 [28638.339563] __ceph_get_caps+0x4a1/0x760 [ceph] [28638.344343] softirqs last enabled at (150749544): [] __do_softirq+0x2be/0x371 [28638.354590] ? __file_remove_privs+0x89/0x110 [28638.359198] softirqs last disabled at (150749539): [] irq_exit_rcu+0xb0/0xd0 [28638.369010] ? wake_bit_function+0x70/0x70 [28638.373533] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28638.382306] ? ceph_write_iter+0x260/0x770 [ceph] [28638.386655] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28638.395252] ceph_write_iter+0x260/0x770 [ceph] [28638.399343] Workqueue: events delayed_work [ceph] [28638.408722] ? lock_is_held_type+0xaa/0x120 [28638.413416] [28638.413417] RIP: 0010:lock_acquire+0xf0/0x2b0 [28638.420891] vfs_write+0x324/0x4f0 [28638.425411] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28638.430113] ksys_pwrite64+0x65/0xa0 [28638.434287] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28638.435776] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28638.440128] [28638.440129] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28638.443528] do_syscall_64+0x38/0x80 [28638.462263] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28638.465834] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28638.471052] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28638.476182] RIP: 0033:0x7f48f0eeb93b [28638.477672] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28638.484796] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28638.488367] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28638.495491] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28638.500537] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28638.507659] ORIG_RAX: 0000000000000012 [28638.511230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28638.518355] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28638.537092] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28638.544217] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28638.549434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28638.557511] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28638.561341] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28638.567080] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28638.574205] Call Trace: [28638.574206] [28638.581327] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28638.588453] ? watchdog_timer_fn+0x1b4/0x220 [28638.595582] [28638.602709] ? __hrtimer_run_queues+0x1d3/0x360 [28638.639511] ? hrtimer_interrupt+0x104/0x240 [28638.643773] ? __local_bh_enable+0x50/0x60 [28638.647867] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28638.653091] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28638.658047] [28638.660144] [28638.662241] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28638.667550] ? lock_acquire+0xf0/0x2b0 [28638.671301] ? igrab+0x43/0x50 [28638.674349] ? lock_release+0xc5/0x260 [28638.678097] _raw_spin_lock+0x30/0x40 [28638.681759] ? igrab+0x1d/0x50 [28638.684810] igrab+0x1d/0x50 [28638.687706] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28638.692856] delayed_work+0x1d8/0x280 [ceph] [28638.697136] ? process_scheduled_works+0x22a/0x510 [28638.701927] process_scheduled_works+0x2a4/0x510 [28638.706543] worker_thread+0x188/0x340 [28638.710291] ? rescuer_thread+0x310/0x310 [28638.714295] kthread+0xf5/0x130 [28638.717432] ? kthread_complete_and_exit+0x20/0x20 [28638.722219] ret_from_fork+0x31/0x40 [28638.725797] ? kthread_complete_and_exit+0x20/0x20 [28638.730589] ret_from_fork_asm+0x11/0x20 [28638.734513] [28665.959811] watchdog: BUG: soft lockup - CPU#0 stuck for 15583s! [dbench:78683] [28665.967114] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28665.971811] watchdog: BUG: soft lockup - CPU#7 stuck for 24580s! [kworker/7:4:78808] [28666.022563] irq event stamp: 47622550 [28666.030295] Modules linked in: ceph libceph dns_resolver fscache netfs [28666.033951] hardirqs last enabled at (47622549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28666.040486] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28666.050642] hardirqs last disabled at (47622550): [] sysvec_apic_timer_interrupt+0xe/0x90 [28666.056726] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [28666.066450] softirqs last enabled at (47622534): [] __do_softirq+0x2be/0x371 [28666.074182] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [28666.082867] softirqs last disabled at (47622529): [] irq_exit_rcu+0xb0/0xd0 [28666.089383] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [28666.097897] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28666.103893] irqbypass i2c_i801 iTCO_wdt [28666.112837] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28666.112839] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28666.116753] iTCO_vendor_support i2c_smbus [28666.124226] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28666.129701] crct10dif_pclmul joydev [28666.133792] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28666.152531] crc32_pclmul ipmi_ssif [28666.156103] [28666.161317] ghash_clmulni_intel mei_me [28666.164802] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28666.166294] mei lpc_ich [28666.170125] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28666.177249] sha512_ssse3 [28666.179778] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28666.186902] acpi_ipmi mfd_core [28666.189519] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28666.196646] wmi ipmi_si [28666.199782] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28666.206907] ipmi_devintf [28666.209436] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28666.216561] ipmi_msghandler [28666.219179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28666.227256] ioatdma [28666.230134] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28666.235872] acpi_pad acpi_power_meter [28666.238055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28666.245178] ixgbe igb [28666.248923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28666.256048] mdio crc32c_intel [28666.258403] Call Trace: [28666.258406] [28666.265529] nvme ptp [28666.268581] ? watchdog_timer_fn+0x1b4/0x220 [28666.271023] nvme_core i2c_algo_bit [28666.273035] ? __hrtimer_run_queues+0x1d3/0x360 [28666.275305] pps_core dca [28666.279573] ? hrtimer_interrupt+0x104/0x240 [28666.283053] fuse [28666.287578] ? __local_bh_enable+0x50/0x60 [28666.290209] [28666.290210] irq event stamp: 150970544 [28666.294494] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28666.296416] hardirqs last enabled at (150970543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28666.300509] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28666.301999] hardirqs last disabled at (150970544): [] sysvec_apic_timer_interrupt+0xe/0x90 [28666.305743] [28666.310959] softirqs last enabled at (150970542): [] __do_softirq+0x2be/0x371 [28666.321213] [28666.326169] softirqs last disabled at (150970537): [] irq_exit_rcu+0xb0/0xd0 [28666.335983] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28666.338079] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28666.346854] ? queued_spin_lock_slowpath+0x18/0x2e0 [28666.348947] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28666.357546] do_raw_spin_lock+0xb5/0xc0 [28666.362849] Workqueue: events delayed_work [ceph] [28666.372227] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28666.377097] [28666.377098] RIP: 0010:lock_acquire+0xf0/0x2b0 [28666.384570] try_get_cap_refs+0x159/0x880 [ceph] [28666.388398] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28666.393100] __ceph_get_caps+0x4a1/0x760 [ceph] [28666.397880] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28666.399375] ? __file_remove_privs+0x89/0x110 [28666.403722] [28666.403722] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28666.408333] ? wake_bit_function+0x70/0x70 [28666.427068] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28666.431596] ? ceph_write_iter+0x260/0x770 [ceph] [28666.436811] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28666.441161] ceph_write_iter+0x260/0x770 [ceph] [28666.442652] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28666.449781] ? lock_is_held_type+0xaa/0x120 [28666.453868] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28666.460995] vfs_write+0x324/0x4f0 [28666.465688] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28666.472818] ksys_pwrite64+0x65/0xa0 [28666.477338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28666.484463] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28666.488638] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28666.495763] do_syscall_64+0x38/0x80 [28666.499160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28666.507239] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28666.510809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28666.516547] RIP: 0033:0x7f48f0eeb93b [28666.521676] Call Trace: [28666.521678] [28666.528801] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28666.532371] ? watchdog_timer_fn+0x1b4/0x220 [28666.539504] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28666.544551] ? __hrtimer_run_queues+0x1d3/0x360 [28666.551671] ORIG_RAX: 0000000000000012 [28666.555247] ? hrtimer_interrupt+0x104/0x240 [28666.557686] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28666.559697] ? __local_bh_enable+0x50/0x60 [28666.578434] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28666.582702] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28666.587917] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28666.592441] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28666.596272] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28666.600535] [28666.607660] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28666.611751] [28666.611752] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28666.618881] [28666.624096] ? lock_acquire+0xf0/0x2b0 [28666.665851] ? igrab+0x43/0x50 [28666.668901] ? lock_release+0xc5/0x260 [28666.672646] _raw_spin_lock+0x30/0x40 [28666.676303] ? igrab+0x1d/0x50 [28666.679352] igrab+0x1d/0x50 [28666.682232] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28666.687380] delayed_work+0x1d8/0x280 [ceph] [28666.691662] ? process_scheduled_works+0x22a/0x510 [28666.696452] process_scheduled_works+0x2a4/0x510 [28666.701069] worker_thread+0x188/0x340 [28666.704817] ? rescuer_thread+0x310/0x310 [28666.708821] kthread+0xf5/0x130 [28666.711967] ? kthread_complete_and_exit+0x20/0x20 [28666.716752] ret_from_fork+0x31/0x40 [28666.720330] ? kthread_complete_and_exit+0x20/0x20 [28666.725115] ret_from_fork_asm+0x11/0x20 [28666.729047] [28693.960319] watchdog: BUG: soft lockup - CPU#0 stuck for 15609s! [dbench:78683] [28693.967619] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28693.972319] watchdog: BUG: soft lockup - CPU#7 stuck for 24606s! [kworker/7:4:78808] [28694.023058] irq event stamp: 47680630 [28694.023059] hardirqs last enabled at (47680629): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28694.030791] Modules linked in: ceph [28694.034447] hardirqs last disabled at (47680630): [] sysvec_apic_timer_interrupt+0xe/0x90 [28694.044606] libceph [28694.048088] softirqs last enabled at (47680572): [] __do_softirq+0x2be/0x371 [28694.057813] dns_resolver fscache [28694.059998] softirqs last disabled at (47680567): [] irq_exit_rcu+0xb0/0xd0 [28694.068681] netfs veth [28694.072008] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28694.080520] nft_chain_nat xt_MASQUERADE [28694.082965] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28694.091907] nf_nat nf_conntrack [28694.095826] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28694.103296] nf_defrag_ipv6 [28694.106519] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28694.112013] nf_defrag_ipv4 nft_compat [28694.114805] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28694.133545] nf_tables [28694.137289] [28694.137290] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28694.142506] nfnetlink bridge [28694.144864] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28694.146353] stp llc [28694.153478] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28694.156442] xfs libcrc32c [28694.163564] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28694.165750] sunrpc [28694.172874] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28694.175578] intel_rapl_msr intel_rapl_common [28694.182715] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28694.184817] sb_edac x86_pkg_temp_thermal [28694.191940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28694.196291] coretemp kvm_intel [28694.204368] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28694.208373] kvm irqbypass [28694.214112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28694.217249] i2c_i801 iTCO_wdt [28694.224371] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28694.227076] iTCO_vendor_support i2c_smbus [28694.234200] Call Trace: [28694.234201] [28694.237252] crct10dif_pclmul joydev [28694.244376] ? watchdog_timer_fn+0x1b4/0x220 [28694.248465] crc32_pclmul ipmi_ssif [28694.250912] ? __hrtimer_run_queues+0x1d3/0x360 [28694.252920] ghash_clmulni_intel mei_me [28694.256494] ? hrtimer_interrupt+0x104/0x240 [28694.260755] mei lpc_ich [28694.264240] ? __local_bh_enable+0x50/0x60 [28694.268764] sha512_ssse3 [28694.272597] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28694.276858] acpi_ipmi mfd_core [28694.279389] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28694.283479] wmi ipmi_si [28694.286096] [28694.291313] ipmi_devintf ipmi_msghandler [28694.294453] [28694.294454] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28694.299409] ioatdma acpi_pad [28694.301943] ? queued_spin_lock_slowpath+0x18/0x2e0 [28694.304037] acpi_power_meter ixgbe [28694.308043] do_raw_spin_lock+0xb5/0xc0 [28694.310140] igb mdio [28694.315444] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28694.318408] crc32c_intel nvme [28694.323281] try_get_cap_refs+0x159/0x880 [ceph] [28694.326761] ptp nvme_core [28694.330596] __ceph_get_caps+0x4a1/0x760 [ceph] [28694.332863] i2c_algo_bit pps_core [28694.337651] ? __file_remove_privs+0x89/0x110 [28694.340715] dca fuse [28694.345328] ? wake_bit_function+0x70/0x70 [28694.348031] [28694.348032] irq event stamp: 151191510 [28694.352556] ? ceph_write_iter+0x260/0x770 [ceph] [28694.355953] hardirqs last enabled at (151191509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28694.360302] ceph_write_iter+0x260/0x770 [ceph] [28694.362574] hardirqs last disabled at (151191510): [] sysvec_apic_timer_interrupt+0xe/0x90 [28694.366668] ? lock_is_held_type+0xaa/0x120 [28694.368155] softirqs last enabled at (151191508): [] __do_softirq+0x2be/0x371 [28694.371901] vfs_write+0x324/0x4f0 [28694.376595] softirqs last disabled at (151191503): [] irq_exit_rcu+0xb0/0xd0 [28694.386846] ksys_pwrite64+0x65/0xa0 [28694.391364] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28694.401174] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28694.405353] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28694.414124] do_syscall_64+0x38/0x80 [28694.417522] Workqueue: events delayed_work [ceph] [28694.426118] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28694.429689] [28694.429690] RIP: 0010:lock_acquire+0xf0/0x2b0 [28694.439066] RIP: 0033:0x7f48f0eeb93b [28694.444199] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28694.451667] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28694.455238] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28694.459936] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28694.464981] [28694.464981] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28694.466488] ORIG_RAX: 0000000000000012 [28694.470840] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28694.474409] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28694.493148] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28694.511886] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28694.517103] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28694.522321] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28694.523812] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28694.523813] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28694.530934] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28694.534765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28694.541889] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28694.549014] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28694.556145] [28694.563263] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28694.629157] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28694.636279] Call Trace: [28694.638725] [28694.640737] ? watchdog_timer_fn+0x1b4/0x220 [28694.645013] ? __hrtimer_run_queues+0x1d3/0x360 [28694.649545] ? hrtimer_interrupt+0x104/0x240 [28694.653813] ? __local_bh_enable+0x50/0x60 [28694.657907] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28694.663130] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28694.668089] [28694.670185] [28694.672283] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28694.677592] ? lock_acquire+0xf0/0x2b0 [28694.681340] ? igrab+0x43/0x50 [28694.684390] ? lock_release+0xc5/0x260 [28694.688136] _raw_spin_lock+0x30/0x40 [28694.691791] ? igrab+0x1d/0x50 [28694.694843] igrab+0x1d/0x50 [28694.697727] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28694.702871] delayed_work+0x1d8/0x280 [ceph] [28694.707150] ? process_scheduled_works+0x22a/0x510 [28694.711934] process_scheduled_works+0x2a4/0x510 [28694.716548] worker_thread+0x188/0x340 [28694.720298] ? rescuer_thread+0x310/0x310 [28694.724301] kthread+0xf5/0x130 [28694.727439] ? kthread_complete_and_exit+0x20/0x20 [28694.732224] ret_from_fork+0x31/0x40 [28694.735792] ? kthread_complete_and_exit+0x20/0x20 [28694.740578] ret_from_fork_asm+0x11/0x20 [28694.744502] [28721.960828] watchdog: BUG: soft lockup - CPU#0 stuck for 15635s! [dbench:78683] [28721.968133] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28721.972828] watchdog: BUG: soft lockup - CPU#7 stuck for 24632s! [kworker/7:4:78808] [28722.023581] irq event stamp: 47738868 [28722.031313] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE [28722.034969] hardirqs last enabled at (47738867): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28722.044356] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [28722.054512] hardirqs last disabled at (47738868): [] sysvec_apic_timer_interrupt+0xe/0x90 [28722.054515] nf_tables [28722.054516] softirqs last enabled at (47738810): [] __do_softirq+0x2be/0x371 [28722.061290] nfnetlink [28722.071015] softirqs last disabled at (47738805): [] irq_exit_rcu+0xb0/0xd0 [28722.073372] bridge [28722.082056] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28722.084415] stp llc [28722.092925] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28722.095021] xfs libcrc32c [28722.103965] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28722.106149] sunrpc [28722.113621] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28722.116324] intel_rapl_msr intel_rapl_common [28722.121803] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28722.123899] sb_edac x86_pkg_temp_thermal [28722.142639] [28722.146988] coretemp kvm_intel [28722.152222] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28722.156228] kvm irqbypass [28722.157717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28722.160855] i2c_i801 iTCO_wdt [28722.167980] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28722.170683] iTCO_vendor_support i2c_smbus [28722.177806] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28722.180857] crct10dif_pclmul [28722.187983] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28722.192073] joydev [28722.199197] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28722.202160] crc32_pclmul [28722.209286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28722.211384] ipmi_ssif ghash_clmulni_intel [28722.219459] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28722.222077] mei_me mei [28722.227816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28722.231908] lpc_ich [28722.239031] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28722.241474] sha512_ssse3 acpi_ipmi [28722.248597] Call Trace: [28722.248600] [28722.250783] mfd_core wmi [28722.257908] ? watchdog_timer_fn+0x1b4/0x220 [28722.261391] ipmi_si ipmi_devintf [28722.263836] ? __hrtimer_run_queues+0x1d3/0x360 [28722.265846] ipmi_msghandler ioatdma [28722.268466] ? hrtimer_interrupt+0x104/0x240 [28722.272726] acpi_pad acpi_power_meter [28722.276036] ? __local_bh_enable+0x50/0x60 [28722.280562] ixgbe igb [28722.284135] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28722.288396] mdio crc32c_intel [28722.292142] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28722.296232] nvme ptp [28722.298589] [28722.303805] nvme_core i2c_algo_bit [28722.306856] [28722.311815] pps_core dca [28722.314086] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28722.316183] fuse [28722.319670] ? queued_spin_lock_slowpath+0x16/0x2e0 [28722.321764] [28722.321765] irq event stamp: 151412430 [28722.324382] do_raw_spin_lock+0xb5/0xc0 [28722.329685] hardirqs last enabled at (151412429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28722.331613] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28722.336480] hardirqs last disabled at (151412430): [] sysvec_apic_timer_interrupt+0xe/0x90 [28722.337991] try_get_cap_refs+0x159/0x880 [ceph] [28722.341733] softirqs last enabled at (151412428): [] __do_softirq+0x2be/0x371 [28722.345569] __ceph_get_caps+0x4a1/0x760 [ceph] [28722.355806] softirqs last disabled at (151412423): [] irq_exit_rcu+0xb0/0xd0 [28722.360596] ? __file_remove_privs+0x89/0x110 [28722.370401] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28722.375014] ? wake_bit_function+0x70/0x70 [28722.383783] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28722.388310] ? ceph_write_iter+0x260/0x770 [ceph] [28722.396905] Workqueue: events delayed_work [ceph] [28722.401256] ceph_write_iter+0x260/0x770 [ceph] [28722.410633] [28722.410634] RIP: 0010:lock_acquire+0xf0/0x2b0 [28722.414730] ? lock_is_held_type+0xaa/0x120 [28722.422196] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28722.426895] vfs_write+0x324/0x4f0 [28722.431590] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28722.436118] ksys_pwrite64+0x65/0xa0 [28722.437604] [28722.437604] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28722.441956] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28722.446132] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28722.464872] do_syscall_64+0x38/0x80 [28722.468266] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28722.473488] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28722.477072] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28722.478564] RIP: 0033:0x7f48f0eeb93b [28722.485688] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28722.490821] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28722.497943] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28722.501514] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28722.508636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28722.513682] ORIG_RAX: 0000000000000012 [28722.520806] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28722.524378] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28722.531501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28722.550256] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28722.558334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28722.563550] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28722.569289] Call Trace: [28722.569290] [28722.573118] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28722.580245] ? watchdog_timer_fn+0x1b4/0x220 [28722.587366] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28722.594508] ? __hrtimer_run_queues+0x1d3/0x360 [28722.601639] [28722.608760] ? hrtimer_interrupt+0x104/0x240 [28722.649821] ? __local_bh_enable+0x50/0x60 [28722.653915] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28722.659140] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28722.664094] [28722.666192] [28722.668289] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28722.673597] ? lock_acquire+0xf0/0x2b0 [28722.677348] ? igrab+0x43/0x50 [28722.680399] ? lock_release+0xc5/0x260 [28722.684144] _raw_spin_lock+0x30/0x40 [28722.687809] ? igrab+0x1d/0x50 [28722.690859] igrab+0x1d/0x50 [28722.693739] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28722.698888] delayed_work+0x1d8/0x280 [ceph] [28722.703168] ? process_scheduled_works+0x22a/0x510 [28722.707958] process_scheduled_works+0x2a4/0x510 [28722.712574] worker_thread+0x188/0x340 [28722.716323] ? rescuer_thread+0x310/0x310 [28722.720326] kthread+0xf5/0x130 [28722.723463] ? kthread_complete_and_exit+0x20/0x20 [28722.728248] ret_from_fork+0x31/0x40 [28722.731827] ? kthread_complete_and_exit+0x20/0x20 [28722.736611] ret_from_fork_asm+0x11/0x20 [28722.740543] [28749.961342] watchdog: BUG: soft lockup - CPU#0 stuck for 15661s! [dbench:78683] [28749.968645] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28749.973342] watchdog: BUG: soft lockup - CPU#7 stuck for 24658s! [kworker/7:4:78808] [28750.024087] irq event stamp: 47796962 [28750.024088] hardirqs last enabled at (47796961): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28750.031818] Modules linked in: [28750.035475] hardirqs last disabled at (47796962): [] sysvec_apic_timer_interrupt+0xe/0x90 [28750.045633] ceph libceph [28750.048682] softirqs last enabled at (47796956): [] __do_softirq+0x2be/0x371 [28750.058406] dns_resolver [28750.061028] softirqs last disabled at (47796951): [] irq_exit_rcu+0xb0/0xd0 [28750.069707] fscache [28750.072327] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28750.080836] netfs veth [28750.083034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28750.091982] nft_chain_nat xt_MASQUERADE [28750.094426] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28750.101898] nf_nat nf_conntrack [28750.105815] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28750.111293] nf_defrag_ipv6 [28750.114517] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28750.133253] nf_defrag_ipv4 nft_compat [28750.136044] [28750.136045] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28750.141263] nf_tables nfnetlink [28750.145021] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28750.146515] bridge [28750.153639] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28750.156862] stp llc [28750.163986] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28750.166083] xfs libcrc32c [28750.173209] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28750.175393] sunrpc [28750.182516] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28750.185220] intel_rapl_msr [28750.192344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28750.194440] intel_rapl_common sb_edac [28750.202519] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28750.205310] x86_pkg_temp_thermal coretemp [28750.211047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28750.214790] kvm_intel kvm [28750.221916] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28750.226019] irqbypass [28750.233140] Call Trace: [28750.233141] [28750.235843] i2c_i801 iTCO_wdt iTCO_vendor_support [28750.242967] ? watchdog_timer_fn+0x1b4/0x220 [28750.245326] i2c_smbus crct10dif_pclmul [28750.247769] ? __hrtimer_run_queues+0x1d3/0x360 [28750.249778] joydev crc32_pclmul [28750.254567] ? hrtimer_interrupt+0x104/0x240 [28750.258828] ipmi_ssif ghash_clmulni_intel [28750.262659] ? __local_bh_enable+0x50/0x60 [28750.267181] mei_me mei [28750.270409] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28750.274670] lpc_ich sha512_ssse3 [28750.278762] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28750.282851] acpi_ipmi mfd_core [28750.285296] [28750.290513] wmi ipmi_si [28750.293824] [28750.293826] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28750.298781] ipmi_devintf ipmi_msghandler [28750.301922] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28750.304023] ioatdma acpi_pad [28750.306559] do_raw_spin_lock+0xb5/0xc0 [28750.308652] acpi_power_meter [28750.313959] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28750.317962] ixgbe igb [28750.322835] try_get_cap_refs+0x159/0x880 [ceph] [28750.325796] mdio crc32c_intel [28750.329631] __ceph_get_caps+0x4a1/0x760 [ceph] [28750.332590] nvme ptp [28750.337379] ? __file_remove_privs+0x89/0x110 [28750.339743] nvme_core [28750.344352] ? wake_bit_function+0x70/0x70 [28750.347402] i2c_algo_bit [28750.351930] ? ceph_write_iter+0x260/0x770 [ceph] [28750.354198] pps_core dca [28750.358547] ceph_write_iter+0x260/0x770 [ceph] [28750.360905] fuse [28750.364999] ? lock_is_held_type+0xaa/0x120 [28750.367613] irq event stamp: 151633360 [28750.372314] vfs_write+0x324/0x4f0 [28750.374928] hardirqs last enabled at (151633359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28750.379457] ksys_pwrite64+0x65/0xa0 [28750.381377] hardirqs last disabled at (151633360): [] sysvec_apic_timer_interrupt+0xe/0x90 [28750.385555] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28750.389297] softirqs last enabled at (151633358): [] __do_softirq+0x2be/0x371 [28750.392697] do_syscall_64+0x38/0x80 [28750.402938] softirqs last disabled at (151633353): [] irq_exit_rcu+0xb0/0xd0 [28750.406512] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28750.416320] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28750.421453] RIP: 0033:0x7f48f0eeb93b [28750.430223] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28750.433793] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28750.442392] Workqueue: events delayed_work [ceph] [28750.447435] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28750.456811] [28750.456812] RIP: 0010:ceph_check_delayed_caps+0x63/0x270 [ceph] [28750.460382] ORIG_RAX: 0000000000000012 [28750.467854] Code: 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 <48> 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 [28750.486592] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28750.491288] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [28750.496506] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28750.497996] [28750.503907] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28750.507745] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [28750.526486] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28750.533608] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [28750.533610] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [28750.538826] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28750.545952] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28750.547448] [28750.554565] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28750.606620] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28750.614696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28750.620433] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28750.627559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28750.634681] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28750.641806] Call Trace: [28750.644250] [28750.646264] ? watchdog_timer_fn+0x1b4/0x220 [28750.650535] ? __hrtimer_run_queues+0x1d3/0x360 [28750.655061] ? hrtimer_interrupt+0x104/0x240 [28750.659322] ? __local_bh_enable+0x50/0x60 [28750.663414] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28750.668630] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28750.673588] [28750.675685] [28750.677782] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28750.683090] ? ceph_check_delayed_caps+0x63/0x270 [ceph] [28750.688408] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [28750.693723] delayed_work+0x1d8/0x280 [ceph] [28750.698023] ? process_scheduled_works+0x22a/0x510 [28750.702830] process_scheduled_works+0x2a4/0x510 [28750.707444] worker_thread+0x188/0x340 [28750.711193] ? rescuer_thread+0x310/0x310 [28750.715196] kthread+0xf5/0x130 [28750.718334] ? kthread_complete_and_exit+0x20/0x20 [28750.723119] ret_from_fork+0x31/0x40 [28750.726698] ? kthread_complete_and_exit+0x20/0x20 [28750.731482] ret_from_fork_asm+0x11/0x20 [28750.735406] [28777.961857] watchdog: BUG: soft lockup - CPU#0 stuck for 15687s! [dbench:78683] [28777.969159] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28777.973857] watchdog: BUG: soft lockup - CPU#7 stuck for 24684s! [kworker/7:4:78808] [28778.024608] irq event stamp: 47855220 [28778.032339] Modules linked in: ceph libceph dns_resolver fscache netfs veth [28778.035998] hardirqs last enabled at (47855219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28778.042957] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28778.053115] hardirqs last disabled at (47855220): [] sysvec_apic_timer_interrupt+0xe/0x90 [28778.058766] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [28778.068490] softirqs last enabled at (47855162): [] __do_softirq+0x2be/0x371 [28778.075266] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [28778.083950] softirqs last disabled at (47855157): [] irq_exit_rcu+0xb0/0xd0 [28778.089861] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [28778.098374] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28778.105583] kvm irqbypass [28778.114534] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28778.114535] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28778.117252] i2c_i801 [28778.124719] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28778.130196] iTCO_wdt iTCO_vendor_support [28778.132467] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28778.151206] i2c_smbus crct10dif_pclmul [28778.155209] [28778.160425] joydev crc32_pclmul [28778.164257] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28778.165749] ipmi_ssif ghash_clmulni_intel [28778.168972] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28778.176095] mei_me [28778.180186] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28778.187311] mei lpc_ich [28778.189409] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28778.196534] sha512_ssse3 [28778.199062] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28778.206188] acpi_ipmi [28778.208805] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28778.215928] mfd_core wmi [28778.218285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28778.226363] ipmi_si ipmi_devintf [28778.228981] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28778.234718] ipmi_msghandler ioatdma [28778.238029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28778.245154] acpi_pad acpi_power_meter [28778.248723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28778.255849] ixgbe igb [28778.259591] Call Trace: [28778.259594] [28778.266716] mdio crc32c_intel [28778.269077] ? watchdog_timer_fn+0x1b4/0x220 [28778.271533] nvme ptp [28778.273553] ? __hrtimer_run_queues+0x1d3/0x360 [28778.276606] nvme_core [28778.280872] ? hrtimer_interrupt+0x104/0x240 [28778.283139] i2c_algo_bit pps_core [28778.287663] ? __local_bh_enable+0x50/0x60 [28778.290021] dca fuse [28778.294288] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28778.297682] [28778.297683] irq event stamp: 151854430 [28778.301773] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28778.304045] hardirqs last enabled at (151854429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28778.309261] [28778.310752] hardirqs last disabled at (151854430): [] sysvec_apic_timer_interrupt+0xe/0x90 [28778.314498] [28778.314499] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28778.319455] softirqs last enabled at (151854428): [] __do_softirq+0x2be/0x371 [28778.329701] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28778.331796] softirqs last disabled at (151854423): [] irq_exit_rcu+0xb0/0xd0 [28778.341608] do_raw_spin_lock+0xb5/0xc0 [28778.343704] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28778.349010] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28778.357779] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28778.362653] try_get_cap_refs+0x159/0x880 [ceph] [28778.371261] Workqueue: events delayed_work [ceph] [28778.375093] __ceph_get_caps+0x4a1/0x760 [ceph] [28778.384463] [28778.384464] RIP: 0010:__list_del_entry_valid_or_report+0x0/0xd0 [28778.389263] ? __file_remove_privs+0x89/0x110 [28778.396729] Code: 1d 82 e8 83 0c b0 ff 0f 0b 31 c0 c3 48 89 f2 48 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 [28778.401338] ? wake_bit_function+0x70/0x70 [28778.406036] RSP: 0018:ffffc90002117db0 EFLAGS: 00000216 [28778.410563] ? ceph_write_iter+0x260/0x770 [ceph] [28778.412051] [28778.412051] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: 0000000101b28380 [28778.417963] ceph_write_iter+0x260/0x770 [ceph] [28778.422312] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [28778.441055] ? lock_is_held_type+0xaa/0x120 [28778.445141] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [28778.450360] vfs_write+0x324/0x4f0 [28778.455054] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28778.456554] ksys_pwrite64+0x65/0xa0 [28778.463669] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28778.468195] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28778.475318] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28778.479498] do_syscall_64+0x38/0x80 [28778.486619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28778.490019] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28778.497141] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28778.500714] RIP: 0033:0x7f48f0eeb93b [28778.507836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28778.512966] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28778.521044] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28778.524616] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28778.530351] Call Trace: [28778.530353] [28778.535396] ORIG_RAX: 0000000000000012 [28778.542522] ? watchdog_timer_fn+0x1b4/0x220 [28778.546092] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28778.553217] ? __hrtimer_run_queues+0x1d3/0x360 [28778.571954] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28778.579081] ? hrtimer_interrupt+0x104/0x240 [28778.584294] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28778.586740] ? __local_bh_enable+0x50/0x60 [28778.588749] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28778.592583] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28778.596843] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28778.603970] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28778.608499] [28778.615617] [28778.659819] [28778.661917] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28778.667225] ? __list_add_valid_or_report+0xa0/0xa0 [28778.672101] ceph_check_delayed_caps+0x8f/0x270 [ceph] [28778.677267] delayed_work+0x1d8/0x280 [ceph] [28778.681548] ? process_scheduled_works+0x22a/0x510 [28778.686339] process_scheduled_works+0x2a4/0x510 [28778.690954] worker_thread+0x188/0x340 [28778.694705] ? rescuer_thread+0x310/0x310 [28778.698714] kthread+0xf5/0x130 [28778.701852] ? kthread_complete_and_exit+0x20/0x20 [28778.706638] ret_from_fork+0x31/0x40 [28778.710216] ? kthread_complete_and_exit+0x20/0x20 [28778.715000] ret_from_fork_asm+0x11/0x20 [28778.718924] [28805.962375] watchdog: BUG: soft lockup - CPU#0 stuck for 15713s! [dbench:78683] [28805.969673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28805.974375] watchdog: BUG: soft lockup - CPU#7 stuck for 24710s! [kworker/7:4:78808] [28806.025114] irq event stamp: 47913328 [28806.025115] hardirqs last enabled at (47913327): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28806.032845] Modules linked in: ceph [28806.036502] hardirqs last disabled at (47913328): [] sysvec_apic_timer_interrupt+0xe/0x90 [28806.046659] libceph [28806.050142] softirqs last enabled at (47913270): [] __do_softirq+0x2be/0x371 [28806.059867] dns_resolver fscache [28806.062058] softirqs last disabled at (47913265): [] irq_exit_rcu+0xb0/0xd0 [28806.070746] netfs [28806.074073] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28806.082584] veth nft_chain_nat [28806.084593] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28806.093539] xt_MASQUERADE [28806.096676] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28806.104145] nf_nat nf_conntrack [28806.106852] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28806.112329] nf_defrag_ipv6 nf_defrag_ipv4 [28806.115551] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28806.134289] nft_compat [28806.138380] [28806.138380] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28806.143597] nf_tables [28806.146053] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28806.147541] nfnetlink bridge [28806.154665] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28806.157024] stp llc [28806.164146] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28806.167111] xfs libcrc32c [28806.174235] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28806.176418] sunrpc intel_rapl_msr [28806.183542] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28806.186247] intel_rapl_common [28806.193371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28806.196774] sb_edac x86_pkg_temp_thermal [28806.204855] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28806.207906] coretemp kvm_intel [28806.213642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28806.217646] kvm irqbypass [28806.224773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28806.227909] i2c_i801 iTCO_wdt [28806.235050] Call Trace: [28806.235051] [28806.237771] iTCO_vendor_support i2c_smbus [28806.244897] ? watchdog_timer_fn+0x1b4/0x220 [28806.247947] crct10dif_pclmul joydev [28806.250391] ? __hrtimer_run_queues+0x1d3/0x360 [28806.252402] crc32_pclmul ipmi_ssif [28806.256496] ? hrtimer_interrupt+0x104/0x240 [28806.260774] ghash_clmulni_intel mei_me [28806.264344] ? __local_bh_enable+0x50/0x60 [28806.268869] mei lpc_ich [28806.272356] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28806.276617] sha512_ssse3 acpi_ipmi [28806.280447] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28806.284538] mfd_core wmi [28806.287068] [28806.292285] ipmi_si ipmi_devintf [28806.295774] [28806.300735] ipmi_msghandler [28806.303353] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28806.305452] ioatdma acpi_pad [28806.308778] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28806.310868] acpi_power_meter ixgbe [28806.313750] do_raw_spin_lock+0xb5/0xc0 [28806.319050] igb mdio crc32c_intel nvme [28806.322032] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28806.326900] ptp [28806.330387] try_get_cap_refs+0x159/0x880 [ceph] [28806.334216] nvme_core [28806.338069] __ceph_get_caps+0x4a1/0x760 [ceph] [28806.342848] i2c_algo_bit pps_core [28806.344690] ? __file_remove_privs+0x89/0x110 [28806.349297] dca fuse [28806.351654] ? wake_bit_function+0x70/0x70 [28806.356177] [28806.356177] irq event stamp: 152075596 [28806.359577] ? ceph_write_iter+0x260/0x770 [ceph] [28806.363925] hardirqs last enabled at (152075595): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28806.366198] ceph_write_iter+0x260/0x770 [ceph] [28806.370288] hardirqs last disabled at (152075596): [] sysvec_apic_timer_interrupt+0xe/0x90 [28806.371781] ? lock_is_held_type+0xaa/0x120 [28806.375523] softirqs last enabled at (152075594): [] __do_softirq+0x2be/0x371 [28806.380221] vfs_write+0x324/0x4f0 [28806.390465] softirqs last disabled at (152075589): [] irq_exit_rcu+0xb0/0xd0 [28806.394994] ksys_pwrite64+0x65/0xa0 [28806.404800] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28806.408977] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28806.417747] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28806.417749] Workqueue: events delayed_work [ceph] [28806.421147] do_syscall_64+0x38/0x80 [28806.429742] [28806.433314] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28806.442690] RIP: 0010:lock_acquire+0xf0/0x2b0 [28806.447820] RIP: 0033:0x7f48f0eeb93b [28806.455292] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28806.459990] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28806.463560] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28806.465065] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28806.470113] [28806.470114] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28806.474463] ORIG_RAX: 0000000000000012 [28806.478033] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28806.496789] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28806.515526] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28806.520744] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28806.525960] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28806.527451] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28806.534577] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28806.538406] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28806.545531] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28806.552655] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28806.559779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28806.566908] [28806.574028] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28806.625647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28806.632783] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28806.639912] Call Trace: [28806.642359] [28806.644368] ? watchdog_timer_fn+0x1b4/0x220 [28806.648631] ? __hrtimer_run_queues+0x1d3/0x360 [28806.653160] ? hrtimer_interrupt+0x104/0x240 [28806.657427] ? __local_bh_enable+0x50/0x60 [28806.661521] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28806.666737] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28806.671693] [28806.673790] [28806.675888] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28806.681197] ? lock_acquire+0xf0/0x2b0 [28806.684947] ? igrab+0x43/0x50 [28806.687996] ? lock_release+0xc5/0x260 [28806.691744] _raw_spin_lock+0x30/0x40 [28806.695407] ? igrab+0x1d/0x50 [28806.698456] igrab+0x1d/0x50 [28806.701337] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28806.706486] delayed_work+0x1d8/0x280 [ceph] [28806.710785] ? process_scheduled_works+0x22a/0x510 [28806.715574] process_scheduled_works+0x2a4/0x510 [28806.720191] worker_thread+0x188/0x340 [28806.723938] ? rescuer_thread+0x310/0x310 [28806.727941] kthread+0xf5/0x130 [28806.731080] ? kthread_complete_and_exit+0x20/0x20 [28806.735865] ret_from_fork+0x31/0x40 [28806.739442] ? kthread_complete_and_exit+0x20/0x20 [28806.744228] ret_from_fork_asm+0x11/0x20 [28806.748150] [28808.791795] rcu: INFO: rcu_preempt self-detected stall on CPU [28808.797555] rcu: 7-....: (16383392 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/16382085 fqs=8248781 [28808.808154] rcu: (t=16871882 jiffies g=1309609 q=508229 ncpus=8) [28808.814239] Sending NMI from CPU 7 to CPUs 0: [28808.818592] NMI backtrace for cpu 0 [28808.818593] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28808.818595] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28808.818595] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28808.818599] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28808.818600] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28808.818601] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28808.818602] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28808.818603] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28808.818604] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28808.818604] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28808.818605] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28808.818606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28808.818607] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28808.818608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28808.818609] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28808.818609] Call Trace: [28808.818610] [28808.818611] ? nmi_cpu_backtrace+0xad/0x120 [28808.818615] ? nmi_cpu_backtrace_handler+0x11/0x20 [28808.818616] ? nmi_handle+0xd4/0x260 [28808.818620] ? default_do_nmi+0x49/0xf0 [28808.818622] ? exc_nmi+0x11e/0x150 [28808.818623] ? end_repeat_nmi+0x16/0x67 [28808.818627] ? queued_spin_lock_slowpath+0x18/0x2e0 [28808.818630] ? queued_spin_lock_slowpath+0x18/0x2e0 [28808.818632] ? queued_spin_lock_slowpath+0x18/0x2e0 [28808.818635] [28808.818635] [28808.818636] do_raw_spin_lock+0xb5/0xc0 [28808.818638] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28808.818652] try_get_cap_refs+0x159/0x880 [ceph] [28808.818666] __ceph_get_caps+0x4a1/0x760 [ceph] [28808.818678] ? __file_remove_privs+0x89/0x110 [28808.818679] ? wake_bit_function+0x70/0x70 [28808.818682] ? ceph_write_iter+0x260/0x770 [ceph] [28808.818692] ceph_write_iter+0x260/0x770 [ceph] [28808.818703] ? lock_is_held_type+0xaa/0x120 [28808.818705] vfs_write+0x324/0x4f0 [28808.818709] ksys_pwrite64+0x65/0xa0 [28808.818711] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28808.818714] do_syscall_64+0x38/0x80 [28808.818716] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28808.818718] RIP: 0033:0x7f48f0eeb93b [28808.818719] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28808.818720] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [28808.818721] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28808.818722] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28808.818722] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28808.818723] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28808.818724] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28808.818727] [28808.819590] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28809.123200] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28809.130671] Workqueue: events delayed_work [ceph] [28809.135386] RIP: 0010:_raw_spin_unlock+0x12/0x40 [28809.139999] Code: 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb <48> 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 [28809.158735] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [28809.163952] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [28809.171076] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34500 [28809.178201] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [28809.185324] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28809.192448] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28809.199574] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28809.207651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28809.213389] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28809.220513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28809.227636] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28809.234760] Call Trace: [28809.237206] [28809.239217] ? rcu_dump_cpu_stacks+0xd1/0x180 [28809.243577] ? rcu_sched_clock_irq+0x633/0x1390 [28809.248112] ? lock_release+0xc5/0x260 [28809.251861] ? lock_is_held_type+0xaa/0x120 [28809.256042] ? lock_is_held_type+0xaa/0x120 [28809.260228] ? tick_sched_do_timer+0xa0/0xa0 [28809.264497] ? rcu_read_lock_sched_held+0x56/0x90 [28809.269197] ? tick_sched_do_timer+0xa0/0xa0 [28809.273468] ? update_process_times+0x5f/0x90 [28809.277830] ? tick_sched_handle.isra.16+0x1e/0x50 [28809.282619] ? tick_sched_timer+0x6f/0x80 [28809.286626] ? __hrtimer_run_queues+0x1d3/0x360 [28809.291164] ? hrtimer_interrupt+0x104/0x240 [28809.295429] ? __local_bh_enable+0x50/0x60 [28809.299534] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28809.304755] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28809.309711] [28809.311827] [28809.313925] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28809.319243] ? igrab+0x43/0x50 [28809.322298] ? _raw_spin_unlock+0x12/0x40 [28809.326312] igrab+0x43/0x50 [28809.329196] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28809.334348] delayed_work+0x1d8/0x280 [ceph] [28809.338636] ? process_scheduled_works+0x22a/0x510 [28809.343424] process_scheduled_works+0x2a4/0x510 [28809.348046] worker_thread+0x188/0x340 [28809.351801] ? rescuer_thread+0x310/0x310 [28809.355832] kthread+0xf5/0x130 [28809.358975] ? kthread_complete_and_exit+0x20/0x20 [28809.363770] ret_from_fork+0x31/0x40 [28809.367345] ? kthread_complete_and_exit+0x20/0x20 [28809.372132] ret_from_fork_asm+0x11/0x20 [28809.376072] [28833.962895] watchdog: BUG: soft lockup - CPU#0 stuck for 15739s! [dbench:78683] [28833.970195] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28833.974894] watchdog: BUG: soft lockup - CPU#7 stuck for 24736s! [kworker/7:4:78808] [28834.025646] irq event stamp: 47971672 [28834.033375] Modules linked in: ceph libceph dns_resolver fscache netfs [28834.037034] hardirqs last enabled at (47971671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28834.043575] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28834.053733] hardirqs last disabled at (47971672): [] sysvec_apic_timer_interrupt+0xe/0x90 [28834.053736] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [28834.069541] softirqs last enabled at (47971630): [] __do_softirq+0x2be/0x371 [28834.076925] stp llc xfs libcrc32c sunrpc intel_rapl_msr [28834.085610] softirqs last disabled at (47971625): [] irq_exit_rcu+0xb0/0xd0 [28834.090915] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [28834.099425] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28834.099426] kvm_intel kvm irqbypass i2c_i801 [28834.114715] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28834.114717] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28834.119064] iTCO_wdt iTCO_vendor_support [28834.126535] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28834.132012] i2c_smbus [28834.136016] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28834.154754] crct10dif_pclmul [28834.157113] [28834.162328] joydev [28834.165292] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28834.166783] crc32_pclmul [28834.168880] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28834.176006] ipmi_ssif [28834.178623] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28834.185746] ghash_clmulni_intel [28834.188104] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28834.195227] mei_me mei [28834.198453] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28834.205575] lpc_ich [28834.208020] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28834.215144] sha512_ssse3 acpi_ipmi [28834.217329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28834.225407] mfd_core [28834.228889] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28834.234627] wmi ipmi_si [28834.236900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28834.244022] ipmi_devintf ipmi_msghandler [28834.246554] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28834.253678] ioatdma acpi_pad [28834.257682] Call Trace: [28834.257684] [28834.264805] acpi_power_meter ixgbe [28834.267773] ? watchdog_timer_fn+0x1b4/0x220 [28834.270214] igb mdio [28834.272224] ? __hrtimer_run_queues+0x1d3/0x360 [28834.275708] crc32c_intel nvme [28834.279975] ? hrtimer_interrupt+0x104/0x240 [28834.282244] ptp nvme_core [28834.286769] ? __local_bh_enable+0x50/0x60 [28834.289819] i2c_algo_bit pps_core [28834.294085] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28834.296785] dca fuse [28834.300879] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28834.304291] [28834.304292] irq event stamp: 152291758 [28834.309509] [28834.311779] hardirqs last enabled at (152291757): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28834.316738] [28834.318227] hardirqs last disabled at (152291758): [] sysvec_apic_timer_interrupt+0xe/0x90 [28834.321973] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28834.324068] softirqs last enabled at (152291756): [] __do_softirq+0x2be/0x371 [28834.334317] ? queued_spin_lock_slowpath+0x16/0x2e0 [28834.336412] softirqs last disabled at (152291751): [] irq_exit_rcu+0xb0/0xd0 [28834.346224] do_raw_spin_lock+0xb5/0xc0 [28834.351525] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28834.360299] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28834.365168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28834.365169] Workqueue: events delayed_work [ceph] [28834.373768] try_get_cap_refs+0x159/0x880 [ceph] [28834.377595] [28834.377596] RIP: 0010:_raw_spin_lock+0x30/0x40 [28834.386977] __ceph_get_caps+0x4a1/0x760 [ceph] [28834.391757] Code: 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c 5c ff <48> 89 df 58 5b e9 76 33 5c ff 66 0f 1f 44 00 00 f3 0f 1e fa 0f 1f [28834.399233] ? __file_remove_privs+0x89/0x110 [28834.403925] RSP: 0018:ffffc90002117d88 EFLAGS: 00000296 [28834.408537] ? wake_bit_function+0x70/0x70 [28834.410026] [28834.410027] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [28834.414466] ? ceph_write_iter+0x260/0x770 [ceph] [28834.418987] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28834.418989] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [28834.437727] ceph_write_iter+0x260/0x770 [ceph] [28834.442078] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28834.447302] ? lock_is_held_type+0xaa/0x120 [28834.451385] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28834.452879] vfs_write+0x324/0x4f0 [28834.459999] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28834.464702] ksys_pwrite64+0x65/0xa0 [28834.471822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28834.478945] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28834.483469] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28834.490596] do_syscall_64+0x38/0x80 [28834.494772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28834.501898] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28834.505297] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28834.513380] RIP: 0033:0x7f48f0eeb93b [28834.516951] Call Trace: [28834.516952] [28834.522689] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28834.527819] ? watchdog_timer_fn+0x1b4/0x220 [28834.534943] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28834.538513] ? __hrtimer_run_queues+0x1d3/0x360 [28834.545636] ORIG_RAX: 0000000000000012 [28834.550684] ? hrtimer_interrupt+0x104/0x240 [28834.557805] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28834.561376] ? __local_bh_enable+0x50/0x60 [28834.563820] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28834.565834] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28834.584567] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28834.588850] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28834.594068] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28834.598592] [28834.602423] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28834.606686] [28834.606688] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28834.613816] [28834.617905] ? _raw_spin_lock+0x30/0x40 [28834.672086] ? _raw_spin_lock+0x30/0x40 [28834.675918] ? igrab+0x1d/0x50 [28834.678968] igrab+0x1d/0x50 [28834.681847] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28834.686995] delayed_work+0x1d8/0x280 [ceph] [28834.691277] ? process_scheduled_works+0x22a/0x510 [28834.696085] process_scheduled_works+0x2a4/0x510 [28834.700699] worker_thread+0x188/0x340 [28834.704450] ? rescuer_thread+0x310/0x310 [28834.708460] kthread+0xf5/0x130 [28834.711600] ? kthread_complete_and_exit+0x20/0x20 [28834.716385] ret_from_fork+0x31/0x40 [28834.719962] ? kthread_complete_and_exit+0x20/0x20 [28834.724746] ret_from_fork_asm+0x11/0x20 [28834.728670] [28861.963414] watchdog: BUG: soft lockup - CPU#0 stuck for 15765s! [dbench:78683] [28861.970716] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28861.975414] watchdog: BUG: soft lockup - CPU#7 stuck for 24762s! [kworker/7:4:78808] [28862.026159] irq event stamp: 48029778 [28862.026160] hardirqs last enabled at (48029777): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28862.033890] Modules linked in: [28862.037547] hardirqs last disabled at (48029778): [] sysvec_apic_timer_interrupt+0xe/0x90 [28862.047704] ceph [28862.050755] softirqs last enabled at (48029720): [] __do_softirq+0x2be/0x371 [28862.060480] libceph [28862.062403] softirqs last disabled at (48029715): [] irq_exit_rcu+0xb0/0xd0 [28862.071087] dns_resolver [28862.073271] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28862.081781] fscache netfs [28862.084415] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28862.093360] veth nft_chain_nat [28862.096064] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [28862.103553] xt_MASQUERADE nf_nat [28862.106689] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [28862.112167] nf_conntrack nf_defrag_ipv6 [28862.115479] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28862.134215] nf_defrag_ipv4 nft_compat [28862.138133] [28862.138134] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28862.143350] nf_tables nfnetlink [28862.147096] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28862.148587] bridge [28862.155710] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28862.158933] stp [28862.166057] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28862.168156] llc xfs [28862.175278] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28862.177117] libcrc32c sunrpc [28862.184242] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28862.186425] intel_rapl_msr intel_rapl_common [28862.193548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28862.196513] sb_edac x86_pkg_temp_thermal [28862.204590] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28862.208942] coretemp [28862.214679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28862.218682] kvm_intel [28862.225813] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28862.228091] kvm irqbypass [28862.235220] Call Trace: [28862.235221] [28862.237577] i2c_i801 iTCO_wdt iTCO_vendor_support [28862.244703] ? watchdog_timer_fn+0x1b4/0x220 [28862.247404] i2c_smbus crct10dif_pclmul [28862.249851] ? __hrtimer_run_queues+0x1d3/0x360 [28862.251860] joydev [28862.256647] ? hrtimer_interrupt+0x104/0x240 [28862.260908] crc32_pclmul ipmi_ssif [28862.264737] ? __local_bh_enable+0x50/0x60 [28862.269261] ghash_clmulni_intel mei_me [28862.271363] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28862.275623] mei lpc_ich [28862.279109] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28862.283198] sha512_ssse3 acpi_ipmi [28862.287029] [28862.292247] mfd_core wmi [28862.294777] [28862.299736] ipmi_si ipmi_devintf [28862.303220] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28862.305316] ipmi_msghandler ioatdma [28862.307939] ? queued_spin_lock_slowpath+0x18/0x2e0 [28862.310032] acpi_pad acpi_power_meter [28862.313343] do_raw_spin_lock+0xb5/0xc0 [28862.318645] ixgbe igb mdio [28862.322219] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28862.327093] crc32c_intel nvme [28862.330842] try_get_cap_refs+0x159/0x880 [ceph] [28862.334671] ptp nvme_core [28862.337466] __ceph_get_caps+0x4a1/0x760 [ceph] [28862.342245] i2c_algo_bit pps_core [28862.345299] ? __file_remove_privs+0x89/0x110 [28862.349908] dca fuse [28862.352613] ? wake_bit_function+0x70/0x70 [28862.357136] [28862.357137] irq event stamp: 152512806 [28862.360536] ? ceph_write_iter+0x260/0x770 [ceph] [28862.364883] hardirqs last enabled at (152512805): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28862.367156] ceph_write_iter+0x260/0x770 [ceph] [28862.371245] hardirqs last disabled at (152512806): [] sysvec_apic_timer_interrupt+0xe/0x90 [28862.372739] ? lock_is_held_type+0xaa/0x120 [28862.376479] softirqs last enabled at (152512804): [] __do_softirq+0x2be/0x371 [28862.381179] vfs_write+0x324/0x4f0 [28862.391422] softirqs last disabled at (152512799): [] irq_exit_rcu+0xb0/0xd0 [28862.395950] ksys_pwrite64+0x65/0xa0 [28862.405757] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28862.409932] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28862.418704] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28862.422105] do_syscall_64+0x38/0x80 [28862.430699] Workqueue: events delayed_work [ceph] [28862.434270] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28862.443648] [28862.443649] RIP: 0010:lock_acquire+0xf0/0x2b0 [28862.448779] RIP: 0033:0x7f48f0eeb93b [28862.456249] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28862.459821] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28862.464519] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28862.469563] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28862.471053] [28862.471054] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28862.475402] ORIG_RAX: 0000000000000012 [28862.478974] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28862.497711] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28862.516448] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28862.521667] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28862.526884] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28862.528375] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28862.535498] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28862.539330] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28862.546453] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28862.553577] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28862.560702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28862.567832] [28862.574950] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28862.626588] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28862.633711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28862.640835] Call Trace: [28862.643281] [28862.645291] ? watchdog_timer_fn+0x1b4/0x220 [28862.649557] ? __hrtimer_run_queues+0x1d3/0x360 [28862.654098] ? hrtimer_interrupt+0x104/0x240 [28862.658370] ? __local_bh_enable+0x50/0x60 [28862.662463] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28862.667686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28862.672644] [28862.674740] [28862.676837] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28862.682147] ? lock_acquire+0xf0/0x2b0 [28862.685896] ? igrab+0x43/0x50 [28862.688946] ? lock_release+0xc5/0x260 [28862.692690] _raw_spin_lock+0x30/0x40 [28862.696346] ? igrab+0x1d/0x50 [28862.699397] igrab+0x1d/0x50 [28862.702275] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28862.707427] delayed_work+0x1d8/0x280 [ceph] [28862.711707] ? process_scheduled_works+0x22a/0x510 [28862.716497] process_scheduled_works+0x2a4/0x510 [28862.721111] worker_thread+0x188/0x340 [28862.724863] ? rescuer_thread+0x310/0x310 [28862.728874] kthread+0xf5/0x130 [28862.732011] ? kthread_complete_and_exit+0x20/0x20 [28862.736797] ret_from_fork+0x31/0x40 [28862.740391] ? kthread_complete_and_exit+0x20/0x20 [28862.745175] ret_from_fork_asm+0x11/0x20 [28862.749109] [28889.963934] watchdog: BUG: soft lockup - CPU#0 stuck for 15791s! [dbench:78683] [28889.971240] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28889.975933] watchdog: BUG: soft lockup - CPU#7 stuck for 24788s! [kworker/7:4:78808] [28890.026689] irq event stamp: 48088006 [28890.034419] Modules linked in: ceph libceph dns_resolver fscache netfs [28890.038078] hardirqs last enabled at (48088005): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28890.044611] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28890.054769] hardirqs last disabled at (48088006): [] sysvec_apic_timer_interrupt+0xe/0x90 [28890.054772] nf_defrag_ipv6 [28890.054773] softirqs last enabled at (48087948): [] __do_softirq+0x2be/0x371 [28890.060854] nf_defrag_ipv4 nft_compat [28890.070578] softirqs last disabled at (48087943): [] irq_exit_rcu+0xb0/0xd0 [28890.073368] nf_tables [28890.082054] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28890.085796] nfnetlink bridge [28890.094326] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28890.096683] stp [28890.105627] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28890.108605] llc xfs [28890.116078] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28890.117915] libcrc32c sunrpc [28890.123394] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28890.125578] intel_rapl_msr intel_rapl_common [28890.144330] [28890.147296] sb_edac [28890.152514] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28890.156865] x86_pkg_temp_thermal coretemp [28890.158355] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28890.160538] kvm_intel kvm [28890.167663] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28890.171753] irqbypass i2c_i801 [28890.178878] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28890.181582] iTCO_wdt iTCO_vendor_support [28890.188706] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28890.191844] i2c_smbus crct10dif_pclmul [28890.198969] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28890.202973] joydev crc32_pclmul [28890.210095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28890.213927] ipmi_ssif ghash_clmulni_intel [28890.222005] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28890.225229] mei_me mei [28890.230966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28890.235056] lpc_ich sha512_ssse3 [28890.242181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28890.244624] acpi_ipmi mfd_core [28890.251749] Call Trace: [28890.251752] [28890.255061] wmi ipmi_si [28890.262187] ? watchdog_timer_fn+0x1b4/0x220 [28890.265330] ipmi_devintf ipmi_msghandler [28890.267775] ? __hrtimer_run_queues+0x1d3/0x360 [28890.269784] ioatdma [28890.272335] ? hrtimer_interrupt+0x104/0x240 [28890.276612] acpi_pad acpi_power_meter [28890.280628] ? __local_bh_enable+0x50/0x60 [28890.285151] ixgbe [28890.287338] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28890.291611] igb mdio [28890.295353] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28890.299445] crc32c_intel [28890.301455] [28890.306672] nvme ptp [28890.308940] [28890.313900] nvme_core i2c_algo_bit [28890.316518] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28890.318613] pps_core dca [28890.320889] ? queued_spin_lock_slowpath+0x16/0x2e0 [28890.322981] fuse [28890.326467] do_raw_spin_lock+0xb5/0xc0 [28890.331770] irq event stamp: 152733784 [28890.334389] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28890.339259] hardirqs last enabled at (152733783): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28890.341185] try_get_cap_refs+0x159/0x880 [ceph] [28890.345012] hardirqs last disabled at (152733784): [] sysvec_apic_timer_interrupt+0xe/0x90 [28890.348763] __ceph_get_caps+0x4a1/0x760 [ceph] [28890.353541] softirqs last enabled at (152733782): [] __do_softirq+0x2be/0x371 [28890.363790] ? __file_remove_privs+0x89/0x110 [28890.368395] softirqs last disabled at (152733777): [] irq_exit_rcu+0xb0/0xd0 [28890.378206] ? wake_bit_function+0x70/0x70 [28890.382732] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28890.391504] ? ceph_write_iter+0x260/0x770 [ceph] [28890.395853] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28890.404451] ceph_write_iter+0x260/0x770 [ceph] [28890.408542] Workqueue: events delayed_work [ceph] [28890.417922] ? lock_is_held_type+0xaa/0x120 [28890.422624] [28890.422625] RIP: 0010:lock_acquire+0xf0/0x2b0 [28890.430096] vfs_write+0x324/0x4f0 [28890.434625] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28890.439338] ksys_pwrite64+0x65/0xa0 [28890.443511] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28890.445003] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28890.449353] [28890.449354] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28890.452750] do_syscall_64+0x38/0x80 [28890.471489] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28890.475060] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28890.480276] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28890.485408] RIP: 0033:0x7f48f0eeb93b [28890.486897] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28890.494022] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28890.497592] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28890.504734] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28890.509777] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28890.516901] ORIG_RAX: 0000000000000012 [28890.520472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28890.527596] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28890.546352] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28890.553477] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28890.558693] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28890.566771] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28890.570601] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28890.576338] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28890.583464] Call Trace: [28890.590589] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28890.597712] [28890.604841] [28890.611959] ? watchdog_timer_fn+0x1b4/0x220 [28890.644236] ? __hrtimer_run_queues+0x1d3/0x360 [28890.648764] ? hrtimer_interrupt+0x104/0x240 [28890.653033] ? __local_bh_enable+0x50/0x60 [28890.657126] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28890.662349] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28890.667306] [28890.669403] [28890.671502] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28890.676808] ? lock_acquire+0xf0/0x2b0 [28890.680552] ? igrab+0x43/0x50 [28890.683619] ? lock_release+0xc5/0x260 [28890.687364] _raw_spin_lock+0x30/0x40 [28890.691030] ? igrab+0x1d/0x50 [28890.694079] igrab+0x1d/0x50 [28890.696959] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28890.702108] delayed_work+0x1d8/0x280 [ceph] [28890.706389] ? process_scheduled_works+0x22a/0x510 [28890.711177] process_scheduled_works+0x2a4/0x510 [28890.715794] worker_thread+0x188/0x340 [28890.719544] ? rescuer_thread+0x310/0x310 [28890.723555] kthread+0xf5/0x130 [28890.726693] ? kthread_complete_and_exit+0x20/0x20 [28890.731477] ret_from_fork+0x31/0x40 [28890.735056] ? kthread_complete_and_exit+0x20/0x20 [28890.739839] ret_from_fork_asm+0x11/0x20 [28890.743765] [28917.964452] watchdog: BUG: soft lockup - CPU#0 stuck for 15817s! [dbench:78683] [28917.971754] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28917.976452] watchdog: BUG: soft lockup - CPU#7 stuck for 24814s! [kworker/7:4:78808] [28918.027193] irq event stamp: 48146134 [28918.027194] hardirqs last enabled at (48146133): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28918.034925] Modules linked in: [28918.038581] hardirqs last disabled at (48146134): [] sysvec_apic_timer_interrupt+0xe/0x90 [28918.048740] ceph [28918.051791] softirqs last enabled at (48146128): [] __do_softirq+0x2be/0x371 [28918.061515] libceph dns_resolver [28918.063439] softirqs last disabled at (48146123): [] irq_exit_rcu+0xb0/0xd0 [28918.072129] fscache netfs [28918.075444] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28918.083954] veth nft_chain_nat [28918.086657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28918.095601] xt_MASQUERADE nf_nat [28918.098739] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28918.106209] nf_conntrack nf_defrag_ipv6 [28918.109520] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28918.114997] nf_defrag_ipv4 nft_compat [28918.118916] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28918.137652] nf_tables [28918.141397] [28918.141398] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28918.146614] nfnetlink [28918.148971] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28918.150461] bridge stp [28918.157587] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28918.159944] llc xfs [28918.167069] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28918.169513] libcrc32c [28918.176636] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28918.178819] sunrpc [28918.185944] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28918.188301] intel_rapl_msr intel_rapl_common [28918.195425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28918.197522] sb_edac x86_pkg_temp_thermal [28918.205600] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28918.209951] coretemp kvm_intel [28918.215689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28918.219693] kvm irqbypass [28918.226818] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28918.229955] i2c_i801 [28918.237078] Call Trace: [28918.239783] iTCO_wdt iTCO_vendor_support [28918.246906] [28918.249178] i2c_smbus crct10dif_pclmul [28918.251623] ? watchdog_timer_fn+0x1b4/0x220 [28918.255625] joydev [28918.257637] ? __hrtimer_run_queues+0x1d3/0x360 [28918.261466] crc32_pclmul ipmi_ssif [28918.265735] ? hrtimer_interrupt+0x104/0x240 [28918.267829] ghash_clmulni_intel mei_me [28918.272370] ? __local_bh_enable+0x50/0x60 [28918.275854] mei lpc_ich [28918.280133] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28918.283957] sha512_ssse3 [28918.288049] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28918.290580] acpi_ipmi mfd_core [28918.295798] [28918.298412] wmi ipmi_si [28918.303371] [28918.303373] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28918.306510] ipmi_devintf [28918.308611] ? queued_spin_lock_slowpath+0x16/0x2e0 [28918.311149] ipmi_msghandler ioatdma [28918.313244] do_raw_spin_lock+0xb5/0xc0 [28918.318548] acpi_pad acpi_power_meter ixgbe [28918.321165] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28918.326034] igb mdio [28918.329607] try_get_cap_refs+0x159/0x880 [ceph] [28918.333435] crc32c_intel nvme [28918.337706] __ceph_get_caps+0x4a1/0x760 [ceph] [28918.342484] ptp nvme_core i2c_algo_bit [28918.344758] ? __file_remove_privs+0x89/0x110 [28918.349367] pps_core dca [28918.352418] ? wake_bit_function+0x70/0x70 [28918.356942] fuse [28918.360774] ? ceph_write_iter+0x260/0x770 [ceph] [28918.365132] [28918.365133] irq event stamp: 152954694 [28918.367748] ceph_write_iter+0x260/0x770 [ceph] [28918.371852] hardirqs last enabled at (152954693): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28918.373774] ? lock_is_held_type+0xaa/0x120 [28918.378469] hardirqs last disabled at (152954694): [] sysvec_apic_timer_interrupt+0xe/0x90 [28918.379962] vfs_write+0x324/0x4f0 [28918.383703] softirqs last enabled at (152954692): [] __do_softirq+0x2be/0x371 [28918.388233] ksys_pwrite64+0x65/0xa0 [28918.398473] softirqs last disabled at (152954687): [] irq_exit_rcu+0xb0/0xd0 [28918.402650] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28918.412460] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28918.415861] do_syscall_64+0x38/0x80 [28918.424629] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28918.424630] Workqueue: events delayed_work [ceph] [28918.428201] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28918.436797] [28918.441927] RIP: 0033:0x7f48f0eeb93b [28918.451306] RIP: 0010:lock_acquire+0xf0/0x2b0 [28918.454875] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28918.462346] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28918.467044] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28918.472087] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28918.473578] ORIG_RAX: 0000000000000012 [28918.477148] [28918.477149] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28918.481500] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28918.500237] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28918.518976] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28918.524193] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28918.529411] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28918.533242] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28918.534732] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28918.541856] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28918.548979] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28918.556103] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28918.563233] [28918.570353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28918.621980] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28918.629105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28918.636228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28918.643355] Call Trace: [28918.645797] [28918.647809] ? watchdog_timer_fn+0x1b4/0x220 [28918.652074] ? __hrtimer_run_queues+0x1d3/0x360 [28918.656601] ? hrtimer_interrupt+0x104/0x240 [28918.660869] ? __local_bh_enable+0x50/0x60 [28918.664962] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28918.670186] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28918.675147] [28918.677249] [28918.679346] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28918.684655] ? lock_acquire+0xf0/0x2b0 [28918.688406] ? igrab+0x43/0x50 [28918.691455] ? lock_release+0xc5/0x260 [28918.695201] _raw_spin_lock+0x30/0x40 [28918.698859] ? igrab+0x1d/0x50 [28918.701907] igrab+0x1d/0x50 [28918.704787] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28918.709936] delayed_work+0x1d8/0x280 [ceph] [28918.714217] ? process_scheduled_works+0x22a/0x510 [28918.719007] process_scheduled_works+0x2a4/0x510 [28918.723621] worker_thread+0x188/0x340 [28918.727372] ? rescuer_thread+0x310/0x310 [28918.731374] kthread+0xf5/0x130 [28918.734513] ? kthread_complete_and_exit+0x20/0x20 [28918.739297] ret_from_fork+0x31/0x40 [28918.742875] ? kthread_complete_and_exit+0x20/0x20 [28918.747658] ret_from_fork_asm+0x11/0x20 [28918.751582] [28945.964970] watchdog: BUG: soft lockup - CPU#0 stuck for 15844s! [dbench:78683] [28945.972274] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28945.976970] watchdog: BUG: soft lockup - CPU#7 stuck for 24841s! [kworker/7:4:78808] [28946.027717] irq event stamp: 48204340 [28946.027719] hardirqs last enabled at (48204339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28946.035446] Modules linked in: [28946.039103] hardirqs last disabled at (48204340): [] sysvec_apic_timer_interrupt+0xe/0x90 [28946.049263] ceph [28946.052311] softirqs last enabled at (48204282): [] __do_softirq+0x2be/0x371 [28946.062036] libceph [28946.063961] softirqs last disabled at (48204277): [] irq_exit_rcu+0xb0/0xd0 [28946.072646] dns_resolver fscache [28946.074828] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28946.083340] netfs [28946.086668] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28946.095611] veth nft_chain_nat [28946.097622] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [28946.105112] xt_MASQUERADE nf_nat [28946.108247] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [28946.113726] nf_conntrack nf_defrag_ipv6 [28946.117036] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28946.135775] nf_defrag_ipv4 nft_compat [28946.139691] [28946.144908] nf_tables nfnetlink [28946.148654] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28946.150143] bridge [28946.153368] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28946.160491] stp [28946.162590] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28946.169713] llc xfs [28946.171550] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28946.178675] libcrc32c sunrpc [28946.180858] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28946.187983] intel_rapl_msr intel_rapl_common [28946.190948] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28946.198071] sb_edac [28946.202421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28946.210498] x86_pkg_temp_thermal coretemp [28946.212683] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28946.218422] kvm_intel [28946.222513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28946.229647] kvm irqbypass [28946.232003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28946.239127] i2c_i801 [28946.241829] Call Trace: [28946.248955] iTCO_wdt [28946.251226] [28946.253669] iTCO_vendor_support i2c_smbus [28946.255941] ? watchdog_timer_fn+0x1b4/0x220 [28946.257950] crct10dif_pclmul joydev [28946.262043] ? __hrtimer_run_queues+0x1d3/0x360 [28946.266305] crc32_pclmul ipmi_ssif [28946.269878] ? hrtimer_interrupt+0x104/0x240 [28946.274400] ghash_clmulni_intel mei_me [28946.277885] ? __local_bh_enable+0x50/0x60 [28946.282148] mei lpc_ich [28946.285982] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28946.290069] sha512_ssse3 acpi_ipmi [28946.292601] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28946.297818] mfd_core wmi [28946.301302] [28946.306260] ipmi_si ipmi_devintf [28946.308877] [28946.308879] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28946.310974] ipmi_msghandler ioatdma [28946.314290] ? queued_spin_lock_slowpath+0x2d/0x2e0 [28946.316383] acpi_pad acpi_power_meter [28946.321688] do_raw_spin_lock+0xb5/0xc0 [28946.325258] ixgbe igb [28946.330129] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28946.333873] mdio crc32c_intel [28946.337705] try_get_cap_refs+0x159/0x880 [ceph] [28946.340060] nvme ptp [28946.344848] __ceph_get_caps+0x4a1/0x760 [ceph] [28946.347894] nvme_core i2c_algo_bit [28946.352511] ? __file_remove_privs+0x89/0x110 [28946.354777] pps_core dca [28946.359302] ? wake_bit_function+0x70/0x70 [28946.362783] fuse [28946.367138] ? ceph_write_iter+0x260/0x770 [ceph] [28946.369752] irq event stamp: 153175554 [28946.373843] ceph_write_iter+0x260/0x770 [ceph] [28946.375768] hardirqs last enabled at (153175553): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28946.380470] ? lock_is_held_type+0xaa/0x120 [28946.384208] hardirqs last disabled at (153175554): [] sysvec_apic_timer_interrupt+0xe/0x90 [28946.388734] vfs_write+0x324/0x4f0 [28946.398977] softirqs last enabled at (153175552): [] __do_softirq+0x2be/0x371 [28946.403160] ksys_pwrite64+0x65/0xa0 [28946.412965] softirqs last disabled at (153175547): [] irq_exit_rcu+0xb0/0xd0 [28946.416373] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28946.425143] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28946.428714] do_syscall_64+0x38/0x80 [28946.437310] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28946.442442] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28946.451818] Workqueue: events delayed_work [ceph] [28946.455389] RIP: 0033:0x7f48f0eeb93b [28946.462861] [28946.462862] RIP: 0010:lock_acquire+0xf0/0x2b0 [28946.467905] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28946.472602] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [28946.476173] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28946.477664] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [28946.482015] ORIG_RAX: 0000000000000012 [28946.500751] [28946.500752] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [28946.519489] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28946.524706] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [28946.529924] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28946.533754] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [28946.535245] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28946.542374] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [28946.549502] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28946.556625] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [28946.563750] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28946.570876] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28946.578005] [28946.585123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28946.622494] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28946.629619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28946.636742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28946.643868] Call Trace: [28946.646310] [28946.648322] ? watchdog_timer_fn+0x1b4/0x220 [28946.652586] ? __hrtimer_run_queues+0x1d3/0x360 [28946.657114] ? hrtimer_interrupt+0x104/0x240 [28946.661383] ? __local_bh_enable+0x50/0x60 [28946.665476] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28946.670701] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28946.675664] [28946.677762] [28946.679860] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28946.685168] ? lock_acquire+0xf0/0x2b0 [28946.688920] ? igrab+0x43/0x50 [28946.691968] ? lock_release+0xc5/0x260 [28946.695714] _raw_spin_lock+0x30/0x40 [28946.699375] ? igrab+0x1d/0x50 [28946.702430] igrab+0x1d/0x50 [28946.705308] ceph_check_delayed_caps+0xb7/0x270 [ceph] [28946.710456] delayed_work+0x1d8/0x280 [ceph] [28946.714739] ? process_scheduled_works+0x22a/0x510 [28946.719527] process_scheduled_works+0x2a4/0x510 [28946.724145] worker_thread+0x188/0x340 [28946.727892] ? rescuer_thread+0x310/0x310 [28946.731896] kthread+0xf5/0x130 [28946.735034] ? kthread_complete_and_exit+0x20/0x20 [28946.739820] ret_from_fork+0x31/0x40 [28946.743396] ? kthread_complete_and_exit+0x20/0x20 [28946.748181] ret_from_fork_asm+0x11/0x20 [28946.752105] [28973.965487] watchdog: BUG: soft lockup - CPU#0 stuck for 15870s! [dbench:78683] [28973.972791] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [28973.977487] watchdog: BUG: soft lockup - CPU#7 stuck for 24867s! [kworker/7:4:78808] [28974.028239] irq event stamp: 48262506 [28974.035968] Modules linked in: ceph libceph dns_resolver fscache netfs [28974.039627] hardirqs last enabled at (48262505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28974.046169] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [28974.056327] hardirqs last disabled at (48262506): [] sysvec_apic_timer_interrupt+0xe/0x90 [28974.062412] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [28974.072136] softirqs last enabled at (48262450): [] __do_softirq+0x2be/0x371 [28974.079528] stp llc xfs libcrc32c sunrpc intel_rapl_msr [28974.088213] softirqs last disabled at (48262445): [] irq_exit_rcu+0xb0/0xd0 [28974.088217] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [28974.102036] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28974.102037] kvm_intel [28974.102039] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28974.108380] kvm irqbypass [28974.117324] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [28974.119681] i2c_i801 iTCO_wdt [28974.127152] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [28974.129883] iTCO_vendor_support i2c_smbus [28974.135360] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [28974.138411] crct10dif_pclmul joydev [28974.157149] [28974.161256] crc32_pclmul ipmi_ssif [28974.166473] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [28974.170046] ghash_clmulni_intel mei_me [28974.171535] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [28974.175018] mei lpc_ich [28974.182145] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [28974.185991] sha512_ssse3 [28974.193115] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [28974.195646] acpi_ipmi mfd_core [28974.202770] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [28974.205389] wmi [28974.212513] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [28974.215651] ipmi_si [28974.222774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28974.224613] ipmi_devintf [28974.232690] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [28974.234884] ipmi_msghandler ioatdma [28974.240618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28974.243237] acpi_pad acpi_power_meter [28974.250361] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28974.253932] ixgbe igb [28974.261055] Call Trace: [28974.261058] [28974.264800] mdio crc32c_intel [28974.271925] ? watchdog_timer_fn+0x1b4/0x220 [28974.274280] nvme ptp [28974.276727] ? __hrtimer_run_queues+0x1d3/0x360 [28974.278736] nvme_core i2c_algo_bit [28974.281789] ? hrtimer_interrupt+0x104/0x240 [28974.286051] pps_core dca [28974.288320] ? __local_bh_enable+0x50/0x60 [28974.292845] fuse [28974.296331] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28974.300593] irq event stamp: 153396248 [28974.303212] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28974.307302] hardirqs last enabled at (153396247): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28974.309225] [28974.314443] hardirqs last disabled at (153396248): [] sysvec_apic_timer_interrupt+0xe/0x90 [28974.318188] [28974.323144] softirqs last enabled at (153396246): [] __do_softirq+0x2be/0x371 [28974.333408] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28974.335503] softirqs last disabled at (153396241): [] irq_exit_rcu+0xb0/0xd0 [28974.345318] ? queued_spin_lock_slowpath+0x16/0x2e0 [28974.347411] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [28974.356186] do_raw_spin_lock+0xb5/0xc0 [28974.361486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [28974.361487] Workqueue: events delayed_work [ceph] [28974.370085] __cap_delay_requeue+0xa7/0x1d0 [ceph] [28974.374956] [28974.374957] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [28974.384334] try_get_cap_refs+0x159/0x880 [ceph] [28974.388169] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [28974.395648] __ceph_get_caps+0x4a1/0x760 [ceph] [28974.400340] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [28974.405131] ? __file_remove_privs+0x89/0x110 [28974.406616] [28974.406617] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [28974.412527] ? wake_bit_function+0x70/0x70 [28974.417137] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [28974.435891] ? ceph_write_iter+0x260/0x770 [ceph] [28974.440415] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [28974.445634] ceph_write_iter+0x260/0x770 [ceph] [28974.449984] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [28974.451479] ? lock_is_held_type+0xaa/0x120 [28974.458598] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [28974.462691] vfs_write+0x324/0x4f0 [28974.469813] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [28974.474515] ksys_pwrite64+0x65/0xa0 [28974.481634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28974.486175] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [28974.493300] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [28974.497479] do_syscall_64+0x38/0x80 [28974.504603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [28974.508000] entry_SYSCALL_64_after_hwframe+0x63/0xcd [28974.516076] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [28974.519648] RIP: 0033:0x7f48f0eeb93b [28974.525385] Call Trace: [28974.525387] [28974.530515] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [28974.537641] ? watchdog_timer_fn+0x1b4/0x220 [28974.541211] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [28974.548337] ? __hrtimer_run_queues+0x1d3/0x360 [28974.553378] ORIG_RAX: 0000000000000012 [28974.560507] ? hrtimer_interrupt+0x104/0x240 [28974.564073] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [28974.566517] ? __local_bh_enable+0x50/0x60 [28974.568527] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [28974.587268] ? __sysvec_apic_timer_interrupt+0x61/0x160 [28974.591529] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [28974.596747] ? sysvec_apic_timer_interrupt+0x6d/0x90 [28974.601272] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [28974.605104] [28974.609366] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [28974.616491] [28974.616492] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [28974.620588] [28974.627711] ? ceph_check_delayed_caps+0x9e/0x270 [ceph] [28974.676241] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [28974.681556] delayed_work+0x1d8/0x280 [ceph] [28974.685838] ? process_scheduled_works+0x22a/0x510 [28974.690627] process_scheduled_works+0x2a4/0x510 [28974.695244] worker_thread+0x188/0x340 [28974.698991] ? rescuer_thread+0x310/0x310 [28974.702995] kthread+0xf5/0x130 [28974.706133] ? kthread_complete_and_exit+0x20/0x20 [28974.710917] ret_from_fork+0x31/0x40 [28974.714495] ? kthread_complete_and_exit+0x20/0x20 [28974.719280] ret_from_fork_asm+0x11/0x20 [28974.723213] [29001.966000] watchdog: BUG: soft lockup - CPU#0 stuck for 15896s! [dbench:78683] [29001.973303] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29001.978000] watchdog: BUG: soft lockup - CPU#7 stuck for 24893s! [kworker/7:4:78808] [29002.028744] irq event stamp: 48320570 [29002.028745] hardirqs last enabled at (48320569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29002.036475] Modules linked in: [29002.040131] hardirqs last disabled at (48320570): [] sysvec_apic_timer_interrupt+0xe/0x90 [29002.050288] ceph [29002.053339] softirqs last enabled at (48320512): [] __do_softirq+0x2be/0x371 [29002.063063] libceph [29002.064989] softirqs last disabled at (48320507): [] irq_exit_rcu+0xb0/0xd0 [29002.073675] dns_resolver fscache [29002.075856] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29002.084366] netfs [29002.087686] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29002.096630] veth nft_chain_nat [29002.098640] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [29002.106113] xt_MASQUERADE [29002.109250] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [29002.114727] nf_nat nf_conntrack [29002.117430] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29002.136170] nf_defrag_ipv6 [29002.139397] [29002.139398] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29002.144618] nf_defrag_ipv4 [29002.147408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29002.148899] nft_compat nf_tables [29002.156023] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29002.158815] nfnetlink bridge [29002.165940] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29002.169250] stp llc [29002.176374] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29002.179355] xfs libcrc32c [29002.186481] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29002.188675] sunrpc intel_rapl_msr [29002.195796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29002.198501] intel_rapl_common sb_edac [29002.206578] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29002.209974] x86_pkg_temp_thermal coretemp [29002.215714] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29002.219456] kvm_intel kvm [29002.226581] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29002.230676] irqbypass i2c_i801 [29002.237797] Call Trace: [29002.237798] [29002.240501] iTCO_wdt iTCO_vendor_support [29002.247624] ? watchdog_timer_fn+0x1b4/0x220 [29002.250761] i2c_smbus crct10dif_pclmul [29002.253206] ? __hrtimer_run_queues+0x1d3/0x360 [29002.255217] joydev [29002.259224] ? hrtimer_interrupt+0x104/0x240 [29002.263484] crc32_pclmul ipmi_ssif [29002.267314] ? __local_bh_enable+0x50/0x60 [29002.271839] ghash_clmulni_intel [29002.273938] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29002.278201] mei_me mei [29002.281685] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29002.285776] lpc_ich [29002.289001] [29002.294216] sha512_ssse3 acpi_ipmi [29002.296677] [29002.296678] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29002.301635] mfd_core wmi [29002.303823] ? queued_spin_lock_slowpath+0x2b/0x2e0 [29002.305918] ipmi_si ipmi_devintf [29002.309403] do_raw_spin_lock+0xb5/0xc0 [29002.311498] ipmi_msghandler ioatdma [29002.316802] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29002.319421] acpi_pad acpi_power_meter ixgbe [29002.324294] try_get_cap_refs+0x159/0x880 [ceph] [29002.327602] igb mdio [29002.331436] __ceph_get_caps+0x4a1/0x760 [ceph] [29002.335002] crc32c_intel nvme [29002.339792] ? __file_remove_privs+0x89/0x110 [29002.344050] ptp [29002.348680] ? wake_bit_function+0x70/0x70 [29002.350950] nvme_core i2c_algo_bit [29002.355477] ? ceph_write_iter+0x260/0x770 [ceph] [29002.358524] pps_core dca [29002.362877] ceph_write_iter+0x260/0x770 [ceph] [29002.364712] fuse [29002.368807] ? lock_is_held_type+0xaa/0x120 [29002.372287] irq event stamp: 153617388 [29002.376987] vfs_write+0x324/0x4f0 [29002.379603] hardirqs last enabled at (153617387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29002.384132] ksys_pwrite64+0x65/0xa0 [29002.386049] hardirqs last disabled at (153617388): [] sysvec_apic_timer_interrupt+0xe/0x90 [29002.390228] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29002.393971] softirqs last enabled at (153617386): [] __do_softirq+0x2be/0x371 [29002.397370] do_syscall_64+0x38/0x80 [29002.407613] softirqs last disabled at (153617381): [] irq_exit_rcu+0xb0/0xd0 [29002.411185] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29002.420995] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29002.426126] RIP: 0033:0x7f48f0eeb93b [29002.434897] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29002.438467] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29002.447065] Workqueue: events delayed_work [ceph] [29002.452108] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29002.461486] [29002.461487] RIP: 0010:lock_acquire+0x50/0x2b0 [29002.465058] ORIG_RAX: 0000000000000012 [29002.472528] Code: 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 0f 82 2b 01 00 00 <8b> 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b 05 db d9 ea 7e 85 [29002.491265] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29002.495963] RSP: 0018:ffffc90002117d18 EFLAGS: 00000202 [29002.501179] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29002.502671] [29002.507023] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29002.510852] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 0000000000000000 [29002.529590] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29002.536715] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [29002.541931] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29002.549056] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [29002.550553] [29002.557672] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [29002.602618] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29002.609741] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29002.617819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29002.623556] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29002.630686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29002.637814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29002.644938] Call Trace: [29002.647382] [29002.649409] ? watchdog_timer_fn+0x1b4/0x220 [29002.653690] ? __hrtimer_run_queues+0x1d3/0x360 [29002.658218] ? hrtimer_interrupt+0x104/0x240 [29002.662479] ? __local_bh_enable+0x50/0x60 [29002.666572] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29002.671796] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29002.676753] [29002.678850] [29002.680948] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29002.686257] ? lock_acquire+0x50/0x2b0 [29002.690006] ? lock_acquire+0x184/0x2b0 [29002.693836] ? igrab+0x43/0x50 [29002.696888] ? lock_release+0xc5/0x260 [29002.700634] _raw_spin_lock+0x30/0x40 [29002.704297] ? igrab+0x1d/0x50 [29002.707349] igrab+0x1d/0x50 [29002.710227] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29002.715377] delayed_work+0x1d8/0x280 [ceph] [29002.719674] ? process_scheduled_works+0x22a/0x510 [29002.724474] process_scheduled_works+0x2a4/0x510 [29002.729089] worker_thread+0x188/0x340 [29002.732837] ? rescuer_thread+0x310/0x310 [29002.736842] kthread+0xf5/0x130 [29002.739979] ? kthread_complete_and_exit+0x20/0x20 [29002.744762] ret_from_fork+0x31/0x40 [29002.748332] ? kthread_complete_and_exit+0x20/0x20 [29002.753119] ret_from_fork_asm+0x11/0x20 [29002.757040] [29004.383411] rcu: INFO: rcu_preempt self-detected stall on CPU [29004.389172] rcu: 7-....: (16573017 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/16571709 fqs=8344667 [29004.399762] rcu: (t=17067470 jiffies g=1309609 q=512211 ncpus=8) [29004.405849] Sending NMI from CPU 7 to CPUs 0: [29004.410199] NMI backtrace for cpu 0 [29004.410200] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29004.410202] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29004.410203] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29004.410206] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29004.410207] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29004.410209] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29004.410210] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29004.410210] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29004.410211] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29004.410212] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29004.410213] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29004.410214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29004.410215] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29004.410216] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29004.410216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29004.410217] Call Trace: [29004.410218] [29004.410219] ? nmi_cpu_backtrace+0xad/0x120 [29004.410222] ? nmi_cpu_backtrace_handler+0x11/0x20 [29004.410224] ? nmi_handle+0xd4/0x260 [29004.410228] ? default_do_nmi+0x49/0xf0 [29004.410230] ? exc_nmi+0x11e/0x150 [29004.410231] ? end_repeat_nmi+0x16/0x67 [29004.410236] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29004.410238] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29004.410241] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29004.410243] [29004.410243] [29004.410244] do_raw_spin_lock+0xb5/0xc0 [29004.410247] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29004.410261] try_get_cap_refs+0x159/0x880 [ceph] [29004.410275] __ceph_get_caps+0x4a1/0x760 [ceph] [29004.410288] ? __file_remove_privs+0x89/0x110 [29004.410290] ? wake_bit_function+0x70/0x70 [29004.410292] ? ceph_write_iter+0x260/0x770 [ceph] [29004.410303] ceph_write_iter+0x260/0x770 [ceph] [29004.410314] ? lock_is_held_type+0xaa/0x120 [29004.410317] vfs_write+0x324/0x4f0 [29004.410321] ksys_pwrite64+0x65/0xa0 [29004.410323] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29004.410326] do_syscall_64+0x38/0x80 [29004.410328] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29004.410330] RIP: 0033:0x7f48f0eeb93b [29004.410331] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29004.410332] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [29004.410334] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29004.410334] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29004.410335] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29004.410336] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29004.410336] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29004.410339] [29004.411197] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29004.714757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29004.722229] Workqueue: events delayed_work [ceph] [29004.726943] RIP: 0010:lock_acquire+0xf0/0x2b0 [29004.731295] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29004.750030] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29004.755248] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29004.762373] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29004.769497] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29004.776622] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29004.783747] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29004.790869] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29004.798947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29004.804686] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29004.811808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29004.818933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29004.826056] Call Trace: [29004.828501] [29004.830513] ? rcu_dump_cpu_stacks+0xd1/0x180 [29004.834873] ? rcu_sched_clock_irq+0x633/0x1390 [29004.839397] ? lock_release+0xc5/0x260 [29004.843141] ? lock_is_held_type+0xaa/0x120 [29004.847318] ? lock_is_held_type+0xaa/0x120 [29004.851495] ? tick_sched_do_timer+0xa0/0xa0 [29004.855766] ? rcu_read_lock_sched_held+0x56/0x90 [29004.860466] ? tick_sched_do_timer+0xa0/0xa0 [29004.864738] ? update_process_times+0x5f/0x90 [29004.869090] ? tick_sched_handle.isra.16+0x1e/0x50 [29004.873881] ? tick_sched_timer+0x6f/0x80 [29004.877885] ? __hrtimer_run_queues+0x1d3/0x360 [29004.882413] ? hrtimer_interrupt+0x104/0x240 [29004.886681] ? __local_bh_enable+0x50/0x60 [29004.890776] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29004.896000] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29004.900956] [29004.903053] [29004.905152] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29004.910468] ? lock_acquire+0xf0/0x2b0 [29004.914218] ? igrab+0x43/0x50 [29004.917267] ? lock_release+0xc5/0x260 [29004.921013] _raw_spin_lock+0x30/0x40 [29004.924678] ? igrab+0x1d/0x50 [29004.927729] igrab+0x1d/0x50 [29004.930607] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29004.935756] delayed_work+0x1d8/0x280 [ceph] [29004.940037] ? process_scheduled_works+0x22a/0x510 [29004.944828] process_scheduled_works+0x2a4/0x510 [29004.949444] worker_thread+0x188/0x340 [29004.953193] ? rescuer_thread+0x310/0x310 [29004.957195] kthread+0xf5/0x130 [29004.960333] ? kthread_complete_and_exit+0x20/0x20 [29004.965118] ret_from_fork+0x31/0x40 [29004.968688] ? kthread_complete_and_exit+0x20/0x20 [29004.973471] ret_from_fork_asm+0x11/0x20 [29004.977394] [29029.966511] watchdog: BUG: soft lockup - CPU#0 stuck for 15922s! [dbench:78683] [29029.973817] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29029.978511] watchdog: BUG: soft lockup - CPU#7 stuck for 24919s! [kworker/7:4:78808] [29030.029268] irq event stamp: 48378736 [29030.036996] Modules linked in: ceph libceph dns_resolver fscache netfs veth [29030.040654] hardirqs last enabled at (48378735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29030.047613] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [29030.057771] hardirqs last disabled at (48378736): [] sysvec_apic_timer_interrupt+0xe/0x90 [29030.057774] nf_defrag_ipv4 [29030.057775] softirqs last enabled at (48378680): [] __do_softirq+0x2be/0x371 [29030.064722] nft_compat nf_tables [29030.074446] softirqs last disabled at (48378675): [] irq_exit_rcu+0xb0/0xd0 [29030.077237] nfnetlink [29030.085921] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29030.089230] bridge stp [29030.097742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29030.100101] llc xfs [29030.109045] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29030.111489] libcrc32c sunrpc [29030.118959] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29030.121141] intel_rapl_msr [29030.126621] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29030.129583] intel_rapl_common sb_edac [29030.148322] [29030.151113] x86_pkg_temp_thermal coretemp [29030.156331] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29030.160074] kvm_intel kvm [29030.161565] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29030.165654] irqbypass i2c_i801 [29030.172779] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29030.175483] iTCO_wdt iTCO_vendor_support [29030.182607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29030.185745] i2c_smbus [29030.192868] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29030.196875] crct10dif_pclmul [29030.203996] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29030.206356] joydev crc32_pclmul [29030.213480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29030.216444] ipmi_ssif ghash_clmulni_intel [29030.224520] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29030.227745] mei_me mei [29030.233483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29030.237573] lpc_ich sha512_ssse3 [29030.244697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29030.247140] acpi_ipmi [29030.254266] Call Trace: [29030.254268] [29030.257575] mfd_core wmi [29030.264701] ? watchdog_timer_fn+0x1b4/0x220 [29030.267057] ipmi_si ipmi_devintf [29030.269503] ? __hrtimer_run_queues+0x1d3/0x360 [29030.271511] ipmi_msghandler ioatdma [29030.274133] ? hrtimer_interrupt+0x104/0x240 [29030.278393] acpi_pad acpi_power_meter [29030.281705] ? __local_bh_enable+0x50/0x60 [29030.286230] ixgbe igb [29030.289803] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29030.294062] mdio crc32c_intel [29030.297809] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29030.301910] nvme ptp [29030.304264] [29030.309481] nvme_core [29030.312532] [29030.312533] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29030.317491] i2c_algo_bit pps_core [29030.319763] ? queued_spin_lock_slowpath+0x16/0x2e0 [29030.321859] dca fuse [29030.324218] do_raw_spin_lock+0xb5/0xc0 [29030.326312] [29030.326313] irq event stamp: 153833410 [29030.331618] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29030.335013] hardirqs last enabled at (153833409): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29030.339907] try_get_cap_refs+0x159/0x880 [ceph] [29030.342172] hardirqs last disabled at (153833410): [] sysvec_apic_timer_interrupt+0xe/0x90 [29030.346011] __ceph_get_caps+0x4a1/0x760 [ceph] [29030.347493] softirqs last enabled at (153833408): [] __do_softirq+0x2be/0x371 [29030.351243] ? __file_remove_privs+0x89/0x110 [29030.356022] softirqs last disabled at (153833403): [] irq_exit_rcu+0xb0/0xd0 [29030.366266] ? wake_bit_function+0x70/0x70 [29030.370878] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29030.380691] ? ceph_write_iter+0x260/0x770 [ceph] [29030.385212] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29030.385214] Workqueue: events delayed_work [ceph] [29030.393984] ceph_write_iter+0x260/0x770 [ceph] [29030.398334] [29030.398335] RIP: 0010:lock_acquire+0xf0/0x2b0 [29030.406936] ? lock_is_held_type+0xaa/0x120 [29030.411021] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29030.420402] vfs_write+0x324/0x4f0 [29030.425097] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29030.432572] ksys_pwrite64+0x65/0xa0 [29030.437265] [29030.437265] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29030.441791] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29030.443281] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29030.447633] do_syscall_64+0x38/0x80 [29030.451808] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29030.470547] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29030.473943] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29030.479160] RIP: 0033:0x7f48f0eeb93b [29030.482732] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29030.484223] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29030.491345] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29030.496478] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29030.503601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29030.507172] ORIG_RAX: 0000000000000012 [29030.514313] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29030.519356] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29030.526481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29030.530052] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29030.537175] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29030.555914] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29030.563991] Call Trace: [29030.563992] [29030.569209] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29030.574948] ? watchdog_timer_fn+0x1b4/0x220 [29030.578778] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29030.585919] ? __hrtimer_run_queues+0x1d3/0x360 [29030.593049] [29030.600170] ? hrtimer_interrupt+0x104/0x240 [29030.655479] ? __local_bh_enable+0x50/0x60 [29030.659573] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29030.664796] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29030.669752] [29030.671852] [29030.673947] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29030.679257] ? lock_acquire+0xf0/0x2b0 [29030.683006] ? igrab+0x43/0x50 [29030.686055] ? lock_release+0xc5/0x260 [29030.689801] _raw_spin_lock+0x30/0x40 [29030.693458] ? igrab+0x1d/0x50 [29030.696508] igrab+0x1d/0x50 [29030.699388] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29030.704538] delayed_work+0x1d8/0x280 [ceph] [29030.708818] ? process_scheduled_works+0x22a/0x510 [29030.713608] process_scheduled_works+0x2a4/0x510 [29030.718224] worker_thread+0x188/0x340 [29030.721972] ? rescuer_thread+0x310/0x310 [29030.725975] kthread+0xf5/0x130 [29030.729121] ? kthread_complete_and_exit+0x20/0x20 [29030.733918] ret_from_fork+0x31/0x40 [29030.737494] ? kthread_complete_and_exit+0x20/0x20 [29030.742277] ret_from_fork_asm+0x11/0x20 [29030.746205] [29057.967023] watchdog: BUG: soft lockup - CPU#0 stuck for 15948s! [dbench:78683] [29057.974320] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29057.979023] watchdog: BUG: soft lockup - CPU#7 stuck for 24945s! [kworker/7:4:78808] [29058.029762] irq event stamp: 48436774 [29058.029764] hardirqs last enabled at (48436773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29058.037493] Modules linked in: [29058.041151] hardirqs last disabled at (48436774): [] sysvec_apic_timer_interrupt+0xe/0x90 [29058.051309] ceph [29058.054358] softirqs last enabled at (48436738): [] __do_softirq+0x2be/0x371 [29058.064083] libceph [29058.066007] softirqs last disabled at (48436733): [] irq_exit_rcu+0xb0/0xd0 [29058.074697] dns_resolver [29058.076884] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29058.085395] fscache netfs [29058.088029] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29058.096973] veth nft_chain_nat [29058.099677] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29058.107148] xt_MASQUERADE [29058.110286] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29058.115764] nf_nat nf_conntrack [29058.118468] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29058.137204] nf_defrag_ipv6 [29058.140429] [29058.140430] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29058.145647] nf_defrag_ipv4 [29058.148436] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29058.149927] nft_compat nf_tables [29058.157052] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29058.159842] nfnetlink bridge [29058.166965] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29058.170277] stp llc [29058.177419] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29058.180383] xfs [29058.187508] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29058.189700] libcrc32c sunrpc [29058.196825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29058.198662] intel_rapl_msr [29058.206739] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29058.209709] intel_rapl_common sb_edac [29058.215450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29058.218239] x86_pkg_temp_thermal [29058.225364] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29058.229109] coretemp kvm_intel [29058.236231] Call Trace: [29058.236233] [29058.239542] kvm irqbypass [29058.246669] ? watchdog_timer_fn+0x1b4/0x220 [29058.249803] i2c_i801 iTCO_wdt [29058.252250] ? __hrtimer_run_queues+0x1d3/0x360 [29058.254258] iTCO_vendor_support i2c_smbus [29058.256966] ? hrtimer_interrupt+0x104/0x240 [29058.261228] crct10dif_pclmul joydev [29058.264279] ? __local_bh_enable+0x50/0x60 [29058.268801] crc32_pclmul ipmi_ssif [29058.272895] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29058.277157] ghash_clmulni_intel mei_me [29058.280728] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29058.284817] mei [29058.288304] [29058.293519] lpc_ich sha512_ssse3 [29058.297350] [29058.302309] acpi_ipmi mfd_core [29058.304145] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29058.306243] wmi ipmi_si [29058.309559] ? queued_spin_lock_slowpath+0x18/0x2e0 [29058.311651] ipmi_devintf [29058.314789] do_raw_spin_lock+0xb5/0xc0 [29058.320091] ipmi_msghandler ioatdma acpi_pad [29058.322624] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29058.327495] acpi_power_meter ixgbe [29058.330112] try_get_cap_refs+0x159/0x880 [ceph] [29058.333943] igb mdio [29058.338296] __ceph_get_caps+0x4a1/0x760 [ceph] [29058.343077] crc32c_intel nvme [29058.346566] ? __file_remove_privs+0x89/0x110 [29058.351170] ptp nvme_core [29058.353441] ? wake_bit_function+0x70/0x70 [29058.357966] i2c_algo_bit [29058.361019] ? ceph_write_iter+0x260/0x770 [ceph] [29058.365368] pps_core dca [29058.368071] ceph_write_iter+0x260/0x770 [ceph] [29058.372162] fuse [29058.374783] ? lock_is_held_type+0xaa/0x120 [29058.379478] irq event stamp: 154054298 [29058.382097] vfs_write+0x324/0x4f0 [29058.386618] hardirqs last enabled at (154054297): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29058.388546] ksys_pwrite64+0x65/0xa0 [29058.392720] hardirqs last disabled at (154054298): [] sysvec_apic_timer_interrupt+0xe/0x90 [29058.396465] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29058.399862] softirqs last enabled at (154054296): [] __do_softirq+0x2be/0x371 [29058.410106] do_syscall_64+0x38/0x80 [29058.413677] softirqs last disabled at (154054291): [] irq_exit_rcu+0xb0/0xd0 [29058.423489] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29058.428618] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29058.437389] RIP: 0033:0x7f48f0eeb93b [29058.440959] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29058.449558] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29058.454600] Workqueue: events delayed_work [ceph] [29058.463979] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29058.467551] [29058.467552] RIP: 0010:lock_acquire+0xf0/0x2b0 [29058.475020] ORIG_RAX: 0000000000000012 [29058.493758] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29058.498456] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29058.503673] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29058.505163] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29058.509515] [29058.513345] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29058.532082] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29058.539208] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29058.544426] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29058.551549] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29058.553040] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29058.560169] [29058.567287] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29058.605111] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29058.612233] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29058.620311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29058.626048] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29058.633173] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29058.640297] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29058.647430] Call Trace: [29058.649875] [29058.651886] ? watchdog_timer_fn+0x1b4/0x220 [29058.656149] ? __hrtimer_run_queues+0x1d3/0x360 [29058.660677] ? hrtimer_interrupt+0x104/0x240 [29058.664946] ? __local_bh_enable+0x50/0x60 [29058.669040] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29058.674262] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29058.679219] [29058.681316] [29058.683431] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29058.688742] ? lock_acquire+0xf0/0x2b0 [29058.692491] ? igrab+0x43/0x50 [29058.695539] ? lock_release+0xc5/0x260 [29058.699284] _raw_spin_lock+0x30/0x40 [29058.702941] ? igrab+0x1d/0x50 [29058.705991] igrab+0x1d/0x50 [29058.708870] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29058.714019] delayed_work+0x1d8/0x280 [ceph] [29058.718301] ? process_scheduled_works+0x22a/0x510 [29058.723091] process_scheduled_works+0x2a4/0x510 [29058.727722] worker_thread+0x188/0x340 [29058.731473] ? rescuer_thread+0x310/0x310 [29058.735476] kthread+0xf5/0x130 [29058.738615] ? kthread_complete_and_exit+0x20/0x20 [29058.743399] ret_from_fork+0x31/0x40 [29058.746976] ? kthread_complete_and_exit+0x20/0x20 [29058.751761] ret_from_fork_asm+0x11/0x20 [29058.755685] [29085.967535] watchdog: BUG: soft lockup - CPU#0 stuck for 15974s! [dbench:78683] [29085.974845] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29085.979535] watchdog: BUG: soft lockup - CPU#7 stuck for 24971s! [kworker/7:4:78808] [29086.030293] irq event stamp: 48494960 [29086.038023] Modules linked in: ceph libceph dns_resolver fscache netfs veth [29086.041680] hardirqs last enabled at (48494959): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29086.048633] nft_chain_nat xt_MASQUERADE nf_nat [29086.058789] hardirqs last disabled at (48494960): [] sysvec_apic_timer_interrupt+0xe/0x90 [29086.063313] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [29086.073038] softirqs last enabled at (48494912): [] __do_softirq+0x2be/0x371 [29086.080943] bridge stp llc xfs libcrc32c [29086.089625] softirqs last disabled at (48494907): [] irq_exit_rcu+0xb0/0xd0 [29086.093631] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [29086.102143] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29086.110393] kvm_intel kvm irqbypass [29086.119337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29086.122906] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [29086.130396] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [29086.130398] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [29086.141524] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [29086.148821] mei_me mei lpc_ich [29086.167560] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29086.167561] sha512_ssse3 acpi_ipmi [29086.170697] [29086.170698] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29086.175929] mfd_core wmi [29086.179414] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29086.180905] ipmi_si ipmi_devintf [29086.188047] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29086.190663] ipmi_msghandler ioatdma [29086.197787] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29086.201100] acpi_pad acpi_power_meter [29086.208222] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29086.211795] ixgbe igb [29086.218932] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29086.222679] mdio crc32c_intel [29086.229804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29086.232160] nvme ptp [29086.240239] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29086.243289] nvme_core i2c_algo_bit pps_core [29086.249027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29086.251297] dca fuse [29086.258422] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29086.262686] [29086.262686] irq event stamp: 154275090 [29086.269809] Call Trace: [29086.269812] [29086.272080] hardirqs last enabled at (154275089): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29086.279206] ? watchdog_timer_fn+0x1b4/0x220 [29086.280713] hardirqs last disabled at (154275090): [] sysvec_apic_timer_interrupt+0xe/0x90 [29086.284457] ? __hrtimer_run_queues+0x1d3/0x360 [29086.286899] softirqs last enabled at (154275088): [] __do_softirq+0x2be/0x371 [29086.288935] ? hrtimer_interrupt+0x104/0x240 [29086.299172] softirqs last disabled at (154275083): [] irq_exit_rcu+0xb0/0xd0 [29086.303436] ? __local_bh_enable+0x50/0x60 [29086.313249] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29086.317775] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29086.326543] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29086.326544] Workqueue: events delayed_work [ceph] [29086.330807] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29086.339406] [29086.339406] RIP: 0010:lock_acquire+0xf0/0x2b0 [29086.343495] [29086.352873] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29086.358091] [29086.365561] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29086.370258] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29086.375225] [29086.375225] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29086.376719] ? queued_spin_lock_slowpath+0x2b/0x2e0 [29086.381064] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29086.383166] do_raw_spin_lock+0xb5/0xc0 [29086.401901] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29086.404000] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29086.409229] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29086.414539] try_get_cap_refs+0x159/0x880 [ceph] [29086.416026] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29086.423157] __ceph_get_caps+0x4a1/0x760 [ceph] [29086.428023] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29086.435151] ? __file_remove_privs+0x89/0x110 [29086.438979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29086.446101] ? wake_bit_function+0x70/0x70 [29086.450886] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29086.458012] ? ceph_write_iter+0x260/0x770 [ceph] [29086.462620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29086.469746] ceph_write_iter+0x260/0x770 [ceph] [29086.474268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29086.482351] ? lock_is_held_type+0xaa/0x120 [29086.486697] Call Trace: [29086.486700] [29086.492438] vfs_write+0x324/0x4f0 [29086.496525] ? watchdog_timer_fn+0x1b4/0x220 [29086.503653] ksys_pwrite64+0x65/0xa0 [29086.508347] ? __hrtimer_run_queues+0x1d3/0x360 [29086.515473] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29086.519998] ? hrtimer_interrupt+0x104/0x240 [29086.527121] do_syscall_64+0x38/0x80 [29086.531296] ? __local_bh_enable+0x50/0x60 [29086.533743] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29086.535755] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29086.539149] RIP: 0033:0x7f48f0eeb93b [29086.543414] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29086.546985] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29086.551508] [29086.556638] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29086.560903] [29086.560904] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29086.564474] ORIG_RAX: 0000000000000012 [29086.568569] ? lock_acquire+0xf0/0x2b0 [29086.573607] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29086.578826] ? igrab+0x43/0x50 [29086.582396] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29086.587355] ? lock_release+0xc5/0x260 [29086.606092] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29086.608191] _raw_spin_lock+0x30/0x40 [29086.613405] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29086.615504] ? igrab+0x1d/0x50 [29086.620807] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29086.624638] igrab+0x1d/0x50 [29086.628389] [29086.635508] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29086.687719] delayed_work+0x1d8/0x280 [ceph] [29086.692000] ? process_scheduled_works+0x22a/0x510 [29086.696789] process_scheduled_works+0x2a4/0x510 [29086.701405] worker_thread+0x188/0x340 [29086.705155] ? rescuer_thread+0x310/0x310 [29086.709157] kthread+0xf5/0x130 [29086.712295] ? kthread_complete_and_exit+0x20/0x20 [29086.717079] ret_from_fork+0x31/0x40 [29086.720659] ? kthread_complete_and_exit+0x20/0x20 [29086.725442] ret_from_fork_asm+0x11/0x20 [29086.729366] [29113.968047] watchdog: BUG: soft lockup - CPU#0 stuck for 16000s! [dbench:78683] [29113.975350] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29113.980047] watchdog: BUG: soft lockup - CPU#7 stuck for 24997s! [kworker/7:4:78808] [29114.030790] irq event stamp: 48552994 [29114.030791] hardirqs last enabled at (48552993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29114.038522] Modules linked in: ceph [29114.042177] hardirqs last disabled at (48552994): [] sysvec_apic_timer_interrupt+0xe/0x90 [29114.052336] libceph [29114.055820] softirqs last enabled at (48552946): [] __do_softirq+0x2be/0x371 [29114.065543] dns_resolver [29114.067728] softirqs last disabled at (48552941): [] irq_exit_rcu+0xb0/0xd0 [29114.076412] fscache netfs [29114.079029] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29114.087539] veth nft_chain_nat [29114.090244] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29114.099187] xt_MASQUERADE [29114.102326] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29114.109797] nf_nat nf_conntrack [29114.112502] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29114.117979] nf_defrag_ipv6 nf_defrag_ipv4 [29114.121201] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29114.139940] nft_compat nf_tables [29114.144031] [29114.144032] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29114.149248] nfnetlink [29114.152557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29114.154049] bridge [29114.161172] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29114.163531] stp llc [29114.170656] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29114.172752] xfs libcrc32c [29114.179876] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29114.182062] sunrpc intel_rapl_msr [29114.189184] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29114.191888] intel_rapl_common [29114.199013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29114.202410] sb_edac x86_pkg_temp_thermal [29114.210489] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29114.213538] coretemp [29114.219277] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29114.223281] kvm_intel kvm [29114.230403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29114.232676] irqbypass i2c_i801 [29114.239798] Call Trace: [29114.239800] [29114.242503] iTCO_wdt iTCO_vendor_support [29114.249627] ? watchdog_timer_fn+0x1b4/0x220 [29114.252764] i2c_smbus crct10dif_pclmul [29114.255211] ? __hrtimer_run_queues+0x1d3/0x360 [29114.257218] joydev crc32_pclmul [29114.261227] ? hrtimer_interrupt+0x104/0x240 [29114.265486] ipmi_ssif ghash_clmulni_intel [29114.269317] ? __local_bh_enable+0x50/0x60 [29114.273841] mei_me mei [29114.277069] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29114.281331] lpc_ich sha512_ssse3 [29114.285422] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29114.289530] acpi_ipmi mfd_core [29114.291974] [29114.297190] wmi [29114.300502] [29114.300504] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29114.305458] ipmi_si [29114.308600] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29114.310695] ipmi_devintf [29114.312533] do_raw_spin_lock+0xb5/0xc0 [29114.314627] ipmi_msghandler [29114.319934] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29114.322117] ioatdma acpi_pad [29114.326990] try_get_cap_refs+0x159/0x880 [ceph] [29114.329606] acpi_power_meter ixgbe [29114.333451] __ceph_get_caps+0x4a1/0x760 [ceph] [29114.336321] igb mdio [29114.341111] ? __file_remove_privs+0x89/0x110 [29114.344070] crc32c_intel [29114.348682] ? wake_bit_function+0x70/0x70 [29114.352163] nvme [29114.356692] ? ceph_write_iter+0x260/0x770 [ceph] [29114.358959] ptp nvme_core [29114.363309] ceph_write_iter+0x260/0x770 [ceph] [29114.365928] i2c_algo_bit [29114.370021] ? lock_is_held_type+0xaa/0x120 [29114.371942] pps_core dca [29114.376642] vfs_write+0x324/0x4f0 [29114.379344] fuse [29114.383873] ksys_pwrite64+0x65/0xa0 [29114.386486] irq event stamp: 154496176 [29114.390663] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29114.393279] hardirqs last enabled at (154496175): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29114.396679] do_syscall_64+0x38/0x80 [29114.398601] hardirqs last disabled at (154496176): [] sysvec_apic_timer_interrupt+0xe/0x90 [29114.402173] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29114.405917] softirqs last enabled at (154496174): [] __do_softirq+0x2be/0x371 [29114.411046] RIP: 0033:0x7f48f0eeb93b [29114.421292] softirqs last disabled at (154496169): [] irq_exit_rcu+0xb0/0xd0 [29114.424861] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29114.434673] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29114.439716] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29114.448504] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29114.452074] ORIG_RAX: 0000000000000012 [29114.460673] Workqueue: events delayed_work [ceph] [29114.479411] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29114.488789] [29114.494005] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29114.501476] RIP: 0010:lock_acquire+0xf0/0x2b0 [29114.505307] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29114.510005] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29114.517130] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29114.518619] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29114.525744] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29114.530094] [29114.530095] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29114.537224] [29114.555957] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29114.593344] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29114.600469] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29114.607592] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29114.614717] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29114.622812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29114.628550] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29114.635674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29114.642799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29114.649921] Call Trace: [29114.652367] [29114.654378] ? watchdog_timer_fn+0x1b4/0x220 [29114.658642] ? __hrtimer_run_queues+0x1d3/0x360 [29114.663169] ? hrtimer_interrupt+0x104/0x240 [29114.667454] ? __local_bh_enable+0x50/0x60 [29114.671548] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29114.676765] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29114.681735] [29114.683836] [29114.685934] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29114.691242] ? lock_acquire+0xf0/0x2b0 [29114.694992] ? igrab+0x43/0x50 [29114.698042] ? lock_release+0xc5/0x260 [29114.701786] _raw_spin_lock+0x30/0x40 [29114.705452] ? igrab+0x1d/0x50 [29114.708501] igrab+0x1d/0x50 [29114.711382] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29114.716530] delayed_work+0x1d8/0x280 [ceph] [29114.720810] ? process_scheduled_works+0x22a/0x510 [29114.725601] process_scheduled_works+0x2a4/0x510 [29114.730216] worker_thread+0x188/0x340 [29114.733967] ? rescuer_thread+0x310/0x310 [29114.737979] kthread+0xf5/0x130 [29114.741124] ? kthread_complete_and_exit+0x20/0x20 [29114.745909] ret_from_fork+0x31/0x40 [29114.749486] ? kthread_complete_and_exit+0x20/0x20 [29114.754271] ret_from_fork_asm+0x11/0x20 [29114.758195] [29141.968558] watchdog: BUG: soft lockup - CPU#0 stuck for 16026s! [dbench:78683] [29141.975863] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29141.980558] watchdog: BUG: soft lockup - CPU#7 stuck for 25023s! [kworker/7:4:78808] [29142.031314] irq event stamp: 48611214 [29142.039044] Modules linked in: ceph libceph dns_resolver fscache [29142.042700] hardirqs last enabled at (48611213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29142.042704] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [29142.058856] hardirqs last disabled at (48611214): [] sysvec_apic_timer_interrupt+0xe/0x90 [29142.065458] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [29142.075183] softirqs last enabled at (48611166): [] __do_softirq+0x2be/0x371 [29142.083260] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [29142.091951] softirqs last disabled at (48611161): [] irq_exit_rcu+0xb0/0xd0 [29142.098819] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [29142.107330] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29142.107332] i2c_i801 [29142.107333] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29142.113500] iTCO_wdt iTCO_vendor_support [29142.122443] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29142.124716] i2c_smbus crct10dif_pclmul [29142.132185] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29142.136190] joydev [29142.141667] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29142.145497] crc32_pclmul ipmi_ssif [29142.164243] [29142.166341] ghash_clmulni_intel [29142.171558] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29142.175042] mei_me mei [29142.176535] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29142.179758] lpc_ich [29142.186881] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29142.189326] sha512_ssse3 [29142.196451] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29142.198634] acpi_ipmi mfd_core [29142.205757] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29142.208374] wmi ipmi_si [29142.215500] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29142.218638] ipmi_devintf ipmi_msghandler [29142.225761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29142.228292] ioatdma acpi_pad [29142.236370] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29142.240373] acpi_power_meter [29142.246110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29142.249074] ixgbe igb [29142.256199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29142.259163] mdio crc32c_intel [29142.266287] Call Trace: [29142.266290] [29142.268644] nvme ptp [29142.275770] ? watchdog_timer_fn+0x1b4/0x220 [29142.278819] nvme_core [29142.281264] ? __hrtimer_run_queues+0x1d3/0x360 [29142.283274] i2c_algo_bit [29142.285549] ? hrtimer_interrupt+0x104/0x240 [29142.289808] pps_core dca [29142.292166] ? __local_bh_enable+0x50/0x60 [29142.296691] fuse [29142.299311] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29142.303573] irq event stamp: 154717104 [29142.306189] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29142.310280] hardirqs last enabled at (154717103): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29142.312204] [29142.317421] hardirqs last disabled at (154717104): [] sysvec_apic_timer_interrupt+0xe/0x90 [29142.321167] [29142.326124] softirqs last enabled at (154717102): [] __do_softirq+0x2be/0x371 [29142.336368] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29142.338466] softirqs last disabled at (154717097): [] irq_exit_rcu+0xb0/0xd0 [29142.348280] ? queued_spin_lock_slowpath+0x16/0x2e0 [29142.350372] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29142.359145] do_raw_spin_lock+0xb5/0xc0 [29142.364447] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29142.373049] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29142.377918] Workqueue: events delayed_work [ceph] [29142.387295] try_get_cap_refs+0x159/0x880 [ceph] [29142.391123] [29142.398599] __ceph_get_caps+0x4a1/0x760 [ceph] [29142.403380] RIP: 0010:ceph_check_delayed_caps+0xc7/0x270 [ceph] [29142.408082] ? __file_remove_privs+0x89/0x110 [29142.412687] Code: 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 48 8b 85 78 04 00 00 <49> 39 c5 75 89 31 c0 48 8b 3c 24 48 89 44 24 08 e8 84 5b 07 e1 48 [29142.414178] ? wake_bit_function+0x70/0x70 [29142.418703] RSP: 0018:ffffc90002117db8 EFLAGS: 00000246 [29142.424616] ? ceph_write_iter+0x260/0x770 [ceph] [29142.428964] [29142.428965] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [29142.447703] ceph_write_iter+0x260/0x770 [ceph] [29142.451793] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [29142.457015] ? lock_is_held_type+0xaa/0x120 [29142.461707] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [29142.463200] vfs_write+0x324/0x4f0 [29142.470322] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [29142.474849] ksys_pwrite64+0x65/0xa0 [29142.481971] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29142.486149] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29142.493271] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29142.496669] do_syscall_64+0x38/0x80 [29142.503793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29142.507366] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29142.514488] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29142.519619] RIP: 0033:0x7f48f0eeb93b [29142.527698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29142.531268] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29142.537005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29142.542049] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29142.549172] Call Trace: [29142.549174] [29142.552744] ORIG_RAX: 0000000000000012 [29142.559868] ? watchdog_timer_fn+0x1b4/0x220 [29142.578604] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29142.585730] ? __hrtimer_run_queues+0x1d3/0x360 [29142.590960] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29142.593411] ? hrtimer_interrupt+0x104/0x240 [29142.595418] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29142.599254] ? __local_bh_enable+0x50/0x60 [29142.603523] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29142.610650] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29142.615169] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29142.622296] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29142.626564] [29142.633683] [29142.666496] [29142.668592] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29142.673901] ? ceph_check_delayed_caps+0xc7/0x270 [ceph] [29142.679219] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [29142.684552] delayed_work+0x1d8/0x280 [ceph] [29142.688834] ? process_scheduled_works+0x22a/0x510 [29142.693624] process_scheduled_works+0x2a4/0x510 [29142.698254] worker_thread+0x188/0x340 [29142.702005] ? rescuer_thread+0x310/0x310 [29142.706008] kthread+0xf5/0x130 [29142.709155] ? kthread_complete_and_exit+0x20/0x20 [29142.713940] ret_from_fork+0x31/0x40 [29142.717536] ? kthread_complete_and_exit+0x20/0x20 [29142.722320] ret_from_fork_asm+0x11/0x20 [29142.726253] [29169.969069] watchdog: BUG: soft lockup - CPU#0 stuck for 16052s! [dbench:78683] [29169.976369] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29169.981069] watchdog: BUG: soft lockup - CPU#7 stuck for 25049s! [kworker/7:4:78808] [29170.031808] irq event stamp: 48669232 [29170.031809] hardirqs last enabled at (48669231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29170.039540] Modules linked in: ceph [29170.043198] hardirqs last disabled at (48669232): [] sysvec_apic_timer_interrupt+0xe/0x90 [29170.053354] libceph [29170.056839] softirqs last enabled at (48669184): [] __do_softirq+0x2be/0x371 [29170.066564] dns_resolver fscache [29170.068746] softirqs last disabled at (48669179): [] irq_exit_rcu+0xb0/0xd0 [29170.077431] netfs [29170.080759] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29170.089269] veth nft_chain_nat [29170.091281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29170.100224] xt_MASQUERADE nf_nat [29170.103362] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29170.110833] nf_conntrack nf_defrag_ipv6 [29170.114143] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29170.119621] nf_defrag_ipv4 nft_compat [29170.123539] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29170.142275] nf_tables nfnetlink [29170.146020] [29170.146021] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29170.151238] bridge stp [29170.154465] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29170.155954] llc xfs [29170.163077] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29170.165522] libcrc32c sunrpc [29170.172645] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29170.174828] intel_rapl_msr [29170.181954] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29170.184917] intel_rapl_common sb_edac [29170.192042] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29170.194833] x86_pkg_temp_thermal [29170.201956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29170.205700] coretemp kvm_intel [29170.213779] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29170.217089] kvm irqbypass [29170.222827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29170.225964] i2c_i801 iTCO_wdt [29170.233087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29170.235792] iTCO_vendor_support i2c_smbus [29170.242915] Call Trace: [29170.242917] [29170.245966] crct10dif_pclmul joydev [29170.253092] ? watchdog_timer_fn+0x1b4/0x220 [29170.257181] crc32_pclmul ipmi_ssif [29170.259627] ? __hrtimer_run_queues+0x1d3/0x360 [29170.261637] ghash_clmulni_intel mei_me [29170.265210] ? hrtimer_interrupt+0x104/0x240 [29170.269470] mei lpc_ich [29170.272956] ? __local_bh_enable+0x50/0x60 [29170.277479] sha512_ssse3 acpi_ipmi [29170.281313] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29170.285573] mfd_core wmi [29170.288104] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29170.292194] ipmi_si ipmi_devintf [29170.295679] [29170.300897] ipmi_msghandler ioatdma [29170.303514] [29170.308471] acpi_pad [29170.311783] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29170.313879] acpi_power_meter ixgbe [29170.317471] ? queued_spin_lock_slowpath+0x16/0x2e0 [29170.319565] igb mdio [29170.321837] do_raw_spin_lock+0xb5/0xc0 [29170.327140] crc32c_intel [29170.330624] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29170.335494] nvme [29170.337767] try_get_cap_refs+0x159/0x880 [ceph] [29170.341595] ptp nvme_core [29170.344218] __ceph_get_caps+0x4a1/0x760 [ceph] [29170.348997] i2c_algo_bit pps_core [29170.350926] ? __file_remove_privs+0x89/0x110 [29170.355533] dca fuse [29170.358237] ? wake_bit_function+0x70/0x70 [29170.362760] [29170.366161] ? ceph_write_iter+0x260/0x770 [ceph] [29170.370509] irq event stamp: 154938054 [29170.372779] ceph_write_iter+0x260/0x770 [ceph] [29170.376869] hardirqs last enabled at (154938053): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29170.378364] ? lock_is_held_type+0xaa/0x120 [29170.383057] hardirqs last disabled at (154938054): [] sysvec_apic_timer_interrupt+0xe/0x90 [29170.386805] vfs_write+0x324/0x4f0 [29170.391326] softirqs last enabled at (154938052): [] __do_softirq+0x2be/0x371 [29170.401576] ksys_pwrite64+0x65/0xa0 [29170.405763] softirqs last disabled at (154938047): [] irq_exit_rcu+0xb0/0xd0 [29170.415575] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29170.418974] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29170.427752] do_syscall_64+0x38/0x80 [29170.431323] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29170.439923] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29170.445052] Workqueue: events delayed_work [ceph] [29170.454428] RIP: 0033:0x7f48f0eeb93b [29170.458001] [29170.465471] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29170.470516] RIP: 0010:lock_release+0x14e/0x260 [29170.475213] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29170.478782] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [29170.480273] ORIG_RAX: 0000000000000012 [29170.499011] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [29170.503450] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29170.508684] [29170.508685] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [29170.527422] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29170.531253] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [29170.536472] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29170.543595] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [29170.545083] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29170.552208] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [29170.559333] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29170.566473] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [29170.573604] [29170.580722] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29170.619482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29170.625217] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29170.632342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29170.639476] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29170.646599] Call Trace: [29170.649045] [29170.651055] ? watchdog_timer_fn+0x1b4/0x220 [29170.655318] ? __hrtimer_run_queues+0x1d3/0x360 [29170.659846] ? hrtimer_interrupt+0x104/0x240 [29170.664114] ? __local_bh_enable+0x50/0x60 [29170.668209] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29170.673434] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29170.678390] [29170.680486] [29170.682585] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29170.687888] ? igrab+0x43/0x50 [29170.690941] ? lock_release+0x14e/0x260 [29170.694771] ? lock_release+0xc5/0x260 [29170.698515] _raw_spin_unlock+0x1b/0x40 [29170.702345] igrab+0x43/0x50 [29170.705224] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29170.710373] delayed_work+0x1d8/0x280 [ceph] [29170.714653] ? process_scheduled_works+0x22a/0x510 [29170.719443] process_scheduled_works+0x2a4/0x510 [29170.724060] worker_thread+0x188/0x340 [29170.727807] ? rescuer_thread+0x310/0x310 [29170.731811] kthread+0xf5/0x130 [29170.734949] ? kthread_complete_and_exit+0x20/0x20 [29170.739734] ret_from_fork+0x31/0x40 [29170.743321] ? kthread_complete_and_exit+0x20/0x20 [29170.748105] ret_from_fork_asm+0x11/0x20 [29170.752029] [29197.969580] watchdog: BUG: soft lockup - CPU#0 stuck for 16078s! [dbench:78683] [29197.976882] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29197.981580] watchdog: BUG: soft lockup - CPU#7 stuck for 25075s! [kworker/7:4:78808] [29198.032331] irq event stamp: 48727350 [29198.032332] hardirqs last enabled at (48727349): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29198.040062] Modules linked in: [29198.043720] hardirqs last disabled at (48727350): [] sysvec_apic_timer_interrupt+0xe/0x90 [29198.053877] ceph [29198.056928] softirqs last enabled at (48727302): [] __do_softirq+0x2be/0x371 [29198.066653] libceph [29198.068576] softirqs last disabled at (48727297): [] irq_exit_rcu+0xb0/0xd0 [29198.077260] dns_resolver [29198.079446] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29198.087972] fscache [29198.090590] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29198.099534] netfs [29198.101718] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29198.109190] veth nft_chain_nat [29198.111200] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29198.116678] xt_MASQUERADE [29198.119814] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29198.138551] nf_nat [29198.141268] [29198.146483] nf_conntrack [29198.148580] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29198.150069] nf_defrag_ipv6 nf_defrag_ipv4 [29198.152689] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29198.159811] nft_compat [29198.163903] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29198.171027] nf_tables [29198.173471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29198.180595] nfnetlink [29198.182953] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29198.190095] bridge stp [29198.192452] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29198.199576] llc [29198.202019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29198.210096] xfs [29198.211933] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29198.217672] libcrc32c [29198.219508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29198.226634] sunrpc intel_rapl_msr [29198.228991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29198.236114] intel_rapl_common [29198.239512] Call Trace: [29198.239514] [29198.246636] sb_edac x86_pkg_temp_thermal [29198.249690] ? watchdog_timer_fn+0x1b4/0x220 [29198.252130] coretemp kvm_intel [29198.254143] ? __hrtimer_run_queues+0x1d3/0x360 [29198.258145] kvm irqbypass [29198.262413] ? hrtimer_interrupt+0x104/0x240 [29198.265548] i2c_i801 iTCO_wdt [29198.270072] ? __local_bh_enable+0x50/0x60 [29198.272775] iTCO_vendor_support i2c_smbus [29198.277042] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29198.280090] crct10dif_pclmul joydev [29198.284181] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29198.288271] crc32_pclmul ipmi_ssif [29198.293491] [29198.297059] ghash_clmulni_intel [29198.302017] [29198.305501] mei_me mei [29198.307600] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29198.310823] lpc_ich [29198.312925] ? queued_spin_lock_slowpath+0x18/0x2e0 [29198.315364] sha512_ssse3 acpi_ipmi [29198.320671] do_raw_spin_lock+0xb5/0xc0 [29198.322852] mfd_core wmi [29198.327726] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29198.331207] ipmi_si ipmi_devintf [29198.335040] try_get_cap_refs+0x159/0x880 [ceph] [29198.337655] ipmi_msghandler ioatdma [29198.342445] __ceph_get_caps+0x4a1/0x760 [ceph] [29198.345751] acpi_pad acpi_power_meter [29198.350366] ? __file_remove_privs+0x89/0x110 [29198.353931] ixgbe igb [29198.358455] ? wake_bit_function+0x70/0x70 [29198.362200] mdio crc32c_intel [29198.366553] ? ceph_write_iter+0x260/0x770 [ceph] [29198.368908] nvme ptp [29198.372998] ceph_write_iter+0x260/0x770 [ceph] [29198.376048] nvme_core [29198.380750] ? lock_is_held_type+0xaa/0x120 [29198.383017] i2c_algo_bit pps_core [29198.387545] vfs_write+0x324/0x4f0 [29198.389899] dca fuse [29198.394080] ksys_pwrite64+0x65/0xa0 [29198.397475] [29198.400873] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29198.403141] irq event stamp: 155159008 [29198.406713] do_syscall_64+0x38/0x80 [29198.408204] hardirqs last enabled at (155159007): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29198.413336] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29198.417078] hardirqs last disabled at (155159008): [] sysvec_apic_timer_interrupt+0xe/0x90 [29198.420650] RIP: 0033:0x7f48f0eeb93b [29198.430893] softirqs last enabled at (155159006): [] __do_softirq+0x2be/0x371 [29198.435937] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29198.445748] softirqs last disabled at (155159001): [] irq_exit_rcu+0xb0/0xd0 [29198.449319] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29198.458090] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29198.476826] ORIG_RAX: 0000000000000012 [29198.485424] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29198.490643] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29198.500021] Workqueue: events delayed_work [ceph] [29198.503850] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29198.511322] [29198.511323] RIP: 0010:lock_acquire+0xf0/0x2b0 [29198.518444] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29198.523142] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29198.530267] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29198.531758] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29198.536107] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29198.543232] [29198.543233] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29198.561988] [29198.569103] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29198.599386] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29198.606509] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29198.613633] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29198.620758] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29198.628836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29198.634572] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29198.641696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29198.648820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29198.655944] Call Trace: [29198.658390] [29198.660399] ? watchdog_timer_fn+0x1b4/0x220 [29198.664666] ? __hrtimer_run_queues+0x1d3/0x360 [29198.669192] ? hrtimer_interrupt+0x104/0x240 [29198.673460] ? __local_bh_enable+0x50/0x60 [29198.677554] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29198.682777] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29198.687735] [29198.689833] [29198.691929] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29198.697239] ? lock_acquire+0xf0/0x2b0 [29198.701004] ? igrab+0x43/0x50 [29198.704056] ? lock_release+0xc5/0x260 [29198.707801] _raw_spin_lock+0x30/0x40 [29198.711466] ? igrab+0x1d/0x50 [29198.714516] igrab+0x1d/0x50 [29198.717396] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29198.722543] delayed_work+0x1d8/0x280 [ceph] [29198.726825] ? process_scheduled_works+0x22a/0x510 [29198.731614] process_scheduled_works+0x2a4/0x510 [29198.736232] worker_thread+0x188/0x340 [29198.739988] ? rescuer_thread+0x310/0x310 [29198.743992] kthread+0xf5/0x130 [29198.747138] ? kthread_complete_and_exit+0x20/0x20 [29198.751921] ret_from_fork+0x31/0x40 [29198.755503] ? kthread_complete_and_exit+0x20/0x20 [29198.760293] ret_from_fork_asm+0x11/0x20 [29198.764219] [29199.984983] rcu: INFO: rcu_preempt self-detected stall on CPU [29199.990740] rcu: 7-....: (16762640 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/16761331 fqs=8440551 [29200.001330] rcu: (t=17263068 jiffies g=1309609 q=517188 ncpus=8) [29200.007417] Sending NMI from CPU 7 to CPUs 0: [29200.011768] NMI backtrace for cpu 0 [29200.011770] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29200.011772] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29200.011772] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29200.011776] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29200.011777] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29200.011779] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29200.011779] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29200.011780] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29200.011781] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29200.011782] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29200.011782] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29200.011784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29200.011784] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29200.011785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29200.011786] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29200.011787] Call Trace: [29200.011788] [29200.011789] ? nmi_cpu_backtrace+0xad/0x120 [29200.011792] ? nmi_cpu_backtrace_handler+0x11/0x20 [29200.011794] ? nmi_handle+0xd4/0x260 [29200.011798] ? default_do_nmi+0x49/0xf0 [29200.011800] ? exc_nmi+0x11e/0x150 [29200.011801] ? end_repeat_nmi+0x16/0x67 [29200.011805] ? queued_spin_lock_slowpath+0x18/0x2e0 [29200.011808] ? queued_spin_lock_slowpath+0x18/0x2e0 [29200.011811] ? queued_spin_lock_slowpath+0x18/0x2e0 [29200.011813] [29200.011813] [29200.011814] do_raw_spin_lock+0xb5/0xc0 [29200.011817] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29200.011831] try_get_cap_refs+0x159/0x880 [ceph] [29200.011846] __ceph_get_caps+0x4a1/0x760 [ceph] [29200.011859] ? __file_remove_privs+0x89/0x110 [29200.011861] ? wake_bit_function+0x70/0x70 [29200.011863] ? ceph_write_iter+0x260/0x770 [ceph] [29200.011874] ceph_write_iter+0x260/0x770 [ceph] [29200.011885] ? lock_is_held_type+0xaa/0x120 [29200.011888] vfs_write+0x324/0x4f0 [29200.011892] ksys_pwrite64+0x65/0xa0 [29200.011894] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29200.011897] do_syscall_64+0x38/0x80 [29200.011899] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29200.011901] RIP: 0033:0x7f48f0eeb93b [29200.011902] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29200.011903] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [29200.011904] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29200.011905] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29200.011906] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29200.011907] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29200.011907] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29200.011910] [29200.012767] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29200.316327] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29200.323798] Workqueue: events delayed_work [ceph] [29200.328512] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [29200.334440] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [29200.353179] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [29200.358396] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [29200.365519] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [29200.372645] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [29200.379767] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [29200.386892] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29200.394015] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29200.402093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29200.407831] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29200.414956] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29200.422079] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29200.429204] Call Trace: [29200.431647] [29200.433659] ? rcu_dump_cpu_stacks+0xd1/0x180 [29200.438018] ? rcu_sched_clock_irq+0x633/0x1390 [29200.442542] ? lock_release+0xc5/0x260 [29200.446285] ? lock_is_held_type+0xaa/0x120 [29200.450465] ? lock_is_held_type+0xaa/0x120 [29200.454642] ? tick_sched_do_timer+0xa0/0xa0 [29200.458913] ? rcu_read_lock_sched_held+0x56/0x90 [29200.463613] ? tick_sched_do_timer+0xa0/0xa0 [29200.467875] ? update_process_times+0x5f/0x90 [29200.472225] ? tick_sched_handle.isra.16+0x1e/0x50 [29200.477011] ? tick_sched_timer+0x6f/0x80 [29200.481014] ? __hrtimer_run_queues+0x1d3/0x360 [29200.485541] ? hrtimer_interrupt+0x104/0x240 [29200.489811] ? __local_bh_enable+0x50/0x60 [29200.493903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29200.499128] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29200.504084] [29200.506181] [29200.508280] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29200.513588] ? ceph_check_delayed_caps+0x9e/0x270 [ceph] [29200.518907] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [29200.524221] delayed_work+0x1d8/0x280 [ceph] [29200.528502] ? process_scheduled_works+0x22a/0x510 [29200.533292] process_scheduled_works+0x2a4/0x510 [29200.537908] worker_thread+0x188/0x340 [29200.541656] ? rescuer_thread+0x310/0x310 [29200.545661] kthread+0xf5/0x130 [29200.548798] ? kthread_complete_and_exit+0x20/0x20 [29200.553582] ret_from_fork+0x31/0x40 [29200.557151] ? kthread_complete_and_exit+0x20/0x20 [29200.561936] ret_from_fork_asm+0x11/0x20 [29200.565861] [29225.970091] watchdog: BUG: soft lockup - CPU#0 stuck for 16104s! [dbench:78683] [29225.977395] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29225.982091] watchdog: BUG: soft lockup - CPU#7 stuck for 25101s! [kworker/7:4:78808] [29226.032838] irq event stamp: 48785536 [29226.032839] hardirqs last enabled at (48785535): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29226.040568] Modules linked in: ceph [29226.044224] hardirqs last disabled at (48785536): [] sysvec_apic_timer_interrupt+0xe/0x90 [29226.054384] libceph [29226.057867] softirqs last enabled at (48785496): [] __do_softirq+0x2be/0x371 [29226.067592] dns_resolver fscache [29226.069777] softirqs last disabled at (48785491): [] irq_exit_rcu+0xb0/0xd0 [29226.078459] netfs veth [29226.081787] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29226.090297] nft_chain_nat xt_MASQUERADE [29226.092742] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29226.101702] nf_nat nf_conntrack [29226.105621] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29226.113092] nf_defrag_ipv6 nf_defrag_ipv4 [29226.116315] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29226.121793] nft_compat [29226.125884] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29226.144622] nf_tables nfnetlink [29226.147066] [29226.147066] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29226.152283] bridge stp [29226.155506] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29226.156998] llc xfs [29226.164121] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29226.166567] libcrc32c sunrpc [29226.173690] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29226.175874] intel_rapl_msr intel_rapl_common [29226.182999] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29226.185963] sb_edac x86_pkg_temp_thermal [29226.193087] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29226.197438] coretemp kvm_intel [29226.204560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29226.208565] kvm irqbypass [29226.216641] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29226.219780] i2c_i801 iTCO_wdt [29226.225518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29226.228221] iTCO_vendor_support i2c_smbus [29226.235346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29226.238397] crct10dif_pclmul joydev [29226.245522] Call Trace: [29226.245523] [29226.249612] crc32_pclmul ipmi_ssif [29226.256737] ? watchdog_timer_fn+0x1b4/0x220 [29226.260306] ghash_clmulni_intel mei_me [29226.262768] ? __hrtimer_run_queues+0x1d3/0x360 [29226.264779] mei lpc_ich [29226.268265] ? hrtimer_interrupt+0x104/0x240 [29226.272526] sha512_ssse3 acpi_ipmi [29226.276358] ? __local_bh_enable+0x50/0x60 [29226.280882] mfd_core wmi [29226.283414] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29226.287675] ipmi_si [29226.291162] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29226.295250] ipmi_devintf ipmi_msghandler [29226.297869] [29226.303085] ioatdma [29226.305269] [29226.310227] acpi_pad acpi_power_meter [29226.314232] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29226.316328] ixgbe igb [29226.318516] ? queued_spin_lock_slowpath+0x16/0x2e0 [29226.320610] mdio crc32c_intel [29226.324357] do_raw_spin_lock+0xb5/0xc0 [29226.329659] nvme ptp [29226.332017] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29226.336885] nvme_core i2c_algo_bit [29226.339940] try_get_cap_refs+0x159/0x880 [ceph] [29226.343770] pps_core dca [29226.346044] __ceph_get_caps+0x4a1/0x760 [ceph] [29226.350822] fuse [29226.354312] ? __file_remove_privs+0x89/0x110 [29226.358918] irq event stamp: 155375002 [29226.361536] ? wake_bit_function+0x70/0x70 [29226.366059] hardirqs last enabled at (155375001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29226.367985] ? ceph_write_iter+0x260/0x770 [ceph] [29226.372334] hardirqs last disabled at (155375002): [] sysvec_apic_timer_interrupt+0xe/0x90 [29226.376079] ceph_write_iter+0x260/0x770 [ceph] [29226.380169] softirqs last enabled at (155375000): [] __do_softirq+0x2be/0x371 [29226.390417] ? lock_is_held_type+0xaa/0x120 [29226.395110] softirqs last disabled at (155374995): [] irq_exit_rcu+0xb0/0xd0 [29226.404923] vfs_write+0x324/0x4f0 [29226.409446] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29226.418219] ksys_pwrite64+0x65/0xa0 [29226.422394] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29226.430991] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29226.434388] Workqueue: events delayed_work [ceph] [29226.443772] do_syscall_64+0x38/0x80 [29226.447344] [29226.447345] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [29226.454816] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29226.459947] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [29226.464644] RIP: 0033:0x7f48f0eeb93b [29226.468214] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [29226.469707] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29226.474316] [29226.474317] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [29226.479362] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29226.498099] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [29226.501670] ORIG_RAX: 0000000000000012 [29226.506885] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [29226.525625] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29226.527115] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [29226.534238] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29226.539455] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29226.546579] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29226.550411] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29226.557536] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29226.564659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29226.571794] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29226.578916] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29226.586045] [29226.593164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29226.637661] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29226.644784] Call Trace: [29226.647228] [29226.649241] ? watchdog_timer_fn+0x1b4/0x220 [29226.653505] ? __hrtimer_run_queues+0x1d3/0x360 [29226.658032] ? hrtimer_interrupt+0x104/0x240 [29226.662301] ? __local_bh_enable+0x50/0x60 [29226.666395] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29226.671616] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29226.676574] [29226.678671] [29226.680776] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29226.686087] ? do_raw_spin_lock+0x42/0xc0 [29226.690097] igrab+0x1d/0x50 [29226.692972] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29226.698124] delayed_work+0x1d8/0x280 [ceph] [29226.702405] ? process_scheduled_works+0x22a/0x510 [29226.707195] process_scheduled_works+0x2a4/0x510 [29226.711808] worker_thread+0x188/0x340 [29226.715559] ? rescuer_thread+0x310/0x310 [29226.719562] kthread+0xf5/0x130 [29226.722699] ? kthread_complete_and_exit+0x20/0x20 [29226.727498] ret_from_fork+0x31/0x40 [29226.731070] ? kthread_complete_and_exit+0x20/0x20 [29226.735856] ret_from_fork_asm+0x11/0x20 [29226.739782] [29253.970602] watchdog: BUG: soft lockup - CPU#0 stuck for 16130s! [dbench:78683] [29253.977910] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29253.982601] watchdog: BUG: soft lockup - CPU#7 stuck for 25127s! [kworker/7:4:78808] [29254.033360] irq event stamp: 48843644 [29254.041092] Modules linked in: ceph libceph dns_resolver fscache [29254.044746] hardirqs last enabled at (48843643): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29254.044751] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [29254.060903] hardirqs last disabled at (48843644): [] sysvec_apic_timer_interrupt+0xe/0x90 [29254.066381] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [29254.076103] softirqs last enabled at (48843606): [] __do_softirq+0x2be/0x371 [29254.083140] nfnetlink bridge stp llc xfs libcrc32c sunrpc [29254.091825] softirqs last disabled at (48843601): [] irq_exit_rcu+0xb0/0xd0 [29254.097302] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [29254.105815] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29254.105817] kvm_intel [29254.105818] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29254.113457] kvm irqbypass [29254.122401] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29254.124759] i2c_i801 [29254.132231] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29254.134933] iTCO_wdt iTCO_vendor_support [29254.140412] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29254.142682] i2c_smbus crct10dif_pclmul [29254.161419] [29254.161420] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29254.165424] joydev [29254.170643] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29254.174473] crc32_pclmul [29254.175964] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29254.183086] ipmi_ssif ghash_clmulni_intel [29254.185183] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29254.192309] mei_me [29254.194926] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29254.202051] mei lpc_ich [29254.206141] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29254.213277] sha512_ssse3 acpi_ipmi [29254.215372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29254.222496] mfd_core wmi [29254.225027] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29254.233104] ipmi_si ipmi_devintf [29254.236587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29254.242325] ipmi_msghandler [29254.244941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29254.252067] ioatdma acpi_pad [29254.255378] Call Trace: [29254.255380] [29254.262502] acpi_power_meter ixgbe [29254.265381] ? watchdog_timer_fn+0x1b4/0x220 [29254.272503] igb mdio [29254.275468] ? __hrtimer_run_queues+0x1d3/0x360 [29254.277912] crc32c_intel nvme [29254.279926] ? hrtimer_interrupt+0x104/0x240 [29254.283407] ptp nvme_core [29254.287670] ? __local_bh_enable+0x50/0x60 [29254.289940] i2c_algo_bit pps_core [29254.294468] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29254.297514] dca fuse [29254.301780] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29254.304483] [29254.304483] irq event stamp: 155595884 [29254.308576] [29254.311970] hardirqs last enabled at (155595883): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29254.317189] [29254.317190] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29254.319458] hardirqs last disabled at (155595884): [] sysvec_apic_timer_interrupt+0xe/0x90 [29254.324422] ? queued_spin_lock_slowpath+0x16/0x2e0 [29254.325908] softirqs last enabled at (155595882): [] __do_softirq+0x2be/0x371 [29254.329653] do_raw_spin_lock+0xb5/0xc0 [29254.331750] softirqs last disabled at (155595877): [] irq_exit_rcu+0xb0/0xd0 [29254.342003] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29254.344101] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29254.349408] try_get_cap_refs+0x159/0x880 [ceph] [29254.359215] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29254.364092] __ceph_get_caps+0x4a1/0x760 [ceph] [29254.372857] Workqueue: events delayed_work [ceph] [29254.376690] ? __file_remove_privs+0x89/0x110 [29254.385291] [29254.385292] RIP: 0010:lock_acquire+0x10c/0x2b0 [29254.390079] ? wake_bit_function+0x70/0x70 [29254.399456] Code: 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 c4 38 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 65 8b 05 b2 e6 eb 7e 89 c0 48 0f a3 05 74 bc [29254.404069] ? ceph_write_iter+0x260/0x770 [ceph] [29254.411537] RSP: 0018:ffffc90002117d68 EFLAGS: 00000296 [29254.416061] ceph_write_iter+0x260/0x770 [ceph] [29254.420757] [29254.420758] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [29254.425113] ? lock_is_held_type+0xaa/0x120 [29254.426598] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29254.431039] vfs_write+0x324/0x4f0 [29254.435126] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [29254.453870] ksys_pwrite64+0x65/0xa0 [29254.458563] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [29254.463781] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29254.468304] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29254.469796] do_syscall_64+0x38/0x80 [29254.476919] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29254.481097] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29254.488221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29254.491617] RIP: 0033:0x7f48f0eeb93b [29254.498741] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29254.502314] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29254.509438] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29254.514568] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29254.521692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29254.525264] ORIG_RAX: 0000000000000012 [29254.533340] Call Trace: [29254.538383] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29254.544123] [29254.547692] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29254.554817] ? watchdog_timer_fn+0x1b4/0x220 [29254.573553] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29254.580678] ? __hrtimer_run_queues+0x1d3/0x360 [29254.585896] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29254.593024] ? hrtimer_interrupt+0x104/0x240 [29254.596851] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29254.599293] ? __local_bh_enable+0x50/0x60 [29254.606425] [29254.608431] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29254.661471] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29254.666428] [29254.668525] [29254.670624] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29254.675931] ? lock_acquire+0x10c/0x2b0 [29254.679770] _raw_spin_lock+0x30/0x40 [29254.683434] ? igrab+0x1d/0x50 [29254.686484] igrab+0x1d/0x50 [29254.689362] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29254.694513] delayed_work+0x1d8/0x280 [ceph] [29254.698794] ? process_scheduled_works+0x22a/0x510 [29254.703582] process_scheduled_works+0x2a4/0x510 [29254.708200] worker_thread+0x188/0x340 [29254.711948] ? rescuer_thread+0x310/0x310 [29254.715950] kthread+0xf5/0x130 [29254.719089] ? kthread_complete_and_exit+0x20/0x20 [29254.723872] ret_from_fork+0x31/0x40 [29254.727452] ? kthread_complete_and_exit+0x20/0x20 [29254.732235] ret_from_fork_asm+0x11/0x20 [29254.736158] [29281.971112] watchdog: BUG: soft lockup - CPU#0 stuck for 16156s! [dbench:78683] [29281.978414] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29281.983112] watchdog: BUG: soft lockup - CPU#7 stuck for 25153s! [kworker/7:4:78808] [29282.033857] irq event stamp: 48901734 [29282.033858] hardirqs last enabled at (48901733): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29282.041586] Modules linked in: ceph [29282.045244] hardirqs last disabled at (48901734): [] sysvec_apic_timer_interrupt+0xe/0x90 [29282.055400] libceph [29282.058885] softirqs last enabled at (48901704): [] __do_softirq+0x2be/0x371 [29282.068610] dns_resolver fscache [29282.070793] softirqs last disabled at (48901699): [] irq_exit_rcu+0xb0/0xd0 [29282.079479] netfs veth [29282.082806] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29282.091316] nft_chain_nat xt_MASQUERADE [29282.093761] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29282.102704] nf_nat nf_conntrack [29282.106621] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29282.114093] nf_defrag_ipv6 nf_defrag_ipv4 [29282.117318] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29282.122801] nft_compat nf_tables [29282.126893] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29282.145633] nfnetlink bridge [29282.148942] [29282.148943] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29282.154160] stp [29282.157125] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29282.158614] llc xfs [29282.165739] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29282.167577] libcrc32c sunrpc [29282.174700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29282.176883] intel_rapl_msr intel_rapl_common [29282.184009] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29282.186971] sb_edac [29282.194096] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29282.198447] x86_pkg_temp_thermal coretemp [29282.205572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29282.207754] kvm_intel kvm [29282.215833] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29282.219925] irqbypass i2c_i801 [29282.225661] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29282.228365] iTCO_wdt iTCO_vendor_support [29282.235506] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29282.238644] i2c_smbus crct10dif_pclmul [29282.245769] Call Trace: [29282.249792] joydev crc32_pclmul [29282.256914] [29282.260745] ipmi_ssif ghash_clmulni_intel [29282.263189] ? watchdog_timer_fn+0x1b4/0x220 [29282.266412] mei_me mei [29282.268424] ? __hrtimer_run_queues+0x1d3/0x360 [29282.272514] lpc_ich [29282.276798] ? hrtimer_interrupt+0x104/0x240 [29282.279240] sha512_ssse3 acpi_ipmi [29282.283764] ? __local_bh_enable+0x50/0x60 [29282.285964] mfd_core wmi [29282.290232] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29282.293714] ipmi_si [29282.297809] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29282.300420] ipmi_devintf [29282.305638] [29282.307823] ipmi_msghandler ioatdma [29282.312779] [29282.312781] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29282.315398] acpi_pad acpi_power_meter [29282.317514] ? queued_spin_lock_slowpath+0x18/0x2e0 [29282.321082] ixgbe igb [29282.323183] do_raw_spin_lock+0xb5/0xc0 [29282.328485] mdio crc32c_intel [29282.332247] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29282.337117] nvme ptp [29282.339475] try_get_cap_refs+0x159/0x880 [ceph] [29282.343305] nvme_core i2c_algo_bit [29282.346361] __ceph_get_caps+0x4a1/0x760 [ceph] [29282.351140] pps_core [29282.353414] ? __file_remove_privs+0x89/0x110 [29282.358020] dca fuse [29282.361513] ? wake_bit_function+0x70/0x70 [29282.366038] [29282.366039] irq event stamp: 155816966 [29282.368310] ? ceph_write_iter+0x260/0x770 [ceph] [29282.372660] hardirqs last enabled at (155816965): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29282.374930] ceph_write_iter+0x260/0x770 [ceph] [29282.379020] hardirqs last disabled at (155816966): [] sysvec_apic_timer_interrupt+0xe/0x90 [29282.380517] ? lock_is_held_type+0xaa/0x120 [29282.384254] softirqs last enabled at (155816964): [] __do_softirq+0x2be/0x371 [29282.388956] vfs_write+0x324/0x4f0 [29282.399197] softirqs last disabled at (155816959): [] irq_exit_rcu+0xb0/0xd0 [29282.403725] ksys_pwrite64+0x65/0xa0 [29282.413532] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29282.417709] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29282.426481] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29282.429879] do_syscall_64+0x38/0x80 [29282.438476] Workqueue: events delayed_work [ceph] [29282.442047] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29282.451423] [29282.451424] RIP: 0010:lock_acquire+0xf0/0x2b0 [29282.456554] RIP: 0033:0x7f48f0eeb93b [29282.464024] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29282.467596] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29282.472294] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29282.477337] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29282.478827] [29282.478828] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29282.483179] ORIG_RAX: 0000000000000012 [29282.486748] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29282.505487] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29282.524241] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29282.529460] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29282.534676] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29282.536167] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29282.543293] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29282.547122] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29282.554247] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29282.561369] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29282.568512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29282.575641] [29282.582761] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29282.634381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29282.641517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29282.648646] Call Trace: [29282.651091] [29282.653102] ? watchdog_timer_fn+0x1b4/0x220 [29282.657366] ? __hrtimer_run_queues+0x1d3/0x360 [29282.661893] ? hrtimer_interrupt+0x104/0x240 [29282.666162] ? __local_bh_enable+0x50/0x60 [29282.670255] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29282.675480] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29282.680436] [29282.682532] [29282.684631] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29282.689938] ? lock_acquire+0xf0/0x2b0 [29282.693688] ? igrab+0x43/0x50 [29282.696740] ? lock_release+0xc5/0x260 [29282.700483] _raw_spin_lock+0x30/0x40 [29282.704140] ? igrab+0x1d/0x50 [29282.707191] igrab+0x1d/0x50 [29282.710070] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29282.715219] delayed_work+0x1d8/0x280 [ceph] [29282.719518] ? process_scheduled_works+0x22a/0x510 [29282.724308] process_scheduled_works+0x2a4/0x510 [29282.728923] worker_thread+0x188/0x340 [29282.732673] ? rescuer_thread+0x310/0x310 [29282.736675] kthread+0xf5/0x130 [29282.739812] ? kthread_complete_and_exit+0x20/0x20 [29282.744597] ret_from_fork+0x31/0x40 [29282.748168] ? kthread_complete_and_exit+0x20/0x20 [29282.752953] ret_from_fork_asm+0x11/0x20 [29282.756885] [29309.971623] watchdog: BUG: soft lockup - CPU#0 stuck for 16183s! [dbench:78683] [29309.978930] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29309.983623] watchdog: BUG: soft lockup - CPU#7 stuck for 25180s! [kworker/7:4:78808] [29310.034378] irq event stamp: 48959872 [29310.034380] hardirqs last enabled at (48959871): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29310.042109] Modules linked in: [29310.045766] hardirqs last disabled at (48959872): [] sysvec_apic_timer_interrupt+0xe/0x90 [29310.055923] ceph [29310.058973] softirqs last enabled at (48959834): [] __do_softirq+0x2be/0x371 [29310.068699] libceph [29310.070622] softirqs last disabled at (48959829): [] irq_exit_rcu+0xb0/0xd0 [29310.079306] dns_resolver [29310.081492] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29310.090016] fscache [29310.092637] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29310.101580] netfs [29310.103765] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [29310.111235] veth nft_chain_nat [29310.113246] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [29310.118724] xt_MASQUERADE [29310.121862] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29310.140600] nf_nat [29310.143314] [29310.143315] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29310.148528] nf_conntrack nf_defrag_ipv6 [29310.150625] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29310.152116] nf_defrag_ipv4 nft_compat [29310.159241] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29310.163158] nf_tables nfnetlink [29310.170283] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29310.174026] bridge stp [29310.181151] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29310.184376] llc xfs [29310.191500] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29310.193942] libcrc32c sunrpc [29310.201067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29310.203251] intel_rapl_msr intel_rapl_common [29310.211329] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29310.214298] sb_edac x86_pkg_temp_thermal [29310.220039] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29310.224389] coretemp kvm_intel [29310.231514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29310.235518] kvm irqbypass [29310.242643] Call Trace: [29310.242645] [29310.245778] i2c_i801 iTCO_wdt [29310.252906] ? watchdog_timer_fn+0x1b4/0x220 [29310.255607] iTCO_vendor_support i2c_smbus [29310.258053] ? __hrtimer_run_queues+0x1d3/0x360 [29310.260061] crct10dif_pclmul joydev [29310.263116] ? hrtimer_interrupt+0x104/0x240 [29310.267377] crc32_pclmul ipmi_ssif [29310.271468] ? __local_bh_enable+0x50/0x60 [29310.275993] ghash_clmulni_intel mei_me [29310.279564] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29310.283827] mei lpc_ich [29310.287310] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29310.291402] sha512_ssse3 acpi_ipmi [29310.295231] [29310.300451] mfd_core wmi [29310.302981] [29310.307937] ipmi_si ipmi_devintf [29310.311421] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29310.313519] ipmi_msghandler ioatdma [29310.316141] ? queued_spin_lock_slowpath+0x2b/0x2e0 [29310.318235] acpi_pad acpi_power_meter [29310.321547] do_raw_spin_lock+0xb5/0xc0 [29310.326848] ixgbe igb mdio [29310.330422] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29310.335290] crc32c_intel nvme [29310.339056] try_get_cap_refs+0x159/0x880 [ceph] [29310.342883] ptp nvme_core [29310.345679] __ceph_get_caps+0x4a1/0x760 [ceph] [29310.350456] i2c_algo_bit pps_core [29310.353512] ? __file_remove_privs+0x89/0x110 [29310.358119] dca fuse [29310.360824] ? wake_bit_function+0x70/0x70 [29310.365346] [29310.365346] irq event stamp: 156037718 [29310.368746] ? ceph_write_iter+0x260/0x770 [ceph] [29310.373095] hardirqs last enabled at (156037717): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29310.375366] ceph_write_iter+0x260/0x770 [ceph] [29310.379457] hardirqs last disabled at (156037718): [] sysvec_apic_timer_interrupt+0xe/0x90 [29310.380950] ? lock_is_held_type+0xaa/0x120 [29310.384690] softirqs last enabled at (156037716): [] __do_softirq+0x2be/0x371 [29310.389390] vfs_write+0x324/0x4f0 [29310.399633] softirqs last disabled at (156037711): [] irq_exit_rcu+0xb0/0xd0 [29310.404161] ksys_pwrite64+0x65/0xa0 [29310.413967] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29310.418145] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29310.426915] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29310.430316] do_syscall_64+0x38/0x80 [29310.438910] Workqueue: events delayed_work [ceph] [29310.442483] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29310.451858] [29310.451859] RIP: 0010:rcu_is_watching+0x1f/0x40 [29310.456990] RIP: 0033:0x7f48f0eeb93b [29310.464461] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [29310.468033] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29310.472727] RSP: 0018:ffffc90002117d40 EFLAGS: 00000202 [29310.477773] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29310.479264] [29310.479265] RAX: 00000000007c1919 RBX: ffff8881a1d34500 RCX: 000000005553a634 [29310.483786] ORIG_RAX: 0000000000000012 [29310.487357] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [29310.506097] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29310.524833] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [29310.530051] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29310.535269] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [29310.536758] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29310.543885] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29310.547715] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29310.554838] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29310.561963] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29310.569086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29310.576217] [29310.583335] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29310.583336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29310.642080] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29310.649202] Call Trace: [29310.651648] [29310.653659] ? watchdog_timer_fn+0x1b4/0x220 [29310.657922] ? __hrtimer_run_queues+0x1d3/0x360 [29310.662449] ? hrtimer_interrupt+0x104/0x240 [29310.666717] ? __local_bh_enable+0x50/0x60 [29310.670812] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29310.676037] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29310.680992] [29310.683091] [29310.685189] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29310.690493] ? igrab+0x43/0x50 [29310.693544] ? igrab+0x43/0x50 [29310.696593] ? rcu_is_watching+0x1f/0x40 [29310.700512] lock_release+0x1d7/0x260 [29310.704172] _raw_spin_unlock+0x1b/0x40 [29310.708027] igrab+0x43/0x50 [29310.710914] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29310.716062] delayed_work+0x1d8/0x280 [ceph] [29310.720344] ? process_scheduled_works+0x22a/0x510 [29310.725132] process_scheduled_works+0x2a4/0x510 [29310.729749] worker_thread+0x188/0x340 [29310.733498] ? rescuer_thread+0x310/0x310 [29310.737502] kthread+0xf5/0x130 [29310.740640] ? kthread_complete_and_exit+0x20/0x20 [29310.745424] ret_from_fork+0x31/0x40 [29310.749004] ? kthread_complete_and_exit+0x20/0x20 [29310.753811] ret_from_fork_asm+0x11/0x20 [29310.757735] [29337.972133] watchdog: BUG: soft lockup - CPU#0 stuck for 16209s! [dbench:78683] [29337.979433] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29337.984133] watchdog: BUG: soft lockup - CPU#7 stuck for 25206s! [kworker/7:4:78808] [29338.034875] irq event stamp: 49018064 [29338.042605] Modules linked in: ceph libceph dns_resolver [29338.046262] hardirqs last enabled at (49018063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29338.051567] fscache netfs veth nft_chain_nat [29338.061724] hardirqs last disabled at (49018064): [] sysvec_apic_timer_interrupt+0xe/0x90 [29338.066076] xt_MASQUERADE nf_nat nf_conntrack [29338.075807] softirqs last enabled at (49018026): [] __do_softirq+0x2be/0x371 [29338.075809] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [29338.088929] softirqs last disabled at (49018021): [] irq_exit_rcu+0xb0/0xd0 [29338.096314] stp llc xfs [29338.104825] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29338.107355] libcrc32c sunrpc [29338.116300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29338.119265] intel_rapl_msr intel_rapl_common [29338.126734] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29338.126735] sb_edac x86_pkg_temp_thermal coretemp [29338.136562] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29338.136563] kvm_intel kvm [29338.141345] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29338.160085] irqbypass i2c_i801 [29338.162789] [29338.162790] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29338.168006] iTCO_wdt iTCO_vendor_support [29338.171142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29338.172634] i2c_smbus crct10dif_pclmul [29338.179758] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29338.183761] joydev crc32_pclmul [29338.190886] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29338.194717] ipmi_ssif ghash_clmulni_intel [29338.201840] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29338.205065] mei_me mei [29338.212189] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29338.216280] lpc_ich sha512_ssse3 [29338.223403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29338.225849] acpi_ipmi mfd_core [29338.233926] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29338.237235] wmi ipmi_si [29338.242973] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29338.246111] ipmi_devintf [29338.253236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29338.255766] ipmi_msghandler ioatdma [29338.262890] Call Trace: [29338.265508] acpi_pad [29338.272649] [29338.276219] acpi_power_meter [29338.278665] ? watchdog_timer_fn+0x1b4/0x220 [29338.280936] ixgbe igb [29338.282945] ? __hrtimer_run_queues+0x1d3/0x360 [29338.285909] mdio crc32c_intel [29338.290176] ? hrtimer_interrupt+0x104/0x240 [29338.292532] nvme ptp [29338.297055] ? __local_bh_enable+0x50/0x60 [29338.300106] nvme_core i2c_algo_bit [29338.304371] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29338.306641] pps_core dca [29338.310731] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29338.314215] fuse [29338.319432] [29338.322051] irq event stamp: 156258578 [29338.327008] [29338.328932] hardirqs last enabled at (156258577): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29338.331029] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29338.334772] hardirqs last disabled at (156258578): [] sysvec_apic_timer_interrupt+0xe/0x90 [29338.336873] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29338.347114] softirqs last enabled at (156258576): [] __do_softirq+0x2be/0x371 [29338.352421] do_raw_spin_lock+0xb5/0xc0 [29338.362229] softirqs last disabled at (156258571): [] irq_exit_rcu+0xb0/0xd0 [29338.367102] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29338.375872] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29338.379705] try_get_cap_refs+0x159/0x880 [ceph] [29338.388298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29338.393089] __ceph_get_caps+0x4a1/0x760 [ceph] [29338.402461] Workqueue: events delayed_work [ceph] [29338.407077] ? __file_remove_privs+0x89/0x110 [29338.414541] [29338.419068] ? wake_bit_function+0x70/0x70 [29338.423763] RIP: 0010:lock_acquire+0xf0/0x2b0 [29338.428118] ? ceph_write_iter+0x260/0x770 [ceph] [29338.429606] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29338.433696] ceph_write_iter+0x260/0x770 [ceph] [29338.438047] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29338.442747] ? lock_is_held_type+0xaa/0x120 [29338.461481] [29338.461482] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29338.466008] vfs_write+0x324/0x4f0 [29338.471222] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29338.475404] ksys_pwrite64+0x65/0xa0 [29338.476890] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29338.484016] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29338.487412] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29338.494539] do_syscall_64+0x38/0x80 [29338.498107] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29338.505232] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29338.510363] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29338.517487] RIP: 0033:0x7f48f0eeb93b [29338.521058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29338.528181] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29338.533226] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29338.541303] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29338.544875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29338.550611] ORIG_RAX: 0000000000000012 [29338.569350] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29338.576474] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29338.581690] Call Trace: [29338.581691] [29338.588829] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29338.592663] ? watchdog_timer_fn+0x1b4/0x220 [29338.599785] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29338.606913] ? __hrtimer_run_queues+0x1d3/0x360 [29338.609354] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29338.611371] ? hrtimer_interrupt+0x104/0x240 [29338.618490] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29338.622754] ? __local_bh_enable+0x50/0x60 [29338.629882] [29338.634405] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29338.664418] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29338.669373] [29338.671471] [29338.673569] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29338.678886] ? lock_acquire+0xf0/0x2b0 [29338.682634] ? igrab+0x43/0x50 [29338.685685] ? lock_release+0xc5/0x260 [29338.689431] _raw_spin_lock+0x30/0x40 [29338.693096] ? igrab+0x1d/0x50 [29338.696146] igrab+0x1d/0x50 [29338.699025] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29338.704174] delayed_work+0x1d8/0x280 [ceph] [29338.708454] ? process_scheduled_works+0x22a/0x510 [29338.713245] process_scheduled_works+0x2a4/0x510 [29338.717860] worker_thread+0x188/0x340 [29338.721610] ? rescuer_thread+0x310/0x310 [29338.725613] kthread+0xf5/0x130 [29338.728750] ? kthread_complete_and_exit+0x20/0x20 [29338.733546] ret_from_fork+0x31/0x40 [29338.737121] ? kthread_complete_and_exit+0x20/0x20 [29338.741906] ret_from_fork_asm+0x11/0x20 [29338.745835] [29365.972644] watchdog: BUG: soft lockup - CPU#0 stuck for 16235s! [dbench:78683] [29365.979947] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29365.984644] watchdog: BUG: soft lockup - CPU#7 stuck for 25232s! [kworker/7:4:78808] [29366.035390] irq event stamp: 49076070 [29366.035391] hardirqs last enabled at (49076069): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29366.043119] Modules linked in: [29366.046776] hardirqs last disabled at (49076070): [] sysvec_apic_timer_interrupt+0xe/0x90 [29366.056934] ceph [29366.059984] softirqs last enabled at (49076032): [] __do_softirq+0x2be/0x371 [29366.069708] libceph dns_resolver [29366.071632] softirqs last disabled at (49076027): [] irq_exit_rcu+0xb0/0xd0 [29366.080320] fscache [29366.083628] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29366.092138] netfs veth [29366.094330] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29366.103275] nft_chain_nat [29366.105721] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29366.113190] xt_MASQUERADE nf_nat [29366.115896] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29366.121373] nf_conntrack nf_defrag_ipv6 [29366.124683] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29366.143421] nf_defrag_ipv4 nft_compat [29366.147339] [29366.147340] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29366.152554] nf_tables nfnetlink [29366.156298] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29366.157808] bridge stp [29366.164932] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29366.168156] llc [29366.175280] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29366.177723] xfs libcrc32c [29366.184848] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29366.186685] sunrpc intel_rapl_msr [29366.193810] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29366.196514] intel_rapl_common sb_edac [29366.203638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29366.207041] x86_pkg_temp_thermal [29366.215122] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29366.218865] coretemp kvm_intel [29366.224603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29366.227912] kvm irqbypass [29366.235042] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29366.238182] i2c_i801 iTCO_wdt [29366.245320] Call Trace: [29366.245321] [29366.248038] iTCO_vendor_support i2c_smbus [29366.255163] ? watchdog_timer_fn+0x1b4/0x220 [29366.258213] crct10dif_pclmul [29366.260657] ? __hrtimer_run_queues+0x1d3/0x360 [29366.262667] joydev crc32_pclmul [29366.266761] ? hrtimer_interrupt+0x104/0x240 [29366.271021] ipmi_ssif ghash_clmulni_intel [29366.274004] ? __local_bh_enable+0x50/0x60 [29366.278527] mei_me mei [29366.281753] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29366.286016] lpc_ich sha512_ssse3 [29366.290125] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29366.294215] acpi_ipmi mfd_core [29366.296659] [29366.301875] wmi ipmi_si [29366.305188] [29366.305189] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29366.310145] ipmi_devintf [29366.313284] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29366.315378] ipmi_msghandler [29366.317911] do_raw_spin_lock+0xb5/0xc0 [29366.320007] ioatdma acpi_pad [29366.325328] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29366.327946] acpi_power_meter [29366.332819] try_get_cap_refs+0x159/0x880 [ceph] [29366.335694] ixgbe igb [29366.339528] __ceph_get_caps+0x4a1/0x760 [ceph] [29366.342488] mdio crc32c_intel [29366.347278] ? __file_remove_privs+0x89/0x110 [29366.350237] nvme ptp [29366.354847] ? wake_bit_function+0x70/0x70 [29366.357204] nvme_core i2c_algo_bit [29366.361732] ? ceph_write_iter+0x260/0x770 [ceph] [29366.364782] pps_core [29366.369131] ceph_write_iter+0x260/0x770 [ceph] [29366.371401] dca [29366.375494] ? lock_is_held_type+0xaa/0x120 [29366.378976] fuse [29366.383677] vfs_write+0x324/0x4f0 [29366.385945] irq event stamp: 156479468 [29366.390471] ksys_pwrite64+0x65/0xa0 [29366.392322] hardirqs last enabled at (156479467): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29366.396501] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29366.398425] hardirqs last disabled at (156479468): [] sysvec_apic_timer_interrupt+0xe/0x90 [29366.401822] do_syscall_64+0x38/0x80 [29366.405567] softirqs last enabled at (156479466): [] __do_softirq+0x2be/0x371 [29366.409138] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29366.419380] softirqs last disabled at (156479461): [] irq_exit_rcu+0xb0/0xd0 [29366.424512] RIP: 0033:0x7f48f0eeb93b [29366.434337] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29366.437910] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29366.446682] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29366.451725] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29366.460335] Workqueue: events delayed_work [ceph] [29366.463902] ORIG_RAX: 0000000000000012 [29366.473280] [29366.473281] RIP: 0010:rcu_is_watching+0x0/0x40 [29366.492017] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29366.499489] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [29366.504705] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29366.509404] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [29366.513233] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29366.514725] [29366.514726] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [29366.519161] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29366.526286] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [29366.545023] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29366.552166] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [29366.557388] [29366.564506] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [29366.610926] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29366.618052] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29366.626127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29366.631866] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29366.638990] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29366.646114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29366.653238] Call Trace: [29366.655684] [29366.657694] ? watchdog_timer_fn+0x1b4/0x220 [29366.661959] ? __hrtimer_run_queues+0x1d3/0x360 [29366.666485] ? hrtimer_interrupt+0x104/0x240 [29366.670754] ? __local_bh_enable+0x50/0x60 [29366.674848] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29366.680071] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29366.685047] [29366.687143] [29366.689240] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29366.694546] ? igrab+0x43/0x50 [29366.697596] ? igrab+0x43/0x50 [29366.700645] ? rcutorture_get_gp_data+0x30/0x30 [29366.705171] lock_release+0x1d7/0x260 [29366.708831] _raw_spin_unlock+0x1b/0x40 [29366.712666] igrab+0x43/0x50 [29366.715543] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29366.720687] delayed_work+0x1d8/0x280 [ceph] [29366.724968] ? process_scheduled_works+0x22a/0x510 [29366.729758] process_scheduled_works+0x2a4/0x510 [29366.734373] worker_thread+0x188/0x340 [29366.738122] ? rescuer_thread+0x310/0x310 [29366.742126] kthread+0xf5/0x130 [29366.745263] ? kthread_complete_and_exit+0x20/0x20 [29366.750053] ret_from_fork+0x31/0x40 [29366.753626] ? kthread_complete_and_exit+0x20/0x20 [29366.758411] ret_from_fork_asm+0x11/0x20 [29366.762337] [29393.973148] watchdog: BUG: soft lockup - CPU#0 stuck for 16261s! [dbench:78683] [29393.980452] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29393.985147] watchdog: BUG: soft lockup - CPU#7 stuck for 25258s! [kworker/7:4:78808] [29394.035903] irq event stamp: 49134226 [29394.043634] Modules linked in: ceph libceph dns_resolver fscache netfs veth [29394.047292] hardirqs last enabled at (49134225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29394.054250] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [29394.064408] hardirqs last disabled at (49134226): [] sysvec_apic_timer_interrupt+0xe/0x90 [29394.070059] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [29394.079783] softirqs last enabled at (49134188): [] __do_softirq+0x2be/0x371 [29394.087538] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [29394.096222] softirqs last disabled at (49134183): [] irq_exit_rcu+0xb0/0xd0 [29394.102740] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [29394.111254] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29394.111255] irqbypass [29394.111256] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29394.117250] i2c_i801 iTCO_wdt [29394.126194] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29394.128550] iTCO_vendor_support i2c_smbus [29394.136022] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29394.139072] crct10dif_pclmul joydev [29394.144549] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29394.148640] crc32_pclmul [29394.167380] [29394.170950] ipmi_ssif [29394.176167] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29394.178785] ghash_clmulni_intel mei_me [29394.180273] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29394.182631] mei lpc_ich [29394.189755] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29394.193587] sha512_ssse3 acpi_ipmi [29394.200711] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29394.203242] mfd_core wmi [29394.210366] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29394.213849] ipmi_si ipmi_devintf [29394.220973] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29394.223590] ipmi_msghandler ioatdma [29394.230716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29394.234025] acpi_pad [29394.242103] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29394.245675] acpi_power_meter ixgbe [29394.251411] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29394.253683] igb mdio [29394.260806] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29394.264291] crc32c_intel nvme [29394.271414] Call Trace: [29394.271416] [29394.273686] ptp nvme_core [29394.280810] ? watchdog_timer_fn+0x1b4/0x220 [29394.283860] i2c_algo_bit pps_core [29394.286304] ? __hrtimer_run_queues+0x1d3/0x360 [29394.288315] dca fuse [29394.291023] ? hrtimer_interrupt+0x104/0x240 [29394.295282] [29394.295283] irq event stamp: 156700414 [29394.298681] ? __local_bh_enable+0x50/0x60 [29394.303205] hardirqs last enabled at (156700413): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29394.305479] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29394.309739] hardirqs last disabled at (156700414): [] sysvec_apic_timer_interrupt+0xe/0x90 [29394.311230] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29394.314973] softirqs last enabled at (156700412): [] __do_softirq+0x2be/0x371 [29394.319065] [29394.329310] softirqs last disabled at (156700407): [] irq_exit_rcu+0xb0/0xd0 [29394.334545] [29394.344356] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29394.349314] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29394.358084] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29394.360184] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29394.368778] Workqueue: events delayed_work [ceph] [29394.370877] do_raw_spin_lock+0xb5/0xc0 [29394.380253] [29394.380254] RIP: 0010:lock_acquire+0xf0/0x2b0 [29394.385560] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29394.393027] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29394.397902] try_get_cap_refs+0x159/0x880 [ceph] [29394.402595] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29394.406431] __ceph_get_caps+0x4a1/0x760 [ceph] [29394.407916] [29394.407917] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29394.412271] ? __file_remove_privs+0x89/0x110 [29394.417051] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29394.435789] ? wake_bit_function+0x70/0x70 [29394.440401] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29394.445620] ? ceph_write_iter+0x260/0x770 [ceph] [29394.450141] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29394.451632] ceph_write_iter+0x260/0x770 [ceph] [29394.458756] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29394.463112] ? lock_is_held_type+0xaa/0x120 [29394.470232] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29394.474324] vfs_write+0x324/0x4f0 [29394.481448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29394.486148] ksys_pwrite64+0x65/0xa0 [29394.493268] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29394.497792] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29394.504915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29394.509094] do_syscall_64+0x38/0x80 [29394.517170] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29394.520568] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29394.526305] Call Trace: [29394.526307] [29394.529877] RIP: 0033:0x7f48f0eeb93b [29394.537002] ? watchdog_timer_fn+0x1b4/0x220 [29394.542133] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29394.549258] ? __hrtimer_run_queues+0x1d3/0x360 [29394.552831] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29394.559962] ? hrtimer_interrupt+0x104/0x240 [29394.565004] ORIG_RAX: 0000000000000012 [29394.567448] ? __local_bh_enable+0x50/0x60 [29394.569459] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29394.573031] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29394.577294] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29394.596031] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29394.600554] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29394.605771] [29394.610036] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29394.613868] [29394.617958] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29394.625082] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29394.630304] [29394.637429] ? lock_acquire+0xf0/0x2b0 [29394.679190] ? igrab+0x43/0x50 [29394.682240] ? lock_release+0xc5/0x260 [29394.685986] _raw_spin_lock+0x30/0x40 [29394.689650] ? igrab+0x1d/0x50 [29394.692701] igrab+0x1d/0x50 [29394.695580] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29394.700730] delayed_work+0x1d8/0x280 [ceph] [29394.705012] ? process_scheduled_works+0x22a/0x510 [29394.709800] process_scheduled_works+0x2a4/0x510 [29394.714417] worker_thread+0x188/0x340 [29394.718167] ? rescuer_thread+0x310/0x310 [29394.722176] kthread+0xf5/0x130 [29394.725314] ? kthread_complete_and_exit+0x20/0x20 [29394.730098] ret_from_fork+0x31/0x40 [29394.733677] ? kthread_complete_and_exit+0x20/0x20 [29394.738462] ret_from_fork_asm+0x11/0x20 [29394.742386] [29395.571543] rcu: INFO: rcu_preempt self-detected stall on CPU [29395.577299] rcu: 7-....: (16952276 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/16950966 fqs=8536439 [29395.587897] rcu: (t=17458651 jiffies g=1309609 q=521143 ncpus=8) [29395.593981] Sending NMI from CPU 7 to CPUs 0: [29395.598334] NMI backtrace for cpu 0 [29395.598335] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29395.598337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29395.598338] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29395.598341] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29395.598342] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29395.598343] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29395.598344] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29395.598345] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29395.598345] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29395.598346] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29395.598347] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29395.598348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29395.598349] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29395.598349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29395.598350] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29395.598351] Call Trace: [29395.598351] [29395.598352] ? nmi_cpu_backtrace+0xad/0x120 [29395.598356] ? nmi_cpu_backtrace_handler+0x11/0x20 [29395.598358] ? nmi_handle+0xd4/0x260 [29395.598362] ? default_do_nmi+0x49/0xf0 [29395.598363] ? exc_nmi+0x11e/0x150 [29395.598365] ? end_repeat_nmi+0x16/0x67 [29395.598369] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29395.598371] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29395.598374] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29395.598376] [29395.598376] [29395.598377] do_raw_spin_lock+0xb5/0xc0 [29395.598379] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29395.598393] try_get_cap_refs+0x159/0x880 [ceph] [29395.598406] __ceph_get_caps+0x4a1/0x760 [ceph] [29395.598419] ? __file_remove_privs+0x89/0x110 [29395.598421] ? wake_bit_function+0x70/0x70 [29395.598423] ? ceph_write_iter+0x260/0x770 [ceph] [29395.598433] ceph_write_iter+0x260/0x770 [ceph] [29395.598444] ? lock_is_held_type+0xaa/0x120 [29395.598447] vfs_write+0x324/0x4f0 [29395.598450] ksys_pwrite64+0x65/0xa0 [29395.598453] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29395.598455] do_syscall_64+0x38/0x80 [29395.598457] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29395.598459] RIP: 0033:0x7f48f0eeb93b [29395.598460] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29395.598461] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [29395.598462] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29395.598463] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29395.598464] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29395.598465] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29395.598465] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29395.598468] [29395.599332] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29395.902953] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29395.910426] Workqueue: events delayed_work [ceph] [29395.915147] RIP: 0010:lock_acquire+0xf0/0x2b0 [29395.919498] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29395.938237] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29395.943455] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29395.950578] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29395.957703] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29395.964827] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29395.971949] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29395.979075] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29395.987152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29395.992889] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29396.000014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29396.007137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29396.014261] Call Trace: [29396.016706] [29396.018718] ? rcu_dump_cpu_stacks+0xd1/0x180 [29396.023077] ? rcu_sched_clock_irq+0x633/0x1390 [29396.027603] ? lock_release+0xc5/0x260 [29396.031354] ? lock_is_held_type+0xaa/0x120 [29396.035542] ? lock_is_held_type+0xaa/0x120 [29396.039729] ? tick_sched_do_timer+0xa0/0xa0 [29396.043997] ? rcu_read_lock_sched_held+0x56/0x90 [29396.048699] ? tick_sched_do_timer+0xa0/0xa0 [29396.052968] ? update_process_times+0x5f/0x90 [29396.057320] ? tick_sched_handle.isra.16+0x1e/0x50 [29396.062112] ? tick_sched_timer+0x6f/0x80 [29396.066118] ? __hrtimer_run_queues+0x1d3/0x360 [29396.070656] ? hrtimer_interrupt+0x104/0x240 [29396.074921] ? __local_bh_enable+0x50/0x60 [29396.079018] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29396.084239] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29396.089195] [29396.091292] [29396.093392] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29396.098713] ? lock_acquire+0xf0/0x2b0 [29396.102468] ? igrab+0x43/0x50 [29396.105526] ? lock_release+0xc5/0x260 [29396.109281] _raw_spin_lock+0x30/0x40 [29396.112942] ? igrab+0x1d/0x50 [29396.115993] igrab+0x1d/0x50 [29396.118872] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29396.124023] delayed_work+0x1d8/0x280 [ceph] [29396.128313] ? process_scheduled_works+0x22a/0x510 [29396.133100] process_scheduled_works+0x2a4/0x510 [29396.137722] worker_thread+0x188/0x340 [29396.141477] ? rescuer_thread+0x310/0x310 [29396.145488] kthread+0xf5/0x130 [29396.148633] ? kthread_complete_and_exit+0x20/0x20 [29396.153429] ret_from_fork+0x31/0x40 [29396.157006] ? kthread_complete_and_exit+0x20/0x20 [29396.161800] ret_from_fork_asm+0x11/0x20 [29396.165737] [29421.973651] watchdog: BUG: soft lockup - CPU#0 stuck for 16287s! [dbench:78683] [29421.980948] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29421.985651] watchdog: BUG: soft lockup - CPU#7 stuck for 25284s! [kworker/7:4:78808] [29422.036391] irq event stamp: 49192238 [29422.036392] hardirqs last enabled at (49192237): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29422.044121] Modules linked in: [29422.047778] hardirqs last disabled at (49192238): [] sysvec_apic_timer_interrupt+0xe/0x90 [29422.057935] ceph [29422.060988] softirqs last enabled at (49192204): [] __do_softirq+0x2be/0x371 [29422.070710] libceph dns_resolver [29422.072634] softirqs last disabled at (49192199): [] irq_exit_rcu+0xb0/0xd0 [29422.081326] fscache netfs [29422.084638] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29422.093148] veth nft_chain_nat [29422.095853] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29422.104798] xt_MASQUERADE [29422.107934] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29422.115406] nf_nat nf_conntrack [29422.118109] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29422.123587] nf_defrag_ipv6 nf_defrag_ipv4 [29422.126811] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29422.145548] nft_compat nf_tables [29422.149640] [29422.154857] nfnetlink bridge [29422.158167] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29422.159657] stp llc [29422.162623] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29422.169747] xfs libcrc32c [29422.171931] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29422.179054] sunrpc [29422.181759] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29422.188882] intel_rapl_msr [29422.190981] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29422.198105] intel_rapl_common sb_edac [29422.200894] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29422.208020] x86_pkg_temp_thermal coretemp [29422.211764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29422.219840] kvm_intel kvm [29422.223932] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29422.229668] irqbypass [29422.232372] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29422.239497] i2c_i801 iTCO_wdt [29422.241854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29422.248978] iTCO_vendor_support i2c_smbus [29422.252048] Call Trace: [29422.259170] crct10dif_pclmul joydev [29422.263264] [29422.265707] crc32_pclmul ipmi_ssif [29422.269277] ? watchdog_timer_fn+0x1b4/0x220 [29422.271288] ghash_clmulni_intel mei_me [29422.274772] ? __hrtimer_run_queues+0x1d3/0x360 [29422.279049] mei lpc_ich [29422.282888] ? hrtimer_interrupt+0x104/0x240 [29422.287408] sha512_ssse3 acpi_ipmi [29422.289937] ? __local_bh_enable+0x50/0x60 [29422.294201] mfd_core wmi [29422.297688] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29422.301777] ipmi_si ipmi_devintf [29422.304395] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29422.309612] ipmi_msghandler [29422.312922] [29422.317881] ioatdma acpi_pad [29422.320758] [29422.320759] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29422.322854] acpi_power_meter [29422.325823] ? queued_spin_lock_slowpath+0x16/0x2e0 [29422.327915] ixgbe igb [29422.333222] do_raw_spin_lock+0xb5/0xc0 [29422.336183] mdio crc32c_intel [29422.341055] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29422.343412] nvme [29422.347246] try_get_cap_refs+0x159/0x880 [ceph] [29422.350294] ptp nvme_core [29422.355083] __ceph_get_caps+0x4a1/0x760 [ceph] [29422.357002] i2c_algo_bit pps_core [29422.361617] ? __file_remove_privs+0x89/0x110 [29422.364316] dca fuse [29422.368859] ? wake_bit_function+0x70/0x70 [29422.372255] [29422.376609] ? ceph_write_iter+0x260/0x770 [ceph] [29422.378876] irq event stamp: 156916652 [29422.382969] ceph_write_iter+0x260/0x770 [ceph] [29422.384457] hardirqs last enabled at (156916651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29422.389159] ? lock_is_held_type+0xaa/0x120 [29422.392900] hardirqs last disabled at (156916652): [] sysvec_apic_timer_interrupt+0xe/0x90 [29422.397427] vfs_write+0x324/0x4f0 [29422.407667] softirqs last enabled at (156916650): [] __do_softirq+0x2be/0x371 [29422.411849] ksys_pwrite64+0x65/0xa0 [29422.421655] softirqs last disabled at (156916645): [] irq_exit_rcu+0xb0/0xd0 [29422.425053] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29422.433825] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29422.437397] do_syscall_64+0x38/0x80 [29422.445992] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29422.451124] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29422.460500] Workqueue: events delayed_work [ceph] [29422.464073] RIP: 0033:0x7f48f0eeb93b [29422.471542] [29422.476587] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29422.481285] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [29422.484856] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29422.486346] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [29422.505085] ORIG_RAX: 0000000000000012 [29422.509693] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [29422.514911] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29422.533650] [29422.533651] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [29422.537481] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29422.542697] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [29422.549822] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29422.551313] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [29422.558454] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29422.565578] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [29422.572702] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29422.579826] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29422.586955] [29422.594074] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29422.625726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29422.631463] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29422.638587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29422.645711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29422.652835] Call Trace: [29422.655280] [29422.657290] ? watchdog_timer_fn+0x1b4/0x220 [29422.661556] ? __hrtimer_run_queues+0x1d3/0x360 [29422.666081] ? hrtimer_interrupt+0x104/0x240 [29422.670342] ? __local_bh_enable+0x50/0x60 [29422.674437] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29422.679661] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29422.684618] [29422.686714] [29422.688811] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29422.694122] ? do_raw_spin_lock+0x42/0xc0 [29422.698131] igrab+0x1d/0x50 [29422.701008] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29422.706158] delayed_work+0x1d8/0x280 [ceph] [29422.710437] ? process_scheduled_works+0x22a/0x510 [29422.715227] process_scheduled_works+0x2a4/0x510 [29422.719844] worker_thread+0x188/0x340 [29422.723592] ? rescuer_thread+0x310/0x310 [29422.727596] kthread+0xf5/0x130 [29422.730734] ? kthread_complete_and_exit+0x20/0x20 [29422.735518] ret_from_fork+0x31/0x40 [29422.739088] ? kthread_complete_and_exit+0x20/0x20 [29422.743874] ret_from_fork_asm+0x11/0x20 [29422.747797] [29449.974158] watchdog: BUG: soft lockup - CPU#0 stuck for 16313s! [dbench:78683] [29449.981464] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29449.986158] watchdog: BUG: soft lockup - CPU#7 stuck for 25310s! [kworker/7:4:78808] [29450.036915] irq event stamp: 49250368 [29450.044644] Modules linked in: ceph libceph dns_resolver fscache netfs [29450.048301] hardirqs last enabled at (49250367): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29450.048306] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [29450.065001] hardirqs last disabled at (49250368): [] sysvec_apic_timer_interrupt+0xe/0x90 [29450.073686] nft_compat nf_tables nfnetlink bridge stp llc xfs [29450.083410] softirqs last enabled at (49250340): [] __do_softirq+0x2be/0x371 [29450.089234] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [29450.097918] softirqs last disabled at (49250335): [] irq_exit_rcu+0xb0/0xd0 [29450.106256] coretemp kvm_intel kvm irqbypass i2c_i801 [29450.114767] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29450.119896] iTCO_wdt iTCO_vendor_support i2c_smbus [29450.128841] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29450.133712] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [29450.141183] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29450.141185] ghash_clmulni_intel mei_me mei lpc_ich [29450.152224] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29450.157096] sha512_ssse3 acpi_ipmi [29450.175848] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29450.175849] mfd_core [29450.175850] [29450.179333] wmi ipmi_si [29450.184553] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29450.186834] ipmi_devintf ipmi_msghandler [29450.188323] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29450.190853] ioatdma acpi_pad [29450.197976] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29450.201981] acpi_power_meter ixgbe [29450.209106] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29450.212069] igb mdio [29450.219194] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29450.222678] crc32c_intel nvme [29450.229802] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29450.232072] ptp nvme_core [29450.239195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29450.242246] i2c_algo_bit pps_core [29450.250325] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29450.253029] dca fuse [29450.258766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29450.262164] [29450.262165] irq event stamp: 157137666 [29450.269287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29450.271559] hardirqs last enabled at (157137665): [] asm_common_interrupt+0x26/0x40 [29450.278681] Call Trace: [29450.278683] [29450.280172] hardirqs last disabled at (157137666): [] sysvec_apic_timer_interrupt+0xe/0x90 [29450.283920] ? watchdog_timer_fn+0x1b4/0x220 [29450.291040] softirqs last enabled at (157137664): [] __do_softirq+0x2be/0x371 [29450.300333] ? __hrtimer_run_queues+0x1d3/0x360 [29450.302775] softirqs last disabled at (157137657): [] irq_exit_rcu+0xb0/0xd0 [29450.304791] ? hrtimer_interrupt+0x104/0x240 [29450.314597] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29450.318862] ? __local_bh_enable+0x50/0x60 [29450.327633] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29450.327634] Workqueue: events delayed_work [ceph] [29450.332159] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29450.340754] [29450.340755] RIP: 0010:lock_acquire+0xf0/0x2b0 [29450.345020] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29450.354396] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29450.358488] [29450.365958] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29450.370655] [29450.375871] [29450.375872] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29450.377362] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29450.381714] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29450.386675] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29450.405409] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29450.407508] do_raw_spin_lock+0xb5/0xc0 [29450.412724] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29450.414840] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29450.416328] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29450.423456] try_get_cap_refs+0x159/0x880 [ceph] [29450.428758] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29450.435885] __ceph_get_caps+0x4a1/0x760 [ceph] [29450.440752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29450.447882] ? __file_remove_privs+0x89/0x110 [29450.451707] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29450.458843] ? wake_bit_function+0x70/0x70 [29450.463624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29450.470751] ? ceph_write_iter+0x260/0x770 [ceph] [29450.475359] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29450.483436] ceph_write_iter+0x260/0x770 [ceph] [29450.487961] Call Trace: [29450.487963] [29450.493701] ? lock_is_held_type+0xaa/0x120 [29450.498050] ? watchdog_timer_fn+0x1b4/0x220 [29450.505174] vfs_write+0x324/0x4f0 [29450.509265] ? __hrtimer_run_queues+0x1d3/0x360 [29450.516391] ksys_pwrite64+0x65/0xa0 [29450.521088] ? hrtimer_interrupt+0x104/0x240 [29450.528211] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29450.532732] ? __local_bh_enable+0x50/0x60 [29450.535178] do_syscall_64+0x38/0x80 [29450.537189] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29450.541366] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29450.545631] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29450.549028] RIP: 0033:0x7f48f0eeb93b [29450.553564] [29450.557130] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29450.561394] [29450.561395] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29450.566526] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29450.570620] ? lock_acquire+0xf0/0x2b0 [29450.574188] ORIG_RAX: 0000000000000012 [29450.579406] ? igrab+0x43/0x50 [29450.584449] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29450.589408] ? lock_release+0xc5/0x260 [29450.592977] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29450.595076] _raw_spin_lock+0x30/0x40 [29450.613812] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29450.615910] ? igrab+0x1d/0x50 [29450.621214] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29450.626431] igrab+0x1d/0x50 [29450.630174] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29450.634008] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29450.637060] [29450.644181] delayed_work+0x1d8/0x280 [ceph] [29450.697589] ? process_scheduled_works+0x22a/0x510 [29450.702378] process_scheduled_works+0x2a4/0x510 [29450.706993] worker_thread+0x188/0x340 [29450.710742] ? rescuer_thread+0x310/0x310 [29450.714746] kthread+0xf5/0x130 [29450.717883] ? kthread_complete_and_exit+0x20/0x20 [29450.722668] ret_from_fork+0x31/0x40 [29450.726247] ? kthread_complete_and_exit+0x20/0x20 [29450.731030] ret_from_fork_asm+0x11/0x20 [29450.734953] [29477.974667] watchdog: BUG: soft lockup - CPU#0 stuck for 16339s! [dbench:78683] [29477.981968] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29477.986667] watchdog: BUG: soft lockup - CPU#7 stuck for 25336s! [kworker/7:4:78808] [29478.037409] irq event stamp: 49308430 [29478.045139] Modules linked in: ceph libceph [29478.048797] hardirqs last enabled at (49308429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29478.052975] dns_resolver fscache netfs [29478.063131] hardirqs last disabled at (49308430): [] sysvec_apic_timer_interrupt+0xe/0x90 [29478.066964] veth nft_chain_nat xt_MASQUERADE [29478.076687] softirqs last enabled at (49308402): [] __do_softirq+0x2be/0x371 [29478.076688] nf_nat [29478.081037] nf_conntrack nf_defrag_ipv6 [29478.089739] softirqs last disabled at (49308397): [] irq_exit_rcu+0xb0/0xd0 [29478.091836] nf_defrag_ipv4 [29478.095755] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29478.104266] nft_compat nf_tables [29478.107064] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29478.116009] nfnetlink bridge [29478.119318] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29478.126806] stp [29478.129770] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29478.135248] llc xfs [29478.137087] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29478.155824] libcrc32c sunrpc [29478.158007] [29478.158008] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29478.163225] intel_rapl_msr intel_rapl_common [29478.166188] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29478.167678] sb_edac x86_pkg_temp_thermal [29478.174804] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29478.179155] coretemp kvm_intel [29478.186277] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29478.190282] kvm irqbypass [29478.197407] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29478.200544] i2c_i801 iTCO_wdt [29478.207668] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29478.210372] iTCO_vendor_support i2c_smbus [29478.217496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29478.220548] crct10dif_pclmul joydev [29478.228623] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29478.232716] crc32_pclmul ipmi_ssif [29478.238454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29478.242024] ghash_clmulni_intel mei_me [29478.249147] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29478.252632] mei lpc_ich [29478.259756] Call Trace: [29478.259758] [29478.263587] sha512_ssse3 [29478.270713] ? watchdog_timer_fn+0x1b4/0x220 [29478.273242] acpi_ipmi mfd_core [29478.275688] ? __hrtimer_run_queues+0x1d3/0x360 [29478.277697] wmi ipmi_si [29478.280316] ? hrtimer_interrupt+0x104/0x240 [29478.284576] ipmi_devintf ipmi_msghandler [29478.287715] ? __local_bh_enable+0x50/0x60 [29478.292239] ioatdma acpi_pad [29478.294772] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29478.299034] acpi_power_meter ixgbe [29478.303040] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29478.307130] igb mdio [29478.310093] [29478.315310] crc32c_intel nvme [29478.318794] [29478.318795] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29478.323751] ptp nvme_core [29478.326026] ? queued_spin_lock_slowpath+0x18/0x2e0 [29478.328120] i2c_algo_bit pps_core [29478.331172] do_raw_spin_lock+0xb5/0xc0 [29478.333268] dca fuse [29478.338572] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29478.341276] [29478.341277] irq event stamp: 157358720 [29478.346148] try_get_cap_refs+0x159/0x880 [ceph] [29478.349545] hardirqs last enabled at (157358719): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29478.353380] __ceph_get_caps+0x4a1/0x760 [ceph] [29478.355645] hardirqs last disabled at (157358720): [] sysvec_apic_timer_interrupt+0xe/0x90 [29478.360436] ? __file_remove_privs+0x89/0x110 [29478.361920] softirqs last enabled at (157358718): [] __do_softirq+0x2be/0x371 [29478.365665] ? wake_bit_function+0x70/0x70 [29478.370275] softirqs last disabled at (157358713): [] irq_exit_rcu+0xb0/0xd0 [29478.380522] ? ceph_write_iter+0x260/0x770 [ceph] [29478.385044] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29478.394872] ceph_write_iter+0x260/0x770 [ceph] [29478.399221] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29478.407997] ? lock_is_held_type+0xaa/0x120 [29478.412083] Workqueue: events delayed_work [ceph] [29478.420685] vfs_write+0x324/0x4f0 [29478.425380] [29478.425381] RIP: 0010:lock_acquire+0xf0/0x2b0 [29478.434760] ksys_pwrite64+0x65/0xa0 [29478.439280] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29478.446751] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29478.450929] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29478.455629] do_syscall_64+0x38/0x80 [29478.459024] [29478.459024] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29478.460516] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29478.464865] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29478.468436] RIP: 0033:0x7f48f0eeb93b [29478.487173] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29478.492306] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29478.497521] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29478.501093] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29478.502584] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29478.509706] ORIG_RAX: 0000000000000012 [29478.514752] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29478.521876] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29478.525447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29478.532571] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29478.551307] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29478.558431] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29478.563648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29478.570773] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29478.574604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29478.582683] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29478.589806] Call Trace: [29478.589807] [29478.595550] [29478.602668] ? watchdog_timer_fn+0x1b4/0x220 [29478.656335] ? __hrtimer_run_queues+0x1d3/0x360 [29478.660862] ? hrtimer_interrupt+0x104/0x240 [29478.665132] ? __local_bh_enable+0x50/0x60 [29478.669231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29478.674456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29478.679414] [29478.681511] [29478.683608] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29478.688916] ? lock_acquire+0xf0/0x2b0 [29478.692665] ? igrab+0x43/0x50 [29478.695715] ? lock_release+0xc5/0x260 [29478.699461] _raw_spin_lock+0x30/0x40 [29478.703125] ? igrab+0x1d/0x50 [29478.706176] igrab+0x1d/0x50 [29478.709071] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29478.714223] delayed_work+0x1d8/0x280 [ceph] [29478.718503] ? process_scheduled_works+0x22a/0x510 [29478.723293] process_scheduled_works+0x2a4/0x510 [29478.727907] worker_thread+0x188/0x340 [29478.731656] ? rescuer_thread+0x310/0x310 [29478.735661] kthread+0xf5/0x130 [29478.738799] ? kthread_complete_and_exit+0x20/0x20 [29478.743583] ret_from_fork+0x31/0x40 [29478.747152] ? kthread_complete_and_exit+0x20/0x20 [29478.751938] ret_from_fork_asm+0x11/0x20 [29478.755861] [29505.975175] watchdog: BUG: soft lockup - CPU#0 stuck for 16365s! [dbench:78683] [29505.982482] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29505.987175] watchdog: BUG: soft lockup - CPU#7 stuck for 25362s! [kworker/7:4:78808] [29506.037931] irq event stamp: 49366544 [29506.045661] Modules linked in: ceph libceph dns_resolver fscache netfs [29506.049320] hardirqs last enabled at (49366543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29506.049325] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [29506.066012] hardirqs last disabled at (49366544): [] sysvec_apic_timer_interrupt+0xe/0x90 [29506.073396] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [29506.083120] softirqs last enabled at (49366516): [] __do_softirq+0x2be/0x371 [29506.089907] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [29506.098589] softirqs last disabled at (49366511): [] irq_exit_rcu+0xb0/0xd0 [29506.107274] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [29506.115788] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29506.121705] iTCO_vendor_support i2c_smbus crct10dif_pclmul [29506.130649] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29506.136212] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [29506.143683] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29506.149508] mei_me mei lpc_ich sha512_ssse3 [29506.154987] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29506.159249] acpi_ipmi mfd_core wmi [29506.177987] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29506.177988] ipmi_si ipmi_devintf [29506.181471] [29506.186688] ipmi_msghandler [29506.189999] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29506.191489] ioatdma acpi_pad [29506.194367] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29506.201490] acpi_power_meter [29506.204454] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29506.211580] ixgbe igb [29506.214542] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29506.221667] mdio crc32c_intel [29506.224023] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29506.231149] nvme ptp [29506.234199] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29506.241324] nvme_core i2c_algo_bit [29506.243595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29506.251673] pps_core dca [29506.255156] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29506.260894] fuse [29506.263510] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29506.270636] [29506.270637] irq event stamp: 157579660 [29506.272575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29506.279700] hardirqs last enabled at (157579659): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29506.281192] Call Trace: [29506.281194] [29506.284936] hardirqs last disabled at (157579660): [] sysvec_apic_timer_interrupt+0xe/0x90 [29506.292062] ? watchdog_timer_fn+0x1b4/0x220 [29506.302304] softirqs last enabled at (157579658): [] __do_softirq+0x2be/0x371 [29506.304749] ? __hrtimer_run_queues+0x1d3/0x360 [29506.306757] softirqs last disabled at (157579653): [] irq_exit_rcu+0xb0/0xd0 [29506.316579] ? hrtimer_interrupt+0x104/0x240 [29506.320856] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29506.329630] ? __local_bh_enable+0x50/0x60 [29506.334155] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29506.334156] Workqueue: events delayed_work [ceph] [29506.342755] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29506.347015] [29506.347016] RIP: 0010:lock_release+0x77/0x260 [29506.356394] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29506.360483] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [29506.367956] [29506.372653] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [29506.377868] [29506.379360] [29506.379361] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [29506.383713] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29506.388669] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [29506.407409] ? queued_spin_lock_slowpath+0x18/0x2e0 [29506.409504] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [29506.414723] do_raw_spin_lock+0xb5/0xc0 [29506.416819] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [29506.418312] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29506.425434] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [29506.430741] try_get_cap_refs+0x159/0x880 [ceph] [29506.437860] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29506.442739] __ceph_get_caps+0x4a1/0x760 [ceph] [29506.449858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29506.453692] ? __file_remove_privs+0x89/0x110 [29506.460811] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29506.465597] ? wake_bit_function+0x70/0x70 [29506.472718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29506.477333] ? ceph_write_iter+0x260/0x770 [ceph] [29506.485406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29506.489933] ceph_write_iter+0x260/0x770 [ceph] [29506.495668] Call Trace: [29506.495670] [29506.500023] ? lock_is_held_type+0xaa/0x120 [29506.507145] ? watchdog_timer_fn+0x1b4/0x220 [29506.511238] vfs_write+0x324/0x4f0 [29506.518359] ? __hrtimer_run_queues+0x1d3/0x360 [29506.523060] ksys_pwrite64+0x65/0xa0 [29506.530185] ? hrtimer_interrupt+0x104/0x240 [29506.534704] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29506.537149] ? __local_bh_enable+0x50/0x60 [29506.539161] do_syscall_64+0x38/0x80 [29506.543338] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29506.547602] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29506.550997] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29506.555523] RIP: 0033:0x7f48f0eeb93b [29506.559093] [29506.563358] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29506.568487] [29506.572579] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29506.576149] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29506.581367] ORIG_RAX: 0000000000000012 [29506.586413] ? igrab+0x43/0x50 [29506.591369] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29506.594943] ? lock_release+0x77/0x260 [29506.597037] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29506.615777] _raw_spin_unlock+0x1b/0x40 [29506.617871] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29506.623088] igrab+0x43/0x50 [29506.628392] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29506.632225] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29506.635274] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29506.642403] delayed_work+0x1d8/0x280 [ceph] [29506.646149] [29506.653270] ? process_scheduled_works+0x22a/0x510 [29506.697710] process_scheduled_works+0x2a4/0x510 [29506.702327] worker_thread+0x188/0x340 [29506.706076] ? rescuer_thread+0x310/0x310 [29506.710087] kthread+0xf5/0x130 [29506.713234] ? kthread_complete_and_exit+0x20/0x20 [29506.718018] ret_from_fork+0x31/0x40 [29506.721597] ? kthread_complete_and_exit+0x20/0x20 [29506.726380] ret_from_fork_asm+0x11/0x20 [29506.730303] [29533.975683] watchdog: BUG: soft lockup - CPU#0 stuck for 16391s! [dbench:78683] [29533.982986] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29533.987683] watchdog: BUG: soft lockup - CPU#7 stuck for 25388s! [kworker/7:4:78808] [29534.038428] irq event stamp: 49424568 [29534.038429] hardirqs last enabled at (49424567): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29534.046157] Modules linked in: ceph [29534.049815] hardirqs last disabled at (49424568): [] sysvec_apic_timer_interrupt+0xe/0x90 [29534.059972] libceph [29534.063456] softirqs last enabled at (49424540): [] __do_softirq+0x2be/0x371 [29534.073182] dns_resolver [29534.075367] softirqs last disabled at (49424535): [] irq_exit_rcu+0xb0/0xd0 [29534.084049] fscache netfs [29534.086667] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29534.095178] veth nft_chain_nat [29534.097882] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29534.106825] xt_MASQUERADE nf_nat [29534.109964] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29534.117434] nf_conntrack nf_defrag_ipv6 [29534.120744] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29534.126223] nf_defrag_ipv4 nft_compat [29534.130139] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29534.148877] nf_tables nfnetlink [29534.152622] [29534.157839] bridge stp [29534.161080] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29534.162571] llc xfs [29534.165015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29534.172139] libcrc32c [29534.174323] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29534.181448] sunrpc intel_rapl_msr [29534.183803] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29534.190928] intel_rapl_common sb_edac [29534.194326] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29534.201451] x86_pkg_temp_thermal coretemp [29534.205195] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29534.212317] kvm_intel [29534.216408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29534.224485] kvm irqbypass [29534.226845] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29534.232582] i2c_i801 iTCO_wdt [29534.235286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29534.242410] iTCO_vendor_support [29534.245460] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29534.252584] i2c_smbus crct10dif_pclmul [29534.255807] Call Trace: [29534.255808] [29534.262933] joydev [29534.266764] ? watchdog_timer_fn+0x1b4/0x220 [29534.269206] crc32_pclmul ipmi_ssif [29534.271218] ? __hrtimer_run_queues+0x1d3/0x360 [29534.273315] ghash_clmulni_intel mei_me [29534.277582] ? hrtimer_interrupt+0x104/0x240 [29534.281079] mei lpc_ich [29534.285604] ? __local_bh_enable+0x50/0x60 [29534.289436] sha512_ssse3 [29534.293701] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29534.296231] acpi_ipmi mfd_core [29534.300321] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29534.302938] wmi ipmi_si [29534.308156] [29534.311294] ipmi_devintf ipmi_msghandler [29534.316249] [29534.316251] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29534.318780] ioatdma acpi_pad [29534.320882] ? queued_spin_lock_slowpath+0x16/0x2e0 [29534.324882] acpi_power_meter ixgbe [29534.326983] do_raw_spin_lock+0xb5/0xc0 [29534.332285] igb mdio [29534.335249] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29534.340120] crc32c_intel [29534.343605] try_get_cap_refs+0x159/0x880 [ceph] [29534.347435] nvme [29534.349710] __ceph_get_caps+0x4a1/0x760 [ceph] [29534.354488] ptp nvme_core [29534.357110] ? __file_remove_privs+0x89/0x110 [29534.361718] i2c_algo_bit pps_core [29534.363641] ? wake_bit_function+0x70/0x70 [29534.368166] dca fuse [29534.370873] ? ceph_write_iter+0x260/0x770 [ceph] [29534.375220] [29534.375220] irq event stamp: 157800820 [29534.378618] ceph_write_iter+0x260/0x770 [ceph] [29534.382708] hardirqs last enabled at (157800819): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29534.384982] ? lock_is_held_type+0xaa/0x120 [29534.389676] hardirqs last disabled at (157800820): [] sysvec_apic_timer_interrupt+0xe/0x90 [29534.391171] vfs_write+0x324/0x4f0 [29534.394910] softirqs last enabled at (157800818): [] __do_softirq+0x2be/0x371 [29534.399440] ksys_pwrite64+0x65/0xa0 [29534.409680] softirqs last disabled at (157800813): [] irq_exit_rcu+0xb0/0xd0 [29534.413858] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29534.423669] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29534.427065] do_syscall_64+0x38/0x80 [29534.435853] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29534.439424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29534.448022] Workqueue: events delayed_work [ceph] [29534.453151] RIP: 0033:0x7f48f0eeb93b [29534.462530] [29534.466100] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29534.473572] RIP: 0010:lock_acquire+0xf0/0x2b0 [29534.478616] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29534.483312] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29534.486884] ORIG_RAX: 0000000000000012 [29534.488375] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29534.507111] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29534.511462] [29534.511463] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29534.516680] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29534.535418] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29534.539249] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29534.544465] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29534.551590] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29534.553085] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29534.560214] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29534.567338] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29534.574467] [29534.581586] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29534.627468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29534.633204] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29534.640332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29534.647455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29534.654577] Call Trace: [29534.657024] [29534.659033] ? watchdog_timer_fn+0x1b4/0x220 [29534.663299] ? __hrtimer_run_queues+0x1d3/0x360 [29534.667825] ? hrtimer_interrupt+0x104/0x240 [29534.672095] ? __local_bh_enable+0x50/0x60 [29534.676197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29534.681419] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29534.686378] [29534.688475] [29534.690572] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29534.695881] ? lock_acquire+0xf0/0x2b0 [29534.699629] ? igrab+0x43/0x50 [29534.702678] ? lock_release+0xc5/0x260 [29534.706426] _raw_spin_lock+0x30/0x40 [29534.710092] ? igrab+0x1d/0x50 [29534.713150] igrab+0x1d/0x50 [29534.716027] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29534.721177] delayed_work+0x1d8/0x280 [ceph] [29534.725459] ? process_scheduled_works+0x22a/0x510 [29534.730248] process_scheduled_works+0x2a4/0x510 [29534.734865] worker_thread+0x188/0x340 [29534.738614] ? rescuer_thread+0x310/0x310 [29534.742625] kthread+0xf5/0x130 [29534.745762] ? kthread_complete_and_exit+0x20/0x20 [29534.750546] ret_from_fork+0x31/0x40 [29534.754118] ? kthread_complete_and_exit+0x20/0x20 [29534.758900] ret_from_fork_asm+0x11/0x20 [29534.762823] [29561.976190] watchdog: BUG: soft lockup - CPU#0 stuck for 16417s! [dbench:78683] [29561.983492] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29561.988190] watchdog: BUG: soft lockup - CPU#7 stuck for 25414s! [kworker/7:4:78808] [29562.038942] irq event stamp: 49482688 [29562.046673] Modules linked in: ceph libceph dns_resolver fscache netfs [29562.050330] hardirqs last enabled at (49482687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29562.056863] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [29562.067022] hardirqs last disabled at (49482688): [] sysvec_apic_timer_interrupt+0xe/0x90 [29562.073107] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [29562.082828] softirqs last enabled at (49482660): [] __do_softirq+0x2be/0x371 [29562.089607] bridge stp llc xfs libcrc32c [29562.098292] softirqs last disabled at (49482655): [] irq_exit_rcu+0xb0/0xd0 [29562.098295] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [29562.110817] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29562.119064] kvm_intel kvm [29562.128009] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29562.128010] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29562.130714] irqbypass i2c_i801 [29562.138185] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29562.143662] iTCO_wdt iTCO_vendor_support [29562.146799] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29562.165536] i2c_smbus crct10dif_pclmul [29562.169542] [29562.174757] joydev crc32_pclmul [29562.178590] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29562.180079] ipmi_ssif [29562.183305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29562.190429] ghash_clmulni_intel mei_me [29562.192784] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29562.199910] mei lpc_ich [29562.203740] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29562.210869] sha512_ssse3 [29562.213403] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29562.220528] acpi_ipmi [29562.223147] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29562.230269] mfd_core wmi [29562.232627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29562.240704] ipmi_si [29562.243322] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29562.249060] ipmi_devintf ipmi_msghandler [29562.251245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29562.258368] ioatdma acpi_pad [29562.262372] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29562.269496] acpi_power_meter ixgbe [29562.272461] Call Trace: [29562.279588] igb [29562.283078] [29562.285520] mdio crc32c_intel [29562.287361] ? watchdog_timer_fn+0x1b4/0x220 [29562.289368] nvme ptp [29562.292421] ? __hrtimer_run_queues+0x1d3/0x360 [29562.296682] nvme_core i2c_algo_bit [29562.298958] ? hrtimer_interrupt+0x104/0x240 [29562.303479] pps_core dca [29562.306962] ? __local_bh_enable+0x50/0x60 [29562.311227] fuse [29562.313846] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29562.317951] irq event stamp: 158021576 [29562.319881] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29562.325102] hardirqs last enabled at (158021575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29562.328847] [29562.333804] hardirqs last disabled at (158021576): [] sysvec_apic_timer_interrupt+0xe/0x90 [29562.344047] [29562.344049] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29562.346146] softirqs last enabled at (158021574): [] __do_softirq+0x2be/0x371 [29562.355959] ? queued_spin_lock_slowpath+0x16/0x2e0 [29562.358052] softirqs last disabled at (158021569): [] irq_exit_rcu+0xb0/0xd0 [29562.363359] do_raw_spin_lock+0xb5/0xc0 [29562.372130] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29562.377001] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29562.385596] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29562.389429] try_get_cap_refs+0x159/0x880 [ceph] [29562.398806] Workqueue: events delayed_work [ceph] [29562.403599] __ceph_get_caps+0x4a1/0x760 [ceph] [29562.411068] [29562.411069] RIP: 0010:lock_acquire+0xf0/0x2b0 [29562.415684] ? __file_remove_privs+0x89/0x110 [29562.420376] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29562.424902] ? wake_bit_function+0x70/0x70 [29562.426391] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29562.430745] ? ceph_write_iter+0x260/0x770 [ceph] [29562.435094] [29562.453831] ceph_write_iter+0x260/0x770 [ceph] [29562.457921] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29562.463144] ? lock_is_held_type+0xaa/0x120 [29562.467836] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29562.469329] vfs_write+0x324/0x4f0 [29562.473867] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29562.480996] ksys_pwrite64+0x65/0xa0 [29562.485169] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29562.492296] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29562.495690] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29562.502817] do_syscall_64+0x38/0x80 [29562.506387] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29562.513512] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29562.518640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29562.525765] RIP: 0033:0x7f48f0eeb93b [29562.529335] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29562.537414] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29562.542458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29562.548195] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29562.551764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29562.558890] ORIG_RAX: 0000000000000012 [29562.577628] Call Trace: [29562.577630] [29562.584752] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29562.589970] ? watchdog_timer_fn+0x1b4/0x220 [29562.597093] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29562.600924] ? __hrtimer_run_queues+0x1d3/0x360 [29562.603368] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29562.605383] ? hrtimer_interrupt+0x104/0x240 [29562.612502] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29562.616766] ? __local_bh_enable+0x50/0x60 [29562.623890] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29562.628418] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29562.635544] [29562.639804] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29562.670503] [29562.672598] [29562.674698] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29562.680005] ? lock_acquire+0xf0/0x2b0 [29562.683757] ? igrab+0x43/0x50 [29562.686813] ? lock_release+0xc5/0x260 [29562.690559] _raw_spin_lock+0x30/0x40 [29562.694224] ? igrab+0x1d/0x50 [29562.697273] igrab+0x1d/0x50 [29562.700152] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29562.705302] delayed_work+0x1d8/0x280 [ceph] [29562.709600] ? process_scheduled_works+0x22a/0x510 [29562.714391] process_scheduled_works+0x2a4/0x510 [29562.719007] worker_thread+0x188/0x340 [29562.722756] ? rescuer_thread+0x310/0x310 [29562.726766] kthread+0xf5/0x130 [29562.729903] ? kthread_complete_and_exit+0x20/0x20 [29562.734689] ret_from_fork+0x31/0x40 [29562.738268] ? kthread_complete_and_exit+0x20/0x20 [29562.743052] ret_from_fork_asm+0x11/0x20 [29562.746985] [29589.976699] watchdog: BUG: soft lockup - CPU#0 stuck for 16443s! [dbench:78683] [29589.984004] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29589.988699] watchdog: BUG: soft lockup - CPU#7 stuck for 25440s! [kworker/7:4:78808] [29590.039447] irq event stamp: 49540718 [29590.039448] hardirqs last enabled at (49540717): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29590.047176] Modules linked in: ceph [29590.050835] hardirqs last disabled at (49540718): [] sysvec_apic_timer_interrupt+0xe/0x90 [29590.060991] libceph [29590.064475] softirqs last enabled at (49540690): [] __do_softirq+0x2be/0x371 [29590.074201] dns_resolver [29590.076384] softirqs last disabled at (49540685): [] irq_exit_rcu+0xb0/0xd0 [29590.085068] fscache netfs [29590.087704] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29590.096213] veth nft_chain_nat [29590.098917] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29590.107863] xt_MASQUERADE nf_nat [29590.111000] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29590.118469] nf_conntrack nf_defrag_ipv6 [29590.121781] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29590.127258] nf_defrag_ipv4 nft_compat [29590.131177] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29590.149912] nf_tables nfnetlink [29590.153656] [29590.153657] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29590.158874] bridge stp [29590.162100] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29590.163588] llc xfs [29590.170714] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29590.173159] libcrc32c [29590.180282] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29590.182464] sunrpc [29590.189591] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29590.191948] intel_rapl_msr [29590.199071] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29590.201187] intel_rapl_common sb_edac [29590.208310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29590.211100] x86_pkg_temp_thermal [29590.219178] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29590.222923] coretemp kvm_intel [29590.228660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29590.231971] kvm irqbypass [29590.239094] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29590.242233] i2c_i801 iTCO_wdt [29590.249373] Call Trace: [29590.252095] iTCO_vendor_support i2c_smbus [29590.259221] [29590.262270] crct10dif_pclmul joydev [29590.264714] ? watchdog_timer_fn+0x1b4/0x220 [29590.268805] crc32_pclmul ipmi_ssif [29590.270815] ? __hrtimer_run_queues+0x1d3/0x360 [29590.274389] ghash_clmulni_intel mei_me [29590.278653] ? hrtimer_interrupt+0x104/0x240 [29590.282133] mei lpc_ich [29590.286658] ? __local_bh_enable+0x50/0x60 [29590.290490] sha512_ssse3 acpi_ipmi [29590.294755] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29590.297284] mfd_core wmi [29590.301380] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29590.304868] ipmi_si ipmi_devintf [29590.310098] [29590.312710] ipmi_msghandler ioatdma [29590.317668] [29590.320978] acpi_pad acpi_power_meter [29590.323095] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29590.326664] ixgbe igb [29590.328767] ? queued_spin_lock_slowpath+0x16/0x2e0 [29590.332506] mdio crc32c_intel [29590.337811] do_raw_spin_lock+0xb5/0xc0 [29590.340167] nvme ptp [29590.345040] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29590.348098] nvme_core [29590.351932] try_get_cap_refs+0x159/0x880 [ceph] [29590.354198] i2c_algo_bit pps_core [29590.358989] __ceph_get_caps+0x4a1/0x760 [ceph] [29590.361340] dca fuse [29590.365957] ? __file_remove_privs+0x89/0x110 [29590.369350] [29590.369350] irq event stamp: 158242494 [29590.373874] ? wake_bit_function+0x70/0x70 [29590.376142] hardirqs last enabled at (158242493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29590.380496] ? ceph_write_iter+0x260/0x770 [ceph] [29590.381984] hardirqs last disabled at (158242494): [] sysvec_apic_timer_interrupt+0xe/0x90 [29590.385728] ceph_write_iter+0x260/0x770 [ceph] [29590.389819] softirqs last enabled at (158242492): [] __do_softirq+0x2be/0x371 [29590.400068] ? lock_is_held_type+0xaa/0x120 [29590.404762] softirqs last disabled at (158242487): [] irq_exit_rcu+0xb0/0xd0 [29590.414574] vfs_write+0x324/0x4f0 [29590.419098] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29590.427872] ksys_pwrite64+0x65/0xa0 [29590.432045] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29590.440642] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29590.444039] Workqueue: events delayed_work [ceph] [29590.453418] do_syscall_64+0x38/0x80 [29590.456987] [29590.456988] RIP: 0010:lock_acquire+0x10c/0x2b0 [29590.464460] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29590.469589] Code: 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 c4 38 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 65 8b 05 b2 e6 eb 7e 89 c0 48 0f a3 05 74 bc [29590.474286] RIP: 0033:0x7f48f0eeb93b [29590.477858] RSP: 0018:ffffc90002117d68 EFLAGS: 00000296 [29590.479349] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29590.483785] [29590.488830] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29590.507566] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [29590.511139] ORIG_RAX: 0000000000000012 [29590.516354] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29590.535103] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29590.536591] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [29590.541811] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29590.548933] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [29590.552766] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29590.559889] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29590.567014] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29590.574137] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29590.581260] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29590.588394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29590.595524] [29590.602643] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29590.640014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29590.647137] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29590.654263] Call Trace: [29590.656706] [29590.658718] ? watchdog_timer_fn+0x1b4/0x220 [29590.662983] ? __hrtimer_run_queues+0x1d3/0x360 [29590.667508] ? hrtimer_interrupt+0x104/0x240 [29590.671770] ? __local_bh_enable+0x50/0x60 [29590.675864] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29590.681104] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29590.686062] [29590.688158] [29590.690256] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29590.695564] ? lock_acquire+0x10c/0x2b0 [29590.699402] _raw_spin_lock+0x30/0x40 [29590.703065] ? igrab+0x1d/0x50 [29590.706115] igrab+0x1d/0x50 [29590.708994] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29590.714145] delayed_work+0x1d8/0x280 [ceph] [29590.718425] ? process_scheduled_works+0x22a/0x510 [29590.723216] process_scheduled_works+0x2a4/0x510 [29590.727832] worker_thread+0x188/0x340 [29590.731582] ? rescuer_thread+0x310/0x310 [29590.735591] kthread+0xf5/0x130 [29590.738730] ? kthread_complete_and_exit+0x20/0x20 [29590.743513] ret_from_fork+0x31/0x40 [29590.747103] ? kthread_complete_and_exit+0x20/0x20 [29590.751885] ret_from_fork_asm+0x11/0x20 [29590.755810] [29591.173086] rcu: INFO: rcu_preempt self-detected stall on CPU [29591.178850] rcu: 0-....: (17257101 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=8632342 [29591.189274] rcu: (t=17654250 jiffies g=1309609 q=525121 ncpus=8) [29591.195360] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29591.204301] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29591.211773] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29591.217251] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29591.235987] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29591.241206] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29591.248329] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29591.255454] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29591.262578] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29591.269701] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29591.276827] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29591.284904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29591.290640] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29591.297766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29591.304890] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29591.312013] Call Trace: [29591.314458] [29591.316468] ? rcu_dump_cpu_stacks+0xd1/0x180 [29591.320827] ? rcu_sched_clock_irq+0x633/0x1390 [29591.325353] ? lock_is_held_type+0xaa/0x120 [29591.329530] ? lock_is_held_type+0xaa/0x120 [29591.333709] ? tick_sched_do_timer+0xa0/0xa0 [29591.337981] ? rcu_read_lock_sched_held+0x56/0x90 [29591.342687] ? tick_sched_do_timer+0xa0/0xa0 [29591.346950] ? update_process_times+0x5f/0x90 [29591.351300] ? tick_sched_handle.isra.16+0x1e/0x50 [29591.356085] ? tick_sched_timer+0x6f/0x80 [29591.360089] ? __hrtimer_run_queues+0x1d3/0x360 [29591.364617] ? hrtimer_interrupt+0x104/0x240 [29591.368884] ? __local_bh_enable+0x50/0x60 [29591.372978] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29591.378194] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29591.383151] [29591.385248] [29591.387346] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29591.392653] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29591.397531] do_raw_spin_lock+0xb5/0xc0 [29591.401360] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29591.406164] try_get_cap_refs+0x159/0x880 [ceph] [29591.410794] __ceph_get_caps+0x4a1/0x760 [ceph] [29591.415335] ? __file_remove_privs+0x89/0x110 [29591.419691] ? wake_bit_function+0x70/0x70 [29591.423783] ? ceph_write_iter+0x260/0x770 [ceph] [29591.428488] ceph_write_iter+0x260/0x770 [ceph] [29591.433023] ? lock_is_held_type+0xaa/0x120 [29591.437208] vfs_write+0x324/0x4f0 [29591.440616] ksys_pwrite64+0x65/0xa0 [29591.444192] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29591.449322] do_syscall_64+0x38/0x80 [29591.452894] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29591.457939] RIP: 0033:0x7f48f0eeb93b [29591.461508] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29591.480245] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [29591.487803] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29591.494928] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29591.502052] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29591.509175] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29591.516298] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29591.523429] [29591.525616] Sending NMI from CPU 0 to CPUs 7: [29591.529970] NMI backtrace for cpu 7 [29591.529972] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29591.529973] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29591.529974] Workqueue: events delayed_work [ceph] [29591.529989] RIP: 0010:lock_acquire+0x86/0x2b0 [29591.529991] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [29591.529992] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [29591.529994] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [29591.529994] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [29591.529995] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [29591.529996] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [29591.529996] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29591.529997] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29591.529998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29591.529999] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29591.529999] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29591.530000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29591.530001] Call Trace: [29591.530001] [29591.530002] ? nmi_cpu_backtrace+0xad/0x120 [29591.530006] ? nmi_cpu_backtrace_handler+0x11/0x20 [29591.530007] ? nmi_handle+0xd4/0x260 [29591.530011] ? default_do_nmi+0x49/0xf0 [29591.530013] ? exc_nmi+0x11e/0x150 [29591.530014] ? end_repeat_nmi+0x16/0x67 [29591.530018] ? lock_acquire+0x86/0x2b0 [29591.530020] ? lock_acquire+0x86/0x2b0 [29591.530022] ? lock_acquire+0x86/0x2b0 [29591.530023] [29591.530024] [29591.530024] ? igrab+0x43/0x50 [29591.530026] ? lock_release+0xc5/0x260 [29591.530027] _raw_spin_lock+0x30/0x40 [29591.530030] ? igrab+0x1d/0x50 [29591.530031] igrab+0x1d/0x50 [29591.530032] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29591.530046] delayed_work+0x1d8/0x280 [ceph] [29591.530059] ? process_scheduled_works+0x22a/0x510 [29591.530062] process_scheduled_works+0x2a4/0x510 [29591.530066] worker_thread+0x188/0x340 [29591.530068] ? rescuer_thread+0x310/0x310 [29591.530071] kthread+0xf5/0x130 [29591.530073] ? kthread_complete_and_exit+0x20/0x20 [29591.530075] ret_from_fork+0x31/0x40 [29591.530078] ? kthread_complete_and_exit+0x20/0x20 [29591.530080] ret_from_fork_asm+0x11/0x20 [29591.530084] [29617.977207] watchdog: BUG: soft lockup - CPU#0 stuck for 16469s! [dbench:78683] [29617.984511] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29617.989207] watchdog: BUG: soft lockup - CPU#7 stuck for 25466s! [kworker/7:4:78808] [29618.039960] irq event stamp: 49598082 [29618.047689] Modules linked in: ceph libceph dns_resolver fscache netfs [29618.051348] hardirqs last enabled at (49598081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29618.057881] veth nft_chain_nat xt_MASQUERADE [29618.068040] hardirqs last disabled at (49598082): [] sysvec_apic_timer_interrupt+0xe/0x90 [29618.068043] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [29618.082116] softirqs last enabled at (49597994): [] __do_softirq+0x2be/0x371 [29618.082120] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [29618.099924] softirqs last disabled at (49597989): [] irq_exit_rcu+0xb0/0xd0 [29618.099928] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [29618.115303] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29618.122945] iTCO_wdt iTCO_vendor_support [29618.131889] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29618.135892] i2c_smbus crct10dif_pclmul joydev crc32_pclmul [29618.143364] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29618.148927] ipmi_ssif ghash_clmulni_intel mei_me mei [29618.154406] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29618.159450] lpc_ich sha512_ssse3 [29618.178187] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29618.178188] acpi_ipmi mfd_core [29618.181499] [29618.186713] wmi ipmi_si [29618.189852] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29618.191343] ipmi_devintf ipmi_msghandler [29618.193873] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29618.200997] ioatdma acpi_pad [29618.205002] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29618.212126] acpi_power_meter ixgbe [29618.215089] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29618.222214] igb mdio [29618.225698] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29618.232822] crc32c_intel nvme [29618.235094] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29618.242218] ptp nvme_core [29618.245269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29618.253346] i2c_algo_bit pps_core [29618.256048] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29618.261786] dca fuse [29618.265183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29618.272308] [29618.272309] irq event stamp: 158461474 [29618.274581] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29618.281720] hardirqs last enabled at (158461473): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29618.283210] Call Trace: [29618.283212] [29618.286956] hardirqs last disabled at (158461474): [] sysvec_apic_timer_interrupt+0xe/0x90 [29618.294082] ? watchdog_timer_fn+0x1b4/0x220 [29618.304324] softirqs last enabled at (158461472): [] __do_softirq+0x2be/0x371 [29618.306770] ? __hrtimer_run_queues+0x1d3/0x360 [29618.308779] softirqs last disabled at (158461467): [] irq_exit_rcu+0xb0/0xd0 [29618.318610] ? hrtimer_interrupt+0x104/0x240 [29618.322871] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29618.331658] ? __local_bh_enable+0x50/0x60 [29618.336182] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29618.336183] Workqueue: events delayed_work [ceph] [29618.344784] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29618.349046] [29618.349046] RIP: 0010:lock_acquire+0xf0/0x2b0 [29618.358424] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29618.362513] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29618.369983] [29618.374682] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29618.379897] [29618.381390] [29618.381390] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29618.385740] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29618.390696] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29618.409438] ? queued_spin_lock_slowpath+0x18/0x2e0 [29618.411531] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29618.416753] do_raw_spin_lock+0xb5/0xc0 [29618.418848] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29618.420341] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29618.427463] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29618.432770] try_get_cap_refs+0x159/0x880 [ceph] [29618.439890] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29618.444767] __ceph_get_caps+0x4a1/0x760 [ceph] [29618.451889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29618.455729] ? __file_remove_privs+0x89/0x110 [29618.462849] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29618.467633] ? wake_bit_function+0x70/0x70 [29618.474757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29618.479369] ? ceph_write_iter+0x260/0x770 [ceph] [29618.487444] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29618.491968] ceph_write_iter+0x260/0x770 [ceph] [29618.497707] Call Trace: [29618.497709] [29618.502062] ? lock_is_held_type+0xaa/0x120 [29618.509183] ? watchdog_timer_fn+0x1b4/0x220 [29618.513276] vfs_write+0x324/0x4f0 [29618.520398] ? __hrtimer_run_queues+0x1d3/0x360 [29618.525098] ksys_pwrite64+0x65/0xa0 [29618.532222] ? hrtimer_interrupt+0x104/0x240 [29618.536742] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29618.539187] ? __local_bh_enable+0x50/0x60 [29618.541199] do_syscall_64+0x38/0x80 [29618.545376] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29618.549640] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29618.553036] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29618.557561] RIP: 0033:0x7f48f0eeb93b [29618.561130] [29618.565394] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29618.570526] [29618.574616] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29618.578188] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29618.583404] ORIG_RAX: 0000000000000012 [29618.588453] ? lock_acquire+0xf0/0x2b0 [29618.593406] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29618.596978] ? igrab+0x43/0x50 [29618.599073] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29618.617811] ? lock_release+0xc5/0x260 [29618.619909] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29618.625129] _raw_spin_lock+0x30/0x40 [29618.630430] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29618.634262] ? igrab+0x1d/0x50 [29618.638005] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29618.645129] igrab+0x1d/0x50 [29618.648185] [29618.655305] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29618.697323] delayed_work+0x1d8/0x280 [ceph] [29618.701617] ? process_scheduled_works+0x22a/0x510 [29618.706405] process_scheduled_works+0x2a4/0x510 [29618.711019] worker_thread+0x188/0x340 [29618.714769] ? rescuer_thread+0x310/0x310 [29618.718773] kthread+0xf5/0x130 [29618.721919] ? kthread_complete_and_exit+0x20/0x20 [29618.726704] ret_from_fork+0x31/0x40 [29618.730282] ? kthread_complete_and_exit+0x20/0x20 [29618.735066] ret_from_fork_asm+0x11/0x20 [29618.738997] [29645.977715] watchdog: BUG: soft lockup - CPU#0 stuck for 16495s! [dbench:78683] [29645.985014] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29645.989715] watchdog: BUG: soft lockup - CPU#7 stuck for 25492s! [kworker/7:4:78808] [29646.040455] irq event stamp: 49656070 [29646.040456] hardirqs last enabled at (49656069): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29646.048185] Modules linked in: ceph [29646.051843] hardirqs last disabled at (49656070): [] sysvec_apic_timer_interrupt+0xe/0x90 [29646.062001] libceph [29646.065484] softirqs last enabled at (49656032): [] __do_softirq+0x2be/0x371 [29646.075209] dns_resolver fscache [29646.077395] softirqs last disabled at (49656027): [] irq_exit_rcu+0xb0/0xd0 [29646.086077] netfs [29646.089403] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29646.097915] veth nft_chain_nat [29646.099928] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29646.108871] xt_MASQUERADE nf_nat [29646.112009] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29646.119480] nf_conntrack nf_defrag_ipv6 [29646.122790] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29646.128267] nf_defrag_ipv4 [29646.132186] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29646.150923] nft_compat nf_tables [29646.153714] [29646.158930] nfnetlink bridge [29646.162243] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29646.163732] stp llc [29646.166696] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29646.173820] xfs libcrc32c [29646.176004] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29646.183128] sunrpc intel_rapl_msr [29646.185833] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29646.192957] intel_rapl_common [29646.196355] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29646.203479] sb_edac x86_pkg_temp_thermal [29646.206530] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29646.213653] coretemp kvm_intel [29646.217658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29646.225735] kvm irqbypass [29646.228873] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29646.234610] i2c_i801 iTCO_wdt [29646.237313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29646.244437] iTCO_vendor_support i2c_smbus [29646.247490] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29646.254613] crct10dif_pclmul [29646.258703] Call Trace: [29646.265827] joydev crc32_pclmul [29646.268792] [29646.271235] ipmi_ssif ghash_clmulni_intel [29646.274460] ? watchdog_timer_fn+0x1b4/0x220 [29646.276471] mei_me mei [29646.280563] ? __hrtimer_run_queues+0x1d3/0x360 [29646.284825] lpc_ich [29646.287274] ? hrtimer_interrupt+0x104/0x240 [29646.291793] sha512_ssse3 acpi_ipmi [29646.293977] ? __local_bh_enable+0x50/0x60 [29646.298242] mfd_core wmi [29646.301727] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29646.305817] ipmi_si ipmi_devintf [29646.308435] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29646.313651] ipmi_msghandler ioatdma [29646.316963] [29646.321918] acpi_pad acpi_power_meter [29646.325489] [29646.325491] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29646.327587] ixgbe igb [29646.331335] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29646.333430] mdio crc32c_intel [29646.338736] do_raw_spin_lock+0xb5/0xc0 [29646.341091] nvme ptp [29646.345981] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29646.349028] nvme_core i2c_algo_bit [29646.352862] try_get_cap_refs+0x159/0x880 [ceph] [29646.355131] pps_core dca [29646.359918] __ceph_get_caps+0x4a1/0x760 [ceph] [29646.363398] fuse [29646.368014] ? __file_remove_privs+0x89/0x110 [29646.370627] irq event stamp: 158682436 [29646.375150] ? wake_bit_function+0x70/0x70 [29646.377076] hardirqs last enabled at (158682435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29646.381428] ? ceph_write_iter+0x260/0x770 [ceph] [29646.385169] hardirqs last disabled at (158682436): [] sysvec_apic_timer_interrupt+0xe/0x90 [29646.389260] ceph_write_iter+0x260/0x770 [ceph] [29646.399504] softirqs last enabled at (158682434): [] __do_softirq+0x2be/0x371 [29646.404206] ? lock_is_held_type+0xaa/0x120 [29646.414013] softirqs last disabled at (158682429): [] irq_exit_rcu+0xb0/0xd0 [29646.418540] vfs_write+0x324/0x4f0 [29646.427308] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29646.431491] ksys_pwrite64+0x65/0xa0 [29646.440082] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29646.443482] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29646.452858] Workqueue: events delayed_work [ceph] [29646.456429] do_syscall_64+0x38/0x80 [29646.463899] [29646.469030] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29646.473729] RIP: 0010:lock_acquire+0x4a/0x2b0 [29646.477299] RIP: 0033:0x7f48f0eeb93b [29646.478789] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [29646.483833] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29646.488185] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [29646.491753] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29646.510493] [29646.510494] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [29646.529228] ORIG_RAX: 0000000000000012 [29646.534448] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [29646.539664] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29646.541157] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [29646.548280] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29646.552118] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [29646.559243] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29646.566367] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29646.573492] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29646.580615] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29646.587739] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29646.594864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29646.601992] [29646.609118] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29646.639368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29646.646491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29646.653617] Call Trace: [29646.656061] [29646.658072] ? watchdog_timer_fn+0x1b4/0x220 [29646.662336] ? __hrtimer_run_queues+0x1d3/0x360 [29646.666862] ? hrtimer_interrupt+0x104/0x240 [29646.671123] ? __local_bh_enable+0x50/0x60 [29646.675218] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29646.680440] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29646.685402] [29646.687504] [29646.689601] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29646.694910] ? lock_acquire+0x4a/0x2b0 [29646.698659] ? igrab+0x43/0x50 [29646.701710] ? lock_release+0xc5/0x260 [29646.705454] _raw_spin_lock+0x30/0x40 [29646.709121] ? igrab+0x1d/0x50 [29646.712179] igrab+0x1d/0x50 [29646.715058] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29646.720207] delayed_work+0x1d8/0x280 [ceph] [29646.724488] ? process_scheduled_works+0x22a/0x510 [29646.729277] process_scheduled_works+0x2a4/0x510 [29646.733894] worker_thread+0x188/0x340 [29646.737642] ? rescuer_thread+0x310/0x310 [29646.741645] kthread+0xf5/0x130 [29646.744782] ? kthread_complete_and_exit+0x20/0x20 [29646.749567] ret_from_fork+0x31/0x40 [29646.753148] ? kthread_complete_and_exit+0x20/0x20 [29646.757932] ret_from_fork_asm+0x11/0x20 [29646.761855] [29673.978223] watchdog: BUG: soft lockup - CPU#0 stuck for 16522s! [dbench:78683] [29673.985528] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29673.990223] watchdog: BUG: soft lockup - CPU#7 stuck for 25519s! [kworker/7:4:78808] [29674.040978] irq event stamp: 49714156 [29674.048709] Modules linked in: ceph libceph dns_resolver [29674.052365] hardirqs last enabled at (49714155): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29674.057669] fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat [29674.067827] hardirqs last disabled at (49714156): [] sysvec_apic_timer_interrupt+0xe/0x90 [29674.073999] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [29674.083723] softirqs last enabled at (49714120): [] __do_softirq+0x2be/0x371 [29674.092589] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [29674.101272] softirqs last disabled at (49714115): [] irq_exit_rcu+0xb0/0xd0 [29674.107791] sb_edac x86_pkg_temp_thermal coretemp [29674.116301] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29674.116302] kvm_intel [29674.116303] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29674.121084] kvm irqbypass [29674.130029] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29674.132387] i2c_i801 [29674.139857] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29674.142562] iTCO_wdt [29674.148038] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29674.150309] iTCO_vendor_support i2c_smbus [29674.169048] [29674.171317] crct10dif_pclmul [29674.176535] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29674.180626] joydev crc32_pclmul [29674.182117] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29674.185082] ipmi_ssif ghash_clmulni_intel [29674.192206] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29674.195429] mei_me mei [29674.202554] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29674.206643] lpc_ich [29674.213767] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29674.216213] sha512_ssse3 acpi_ipmi [29674.223336] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29674.225522] mfd_core wmi [29674.232643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29674.236130] ipmi_si ipmi_devintf [29674.244205] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29674.246822] ipmi_msghandler [29674.252561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29674.255872] ioatdma acpi_pad [29674.262996] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29674.265874] acpi_power_meter ixgbe [29674.272999] Call Trace: [29674.275962] igb mdio [29674.283086] [29674.286570] crc32c_intel [29674.289015] ? watchdog_timer_fn+0x1b4/0x220 [29674.291285] nvme ptp [29674.293297] ? __hrtimer_run_queues+0x1d3/0x360 [29674.295912] nvme_core i2c_algo_bit [29674.300179] ? hrtimer_interrupt+0x104/0x240 [29674.302448] pps_core [29674.306970] ? __local_bh_enable+0x50/0x60 [29674.310455] dca fuse [29674.314722] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29674.316991] [29674.316992] irq event stamp: 158903204 [29674.321080] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29674.323351] hardirqs last enabled at (158903203): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29674.328570] [29674.330060] hardirqs last disabled at (158903204): [] sysvec_apic_timer_interrupt+0xe/0x90 [29674.333803] [29674.338762] softirqs last enabled at (158903202): [] __do_softirq+0x2be/0x371 [29674.349008] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29674.351104] softirqs last disabled at (158903197): [] irq_exit_rcu+0xb0/0xd0 [29674.360917] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29674.363010] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29674.371784] do_raw_spin_lock+0xb5/0xc0 [29674.377086] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29674.385686] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29674.390553] Workqueue: events delayed_work [ceph] [29674.399934] try_get_cap_refs+0x159/0x880 [ceph] [29674.403763] [29674.403764] RIP: 0010:lock_acquire+0xf0/0x2b0 [29674.411239] __ceph_get_caps+0x4a1/0x760 [ceph] [29674.416018] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29674.420719] ? __file_remove_privs+0x89/0x110 [29674.425325] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29674.426816] ? wake_bit_function+0x70/0x70 [29674.431167] [29674.431168] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29674.435695] ? ceph_write_iter+0x260/0x770 [ceph] [29674.454429] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29674.458781] ceph_write_iter+0x260/0x770 [ceph] [29674.463997] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29674.468093] ? lock_is_held_type+0xaa/0x120 [29674.469579] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29674.476706] vfs_write+0x324/0x4f0 [29674.481400] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29674.488529] ksys_pwrite64+0x65/0xa0 [29674.493049] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29674.500173] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29674.504349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29674.511474] do_syscall_64+0x38/0x80 [29674.514871] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29674.521998] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29674.525566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29674.533643] RIP: 0033:0x7f48f0eeb93b [29674.538773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29674.544511] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29674.548084] Call Trace: [29674.555206] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29674.560253] [29674.567376] ORIG_RAX: 0000000000000012 [29674.570947] ? watchdog_timer_fn+0x1b4/0x220 [29674.578070] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29674.596810] ? __hrtimer_run_queues+0x1d3/0x360 [29674.599252] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29674.604473] ? hrtimer_interrupt+0x104/0x240 [29674.606480] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29674.610311] ? __local_bh_enable+0x50/0x60 [29674.614574] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29674.621701] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29674.626221] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29674.633346] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29674.637634] [29674.644752] [29674.677549] [29674.679645] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29674.684953] ? lock_acquire+0xf0/0x2b0 [29674.688704] ? igrab+0x43/0x50 [29674.691763] ? lock_release+0xc5/0x260 [29674.695508] _raw_spin_lock+0x30/0x40 [29674.699164] ? igrab+0x1d/0x50 [29674.702213] igrab+0x1d/0x50 [29674.705095] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29674.710242] delayed_work+0x1d8/0x280 [ceph] [29674.714525] ? process_scheduled_works+0x22a/0x510 [29674.719312] process_scheduled_works+0x2a4/0x510 [29674.723928] worker_thread+0x188/0x340 [29674.727677] ? rescuer_thread+0x310/0x310 [29674.731682] kthread+0xf5/0x130 [29674.734818] ? kthread_complete_and_exit+0x20/0x20 [29674.739604] ret_from_fork+0x31/0x40 [29674.743199] ? kthread_complete_and_exit+0x20/0x20 [29674.747983] ret_from_fork_asm+0x11/0x20 [29674.751922] [29701.978732] watchdog: BUG: soft lockup - CPU#0 stuck for 16548s! [dbench:78683] [29701.986033] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29701.990731] watchdog: BUG: soft lockup - CPU#7 stuck for 25545s! [kworker/7:4:78808] [29702.041473] irq event stamp: 49772178 [29702.049204] Modules linked in: ceph libceph dns_resolver [29702.052863] hardirqs last enabled at (49772177): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29702.058166] fscache netfs veth [29702.068325] hardirqs last disabled at (49772178): [] sysvec_apic_timer_interrupt+0xe/0x90 [29702.071460] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [29702.081187] softirqs last enabled at (49772140): [] __do_softirq+0x2be/0x371 [29702.086835] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [29702.095520] softirqs last disabled at (49772135): [] irq_exit_rcu+0xb0/0xd0 [29702.101432] nfnetlink bridge stp [29702.109942] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29702.113252] llc xfs [29702.122196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29702.124380] libcrc32c sunrpc intel_rapl_msr intel_rapl_common [29702.131851] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29702.137677] sb_edac x86_pkg_temp_thermal [29702.143153] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29702.143153] coretemp kvm_intel [29702.147156] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29702.165895] kvm irqbypass [29702.169033] [29702.169034] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29702.174249] i2c_i801 iTCO_wdt [29702.176953] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29702.178444] iTCO_vendor_support i2c_smbus [29702.185567] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29702.188619] crct10dif_pclmul [29702.195742] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29702.199833] joydev crc32_pclmul [29702.206959] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29702.209922] ipmi_ssif ghash_clmulni_intel [29702.217047] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29702.220270] mei_me mei [29702.227410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29702.231503] lpc_ich sha512_ssse3 [29702.239580] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29702.242023] acpi_ipmi mfd_core [29702.247761] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29702.251072] wmi ipmi_si [29702.258196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29702.261334] ipmi_devintf ipmi_msghandler [29702.268458] Call Trace: [29702.270989] ioatdma [29702.278130] [29702.282133] acpi_pad acpi_power_meter [29702.284578] ? watchdog_timer_fn+0x1b4/0x220 [29702.286762] ixgbe igb [29702.288774] ? __hrtimer_run_queues+0x1d3/0x360 [29702.292518] mdio crc32c_intel [29702.296784] ? hrtimer_interrupt+0x104/0x240 [29702.299140] nvme ptp [29702.303662] ? __local_bh_enable+0x50/0x60 [29702.306713] nvme_core i2c_algo_bit [29702.310980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29702.313249] pps_core dca [29702.317340] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29702.320822] fuse [29702.326042] [29702.328658] irq event stamp: 159124136 [29702.333614] [29702.335540] hardirqs last enabled at (159124135): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29702.337637] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29702.341382] hardirqs last disabled at (159124136): [] sysvec_apic_timer_interrupt+0xe/0x90 [29702.343481] ? queued_spin_lock_slowpath+0x18/0x2e0 [29702.353722] softirqs last enabled at (159124134): [] __do_softirq+0x2be/0x371 [29702.359028] do_raw_spin_lock+0xb5/0xc0 [29702.368836] softirqs last disabled at (159124129): [] irq_exit_rcu+0xb0/0xd0 [29702.373711] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29702.382480] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29702.386312] try_get_cap_refs+0x159/0x880 [ceph] [29702.394908] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29702.399697] __ceph_get_caps+0x4a1/0x760 [ceph] [29702.409070] Workqueue: events delayed_work [ceph] [29702.413683] ? __file_remove_privs+0x89/0x110 [29702.421151] [29702.425676] ? wake_bit_function+0x70/0x70 [29702.430371] RIP: 0010:lock_acquire+0xf0/0x2b0 [29702.434724] ? ceph_write_iter+0x260/0x770 [ceph] [29702.436214] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29702.440305] ceph_write_iter+0x260/0x770 [ceph] [29702.444656] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29702.449356] ? lock_is_held_type+0xaa/0x120 [29702.468090] [29702.468090] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29702.472617] vfs_write+0x324/0x4f0 [29702.477831] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29702.482013] ksys_pwrite64+0x65/0xa0 [29702.483498] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29702.490625] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29702.494022] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29702.501147] do_syscall_64+0x38/0x80 [29702.504717] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29702.511840] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29702.516972] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29702.524096] RIP: 0033:0x7f48f0eeb93b [29702.527665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29702.534790] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29702.539834] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29702.547912] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29702.551481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29702.557220] ORIG_RAX: 0000000000000012 [29702.575958] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29702.583082] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29702.588299] Call Trace: [29702.588301] [29702.595432] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29702.599264] ? watchdog_timer_fn+0x1b4/0x220 [29702.606387] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29702.613511] ? __hrtimer_run_queues+0x1d3/0x360 [29702.615956] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29702.617968] ? hrtimer_interrupt+0x104/0x240 [29702.625089] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29702.629352] ? __local_bh_enable+0x50/0x60 [29702.636483] [29702.641004] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29702.671034] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29702.675991] [29702.678088] [29702.680185] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29702.685493] ? lock_acquire+0xf0/0x2b0 [29702.689243] ? igrab+0x43/0x50 [29702.692293] ? lock_release+0xc5/0x260 [29702.696038] _raw_spin_lock+0x30/0x40 [29702.699693] ? igrab+0x1d/0x50 [29702.702744] igrab+0x1d/0x50 [29702.705625] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29702.710773] delayed_work+0x1d8/0x280 [ceph] [29702.715053] ? process_scheduled_works+0x22a/0x510 [29702.719844] process_scheduled_works+0x2a4/0x510 [29702.724462] worker_thread+0x188/0x340 [29702.728208] ? rescuer_thread+0x310/0x310 [29702.732211] kthread+0xf5/0x130 [29702.735350] ? kthread_complete_and_exit+0x20/0x20 [29702.740139] ret_from_fork+0x31/0x40 [29702.743712] ? kthread_complete_and_exit+0x20/0x20 [29702.748498] ret_from_fork_asm+0x11/0x20 [29702.752428] [29729.979241] watchdog: BUG: soft lockup - CPU#0 stuck for 16574s! [dbench:78683] [29729.986547] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29729.991240] watchdog: BUG: soft lockup - CPU#7 stuck for 25571s! [kworker/7:4:78808] [29730.041998] irq event stamp: 49830180 [29730.049728] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [29730.053385] hardirqs last enabled at (49830179): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29730.061559] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [29730.071715] hardirqs last disabled at (49830180): [] sysvec_apic_timer_interrupt+0xe/0x90 [29730.071718] nft_compat [29730.071719] softirqs last enabled at (49830142): [] __do_softirq+0x2be/0x371 [29730.078753] nf_tables nfnetlink [29730.088477] softirqs last disabled at (49830137): [] irq_exit_rcu+0xb0/0xd0 [29730.090930] bridge stp [29730.099634] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29730.102854] llc xfs [29730.111365] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29730.113809] libcrc32c sunrpc [29730.122753] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29730.124939] intel_rapl_msr intel_rapl_common [29730.132410] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29730.135372] sb_edac x86_pkg_temp_thermal [29730.140850] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29730.145200] coretemp kvm_intel [29730.163939] [29730.163940] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29730.167942] kvm irqbypass [29730.173159] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29730.176298] i2c_i801 iTCO_wdt [29730.177788] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29730.184924] iTCO_vendor_support i2c_smbus [29730.187637] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29730.194757] crct10dif_pclmul [29730.197809] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29730.204932] joydev [29730.209023] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29730.216148] crc32_pclmul ipmi_ssif [29730.219111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29730.226235] ghash_clmulni_intel mei_me [29730.228333] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29730.236411] mei lpc_ich [29730.239894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29730.245649] sha512_ssse3 [29730.249481] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29730.256604] acpi_ipmi mfd_core [29730.259134] Call Trace: [29730.259136] [29730.266260] wmi ipmi_si [29730.268878] ? watchdog_timer_fn+0x1b4/0x220 [29730.276001] ipmi_devintf ipmi_msghandler [29730.279140] ? __hrtimer_run_queues+0x1d3/0x360 [29730.281583] ioatdma acpi_pad [29730.283596] ? hrtimer_interrupt+0x104/0x240 [29730.286122] acpi_power_meter ixgbe [29730.290387] ? __local_bh_enable+0x50/0x60 [29730.294391] igb [29730.298923] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29730.301889] mdio crc32c_intel [29730.306153] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29730.309637] nvme [29730.313728] [29730.315565] ptp nvme_core [29730.320783] [29730.323833] i2c_algo_bit pps_core [29730.328791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29730.330715] dca fuse [29730.332816] ? queued_spin_lock_slowpath+0x16/0x2e0 [29730.335517] irq event stamp: 159345126 [29730.337617] do_raw_spin_lock+0xb5/0xc0 [29730.341029] hardirqs last enabled at (159345125): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29730.346334] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29730.348602] hardirqs last disabled at (159345126): [] sysvec_apic_timer_interrupt+0xe/0x90 [29730.353478] try_get_cap_refs+0x159/0x880 [ceph] [29730.357217] softirqs last enabled at (159345124): [] __do_softirq+0x2be/0x371 [29730.361053] __ceph_get_caps+0x4a1/0x760 [ceph] [29730.371292] softirqs last disabled at (159345119): [] irq_exit_rcu+0xb0/0xd0 [29730.376080] ? __file_remove_privs+0x89/0x110 [29730.385886] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29730.390499] ? wake_bit_function+0x70/0x70 [29730.399269] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29730.403795] ? ceph_write_iter+0x260/0x770 [ceph] [29730.412393] Workqueue: events delayed_work [ceph] [29730.416741] ceph_write_iter+0x260/0x770 [ceph] [29730.426120] [29730.426120] RIP: 0010:lock_acquire+0x85/0x2b0 [29730.430213] ? lock_is_held_type+0xaa/0x120 [29730.437679] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [29730.442380] vfs_write+0x324/0x4f0 [29730.447076] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [29730.451605] ksys_pwrite64+0x65/0xa0 [29730.453090] [29730.453091] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [29730.457441] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29730.461618] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [29730.480376] do_syscall_64+0x38/0x80 [29730.483771] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [29730.488989] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29730.492558] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [29730.494051] RIP: 0033:0x7f48f0eeb93b [29730.501173] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29730.506305] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29730.513429] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29730.516999] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29730.524124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29730.529167] ORIG_RAX: 0000000000000012 [29730.536293] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29730.539861] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29730.546985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29730.565723] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29730.573801] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29730.579018] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29730.584757] Call Trace: [29730.584759] [29730.588588] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29730.595711] ? watchdog_timer_fn+0x1b4/0x220 [29730.602835] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29730.609960] ? __hrtimer_run_queues+0x1d3/0x360 [29730.617089] [29730.624211] ? hrtimer_interrupt+0x104/0x240 [29730.665290] ? __local_bh_enable+0x50/0x60 [29730.669381] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29730.674599] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29730.679555] [29730.681653] [29730.683751] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29730.689057] ? lock_acquire+0x85/0x2b0 [29730.692808] ? igrab+0x43/0x50 [29730.695857] ? lock_release+0xc5/0x260 [29730.699602] _raw_spin_lock+0x30/0x40 [29730.703259] ? igrab+0x1d/0x50 [29730.706308] igrab+0x1d/0x50 [29730.709188] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29730.714339] delayed_work+0x1d8/0x280 [ceph] [29730.718619] ? process_scheduled_works+0x22a/0x510 [29730.723425] process_scheduled_works+0x2a4/0x510 [29730.728042] worker_thread+0x188/0x340 [29730.731792] ? rescuer_thread+0x310/0x310 [29730.735804] kthread+0xf5/0x130 [29730.738948] ? kthread_complete_and_exit+0x20/0x20 [29730.743733] ret_from_fork+0x31/0x40 [29730.747313] ? kthread_complete_and_exit+0x20/0x20 [29730.752098] ret_from_fork_asm+0x11/0x20 [29730.756028] [29757.979748] watchdog: BUG: soft lockup - CPU#0 stuck for 16600s! [dbench:78683] [29757.987051] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29757.991748] watchdog: BUG: soft lockup - CPU#7 stuck for 25597s! [kworker/7:4:78808] [29758.042494] irq event stamp: 49888266 [29758.042495] hardirqs last enabled at (49888265): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29758.050224] Modules linked in: [29758.053880] hardirqs last disabled at (49888266): [] sysvec_apic_timer_interrupt+0xe/0x90 [29758.064039] ceph [29758.067090] softirqs last enabled at (49888256): [] __do_softirq+0x2be/0x371 [29758.076814] libceph dns_resolver [29758.078737] softirqs last disabled at (49888251): [] irq_exit_rcu+0xb0/0xd0 [29758.087423] fscache netfs [29758.090734] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29758.099242] veth nft_chain_nat [29758.101947] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29758.110890] xt_MASQUERADE nf_nat [29758.114029] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29758.121500] nf_conntrack nf_defrag_ipv6 [29758.124810] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29758.130289] nf_defrag_ipv4 nft_compat [29758.134205] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29758.152943] nf_tables nfnetlink [29758.156686] [29758.156687] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29758.161904] bridge [29758.165145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29758.166637] stp [29758.173762] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29758.175858] llc xfs [29758.182982] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29758.184819] libcrc32c sunrpc [29758.191944] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29758.194143] intel_rapl_msr intel_rapl_common [29758.201269] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29758.204234] sb_edac x86_pkg_temp_thermal [29758.211357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29758.215708] coretemp kvm_intel [29758.223787] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29758.227790] kvm irqbypass [29758.233527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29758.236664] i2c_i801 iTCO_wdt [29758.243787] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29758.246493] iTCO_vendor_support i2c_smbus [29758.253617] Call Trace: [29758.253619] [29758.256668] crct10dif_pclmul joydev [29758.263793] ? watchdog_timer_fn+0x1b4/0x220 [29758.267882] crc32_pclmul ipmi_ssif [29758.270329] ? __hrtimer_run_queues+0x1d3/0x360 [29758.272336] ghash_clmulni_intel mei_me [29758.275912] ? hrtimer_interrupt+0x104/0x240 [29758.280171] mei lpc_ich [29758.283656] ? __local_bh_enable+0x50/0x60 [29758.288179] sha512_ssse3 acpi_ipmi [29758.292015] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29758.296275] mfd_core wmi [29758.298807] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29758.302896] ipmi_si ipmi_devintf [29758.306381] [29758.311599] ipmi_msghandler ioatdma [29758.314216] [29758.314217] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29758.319173] acpi_pad [29758.322487] ? queued_spin_lock_slowpath+0x16/0x2e0 [29758.324580] acpi_power_meter ixgbe [29758.328155] do_raw_spin_lock+0xb5/0xc0 [29758.330248] igb mdio [29758.335555] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29758.337823] crc32c_intel nvme [29758.342697] try_get_cap_refs+0x159/0x880 [ceph] [29758.346178] ptp nvme_core [29758.350014] __ceph_get_caps+0x4a1/0x760 [ceph] [29758.352279] i2c_algo_bit pps_core [29758.357068] ? __file_remove_privs+0x89/0x110 [29758.360115] dca fuse [29758.364725] ? wake_bit_function+0x70/0x70 [29758.367430] [29758.367430] irq event stamp: 159565998 [29758.371955] ? ceph_write_iter+0x260/0x770 [ceph] [29758.375351] hardirqs last enabled at (159565997): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29758.379702] ceph_write_iter+0x260/0x770 [ceph] [29758.381971] hardirqs last disabled at (159565998): [] sysvec_apic_timer_interrupt+0xe/0x90 [29758.386067] ? lock_is_held_type+0xaa/0x120 [29758.387555] softirqs last enabled at (159565996): [] __do_softirq+0x2be/0x371 [29758.391301] vfs_write+0x324/0x4f0 [29758.395994] softirqs last disabled at (159565991): [] irq_exit_rcu+0xb0/0xd0 [29758.406244] ksys_pwrite64+0x65/0xa0 [29758.410764] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29758.420574] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29758.424753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29758.433523] do_syscall_64+0x38/0x80 [29758.436921] Workqueue: events delayed_work [ceph] [29758.445520] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29758.449088] [29758.458467] RIP: 0033:0x7f48f0eeb93b [29758.463596] RIP: 0010:lock_acquire+0xf0/0x2b0 [29758.471068] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29758.474637] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29758.479336] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29758.484378] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29758.485869] ORIG_RAX: 0000000000000012 [29758.489442] [29758.489442] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29758.493792] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29758.512529] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29758.531268] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29758.536483] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29758.541701] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29758.545534] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29758.547024] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29758.554159] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29758.561280] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29758.568406] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29758.575550] [29758.582671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29758.634289] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29758.641415] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29758.648539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29758.655663] Call Trace: [29758.658109] [29758.660117] ? watchdog_timer_fn+0x1b4/0x220 [29758.664383] ? __hrtimer_run_queues+0x1d3/0x360 [29758.668908] ? hrtimer_interrupt+0x104/0x240 [29758.673179] ? __local_bh_enable+0x50/0x60 [29758.677280] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29758.682504] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29758.687462] [29758.689557] [29758.691657] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29758.696963] ? lock_acquire+0xf0/0x2b0 [29758.700715] ? igrab+0x43/0x50 [29758.703764] ? lock_release+0xc5/0x260 [29758.707510] _raw_spin_lock+0x30/0x40 [29758.711172] ? igrab+0x1d/0x50 [29758.714224] igrab+0x1d/0x50 [29758.717103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29758.722254] delayed_work+0x1d8/0x280 [ceph] [29758.726534] ? process_scheduled_works+0x22a/0x510 [29758.731323] process_scheduled_works+0x2a4/0x510 [29758.735939] worker_thread+0x188/0x340 [29758.739689] ? rescuer_thread+0x310/0x310 [29758.743700] kthread+0xf5/0x130 [29758.746839] ? kthread_complete_and_exit+0x20/0x20 [29758.751623] ret_from_fork+0x31/0x40 [29758.755202] ? kthread_complete_and_exit+0x20/0x20 [29758.759987] ret_from_fork_asm+0x11/0x20 [29758.763919] [29785.980257] watchdog: BUG: soft lockup - CPU#0 stuck for 16626s! [dbench:78683] [29785.987556] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29785.992257] watchdog: BUG: soft lockup - CPU#7 stuck for 25623s! [kworker/7:4:78808] [29786.042997] irq event stamp: 49946402 [29786.042998] hardirqs last enabled at (49946401): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29786.050729] Modules linked in: [29786.054387] hardirqs last disabled at (49946402): [] sysvec_apic_timer_interrupt+0xe/0x90 [29786.064544] ceph [29786.067595] softirqs last enabled at (49946364): [] __do_softirq+0x2be/0x371 [29786.077319] libceph [29786.079243] softirqs last disabled at (49946359): [] irq_exit_rcu+0xb0/0xd0 [29786.087933] dns_resolver [29786.090118] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29786.098630] fscache netfs [29786.101265] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29786.110209] veth nft_chain_nat [29786.112912] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29786.120384] xt_MASQUERADE nf_nat [29786.123522] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29786.129000] nf_conntrack [29786.132310] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29786.151047] nf_defrag_ipv6 nf_defrag_ipv4 [29786.153664] [29786.153665] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29786.158881] nft_compat nf_tables [29786.162972] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29786.164464] nfnetlink bridge [29786.171588] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29786.174899] stp llc [29786.182022] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29786.184987] xfs libcrc32c [29786.192111] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29786.194294] sunrpc intel_rapl_msr [29786.201420] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29786.204122] intel_rapl_common sb_edac [29786.211248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29786.214654] x86_pkg_temp_thermal [29786.222731] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29786.226473] coretemp kvm_intel [29786.232211] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29786.235522] kvm irqbypass [29786.242655] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29786.245792] i2c_i801 iTCO_wdt [29786.252917] Call Trace: [29786.252918] [29786.255622] iTCO_vendor_support [29786.262745] ? watchdog_timer_fn+0x1b4/0x220 [29786.265795] i2c_smbus crct10dif_pclmul [29786.268241] ? __hrtimer_run_queues+0x1d3/0x360 [29786.270250] joydev crc32_pclmul [29786.273478] ? hrtimer_interrupt+0x104/0x240 [29786.277740] ipmi_ssif ghash_clmulni_intel [29786.281570] ? __local_bh_enable+0x50/0x60 [29786.286092] mei_me mei [29786.289319] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29786.293581] lpc_ich [29786.297673] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29786.301763] sha512_ssse3 acpi_ipmi [29786.304207] [29786.309425] mfd_core wmi [29786.311609] [29786.311611] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29786.316566] ipmi_si ipmi_devintf [29786.320053] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29786.322147] ipmi_msghandler [29786.324768] do_raw_spin_lock+0xb5/0xc0 [29786.326862] ioatdma acpi_pad [29786.332167] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29786.335478] acpi_power_meter ixgbe [29786.340350] try_get_cap_refs+0x159/0x880 [ceph] [29786.343224] igb mdio [29786.347062] __ceph_get_caps+0x4a1/0x760 [ceph] [29786.350020] crc32c_intel nvme [29786.354809] ? __file_remove_privs+0x89/0x110 [29786.358289] ptp nvme_core [29786.362901] ? wake_bit_function+0x70/0x70 [29786.365170] i2c_algo_bit pps_core [29786.369697] ? ceph_write_iter+0x260/0x770 [ceph] [29786.372745] dca fuse [29786.377096] ceph_write_iter+0x260/0x770 [ceph] [29786.379800] [29786.379800] irq event stamp: 159786874 [29786.383893] ? lock_is_held_type+0xaa/0x120 [29786.387289] hardirqs last enabled at (159786873): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29786.391988] vfs_write+0x324/0x4f0 [29786.394255] hardirqs last disabled at (159786874): [] sysvec_apic_timer_interrupt+0xe/0x90 [29786.398784] ksys_pwrite64+0x65/0xa0 [29786.400270] softirqs last enabled at (159786872): [] __do_softirq+0x2be/0x371 [29786.404015] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29786.408192] softirqs last disabled at (159786867): [] irq_exit_rcu+0xb0/0xd0 [29786.418437] do_syscall_64+0x38/0x80 [29786.421834] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29786.431660] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29786.435233] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29786.435234] Workqueue: events delayed_work [ceph] [29786.444003] RIP: 0033:0x7f48f0eeb93b [29786.449135] [29786.449135] RIP: 0010:lock_acquire+0xf0/0x2b0 [29786.457731] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29786.461302] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29786.470680] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29786.475724] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29786.483195] ORIG_RAX: 0000000000000012 [29786.487893] [29786.487893] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29786.491464] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29786.492954] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29786.497303] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29786.516042] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29786.534781] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29786.539998] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29786.541635] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [29786.541638] rcu: 0-....: (17447629 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=8728232 [29786.541659] rcu: 7-....: (17331942 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/17330631 fqs=8728232 [29786.541663] rcu: (detected by 3, t=17849599 jiffies, g=1309609, q=528840 ncpus=8) [29786.541666] Sending NMI from CPU 3 to CPUs 0: [29786.541669] NMI backtrace for cpu 0 [29786.541670] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29786.541671] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29786.541672] RIP: 0010:delay_tsc+0x4d/0x90 [29786.541675] Code: c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e 65 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 48 89 c2 4c 29 c2 48 39 fa 72 c8 65 ff [29786.541676] RSP: 0018:ffffc90000003b50 EFLAGS: 00000046 [29786.541677] RAX: 00000000ebb9911b RBX: 0000000000000000 RCX: 0000000000000000 [29786.541678] RDX: 000000000000606a RSI: 0000000000000000 RDI: 0000000000000dac [29786.541679] RBP: 0000000000002676 R08: 0000606aebb988d8 R09: 0000000000000000 [29786.541680] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [29786.541680] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [29786.541681] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29786.541682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29786.541683] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29786.541684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29786.541685] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29786.541686] Call Trace: [29786.541687] [29786.541687] ? nmi_cpu_backtrace+0xad/0x120 [29786.541691] ? nmi_cpu_backtrace_handler+0x11/0x20 [29786.541692] ? nmi_handle+0xd4/0x260 [29786.541697] ? default_do_nmi+0x49/0xf0 [29786.541698] ? exc_nmi+0x11e/0x150 [29786.541700] ? end_repeat_nmi+0x16/0x67 [29786.541704] ? delay_tsc+0x4d/0x90 [29786.541706] ? delay_tsc+0x4d/0x90 [29786.541708] ? delay_tsc+0x4d/0x90 [29786.541710] [29786.541710] [29786.541710] wait_for_lsr+0x27/0x70 [29786.541714] serial8250_console_write+0x341/0x3f0 [29786.541717] ? lock_release+0xc5/0x260 [29786.541720] console_flush_all+0x227/0x5a0 [29786.541722] ? console_flush_all+0x1ff/0x5a0 [29786.541723] ? console_flush_all+0x1ff/0x5a0 [29786.541726] console_unlock+0x7b/0x120 [29786.541728] vprintk_emit+0x246/0x340 [29786.541731] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [29786.541733] _printk+0x5c/0x80 [29786.541735] __show_regs+0x93/0x2e0 [29786.541738] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [29786.541739] ? unwind_next_frame+0x379/0x830 [29786.541743] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [29786.541744] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [29786.541746] show_trace_log_lvl+0x2b5/0x3f0 [29786.541751] ? touch_softlockup_watchdog+0x20/0x20 [29786.541753] watchdog_timer_fn+0x1b4/0x220 [29786.541755] __hrtimer_run_queues+0x1d3/0x360 [29786.541758] hrtimer_interrupt+0x104/0x240 [29786.541759] ? __local_bh_enable+0x50/0x60 [29786.541762] __sysvec_apic_timer_interrupt+0x61/0x160 [29786.541765] sysvec_apic_timer_interrupt+0x6d/0x90 [29786.541767] [29786.541767] [29786.541768] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29786.541770] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29786.541772] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29786.541774] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29786.541775] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29786.541775] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29786.541776] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29786.541777] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29786.541778] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29786.541781] do_raw_spin_lock+0xb5/0xc0 [29786.541783] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29786.541797] try_get_cap_refs+0x159/0x880 [ceph] [29786.541811] __ceph_get_caps+0x4a1/0x760 [ceph] [29786.541824] ? __file_remove_privs+0x89/0x110 [29786.541826] ? wake_bit_function+0x70/0x70 [29786.541828] ? ceph_write_iter+0x260/0x770 [ceph] [29786.541839] ceph_write_iter+0x260/0x770 [ceph] [29786.541850] ? lock_is_held_type+0xaa/0x120 [29786.541853] vfs_write+0x324/0x4f0 [29786.541857] ksys_pwrite64+0x65/0xa0 [29786.541859] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29786.541862] do_syscall_64+0x38/0x80 [29786.541864] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29786.541865] RIP: 0033:0x7f48f0eeb93b [29786.541866] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29786.541867] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [29786.541869] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29786.541869] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29786.541870] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29786.541871] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29786.541871] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29786.541874] [29786.542668] Sending NMI from CPU 3 to CPUs 7: [29786.542671] NMI backtrace for cpu 7 [29786.542671] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29786.542673] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29786.542674] Workqueue: events delayed_work [ceph] [29786.542693] RIP: 0010:vprintk_emit+0x1ee/0x340 [29786.542696] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [29786.542697] RSP: 0018:ffffc90000264db8 EFLAGS: 00000002 [29786.542699] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [29786.542700] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb578 [29786.542701] RBP: 0000000000000041 R08: 0000000000005ef4 R09: 0000000000000004 [29786.542702] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [29786.542703] R13: 0000000000000000 R14: ffffffff82188750 R15: ffffc90000264e08 [29786.542704] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29786.542706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29786.542707] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29786.542709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29786.542710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29786.542711] Call Trace: [29786.542712] [29786.542713] ? nmi_cpu_backtrace+0xad/0x120 [29786.542717] ? nmi_cpu_backtrace_handler+0x11/0x20 [29786.542719] ? nmi_handle+0xd4/0x260 [29786.542724] ? default_do_nmi+0x49/0xf0 [29786.542726] ? exc_nmi+0x11e/0x150 [29786.542729] ? end_repeat_nmi+0x16/0x67 [29786.542735] ? vprintk_emit+0x1ee/0x340 [29786.542738] ? vprintk_emit+0x1ee/0x340 [29786.542741] ? vprintk_emit+0x1ee/0x340 [29786.542744] [29786.542744] [29786.542746] ? touch_softlockup_watchdog+0x20/0x20 [29786.542748] _printk+0x5c/0x80 [29786.542753] __show_regs+0xae/0x2e0 [29786.542757] ? touch_softlockup_watchdog+0x20/0x20 [29786.542759] show_regs+0x38/0x60 [29786.542762] watchdog_timer_fn+0x1b4/0x220 [29786.542764] __hrtimer_run_queues+0x1d3/0x360 [29786.542769] hrtimer_interrupt+0x104/0x240 [29786.542771] ? __local_bh_enable+0x50/0x60 [29786.542776] __sysvec_apic_timer_interrupt+0x61/0x160 [29786.542779] sysvec_apic_timer_interrupt+0x6d/0x90 [29786.542782] [29786.542783] [29786.542784] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29786.542786] RIP: 0010:lock_acquire+0xf0/0x2b0 [29786.542788] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29786.542790] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29786.542791] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29786.542793] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29786.542794] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29786.542795] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29786.542796] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29786.542802] ? igrab+0x43/0x50 [29786.542805] ? lock_release+0xc5/0x260 [29786.542808] _raw_spin_lock+0x30/0x40 [29786.542811] ? igrab+0x1d/0x50 [29786.542812] igrab+0x1d/0x50 [29786.542814] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29786.542834] delayed_work+0x1d8/0x280 [ceph] [29786.542854] ? process_scheduled_works+0x22a/0x510 [29786.542856] process_scheduled_works+0x2a4/0x510 [29786.542863] worker_thread+0x188/0x340 [29786.542866] ? rescuer_thread+0x310/0x310 [29786.542869] kthread+0xf5/0x130 [29786.542872] ? kthread_complete_and_exit+0x20/0x20 [29786.542875] ret_from_fork+0x31/0x40 [29786.542878] ? kthread_complete_and_exit+0x20/0x20 [29786.542881] ret_from_fork_asm+0x11/0x20 [29786.542888] [29786.545214] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29786.549045] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29786.550537] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29786.557661] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29786.564790] [29786.571909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29787.495801] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29787.502925] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29787.510050] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29787.517174] Call Trace: [29787.519619] [29787.521628] ? watchdog_timer_fn+0x1b4/0x220 [29787.525893] ? __hrtimer_run_queues+0x1d3/0x360 [29787.530420] ? hrtimer_interrupt+0x104/0x240 [29787.534688] ? __local_bh_enable+0x50/0x60 [29787.538782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29787.544007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29787.548963] [29787.551060] [29787.553159] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29787.558467] ? lock_acquire+0xf0/0x2b0 [29787.562216] ? igrab+0x43/0x50 [29787.565266] ? lock_release+0xc5/0x260 [29787.569011] _raw_spin_lock+0x30/0x40 [29787.572680] ? igrab+0x1d/0x50 [29787.575734] igrab+0x1d/0x50 [29787.578613] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29787.583763] delayed_work+0x1d8/0x280 [ceph] [29787.588043] ? process_scheduled_works+0x22a/0x510 [29787.592825] process_scheduled_works+0x2a4/0x510 [29787.597441] worker_thread+0x188/0x340 [29787.601191] ? rescuer_thread+0x310/0x310 [29787.605195] kthread+0xf5/0x130 [29787.608331] ? kthread_complete_and_exit+0x20/0x20 [29787.613116] ret_from_fork+0x31/0x40 [29787.616685] ? kthread_complete_and_exit+0x20/0x20 [29787.621471] ret_from_fork_asm+0x11/0x20 [29787.625392] [29813.980765] watchdog: BUG: soft lockup - CPU#0 stuck for 16652s! [dbench:78683] [29813.988070] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29813.992765] watchdog: BUG: soft lockup - CPU#7 stuck for 25649s! [kworker/7:4:78808] [29814.043521] irq event stamp: 50004576 [29814.043522] hardirqs last enabled at (50004575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29814.051251] Modules linked in: [29814.054909] hardirqs last disabled at (50004576): [] sysvec_apic_timer_interrupt+0xe/0x90 [29814.065066] ceph [29814.068117] softirqs last enabled at (50004538): [] __do_softirq+0x2be/0x371 [29814.077841] libceph [29814.079764] softirqs last disabled at (50004533): [] irq_exit_rcu+0xb0/0xd0 [29814.088450] dns_resolver fscache [29814.090636] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29814.099161] netfs veth [29814.102472] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29814.111417] nft_chain_nat [29814.113879] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29814.121349] xt_MASQUERADE nf_nat [29814.124052] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29814.129529] nf_conntrack nf_defrag_ipv6 [29814.132840] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29814.151579] nf_defrag_ipv4 [29814.155496] [29814.160713] nft_compat nf_tables [29814.163502] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29814.164993] nfnetlink [29814.168305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29814.175443] bridge [29814.177805] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29814.184927] stp [29814.187025] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29814.194162] llc [29814.195996] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29814.203121] xfs libcrc32c [29814.204958] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29814.212080] sunrpc intel_rapl_msr [29814.214785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29814.222864] intel_rapl_common [29814.226261] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29814.231996] sb_edac x86_pkg_temp_thermal [29814.235049] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29814.242173] coretemp [29814.246177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29814.253301] kvm_intel [29814.255572] Call Trace: [29814.262697] kvm irqbypass [29814.265056] [29814.267498] i2c_i801 iTCO_wdt [29814.270204] ? watchdog_timer_fn+0x1b4/0x220 [29814.272213] iTCO_vendor_support [29814.275265] ? __hrtimer_run_queues+0x1d3/0x360 [29814.279525] i2c_smbus crct10dif_pclmul [29814.282754] ? hrtimer_interrupt+0x104/0x240 [29814.287275] joydev crc32_pclmul [29814.291105] ? __local_bh_enable+0x50/0x60 [29814.295369] ipmi_ssif ghash_clmulni_intel [29814.298596] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29814.302684] mei_me [29814.306775] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29814.311992] mei lpc_ich [29814.314090] [29814.319047] sha512_ssse3 [29814.321579] [29814.323674] acpi_ipmi mfd_core [29814.326294] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29814.328391] wmi ipmi_si [29814.331532] ? queued_spin_lock_slowpath+0x16/0x2e0 [29814.336831] ipmi_devintf ipmi_msghandler [29814.339364] do_raw_spin_lock+0xb5/0xc0 [29814.344234] ioatdma acpi_pad [29814.348239] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29814.352068] acpi_power_meter ixgbe [29814.355036] try_get_cap_refs+0x159/0x880 [ceph] [29814.359817] igb [29814.363305] __ceph_get_caps+0x4a1/0x760 [ceph] [29814.367910] mdio crc32c_intel [29814.369753] ? __file_remove_privs+0x89/0x110 [29814.374273] nvme ptp [29814.377322] ? wake_bit_function+0x70/0x70 [29814.381673] nvme_core i2c_algo_bit [29814.383947] ? ceph_write_iter+0x260/0x770 [ceph] [29814.388036] pps_core [29814.391521] ceph_write_iter+0x260/0x770 [ceph] [29814.396217] dca fuse [29814.398491] ? lock_is_held_type+0xaa/0x120 [29814.403012] [29814.403013] irq event stamp: 160000718 [29814.405285] vfs_write+0x324/0x4f0 [29814.409459] hardirqs last enabled at (160000717): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29814.410956] ksys_pwrite64+0x65/0xa0 [29814.414695] hardirqs last disabled at (160000718): [] sysvec_apic_timer_interrupt+0xe/0x90 [29814.418092] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29814.428335] softirqs last enabled at (160000716): [] __do_softirq+0x2be/0x371 [29814.431907] do_syscall_64+0x38/0x80 [29814.441716] softirqs last disabled at (160000711): [] irq_exit_rcu+0xb0/0xd0 [29814.446849] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29814.455619] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29814.459189] RIP: 0033:0x7f48f0eeb93b [29814.467788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29814.472833] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29814.482209] Workqueue: events delayed_work [ceph] [29814.485780] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29814.493252] [29814.493253] RIP: 0010:lock_release+0x14e/0x260 [29814.511987] ORIG_RAX: 0000000000000012 [29814.516685] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [29814.521904] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29814.523394] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [29814.527830] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29814.531661] [29814.531662] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [29814.550400] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29814.557523] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [29814.562741] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29814.569865] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [29814.571355] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29814.578481] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [29814.585609] [29814.592728] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [29814.630533] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29814.638611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29814.644348] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29814.651473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29814.658597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29814.665721] Call Trace: [29814.668171] [29814.670184] ? watchdog_timer_fn+0x1b4/0x220 [29814.674454] ? __hrtimer_run_queues+0x1d3/0x360 [29814.678984] ? hrtimer_interrupt+0x104/0x240 [29814.683254] ? __local_bh_enable+0x50/0x60 [29814.687347] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29814.692572] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29814.697527] [29814.699626] [29814.701722] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29814.707028] ? igrab+0x43/0x50 [29814.710082] ? lock_release+0x14e/0x260 [29814.713919] ? lock_release+0xc5/0x260 [29814.717663] _raw_spin_unlock+0x1b/0x40 [29814.721492] igrab+0x43/0x50 [29814.724372] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29814.729520] delayed_work+0x1d8/0x280 [ceph] [29814.733803] ? process_scheduled_works+0x22a/0x510 [29814.738590] process_scheduled_works+0x2a4/0x510 [29814.743206] worker_thread+0x188/0x340 [29814.746957] ? rescuer_thread+0x310/0x310 [29814.750969] kthread+0xf5/0x130 [29814.754115] ? kthread_complete_and_exit+0x20/0x20 [29814.758899] ret_from_fork+0x31/0x40 [29814.762479] ? kthread_complete_and_exit+0x20/0x20 [29814.767262] ret_from_fork_asm+0x11/0x20 [29814.771196] [29841.981270] watchdog: BUG: soft lockup - CPU#0 stuck for 16678s! [dbench:78683] [29841.988575] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29841.993270] watchdog: BUG: soft lockup - CPU#7 stuck for 25675s! [kworker/7:4:78808] [29842.044015] irq event stamp: 50062704 [29842.044016] hardirqs last enabled at (50062703): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29842.051748] Modules linked in: [29842.055403] hardirqs last disabled at (50062704): [] sysvec_apic_timer_interrupt+0xe/0x90 [29842.065562] ceph [29842.068611] softirqs last enabled at (50062666): [] __do_softirq+0x2be/0x371 [29842.078337] libceph [29842.080261] softirqs last disabled at (50062661): [] irq_exit_rcu+0xb0/0xd0 [29842.088945] dns_resolver [29842.091129] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29842.099639] fscache netfs [29842.102275] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29842.111218] veth [29842.113922] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29842.121393] nft_chain_nat [29842.123318] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29842.128795] xt_MASQUERADE nf_nat [29842.131500] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29842.150237] nf_conntrack nf_defrag_ipv6 [29842.153548] [29842.153548] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29842.158764] nf_defrag_ipv4 [29842.162683] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29842.164174] nft_compat nf_tables [29842.171296] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29842.174087] nfnetlink bridge [29842.181212] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29842.184524] stp llc [29842.191664] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29842.194627] xfs [29842.201753] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29842.203948] libcrc32c sunrpc [29842.211069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29842.212907] intel_rapl_msr intel_rapl_common [29842.220985] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29842.223949] sb_edac x86_pkg_temp_thermal [29842.229686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29842.234037] coretemp kvm_intel [29842.241161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29842.245165] kvm irqbypass [29842.252290] Call Trace: [29842.252291] [29842.255426] i2c_i801 iTCO_wdt [29842.262552] ? watchdog_timer_fn+0x1b4/0x220 [29842.265255] iTCO_vendor_support i2c_smbus [29842.267698] ? __hrtimer_run_queues+0x1d3/0x360 [29842.269708] crct10dif_pclmul [29842.272762] ? hrtimer_interrupt+0x104/0x240 [29842.277023] joydev crc32_pclmul [29842.281114] ? __local_bh_enable+0x50/0x60 [29842.285640] ipmi_ssif ghash_clmulni_intel [29842.288604] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29842.292867] mei_me mei [29842.296092] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29842.300180] lpc_ich sha512_ssse3 [29842.304273] [29842.309489] acpi_ipmi mfd_core [29842.311949] [29842.316909] wmi ipmi_si [29842.320219] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29842.322318] ipmi_devintf [29842.325457] ? queued_spin_lock_slowpath+0x16/0x2e0 [29842.327551] ipmi_msghandler ioatdma [29842.330085] do_raw_spin_lock+0xb5/0xc0 [29842.335386] acpi_pad acpi_power_meter [29842.338004] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29842.342875] ixgbe igb [29842.346448] try_get_cap_refs+0x159/0x880 [ceph] [29842.350277] mdio crc32c_intel [29842.354024] __ceph_get_caps+0x4a1/0x760 [ceph] [29842.358805] nvme ptp [29842.361166] ? __file_remove_privs+0x89/0x110 [29842.365771] nvme_core i2c_algo_bit [29842.368824] ? wake_bit_function+0x70/0x70 [29842.373348] pps_core dca [29842.375621] ? ceph_write_iter+0x260/0x770 [ceph] [29842.379968] fuse [29842.383453] ceph_write_iter+0x260/0x770 [ceph] [29842.387543] irq event stamp: 160221674 [29842.390164] ? lock_is_held_type+0xaa/0x120 [29842.394858] hardirqs last enabled at (160221673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29842.396784] vfs_write+0x324/0x4f0 [29842.401306] hardirqs last disabled at (160221674): [] sysvec_apic_timer_interrupt+0xe/0x90 [29842.405055] ksys_pwrite64+0x65/0xa0 [29842.409226] softirqs last enabled at (160221672): [] __do_softirq+0x2be/0x371 [29842.419472] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29842.422869] softirqs last disabled at (160221667): [] irq_exit_rcu+0xb0/0xd0 [29842.432681] do_syscall_64+0x38/0x80 [29842.436250] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29842.445021] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29842.450152] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29842.458750] RIP: 0033:0x7f48f0eeb93b [29842.462320] Workqueue: events delayed_work [ceph] [29842.471699] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29842.476742] [29842.476743] RIP: 0010:lock_acquire+0xf0/0x2b0 [29842.484214] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29842.487785] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29842.492480] ORIG_RAX: 0000000000000012 [29842.511218] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29842.512710] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29842.517061] [29842.517062] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29842.522277] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29842.541016] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29842.544847] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29842.550064] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29842.557187] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29842.558683] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29842.565812] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29842.572934] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29842.580065] [29842.587183] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29842.633065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29842.638804] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29842.645927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29842.653050] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29842.660175] Call Trace: [29842.662622] [29842.664632] ? watchdog_timer_fn+0x1b4/0x220 [29842.668895] ? __hrtimer_run_queues+0x1d3/0x360 [29842.673423] ? hrtimer_interrupt+0x104/0x240 [29842.677692] ? __local_bh_enable+0x50/0x60 [29842.681785] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29842.687010] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29842.691965] [29842.694062] [29842.696161] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29842.701469] ? lock_acquire+0xf0/0x2b0 [29842.705219] ? igrab+0x43/0x50 [29842.708269] ? lock_release+0xc5/0x260 [29842.712014] _raw_spin_lock+0x30/0x40 [29842.715678] ? igrab+0x1d/0x50 [29842.718728] igrab+0x1d/0x50 [29842.721608] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29842.726757] delayed_work+0x1d8/0x280 [ceph] [29842.731039] ? process_scheduled_works+0x22a/0x510 [29842.735829] process_scheduled_works+0x2a4/0x510 [29842.740445] worker_thread+0x188/0x340 [29842.744194] ? rescuer_thread+0x310/0x310 [29842.748196] kthread+0xf5/0x130 [29842.751335] ? kthread_complete_and_exit+0x20/0x20 [29842.756119] ret_from_fork+0x31/0x40 [29842.759689] ? kthread_complete_and_exit+0x20/0x20 [29842.764473] ret_from_fork_asm+0x11/0x20 [29842.768395] [29869.981775] watchdog: BUG: soft lockup - CPU#0 stuck for 16704s! [dbench:78683] [29869.989080] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29869.993775] watchdog: BUG: soft lockup - CPU#7 stuck for 25701s! [kworker/7:4:78808] [29870.044529] irq event stamp: 50120884 [29870.044530] hardirqs last enabled at (50120883): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29870.052262] Modules linked in: [29870.055918] hardirqs last disabled at (50120884): [] sysvec_apic_timer_interrupt+0xe/0x90 [29870.066075] ceph [29870.069126] softirqs last enabled at (50120846): [] __do_softirq+0x2be/0x371 [29870.078851] libceph [29870.080773] softirqs last disabled at (50120841): [] irq_exit_rcu+0xb0/0xd0 [29870.089458] dns_resolver [29870.091644] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29870.100170] fscache netfs [29870.102788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29870.111733] veth nft_chain_nat [29870.114452] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29870.121924] xt_MASQUERADE nf_nat [29870.125062] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29870.130540] nf_conntrack nf_defrag_ipv6 [29870.133849] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29870.152588] nf_defrag_ipv4 [29870.156505] [29870.161723] nft_compat nf_tables [29870.164513] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29870.166004] nfnetlink bridge [29870.169315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29870.176451] stp llc [29870.179412] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29870.186536] xfs libcrc32c [29870.188720] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29870.195843] sunrpc intel_rapl_msr [29870.198547] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29870.205671] intel_rapl_common [29870.209068] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29870.216194] sb_edac [29870.219244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29870.227322] x86_pkg_temp_thermal [29870.229504] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29870.235244] coretemp kvm_intel [29870.238555] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29870.245677] kvm irqbypass [29870.248816] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29870.255939] i2c_i801 iTCO_wdt [29870.258644] Call Trace: [29870.265767] iTCO_vendor_support [29870.268821] [29870.271262] i2c_smbus crct10dif_pclmul [29870.274488] ? watchdog_timer_fn+0x1b4/0x220 [29870.276496] joydev crc32_pclmul [29870.280328] ? __hrtimer_run_queues+0x1d3/0x360 [29870.284592] ipmi_ssif ghash_clmulni_intel [29870.287818] ? hrtimer_interrupt+0x104/0x240 [29870.292339] mei_me mei [29870.296430] ? __local_bh_enable+0x50/0x60 [29870.300714] lpc_ich [29870.303176] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29870.307265] sha512_ssse3 acpi_ipmi [29870.309456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29870.314674] mfd_core wmi [29870.318174] [29870.318175] [29870.323132] ipmi_si [29870.325750] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29870.327848] ipmi_devintf ipmi_msghandler [29870.329951] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29870.332130] ioatdma acpi_pad [29870.337435] do_raw_spin_lock+0xb5/0xc0 [29870.341454] acpi_power_meter [29870.346328] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29870.349289] ixgbe igb [29870.353122] try_get_cap_refs+0x159/0x880 [ceph] [29870.356084] mdio [29870.360873] __ceph_get_caps+0x4a1/0x760 [ceph] [29870.363226] crc32c_intel [29870.367842] ? __file_remove_privs+0x89/0x110 [29870.369761] nvme ptp [29870.374286] ? wake_bit_function+0x70/0x70 [29870.376903] nvme_core i2c_algo_bit [29870.381256] ? ceph_write_iter+0x260/0x770 [ceph] [29870.383525] pps_core [29870.387614] ceph_write_iter+0x260/0x770 [ceph] [29870.391098] dca fuse [29870.395801] ? lock_is_held_type+0xaa/0x120 [29870.398067] [29870.398068] irq event stamp: 160442414 [29870.402594] vfs_write+0x324/0x4f0 [29870.404861] hardirqs last enabled at (160442413): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29870.409043] ksys_pwrite64+0x65/0xa0 [29870.410530] hardirqs last disabled at (160442414): [] sysvec_apic_timer_interrupt+0xe/0x90 [29870.414276] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29870.417672] softirqs last enabled at (160442412): [] __do_softirq+0x2be/0x371 [29870.427916] do_syscall_64+0x38/0x80 [29870.431486] softirqs last disabled at (160442407): [] irq_exit_rcu+0xb0/0xd0 [29870.441299] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29870.446429] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29870.455199] RIP: 0033:0x7f48f0eeb93b [29870.458768] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29870.467367] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29870.472411] Workqueue: events delayed_work [ceph] [29870.481788] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29870.485360] [29870.485361] RIP: 0010:lock_acquire+0xf0/0x2b0 [29870.492831] ORIG_RAX: 0000000000000012 [29870.511567] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29870.516265] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29870.521481] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29870.522973] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29870.527325] [29870.527326] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29870.531155] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29870.549911] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29870.557034] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29870.562250] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29870.569376] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29870.570866] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29870.577996] [29870.585115] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29870.630044] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29870.638121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29870.643858] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29870.650982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29870.658106] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29870.665232] Call Trace: [29870.667676] [29870.669687] ? watchdog_timer_fn+0x1b4/0x220 [29870.673950] ? __hrtimer_run_queues+0x1d3/0x360 [29870.678478] ? hrtimer_interrupt+0x104/0x240 [29870.682747] ? __local_bh_enable+0x50/0x60 [29870.686839] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29870.692064] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29870.697022] [29870.699119] [29870.701216] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29870.706523] ? lock_acquire+0xf0/0x2b0 [29870.710275] ? igrab+0x43/0x50 [29870.713324] ? lock_release+0xc5/0x260 [29870.717068] _raw_spin_lock+0x30/0x40 [29870.720723] ? igrab+0x1d/0x50 [29870.723776] igrab+0x1d/0x50 [29870.726654] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29870.731805] delayed_work+0x1d8/0x280 [ceph] [29870.736085] ? process_scheduled_works+0x22a/0x510 [29870.740874] process_scheduled_works+0x2a4/0x510 [29870.745490] worker_thread+0x188/0x340 [29870.749238] ? rescuer_thread+0x310/0x310 [29870.753242] kthread+0xf5/0x130 [29870.756379] ? kthread_complete_and_exit+0x20/0x20 [29870.761166] ret_from_fork+0x31/0x40 [29870.764744] ? kthread_complete_and_exit+0x20/0x20 [29870.769528] ret_from_fork_asm+0x11/0x20 [29870.773451] [29897.982280] watchdog: BUG: soft lockup - CPU#0 stuck for 16730s! [dbench:78683] [29897.989585] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29897.994280] watchdog: BUG: soft lockup - CPU#7 stuck for 25727s! [kworker/7:4:78808] [29898.045026] irq event stamp: 50179036 [29898.045027] hardirqs last enabled at (50179035): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29898.052757] Modules linked in: [29898.056413] hardirqs last disabled at (50179036): [] sysvec_apic_timer_interrupt+0xe/0x90 [29898.066571] ceph [29898.069622] softirqs last enabled at (50178998): [] __do_softirq+0x2be/0x371 [29898.079345] libceph [29898.081272] softirqs last disabled at (50178993): [] irq_exit_rcu+0xb0/0xd0 [29898.089956] dns_resolver [29898.092140] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29898.100650] fscache [29898.103284] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29898.112227] netfs veth [29898.114412] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29898.121882] nft_chain_nat xt_MASQUERADE [29898.124326] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29898.129806] nf_nat [29898.133723] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29898.152460] nf_conntrack nf_defrag_ipv6 [29898.154558] [29898.154559] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29898.159774] nf_defrag_ipv4 [29898.163691] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29898.165182] nft_compat nf_tables [29898.172306] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29898.175097] nfnetlink [29898.182221] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29898.185532] bridge stp [29898.192673] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29898.195032] llc [29898.202155] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29898.204599] xfs libcrc32c [29898.211724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29898.213560] sunrpc intel_rapl_msr [29898.221639] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29898.224343] intel_rapl_common [29898.230080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29898.233477] sb_edac x86_pkg_temp_thermal [29898.240602] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29898.243651] coretemp kvm_intel [29898.250792] Call Trace: [29898.250794] [29898.254797] kvm irqbypass [29898.261924] ? watchdog_timer_fn+0x1b4/0x220 [29898.265060] i2c_i801 iTCO_wdt [29898.267505] ? __hrtimer_run_queues+0x1d3/0x360 [29898.269514] iTCO_vendor_support i2c_smbus [29898.272221] ? hrtimer_interrupt+0x104/0x240 [29898.276481] crct10dif_pclmul [29898.279533] ? __local_bh_enable+0x50/0x60 [29898.284058] joydev crc32_pclmul [29898.288149] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29898.292412] ipmi_ssif ghash_clmulni_intel [29898.295377] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29898.299467] mei_me mei [29898.302691] [29898.307908] lpc_ich sha512_ssse3 [29898.311998] [29898.316959] acpi_ipmi [29898.319400] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29898.321498] mfd_core wmi [29898.324813] ? queued_spin_lock_slowpath+0x18/0x2e0 [29898.326905] ipmi_si ipmi_devintf [29898.329265] do_raw_spin_lock+0xb5/0xc0 [29898.334567] ipmi_msghandler ioatdma [29898.337187] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29898.342055] acpi_pad acpi_power_meter [29898.345368] try_get_cap_refs+0x159/0x880 [ceph] [29898.349197] ixgbe igb [29898.352773] __ceph_get_caps+0x4a1/0x760 [ceph] [29898.357550] mdio crc32c_intel [29898.361300] ? __file_remove_privs+0x89/0x110 [29898.365907] nvme ptp [29898.368264] ? wake_bit_function+0x70/0x70 [29898.372789] nvme_core [29898.375842] ? ceph_write_iter+0x260/0x770 [ceph] [29898.380189] i2c_algo_bit pps_core [29898.382459] ceph_write_iter+0x260/0x770 [ceph] [29898.386551] dca fuse [29898.388911] ? lock_is_held_type+0xaa/0x120 [29898.393605] [29898.393606] irq event stamp: 160663362 [29898.397006] vfs_write+0x324/0x4f0 [29898.401528] hardirqs last enabled at (160663361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29898.403803] ksys_pwrite64+0x65/0xa0 [29898.407974] hardirqs last disabled at (160663362): [] sysvec_apic_timer_interrupt+0xe/0x90 [29898.409465] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29898.413211] softirqs last enabled at (160663360): [] __do_softirq+0x2be/0x371 [29898.416608] do_syscall_64+0x38/0x80 [29898.426852] softirqs last disabled at (160663355): [] irq_exit_rcu+0xb0/0xd0 [29898.430423] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29898.440232] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29898.445364] RIP: 0033:0x7f48f0eeb93b [29898.454135] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29898.457705] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29898.466304] Workqueue: events delayed_work [ceph] [29898.471347] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29898.480726] [29898.484294] ORIG_RAX: 0000000000000012 [29898.491767] RIP: 0010:lock_acquire+0xf0/0x2b0 [29898.510504] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29898.515200] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29898.520419] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29898.521909] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29898.525741] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29898.530090] [29898.530090] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29898.537215] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29898.555961] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29898.563085] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29898.568302] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29898.575432] [29898.576917] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29898.621846] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29898.628979] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29898.637056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29898.642794] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29898.649918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29898.657043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29898.664166] Call Trace: [29898.666612] [29898.668622] ? watchdog_timer_fn+0x1b4/0x220 [29898.672888] ? __hrtimer_run_queues+0x1d3/0x360 [29898.677412] ? hrtimer_interrupt+0x104/0x240 [29898.681686] ? __local_bh_enable+0x50/0x60 [29898.685786] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29898.691008] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29898.695966] [29898.698062] [29898.700161] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29898.705470] ? lock_acquire+0xf0/0x2b0 [29898.709219] ? igrab+0x43/0x50 [29898.712270] ? lock_release+0xc5/0x260 [29898.716013] _raw_spin_lock+0x30/0x40 [29898.719688] ? igrab+0x1d/0x50 [29898.722738] igrab+0x1d/0x50 [29898.725616] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29898.730765] delayed_work+0x1d8/0x280 [ceph] [29898.735047] ? process_scheduled_works+0x22a/0x510 [29898.739838] process_scheduled_works+0x2a4/0x510 [29898.744454] worker_thread+0x188/0x340 [29898.748203] ? rescuer_thread+0x310/0x310 [29898.752212] kthread+0xf5/0x130 [29898.755350] ? kthread_complete_and_exit+0x20/0x20 [29898.760135] ret_from_fork+0x31/0x40 [29898.763714] ? kthread_complete_and_exit+0x20/0x20 [29898.768500] ret_from_fork_asm+0x11/0x20 [29898.772421] [29925.982787] watchdog: BUG: soft lockup - CPU#0 stuck for 16756s! [dbench:78683] [29925.990090] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29925.994787] watchdog: BUG: soft lockup - CPU#7 stuck for 25753s! [kworker/7:4:78808] [29926.045541] irq event stamp: 50237206 [29926.045542] hardirqs last enabled at (50237205): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29926.053271] Modules linked in: [29926.056927] hardirqs last disabled at (50237206): [] sysvec_apic_timer_interrupt+0xe/0x90 [29926.067086] ceph [29926.070135] softirqs last enabled at (50237168): [] __do_softirq+0x2be/0x371 [29926.079859] libceph [29926.081784] softirqs last disabled at (50237163): [] irq_exit_rcu+0xb0/0xd0 [29926.090468] dns_resolver fscache [29926.092655] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29926.101181] netfs [29926.104492] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29926.113435] veth nft_chain_nat [29926.115462] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [29926.122935] xt_MASQUERADE [29926.126074] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [29926.131549] nf_nat nf_conntrack [29926.134253] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29926.152991] nf_defrag_ipv6 [29926.156214] [29926.161433] nf_defrag_ipv4 nft_compat [29926.164222] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29926.165713] nf_tables nfnetlink [29926.169457] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29926.176600] bridge stp [29926.179823] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29926.186948] llc xfs [29926.189392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29926.196515] libcrc32c sunrpc [29926.198699] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29926.205824] intel_rapl_msr intel_rapl_common [29926.208789] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29926.215911] sb_edac x86_pkg_temp_thermal [29926.220263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29926.228340] coretemp kvm_intel [29926.232344] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29926.238083] kvm irqbypass [29926.241218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29926.248342] i2c_i801 iTCO_wdt [29926.251047] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29926.258185] iTCO_vendor_support i2c_smbus [29926.261239] Call Trace: [29926.268364] crct10dif_pclmul joydev [29926.272455] [29926.274914] crc32_pclmul ipmi_ssif [29926.278488] ? watchdog_timer_fn+0x1b4/0x220 [29926.280497] ghash_clmulni_intel [29926.283982] ? __hrtimer_run_queues+0x1d3/0x360 [29926.288244] mei_me mei [29926.291478] ? hrtimer_interrupt+0x104/0x240 [29926.296002] lpc_ich [29926.298462] ? __local_bh_enable+0x50/0x60 [29926.302728] sha512_ssse3 acpi_ipmi [29926.304914] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29926.309001] mfd_core wmi [29926.312486] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29926.317704] ipmi_si ipmi_devintf [29926.320321] [29926.325278] ipmi_msghandler [29926.328588] [29926.330687] ioatdma acpi_pad [29926.333565] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29926.335661] acpi_power_meter ixgbe igb [29926.338630] ? queued_spin_lock_slowpath+0x18/0x2e0 [29926.343929] mdio crc32c_intel [29926.347763] do_raw_spin_lock+0xb5/0xc0 [29926.352631] nvme [29926.355684] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29926.359512] ptp nvme_core [29926.361439] try_get_cap_refs+0x159/0x880 [ceph] [29926.366220] i2c_algo_bit [29926.368932] __ceph_get_caps+0x4a1/0x760 [ceph] [29926.373535] pps_core dca [29926.376156] ? __file_remove_privs+0x89/0x110 [29926.380677] fuse [29926.383294] ? wake_bit_function+0x70/0x70 [29926.387645] irq event stamp: 160884044 [29926.389571] ? ceph_write_iter+0x260/0x770 [ceph] [29926.393661] hardirqs last enabled at (160884043): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29926.397404] ceph_write_iter+0x260/0x770 [ceph] [29926.402100] hardirqs last disabled at (160884044): [] sysvec_apic_timer_interrupt+0xe/0x90 [29926.412349] ? lock_is_held_type+0xaa/0x120 [29926.416870] softirqs last enabled at (160884042): [] __do_softirq+0x2be/0x371 [29926.426683] vfs_write+0x324/0x4f0 [29926.430859] softirqs last disabled at (160884037): [] irq_exit_rcu+0xb0/0xd0 [29926.439633] ksys_pwrite64+0x65/0xa0 [29926.443026] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29926.451625] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29926.455194] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29926.464572] do_syscall_64+0x38/0x80 [29926.469702] Workqueue: events delayed_work [ceph] [29926.477192] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29926.480763] [29926.480763] RIP: 0010:lock_acquire+0xf0/0x2b0 [29926.485469] RIP: 0033:0x7f48f0eeb93b [29926.490512] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29926.492003] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29926.496353] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29926.499924] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29926.518661] [29926.537398] ORIG_RAX: 0000000000000012 [29926.542617] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29926.547835] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29926.549325] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29926.553155] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29926.560279] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29926.567403] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29926.574528] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29926.581653] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29926.588776] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29926.595900] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29926.603025] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29926.610154] [29926.617273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29926.640406] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29926.647530] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29926.654652] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29926.661776] Call Trace: [29926.664221] [29926.666234] ? watchdog_timer_fn+0x1b4/0x220 [29926.670496] ? __hrtimer_run_queues+0x1d3/0x360 [29926.675022] ? hrtimer_interrupt+0x104/0x240 [29926.679285] ? __local_bh_enable+0x50/0x60 [29926.683377] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29926.688601] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29926.693560] [29926.695657] [29926.697754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29926.703060] ? lock_acquire+0xf0/0x2b0 [29926.706804] ? igrab+0x43/0x50 [29926.709861] ? lock_release+0xc5/0x260 [29926.713606] _raw_spin_lock+0x30/0x40 [29926.717263] ? igrab+0x1d/0x50 [29926.720313] igrab+0x1d/0x50 [29926.723195] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29926.728341] delayed_work+0x1d8/0x280 [ceph] [29926.732623] ? process_scheduled_works+0x22a/0x510 [29926.737412] process_scheduled_works+0x2a4/0x510 [29926.742029] worker_thread+0x188/0x340 [29926.745779] ? rescuer_thread+0x310/0x310 [29926.749790] kthread+0xf5/0x130 [29926.752936] ? kthread_complete_and_exit+0x20/0x20 [29926.757720] ret_from_fork+0x31/0x40 [29926.761298] ? kthread_complete_and_exit+0x20/0x20 [29926.766082] ret_from_fork_asm+0x11/0x20 [29926.770006] [29953.983295] watchdog: BUG: soft lockup - CPU#0 stuck for 16782s! [dbench:78683] [29953.990594] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [29953.995295] watchdog: BUG: soft lockup - CPU#7 stuck for 25779s! [kworker/7:4:78808] [29954.046035] irq event stamp: 50295284 [29954.046036] hardirqs last enabled at (50295283): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29954.053767] Modules linked in: [29954.057422] hardirqs last disabled at (50295284): [] sysvec_apic_timer_interrupt+0xe/0x90 [29954.067582] ceph [29954.070633] softirqs last enabled at (50295258): [] __do_softirq+0x2be/0x371 [29954.080357] libceph [29954.082280] softirqs last disabled at (50295253): [] irq_exit_rcu+0xb0/0xd0 [29954.090970] dns_resolver [29954.093157] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29954.101667] fscache netfs [29954.104302] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29954.113247] veth nft_chain_nat [29954.115951] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [29954.123421] xt_MASQUERADE nf_nat [29954.126559] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [29954.132035] nf_conntrack nf_defrag_ipv6 [29954.135346] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29954.154086] nf_defrag_ipv4 [29954.158003] [29954.158004] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29954.163221] nft_compat nf_tables [29954.166011] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29954.167501] nfnetlink bridge [29954.174624] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29954.177936] stp llc [29954.185061] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29954.188023] xfs libcrc32c [29954.195149] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29954.197333] sunrpc [29954.204456] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29954.207161] intel_rapl_msr intel_rapl_common [29954.214284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29954.216382] sb_edac x86_pkg_temp_thermal [29954.224459] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29954.228810] coretemp kvm_intel [29954.234549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29954.238553] kvm irqbypass [29954.245694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29954.248830] i2c_i801 iTCO_wdt [29954.255971] Call Trace: [29954.255973] [29954.258691] iTCO_vendor_support [29954.265818] ? watchdog_timer_fn+0x1b4/0x220 [29954.268868] i2c_smbus crct10dif_pclmul [29954.271312] ? __hrtimer_run_queues+0x1d3/0x360 [29954.273323] joydev crc32_pclmul [29954.276551] ? hrtimer_interrupt+0x104/0x240 [29954.280810] ipmi_ssif ghash_clmulni_intel [29954.284642] ? __local_bh_enable+0x50/0x60 [29954.289167] mei_me mei [29954.292392] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29954.296653] lpc_ich sha512_ssse3 [29954.300746] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29954.304836] acpi_ipmi mfd_core [29954.307279] [29954.312497] wmi [29954.315809] [29954.320765] ipmi_si ipmi_devintf [29954.323904] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29954.325999] ipmi_msghandler [29954.327842] ? queued_spin_lock_slowpath+0x16/0x2e0 [29954.329935] ioatdma acpi_pad [29954.333246] do_raw_spin_lock+0xb5/0xc0 [29954.338550] acpi_power_meter ixgbe [29954.341429] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29954.346298] igb mdio [29954.349264] try_get_cap_refs+0x159/0x880 [ceph] [29954.353092] crc32c_intel [29954.356581] __ceph_get_caps+0x4a1/0x760 [ceph] [29954.361362] nvme ptp [29954.363637] ? __file_remove_privs+0x89/0x110 [29954.368243] nvme_core [29954.370860] ? wake_bit_function+0x70/0x70 [29954.375383] i2c_algo_bit pps_core [29954.377656] ? ceph_write_iter+0x260/0x770 [ceph] [29954.382004] dca fuse [29954.384363] ceph_write_iter+0x260/0x770 [ceph] [29954.388454] [29954.391853] ? lock_is_held_type+0xaa/0x120 [29954.396549] irq event stamp: 161104816 [29954.398822] vfs_write+0x324/0x4f0 [29954.403344] hardirqs last enabled at (161104815): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29954.404837] ksys_pwrite64+0x65/0xa0 [29954.409010] hardirqs last disabled at (161104816): [] sysvec_apic_timer_interrupt+0xe/0x90 [29954.412754] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29954.416152] softirqs last enabled at (161104814): [] __do_softirq+0x2be/0x371 [29954.426397] do_syscall_64+0x38/0x80 [29954.429981] softirqs last disabled at (161104809): [] irq_exit_rcu+0xb0/0xd0 [29954.439796] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29954.444927] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29954.453696] RIP: 0033:0x7f48f0eeb93b [29954.457269] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29954.465866] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29954.470909] Workqueue: events delayed_work [ceph] [29954.480287] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [29954.483858] [29954.491328] ORIG_RAX: 0000000000000012 [29954.510066] RIP: 0010:lock_acquire+0xf0/0x2b0 [29954.514764] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29954.519987] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29954.521481] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29954.525312] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29954.529662] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29954.536786] [29954.536786] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29954.555522] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29954.562648] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29954.567866] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29954.574990] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29954.576486] [29954.583603] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29954.621419] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29954.628542] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29954.636619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29954.642357] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29954.649481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29954.656605] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29954.663730] Call Trace: [29954.666173] [29954.668184] ? watchdog_timer_fn+0x1b4/0x220 [29954.672449] ? __hrtimer_run_queues+0x1d3/0x360 [29954.676985] ? hrtimer_interrupt+0x104/0x240 [29954.681255] ? __local_bh_enable+0x50/0x60 [29954.685348] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29954.690571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29954.695527] [29954.697624] [29954.699724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29954.705031] ? lock_acquire+0xf0/0x2b0 [29954.708780] ? igrab+0x43/0x50 [29954.711830] ? lock_release+0xc5/0x260 [29954.715577] _raw_spin_lock+0x30/0x40 [29954.719242] ? igrab+0x1d/0x50 [29954.722291] igrab+0x1d/0x50 [29954.725170] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29954.730320] delayed_work+0x1d8/0x280 [ceph] [29954.734600] ? process_scheduled_works+0x22a/0x510 [29954.739389] process_scheduled_works+0x2a4/0x510 [29954.744006] worker_thread+0x188/0x340 [29954.747757] ? rescuer_thread+0x310/0x310 [29954.751768] kthread+0xf5/0x130 [29954.754904] ? kthread_complete_and_exit+0x20/0x20 [29954.759705] ret_from_fork+0x31/0x40 [29954.763287] ? kthread_complete_and_exit+0x20/0x20 [29954.768070] ret_from_fork_asm+0x11/0x20 [29954.772002] [29981.548163] rcu: INFO: rcu_preempt self-detected stall on CPU [29981.553928] rcu: 7-....: (17521190 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/17519878 fqs=8824001 [29981.564528] rcu: (t=18044617 jiffies g=1309609 q=533664 ncpus=8) [29981.570612] Sending NMI from CPU 7 to CPUs 0: [29981.574963] NMI backtrace for cpu 0 [29981.574965] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29981.574967] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29981.574968] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [29981.574973] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [29981.574974] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [29981.574975] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [29981.574976] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [29981.574977] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [29981.574978] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [29981.574979] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [29981.574979] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [29981.574981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29981.574981] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [29981.574982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29981.574983] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29981.574984] Call Trace: [29981.574985] [29981.574986] ? nmi_cpu_backtrace+0xad/0x120 [29981.574990] ? nmi_cpu_backtrace_handler+0x11/0x20 [29981.574992] ? nmi_handle+0xd4/0x260 [29981.574996] ? default_do_nmi+0x49/0xf0 [29981.574998] ? exc_nmi+0x11e/0x150 [29981.574999] ? end_repeat_nmi+0x16/0x67 [29981.575003] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29981.575006] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29981.575008] ? queued_spin_lock_slowpath+0x2d/0x2e0 [29981.575011] [29981.575011] [29981.575012] do_raw_spin_lock+0xb5/0xc0 [29981.575016] __cap_delay_requeue+0xa7/0x1d0 [ceph] [29981.575036] try_get_cap_refs+0x159/0x880 [ceph] [29981.575050] __ceph_get_caps+0x4a1/0x760 [ceph] [29981.575062] ? __file_remove_privs+0x89/0x110 [29981.575064] ? wake_bit_function+0x70/0x70 [29981.575067] ? ceph_write_iter+0x260/0x770 [ceph] [29981.575077] ceph_write_iter+0x260/0x770 [ceph] [29981.575088] ? lock_is_held_type+0xaa/0x120 [29981.575091] vfs_write+0x324/0x4f0 [29981.575095] ksys_pwrite64+0x65/0xa0 [29981.575097] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [29981.575100] do_syscall_64+0x38/0x80 [29981.575103] entry_SYSCALL_64_after_hwframe+0x63/0xcd [29981.575105] RIP: 0033:0x7f48f0eeb93b [29981.575106] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [29981.575107] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [29981.575109] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [29981.575110] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [29981.575110] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [29981.575111] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [29981.575112] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [29981.575114] [29981.575962] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [29981.879574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [29981.887047] Workqueue: events delayed_work [ceph] [29981.891769] RIP: 0010:lock_acquire+0xf0/0x2b0 [29981.896120] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [29981.914858] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [29981.920073] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [29981.927198] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [29981.934324] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [29981.941448] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [29981.948571] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [29981.955696] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [29981.963772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [29981.969511] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [29981.976635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [29981.983759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [29981.990882] Call Trace: [29981.993328] [29981.995340] ? rcu_dump_cpu_stacks+0xd1/0x180 [29981.999699] ? rcu_sched_clock_irq+0x633/0x1390 [29982.004231] ? lock_release+0xc5/0x260 [29982.007982] ? lock_is_held_type+0xaa/0x120 [29982.012161] ? lock_is_held_type+0xaa/0x120 [29982.016350] ? tick_sched_do_timer+0xa0/0xa0 [29982.020620] ? rcu_read_lock_sched_held+0x56/0x90 [29982.025327] ? tick_sched_do_timer+0xa0/0xa0 [29982.029598] ? update_process_times+0x5f/0x90 [29982.033958] ? tick_sched_handle.isra.16+0x1e/0x50 [29982.038750] ? tick_sched_timer+0x6f/0x80 [29982.042756] ? __hrtimer_run_queues+0x1d3/0x360 [29982.047295] ? hrtimer_interrupt+0x104/0x240 [29982.051559] ? __local_bh_enable+0x50/0x60 [29982.055665] ? __sysvec_apic_timer_interrupt+0x61/0x160 [29982.060885] ? sysvec_apic_timer_interrupt+0x6d/0x90 [29982.065843] [29982.067938] [29982.070038] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [29982.075361] ? lock_acquire+0xf0/0x2b0 [29982.079115] ? igrab+0x43/0x50 [29982.082171] ? lock_release+0xc5/0x260 [29982.085936] _raw_spin_lock+0x30/0x40 [29982.089597] ? igrab+0x1d/0x50 [29982.092649] igrab+0x1d/0x50 [29982.095529] ceph_check_delayed_caps+0xb7/0x270 [ceph] [29982.100690] delayed_work+0x1d8/0x280 [ceph] [29982.104978] ? process_scheduled_works+0x22a/0x510 [29982.109766] process_scheduled_works+0x2a4/0x510 [29982.114387] worker_thread+0x188/0x340 [29982.118141] ? rescuer_thread+0x310/0x310 [29982.122152] kthread+0xf5/0x130 [29982.125297] ? kthread_complete_and_exit+0x20/0x20 [29982.130092] ret_from_fork+0x31/0x40 [29982.133668] ? kthread_complete_and_exit+0x20/0x20 [29982.138455] ret_from_fork_asm+0x11/0x20 [29982.142395] [30005.984239] watchdog: BUG: soft lockup - CPU#0 stuck for 16831s! [dbench:78683] [30005.991543] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30005.996239] watchdog: BUG: soft lockup - CPU#7 stuck for 25828s! [kworker/7:4:78808] [30006.046994] irq event stamp: 50404336 [30006.054724] Modules linked in: ceph libceph [30006.058381] hardirqs last enabled at (50404335): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30006.062558] dns_resolver fscache netfs [30006.072716] hardirqs last disabled at (50404336): [] sysvec_apic_timer_interrupt+0xe/0x90 [30006.076547] veth nft_chain_nat xt_MASQUERADE [30006.086272] softirqs last enabled at (50404282): [] __do_softirq+0x2be/0x371 [30006.086273] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [30006.099316] softirqs last disabled at (50404277): [] irq_exit_rcu+0xb0/0xd0 [30006.106092] nf_tables nfnetlink [30006.114602] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30006.117828] bridge stp [30006.126772] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30006.129216] llc xfs libcrc32c [30006.136687] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30006.139738] sunrpc intel_rapl_msr [30006.145214] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30006.148611] intel_rapl_common sb_edac [30006.167367] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30006.171111] x86_pkg_temp_thermal coretemp [30006.176327] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30006.180420] kvm_intel kvm [30006.187543] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30006.187544] irqbypass [30006.190247] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30006.197371] i2c_i801 iTCO_wdt [30006.199728] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30006.206853] iTCO_vendor_support i2c_smbus [30006.209917] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30006.217046] crct10dif_pclmul joydev [30006.221137] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30006.228260] crc32_pclmul ipmi_ssif [30006.231830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30006.239918] ghash_clmulni_intel [30006.243400] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30006.249139] mei_me mei [30006.252362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30006.259486] lpc_ich sha512_ssse3 [30006.261931] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30006.269054] acpi_ipmi mfd_core [30006.272366] Call Trace: [30006.272367] [30006.279490] wmi [30006.282638] ? watchdog_timer_fn+0x1b4/0x220 [30006.285080] ipmi_si ipmi_devintf [30006.287092] ? __hrtimer_run_queues+0x1d3/0x360 [30006.288928] ipmi_msghandler ioatdma [30006.293194] ? hrtimer_interrupt+0x104/0x240 [30006.296501] acpi_pad [30006.301028] ? __local_bh_enable+0x50/0x60 [30006.304598] acpi_power_meter [30006.308864] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30006.311133] ixgbe igb [30006.315223] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30006.318187] mdio crc32c_intel [30006.323403] [30006.325761] nvme [30006.330720] [30006.330721] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30006.333771] ptp nvme_core [30006.335877] ? queued_spin_lock_slowpath+0x16/0x2e0 [30006.337790] i2c_algo_bit pps_core [30006.339891] do_raw_spin_lock+0xb5/0xc0 [30006.345192] dca fuse [30006.347897] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30006.352784] [30006.352785] irq event stamp: 161515308 [30006.356184] try_get_cap_refs+0x159/0x880 [ceph] [30006.360012] hardirqs last enabled at (161515307): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30006.362288] __ceph_get_caps+0x4a1/0x760 [ceph] [30006.367067] hardirqs last disabled at (161515308): [] sysvec_apic_timer_interrupt+0xe/0x90 [30006.368562] ? __file_remove_privs+0x89/0x110 [30006.372301] softirqs last enabled at (161515306): [] __do_softirq+0x2be/0x371 [30006.376920] ? wake_bit_function+0x70/0x70 [30006.387165] softirqs last disabled at (161515301): [] irq_exit_rcu+0xb0/0xd0 [30006.391692] ? ceph_write_iter+0x260/0x770 [ceph] [30006.401500] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30006.405853] ceph_write_iter+0x260/0x770 [ceph] [30006.414638] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30006.418734] ? lock_is_held_type+0xaa/0x120 [30006.427330] Workqueue: events delayed_work [ceph] [30006.432028] vfs_write+0x324/0x4f0 [30006.441402] [30006.441403] RIP: 0010:lock_acquire+0xf0/0x2b0 [30006.445944] ksys_pwrite64+0x65/0xa0 [30006.453408] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30006.457583] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30006.462281] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30006.465681] do_syscall_64+0x38/0x80 [30006.467168] [30006.467169] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30006.471521] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30006.475091] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30006.493829] RIP: 0033:0x7f48f0eeb93b [30006.498960] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30006.504177] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30006.507747] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30006.509238] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30006.516362] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30006.521407] ORIG_RAX: 0000000000000012 [30006.528531] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30006.532102] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30006.539225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30006.557963] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30006.565088] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30006.570306] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30006.577430] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30006.581260] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30006.589337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30006.596462] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30006.602199] Call Trace: [30006.609328] [30006.616449] [30006.658708] ? watchdog_timer_fn+0x1b4/0x220 [30006.662973] ? __hrtimer_run_queues+0x1d3/0x360 [30006.667498] ? hrtimer_interrupt+0x104/0x240 [30006.671758] ? __local_bh_enable+0x50/0x60 [30006.675853] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30006.681075] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30006.686033] [30006.688131] [30006.690229] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30006.695537] ? lock_acquire+0xf0/0x2b0 [30006.699286] ? igrab+0x43/0x50 [30006.702337] ? lock_release+0xc5/0x260 [30006.706082] _raw_spin_lock+0x30/0x40 [30006.709746] ? igrab+0x1d/0x50 [30006.712798] igrab+0x1d/0x50 [30006.715675] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30006.720823] delayed_work+0x1d8/0x280 [ceph] [30006.725104] ? process_scheduled_works+0x22a/0x510 [30006.729886] process_scheduled_works+0x2a4/0x510 [30006.734503] worker_thread+0x188/0x340 [30006.738253] ? rescuer_thread+0x310/0x310 [30006.742262] kthread+0xf5/0x130 [30006.745401] ? kthread_complete_and_exit+0x20/0x20 [30006.750187] ret_from_fork+0x31/0x40 [30006.753755] ? kthread_complete_and_exit+0x20/0x20 [30006.758539] ret_from_fork_asm+0x11/0x20 [30006.762463] [30033.984746] watchdog: BUG: soft lockup - CPU#0 stuck for 16857s! [dbench:78683] [30033.992048] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30033.996746] watchdog: BUG: soft lockup - CPU#7 stuck for 25854s! [kworker/7:4:78808] [30034.047498] irq event stamp: 50462402 [30034.055230] Modules linked in: ceph libceph dns_resolver fscache netfs [30034.058886] hardirqs last enabled at (50462401): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30034.065422] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [30034.075578] hardirqs last disabled at (50462402): [] sysvec_apic_timer_interrupt+0xe/0x90 [30034.082962] nf_defrag_ipv4 nft_compat nf_tables [30034.092687] softirqs last enabled at (50462366): [] __do_softirq+0x2be/0x371 [30034.097298] nfnetlink bridge stp llc xfs libcrc32c [30034.105982] softirqs last disabled at (50462361): [] irq_exit_rcu+0xb0/0xd0 [30034.110852] sunrpc intel_rapl_msr intel_rapl_common [30034.119365] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30034.119365] sb_edac [30034.119367] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30034.124322] x86_pkg_temp_thermal coretemp [30034.133264] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30034.135450] kvm_intel [30034.142919] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30034.147011] kvm irqbypass [30034.152487] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30034.154847] i2c_i801 iTCO_wdt [30034.173582] [30034.173583] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30034.176288] iTCO_vendor_support i2c_smbus [30034.181505] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30034.184554] crct10dif_pclmul joydev [30034.186045] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30034.193169] crc32_pclmul ipmi_ssif [30034.197262] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30034.204384] ghash_clmulni_intel mei_me [30034.207955] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30034.215079] mei lpc_ich [30034.218563] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30034.225687] sha512_ssse3 [30034.229518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30034.236643] acpi_ipmi [30034.239174] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30034.247250] mfd_core wmi [30034.249868] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30034.255607] ipmi_si [30034.257962] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30034.265088] ipmi_devintf [30034.267705] Call Trace: [30034.267707] [30034.274830] ipmi_msghandler ioatdma [30034.277016] ? watchdog_timer_fn+0x1b4/0x220 [30034.284146] acpi_pad acpi_power_meter [30034.286766] ? __hrtimer_run_queues+0x1d3/0x360 [30034.289207] ixgbe igb [30034.291220] ? hrtimer_interrupt+0x104/0x240 [30034.294790] mdio [30034.299052] ? __local_bh_enable+0x50/0x60 [30034.302797] crc32c_intel nvme [30034.307323] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30034.309677] ptp nvme_core [30034.313944] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30034.315868] i2c_algo_bit pps_core [30034.319957] [30034.323009] dca [30034.328226] [30034.330930] fuse [30034.335888] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30034.339283] irq event stamp: 161736178 [30034.341386] ? queued_spin_lock_slowpath+0x16/0x2e0 [30034.343219] hardirqs last enabled at (161736177): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30034.345318] do_raw_spin_lock+0xb5/0xc0 [30034.347241] hardirqs last disabled at (161736178): [] sysvec_apic_timer_interrupt+0xe/0x90 [30034.352546] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30034.356288] softirqs last enabled at (161736176): [] __do_softirq+0x2be/0x371 [30034.361164] try_get_cap_refs+0x159/0x880 [ceph] [30034.371402] softirqs last disabled at (161736171): [] irq_exit_rcu+0xb0/0xd0 [30034.375239] __ceph_get_caps+0x4a1/0x760 [ceph] [30034.385045] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30034.389834] ? __file_remove_privs+0x89/0x110 [30034.398599] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30034.403211] ? wake_bit_function+0x70/0x70 [30034.411810] Workqueue: events delayed_work [ceph] [30034.416335] ? ceph_write_iter+0x260/0x770 [ceph] [30034.425710] [30034.430060] ceph_write_iter+0x260/0x770 [ceph] [30034.437532] RIP: 0010:lock_acquire+0xf0/0x2b0 [30034.441625] ? lock_is_held_type+0xaa/0x120 [30034.446320] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30034.451020] vfs_write+0x324/0x4f0 [30034.452509] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30034.457035] ksys_pwrite64+0x65/0xa0 [30034.461382] [30034.461383] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30034.465562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30034.484297] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30034.487697] do_syscall_64+0x38/0x80 [30034.492913] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30034.496484] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30034.497973] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30034.505099] RIP: 0033:0x7f48f0eeb93b [30034.510228] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30034.517353] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30034.520923] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30034.528046] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30034.533092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30034.540216] ORIG_RAX: 0000000000000012 [30034.543786] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30034.550910] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30034.569649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30034.577726] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30034.582944] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30034.588680] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30034.592513] Call Trace: [30034.592514] [30034.599636] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30034.606760] ? watchdog_timer_fn+0x1b4/0x220 [30034.613883] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30034.621008] ? __hrtimer_run_queues+0x1d3/0x360 [30034.628154] [30034.635277] ? hrtimer_interrupt+0x104/0x240 [30034.669212] ? __local_bh_enable+0x50/0x60 [30034.673307] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30034.678532] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30034.683488] [30034.685586] [30034.687682] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30034.692990] ? lock_acquire+0xf0/0x2b0 [30034.696741] ? igrab+0x43/0x50 [30034.699790] ? lock_release+0xc5/0x260 [30034.703536] _raw_spin_lock+0x30/0x40 [30034.707201] ? igrab+0x1d/0x50 [30034.710251] igrab+0x1d/0x50 [30034.713131] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30034.718298] delayed_work+0x1d8/0x280 [ceph] [30034.722576] ? process_scheduled_works+0x22a/0x510 [30034.727367] process_scheduled_works+0x2a4/0x510 [30034.731983] worker_thread+0x188/0x340 [30034.735733] ? rescuer_thread+0x310/0x310 [30034.739744] kthread+0xf5/0x130 [30034.742883] ? kthread_complete_and_exit+0x20/0x20 [30034.747666] ret_from_fork+0x31/0x40 [30034.751245] ? kthread_complete_and_exit+0x20/0x20 [30034.756028] ret_from_fork_asm+0x11/0x20 [30034.759952] [30061.985253] watchdog: BUG: soft lockup - CPU#0 stuck for 16883s! [dbench:78683] [30061.992553] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30061.997253] watchdog: BUG: soft lockup - CPU#7 stuck for 25880s! [kworker/7:4:78808] [30062.047995] irq event stamp: 50520480 [30062.055726] Modules linked in: ceph libceph [30062.059383] hardirqs last enabled at (50520479): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30062.063560] dns_resolver fscache netfs [30062.073718] hardirqs last disabled at (50520480): [] sysvec_apic_timer_interrupt+0xe/0x90 [30062.077548] veth nft_chain_nat xt_MASQUERADE nf_nat [30062.087273] softirqs last enabled at (50520470): [] __do_softirq+0x2be/0x371 [30062.092229] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [30062.100926] softirqs last disabled at (50520465): [] irq_exit_rcu+0xb0/0xd0 [30062.107103] nf_tables nfnetlink bridge [30062.115613] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30062.119444] stp llc [30062.128389] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30062.130572] xfs libcrc32c sunrpc intel_rapl_msr [30062.138041] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30062.142653] intel_rapl_common sb_edac [30062.148132] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30062.148132] x86_pkg_temp_thermal coretemp [30062.151876] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30062.170612] kvm_intel kvm [30062.174703] [30062.174704] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30062.179929] irqbypass i2c_i801 [30062.182650] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30062.184140] iTCO_wdt iTCO_vendor_support [30062.191267] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30062.194404] i2c_smbus crct10dif_pclmul [30062.201528] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30062.205531] joydev [30062.212654] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30062.216487] crc32_pclmul ipmi_ssif [30062.223611] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30062.225707] ghash_clmulni_intel mei_me [30062.232833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30062.236316] mei [30062.244393] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30062.248223] lpc_ich sha512_ssse3 [30062.253962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30062.255799] acpi_ipmi mfd_core [30062.262930] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30062.266242] wmi ipmi_si [30062.273365] Call Trace: [30062.273367] [30062.276503] ipmi_devintf ipmi_msghandler [30062.283647] ? watchdog_timer_fn+0x1b4/0x220 [30062.286175] ioatdma acpi_pad [30062.288622] ? __hrtimer_run_queues+0x1d3/0x360 [30062.290647] acpi_power_meter [30062.294657] ? hrtimer_interrupt+0x104/0x240 [30062.298915] ixgbe igb [30062.301881] ? __local_bh_enable+0x50/0x60 [30062.306403] mdio crc32c_intel [30062.309370] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30062.313650] nvme ptp [30062.316007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30062.320097] nvme_core i2c_algo_bit [30062.323150] [30062.328366] pps_core dca [30062.330650] [30062.335612] fuse [30062.339096] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30062.341192] irq event stamp: 161956980 [30062.343814] ? queued_spin_lock_slowpath+0x18/0x2e0 [30062.345908] hardirqs last enabled at (161956979): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30062.347852] do_raw_spin_lock+0xb5/0xc0 [30062.353153] hardirqs last disabled at (161956980): [] sysvec_apic_timer_interrupt+0xe/0x90 [30062.356900] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30062.361769] softirqs last enabled at (161956978): [] __do_softirq+0x2be/0x371 [30062.372015] try_get_cap_refs+0x159/0x880 [ceph] [30062.375842] softirqs last disabled at (161956973): [] irq_exit_rcu+0xb0/0xd0 [30062.385659] __ceph_get_caps+0x4a1/0x760 [ceph] [30062.390439] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30062.399214] ? __file_remove_privs+0x89/0x110 [30062.403820] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30062.412417] ? wake_bit_function+0x70/0x70 [30062.416957] Workqueue: events delayed_work [ceph] [30062.426337] ? ceph_write_iter+0x260/0x770 [ceph] [30062.430686] [30062.430687] RIP: 0010:lock_acquire+0xf0/0x2b0 [30062.438159] ceph_write_iter+0x260/0x770 [ceph] [30062.442249] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30062.446954] ? lock_is_held_type+0xaa/0x120 [30062.451654] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30062.453145] vfs_write+0x324/0x4f0 [30062.457493] [30062.457493] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30062.462020] ksys_pwrite64+0x65/0xa0 [30062.480755] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30062.484946] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30062.490158] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30062.493557] do_syscall_64+0x38/0x80 [30062.495047] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30062.502173] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30062.505741] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30062.512866] RIP: 0033:0x7f48f0eeb93b [30062.517996] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30062.525121] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30062.528692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30062.535816] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30062.540858] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30062.547984] ORIG_RAX: 0000000000000012 [30062.551555] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30062.559632] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30062.578386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30062.584124] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30062.589340] Call Trace: [30062.589342] [30062.596467] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30062.600298] ? watchdog_timer_fn+0x1b4/0x220 [30062.607421] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30062.614545] ? __hrtimer_run_queues+0x1d3/0x360 [30062.621669] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30062.628797] ? hrtimer_interrupt+0x104/0x240 [30062.631244] [30062.633247] ? __local_bh_enable+0x50/0x60 [30062.673967] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30062.679190] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30062.684148] [30062.686246] [30062.688343] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30062.693665] ? lock_acquire+0xf0/0x2b0 [30062.697411] ? igrab+0x43/0x50 [30062.700461] ? lock_release+0xc5/0x260 [30062.704206] _raw_spin_lock+0x30/0x40 [30062.707871] ? igrab+0x1d/0x50 [30062.710937] igrab+0x1d/0x50 [30062.713815] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30062.718966] delayed_work+0x1d8/0x280 [ceph] [30062.723247] ? process_scheduled_works+0x22a/0x510 [30062.728037] process_scheduled_works+0x2a4/0x510 [30062.732667] worker_thread+0x188/0x340 [30062.736412] ? rescuer_thread+0x310/0x310 [30062.740413] kthread+0xf5/0x130 [30062.743550] ? kthread_complete_and_exit+0x20/0x20 [30062.748335] ret_from_fork+0x31/0x40 [30062.751915] ? kthread_complete_and_exit+0x20/0x20 [30062.756700] ret_from_fork_asm+0x11/0x20 [30062.760622] [30089.985760] watchdog: BUG: soft lockup - CPU#0 stuck for 16909s! [dbench:78683] [30089.993068] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30089.997759] watchdog: BUG: soft lockup - CPU#7 stuck for 25906s! [kworker/7:4:78808] [30090.048516] irq event stamp: 50578528 [30090.048517] hardirqs last enabled at (50578527): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30090.056247] Modules linked in: [30090.059905] hardirqs last disabled at (50578528): [] sysvec_apic_timer_interrupt+0xe/0x90 [30090.070062] ceph [30090.073114] softirqs last enabled at (50578518): [] __do_softirq+0x2be/0x371 [30090.082836] libceph [30090.084762] softirqs last disabled at (50578513): [] irq_exit_rcu+0xb0/0xd0 [30090.093445] dns_resolver [30090.095631] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30090.104155] fscache netfs [30090.106775] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30090.115719] veth nft_chain_nat [30090.118438] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30090.125911] xt_MASQUERADE nf_nat [30090.129048] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30090.134526] nf_conntrack [30090.137837] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30090.156574] nf_defrag_ipv6 [30090.159191] [30090.159192] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30090.164410] nf_defrag_ipv4 nft_compat [30090.167217] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30090.168708] nf_tables nfnetlink [30090.175833] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30090.179576] bridge stp [30090.186699] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30090.189924] llc xfs [30090.197050] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30090.199492] libcrc32c sunrpc [30090.206617] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30090.208800] intel_rapl_msr [30090.215925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30090.218889] intel_rapl_common [30090.226966] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30090.229756] sb_edac x86_pkg_temp_thermal [30090.235495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30090.238544] coretemp kvm_intel [30090.245668] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30090.249674] kvm [30090.256799] Call Trace: [30090.256801] [30090.259935] irqbypass i2c_i801 [30090.267062] ? watchdog_timer_fn+0x1b4/0x220 [30090.268896] iTCO_wdt iTCO_vendor_support [30090.271343] ? __hrtimer_run_queues+0x1d3/0x360 [30090.273351] i2c_smbus crct10dif_pclmul [30090.276493] ? hrtimer_interrupt+0x104/0x240 [30090.280754] joydev crc32_pclmul [30090.284757] ? __local_bh_enable+0x50/0x60 [30090.289282] ipmi_ssif ghash_clmulni_intel [30090.293115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30090.297375] mei_me mei [30090.300601] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30090.304690] lpc_ich [30090.308782] [30090.314000] sha512_ssse3 [30090.316444] [30090.321399] acpi_ipmi mfd_core [30090.323585] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30090.325683] wmi ipmi_si [30090.328303] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30090.330395] ipmi_devintf [30090.333535] do_raw_spin_lock+0xb5/0xc0 [30090.338837] ipmi_msghandler [30090.341371] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30090.346239] ioatdma acpi_pad [30090.348859] try_get_cap_refs+0x159/0x880 [ceph] [30090.352689] acpi_power_meter [30090.355569] __ceph_get_caps+0x4a1/0x760 [ceph] [30090.360349] ixgbe igb [30090.363319] ? __file_remove_privs+0x89/0x110 [30090.367923] mdio [30090.370887] ? wake_bit_function+0x70/0x70 [30090.375413] crc32c_intel nvme [30090.377772] ? ceph_write_iter+0x260/0x770 [ceph] [30090.382120] ptp nvme_core [30090.384044] ceph_write_iter+0x260/0x770 [ceph] [30090.388135] i2c_algo_bit [30090.391207] ? lock_is_held_type+0xaa/0x120 [30090.395901] pps_core [30090.398608] vfs_write+0x324/0x4f0 [30090.403128] dca [30090.405750] ksys_pwrite64+0x65/0xa0 [30090.409924] fuse [30090.412196] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30090.415590] irq event stamp: 162177974 [30090.417430] do_syscall_64+0x38/0x80 [30090.421016] hardirqs last enabled at (162177973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30090.422943] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30090.428072] hardirqs last disabled at (162177974): [] sysvec_apic_timer_interrupt+0xe/0x90 [30090.431818] RIP: 0033:0x7f48f0eeb93b [30090.435386] softirqs last enabled at (162177972): [] __do_softirq+0x2be/0x371 [30090.445632] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30090.450676] softirqs last disabled at (162177967): [] irq_exit_rcu+0xb0/0xd0 [30090.460486] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30090.464056] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30090.472826] ORIG_RAX: 0000000000000012 [30090.491565] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30090.500163] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30090.505380] Workqueue: events delayed_work [ceph] [30090.514758] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30090.518588] [30090.526059] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30090.533182] RIP: 0010:igrab+0x46/0x50 [30090.537882] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30090.545004] Code: 00 f6 83 d8 00 00 00 30 75 15 f0 ff 83 08 02 00 00 48 89 ef e8 8b 60 7f 00 48 89 d8 5b 5d c3 48 89 ef e8 7d 60 7f 00 31 c0 5b <5d> c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 41 56 49 [30090.546495] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30090.553619] RSP: 0018:ffffc90002117da8 EFLAGS: 00000246 [30090.557282] [30090.564400] [30090.564401] RAX: 0000000000000000 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [30090.606297] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [30090.613421] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [30090.620546] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [30090.627669] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30090.634792] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30090.642870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30090.648608] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30090.655734] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30090.662856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30090.669981] Call Trace: [30090.672428] [30090.674454] ? watchdog_timer_fn+0x1b4/0x220 [30090.678718] ? __hrtimer_run_queues+0x1d3/0x360 [30090.683243] ? hrtimer_interrupt+0x104/0x240 [30090.687504] ? __local_bh_enable+0x50/0x60 [30090.691599] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30090.696822] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30090.701779] [30090.703876] [30090.705974] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30090.711283] ? igrab+0x46/0x50 [30090.714340] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30090.719490] delayed_work+0x1d8/0x280 [ceph] [30090.723770] ? process_scheduled_works+0x22a/0x510 [30090.728559] process_scheduled_works+0x2a4/0x510 [30090.733175] worker_thread+0x188/0x340 [30090.736926] ? rescuer_thread+0x310/0x310 [30090.740936] kthread+0xf5/0x130 [30090.744075] ? kthread_complete_and_exit+0x20/0x20 [30090.748857] ret_from_fork+0x31/0x40 [30090.752452] ? kthread_complete_and_exit+0x20/0x20 [30090.757239] ret_from_fork_asm+0x11/0x20 [30090.761173] [30117.986265] watchdog: BUG: soft lockup - CPU#0 stuck for 16935s! [dbench:78683] [30117.993564] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30117.998265] watchdog: BUG: soft lockup - CPU#7 stuck for 25932s! [kworker/7:4:78808] [30118.049005] irq event stamp: 50636726 [30118.049006] hardirqs last enabled at (50636725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30118.056736] Modules linked in: [30118.060392] hardirqs last disabled at (50636726): [] sysvec_apic_timer_interrupt+0xe/0x90 [30118.070552] ceph [30118.073600] softirqs last enabled at (50636628): [] __do_softirq+0x2be/0x371 [30118.083325] libceph [30118.085248] softirqs last disabled at (50636623): [] irq_exit_rcu+0xb0/0xd0 [30118.093941] dns_resolver [30118.096125] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30118.104637] fscache [30118.107271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30118.116217] netfs [30118.118401] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30118.125871] veth nft_chain_nat [30118.127882] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30118.133358] xt_MASQUERADE nf_nat [30118.136497] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30118.155234] nf_conntrack [30118.158545] [30118.158546] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30118.163761] nf_defrag_ipv6 [30118.166379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30118.167871] nf_defrag_ipv4 nft_compat [30118.174994] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30118.177785] nf_tables nfnetlink [30118.184909] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30118.188662] bridge stp [30118.195785] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30118.199011] llc [30118.206134] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30118.208577] xfs libcrc32c [30118.215703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30118.217540] sunrpc intel_rapl_msr [30118.225617] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30118.228320] intel_rapl_common [30118.234058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30118.237455] sb_edac x86_pkg_temp_thermal [30118.244581] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30118.247632] coretemp kvm_intel [30118.254756] Call Trace: [30118.254758] [30118.258760] kvm irqbypass [30118.265884] ? watchdog_timer_fn+0x1b4/0x220 [30118.269021] i2c_i801 iTCO_wdt [30118.271466] ? __hrtimer_run_queues+0x1d3/0x360 [30118.273476] iTCO_vendor_support [30118.276183] ? hrtimer_interrupt+0x104/0x240 [30118.280444] i2c_smbus crct10dif_pclmul [30118.283495] ? __local_bh_enable+0x50/0x60 [30118.288017] joydev crc32_pclmul [30118.291245] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30118.295505] ipmi_ssif ghash_clmulni_intel [30118.299338] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30118.303427] mei_me mei [30118.306664] [30118.311877] lpc_ich [30118.315968] [30118.315969] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30118.320925] sha512_ssse3 acpi_ipmi [30118.323392] ? queued_spin_lock_slowpath+0x16/0x2e0 [30118.325484] mfd_core wmi [30118.327672] do_raw_spin_lock+0xb5/0xc0 [30118.329766] ipmi_si ipmi_devintf [30118.335072] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30118.338554] ipmi_msghandler ioatdma [30118.343427] try_get_cap_refs+0x159/0x880 [ceph] [30118.346042] acpi_pad acpi_power_meter [30118.349879] __ceph_get_caps+0x4a1/0x760 [ceph] [30118.353185] ixgbe [30118.357972] ? __file_remove_privs+0x89/0x110 [30118.361539] igb mdio [30118.366150] ? wake_bit_function+0x70/0x70 [30118.369895] crc32c_intel nvme [30118.374422] ? ceph_write_iter+0x260/0x770 [ceph] [30118.376427] ptp nvme_core [30118.380780] ceph_write_iter+0x260/0x770 [ceph] [30118.383051] i2c_algo_bit pps_core [30118.387145] ? lock_is_held_type+0xaa/0x120 [30118.390192] dca fuse [30118.394892] vfs_write+0x324/0x4f0 [30118.397594] [30118.397595] irq event stamp: 162398796 [30118.402120] ksys_pwrite64+0x65/0xa0 [30118.405515] hardirqs last enabled at (162398795): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30118.409693] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30118.411964] hardirqs last disabled at (162398796): [] sysvec_apic_timer_interrupt+0xe/0x90 [30118.415360] do_syscall_64+0x38/0x80 [30118.416849] softirqs last enabled at (162398794): [] __do_softirq+0x2be/0x371 [30118.420596] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30118.424165] softirqs last disabled at (162398789): [] irq_exit_rcu+0xb0/0xd0 [30118.434409] RIP: 0033:0x7f48f0eeb93b [30118.439541] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30118.449352] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30118.452921] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30118.461693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30118.466737] Workqueue: events delayed_work [ceph] [30118.475336] ORIG_RAX: 0000000000000012 [30118.478904] [30118.478905] RIP: 0010:lock_acquire+0xf0/0x2b0 [30118.488282] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30118.507021] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30118.514490] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30118.519708] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30118.524407] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30118.528237] [30118.528238] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30118.529727] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30118.534077] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30118.541202] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30118.559948] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30118.567078] [30118.572290] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30118.625833] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30118.632971] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30118.641054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30118.646791] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30118.653916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30118.661040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30118.668164] Call Trace: [30118.670608] [30118.672618] ? watchdog_timer_fn+0x1b4/0x220 [30118.676884] ? __hrtimer_run_queues+0x1d3/0x360 [30118.681419] ? hrtimer_interrupt+0x104/0x240 [30118.685687] ? __local_bh_enable+0x50/0x60 [30118.689780] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30118.695005] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30118.699961] [30118.702058] [30118.704156] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30118.709464] ? lock_acquire+0xf0/0x2b0 [30118.713215] ? igrab+0x43/0x50 [30118.716264] ? lock_release+0xc5/0x260 [30118.720009] _raw_spin_lock+0x30/0x40 [30118.723672] ? igrab+0x1d/0x50 [30118.726726] igrab+0x1d/0x50 [30118.729603] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30118.734752] delayed_work+0x1d8/0x280 [ceph] [30118.739034] ? process_scheduled_works+0x22a/0x510 [30118.743825] process_scheduled_works+0x2a4/0x510 [30118.748441] worker_thread+0x188/0x340 [30118.752190] ? rescuer_thread+0x310/0x310 [30118.756200] kthread+0xf5/0x130 [30118.759338] ? kthread_complete_and_exit+0x20/0x20 [30118.764122] ret_from_fork+0x31/0x40 [30118.767693] ? kthread_complete_and_exit+0x20/0x20 [30118.772478] ret_from_fork_asm+0x11/0x20 [30118.776408] [30145.986772] watchdog: BUG: soft lockup - CPU#0 stuck for 16961s! [dbench:78683] [30145.994077] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30145.998772] watchdog: BUG: soft lockup - CPU#7 stuck for 25958s! [kworker/7:4:78808] [30146.049518] irq event stamp: 50694902 [30146.049519] hardirqs last enabled at (50694901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30146.057249] Modules linked in: ceph [30146.060905] hardirqs last disabled at (50694902): [] sysvec_apic_timer_interrupt+0xe/0x90 [30146.071064] libceph [30146.074548] softirqs last enabled at (50694804): [] __do_softirq+0x2be/0x371 [30146.084271] dns_resolver [30146.086457] softirqs last disabled at (50694799): [] irq_exit_rcu+0xb0/0xd0 [30146.095139] fscache netfs [30146.097776] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30146.106287] veth nft_chain_nat [30146.108989] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30146.117933] xt_MASQUERADE nf_nat [30146.121071] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30146.128542] nf_conntrack nf_defrag_ipv6 [30146.131854] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30146.137331] nf_defrag_ipv4 [30146.141248] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30146.159986] nft_compat nf_tables [30146.162776] [30146.162777] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30146.167993] nfnetlink bridge [30146.171304] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30146.172795] stp llc [30146.179920] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30146.182882] xfs libcrc32c [30146.190008] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30146.192191] sunrpc intel_rapl_msr [30146.199316] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30146.202019] intel_rapl_common sb_edac [30146.209145] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30146.212558] x86_pkg_temp_thermal coretemp [30146.219683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30146.223426] kvm_intel kvm [30146.231504] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30146.235594] irqbypass i2c_i801 [30146.241333] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30146.244035] iTCO_wdt iTCO_vendor_support [30146.251170] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30146.254306] i2c_smbus crct10dif_pclmul [30146.261431] Call Trace: [30146.261432] [30146.265453] joydev crc32_pclmul [30146.272578] ? watchdog_timer_fn+0x1b4/0x220 [30146.276407] ipmi_ssif [30146.278852] ? __hrtimer_run_queues+0x1d3/0x360 [30146.280862] ghash_clmulni_intel mei_me [30146.284088] ? hrtimer_interrupt+0x104/0x240 [30146.288351] mei lpc_ich [30146.290706] ? __local_bh_enable+0x50/0x60 [30146.295230] sha512_ssse3 acpi_ipmi [30146.299064] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30146.303326] mfd_core [30146.305857] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30146.309948] wmi ipmi_si [30146.313433] [30146.318648] ipmi_devintf ipmi_msghandler [30146.320921] [30146.320922] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30146.325878] ioatdma acpi_pad [30146.328412] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30146.330505] acpi_power_meter [30146.334512] do_raw_spin_lock+0xb5/0xc0 [30146.336606] ixgbe igb [30146.341912] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30146.344874] mdio crc32c_intel [30146.349748] try_get_cap_refs+0x159/0x880 [ceph] [30146.352709] nvme ptp [30146.356546] __ceph_get_caps+0x4a1/0x760 [ceph] [30146.358897] nvme_core i2c_algo_bit [30146.363685] ? __file_remove_privs+0x89/0x110 [30146.366733] pps_core dca [30146.371343] ? wake_bit_function+0x70/0x70 [30146.373615] fuse [30146.378140] ? ceph_write_iter+0x260/0x770 [ceph] [30146.381622] irq event stamp: 162619666 [30146.385973] ceph_write_iter+0x260/0x770 [ceph] [30146.388589] hardirqs last enabled at (162619665): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30146.392684] ? lock_is_held_type+0xaa/0x120 [30146.394606] hardirqs last disabled at (162619666): [] sysvec_apic_timer_interrupt+0xe/0x90 [30146.399305] vfs_write+0x324/0x4f0 [30146.403046] softirqs last enabled at (162619664): [] __do_softirq+0x2be/0x371 [30146.407574] ksys_pwrite64+0x65/0xa0 [30146.417815] softirqs last disabled at (162619659): [] irq_exit_rcu+0xb0/0xd0 [30146.421993] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30146.431803] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30146.435203] do_syscall_64+0x38/0x80 [30146.443970] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30146.447542] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30146.456141] Workqueue: events delayed_work [ceph] [30146.461270] RIP: 0033:0x7f48f0eeb93b [30146.470648] [30146.470648] RIP: 0010:lock_acquire+0xf0/0x2b0 [30146.474218] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30146.481690] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30146.486734] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30146.491430] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30146.495001] ORIG_RAX: 0000000000000012 [30146.496492] [30146.496493] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30146.500844] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30146.519582] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30146.538318] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30146.543535] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30146.548752] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30146.552583] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30146.554073] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30146.561199] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30146.568322] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30146.575454] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30146.582585] [30146.589705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30146.641324] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30146.648449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30146.655572] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30146.662695] Call Trace: [30146.665140] [30146.667152] ? watchdog_timer_fn+0x1b4/0x220 [30146.671434] ? __hrtimer_run_queues+0x1d3/0x360 [30146.675978] ? hrtimer_interrupt+0x104/0x240 [30146.680247] ? __local_bh_enable+0x50/0x60 [30146.684340] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30146.689564] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30146.694522] [30146.696619] [30146.698715] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30146.704023] ? lock_acquire+0xf0/0x2b0 [30146.707775] ? igrab+0x43/0x50 [30146.710825] ? lock_release+0xc5/0x260 [30146.714569] _raw_spin_lock+0x30/0x40 [30146.718225] ? igrab+0x1d/0x50 [30146.721276] igrab+0x1d/0x50 [30146.724155] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30146.729320] delayed_work+0x1d8/0x280 [ceph] [30146.733602] ? process_scheduled_works+0x22a/0x510 [30146.738392] process_scheduled_works+0x2a4/0x510 [30146.743008] worker_thread+0x188/0x340 [30146.746757] ? rescuer_thread+0x310/0x310 [30146.750760] kthread+0xf5/0x130 [30146.753898] ? kthread_complete_and_exit+0x20/0x20 [30146.758681] ret_from_fork+0x31/0x40 [30146.762252] ? kthread_complete_and_exit+0x20/0x20 [30146.767037] ret_from_fork_asm+0x11/0x20 [30146.770959] [30173.987278] watchdog: BUG: soft lockup - CPU#0 stuck for 16987s! [dbench:78683] [30173.994581] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30173.999278] watchdog: BUG: soft lockup - CPU#7 stuck for 25984s! [kworker/7:4:78808] [30174.050023] irq event stamp: 50753054 [30174.057755] Modules linked in: ceph libceph dns_resolver fscache netfs [30174.061411] hardirqs last enabled at (50753053): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30174.067951] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [30174.078111] hardirqs last disabled at (50753054): [] sysvec_apic_timer_interrupt+0xe/0x90 [30174.084196] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [30174.093921] softirqs last enabled at (50752958): [] __do_softirq+0x2be/0x371 [30174.100699] bridge stp llc xfs libcrc32c [30174.109382] softirqs last disabled at (50752953): [] irq_exit_rcu+0xb0/0xd0 [30174.113385] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [30174.121898] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30174.129368] coretemp [30174.138313] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30174.138314] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30174.140583] kvm_intel kvm [30174.148054] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30174.153531] irqbypass i2c_i801 [30174.156234] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30174.174972] iTCO_wdt [30174.178109] [30174.178110] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30174.183328] iTCO_vendor_support [30174.185599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30174.187088] i2c_smbus crct10dif_pclmul [30174.194214] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30174.197437] joydev crc32_pclmul [30174.204560] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30174.208393] ipmi_ssif ghash_clmulni_intel [30174.215516] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30174.218741] mei_me mei [30174.225864] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30174.229956] lpc_ich [30174.237078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30174.239523] sha512_ssse3 [30174.247599] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30174.249783] acpi_ipmi mfd_core [30174.255523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30174.258140] wmi ipmi_si [30174.265265] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30174.268400] ipmi_devintf ipmi_msghandler [30174.275524] Call Trace: [30174.275527] [30174.278056] ioatdma acpi_pad [30174.285182] ? watchdog_timer_fn+0x1b4/0x220 [30174.289183] acpi_power_meter ixgbe [30174.291630] ? __hrtimer_run_queues+0x1d3/0x360 [30174.293639] igb mdio [30174.296606] ? hrtimer_interrupt+0x104/0x240 [30174.300866] crc32c_intel [30174.304352] ? __local_bh_enable+0x50/0x60 [30174.308876] nvme ptp [30174.311148] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30174.315409] nvme_core i2c_algo_bit [30174.318029] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30174.322118] pps_core [30174.324389] [30174.329605] dca fuse [30174.333091] [30174.338047] [30174.338047] irq event stamp: 162840524 [30174.340320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30174.342415] hardirqs last enabled at (162840523): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30174.344690] ? queued_spin_lock_slowpath+0x18/0x2e0 [30174.346784] hardirqs last disabled at (162840524): [] sysvec_apic_timer_interrupt+0xe/0x90 [30174.348276] do_raw_spin_lock+0xb5/0xc0 [30174.352018] softirqs last enabled at (162840522): [] __do_softirq+0x2be/0x371 [30174.357324] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30174.367566] softirqs last disabled at (162840517): [] irq_exit_rcu+0xb0/0xd0 [30174.372441] try_get_cap_refs+0x159/0x880 [ceph] [30174.382249] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30174.386084] __ceph_get_caps+0x4a1/0x760 [ceph] [30174.394850] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30174.399639] ? __file_remove_privs+0x89/0x110 [30174.408231] Workqueue: events delayed_work [ceph] [30174.412843] ? wake_bit_function+0x70/0x70 [30174.422219] [30174.426747] ? ceph_write_iter+0x260/0x770 [ceph] [30174.434215] RIP: 0010:lock_acquire+0xf0/0x2b0 [30174.438567] ceph_write_iter+0x260/0x770 [ceph] [30174.443262] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30174.447359] ? lock_is_held_type+0xaa/0x120 [30174.448844] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30174.453545] vfs_write+0x324/0x4f0 [30174.457893] [30174.457894] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30174.462422] ksys_pwrite64+0x65/0xa0 [30174.481153] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30174.485333] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30174.490548] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30174.493948] do_syscall_64+0x38/0x80 [30174.495438] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30174.502563] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30174.506132] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30174.513258] RIP: 0033:0x7f48f0eeb93b [30174.518386] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30174.525511] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30174.529081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30174.536205] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30174.541251] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30174.548373] ORIG_RAX: 0000000000000012 [30174.551946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30174.560039] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30174.578778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30174.584515] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30174.589733] Call Trace: [30174.589734] [30174.596855] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30174.600688] ? watchdog_timer_fn+0x1b4/0x220 [30174.607810] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30174.614936] ? __hrtimer_run_queues+0x1d3/0x360 [30174.622061] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30174.629188] ? hrtimer_interrupt+0x104/0x240 [30174.631633] [30174.633637] ? __local_bh_enable+0x50/0x60 [30174.674350] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30174.679573] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30174.684529] [30174.686627] [30174.688725] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30174.694033] ? lock_acquire+0xf0/0x2b0 [30174.697783] ? igrab+0x43/0x50 [30174.700834] ? lock_release+0xc5/0x260 [30174.704579] _raw_spin_lock+0x30/0x40 [30174.708242] ? igrab+0x1d/0x50 [30174.711293] igrab+0x1d/0x50 [30174.714172] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30174.719321] delayed_work+0x1d8/0x280 [ceph] [30174.723602] ? process_scheduled_works+0x22a/0x510 [30174.728393] process_scheduled_works+0x2a4/0x510 [30174.733009] worker_thread+0x188/0x340 [30174.736757] ? rescuer_thread+0x310/0x310 [30174.740762] kthread+0xf5/0x130 [30174.743908] ? kthread_complete_and_exit+0x20/0x20 [30174.748691] ret_from_fork+0x31/0x40 [30174.752271] ? kthread_complete_and_exit+0x20/0x20 [30174.757054] ret_from_fork_asm+0x11/0x20 [30174.760978] [30177.149702] rcu: INFO: rcu_preempt self-detected stall on CPU [30177.155460] rcu: 7-....: (17710815 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/17709501 fqs=8919885 [30177.166051] rcu: (t=18240215 jiffies g=1309609 q=537473 ncpus=8) [30177.172136] Sending NMI from CPU 7 to CPUs 0: [30177.176489] NMI backtrace for cpu 0 [30177.176490] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30177.176492] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30177.176492] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30177.176495] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30177.176497] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30177.176498] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30177.176499] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30177.176500] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30177.176500] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30177.176501] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30177.176502] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30177.176503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30177.176504] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30177.176505] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30177.176505] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30177.176506] Call Trace: [30177.176507] [30177.176508] ? nmi_cpu_backtrace+0xad/0x120 [30177.176512] ? nmi_cpu_backtrace_handler+0x11/0x20 [30177.176513] ? nmi_handle+0xd4/0x260 [30177.176517] ? default_do_nmi+0x49/0xf0 [30177.176519] ? exc_nmi+0x11e/0x150 [30177.176520] ? end_repeat_nmi+0x16/0x67 [30177.176524] ? queued_spin_lock_slowpath+0x16/0x2e0 [30177.176527] ? queued_spin_lock_slowpath+0x16/0x2e0 [30177.176529] ? queued_spin_lock_slowpath+0x16/0x2e0 [30177.176531] [30177.176532] [30177.176533] do_raw_spin_lock+0xb5/0xc0 [30177.176535] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30177.176549] try_get_cap_refs+0x159/0x880 [ceph] [30177.176562] __ceph_get_caps+0x4a1/0x760 [ceph] [30177.176574] ? __file_remove_privs+0x89/0x110 [30177.176576] ? wake_bit_function+0x70/0x70 [30177.176579] ? ceph_write_iter+0x260/0x770 [ceph] [30177.176589] ceph_write_iter+0x260/0x770 [ceph] [30177.176600] ? lock_is_held_type+0xaa/0x120 [30177.176602] vfs_write+0x324/0x4f0 [30177.176606] ksys_pwrite64+0x65/0xa0 [30177.176608] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30177.176611] do_syscall_64+0x38/0x80 [30177.176613] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30177.176615] RIP: 0033:0x7f48f0eeb93b [30177.176616] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30177.176617] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [30177.176618] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30177.176619] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30177.176619] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30177.176620] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30177.176621] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30177.176623] [30177.177487] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30177.481116] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30177.488586] Workqueue: events delayed_work [ceph] [30177.493302] RIP: 0010:lock_acquire+0xf0/0x2b0 [30177.497652] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30177.516390] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30177.521608] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30177.528736] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30177.535863] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30177.542987] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30177.550130] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30177.557255] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30177.565332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30177.571070] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30177.578193] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30177.585318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30177.592442] Call Trace: [30177.594886] [30177.596897] ? rcu_dump_cpu_stacks+0xd1/0x180 [30177.601258] ? rcu_sched_clock_irq+0x633/0x1390 [30177.605789] ? lock_release+0xc5/0x260 [30177.609532] ? lock_is_held_type+0xaa/0x120 [30177.613712] ? lock_is_held_type+0xaa/0x120 [30177.617916] ? tick_sched_do_timer+0xa0/0xa0 [30177.622187] ? rcu_read_lock_sched_held+0x56/0x90 [30177.626896] ? tick_sched_do_timer+0xa0/0xa0 [30177.631165] ? update_process_times+0x5f/0x90 [30177.635516] ? tick_sched_handle.isra.16+0x1e/0x50 [30177.640310] ? tick_sched_timer+0x6f/0x80 [30177.644324] ? __hrtimer_run_queues+0x1d3/0x360 [30177.648862] ? hrtimer_interrupt+0x104/0x240 [30177.653126] ? __local_bh_enable+0x50/0x60 [30177.657224] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30177.662445] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30177.667410] [30177.669506] [30177.671605] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30177.676927] ? lock_acquire+0xf0/0x2b0 [30177.680682] ? igrab+0x43/0x50 [30177.683741] ? lock_release+0xc5/0x260 [30177.687495] _raw_spin_lock+0x30/0x40 [30177.691158] ? igrab+0x1d/0x50 [30177.694209] igrab+0x1d/0x50 [30177.697087] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30177.702246] delayed_work+0x1d8/0x280 [ceph] [30177.706537] ? process_scheduled_works+0x22a/0x510 [30177.711324] process_scheduled_works+0x2a4/0x510 [30177.715945] worker_thread+0x188/0x340 [30177.719700] ? rescuer_thread+0x310/0x310 [30177.723710] kthread+0xf5/0x130 [30177.726865] ? kthread_complete_and_exit+0x20/0x20 [30177.731658] ret_from_fork+0x31/0x40 [30177.735235] ? kthread_complete_and_exit+0x20/0x20 [30177.740028] ret_from_fork_asm+0x11/0x20 [30177.743959] [30201.987784] watchdog: BUG: soft lockup - CPU#0 stuck for 17013s! [dbench:78683] [30201.995087] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30201.999784] watchdog: BUG: soft lockup - CPU#7 stuck for 26010s! [kworker/7:4:78808] [30202.050528] irq event stamp: 50811100 [30202.058258] Modules linked in: ceph [30202.061917] hardirqs last enabled at (50811099): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30202.061919] libceph dns_resolver fscache netfs [30202.075558] hardirqs last disabled at (50811100): [] sysvec_apic_timer_interrupt+0xe/0x90 [30202.080081] veth nft_chain_nat [30202.089805] softirqs last enabled at (50811004): [] __do_softirq+0x2be/0x371 [30202.089806] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [30202.101628] softirqs last disabled at (50810999): [] irq_exit_rcu+0xb0/0xd0 [30202.109618] nf_tables nfnetlink bridge [30202.118130] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30202.121961] stp llc [30202.130905] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30202.133088] xfs libcrc32c sunrpc intel_rapl_msr [30202.140559] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30202.145179] intel_rapl_common [30202.150656] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30202.150656] sb_edac [30202.153706] x86_pkg_temp_thermal coretemp [30202.172445] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30202.174645] kvm_intel kvm [30202.178736] [30202.178737] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30202.183953] irqbypass i2c_i801 [30202.186658] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30202.188149] iTCO_wdt iTCO_vendor_support [30202.195272] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30202.198408] i2c_smbus crct10dif_pclmul [30202.205534] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30202.209537] joydev [30202.216662] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30202.220493] crc32_pclmul [30202.227618] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30202.229715] ipmi_ssif ghash_clmulni_intel [30202.236839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30202.239471] mei_me mei [30202.247551] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30202.251641] lpc_ich sha512_ssse3 [30202.257378] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30202.259823] acpi_ipmi [30202.266948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30202.270257] mfd_core [30202.277382] Call Trace: [30202.277384] [30202.279740] wmi ipmi_si [30202.286864] ? watchdog_timer_fn+0x1b4/0x220 [30202.289135] ipmi_devintf [30202.291579] ? __hrtimer_run_queues+0x1d3/0x360 [30202.293590] ipmi_msghandler [30202.296124] ? hrtimer_interrupt+0x104/0x240 [30202.300385] ioatdma acpi_pad [30202.303002] ? __local_bh_enable+0x50/0x60 [30202.307524] acpi_power_meter [30202.310405] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30202.314667] ixgbe igb [30202.317632] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30202.321720] mdio crc32c_intel [30202.324685] [30202.329904] nvme [30202.332259] [30202.332260] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30202.337218] ptp [30202.340274] ? queued_spin_lock_slowpath+0x16/0x2e0 [30202.342366] nvme_core i2c_algo_bit [30202.344292] do_raw_spin_lock+0xb5/0xc0 [30202.346386] pps_core [30202.351693] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30202.353529] dca fuse [30202.358402] try_get_cap_refs+0x159/0x880 [ceph] [30202.361884] [30202.361885] irq event stamp: 163056642 [30202.365719] __ceph_get_caps+0x4a1/0x760 [ceph] [30202.367985] hardirqs last enabled at (163056641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30202.372774] ? __file_remove_privs+0x89/0x110 [30202.375040] hardirqs last disabled at (163056642): [] sysvec_apic_timer_interrupt+0xe/0x90 [30202.379650] ? wake_bit_function+0x70/0x70 [30202.381140] softirqs last enabled at (163056640): [] __do_softirq+0x2be/0x371 [30202.384888] ? ceph_write_iter+0x260/0x770 [ceph] [30202.389409] softirqs last disabled at (163056635): [] irq_exit_rcu+0xb0/0xd0 [30202.399654] ceph_write_iter+0x260/0x770 [ceph] [30202.404004] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30202.413819] ? lock_is_held_type+0xaa/0x120 [30202.417907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30202.426679] vfs_write+0x324/0x4f0 [30202.431375] Workqueue: events delayed_work [ceph] [30202.439975] ksys_pwrite64+0x65/0xa0 [30202.444496] [30202.444497] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [30202.453874] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30202.458051] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [30202.465523] do_syscall_64+0x38/0x80 [30202.468920] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [30202.473616] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30202.477187] [30202.477188] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [30202.478678] RIP: 0033:0x7f48f0eeb93b [30202.484674] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [30202.489806] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30202.508544] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [30202.512114] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30202.517332] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [30202.522375] ORIG_RAX: 0000000000000012 [30202.523866] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30202.530991] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30202.534561] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30202.541687] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30202.560423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30202.567548] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30202.572764] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30202.579889] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30202.583719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30202.590843] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30202.597969] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30202.606050] [30202.613186] Call Trace: [30202.613187] [30202.668327] ? watchdog_timer_fn+0x1b4/0x220 [30202.672591] ? __hrtimer_run_queues+0x1d3/0x360 [30202.677118] ? hrtimer_interrupt+0x104/0x240 [30202.681387] ? __local_bh_enable+0x50/0x60 [30202.685480] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30202.690703] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30202.695662] [30202.697758] [30202.699855] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30202.705165] ? __list_del_entry_valid_or_report+0x10/0xd0 [30202.710576] ceph_check_delayed_caps+0x8f/0x270 [ceph] [30202.715728] delayed_work+0x1d8/0x280 [ceph] [30202.720008] ? process_scheduled_works+0x22a/0x510 [30202.724800] process_scheduled_works+0x2a4/0x510 [30202.729414] worker_thread+0x188/0x340 [30202.733165] ? rescuer_thread+0x310/0x310 [30202.737193] kthread+0xf5/0x130 [30202.740330] ? kthread_complete_and_exit+0x20/0x20 [30202.745115] ret_from_fork+0x31/0x40 [30202.748685] ? kthread_complete_and_exit+0x20/0x20 [30202.753471] ret_from_fork_asm+0x11/0x20 [30202.757401] [30229.988291] watchdog: BUG: soft lockup - CPU#0 stuck for 17039s! [dbench:78683] [30229.995592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30230.000291] watchdog: BUG: soft lockup - CPU#7 stuck for 26036s! [kworker/7:4:78808] [30230.051043] irq event stamp: 50869200 [30230.058774] Modules linked in: ceph libceph dns_resolver fscache [30230.062431] hardirqs last enabled at (50869199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30230.062435] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [30230.078594] hardirqs last disabled at (50869200): [] sysvec_apic_timer_interrupt+0xe/0x90 [30230.086506] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [30230.096231] softirqs last enabled at (50869144): [] __do_softirq+0x2be/0x371 [30230.103017] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [30230.111699] softirqs last disabled at (50869139): [] irq_exit_rcu+0xb0/0xd0 [30230.111703] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [30230.127077] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30230.134025] iTCO_wdt iTCO_vendor_support [30230.142969] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30230.146978] i2c_smbus crct10dif_pclmul joydev crc32_pclmul [30230.154445] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30230.160009] ipmi_ssif ghash_clmulni_intel mei_me mei [30230.165489] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30230.170532] lpc_ich sha512_ssse3 [30230.189269] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30230.189269] acpi_ipmi mfd_core [30230.192581] [30230.197797] wmi ipmi_si [30230.200935] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30230.202424] ipmi_devintf ipmi_msghandler [30230.204956] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30230.212080] ioatdma [30230.216083] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30230.223207] acpi_pad acpi_power_meter [30230.225392] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30230.232515] ixgbe igb [30230.236261] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30230.243385] mdio crc32c_intel [30230.245742] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30230.252866] nvme ptp [30230.255918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30230.263994] nvme_core i2c_algo_bit [30230.266266] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30230.272004] pps_core [30230.275487] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30230.282610] dca fuse [30230.284881] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30230.292004] [30230.294276] Call Trace: [30230.294278] [30230.301400] irq event stamp: 163277588 [30230.302891] ? watchdog_timer_fn+0x1b4/0x220 [30230.305335] hardirqs last enabled at (163277587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30230.307346] ? __hrtimer_run_queues+0x1d3/0x360 [30230.311089] hardirqs last disabled at (163277588): [] sysvec_apic_timer_interrupt+0xe/0x90 [30230.315356] ? hrtimer_interrupt+0x104/0x240 [30230.325598] softirqs last enabled at (163277586): [] __do_softirq+0x2be/0x371 [30230.330122] ? __local_bh_enable+0x50/0x60 [30230.339932] softirqs last disabled at (163277581): [] irq_exit_rcu+0xb0/0xd0 [30230.344200] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30230.352979] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30230.357068] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30230.365664] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30230.370900] [30230.380278] Workqueue: events delayed_work [ceph] [30230.385235] [30230.385236] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30230.392705] [30230.394807] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30230.399499] RIP: 0010:lock_acquire+0xf0/0x2b0 [30230.401599] do_raw_spin_lock+0xb5/0xc0 [30230.406902] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30230.408394] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30230.413262] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30230.417616] try_get_cap_refs+0x159/0x880 [ceph] [30230.421444] [30230.421445] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30230.440190] __ceph_get_caps+0x4a1/0x760 [ceph] [30230.444965] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30230.450206] ? __file_remove_privs+0x89/0x110 [30230.454811] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30230.456302] ? wake_bit_function+0x70/0x70 [30230.463426] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30230.467969] ? ceph_write_iter+0x260/0x770 [ceph] [30230.475092] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30230.479444] ceph_write_iter+0x260/0x770 [ceph] [30230.486565] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30230.490662] ? lock_is_held_type+0xaa/0x120 [30230.497782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30230.502482] vfs_write+0x324/0x4f0 [30230.509603] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30230.514132] ksys_pwrite64+0x65/0xa0 [30230.522204] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30230.526382] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30230.532119] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30230.535518] do_syscall_64+0x38/0x80 [30230.542640] Call Trace: [30230.542642] [30230.546212] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30230.553338] ? watchdog_timer_fn+0x1b4/0x220 [30230.558466] RIP: 0033:0x7f48f0eeb93b [30230.565591] ? __hrtimer_run_queues+0x1d3/0x360 [30230.569162] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30230.571609] ? hrtimer_interrupt+0x104/0x240 [30230.573615] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30230.578661] ? __local_bh_enable+0x50/0x60 [30230.582924] ORIG_RAX: 0000000000000012 [30230.586498] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30230.591019] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30230.609758] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30230.614022] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30230.619240] [30230.623330] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30230.627160] [30230.632378] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30230.639503] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30230.644460] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30230.651588] ? lock_acquire+0xf0/0x2b0 [30230.653703] [30230.660823] ? igrab+0x43/0x50 [30230.691453] ? lock_release+0xc5/0x260 [30230.695197] _raw_spin_lock+0x30/0x40 [30230.698854] ? igrab+0x1d/0x50 [30230.701904] igrab+0x1d/0x50 [30230.704782] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30230.709932] delayed_work+0x1d8/0x280 [ceph] [30230.714212] ? process_scheduled_works+0x22a/0x510 [30230.719004] process_scheduled_works+0x2a4/0x510 [30230.723618] worker_thread+0x188/0x340 [30230.727368] ? rescuer_thread+0x310/0x310 [30230.731370] kthread+0xf5/0x130 [30230.734508] ? kthread_complete_and_exit+0x20/0x20 [30230.739293] ret_from_fork+0x31/0x40 [30230.742872] ? kthread_complete_and_exit+0x20/0x20 [30230.747656] ret_from_fork_asm+0x11/0x20 [30230.751580] [30257.988797] watchdog: BUG: soft lockup - CPU#0 stuck for 17065s! [dbench:78683] [30257.996096] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30258.000797] watchdog: BUG: soft lockup - CPU#7 stuck for 26062s! [kworker/7:4:78808] [30258.051538] irq event stamp: 50927232 [30258.051540] hardirqs last enabled at (50927231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30258.059270] Modules linked in: [30258.062925] hardirqs last disabled at (50927232): [] sysvec_apic_timer_interrupt+0xe/0x90 [30258.073083] ceph [30258.076133] softirqs last enabled at (50927174): [] __do_softirq+0x2be/0x371 [30258.085859] libceph dns_resolver [30258.087782] softirqs last disabled at (50927169): [] irq_exit_rcu+0xb0/0xd0 [30258.096471] fscache netfs [30258.099786] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30258.108295] veth nft_chain_nat [30258.111001] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30258.119945] xt_MASQUERADE nf_nat [30258.123082] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30258.130554] nf_conntrack nf_defrag_ipv6 [30258.133864] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30258.139342] nf_defrag_ipv4 nft_compat [30258.143257] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30258.161996] nf_tables nfnetlink [30258.165740] [30258.165741] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30258.170958] bridge stp [30258.174193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30258.175682] llc xfs [30258.182804] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30258.185249] libcrc32c [30258.192374] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30258.194557] sunrpc intel_rapl_msr [30258.201682] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30258.204039] intel_rapl_common sb_edac [30258.211164] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30258.214561] x86_pkg_temp_thermal coretemp [30258.221684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30258.225429] kvm_intel kvm [30258.233506] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30258.237597] irqbypass i2c_i801 [30258.243334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30258.246039] iTCO_wdt iTCO_vendor_support [30258.253162] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30258.256301] i2c_smbus crct10dif_pclmul [30258.263424] Call Trace: [30258.263426] [30258.267428] joydev crc32_pclmul [30258.274554] ? watchdog_timer_fn+0x1b4/0x220 [30258.278383] ipmi_ssif ghash_clmulni_intel [30258.280830] ? __hrtimer_run_queues+0x1d3/0x360 [30258.282837] mei_me mei [30258.286065] ? hrtimer_interrupt+0x104/0x240 [30258.290325] lpc_ich sha512_ssse3 [30258.294417] ? __local_bh_enable+0x50/0x60 [30258.298940] acpi_ipmi mfd_core [30258.301387] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30258.305648] wmi [30258.308959] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30258.313052] ipmi_si ipmi_devintf [30258.316196] [30258.321415] ipmi_msghandler [30258.323251] [30258.323253] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30258.328208] ioatdma acpi_pad [30258.331522] ? queued_spin_lock_slowpath+0x16/0x2e0 [30258.333617] acpi_power_meter ixgbe [30258.336496] do_raw_spin_lock+0xb5/0xc0 [30258.338590] igb mdio [30258.343897] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30258.346859] crc32c_intel nvme [30258.351735] try_get_cap_refs+0x159/0x880 [ceph] [30258.355213] ptp nvme_core [30258.359049] __ceph_get_caps+0x4a1/0x760 [ceph] [30258.361316] i2c_algo_bit pps_core [30258.366104] ? __file_remove_privs+0x89/0x110 [30258.369151] dca fuse [30258.373762] ? wake_bit_function+0x70/0x70 [30258.376478] [30258.376479] irq event stamp: 163498658 [30258.381001] ? ceph_write_iter+0x260/0x770 [ceph] [30258.384397] hardirqs last enabled at (163498657): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30258.388747] ceph_write_iter+0x260/0x770 [ceph] [30258.391016] hardirqs last disabled at (163498658): [] sysvec_apic_timer_interrupt+0xe/0x90 [30258.395112] ? lock_is_held_type+0xaa/0x120 [30258.396598] softirqs last enabled at (163498656): [] __do_softirq+0x2be/0x371 [30258.400345] vfs_write+0x324/0x4f0 [30258.405040] softirqs last disabled at (163498651): [] irq_exit_rcu+0xb0/0xd0 [30258.415290] ksys_pwrite64+0x65/0xa0 [30258.419809] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30258.429620] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30258.433797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30258.442568] do_syscall_64+0x38/0x80 [30258.445965] Workqueue: events delayed_work [ceph] [30258.454564] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30258.458134] [30258.458135] RIP: 0010:lock_release+0x14e/0x260 [30258.467510] RIP: 0033:0x7f48f0eeb93b [30258.472641] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [30258.480113] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30258.483682] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [30258.488382] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30258.493424] [30258.493425] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [30258.494915] ORIG_RAX: 0000000000000012 [30258.499352] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [30258.502924] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30258.521662] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [30258.540399] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30258.545616] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [30258.550834] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30258.552324] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [30258.559448] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30258.563278] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30258.570403] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30258.577529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30258.584658] [30258.591774] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30258.650520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30258.657645] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30258.664767] Call Trace: [30258.667213] [30258.669224] ? watchdog_timer_fn+0x1b4/0x220 [30258.673488] ? __hrtimer_run_queues+0x1d3/0x360 [30258.678015] ? hrtimer_interrupt+0x104/0x240 [30258.682283] ? __local_bh_enable+0x50/0x60 [30258.686377] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30258.691602] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30258.696557] [30258.698655] [30258.700754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30258.706066] ? igrab+0x43/0x50 [30258.709120] ? lock_release+0x14e/0x260 [30258.712958] ? lock_release+0xc5/0x260 [30258.716701] _raw_spin_unlock+0x1b/0x40 [30258.720531] igrab+0x43/0x50 [30258.723409] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30258.728557] delayed_work+0x1d8/0x280 [ceph] [30258.732840] ? process_scheduled_works+0x22a/0x510 [30258.737630] process_scheduled_works+0x2a4/0x510 [30258.742246] worker_thread+0x188/0x340 [30258.745995] ? rescuer_thread+0x310/0x310 [30258.749999] kthread+0xf5/0x130 [30258.753135] ? kthread_complete_and_exit+0x20/0x20 [30258.757920] ret_from_fork+0x31/0x40 [30258.761498] ? kthread_complete_and_exit+0x20/0x20 [30258.766282] ret_from_fork_asm+0x11/0x20 [30258.770209] [30285.989306] watchdog: BUG: soft lockup - CPU#0 stuck for 17092s! [dbench:78683] [30285.996612] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30286.001306] watchdog: BUG: soft lockup - CPU#7 stuck for 26089s! [kworker/7:4:78808] [30286.052062] irq event stamp: 50985378 [30286.059791] Modules linked in: ceph libceph dns_resolver fscache [30286.063447] hardirqs last enabled at (50985377): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30286.063452] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [30286.079612] hardirqs last disabled at (50985378): [] sysvec_apic_timer_interrupt+0xe/0x90 [30286.086217] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [30286.095940] softirqs last enabled at (50985320): [] __do_softirq+0x2be/0x371 [30286.095944] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [30286.112380] softirqs last disabled at (50985315): [] irq_exit_rcu+0xb0/0xd0 [30286.112384] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [30286.128104] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30286.135054] iTCO_wdt [30286.143997] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30286.146270] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [30286.153739] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30286.159909] crc32_pclmul ipmi_ssif [30286.165388] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30286.165389] ghash_clmulni_intel [30286.168871] mei_me [30286.187609] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30286.190834] mei lpc_ich [30286.192930] [30286.192931] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30286.198147] sha512_ssse3 acpi_ipmi [30286.200678] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30286.202188] mfd_core wmi [30286.209312] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30286.212794] ipmi_si ipmi_devintf [30286.219918] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30286.222536] ipmi_msghandler ioatdma [30286.229660] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30286.232985] acpi_pad [30286.240113] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30286.243701] acpi_power_meter [30286.250825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30286.253095] ixgbe igb [30286.261172] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30286.264136] mdio crc32c_intel [30286.269875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30286.272233] nvme ptp [30286.279355] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30286.282408] nvme_core [30286.289532] Call Trace: [30286.289534] [30286.291802] i2c_algo_bit [30286.298928] ? watchdog_timer_fn+0x1b4/0x220 [30286.301283] pps_core dca [30286.303729] ? __hrtimer_run_queues+0x1d3/0x360 [30286.305739] fuse [30286.308357] ? hrtimer_interrupt+0x104/0x240 [30286.312620] [30286.315237] ? __local_bh_enable+0x50/0x60 [30286.319761] irq event stamp: 163719454 [30286.321706] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30286.325967] hardirqs last enabled at (163719453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30286.327458] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30286.331549] hardirqs last disabled at (163719454): [] sysvec_apic_timer_interrupt+0xe/0x90 [30286.335293] [30286.340509] softirqs last enabled at (163719452): [] __do_softirq+0x2be/0x371 [30286.350753] [30286.355711] softirqs last disabled at (163719447): [] irq_exit_rcu+0xb0/0xd0 [30286.365523] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30286.367619] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30286.376394] ? queued_spin_lock_slowpath+0x16/0x2e0 [30286.378486] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30286.387086] do_raw_spin_lock+0xb5/0xc0 [30286.392388] Workqueue: events delayed_work [ceph] [30286.401768] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30286.406636] [30286.406638] RIP: 0010:lock_acquire+0xf0/0x2b0 [30286.414110] try_get_cap_refs+0x159/0x880 [ceph] [30286.417939] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30286.422641] __ceph_get_caps+0x4a1/0x760 [ceph] [30286.427421] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30286.428916] ? __file_remove_privs+0x89/0x110 [30286.433261] [30286.433262] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30286.437874] ? wake_bit_function+0x70/0x70 [30286.456610] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30286.461138] ? ceph_write_iter+0x260/0x770 [ceph] [30286.466350] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30286.470709] ceph_write_iter+0x260/0x770 [ceph] [30286.472201] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30286.479329] ? lock_is_held_type+0xaa/0x120 [30286.483417] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30286.490543] vfs_write+0x324/0x4f0 [30286.495237] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30286.502367] ksys_pwrite64+0x65/0xa0 [30286.506887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30286.514012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30286.518189] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30286.525312] do_syscall_64+0x38/0x80 [30286.528709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30286.536788] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30286.540358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30286.546096] RIP: 0033:0x7f48f0eeb93b [30286.551225] Call Trace: [30286.551226] [30286.558350] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30286.561921] ? watchdog_timer_fn+0x1b4/0x220 [30286.569045] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30286.574091] ? __hrtimer_run_queues+0x1d3/0x360 [30286.581212] ORIG_RAX: 0000000000000012 [30286.581213] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30286.584788] ? hrtimer_interrupt+0x104/0x240 [30286.587228] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30286.589237] ? __local_bh_enable+0x50/0x60 [30286.607992] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30286.612261] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30286.617474] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30286.622002] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30286.625829] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30286.632956] [30286.637224] [30286.644343] [30286.686369] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30286.691676] ? lock_acquire+0xf0/0x2b0 [30286.695425] ? igrab+0x43/0x50 [30286.698476] ? lock_release+0xc5/0x260 [30286.702221] _raw_spin_lock+0x30/0x40 [30286.705876] ? igrab+0x1d/0x50 [30286.708929] igrab+0x1d/0x50 [30286.711806] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30286.716957] delayed_work+0x1d8/0x280 [ceph] [30286.721254] ? process_scheduled_works+0x22a/0x510 [30286.726045] process_scheduled_works+0x2a4/0x510 [30286.730662] worker_thread+0x188/0x340 [30286.734410] ? rescuer_thread+0x310/0x310 [30286.738413] kthread+0xf5/0x130 [30286.741558] ? kthread_complete_and_exit+0x20/0x20 [30286.746342] ret_from_fork+0x31/0x40 [30286.749922] ? kthread_complete_and_exit+0x20/0x20 [30286.754714] ret_from_fork_asm+0x11/0x20 [30286.758639] [30313.989817] watchdog: BUG: soft lockup - CPU#0 stuck for 17118s! [dbench:78683] [30313.997115] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30314.001817] watchdog: BUG: soft lockup - CPU#7 stuck for 26115s! [kworker/7:4:78808] [30314.052558] irq event stamp: 51043414 [30314.060288] Modules linked in: ceph libceph [30314.063945] hardirqs last enabled at (51043413): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30314.068122] dns_resolver fscache [30314.078279] hardirqs last disabled at (51043414): [] sysvec_apic_timer_interrupt+0xe/0x90 [30314.081589] netfs veth nft_chain_nat [30314.091315] softirqs last enabled at (51043356): [] __do_softirq+0x2be/0x371 [30314.091317] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [30314.103665] softirqs last disabled at (51043351): [] irq_exit_rcu+0xb0/0xd0 [30314.110701] nft_compat nf_tables nfnetlink [30314.119214] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30314.123389] bridge stp [30314.132334] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30314.134777] llc xfs libcrc32c sunrpc [30314.142249] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30314.145905] intel_rapl_msr intel_rapl_common sb_edac [30314.151385] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30314.156428] x86_pkg_temp_thermal [30314.175166] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30314.175167] coretemp kvm_intel [30314.178476] [30314.178477] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30314.183694] kvm irqbypass [30314.186830] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30314.188322] i2c_i801 iTCO_wdt [30314.195446] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30314.198149] iTCO_vendor_support [30314.205273] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30314.208324] i2c_smbus crct10dif_pclmul [30314.215449] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30314.218673] joydev crc32_pclmul [30314.225798] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30314.229627] ipmi_ssif ghash_clmulni_intel [30314.236751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30314.239976] mei_me mei [30314.248054] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30314.252144] lpc_ich sha512_ssse3 [30314.257881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30314.260325] acpi_ipmi mfd_core [30314.267450] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30314.270760] wmi ipmi_si [30314.277884] Call Trace: [30314.277886] [30314.281022] ipmi_devintf ipmi_msghandler [30314.288148] ? watchdog_timer_fn+0x1b4/0x220 [30314.290677] ioatdma acpi_pad [30314.293122] ? __hrtimer_run_queues+0x1d3/0x360 [30314.295132] acpi_power_meter [30314.299139] ? hrtimer_interrupt+0x104/0x240 [30314.303399] ixgbe [30314.306365] ? __local_bh_enable+0x50/0x60 [30314.310888] igb mdio [30314.313856] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30314.318117] crc32c_intel nvme [30314.320128] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30314.324218] ptp [30314.326488] [30314.331705] nvme_core [30314.334757] [30314.334758] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30314.339715] i2c_algo_bit pps_core [30314.341556] ? queued_spin_lock_slowpath+0x18/0x2e0 [30314.343650] dca fuse [30314.346008] do_raw_spin_lock+0xb5/0xc0 [30314.348103] [30314.348103] irq event stamp: 163940350 [30314.353409] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30314.356806] hardirqs last enabled at (163940349): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30314.361677] try_get_cap_refs+0x159/0x880 [ceph] [30314.363945] hardirqs last disabled at (163940350): [] sysvec_apic_timer_interrupt+0xe/0x90 [30314.367782] __ceph_get_caps+0x4a1/0x760 [ceph] [30314.369269] softirqs last enabled at (163940348): [] __do_softirq+0x2be/0x371 [30314.373017] ? __file_remove_privs+0x89/0x110 [30314.377795] softirqs last disabled at (163940343): [] irq_exit_rcu+0xb0/0xd0 [30314.388040] ? wake_bit_function+0x70/0x70 [30314.392651] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30314.402465] ? ceph_write_iter+0x260/0x770 [ceph] [30314.406986] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30314.415756] ceph_write_iter+0x260/0x770 [ceph] [30314.420108] Workqueue: events delayed_work [ceph] [30314.428708] ? lock_is_held_type+0xaa/0x120 [30314.432795] [30314.442175] vfs_write+0x324/0x4f0 [30314.446871] RIP: 0010:lock_release+0x32/0x260 [30314.454345] ksys_pwrite64+0x65/0xa0 [30314.458866] Code: 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e <89> c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 [30314.463562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30314.467741] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [30314.469233] do_syscall_64+0x38/0x80 [30314.472628] [30314.472629] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [30314.476979] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30314.480548] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [30314.499288] RIP: 0033:0x7f48f0eeb93b [30314.504417] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [30314.509635] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30314.513221] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [30314.514715] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30314.521839] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30314.526884] ORIG_RAX: 0000000000000012 [30314.534006] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30314.537578] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30314.544701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30314.563439] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30314.570563] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30314.575781] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30314.582904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30314.586736] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30314.594812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30314.601936] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30314.607675] Call Trace: [30314.607677] [30314.614804] [30314.621924] ? watchdog_timer_fn+0x1b4/0x220 [30314.668449] ? __hrtimer_run_queues+0x1d3/0x360 [30314.672973] ? hrtimer_interrupt+0x104/0x240 [30314.677234] ? __local_bh_enable+0x50/0x60 [30314.681329] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30314.686551] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30314.691517] [30314.693614] [30314.695715] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30314.701018] ? igrab+0x43/0x50 [30314.704069] ? igrab+0x43/0x50 [30314.707119] ? lock_release+0x32/0x260 [30314.710864] ? lock_release+0xc5/0x260 [30314.714608] _raw_spin_unlock+0x1b/0x40 [30314.718439] igrab+0x43/0x50 [30314.721316] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30314.726466] delayed_work+0x1d8/0x280 [ceph] [30314.730748] ? process_scheduled_works+0x22a/0x510 [30314.735535] process_scheduled_works+0x2a4/0x510 [30314.740152] worker_thread+0x188/0x340 [30314.743900] ? rescuer_thread+0x310/0x310 [30314.747905] kthread+0xf5/0x130 [30314.751043] ? kthread_complete_and_exit+0x20/0x20 [30314.755827] ret_from_fork+0x31/0x40 [30314.759404] ? kthread_complete_and_exit+0x20/0x20 [30314.764191] ret_from_fork_asm+0x11/0x20 [30314.768122] [30341.990329] watchdog: BUG: soft lockup - CPU#0 stuck for 17144s! [dbench:78683] [30341.997639] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30342.002329] watchdog: BUG: soft lockup - CPU#7 stuck for 26141s! [kworker/7:4:78808] [30342.053088] irq event stamp: 51101526 [30342.053089] hardirqs last enabled at (51101525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30342.060819] Modules linked in: [30342.064475] hardirqs last disabled at (51101526): [] sysvec_apic_timer_interrupt+0xe/0x90 [30342.074632] ceph [30342.077683] softirqs last enabled at (51101490): [] __do_softirq+0x2be/0x371 [30342.087407] libceph [30342.089330] softirqs last disabled at (51101485): [] irq_exit_rcu+0xb0/0xd0 [30342.098015] dns_resolver fscache [30342.100201] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30342.108724] netfs [30342.112039] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30342.120983] veth nft_chain_nat [30342.123006] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [30342.130482] xt_MASQUERADE [30342.133619] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [30342.139096] nf_nat nf_conntrack [30342.141799] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30342.160538] nf_defrag_ipv6 [30342.163761] [30342.168980] nf_defrag_ipv4 nft_compat [30342.171771] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30342.173260] nf_tables nfnetlink [30342.177013] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30342.184137] bridge stp [30342.187363] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30342.194487] llc xfs [30342.196930] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30342.204055] libcrc32c sunrpc [30342.206239] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30342.213361] intel_rapl_msr intel_rapl_common [30342.216326] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30342.223451] sb_edac [30342.227802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30342.235878] x86_pkg_temp_thermal coretemp [30342.238063] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30342.243801] kvm_intel [30342.247890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30342.255015] kvm [30342.257373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30342.264497] irqbypass i2c_i801 [30342.266333] Call Trace: [30342.266335] [30342.273459] iTCO_wdt iTCO_vendor_support [30342.276597] ? watchdog_timer_fn+0x1b4/0x220 [30342.279039] i2c_smbus crct10dif_pclmul [30342.281053] ? __hrtimer_run_queues+0x1d3/0x360 [30342.285054] joydev [30342.289322] ? hrtimer_interrupt+0x104/0x240 [30342.293149] crc32_pclmul ipmi_ssif [30342.297673] ? __local_bh_enable+0x50/0x60 [30342.299771] ghash_clmulni_intel [30342.304036] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30342.307519] mei_me mei [30342.311610] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30342.314834] lpc_ich [30342.320052] [30342.322495] sha512_ssse3 acpi_ipmi [30342.327452] [30342.329637] mfd_core wmi [30342.331734] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30342.335218] ipmi_si ipmi_devintf [30342.337318] ? queued_spin_lock_slowpath+0x2b/0x2e0 [30342.339932] ipmi_msghandler ioatdma [30342.345240] do_raw_spin_lock+0xb5/0xc0 [30342.348547] acpi_pad acpi_power_meter [30342.353419] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30342.356990] ixgbe igb [30342.360823] try_get_cap_refs+0x159/0x880 [ceph] [30342.364565] mdio crc32c_intel [30342.369354] __ceph_get_caps+0x4a1/0x760 [ceph] [30342.371704] nvme ptp [30342.376337] ? __file_remove_privs+0x89/0x110 [30342.379384] nvme_core i2c_algo_bit [30342.383908] ? wake_bit_function+0x70/0x70 [30342.386178] pps_core dca [30342.390532] ? ceph_write_iter+0x260/0x770 [ceph] [30342.394019] fuse [30342.398114] ceph_write_iter+0x260/0x770 [ceph] [30342.400731] irq event stamp: 164161276 [30342.405431] ? lock_is_held_type+0xaa/0x120 [30342.407352] hardirqs last enabled at (164161275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30342.411880] vfs_write+0x324/0x4f0 [30342.415620] hardirqs last disabled at (164161276): [] sysvec_apic_timer_interrupt+0xe/0x90 [30342.419802] ksys_pwrite64+0x65/0xa0 [30342.430041] softirqs last enabled at (164161274): [] __do_softirq+0x2be/0x371 [30342.433441] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30342.443250] softirqs last disabled at (164161269): [] irq_exit_rcu+0xb0/0xd0 [30342.446822] do_syscall_64+0x38/0x80 [30342.455591] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30342.460732] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30342.469329] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30342.472900] RIP: 0033:0x7f48f0eeb93b [30342.482277] Workqueue: events delayed_work [ceph] [30342.487321] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30342.494792] [30342.494793] RIP: 0010:lock_acquire+0xf0/0x2b0 [30342.498361] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30342.503061] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30342.521797] ORIG_RAX: 0000000000000012 [30342.523288] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30342.527640] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30342.532856] [30342.532856] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30342.551594] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30342.555425] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30342.560643] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30342.567765] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30342.569256] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30342.576381] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30342.583504] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30342.590629] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30342.597759] [30342.604878] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30342.643655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30342.649390] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30342.656514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30342.663638] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30342.670762] Call Trace: [30342.673208] [30342.675219] ? watchdog_timer_fn+0x1b4/0x220 [30342.679482] ? __hrtimer_run_queues+0x1d3/0x360 [30342.684017] ? hrtimer_interrupt+0x104/0x240 [30342.688280] ? __local_bh_enable+0x50/0x60 [30342.692372] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30342.697596] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30342.702552] [30342.704651] [30342.706749] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30342.712055] ? lock_acquire+0xf0/0x2b0 [30342.715807] ? igrab+0x43/0x50 [30342.718856] ? lock_release+0xc5/0x260 [30342.722600] _raw_spin_lock+0x30/0x40 [30342.726257] ? igrab+0x1d/0x50 [30342.729308] igrab+0x1d/0x50 [30342.732187] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30342.737337] delayed_work+0x1d8/0x280 [ceph] [30342.741618] ? process_scheduled_works+0x22a/0x510 [30342.746407] process_scheduled_works+0x2a4/0x510 [30342.751024] worker_thread+0x188/0x340 [30342.754772] ? rescuer_thread+0x310/0x310 [30342.758774] kthread+0xf5/0x130 [30342.761913] ? kthread_complete_and_exit+0x20/0x20 [30342.766696] ret_from_fork+0x31/0x40 [30342.770276] ? kthread_complete_and_exit+0x20/0x20 [30342.775061] ret_from_fork_asm+0x11/0x20 [30342.778992] [30369.990840] watchdog: BUG: soft lockup - CPU#0 stuck for 17170s! [dbench:78683] [30369.998143] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30370.002840] watchdog: BUG: soft lockup - CPU#7 stuck for 26167s! [kworker/7:4:78808] [30370.053583] irq event stamp: 51159644 [30370.061314] Modules linked in: ceph libceph dns_resolver [30370.064970] hardirqs last enabled at (51159643): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30370.070276] fscache netfs veth [30370.080432] hardirqs last disabled at (51159644): [] sysvec_apic_timer_interrupt+0xe/0x90 [30370.083571] nft_chain_nat xt_MASQUERADE nf_nat [30370.093295] softirqs last enabled at (51159586): [] __do_softirq+0x2be/0x371 [30370.097819] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [30370.106504] softirqs last disabled at (51159581): [] irq_exit_rcu+0xb0/0xd0 [30370.112691] nf_tables nfnetlink bridge [30370.121202] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30370.125031] stp llc [30370.133975] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30370.133976] xfs [30370.136161] libcrc32c sunrpc [30370.143631] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30370.145469] intel_rapl_msr intel_rapl_common [30370.148433] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30370.153910] sb_edac x86_pkg_temp_thermal [30370.158260] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30370.176999] coretemp [30370.181002] [30370.181003] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30370.186236] kvm_intel [30370.188519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30370.190008] kvm irqbypass [30370.197131] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30370.199489] i2c_i801 iTCO_wdt [30370.206614] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30370.209318] iTCO_vendor_support i2c_smbus [30370.216440] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30370.219492] crct10dif_pclmul joydev [30370.226615] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30370.230708] crc32_pclmul ipmi_ssif [30370.237830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30370.241401] ghash_clmulni_intel mei_me [30370.249480] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30370.252964] mei lpc_ich [30370.258702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30370.262536] sha512_ssse3 [30370.269663] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30370.272194] acpi_ipmi mfd_core [30370.279318] Call Trace: [30370.279320] [30370.281937] wmi ipmi_si [30370.289062] ? watchdog_timer_fn+0x1b4/0x220 [30370.292198] ipmi_devintf ipmi_msghandler [30370.294642] ? __hrtimer_run_queues+0x1d3/0x360 [30370.296654] ioatdma acpi_pad [30370.299188] ? hrtimer_interrupt+0x104/0x240 [30370.303447] acpi_power_meter ixgbe [30370.307452] ? __local_bh_enable+0x50/0x60 [30370.311975] igb mdio [30370.314942] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30370.319205] crc32c_intel [30370.322690] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30370.326778] nvme ptp [30370.329048] [30370.334267] nvme_core i2c_algo_bit [30370.336884] [30370.336885] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30370.341841] pps_core dca [30370.344117] ? queued_spin_lock_slowpath+0x16/0x2e0 [30370.346210] fuse [30370.349695] do_raw_spin_lock+0xb5/0xc0 [30370.351791] [30370.351792] irq event stamp: 164382132 [30370.357096] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30370.359711] hardirqs last enabled at (164382131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30370.364585] try_get_cap_refs+0x159/0x880 [ceph] [30370.366521] hardirqs last disabled at (164382132): [] sysvec_apic_timer_interrupt+0xe/0x90 [30370.370361] __ceph_get_caps+0x4a1/0x760 [ceph] [30370.371846] softirqs last enabled at (164382130): [] __do_softirq+0x2be/0x371 [30370.375596] ? __file_remove_privs+0x89/0x110 [30370.380375] softirqs last disabled at (164382125): [] irq_exit_rcu+0xb0/0xd0 [30370.390619] ? wake_bit_function+0x70/0x70 [30370.395242] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30370.405052] ? ceph_write_iter+0x260/0x770 [ceph] [30370.409574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30370.418344] ceph_write_iter+0x260/0x770 [ceph] [30370.422694] Workqueue: events delayed_work [ceph] [30370.431297] ? lock_is_held_type+0xaa/0x120 [30370.435382] [30370.435383] RIP: 0010:lock_acquire+0xf0/0x2b0 [30370.444763] vfs_write+0x324/0x4f0 [30370.449457] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30370.456933] ksys_pwrite64+0x65/0xa0 [30370.461453] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30370.466152] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30370.470328] [30370.471819] do_syscall_64+0x38/0x80 [30370.476170] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30370.479568] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30370.498304] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30370.501874] RIP: 0033:0x7f48f0eeb93b [30370.507091] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30370.512239] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30370.513730] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30370.517302] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30370.524425] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30370.529469] ORIG_RAX: 0000000000000012 [30370.536593] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30370.540166] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30370.547288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30370.566026] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30370.573150] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30370.578367] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30370.585493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30370.589322] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30370.597400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30370.604526] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30370.610262] Call Trace: [30370.610264] [30370.617392] [30370.624524] ? watchdog_timer_fn+0x1b4/0x220 [30370.671054] ? __hrtimer_run_queues+0x1d3/0x360 [30370.675580] ? hrtimer_interrupt+0x104/0x240 [30370.679847] ? __local_bh_enable+0x50/0x60 [30370.683940] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30370.689156] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30370.694114] [30370.696210] [30370.698310] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30370.703617] ? lock_acquire+0xf0/0x2b0 [30370.707366] ? igrab+0x43/0x50 [30370.710418] ? lock_release+0xc5/0x260 [30370.714161] _raw_spin_lock+0x30/0x40 [30370.717816] ? igrab+0x1d/0x50 [30370.720867] igrab+0x1d/0x50 [30370.723748] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30370.728895] delayed_work+0x1d8/0x280 [ceph] [30370.733178] ? process_scheduled_works+0x22a/0x510 [30370.737969] process_scheduled_works+0x2a4/0x510 [30370.742585] worker_thread+0x188/0x340 [30370.746333] ? rescuer_thread+0x310/0x310 [30370.750336] kthread+0xf5/0x130 [30370.753473] ? kthread_complete_and_exit+0x20/0x20 [30370.758257] ret_from_fork+0x31/0x40 [30370.761827] ? kthread_complete_and_exit+0x20/0x20 [30370.766612] ret_from_fork_asm+0x11/0x20 [30370.770535] [30372.751257] rcu: INFO: rcu_preempt self-detected stall on CPU [30372.757018] rcu: 0-....: (18021234 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9015777 [30372.767444] rcu: (t=18435814 jiffies g=1309609 q=541643 ncpus=8) [30372.773529] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30372.782473] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30372.789943] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30372.795421] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30372.814159] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30372.819375] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30372.826499] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30372.833623] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30372.840747] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30372.847871] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30372.854997] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30372.863074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30372.868812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30372.875935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30372.883058] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30372.890185] Call Trace: [30372.892628] [30372.894638] ? rcu_dump_cpu_stacks+0xd1/0x180 [30372.898999] ? rcu_sched_clock_irq+0x633/0x1390 [30372.903530] ? lock_is_held_type+0xaa/0x120 [30372.907709] ? lock_is_held_type+0xaa/0x120 [30372.911886] ? tick_sched_do_timer+0xa0/0xa0 [30372.916157] ? rcu_read_lock_sched_held+0x56/0x90 [30372.920858] ? tick_sched_do_timer+0xa0/0xa0 [30372.925129] ? update_process_times+0x5f/0x90 [30372.929479] ? tick_sched_handle.isra.16+0x1e/0x50 [30372.934264] ? tick_sched_timer+0x6f/0x80 [30372.938268] ? __hrtimer_run_queues+0x1d3/0x360 [30372.942796] ? hrtimer_interrupt+0x104/0x240 [30372.947062] ? __local_bh_enable+0x50/0x60 [30372.951158] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30372.956380] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30372.961339] [30372.963435] [30372.965532] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30372.970843] ? queued_spin_lock_slowpath+0x16/0x2e0 [30372.975720] do_raw_spin_lock+0xb5/0xc0 [30372.979557] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30372.984359] try_get_cap_refs+0x159/0x880 [ceph] [30372.988991] __ceph_get_caps+0x4a1/0x760 [ceph] [30372.993532] ? __file_remove_privs+0x89/0x110 [30372.997887] ? wake_bit_function+0x70/0x70 [30373.001980] ? ceph_write_iter+0x260/0x770 [ceph] [30373.006691] ceph_write_iter+0x260/0x770 [ceph] [30373.011228] ? lock_is_held_type+0xaa/0x120 [30373.015412] vfs_write+0x324/0x4f0 [30373.018813] ksys_pwrite64+0x65/0xa0 [30373.022388] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30373.027518] do_syscall_64+0x38/0x80 [30373.031088] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30373.036132] RIP: 0033:0x7f48f0eeb93b [30373.039703] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30373.058441] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [30373.065999] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30373.073123] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30373.080246] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30373.087370] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30373.094495] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30373.101624] [30373.103813] Sending NMI from CPU 0 to CPUs 7: [30373.108165] NMI backtrace for cpu 7 [30373.108167] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30373.108168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30373.108169] Workqueue: events delayed_work [ceph] [30373.108184] RIP: 0010:mark_lock+0xa/0x4c0 [30373.108186] Code: 14 82 48 c7 c7 8a 3f 12 82 e8 42 bf f7 ff 0f 0b 31 c0 e9 8b fe ff ff 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 <41> 54 53 48 83 e4 f0 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 [30373.108187] RSP: 0018:ffffc90002117c38 EFLAGS: 00000002 [30373.108188] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: 0000000000000000 [30373.108189] RDX: 0000000000000006 RSI: ffff88810debb550 RDI: ffff88810debaa80 [30373.108190] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [30373.108190] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [30373.108191] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [30373.108192] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30373.108193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30373.108193] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30373.108194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30373.108195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30373.108195] Call Trace: [30373.108196] [30373.108197] ? nmi_cpu_backtrace+0xad/0x120 [30373.108200] ? nmi_cpu_backtrace_handler+0x11/0x20 [30373.108201] ? nmi_handle+0xd4/0x260 [30373.108205] ? default_do_nmi+0x49/0xf0 [30373.108207] ? exc_nmi+0x11e/0x150 [30373.108208] ? end_repeat_nmi+0x16/0x67 [30373.108212] ? mark_lock+0xa/0x4c0 [30373.108213] ? mark_lock+0xa/0x4c0 [30373.108215] ? mark_lock+0xa/0x4c0 [30373.108216] [30373.108216] [30373.108217] __lock_acquire+0x4be/0x1e20 [30373.108220] lock_acquire+0xcb/0x2b0 [30373.108221] ? igrab+0x1d/0x50 [30373.108223] ? igrab+0x43/0x50 [30373.108224] ? lock_release+0xc5/0x260 [30373.108226] _raw_spin_lock+0x30/0x40 [30373.108228] ? igrab+0x1d/0x50 [30373.108229] igrab+0x1d/0x50 [30373.108230] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30373.108244] delayed_work+0x1d8/0x280 [ceph] [30373.108256] ? process_scheduled_works+0x22a/0x510 [30373.108259] process_scheduled_works+0x2a4/0x510 [30373.108263] worker_thread+0x188/0x340 [30373.108267] ? rescuer_thread+0x310/0x310 [30373.108269] kthread+0xf5/0x130 [30373.108271] ? kthread_complete_and_exit+0x20/0x20 [30373.108274] ret_from_fork+0x31/0x40 [30373.108276] ? kthread_complete_and_exit+0x20/0x20 [30373.108278] ret_from_fork_asm+0x11/0x20 [30373.108281] [30397.991352] watchdog: BUG: soft lockup - CPU#0 stuck for 17196s! [dbench:78683] [30397.998656] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30398.003352] watchdog: BUG: soft lockup - CPU#7 stuck for 26193s! [kworker/7:4:78808] [30398.054106] irq event stamp: 51216986 [30398.054107] hardirqs last enabled at (51216985): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30398.061837] Modules linked in: [30398.065493] hardirqs last disabled at (51216986): [] sysvec_apic_timer_interrupt+0xe/0x90 [30398.075651] ceph [30398.078702] softirqs last enabled at (51216876): [] __do_softirq+0x2be/0x371 [30398.088425] libceph dns_resolver [30398.090350] softirqs last disabled at (51216871): [] irq_exit_rcu+0xb0/0xd0 [30398.099033] fscache netfs [30398.102347] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30398.110855] veth nft_chain_nat [30398.113560] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30398.122504] xt_MASQUERADE [30398.125642] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30398.133111] nf_nat nf_conntrack [30398.135815] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30398.141294] nf_defrag_ipv6 nf_defrag_ipv4 [30398.144517] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30398.163255] nft_compat nf_tables [30398.167345] [30398.172564] nfnetlink bridge [30398.175874] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30398.177363] stp llc [30398.180330] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30398.187454] xfs libcrc32c [30398.189636] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30398.196761] sunrpc [30398.199466] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30398.206590] intel_rapl_msr [30398.208686] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30398.215810] intel_rapl_common sb_edac [30398.218602] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30398.225726] x86_pkg_temp_thermal coretemp [30398.229486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30398.237564] kvm_intel kvm [30398.241654] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30398.247393] irqbypass i2c_i801 [30398.250097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30398.257222] iTCO_wdt iTCO_vendor_support [30398.260358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30398.267482] i2c_smbus crct10dif_pclmul [30398.271487] Call Trace: [30398.271489] [30398.278610] joydev [30398.282442] ? watchdog_timer_fn+0x1b4/0x220 [30398.284885] crc32_pclmul ipmi_ssif [30398.286897] ? __hrtimer_run_queues+0x1d3/0x360 [30398.288994] ghash_clmulni_intel mei_me [30398.293261] ? hrtimer_interrupt+0x104/0x240 [30398.296751] mei lpc_ich [30398.301273] ? __local_bh_enable+0x50/0x60 [30398.305104] sha512_ssse3 acpi_ipmi [30398.309371] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30398.311899] mfd_core wmi [30398.315991] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30398.319474] ipmi_si ipmi_devintf [30398.324692] [30398.327310] ipmi_msghandler ioatdma [30398.332268] [30398.332269] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30398.335577] acpi_pad acpi_power_meter [30398.337678] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30398.341246] ixgbe igb [30398.343345] do_raw_spin_lock+0xb5/0xc0 [30398.348646] mdio crc32c_intel [30398.352393] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30398.357262] nvme ptp [30398.359621] try_get_cap_refs+0x159/0x880 [ceph] [30398.363451] nvme_core i2c_algo_bit [30398.366507] __ceph_get_caps+0x4a1/0x760 [ceph] [30398.371284] pps_core dca [30398.373559] ? __file_remove_privs+0x89/0x110 [30398.378168] fuse [30398.381652] ? wake_bit_function+0x70/0x70 [30398.386174] [30398.386175] irq event stamp: 164601130 [30398.388793] ? ceph_write_iter+0x260/0x770 [ceph] [30398.393142] hardirqs last enabled at (164601129): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30398.395068] ceph_write_iter+0x260/0x770 [ceph] [30398.399158] hardirqs last disabled at (164601130): [] sysvec_apic_timer_interrupt+0xe/0x90 [30398.400653] ? lock_is_held_type+0xaa/0x120 [30398.404391] softirqs last enabled at (164601128): [] __do_softirq+0x2be/0x371 [30398.409091] vfs_write+0x324/0x4f0 [30398.419333] softirqs last disabled at (164601123): [] irq_exit_rcu+0xb0/0xd0 [30398.423863] ksys_pwrite64+0x65/0xa0 [30398.433669] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30398.437846] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30398.446616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30398.446617] Workqueue: events delayed_work [ceph] [30398.450032] do_syscall_64+0x38/0x80 [30398.458628] [30398.458629] RIP: 0010:do_raw_spin_lock+0x37/0xc0 [30398.462201] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30398.471578] Code: 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a <31> c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 [30398.476709] RIP: 0033:0x7f48f0eeb93b [30398.484178] RSP: 0018:ffffc90002117d90 EFLAGS: 00000282 [30398.488877] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30398.492448] [30398.492448] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [30398.493938] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30398.498549] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [30398.503594] ORIG_RAX: 0000000000000012 [30398.503595] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30398.522329] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [30398.525901] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30398.531117] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [30398.549856] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30398.551346] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30398.558470] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30398.563688] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30398.570813] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30398.574644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30398.581774] [30398.588891] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30398.654794] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30398.661919] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30398.669043] Call Trace: [30398.671487] [30398.673498] ? watchdog_timer_fn+0x1b4/0x220 [30398.677761] ? __hrtimer_run_queues+0x1d3/0x360 [30398.682290] ? hrtimer_interrupt+0x104/0x240 [30398.686558] ? __local_bh_enable+0x50/0x60 [30398.690652] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30398.695876] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30398.700833] [30398.702930] [30398.705040] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30398.710354] ? do_raw_spin_lock+0x37/0xc0 [30398.714362] igrab+0x1d/0x50 [30398.717241] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30398.722390] delayed_work+0x1d8/0x280 [ceph] [30398.726672] ? process_scheduled_works+0x22a/0x510 [30398.731462] process_scheduled_works+0x2a4/0x510 [30398.736076] worker_thread+0x188/0x340 [30398.739826] ? rescuer_thread+0x310/0x310 [30398.743828] kthread+0xf5/0x130 [30398.746968] ? kthread_complete_and_exit+0x20/0x20 [30398.751759] ret_from_fork+0x31/0x40 [30398.755339] ? kthread_complete_and_exit+0x20/0x20 [30398.760124] ret_from_fork_asm+0x11/0x20 [30398.764056] [30425.991851] watchdog: BUG: soft lockup - CPU#0 stuck for 17222s! [dbench:78683] [30425.999151] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30426.003852] watchdog: BUG: soft lockup - CPU#7 stuck for 26219s! [kworker/7:4:78808] [30426.054594] irq event stamp: 51275150 [30426.062324] Modules linked in: ceph libceph dns_resolver fscache [30426.065980] hardirqs last enabled at (51275149): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30426.071979] netfs veth nft_chain_nat [30426.082134] hardirqs last disabled at (51275150): [] sysvec_apic_timer_interrupt+0xe/0x90 [30426.085793] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [30426.095527] softirqs last enabled at (51275044): [] __do_softirq+0x2be/0x371 [30426.095530] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [30426.109957] softirqs last disabled at (51275039): [] irq_exit_rcu+0xb0/0xd0 [30426.115434] bridge stp llc xfs [30426.123947] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30426.127083] libcrc32c sunrpc [30426.136027] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30426.138989] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [30426.146462] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30426.153325] coretemp kvm_intel [30426.158802] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30426.158803] kvm irqbypass i2c_i801 [30426.180677] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30426.184163] iTCO_wdt iTCO_vendor_support [30426.189378] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30426.189379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30426.193384] i2c_smbus crct10dif_pclmul [30426.200524] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30426.207649] joydev crc32_pclmul [30426.211480] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30426.218604] ipmi_ssif ghash_clmulni_intel [30426.221828] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30426.228950] mei_me mei [30426.233043] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30426.240167] lpc_ich sha512_ssse3 [30426.242610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30426.250688] acpi_ipmi mfd_core [30426.253998] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30426.259736] wmi ipmi_si [30426.262873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30426.269998] ipmi_devintf ipmi_msghandler [30426.272531] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30426.279652] ioatdma acpi_pad [30426.283657] Call Trace: [30426.283658] [30426.290781] acpi_power_meter ixgbe [30426.293746] ? watchdog_timer_fn+0x1b4/0x220 [30426.296188] igb [30426.298202] ? __hrtimer_run_queues+0x1d3/0x360 [30426.301684] mdio crc32c_intel [30426.305952] ? hrtimer_interrupt+0x104/0x240 [30426.307784] nvme ptp [30426.312308] ? __local_bh_enable+0x50/0x60 [30426.315359] nvme_core i2c_algo_bit [30426.319627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30426.321894] pps_core [30426.325986] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30426.329468] dca fuse [30426.334688] [30426.336958] [30426.336959] irq event stamp: 164822102 [30426.341915] [30426.341917] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30426.344186] hardirqs last enabled at (164822101): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30426.346288] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30426.347773] hardirqs last disabled at (164822102): [] sysvec_apic_timer_interrupt+0xe/0x90 [30426.351534] do_raw_spin_lock+0xb5/0xc0 [30426.353633] softirqs last enabled at (164822100): [] __do_softirq+0x2be/0x371 [30426.358937] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30426.369181] softirqs last disabled at (164822095): [] irq_exit_rcu+0xb0/0xd0 [30426.374053] try_get_cap_refs+0x159/0x880 [ceph] [30426.383862] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30426.387699] __ceph_get_caps+0x4a1/0x760 [ceph] [30426.396464] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30426.401259] ? __file_remove_privs+0x89/0x110 [30426.409854] Workqueue: events delayed_work [ceph] [30426.414465] ? wake_bit_function+0x70/0x70 [30426.423843] [30426.423844] RIP: 0010:lock_acquire+0xf0/0x2b0 [30426.428368] ? ceph_write_iter+0x260/0x770 [ceph] [30426.435839] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30426.440188] ceph_write_iter+0x260/0x770 [ceph] [30426.444886] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30426.448980] ? lock_is_held_type+0xaa/0x120 [30426.450466] [30426.450467] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30426.454820] vfs_write+0x324/0x4f0 [30426.459531] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30426.478274] ksys_pwrite64+0x65/0xa0 [30426.482795] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30426.488012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30426.492190] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30426.493682] do_syscall_64+0x38/0x80 [30426.500804] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30426.504202] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30426.511324] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30426.514896] RIP: 0033:0x7f48f0eeb93b [30426.522021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30426.527151] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30426.534275] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30426.537846] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30426.544971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30426.550013] ORIG_RAX: 0000000000000012 [30426.558092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30426.561663] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30426.567401] Call Trace: [30426.567402] [30426.586138] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30426.593262] ? watchdog_timer_fn+0x1b4/0x220 [30426.598480] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30426.605605] ? __hrtimer_run_queues+0x1d3/0x360 [30426.609435] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30426.616562] ? hrtimer_interrupt+0x104/0x240 [30426.623683] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30426.626126] ? __local_bh_enable+0x50/0x60 [30426.628143] [30426.635264] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30426.681197] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30426.686155] [30426.688259] [30426.690356] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30426.695665] ? lock_acquire+0xf0/0x2b0 [30426.699416] ? igrab+0x43/0x50 [30426.702475] ? lock_release+0xc5/0x260 [30426.706219] _raw_spin_lock+0x30/0x40 [30426.709883] ? igrab+0x1d/0x50 [30426.712934] igrab+0x1d/0x50 [30426.715812] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30426.720963] delayed_work+0x1d8/0x280 [ceph] [30426.725260] ? process_scheduled_works+0x22a/0x510 [30426.730051] process_scheduled_works+0x2a4/0x510 [30426.734667] worker_thread+0x188/0x340 [30426.738416] ? rescuer_thread+0x310/0x310 [30426.742427] kthread+0xf5/0x130 [30426.745564] ? kthread_complete_and_exit+0x20/0x20 [30426.750350] ret_from_fork+0x31/0x40 [30426.753928] ? kthread_complete_and_exit+0x20/0x20 [30426.758711] ret_from_fork_asm+0x11/0x20 [30426.762635] [30453.992347] watchdog: BUG: soft lockup - CPU#0 stuck for 17248s! [dbench:78683] [30453.999648] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30454.004347] watchdog: BUG: soft lockup - CPU#7 stuck for 26245s! [kworker/7:4:78808] [30454.055097] irq event stamp: 51333214 [30454.055098] hardirqs last enabled at (51333213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30454.062829] Modules linked in: [30454.066485] hardirqs last disabled at (51333214): [] sysvec_apic_timer_interrupt+0xe/0x90 [30454.076643] ceph [30454.079694] softirqs last enabled at (51333194): [] __do_softirq+0x2be/0x371 [30454.089418] libceph [30454.091341] softirqs last disabled at (51333189): [] irq_exit_rcu+0xb0/0xd0 [30454.100031] dns_resolver fscache [30454.102219] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30454.110744] netfs veth [30454.114057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30454.123001] nft_chain_nat [30454.125463] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30454.132933] xt_MASQUERADE [30454.135638] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30454.141116] nf_nat nf_conntrack [30454.143819] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30454.162557] nf_defrag_ipv6 nf_defrag_ipv4 [30454.165783] [30454.170999] nft_compat [30454.175106] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30454.176598] nf_tables nfnetlink [30454.179042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30454.186166] bridge stp [30454.189389] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30454.196514] llc xfs [30454.198959] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30454.206081] libcrc32c sunrpc [30454.208267] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30454.215391] intel_rapl_msr [30454.218353] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30454.225479] intel_rapl_common [30454.228269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30454.236346] sb_edac [30454.239397] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30454.245136] x86_pkg_temp_thermal [30454.247320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30454.254444] coretemp kvm_intel [30454.257755] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30454.264879] kvm irqbypass [30454.268031] Call Trace: [30454.268033] [30454.275157] i2c_i801 iTCO_wdt [30454.277862] ? watchdog_timer_fn+0x1b4/0x220 [30454.280305] iTCO_vendor_support [30454.282316] ? __hrtimer_run_queues+0x1d3/0x360 [30454.285366] i2c_smbus crct10dif_pclmul [30454.289633] ? hrtimer_interrupt+0x104/0x240 [30454.292853] joydev crc32_pclmul [30454.297379] ? __local_bh_enable+0x50/0x60 [30454.301209] ipmi_ssif ghash_clmulni_intel [30454.305476] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30454.308696] mei_me mei [30454.312789] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30454.316880] lpc_ich sha512_ssse3 [30454.322096] [30454.324539] acpi_ipmi mfd_core [30454.329497] [30454.329498] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30454.332809] wmi ipmi_si [30454.334910] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30454.338044] ipmi_devintf [30454.340142] do_raw_spin_lock+0xb5/0xc0 [30454.345445] ipmi_msghandler [30454.347978] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30454.352846] ioatdma acpi_pad [30454.355467] try_get_cap_refs+0x159/0x880 [ceph] [30454.359295] acpi_power_meter ixgbe [30454.362178] __ceph_get_caps+0x4a1/0x760 [ceph] [30454.366957] igb mdio [30454.369926] ? __file_remove_privs+0x89/0x110 [30454.374531] crc32c_intel nvme [30454.378027] ? wake_bit_function+0x70/0x70 [30454.382548] ptp nvme_core [30454.384820] ? ceph_write_iter+0x260/0x770 [ceph] [30454.389169] i2c_algo_bit pps_core [30454.392219] ceph_write_iter+0x260/0x770 [ceph] [30454.396311] dca fuse [30454.399035] ? lock_is_held_type+0xaa/0x120 [30454.403745] [30454.407147] vfs_write+0x324/0x4f0 [30454.411668] irq event stamp: 165043112 [30454.413944] ksys_pwrite64+0x65/0xa0 [30454.418116] hardirqs last enabled at (165043111): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30454.419607] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30454.423005] hardirqs last disabled at (165043112): [] sysvec_apic_timer_interrupt+0xe/0x90 [30454.426766] do_syscall_64+0x38/0x80 [30454.430336] softirqs last enabled at (165043110): [] __do_softirq+0x2be/0x371 [30454.440583] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30454.445710] softirqs last disabled at (165043105): [] irq_exit_rcu+0xb0/0xd0 [30454.455522] RIP: 0033:0x7f48f0eeb93b [30454.459092] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30454.467865] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30454.472907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30454.481506] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30454.485076] Workqueue: events delayed_work [ceph] [30454.494452] ORIG_RAX: 0000000000000012 [30454.513191] [30454.520663] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30454.525879] RIP: 0010:lock_acquire+0xf0/0x2b0 [30454.530576] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30454.534408] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30454.535897] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30454.543029] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30454.547382] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30454.554506] [30454.554506] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30454.573243] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30454.580368] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30454.585589] [30454.592710] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30454.624881] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30454.632005] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30454.639129] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30454.647205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30454.652945] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30454.660068] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30454.667192] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30454.674316] Call Trace: [30454.676762] [30454.678772] ? watchdog_timer_fn+0x1b4/0x220 [30454.683036] ? __hrtimer_run_queues+0x1d3/0x360 [30454.687563] ? hrtimer_interrupt+0x104/0x240 [30454.691832] ? __local_bh_enable+0x50/0x60 [30454.695925] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30454.701149] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30454.706106] [30454.708205] [30454.710301] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30454.715610] ? lock_acquire+0xf0/0x2b0 [30454.719360] ? igrab+0x43/0x50 [30454.722417] ? lock_release+0xc5/0x260 [30454.726164] _raw_spin_lock+0x30/0x40 [30454.729829] ? igrab+0x1d/0x50 [30454.732877] igrab+0x1d/0x50 [30454.735758] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30454.740908] delayed_work+0x1d8/0x280 [ceph] [30454.745188] ? process_scheduled_works+0x22a/0x510 [30454.749977] process_scheduled_works+0x2a4/0x510 [30454.754594] worker_thread+0x188/0x340 [30454.758343] ? rescuer_thread+0x310/0x310 [30454.762346] kthread+0xf5/0x130 [30454.765492] ? kthread_complete_and_exit+0x20/0x20 [30454.770276] ret_from_fork+0x31/0x40 [30454.773857] ? kthread_complete_and_exit+0x20/0x20 [30454.778640] ret_from_fork_asm+0x11/0x20 [30454.782572] [30481.992843] watchdog: BUG: soft lockup - CPU#0 stuck for 17274s! [dbench:78683] [30482.000143] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30482.004843] watchdog: BUG: soft lockup - CPU#7 stuck for 26271s! [kworker/7:4:78808] [30482.055586] irq event stamp: 51391388 [30482.055588] hardirqs last enabled at (51391387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30482.063316] Modules linked in: [30482.066972] hardirqs last disabled at (51391388): [] sysvec_apic_timer_interrupt+0xe/0x90 [30482.077131] ceph libceph [30482.080182] softirqs last enabled at (51391282): [] __do_softirq+0x2be/0x371 [30482.089904] dns_resolver fscache [30482.092527] softirqs last disabled at (51391277): [] irq_exit_rcu+0xb0/0xd0 [30482.101207] netfs [30482.104535] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30482.113045] veth nft_chain_nat [30482.115055] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30482.124001] xt_MASQUERADE [30482.127138] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30482.134609] nf_nat nf_conntrack [30482.137313] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30482.142790] nf_defrag_ipv6 nf_defrag_ipv4 [30482.146015] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30482.164753] nft_compat nf_tables [30482.168844] [30482.168844] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30482.174060] nfnetlink bridge [30482.177371] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30482.178862] stp llc [30482.185987] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30482.188951] xfs libcrc32c [30482.196074] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30482.198257] sunrpc intel_rapl_msr [30482.205382] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30482.208087] intel_rapl_common sb_edac [30482.215211] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30482.218607] x86_pkg_temp_thermal coretemp [30482.225733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30482.229476] kvm_intel kvm [30482.237554] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30482.241645] irqbypass i2c_i801 [30482.247381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30482.250085] iTCO_wdt iTCO_vendor_support [30482.257210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30482.260346] i2c_smbus crct10dif_pclmul [30482.267471] Call Trace: [30482.267473] [30482.271476] joydev [30482.278600] ? watchdog_timer_fn+0x1b4/0x220 [30482.282431] crc32_pclmul ipmi_ssif [30482.284875] ? __hrtimer_run_queues+0x1d3/0x360 [30482.286886] ghash_clmulni_intel [30482.288986] ? hrtimer_interrupt+0x104/0x240 [30482.293246] mei_me mei [30482.296730] ? __local_bh_enable+0x50/0x60 [30482.301254] lpc_ich [30482.304481] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30482.308742] sha512_ssse3 acpi_ipmi [30482.311188] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30482.315278] mfd_core [30482.317461] [30482.322680] wmi ipmi_si [30482.326164] [30482.326166] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30482.331121] ipmi_devintf ipmi_msghandler [30482.333396] ? queued_spin_lock_slowpath+0x18/0x2e0 [30482.335488] ioatdma acpi_pad [30482.338020] do_raw_spin_lock+0xb5/0xc0 [30482.340117] acpi_power_meter ixgbe [30482.345421] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30482.349425] igb mdio crc32c_intel [30482.354297] try_get_cap_refs+0x159/0x880 [ceph] [30482.357260] nvme ptp [30482.361095] __ceph_get_caps+0x4a1/0x760 [ceph] [30482.364573] nvme_core i2c_algo_bit [30482.369365] ? __file_remove_privs+0x89/0x110 [30482.372755] pps_core dca [30482.377367] ? wake_bit_function+0x70/0x70 [30482.379637] fuse [30482.384164] ? ceph_write_iter+0x260/0x770 [ceph] [30482.387646] irq event stamp: 165263994 [30482.391998] ceph_write_iter+0x260/0x770 [ceph] [30482.394614] hardirqs last enabled at (165263993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30482.398710] ? lock_is_held_type+0xaa/0x120 [30482.400627] hardirqs last disabled at (165263994): [] sysvec_apic_timer_interrupt+0xe/0x90 [30482.405329] vfs_write+0x324/0x4f0 [30482.409070] softirqs last enabled at (165263992): [] __do_softirq+0x2be/0x371 [30482.413598] ksys_pwrite64+0x65/0xa0 [30482.423839] softirqs last disabled at (165263987): [] irq_exit_rcu+0xb0/0xd0 [30482.428017] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30482.437826] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30482.441242] do_syscall_64+0x38/0x80 [30482.450012] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30482.453584] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30482.462180] Workqueue: events delayed_work [ceph] [30482.467311] RIP: 0033:0x7f48f0eeb93b [30482.476688] [30482.476689] RIP: 0010:lock_acquire+0xf0/0x2b0 [30482.480260] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30482.487731] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30482.492773] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30482.497472] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30482.501043] ORIG_RAX: 0000000000000012 [30482.502534] [30482.502534] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30482.506885] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30482.525622] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30482.544358] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30482.549578] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30482.554794] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30482.558625] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30482.560115] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30482.567247] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30482.574373] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30482.581496] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30482.588626] [30482.595746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30482.647374] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30482.654496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30482.661621] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30482.668746] Call Trace: [30482.671189] [30482.673202] ? watchdog_timer_fn+0x1b4/0x220 [30482.677466] ? __hrtimer_run_queues+0x1d3/0x360 [30482.681992] ? hrtimer_interrupt+0x104/0x240 [30482.686252] ? __local_bh_enable+0x50/0x60 [30482.690346] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30482.695571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30482.700531] [30482.702633] [30482.704730] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30482.710038] ? lock_acquire+0xf0/0x2b0 [30482.713788] ? igrab+0x43/0x50 [30482.716839] ? lock_release+0xc5/0x260 [30482.720584] _raw_spin_lock+0x30/0x40 [30482.724249] ? igrab+0x1d/0x50 [30482.727300] igrab+0x1d/0x50 [30482.730178] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30482.735326] delayed_work+0x1d8/0x280 [ceph] [30482.739609] ? process_scheduled_works+0x22a/0x510 [30482.744397] process_scheduled_works+0x2a4/0x510 [30482.749013] worker_thread+0x188/0x340 [30482.752765] ? rescuer_thread+0x310/0x310 [30482.756777] kthread+0xf5/0x130 [30482.759913] ? kthread_complete_and_exit+0x20/0x20 [30482.764698] ret_from_fork+0x31/0x40 [30482.768276] ? kthread_complete_and_exit+0x20/0x20 [30482.773059] ret_from_fork_asm+0x11/0x20 [30482.776982] [30509.993339] watchdog: BUG: soft lockup - CPU#0 stuck for 17300s! [dbench:78683] [30510.000640] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30510.005339] watchdog: BUG: soft lockup - CPU#7 stuck for 26297s! [kworker/7:4:78808] [30510.056091] irq event stamp: 51449546 [30510.063822] Modules linked in: ceph libceph dns_resolver fscache netfs veth [30510.067479] hardirqs last enabled at (51449545): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30510.074438] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [30510.084596] hardirqs last disabled at (51449546): [] sysvec_apic_timer_interrupt+0xe/0x90 [30510.084599] nf_defrag_ipv6 [30510.084600] softirqs last enabled at (51449450): [] __do_softirq+0x2be/0x371 [30510.090254] nf_defrag_ipv4 [30510.099978] softirqs last disabled at (51449445): [] irq_exit_rcu+0xb0/0xd0 [30510.102769] nft_compat [30510.111453] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30510.114244] nf_tables [30510.122755] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30510.125199] nfnetlink bridge [30510.134142] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30510.136502] stp llc [30510.143970] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30510.146935] xfs libcrc32c [30510.152412] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30510.154596] sunrpc intel_rapl_msr [30510.173335] [30510.173336] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30510.176038] intel_rapl_common [30510.181256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30510.184654] sb_edac x86_pkg_temp_thermal [30510.186144] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30510.193267] coretemp kvm_intel [30510.196318] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30510.203443] kvm irqbypass [30510.207447] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30510.214572] i2c_i801 iTCO_wdt [30510.217708] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30510.224832] iTCO_vendor_support [30510.227536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30510.234662] i2c_smbus [30510.237712] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30510.245807] crct10dif_pclmul joydev [30510.249030] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30510.254769] crc32_pclmul [30510.257125] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30510.264249] ipmi_ssif ghash_clmulni_intel [30510.267821] Call Trace: [30510.274944] mei_me mei [30510.277563] [30510.284686] lpc_ich sha512_ssse3 [30510.288779] ? watchdog_timer_fn+0x1b4/0x220 [30510.291221] acpi_ipmi mfd_core [30510.293666] ? __hrtimer_run_queues+0x1d3/0x360 [30510.295676] wmi ipmi_si [30510.298991] ? hrtimer_interrupt+0x104/0x240 [30510.303251] ipmi_devintf ipmi_msghandler [30510.306387] ? __local_bh_enable+0x50/0x60 [30510.310911] ioatdma acpi_pad [30510.313445] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30510.317706] acpi_power_meter [30510.321712] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30510.325820] ixgbe igb [30510.328784] [30510.333999] mdio crc32c_intel [30510.336965] [30510.341920] nvme ptp [30510.344281] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30510.346377] nvme_core i2c_algo_bit [30510.349430] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30510.351525] pps_core [30510.353798] do_raw_spin_lock+0xb5/0xc0 [30510.359098] dca fuse [30510.362587] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30510.367453] [30510.367454] irq event stamp: 165485404 [30510.369744] try_get_cap_refs+0x159/0x880 [ceph] [30510.373572] hardirqs last enabled at (165485403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30510.375848] __ceph_get_caps+0x4a1/0x760 [ceph] [30510.380627] hardirqs last disabled at (165485404): [] sysvec_apic_timer_interrupt+0xe/0x90 [30510.382123] ? __file_remove_privs+0x89/0x110 [30510.385862] softirqs last enabled at (165485402): [] __do_softirq+0x2be/0x371 [30510.390475] ? wake_bit_function+0x70/0x70 [30510.400736] softirqs last disabled at (165485397): [] irq_exit_rcu+0xb0/0xd0 [30510.405263] ? ceph_write_iter+0x260/0x770 [ceph] [30510.415069] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30510.419421] ceph_write_iter+0x260/0x770 [ceph] [30510.428192] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30510.432286] ? lock_is_held_type+0xaa/0x120 [30510.440881] Workqueue: events delayed_work [ceph] [30510.445579] vfs_write+0x324/0x4f0 [30510.454955] [30510.454956] RIP: 0010:lock_acquire+0x16/0x2b0 [30510.459482] ksys_pwrite64+0x65/0xa0 [30510.466949] Code: 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd <41> 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 [30510.471128] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30510.475823] RSP: 0018:ffffc90002117d68 EFLAGS: 00000246 [30510.479223] do_syscall_64+0x38/0x80 [30510.480712] [30510.480714] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [30510.485083] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30510.488652] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [30510.507389] RIP: 0033:0x7f48f0eeb93b [30510.512519] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [30510.517742] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30510.521316] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [30510.522807] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30510.529930] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30510.534974] ORIG_RAX: 0000000000000012 [30510.542100] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30510.545671] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30510.552793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30510.571531] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30510.578655] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30510.583872] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30510.590999] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30510.594846] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30510.602923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30510.610048] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30510.615786] Call Trace: [30510.615788] [30510.622915] [30510.630036] ? watchdog_timer_fn+0x1b4/0x220 [30510.676559] ? __hrtimer_run_queues+0x1d3/0x360 [30510.681085] ? hrtimer_interrupt+0x104/0x240 [30510.685354] ? __local_bh_enable+0x50/0x60 [30510.689446] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30510.694671] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30510.699627] [30510.701744] [30510.703842] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30510.709147] ? lock_acquire+0x16/0x2b0 [30510.712899] _raw_spin_lock+0x30/0x40 [30510.716563] ? igrab+0x1d/0x50 [30510.719614] igrab+0x1d/0x50 [30510.722493] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30510.727642] delayed_work+0x1d8/0x280 [ceph] [30510.731923] ? process_scheduled_works+0x22a/0x510 [30510.736714] process_scheduled_works+0x2a4/0x510 [30510.741355] worker_thread+0x188/0x340 [30510.745105] ? rescuer_thread+0x310/0x310 [30510.749116] kthread+0xf5/0x130 [30510.752255] ? kthread_complete_and_exit+0x20/0x20 [30510.757046] ret_from_fork+0x31/0x40 [30510.760626] ? kthread_complete_and_exit+0x20/0x20 [30510.765409] ret_from_fork_asm+0x11/0x20 [30510.769334] [30537.993837] watchdog: BUG: soft lockup - CPU#0 stuck for 17326s! [dbench:78683] [30538.001137] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30538.005837] watchdog: BUG: soft lockup - CPU#7 stuck for 26323s! [kworker/7:4:78808] [30538.056578] irq event stamp: 51507604 [30538.056579] hardirqs last enabled at (51507603): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30538.064308] Modules linked in: [30538.067966] hardirqs last disabled at (51507604): [] sysvec_apic_timer_interrupt+0xe/0x90 [30538.078123] ceph [30538.081173] softirqs last enabled at (51507578): [] __do_softirq+0x2be/0x371 [30538.090898] libceph [30538.092821] softirqs last disabled at (51507573): [] irq_exit_rcu+0xb0/0xd0 [30538.101513] dns_resolver fscache [30538.103699] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30538.112209] netfs [30538.115537] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30538.115538] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30538.124482] veth nft_chain_nat [30538.126510] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30538.133981] xt_MASQUERADE nf_nat [30538.139458] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30538.142597] nf_conntrack nf_defrag_ipv6 [30538.161333] [30538.161334] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30538.164644] nf_defrag_ipv4 nft_compat [30538.169862] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30538.173780] nf_tables [30538.175269] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30538.182394] nfnetlink bridge [30538.186138] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30538.193262] stp [30538.195619] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30538.202744] llc [30538.205708] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30538.212832] xfs libcrc32c [30538.214669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30538.221793] sunrpc intel_rapl_msr [30538.223629] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30538.231707] intel_rapl_common sb_edac [30538.234412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30538.240150] x86_pkg_temp_thermal coretemp [30538.243546] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30538.250671] kvm_intel kvm [30538.254416] Call Trace: [30538.261540] irqbypass i2c_i801 [30538.265630] [30538.272754] iTCO_wdt iTCO_vendor_support [30538.275459] ? watchdog_timer_fn+0x1b4/0x220 [30538.277902] i2c_smbus crct10dif_pclmul [30538.281039] ? __hrtimer_run_queues+0x1d3/0x360 [30538.283050] joydev crc32_pclmul [30538.287057] ? hrtimer_interrupt+0x104/0x240 [30538.291319] ipmi_ssif ghash_clmulni_intel [30538.295150] ? __local_bh_enable+0x50/0x60 [30538.299673] mei_me mei [30538.302900] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30538.307160] lpc_ich sha512_ssse3 [30538.311254] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30538.315342] acpi_ipmi mfd_core [30538.317786] [30538.323005] wmi [30538.326314] [30538.326316] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30538.331273] ipmi_si [30538.334413] ? queued_spin_lock_slowpath+0x16/0x2e0 [30538.336516] ipmi_devintf ipmi_msghandler [30538.338354] do_raw_spin_lock+0xb5/0xc0 [30538.340452] ioatdma acpi_pad [30538.345756] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30538.347939] acpi_power_meter ixgbe [30538.352811] try_get_cap_refs+0x159/0x880 [ceph] [30538.356815] igb mdio [30538.360650] __ceph_get_caps+0x4a1/0x760 [ceph] [30538.363608] crc32c_intel nvme [30538.368396] ? __file_remove_privs+0x89/0x110 [30538.371877] ptp nvme_core [30538.376488] ? wake_bit_function+0x70/0x70 [30538.378758] i2c_algo_bit pps_core [30538.383286] ? ceph_write_iter+0x260/0x770 [ceph] [30538.386334] dca fuse [30538.390684] ceph_write_iter+0x260/0x770 [ceph] [30538.393388] [30538.393389] irq event stamp: 165706400 [30538.397482] ? lock_is_held_type+0xaa/0x120 [30538.400876] hardirqs last enabled at (165706399): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30538.405577] vfs_write+0x324/0x4f0 [30538.407844] hardirqs last disabled at (165706400): [] sysvec_apic_timer_interrupt+0xe/0x90 [30538.412373] ksys_pwrite64+0x65/0xa0 [30538.413859] softirqs last enabled at (165706398): [] __do_softirq+0x2be/0x371 [30538.417603] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30538.421779] softirqs last disabled at (165706393): [] irq_exit_rcu+0xb0/0xd0 [30538.432025] do_syscall_64+0x38/0x80 [30538.435421] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30538.445238] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30538.448811] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30538.457584] RIP: 0033:0x7f48f0eeb93b [30538.462714] Workqueue: events delayed_work [ceph] [30538.471311] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30538.474881] [30538.474882] RIP: 0010:lock_acquire+0xf0/0x2b0 [30538.484259] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30538.489304] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30538.496773] ORIG_RAX: 0000000000000012 [30538.500345] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30538.505043] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30538.523780] [30538.523780] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30538.525272] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30538.529622] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30538.534838] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30538.553577] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30538.557406] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30538.562623] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30538.569749] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30538.571239] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30538.578369] [30538.585488] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30538.645627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30538.651365] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30538.658488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30538.665612] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30538.672738] Call Trace: [30538.675182] [30538.677193] ? watchdog_timer_fn+0x1b4/0x220 [30538.681456] ? __hrtimer_run_queues+0x1d3/0x360 [30538.685982] ? hrtimer_interrupt+0x104/0x240 [30538.690243] ? __local_bh_enable+0x50/0x60 [30538.694338] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30538.699562] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30538.704526] [30538.706625] [30538.708723] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30538.714031] ? lock_acquire+0xf0/0x2b0 [30538.717780] ? igrab+0x43/0x50 [30538.720830] ? lock_release+0xc5/0x260 [30538.724574] _raw_spin_lock+0x30/0x40 [30538.728244] ? igrab+0x1d/0x50 [30538.731298] igrab+0x1d/0x50 [30538.734179] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30538.739326] delayed_work+0x1d8/0x280 [ceph] [30538.743610] ? process_scheduled_works+0x22a/0x510 [30538.748399] process_scheduled_works+0x2a4/0x510 [30538.753013] worker_thread+0x188/0x340 [30538.756764] ? rescuer_thread+0x310/0x310 [30538.760766] kthread+0xf5/0x130 [30538.763904] ? kthread_complete_and_exit+0x20/0x20 [30538.768687] ret_from_fork+0x31/0x40 [30538.772266] ? kthread_complete_and_exit+0x20/0x20 [30538.777051] ret_from_fork_asm+0x11/0x20 [30538.780973] [30565.994343] watchdog: BUG: soft lockup - CPU#0 stuck for 17352s! [dbench:78683] [30566.001641] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30566.006343] watchdog: BUG: soft lockup - CPU#7 stuck for 26349s! [kworker/7:4:78808] [30566.057084] irq event stamp: 51565734 [30566.057085] hardirqs last enabled at (51565733): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30566.064815] Modules linked in: ceph [30566.068470] hardirqs last disabled at (51565734): [] sysvec_apic_timer_interrupt+0xe/0x90 [30566.078628] libceph [30566.082112] softirqs last enabled at (51565626): [] __do_softirq+0x2be/0x371 [30566.091836] dns_resolver [30566.094021] softirqs last disabled at (51565621): [] irq_exit_rcu+0xb0/0xd0 [30566.102704] fscache netfs [30566.105322] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30566.113832] veth [30566.116535] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30566.125480] nft_chain_nat [30566.127407] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30566.134876] xt_MASQUERADE nf_nat [30566.137581] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30566.143057] nf_conntrack nf_defrag_ipv6 [30566.146369] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30566.165105] nf_defrag_ipv4 [30566.169022] [30566.169023] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30566.174239] nft_compat nf_tables [30566.177031] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30566.178522] nfnetlink bridge [30566.185645] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30566.188956] stp llc [30566.196081] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30566.199045] xfs libcrc32c [30566.206170] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30566.208353] sunrpc intel_rapl_msr [30566.215478] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30566.218180] intel_rapl_common [30566.225304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30566.228702] sb_edac x86_pkg_temp_thermal [30566.236780] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30566.239831] coretemp kvm_intel [30566.245567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30566.249573] kvm [30566.256697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30566.259833] irqbypass i2c_i801 [30566.266957] Call Trace: [30566.268796] iTCO_wdt [30566.275921] [30566.279057] iTCO_vendor_support i2c_smbus [30566.281502] ? watchdog_timer_fn+0x1b4/0x220 [30566.283771] crct10dif_pclmul joydev [30566.285783] ? __hrtimer_run_queues+0x1d3/0x360 [30566.289872] crc32_pclmul ipmi_ssif [30566.294139] ? hrtimer_interrupt+0x104/0x240 [30566.297708] ghash_clmulni_intel mei_me [30566.302232] ? __local_bh_enable+0x50/0x60 [30566.305717] mei lpc_ich [30566.310000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30566.313846] sha512_ssse3 acpi_ipmi [30566.317938] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30566.320467] mfd_core wmi [30566.325684] [30566.329169] ipmi_si ipmi_devintf [30566.334126] [30566.336742] ipmi_msghandler [30566.338842] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30566.342151] ioatdma acpi_pad [30566.344252] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30566.347126] acpi_power_meter ixgbe [30566.352431] do_raw_spin_lock+0xb5/0xc0 [30566.355394] igb mdio [30566.360268] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30566.363749] crc32c_intel nvme [30566.367583] try_get_cap_refs+0x159/0x880 [ceph] [30566.369850] ptp nvme_core [30566.374639] __ceph_get_caps+0x4a1/0x760 [ceph] [30566.377686] i2c_algo_bit pps_core [30566.382302] ? __file_remove_privs+0x89/0x110 [30566.384999] dca fuse [30566.389524] ? wake_bit_function+0x70/0x70 [30566.392920] [30566.392921] irq event stamp: 165927092 [30566.397274] ? ceph_write_iter+0x260/0x770 [ceph] [30566.399544] hardirqs last enabled at (165927091): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30566.403634] ceph_write_iter+0x260/0x770 [ceph] [30566.405124] hardirqs last disabled at (165927092): [] sysvec_apic_timer_interrupt+0xe/0x90 [30566.408874] ? lock_is_held_type+0xaa/0x120 [30566.413566] softirqs last enabled at (165927090): [] __do_softirq+0x2be/0x371 [30566.423814] vfs_write+0x324/0x4f0 [30566.428336] softirqs last disabled at (165927085): [] irq_exit_rcu+0xb0/0xd0 [30566.438151] ksys_pwrite64+0x65/0xa0 [30566.442323] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30566.451095] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30566.454490] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30566.463090] do_syscall_64+0x38/0x80 [30566.466659] Workqueue: events delayed_work [ceph] [30566.476044] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30566.481175] [30566.488647] RIP: 0033:0x7f48f0eeb93b [30566.492217] RIP: 0010:lock_acquire+0xf0/0x2b0 [30566.496915] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30566.501959] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30566.503449] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30566.507024] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30566.511371] ORIG_RAX: 0000000000000012 [30566.530108] [30566.530109] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30566.548847] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30566.554063] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30566.559280] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30566.563113] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30566.564603] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30566.571745] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30566.578869] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30566.585994] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30566.593117] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30566.600240] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30566.607370] [30566.614490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30566.651869] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30566.658992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30566.666118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30566.673242] Call Trace: [30566.675688] [30566.677698] ? watchdog_timer_fn+0x1b4/0x220 [30566.681961] ? __hrtimer_run_queues+0x1d3/0x360 [30566.686487] ? hrtimer_interrupt+0x104/0x240 [30566.690751] ? __local_bh_enable+0x50/0x60 [30566.694842] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30566.700067] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30566.705038] [30566.707137] [30566.709234] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30566.714544] ? lock_acquire+0xf0/0x2b0 [30566.718294] ? igrab+0x43/0x50 [30566.721342] ? lock_release+0xc5/0x260 [30566.725088] _raw_spin_lock+0x30/0x40 [30566.728749] ? igrab+0x1d/0x50 [30566.731803] igrab+0x1d/0x50 [30566.734683] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30566.739832] delayed_work+0x1d8/0x280 [ceph] [30566.744114] ? process_scheduled_works+0x22a/0x510 [30566.748902] process_scheduled_works+0x2a4/0x510 [30566.753520] worker_thread+0x188/0x340 [30566.757267] ? rescuer_thread+0x310/0x310 [30566.761272] kthread+0xf5/0x130 [30566.764410] ? kthread_complete_and_exit+0x20/0x20 [30566.769194] ret_from_fork+0x31/0x40 [30566.772771] ? kthread_complete_and_exit+0x20/0x20 [30566.777557] ret_from_fork_asm+0x11/0x20 [30566.781480] [30568.114747] rcu: INFO: rcu_preempt self-detected stall on CPU [30568.120508] rcu: 0-....: (18211910 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9111592 [30568.130933] rcu: (t=18631174 jiffies g=1309609 q=546228 ncpus=8) [30568.137018] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30568.145961] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30568.153432] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30568.158908] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30568.177648] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30568.182865] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30568.189989] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30568.197112] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30568.204236] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30568.211361] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30568.218485] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30568.226564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30568.232300] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30568.239423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30568.246549] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30568.253673] Call Trace: [30568.256116] [30568.258129] ? rcu_dump_cpu_stacks+0xd1/0x180 [30568.262488] ? rcu_sched_clock_irq+0x633/0x1390 [30568.267019] ? lock_is_held_type+0xaa/0x120 [30568.271197] ? lock_is_held_type+0xaa/0x120 [30568.275375] ? tick_sched_do_timer+0xa0/0xa0 [30568.279647] ? rcu_read_lock_sched_held+0x56/0x90 [30568.284344] ? tick_sched_do_timer+0xa0/0xa0 [30568.288607] ? update_process_times+0x5f/0x90 [30568.292960] ? tick_sched_handle.isra.16+0x1e/0x50 [30568.297743] ? tick_sched_timer+0x6f/0x80 [30568.301750] ? __hrtimer_run_queues+0x1d3/0x360 [30568.306284] ? hrtimer_interrupt+0x104/0x240 [30568.310552] ? __local_bh_enable+0x50/0x60 [30568.314645] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30568.319862] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30568.324818] [30568.326914] [30568.329014] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30568.334321] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30568.339199] do_raw_spin_lock+0xb5/0xc0 [30568.343037] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30568.347841] try_get_cap_refs+0x159/0x880 [ceph] [30568.352472] __ceph_get_caps+0x4a1/0x760 [ceph] [30568.357011] ? __file_remove_privs+0x89/0x110 [30568.361366] ? wake_bit_function+0x70/0x70 [30568.365461] ? ceph_write_iter+0x260/0x770 [ceph] [30568.370172] ceph_write_iter+0x260/0x770 [ceph] [30568.374708] ? lock_is_held_type+0xaa/0x120 [30568.378893] vfs_write+0x324/0x4f0 [30568.382302] ksys_pwrite64+0x65/0xa0 [30568.385877] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30568.391008] do_syscall_64+0x38/0x80 [30568.394578] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30568.399623] RIP: 0033:0x7f48f0eeb93b [30568.403192] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30568.421929] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [30568.429486] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30568.436611] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30568.443737] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30568.450859] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30568.457984] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30568.465115] [30568.467302] Sending NMI from CPU 0 to CPUs 7: [30568.471653] NMI backtrace for cpu 7 [30568.471655] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30568.471657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30568.471657] Workqueue: events delayed_work [ceph] [30568.471673] RIP: 0010:__lock_acquire+0xf7/0x1e20 [30568.471675] Code: ec c4 4e 49 c1 fc 04 48 89 34 24 4c 0f af e0 8b 44 24 10 4c 8d 04 80 49 c1 e0 03 85 c0 74 25 41 f6 c5 01 75 1f 49 8d 74 30 d8 <0f> b7 46 20 25 ff 1f 00 00 44 39 e0 75 0c 48 83 7c 24 08 00 0f 85 [30568.471676] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [30568.471677] RAX: 0000000000000003 RBX: ffff88810debaa80 RCX: 0000000000000000 [30568.471678] RDX: 0000000000000000 RSI: ffff88810debb528 RDI: ffff8881a1d34518 [30568.471679] RBP: 0000000000000001 R08: 0000000000000078 R09: 0000000000000000 [30568.471679] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [30568.471680] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8881a1d34518 [30568.471681] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30568.471682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30568.471682] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30568.471683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30568.471684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30568.471685] Call Trace: [30568.471685] [30568.471686] ? nmi_cpu_backtrace+0xad/0x120 [30568.471689] ? nmi_cpu_backtrace_handler+0x11/0x20 [30568.471691] ? nmi_handle+0xd4/0x260 [30568.471695] ? default_do_nmi+0x49/0xf0 [30568.471696] ? exc_nmi+0x11e/0x150 [30568.471698] ? end_repeat_nmi+0x16/0x67 [30568.471701] ? __lock_acquire+0xf7/0x1e20 [30568.471703] ? __lock_acquire+0xf7/0x1e20 [30568.471704] ? __lock_acquire+0xf7/0x1e20 [30568.471706] [30568.471706] [30568.471708] lock_acquire+0xcb/0x2b0 [30568.471709] ? igrab+0x1d/0x50 [30568.471711] ? igrab+0x43/0x50 [30568.471712] ? lock_release+0xc5/0x260 [30568.471713] _raw_spin_lock+0x30/0x40 [30568.471715] ? igrab+0x1d/0x50 [30568.471716] igrab+0x1d/0x50 [30568.471718] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30568.471731] delayed_work+0x1d8/0x280 [ceph] [30568.471745] ? process_scheduled_works+0x22a/0x510 [30568.471747] process_scheduled_works+0x2a4/0x510 [30568.471751] worker_thread+0x188/0x340 [30568.471754] ? rescuer_thread+0x310/0x310 [30568.471757] kthread+0xf5/0x130 [30568.471759] ? kthread_complete_and_exit+0x20/0x20 [30568.471762] ret_from_fork+0x31/0x40 [30568.471764] ? kthread_complete_and_exit+0x20/0x20 [30568.471766] ret_from_fork_asm+0x11/0x20 [30568.471770] [30593.994849] watchdog: BUG: soft lockup - CPU#0 stuck for 17378s! [dbench:78683] [30594.002155] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30594.006849] watchdog: BUG: soft lockup - CPU#7 stuck for 26375s! [kworker/7:4:78808] [30594.057606] irq event stamp: 51623110 [30594.057607] hardirqs last enabled at (51623109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30594.065336] Modules linked in: [30594.068994] hardirqs last disabled at (51623110): [] sysvec_apic_timer_interrupt+0xe/0x90 [30594.079150] ceph [30594.082201] softirqs last enabled at (51623106): [] __do_softirq+0x2be/0x371 [30594.091926] libceph [30594.093850] softirqs last disabled at (51623101): [] irq_exit_rcu+0xb0/0xd0 [30594.102534] dns_resolver fscache [30594.104718] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30594.113244] netfs veth [30594.116557] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30594.125499] nft_chain_nat [30594.127964] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [30594.135432] xt_MASQUERADE nf_nat [30594.138137] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [30594.143615] nf_conntrack nf_defrag_ipv6 [30594.146924] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30594.165664] nf_defrag_ipv4 nft_compat [30594.169579] [30594.169580] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30594.174797] nf_tables [30594.178540] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30594.180031] nfnetlink bridge [30594.187157] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30594.189526] stp [30594.196646] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30594.199609] llc xfs [30594.206735] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30594.208571] libcrc32c sunrpc [30594.215696] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30594.217881] intel_rapl_msr [30594.225005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30594.227969] intel_rapl_common sb_edac [30594.236046] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30594.238835] x86_pkg_temp_thermal [30594.244573] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30594.248317] coretemp kvm_intel [30594.255443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30594.258752] kvm irqbypass [30594.265876] Call Trace: [30594.265878] [30594.269014] i2c_i801 iTCO_wdt [30594.276141] ? watchdog_timer_fn+0x1b4/0x220 [30594.278843] iTCO_vendor_support i2c_smbus [30594.281287] ? __hrtimer_run_queues+0x1d3/0x360 [30594.283297] crct10dif_pclmul [30594.286351] ? hrtimer_interrupt+0x104/0x240 [30594.290613] joydev [30594.294702] ? __local_bh_enable+0x50/0x60 [30594.299244] crc32_pclmul ipmi_ssif [30594.302211] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30594.306472] ghash_clmulni_intel [30594.308569] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30594.312660] mei_me mei [30594.316144] [30594.321362] lpc_ich [30594.324587] [30594.324589] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30594.329542] sha512_ssse3 acpi_ipmi [30594.331991] ? queued_spin_lock_slowpath+0x2b/0x2e0 [30594.334086] mfd_core wmi [30594.336270] do_raw_spin_lock+0xb5/0xc0 [30594.338367] ipmi_si ipmi_devintf [30594.343673] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30594.347153] ipmi_msghandler ioatdma acpi_pad [30594.352028] try_get_cap_refs+0x159/0x880 [ceph] [30594.354643] acpi_power_meter ixgbe [30594.358479] __ceph_get_caps+0x4a1/0x760 [ceph] [30594.361784] igb mdio [30594.366573] ? __file_remove_privs+0x89/0x110 [30594.370919] crc32c_intel nvme [30594.375538] ? wake_bit_function+0x70/0x70 [30594.379022] ptp nvme_core [30594.383549] ? ceph_write_iter+0x260/0x770 [ceph] [30594.385816] i2c_algo_bit pps_core [30594.390170] ceph_write_iter+0x260/0x770 [ceph] [30594.393218] dca [30594.397314] ? lock_is_held_type+0xaa/0x120 [30594.400013] fuse [30594.404714] vfs_write+0x324/0x4f0 [30594.408107] [30594.408108] irq event stamp: 166145952 [30594.412638] ksys_pwrite64+0x65/0xa0 [30594.414469] hardirqs last enabled at (166145951): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30594.418649] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30594.420570] hardirqs last disabled at (166145952): [] sysvec_apic_timer_interrupt+0xe/0x90 [30594.423971] do_syscall_64+0x38/0x80 [30594.425460] softirqs last enabled at (166145950): [] __do_softirq+0x2be/0x371 [30594.429203] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30594.432774] softirqs last disabled at (166145945): [] irq_exit_rcu+0xb0/0xd0 [30594.443019] RIP: 0033:0x7f48f0eeb93b [30594.448149] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30594.457961] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30594.461537] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30594.461538] Workqueue: events delayed_work [ceph] [30594.470309] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30594.475354] [30594.475355] RIP: 0010:lock_acquire+0xf0/0x2b0 [30594.483952] ORIG_RAX: 0000000000000012 [30594.487530] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30594.496921] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30594.515663] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30594.523134] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30594.527832] [30594.527832] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30594.533050] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30594.534541] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30594.538892] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30594.542720] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30594.561459] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30594.568582] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30594.573805] [30594.580925] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30594.641594] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30594.649670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30594.655409] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30594.662544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30594.669664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30594.676789] Call Trace: [30594.679252] [30594.681263] ? watchdog_timer_fn+0x1b4/0x220 [30594.685536] ? __hrtimer_run_queues+0x1d3/0x360 [30594.690060] ? hrtimer_interrupt+0x104/0x240 [30594.694321] ? __local_bh_enable+0x50/0x60 [30594.698415] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30594.703640] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30594.708596] [30594.710694] [30594.712791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30594.718099] ? lock_acquire+0xf0/0x2b0 [30594.721850] ? igrab+0x43/0x50 [30594.724900] ? lock_release+0xc5/0x260 [30594.728644] _raw_spin_lock+0x30/0x40 [30594.732299] ? igrab+0x1d/0x50 [30594.735351] igrab+0x1d/0x50 [30594.738232] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30594.743397] delayed_work+0x1d8/0x280 [ceph] [30594.747679] ? process_scheduled_works+0x22a/0x510 [30594.752466] process_scheduled_works+0x2a4/0x510 [30594.757083] worker_thread+0x188/0x340 [30594.760834] ? rescuer_thread+0x310/0x310 [30594.764845] kthread+0xf5/0x130 [30594.767992] ? kthread_complete_and_exit+0x20/0x20 [30594.772775] ret_from_fork+0x31/0x40 [30594.776355] ? kthread_complete_and_exit+0x20/0x20 [30594.781137] ret_from_fork_asm+0x11/0x20 [30594.785061] [30621.995359] watchdog: BUG: soft lockup - CPU#0 stuck for 17404s! [dbench:78683] [30622.002660] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30622.007359] watchdog: BUG: soft lockup - CPU#7 stuck for 26401s! [kworker/7:4:78808] [30622.058101] irq event stamp: 51681350 [30622.058103] hardirqs last enabled at (51681349): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30622.065830] Modules linked in: [30622.069490] hardirqs last disabled at (51681350): [] sysvec_apic_timer_interrupt+0xe/0x90 [30622.079648] ceph [30622.082696] softirqs last enabled at (51681344): [] __do_softirq+0x2be/0x371 [30622.092421] libceph [30622.094344] softirqs last disabled at (51681339): [] irq_exit_rcu+0xb0/0xd0 [30622.103032] dns_resolver fscache [30622.105214] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30622.113724] netfs veth [30622.117050] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30622.125997] nft_chain_nat [30622.128441] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [30622.135912] xt_MASQUERADE nf_nat [30622.138614] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [30622.144094] nf_conntrack nf_defrag_ipv6 [30622.147403] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30622.166142] nf_defrag_ipv4 [30622.170060] [30622.170061] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30622.175275] nft_compat nf_tables [30622.178068] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30622.179557] nfnetlink bridge [30622.186683] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30622.189992] stp llc [30622.197116] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30622.200081] xfs libcrc32c [30622.207204] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30622.209390] sunrpc intel_rapl_msr [30622.216513] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30622.219217] intel_rapl_common sb_edac [30622.226342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30622.229755] x86_pkg_temp_thermal coretemp [30622.237834] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30622.241578] kvm_intel kvm [30622.247315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30622.251406] irqbypass i2c_i801 [30622.258529] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30622.261233] iTCO_wdt iTCO_vendor_support [30622.268359] Call Trace: [30622.271495] i2c_smbus crct10dif_pclmul [30622.278619] [30622.282624] joydev crc32_pclmul [30622.285069] ? watchdog_timer_fn+0x1b4/0x220 [30622.288897] ipmi_ssif ghash_clmulni_intel [30622.290911] ? __hrtimer_run_queues+0x1d3/0x360 [30622.294132] mei_me mei [30622.298400] ? hrtimer_interrupt+0x104/0x240 [30622.302488] lpc_ich sha512_ssse3 [30622.307013] ? __local_bh_enable+0x50/0x60 [30622.309457] acpi_ipmi mfd_core [30622.313723] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30622.317030] wmi ipmi_si ipmi_devintf [30622.321139] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30622.324276] ipmi_msghandler ioatdma [30622.329495] [30622.333151] acpi_pad acpi_power_meter [30622.338108] [30622.338109] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30622.341680] ixgbe igb [30622.343780] ? queued_spin_lock_slowpath+0x2b/0x2e0 [30622.347520] mdio crc32c_intel nvme [30622.349621] do_raw_spin_lock+0xb5/0xc0 [30622.354923] ptp nvme_core [30622.357282] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30622.362150] i2c_algo_bit pps_core [30622.365636] try_get_cap_refs+0x159/0x880 [ceph] [30622.369465] dca [30622.372174] __ceph_get_caps+0x4a1/0x760 [ceph] [30622.376952] fuse [30622.380354] ? __file_remove_privs+0x89/0x110 [30622.384963] irq event stamp: 166366820 [30622.386800] ? wake_bit_function+0x70/0x70 [30622.391322] hardirqs last enabled at (166366819): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30622.393250] ? ceph_write_iter+0x260/0x770 [ceph] [30622.397598] hardirqs last disabled at (166366820): [] sysvec_apic_timer_interrupt+0xe/0x90 [30622.401341] ceph_write_iter+0x260/0x770 [ceph] [30622.405433] softirqs last enabled at (166366818): [] __do_softirq+0x2be/0x371 [30622.415681] ? lock_is_held_type+0xaa/0x120 [30622.420373] softirqs last disabled at (166366813): [] irq_exit_rcu+0xb0/0xd0 [30622.430188] vfs_write+0x324/0x4f0 [30622.434710] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30622.443485] ksys_pwrite64+0x65/0xa0 [30622.447658] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30622.456256] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30622.459653] Workqueue: events delayed_work [ceph] [30622.469042] do_syscall_64+0x38/0x80 [30622.472609] [30622.472610] RIP: 0010:lock_acquire+0xf0/0x2b0 [30622.480081] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30622.485211] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30622.489908] RIP: 0033:0x7f48f0eeb93b [30622.493478] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30622.494971] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30622.499321] [30622.499321] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30622.504364] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30622.523102] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30622.526672] ORIG_RAX: 0000000000000012 [30622.531891] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30622.550627] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30622.552119] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30622.559242] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30622.564460] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30622.571583] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30622.575416] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30622.582538] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30622.589664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30622.596788] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30622.603912] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30622.611058] [30622.618177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30622.662674] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30622.669797] Call Trace: [30622.672243] [30622.674253] ? watchdog_timer_fn+0x1b4/0x220 [30622.678516] ? __hrtimer_run_queues+0x1d3/0x360 [30622.683049] ? hrtimer_interrupt+0x104/0x240 [30622.687311] ? __local_bh_enable+0x50/0x60 [30622.691406] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30622.696621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30622.701579] [30622.703674] [30622.705773] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30622.711082] ? lock_acquire+0xf0/0x2b0 [30622.714832] ? igrab+0x43/0x50 [30622.717882] ? lock_release+0xc5/0x260 [30622.721627] _raw_spin_lock+0x30/0x40 [30622.725282] ? igrab+0x1d/0x50 [30622.728332] igrab+0x1d/0x50 [30622.731212] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30622.736360] delayed_work+0x1d8/0x280 [ceph] [30622.740642] ? process_scheduled_works+0x22a/0x510 [30622.745434] process_scheduled_works+0x2a4/0x510 [30622.750050] worker_thread+0x188/0x340 [30622.753796] ? rescuer_thread+0x310/0x310 [30622.757801] kthread+0xf5/0x130 [30622.760939] ? kthread_complete_and_exit+0x20/0x20 [30622.765721] ret_from_fork+0x31/0x40 [30622.769302] ? kthread_complete_and_exit+0x20/0x20 [30622.774085] ret_from_fork_asm+0x11/0x20 [30622.778007] [30649.995870] watchdog: BUG: soft lockup - CPU#0 stuck for 17431s! [dbench:78683] [30650.003174] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30650.007869] watchdog: BUG: soft lockup - CPU#7 stuck for 26428s! [kworker/7:4:78808] [30650.058624] irq event stamp: 51739468 [30650.058625] hardirqs last enabled at (51739467): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30650.066355] Modules linked in: ceph [30650.070010] hardirqs last disabled at (51739468): [] sysvec_apic_timer_interrupt+0xe/0x90 [30650.080169] libceph [30650.083652] softirqs last enabled at (51739412): [] __do_softirq+0x2be/0x371 [30650.093376] dns_resolver [30650.095560] softirqs last disabled at (51739407): [] irq_exit_rcu+0xb0/0xd0 [30650.104263] fscache [30650.106879] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30650.115391] netfs veth [30650.117574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30650.126519] nft_chain_nat [30650.128980] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30650.136451] xt_MASQUERADE nf_nat [30650.139156] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30650.144633] nf_conntrack [30650.147944] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30650.166681] nf_defrag_ipv6 [30650.169298] [30650.169298] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30650.174515] nf_defrag_ipv4 nft_compat [30650.177306] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30650.178797] nf_tables nfnetlink [30650.185922] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30650.189664] bridge stp [30650.196790] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30650.200014] llc xfs [30650.207139] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30650.209581] libcrc32c sunrpc [30650.216706] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30650.218890] intel_rapl_msr intel_rapl_common [30650.226015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30650.228979] sb_edac x86_pkg_temp_thermal [30650.237055] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30650.241406] coretemp kvm_intel [30650.247143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30650.251148] kvm irqbypass [30650.258272] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30650.261409] i2c_i801 iTCO_wdt [30650.268546] Call Trace: [30650.268549] [30650.271267] iTCO_vendor_support [30650.278389] ? watchdog_timer_fn+0x1b4/0x220 [30650.281438] i2c_smbus [30650.283885] ? __hrtimer_run_queues+0x1d3/0x360 [30650.285893] crct10dif_pclmul joydev [30650.289121] ? hrtimer_interrupt+0x104/0x240 [30650.293382] crc32_pclmul [30650.295739] ? __local_bh_enable+0x50/0x60 [30650.300269] ipmi_ssif [30650.303844] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30650.308105] ghash_clmulni_intel [30650.310724] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30650.314815] mei_me [30650.317172] [30650.322388] mei [30650.325613] [30650.325614] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30650.330571] lpc_ich [30650.332672] ? queued_spin_lock_slowpath+0x16/0x2e0 [30650.334766] sha512_ssse3 acpi_ipmi [30650.336604] do_raw_spin_lock+0xb5/0xc0 [30650.338699] mfd_core wmi [30650.344006] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30650.346187] ipmi_si ipmi_devintf [30650.351062] try_get_cap_refs+0x159/0x880 [ceph] [30650.354557] ipmi_msghandler ioatdma [30650.358396] __ceph_get_caps+0x4a1/0x760 [ceph] [30650.361008] acpi_pad acpi_power_meter [30650.365798] ? __file_remove_privs+0x89/0x110 [30650.369104] ixgbe igb [30650.373715] ? wake_bit_function+0x70/0x70 [30650.377284] mdio [30650.381811] ? ceph_write_iter+0x260/0x770 [ceph] [30650.385554] crc32c_intel nvme [30650.389904] ceph_write_iter+0x260/0x770 [ceph] [30650.392269] ptp [30650.396365] ? lock_is_held_type+0xaa/0x120 [30650.398285] nvme_core [30650.402986] vfs_write+0x324/0x4f0 [30650.406031] i2c_algo_bit pps_core [30650.410566] ksys_pwrite64+0x65/0xa0 [30650.412395] dca fuse [30650.416573] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30650.418929] [30650.418930] irq event stamp: 166587618 [30650.422327] do_syscall_64+0x38/0x80 [30650.425723] hardirqs last enabled at (166587617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30650.429296] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30650.431566] hardirqs last disabled at (166587618): [] sysvec_apic_timer_interrupt+0xe/0x90 [30650.436697] RIP: 0033:0x7f48f0eeb93b [30650.438187] softirqs last enabled at (166587616): [] __do_softirq+0x2be/0x371 [30650.441933] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30650.445500] softirqs last disabled at (166587611): [] irq_exit_rcu+0xb0/0xd0 [30650.455746] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30650.460789] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30650.470601] ORIG_RAX: 0000000000000012 [30650.474172] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30650.482943] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30650.501680] Workqueue: events delayed_work [ceph] [30650.510276] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30650.515495] [30650.515496] RIP: 0010:do_raw_spin_lock+0x24/0xc0 [30650.524871] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30650.528704] Code: 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 <48> 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba [30650.536175] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30650.543298] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [30650.547996] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30650.555120] [30650.555121] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [30650.556615] [30650.561222] RDX: ffffffffffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [30650.624473] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [30650.631597] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [30650.638721] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30650.645845] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30650.653921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30650.659660] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30650.666783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30650.673907] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30650.681030] Call Trace: [30650.683474] [30650.685487] ? watchdog_timer_fn+0x1b4/0x220 [30650.689753] ? __hrtimer_run_queues+0x1d3/0x360 [30650.694287] ? hrtimer_interrupt+0x104/0x240 [30650.698556] ? __local_bh_enable+0x50/0x60 [30650.702648] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30650.707873] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30650.712831] [30650.714927] [30650.717024] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30650.722335] ? do_raw_spin_lock+0x24/0xc0 [30650.726342] igrab+0x1d/0x50 [30650.729220] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30650.734370] delayed_work+0x1d8/0x280 [ceph] [30650.738652] ? process_scheduled_works+0x22a/0x510 [30650.743441] process_scheduled_works+0x2a4/0x510 [30650.748057] worker_thread+0x188/0x340 [30650.751806] ? rescuer_thread+0x310/0x310 [30650.755808] kthread+0xf5/0x130 [30650.758947] ? kthread_complete_and_exit+0x20/0x20 [30650.763731] ret_from_fork+0x31/0x40 [30650.767311] ? kthread_complete_and_exit+0x20/0x20 [30650.772095] ret_from_fork_asm+0x11/0x20 [30650.776025] [30677.996381] watchdog: BUG: soft lockup - CPU#0 stuck for 17457s! [dbench:78683] [30678.003688] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30678.008381] watchdog: BUG: soft lockup - CPU#7 stuck for 26454s! [kworker/7:4:78808] [30678.059136] irq event stamp: 51797710 [30678.059138] hardirqs last enabled at (51797709): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30678.066867] Modules linked in: [30678.070525] hardirqs last disabled at (51797710): [] sysvec_apic_timer_interrupt+0xe/0x90 [30678.080681] ceph [30678.083732] softirqs last enabled at (51797674): [] __do_softirq+0x2be/0x371 [30678.093456] libceph [30678.095382] softirqs last disabled at (51797669): [] irq_exit_rcu+0xb0/0xd0 [30678.104065] dns_resolver fscache [30678.106248] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30678.114776] netfs veth [30678.118089] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30678.127033] nft_chain_nat [30678.129495] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30678.136963] xt_MASQUERADE nf_nat [30678.139667] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30678.145145] nf_conntrack nf_defrag_ipv6 [30678.148455] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30678.167195] nf_defrag_ipv4 [30678.171111] [30678.171111] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30678.176329] nft_compat nf_tables [30678.179119] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30678.180610] nfnetlink [30678.187736] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30678.191057] bridge stp [30678.198178] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30678.200537] llc xfs [30678.207661] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30678.210103] libcrc32c sunrpc [30678.217227] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30678.219413] intel_rapl_msr intel_rapl_common [30678.226535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30678.229501] sb_edac x86_pkg_temp_thermal [30678.237578] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30678.241929] coretemp kvm_intel [30678.247666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30678.251669] kvm irqbypass [30678.258795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30678.261930] i2c_i801 iTCO_wdt [30678.269060] Call Trace: [30678.269061] [30678.271779] iTCO_vendor_support [30678.278903] ? watchdog_timer_fn+0x1b4/0x220 [30678.281952] i2c_smbus crct10dif_pclmul [30678.284397] ? __hrtimer_run_queues+0x1d3/0x360 [30678.286406] joydev crc32_pclmul [30678.289635] ? hrtimer_interrupt+0x104/0x240 [30678.293895] ipmi_ssif ghash_clmulni_intel [30678.297724] ? __local_bh_enable+0x50/0x60 [30678.302249] mei_me mei [30678.305476] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30678.309738] lpc_ich sha512_ssse3 [30678.313828] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30678.317919] acpi_ipmi mfd_core [30678.320363] [30678.325582] wmi ipmi_si [30678.328892] [30678.328894] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30678.333847] ipmi_devintf [30678.336990] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30678.339082] ipmi_msghandler [30678.341615] do_raw_spin_lock+0xb5/0xc0 [30678.343712] ioatdma [30678.349016] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30678.351633] acpi_pad acpi_power_meter [30678.356505] try_get_cap_refs+0x159/0x880 [ceph] [30678.359382] ixgbe igb [30678.363216] __ceph_get_caps+0x4a1/0x760 [ceph] [30678.365396] mdio crc32c_intel [30678.370184] ? __file_remove_privs+0x89/0x110 [30678.373923] nvme [30678.378535] ? wake_bit_function+0x70/0x70 [30678.380892] ptp nvme_core [30678.385420] ? ceph_write_iter+0x260/0x770 [ceph] [30678.388468] i2c_algo_bit pps_core [30678.392818] ceph_write_iter+0x260/0x770 [ceph] [30678.394741] dca fuse [30678.398837] ? lock_is_held_type+0xaa/0x120 [30678.401536] [30678.401537] irq event stamp: 166808548 [30678.406236] vfs_write+0x324/0x4f0 [30678.409631] hardirqs last enabled at (166808547): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30678.414160] ksys_pwrite64+0x65/0xa0 [30678.416426] hardirqs last disabled at (166808548): [] sysvec_apic_timer_interrupt+0xe/0x90 [30678.420605] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30678.422093] softirqs last enabled at (166808546): [] __do_softirq+0x2be/0x371 [30678.425840] do_syscall_64+0x38/0x80 [30678.429236] softirqs last disabled at (166808541): [] irq_exit_rcu+0xb0/0xd0 [30678.439482] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30678.443065] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30678.452880] RIP: 0033:0x7f48f0eeb93b [30678.458010] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30678.458012] Workqueue: events delayed_work [ceph] [30678.466781] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30678.470352] [30678.478949] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30678.483994] RIP: 0010:lock_acquire+0xf0/0x2b0 [30678.493371] ORIG_RAX: 0000000000000012 [30678.496941] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30678.504412] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30678.509108] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30678.527846] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30678.529339] [30678.529339] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30678.534555] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30678.538905] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30678.542736] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30678.561473] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30678.568598] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30678.573816] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30678.580946] [30678.582431] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30678.641624] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30678.649702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30678.655440] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30678.662566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30678.669689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30678.676813] Call Trace: [30678.679258] [30678.681268] ? watchdog_timer_fn+0x1b4/0x220 [30678.685532] ? __hrtimer_run_queues+0x1d3/0x360 [30678.690070] ? hrtimer_interrupt+0x104/0x240 [30678.694337] ? __local_bh_enable+0x50/0x60 [30678.698430] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30678.703654] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30678.708612] [30678.710709] [30678.712805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30678.718114] ? lock_acquire+0xf0/0x2b0 [30678.721865] ? igrab+0x43/0x50 [30678.724914] ? lock_release+0xc5/0x260 [30678.728661] _raw_spin_lock+0x30/0x40 [30678.732324] ? igrab+0x1d/0x50 [30678.735374] igrab+0x1d/0x50 [30678.738252] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30678.743402] delayed_work+0x1d8/0x280 [ceph] [30678.747684] ? process_scheduled_works+0x22a/0x510 [30678.752474] process_scheduled_works+0x2a4/0x510 [30678.757090] worker_thread+0x188/0x340 [30678.760840] ? rescuer_thread+0x310/0x310 [30678.764851] kthread+0xf5/0x130 [30678.767988] ? kthread_complete_and_exit+0x20/0x20 [30678.772790] ret_from_fork+0x31/0x40 [30678.776368] ? kthread_complete_and_exit+0x20/0x20 [30678.781153] ret_from_fork_asm+0x11/0x20 [30678.785077] [30705.996895] watchdog: BUG: soft lockup - CPU#0 stuck for 17483s! [dbench:78683] [30706.004200] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30706.008894] watchdog: BUG: soft lockup - CPU#7 stuck for 26480s! [kworker/7:4:78808] [30706.059649] irq event stamp: 51855814 [30706.059650] hardirqs last enabled at (51855813): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30706.067382] Modules linked in: [30706.071039] hardirqs last disabled at (51855814): [] sysvec_apic_timer_interrupt+0xe/0x90 [30706.081194] ceph [30706.084246] softirqs last enabled at (51855706): [] __do_softirq+0x2be/0x371 [30706.093972] libceph [30706.095894] softirqs last disabled at (51855701): [] irq_exit_rcu+0xb0/0xd0 [30706.104578] dns_resolver [30706.106764] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30706.115290] fscache netfs [30706.117907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30706.126852] veth nft_chain_nat [30706.129568] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30706.137035] xt_MASQUERADE nf_nat [30706.140175] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30706.145651] nf_conntrack [30706.148961] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30706.167700] nf_defrag_ipv6 nf_defrag_ipv4 [30706.170318] [30706.175535] nft_compat nf_tables [30706.179624] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30706.181116] nfnetlink bridge [30706.184425] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30706.191549] stp llc [30706.194513] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30706.201638] xfs libcrc32c [30706.203822] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30706.210947] sunrpc intel_rapl_msr [30706.213650] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30706.220774] intel_rapl_common sb_edac [30706.224172] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30706.231296] x86_pkg_temp_thermal [30706.235041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30706.243117] coretemp kvm_intel [30706.246428] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30706.252166] kvm irqbypass [30706.255303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30706.262428] i2c_i801 iTCO_wdt [30706.265131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30706.272256] iTCO_vendor_support i2c_smbus [30706.275306] Call Trace: [30706.282430] crct10dif_pclmul joydev [30706.286524] [30706.288965] crc32_pclmul ipmi_ssif [30706.292538] ? watchdog_timer_fn+0x1b4/0x220 [30706.294547] ghash_clmulni_intel mei_me [30706.298032] ? __hrtimer_run_queues+0x1d3/0x360 [30706.302295] mei lpc_ich [30706.306130] ? hrtimer_interrupt+0x104/0x240 [30706.310648] sha512_ssse3 acpi_ipmi [30706.313180] ? __local_bh_enable+0x50/0x60 [30706.317446] mfd_core wmi [30706.320930] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30706.325018] ipmi_si [30706.327637] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30706.332855] ipmi_devintf ipmi_msghandler [30706.335038] [30706.339996] ioatdma acpi_pad [30706.343999] [30706.346098] acpi_power_meter ixgbe [30706.349062] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30706.351158] igb mdio [30706.354646] ? queued_spin_lock_slowpath+0x16/0x2e0 [30706.359946] crc32c_intel [30706.362220] do_raw_spin_lock+0xb5/0xc0 [30706.367088] nvme [30706.369708] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30706.373537] ptp nvme_core [30706.375464] try_get_cap_refs+0x159/0x880 [ceph] [30706.380243] i2c_algo_bit pps_core [30706.382954] __ceph_get_caps+0x4a1/0x760 [ceph] [30706.387558] dca fuse [30706.390978] ? __file_remove_privs+0x89/0x110 [30706.395498] [30706.395499] irq event stamp: 167029362 [30706.397769] ? wake_bit_function+0x70/0x70 [30706.402119] hardirqs last enabled at (167029361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30706.403612] ? ceph_write_iter+0x260/0x770 [ceph] [30706.407355] hardirqs last disabled at (167029362): [] sysvec_apic_timer_interrupt+0xe/0x90 [30706.411445] ceph_write_iter+0x260/0x770 [ceph] [30706.421690] softirqs last enabled at (167029360): [] __do_softirq+0x2be/0x371 [30706.426389] ? lock_is_held_type+0xaa/0x120 [30706.436198] softirqs last disabled at (167029355): [] irq_exit_rcu+0xb0/0xd0 [30706.440723] vfs_write+0x324/0x4f0 [30706.449493] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30706.453673] ksys_pwrite64+0x65/0xa0 [30706.462267] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30706.465665] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30706.475041] Workqueue: events delayed_work [ceph] [30706.478614] do_syscall_64+0x38/0x80 [30706.486085] [30706.486085] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [30706.491215] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30706.495913] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [30706.499482] RIP: 0033:0x7f48f0eeb93b [30706.500973] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [30706.505585] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30706.510629] [30706.510630] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [30706.529364] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30706.532936] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [30706.538153] ORIG_RAX: 0000000000000012 [30706.556892] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [30706.558382] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30706.565507] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [30706.570723] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30706.577848] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30706.581678] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30706.588804] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30706.595928] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30706.603050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30706.610175] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30706.617300] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30706.624429] [30706.632501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30706.668918] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30706.676044] Call Trace: [30706.678489] [30706.680499] ? watchdog_timer_fn+0x1b4/0x220 [30706.684764] ? __hrtimer_run_queues+0x1d3/0x360 [30706.689304] ? hrtimer_interrupt+0x104/0x240 [30706.693567] ? __local_bh_enable+0x50/0x60 [30706.697678] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30706.702904] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30706.707858] [30706.709958] [30706.712054] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30706.717361] ? do_raw_spin_lock+0x42/0xc0 [30706.721372] igrab+0x1d/0x50 [30706.724249] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30706.729401] delayed_work+0x1d8/0x280 [ceph] [30706.733680] ? process_scheduled_works+0x22a/0x510 [30706.738471] process_scheduled_works+0x2a4/0x510 [30706.743085] worker_thread+0x188/0x340 [30706.746836] ? rescuer_thread+0x310/0x310 [30706.750838] kthread+0xf5/0x130 [30706.753976] ? kthread_complete_and_exit+0x20/0x20 [30706.758761] ret_from_fork+0x31/0x40 [30706.762339] ? kthread_complete_and_exit+0x20/0x20 [30706.767123] ret_from_fork_asm+0x11/0x20 [30706.771046] [30733.997404] watchdog: BUG: soft lockup - CPU#0 stuck for 17509s! [dbench:78683] [30734.004706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30734.009404] watchdog: BUG: soft lockup - CPU#7 stuck for 26506s! [kworker/7:4:78808] [30734.060145] irq event stamp: 51913918 [30734.060146] hardirqs last enabled at (51913917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30734.067877] Modules linked in: [30734.071535] hardirqs last disabled at (51913918): [] sysvec_apic_timer_interrupt+0xe/0x90 [30734.081693] ceph [30734.084742] softirqs last enabled at (51913892): [] __do_softirq+0x2be/0x371 [30734.094466] libceph [30734.096392] softirqs last disabled at (51913887): [] irq_exit_rcu+0xb0/0xd0 [30734.105088] dns_resolver [30734.107275] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30734.115801] fscache netfs [30734.118421] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30734.127367] veth nft_chain_nat [30734.130078] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30734.137548] xt_MASQUERADE nf_nat [30734.140687] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30734.146163] nf_conntrack nf_defrag_ipv6 [30734.149475] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30734.168212] nf_defrag_ipv4 nft_compat [30734.172129] [30734.172130] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30734.177348] nf_tables nfnetlink [30734.181092] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30734.182581] bridge stp [30734.189706] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30734.192931] llc [30734.200054] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30734.202498] xfs libcrc32c [30734.209622] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30734.211460] sunrpc [30734.218583] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30734.221288] intel_rapl_msr intel_rapl_common [30734.228412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30734.230509] sb_edac x86_pkg_temp_thermal [30734.238587] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30734.242938] coretemp kvm_intel [30734.248677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30734.252679] kvm irqbypass [30734.259804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30734.262942] i2c_i801 iTCO_wdt [30734.270080] Call Trace: [30734.270081] [30734.272803] iTCO_vendor_support i2c_smbus [30734.279929] ? watchdog_timer_fn+0x1b4/0x220 [30734.282979] crct10dif_pclmul joydev [30734.285424] ? __hrtimer_run_queues+0x1d3/0x360 [30734.287435] crc32_pclmul ipmi_ssif [30734.291527] ? hrtimer_interrupt+0x104/0x240 [30734.295802] ghash_clmulni_intel [30734.299375] ? __local_bh_enable+0x50/0x60 [30734.303900] mei_me [30734.307387] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30734.311647] mei lpc_ich [30734.314872] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30734.318964] sha512_ssse3 acpi_ipmi [30734.321078] [30734.326294] mfd_core [30734.328826] [30734.333801] wmi [30734.337286] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30734.339384] ipmi_si [30734.341656] ? queued_spin_lock_slowpath+0x16/0x2e0 [30734.343750] ipmi_devintf ipmi_msghandler [30734.345591] do_raw_spin_lock+0xb5/0xc0 [30734.350891] ioatdma [30734.353084] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30734.357955] acpi_pad [30734.361963] try_get_cap_refs+0x159/0x880 [ceph] [30734.365805] acpi_power_meter [30734.367998] __ceph_get_caps+0x4a1/0x760 [ceph] [30734.372777] ixgbe [30734.375084] ? __file_remove_privs+0x89/0x110 [30734.379692] igb mdio [30734.382657] ? wake_bit_function+0x70/0x70 [30734.387180] crc32c_intel nvme [30734.389193] ? ceph_write_iter+0x260/0x770 [ceph] [30734.393541] ptp nvme_core [30734.395812] ceph_write_iter+0x260/0x770 [ceph] [30734.399904] i2c_algo_bit pps_core [30734.402957] ? lock_is_held_type+0xaa/0x120 [30734.407650] dca fuse [30734.410357] vfs_write+0x324/0x4f0 [30734.414879] [30734.414880] irq event stamp: 167250622 [30734.418281] ksys_pwrite64+0x65/0xa0 [30734.422454] hardirqs last enabled at (167250621): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30734.424725] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30734.428123] hardirqs last disabled at (167250622): [] sysvec_apic_timer_interrupt+0xe/0x90 [30734.429615] do_syscall_64+0x38/0x80 [30734.433358] softirqs last enabled at (167250620): [] __do_softirq+0x2be/0x371 [30734.436928] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30734.447171] softirqs last disabled at (167250615): [] irq_exit_rcu+0xb0/0xd0 [30734.452302] RIP: 0033:0x7f48f0eeb93b [30734.462114] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30734.465683] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30734.474454] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30734.479499] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30734.488097] Workqueue: events delayed_work [ceph] [30734.491667] ORIG_RAX: 0000000000000012 [30734.491668] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30734.501045] [30734.501046] RIP: 0010:lock_acquire+0xf0/0x2b0 [30734.519782] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30734.527271] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30734.532487] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30734.537186] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30734.541017] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30734.548141] [30734.548142] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30734.549632] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30734.553981] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30734.561110] [30734.579844] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30734.631490] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30734.638612] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30734.645737] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30734.653814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30734.659552] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30734.666676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30734.673811] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30734.680935] Call Trace: [30734.683378] [30734.685389] ? watchdog_timer_fn+0x1b4/0x220 [30734.689653] ? __hrtimer_run_queues+0x1d3/0x360 [30734.694181] ? hrtimer_interrupt+0x104/0x240 [30734.698449] ? __local_bh_enable+0x50/0x60 [30734.702543] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30734.707767] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30734.712724] [30734.714821] [30734.716919] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30734.722228] ? lock_acquire+0xf0/0x2b0 [30734.725977] ? igrab+0x43/0x50 [30734.729028] ? lock_release+0xc5/0x260 [30734.732773] _raw_spin_lock+0x30/0x40 [30734.736436] ? igrab+0x1d/0x50 [30734.739486] igrab+0x1d/0x50 [30734.742365] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30734.747515] delayed_work+0x1d8/0x280 [ceph] [30734.751813] ? process_scheduled_works+0x22a/0x510 [30734.756604] process_scheduled_works+0x2a4/0x510 [30734.761220] worker_thread+0x188/0x340 [30734.764969] ? rescuer_thread+0x310/0x310 [30734.768980] kthread+0xf5/0x130 [30734.772118] ? kthread_complete_and_exit+0x20/0x20 [30734.776903] ret_from_fork+0x31/0x40 [30734.780482] ? kthread_complete_and_exit+0x20/0x20 [30734.785264] ret_from_fork_asm+0x11/0x20 [30734.789189] [30761.997907] watchdog: BUG: soft lockup - CPU#0 stuck for 17535s! [dbench:78683] [30762.005211] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30762.009906] watchdog: BUG: soft lockup - CPU#7 stuck for 26532s! [kworker/7:4:78808] [30762.060660] irq event stamp: 51972080 [30762.060661] hardirqs last enabled at (51972079): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30762.068391] Modules linked in: [30762.072048] hardirqs last disabled at (51972080): [] sysvec_apic_timer_interrupt+0xe/0x90 [30762.082205] ceph [30762.085257] softirqs last enabled at (51972074): [] __do_softirq+0x2be/0x371 [30762.094980] libceph dns_resolver [30762.096903] softirqs last disabled at (51972069): [] irq_exit_rcu+0xb0/0xd0 [30762.105587] fscache [30762.108900] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30762.117409] netfs veth [30762.119597] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30762.128537] nft_chain_nat [30762.130983] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30762.138452] xt_MASQUERADE nf_nat [30762.141157] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30762.146634] nf_conntrack nf_defrag_ipv6 [30762.149944] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30762.168684] nf_defrag_ipv4 nft_compat [30762.172601] [30762.172602] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30762.177819] nf_tables nfnetlink [30762.181562] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30762.183071] bridge stp [30762.190193] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30762.193419] llc xfs [30762.200543] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30762.202987] libcrc32c sunrpc [30762.210111] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30762.212301] intel_rapl_msr intel_rapl_common [30762.219426] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30762.222390] sb_edac x86_pkg_temp_thermal [30762.229516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30762.233866] coretemp kvm_intel [30762.241944] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30762.245948] kvm irqbypass [30762.251684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30762.254822] i2c_i801 iTCO_wdt [30762.261947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30762.264650] iTCO_vendor_support i2c_smbus [30762.271775] Call Trace: [30762.274824] crct10dif_pclmul joydev [30762.281952] [30762.286040] crc32_pclmul ipmi_ssif [30762.288487] ? watchdog_timer_fn+0x1b4/0x220 [30762.292056] ghash_clmulni_intel mei_me [30762.294068] ? __hrtimer_run_queues+0x1d3/0x360 [30762.297550] mei lpc_ich [30762.301817] ? hrtimer_interrupt+0x104/0x240 [30762.305644] sha512_ssse3 acpi_ipmi [30762.310168] ? __local_bh_enable+0x50/0x60 [30762.312699] mfd_core wmi [30762.316967] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30762.320449] ipmi_si ipmi_devintf [30762.324540] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30762.327155] ipmi_msghandler ioatdma [30762.332374] [30762.335684] acpi_pad acpi_power_meter [30762.340641] [30762.344213] ixgbe igb [30762.346310] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30762.350055] mdio crc32c_intel nvme [30762.352154] ? queued_spin_lock_slowpath+0x16/0x2e0 [30762.354509] ptp nvme_core [30762.359815] do_raw_spin_lock+0xb5/0xc0 [30762.363306] i2c_algo_bit pps_core [30762.368178] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30762.370880] dca fuse [30762.374714] try_get_cap_refs+0x159/0x880 [ceph] [30762.378107] [30762.378108] irq event stamp: 167471510 [30762.382898] __ceph_get_caps+0x4a1/0x760 [ceph] [30762.385163] hardirqs last enabled at (167471509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30762.389778] ? __file_remove_privs+0x89/0x110 [30762.391263] hardirqs last disabled at (167471510): [] sysvec_apic_timer_interrupt+0xe/0x90 [30762.395010] ? wake_bit_function+0x70/0x70 [30762.399532] softirqs last enabled at (167471508): [] __do_softirq+0x2be/0x371 [30762.409781] ? ceph_write_iter+0x260/0x770 [ceph] [30762.414126] softirqs last disabled at (167471503): [] irq_exit_rcu+0xb0/0xd0 [30762.423939] ceph_write_iter+0x260/0x770 [ceph] [30762.428029] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30762.436805] ? lock_is_held_type+0xaa/0x120 [30762.441496] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30762.450096] vfs_write+0x324/0x4f0 [30762.454618] Workqueue: events delayed_work [ceph] [30762.464001] ksys_pwrite64+0x65/0xa0 [30762.468174] [30762.468175] RIP: 0010:lock_acquire+0xf0/0x2b0 [30762.475644] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30762.479041] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30762.483742] do_syscall_64+0x38/0x80 [30762.487310] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30762.488801] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30762.493151] [30762.493152] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30762.498284] RIP: 0033:0x7f48f0eeb93b [30762.517038] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30762.520610] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30762.525825] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30762.530868] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30762.532361] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30762.539483] ORIG_RAX: 0000000000000012 [30762.543056] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30762.550178] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30762.568916] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30762.576042] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30762.581258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30762.588381] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30762.592212] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30762.599338] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30762.606462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30762.614538] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30762.621664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30762.627405] [30762.634526] Call Trace: [30762.634527] [30762.676785] ? watchdog_timer_fn+0x1b4/0x220 [30762.681050] ? __hrtimer_run_queues+0x1d3/0x360 [30762.685592] ? hrtimer_interrupt+0x104/0x240 [30762.689853] ? __local_bh_enable+0x50/0x60 [30762.693948] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30762.699172] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30762.704129] [30762.706227] [30762.708324] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30762.713632] ? lock_acquire+0xf0/0x2b0 [30762.717381] ? igrab+0x43/0x50 [30762.720431] ? lock_release+0xc5/0x260 [30762.724176] _raw_spin_lock+0x30/0x40 [30762.727833] ? igrab+0x1d/0x50 [30762.730883] igrab+0x1d/0x50 [30762.733763] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30762.738912] delayed_work+0x1d8/0x280 [ceph] [30762.743191] ? process_scheduled_works+0x22a/0x510 [30762.747983] process_scheduled_works+0x2a4/0x510 [30762.752598] worker_thread+0x188/0x340 [30762.756348] ? rescuer_thread+0x310/0x310 [30762.760360] kthread+0xf5/0x130 [30762.763506] ? kthread_complete_and_exit+0x20/0x20 [30762.768291] ret_from_fork+0x31/0x40 [30762.771886] ? kthread_complete_and_exit+0x20/0x20 [30762.776671] ret_from_fork_asm+0x11/0x20 [30762.780602] [30763.478299] rcu: INFO: rcu_preempt self-detected stall on CPU [30763.484057] rcu: 0-....: (18402659 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9207395 [30763.494483] rcu: (t=18826534 jiffies g=1309609 q=551605 ncpus=8) [30763.500569] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30763.509510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30763.516980] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30763.522459] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30763.541196] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30763.546415] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30763.553539] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30763.560662] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30763.567786] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30763.574910] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30763.582036] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30763.590113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30763.595849] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30763.602973] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30763.610097] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30763.617223] Call Trace: [30763.619666] [30763.621676] ? rcu_dump_cpu_stacks+0xd1/0x180 [30763.626029] ? rcu_sched_clock_irq+0x633/0x1390 [30763.630560] ? lock_is_held_type+0xaa/0x120 [30763.634739] ? lock_is_held_type+0xaa/0x120 [30763.638916] ? tick_sched_do_timer+0xa0/0xa0 [30763.643188] ? rcu_read_lock_sched_held+0x56/0x90 [30763.647885] ? tick_sched_do_timer+0xa0/0xa0 [30763.652149] ? update_process_times+0x5f/0x90 [30763.656499] ? tick_sched_handle.isra.16+0x1e/0x50 [30763.661285] ? tick_sched_timer+0x6f/0x80 [30763.665289] ? __hrtimer_run_queues+0x1d3/0x360 [30763.669817] ? hrtimer_interrupt+0x104/0x240 [30763.674084] ? __local_bh_enable+0x50/0x60 [30763.678178] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30763.683402] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30763.688360] [30763.690456] [30763.692554] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30763.697864] ? queued_spin_lock_slowpath+0x16/0x2e0 [30763.702739] do_raw_spin_lock+0xb5/0xc0 [30763.706568] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30763.711372] try_get_cap_refs+0x159/0x880 [ceph] [30763.716004] __ceph_get_caps+0x4a1/0x760 [ceph] [30763.720545] ? __file_remove_privs+0x89/0x110 [30763.724898] ? wake_bit_function+0x70/0x70 [30763.728992] ? ceph_write_iter+0x260/0x770 [ceph] [30763.733705] ceph_write_iter+0x260/0x770 [ceph] [30763.738242] ? lock_is_held_type+0xaa/0x120 [30763.742426] vfs_write+0x324/0x4f0 [30763.745835] ksys_pwrite64+0x65/0xa0 [30763.749408] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30763.754541] do_syscall_64+0x38/0x80 [30763.758111] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30763.763153] RIP: 0033:0x7f48f0eeb93b [30763.766725] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30763.785463] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [30763.793020] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30763.800144] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30763.807267] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30763.814393] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30763.821516] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30763.828646] [30763.830834] Sending NMI from CPU 0 to CPUs 7: [30763.835186] NMI backtrace for cpu 7 [30763.835187] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30763.835189] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30763.835190] Workqueue: events delayed_work [ceph] [30763.835205] RIP: 0010:hlock_class+0xb/0x60 [30763.835208] Code: 07 00 48 c7 c7 38 50 12 82 48 89 c6 e8 2e 65 01 00 eb 9c e8 d7 ed a2 00 0f 1f 80 00 00 00 00 0f b7 47 20 66 25 ff 1f 0f b7 c0 <48> 0f a3 05 6d c0 e0 01 73 13 48 8d 14 40 48 8d 04 90 48 c1 e0 04 [30763.835210] RSP: 0018:ffffc90002117c58 EFLAGS: 00000002 [30763.835211] RAX: 0000000000000702 RBX: ffff88810debaa80 RCX: 0000000000000003 [30763.835212] RDX: 00000000000014df RSI: 0000000000000000 RDI: ffff88810debb528 [30763.835212] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [30763.835213] R10: 0000000000000000 R11: 0000000000000001 R12: b656a521d3820f2c [30763.835214] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [30763.835214] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30763.835215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30763.835216] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30763.835217] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30763.835218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30763.835218] Call Trace: [30763.835219] [30763.835220] ? nmi_cpu_backtrace+0xad/0x120 [30763.835224] ? nmi_cpu_backtrace_handler+0x11/0x20 [30763.835225] ? nmi_handle+0xd4/0x260 [30763.835229] ? default_do_nmi+0x49/0xf0 [30763.835231] ? exc_nmi+0x11e/0x150 [30763.835232] ? end_repeat_nmi+0x16/0x67 [30763.835236] ? hlock_class+0xb/0x60 [30763.835239] ? hlock_class+0xb/0x60 [30763.835241] ? hlock_class+0xb/0x60 [30763.835243] [30763.835244] [30763.835244] __lock_acquire+0xb15/0x1e20 [30763.835247] lock_acquire+0xcb/0x2b0 [30763.835248] ? igrab+0x1d/0x50 [30763.835250] ? igrab+0x43/0x50 [30763.835251] ? lock_release+0xc5/0x260 [30763.835253] _raw_spin_lock+0x30/0x40 [30763.835255] ? igrab+0x1d/0x50 [30763.835256] igrab+0x1d/0x50 [30763.835257] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30763.835271] delayed_work+0x1d8/0x280 [ceph] [30763.835284] ? process_scheduled_works+0x22a/0x510 [30763.835286] process_scheduled_works+0x2a4/0x510 [30763.835290] worker_thread+0x188/0x340 [30763.835292] ? rescuer_thread+0x310/0x310 [30763.835295] kthread+0xf5/0x130 [30763.835297] ? kthread_complete_and_exit+0x20/0x20 [30763.835299] ret_from_fork+0x31/0x40 [30763.835301] ? kthread_complete_and_exit+0x20/0x20 [30763.835303] ret_from_fork_asm+0x11/0x20 [30763.835309] [30789.998409] watchdog: BUG: soft lockup - CPU#0 stuck for 17561s! [dbench:78683] [30790.005706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30790.010409] watchdog: BUG: soft lockup - CPU#7 stuck for 26558s! [kworker/7:4:78808] [30790.061148] irq event stamp: 52029454 [30790.061150] hardirqs last enabled at (52029453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30790.068878] Modules linked in: [30790.072536] hardirqs last disabled at (52029454): [] sysvec_apic_timer_interrupt+0xe/0x90 [30790.082692] ceph [30790.085742] softirqs last enabled at (52029430): [] __do_softirq+0x2be/0x371 [30790.095468] libceph [30790.097393] softirqs last disabled at (52029425): [] irq_exit_rcu+0xb0/0xd0 [30790.106083] dns_resolver [30790.108269] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30790.116779] fscache [30790.119414] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30790.119415] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30790.128358] netfs [30790.130542] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30790.138012] veth nft_chain_nat [30790.143491] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30790.145500] xt_MASQUERADE nf_nat [30790.164238] [30790.164239] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30790.167377] nf_conntrack nf_defrag_ipv6 [30790.172594] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30790.175905] nf_defrag_ipv4 nft_compat [30790.177394] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30790.184519] nf_tables nfnetlink [30790.188436] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30790.195562] bridge stp [30790.199305] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30790.206429] llc xfs [30790.209654] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30790.216778] libcrc32c sunrpc [30790.219237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30790.226361] intel_rapl_msr [30790.228547] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30790.236624] intel_rapl_common sb_edac [30790.239589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30790.245325] x86_pkg_temp_thermal coretemp [30790.248117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30790.255240] kvm_intel kvm [30790.258985] Call Trace: [30790.266110] irqbypass i2c_i801 [30790.270200] [30790.277324] iTCO_wdt iTCO_vendor_support [30790.280029] ? watchdog_timer_fn+0x1b4/0x220 [30790.282473] i2c_smbus crct10dif_pclmul [30790.285609] ? __hrtimer_run_queues+0x1d3/0x360 [30790.287619] joydev [30790.291627] ? hrtimer_interrupt+0x104/0x240 [30790.295888] crc32_pclmul ipmi_ssif [30790.299719] ? __local_bh_enable+0x50/0x60 [30790.304241] ghash_clmulni_intel mei_me [30790.306342] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30790.310605] mei [30790.314090] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30790.318180] lpc_ich [30790.322009] [30790.327227] sha512_ssse3 acpi_ipmi [30790.329066] [30790.334039] mfd_core wmi [30790.336223] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30790.338321] ipmi_si ipmi_devintf [30790.341812] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30790.343901] ipmi_msghandler ioatdma [30790.346521] do_raw_spin_lock+0xb5/0xc0 [30790.351825] acpi_pad acpi_power_meter ixgbe [30790.355135] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30790.360005] igb mdio [30790.363578] try_get_cap_refs+0x159/0x880 [ceph] [30790.367407] crc32c_intel nvme [30790.371676] __ceph_get_caps+0x4a1/0x760 [ceph] [30790.376456] ptp nvme_core [30790.378731] ? __file_remove_privs+0x89/0x110 [30790.383351] i2c_algo_bit pps_core [30790.386405] ? wake_bit_function+0x70/0x70 [30790.390929] dca fuse [30790.393634] ? ceph_write_iter+0x260/0x770 [ceph] [30790.397982] [30790.401380] ceph_write_iter+0x260/0x770 [ceph] [30790.405471] irq event stamp: 167690408 [30790.407746] ? lock_is_held_type+0xaa/0x120 [30790.412439] hardirqs last enabled at (167690407): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30790.413934] vfs_write+0x324/0x4f0 [30790.418455] hardirqs last disabled at (167690408): [] sysvec_apic_timer_interrupt+0xe/0x90 [30790.422202] ksys_pwrite64+0x65/0xa0 [30790.426375] softirqs last enabled at (167690406): [] __do_softirq+0x2be/0x371 [30790.436620] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30790.440018] softirqs last disabled at (167690401): [] irq_exit_rcu+0xb0/0xd0 [30790.449829] do_syscall_64+0x38/0x80 [30790.453400] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30790.462170] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30790.467300] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30790.475899] RIP: 0033:0x7f48f0eeb93b [30790.479470] Workqueue: events delayed_work [ceph] [30790.488847] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30790.493891] [30790.493892] RIP: 0010:lock_acquire+0xf0/0x2b0 [30790.501361] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30790.504931] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30790.509630] ORIG_RAX: 0000000000000012 [30790.528367] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30790.529857] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30790.534209] [30790.534210] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30790.539426] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30790.558163] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30790.561994] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30790.567211] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30790.574334] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30790.575825] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30790.582950] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30790.590074] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30790.597203] [30790.604324] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30790.650204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30790.655943] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30790.663068] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30790.670209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30790.677333] Call Trace: [30790.679776] [30790.681788] ? watchdog_timer_fn+0x1b4/0x220 [30790.686070] ? __hrtimer_run_queues+0x1d3/0x360 [30790.690596] ? hrtimer_interrupt+0x104/0x240 [30790.694857] ? __local_bh_enable+0x50/0x60 [30790.698952] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30790.704174] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30790.709131] [30790.711228] [30790.713327] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30790.718634] ? lock_acquire+0xf0/0x2b0 [30790.722385] ? igrab+0x43/0x50 [30790.725434] ? lock_release+0xc5/0x260 [30790.729178] _raw_spin_lock+0x30/0x40 [30790.732834] ? igrab+0x1d/0x50 [30790.735885] igrab+0x1d/0x50 [30790.738763] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30790.743905] delayed_work+0x1d8/0x280 [ceph] [30790.748186] ? process_scheduled_works+0x22a/0x510 [30790.752975] process_scheduled_works+0x2a4/0x510 [30790.757592] worker_thread+0x188/0x340 [30790.761340] ? rescuer_thread+0x310/0x310 [30790.765344] kthread+0xf5/0x130 [30790.768481] ? kthread_complete_and_exit+0x20/0x20 [30790.773266] ret_from_fork+0x31/0x40 [30790.776838] ? kthread_complete_and_exit+0x20/0x20 [30790.781622] ret_from_fork_asm+0x11/0x20 [30790.785545] [30817.998910] watchdog: BUG: soft lockup - CPU#0 stuck for 17587s! [dbench:78683] [30818.006211] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30818.010909] watchdog: BUG: soft lockup - CPU#7 stuck for 26584s! [kworker/7:4:78808] [30818.061660] irq event stamp: 52087558 [30818.069391] Modules linked in: ceph libceph dns_resolver fscache netfs [30818.073048] hardirqs last enabled at (52087557): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30818.079582] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [30818.089742] hardirqs last disabled at (52087558): [] sysvec_apic_timer_interrupt+0xe/0x90 [30818.089745] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [30818.105549] softirqs last enabled at (52087472): [] __do_softirq+0x2be/0x371 [30818.113627] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [30818.122310] softirqs last disabled at (52087467): [] irq_exit_rcu+0xb0/0xd0 [30818.122314] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [30818.137687] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30818.144637] iTCO_wdt [30818.153585] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30818.155859] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul [30818.163333] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30818.170630] ipmi_ssif ghash_clmulni_intel mei_me mei [30818.176107] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30818.181151] lpc_ich sha512_ssse3 [30818.199888] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30818.199888] acpi_ipmi mfd_core [30818.203197] [30818.203198] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30818.208417] wmi ipmi_si [30818.211552] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30818.213043] ipmi_devintf [30818.220169] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30818.222699] ipmi_msghandler [30818.229823] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30818.232440] ioatdma acpi_pad [30818.239564] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30818.242459] acpi_power_meter ixgbe [30818.249595] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30818.252556] igb [30818.259680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30818.263165] mdio crc32c_intel [30818.271241] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30818.273079] nvme [30818.278816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30818.281866] ptp nvme_core [30818.288991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30818.290916] i2c_algo_bit [30818.298041] Call Trace: [30818.298043] [30818.300743] pps_core dca [30818.307870] ? watchdog_timer_fn+0x1b4/0x220 [30818.310486] fuse [30818.312930] ? __hrtimer_run_queues+0x1d3/0x360 [30818.314940] irq event stamp: 167911286 [30818.317560] ? hrtimer_interrupt+0x104/0x240 [30818.321822] hardirqs last enabled at (167911285): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30818.323745] ? __local_bh_enable+0x50/0x60 [30818.328269] hardirqs last disabled at (167911286): [] sysvec_apic_timer_interrupt+0xe/0x90 [30818.332017] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30818.336278] softirqs last enabled at (167911284): [] __do_softirq+0x2be/0x371 [30818.346523] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30818.350612] softirqs last disabled at (167911279): [] irq_exit_rcu+0xb0/0xd0 [30818.360424] [30818.365640] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30818.374412] [30818.379368] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30818.387967] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30818.390064] Workqueue: events delayed_work [ceph] [30818.399445] ? queued_spin_lock_slowpath+0x16/0x2e0 [30818.401539] [30818.409012] do_raw_spin_lock+0xb5/0xc0 [30818.414314] RIP: 0010:lock_acquire+0xf0/0x2b0 [30818.419013] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30818.423883] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30818.425376] try_get_cap_refs+0x159/0x880 [ceph] [30818.429203] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30818.433560] __ceph_get_caps+0x4a1/0x760 [ceph] [30818.438339] [30818.438339] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30818.457080] ? __file_remove_privs+0x89/0x110 [30818.461686] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30818.466905] ? wake_bit_function+0x70/0x70 [30818.471427] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30818.472922] ? ceph_write_iter+0x260/0x770 [ceph] [30818.480043] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30818.484394] ceph_write_iter+0x260/0x770 [ceph] [30818.491517] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30818.495612] ? lock_is_held_type+0xaa/0x120 [30818.502731] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30818.507431] vfs_write+0x324/0x4f0 [30818.514553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30818.519085] ksys_pwrite64+0x65/0xa0 [30818.526201] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30818.530379] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30818.538458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30818.541856] do_syscall_64+0x38/0x80 [30818.547594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30818.551164] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30818.558286] Call Trace: [30818.558289] [30818.563437] RIP: 0033:0x7f48f0eeb93b [30818.570560] ? watchdog_timer_fn+0x1b4/0x220 [30818.574130] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30818.581256] ? __hrtimer_run_queues+0x1d3/0x360 [30818.586313] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30818.588764] ? hrtimer_interrupt+0x104/0x240 [30818.590770] ORIG_RAX: 0000000000000012 [30818.594340] ? __local_bh_enable+0x50/0x60 [30818.598605] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30818.617345] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30818.621866] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30818.627084] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30818.631347] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30818.635179] [30818.639269] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30818.646393] [30818.651612] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30818.658736] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30818.663699] [30818.670820] ? lock_acquire+0xf0/0x2b0 [30818.700494] ? igrab+0x43/0x50 [30818.703544] ? lock_release+0xc5/0x260 [30818.707292] _raw_spin_lock+0x30/0x40 [30818.710972] ? igrab+0x1d/0x50 [30818.714021] igrab+0x1d/0x50 [30818.716900] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30818.722051] delayed_work+0x1d8/0x280 [ceph] [30818.726333] ? process_scheduled_works+0x22a/0x510 [30818.731121] process_scheduled_works+0x2a4/0x510 [30818.735737] worker_thread+0x188/0x340 [30818.739486] ? rescuer_thread+0x310/0x310 [30818.743489] kthread+0xf5/0x130 [30818.746635] ? kthread_complete_and_exit+0x20/0x20 [30818.751420] ret_from_fork+0x31/0x40 [30818.755001] ? kthread_complete_and_exit+0x20/0x20 [30818.759792] ret_from_fork_asm+0x11/0x20 [30818.763723] [30845.999409] watchdog: BUG: soft lockup - CPU#0 stuck for 17613s! [dbench:78683] [30846.006716] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30846.011409] watchdog: BUG: soft lockup - CPU#7 stuck for 26610s! [kworker/7:4:78808] [30846.062157] irq event stamp: 52145590 [30846.062159] hardirqs last enabled at (52145589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30846.069886] Modules linked in: [30846.073544] hardirqs last disabled at (52145590): [] sysvec_apic_timer_interrupt+0xe/0x90 [30846.083701] ceph [30846.086753] softirqs last enabled at (52145532): [] __do_softirq+0x2be/0x371 [30846.096476] libceph [30846.098402] softirqs last disabled at (52145527): [] irq_exit_rcu+0xb0/0xd0 [30846.107086] dns_resolver fscache [30846.109270] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30846.117780] netfs [30846.121109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30846.130052] veth nft_chain_nat [30846.132063] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30846.139535] xt_MASQUERADE [30846.142671] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30846.148149] nf_nat nf_conntrack [30846.150852] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30846.169590] nf_defrag_ipv6 nf_defrag_ipv4 [30846.172814] [30846.172814] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30846.178031] nft_compat nf_tables [30846.182122] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30846.183613] nfnetlink bridge [30846.190738] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30846.194047] stp llc [30846.201172] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30846.204137] xfs libcrc32c [30846.211261] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30846.213444] sunrpc intel_rapl_msr [30846.220567] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30846.223273] intel_rapl_common sb_edac [30846.230397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30846.233806] x86_pkg_temp_thermal [30846.241881] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30846.245625] coretemp kvm_intel [30846.251362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30846.254673] kvm irqbypass [30846.261806] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30846.264941] i2c_i801 iTCO_wdt [30846.272068] Call Trace: [30846.274771] iTCO_vendor_support i2c_smbus [30846.281894] [30846.284946] crct10dif_pclmul joydev [30846.287390] ? watchdog_timer_fn+0x1b4/0x220 [30846.291480] crc32_pclmul ipmi_ssif [30846.293492] ? __hrtimer_run_queues+0x1d3/0x360 [30846.297062] ghash_clmulni_intel mei_me [30846.301346] ? hrtimer_interrupt+0x104/0x240 [30846.304828] mei lpc_ich [30846.309351] ? __local_bh_enable+0x50/0x60 [30846.313182] sha512_ssse3 acpi_ipmi [30846.317448] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30846.319977] mfd_core wmi [30846.324086] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30846.327570] ipmi_si ipmi_devintf [30846.332786] [30846.335422] ipmi_msghandler ioatdma [30846.340378] [30846.343690] acpi_pad acpi_power_meter [30846.345805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30846.349376] ixgbe igb [30846.351476] ? queued_spin_lock_slowpath+0x18/0x2e0 [30846.355216] mdio crc32c_intel [30846.360524] do_raw_spin_lock+0xb5/0xc0 [30846.362877] nvme ptp [30846.367751] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30846.370809] nvme_core i2c_algo_bit [30846.374642] try_get_cap_refs+0x159/0x880 [ceph] [30846.376909] pps_core dca [30846.381698] __ceph_get_caps+0x4a1/0x760 [ceph] [30846.385178] fuse [30846.389811] ? __file_remove_privs+0x89/0x110 [30846.392424] irq event stamp: 168132270 [30846.396948] ? wake_bit_function+0x70/0x70 [30846.398872] hardirqs last enabled at (168132269): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30846.403224] ? ceph_write_iter+0x260/0x770 [ceph] [30846.406965] hardirqs last disabled at (168132270): [] sysvec_apic_timer_interrupt+0xe/0x90 [30846.411058] ceph_write_iter+0x260/0x770 [ceph] [30846.421301] softirqs last enabled at (168132268): [] __do_softirq+0x2be/0x371 [30846.426001] ? lock_is_held_type+0xaa/0x120 [30846.435810] softirqs last disabled at (168132263): [] irq_exit_rcu+0xb0/0xd0 [30846.440336] vfs_write+0x324/0x4f0 [30846.449104] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30846.453285] ksys_pwrite64+0x65/0xa0 [30846.461879] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30846.465275] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30846.474653] Workqueue: events delayed_work [ceph] [30846.478226] do_syscall_64+0x38/0x80 [30846.485695] [30846.490826] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30846.495523] RIP: 0010:lock_acquire+0xf0/0x2b0 [30846.499094] RIP: 0033:0x7f48f0eeb93b [30846.500584] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30846.505628] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30846.509979] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30846.513550] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30846.532287] [30846.532288] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30846.551026] ORIG_RAX: 0000000000000012 [30846.556243] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30846.561462] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30846.562952] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30846.570093] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30846.573923] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30846.581049] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30846.588171] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30846.595296] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30846.602419] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30846.609543] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30846.616669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30846.623814] [30846.630935] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30846.661183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30846.668305] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30846.675431] Call Trace: [30846.677874] [30846.679885] ? watchdog_timer_fn+0x1b4/0x220 [30846.684149] ? __hrtimer_run_queues+0x1d3/0x360 [30846.688675] ? hrtimer_interrupt+0x104/0x240 [30846.692937] ? __local_bh_enable+0x50/0x60 [30846.697030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30846.702254] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30846.707211] [30846.709308] [30846.711405] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30846.716713] ? lock_acquire+0xf0/0x2b0 [30846.720463] ? igrab+0x43/0x50 [30846.723515] ? lock_release+0xc5/0x260 [30846.727260] _raw_spin_lock+0x30/0x40 [30846.730924] ? igrab+0x1d/0x50 [30846.733975] igrab+0x1d/0x50 [30846.736853] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30846.742003] delayed_work+0x1d8/0x280 [ceph] [30846.746283] ? process_scheduled_works+0x22a/0x510 [30846.751075] process_scheduled_works+0x2a4/0x510 [30846.755690] worker_thread+0x188/0x340 [30846.759439] ? rescuer_thread+0x310/0x310 [30846.763441] kthread+0xf5/0x130 [30846.766580] ? kthread_complete_and_exit+0x20/0x20 [30846.771365] ret_from_fork+0x31/0x40 [30846.774942] ? kthread_complete_and_exit+0x20/0x20 [30846.779728] ret_from_fork_asm+0x11/0x20 [30846.783660] [30873.999908] watchdog: BUG: soft lockup - CPU#0 stuck for 17639s! [dbench:78683] [30874.007212] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30874.011907] watchdog: BUG: soft lockup - CPU#7 stuck for 26636s! [kworker/7:4:78808] [30874.062660] irq event stamp: 52203654 [30874.070392] Modules linked in: ceph libceph dns_resolver fscache netfs [30874.074048] hardirqs last enabled at (52203653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30874.080591] veth nft_chain_nat xt_MASQUERADE nf_nat [30874.090750] hardirqs last disabled at (52203654): [] sysvec_apic_timer_interrupt+0xe/0x90 [30874.095708] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [30874.105432] softirqs last enabled at (52203618): [] __do_softirq+0x2be/0x371 [30874.113951] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [30874.122634] softirqs last disabled at (52203613): [] irq_exit_rcu+0xb0/0xd0 [30874.129498] sb_edac x86_pkg_temp_thermal coretemp [30874.138012] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30874.138012] kvm_intel [30874.138014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30874.142794] kvm [30874.151739] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30874.154097] irqbypass [30874.161584] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30874.163422] i2c_i801 [30874.168900] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30874.171258] iTCO_wdt iTCO_vendor_support [30874.189994] [30874.189995] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30874.192264] i2c_smbus crct10dif_pclmul [30874.197483] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30874.201487] joydev [30874.202976] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30874.210100] crc32_pclmul ipmi_ssif [30874.213932] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30874.221056] ghash_clmulni_intel mei_me [30874.223154] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30874.230278] mei lpc_ich [30874.233778] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30874.240903] sha512_ssse3 acpi_ipmi [30874.244735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30874.251857] mfd_core wmi [30874.254387] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30874.262465] ipmi_si ipmi_devintf [30874.265951] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30874.271688] ipmi_msghandler ioatdma acpi_pad [30874.274305] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30874.281429] acpi_power_meter [30874.284739] Call Trace: [30874.284742] [30874.291863] ixgbe igb [30874.296218] ? watchdog_timer_fn+0x1b4/0x220 [30874.303340] mdio crc32c_intel [30874.306305] ? __hrtimer_run_queues+0x1d3/0x360 [30874.308747] nvme [30874.310760] ? hrtimer_interrupt+0x104/0x240 [30874.313114] ptp nvme_core [30874.317379] ? __local_bh_enable+0x50/0x60 [30874.320431] i2c_algo_bit [30874.324957] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30874.326877] pps_core [30874.331142] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30874.333848] dca [30874.337937] [30874.340553] fuse [30874.345771] [30874.348043] irq event stamp: 168353204 [30874.353000] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30874.354838] hardirqs last enabled at (168353203): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30874.356939] ? queued_spin_lock_slowpath+0x18/0x2e0 [30874.358858] hardirqs last disabled at (168353204): [] sysvec_apic_timer_interrupt+0xe/0x90 [30874.360957] do_raw_spin_lock+0xb5/0xc0 [30874.364699] softirqs last enabled at (168353202): [] __do_softirq+0x2be/0x371 [30874.370007] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30874.380249] softirqs last disabled at (168353197): [] irq_exit_rcu+0xb0/0xd0 [30874.385122] try_get_cap_refs+0x159/0x880 [ceph] [30874.394930] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30874.398767] __ceph_get_caps+0x4a1/0x760 [ceph] [30874.407531] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30874.412320] ? __file_remove_privs+0x89/0x110 [30874.420913] Workqueue: events delayed_work [ceph] [30874.425524] ? wake_bit_function+0x70/0x70 [30874.434902] [30874.439428] ? ceph_write_iter+0x260/0x770 [ceph] [30874.446896] RIP: 0010:lock_acquire+0xf0/0x2b0 [30874.451247] ceph_write_iter+0x260/0x770 [ceph] [30874.455945] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30874.460039] ? lock_is_held_type+0xaa/0x120 [30874.461525] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30874.466225] vfs_write+0x324/0x4f0 [30874.470574] [30874.470574] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30874.475121] ksys_pwrite64+0x65/0xa0 [30874.493852] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30874.498030] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30874.503249] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30874.506646] do_syscall_64+0x38/0x80 [30874.508135] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30874.515260] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30874.518830] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30874.525957] RIP: 0033:0x7f48f0eeb93b [30874.531086] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30874.538210] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30874.541780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30874.548903] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30874.553950] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30874.561074] ORIG_RAX: 0000000000000012 [30874.564643] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30874.572721] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30874.591459] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30874.597195] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30874.602412] Call Trace: [30874.602414] [30874.609537] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30874.613369] ? watchdog_timer_fn+0x1b4/0x220 [30874.620492] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30874.627617] ? __hrtimer_run_queues+0x1d3/0x360 [30874.634741] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30874.641869] ? hrtimer_interrupt+0x104/0x240 [30874.644315] [30874.646320] ? __local_bh_enable+0x50/0x60 [30874.687023] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30874.692246] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30874.697204] [30874.699311] [30874.701408] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30874.706724] ? lock_acquire+0xf0/0x2b0 [30874.710475] ? igrab+0x43/0x50 [30874.713523] ? lock_release+0xc5/0x260 [30874.717268] _raw_spin_lock+0x30/0x40 [30874.720934] ? igrab+0x1d/0x50 [30874.723984] igrab+0x1d/0x50 [30874.726862] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30874.732013] delayed_work+0x1d8/0x280 [ceph] [30874.736313] ? process_scheduled_works+0x22a/0x510 [30874.741101] process_scheduled_works+0x2a4/0x510 [30874.745716] worker_thread+0x188/0x340 [30874.749465] ? rescuer_thread+0x310/0x310 [30874.753469] kthread+0xf5/0x130 [30874.756617] ? kthread_complete_and_exit+0x20/0x20 [30874.761409] ret_from_fork+0x31/0x40 [30874.764988] ? kthread_complete_and_exit+0x20/0x20 [30874.769771] ret_from_fork_asm+0x11/0x20 [30874.773694] [30902.000403] watchdog: BUG: soft lockup - CPU#0 stuck for 17665s! [dbench:78683] [30902.007708] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30902.012403] watchdog: BUG: soft lockup - CPU#7 stuck for 26662s! [kworker/7:4:78808] [30902.063149] irq event stamp: 52261700 [30902.063150] hardirqs last enabled at (52261699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30902.070880] Modules linked in: [30902.074535] hardirqs last disabled at (52261700): [] sysvec_apic_timer_interrupt+0xe/0x90 [30902.084693] ceph [30902.087744] softirqs last enabled at (52261690): [] __do_softirq+0x2be/0x371 [30902.097470] libceph [30902.099392] softirqs last disabled at (52261685): [] irq_exit_rcu+0xb0/0xd0 [30902.108079] dns_resolver fscache [30902.110261] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30902.118773] netfs veth [30902.122099] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30902.122100] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30902.131043] nft_chain_nat xt_MASQUERADE [30902.133488] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30902.140959] nf_nat nf_conntrack [30902.146438] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30902.150354] nf_defrag_ipv6 nf_defrag_ipv4 [30902.169092] [30902.169093] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30902.172317] nft_compat nf_tables [30902.177534] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30902.181625] nfnetlink bridge [30902.183115] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30902.190238] stp llc [30902.193549] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30902.200673] xfs libcrc32c [30902.203638] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30902.210762] sunrpc intel_rapl_msr [30902.212946] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30902.220078] intel_rapl_common [30902.222800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30902.229925] sb_edac x86_pkg_temp_thermal [30902.233322] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30902.241400] coretemp [30902.244449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30902.250187] kvm_intel kvm [30902.254191] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30902.261317] irqbypass i2c_i801 [30902.263585] Call Trace: [30902.263587] [30902.270710] iTCO_wdt iTCO_vendor_support [30902.273415] ? watchdog_timer_fn+0x1b4/0x220 [30902.280540] i2c_smbus [30902.283677] ? __hrtimer_run_queues+0x1d3/0x360 [30902.286120] crct10dif_pclmul [30902.288135] ? hrtimer_interrupt+0x104/0x240 [30902.292135] joydev crc32_pclmul [30902.296398] ? __local_bh_enable+0x50/0x60 [30902.298755] ipmi_ssif ghash_clmulni_intel [30902.303282] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30902.306244] mei_me mei [30902.310509] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30902.313731] lpc_ich [30902.317822] [30902.321914] sha512_ssse3 acpi_ipmi [30902.327131] [30902.327133] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30902.329575] mfd_core wmi [30902.334536] ? queued_spin_lock_slowpath+0x16/0x2e0 [30902.336718] ipmi_si ipmi_devintf [30902.338816] do_raw_spin_lock+0xb5/0xc0 [30902.342298] ipmi_msghandler ioatdma [30902.344397] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30902.349699] acpi_pad acpi_power_meter [30902.352318] try_get_cap_refs+0x159/0x880 [ceph] [30902.357188] ixgbe igb [30902.360503] __ceph_get_caps+0x4a1/0x760 [ceph] [30902.364329] mdio crc32c_intel [30902.367905] ? __file_remove_privs+0x89/0x110 [30902.372684] nvme ptp [30902.376430] ? wake_bit_function+0x70/0x70 [30902.381038] nvme_core i2c_algo_bit [30902.383399] ? ceph_write_iter+0x260/0x770 [ceph] [30902.387921] pps_core dca [30902.390970] ceph_write_iter+0x260/0x770 [ceph] [30902.395322] fuse [30902.397595] ? lock_is_held_type+0xaa/0x120 [30902.401684] irq event stamp: 168574138 [30902.405169] vfs_write+0x324/0x4f0 [30902.409865] hardirqs last enabled at (168574137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30902.412487] ksys_pwrite64+0x65/0xa0 [30902.417006] hardirqs last disabled at (168574138): [] sysvec_apic_timer_interrupt+0xe/0x90 [30902.418932] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30902.423108] softirqs last enabled at (168574136): [] __do_softirq+0x2be/0x371 [30902.426854] do_syscall_64+0x38/0x80 [30902.430249] softirqs last disabled at (168574131): [] irq_exit_rcu+0xb0/0xd0 [30902.440495] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30902.444063] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30902.453874] RIP: 0033:0x7f48f0eeb93b [30902.459006] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30902.467778] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30902.471365] Workqueue: events delayed_work [ceph] [30902.479963] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30902.485006] [30902.485007] RIP: 0010:lock_acquire+0x9/0x2b0 [30902.494383] ORIG_RAX: 0000000000000012 [30902.497955] Code: 01 75 0e 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca <41> 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 [30902.505424] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30902.524162] RSP: 0018:ffffc90002117d78 EFLAGS: 00000246 [30902.528859] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30902.534093] [30902.534093] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [30902.535587] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30902.539850] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [30902.543680] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30902.562419] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [30902.569542] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30902.574759] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [30902.581891] [30902.583373] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30902.642553] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30902.650630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30902.656366] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30902.663490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30902.670617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30902.677740] Call Trace: [30902.680183] [30902.682194] ? watchdog_timer_fn+0x1b4/0x220 [30902.686459] ? __hrtimer_run_queues+0x1d3/0x360 [30902.690985] ? hrtimer_interrupt+0x104/0x240 [30902.695247] ? __local_bh_enable+0x50/0x60 [30902.699341] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30902.704565] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30902.709520] [30902.711617] [30902.713717] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30902.719034] ? lock_acquire+0x9/0x2b0 [30902.722695] _raw_spin_lock+0x30/0x40 [30902.726352] ? igrab+0x1d/0x50 [30902.729402] igrab+0x1d/0x50 [30902.732281] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30902.737431] delayed_work+0x1d8/0x280 [ceph] [30902.741711] ? process_scheduled_works+0x22a/0x510 [30902.746494] process_scheduled_works+0x2a4/0x510 [30902.751108] worker_thread+0x188/0x340 [30902.754859] ? rescuer_thread+0x310/0x310 [30902.758870] kthread+0xf5/0x130 [30902.762007] ? kthread_complete_and_exit+0x20/0x20 [30902.766810] ret_from_fork+0x31/0x40 [30902.770379] ? kthread_complete_and_exit+0x20/0x20 [30902.775164] ret_from_fork_asm+0x11/0x20 [30902.779094] [30930.000899] watchdog: BUG: soft lockup - CPU#0 stuck for 17691s! [dbench:78683] [30930.008204] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30930.012899] watchdog: BUG: soft lockup - CPU#7 stuck for 26688s! [kworker/7:4:78808] [30930.063646] irq event stamp: 52319868 [30930.063647] hardirqs last enabled at (52319867): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30930.071375] Modules linked in: [30930.075033] hardirqs last disabled at (52319868): [] sysvec_apic_timer_interrupt+0xe/0x90 [30930.085192] ceph [30930.088242] softirqs last enabled at (52319862): [] __do_softirq+0x2be/0x371 [30930.097966] libceph dns_resolver [30930.099888] softirqs last disabled at (52319857): [] irq_exit_rcu+0xb0/0xd0 [30930.108583] fscache [30930.111893] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30930.120405] netfs [30930.122589] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30930.131532] veth [30930.133542] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [30930.141013] nft_chain_nat [30930.142937] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [30930.148415] xt_MASQUERADE nf_nat [30930.151118] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30930.169856] nf_conntrack [30930.173168] [30930.173169] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30930.178383] nf_defrag_ipv6 nf_defrag_ipv4 [30930.181003] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30930.182494] nft_compat [30930.189617] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30930.193709] nf_tables [30930.200832] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30930.203294] nfnetlink [30930.210417] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30930.212775] bridge stp [30930.219899] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30930.222256] llc xfs [30930.229381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30930.231825] libcrc32c [30930.239901] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30930.242085] sunrpc intel_rapl_msr [30930.247823] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30930.250180] intel_rapl_common [30930.257305] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30930.260703] sb_edac x86_pkg_temp_thermal [30930.267827] Call Trace: [30930.267829] [30930.270876] coretemp kvm_intel [30930.278002] ? watchdog_timer_fn+0x1b4/0x220 [30930.282005] kvm [30930.284451] ? __hrtimer_run_queues+0x1d3/0x360 [30930.286460] irqbypass i2c_i801 [30930.289600] ? hrtimer_interrupt+0x104/0x240 [30930.293863] iTCO_wdt [30930.295698] ? __local_bh_enable+0x50/0x60 [30930.300224] iTCO_vendor_support [30930.303363] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30930.307625] i2c_smbus crct10dif_pclmul [30930.309896] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30930.313987] joydev [30930.317211] [30930.322428] crc32_pclmul ipmi_ssif [30930.326258] [30930.331217] ghash_clmulni_intel [30930.333312] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30930.335411] mei_me [30930.338899] ? queued_spin_lock_slowpath+0x18/0x2e0 [30930.340992] mei lpc_ich [30930.344217] do_raw_spin_lock+0xb5/0xc0 [30930.349519] sha512_ssse3 [30930.351620] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30930.356487] acpi_ipmi [30930.359023] try_get_cap_refs+0x159/0x880 [ceph] [30930.362850] mfd_core [30930.365473] __ceph_get_caps+0x4a1/0x760 [ceph] [30930.370250] wmi ipmi_si [30930.372613] ? __file_remove_privs+0x89/0x110 [30930.377219] ipmi_devintf [30930.379489] ? wake_bit_function+0x70/0x70 [30930.384013] ipmi_msghandler [30930.386548] ? ceph_write_iter+0x260/0x770 [ceph] [30930.390896] ioatdma [30930.393512] ceph_write_iter+0x260/0x770 [ceph] [30930.397603] acpi_pad [30930.400485] ? lock_is_held_type+0xaa/0x120 [30930.405178] acpi_power_meter [30930.407364] vfs_write+0x324/0x4f0 [30930.411886] ixgbe igb [30930.414163] ksys_pwrite64+0x65/0xa0 [30930.418333] mdio [30930.421301] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30930.424697] crc32c_intel nvme [30930.427054] do_syscall_64+0x38/0x80 [30930.430624] ptp [30930.432550] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30930.437680] nvme_core i2c_algo_bit [30930.440730] RIP: 0033:0x7f48f0eeb93b [30930.444300] pps_core [30930.446140] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30930.451182] dca [30930.454666] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30930.458238] fuse [30930.460506] ORIG_RAX: 0000000000000012 [30930.479244] irq event stamp: 168795046 [30930.481082] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30930.486300] hardirqs last enabled at (168795045): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30930.488224] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30930.492054] hardirqs last disabled at (168795046): [] sysvec_apic_timer_interrupt+0xe/0x90 [30930.495797] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30930.502923] softirqs last enabled at (168795044): [] __do_softirq+0x2be/0x371 [30930.513166] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30930.520302] softirqs last disabled at (168795039): [] irq_exit_rcu+0xb0/0xd0 [30930.530110] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30930.537235] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30930.546011] [30930.553130] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30930.587884] Workqueue: events delayed_work [ceph] [30930.592599] RIP: 0010:lock_release+0x77/0x260 [30930.596950] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [30930.615688] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [30930.620903] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [30930.628029] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [30930.635153] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [30930.642277] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [30930.649418] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [30930.656541] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30930.664619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30930.670357] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30930.677482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30930.684605] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30930.691729] Call Trace: [30930.694174] [30930.696185] ? watchdog_timer_fn+0x1b4/0x220 [30930.700449] ? __hrtimer_run_queues+0x1d3/0x360 [30930.704975] ? hrtimer_interrupt+0x104/0x240 [30930.709237] ? __local_bh_enable+0x50/0x60 [30930.713330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30930.718554] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30930.723511] [30930.725607] [30930.727707] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30930.733010] ? igrab+0x43/0x50 [30930.736065] ? lock_release+0x77/0x260 [30930.739817] _raw_spin_unlock+0x1b/0x40 [30930.743654] igrab+0x43/0x50 [30930.746532] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30930.751682] delayed_work+0x1d8/0x280 [ceph] [30930.755962] ? process_scheduled_works+0x22a/0x510 [30930.760753] process_scheduled_works+0x2a4/0x510 [30930.765368] worker_thread+0x188/0x340 [30930.769117] ? rescuer_thread+0x310/0x310 [30930.773120] kthread+0xf5/0x130 [30930.776259] ? kthread_complete_and_exit+0x20/0x20 [30930.781042] ret_from_fork+0x31/0x40 [30930.784621] ? kthread_complete_and_exit+0x20/0x20 [30930.789406] ret_from_fork_asm+0x11/0x20 [30930.793338] [30958.001393] watchdog: BUG: soft lockup - CPU#0 stuck for 17717s! [dbench:78683] [30958.008699] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30958.013394] watchdog: BUG: soft lockup - CPU#7 stuck for 26714s! [kworker/7:4:78808] [30958.064140] irq event stamp: 52378116 [30958.064141] hardirqs last enabled at (52378115): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30958.071872] Modules linked in: [30958.075529] hardirqs last disabled at (52378116): [] sysvec_apic_timer_interrupt+0xe/0x90 [30958.085686] ceph [30958.088738] softirqs last enabled at (52378058): [] __do_softirq+0x2be/0x371 [30958.098461] libceph [30958.100384] softirqs last disabled at (52378053): [] irq_exit_rcu+0xb0/0xd0 [30958.109069] dns_resolver fscache [30958.111254] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30958.119765] netfs veth [30958.123091] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30958.132035] nft_chain_nat [30958.134481] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [30958.141951] xt_MASQUERADE [30958.144655] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [30958.150131] nf_nat nf_conntrack [30958.152835] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30958.171573] nf_defrag_ipv6 nf_defrag_ipv4 [30958.174798] [30958.180014] nft_compat nf_tables [30958.184108] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30958.185598] nfnetlink [30958.188907] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30958.196032] bridge stp [30958.198389] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30958.205514] llc xfs [30958.207956] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30958.215082] libcrc32c sunrpc [30958.217266] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30958.224389] intel_rapl_msr intel_rapl_common [30958.227354] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30958.234479] sb_edac x86_pkg_temp_thermal [30958.238829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30958.246906] coretemp kvm_intel [30958.250910] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30958.256649] kvm irqbypass [30958.259789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30958.266909] i2c_i801 [30958.269613] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30958.276737] iTCO_wdt iTCO_vendor_support [30958.279008] Call Trace: [30958.279009] [30958.286132] i2c_smbus crct10dif_pclmul [30958.290137] ? watchdog_timer_fn+0x1b4/0x220 [30958.292580] joydev crc32_pclmul [30958.294592] ? __hrtimer_run_queues+0x1d3/0x360 [30958.298422] ipmi_ssif ghash_clmulni_intel [30958.302689] ? hrtimer_interrupt+0x104/0x240 [30958.305909] mei_me mei [30958.310435] ? __local_bh_enable+0x50/0x60 [30958.314525] lpc_ich [30958.318793] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30958.321232] sha512_ssse3 acpi_ipmi [30958.325325] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30958.327508] mfd_core wmi [30958.332726] [30958.336210] ipmi_si ipmi_devintf [30958.341167] [30958.341169] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30958.343792] ipmi_msghandler ioatdma [30958.345894] ? queued_spin_lock_slowpath+0x2d/0x2e0 [30958.349200] acpi_pad acpi_power_meter [30958.351300] do_raw_spin_lock+0xb5/0xc0 [30958.356603] ixgbe igb mdio [30958.360175] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30958.365045] crc32c_intel nvme [30958.368793] try_get_cap_refs+0x159/0x880 [ceph] [30958.372617] ptp nvme_core [30958.375413] __ceph_get_caps+0x4a1/0x760 [ceph] [30958.380194] i2c_algo_bit [30958.383247] ? __file_remove_privs+0x89/0x110 [30958.387854] pps_core dca [30958.390560] ? wake_bit_function+0x70/0x70 [30958.395084] fuse [30958.397703] ? ceph_write_iter+0x260/0x770 [ceph] [30958.402051] irq event stamp: 169015802 [30958.404686] ceph_write_iter+0x260/0x770 [ceph] [30958.408792] hardirqs last enabled at (169015801): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30958.410723] ? lock_is_held_type+0xaa/0x120 [30958.415415] hardirqs last disabled at (169015802): [] sysvec_apic_timer_interrupt+0xe/0x90 [30958.419162] vfs_write+0x324/0x4f0 [30958.423684] softirqs last enabled at (169015800): [] __do_softirq+0x2be/0x371 [30958.433932] ksys_pwrite64+0x65/0xa0 [30958.438104] softirqs last disabled at (169015795): [] irq_exit_rcu+0xb0/0xd0 [30958.447915] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30958.451312] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30958.460085] do_syscall_64+0x38/0x80 [30958.463656] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30958.472252] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30958.477383] Workqueue: events delayed_work [ceph] [30958.486761] RIP: 0033:0x7f48f0eeb93b [30958.490330] [30958.497803] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30958.502846] RIP: 0010:lock_acquire+0xf0/0x2b0 [30958.507543] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30958.511114] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30958.512605] ORIG_RAX: 0000000000000012 [30958.531344] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30958.535694] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30958.540910] [30958.540911] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30958.559648] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30958.563479] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30958.568696] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30958.575822] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30958.577311] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30958.584436] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30958.591561] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30958.598683] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30958.605813] [30958.612934] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30958.651708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30958.657445] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30958.664570] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30958.671694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30958.678817] Call Trace: [30958.681264] [30958.683275] ? watchdog_timer_fn+0x1b4/0x220 [30958.687547] ? __hrtimer_run_queues+0x1d3/0x360 [30958.692085] ? hrtimer_interrupt+0x104/0x240 [30958.696351] ? __local_bh_enable+0x50/0x60 [30958.700445] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30958.705669] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30958.710625] [30958.712723] [30958.714822] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30958.720138] ? lock_acquire+0xf0/0x2b0 [30958.723888] ? igrab+0x43/0x50 [30958.726937] ? lock_release+0xc5/0x260 [30958.730683] _raw_spin_lock+0x30/0x40 [30958.734348] ? igrab+0x1d/0x50 [30958.737397] igrab+0x1d/0x50 [30958.740274] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30958.745426] delayed_work+0x1d8/0x280 [ceph] [30958.749707] ? process_scheduled_works+0x22a/0x510 [30958.754497] process_scheduled_works+0x2a4/0x510 [30958.759113] worker_thread+0x188/0x340 [30958.762861] ? rescuer_thread+0x310/0x310 [30958.766865] kthread+0xf5/0x130 [30958.770003] ? kthread_complete_and_exit+0x20/0x20 [30958.774802] ret_from_fork+0x31/0x40 [30958.778374] ? kthread_complete_and_exit+0x20/0x20 [30958.783159] ret_from_fork_asm+0x11/0x20 [30958.787087] [30958.841775] rcu: INFO: rcu_preempt self-detected stall on CPU [30958.847535] rcu: 7-....: (18469621 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/18468303 fqs=9303206 [30958.858127] rcu: (t=19021893 jiffies g=1309609 q=555646 ncpus=8) [30958.864212] Sending NMI from CPU 7 to CPUs 0: [30958.868565] NMI backtrace for cpu 0 [30958.868566] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30958.868568] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30958.868568] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30958.868571] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30958.868573] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30958.868574] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30958.868575] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30958.868576] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30958.868577] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30958.868577] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30958.868578] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30958.868579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30958.868580] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30958.868581] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30958.868582] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30958.868582] Call Trace: [30958.868583] [30958.868584] ? nmi_cpu_backtrace+0xad/0x120 [30958.868588] ? nmi_cpu_backtrace_handler+0x11/0x20 [30958.868589] ? nmi_handle+0xd4/0x260 [30958.868594] ? default_do_nmi+0x49/0xf0 [30958.868595] ? exc_nmi+0x11e/0x150 [30958.868597] ? end_repeat_nmi+0x16/0x67 [30958.868601] ? queued_spin_lock_slowpath+0x16/0x2e0 [30958.868603] ? queued_spin_lock_slowpath+0x16/0x2e0 [30958.868606] ? queued_spin_lock_slowpath+0x16/0x2e0 [30958.868608] [30958.868609] [30958.868609] do_raw_spin_lock+0xb5/0xc0 [30958.868612] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30958.868626] try_get_cap_refs+0x159/0x880 [ceph] [30958.868640] __ceph_get_caps+0x4a1/0x760 [ceph] [30958.868653] ? __file_remove_privs+0x89/0x110 [30958.868655] ? wake_bit_function+0x70/0x70 [30958.868657] ? ceph_write_iter+0x260/0x770 [ceph] [30958.868668] ceph_write_iter+0x260/0x770 [ceph] [30958.868679] ? lock_is_held_type+0xaa/0x120 [30958.868682] vfs_write+0x324/0x4f0 [30958.868686] ksys_pwrite64+0x65/0xa0 [30958.868688] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30958.868691] do_syscall_64+0x38/0x80 [30958.868693] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30958.868695] RIP: 0033:0x7f48f0eeb93b [30958.868696] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30958.868697] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [30958.868698] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30958.868699] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30958.868700] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30958.868700] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30958.868701] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30958.868704] [30958.869562] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30959.173123] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30959.180594] Workqueue: events delayed_work [ceph] [30959.185308] RIP: 0010:lock_acquire+0xf0/0x2b0 [30959.189657] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30959.208395] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30959.213612] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30959.220736] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30959.227861] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30959.234986] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30959.242109] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30959.249234] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30959.257311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30959.263048] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30959.270174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30959.277297] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30959.284422] Call Trace: [30959.286864] [30959.288877] ? rcu_dump_cpu_stacks+0xd1/0x180 [30959.293235] ? rcu_sched_clock_irq+0x633/0x1390 [30959.297759] ? lock_release+0xc5/0x260 [30959.301503] ? lock_is_held_type+0xaa/0x120 [30959.305681] ? lock_is_held_type+0xaa/0x120 [30959.309859] ? tick_sched_do_timer+0xa0/0xa0 [30959.314132] ? rcu_read_lock_sched_held+0x56/0x90 [30959.318830] ? tick_sched_do_timer+0xa0/0xa0 [30959.323093] ? update_process_times+0x5f/0x90 [30959.327444] ? tick_sched_handle.isra.16+0x1e/0x50 [30959.332228] ? tick_sched_timer+0x6f/0x80 [30959.336231] ? __hrtimer_run_queues+0x1d3/0x360 [30959.340761] ? hrtimer_interrupt+0x104/0x240 [30959.345028] ? __local_bh_enable+0x50/0x60 [30959.349121] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30959.354345] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30959.359302] [30959.361401] [30959.363498] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30959.368805] ? lock_acquire+0xf0/0x2b0 [30959.372556] ? igrab+0x43/0x50 [30959.375605] ? lock_release+0xc5/0x260 [30959.379351] _raw_spin_lock+0x30/0x40 [30959.383014] ? igrab+0x1d/0x50 [30959.386066] igrab+0x1d/0x50 [30959.388944] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30959.394094] delayed_work+0x1d8/0x280 [ceph] [30959.398375] ? process_scheduled_works+0x22a/0x510 [30959.403166] process_scheduled_works+0x2a4/0x510 [30959.407782] worker_thread+0x188/0x340 [30959.411531] ? rescuer_thread+0x310/0x310 [30959.415541] kthread+0xf5/0x130 [30959.418680] ? kthread_complete_and_exit+0x20/0x20 [30959.423463] ret_from_fork+0x31/0x40 [30959.427033] ? kthread_complete_and_exit+0x20/0x20 [30959.431820] ret_from_fork_asm+0x11/0x20 [30959.435741] [30986.001889] watchdog: BUG: soft lockup - CPU#0 stuck for 17743s! [dbench:78683] [30986.009196] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [30986.013888] watchdog: BUG: soft lockup - CPU#7 stuck for 26740s! [kworker/7:4:78808] [30986.064637] irq event stamp: 52436272 [30986.064639] hardirqs last enabled at (52436271): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30986.072366] Modules linked in: [30986.076025] hardirqs last disabled at (52436272): [] sysvec_apic_timer_interrupt+0xe/0x90 [30986.086182] ceph [30986.089234] softirqs last enabled at (52436234): [] __do_softirq+0x2be/0x371 [30986.098956] libceph [30986.100882] softirqs last disabled at (52436229): [] irq_exit_rcu+0xb0/0xd0 [30986.109566] dns_resolver fscache [30986.111749] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30986.120278] netfs [30986.123587] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30986.132533] veth nft_chain_nat [30986.134542] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [30986.142031] xt_MASQUERADE nf_nat [30986.145168] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [30986.150646] nf_conntrack nf_defrag_ipv6 [30986.153956] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [30986.172694] nf_defrag_ipv4 [30986.176611] [30986.181828] nft_compat [30986.184619] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [30986.186110] nf_tables nfnetlink [30986.188555] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [30986.195697] bridge [30986.198919] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [30986.206045] stp llc [30986.208141] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [30986.215282] xfs [30986.217466] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [30986.224590] libcrc32c sunrpc [30986.226427] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [30986.233553] intel_rapl_msr [30986.236535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30986.244611] intel_rapl_common [30986.247401] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [30986.253140] sb_edac x86_pkg_temp_thermal [30986.256191] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30986.263315] coretemp kvm_intel [30986.267317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30986.274442] kvm irqbypass [30986.277580] Call Trace: [30986.277581] [30986.284705] i2c_i801 [30986.287409] ? watchdog_timer_fn+0x1b4/0x220 [30986.289851] iTCO_wdt iTCO_vendor_support [30986.291863] ? __hrtimer_run_queues+0x1d3/0x360 [30986.294134] i2c_smbus crct10dif_pclmul [30986.298401] ? hrtimer_interrupt+0x104/0x240 [30986.302402] joydev crc32_pclmul [30986.306927] ? __local_bh_enable+0x50/0x60 [30986.310756] ipmi_ssif ghash_clmulni_intel [30986.315023] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30986.318245] mei_me mei [30986.322335] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30986.326427] lpc_ich sha512_ssse3 [30986.331645] [30986.334088] acpi_ipmi mfd_core [30986.339045] [30986.339047] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30986.342357] wmi ipmi_si [30986.344458] ? queued_spin_lock_slowpath+0x16/0x2e0 [30986.347591] ipmi_devintf ipmi_msghandler [30986.349690] do_raw_spin_lock+0xb5/0xc0 [30986.354993] ioatdma acpi_pad acpi_power_meter [30986.357526] __cap_delay_requeue+0xa7/0x1d0 [ceph] [30986.362394] ixgbe igb [30986.366399] try_get_cap_refs+0x159/0x880 [ceph] [30986.370228] mdio crc32c_intel [30986.374672] __ceph_get_caps+0x4a1/0x760 [ceph] [30986.379450] nvme ptp [30986.381811] ? __file_remove_privs+0x89/0x110 [30986.386418] nvme_core i2c_algo_bit [30986.389471] ? wake_bit_function+0x70/0x70 [30986.393993] pps_core dca [30986.396267] ? ceph_write_iter+0x260/0x770 [ceph] [30986.400631] fuse [30986.404115] ceph_write_iter+0x260/0x770 [ceph] [30986.408207] irq event stamp: 169231620 [30986.410827] ? lock_is_held_type+0xaa/0x120 [30986.415521] hardirqs last enabled at (169231619): [] asm_common_interrupt+0x26/0x40 [30986.417447] vfs_write+0x324/0x4f0 [30986.421970] hardirqs last disabled at (169231620): [] sysvec_apic_timer_interrupt+0xe/0x90 [30986.425718] ksys_pwrite64+0x65/0xa0 [30986.429892] softirqs last enabled at (169231618): [] __do_softirq+0x2be/0x371 [30986.439181] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [30986.442579] softirqs last disabled at (169231611): [] irq_exit_rcu+0xb0/0xd0 [30986.452391] do_syscall_64+0x38/0x80 [30986.455960] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [30986.464732] entry_SYSCALL_64_after_hwframe+0x63/0xcd [30986.469862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [30986.478460] RIP: 0033:0x7f48f0eeb93b [30986.482031] Workqueue: events delayed_work [ceph] [30986.491408] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [30986.496453] [30986.503922] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [30986.507493] RIP: 0010:lock_acquire+0xf0/0x2b0 [30986.512190] ORIG_RAX: 0000000000000012 [30986.530929] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [30986.532419] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [30986.537637] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [30986.541988] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [30986.545819] [30986.564571] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [30986.571698] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [30986.576916] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [30986.584040] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [30986.585530] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [30986.592655] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [30986.599783] [30986.606901] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [30986.637601] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [30986.644725] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [30986.652801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30986.658539] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [30986.665664] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [30986.672787] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [30986.679911] Call Trace: [30986.682355] [30986.684368] ? watchdog_timer_fn+0x1b4/0x220 [30986.688642] ? __hrtimer_run_queues+0x1d3/0x360 [30986.693168] ? hrtimer_interrupt+0x104/0x240 [30986.697436] ? __local_bh_enable+0x50/0x60 [30986.701532] ? __sysvec_apic_timer_interrupt+0x61/0x160 [30986.706755] ? sysvec_apic_timer_interrupt+0x6d/0x90 [30986.711710] [30986.713808] [30986.715906] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [30986.721213] ? lock_acquire+0xf0/0x2b0 [30986.724965] ? igrab+0x43/0x50 [30986.728014] ? lock_release+0xc5/0x260 [30986.731760] _raw_spin_lock+0x30/0x40 [30986.735424] ? igrab+0x1d/0x50 [30986.738473] igrab+0x1d/0x50 [30986.741353] ceph_check_delayed_caps+0xb7/0x270 [ceph] [30986.746503] delayed_work+0x1d8/0x280 [ceph] [30986.750782] ? process_scheduled_works+0x22a/0x510 [30986.755573] process_scheduled_works+0x2a4/0x510 [30986.760189] worker_thread+0x188/0x340 [30986.763938] ? rescuer_thread+0x310/0x310 [30986.767941] kthread+0xf5/0x130 [30986.771078] ? kthread_complete_and_exit+0x20/0x20 [30986.775864] ret_from_fork+0x31/0x40 [30986.779434] ? kthread_complete_and_exit+0x20/0x20 [30986.784217] ret_from_fork_asm+0x11/0x20 [30986.788139] [31014.002385] watchdog: BUG: soft lockup - CPU#0 stuck for 17770s! [dbench:78683] [31014.009693] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31014.014384] watchdog: BUG: soft lockup - CPU#7 stuck for 26767s! [kworker/7:4:78808] [31014.065141] irq event stamp: 52494358 [31014.065142] hardirqs last enabled at (52494357): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31014.072872] Modules linked in: [31014.076530] hardirqs last disabled at (52494358): [] sysvec_apic_timer_interrupt+0xe/0x90 [31014.086687] ceph [31014.089739] softirqs last enabled at (52494310): [] __do_softirq+0x2be/0x371 [31014.099463] libceph [31014.101386] softirqs last disabled at (52494305): [] irq_exit_rcu+0xb0/0xd0 [31014.110071] dns_resolver fscache [31014.112254] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31014.120778] netfs [31014.124083] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31014.133027] veth nft_chain_nat [31014.135055] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31014.142528] xt_MASQUERADE [31014.145666] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31014.151143] nf_nat nf_conntrack [31014.153846] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31014.172585] nf_defrag_ipv6 [31014.175808] [31014.181025] nf_defrag_ipv4 nft_compat [31014.183817] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31014.185306] nf_tables [31014.189049] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31014.196193] nfnetlink bridge [31014.198548] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31014.205674] stp llc [31014.208638] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31014.215761] xfs libcrc32c [31014.217962] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31014.225087] sunrpc [31014.227790] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31014.234916] intel_rapl_msr [31014.237012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31014.245089] intel_rapl_common [31014.247880] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31014.253619] sb_edac [31014.256670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31014.263794] x86_pkg_temp_thermal [31014.265978] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31014.273101] coretemp kvm_intel [31014.276412] Call Trace: [31014.283538] kvm irqbypass [31014.286677] [31014.289117] i2c_i801 iTCO_wdt [31014.291825] ? watchdog_timer_fn+0x1b4/0x220 [31014.293832] iTCO_vendor_support [31014.296884] ? __hrtimer_run_queues+0x1d3/0x360 [31014.301147] i2c_smbus [31014.304375] ? hrtimer_interrupt+0x104/0x240 [31014.308895] crct10dif_pclmul joydev [31014.311254] ? __local_bh_enable+0x50/0x60 [31014.315517] crc32_pclmul ipmi_ssif [31014.319091] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31014.323178] ghash_clmulni_intel mei_me [31014.326663] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31014.331880] mei lpc_ich [31014.335711] [31014.340667] sha512_ssse3 acpi_ipmi [31014.343199] [31014.343200] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31014.345296] mfd_core [31014.348787] ? queued_spin_lock_slowpath+0x16/0x2e0 [31014.350878] wmi ipmi_si [31014.356185] do_raw_spin_lock+0xb5/0xc0 [31014.358454] ipmi_devintf ipmi_msghandler [31014.363326] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31014.365852] ioatdma [31014.369689] try_get_cap_refs+0x159/0x880 [ceph] [31014.373689] acpi_pad acpi_power_meter [31014.378479] __ceph_get_caps+0x4a1/0x760 [ceph] [31014.380657] ixgbe igb [31014.385272] ? __file_remove_privs+0x89/0x110 [31014.389013] mdio crc32c_intel [31014.393536] ? wake_bit_function+0x70/0x70 [31014.395894] nvme [31014.400247] ? ceph_write_iter+0x260/0x770 [ceph] [31014.403294] ptp nvme_core [31014.407385] ceph_write_iter+0x260/0x770 [ceph] [31014.409309] i2c_algo_bit pps_core [31014.414009] ? lock_is_held_type+0xaa/0x120 [31014.416710] dca fuse [31014.421238] vfs_write+0x324/0x4f0 [31014.424632] [31014.424633] irq event stamp: 169452578 [31014.428816] ksys_pwrite64+0x65/0xa0 [31014.431081] hardirqs last enabled at (169452577): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31014.434477] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31014.435967] hardirqs last disabled at (169452578): [] sysvec_apic_timer_interrupt+0xe/0x90 [31014.439714] do_syscall_64+0x38/0x80 [31014.443284] softirqs last enabled at (169452576): [] __do_softirq+0x2be/0x371 [31014.453529] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31014.458657] softirqs last disabled at (169452571): [] irq_exit_rcu+0xb0/0xd0 [31014.468470] RIP: 0033:0x7f48f0eeb93b [31014.472041] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31014.480812] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31014.485854] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31014.494452] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31014.498023] Workqueue: events delayed_work [ceph] [31014.507400] ORIG_RAX: 0000000000000012 [31014.526137] [31014.526138] RIP: 0010:lock_acquire+0xf0/0x2b0 [31014.533608] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31014.538825] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31014.543523] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31014.547353] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31014.548844] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31014.553195] [31014.553195] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31014.560320] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31014.579057] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31014.586181] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31014.591400] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31014.598529] [31014.600013] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31014.644943] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31014.652076] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31014.660153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31014.665891] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31014.673016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31014.680138] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31014.687263] Call Trace: [31014.689708] [31014.691719] ? watchdog_timer_fn+0x1b4/0x220 [31014.695983] ? __hrtimer_run_queues+0x1d3/0x360 [31014.700509] ? hrtimer_interrupt+0x104/0x240 [31014.704793] ? __local_bh_enable+0x50/0x60 [31014.708890] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31014.714112] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31014.719087] [31014.721186] [31014.723283] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31014.728592] ? lock_acquire+0xf0/0x2b0 [31014.732342] ? igrab+0x43/0x50 [31014.735392] ? lock_release+0xc5/0x260 [31014.739137] _raw_spin_lock+0x30/0x40 [31014.742802] ? igrab+0x1d/0x50 [31014.745852] igrab+0x1d/0x50 [31014.748729] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31014.753881] delayed_work+0x1d8/0x280 [ceph] [31014.758162] ? process_scheduled_works+0x22a/0x510 [31014.762950] process_scheduled_works+0x2a4/0x510 [31014.767565] worker_thread+0x188/0x340 [31014.771315] ? rescuer_thread+0x310/0x310 [31014.775328] kthread+0xf5/0x130 [31014.778474] ? kthread_complete_and_exit+0x20/0x20 [31014.783259] ret_from_fork+0x31/0x40 [31014.786836] ? kthread_complete_and_exit+0x20/0x20 [31014.791620] ret_from_fork_asm+0x11/0x20 [31014.795544] [31042.002879] watchdog: BUG: soft lockup - CPU#0 stuck for 17796s! [dbench:78683] [31042.010179] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31042.014879] watchdog: BUG: soft lockup - CPU#7 stuck for 26793s! [kworker/7:4:78808] [31042.065620] irq event stamp: 52552496 [31042.073350] Modules linked in: ceph libceph [31042.077008] hardirqs last enabled at (52552495): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31042.081184] dns_resolver fscache netfs [31042.091342] hardirqs last disabled at (52552496): [] sysvec_apic_timer_interrupt+0xe/0x90 [31042.095174] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [31042.104898] softirqs last enabled at (52552448): [] __do_softirq+0x2be/0x371 [31042.110981] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [31042.119665] softirqs last disabled at (52552443): [] irq_exit_rcu+0xb0/0xd0 [31042.119666] nf_tables nfnetlink bridge stp llc [31042.133221] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31042.133222] xfs libcrc32c sunrpc [31042.146690] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31042.146691] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31042.149999] intel_rapl_msr intel_rapl_common [31042.157472] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31042.162947] sb_edac x86_pkg_temp_thermal [31042.167299] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31042.186036] coretemp [31042.190041] [31042.195274] kvm_intel kvm [31042.197557] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31042.199046] irqbypass i2c_i801 [31042.201751] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31042.208873] iTCO_wdt iTCO_vendor_support [31042.212011] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31042.219136] i2c_smbus crct10dif_pclmul [31042.223140] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31042.230277] joydev crc32_pclmul [31042.234112] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31042.241235] ipmi_ssif ghash_clmulni_intel [31042.244460] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31042.251583] mei_me mei [31042.255676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31042.263752] lpc_ich sha512_ssse3 [31042.266196] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31042.271933] acpi_ipmi [31042.275245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31042.282368] mfd_core wmi [31042.284727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31042.291850] ipmi_si ipmi_devintf [31042.294468] Call Trace: [31042.301593] ipmi_msghandler ioatdma [31042.304902] [31042.307345] acpi_pad acpi_power_meter [31042.310917] ? watchdog_timer_fn+0x1b4/0x220 [31042.312929] ixgbe igb [31042.316674] ? __hrtimer_run_queues+0x1d3/0x360 [31042.320935] mdio crc32c_intel [31042.323295] ? hrtimer_interrupt+0x104/0x240 [31042.327817] nvme ptp [31042.330869] ? __local_bh_enable+0x50/0x60 [31042.335133] nvme_core i2c_algo_bit [31042.337406] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31042.341494] pps_core dca [31042.344979] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31042.350195] fuse [31042.352812] [31042.357769] irq event stamp: 169673290 [31042.359693] [31042.359694] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31042.361791] hardirqs last enabled at (169673289): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31042.365556] ? queued_spin_lock_slowpath+0x16/0x2e0 [31042.367651] hardirqs last disabled at (169673290): [] sysvec_apic_timer_interrupt+0xe/0x90 [31042.372956] do_raw_spin_lock+0xb5/0xc0 [31042.383198] softirqs last enabled at (169673288): [] __do_softirq+0x2be/0x371 [31042.388069] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31042.397879] softirqs last disabled at (169673283): [] irq_exit_rcu+0xb0/0xd0 [31042.401713] try_get_cap_refs+0x159/0x880 [ceph] [31042.410481] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31042.415284] __ceph_get_caps+0x4a1/0x760 [ceph] [31042.423880] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31042.428496] ? __file_remove_privs+0x89/0x110 [31042.437868] Workqueue: events delayed_work [ceph] [31042.442393] ? wake_bit_function+0x70/0x70 [31042.449864] [31042.449865] RIP: 0010:lock_acquire+0xf0/0x2b0 [31042.454217] ? ceph_write_iter+0x260/0x770 [ceph] [31042.458913] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31042.463003] ceph_write_iter+0x260/0x770 [ceph] [31042.464493] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31042.468848] ? lock_is_held_type+0xaa/0x120 [31042.473557] [31042.473557] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31042.492298] vfs_write+0x324/0x4f0 [31042.496820] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31042.502043] ksys_pwrite64+0x65/0xa0 [31042.506216] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31042.507706] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31042.514831] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31042.518229] do_syscall_64+0x38/0x80 [31042.525352] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31042.528924] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31042.536047] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31042.541177] RIP: 0033:0x7f48f0eeb93b [31042.548301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31042.551872] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31042.558996] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31042.564040] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31042.572119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31042.575688] ORIG_RAX: 0000000000000012 [31042.581427] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31042.600164] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31042.607288] Call Trace: [31042.607289] [31042.612506] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31042.619629] ? watchdog_timer_fn+0x1b4/0x220 [31042.623460] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31042.630584] ? __hrtimer_run_queues+0x1d3/0x360 [31042.637709] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31042.640157] ? hrtimer_interrupt+0x104/0x240 [31042.642163] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31042.649287] ? __local_bh_enable+0x50/0x60 [31042.653572] [31042.660696] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31042.695221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31042.700180] [31042.702285] [31042.704383] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31042.709691] ? lock_acquire+0xf0/0x2b0 [31042.713441] ? igrab+0x43/0x50 [31042.716491] ? lock_release+0xc5/0x260 [31042.720235] _raw_spin_lock+0x30/0x40 [31042.723890] ? igrab+0x1d/0x50 [31042.726942] igrab+0x1d/0x50 [31042.729820] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31042.734971] delayed_work+0x1d8/0x280 [ceph] [31042.739250] ? process_scheduled_works+0x22a/0x510 [31042.744033] process_scheduled_works+0x2a4/0x510 [31042.748648] worker_thread+0x188/0x340 [31042.752398] ? rescuer_thread+0x310/0x310 [31042.756402] kthread+0xf5/0x130 [31042.759538] ? kthread_complete_and_exit+0x20/0x20 [31042.764325] ret_from_fork+0x31/0x40 [31042.767901] ? kthread_complete_and_exit+0x20/0x20 [31042.772686] ret_from_fork_asm+0x11/0x20 [31042.776610] [31070.003377] watchdog: BUG: soft lockup - CPU#0 stuck for 17822s! [dbench:78683] [31070.010684] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31070.015376] watchdog: BUG: soft lockup - CPU#7 stuck for 26819s! [kworker/7:4:78808] [31070.066133] irq event stamp: 52610520 [31070.066135] hardirqs last enabled at (52610519): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31070.073865] Modules linked in: [31070.077522] hardirqs last disabled at (52610520): [] sysvec_apic_timer_interrupt+0xe/0x90 [31070.087680] ceph [31070.090729] softirqs last enabled at (52610472): [] __do_softirq+0x2be/0x371 [31070.100455] libceph [31070.102378] softirqs last disabled at (52610467): [] irq_exit_rcu+0xb0/0xd0 [31070.111062] dns_resolver fscache [31070.113248] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31070.121772] netfs [31070.125084] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31070.134028] veth nft_chain_nat [31070.136061] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31070.143527] xt_MASQUERADE [31070.146668] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31070.152144] nf_nat nf_conntrack [31070.154847] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31070.173585] nf_defrag_ipv6 [31070.176808] [31070.182027] nf_defrag_ipv4 nft_compat [31070.184818] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31070.186309] nf_tables nfnetlink [31070.190064] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31070.197184] bridge stp [31070.200407] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31070.207532] llc [31070.209977] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31070.217101] xfs libcrc32c [31070.218939] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31070.226062] sunrpc intel_rapl_msr [31070.228771] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31070.235898] intel_rapl_common [31070.239295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31070.247373] sb_edac [31070.250424] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31070.256162] x86_pkg_temp_thermal coretemp [31070.258347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31070.265471] kvm_intel kvm [31070.269560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31070.276685] irqbypass i2c_i801 [31070.279388] Call Trace: [31070.286512] iTCO_wdt [31070.289654] [31070.292095] iTCO_vendor_support i2c_smbus [31070.294368] ? watchdog_timer_fn+0x1b4/0x220 [31070.296377] crct10dif_pclmul joydev [31070.300469] ? __hrtimer_run_queues+0x1d3/0x360 [31070.304732] crc32_pclmul [31070.308306] ? hrtimer_interrupt+0x104/0x240 [31070.312826] ipmi_ssif [31070.315443] ? __local_bh_enable+0x50/0x60 [31070.319708] ghash_clmulni_intel [31070.322066] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31070.326155] mei_me [31070.329381] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31070.334596] mei [31070.336695] [31070.341651] lpc_ich [31070.343490] [31070.343491] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31070.345586] sha512_ssse3 acpi_ipmi [31070.347779] ? queued_spin_lock_slowpath+0x16/0x2e0 [31070.349876] mfd_core [31070.355182] do_raw_spin_lock+0xb5/0xc0 [31070.358665] wmi [31070.363538] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31070.365805] ipmi_si ipmi_devintf [31070.369639] try_get_cap_refs+0x159/0x880 [ceph] [31070.371474] ipmi_msghandler ioatdma [31070.376264] __ceph_get_caps+0x4a1/0x760 [ceph] [31070.379568] acpi_pad acpi_power_meter [31070.384184] ? __file_remove_privs+0x89/0x110 [31070.387751] ixgbe [31070.392292] ? wake_bit_function+0x70/0x70 [31070.396052] igb mdio [31070.400406] ? ceph_write_iter+0x260/0x770 [ceph] [31070.402415] crc32c_intel nvme [31070.406505] ceph_write_iter+0x260/0x770 [ceph] [31070.408777] ptp nvme_core [31070.413477] ? lock_is_held_type+0xaa/0x120 [31070.416523] i2c_algo_bit pps_core [31070.421058] vfs_write+0x324/0x4f0 [31070.423778] dca [31070.427960] ksys_pwrite64+0x65/0xa0 [31070.431354] fuse [31070.434753] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31070.436606] [31070.436607] irq event stamp: 169894250 [31070.440178] do_syscall_64+0x38/0x80 [31070.442099] hardirqs last enabled at (169894249): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31070.447232] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31070.448721] hardirqs last disabled at (169894250): [] sysvec_apic_timer_interrupt+0xe/0x90 [31070.452467] RIP: 0033:0x7f48f0eeb93b [31070.456036] softirqs last enabled at (169894248): [] __do_softirq+0x2be/0x371 [31070.466300] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31070.471341] softirqs last disabled at (169894243): [] irq_exit_rcu+0xb0/0xd0 [31070.481152] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31070.484725] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31070.493494] ORIG_RAX: 0000000000000012 [31070.512231] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31070.520828] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31070.526056] Workqueue: events delayed_work [ceph] [31070.535433] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31070.539265] [31070.546734] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31070.553858] RIP: 0010:lock_acquire+0xf0/0x2b0 [31070.558556] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31070.565679] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31070.567171] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31070.574294] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31070.578652] [31070.585781] [31070.585781] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31070.627674] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31070.634798] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31070.641923] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31070.649048] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31070.656188] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31070.664266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31070.670003] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31070.677129] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31070.684252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31070.691376] Call Trace: [31070.693820] [31070.695831] ? watchdog_timer_fn+0x1b4/0x220 [31070.700096] ? __hrtimer_run_queues+0x1d3/0x360 [31070.704623] ? hrtimer_interrupt+0x104/0x240 [31070.708890] ? __local_bh_enable+0x50/0x60 [31070.712986] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31070.718208] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31070.723166] [31070.725262] [31070.727360] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31070.732668] ? lock_acquire+0xf0/0x2b0 [31070.736419] ? igrab+0x43/0x50 [31070.739470] ? lock_release+0xc5/0x260 [31070.743213] _raw_spin_lock+0x30/0x40 [31070.746871] ? igrab+0x1d/0x50 [31070.749920] igrab+0x1d/0x50 [31070.752800] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31070.757949] delayed_work+0x1d8/0x280 [ceph] [31070.762229] ? process_scheduled_works+0x22a/0x510 [31070.767019] process_scheduled_works+0x2a4/0x510 [31070.771637] worker_thread+0x188/0x340 [31070.775384] ? rescuer_thread+0x310/0x310 [31070.779386] kthread+0xf5/0x130 [31070.782527] ? kthread_complete_and_exit+0x20/0x20 [31070.787311] ret_from_fork+0x31/0x40 [31070.790891] ? kthread_complete_and_exit+0x20/0x20 [31070.795682] ret_from_fork_asm+0x11/0x20 [31070.799613] [31098.003873] watchdog: BUG: soft lockup - CPU#0 stuck for 17848s! [dbench:78683] [31098.011170] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31098.015873] watchdog: BUG: soft lockup - CPU#7 stuck for 26845s! [kworker/7:4:78808] [31098.066612] irq event stamp: 52668656 [31098.066613] hardirqs last enabled at (52668655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31098.074343] Modules linked in: [31098.078000] hardirqs last disabled at (52668656): [] sysvec_apic_timer_interrupt+0xe/0x90 [31098.088157] ceph [31098.091209] softirqs last enabled at (52668608): [] __do_softirq+0x2be/0x371 [31098.100932] libceph [31098.102858] softirqs last disabled at (52668603): [] irq_exit_rcu+0xb0/0xd0 [31098.111549] dns_resolver fscache [31098.113734] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31098.122246] netfs veth [31098.125574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31098.134518] nft_chain_nat xt_MASQUERADE [31098.136961] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31098.144432] nf_nat [31098.148349] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31098.153826] nf_conntrack nf_defrag_ipv6 [31098.155923] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31098.174660] nf_defrag_ipv4 nft_compat [31098.178580] [31098.178580] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31098.183795] nf_tables nfnetlink [31098.187556] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31098.189049] bridge stp [31098.196173] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31098.199397] llc xfs [31098.206520] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31098.208964] libcrc32c [31098.216090] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31098.218272] sunrpc intel_rapl_msr [31098.225397] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31098.227753] intel_rapl_common sb_edac [31098.234879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31098.238276] x86_pkg_temp_thermal [31098.246353] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31098.250098] coretemp kvm_intel [31098.255836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31098.259145] kvm [31098.266271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31098.269407] irqbypass i2c_i801 [31098.276531] Call Trace: [31098.278369] iTCO_wdt iTCO_vendor_support [31098.285492] [31098.288630] i2c_smbus crct10dif_pclmul [31098.291076] ? watchdog_timer_fn+0x1b4/0x220 [31098.295078] joydev crc32_pclmul [31098.297089] ? __hrtimer_run_queues+0x1d3/0x360 [31098.300920] ipmi_ssif ghash_clmulni_intel [31098.305188] ? hrtimer_interrupt+0x104/0x240 [31098.308407] mei_me [31098.312932] ? __local_bh_enable+0x50/0x60 [31098.317023] mei [31098.321290] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31098.323384] lpc_ich [31098.327475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31098.329313] sha512_ssse3 acpi_ipmi [31098.334531] [31098.336715] mfd_core wmi [31098.341670] [31098.341671] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31098.345154] ipmi_si ipmi_devintf [31098.347273] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31098.349887] ipmi_msghandler [31098.351986] do_raw_spin_lock+0xb5/0xc0 [31098.357288] ioatdma acpi_pad acpi_power_meter [31098.360601] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31098.365471] ixgbe igb [31098.368350] try_get_cap_refs+0x159/0x880 [ceph] [31098.372178] mdio crc32c_intel [31098.376620] __ceph_get_caps+0x4a1/0x760 [ceph] [31098.381399] nvme [31098.383763] ? __file_remove_privs+0x89/0x110 [31098.388368] ptp nvme_core [31098.391418] ? wake_bit_function+0x70/0x70 [31098.395943] i2c_algo_bit [31098.397871] ? ceph_write_iter+0x260/0x770 [ceph] [31098.402217] pps_core [31098.404923] ceph_write_iter+0x260/0x770 [ceph] [31098.409013] dca fuse [31098.411634] ? lock_is_held_type+0xaa/0x120 [31098.416328] [31098.418600] vfs_write+0x324/0x4f0 [31098.423121] irq event stamp: 170114946 [31098.425398] ksys_pwrite64+0x65/0xa0 [31098.429571] hardirqs last enabled at (170114945): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31098.431062] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31098.434458] hardirqs last disabled at (170114946): [] sysvec_apic_timer_interrupt+0xe/0x90 [31098.438204] do_syscall_64+0x38/0x80 [31098.441772] softirqs last enabled at (170114944): [] __do_softirq+0x2be/0x371 [31098.452019] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31098.457149] softirqs last disabled at (170114939): [] irq_exit_rcu+0xb0/0xd0 [31098.466960] RIP: 0033:0x7f48f0eeb93b [31098.470530] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31098.479301] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31098.484345] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31098.492943] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31098.496513] Workqueue: events delayed_work [ceph] [31098.505891] ORIG_RAX: 0000000000000012 [31098.524629] [31098.524630] RIP: 0010:lock_acquire+0xf0/0x2b0 [31098.532098] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31098.537316] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31098.542013] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31098.545844] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31098.547335] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31098.551686] [31098.551686] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31098.558810] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31098.577563] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31098.584690] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31098.589907] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31098.597035] [31098.598521] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31098.643450] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31098.650574] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31098.658652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31098.664390] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31098.671514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31098.678640] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31098.685761] Call Trace: [31098.688207] [31098.690216] ? watchdog_timer_fn+0x1b4/0x220 [31098.694483] ? __hrtimer_run_queues+0x1d3/0x360 [31098.699008] ? hrtimer_interrupt+0x104/0x240 [31098.703279] ? __local_bh_enable+0x50/0x60 [31098.707372] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31098.712595] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31098.717567] [31098.719666] [31098.721764] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31098.727071] ? lock_acquire+0xf0/0x2b0 [31098.730821] ? igrab+0x43/0x50 [31098.733873] ? lock_release+0xc5/0x260 [31098.737618] _raw_spin_lock+0x30/0x40 [31098.741281] ? igrab+0x1d/0x50 [31098.744332] igrab+0x1d/0x50 [31098.747212] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31098.752361] delayed_work+0x1d8/0x280 [ceph] [31098.756641] ? process_scheduled_works+0x22a/0x510 [31098.761433] process_scheduled_works+0x2a4/0x510 [31098.766049] worker_thread+0x188/0x340 [31098.769797] ? rescuer_thread+0x310/0x310 [31098.773801] kthread+0xf5/0x130 [31098.776936] ? kthread_complete_and_exit+0x20/0x20 [31098.781722] ret_from_fork+0x31/0x40 [31098.785293] ? kthread_complete_and_exit+0x20/0x20 [31098.790076] ret_from_fork_asm+0x11/0x20 [31098.793999] [31126.004371] watchdog: BUG: soft lockup - CPU#0 stuck for 17874s! [dbench:78683] [31126.011678] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31126.016371] watchdog: BUG: soft lockup - CPU#7 stuck for 26871s! [kworker/7:4:78808] [31126.067126] irq event stamp: 52726814 [31126.067128] hardirqs last enabled at (52726813): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31126.074858] Modules linked in: ceph [31126.078515] hardirqs last disabled at (52726814): [] sysvec_apic_timer_interrupt+0xe/0x90 [31126.088671] libceph [31126.092155] softirqs last enabled at (52726766): [] __do_softirq+0x2be/0x371 [31126.101879] dns_resolver fscache [31126.104064] softirqs last disabled at (52726761): [] irq_exit_rcu+0xb0/0xd0 [31126.112765] netfs veth [31126.116076] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31126.124586] nft_chain_nat xt_MASQUERADE [31126.127047] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31126.135992] nf_nat nf_conntrack [31126.139909] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31126.147381] nf_defrag_ipv6 nf_defrag_ipv4 [31126.150606] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31126.156081] nft_compat nf_tables [31126.160173] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31126.178909] nfnetlink bridge [31126.182222] [31126.187438] stp [31126.190402] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31126.191892] llc xfs [31126.193731] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31126.200855] libcrc32c sunrpc [31126.203048] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31126.210172] intel_rapl_msr intel_rapl_common [31126.213137] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31126.220261] sb_edac [31126.224611] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31126.231734] x86_pkg_temp_thermal [31126.233918] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31126.241042] coretemp kvm_intel [31126.244371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31126.252448] kvm [31126.255586] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31126.261324] irqbypass i2c_i801 [31126.263160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31126.270284] iTCO_wdt [31126.273421] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31126.280548] iTCO_vendor_support i2c_smbus [31126.282817] Call Trace: [31126.289940] crct10dif_pclmul joydev [31126.294051] [31126.296493] crc32_pclmul [31126.300067] ? watchdog_timer_fn+0x1b4/0x220 [31126.302074] ipmi_ssif ghash_clmulni_intel [31126.304694] ? __hrtimer_run_queues+0x1d3/0x360 [31126.308955] mei_me mei [31126.313060] ? hrtimer_interrupt+0x104/0x240 [31126.317579] lpc_ich sha512_ssse3 [31126.320024] ? __local_bh_enable+0x50/0x60 [31126.324305] acpi_ipmi mfd_core [31126.327619] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31126.331707] wmi ipmi_si [31126.334845] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31126.340061] ipmi_devintf [31126.342592] [31126.347550] ipmi_msghandler ioatdma [31126.350167] [31126.350168] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31126.352264] acpi_pad acpi_power_meter [31126.355839] ? queued_spin_lock_slowpath+0x16/0x2e0 [31126.357931] ixgbe igb [31126.363240] do_raw_spin_lock+0xb5/0xc0 [31126.366981] mdio crc32c_intel [31126.371853] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31126.374208] nvme ptp [31126.378054] try_get_cap_refs+0x159/0x880 [ceph] [31126.381100] nvme_core i2c_algo_bit [31126.385889] __ceph_get_caps+0x4a1/0x760 [ceph] [31126.388154] pps_core [31126.392774] ? __file_remove_privs+0x89/0x110 [31126.396257] dca fuse [31126.400781] ? wake_bit_function+0x70/0x70 [31126.403061] [31126.403062] irq event stamp: 170335828 [31126.407414] ? ceph_write_iter+0x260/0x770 [ceph] [31126.409683] hardirqs last enabled at (170335827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31126.413774] ceph_write_iter+0x260/0x770 [ceph] [31126.415263] hardirqs last disabled at (170335828): [] sysvec_apic_timer_interrupt+0xe/0x90 [31126.419011] ? lock_is_held_type+0xaa/0x120 [31126.423704] softirqs last enabled at (170335826): [] __do_softirq+0x2be/0x371 [31126.433953] vfs_write+0x324/0x4f0 [31126.438474] softirqs last disabled at (170335821): [] irq_exit_rcu+0xb0/0xd0 [31126.448290] ksys_pwrite64+0x65/0xa0 [31126.452461] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31126.461232] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31126.464630] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31126.473229] do_syscall_64+0x38/0x80 [31126.476797] Workqueue: events delayed_work [ceph] [31126.486176] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31126.491305] [31126.498777] RIP: 0033:0x7f48f0eeb93b [31126.502349] RIP: 0010:lock_acquire+0xf0/0x2b0 [31126.507055] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31126.512098] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31126.513589] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31126.517160] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31126.521511] ORIG_RAX: 0000000000000012 [31126.540248] [31126.558985] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31126.564204] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31126.569420] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31126.573252] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31126.574743] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31126.581866] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31126.588991] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31126.596113] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31126.603237] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31126.610361] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31126.617493] [31126.624610] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31126.656245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31126.661982] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31126.669107] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31126.676231] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31126.683354] Call Trace: [31126.685800] [31126.687812] ? watchdog_timer_fn+0x1b4/0x220 [31126.692073] ? __hrtimer_run_queues+0x1d3/0x360 [31126.696601] ? hrtimer_interrupt+0x104/0x240 [31126.700871] ? __local_bh_enable+0x50/0x60 [31126.704964] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31126.710187] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31126.715145] [31126.717242] [31126.719340] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31126.724649] ? lock_acquire+0xf0/0x2b0 [31126.728397] ? igrab+0x43/0x50 [31126.731448] ? lock_release+0xc5/0x260 [31126.735192] _raw_spin_lock+0x30/0x40 [31126.738848] ? igrab+0x1d/0x50 [31126.741900] igrab+0x1d/0x50 [31126.744780] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31126.749927] delayed_work+0x1d8/0x280 [ceph] [31126.754210] ? process_scheduled_works+0x22a/0x510 [31126.759000] process_scheduled_works+0x2a4/0x510 [31126.763615] worker_thread+0x188/0x340 [31126.767364] ? rescuer_thread+0x310/0x310 [31126.771366] kthread+0xf5/0x130 [31126.774505] ? kthread_complete_and_exit+0x20/0x20 [31126.779290] ret_from_fork+0x31/0x40 [31126.782868] ? kthread_complete_and_exit+0x20/0x20 [31126.787653] ret_from_fork_asm+0x11/0x20 [31126.791582] [31154.004870] watchdog: BUG: soft lockup - CPU#0 stuck for 17900s! [dbench:78683] [31154.012173] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31154.016870] watchdog: BUG: soft lockup - CPU#7 stuck for 26897s! [kworker/7:4:78808] [31154.067613] irq event stamp: 52784938 [31154.067614] hardirqs last enabled at (52784937): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31154.075344] Modules linked in: [31154.079002] hardirqs last disabled at (52784938): [] sysvec_apic_timer_interrupt+0xe/0x90 [31154.089159] ceph [31154.092210] softirqs last enabled at (52784890): [] __do_softirq+0x2be/0x371 [31154.101934] libceph [31154.103857] softirqs last disabled at (52784885): [] irq_exit_rcu+0xb0/0xd0 [31154.112544] dns_resolver fscache [31154.114727] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31154.123238] netfs veth [31154.126565] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31154.135510] nft_chain_nat [31154.137955] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31154.145423] xt_MASQUERADE nf_nat [31154.148128] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31154.153605] nf_conntrack nf_defrag_ipv6 [31154.156918] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31154.175653] nf_defrag_ipv4 [31154.179572] [31154.179573] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31154.184789] nft_compat nf_tables [31154.187580] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31154.189071] nfnetlink bridge [31154.196195] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31154.199504] stp llc [31154.206630] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31154.209593] xfs libcrc32c [31154.216718] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31154.218901] sunrpc intel_rapl_msr [31154.226026] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31154.228730] intel_rapl_common sb_edac [31154.235854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31154.239266] x86_pkg_temp_thermal [31154.247347] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31154.251089] coretemp [31154.256828] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31154.260139] kvm_intel kvm [31154.267269] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31154.269548] irqbypass i2c_i801 [31154.276675] Call Trace: [31154.279379] iTCO_wdt iTCO_vendor_support [31154.286502] [31154.289639] i2c_smbus crct10dif_pclmul [31154.292085] ? watchdog_timer_fn+0x1b4/0x220 [31154.296089] joydev crc32_pclmul [31154.298101] ? __hrtimer_run_queues+0x1d3/0x360 [31154.301928] ipmi_ssif ghash_clmulni_intel [31154.306196] ? hrtimer_interrupt+0x104/0x240 [31154.309419] mei_me mei [31154.313943] ? __local_bh_enable+0x50/0x60 [31154.318032] lpc_ich [31154.322300] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31154.324741] sha512_ssse3 acpi_ipmi [31154.328831] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31154.331015] mfd_core wmi [31154.336234] [31154.339718] ipmi_si ipmi_devintf [31154.344674] [31154.344676] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31154.347291] ipmi_msghandler [31154.349393] ? queued_spin_lock_slowpath+0x18/0x2e0 [31154.352700] ioatdma acpi_pad [31154.354799] do_raw_spin_lock+0xb5/0xc0 [31154.360103] acpi_power_meter ixgbe igb [31154.362982] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31154.367850] mdio [31154.370816] try_get_cap_refs+0x159/0x880 [ceph] [31154.374644] crc32c_intel nvme [31154.378480] __ceph_get_caps+0x4a1/0x760 [ceph] [31154.383270] ptp nvme_core [31154.385196] ? __file_remove_privs+0x89/0x110 [31154.389803] i2c_algo_bit pps_core [31154.392854] ? wake_bit_function+0x70/0x70 [31154.397378] dca fuse [31154.400085] ? ceph_write_iter+0x260/0x770 [ceph] [31154.404432] [31154.404433] irq event stamp: 170556644 [31154.407831] ceph_write_iter+0x260/0x770 [ceph] [31154.411920] hardirqs last enabled at (170556643): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31154.414195] ? lock_is_held_type+0xaa/0x120 [31154.418890] hardirqs last disabled at (170556644): [] sysvec_apic_timer_interrupt+0xe/0x90 [31154.420382] vfs_write+0x324/0x4f0 [31154.424124] softirqs last enabled at (170556642): [] __do_softirq+0x2be/0x371 [31154.428651] ksys_pwrite64+0x65/0xa0 [31154.438891] softirqs last disabled at (170556637): [] irq_exit_rcu+0xb0/0xd0 [31154.443069] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31154.446244] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [31154.446247] rcu: 0-....: (18784773 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9399177 [31154.446269] rcu: 7-....: (18659534 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/18658216 fqs=9399178 [31154.446273] rcu: (detected by 4, t=19217479 jiffies, g=1309609, q=559699 ncpus=8) [31154.446276] Sending NMI from CPU 4 to CPUs 0: [31154.446279] NMI backtrace for cpu 0 [31154.446280] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31154.446282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31154.446283] RIP: 0010:vprintk_emit+0x1eb/0x340 [31154.446286] Code: 50 41 b8 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 <0f> b6 11 84 d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 [31154.446289] RSP: 0018:ffffc90000003d30 EFLAGS: 00000002 [31154.446290] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [31154.446292] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [31154.446293] RBP: 000000000000002a R08: 0000000000005ef4 R09: 0000000000000004 [31154.446294] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [31154.446295] R13: 0000000000000000 R14: ffffffff82123e9c R15: ffffc90000003d80 [31154.446297] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31154.446298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31154.446299] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31154.446301] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31154.446302] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31154.446303] Call Trace: [31154.446304] [31154.446305] ? nmi_cpu_backtrace+0xad/0x120 [31154.446309] ? nmi_cpu_backtrace_handler+0x11/0x20 [31154.446311] ? nmi_handle+0xd4/0x260 [31154.446317] ? default_do_nmi+0x49/0xf0 [31154.446319] ? exc_nmi+0x11e/0x150 [31154.446322] ? end_repeat_nmi+0x16/0x67 [31154.446328] ? vprintk_emit+0x1eb/0x340 [31154.446331] ? vprintk_emit+0x1eb/0x340 [31154.446334] ? vprintk_emit+0x1eb/0x340 [31154.446337] [31154.446338] [31154.446339] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31154.446342] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31154.446345] _printk+0x5c/0x80 [31154.446348] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31154.446351] show_trace_log_lvl+0x1ac/0x3f0 [31154.446359] ? do_syscall_64+0x38/0x80 [31154.446363] ? touch_softlockup_watchdog+0x20/0x20 [31154.446365] watchdog_timer_fn+0x1b4/0x220 [31154.446368] __hrtimer_run_queues+0x1d3/0x360 [31154.446373] hrtimer_interrupt+0x104/0x240 [31154.446375] ? __local_bh_enable+0x50/0x60 [31154.446379] __sysvec_apic_timer_interrupt+0x61/0x160 [31154.446383] sysvec_apic_timer_interrupt+0x6d/0x90 [31154.446385] [31154.446386] [31154.446387] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31154.446389] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31154.446392] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31154.446394] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31154.446395] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31154.446396] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31154.446397] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31154.446399] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31154.446400] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31154.446406] do_raw_spin_lock+0xb5/0xc0 [31154.446409] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31154.446428] try_get_cap_refs+0x159/0x880 [ceph] [31154.446449] __ceph_get_caps+0x4a1/0x760 [ceph] [31154.446468] ? __file_remove_privs+0x89/0x110 [31154.446471] ? wake_bit_function+0x70/0x70 [31154.446475] ? ceph_write_iter+0x260/0x770 [ceph] [31154.446490] ceph_write_iter+0x260/0x770 [ceph] [31154.446508] ? lock_is_held_type+0xaa/0x120 [31154.446512] vfs_write+0x324/0x4f0 [31154.446519] ksys_pwrite64+0x65/0xa0 [31154.446521] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31154.446525] do_syscall_64+0x38/0x80 [31154.446528] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31154.446530] RIP: 0033:0x7f48f0eeb93b [31154.446532] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31154.446534] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [31154.446552] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31154.446553] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31154.446554] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31154.446555] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31154.446556] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31154.446562] [31154.447278] Sending NMI from CPU 4 to CPUs 7: [31154.447281] NMI backtrace for cpu 7 [31154.447282] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31154.447284] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31154.447285] Workqueue: events delayed_work [ceph] [31154.447300] RIP: 0010:io_serial_in+0x19/0x20 [31154.447304] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [31154.447305] RSP: 0018:ffffc90000264cc8 EFLAGS: 00000002 [31154.447306] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [31154.447307] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [31154.447308] RBP: 00000000000026d0 R08: 000064c5a2bf1a92 R09: 0000000000000007 [31154.447309] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [31154.447309] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [31154.447310] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31154.447311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31154.447312] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31154.447313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31154.447314] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31154.447315] Call Trace: [31154.447316] [31154.447317] ? nmi_cpu_backtrace+0xad/0x120 [31154.447320] ? nmi_cpu_backtrace_handler+0x11/0x20 [31154.447321] ? nmi_handle+0xd4/0x260 [31154.447324] ? default_do_nmi+0x49/0xf0 [31154.447326] ? exc_nmi+0x11e/0x150 [31154.447328] ? end_repeat_nmi+0x16/0x67 [31154.447331] ? io_serial_in+0x19/0x20 [31154.447333] ? io_serial_in+0x19/0x20 [31154.447335] ? io_serial_in+0x19/0x20 [31154.447337] [31154.447337] [31154.447337] wait_for_lsr+0x45/0x70 [31154.447339] serial8250_console_write+0x341/0x3f0 [31154.447343] ? lock_release+0xc5/0x260 [31154.447345] console_flush_all+0x227/0x5a0 [31154.447347] ? console_flush_all+0x1ff/0x5a0 [31154.447348] ? console_flush_all+0x1ff/0x5a0 [31154.447351] console_unlock+0x7b/0x120 [31154.447353] vprintk_emit+0x246/0x340 [31154.447355] ? touch_softlockup_watchdog+0x20/0x20 [31154.447357] _printk+0x5c/0x80 [31154.447359] ? __do_softirq+0x2be/0x371 [31154.447361] ? irq_exit_rcu+0xb0/0xd0 [31154.447363] ? irq_exit_rcu+0xb0/0xd0 [31154.447366] watchdog_timer_fn+0x1a7/0x220 [31154.447367] __hrtimer_run_queues+0x1d3/0x360 [31154.447370] hrtimer_interrupt+0x104/0x240 [31154.447371] ? __local_bh_enable+0x50/0x60 [31154.447374] __sysvec_apic_timer_interrupt+0x61/0x160 [31154.447377] sysvec_apic_timer_interrupt+0x6d/0x90 [31154.447379] [31154.447379] [31154.447380] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31154.447382] RIP: 0010:lock_acquire+0xf0/0x2b0 [31154.447383] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31154.447385] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31154.447386] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31154.447386] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31154.447387] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31154.447388] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31154.447388] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31154.447391] ? igrab+0x43/0x50 [31154.447393] ? lock_release+0xc5/0x260 [31154.447395] _raw_spin_lock+0x30/0x40 [31154.447397] ? igrab+0x1d/0x50 [31154.447398] igrab+0x1d/0x50 [31154.447399] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31154.447414] delayed_work+0x1d8/0x280 [ceph] [31154.447427] ? process_scheduled_works+0x22a/0x510 [31154.447430] process_scheduled_works+0x2a4/0x510 [31154.447434] worker_thread+0x188/0x340 [31154.447437] ? rescuer_thread+0x310/0x310 [31154.447439] kthread+0xf5/0x130 [31154.447441] ? kthread_complete_and_exit+0x20/0x20 [31154.447443] ret_from_fork+0x31/0x40 [31154.447446] ? kthread_complete_and_exit+0x20/0x20 [31154.447448] ret_from_fork_asm+0x11/0x20 [31154.447452] [31154.452880] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31154.456281] do_syscall_64+0x38/0x80 [31154.465047] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31154.468620] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31154.477215] Workqueue: events delayed_work [ceph] [31154.482347] RIP: 0033:0x7f48f0eeb93b [31154.488431] [31154.498848] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31154.509438] RIP: 0010:lock_acquire+0xf0/0x2b0 [31154.516997] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31154.521348] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31154.524833] ORIG_RAX: 0000000000000012 [31154.533777] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31154.541248] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31154.545700] [31154.545701] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31154.564438] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31154.569655] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31154.576780] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31154.583906] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31154.591030] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31154.598154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31154.605278] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31154.613355] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31154.619098] [31154.626216] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31155.505959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31155.511699] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31155.518821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31155.525945] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31155.533071] Call Trace: [31155.535515] [31155.537525] ? watchdog_timer_fn+0x1b4/0x220 [31155.541791] ? __hrtimer_run_queues+0x1d3/0x360 [31155.546315] ? hrtimer_interrupt+0x104/0x240 [31155.550580] ? __local_bh_enable+0x50/0x60 [31155.554678] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31155.559894] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31155.564852] [31155.566948] [31155.569046] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31155.574354] ? lock_acquire+0xf0/0x2b0 [31155.578106] ? igrab+0x43/0x50 [31155.581153] ? lock_release+0xc5/0x260 [31155.584899] _raw_spin_lock+0x30/0x40 [31155.588570] ? igrab+0x1d/0x50 [31155.591623] igrab+0x1d/0x50 [31155.594501] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31155.599650] delayed_work+0x1d8/0x280 [ceph] [31155.603932] ? process_scheduled_works+0x22a/0x510 [31155.608723] process_scheduled_works+0x2a4/0x510 [31155.613338] worker_thread+0x188/0x340 [31155.617088] ? rescuer_thread+0x310/0x310 [31155.621091] kthread+0xf5/0x130 [31155.624228] ? kthread_complete_and_exit+0x20/0x20 [31155.629013] ret_from_fork+0x31/0x40 [31155.632582] ? kthread_complete_and_exit+0x20/0x20 [31155.637368] ret_from_fork_asm+0x11/0x20 [31155.641303] [31182.005369] watchdog: BUG: soft lockup - CPU#0 stuck for 17926s! [dbench:78683] [31182.012678] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31182.017369] watchdog: BUG: soft lockup - CPU#7 stuck for 26923s! [kworker/7:4:78808] [31182.068126] irq event stamp: 52843030 [31182.075859] Modules linked in: ceph libceph dns_resolver fscache netfs [31182.079515] hardirqs last enabled at (52843029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31182.086042] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [31182.096199] hardirqs last disabled at (52843030): [] sysvec_apic_timer_interrupt+0xe/0x90 [31182.102282] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [31182.112008] softirqs last enabled at (52843026): [] __do_softirq+0x2be/0x371 [31182.112012] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [31182.128084] softirqs last disabled at (52843021): [] irq_exit_rcu+0xb0/0xd0 [31182.135642] x86_pkg_temp_thermal coretemp kvm_intel kvm [31182.144154] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31182.149457] irqbypass i2c_i801 [31182.158402] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31182.158403] iTCO_wdt iTCO_vendor_support [31182.161540] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31182.169008] i2c_smbus crct10dif_pclmul [31182.173015] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31182.178491] joydev crc32_pclmul [31182.182320] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31182.201058] ipmi_ssif [31182.204282] [31182.204283] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31182.209502] ghash_clmulni_intel [31182.211859] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31182.213350] mei_me mei [31182.220473] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31182.223695] lpc_ich sha512_ssse3 [31182.230820] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31182.233264] acpi_ipmi mfd_core [31182.240390] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31182.243700] wmi [31182.250824] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31182.253960] ipmi_si ipmi_devintf [31182.261085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31182.262923] ipmi_msghandler ioatdma [31182.271000] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31182.274310] acpi_pad acpi_power_meter [31182.280048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31182.283619] ixgbe igb [31182.290742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31182.294503] mdio crc32c_intel [31182.301627] Call Trace: [31182.303987] nvme ptp [31182.311112] [31182.314162] nvme_core [31182.316607] ? watchdog_timer_fn+0x1b4/0x220 [31182.318877] i2c_algo_bit [31182.320888] ? __hrtimer_run_queues+0x1d3/0x360 [31182.323244] pps_core dca [31182.327510] ? hrtimer_interrupt+0x104/0x240 [31182.330125] fuse [31182.334649] ? __local_bh_enable+0x50/0x60 [31182.337267] irq event stamp: 170771500 [31182.341534] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31182.343455] hardirqs last enabled at (170771499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31182.347546] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31182.351289] hardirqs last disabled at (170771500): [] sysvec_apic_timer_interrupt+0xe/0x90 [31182.356507] [31182.366768] softirqs last enabled at (170771498): [] __do_softirq+0x2be/0x371 [31182.371727] [31182.381536] softirqs last disabled at (170771493): [] irq_exit_rcu+0xb0/0xd0 [31182.383634] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31182.392406] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31182.394505] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31182.403099] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31182.408405] do_raw_spin_lock+0xb5/0xc0 [31182.417783] Workqueue: events delayed_work [ceph] [31182.422654] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31182.430122] [31182.430123] RIP: 0010:lock_acquire+0xf0/0x2b0 [31182.433958] try_get_cap_refs+0x159/0x880 [ceph] [31182.438651] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31182.443440] __ceph_get_caps+0x4a1/0x760 [ceph] [31182.444925] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31182.449280] ? __file_remove_privs+0x89/0x110 [31182.453887] [31182.453887] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31182.472626] ? wake_bit_function+0x70/0x70 [31182.477148] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31182.482369] ? ceph_write_iter+0x260/0x770 [ceph] [31182.486716] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31182.488208] ceph_write_iter+0x260/0x770 [ceph] [31182.495332] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31182.499426] ? lock_is_held_type+0xaa/0x120 [31182.506547] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31182.506548] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31182.511248] vfs_write+0x324/0x4f0 [31182.518369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31182.522899] ksys_pwrite64+0x65/0xa0 [31182.530018] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31182.534194] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31182.541317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31182.549397] do_syscall_64+0x38/0x80 [31182.552794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31182.558532] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31182.562102] Call Trace: [31182.562104] [31182.569227] RIP: 0033:0x7f48f0eeb93b [31182.574358] ? watchdog_timer_fn+0x1b4/0x220 [31182.581483] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31182.585065] ? __hrtimer_run_queues+0x1d3/0x360 [31182.592184] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31182.597233] ? hrtimer_interrupt+0x104/0x240 [31182.599672] ORIG_RAX: 0000000000000012 [31182.601683] ? __local_bh_enable+0x50/0x60 [31182.605254] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31182.609520] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31182.628256] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31182.632781] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31182.637995] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31182.642261] [31182.646090] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31182.650182] [31182.657307] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31182.662525] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31182.669654] [31182.674610] ? lock_acquire+0xf0/0x2b0 [31182.711415] ? igrab+0x43/0x50 [31182.714467] ? lock_release+0xc5/0x260 [31182.718210] _raw_spin_lock+0x30/0x40 [31182.721876] ? igrab+0x1d/0x50 [31182.724926] igrab+0x1d/0x50 [31182.727805] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31182.732954] delayed_work+0x1d8/0x280 [ceph] [31182.737234] ? process_scheduled_works+0x22a/0x510 [31182.742026] process_scheduled_works+0x2a4/0x510 [31182.746640] worker_thread+0x188/0x340 [31182.750389] ? rescuer_thread+0x310/0x310 [31182.754393] kthread+0xf5/0x130 [31182.757540] ? kthread_complete_and_exit+0x20/0x20 [31182.762324] ret_from_fork+0x31/0x40 [31182.765904] ? kthread_complete_and_exit+0x20/0x20 [31182.770687] ret_from_fork_asm+0x11/0x20 [31182.774617] [31210.005869] watchdog: BUG: soft lockup - CPU#0 stuck for 17952s! [dbench:78683] [31210.013174] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31210.017869] watchdog: BUG: soft lockup - CPU#7 stuck for 26949s! [kworker/7:4:78808] [31210.068622] irq event stamp: 52900998 [31210.068623] hardirqs last enabled at (52900997): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31210.076354] Modules linked in: [31210.080010] hardirqs last disabled at (52900998): [] sysvec_apic_timer_interrupt+0xe/0x90 [31210.090168] ceph [31210.093220] softirqs last enabled at (52900950): [] __do_softirq+0x2be/0x371 [31210.102944] libceph [31210.104868] softirqs last disabled at (52900945): [] irq_exit_rcu+0xb0/0xd0 [31210.113553] dns_resolver fscache [31210.115735] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31210.124262] netfs veth [31210.127574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31210.136520] nft_chain_nat xt_MASQUERADE [31210.138980] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31210.146452] nf_nat nf_conntrack [31210.150369] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31210.155847] nf_defrag_ipv6 nf_defrag_ipv4 [31210.159071] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31210.177808] nft_compat nf_tables nfnetlink [31210.181900] [31210.187116] bridge stp [31210.191292] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31210.192783] llc xfs [31210.195229] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31210.202353] libcrc32c sunrpc [31210.204553] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31210.211678] intel_rapl_msr intel_rapl_common [31210.214641] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31210.221765] sb_edac x86_pkg_temp_thermal [31210.226117] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31210.233242] coretemp kvm_intel [31210.237266] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31210.244387] kvm irqbypass [31210.247524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31210.255602] i2c_i801 iTCO_wdt [31210.258305] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31210.264043] iTCO_vendor_support [31210.267094] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31210.274218] i2c_smbus crct10dif_pclmul [31210.277443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31210.284565] joydev crc32_pclmul [31210.288396] Call Trace: [31210.295520] ipmi_ssif ghash_clmulni_intel [31210.298763] [31210.301207] mei_me mei [31210.305297] ? watchdog_timer_fn+0x1b4/0x220 [31210.307308] lpc_ich sha512_ssse3 [31210.309753] ? __hrtimer_run_queues+0x1d3/0x360 [31210.314017] acpi_ipmi [31210.317329] ? hrtimer_interrupt+0x104/0x240 [31210.321850] mfd_core [31210.324208] ? __local_bh_enable+0x50/0x60 [31210.328471] wmi ipmi_si [31210.330746] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31210.334833] ipmi_devintf ipmi_msghandler [31210.337363] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31210.342581] ioatdma acpi_pad [31210.346586] [31210.351549] acpi_power_meter ixgbe [31210.354516] [31210.354517] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31210.356612] igb mdio [31210.360102] ? queued_spin_lock_slowpath+0x16/0x2e0 [31210.362195] crc32c_intel nvme [31210.367501] do_raw_spin_lock+0xb5/0xc0 [31210.369770] ptp nvme_core [31210.374641] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31210.377691] i2c_algo_bit pps_core [31210.381525] try_get_cap_refs+0x159/0x880 [ceph] [31210.384226] dca [31210.389013] __ceph_get_caps+0x4a1/0x760 [ceph] [31210.392406] fuse [31210.397021] ? __file_remove_privs+0x89/0x110 [31210.398855] irq event stamp: 170993036 [31210.403381] ? wake_bit_function+0x70/0x70 [31210.405302] hardirqs last enabled at (170993035): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31210.409656] ? ceph_write_iter+0x260/0x770 [ceph] [31210.413398] hardirqs last disabled at (170993036): [] sysvec_apic_timer_interrupt+0xe/0x90 [31210.417489] ceph_write_iter+0x260/0x770 [ceph] [31210.427733] softirqs last enabled at (170993034): [] __do_softirq+0x2be/0x371 [31210.432433] ? lock_is_held_type+0xaa/0x120 [31210.442241] softirqs last disabled at (170993029): [] irq_exit_rcu+0xb0/0xd0 [31210.446768] vfs_write+0x324/0x4f0 [31210.455537] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31210.459736] ksys_pwrite64+0x65/0xa0 [31210.468330] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31210.471725] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31210.481104] Workqueue: events delayed_work [ceph] [31210.484676] do_syscall_64+0x38/0x80 [31210.492146] [31210.492147] RIP: 0010:lock_release+0x14e/0x260 [31210.497277] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31210.501974] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [31210.505560] RIP: 0033:0x7f48f0eeb93b [31210.507051] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [31210.511490] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31210.516548] [31210.516549] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [31210.535289] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31210.538859] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [31210.544077] ORIG_RAX: 0000000000000012 [31210.562813] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [31210.564305] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31210.571429] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [31210.576646] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31210.583771] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [31210.587602] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31210.594725] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31210.601848] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31210.608973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31210.616097] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31210.623221] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31210.630352] [31210.638424] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31210.674860] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31210.681982] Call Trace: [31210.684428] [31210.686440] ? watchdog_timer_fn+0x1b4/0x220 [31210.690704] ? __hrtimer_run_queues+0x1d3/0x360 [31210.695230] ? hrtimer_interrupt+0x104/0x240 [31210.699499] ? __local_bh_enable+0x50/0x60 [31210.703592] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31210.708816] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31210.713773] [31210.715871] [31210.717970] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31210.723282] ? igrab+0x43/0x50 [31210.726334] ? lock_release+0x14e/0x260 [31210.730163] ? lock_release+0xc5/0x260 [31210.733908] _raw_spin_unlock+0x1b/0x40 [31210.737740] igrab+0x43/0x50 [31210.740614] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31210.745765] delayed_work+0x1d8/0x280 [ceph] [31210.750046] ? process_scheduled_works+0x22a/0x510 [31210.754838] process_scheduled_works+0x2a4/0x510 [31210.759451] worker_thread+0x188/0x340 [31210.763200] ? rescuer_thread+0x310/0x310 [31210.767206] kthread+0xf5/0x130 [31210.770342] ? kthread_complete_and_exit+0x20/0x20 [31210.775126] ret_from_fork+0x31/0x40 [31210.778698] ? kthread_complete_and_exit+0x20/0x20 [31210.783482] ret_from_fork_asm+0x11/0x20 [31210.787403] [31238.006370] watchdog: BUG: soft lockup - CPU#0 stuck for 17978s! [dbench:78683] [31238.013678] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31238.018369] watchdog: BUG: soft lockup - CPU#7 stuck for 26975s! [kworker/7:4:78808] [31238.069129] irq event stamp: 52959110 [31238.076859] Modules linked in: ceph libceph dns_resolver fscache netfs [31238.080517] hardirqs last enabled at (52959109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31238.087045] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [31238.097200] hardirqs last disabled at (52959110): [] sysvec_apic_timer_interrupt+0xe/0x90 [31238.103284] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [31238.113009] softirqs last enabled at (52959074): [] __do_softirq+0x2be/0x371 [31238.120763] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [31238.129450] softirqs last disabled at (52959069): [] irq_exit_rcu+0xb0/0xd0 [31238.136670] x86_pkg_temp_thermal coretemp kvm_intel kvm [31238.145180] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31238.150484] irqbypass i2c_i801 [31238.159427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31238.159428] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31238.162565] iTCO_wdt [31238.170048] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31238.175522] iTCO_vendor_support [31238.177793] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31238.196531] i2c_smbus crct10dif_pclmul [31238.199766] [31238.199767] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31238.204981] joydev crc32_pclmul [31238.208811] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31238.210303] ipmi_ssif ghash_clmulni_intel [31238.217425] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31238.220649] mei_me mei [31238.227774] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31238.231865] lpc_ich sha512_ssse3 [31238.238990] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31238.241433] acpi_ipmi mfd_core [31238.248556] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31238.251868] wmi [31238.258991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31238.262130] ipmi_si ipmi_devintf [31238.270207] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31238.272046] ipmi_msghandler ioatdma [31238.277781] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31238.281091] acpi_pad acpi_power_meter [31238.288215] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31238.291788] ixgbe igb [31238.298910] Call Trace: [31238.298912] [31238.302655] mdio [31238.309782] ? watchdog_timer_fn+0x1b4/0x220 [31238.312137] crc32c_intel [31238.314581] ? __hrtimer_run_queues+0x1d3/0x360 [31238.316591] nvme ptp [31238.318520] ? hrtimer_interrupt+0x104/0x240 [31238.322781] nvme_core [31238.325397] ? __local_bh_enable+0x50/0x60 [31238.329922] i2c_algo_bit [31238.332194] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31238.336457] pps_core [31238.338815] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31238.342903] dca fuse [31238.345521] [31238.350740] [31238.350741] irq event stamp: 171214138 [31238.353009] [31238.353010] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31238.357966] hardirqs last enabled at (171214137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31238.360243] ? queued_spin_lock_slowpath+0x16/0x2e0 [31238.362335] hardirqs last disabled at (171214138): [] sysvec_apic_timer_interrupt+0xe/0x90 [31238.363829] do_raw_spin_lock+0xb5/0xc0 [31238.367571] softirqs last enabled at (171214136): [] __do_softirq+0x2be/0x371 [31238.369670] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31238.374972] softirqs last disabled at (171214131): [] irq_exit_rcu+0xb0/0xd0 [31238.385220] try_get_cap_refs+0x159/0x880 [ceph] [31238.390086] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31238.399903] __ceph_get_caps+0x4a1/0x760 [ceph] [31238.403728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31238.412503] ? __file_remove_privs+0x89/0x110 [31238.417282] Workqueue: events delayed_work [ceph] [31238.425882] ? wake_bit_function+0x70/0x70 [31238.430491] [31238.439872] ? ceph_write_iter+0x260/0x770 [ceph] [31238.444394] RIP: 0010:lock_release+0x14e/0x260 [31238.451864] ceph_write_iter+0x260/0x770 [ceph] [31238.456213] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [31238.460915] ? lock_is_held_type+0xaa/0x120 [31238.465003] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [31238.466495] vfs_write+0x324/0x4f0 [31238.471192] [31238.471192] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [31238.475635] ksys_pwrite64+0x65/0xa0 [31238.480151] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [31238.498891] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31238.503068] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [31238.508287] do_syscall_64+0x38/0x80 [31238.511683] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [31238.513175] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31238.520296] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [31238.523870] RIP: 0033:0x7f48f0eeb93b [31238.530993] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31238.536124] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31238.543247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31238.546817] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31238.553941] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31238.558984] ORIG_RAX: 0000000000000012 [31238.566111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31238.569681] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31238.577773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31238.596514] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31238.602251] Call Trace: [31238.607466] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31238.614594] [31238.618424] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31238.625547] ? watchdog_timer_fn+0x1b4/0x220 [31238.632670] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31238.639797] ? __hrtimer_run_queues+0x1d3/0x360 [31238.646924] [31238.649368] ? hrtimer_interrupt+0x104/0x240 [31238.687982] ? __local_bh_enable+0x50/0x60 [31238.692076] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31238.697300] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31238.702256] [31238.704353] [31238.706453] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31238.711773] ? igrab+0x43/0x50 [31238.714827] ? lock_release+0x14e/0x260 [31238.718666] ? lock_release+0xc5/0x260 [31238.722410] _raw_spin_unlock+0x1b/0x40 [31238.726249] igrab+0x43/0x50 [31238.729127] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31238.734275] delayed_work+0x1d8/0x280 [ceph] [31238.738556] ? process_scheduled_works+0x22a/0x510 [31238.743347] process_scheduled_works+0x2a4/0x510 [31238.747963] worker_thread+0x188/0x340 [31238.751712] ? rescuer_thread+0x310/0x310 [31238.755713] kthread+0xf5/0x130 [31238.758853] ? kthread_complete_and_exit+0x20/0x20 [31238.763638] ret_from_fork+0x31/0x40 [31238.767217] ? kthread_complete_and_exit+0x20/0x20 [31238.771999] ret_from_fork_asm+0x11/0x20 [31238.775923] [31266.006870] watchdog: BUG: soft lockup - CPU#0 stuck for 18004s! [dbench:78683] [31266.014173] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31266.018870] watchdog: BUG: soft lockup - CPU#7 stuck for 27001s! [kworker/7:4:78808] [31266.069616] irq event stamp: 53017138 [31266.069617] hardirqs last enabled at (53017137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31266.077347] Modules linked in: [31266.081003] hardirqs last disabled at (53017138): [] sysvec_apic_timer_interrupt+0xe/0x90 [31266.091161] ceph [31266.094213] softirqs last enabled at (53017118): [] __do_softirq+0x2be/0x371 [31266.103937] libceph dns_resolver [31266.105860] softirqs last disabled at (53017113): [] irq_exit_rcu+0xb0/0xd0 [31266.114545] fscache netfs [31266.117856] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31266.126365] veth nft_chain_nat [31266.129070] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31266.138013] xt_MASQUERADE nf_nat [31266.141152] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31266.148623] nf_conntrack nf_defrag_ipv6 [31266.151933] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31266.157411] nf_defrag_ipv4 nft_compat [31266.161329] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31266.180065] nf_tables nfnetlink [31266.183811] [31266.183812] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31266.189028] bridge stp [31266.192267] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31266.193759] llc xfs [31266.200883] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31266.203327] libcrc32c sunrpc [31266.210451] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31266.212634] intel_rapl_msr [31266.219759] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31266.222723] intel_rapl_common sb_edac [31266.229847] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31266.232639] x86_pkg_temp_thermal coretemp [31266.239763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31266.243507] kvm_intel kvm [31266.251583] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31266.255674] irqbypass i2c_i801 [31266.261413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31266.264116] iTCO_wdt iTCO_vendor_support [31266.271241] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31266.274395] i2c_smbus crct10dif_pclmul [31266.281520] Call Trace: [31266.281521] [31266.285524] joydev crc32_pclmul [31266.292666] ? watchdog_timer_fn+0x1b4/0x220 [31266.296494] ipmi_ssif ghash_clmulni_intel [31266.298941] ? __hrtimer_run_queues+0x1d3/0x360 [31266.300951] mei_me mei [31266.304177] ? hrtimer_interrupt+0x104/0x240 [31266.308440] lpc_ich sha512_ssse3 [31266.312545] ? __local_bh_enable+0x50/0x60 [31266.317070] acpi_ipmi mfd_core [31266.319517] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31266.323778] wmi ipmi_si [31266.327090] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31266.331180] ipmi_devintf ipmi_msghandler [31266.334317] [31266.339547] ioatdma acpi_pad [31266.342075] [31266.347033] acpi_power_meter ixgbe [31266.351036] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31266.353134] igb mdio [31266.356101] ? queued_spin_lock_slowpath+0x18/0x2e0 [31266.358195] crc32c_intel nvme ptp [31266.361682] do_raw_spin_lock+0xb5/0xc0 [31266.366983] nvme_core i2c_algo_bit [31266.369270] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31266.374142] pps_core dca [31266.377550] try_get_cap_refs+0x159/0x880 [ceph] [31266.381378] fuse [31266.384868] __ceph_get_caps+0x4a1/0x760 [ceph] [31266.389648] irq event stamp: 171435258 [31266.392274] ? __file_remove_privs+0x89/0x110 [31266.396884] hardirqs last enabled at (171435257): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31266.398807] ? wake_bit_function+0x70/0x70 [31266.403331] hardirqs last disabled at (171435258): [] sysvec_apic_timer_interrupt+0xe/0x90 [31266.407077] ? ceph_write_iter+0x260/0x770 [ceph] [31266.411425] softirqs last enabled at (171435256): [] __do_softirq+0x2be/0x371 [31266.421672] ceph_write_iter+0x260/0x770 [ceph] [31266.425760] softirqs last disabled at (171435251): [] irq_exit_rcu+0xb0/0xd0 [31266.435575] ? lock_is_held_type+0xaa/0x120 [31266.440271] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31266.449043] vfs_write+0x324/0x4f0 [31266.453569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31266.462175] ksys_pwrite64+0x65/0xa0 [31266.466348] Workqueue: events delayed_work [ceph] [31266.475725] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31266.479123] [31266.479124] RIP: 0010:lock_release+0x32/0x260 [31266.486595] do_syscall_64+0x38/0x80 [31266.490166] Code: 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e <89> c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 [31266.494862] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31266.499994] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [31266.501483] RIP: 0033:0x7f48f0eeb93b [31266.505833] [31266.505834] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [31266.509406] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31266.528142] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [31266.533187] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31266.538404] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [31266.541974] ORIG_RAX: 0000000000000012 [31266.543466] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [31266.550589] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31266.569328] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31266.576452] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31266.581669] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31266.588794] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31266.592624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31266.599748] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31266.606872] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31266.613997] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31266.621120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31266.629204] [31266.636322] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31266.679866] Call Trace: [31266.682309] [31266.684320] ? watchdog_timer_fn+0x1b4/0x220 [31266.688585] ? __hrtimer_run_queues+0x1d3/0x360 [31266.693110] ? hrtimer_interrupt+0x104/0x240 [31266.697372] ? __local_bh_enable+0x50/0x60 [31266.701466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31266.706690] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31266.711646] [31266.713744] [31266.715841] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31266.721146] ? igrab+0x43/0x50 [31266.724197] ? igrab+0x43/0x50 [31266.727245] ? lock_release+0x32/0x260 [31266.730992] ? lock_release+0xc5/0x260 [31266.734737] _raw_spin_unlock+0x1b/0x40 [31266.738575] igrab+0x43/0x50 [31266.741452] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31266.746602] delayed_work+0x1d8/0x280 [ceph] [31266.750883] ? process_scheduled_works+0x22a/0x510 [31266.755673] process_scheduled_works+0x2a4/0x510 [31266.760289] worker_thread+0x188/0x340 [31266.764038] ? rescuer_thread+0x310/0x310 [31266.768041] kthread+0xf5/0x130 [31266.771179] ? kthread_complete_and_exit+0x20/0x20 [31266.775964] ret_from_fork+0x31/0x40 [31266.779533] ? kthread_complete_and_exit+0x20/0x20 [31266.784334] ret_from_fork_asm+0x11/0x20 [31266.788276] [31294.007371] watchdog: BUG: soft lockup - CPU#0 stuck for 18030s! [dbench:78683] [31294.014672] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31294.019370] watchdog: BUG: soft lockup - CPU#7 stuck for 27027s! [kworker/7:4:78808] [31294.070120] irq event stamp: 53075260 [31294.077852] Modules linked in: ceph libceph dns_resolver fscache netfs [31294.081509] hardirqs last enabled at (53075259): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31294.088044] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [31294.098201] hardirqs last disabled at (53075260): [] sysvec_apic_timer_interrupt+0xe/0x90 [31294.105584] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [31294.115310] softirqs last enabled at (53075214): [] __do_softirq+0x2be/0x371 [31294.122096] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [31294.130779] softirqs last disabled at (53075209): [] irq_exit_rcu+0xb0/0xd0 [31294.137643] x86_pkg_temp_thermal coretemp kvm_intel kvm [31294.146157] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31294.151460] irqbypass i2c_i801 [31294.160403] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31294.160404] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31294.163541] iTCO_wdt [31294.171012] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31294.176488] iTCO_vendor_support [31294.178766] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31294.197507] i2c_smbus crct10dif_pclmul [31294.200729] [31294.200730] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31294.205947] joydev crc32_pclmul [31294.209777] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31294.211269] ipmi_ssif ghash_clmulni_intel [31294.218393] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31294.221619] mei_me [31294.228740] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31294.232848] mei lpc_ich [31294.239972] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31294.242070] sha512_ssse3 acpi_ipmi [31294.249195] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31294.251725] mfd_core [31294.258848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31294.262335] wmi ipmi_si [31294.270410] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31294.272681] ipmi_devintf ipmi_msghandler [31294.278420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31294.280949] ioatdma acpi_pad [31294.288075] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31294.292077] acpi_power_meter [31294.299201] Call Trace: [31294.299203] [31294.302166] ixgbe igb [31294.309294] ? watchdog_timer_fn+0x1b4/0x220 [31294.312255] mdio crc32c_intel [31294.314699] ? __hrtimer_run_queues+0x1d3/0x360 [31294.316709] nvme [31294.319070] ? hrtimer_interrupt+0x104/0x240 [31294.323331] ptp nvme_core [31294.326382] ? __local_bh_enable+0x50/0x60 [31294.330906] i2c_algo_bit [31294.332831] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31294.337094] pps_core dca [31294.339799] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31294.343888] fuse [31294.346507] [31294.351723] irq event stamp: 171656100 [31294.354340] [31294.359299] hardirqs last enabled at (171656099): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31294.361224] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31294.363320] hardirqs last disabled at (171656100): [] sysvec_apic_timer_interrupt+0xe/0x90 [31294.367067] ? queued_spin_lock_slowpath+0x18/0x2e0 [31294.369161] softirqs last enabled at (171656098): [] __do_softirq+0x2be/0x371 [31294.379406] do_raw_spin_lock+0xb5/0xc0 [31294.384710] softirqs last disabled at (171656093): [] irq_exit_rcu+0xb0/0xd0 [31294.394523] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31294.399391] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31294.408164] try_get_cap_refs+0x159/0x880 [ceph] [31294.411992] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31294.420595] __ceph_get_caps+0x4a1/0x760 [ceph] [31294.425375] Workqueue: events delayed_work [ceph] [31294.434774] ? __file_remove_privs+0x89/0x110 [31294.439380] [31294.439380] RIP: 0010:rcu_is_watching+0x29/0x40 [31294.446849] ? wake_bit_function+0x70/0x70 [31294.451375] Code: 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e <74> 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [31294.456073] ? ceph_write_iter+0x260/0x770 [ceph] [31294.460422] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [31294.461913] ceph_write_iter+0x260/0x770 [ceph] [31294.466437] [31294.466437] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 0000000000000000 [31294.470533] ? lock_is_held_type+0xaa/0x120 [31294.489264] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31294.493966] vfs_write+0x324/0x4f0 [31294.499180] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [31294.503709] ksys_pwrite64+0x65/0xa0 [31294.505196] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [31294.512320] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31294.516497] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31294.523622] do_syscall_64+0x38/0x80 [31294.527018] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31294.534144] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31294.537714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31294.544838] RIP: 0033:0x7f48f0eeb93b [31294.549968] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31294.557093] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31294.560663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31294.568741] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31294.573784] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31294.579523] ORIG_RAX: 0000000000000012 [31294.583092] Call Trace: [31294.590216] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31294.608955] [31294.616078] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31294.621298] ? watchdog_timer_fn+0x1b4/0x220 [31294.628420] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31294.632252] ? __hrtimer_run_queues+0x1d3/0x360 [31294.634695] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31294.641823] ? hrtimer_interrupt+0x104/0x240 [31294.643829] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31294.650955] ? __local_bh_enable+0x50/0x60 [31294.655223] [31294.662346] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31294.696872] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31294.701828] [31294.703927] [31294.706024] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31294.711331] ? rcu_is_watching+0x29/0x40 [31294.715254] lock_acquire+0x184/0x2b0 [31294.718913] ? igrab+0x43/0x50 [31294.721963] ? lock_release+0xc5/0x260 [31294.725708] _raw_spin_lock+0x30/0x40 [31294.729372] ? igrab+0x1d/0x50 [31294.732422] igrab+0x1d/0x50 [31294.735302] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31294.740452] delayed_work+0x1d8/0x280 [ceph] [31294.744733] ? process_scheduled_works+0x22a/0x510 [31294.749521] process_scheduled_works+0x2a4/0x510 [31294.754139] worker_thread+0x188/0x340 [31294.757886] ? rescuer_thread+0x310/0x310 [31294.761889] kthread+0xf5/0x130 [31294.765028] ? kthread_complete_and_exit+0x20/0x20 [31294.769812] ret_from_fork+0x31/0x40 [31294.773392] ? kthread_complete_and_exit+0x20/0x20 [31294.778175] ret_from_fork_asm+0x11/0x20 [31294.782108] [31322.007896] watchdog: BUG: soft lockup - CPU#0 stuck for 18056s! [dbench:78683] [31322.015194] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31322.019896] watchdog: BUG: soft lockup - CPU#7 stuck for 27053s! [kworker/7:4:78808] [31322.070634] irq event stamp: 53133248 [31322.070636] hardirqs last enabled at (53133247): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31322.078365] Modules linked in: [31322.082023] hardirqs last disabled at (53133248): [] sysvec_apic_timer_interrupt+0xe/0x90 [31322.092181] ceph [31322.095231] softirqs last enabled at (53133200): [] __do_softirq+0x2be/0x371 [31322.104956] libceph [31322.106880] softirqs last disabled at (53133195): [] irq_exit_rcu+0xb0/0xd0 [31322.115577] dns_resolver fscache [31322.117764] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31322.126290] netfs veth nft_chain_nat [31322.129603] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31322.138547] xt_MASQUERADE nf_nat [31322.142223] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31322.149693] nf_conntrack nf_defrag_ipv6 [31322.153003] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31322.158481] nf_defrag_ipv4 nft_compat [31322.162398] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31322.181137] nf_tables nfnetlink [31322.184881] [31322.184882] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31322.190096] bridge stp [31322.193321] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31322.194812] llc xfs [31322.201937] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31322.204381] libcrc32c [31322.211506] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31322.213689] sunrpc intel_rapl_msr [31322.220813] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31322.223170] intel_rapl_common [31322.230295] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31322.233692] sb_edac x86_pkg_temp_thermal [31322.240817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31322.243866] coretemp kvm_intel [31322.251943] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31322.255947] kvm irqbypass [31322.261687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31322.264824] i2c_i801 iTCO_wdt [31322.271946] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31322.274651] iTCO_vendor_support [31322.281775] Call Trace: [31322.281776] [31322.284827] i2c_smbus crct10dif_pclmul [31322.291951] ? watchdog_timer_fn+0x1b4/0x220 [31322.295174] joydev crc32_pclmul [31322.297620] ? __hrtimer_run_queues+0x1d3/0x360 [31322.299628] ipmi_ssif ghash_clmulni_intel [31322.303464] ? hrtimer_interrupt+0x104/0x240 [31322.307723] mei_me mei [31322.310948] ? __local_bh_enable+0x50/0x60 [31322.315472] lpc_ich sha512_ssse3 [31322.319566] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31322.323827] acpi_ipmi mfd_core [31322.326272] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31322.330379] wmi ipmi_si [31322.333691] [31322.338907] ipmi_devintf ipmi_msghandler [31322.342046] [31322.347001] ioatdma acpi_pad [31322.349533] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31322.351629] acpi_power_meter ixgbe [31322.355639] ? queued_spin_lock_slowpath+0x18/0x2e0 [31322.357731] igb mdio [31322.360699] do_raw_spin_lock+0xb5/0xc0 [31322.366000] crc32c_intel [31322.369486] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31322.374355] nvme ptp [31322.376628] try_get_cap_refs+0x159/0x880 [ceph] [31322.380456] nvme_core i2c_algo_bit [31322.383078] __ceph_get_caps+0x4a1/0x760 [ceph] [31322.387857] pps_core [31322.390133] ? __file_remove_privs+0x89/0x110 [31322.394739] dca fuse [31322.398224] ? wake_bit_function+0x70/0x70 [31322.402747] [31322.402748] irq event stamp: 171876832 [31322.405019] ? ceph_write_iter+0x260/0x770 [ceph] [31322.409369] hardirqs last enabled at (171876831): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31322.411639] ceph_write_iter+0x260/0x770 [ceph] [31322.415730] hardirqs last disabled at (171876832): [] sysvec_apic_timer_interrupt+0xe/0x90 [31322.417225] ? lock_is_held_type+0xaa/0x120 [31322.420964] softirqs last enabled at (171876830): [] __do_softirq+0x2be/0x371 [31322.425666] vfs_write+0x324/0x4f0 [31322.435906] softirqs last disabled at (171876825): [] irq_exit_rcu+0xb0/0xd0 [31322.440437] ksys_pwrite64+0x65/0xa0 [31322.450242] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31322.454417] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31322.463190] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31322.466590] do_syscall_64+0x38/0x80 [31322.475186] Workqueue: events delayed_work [ceph] [31322.478757] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31322.488133] [31322.488134] RIP: 0010:lock_release+0x1f/0x260 [31322.493264] RIP: 0033:0x7f48f0eeb93b [31322.500733] Code: a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 <48> 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 [31322.504306] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31322.509001] RSP: 0018:ffffc90002117d48 EFLAGS: 00000296 [31322.514046] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31322.515538] [31322.515539] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 000000005553a634 [31322.519889] ORIG_RAX: 0000000000000012 [31322.523458] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [31322.542197] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31322.560934] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [31322.566150] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31322.571368] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [31322.572860] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31322.579983] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31322.583814] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31322.590940] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31322.598062] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31322.605188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31322.612316] [31322.619435] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31322.671056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31322.678180] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31322.685303] Call Trace: [31322.687748] [31322.689758] ? watchdog_timer_fn+0x1b4/0x220 [31322.694023] ? __hrtimer_run_queues+0x1d3/0x360 [31322.698550] ? hrtimer_interrupt+0x104/0x240 [31322.702820] ? __local_bh_enable+0x50/0x60 [31322.706911] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31322.712128] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31322.717084] [31322.719183] [31322.721298] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31322.726601] ? igrab+0x43/0x50 [31322.729654] ? igrab+0x43/0x50 [31322.732704] ? lock_release+0x1f/0x260 [31322.736447] ? lock_release+0xc5/0x260 [31322.740192] _raw_spin_unlock+0x1b/0x40 [31322.744023] igrab+0x43/0x50 [31322.746901] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31322.752049] delayed_work+0x1d8/0x280 [ceph] [31322.756330] ? process_scheduled_works+0x22a/0x510 [31322.761121] process_scheduled_works+0x2a4/0x510 [31322.765735] worker_thread+0x188/0x340 [31322.769484] ? rescuer_thread+0x310/0x310 [31322.773489] kthread+0xf5/0x130 [31322.776626] ? kthread_complete_and_exit+0x20/0x20 [31322.781410] ret_from_fork+0x31/0x40 [31322.784982] ? kthread_complete_and_exit+0x20/0x20 [31322.789764] ret_from_fork_asm+0x11/0x20 [31322.793687] [31349.452806] rcu: INFO: rcu_preempt self-detected stall on CPU [31349.458570] rcu: 0-....: (18975716 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9494940 [31349.468994] rcu: (t=19412498 jiffies g=1309609 q=563533 ncpus=8) [31349.475079] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31349.484023] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31349.491494] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [31349.496979] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [31349.515718] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31349.520935] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31349.528059] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31349.535183] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31349.542307] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31349.549431] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31349.556556] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31349.564633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31349.570371] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31349.577494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31349.584620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31349.591741] Call Trace: [31349.594186] [31349.596199] ? rcu_dump_cpu_stacks+0xd1/0x180 [31349.600559] ? rcu_sched_clock_irq+0x633/0x1390 [31349.605082] ? lock_is_held_type+0xaa/0x120 [31349.609259] ? lock_is_held_type+0xaa/0x120 [31349.613436] ? tick_sched_do_timer+0xa0/0xa0 [31349.617708] ? rcu_read_lock_sched_held+0x56/0x90 [31349.622409] ? tick_sched_do_timer+0xa0/0xa0 [31349.626679] ? update_process_times+0x5f/0x90 [31349.631031] ? tick_sched_handle.isra.16+0x1e/0x50 [31349.635813] ? tick_sched_timer+0x6f/0x80 [31349.639839] ? __hrtimer_run_queues+0x1d3/0x360 [31349.644372] ? hrtimer_interrupt+0x104/0x240 [31349.648640] ? __local_bh_enable+0x50/0x60 [31349.652733] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31349.657957] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31349.662916] [31349.665014] [31349.667125] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31349.672434] ? queued_spin_lock_slowpath+0x2b/0x2e0 [31349.677313] do_raw_spin_lock+0xb5/0xc0 [31349.681142] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31349.685944] try_get_cap_refs+0x159/0x880 [ceph] [31349.690575] __ceph_get_caps+0x4a1/0x760 [ceph] [31349.695127] ? __file_remove_privs+0x89/0x110 [31349.699481] ? wake_bit_function+0x70/0x70 [31349.703573] ? ceph_write_iter+0x260/0x770 [ceph] [31349.708285] ceph_write_iter+0x260/0x770 [ceph] [31349.712841] ? lock_is_held_type+0xaa/0x120 [31349.717025] vfs_write+0x324/0x4f0 [31349.720432] ksys_pwrite64+0x65/0xa0 [31349.724006] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31349.729139] do_syscall_64+0x38/0x80 [31349.732709] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31349.737754] RIP: 0033:0x7f48f0eeb93b [31349.741323] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31349.760062] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [31349.767619] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31349.774742] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31349.781866] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31349.788992] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31349.796121] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31349.803255] [31349.805442] Sending NMI from CPU 0 to CPUs 7: [31349.809793] NMI backtrace for cpu 7 [31349.809795] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31349.809797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31349.809798] Workqueue: events delayed_work [ceph] [31349.809836] RIP: 0010:lock_acquire+0xc4/0x2b0 [31349.809839] Code: 6a 00 81 e3 00 02 00 00 41 89 e8 44 89 e9 6a 00 41 0f 94 c1 44 89 f2 44 89 fe 6a 00 45 0f b6 c9 4c 89 e7 ff b4 24 88 00 00 00 <41> 52 e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 [31349.809841] RSP: 0018:ffffc90002117cf8 EFLAGS: 00000006 [31349.809843] RAX: 0000000000000246 RBX: 0000000000000200 RCX: 0000000000000000 [31349.809844] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31349.809845] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [31349.809846] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [31349.809847] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31349.809849] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31349.809850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31349.809852] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31349.809853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31349.809854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31349.809856] Call Trace: [31349.809857] [31349.809858] ? nmi_cpu_backtrace+0xad/0x120 [31349.809863] ? nmi_cpu_backtrace_handler+0x11/0x20 [31349.809865] ? nmi_handle+0xd4/0x260 [31349.809871] ? default_do_nmi+0x49/0xf0 [31349.809873] ? exc_nmi+0x11e/0x150 [31349.809875] ? end_repeat_nmi+0x16/0x67 [31349.809881] ? lock_acquire+0xc4/0x2b0 [31349.809884] ? lock_acquire+0xc4/0x2b0 [31349.809887] ? lock_acquire+0xc4/0x2b0 [31349.809889] [31349.809890] [31349.809890] ? igrab+0x1d/0x50 [31349.809893] ? igrab+0x43/0x50 [31349.809895] ? lock_release+0xc5/0x260 [31349.809898] _raw_spin_lock+0x30/0x40 [31349.809901] ? igrab+0x1d/0x50 [31349.809902] igrab+0x1d/0x50 [31349.809905] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31349.809925] delayed_work+0x1d8/0x280 [ceph] [31349.809945] ? process_scheduled_works+0x22a/0x510 [31349.809948] process_scheduled_works+0x2a4/0x510 [31349.809954] worker_thread+0x188/0x340 [31349.809957] ? rescuer_thread+0x310/0x310 [31349.809960] kthread+0xf5/0x130 [31349.809963] ? kthread_complete_and_exit+0x20/0x20 [31349.809967] ret_from_fork+0x31/0x40 [31349.809969] ? kthread_complete_and_exit+0x20/0x20 [31349.809972] ret_from_fork_asm+0x11/0x20 [31349.809979] [31374.008928] watchdog: BUG: soft lockup - CPU#0 stuck for 18105s! [dbench:78683] [31374.016230] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31374.020927] watchdog: BUG: soft lockup - CPU#7 stuck for 27102s! [kworker/7:4:78808] [31374.071679] irq event stamp: 53241510 [31374.079410] Modules linked in: ceph libceph dns_resolver fscache [31374.083067] hardirqs last enabled at (53241509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31374.089073] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [31374.099229] hardirqs last disabled at (53241510): [] sysvec_apic_timer_interrupt+0xe/0x90 [31374.104709] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [31374.114431] softirqs last enabled at (53241508): [] __do_softirq+0x2be/0x371 [31374.114435] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [31374.131022] softirqs last disabled at (53241503): [] irq_exit_rcu+0xb0/0xd0 [31374.131025] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [31374.145452] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31374.145455] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31374.152661] kvm irqbypass [31374.161605] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31374.169076] i2c_i801 [31374.171781] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31374.177258] iTCO_wdt iTCO_vendor_support [31374.179529] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31374.198267] i2c_smbus crct10dif_pclmul [31374.202269] [31374.207489] joydev crc32_pclmul [31374.211325] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31374.212818] ipmi_ssif ghash_clmulni_intel [31374.216043] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31374.223167] mei_me mei [31374.227256] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31374.234381] lpc_ich sha512_ssse3 [31374.236824] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31374.243949] acpi_ipmi mfd_core [31374.247259] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31374.254384] wmi ipmi_si [31374.257521] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31374.264646] ipmi_devintf ipmi_msghandler [31374.267176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31374.275254] ioatdma acpi_pad [31374.279258] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31374.284995] acpi_power_meter [31374.287959] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31374.295083] ixgbe [31374.298048] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31374.305171] igb mdio [31374.307182] Call Trace: [31374.314323] crc32c_intel [31374.316608] [31374.319047] nvme [31374.321666] ? watchdog_timer_fn+0x1b4/0x220 [31374.323674] ptp nvme_core [31374.325601] ? __hrtimer_run_queues+0x1d3/0x360 [31374.329880] i2c_algo_bit pps_core [31374.332587] ? hrtimer_interrupt+0x104/0x240 [31374.337109] dca fuse [31374.340505] ? __local_bh_enable+0x50/0x60 [31374.344769] [31374.344770] irq event stamp: 172290348 [31374.347043] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31374.351133] hardirqs last enabled at (172290347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31374.352624] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31374.356366] hardirqs last disabled at (172290348): [] sysvec_apic_timer_interrupt+0xe/0x90 [31374.361585] [31374.371845] softirqs last enabled at (172290346): [] __do_softirq+0x2be/0x371 [31374.376803] [31374.376804] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31374.386616] softirqs last disabled at (172290341): [] irq_exit_rcu+0xb0/0xd0 [31374.388716] ? queued_spin_lock_slowpath+0x18/0x2e0 [31374.397482] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31374.399582] do_raw_spin_lock+0xb5/0xc0 [31374.404884] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31374.413483] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31374.418353] Workqueue: events delayed_work [ceph] [31374.427734] try_get_cap_refs+0x159/0x880 [ceph] [31374.431559] [31374.431560] RIP: 0010:lock_acquire+0xf0/0x2b0 [31374.439036] __ceph_get_caps+0x4a1/0x760 [ceph] [31374.443815] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31374.448516] ? __file_remove_privs+0x89/0x110 [31374.453122] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31374.454613] ? wake_bit_function+0x70/0x70 [31374.458964] [31374.458965] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31374.463491] ? ceph_write_iter+0x260/0x770 [ceph] [31374.482226] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31374.486577] ceph_write_iter+0x260/0x770 [ceph] [31374.491795] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31374.495890] ? lock_is_held_type+0xaa/0x120 [31374.497376] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31374.504504] vfs_write+0x324/0x4f0 [31374.509197] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31374.516334] ksys_pwrite64+0x65/0xa0 [31374.520855] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31374.527980] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31374.532156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31374.539281] do_syscall_64+0x38/0x80 [31374.542676] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31374.549802] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31374.553373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31374.561452] RIP: 0033:0x7f48f0eeb93b [31374.566580] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31374.572331] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31374.575897] Call Trace: [31374.575898] [31374.583021] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31374.588066] ? watchdog_timer_fn+0x1b4/0x220 [31374.595191] ORIG_RAX: 0000000000000012 [31374.598762] ? __hrtimer_run_queues+0x1d3/0x360 [31374.605885] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31374.624626] ? hrtimer_interrupt+0x104/0x240 [31374.627067] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31374.629077] ? __local_bh_enable+0x50/0x60 [31374.634295] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31374.638563] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31374.642390] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31374.646914] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31374.654038] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31374.658302] [31374.665432] [31374.669517] [31374.669518] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31374.712752] ? lock_acquire+0xf0/0x2b0 [31374.716501] ? igrab+0x43/0x50 [31374.719552] ? lock_release+0xc5/0x260 [31374.723296] _raw_spin_lock+0x30/0x40 [31374.726953] ? igrab+0x1d/0x50 [31374.730004] igrab+0x1d/0x50 [31374.732882] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31374.738030] delayed_work+0x1d8/0x280 [ceph] [31374.742330] ? process_scheduled_works+0x22a/0x510 [31374.747120] process_scheduled_works+0x2a4/0x510 [31374.751735] worker_thread+0x188/0x340 [31374.755485] ? rescuer_thread+0x310/0x310 [31374.759497] kthread+0xf5/0x130 [31374.762642] ? kthread_complete_and_exit+0x20/0x20 [31374.767427] ret_from_fork+0x31/0x40 [31374.771006] ? kthread_complete_and_exit+0x20/0x20 [31374.775789] ret_from_fork_asm+0x11/0x20 [31374.779712] [31402.009477] watchdog: BUG: soft lockup - CPU#0 stuck for 18131s! [dbench:78683] [31402.016778] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31402.021477] watchdog: BUG: soft lockup - CPU#7 stuck for 27128s! [kworker/7:4:78808] [31402.072218] irq event stamp: 53299506 [31402.072220] hardirqs last enabled at (53299505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31402.079950] Modules linked in: [31402.083608] hardirqs last disabled at (53299506): [] sysvec_apic_timer_interrupt+0xe/0x90 [31402.093765] ceph [31402.096814] softirqs last enabled at (53299438): [] __do_softirq+0x2be/0x371 [31402.106538] libceph [31402.108463] softirqs last disabled at (53299433): [] irq_exit_rcu+0xb0/0xd0 [31402.117152] dns_resolver fscache [31402.119340] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31402.127850] netfs [31402.131179] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31402.140122] veth nft_chain_nat [31402.142136] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31402.149605] xt_MASQUERADE nf_nat [31402.152742] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31402.158220] nf_conntrack [31402.161529] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31402.180268] nf_defrag_ipv6 nf_defrag_ipv4 [31402.182884] [31402.182885] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31402.188102] nft_compat nf_tables [31402.192193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31402.193685] nfnetlink bridge [31402.200807] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31402.204118] stp llc [31402.211242] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31402.214206] xfs libcrc32c [31402.221330] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31402.223514] sunrpc intel_rapl_msr [31402.230639] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31402.233342] intel_rapl_common [31402.240467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31402.243876] sb_edac x86_pkg_temp_thermal [31402.251952] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31402.255001] coretemp kvm_intel [31402.260739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31402.264744] kvm irqbypass [31402.271876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31402.275013] i2c_i801 iTCO_wdt [31402.282153] Call Trace: [31402.282155] [31402.284873] iTCO_vendor_support i2c_smbus [31402.292001] ? watchdog_timer_fn+0x1b4/0x220 [31402.295050] crct10dif_pclmul [31402.297496] ? __hrtimer_run_queues+0x1d3/0x360 [31402.299506] joydev crc32_pclmul [31402.303601] ? hrtimer_interrupt+0x104/0x240 [31402.307875] ipmi_ssif [31402.310842] ? __local_bh_enable+0x50/0x60 [31402.315367] ghash_clmulni_intel [31402.318593] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31402.322855] mei_me mei [31402.325229] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31402.329321] lpc_ich sha512_ssse3 [31402.332545] [31402.337760] acpi_ipmi [31402.340205] [31402.345162] mfd_core wmi [31402.348475] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31402.350570] ipmi_si ipmi_devintf [31402.352931] ? queued_spin_lock_slowpath+0x18/0x2e0 [31402.355026] ipmi_msghandler [31402.357645] do_raw_spin_lock+0xb5/0xc0 [31402.362948] ioatdma [31402.366258] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31402.371128] acpi_pad [31402.374010] try_get_cap_refs+0x159/0x880 [ceph] [31402.377836] acpi_power_meter [31402.380025] __ceph_get_caps+0x4a1/0x760 [ceph] [31402.384805] ixgbe igb [31402.387079] ? __file_remove_privs+0x89/0x110 [31402.391685] mdio [31402.394651] ? wake_bit_function+0x70/0x70 [31402.399175] crc32c_intel [31402.401534] ? ceph_write_iter+0x260/0x770 [ceph] [31402.405882] nvme [31402.407807] ceph_write_iter+0x260/0x770 [ceph] [31402.411897] ptp [31402.414518] ? lock_is_held_type+0xaa/0x120 [31402.419211] nvme_core [31402.421139] vfs_write+0x324/0x4f0 [31402.425659] i2c_algo_bit pps_core [31402.427503] ksys_pwrite64+0x65/0xa0 [31402.431676] dca [31402.434033] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31402.437431] fuse [31402.440828] do_syscall_64+0x38/0x80 [31402.444398] irq event stamp: 172511420 [31402.446237] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31402.451365] hardirqs last enabled at (172511419): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31402.453291] RIP: 0033:0x7f48f0eeb93b [31402.456877] hardirqs last disabled at (172511420): [] sysvec_apic_timer_interrupt+0xe/0x90 [31402.460622] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31402.465666] softirqs last enabled at (172511418): [] __do_softirq+0x2be/0x371 [31402.475911] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31402.479482] softirqs last disabled at (172511413): [] irq_exit_rcu+0xb0/0xd0 [31402.489292] ORIG_RAX: 0000000000000012 [31402.508030] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31402.516800] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31402.522018] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31402.530617] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31402.534446] Workqueue: events delayed_work [ceph] [31402.543822] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31402.550948] [31402.558419] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31402.565543] RIP: 0010:lock_acquire+0xf0/0x2b0 [31402.570241] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31402.577364] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31402.578881] [31402.585996] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31402.623611] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31402.630735] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31402.637859] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31402.645001] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31402.652124] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31402.659249] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31402.667326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31402.673063] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31402.680189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31402.687312] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31402.694435] Call Trace: [31402.696883] [31402.698891] ? watchdog_timer_fn+0x1b4/0x220 [31402.703164] ? __hrtimer_run_queues+0x1d3/0x360 [31402.707690] ? hrtimer_interrupt+0x104/0x240 [31402.711953] ? __local_bh_enable+0x50/0x60 [31402.716046] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31402.721270] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31402.726227] [31402.728324] [31402.730421] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31402.735729] ? lock_acquire+0xf0/0x2b0 [31402.739480] ? igrab+0x43/0x50 [31402.742528] ? lock_release+0xc5/0x260 [31402.746274] _raw_spin_lock+0x30/0x40 [31402.749931] ? igrab+0x1d/0x50 [31402.752981] igrab+0x1d/0x50 [31402.755861] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31402.761036] delayed_work+0x1d8/0x280 [ceph] [31402.765315] ? process_scheduled_works+0x22a/0x510 [31402.770098] process_scheduled_works+0x2a4/0x510 [31402.774713] worker_thread+0x188/0x340 [31402.778463] ? rescuer_thread+0x310/0x310 [31402.782475] kthread+0xf5/0x130 [31402.785612] ? kthread_complete_and_exit+0x20/0x20 [31402.790398] ret_from_fork+0x31/0x40 [31402.793975] ? kthread_complete_and_exit+0x20/0x20 [31402.798759] ret_from_fork_asm+0x11/0x20 [31402.802682] [31430.010028] watchdog: BUG: soft lockup - CPU#0 stuck for 18157s! [dbench:78683] [31430.017335] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31430.022027] watchdog: BUG: soft lockup - CPU#7 stuck for 27154s! [kworker/7:4:78808] [31430.072783] irq event stamp: 53357686 [31430.072784] hardirqs last enabled at (53357685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31430.080516] Modules linked in: [31430.084172] hardirqs last disabled at (53357686): [] sysvec_apic_timer_interrupt+0xe/0x90 [31430.094330] ceph [31430.097380] softirqs last enabled at (53357660): [] __do_softirq+0x2be/0x371 [31430.107105] libceph [31430.109028] softirqs last disabled at (53357655): [] irq_exit_rcu+0xb0/0xd0 [31430.117713] dns_resolver fscache [31430.119897] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31430.128422] netfs veth [31430.131737] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31430.140679] nft_chain_nat [31430.143126] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31430.150595] xt_MASQUERADE nf_nat [31430.153300] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31430.158776] nf_conntrack nf_defrag_ipv6 [31430.162087] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31430.180824] nf_defrag_ipv4 nft_compat [31430.184743] [31430.184744] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31430.189959] nf_tables nfnetlink [31430.193706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31430.195195] bridge [31430.202318] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31430.205543] stp llc [31430.212668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31430.214764] xfs libcrc32c [31430.221888] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31430.224073] sunrpc [31430.231198] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31430.233900] intel_rapl_msr intel_rapl_common [31430.241025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31430.243122] sb_edac [31430.251199] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31430.255549] x86_pkg_temp_thermal coretemp [31430.261287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31430.263473] kvm_intel [31430.270594] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31430.274687] kvm irqbypass [31430.281811] Call Trace: [31430.281813] [31430.284168] i2c_i801 iTCO_wdt iTCO_vendor_support [31430.291294] ? watchdog_timer_fn+0x1b4/0x220 [31430.293995] i2c_smbus [31430.296441] ? __hrtimer_run_queues+0x1d3/0x360 [31430.298451] crct10dif_pclmul joydev [31430.303238] ? hrtimer_interrupt+0x104/0x240 [31430.307499] crc32_pclmul ipmi_ssif [31430.309856] ? __local_bh_enable+0x50/0x60 [31430.314380] ghash_clmulni_intel mei_me [31430.317953] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31430.322214] mei lpc_ich [31430.325715] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31430.329806] sha512_ssse3 acpi_ipmi [31430.333638] [31430.338857] mfd_core [31430.341387] [31430.341388] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31430.346343] wmi ipmi_si [31430.349832] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31430.351926] ipmi_devintf ipmi_msghandler [31430.354199] do_raw_spin_lock+0xb5/0xc0 [31430.356294] ioatdma acpi_pad [31430.361599] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31430.364127] acpi_power_meter ixgbe [31430.369003] try_get_cap_refs+0x159/0x880 [ceph] [31430.373004] igb mdio [31430.376839] __ceph_get_caps+0x4a1/0x760 [ceph] [31430.379798] crc32c_intel nvme [31430.384586] ? __file_remove_privs+0x89/0x110 [31430.388065] ptp [31430.392677] ? wake_bit_function+0x70/0x70 [31430.394947] nvme_core i2c_algo_bit [31430.399475] ? ceph_write_iter+0x260/0x770 [ceph] [31430.402523] pps_core dca [31430.406873] ceph_write_iter+0x260/0x770 [ceph] [31430.408726] fuse [31430.412822] ? lock_is_held_type+0xaa/0x120 [31430.416303] irq event stamp: 172732102 [31430.421002] vfs_write+0x324/0x4f0 [31430.423617] hardirqs last enabled at (172732101): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31430.428147] ksys_pwrite64+0x65/0xa0 [31430.430065] hardirqs last disabled at (172732102): [] sysvec_apic_timer_interrupt+0xe/0x90 [31430.434245] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31430.437987] softirqs last enabled at (172732100): [] __do_softirq+0x2be/0x371 [31430.441386] do_syscall_64+0x38/0x80 [31430.451629] softirqs last disabled at (172732091): [] irq_exit_rcu+0xb0/0xd0 [31430.455199] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31430.465011] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31430.470141] RIP: 0033:0x7f48f0eeb93b [31430.478911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31430.482482] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31430.491079] Workqueue: events delayed_work [ceph] [31430.496124] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31430.505501] [31430.509071] ORIG_RAX: 0000000000000012 [31430.516544] RIP: 0010:lock_acquire+0xf0/0x2b0 [31430.535281] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31430.539978] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31430.545196] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31430.546687] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31430.550518] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31430.554868] [31430.561991] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31430.580728] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31430.587853] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31430.593071] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31430.600202] [31430.601686] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31430.639490] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31430.646615] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31430.653739] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31430.661817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31430.667553] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31430.674677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31430.681801] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31430.688925] Call Trace: [31430.691372] [31430.693382] ? watchdog_timer_fn+0x1b4/0x220 [31430.697647] ? __hrtimer_run_queues+0x1d3/0x360 [31430.702174] ? hrtimer_interrupt+0x104/0x240 [31430.706443] ? __local_bh_enable+0x50/0x60 [31430.710535] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31430.715759] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31430.720717] [31430.722813] [31430.724912] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31430.730220] ? lock_acquire+0xf0/0x2b0 [31430.733969] ? igrab+0x43/0x50 [31430.737019] ? lock_release+0xc5/0x260 [31430.740765] _raw_spin_lock+0x30/0x40 [31430.744436] ? igrab+0x1d/0x50 [31430.747490] igrab+0x1d/0x50 [31430.750367] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31430.755515] delayed_work+0x1d8/0x280 [ceph] [31430.759798] ? process_scheduled_works+0x22a/0x510 [31430.764588] process_scheduled_works+0x2a4/0x510 [31430.769204] worker_thread+0x188/0x340 [31430.772952] ? rescuer_thread+0x310/0x310 [31430.776956] kthread+0xf5/0x130 [31430.780102] ? kthread_complete_and_exit+0x20/0x20 [31430.784887] ret_from_fork+0x31/0x40 [31430.788466] ? kthread_complete_and_exit+0x20/0x20 [31430.793250] ret_from_fork_asm+0x11/0x20 [31430.797181] [31458.010575] watchdog: BUG: soft lockup - CPU#0 stuck for 18183s! [dbench:78683] [31458.017874] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31458.022575] watchdog: BUG: soft lockup - CPU#7 stuck for 27180s! [kworker/7:4:78808] [31458.073315] irq event stamp: 53415800 [31458.073317] hardirqs last enabled at (53415799): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31458.081046] Modules linked in: [31458.084703] hardirqs last disabled at (53415800): [] sysvec_apic_timer_interrupt+0xe/0x90 [31458.094861] ceph libceph [31458.097912] softirqs last enabled at (53415732): [] __do_softirq+0x2be/0x371 [31458.107636] dns_resolver fscache [31458.110253] softirqs last disabled at (53415727): [] irq_exit_rcu+0xb0/0xd0 [31458.118937] netfs veth [31458.122264] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31458.130777] nft_chain_nat [31458.133221] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31458.142164] xt_MASQUERADE nf_nat [31458.144869] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31458.152340] nf_conntrack nf_defrag_ipv6 [31458.155650] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31458.161127] nf_defrag_ipv4 nft_compat [31458.165045] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31458.183783] nf_tables nfnetlink [31458.187526] [31458.187526] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31458.192743] bridge stp [31458.195972] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31458.197460] llc xfs [31458.204582] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31458.207028] libcrc32c sunrpc [31458.214151] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31458.216335] intel_rapl_msr intel_rapl_common [31458.223460] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31458.226423] sb_edac x86_pkg_temp_thermal [31458.233548] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31458.237899] coretemp kvm_intel [31458.245023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31458.249027] kvm irqbypass [31458.257105] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31458.260253] i2c_i801 iTCO_wdt [31458.265988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31458.268693] iTCO_vendor_support i2c_smbus [31458.275815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31458.278866] crct10dif_pclmul joydev [31458.285992] Call Trace: [31458.290082] crc32_pclmul ipmi_ssif ghash_clmulni_intel [31458.297206] [31458.300776] mei_me mei [31458.303223] ? watchdog_timer_fn+0x1b4/0x220 [31458.308437] lpc_ich sha512_ssse3 [31458.310448] ? __hrtimer_run_queues+0x1d3/0x360 [31458.312892] acpi_ipmi mfd_core [31458.317159] ? hrtimer_interrupt+0x104/0x240 [31458.320467] wmi ipmi_si [31458.324992] ? __local_bh_enable+0x50/0x60 [31458.328128] ipmi_devintf ipmi_msghandler [31458.332395] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31458.334923] ioatdma acpi_pad [31458.339016] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31458.343019] acpi_power_meter ixgbe [31458.348237] [31458.351201] igb mdio [31458.356158] [31458.359640] crc32c_intel nvme [31458.361739] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31458.364010] ptp nvme_core [31458.366112] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31458.369158] i2c_algo_bit pps_core [31458.374465] do_raw_spin_lock+0xb5/0xc0 [31458.377164] dca fuse [31458.382038] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31458.385433] [31458.385434] irq event stamp: 172953014 [31458.389267] try_get_cap_refs+0x159/0x880 [ceph] [31458.391534] hardirqs last enabled at (172953013): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31458.396324] __ceph_get_caps+0x4a1/0x760 [ceph] [31458.397809] hardirqs last disabled at (172953014): [] sysvec_apic_timer_interrupt+0xe/0x90 [31458.401558] ? __file_remove_privs+0x89/0x110 [31458.406165] softirqs last enabled at (172953012): [] __do_softirq+0x2be/0x371 [31458.416410] ? wake_bit_function+0x70/0x70 [31458.420932] softirqs last disabled at (172953007): [] irq_exit_rcu+0xb0/0xd0 [31458.430747] ? ceph_write_iter+0x260/0x770 [ceph] [31458.435094] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31458.443866] ceph_write_iter+0x260/0x770 [ceph] [31458.447973] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31458.456576] ? lock_is_held_type+0xaa/0x120 [31458.461268] Workqueue: events delayed_work [ceph] [31458.470648] vfs_write+0x324/0x4f0 [31458.475169] [31458.475170] RIP: 0010:lock_acquire+0xf0/0x2b0 [31458.482646] ksys_pwrite64+0x65/0xa0 [31458.486818] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31458.491515] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31458.494914] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31458.496404] do_syscall_64+0x38/0x80 [31458.500753] [31458.500754] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31458.504326] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31458.523064] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31458.528195] RIP: 0033:0x7f48f0eeb93b [31458.533412] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31458.536983] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31458.538472] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31458.545598] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31458.550640] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31458.557764] ORIG_RAX: 0000000000000012 [31458.561337] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31458.568460] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31458.587196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31458.594321] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31458.599538] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31458.606664] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31458.610495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31458.618571] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31458.625696] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31458.631434] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31458.638557] Call Trace: [31458.638559] [31458.645686] [31458.652807] ? watchdog_timer_fn+0x1b4/0x220 [31458.692207] ? __hrtimer_run_queues+0x1d3/0x360 [31458.696733] ? hrtimer_interrupt+0x104/0x240 [31458.701002] ? __local_bh_enable+0x50/0x60 [31458.705095] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31458.710319] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31458.715277] [31458.717374] [31458.719473] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31458.724780] ? lock_acquire+0xf0/0x2b0 [31458.728530] ? igrab+0x43/0x50 [31458.731580] ? lock_release+0xc5/0x260 [31458.735325] _raw_spin_lock+0x30/0x40 [31458.738989] ? igrab+0x1d/0x50 [31458.742039] igrab+0x1d/0x50 [31458.744919] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31458.750067] delayed_work+0x1d8/0x280 [ceph] [31458.754347] ? process_scheduled_works+0x22a/0x510 [31458.759131] process_scheduled_works+0x2a4/0x510 [31458.763745] worker_thread+0x188/0x340 [31458.767495] ? rescuer_thread+0x310/0x310 [31458.771497] kthread+0xf5/0x130 [31458.774636] ? kthread_complete_and_exit+0x20/0x20 [31458.779421] ret_from_fork+0x31/0x40 [31458.782998] ? kthread_complete_and_exit+0x20/0x20 [31458.787782] ret_from_fork_asm+0x11/0x20 [31458.791707] [31486.011121] watchdog: BUG: soft lockup - CPU#0 stuck for 18209s! [dbench:78683] [31486.018424] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31486.023121] watchdog: BUG: soft lockup - CPU#7 stuck for 27206s! [kworker/7:4:78808] [31486.073874] irq event stamp: 53473922 [31486.081602] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE [31486.085261] hardirqs last enabled at (53473921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31486.094648] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [31486.104804] hardirqs last disabled at (53473922): [] sysvec_apic_timer_interrupt+0xe/0x90 [31486.111582] nf_tables nfnetlink bridge stp llc xfs libcrc32c [31486.121306] softirqs last enabled at (53473906): [] __do_softirq+0x2be/0x371 [31486.127052] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [31486.135737] softirqs last disabled at (53473901): [] irq_exit_rcu+0xb0/0xd0 [31486.144862] kvm irqbypass i2c_i801 iTCO_wdt [31486.153373] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31486.153374] iTCO_vendor_support [31486.153376] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31486.157636] i2c_smbus [31486.166583] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31486.169806] crct10dif_pclmul joydev [31486.177278] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31486.179634] crc32_pclmul ipmi_ssif [31486.185111] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31486.188683] ghash_clmulni_intel mei_me [31486.207419] [31486.210905] mei lpc_ich [31486.216120] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31486.219952] sha512_ssse3 [31486.221442] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31486.223974] acpi_ipmi mfd_core [31486.231097] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31486.233716] wmi ipmi_si [31486.240839] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31486.243977] ipmi_devintf ipmi_msghandler [31486.251100] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31486.253630] ioatdma acpi_pad [31486.260756] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31486.264758] acpi_power_meter [31486.271882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31486.274847] ixgbe igb [31486.282924] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31486.285889] mdio crc32c_intel [31486.291626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31486.293985] nvme [31486.301108] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31486.304159] ptp [31486.311283] Call Trace: [31486.311284] [31486.313208] nvme_core i2c_algo_bit [31486.320333] ? watchdog_timer_fn+0x1b4/0x220 [31486.322167] pps_core dca [31486.324614] ? __hrtimer_run_queues+0x1d3/0x360 [31486.326624] fuse [31486.330111] ? hrtimer_interrupt+0x104/0x240 [31486.334371] irq event stamp: 173173818 [31486.336988] ? __local_bh_enable+0x50/0x60 [31486.341521] hardirqs last enabled at (173173817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31486.343449] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31486.347710] hardirqs last disabled at (173173818): [] sysvec_apic_timer_interrupt+0xe/0x90 [31486.351453] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31486.355544] softirqs last enabled at (173173816): [] __do_softirq+0x2be/0x371 [31486.365802] [31486.371023] softirqs last disabled at (173173811): [] irq_exit_rcu+0xb0/0xd0 [31486.380835] [31486.385804] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31486.394571] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31486.396669] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31486.405271] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31486.407364] Workqueue: events delayed_work [ceph] [31486.416744] do_raw_spin_lock+0xb5/0xc0 [31486.422045] [31486.429524] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31486.434395] RIP: 0010:lock_acquire+0x42/0x2b0 [31486.439094] try_get_cap_refs+0x159/0x880 [ceph] [31486.442924] Code: 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 <48> 0f a3 05 4e bd 53 01 0f 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 [31486.444419] __ceph_get_caps+0x4a1/0x760 [ceph] [31486.449198] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [31486.453553] ? __file_remove_privs+0x89/0x110 [31486.458159] [31486.458160] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [31486.476899] ? wake_bit_function+0x70/0x70 [31486.481422] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31486.486642] ? ceph_write_iter+0x260/0x770 [ceph] [31486.490990] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [31486.492481] ceph_write_iter+0x260/0x770 [ceph] [31486.499603] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [31486.503700] ? lock_is_held_type+0xaa/0x120 [31486.510820] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31486.515526] vfs_write+0x324/0x4f0 [31486.522650] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31486.527178] ksys_pwrite64+0x65/0xa0 [31486.534299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31486.538476] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31486.545598] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31486.548998] do_syscall_64+0x38/0x80 [31486.557074] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31486.560645] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31486.566381] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31486.571526] RIP: 0033:0x7f48f0eeb93b [31486.578644] Call Trace: [31486.578646] [31486.582217] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31486.589342] ? watchdog_timer_fn+0x1b4/0x220 [31486.594384] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31486.601525] ? __hrtimer_run_queues+0x1d3/0x360 [31486.605097] ORIG_RAX: 0000000000000012 [31486.607546] ? hrtimer_interrupt+0x104/0x240 [31486.609551] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31486.628288] ? __local_bh_enable+0x50/0x60 [31486.632553] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31486.637773] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31486.642295] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31486.646127] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31486.650389] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31486.657527] [31486.657528] [31486.661621] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31486.668746] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31486.673969] [31486.681092] ? lock_acquire+0x42/0x2b0 [31486.715739] ? igrab+0x43/0x50 [31486.718804] ? lock_release+0xc5/0x260 [31486.722552] _raw_spin_lock+0x30/0x40 [31486.726218] ? igrab+0x1d/0x50 [31486.729268] igrab+0x1d/0x50 [31486.732147] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31486.737297] delayed_work+0x1d8/0x280 [ceph] [31486.741576] ? process_scheduled_works+0x22a/0x510 [31486.746365] process_scheduled_works+0x2a4/0x510 [31486.750982] worker_thread+0x188/0x340 [31486.754732] ? rescuer_thread+0x310/0x310 [31486.758744] kthread+0xf5/0x130 [31486.761889] ? kthread_complete_and_exit+0x20/0x20 [31486.766676] ret_from_fork+0x31/0x40 [31486.770254] ? kthread_complete_and_exit+0x20/0x20 [31486.775036] ret_from_fork_asm+0x11/0x20 [31486.778960] [31514.011666] watchdog: BUG: soft lockup - CPU#0 stuck for 18235s! [dbench:78683] [31514.018972] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31514.023666] watchdog: BUG: soft lockup - CPU#7 stuck for 27232s! [kworker/7:4:78808] [31514.074414] irq event stamp: 53531906 [31514.074415] hardirqs last enabled at (53531905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31514.082143] Modules linked in: [31514.085801] hardirqs last disabled at (53531906): [] sysvec_apic_timer_interrupt+0xe/0x90 [31514.095957] ceph [31514.099009] softirqs last enabled at (53531846): [] __do_softirq+0x2be/0x371 [31514.108733] libceph [31514.110657] softirqs last disabled at (53531841): [] irq_exit_rcu+0xb0/0xd0 [31514.119351] dns_resolver fscache [31514.121533] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31514.130059] netfs veth [31514.133374] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31514.133375] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31514.142318] nft_chain_nat [31514.144777] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31514.152249] xt_MASQUERADE [31514.157726] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31514.160430] nf_nat nf_conntrack [31514.179168] [31514.179169] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31514.181873] nf_defrag_ipv6 nf_defrag_ipv4 [31514.187089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31514.190312] nft_compat nf_tables [31514.191804] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31514.198929] nfnetlink bridge [31514.203018] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31514.210142] stp [31514.213455] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31514.220579] llc xfs [31514.223543] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31514.230666] libcrc32c [31514.232505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31514.239627] sunrpc [31514.241812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31514.249890] intel_rapl_msr [31514.252248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31514.257983] intel_rapl_common [31514.260081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31514.267206] sb_edac x86_pkg_temp_thermal [31514.269997] Call Trace: [31514.269999] [31514.277122] coretemp [31514.280171] ? watchdog_timer_fn+0x1b4/0x220 [31514.287296] kvm_intel kvm [31514.291301] ? __hrtimer_run_queues+0x1d3/0x360 [31514.293744] irqbypass i2c_i801 [31514.295758] ? hrtimer_interrupt+0x104/0x240 [31514.298024] iTCO_wdt iTCO_vendor_support [31514.302290] ? __local_bh_enable+0x50/0x60 [31514.304993] i2c_smbus [31514.309519] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31514.312655] crct10dif_pclmul joydev [31514.316919] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31514.320922] crc32_pclmul [31514.325015] [31514.327371] ipmi_ssif [31514.332588] [31514.332590] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31514.336158] ghash_clmulni_intel mei_me [31514.341121] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31514.343733] mei lpc_ich [31514.345833] do_raw_spin_lock+0xb5/0xc0 [31514.348189] sha512_ssse3 acpi_ipmi [31514.350286] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31514.355590] mfd_core wmi [31514.359422] try_get_cap_refs+0x159/0x880 [ceph] [31514.364291] ipmi_si ipmi_devintf ipmi_msghandler [31514.366828] __ceph_get_caps+0x4a1/0x760 [ceph] [31514.370653] ioatdma acpi_pad [31514.374142] ? __file_remove_privs+0x89/0x110 [31514.378922] acpi_power_meter ixgbe [31514.381540] ? wake_bit_function+0x70/0x70 [31514.386150] igb mdio [31514.390850] ? ceph_write_iter+0x260/0x770 [ceph] [31514.395371] crc32c_intel nvme [31514.398335] ceph_write_iter+0x260/0x770 [ceph] [31514.402702] ptp nvme_core [31514.406191] ? lock_is_held_type+0xaa/0x120 [31514.410277] i2c_algo_bit [31514.412552] vfs_write+0x324/0x4f0 [31514.417245] pps_core dca [31514.420301] ksys_pwrite64+0x65/0xa0 [31514.424821] fuse [31514.427524] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31514.431702] irq event stamp: 173394974 [31514.434321] do_syscall_64+0x38/0x80 [31514.437717] hardirqs last enabled at (173394973): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31514.440347] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31514.443914] hardirqs last disabled at (173394974): [] sysvec_apic_timer_interrupt+0xe/0x90 [31514.445839] RIP: 0033:0x7f48f0eeb93b [31514.450969] softirqs last enabled at (173394972): [] __do_softirq+0x2be/0x371 [31514.454713] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31514.458283] softirqs last disabled at (173394967): [] irq_exit_rcu+0xb0/0xd0 [31514.468527] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31514.473571] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31514.483383] ORIG_RAX: 0000000000000012 [31514.486954] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31514.495724] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31514.514461] Workqueue: events delayed_work [ceph] [31514.523069] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31514.528285] [31514.537663] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31514.541493] RIP: 0010:lock_acquire+0xf0/0x2b0 [31514.548965] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31514.556088] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31514.560786] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31514.567909] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31514.569406] [31514.576525] [31514.576526] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31514.629878] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31514.637002] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31514.644127] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31514.651251] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31514.658374] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31514.666452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31514.672188] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31514.679314] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31514.686437] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31514.693562] Call Trace: [31514.696006] [31514.698017] ? watchdog_timer_fn+0x1b4/0x220 [31514.702282] ? __hrtimer_run_queues+0x1d3/0x360 [31514.706810] ? hrtimer_interrupt+0x104/0x240 [31514.711077] ? __local_bh_enable+0x50/0x60 [31514.715171] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31514.720396] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31514.725353] [31514.727450] [31514.729549] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31514.734864] ? lock_acquire+0xf0/0x2b0 [31514.738613] ? igrab+0x43/0x50 [31514.741664] ? lock_release+0xc5/0x260 [31514.745409] _raw_spin_lock+0x30/0x40 [31514.749073] ? igrab+0x1d/0x50 [31514.752125] igrab+0x1d/0x50 [31514.755002] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31514.760152] delayed_work+0x1d8/0x280 [ceph] [31514.764434] ? process_scheduled_works+0x22a/0x510 [31514.769223] process_scheduled_works+0x2a4/0x510 [31514.773838] worker_thread+0x188/0x340 [31514.777589] ? rescuer_thread+0x310/0x310 [31514.781601] kthread+0xf5/0x130 [31514.784736] ? kthread_complete_and_exit+0x20/0x20 [31514.789523] ret_from_fork+0x31/0x40 [31514.793093] ? kthread_complete_and_exit+0x20/0x20 [31514.797875] ret_from_fork_asm+0x11/0x20 [31514.801798] [31542.012208] watchdog: BUG: soft lockup - CPU#0 stuck for 18261s! [dbench:78683] [31542.019511] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31542.024207] watchdog: BUG: soft lockup - CPU#7 stuck for 27258s! [kworker/7:4:78808] [31542.074970] irq event stamp: 53590204 [31542.082700] Modules linked in: ceph libceph dns_resolver fscache [31542.086357] hardirqs last enabled at (53590203): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31542.092363] netfs veth nft_chain_nat xt_MASQUERADE [31542.102522] hardirqs last disabled at (53590204): [] sysvec_apic_timer_interrupt+0xe/0x90 [31542.107392] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [31542.117116] softirqs last enabled at (53590158): [] __do_softirq+0x2be/0x371 [31542.126241] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [31542.134926] softirqs last disabled at (53590153): [] irq_exit_rcu+0xb0/0xd0 [31542.134930] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [31542.150311] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31542.155960] kvm [31542.164906] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31542.164908] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31542.166742] irqbypass i2c_i801 [31542.174214] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31542.179691] iTCO_wdt iTCO_vendor_support [31542.182829] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31542.201565] i2c_smbus [31542.205569] [31542.205570] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31542.210788] crct10dif_pclmul joydev [31542.213145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31542.214635] crc32_pclmul [31542.221758] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31542.225331] ipmi_ssif [31542.232454] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31542.235072] ghash_clmulni_intel [31542.242197] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31542.244554] mei_me [31542.251678] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31542.254901] mei lpc_ich [31542.262027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31542.264123] sha512_ssse3 acpi_ipmi [31542.272201] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31542.274731] mfd_core wmi [31542.280468] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31542.283953] ipmi_si ipmi_devintf [31542.291077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31542.293695] ipmi_msghandler ioatdma [31542.300818] Call Trace: [31542.300821] [31542.304130] acpi_pad acpi_power_meter [31542.311256] ? watchdog_timer_fn+0x1b4/0x220 [31542.314824] ixgbe igb [31542.317268] ? __hrtimer_run_queues+0x1d3/0x360 [31542.319279] mdio crc32c_intel [31542.323025] ? hrtimer_interrupt+0x104/0x240 [31542.327288] nvme [31542.329644] ? __local_bh_enable+0x50/0x60 [31542.334167] ptp [31542.337221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31542.341484] nvme_core i2c_algo_bit [31542.343407] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31542.347497] pps_core dca [31542.349335] [31542.354553] fuse [31542.358036] [31542.358037] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31542.362994] irq event stamp: 173615850 [31542.365616] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31542.367708] hardirqs last enabled at (173615849): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31542.369635] do_raw_spin_lock+0xb5/0xc0 [31542.371729] hardirqs last disabled at (173615850): [] sysvec_apic_timer_interrupt+0xe/0x90 [31542.377035] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31542.380779] softirqs last enabled at (173615848): [] __do_softirq+0x2be/0x371 [31542.385652] try_get_cap_refs+0x159/0x880 [ceph] [31542.395897] softirqs last disabled at (173615843): [] irq_exit_rcu+0xb0/0xd0 [31542.399730] __ceph_get_caps+0x4a1/0x760 [ceph] [31542.409536] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31542.414324] ? __file_remove_privs+0x89/0x110 [31542.423090] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31542.427702] ? wake_bit_function+0x70/0x70 [31542.436298] Workqueue: events delayed_work [ceph] [31542.440826] ? ceph_write_iter+0x260/0x770 [ceph] [31542.450200] [31542.450201] RIP: 0010:lock_acquire+0xf0/0x2b0 [31542.454550] ceph_write_iter+0x260/0x770 [ceph] [31542.462021] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31542.466117] ? lock_is_held_type+0xaa/0x120 [31542.470810] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31542.475510] vfs_write+0x324/0x4f0 [31542.476997] [31542.476998] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31542.481353] ksys_pwrite64+0x65/0xa0 [31542.485889] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31542.504627] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31542.508804] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31542.514025] do_syscall_64+0x38/0x80 [31542.517419] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31542.518910] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31542.526034] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31542.529610] RIP: 0033:0x7f48f0eeb93b [31542.536729] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31542.541861] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31542.548985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31542.552554] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31542.559680] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31542.564723] ORIG_RAX: 0000000000000012 [31542.571848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31542.575417] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31542.583496] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31542.602233] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31542.607971] Call Trace: [31542.607972] [31542.613188] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31542.620313] ? watchdog_timer_fn+0x1b4/0x220 [31542.624143] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31542.631269] ? __hrtimer_run_queues+0x1d3/0x360 [31542.638392] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31542.645519] ? hrtimer_interrupt+0x104/0x240 [31542.652646] [31542.655083] ? __local_bh_enable+0x50/0x60 [31542.697806] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31542.703029] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31542.707987] [31542.710084] [31542.712181] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31542.717489] ? lock_acquire+0xf0/0x2b0 [31542.721238] ? igrab+0x43/0x50 [31542.724288] ? lock_release+0xc5/0x260 [31542.728034] _raw_spin_lock+0x30/0x40 [31542.731699] ? igrab+0x1d/0x50 [31542.734750] igrab+0x1d/0x50 [31542.737630] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31542.742778] delayed_work+0x1d8/0x280 [ceph] [31542.747058] ? process_scheduled_works+0x22a/0x510 [31542.751848] process_scheduled_works+0x2a4/0x510 [31542.756465] worker_thread+0x188/0x340 [31542.760214] ? rescuer_thread+0x310/0x310 [31542.764218] kthread+0xf5/0x130 [31542.767365] ? kthread_complete_and_exit+0x20/0x20 [31542.772157] ret_from_fork+0x31/0x40 [31542.775735] ? kthread_complete_and_exit+0x20/0x20 [31542.780520] ret_from_fork_asm+0x11/0x20 [31542.784443] [31544.816627] rcu: INFO: rcu_preempt self-detected stall on CPU [31544.822389] rcu: 0-....: (19166341 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9590754 [31544.832817] rcu: (t=19607858 jiffies g=1309609 q=570092 ncpus=8) [31544.838910] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31544.847868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31544.855339] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31544.860825] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31544.879563] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31544.884781] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31544.891906] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31544.899039] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31544.906162] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31544.913286] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31544.920411] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31544.928488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31544.934224] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31544.941349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31544.948474] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31544.955597] Call Trace: [31544.958041] [31544.960054] ? rcu_dump_cpu_stacks+0xd1/0x180 [31544.964413] ? rcu_sched_clock_irq+0x633/0x1390 [31544.968947] ? lock_is_held_type+0xaa/0x120 [31544.973133] ? lock_is_held_type+0xaa/0x120 [31544.977320] ? tick_sched_do_timer+0xa0/0xa0 [31544.981591] ? rcu_read_lock_sched_held+0x56/0x90 [31544.986301] ? tick_sched_do_timer+0xa0/0xa0 [31544.990568] ? update_process_times+0x5f/0x90 [31544.994922] ? tick_sched_handle.isra.16+0x1e/0x50 [31544.999732] ? tick_sched_timer+0x6f/0x80 [31545.003744] ? __hrtimer_run_queues+0x1d3/0x360 [31545.008281] ? hrtimer_interrupt+0x104/0x240 [31545.012549] ? __local_bh_enable+0x50/0x60 [31545.016665] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31545.021892] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31545.026857] [31545.028953] [31545.031052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31545.036365] ? queued_spin_lock_slowpath+0x16/0x2e0 [31545.041248] do_raw_spin_lock+0xb5/0xc0 [31545.045085] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31545.049888] try_get_cap_refs+0x159/0x880 [ceph] [31545.054532] __ceph_get_caps+0x4a1/0x760 [ceph] [31545.059079] ? __file_remove_privs+0x89/0x110 [31545.063433] ? wake_bit_function+0x70/0x70 [31545.067536] ? ceph_write_iter+0x260/0x770 [ceph] [31545.072255] ceph_write_iter+0x260/0x770 [ceph] [31545.076802] ? lock_is_held_type+0xaa/0x120 [31545.080987] vfs_write+0x324/0x4f0 [31545.084396] ksys_pwrite64+0x65/0xa0 [31545.087976] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31545.093116] do_syscall_64+0x38/0x80 [31545.096695] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31545.101747] RIP: 0033:0x7f48f0eeb93b [31545.105317] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31545.124056] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [31545.131620] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31545.138746] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31545.145868] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31545.152993] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31545.160117] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31545.167255] [31545.169444] Sending NMI from CPU 0 to CPUs 7: [31545.173796] NMI backtrace for cpu 7 [31545.173798] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31545.173800] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31545.173801] Workqueue: events delayed_work [ceph] [31545.173817] RIP: 0010:hlock_class+0x13/0x60 [31545.173821] Code: 82 48 89 c6 e8 2e 65 01 00 eb 9c e8 d7 ed a2 00 0f 1f 80 00 00 00 00 0f b7 47 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 6d c0 e0 01 <73> 13 48 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 c3 8b [31545.173822] RSP: 0018:ffffc90002117c58 EFLAGS: 00000003 [31545.173823] RAX: 0000000000000702 RBX: ffff88810debaa80 RCX: 0000000000000003 [31545.173824] RDX: 00000000000014df RSI: 0000000000000000 RDI: ffff88810debb528 [31545.173825] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [31545.173826] R10: 0000000000000000 R11: 0000000000000001 R12: b656a521d3820f2c [31545.173826] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [31545.173827] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31545.173828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31545.173829] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31545.173830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31545.173831] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31545.173831] Call Trace: [31545.173832] [31545.173833] ? nmi_cpu_backtrace+0xad/0x120 [31545.173837] ? nmi_cpu_backtrace_handler+0x11/0x20 [31545.173839] ? nmi_handle+0xd4/0x260 [31545.173843] ? default_do_nmi+0x49/0xf0 [31545.173844] ? exc_nmi+0x11e/0x150 [31545.173846] ? end_repeat_nmi+0x16/0x67 [31545.173850] ? hlock_class+0x13/0x60 [31545.173852] ? hlock_class+0x13/0x60 [31545.173855] ? hlock_class+0x13/0x60 [31545.173857] [31545.173858] [31545.173858] __lock_acquire+0xb15/0x1e20 [31545.173861] lock_acquire+0xcb/0x2b0 [31545.173863] ? igrab+0x1d/0x50 [31545.173864] ? igrab+0x43/0x50 [31545.173865] ? lock_release+0xc5/0x260 [31545.173867] _raw_spin_lock+0x30/0x40 [31545.173869] ? igrab+0x1d/0x50 [31545.173870] igrab+0x1d/0x50 [31545.173872] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31545.173888] delayed_work+0x1d8/0x280 [ceph] [31545.173904] ? process_scheduled_works+0x22a/0x510 [31545.173907] process_scheduled_works+0x2a4/0x510 [31545.173910] worker_thread+0x188/0x340 [31545.173913] ? rescuer_thread+0x310/0x310 [31545.173915] kthread+0xf5/0x130 [31545.173917] ? kthread_complete_and_exit+0x20/0x20 [31545.173920] ret_from_fork+0x31/0x40 [31545.173938] ? kthread_complete_and_exit+0x20/0x20 [31545.173940] ret_from_fork_asm+0x11/0x20 [31545.173944] [31570.012748] watchdog: BUG: soft lockup - CPU#0 stuck for 18287s! [dbench:78683] [31570.020051] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31570.024747] watchdog: BUG: soft lockup - CPU#7 stuck for 27284s! [kworker/7:4:78808] [31570.075492] irq event stamp: 53647520 [31570.075493] hardirqs last enabled at (53647519): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31570.083224] Modules linked in: ceph [31570.086881] hardirqs last disabled at (53647520): [] sysvec_apic_timer_interrupt+0xe/0x90 [31570.097037] libceph [31570.100522] softirqs last enabled at (53647500): [] __do_softirq+0x2be/0x371 [31570.110247] dns_resolver [31570.112430] softirqs last disabled at (53647495): [] irq_exit_rcu+0xb0/0xd0 [31570.121115] fscache netfs [31570.123749] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31570.132260] veth nft_chain_nat [31570.134963] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31570.143908] xt_MASQUERADE nf_nat [31570.147046] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [31570.154516] nf_conntrack [31570.157827] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [31570.163304] nf_defrag_ipv6 nf_defrag_ipv4 [31570.165920] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31570.184659] nft_compat nf_tables [31570.188750] [31570.188750] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31570.193967] nfnetlink bridge [31570.197279] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31570.198768] stp llc [31570.205893] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31570.208858] xfs libcrc32c [31570.215980] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31570.218166] sunrpc intel_rapl_msr [31570.225289] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31570.227994] intel_rapl_common [31570.235118] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31570.238533] sb_edac x86_pkg_temp_thermal [31570.245657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31570.248708] coretemp kvm_intel [31570.256785] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31570.260789] kvm irqbypass [31570.266527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31570.269664] i2c_i801 iTCO_wdt [31570.276788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31570.279492] iTCO_vendor_support i2c_smbus [31570.286615] Call Trace: [31570.286616] [31570.289666] crct10dif_pclmul joydev [31570.296791] ? watchdog_timer_fn+0x1b4/0x220 [31570.300883] crc32_pclmul ipmi_ssif [31570.303327] ? __hrtimer_run_queues+0x1d3/0x360 [31570.305337] ghash_clmulni_intel [31570.308909] ? hrtimer_interrupt+0x104/0x240 [31570.313171] mei_me mei [31570.316654] ? __local_bh_enable+0x50/0x60 [31570.321180] lpc_ich [31570.324406] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31570.328668] sha512_ssse3 acpi_ipmi [31570.331112] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31570.335202] mfd_core wmi [31570.337386] [31570.342603] ipmi_si ipmi_devintf [31570.346086] [31570.351044] ipmi_msghandler ioatdma [31570.353664] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31570.355761] acpi_pad acpi_power_meter [31570.359074] ? queued_spin_lock_slowpath+0x2b/0x2e0 [31570.361168] ixgbe igb [31570.364740] do_raw_spin_lock+0xb5/0xc0 [31570.370044] mdio crc32c_intel [31570.373788] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31570.378657] nvme ptp [31570.381018] try_get_cap_refs+0x159/0x880 [ceph] [31570.384845] nvme_core i2c_algo_bit [31570.387902] __ceph_get_caps+0x4a1/0x760 [ceph] [31570.392680] pps_core dca [31570.394956] ? __file_remove_privs+0x89/0x110 [31570.399562] fuse [31570.403048] ? wake_bit_function+0x70/0x70 [31570.407571] irq event stamp: 173835666 [31570.410190] ? ceph_write_iter+0x260/0x770 [ceph] [31570.414539] hardirqs last enabled at (173835665): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31570.416462] ceph_write_iter+0x260/0x770 [ceph] [31570.420554] hardirqs last disabled at (173835666): [] sysvec_apic_timer_interrupt+0xe/0x90 [31570.424302] ? lock_is_held_type+0xaa/0x120 [31570.428995] softirqs last enabled at (173835664): [] __do_softirq+0x2be/0x371 [31570.439242] vfs_write+0x324/0x4f0 [31570.443763] softirqs last disabled at (173835659): [] irq_exit_rcu+0xb0/0xd0 [31570.453578] ksys_pwrite64+0x65/0xa0 [31570.457751] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31570.466522] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31570.469919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31570.478519] do_syscall_64+0x38/0x80 [31570.482088] Workqueue: events delayed_work [ceph] [31570.491465] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31570.496596] [31570.496597] RIP: 0010:_raw_spin_lock+0x21/0x40 [31570.504066] RIP: 0033:0x7f48f0eeb93b [31570.507639] Code: cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 <41> b8 01 00 00 00 31 d2 31 f6 e8 30 0c 5c ff 48 89 df 58 5b e9 76 [31570.512335] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31570.517380] RSP: 0018:ffffc90002117d88 EFLAGS: 00000246 [31570.518869] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31570.523307] [31570.523308] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [31570.526877] ORIG_RAX: 0000000000000012 [31570.545615] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34518 [31570.564353] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31570.569571] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000000 [31570.574788] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31570.576278] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [31570.583402] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31570.587234] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31570.594356] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31570.601482] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31570.608605] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31570.615730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31570.622860] [31570.629979] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31570.674491] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31570.681615] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31570.688741] Call Trace: [31570.691184] [31570.693195] ? watchdog_timer_fn+0x1b4/0x220 [31570.697461] ? __hrtimer_run_queues+0x1d3/0x360 [31570.701985] ? hrtimer_interrupt+0x104/0x240 [31570.706246] ? __local_bh_enable+0x50/0x60 [31570.710340] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31570.715564] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31570.720522] [31570.722618] [31570.724715] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31570.730025] ? _raw_spin_lock+0x21/0x40 [31570.733861] ? igrab+0x1d/0x50 [31570.736911] igrab+0x1d/0x50 [31570.739788] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31570.744939] delayed_work+0x1d8/0x280 [ceph] [31570.749220] ? process_scheduled_works+0x22a/0x510 [31570.754010] process_scheduled_works+0x2a4/0x510 [31570.758624] worker_thread+0x188/0x340 [31570.762374] ? rescuer_thread+0x310/0x310 [31570.766378] kthread+0xf5/0x130 [31570.769516] ? kthread_complete_and_exit+0x20/0x20 [31570.774299] ret_from_fork+0x31/0x40 [31570.777870] ? kthread_complete_and_exit+0x20/0x20 [31570.782653] ret_from_fork_asm+0x11/0x20 [31570.786578] [31598.013268] watchdog: BUG: soft lockup - CPU#0 stuck for 18313s! [dbench:78683] [31598.020575] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31598.025267] watchdog: BUG: soft lockup - CPU#7 stuck for 27310s! [kworker/7:4:78808] [31598.076025] irq event stamp: 53705636 [31598.083753] Modules linked in: ceph libceph dns_resolver fscache netfs [31598.087413] hardirqs last enabled at (53705635): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31598.093941] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [31598.104094] hardirqs last disabled at (53705636): [] sysvec_apic_timer_interrupt+0xe/0x90 [31598.110179] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [31598.119904] softirqs last enabled at (53705578): [] __do_softirq+0x2be/0x371 [31598.127660] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [31598.136345] softirqs last disabled at (53705573): [] irq_exit_rcu+0xb0/0xd0 [31598.145383] coretemp kvm_intel kvm [31598.153895] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31598.153897] irqbypass [31598.153898] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31598.157379] i2c_i801 [31598.166324] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31598.168681] iTCO_wdt [31598.176152] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31598.178422] iTCO_vendor_support i2c_smbus [31598.183899] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31598.186169] crct10dif_pclmul joydev [31598.204907] [31598.208998] crc32_pclmul ipmi_ssif [31598.214215] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31598.217786] ghash_clmulni_intel mei_me [31598.219277] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31598.222762] mei lpc_ich [31598.229885] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31598.233715] sha512_ssse3 acpi_ipmi [31598.240841] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31598.243371] mfd_core wmi [31598.250494] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31598.253980] ipmi_si ipmi_devintf [31598.261102] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31598.263721] ipmi_msghandler ioatdma [31598.270844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31598.274155] acpi_pad acpi_power_meter [31598.282234] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31598.285805] ixgbe igb [31598.291541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31598.295286] mdio crc32c_intel [31598.302409] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31598.304767] nvme [31598.311891] Call Trace: [31598.311893] [31598.314957] ptp nvme_core [31598.322085] ? watchdog_timer_fn+0x1b4/0x220 [31598.324008] i2c_algo_bit [31598.326454] ? __hrtimer_run_queues+0x1d3/0x360 [31598.328461] pps_core [31598.331168] ? hrtimer_interrupt+0x104/0x240 [31598.335430] dca [31598.338047] ? __local_bh_enable+0x50/0x60 [31598.342572] fuse [31598.344845] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31598.349106] irq event stamp: 174056852 [31598.350948] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31598.355035] hardirqs last enabled at (174056851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31598.356958] [31598.362176] hardirqs last disabled at (174056852): [] sysvec_apic_timer_interrupt+0xe/0x90 [31598.365920] [31598.370895] softirqs last enabled at (174056850): [] __do_softirq+0x2be/0x371 [31598.381139] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31598.383236] softirqs last disabled at (174056845): [] irq_exit_rcu+0xb0/0xd0 [31598.393052] ? queued_spin_lock_slowpath+0x18/0x2e0 [31598.395145] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31598.403916] do_raw_spin_lock+0xb5/0xc0 [31598.409220] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31598.417819] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31598.422687] Workqueue: events delayed_work [ceph] [31598.432068] try_get_cap_refs+0x159/0x880 [ceph] [31598.435896] [31598.435897] RIP: 0010:lock_acquire+0xf0/0x2b0 [31598.443373] __ceph_get_caps+0x4a1/0x760 [ceph] [31598.448150] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31598.452852] ? __file_remove_privs+0x89/0x110 [31598.457459] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31598.458951] ? wake_bit_function+0x70/0x70 [31598.463301] [31598.463302] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31598.467826] ? ceph_write_iter+0x260/0x770 [ceph] [31598.486562] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31598.490912] ceph_write_iter+0x260/0x770 [ceph] [31598.496129] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31598.500225] ? lock_is_held_type+0xaa/0x120 [31598.501711] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31598.508840] vfs_write+0x324/0x4f0 [31598.513534] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31598.520677] ksys_pwrite64+0x65/0xa0 [31598.525199] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31598.532324] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31598.536501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31598.543626] do_syscall_64+0x38/0x80 [31598.547022] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31598.554148] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31598.557718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31598.565796] RIP: 0033:0x7f48f0eeb93b [31598.570925] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31598.576682] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31598.580251] Call Trace: [31598.580253] [31598.587375] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31598.592420] ? watchdog_timer_fn+0x1b4/0x220 [31598.599542] ORIG_RAX: 0000000000000012 [31598.603115] ? __hrtimer_run_queues+0x1d3/0x360 [31598.610237] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31598.628980] ? hrtimer_interrupt+0x104/0x240 [31598.631419] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31598.633430] ? __local_bh_enable+0x50/0x60 [31598.638647] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31598.642949] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31598.646778] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31598.651301] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31598.658424] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31598.662689] [31598.669820] [31598.673904] [31598.711832] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31598.717140] ? lock_acquire+0xf0/0x2b0 [31598.720889] ? igrab+0x43/0x50 [31598.723940] ? lock_release+0xc5/0x260 [31598.727702] _raw_spin_lock+0x30/0x40 [31598.731366] ? igrab+0x1d/0x50 [31598.734418] igrab+0x1d/0x50 [31598.737297] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31598.742445] delayed_work+0x1d8/0x280 [ceph] [31598.746727] ? process_scheduled_works+0x22a/0x510 [31598.751516] process_scheduled_works+0x2a4/0x510 [31598.756130] worker_thread+0x188/0x340 [31598.759881] ? rescuer_thread+0x310/0x310 [31598.763883] kthread+0xf5/0x130 [31598.767022] ? kthread_complete_and_exit+0x20/0x20 [31598.771806] ret_from_fork+0x31/0x40 [31598.775386] ? kthread_complete_and_exit+0x20/0x20 [31598.780169] ret_from_fork_asm+0x11/0x20 [31598.784100] [31626.013787] watchdog: BUG: soft lockup - CPU#0 stuck for 18340s! [dbench:78683] [31626.021087] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31626.025786] watchdog: BUG: soft lockup - CPU#7 stuck for 27337s! [kworker/7:4:78808] [31626.076529] irq event stamp: 53763676 [31626.076530] hardirqs last enabled at (53763675): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31626.084260] Modules linked in: [31626.087916] hardirqs last disabled at (53763676): [] sysvec_apic_timer_interrupt+0xe/0x90 [31626.098074] ceph [31626.101124] softirqs last enabled at (53763618): [] __do_softirq+0x2be/0x371 [31626.110849] libceph [31626.112773] softirqs last disabled at (53763613): [] irq_exit_rcu+0xb0/0xd0 [31626.121460] dns_resolver [31626.123643] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31626.132153] fscache netfs [31626.134786] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31626.143731] veth [31626.146435] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31626.153905] nft_chain_nat xt_MASQUERADE [31626.155829] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31626.161307] nf_nat nf_conntrack [31626.165224] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31626.183961] nf_defrag_ipv6 nf_defrag_ipv4 [31626.187188] [31626.192404] nft_compat [31626.196495] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31626.197986] nf_tables nfnetlink [31626.200429] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31626.207554] bridge stp [31626.210778] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31626.217901] llc xfs [31626.220347] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31626.227469] libcrc32c sunrpc [31626.229654] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31626.236777] intel_rapl_msr intel_rapl_common [31626.239743] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31626.246865] sb_edac x86_pkg_temp_thermal [31626.251216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31626.259295] coretemp kvm_intel [31626.263299] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31626.269035] kvm irqbypass [31626.272186] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31626.279307] i2c_i801 iTCO_wdt [31626.282011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31626.289136] iTCO_vendor_support [31626.292184] Call Trace: [31626.299308] i2c_smbus crct10dif_pclmul [31626.302534] [31626.304978] joydev crc32_pclmul [31626.308808] ? watchdog_timer_fn+0x1b4/0x220 [31626.310819] ipmi_ssif ghash_clmulni_intel [31626.314043] ? __hrtimer_run_queues+0x1d3/0x360 [31626.318308] mei_me mei [31626.322400] ? hrtimer_interrupt+0x104/0x240 [31626.326923] lpc_ich sha512_ssse3 [31626.329365] ? __local_bh_enable+0x50/0x60 [31626.333629] acpi_ipmi mfd_core [31626.336942] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31626.341032] wmi ipmi_si [31626.344186] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31626.349402] ipmi_devintf ipmi_msghandler [31626.351935] [31626.356891] ioatdma acpi_pad [31626.360896] [31626.362992] acpi_power_meter [31626.365957] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31626.368053] ixgbe igb [31626.371022] ? queued_spin_lock_slowpath+0x18/0x2e0 [31626.376324] mdio crc32c_intel [31626.378682] do_raw_spin_lock+0xb5/0xc0 [31626.383551] nvme ptp [31626.386604] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31626.390433] nvme_core [31626.392705] try_get_cap_refs+0x159/0x880 [ceph] [31626.397488] i2c_algo_bit pps_core [31626.399850] __ceph_get_caps+0x4a1/0x760 [ceph] [31626.404466] dca fuse [31626.407864] ? __file_remove_privs+0x89/0x110 [31626.412385] [31626.412385] irq event stamp: 174277656 [31626.414658] ? wake_bit_function+0x70/0x70 [31626.419007] hardirqs last enabled at (174277655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31626.420500] ? ceph_write_iter+0x260/0x770 [ceph] [31626.424242] hardirqs last disabled at (174277656): [] sysvec_apic_timer_interrupt+0xe/0x90 [31626.428333] ceph_write_iter+0x260/0x770 [ceph] [31626.438578] softirqs last enabled at (174277654): [] __do_softirq+0x2be/0x371 [31626.443277] ? lock_is_held_type+0xaa/0x120 [31626.453086] softirqs last disabled at (174277649): [] irq_exit_rcu+0xb0/0xd0 [31626.457612] vfs_write+0x324/0x4f0 [31626.466381] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31626.470561] ksys_pwrite64+0x65/0xa0 [31626.479154] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31626.482552] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31626.491929] Workqueue: events delayed_work [ceph] [31626.495502] do_syscall_64+0x38/0x80 [31626.502971] [31626.502972] RIP: 0010:lock_acquire+0xf0/0x2b0 [31626.508103] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31626.512800] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31626.516371] RIP: 0033:0x7f48f0eeb93b [31626.517860] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31626.522211] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31626.527254] [31626.527255] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31626.545993] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31626.549564] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31626.554781] ORIG_RAX: 0000000000000012 [31626.573519] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31626.575010] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31626.582134] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31626.587352] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31626.594475] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31626.598306] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31626.605431] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31626.612555] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31626.619678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31626.626802] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31626.633926] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31626.641057] [31626.649128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31626.685547] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31626.692671] Call Trace: [31626.695115] [31626.697126] ? watchdog_timer_fn+0x1b4/0x220 [31626.701392] ? __hrtimer_run_queues+0x1d3/0x360 [31626.705916] ? hrtimer_interrupt+0x104/0x240 [31626.710193] ? __local_bh_enable+0x50/0x60 [31626.714288] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31626.719512] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31626.724481] [31626.726575] [31626.728674] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31626.733983] ? lock_acquire+0xf0/0x2b0 [31626.737731] ? igrab+0x43/0x50 [31626.740782] ? lock_release+0xc5/0x260 [31626.744527] _raw_spin_lock+0x30/0x40 [31626.748194] ? igrab+0x1d/0x50 [31626.751250] igrab+0x1d/0x50 [31626.754127] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31626.759279] delayed_work+0x1d8/0x280 [ceph] [31626.763560] ? process_scheduled_works+0x22a/0x510 [31626.768350] process_scheduled_works+0x2a4/0x510 [31626.772965] worker_thread+0x188/0x340 [31626.776714] ? rescuer_thread+0x310/0x310 [31626.780719] kthread+0xf5/0x130 [31626.783856] ? kthread_complete_and_exit+0x20/0x20 [31626.788640] ret_from_fork+0x31/0x40 [31626.792217] ? kthread_complete_and_exit+0x20/0x20 [31626.797003] ret_from_fork_asm+0x11/0x20 [31626.800926] [31654.014291] watchdog: BUG: soft lockup - CPU#0 stuck for 18366s! [dbench:78683] [31654.021594] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31654.026290] watchdog: BUG: soft lockup - CPU#7 stuck for 27363s! [kworker/7:4:78808] [31654.077043] irq event stamp: 53821742 [31654.084773] Modules linked in: ceph libceph dns_resolver fscache [31654.088431] hardirqs last enabled at (53821741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31654.094427] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [31654.104586] hardirqs last disabled at (53821742): [] sysvec_apic_timer_interrupt+0xe/0x90 [31654.104589] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [31654.119795] softirqs last enabled at (53821716): [] __do_softirq+0x2be/0x371 [31654.128654] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [31654.137338] softirqs last disabled at (53821711): [] irq_exit_rcu+0xb0/0xd0 [31654.146376] coretemp kvm_intel kvm irqbypass i2c_i801 [31654.154890] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31654.160020] iTCO_wdt [31654.168964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31654.168966] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31654.171232] iTCO_vendor_support i2c_smbus [31654.178704] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31654.184181] crct10dif_pclmul joydev [31654.188273] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31654.207010] crc32_pclmul ipmi_ssif [31654.210581] [31654.215798] ghash_clmulni_intel [31654.219282] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31654.220775] mei_me [31654.223997] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31654.231122] mei lpc_ich [31654.233219] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31654.240341] sha512_ssse3 acpi_ipmi [31654.242874] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31654.249997] mfd_core wmi [31654.253480] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31654.260604] ipmi_si ipmi_devintf [31654.263224] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31654.270348] ipmi_msghandler [31654.273659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31654.281736] ioatdma acpi_pad [31654.284614] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31654.290351] acpi_power_meter [31654.293316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31654.300438] ixgbe igb [31654.303403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31654.310527] mdio crc32c_intel [31654.312884] Call Trace: [31654.312887] [31654.320009] nvme ptp [31654.323062] ? watchdog_timer_fn+0x1b4/0x220 [31654.325502] nvme_core i2c_algo_bit [31654.327515] ? __hrtimer_run_queues+0x1d3/0x360 [31654.329785] pps_core [31654.334051] ? hrtimer_interrupt+0x104/0x240 [31654.337533] dca fuse [31654.342058] ? __local_bh_enable+0x50/0x60 [31654.344328] [31654.344328] irq event stamp: 174498688 [31654.348593] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31654.350863] hardirqs last enabled at (174498687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31654.354968] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31654.356461] hardirqs last disabled at (174498688): [] sysvec_apic_timer_interrupt+0xe/0x90 [31654.360205] [31654.365421] softirqs last enabled at (174498686): [] __do_softirq+0x2be/0x371 [31654.375667] [31654.380641] softirqs last disabled at (174498681): [] irq_exit_rcu+0xb0/0xd0 [31654.390454] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31654.392549] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31654.401324] ? queued_spin_lock_slowpath+0x16/0x2e0 [31654.403417] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31654.412018] do_raw_spin_lock+0xb5/0xc0 [31654.417321] Workqueue: events delayed_work [ceph] [31654.426699] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31654.431568] [31654.431569] RIP: 0010:lock_acquire+0xf0/0x2b0 [31654.439040] try_get_cap_refs+0x159/0x880 [ceph] [31654.442869] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31654.447573] __ceph_get_caps+0x4a1/0x760 [ceph] [31654.452351] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31654.453846] ? __file_remove_privs+0x89/0x110 [31654.458191] [31654.458192] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31654.462803] ? wake_bit_function+0x70/0x70 [31654.481539] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31654.486066] ? ceph_write_iter+0x260/0x770 [ceph] [31654.491282] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31654.495634] ceph_write_iter+0x260/0x770 [ceph] [31654.497123] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31654.504250] ? lock_is_held_type+0xaa/0x120 [31654.508338] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31654.515464] vfs_write+0x324/0x4f0 [31654.520161] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31654.527288] ksys_pwrite64+0x65/0xa0 [31654.531808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31654.538932] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31654.543110] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31654.550234] do_syscall_64+0x38/0x80 [31654.553631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31654.561710] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31654.565278] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31654.571018] RIP: 0033:0x7f48f0eeb93b [31654.576148] Call Trace: [31654.576149] [31654.583271] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31654.586842] ? watchdog_timer_fn+0x1b4/0x220 [31654.593974] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31654.599020] ? __hrtimer_run_queues+0x1d3/0x360 [31654.606143] ORIG_RAX: 0000000000000012 [31654.609719] ? hrtimer_interrupt+0x104/0x240 [31654.612159] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31654.614168] ? __local_bh_enable+0x50/0x60 [31654.632908] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31654.637174] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31654.642389] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31654.646913] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31654.650742] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31654.655007] [31654.662132] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31654.666223] [31654.666224] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31654.673350] [31654.678567] ? lock_acquire+0xf0/0x2b0 [31654.720330] ? igrab+0x43/0x50 [31654.723380] ? lock_release+0xc5/0x260 [31654.727125] _raw_spin_lock+0x30/0x40 [31654.730782] ? igrab+0x1d/0x50 [31654.733832] igrab+0x1d/0x50 [31654.736711] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31654.741862] delayed_work+0x1d8/0x280 [ceph] [31654.746142] ? process_scheduled_works+0x22a/0x510 [31654.750932] process_scheduled_works+0x2a4/0x510 [31654.755546] worker_thread+0x188/0x340 [31654.759296] ? rescuer_thread+0x310/0x310 [31654.763298] kthread+0xf5/0x130 [31654.766436] ? kthread_complete_and_exit+0x20/0x20 [31654.771221] ret_from_fork+0x31/0x40 [31654.774801] ? kthread_complete_and_exit+0x20/0x20 [31654.779584] ret_from_fork_asm+0x11/0x20 [31654.783509] [31682.014787] watchdog: BUG: soft lockup - CPU#0 stuck for 18392s! [dbench:78683] [31682.022089] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31682.026787] watchdog: BUG: soft lockup - CPU#7 stuck for 27389s! [kworker/7:4:78808] [31682.077530] irq event stamp: 53879772 [31682.085260] Modules linked in: ceph libceph [31682.088918] hardirqs last enabled at (53879771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31682.093096] dns_resolver fscache [31682.103254] hardirqs last disabled at (53879772): [] sysvec_apic_timer_interrupt+0xe/0x90 [31682.106568] netfs veth [31682.116298] softirqs last enabled at (53879714): [] __do_softirq+0x2be/0x371 [31682.116299] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [31682.127426] softirqs last disabled at (53879709): [] irq_exit_rcu+0xb0/0xd0 [31682.127427] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [31682.141586] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31682.147497] nfnetlink [31682.156440] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31682.158798] bridge stp llc xfs [31682.166270] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31682.169407] libcrc32c sunrpc intel_rapl_msr [31682.174884] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31682.179149] intel_rapl_common [31682.197885] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31682.200937] sb_edac x86_pkg_temp_thermal [31682.206156] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31682.206158] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31682.210183] coretemp kvm_intel [31682.217308] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31682.224432] kvm irqbypass [31682.227569] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31682.234695] i2c_i801 [31682.237397] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31682.244523] iTCO_wdt iTCO_vendor_support [31682.246792] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31682.253917] i2c_smbus [31682.257921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31682.266000] crct10dif_pclmul joydev [31682.268355] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31682.274093] crc32_pclmul [31682.277664] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31682.284790] ipmi_ssif ghash_clmulni_intel [31682.287405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31682.294530] mei_me mei [31682.298621] Call Trace: [31682.298622] [31682.305745] lpc_ich sha512_ssse3 [31682.308191] ? watchdog_timer_fn+0x1b4/0x220 [31682.310632] acpi_ipmi mfd_core [31682.312646] ? __hrtimer_run_queues+0x1d3/0x360 [31682.315954] wmi ipmi_si [31682.320223] ? hrtimer_interrupt+0x104/0x240 [31682.323355] ipmi_devintf ipmi_msghandler [31682.327881] ? __local_bh_enable+0x50/0x60 [31682.330411] ioatdma acpi_pad [31682.334678] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31682.338678] acpi_power_meter [31682.342769] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31682.345733] ixgbe igb [31682.350952] [31682.353914] mdio [31682.358872] [31682.361231] crc32c_intel [31682.363327] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31682.365251] nvme ptp [31682.367353] ? queued_spin_lock_slowpath+0x16/0x2e0 [31682.369967] nvme_core i2c_algo_bit [31682.375272] do_raw_spin_lock+0xb5/0xc0 [31682.377542] pps_core dca [31682.382413] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31682.385897] fuse [31682.389729] try_get_cap_refs+0x159/0x880 [ceph] [31682.392344] irq event stamp: 174720218 [31682.397133] __ceph_get_caps+0x4a1/0x760 [ceph] [31682.399051] hardirqs last enabled at (174720217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31682.403666] ? __file_remove_privs+0x89/0x110 [31682.407407] hardirqs last disabled at (174720218): [] sysvec_apic_timer_interrupt+0xe/0x90 [31682.411931] ? wake_bit_function+0x70/0x70 [31682.422190] softirqs last enabled at (174720216): [] __do_softirq+0x2be/0x371 [31682.426545] ? ceph_write_iter+0x260/0x770 [ceph] [31682.436354] softirqs last disabled at (174720211): [] irq_exit_rcu+0xb0/0xd0 [31682.440446] ceph_write_iter+0x260/0x770 [ceph] [31682.449216] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31682.453917] ? lock_is_held_type+0xaa/0x120 [31682.462509] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31682.467036] vfs_write+0x324/0x4f0 [31682.476411] Workqueue: events delayed_work [ceph] [31682.480595] ksys_pwrite64+0x65/0xa0 [31682.488060] [31682.491474] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31682.496188] RIP: 0010:lock_release+0x14e/0x260 [31682.499761] do_syscall_64+0x38/0x80 [31682.501250] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [31682.506383] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31682.510818] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [31682.514390] RIP: 0033:0x7f48f0eeb93b [31682.533129] [31682.538188] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31682.543408] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [31682.546978] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31682.548471] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [31682.567207] ORIG_RAX: 0000000000000012 [31682.574331] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [31682.579548] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31682.586671] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [31682.590502] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31682.597627] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [31682.604751] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31682.611874] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31682.618998] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31682.626123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31682.633246] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31682.641325] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31682.648471] [31682.654203] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31682.677778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31682.684902] Call Trace: [31682.687347] [31682.689358] ? watchdog_timer_fn+0x1b4/0x220 [31682.693621] ? __hrtimer_run_queues+0x1d3/0x360 [31682.698149] ? hrtimer_interrupt+0x104/0x240 [31682.702417] ? __local_bh_enable+0x50/0x60 [31682.706510] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31682.711734] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31682.716693] [31682.718789] [31682.720886] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31682.726196] ? igrab+0x43/0x50 [31682.729254] ? lock_release+0x14e/0x260 [31682.733091] ? lock_release+0xc5/0x260 [31682.736835] _raw_spin_unlock+0x1b/0x40 [31682.740675] igrab+0x43/0x50 [31682.743551] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31682.748701] delayed_work+0x1d8/0x280 [ceph] [31682.752982] ? process_scheduled_works+0x22a/0x510 [31682.757772] process_scheduled_works+0x2a4/0x510 [31682.762388] worker_thread+0x188/0x340 [31682.766136] ? rescuer_thread+0x310/0x310 [31682.770139] kthread+0xf5/0x130 [31682.773277] ? kthread_complete_and_exit+0x20/0x20 [31682.778062] ret_from_fork+0x31/0x40 [31682.781633] ? kthread_complete_and_exit+0x20/0x20 [31682.786418] ret_from_fork_asm+0x11/0x20 [31682.790349] [31710.015277] watchdog: BUG: soft lockup - CPU#0 stuck for 18418s! [dbench:78683] [31710.022578] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31710.027276] watchdog: BUG: soft lockup - CPU#7 stuck for 27415s! [kworker/7:4:78808] [31710.078019] irq event stamp: 53937798 [31710.078020] hardirqs last enabled at (53937797): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31710.085749] Modules linked in: [31710.089407] hardirqs last disabled at (53937798): [] sysvec_apic_timer_interrupt+0xe/0x90 [31710.099563] ceph [31710.102613] softirqs last enabled at (53937782): [] __do_softirq+0x2be/0x371 [31710.112337] libceph [31710.114264] softirqs last disabled at (53937777): [] irq_exit_rcu+0xb0/0xd0 [31710.122961] dns_resolver [31710.125148] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31710.133673] fscache netfs [31710.136293] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31710.145238] veth nft_chain_nat [31710.147953] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31710.155420] xt_MASQUERADE [31710.158558] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31710.164036] nf_nat nf_conntrack [31710.166741] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31710.185478] nf_defrag_ipv6 [31710.188701] [31710.188702] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31710.193920] nf_defrag_ipv4 [31710.196711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31710.198201] nft_compat nf_tables [31710.205325] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31710.208115] nfnetlink bridge [31710.215241] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31710.218550] stp llc [31710.225674] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31710.228639] xfs libcrc32c [31710.235763] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31710.237962] sunrpc intel_rapl_msr [31710.245089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31710.247791] intel_rapl_common sb_edac [31710.255871] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31710.259268] x86_pkg_temp_thermal coretemp [31710.265004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31710.268749] kvm_intel [31710.275873] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31710.279964] kvm irqbypass [31710.287087] Call Trace: [31710.289445] i2c_i801 iTCO_wdt [31710.296571] [31710.299272] iTCO_vendor_support i2c_smbus [31710.301717] ? watchdog_timer_fn+0x1b4/0x220 [31710.304768] crct10dif_pclmul [31710.306779] ? __hrtimer_run_queues+0x1d3/0x360 [31710.310870] joydev crc32_pclmul [31710.315138] ? hrtimer_interrupt+0x104/0x240 [31710.318098] ipmi_ssif [31710.322623] ? __local_bh_enable+0x50/0x60 [31710.325847] ghash_clmulni_intel mei_me [31710.330112] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31710.332468] mei lpc_ich [31710.336558] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31710.340389] sha512_ssse3 acpi_ipmi [31710.345606] [31710.348136] mfd_core wmi [31710.353094] [31710.353095] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31710.356577] ipmi_si [31710.358683] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31710.361292] ipmi_devintf [31710.363392] do_raw_spin_lock+0xb5/0xc0 [31710.368693] ipmi_msghandler ioatdma [31710.370880] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31710.375750] acpi_pad acpi_power_meter [31710.378369] try_get_cap_refs+0x159/0x880 [ceph] [31710.382198] ixgbe igb [31710.385774] __ceph_get_caps+0x4a1/0x760 [ceph] [31710.390552] mdio [31710.394301] ? __file_remove_privs+0x89/0x110 [31710.398906] crc32c_intel [31710.401264] ? wake_bit_function+0x70/0x70 [31710.405788] nvme ptp [31710.407716] ? ceph_write_iter+0x260/0x770 [ceph] [31710.412065] nvme_core [31710.414682] ceph_write_iter+0x260/0x770 [ceph] [31710.418772] i2c_algo_bit [31710.421046] ? lock_is_held_type+0xaa/0x120 [31710.425739] pps_core [31710.428100] vfs_write+0x324/0x4f0 [31710.432622] dca [31710.435242] ksys_pwrite64+0x65/0xa0 [31710.439417] fuse [31710.441688] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31710.445083] [31710.446922] do_syscall_64+0x38/0x80 [31710.450493] irq event stamp: 174941672 [31710.452417] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31710.457546] hardirqs last enabled at (174941671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31710.459040] RIP: 0033:0x7f48f0eeb93b [31710.462608] hardirqs last disabled at (174941672): [] sysvec_apic_timer_interrupt+0xe/0x90 [31710.466355] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31710.471396] softirqs last enabled at (174941670): [] __do_softirq+0x2be/0x371 [31710.481642] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31710.485212] softirqs last disabled at (174941665): [] irq_exit_rcu+0xb0/0xd0 [31710.495023] ORIG_RAX: 0000000000000012 [31710.513760] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31710.522532] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31710.527748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31710.536344] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31710.540177] Workqueue: events delayed_work [ceph] [31710.549553] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31710.556679] [31710.564148] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31710.571272] RIP: 0010:_raw_spin_lock+0xd/0x40 [31710.575972] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31710.583094] Code: 69 5d ff ff 48 89 ea 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb <65> ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 [31710.584591] [31710.591709] RSP: 0018:ffffc90002117d90 EFLAGS: 00000286 [31710.629325] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [31710.636447] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [31710.643573] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [31710.650696] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [31710.657820] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31710.664943] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31710.673040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31710.678777] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31710.685900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31710.693025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31710.700149] Call Trace: [31710.702596] [31710.704604] ? watchdog_timer_fn+0x1b4/0x220 [31710.708868] ? __hrtimer_run_queues+0x1d3/0x360 [31710.713396] ? hrtimer_interrupt+0x104/0x240 [31710.717682] ? __local_bh_enable+0x50/0x60 [31710.721776] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31710.726998] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31710.731973] [31710.734071] [31710.736168] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31710.741477] ? _raw_spin_lock+0xd/0x40 [31710.745226] igrab+0x1d/0x50 [31710.748103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31710.753254] delayed_work+0x1d8/0x280 [ceph] [31710.757535] ? process_scheduled_works+0x22a/0x510 [31710.762326] process_scheduled_works+0x2a4/0x510 [31710.766940] worker_thread+0x188/0x340 [31710.770706] ? rescuer_thread+0x310/0x310 [31710.774709] kthread+0xf5/0x130 [31710.777849] ? kthread_complete_and_exit+0x20/0x20 [31710.782633] ret_from_fork+0x31/0x40 [31710.786211] ? kthread_complete_and_exit+0x20/0x20 [31710.790995] ret_from_fork_asm+0x11/0x20 [31710.794917] [31738.015756] watchdog: BUG: soft lockup - CPU#0 stuck for 18444s! [dbench:78683] [31738.023057] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31738.027756] watchdog: BUG: soft lockup - CPU#7 stuck for 27441s! [kworker/7:4:78808] [31738.078507] irq event stamp: 53996036 [31738.078508] hardirqs last enabled at (53996035): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31738.086238] Modules linked in: [31738.089894] hardirqs last disabled at (53996036): [] sysvec_apic_timer_interrupt+0xe/0x90 [31738.100051] ceph [31738.103101] softirqs last enabled at (53996010): [] __do_softirq+0x2be/0x371 [31738.112826] libceph dns_resolver [31738.114750] softirqs last disabled at (53996005): [] irq_exit_rcu+0xb0/0xd0 [31738.123434] fscache netfs [31738.126745] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31738.135256] veth [31738.137961] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31738.146905] nft_chain_nat [31738.148830] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31738.156298] xt_MASQUERADE nf_nat [31738.159003] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31738.164480] nf_conntrack nf_defrag_ipv6 [31738.167790] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31738.186529] nf_defrag_ipv4 nft_compat [31738.190446] [31738.195665] nf_tables nfnetlink [31738.199408] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31738.200917] bridge stp [31738.204151] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31738.211274] llc xfs [31738.213716] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31738.220841] libcrc32c sunrpc [31738.223026] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31738.230153] intel_rapl_msr intel_rapl_common [31738.233113] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31738.240237] sb_edac x86_pkg_temp_thermal [31738.244588] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31738.251712] coretemp kvm_intel [31738.255717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31738.263793] kvm irqbypass [31738.266931] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31738.272669] i2c_i801 iTCO_wdt [31738.275374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31738.282497] iTCO_vendor_support i2c_smbus [31738.285547] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31738.292672] crct10dif_pclmul joydev [31738.296763] Call Trace: [31738.296765] [31738.303886] crc32_pclmul ipmi_ssif [31738.307457] ? watchdog_timer_fn+0x1b4/0x220 [31738.309903] ghash_clmulni_intel [31738.311914] ? __hrtimer_run_queues+0x1d3/0x360 [31738.315397] mei_me [31738.319664] ? hrtimer_interrupt+0x104/0x240 [31738.322885] mei lpc_ich [31738.327409] ? __local_bh_enable+0x50/0x60 [31738.329504] sha512_ssse3 acpi_ipmi [31738.333773] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31738.336301] mfd_core wmi [31738.340391] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31738.343875] ipmi_si [31738.349093] [31738.351711] ipmi_devintf ipmi_msghandler [31738.356667] [31738.358852] ioatdma [31738.360949] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31738.364952] acpi_pad acpi_power_meter [31738.367054] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31738.369234] ixgbe igb [31738.374542] do_raw_spin_lock+0xb5/0xc0 [31738.378283] mdio crc32c_intel [31738.383157] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31738.385510] nvme [31738.389344] try_get_cap_refs+0x159/0x880 [ceph] [31738.392392] ptp nvme_core [31738.397181] __ceph_get_caps+0x4a1/0x760 [ceph] [31738.399100] i2c_algo_bit pps_core [31738.403714] ? __file_remove_privs+0x89/0x110 [31738.406414] dca fuse [31738.410958] ? wake_bit_function+0x70/0x70 [31738.414353] irq event stamp: 175162910 [31738.418707] ? ceph_write_iter+0x260/0x770 [ceph] [31738.420976] hardirqs last enabled at (175162909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31738.425066] ceph_write_iter+0x260/0x770 [ceph] [31738.428809] hardirqs last disabled at (175162910): [] sysvec_apic_timer_interrupt+0xe/0x90 [31738.433510] ? lock_is_held_type+0xaa/0x120 [31738.443751] softirqs last enabled at (175162908): [] __do_softirq+0x2be/0x371 [31738.448279] vfs_write+0x324/0x4f0 [31738.458086] softirqs last disabled at (175162903): [] irq_exit_rcu+0xb0/0xd0 [31738.462269] ksys_pwrite64+0x65/0xa0 [31738.471034] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31738.474439] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31738.483038] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31738.486610] do_syscall_64+0x38/0x80 [31738.495988] Workqueue: events delayed_work [ceph] [31738.501119] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31738.508587] [31738.512162] RIP: 0033:0x7f48f0eeb93b [31738.516855] RIP: 0010:lock_acquire+0x1c/0x2b0 [31738.521901] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31738.523391] Code: 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 <44> 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 [31738.526961] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31738.531312] RSP: 0018:ffffc90002117d58 EFLAGS: 00000246 [31738.550050] ORIG_RAX: 0000000000000012 [31738.568788] [31738.568789] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [31738.574007] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31738.579224] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31738.583054] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31738.584543] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [31738.591668] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31738.598793] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [31738.605916] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31738.613042] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31738.620165] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31738.627290] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31738.634420] [31738.641556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31738.671801] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31738.678927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31738.686050] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31738.693176] Call Trace: [31738.695620] [31738.697629] ? watchdog_timer_fn+0x1b4/0x220 [31738.701896] ? __hrtimer_run_queues+0x1d3/0x360 [31738.706422] ? hrtimer_interrupt+0x104/0x240 [31738.710699] ? __local_bh_enable+0x50/0x60 [31738.714794] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31738.720016] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31738.724972] [31738.727070] [31738.729168] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31738.734476] ? lock_acquire+0x1c/0x2b0 [31738.738228] _raw_spin_lock+0x30/0x40 [31738.741891] ? igrab+0x1d/0x50 [31738.744942] igrab+0x1d/0x50 [31738.747821] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31738.752971] delayed_work+0x1d8/0x280 [ceph] [31738.757252] ? process_scheduled_works+0x22a/0x510 [31738.762040] process_scheduled_works+0x2a4/0x510 [31738.766658] worker_thread+0x188/0x340 [31738.770405] ? rescuer_thread+0x310/0x310 [31738.774409] kthread+0xf5/0x130 [31738.777548] ? kthread_complete_and_exit+0x20/0x20 [31738.782332] ret_from_fork+0x31/0x40 [31738.785910] ? kthread_complete_and_exit+0x20/0x20 [31738.790695] ret_from_fork_asm+0x11/0x20 [31738.794625] [31740.181159] rcu: INFO: rcu_preempt self-detected stall on CPU [31740.186922] rcu: 0-....: (19356884 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9686575 [31740.197348] rcu: (t=19803219 jiffies g=1309609 q=574442 ncpus=8) [31740.203440] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31740.212402] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31740.219872] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31740.225356] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31740.244095] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31740.249311] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31740.256436] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31740.263561] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31740.270683] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31740.277809] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31740.284933] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31740.293011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31740.298748] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31740.305872] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31740.312997] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31740.320121] Call Trace: [31740.322564] [31740.324578] ? rcu_dump_cpu_stacks+0xd1/0x180 [31740.328936] ? rcu_sched_clock_irq+0x633/0x1390 [31740.333481] ? lock_is_held_type+0xaa/0x120 [31740.337665] ? lock_is_held_type+0xaa/0x120 [31740.341850] ? tick_sched_do_timer+0xa0/0xa0 [31740.346121] ? rcu_read_lock_sched_held+0x56/0x90 [31740.350822] ? tick_sched_do_timer+0xa0/0xa0 [31740.355091] ? update_process_times+0x5f/0x90 [31740.359444] ? tick_sched_handle.isra.16+0x1e/0x50 [31740.364235] ? tick_sched_timer+0x6f/0x80 [31740.368251] ? __hrtimer_run_queues+0x1d3/0x360 [31740.372786] ? hrtimer_interrupt+0x104/0x240 [31740.377053] ? __local_bh_enable+0x50/0x60 [31740.381149] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31740.386370] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31740.391329] [31740.393423] [31740.395522] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31740.400844] ? queued_spin_lock_slowpath+0x16/0x2e0 [31740.405727] do_raw_spin_lock+0xb5/0xc0 [31740.409564] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31740.414368] try_get_cap_refs+0x159/0x880 [ceph] [31740.419009] __ceph_get_caps+0x4a1/0x760 [ceph] [31740.423560] ? __file_remove_privs+0x89/0x110 [31740.427919] ? wake_bit_function+0x70/0x70 [31740.432016] ? ceph_write_iter+0x260/0x770 [ceph] [31740.436732] ceph_write_iter+0x260/0x770 [ceph] [31740.441281] ? lock_is_held_type+0xaa/0x120 [31740.445482] vfs_write+0x324/0x4f0 [31740.448892] ksys_pwrite64+0x65/0xa0 [31740.452479] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31740.457615] do_syscall_64+0x38/0x80 [31740.461191] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31740.466243] RIP: 0033:0x7f48f0eeb93b [31740.469813] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31740.488551] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [31740.496108] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31740.503232] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31740.510357] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31740.517483] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31740.524614] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31740.531750] [31740.533941] Sending NMI from CPU 0 to CPUs 7: [31740.538301] NMI backtrace for cpu 7 [31740.538303] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31740.538304] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31740.538305] Workqueue: events delayed_work [ceph] [31740.538320] RIP: 0010:__lock_acquire+0x6fa/0x1e20 [31740.538322] Code: 00 83 f8 2f 77 46 3b 05 98 71 e0 01 0f 87 b6 04 00 00 41 be 01 00 00 00 e9 85 00 00 00 48 81 3f c0 50 d7 82 41 bb 00 00 00 00 <45> 0f 45 d8 83 fe 01 0f 87 64 f9 ff ff 89 f0 4d 8b 64 c7 08 4d 85 [31740.538324] RSP: 0018:ffffc90002117c60 EFLAGS: 00000006 [31740.538325] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 0000000000000000 [31740.538326] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31740.538326] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [31740.538327] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8881a1d34518 [31740.538327] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8881a1d34518 [31740.538328] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31740.538329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31740.538330] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31740.538331] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31740.538331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31740.538332] Call Trace: [31740.538333] [31740.538334] ? nmi_cpu_backtrace+0xad/0x120 [31740.538337] ? nmi_cpu_backtrace_handler+0x11/0x20 [31740.538339] ? nmi_handle+0xd4/0x260 [31740.538343] ? default_do_nmi+0x49/0xf0 [31740.538344] ? exc_nmi+0x11e/0x150 [31740.538345] ? end_repeat_nmi+0x16/0x67 [31740.538349] ? __lock_acquire+0x6fa/0x1e20 [31740.538351] ? __lock_acquire+0x6fa/0x1e20 [31740.538352] ? __lock_acquire+0x6fa/0x1e20 [31740.538354] [31740.538354] [31740.538356] lock_acquire+0xcb/0x2b0 [31740.538357] ? igrab+0x1d/0x50 [31740.538359] ? igrab+0x43/0x50 [31740.538360] ? lock_release+0xc5/0x260 [31740.538361] _raw_spin_lock+0x30/0x40 [31740.538364] ? igrab+0x1d/0x50 [31740.538364] igrab+0x1d/0x50 [31740.538366] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31740.538380] delayed_work+0x1d8/0x280 [ceph] [31740.538393] ? process_scheduled_works+0x22a/0x510 [31740.538395] process_scheduled_works+0x2a4/0x510 [31740.538399] worker_thread+0x188/0x340 [31740.538402] ? rescuer_thread+0x310/0x310 [31740.538404] kthread+0xf5/0x130 [31740.538406] ? kthread_complete_and_exit+0x20/0x20 [31740.538408] ret_from_fork+0x31/0x40 [31740.538410] ? kthread_complete_and_exit+0x20/0x20 [31740.538412] ret_from_fork_asm+0x11/0x20 [31740.538416] [31766.016235] watchdog: BUG: soft lockup - CPU#0 stuck for 18470s! [dbench:78683] [31766.023536] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31766.028235] watchdog: BUG: soft lockup - CPU#7 stuck for 27467s! [kworker/7:4:78808] [31766.078977] irq event stamp: 54053366 [31766.078979] hardirqs last enabled at (54053365): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31766.086707] Modules linked in: ceph [31766.090364] hardirqs last disabled at (54053366): [] sysvec_apic_timer_interrupt+0xe/0x90 [31766.100523] libceph [31766.104007] softirqs last enabled at (54053328): [] __do_softirq+0x2be/0x371 [31766.113730] dns_resolver fscache [31766.115918] softirqs last disabled at (54053323): [] irq_exit_rcu+0xb0/0xd0 [31766.124597] netfs veth [31766.127921] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31766.136430] nft_chain_nat xt_MASQUERADE [31766.138874] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31766.147817] nf_nat nf_conntrack [31766.151735] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31766.159204] nf_defrag_ipv6 nf_defrag_ipv4 [31766.162429] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31766.167922] nft_compat nf_tables [31766.172016] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31766.190752] nfnetlink bridge [31766.194064] [31766.194065] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31766.199281] stp llc [31766.202244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31766.203734] xfs libcrc32c [31766.210859] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31766.213042] sunrpc [31766.220167] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31766.222872] intel_rapl_msr intel_rapl_common [31766.229997] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31766.232093] sb_edac x86_pkg_temp_thermal [31766.239217] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31766.243567] coretemp kvm_intel [31766.250692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31766.254697] kvm irqbypass [31766.262775] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31766.265912] i2c_i801 iTCO_wdt [31766.271647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31766.274353] iTCO_vendor_support [31766.281476] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31766.284527] i2c_smbus crct10dif_pclmul [31766.291651] Call Trace: [31766.294875] joydev crc32_pclmul [31766.301999] [31766.305831] ipmi_ssif ghash_clmulni_intel [31766.308276] ? watchdog_timer_fn+0x1b4/0x220 [31766.311499] mei_me mei [31766.313511] ? __hrtimer_run_queues+0x1d3/0x360 [31766.317599] lpc_ich sha512_ssse3 [31766.321866] ? hrtimer_interrupt+0x104/0x240 [31766.324309] acpi_ipmi mfd_core [31766.328832] ? __local_bh_enable+0x50/0x60 [31766.332143] wmi ipmi_si [31766.336410] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31766.339544] ipmi_devintf ipmi_msghandler [31766.343636] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31766.346167] ioatdma acpi_pad [31766.351384] [31766.355386] acpi_power_meter [31766.360344] [31766.363308] ixgbe igb [31766.365406] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31766.368370] mdio crc32c_intel [31766.370471] ? queued_spin_lock_slowpath+0x18/0x2e0 [31766.372825] nvme ptp [31766.378130] do_raw_spin_lock+0xb5/0xc0 [31766.381179] nvme_core [31766.386052] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31766.388320] i2c_algo_bit pps_core [31766.392154] try_get_cap_refs+0x159/0x880 [ceph] [31766.394510] dca fuse [31766.399298] __ceph_get_caps+0x4a1/0x760 [ceph] [31766.402691] [31766.402692] irq event stamp: 175382358 [31766.407305] ? __file_remove_privs+0x89/0x110 [31766.409573] hardirqs last enabled at (175382357): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31766.414097] ? wake_bit_function+0x70/0x70 [31766.415587] hardirqs last disabled at (175382358): [] sysvec_apic_timer_interrupt+0xe/0x90 [31766.419335] ? ceph_write_iter+0x260/0x770 [ceph] [31766.423681] softirqs last enabled at (175382356): [] __do_softirq+0x2be/0x371 [31766.433926] ceph_write_iter+0x260/0x770 [ceph] [31766.438018] softirqs last disabled at (175382351): [] irq_exit_rcu+0xb0/0xd0 [31766.447833] ? lock_is_held_type+0xaa/0x120 [31766.452525] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31766.461299] vfs_write+0x324/0x4f0 [31766.465819] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31766.474421] ksys_pwrite64+0x65/0xa0 [31766.478594] Workqueue: events delayed_work [ceph] [31766.487973] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31766.491371] [31766.498842] do_syscall_64+0x38/0x80 [31766.502411] RIP: 0010:lock_acquire+0xf0/0x2b0 [31766.507108] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31766.512240] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31766.513730] RIP: 0033:0x7f48f0eeb93b [31766.517300] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31766.521652] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31766.526695] [31766.526696] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31766.545433] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31766.549004] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31766.554222] ORIG_RAX: 0000000000000012 [31766.572959] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31766.574450] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31766.581575] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31766.586792] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31766.593928] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31766.597754] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31766.604879] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31766.612005] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31766.619129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31766.626253] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31766.633376] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31766.640505] [31766.648578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31766.684996] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31766.692120] Call Trace: [31766.694563] [31766.696577] ? watchdog_timer_fn+0x1b4/0x220 [31766.700839] ? __hrtimer_run_queues+0x1d3/0x360 [31766.705367] ? hrtimer_interrupt+0x104/0x240 [31766.709641] ? __local_bh_enable+0x50/0x60 [31766.713739] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31766.718962] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31766.723920] [31766.726016] [31766.728114] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31766.733422] ? lock_acquire+0xf0/0x2b0 [31766.737171] ? igrab+0x43/0x50 [31766.740221] ? lock_release+0xc5/0x260 [31766.743968] _raw_spin_lock+0x30/0x40 [31766.747643] ? igrab+0x1d/0x50 [31766.750692] igrab+0x1d/0x50 [31766.753569] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31766.758720] delayed_work+0x1d8/0x280 [ceph] [31766.763000] ? process_scheduled_works+0x22a/0x510 [31766.767789] process_scheduled_works+0x2a4/0x510 [31766.772405] worker_thread+0x188/0x340 [31766.776156] ? rescuer_thread+0x310/0x310 [31766.780167] kthread+0xf5/0x130 [31766.783305] ? kthread_complete_and_exit+0x20/0x20 [31766.788090] ret_from_fork+0x31/0x40 [31766.791667] ? kthread_complete_and_exit+0x20/0x20 [31766.796452] ret_from_fork_asm+0x11/0x20 [31766.800375] [31794.016710] watchdog: BUG: soft lockup - CPU#0 stuck for 18496s! [dbench:78683] [31794.024017] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31794.028709] watchdog: BUG: soft lockup - CPU#7 stuck for 27493s! [kworker/7:4:78808] [31794.079465] irq event stamp: 54111470 [31794.087194] Modules linked in: ceph libceph dns_resolver fscache netfs veth [31794.090851] hardirqs last enabled at (54111469): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31794.090857] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [31794.107969] hardirqs last disabled at (54111470): [] sysvec_apic_timer_interrupt+0xe/0x90 [31794.117174] nf_tables nfnetlink bridge stp llc xfs libcrc32c [31794.126898] softirqs last enabled at (54111434): [] __do_softirq+0x2be/0x371 [31794.132643] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [31794.141327] softirqs last disabled at (54111429): [] irq_exit_rcu+0xb0/0xd0 [31794.141330] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [31794.157312] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31794.163221] iTCO_vendor_support [31794.172164] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31794.175388] i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [31794.182861] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31794.182863] ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 [31794.194770] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31794.194770] acpi_ipmi [31794.194771] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31794.200766] mfd_core wmi [31794.219502] [31794.221860] ipmi_si ipmi_devintf [31794.227077] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31794.229695] ipmi_msghandler ioatdma [31794.231185] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31794.234497] acpi_pad acpi_power_meter [31794.241621] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31794.245192] ixgbe igb [31794.252316] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31794.256060] mdio crc32c_intel [31794.263183] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31794.265543] nvme [31794.272665] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31794.275715] ptp nvme_core [31794.282840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31794.284764] i2c_algo_bit pps_core [31794.292841] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31794.295545] dca fuse [31794.301282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31794.304680] [31794.304681] irq event stamp: 175604242 [31794.311804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31794.314074] hardirqs last enabled at (175604241): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31794.321199] Call Trace: [31794.321201] [31794.322691] hardirqs last disabled at (175604242): [] sysvec_apic_timer_interrupt+0xe/0x90 [31794.326436] ? watchdog_timer_fn+0x1b4/0x220 [31794.333559] softirqs last enabled at (175604240): [] __do_softirq+0x2be/0x371 [31794.343803] ? __hrtimer_run_queues+0x1d3/0x360 [31794.346246] softirqs last disabled at (175604235): [] irq_exit_rcu+0xb0/0xd0 [31794.348260] ? hrtimer_interrupt+0x104/0x240 [31794.358068] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31794.362333] ? __local_bh_enable+0x50/0x60 [31794.371110] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31794.371111] Workqueue: events delayed_work [ceph] [31794.375637] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31794.384234] [31794.384235] RIP: 0010:lock_acquire+0xf0/0x2b0 [31794.388499] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31794.397874] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31794.401967] [31794.409435] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31794.414135] [31794.419350] [31794.419351] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31794.420842] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31794.425191] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31794.430155] ? queued_spin_lock_slowpath+0x16/0x2e0 [31794.448888] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31794.450987] do_raw_spin_lock+0xb5/0xc0 [31794.456202] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31794.458303] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31794.459792] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31794.466918] try_get_cap_refs+0x159/0x880 [ceph] [31794.472218] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31794.479348] __ceph_get_caps+0x4a1/0x760 [ceph] [31794.484213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31794.491343] ? __file_remove_privs+0x89/0x110 [31794.495169] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31794.502292] ? wake_bit_function+0x70/0x70 [31794.507078] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31794.514203] ? ceph_write_iter+0x260/0x770 [ceph] [31794.518812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31794.526890] ceph_write_iter+0x260/0x770 [ceph] [31794.531412] Call Trace: [31794.531414] [31794.537155] ? lock_is_held_type+0xaa/0x120 [31794.541501] ? watchdog_timer_fn+0x1b4/0x220 [31794.548629] vfs_write+0x324/0x4f0 [31794.552717] ? __hrtimer_run_queues+0x1d3/0x360 [31794.559846] ksys_pwrite64+0x65/0xa0 [31794.564543] ? hrtimer_interrupt+0x104/0x240 [31794.571663] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31794.576187] ? __local_bh_enable+0x50/0x60 [31794.578630] do_syscall_64+0x38/0x80 [31794.580643] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31794.584819] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31794.589083] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31794.592481] RIP: 0033:0x7f48f0eeb93b [31794.597003] [31794.600575] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31794.604840] [31794.609969] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31794.614060] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31794.617630] ORIG_RAX: 0000000000000012 [31794.622851] ? lock_acquire+0xf0/0x2b0 [31794.627893] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31794.632851] ? igrab+0x43/0x50 [31794.636420] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31794.638518] ? lock_release+0xc5/0x260 [31794.657254] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31794.659355] _raw_spin_lock+0x30/0x40 [31794.664570] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31794.669876] ? igrab+0x1d/0x50 [31794.673706] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31794.677450] igrab+0x1d/0x50 [31794.684578] [31794.687627] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31794.736769] delayed_work+0x1d8/0x280 [ceph] [31794.741049] ? process_scheduled_works+0x22a/0x510 [31794.745840] process_scheduled_works+0x2a4/0x510 [31794.750456] worker_thread+0x188/0x340 [31794.754205] ? rescuer_thread+0x310/0x310 [31794.758208] kthread+0xf5/0x130 [31794.761346] ? kthread_complete_and_exit+0x20/0x20 [31794.766129] ret_from_fork+0x31/0x40 [31794.769708] ? kthread_complete_and_exit+0x20/0x20 [31794.774491] ret_from_fork_asm+0x11/0x20 [31794.778414] [31822.017183] watchdog: BUG: soft lockup - CPU#0 stuck for 18522s! [dbench:78683] [31822.024486] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31822.029183] watchdog: BUG: soft lockup - CPU#7 stuck for 27519s! [kworker/7:4:78808] [31822.079927] irq event stamp: 54169494 [31822.079928] hardirqs last enabled at (54169493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31822.087656] Modules linked in: [31822.091314] hardirqs last disabled at (54169494): [] sysvec_apic_timer_interrupt+0xe/0x90 [31822.101471] ceph [31822.104523] softirqs last enabled at (54169446): [] __do_softirq+0x2be/0x371 [31822.114246] libceph [31822.116171] softirqs last disabled at (54169441): [] irq_exit_rcu+0xb0/0xd0 [31822.124858] dns_resolver fscache [31822.127040] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31822.135549] netfs veth [31822.138873] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31822.147813] nft_chain_nat xt_MASQUERADE [31822.150256] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31822.157728] nf_nat [31822.161645] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31822.167123] nf_conntrack [31822.169220] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31822.187957] nf_defrag_ipv6 [31822.190579] [31822.190580] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31822.195792] nf_defrag_ipv4 nft_compat [31822.198583] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31822.200074] nf_tables nfnetlink [31822.207198] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31822.210943] bridge [31822.218068] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31822.221290] stp [31822.228414] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31822.230513] llc xfs [31822.237637] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31822.239475] libcrc32c sunrpc [31822.246598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31822.248782] intel_rapl_msr intel_rapl_common [31822.256862] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31822.259824] sb_edac [31822.265578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31822.269929] x86_pkg_temp_thermal coretemp [31822.277052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31822.279237] kvm_intel kvm [31822.286362] Call Trace: [31822.286363] [31822.290452] irqbypass [31822.297583] ? watchdog_timer_fn+0x1b4/0x220 [31822.300288] i2c_i801 iTCO_wdt [31822.302733] ? __hrtimer_run_queues+0x1d3/0x360 [31822.304743] iTCO_vendor_support [31822.307104] ? hrtimer_interrupt+0x104/0x240 [31822.311364] i2c_smbus crct10dif_pclmul [31822.314416] ? __local_bh_enable+0x50/0x60 [31822.318939] joydev crc32_pclmul [31822.322165] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31822.326428] ipmi_ssif ghash_clmulni_intel [31822.330258] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31822.334349] mei_me mei [31822.337584] [31822.342798] lpc_ich sha512_ssse3 [31822.346891] [31822.346892] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31822.351864] acpi_ipmi mfd_core [31822.354314] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31822.356407] wmi ipmi_si [31822.359719] do_raw_spin_lock+0xb5/0xc0 [31822.361814] ipmi_devintf ipmi_msghandler [31822.367121] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31822.370255] ioatdma acpi_pad [31822.375129] try_get_cap_refs+0x159/0x880 [ceph] [31822.377658] acpi_power_meter ixgbe [31822.381492] __ceph_get_caps+0x4a1/0x760 [ceph] [31822.385493] igb mdio [31822.390280] ? __file_remove_privs+0x89/0x110 [31822.393240] crc32c_intel nvme [31822.397862] ? wake_bit_function+0x70/0x70 [31822.401344] ptp nvme_core [31822.405870] ? ceph_write_iter+0x260/0x770 [ceph] [31822.408140] i2c_algo_bit [31822.412490] ceph_write_iter+0x260/0x770 [ceph] [31822.415539] pps_core [31822.419635] ? lock_is_held_type+0xaa/0x120 [31822.422336] dca fuse [31822.427034] vfs_write+0x324/0x4f0 [31822.429650] [31822.434179] ksys_pwrite64+0x65/0xa0 [31822.436445] irq event stamp: 175825680 [31822.440622] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31822.442893] hardirqs last enabled at (175825679): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31822.446292] do_syscall_64+0x38/0x80 [31822.447782] hardirqs last disabled at (175825680): [] sysvec_apic_timer_interrupt+0xe/0x90 [31822.451353] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31822.455096] softirqs last enabled at (175825678): [] __do_softirq+0x2be/0x371 [31822.460228] RIP: 0033:0x7f48f0eeb93b [31822.470472] softirqs last disabled at (175825673): [] irq_exit_rcu+0xb0/0xd0 [31822.474042] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31822.483869] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31822.488914] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31822.497685] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31822.501254] ORIG_RAX: 0000000000000012 [31822.509866] Workqueue: events delayed_work [ceph] [31822.528609] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31822.537985] [31822.543202] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31822.550674] RIP: 0010:_raw_spin_unlock+0x1e/0x40 [31822.554504] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31822.559202] Code: 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b [31822.566325] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31822.567816] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [31822.574941] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31822.579551] [31822.586680] [31822.605412] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [31822.635679] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [31822.642803] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [31822.649927] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [31822.657051] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31822.664174] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31822.672253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31822.677990] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31822.685113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31822.692237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31822.699362] Call Trace: [31822.701806] [31822.703819] ? watchdog_timer_fn+0x1b4/0x220 [31822.708091] ? __hrtimer_run_queues+0x1d3/0x360 [31822.712618] ? hrtimer_interrupt+0x104/0x240 [31822.716885] ? __local_bh_enable+0x50/0x60 [31822.720980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31822.726204] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31822.731161] [31822.733257] [31822.735357] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31822.740673] ? _raw_spin_unlock+0x1e/0x40 [31822.744682] igrab+0x43/0x50 [31822.747559] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31822.752726] delayed_work+0x1d8/0x280 [ceph] [31822.757008] ? process_scheduled_works+0x22a/0x510 [31822.761798] process_scheduled_works+0x2a4/0x510 [31822.766413] worker_thread+0x188/0x340 [31822.770162] ? rescuer_thread+0x310/0x310 [31822.774167] kthread+0xf5/0x130 [31822.777303] ? kthread_complete_and_exit+0x20/0x20 [31822.782089] ret_from_fork+0x31/0x40 [31822.785666] ? kthread_complete_and_exit+0x20/0x20 [31822.790451] ret_from_fork_asm+0x11/0x20 [31822.794372] [31850.017659] watchdog: BUG: soft lockup - CPU#0 stuck for 18548s! [dbench:78683] [31850.024964] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31850.029659] watchdog: BUG: soft lockup - CPU#7 stuck for 27545s! [kworker/7:4:78808] [31850.080415] irq event stamp: 54227680 [31850.080416] hardirqs last enabled at (54227679): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31850.088145] Modules linked in: [31850.091801] hardirqs last disabled at (54227680): [] sysvec_apic_timer_interrupt+0xe/0x90 [31850.101960] ceph [31850.105011] softirqs last enabled at (54227632): [] __do_softirq+0x2be/0x371 [31850.114734] libceph [31850.116660] softirqs last disabled at (54227627): [] irq_exit_rcu+0xb0/0xd0 [31850.125343] dns_resolver [31850.127530] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31850.136053] fscache netfs [31850.138674] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31850.147615] veth nft_chain_nat [31850.150337] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31850.157807] xt_MASQUERADE nf_nat [31850.160948] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31850.166423] nf_conntrack nf_defrag_ipv6 [31850.169735] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31850.188471] nf_defrag_ipv4 nft_compat [31850.192390] [31850.192391] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31850.197608] nf_tables nfnetlink [31850.201350] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31850.202841] bridge stp [31850.209965] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31850.213189] llc xfs [31850.220314] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31850.222758] libcrc32c sunrpc [31850.229882] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31850.232066] intel_rapl_msr intel_rapl_common [31850.239190] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31850.242153] sb_edac [31850.249278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31850.253630] x86_pkg_temp_thermal coretemp [31850.261707] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31850.263891] kvm_intel kvm [31850.269628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31850.273718] irqbypass i2c_i801 [31850.280844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31850.283547] iTCO_wdt [31850.290672] Call Trace: [31850.290674] [31850.293807] iTCO_vendor_support i2c_smbus [31850.300936] ? watchdog_timer_fn+0x1b4/0x220 [31850.303202] crct10dif_pclmul joydev [31850.305648] ? __hrtimer_run_queues+0x1d3/0x360 [31850.307657] crc32_pclmul ipmi_ssif [31850.311752] ? hrtimer_interrupt+0x104/0x240 [31850.316015] ghash_clmulni_intel [31850.319584] ? __local_bh_enable+0x50/0x60 [31850.324108] mei_me mei [31850.327594] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31850.331855] lpc_ich sha512_ssse3 [31850.335080] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31850.339170] acpi_ipmi mfd_core [31850.341614] [31850.346833] wmi ipmi_si [31850.350144] [31850.350145] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31850.355100] ipmi_devintf ipmi_msghandler [31850.358243] ? queued_spin_lock_slowpath+0x16/0x2e0 [31850.360337] ioatdma acpi_pad [31850.362869] do_raw_spin_lock+0xb5/0xc0 [31850.364963] acpi_power_meter [31850.370269] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31850.374271] ixgbe igb [31850.379145] try_get_cap_refs+0x159/0x880 [ceph] [31850.382105] mdio crc32c_intel [31850.385942] __ceph_get_caps+0x4a1/0x760 [ceph] [31850.388901] nvme [31850.393691] ? __file_remove_privs+0x89/0x110 [31850.396060] ptp nvme_core [31850.400672] ? wake_bit_function+0x70/0x70 [31850.403720] i2c_algo_bit pps_core [31850.408249] ? ceph_write_iter+0x260/0x770 [ceph] [31850.410169] dca [31850.414520] ceph_write_iter+0x260/0x770 [ceph] [31850.417224] fuse [31850.421335] ? lock_is_held_type+0xaa/0x120 [31850.424729] irq event stamp: 176047024 [31850.429430] vfs_write+0x324/0x4f0 [31850.431265] hardirqs last enabled at (176047023): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31850.435793] ksys_pwrite64+0x65/0xa0 [31850.437714] hardirqs last disabled at (176047024): [] sysvec_apic_timer_interrupt+0xe/0x90 [31850.441891] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31850.445634] softirqs last enabled at (176047022): [] __do_softirq+0x2be/0x371 [31850.449033] do_syscall_64+0x38/0x80 [31850.459275] softirqs last disabled at (176047017): [] irq_exit_rcu+0xb0/0xd0 [31850.462847] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31850.472656] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31850.477788] RIP: 0033:0x7f48f0eeb93b [31850.486558] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31850.490132] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31850.498728] Workqueue: events delayed_work [ceph] [31850.503772] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31850.513149] [31850.516720] ORIG_RAX: 0000000000000012 [31850.524189] RIP: 0010:lock_release+0x14e/0x260 [31850.542927] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31850.547624] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [31850.552842] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31850.554346] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [31850.558172] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31850.562611] [31850.569734] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31850.588472] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [31850.595597] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31850.600814] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [31850.607944] [31850.609429] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [31850.647233] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [31850.654359] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [31850.661483] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31850.669559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31850.675297] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31850.682420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31850.689545] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31850.696669] Call Trace: [31850.699114] [31850.701123] ? watchdog_timer_fn+0x1b4/0x220 [31850.705388] ? __hrtimer_run_queues+0x1d3/0x360 [31850.709916] ? hrtimer_interrupt+0x104/0x240 [31850.714185] ? __local_bh_enable+0x50/0x60 [31850.718277] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31850.723501] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31850.728459] [31850.730555] [31850.732655] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31850.737959] ? igrab+0x43/0x50 [31850.741011] ? lock_release+0x14e/0x260 [31850.744841] ? lock_release+0xc5/0x260 [31850.748586] _raw_spin_unlock+0x1b/0x40 [31850.752425] igrab+0x43/0x50 [31850.755302] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31850.760467] delayed_work+0x1d8/0x280 [ceph] [31850.764749] ? process_scheduled_works+0x22a/0x510 [31850.769541] process_scheduled_works+0x2a4/0x510 [31850.774156] worker_thread+0x188/0x340 [31850.777904] ? rescuer_thread+0x310/0x310 [31850.781908] kthread+0xf5/0x130 [31850.785065] ? kthread_complete_and_exit+0x20/0x20 [31850.789857] ret_from_fork+0x31/0x40 [31850.793435] ? kthread_complete_and_exit+0x20/0x20 [31850.798221] ret_from_fork_asm+0x11/0x20 [31850.802152] [31878.018134] watchdog: BUG: soft lockup - CPU#0 stuck for 18574s! [dbench:78683] [31878.025435] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31878.030134] watchdog: BUG: soft lockup - CPU#7 stuck for 27571s! [kworker/7:4:78808] [31878.080875] irq event stamp: 54285852 [31878.080876] hardirqs last enabled at (54285851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31878.088607] Modules linked in: [31878.092263] hardirqs last disabled at (54285852): [] sysvec_apic_timer_interrupt+0xe/0x90 [31878.102422] ceph [31878.105472] softirqs last enabled at (54285804): [] __do_softirq+0x2be/0x371 [31878.115195] libceph [31878.117120] softirqs last disabled at (54285799): [] irq_exit_rcu+0xb0/0xd0 [31878.125808] dns_resolver [31878.127996] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31878.136525] fscache netfs [31878.139142] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31878.148086] veth [31878.150807] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31878.158277] nft_chain_nat [31878.160204] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31878.165679] xt_MASQUERADE nf_nat [31878.168384] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31878.187121] nf_conntrack nf_defrag_ipv6 [31878.190433] [31878.190434] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31878.195649] nf_defrag_ipv4 [31878.199568] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31878.201059] nft_compat nf_tables [31878.208181] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31878.210972] nfnetlink [31878.218096] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31878.221409] bridge [31878.228533] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31878.230890] stp llc [31878.238012] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31878.240112] xfs libcrc32c [31878.247235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31878.249418] sunrpc intel_rapl_msr [31878.257497] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31878.260200] intel_rapl_common [31878.265939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31878.269335] sb_edac x86_pkg_temp_thermal [31878.276459] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31878.279527] coretemp [31878.286650] Call Trace: [31878.286652] [31878.290656] kvm_intel kvm [31878.297781] ? watchdog_timer_fn+0x1b4/0x220 [31878.300050] irqbypass i2c_i801 [31878.302495] ? __hrtimer_run_queues+0x1d3/0x360 [31878.304505] iTCO_wdt iTCO_vendor_support [31878.307230] ? hrtimer_interrupt+0x104/0x240 [31878.311490] i2c_smbus crct10dif_pclmul [31878.314628] ? __local_bh_enable+0x50/0x60 [31878.319151] joydev crc32_pclmul [31878.323160] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31878.327420] ipmi_ssif ghash_clmulni_intel [31878.331250] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31878.335342] mei_me mei [31878.338566] [31878.343783] lpc_ich sha512_ssse3 [31878.347874] [31878.352830] acpi_ipmi [31878.355275] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31878.357373] mfd_core wmi [31878.360688] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31878.362780] ipmi_si ipmi_devintf [31878.365141] do_raw_spin_lock+0xb5/0xc0 [31878.370443] ipmi_msghandler ioatdma [31878.373061] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31878.377931] acpi_pad acpi_power_meter [31878.381244] try_get_cap_refs+0x159/0x880 [ceph] [31878.385073] ixgbe [31878.388649] __ceph_get_caps+0x4a1/0x760 [ceph] [31878.393427] igb [31878.397175] ? __file_remove_privs+0x89/0x110 [31878.401781] mdio [31878.403792] ? wake_bit_function+0x70/0x70 [31878.408316] crc32c_intel [31878.410157] ? ceph_write_iter+0x260/0x770 [ceph] [31878.414504] nvme [31878.416428] ceph_write_iter+0x260/0x770 [ceph] [31878.420532] ptp [31878.423159] ? lock_is_held_type+0xaa/0x120 [31878.427850] nvme_core i2c_algo_bit [31878.429779] vfs_write+0x324/0x4f0 [31878.434299] pps_core dca [31878.436142] ksys_pwrite64+0x65/0xa0 [31878.440315] fuse [31878.443814] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31878.447214] irq event stamp: 176268332 [31878.449837] do_syscall_64+0x38/0x80 [31878.453410] hardirqs last enabled at (176268331): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31878.455334] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31878.460464] hardirqs last disabled at (176268332): [] sysvec_apic_timer_interrupt+0xe/0x90 [31878.464210] RIP: 0033:0x7f48f0eeb93b [31878.467780] softirqs last enabled at (176268330): [] __do_softirq+0x2be/0x371 [31878.478024] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31878.483069] softirqs last disabled at (176268325): [] irq_exit_rcu+0xb0/0xd0 [31878.492878] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31878.496451] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31878.505220] ORIG_RAX: 0000000000000012 [31878.523959] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31878.532556] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31878.537774] Workqueue: events delayed_work [ceph] [31878.547151] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31878.550982] [31878.550983] RIP: 0010:ceph_check_delayed_caps+0xb2/0x270 [ceph] [31878.558453] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31878.565576] Code: 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 48 8b 85 78 04 00 00 49 [31878.570274] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31878.577397] RSP: 0018:ffffc90002117db8 EFLAGS: 00000286 [31878.578889] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31878.584800] [31878.584800] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [31878.591929] [31878.610662] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [31878.648051] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [31878.655173] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [31878.662299] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31878.669422] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31878.677499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31878.683237] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31878.690362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31878.697486] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31878.704610] Call Trace: [31878.707054] [31878.709065] ? watchdog_timer_fn+0x1b4/0x220 [31878.713331] ? __hrtimer_run_queues+0x1d3/0x360 [31878.717857] ? hrtimer_interrupt+0x104/0x240 [31878.722125] ? __local_bh_enable+0x50/0x60 [31878.726219] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31878.731442] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31878.736400] [31878.738497] [31878.740595] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31878.745902] ? ceph_check_delayed_caps+0xb2/0x270 [ceph] [31878.751221] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [31878.756543] delayed_work+0x1d8/0x280 [ceph] [31878.760832] ? process_scheduled_works+0x22a/0x510 [31878.765616] process_scheduled_works+0x2a4/0x510 [31878.770231] worker_thread+0x188/0x340 [31878.773981] ? rescuer_thread+0x310/0x310 [31878.777982] kthread+0xf5/0x130 [31878.781120] ? kthread_complete_and_exit+0x20/0x20 [31878.785905] ret_from_fork+0x31/0x40 [31878.789476] ? kthread_complete_and_exit+0x20/0x20 [31878.794259] ret_from_fork_asm+0x11/0x20 [31878.798184] [31906.018611] watchdog: BUG: soft lockup - CPU#0 stuck for 18600s! [dbench:78683] [31906.025913] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31906.030610] watchdog: BUG: soft lockup - CPU#7 stuck for 27597s! [kworker/7:4:78808] [31906.081363] irq event stamp: 54344002 [31906.081365] hardirqs last enabled at (54344001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31906.089094] Modules linked in: [31906.092750] hardirqs last disabled at (54344002): [] sysvec_apic_timer_interrupt+0xe/0x90 [31906.102907] ceph [31906.105958] softirqs last enabled at (54343954): [] __do_softirq+0x2be/0x371 [31906.115683] libceph dns_resolver [31906.117607] softirqs last disabled at (54343949): [] irq_exit_rcu+0xb0/0xd0 [31906.126292] fscache netfs [31906.129604] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31906.138113] veth nft_chain_nat [31906.140816] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31906.149760] xt_MASQUERADE nf_nat [31906.152901] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31906.160369] nf_conntrack nf_defrag_ipv6 [31906.163682] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31906.169157] nf_defrag_ipv4 nft_compat [31906.173075] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31906.191813] nf_tables nfnetlink [31906.195556] [31906.200774] bridge stp [31906.204004] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31906.205499] llc xfs [31906.207941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31906.215067] libcrc32c sunrpc [31906.217250] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31906.224375] intel_rapl_msr intel_rapl_common [31906.227337] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31906.234463] sb_edac x86_pkg_temp_thermal [31906.238813] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31906.245936] coretemp kvm_intel [31906.249941] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31906.257066] kvm irqbypass [31906.260203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31906.268286] i2c_i801 iTCO_wdt [31906.271006] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31906.276749] iTCO_vendor_support i2c_smbus [31906.279798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31906.286921] crct10dif_pclmul joydev [31906.291012] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31906.298138] crc32_pclmul [31906.301707] Call Trace: [31906.301710] [31906.308833] ipmi_ssif ghash_clmulni_intel [31906.311452] ? watchdog_timer_fn+0x1b4/0x220 [31906.313892] mei_me mei [31906.315906] ? __hrtimer_run_queues+0x1d3/0x360 [31906.320008] lpc_ich sha512_ssse3 acpi_ipmi [31906.324292] ? hrtimer_interrupt+0x104/0x240 [31906.326728] mfd_core wmi [31906.331254] ? __local_bh_enable+0x50/0x60 [31906.335432] ipmi_si [31906.339697] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31906.342312] ipmi_devintf ipmi_msghandler [31906.346402] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31906.348587] ioatdma [31906.353804] [31906.357807] acpi_pad acpi_power_meter [31906.362766] [31906.364949] ixgbe igb [31906.367047] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31906.370790] mdio crc32c_intel [31906.372892] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31906.375247] nvme [31906.380552] do_raw_spin_lock+0xb5/0xc0 [31906.383601] ptp nvme_core [31906.388473] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31906.390395] i2c_algo_bit pps_core [31906.394230] try_get_cap_refs+0x159/0x880 [ceph] [31906.396929] dca fuse [31906.401721] __ceph_get_caps+0x4a1/0x760 [ceph] [31906.405111] [31906.405112] irq event stamp: 176489746 [31906.409726] ? __file_remove_privs+0x89/0x110 [31906.412010] hardirqs last enabled at (176489745): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31906.416536] ? wake_bit_function+0x70/0x70 [31906.418026] hardirqs last disabled at (176489746): [] sysvec_apic_timer_interrupt+0xe/0x90 [31906.421772] ? ceph_write_iter+0x260/0x770 [ceph] [31906.426120] softirqs last enabled at (176489744): [] __do_softirq+0x2be/0x371 [31906.436366] ceph_write_iter+0x260/0x770 [ceph] [31906.440454] softirqs last disabled at (176489739): [] irq_exit_rcu+0xb0/0xd0 [31906.450272] ? lock_is_held_type+0xaa/0x120 [31906.454964] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31906.463738] vfs_write+0x324/0x4f0 [31906.468257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31906.476862] ksys_pwrite64+0x65/0xa0 [31906.481036] Workqueue: events delayed_work [ceph] [31906.490412] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31906.493808] [31906.493808] RIP: 0010:lock_acquire+0x86/0x2b0 [31906.501293] do_syscall_64+0x38/0x80 [31906.504859] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [31906.509557] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31906.514686] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [31906.516178] RIP: 0033:0x7f48f0eeb93b [31906.520529] [31906.524101] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31906.542837] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [31906.547880] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31906.553098] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31906.556670] ORIG_RAX: 0000000000000012 [31906.558159] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [31906.576898] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31906.584022] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [31906.589237] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31906.596363] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31906.600194] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31906.607318] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31906.614441] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31906.621566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31906.628691] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31906.635813] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31906.642943] [31906.651016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31906.687451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31906.694575] Call Trace: [31906.697020] [31906.699030] ? watchdog_timer_fn+0x1b4/0x220 [31906.703295] ? __hrtimer_run_queues+0x1d3/0x360 [31906.707822] ? hrtimer_interrupt+0x104/0x240 [31906.712092] ? __local_bh_enable+0x50/0x60 [31906.716184] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31906.721408] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31906.726365] [31906.728464] [31906.730561] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31906.735869] ? lock_acquire+0x86/0x2b0 [31906.739619] ? igrab+0x43/0x50 [31906.742668] ? lock_release+0xc5/0x260 [31906.746414] _raw_spin_lock+0x30/0x40 [31906.750078] ? igrab+0x1d/0x50 [31906.753130] igrab+0x1d/0x50 [31906.756018] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31906.761166] delayed_work+0x1d8/0x280 [ceph] [31906.765448] ? process_scheduled_works+0x22a/0x510 [31906.770238] process_scheduled_works+0x2a4/0x510 [31906.774852] worker_thread+0x188/0x340 [31906.778600] ? rescuer_thread+0x310/0x310 [31906.782604] kthread+0xf5/0x130 [31906.785745] ? kthread_complete_and_exit+0x20/0x20 [31906.790536] ret_from_fork+0x31/0x40 [31906.794115] ? kthread_complete_and_exit+0x20/0x20 [31906.798899] ret_from_fork_asm+0x11/0x20 [31906.802830] [31934.019085] watchdog: BUG: soft lockup - CPU#0 stuck for 18626s! [dbench:78683] [31934.026383] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31934.031085] watchdog: BUG: soft lockup - CPU#7 stuck for 27623s! [kworker/7:4:78808] [31934.081825] irq event stamp: 54402076 [31934.081826] hardirqs last enabled at (54402075): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31934.089556] Modules linked in: [31934.093213] hardirqs last disabled at (54402076): [] sysvec_apic_timer_interrupt+0xe/0x90 [31934.103369] ceph [31934.106419] softirqs last enabled at (54402040): [] __do_softirq+0x2be/0x371 [31934.116146] libceph [31934.118068] softirqs last disabled at (54402035): [] irq_exit_rcu+0xb0/0xd0 [31934.126768] dns_resolver [31934.128955] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31934.137480] fscache netfs [31934.140099] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31934.149043] veth nft_chain_nat [31934.151763] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [31934.159236] xt_MASQUERADE [31934.162373] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [31934.167852] nf_nat [31934.170554] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31934.189293] nf_conntrack [31934.191390] [31934.191390] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31934.196606] nf_defrag_ipv6 [31934.199224] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31934.200715] nf_defrag_ipv4 nft_compat [31934.207840] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31934.210630] nf_tables nfnetlink [31934.217762] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31934.221507] bridge [31934.228631] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31934.231855] stp llc [31934.238980] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31934.241078] xfs libcrc32c [31934.248201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31934.250384] sunrpc intel_rapl_msr [31934.258480] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31934.261184] intel_rapl_common sb_edac [31934.266921] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31934.270320] x86_pkg_temp_thermal coretemp [31934.277442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31934.281187] kvm_intel [31934.288312] Call Trace: [31934.288313] [31934.292402] kvm [31934.299525] ? watchdog_timer_fn+0x1b4/0x220 [31934.301883] irqbypass [31934.304329] ? __hrtimer_run_queues+0x1d3/0x360 [31934.306338] i2c_i801 iTCO_wdt [31934.308179] ? hrtimer_interrupt+0x104/0x240 [31934.312440] iTCO_vendor_support [31934.314796] ? __local_bh_enable+0x50/0x60 [31934.319320] i2c_smbus crct10dif_pclmul [31934.322374] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31934.326635] joydev crc32_pclmul [31934.329861] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31934.333951] ipmi_ssif [31934.337781] [31934.343000] ghash_clmulni_intel [31934.346222] [31934.346223] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31934.351181] mei_me [31934.353542] ? queued_spin_lock_slowpath+0x2b/0x2e0 [31934.355635] mei lpc_ich [31934.358860] do_raw_spin_lock+0xb5/0xc0 [31934.360956] sha512_ssse3 acpi_ipmi [31934.366262] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31934.368357] mfd_core wmi [31934.373232] try_get_cap_refs+0x159/0x880 [ceph] [31934.375773] ipmi_si ipmi_devintf [31934.379614] __ceph_get_caps+0x4a1/0x760 [ceph] [31934.383092] ipmi_msghandler ioatdma [31934.387881] ? __file_remove_privs+0x89/0x110 [31934.390494] acpi_pad [31934.395105] ? wake_bit_function+0x70/0x70 [31934.398415] acpi_power_meter ixgbe [31934.402941] ? ceph_write_iter+0x260/0x770 [ceph] [31934.406509] igb mdio [31934.410860] ceph_write_iter+0x260/0x770 [ceph] [31934.413131] crc32c_intel nvme [31934.417226] ? lock_is_held_type+0xaa/0x120 [31934.420705] ptp nvme_core [31934.425406] vfs_write+0x324/0x4f0 [31934.427673] i2c_algo_bit pps_core [31934.432201] ksys_pwrite64+0x65/0xa0 [31934.435248] dca fuse [31934.439425] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31934.442129] [31934.442130] irq event stamp: 176710890 [31934.445529] do_syscall_64+0x38/0x80 [31934.448924] hardirqs last enabled at (176710889): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31934.452496] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31934.454766] hardirqs last disabled at (176710890): [] sysvec_apic_timer_interrupt+0xe/0x90 [31934.459898] RIP: 0033:0x7f48f0eeb93b [31934.461387] softirqs last enabled at (176710888): [] __do_softirq+0x2be/0x371 [31934.465132] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31934.468703] softirqs last disabled at (176710883): [] irq_exit_rcu+0xb0/0xd0 [31934.478946] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31934.483990] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31934.493802] ORIG_RAX: 0000000000000012 [31934.497373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31934.506144] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31934.524881] Workqueue: events delayed_work [ceph] [31934.533486] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31934.538703] [31934.538704] RIP: 0010:rcu_is_watching+0x1c/0x40 [31934.548081] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31934.551912] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 e8 02 83 e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 [31934.559384] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31934.566508] RSP: 0018:ffffc90002117d10 EFLAGS: 00000286 [31934.571205] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31934.578330] [31934.578331] RAX: 0000000001f06464 RBX: ffff8881a1d34500 RCX: 0000000000000000 [31934.579826] [31934.584343] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [31934.647611] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [31934.654737] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [31934.661860] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31934.668986] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31934.677061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31934.682799] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31934.689923] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31934.697046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31934.704170] Call Trace: [31934.706617] [31934.708627] ? watchdog_timer_fn+0x1b4/0x220 [31934.712893] ? __hrtimer_run_queues+0x1d3/0x360 [31934.717418] ? hrtimer_interrupt+0x104/0x240 [31934.721677] ? __local_bh_enable+0x50/0x60 [31934.725783] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31934.731005] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31934.735963] [31934.738060] [31934.740157] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31934.745469] ? rcu_is_watching+0x1c/0x40 [31934.749405] lock_acquire+0x184/0x2b0 [31934.753064] ? igrab+0x43/0x50 [31934.756113] ? lock_release+0xc5/0x260 [31934.759858] _raw_spin_lock+0x30/0x40 [31934.763522] ? igrab+0x1d/0x50 [31934.766572] igrab+0x1d/0x50 [31934.769450] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31934.774619] delayed_work+0x1d8/0x280 [ceph] [31934.778900] ? process_scheduled_works+0x22a/0x510 [31934.783691] process_scheduled_works+0x2a4/0x510 [31934.788305] worker_thread+0x188/0x340 [31934.792054] ? rescuer_thread+0x310/0x310 [31934.796058] kthread+0xf5/0x130 [31934.799196] ? kthread_complete_and_exit+0x20/0x20 [31934.803979] ret_from_fork+0x31/0x40 [31934.807549] ? kthread_complete_and_exit+0x20/0x20 [31934.812333] ret_from_fork_asm+0x11/0x20 [31934.816258] [31935.545477] rcu: INFO: rcu_preempt self-detected stall on CPU [31935.551234] rcu: 0-....: (19547570 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9782385 [31935.561660] rcu: (t=19998580 jiffies g=1309609 q=578350 ncpus=8) [31935.567743] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31935.576689] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31935.584159] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [31935.589645] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [31935.608390] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31935.613609] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31935.620731] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31935.627857] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31935.634981] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31935.642104] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31935.649230] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31935.657307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31935.663044] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31935.670169] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31935.677293] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31935.684417] Call Trace: [31935.686861] [31935.688872] ? rcu_dump_cpu_stacks+0xd1/0x180 [31935.693231] ? rcu_sched_clock_irq+0x633/0x1390 [31935.697756] ? lock_is_held_type+0xaa/0x120 [31935.701944] ? lock_is_held_type+0xaa/0x120 [31935.706130] ? tick_sched_do_timer+0xa0/0xa0 [31935.710401] ? rcu_read_lock_sched_held+0x56/0x90 [31935.715109] ? tick_sched_do_timer+0xa0/0xa0 [31935.719379] ? update_process_times+0x5f/0x90 [31935.723730] ? tick_sched_handle.isra.16+0x1e/0x50 [31935.728523] ? tick_sched_timer+0x6f/0x80 [31935.732537] ? __hrtimer_run_queues+0x1d3/0x360 [31935.737074] ? hrtimer_interrupt+0x104/0x240 [31935.741339] ? __local_bh_enable+0x50/0x60 [31935.745436] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31935.750657] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31935.755614] [31935.757712] [31935.759809] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31935.765131] ? queued_spin_lock_slowpath+0x16/0x2e0 [31935.770006] do_raw_spin_lock+0xb5/0xc0 [31935.773842] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31935.778646] try_get_cap_refs+0x159/0x880 [ceph] [31935.783289] __ceph_get_caps+0x4a1/0x760 [ceph] [31935.787838] ? __file_remove_privs+0x89/0x110 [31935.792188] ? wake_bit_function+0x70/0x70 [31935.796283] ? ceph_write_iter+0x260/0x770 [ceph] [31935.800995] ceph_write_iter+0x260/0x770 [ceph] [31935.805542] ? lock_is_held_type+0xaa/0x120 [31935.809727] vfs_write+0x324/0x4f0 [31935.813136] ksys_pwrite64+0x65/0xa0 [31935.816706] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31935.821839] do_syscall_64+0x38/0x80 [31935.825418] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31935.830468] RIP: 0033:0x7f48f0eeb93b [31935.834040] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31935.852790] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [31935.860352] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31935.867476] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31935.874601] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31935.881725] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31935.888850] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31935.895985] [31935.898173] Sending NMI from CPU 0 to CPUs 7: [31935.902529] NMI backtrace for cpu 7 [31935.902530] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31935.902532] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31935.902533] Workqueue: events delayed_work [ceph] [31935.902548] RIP: 0010:__lock_acquire+0x111/0x1e20 [31935.902550] Code: 03 85 c0 74 25 41 f6 c5 01 75 1f 49 8d 74 30 d8 0f b7 46 20 25 ff 1f 00 00 44 39 e0 75 0c 48 83 7c 24 08 00 0f 85 0c 07 00 00 <48> 8b 04 24 44 89 e2 66 81 e2 ff 1f 4a 8d 2c 00 0f b7 45 20 4c 89 [31935.902551] RSP: 0018:ffffc90002117c60 EFLAGS: 00000012 [31935.902552] RAX: 0000000000000702 RBX: ffff88810debaa80 RCX: 0000000000000000 [31935.902553] RDX: 0000000000000000 RSI: ffff88810debb528 RDI: ffff8881a1d34518 [31935.902553] RBP: 0000000000000001 R08: 0000000000000078 R09: 0000000000000000 [31935.902554] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [31935.902555] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8881a1d34518 [31935.902555] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31935.902556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31935.902557] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31935.902558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31935.902558] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31935.902559] Call Trace: [31935.902560] [31935.902561] ? nmi_cpu_backtrace+0xad/0x120 [31935.902563] ? nmi_cpu_backtrace_handler+0x11/0x20 [31935.902565] ? nmi_handle+0xd4/0x260 [31935.902569] ? default_do_nmi+0x49/0xf0 [31935.902570] ? exc_nmi+0x11e/0x150 [31935.902572] ? end_repeat_nmi+0x16/0x67 [31935.902575] ? __lock_acquire+0x111/0x1e20 [31935.902577] ? __lock_acquire+0x111/0x1e20 [31935.902578] ? __lock_acquire+0x111/0x1e20 [31935.902580] [31935.902580] [31935.902582] lock_acquire+0xcb/0x2b0 [31935.902583] ? igrab+0x1d/0x50 [31935.902585] ? igrab+0x43/0x50 [31935.902586] ? lock_release+0xc5/0x260 [31935.902588] _raw_spin_lock+0x30/0x40 [31935.902590] ? igrab+0x1d/0x50 [31935.902591] igrab+0x1d/0x50 [31935.902592] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31935.902606] delayed_work+0x1d8/0x280 [ceph] [31935.902618] ? process_scheduled_works+0x22a/0x510 [31935.902621] process_scheduled_works+0x2a4/0x510 [31935.902625] worker_thread+0x188/0x340 [31935.902627] ? rescuer_thread+0x310/0x310 [31935.902629] kthread+0xf5/0x130 [31935.902631] ? kthread_complete_and_exit+0x20/0x20 [31935.902634] ret_from_fork+0x31/0x40 [31935.902636] ? kthread_complete_and_exit+0x20/0x20 [31935.902638] ret_from_fork_asm+0x11/0x20 [31935.902642] [31962.019561] watchdog: BUG: soft lockup - CPU#0 stuck for 18652s! [dbench:78683] [31962.026862] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31962.031560] watchdog: BUG: soft lockup - CPU#7 stuck for 27649s! [kworker/7:4:78808] [31962.082311] irq event stamp: 54459556 [31962.082313] hardirqs last enabled at (54459555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31962.090042] Modules linked in: [31962.093701] hardirqs last disabled at (54459556): [] sysvec_apic_timer_interrupt+0xe/0x90 [31962.103858] ceph [31962.106907] softirqs last enabled at (54459514): [] __do_softirq+0x2be/0x371 [31962.116632] libceph dns_resolver [31962.118556] softirqs last disabled at (54459509): [] irq_exit_rcu+0xb0/0xd0 [31962.127239] fscache netfs [31962.130553] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31962.139060] veth nft_chain_nat [31962.141766] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31962.150710] xt_MASQUERADE [31962.153848] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [31962.161317] nf_nat nf_conntrack [31962.164024] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [31962.169500] nf_defrag_ipv6 nf_defrag_ipv4 [31962.172724] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31962.191460] nft_compat nf_tables [31962.195552] [31962.195553] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31962.200769] nfnetlink bridge [31962.204080] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31962.205571] stp llc [31962.212695] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31962.215659] xfs libcrc32c [31962.222783] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31962.224968] sunrpc intel_rapl_msr [31962.232091] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31962.234796] intel_rapl_common [31962.241920] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31962.245316] sb_edac x86_pkg_temp_thermal [31962.252442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31962.255493] coretemp kvm_intel [31962.263569] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31962.267574] kvm [31962.273311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31962.276449] irqbypass i2c_i801 [31962.283572] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31962.285411] iTCO_wdt [31962.292534] Call Trace: [31962.292536] [31962.295672] iTCO_vendor_support [31962.302797] ? watchdog_timer_fn+0x1b4/0x220 [31962.305068] i2c_smbus [31962.307511] ? __hrtimer_run_queues+0x1d3/0x360 [31962.309520] crct10dif_pclmul joydev [31962.312748] ? hrtimer_interrupt+0x104/0x240 [31962.317010] crc32_pclmul ipmi_ssif [31962.319366] ? __local_bh_enable+0x50/0x60 [31962.323891] ghash_clmulni_intel mei_me [31962.327465] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31962.331726] mei lpc_ich [31962.335210] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31962.339299] sha512_ssse3 acpi_ipmi [31962.343131] [31962.348348] mfd_core wmi [31962.350878] [31962.355837] ipmi_si [31962.359320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31962.361418] ipmi_devintf ipmi_msghandler [31962.364039] ? queued_spin_lock_slowpath+0x18/0x2e0 [31962.366134] ioatdma acpi_pad [31962.368318] do_raw_spin_lock+0xb5/0xc0 [31962.373621] acpi_power_meter ixgbe [31962.377628] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31962.382494] igb [31962.385464] try_get_cap_refs+0x159/0x880 [ceph] [31962.389291] mdio crc32c_intel [31962.392780] __ceph_get_caps+0x4a1/0x760 [ceph] [31962.397558] nvme ptp [31962.399401] ? __file_remove_privs+0x89/0x110 [31962.404008] nvme_core i2c_algo_bit [31962.407058] ? wake_bit_function+0x70/0x70 [31962.411581] pps_core dca [31962.413854] ? ceph_write_iter+0x260/0x770 [ceph] [31962.418202] fuse [31962.421686] ceph_write_iter+0x260/0x770 [ceph] [31962.425777] irq event stamp: 176929978 [31962.428400] ? lock_is_held_type+0xaa/0x120 [31962.433092] hardirqs last enabled at (176929977): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31962.435020] vfs_write+0x324/0x4f0 [31962.439540] hardirqs last disabled at (176929978): [] sysvec_apic_timer_interrupt+0xe/0x90 [31962.443291] ksys_pwrite64+0x65/0xa0 [31962.447461] softirqs last enabled at (176929976): [] __do_softirq+0x2be/0x371 [31962.457706] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31962.461103] softirqs last disabled at (176929971): [] irq_exit_rcu+0xb0/0xd0 [31962.470917] do_syscall_64+0x38/0x80 [31962.474486] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31962.483258] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31962.488387] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31962.496986] RIP: 0033:0x7f48f0eeb93b [31962.500556] Workqueue: events delayed_work [ceph] [31962.509934] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31962.514977] [31962.514978] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [31962.522448] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31962.526017] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [31962.530715] ORIG_RAX: 0000000000000012 [31962.549453] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [31962.550962] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31962.555572] [31962.560789] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31962.579526] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [31962.583357] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31962.588576] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [31962.595701] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31962.597191] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [31962.604314] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31962.611440] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [31962.618567] [31962.625687] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [31962.663492] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31962.671570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31962.677308] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31962.684432] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31962.691556] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31962.698680] Call Trace: [31962.701124] [31962.703134] ? watchdog_timer_fn+0x1b4/0x220 [31962.707398] ? __hrtimer_run_queues+0x1d3/0x360 [31962.711924] ? hrtimer_interrupt+0x104/0x240 [31962.716195] ? __local_bh_enable+0x50/0x60 [31962.720287] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31962.725503] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31962.730461] [31962.732558] [31962.734655] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31962.739970] ? do_raw_spin_lock+0x42/0xc0 [31962.743981] igrab+0x1d/0x50 [31962.746861] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31962.752010] delayed_work+0x1d8/0x280 [ceph] [31962.756289] ? process_scheduled_works+0x22a/0x510 [31962.761081] process_scheduled_works+0x2a4/0x510 [31962.765695] worker_thread+0x188/0x340 [31962.769445] ? rescuer_thread+0x310/0x310 [31962.773448] kthread+0xf5/0x130 [31962.776597] ? kthread_complete_and_exit+0x20/0x20 [31962.781387] ret_from_fork+0x31/0x40 [31962.784967] ? kthread_complete_and_exit+0x20/0x20 [31962.789752] ret_from_fork_asm+0x11/0x20 [31962.793684] [31990.020036] watchdog: BUG: soft lockup - CPU#0 stuck for 18679s! [dbench:78683] [31990.027341] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [31990.032036] watchdog: BUG: soft lockup - CPU#7 stuck for 27676s! [kworker/7:4:78808] [31990.082781] irq event stamp: 54517684 [31990.082782] hardirqs last enabled at (54517683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31990.090511] Modules linked in: [31990.094169] hardirqs last disabled at (54517684): [] sysvec_apic_timer_interrupt+0xe/0x90 [31990.104328] ceph [31990.107377] softirqs last enabled at (54517646): [] __do_softirq+0x2be/0x371 [31990.117102] libceph [31990.119026] softirqs last disabled at (54517641): [] irq_exit_rcu+0xb0/0xd0 [31990.127712] dns_resolver fscache [31990.129894] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31990.138406] netfs [31990.141732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31990.150676] veth nft_chain_nat [31990.152689] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [31990.160178] xt_MASQUERADE nf_nat [31990.163314] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [31990.168790] nf_conntrack nf_defrag_ipv6 [31990.172102] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [31990.190840] nf_defrag_ipv4 nft_compat [31990.194757] [31990.194758] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [31990.199974] nf_tables nfnetlink [31990.203719] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [31990.205210] bridge stp [31990.212332] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [31990.215557] llc [31990.222680] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [31990.225124] xfs libcrc32c [31990.232250] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [31990.234088] sunrpc intel_rapl_msr [31990.241212] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [31990.243916] intel_rapl_common sb_edac [31990.251040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31990.254438] x86_pkg_temp_thermal [31990.262515] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [31990.266259] coretemp kvm_intel [31990.271996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31990.275307] kvm irqbypass [31990.282433] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31990.285568] i2c_i801 iTCO_wdt [31990.292693] Call Trace: [31990.292711] [31990.295431] iTCO_vendor_support i2c_smbus [31990.302555] ? watchdog_timer_fn+0x1b4/0x220 [31990.305606] crct10dif_pclmul joydev [31990.308050] ? __hrtimer_run_queues+0x1d3/0x360 [31990.310060] crc32_pclmul ipmi_ssif [31990.314155] ? hrtimer_interrupt+0x104/0x240 [31990.318432] ghash_clmulni_intel mei_me [31990.322002] ? __local_bh_enable+0x50/0x60 [31990.326526] mei lpc_ich [31990.330013] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31990.334276] sha512_ssse3 acpi_ipmi [31990.338106] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31990.342197] mfd_core wmi [31990.344729] [31990.349945] ipmi_si ipmi_devintf [31990.353435] [31990.358394] ipmi_msghandler ioatdma [31990.361014] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31990.363109] acpi_pad acpi_power_meter [31990.366441] ? queued_spin_lock_slowpath+0x2d/0x2e0 [31990.368536] ixgbe igb [31990.372110] do_raw_spin_lock+0xb5/0xc0 [31990.377411] mdio crc32c_intel [31990.381174] __cap_delay_requeue+0xa7/0x1d0 [ceph] [31990.386042] nvme ptp [31990.388403] try_get_cap_refs+0x159/0x880 [ceph] [31990.392231] nvme_core i2c_algo_bit [31990.395286] __ceph_get_caps+0x4a1/0x760 [ceph] [31990.400066] pps_core dca [31990.402341] ? __file_remove_privs+0x89/0x110 [31990.406948] fuse [31990.410434] ? wake_bit_function+0x70/0x70 [31990.414956] irq event stamp: 177151182 [31990.417576] ? ceph_write_iter+0x260/0x770 [ceph] [31990.421923] hardirqs last enabled at (177151181): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31990.423849] ceph_write_iter+0x260/0x770 [ceph] [31990.427939] hardirqs last disabled at (177151182): [] sysvec_apic_timer_interrupt+0xe/0x90 [31990.431685] ? lock_is_held_type+0xaa/0x120 [31990.436381] softirqs last enabled at (177151180): [] __do_softirq+0x2be/0x371 [31990.446627] vfs_write+0x324/0x4f0 [31990.451148] softirqs last disabled at (177151175): [] irq_exit_rcu+0xb0/0xd0 [31990.460964] ksys_pwrite64+0x65/0xa0 [31990.465137] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [31990.473906] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [31990.477305] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [31990.477306] Workqueue: events delayed_work [ceph] [31990.485903] do_syscall_64+0x38/0x80 [31990.489472] [31990.498850] entry_SYSCALL_64_after_hwframe+0x63/0xcd [31990.503980] RIP: 0010:lock_acquire+0xf0/0x2b0 [31990.511451] RIP: 0033:0x7f48f0eeb93b [31990.516148] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [31990.519727] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [31990.521218] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [31990.526263] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [31990.530614] [31990.534185] ORIG_RAX: 0000000000000012 [31990.552923] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [31990.571659] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [31990.576878] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [31990.582095] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [31990.583586] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [31990.587434] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [31990.594557] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [31990.601682] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [31990.608804] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [31990.615931] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [31990.623054] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [31990.630184] [31990.637303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31990.674682] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [31990.681807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [31990.688930] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [31990.696055] Call Trace: [31990.698501] [31990.700512] ? watchdog_timer_fn+0x1b4/0x220 [31990.704776] ? __hrtimer_run_queues+0x1d3/0x360 [31990.709301] ? hrtimer_interrupt+0x104/0x240 [31990.713562] ? __local_bh_enable+0x50/0x60 [31990.717656] ? __sysvec_apic_timer_interrupt+0x61/0x160 [31990.722879] ? sysvec_apic_timer_interrupt+0x6d/0x90 [31990.727837] [31990.729933] [31990.732032] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [31990.737340] ? lock_acquire+0xf0/0x2b0 [31990.741089] ? igrab+0x43/0x50 [31990.744139] ? lock_release+0xc5/0x260 [31990.747885] _raw_spin_lock+0x30/0x40 [31990.751549] ? igrab+0x1d/0x50 [31990.754599] igrab+0x1d/0x50 [31990.757478] ceph_check_delayed_caps+0xb7/0x270 [ceph] [31990.762627] delayed_work+0x1d8/0x280 [ceph] [31990.766910] ? process_scheduled_works+0x22a/0x510 [31990.771700] process_scheduled_works+0x2a4/0x510 [31990.776314] worker_thread+0x188/0x340 [31990.780066] ? rescuer_thread+0x310/0x310 [31990.784076] kthread+0xf5/0x130 [31990.787215] ? kthread_complete_and_exit+0x20/0x20 [31990.791998] ret_from_fork+0x31/0x40 [31990.795577] ? kthread_complete_and_exit+0x20/0x20 [31990.800360] ret_from_fork_asm+0x11/0x20 [31990.804285] [32018.020512] watchdog: BUG: soft lockup - CPU#0 stuck for 18705s! [dbench:78683] [32018.027821] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32018.032511] watchdog: BUG: soft lockup - CPU#7 stuck for 27702s! [kworker/7:4:78808] [32018.083269] irq event stamp: 54575838 [32018.083271] hardirqs last enabled at (54575837): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32018.091000] Modules linked in: [32018.094657] hardirqs last disabled at (54575838): [] sysvec_apic_timer_interrupt+0xe/0x90 [32018.104816] ceph [32018.107867] softirqs last enabled at (54575800): [] __do_softirq+0x2be/0x371 [32018.117589] libceph [32018.119513] softirqs last disabled at (54575795): [] irq_exit_rcu+0xb0/0xd0 [32018.128199] dns_resolver fscache [32018.130382] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32018.138909] netfs veth [32018.142220] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32018.151166] nft_chain_nat xt_MASQUERADE [32018.153610] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32018.161079] nf_nat nf_conntrack [32018.164999] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32018.170475] nf_defrag_ipv6 nf_defrag_ipv4 [32018.173700] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32018.192437] nft_compat nf_tables [32018.196528] [32018.201744] nfnetlink bridge [32018.205055] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32018.206546] stp llc [32018.209511] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32018.216634] xfs libcrc32c [32018.218819] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32018.225945] sunrpc [32018.228646] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32018.235772] intel_rapl_msr intel_rapl_common [32018.237867] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32018.244993] sb_edac x86_pkg_temp_thermal [32018.249344] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32018.256466] coretemp kvm_intel [32018.260472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32018.268548] kvm irqbypass [32018.271687] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32018.277422] i2c_i801 iTCO_wdt [32018.280128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32018.287253] iTCO_vendor_support [32018.290301] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32018.297426] i2c_smbus crct10dif_pclmul [32018.300649] Call Trace: [32018.300651] [32018.307775] joydev crc32_pclmul [32018.311608] ? watchdog_timer_fn+0x1b4/0x220 [32018.314051] ipmi_ssif ghash_clmulni_intel [32018.316060] ? __hrtimer_run_queues+0x1d3/0x360 [32018.319285] mei_me mei [32018.323551] ? hrtimer_interrupt+0x104/0x240 [32018.327639] lpc_ich sha512_ssse3 [32018.332163] ? __local_bh_enable+0x50/0x60 [32018.334608] acpi_ipmi mfd_core [32018.338873] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32018.342199] wmi [32018.346290] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32018.349428] ipmi_si [32018.354644] [32018.356483] ipmi_devintf ipmi_msghandler [32018.361440] [32018.363625] ioatdma acpi_pad [32018.365722] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32018.369725] acpi_power_meter [32018.371827] ? queued_spin_lock_slowpath+0x18/0x2e0 [32018.374785] ixgbe igb [32018.380092] do_raw_spin_lock+0xb5/0xc0 [32018.383055] mdio crc32c_intel [32018.387927] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32018.390283] nvme [32018.394117] try_get_cap_refs+0x159/0x880 [ceph] [32018.397164] ptp nvme_core [32018.401954] __ceph_get_caps+0x4a1/0x760 [ceph] [32018.403872] i2c_algo_bit [32018.408488] ? __file_remove_privs+0x89/0x110 [32018.411193] pps_core [32018.415722] ? wake_bit_function+0x70/0x70 [32018.418336] dca [32018.422691] ? ceph_write_iter+0x260/0x770 [ceph] [32018.424958] fuse [32018.429051] ceph_write_iter+0x260/0x770 [ceph] [32018.430887] irq event stamp: 177372064 [32018.435606] ? lock_is_held_type+0xaa/0x120 [32018.437526] hardirqs last enabled at (177372063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32018.442052] vfs_write+0x324/0x4f0 [32018.445793] hardirqs last disabled at (177372064): [] sysvec_apic_timer_interrupt+0xe/0x90 [32018.449977] ksys_pwrite64+0x65/0xa0 [32018.460214] softirqs last enabled at (177372062): [] __do_softirq+0x2be/0x371 [32018.463612] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32018.473425] softirqs last disabled at (177372057): [] irq_exit_rcu+0xb0/0xd0 [32018.476997] do_syscall_64+0x38/0x80 [32018.485765] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32018.490915] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32018.499510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32018.503083] RIP: 0033:0x7f48f0eeb93b [32018.512460] Workqueue: events delayed_work [ceph] [32018.517505] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32018.524974] [32018.524975] RIP: 0010:lock_acquire+0x104/0x2b0 [32018.528545] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32018.533243] Code: ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 <48> 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 5f c3 65 8b 05 b2 e6 eb 7e [32018.551980] ORIG_RAX: 0000000000000012 [32018.553470] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [32018.557913] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32018.563133] [32018.563134] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 000000005553a634 [32018.581873] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32018.585703] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32018.590919] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32018.598043] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32018.599535] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32018.606659] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32018.613783] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32018.620914] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32018.628047] [32018.635165] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32018.673922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32018.679659] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32018.686783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32018.693916] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32018.701040] Call Trace: [32018.703485] [32018.705495] ? watchdog_timer_fn+0x1b4/0x220 [32018.709760] ? __hrtimer_run_queues+0x1d3/0x360 [32018.714288] ? hrtimer_interrupt+0x104/0x240 [32018.718555] ? __local_bh_enable+0x50/0x60 [32018.722650] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32018.727875] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32018.732831] [32018.734927] [32018.737027] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32018.742333] ? lock_acquire+0x104/0x2b0 [32018.746170] ? igrab+0x43/0x50 [32018.749219] ? lock_release+0xc5/0x260 [32018.752965] _raw_spin_lock+0x30/0x40 [32018.756632] ? igrab+0x1d/0x50 [32018.759682] igrab+0x1d/0x50 [32018.762559] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32018.767709] delayed_work+0x1d8/0x280 [ceph] [32018.771991] ? process_scheduled_works+0x22a/0x510 [32018.776780] process_scheduled_works+0x2a4/0x510 [32018.781396] worker_thread+0x188/0x340 [32018.785145] ? rescuer_thread+0x310/0x310 [32018.789147] kthread+0xf5/0x130 [32018.792286] ? kthread_complete_and_exit+0x20/0x20 [32018.797072] ret_from_fork+0x31/0x40 [32018.800651] ? kthread_complete_and_exit+0x20/0x20 [32018.805434] ret_from_fork_asm+0x11/0x20 [32018.809366] [32046.020984] watchdog: BUG: soft lockup - CPU#0 stuck for 18731s! [dbench:78683] [32046.028281] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32046.032984] watchdog: BUG: soft lockup - CPU#7 stuck for 27728s! [kworker/7:4:78808] [32046.083722] irq event stamp: 54633906 [32046.083723] hardirqs last enabled at (54633905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32046.091452] Modules linked in: ceph [32046.095110] hardirqs last disabled at (54633906): [] sysvec_apic_timer_interrupt+0xe/0x90 [32046.105269] libceph [32046.108753] softirqs last enabled at (54633868): [] __do_softirq+0x2be/0x371 [32046.118476] dns_resolver fscache [32046.120661] softirqs last disabled at (54633863): [] irq_exit_rcu+0xb0/0xd0 [32046.129346] netfs veth [32046.132671] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32046.141184] nft_chain_nat xt_MASQUERADE [32046.143628] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32046.152570] nf_nat nf_conntrack [32046.156489] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32046.163960] nf_defrag_ipv6 nf_defrag_ipv4 [32046.167183] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32046.172670] nft_compat [32046.176760] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32046.195498] nf_tables nfnetlink [32046.197943] [32046.203161] bridge stp [32046.206383] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32046.207874] llc xfs [32046.210319] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32046.217442] libcrc32c [32046.219627] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32046.226752] sunrpc [32046.229109] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32046.236232] intel_rapl_msr intel_rapl_common [32046.238330] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32046.245454] sb_edac x86_pkg_temp_thermal [32046.249804] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32046.256927] coretemp kvm_intel [32046.260934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32046.269011] kvm irqbypass [32046.272148] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32046.277885] i2c_i801 iTCO_wdt [32046.280589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32046.287712] iTCO_vendor_support i2c_smbus [32046.290764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32046.297889] crct10dif_pclmul joydev [32046.301979] Call Trace: [32046.301980] [32046.309102] crc32_pclmul ipmi_ssif [32046.312677] ? watchdog_timer_fn+0x1b4/0x220 [32046.315117] ghash_clmulni_intel mei_me [32046.317131] ? __hrtimer_run_queues+0x1d3/0x360 [32046.320614] mei lpc_ich [32046.324881] ? hrtimer_interrupt+0x104/0x240 [32046.328708] sha512_ssse3 acpi_ipmi [32046.333233] ? __local_bh_enable+0x50/0x60 [32046.335763] mfd_core wmi [32046.340028] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32046.343509] ipmi_si ipmi_devintf [32046.347602] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32046.350218] ipmi_msghandler ioatdma [32046.355436] [32046.358746] acpi_pad acpi_power_meter [32046.363705] [32046.367275] ixgbe igb [32046.369384] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32046.373124] mdio [32046.375226] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32046.377581] crc32c_intel nvme [32046.382886] do_raw_spin_lock+0xb5/0xc0 [32046.384807] ptp nvme_core [32046.389680] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32046.392729] i2c_algo_bit pps_core [32046.396561] try_get_cap_refs+0x159/0x880 [ceph] [32046.399265] dca fuse [32046.404053] __ceph_get_caps+0x4a1/0x760 [ceph] [32046.407446] [32046.407447] irq event stamp: 177592810 [32046.412061] ? __file_remove_privs+0x89/0x110 [32046.414327] hardirqs last enabled at (177592809): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32046.418852] ? wake_bit_function+0x70/0x70 [32046.420342] hardirqs last disabled at (177592810): [] sysvec_apic_timer_interrupt+0xe/0x90 [32046.424087] ? ceph_write_iter+0x260/0x770 [ceph] [32046.428437] softirqs last enabled at (177592808): [] __do_softirq+0x2be/0x371 [32046.438681] ceph_write_iter+0x260/0x770 [ceph] [32046.442772] softirqs last disabled at (177592803): [] irq_exit_rcu+0xb0/0xd0 [32046.452586] ? lock_is_held_type+0xaa/0x120 [32046.457281] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32046.466054] vfs_write+0x324/0x4f0 [32046.470575] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32046.479176] ksys_pwrite64+0x65/0xa0 [32046.483350] Workqueue: events delayed_work [ceph] [32046.492728] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32046.496124] [32046.496125] RIP: 0010:lock_acquire+0xf0/0x2b0 [32046.503597] do_syscall_64+0x38/0x80 [32046.507165] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32046.511863] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32046.516995] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32046.518484] RIP: 0033:0x7f48f0eeb93b [32046.522836] [32046.522837] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32046.526407] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32046.545144] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32046.550188] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32046.555406] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32046.558977] ORIG_RAX: 0000000000000012 [32046.560466] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32046.567590] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32046.586329] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32046.593453] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32046.598675] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32046.605804] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32046.609634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32046.616758] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32046.623881] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32046.631006] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32046.638131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32046.646214] [32046.653331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32046.696874] Call Trace: [32046.699318] [32046.701331] ? watchdog_timer_fn+0x1b4/0x220 [32046.705595] ? __hrtimer_run_queues+0x1d3/0x360 [32046.710122] ? hrtimer_interrupt+0x104/0x240 [32046.714391] ? __local_bh_enable+0x50/0x60 [32046.718483] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32046.723698] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32046.728671] [32046.730770] [32046.732869] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32046.738177] ? lock_acquire+0xf0/0x2b0 [32046.741925] ? igrab+0x43/0x50 [32046.744975] ? lock_release+0xc5/0x260 [32046.748722] _raw_spin_lock+0x30/0x40 [32046.752390] ? igrab+0x1d/0x50 [32046.755445] igrab+0x1d/0x50 [32046.758324] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32046.763474] delayed_work+0x1d8/0x280 [ceph] [32046.767755] ? process_scheduled_works+0x22a/0x510 [32046.772544] process_scheduled_works+0x2a4/0x510 [32046.777159] worker_thread+0x188/0x340 [32046.780908] ? rescuer_thread+0x310/0x310 [32046.784913] kthread+0xf5/0x130 [32046.788050] ? kthread_complete_and_exit+0x20/0x20 [32046.792835] ret_from_fork+0x31/0x40 [32046.796413] ? kthread_complete_and_exit+0x20/0x20 [32046.801197] ret_from_fork_asm+0x11/0x20 [32046.805119] [32074.021457] watchdog: BUG: soft lockup - CPU#0 stuck for 18757s! [dbench:78683] [32074.028761] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32074.033457] watchdog: BUG: soft lockup - CPU#7 stuck for 27754s! [kworker/7:4:78808] [32074.084211] irq event stamp: 54692024 [32074.084212] hardirqs last enabled at (54692023): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32074.091941] Modules linked in: [32074.095598] hardirqs last disabled at (54692024): [] sysvec_apic_timer_interrupt+0xe/0x90 [32074.105755] ceph [32074.108806] softirqs last enabled at (54691988): [] __do_softirq+0x2be/0x371 [32074.118531] libceph dns_resolver [32074.120454] softirqs last disabled at (54691983): [] irq_exit_rcu+0xb0/0xd0 [32074.129141] fscache netfs [32074.132452] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32074.140959] veth [32074.143663] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32074.152608] nft_chain_nat [32074.154533] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32074.162003] xt_MASQUERADE nf_nat [32074.164708] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32074.170185] nf_conntrack nf_defrag_ipv6 [32074.173497] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32074.192233] nf_defrag_ipv4 nft_compat [32074.196151] [32074.201369] nf_tables nfnetlink [32074.205114] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32074.206604] bridge stp [32074.209826] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32074.216968] llc xfs [32074.219413] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32074.226537] libcrc32c sunrpc [32074.228720] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32074.235856] intel_rapl_msr [32074.238818] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32074.245940] intel_rapl_common [32074.248733] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32074.255858] sb_edac x86_pkg_temp_thermal [32074.258906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32074.266986] coretemp kvm_intel [32074.270988] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32074.276726] kvm irqbypass [32074.279864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32074.286988] i2c_i801 [32074.289693] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32074.296816] iTCO_wdt [32074.299088] Call Trace: [32074.299090] [32074.306211] iTCO_vendor_support [32074.308484] ? watchdog_timer_fn+0x1b4/0x220 [32074.310926] i2c_smbus crct10dif_pclmul [32074.312937] ? __hrtimer_run_queues+0x1d3/0x360 [32074.316161] joydev crc32_pclmul [32074.320427] ? hrtimer_interrupt+0x104/0x240 [32074.324256] ipmi_ssif ghash_clmulni_intel [32074.328779] ? __local_bh_enable+0x50/0x60 [32074.332004] mei_me mei [32074.336269] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32074.340358] lpc_ich sha512_ssse3 [32074.344449] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32074.346892] acpi_ipmi mfd_core [32074.352111] [32074.355437] wmi ipmi_si [32074.360396] [32074.363534] ipmi_devintf [32074.365631] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32074.368160] ipmi_msghandler [32074.370262] ? queued_spin_lock_slowpath+0x18/0x2e0 [32074.372877] ioatdma acpi_pad [32074.378183] do_raw_spin_lock+0xb5/0xc0 [32074.381058] acpi_power_meter [32074.385929] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32074.388891] ixgbe igb [32074.392726] try_get_cap_refs+0x159/0x880 [ceph] [32074.395688] mdio crc32c_intel [32074.400476] __ceph_get_caps+0x4a1/0x760 [ceph] [32074.402828] nvme [32074.407443] ? __file_remove_privs+0x89/0x110 [32074.410492] ptp [32074.415014] ? wake_bit_function+0x70/0x70 [32074.416938] nvme_core i2c_algo_bit [32074.421293] ? ceph_write_iter+0x260/0x770 [ceph] [32074.423141] pps_core dca [32074.427235] ceph_write_iter+0x260/0x770 [ceph] [32074.430719] fuse [32074.435419] ? lock_is_held_type+0xaa/0x120 [32074.438034] irq event stamp: 177813556 [32074.442560] vfs_write+0x324/0x4f0 [32074.444480] hardirqs last enabled at (177813555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32074.448665] ksys_pwrite64+0x65/0xa0 [32074.452403] hardirqs last disabled at (177813556): [] sysvec_apic_timer_interrupt+0xe/0x90 [32074.455800] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32074.466045] softirqs last enabled at (177813554): [] __do_softirq+0x2be/0x371 [32074.469616] do_syscall_64+0x38/0x80 [32074.479427] softirqs last disabled at (177813549): [] irq_exit_rcu+0xb0/0xd0 [32074.484559] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32074.493328] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32074.496900] RIP: 0033:0x7f48f0eeb93b [32074.505496] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32074.510541] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32074.519916] Workqueue: events delayed_work [ceph] [32074.523487] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32074.530959] [32074.530960] RIP: 0010:lock_acquire+0xf0/0x2b0 [32074.549696] ORIG_RAX: 0000000000000012 [32074.554394] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32074.559611] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32074.561102] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32074.565454] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32074.569282] [32074.569283] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32074.588021] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32074.595146] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32074.600362] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32074.607486] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32074.608979] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32074.616102] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32074.623231] [32074.630349] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32074.668171] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32074.676250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32074.681986] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32074.689111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32074.696236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32074.703359] Call Trace: [32074.705806] [32074.707816] ? watchdog_timer_fn+0x1b4/0x220 [32074.712080] ? __hrtimer_run_queues+0x1d3/0x360 [32074.716605] ? hrtimer_interrupt+0x104/0x240 [32074.720868] ? __local_bh_enable+0x50/0x60 [32074.724960] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32074.730183] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32074.735151] [32074.737247] [32074.739344] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32074.744654] ? lock_acquire+0xf0/0x2b0 [32074.748402] ? igrab+0x43/0x50 [32074.751452] ? lock_release+0xc5/0x260 [32074.755197] _raw_spin_lock+0x30/0x40 [32074.758864] ? igrab+0x1d/0x50 [32074.761913] igrab+0x1d/0x50 [32074.764794] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32074.769941] delayed_work+0x1d8/0x280 [ceph] [32074.774222] ? process_scheduled_works+0x22a/0x510 [32074.779012] process_scheduled_works+0x2a4/0x510 [32074.783628] worker_thread+0x188/0x340 [32074.787377] ? rescuer_thread+0x310/0x310 [32074.791380] kthread+0xf5/0x130 [32074.794518] ? kthread_complete_and_exit+0x20/0x20 [32074.799302] ret_from_fork+0x31/0x40 [32074.802882] ? kthread_complete_and_exit+0x20/0x20 [32074.807667] ret_from_fork_asm+0x11/0x20 [32074.811599] [32102.021929] watchdog: BUG: soft lockup - CPU#0 stuck for 18783s! [dbench:78683] [32102.029230] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32102.033929] watchdog: BUG: soft lockup - CPU#7 stuck for 27780s! [kworker/7:4:78808] [32102.084673] irq event stamp: 54750142 [32102.084675] hardirqs last enabled at (54750141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32102.092403] Modules linked in: ceph [32102.096059] hardirqs last disabled at (54750142): [] sysvec_apic_timer_interrupt+0xe/0x90 [32102.106217] libceph [32102.109701] softirqs last enabled at (54750104): [] __do_softirq+0x2be/0x371 [32102.119425] dns_resolver [32102.121611] softirqs last disabled at (54750099): [] irq_exit_rcu+0xb0/0xd0 [32102.130293] fscache netfs [32102.132911] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32102.141423] veth nft_chain_nat [32102.144126] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32102.153070] xt_MASQUERADE nf_nat [32102.156207] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32102.163677] nf_conntrack [32102.166990] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32102.172467] nf_defrag_ipv6 nf_defrag_ipv4 [32102.175085] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32102.193822] nft_compat nf_tables [32102.197912] [32102.203129] nfnetlink bridge [32102.206441] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32102.207932] stp llc [32102.210896] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32102.218019] xfs libcrc32c [32102.220204] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32102.227329] sunrpc [32102.230033] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32102.237156] intel_rapl_msr intel_rapl_common [32102.239253] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32102.246378] sb_edac x86_pkg_temp_thermal [32102.250728] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32102.257853] coretemp kvm_intel [32102.261856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32102.269933] kvm irqbypass [32102.273072] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32102.278809] i2c_i801 [32102.281512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32102.288638] iTCO_wdt [32102.290909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32102.298032] iTCO_vendor_support [32102.300302] Call Trace: [32102.307444] i2c_smbus crct10dif_pclmul [32102.310670] [32102.313113] joydev crc32_pclmul [32102.316944] ? watchdog_timer_fn+0x1b4/0x220 [32102.318953] ipmi_ssif [32102.322179] ? __hrtimer_run_queues+0x1d3/0x360 [32102.326442] ghash_clmulni_intel mei_me [32102.328801] ? hrtimer_interrupt+0x104/0x240 [32102.333327] mei lpc_ich sha512_ssse3 [32102.337164] ? __local_bh_enable+0x50/0x60 [32102.341427] acpi_ipmi [32102.345086] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32102.349174] mfd_core [32102.351531] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32102.356749] wmi ipmi_si [32102.359021] [32102.363977] ipmi_devintf ipmi_msghandler [32102.366508] [32102.366510] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32102.368613] ioatdma acpi_pad [32102.372623] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32102.374716] acpi_power_meter [32102.380022] do_raw_spin_lock+0xb5/0xc0 [32102.382985] ixgbe igb [32102.387855] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32102.390820] mdio crc32c_intel [32102.394651] try_get_cap_refs+0x159/0x880 [ceph] [32102.397007] nvme ptp [32102.401796] __ceph_get_caps+0x4a1/0x760 [ceph] [32102.404841] nvme_core i2c_algo_bit [32102.409456] ? __file_remove_privs+0x89/0x110 [32102.411724] pps_core dca [32102.416247] ? wake_bit_function+0x70/0x70 [32102.419732] fuse [32102.424084] ? ceph_write_iter+0x260/0x770 [ceph] [32102.426700] irq event stamp: 178034272 [32102.430790] ceph_write_iter+0x260/0x770 [ceph] [32102.432714] hardirqs last enabled at (178034271): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32102.437414] ? lock_is_held_type+0xaa/0x120 [32102.441157] hardirqs last disabled at (178034272): [] sysvec_apic_timer_interrupt+0xe/0x90 [32102.445683] vfs_write+0x324/0x4f0 [32102.455923] softirqs last enabled at (178034270): [] __do_softirq+0x2be/0x371 [32102.460106] ksys_pwrite64+0x65/0xa0 [32102.469912] softirqs last disabled at (178034265): [] irq_exit_rcu+0xb0/0xd0 [32102.473326] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32102.482097] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32102.485671] do_syscall_64+0x38/0x80 [32102.494266] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32102.499398] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32102.508774] Workqueue: events delayed_work [ceph] [32102.512346] RIP: 0033:0x7f48f0eeb93b [32102.519817] [32102.519817] RIP: 0010:lock_release+0x77/0x260 [32102.524860] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32102.529557] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [32102.533127] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32102.534618] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [32102.538971] ORIG_RAX: 0000000000000012 [32102.557707] [32102.557708] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [32102.576444] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32102.581661] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [32102.586879] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32102.590709] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [32102.592201] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32102.599332] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [32102.606459] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32102.613582] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32102.620705] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32102.627831] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32102.634960] [32102.642079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32102.679460] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32102.686584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32102.693707] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32102.700832] Call Trace: [32102.703278] [32102.705287] ? watchdog_timer_fn+0x1b4/0x220 [32102.709551] ? __hrtimer_run_queues+0x1d3/0x360 [32102.714077] ? hrtimer_interrupt+0x104/0x240 [32102.718337] ? __local_bh_enable+0x50/0x60 [32102.722431] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32102.727656] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32102.732613] [32102.734709] [32102.736809] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32102.742113] ? igrab+0x43/0x50 [32102.745166] ? lock_release+0x77/0x260 [32102.748920] _raw_spin_unlock+0x1b/0x40 [32102.752756] igrab+0x43/0x50 [32102.755634] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32102.760783] delayed_work+0x1d8/0x280 [ceph] [32102.765064] ? process_scheduled_works+0x22a/0x510 [32102.769854] process_scheduled_works+0x2a4/0x510 [32102.774471] worker_thread+0x188/0x340 [32102.778218] ? rescuer_thread+0x310/0x310 [32102.782222] kthread+0xf5/0x130 [32102.785360] ? kthread_complete_and_exit+0x20/0x20 [32102.790144] ret_from_fork+0x31/0x40 [32102.793722] ? kthread_complete_and_exit+0x20/0x20 [32102.798507] ret_from_fork_asm+0x11/0x20 [32102.802431] [32130.022400] watchdog: BUG: soft lockup - CPU#0 stuck for 18809s! [dbench:78683] [32130.029702] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32130.034400] watchdog: BUG: soft lockup - CPU#7 stuck for 27806s! [kworker/7:4:78808] [32130.085142] irq event stamp: 54808316 [32130.085144] hardirqs last enabled at (54808315): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32130.092873] Modules linked in: [32130.096530] hardirqs last disabled at (54808316): [] sysvec_apic_timer_interrupt+0xe/0x90 [32130.106688] ceph [32130.109739] softirqs last enabled at (54808278): [] __do_softirq+0x2be/0x371 [32130.119462] libceph [32130.121388] softirqs last disabled at (54808273): [] irq_exit_rcu+0xb0/0xd0 [32130.130076] dns_resolver fscache [32130.132265] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32130.140792] netfs veth [32130.144103] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32130.153046] nft_chain_nat xt_MASQUERADE [32130.155492] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32130.162961] nf_nat nf_conntrack [32130.166880] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32130.172357] nf_defrag_ipv6 nf_defrag_ipv4 [32130.175580] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32130.194319] nft_compat nf_tables [32130.198409] [32130.198410] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32130.203626] nfnetlink [32130.206938] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32130.208429] bridge stp [32130.215551] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32130.217909] llc xfs [32130.225033] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32130.227477] libcrc32c sunrpc [32130.234603] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32130.236796] intel_rapl_msr [32130.243919] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32130.246883] intel_rapl_common sb_edac [32130.254007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32130.256797] x86_pkg_temp_thermal coretemp [32130.264874] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32130.268619] kvm_intel [32130.274357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32130.278447] kvm irqbypass [32130.285571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32130.287928] i2c_i801 iTCO_wdt [32130.295053] Call Trace: [32130.295054] [32130.297758] iTCO_vendor_support i2c_smbus [32130.304882] ? watchdog_timer_fn+0x1b4/0x220 [32130.307933] crct10dif_pclmul joydev [32130.310378] ? __hrtimer_run_queues+0x1d3/0x360 [32130.312386] crc32_pclmul ipmi_ssif [32130.316480] ? hrtimer_interrupt+0x104/0x240 [32130.320742] ghash_clmulni_intel mei_me [32130.324312] ? __local_bh_enable+0x50/0x60 [32130.328835] mei lpc_ich [32130.332322] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32130.336583] sha512_ssse3 acpi_ipmi [32130.340415] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32130.344506] mfd_core wmi [32130.347037] [32130.352254] ipmi_si ipmi_devintf [32130.355738] [32130.355739] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32130.360696] ipmi_msghandler ioatdma [32130.363316] ? queued_spin_lock_slowpath+0x18/0x2e0 [32130.365410] acpi_pad acpi_power_meter [32130.368723] do_raw_spin_lock+0xb5/0xc0 [32130.370819] ixgbe [32130.376122] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32130.379693] igb mdio [32130.384565] try_get_cap_refs+0x159/0x880 [ceph] [32130.388308] crc32c_intel nvme [32130.392143] __ceph_get_caps+0x4a1/0x760 [ceph] [32130.394150] ptp nvme_core [32130.398938] ? __file_remove_privs+0x89/0x110 [32130.401203] i2c_algo_bit pps_core [32130.405816] ? wake_bit_function+0x70/0x70 [32130.408866] dca fuse [32130.413393] ? ceph_write_iter+0x260/0x770 [ceph] [32130.416093] [32130.420445] ceph_write_iter+0x260/0x770 [ceph] [32130.423842] irq event stamp: 178255070 [32130.427937] ? lock_is_held_type+0xaa/0x120 [32130.430202] hardirqs last enabled at (178255069): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32130.434903] vfs_write+0x324/0x4f0 [32130.436392] hardirqs last disabled at (178255070): [] sysvec_apic_timer_interrupt+0xe/0x90 [32130.440919] ksys_pwrite64+0x65/0xa0 [32130.444659] softirqs last enabled at (178255068): [] __do_softirq+0x2be/0x371 [32130.448836] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32130.459085] softirqs last disabled at (178255063): [] irq_exit_rcu+0xb0/0xd0 [32130.462480] do_syscall_64+0x38/0x80 [32130.472290] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32130.475862] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32130.484631] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32130.489762] RIP: 0033:0x7f48f0eeb93b [32130.498359] Workqueue: events delayed_work [ceph] [32130.501930] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32130.511307] [32130.511307] RIP: 0010:lock_acquire+0xf0/0x2b0 [32130.516351] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32130.523823] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32130.527394] ORIG_RAX: 0000000000000012 [32130.532091] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32130.550827] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32130.552319] [32130.552320] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32130.556669] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32130.561887] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32130.580626] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32130.584455] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32130.589674] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32130.589675] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32130.596799] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32130.598293] [32130.605413] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32130.664608] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32130.672685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32130.678422] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32130.685546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32130.692669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32130.699805] Call Trace: [32130.702246] [32130.704259] ? watchdog_timer_fn+0x1b4/0x220 [32130.708532] ? __hrtimer_run_queues+0x1d3/0x360 [32130.713084] ? hrtimer_interrupt+0x104/0x240 [32130.717352] ? __local_bh_enable+0x50/0x60 [32130.721446] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32130.726672] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32130.731636] [32130.733734] [32130.735830] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32130.741139] ? lock_acquire+0xf0/0x2b0 [32130.744889] ? igrab+0x43/0x50 [32130.747947] ? lock_release+0xc5/0x260 [32130.751694] _raw_spin_lock+0x30/0x40 [32130.755357] ? igrab+0x1d/0x50 [32130.758408] igrab+0x1d/0x50 [32130.761287] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32130.766437] delayed_work+0x1d8/0x280 [ceph] [32130.770717] ? process_scheduled_works+0x22a/0x510 [32130.775507] process_scheduled_works+0x2a4/0x510 [32130.780124] worker_thread+0x188/0x340 [32130.783873] ? rescuer_thread+0x310/0x310 [32130.787883] kthread+0xf5/0x130 [32130.791022] ? kthread_complete_and_exit+0x20/0x20 [32130.795808] ret_from_fork+0x31/0x40 [32130.799386] ? kthread_complete_and_exit+0x20/0x20 [32130.804170] ret_from_fork_asm+0x11/0x20 [32130.808100] [32130.909781] rcu: INFO: rcu_preempt self-detected stall on CPU [32130.915520] rcu: 0-....: (19738250 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9878190 [32130.925939] rcu: (t=20193941 jiffies g=1309609 q=583248 ncpus=8) [32130.932024] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32130.940967] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32130.948439] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32130.953923] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32130.972661] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32130.977878] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32130.985003] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32130.992126] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32130.999250] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32131.006374] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32131.013500] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32131.021575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32131.027314] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32131.034437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32131.041562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32131.048686] Call Trace: [32131.051132] [32131.053143] ? rcu_dump_cpu_stacks+0xd1/0x180 [32131.057502] ? rcu_sched_clock_irq+0x633/0x1390 [32131.062035] ? lock_is_held_type+0xaa/0x120 [32131.066220] ? lock_is_held_type+0xaa/0x120 [32131.070407] ? tick_sched_do_timer+0xa0/0xa0 [32131.074679] ? rcu_read_lock_sched_held+0x56/0x90 [32131.079389] ? tick_sched_do_timer+0xa0/0xa0 [32131.083657] ? update_process_times+0x5f/0x90 [32131.088008] ? tick_sched_handle.isra.16+0x1e/0x50 [32131.092812] ? tick_sched_timer+0x6f/0x80 [32131.096825] ? __hrtimer_run_queues+0x1d3/0x360 [32131.101362] ? hrtimer_interrupt+0x104/0x240 [32131.105626] ? __local_bh_enable+0x50/0x60 [32131.109724] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32131.114944] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32131.119902] [32131.121998] [32131.124103] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32131.129417] ? queued_spin_lock_slowpath+0x16/0x2e0 [32131.134292] do_raw_spin_lock+0xb5/0xc0 [32131.138128] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32131.142933] try_get_cap_refs+0x159/0x880 [ceph] [32131.147575] __ceph_get_caps+0x4a1/0x760 [ceph] [32131.152126] ? __file_remove_privs+0x89/0x110 [32131.156486] ? wake_bit_function+0x70/0x70 [32131.160589] ? ceph_write_iter+0x260/0x770 [ceph] [32131.165308] ceph_write_iter+0x260/0x770 [ceph] [32131.169857] ? lock_is_held_type+0xaa/0x120 [32131.174041] vfs_write+0x324/0x4f0 [32131.177450] ksys_pwrite64+0x65/0xa0 [32131.181030] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32131.186171] do_syscall_64+0x38/0x80 [32131.189750] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32131.194816] RIP: 0033:0x7f48f0eeb93b [32131.198389] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32131.217125] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [32131.224683] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32131.231813] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32131.238941] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32131.246065] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32131.253187] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32131.260324] [32131.262505] Sending NMI from CPU 0 to CPUs 7: [32131.266859] NMI backtrace for cpu 7 [32131.266861] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32131.266862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32131.266863] Workqueue: events delayed_work [ceph] [32131.266878] RIP: 0010:__lock_acquire+0x141/0x1e20 [32131.266880] Code: e2 66 81 e2 ff 1f 4a 8d 2c 00 0f b7 45 20 4c 89 7d 10 66 25 00 e0 09 d0 31 d2 66 89 45 20 48 8b 84 24 98 00 00 00 48 89 45 08 <48> 8b 44 24 08 48 89 45 18 8b b3 3c 0a 00 00 65 8b 05 51 fc ea 7e [32131.266881] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [32131.266882] RAX: ffffffff813b940d RBX: ffff88810debaa80 RCX: 0000000000000000 [32131.266883] RDX: 0000000000000000 RSI: ffff88810debb528 RDI: ffff8881a1d34518 [32131.266884] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [32131.266885] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [32131.266885] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8881a1d34518 [32131.266886] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32131.266887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32131.266888] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32131.266889] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32131.266889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32131.266890] Call Trace: [32131.266891] [32131.266892] ? nmi_cpu_backtrace+0xad/0x120 [32131.266895] ? nmi_cpu_backtrace_handler+0x11/0x20 [32131.266896] ? nmi_handle+0xd4/0x260 [32131.266900] ? default_do_nmi+0x49/0xf0 [32131.266902] ? exc_nmi+0x11e/0x150 [32131.266903] ? end_repeat_nmi+0x16/0x67 [32131.266906] ? igrab+0x1d/0x50 [32131.266908] ? __lock_acquire+0x141/0x1e20 [32131.266910] ? __lock_acquire+0x141/0x1e20 [32131.266911] ? __lock_acquire+0x141/0x1e20 [32131.266912] [32131.266913] [32131.266915] lock_acquire+0xcb/0x2b0 [32131.266916] ? igrab+0x1d/0x50 [32131.266918] ? igrab+0x43/0x50 [32131.266919] ? lock_release+0xc5/0x260 [32131.266920] _raw_spin_lock+0x30/0x40 [32131.266922] ? igrab+0x1d/0x50 [32131.266923] igrab+0x1d/0x50 [32131.266925] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32131.266938] delayed_work+0x1d8/0x280 [ceph] [32131.266952] ? process_scheduled_works+0x22a/0x510 [32131.266954] process_scheduled_works+0x2a4/0x510 [32131.266958] worker_thread+0x188/0x340 [32131.266960] ? rescuer_thread+0x310/0x310 [32131.266963] kthread+0xf5/0x130 [32131.266965] ? kthread_complete_and_exit+0x20/0x20 [32131.266967] ret_from_fork+0x31/0x40 [32131.266969] ? kthread_complete_and_exit+0x20/0x20 [32131.266971] ret_from_fork_asm+0x11/0x20 [32131.266975] [32158.022870] watchdog: BUG: soft lockup - CPU#0 stuck for 18835s! [dbench:78683] [32158.030173] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32158.034869] watchdog: BUG: soft lockup - CPU#7 stuck for 27832s! [kworker/7:4:78808] [32158.085621] irq event stamp: 54865822 [32158.093352] Modules linked in: ceph libceph [32158.097010] hardirqs last enabled at (54865821): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32158.101188] dns_resolver fscache netfs [32158.111345] hardirqs last disabled at (54865822): [] sysvec_apic_timer_interrupt+0xe/0x90 [32158.115174] veth nft_chain_nat xt_MASQUERADE nf_nat [32158.124901] softirqs last enabled at (54865792): [] __do_softirq+0x2be/0x371 [32158.129857] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [32158.138543] softirqs last disabled at (54865787): [] irq_exit_rcu+0xb0/0xd0 [32158.146453] bridge stp [32158.154966] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32158.157409] llc xfs [32158.166354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32158.168546] libcrc32c sunrpc intel_rapl_msr [32158.176017] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32158.176018] intel_rapl_common sb_edac x86_pkg_temp_thermal [32158.185758] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32158.185759] coretemp [32158.191321] kvm_intel kvm [32158.210060] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32158.212331] irqbypass i2c_i801 [32158.215036] [32158.215036] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32158.220268] iTCO_wdt [32158.223408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32158.224898] iTCO_vendor_support i2c_smbus [32158.232023] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32158.234292] crct10dif_pclmul joydev [32158.241417] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32158.245508] crc32_pclmul ipmi_ssif [32158.252630] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32158.256203] ghash_clmulni_intel mei_me [32158.263326] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32158.266809] mei lpc_ich [32158.273934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32158.277764] sha512_ssse3 acpi_ipmi [32158.285842] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32158.288373] mfd_core wmi [32158.294111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32158.297596] ipmi_si [32158.304718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32158.307337] ipmi_devintf ipmi_msghandler [32158.314462] Call Trace: [32158.316644] ioatdma [32158.323771] [32158.327772] acpi_pad acpi_power_meter [32158.330217] ? watchdog_timer_fn+0x1b4/0x220 [32158.332399] ixgbe igb [32158.334414] ? __hrtimer_run_queues+0x1d3/0x360 [32158.338156] mdio crc32c_intel [32158.342424] ? hrtimer_interrupt+0x104/0x240 [32158.344778] nvme [32158.349300] ? __local_bh_enable+0x50/0x60 [32158.352352] ptp [32158.356618] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32158.358555] nvme_core [32158.362649] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32158.364484] i2c_algo_bit pps_core [32158.369704] [32158.372061] dca fuse [32158.377017] [32158.380414] [32158.380415] irq event stamp: 178473844 [32158.382514] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32158.384783] hardirqs last enabled at (178473843): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32158.386884] ? queued_spin_lock_slowpath+0x16/0x2e0 [32158.388370] hardirqs last disabled at (178473844): [] sysvec_apic_timer_interrupt+0xe/0x90 [32158.392118] do_raw_spin_lock+0xb5/0xc0 [32158.397418] softirqs last enabled at (178473842): [] __do_softirq+0x2be/0x371 [32158.407665] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32158.412550] softirqs last disabled at (178473833): [] irq_exit_rcu+0xb0/0xd0 [32158.422364] try_get_cap_refs+0x159/0x880 [ceph] [32158.426193] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32158.434969] __ceph_get_caps+0x4a1/0x760 [ceph] [32158.439749] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32158.439750] Workqueue: events delayed_work [ceph] [32158.448350] ? __file_remove_privs+0x89/0x110 [32158.452956] [32158.452957] RIP: 0010:lock_acquire+0xf0/0x2b0 [32158.462335] ? wake_bit_function+0x70/0x70 [32158.466859] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32158.474332] ? ceph_write_iter+0x260/0x770 [ceph] [32158.479026] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32158.483377] ceph_write_iter+0x260/0x770 [ceph] [32158.484868] [32158.484868] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32158.489221] ? lock_is_held_type+0xaa/0x120 [32158.493309] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32158.512049] vfs_write+0x324/0x4f0 [32158.516745] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32158.521965] ksys_pwrite64+0x65/0xa0 [32158.526486] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32158.527978] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32158.535101] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32158.539281] do_syscall_64+0x38/0x80 [32158.546403] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32158.549801] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32158.556924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32158.560494] RIP: 0033:0x7f48f0eeb93b [32158.567617] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32158.572750] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32158.579873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32158.583444] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32158.591521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32158.596564] ORIG_RAX: 0000000000000012 [32158.602303] Call Trace: [32158.605874] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32158.612999] [32158.631734] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32158.638861] ? watchdog_timer_fn+0x1b4/0x220 [32158.644078] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32158.651202] ? __hrtimer_run_queues+0x1d3/0x360 [32158.655033] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32158.657480] ? hrtimer_interrupt+0x104/0x240 [32158.664601] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32158.666610] ? __local_bh_enable+0x50/0x60 [32158.673740] [32158.678001] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32158.719670] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32158.724626] [32158.726724] [32158.728822] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32158.734129] ? lock_acquire+0xf0/0x2b0 [32158.737879] ? igrab+0x43/0x50 [32158.740930] ? lock_release+0xc5/0x260 [32158.744674] _raw_spin_lock+0x30/0x40 [32158.748332] ? igrab+0x1d/0x50 [32158.751381] igrab+0x1d/0x50 [32158.754277] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32158.759426] delayed_work+0x1d8/0x280 [ceph] [32158.763709] ? process_scheduled_works+0x22a/0x510 [32158.768499] process_scheduled_works+0x2a4/0x510 [32158.773113] worker_thread+0x188/0x340 [32158.776862] ? rescuer_thread+0x310/0x310 [32158.780865] kthread+0xf5/0x130 [32158.784005] ? kthread_complete_and_exit+0x20/0x20 [32158.788788] ret_from_fork+0x31/0x40 [32158.792359] ? kthread_complete_and_exit+0x20/0x20 [32158.797143] ret_from_fork_asm+0x11/0x20 [32158.801066] [32186.023336] watchdog: BUG: soft lockup - CPU#0 stuck for 18861s! [dbench:78683] [32186.030634] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32186.035335] watchdog: BUG: soft lockup - CPU#7 stuck for 27858s! [kworker/7:4:78808] [32186.086074] irq event stamp: 54923884 [32186.086075] hardirqs last enabled at (54923883): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32186.093807] Modules linked in: ceph [32186.097463] hardirqs last disabled at (54923884): [] sysvec_apic_timer_interrupt+0xe/0x90 [32186.107620] libceph [32186.111105] softirqs last enabled at (54923856): [] __do_softirq+0x2be/0x371 [32186.120828] dns_resolver fscache [32186.123019] softirqs last disabled at (54923851): [] irq_exit_rcu+0xb0/0xd0 [32186.131705] netfs veth [32186.135035] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32186.143544] nft_chain_nat xt_MASQUERADE [32186.145990] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32186.154949] nf_nat nf_conntrack [32186.158868] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32186.166338] nf_defrag_ipv6 nf_defrag_ipv4 [32186.169561] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32186.175041] nft_compat nf_tables [32186.179131] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32186.197867] nfnetlink bridge [32186.201178] [32186.201179] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32186.206396] stp llc [32186.209360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32186.210852] xfs [32186.217974] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32186.220160] libcrc32c sunrpc [32186.227283] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32186.229121] intel_rapl_msr intel_rapl_common [32186.236246] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32186.239208] sb_edac x86_pkg_temp_thermal [32186.246334] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32186.250684] coretemp kvm_intel [32186.257807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32186.261811] kvm irqbypass [32186.269889] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32186.273028] i2c_i801 iTCO_wdt [32186.278764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32186.281469] iTCO_vendor_support i2c_smbus [32186.288593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32186.291642] crct10dif_pclmul joydev [32186.298767] Call Trace: [32186.302859] crc32_pclmul [32186.309983] [32186.313552] ipmi_ssif [32186.316014] ? watchdog_timer_fn+0x1b4/0x220 [32186.318631] ghash_clmulni_intel mei_me [32186.320644] ? __hrtimer_run_queues+0x1d3/0x360 [32186.322999] mei lpc_ich [32186.327285] ? hrtimer_interrupt+0x104/0x240 [32186.331112] sha512_ssse3 acpi_ipmi [32186.335635] ? __local_bh_enable+0x50/0x60 [32186.338168] mfd_core wmi [32186.342433] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32186.345915] ipmi_si ipmi_devintf [32186.350019] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32186.352633] ipmi_msghandler ioatdma [32186.357848] [32186.361161] acpi_pad [32186.366118] [32186.366120] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32186.369688] acpi_power_meter [32186.371789] ? queued_spin_lock_slowpath+0x16/0x2e0 [32186.374055] ixgbe [32186.376155] do_raw_spin_lock+0xb5/0xc0 [32186.381458] igb mdio [32186.384422] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32186.389293] crc32c_intel nvme [32186.391306] try_get_cap_refs+0x159/0x880 [ceph] [32186.395135] ptp nvme_core [32186.397410] __ceph_get_caps+0x4a1/0x760 [ceph] [32186.402189] i2c_algo_bit [32186.405244] ? __file_remove_privs+0x89/0x110 [32186.409851] pps_core dca [32186.412554] ? wake_bit_function+0x70/0x70 [32186.417079] fuse [32186.419699] ? ceph_write_iter+0x260/0x770 [ceph] [32186.424045] irq event stamp: 178694622 [32186.426664] ceph_write_iter+0x260/0x770 [ceph] [32186.430754] hardirqs last enabled at (178694621): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32186.432683] ? lock_is_held_type+0xaa/0x120 [32186.437377] hardirqs last disabled at (178694622): [] sysvec_apic_timer_interrupt+0xe/0x90 [32186.441123] vfs_write+0x324/0x4f0 [32186.445645] softirqs last enabled at (178694620): [] __do_softirq+0x2be/0x371 [32186.455893] ksys_pwrite64+0x65/0xa0 [32186.460066] softirqs last disabled at (178694615): [] irq_exit_rcu+0xb0/0xd0 [32186.469875] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32186.473274] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32186.482046] do_syscall_64+0x38/0x80 [32186.485615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32186.494213] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32186.499344] Workqueue: events delayed_work [ceph] [32186.508736] RIP: 0033:0x7f48f0eeb93b [32186.512308] [32186.512309] RIP: 0010:lock_acquire+0xf0/0x2b0 [32186.519781] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32186.524825] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32186.529522] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32186.533093] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32186.534584] ORIG_RAX: 0000000000000012 [32186.538934] [32186.538935] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32186.557670] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32186.576410] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32186.581626] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32186.586843] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32186.590675] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32186.592165] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32186.599291] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32186.606413] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32186.613537] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32186.620663] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32186.627792] [32186.634911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32186.679407] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32186.686531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32186.693653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32186.700779] Call Trace: [32186.703223] [32186.705234] ? watchdog_timer_fn+0x1b4/0x220 [32186.709499] ? __hrtimer_run_queues+0x1d3/0x360 [32186.714025] ? hrtimer_interrupt+0x104/0x240 [32186.718295] ? __local_bh_enable+0x50/0x60 [32186.722387] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32186.727613] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32186.732569] [32186.734666] [32186.736765] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32186.742081] ? lock_acquire+0xf0/0x2b0 [32186.745830] ? igrab+0x43/0x50 [32186.748881] ? lock_release+0xc5/0x260 [32186.752625] _raw_spin_lock+0x30/0x40 [32186.756281] ? igrab+0x1d/0x50 [32186.759332] igrab+0x1d/0x50 [32186.762210] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32186.767359] delayed_work+0x1d8/0x280 [ceph] [32186.771640] ? process_scheduled_works+0x22a/0x510 [32186.776424] process_scheduled_works+0x2a4/0x510 [32186.781038] worker_thread+0x188/0x340 [32186.784786] ? rescuer_thread+0x310/0x310 [32186.788790] kthread+0xf5/0x130 [32186.791929] ? kthread_complete_and_exit+0x20/0x20 [32186.796713] ret_from_fork+0x31/0x40 [32186.800300] ? kthread_complete_and_exit+0x20/0x20 [32186.805084] ret_from_fork_asm+0x11/0x20 [32186.809008] [32214.023803] watchdog: BUG: soft lockup - CPU#0 stuck for 18887s! [dbench:78683] [32214.031107] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32214.035802] watchdog: BUG: soft lockup - CPU#7 stuck for 27884s! [kworker/7:4:78808] [32214.086554] irq event stamp: 54981988 [32214.094286] Modules linked in: ceph libceph dns_resolver fscache netfs [32214.097943] hardirqs last enabled at (54981987): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32214.104474] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [32214.114635] hardirqs last disabled at (54981988): [] sysvec_apic_timer_interrupt+0xe/0x90 [32214.120719] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [32214.130444] softirqs last enabled at (54981960): [] __do_softirq+0x2be/0x371 [32214.137219] bridge stp llc xfs libcrc32c sunrpc [32214.145904] softirqs last disabled at (54981955): [] irq_exit_rcu+0xb0/0xd0 [32214.150515] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [32214.159029] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32214.166672] kvm_intel [32214.175615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32214.175616] kvm irqbypass [32214.177973] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32214.185443] i2c_i801 iTCO_wdt [32214.188148] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32214.193624] iTCO_vendor_support i2c_smbus [32214.196675] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32214.215411] crct10dif_pclmul [32214.219504] [32214.219505] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32214.224720] joydev crc32_pclmul [32214.227684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32214.229177] ipmi_ssif [32214.236318] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32214.239540] ghash_clmulni_intel mei_me [32214.246666] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32214.249022] mei [32214.256147] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32214.259978] lpc_ich sha512_ssse3 [32214.267102] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32214.268938] acpi_ipmi [32214.276063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32214.279374] mfd_core wmi [32214.287452] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32214.289809] ipmi_si [32214.295545] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32214.298163] ipmi_devintf ipmi_msghandler [32214.305288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32214.307478] ioatdma acpi_pad [32214.314605] Call Trace: [32214.314607] [32214.318607] acpi_power_meter [32214.325735] ? watchdog_timer_fn+0x1b4/0x220 [32214.328696] ixgbe igb [32214.331143] ? __hrtimer_run_queues+0x1d3/0x360 [32214.333151] mdio crc32c_intel [32214.336120] ? hrtimer_interrupt+0x104/0x240 [32214.340379] nvme ptp [32214.342736] ? __local_bh_enable+0x50/0x60 [32214.347262] nvme_core [32214.350313] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32214.354578] i2c_algo_bit [32214.356847] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32214.360938] pps_core [32214.363296] [32214.368511] dca fuse [32214.371129] [32214.371130] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32214.376086] [32214.376087] irq event stamp: 178915330 [32214.378362] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32214.380455] hardirqs last enabled at (178915329): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32214.382727] do_raw_spin_lock+0xb5/0xc0 [32214.384823] hardirqs last disabled at (178915330): [] sysvec_apic_timer_interrupt+0xe/0x90 [32214.390129] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32214.391618] softirqs last enabled at (178915328): [] __do_softirq+0x2be/0x371 [32214.395363] try_get_cap_refs+0x159/0x880 [ceph] [32214.400232] softirqs last disabled at (178915323): [] irq_exit_rcu+0xb0/0xd0 [32214.410495] __ceph_get_caps+0x4a1/0x760 [ceph] [32214.414317] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32214.424132] ? __file_remove_privs+0x89/0x110 [32214.428911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32214.437682] ? wake_bit_function+0x70/0x70 [32214.442292] Workqueue: events delayed_work [ceph] [32214.450893] ? ceph_write_iter+0x260/0x770 [ceph] [32214.455415] [32214.455415] RIP: 0010:lock_acquire+0xf0/0x2b0 [32214.464793] ceph_write_iter+0x260/0x770 [ceph] [32214.469143] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32214.476617] ? lock_is_held_type+0xaa/0x120 [32214.480704] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32214.485404] vfs_write+0x324/0x4f0 [32214.490098] [32214.490099] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32214.491595] ksys_pwrite64+0x65/0xa0 [32214.495940] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32214.500466] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32214.519203] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32214.523380] do_syscall_64+0x38/0x80 [32214.528596] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32214.531996] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32214.533486] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32214.540611] RIP: 0033:0x7f48f0eeb93b [32214.544181] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32214.551323] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32214.556452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32214.563577] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32214.567146] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32214.574272] ORIG_RAX: 0000000000000012 [32214.579315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32214.586440] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32214.590011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32214.598089] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32214.616825] Call Trace: [32214.616826] [32214.622564] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32214.627781] ? watchdog_timer_fn+0x1b4/0x220 [32214.634904] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32214.638737] ? __hrtimer_run_queues+0x1d3/0x360 [32214.645859] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32214.652986] ? hrtimer_interrupt+0x104/0x240 [32214.660114] [32214.667233] ? __local_bh_enable+0x50/0x60 [32214.712397] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32214.717620] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32214.722579] [32214.724675] [32214.726773] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32214.732081] ? lock_acquire+0xf0/0x2b0 [32214.735831] ? igrab+0x43/0x50 [32214.738881] ? lock_release+0xc5/0x260 [32214.742628] _raw_spin_lock+0x30/0x40 [32214.746292] ? igrab+0x1d/0x50 [32214.749341] igrab+0x1d/0x50 [32214.752222] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32214.757370] delayed_work+0x1d8/0x280 [ceph] [32214.761651] ? process_scheduled_works+0x22a/0x510 [32214.766442] process_scheduled_works+0x2a4/0x510 [32214.771057] worker_thread+0x188/0x340 [32214.774807] ? rescuer_thread+0x310/0x310 [32214.778816] kthread+0xf5/0x130 [32214.781957] ? kthread_complete_and_exit+0x20/0x20 [32214.786739] ret_from_fork+0x31/0x40 [32214.790321] ? kthread_complete_and_exit+0x20/0x20 [32214.795112] ret_from_fork_asm+0x11/0x20 [32214.799042] [32242.024274] watchdog: BUG: soft lockup - CPU#0 stuck for 18913s! [dbench:78683] [32242.031576] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32242.036274] watchdog: BUG: soft lockup - CPU#7 stuck for 27910s! [kworker/7:4:78808] [32242.087017] irq event stamp: 55040074 [32242.087018] hardirqs last enabled at (55040073): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32242.094746] Modules linked in: [32242.098404] hardirqs last disabled at (55040074): [] sysvec_apic_timer_interrupt+0xe/0x90 [32242.108562] ceph [32242.111613] softirqs last enabled at (55040048): [] __do_softirq+0x2be/0x371 [32242.121336] libceph [32242.123259] softirqs last disabled at (55040043): [] irq_exit_rcu+0xb0/0xd0 [32242.131949] dns_resolver fscache [32242.134138] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32242.142665] netfs [32242.145978] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32242.154920] veth nft_chain_nat [32242.156947] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32242.164419] xt_MASQUERADE nf_nat [32242.167556] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32242.173035] nf_conntrack [32242.176345] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32242.195082] nf_defrag_ipv6 [32242.197700] [32242.197701] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32242.202916] nf_defrag_ipv4 nft_compat [32242.205707] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32242.207198] nf_tables [32242.214323] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32242.218068] nfnetlink bridge [32242.225190] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32242.227548] stp llc [32242.234674] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32242.237638] xfs libcrc32c [32242.244760] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32242.246959] sunrpc intel_rapl_msr [32242.254086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32242.256790] intel_rapl_common sb_edac [32242.264867] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32242.268265] x86_pkg_temp_thermal [32242.274003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32242.277748] coretemp kvm_intel [32242.284870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32242.288183] kvm irqbypass [32242.295305] Call Trace: [32242.295306] [32242.298444] i2c_i801 [32242.305567] ? watchdog_timer_fn+0x1b4/0x220 [32242.308271] iTCO_wdt [32242.310718] ? __hrtimer_run_queues+0x1d3/0x360 [32242.312726] iTCO_vendor_support [32242.315001] ? hrtimer_interrupt+0x104/0x240 [32242.319261] i2c_smbus [32242.321530] ? __local_bh_enable+0x50/0x60 [32242.326055] crct10dif_pclmul [32242.329282] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32242.333544] joydev [32242.335902] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32242.339992] crc32_pclmul [32242.342955] [32242.348173] ipmi_ssif ghash_clmulni_intel [32242.350270] [32242.355227] mei_me mei [32242.357846] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32242.359954] lpc_ich [32242.364046] ? queued_spin_lock_slowpath+0x16/0x2e0 [32242.366139] sha512_ssse3 acpi_ipmi [32242.368586] do_raw_spin_lock+0xb5/0xc0 [32242.373889] mfd_core [32242.376072] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32242.380957] wmi ipmi_si [32242.384446] try_get_cap_refs+0x159/0x880 [ceph] [32242.388274] ipmi_devintf ipmi_msghandler [32242.390550] __ceph_get_caps+0x4a1/0x760 [ceph] [32242.395329] ioatdma acpi_pad [32242.397865] ? __file_remove_privs+0x89/0x110 [32242.402470] acpi_power_meter [32242.406476] ? wake_bit_function+0x70/0x70 [32242.411000] ixgbe igb [32242.413965] ? ceph_write_iter+0x260/0x770 [ceph] [32242.418314] mdio [32242.421277] ceph_write_iter+0x260/0x770 [ceph] [32242.425368] crc32c_intel nvme [32242.427729] ? lock_is_held_type+0xaa/0x120 [32242.432423] ptp nvme_core [32242.434349] vfs_write+0x324/0x4f0 [32242.438872] i2c_algo_bit pps_core [32242.441926] ksys_pwrite64+0x65/0xa0 [32242.446099] dca fuse [32242.448805] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32242.452201] [32242.452202] irq event stamp: 179136168 [32242.455600] do_syscall_64+0x38/0x80 [32242.459169] hardirqs last enabled at (179136167): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32242.461441] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32242.466570] hardirqs last disabled at (179136168): [] sysvec_apic_timer_interrupt+0xe/0x90 [32242.468060] RIP: 0033:0x7f48f0eeb93b [32242.471806] softirqs last enabled at (179136166): [] __do_softirq+0x2be/0x371 [32242.475378] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32242.485622] softirqs last disabled at (179136161): [] irq_exit_rcu+0xb0/0xd0 [32242.490677] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32242.500493] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32242.504065] ORIG_RAX: 0000000000000012 [32242.512833] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32242.512835] Workqueue: events delayed_work [ceph] [32242.531571] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32242.540169] [32242.545388] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32242.554764] RIP: 0010:lock_acquire+0xf0/0x2b0 [32242.558595] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32242.566067] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32242.570763] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32242.577887] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32242.579379] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32242.586502] [32242.586503] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32242.590859] [32242.597978] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32242.654113] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32242.661238] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32242.668361] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32242.675485] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32242.683563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32242.689299] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32242.696423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32242.703547] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32242.710680] Call Trace: [32242.713124] [32242.715137] ? watchdog_timer_fn+0x1b4/0x220 [32242.719402] ? __hrtimer_run_queues+0x1d3/0x360 [32242.723937] ? hrtimer_interrupt+0x104/0x240 [32242.728205] ? __local_bh_enable+0x50/0x60 [32242.732299] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32242.737523] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32242.742480] [32242.744577] [32242.746681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32242.751992] ? lock_acquire+0xf0/0x2b0 [32242.755743] ? igrab+0x43/0x50 [32242.758801] ? lock_release+0xc5/0x260 [32242.762545] _raw_spin_lock+0x30/0x40 [32242.766201] ? igrab+0x1d/0x50 [32242.769251] igrab+0x1d/0x50 [32242.772131] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32242.777279] delayed_work+0x1d8/0x280 [ceph] [32242.781562] ? process_scheduled_works+0x22a/0x510 [32242.786351] process_scheduled_works+0x2a4/0x510 [32242.790969] worker_thread+0x188/0x340 [32242.794714] ? rescuer_thread+0x310/0x310 [32242.798719] kthread+0xf5/0x130 [32242.801857] ? kthread_complete_and_exit+0x20/0x20 [32242.806640] ret_from_fork+0x31/0x40 [32242.810211] ? kthread_complete_and_exit+0x20/0x20 [32242.814996] ret_from_fork_asm+0x11/0x20 [32242.818918] [32270.024747] watchdog: BUG: soft lockup - CPU#0 stuck for 18939s! [dbench:78683] [32270.032055] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32270.036746] watchdog: BUG: soft lockup - CPU#7 stuck for 27936s! [kworker/7:4:78808] [32270.087504] irq event stamp: 55098294 [32270.095235] Modules linked in: ceph libceph dns_resolver fscache netfs [32270.098890] hardirqs last enabled at (55098293): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32270.098895] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [32270.115575] hardirqs last disabled at (55098294): [] sysvec_apic_timer_interrupt+0xe/0x90 [32270.122960] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [32270.132683] softirqs last enabled at (55098266): [] __do_softirq+0x2be/0x371 [32270.132686] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [32270.147453] softirqs last disabled at (55098261): [] irq_exit_rcu+0xb0/0xd0 [32270.154324] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [32270.162836] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32270.168832] irqbypass i2c_i801 [32270.177776] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32270.177777] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32270.180915] iTCO_wdt [32270.188386] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32270.193863] iTCO_vendor_support [32270.196143] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32270.214880] i2c_smbus crct10dif_pclmul [32270.218105] [32270.218105] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32270.223322] joydev crc32_pclmul [32270.227153] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32270.228644] ipmi_ssif [32270.235767] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32270.238992] ghash_clmulni_intel [32270.246115] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32270.248489] mei_me mei [32270.255614] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32270.258838] lpc_ich sha512_ssse3 [32270.265962] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32270.268421] acpi_ipmi mfd_core [32270.275547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32270.278858] wmi ipmi_si [32270.286936] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32270.290074] ipmi_devintf ipmi_msghandler [32270.295809] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32270.298342] ioatdma acpi_pad [32270.305466] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32270.309468] acpi_power_meter [32270.316593] Call Trace: [32270.316595] [32270.319557] ixgbe igb [32270.326684] ? watchdog_timer_fn+0x1b4/0x220 [32270.329645] mdio [32270.332092] ? __hrtimer_run_queues+0x1d3/0x360 [32270.334100] crc32c_intel nvme [32270.336461] ? hrtimer_interrupt+0x104/0x240 [32270.340722] ptp nvme_core [32270.342646] ? __local_bh_enable+0x50/0x60 [32270.347171] i2c_algo_bit [32270.350225] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32270.354484] pps_core [32270.357190] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32270.361280] dca fuse [32270.363898] [32270.369114] [32270.369115] irq event stamp: 179356736 [32270.371386] [32270.371387] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32270.376343] hardirqs last enabled at (179356735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32270.378616] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32270.380710] hardirqs last disabled at (179356736): [] sysvec_apic_timer_interrupt+0xe/0x90 [32270.382202] do_raw_spin_lock+0xb5/0xc0 [32270.385945] softirqs last enabled at (179356734): [] __do_softirq+0x2be/0x371 [32270.388043] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32270.393346] softirqs last disabled at (179356729): [] irq_exit_rcu+0xb0/0xd0 [32270.403595] try_get_cap_refs+0x159/0x880 [ceph] [32270.408462] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32270.418279] __ceph_get_caps+0x4a1/0x760 [ceph] [32270.422103] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32270.430879] ? __file_remove_privs+0x89/0x110 [32270.435659] Workqueue: events delayed_work [ceph] [32270.444255] ? wake_bit_function+0x70/0x70 [32270.448867] [32270.448868] RIP: 0010:lock_acquire+0x4a/0x2b0 [32270.458247] ? ceph_write_iter+0x260/0x770 [ceph] [32270.462768] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [32270.470240] ceph_write_iter+0x260/0x770 [ceph] [32270.474591] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [32270.479291] ? lock_is_held_type+0xaa/0x120 [32270.483378] [32270.483379] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [32270.484872] vfs_write+0x324/0x4f0 [32270.489220] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [32270.493923] ksys_pwrite64+0x65/0xa0 [32270.512655] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [32270.517179] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32270.522397] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [32270.526574] do_syscall_64+0x38/0x80 [32270.528064] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32270.535190] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32270.538585] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32270.545712] RIP: 0033:0x7f48f0eeb93b [32270.549281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32270.556405] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32270.561535] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32270.568659] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32270.572229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32270.579354] ORIG_RAX: 0000000000000012 [32270.584400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32270.592476] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32270.596046] Call Trace: [32270.596048] [32270.601784] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32270.620522] ? watchdog_timer_fn+0x1b4/0x220 [32270.627646] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32270.632865] ? __hrtimer_run_queues+0x1d3/0x360 [32270.639987] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32270.643822] ? hrtimer_interrupt+0x104/0x240 [32270.650943] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32270.658067] ? __local_bh_enable+0x50/0x60 [32270.660515] [32270.662523] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32270.715564] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32270.720520] [32270.722616] [32270.724716] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32270.730023] ? lock_acquire+0x4a/0x2b0 [32270.733773] ? igrab+0x43/0x50 [32270.736824] ? lock_release+0xc5/0x260 [32270.740567] _raw_spin_lock+0x30/0x40 [32270.744225] ? igrab+0x1d/0x50 [32270.747275] igrab+0x1d/0x50 [32270.750153] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32270.755302] delayed_work+0x1d8/0x280 [ceph] [32270.759584] ? process_scheduled_works+0x22a/0x510 [32270.764374] process_scheduled_works+0x2a4/0x510 [32270.768990] worker_thread+0x188/0x340 [32270.772739] ? rescuer_thread+0x310/0x310 [32270.776741] kthread+0xf5/0x130 [32270.779880] ? kthread_complete_and_exit+0x20/0x20 [32270.784665] ret_from_fork+0x31/0x40 [32270.788242] ? kthread_complete_and_exit+0x20/0x20 [32270.793027] ret_from_fork_asm+0x11/0x20 [32270.796959] [32298.025220] watchdog: BUG: soft lockup - CPU#0 stuck for 18965s! [dbench:78683] [32298.032523] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32298.037220] watchdog: BUG: soft lockup - CPU#7 stuck for 27962s! [kworker/7:4:78808] [32298.087964] irq event stamp: 55156342 [32298.095696] Modules linked in: ceph [32298.099353] hardirqs last enabled at (55156341): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32298.099355] libceph dns_resolver fscache netfs [32298.112993] hardirqs last disabled at (55156342): [] sysvec_apic_timer_interrupt+0xe/0x90 [32298.117519] veth nft_chain_nat [32298.127242] softirqs last enabled at (55156316): [] __do_softirq+0x2be/0x371 [32298.127244] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [32298.139064] softirqs last disabled at (55156311): [] irq_exit_rcu+0xb0/0xd0 [32298.147055] nf_tables nfnetlink [32298.155566] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32298.158789] bridge stp [32298.167734] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32298.170178] llc xfs libcrc32c sunrpc [32298.177648] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32298.181307] intel_rapl_msr intel_rapl_common [32298.186784] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32298.186784] sb_edac [32298.191134] x86_pkg_temp_thermal coretemp [32298.209872] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32298.212074] kvm_intel [32298.216164] [32298.216165] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32298.221381] kvm irqbypass [32298.223740] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32298.225231] i2c_i801 iTCO_wdt [32298.232355] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32298.235059] iTCO_vendor_support [32298.242182] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32298.245234] i2c_smbus [32298.252357] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32298.255582] crct10dif_pclmul joydev [32298.262706] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32298.265062] crc32_pclmul ipmi_ssif [32298.272187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32298.275758] ghash_clmulni_intel mei_me [32298.283836] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32298.287320] mei lpc_ich [32298.293058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32298.296904] sha512_ssse3 [32298.304029] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32298.306559] acpi_ipmi mfd_core [32298.313684] Call Trace: [32298.313687] [32298.316301] wmi ipmi_si [32298.323426] ? watchdog_timer_fn+0x1b4/0x220 [32298.326561] ipmi_devintf [32298.329006] ? __hrtimer_run_queues+0x1d3/0x360 [32298.331018] ipmi_msghandler ioatdma [32298.333552] ? hrtimer_interrupt+0x104/0x240 [32298.337812] acpi_pad [32298.340429] ? __local_bh_enable+0x50/0x60 [32298.344953] acpi_power_meter ixgbe [32298.348528] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32298.352788] igb mdio [32298.355060] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32298.359150] crc32c_intel nvme [32298.362633] [32298.367850] ptp nvme_core [32298.370122] [32298.370124] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32298.375079] i2c_algo_bit pps_core [32298.378135] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32298.380228] dca fuse [32298.382933] do_raw_spin_lock+0xb5/0xc0 [32298.385030] [32298.390333] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32298.393729] irq event stamp: 179577590 [32298.398620] try_get_cap_refs+0x159/0x880 [ceph] [32298.400888] hardirqs last enabled at (179577589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32298.404741] __ceph_get_caps+0x4a1/0x760 [ceph] [32298.406227] hardirqs last disabled at (179577590): [] sysvec_apic_timer_interrupt+0xe/0x90 [32298.411016] ? __file_remove_privs+0x89/0x110 [32298.414755] softirqs last enabled at (179577588): [] __do_softirq+0x2be/0x371 [32298.419368] ? wake_bit_function+0x70/0x70 [32298.429620] softirqs last disabled at (179577583): [] irq_exit_rcu+0xb0/0xd0 [32298.434146] ? ceph_write_iter+0x260/0x770 [ceph] [32298.443955] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32298.448306] ceph_write_iter+0x260/0x770 [ceph] [32298.457075] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32298.461170] ? lock_is_held_type+0xaa/0x120 [32298.469765] Workqueue: events delayed_work [ceph] [32298.474464] vfs_write+0x324/0x4f0 [32298.483839] [32298.488367] ksys_pwrite64+0x65/0xa0 [32298.495834] RIP: 0010:do_raw_spin_lock+0xd/0xc0 [32298.500012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32298.504710] Code: 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 <48> 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 [32298.508107] do_syscall_64+0x38/0x80 [32298.509597] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [32298.513185] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32298.517708] [32298.517709] RAX: 00000000dead4ead RBX: ffff8881a1d34478 RCX: 000000005553a634 [32298.522841] RIP: 0033:0x7f48f0eeb93b [32298.541577] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [32298.545149] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32298.550367] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [32298.555409] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32298.556912] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32298.564032] ORIG_RAX: 0000000000000012 [32298.567620] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32298.574745] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32298.593485] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32298.600622] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32298.605843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32298.612966] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32298.616797] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32298.623922] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32298.631045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32298.639123] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32298.646246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32298.651990] [32298.659108] Call Trace: [32298.659109] [32298.701386] ? watchdog_timer_fn+0x1b4/0x220 [32298.705650] ? __hrtimer_run_queues+0x1d3/0x360 [32298.710177] ? hrtimer_interrupt+0x104/0x240 [32298.714448] ? __local_bh_enable+0x50/0x60 [32298.718539] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32298.723755] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32298.728712] [32298.730810] [32298.732913] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32298.738224] ? do_raw_spin_lock+0xd/0xc0 [32298.742147] igrab+0x1d/0x50 [32298.745024] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32298.750174] delayed_work+0x1d8/0x280 [ceph] [32298.754455] ? process_scheduled_works+0x22a/0x510 [32298.759245] process_scheduled_works+0x2a4/0x510 [32298.763860] worker_thread+0x188/0x340 [32298.767626] ? rescuer_thread+0x310/0x310 [32298.771631] kthread+0xf5/0x130 [32298.774770] ? kthread_complete_and_exit+0x20/0x20 [32298.779553] ret_from_fork+0x31/0x40 [32298.783131] ? kthread_complete_and_exit+0x20/0x20 [32298.787916] ret_from_fork_asm+0x11/0x20 [32298.791838] [32326.025697] watchdog: BUG: soft lockup - CPU#0 stuck for 18991s! [dbench:78683] [32326.033004] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32326.037696] watchdog: BUG: soft lockup - CPU#7 stuck for 27988s! [kworker/7:4:78808] [32326.088454] irq event stamp: 55214460 [32326.088455] hardirqs last enabled at (55214459): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32326.096183] Modules linked in: [32326.099840] hardirqs last disabled at (55214460): [] sysvec_apic_timer_interrupt+0xe/0x90 [32326.109998] ceph [32326.113048] softirqs last enabled at (55214432): [] __do_softirq+0x2be/0x371 [32326.122773] libceph [32326.124696] softirqs last disabled at (55214427): [] irq_exit_rcu+0xb0/0xd0 [32326.133380] dns_resolver fscache [32326.135566] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32326.144092] netfs [32326.147405] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32326.156348] veth [32326.158376] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32326.165846] nft_chain_nat [32326.167773] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32326.173248] xt_MASQUERADE nf_nat [32326.175953] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32326.194691] nf_conntrack nf_defrag_ipv6 [32326.198000] [32326.198001] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32326.203220] nf_defrag_ipv4 [32326.207135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32326.208625] nft_compat nf_tables [32326.215749] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32326.218542] nfnetlink bridge [32326.225666] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32326.228975] stp llc [32326.236100] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32326.239064] xfs libcrc32c [32326.246189] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32326.248375] sunrpc [32326.255497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32326.258200] intel_rapl_msr intel_rapl_common [32326.266277] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32326.268376] sb_edac x86_pkg_temp_thermal [32326.274114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32326.277069] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [32326.277089] rcu: 0-....: (19929205 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=9974160 [32326.277094] rcu: 7-....: (19797950 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/19796621 fqs=9974160 [32326.277097] rcu: (detected by 4, t=20389289 jiffies, g=1309609, q=587068 ncpus=8) [32326.277100] Sending NMI from CPU 4 to CPUs 0: [32326.277103] NMI backtrace for cpu 0 [32326.277104] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32326.277106] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32326.277107] RIP: 0010:vprintk_emit+0x1eb/0x340 [32326.277110] Code: 50 41 b8 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 <0f> b6 11 84 d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 [32326.277112] RSP: 0018:ffffc90000003db8 EFLAGS: 00000002 [32326.277114] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [32326.277115] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [32326.277116] RBP: 0000000000000041 R08: 0000000000005ef4 R09: 0000000000000004 [32326.277118] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [32326.277119] R13: 0000000000000000 R14: ffffffff82188830 R15: ffffc90000003e08 [32326.277120] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32326.277122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32326.277123] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32326.277124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32326.277125] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32326.277127] Call Trace: [32326.277128] [32326.277130] ? nmi_cpu_backtrace+0xad/0x120 [32326.277134] ? nmi_cpu_backtrace_handler+0x11/0x20 [32326.277137] ? nmi_handle+0xd4/0x260 [32326.277142] ? default_do_nmi+0x49/0xf0 [32326.277145] ? exc_nmi+0x11e/0x150 [32326.277147] ? end_repeat_nmi+0x16/0x67 [32326.277154] ? vprintk_emit+0x1eb/0x340 [32326.277157] ? vprintk_emit+0x1eb/0x340 [32326.277160] ? vprintk_emit+0x1eb/0x340 [32326.277162] [32326.277163] [32326.277166] _printk+0x5c/0x80 [32326.277170] __show_regs+0x251/0x2e0 [32326.277175] ? touch_softlockup_watchdog+0x20/0x20 [32326.277177] show_regs+0x38/0x60 [32326.277180] watchdog_timer_fn+0x1b4/0x220 [32326.277183] __hrtimer_run_queues+0x1d3/0x360 [32326.277188] hrtimer_interrupt+0x104/0x240 [32326.277190] ? __local_bh_enable+0x50/0x60 [32326.277195] __sysvec_apic_timer_interrupt+0x61/0x160 [32326.277199] sysvec_apic_timer_interrupt+0x6d/0x90 [32326.277201] [32326.277202] [32326.277203] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32326.277206] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32326.277209] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32326.277210] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32326.277212] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32326.277213] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32326.277214] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32326.277215] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32326.277216] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32326.277223] do_raw_spin_lock+0xb5/0xc0 [32326.277227] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32326.277253] try_get_cap_refs+0x159/0x880 [ceph] [32326.277274] __ceph_get_caps+0x4a1/0x760 [ceph] [32326.277294] ? __file_remove_privs+0x89/0x110 [32326.277297] ? wake_bit_function+0x70/0x70 [32326.277301] ? ceph_write_iter+0x260/0x770 [ceph] [32326.277316] ceph_write_iter+0x260/0x770 [ceph] [32326.277334] ? lock_is_held_type+0xaa/0x120 [32326.277339] vfs_write+0x324/0x4f0 [32326.277345] ksys_pwrite64+0x65/0xa0 [32326.277348] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32326.277352] do_syscall_64+0x38/0x80 [32326.277355] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32326.277357] RIP: 0033:0x7f48f0eeb93b [32326.277359] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32326.277361] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [32326.277362] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32326.277364] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32326.277365] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32326.277366] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32326.277367] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32326.277373] [32326.278102] Sending NMI from CPU 4 to CPUs 7: [32326.278106] NMI backtrace for cpu 7 [32326.278107] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32326.278109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32326.278109] Workqueue: events delayed_work [ceph] [32326.278127] RIP: 0010:io_serial_in+0x19/0x20 [32326.278130] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [32326.278131] RSP: 0018:ffffc90000264c98 EFLAGS: 00000002 [32326.278133] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [32326.278133] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [32326.278134] RBP: 000000000000250e R08: 00006880916d5ccc R09: 0000000000000007 [32326.278135] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [32326.278136] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [32326.278136] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32326.278137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32326.278138] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32326.278139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32326.278140] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32326.278140] Call Trace: [32326.278141] [32326.278143] ? nmi_cpu_backtrace+0xad/0x120 [32326.278146] ? nmi_cpu_backtrace_handler+0x11/0x20 [32326.278147] ? nmi_handle+0xd4/0x260 [32326.278151] ? default_do_nmi+0x49/0xf0 [32326.278153] ? exc_nmi+0x11e/0x150 [32326.278154] ? end_repeat_nmi+0x16/0x67 [32326.278158] ? io_serial_in+0x19/0x20 [32326.278160] ? io_serial_in+0x19/0x20 [32326.278162] ? io_serial_in+0x19/0x20 [32326.278164] [32326.278164] [32326.278164] wait_for_lsr+0x45/0x70 [32326.278166] serial8250_console_write+0x341/0x3f0 [32326.278169] ? lock_release+0xc5/0x260 [32326.278172] console_flush_all+0x227/0x5a0 [32326.278174] ? console_flush_all+0x1ff/0x5a0 [32326.278175] ? console_flush_all+0x1ff/0x5a0 [32326.278177] console_unlock+0x7b/0x120 [32326.278180] vprintk_emit+0x246/0x340 [32326.278181] ? irq_work_queue+0x2c/0x50 [32326.278185] ? touch_softlockup_watchdog+0x20/0x20 [32326.278187] _printk+0x5c/0x80 [32326.278190] print_modules+0x6e/0xe0 [32326.278192] watchdog_timer_fn+0x19f/0x220 [32326.278194] __hrtimer_run_queues+0x1d3/0x360 [32326.278197] hrtimer_interrupt+0x104/0x240 [32326.278198] ? __local_bh_enable+0x50/0x60 [32326.278202] __sysvec_apic_timer_interrupt+0x61/0x160 [32326.278204] sysvec_apic_timer_interrupt+0x6d/0x90 [32326.278206] [32326.278206] [32326.278207] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32326.278209] RIP: 0010:lock_acquire+0xf0/0x2b0 [32326.278211] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32326.278212] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32326.278213] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32326.278214] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32326.278215] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32326.278215] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32326.278216] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32326.278219] ? igrab+0x43/0x50 [32326.278221] ? lock_release+0xc5/0x260 [32326.278223] _raw_spin_lock+0x30/0x40 [32326.278225] ? igrab+0x1d/0x50 [32326.278226] igrab+0x1d/0x50 [32326.278227] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32326.278242] delayed_work+0x1d8/0x280 [ceph] [32326.278255] ? process_scheduled_works+0x22a/0x510 [32326.278259] process_scheduled_works+0x2a4/0x510 [32326.278263] worker_thread+0x188/0x340 [32326.278265] ? rescuer_thread+0x310/0x310 [32326.278267] kthread+0xf5/0x130 [32326.278270] ? kthread_complete_and_exit+0x20/0x20 [32326.278273] ret_from_fork+0x31/0x40 [32326.278275] ? kthread_complete_and_exit+0x20/0x20 [32326.278277] ret_from_fork_asm+0x11/0x20 [32326.278282] [32326.278472] coretemp kvm_intel [32326.285597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32326.289602] kvm irqbypass [32326.296724] Call Trace: [32326.296725] [32326.302808] i2c_i801 iTCO_wdt [32326.313227] ? watchdog_timer_fn+0x1b4/0x220 [32326.323816] iTCO_vendor_support i2c_smbus [32326.331378] ? __hrtimer_run_queues+0x1d3/0x360 [32326.335725] crct10dif_pclmul [32326.339212] ? hrtimer_interrupt+0x104/0x240 [32326.348154] joydev crc32_pclmul [32326.355624] ? __local_bh_enable+0x50/0x60 [32326.360063] ipmi_ssif [32326.378802] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32326.384016] ghash_clmulni_intel mei_me [32326.391141] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32326.398266] mei [32326.405389] [32326.412514] lpc_ich sha512_ssse3 [32326.419638] [32326.419639] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32326.427715] acpi_ipmi mfd_core [32326.433457] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32326.440576] wmi ipmi_si [32326.447702] do_raw_spin_lock+0xb5/0xc0 [32326.454825] ipmi_devintf ipmi_msghandler [32326.457271] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32326.459279] ioatdma acpi_pad [32326.463458] try_get_cap_refs+0x159/0x880 [ceph] [32326.468242] acpi_power_meter [32326.471817] __ceph_get_caps+0x4a1/0x760 [ceph] [32326.475642] ixgbe igb [32326.479045] ? __file_remove_privs+0x89/0x110 [32326.482871] mdio crc32c_intel [32326.486702] ? wake_bit_function+0x70/0x70 [32326.490532] nvme [32326.494382] ? ceph_write_iter+0x260/0x770 [ceph] [32326.496478] ptp [32326.498489] ceph_write_iter+0x260/0x770 [ceph] [32326.501538] nvme_core i2c_algo_bit [32326.505113] ? lock_is_held_type+0xaa/0x120 [32326.509894] pps_core [32326.513121] vfs_write+0x324/0x4f0 [32326.517209] dca fuse [32326.521564] ksys_pwrite64+0x65/0xa0 [32326.525650] [32326.525651] irq event stamp: 179798316 [32326.529741] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32326.534785] hardirqs last enabled at (179798315): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32326.539572] do_syscall_64+0x38/0x80 [32326.541666] hardirqs last disabled at (179798316): [] sysvec_apic_timer_interrupt+0xe/0x90 [32326.543764] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32326.548894] softirqs last enabled at (179798314): [] __do_softirq+0x2be/0x371 [32326.554380] RIP: 0033:0x7f48f0eeb93b [32326.573119] softirqs last disabled at (179798309): [] irq_exit_rcu+0xb0/0xd0 [32326.578335] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32326.585459] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32326.592584] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32326.599709] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32326.606834] ORIG_RAX: 0000000000000012 [32326.613958] Workqueue: events delayed_work [ceph] [32326.617788] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32326.622572] [32326.622573] RIP: 0010:lock_acquire+0xf0/0x2b0 [32326.627182] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32326.631706] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32326.636057] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32326.640147] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32326.644846] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32326.649379] [32326.649380] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32326.653554] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32326.656953] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32326.660528] [32326.665654] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32327.488621] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32327.495745] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32327.502871] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32327.510948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32327.516686] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32327.523809] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32327.530934] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32327.538058] Call Trace: [32327.540503] [32327.542515] ? watchdog_timer_fn+0x1b4/0x220 [32327.546787] ? __hrtimer_run_queues+0x1d3/0x360 [32327.551312] ? hrtimer_interrupt+0x104/0x240 [32327.555583] ? __local_bh_enable+0x50/0x60 [32327.559675] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32327.564898] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32327.569855] [32327.571952] [32327.574052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32327.579360] ? lock_acquire+0xf0/0x2b0 [32327.583117] ? igrab+0x43/0x50 [32327.586167] ? lock_release+0xc5/0x260 [32327.589912] _raw_spin_lock+0x30/0x40 [32327.593567] ? igrab+0x1d/0x50 [32327.596620] igrab+0x1d/0x50 [32327.599496] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32327.604648] delayed_work+0x1d8/0x280 [ceph] [32327.608928] ? process_scheduled_works+0x22a/0x510 [32327.613718] process_scheduled_works+0x2a4/0x510 [32327.618335] worker_thread+0x188/0x340 [32327.622083] ? rescuer_thread+0x310/0x310 [32327.626085] kthread+0xf5/0x130 [32327.629225] ? kthread_complete_and_exit+0x20/0x20 [32327.634009] ret_from_fork+0x31/0x40 [32327.637578] ? kthread_complete_and_exit+0x20/0x20 [32327.642363] ret_from_fork_asm+0x11/0x20 [32327.646285] [32354.026173] watchdog: BUG: soft lockup - CPU#0 stuck for 19018s! [dbench:78683] [32354.033474] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32354.038172] watchdog: BUG: soft lockup - CPU#7 stuck for 28015s! [kworker/7:4:78808] [32354.088915] irq event stamp: 55272500 [32354.088917] hardirqs last enabled at (55272499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32354.096644] Modules linked in: [32354.100303] hardirqs last disabled at (55272500): [] sysvec_apic_timer_interrupt+0xe/0x90 [32354.110459] ceph [32354.113510] softirqs last enabled at (55272472): [] __do_softirq+0x2be/0x371 [32354.123234] libceph [32354.125158] softirqs last disabled at (55272467): [] irq_exit_rcu+0xb0/0xd0 [32354.133857] dns_resolver [32354.136045] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32354.144569] fscache [32354.147188] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32354.156133] netfs veth [32354.158318] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32354.165789] nft_chain_nat [32354.168234] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32354.173711] xt_MASQUERADE nf_nat [32354.176415] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32354.195151] nf_conntrack [32354.198463] [32354.198464] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32354.203679] nf_defrag_ipv6 nf_defrag_ipv4 [32354.206297] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32354.207787] nft_compat nf_tables [32354.214913] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32354.219002] nfnetlink bridge [32354.226128] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32354.229437] stp llc [32354.236569] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32354.239535] xfs [32354.246660] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32354.248857] libcrc32c sunrpc [32354.255983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32354.257823] intel_rapl_msr intel_rapl_common [32354.265898] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32354.268863] sb_edac x86_pkg_temp_thermal [32354.274601] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32354.278951] coretemp [32354.286075] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32354.290079] kvm_intel kvm [32354.297205] Call Trace: [32354.297206] [32354.299473] irqbypass i2c_i801 [32354.306599] ? watchdog_timer_fn+0x1b4/0x220 [32354.309303] iTCO_wdt iTCO_vendor_support [32354.311747] ? __hrtimer_run_queues+0x1d3/0x360 [32354.313758] i2c_smbus crct10dif_pclmul [32354.316897] ? hrtimer_interrupt+0x104/0x240 [32354.321158] joydev crc32_pclmul [32354.325164] ? __local_bh_enable+0x50/0x60 [32354.329687] ipmi_ssif ghash_clmulni_intel [32354.333520] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32354.337782] mei_me mei [32354.341006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32354.345096] lpc_ich sha512_ssse3 [32354.349187] [32354.354406] acpi_ipmi mfd_core [32354.356859] [32354.361816] wmi [32354.365127] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32354.367222] ipmi_si ipmi_devintf [32354.370365] ? queued_spin_lock_slowpath+0x18/0x2e0 [32354.372457] ipmi_msghandler ioatdma [32354.374298] do_raw_spin_lock+0xb5/0xc0 [32354.379599] acpi_pad [32354.382912] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32354.387779] acpi_power_meter [32354.391354] try_get_cap_refs+0x159/0x880 [ceph] [32354.395183] ixgbe igb [32354.397457] __ceph_get_caps+0x4a1/0x760 [ceph] [32354.402236] mdio crc32c_intel [32354.405205] ? __file_remove_privs+0x89/0x110 [32354.409811] nvme ptp [32354.412170] ? wake_bit_function+0x70/0x70 [32354.416693] nvme_core i2c_algo_bit [32354.419747] ? ceph_write_iter+0x260/0x770 [ceph] [32354.424095] pps_core [32354.426366] ceph_write_iter+0x260/0x770 [ceph] [32354.430457] dca fuse [32354.433945] ? lock_is_held_type+0xaa/0x120 [32354.438638] [32354.438639] irq event stamp: 180012292 [32354.440910] vfs_write+0x324/0x4f0 [32354.445432] hardirqs last enabled at (180012291): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32354.447707] ksys_pwrite64+0x65/0xa0 [32354.451881] hardirqs last disabled at (180012292): [] sysvec_apic_timer_interrupt+0xe/0x90 [32354.453373] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32354.457116] softirqs last enabled at (180012290): [] __do_softirq+0x2be/0x371 [32354.460515] do_syscall_64+0x38/0x80 [32354.470756] softirqs last disabled at (180012285): [] irq_exit_rcu+0xb0/0xd0 [32354.474328] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32354.484137] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32354.489269] RIP: 0033:0x7f48f0eeb93b [32354.498039] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32354.501612] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32354.510208] Workqueue: events delayed_work [ceph] [32354.515253] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32354.524630] [32354.524630] RIP: 0010:lock_acquire+0xf0/0x2b0 [32354.528201] ORIG_RAX: 0000000000000012 [32354.535671] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32354.554409] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32354.559106] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32354.564324] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32354.565814] [32354.565815] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32354.570166] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32354.573997] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32354.592732] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32354.599863] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32354.605084] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32354.612208] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32354.613704] [32354.620823] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32354.672885] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32354.680961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32354.686701] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32354.693824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32354.700947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32354.708071] Call Trace: [32354.710517] [32354.712528] ? watchdog_timer_fn+0x1b4/0x220 [32354.716793] ? __hrtimer_run_queues+0x1d3/0x360 [32354.721318] ? hrtimer_interrupt+0x104/0x240 [32354.725588] ? __local_bh_enable+0x50/0x60 [32354.729681] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32354.734906] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32354.739868] [32354.741967] [32354.744065] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32354.749375] ? lock_acquire+0xf0/0x2b0 [32354.753124] ? igrab+0x43/0x50 [32354.756174] ? lock_release+0xc5/0x260 [32354.759918] _raw_spin_lock+0x30/0x40 [32354.763577] ? igrab+0x1d/0x50 [32354.766627] igrab+0x1d/0x50 [32354.769504] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32354.774653] delayed_work+0x1d8/0x280 [ceph] [32354.778935] ? process_scheduled_works+0x22a/0x510 [32354.783724] process_scheduled_works+0x2a4/0x510 [32354.788341] worker_thread+0x188/0x340 [32354.792090] ? rescuer_thread+0x310/0x310 [32354.796092] kthread+0xf5/0x130 [32354.799230] ? kthread_complete_and_exit+0x20/0x20 [32354.804015] ret_from_fork+0x31/0x40 [32354.807594] ? kthread_complete_and_exit+0x20/0x20 [32354.812379] ret_from_fork_asm+0x11/0x20 [32354.816302] [32382.026650] watchdog: BUG: soft lockup - CPU#0 stuck for 19044s! [dbench:78683] [32382.033951] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32382.038649] watchdog: BUG: soft lockup - CPU#7 stuck for 28041s! [kworker/7:4:78808] [32382.089393] irq event stamp: 55330690 [32382.089394] hardirqs last enabled at (55330689): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32382.097123] Modules linked in: [32382.100781] hardirqs last disabled at (55330690): [] sysvec_apic_timer_interrupt+0xe/0x90 [32382.110938] ceph [32382.113989] softirqs last enabled at (55330662): [] __do_softirq+0x2be/0x371 [32382.123714] libceph [32382.125636] softirqs last disabled at (55330657): [] irq_exit_rcu+0xb0/0xd0 [32382.134324] dns_resolver [32382.136507] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32382.145016] fscache [32382.147635] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32382.156577] netfs veth [32382.158762] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32382.166234] nft_chain_nat xt_MASQUERADE [32382.168679] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32382.174154] nf_nat nf_conntrack [32382.178071] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32382.196809] nf_defrag_ipv6 nf_defrag_ipv4 [32382.200044] [32382.205259] nft_compat nf_tables [32382.209350] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32382.210842] nfnetlink bridge [32382.214152] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32382.221276] stp llc [32382.224241] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32382.231363] xfs libcrc32c [32382.233547] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32382.240674] sunrpc [32382.243377] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32382.250500] intel_rapl_msr intel_rapl_common [32382.252598] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32382.259721] sb_edac x86_pkg_temp_thermal [32382.264071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32382.272151] coretemp kvm_intel [32382.276154] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32382.281891] kvm irqbypass [32382.285045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32382.292170] i2c_i801 iTCO_wdt [32382.294875] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32382.301999] iTCO_vendor_support [32382.305051] Call Trace: [32382.312172] i2c_smbus crct10dif_pclmul [32382.315399] [32382.317842] joydev crc32_pclmul [32382.321674] ? watchdog_timer_fn+0x1b4/0x220 [32382.323684] ipmi_ssif ghash_clmulni_intel [32382.326907] ? __hrtimer_run_queues+0x1d3/0x360 [32382.331172] mei_me mei [32382.335266] ? hrtimer_interrupt+0x104/0x240 [32382.339785] lpc_ich [32382.342230] ? __local_bh_enable+0x50/0x60 [32382.346494] sha512_ssse3 acpi_ipmi [32382.348681] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32382.352771] mfd_core [32382.356254] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32382.361471] wmi ipmi_si [32382.363741] [32382.368698] ipmi_devintf ipmi_msghandler [32382.371229] [32382.371231] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32382.373342] ioatdma acpi_pad [32382.377352] ? queued_spin_lock_slowpath+0x18/0x2e0 [32382.379445] acpi_power_meter ixgbe [32382.384752] do_raw_spin_lock+0xb5/0xc0 [32382.387714] igb mdio [32382.392586] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32382.396069] crc32c_intel nvme [32382.399901] try_get_cap_refs+0x159/0x880 [ceph] [32382.402170] ptp [32382.406961] __ceph_get_caps+0x4a1/0x760 [ceph] [32382.410005] nvme_core i2c_algo_bit [32382.414619] ? __file_remove_privs+0x89/0x110 [32382.416453] pps_core dca [32382.420976] ? wake_bit_function+0x70/0x70 [32382.424462] fuse [32382.428815] ? ceph_write_iter+0x260/0x770 [ceph] [32382.431428] irq event stamp: 180233004 [32382.435521] ceph_write_iter+0x260/0x770 [ceph] [32382.437443] hardirqs last enabled at (180233003): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32382.442144] ? lock_is_held_type+0xaa/0x120 [32382.445884] hardirqs last disabled at (180233004): [] sysvec_apic_timer_interrupt+0xe/0x90 [32382.450411] vfs_write+0x324/0x4f0 [32382.460652] softirqs last enabled at (180233002): [] __do_softirq+0x2be/0x371 [32382.464837] ksys_pwrite64+0x65/0xa0 [32382.474642] softirqs last disabled at (180232997): [] irq_exit_rcu+0xb0/0xd0 [32382.478051] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32382.486819] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32382.490390] do_syscall_64+0x38/0x80 [32382.498987] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32382.504117] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32382.513495] Workqueue: events delayed_work [ceph] [32382.517066] RIP: 0033:0x7f48f0eeb93b [32382.524538] [32382.524539] RIP: 0010:lock_acquire+0xf0/0x2b0 [32382.529583] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32382.534279] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32382.537850] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32382.539340] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32382.543691] ORIG_RAX: 0000000000000012 [32382.543692] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32382.562427] [32382.562427] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32382.581165] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32382.586384] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32382.591601] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32382.595431] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32382.602555] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32382.604052] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32382.611178] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32382.618303] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32382.625434] [32382.632552] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32382.678435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32382.684172] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32382.691296] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32382.698420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32382.705544] Call Trace: [32382.707989] [32382.710000] ? watchdog_timer_fn+0x1b4/0x220 [32382.714264] ? __hrtimer_run_queues+0x1d3/0x360 [32382.718789] ? hrtimer_interrupt+0x104/0x240 [32382.723055] ? __local_bh_enable+0x50/0x60 [32382.727152] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32382.732367] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32382.737336] [32382.739430] [32382.741530] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32382.746845] ? lock_acquire+0xf0/0x2b0 [32382.750595] ? igrab+0x43/0x50 [32382.753646] ? lock_release+0xc5/0x260 [32382.757391] _raw_spin_lock+0x30/0x40 [32382.761057] ? igrab+0x1d/0x50 [32382.764105] igrab+0x1d/0x50 [32382.766986] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32382.772135] delayed_work+0x1d8/0x280 [ceph] [32382.776416] ? process_scheduled_works+0x22a/0x510 [32382.781206] process_scheduled_works+0x2a4/0x510 [32382.785820] worker_thread+0x188/0x340 [32382.789570] ? rescuer_thread+0x310/0x310 [32382.793574] kthread+0xf5/0x130 [32382.796720] ? kthread_complete_and_exit+0x20/0x20 [32382.801504] ret_from_fork+0x31/0x40 [32382.805082] ? kthread_complete_and_exit+0x20/0x20 [32382.809867] ret_from_fork_asm+0x11/0x20 [32382.813798] [32410.027127] watchdog: BUG: soft lockup - CPU#0 stuck for 19070s! [dbench:78683] [32410.034431] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32410.039126] watchdog: BUG: soft lockup - CPU#7 stuck for 28067s! [kworker/7:4:78808] [32410.089871] irq event stamp: 55388834 [32410.089872] hardirqs last enabled at (55388833): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32410.097603] Modules linked in: [32410.101260] hardirqs last disabled at (55388834): [] sysvec_apic_timer_interrupt+0xe/0x90 [32410.111417] ceph [32410.114468] softirqs last enabled at (55388806): [] __do_softirq+0x2be/0x371 [32410.124193] libceph [32410.126116] softirqs last disabled at (55388801): [] irq_exit_rcu+0xb0/0xd0 [32410.134800] dns_resolver fscache [32410.136984] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32410.145494] netfs veth [32410.148823] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32410.157767] nft_chain_nat xt_MASQUERADE [32410.160211] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32410.167682] nf_nat [32410.171600] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32410.177078] nf_conntrack nf_defrag_ipv6 [32410.179174] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32410.197911] nf_defrag_ipv4 [32410.201829] [32410.201830] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32410.207046] nft_compat nf_tables [32410.209839] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32410.211329] nfnetlink bridge [32410.218453] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32410.221762] stp llc [32410.228888] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32410.231852] xfs libcrc32c [32410.238976] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32410.241159] sunrpc intel_rapl_msr [32410.248285] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32410.250988] intel_rapl_common sb_edac [32410.258111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32410.261523] x86_pkg_temp_thermal [32410.269604] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32410.273348] coretemp [32410.279084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32410.282396] kvm_intel kvm [32410.289524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32410.291805] irqbypass i2c_i801 [32410.298932] Call Trace: [32410.298933] [32410.301636] iTCO_wdt iTCO_vendor_support [32410.308761] ? watchdog_timer_fn+0x1b4/0x220 [32410.311898] i2c_smbus crct10dif_pclmul [32410.314344] ? __hrtimer_run_queues+0x1d3/0x360 [32410.316352] joydev [32410.320360] ? hrtimer_interrupt+0x104/0x240 [32410.324621] crc32_pclmul [32410.328451] ? __local_bh_enable+0x50/0x60 [32410.332976] ipmi_ssif ghash_clmulni_intel [32410.335077] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32410.339338] mei_me [32410.341957] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32410.346045] mei [32410.350137] [32410.355354] lpc_ich [32410.357451] [32410.357453] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32410.362408] sha512_ssse3 acpi_ipmi [32410.364249] ? queued_spin_lock_slowpath+0x18/0x2e0 [32410.366344] mfd_core wmi [32410.368530] do_raw_spin_lock+0xb5/0xc0 [32410.370625] ipmi_si ipmi_devintf [32410.375929] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32410.379413] ipmi_msghandler ioatdma [32410.384286] try_get_cap_refs+0x159/0x880 [ceph] [32410.386901] acpi_pad acpi_power_meter [32410.390735] __ceph_get_caps+0x4a1/0x760 [ceph] [32410.394044] ixgbe igb [32410.398831] ? __file_remove_privs+0x89/0x110 [32410.402397] mdio crc32c_intel [32410.407008] ? wake_bit_function+0x70/0x70 [32410.410751] nvme ptp [32410.415278] ? ceph_write_iter+0x260/0x770 [ceph] [32410.417633] nvme_core i2c_algo_bit [32410.421984] ceph_write_iter+0x260/0x770 [ceph] [32410.425034] pps_core dca [32410.429130] ? lock_is_held_type+0xaa/0x120 [32410.431397] fuse [32410.436097] vfs_write+0x324/0x4f0 [32410.439578] irq event stamp: 180453684 [32410.444106] ksys_pwrite64+0x65/0xa0 [32410.446720] hardirqs last enabled at (180453683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32410.450897] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32410.452821] hardirqs last disabled at (180453684): [] sysvec_apic_timer_interrupt+0xe/0x90 [32410.456219] do_syscall_64+0x38/0x80 [32410.459962] softirqs last enabled at (180453682): [] __do_softirq+0x2be/0x371 [32410.463534] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32410.473777] softirqs last disabled at (180453677): [] irq_exit_rcu+0xb0/0xd0 [32410.478908] RIP: 0033:0x7f48f0eeb93b [32410.488719] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32410.492291] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32410.501061] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32410.506103] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32410.514701] Workqueue: events delayed_work [ceph] [32410.518272] ORIG_RAX: 0000000000000012 [32410.527650] [32410.527651] RIP: 0010:_raw_spin_unlock+0xf/0x40 [32410.546387] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32410.553859] Code: 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 <48> 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff [32410.559077] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32410.563774] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [32410.567604] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32410.569096] [32410.569097] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [32410.573620] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32410.580744] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34500 [32410.599480] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32410.606606] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [32410.611828] [32410.618947] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32410.665366] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32410.672490] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32410.680569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32410.686306] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32410.693429] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32410.700555] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32410.707678] Call Trace: [32410.710122] [32410.712133] ? watchdog_timer_fn+0x1b4/0x220 [32410.716398] ? __hrtimer_run_queues+0x1d3/0x360 [32410.720924] ? hrtimer_interrupt+0x104/0x240 [32410.725193] ? __local_bh_enable+0x50/0x60 [32410.729288] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32410.734528] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32410.739487] [32410.741582] [32410.743681] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32410.748987] ? igrab+0x43/0x50 [32410.752044] ? _raw_spin_unlock+0xf/0x40 [32410.755962] igrab+0x43/0x50 [32410.758840] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32410.763989] delayed_work+0x1d8/0x280 [ceph] [32410.768270] ? process_scheduled_works+0x22a/0x510 [32410.773060] process_scheduled_works+0x2a4/0x510 [32410.777677] worker_thread+0x188/0x340 [32410.781426] ? rescuer_thread+0x310/0x310 [32410.785438] kthread+0xf5/0x130 [32410.788575] ? kthread_complete_and_exit+0x20/0x20 [32410.793359] ret_from_fork+0x31/0x40 [32410.796938] ? kthread_complete_and_exit+0x20/0x20 [32410.801724] ret_from_fork_asm+0x11/0x20 [32410.805654] [32438.027603] watchdog: BUG: soft lockup - CPU#0 stuck for 19096s! [dbench:78683] [32438.034910] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32438.039603] watchdog: BUG: soft lockup - CPU#7 stuck for 28093s! [kworker/7:4:78808] [32438.090359] irq event stamp: 55447068 [32438.098090] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [32438.101748] hardirqs last enabled at (55447067): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32438.109919] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [32438.120079] hardirqs last disabled at (55447068): [] sysvec_apic_timer_interrupt+0xe/0x90 [32438.125814] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [32438.135539] softirqs last enabled at (55447040): [] __do_softirq+0x2be/0x371 [32438.135542] llc xfs libcrc32c sunrpc intel_rapl_msr [32438.150653] softirqs last disabled at (55447035): [] irq_exit_rcu+0xb0/0xd0 [32438.150655] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [32438.164125] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32438.171679] irqbypass i2c_i801 [32438.180623] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32438.180624] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32438.183763] iTCO_wdt [32438.191234] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32438.196711] iTCO_vendor_support i2c_smbus [32438.198998] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32438.217736] crct10dif_pclmul [32438.221825] [32438.221826] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32438.227044] joydev [32438.230007] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32438.231498] crc32_pclmul ipmi_ssif [32438.238623] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32438.240719] ghash_clmulni_intel mei_me [32438.247844] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32438.251329] mei lpc_ich [32438.258452] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32438.262288] sha512_ssse3 [32438.269415] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32438.271948] acpi_ipmi mfd_core [32438.279069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32438.281688] wmi ipmi_si [32438.289765] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32438.292904] ipmi_devintf ipmi_msghandler [32438.298641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32438.301170] ioatdma acpi_pad [32438.308294] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32438.312298] acpi_power_meter ixgbe [32438.319424] Call Trace: [32438.319427] [32438.322386] igb mdio [32438.329514] ? watchdog_timer_fn+0x1b4/0x220 [32438.333000] crc32c_intel nvme [32438.335450] ? __hrtimer_run_queues+0x1d3/0x360 [32438.337459] ptp [32438.339733] ? hrtimer_interrupt+0x104/0x240 [32438.344003] nvme_core i2c_algo_bit [32438.347052] ? __local_bh_enable+0x50/0x60 [32438.351577] pps_core dca [32438.353416] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32438.357679] fuse [32438.361164] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32438.365255] irq event stamp: 180674502 [32438.367872] [32438.373089] hardirqs last enabled at (180674501): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32438.375012] [32438.379970] hardirqs last disabled at (180674502): [] sysvec_apic_timer_interrupt+0xe/0x90 [32438.383715] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32438.385812] softirqs last enabled at (180674500): [] __do_softirq+0x2be/0x371 [32438.396060] ? queued_spin_lock_slowpath+0x16/0x2e0 [32438.398152] softirqs last disabled at (180674495): [] irq_exit_rcu+0xb0/0xd0 [32438.407967] do_raw_spin_lock+0xb5/0xc0 [32438.413284] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32438.422057] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32438.426926] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32438.435526] try_get_cap_refs+0x159/0x880 [ceph] [32438.439355] Workqueue: events delayed_work [ceph] [32438.448738] __ceph_get_caps+0x4a1/0x760 [ceph] [32438.453517] [32438.453518] RIP: 0010:lock_acquire+0xf0/0x2b0 [32438.461009] ? __file_remove_privs+0x89/0x110 [32438.465615] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32438.470313] ? wake_bit_function+0x70/0x70 [32438.474837] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32438.476329] ? ceph_write_iter+0x260/0x770 [ceph] [32438.480679] [32438.480680] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32438.485029] ceph_write_iter+0x260/0x770 [ceph] [32438.503766] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32438.507860] ? lock_is_held_type+0xaa/0x120 [32438.513075] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32438.517775] vfs_write+0x324/0x4f0 [32438.519263] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32438.526392] ksys_pwrite64+0x65/0xa0 [32438.530912] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32438.538036] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32438.542212] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32438.549339] do_syscall_64+0x38/0x80 [32438.552735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32438.559860] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32438.563430] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32438.570555] RIP: 0033:0x7f48f0eeb93b [32438.575684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32438.583762] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32438.587332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32438.593070] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32438.598113] Call Trace: [32438.598115] [32438.605238] ORIG_RAX: 0000000000000012 [32438.608810] ? watchdog_timer_fn+0x1b4/0x220 [32438.615934] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32438.634672] ? __hrtimer_run_queues+0x1d3/0x360 [32438.641795] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32438.647016] ? hrtimer_interrupt+0x104/0x240 [32438.649456] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32438.651467] ? __local_bh_enable+0x50/0x60 [32438.655297] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32438.659564] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32438.666686] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32438.671210] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32438.678340] [32438.682598] [32438.722526] [32438.724624] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32438.729933] ? lock_acquire+0xf0/0x2b0 [32438.733683] ? igrab+0x43/0x50 [32438.736732] ? lock_release+0xc5/0x260 [32438.740478] _raw_spin_lock+0x30/0x40 [32438.744142] ? igrab+0x1d/0x50 [32438.747192] igrab+0x1d/0x50 [32438.750072] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32438.755222] delayed_work+0x1d8/0x280 [ceph] [32438.759501] ? process_scheduled_works+0x22a/0x510 [32438.764308] process_scheduled_works+0x2a4/0x510 [32438.768926] worker_thread+0x188/0x340 [32438.772673] ? rescuer_thread+0x310/0x310 [32438.776677] kthread+0xf5/0x130 [32438.779815] ? kthread_complete_and_exit+0x20/0x20 [32438.784599] ret_from_fork+0x31/0x40 [32438.788179] ? kthread_complete_and_exit+0x20/0x20 [32438.792961] ret_from_fork_asm+0x11/0x20 [32438.796884] [32466.028080] watchdog: BUG: soft lockup - CPU#0 stuck for 19122s! [dbench:78683] [32466.035380] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32466.040080] watchdog: BUG: soft lockup - CPU#7 stuck for 28119s! [kworker/7:4:78808] [32466.090821] irq event stamp: 55505102 [32466.090823] hardirqs last enabled at (55505101): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32466.098552] Modules linked in: [32466.102209] hardirqs last disabled at (55505102): [] sysvec_apic_timer_interrupt+0xe/0x90 [32466.112365] ceph [32466.115416] softirqs last enabled at (55505074): [] __do_softirq+0x2be/0x371 [32466.125142] libceph [32466.127066] softirqs last disabled at (55505069): [] irq_exit_rcu+0xb0/0xd0 [32466.135754] dns_resolver [32466.137942] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32466.146452] fscache [32466.149088] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32466.158033] netfs veth [32466.160215] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32466.167688] nft_chain_nat xt_MASQUERADE [32466.170130] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32466.175609] nf_nat nf_conntrack [32466.179524] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32466.198263] nf_defrag_ipv6 [32466.201488] [32466.201489] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32466.206704] nf_defrag_ipv4 nft_compat [32466.209495] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32466.210986] nf_tables nfnetlink [32466.218111] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32466.221854] bridge stp [32466.228978] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32466.232202] llc [32466.239327] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32466.241771] xfs libcrc32c [32466.248894] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32466.250732] sunrpc intel_rapl_msr [32466.257873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32466.260579] intel_rapl_common sb_edac [32466.268655] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32466.272053] x86_pkg_temp_thermal [32466.277791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32466.281534] coretemp kvm_intel [32466.288657] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32466.291968] kvm irqbypass [32466.299094] Call Trace: [32466.299096] [32466.302231] i2c_i801 iTCO_wdt [32466.309356] ? watchdog_timer_fn+0x1b4/0x220 [32466.312059] iTCO_vendor_support [32466.314504] ? __hrtimer_run_queues+0x1d3/0x360 [32466.316513] i2c_smbus [32466.319568] ? hrtimer_interrupt+0x104/0x240 [32466.323829] crct10dif_pclmul joydev [32466.327052] ? __local_bh_enable+0x50/0x60 [32466.331576] crc32_pclmul [32466.333935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32466.338197] ipmi_ssif [32466.341770] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32466.345860] ghash_clmulni_intel [32466.348481] [32466.353694] mei_me [32466.356050] [32466.356052] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32466.361008] mei lpc_ich [32466.364238] ? queued_spin_lock_slowpath+0x18/0x2e0 [32466.366330] sha512_ssse3 [32466.368430] do_raw_spin_lock+0xb5/0xc0 [32466.370525] acpi_ipmi mfd_core [32466.375830] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32466.378359] wmi ipmi_si [32466.383234] try_get_cap_refs+0x159/0x880 [ceph] [32466.385849] ipmi_devintf ipmi_msghandler [32466.389685] __ceph_get_caps+0x4a1/0x760 [ceph] [32466.392817] ioatdma acpi_pad [32466.397605] ? __file_remove_privs+0x89/0x110 [32466.400130] acpi_power_meter ixgbe [32466.404743] ? wake_bit_function+0x70/0x70 [32466.408760] igb mdio [32466.413291] ? ceph_write_iter+0x260/0x770 [ceph] [32466.416252] crc32c_intel nvme [32466.420602] ceph_write_iter+0x260/0x770 [ceph] [32466.424087] ptp nvme_core [32466.428182] ? lock_is_held_type+0xaa/0x120 [32466.430447] i2c_algo_bit pps_core [32466.435147] vfs_write+0x324/0x4f0 [32466.438195] dca fuse [32466.442724] ksys_pwrite64+0x65/0xa0 [32466.445423] [32466.445424] irq event stamp: 180895340 [32466.449602] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32466.452999] hardirqs last enabled at (180895339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32466.456397] do_syscall_64+0x38/0x80 [32466.458666] hardirqs last disabled at (180895340): [] sysvec_apic_timer_interrupt+0xe/0x90 [32466.462240] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32466.463728] softirqs last enabled at (180895338): [] __do_softirq+0x2be/0x371 [32466.467481] RIP: 0033:0x7f48f0eeb93b [32466.472611] softirqs last disabled at (180895333): [] irq_exit_rcu+0xb0/0xd0 [32466.482855] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32466.486427] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32466.496238] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32466.501282] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32466.510052] ORIG_RAX: 0000000000000012 [32466.513623] Workqueue: events delayed_work [ceph] [32466.522221] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32466.540959] [32466.550336] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32466.555554] RIP: 0010:lock_acquire+0xf0/0x2b0 [32466.563024] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32466.566854] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32466.571553] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32466.578676] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32466.580166] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32466.587291] [32466.587292] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32466.591647] [32466.598767] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32466.654911] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32466.662035] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32466.669159] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32466.676282] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32466.684361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32466.690097] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32466.697221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32466.704347] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32466.711487] Call Trace: [32466.713932] [32466.715944] ? watchdog_timer_fn+0x1b4/0x220 [32466.720206] ? __hrtimer_run_queues+0x1d3/0x360 [32466.724733] ? hrtimer_interrupt+0x104/0x240 [32466.729002] ? __local_bh_enable+0x50/0x60 [32466.733097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32466.738320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32466.743278] [32466.745374] [32466.747486] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32466.752798] ? lock_acquire+0xf0/0x2b0 [32466.756547] ? igrab+0x43/0x50 [32466.759597] ? lock_release+0xc5/0x260 [32466.763343] _raw_spin_lock+0x30/0x40 [32466.767006] ? igrab+0x1d/0x50 [32466.770058] igrab+0x1d/0x50 [32466.772936] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32466.778086] delayed_work+0x1d8/0x280 [ceph] [32466.782366] ? process_scheduled_works+0x22a/0x510 [32466.787157] process_scheduled_works+0x2a4/0x510 [32466.791774] worker_thread+0x188/0x340 [32466.795521] ? rescuer_thread+0x310/0x310 [32466.799526] kthread+0xf5/0x130 [32466.802661] ? kthread_complete_and_exit+0x20/0x20 [32466.807447] ret_from_fork+0x31/0x40 [32466.811025] ? kthread_complete_and_exit+0x20/0x20 [32466.815810] ret_from_fork_asm+0x11/0x20 [32466.819733] [32494.028558] watchdog: BUG: soft lockup - CPU#0 stuck for 19148s! [dbench:78683] [32494.035860] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32494.040557] watchdog: BUG: soft lockup - CPU#7 stuck for 28145s! [kworker/7:4:78808] [32494.091309] irq event stamp: 55563288 [32494.099041] Modules linked in: ceph libceph dns_resolver [32494.102696] hardirqs last enabled at (55563287): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32494.108001] fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat [32494.118158] hardirqs last disabled at (55563288): [] sysvec_apic_timer_interrupt+0xe/0x90 [32494.124328] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [32494.134052] softirqs last enabled at (55563282): [] __do_softirq+0x2be/0x371 [32494.141958] bridge stp llc xfs libcrc32c sunrpc [32494.150643] softirqs last disabled at (55563277): [] irq_exit_rcu+0xb0/0xd0 [32494.155251] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [32494.163766] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32494.171408] kvm_intel [32494.180351] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32494.180353] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32494.182709] kvm irqbypass [32494.190180] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32494.195657] i2c_i801 [32494.198362] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32494.217099] iTCO_wdt [32494.219369] [32494.219370] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32494.224587] iTCO_vendor_support [32494.226859] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32494.228349] i2c_smbus crct10dif_pclmul [32494.235472] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32494.238696] joydev crc32_pclmul [32494.245820] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32494.249651] ipmi_ssif ghash_clmulni_intel [32494.256776] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32494.259999] mei_me mei [32494.267123] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32494.271214] lpc_ich [32494.278356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32494.280801] sha512_ssse3 acpi_ipmi [32494.288878] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32494.291062] mfd_core [32494.296800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32494.300282] wmi [32494.307408] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32494.309679] ipmi_si [32494.316802] Call Trace: [32494.316805] [32494.318639] ipmi_devintf ipmi_msghandler [32494.325767] ? watchdog_timer_fn+0x1b4/0x220 [32494.327956] ioatdma [32494.330400] ? __hrtimer_run_queues+0x1d3/0x360 [32494.332410] acpi_pad acpi_power_meter [32494.336419] ? hrtimer_interrupt+0x104/0x240 [32494.340680] ixgbe igb [32494.342862] ? __local_bh_enable+0x50/0x60 [32494.347388] mdio [32494.351133] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32494.355394] crc32c_intel nvme [32494.357753] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32494.361845] ptp [32494.363769] [32494.368984] nvme_core [32494.372034] [32494.376993] i2c_algo_bit pps_core [32494.378831] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32494.380927] dca fuse [32494.383288] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32494.385382] [32494.385383] irq event stamp: 181116112 [32494.388783] do_raw_spin_lock+0xb5/0xc0 [32494.394084] hardirqs last enabled at (181116111): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32494.396358] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32494.401236] hardirqs last disabled at (181116112): [] sysvec_apic_timer_interrupt+0xe/0x90 [32494.402728] try_get_cap_refs+0x159/0x880 [ceph] [32494.406469] softirqs last enabled at (181116110): [] __do_softirq+0x2be/0x371 [32494.410304] __ceph_get_caps+0x4a1/0x760 [ceph] [32494.420543] softirqs last disabled at (181116105): [] irq_exit_rcu+0xb0/0xd0 [32494.425332] ? __file_remove_privs+0x89/0x110 [32494.435138] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32494.439751] ? wake_bit_function+0x70/0x70 [32494.448519] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32494.453048] ? ceph_write_iter+0x260/0x770 [ceph] [32494.461643] Workqueue: events delayed_work [ceph] [32494.465994] ceph_write_iter+0x260/0x770 [ceph] [32494.475369] [32494.475370] RIP: 0010:lock_acquire+0xf0/0x2b0 [32494.479464] ? lock_is_held_type+0xaa/0x120 [32494.486933] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32494.491649] vfs_write+0x324/0x4f0 [32494.496343] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32494.500874] ksys_pwrite64+0x65/0xa0 [32494.502359] [32494.506711] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32494.510887] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32494.529625] do_syscall_64+0x38/0x80 [32494.533022] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32494.538240] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32494.541810] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32494.543302] RIP: 0033:0x7f48f0eeb93b [32494.548432] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32494.555557] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32494.559127] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32494.566250] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32494.571295] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32494.578418] ORIG_RAX: 0000000000000012 [32494.581989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32494.589113] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32494.607851] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32494.614976] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32494.620192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32494.628271] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32494.632100] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32494.637838] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32494.644962] Call Trace: [32494.644964] [32494.652088] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32494.659211] ? watchdog_timer_fn+0x1b4/0x220 [32494.666341] [32494.673462] ? __hrtimer_run_queues+0x1d3/0x360 [32494.710262] ? hrtimer_interrupt+0x104/0x240 [32494.714532] ? __local_bh_enable+0x50/0x60 [32494.718625] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32494.723848] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32494.728806] [32494.730903] [32494.733002] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32494.738308] ? lock_acquire+0xf0/0x2b0 [32494.742059] ? igrab+0x43/0x50 [32494.745109] ? lock_release+0xc5/0x260 [32494.748855] _raw_spin_lock+0x30/0x40 [32494.752519] ? igrab+0x1d/0x50 [32494.755569] igrab+0x1d/0x50 [32494.758448] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32494.763597] delayed_work+0x1d8/0x280 [ceph] [32494.767879] ? process_scheduled_works+0x22a/0x510 [32494.772669] process_scheduled_works+0x2a4/0x510 [32494.777283] worker_thread+0x188/0x340 [32494.781032] ? rescuer_thread+0x310/0x310 [32494.785037] kthread+0xf5/0x130 [32494.788184] ? kthread_complete_and_exit+0x20/0x20 [32494.792967] ret_from_fork+0x31/0x40 [32494.796547] ? kthread_complete_and_exit+0x20/0x20 [32494.801330] ret_from_fork_asm+0x11/0x20 [32494.805267] [32521.283387] rcu: INFO: rcu_preempt self-detected stall on CPU [32521.289145] rcu: 0-....: (20120111 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10069861 [32521.299657] rcu: (t=20584308 jiffies g=1309609 q=590921 ncpus=8) [32521.305743] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32521.314686] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32521.322155] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32521.327634] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32521.346372] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32521.351588] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32521.358714] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32521.365837] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32521.372961] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32521.380085] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32521.387209] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32521.395286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32521.401025] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32521.408150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32521.415273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32521.422396] Call Trace: [32521.424842] [32521.426852] ? rcu_dump_cpu_stacks+0xd1/0x180 [32521.431211] ? rcu_sched_clock_irq+0x633/0x1390 [32521.435736] ? lock_is_held_type+0xaa/0x120 [32521.439924] ? lock_is_held_type+0xaa/0x120 [32521.444101] ? tick_sched_do_timer+0xa0/0xa0 [32521.448373] ? rcu_read_lock_sched_held+0x56/0x90 [32521.453071] ? tick_sched_do_timer+0xa0/0xa0 [32521.457334] ? update_process_times+0x5f/0x90 [32521.461685] ? tick_sched_handle.isra.16+0x1e/0x50 [32521.466467] ? tick_sched_timer+0x6f/0x80 [32521.470474] ? __hrtimer_run_queues+0x1d3/0x360 [32521.474998] ? hrtimer_interrupt+0x104/0x240 [32521.479260] ? __local_bh_enable+0x50/0x60 [32521.483353] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32521.488578] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32521.493533] [32521.495632] [32521.497729] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32521.503039] ? queued_spin_lock_slowpath+0x18/0x2e0 [32521.507914] do_raw_spin_lock+0xb5/0xc0 [32521.511743] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32521.516546] try_get_cap_refs+0x159/0x880 [ceph] [32521.521178] __ceph_get_caps+0x4a1/0x760 [ceph] [32521.525719] ? __file_remove_privs+0x89/0x110 [32521.530075] ? wake_bit_function+0x70/0x70 [32521.534167] ? ceph_write_iter+0x260/0x770 [ceph] [32521.538879] ceph_write_iter+0x260/0x770 [ceph] [32521.543416] ? lock_is_held_type+0xaa/0x120 [32521.547601] vfs_write+0x324/0x4f0 [32521.551008] ksys_pwrite64+0x65/0xa0 [32521.554583] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32521.559715] do_syscall_64+0x38/0x80 [32521.563286] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32521.568328] RIP: 0033:0x7f48f0eeb93b [32521.571901] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32521.590636] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [32521.598196] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32521.605320] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32521.612444] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32521.619568] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32521.626692] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32521.633822] [32521.636008] Sending NMI from CPU 0 to CPUs 7: [32521.640362] NMI backtrace for cpu 7 [32521.640363] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32521.640365] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32521.640366] Workqueue: events delayed_work [ceph] [32521.640382] RIP: 0010:__lock_acquire+0x1cc/0x1e20 [32521.640385] Code: f6 83 e1 f8 44 88 75 21 09 c1 09 d1 83 e1 e7 44 09 e9 44 09 c9 88 4d 22 0f b7 94 24 a0 00 00 00 0f b7 45 22 c1 e2 05 83 e0 1f <09> d0 66 89 45 22 8b 84 24 a8 00 00 00 89 45 24 4c 89 e0 25 ff 1f [32521.640386] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [32521.640387] RAX: 0000000000000004 RBX: ffff88810debaa80 RCX: 0000000000000004 [32521.640388] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [32521.640388] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [32521.640389] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [32521.640390] R13: 0000000000000000 R14: 0000000000000006 R15: ffff8881a1d34518 [32521.640390] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32521.640391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32521.640393] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32521.640394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32521.640395] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32521.640396] Call Trace: [32521.640398] [32521.640399] ? nmi_cpu_backtrace+0xad/0x120 [32521.640402] ? nmi_cpu_backtrace_handler+0x11/0x20 [32521.640404] ? nmi_handle+0xd4/0x260 [32521.640408] ? default_do_nmi+0x49/0xf0 [32521.640409] ? exc_nmi+0x11e/0x150 [32521.640411] ? end_repeat_nmi+0x16/0x67 [32521.640414] ? __lock_acquire+0x1cc/0x1e20 [32521.640416] ? __lock_acquire+0x1cc/0x1e20 [32521.640418] ? __lock_acquire+0x1cc/0x1e20 [32521.640419] [32521.640419] [32521.640421] lock_acquire+0xcb/0x2b0 [32521.640423] ? igrab+0x1d/0x50 [32521.640424] ? igrab+0x43/0x50 [32521.640426] ? lock_release+0xc5/0x260 [32521.640427] _raw_spin_lock+0x30/0x40 [32521.640429] ? igrab+0x1d/0x50 [32521.640430] igrab+0x1d/0x50 [32521.640432] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32521.640446] delayed_work+0x1d8/0x280 [ceph] [32521.640459] ? process_scheduled_works+0x22a/0x510 [32521.640462] process_scheduled_works+0x2a4/0x510 [32521.640465] worker_thread+0x188/0x340 [32521.640468] ? rescuer_thread+0x310/0x310 [32521.640470] kthread+0xf5/0x130 [32521.640472] ? kthread_complete_and_exit+0x20/0x20 [32521.640474] ret_from_fork+0x31/0x40 [32521.640477] ? kthread_complete_and_exit+0x20/0x20 [32521.640479] ret_from_fork_asm+0x11/0x20 [32521.640482] [32546.029443] watchdog: BUG: soft lockup - CPU#0 stuck for 19196s! [dbench:78683] [32546.036748] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32546.041443] watchdog: BUG: soft lockup - CPU#7 stuck for 28193s! [kworker/7:4:78808] [32546.092188] irq event stamp: 55671526 [32546.092189] hardirqs last enabled at (55671525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32546.099919] Modules linked in: [32546.103577] hardirqs last disabled at (55671526): [] sysvec_apic_timer_interrupt+0xe/0x90 [32546.113734] ceph [32546.116786] softirqs last enabled at (55671522): [] __do_softirq+0x2be/0x371 [32546.126509] libceph [32546.128432] softirqs last disabled at (55671517): [] irq_exit_rcu+0xb0/0xd0 [32546.137116] dns_resolver [32546.139300] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32546.147812] fscache netfs [32546.150448] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32546.159391] veth nft_chain_nat [32546.162094] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32546.169583] xt_MASQUERADE nf_nat [32546.172721] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32546.178197] nf_conntrack nf_defrag_ipv6 [32546.181508] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32546.200247] nf_defrag_ipv4 nft_compat [32546.204162] [32546.204163] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32546.209382] nf_tables nfnetlink [32546.213124] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32546.214616] bridge stp [32546.221739] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32546.224963] llc xfs [32546.232087] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32546.234532] libcrc32c sunrpc [32546.241656] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32546.243841] intel_rapl_msr intel_rapl_common [32546.250965] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32546.253929] sb_edac x86_pkg_temp_thermal [32546.261051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32546.265402] coretemp kvm_intel [32546.273481] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32546.277484] kvm irqbypass [32546.283222] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32546.286360] i2c_i801 iTCO_wdt [32546.293483] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32546.296187] iTCO_vendor_support i2c_smbus [32546.303313] Call Trace: [32546.303314] [32546.306364] crct10dif_pclmul joydev [32546.313489] ? watchdog_timer_fn+0x1b4/0x220 [32546.317578] crc32_pclmul [32546.320024] ? __hrtimer_run_queues+0x1d3/0x360 [32546.322033] ipmi_ssif ghash_clmulni_intel [32546.325605] ? hrtimer_interrupt+0x104/0x240 [32546.329868] mei_me mei [32546.332484] ? __local_bh_enable+0x50/0x60 [32546.337008] lpc_ich sha512_ssse3 [32546.341103] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32546.345362] acpi_ipmi [32546.347808] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32546.351899] mfd_core wmi [32546.355208] [32546.360427] ipmi_si ipmi_devintf [32546.362784] [32546.362785] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32546.367742] ipmi_msghandler ioatdma [32546.370363] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32546.372456] acpi_pad acpi_power_meter [32546.375768] do_raw_spin_lock+0xb5/0xc0 [32546.377864] ixgbe igb [32546.383171] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32546.386740] mdio crc32c_intel [32546.391612] try_get_cap_refs+0x159/0x880 [ceph] [32546.395354] nvme ptp [32546.399190] __ceph_get_caps+0x4a1/0x760 [ceph] [32546.401543] nvme_core i2c_algo_bit [32546.406329] ? __file_remove_privs+0x89/0x110 [32546.409376] pps_core dca [32546.413987] ? wake_bit_function+0x70/0x70 [32546.416257] fuse [32546.420785] ? ceph_write_iter+0x260/0x770 [ceph] [32546.424267] irq event stamp: 181529512 [32546.428617] ceph_write_iter+0x260/0x770 [ceph] [32546.431234] hardirqs last enabled at (181529511): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32546.435329] ? lock_is_held_type+0xaa/0x120 [32546.437250] hardirqs last disabled at (181529512): [] sysvec_apic_timer_interrupt+0xe/0x90 [32546.441948] vfs_write+0x324/0x4f0 [32546.445690] softirqs last enabled at (181529510): [] __do_softirq+0x2be/0x371 [32546.450218] ksys_pwrite64+0x65/0xa0 [32546.460459] softirqs last disabled at (181529505): [] irq_exit_rcu+0xb0/0xd0 [32546.464636] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32546.474447] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32546.477847] do_syscall_64+0x38/0x80 [32546.486614] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32546.490187] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32546.498785] Workqueue: events delayed_work [ceph] [32546.503914] RIP: 0033:0x7f48f0eeb93b [32546.513291] [32546.513292] RIP: 0010:lock_acquire+0xf0/0x2b0 [32546.516864] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32546.524334] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32546.529378] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32546.534076] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32546.537645] ORIG_RAX: 0000000000000012 [32546.539136] [32546.539136] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32546.543487] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32546.562224] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32546.580961] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32546.586180] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32546.591398] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32546.595228] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32546.596719] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32546.603845] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32546.610966] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32546.618091] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32546.625221] [32546.632339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32546.683967] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32546.691093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32546.698215] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32546.705340] Call Trace: [32546.707786] [32546.709796] ? watchdog_timer_fn+0x1b4/0x220 [32546.714059] ? __hrtimer_run_queues+0x1d3/0x360 [32546.718586] ? hrtimer_interrupt+0x104/0x240 [32546.722857] ? __local_bh_enable+0x50/0x60 [32546.726949] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32546.732172] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32546.737137] [32546.739236] [32546.741333] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32546.746641] ? lock_acquire+0xf0/0x2b0 [32546.750393] ? igrab+0x43/0x50 [32546.753442] ? lock_release+0xc5/0x260 [32546.757187] _raw_spin_lock+0x30/0x40 [32546.760851] ? igrab+0x1d/0x50 [32546.763902] igrab+0x1d/0x50 [32546.766781] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32546.771930] delayed_work+0x1d8/0x280 [ceph] [32546.776211] ? process_scheduled_works+0x22a/0x510 [32546.780993] process_scheduled_works+0x2a4/0x510 [32546.785609] worker_thread+0x188/0x340 [32546.789359] ? rescuer_thread+0x310/0x310 [32546.793360] kthread+0xf5/0x130 [32546.796498] ? kthread_complete_and_exit+0x20/0x20 [32546.801284] ret_from_fork+0x31/0x40 [32546.804862] ? kthread_complete_and_exit+0x20/0x20 [32546.809646] ret_from_fork_asm+0x11/0x20 [32546.813570] [32574.029921] watchdog: BUG: soft lockup - CPU#0 stuck for 19222s! [dbench:78683] [32574.037226] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32574.041920] watchdog: BUG: soft lockup - CPU#7 stuck for 28219s! [kworker/7:4:78808] [32574.092675] irq event stamp: 55729684 [32574.100407] Modules linked in: ceph libceph dns_resolver fscache netfs veth [32574.104064] hardirqs last enabled at (55729683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32574.111025] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [32574.121182] hardirqs last disabled at (55729684): [] sysvec_apic_timer_interrupt+0xe/0x90 [32574.121185] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [32574.136564] softirqs last enabled at (55729588): [] __do_softirq+0x2be/0x371 [32574.144641] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [32574.153326] softirqs last disabled at (55729583): [] irq_exit_rcu+0xb0/0xd0 [32574.159496] sb_edac x86_pkg_temp_thermal coretemp [32574.168009] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32574.168010] kvm_intel [32574.168011] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32574.172793] kvm irqbypass [32574.181736] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32574.184093] i2c_i801 iTCO_wdt [32574.191564] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32574.194269] iTCO_vendor_support i2c_smbus [32574.199745] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32574.202797] crct10dif_pclmul joydev [32574.221533] [32574.221534] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32574.225624] crc32_pclmul [32574.230843] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32574.234412] ipmi_ssif ghash_clmulni_intel [32574.235904] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32574.243028] mei_me [32574.245646] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32574.252771] mei [32574.256859] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32574.263985] lpc_ich sha512_ssse3 [32574.266083] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32574.273206] acpi_ipmi [32574.275043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32574.282167] mfd_core wmi [32574.285478] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32574.293557] ipmi_si [32574.295912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32574.301650] ipmi_devintf ipmi_msghandler [32574.304268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32574.311391] ioatdma acpi_pad [32574.313575] Call Trace: [32574.320716] acpi_power_meter [32574.324723] [32574.331846] ixgbe igb [32574.334813] ? watchdog_timer_fn+0x1b4/0x220 [32574.337253] mdio crc32c_intel [32574.340220] ? __hrtimer_run_queues+0x1d3/0x360 [32574.342229] nvme [32574.344600] ? hrtimer_interrupt+0x104/0x240 [32574.348860] ptp nvme_core [32574.351910] ? __local_bh_enable+0x50/0x60 [32574.356432] i2c_algo_bit [32574.358361] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32574.362621] pps_core dca [32574.365325] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32574.369416] fuse [32574.372034] [32574.377252] irq event stamp: 181750306 [32574.379869] [32574.384825] hardirqs last enabled at (181750305): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32574.386750] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32574.388848] hardirqs last disabled at (181750306): [] sysvec_apic_timer_interrupt+0xe/0x90 [32574.392604] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32574.394696] softirqs last enabled at (181750304): [] __do_softirq+0x2be/0x371 [32574.404945] do_raw_spin_lock+0xb5/0xc0 [32574.410245] softirqs last disabled at (181750299): [] irq_exit_rcu+0xb0/0xd0 [32574.420058] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32574.424929] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32574.433701] try_get_cap_refs+0x159/0x880 [ceph] [32574.437528] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32574.446131] __ceph_get_caps+0x4a1/0x760 [ceph] [32574.450910] Workqueue: events delayed_work [ceph] [32574.460293] ? __file_remove_privs+0x89/0x110 [32574.464899] [32574.464900] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [32574.472371] ? wake_bit_function+0x70/0x70 [32574.476893] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [32574.481595] ? ceph_write_iter+0x260/0x770 [ceph] [32574.485959] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [32574.487449] ceph_write_iter+0x260/0x770 [ceph] [32574.492061] [32574.496154] ? lock_is_held_type+0xaa/0x120 [32574.514889] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [32574.519604] vfs_write+0x324/0x4f0 [32574.524821] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [32574.529352] ksys_pwrite64+0x65/0xa0 [32574.530835] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [32574.535014] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32574.542137] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32574.545536] do_syscall_64+0x38/0x80 [32574.552660] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32574.556229] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32574.563355] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32574.568485] RIP: 0033:0x7f48f0eeb93b [32574.575612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32574.579181] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32574.586320] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32574.591364] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32574.599443] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32574.603013] ORIG_RAX: 0000000000000012 [32574.608751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32574.627487] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32574.634611] Call Trace: [32574.634613] [32574.639831] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32574.646954] ? watchdog_timer_fn+0x1b4/0x220 [32574.650785] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32574.657911] ? __hrtimer_run_queues+0x1d3/0x360 [32574.665033] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32574.667480] ? hrtimer_interrupt+0x104/0x240 [32574.669489] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32574.676619] ? __local_bh_enable+0x50/0x60 [32574.680890] [32574.688012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32574.722538] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32574.727496] [32574.729603] [32574.731698] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32574.737008] ? do_raw_spin_lock+0x42/0xc0 [32574.741018] igrab+0x1d/0x50 [32574.743894] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32574.749043] delayed_work+0x1d8/0x280 [ceph] [32574.753327] ? process_scheduled_works+0x22a/0x510 [32574.758115] process_scheduled_works+0x2a4/0x510 [32574.762731] worker_thread+0x188/0x340 [32574.766481] ? rescuer_thread+0x310/0x310 [32574.770491] kthread+0xf5/0x130 [32574.773630] ? kthread_complete_and_exit+0x20/0x20 [32574.778415] ret_from_fork+0x31/0x40 [32574.781995] ? kthread_complete_and_exit+0x20/0x20 [32574.786786] ret_from_fork_asm+0x11/0x20 [32574.790717] [32602.030397] watchdog: BUG: soft lockup - CPU#0 stuck for 19249s! [dbench:78683] [32602.037697] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32602.042397] watchdog: BUG: soft lockup - CPU#7 stuck for 28246s! [kworker/7:4:78808] [32602.093138] irq event stamp: 55787656 [32602.093139] hardirqs last enabled at (55787655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32602.100869] Modules linked in: ceph [32602.104524] hardirqs last disabled at (55787656): [] sysvec_apic_timer_interrupt+0xe/0x90 [32602.114682] libceph dns_resolver [32602.118166] softirqs last enabled at (55787578): [] __do_softirq+0x2be/0x371 [32602.127892] fscache netfs [32602.131201] softirqs last disabled at (55787573): [] irq_exit_rcu+0xb0/0xd0 [32602.139886] veth nft_chain_nat [32602.142591] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32602.151101] xt_MASQUERADE nf_nat [32602.154239] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32602.163183] nf_conntrack nf_defrag_ipv6 [32602.166494] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32602.173964] nf_defrag_ipv4 nft_compat [32602.177881] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32602.183358] nf_tables nfnetlink [32602.187102] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32602.205840] bridge stp [32602.209071] [32602.209072] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32602.214291] llc xfs [32602.216734] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32602.218227] libcrc32c [32602.225350] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32602.227533] sunrpc intel_rapl_msr [32602.234659] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32602.237016] intel_rapl_common sb_edac [32602.244139] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32602.247536] x86_pkg_temp_thermal coretemp [32602.254661] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32602.258404] kvm_intel kvm [32602.265528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32602.269619] irqbypass [32602.277696] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32602.280400] i2c_i801 iTCO_wdt [32602.286138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32602.288497] iTCO_vendor_support i2c_smbus [32602.295620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32602.298672] crct10dif_pclmul joydev [32602.305796] Call Trace: [32602.309885] crc32_pclmul ipmi_ssif [32602.317012] [32602.320581] ghash_clmulni_intel mei_me [32602.323027] ? watchdog_timer_fn+0x1b4/0x220 [32602.326509] mei lpc_ich [32602.328520] ? __hrtimer_run_queues+0x1d3/0x360 [32602.332349] sha512_ssse3 acpi_ipmi [32602.336618] ? hrtimer_interrupt+0x104/0x240 [32602.339146] mfd_core wmi [32602.343670] ? __local_bh_enable+0x50/0x60 [32602.347153] ipmi_si [32602.351419] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32602.354034] ipmi_devintf ipmi_msghandler [32602.358127] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32602.360309] ioatdma [32602.365528] [32602.369531] acpi_pad acpi_power_meter [32602.374489] [32602.374490] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32602.376672] ixgbe igb [32602.378794] ? queued_spin_lock_slowpath+0x16/0x2e0 [32602.382531] mdio crc32c_intel nvme [32602.384631] do_raw_spin_lock+0xb5/0xc0 [32602.389932] ptp nvme_core [32602.392292] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32602.397162] i2c_algo_bit pps_core [32602.400648] try_get_cap_refs+0x159/0x880 [ceph] [32602.404476] dca fuse [32602.407184] __ceph_get_caps+0x4a1/0x760 [ceph] [32602.411963] [32602.411964] irq event stamp: 181971066 [32602.415366] ? __file_remove_privs+0x89/0x110 [32602.419972] hardirqs last enabled at (181971065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32602.422242] ? wake_bit_function+0x70/0x70 [32602.426766] hardirqs last disabled at (181971066): [] sysvec_apic_timer_interrupt+0xe/0x90 [32602.428261] ? ceph_write_iter+0x260/0x770 [ceph] [32602.432001] softirqs last enabled at (181971064): [] __do_softirq+0x2be/0x371 [32602.436353] ceph_write_iter+0x260/0x770 [ceph] [32602.446596] softirqs last disabled at (181971059): [] irq_exit_rcu+0xb0/0xd0 [32602.450690] ? lock_is_held_type+0xaa/0x120 [32602.460499] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32602.465199] vfs_write+0x324/0x4f0 [32602.473967] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32602.478495] ksys_pwrite64+0x65/0xa0 [32602.487097] Workqueue: events delayed_work [ceph] [32602.491274] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32602.500651] [32602.500652] RIP: 0010:lock_acquire+0x0/0x2b0 [32602.504050] do_syscall_64+0x38/0x80 [32602.511521] Code: 0f c1 05 63 da ea 7e 83 f8 01 75 0e 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd [32602.515092] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32602.519800] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [32602.524928] RIP: 0033:0x7f48f0eeb93b [32602.526418] [32602.526419] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [32602.530684] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32602.534254] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [32602.552990] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32602.558035] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [32602.563253] ORIG_RAX: 0000000000000012 [32602.566822] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32602.568314] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32602.575438] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32602.594174] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32602.601299] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32602.606516] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32602.613640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32602.617473] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32602.624597] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32602.631720] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32602.638843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32602.645973] [32602.654046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32602.704714] Call Trace: [32602.707157] [32602.709169] ? watchdog_timer_fn+0x1b4/0x220 [32602.713432] ? __hrtimer_run_queues+0x1d3/0x360 [32602.717959] ? hrtimer_interrupt+0x104/0x240 [32602.722227] ? __local_bh_enable+0x50/0x60 [32602.726321] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32602.731546] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32602.736502] [32602.738600] [32602.740698] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32602.746005] ? lock_sync+0xa0/0xa0 [32602.749407] _raw_spin_lock+0x30/0x40 [32602.753066] ? igrab+0x1d/0x50 [32602.756134] igrab+0x1d/0x50 [32602.759011] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32602.764161] delayed_work+0x1d8/0x280 [ceph] [32602.768441] ? process_scheduled_works+0x22a/0x510 [32602.773224] process_scheduled_works+0x2a4/0x510 [32602.777840] worker_thread+0x188/0x340 [32602.781588] ? rescuer_thread+0x310/0x310 [32602.785591] kthread+0xf5/0x130 [32602.788730] ? kthread_complete_and_exit+0x20/0x20 [32602.793523] ret_from_fork+0x31/0x40 [32602.797100] ? kthread_complete_and_exit+0x20/0x20 [32602.801885] ret_from_fork_asm+0x11/0x20 [32602.805810] [32630.030876] watchdog: BUG: soft lockup - CPU#0 stuck for 19275s! [dbench:78683] [32630.038176] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32630.042875] watchdog: BUG: soft lockup - CPU#7 stuck for 28272s! [kworker/7:4:78808] [32630.093625] irq event stamp: 55845748 [32630.093627] hardirqs last enabled at (55845747): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32630.101356] Modules linked in: [32630.105012] hardirqs last disabled at (55845748): [] sysvec_apic_timer_interrupt+0xe/0x90 [32630.115170] ceph [32630.118222] softirqs last enabled at (55845692): [] __do_softirq+0x2be/0x371 [32630.127946] libceph [32630.129870] softirqs last disabled at (55845687): [] irq_exit_rcu+0xb0/0xd0 [32630.138558] dns_resolver [32630.140738] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32630.149267] fscache [32630.151882] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32630.160828] netfs [32630.163012] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32630.170482] veth [32630.172495] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32630.177970] nft_chain_nat [32630.179896] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32630.198632] xt_MASQUERADE nf_nat [32630.201336] [32630.206559] nf_conntrack nf_defrag_ipv6 [32630.209873] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32630.211365] nf_defrag_ipv4 nft_compat [32630.215281] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32630.222406] nf_tables [32630.226148] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32630.233273] nfnetlink [32630.235631] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32630.242754] bridge stp [32630.245112] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32630.252238] llc xfs [32630.254681] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32630.261805] libcrc32c sunrpc [32630.263990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32630.272065] intel_rapl_msr [32630.275031] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32630.280767] intel_rapl_common sb_edac [32630.283560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32630.290682] x86_pkg_temp_thermal [32630.294427] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32630.301558] coretemp kvm_intel [32630.304871] Call Trace: [32630.304873] [32630.311994] kvm irqbypass [32630.315134] ? watchdog_timer_fn+0x1b4/0x220 [32630.317575] i2c_i801 iTCO_wdt [32630.319587] ? __hrtimer_run_queues+0x1d3/0x360 [32630.322289] iTCO_vendor_support i2c_smbus [32630.326559] ? hrtimer_interrupt+0x104/0x240 [32630.329606] crct10dif_pclmul joydev [32630.334128] ? __local_bh_enable+0x50/0x60 [32630.338221] crc32_pclmul ipmi_ssif [32630.342487] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32630.346054] ghash_clmulni_intel mei_me [32630.350145] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32630.353630] mei lpc_ich [32630.358847] [32630.362677] sha512_ssse3 [32630.367636] [32630.367637] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32630.370165] acpi_ipmi mfd_core [32630.372285] ? queued_spin_lock_slowpath+0x18/0x2e0 [32630.374898] wmi ipmi_si [32630.376997] do_raw_spin_lock+0xb5/0xc0 [32630.382300] ipmi_devintf ipmi_msghandler [32630.385439] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32630.390307] ioatdma acpi_pad [32630.392840] try_get_cap_refs+0x159/0x880 [ceph] [32630.396669] acpi_power_meter [32630.400677] __ceph_get_caps+0x4a1/0x760 [ceph] [32630.405458] ixgbe igb [32630.408425] ? __file_remove_privs+0x89/0x110 [32630.413031] mdio [32630.415998] ? wake_bit_function+0x70/0x70 [32630.420521] crc32c_intel [32630.422881] ? ceph_write_iter+0x260/0x770 [ceph] [32630.427228] nvme ptp [32630.429154] ceph_write_iter+0x260/0x770 [ceph] [32630.433243] nvme_core [32630.435864] ? lock_is_held_type+0xaa/0x120 [32630.440557] i2c_algo_bit pps_core [32630.442832] vfs_write+0x324/0x4f0 [32630.447352] dca fuse [32630.449716] ksys_pwrite64+0x65/0xa0 [32630.453888] [32630.453889] irq event stamp: 182191906 [32630.457285] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32630.460683] hardirqs last enabled at (182191905): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32630.462953] do_syscall_64+0x38/0x80 [32630.466525] hardirqs last disabled at (182191906): [] sysvec_apic_timer_interrupt+0xe/0x90 [32630.468014] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32630.471760] softirqs last enabled at (182191904): [] __do_softirq+0x2be/0x371 [32630.476892] RIP: 0033:0x7f48f0eeb93b [32630.487133] softirqs last disabled at (182191899): [] irq_exit_rcu+0xb0/0xd0 [32630.490706] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32630.500516] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32630.505560] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32630.514330] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32630.517900] ORIG_RAX: 0000000000000012 [32630.526498] Workqueue: events delayed_work [ceph] [32630.545235] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32630.554614] [32630.559831] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32630.567301] RIP: 0010:lock_release+0x78/0x260 [32630.571134] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32630.575831] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [32630.582954] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32630.584444] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [32630.591568] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32630.595919] [32630.595920] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [32630.603050] [32630.621782] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [32630.659171] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [32630.666296] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [32630.673419] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32630.680559] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32630.688638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32630.694374] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32630.701498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32630.708623] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32630.715749] Call Trace: [32630.718192] [32630.720202] ? watchdog_timer_fn+0x1b4/0x220 [32630.724467] ? __hrtimer_run_queues+0x1d3/0x360 [32630.728993] ? hrtimer_interrupt+0x104/0x240 [32630.733253] ? __local_bh_enable+0x50/0x60 [32630.737365] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32630.742588] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32630.747563] [32630.749660] [32630.751760] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32630.757064] ? igrab+0x43/0x50 [32630.760117] ? lock_release+0x78/0x260 [32630.763869] _raw_spin_unlock+0x1b/0x40 [32630.767707] igrab+0x43/0x50 [32630.770584] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32630.775734] delayed_work+0x1d8/0x280 [ceph] [32630.780016] ? process_scheduled_works+0x22a/0x510 [32630.784803] process_scheduled_works+0x2a4/0x510 [32630.789421] worker_thread+0x188/0x340 [32630.793171] ? rescuer_thread+0x310/0x310 [32630.797182] kthread+0xf5/0x130 [32630.800329] ? kthread_complete_and_exit+0x20/0x20 [32630.805113] ret_from_fork+0x31/0x40 [32630.808692] ? kthread_complete_and_exit+0x20/0x20 [32630.813476] ret_from_fork_asm+0x11/0x20 [32630.817407] [32658.031353] watchdog: BUG: soft lockup - CPU#0 stuck for 19301s! [dbench:78683] [32658.038655] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32658.043353] watchdog: BUG: soft lockup - CPU#7 stuck for 28298s! [kworker/7:4:78808] [32658.094094] irq event stamp: 55903908 [32658.094096] hardirqs last enabled at (55903907): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32658.101827] Modules linked in: [32658.105483] hardirqs last disabled at (55903908): [] sysvec_apic_timer_interrupt+0xe/0x90 [32658.115640] ceph [32658.118691] softirqs last enabled at (55903830): [] __do_softirq+0x2be/0x371 [32658.128415] libceph [32658.130340] softirqs last disabled at (55903825): [] irq_exit_rcu+0xb0/0xd0 [32658.139037] dns_resolver [32658.141216] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32658.149745] fscache [32658.152362] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32658.152364] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32658.161306] netfs veth [32658.163491] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32658.170961] nft_chain_nat xt_MASQUERADE [32658.176438] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32658.178882] nf_nat nf_conntrack [32658.197619] [32658.197620] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32658.201537] nf_defrag_ipv6 [32658.206754] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32658.209979] nf_defrag_ipv4 [32658.211470] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32658.218595] nft_compat nf_tables [32658.221386] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32658.228509] nfnetlink bridge [32658.231300] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32658.238425] stp llc [32658.241751] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32658.248875] xfs libcrc32c [32658.251841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32658.258964] sunrpc intel_rapl_msr [32658.261149] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32658.269225] intel_rapl_common [32658.271930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32658.277668] sb_edac x86_pkg_temp_thermal [32658.281065] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32658.288188] coretemp kvm_intel [32658.291240] Call Trace: [32658.291241] [32658.298364] kvm irqbypass [32658.302369] ? watchdog_timer_fn+0x1b4/0x220 [32658.309490] i2c_i801 iTCO_wdt [32658.312632] ? __hrtimer_run_queues+0x1d3/0x360 [32658.315072] iTCO_vendor_support i2c_smbus [32658.317086] ? hrtimer_interrupt+0x104/0x240 [32658.319789] crct10dif_pclmul joydev [32658.324052] ? __local_bh_enable+0x50/0x60 [32658.327102] crc32_pclmul ipmi_ssif [32658.331630] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32658.335717] ghash_clmulni_intel [32658.339983] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32658.343553] mei_me mei [32658.347643] [32658.351126] lpc_ich sha512_ssse3 [32658.356343] [32658.356345] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32658.359568] acpi_ipmi mfd_core [32658.364531] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32658.366969] wmi ipmi_si [32658.369068] do_raw_spin_lock+0xb5/0xc0 [32658.372379] ipmi_devintf ipmi_msghandler [32658.374477] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32658.379780] ioatdma acpi_pad [32658.382920] try_get_cap_refs+0x159/0x880 [ceph] [32658.387787] acpi_power_meter ixgbe [32658.390323] __ceph_get_caps+0x4a1/0x760 [ceph] [32658.394150] igb mdio [32658.398158] ? __file_remove_privs+0x89/0x110 [32658.402937] crc32c_intel nvme [32658.405902] ? wake_bit_function+0x70/0x70 [32658.410514] ptp nvme_core [32658.413998] ? ceph_write_iter+0x260/0x770 [ceph] [32658.418520] i2c_algo_bit [32658.420792] ceph_write_iter+0x260/0x770 [ceph] [32658.425143] pps_core dca [32658.428195] ? lock_is_held_type+0xaa/0x120 [32658.432282] fuse [32658.434991] vfs_write+0x324/0x4f0 [32658.439685] irq event stamp: 182412552 [32658.442307] ksys_pwrite64+0x65/0xa0 [32658.446826] hardirqs last enabled at (182412551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32658.449443] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32658.453621] hardirqs last disabled at (182412552): [] sysvec_apic_timer_interrupt+0xe/0x90 [32658.455547] do_syscall_64+0x38/0x80 [32658.458942] softirqs last enabled at (182412550): [] __do_softirq+0x2be/0x371 [32658.462686] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32658.466258] softirqs last disabled at (182412545): [] irq_exit_rcu+0xb0/0xd0 [32658.476501] RIP: 0033:0x7f48f0eeb93b [32658.481633] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32658.491443] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32658.495014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32658.495031] Workqueue: events delayed_work [ceph] [32658.503802] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32658.508845] [32658.508846] RIP: 0010:ceph_check_delayed_caps+0x66/0x270 [ceph] [32658.517442] ORIG_RAX: 0000000000000012 [32658.521013] Code: 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 <4c> 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 [32658.530393] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32658.549130] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [32658.556600] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32658.561298] [32658.561299] RAX: 0000000000023e20 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [32658.566514] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32658.568007] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [32658.573917] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32658.577756] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [32658.596493] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32658.603617] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32658.608841] [32658.615958] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32658.676628] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32658.684704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32658.690442] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32658.697566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32658.704690] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32658.711815] Call Trace: [32658.714260] [32658.716270] ? watchdog_timer_fn+0x1b4/0x220 [32658.720535] ? __hrtimer_run_queues+0x1d3/0x360 [32658.725060] ? hrtimer_interrupt+0x104/0x240 [32658.729324] ? __local_bh_enable+0x50/0x60 [32658.733415] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32658.738638] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32658.743596] [32658.745692] [32658.747791] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32658.753100] ? ceph_check_delayed_caps+0x66/0x270 [ceph] [32658.758419] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [32658.763734] delayed_work+0x1d8/0x280 [ceph] [32658.768030] ? process_scheduled_works+0x22a/0x510 [32658.772822] process_scheduled_works+0x2a4/0x510 [32658.777438] worker_thread+0x188/0x340 [32658.781185] ? rescuer_thread+0x310/0x310 [32658.785189] kthread+0xf5/0x130 [32658.788327] ? kthread_complete_and_exit+0x20/0x20 [32658.793110] ret_from_fork+0x31/0x40 [32658.796680] ? kthread_complete_and_exit+0x20/0x20 [32658.801464] ret_from_fork_asm+0x11/0x20 [32658.805387] [32686.031833] watchdog: BUG: soft lockup - CPU#0 stuck for 19327s! [dbench:78683] [32686.039133] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32686.043832] watchdog: BUG: soft lockup - CPU#7 stuck for 28324s! [kworker/7:4:78808] [32686.094582] irq event stamp: 55962068 [32686.094584] hardirqs last enabled at (55962067): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32686.102315] Modules linked in: [32686.105971] hardirqs last disabled at (55962068): [] sysvec_apic_timer_interrupt+0xe/0x90 [32686.116127] ceph [32686.119180] softirqs last enabled at (55962052): [] __do_softirq+0x2be/0x371 [32686.128904] libceph [32686.130827] softirqs last disabled at (55962047): [] irq_exit_rcu+0xb0/0xd0 [32686.139516] dns_resolver [32686.141707] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32686.150227] fscache netfs [32686.152840] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32686.161784] veth nft_chain_nat [32686.164490] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32686.171959] xt_MASQUERADE nf_nat [32686.175098] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32686.180575] nf_conntrack nf_defrag_ipv6 [32686.183886] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32686.202625] nf_defrag_ipv4 nft_compat [32686.206541] [32686.206542] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32686.211757] nf_tables [32686.215517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32686.217010] nfnetlink bridge [32686.224133] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32686.226507] stp llc [32686.233633] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32686.236597] xfs libcrc32c [32686.243723] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32686.245906] sunrpc [32686.253030] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32686.255733] intel_rapl_msr intel_rapl_common [32686.262858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32686.264954] sb_edac x86_pkg_temp_thermal [32686.273033] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32686.277384] coretemp kvm_intel [32686.283122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32686.287124] kvm irqbypass [32686.294248] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32686.297386] i2c_i801 iTCO_wdt [32686.304511] Call Trace: [32686.307231] iTCO_vendor_support i2c_smbus [32686.314357] [32686.317406] crct10dif_pclmul joydev [32686.319853] ? watchdog_timer_fn+0x1b4/0x220 [32686.323941] crc32_pclmul ipmi_ssif [32686.325953] ? __hrtimer_run_queues+0x1d3/0x360 [32686.329524] ghash_clmulni_intel mei_me [32686.333790] ? hrtimer_interrupt+0x104/0x240 [32686.337272] mei lpc_ich [32686.341795] ? __local_bh_enable+0x50/0x60 [32686.345627] sha512_ssse3 acpi_ipmi [32686.349894] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32686.352422] mfd_core wmi [32686.356519] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32686.360005] ipmi_si ipmi_devintf [32686.365231] [32686.367847] ipmi_msghandler ioatdma [32686.372805] [32686.376115] acpi_pad acpi_power_meter [32686.378231] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32686.381800] ixgbe igb [32686.383902] ? queued_spin_lock_slowpath+0x18/0x2e0 [32686.387643] mdio crc32c_intel [32686.392949] do_raw_spin_lock+0xb5/0xc0 [32686.395305] nvme ptp [32686.400177] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32686.403233] nvme_core i2c_algo_bit [32686.407068] try_get_cap_refs+0x159/0x880 [ceph] [32686.409336] pps_core dca [32686.414126] __ceph_get_caps+0x4a1/0x760 [ceph] [32686.417603] fuse [32686.422236] ? __file_remove_privs+0x89/0x110 [32686.424851] irq event stamp: 182633434 [32686.429373] ? wake_bit_function+0x70/0x70 [32686.431298] hardirqs last enabled at (182633433): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32686.435650] ? ceph_write_iter+0x260/0x770 [ceph] [32686.439392] hardirqs last disabled at (182633434): [] sysvec_apic_timer_interrupt+0xe/0x90 [32686.443483] ceph_write_iter+0x260/0x770 [ceph] [32686.453727] softirqs last enabled at (182633432): [] __do_softirq+0x2be/0x371 [32686.458430] ? lock_is_held_type+0xaa/0x120 [32686.468236] softirqs last disabled at (182633427): [] irq_exit_rcu+0xb0/0xd0 [32686.472762] vfs_write+0x324/0x4f0 [32686.481531] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32686.485714] ksys_pwrite64+0x65/0xa0 [32686.494305] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32686.497705] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32686.507080] Workqueue: events delayed_work [ceph] [32686.510652] do_syscall_64+0x38/0x80 [32686.518123] [32686.523254] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32686.527951] RIP: 0010:ceph_check_delayed_caps+0x6f/0x270 [ceph] [32686.531523] RIP: 0033:0x7f48f0eeb93b [32686.533010] Code: 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 <48> 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 [32686.538058] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32686.543967] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [32686.547537] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32686.566274] [32686.566275] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [32686.585013] ORIG_RAX: 0000000000000012 [32686.590235] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [32686.595455] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32686.596946] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [32686.604072] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32686.607903] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32686.615027] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32686.622151] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32686.629273] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32686.636397] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32686.643532] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32686.650655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32686.657785] [32686.664903] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32686.695151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32686.702274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32686.709400] Call Trace: [32686.711846] [32686.713854] ? watchdog_timer_fn+0x1b4/0x220 [32686.718120] ? __hrtimer_run_queues+0x1d3/0x360 [32686.722646] ? hrtimer_interrupt+0x104/0x240 [32686.726915] ? __local_bh_enable+0x50/0x60 [32686.731009] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32686.736236] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32686.741197] [32686.743295] [32686.745393] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32686.750700] ? ceph_check_delayed_caps+0x6f/0x270 [ceph] [32686.756020] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [32686.761343] delayed_work+0x1d8/0x280 [ceph] [32686.765625] ? process_scheduled_works+0x22a/0x510 [32686.770413] process_scheduled_works+0x2a4/0x510 [32686.775030] worker_thread+0x188/0x340 [32686.778777] ? rescuer_thread+0x310/0x310 [32686.782782] kthread+0xf5/0x130 [32686.785928] ? kthread_complete_and_exit+0x20/0x20 [32686.790713] ret_from_fork+0x31/0x40 [32686.794292] ? kthread_complete_and_exit+0x20/0x20 [32686.799075] ret_from_fork_asm+0x11/0x20 [32686.803008] [32714.032311] watchdog: BUG: soft lockup - CPU#0 stuck for 19353s! [dbench:78683] [32714.039611] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32714.044311] watchdog: BUG: soft lockup - CPU#7 stuck for 28350s! [kworker/7:4:78808] [32714.095054] irq event stamp: 56020164 [32714.095055] hardirqs last enabled at (56020163): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32714.102785] Modules linked in: [32714.106440] hardirqs last disabled at (56020164): [] sysvec_apic_timer_interrupt+0xe/0x90 [32714.116599] ceph [32714.119648] softirqs last enabled at (56020128): [] __do_softirq+0x2be/0x371 [32714.129373] libceph dns_resolver [32714.131298] softirqs last disabled at (56020123): [] irq_exit_rcu+0xb0/0xd0 [32714.139994] fscache netfs [32714.143303] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32714.151813] veth nft_chain_nat [32714.154517] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32714.154518] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32714.163459] xt_MASQUERADE nf_nat [32714.166598] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32714.174070] nf_conntrack [32714.179547] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32714.182856] nf_defrag_ipv6 nf_defrag_ipv4 [32714.201595] [32714.201596] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32714.204212] nft_compat [32714.209429] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32714.213520] nf_tables nfnetlink [32714.215009] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32714.222135] bridge stp [32714.224578] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32714.231708] llc [32714.234937] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32714.242059] xfs libcrc32c [32714.244504] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32714.251627] sunrpc intel_rapl_msr [32714.253466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32714.260589] intel_rapl_common sb_edac [32714.263294] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32714.271371] x86_pkg_temp_thermal [32714.274768] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32714.280506] coretemp kvm_intel [32714.284249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32714.291375] kvm irqbypass [32714.294684] Call Trace: [32714.294686] [32714.301825] i2c_i801 iTCO_wdt [32714.304964] ? watchdog_timer_fn+0x1b4/0x220 [32714.312106] iTCO_vendor_support i2c_smbus [32714.314810] ? __hrtimer_run_queues+0x1d3/0x360 [32714.317252] crct10dif_pclmul joydev [32714.319266] ? hrtimer_interrupt+0x104/0x240 [32714.322314] crc32_pclmul ipmi_ssif [32714.326579] ? __local_bh_enable+0x50/0x60 [32714.330670] ghash_clmulni_intel mei_me [32714.335197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32714.338763] mei lpc_ich [32714.343030] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32714.346512] sha512_ssse3 acpi_ipmi [32714.350604] [32714.354434] mfd_core wmi [32714.359652] [32714.362182] ipmi_si ipmi_devintf ipmi_msghandler [32714.367141] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32714.370623] ioatdma acpi_pad [32714.372724] ? queued_spin_lock_slowpath+0x16/0x2e0 [32714.375339] acpi_power_meter [32714.377437] do_raw_spin_lock+0xb5/0xc0 [32714.382132] ixgbe igb [32714.387439] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32714.390400] mdio crc32c_intel [32714.395274] try_get_cap_refs+0x159/0x880 [ceph] [32714.398236] nvme ptp [32714.402071] __ceph_get_caps+0x4a1/0x760 [ceph] [32714.404424] nvme_core [32714.409213] ? __file_remove_privs+0x89/0x110 [32714.412259] i2c_algo_bit pps_core [32714.416871] ? wake_bit_function+0x70/0x70 [32714.419141] dca fuse [32714.423666] ? ceph_write_iter+0x260/0x770 [ceph] [32714.426021] [32714.426022] irq event stamp: 182854212 [32714.430374] ceph_write_iter+0x260/0x770 [ceph] [32714.433771] hardirqs last enabled at (182854211): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32714.437866] ? lock_is_held_type+0xaa/0x120 [32714.440131] hardirqs last disabled at (182854212): [] sysvec_apic_timer_interrupt+0xe/0x90 [32714.444832] vfs_write+0x324/0x4f0 [32714.446320] softirqs last enabled at (182854210): [] __do_softirq+0x2be/0x371 [32714.450069] ksys_pwrite64+0x65/0xa0 [32714.454588] softirqs last disabled at (182854205): [] irq_exit_rcu+0xb0/0xd0 [32714.464832] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32714.469010] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32714.478821] do_syscall_64+0x38/0x80 [32714.482218] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32714.490996] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32714.494569] Workqueue: events delayed_work [ceph] [32714.503165] RIP: 0033:0x7f48f0eeb93b [32714.508295] [32714.517674] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32714.521245] RIP: 0010:lock_acquire+0xf0/0x2b0 [32714.528716] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32714.533760] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32714.538456] ORIG_RAX: 0000000000000012 [32714.542026] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32714.543518] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32714.562255] [32714.562256] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32714.566606] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32714.571823] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32714.590561] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32714.594393] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32714.599609] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32714.606733] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32714.608224] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32714.615350] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32714.622477] [32714.629598] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32714.682605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32714.688341] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32714.695466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32714.702588] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32714.709718] Call Trace: [32714.712167] [32714.714178] ? watchdog_timer_fn+0x1b4/0x220 [32714.718442] ? __hrtimer_run_queues+0x1d3/0x360 [32714.722968] ? hrtimer_interrupt+0x104/0x240 [32714.727238] ? __local_bh_enable+0x50/0x60 [32714.731330] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32714.736548] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32714.741504] [32714.743601] [32714.745715] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32714.751023] ? lock_acquire+0xf0/0x2b0 [32714.754773] ? igrab+0x43/0x50 [32714.757823] ? lock_release+0xc5/0x260 [32714.761569] _raw_spin_lock+0x30/0x40 [32714.765233] ? igrab+0x1d/0x50 [32714.768283] igrab+0x1d/0x50 [32714.771165] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32714.776312] delayed_work+0x1d8/0x280 [ceph] [32714.780593] ? process_scheduled_works+0x22a/0x510 [32714.785383] process_scheduled_works+0x2a4/0x510 [32714.790002] worker_thread+0x188/0x340 [32714.793748] ? rescuer_thread+0x310/0x310 [32714.797752] kthread+0xf5/0x130 [32714.800889] ? kthread_complete_and_exit+0x20/0x20 [32714.805673] ret_from_fork+0x31/0x40 [32714.809243] ? kthread_complete_and_exit+0x20/0x20 [32714.814027] ret_from_fork_asm+0x11/0x20 [32714.817952] [32716.646723] rcu: INFO: rcu_preempt self-detected stall on CPU [32716.652478] rcu: 7-....: (20176712 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/20175381 fqs=10165683 [32716.663158] rcu: (t=20779667 jiffies g=1309609 q=595381 ncpus=8) [32716.669250] Sending NMI from CPU 7 to CPUs 0: [32716.673602] NMI backtrace for cpu 0 [32716.673604] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32716.673605] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32716.673606] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32716.673609] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32716.673610] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32716.673611] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32716.673612] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32716.673613] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32716.673614] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32716.673614] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32716.673615] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32716.673616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32716.673617] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32716.673618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32716.673619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32716.673619] Call Trace: [32716.673620] [32716.673621] ? nmi_cpu_backtrace+0xad/0x120 [32716.673624] ? nmi_cpu_backtrace_handler+0x11/0x20 [32716.673626] ? nmi_handle+0xd4/0x260 [32716.673630] ? default_do_nmi+0x49/0xf0 [32716.673631] ? exc_nmi+0x11e/0x150 [32716.673633] ? end_repeat_nmi+0x16/0x67 [32716.673637] ? queued_spin_lock_slowpath+0x16/0x2e0 [32716.673639] ? queued_spin_lock_slowpath+0x16/0x2e0 [32716.673642] ? queued_spin_lock_slowpath+0x16/0x2e0 [32716.673644] [32716.673644] [32716.673645] do_raw_spin_lock+0xb5/0xc0 [32716.673647] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32716.673661] try_get_cap_refs+0x159/0x880 [ceph] [32716.673675] __ceph_get_caps+0x4a1/0x760 [ceph] [32716.673687] ? __file_remove_privs+0x89/0x110 [32716.673689] ? wake_bit_function+0x70/0x70 [32716.673691] ? ceph_write_iter+0x260/0x770 [ceph] [32716.673701] ceph_write_iter+0x260/0x770 [ceph] [32716.673712] ? lock_is_held_type+0xaa/0x120 [32716.673714] vfs_write+0x324/0x4f0 [32716.673718] ksys_pwrite64+0x65/0xa0 [32716.673720] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32716.673724] do_syscall_64+0x38/0x80 [32716.673727] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32716.673729] RIP: 0033:0x7f48f0eeb93b [32716.673748] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32716.673749] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [32716.673750] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32716.673751] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32716.673751] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32716.673752] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32716.673753] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32716.673756] [32716.674600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32716.978134] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32716.985606] Workqueue: events delayed_work [ceph] [32716.990329] RIP: 0010:lock_release+0x1d2/0x260 [32716.994765] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [32717.013503] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [32717.018720] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [32717.025845] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [32717.032969] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [32717.040094] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [32717.047216] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32717.054342] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32717.062419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32717.068156] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32717.075282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32717.082403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32717.089528] Call Trace: [32717.091975] [32717.093985] ? rcu_dump_cpu_stacks+0xd1/0x180 [32717.098344] ? rcu_sched_clock_irq+0x633/0x1390 [32717.102869] ? lock_release+0xc5/0x260 [32717.106619] ? lock_is_held_type+0xaa/0x120 [32717.110800] ? lock_is_held_type+0xaa/0x120 [32717.114986] ? tick_sched_do_timer+0xa0/0xa0 [32717.119256] ? rcu_read_lock_sched_held+0x56/0x90 [32717.123956] ? tick_sched_do_timer+0xa0/0xa0 [32717.128226] ? update_process_times+0x5f/0x90 [32717.132579] ? tick_sched_handle.isra.16+0x1e/0x50 [32717.137371] ? tick_sched_timer+0x6f/0x80 [32717.141385] ? __hrtimer_run_queues+0x1d3/0x360 [32717.145925] ? hrtimer_interrupt+0x104/0x240 [32717.150188] ? __local_bh_enable+0x50/0x60 [32717.154284] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32717.159506] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32717.164462] [32717.166560] [32717.168659] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32717.173973] ? igrab+0x43/0x50 [32717.177050] ? igrab+0x43/0x50 [32717.180107] ? lock_release+0x1d2/0x260 [32717.183940] ? lock_release+0xc5/0x260 [32717.187694] _raw_spin_unlock+0x1b/0x40 [32717.191531] igrab+0x43/0x50 [32717.194417] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32717.199569] delayed_work+0x1d8/0x280 [ceph] [32717.203858] ? process_scheduled_works+0x22a/0x510 [32717.208646] process_scheduled_works+0x2a4/0x510 [32717.213266] worker_thread+0x188/0x340 [32717.217021] ? rescuer_thread+0x310/0x310 [32717.221040] kthread+0xf5/0x130 [32717.224178] ? kthread_complete_and_exit+0x20/0x20 [32717.228972] ret_from_fork+0x31/0x40 [32717.232548] ? kthread_complete_and_exit+0x20/0x20 [32717.237334] ret_from_fork_asm+0x11/0x20 [32717.241263] [32742.032791] watchdog: BUG: soft lockup - CPU#0 stuck for 19379s! [dbench:78683] [32742.040100] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32742.044791] watchdog: BUG: soft lockup - CPU#7 stuck for 28376s! [kworker/7:4:78808] [32742.095550] irq event stamp: 56078296 [32742.103281] Modules linked in: ceph libceph dns_resolver fscache netfs [32742.106939] hardirqs last enabled at (56078295): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32742.113465] veth nft_chain_nat xt_MASQUERADE [32742.123621] hardirqs last disabled at (56078296): [] sysvec_apic_timer_interrupt+0xe/0x90 [32742.123624] nf_nat [32742.123625] softirqs last enabled at (56078232): [] __do_softirq+0x2be/0x371 [32742.127972] nf_conntrack nf_defrag_ipv6 [32742.137697] softirqs last disabled at (56078227): [] irq_exit_rcu+0xb0/0xd0 [32742.139794] nf_defrag_ipv4 [32742.148478] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32742.152396] nft_compat nf_tables [32742.160906] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32742.163695] nfnetlink bridge [32742.172641] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32742.175952] stp llc [32742.183422] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32742.186386] xfs [32742.191865] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32742.194048] libcrc32c sunrpc [32742.212786] [32742.212787] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32742.214624] intel_rapl_msr intel_rapl_common [32742.219840] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32742.222806] sb_edac x86_pkg_temp_thermal [32742.224296] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32742.231418] coretemp kvm_intel [32742.235769] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32742.242894] kvm irqbypass [32742.246898] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32742.254023] i2c_i801 iTCO_wdt [32742.257159] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32742.264300] iTCO_vendor_support [32742.267006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32742.274130] i2c_smbus crct10dif_pclmul [32742.277189] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32742.285267] joydev crc32_pclmul [32742.288490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32742.294229] ipmi_ssif [32742.298059] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32742.305189] ghash_clmulni_intel [32742.308415] Call Trace: [32742.315539] mei_me mei [32742.317899] [32742.325021] lpc_ich sha512_ssse3 [32742.328248] ? watchdog_timer_fn+0x1b4/0x220 [32742.330689] acpi_ipmi mfd_core [32742.333135] ? __hrtimer_run_queues+0x1d3/0x360 [32742.335143] wmi [32742.338475] ? hrtimer_interrupt+0x104/0x240 [32742.342737] ipmi_si [32742.345873] ? __local_bh_enable+0x50/0x60 [32742.350397] ipmi_devintf [32742.352239] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32742.356500] ipmi_msghandler [32742.358684] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32742.362775] ioatdma acpi_pad acpi_power_meter [32742.365393] [32742.370610] ixgbe igb [32742.373487] [32742.378442] mdio crc32c_intel [32742.382899] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32742.384995] nvme [32742.387357] ? queued_spin_lock_slowpath+0x18/0x2e0 [32742.389449] ptp [32742.392503] do_raw_spin_lock+0xb5/0xc0 [32742.397804] nvme_core i2c_algo_bit [32742.399730] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32742.404601] pps_core [32742.406441] try_get_cap_refs+0x159/0x880 [ceph] [32742.410268] dca [32742.413757] __ceph_get_caps+0x4a1/0x760 [ceph] [32742.418536] fuse [32742.420811] ? __file_remove_privs+0x89/0x110 [32742.425416] [32742.425417] irq event stamp: 183070278 [32742.427256] ? wake_bit_function+0x70/0x70 [32742.431779] hardirqs last enabled at (183070277): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32742.433705] ? ceph_write_iter+0x260/0x770 [ceph] [32742.438055] hardirqs last disabled at (183070278): [] sysvec_apic_timer_interrupt+0xe/0x90 [32742.439545] ceph_write_iter+0x260/0x770 [ceph] [32742.443289] softirqs last enabled at (183070276): [] __do_softirq+0x2be/0x371 [32742.447384] ? lock_is_held_type+0xaa/0x120 [32742.457623] softirqs last disabled at (183070271): [] irq_exit_rcu+0xb0/0xd0 [32742.462325] vfs_write+0x324/0x4f0 [32742.472131] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32742.476660] ksys_pwrite64+0x65/0xa0 [32742.485427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32742.489606] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32742.498201] Workqueue: events delayed_work [ceph] [32742.501601] do_syscall_64+0x38/0x80 [32742.510975] [32742.514549] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32742.522017] RIP: 0010:lock_acquire+0xf0/0x2b0 [32742.527150] RIP: 0033:0x7f48f0eeb93b [32742.531846] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32742.535417] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32742.536907] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32742.541953] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32742.546303] [32742.546304] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32742.549872] ORIG_RAX: 0000000000000012 [32742.568610] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32742.587349] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32742.592566] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32742.597783] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32742.599275] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32742.606398] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32742.610230] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32742.617353] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32742.624487] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32742.631610] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32742.638735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32742.645863] [32742.652983] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32742.697496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32742.704619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32742.711743] Call Trace: [32742.714196] [32742.716208] ? watchdog_timer_fn+0x1b4/0x220 [32742.720488] ? __hrtimer_run_queues+0x1d3/0x360 [32742.725016] ? hrtimer_interrupt+0x104/0x240 [32742.729286] ? __local_bh_enable+0x50/0x60 [32742.733378] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32742.738603] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32742.743561] [32742.745657] [32742.747755] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32742.753063] ? lock_acquire+0xf0/0x2b0 [32742.756814] ? igrab+0x43/0x50 [32742.759863] ? lock_release+0xc5/0x260 [32742.763609] _raw_spin_lock+0x30/0x40 [32742.767274] ? igrab+0x1d/0x50 [32742.770323] igrab+0x1d/0x50 [32742.773204] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32742.778353] delayed_work+0x1d8/0x280 [ceph] [32742.782633] ? process_scheduled_works+0x22a/0x510 [32742.787423] process_scheduled_works+0x2a4/0x510 [32742.792038] worker_thread+0x188/0x340 [32742.795788] ? rescuer_thread+0x310/0x310 [32742.799798] kthread+0xf5/0x130 [32742.802937] ? kthread_complete_and_exit+0x20/0x20 [32742.807720] ret_from_fork+0x31/0x40 [32742.811302] ? kthread_complete_and_exit+0x20/0x20 [32742.816093] ret_from_fork_asm+0x11/0x20 [32742.820023] [32770.033269] watchdog: BUG: soft lockup - CPU#0 stuck for 19405s! [dbench:78683] [32770.040569] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32770.045269] watchdog: BUG: soft lockup - CPU#7 stuck for 28402s! [kworker/7:4:78808] [32770.096011] irq event stamp: 56136498 [32770.096012] hardirqs last enabled at (56136497): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32770.103741] Modules linked in: [32770.107399] hardirqs last disabled at (56136498): [] sysvec_apic_timer_interrupt+0xe/0x90 [32770.117558] ceph [32770.120609] softirqs last enabled at (56136442): [] __do_softirq+0x2be/0x371 [32770.130332] libceph [32770.132256] softirqs last disabled at (56136437): [] irq_exit_rcu+0xb0/0xd0 [32770.140945] dns_resolver fscache [32770.143133] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32770.151662] netfs veth [32770.154972] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32770.154973] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32770.163916] nft_chain_nat xt_MASQUERADE [32770.166359] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32770.173831] nf_nat nf_conntrack [32770.179308] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32770.183225] nf_defrag_ipv6 nf_defrag_ipv4 [32770.201962] [32770.201963] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32770.205186] nft_compat nf_tables [32770.210404] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32770.214494] nfnetlink bridge [32770.215987] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32770.223109] stp llc [32770.226421] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32770.233546] xfs libcrc32c [32770.236509] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32770.243632] sunrpc [32770.245818] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32770.252944] intel_rapl_msr [32770.255663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32770.262786] intel_rapl_common sb_edac [32770.264883] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32770.272961] x86_pkg_temp_thermal [32770.275752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32770.281491] coretemp kvm_intel [32770.285233] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32770.292359] kvm irqbypass [32770.295668] Call Trace: [32770.295669] [32770.302792] i2c_i801 iTCO_wdt [32770.305949] ? watchdog_timer_fn+0x1b4/0x220 [32770.313071] iTCO_vendor_support i2c_smbus [32770.315777] ? __hrtimer_run_queues+0x1d3/0x360 [32770.318219] crct10dif_pclmul joydev [32770.320235] ? hrtimer_interrupt+0x104/0x240 [32770.323280] crc32_pclmul ipmi_ssif [32770.327545] ? __local_bh_enable+0x50/0x60 [32770.331635] ghash_clmulni_intel mei_me [32770.336164] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32770.339732] mei lpc_ich [32770.343997] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32770.347479] sha512_ssse3 acpi_ipmi [32770.351569] [32770.355399] mfd_core wmi [32770.360617] [32770.360619] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32770.363148] ipmi_si ipmi_devintf ipmi_msghandler [32770.368111] ? queued_spin_lock_slowpath+0x18/0x2e0 [32770.371591] ioatdma acpi_pad [32770.373689] do_raw_spin_lock+0xb5/0xc0 [32770.376305] acpi_power_meter [32770.378405] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32770.383705] ixgbe igb mdio [32770.388406] try_get_cap_refs+0x159/0x880 [ceph] [32770.393276] crc32c_intel nvme [32770.396244] __ceph_get_caps+0x4a1/0x760 [ceph] [32770.400069] ptp [32770.403037] ? __file_remove_privs+0x89/0x110 [32770.407817] nvme_core i2c_algo_bit [32770.410608] ? wake_bit_function+0x70/0x70 [32770.415219] pps_core dca [32770.418273] ? ceph_write_iter+0x260/0x770 [ceph] [32770.422794] fuse [32770.424632] ceph_write_iter+0x260/0x770 [ceph] [32770.428982] irq event stamp: 183290940 [32770.432471] ? lock_is_held_type+0xaa/0x120 [32770.436558] hardirqs last enabled at (183290939): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32770.439178] vfs_write+0x324/0x4f0 [32770.443872] hardirqs last disabled at (183290940): [] sysvec_apic_timer_interrupt+0xe/0x90 [32770.445800] ksys_pwrite64+0x65/0xa0 [32770.450321] softirqs last enabled at (183290938): [] __do_softirq+0x2be/0x371 [32770.454064] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32770.458242] softirqs last disabled at (183290933): [] irq_exit_rcu+0xb0/0xd0 [32770.468487] do_syscall_64+0x38/0x80 [32770.471883] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32770.481694] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32770.485263] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32770.485264] Workqueue: events delayed_work [ceph] [32770.494036] RIP: 0033:0x7f48f0eeb93b [32770.499166] [32770.499167] RIP: 0010:lock_acquire+0x4a/0x2b0 [32770.507766] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32770.511335] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [32770.520711] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32770.525756] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [32770.533225] ORIG_RAX: 0000000000000012 [32770.537925] [32770.537925] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [32770.541495] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32770.542984] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [32770.547335] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32770.566073] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [32770.584812] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32770.590028] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [32770.595245] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32770.599076] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32770.600567] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32770.607692] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32770.614821] [32770.621955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32770.687826] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32770.694949] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32770.702090] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32770.709214] Call Trace: [32770.711676] [32770.713689] ? watchdog_timer_fn+0x1b4/0x220 [32770.717957] ? __hrtimer_run_queues+0x1d3/0x360 [32770.722489] ? hrtimer_interrupt+0x104/0x240 [32770.726756] ? __local_bh_enable+0x50/0x60 [32770.730851] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32770.736073] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32770.741033] [32770.743129] [32770.745227] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32770.750534] ? lock_acquire+0x4a/0x2b0 [32770.754285] ? igrab+0x43/0x50 [32770.757335] ? lock_release+0xc5/0x260 [32770.761081] _raw_spin_lock+0x30/0x40 [32770.764745] ? igrab+0x1d/0x50 [32770.767795] igrab+0x1d/0x50 [32770.770677] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32770.775823] delayed_work+0x1d8/0x280 [ceph] [32770.780104] ? process_scheduled_works+0x22a/0x510 [32770.784893] process_scheduled_works+0x2a4/0x510 [32770.789509] worker_thread+0x188/0x340 [32770.793260] ? rescuer_thread+0x310/0x310 [32770.797271] kthread+0xf5/0x130 [32770.800409] ? kthread_complete_and_exit+0x20/0x20 [32770.805193] ret_from_fork+0x31/0x40 [32770.808763] ? kthread_complete_and_exit+0x20/0x20 [32770.813547] ret_from_fork_asm+0x11/0x20 [32770.817471] [32798.033749] watchdog: BUG: soft lockup - CPU#0 stuck for 19431s! [dbench:78683] [32798.041051] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32798.045748] watchdog: BUG: soft lockup - CPU#7 stuck for 28428s! [kworker/7:4:78808] [32798.096499] irq event stamp: 56194652 [32798.096500] hardirqs last enabled at (56194651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32798.104231] Modules linked in: [32798.107888] hardirqs last disabled at (56194652): [] sysvec_apic_timer_interrupt+0xe/0x90 [32798.118046] ceph [32798.121095] softirqs last enabled at (56194584): [] __do_softirq+0x2be/0x371 [32798.130819] libceph [32798.132742] softirqs last disabled at (56194579): [] irq_exit_rcu+0xb0/0xd0 [32798.141430] dns_resolver [32798.143612] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32798.152140] fscache netfs [32798.154757] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32798.154759] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32798.163701] veth nft_chain_nat [32798.166406] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32798.173893] xt_MASQUERADE nf_nat [32798.179372] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32798.182509] nf_conntrack nf_defrag_ipv6 [32798.201246] [32798.201247] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32798.204558] nf_defrag_ipv4 [32798.209775] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32798.213691] nft_compat [32798.215181] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32798.222305] nf_tables nfnetlink [32798.225097] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32798.232220] bridge stp [32798.234666] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32798.241790] llc [32798.245012] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32798.252146] xfs libcrc32c [32798.254591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32798.261714] sunrpc intel_rapl_msr [32798.263553] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32798.271630] intel_rapl_common [32798.274334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32798.280070] sb_edac [32798.283467] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32798.290591] x86_pkg_temp_thermal coretemp [32798.293644] Call Trace: [32798.293646] [32798.300767] kvm_intel [32798.302955] ? watchdog_timer_fn+0x1b4/0x220 [32798.310075] kvm irqbypass i2c_i801 [32798.314167] ? __hrtimer_run_queues+0x1d3/0x360 [32798.316611] iTCO_wdt [32798.318624] ? hrtimer_interrupt+0x104/0x240 [32798.320977] iTCO_vendor_support [32798.325243] ? __local_bh_enable+0x50/0x60 [32798.328727] i2c_smbus [32798.333253] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32798.335521] crct10dif_pclmul joydev [32798.339787] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32798.343009] crc32_pclmul ipmi_ssif [32798.347100] [32798.349457] ghash_clmulni_intel [32798.354676] [32798.358245] mei_me [32798.363204] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32798.366686] mei lpc_ich [32798.368788] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32798.372009] sha512_ssse3 [32798.374109] do_raw_spin_lock+0xb5/0xc0 [32798.376204] acpi_ipmi mfd_core [32798.381509] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32798.384038] wmi ipmi_si [32798.388912] try_get_cap_refs+0x159/0x880 [ceph] [32798.391527] ipmi_devintf ipmi_msghandler [32798.395362] __ceph_get_caps+0x4a1/0x760 [ceph] [32798.398495] ioatdma acpi_pad [32798.403284] ? __file_remove_privs+0x89/0x110 [32798.405809] acpi_power_meter ixgbe [32798.410437] ? wake_bit_function+0x70/0x70 [32798.414442] igb mdio [32798.418967] ? ceph_write_iter+0x260/0x770 [ceph] [32798.421931] crc32c_intel nvme [32798.426280] ceph_write_iter+0x260/0x770 [ceph] [32798.429765] ptp nvme_core [32798.433860] ? lock_is_held_type+0xaa/0x120 [32798.436125] i2c_algo_bit pps_core [32798.440844] vfs_write+0x324/0x4f0 [32798.443893] dca [32798.448435] ksys_pwrite64+0x65/0xa0 [32798.451148] fuse [32798.455324] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32798.458721] [32798.458722] irq event stamp: 183511716 [32798.462118] do_syscall_64+0x38/0x80 [32798.463956] hardirqs last enabled at (183511715): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32798.467527] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32798.469450] hardirqs last disabled at (183511716): [] sysvec_apic_timer_interrupt+0xe/0x90 [32798.474583] RIP: 0033:0x7f48f0eeb93b [32798.476070] softirqs last enabled at (183511714): [] __do_softirq+0x2be/0x371 [32798.479814] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32798.483387] softirqs last disabled at (183511709): [] irq_exit_rcu+0xb0/0xd0 [32798.493631] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32798.498674] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32798.508486] ORIG_RAX: 0000000000000012 [32798.512056] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32798.520827] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32798.539564] Workqueue: events delayed_work [ceph] [32798.548163] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32798.553379] [32798.553380] RIP: 0010:lock_acquire+0xf0/0x2b0 [32798.562757] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32798.566588] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32798.574058] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32798.581182] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32798.585879] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32798.593004] [32798.593004] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32798.594500] [32798.598846] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32798.662098] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32798.669220] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32798.676344] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32798.683470] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32798.691546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32798.697285] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32798.704408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32798.711550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32798.718672] Call Trace: [32798.721118] [32798.723130] ? watchdog_timer_fn+0x1b4/0x220 [32798.727411] ? __hrtimer_run_queues+0x1d3/0x360 [32798.731937] ? hrtimer_interrupt+0x104/0x240 [32798.736207] ? __local_bh_enable+0x50/0x60 [32798.740299] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32798.745523] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32798.750481] [32798.752578] [32798.754677] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32798.759985] ? lock_acquire+0xf0/0x2b0 [32798.763733] ? igrab+0x43/0x50 [32798.766785] ? lock_release+0xc5/0x260 [32798.770530] _raw_spin_lock+0x30/0x40 [32798.774195] ? igrab+0x1d/0x50 [32798.777245] igrab+0x1d/0x50 [32798.780123] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32798.785290] delayed_work+0x1d8/0x280 [ceph] [32798.789570] ? process_scheduled_works+0x22a/0x510 [32798.794360] process_scheduled_works+0x2a4/0x510 [32798.798977] worker_thread+0x188/0x340 [32798.802726] ? rescuer_thread+0x310/0x310 [32798.806738] kthread+0xf5/0x130 [32798.809884] ? kthread_complete_and_exit+0x20/0x20 [32798.814669] ret_from_fork+0x31/0x40 [32798.818247] ? kthread_complete_and_exit+0x20/0x20 [32798.823032] ret_from_fork_asm+0x11/0x20 [32798.826962] [32826.034228] watchdog: BUG: soft lockup - CPU#0 stuck for 19457s! [dbench:78683] [32826.041529] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32826.046228] watchdog: BUG: soft lockup - CPU#7 stuck for 28454s! [kworker/7:4:78808] [32826.096969] irq event stamp: 56252902 [32826.096970] hardirqs last enabled at (56252901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32826.104699] Modules linked in: [32826.108356] hardirqs last disabled at (56252902): [] sysvec_apic_timer_interrupt+0xe/0x90 [32826.118515] ceph [32826.121566] softirqs last enabled at (56252866): [] __do_softirq+0x2be/0x371 [32826.131289] libceph [32826.133213] softirqs last disabled at (56252861): [] irq_exit_rcu+0xb0/0xd0 [32826.141902] dns_resolver fscache [32826.144091] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32826.152601] netfs veth [32826.155928] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32826.155929] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32826.164874] nft_chain_nat xt_MASQUERADE [32826.167317] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32826.174789] nf_nat nf_conntrack [32826.180266] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32826.184183] nf_defrag_ipv6 nf_defrag_ipv4 [32826.202922] [32826.202923] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32826.206144] nft_compat nf_tables [32826.211363] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32826.215453] nfnetlink bridge [32826.216945] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32826.224067] stp [32826.227379] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32826.234502] llc xfs [32826.237466] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32826.244592] libcrc32c sunrpc [32826.246428] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32826.253553] intel_rapl_msr intel_rapl_common [32826.255737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32826.262862] sb_edac [32826.265824] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32826.273905] x86_pkg_temp_thermal [32826.278253] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32826.283991] coretemp [32826.286175] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32826.293298] kvm_intel kvm [32826.296628] Call Trace: [32826.296629] [32826.303751] irqbypass i2c_i801 [32826.306024] ? watchdog_timer_fn+0x1b4/0x220 [32826.313145] iTCO_wdt iTCO_vendor_support [32826.315850] ? __hrtimer_run_queues+0x1d3/0x360 [32826.318294] i2c_smbus crct10dif_pclmul [32826.320308] ? hrtimer_interrupt+0x104/0x240 [32826.323443] joydev crc32_pclmul [32826.327705] ? __local_bh_enable+0x50/0x60 [32826.331710] ipmi_ssif ghash_clmulni_intel [32826.336237] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32826.340066] mei_me mei [32826.344330] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32826.347553] lpc_ich sha512_ssse3 [32826.351644] [32826.355734] acpi_ipmi mfd_core [32826.360952] [32826.360953] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32826.363396] wmi ipmi_si [32826.368357] ? queued_spin_lock_slowpath+0x16/0x2e0 [32826.371664] ipmi_devintf ipmi_msghandler [32826.373764] do_raw_spin_lock+0xb5/0xc0 [32826.376905] ioatdma acpi_pad [32826.379005] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32826.384309] acpi_power_meter ixgbe [32826.386842] try_get_cap_refs+0x159/0x880 [ceph] [32826.391710] igb mdio crc32c_intel [32826.395721] __ceph_get_caps+0x4a1/0x760 [ceph] [32826.399546] nvme ptp [32826.402512] ? __file_remove_privs+0x89/0x110 [32826.407293] nvme_core [32826.410778] ? wake_bit_function+0x70/0x70 [32826.415389] i2c_algo_bit [32826.418788] ? ceph_write_iter+0x260/0x770 [ceph] [32826.423311] pps_core dca [32826.425580] ceph_write_iter+0x260/0x770 [ceph] [32826.429932] fuse [32826.432292] ? lock_is_held_type+0xaa/0x120 [32826.436380] irq event stamp: 183732342 [32826.439000] vfs_write+0x324/0x4f0 [32826.443695] hardirqs last enabled at (183732341): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32826.446316] ksys_pwrite64+0x65/0xa0 [32826.450835] hardirqs last disabled at (183732342): [] sysvec_apic_timer_interrupt+0xe/0x90 [32826.452761] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32826.456937] softirqs last enabled at (183732340): [] __do_softirq+0x2be/0x371 [32826.460681] do_syscall_64+0x38/0x80 [32826.464078] softirqs last disabled at (183732331): [] irq_exit_rcu+0xb0/0xd0 [32826.474323] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32826.477910] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32826.487722] RIP: 0033:0x7f48f0eeb93b [32826.492852] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32826.501630] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32826.505204] Workqueue: events delayed_work [ceph] [32826.513800] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32826.518843] [32826.518844] RIP: 0010:lock_acquire+0xf0/0x2b0 [32826.528221] ORIG_RAX: 0000000000000012 [32826.531792] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32826.539264] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32826.558000] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32826.562698] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32826.567917] [32826.567918] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32826.569407] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32826.573757] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32826.577589] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32826.596326] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32826.603449] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32826.608668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32826.615797] [32826.617283] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32826.676469] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32826.684545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32826.690283] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32826.697406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32826.704532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32826.711655] Call Trace: [32826.714101] [32826.716112] ? watchdog_timer_fn+0x1b4/0x220 [32826.720376] ? __hrtimer_run_queues+0x1d3/0x360 [32826.724916] ? hrtimer_interrupt+0x104/0x240 [32826.729179] ? __local_bh_enable+0x50/0x60 [32826.733273] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32826.738496] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32826.743453] [32826.745552] [32826.747651] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32826.752958] ? lock_acquire+0xf0/0x2b0 [32826.756706] ? igrab+0x43/0x50 [32826.759756] ? lock_release+0xc5/0x260 [32826.763501] _raw_spin_lock+0x30/0x40 [32826.767157] ? igrab+0x1d/0x50 [32826.770208] igrab+0x1d/0x50 [32826.773086] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32826.778236] delayed_work+0x1d8/0x280 [ceph] [32826.782518] ? process_scheduled_works+0x22a/0x510 [32826.787309] process_scheduled_works+0x2a4/0x510 [32826.791930] worker_thread+0x188/0x340 [32826.795680] ? rescuer_thread+0x310/0x310 [32826.799683] kthread+0xf5/0x130 [32826.802821] ? kthread_complete_and_exit+0x20/0x20 [32826.807608] ret_from_fork+0x31/0x40 [32826.811202] ? kthread_complete_and_exit+0x20/0x20 [32826.815987] ret_from_fork_asm+0x11/0x20 [32826.819923] [32854.034709] watchdog: BUG: soft lockup - CPU#0 stuck for 19483s! [dbench:78683] [32854.042017] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32854.046709] watchdog: BUG: soft lockup - CPU#7 stuck for 28480s! [kworker/7:4:78808] [32854.097465] irq event stamp: 56311088 [32854.097466] hardirqs last enabled at (56311087): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32854.105195] Modules linked in: [32854.108854] hardirqs last disabled at (56311088): [] sysvec_apic_timer_interrupt+0xe/0x90 [32854.119011] ceph [32854.122062] softirqs last enabled at (56311082): [] __do_softirq+0x2be/0x371 [32854.131785] libceph [32854.133710] softirqs last disabled at (56311077): [] irq_exit_rcu+0xb0/0xd0 [32854.142395] dns_resolver [32854.144580] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32854.153104] fscache netfs [32854.155725] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32854.164669] veth nft_chain_nat [32854.167388] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32854.174860] xt_MASQUERADE nf_nat [32854.177997] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32854.183475] nf_conntrack nf_defrag_ipv6 [32854.186787] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32854.205524] nf_defrag_ipv4 nft_compat [32854.209441] [32854.209442] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32854.214658] nf_tables nfnetlink [32854.218402] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32854.219892] bridge stp [32854.227016] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32854.230240] llc [32854.237365] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32854.239827] xfs libcrc32c [32854.246950] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32854.248788] sunrpc intel_rapl_msr [32854.255912] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32854.258616] intel_rapl_common sb_edac [32854.265739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32854.269139] x86_pkg_temp_thermal [32854.277214] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32854.280958] coretemp kvm_intel [32854.286698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32854.290007] kvm irqbypass [32854.297131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32854.300271] i2c_i801 [32854.307399] Call Trace: [32854.307401] [32854.310106] iTCO_wdt [32854.317231] ? watchdog_timer_fn+0x1b4/0x220 [32854.319500] iTCO_vendor_support i2c_smbus [32854.321945] ? __hrtimer_run_queues+0x1d3/0x360 [32854.323954] crct10dif_pclmul joydev [32854.326228] ? hrtimer_interrupt+0x104/0x240 [32854.330491] crc32_pclmul ipmi_ssif [32854.334581] ? __local_bh_enable+0x50/0x60 [32854.339108] ghash_clmulni_intel [32854.342678] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32854.346940] mei_me [32854.350424] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32854.354514] mei lpc_ich [32854.357738] [32854.362955] sha512_ssse3 [32854.365053] [32854.370012] acpi_ipmi mfd_core [32854.372541] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32854.374639] wmi [32854.377261] ? queued_spin_lock_slowpath+0x18/0x2e0 [32854.379354] ipmi_si ipmi_devintf [32854.382493] do_raw_spin_lock+0xb5/0xc0 [32854.387796] ipmi_msghandler ioatdma [32854.389635] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32854.394504] acpi_pad [32854.397819] try_get_cap_refs+0x159/0x880 [ceph] [32854.401644] acpi_power_meter ixgbe [32854.405220] __ceph_get_caps+0x4a1/0x760 [ceph] [32854.410000] igb [32854.412276] ? __file_remove_privs+0x89/0x110 [32854.416880] mdio [32854.420366] ? wake_bit_function+0x70/0x70 [32854.424890] crc32c_intel [32854.426728] ? ceph_write_iter+0x260/0x770 [ceph] [32854.431078] nvme [32854.433002] ceph_write_iter+0x260/0x770 [ceph] [32854.437107] ptp nvme_core [32854.439732] ? lock_is_held_type+0xaa/0x120 [32854.444425] i2c_algo_bit [32854.446350] vfs_write+0x324/0x4f0 [32854.450871] pps_core [32854.453581] ksys_pwrite64+0x65/0xa0 [32854.457754] dca [32854.460389] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32854.463786] fuse [32854.466059] do_syscall_64+0x38/0x80 [32854.469628] irq event stamp: 183953404 [32854.471464] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32854.476595] hardirqs last enabled at (183953403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32854.478522] RIP: 0033:0x7f48f0eeb93b [32854.482108] hardirqs last disabled at (183953404): [] sysvec_apic_timer_interrupt+0xe/0x90 [32854.485853] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32854.490896] softirqs last enabled at (183953402): [] __do_softirq+0x2be/0x371 [32854.501140] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32854.504710] softirqs last disabled at (183953397): [] irq_exit_rcu+0xb0/0xd0 [32854.514522] ORIG_RAX: 0000000000000012 [32854.533260] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32854.542029] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32854.547248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32854.555846] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32854.559676] Workqueue: events delayed_work [ceph] [32854.569054] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32854.576178] [32854.576179] RIP: 0010:lock_acquire+0xf0/0x2b0 [32854.583647] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32854.590772] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32854.595471] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32854.602593] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32854.604110] [32854.608453] [32854.608454] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32854.657455] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32854.664580] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32854.671703] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32854.678828] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32854.685952] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32854.694028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32854.699765] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32854.706890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32854.714014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32854.721140] Call Trace: [32854.723583] [32854.725594] ? watchdog_timer_fn+0x1b4/0x220 [32854.729858] ? __hrtimer_run_queues+0x1d3/0x360 [32854.734401] ? hrtimer_interrupt+0x104/0x240 [32854.738664] ? __local_bh_enable+0x50/0x60 [32854.742756] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32854.747972] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32854.752930] [32854.755025] [32854.757125] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32854.762432] ? lock_acquire+0xf0/0x2b0 [32854.766182] ? igrab+0x43/0x50 [32854.769232] ? lock_release+0xc5/0x260 [32854.772978] _raw_spin_lock+0x30/0x40 [32854.776644] ? igrab+0x1d/0x50 [32854.779691] igrab+0x1d/0x50 [32854.782571] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32854.787720] delayed_work+0x1d8/0x280 [ceph] [32854.792002] ? process_scheduled_works+0x22a/0x510 [32854.796793] process_scheduled_works+0x2a4/0x510 [32854.801415] worker_thread+0x188/0x340 [32854.805165] ? rescuer_thread+0x310/0x310 [32854.809169] kthread+0xf5/0x130 [32854.812314] ? kthread_complete_and_exit+0x20/0x20 [32854.817115] ret_from_fork+0x31/0x40 [32854.820695] ? kthread_complete_and_exit+0x20/0x20 [32854.825480] ret_from_fork_asm+0x11/0x20 [32854.829411] [32882.035191] watchdog: BUG: soft lockup - CPU#0 stuck for 19509s! [dbench:78683] [32882.042493] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32882.047191] watchdog: BUG: soft lockup - CPU#7 stuck for 28506s! [kworker/7:4:78808] [32882.097936] irq event stamp: 56369276 [32882.097937] hardirqs last enabled at (56369275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32882.105666] Modules linked in: [32882.109324] hardirqs last disabled at (56369276): [] sysvec_apic_timer_interrupt+0xe/0x90 [32882.119481] ceph [32882.122531] softirqs last enabled at (56369250): [] __do_softirq+0x2be/0x371 [32882.132255] libceph [32882.134180] softirqs last disabled at (56369245): [] irq_exit_rcu+0xb0/0xd0 [32882.142865] dns_resolver fscache [32882.145049] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32882.153558] netfs [32882.156881] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32882.165822] veth nft_chain_nat [32882.167834] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32882.175303] xt_MASQUERADE [32882.178441] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32882.183918] nf_nat [32882.186622] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32882.205361] nf_conntrack [32882.207458] [32882.207458] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32882.212674] nf_defrag_ipv6 [32882.215293] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32882.216782] nf_defrag_ipv4 [32882.223908] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32882.226699] nft_compat nf_tables [32882.233821] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32882.236613] nfnetlink [32882.243738] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32882.247048] bridge [32882.254172] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32882.256529] stp llc [32882.263654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32882.265751] xfs libcrc32c [32882.273829] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32882.276013] sunrpc intel_rapl_msr [32882.281751] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32882.284454] intel_rapl_common sb_edac [32882.291590] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32882.291591] Call Trace: [32882.294984] x86_pkg_temp_thermal [32882.302109] [32882.305867] coretemp [32882.312994] ? watchdog_timer_fn+0x1b4/0x220 [32882.315437] kvm_intel kvm [32882.318751] ? __hrtimer_run_queues+0x1d3/0x360 [32882.320759] irqbypass i2c_i801 [32882.323032] ? hrtimer_interrupt+0x104/0x240 [32882.327293] iTCO_wdt iTCO_vendor_support [32882.329997] ? __local_bh_enable+0x50/0x60 [32882.334521] i2c_smbus [32882.337663] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32882.341923] crct10dif_pclmul joydev [32882.345927] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32882.350018] crc32_pclmul ipmi_ssif [32882.352376] [32882.357594] ghash_clmulni_intel [32882.361164] [32882.366120] mei_me mei [32882.369606] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32882.371704] lpc_ich sha512_ssse3 [32882.374930] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32882.377025] acpi_ipmi mfd_core [32882.379469] do_raw_spin_lock+0xb5/0xc0 [32882.384771] wmi ipmi_si [32882.388086] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32882.392953] ipmi_devintf [32882.396094] try_get_cap_refs+0x159/0x880 [ceph] [32882.399923] ipmi_msghandler ioatdma [32882.402456] __ceph_get_caps+0x4a1/0x760 [ceph] [32882.407237] acpi_pad acpi_power_meter [32882.409860] ? __file_remove_privs+0x89/0x110 [32882.414483] ixgbe [32882.418054] ? wake_bit_function+0x70/0x70 [32882.422590] igb [32882.426340] ? ceph_write_iter+0x260/0x770 [ceph] [32882.430688] mdio crc32c_intel [32882.432701] ceph_write_iter+0x260/0x770 [ceph] [32882.436790] nvme ptp [32882.438631] ? lock_is_held_type+0xaa/0x120 [32882.443324] nvme_core [32882.446378] vfs_write+0x324/0x4f0 [32882.450899] i2c_algo_bit pps_core [32882.453175] ksys_pwrite64+0x65/0xa0 [32882.457348] dca [32882.459707] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32882.463103] fuse [32882.466501] do_syscall_64+0x38/0x80 [32882.470072] irq event stamp: 184174022 [32882.471909] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32882.477040] hardirqs last enabled at (184174021): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32882.478964] RIP: 0033:0x7f48f0eeb93b [32882.482534] hardirqs last disabled at (184174022): [] sysvec_apic_timer_interrupt+0xe/0x90 [32882.486278] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32882.491322] softirqs last enabled at (184174020): [] __do_softirq+0x2be/0x371 [32882.501566] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32882.505154] softirqs last disabled at (184174015): [] irq_exit_rcu+0xb0/0xd0 [32882.514965] ORIG_RAX: 0000000000000012 [32882.533704] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32882.542474] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32882.547692] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32882.556288] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32882.560121] Workqueue: events delayed_work [ceph] [32882.569497] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32882.576620] [32882.584091] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32882.591217] RIP: 0010:lock_acquire+0xf0/0x2b0 [32882.595914] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32882.603039] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32882.604534] [32882.611652] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32882.649285] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32882.656408] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32882.663533] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32882.670657] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32882.677782] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32882.684905] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32882.692983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32882.698719] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32882.705843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32882.712968] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32882.720091] Call Trace: [32882.722537] [32882.724548] ? watchdog_timer_fn+0x1b4/0x220 [32882.728812] ? __hrtimer_run_queues+0x1d3/0x360 [32882.733338] ? hrtimer_interrupt+0x104/0x240 [32882.737608] ? __local_bh_enable+0x50/0x60 [32882.741701] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32882.746917] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32882.751890] [32882.753989] [32882.756085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32882.761393] ? lock_acquire+0xf0/0x2b0 [32882.765145] ? igrab+0x43/0x50 [32882.768193] ? lock_release+0xc5/0x260 [32882.771940] _raw_spin_lock+0x30/0x40 [32882.775603] ? igrab+0x1d/0x50 [32882.778655] igrab+0x1d/0x50 [32882.781532] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32882.786683] delayed_work+0x1d8/0x280 [ceph] [32882.790965] ? process_scheduled_works+0x22a/0x510 [32882.795753] process_scheduled_works+0x2a4/0x510 [32882.800368] worker_thread+0x188/0x340 [32882.804119] ? rescuer_thread+0x310/0x310 [32882.808131] kthread+0xf5/0x130 [32882.811269] ? kthread_complete_and_exit+0x20/0x20 [32882.816052] ret_from_fork+0x31/0x40 [32882.819631] ? kthread_complete_and_exit+0x20/0x20 [32882.824415] ret_from_fork_asm+0x11/0x20 [32882.828339] [32910.035675] watchdog: BUG: soft lockup - CPU#0 stuck for 19535s! [dbench:78683] [32910.042973] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32910.047675] watchdog: BUG: soft lockup - CPU#7 stuck for 28532s! [kworker/7:4:78808] [32910.098414] irq event stamp: 56427494 [32910.098415] hardirqs last enabled at (56427493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32910.106144] Modules linked in: [32910.109803] hardirqs last disabled at (56427494): [] sysvec_apic_timer_interrupt+0xe/0x90 [32910.119961] ceph [32910.123011] softirqs last enabled at (56427468): [] __do_softirq+0x2be/0x371 [32910.132735] libceph [32910.134659] softirqs last disabled at (56427463): [] irq_exit_rcu+0xb0/0xd0 [32910.143359] dns_resolver [32910.145543] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32910.154070] fscache netfs [32910.156689] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32910.156690] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32910.165634] veth nft_chain_nat [32910.168337] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32910.175825] xt_MASQUERADE nf_nat [32910.181304] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32910.184440] nf_conntrack [32910.203180] [32910.203181] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32910.206489] nf_defrag_ipv6 nf_defrag_ipv4 [32910.211706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32910.214325] nft_compat nf_tables [32910.215813] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32910.222940] nfnetlink [32910.227029] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32910.234154] bridge stp [32910.237465] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32910.244588] llc xfs [32910.246947] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32910.254070] libcrc32c sunrpc [32910.256515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32910.263638] intel_rapl_msr intel_rapl_common [32910.265821] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32910.273899] sb_edac x86_pkg_temp_thermal [32910.276863] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32910.282601] coretemp [32910.286953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32910.294076] kvm_intel kvm [32910.298080] Call Trace: [32910.298082] [32910.305205] irqbypass i2c_i801 [32910.307475] ? watchdog_timer_fn+0x1b4/0x220 [32910.314599] iTCO_wdt [32910.317303] ? __hrtimer_run_queues+0x1d3/0x360 [32910.319748] iTCO_vendor_support [32910.321762] ? hrtimer_interrupt+0x104/0x240 [32910.324896] i2c_smbus crct10dif_pclmul [32910.329159] ? __local_bh_enable+0x50/0x60 [32910.331429] joydev crc32_pclmul [32910.335958] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32910.339178] ipmi_ssif ghash_clmulni_intel [32910.343443] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32910.347272] mei_me [32910.351364] [32910.354588] mei lpc_ich [32910.359806] [32910.359807] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32910.363895] sha512_ssse3 acpi_ipmi [32910.368859] ? queued_spin_lock_slowpath+0x16/0x2e0 [32910.370951] mfd_core [32910.373051] do_raw_spin_lock+0xb5/0xc0 [32910.375597] wmi ipmi_si [32910.377695] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32910.382999] ipmi_devintf ipmi_msghandler [32910.386483] try_get_cap_refs+0x159/0x880 [ceph] [32910.391355] ioatdma acpi_pad [32910.393629] __ceph_get_caps+0x4a1/0x760 [ceph] [32910.397453] acpi_power_meter ixgbe [32910.399990] ? __file_remove_privs+0x89/0x110 [32910.404769] igb mdio [32910.408773] ? wake_bit_function+0x70/0x70 [32910.413384] crc32c_intel [32910.416359] ? ceph_write_iter+0x260/0x770 [ceph] [32910.420881] nvme ptp [32910.424365] ceph_write_iter+0x260/0x770 [ceph] [32910.428715] nvme_core [32910.430990] ? lock_is_held_type+0xaa/0x120 [32910.435076] i2c_algo_bit [32910.437697] vfs_write+0x324/0x4f0 [32910.442392] pps_core dca [32910.444667] ksys_pwrite64+0x65/0xa0 [32910.449186] fuse [32910.451543] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32910.455721] irq event stamp: 184394952 [32910.458339] do_syscall_64+0x38/0x80 [32910.461753] hardirqs last enabled at (184394951): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32910.464370] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32910.467941] hardirqs last disabled at (184394952): [] sysvec_apic_timer_interrupt+0xe/0x90 [32910.469866] RIP: 0033:0x7f48f0eeb93b [32910.474995] softirqs last enabled at (184394950): [] __do_softirq+0x2be/0x371 [32910.478741] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32910.482310] softirqs last disabled at (184394945): [] irq_exit_rcu+0xb0/0xd0 [32910.492556] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32910.497600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32910.507410] ORIG_RAX: 0000000000000012 [32910.510981] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32910.510982] Workqueue: events delayed_work [ceph] [32910.519752] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32910.538490] [32910.538491] RIP: 0010:lock_acquire+0xf0/0x2b0 [32910.547088] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32910.552305] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32910.561681] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32910.565513] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32910.572982] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32910.577680] [32910.577681] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32910.584804] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32910.586295] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32910.590652] [32910.597771] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32910.668144] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32910.675269] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32910.682394] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32910.690470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32910.696208] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32910.703333] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32910.710458] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32910.717580] Call Trace: [32910.720025] [32910.722037] ? watchdog_timer_fn+0x1b4/0x220 [32910.726300] ? __hrtimer_run_queues+0x1d3/0x360 [32910.730827] ? hrtimer_interrupt+0x104/0x240 [32910.735097] ? __local_bh_enable+0x50/0x60 [32910.739190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32910.744413] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32910.749371] [32910.751469] [32910.753567] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32910.758873] ? lock_acquire+0xf0/0x2b0 [32910.762623] ? igrab+0x43/0x50 [32910.765674] ? lock_release+0xc5/0x260 [32910.769418] _raw_spin_lock+0x30/0x40 [32910.773081] ? igrab+0x1d/0x50 [32910.776134] igrab+0x1d/0x50 [32910.779013] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32910.784162] delayed_work+0x1d8/0x280 [ceph] [32910.788442] ? process_scheduled_works+0x22a/0x510 [32910.793226] process_scheduled_works+0x2a4/0x510 [32910.797841] worker_thread+0x188/0x340 [32910.801588] ? rescuer_thread+0x310/0x310 [32910.805592] kthread+0xf5/0x130 [32910.808729] ? kthread_complete_and_exit+0x20/0x20 [32910.813514] ret_from_fork+0x31/0x40 [32910.817086] ? kthread_complete_and_exit+0x20/0x20 [32910.821869] ret_from_fork_asm+0x11/0x20 [32910.825791] [32912.248080] rcu: INFO: rcu_preempt self-detected stall on CPU [32912.253838] rcu: 0-....: (20502263 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10261544 [32912.264349] rcu: (t=20975266 jiffies g=1309609 q=600727 ncpus=8) [32912.270436] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32912.279386] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32912.286858] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [32912.292344] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [32912.311081] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32912.316307] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32912.323432] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32912.330556] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32912.337679] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32912.344803] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32912.351927] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32912.360005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32912.365743] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32912.372866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32912.379992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32912.387116] Call Trace: [32912.389559] [32912.391574] ? rcu_dump_cpu_stacks+0xd1/0x180 [32912.395929] ? rcu_sched_clock_irq+0x633/0x1390 [32912.400455] ? lock_is_held_type+0xaa/0x120 [32912.404640] ? lock_is_held_type+0xaa/0x120 [32912.408818] ? tick_sched_do_timer+0xa0/0xa0 [32912.413089] ? rcu_read_lock_sched_held+0x56/0x90 [32912.417789] ? tick_sched_do_timer+0xa0/0xa0 [32912.422052] ? update_process_times+0x5f/0x90 [32912.426403] ? tick_sched_handle.isra.16+0x1e/0x50 [32912.431187] ? tick_sched_timer+0x6f/0x80 [32912.435191] ? __hrtimer_run_queues+0x1d3/0x360 [32912.439718] ? hrtimer_interrupt+0x104/0x240 [32912.443985] ? __local_bh_enable+0x50/0x60 [32912.448080] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32912.453305] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32912.458261] [32912.460357] [32912.462455] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32912.467764] ? queued_spin_lock_slowpath+0x18/0x2e0 [32912.472642] do_raw_spin_lock+0xb5/0xc0 [32912.476481] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32912.481292] try_get_cap_refs+0x159/0x880 [ceph] [32912.485923] __ceph_get_caps+0x4a1/0x760 [ceph] [32912.490464] ? __file_remove_privs+0x89/0x110 [32912.494818] ? wake_bit_function+0x70/0x70 [32912.498912] ? ceph_write_iter+0x260/0x770 [ceph] [32912.503622] ceph_write_iter+0x260/0x770 [ceph] [32912.508159] ? lock_is_held_type+0xaa/0x120 [32912.512346] vfs_write+0x324/0x4f0 [32912.515753] ksys_pwrite64+0x65/0xa0 [32912.519329] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32912.524459] do_syscall_64+0x38/0x80 [32912.528029] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32912.533075] RIP: 0033:0x7f48f0eeb93b [32912.536644] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32912.555381] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [32912.562939] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32912.570062] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32912.577186] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32912.584312] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32912.591435] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32912.598565] [32912.600751] Sending NMI from CPU 0 to CPUs 7: [32912.605105] NMI backtrace for cpu 7 [32912.605107] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32912.605109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32912.605110] Workqueue: events delayed_work [ceph] [32912.605127] RIP: 0010:__lock_acquire+0x4cb/0x1e20 [32912.605130] Code: 00 44 8b 9b 38 0a 00 00 45 85 db 74 18 ba 06 00 00 00 48 89 ee 48 89 df e8 a2 f4 ff ff 85 c0 0f 84 a9 02 00 00 ba 08 00 00 00 <48> 89 ee 48 89 df e8 8a f4 ff ff 85 c0 0f 84 91 02 00 00 44 8b 15 [32912.605131] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [32912.605132] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: 0000000000000000 [32912.605133] RDX: 0000000000000008 RSI: ffff88810debb550 RDI: ffff88810debaa80 [32912.605134] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [32912.605135] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [32912.605135] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [32912.605136] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32912.605137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32912.605138] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32912.605139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32912.605139] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32912.605140] Call Trace: [32912.605141] [32912.605142] ? nmi_cpu_backtrace+0xad/0x120 [32912.605145] ? nmi_cpu_backtrace_handler+0x11/0x20 [32912.605146] ? nmi_handle+0xd4/0x260 [32912.605150] ? default_do_nmi+0x49/0xf0 [32912.605152] ? exc_nmi+0x11e/0x150 [32912.605153] ? end_repeat_nmi+0x16/0x67 [32912.605157] ? __lock_acquire+0x4cb/0x1e20 [32912.605158] ? __lock_acquire+0x4cb/0x1e20 [32912.605160] ? __lock_acquire+0x4cb/0x1e20 [32912.605161] [32912.605162] [32912.605164] lock_acquire+0xcb/0x2b0 [32912.605165] ? igrab+0x1d/0x50 [32912.605167] ? igrab+0x43/0x50 [32912.605168] ? lock_release+0xc5/0x260 [32912.605170] _raw_spin_lock+0x30/0x40 [32912.605172] ? igrab+0x1d/0x50 [32912.605173] igrab+0x1d/0x50 [32912.605174] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32912.605189] delayed_work+0x1d8/0x280 [ceph] [32912.605202] ? process_scheduled_works+0x22a/0x510 [32912.605205] process_scheduled_works+0x2a4/0x510 [32912.605209] worker_thread+0x188/0x340 [32912.605211] ? rescuer_thread+0x310/0x310 [32912.605214] kthread+0xf5/0x130 [32912.605217] ? kthread_complete_and_exit+0x20/0x20 [32912.605219] ret_from_fork+0x31/0x40 [32912.605222] ? kthread_complete_and_exit+0x20/0x20 [32912.605224] ret_from_fork_asm+0x11/0x20 [32912.605228] [32938.036158] watchdog: BUG: soft lockup - CPU#0 stuck for 19561s! [dbench:78683] [32938.043460] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32938.048159] watchdog: BUG: soft lockup - CPU#7 stuck for 28558s! [kworker/7:4:78808] [32938.098909] irq event stamp: 56484952 [32938.106640] Modules linked in: ceph [32938.110299] hardirqs last enabled at (56484951): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32938.113783] libceph dns_resolver [32938.123941] hardirqs last disabled at (56484952): [] sysvec_apic_timer_interrupt+0xe/0x90 [32938.127251] fscache netfs veth nft_chain_nat [32938.136975] softirqs last enabled at (56484892): [] __do_softirq+0x2be/0x371 [32938.141326] xt_MASQUERADE nf_nat nf_conntrack [32938.150010] softirqs last disabled at (56484887): [] irq_exit_rcu+0xb0/0xd0 [32938.150012] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [32938.162959] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32938.168870] nfnetlink bridge [32938.177813] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32938.177814] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [32938.180776] stp llc [32938.188247] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [32938.193726] xfs libcrc32c [32938.195910] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32938.214649] sunrpc [32938.217350] [32938.217351] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32938.222567] intel_rapl_msr [32938.224665] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32938.226156] intel_rapl_common sb_edac [32938.233281] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32938.236071] x86_pkg_temp_thermal [32938.243197] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32938.246940] coretemp kvm_intel [32938.254063] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32938.257374] kvm irqbypass [32938.264498] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32938.267635] i2c_i801 iTCO_wdt [32938.274759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32938.277463] iTCO_vendor_support i2c_smbus [32938.285556] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32938.288610] crct10dif_pclmul joydev [32938.294347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32938.298438] crc32_pclmul ipmi_ssif [32938.305563] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32938.309133] ghash_clmulni_intel mei_me [32938.316256] Call Trace: [32938.319742] mei lpc_ich [32938.326866] [32938.330696] sha512_ssse3 acpi_ipmi [32938.333141] ? watchdog_timer_fn+0x1b4/0x220 [32938.335669] mfd_core wmi [32938.337681] ? __hrtimer_run_queues+0x1d3/0x360 [32938.341165] ipmi_si ipmi_devintf [32938.345432] ? hrtimer_interrupt+0x104/0x240 [32938.348047] ipmi_msghandler ioatdma [32938.352570] ? __local_bh_enable+0x50/0x60 [32938.355883] acpi_pad [32938.360149] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32938.363716] acpi_power_meter ixgbe [32938.367807] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32938.370079] igb mdio [32938.375296] [32938.378778] crc32c_intel nvme [32938.383736] [32938.383737] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32938.386007] ptp nvme_core [32938.388110] ? queued_spin_lock_slowpath+0x16/0x2e0 [32938.391155] i2c_algo_bit pps_core [32938.393256] do_raw_spin_lock+0xb5/0xc0 [32938.398558] dca fuse [32938.401263] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32938.406132] [32938.409534] try_get_cap_refs+0x159/0x880 [ceph] [32938.413359] irq event stamp: 184613830 [32938.415635] __ceph_get_caps+0x4a1/0x760 [ceph] [32938.420414] hardirqs last enabled at (184613829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32938.421909] ? __file_remove_privs+0x89/0x110 [32938.426516] hardirqs last disabled at (184613830): [] sysvec_apic_timer_interrupt+0xe/0x90 [32938.430260] ? wake_bit_function+0x70/0x70 [32938.434784] softirqs last enabled at (184613828): [] __do_softirq+0x2be/0x371 [32938.445030] ? ceph_write_iter+0x260/0x770 [ceph] [32938.449380] softirqs last disabled at (184613823): [] irq_exit_rcu+0xb0/0xd0 [32938.459192] ceph_write_iter+0x260/0x770 [ceph] [32938.463280] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32938.472055] ? lock_is_held_type+0xaa/0x120 [32938.476749] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32938.476750] Workqueue: events delayed_work [ceph] [32938.485350] vfs_write+0x324/0x4f0 [32938.489871] [32938.489872] RIP: 0010:lock_acquire+0xf0/0x2b0 [32938.499253] ksys_pwrite64+0x65/0xa0 [32938.503426] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [32938.510896] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32938.515594] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [32938.518992] do_syscall_64+0x38/0x80 [32938.520482] [32938.524842] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32938.528411] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [32938.547150] RIP: 0033:0x7f48f0eeb93b [32938.552281] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [32938.557499] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32938.561069] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [32938.562561] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32938.567603] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [32938.574727] ORIG_RAX: 0000000000000012 [32938.578299] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [32938.585423] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32938.604161] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32938.611284] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32938.616502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32938.623627] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32938.627456] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32938.634582] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32938.641704] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32938.649782] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32938.656907] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32938.662649] [32938.669767] Call Trace: [32938.669768] [32938.712028] ? watchdog_timer_fn+0x1b4/0x220 [32938.716293] ? __hrtimer_run_queues+0x1d3/0x360 [32938.720821] ? hrtimer_interrupt+0x104/0x240 [32938.725105] ? __local_bh_enable+0x50/0x60 [32938.729199] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32938.734424] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32938.739380] [32938.741476] [32938.743576] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32938.748883] ? lock_acquire+0xf0/0x2b0 [32938.752634] ? igrab+0x43/0x50 [32938.755682] ? lock_release+0xc5/0x260 [32938.759430] _raw_spin_lock+0x30/0x40 [32938.763094] ? igrab+0x1d/0x50 [32938.766144] igrab+0x1d/0x50 [32938.769023] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32938.774173] delayed_work+0x1d8/0x280 [ceph] [32938.778453] ? process_scheduled_works+0x22a/0x510 [32938.783242] process_scheduled_works+0x2a4/0x510 [32938.787860] worker_thread+0x188/0x340 [32938.791609] ? rescuer_thread+0x310/0x310 [32938.795619] kthread+0xf5/0x130 [32938.798757] ? kthread_complete_and_exit+0x20/0x20 [32938.803543] ret_from_fork+0x31/0x40 [32938.807139] ? kthread_complete_and_exit+0x20/0x20 [32938.811923] ret_from_fork_asm+0x11/0x20 [32938.815860] [32966.036642] watchdog: BUG: soft lockup - CPU#0 stuck for 19588s! [dbench:78683] [32966.043939] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32966.048642] watchdog: BUG: soft lockup - CPU#7 stuck for 28585s! [kworker/7:4:78808] [32966.099380] irq event stamp: 56543020 [32966.099382] hardirqs last enabled at (56543019): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32966.107111] Modules linked in: ceph [32966.110769] hardirqs last disabled at (56543020): [] sysvec_apic_timer_interrupt+0xe/0x90 [32966.120927] libceph [32966.124410] softirqs last enabled at (56542984): [] __do_softirq+0x2be/0x371 [32966.134135] dns_resolver fscache [32966.136320] softirqs last disabled at (56542979): [] irq_exit_rcu+0xb0/0xd0 [32966.145003] netfs veth [32966.148330] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32966.156841] nft_chain_nat [32966.159286] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32966.168231] xt_MASQUERADE nf_nat [32966.170935] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32966.178405] nf_conntrack nf_defrag_ipv6 [32966.181714] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32966.187193] nf_defrag_ipv4 nft_compat [32966.191111] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32966.209848] nf_tables [32966.213591] [32966.213592] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32966.218810] nfnetlink bridge [32966.221166] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32966.222656] stp llc [32966.229781] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32966.232746] xfs libcrc32c [32966.239870] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32966.242053] sunrpc intel_rapl_msr [32966.249178] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32966.251882] intel_rapl_common sb_edac [32966.259005] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32966.262404] x86_pkg_temp_thermal coretemp [32966.269529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32966.273273] kvm_intel kvm [32966.281350] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32966.285440] irqbypass i2c_i801 [32966.291177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32966.293881] iTCO_wdt iTCO_vendor_support [32966.301004] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32966.304143] i2c_smbus crct10dif_pclmul [32966.311266] Call Trace: [32966.311268] [32966.315270] joydev crc32_pclmul [32966.322397] ? watchdog_timer_fn+0x1b4/0x220 [32966.326225] ipmi_ssif ghash_clmulni_intel [32966.328671] ? __hrtimer_run_queues+0x1d3/0x360 [32966.330680] mei_me mei [32966.333909] ? hrtimer_interrupt+0x104/0x240 [32966.338168] lpc_ich sha512_ssse3 [32966.342260] ? __local_bh_enable+0x50/0x60 [32966.346784] acpi_ipmi mfd_core [32966.349229] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32966.353492] wmi ipmi_si [32966.356803] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32966.360894] ipmi_devintf ipmi_msghandler [32966.364041] [32966.369256] ioatdma acpi_pad [32966.371787] [32966.371788] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32966.376746] acpi_power_meter ixgbe [32966.380753] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32966.382845] igb mdio [32966.385814] do_raw_spin_lock+0xb5/0xc0 [32966.387907] crc32c_intel nvme ptp [32966.393213] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32966.396696] nvme_core i2c_algo_bit [32966.401569] try_get_cap_refs+0x159/0x880 [ceph] [32966.403838] pps_core dca [32966.407674] __ceph_get_caps+0x4a1/0x760 [ceph] [32966.411065] fuse [32966.415853] ? __file_remove_privs+0x89/0x110 [32966.419334] irq event stamp: 184834456 [32966.423944] ? wake_bit_function+0x70/0x70 [32966.426563] hardirqs last enabled at (184834455): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32966.431088] ? ceph_write_iter+0x260/0x770 [ceph] [32966.433009] hardirqs last disabled at (184834456): [] sysvec_apic_timer_interrupt+0xe/0x90 [32966.437360] ceph_write_iter+0x260/0x770 [ceph] [32966.441105] softirqs last enabled at (184834454): [] __do_softirq+0x2be/0x371 [32966.445198] ? lock_is_held_type+0xaa/0x120 [32966.455440] softirqs last disabled at (184834449): [] irq_exit_rcu+0xb0/0xd0 [32966.460141] vfs_write+0x324/0x4f0 [32966.469947] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32966.474475] ksys_pwrite64+0x65/0xa0 [32966.483242] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32966.487422] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32966.496017] Workqueue: events delayed_work [ceph] [32966.499433] do_syscall_64+0x38/0x80 [32966.508811] [32966.508812] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [32966.512382] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32966.519853] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [32966.524983] RIP: 0033:0x7f48f0eeb93b [32966.529680] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [32966.533251] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32966.534741] [32966.534742] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [32966.540739] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32966.545783] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [32966.564519] ORIG_RAX: 0000000000000012 [32966.568091] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [32966.573310] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32966.592047] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32966.593536] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32966.600660] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32966.605878] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32966.613002] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32966.616832] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32966.623957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32966.631081] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32966.638205] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32966.645334] [32966.652454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32966.704074] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32966.711197] Call Trace: [32966.713645] [32966.715655] ? watchdog_timer_fn+0x1b4/0x220 [32966.719918] ? __hrtimer_run_queues+0x1d3/0x360 [32966.724444] ? hrtimer_interrupt+0x104/0x240 [32966.728713] ? __local_bh_enable+0x50/0x60 [32966.732807] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32966.738047] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32966.743005] [32966.745104] [32966.747201] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32966.752510] ? __list_del_entry_valid_or_report+0x10/0xd0 [32966.757904] ceph_check_delayed_caps+0x8f/0x270 [ceph] [32966.763055] delayed_work+0x1d8/0x280 [ceph] [32966.767339] ? process_scheduled_works+0x22a/0x510 [32966.772125] process_scheduled_works+0x2a4/0x510 [32966.776740] worker_thread+0x188/0x340 [32966.780489] ? rescuer_thread+0x310/0x310 [32966.784494] kthread+0xf5/0x130 [32966.787632] ? kthread_complete_and_exit+0x20/0x20 [32966.792415] ret_from_fork+0x31/0x40 [32966.795985] ? kthread_complete_and_exit+0x20/0x20 [32966.800771] ret_from_fork_asm+0x11/0x20 [32966.804693] [32994.037126] watchdog: BUG: soft lockup - CPU#0 stuck for 19614s! [dbench:78683] [32994.044427] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [32994.049125] watchdog: BUG: soft lockup - CPU#7 stuck for 28611s! [kworker/7:4:78808] [32994.099877] irq event stamp: 56601170 [32994.099878] hardirqs last enabled at (56601169): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32994.107608] Modules linked in: [32994.111264] hardirqs last disabled at (56601170): [] sysvec_apic_timer_interrupt+0xe/0x90 [32994.121421] ceph [32994.124473] softirqs last enabled at (56601112): [] __do_softirq+0x2be/0x371 [32994.134197] libceph [32994.136122] softirqs last disabled at (56601107): [] irq_exit_rcu+0xb0/0xd0 [32994.144807] dns_resolver fscache [32994.146990] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32994.155517] netfs veth [32994.158828] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32994.167772] nft_chain_nat xt_MASQUERADE [32994.170217] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [32994.177687] nf_nat [32994.181607] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [32994.187081] nf_conntrack nf_defrag_ipv6 [32994.189181] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [32994.207918] nf_defrag_ipv4 [32994.211834] [32994.211835] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [32994.217053] nft_compat nf_tables [32994.219844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [32994.221333] nfnetlink bridge [32994.228458] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [32994.231769] stp llc [32994.238892] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [32994.241857] xfs [32994.248981] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [32994.251164] libcrc32c sunrpc [32994.258289] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [32994.260125] intel_rapl_msr [32994.267251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32994.270214] intel_rapl_common sb_edac [32994.278292] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [32994.281083] x86_pkg_temp_thermal [32994.286820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32994.290565] coretemp kvm_intel [32994.297690] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32994.301000] kvm irqbypass [32994.308125] Call Trace: [32994.311260] i2c_i801 iTCO_wdt [32994.318387] [32994.321090] iTCO_vendor_support i2c_smbus [32994.323535] ? watchdog_timer_fn+0x1b4/0x220 [32994.326584] crct10dif_pclmul joydev [32994.328596] ? __hrtimer_run_queues+0x1d3/0x360 [32994.332685] crc32_pclmul ipmi_ssif [32994.336953] ? hrtimer_interrupt+0x104/0x240 [32994.340525] ghash_clmulni_intel mei_me [32994.345054] ? __local_bh_enable+0x50/0x60 [32994.348537] mei lpc_ich [32994.352807] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32994.356630] sha512_ssse3 acpi_ipmi [32994.360723] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32994.363254] mfd_core wmi [32994.368471] [32994.371955] ipmi_si ipmi_devintf [32994.376911] [32994.379529] ipmi_msghandler ioatdma [32994.381628] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32994.384936] acpi_pad acpi_power_meter [32994.387040] ? queued_spin_lock_slowpath+0x2d/0x2e0 [32994.390605] ixgbe igb mdio [32994.395912] do_raw_spin_lock+0xb5/0xc0 [32994.399652] crc32c_intel nvme [32994.404527] __cap_delay_requeue+0xa7/0x1d0 [ceph] [32994.407315] ptp [32994.411148] try_get_cap_refs+0x159/0x880 [ceph] [32994.414197] nvme_core i2c_algo_bit [32994.418985] __ceph_get_caps+0x4a1/0x760 [ceph] [32994.420818] pps_core dca [32994.425434] ? __file_remove_privs+0x89/0x110 [32994.428913] fuse [32994.433438] ? wake_bit_function+0x70/0x70 [32994.436055] irq event stamp: 185055360 [32994.440409] ? ceph_write_iter+0x260/0x770 [ceph] [32994.442328] hardirqs last enabled at (185055359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32994.446420] ceph_write_iter+0x260/0x770 [ceph] [32994.450165] hardirqs last disabled at (185055360): [] sysvec_apic_timer_interrupt+0xe/0x90 [32994.454867] ? lock_is_held_type+0xaa/0x120 [32994.465105] softirqs last enabled at (185055358): [] __do_softirq+0x2be/0x371 [32994.469632] vfs_write+0x324/0x4f0 [32994.479441] softirqs last disabled at (185055353): [] irq_exit_rcu+0xb0/0xd0 [32994.483623] ksys_pwrite64+0x65/0xa0 [32994.492389] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [32994.495803] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [32994.504400] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [32994.507974] do_syscall_64+0x38/0x80 [32994.517351] Workqueue: events delayed_work [ceph] [32994.522481] entry_SYSCALL_64_after_hwframe+0x63/0xcd [32994.529952] [32994.533527] RIP: 0033:0x7f48f0eeb93b [32994.538219] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [32994.543263] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [32994.544754] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [32994.548326] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [32994.552934] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [32994.571673] ORIG_RAX: 0000000000000012 [32994.590410] [32994.595627] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [32994.600845] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [32994.604675] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [32994.606166] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [32994.613292] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [32994.620414] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [32994.627539] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [32994.634663] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [32994.641788] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [32994.648912] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [32994.656041] [32994.663160] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [32994.687669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32994.693408] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [32994.700532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32994.707657] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [32994.714781] Call Trace: [32994.717224] [32994.719237] ? watchdog_timer_fn+0x1b4/0x220 [32994.723499] ? __hrtimer_run_queues+0x1d3/0x360 [32994.728027] ? hrtimer_interrupt+0x104/0x240 [32994.732296] ? __local_bh_enable+0x50/0x60 [32994.736388] ? __sysvec_apic_timer_interrupt+0x61/0x160 [32994.741612] ? sysvec_apic_timer_interrupt+0x6d/0x90 [32994.746569] [32994.748667] [32994.750766] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [32994.756072] ? do_raw_spin_lock+0x42/0xc0 [32994.760082] igrab+0x1d/0x50 [32994.762961] ceph_check_delayed_caps+0xb7/0x270 [ceph] [32994.768111] delayed_work+0x1d8/0x280 [ceph] [32994.772391] ? process_scheduled_works+0x22a/0x510 [32994.777181] process_scheduled_works+0x2a4/0x510 [32994.781812] worker_thread+0x188/0x340 [32994.785564] ? rescuer_thread+0x310/0x310 [32994.789567] kthread+0xf5/0x130 [32994.792713] ? kthread_complete_and_exit+0x20/0x20 [32994.797496] ret_from_fork+0x31/0x40 [32994.801078] ? kthread_complete_and_exit+0x20/0x20 [32994.805868] ret_from_fork_asm+0x11/0x20 [32994.809792] [33022.037605] watchdog: BUG: soft lockup - CPU#0 stuck for 19640s! [dbench:78683] [33022.044906] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33022.049605] watchdog: BUG: soft lockup - CPU#7 stuck for 28637s! [kworker/7:4:78808] [33022.100346] irq event stamp: 56659306 [33022.100347] hardirqs last enabled at (56659305): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33022.108078] Modules linked in: [33022.111735] hardirqs last disabled at (56659306): [] sysvec_apic_timer_interrupt+0xe/0x90 [33022.121892] ceph [33022.124943] softirqs last enabled at (56659260): [] __do_softirq+0x2be/0x371 [33022.134669] libceph [33022.136592] softirqs last disabled at (56659255): [] irq_exit_rcu+0xb0/0xd0 [33022.145281] dns_resolver fscache [33022.147468] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33022.155996] netfs veth [33022.159308] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33022.159309] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33022.168253] nft_chain_nat xt_MASQUERADE [33022.170696] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33022.178167] nf_nat nf_conntrack [33022.183644] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33022.187560] nf_defrag_ipv6 nf_defrag_ipv4 [33022.206299] [33022.206300] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33022.209522] nft_compat nf_tables [33022.214741] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33022.218832] nfnetlink [33022.220322] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33022.227445] bridge stp [33022.230757] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33022.237880] llc xfs [33022.240239] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33022.247362] libcrc32c sunrpc [33022.249807] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33022.256931] intel_rapl_msr [33022.259114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33022.266238] intel_rapl_common [33022.269203] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33022.277284] sb_edac [33022.280071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33022.285808] x86_pkg_temp_thermal coretemp [33022.288859] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33022.295999] kvm_intel [33022.298186] Call Trace: [33022.298187] [33022.305310] kvm irqbypass [33022.309400] ? watchdog_timer_fn+0x1b4/0x220 [33022.316524] i2c_i801 iTCO_wdt iTCO_vendor_support [33022.318883] ? __hrtimer_run_queues+0x1d3/0x360 [33022.321324] i2c_smbus crct10dif_pclmul [33022.323339] ? hrtimer_interrupt+0x104/0x240 [33022.326039] joydev crc32_pclmul [33022.330303] ? __local_bh_enable+0x50/0x60 [33022.335088] ipmi_ssif ghash_clmulni_intel [33022.339614] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33022.343444] mei_me mei [33022.347708] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33022.350932] lpc_ich sha512_ssse3 [33022.355023] [33022.359114] acpi_ipmi mfd_core [33022.364331] [33022.366775] wmi ipmi_si [33022.371731] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33022.375041] ipmi_devintf ipmi_msghandler [33022.377143] ? queued_spin_lock_slowpath+0x18/0x2e0 [33022.380291] ioatdma [33022.382393] do_raw_spin_lock+0xb5/0xc0 [33022.384922] acpi_pad acpi_power_meter [33022.390228] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33022.394229] ixgbe igb [33022.399104] try_get_cap_refs+0x159/0x880 [ceph] [33022.401292] mdio [33022.405130] __ceph_get_caps+0x4a1/0x760 [ceph] [33022.408868] crc32c_intel nvme [33022.413656] ? __file_remove_privs+0x89/0x110 [33022.416010] ptp nvme_core [33022.420623] ? wake_bit_function+0x70/0x70 [33022.422546] i2c_algo_bit pps_core [33022.427072] ? ceph_write_iter+0x260/0x770 [ceph] [33022.430119] dca fuse [33022.434470] ceph_write_iter+0x260/0x770 [ceph] [33022.437174] [33022.437175] irq event stamp: 185276366 [33022.441284] ? lock_is_held_type+0xaa/0x120 [33022.444679] hardirqs last enabled at (185276365): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33022.449380] vfs_write+0x324/0x4f0 [33022.451649] hardirqs last disabled at (185276366): [] sysvec_apic_timer_interrupt+0xe/0x90 [33022.456177] ksys_pwrite64+0x65/0xa0 [33022.457663] softirqs last enabled at (185276364): [] __do_softirq+0x2be/0x371 [33022.461406] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33022.465586] softirqs last disabled at (185276359): [] irq_exit_rcu+0xb0/0xd0 [33022.475830] do_syscall_64+0x38/0x80 [33022.479227] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33022.489037] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33022.492607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33022.501378] RIP: 0033:0x7f48f0eeb93b [33022.506509] Workqueue: events delayed_work [ceph] [33022.515106] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33022.518677] [33022.518678] RIP: 0010:lock_release+0x42/0x260 [33022.528055] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33022.533100] Code: 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 <8b> 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 [33022.540571] ORIG_RAX: 0000000000000012 [33022.544140] RSP: 0018:ffffc90002117d48 EFLAGS: 00000202 [33022.548838] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33022.567576] [33022.567577] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [33022.569067] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33022.573417] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [33022.578635] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33022.597372] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [33022.601203] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33022.606419] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [33022.613545] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33022.615035] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33022.622164] [33022.629285] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33022.689432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33022.695168] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33022.702293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33022.709416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33022.716542] Call Trace: [33022.718987] [33022.721014] ? watchdog_timer_fn+0x1b4/0x220 [33022.725295] ? __hrtimer_run_queues+0x1d3/0x360 [33022.729823] ? hrtimer_interrupt+0x104/0x240 [33022.734091] ? __local_bh_enable+0x50/0x60 [33022.738184] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33022.743400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33022.748356] [33022.750455] [33022.752553] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33022.757856] ? igrab+0x43/0x50 [33022.760910] ? igrab+0x43/0x50 [33022.763958] ? lock_release+0x42/0x260 [33022.767703] ? lock_release+0x1d7/0x260 [33022.771535] _raw_spin_unlock+0x1b/0x40 [33022.775372] igrab+0x43/0x50 [33022.778250] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33022.783401] delayed_work+0x1d8/0x280 [ceph] [33022.787681] ? process_scheduled_works+0x22a/0x510 [33022.792471] process_scheduled_works+0x2a4/0x510 [33022.797087] worker_thread+0x188/0x340 [33022.800836] ? rescuer_thread+0x310/0x310 [33022.804841] kthread+0xf5/0x130 [33022.807978] ? kthread_complete_and_exit+0x20/0x20 [33022.812763] ret_from_fork+0x31/0x40 [33022.816339] ? kthread_complete_and_exit+0x20/0x20 [33022.821125] ret_from_fork_asm+0x11/0x20 [33022.825049] [33050.038087] watchdog: BUG: soft lockup - CPU#0 stuck for 19666s! [dbench:78683] [33050.045394] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33050.050086] watchdog: BUG: soft lockup - CPU#7 stuck for 28663s! [kworker/7:4:78808] [33050.100842] irq event stamp: 56717454 [33050.100843] hardirqs last enabled at (56717453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33050.108575] Modules linked in: [33050.112231] hardirqs last disabled at (56717454): [] sysvec_apic_timer_interrupt+0xe/0x90 [33050.122390] ceph [33050.125441] softirqs last enabled at (56717396): [] __do_softirq+0x2be/0x371 [33050.135164] libceph [33050.137087] softirqs last disabled at (56717391): [] irq_exit_rcu+0xb0/0xd0 [33050.145771] dns_resolver [33050.147957] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33050.156482] fscache [33050.159100] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33050.168046] netfs veth [33050.170231] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33050.177700] nft_chain_nat xt_MASQUERADE [33050.180144] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33050.185621] nf_nat nf_conntrack [33050.189538] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33050.208276] nf_defrag_ipv6 nf_defrag_ipv4 [33050.211501] [33050.211502] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33050.216718] nft_compat nf_tables [33050.220809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33050.222300] nfnetlink bridge [33050.229423] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33050.232735] stp llc [33050.239859] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33050.242823] xfs libcrc32c [33050.249947] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33050.252133] sunrpc [33050.259255] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33050.261959] intel_rapl_msr intel_rapl_common [33050.269083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33050.271180] sb_edac x86_pkg_temp_thermal [33050.279257] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33050.283609] coretemp [33050.289347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33050.293350] kvm_intel [33050.300486] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33050.302763] kvm irqbypass [33050.309886] Call Trace: [33050.309889] [33050.312244] i2c_i801 iTCO_wdt [33050.319372] ? watchdog_timer_fn+0x1b4/0x220 [33050.322073] iTCO_vendor_support [33050.324517] ? __hrtimer_run_queues+0x1d3/0x360 [33050.326527] i2c_smbus crct10dif_pclmul [33050.329582] ? hrtimer_interrupt+0x104/0x240 [33050.333842] joydev crc32_pclmul [33050.337067] ? __local_bh_enable+0x50/0x60 [33050.341589] ipmi_ssif [33050.345424] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33050.349685] ghash_clmulni_intel [33050.352908] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33050.356999] mei_me mei [33050.359357] [33050.364575] lpc_ich sha512_ssse3 [33050.367798] [33050.372770] acpi_ipmi mfd_core [33050.375217] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33050.377316] wmi ipmi_si [33050.380631] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33050.382723] ipmi_devintf [33050.385863] do_raw_spin_lock+0xb5/0xc0 [33050.391163] ipmi_msghandler [33050.393697] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33050.398565] ioatdma acpi_pad [33050.401188] try_get_cap_refs+0x159/0x880 [ceph] [33050.405013] acpi_power_meter ixgbe [33050.407898] __ceph_get_caps+0x4a1/0x760 [ceph] [33050.412675] igb mdio [33050.415645] ? __file_remove_privs+0x89/0x110 [33050.420252] crc32c_intel [33050.423734] ? wake_bit_function+0x70/0x70 [33050.428258] nvme [33050.430533] ? ceph_write_iter+0x260/0x770 [ceph] [33050.434880] ptp [33050.437498] ceph_write_iter+0x260/0x770 [ceph] [33050.441588] nvme_core i2c_algo_bit [33050.443516] ? lock_is_held_type+0xaa/0x120 [33050.448209] pps_core [33050.450050] vfs_write+0x324/0x4f0 [33050.454570] dca fuse [33050.458058] ksys_pwrite64+0x65/0xa0 [33050.462232] [33050.462233] irq event stamp: 185497098 [33050.464504] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33050.467900] hardirqs last enabled at (185497097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33050.470171] do_syscall_64+0x38/0x80 [33050.473741] hardirqs last disabled at (185497098): [] sysvec_apic_timer_interrupt+0xe/0x90 [33050.475233] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33050.478976] softirqs last enabled at (185497096): [] __do_softirq+0x2be/0x371 [33050.484108] RIP: 0033:0x7f48f0eeb93b [33050.494352] softirqs last disabled at (185497091): [] irq_exit_rcu+0xb0/0xd0 [33050.497924] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33050.507733] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33050.512778] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33050.521547] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33050.525118] ORIG_RAX: 0000000000000012 [33050.533716] Workqueue: events delayed_work [ceph] [33050.552453] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33050.561833] [33050.567050] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33050.574519] RIP: 0010:do_raw_spin_unlock+0x16/0xc0 [33050.578349] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33050.583048] Code: 00 48 89 47 10 b8 01 00 00 00 c3 31 c0 c3 66 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 53 81 7f 04 ad 4e ad de 48 89 fb 75 34 <8b> 03 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 [33050.590172] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33050.591662] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [33050.598787] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33050.603572] [33050.603573] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [33050.610699] [33050.629433] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [33050.666822] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [33050.673946] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33050.681071] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33050.688194] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33050.696273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33050.702010] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33050.709133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33050.716257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33050.723382] Call Trace: [33050.725827] [33050.727838] ? watchdog_timer_fn+0x1b4/0x220 [33050.732102] ? __hrtimer_run_queues+0x1d3/0x360 [33050.736628] ? hrtimer_interrupt+0x104/0x240 [33050.740896] ? __local_bh_enable+0x50/0x60 [33050.744991] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33050.750214] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33050.755171] [33050.757270] [33050.759367] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33050.764674] ? do_raw_spin_unlock+0x16/0xc0 [33050.768858] _raw_spin_unlock+0x23/0x40 [33050.772698] igrab+0x43/0x50 [33050.775575] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33050.780725] delayed_work+0x1d8/0x280 [ceph] [33050.785006] ? process_scheduled_works+0x22a/0x510 [33050.789795] process_scheduled_works+0x2a4/0x510 [33050.794411] worker_thread+0x188/0x340 [33050.798160] ? rescuer_thread+0x310/0x310 [33050.802163] kthread+0xf5/0x130 [33050.805301] ? kthread_complete_and_exit+0x20/0x20 [33050.810087] ret_from_fork+0x31/0x40 [33050.813665] ? kthread_complete_and_exit+0x20/0x20 [33050.818449] ret_from_fork_asm+0x11/0x20 [33050.822381] [33078.038567] watchdog: BUG: soft lockup - CPU#0 stuck for 19692s! [dbench:78683] [33078.045873] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33078.050567] watchdog: BUG: soft lockup - CPU#7 stuck for 28689s! [kworker/7:4:78808] [33078.101314] irq event stamp: 56775612 [33078.101315] hardirqs last enabled at (56775611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33078.109044] Modules linked in: [33078.112703] hardirqs last disabled at (56775612): [] sysvec_apic_timer_interrupt+0xe/0x90 [33078.122858] ceph [33078.125909] softirqs last enabled at (56775554): [] __do_softirq+0x2be/0x371 [33078.135633] libceph [33078.137559] softirqs last disabled at (56775549): [] irq_exit_rcu+0xb0/0xd0 [33078.146250] dns_resolver fscache [33078.148434] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33078.156961] netfs [33078.160273] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33078.169218] veth nft_chain_nat [33078.171245] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [33078.178716] xt_MASQUERADE nf_nat [33078.181853] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [33078.187331] nf_conntrack [33078.190643] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33078.209380] nf_defrag_ipv6 nf_defrag_ipv4 [33078.211996] [33078.211997] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33078.217213] nft_compat [33078.221305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33078.222796] nf_tables nfnetlink [33078.229920] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33078.232365] bridge stp [33078.239489] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33078.242713] llc xfs [33078.249836] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33078.252279] libcrc32c [33078.259406] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33078.261589] sunrpc intel_rapl_msr [33078.268712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33078.271071] intel_rapl_common sb_edac [33078.279147] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33078.282545] x86_pkg_temp_thermal [33078.288283] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33078.292026] coretemp kvm_intel [33078.299152] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33078.302462] kvm irqbypass [33078.309587] Call Trace: [33078.309588] [33078.312722] i2c_i801 iTCO_wdt [33078.319847] ? watchdog_timer_fn+0x1b4/0x220 [33078.322552] iTCO_vendor_support [33078.324995] ? __hrtimer_run_queues+0x1d3/0x360 [33078.327006] i2c_smbus crct10dif_pclmul [33078.330060] ? hrtimer_interrupt+0x104/0x240 [33078.334321] joydev [33078.337545] ? __local_bh_enable+0x50/0x60 [33078.342068] crc32_pclmul ipmi_ssif [33078.345901] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33078.350164] ghash_clmulni_intel [33078.352262] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33078.356354] mei_me [33078.359837] [33078.365052] mei lpc_ich [33078.368277] [33078.373250] sha512_ssse3 [33078.375351] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33078.377448] acpi_ipmi mfd_core [33078.379983] ? queued_spin_lock_slowpath+0x2b/0x2e0 [33078.382076] wmi ipmi_si [33078.384693] do_raw_spin_lock+0xb5/0xc0 [33078.389995] ipmi_devintf ipmi_msghandler [33078.393136] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33078.398003] ioatdma acpi_pad [33078.400537] try_get_cap_refs+0x159/0x880 [ceph] [33078.404366] acpi_power_meter ixgbe [33078.408374] __ceph_get_caps+0x4a1/0x760 [ceph] [33078.413155] igb mdio [33078.416122] ? __file_remove_privs+0x89/0x110 [33078.420728] crc32c_intel nvme [33078.424215] ? wake_bit_function+0x70/0x70 [33078.428737] ptp nvme_core [33078.431010] ? ceph_write_iter+0x260/0x770 [ceph] [33078.435357] i2c_algo_bit pps_core [33078.438410] ceph_write_iter+0x260/0x770 [ceph] [33078.442501] dca fuse [33078.445207] ? lock_is_held_type+0xaa/0x120 [33078.449901] [33078.453301] vfs_write+0x324/0x4f0 [33078.457823] irq event stamp: 185717830 [33078.460099] ksys_pwrite64+0x65/0xa0 [33078.464271] hardirqs last enabled at (185717829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33078.465762] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33078.469160] hardirqs last disabled at (185717830): [] sysvec_apic_timer_interrupt+0xe/0x90 [33078.472906] do_syscall_64+0x38/0x80 [33078.476473] softirqs last enabled at (185717828): [] __do_softirq+0x2be/0x371 [33078.486719] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33078.491849] softirqs last disabled at (185717823): [] irq_exit_rcu+0xb0/0xd0 [33078.501659] RIP: 0033:0x7f48f0eeb93b [33078.505230] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33078.514002] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33078.519046] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33078.527643] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33078.531215] Workqueue: events delayed_work [ceph] [33078.540590] ORIG_RAX: 0000000000000012 [33078.559329] [33078.559330] RIP: 0010:lock_acquire+0xf0/0x2b0 [33078.566800] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33078.572018] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33078.576715] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33078.580546] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33078.582035] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33078.586386] [33078.586387] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33078.593511] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33078.612250] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33078.619372] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33078.624591] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33078.631720] [33078.633204] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33078.678134] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33078.685272] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33078.693354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33078.699091] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33078.706215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33078.713340] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33078.720464] Call Trace: [33078.722908] [33078.724919] ? watchdog_timer_fn+0x1b4/0x220 [33078.729183] ? __hrtimer_run_queues+0x1d3/0x360 [33078.733710] ? hrtimer_interrupt+0x104/0x240 [33078.737978] ? __local_bh_enable+0x50/0x60 [33078.742072] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33078.747295] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33078.752260] [33078.754358] [33078.756456] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33078.761764] ? lock_acquire+0xf0/0x2b0 [33078.765514] ? igrab+0x43/0x50 [33078.768564] ? lock_release+0xc5/0x260 [33078.772310] _raw_spin_lock+0x30/0x40 [33078.775973] ? igrab+0x1d/0x50 [33078.779026] igrab+0x1d/0x50 [33078.781902] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33078.787052] delayed_work+0x1d8/0x280 [ceph] [33078.791335] ? process_scheduled_works+0x22a/0x510 [33078.796126] process_scheduled_works+0x2a4/0x510 [33078.800740] worker_thread+0x188/0x340 [33078.804489] ? rescuer_thread+0x310/0x310 [33078.808493] kthread+0xf5/0x130 [33078.811629] ? kthread_complete_and_exit+0x20/0x20 [33078.816414] ret_from_fork+0x31/0x40 [33078.819993] ? kthread_complete_and_exit+0x20/0x20 [33078.824777] ret_from_fork_asm+0x11/0x20 [33078.828700] [33106.039048] watchdog: BUG: soft lockup - CPU#0 stuck for 19718s! [dbench:78683] [33106.046352] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33106.051047] watchdog: BUG: soft lockup - CPU#7 stuck for 28715s! [kworker/7:4:78808] [33106.101800] irq event stamp: 56833766 [33106.101801] hardirqs last enabled at (56833765): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33106.109532] Modules linked in: [33106.113188] hardirqs last disabled at (56833766): [] sysvec_apic_timer_interrupt+0xe/0x90 [33106.123346] ceph [33106.126396] softirqs last enabled at (56833708): [] __do_softirq+0x2be/0x371 [33106.136122] libceph [33106.138045] softirqs last disabled at (56833703): [] irq_exit_rcu+0xb0/0xd0 [33106.146730] dns_resolver [33106.148916] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33106.157441] fscache netfs [33106.160059] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33106.169004] veth [33106.171725] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33106.179195] nft_chain_nat xt_MASQUERADE [33106.181121] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33106.186596] nf_nat nf_conntrack [33106.190513] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33106.209251] nf_defrag_ipv6 nf_defrag_ipv4 [33106.212477] [33106.212477] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33106.217695] nft_compat nf_tables [33106.221785] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33106.223276] nfnetlink [33106.230400] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33106.233722] bridge [33106.240843] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33106.243200] stp [33106.250324] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33106.252421] llc [33106.259563] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33106.261400] xfs libcrc32c [33106.268525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33106.270362] sunrpc [33106.278443] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33106.281144] intel_rapl_msr intel_rapl_common [33106.286881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33106.288978] sb_edac x86_pkg_temp_thermal [33106.296102] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33106.300453] coretemp [33106.307578] Call Trace: [33106.311582] kvm_intel kvm [33106.318725] [33106.320992] irqbypass i2c_i801 [33106.323446] ? watchdog_timer_fn+0x1b4/0x220 [33106.326151] iTCO_wdt [33106.328162] ? __hrtimer_run_queues+0x1d3/0x360 [33106.331299] iTCO_vendor_support i2c_smbus [33106.335566] ? hrtimer_interrupt+0x104/0x240 [33106.337833] crct10dif_pclmul [33106.342356] ? __local_bh_enable+0x50/0x60 [33106.346448] joydev [33106.350715] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33106.353676] crc32_pclmul [33106.357767] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33106.359863] ipmi_ssif ghash_clmulni_intel [33106.365082] [33106.367700] mei_me [33106.372656] [33106.372658] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33106.376746] mei [33106.378849] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33106.380943] lpc_ich sha512_ssse3 [33106.383042] do_raw_spin_lock+0xb5/0xc0 [33106.388344] acpi_ipmi mfd_core [33106.390183] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33106.395052] wmi ipmi_si [33106.398365] try_get_cap_refs+0x159/0x880 [ceph] [33106.402194] ipmi_devintf ipmi_msghandler [33106.405337] __ceph_get_caps+0x4a1/0x760 [ceph] [33106.410115] ioatdma acpi_pad [33106.412649] ? __file_remove_privs+0x89/0x110 [33106.417257] acpi_power_meter ixgbe [33106.421261] ? wake_bit_function+0x70/0x70 [33106.425785] igb mdio [33106.428752] ? ceph_write_iter+0x260/0x770 [ceph] [33106.433100] crc32c_intel [33106.436582] ceph_write_iter+0x260/0x770 [ceph] [33106.440674] nvme ptp [33106.442949] ? lock_is_held_type+0xaa/0x120 [33106.447643] nvme_core [33106.450261] vfs_write+0x324/0x4f0 [33106.454784] i2c_algo_bit pps_core [33106.457058] ksys_pwrite64+0x65/0xa0 [33106.461230] dca [33106.463588] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33106.466986] fuse [33106.470386] do_syscall_64+0x38/0x80 [33106.473953] irq event stamp: 185938518 [33106.475792] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33106.480923] hardirqs last enabled at (185938517): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33106.482847] RIP: 0033:0x7f48f0eeb93b [33106.486418] hardirqs last disabled at (185938518): [] sysvec_apic_timer_interrupt+0xe/0x90 [33106.490162] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33106.495206] softirqs last enabled at (185938516): [] __do_softirq+0x2be/0x371 [33106.505451] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33106.509020] softirqs last disabled at (185938511): [] irq_exit_rcu+0xb0/0xd0 [33106.518831] ORIG_RAX: 0000000000000012 [33106.537569] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33106.546341] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33106.551558] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33106.560154] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33106.563985] Workqueue: events delayed_work [ceph] [33106.573362] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33106.580487] [33106.587957] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33106.595081] RIP: 0010:lock_acquire+0xf0/0x2b0 [33106.599780] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33106.606904] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33106.608400] [33106.615518] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33106.653151] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33106.660273] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33106.667397] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33106.674523] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33106.681647] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33106.688771] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33106.696849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33106.702586] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33106.709709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33106.716851] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33106.723975] Call Trace: [33106.726421] [33106.728430] ? watchdog_timer_fn+0x1b4/0x220 [33106.732714] ? __hrtimer_run_queues+0x1d3/0x360 [33106.737256] ? hrtimer_interrupt+0x104/0x240 [33106.741525] ? __local_bh_enable+0x50/0x60 [33106.745620] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33106.750842] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33106.755799] [33106.757898] [33106.759995] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33106.765303] ? lock_acquire+0xf0/0x2b0 [33106.769053] ? igrab+0x43/0x50 [33106.772104] ? lock_release+0xc5/0x260 [33106.775847] _raw_spin_lock+0x30/0x40 [33106.779506] ? igrab+0x1d/0x50 [33106.782554] igrab+0x1d/0x50 [33106.785452] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33106.790600] delayed_work+0x1d8/0x280 [ceph] [33106.794881] ? process_scheduled_works+0x22a/0x510 [33106.799670] process_scheduled_works+0x2a4/0x510 [33106.804288] worker_thread+0x188/0x340 [33106.808037] ? rescuer_thread+0x310/0x310 [33106.812038] kthread+0xf5/0x130 [33106.815178] ? kthread_complete_and_exit+0x20/0x20 [33106.819962] ret_from_fork+0x31/0x40 [33106.823541] ? kthread_complete_and_exit+0x20/0x20 [33106.828323] ret_from_fork_asm+0x11/0x20 [33106.832247] [33107.612440] rcu: INFO: rcu_preempt self-detected stall on CPU [33107.618204] rcu: 0-....: (20692996 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10357356 [33107.628715] rcu: (t=21170627 jiffies g=1309609 q=605006 ncpus=8) [33107.634800] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33107.643748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33107.651222] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33107.656701] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33107.675437] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33107.680655] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33107.687777] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33107.694902] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33107.702026] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33107.709151] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33107.716274] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33107.724351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33107.730090] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33107.737215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33107.744339] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33107.751470] Call Trace: [33107.753916] [33107.755927] ? rcu_dump_cpu_stacks+0xd1/0x180 [33107.760286] ? rcu_sched_clock_irq+0x633/0x1390 [33107.764811] ? lock_is_held_type+0xaa/0x120 [33107.768988] ? lock_is_held_type+0xaa/0x120 [33107.773166] ? tick_sched_do_timer+0xa0/0xa0 [33107.777438] ? rcu_read_lock_sched_held+0x56/0x90 [33107.782135] ? tick_sched_do_timer+0xa0/0xa0 [33107.786398] ? update_process_times+0x5f/0x90 [33107.790766] ? tick_sched_handle.isra.16+0x1e/0x50 [33107.795550] ? tick_sched_timer+0x6f/0x80 [33107.799556] ? __hrtimer_run_queues+0x1d3/0x360 [33107.804082] ? hrtimer_interrupt+0x104/0x240 [33107.808351] ? __local_bh_enable+0x50/0x60 [33107.812448] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33107.817686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33107.822644] [33107.824740] [33107.826837] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33107.832146] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33107.837022] do_raw_spin_lock+0xb5/0xc0 [33107.840853] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33107.845656] try_get_cap_refs+0x159/0x880 [ceph] [33107.850285] __ceph_get_caps+0x4a1/0x760 [ceph] [33107.854827] ? __file_remove_privs+0x89/0x110 [33107.859183] ? wake_bit_function+0x70/0x70 [33107.863275] ? ceph_write_iter+0x260/0x770 [ceph] [33107.867988] ceph_write_iter+0x260/0x770 [ceph] [33107.872523] ? lock_is_held_type+0xaa/0x120 [33107.876708] vfs_write+0x324/0x4f0 [33107.880109] ksys_pwrite64+0x65/0xa0 [33107.883684] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33107.888816] do_syscall_64+0x38/0x80 [33107.892386] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33107.897428] RIP: 0033:0x7f48f0eeb93b [33107.900999] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33107.919752] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [33107.927312] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33107.934436] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33107.941561] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33107.948685] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33107.955809] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33107.962939] [33107.965126] Sending NMI from CPU 0 to CPUs 7: [33107.969479] NMI backtrace for cpu 7 [33107.969481] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33107.969484] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33107.969485] Workqueue: events delayed_work [ceph] [33107.969505] RIP: 0010:mark_lock+0x55/0x4c0 [33107.969508] Code: 00 00 00 49 89 ff 49 89 f5 89 d3 83 fa 08 74 6c 41 be 01 00 00 00 89 d1 41 d3 e6 4d 63 f6 41 0f b7 45 20 66 25 ff 1f 0f b7 c0 <48> 0f a3 05 d3 7e e0 01 73 79 48 8d 14 40 48 8d 04 90 48 c1 e0 04 [33107.969510] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000006 [33107.969512] RAX: 00000000000006f5 RBX: 0000000000000008 RCX: 0000000000000000 [33107.969513] RDX: 0000000000000008 RSI: ffff88810debb550 RDI: ffff88810debaa80 [33107.969515] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [33107.969516] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [33107.969517] R13: ffff88810debb550 R14: 0000000000000100 R15: ffff88810debaa80 [33107.969519] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33107.969520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33107.969522] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33107.969523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33107.969524] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33107.969526] Call Trace: [33107.969527] [33107.969528] ? nmi_cpu_backtrace+0xad/0x120 [33107.969532] ? nmi_cpu_backtrace_handler+0x11/0x20 [33107.969534] ? nmi_handle+0xd4/0x260 [33107.969540] ? default_do_nmi+0x49/0xf0 [33107.969542] ? exc_nmi+0x11e/0x150 [33107.969544] ? end_repeat_nmi+0x16/0x67 [33107.969550] ? mark_lock+0x55/0x4c0 [33107.969553] ? mark_lock+0x55/0x4c0 [33107.969556] ? mark_lock+0x55/0x4c0 [33107.969558] [33107.969559] [33107.969559] ? _raw_spin_unlock+0x23/0x40 [33107.969562] ? ceph_check_caps+0x275/0xae0 [ceph] [33107.969584] __lock_acquire+0x4d6/0x1e20 [33107.969589] lock_acquire+0xcb/0x2b0 [33107.969591] ? igrab+0x1d/0x50 [33107.969594] ? igrab+0x43/0x50 [33107.969596] ? lock_release+0xc5/0x260 [33107.969599] _raw_spin_lock+0x30/0x40 [33107.969601] ? igrab+0x1d/0x50 [33107.969603] igrab+0x1d/0x50 [33107.969605] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33107.969625] delayed_work+0x1d8/0x280 [ceph] [33107.969644] ? process_scheduled_works+0x22a/0x510 [33107.969648] process_scheduled_works+0x2a4/0x510 [33107.969654] worker_thread+0x188/0x340 [33107.969657] ? rescuer_thread+0x310/0x310 [33107.969660] kthread+0xf5/0x130 [33107.969663] ? kthread_complete_and_exit+0x20/0x20 [33107.969666] ret_from_fork+0x31/0x40 [33107.969669] ? kthread_complete_and_exit+0x20/0x20 [33107.969672] ret_from_fork_asm+0x11/0x20 [33107.969678] [33134.039537] watchdog: BUG: soft lockup - CPU#0 stuck for 19744s! [dbench:78683] [33134.046839] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33134.051536] watchdog: BUG: soft lockup - CPU#7 stuck for 28741s! [kworker/7:4:78808] [33134.102279] irq event stamp: 56891266 [33134.102280] hardirqs last enabled at (56891265): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33134.110011] Modules linked in: [33134.113668] hardirqs last disabled at (56891266): [] sysvec_apic_timer_interrupt+0xe/0x90 [33134.123826] ceph [33134.126877] softirqs last enabled at (56891208): [] __do_softirq+0x2be/0x371 [33134.136602] libceph [33134.138525] softirqs last disabled at (56891203): [] irq_exit_rcu+0xb0/0xd0 [33134.147212] dns_resolver fscache [33134.149391] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33134.157904] netfs veth [33134.161231] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33134.170176] nft_chain_nat [33134.172620] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33134.180089] xt_MASQUERADE [33134.182794] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33134.188271] nf_nat [33134.190977] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33134.209714] nf_conntrack nf_defrag_ipv6 [33134.211812] [33134.211813] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33134.217029] nf_defrag_ipv4 [33134.220945] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33134.222437] nft_compat nf_tables [33134.229560] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33134.232351] nfnetlink bridge [33134.239475] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33134.242786] stp llc [33134.249911] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33134.252892] xfs libcrc32c [33134.260015] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33134.262215] sunrpc intel_rapl_msr [33134.269340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33134.272045] intel_rapl_common sb_edac [33134.280123] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33134.283521] x86_pkg_temp_thermal [33134.289258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33134.293001] coretemp kvm_intel [33134.300127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33134.303436] kvm irqbypass [33134.310562] Call Trace: [33134.313697] i2c_i801 [33134.320823] [33134.323526] iTCO_wdt iTCO_vendor_support [33134.325972] ? watchdog_timer_fn+0x1b4/0x220 [33134.328240] i2c_smbus crct10dif_pclmul [33134.330254] ? __hrtimer_run_queues+0x1d3/0x360 [33134.334257] joydev [33134.338522] ? hrtimer_interrupt+0x104/0x240 [33134.342351] crc32_pclmul ipmi_ssif [33134.346875] ? __local_bh_enable+0x50/0x60 [33134.348972] ghash_clmulni_intel mei_me [33134.353238] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33134.356720] mei lpc_ich [33134.360811] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33134.364641] sha512_ssse3 acpi_ipmi [33134.369859] [33134.372389] mfd_core wmi [33134.377347] [33134.377348] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33134.380830] ipmi_si ipmi_devintf [33134.382940] ? queued_spin_lock_slowpath+0x18/0x2e0 [33134.385554] ipmi_msghandler ioatdma [33134.387655] do_raw_spin_lock+0xb5/0xc0 [33134.392957] acpi_pad [33134.396268] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33134.401139] acpi_power_meter ixgbe [33134.404711] try_get_cap_refs+0x159/0x880 [ceph] [33134.408539] igb mdio [33134.410815] __ceph_get_caps+0x4a1/0x760 [ceph] [33134.415595] crc32c_intel nvme [33134.419082] ? __file_remove_privs+0x89/0x110 [33134.423690] ptp [33134.425961] ? wake_bit_function+0x70/0x70 [33134.430483] nvme_core i2c_algo_bit [33134.433537] ? ceph_write_iter+0x260/0x770 [ceph] [33134.437884] pps_core dca [33134.439723] ceph_write_iter+0x260/0x770 [ceph] [33134.443813] fuse [33134.447301] ? lock_is_held_type+0xaa/0x120 [33134.451994] irq event stamp: 186157164 [33134.454615] vfs_write+0x324/0x4f0 [33134.459135] hardirqs last enabled at (186157163): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33134.461065] ksys_pwrite64+0x65/0xa0 [33134.465236] hardirqs last disabled at (186157164): [] sysvec_apic_timer_interrupt+0xe/0x90 [33134.468981] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33134.472380] softirqs last enabled at (186157162): [] __do_softirq+0x2be/0x371 [33134.482624] do_syscall_64+0x38/0x80 [33134.486193] softirqs last disabled at (186157157): [] irq_exit_rcu+0xb0/0xd0 [33134.496006] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33134.501135] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33134.509907] RIP: 0033:0x7f48f0eeb93b [33134.513477] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33134.522075] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33134.527119] Workqueue: events delayed_work [ceph] [33134.536495] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33134.540066] [33134.540067] RIP: 0010:lock_acquire+0xf0/0x2b0 [33134.547537] ORIG_RAX: 0000000000000012 [33134.547538] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33134.566276] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33134.570974] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33134.576190] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33134.577681] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33134.582033] [33134.582033] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33134.585862] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33134.592985] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33134.611725] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33134.618849] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33134.624071] [33134.631191] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33134.677619] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33134.684742] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33134.692820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33134.698558] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33134.705682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33134.712806] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33134.719940] Call Trace: [33134.722384] [33134.724396] ? watchdog_timer_fn+0x1b4/0x220 [33134.728668] ? __hrtimer_run_queues+0x1d3/0x360 [33134.733193] ? hrtimer_interrupt+0x104/0x240 [33134.737454] ? __local_bh_enable+0x50/0x60 [33134.741547] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33134.746772] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33134.751728] [33134.753826] [33134.755941] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33134.761249] ? lock_acquire+0xf0/0x2b0 [33134.765000] ? igrab+0x43/0x50 [33134.768050] ? lock_release+0xc5/0x260 [33134.771795] _raw_spin_lock+0x30/0x40 [33134.775450] ? igrab+0x1d/0x50 [33134.778502] igrab+0x1d/0x50 [33134.781378] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33134.786528] delayed_work+0x1d8/0x280 [ceph] [33134.790810] ? process_scheduled_works+0x22a/0x510 [33134.795599] process_scheduled_works+0x2a4/0x510 [33134.800230] worker_thread+0x188/0x340 [33134.803975] ? rescuer_thread+0x310/0x310 [33134.807986] kthread+0xf5/0x130 [33134.811122] ? kthread_complete_and_exit+0x20/0x20 [33134.815908] ret_from_fork+0x31/0x40 [33134.819485] ? kthread_complete_and_exit+0x20/0x20 [33134.824270] ret_from_fork_asm+0x11/0x20 [33134.828193] [33162.040032] watchdog: BUG: soft lockup - CPU#0 stuck for 19770s! [dbench:78683] [33162.047336] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33162.052031] watchdog: BUG: soft lockup - CPU#7 stuck for 28767s! [kworker/7:4:78808] [33162.102784] irq event stamp: 56949418 [33162.110517] Modules linked in: ceph libceph dns_resolver fscache netfs [33162.114174] hardirqs last enabled at (56949417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33162.120708] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [33162.130865] hardirqs last disabled at (56949418): [] sysvec_apic_timer_interrupt+0xe/0x90 [33162.130868] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [33162.146674] softirqs last enabled at (56949360): [] __do_softirq+0x2be/0x371 [33162.146678] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [33162.163115] softirqs last disabled at (56949355): [] irq_exit_rcu+0xb0/0xd0 [33162.172155] coretemp kvm_intel kvm irqbypass [33162.180667] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33162.185016] i2c_i801 iTCO_wdt [33162.193959] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33162.197010] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul [33162.204481] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33162.211779] ipmi_ssif ghash_clmulni_intel mei_me mei [33162.217257] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33162.222300] lpc_ich [33162.241037] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33162.241039] sha512_ssse3 [33162.243221] [33162.248440] acpi_ipmi mfd_core [33162.251058] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33162.252547] wmi ipmi_si [33162.255684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33162.262825] ipmi_devintf ipmi_msghandler [33162.265356] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33162.272481] ioatdma [33162.276486] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33162.283610] acpi_pad acpi_power_meter [33162.285792] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33162.292916] ixgbe igb [33162.296661] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33162.303787] mdio crc32c_intel [33162.306144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33162.314220] nvme ptp nvme_core [33162.317272] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33162.323009] i2c_algo_bit [33162.326145] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33162.333271] pps_core dca [33162.335887] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33162.343011] fuse [33162.345628] Call Trace: [33162.352752] irq event stamp: 186377890 [33162.354680] [33162.357121] hardirqs last enabled at (186377889): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33162.360869] ? watchdog_timer_fn+0x1b4/0x220 [33162.362875] hardirqs last disabled at (186377890): [] sysvec_apic_timer_interrupt+0xe/0x90 [33162.373121] ? __hrtimer_run_queues+0x1d3/0x360 [33162.377385] softirqs last enabled at (186377888): [] __do_softirq+0x2be/0x371 [33162.387199] ? hrtimer_interrupt+0x104/0x240 [33162.391731] softirqs last disabled at (186377883): [] irq_exit_rcu+0xb0/0xd0 [33162.400500] ? __local_bh_enable+0x50/0x60 [33162.404763] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33162.413364] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33162.417452] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33162.426829] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33162.432047] Workqueue: events delayed_work [ceph] [33162.439517] [33162.444475] [33162.444476] RIP: 0010:lock_release+0x14e/0x260 [33162.449171] [33162.451270] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [33162.452760] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33162.457198] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [33162.459299] ? queued_spin_lock_slowpath+0x16/0x2e0 [33162.478031] [33162.478032] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [33162.483339] do_raw_spin_lock+0xb5/0xc0 [33162.488554] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [33162.493435] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33162.494923] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [33162.502051] try_get_cap_refs+0x159/0x880 [ceph] [33162.505879] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [33162.513007] __ceph_get_caps+0x4a1/0x760 [ceph] [33162.517788] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [33162.524915] ? __file_remove_privs+0x89/0x110 [33162.529523] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33162.536646] ? wake_bit_function+0x70/0x70 [33162.541170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33162.548296] ? ceph_write_iter+0x260/0x770 [ceph] [33162.552644] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33162.560722] ceph_write_iter+0x260/0x770 [ceph] [33162.564814] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33162.570554] ? lock_is_held_type+0xaa/0x120 [33162.575247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33162.582375] vfs_write+0x324/0x4f0 [33162.586895] Call Trace: [33162.586897] [33162.594024] ksys_pwrite64+0x65/0xa0 [33162.598199] ? watchdog_timer_fn+0x1b4/0x220 [33162.605323] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33162.608722] ? __hrtimer_run_queues+0x1d3/0x360 [33162.611164] do_syscall_64+0x38/0x80 [33162.613178] ? hrtimer_interrupt+0x104/0x240 [33162.616746] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33162.621008] ? __local_bh_enable+0x50/0x60 [33162.626141] RIP: 0033:0x7f48f0eeb93b [33162.630667] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33162.634235] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33162.638498] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33162.643541] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33162.647632] [33162.651203] ORIG_RAX: 0000000000000012 [33162.656436] [33162.656438] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33162.675177] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33162.680136] ? igrab+0x43/0x50 [33162.685351] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33162.687453] ? lock_release+0x14e/0x260 [33162.691279] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33162.693377] ? lock_release+0xc5/0x260 [33162.698680] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33162.705808] _raw_spin_unlock+0x1b/0x40 [33162.708857] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33162.715981] igrab+0x43/0x50 [33162.719815] [33162.726937] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33162.758953] delayed_work+0x1d8/0x280 [ceph] [33162.763234] ? process_scheduled_works+0x22a/0x510 [33162.768023] process_scheduled_works+0x2a4/0x510 [33162.772639] worker_thread+0x188/0x340 [33162.776389] ? rescuer_thread+0x310/0x310 [33162.780392] kthread+0xf5/0x130 [33162.783532] ? kthread_complete_and_exit+0x20/0x20 [33162.788324] ret_from_fork+0x31/0x40 [33162.791901] ? kthread_complete_and_exit+0x20/0x20 [33162.796685] ret_from_fork_asm+0x11/0x20 [33162.800610] [33190.040526] watchdog: BUG: soft lockup - CPU#0 stuck for 19796s! [dbench:78683] [33190.047831] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33190.052526] watchdog: BUG: soft lockup - CPU#7 stuck for 28793s! [kworker/7:4:78808] [33190.103273] irq event stamp: 57007434 [33190.103274] hardirqs last enabled at (57007433): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33190.111003] Modules linked in: [33190.114661] hardirqs last disabled at (57007434): [] sysvec_apic_timer_interrupt+0xe/0x90 [33190.124819] ceph [33190.127870] softirqs last enabled at (57007406): [] __do_softirq+0x2be/0x371 [33190.137592] libceph [33190.139517] softirqs last disabled at (57007401): [] irq_exit_rcu+0xb0/0xd0 [33190.148200] dns_resolver fscache [33190.150386] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33190.158895] netfs [33190.162223] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33190.171168] veth nft_chain_nat [33190.173180] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33190.180666] xt_MASQUERADE [33190.183803] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33190.189281] nf_nat nf_conntrack [33190.191985] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33190.210723] nf_defrag_ipv6 [33190.213948] [33190.213948] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33190.219165] nf_defrag_ipv4 nft_compat [33190.221956] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33190.223446] nf_tables nfnetlink [33190.230570] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33190.234315] bridge stp [33190.241439] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33190.244663] llc xfs [33190.251786] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33190.254230] libcrc32c sunrpc [33190.261356] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33190.263539] intel_rapl_msr intel_rapl_common [33190.270663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33190.273628] sb_edac x86_pkg_temp_thermal [33190.281706] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33190.286054] coretemp kvm_intel [33190.291793] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33190.295798] kvm irqbypass [33190.302924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33190.306058] i2c_i801 iTCO_wdt [33190.313183] Call Trace: [33190.315921] iTCO_vendor_support [33190.323046] [33190.326096] i2c_smbus crct10dif_pclmul [33190.328541] ? watchdog_timer_fn+0x1b4/0x220 [33190.331765] joydev [33190.333776] ? __hrtimer_run_queues+0x1d3/0x360 [33190.337605] crc32_pclmul ipmi_ssif [33190.341873] ? hrtimer_interrupt+0x104/0x240 [33190.343967] ghash_clmulni_intel [33190.348491] ? __local_bh_enable+0x50/0x60 [33190.351975] mei_me [33190.356243] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33190.359464] mei lpc_ich [33190.363554] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33190.365651] sha512_ssse3 acpi_ipmi [33190.370870] [33190.373399] mfd_core wmi [33190.378359] [33190.378360] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33190.381840] ipmi_si ipmi_devintf [33190.383943] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33190.386556] ipmi_msghandler [33190.388655] do_raw_spin_lock+0xb5/0xc0 [33190.393957] ioatdma [33190.397269] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33190.402139] acpi_pad acpi_power_meter [33190.405018] try_get_cap_refs+0x159/0x880 [ceph] [33190.408846] ixgbe igb [33190.411035] __ceph_get_caps+0x4a1/0x760 [ceph] [33190.415815] mdio [33190.419564] ? __file_remove_privs+0x89/0x110 [33190.424171] crc32c_intel nvme [33190.426529] ? wake_bit_function+0x70/0x70 [33190.431052] ptp nvme_core [33190.432978] ? ceph_write_iter+0x260/0x770 [ceph] [33190.437326] i2c_algo_bit pps_core [33190.440378] ceph_write_iter+0x260/0x770 [ceph] [33190.444468] dca fuse [33190.447175] ? lock_is_held_type+0xaa/0x120 [33190.451869] [33190.451870] irq event stamp: 186598772 [33190.455270] vfs_write+0x324/0x4f0 [33190.459792] hardirqs last enabled at (186598771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33190.462067] ksys_pwrite64+0x65/0xa0 [33190.466240] hardirqs last disabled at (186598772): [] sysvec_apic_timer_interrupt+0xe/0x90 [33190.467731] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33190.471473] softirqs last enabled at (186598770): [] __do_softirq+0x2be/0x371 [33190.474874] do_syscall_64+0x38/0x80 [33190.485116] softirqs last disabled at (186598765): [] irq_exit_rcu+0xb0/0xd0 [33190.488688] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33190.498496] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33190.503627] RIP: 0033:0x7f48f0eeb93b [33190.512398] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33190.515970] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33190.524566] Workqueue: events delayed_work [ceph] [33190.529610] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33190.538989] [33190.538990] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [33190.542559] ORIG_RAX: 0000000000000012 [33190.550030] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [33190.568768] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33190.573465] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [33190.578683] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33190.580174] [33190.580175] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [33190.584784] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33190.588613] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [33190.607352] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33190.614477] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [33190.619692] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33190.626819] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33190.628314] [33190.635434] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33190.687486] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33190.695562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33190.701300] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33190.708426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33190.715550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33190.722674] Call Trace: [33190.725119] [33190.727129] ? watchdog_timer_fn+0x1b4/0x220 [33190.731393] ? __hrtimer_run_queues+0x1d3/0x360 [33190.735937] ? hrtimer_interrupt+0x104/0x240 [33190.740211] ? __local_bh_enable+0x50/0x60 [33190.744309] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33190.749533] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33190.754490] [33190.756586] [33190.758684] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33190.763993] ? do_raw_spin_lock+0x42/0xc0 [33190.768001] igrab+0x1d/0x50 [33190.770879] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33190.776028] delayed_work+0x1d8/0x280 [ceph] [33190.780310] ? process_scheduled_works+0x22a/0x510 [33190.785100] process_scheduled_works+0x2a4/0x510 [33190.789717] worker_thread+0x188/0x340 [33190.793466] ? rescuer_thread+0x310/0x310 [33190.797468] kthread+0xf5/0x130 [33190.800607] ? kthread_complete_and_exit+0x20/0x20 [33190.805390] ret_from_fork+0x31/0x40 [33190.808969] ? kthread_complete_and_exit+0x20/0x20 [33190.813754] ret_from_fork_asm+0x11/0x20 [33190.817677] [33218.041022] watchdog: BUG: soft lockup - CPU#0 stuck for 19822s! [dbench:78683] [33218.048329] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33218.053021] watchdog: BUG: soft lockup - CPU#7 stuck for 28819s! [kworker/7:4:78808] [33218.103777] irq event stamp: 57065580 [33218.111508] Modules linked in: ceph libceph dns_resolver fscache netfs [33218.115165] hardirqs last enabled at (57065579): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33218.115170] veth nft_chain_nat xt_MASQUERADE nf_nat [33218.131850] hardirqs last disabled at (57065580): [] sysvec_apic_timer_interrupt+0xe/0x90 [33218.131851] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [33218.146531] softirqs last enabled at (57065552): [] __do_softirq+0x2be/0x371 [33218.155414] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [33218.164098] softirqs last disabled at (57065547): [] irq_exit_rcu+0xb0/0xd0 [33218.170615] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [33218.179129] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33218.184779] kvm irqbypass [33218.193721] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33218.193724] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33218.196427] i2c_i801 [33218.203897] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33218.209376] iTCO_wdt [33218.211646] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33218.230383] iTCO_vendor_support i2c_smbus [33218.232652] [33218.237870] crct10dif_pclmul [33218.241962] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33218.243452] joydev crc32_pclmul [33218.246418] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33218.253541] ipmi_ssif ghash_clmulni_intel [33218.256765] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33218.263887] mei_me mei [33218.267979] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33218.275102] lpc_ich [33218.277546] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33218.284672] sha512_ssse3 acpi_ipmi [33218.286856] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33218.293980] mfd_core wmi [33218.297463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33218.305540] ipmi_si ipmi_devintf [33218.308158] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33218.313896] ipmi_msghandler [33218.317207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33218.324331] ioatdma acpi_pad [33218.327207] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33218.334333] acpi_power_meter ixgbe [33218.337297] Call Trace: [33218.337299] [33218.344422] igb [33218.347906] ? watchdog_timer_fn+0x1b4/0x220 [33218.350347] mdio crc32c_intel [33218.352360] ? __hrtimer_run_queues+0x1d3/0x360 [33218.354198] nvme [33218.358465] ? hrtimer_interrupt+0x104/0x240 [33218.361513] ptp [33218.366036] ? __local_bh_enable+0x50/0x60 [33218.367959] nvme_core i2c_algo_bit [33218.372226] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33218.374061] pps_core dca [33218.378151] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33218.381635] fuse [33218.386854] [33218.389470] irq event stamp: 186819576 [33218.394428] [33218.396353] hardirqs last enabled at (186819575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33218.398450] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33218.402193] hardirqs last disabled at (186819576): [] sysvec_apic_timer_interrupt+0xe/0x90 [33218.404295] ? queued_spin_lock_slowpath+0x16/0x2e0 [33218.414536] softirqs last enabled at (186819574): [] __do_softirq+0x2be/0x371 [33218.419841] do_raw_spin_lock+0xb5/0xc0 [33218.429651] softirqs last disabled at (186819569): [] irq_exit_rcu+0xb0/0xd0 [33218.434522] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33218.443291] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33218.447125] try_get_cap_refs+0x159/0x880 [ceph] [33218.455720] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33218.460508] __ceph_get_caps+0x4a1/0x760 [ceph] [33218.469882] Workqueue: events delayed_work [ceph] [33218.474496] ? __file_remove_privs+0x89/0x110 [33218.481962] [33218.481963] RIP: 0010:lock_acquire+0x86/0x2b0 [33218.486487] ? wake_bit_function+0x70/0x70 [33218.491185] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [33218.495539] ? ceph_write_iter+0x260/0x770 [ceph] [33218.497025] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [33218.501378] ceph_write_iter+0x260/0x770 [ceph] [33218.505467] [33218.505468] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [33218.524209] ? lock_is_held_type+0xaa/0x120 [33218.528903] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [33218.534122] vfs_write+0x324/0x4f0 [33218.538643] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [33218.540138] ksys_pwrite64+0x65/0xa0 [33218.547258] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [33218.551437] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33218.558561] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33218.561960] do_syscall_64+0x38/0x80 [33218.569082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33218.572652] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33218.579777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33218.584909] RIP: 0033:0x7f48f0eeb93b [33218.592031] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33218.595603] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33218.603680] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33218.608742] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33218.614478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33218.618048] ORIG_RAX: 0000000000000012 [33218.625174] Call Trace: [33218.625175] [33218.643912] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33218.651036] ? watchdog_timer_fn+0x1b4/0x220 [33218.656253] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33218.663377] ? __hrtimer_run_queues+0x1d3/0x360 [33218.667207] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33218.669654] ? hrtimer_interrupt+0x104/0x240 [33218.671662] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33218.678786] ? __local_bh_enable+0x50/0x60 [33218.683051] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33218.690177] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33218.694712] [33218.701832] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33218.736795] [33218.738890] [33218.740989] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33218.746296] ? lock_acquire+0x86/0x2b0 [33218.750048] ? igrab+0x43/0x50 [33218.753098] ? lock_release+0xc5/0x260 [33218.756843] _raw_spin_lock+0x30/0x40 [33218.760508] ? igrab+0x1d/0x50 [33218.763556] igrab+0x1d/0x50 [33218.766437] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33218.771584] delayed_work+0x1d8/0x280 [ceph] [33218.775867] ? process_scheduled_works+0x22a/0x510 [33218.780657] process_scheduled_works+0x2a4/0x510 [33218.785273] worker_thread+0x188/0x340 [33218.789022] ? rescuer_thread+0x310/0x310 [33218.793025] kthread+0xf5/0x130 [33218.796173] ? kthread_complete_and_exit+0x20/0x20 [33218.800965] ret_from_fork+0x31/0x40 [33218.804545] ? kthread_complete_and_exit+0x20/0x20 [33218.809336] ret_from_fork_asm+0x11/0x20 [33218.813269] [33246.041516] watchdog: BUG: soft lockup - CPU#0 stuck for 19848s! [dbench:78683] [33246.048815] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33246.053516] watchdog: BUG: soft lockup - CPU#7 stuck for 28845s! [kworker/7:4:78808] [33246.104255] irq event stamp: 57123628 [33246.104256] hardirqs last enabled at (57123627): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33246.111988] Modules linked in: [33246.115644] hardirqs last disabled at (57123628): [] sysvec_apic_timer_interrupt+0xe/0x90 [33246.125803] ceph [33246.128853] softirqs last enabled at (57123600): [] __do_softirq+0x2be/0x371 [33246.138576] libceph [33246.140501] softirqs last disabled at (57123595): [] irq_exit_rcu+0xb0/0xd0 [33246.149200] dns_resolver [33246.151388] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33246.159911] fscache netfs [33246.162531] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33246.171477] veth nft_chain_nat [33246.174190] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33246.181660] xt_MASQUERADE nf_nat [33246.184796] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33246.190275] nf_conntrack [33246.193585] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33246.212322] nf_defrag_ipv6 nf_defrag_ipv4 [33246.214941] [33246.214942] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33246.220157] nft_compat nf_tables [33246.224248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33246.225739] nfnetlink bridge [33246.232864] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33246.236174] stp llc [33246.243298] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33246.246261] xfs libcrc32c [33246.253387] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33246.255571] sunrpc intel_rapl_msr [33246.262695] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33246.265398] intel_rapl_common sb_edac [33246.272522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33246.275920] x86_pkg_temp_thermal [33246.283997] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33246.287743] coretemp kvm_intel [33246.293478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33246.296788] kvm [33246.303914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33246.307051] irqbypass i2c_i801 [33246.314189] Call Trace: [33246.316030] iTCO_wdt iTCO_vendor_support [33246.323155] [33246.326290] i2c_smbus crct10dif_pclmul [33246.328736] ? watchdog_timer_fn+0x1b4/0x220 [33246.332739] joydev crc32_pclmul [33246.334750] ? __hrtimer_run_queues+0x1d3/0x360 [33246.338582] ipmi_ssif ghash_clmulni_intel [33246.342849] ? hrtimer_interrupt+0x104/0x240 [33246.346069] mei_me mei [33246.350593] ? __local_bh_enable+0x50/0x60 [33246.354684] lpc_ich sha512_ssse3 [33246.358951] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33246.361393] acpi_ipmi mfd_core [33246.365484] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33246.368795] wmi ipmi_si [33246.374012] [33246.377149] ipmi_devintf ipmi_msghandler [33246.382105] [33246.384637] ioatdma acpi_pad [33246.386735] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33246.390738] acpi_power_meter [33246.392840] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33246.395799] ixgbe [33246.401107] do_raw_spin_lock+0xb5/0xc0 [33246.404069] igb mdio [33246.408939] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33246.410948] crc32c_intel nvme [33246.414782] try_get_cap_refs+0x159/0x880 [ceph] [33246.417051] ptp nvme_core [33246.421840] __ceph_get_caps+0x4a1/0x760 [ceph] [33246.424886] i2c_algo_bit [33246.429501] ? __file_remove_privs+0x89/0x110 [33246.432201] pps_core dca [33246.436725] ? wake_bit_function+0x70/0x70 [33246.439341] fuse [33246.443694] ? ceph_write_iter+0x260/0x770 [ceph] [33246.446310] [33246.446311] irq event stamp: 187040322 [33246.450402] ceph_write_iter+0x260/0x770 [ceph] [33246.452325] hardirqs last enabled at (187040321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33246.457026] ? lock_is_held_type+0xaa/0x120 [33246.458513] hardirqs last disabled at (187040322): [] sysvec_apic_timer_interrupt+0xe/0x90 [33246.462259] vfs_write+0x324/0x4f0 [33246.466781] softirqs last enabled at (187040320): [] __do_softirq+0x2be/0x371 [33246.477031] ksys_pwrite64+0x65/0xa0 [33246.481202] softirqs last disabled at (187040315): [] irq_exit_rcu+0xb0/0xd0 [33246.491012] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33246.494411] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33246.503183] do_syscall_64+0x38/0x80 [33246.506753] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33246.515350] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33246.520480] Workqueue: events delayed_work [ceph] [33246.529857] RIP: 0033:0x7f48f0eeb93b [33246.533429] [33246.533431] RIP: 0010:lock_acquire+0xf0/0x2b0 [33246.540916] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33246.545961] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33246.550658] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33246.554229] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33246.555720] ORIG_RAX: 0000000000000012 [33246.560070] [33246.560070] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33246.578809] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33246.597546] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33246.602763] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33246.607981] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33246.611811] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33246.613301] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33246.620425] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33246.627550] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33246.634674] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33246.641800] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33246.648929] [33246.656048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33246.700543] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33246.707667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33246.714791] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33246.721921] Call Trace: [33246.724368] [33246.726379] ? watchdog_timer_fn+0x1b4/0x220 [33246.730643] ? __hrtimer_run_queues+0x1d3/0x360 [33246.735170] ? hrtimer_interrupt+0x104/0x240 [33246.739431] ? __local_bh_enable+0x50/0x60 [33246.743525] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33246.748747] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33246.753706] [33246.755803] [33246.757919] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33246.763234] ? lock_acquire+0xf0/0x2b0 [33246.766985] ? igrab+0x43/0x50 [33246.770033] ? lock_release+0xc5/0x260 [33246.773779] _raw_spin_lock+0x30/0x40 [33246.777445] ? igrab+0x1d/0x50 [33246.780494] igrab+0x1d/0x50 [33246.783373] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33246.788521] delayed_work+0x1d8/0x280 [ceph] [33246.792803] ? process_scheduled_works+0x22a/0x510 [33246.797595] process_scheduled_works+0x2a4/0x510 [33246.802209] worker_thread+0x188/0x340 [33246.805959] ? rescuer_thread+0x310/0x310 [33246.809960] kthread+0xf5/0x130 [33246.813100] ? kthread_complete_and_exit+0x20/0x20 [33246.817884] ret_from_fork+0x31/0x40 [33246.821464] ? kthread_complete_and_exit+0x20/0x20 [33246.826246] ret_from_fork_asm+0x11/0x20 [33246.830170] [33274.042037] watchdog: BUG: soft lockup - CPU#0 stuck for 19874s! [dbench:78683] [33274.049338] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33274.054036] watchdog: BUG: soft lockup - CPU#7 stuck for 28871s! [kworker/7:4:78808] [33274.104788] irq event stamp: 57181804 [33274.112518] Modules linked in: ceph libceph dns_resolver [33274.116176] hardirqs last enabled at (57181803): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33274.116180] fscache netfs veth nft_chain_nat xt_MASQUERADE [33274.131646] hardirqs last disabled at (57181804): [] sysvec_apic_timer_interrupt+0xe/0x90 [33274.131650] nf_nat [33274.131651] softirqs last enabled at (57181776): [] __do_softirq+0x2be/0x371 [33274.137218] nf_conntrack nf_defrag_ipv6 [33274.146943] softirqs last disabled at (57181771): [] irq_exit_rcu+0xb0/0xd0 [33274.149039] nf_defrag_ipv4 [33274.157726] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33274.161643] nft_compat nf_tables [33274.170152] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33274.172943] nfnetlink bridge [33274.181888] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33274.185198] stp llc [33274.192670] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33274.195634] xfs libcrc32c [33274.201111] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33274.203294] sunrpc intel_rapl_msr [33274.222031] [33274.224736] intel_rapl_common sb_edac [33274.229955] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33274.233351] x86_pkg_temp_thermal [33274.234841] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33274.238586] coretemp kvm_intel [33274.245716] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33274.249030] kvm irqbypass [33274.256154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33274.259291] i2c_i801 [33274.266432] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33274.269136] iTCO_wdt [33274.276261] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33274.278532] iTCO_vendor_support i2c_smbus [33274.285655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33274.287926] crct10dif_pclmul joydev [33274.296003] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33274.300094] crc32_pclmul ipmi_ssif [33274.305833] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33274.309404] ghash_clmulni_intel [33274.316525] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33274.320011] mei_me [33274.327134] Call Trace: [33274.327137] [33274.330359] mei lpc_ich [33274.337486] ? watchdog_timer_fn+0x1b4/0x220 [33274.339581] sha512_ssse3 acpi_ipmi [33274.342027] ? __hrtimer_run_queues+0x1d3/0x360 [33274.344036] mfd_core wmi [33274.346569] ? hrtimer_interrupt+0x104/0x240 [33274.350829] ipmi_si [33274.354314] ? __local_bh_enable+0x50/0x60 [33274.358839] ipmi_devintf ipmi_msghandler [33274.361457] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33274.365720] ioatdma acpi_pad [33274.367903] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33274.371995] acpi_power_meter ixgbe [33274.375999] [33274.381216] igb mdio [33274.384180] [33274.389136] crc32c_intel [33274.392621] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33274.394727] nvme ptp [33274.397002] ? queued_spin_lock_slowpath+0x16/0x2e0 [33274.399095] nvme_core [33274.401729] do_raw_spin_lock+0xb5/0xc0 [33274.407033] i2c_algo_bit [33274.409307] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33274.414176] pps_core [33274.416536] try_get_cap_refs+0x159/0x880 [ceph] [33274.420365] dca fuse [33274.422987] __ceph_get_caps+0x4a1/0x760 [ceph] [33274.427764] [33274.427765] irq event stamp: 187261052 [33274.430042] ? __file_remove_privs+0x89/0x110 [33274.434648] hardirqs last enabled at (187261051): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33274.436917] ? wake_bit_function+0x70/0x70 [33274.441440] hardirqs last disabled at (187261052): [] sysvec_apic_timer_interrupt+0xe/0x90 [33274.442934] ? ceph_write_iter+0x260/0x770 [ceph] [33274.446677] softirqs last enabled at (187261050): [] __do_softirq+0x2be/0x371 [33274.451027] ceph_write_iter+0x260/0x770 [ceph] [33274.461271] softirqs last disabled at (187261045): [] irq_exit_rcu+0xb0/0xd0 [33274.465364] ? lock_is_held_type+0xaa/0x120 [33274.475173] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33274.479873] vfs_write+0x324/0x4f0 [33274.488641] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33274.493172] ksys_pwrite64+0x65/0xa0 [33274.501763] Workqueue: events delayed_work [ceph] [33274.505942] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33274.515318] [33274.518732] do_syscall_64+0x38/0x80 [33274.526203] RIP: 0010:ceph_check_delayed_caps+0x5f/0x270 [ceph] [33274.529775] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33274.534471] Code: 1f 44 00 00 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 <48> 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff [33274.539602] RIP: 0033:0x7f48f0eeb93b [33274.541092] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [33274.544664] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33274.550573] [33274.550574] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [33274.555617] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33274.574357] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [33274.577927] ORIG_RAX: 0000000000000012 [33274.583144] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [33274.601882] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33274.603372] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33274.610497] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33274.615722] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33274.622846] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33274.626677] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33274.633802] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33274.640927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33274.648051] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33274.655173] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33274.662303] [33274.669439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33274.713936] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33274.721060] Call Trace: [33274.723504] [33274.725514] ? watchdog_timer_fn+0x1b4/0x220 [33274.729779] ? __hrtimer_run_queues+0x1d3/0x360 [33274.734307] ? hrtimer_interrupt+0x104/0x240 [33274.738574] ? __local_bh_enable+0x50/0x60 [33274.742670] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33274.747893] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33274.752849] [33274.754948] [33274.757045] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33274.762353] ? ceph_check_delayed_caps+0x5f/0x270 [ceph] [33274.767672] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [33274.772993] delayed_work+0x1d8/0x280 [ceph] [33274.777277] ? process_scheduled_works+0x22a/0x510 [33274.782065] process_scheduled_works+0x2a4/0x510 [33274.786681] worker_thread+0x188/0x340 [33274.790444] ? rescuer_thread+0x310/0x310 [33274.794450] kthread+0xf5/0x130 [33274.797589] ? kthread_complete_and_exit+0x20/0x20 [33274.802372] ret_from_fork+0x31/0x40 [33274.805951] ? kthread_complete_and_exit+0x20/0x20 [33274.810737] ret_from_fork_asm+0x11/0x20 [33274.814668] [33302.042557] watchdog: BUG: soft lockup - CPU#0 stuck for 19900s! [dbench:78683] [33302.049860] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33302.054557] watchdog: BUG: soft lockup - CPU#7 stuck for 28897s! [kworker/7:4:78808] [33302.105302] irq event stamp: 57239900 [33302.105303] hardirqs last enabled at (57239899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33302.113032] Modules linked in: [33302.116691] hardirqs last disabled at (57239900): [] sysvec_apic_timer_interrupt+0xe/0x90 [33302.126846] ceph [33302.129896] softirqs last enabled at (57239872): [] __do_softirq+0x2be/0x371 [33302.139623] libceph [33302.141546] softirqs last disabled at (57239867): [] irq_exit_rcu+0xb0/0xd0 [33302.150234] dns_resolver [33302.152413] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33302.160925] fscache [33302.163560] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33302.172503] netfs [33302.174687] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33302.182159] veth nft_chain_nat [33302.184169] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33302.189647] xt_MASQUERADE nf_nat [33302.192784] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33302.211522] nf_conntrack nf_defrag_ipv6 [33302.214833] [33302.214834] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33302.220049] nf_defrag_ipv4 nft_compat [33302.223966] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33302.225458] nf_tables nfnetlink [33302.232583] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33302.236327] bridge stp [33302.243449] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33302.246674] llc [33302.253797] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33302.256243] xfs libcrc32c [33302.263367] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33302.265205] sunrpc intel_rapl_msr [33302.272327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33302.275032] intel_rapl_common sb_edac [33302.283109] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33302.286506] x86_pkg_temp_thermal coretemp [33302.292246] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33302.295990] kvm_intel kvm [33302.303111] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33302.307203] irqbypass [33302.314327] Call Trace: [33302.314328] [33302.317032] i2c_i801 iTCO_wdt [33302.324158] ? watchdog_timer_fn+0x1b4/0x220 [33302.326513] iTCO_vendor_support i2c_smbus [33302.328957] ? __hrtimer_run_queues+0x1d3/0x360 [33302.330969] crct10dif_pclmul [33302.334022] ? hrtimer_interrupt+0x104/0x240 [33302.338282] joydev crc32_pclmul [33302.342375] ? __local_bh_enable+0x50/0x60 [33302.346898] ipmi_ssif ghash_clmulni_intel [33302.349865] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33302.354126] mei_me [33302.357350] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33302.361439] mei lpc_ich [33302.365531] [33302.370748] sha512_ssse3 acpi_ipmi [33302.372846] [33302.372847] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33302.377804] mfd_core [33302.380338] ? queued_spin_lock_slowpath+0x16/0x2e0 [33302.382431] wmi ipmi_si [33302.385917] do_raw_spin_lock+0xb5/0xc0 [33302.388014] ipmi_devintf ipmi_msghandler [33302.393318] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33302.395588] ioatdma acpi_pad [33302.400461] try_get_cap_refs+0x159/0x880 [ceph] [33302.402989] acpi_power_meter [33302.406825] __ceph_get_caps+0x4a1/0x760 [ceph] [33302.410824] ixgbe igb [33302.415612] ? __file_remove_privs+0x89/0x110 [33302.418572] mdio crc32c_intel [33302.423185] ? wake_bit_function+0x70/0x70 [33302.426147] nvme ptp [33302.430674] ? ceph_write_iter+0x260/0x770 [ceph] [33302.433028] nvme_core i2c_algo_bit [33302.437379] ceph_write_iter+0x260/0x770 [ceph] [33302.440430] pps_core [33302.444524] ? lock_is_held_type+0xaa/0x120 [33302.446791] dca fuse [33302.451492] vfs_write+0x324/0x4f0 [33302.454973] [33302.454974] irq event stamp: 187481722 [33302.459500] ksys_pwrite64+0x65/0xa0 [33302.461768] hardirqs last enabled at (187481721): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33302.465958] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33302.468238] hardirqs last disabled at (187481722): [] sysvec_apic_timer_interrupt+0xe/0x90 [33302.471641] do_syscall_64+0x38/0x80 [33302.473131] softirqs last enabled at (187481720): [] __do_softirq+0x2be/0x371 [33302.476874] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33302.480446] softirqs last disabled at (187481715): [] irq_exit_rcu+0xb0/0xd0 [33302.490688] RIP: 0033:0x7f48f0eeb93b [33302.495819] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33302.505632] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33302.509202] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33302.517972] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33302.523016] Workqueue: events delayed_work [ceph] [33302.531612] ORIG_RAX: 0000000000000012 [33302.535186] [33302.544562] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33302.563300] RIP: 0010:lock_acquire+0xf0/0x2b0 [33302.570770] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33302.575989] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33302.580685] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33302.584517] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33302.586007] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33302.593131] [33302.593132] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33302.597482] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33302.604606] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33302.623348] [33302.630469] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33302.674982] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33302.682106] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33302.689229] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33302.697307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33302.703043] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33302.710168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33302.717293] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33302.724415] Call Trace: [33302.726861] [33302.728871] ? watchdog_timer_fn+0x1b4/0x220 [33302.733136] ? __hrtimer_run_queues+0x1d3/0x360 [33302.737663] ? hrtimer_interrupt+0x104/0x240 [33302.741933] ? __local_bh_enable+0x50/0x60 [33302.746025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33302.751250] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33302.756205] [33302.758304] [33302.760401] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33302.765709] ? lock_acquire+0xf0/0x2b0 [33302.769458] ? igrab+0x43/0x50 [33302.772510] ? lock_release+0xc5/0x260 [33302.776254] _raw_spin_lock+0x30/0x40 [33302.779910] ? igrab+0x1d/0x50 [33302.782967] igrab+0x1d/0x50 [33302.785849] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33302.790999] delayed_work+0x1d8/0x280 [ceph] [33302.795278] ? process_scheduled_works+0x22a/0x510 [33302.800071] process_scheduled_works+0x2a4/0x510 [33302.804684] worker_thread+0x188/0x340 [33302.808433] ? rescuer_thread+0x310/0x310 [33302.812436] kthread+0xf5/0x130 [33302.815576] ? kthread_complete_and_exit+0x20/0x20 [33302.820359] ret_from_fork+0x31/0x40 [33302.823929] ? kthread_complete_and_exit+0x20/0x20 [33302.828714] ret_from_fork_asm+0x11/0x20 [33302.832637] [33302.976941] rcu: INFO: rcu_preempt self-detected stall on CPU [33302.982698] rcu: 0-....: (20883605 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10453172 [33302.993211] rcu: (t=21365988 jiffies g=1309609 q=608988 ncpus=8) [33302.999295] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33303.008236] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33303.015709] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33303.021187] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33303.039923] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33303.045141] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33303.052265] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33303.059388] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33303.066514] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33303.073637] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33303.080761] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33303.088838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33303.094577] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33303.101702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33303.108825] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33303.115966] Call Trace: [33303.118410] [33303.120422] ? rcu_dump_cpu_stacks+0xd1/0x180 [33303.124782] ? rcu_sched_clock_irq+0x633/0x1390 [33303.129304] ? lock_is_held_type+0xaa/0x120 [33303.133483] ? lock_is_held_type+0xaa/0x120 [33303.137661] ? tick_sched_do_timer+0xa0/0xa0 [33303.141933] ? rcu_read_lock_sched_held+0x56/0x90 [33303.146631] ? tick_sched_do_timer+0xa0/0xa0 [33303.150895] ? update_process_times+0x5f/0x90 [33303.155254] ? tick_sched_handle.isra.16+0x1e/0x50 [33303.160037] ? tick_sched_timer+0x6f/0x80 [33303.164042] ? __hrtimer_run_queues+0x1d3/0x360 [33303.168569] ? hrtimer_interrupt+0x104/0x240 [33303.172837] ? __local_bh_enable+0x50/0x60 [33303.176932] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33303.182156] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33303.187112] [33303.189210] [33303.191307] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33303.196615] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33303.201494] do_raw_spin_lock+0xb5/0xc0 [33303.205330] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33303.210132] try_get_cap_refs+0x159/0x880 [ceph] [33303.214764] __ceph_get_caps+0x4a1/0x760 [ceph] [33303.219305] ? __file_remove_privs+0x89/0x110 [33303.223660] ? wake_bit_function+0x70/0x70 [33303.227753] ? ceph_write_iter+0x260/0x770 [ceph] [33303.232458] ceph_write_iter+0x260/0x770 [ceph] [33303.236994] ? lock_is_held_type+0xaa/0x120 [33303.241178] vfs_write+0x324/0x4f0 [33303.244577] ksys_pwrite64+0x65/0xa0 [33303.248153] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33303.253284] do_syscall_64+0x38/0x80 [33303.256853] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33303.261898] RIP: 0033:0x7f48f0eeb93b [33303.265469] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33303.284206] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [33303.291764] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33303.298887] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33303.306012] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33303.313135] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33303.320268] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33303.327399] [33303.329586] Sending NMI from CPU 0 to CPUs 7: [33303.333939] NMI backtrace for cpu 7 [33303.333940] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33303.333942] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33303.333943] Workqueue: events delayed_work [ceph] [33303.333980] RIP: 0010:mark_lock+0xc/0x4c0 [33303.333982] Code: 48 c7 c7 8a 3f 12 82 e8 42 bf f7 ff 0f 0b 31 c0 e9 8b fe ff ff 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 <53> 48 83 e4 f0 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 44 24 [33303.333984] RSP: 0018:ffffc90002117c30 EFLAGS: 00000046 [33303.333986] RAX: 0000000000000004 RBX: ffff88810debaa80 RCX: 00000000000006f5 [33303.333988] RDX: 0000000000000002 RSI: ffff88810debb550 RDI: ffff88810debaa80 [33303.333989] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [33303.333990] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [33303.333991] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [33303.333992] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33303.333994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33303.333996] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33303.333997] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33303.333998] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33303.333999] Call Trace: [33303.334000] [33303.334002] ? nmi_cpu_backtrace+0xad/0x120 [33303.334006] ? nmi_cpu_backtrace_handler+0x11/0x20 [33303.334008] ? nmi_handle+0xd4/0x260 [33303.334013] ? default_do_nmi+0x49/0xf0 [33303.334016] ? exc_nmi+0x11e/0x150 [33303.334018] ? end_repeat_nmi+0x16/0x67 [33303.334024] ? mark_lock+0xc/0x4c0 [33303.334026] ? mark_lock+0xc/0x4c0 [33303.334029] ? mark_lock+0xc/0x4c0 [33303.334031] [33303.334032] [33303.334034] __lock_acquire+0x49a/0x1e20 [33303.334039] lock_acquire+0xcb/0x2b0 [33303.334041] ? igrab+0x1d/0x50 [33303.334044] ? igrab+0x43/0x50 [33303.334046] ? lock_release+0xc5/0x260 [33303.334049] _raw_spin_lock+0x30/0x40 [33303.334052] ? igrab+0x1d/0x50 [33303.334053] igrab+0x1d/0x50 [33303.334056] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33303.334076] delayed_work+0x1d8/0x280 [ceph] [33303.334095] ? process_scheduled_works+0x22a/0x510 [33303.334098] process_scheduled_works+0x2a4/0x510 [33303.334104] worker_thread+0x188/0x340 [33303.334108] ? rescuer_thread+0x310/0x310 [33303.334111] kthread+0xf5/0x130 [33303.334113] ? kthread_complete_and_exit+0x20/0x20 [33303.334117] ret_from_fork+0x31/0x40 [33303.334120] ? kthread_complete_and_exit+0x20/0x20 [33303.334122] ret_from_fork_asm+0x11/0x20 [33303.334129] [33330.043078] watchdog: BUG: soft lockup - CPU#0 stuck for 19927s! [dbench:78683] [33330.050381] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33330.055078] watchdog: BUG: soft lockup - CPU#7 stuck for 28924s! [kworker/7:4:78808] [33330.105823] irq event stamp: 57297388 [33330.105824] hardirqs last enabled at (57297387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33330.113554] Modules linked in: [33330.117212] hardirqs last disabled at (57297388): [] sysvec_apic_timer_interrupt+0xe/0x90 [33330.127370] ceph [33330.130420] softirqs last enabled at (57297360): [] __do_softirq+0x2be/0x371 [33330.140145] libceph [33330.142068] softirqs last disabled at (57297355): [] irq_exit_rcu+0xb0/0xd0 [33330.150754] dns_resolver fscache [33330.152937] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33330.161448] netfs veth [33330.164775] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33330.173719] nft_chain_nat [33330.176165] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33330.183633] xt_MASQUERADE [33330.186338] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33330.191816] nf_nat nf_conntrack [33330.194521] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33330.213257] nf_defrag_ipv6 nf_defrag_ipv4 [33330.216481] [33330.216482] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33330.221699] nft_compat nf_tables [33330.225789] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33330.227280] nfnetlink bridge [33330.234405] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33330.237716] stp llc [33330.244840] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33330.247804] xfs libcrc32c [33330.254928] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33330.257111] sunrpc [33330.264237] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33330.266940] intel_rapl_msr intel_rapl_common [33330.274065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33330.276161] sb_edac x86_pkg_temp_thermal [33330.284239] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33330.288589] coretemp kvm_intel [33330.294328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33330.298330] kvm irqbypass [33330.305473] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33330.308609] i2c_i801 iTCO_wdt [33330.315734] Call Trace: [33330.315735] [33330.318455] iTCO_vendor_support i2c_smbus [33330.325596] ? watchdog_timer_fn+0x1b4/0x220 [33330.328647] crct10dif_pclmul [33330.331093] ? __hrtimer_run_queues+0x1d3/0x360 [33330.333103] joydev crc32_pclmul [33330.337198] ? hrtimer_interrupt+0x104/0x240 [33330.341476] ipmi_ssif [33330.344438] ? __local_bh_enable+0x50/0x60 [33330.348962] ghash_clmulni_intel [33330.352189] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33330.356452] mei_me [33330.358825] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33330.362917] mei lpc_ich [33330.366140] [33330.371359] sha512_ssse3 acpi_ipmi [33330.373454] [33330.378430] mfd_core wmi [33330.380960] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33330.383058] ipmi_si ipmi_devintf [33330.386547] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33330.388639] ipmi_msghandler ioatdma [33330.391259] do_raw_spin_lock+0xb5/0xc0 [33330.396561] acpi_pad acpi_power_meter [33330.399872] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33330.404743] ixgbe [33330.408316] try_get_cap_refs+0x159/0x880 [ceph] [33330.412144] igb mdio [33330.415893] __ceph_get_caps+0x4a1/0x760 [ceph] [33330.420673] crc32c_intel nvme [33330.422688] ? __file_remove_privs+0x89/0x110 [33330.427292] ptp nvme_core [33330.429566] ? wake_bit_function+0x70/0x70 [33330.434089] i2c_algo_bit pps_core [33330.437142] ? ceph_write_iter+0x260/0x770 [ceph] [33330.441488] dca fuse [33330.444194] ceph_write_iter+0x260/0x770 [ceph] [33330.448284] [33330.448285] irq event stamp: 187700474 [33330.451684] ? lock_is_held_type+0xaa/0x120 [33330.456378] hardirqs last enabled at (187700473): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33330.458654] vfs_write+0x324/0x4f0 [33330.463173] hardirqs last disabled at (187700474): [] sysvec_apic_timer_interrupt+0xe/0x90 [33330.464668] ksys_pwrite64+0x65/0xa0 [33330.468410] softirqs last enabled at (187700472): [] __do_softirq+0x2be/0x371 [33330.472586] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33330.482830] softirqs last disabled at (187700467): [] irq_exit_rcu+0xb0/0xd0 [33330.486228] do_syscall_64+0x38/0x80 [33330.496040] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33330.499610] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33330.508381] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33330.513512] RIP: 0033:0x7f48f0eeb93b [33330.522108] Workqueue: events delayed_work [ceph] [33330.525679] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33330.535056] [33330.535057] RIP: 0010:lock_acquire+0xf0/0x2b0 [33330.540100] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33330.547573] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33330.551143] ORIG_RAX: 0000000000000012 [33330.555839] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33330.574577] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33330.576068] [33330.576069] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33330.580419] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33330.585635] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33330.604373] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33330.608204] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33330.613422] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33330.620546] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33330.622038] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33330.629160] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33330.636290] [33330.643410] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33330.696415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33330.702153] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33330.709278] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33330.716402] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33330.723525] Call Trace: [33330.725970] [33330.727981] ? watchdog_timer_fn+0x1b4/0x220 [33330.732245] ? __hrtimer_run_queues+0x1d3/0x360 [33330.736773] ? hrtimer_interrupt+0x104/0x240 [33330.741043] ? __local_bh_enable+0x50/0x60 [33330.745136] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33330.750358] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33330.755315] [33330.757412] [33330.759511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33330.764819] ? lock_acquire+0xf0/0x2b0 [33330.768568] ? igrab+0x43/0x50 [33330.771620] ? lock_release+0xc5/0x260 [33330.775364] _raw_spin_lock+0x30/0x40 [33330.779020] ? igrab+0x1d/0x50 [33330.782071] igrab+0x1d/0x50 [33330.784949] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33330.790099] delayed_work+0x1d8/0x280 [ceph] [33330.794379] ? process_scheduled_works+0x22a/0x510 [33330.799162] process_scheduled_works+0x2a4/0x510 [33330.803778] worker_thread+0x188/0x340 [33330.807526] ? rescuer_thread+0x310/0x310 [33330.811529] kthread+0xf5/0x130 [33330.814666] ? kthread_complete_and_exit+0x20/0x20 [33330.819452] ret_from_fork+0x31/0x40 [33330.823030] ? kthread_complete_and_exit+0x20/0x20 [33330.827814] ret_from_fork_asm+0x11/0x20 [33330.831738] [33358.043600] watchdog: BUG: soft lockup - CPU#0 stuck for 19953s! [dbench:78683] [33358.050907] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33358.055599] watchdog: BUG: soft lockup - CPU#7 stuck for 28950s! [kworker/7:4:78808] [33358.106354] irq event stamp: 57355584 [33358.114086] Modules linked in: ceph libceph dns_resolver fscache netfs [33358.117742] hardirqs last enabled at (57355583): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33358.124260] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [33358.134436] hardirqs last disabled at (57355584): [] sysvec_apic_timer_interrupt+0xe/0x90 [33358.141829] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [33358.151553] softirqs last enabled at (57355556): [] __do_softirq+0x2be/0x371 [33358.151556] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [33358.166329] softirqs last disabled at (57355551): [] irq_exit_rcu+0xb0/0xd0 [33358.173193] sb_edac x86_pkg_temp_thermal coretemp [33358.181704] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33358.181705] kvm_intel [33358.181706] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33358.186488] kvm irqbypass [33358.195435] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33358.197788] i2c_i801 iTCO_wdt [33358.205260] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33358.207965] iTCO_vendor_support [33358.213441] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33358.216492] i2c_smbus crct10dif_pclmul [33358.235231] [33358.235232] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33358.238455] joydev crc32_pclmul [33358.243672] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33358.247503] ipmi_ssif [33358.248996] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33358.256119] ghash_clmulni_intel [33358.259342] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33358.266465] mei_me mei [33358.268823] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33358.275947] lpc_ich sha512_ssse3 [33358.279171] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33358.286296] acpi_ipmi mfd_core [33358.288740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33358.295864] wmi ipmi_si [33358.299174] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33358.307252] ipmi_devintf [33358.310390] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33358.316126] ipmi_msghandler ioatdma [33358.318656] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33358.325781] acpi_pad acpi_power_meter [33358.328398] Call Trace: [33358.328400] [33358.335524] ixgbe igb [33358.339097] ? watchdog_timer_fn+0x1b4/0x220 [33358.346218] mdio crc32c_intel [33358.349964] ? __hrtimer_run_queues+0x1d3/0x360 [33358.352406] nvme [33358.354419] ? hrtimer_interrupt+0x104/0x240 [33358.356775] ptp nvme_core [33358.361039] ? __local_bh_enable+0x50/0x60 [33358.364089] i2c_algo_bit pps_core [33358.368616] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33358.370538] dca [33358.374801] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33358.377505] fuse [33358.381595] [33358.384999] irq event stamp: 187921182 [33358.390219] [33358.390221] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33358.392057] hardirqs last enabled at (187921181): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33358.397017] ? queued_spin_lock_slowpath+0x18/0x2e0 [33358.398939] hardirqs last disabled at (187921182): [] sysvec_apic_timer_interrupt+0xe/0x90 [33358.401037] do_raw_spin_lock+0xb5/0xc0 [33358.404779] softirqs last enabled at (187921180): [] __do_softirq+0x2be/0x371 [33358.406878] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33358.412182] softirqs last disabled at (187921175): [] irq_exit_rcu+0xb0/0xd0 [33358.422429] try_get_cap_refs+0x159/0x880 [ceph] [33358.427297] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33358.437111] __ceph_get_caps+0x4a1/0x760 [ceph] [33358.440937] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33358.449713] ? __file_remove_privs+0x89/0x110 [33358.454492] Workqueue: events delayed_work [ceph] [33358.463090] ? wake_bit_function+0x70/0x70 [33358.467700] [33358.467701] RIP: 0010:lock_acquire+0xf0/0x2b0 [33358.477082] ? ceph_write_iter+0x260/0x770 [ceph] [33358.481602] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33358.489074] ceph_write_iter+0x260/0x770 [ceph] [33358.493425] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33358.498126] ? lock_is_held_type+0xaa/0x120 [33358.502212] [33358.502213] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33358.503706] vfs_write+0x324/0x4f0 [33358.508055] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33358.512755] ksys_pwrite64+0x65/0xa0 [33358.531487] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33358.536013] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33358.541231] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33358.545408] do_syscall_64+0x38/0x80 [33358.546897] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33358.554022] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33358.557420] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33358.564546] RIP: 0033:0x7f48f0eeb93b [33358.568115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33358.575241] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33358.580371] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33358.587493] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33358.591064] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33358.598188] ORIG_RAX: 0000000000000012 [33358.603231] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33358.611310] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33358.614880] Call Trace: [33358.614881] [33358.620619] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33358.639358] ? watchdog_timer_fn+0x1b4/0x220 [33358.646480] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33358.651699] ? __hrtimer_run_queues+0x1d3/0x360 [33358.658821] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33358.662656] ? hrtimer_interrupt+0x104/0x240 [33358.669775] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33358.676901] ? __local_bh_enable+0x50/0x60 [33358.679349] [33358.681359] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33358.734406] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33358.739364] [33358.741461] [33358.743558] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33358.748866] ? lock_acquire+0xf0/0x2b0 [33358.752617] ? igrab+0x43/0x50 [33358.755665] ? lock_release+0xc5/0x260 [33358.759412] _raw_spin_lock+0x30/0x40 [33358.763076] ? igrab+0x1d/0x50 [33358.766125] igrab+0x1d/0x50 [33358.769007] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33358.774153] delayed_work+0x1d8/0x280 [ceph] [33358.778436] ? process_scheduled_works+0x22a/0x510 [33358.783225] process_scheduled_works+0x2a4/0x510 [33358.787841] worker_thread+0x188/0x340 [33358.791591] ? rescuer_thread+0x310/0x310 [33358.795601] kthread+0xf5/0x130 [33358.798740] ? kthread_complete_and_exit+0x20/0x20 [33358.803523] ret_from_fork+0x31/0x40 [33358.807103] ? kthread_complete_and_exit+0x20/0x20 [33358.811888] ret_from_fork_asm+0x11/0x20 [33358.815819] [33386.044121] watchdog: BUG: soft lockup - CPU#0 stuck for 19979s! [dbench:78683] [33386.051418] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33386.056121] watchdog: BUG: soft lockup - CPU#7 stuck for 28976s! [kworker/7:4:78808] [33386.106878] irq event stamp: 57413626 [33386.106879] hardirqs last enabled at (57413625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33386.114608] Modules linked in: [33386.118265] hardirqs last disabled at (57413626): [] sysvec_apic_timer_interrupt+0xe/0x90 [33386.128422] ceph [33386.131473] softirqs last enabled at (57413600): [] __do_softirq+0x2be/0x371 [33386.141197] libceph [33386.143121] softirqs last disabled at (57413595): [] irq_exit_rcu+0xb0/0xd0 [33386.151805] dns_resolver [33386.153990] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33386.162517] fscache [33386.165135] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33386.174080] netfs veth [33386.176264] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33386.183734] nft_chain_nat [33386.186179] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33386.191656] xt_MASQUERADE nf_nat [33386.194359] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33386.213099] nf_conntrack [33386.216408] [33386.216408] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33386.221626] nf_defrag_ipv6 nf_defrag_ipv4 [33386.224243] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33386.225734] nft_compat nf_tables [33386.232859] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33386.236949] nfnetlink bridge [33386.244073] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33386.247383] stp llc [33386.254516] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33386.257479] xfs [33386.264604] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33386.266801] libcrc32c sunrpc [33386.273921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33386.275758] intel_rapl_msr intel_rapl_common [33386.283835] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33386.286800] sb_edac x86_pkg_temp_thermal [33386.292539] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33386.296889] coretemp kvm_intel [33386.304012] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33386.308016] kvm irqbypass [33386.315142] Call Trace: [33386.315143] [33386.318277] i2c_i801 iTCO_wdt [33386.325404] ? watchdog_timer_fn+0x1b4/0x220 [33386.328106] iTCO_vendor_support [33386.330551] ? __hrtimer_run_queues+0x1d3/0x360 [33386.332562] i2c_smbus [33386.335614] ? hrtimer_interrupt+0x104/0x240 [33386.339877] crct10dif_pclmul joydev [33386.343100] ? __local_bh_enable+0x50/0x60 [33386.347624] crc32_pclmul [33386.349984] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33386.354246] ipmi_ssif [33386.357817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33386.361907] ghash_clmulni_intel mei_me [33386.364524] [33386.369741] mei lpc_ich [33386.372099] [33386.377056] sha512_ssse3 [33386.380887] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33386.382985] acpi_ipmi mfd_core [33386.385524] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33386.387622] wmi ipmi_si [33386.390240] do_raw_spin_lock+0xb5/0xc0 [33386.395544] ipmi_devintf ipmi_msghandler [33386.398681] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33386.403552] ioatdma [33386.406084] try_get_cap_refs+0x159/0x880 [ceph] [33386.409913] acpi_pad acpi_power_meter [33386.413921] __ceph_get_caps+0x4a1/0x760 [ceph] [33386.418699] ixgbe igb [33386.420889] ? __file_remove_privs+0x89/0x110 [33386.425496] mdio crc32c_intel [33386.429257] ? wake_bit_function+0x70/0x70 [33386.433782] nvme ptp [33386.436141] ? ceph_write_iter+0x260/0x770 [ceph] [33386.440489] nvme_core i2c_algo_bit [33386.443539] ceph_write_iter+0x260/0x770 [ceph] [33386.447631] pps_core dca [33386.449905] ? lock_is_held_type+0xaa/0x120 [33386.454599] fuse [33386.458086] vfs_write+0x324/0x4f0 [33386.462606] irq event stamp: 188142064 [33386.465228] ksys_pwrite64+0x65/0xa0 [33386.469402] hardirqs last enabled at (188142063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33386.471325] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33386.474722] hardirqs last disabled at (188142064): [] sysvec_apic_timer_interrupt+0xe/0x90 [33386.478468] do_syscall_64+0x38/0x80 [33386.482037] softirqs last enabled at (188142062): [] __do_softirq+0x2be/0x371 [33386.492282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33386.497412] softirqs last disabled at (188142057): [] irq_exit_rcu+0xb0/0xd0 [33386.507225] RIP: 0033:0x7f48f0eeb93b [33386.510795] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33386.519582] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33386.524627] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33386.533224] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33386.536802] Workqueue: events delayed_work [ceph] [33386.546181] ORIG_RAX: 0000000000000012 [33386.546182] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33386.564919] [33386.572389] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33386.577606] RIP: 0010:lock_release+0x8/0x260 [33386.582305] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33386.586136] Code: 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 <41> 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 [33386.593259] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33386.594751] RSP: 0018:ffffc90002117d78 EFLAGS: 00000286 [33386.601874] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33386.606139] [33386.613269] [33386.632000] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [33386.662266] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [33386.669389] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [33386.676529] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33386.683655] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33386.690780] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33386.698857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33386.704593] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33386.711719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33386.718842] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33386.725966] Call Trace: [33386.728410] [33386.730422] ? watchdog_timer_fn+0x1b4/0x220 [33386.734687] ? __hrtimer_run_queues+0x1d3/0x360 [33386.739212] ? hrtimer_interrupt+0x104/0x240 [33386.743472] ? __local_bh_enable+0x50/0x60 [33386.747566] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33386.752807] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33386.757764] [33386.759862] [33386.761961] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33386.767268] ? igrab+0x43/0x50 [33386.770324] ? lock_release+0x8/0x260 [33386.773983] _raw_spin_unlock+0x1b/0x40 [33386.777822] igrab+0x43/0x50 [33386.780697] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33386.785848] delayed_work+0x1d8/0x280 [ceph] [33386.790128] ? process_scheduled_works+0x22a/0x510 [33386.794909] process_scheduled_works+0x2a4/0x510 [33386.799535] worker_thread+0x188/0x340 [33386.803285] ? rescuer_thread+0x310/0x310 [33386.807289] kthread+0xf5/0x130 [33386.810425] ? kthread_complete_and_exit+0x20/0x20 [33386.815210] ret_from_fork+0x31/0x40 [33386.818804] ? kthread_complete_and_exit+0x20/0x20 [33386.823589] ret_from_fork_asm+0x11/0x20 [33386.827531] [33414.044643] watchdog: BUG: soft lockup - CPU#0 stuck for 20005s! [dbench:78683] [33414.051951] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33414.056642] watchdog: BUG: soft lockup - CPU#7 stuck for 29002s! [kworker/7:4:78808] [33414.107399] irq event stamp: 57471862 [33414.115132] Modules linked in: ceph libceph dns_resolver fscache [33414.118787] hardirqs last enabled at (57471861): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33414.124786] netfs veth nft_chain_nat xt_MASQUERADE [33414.134943] hardirqs last disabled at (57471862): [] sysvec_apic_timer_interrupt+0xe/0x90 [33414.134947] softirqs last enabled at (57471834): [] __do_softirq+0x2be/0x371 [33414.139813] nf_nat [33414.149537] softirqs last disabled at (57471829): [] irq_exit_rcu+0xb0/0xd0 [33414.158223] nf_conntrack nf_defrag_ipv6 [33414.160320] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33414.168829] nf_defrag_ipv4 nft_compat [33414.172748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33414.181692] nf_tables [33414.185437] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33414.192907] nfnetlink [33414.195266] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33414.200742] bridge [33414.203097] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33414.221837] stp [33414.223933] [33414.223934] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33414.229150] llc xfs [33414.230987] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33414.232478] libcrc32c sunrpc [33414.239604] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33414.241788] intel_rapl_msr [33414.248912] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33414.251876] intel_rapl_common [33414.259000] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33414.261790] sb_edac [33414.268916] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33414.271966] x86_pkg_temp_thermal [33414.279089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33414.281273] coretemp [33414.289351] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33414.292663] kvm_intel [33414.298400] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33414.300669] kvm [33414.307794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33414.310151] irqbypass i2c_i801 [33414.317275] Call Trace: [33414.319113] iTCO_wdt [33414.326239] [33414.329374] iTCO_vendor_support i2c_smbus [33414.331821] ? watchdog_timer_fn+0x1b4/0x220 [33414.334088] crct10dif_pclmul [33414.336101] ? __hrtimer_run_queues+0x1d3/0x360 [33414.340190] joydev crc32_pclmul [33414.344457] ? hrtimer_interrupt+0x104/0x240 [33414.347418] ipmi_ssif ghash_clmulni_intel [33414.351943] ? __local_bh_enable+0x50/0x60 [33414.355166] mei_me [33414.359435] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33414.363522] mei [33414.367614] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33414.369709] lpc_ich sha512_ssse3 [33414.374928] [33414.376763] acpi_ipmi mfd_core [33414.381722] [33414.385043] wmi ipmi_si [33414.387140] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33414.390276] ipmi_devintf [33414.392377] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33414.394903] ipmi_msghandler ioatdma [33414.400212] do_raw_spin_lock+0xb5/0xc0 [33414.402825] acpi_pad acpi_power_meter [33414.407698] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33414.411266] ixgbe igb [33414.415101] try_get_cap_refs+0x159/0x880 [ceph] [33414.418842] mdio crc32c_intel [33414.423632] __ceph_get_caps+0x4a1/0x760 [ceph] [33414.425985] nvme [33414.430599] ? __file_remove_privs+0x89/0x110 [33414.433645] ptp [33414.438168] ? wake_bit_function+0x70/0x70 [33414.440093] nvme_core i2c_algo_bit [33414.444446] ? ceph_write_iter+0x260/0x770 [ceph] [33414.446282] pps_core [33414.450371] ceph_write_iter+0x260/0x770 [ceph] [33414.453857] dca fuse [33414.458557] ? lock_is_held_type+0xaa/0x120 [33414.460823] [33414.460824] irq event stamp: 188362830 [33414.465350] vfs_write+0x324/0x4f0 [33414.467619] hardirqs last enabled at (188362829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33414.471800] ksys_pwrite64+0x65/0xa0 [33414.473286] hardirqs last disabled at (188362830): [] sysvec_apic_timer_interrupt+0xe/0x90 [33414.477044] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33414.480437] softirqs last enabled at (188362828): [] __do_softirq+0x2be/0x371 [33414.490681] do_syscall_64+0x38/0x80 [33414.494253] softirqs last disabled at (188362823): [] irq_exit_rcu+0xb0/0xd0 [33414.504065] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33414.509192] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33414.517967] RIP: 0033:0x7f48f0eeb93b [33414.521536] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33414.530133] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33414.535178] Workqueue: events delayed_work [ceph] [33414.544555] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33414.548124] [33414.548125] RIP: 0010:do_raw_spin_lock+0xd/0xc0 [33414.555595] ORIG_RAX: 0000000000000012 [33414.574333] Code: 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 66 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 <48> 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 [33414.579046] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33414.584267] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [33414.585755] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33414.590280] [33414.590280] RAX: 00000000dead4ead RBX: ffff8881a1d34478 RCX: 000000005553a634 [33414.594111] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33414.612848] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [33414.619974] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33414.625191] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [33414.632313] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33414.633823] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33414.640952] [33414.648072] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33414.693009] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33414.701086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33414.706824] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33414.713949] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33414.721072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33414.728197] Call Trace: [33414.730641] [33414.732653] ? watchdog_timer_fn+0x1b4/0x220 [33414.736916] ? __hrtimer_run_queues+0x1d3/0x360 [33414.741444] ? hrtimer_interrupt+0x104/0x240 [33414.745712] ? __local_bh_enable+0x50/0x60 [33414.749805] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33414.755046] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33414.760003] [33414.762102] [33414.764200] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33414.769516] ? do_raw_spin_lock+0xd/0xc0 [33414.773439] igrab+0x1d/0x50 [33414.776340] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33414.781491] delayed_work+0x1d8/0x280 [ceph] [33414.785773] ? process_scheduled_works+0x22a/0x510 [33414.790563] process_scheduled_works+0x2a4/0x510 [33414.795177] worker_thread+0x188/0x340 [33414.798928] ? rescuer_thread+0x310/0x310 [33414.802938] kthread+0xf5/0x130 [33414.806076] ? kthread_complete_and_exit+0x20/0x20 [33414.810862] ret_from_fork+0x31/0x40 [33414.814441] ? kthread_complete_and_exit+0x20/0x20 [33414.819225] ret_from_fork_asm+0x11/0x20 [33414.823158] [33442.045163] watchdog: BUG: soft lockup - CPU#0 stuck for 20031s! [dbench:78683] [33442.052463] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33442.057163] watchdog: BUG: soft lockup - CPU#7 stuck for 29028s! [kworker/7:4:78808] [33442.107905] irq event stamp: 57530042 [33442.107907] hardirqs last enabled at (57530041): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33442.115635] Modules linked in: [33442.119293] hardirqs last disabled at (57530042): [] sysvec_apic_timer_interrupt+0xe/0x90 [33442.129451] ceph [33442.132500] softirqs last enabled at (57530014): [] __do_softirq+0x2be/0x371 [33442.142226] libceph [33442.144149] softirqs last disabled at (57530009): [] irq_exit_rcu+0xb0/0xd0 [33442.152849] dns_resolver [33442.155034] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33442.163559] fscache netfs [33442.166180] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33442.175125] veth nft_chain_nat [33442.177843] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33442.185316] xt_MASQUERADE nf_nat [33442.188455] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33442.193931] nf_conntrack [33442.197242] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33442.215980] nf_defrag_ipv6 [33442.218598] [33442.218599] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33442.223814] nf_defrag_ipv4 nft_compat [33442.226622] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33442.228114] nf_tables nfnetlink [33442.235238] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33442.238981] bridge [33442.246106] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33442.249330] stp llc [33442.256454] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33442.258561] xfs libcrc32c [33442.265685] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33442.267868] sunrpc intel_rapl_msr [33442.274993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33442.277696] intel_rapl_common sb_edac [33442.285774] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33442.289171] x86_pkg_temp_thermal [33442.294909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33442.298652] coretemp kvm_intel [33442.305776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33442.309089] kvm irqbypass [33442.316213] Call Trace: [33442.316214] [33442.319350] i2c_i801 iTCO_wdt iTCO_vendor_support [33442.326474] ? watchdog_timer_fn+0x1b4/0x220 [33442.329177] i2c_smbus crct10dif_pclmul [33442.331621] ? __hrtimer_run_queues+0x1d3/0x360 [33442.333631] joydev crc32_pclmul [33442.338419] ? hrtimer_interrupt+0x104/0x240 [33442.342679] ipmi_ssif ghash_clmulni_intel [33442.346510] ? __local_bh_enable+0x50/0x60 [33442.351036] mei_me mei [33442.354263] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33442.358522] lpc_ich sha512_ssse3 [33442.362615] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33442.366705] acpi_ipmi mfd_core [33442.369149] [33442.374367] wmi ipmi_si [33442.377678] [33442.382634] ipmi_devintf [33442.385774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33442.387869] ipmi_msghandler ioatdma [33442.390405] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33442.392497] acpi_pad acpi_power_meter [33442.395117] do_raw_spin_lock+0xb5/0xc0 [33442.400419] ixgbe igb mdio [33442.403991] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33442.408859] crc32c_intel nvme [33442.412608] try_get_cap_refs+0x159/0x880 [ceph] [33442.416434] ptp nvme_core [33442.419230] __ceph_get_caps+0x4a1/0x760 [ceph] [33442.424011] i2c_algo_bit pps_core [33442.427066] ? __file_remove_privs+0x89/0x110 [33442.431671] dca fuse [33442.434376] ? wake_bit_function+0x70/0x70 [33442.438900] [33442.438901] irq event stamp: 188583576 [33442.442300] ? ceph_write_iter+0x260/0x770 [ceph] [33442.446647] hardirqs last enabled at (188583575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33442.448919] ceph_write_iter+0x260/0x770 [ceph] [33442.453008] hardirqs last disabled at (188583576): [] sysvec_apic_timer_interrupt+0xe/0x90 [33442.454502] ? lock_is_held_type+0xaa/0x120 [33442.458245] softirqs last enabled at (188583574): [] __do_softirq+0x2be/0x371 [33442.462943] vfs_write+0x324/0x4f0 [33442.473185] softirqs last disabled at (188583569): [] irq_exit_rcu+0xb0/0xd0 [33442.477716] ksys_pwrite64+0x65/0xa0 [33442.487521] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33442.491699] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33442.500470] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33442.500471] Workqueue: events delayed_work [ceph] [33442.503868] do_syscall_64+0x38/0x80 [33442.512465] [33442.516034] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33442.525412] RIP: 0010:lock_acquire+0xf0/0x2b0 [33442.530544] RIP: 0033:0x7f48f0eeb93b [33442.538030] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33442.542729] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33442.546299] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33442.547789] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33442.552834] [33442.552834] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33442.557202] ORIG_RAX: 0000000000000012 [33442.560772] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33442.560773] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33442.579511] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33442.598247] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33442.603466] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33442.608683] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33442.610174] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33442.617297] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33442.621129] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33442.628253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33442.635375] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33442.642500] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33442.649632] [33442.656749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33442.715493] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33442.722616] Call Trace: [33442.725062] [33442.727074] ? watchdog_timer_fn+0x1b4/0x220 [33442.731337] ? __hrtimer_run_queues+0x1d3/0x360 [33442.735863] ? hrtimer_interrupt+0x104/0x240 [33442.740133] ? __local_bh_enable+0x50/0x60 [33442.744227] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33442.749451] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33442.754406] [33442.756505] [33442.758602] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33442.763909] ? lock_acquire+0xf0/0x2b0 [33442.767660] ? igrab+0x43/0x50 [33442.770711] ? lock_release+0xc5/0x260 [33442.774456] _raw_spin_lock+0x30/0x40 [33442.778121] ? igrab+0x1d/0x50 [33442.781171] igrab+0x1d/0x50 [33442.784049] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33442.789199] delayed_work+0x1d8/0x280 [ceph] [33442.793479] ? process_scheduled_works+0x22a/0x510 [33442.798270] process_scheduled_works+0x2a4/0x510 [33442.802886] worker_thread+0x188/0x340 [33442.806634] ? rescuer_thread+0x310/0x310 [33442.810636] kthread+0xf5/0x130 [33442.813776] ? kthread_complete_and_exit+0x20/0x20 [33442.818574] ret_from_fork+0x31/0x40 [33442.822146] ? kthread_complete_and_exit+0x20/0x20 [33442.826931] ret_from_fork_asm+0x11/0x20 [33442.830857] [33470.045686] watchdog: BUG: soft lockup - CPU#0 stuck for 20057s! [dbench:78683] [33470.052988] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33470.057685] watchdog: BUG: soft lockup - CPU#7 stuck for 29054s! [kworker/7:4:78808] [33470.108436] irq event stamp: 57588202 [33470.108438] hardirqs last enabled at (57588201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33470.116168] Modules linked in: [33470.119823] hardirqs last disabled at (57588202): [] sysvec_apic_timer_interrupt+0xe/0x90 [33470.129981] ceph [33470.133031] softirqs last enabled at (57588174): [] __do_softirq+0x2be/0x371 [33470.142756] libceph [33470.144680] softirqs last disabled at (57588169): [] irq_exit_rcu+0xb0/0xd0 [33470.153364] dns_resolver [33470.155549] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33470.164077] fscache [33470.166694] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33470.175638] netfs [33470.177823] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33470.185293] veth nft_chain_nat [33470.187304] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33470.192781] xt_MASQUERADE nf_nat [33470.195919] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33470.214657] nf_conntrack nf_defrag_ipv6 [33470.217968] [33470.217969] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33470.223185] nf_defrag_ipv4 nft_compat [33470.227101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33470.228593] nf_tables [33470.235716] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33470.239462] nfnetlink bridge [33470.246586] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33470.248942] stp llc [33470.256080] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33470.259048] xfs [33470.266173] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33470.268360] libcrc32c [33470.275480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33470.277319] sunrpc [33470.285394] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33470.287752] intel_rapl_msr [33470.293491] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33470.295587] intel_rapl_common [33470.302712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33470.305501] sb_edac x86_pkg_temp_thermal [33470.312625] Call Trace: [33470.312627] [33470.315678] coretemp kvm_intel [33470.322803] ? watchdog_timer_fn+0x1b4/0x220 [33470.326805] kvm irqbypass [33470.329250] ? __hrtimer_run_queues+0x1d3/0x360 [33470.331259] i2c_i801 iTCO_wdt [33470.334402] ? hrtimer_interrupt+0x104/0x240 [33470.338661] iTCO_vendor_support [33470.341366] ? __local_bh_enable+0x50/0x60 [33470.345891] i2c_smbus [33470.348943] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33470.353203] crct10dif_pclmul joydev [33470.356429] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33470.360520] crc32_pclmul ipmi_ssif [33470.362878] [33470.368094] ghash_clmulni_intel [33470.371665] [33470.376623] mei_me mei [33470.380107] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33470.382203] lpc_ich sha512_ssse3 [33470.385431] ? queued_spin_lock_slowpath+0x18/0x2e0 [33470.387525] acpi_ipmi [33470.389971] do_raw_spin_lock+0xb5/0xc0 [33470.395274] mfd_core [33470.398587] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33470.403455] wmi ipmi_si [33470.405815] try_get_cap_refs+0x159/0x880 [ceph] [33470.409642] ipmi_devintf ipmi_msghandler [33470.411919] __ceph_get_caps+0x4a1/0x760 [ceph] [33470.416697] ioatdma acpi_pad [33470.419233] ? __file_remove_privs+0x89/0x110 [33470.423840] acpi_power_meter ixgbe [33470.427845] ? wake_bit_function+0x70/0x70 [33470.432382] igb mdio [33470.435368] ? ceph_write_iter+0x260/0x770 [ceph] [33470.439717] crc32c_intel [33470.443201] ceph_write_iter+0x260/0x770 [ceph] [33470.447293] nvme [33470.449566] ? lock_is_held_type+0xaa/0x120 [33470.454259] ptp nvme_core [33470.456880] vfs_write+0x324/0x4f0 [33470.461401] i2c_algo_bit [33470.463329] ksys_pwrite64+0x65/0xa0 [33470.467504] pps_core dca [33470.470208] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33470.473605] fuse [33470.476222] do_syscall_64+0x38/0x80 [33470.479791] [33470.482410] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33470.487540] irq event stamp: 188804332 [33470.489464] RIP: 0033:0x7f48f0eeb93b [33470.493035] hardirqs last enabled at (188804331): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33470.494527] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33470.499571] hardirqs last disabled at (188804332): [] sysvec_apic_timer_interrupt+0xe/0x90 [33470.503313] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33470.506886] softirqs last enabled at (188804330): [] __do_softirq+0x2be/0x371 [33470.517130] ORIG_RAX: 0000000000000012 [33470.535867] softirqs last disabled at (188804325): [] irq_exit_rcu+0xb0/0xd0 [33470.545677] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33470.550894] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33470.559667] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33470.563498] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33470.572095] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33470.579218] Workqueue: events delayed_work [ceph] [33470.588596] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33470.595719] [33470.595720] RIP: 0010:lock_acquire+0xf0/0x2b0 [33470.603190] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33470.610315] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33470.615017] [33470.622136] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33470.661242] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33470.668371] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33470.675499] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33470.682621] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33470.689747] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33470.696870] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33470.704948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33470.710685] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33470.717810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33470.724935] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33470.732057] Call Trace: [33470.734503] [33470.736513] ? watchdog_timer_fn+0x1b4/0x220 [33470.740780] ? __hrtimer_run_queues+0x1d3/0x360 [33470.745313] ? hrtimer_interrupt+0x104/0x240 [33470.749582] ? __local_bh_enable+0x50/0x60 [33470.753676] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33470.758899] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33470.763857] [33470.765954] [33470.768051] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33470.773376] ? lock_acquire+0xf0/0x2b0 [33470.777127] ? igrab+0x43/0x50 [33470.780178] ? lock_release+0xc5/0x260 [33470.783922] _raw_spin_lock+0x30/0x40 [33470.787586] ? igrab+0x1d/0x50 [33470.790638] igrab+0x1d/0x50 [33470.793516] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33470.798665] delayed_work+0x1d8/0x280 [ceph] [33470.802948] ? process_scheduled_works+0x22a/0x510 [33470.807738] process_scheduled_works+0x2a4/0x510 [33470.812351] worker_thread+0x188/0x340 [33470.816102] ? rescuer_thread+0x310/0x310 [33470.820105] kthread+0xf5/0x130 [33470.823251] ? kthread_complete_and_exit+0x20/0x20 [33470.828035] ret_from_fork+0x31/0x40 [33470.831615] ? kthread_complete_and_exit+0x20/0x20 [33470.836398] ret_from_fork_asm+0x11/0x20 [33470.840322] [33498.046206] watchdog: BUG: soft lockup - CPU#0 stuck for 20083s! [dbench:78683] [33498.053510] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33498.058206] watchdog: BUG: soft lockup - CPU#7 stuck for 29080s! [kworker/7:4:78808] [33498.108950] irq event stamp: 57646412 [33498.108952] hardirqs last enabled at (57646411): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33498.116682] Modules linked in: [33498.120339] hardirqs last disabled at (57646412): [] sysvec_apic_timer_interrupt+0xe/0x90 [33498.130495] ceph [33498.133545] softirqs last enabled at (57646386): [] __do_softirq+0x2be/0x371 [33498.143272] libceph [33498.145194] softirqs last disabled at (57646381): [] irq_exit_rcu+0xb0/0xd0 [33498.153884] dns_resolver [33498.156071] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33498.164599] fscache [33498.167218] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33498.176160] netfs veth [33498.178345] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33498.185816] nft_chain_nat [33498.188261] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33498.193737] xt_MASQUERADE nf_nat [33498.196443] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33498.215178] nf_conntrack nf_defrag_ipv6 [33498.218489] [33498.218489] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33498.223707] nf_defrag_ipv4 [33498.227625] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33498.229116] nft_compat nf_tables [33498.236240] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33498.239030] nfnetlink [33498.246154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33498.249464] bridge stp [33498.256602] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33498.258956] llc [33498.266079] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33498.268523] xfs libcrc32c [33498.275647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33498.277485] sunrpc intel_rapl_msr [33498.285563] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33498.288267] intel_rapl_common [33498.294005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33498.297400] sb_edac x86_pkg_temp_thermal [33498.304526] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33498.307575] coretemp kvm_intel [33498.314701] Call Trace: [33498.318704] kvm irqbypass [33498.325828] [33498.328966] i2c_i801 [33498.331413] ? watchdog_timer_fn+0x1b4/0x220 [33498.334115] iTCO_wdt iTCO_vendor_support [33498.336125] ? __hrtimer_run_queues+0x1d3/0x360 [33498.338396] i2c_smbus crct10dif_pclmul [33498.342579] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [33498.342582] rcu: 0-....: (21074664 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10549122 [33498.342604] rcu: 7-....: (20935914 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/20934580 fqs=10549122 [33498.342607] rcu: (detected by 3, t=21561334 jiffies, g=1309609, q=612916 ncpus=8) [33498.342610] Sending NMI from CPU 3 to CPUs 0: [33498.342614] NMI backtrace for cpu 0 [33498.342615] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33498.342616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33498.342617] RIP: 0010:delay_tsc+0x4d/0x90 [33498.342619] Code: c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e 65 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 48 89 c2 4c 29 c2 48 39 fa 72 c8 65 ff [33498.342620] RSP: 0018:ffffc90000003ba8 EFLAGS: 00000046 [33498.342622] RAX: 00000000b0dbc1b9 RBX: 0000000000000000 RCX: 0000000000000000 [33498.342622] RDX: 0000000000006c3b RSI: 0000000000000000 RDI: 0000000000000dac [33498.342623] RBP: 00000000000026c4 R08: 00006c3bb0dbbc92 R09: 0000000000000000 [33498.342624] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000060 [33498.342624] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [33498.342625] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33498.342626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33498.342627] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33498.342628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33498.342628] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33498.342629] Call Trace: [33498.342630] [33498.342630] ? nmi_cpu_backtrace+0xad/0x120 [33498.342634] ? nmi_cpu_backtrace_handler+0x11/0x20 [33498.342635] ? nmi_handle+0xd4/0x260 [33498.342639] ? default_do_nmi+0x49/0xf0 [33498.342641] ? exc_nmi+0x11e/0x150 [33498.342642] ? end_repeat_nmi+0x16/0x67 [33498.342646] ? delay_tsc+0x4d/0x90 [33498.342648] ? delay_tsc+0x4d/0x90 [33498.342650] ? delay_tsc+0x4d/0x90 [33498.342652] [33498.342653] [33498.342653] wait_for_lsr+0x27/0x70 [33498.342657] wait_for_xmitr+0xf/0x60 [33498.342658] serial8250_console_write+0x22a/0x3f0 [33498.342661] ? lock_release+0xc5/0x260 [33498.342664] console_flush_all+0x227/0x5a0 [33498.342666] ? console_flush_all+0x1ff/0x5a0 [33498.342668] ? console_flush_all+0x1ff/0x5a0 [33498.342670] console_unlock+0x7b/0x120 [33498.342673] vprintk_emit+0x246/0x340 [33498.342675] ? __hrtimer_run_queues+0x1d3/0x360 [33498.342676] ? __hrtimer_run_queues+0x1d3/0x360 [33498.342677] _printk+0x5c/0x80 [33498.342680] ? __hrtimer_run_queues+0x1d3/0x360 [33498.342681] show_trace_log_lvl+0x1ac/0x3f0 [33498.342686] ? do_raw_spin_lock+0xb5/0xc0 [33498.342688] ? touch_softlockup_watchdog+0x20/0x20 [33498.342690] watchdog_timer_fn+0x1b4/0x220 [33498.342692] __hrtimer_run_queues+0x1d3/0x360 [33498.342694] hrtimer_interrupt+0x104/0x240 [33498.342695] ? __local_bh_enable+0x50/0x60 [33498.342699] __sysvec_apic_timer_interrupt+0x61/0x160 [33498.342701] sysvec_apic_timer_interrupt+0x6d/0x90 [33498.342703] [33498.342703] [33498.342704] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33498.342706] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33498.342709] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33498.342710] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33498.342711] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33498.342711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33498.342712] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33498.342713] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33498.342713] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33498.342717] do_raw_spin_lock+0xb5/0xc0 [33498.342719] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33498.342734] try_get_cap_refs+0x159/0x880 [ceph] [33498.342751] __ceph_get_caps+0x4a1/0x760 [ceph] [33498.342768] ? __file_remove_privs+0x89/0x110 [33498.342770] ? wake_bit_function+0x70/0x70 [33498.342773] ? ceph_write_iter+0x260/0x770 [ceph] [33498.342785] ceph_write_iter+0x260/0x770 [ceph] [33498.342799] ? lock_is_held_type+0xaa/0x120 [33498.342802] vfs_write+0x324/0x4f0 [33498.342806] ksys_pwrite64+0x65/0xa0 [33498.342808] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33498.342811] do_syscall_64+0x38/0x80 [33498.342813] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33498.342815] RIP: 0033:0x7f48f0eeb93b [33498.342817] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33498.342818] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [33498.342819] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33498.342820] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33498.342821] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33498.342821] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33498.342822] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33498.342825] [33498.342835] ? hrtimer_interrupt+0x104/0x240 [33498.343612] Sending NMI from CPU 3 to CPUs 7: [33498.343617] NMI backtrace for cpu 7 [33498.343618] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33498.343620] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33498.343621] Workqueue: events delayed_work [ceph] [33498.343642] RIP: 0010:io_serial_in+0x19/0x20 [33498.343645] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [33498.343647] RSP: 0018:ffffc90000264c98 EFLAGS: 00000002 [33498.343648] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [33498.343649] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [33498.343651] RBP: 00000000000025fc R08: 00006c3bb1112e9f R09: 0000000000000007 [33498.343652] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [33498.343653] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [33498.343654] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33498.343656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33498.343657] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33498.343659] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33498.343660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33498.343661] Call Trace: [33498.343662] [33498.343663] ? nmi_cpu_backtrace+0xad/0x120 [33498.343667] ? nmi_cpu_backtrace_handler+0x11/0x20 [33498.343669] ? nmi_handle+0xd4/0x260 [33498.343674] ? default_do_nmi+0x49/0xf0 [33498.343676] ? exc_nmi+0x11e/0x150 [33498.343679] ? end_repeat_nmi+0x16/0x67 [33498.343685] ? io_serial_in+0x19/0x20 [33498.343688] ? io_serial_in+0x19/0x20 [33498.343690] ? io_serial_in+0x19/0x20 [33498.343693] [33498.343694] [33498.343694] wait_for_lsr+0x45/0x70 [33498.343698] serial8250_console_write+0x341/0x3f0 [33498.343701] ? lock_release+0xc5/0x260 [33498.343705] console_flush_all+0x227/0x5a0 [33498.343708] ? console_flush_all+0x1ff/0x5a0 [33498.343709] ? console_flush_all+0x1ff/0x5a0 [33498.343714] console_unlock+0x7b/0x120 [33498.343717] vprintk_emit+0x246/0x340 [33498.343720] ? irq_work_queue+0x2c/0x50 [33498.343725] ? touch_softlockup_watchdog+0x20/0x20 [33498.343727] _printk+0x5c/0x80 [33498.343732] print_modules+0x6e/0xe0 [33498.343735] watchdog_timer_fn+0x19f/0x220 [33498.343738] __hrtimer_run_queues+0x1d3/0x360 [33498.343742] hrtimer_interrupt+0x104/0x240 [33498.343744] ? __local_bh_enable+0x50/0x60 [33498.343749] __sysvec_apic_timer_interrupt+0x61/0x160 [33498.343752] sysvec_apic_timer_interrupt+0x6d/0x90 [33498.343755] [33498.343755] [33498.343756] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33498.343759] RIP: 0010:lock_acquire+0xf0/0x2b0 [33498.343761] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33498.343763] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33498.343764] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33498.343765] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33498.343767] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33498.343768] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33498.343769] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33498.343775] ? igrab+0x43/0x50 [33498.343777] ? lock_release+0xc5/0x260 [33498.343780] _raw_spin_lock+0x30/0x40 [33498.343783] ? igrab+0x1d/0x50 [33498.343785] igrab+0x1d/0x50 [33498.343787] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33498.343807] delayed_work+0x1d8/0x280 [ceph] [33498.343826] ? process_scheduled_works+0x22a/0x510 [33498.343829] process_scheduled_works+0x2a4/0x510 [33498.343836] worker_thread+0x188/0x340 [33498.343839] ? rescuer_thread+0x310/0x310 [33498.343842] kthread+0xf5/0x130 [33498.343845] ? kthread_complete_and_exit+0x20/0x20 [33498.343849] ret_from_fork+0x31/0x40 [33498.343851] ? kthread_complete_and_exit+0x20/0x20 [33498.343854] ret_from_fork_asm+0x11/0x20 [33498.343861] [33498.346837] joydev crc32_pclmul [33498.351360] ? __local_bh_enable+0x50/0x60 [33498.355191] ipmi_ssif ghash_clmulni_intel [33498.361279] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33498.371780] mei_me mei [33498.382458] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33498.390016] lpc_ich sha512_ssse3 [33498.394367] [33498.397850] acpi_ipmi mfd_core wmi [33498.406794] [33498.414264] ipmi_si ipmi_devintf [33498.418271] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33498.437008] ipmi_msghandler [33498.442227] ? queued_spin_lock_slowpath+0x16/0x2e0 [33498.449349] ioatdma acpi_pad [33498.456475] do_raw_spin_lock+0xb5/0xc0 [33498.463607] acpi_power_meter ixgbe [33498.470731] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33498.477853] igb mdio [33498.485933] try_get_cap_refs+0x159/0x880 [ceph] [33498.491669] crc32c_intel nvme [33498.498798] __ceph_get_caps+0x4a1/0x760 [ceph] [33498.505917] ptp nvme_core [33498.513044] ? __file_remove_privs+0x89/0x110 [33498.515485] i2c_algo_bit pps_core [33498.517496] ? wake_bit_function+0x70/0x70 [33498.521672] dca [33498.526461] ? ceph_write_iter+0x260/0x770 [ceph] [33498.530027] fuse [33498.533860] ceph_write_iter+0x260/0x770 [ceph] [33498.537256] irq event stamp: 189024920 [33498.541091] ? lock_is_held_type+0xaa/0x120 [33498.544485] hardirqs last enabled at (189024919): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33498.547901] vfs_write+0x324/0x4f0 [33498.551296] hardirqs last disabled at (189024920): [] sysvec_apic_timer_interrupt+0xe/0x90 [33498.553397] ksys_pwrite64+0x65/0xa0 [33498.555404] softirqs last enabled at (189024918): [] __do_softirq+0x2be/0x371 [33498.558890] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33498.562458] softirqs last disabled at (189024913): [] irq_exit_rcu+0xb0/0xd0 [33498.567158] do_syscall_64+0x38/0x80 [33498.570901] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33498.574993] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33498.579256] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33498.583520] RIP: 0033:0x7f48f0eeb93b [33498.587263] Workqueue: events delayed_work [ceph] [33498.590921] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33498.595446] [33498.595447] RIP: 0010:lock_acquire+0xf0/0x2b0 [33498.599970] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33498.603019] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33498.607543] ORIG_RAX: 0000000000000012 [33498.611722] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33498.615726] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33498.620510] [33498.620511] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33498.624613] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33498.628968] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33498.633058] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33498.637150] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33498.642193] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33498.646978] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33498.649075] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33498.651174] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33498.656308] [33498.661782] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33499.577562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33499.583298] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33499.590422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33499.597546] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33499.604670] Call Trace: [33499.607116] [33499.609128] ? watchdog_timer_fn+0x1b4/0x220 [33499.613392] ? __hrtimer_run_queues+0x1d3/0x360 [33499.617930] ? hrtimer_interrupt+0x104/0x240 [33499.622195] ? __local_bh_enable+0x50/0x60 [33499.626291] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33499.631514] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33499.636469] [33499.638568] [33499.640665] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33499.645972] ? lock_acquire+0xf0/0x2b0 [33499.649723] ? igrab+0x43/0x50 [33499.652774] ? lock_release+0xc5/0x260 [33499.656518] _raw_spin_lock+0x30/0x40 [33499.660173] ? igrab+0x1d/0x50 [33499.663224] igrab+0x1d/0x50 [33499.666103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33499.671252] delayed_work+0x1d8/0x280 [ceph] [33499.675535] ? process_scheduled_works+0x22a/0x510 [33499.680324] process_scheduled_works+0x2a4/0x510 [33499.684939] worker_thread+0x188/0x340 [33499.688691] ? rescuer_thread+0x310/0x310 [33499.692701] kthread+0xf5/0x130 [33499.695838] ? kthread_complete_and_exit+0x20/0x20 [33499.700634] ret_from_fork+0x31/0x40 [33499.704209] ? kthread_complete_and_exit+0x20/0x20 [33499.708993] ret_from_fork_asm+0x11/0x20 [33499.712931] [33526.046725] watchdog: BUG: soft lockup - CPU#0 stuck for 20109s! [dbench:78683] [33526.054031] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33526.058724] watchdog: BUG: soft lockup - CPU#7 stuck for 29106s! [kworker/7:4:78808] [33526.109481] irq event stamp: 57704544 [33526.109483] hardirqs last enabled at (57704543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33526.117213] Modules linked in: [33526.120870] hardirqs last disabled at (57704544): [] sysvec_apic_timer_interrupt+0xe/0x90 [33526.131026] ceph [33526.134077] softirqs last enabled at (57704516): [] __do_softirq+0x2be/0x371 [33526.143802] libceph [33526.145725] softirqs last disabled at (57704511): [] irq_exit_rcu+0xb0/0xd0 [33526.154410] dns_resolver [33526.156595] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33526.165120] fscache netfs [33526.167740] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33526.176683] veth nft_chain_nat [33526.179401] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33526.186867] xt_MASQUERADE nf_nat [33526.190005] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33526.195482] nf_conntrack nf_defrag_ipv6 [33526.198792] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33526.217530] nf_defrag_ipv4 [33526.221447] [33526.221448] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33526.226666] nft_compat nf_tables [33526.229457] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33526.230946] nfnetlink bridge [33526.238070] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33526.241398] stp llc [33526.248523] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33526.251486] xfs libcrc32c [33526.258612] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33526.260796] sunrpc [33526.267920] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33526.270622] intel_rapl_msr [33526.277747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33526.279846] intel_rapl_common sb_edac [33526.287921] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33526.290712] x86_pkg_temp_thermal coretemp [33526.296450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33526.300194] kvm_intel kvm [33526.307318] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33526.311410] irqbypass [33526.318534] Call Trace: [33526.318536] [33526.321237] i2c_i801 iTCO_wdt [33526.328365] ? watchdog_timer_fn+0x1b4/0x220 [33526.330720] iTCO_vendor_support [33526.333164] ? __hrtimer_run_queues+0x1d3/0x360 [33526.335173] i2c_smbus [33526.338228] ? hrtimer_interrupt+0x104/0x240 [33526.342489] crct10dif_pclmul joydev [33526.345714] ? __local_bh_enable+0x50/0x60 [33526.350238] crc32_pclmul ipmi_ssif [33526.352597] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33526.356858] ghash_clmulni_intel mei_me [33526.360429] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33526.364519] mei lpc_ich [33526.368005] [33526.373221] sha512_ssse3 acpi_ipmi [33526.377052] [33526.377054] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33526.382009] mfd_core wmi [33526.384545] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33526.386638] ipmi_si [33526.390126] do_raw_spin_lock+0xb5/0xc0 [33526.392218] ipmi_devintf ipmi_msghandler [33526.397525] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33526.400141] ioatdma acpi_pad [33526.405013] try_get_cap_refs+0x159/0x880 [ceph] [33526.407195] acpi_power_meter ixgbe [33526.411032] __ceph_get_caps+0x4a1/0x760 [ceph] [33526.415031] igb mdio [33526.419818] ? __file_remove_privs+0x89/0x110 [33526.422779] crc32c_intel [33526.427405] ? wake_bit_function+0x70/0x70 [33526.430891] nvme [33526.435417] ? ceph_write_iter+0x260/0x770 [ceph] [33526.437686] ptp nvme_core [33526.442037] ceph_write_iter+0x260/0x770 [ceph] [33526.444653] i2c_algo_bit pps_core [33526.448748] ? lock_is_held_type+0xaa/0x120 [33526.450667] dca fuse [33526.455367] vfs_write+0x324/0x4f0 [33526.458070] [33526.458071] irq event stamp: 189238614 [33526.462597] ksys_pwrite64+0x65/0xa0 [33526.465991] hardirqs last enabled at (189238613): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33526.470168] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33526.472439] hardirqs last disabled at (189238614): [] sysvec_apic_timer_interrupt+0xe/0x90 [33526.475839] do_syscall_64+0x38/0x80 [33526.477328] softirqs last enabled at (189238612): [] __do_softirq+0x2be/0x371 [33526.481072] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33526.484642] softirqs last disabled at (189238607): [] irq_exit_rcu+0xb0/0xd0 [33526.494888] RIP: 0033:0x7f48f0eeb93b [33526.500017] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33526.509829] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33526.513399] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33526.522187] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33526.527231] Workqueue: events delayed_work [ceph] [33526.535828] ORIG_RAX: 0000000000000012 [33526.539406] [33526.539407] RIP: 0010:lock_acquire+0xf0/0x2b0 [33526.548786] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33526.567524] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33526.574994] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33526.580211] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33526.584909] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33526.588740] [33526.588741] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33526.590230] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33526.594582] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33526.601704] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33526.620441] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33526.627572] [33526.632785] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33526.686344] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33526.693470] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33526.701547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33526.707283] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33526.714412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33526.721533] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33526.728658] Call Trace: [33526.731101] [33526.733130] ? watchdog_timer_fn+0x1b4/0x220 [33526.737411] ? __hrtimer_run_queues+0x1d3/0x360 [33526.741939] ? hrtimer_interrupt+0x104/0x240 [33526.746207] ? __local_bh_enable+0x50/0x60 [33526.750301] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33526.755524] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33526.760483] [33526.762578] [33526.764676] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33526.769985] ? lock_acquire+0xf0/0x2b0 [33526.773734] ? igrab+0x43/0x50 [33526.776783] ? lock_release+0xc5/0x260 [33526.780530] _raw_spin_lock+0x30/0x40 [33526.784194] ? igrab+0x1d/0x50 [33526.787244] igrab+0x1d/0x50 [33526.790133] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33526.795282] delayed_work+0x1d8/0x280 [ceph] [33526.799562] ? process_scheduled_works+0x22a/0x510 [33526.804353] process_scheduled_works+0x2a4/0x510 [33526.808968] worker_thread+0x188/0x340 [33526.812718] ? rescuer_thread+0x310/0x310 [33526.816722] kthread+0xf5/0x130 [33526.819867] ? kthread_complete_and_exit+0x20/0x20 [33526.824652] ret_from_fork+0x31/0x40 [33526.828229] ? kthread_complete_and_exit+0x20/0x20 [33526.833014] ret_from_fork_asm+0x11/0x20 [33526.836938] [33554.047241] watchdog: BUG: soft lockup - CPU#0 stuck for 20135s! [dbench:78683] [33554.054544] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33554.059241] watchdog: BUG: soft lockup - CPU#7 stuck for 29132s! [kworker/7:4:78808] [33554.109986] irq event stamp: 57762766 [33554.109987] hardirqs last enabled at (57762765): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33554.117716] Modules linked in: [33554.121374] hardirqs last disabled at (57762766): [] sysvec_apic_timer_interrupt+0xe/0x90 [33554.131531] ceph [33554.134582] softirqs last enabled at (57762738): [] __do_softirq+0x2be/0x371 [33554.144307] libceph [33554.146230] softirqs last disabled at (57762733): [] irq_exit_rcu+0xb0/0xd0 [33554.154915] dns_resolver [33554.157100] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33554.165610] fscache netfs [33554.168243] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33554.177187] veth nft_chain_nat [33554.179892] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33554.187381] xt_MASQUERADE [33554.190519] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33554.195996] nf_nat nf_conntrack [33554.198699] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33554.217437] nf_defrag_ipv6 nf_defrag_ipv4 [33554.220662] [33554.220663] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33554.225878] nft_compat nf_tables [33554.229970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33554.231460] nfnetlink [33554.238585] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33554.241894] bridge stp [33554.249036] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33554.251394] llc xfs [33554.258518] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33554.260961] libcrc32c [33554.268086] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33554.270269] sunrpc intel_rapl_msr [33554.277395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33554.279751] intel_rapl_common sb_edac [33554.287828] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33554.291226] x86_pkg_temp_thermal coretemp [33554.296963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33554.300709] kvm_intel [33554.307832] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33554.311927] kvm irqbypass [33554.319055] Call Trace: [33554.321412] i2c_i801 iTCO_wdt [33554.328538] [33554.331240] iTCO_vendor_support i2c_smbus [33554.333686] ? watchdog_timer_fn+0x1b4/0x220 [33554.336736] crct10dif_pclmul joydev [33554.338749] ? __hrtimer_run_queues+0x1d3/0x360 [33554.342837] crc32_pclmul [33554.347106] ? hrtimer_interrupt+0x104/0x240 [33554.350672] ipmi_ssif [33554.355195] ? __local_bh_enable+0x50/0x60 [33554.357813] ghash_clmulni_intel mei_me [33554.362081] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33554.364435] mei lpc_ich [33554.368527] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33554.372356] sha512_ssse3 [33554.377574] [33554.380104] acpi_ipmi mfd_core [33554.385063] [33554.385065] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33554.387680] wmi ipmi_si [33554.389782] ? queued_spin_lock_slowpath+0x18/0x2e0 [33554.392920] ipmi_devintf ipmi_msghandler [33554.395022] do_raw_spin_lock+0xb5/0xc0 [33554.400323] ioatdma acpi_pad acpi_power_meter [33554.402857] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33554.407725] ixgbe igb [33554.411731] try_get_cap_refs+0x159/0x880 [ceph] [33554.415559] mdio crc32c_intel [33554.420002] __ceph_get_caps+0x4a1/0x760 [ceph] [33554.424782] nvme ptp [33554.427143] ? __file_remove_privs+0x89/0x110 [33554.431750] nvme_core i2c_algo_bit [33554.434802] ? wake_bit_function+0x70/0x70 [33554.439325] pps_core dca [33554.441599] ? ceph_write_iter+0x260/0x770 [ceph] [33554.445947] fuse [33554.449432] ceph_write_iter+0x260/0x770 [ceph] [33554.453522] irq event stamp: 189459202 [33554.456143] ? lock_is_held_type+0xaa/0x120 [33554.460837] hardirqs last enabled at (189459201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33554.462762] vfs_write+0x324/0x4f0 [33554.467284] hardirqs last disabled at (189459202): [] sysvec_apic_timer_interrupt+0xe/0x90 [33554.471033] ksys_pwrite64+0x65/0xa0 [33554.475206] softirqs last enabled at (189459200): [] __do_softirq+0x2be/0x371 [33554.485450] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33554.488847] softirqs last disabled at (189459195): [] irq_exit_rcu+0xb0/0xd0 [33554.498660] do_syscall_64+0x38/0x80 [33554.502230] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33554.511001] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33554.516131] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33554.524729] RIP: 0033:0x7f48f0eeb93b [33554.528300] Workqueue: events delayed_work [ceph] [33554.537677] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33554.542720] [33554.542721] RIP: 0010:lock_acquire+0xf0/0x2b0 [33554.550191] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33554.553762] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33554.558459] ORIG_RAX: 0000000000000012 [33554.577197] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33554.578689] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33554.583038] [33554.583039] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33554.588255] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33554.606992] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33554.610825] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33554.616042] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33554.623165] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33554.624657] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33554.631782] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33554.638905] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33554.646035] [33554.653154] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33554.699035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33554.704773] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33554.711897] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33554.719020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33554.726146] Call Trace: [33554.728589] [33554.730602] ? watchdog_timer_fn+0x1b4/0x220 [33554.734866] ? __hrtimer_run_queues+0x1d3/0x360 [33554.739391] ? hrtimer_interrupt+0x104/0x240 [33554.743653] ? __local_bh_enable+0x50/0x60 [33554.747747] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33554.752971] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33554.757928] [33554.760024] [33554.762123] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33554.767430] ? lock_acquire+0xf0/0x2b0 [33554.771179] ? igrab+0x43/0x50 [33554.774229] ? lock_release+0xc5/0x260 [33554.777975] _raw_spin_lock+0x30/0x40 [33554.781646] ? igrab+0x1d/0x50 [33554.784699] igrab+0x1d/0x50 [33554.787577] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33554.792728] delayed_work+0x1d8/0x280 [ceph] [33554.797009] ? process_scheduled_works+0x22a/0x510 [33554.801797] process_scheduled_works+0x2a4/0x510 [33554.806414] worker_thread+0x188/0x340 [33554.810163] ? rescuer_thread+0x310/0x310 [33554.814167] kthread+0xf5/0x130 [33554.817303] ? kthread_complete_and_exit+0x20/0x20 [33554.822088] ret_from_fork+0x31/0x40 [33554.825658] ? kthread_complete_and_exit+0x20/0x20 [33554.830444] ret_from_fork_asm+0x11/0x20 [33554.834365] [33582.047757] watchdog: BUG: soft lockup - CPU#0 stuck for 20161s! [dbench:78683] [33582.055059] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33582.059757] watchdog: BUG: soft lockup - CPU#7 stuck for 29158s! [kworker/7:4:78808] [33582.110507] irq event stamp: 57820910 [33582.118238] Modules linked in: ceph libceph dns_resolver [33582.121897] hardirqs last enabled at (57820909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33582.121901] fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat [33582.137367] hardirqs last disabled at (57820910): [] sysvec_apic_timer_interrupt+0xe/0x90 [33582.143537] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [33582.153261] softirqs last enabled at (57820882): [] __do_softirq+0x2be/0x371 [33582.161772] stp llc xfs libcrc32c sunrpc [33582.170456] softirqs last disabled at (57820877): [] irq_exit_rcu+0xb0/0xd0 [33582.170459] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [33582.182972] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33582.191482] kvm irqbypass [33582.200432] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33582.200434] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33582.203155] i2c_i801 [33582.210627] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33582.216105] iTCO_wdt [33582.218374] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33582.237114] iTCO_vendor_support [33582.239382] [33582.239383] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33582.244600] i2c_smbus crct10dif_pclmul [33582.247824] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33582.249315] joydev [33582.256440] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33582.260271] crc32_pclmul ipmi_ssif [33582.267395] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33582.269493] ghash_clmulni_intel [33582.276617] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33582.280101] mei_me mei [33582.287224] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33582.290448] lpc_ich sha512_ssse3 [33582.297572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33582.300016] acpi_ipmi mfd_core [33582.308093] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33582.311404] wmi ipmi_si [33582.317155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33582.320288] ipmi_devintf ipmi_msghandler [33582.327412] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33582.329961] ioatdma acpi_pad [33582.337086] Call Trace: [33582.337088] [33582.341089] acpi_power_meter [33582.348216] ? watchdog_timer_fn+0x1b4/0x220 [33582.351176] ixgbe [33582.353621] ? __hrtimer_run_queues+0x1d3/0x360 [33582.355633] igb mdio [33582.358600] ? hrtimer_interrupt+0x104/0x240 [33582.362860] crc32c_intel [33582.364869] ? __local_bh_enable+0x50/0x60 [33582.369396] nvme [33582.371667] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33582.375929] ptp nvme_core [33582.378549] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33582.382637] i2c_algo_bit [33582.384563] [33582.389778] pps_core dca [33582.392484] [33582.397441] fuse [33582.400058] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33582.402155] [33582.404777] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33582.406870] irq event stamp: 189679928 [33582.408797] do_raw_spin_lock+0xb5/0xc0 [33582.414097] hardirqs last enabled at (189679927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33582.415590] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33582.420460] hardirqs last disabled at (189679928): [] sysvec_apic_timer_interrupt+0xe/0x90 [33582.424207] try_get_cap_refs+0x159/0x880 [ceph] [33582.428035] softirqs last enabled at (189679926): [] __do_softirq+0x2be/0x371 [33582.438283] __ceph_get_caps+0x4a1/0x760 [ceph] [33582.443063] softirqs last disabled at (189679921): [] irq_exit_rcu+0xb0/0xd0 [33582.452878] ? __file_remove_privs+0x89/0x110 [33582.457485] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33582.466255] ? wake_bit_function+0x70/0x70 [33582.470780] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33582.479381] ? ceph_write_iter+0x260/0x770 [ceph] [33582.483726] Workqueue: events delayed_work [ceph] [33582.493106] ceph_write_iter+0x260/0x770 [ceph] [33582.497196] [33582.497198] RIP: 0010:lock_acquire+0x42/0x2b0 [33582.504670] ? lock_is_held_type+0xaa/0x120 [33582.509364] Code: 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 <48> 0f a3 05 4e bd 53 01 0f 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 [33582.514064] vfs_write+0x324/0x4f0 [33582.518585] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [33582.520081] ksys_pwrite64+0x65/0xa0 [33582.524427] [33582.524428] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [33582.528605] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33582.547342] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [33582.550740] do_syscall_64+0x38/0x80 [33582.555957] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [33582.559529] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33582.561019] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [33582.568162] RIP: 0033:0x7f48f0eeb93b [33582.573290] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33582.580416] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33582.584002] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33582.591127] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33582.596170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33582.603295] ORIG_RAX: 0000000000000012 [33582.606864] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33582.613991] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33582.632727] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33582.640804] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33582.646022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33582.651760] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33582.655590] Call Trace: [33582.655591] [33582.662714] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33582.669841] ? watchdog_timer_fn+0x1b4/0x220 [33582.676964] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33582.684089] ? __hrtimer_run_queues+0x1d3/0x360 [33582.691217] [33582.698339] ? hrtimer_interrupt+0x104/0x240 [33582.732274] ? __local_bh_enable+0x50/0x60 [33582.736369] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33582.741592] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33582.746550] [33582.748647] [33582.750744] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33582.756052] ? lock_acquire+0x42/0x2b0 [33582.759803] ? igrab+0x43/0x50 [33582.762853] ? lock_release+0xc5/0x260 [33582.766598] _raw_spin_lock+0x30/0x40 [33582.770263] ? igrab+0x1d/0x50 [33582.773312] igrab+0x1d/0x50 [33582.776192] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33582.781341] delayed_work+0x1d8/0x280 [ceph] [33582.785622] ? process_scheduled_works+0x22a/0x510 [33582.790413] process_scheduled_works+0x2a4/0x510 [33582.795029] worker_thread+0x188/0x340 [33582.798777] ? rescuer_thread+0x310/0x310 [33582.802779] kthread+0xf5/0x130 [33582.805917] ? kthread_complete_and_exit+0x20/0x20 [33582.810703] ret_from_fork+0x31/0x40 [33582.814282] ? kthread_complete_and_exit+0x20/0x20 [33582.819065] ret_from_fork_asm+0x11/0x20 [33582.822995] [33610.048274] watchdog: BUG: soft lockup - CPU#0 stuck for 20187s! [dbench:78683] [33610.055571] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33610.060274] watchdog: BUG: soft lockup - CPU#7 stuck for 29184s! [kworker/7:4:78808] [33610.111013] irq event stamp: 57878944 [33610.111014] hardirqs last enabled at (57878943): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33610.118744] Modules linked in: [33610.122400] hardirqs last disabled at (57878944): [] sysvec_apic_timer_interrupt+0xe/0x90 [33610.132559] ceph [33610.135609] softirqs last enabled at (57878934): [] __do_softirq+0x2be/0x371 [33610.145333] libceph [33610.147257] softirqs last disabled at (57878929): [] irq_exit_rcu+0xb0/0xd0 [33610.155956] dns_resolver fscache [33610.158143] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33610.166668] netfs veth [33610.169982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33610.178926] nft_chain_nat xt_MASQUERADE [33610.181388] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33610.188859] nf_nat nf_conntrack [33610.192776] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33610.198253] nf_defrag_ipv6 nf_defrag_ipv4 [33610.201478] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33610.220215] nft_compat [33610.224305] [33610.224306] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33610.229522] nf_tables [33610.231966] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33610.233458] nfnetlink bridge [33610.240582] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33610.242951] stp llc [33610.250071] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33610.253036] xfs libcrc32c [33610.260161] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33610.262344] sunrpc [33610.269467] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33610.272172] intel_rapl_msr intel_rapl_common [33610.279298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33610.281394] sb_edac x86_pkg_temp_thermal [33610.289472] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33610.293822] coretemp kvm_intel [33610.299559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33610.303564] kvm [33610.310688] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33610.313826] irqbypass i2c_i801 [33610.320952] Call Trace: [33610.322787] iTCO_wdt iTCO_vendor_support [33610.329911] [33610.333047] i2c_smbus [33610.335493] ? watchdog_timer_fn+0x1b4/0x220 [33610.339497] crct10dif_pclmul joydev [33610.341508] ? __hrtimer_run_queues+0x1d3/0x360 [33610.343863] crc32_pclmul ipmi_ssif [33610.348131] ? hrtimer_interrupt+0x104/0x240 [33610.351700] ghash_clmulni_intel mei_me [33610.356222] ? __local_bh_enable+0x50/0x60 [33610.359708] mei lpc_ich [33610.363973] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33610.367801] sha512_ssse3 [33610.371893] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33610.374423] acpi_ipmi mfd_core [33610.379642] [33610.382257] wmi ipmi_si [33610.387215] [33610.387216] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33610.390353] ipmi_devintf ipmi_msghandler [33610.392454] ? queued_spin_lock_slowpath+0x18/0x2e0 [33610.394980] ioatdma acpi_pad [33610.397082] do_raw_spin_lock+0xb5/0xc0 [33610.402382] acpi_power_meter ixgbe [33610.406388] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33610.411257] igb mdio [33610.414223] try_get_cap_refs+0x159/0x880 [ceph] [33610.418052] crc32c_intel [33610.421541] __ceph_get_caps+0x4a1/0x760 [ceph] [33610.426320] nvme ptp [33610.428596] ? __file_remove_privs+0x89/0x110 [33610.433203] nvme_core i2c_algo_bit [33610.435819] ? wake_bit_function+0x70/0x70 [33610.440343] pps_core dca [33610.442618] ? ceph_write_iter+0x260/0x770 [ceph] [33610.446964] fuse [33610.450449] ceph_write_iter+0x260/0x770 [ceph] [33610.454539] irq event stamp: 189900734 [33610.457162] ? lock_is_held_type+0xaa/0x120 [33610.461854] hardirqs last enabled at (189900733): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33610.463780] vfs_write+0x324/0x4f0 [33610.468303] hardirqs last disabled at (189900734): [] sysvec_apic_timer_interrupt+0xe/0x90 [33610.472052] ksys_pwrite64+0x65/0xa0 [33610.476223] softirqs last enabled at (189900732): [] __do_softirq+0x2be/0x371 [33610.486468] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33610.489866] softirqs last disabled at (189900727): [] irq_exit_rcu+0xb0/0xd0 [33610.499677] do_syscall_64+0x38/0x80 [33610.503247] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33610.512019] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33610.517148] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33610.525747] RIP: 0033:0x7f48f0eeb93b [33610.529317] Workqueue: events delayed_work [ceph] [33610.538694] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33610.543739] [33610.551209] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33610.554779] RIP: 0010:lock_acquire+0x17b/0x2b0 [33610.559479] ORIG_RAX: 0000000000000012 [33610.578216] Code: c8 89 d1 89 f2 4c 89 e6 e8 62 5b ff ff 59 5e 4c 8b 14 24 65 ff 0d 5d e6 eb 7e 0f 85 c8 fe ff ff 0f 1f 44 00 00 e9 be fe ff ff <4c> 89 14 24 e8 9c 8a 02 00 4c 8b 14 24 84 c0 0f 85 c0 fe ff ff 0f [33610.579706] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33610.584924] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [33610.589362] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33610.593191] [33610.593192] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [33610.611930] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33610.619053] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [33610.624271] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33610.631394] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [33610.632885] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33610.640009] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [33610.647140] [33610.654259] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33610.692063] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33610.700141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33610.705878] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33610.713001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33610.720127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33610.727251] Call Trace: [33610.729694] [33610.731705] ? watchdog_timer_fn+0x1b4/0x220 [33610.735971] ? __hrtimer_run_queues+0x1d3/0x360 [33610.740497] ? hrtimer_interrupt+0x104/0x240 [33610.744766] ? __local_bh_enable+0x50/0x60 [33610.748860] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33610.754084] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33610.759041] [33610.761137] [33610.763236] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33610.768543] ? lock_acquire+0x17b/0x2b0 [33610.772380] ? igrab+0x43/0x50 [33610.775429] ? lock_release+0xc5/0x260 [33610.779175] _raw_spin_lock+0x30/0x40 [33610.782832] ? igrab+0x1d/0x50 [33610.785881] igrab+0x1d/0x50 [33610.788760] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33610.793909] delayed_work+0x1d8/0x280 [ceph] [33610.798192] ? process_scheduled_works+0x22a/0x510 [33610.802982] process_scheduled_works+0x2a4/0x510 [33610.807595] worker_thread+0x188/0x340 [33610.811347] ? rescuer_thread+0x310/0x310 [33610.815357] kthread+0xf5/0x130 [33610.818494] ? kthread_complete_and_exit+0x20/0x20 [33610.823280] ret_from_fork+0x31/0x40 [33610.826850] ? kthread_complete_and_exit+0x20/0x20 [33610.831634] ret_from_fork_asm+0x11/0x20 [33610.835558] [33638.048791] watchdog: BUG: soft lockup - CPU#0 stuck for 20213s! [dbench:78683] [33638.056095] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33638.060791] watchdog: BUG: soft lockup - CPU#7 stuck for 29210s! [kworker/7:4:78808] [33638.111543] irq event stamp: 57937166 [33638.119274] Modules linked in: ceph libceph dns_resolver fscache netfs [33638.122932] hardirqs last enabled at (57937165): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33638.129467] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [33638.139625] hardirqs last disabled at (57937166): [] sysvec_apic_timer_interrupt+0xe/0x90 [33638.145708] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [33638.155432] softirqs last enabled at (57937138): [] __do_softirq+0x2be/0x371 [33638.155436] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [33638.171501] softirqs last disabled at (57937133): [] irq_exit_rcu+0xb0/0xd0 [33638.179068] x86_pkg_temp_thermal coretemp kvm_intel kvm [33638.187580] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33638.192883] irqbypass i2c_i801 [33638.201826] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33638.204964] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [33638.212435] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33638.218778] joydev crc32_pclmul [33638.224257] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33638.224258] ipmi_ssif [33638.224259] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33638.227480] ghash_clmulni_intel [33638.246216] [33638.246217] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33638.248575] mei_me [33638.253791] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33638.257015] mei [33638.258508] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33638.265632] lpc_ich [33638.267730] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33638.274851] sha512_ssse3 acpi_ipmi [33638.276690] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33638.283814] mfd_core [33638.285999] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33638.293123] wmi [33638.296605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33638.303730] ipmi_si [33638.306001] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33638.314079] ipmi_devintf [33638.315917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33638.321653] ipmi_msghandler ioatdma [33638.323837] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33638.330962] acpi_pad acpi_power_meter [33638.333579] Call Trace: [33638.333581] [33638.340703] ixgbe igb [33638.344275] ? watchdog_timer_fn+0x1b4/0x220 [33638.351399] mdio crc32c_intel [33638.355144] ? __hrtimer_run_queues+0x1d3/0x360 [33638.357587] nvme [33638.359601] ? hrtimer_interrupt+0x104/0x240 [33638.361954] ptp nvme_core [33638.366217] ? __local_bh_enable+0x50/0x60 [33638.369269] i2c_algo_bit [33638.373796] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33638.375718] pps_core dca [33638.379982] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33638.382685] fuse [33638.386777] [33638.389393] irq event stamp: 190121246 [33638.394611] [33638.394612] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33638.397229] hardirqs last enabled at (190121245): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33638.402193] ? queued_spin_lock_slowpath+0x18/0x2e0 [33638.404110] hardirqs last disabled at (190121246): [] sysvec_apic_timer_interrupt+0xe/0x90 [33638.406209] do_raw_spin_lock+0xb5/0xc0 [33638.409950] softirqs last enabled at (190121244): [] __do_softirq+0x2be/0x371 [33638.412049] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33638.417353] softirqs last disabled at (190121239): [] irq_exit_rcu+0xb0/0xd0 [33638.427601] try_get_cap_refs+0x159/0x880 [ceph] [33638.432469] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33638.442284] __ceph_get_caps+0x4a1/0x760 [ceph] [33638.446108] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33638.454883] ? __file_remove_privs+0x89/0x110 [33638.459664] Workqueue: events delayed_work [ceph] [33638.468261] ? wake_bit_function+0x70/0x70 [33638.472872] [33638.482252] ? ceph_write_iter+0x260/0x770 [ceph] [33638.486773] RIP: 0010:lock_release+0x162/0x260 [33638.494246] ceph_write_iter+0x260/0x770 [ceph] [33638.498596] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [33638.503297] ? lock_is_held_type+0xaa/0x120 [33638.507384] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [33638.508878] vfs_write+0x324/0x4f0 [33638.513572] [33638.513572] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [33638.518015] ksys_pwrite64+0x65/0xa0 [33638.522534] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [33638.541273] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33638.545449] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [33638.550666] do_syscall_64+0x38/0x80 [33638.554063] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [33638.555554] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33638.562678] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [33638.566249] RIP: 0033:0x7f48f0eeb93b [33638.573372] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33638.578503] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33638.585627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33638.589199] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33638.596322] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33638.601366] ORIG_RAX: 0000000000000012 [33638.608490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33638.612063] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33638.620138] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33638.638876] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33638.644615] Call Trace: [33638.644616] [33638.649831] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33638.656958] ? watchdog_timer_fn+0x1b4/0x220 [33638.660785] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33638.667913] ? __hrtimer_run_queues+0x1d3/0x360 [33638.675033] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33638.682163] ? hrtimer_interrupt+0x104/0x240 [33638.689289] [33638.691727] ? __local_bh_enable+0x50/0x60 [33638.734439] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33638.739665] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33638.744620] [33638.746718] [33638.748816] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33638.754120] ? igrab+0x43/0x50 [33638.757193] ? lock_release+0x162/0x260 [33638.761028] ? lock_release+0xc5/0x260 [33638.764774] _raw_spin_unlock+0x1b/0x40 [33638.768612] igrab+0x43/0x50 [33638.771492] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33638.776640] delayed_work+0x1d8/0x280 [ceph] [33638.780920] ? process_scheduled_works+0x22a/0x510 [33638.785711] process_scheduled_works+0x2a4/0x510 [33638.790326] worker_thread+0x188/0x340 [33638.794074] ? rescuer_thread+0x310/0x310 [33638.798077] kthread+0xf5/0x130 [33638.801216] ? kthread_complete_and_exit+0x20/0x20 [33638.806000] ret_from_fork+0x31/0x40 [33638.809580] ? kthread_complete_and_exit+0x20/0x20 [33638.814364] ret_from_fork_asm+0x11/0x20 [33638.818297] [33666.049307] watchdog: BUG: soft lockup - CPU#0 stuck for 20239s! [dbench:78683] [33666.056606] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33666.061307] watchdog: BUG: soft lockup - CPU#7 stuck for 29236s! [kworker/7:4:78808] [33666.112050] irq event stamp: 57995200 [33666.112051] hardirqs last enabled at (57995199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33666.119779] Modules linked in: [33666.123438] hardirqs last disabled at (57995200): [] sysvec_apic_timer_interrupt+0xe/0x90 [33666.133594] ceph [33666.136644] softirqs last enabled at (57995174): [] __do_softirq+0x2be/0x371 [33666.146370] libceph [33666.148294] softirqs last disabled at (57995169): [] irq_exit_rcu+0xb0/0xd0 [33666.156990] dns_resolver fscache [33666.159170] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33666.167681] netfs veth [33666.171008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33666.179953] nft_chain_nat [33666.182397] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [33666.189868] xt_MASQUERADE [33666.192573] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [33666.198049] nf_nat nf_conntrack [33666.200754] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33666.219491] nf_defrag_ipv6 [33666.222714] [33666.222715] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33666.227932] nf_defrag_ipv4 nft_compat [33666.230723] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33666.232213] nf_tables nfnetlink [33666.239337] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33666.243083] bridge stp [33666.250208] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33666.253430] llc xfs [33666.260555] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33666.262999] libcrc32c sunrpc [33666.270122] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33666.272306] intel_rapl_msr [33666.279431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33666.282395] intel_rapl_common sb_edac [33666.290473] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33666.293264] x86_pkg_temp_thermal [33666.299002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33666.302744] coretemp kvm_intel [33666.309869] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33666.313181] kvm irqbypass [33666.320304] Call Trace: [33666.320305] [33666.323442] i2c_i801 iTCO_wdt [33666.330567] ? watchdog_timer_fn+0x1b4/0x220 [33666.333270] iTCO_vendor_support i2c_smbus [33666.335715] ? __hrtimer_run_queues+0x1d3/0x360 [33666.337725] crct10dif_pclmul joydev [33666.340778] ? hrtimer_interrupt+0x104/0x240 [33666.345039] crc32_pclmul [33666.349130] ? __local_bh_enable+0x50/0x60 [33666.353652] ipmi_ssif ghash_clmulni_intel [33666.357228] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33666.361489] mei_me mei [33666.364107] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33666.368197] lpc_ich sha512_ssse3 [33666.372289] [33666.377505] acpi_ipmi [33666.379948] [33666.384906] mfd_core wmi [33666.388217] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33666.390315] ipmi_si ipmi_devintf [33666.392677] ? queued_spin_lock_slowpath+0x2b/0x2e0 [33666.394768] ipmi_msghandler [33666.397388] do_raw_spin_lock+0xb5/0xc0 [33666.402706] ioatdma acpi_pad [33666.406020] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33666.410890] acpi_power_meter [33666.413769] try_get_cap_refs+0x159/0x880 [ceph] [33666.417597] ixgbe igb [33666.420566] __ceph_get_caps+0x4a1/0x760 [ceph] [33666.425346] mdio [33666.428314] ? __file_remove_privs+0x89/0x110 [33666.432922] crc32c_intel [33666.435279] ? wake_bit_function+0x70/0x70 [33666.439802] nvme ptp [33666.441730] ? ceph_write_iter+0x260/0x770 [ceph] [33666.446076] nvme_core i2c_algo_bit [33666.448708] ceph_write_iter+0x260/0x770 [ceph] [33666.452802] pps_core dca [33666.455075] ? lock_is_held_type+0xaa/0x120 [33666.459771] fuse [33666.463258] vfs_write+0x324/0x4f0 [33666.467778] [33666.467779] irq event stamp: 190342092 [33666.470399] ksys_pwrite64+0x65/0xa0 [33666.474574] hardirqs last enabled at (190342091): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33666.476497] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33666.479896] hardirqs last disabled at (190342092): [] sysvec_apic_timer_interrupt+0xe/0x90 [33666.481386] do_syscall_64+0x38/0x80 [33666.485130] softirqs last enabled at (190342090): [] __do_softirq+0x2be/0x371 [33666.488710] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33666.498954] softirqs last disabled at (190342085): [] irq_exit_rcu+0xb0/0xd0 [33666.504084] RIP: 0033:0x7f48f0eeb93b [33666.513895] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33666.517465] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33666.526237] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33666.531280] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33666.539877] Workqueue: events delayed_work [ceph] [33666.543447] ORIG_RAX: 0000000000000012 [33666.543448] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33666.552825] [33666.552826] RIP: 0010:lock_release+0x0/0x260 [33666.571564] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33666.579034] Code: ff ff 48 c7 c6 c9 df 12 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 [33666.584252] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33666.588949] RSP: 0018:ffffc90002117d88 EFLAGS: 00000286 [33666.592780] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33666.599903] [33666.599904] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [33666.601394] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33666.605658] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [33666.612788] [33666.631521] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [33666.683176] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33666.690300] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33666.697423] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33666.705502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33666.711239] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33666.718364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33666.725487] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33666.732612] Call Trace: [33666.735054] [33666.737068] ? watchdog_timer_fn+0x1b4/0x220 [33666.741331] ? __hrtimer_run_queues+0x1d3/0x360 [33666.745859] ? hrtimer_interrupt+0x104/0x240 [33666.750126] ? __local_bh_enable+0x50/0x60 [33666.754221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33666.759444] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33666.764402] [33666.766499] [33666.768597] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33666.773904] ? igrab+0x43/0x50 [33666.776960] ? lock_downgrade+0x250/0x250 [33666.780965] ? _raw_spin_lock+0x30/0x40 [33666.784796] _raw_spin_unlock+0x1b/0x40 [33666.788624] igrab+0x43/0x50 [33666.791502] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33666.796654] delayed_work+0x1d8/0x280 [ceph] [33666.800935] ? process_scheduled_works+0x22a/0x510 [33666.805725] process_scheduled_works+0x2a4/0x510 [33666.810339] worker_thread+0x188/0x340 [33666.814090] ? rescuer_thread+0x310/0x310 [33666.818102] kthread+0xf5/0x130 [33666.821239] ? kthread_complete_and_exit+0x20/0x20 [33666.826023] ret_from_fork+0x31/0x40 [33666.829593] ? kthread_complete_and_exit+0x20/0x20 [33666.834378] ret_from_fork_asm+0x11/0x20 [33666.838301] [33693.349178] rcu: INFO: rcu_preempt self-detected stall on CPU [33693.354945] rcu: 0-....: (21265631 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10644820 [33693.365458] rcu: (t=21756353 jiffies g=1309609 q=617775 ncpus=8) [33693.371542] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33693.380484] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33693.387956] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33693.393443] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33693.412179] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33693.417424] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33693.424547] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33693.431669] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33693.438796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33693.445919] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33693.453044] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33693.461122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33693.466858] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33693.473981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33693.481106] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33693.488230] Call Trace: [33693.490675] [33693.492688] ? rcu_dump_cpu_stacks+0xd1/0x180 [33693.497045] ? rcu_sched_clock_irq+0x633/0x1390 [33693.501570] ? lock_is_held_type+0xaa/0x120 [33693.505755] ? lock_is_held_type+0xaa/0x120 [33693.509934] ? tick_sched_do_timer+0xa0/0xa0 [33693.514207] ? rcu_read_lock_sched_held+0x56/0x90 [33693.518913] ? tick_sched_do_timer+0xa0/0xa0 [33693.523176] ? update_process_times+0x5f/0x90 [33693.527534] ? tick_sched_handle.isra.16+0x1e/0x50 [33693.532318] ? tick_sched_timer+0x6f/0x80 [33693.536324] ? __hrtimer_run_queues+0x1d3/0x360 [33693.540849] ? hrtimer_interrupt+0x104/0x240 [33693.545111] ? __local_bh_enable+0x50/0x60 [33693.549213] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33693.554437] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33693.559394] [33693.561505] [33693.563606] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33693.568915] ? queued_spin_lock_slowpath+0x18/0x2e0 [33693.573791] do_raw_spin_lock+0xb5/0xc0 [33693.577632] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33693.582442] try_get_cap_refs+0x159/0x880 [ceph] [33693.587074] __ceph_get_caps+0x4a1/0x760 [ceph] [33693.591613] ? __file_remove_privs+0x89/0x110 [33693.595969] ? wake_bit_function+0x70/0x70 [33693.600070] ? ceph_write_iter+0x260/0x770 [ceph] [33693.604783] ceph_write_iter+0x260/0x770 [ceph] [33693.609320] ? lock_is_held_type+0xaa/0x120 [33693.613509] vfs_write+0x324/0x4f0 [33693.616912] ksys_pwrite64+0x65/0xa0 [33693.620500] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33693.625636] do_syscall_64+0x38/0x80 [33693.629215] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33693.634259] RIP: 0033:0x7f48f0eeb93b [33693.637828] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33693.656565] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [33693.664123] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33693.671249] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33693.678371] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33693.685506] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33693.692630] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33693.699758] [33693.701945] Sending NMI from CPU 0 to CPUs 7: [33693.706298] NMI backtrace for cpu 7 [33693.706301] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33693.706303] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33693.706305] Workqueue: events delayed_work [ceph] [33693.706328] RIP: 0010:ceph_check_delayed_caps+0x55/0x270 [ceph] [33693.706347] Code: 18 44 69 60 18 e8 03 00 00 0f 1f 44 00 00 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b <48> 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 [33693.706349] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [33693.706350] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [33693.706352] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [33693.706353] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [33693.706354] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [33693.706356] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [33693.706357] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33693.706359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33693.706360] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33693.706362] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33693.706363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33693.706364] Call Trace: [33693.706366] [33693.706367] ? nmi_cpu_backtrace+0xad/0x120 [33693.706373] ? nmi_cpu_backtrace_handler+0x11/0x20 [33693.706375] ? nmi_handle+0xd4/0x260 [33693.706381] ? default_do_nmi+0x49/0xf0 [33693.706384] ? exc_nmi+0x11e/0x150 [33693.706387] ? end_repeat_nmi+0x16/0x67 [33693.706394] ? ceph_check_delayed_caps+0x55/0x270 [ceph] [33693.706412] ? ceph_check_delayed_caps+0x55/0x270 [ceph] [33693.706429] ? ceph_check_delayed_caps+0x55/0x270 [ceph] [33693.706445] [33693.706446] [33693.706449] delayed_work+0x1d8/0x280 [ceph] [33693.706469] ? process_scheduled_works+0x22a/0x510 [33693.706473] process_scheduled_works+0x2a4/0x510 [33693.706480] worker_thread+0x188/0x340 [33693.706500] ? rescuer_thread+0x310/0x310 [33693.706502] kthread+0xf5/0x130 [33693.706506] ? kthread_complete_and_exit+0x20/0x20 [33693.706509] ret_from_fork+0x31/0x40 [33693.706512] ? kthread_complete_and_exit+0x20/0x20 [33693.706515] ret_from_fork_asm+0x11/0x20 [33693.706524] [33718.050267] watchdog: BUG: soft lockup - CPU#0 stuck for 20288s! [dbench:78683] [33718.057564] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33718.062267] watchdog: BUG: soft lockup - CPU#7 stuck for 29285s! [kworker/7:4:78808] [33718.113015] irq event stamp: 58103526 [33718.120747] Modules linked in: ceph libceph [33718.124403] hardirqs last enabled at (58103525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33718.128581] dns_resolver fscache netfs [33718.138739] hardirqs last disabled at (58103526): [] sysvec_apic_timer_interrupt+0xe/0x90 [33718.142569] veth nft_chain_nat xt_MASQUERADE [33718.152292] softirqs last enabled at (58103488): [] __do_softirq+0x2be/0x371 [33718.152294] nf_nat nf_conntrack nf_defrag_ipv6 [33718.165346] softirqs last disabled at (58103483): [] irq_exit_rcu+0xb0/0xd0 [33718.165347] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [33718.178381] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33718.183857] bridge stp [33718.192803] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33718.195245] llc xfs libcrc32c sunrpc [33718.202717] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [33718.206373] intel_rapl_msr [33718.211852] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [33718.211853] intel_rapl_common sb_edac [33718.214660] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33718.233397] x86_pkg_temp_thermal [33718.237142] [33718.237143] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33718.242358] coretemp [33718.245669] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33718.247161] kvm_intel [33718.254284] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33718.256555] kvm irqbypass [33718.263679] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33718.266036] i2c_i801 iTCO_wdt [33718.273162] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33718.275863] iTCO_vendor_support i2c_smbus [33718.282988] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33718.286040] crct10dif_pclmul [33718.293163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33718.297255] joydev crc32_pclmul [33718.305331] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33718.308295] ipmi_ssif ghash_clmulni_intel [33718.314033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33718.317258] mei_me [33718.324381] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33718.328471] mei lpc_ich [33718.335597] Call Trace: [33718.335599] [33718.337695] sha512_ssse3 [33718.344817] ? watchdog_timer_fn+0x1b4/0x220 [33718.347349] acpi_ipmi mfd_core [33718.349795] ? __hrtimer_run_queues+0x1d3/0x360 [33718.351802] wmi [33718.354424] ? hrtimer_interrupt+0x104/0x240 [33718.358686] ipmi_si [33718.361822] ? __local_bh_enable+0x50/0x60 [33718.366345] ipmi_devintf [33718.368186] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33718.372448] ipmi_msghandler [33718.374633] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33718.378722] ioatdma acpi_pad [33718.381341] [33718.386558] acpi_power_meter ixgbe [33718.389436] [33718.394391] igb mdio [33718.397358] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33718.399454] crc32c_intel nvme [33718.402951] ? queued_spin_lock_slowpath+0x18/0x2e0 [33718.405044] ptp [33718.407316] do_raw_spin_lock+0xb5/0xc0 [33718.412618] nvme_core [33718.415671] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33718.420540] i2c_algo_bit pps_core [33718.422379] try_get_cap_refs+0x159/0x880 [ceph] [33718.426209] dca fuse [33718.428569] __ceph_get_caps+0x4a1/0x760 [ceph] [33718.433351] [33718.436750] ? __file_remove_privs+0x89/0x110 [33718.441357] irq event stamp: 190755626 [33718.443629] ? wake_bit_function+0x70/0x70 [33718.448152] hardirqs last enabled at (190755625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33718.449665] ? ceph_write_iter+0x260/0x770 [ceph] [33718.454012] hardirqs last disabled at (190755626): [] sysvec_apic_timer_interrupt+0xe/0x90 [33718.457755] ceph_write_iter+0x260/0x770 [ceph] [33718.461845] softirqs last enabled at (190755624): [] __do_softirq+0x2be/0x371 [33718.472094] ? lock_is_held_type+0xaa/0x120 [33718.476788] softirqs last disabled at (190755619): [] irq_exit_rcu+0xb0/0xd0 [33718.486601] vfs_write+0x324/0x4f0 [33718.491123] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33718.499903] ksys_pwrite64+0x65/0xa0 [33718.504070] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33718.512669] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33718.516066] Workqueue: events delayed_work [ceph] [33718.525444] do_syscall_64+0x38/0x80 [33718.529015] [33718.529016] RIP: 0010:lock_acquire+0x86/0x2b0 [33718.536484] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33718.541616] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [33718.546314] RIP: 0033:0x7f48f0eeb93b [33718.549882] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [33718.551374] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33718.555725] [33718.555726] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [33718.560768] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33718.579507] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [33718.583077] ORIG_RAX: 0000000000000012 [33718.588295] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [33718.607034] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33718.608522] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [33718.615648] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33718.620881] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33718.628006] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33718.631838] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33718.638962] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33718.646086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33718.653209] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33718.660333] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33718.667464] [33718.674582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33718.719104] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33718.726228] Call Trace: [33718.728674] [33718.730683] ? watchdog_timer_fn+0x1b4/0x220 [33718.734952] ? __hrtimer_run_queues+0x1d3/0x360 [33718.739484] ? hrtimer_interrupt+0x104/0x240 [33718.743751] ? __local_bh_enable+0x50/0x60 [33718.747845] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33718.753069] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33718.758027] [33718.760123] [33718.762221] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33718.767531] ? lock_acquire+0x86/0x2b0 [33718.771280] ? igrab+0x43/0x50 [33718.774330] ? lock_release+0xc5/0x260 [33718.778076] _raw_spin_lock+0x30/0x40 [33718.781739] ? igrab+0x1d/0x50 [33718.784791] igrab+0x1d/0x50 [33718.787676] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33718.792828] delayed_work+0x1d8/0x280 [ceph] [33718.797107] ? process_scheduled_works+0x22a/0x510 [33718.801889] process_scheduled_works+0x2a4/0x510 [33718.806504] worker_thread+0x188/0x340 [33718.810253] ? rescuer_thread+0x310/0x310 [33718.814258] kthread+0xf5/0x130 [33718.817395] ? kthread_complete_and_exit+0x20/0x20 [33718.822180] ret_from_fork+0x31/0x40 [33718.825758] ? kthread_complete_and_exit+0x20/0x20 [33718.830543] ret_from_fork_asm+0x11/0x20 [33718.834476] [33746.050784] watchdog: BUG: soft lockup - CPU#0 stuck for 20314s! [dbench:78683] [33746.058086] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33746.062784] watchdog: BUG: soft lockup - CPU#7 stuck for 29311s! [kworker/7:4:78808] [33746.113529] irq event stamp: 58161632 [33746.113531] hardirqs last enabled at (58161631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33746.121261] Modules linked in: [33746.124918] hardirqs last disabled at (58161632): [] sysvec_apic_timer_interrupt+0xe/0x90 [33746.135076] ceph [33746.138124] softirqs last enabled at (58161586): [] __do_softirq+0x2be/0x371 [33746.147850] libceph [33746.149772] softirqs last disabled at (58161581): [] irq_exit_rcu+0xb0/0xd0 [33746.158456] dns_resolver [33746.160641] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33746.169153] fscache [33746.171786] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33746.171787] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33746.180732] netfs veth [33746.182917] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33746.190386] nft_chain_nat xt_MASQUERADE [33746.195863] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33746.198308] nf_nat nf_conntrack [33746.217045] [33746.217046] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33746.220963] nf_defrag_ipv6 [33746.226179] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33746.229405] nf_defrag_ipv4 nft_compat [33746.230894] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33746.238019] nf_tables [33746.240810] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33746.247933] nfnetlink bridge [33746.251678] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33746.258803] stp llc [33746.261177] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33746.268301] xfs libcrc32c [33746.271264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33746.278390] sunrpc [33746.280573] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33746.288652] intel_rapl_msr [33746.291355] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33746.297091] intel_rapl_common [33746.299191] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33746.306314] sb_edac x86_pkg_temp_thermal [33746.309104] Call Trace: [33746.309105] [33746.316228] coretemp [33746.319281] ? watchdog_timer_fn+0x1b4/0x220 [33746.326404] kvm_intel kvm [33746.330409] ? __hrtimer_run_queues+0x1d3/0x360 [33746.332852] irqbypass [33746.334867] ? hrtimer_interrupt+0x104/0x240 [33746.337133] i2c_i801 iTCO_wdt [33746.341396] ? __local_bh_enable+0x50/0x60 [33746.344102] iTCO_vendor_support i2c_smbus [33746.348627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33746.350983] crct10dif_pclmul joydev [33746.355246] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33746.358296] crc32_pclmul ipmi_ssif [33746.362387] [33746.366480] ghash_clmulni_intel mei_me [33746.371696] [33746.371697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33746.375267] mei lpc_ich [33746.380227] ? queued_spin_lock_slowpath+0x16/0x2e0 [33746.383709] sha512_ssse3 acpi_ipmi [33746.385807] do_raw_spin_lock+0xb5/0xc0 [33746.389638] mfd_core wmi [33746.391736] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33746.397038] ipmi_si ipmi_devintf [33746.399573] try_get_cap_refs+0x159/0x880 [ceph] [33746.404439] ipmi_msghandler ioatdma acpi_pad [33746.407929] __ceph_get_caps+0x4a1/0x760 [ceph] [33746.411754] acpi_power_meter [33746.414375] ? __file_remove_privs+0x89/0x110 [33746.419155] ixgbe igb [33746.422480] ? wake_bit_function+0x70/0x70 [33746.427086] mdio crc32c_intel [33746.431439] ? ceph_write_iter+0x260/0x770 [ceph] [33746.435962] nvme ptp [33746.438924] ceph_write_iter+0x260/0x770 [ceph] [33746.443275] nvme_core i2c_algo_bit [33746.445637] ? lock_is_held_type+0xaa/0x120 [33746.449724] pps_core dca [33746.452778] vfs_write+0x324/0x4f0 [33746.457481] fuse [33746.459754] ksys_pwrite64+0x65/0xa0 [33746.464276] irq event stamp: 190976282 [33746.467759] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33746.471937] hardirqs last enabled at (190976281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33746.474556] do_syscall_64+0x38/0x80 [33746.477951] hardirqs last disabled at (190976282): [] sysvec_apic_timer_interrupt+0xe/0x90 [33746.479876] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33746.483462] softirqs last enabled at (190976280): [] __do_softirq+0x2be/0x371 [33746.487207] RIP: 0033:0x7f48f0eeb93b [33746.492338] softirqs last disabled at (190976275): [] irq_exit_rcu+0xb0/0xd0 [33746.502581] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33746.506154] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33746.515964] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33746.521008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33746.529778] ORIG_RAX: 0000000000000012 [33746.533350] Workqueue: events delayed_work [ceph] [33746.541948] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33746.560685] [33746.570063] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33746.575279] RIP: 0010:lock_release+0x14e/0x260 [33746.582750] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33746.586581] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [33746.591279] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33746.598403] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [33746.599894] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33746.607018] [33746.607018] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [33746.611475] [33746.618596] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [33746.674732] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [33746.681856] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [33746.688980] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [33746.696105] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33746.704188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33746.709928] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33746.717053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33746.724187] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33746.731308] Call Trace: [33746.733754] [33746.735764] ? watchdog_timer_fn+0x1b4/0x220 [33746.740030] ? __hrtimer_run_queues+0x1d3/0x360 [33746.744557] ? hrtimer_interrupt+0x104/0x240 [33746.748824] ? __local_bh_enable+0x50/0x60 [33746.752917] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33746.758134] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33746.763092] [33746.765190] [33746.767286] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33746.772589] ? igrab+0x43/0x50 [33746.775642] ? lock_release+0x14e/0x260 [33746.779472] ? lock_release+0xc5/0x260 [33746.783218] _raw_spin_unlock+0x1b/0x40 [33746.787056] igrab+0x43/0x50 [33746.789934] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33746.795084] delayed_work+0x1d8/0x280 [ceph] [33746.799362] ? process_scheduled_works+0x22a/0x510 [33746.804145] process_scheduled_works+0x2a4/0x510 [33746.808760] worker_thread+0x188/0x340 [33746.812509] ? rescuer_thread+0x310/0x310 [33746.816514] kthread+0xf5/0x130 [33746.819651] ? kthread_complete_and_exit+0x20/0x20 [33746.824436] ret_from_fork+0x31/0x40 [33746.828015] ? kthread_complete_and_exit+0x20/0x20 [33746.832799] ret_from_fork_asm+0x11/0x20 [33746.836722] [33774.051287] watchdog: BUG: soft lockup - CPU#0 stuck for 20340s! [dbench:78683] [33774.058593] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33774.063286] watchdog: BUG: soft lockup - CPU#7 stuck for 29337s! [kworker/7:4:78808] [33774.114043] irq event stamp: 58219846 [33774.114044] hardirqs last enabled at (58219845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33774.121774] Modules linked in: [33774.125430] hardirqs last disabled at (58219846): [] sysvec_apic_timer_interrupt+0xe/0x90 [33774.135587] ceph [33774.138640] softirqs last enabled at (58219840): [] __do_softirq+0x2be/0x371 [33774.148363] libceph [33774.150286] softirqs last disabled at (58219835): [] irq_exit_rcu+0xb0/0xd0 [33774.158971] dns_resolver [33774.161157] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33774.169683] fscache netfs [33774.172299] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33774.181246] veth nft_chain_nat [33774.183963] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33774.191437] xt_MASQUERADE nf_nat [33774.194576] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33774.200052] nf_conntrack nf_defrag_ipv6 [33774.203362] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33774.222101] nf_defrag_ipv4 [33774.226018] [33774.226019] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33774.231236] nft_compat [33774.234025] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33774.235517] nf_tables [33774.242642] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33774.245084] nfnetlink bridge [33774.252209] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33774.254566] stp llc [33774.261691] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33774.264654] xfs libcrc32c [33774.271795] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33774.273981] sunrpc [33774.281105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33774.283809] intel_rapl_msr intel_rapl_common [33774.291886] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33774.293982] sb_edac x86_pkg_temp_thermal [33774.299720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33774.304071] coretemp kvm_intel [33774.311194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33774.315200] kvm [33774.322323] Call Trace: [33774.322326] [33774.325462] irqbypass i2c_i801 [33774.332587] ? watchdog_timer_fn+0x1b4/0x220 [33774.334423] iTCO_wdt [33774.336867] ? __hrtimer_run_queues+0x1d3/0x360 [33774.338877] iTCO_vendor_support i2c_smbus [33774.342017] ? hrtimer_interrupt+0x104/0x240 [33774.346279] crct10dif_pclmul [33774.348550] ? __local_bh_enable+0x50/0x60 [33774.353074] joydev crc32_pclmul [33774.357168] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33774.361429] ipmi_ssif ghash_clmulni_intel [33774.364393] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33774.368483] mei_me mei [33774.371708] [33774.376925] lpc_ich [33774.381015] [33774.381017] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33774.385971] sha512_ssse3 acpi_ipmi [33774.388421] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33774.390513] mfd_core wmi [33774.392701] do_raw_spin_lock+0xb5/0xc0 [33774.394794] ipmi_si [33774.400102] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33774.403582] ipmi_devintf ipmi_msghandler ioatdma [33774.408458] try_get_cap_refs+0x159/0x880 [ceph] [33774.411071] acpi_pad acpi_power_meter [33774.414908] __ceph_get_caps+0x4a1/0x760 [ceph] [33774.417085] ixgbe igb [33774.421876] ? __file_remove_privs+0x89/0x110 [33774.426567] mdio crc32c_intel [33774.431179] ? wake_bit_function+0x70/0x70 [33774.434923] nvme [33774.439450] ? ceph_write_iter+0x260/0x770 [ceph] [33774.441804] ptp nvme_core [33774.446155] ceph_write_iter+0x260/0x770 [ceph] [33774.449205] i2c_algo_bit pps_core [33774.453300] ? lock_is_held_type+0xaa/0x120 [33774.455221] dca fuse [33774.459920] vfs_write+0x324/0x4f0 [33774.462622] [33774.462623] irq event stamp: 191197086 [33774.467152] ksys_pwrite64+0x65/0xa0 [33774.470543] hardirqs last enabled at (191197085): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33774.474721] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33774.476992] hardirqs last disabled at (191197086): [] sysvec_apic_timer_interrupt+0xe/0x90 [33774.480390] do_syscall_64+0x38/0x80 [33774.481879] softirqs last enabled at (191197084): [] __do_softirq+0x2be/0x371 [33774.485625] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33774.489193] softirqs last disabled at (191197079): [] irq_exit_rcu+0xb0/0xd0 [33774.499440] RIP: 0033:0x7f48f0eeb93b [33774.504570] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33774.514380] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33774.517950] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33774.526721] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33774.531766] Workqueue: events delayed_work [ceph] [33774.540363] ORIG_RAX: 0000000000000012 [33774.543933] [33774.543934] RIP: 0010:lock_acquire+0x86/0x2b0 [33774.553313] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33774.572050] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [33774.579519] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33774.584737] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [33774.589435] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33774.593265] [33774.593266] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [33774.594756] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33774.599107] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [33774.606232] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33774.624968] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [33774.632100] [33774.637311] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [33774.690856] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33774.697986] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33774.706063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33774.711802] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33774.718926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33774.726051] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33774.733176] Call Trace: [33774.735621] [33774.737631] ? watchdog_timer_fn+0x1b4/0x220 [33774.741894] ? __hrtimer_run_queues+0x1d3/0x360 [33774.746422] ? hrtimer_interrupt+0x104/0x240 [33774.750693] ? __local_bh_enable+0x50/0x60 [33774.754792] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33774.760016] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33774.764983] [33774.767080] [33774.769177] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33774.774484] ? lock_acquire+0x86/0x2b0 [33774.778236] ? igrab+0x43/0x50 [33774.781284] ? lock_release+0xc5/0x260 [33774.785030] _raw_spin_lock+0x30/0x40 [33774.788694] ? igrab+0x1d/0x50 [33774.791746] igrab+0x1d/0x50 [33774.794625] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33774.799773] delayed_work+0x1d8/0x280 [ceph] [33774.804055] ? process_scheduled_works+0x22a/0x510 [33774.808845] process_scheduled_works+0x2a4/0x510 [33774.813461] worker_thread+0x188/0x340 [33774.817210] ? rescuer_thread+0x310/0x310 [33774.821213] kthread+0xf5/0x130 [33774.824361] ? kthread_complete_and_exit+0x20/0x20 [33774.829151] ret_from_fork+0x31/0x40 [33774.832731] ? kthread_complete_and_exit+0x20/0x20 [33774.837514] ret_from_fork_asm+0x11/0x20 [33774.841437] [33802.051773] watchdog: BUG: soft lockup - CPU#0 stuck for 20366s! [dbench:78683] [33802.059071] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33802.063773] watchdog: BUG: soft lockup - CPU#7 stuck for 29363s! [kworker/7:4:78808] [33802.114513] irq event stamp: 58278014 [33802.122243] Modules linked in: ceph libceph [33802.125901] hardirqs last enabled at (58278013): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33802.130077] dns_resolver fscache netfs [33802.140235] hardirqs last disabled at (58278014): [] sysvec_apic_timer_interrupt+0xe/0x90 [33802.144066] veth nft_chain_nat xt_MASQUERADE nf_nat [33802.153790] softirqs last enabled at (58277892): [] __do_softirq+0x2be/0x371 [33802.158746] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [33802.167447] softirqs last disabled at (58277887): [] irq_exit_rcu+0xb0/0xd0 [33802.173621] nf_tables nfnetlink bridge [33802.182132] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33802.185963] stp llc [33802.194907] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33802.194908] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33802.197091] xfs libcrc32c [33802.204559] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33802.210037] sunrpc [33802.212742] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33802.231481] intel_rapl_msr intel_rapl_common [33802.233577] [33802.233578] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33802.238794] sb_edac x86_pkg_temp_thermal [33802.243145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33802.244654] coretemp kvm_intel [33802.251776] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33802.255781] kvm irqbypass [33802.262905] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33802.266042] i2c_i801 [33802.273172] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33802.275879] iTCO_wdt iTCO_vendor_support [33802.283004] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33802.285275] i2c_smbus [33802.292398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33802.296403] crct10dif_pclmul joydev [33802.304481] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33802.306838] crc32_pclmul ipmi_ssif [33802.312575] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33802.316147] ghash_clmulni_intel mei_me [33802.323288] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33802.326772] mei lpc_ich [33802.333896] Call Trace: [33802.333898] [33802.337726] sha512_ssse3 acpi_ipmi [33802.344852] ? watchdog_timer_fn+0x1b4/0x220 [33802.347381] mfd_core wmi [33802.349826] ? __hrtimer_run_queues+0x1d3/0x360 [33802.351837] ipmi_si ipmi_devintf [33802.355323] ? hrtimer_interrupt+0x104/0x240 [33802.359583] ipmi_msghandler ioatdma [33802.362202] ? __local_bh_enable+0x50/0x60 [33802.366725] acpi_pad acpi_power_meter [33802.370038] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33802.374301] ixgbe igb [33802.377873] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33802.381963] mdio crc32c_intel nvme [33802.385708] [33802.390923] ptp nvme_core [33802.393281] [33802.398239] i2c_algo_bit pps_core [33802.401723] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33802.403818] dca fuse [33802.406526] ? queued_spin_lock_slowpath+0x16/0x2e0 [33802.408621] [33802.408622] irq event stamp: 191417714 [33802.412019] do_raw_spin_lock+0xb5/0xc0 [33802.417322] hardirqs last enabled at (191417713): [] asm_common_interrupt+0x26/0x40 [33802.419595] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33802.424464] hardirqs last disabled at (191417714): [] sysvec_apic_timer_interrupt+0xe/0x90 [33802.425957] try_get_cap_refs+0x159/0x880 [ceph] [33802.429700] softirqs last enabled at (191417712): [] __do_softirq+0x2be/0x371 [33802.433534] __ceph_get_caps+0x4a1/0x760 [ceph] [33802.442820] softirqs last disabled at (191417707): [] irq_exit_rcu+0xb0/0xd0 [33802.447608] ? __file_remove_privs+0x89/0x110 [33802.457414] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33802.462026] ? wake_bit_function+0x70/0x70 [33802.470797] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33802.475323] ? ceph_write_iter+0x260/0x770 [ceph] [33802.483920] Workqueue: events delayed_work [ceph] [33802.488270] ceph_write_iter+0x260/0x770 [ceph] [33802.497647] [33802.501742] ? lock_is_held_type+0xaa/0x120 [33802.509209] RIP: 0010:lock_acquire+0xf0/0x2b0 [33802.513908] vfs_write+0x324/0x4f0 [33802.518602] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33802.523130] ksys_pwrite64+0x65/0xa0 [33802.524617] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33802.528795] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33802.533145] [33802.533146] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33802.536544] do_syscall_64+0x38/0x80 [33802.555281] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33802.558853] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33802.564069] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33802.569200] RIP: 0033:0x7f48f0eeb93b [33802.570690] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33802.577816] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33802.581386] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33802.588509] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33802.593554] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33802.600677] ORIG_RAX: 0000000000000012 [33802.604248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33802.611373] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33802.630110] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33802.637235] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33802.642464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33802.650538] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33802.654368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33802.660107] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33802.667229] Call Trace: [33802.667231] [33802.674354] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33802.681478] ? watchdog_timer_fn+0x1b4/0x220 [33802.688609] [33802.695727] ? __hrtimer_run_queues+0x1d3/0x360 [33802.732530] ? hrtimer_interrupt+0x104/0x240 [33802.736798] ? __local_bh_enable+0x50/0x60 [33802.740894] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33802.746118] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33802.751075] [33802.753178] [33802.755278] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33802.760585] ? lock_acquire+0xf0/0x2b0 [33802.764334] ? igrab+0x43/0x50 [33802.767385] ? lock_release+0xc5/0x260 [33802.771130] _raw_spin_lock+0x30/0x40 [33802.774787] ? igrab+0x1d/0x50 [33802.777838] igrab+0x1d/0x50 [33802.780716] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33802.785865] delayed_work+0x1d8/0x280 [ceph] [33802.790146] ? process_scheduled_works+0x22a/0x510 [33802.794935] process_scheduled_works+0x2a4/0x510 [33802.799551] worker_thread+0x188/0x340 [33802.803302] ? rescuer_thread+0x310/0x310 [33802.807304] kthread+0xf5/0x130 [33802.810457] ? kthread_complete_and_exit+0x20/0x20 [33802.815245] ret_from_fork+0x31/0x40 [33802.818821] ? kthread_complete_and_exit+0x20/0x20 [33802.823605] ret_from_fork_asm+0x11/0x20 [33802.827529] [33830.052259] watchdog: BUG: soft lockup - CPU#0 stuck for 20392s! [dbench:78683] [33830.059567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33830.064258] watchdog: BUG: soft lockup - CPU#7 stuck for 29389s! [kworker/7:4:78808] [33830.115016] irq event stamp: 58336066 [33830.122748] Modules linked in: ceph libceph dns_resolver fscache netfs veth [33830.126406] hardirqs last enabled at (58336065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33830.133364] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [33830.143521] hardirqs last disabled at (58336066): [] sysvec_apic_timer_interrupt+0xe/0x90 [33830.151773] nft_compat nf_tables nfnetlink bridge stp llc xfs [33830.161497] softirqs last enabled at (58336050): [] __do_softirq+0x2be/0x371 [33830.167320] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [33830.176006] softirqs last disabled at (58336045): [] irq_exit_rcu+0xb0/0xd0 [33830.184344] coretemp kvm_intel kvm irqbypass [33830.192854] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33830.192855] i2c_i801 [33830.192856] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33830.197205] iTCO_wdt [33830.206151] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33830.208419] iTCO_vendor_support i2c_smbus [33830.215890] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33830.218161] crct10dif_pclmul joydev [33830.223654] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33830.227746] crc32_pclmul [33830.246485] [33830.246486] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33830.250055] ipmi_ssif ghash_clmulni_intel [33830.255273] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33830.257891] mei_me [33830.259379] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33830.266505] mei lpc_ich [33830.270594] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33830.277718] sha512_ssse3 acpi_ipmi [33830.279817] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33830.286941] mfd_core wmi [33830.289473] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33830.296595] ipmi_si [33830.300079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33830.307204] ipmi_devintf [33830.309822] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33830.317900] ipmi_msghandler [33830.320082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33830.325821] ioatdma acpi_pad [33830.328437] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33830.335562] acpi_power_meter [33830.338440] Call Trace: [33830.338443] [33830.345562] ixgbe igb [33830.348529] ? watchdog_timer_fn+0x1b4/0x220 [33830.355656] mdio [33830.358626] ? __hrtimer_run_queues+0x1d3/0x360 [33830.361069] crc32c_intel nvme [33830.363082] ? hrtimer_interrupt+0x104/0x240 [33830.365436] ptp nvme_core [33830.369701] ? __local_bh_enable+0x50/0x60 [33830.371625] i2c_algo_bit pps_core [33830.376151] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33830.379199] dca fuse [33830.383465] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33830.386168] [33830.386168] irq event stamp: 191638570 [33830.390257] [33830.393655] hardirqs last enabled at (191638569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33830.398873] [33830.401143] hardirqs last disabled at (191638570): [] sysvec_apic_timer_interrupt+0xe/0x90 [33830.406102] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33830.407592] softirqs last enabled at (191638568): [] __do_softirq+0x2be/0x371 [33830.411340] ? queued_spin_lock_slowpath+0x16/0x2e0 [33830.413433] softirqs last disabled at (191638563): [] irq_exit_rcu+0xb0/0xd0 [33830.423678] do_raw_spin_lock+0xb5/0xc0 [33830.425774] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33830.435588] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33830.440889] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33830.449664] try_get_cap_refs+0x159/0x880 [ceph] [33830.454532] Workqueue: events delayed_work [ceph] [33830.463135] __ceph_get_caps+0x4a1/0x760 [ceph] [33830.466960] [33830.466960] RIP: 0010:lock_acquire+0xf0/0x2b0 [33830.476341] ? __file_remove_privs+0x89/0x110 [33830.481121] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33830.488593] ? wake_bit_function+0x70/0x70 [33830.493203] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33830.497903] ? ceph_write_iter+0x260/0x770 [ceph] [33830.502424] [33830.502424] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33830.503915] ceph_write_iter+0x260/0x770 [ceph] [33830.508265] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33830.512620] ? lock_is_held_type+0xaa/0x120 [33830.531354] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33830.535447] vfs_write+0x324/0x4f0 [33830.540663] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33830.545366] ksys_pwrite64+0x65/0xa0 [33830.546850] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33830.546851] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33830.553976] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33830.558499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33830.565623] do_syscall_64+0x38/0x80 [33830.569799] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33830.576941] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33830.580338] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33830.587463] RIP: 0033:0x7f48f0eeb93b [33830.591033] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33830.598159] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33830.606236] Call Trace: [33830.606238] [33830.611366] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33830.617105] ? watchdog_timer_fn+0x1b4/0x220 [33830.620674] ORIG_RAX: 0000000000000012 [33830.627801] ? __hrtimer_run_queues+0x1d3/0x360 [33830.632843] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33830.639970] ? hrtimer_interrupt+0x104/0x240 [33830.643537] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33830.650661] ? __local_bh_enable+0x50/0x60 [33830.669400] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33830.671846] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33830.673854] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33830.679072] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33830.683336] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33830.687167] [33830.691696] [33830.698815] [33830.752228] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33830.757537] ? lock_acquire+0xf0/0x2b0 [33830.761288] ? igrab+0x43/0x50 [33830.764336] ? lock_release+0xc5/0x260 [33830.768081] _raw_spin_lock+0x30/0x40 [33830.771738] ? igrab+0x1d/0x50 [33830.774789] igrab+0x1d/0x50 [33830.777668] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33830.782817] delayed_work+0x1d8/0x280 [ceph] [33830.787098] ? process_scheduled_works+0x22a/0x510 [33830.791887] process_scheduled_works+0x2a4/0x510 [33830.796503] worker_thread+0x188/0x340 [33830.800252] ? rescuer_thread+0x310/0x310 [33830.804255] kthread+0xf5/0x130 [33830.807394] ? kthread_complete_and_exit+0x20/0x20 [33830.812178] ret_from_fork+0x31/0x40 [33830.815757] ? kthread_complete_and_exit+0x20/0x20 [33830.820542] ret_from_fork_asm+0x11/0x20 [33830.824472] [33858.052732] watchdog: BUG: soft lockup - CPU#0 stuck for 20418s! [dbench:78683] [33858.060028] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33858.064732] watchdog: BUG: soft lockup - CPU#7 stuck for 29415s! [kworker/7:4:78808] [33858.115469] irq event stamp: 58394114 [33858.115471] hardirqs last enabled at (58394113): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33858.123201] Modules linked in: [33858.126859] hardirqs last disabled at (58394114): [] sysvec_apic_timer_interrupt+0xe/0x90 [33858.137016] ceph [33858.140066] softirqs last enabled at (58394096): [] __do_softirq+0x2be/0x371 [33858.149790] libceph [33858.151716] softirqs last disabled at (58394091): [] irq_exit_rcu+0xb0/0xd0 [33858.160407] dns_resolver fscache [33858.162591] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33858.171103] netfs veth [33858.174430] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33858.174431] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33858.183375] nft_chain_nat xt_MASQUERADE [33858.185819] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33858.193289] nf_nat [33858.198767] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33858.202685] nf_conntrack nf_defrag_ipv6 [33858.221423] [33858.223519] nf_defrag_ipv4 [33858.228736] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33858.232654] nft_compat nf_tables [33858.234144] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33858.236935] nfnetlink bridge [33858.244060] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33858.247370] stp llc [33858.254494] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33858.257458] xfs libcrc32c [33858.264582] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33858.266768] sunrpc intel_rapl_msr [33858.273892] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33858.276595] intel_rapl_common sb_edac [33858.283719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33858.287129] x86_pkg_temp_thermal [33858.295202] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33858.298945] coretemp kvm_intel [33858.304683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33858.307994] kvm irqbypass [33858.315129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33858.318265] i2c_i801 [33858.325406] Call Trace: [33858.328127] iTCO_wdt iTCO_vendor_support [33858.335252] [33858.337522] i2c_smbus [33858.339967] ? watchdog_timer_fn+0x1b4/0x220 [33858.343972] crct10dif_pclmul joydev [33858.345983] ? __hrtimer_run_queues+0x1d3/0x360 [33858.348338] crc32_pclmul [33858.352606] ? hrtimer_interrupt+0x104/0x240 [33858.356174] ipmi_ssif [33858.360699] ? __local_bh_enable+0x50/0x60 [33858.363315] ghash_clmulni_intel mei_me [33858.367582] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33858.369937] mei lpc_ich [33858.374027] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33858.377858] sha512_ssse3 acpi_ipmi [33858.383076] [33858.385606] mfd_core wmi [33858.390563] [33858.394047] ipmi_si ipmi_devintf [33858.396146] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33858.398763] ipmi_msghandler [33858.400865] ? queued_spin_lock_slowpath+0x16/0x2e0 [33858.404170] ioatdma [33858.409477] do_raw_spin_lock+0xb5/0xc0 [33858.412351] acpi_pad acpi_power_meter [33858.417224] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33858.419407] ixgbe igb [33858.423256] try_get_cap_refs+0x159/0x880 [ceph] [33858.426999] mdio crc32c_intel [33858.431787] __ceph_get_caps+0x4a1/0x760 [ceph] [33858.434139] nvme [33858.438754] ? __file_remove_privs+0x89/0x110 [33858.441801] ptp nvme_core [33858.446325] ? wake_bit_function+0x70/0x70 [33858.448249] i2c_algo_bit pps_core [33858.452602] ? ceph_write_iter+0x260/0x770 [ceph] [33858.455303] dca fuse [33858.459397] ceph_write_iter+0x260/0x770 [ceph] [33858.462794] [33858.467494] ? lock_is_held_type+0xaa/0x120 [33858.469760] irq event stamp: 191859428 [33858.474288] vfs_write+0x324/0x4f0 [33858.475776] hardirqs last enabled at (191859427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33858.479958] ksys_pwrite64+0x65/0xa0 [33858.483698] hardirqs last disabled at (191859428): [] sysvec_apic_timer_interrupt+0xe/0x90 [33858.487096] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33858.497339] softirqs last enabled at (191859426): [] __do_softirq+0x2be/0x371 [33858.500911] do_syscall_64+0x38/0x80 [33858.510721] softirqs last disabled at (191859421): [] irq_exit_rcu+0xb0/0xd0 [33858.515852] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33858.524622] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33858.528194] RIP: 0033:0x7f48f0eeb93b [33858.536789] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33858.541833] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33858.551212] Workqueue: events delayed_work [ceph] [33858.554783] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33858.562253] [33858.562254] RIP: 0010:lock_acquire+0xf0/0x2b0 [33858.580991] ORIG_RAX: 0000000000000012 [33858.580992] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33858.585688] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33858.590905] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33858.592395] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33858.596765] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33858.600595] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33858.607718] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33858.626456] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33858.633582] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33858.638800] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33858.645929] [33858.653048] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33858.690868] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33858.697994] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33858.706071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33858.711808] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33858.718932] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33858.726057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33858.733181] Call Trace: [33858.735624] [33858.737637] ? watchdog_timer_fn+0x1b4/0x220 [33858.741902] ? __hrtimer_run_queues+0x1d3/0x360 [33858.746427] ? hrtimer_interrupt+0x104/0x240 [33858.750696] ? __local_bh_enable+0x50/0x60 [33858.754789] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33858.760004] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33858.764962] [33858.767061] [33858.769156] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33858.774465] ? lock_acquire+0xf0/0x2b0 [33858.778215] ? igrab+0x43/0x50 [33858.781266] ? lock_release+0xc5/0x260 [33858.785009] _raw_spin_lock+0x30/0x40 [33858.788665] ? igrab+0x1d/0x50 [33858.791717] igrab+0x1d/0x50 [33858.794596] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33858.799746] delayed_work+0x1d8/0x280 [ceph] [33858.804026] ? process_scheduled_works+0x22a/0x510 [33858.808817] process_scheduled_works+0x2a4/0x510 [33858.813431] worker_thread+0x188/0x340 [33858.817181] ? rescuer_thread+0x310/0x310 [33858.821183] kthread+0xf5/0x130 [33858.824321] ? kthread_complete_and_exit+0x20/0x20 [33858.829107] ret_from_fork+0x31/0x40 [33858.832701] ? kthread_complete_and_exit+0x20/0x20 [33858.837485] ret_from_fork_asm+0x11/0x20 [33858.841424] [33886.053206] watchdog: BUG: soft lockup - CPU#0 stuck for 20444s! [dbench:78683] [33886.060509] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33886.065205] watchdog: BUG: soft lockup - CPU#7 stuck for 29441s! [kworker/7:4:78808] [33886.115956] irq event stamp: 58452320 [33886.115958] hardirqs last enabled at (58452319): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33886.123688] Modules linked in: [33886.127346] hardirqs last disabled at (58452320): [] sysvec_apic_timer_interrupt+0xe/0x90 [33886.137504] ceph [33886.140553] softirqs last enabled at (58452224): [] __do_softirq+0x2be/0x371 [33886.150278] libceph [33886.152202] softirqs last disabled at (58452219): [] irq_exit_rcu+0xb0/0xd0 [33886.160886] dns_resolver [33886.163072] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33886.171599] fscache netfs [33886.174215] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33886.183160] veth nft_chain_nat [33886.185881] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33886.193352] xt_MASQUERADE nf_nat [33886.196491] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33886.201967] nf_conntrack nf_defrag_ipv6 [33886.205278] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33886.224016] nf_defrag_ipv4 [33886.227931] [33886.227932] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33886.233151] nft_compat [33886.235941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33886.237430] nf_tables nfnetlink [33886.244555] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33886.246999] bridge [33886.254124] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33886.257349] stp [33886.264472] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33886.266568] llc xfs [33886.273694] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33886.275531] libcrc32c sunrpc [33886.282654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33886.284840] intel_rapl_msr [33886.292917] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33886.295891] intel_rapl_common sb_edac [33886.301626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33886.304418] x86_pkg_temp_thermal [33886.311542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33886.315284] coretemp kvm_intel [33886.322411] Call Trace: [33886.322413] [33886.325721] kvm irqbypass [33886.332847] ? watchdog_timer_fn+0x1b4/0x220 [33886.335983] i2c_i801 iTCO_wdt [33886.338428] ? __hrtimer_run_queues+0x1d3/0x360 [33886.340437] iTCO_vendor_support i2c_smbus [33886.343143] ? hrtimer_interrupt+0x104/0x240 [33886.347404] crct10dif_pclmul joydev [33886.350455] ? __local_bh_enable+0x50/0x60 [33886.354980] crc32_pclmul ipmi_ssif [33886.359073] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33886.363334] ghash_clmulni_intel [33886.366906] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33886.370997] mei_me [33886.374480] [33886.379696] mei lpc_ich [33886.382920] [33886.387884] sha512_ssse3 [33886.389987] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33886.392082] acpi_ipmi mfd_core [33886.394616] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33886.396711] wmi ipmi_si [33886.399329] do_raw_spin_lock+0xb5/0xc0 [33886.404632] ipmi_devintf ipmi_msghandler [33886.407772] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33886.412639] ioatdma acpi_pad [33886.415175] try_get_cap_refs+0x159/0x880 [ceph] [33886.419002] acpi_power_meter [33886.423011] __ceph_get_caps+0x4a1/0x760 [ceph] [33886.427789] ixgbe igb [33886.430757] ? __file_remove_privs+0x89/0x110 [33886.435364] mdio crc32c_intel [33886.438329] ? wake_bit_function+0x70/0x70 [33886.442852] nvme [33886.445213] ? ceph_write_iter+0x260/0x770 [ceph] [33886.449561] ptp nvme_core [33886.452612] ceph_write_iter+0x260/0x770 [ceph] [33886.456704] i2c_algo_bit [33886.458631] ? lock_is_held_type+0xaa/0x120 [33886.463323] pps_core dca [33886.466029] vfs_write+0x324/0x4f0 [33886.470552] fuse [33886.473174] ksys_pwrite64+0x65/0xa0 [33886.477347] irq event stamp: 192080068 [33886.479965] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33886.483361] hardirqs last enabled at (192080067): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33886.485288] do_syscall_64+0x38/0x80 [33886.488856] hardirqs last disabled at (192080068): [] sysvec_apic_timer_interrupt+0xe/0x90 [33886.492607] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33886.497738] softirqs last enabled at (192080066): [] __do_softirq+0x2be/0x371 [33886.507985] RIP: 0033:0x7f48f0eeb93b [33886.511554] softirqs last disabled at (192080061): [] irq_exit_rcu+0xb0/0xd0 [33886.521366] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33886.526410] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33886.535181] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33886.538750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33886.547349] ORIG_RAX: 0000000000000012 [33886.566087] Workqueue: events delayed_work [ceph] [33886.575464] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33886.580682] [33886.580683] RIP: 0010:lock_acquire+0xf0/0x2b0 [33886.588152] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33886.591983] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33886.596679] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33886.603804] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33886.605294] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33886.609644] [33886.609645] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33886.616770] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33886.635508] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33886.642638] [33886.647850] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33886.687161] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33886.694286] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33886.701411] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33886.709488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33886.715225] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33886.722349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33886.729474] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33886.736610] Call Trace: [33886.739060] [33886.741071] ? watchdog_timer_fn+0x1b4/0x220 [33886.745335] ? __hrtimer_run_queues+0x1d3/0x360 [33886.749861] ? hrtimer_interrupt+0x104/0x240 [33886.754121] ? __local_bh_enable+0x50/0x60 [33886.758216] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33886.763439] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33886.768396] [33886.770493] [33886.772608] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33886.777918] ? lock_acquire+0xf0/0x2b0 [33886.781666] ? igrab+0x43/0x50 [33886.784716] ? lock_release+0xc5/0x260 [33886.788463] _raw_spin_lock+0x30/0x40 [33886.792126] ? igrab+0x1d/0x50 [33886.795176] igrab+0x1d/0x50 [33886.798057] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33886.803205] delayed_work+0x1d8/0x280 [ceph] [33886.807487] ? process_scheduled_works+0x22a/0x510 [33886.812276] process_scheduled_works+0x2a4/0x510 [33886.816903] worker_thread+0x188/0x340 [33886.820650] ? rescuer_thread+0x310/0x310 [33886.824654] kthread+0xf5/0x130 [33886.827800] ? kthread_complete_and_exit+0x20/0x20 [33886.832585] ret_from_fork+0x31/0x40 [33886.836182] ? kthread_complete_and_exit+0x20/0x20 [33886.840973] ret_from_fork_asm+0x11/0x20 [33886.844916] [33888.713617] rcu: INFO: rcu_preempt self-detected stall on CPU [33888.719377] rcu: 7-....: (21314642 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/21313306 fqs=10740633 [33888.730057] rcu: (t=21951713 jiffies g=1309609 q=621575 ncpus=8) [33888.736148] Sending NMI from CPU 7 to CPUs 0: [33888.740499] NMI backtrace for cpu 0 [33888.740501] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33888.740503] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33888.740503] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33888.740506] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33888.740508] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33888.740509] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33888.740510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33888.740510] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33888.740511] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33888.740512] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33888.740513] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33888.740514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33888.740514] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33888.740515] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33888.740516] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33888.740517] Call Trace: [33888.740517] [33888.740518] ? nmi_cpu_backtrace+0xad/0x120 [33888.740522] ? nmi_cpu_backtrace_handler+0x11/0x20 [33888.740523] ? nmi_handle+0xd4/0x260 [33888.740528] ? default_do_nmi+0x49/0xf0 [33888.740529] ? exc_nmi+0x11e/0x150 [33888.740530] ? end_repeat_nmi+0x16/0x67 [33888.740534] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33888.740537] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33888.740539] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33888.740541] [33888.740542] [33888.740543] do_raw_spin_lock+0xb5/0xc0 [33888.740545] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33888.740558] try_get_cap_refs+0x159/0x880 [ceph] [33888.740572] __ceph_get_caps+0x4a1/0x760 [ceph] [33888.740584] ? __file_remove_privs+0x89/0x110 [33888.740586] ? wake_bit_function+0x70/0x70 [33888.740588] ? ceph_write_iter+0x260/0x770 [ceph] [33888.740599] ceph_write_iter+0x260/0x770 [ceph] [33888.740609] ? lock_is_held_type+0xaa/0x120 [33888.740612] vfs_write+0x324/0x4f0 [33888.740617] ksys_pwrite64+0x65/0xa0 [33888.740620] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33888.740622] do_syscall_64+0x38/0x80 [33888.740642] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33888.740644] RIP: 0033:0x7f48f0eeb93b [33888.740645] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33888.740646] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [33888.740648] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33888.740649] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33888.740649] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33888.740650] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33888.740651] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33888.740654] [33888.741498] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33889.045024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33889.052497] Workqueue: events delayed_work [ceph] [33889.057220] RIP: 0010:lock_acquire+0xf0/0x2b0 [33889.061577] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33889.080315] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33889.085534] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33889.092656] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33889.099781] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33889.106906] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33889.114048] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33889.121172] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33889.129248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33889.134986] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33889.142111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33889.149233] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33889.156359] Call Trace: [33889.158803] [33889.160815] ? rcu_dump_cpu_stacks+0xd1/0x180 [33889.165175] ? rcu_sched_clock_irq+0x633/0x1390 [33889.169708] ? lock_release+0xc5/0x260 [33889.173458] ? lock_is_held_type+0xaa/0x120 [33889.177654] ? lock_is_held_type+0xaa/0x120 [33889.181833] ? tick_sched_do_timer+0xa0/0xa0 [33889.186103] ? rcu_read_lock_sched_held+0x56/0x90 [33889.190803] ? tick_sched_do_timer+0xa0/0xa0 [33889.195074] ? update_process_times+0x5f/0x90 [33889.199424] ? tick_sched_handle.isra.16+0x1e/0x50 [33889.204210] ? tick_sched_timer+0x6f/0x80 [33889.208223] ? __hrtimer_run_queues+0x1d3/0x360 [33889.212762] ? hrtimer_interrupt+0x104/0x240 [33889.217027] ? __local_bh_enable+0x50/0x60 [33889.221124] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33889.226344] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33889.231302] [33889.233397] [33889.235496] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33889.240811] ? lock_acquire+0xf0/0x2b0 [33889.244564] ? igrab+0x43/0x50 [33889.247622] ? lock_release+0xc5/0x260 [33889.251368] _raw_spin_lock+0x30/0x40 [33889.255031] ? igrab+0x1d/0x50 [33889.258081] igrab+0x1d/0x50 [33889.260959] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33889.266112] delayed_work+0x1d8/0x280 [ceph] [33889.270400] ? process_scheduled_works+0x22a/0x510 [33889.275189] process_scheduled_works+0x2a4/0x510 [33889.279812] worker_thread+0x188/0x340 [33889.283564] ? rescuer_thread+0x310/0x310 [33889.287575] kthread+0xf5/0x130 [33889.290712] ? kthread_complete_and_exit+0x20/0x20 [33889.295497] ret_from_fork+0x31/0x40 [33889.299075] ? kthread_complete_and_exit+0x20/0x20 [33889.303861] ret_from_fork_asm+0x11/0x20 [33889.307799] [33914.053678] watchdog: BUG: soft lockup - CPU#0 stuck for 20470s! [dbench:78683] [33914.060978] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33914.065678] watchdog: BUG: soft lockup - CPU#7 stuck for 29467s! [kworker/7:4:78808] [33914.116419] irq event stamp: 58510444 [33914.116420] hardirqs last enabled at (58510443): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33914.124150] Modules linked in: [33914.127807] hardirqs last disabled at (58510444): [] sysvec_apic_timer_interrupt+0xe/0x90 [33914.137965] ceph [33914.141014] softirqs last enabled at (58510372): [] __do_softirq+0x2be/0x371 [33914.150739] libceph dns_resolver [33914.152662] softirqs last disabled at (58510367): [] irq_exit_rcu+0xb0/0xd0 [33914.161352] fscache netfs [33914.164667] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33914.173179] veth nft_chain_nat [33914.175881] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33914.184825] xt_MASQUERADE nf_nat [33914.187963] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [33914.195435] nf_conntrack [33914.198745] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [33914.204223] nf_defrag_ipv6 nf_defrag_ipv4 [33914.206838] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33914.225578] nft_compat [33914.229669] [33914.229669] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33914.234885] nf_tables nfnetlink [33914.237330] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33914.238836] bridge stp [33914.245962] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33914.249185] llc xfs [33914.256311] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33914.258755] libcrc32c sunrpc [33914.265878] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33914.268075] intel_rapl_msr intel_rapl_common [33914.275205] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33914.278168] sb_edac x86_pkg_temp_thermal [33914.285292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33914.289643] coretemp kvm_intel [33914.297721] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33914.301723] kvm irqbypass [33914.307462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33914.310600] i2c_i801 iTCO_wdt [33914.317723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33914.320428] iTCO_vendor_support [33914.327551] Call Trace: [33914.330601] i2c_smbus crct10dif_pclmul [33914.337726] [33914.340950] joydev crc32_pclmul [33914.343395] ? watchdog_timer_fn+0x1b4/0x220 [33914.347224] ipmi_ssif ghash_clmulni_intel [33914.349238] ? __hrtimer_run_queues+0x1d3/0x360 [33914.352460] mei_me [33914.356728] ? hrtimer_interrupt+0x104/0x240 [33914.360816] mei lpc_ich [33914.365340] ? __local_bh_enable+0x50/0x60 [33914.367436] sha512_ssse3 acpi_ipmi [33914.371703] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33914.374231] mfd_core wmi [33914.378321] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33914.381805] ipmi_si [33914.387024] [33914.389640] ipmi_devintf ipmi_msghandler [33914.394597] [33914.396783] ioatdma acpi_pad [33914.398881] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33914.402882] acpi_power_meter ixgbe [33914.404985] ? queued_spin_lock_slowpath+0x2d/0x2e0 [33914.407945] igb mdio [33914.413251] do_raw_spin_lock+0xb5/0xc0 [33914.416733] crc32c_intel [33914.421604] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33914.423875] nvme ptp [33914.427706] try_get_cap_refs+0x159/0x880 [ceph] [33914.430322] nvme_core i2c_algo_bit [33914.435110] __ceph_get_caps+0x4a1/0x760 [ceph] [33914.437377] pps_core dca [33914.441992] ? __file_remove_privs+0x89/0x110 [33914.445471] fuse [33914.449996] ? wake_bit_function+0x70/0x70 [33914.452615] irq event stamp: 192296018 [33914.456967] ? ceph_write_iter+0x260/0x770 [ceph] [33914.458887] hardirqs last enabled at (192296017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33914.462980] ceph_write_iter+0x260/0x770 [ceph] [33914.466724] hardirqs last disabled at (192296018): [] sysvec_apic_timer_interrupt+0xe/0x90 [33914.471425] ? lock_is_held_type+0xaa/0x120 [33914.481665] softirqs last enabled at (192296016): [] __do_softirq+0x2be/0x371 [33914.486191] vfs_write+0x324/0x4f0 [33914.495999] softirqs last disabled at (192296011): [] irq_exit_rcu+0xb0/0xd0 [33914.500182] ksys_pwrite64+0x65/0xa0 [33914.508949] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33914.512345] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33914.520960] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33914.524532] do_syscall_64+0x38/0x80 [33914.533909] Workqueue: events delayed_work [ceph] [33914.539039] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33914.546511] [33914.546512] RIP: 0010:lock_acquire+0xf0/0x2b0 [33914.550090] RIP: 0033:0x7f48f0eeb93b [33914.554787] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33914.559832] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33914.561322] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33914.565672] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33914.569242] [33914.587981] ORIG_RAX: 0000000000000012 [33914.587982] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33914.606719] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33914.611934] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33914.617154] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33914.618644] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33914.622474] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33914.629599] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33914.636723] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33914.643848] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33914.650971] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33914.658100] [33914.665221] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33914.703996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33914.709734] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33914.716857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33914.723980] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33914.731104] Call Trace: [33914.733550] [33914.735560] ? watchdog_timer_fn+0x1b4/0x220 [33914.739825] ? __hrtimer_run_queues+0x1d3/0x360 [33914.744366] ? hrtimer_interrupt+0x104/0x240 [33914.748630] ? __local_bh_enable+0x50/0x60 [33914.752724] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33914.757947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33914.762905] [33914.765000] [33914.767098] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33914.772406] ? lock_acquire+0xf0/0x2b0 [33914.776158] ? igrab+0x43/0x50 [33914.779206] ? lock_release+0xc5/0x260 [33914.782953] _raw_spin_lock+0x30/0x40 [33914.786617] ? igrab+0x1d/0x50 [33914.789667] igrab+0x1d/0x50 [33914.792545] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33914.797696] delayed_work+0x1d8/0x280 [ceph] [33914.801975] ? process_scheduled_works+0x22a/0x510 [33914.806765] process_scheduled_works+0x2a4/0x510 [33914.811381] worker_thread+0x188/0x340 [33914.815131] ? rescuer_thread+0x310/0x310 [33914.819135] kthread+0xf5/0x130 [33914.822272] ? kthread_complete_and_exit+0x20/0x20 [33914.827056] ret_from_fork+0x31/0x40 [33914.830625] ? kthread_complete_and_exit+0x20/0x20 [33914.835411] ret_from_fork_asm+0x11/0x20 [33914.839333] [33942.054152] watchdog: BUG: soft lockup - CPU#0 stuck for 20497s! [dbench:78683] [33942.061458] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33942.066152] watchdog: BUG: soft lockup - CPU#7 stuck for 29494s! [kworker/7:4:78808] [33942.116908] irq event stamp: 58568624 [33942.116909] hardirqs last enabled at (58568623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33942.124636] Modules linked in: [33942.128294] hardirqs last disabled at (58568624): [] sysvec_apic_timer_interrupt+0xe/0x90 [33942.138451] ceph [33942.141504] softirqs last enabled at (58568548): [] __do_softirq+0x2be/0x371 [33942.151226] libceph [33942.153151] softirqs last disabled at (58568543): [] irq_exit_rcu+0xb0/0xd0 [33942.161834] dns_resolver fscache [33942.164019] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33942.172547] netfs [33942.175858] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33942.184802] veth nft_chain_nat [33942.186831] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33942.194301] xt_MASQUERADE nf_nat [33942.197440] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33942.202916] nf_conntrack nf_defrag_ipv6 [33942.206227] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33942.224963] nf_defrag_ipv4 [33942.228881] [33942.234100] nft_compat nf_tables [33942.236891] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33942.238380] nfnetlink [33942.241692] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33942.248827] bridge [33942.251181] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33942.258304] stp llc [33942.260402] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33942.267545] xfs libcrc32c [33942.269730] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33942.276852] sunrpc [33942.279557] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33942.286680] intel_rapl_msr [33942.288777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33942.296855] intel_rapl_common [33942.299645] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33942.305383] sb_edac x86_pkg_temp_thermal [33942.308434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33942.315558] coretemp [33942.319561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33942.326687] kvm_intel [33942.328956] Call Trace: [33942.336082] kvm irqbypass [33942.338441] [33942.340883] i2c_i801 iTCO_wdt [33942.343589] ? watchdog_timer_fn+0x1b4/0x220 [33942.345597] iTCO_vendor_support i2c_smbus [33942.348649] ? __hrtimer_run_queues+0x1d3/0x360 [33942.352914] crct10dif_pclmul joydev [33942.357007] ? hrtimer_interrupt+0x104/0x240 [33942.361546] crc32_pclmul [33942.365116] ? __local_bh_enable+0x50/0x60 [33942.369380] ipmi_ssif [33942.371999] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33942.376088] ghash_clmulni_intel [33942.378447] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33942.383661] mei_me mei [33942.386888] [33942.391843] lpc_ich sha512_ssse3 [33942.394288] [33942.396385] acpi_ipmi mfd_core [33942.399697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33942.401795] wmi [33942.404934] ? queued_spin_lock_slowpath+0x16/0x2e0 [33942.410236] ipmi_si [33942.412074] do_raw_spin_lock+0xb5/0xc0 [33942.416944] ipmi_devintf ipmi_msghandler [33942.419130] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33942.422958] ioatdma acpi_pad [33942.426965] try_get_cap_refs+0x159/0x880 [ceph] [33942.431746] acpi_power_meter ixgbe [33942.434715] __ceph_get_caps+0x4a1/0x760 [ceph] [33942.439320] igb mdio [33942.442808] ? __file_remove_privs+0x89/0x110 [33942.447328] crc32c_intel nvme [33942.449600] ? wake_bit_function+0x70/0x70 [33942.453950] ptp nvme_core [33942.457003] ? ceph_write_iter+0x260/0x770 [ceph] [33942.461091] i2c_algo_bit pps_core [33942.463795] ceph_write_iter+0x260/0x770 [ceph] [33942.468493] dca fuse [33942.471893] ? lock_is_held_type+0xaa/0x120 [33942.476416] [33942.476417] irq event stamp: 192516586 [33942.478687] vfs_write+0x324/0x4f0 [33942.482864] hardirqs last enabled at (192516585): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33942.484359] ksys_pwrite64+0x65/0xa0 [33942.488098] hardirqs last disabled at (192516586): [] sysvec_apic_timer_interrupt+0xe/0x90 [33942.491495] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33942.501740] softirqs last enabled at (192516584): [] __do_softirq+0x2be/0x371 [33942.505311] do_syscall_64+0x38/0x80 [33942.515122] softirqs last disabled at (192516579): [] irq_exit_rcu+0xb0/0xd0 [33942.520252] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33942.529024] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33942.532594] RIP: 0033:0x7f48f0eeb93b [33942.541190] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33942.546237] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33942.555613] Workqueue: events delayed_work [ceph] [33942.559184] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33942.566654] [33942.566655] RIP: 0010:lock_acquire+0xf0/0x2b0 [33942.585392] ORIG_RAX: 0000000000000012 [33942.590088] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33942.595305] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33942.596796] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33942.601147] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33942.604979] [33942.604979] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33942.623717] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33942.630854] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33942.636075] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33942.643200] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33942.644690] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33942.651814] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33942.651831] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33942.658960] [33942.666079] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33942.711963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33942.717700] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33942.724834] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33942.731956] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33942.739080] Call Trace: [33942.741525] [33942.743554] ? watchdog_timer_fn+0x1b4/0x220 [33942.747834] ? __hrtimer_run_queues+0x1d3/0x360 [33942.752363] ? hrtimer_interrupt+0x104/0x240 [33942.756631] ? __local_bh_enable+0x50/0x60 [33942.760723] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33942.765948] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33942.770905] [33942.773004] [33942.775101] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33942.780408] ? lock_acquire+0xf0/0x2b0 [33942.784160] ? igrab+0x43/0x50 [33942.787210] ? lock_release+0xc5/0x260 [33942.790954] _raw_spin_lock+0x30/0x40 [33942.794620] ? igrab+0x1d/0x50 [33942.797668] igrab+0x1d/0x50 [33942.800562] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33942.805716] delayed_work+0x1d8/0x280 [ceph] [33942.809996] ? process_scheduled_works+0x22a/0x510 [33942.814786] process_scheduled_works+0x2a4/0x510 [33942.819401] worker_thread+0x188/0x340 [33942.823151] ? rescuer_thread+0x310/0x310 [33942.827153] kthread+0xf5/0x130 [33942.830301] ? kthread_complete_and_exit+0x20/0x20 [33942.835083] ret_from_fork+0x31/0x40 [33942.838664] ? kthread_complete_and_exit+0x20/0x20 [33942.843447] ret_from_fork_asm+0x11/0x20 [33942.847379] [33970.054625] watchdog: BUG: soft lockup - CPU#0 stuck for 20523s! [dbench:78683] [33970.061928] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33970.066625] watchdog: BUG: soft lockup - CPU#7 stuck for 29520s! [kworker/7:4:78808] [33970.117369] irq event stamp: 58626772 [33970.117370] hardirqs last enabled at (58626771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33970.125099] Modules linked in: [33970.128755] hardirqs last disabled at (58626772): [] sysvec_apic_timer_interrupt+0xe/0x90 [33970.138913] ceph [33970.141964] softirqs last enabled at (58626696): [] __do_softirq+0x2be/0x371 [33970.151688] libceph [33970.153611] softirqs last disabled at (58626691): [] irq_exit_rcu+0xb0/0xd0 [33970.162309] dns_resolver fscache [33970.164491] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33970.173017] netfs veth [33970.176329] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33970.176331] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [33970.185273] nft_chain_nat [33970.187717] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [33970.195187] xt_MASQUERADE nf_nat [33970.200665] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33970.203368] nf_conntrack nf_defrag_ipv6 [33970.222106] [33970.222107] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33970.225418] nf_defrag_ipv4 nft_compat [33970.230634] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33970.234552] nf_tables [33970.236044] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33970.243167] nfnetlink [33970.246912] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33970.254036] bridge stp [33970.256392] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33970.263516] llc [33970.265875] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33970.272997] xfs libcrc32c [33970.275460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33970.282582] sunrpc intel_rapl_msr [33970.284420] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33970.292498] intel_rapl_common sb_edac [33970.295203] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33970.300939] x86_pkg_temp_thermal [33970.304338] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33970.311461] coretemp kvm_intel [33970.315204] Call Trace: [33970.315206] [33970.322329] kvm irqbypass [33970.325643] ? watchdog_timer_fn+0x1b4/0x220 [33970.332765] i2c_i801 iTCO_wdt [33970.335903] ? __hrtimer_run_queues+0x1d3/0x360 [33970.338345] iTCO_vendor_support [33970.340361] ? hrtimer_interrupt+0x104/0x240 [33970.343060] i2c_smbus crct10dif_pclmul [33970.347324] ? __local_bh_enable+0x50/0x60 [33970.350375] joydev crc32_pclmul [33970.354901] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33970.358123] ipmi_ssif ghash_clmulni_intel [33970.362388] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33970.366218] mei_me [33970.370309] [33970.373534] mei lpc_ich [33970.378750] [33970.378752] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33970.382841] sha512_ssse3 acpi_ipmi [33970.387802] ? queued_spin_lock_slowpath+0x2b/0x2e0 [33970.389897] mfd_core wmi [33970.391995] do_raw_spin_lock+0xb5/0xc0 [33970.394524] ipmi_si ipmi_devintf [33970.396623] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33970.401926] ipmi_msghandler [33970.405414] try_get_cap_refs+0x159/0x880 [ceph] [33970.410280] ioatdma acpi_pad [33970.412902] __ceph_get_caps+0x4a1/0x760 [ceph] [33970.416728] acpi_power_meter ixgbe [33970.420044] ? __file_remove_privs+0x89/0x110 [33970.424823] igb [33970.427702] ? wake_bit_function+0x70/0x70 [33970.432311] mdio crc32c_intel [33970.435278] ? ceph_write_iter+0x260/0x770 [ceph] [33970.439799] nvme [33970.443285] ceph_write_iter+0x260/0x770 [ceph] [33970.447635] ptp nvme_core [33970.449476] ? lock_is_held_type+0xaa/0x120 [33970.453562] i2c_algo_bit [33970.456616] vfs_write+0x324/0x4f0 [33970.461322] pps_core dca [33970.463247] ksys_pwrite64+0x65/0xa0 [33970.467769] fuse [33970.470471] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33970.474650] irq event stamp: 192737352 [33970.477268] do_syscall_64+0x38/0x80 [33970.480665] hardirqs last enabled at (192737351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33970.483282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33970.486869] hardirqs last disabled at (192737352): [] sysvec_apic_timer_interrupt+0xe/0x90 [33970.488793] RIP: 0033:0x7f48f0eeb93b [33970.493925] softirqs last enabled at (192737350): [] __do_softirq+0x2be/0x371 [33970.497667] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33970.501240] softirqs last disabled at (192737341): [] irq_exit_rcu+0xb0/0xd0 [33970.511482] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33970.516527] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33970.526337] ORIG_RAX: 0000000000000012 [33970.529909] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33970.538678] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33970.557418] Workqueue: events delayed_work [ceph] [33970.566026] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33970.571242] [33970.571243] RIP: 0010:lock_acquire+0xf0/0x2b0 [33970.580619] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33970.584449] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [33970.591920] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33970.599045] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [33970.603741] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33970.610865] [33970.612361] [33970.616708] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [33970.672833] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [33970.679957] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [33970.687082] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [33970.694205] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [33970.701331] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33970.709408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33970.715145] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33970.722269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33970.729393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33970.736518] Call Trace: [33970.738963] [33970.740973] ? watchdog_timer_fn+0x1b4/0x220 [33970.745238] ? __hrtimer_run_queues+0x1d3/0x360 [33970.749765] ? hrtimer_interrupt+0x104/0x240 [33970.754033] ? __local_bh_enable+0x50/0x60 [33970.758127] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33970.763350] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33970.768310] [33970.770404] [33970.772504] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33970.777820] ? lock_acquire+0xf0/0x2b0 [33970.781570] ? igrab+0x43/0x50 [33970.784618] ? lock_release+0xc5/0x260 [33970.788365] _raw_spin_lock+0x30/0x40 [33970.792030] ? igrab+0x1d/0x50 [33970.795080] igrab+0x1d/0x50 [33970.797959] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33970.803108] delayed_work+0x1d8/0x280 [ceph] [33970.807391] ? process_scheduled_works+0x22a/0x510 [33970.812178] process_scheduled_works+0x2a4/0x510 [33970.816796] worker_thread+0x188/0x340 [33970.820543] ? rescuer_thread+0x310/0x310 [33970.824546] kthread+0xf5/0x130 [33970.827685] ? kthread_complete_and_exit+0x20/0x20 [33970.832468] ret_from_fork+0x31/0x40 [33970.836039] ? kthread_complete_and_exit+0x20/0x20 [33970.840822] ret_from_fork_asm+0x11/0x20 [33970.844745] [33998.055097] watchdog: BUG: soft lockup - CPU#0 stuck for 20549s! [dbench:78683] [33998.062407] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [33998.067097] watchdog: BUG: soft lockup - CPU#7 stuck for 29546s! [kworker/7:4:78808] [33998.117856] irq event stamp: 58684998 [33998.125587] Modules linked in: ceph libceph dns_resolver fscache netfs veth [33998.129243] hardirqs last enabled at (58684997): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33998.129248] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [33998.146361] hardirqs last disabled at (58684998): [] sysvec_apic_timer_interrupt+0xe/0x90 [33998.155566] nf_tables nfnetlink bridge stp llc xfs libcrc32c [33998.165289] softirqs last enabled at (58684952): [] __do_softirq+0x2be/0x371 [33998.171027] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [33998.179711] softirqs last disabled at (58684947): [] irq_exit_rcu+0xb0/0xd0 [33998.187180] coretemp kvm_intel kvm irqbypass i2c_i801 [33998.195693] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33998.200823] iTCO_wdt [33998.209783] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33998.212056] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [33998.219528] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [33998.227689] ghash_clmulni_intel mei_me mei lpc_ich [33998.233169] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [33998.238037] sha512_ssse3 acpi_ipmi [33998.256793] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [33998.256794] mfd_core wmi [33998.260278] [33998.260278] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [33998.265496] ipmi_si [33998.268113] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [33998.269603] ipmi_devintf ipmi_msghandler [33998.276728] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [33998.278911] ioatdma acpi_pad [33998.286035] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [33998.290039] acpi_power_meter [33998.297164] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [33998.300127] ixgbe igb [33998.307251] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [33998.310214] mdio crc32c_intel [33998.317339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33998.319698] nvme [33998.327786] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [33998.330834] ptp nvme_core [33998.336571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33998.338496] i2c_algo_bit [33998.345619] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33998.348323] pps_core dca [33998.355449] Call Trace: [33998.355452] [33998.358064] fuse [33998.365193] ? watchdog_timer_fn+0x1b4/0x220 [33998.367806] irq event stamp: 192958038 [33998.370252] ? __hrtimer_run_queues+0x1d3/0x360 [33998.372262] hardirqs last enabled at (192958037): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33998.374190] ? hrtimer_interrupt+0x104/0x240 [33998.378448] hardirqs last disabled at (192958038): [] sysvec_apic_timer_interrupt+0xe/0x90 [33998.382193] ? __local_bh_enable+0x50/0x60 [33998.386717] softirqs last enabled at (192958036): [] __do_softirq+0x2be/0x371 [33998.396965] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33998.401225] softirqs last disabled at (192958031): [] irq_exit_rcu+0xb0/0xd0 [33998.411037] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33998.415127] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [33998.423899] [33998.429115] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [33998.437714] [33998.442672] Workqueue: events delayed_work [ceph] [33998.452050] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33998.454145] [33998.461622] ? queued_spin_lock_slowpath+0x16/0x2e0 [33998.463713] RIP: 0010:lock_release+0x14e/0x260 [33998.468415] do_raw_spin_lock+0xb5/0xc0 [33998.473716] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [33998.475207] __cap_delay_requeue+0xa7/0x1d0 [ceph] [33998.480076] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [33998.484518] try_get_cap_refs+0x159/0x880 [ceph] [33998.488344] [33998.488345] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [33998.507089] __ceph_get_caps+0x4a1/0x760 [ceph] [33998.511866] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [33998.517089] ? __file_remove_privs+0x89/0x110 [33998.521695] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [33998.523187] ? wake_bit_function+0x70/0x70 [33998.530310] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [33998.534836] ? ceph_write_iter+0x260/0x770 [ceph] [33998.541959] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [33998.546308] ceph_write_iter+0x260/0x770 [ceph] [33998.553432] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [33998.557529] ? lock_is_held_type+0xaa/0x120 [33998.564647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33998.569347] vfs_write+0x324/0x4f0 [33998.576469] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [33998.580998] ksys_pwrite64+0x65/0xa0 [33998.589071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [33998.593250] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [33998.598985] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [33998.602386] do_syscall_64+0x38/0x80 [33998.609507] Call Trace: [33998.609508] [33998.613080] entry_SYSCALL_64_after_hwframe+0x63/0xcd [33998.620202] ? watchdog_timer_fn+0x1b4/0x220 [33998.625333] RIP: 0033:0x7f48f0eeb93b [33998.632459] ? __hrtimer_run_queues+0x1d3/0x360 [33998.636029] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [33998.638478] ? hrtimer_interrupt+0x104/0x240 [33998.640501] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [33998.645543] ? __local_bh_enable+0x50/0x60 [33998.649807] ORIG_RAX: 0000000000000012 [33998.653382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [33998.657902] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [33998.676641] ? sysvec_apic_timer_interrupt+0x6d/0x90 [33998.680904] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [33998.686122] [33998.690212] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [33998.694043] [33998.699260] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [33998.706386] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [33998.711342] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [33998.718467] ? igrab+0x43/0x50 [33998.720570] [33998.727691] ? lock_release+0x14e/0x260 [33998.758404] ? lock_release+0xc5/0x260 [33998.762150] _raw_spin_unlock+0x1b/0x40 [33998.765987] igrab+0x43/0x50 [33998.768867] ceph_check_delayed_caps+0xb7/0x270 [ceph] [33998.774015] delayed_work+0x1d8/0x280 [ceph] [33998.778298] ? process_scheduled_works+0x22a/0x510 [33998.783085] process_scheduled_works+0x2a4/0x510 [33998.787704] worker_thread+0x188/0x340 [33998.791450] ? rescuer_thread+0x310/0x310 [33998.795454] kthread+0xf5/0x130 [33998.798603] ? kthread_complete_and_exit+0x20/0x20 [33998.803393] ret_from_fork+0x31/0x40 [33998.806972] ? kthread_complete_and_exit+0x20/0x20 [33998.811756] ret_from_fork_asm+0x11/0x20 [33998.815690] [34026.055568] watchdog: BUG: soft lockup - CPU#0 stuck for 20575s! [dbench:78683] [34026.062867] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34026.067569] watchdog: BUG: soft lockup - CPU#7 stuck for 29572s! [kworker/7:4:78808] [34026.118308] irq event stamp: 58743094 [34026.118309] hardirqs last enabled at (58743093): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34026.126038] Modules linked in: [34026.129698] hardirqs last disabled at (58743094): [] sysvec_apic_timer_interrupt+0xe/0x90 [34026.139855] ceph [34026.142905] softirqs last enabled at (58743036): [] __do_softirq+0x2be/0x371 [34026.152628] libceph [34026.154552] softirqs last disabled at (58743031): [] irq_exit_rcu+0xb0/0xd0 [34026.163243] dns_resolver fscache [34026.165430] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34026.173940] netfs veth [34026.177268] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34026.186212] nft_chain_nat [34026.188656] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34026.196127] xt_MASQUERADE [34026.198832] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34026.204309] nf_nat nf_conntrack [34026.207013] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34026.225750] nf_defrag_ipv6 [34026.228976] [34026.228977] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34026.234193] nf_defrag_ipv4 nft_compat [34026.236983] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34026.238474] nf_tables nfnetlink [34026.245597] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34026.249342] bridge stp [34026.256466] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34026.259690] llc xfs [34026.266815] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34026.269258] libcrc32c sunrpc [34026.276382] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34026.278567] intel_rapl_msr intel_rapl_common [34026.285691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34026.288656] sb_edac x86_pkg_temp_thermal [34026.296733] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34026.301082] coretemp kvm_intel [34026.306820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34026.310824] kvm [34026.317964] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34026.321104] irqbypass i2c_i801 [34026.328243] Call Trace: [34026.328244] [34026.330081] iTCO_wdt [34026.337208] ? watchdog_timer_fn+0x1b4/0x220 [34026.340344] iTCO_vendor_support [34026.342788] ? __hrtimer_run_queues+0x1d3/0x360 [34026.344799] i2c_smbus [34026.347073] ? hrtimer_interrupt+0x104/0x240 [34026.351334] crct10dif_pclmul joydev [34026.354557] ? __local_bh_enable+0x50/0x60 [34026.359082] crc32_pclmul ipmi_ssif [34026.361441] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34026.365703] ghash_clmulni_intel [34026.369275] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34026.373364] mei_me mei [34026.376849] [34026.382066] lpc_ich sha512_ssse3 [34026.385291] [34026.390248] acpi_ipmi mfd_core [34026.392691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34026.394789] wmi ipmi_si [34026.398105] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34026.400197] ipmi_devintf ipmi_msghandler [34026.403336] do_raw_spin_lock+0xb5/0xc0 [34026.408638] ioatdma acpi_pad acpi_power_meter [34026.411171] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34026.416040] ixgbe igb [34026.420047] try_get_cap_refs+0x159/0x880 [ceph] [34026.423874] mdio crc32c_intel [34026.428317] __ceph_get_caps+0x4a1/0x760 [ceph] [34026.433097] nvme ptp [34026.435457] ? __file_remove_privs+0x89/0x110 [34026.440063] nvme_core [34026.443116] ? wake_bit_function+0x70/0x70 [34026.447640] i2c_algo_bit [34026.449914] ? ceph_write_iter+0x260/0x770 [ceph] [34026.454261] pps_core dca [34026.456619] ceph_write_iter+0x260/0x770 [ceph] [34026.460709] fuse [34026.463328] ? lock_is_held_type+0xaa/0x120 [34026.468023] irq event stamp: 193179018 [34026.470643] vfs_write+0x324/0x4f0 [34026.475164] hardirqs last enabled at (193179017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34026.477095] ksys_pwrite64+0x65/0xa0 [34026.481266] hardirqs last disabled at (193179018): [] sysvec_apic_timer_interrupt+0xe/0x90 [34026.485011] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34026.488408] softirqs last enabled at (193179016): [] __do_softirq+0x2be/0x371 [34026.498653] do_syscall_64+0x38/0x80 [34026.502224] softirqs last disabled at (193179011): [] irq_exit_rcu+0xb0/0xd0 [34026.512034] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34026.517165] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34026.525936] RIP: 0033:0x7f48f0eeb93b [34026.529506] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34026.538103] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34026.543148] Workqueue: events delayed_work [ceph] [34026.552525] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34026.556097] [34026.556098] RIP: 0010:lock_acquire+0xf0/0x2b0 [34026.563568] ORIG_RAX: 0000000000000012 [34026.582304] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34026.587002] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34026.592219] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34026.593709] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34026.598059] [34026.598060] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34026.601891] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34026.620629] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34026.627752] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34026.632971] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34026.640094] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34026.641585] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34026.648715] [34026.655834] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34026.700779] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34026.708857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34026.714596] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34026.721719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34026.728843] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34026.735974] Call Trace: [34026.738421] [34026.740432] ? watchdog_timer_fn+0x1b4/0x220 [34026.744696] ? __hrtimer_run_queues+0x1d3/0x360 [34026.749222] ? hrtimer_interrupt+0x104/0x240 [34026.753484] ? __local_bh_enable+0x50/0x60 [34026.757576] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34026.762801] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34026.767758] [34026.769855] [34026.771971] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34026.777286] ? lock_acquire+0xf0/0x2b0 [34026.781038] ? igrab+0x43/0x50 [34026.784086] ? lock_release+0xc5/0x260 [34026.787831] _raw_spin_lock+0x30/0x40 [34026.791487] ? igrab+0x1d/0x50 [34026.794539] igrab+0x1d/0x50 [34026.797417] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34026.802566] delayed_work+0x1d8/0x280 [ceph] [34026.806848] ? process_scheduled_works+0x22a/0x510 [34026.811640] process_scheduled_works+0x2a4/0x510 [34026.816262] worker_thread+0x188/0x340 [34026.820010] ? rescuer_thread+0x310/0x310 [34026.824014] kthread+0xf5/0x130 [34026.827151] ? kthread_complete_and_exit+0x20/0x20 [34026.831935] ret_from_fork+0x31/0x40 [34026.835507] ? kthread_complete_and_exit+0x20/0x20 [34026.840290] ret_from_fork_asm+0x11/0x20 [34026.844213] [34054.056037] watchdog: BUG: soft lockup - CPU#0 stuck for 20601s! [dbench:78683] [34054.063337] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34054.068036] watchdog: BUG: soft lockup - CPU#7 stuck for 29598s! [kworker/7:4:78808] [34054.118786] irq event stamp: 58801226 [34054.126519] Modules linked in: ceph libceph dns_resolver fscache netfs [34054.130175] hardirqs last enabled at (58801225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34054.136709] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [34054.146867] hardirqs last disabled at (58801226): [] sysvec_apic_timer_interrupt+0xe/0x90 [34054.154252] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [34054.163975] softirqs last enabled at (58801168): [] __do_softirq+0x2be/0x371 [34054.170754] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [34054.179438] softirqs last disabled at (58801163): [] irq_exit_rcu+0xb0/0xd0 [34054.186302] x86_pkg_temp_thermal coretemp kvm_intel kvm [34054.194813] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34054.200116] irqbypass i2c_i801 [34054.209062] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34054.209063] iTCO_wdt iTCO_vendor_support [34054.212200] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34054.219670] i2c_smbus crct10dif_pclmul [34054.223674] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34054.229150] joydev crc32_pclmul [34054.232981] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34054.251720] ipmi_ssif ghash_clmulni_intel [34054.254942] [34054.260162] mei_me mei [34054.264250] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34054.265741] lpc_ich sha512_ssse3 [34054.268185] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34054.275311] acpi_ipmi mfd_core [34054.278620] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34054.285745] wmi [34054.288882] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34054.296005] ipmi_si ipmi_devintf [34054.297844] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34054.304969] ipmi_msghandler ioatdma [34054.308278] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34054.315404] acpi_pad acpi_power_meter [34054.318974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34054.327052] ixgbe igb [34054.330795] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34054.336532] mdio crc32c_intel [34054.338891] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34054.346016] nvme [34054.349066] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34054.356190] ptp nvme_core [34054.358112] Call Trace: [34054.358114] [34054.365238] i2c_algo_bit [34054.367944] ? watchdog_timer_fn+0x1b4/0x220 [34054.370386] pps_core dca [34054.372398] ? __hrtimer_run_queues+0x1d3/0x360 [34054.375014] fuse [34054.379281] ? hrtimer_interrupt+0x104/0x240 [34054.381894] irq event stamp: 193399752 [34054.386435] ? __local_bh_enable+0x50/0x60 [34054.388361] hardirqs last enabled at (193399751): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34054.392626] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34054.396370] hardirqs last disabled at (193399752): [] sysvec_apic_timer_interrupt+0xe/0x90 [34054.400459] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34054.410714] softirqs last enabled at (193399750): [] __do_softirq+0x2be/0x371 [34054.415930] [34054.425741] softirqs last disabled at (193399745): [] irq_exit_rcu+0xb0/0xd0 [34054.430713] [34054.439487] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34054.441584] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34054.450180] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34054.452282] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34054.461656] Workqueue: events delayed_work [ceph] [34054.466961] do_raw_spin_lock+0xb5/0xc0 [34054.474436] [34054.474437] RIP: 0010:lock_release+0x3c/0x260 [34054.479312] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34054.484007] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [34054.487841] try_get_cap_refs+0x159/0x880 [ceph] [34054.489329] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [34054.493684] __ceph_get_caps+0x4a1/0x760 [ceph] [34054.498463] [34054.498464] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [34054.517207] ? __file_remove_privs+0x89/0x110 [34054.521812] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [34054.527030] ? wake_bit_function+0x70/0x70 [34054.531552] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [34054.533046] ? ceph_write_iter+0x260/0x770 [ceph] [34054.540167] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [34054.544519] ceph_write_iter+0x260/0x770 [ceph] [34054.551643] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34054.555739] ? lock_is_held_type+0xaa/0x120 [34054.562857] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34054.567557] vfs_write+0x324/0x4f0 [34054.574679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34054.579210] ksys_pwrite64+0x65/0xa0 [34054.586328] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34054.590508] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34054.598584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34054.601982] do_syscall_64+0x38/0x80 [34054.607719] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34054.611307] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34054.618438] Call Trace: [34054.623569] RIP: 0033:0x7f48f0eeb93b [34054.630695] [34054.634283] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34054.641407] ? watchdog_timer_fn+0x1b4/0x220 [34054.646449] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34054.648895] ? __hrtimer_run_queues+0x1d3/0x360 [34054.652465] ORIG_RAX: 0000000000000012 [34054.654478] ? hrtimer_interrupt+0x104/0x240 [34054.673212] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34054.677476] ? __local_bh_enable+0x50/0x60 [34054.682695] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34054.687220] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34054.691049] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34054.695313] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34054.702442] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34054.706528] [34054.713654] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34054.718871] [34054.718872] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34054.725999] [34054.730952] ? igrab+0x43/0x50 [34054.759937] ? igrab+0x43/0x50 [34054.762993] ? lock_release+0x3c/0x260 [34054.766738] ? lock_release+0xc5/0x260 [34054.770484] _raw_spin_unlock+0x1b/0x40 [34054.774321] igrab+0x43/0x50 [34054.777199] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34054.782348] delayed_work+0x1d8/0x280 [ceph] [34054.786630] ? process_scheduled_works+0x22a/0x510 [34054.791419] process_scheduled_works+0x2a4/0x510 [34054.796055] worker_thread+0x188/0x340 [34054.799803] ? rescuer_thread+0x310/0x310 [34054.803815] kthread+0xf5/0x130 [34054.806961] ? kthread_complete_and_exit+0x20/0x20 [34054.811744] ret_from_fork+0x31/0x40 [34054.815325] ? kthread_complete_and_exit+0x20/0x20 [34054.820107] ret_from_fork_asm+0x11/0x20 [34054.824032] [34082.056504] watchdog: BUG: soft lockup - CPU#0 stuck for 20627s! [dbench:78683] [34082.063809] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34082.068504] watchdog: BUG: soft lockup - CPU#7 stuck for 29624s! [kworker/7:4:78808] [34082.119248] irq event stamp: 58859196 [34082.126980] Modules linked in: ceph libceph [34082.130636] hardirqs last enabled at (58859195): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34082.134814] dns_resolver fscache netfs [34082.144972] hardirqs last disabled at (58859196): [] sysvec_apic_timer_interrupt+0xe/0x90 [34082.148803] veth nft_chain_nat xt_MASQUERADE nf_nat [34082.158527] softirqs last enabled at (58859158): [] __do_softirq+0x2be/0x371 [34082.163484] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [34082.172180] softirqs last disabled at (58859153): [] irq_exit_rcu+0xb0/0xd0 [34082.178357] nf_tables nfnetlink bridge [34082.186868] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34082.190699] stp llc [34082.199641] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34082.199643] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34082.201826] xfs libcrc32c [34082.209296] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34082.214776] sunrpc [34082.217479] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34082.236215] intel_rapl_msr intel_rapl_common [34082.238315] [34082.238315] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34082.243531] sb_edac x86_pkg_temp_thermal [34082.247899] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34082.249390] coretemp kvm_intel [34082.256515] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34082.260519] kvm irqbypass [34082.267642] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34082.270780] i2c_i801 iTCO_wdt [34082.277905] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34082.280607] iTCO_vendor_support i2c_smbus [34082.287732] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34082.290783] crct10dif_pclmul joydev [34082.297908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34082.301997] crc32_pclmul ipmi_ssif [34082.310075] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34082.313645] ghash_clmulni_intel mei_me [34082.319383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34082.322866] mei lpc_ich [34082.329992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34082.333821] sha512_ssse3 acpi_ipmi [34082.340947] Call Trace: [34082.340948] [34082.343478] mfd_core [34082.350602] ? watchdog_timer_fn+0x1b4/0x220 [34082.354084] wmi ipmi_si [34082.356531] ? __hrtimer_run_queues+0x1d3/0x360 [34082.358539] ipmi_devintf ipmi_msghandler [34082.360814] ? hrtimer_interrupt+0x104/0x240 [34082.365074] ioatdma acpi_pad [34082.367604] ? __local_bh_enable+0x50/0x60 [34082.372131] acpi_power_meter ixgbe [34082.376137] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34082.380397] igb [34082.383363] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34082.387452] mdio crc32c_intel nvme [34082.390937] [34082.396153] ptp nvme_core [34082.397992] [34082.402949] i2c_algo_bit pps_core [34082.406434] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34082.408530] dca [34082.411240] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34082.413331] fuse [34082.416730] do_raw_spin_lock+0xb5/0xc0 [34082.422034] irq event stamp: 193620626 [34082.423872] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34082.428742] hardirqs last enabled at (193620625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34082.430668] try_get_cap_refs+0x159/0x880 [ceph] [34082.434496] hardirqs last disabled at (193620626): [] sysvec_apic_timer_interrupt+0xe/0x90 [34082.438245] __ceph_get_caps+0x4a1/0x760 [ceph] [34082.443025] softirqs last enabled at (193620624): [] __do_softirq+0x2be/0x371 [34082.453272] ? __file_remove_privs+0x89/0x110 [34082.457880] softirqs last disabled at (193620619): [] irq_exit_rcu+0xb0/0xd0 [34082.467707] ? wake_bit_function+0x70/0x70 [34082.472233] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34082.481004] ? ceph_write_iter+0x260/0x770 [ceph] [34082.485354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34082.493950] ceph_write_iter+0x260/0x770 [ceph] [34082.498041] Workqueue: events delayed_work [ceph] [34082.507423] ? lock_is_held_type+0xaa/0x120 [34082.512116] [34082.512117] RIP: 0010:lock_acquire+0xf0/0x2b0 [34082.519590] vfs_write+0x324/0x4f0 [34082.524110] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34082.528814] ksys_pwrite64+0x65/0xa0 [34082.532985] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34082.534477] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34082.538828] [34082.538828] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34082.542226] do_syscall_64+0x38/0x80 [34082.560962] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34082.564533] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34082.569750] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34082.574881] RIP: 0033:0x7f48f0eeb93b [34082.576389] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34082.583513] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34082.587083] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34082.594207] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34082.599253] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34082.606376] ORIG_RAX: 0000000000000012 [34082.609947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34082.617072] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34082.635808] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34082.642934] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34082.648150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34082.656228] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34082.660059] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34082.665798] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34082.672921] Call Trace: [34082.680045] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34082.687186] [34082.694315] [34082.701435] ? watchdog_timer_fn+0x1b4/0x220 [34082.733711] ? __hrtimer_run_queues+0x1d3/0x360 [34082.738238] ? hrtimer_interrupt+0x104/0x240 [34082.742507] ? __local_bh_enable+0x50/0x60 [34082.746600] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34082.751815] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34082.756773] [34082.758870] [34082.760968] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34082.766274] ? lock_acquire+0xf0/0x2b0 [34082.770025] ? igrab+0x43/0x50 [34082.773076] ? lock_release+0xc5/0x260 [34082.776821] _raw_spin_lock+0x30/0x40 [34082.780486] ? igrab+0x1d/0x50 [34082.783537] igrab+0x1d/0x50 [34082.786413] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34082.791563] delayed_work+0x1d8/0x280 [ceph] [34082.795845] ? process_scheduled_works+0x22a/0x510 [34082.800634] process_scheduled_works+0x2a4/0x510 [34082.805249] worker_thread+0x188/0x340 [34082.809000] ? rescuer_thread+0x310/0x310 [34082.813002] kthread+0xf5/0x130 [34082.816143] ? kthread_complete_and_exit+0x20/0x20 [34082.820933] ret_from_fork+0x31/0x40 [34082.824502] ? kthread_complete_and_exit+0x20/0x20 [34082.829289] ret_from_fork_asm+0x11/0x20 [34082.833221] [34084.314908] rcu: INFO: rcu_preempt self-detected stall on CPU [34084.320669] rcu: 0-....: (21647496 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10836517 [34084.331180] rcu: (t=22147312 jiffies g=1309609 q=625479 ncpus=8) [34084.337264] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34084.346208] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34084.353679] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34084.359154] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34084.377894] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34084.383111] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34084.390235] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34084.397359] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34084.404484] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34084.411608] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34084.418731] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34084.426809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34084.432546] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34084.439671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34084.446795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34084.453919] Call Trace: [34084.456364] [34084.458374] ? rcu_dump_cpu_stacks+0xd1/0x180 [34084.462735] ? rcu_sched_clock_irq+0x633/0x1390 [34084.467267] ? lock_is_held_type+0xaa/0x120 [34084.471444] ? lock_is_held_type+0xaa/0x120 [34084.475622] ? tick_sched_do_timer+0xa0/0xa0 [34084.479894] ? rcu_read_lock_sched_held+0x56/0x90 [34084.484593] ? tick_sched_do_timer+0xa0/0xa0 [34084.488856] ? update_process_times+0x5f/0x90 [34084.493206] ? tick_sched_handle.isra.16+0x1e/0x50 [34084.497989] ? tick_sched_timer+0x6f/0x80 [34084.501994] ? __hrtimer_run_queues+0x1d3/0x360 [34084.506521] ? hrtimer_interrupt+0x104/0x240 [34084.510782] ? __local_bh_enable+0x50/0x60 [34084.514875] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34084.520099] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34084.525056] [34084.527153] [34084.529253] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34084.534560] ? queued_spin_lock_slowpath+0x16/0x2e0 [34084.539436] do_raw_spin_lock+0xb5/0xc0 [34084.543267] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34084.548069] try_get_cap_refs+0x159/0x880 [ceph] [34084.552699] __ceph_get_caps+0x4a1/0x760 [ceph] [34084.557240] ? __file_remove_privs+0x89/0x110 [34084.561596] ? wake_bit_function+0x70/0x70 [34084.565690] ? ceph_write_iter+0x260/0x770 [ceph] [34084.570401] ceph_write_iter+0x260/0x770 [ceph] [34084.574938] ? lock_is_held_type+0xaa/0x120 [34084.579122] vfs_write+0x324/0x4f0 [34084.582532] ksys_pwrite64+0x65/0xa0 [34084.586105] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34084.591238] do_syscall_64+0x38/0x80 [34084.594808] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34084.599852] RIP: 0033:0x7f48f0eeb93b [34084.603421] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34084.622158] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [34084.629717] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34084.636841] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34084.643965] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34084.651089] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34084.658213] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34084.665342] [34084.667531] Sending NMI from CPU 0 to CPUs 7: [34084.671882] NMI backtrace for cpu 7 [34084.671884] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34084.671886] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34084.671887] Workqueue: events delayed_work [ceph] [34084.671903] RIP: 0010:__lock_acquire+0x451/0x1e20 [34084.671905] Code: 89 df e8 62 d3 ff ff 48 c7 c7 5b df 12 82 e8 f6 18 01 00 e8 41 ec 9f 00 44 8b 5c 24 18 45 85 db 74 7d 80 7d 21 00 0f b6 45 22 <78> 2b a8 03 0f 84 64 05 00 00 65 8b 05 46 f9 ea 7e 85 c0 0f 85 bd [34084.671906] RSP: 0018:ffffc90002117c60 EFLAGS: 00000006 [34084.671907] RAX: 0000000000000004 RBX: ffff88810debaa80 RCX: 00000000000006f5 [34084.671908] RDX: ffff88810debb550 RSI: 0000000000000002 RDI: 0000000000000003 [34084.671909] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [34084.671909] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [34084.671910] R13: 0000000000000003 R14: 0000000000000002 R15: 0000000000000002 [34084.671911] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34084.671912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34084.671912] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34084.671914] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34084.671915] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34084.671916] Call Trace: [34084.671917] [34084.671918] ? nmi_cpu_backtrace+0xad/0x120 [34084.671922] ? nmi_cpu_backtrace_handler+0x11/0x20 [34084.671923] ? nmi_handle+0xd4/0x260 [34084.671928] ? default_do_nmi+0x49/0xf0 [34084.671929] ? exc_nmi+0x11e/0x150 [34084.671931] ? end_repeat_nmi+0x16/0x67 [34084.671934] ? __lock_acquire+0x451/0x1e20 [34084.671936] ? __lock_acquire+0x451/0x1e20 [34084.671937] ? __lock_acquire+0x451/0x1e20 [34084.671939] [34084.671939] [34084.671941] lock_acquire+0xcb/0x2b0 [34084.671942] ? igrab+0x1d/0x50 [34084.671944] ? igrab+0x43/0x50 [34084.671945] ? lock_release+0xc5/0x260 [34084.671947] _raw_spin_lock+0x30/0x40 [34084.671949] ? igrab+0x1d/0x50 [34084.671950] igrab+0x1d/0x50 [34084.671951] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34084.671965] delayed_work+0x1d8/0x280 [ceph] [34084.671978] ? process_scheduled_works+0x22a/0x510 [34084.671981] process_scheduled_works+0x2a4/0x510 [34084.671984] worker_thread+0x188/0x340 [34084.671987] ? rescuer_thread+0x310/0x310 [34084.671989] kthread+0xf5/0x130 [34084.671991] ? kthread_complete_and_exit+0x20/0x20 [34084.671994] ret_from_fork+0x31/0x40 [34084.671996] ? kthread_complete_and_exit+0x20/0x20 [34084.671998] ret_from_fork_asm+0x11/0x20 [34084.672001] [34110.056973] watchdog: BUG: soft lockup - CPU#0 stuck for 20653s! [dbench:78683] [34110.064278] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34110.068973] watchdog: BUG: soft lockup - CPU#7 stuck for 29650s! [kworker/7:4:78808] [34110.119720] irq event stamp: 58916502 [34110.119721] hardirqs last enabled at (58916501): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34110.127451] Modules linked in: [34110.131106] hardirqs last disabled at (58916502): [] sysvec_apic_timer_interrupt+0xe/0x90 [34110.141264] ceph [34110.144315] softirqs last enabled at (58916472): [] __do_softirq+0x2be/0x371 [34110.154041] libceph [34110.155964] softirqs last disabled at (58916467): [] irq_exit_rcu+0xb0/0xd0 [34110.164648] dns_resolver fscache [34110.166833] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34110.175344] netfs [34110.178653] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34110.178654] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34110.187598] veth nft_chain_nat [34110.189610] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34110.197080] xt_MASQUERADE nf_nat [34110.202556] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34110.205693] nf_conntrack nf_defrag_ipv6 [34110.224432] [34110.224433] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34110.227743] nf_defrag_ipv4 nft_compat [34110.232960] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34110.236878] nf_tables nfnetlink [34110.238368] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34110.238369] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34110.245492] bridge [34110.249237] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34110.256371] stp llc [34110.259594] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34110.266718] xfs libcrc32c [34110.273841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34110.275938] sunrpc intel_rapl_msr [34110.283062] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34110.285246] intel_rapl_common sb_edac [34110.293323] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34110.296029] x86_pkg_temp_thermal coretemp [34110.301766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34110.305164] kvm_intel [34110.312287] Call Trace: [34110.316031] kvm irqbypass [34110.323157] [34110.327247] i2c_i801 iTCO_wdt [34110.334371] ? watchdog_timer_fn+0x1b4/0x220 [34110.336729] iTCO_vendor_support [34110.339173] ? __hrtimer_run_queues+0x1d3/0x360 [34110.341877] i2c_smbus crct10dif_pclmul [34110.343890] ? hrtimer_interrupt+0x104/0x240 [34110.346937] joydev crc32_pclmul [34110.351202] ? __local_bh_enable+0x50/0x60 [34110.354425] ipmi_ssif ghash_clmulni_intel [34110.358951] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34110.362780] mei_me mei [34110.367044] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34110.370269] lpc_ich sha512_ssse3 [34110.374373] [34110.378468] acpi_ipmi mfd_core [34110.383686] [34110.386130] wmi ipmi_si [34110.391087] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34110.394398] ipmi_devintf [34110.396499] ? queued_spin_lock_slowpath+0x16/0x2e0 [34110.399632] ipmi_msghandler ioatdma acpi_pad [34110.401749] do_raw_spin_lock+0xb5/0xc0 [34110.404277] acpi_power_meter ixgbe [34110.409583] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34110.412198] igb mdio [34110.417070] try_get_cap_refs+0x159/0x880 [ceph] [34110.421421] crc32c_intel [34110.425256] __ceph_get_caps+0x4a1/0x760 [ceph] [34110.428736] nvme ptp [34110.433523] ? __file_remove_privs+0x89/0x110 [34110.435789] nvme_core i2c_algo_bit [34110.440401] ? wake_bit_function+0x70/0x70 [34110.443017] pps_core dca [34110.447545] ? ceph_write_iter+0x260/0x770 [ceph] [34110.449812] fuse [34110.454165] ceph_write_iter+0x260/0x770 [ceph] [34110.457653] irq event stamp: 193839542 [34110.461751] ? lock_is_held_type+0xaa/0x120 [34110.464375] hardirqs last enabled at (193839541): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34110.469073] vfs_write+0x324/0x4f0 [34110.470995] hardirqs last disabled at (193839542): [] sysvec_apic_timer_interrupt+0xe/0x90 [34110.475523] ksys_pwrite64+0x65/0xa0 [34110.479262] softirqs last enabled at (193839540): [] __do_softirq+0x2be/0x371 [34110.483441] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34110.493684] softirqs last disabled at (193839535): [] irq_exit_rcu+0xb0/0xd0 [34110.497082] do_syscall_64+0x38/0x80 [34110.506892] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34110.510465] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34110.519234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34110.524373] RIP: 0033:0x7f48f0eeb93b [34110.532971] Workqueue: events delayed_work [ceph] [34110.536540] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34110.545920] [34110.545921] RIP: 0010:lock_acquire+0xf0/0x2b0 [34110.550963] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34110.558433] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34110.562006] ORIG_RAX: 0000000000000012 [34110.566701] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34110.585441] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34110.586930] [34110.586930] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34110.591280] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34110.596499] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34110.615237] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34110.619067] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34110.624286] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34110.631408] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34110.632898] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34110.640023] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34110.647153] [34110.654273] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34110.707286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34110.713025] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34110.720149] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34110.727274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34110.734396] Call Trace: [34110.736841] [34110.738853] ? watchdog_timer_fn+0x1b4/0x220 [34110.743118] ? __hrtimer_run_queues+0x1d3/0x360 [34110.747660] ? hrtimer_interrupt+0x104/0x240 [34110.751921] ? __local_bh_enable+0x50/0x60 [34110.756015] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34110.761240] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34110.766197] [34110.768293] [34110.770390] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34110.775699] ? lock_acquire+0xf0/0x2b0 [34110.779447] ? igrab+0x43/0x50 [34110.782498] ? lock_release+0xc5/0x260 [34110.786243] _raw_spin_lock+0x30/0x40 [34110.789900] ? igrab+0x1d/0x50 [34110.792951] igrab+0x1d/0x50 [34110.795828] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34110.800979] delayed_work+0x1d8/0x280 [ceph] [34110.805258] ? process_scheduled_works+0x22a/0x510 [34110.810042] process_scheduled_works+0x2a4/0x510 [34110.814662] worker_thread+0x188/0x340 [34110.818407] ? rescuer_thread+0x310/0x310 [34110.822418] kthread+0xf5/0x130 [34110.825554] ? kthread_complete_and_exit+0x20/0x20 [34110.830339] ret_from_fork+0x31/0x40 [34110.833917] ? kthread_complete_and_exit+0x20/0x20 [34110.838704] ret_from_fork_asm+0x11/0x20 [34110.842625] [34138.057442] watchdog: BUG: soft lockup - CPU#0 stuck for 20679s! [dbench:78683] [34138.064750] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34138.069441] watchdog: BUG: soft lockup - CPU#7 stuck for 29676s! [kworker/7:4:78808] [34138.120199] irq event stamp: 58974702 [34138.127928] Modules linked in: ceph libceph dns_resolver fscache [34138.131586] hardirqs last enabled at (58974701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34138.137584] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [34138.147740] hardirqs last disabled at (58974702): [] sysvec_apic_timer_interrupt+0xe/0x90 [34138.147743] nf_conntrack [34138.147744] softirqs last enabled at (58974674): [] __do_softirq+0x2be/0x371 [34138.153219] nf_defrag_ipv6 [34138.162942] softirqs last disabled at (58974669): [] irq_exit_rcu+0xb0/0xd0 [34138.165561] nf_defrag_ipv4 nft_compat [34138.174246] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34138.177035] nf_tables nfnetlink [34138.185547] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34138.189290] bridge stp [34138.198236] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34138.201457] llc [34138.208929] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34138.211373] xfs libcrc32c [34138.216851] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34138.218689] sunrpc [34138.237426] [34138.237427] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34138.240131] intel_rapl_msr [34138.245348] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34138.247444] intel_rapl_common [34138.248934] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34138.256060] sb_edac x86_pkg_temp_thermal [34138.258850] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34138.265974] coretemp kvm_intel [34138.269024] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34138.276149] kvm [34138.280154] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34138.287277] irqbypass [34138.290415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34138.297538] i2c_i801 iTCO_wdt [34138.299376] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34138.307452] iTCO_vendor_support [34138.309811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34138.315565] i2c_smbus [34138.318616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34138.325740] crct10dif_pclmul joydev [34138.328964] Call Trace: [34138.328965] [34138.336090] crc32_pclmul [34138.338448] ? watchdog_timer_fn+0x1b4/0x220 [34138.345569] ipmi_ssif ghash_clmulni_intel [34138.349142] ? __hrtimer_run_queues+0x1d3/0x360 [34138.351585] mei_me mei [34138.353598] ? hrtimer_interrupt+0x104/0x240 [34138.356214] lpc_ich [34138.360476] ? __local_bh_enable+0x50/0x60 [34138.364568] sha512_ssse3 acpi_ipmi [34138.369094] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34138.371537] mfd_core wmi [34138.375802] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34138.377984] ipmi_si ipmi_devintf [34138.382076] [34138.385559] ipmi_msghandler [34138.390777] [34138.393393] ioatdma acpi_pad [34138.398351] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34138.401661] acpi_power_meter [34138.403763] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34138.406635] ixgbe igb [34138.408737] do_raw_spin_lock+0xb5/0xc0 [34138.411698] mdio crc32c_intel [34138.417005] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34138.419968] nvme [34138.424846] try_get_cap_refs+0x159/0x880 [ceph] [34138.427202] ptp nvme_core [34138.431040] __ceph_get_caps+0x4a1/0x760 [ceph] [34138.434085] i2c_algo_bit [34138.438873] ? __file_remove_privs+0x89/0x110 [34138.440792] pps_core dca [34138.445403] ? wake_bit_function+0x70/0x70 [34138.448120] fuse [34138.452642] ? ceph_write_iter+0x260/0x770 [ceph] [34138.455257] irq event stamp: 194060280 [34138.459608] ceph_write_iter+0x260/0x770 [ceph] [34138.462227] hardirqs last enabled at (194060279): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34138.466320] ? lock_is_held_type+0xaa/0x120 [34138.468242] hardirqs last disabled at (194060280): [] sysvec_apic_timer_interrupt+0xe/0x90 [34138.472940] vfs_write+0x324/0x4f0 [34138.476681] softirqs last enabled at (194060278): [] __do_softirq+0x2be/0x371 [34138.481212] ksys_pwrite64+0x65/0xa0 [34138.491449] softirqs last disabled at (194060273): [] irq_exit_rcu+0xb0/0xd0 [34138.495628] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34138.505439] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34138.508844] do_syscall_64+0x38/0x80 [34138.517616] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34138.521187] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34138.529785] Workqueue: events delayed_work [ceph] [34138.534915] RIP: 0033:0x7f48f0eeb93b [34138.544291] [34138.544292] RIP: 0010:lock_acquire+0xf0/0x2b0 [34138.547862] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34138.555333] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34138.560377] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34138.565076] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34138.568645] ORIG_RAX: 0000000000000012 [34138.570135] [34138.570136] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34138.574488] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34138.593224] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34138.611963] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34138.617179] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34138.622397] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34138.626227] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34138.627718] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34138.634845] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34138.641967] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34138.649091] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34138.656222] [34138.663340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34138.714969] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34138.722092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34138.729217] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34138.736340] Call Trace: [34138.738785] [34138.740797] ? watchdog_timer_fn+0x1b4/0x220 [34138.745060] ? __hrtimer_run_queues+0x1d3/0x360 [34138.749587] ? hrtimer_interrupt+0x104/0x240 [34138.753856] ? __local_bh_enable+0x50/0x60 [34138.757950] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34138.763174] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34138.768132] [34138.770228] [34138.772326] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34138.777634] ? lock_acquire+0xf0/0x2b0 [34138.781383] ? igrab+0x43/0x50 [34138.784433] ? lock_release+0xc5/0x260 [34138.788180] _raw_spin_lock+0x30/0x40 [34138.791848] ? igrab+0x1d/0x50 [34138.794902] igrab+0x1d/0x50 [34138.797783] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34138.802931] delayed_work+0x1d8/0x280 [ceph] [34138.807213] ? process_scheduled_works+0x22a/0x510 [34138.812003] process_scheduled_works+0x2a4/0x510 [34138.816617] worker_thread+0x188/0x340 [34138.820367] ? rescuer_thread+0x310/0x310 [34138.824379] kthread+0xf5/0x130 [34138.827518] ? kthread_complete_and_exit+0x20/0x20 [34138.832308] ret_from_fork+0x31/0x40 [34138.835888] ? kthread_complete_and_exit+0x20/0x20 [34138.840671] ret_from_fork_asm+0x11/0x20 [34138.844595] [34166.057908] watchdog: BUG: soft lockup - CPU#0 stuck for 20705s! [dbench:78683] [34166.065209] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34166.069908] watchdog: BUG: soft lockup - CPU#7 stuck for 29702s! [kworker/7:4:78808] [34166.120652] irq event stamp: 59032894 [34166.120653] hardirqs last enabled at (59032893): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34166.128382] Modules linked in: [34166.132040] hardirqs last disabled at (59032894): [] sysvec_apic_timer_interrupt+0xe/0x90 [34166.142197] ceph [34166.145247] softirqs last enabled at (59032866): [] __do_softirq+0x2be/0x371 [34166.154971] libceph dns_resolver [34166.156895] softirqs last disabled at (59032861): [] irq_exit_rcu+0xb0/0xd0 [34166.165590] fscache netfs [34166.168899] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34166.177410] veth nft_chain_nat [34166.180115] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34166.189059] xt_MASQUERADE nf_nat [34166.192195] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34166.199667] nf_conntrack nf_defrag_ipv6 [34166.202978] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34166.208456] nf_defrag_ipv4 nft_compat [34166.212372] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34166.231109] nf_tables nfnetlink [34166.234854] [34166.234855] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34166.240073] bridge [34166.243305] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34166.244794] stp llc [34166.251919] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34166.254016] xfs libcrc32c [34166.261141] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34166.263325] sunrpc intel_rapl_msr [34166.270448] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34166.273152] intel_rapl_common [34166.280278] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34166.283691] sb_edac x86_pkg_temp_thermal [34166.290816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34166.293865] coretemp kvm_intel [34166.301944] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34166.305947] kvm irqbypass [34166.311685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34166.314823] i2c_i801 iTCO_wdt [34166.321947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34166.324651] iTCO_vendor_support i2c_smbus [34166.331775] Call Trace: [34166.331776] [34166.334826] crct10dif_pclmul [34166.341949] ? watchdog_timer_fn+0x1b4/0x220 [34166.346040] joydev [34166.348486] ? __hrtimer_run_queues+0x1d3/0x360 [34166.350497] crc32_pclmul ipmi_ssif [34166.353463] ? hrtimer_interrupt+0x104/0x240 [34166.357724] ghash_clmulni_intel mei_me [34166.359820] ? __local_bh_enable+0x50/0x60 [34166.364345] mei [34166.367832] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34166.372092] lpc_ich [34166.375923] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34166.380014] sha512_ssse3 acpi_ipmi [34166.381853] [34166.387069] mfd_core wmi [34166.389253] [34166.394210] ipmi_si ipmi_devintf [34166.397697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34166.399792] ipmi_msghandler [34166.402413] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34166.404507] ioatdma acpi_pad [34166.407820] do_raw_spin_lock+0xb5/0xc0 [34166.413122] acpi_power_meter ixgbe [34166.416001] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34166.420871] igb mdio [34166.423836] try_get_cap_refs+0x159/0x880 [ceph] [34166.427665] crc32c_intel nvme [34166.431153] __ceph_get_caps+0x4a1/0x760 [ceph] [34166.435932] ptp nvme_core [34166.438208] ? __file_remove_privs+0x89/0x110 [34166.442816] i2c_algo_bit pps_core [34166.445865] ? wake_bit_function+0x70/0x70 [34166.450390] dca fuse [34166.453095] ? ceph_write_iter+0x260/0x770 [ceph] [34166.457443] [34166.457444] irq event stamp: 194281010 [34166.460841] ceph_write_iter+0x260/0x770 [ceph] [34166.464933] hardirqs last enabled at (194281009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34166.467207] ? lock_is_held_type+0xaa/0x120 [34166.471901] hardirqs last disabled at (194281010): [] sysvec_apic_timer_interrupt+0xe/0x90 [34166.473393] vfs_write+0x324/0x4f0 [34166.477136] softirqs last enabled at (194281008): [] __do_softirq+0x2be/0x371 [34166.481662] ksys_pwrite64+0x65/0xa0 [34166.491904] softirqs last disabled at (194281003): [] irq_exit_rcu+0xb0/0xd0 [34166.496080] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34166.505892] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34166.509307] do_syscall_64+0x38/0x80 [34166.518077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34166.521650] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34166.530247] Workqueue: events delayed_work [ceph] [34166.535377] RIP: 0033:0x7f48f0eeb93b [34166.544754] [34166.544755] RIP: 0010:lock_acquire+0xf0/0x2b0 [34166.548324] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34166.555794] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34166.560840] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34166.565537] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34166.569108] ORIG_RAX: 0000000000000012 [34166.570598] [34166.570599] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34166.574948] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34166.593686] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34166.612423] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34166.617642] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34166.622860] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34166.626691] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34166.628180] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34166.635308] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34166.642427] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34166.649553] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34166.656682] [34166.663801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34166.715430] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34166.722554] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34166.729677] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34166.736803] Call Trace: [34166.739248] [34166.741258] ? watchdog_timer_fn+0x1b4/0x220 [34166.745522] ? __hrtimer_run_queues+0x1d3/0x360 [34166.750049] ? hrtimer_interrupt+0x104/0x240 [34166.754317] ? __local_bh_enable+0x50/0x60 [34166.758411] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34166.763634] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34166.768594] [34166.770688] [34166.772789] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34166.778095] ? lock_acquire+0xf0/0x2b0 [34166.781845] ? igrab+0x43/0x50 [34166.784895] ? lock_release+0xc5/0x260 [34166.788642] _raw_spin_lock+0x30/0x40 [34166.792312] ? igrab+0x1d/0x50 [34166.795364] igrab+0x1d/0x50 [34166.798243] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34166.803391] delayed_work+0x1d8/0x280 [ceph] [34166.807674] ? process_scheduled_works+0x22a/0x510 [34166.812463] process_scheduled_works+0x2a4/0x510 [34166.817080] worker_thread+0x188/0x340 [34166.820827] ? rescuer_thread+0x310/0x310 [34166.824832] kthread+0xf5/0x130 [34166.827969] ? kthread_complete_and_exit+0x20/0x20 [34166.832754] ret_from_fork+0x31/0x40 [34166.836322] ? kthread_complete_and_exit+0x20/0x20 [34166.841109] ret_from_fork_asm+0x11/0x20 [34166.845030] [34194.058374] watchdog: BUG: soft lockup - CPU#0 stuck for 20731s! [dbench:78683] [34194.065682] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34194.070373] watchdog: BUG: soft lockup - CPU#7 stuck for 29728s! [kworker/7:4:78808] [34194.121130] irq event stamp: 59091082 [34194.128862] Modules linked in: ceph libceph dns_resolver fscache netfs veth [34194.132518] hardirqs last enabled at (59091081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34194.139478] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [34194.149636] hardirqs last disabled at (59091082): [] sysvec_apic_timer_interrupt+0xe/0x90 [34194.155287] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [34194.165011] softirqs last enabled at (59091054): [] __do_softirq+0x2be/0x371 [34194.165015] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [34194.181086] softirqs last disabled at (59091049): [] irq_exit_rcu+0xb0/0xd0 [34194.187951] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [34194.196464] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34194.196465] kvm [34194.196467] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34194.202112] irqbypass i2c_i801 [34194.211058] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34194.212896] iTCO_wdt [34194.220366] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34194.223502] iTCO_vendor_support [34194.228979] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34194.231252] i2c_smbus crct10dif_pclmul [34194.249988] [34194.249989] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34194.253214] joydev [34194.258430] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34194.262261] crc32_pclmul [34194.263768] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34194.270893] ipmi_ssif [34194.272991] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34194.280116] ghash_clmulni_intel mei_me [34194.282732] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34194.289855] mei lpc_ich [34194.292212] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34194.299338] sha512_ssse3 [34194.303167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34194.310291] acpi_ipmi mfd_core [34194.312823] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34194.320900] wmi ipmi_si [34194.323517] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34194.329256] ipmi_devintf ipmi_msghandler [34194.332393] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34194.339516] ioatdma acpi_pad [34194.342051] Call Trace: [34194.349173] acpi_power_meter ixgbe [34194.353177] [34194.360299] igb mdio [34194.363267] ? watchdog_timer_fn+0x1b4/0x220 [34194.365709] crc32c_intel nvme [34194.369193] ? __hrtimer_run_queues+0x1d3/0x360 [34194.371204] ptp [34194.373476] ? hrtimer_interrupt+0x104/0x240 [34194.377737] nvme_core i2c_algo_bit [34194.380789] ? __local_bh_enable+0x50/0x60 [34194.385312] pps_core [34194.387153] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34194.391414] dca fuse [34194.394899] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34194.398987] [34194.398988] irq event stamp: 194501738 [34194.401260] [34194.406478] hardirqs last enabled at (194501737): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34194.408747] [34194.413722] hardirqs last disabled at (194501738): [] sysvec_apic_timer_interrupt+0xe/0x90 [34194.415215] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34194.418957] softirqs last enabled at (194501736): [] __do_softirq+0x2be/0x371 [34194.421059] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34194.431297] softirqs last disabled at (194501731): [] irq_exit_rcu+0xb0/0xd0 [34194.433397] do_raw_spin_lock+0xb5/0xc0 [34194.443207] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34194.448512] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34194.457281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34194.462155] try_get_cap_refs+0x159/0x880 [ceph] [34194.470768] Workqueue: events delayed_work [ceph] [34194.474603] __ceph_get_caps+0x4a1/0x760 [ceph] [34194.483976] [34194.488778] ? __file_remove_privs+0x89/0x110 [34194.496247] RIP: 0010:lock_acquire+0xf0/0x2b0 [34194.500860] ? wake_bit_function+0x70/0x70 [34194.505555] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34194.510083] ? ceph_write_iter+0x260/0x770 [ceph] [34194.511572] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34194.515922] ceph_write_iter+0x260/0x770 [ceph] [34194.520272] [34194.520273] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34194.524368] ? lock_is_held_type+0xaa/0x120 [34194.543099] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34194.547801] vfs_write+0x324/0x4f0 [34194.553016] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34194.557545] ksys_pwrite64+0x65/0xa0 [34194.559030] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34194.566156] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34194.570331] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34194.577457] do_syscall_64+0x38/0x80 [34194.580852] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34194.587979] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34194.591549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34194.598672] RIP: 0033:0x7f48f0eeb93b [34194.603803] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34194.610927] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34194.614498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34194.622575] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34194.627620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34194.633357] ORIG_RAX: 0000000000000012 [34194.636927] Call Trace: [34194.636928] [34194.644059] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34194.662800] ? watchdog_timer_fn+0x1b4/0x220 [34194.669921] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34194.675141] ? __hrtimer_run_queues+0x1d3/0x360 [34194.682265] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34194.686098] ? hrtimer_interrupt+0x104/0x240 [34194.688538] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34194.690549] ? __local_bh_enable+0x50/0x60 [34194.697674] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34194.701940] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34194.709070] [34194.713587] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34194.755672] [34194.757779] [34194.759877] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34194.765183] ? lock_acquire+0xf0/0x2b0 [34194.768933] ? igrab+0x43/0x50 [34194.771985] ? lock_release+0xc5/0x260 [34194.775730] _raw_spin_lock+0x30/0x40 [34194.779395] ? igrab+0x1d/0x50 [34194.782445] igrab+0x1d/0x50 [34194.785323] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34194.790472] delayed_work+0x1d8/0x280 [ceph] [34194.794774] ? process_scheduled_works+0x22a/0x510 [34194.799561] process_scheduled_works+0x2a4/0x510 [34194.804176] worker_thread+0x188/0x340 [34194.807926] ? rescuer_thread+0x310/0x310 [34194.811928] kthread+0xf5/0x130 [34194.815080] ? kthread_complete_and_exit+0x20/0x20 [34194.819868] ret_from_fork+0x31/0x40 [34194.823448] ? kthread_complete_and_exit+0x20/0x20 [34194.828232] ret_from_fork_asm+0x11/0x20 [34194.832164] [34222.058839] watchdog: BUG: soft lockup - CPU#0 stuck for 20757s! [dbench:78683] [34222.066143] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34222.070839] watchdog: BUG: soft lockup - CPU#7 stuck for 29754s! [kworker/7:4:78808] [34222.121583] irq event stamp: 59149146 [34222.121584] hardirqs last enabled at (59149145): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34222.129314] Modules linked in: [34222.132970] hardirqs last disabled at (59149146): [] sysvec_apic_timer_interrupt+0xe/0x90 [34222.143129] ceph [34222.146180] softirqs last enabled at (59149118): [] __do_softirq+0x2be/0x371 [34222.155905] libceph dns_resolver [34222.157828] softirqs last disabled at (59149113): [] irq_exit_rcu+0xb0/0xd0 [34222.166523] fscache netfs [34222.169830] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34222.178343] veth nft_chain_nat [34222.181047] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34222.189991] xt_MASQUERADE [34222.193127] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34222.200598] nf_nat nf_conntrack [34222.203304] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34222.208780] nf_defrag_ipv6 nf_defrag_ipv4 [34222.212005] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34222.230741] nft_compat nf_tables [34222.234831] [34222.234832] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34222.240049] nfnetlink bridge [34222.243362] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34222.244853] stp llc [34222.251976] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34222.254940] xfs libcrc32c [34222.262064] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34222.264247] sunrpc intel_rapl_msr [34222.271371] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34222.274076] intel_rapl_common [34222.281201] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34222.284596] sb_edac x86_pkg_temp_thermal [34222.291720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34222.294772] coretemp kvm_intel [34222.302851] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34222.306855] kvm irqbypass [34222.312591] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34222.315729] i2c_i801 [34222.322854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34222.325558] iTCO_wdt iTCO_vendor_support [34222.332682] Call Trace: [34222.334953] i2c_smbus crct10dif_pclmul [34222.342076] [34222.346080] joydev crc32_pclmul [34222.348524] ? watchdog_timer_fn+0x1b4/0x220 [34222.352354] ipmi_ssif ghash_clmulni_intel [34222.354366] ? __hrtimer_run_queues+0x1d3/0x360 [34222.357590] mei_me [34222.361856] ? hrtimer_interrupt+0x104/0x240 [34222.365945] mei lpc_ich [34222.370469] ? __local_bh_enable+0x50/0x60 [34222.372565] sha512_ssse3 [34222.376833] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34222.379360] acpi_ipmi mfd_core [34222.383452] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34222.386069] wmi ipmi_si [34222.391286] [34222.394424] ipmi_devintf ipmi_msghandler [34222.399382] [34222.399383] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34222.401913] ioatdma [34222.404014] ? queued_spin_lock_slowpath+0x16/0x2e0 [34222.408013] acpi_pad acpi_power_meter [34222.410112] do_raw_spin_lock+0xb5/0xc0 [34222.415414] ixgbe [34222.417600] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34222.422469] igb mdio [34222.426235] try_get_cap_refs+0x159/0x880 [ceph] [34222.430062] crc32c_intel [34222.432076] __ceph_get_caps+0x4a1/0x760 [ceph] [34222.436857] nvme ptp [34222.439132] ? __file_remove_privs+0x89/0x110 [34222.443738] nvme_core i2c_algo_bit [34222.446354] ? wake_bit_function+0x70/0x70 [34222.450879] pps_core [34222.453153] ? ceph_write_iter+0x260/0x770 [ceph] [34222.457500] dca fuse [34222.460985] ceph_write_iter+0x260/0x770 [ceph] [34222.465075] [34222.465075] irq event stamp: 194722534 [34222.467350] ? lock_is_held_type+0xaa/0x120 [34222.472043] hardirqs last enabled at (194722533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34222.474318] vfs_write+0x324/0x4f0 [34222.478839] hardirqs last disabled at (194722534): [] sysvec_apic_timer_interrupt+0xe/0x90 [34222.480334] ksys_pwrite64+0x65/0xa0 [34222.484073] softirqs last enabled at (194722532): [] __do_softirq+0x2be/0x371 [34222.488249] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34222.498495] softirqs last disabled at (194722527): [] irq_exit_rcu+0xb0/0xd0 [34222.501894] do_syscall_64+0x38/0x80 [34222.511703] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34222.515274] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34222.524045] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34222.529177] RIP: 0033:0x7f48f0eeb93b [34222.537774] Workqueue: events delayed_work [ceph] [34222.541344] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34222.550722] [34222.550723] RIP: 0010:lock_acquire+0xf0/0x2b0 [34222.555766] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34222.563241] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34222.566815] ORIG_RAX: 0000000000000012 [34222.571528] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34222.590268] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34222.591759] [34222.591760] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34222.596108] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34222.601327] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34222.620064] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34222.623894] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34222.629114] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34222.636241] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34222.637736] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34222.644861] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34222.651989] [34222.659109] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34222.712123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34222.717860] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34222.724985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34222.732110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34222.739242] Call Trace: [34222.741687] [34222.743699] ? watchdog_timer_fn+0x1b4/0x220 [34222.747962] ? __hrtimer_run_queues+0x1d3/0x360 [34222.752489] ? hrtimer_interrupt+0x104/0x240 [34222.756758] ? __local_bh_enable+0x50/0x60 [34222.760853] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34222.766074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34222.771032] [34222.773131] [34222.775245] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34222.780552] ? lock_acquire+0xf0/0x2b0 [34222.784304] ? igrab+0x43/0x50 [34222.787352] ? lock_release+0xc5/0x260 [34222.791099] _raw_spin_lock+0x30/0x40 [34222.794763] ? igrab+0x1d/0x50 [34222.797813] igrab+0x1d/0x50 [34222.800690] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34222.805841] delayed_work+0x1d8/0x280 [ceph] [34222.810122] ? process_scheduled_works+0x22a/0x510 [34222.814912] process_scheduled_works+0x2a4/0x510 [34222.819530] worker_thread+0x188/0x340 [34222.823276] ? rescuer_thread+0x310/0x310 [34222.827281] kthread+0xf5/0x130 [34222.830417] ? kthread_complete_and_exit+0x20/0x20 [34222.835203] ret_from_fork+0x31/0x40 [34222.838772] ? kthread_complete_and_exit+0x20/0x20 [34222.843557] ret_from_fork_asm+0x11/0x20 [34222.847481] [34250.059305] watchdog: BUG: soft lockup - CPU#0 stuck for 20783s! [dbench:78683] [34250.066614] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34250.071304] watchdog: BUG: soft lockup - CPU#7 stuck for 29780s! [kworker/7:4:78808] [34250.122063] irq event stamp: 59207358 [34250.129792] Modules linked in: ceph libceph dns_resolver fscache netfs [34250.133451] hardirqs last enabled at (59207357): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34250.139977] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [34250.150133] hardirqs last disabled at (59207358): [] sysvec_apic_timer_interrupt+0xe/0x90 [34250.157518] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [34250.167242] softirqs last enabled at (59207330): [] __do_softirq+0x2be/0x371 [34250.173673] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [34250.182375] softirqs last disabled at (59207325): [] irq_exit_rcu+0xb0/0xd0 [34250.188902] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [34250.197413] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34250.197414] kvm [34250.197415] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34250.203063] irqbypass i2c_i801 [34250.212008] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34250.213845] iTCO_wdt [34250.221316] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34250.224453] iTCO_vendor_support i2c_smbus [34250.229930] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34250.232200] crct10dif_pclmul joydev [34250.250937] [34250.250938] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34250.255029] crc32_pclmul ipmi_ssif [34250.260246] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34250.263819] ghash_clmulni_intel [34250.265308] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34250.272432] mei_me mei [34250.275915] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34250.283040] lpc_ich [34250.286263] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34250.293389] sha512_ssse3 acpi_ipmi [34250.295833] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34250.302957] mfd_core wmi [34250.305158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34250.312282] ipmi_si [34250.315767] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34250.323844] ipmi_devintf [34250.326462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34250.332198] ipmi_msghandler [34250.334383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34250.341507] ioatdma [34250.344125] Call Trace: [34250.344127] [34250.351250] acpi_pad [34250.354129] ? watchdog_timer_fn+0x1b4/0x220 [34250.361249] acpi_power_meter [34250.363434] ? __hrtimer_run_queues+0x1d3/0x360 [34250.365876] ixgbe igb [34250.367893] ? hrtimer_interrupt+0x104/0x240 [34250.370160] mdio crc32c_intel [34250.374423] ? __local_bh_enable+0x50/0x60 [34250.377388] nvme ptp [34250.381914] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34250.384267] nvme_core i2c_algo_bit [34250.388533] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34250.391582] pps_core dca [34250.395673] [34250.397945] fuse [34250.403163] [34250.403164] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34250.406646] [34250.406647] irq event stamp: 194943244 [34250.411608] ? queued_spin_lock_slowpath+0x16/0x2e0 [34250.414221] hardirqs last enabled at (194943243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34250.416321] do_raw_spin_lock+0xb5/0xc0 [34250.418243] hardirqs last disabled at (194943244): [] sysvec_apic_timer_interrupt+0xe/0x90 [34250.420342] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34250.425644] softirqs last enabled at (194943242): [] __do_softirq+0x2be/0x371 [34250.427137] try_get_cap_refs+0x159/0x880 [ceph] [34250.430878] softirqs last disabled at (194943237): [] irq_exit_rcu+0xb0/0xd0 [34250.435754] __ceph_get_caps+0x4a1/0x760 [ceph] [34250.446001] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34250.449837] ? __file_remove_privs+0x89/0x110 [34250.459644] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34250.464428] ? wake_bit_function+0x70/0x70 [34250.473199] Workqueue: events delayed_work [ceph] [34250.477811] ? ceph_write_iter+0x260/0x770 [ceph] [34250.486407] [34250.486408] RIP: 0010:lock_release+0x5f/0x260 [34250.490931] ceph_write_iter+0x260/0x770 [ceph] [34250.500309] Code: 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 <65> 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 [34250.504663] ? lock_is_held_type+0xaa/0x120 [34250.512129] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [34250.516222] vfs_write+0x324/0x4f0 [34250.520917] [34250.520918] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [34250.525620] ksys_pwrite64+0x65/0xa0 [34250.527106] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: 0000000000000001 [34250.531459] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34250.535980] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [34250.554738] do_syscall_64+0x38/0x80 [34250.558914] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [34250.564132] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34250.567528] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34250.569019] RIP: 0033:0x7f48f0eeb93b [34250.576143] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34250.579716] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34250.586839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34250.591985] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34250.599109] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34250.602681] ORIG_RAX: 0000000000000012 [34250.609822] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34250.614865] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34250.621990] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34250.625562] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34250.633640] Call Trace: [34250.633642] [34250.652376] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34250.658116] ? watchdog_timer_fn+0x1b4/0x220 [34250.663332] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34250.670456] ? __hrtimer_run_queues+0x1d3/0x360 [34250.674287] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34250.681413] ? hrtimer_interrupt+0x104/0x240 [34250.688540] [34250.695658] ? __local_bh_enable+0x50/0x60 [34250.747940] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34250.753164] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34250.758121] [34250.760218] [34250.762316] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34250.767621] ? igrab+0x43/0x50 [34250.770672] ? igrab+0x43/0x50 [34250.773721] ? lock_release+0x5f/0x260 [34250.777467] ? lock_release+0x1d7/0x260 [34250.781298] _raw_spin_unlock+0x1b/0x40 [34250.785138] igrab+0x43/0x50 [34250.788015] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34250.793164] delayed_work+0x1d8/0x280 [ceph] [34250.797447] ? process_scheduled_works+0x22a/0x510 [34250.802234] process_scheduled_works+0x2a4/0x510 [34250.806851] worker_thread+0x188/0x340 [34250.810599] ? rescuer_thread+0x310/0x310 [34250.814602] kthread+0xf5/0x130 [34250.817742] ? kthread_complete_and_exit+0x20/0x20 [34250.822526] ret_from_fork+0x31/0x40 [34250.826103] ? kthread_complete_and_exit+0x20/0x20 [34250.830888] ret_from_fork_asm+0x11/0x20 [34250.834819] [34278.059770] watchdog: BUG: soft lockup - CPU#0 stuck for 20809s! [dbench:78683] [34278.067073] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34278.071770] watchdog: BUG: soft lockup - CPU#7 stuck for 29806s! [kworker/7:4:78808] [34278.122514] irq event stamp: 59265506 [34278.122515] hardirqs last enabled at (59265505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34278.130247] Modules linked in: [34278.133902] hardirqs last disabled at (59265506): [] sysvec_apic_timer_interrupt+0xe/0x90 [34278.144061] ceph [34278.147111] softirqs last enabled at (59265480): [] __do_softirq+0x2be/0x371 [34278.156836] libceph [34278.158761] softirqs last disabled at (59265475): [] irq_exit_rcu+0xb0/0xd0 [34278.167444] dns_resolver [34278.169628] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34278.178139] fscache [34278.180774] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34278.180775] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34278.189717] netfs veth [34278.191902] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34278.199373] nft_chain_nat [34278.204849] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34278.207293] xt_MASQUERADE [34278.226032] [34278.226033] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34278.228735] nf_nat nf_conntrack [34278.233953] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34278.236657] nf_defrag_ipv6 nf_defrag_ipv4 [34278.238164] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34278.245289] nft_compat nf_tables [34278.248512] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34278.255637] nfnetlink bridge [34278.259730] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34278.266853] stp llc [34278.270167] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34278.277288] xfs libcrc32c [34278.280251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34278.287375] sunrpc [34278.289559] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34278.297637] intel_rapl_msr [34278.300342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34278.306079] intel_rapl_common [34278.308177] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34278.315299] sb_edac x86_pkg_temp_thermal [34278.318090] Call Trace: [34278.318091] [34278.325215] coretemp [34278.328267] ? watchdog_timer_fn+0x1b4/0x220 [34278.335390] kvm_intel [34278.339395] ? __hrtimer_run_queues+0x1d3/0x360 [34278.341838] kvm [34278.343852] ? hrtimer_interrupt+0x104/0x240 [34278.346119] irqbypass i2c_i801 [34278.350383] ? __local_bh_enable+0x50/0x60 [34278.352740] iTCO_wdt [34278.357269] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34278.359102] iTCO_vendor_support [34278.363366] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34278.366504] i2c_smbus [34278.370595] [34278.372865] crct10dif_pclmul joydev [34278.378083] [34278.378085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34278.381305] crc32_pclmul ipmi_ssif [34278.386269] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34278.388621] ghash_clmulni_intel [34278.390721] do_raw_spin_lock+0xb5/0xc0 [34278.394291] mei_me [34278.396388] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34278.401690] mei lpc_ich [34278.405177] try_get_cap_refs+0x159/0x880 [ceph] [34278.410046] sha512_ssse3 acpi_ipmi [34278.413276] __ceph_get_caps+0x4a1/0x760 [ceph] [34278.417101] mfd_core wmi [34278.419203] ? __file_remove_privs+0x89/0x110 [34278.423983] ipmi_si ipmi_devintf [34278.426513] ? wake_bit_function+0x70/0x70 [34278.431124] ipmi_msghandler ioatdma [34278.434609] ? ceph_write_iter+0x260/0x770 [ceph] [34278.439131] acpi_pad acpi_power_meter [34278.441749] ceph_write_iter+0x260/0x770 [ceph] [34278.446101] ixgbe igb [34278.449416] ? lock_is_held_type+0xaa/0x120 [34278.453501] mdio crc32c_intel [34278.457075] vfs_write+0x324/0x4f0 [34278.461770] nvme ptp [34278.465517] ksys_pwrite64+0x65/0xa0 [34278.470036] nvme_core i2c_algo_bit [34278.472396] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34278.476573] pps_core dca [34278.479624] do_syscall_64+0x38/0x80 [34278.483021] fuse [34278.485291] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34278.488869] [34278.488870] irq event stamp: 195163966 [34278.492354] RIP: 0033:0x7f48f0eeb93b [34278.497486] hardirqs last enabled at (195163965): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34278.500103] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34278.503674] hardirqs last disabled at (195163966): [] sysvec_apic_timer_interrupt+0xe/0x90 [34278.505597] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34278.510640] softirqs last enabled at (195163964): [] __do_softirq+0x2be/0x371 [34278.512134] ORIG_RAX: 0000000000000012 [34278.515875] softirqs last disabled at (195163959): [] irq_exit_rcu+0xb0/0xd0 [34278.519450] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34278.529692] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34278.548430] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34278.558257] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34278.563474] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34278.572245] Workqueue: events delayed_work [ceph] [34278.576075] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34278.584673] [34278.591799] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34278.601174] RIP: 0010:lock_release+0x77/0x260 [34278.608305] [34278.615771] Code: 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c <58> 0f 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 [34278.668605] RSP: 0018:ffffc90002117d40 EFLAGS: 00000246 [34278.673822] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 000000005553a634 [34278.680946] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [34278.688070] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [34278.695194] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [34278.702318] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34278.709457] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34278.717536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34278.723273] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34278.730399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34278.737521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34278.744648] Call Trace: [34278.747089] [34278.749102] ? watchdog_timer_fn+0x1b4/0x220 [34278.753366] ? __hrtimer_run_queues+0x1d3/0x360 [34278.757892] ? hrtimer_interrupt+0x104/0x240 [34278.762172] ? __local_bh_enable+0x50/0x60 [34278.766263] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34278.771480] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34278.776438] [34278.778532] [34278.780632] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34278.785935] ? igrab+0x43/0x50 [34278.788990] ? lock_release+0x77/0x260 [34278.792741] _raw_spin_unlock+0x1b/0x40 [34278.796579] igrab+0x43/0x50 [34278.799457] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34278.804605] delayed_work+0x1d8/0x280 [ceph] [34278.808886] ? process_scheduled_works+0x22a/0x510 [34278.813668] process_scheduled_works+0x2a4/0x510 [34278.818285] worker_thread+0x188/0x340 [34278.822032] ? rescuer_thread+0x310/0x310 [34278.826038] kthread+0xf5/0x130 [34278.829177] ? kthread_complete_and_exit+0x20/0x20 [34278.833967] ret_from_fork+0x31/0x40 [34278.837539] ? kthread_complete_and_exit+0x20/0x20 [34278.842323] ret_from_fork_asm+0x11/0x20 [34278.846246] [34279.678163] rcu: INFO: rcu_preempt self-detected stall on CPU [34279.683924] rcu: 0-....: (21838199 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=10932325 [34279.694435] rcu: (t=22342672 jiffies g=1309609 q=629454 ncpus=8) [34279.700521] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34279.709463] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34279.716951] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34279.722430] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34279.741166] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34279.746401] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34279.753525] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34279.760650] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34279.767773] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34279.774898] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34279.782021] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34279.790099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34279.795837] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34279.802961] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34279.810084] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34279.817208] Call Trace: [34279.819655] [34279.821666] ? rcu_dump_cpu_stacks+0xd1/0x180 [34279.826025] ? rcu_sched_clock_irq+0x633/0x1390 [34279.830548] ? lock_is_held_type+0xaa/0x120 [34279.834727] ? lock_is_held_type+0xaa/0x120 [34279.838903] ? tick_sched_do_timer+0xa0/0xa0 [34279.843192] ? rcu_read_lock_sched_held+0x56/0x90 [34279.847890] ? tick_sched_do_timer+0xa0/0xa0 [34279.852155] ? update_process_times+0x5f/0x90 [34279.856504] ? tick_sched_handle.isra.16+0x1e/0x50 [34279.861290] ? tick_sched_timer+0x6f/0x80 [34279.865293] ? __hrtimer_run_queues+0x1d3/0x360 [34279.869821] ? hrtimer_interrupt+0x104/0x240 [34279.874090] ? __local_bh_enable+0x50/0x60 [34279.878196] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34279.883416] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34279.888372] [34279.890484] [34279.892585] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34279.897894] ? queued_spin_lock_slowpath+0x18/0x2e0 [34279.902772] do_raw_spin_lock+0xb5/0xc0 [34279.906608] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34279.911413] try_get_cap_refs+0x159/0x880 [ceph] [34279.916042] __ceph_get_caps+0x4a1/0x760 [ceph] [34279.920582] ? __file_remove_privs+0x89/0x110 [34279.924938] ? wake_bit_function+0x70/0x70 [34279.929031] ? ceph_write_iter+0x260/0x770 [ceph] [34279.933744] ceph_write_iter+0x260/0x770 [ceph] [34279.938281] ? lock_is_held_type+0xaa/0x120 [34279.942481] vfs_write+0x324/0x4f0 [34279.945881] ksys_pwrite64+0x65/0xa0 [34279.949457] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34279.954588] do_syscall_64+0x38/0x80 [34279.958159] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34279.963202] RIP: 0033:0x7f48f0eeb93b [34279.966773] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34279.985510] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [34279.993069] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34280.000193] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34280.007317] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34280.014439] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34280.021565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34280.028694] [34280.030882] Sending NMI from CPU 0 to CPUs 7: [34280.035235] NMI backtrace for cpu 7 [34280.035237] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34280.035239] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34280.035240] Workqueue: events delayed_work [ceph] [34280.035261] RIP: 0010:mark_lock+0x55/0x4c0 [34280.035264] Code: 00 00 00 49 89 ff 49 89 f5 89 d3 83 fa 08 74 6c 41 be 01 00 00 00 89 d1 41 d3 e6 4d 63 f6 41 0f b7 45 20 66 25 ff 1f 0f b7 c0 <48> 0f a3 05 d3 7e e0 01 73 79 48 8d 14 40 48 8d 04 90 48 c1 e0 04 [34280.035266] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000006 [34280.035268] RAX: 00000000000006f5 RBX: 0000000000000002 RCX: 0000000000000002 [34280.035269] RDX: 0000000000000002 RSI: ffff88810debb550 RDI: ffff88810debaa80 [34280.035270] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [34280.035272] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [34280.035273] R13: ffff88810debb550 R14: 0000000000000004 R15: ffff88810debaa80 [34280.035274] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34280.035276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34280.035277] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34280.035279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34280.035280] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34280.035281] Call Trace: [34280.035282] [34280.035284] ? nmi_cpu_backtrace+0xad/0x120 [34280.035288] ? nmi_cpu_backtrace_handler+0x11/0x20 [34280.035290] ? nmi_handle+0xd4/0x260 [34280.035295] ? default_do_nmi+0x49/0xf0 [34280.035298] ? exc_nmi+0x11e/0x150 [34280.035300] ? end_repeat_nmi+0x16/0x67 [34280.035306] ? mark_lock+0x55/0x4c0 [34280.035309] ? mark_lock+0x55/0x4c0 [34280.035312] ? mark_lock+0x55/0x4c0 [34280.035314] [34280.035315] [34280.035316] ? _raw_spin_unlock+0x23/0x40 [34280.035319] ? ceph_check_caps+0x275/0xae0 [ceph] [34280.035340] __lock_acquire+0x49a/0x1e20 [34280.035345] lock_acquire+0xcb/0x2b0 [34280.035347] ? igrab+0x1d/0x50 [34280.035350] ? igrab+0x43/0x50 [34280.035352] ? lock_release+0xc5/0x260 [34280.035355] _raw_spin_lock+0x30/0x40 [34280.035358] ? igrab+0x1d/0x50 [34280.035359] igrab+0x1d/0x50 [34280.035361] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34280.035380] delayed_work+0x1d8/0x280 [ceph] [34280.035400] ? process_scheduled_works+0x22a/0x510 [34280.035403] process_scheduled_works+0x2a4/0x510 [34280.035409] worker_thread+0x188/0x340 [34280.035412] ? rescuer_thread+0x310/0x310 [34280.035415] kthread+0xf5/0x130 [34280.035418] ? kthread_complete_and_exit+0x20/0x20 [34280.035422] ret_from_fork+0x31/0x40 [34280.035424] ? kthread_complete_and_exit+0x20/0x20 [34280.035427] ret_from_fork_asm+0x11/0x20 [34280.035434] [34306.060234] watchdog: BUG: soft lockup - CPU#0 stuck for 20836s! [dbench:78683] [34306.067536] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34306.072233] watchdog: BUG: soft lockup - CPU#7 stuck for 29833s! [kworker/7:4:78808] [34306.122985] irq event stamp: 59322982 [34306.122986] hardirqs last enabled at (59322981): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34306.130716] Modules linked in: [34306.134373] hardirqs last disabled at (59322982): [] sysvec_apic_timer_interrupt+0xe/0x90 [34306.144531] ceph [34306.147581] softirqs last enabled at (59322954): [] __do_softirq+0x2be/0x371 [34306.157307] libceph [34306.159230] softirqs last disabled at (59322949): [] irq_exit_rcu+0xb0/0xd0 [34306.167913] dns_resolver fscache [34306.170100] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34306.178626] netfs veth [34306.181937] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34306.190882] nft_chain_nat [34306.193328] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34306.200795] xt_MASQUERADE nf_nat [34306.203501] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34306.208977] nf_conntrack [34306.212290] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34306.231026] nf_defrag_ipv6 nf_defrag_ipv4 [34306.233644] [34306.233645] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34306.238861] nft_compat [34306.242952] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34306.244444] nf_tables [34306.251567] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34306.254011] nfnetlink bridge [34306.261134] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34306.263491] stp llc [34306.270634] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34306.273597] xfs libcrc32c [34306.280721] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34306.282919] sunrpc [34306.290047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34306.292751] intel_rapl_msr [34306.300828] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34306.302927] intel_rapl_common [34306.308664] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34306.311453] sb_edac [34306.318579] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34306.321633] x86_pkg_temp_thermal [34306.328753] Call Trace: [34306.328755] [34306.330936] coretemp kvm_intel [34306.338064] ? watchdog_timer_fn+0x1b4/0x220 [34306.341372] kvm [34306.343819] ? __hrtimer_run_queues+0x1d3/0x360 [34306.345827] irqbypass i2c_i801 [34306.348968] ? hrtimer_interrupt+0x104/0x240 [34306.353228] iTCO_wdt [34306.355065] ? __local_bh_enable+0x50/0x60 [34306.359589] iTCO_vendor_support [34306.362730] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34306.366992] i2c_smbus [34306.369263] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34306.373352] crct10dif_pclmul [34306.376576] [34306.381795] joydev [34306.384151] [34306.384153] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34306.389110] crc32_pclmul ipmi_ssif [34306.392078] ? queued_spin_lock_slowpath+0x16/0x2e0 [34306.394170] ghash_clmulni_intel [34306.396270] do_raw_spin_lock+0xb5/0xc0 [34306.398365] mei_me [34306.403673] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34306.407153] mei lpc_ich [34306.412028] try_get_cap_refs+0x159/0x880 [ceph] [34306.415248] sha512_ssse3 acpi_ipmi [34306.419085] __ceph_get_caps+0x4a1/0x760 [ceph] [34306.421176] mfd_core wmi [34306.425965] ? __file_remove_privs+0x89/0x110 [34306.428491] ipmi_si ipmi_devintf [34306.433104] ? wake_bit_function+0x70/0x70 [34306.436586] ipmi_msghandler ioatdma [34306.441112] ? ceph_write_iter+0x260/0x770 [ceph] [34306.443727] acpi_pad acpi_power_meter [34306.448080] ceph_write_iter+0x260/0x770 [ceph] [34306.451389] ixgbe igb [34306.455484] ? lock_is_held_type+0xaa/0x120 [34306.459049] mdio [34306.463750] vfs_write+0x324/0x4f0 [34306.467491] crc32c_intel [34306.472019] ksys_pwrite64+0x65/0xa0 [34306.474375] nvme [34306.478553] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34306.480475] ptp nvme_core [34306.483873] do_syscall_64+0x38/0x80 [34306.486490] i2c_algo_bit [34306.490062] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34306.491983] pps_core dca [34306.497117] RIP: 0033:0x7f48f0eeb93b [34306.499819] fuse [34306.503391] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34306.506008] irq event stamp: 195382642 [34306.511051] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34306.513669] hardirqs last enabled at (195382641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34306.517238] ORIG_RAX: 0000000000000012 [34306.519164] hardirqs last disabled at (195382642): [] sysvec_apic_timer_interrupt+0xe/0x90 [34306.537914] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34306.541663] softirqs last enabled at (195382640): [] __do_softirq+0x2be/0x371 [34306.546879] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34306.557125] softirqs last disabled at (195382635): [] irq_exit_rcu+0xb0/0xd0 [34306.560955] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34306.570766] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34306.577890] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34306.586678] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34306.593802] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34306.602399] Workqueue: events delayed_work [ceph] [34306.609530] [34306.618902] [34306.618919] RIP: 0010:lock_acquire+0xf0/0x2b0 [34306.653361] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34306.672098] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34306.677317] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34306.684439] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34306.691563] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34306.698688] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34306.705812] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34306.712937] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34306.721015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34306.726751] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34306.733876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34306.740999] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34306.748124] Call Trace: [34306.750569] [34306.752581] ? watchdog_timer_fn+0x1b4/0x220 [34306.756844] ? __hrtimer_run_queues+0x1d3/0x360 [34306.761372] ? hrtimer_interrupt+0x104/0x240 [34306.765638] ? __local_bh_enable+0x50/0x60 [34306.769734] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34306.774958] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34306.779919] [34306.782019] [34306.784118] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34306.789427] ? lock_acquire+0xf0/0x2b0 [34306.793178] ? igrab+0x43/0x50 [34306.796233] ? lock_release+0xc5/0x260 [34306.799980] _raw_spin_lock+0x30/0x40 [34306.803646] ? igrab+0x1d/0x50 [34306.806695] igrab+0x1d/0x50 [34306.809574] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34306.814722] delayed_work+0x1d8/0x280 [ceph] [34306.819005] ? process_scheduled_works+0x22a/0x510 [34306.823793] process_scheduled_works+0x2a4/0x510 [34306.828409] worker_thread+0x188/0x340 [34306.832158] ? rescuer_thread+0x310/0x310 [34306.836162] kthread+0xf5/0x130 [34306.839300] ? kthread_complete_and_exit+0x20/0x20 [34306.844083] ret_from_fork+0x31/0x40 [34306.847663] ? kthread_complete_and_exit+0x20/0x20 [34306.852447] ret_from_fork_asm+0x11/0x20 [34306.856380] [34334.060697] watchdog: BUG: soft lockup - CPU#0 stuck for 20862s! [dbench:78683] [34334.067997] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34334.072697] watchdog: BUG: soft lockup - CPU#7 stuck for 29859s! [kworker/7:4:78808] [34334.123437] irq event stamp: 59381210 [34334.123439] hardirqs last enabled at (59381209): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34334.131171] Modules linked in: [34334.134826] hardirqs last disabled at (59381210): [] sysvec_apic_timer_interrupt+0xe/0x90 [34334.144984] ceph [34334.148034] softirqs last enabled at (59381182): [] __do_softirq+0x2be/0x371 [34334.157759] libceph [34334.159682] softirqs last disabled at (59381177): [] irq_exit_rcu+0xb0/0xd0 [34334.168371] dns_resolver [34334.170550] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34334.179063] fscache netfs [34334.181698] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34334.190640] veth [34334.193346] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34334.200834] nft_chain_nat [34334.202758] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34334.208235] xt_MASQUERADE [34334.210938] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34334.229676] nf_nat nf_conntrack [34334.232387] [34334.232388] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34334.237607] nf_defrag_ipv6 [34334.240830] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34334.242322] nf_defrag_ipv4 nft_compat [34334.249445] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34334.252237] nf_tables nfnetlink [34334.259359] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34334.263121] bridge [34334.270245] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34334.273471] stp llc [34334.280594] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34334.282693] xfs libcrc32c [34334.289816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34334.291999] sunrpc intel_rapl_msr [34334.300094] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34334.302798] intel_rapl_common [34334.308536] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34334.311933] sb_edac x86_pkg_temp_thermal [34334.319057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34334.322108] coretemp kvm_intel [34334.329233] Call Trace: [34334.329234] [34334.333237] kvm irqbypass [34334.340362] ? watchdog_timer_fn+0x1b4/0x220 [34334.343516] i2c_i801 iTCO_wdt [34334.345960] ? __hrtimer_run_queues+0x1d3/0x360 [34334.347971] iTCO_vendor_support [34334.350676] ? hrtimer_interrupt+0x104/0x240 [34334.354939] i2c_smbus crct10dif_pclmul [34334.357989] ? __local_bh_enable+0x50/0x60 [34334.362513] joydev [34334.365741] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34334.370000] crc32_pclmul [34334.373833] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34334.377922] ipmi_ssif ghash_clmulni_intel [34334.380020] [34334.385238] mei_me mei [34334.387855] [34334.387857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34334.392812] lpc_ich sha512_ssse3 [34334.396908] ? queued_spin_lock_slowpath+0x18/0x2e0 [34334.399000] acpi_ipmi [34334.401447] do_raw_spin_lock+0xb5/0xc0 [34334.403542] mfd_core wmi [34334.408848] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34334.412158] ipmi_si ipmi_devintf [34334.417030] try_get_cap_refs+0x159/0x880 [ceph] [34334.419386] ipmi_msghandler [34334.423219] __ceph_get_caps+0x4a1/0x760 [ceph] [34334.425833] ioatdma acpi_pad [34334.430622] ? __file_remove_privs+0x89/0x110 [34334.433929] acpi_power_meter ixgbe [34334.438539] ? wake_bit_function+0x70/0x70 [34334.441415] igb mdio [34334.445943] ? ceph_write_iter+0x260/0x770 [ceph] [34334.448904] crc32c_intel [34334.453255] ceph_write_iter+0x260/0x770 [ceph] [34334.456737] nvme ptp [34334.460833] ? lock_is_held_type+0xaa/0x120 [34334.463099] nvme_core [34334.467801] vfs_write+0x324/0x4f0 [34334.470416] i2c_algo_bit pps_core [34334.474944] ksys_pwrite64+0x65/0xa0 [34334.477211] dca fuse [34334.481386] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34334.483743] [34334.487143] do_syscall_64+0x38/0x80 [34334.490540] irq event stamp: 195603284 [34334.494111] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34334.496381] hardirqs last enabled at (195603283): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34334.501512] RIP: 0033:0x7f48f0eeb93b [34334.503003] hardirqs last disabled at (195603284): [] sysvec_apic_timer_interrupt+0xe/0x90 [34334.506573] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34334.510317] softirqs last enabled at (195603282): [] __do_softirq+0x2be/0x371 [34334.515361] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34334.525623] softirqs last disabled at (195603277): [] irq_exit_rcu+0xb0/0xd0 [34334.529194] ORIG_RAX: 0000000000000012 [34334.539005] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34334.557742] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34334.566512] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34334.571731] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34334.580329] Workqueue: events delayed_work [ceph] [34334.584159] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34334.593536] [34334.600660] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34334.608131] RIP: 0010:lock_acquire+0xf0/0x2b0 [34334.615254] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34334.619953] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34334.627099] [34334.628585] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34334.673322] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34334.680448] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34334.687572] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34334.694696] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34334.701820] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34334.708943] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34334.717022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34334.722758] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34334.729883] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34334.737007] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34334.744130] Call Trace: [34334.746574] [34334.748588] ? watchdog_timer_fn+0x1b4/0x220 [34334.752850] ? __hrtimer_run_queues+0x1d3/0x360 [34334.757378] ? hrtimer_interrupt+0x104/0x240 [34334.761656] ? __local_bh_enable+0x50/0x60 [34334.765750] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34334.770972] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34334.775930] [34334.778027] [34334.780126] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34334.785432] ? lock_acquire+0xf0/0x2b0 [34334.789184] ? igrab+0x43/0x50 [34334.792231] ? lock_release+0xc5/0x260 [34334.795977] _raw_spin_lock+0x30/0x40 [34334.799633] ? igrab+0x1d/0x50 [34334.802685] igrab+0x1d/0x50 [34334.805562] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34334.810711] delayed_work+0x1d8/0x280 [ceph] [34334.814993] ? process_scheduled_works+0x22a/0x510 [34334.819776] process_scheduled_works+0x2a4/0x510 [34334.824390] worker_thread+0x188/0x340 [34334.828141] ? rescuer_thread+0x310/0x310 [34334.832152] kthread+0xf5/0x130 [34334.835290] ? kthread_complete_and_exit+0x20/0x20 [34334.840073] ret_from_fork+0x31/0x40 [34334.843661] ? kthread_complete_and_exit+0x20/0x20 [34334.848445] ret_from_fork_asm+0x11/0x20 [34334.852369] [34362.061157] watchdog: BUG: soft lockup - CPU#0 stuck for 20888s! [dbench:78683] [34362.068461] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34362.073156] watchdog: BUG: soft lockup - CPU#7 stuck for 29885s! [kworker/7:4:78808] [34362.123909] irq event stamp: 59439400 [34362.131639] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [34362.135298] hardirqs last enabled at (59439399): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34362.143470] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [34362.153628] hardirqs last disabled at (59439400): [] sysvec_apic_timer_interrupt+0xe/0x90 [34362.161619] nf_tables nfnetlink bridge stp llc xfs libcrc32c [34362.171342] softirqs last enabled at (59439372): [] __do_softirq+0x2be/0x371 [34362.171346] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [34362.185764] softirqs last disabled at (59439367): [] irq_exit_rcu+0xb0/0xd0 [34362.194881] kvm irqbypass i2c_i801 iTCO_wdt [34362.203393] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34362.207658] iTCO_vendor_support i2c_smbus [34362.216601] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34362.216603] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34362.220691] crct10dif_pclmul joydev [34362.228163] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34362.233640] crc32_pclmul ipmi_ssif [34362.237210] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34362.255948] ghash_clmulni_intel mei_me [34362.259431] [34362.259432] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34362.264649] mei lpc_ich [34362.268481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34362.269972] sha512_ssse3 acpi_ipmi [34362.277094] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34362.279627] mfd_core wmi [34362.286749] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34362.290233] ipmi_si ipmi_devintf [34362.297357] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34362.299975] ipmi_msghandler [34362.307101] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34362.310411] ioatdma acpi_pad [34362.317551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34362.320429] acpi_power_meter [34362.328506] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34362.331470] ixgbe [34362.337208] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34362.340174] igb mdio [34362.347296] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34362.349308] crc32c_intel [34362.356430] Call Trace: [34362.358703] nvme ptp [34362.365836] [34362.368453] nvme_core [34362.370898] ? watchdog_timer_fn+0x1b4/0x220 [34362.373166] i2c_algo_bit [34362.375179] ? __hrtimer_run_queues+0x1d3/0x360 [34362.377534] pps_core [34362.381837] ? hrtimer_interrupt+0x104/0x240 [34362.384451] dca fuse [34362.388974] ? __local_bh_enable+0x50/0x60 [34362.391245] [34362.391246] irq event stamp: 195823960 [34362.395512] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34362.397781] hardirqs last enabled at (195823959): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34362.401873] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34362.403361] hardirqs last disabled at (195823960): [] sysvec_apic_timer_interrupt+0xe/0x90 [34362.407107] [34362.412324] softirqs last enabled at (195823958): [] __do_softirq+0x2be/0x371 [34362.422567] [34362.427526] softirqs last disabled at (195823953): [] irq_exit_rcu+0xb0/0xd0 [34362.437338] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34362.439435] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34362.448209] ? queued_spin_lock_slowpath+0x18/0x2e0 [34362.450302] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34362.458902] do_raw_spin_lock+0xb5/0xc0 [34362.464203] Workqueue: events delayed_work [ceph] [34362.473583] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34362.478450] [34362.485925] try_get_cap_refs+0x159/0x880 [ceph] [34362.489753] RIP: 0010:lock_acquire+0xf0/0x2b0 [34362.494455] __ceph_get_caps+0x4a1/0x760 [ceph] [34362.499235] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34362.500730] ? __file_remove_privs+0x89/0x110 [34362.505335] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34362.509687] ? wake_bit_function+0x70/0x70 [34362.514210] [34362.514211] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34362.532952] ? ceph_write_iter+0x260/0x770 [ceph] [34362.537300] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34362.542518] ceph_write_iter+0x260/0x770 [ceph] [34362.546606] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34362.548102] ? lock_is_held_type+0xaa/0x120 [34362.555221] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34362.559922] vfs_write+0x324/0x4f0 [34362.567043] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34362.571572] ksys_pwrite64+0x65/0xa0 [34362.578693] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34362.582871] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34362.589992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34362.593391] do_syscall_64+0x38/0x80 [34362.600514] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34362.604086] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34362.612162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34362.617295] RIP: 0033:0x7f48f0eeb93b [34362.623031] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34362.626603] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34362.633725] Call Trace: [34362.633727] [34362.638769] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34362.645897] ? watchdog_timer_fn+0x1b4/0x220 [34362.649466] ORIG_RAX: 0000000000000012 [34362.656590] ? __hrtimer_run_queues+0x1d3/0x360 [34362.675326] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34362.677776] ? hrtimer_interrupt+0x104/0x240 [34362.679782] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34362.684998] ? __local_bh_enable+0x50/0x60 [34362.689264] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34362.693097] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34362.697618] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34362.704742] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34362.709005] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34362.716131] [34362.720225] [34362.727345] [34362.758149] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34362.763466] ? lock_acquire+0xf0/0x2b0 [34362.767215] ? igrab+0x43/0x50 [34362.770265] ? lock_release+0xc5/0x260 [34362.774010] _raw_spin_lock+0x30/0x40 [34362.777674] ? igrab+0x1d/0x50 [34362.780724] igrab+0x1d/0x50 [34362.783603] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34362.788753] delayed_work+0x1d8/0x280 [ceph] [34362.793035] ? process_scheduled_works+0x22a/0x510 [34362.797824] process_scheduled_works+0x2a4/0x510 [34362.802439] worker_thread+0x188/0x340 [34362.806190] ? rescuer_thread+0x310/0x310 [34362.810202] kthread+0xf5/0x130 [34362.813349] ? kthread_complete_and_exit+0x20/0x20 [34362.818141] ret_from_fork+0x31/0x40 [34362.821719] ? kthread_complete_and_exit+0x20/0x20 [34362.826504] ret_from_fork_asm+0x11/0x20 [34362.830428] [34390.061612] watchdog: BUG: soft lockup - CPU#0 stuck for 20914s! [dbench:78683] [34390.068912] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34390.073613] watchdog: BUG: soft lockup - CPU#7 stuck for 29911s! [kworker/7:4:78808] [34390.124354] irq event stamp: 59497442 [34390.124355] hardirqs last enabled at (59497441): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34390.132083] Modules linked in: [34390.135742] hardirqs last disabled at (59497442): [] sysvec_apic_timer_interrupt+0xe/0x90 [34390.145899] ceph [34390.148950] softirqs last enabled at (59497414): [] __do_softirq+0x2be/0x371 [34390.158673] libceph [34390.160599] softirqs last disabled at (59497409): [] irq_exit_rcu+0xb0/0xd0 [34390.169294] dns_resolver fscache [34390.171475] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34390.180003] netfs [34390.183313] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34390.192257] veth nft_chain_nat [34390.194286] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34390.201756] xt_MASQUERADE nf_nat [34390.204893] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34390.210372] nf_conntrack nf_defrag_ipv6 [34390.213682] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34390.232420] nf_defrag_ipv4 nft_compat [34390.236336] [34390.241556] nf_tables [34390.245298] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34390.246788] nfnetlink bridge [34390.249146] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34390.256288] stp llc [34390.259253] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34390.266377] xfs libcrc32c [34390.268562] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34390.275685] sunrpc intel_rapl_msr [34390.278390] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34390.285513] intel_rapl_common sb_edac [34390.288910] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34390.296034] x86_pkg_temp_thermal [34390.299779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34390.307856] coretemp kvm_intel [34390.311166] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34390.316905] kvm irqbypass [34390.320042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34390.327165] i2c_i801 iTCO_wdt [34390.329870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34390.337011] iTCO_vendor_support [34390.340063] Call Trace: [34390.347187] i2c_smbus crct10dif_pclmul [34390.350411] [34390.352853] joydev crc32_pclmul [34390.356686] ? watchdog_timer_fn+0x1b4/0x220 [34390.358694] ipmi_ssif ghash_clmulni_intel [34390.361920] ? __hrtimer_run_queues+0x1d3/0x360 [34390.366184] mei_me mei [34390.370278] ? hrtimer_interrupt+0x104/0x240 [34390.374815] lpc_ich sha512_ssse3 [34390.377260] ? __local_bh_enable+0x50/0x60 [34390.381523] acpi_ipmi mfd_core [34390.384837] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34390.388924] wmi ipmi_si [34390.392065] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34390.397290] ipmi_devintf ipmi_msghandler [34390.399821] [34390.404777] ioatdma acpi_pad [34390.408782] [34390.408784] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34390.410879] acpi_power_meter [34390.413847] ? queued_spin_lock_slowpath+0x16/0x2e0 [34390.415941] ixgbe igb [34390.421247] do_raw_spin_lock+0xb5/0xc0 [34390.424209] mdio crc32c_intel [34390.429080] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34390.431437] nvme [34390.435271] try_get_cap_refs+0x159/0x880 [ceph] [34390.438335] ptp nvme_core i2c_algo_bit [34390.443123] __ceph_get_caps+0x4a1/0x760 [ceph] [34390.445043] pps_core dca [34390.449657] ? __file_remove_privs+0x89/0x110 [34390.453485] fuse [34390.458014] ? wake_bit_function+0x70/0x70 [34390.460634] irq event stamp: 196044830 [34390.464990] ? ceph_write_iter+0x260/0x770 [ceph] [34390.466926] hardirqs last enabled at (196044829): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34390.471017] ceph_write_iter+0x260/0x770 [ceph] [34390.474762] hardirqs last disabled at (196044830): [] sysvec_apic_timer_interrupt+0xe/0x90 [34390.479463] ? lock_is_held_type+0xaa/0x120 [34390.489704] softirqs last enabled at (196044828): [] __do_softirq+0x2be/0x371 [34390.494231] vfs_write+0x324/0x4f0 [34390.504039] softirqs last disabled at (196044823): [] irq_exit_rcu+0xb0/0xd0 [34390.508221] ksys_pwrite64+0x65/0xa0 [34390.516987] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34390.520402] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34390.529011] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34390.532579] do_syscall_64+0x38/0x80 [34390.541957] Workqueue: events delayed_work [ceph] [34390.547086] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34390.554556] [34390.558129] RIP: 0033:0x7f48f0eeb93b [34390.562826] RIP: 0010:lock_acquire+0xf0/0x2b0 [34390.567868] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34390.569361] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34390.572932] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34390.577295] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34390.596037] ORIG_RAX: 0000000000000012 [34390.614774] [34390.614775] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34390.619992] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34390.625227] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34390.629056] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34390.630548] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34390.637672] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34390.644797] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34390.651920] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34390.659045] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34390.666168] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34390.673308] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34390.680439] [34390.687558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34390.717805] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34390.724931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34390.732054] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34390.739178] Call Trace: [34390.741622] [34390.743635] ? watchdog_timer_fn+0x1b4/0x220 [34390.747899] ? __hrtimer_run_queues+0x1d3/0x360 [34390.752424] ? hrtimer_interrupt+0x104/0x240 [34390.756693] ? __local_bh_enable+0x50/0x60 [34390.760788] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34390.766018] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34390.770976] [34390.773073] [34390.775172] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34390.780480] ? lock_acquire+0xf0/0x2b0 [34390.784229] ? igrab+0x43/0x50 [34390.787295] ? lock_release+0xc5/0x260 [34390.791043] _raw_spin_lock+0x30/0x40 [34390.794708] ? igrab+0x1d/0x50 [34390.797758] igrab+0x1d/0x50 [34390.800637] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34390.805786] delayed_work+0x1d8/0x280 [ceph] [34390.810067] ? process_scheduled_works+0x22a/0x510 [34390.814857] process_scheduled_works+0x2a4/0x510 [34390.819471] worker_thread+0x188/0x340 [34390.823221] ? rescuer_thread+0x310/0x310 [34390.827226] kthread+0xf5/0x130 [34390.830364] ? kthread_complete_and_exit+0x20/0x20 [34390.835146] ret_from_fork+0x31/0x40 [34390.838717] ? kthread_complete_and_exit+0x20/0x20 [34390.843500] ret_from_fork_asm+0x11/0x20 [34390.847423] [34418.062067] watchdog: BUG: soft lockup - CPU#0 stuck for 20940s! [dbench:78683] [34418.069375] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34418.074067] watchdog: BUG: soft lockup - CPU#7 stuck for 29937s! [kworker/7:4:78808] [34418.124824] irq event stamp: 59555572 [34418.132555] Modules linked in: ceph libceph dns_resolver fscache netfs [34418.136212] hardirqs last enabled at (59555571): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34418.142750] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [34418.152905] hardirqs last disabled at (59555572): [] sysvec_apic_timer_interrupt+0xe/0x90 [34418.158989] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [34418.168711] softirqs last enabled at (59555544): [] __do_softirq+0x2be/0x371 [34418.176097] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [34418.184782] softirqs last disabled at (59555539): [] irq_exit_rcu+0xb0/0xd0 [34418.191653] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [34418.200166] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34418.200167] irqbypass [34418.200168] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34418.206162] i2c_i801 iTCO_wdt [34418.215107] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34418.217464] iTCO_vendor_support [34418.224935] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34418.227985] i2c_smbus crct10dif_pclmul [34418.233462] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34418.236686] joydev crc32_pclmul [34418.255423] [34418.255424] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34418.259254] ipmi_ssif ghash_clmulni_intel [34418.264471] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34418.267695] mei_me mei [34418.269187] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34418.276310] lpc_ich sha512_ssse3 [34418.280402] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34418.287526] acpi_ipmi mfd_core [34418.289970] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34418.297095] wmi [34418.300404] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34418.307529] ipmi_si [34418.310667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34418.317790] ipmi_devintf [34418.319628] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34418.327706] ipmi_msghandler ioatdma [34418.329889] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34418.335626] acpi_pad acpi_power_meter [34418.338245] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34418.345368] ixgbe igb [34418.348939] Call Trace: [34418.348941] [34418.356063] mdio crc32c_intel [34418.359810] ? watchdog_timer_fn+0x1b4/0x220 [34418.366931] nvme ptp [34418.369291] ? __hrtimer_run_queues+0x1d3/0x360 [34418.371743] nvme_core i2c_algo_bit [34418.373764] ? hrtimer_interrupt+0x104/0x240 [34418.376811] pps_core dca [34418.381077] ? __local_bh_enable+0x50/0x60 [34418.383345] fuse [34418.387874] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34418.391354] irq event stamp: 196265566 [34418.395619] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34418.398236] hardirqs last enabled at (196265565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34418.402326] [34418.404250] hardirqs last disabled at (196265566): [] sysvec_apic_timer_interrupt+0xe/0x90 [34418.409468] [34418.413213] softirqs last enabled at (196265564): [] __do_softirq+0x2be/0x371 [34418.418169] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34418.428415] softirqs last disabled at (196265559): [] irq_exit_rcu+0xb0/0xd0 [34418.430515] ? queued_spin_lock_slowpath+0x18/0x2e0 [34418.440323] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34418.442422] do_raw_spin_lock+0xb5/0xc0 [34418.451190] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34418.456497] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34418.465092] Workqueue: events delayed_work [ceph] [34418.469967] try_get_cap_refs+0x159/0x880 [ceph] [34418.479341] [34418.479342] RIP: 0010:lock_acquire+0xf0/0x2b0 [34418.483176] __ceph_get_caps+0x4a1/0x760 [ceph] [34418.490641] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34418.495430] ? __file_remove_privs+0x89/0x110 [34418.500123] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34418.504735] ? wake_bit_function+0x70/0x70 [34418.506242] [34418.506243] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34418.510594] ? ceph_write_iter+0x260/0x770 [ceph] [34418.515117] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34418.533856] ceph_write_iter+0x260/0x770 [ceph] [34418.538206] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34418.543425] ? lock_is_held_type+0xaa/0x120 [34418.547514] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34418.549008] vfs_write+0x324/0x4f0 [34418.556129] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34418.560832] ksys_pwrite64+0x65/0xa0 [34418.567949] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34418.572475] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34418.579597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34418.583775] do_syscall_64+0x38/0x80 [34418.590900] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34418.594299] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34418.601421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34418.604993] RIP: 0033:0x7f48f0eeb93b [34418.613069] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34418.618202] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34418.623937] Call Trace: [34418.623938] [34418.627508] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34418.634632] ? watchdog_timer_fn+0x1b4/0x220 [34418.639676] ORIG_RAX: 0000000000000012 [34418.646801] ? __hrtimer_run_queues+0x1d3/0x360 [34418.650372] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34418.657498] ? hrtimer_interrupt+0x104/0x240 [34418.676234] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34418.678678] ? __local_bh_enable+0x50/0x60 [34418.680687] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34418.685908] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34418.690169] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34418.694000] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34418.698523] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34418.705648] [34418.709917] [34418.717036] [34418.759054] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34418.764361] ? lock_acquire+0xf0/0x2b0 [34418.768112] ? igrab+0x43/0x50 [34418.771162] ? lock_release+0xc5/0x260 [34418.774907] _raw_spin_lock+0x30/0x40 [34418.778563] ? igrab+0x1d/0x50 [34418.781614] igrab+0x1d/0x50 [34418.784494] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34418.789642] delayed_work+0x1d8/0x280 [ceph] [34418.793923] ? process_scheduled_works+0x22a/0x510 [34418.798714] process_scheduled_works+0x2a4/0x510 [34418.803330] worker_thread+0x188/0x340 [34418.807077] ? rescuer_thread+0x310/0x310 [34418.811080] kthread+0xf5/0x130 [34418.814219] ? kthread_complete_and_exit+0x20/0x20 [34418.819002] ret_from_fork+0x31/0x40 [34418.822581] ? kthread_complete_and_exit+0x20/0x20 [34418.827366] ret_from_fork_asm+0x11/0x20 [34418.831298] [34446.062516] watchdog: BUG: soft lockup - CPU#0 stuck for 20966s! [dbench:78683] [34446.069819] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34446.074516] watchdog: BUG: soft lockup - CPU#7 stuck for 29963s! [kworker/7:4:78808] [34446.125260] irq event stamp: 59613608 [34446.125262] hardirqs last enabled at (59613607): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34446.132989] Modules linked in: [34446.136648] hardirqs last disabled at (59613608): [] sysvec_apic_timer_interrupt+0xe/0x90 [34446.146805] ceph [34446.149856] softirqs last enabled at (59613580): [] __do_softirq+0x2be/0x371 [34446.159580] libceph [34446.161503] softirqs last disabled at (59613575): [] irq_exit_rcu+0xb0/0xd0 [34446.170192] dns_resolver [34446.172372] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34446.180884] fscache [34446.183518] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34446.192462] netfs [34446.194647] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34446.202115] veth [34446.204128] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34446.209605] nft_chain_nat xt_MASQUERADE [34446.211530] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34446.230267] nf_nat nf_conntrack [34446.234183] [34446.234183] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34446.239400] nf_defrag_ipv6 nf_defrag_ipv4 [34446.242626] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34446.244117] nft_compat nf_tables [34446.251240] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34446.255332] nfnetlink bridge [34446.262454] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34446.265766] stp llc [34446.272907] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34446.275871] xfs libcrc32c [34446.282995] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34446.285192] sunrpc intel_rapl_msr [34446.292320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34446.295025] intel_rapl_common sb_edac [34446.303103] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34446.306500] x86_pkg_temp_thermal coretemp [34446.312237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34446.315981] kvm_intel [34446.323105] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34446.327197] kvm irqbypass [34446.334320] Call Trace: [34446.334322] [34446.336678] i2c_i801 iTCO_wdt [34446.343804] ? watchdog_timer_fn+0x1b4/0x220 [34446.346507] iTCO_vendor_support [34446.348951] ? __hrtimer_run_queues+0x1d3/0x360 [34446.350961] i2c_smbus crct10dif_pclmul [34446.354014] ? hrtimer_interrupt+0x104/0x240 [34446.358276] joydev crc32_pclmul [34446.361500] ? __local_bh_enable+0x50/0x60 [34446.366024] ipmi_ssif [34446.369857] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34446.374119] ghash_clmulni_intel [34446.377344] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34446.381433] mei_me [34446.383791] [34446.389009] mei lpc_ich [34446.392232] [34446.397194] sha512_ssse3 acpi_ipmi [34446.399295] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34446.401394] mfd_core wmi [34446.403928] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34446.406020] ipmi_si ipmi_devintf [34446.409506] do_raw_spin_lock+0xb5/0xc0 [34446.414809] ipmi_msghandler ioatdma acpi_pad [34446.417429] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34446.422297] acpi_power_meter ixgbe [34446.425611] try_get_cap_refs+0x159/0x880 [ceph] [34446.429438] igb mdio [34446.433794] __ceph_get_caps+0x4a1/0x760 [ceph] [34446.438573] crc32c_intel nvme [34446.442063] ? __file_remove_privs+0x89/0x110 [34446.446668] ptp nvme_core [34446.448939] ? wake_bit_function+0x70/0x70 [34446.453462] i2c_algo_bit pps_core [34446.456517] ? ceph_write_iter+0x260/0x770 [ceph] [34446.460866] dca fuse [34446.463570] ceph_write_iter+0x260/0x770 [ceph] [34446.467660] [34446.471061] ? lock_is_held_type+0xaa/0x120 [34446.475754] irq event stamp: 196486426 [34446.478027] vfs_write+0x324/0x4f0 [34446.482549] hardirqs last enabled at (196486425): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34446.484044] ksys_pwrite64+0x65/0xa0 [34446.488216] hardirqs last disabled at (196486426): [] sysvec_apic_timer_interrupt+0xe/0x90 [34446.491961] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34446.495359] softirqs last enabled at (196486424): [] __do_softirq+0x2be/0x371 [34446.505605] do_syscall_64+0x38/0x80 [34446.509174] softirqs last disabled at (196486419): [] irq_exit_rcu+0xb0/0xd0 [34446.518986] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34446.524116] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34446.532886] RIP: 0033:0x7f48f0eeb93b [34446.536458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34446.545055] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34446.550097] Workqueue: events delayed_work [ceph] [34446.559477] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34446.563046] [34446.563047] RIP: 0010:lock_acquire+0xf0/0x2b0 [34446.570517] ORIG_RAX: 0000000000000012 [34446.589256] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34446.593952] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34446.599170] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34446.600661] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34446.605011] [34446.608842] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34446.627580] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34446.634705] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34446.639921] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34446.647045] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34446.648536] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34446.655665] [34446.662785] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34446.700597] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34446.707723] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34446.715800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34446.721538] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34446.728661] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34446.735785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34446.742918] Call Trace: [34446.745364] [34446.747375] ? watchdog_timer_fn+0x1b4/0x220 [34446.751638] ? __hrtimer_run_queues+0x1d3/0x360 [34446.756166] ? hrtimer_interrupt+0x104/0x240 [34446.760435] ? __local_bh_enable+0x50/0x60 [34446.764528] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34446.769751] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34446.774709] [34446.776805] [34446.778920] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34446.784228] ? lock_acquire+0xf0/0x2b0 [34446.787979] ? igrab+0x43/0x50 [34446.791029] ? lock_release+0xc5/0x260 [34446.794773] _raw_spin_lock+0x30/0x40 [34446.798430] ? igrab+0x1d/0x50 [34446.801481] igrab+0x1d/0x50 [34446.804359] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34446.809509] delayed_work+0x1d8/0x280 [ceph] [34446.813790] ? process_scheduled_works+0x22a/0x510 [34446.818578] process_scheduled_works+0x2a4/0x510 [34446.823207] worker_thread+0x188/0x340 [34446.826953] ? rescuer_thread+0x310/0x310 [34446.830957] kthread+0xf5/0x130 [34446.834095] ? kthread_complete_and_exit+0x20/0x20 [34446.838878] ret_from_fork+0x31/0x40 [34446.842448] ? kthread_complete_and_exit+0x20/0x20 [34446.847232] ret_from_fork_asm+0x11/0x20 [34446.851157] [34474.062966] watchdog: BUG: soft lockup - CPU#0 stuck for 20992s! [dbench:78683] [34474.070273] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34474.074965] watchdog: BUG: soft lockup - CPU#7 stuck for 29989s! [kworker/7:4:78808] [34474.125721] irq event stamp: 59671788 [34474.133452] Modules linked in: ceph libceph dns_resolver fscache netfs [34474.137109] hardirqs last enabled at (59671787): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34474.143638] veth nft_chain_nat xt_MASQUERADE [34474.153792] hardirqs last disabled at (59671788): [] sysvec_apic_timer_interrupt+0xe/0x90 [34474.153794] nf_nat [34474.153795] softirqs last enabled at (59671760): [] __do_softirq+0x2be/0x371 [34474.158144] nf_conntrack nf_defrag_ipv6 [34474.167869] softirqs last disabled at (59671755): [] irq_exit_rcu+0xb0/0xd0 [34474.169965] nf_defrag_ipv4 [34474.178651] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34474.182566] nft_compat nf_tables [34474.191078] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34474.193868] nfnetlink [34474.202812] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34474.206122] bridge stp [34474.213594] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34474.215952] llc xfs [34474.221430] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34474.223873] libcrc32c sunrpc [34474.242612] [34474.242612] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34474.244795] intel_rapl_msr [34474.250013] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34474.252976] intel_rapl_common [34474.254465] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34474.261591] sb_edac x86_pkg_temp_thermal [34474.264382] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34474.271506] coretemp [34474.274557] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34474.281682] kvm_intel [34474.285684] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34474.292809] kvm [34474.295080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34474.302204] irqbypass [34474.304561] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34474.312643] i2c_i801 iTCO_wdt [34474.314484] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34474.320222] iTCO_vendor_support [34474.322580] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34474.329704] i2c_smbus [34474.332754] Call Trace: [34474.339879] crct10dif_pclmul joydev [34474.343106] [34474.350227] crc32_pclmul ipmi_ssif [34474.352587] ? watchdog_timer_fn+0x1b4/0x220 [34474.355027] ghash_clmulni_intel mei_me [34474.358600] ? __hrtimer_run_queues+0x1d3/0x360 [34474.360609] mei lpc_ich [34474.364097] ? hrtimer_interrupt+0x104/0x240 [34474.368365] sha512_ssse3 [34474.372197] ? __local_bh_enable+0x50/0x60 [34474.376720] acpi_ipmi mfd_core [34474.379255] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34474.383515] wmi ipmi_si [34474.386133] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34474.390224] ipmi_devintf ipmi_msghandler [34474.393363] [34474.398579] ioatdma [34474.401109] [34474.406068] acpi_pad acpi_power_meter [34474.410072] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34474.412168] ixgbe [34474.414368] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34474.416458] igb mdio [34474.420203] do_raw_spin_lock+0xb5/0xc0 [34474.425507] crc32c_intel nvme [34474.427519] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34474.432387] ptp nvme_core [34474.434661] try_get_cap_refs+0x159/0x880 [ceph] [34474.438489] i2c_algo_bit pps_core [34474.441545] __ceph_get_caps+0x4a1/0x760 [ceph] [34474.446324] dca fuse [34474.449031] ? __file_remove_privs+0x89/0x110 [34474.453644] [34474.453645] irq event stamp: 196707062 [34474.457046] ? wake_bit_function+0x70/0x70 [34474.461569] hardirqs last enabled at (196707061): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34474.463842] ? ceph_write_iter+0x260/0x770 [ceph] [34474.468190] hardirqs last disabled at (196707062): [] sysvec_apic_timer_interrupt+0xe/0x90 [34474.469682] ceph_write_iter+0x260/0x770 [ceph] [34474.473424] softirqs last enabled at (196707060): [] __do_softirq+0x2be/0x371 [34474.477521] ? lock_is_held_type+0xaa/0x120 [34474.487760] softirqs last disabled at (196707055): [] irq_exit_rcu+0xb0/0xd0 [34474.492461] vfs_write+0x324/0x4f0 [34474.502269] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34474.506797] ksys_pwrite64+0x65/0xa0 [34474.515564] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34474.519742] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34474.528338] Workqueue: events delayed_work [ceph] [34474.531755] do_syscall_64+0x38/0x80 [34474.541130] [34474.544703] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34474.552172] RIP: 0010:lock_acquire+0xf0/0x2b0 [34474.557305] RIP: 0033:0x7f48f0eeb93b [34474.562001] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34474.565572] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34474.567060] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34474.572107] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34474.576455] [34474.576456] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34474.580028] ORIG_RAX: 0000000000000012 [34474.598766] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34474.617502] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34474.622720] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34474.627937] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34474.629427] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34474.636551] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34474.640383] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34474.647507] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34474.654633] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34474.661756] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34474.668880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34474.676008] [34474.683127] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34474.727623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34474.734765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34474.741888] Call Trace: [34474.744334] [34474.746345] ? watchdog_timer_fn+0x1b4/0x220 [34474.750628] ? __hrtimer_run_queues+0x1d3/0x360 [34474.755172] ? hrtimer_interrupt+0x104/0x240 [34474.759439] ? __local_bh_enable+0x50/0x60 [34474.763533] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34474.768757] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34474.773715] [34474.775810] [34474.777909] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34474.783217] ? lock_acquire+0xf0/0x2b0 [34474.786967] ? igrab+0x43/0x50 [34474.790017] ? lock_release+0xc5/0x260 [34474.793763] _raw_spin_lock+0x30/0x40 [34474.797427] ? igrab+0x1d/0x50 [34474.800477] igrab+0x1d/0x50 [34474.803372] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34474.808523] delayed_work+0x1d8/0x280 [ceph] [34474.812805] ? process_scheduled_works+0x22a/0x510 [34474.817593] process_scheduled_works+0x2a4/0x510 [34474.822209] worker_thread+0x188/0x340 [34474.825960] ? rescuer_thread+0x310/0x310 [34474.829962] kthread+0xf5/0x130 [34474.833101] ? kthread_complete_and_exit+0x20/0x20 [34474.837883] ret_from_fork+0x31/0x40 [34474.841463] ? kthread_complete_and_exit+0x20/0x20 [34474.846247] ret_from_fork_asm+0x11/0x20 [34474.850171] [34475.042346] rcu: INFO: rcu_preempt self-detected stall on CPU [34475.048108] rcu: 0-....: (22028870 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11028127 [34475.058620] rcu: (t=22538033 jiffies g=1309609 q=634582 ncpus=8) [34475.064705] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34475.073646] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34475.081136] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34475.086611] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34475.105349] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34475.110585] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34475.117710] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34475.124832] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34475.131958] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34475.139081] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34475.146205] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34475.154282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34475.160020] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34475.167144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34475.174269] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34475.181393] Call Trace: [34475.183838] [34475.185847] ? rcu_dump_cpu_stacks+0xd1/0x180 [34475.190199] ? rcu_sched_clock_irq+0x633/0x1390 [34475.194723] ? lock_is_held_type+0xaa/0x120 [34475.198902] ? lock_is_held_type+0xaa/0x120 [34475.203079] ? tick_sched_do_timer+0xa0/0xa0 [34475.207350] ? rcu_read_lock_sched_held+0x56/0x90 [34475.212066] ? tick_sched_do_timer+0xa0/0xa0 [34475.216329] ? update_process_times+0x5f/0x90 [34475.220680] ? tick_sched_handle.isra.16+0x1e/0x50 [34475.225463] ? tick_sched_timer+0x6f/0x80 [34475.229469] ? __hrtimer_run_queues+0x1d3/0x360 [34475.233996] ? hrtimer_interrupt+0x104/0x240 [34475.238264] ? __local_bh_enable+0x50/0x60 [34475.242376] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34475.247599] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34475.252556] [34475.254663] [34475.256761] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34475.262070] ? queued_spin_lock_slowpath+0x16/0x2e0 [34475.266944] do_raw_spin_lock+0xb5/0xc0 [34475.270775] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34475.275578] try_get_cap_refs+0x159/0x880 [ceph] [34475.280207] __ceph_get_caps+0x4a1/0x760 [ceph] [34475.284750] ? __file_remove_privs+0x89/0x110 [34475.289105] ? wake_bit_function+0x70/0x70 [34475.293197] ? ceph_write_iter+0x260/0x770 [ceph] [34475.297909] ceph_write_iter+0x260/0x770 [ceph] [34475.302446] ? lock_is_held_type+0xaa/0x120 [34475.306630] vfs_write+0x324/0x4f0 [34475.310031] ksys_pwrite64+0x65/0xa0 [34475.313604] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34475.318737] do_syscall_64+0x38/0x80 [34475.322307] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34475.327352] RIP: 0033:0x7f48f0eeb93b [34475.330938] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34475.349676] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [34475.357234] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34475.364376] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34475.371500] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34475.378624] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34475.385748] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34475.392877] [34475.395066] Sending NMI from CPU 0 to CPUs 7: [34475.399416] NMI backtrace for cpu 7 [34475.399418] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34475.399420] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34475.399421] Workqueue: events delayed_work [ceph] [34475.399441] RIP: 0010:hlock_class+0x15/0x60 [34475.399445] Code: 89 c6 e8 2e 65 01 00 eb 9c e8 d7 ed a2 00 0f 1f 80 00 00 00 00 0f b7 47 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 6d c0 e0 01 73 13 <48> 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 c3 8b 15 0a [34475.399447] RSP: 0018:ffffc90002117c58 EFLAGS: 00000007 [34475.399449] RAX: 00000000000006f5 RBX: ffff88810debaa80 RCX: 0000000000000003 [34475.399450] RDX: 0000000000001506 RSI: 0000000000000000 RDI: ffff88810debb550 [34475.399451] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000003 [34475.399453] R10: 0000000000000000 R11: 0000000000000001 R12: b656a521d3820f2c [34475.399454] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [34475.399455] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34475.399457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34475.399458] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34475.399460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34475.399461] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34475.399462] Call Trace: [34475.399463] [34475.399464] ? nmi_cpu_backtrace+0xad/0x120 [34475.399469] ? nmi_cpu_backtrace_handler+0x11/0x20 [34475.399471] ? nmi_handle+0xd4/0x260 [34475.399477] ? default_do_nmi+0x49/0xf0 [34475.399479] ? exc_nmi+0x11e/0x150 [34475.399482] ? end_repeat_nmi+0x16/0x67 [34475.399488] ? hlock_class+0x15/0x60 [34475.399491] ? hlock_class+0x15/0x60 [34475.399495] ? hlock_class+0x15/0x60 [34475.399498] [34475.399499] [34475.399500] __lock_acquire+0x611/0x1e20 [34475.399505] lock_acquire+0xcb/0x2b0 [34475.399507] ? igrab+0x1d/0x50 [34475.399509] ? igrab+0x43/0x50 [34475.399511] ? lock_release+0xc5/0x260 [34475.399515] _raw_spin_lock+0x30/0x40 [34475.399517] ? igrab+0x1d/0x50 [34475.399519] igrab+0x1d/0x50 [34475.399521] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34475.399541] delayed_work+0x1d8/0x280 [ceph] [34475.399561] ? process_scheduled_works+0x22a/0x510 [34475.399564] process_scheduled_works+0x2a4/0x510 [34475.399570] worker_thread+0x188/0x340 [34475.399574] ? rescuer_thread+0x310/0x310 [34475.399576] kthread+0xf5/0x130 [34475.399579] ? kthread_complete_and_exit+0x20/0x20 [34475.399583] ret_from_fork+0x31/0x40 [34475.399586] ? kthread_complete_and_exit+0x20/0x20 [34475.399589] ret_from_fork_asm+0x11/0x20 [34475.399595] [34502.063404] watchdog: BUG: soft lockup - CPU#0 stuck for 21018s! [dbench:78683] [34502.070706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34502.075404] watchdog: BUG: soft lockup - CPU#7 stuck for 30015s! [kworker/7:4:78808] [34502.126147] irq event stamp: 59729150 [34502.126148] hardirqs last enabled at (59729149): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34502.133879] Modules linked in: [34502.137536] hardirqs last disabled at (59729150): [] sysvec_apic_timer_interrupt+0xe/0x90 [34502.147694] ceph [34502.150743] softirqs last enabled at (59729134): [] __do_softirq+0x2be/0x371 [34502.160470] libceph [34502.162392] softirqs last disabled at (59729129): [] irq_exit_rcu+0xb0/0xd0 [34502.171079] dns_resolver [34502.173261] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34502.181772] fscache netfs [34502.184389] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34502.193332] veth nft_chain_nat [34502.196036] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34502.203509] xt_MASQUERADE [34502.206646] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34502.212123] nf_nat [34502.214828] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34502.233565] nf_conntrack nf_defrag_ipv6 [34502.235662] [34502.235663] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34502.240879] nf_defrag_ipv4 [34502.244800] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34502.244802] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34502.246295] nft_compat nf_tables [34502.253420] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34502.256212] nfnetlink [34502.263336] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34502.270458] bridge stp [34502.273770] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34502.280894] llc xfs [34502.283252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34502.290376] libcrc32c sunrpc [34502.292820] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34502.300897] intel_rapl_msr intel_rapl_common [34502.303082] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34502.308818] sb_edac x86_pkg_temp_thermal [34502.311800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34502.318923] coretemp kvm_intel [34502.323275] Call Trace: [34502.323277] [34502.330399] kvm irqbypass [34502.334405] ? watchdog_timer_fn+0x1b4/0x220 [34502.341528] i2c_i801 iTCO_wdt [34502.344666] ? __hrtimer_run_queues+0x1d3/0x360 [34502.347108] iTCO_vendor_support i2c_smbus [34502.349122] ? hrtimer_interrupt+0x104/0x240 [34502.351823] crct10dif_pclmul joydev [34502.356087] ? __local_bh_enable+0x50/0x60 [34502.359139] crc32_pclmul ipmi_ssif [34502.363666] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34502.367752] ghash_clmulni_intel mei_me [34502.372018] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34502.375588] mei lpc_ich [34502.379679] [34502.383164] sha512_ssse3 acpi_ipmi [34502.388381] [34502.388382] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34502.392212] mfd_core wmi [34502.397173] ? queued_spin_lock_slowpath+0x16/0x2e0 [34502.399699] ipmi_si ipmi_devintf [34502.401805] do_raw_spin_lock+0xb5/0xc0 [34502.405288] ipmi_msghandler ioatdma [34502.407387] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34502.412692] acpi_pad acpi_power_meter [34502.415310] try_get_cap_refs+0x159/0x880 [ceph] [34502.420179] ixgbe igb [34502.423494] __ceph_get_caps+0x4a1/0x760 [ceph] [34502.427320] mdio crc32c_intel [34502.430896] ? __file_remove_privs+0x89/0x110 [34502.435674] nvme ptp [34502.439419] ? wake_bit_function+0x70/0x70 [34502.444029] nvme_core i2c_algo_bit [34502.446389] ? ceph_write_iter+0x260/0x770 [ceph] [34502.450910] pps_core [34502.453962] ceph_write_iter+0x260/0x770 [ceph] [34502.458313] dca [34502.460586] ? lock_is_held_type+0xaa/0x120 [34502.464675] fuse [34502.468162] vfs_write+0x324/0x4f0 [34502.472855] irq event stamp: 196925852 [34502.475131] ksys_pwrite64+0x65/0xa0 [34502.479651] hardirqs last enabled at (196925851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34502.481488] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34502.485666] hardirqs last disabled at (196925852): [] sysvec_apic_timer_interrupt+0xe/0x90 [34502.487590] do_syscall_64+0x38/0x80 [34502.490987] softirqs last enabled at (196925850): [] __do_softirq+0x2be/0x371 [34502.494732] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34502.498301] softirqs last disabled at (196925845): [] irq_exit_rcu+0xb0/0xd0 [34502.508545] RIP: 0033:0x7f48f0eeb93b [34502.513678] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34502.523487] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34502.527057] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34502.535829] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34502.540873] Workqueue: events delayed_work [ceph] [34502.549470] ORIG_RAX: 0000000000000012 [34502.553041] [34502.553042] RIP: 0010:lock_acquire+0xf0/0x2b0 [34502.562418] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34502.581156] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34502.588628] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34502.593845] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34502.598542] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34502.602373] [34502.602374] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34502.603863] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34502.608213] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34502.615337] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34502.634087] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34502.641215] [34502.646427] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34502.699971] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34502.707094] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34502.715171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34502.720910] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34502.728034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34502.735158] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34502.742282] Call Trace: [34502.744727] [34502.746737] ? watchdog_timer_fn+0x1b4/0x220 [34502.751000] ? __hrtimer_run_queues+0x1d3/0x360 [34502.755529] ? hrtimer_interrupt+0x104/0x240 [34502.759806] ? __local_bh_enable+0x50/0x60 [34502.763907] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34502.769131] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34502.774089] [34502.776185] [34502.778283] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34502.783593] ? lock_acquire+0xf0/0x2b0 [34502.787343] ? igrab+0x43/0x50 [34502.790393] ? lock_release+0xc5/0x260 [34502.794138] _raw_spin_lock+0x30/0x40 [34502.797807] ? igrab+0x1d/0x50 [34502.800860] igrab+0x1d/0x50 [34502.803740] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34502.808888] delayed_work+0x1d8/0x280 [ceph] [34502.813170] ? process_scheduled_works+0x22a/0x510 [34502.817960] process_scheduled_works+0x2a4/0x510 [34502.822576] worker_thread+0x188/0x340 [34502.826324] ? rescuer_thread+0x310/0x310 [34502.830327] kthread+0xf5/0x130 [34502.833466] ? kthread_complete_and_exit+0x20/0x20 [34502.838251] ret_from_fork+0x31/0x40 [34502.841828] ? kthread_complete_and_exit+0x20/0x20 [34502.846614] ret_from_fork_asm+0x11/0x20 [34502.850537] [34530.063843] watchdog: BUG: soft lockup - CPU#0 stuck for 21044s! [dbench:78683] [34530.071143] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34530.075843] watchdog: BUG: soft lockup - CPU#7 stuck for 30041s! [kworker/7:4:78808] [34530.126584] irq event stamp: 59787404 [34530.126586] hardirqs last enabled at (59787403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34530.134314] Modules linked in: [34530.137973] hardirqs last disabled at (59787404): [] sysvec_apic_timer_interrupt+0xe/0x90 [34530.148130] ceph [34530.151180] softirqs last enabled at (59787398): [] __do_softirq+0x2be/0x371 [34530.160905] libceph [34530.162828] softirqs last disabled at (59787393): [] irq_exit_rcu+0xb0/0xd0 [34530.171519] dns_resolver fscache [34530.173706] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34530.182232] netfs veth [34530.185545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34530.194487] nft_chain_nat [34530.196932] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34530.204404] xt_MASQUERADE [34530.207108] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34530.212584] nf_nat nf_conntrack [34530.215289] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34530.234026] nf_defrag_ipv6 nf_defrag_ipv4 [34530.237250] [34530.237251] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34530.242469] nft_compat [34530.246557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34530.248049] nf_tables nfnetlink [34530.255172] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34530.257618] bridge stp [34530.264742] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34530.267965] llc xfs [34530.275090] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34530.277534] libcrc32c sunrpc [34530.284657] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34530.286841] intel_rapl_msr [34530.293967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34530.296931] intel_rapl_common [34530.305007] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34530.307798] sb_edac x86_pkg_temp_thermal [34530.313536] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34530.316587] coretemp kvm_intel [34530.323710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34530.327716] kvm [34530.334840] Call Trace: [34530.334841] [34530.337977] irqbypass [34530.345101] ? watchdog_timer_fn+0x1b4/0x220 [34530.346938] i2c_i801 iTCO_wdt [34530.349382] ? __hrtimer_run_queues+0x1d3/0x360 [34530.351393] iTCO_vendor_support [34530.353753] ? hrtimer_interrupt+0x104/0x240 [34530.358014] i2c_smbus crct10dif_pclmul [34530.361064] ? __local_bh_enable+0x50/0x60 [34530.365589] joydev crc32_pclmul [34530.368815] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34530.373077] ipmi_ssif [34530.376909] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34530.380998] ghash_clmulni_intel mei_me [34530.384239] [34530.389457] mei [34530.391814] [34530.391815] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34530.396771] lpc_ich sha512_ssse3 [34530.400606] ? queued_spin_lock_slowpath+0x16/0x2e0 [34530.402699] acpi_ipmi mfd_core [34530.404539] do_raw_spin_lock+0xb5/0xc0 [34530.406633] wmi ipmi_si [34530.411940] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34530.415250] ipmi_devintf ipmi_msghandler ioatdma [34530.420123] try_get_cap_refs+0x159/0x880 [ceph] [34530.423257] acpi_pad acpi_power_meter [34530.427092] __ceph_get_caps+0x4a1/0x760 [ceph] [34530.429619] ixgbe [34530.434408] ? __file_remove_privs+0x89/0x110 [34530.439102] igb mdio [34530.443711] ? wake_bit_function+0x70/0x70 [34530.447456] crc32c_intel nvme [34530.451984] ? ceph_write_iter+0x260/0x770 [ceph] [34530.453991] ptp nvme_core [34530.458341] ceph_write_iter+0x260/0x770 [ceph] [34530.460610] i2c_algo_bit pps_core [34530.464705] ? lock_is_held_type+0xaa/0x120 [34530.467752] dca fuse [34530.472452] vfs_write+0x324/0x4f0 [34530.475155] [34530.475156] irq event stamp: 197146624 [34530.479682] ksys_pwrite64+0x65/0xa0 [34530.483077] hardirqs last enabled at (197146623): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34530.487253] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34530.489524] hardirqs last disabled at (197146624): [] sysvec_apic_timer_interrupt+0xe/0x90 [34530.492923] do_syscall_64+0x38/0x80 [34530.494413] softirqs last enabled at (197146622): [] __do_softirq+0x2be/0x371 [34530.498157] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34530.501726] softirqs last disabled at (197146617): [] irq_exit_rcu+0xb0/0xd0 [34530.511970] RIP: 0033:0x7f48f0eeb93b [34530.517103] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34530.526913] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34530.530485] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34530.539256] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34530.544298] Workqueue: events delayed_work [ceph] [34530.552896] ORIG_RAX: 0000000000000012 [34530.556467] [34530.556468] RIP: 0010:lock_release+0x5f/0x260 [34530.565845] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34530.584581] Code: 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 <65> 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 [34530.592054] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34530.597270] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [34530.601967] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34530.605797] [34530.605798] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [34530.607289] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34530.611640] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: 0000000000000001 [34530.618765] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34530.637502] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [34530.637520] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [34530.644649] [34530.649860] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34530.710547] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34530.718622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34530.724361] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34530.731486] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34530.738608] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34530.745733] Call Trace: [34530.748179] [34530.750190] ? watchdog_timer_fn+0x1b4/0x220 [34530.754452] ? __hrtimer_run_queues+0x1d3/0x360 [34530.758980] ? hrtimer_interrupt+0x104/0x240 [34530.763250] ? __local_bh_enable+0x50/0x60 [34530.767343] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34530.772567] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34530.777528] [34530.779630] [34530.781727] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34530.787031] ? igrab+0x43/0x50 [34530.790083] ? igrab+0x43/0x50 [34530.793132] ? lock_release+0x5f/0x260 [34530.796879] ? lock_release+0x1d7/0x260 [34530.800710] _raw_spin_unlock+0x1b/0x40 [34530.804547] igrab+0x43/0x50 [34530.807426] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34530.812576] delayed_work+0x1d8/0x280 [ceph] [34530.816855] ? process_scheduled_works+0x22a/0x510 [34530.821647] process_scheduled_works+0x2a4/0x510 [34530.826261] worker_thread+0x188/0x340 [34530.830013] ? rescuer_thread+0x310/0x310 [34530.834023] kthread+0xf5/0x130 [34530.837160] ? kthread_complete_and_exit+0x20/0x20 [34530.841946] ret_from_fork+0x31/0x40 [34530.845523] ? kthread_complete_and_exit+0x20/0x20 [34530.850324] ret_from_fork_asm+0x11/0x20 [34530.854257] [34558.064280] watchdog: BUG: soft lockup - CPU#0 stuck for 21070s! [dbench:78683] [34558.071588] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34558.076279] watchdog: BUG: soft lockup - CPU#7 stuck for 30067s! [kworker/7:4:78808] [34558.127038] irq event stamp: 59845616 [34558.134768] Modules linked in: ceph libceph dns_resolver fscache netfs [34558.138425] hardirqs last enabled at (59845615): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34558.138430] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [34558.155118] hardirqs last disabled at (59845616): [] sysvec_apic_timer_interrupt+0xe/0x90 [34558.155121] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [34558.172234] softirqs last enabled at (59845598): [] __do_softirq+0x2be/0x371 [34558.172238] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [34558.187705] softirqs last disabled at (59845593): [] irq_exit_rcu+0xb0/0xd0 [34558.197176] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [34558.205689] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34558.210819] iTCO_vendor_support [34558.219762] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34558.222986] i2c_smbus crct10dif_pclmul joydev crc32_pclmul [34558.230458] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34558.236023] ipmi_ssif ghash_clmulni_intel mei_me mei [34558.241501] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34558.246543] lpc_ich [34558.265282] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34558.267466] sha512_ssse3 acpi_ipmi mfd_core [34558.272683] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34558.272685] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34558.276954] wmi [34558.284079] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34558.291202] ipmi_si ipmi_devintf [34558.293041] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34558.300164] ipmi_msghandler [34558.303476] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34558.310600] ioatdma acpi_pad [34558.313477] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34558.320600] acpi_power_meter [34558.323565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34558.331643] ixgbe [34558.334606] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34558.340346] igb mdio [34558.342356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34558.349478] crc32c_intel [34558.351751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34558.358873] nvme [34558.361491] Call Trace: [34558.368616] ptp nvme_core [34558.370541] [34558.372984] i2c_algo_bit pps_core [34558.375690] ? watchdog_timer_fn+0x1b4/0x220 [34558.377699] dca fuse [34558.381098] ? __hrtimer_run_queues+0x1d3/0x360 [34558.385361] [34558.387635] ? hrtimer_interrupt+0x104/0x240 [34558.392155] irq event stamp: 197367330 [34558.393647] ? __local_bh_enable+0x50/0x60 [34558.397908] hardirqs last enabled at (197367329): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34558.401674] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34558.405760] hardirqs last disabled at (197367330): [] sysvec_apic_timer_interrupt+0xe/0x90 [34558.416006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34558.421223] softirqs last enabled at (197367328): [] __do_softirq+0x2be/0x371 [34558.431036] [34558.435992] softirqs last disabled at (197367323): [] irq_exit_rcu+0xb0/0xd0 [34558.444763] [34558.446859] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34558.455457] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34558.457554] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34558.466936] ? queued_spin_lock_slowpath+0x16/0x2e0 [34558.472236] Workqueue: events delayed_work [ceph] [34558.479710] do_raw_spin_lock+0xb5/0xc0 [34558.484577] [34558.484578] RIP: 0010:lock_acquire+0xf0/0x2b0 [34558.489278] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34558.493105] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34558.494599] try_get_cap_refs+0x159/0x880 [ceph] [34558.498960] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34558.503747] __ceph_get_caps+0x4a1/0x760 [ceph] [34558.522479] [34558.522480] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34558.527093] ? __file_remove_privs+0x89/0x110 [34558.532307] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34558.536830] ? wake_bit_function+0x70/0x70 [34558.538320] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34558.545447] ? ceph_write_iter+0x260/0x770 [ceph] [34558.549796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34558.556920] ceph_write_iter+0x260/0x770 [ceph] [34558.561009] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34558.568138] ? lock_is_held_type+0xaa/0x120 [34558.572832] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34558.579964] vfs_write+0x324/0x4f0 [34558.584490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34558.591619] ksys_pwrite64+0x65/0xa0 [34558.595791] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34558.603869] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34558.607267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34558.613004] do_syscall_64+0x38/0x80 [34558.616575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34558.623699] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34558.628828] Call Trace: [34558.628830] [34558.635954] RIP: 0033:0x7f48f0eeb93b [34558.639524] ? watchdog_timer_fn+0x1b4/0x220 [34558.646649] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34558.651692] ? __hrtimer_run_queues+0x1d3/0x360 [34558.654134] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34558.656150] ? hrtimer_interrupt+0x104/0x240 [34558.659717] ORIG_RAX: 0000000000000012 [34558.663980] ? __local_bh_enable+0x50/0x60 [34558.682719] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34558.687245] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34558.692460] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34558.696725] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34558.700556] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34558.704647] [34558.711770] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34558.716987] [34558.724111] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34558.729069] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34558.736199] [34558.738296] ? lock_acquire+0xf0/0x2b0 [34558.765870] ? igrab+0x43/0x50 [34558.768921] ? lock_release+0xc5/0x260 [34558.772682] _raw_spin_lock+0x30/0x40 [34558.776340] ? igrab+0x1d/0x50 [34558.779390] igrab+0x1d/0x50 [34558.782270] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34558.787418] delayed_work+0x1d8/0x280 [ceph] [34558.791698] ? process_scheduled_works+0x22a/0x510 [34558.796489] process_scheduled_works+0x2a4/0x510 [34558.801105] worker_thread+0x188/0x340 [34558.804853] ? rescuer_thread+0x310/0x310 [34558.808856] kthread+0xf5/0x130 [34558.812002] ? kthread_complete_and_exit+0x20/0x20 [34558.816787] ret_from_fork+0x31/0x40 [34558.820366] ? kthread_complete_and_exit+0x20/0x20 [34558.825150] ret_from_fork_asm+0x11/0x20 [34558.829073] [34586.064714] watchdog: BUG: soft lockup - CPU#0 stuck for 21096s! [dbench:78683] [34586.072014] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34586.076714] watchdog: BUG: soft lockup - CPU#7 stuck for 30093s! [kworker/7:4:78808] [34586.127455] irq event stamp: 59903638 [34586.127456] hardirqs last enabled at (59903637): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34586.135185] Modules linked in: [34586.138843] hardirqs last disabled at (59903638): [] sysvec_apic_timer_interrupt+0xe/0x90 [34586.149002] ceph [34586.152051] softirqs last enabled at (59903620): [] __do_softirq+0x2be/0x371 [34586.161775] libceph [34586.163701] softirqs last disabled at (59903615): [] irq_exit_rcu+0xb0/0xd0 [34586.172389] dns_resolver [34586.174577] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34586.183087] fscache [34586.185705] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34586.185707] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34586.194651] netfs [34586.196834] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34586.204304] veth nft_chain_nat [34586.209782] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34586.211791] xt_MASQUERADE nf_nat [34586.230530] [34586.233668] nf_conntrack nf_defrag_ipv6 [34586.238884] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34586.242195] nf_defrag_ipv4 [34586.243687] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34586.247603] nft_compat nf_tables [34586.254728] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34586.257519] nfnetlink [34586.264643] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34586.267952] bridge stp [34586.275076] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34586.277433] llc xfs [34586.284558] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34586.287003] libcrc32c sunrpc [34586.294127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34586.296311] intel_rapl_msr intel_rapl_common [34586.304390] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34586.307352] sb_edac x86_pkg_temp_thermal [34586.313089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34586.317459] coretemp kvm_intel [34586.324583] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34586.328586] kvm [34586.335710] Call Trace: [34586.335711] [34586.338848] irqbypass [34586.345971] ? watchdog_timer_fn+0x1b4/0x220 [34586.347809] i2c_i801 iTCO_wdt [34586.350253] ? __hrtimer_run_queues+0x1d3/0x360 [34586.352263] iTCO_vendor_support [34586.354625] ? hrtimer_interrupt+0x104/0x240 [34586.358886] i2c_smbus crct10dif_pclmul [34586.361937] ? __local_bh_enable+0x50/0x60 [34586.366460] joydev crc32_pclmul [34586.369688] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34586.373948] ipmi_ssif [34586.377780] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34586.381869] ghash_clmulni_intel [34586.385110] [34586.390329] mei_me [34586.392685] [34586.392686] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34586.397644] mei [34586.400871] ? queued_spin_lock_slowpath+0x18/0x2e0 [34586.402966] lpc_ich sha512_ssse3 [34586.405063] do_raw_spin_lock+0xb5/0xc0 [34586.407159] acpi_ipmi mfd_core [34586.412464] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34586.414301] wmi ipmi_si [34586.419173] try_get_cap_refs+0x159/0x880 [ceph] [34586.422482] ipmi_devintf ipmi_msghandler [34586.426318] __ceph_get_caps+0x4a1/0x760 [ceph] [34586.429451] ioatdma acpi_pad [34586.434238] ? __file_remove_privs+0x89/0x110 [34586.436765] acpi_power_meter ixgbe [34586.441376] ? wake_bit_function+0x70/0x70 [34586.445381] igb mdio [34586.449925] ? ceph_write_iter+0x260/0x770 [ceph] [34586.452885] crc32c_intel [34586.457238] ceph_write_iter+0x260/0x770 [ceph] [34586.460720] nvme [34586.464815] ? lock_is_held_type+0xaa/0x120 [34586.467083] ptp [34586.471781] vfs_write+0x324/0x4f0 [34586.474397] nvme_core i2c_algo_bit [34586.478925] ksys_pwrite64+0x65/0xa0 [34586.480845] pps_core dca [34586.485023] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34586.486860] fuse [34586.490259] do_syscall_64+0x38/0x80 [34586.493741] irq event stamp: 197588152 [34586.497312] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34586.499929] hardirqs last enabled at (197588151): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34586.505061] RIP: 0033:0x7f48f0eeb93b [34586.506984] hardirqs last disabled at (197588152): [] sysvec_apic_timer_interrupt+0xe/0x90 [34586.510556] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34586.514297] softirqs last enabled at (197588150): [] __do_softirq+0x2be/0x371 [34586.519344] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34586.529586] softirqs last disabled at (197588145): [] irq_exit_rcu+0xb0/0xd0 [34586.533159] ORIG_RAX: 0000000000000012 [34586.542968] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34586.561707] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34586.570478] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34586.575695] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34586.584293] Workqueue: events delayed_work [ceph] [34586.588123] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34586.597500] [34586.597501] RIP: 0010:lock_acquire+0xf0/0x2b0 [34586.604624] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34586.612112] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34586.619236] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34586.623935] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34586.631064] [34586.632549] [34586.632549] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34586.685920] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34586.693045] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34586.700167] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34586.707291] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34586.714416] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34586.722494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34586.728232] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34586.735354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34586.742479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34586.749604] Call Trace: [34586.752049] [34586.754060] ? watchdog_timer_fn+0x1b4/0x220 [34586.758323] ? __hrtimer_run_queues+0x1d3/0x360 [34586.762850] ? hrtimer_interrupt+0x104/0x240 [34586.767120] ? __local_bh_enable+0x50/0x60 [34586.771214] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34586.776438] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34586.781393] [34586.783507] [34586.785605] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34586.790913] ? lock_acquire+0xf0/0x2b0 [34586.794663] ? igrab+0x43/0x50 [34586.797714] ? lock_release+0xc5/0x260 [34586.801460] _raw_spin_lock+0x30/0x40 [34586.805123] ? igrab+0x1d/0x50 [34586.808175] igrab+0x1d/0x50 [34586.811053] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34586.816202] delayed_work+0x1d8/0x280 [ceph] [34586.820482] ? process_scheduled_works+0x22a/0x510 [34586.825264] process_scheduled_works+0x2a4/0x510 [34586.829880] worker_thread+0x188/0x340 [34586.833629] ? rescuer_thread+0x310/0x310 [34586.837633] kthread+0xf5/0x130 [34586.840771] ? kthread_complete_and_exit+0x20/0x20 [34586.845554] ret_from_fork+0x31/0x40 [34586.849125] ? kthread_complete_and_exit+0x20/0x20 [34586.853910] ret_from_fork_asm+0x11/0x20 [34586.857840] [34614.065150] watchdog: BUG: soft lockup - CPU#0 stuck for 21122s! [dbench:78683] [34614.072458] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34614.077150] watchdog: BUG: soft lockup - CPU#7 stuck for 30119s! [kworker/7:4:78808] [34614.127908] irq event stamp: 59961838 [34614.135638] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE [34614.139297] hardirqs last enabled at (59961837): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34614.148682] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [34614.158841] hardirqs last disabled at (59961838): [] sysvec_apic_timer_interrupt+0xe/0x90 [34614.166484] nfnetlink bridge stp llc xfs libcrc32c sunrpc [34614.176208] softirqs last enabled at (59961820): [] __do_softirq+0x2be/0x371 [34614.181685] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [34614.190369] softirqs last disabled at (59961815): [] irq_exit_rcu+0xb0/0xd0 [34614.198880] kvm irqbypass i2c_i801 [34614.207392] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34614.207393] iTCO_wdt [34614.207394] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34614.210875] iTCO_vendor_support i2c_smbus [34614.219827] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34614.222100] crct10dif_pclmul [34614.229570] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34614.233661] joydev crc32_pclmul [34614.239138] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34614.242102] ipmi_ssif ghash_clmulni_intel [34614.260839] [34614.264064] mei_me [34614.269281] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34614.273372] mei lpc_ich [34614.274864] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34614.276959] sha512_ssse3 [34614.284085] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34614.286615] acpi_ipmi mfd_core [34614.293739] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34614.296357] wmi ipmi_si [34614.303480] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34614.306617] ipmi_devintf [34614.313741] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34614.316274] ipmi_msghandler [34614.323397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34614.326014] ioatdma acpi_pad [34614.334091] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34614.336970] acpi_power_meter [34614.342706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34614.345671] ixgbe [34614.352794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34614.355759] igb mdio [34614.362883] Call Trace: [34614.362885] [34614.364893] crc32c_intel nvme [34614.372020] ? watchdog_timer_fn+0x1b4/0x220 [34614.374288] ptp nvme_core [34614.376733] ? __hrtimer_run_queues+0x1d3/0x360 [34614.378744] i2c_algo_bit [34614.381797] ? hrtimer_interrupt+0x104/0x240 [34614.386059] pps_core [34614.388761] ? __local_bh_enable+0x50/0x60 [34614.393287] dca fuse [34614.395907] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34614.400167] [34614.400168] irq event stamp: 197809018 [34614.402438] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34614.406546] hardirqs last enabled at (197809017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34614.408830] [34614.414042] hardirqs last disabled at (197809018): [] sysvec_apic_timer_interrupt+0xe/0x90 [34614.415547] [34614.419296] softirqs last enabled at (197809016): [] __do_softirq+0x2be/0x371 [34614.424255] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34614.434496] softirqs last disabled at (197809011): [] irq_exit_rcu+0xb0/0xd0 [34614.436600] ? queued_spin_lock_slowpath+0x16/0x2e0 [34614.446406] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34614.448505] do_raw_spin_lock+0xb5/0xc0 [34614.457273] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34614.462579] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34614.471176] Workqueue: events delayed_work [ceph] [34614.476048] try_get_cap_refs+0x159/0x880 [ceph] [34614.485424] [34614.489259] __ceph_get_caps+0x4a1/0x760 [ceph] [34614.496724] RIP: 0010:rcu_is_watching+0x0/0x40 [34614.501513] ? __file_remove_privs+0x89/0x110 [34614.506207] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [34614.510830] ? wake_bit_function+0x70/0x70 [34614.512317] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [34614.516855] ? ceph_write_iter+0x260/0x770 [ceph] [34614.521288] [34614.525638] ceph_write_iter+0x260/0x770 [ceph] [34614.544374] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [34614.548470] ? lock_is_held_type+0xaa/0x120 [34614.553682] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [34614.558384] vfs_write+0x324/0x4f0 [34614.559870] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [34614.564401] ksys_pwrite64+0x65/0xa0 [34614.571519] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [34614.575697] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34614.582829] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34614.586230] do_syscall_64+0x38/0x80 [34614.593352] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34614.596924] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34614.604045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34614.609178] RIP: 0033:0x7f48f0eeb93b [34614.616301] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34614.619873] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34614.627949] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34614.632992] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34614.638731] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34614.642302] ORIG_RAX: 0000000000000012 [34614.649426] Call Trace: [34614.649427] [34614.668164] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34614.675288] ? watchdog_timer_fn+0x1b4/0x220 [34614.680504] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34614.687632] ? __hrtimer_run_queues+0x1d3/0x360 [34614.691460] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34614.693908] ? hrtimer_interrupt+0x104/0x240 [34614.695914] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34614.703038] ? __local_bh_enable+0x50/0x60 [34614.707304] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34614.714429] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34614.718956] [34614.726075] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34614.761038] [34614.763135] [34614.765233] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34614.770555] ? igrab+0x43/0x50 [34614.773608] ? igrab+0x43/0x50 [34614.776663] ? rcutorture_get_gp_data+0x30/0x30 [34614.781188] lock_release+0x1d7/0x260 [34614.784849] _raw_spin_unlock+0x1b/0x40 [34614.788686] igrab+0x43/0x50 [34614.791563] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34614.796713] delayed_work+0x1d8/0x280 [ceph] [34614.800994] ? process_scheduled_works+0x22a/0x510 [34614.805785] process_scheduled_works+0x2a4/0x510 [34614.810400] worker_thread+0x188/0x340 [34614.814148] ? rescuer_thread+0x310/0x310 [34614.818152] kthread+0xf5/0x130 [34614.821290] ? kthread_complete_and_exit+0x20/0x20 [34614.826075] ret_from_fork+0x31/0x40 [34614.829655] ? kthread_complete_and_exit+0x20/0x20 [34614.834446] ret_from_fork_asm+0x11/0x20 [34614.838377] [34642.065586] watchdog: BUG: soft lockup - CPU#0 stuck for 21148s! [dbench:78683] [34642.072886] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34642.077586] watchdog: BUG: soft lockup - CPU#7 stuck for 30145s! [kworker/7:4:78808] [34642.128327] irq event stamp: 60019886 [34642.136058] Modules linked in: ceph [34642.139714] hardirqs last enabled at (60019885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34642.139716] libceph dns_resolver fscache [34642.153357] hardirqs last disabled at (60019886): [] sysvec_apic_timer_interrupt+0xe/0x90 [34642.157360] netfs veth [34642.167085] softirqs last enabled at (60019868): [] __do_softirq+0x2be/0x371 [34642.167087] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [34642.178211] softirqs last disabled at (60019863): [] irq_exit_rcu+0xb0/0xd0 [34642.178213] nf_defrag_ipv6 [34642.178215] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34642.183865] nf_defrag_ipv4 nft_compat [34642.192374] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34642.195165] nf_tables nfnetlink [34642.204108] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34642.207854] bridge [34642.215324] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34642.218547] stp llc [34642.224026] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34642.226121] xfs libcrc32c [34642.244860] [34642.247044] sunrpc intel_rapl_msr [34642.252277] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34642.254982] intel_rapl_common [34642.256474] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34642.259871] sb_edac x86_pkg_temp_thermal [34642.266995] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34642.270047] coretemp kvm_intel [34642.277171] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34642.281175] kvm irqbypass [34642.288299] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34642.291437] i2c_i801 iTCO_wdt [34642.298559] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34642.301264] iTCO_vendor_support i2c_smbus [34642.308388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34642.311438] crct10dif_pclmul [34642.319516] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34642.323608] joydev crc32_pclmul [34642.329344] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34642.332308] ipmi_ssif [34642.339434] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34642.342657] ghash_clmulni_intel mei_me [34642.349780] Call Trace: [34642.349781] [34642.352137] mei lpc_ich [34642.359263] ? watchdog_timer_fn+0x1b4/0x220 [34642.363092] sha512_ssse3 acpi_ipmi [34642.365538] ? __hrtimer_run_queues+0x1d3/0x360 [34642.367549] mfd_core wmi [34642.370081] ? hrtimer_interrupt+0x104/0x240 [34642.374342] ipmi_si ipmi_devintf [34642.377828] ? __local_bh_enable+0x50/0x60 [34642.382351] ipmi_msghandler [34642.384988] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34642.389265] ioatdma acpi_pad [34642.392578] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34642.396668] acpi_power_meter ixgbe [34642.399546] [34642.404763] igb mdio [34642.407728] [34642.407730] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34642.412684] crc32c_intel nvme [34642.416173] ? queued_spin_lock_slowpath+0x18/0x2e0 [34642.418267] ptp nvme_core [34642.420538] do_raw_spin_lock+0xb5/0xc0 [34642.422635] i2c_algo_bit pps_core [34642.427941] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34642.430990] dca fuse [34642.435864] try_get_cap_refs+0x159/0x880 [ceph] [34642.438565] [34642.438566] irq event stamp: 198029876 [34642.442400] __ceph_get_caps+0x4a1/0x760 [ceph] [34642.445793] hardirqs last enabled at (198029875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34642.450579] ? __file_remove_privs+0x89/0x110 [34642.452848] hardirqs last disabled at (198029876): [] sysvec_apic_timer_interrupt+0xe/0x90 [34642.457458] ? wake_bit_function+0x70/0x70 [34642.458948] softirqs last enabled at (198029874): [] __do_softirq+0x2be/0x371 [34642.462694] ? ceph_write_iter+0x260/0x770 [ceph] [34642.467216] softirqs last disabled at (198029869): [] irq_exit_rcu+0xb0/0xd0 [34642.477461] ceph_write_iter+0x260/0x770 [ceph] [34642.481811] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34642.491627] ? lock_is_held_type+0xaa/0x120 [34642.495713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34642.504487] vfs_write+0x324/0x4f0 [34642.509182] Workqueue: events delayed_work [ceph] [34642.517783] ksys_pwrite64+0x65/0xa0 [34642.522304] [34642.531680] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34642.535857] RIP: 0010:_raw_spin_unlock+0x2d/0x40 [34642.543331] do_syscall_64+0x38/0x80 [34642.546727] Code: fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b 0f 1f 44 00 00 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e [34642.551424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34642.554994] RSP: 0018:ffffc90002117d98 EFLAGS: 00000202 [34642.556486] RIP: 0033:0x7f48f0eeb93b [34642.561617] [34642.561617] RAX: ffffffffffffffff RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [34642.566227] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34642.569796] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [34642.588534] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34642.593580] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [34642.598797] ORIG_RAX: 0000000000000012 [34642.602366] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [34642.603857] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34642.610987] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34642.629728] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34642.636853] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34642.642070] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34642.649193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34642.653024] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34642.660149] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34642.667283] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34642.674407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34642.681534] [34642.689606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34642.740292] Call Trace: [34642.742736] [34642.744747] ? watchdog_timer_fn+0x1b4/0x220 [34642.749010] ? __hrtimer_run_queues+0x1d3/0x360 [34642.753537] ? hrtimer_interrupt+0x104/0x240 [34642.757808] ? __local_bh_enable+0x50/0x60 [34642.761901] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34642.767124] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34642.772081] [34642.774178] [34642.776276] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34642.781585] ? _raw_spin_unlock+0x2d/0x40 [34642.785593] igrab+0x43/0x50 [34642.788471] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34642.793622] delayed_work+0x1d8/0x280 [ceph] [34642.797901] ? process_scheduled_works+0x22a/0x510 [34642.802692] process_scheduled_works+0x2a4/0x510 [34642.807307] worker_thread+0x188/0x340 [34642.811058] ? rescuer_thread+0x310/0x310 [34642.815068] kthread+0xf5/0x130 [34642.818207] ? kthread_complete_and_exit+0x20/0x20 [34642.823000] ret_from_fork+0x31/0x40 [34642.826571] ? kthread_complete_and_exit+0x20/0x20 [34642.831355] ret_from_fork_asm+0x11/0x20 [34642.835279] [34670.066023] watchdog: BUG: soft lockup - CPU#0 stuck for 21175s! [dbench:78683] [34670.073331] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34670.078023] watchdog: BUG: soft lockup - CPU#7 stuck for 30172s! [kworker/7:4:78808] [34670.128779] irq event stamp: 60077950 [34670.136511] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [34670.140167] hardirqs last enabled at (60077949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34670.148339] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [34670.158498] hardirqs last disabled at (60077950): [] sysvec_apic_timer_interrupt+0xe/0x90 [34670.164234] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [34670.173958] softirqs last enabled at (60077932): [] __do_softirq+0x2be/0x371 [34670.173961] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [34670.188120] softirqs last disabled at (60077927): [] irq_exit_rcu+0xb0/0xd0 [34670.194030] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [34670.202544] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34670.209753] kvm irqbypass [34670.218707] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34670.218709] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34670.221419] i2c_i801 [34670.228891] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34670.234367] iTCO_wdt [34670.236637] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34670.255376] iTCO_vendor_support i2c_smbus [34670.257646] [34670.257647] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34670.262864] crct10dif_pclmul joydev [34670.266953] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34670.268445] crc32_pclmul [34670.275569] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34670.279140] ipmi_ssif ghash_clmulni_intel [34670.286265] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34670.288881] mei_me [34670.296005] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34670.300095] mei [34670.307220] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34670.309317] lpc_ich sha512_ssse3 [34670.316442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34670.318279] acpi_ipmi mfd_core [34670.326357] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34670.329668] wmi ipmi_si [34670.335421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34670.338559] ipmi_devintf [34670.345700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34670.348232] ipmi_msghandler [34670.355357] Call Trace: [34670.355360] [34670.357974] ioatdma acpi_pad [34670.365100] ? watchdog_timer_fn+0x1b4/0x220 [34670.367975] acpi_power_meter [34670.370423] ? __hrtimer_run_queues+0x1d3/0x360 [34670.372431] ixgbe igb [34670.375399] ? hrtimer_interrupt+0x104/0x240 [34670.379674] mdio crc32c_intel [34670.382657] ? __local_bh_enable+0x50/0x60 [34670.387181] nvme [34670.389540] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34670.393801] ptp nvme_core [34670.396854] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34670.400943] i2c_algo_bit pps_core [34670.402868] [34670.408086] dca fuse [34670.410790] [34670.412396] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [34670.412399] rcu: 0-....: (22219807 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11124072 [34670.412421] rcu: 7-....: (22073842 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/22072503 fqs=11124073 [34670.412425] rcu: (detected by 4, t=22733384 jiffies, g=1309609, q=638608 ncpus=8) [34670.412427] Sending NMI from CPU 4 to CPUs 0: [34670.412431] NMI backtrace for cpu 0 [34670.412432] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34670.412434] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34670.412434] RIP: 0010:vprintk_emit+0x1eb/0x340 [34670.412438] Code: 50 41 b8 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 <0f> b6 11 84 d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 [34670.412440] RSP: 0018:ffffc90000003d30 EFLAGS: 00000002 [34670.412441] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [34670.412443] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [34670.412444] RBP: 0000000000000007 R08: 0000000000005ef4 R09: 0000000000000004 [34670.412445] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [34670.412446] R13: 0000000000000000 R14: ffffffff82123e93 R15: ffffc90000003d80 [34670.412447] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34670.412449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34670.412450] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34670.412452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34670.412453] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34670.412454] Call Trace: [34670.412455] [34670.412456] ? nmi_cpu_backtrace+0xad/0x120 [34670.412460] ? nmi_cpu_backtrace_handler+0x11/0x20 [34670.412462] ? nmi_handle+0xd4/0x260 [34670.412469] ? default_do_nmi+0x49/0xf0 [34670.412471] ? exc_nmi+0x11e/0x150 [34670.412473] ? end_repeat_nmi+0x16/0x67 [34670.412480] ? vprintk_emit+0x1eb/0x340 [34670.412483] ? vprintk_emit+0x1eb/0x340 [34670.412486] ? vprintk_emit+0x1eb/0x340 [34670.412488] [34670.412489] [34670.412491] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34670.412493] _printk+0x5c/0x80 [34670.412498] show_trace_log_lvl+0x15f/0x3f0 [34670.412506] ? do_raw_spin_lock+0xb5/0xc0 [34670.412509] ? touch_softlockup_watchdog+0x20/0x20 [34670.412512] watchdog_timer_fn+0x1b4/0x220 [34670.412514] __hrtimer_run_queues+0x1d3/0x360 [34670.412519] hrtimer_interrupt+0x104/0x240 [34670.412521] ? __local_bh_enable+0x50/0x60 [34670.412526] __sysvec_apic_timer_interrupt+0x61/0x160 [34670.412529] sysvec_apic_timer_interrupt+0x6d/0x90 [34670.412532] [34670.412532] [34670.412533] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34670.412536] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34670.412539] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34670.412541] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34670.412542] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34670.412543] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34670.412544] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34670.412546] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34670.412547] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34670.412553] do_raw_spin_lock+0xb5/0xc0 [34670.412556] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34670.412584] try_get_cap_refs+0x159/0x880 [ceph] [34670.412605] __ceph_get_caps+0x4a1/0x760 [ceph] [34670.412625] ? __file_remove_privs+0x89/0x110 [34670.412628] ? wake_bit_function+0x70/0x70 [34670.412632] ? ceph_write_iter+0x260/0x770 [ceph] [34670.412648] ceph_write_iter+0x260/0x770 [ceph] [34670.412666] ? lock_is_held_type+0xaa/0x120 [34670.412670] vfs_write+0x324/0x4f0 [34670.412677] ksys_pwrite64+0x65/0xa0 [34670.412680] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34670.412684] do_syscall_64+0x38/0x80 [34670.412687] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34670.412690] RIP: 0033:0x7f48f0eeb93b [34670.412692] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34670.412693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [34670.412695] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34670.412696] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34670.412697] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34670.412698] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34670.412715] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34670.412721] [34670.413430] Sending NMI from CPU 4 to CPUs 7: [34670.413433] NMI backtrace for cpu 7 [34670.413434] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34670.413436] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34670.413437] Workqueue: events delayed_work [ceph] [34670.413454] RIP: 0010:io_serial_out+0x1b/0x20 [34670.413457] Code: b7 57 40 d3 e6 01 f2 ec 0f b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 89 d0 0f b7 57 40 d3 e6 01 f2 ee 0f 1f 40 00 0f 1f 44 00 00 0f b6 87 fa 00 00 00 48 c7 87 e8 02 [34670.413458] RSP: 0018:ffffc90000264cc0 EFLAGS: 00000002 [34670.413460] RAX: 0000000000000072 RBX: ffffffff838a9560 RCX: 0000000000000000 [34670.413460] RDX: 00000000000002f8 RSI: 0000000000000000 RDI: ffffffff838a9560 [34670.413461] RBP: ffffffff837f4b09 R08: 00006ff6d1995660 R09: 0000000000000007 [34670.413462] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000009 [34670.413463] R13: ffffffff837f4b18 R14: 0000000000000010 R15: 0000000000000000 [34670.413463] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34670.413464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34670.413465] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34670.413466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34670.413467] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34670.413468] Call Trace: [34670.413468] [34670.413469] ? nmi_cpu_backtrace+0xad/0x120 [34670.413473] ? nmi_cpu_backtrace_handler+0x11/0x20 [34670.413474] ? nmi_handle+0xd4/0x260 [34670.413478] ? default_do_nmi+0x49/0xf0 [34670.413479] ? exc_nmi+0x11e/0x150 [34670.413481] ? end_repeat_nmi+0x16/0x67 [34670.413485] ? io_serial_out+0x1b/0x20 [34670.413487] ? io_serial_out+0x1b/0x20 [34670.413489] ? io_serial_out+0x1b/0x20 [34670.413490] [34670.413491] [34670.413491] serial8250_console_write+0x35e/0x3f0 [34670.413494] ? lock_release+0xc5/0x260 [34670.413497] console_flush_all+0x227/0x5a0 [34670.413498] ? console_flush_all+0x1ff/0x5a0 [34670.413500] ? console_flush_all+0x1ff/0x5a0 [34670.413502] console_unlock+0x7b/0x120 [34670.413505] vprintk_emit+0x246/0x340 [34670.413506] ? irq_work_queue+0x2c/0x50 [34670.413511] ? touch_softlockup_watchdog+0x20/0x20 [34670.413512] _printk+0x5c/0x80 [34670.413515] print_modules+0x6e/0xe0 [34670.413517] watchdog_timer_fn+0x19f/0x220 [34670.413519] __hrtimer_run_queues+0x1d3/0x360 [34670.413522] hrtimer_interrupt+0x104/0x240 [34670.413523] ? __local_bh_enable+0x50/0x60 [34670.413526] __sysvec_apic_timer_interrupt+0x61/0x160 [34670.413529] sysvec_apic_timer_interrupt+0x6d/0x90 [34670.413531] [34670.413531] [34670.413532] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34670.413534] RIP: 0010:_raw_spin_lock+0x14/0x40 [34670.413536] Code: 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 [34670.413537] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [34670.413538] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [34670.413539] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [34670.413540] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [34670.413541] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [34670.413541] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34670.413545] igrab+0x1d/0x50 [34670.413547] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34670.413561] delayed_work+0x1d8/0x280 [ceph] [34670.413575] ? process_scheduled_works+0x22a/0x510 [34670.413578] process_scheduled_works+0x2a4/0x510 [34670.413582] worker_thread+0x188/0x340 [34670.413585] ? rescuer_thread+0x310/0x310 [34670.413587] kthread+0xf5/0x130 [34670.413590] ? kthread_complete_and_exit+0x20/0x20 [34670.413592] ret_from_fork+0x31/0x40 [34670.413595] ? kthread_complete_and_exit+0x20/0x20 [34670.413597] ret_from_fork_asm+0x11/0x20 [34670.413601] [34670.415747] [34670.415748] irq event stamp: 198250750 [34670.419144] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34670.421242] hardirqs last enabled at (198250749): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34670.423515] ? queued_spin_lock_slowpath+0x18/0x2e0 [34670.425608] hardirqs last disabled at (198250750): [] sysvec_apic_timer_interrupt+0xe/0x90 [34670.431711] do_raw_spin_lock+0xb5/0xc0 [34670.442214] softirqs last enabled at (198250748): [] __do_softirq+0x2be/0x371 [34670.452894] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34670.460451] softirqs last disabled at (198250743): [] irq_exit_rcu+0xb0/0xd0 [34670.464804] try_get_cap_refs+0x159/0x880 [ceph] [34670.468284] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34670.477233] __ceph_get_caps+0x4a1/0x760 [ceph] [34670.484712] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34670.489151] ? __file_remove_privs+0x89/0x110 [34670.507883] Workqueue: events delayed_work [ceph] [34670.513102] ? wake_bit_function+0x70/0x70 [34670.520224] [34670.520225] RIP: 0010:_raw_spin_lock+0x14/0x40 [34670.527353] ? ceph_write_iter+0x260/0x770 [ceph] [34670.534472] Code: 48 89 df 31 f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 [34670.541597] ceph_write_iter+0x260/0x770 [ceph] [34670.548722] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [34670.556804] ? lock_is_held_type+0xaa/0x120 [34670.562536] [34670.562537] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [34670.569664] vfs_write+0x324/0x4f0 [34670.576783] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [34670.583914] ksys_pwrite64+0x65/0xa0 [34670.586353] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [34670.588363] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34670.592541] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [34670.597327] do_syscall_64+0x38/0x80 [34670.600895] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34670.604728] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34670.608123] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34670.611954] RIP: 0033:0x7f48f0eeb93b [34670.615784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34670.619615] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34670.623448] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34670.625545] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34670.627554] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34670.632512] ORIG_RAX: 0000000000000012 [34670.635562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34670.639740] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34670.643744] Call Trace: [34670.648528] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34670.652619] [34670.656970] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34670.661061] ? watchdog_timer_fn+0x1b4/0x220 [34670.665152] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34670.670196] ? __hrtimer_run_queues+0x1d3/0x360 [34670.674979] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34670.677080] ? hrtimer_interrupt+0x104/0x240 [34670.679179] [34670.684304] ? __local_bh_enable+0x50/0x60 [34671.572164] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34671.577388] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34671.582345] [34671.584443] [34671.586540] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34671.591848] ? _raw_spin_lock+0x14/0x40 [34671.595685] igrab+0x1d/0x50 [34671.598562] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34671.603724] delayed_work+0x1d8/0x280 [ceph] [34671.608001] ? process_scheduled_works+0x22a/0x510 [34671.612791] process_scheduled_works+0x2a4/0x510 [34671.617406] worker_thread+0x188/0x340 [34671.621155] ? rescuer_thread+0x310/0x310 [34671.625160] kthread+0xf5/0x130 [34671.628297] ? kthread_complete_and_exit+0x20/0x20 [34671.633082] ret_from_fork+0x31/0x40 [34671.636650] ? kthread_complete_and_exit+0x20/0x20 [34671.641444] ret_from_fork_asm+0x11/0x20 [34671.645367] [34698.066473] watchdog: BUG: soft lockup - CPU#0 stuck for 21201s! [dbench:78683] [34698.073773] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34698.078473] watchdog: BUG: soft lockup - CPU#7 stuck for 30198s! [kworker/7:4:78808] [34698.129214] irq event stamp: 60135990 [34698.129215] hardirqs last enabled at (60135989): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34698.136946] Modules linked in: [34698.140603] hardirqs last disabled at (60135990): [] sysvec_apic_timer_interrupt+0xe/0x90 [34698.150759] ceph [34698.153810] softirqs last enabled at (60135972): [] __do_softirq+0x2be/0x371 [34698.163535] libceph [34698.165459] softirqs last disabled at (60135967): [] irq_exit_rcu+0xb0/0xd0 [34698.174155] dns_resolver fscache [34698.176336] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34698.184847] netfs veth [34698.188173] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34698.197119] nft_chain_nat [34698.199564] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34698.207035] xt_MASQUERADE nf_nat [34698.209737] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34698.215215] nf_conntrack [34698.218525] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34698.237265] nf_defrag_ipv6 nf_defrag_ipv4 [34698.239881] [34698.245097] nft_compat nf_tables [34698.249188] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34698.250679] nfnetlink [34698.253991] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34698.261115] bridge stp [34698.263472] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34698.270595] llc xfs [34698.273040] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34698.280164] libcrc32c sunrpc [34698.282348] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34698.289474] intel_rapl_msr intel_rapl_common [34698.292437] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34698.299560] sb_edac x86_pkg_temp_thermal [34698.303913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34698.311990] coretemp kvm_intel [34698.315993] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34698.321729] kvm irqbypass [34698.324871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34698.331992] i2c_i801 iTCO_wdt [34698.334696] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34698.341819] iTCO_vendor_support [34698.344871] Call Trace: [34698.344872] [34698.351995] i2c_smbus crct10dif_pclmul [34698.355220] ? watchdog_timer_fn+0x1b4/0x220 [34698.357664] joydev crc32_pclmul [34698.359675] ? __hrtimer_run_queues+0x1d3/0x360 [34698.363504] ipmi_ssif ghash_clmulni_intel [34698.367773] ? hrtimer_interrupt+0x104/0x240 [34698.370993] mei_me mei [34698.375518] ? __local_bh_enable+0x50/0x60 [34698.379607] lpc_ich sha512_ssse3 [34698.383875] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34698.386317] acpi_ipmi mfd_core [34698.390408] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34698.393717] wmi [34698.398936] [34698.402072] ipmi_si ipmi_devintf [34698.407029] [34698.407031] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34698.408871] ipmi_msghandler [34698.410969] ? queued_spin_lock_slowpath+0x16/0x2e0 [34698.414275] ioatdma acpi_pad [34698.416374] do_raw_spin_lock+0xb5/0xc0 [34698.421675] acpi_power_meter ixgbe igb [34698.424555] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34698.429424] mdio crc32c_intel [34698.432390] try_get_cap_refs+0x159/0x880 [ceph] [34698.436220] nvme ptp [34698.440056] __ceph_get_caps+0x4a1/0x760 [ceph] [34698.444833] nvme_core [34698.447890] ? __file_remove_privs+0x89/0x110 [34698.452496] i2c_algo_bit pps_core [34698.454767] ? wake_bit_function+0x70/0x70 [34698.459291] dca fuse [34698.461651] ? ceph_write_iter+0x260/0x770 [ceph] [34698.465998] [34698.465999] irq event stamp: 198465096 [34698.469397] ceph_write_iter+0x260/0x770 [ceph] [34698.473487] hardirqs last enabled at (198465095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34698.475761] ? lock_is_held_type+0xaa/0x120 [34698.480455] hardirqs last disabled at (198465096): [] sysvec_apic_timer_interrupt+0xe/0x90 [34698.481949] vfs_write+0x324/0x4f0 [34698.485690] softirqs last enabled at (198465094): [] __do_softirq+0x2be/0x371 [34698.490217] ksys_pwrite64+0x65/0xa0 [34698.500456] softirqs last disabled at (198465089): [] irq_exit_rcu+0xb0/0xd0 [34698.504636] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34698.514446] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34698.517845] do_syscall_64+0x38/0x80 [34698.526614] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34698.530185] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34698.538782] Workqueue: events delayed_work [ceph] [34698.543913] RIP: 0033:0x7f48f0eeb93b [34698.553291] [34698.553292] RIP: 0010:lock_acquire+0xf0/0x2b0 [34698.556877] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34698.564350] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34698.569394] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34698.574091] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34698.577662] ORIG_RAX: 0000000000000012 [34698.579155] [34698.579156] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34698.583502] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34698.602241] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34698.620977] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34698.626195] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34698.631414] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34698.635243] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34698.636735] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34698.643877] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34698.651002] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34698.658125] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34698.665255] [34698.672372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34698.724002] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34698.731126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34698.738248] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34698.745374] Call Trace: [34698.747819] [34698.749831] ? watchdog_timer_fn+0x1b4/0x220 [34698.754093] ? __hrtimer_run_queues+0x1d3/0x360 [34698.758621] ? hrtimer_interrupt+0x104/0x240 [34698.762889] ? __local_bh_enable+0x50/0x60 [34698.766982] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34698.772199] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34698.777160] [34698.779261] [34698.781361] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34698.786667] ? lock_acquire+0xf0/0x2b0 [34698.790418] ? igrab+0x43/0x50 [34698.793467] ? lock_release+0xc5/0x260 [34698.797212] _raw_spin_lock+0x30/0x40 [34698.800880] ? igrab+0x1d/0x50 [34698.803937] igrab+0x1d/0x50 [34698.806814] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34698.811964] delayed_work+0x1d8/0x280 [ceph] [34698.816244] ? process_scheduled_works+0x22a/0x510 [34698.821027] process_scheduled_works+0x2a4/0x510 [34698.825642] worker_thread+0x188/0x340 [34698.829391] ? rescuer_thread+0x310/0x310 [34698.833394] kthread+0xf5/0x130 [34698.836531] ? kthread_complete_and_exit+0x20/0x20 [34698.841317] ret_from_fork+0x31/0x40 [34698.844886] ? kthread_complete_and_exit+0x20/0x20 [34698.849671] ret_from_fork_asm+0x11/0x20 [34698.853594] [34726.066924] watchdog: BUG: soft lockup - CPU#0 stuck for 21227s! [dbench:78683] [34726.074227] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34726.078923] watchdog: BUG: soft lockup - CPU#7 stuck for 30224s! [kworker/7:4:78808] [34726.129675] irq event stamp: 60194164 [34726.137406] Modules linked in: ceph libceph dns_resolver fscache netfs veth [34726.141064] hardirqs last enabled at (60194163): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34726.148024] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [34726.158180] hardirqs last disabled at (60194164): [] sysvec_apic_timer_interrupt+0xe/0x90 [34726.165131] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [34726.174855] softirqs last enabled at (60194146): [] __do_softirq+0x2be/0x371 [34726.181286] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [34726.189970] softirqs last disabled at (60194141): [] irq_exit_rcu+0xb0/0xd0 [34726.199002] coretemp kvm_intel kvm irqbypass [34726.207513] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34726.207515] i2c_i801 [34726.207516] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34726.211863] iTCO_wdt iTCO_vendor_support [34726.220810] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34726.223079] i2c_smbus crct10dif_pclmul [34726.230550] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34726.234554] joydev crc32_pclmul [34726.240030] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34726.243862] ipmi_ssif ghash_clmulni_intel [34726.262608] [34726.265830] mei_me mei [34726.271049] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34726.275139] lpc_ich sha512_ssse3 [34726.276629] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34726.279073] acpi_ipmi mfd_core [34726.286198] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34726.289509] wmi ipmi_si [34726.296634] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34726.299772] ipmi_devintf ipmi_msghandler [34726.306894] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34726.309426] ioatdma acpi_pad [34726.316550] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34726.320554] acpi_power_meter [34726.327678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34726.330643] ixgbe igb [34726.338719] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34726.341684] mdio crc32c_intel [34726.347422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34726.349778] nvme ptp [34726.356901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34726.359952] nvme_core i2c_algo_bit [34726.367077] Call Trace: [34726.369349] pps_core dca [34726.376476] [34726.379956] fuse [34726.382401] ? watchdog_timer_fn+0x1b4/0x220 [34726.385018] irq event stamp: 198685892 [34726.387030] ? __hrtimer_run_queues+0x1d3/0x360 [34726.388952] hardirqs last enabled at (198685891): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34726.393220] ? hrtimer_interrupt+0x104/0x240 [34726.396961] hardirqs last disabled at (198685892): [] sysvec_apic_timer_interrupt+0xe/0x90 [34726.401485] ? __local_bh_enable+0x50/0x60 [34726.411728] softirqs last enabled at (198685890): [] __do_softirq+0x2be/0x371 [34726.415996] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34726.425803] softirqs last disabled at (198685885): [] irq_exit_rcu+0xb0/0xd0 [34726.429896] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34726.438666] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34726.443882] [34726.452480] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34726.457438] [34726.457440] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34726.466815] Workqueue: events delayed_work [ceph] [34726.468916] ? queued_spin_lock_slowpath+0x16/0x2e0 [34726.476383] [34726.478482] do_raw_spin_lock+0xb5/0xc0 [34726.483785] RIP: 0010:lock_acquire+0xf0/0x2b0 [34726.488483] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34726.493353] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34726.494848] try_get_cap_refs+0x159/0x880 [ceph] [34726.498674] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34726.503030] __ceph_get_caps+0x4a1/0x760 [ceph] [34726.507808] [34726.507809] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34726.526551] ? __file_remove_privs+0x89/0x110 [34726.531157] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34726.536374] ? wake_bit_function+0x70/0x70 [34726.540898] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34726.542392] ? ceph_write_iter+0x260/0x770 [ceph] [34726.549515] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34726.553865] ceph_write_iter+0x260/0x770 [ceph] [34726.560990] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34726.565084] ? lock_is_held_type+0xaa/0x120 [34726.572205] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34726.576902] vfs_write+0x324/0x4f0 [34726.584024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34726.588555] ksys_pwrite64+0x65/0xa0 [34726.595674] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34726.599853] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34726.607928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34726.611329] do_syscall_64+0x38/0x80 [34726.617063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34726.620635] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34726.627758] Call Trace: [34726.627760] [34726.632891] RIP: 0033:0x7f48f0eeb93b [34726.640015] ? watchdog_timer_fn+0x1b4/0x220 [34726.643584] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34726.650710] ? __hrtimer_run_queues+0x1d3/0x360 [34726.655753] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34726.658200] ? hrtimer_interrupt+0x104/0x240 [34726.660207] ORIG_RAX: 0000000000000012 [34726.663777] ? __local_bh_enable+0x50/0x60 [34726.668042] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34726.686781] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34726.691302] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34726.696538] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34726.700802] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34726.704632] [34726.708724] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34726.715847] [34726.715849] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34726.721065] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34726.728194] ? lock_acquire+0xf0/0x2b0 [34726.733153] [34726.740273] ? igrab+0x43/0x50 [34726.773016] ? lock_release+0xc5/0x260 [34726.776761] _raw_spin_lock+0x30/0x40 [34726.780425] ? igrab+0x1d/0x50 [34726.783475] igrab+0x1d/0x50 [34726.786354] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34726.791502] delayed_work+0x1d8/0x280 [ceph] [34726.795786] ? process_scheduled_works+0x22a/0x510 [34726.800576] process_scheduled_works+0x2a4/0x510 [34726.805191] worker_thread+0x188/0x340 [34726.808941] ? rescuer_thread+0x310/0x310 [34726.812942] kthread+0xf5/0x130 [34726.816082] ? kthread_complete_and_exit+0x20/0x20 [34726.820873] ret_from_fork+0x31/0x40 [34726.824452] ? kthread_complete_and_exit+0x20/0x20 [34726.829238] ret_from_fork_asm+0x11/0x20 [34726.833168] [34754.067373] watchdog: BUG: soft lockup - CPU#0 stuck for 21253s! [dbench:78683] [34754.074671] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34754.079373] watchdog: BUG: soft lockup - CPU#7 stuck for 30250s! [kworker/7:4:78808] [34754.130111] irq event stamp: 60252126 [34754.130112] hardirqs last enabled at (60252125): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34754.137842] Modules linked in: [34754.141500] hardirqs last disabled at (60252126): [] sysvec_apic_timer_interrupt+0xe/0x90 [34754.151655] ceph [34754.154707] softirqs last enabled at (60252108): [] __do_softirq+0x2be/0x371 [34754.164432] libceph dns_resolver [34754.166355] softirqs last disabled at (60252103): [] irq_exit_rcu+0xb0/0xd0 [34754.175048] fscache netfs [34754.178358] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34754.186869] veth nft_chain_nat [34754.189575] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34754.198519] xt_MASQUERADE nf_nat [34754.201656] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34754.209126] nf_conntrack [34754.212438] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34754.217914] nf_defrag_ipv6 [34754.220533] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34754.239271] nf_defrag_ipv4 nft_compat [34754.242060] [34754.247278] nf_tables nfnetlink [34754.251022] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34754.252513] bridge stp [34754.255737] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34754.262861] llc xfs [34754.265304] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34754.272429] libcrc32c sunrpc [34754.274612] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34754.281737] intel_rapl_msr [34754.284701] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34754.291826] intel_rapl_common sb_edac [34754.294615] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34754.301741] x86_pkg_temp_thermal coretemp [34754.305500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34754.313579] kvm_intel kvm [34754.317670] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34754.323407] irqbypass i2c_i801 [34754.326111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34754.333234] iTCO_wdt iTCO_vendor_support [34754.336372] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34754.343498] i2c_smbus [34754.347502] Call Trace: [34754.354624] crct10dif_pclmul [34754.356982] [34754.359426] joydev [34754.362392] ? watchdog_timer_fn+0x1b4/0x220 [34754.364400] crc32_pclmul ipmi_ssif [34754.366499] ? __hrtimer_run_queues+0x1d3/0x360 [34754.370770] ghash_clmulni_intel [34754.374259] ? hrtimer_interrupt+0x104/0x240 [34754.378778] mei_me [34754.382004] ? __local_bh_enable+0x50/0x60 [34754.386267] mei [34754.388368] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34754.392455] lpc_ich sha512_ssse3 [34754.394294] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34754.399511] acpi_ipmi mfd_core [34754.402823] [34754.407778] wmi ipmi_si [34754.410915] [34754.413014] ipmi_devintf [34754.415544] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34754.417642] ipmi_msghandler ioatdma [34754.420262] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34754.425562] acpi_pad [34754.429136] do_raw_spin_lock+0xb5/0xc0 [34754.434003] acpi_power_meter [34754.436275] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34754.440105] ixgbe igb [34754.443073] try_get_cap_refs+0x159/0x880 [ceph] [34754.447855] mdio crc32c_intel [34754.450217] __ceph_get_caps+0x4a1/0x760 [ceph] [34754.454823] nvme [34754.457877] ? __file_remove_privs+0x89/0x110 [34754.462396] ptp nvme_core [34754.464322] ? wake_bit_function+0x70/0x70 [34754.468672] i2c_algo_bit pps_core [34754.471379] ? ceph_write_iter+0x260/0x770 [ceph] [34754.475466] dca fuse [34754.478865] ceph_write_iter+0x260/0x770 [ceph] [34754.483562] [34754.485837] ? lock_is_held_type+0xaa/0x120 [34754.490356] irq event stamp: 198906868 [34754.491848] vfs_write+0x324/0x4f0 [34754.496023] hardirqs last enabled at (198906867): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34754.499776] ksys_pwrite64+0x65/0xa0 [34754.503165] hardirqs last disabled at (198906868): [] sysvec_apic_timer_interrupt+0xe/0x90 [34754.513410] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34754.516981] softirqs last enabled at (198906866): [] __do_softirq+0x2be/0x371 [34754.526794] do_syscall_64+0x38/0x80 [34754.531922] softirqs last disabled at (198906861): [] irq_exit_rcu+0xb0/0xd0 [34754.540694] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34754.544264] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34754.552862] RIP: 0033:0x7f48f0eeb93b [34754.557904] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34754.567282] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34754.570853] Workqueue: events delayed_work [ceph] [34754.578323] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34754.597063] [34754.601776] ORIG_RAX: 0000000000000012 [34754.606994] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [34754.608485] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34754.612315] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [34754.616926] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34754.624051] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [34754.642787] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34754.649911] [34754.649912] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [34754.655129] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34754.662255] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [34754.663746] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34754.670868] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [34754.677999] [34754.685116] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [34754.708674] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34754.715799] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34754.723874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34754.729613] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34754.736738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34754.743860] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34754.750984] Call Trace: [34754.753430] [34754.755440] ? watchdog_timer_fn+0x1b4/0x220 [34754.759706] ? __hrtimer_run_queues+0x1d3/0x360 [34754.764239] ? hrtimer_interrupt+0x104/0x240 [34754.768501] ? __local_bh_enable+0x50/0x60 [34754.772595] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34754.777817] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34754.782779] [34754.784872] [34754.786969] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34754.792279] ? do_raw_spin_lock+0x42/0xc0 [34754.796289] igrab+0x1d/0x50 [34754.799165] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34754.804315] delayed_work+0x1d8/0x280 [ceph] [34754.808597] ? process_scheduled_works+0x22a/0x510 [34754.813386] process_scheduled_works+0x2a4/0x510 [34754.818001] worker_thread+0x188/0x340 [34754.821752] ? rescuer_thread+0x310/0x310 [34754.825781] kthread+0xf5/0x130 [34754.828918] ? kthread_complete_and_exit+0x20/0x20 [34754.833702] ret_from_fork+0x31/0x40 [34754.837272] ? kthread_complete_and_exit+0x20/0x20 [34754.842060] ret_from_fork_asm+0x11/0x20 [34754.845988] [34782.067823] watchdog: BUG: soft lockup - CPU#0 stuck for 21279s! [dbench:78683] [34782.075123] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34782.079823] watchdog: BUG: soft lockup - CPU#7 stuck for 30276s! [kworker/7:4:78808] [34782.130573] irq event stamp: 60310268 [34782.138303] Modules linked in: ceph libceph dns_resolver fscache [34782.141959] hardirqs last enabled at (60310267): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34782.141964] netfs veth nft_chain_nat xt_MASQUERADE [34782.158114] hardirqs last disabled at (60310268): [] sysvec_apic_timer_interrupt+0xe/0x90 [34782.162987] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [34782.172711] softirqs last enabled at (60310250): [] __do_softirq+0x2be/0x371 [34782.179487] nf_tables nfnetlink bridge stp [34782.188190] softirqs last disabled at (60310245): [] irq_exit_rcu+0xb0/0xd0 [34782.188193] llc xfs libcrc32c sunrpc intel_rapl_msr [34782.200905] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34782.200906] intel_rapl_common sb_edac x86_pkg_temp_thermal [34782.214806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34782.214808] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34782.220368] coretemp kvm_intel [34782.227842] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34782.233316] kvm irqbypass [34782.236455] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34782.255191] i2c_i801 [34782.257913] [34782.257914] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34782.263131] iTCO_wdt [34782.265401] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34782.266892] iTCO_vendor_support i2c_smbus [34782.274016] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34782.276287] crct10dif_pclmul [34782.283412] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34782.287503] joydev crc32_pclmul [34782.294627] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34782.297591] ipmi_ssif ghash_clmulni_intel [34782.304714] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34782.307939] mei_me mei [34782.315063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34782.319155] lpc_ich sha512_ssse3 [34782.327230] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34782.329675] acpi_ipmi mfd_core [34782.335413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34782.338723] wmi ipmi_si [34782.345848] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34782.348984] ipmi_devintf ipmi_msghandler [34782.356110] Call Trace: [34782.356111] [34782.358640] ioatdma acpi_pad [34782.365767] ? watchdog_timer_fn+0x1b4/0x220 [34782.369767] acpi_power_meter ixgbe [34782.372221] ? __hrtimer_run_queues+0x1d3/0x360 [34782.374231] igb mdio [34782.377219] ? hrtimer_interrupt+0x104/0x240 [34782.381476] crc32c_intel [34782.384961] ? __local_bh_enable+0x50/0x60 [34782.389486] nvme [34782.391757] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34782.396019] ptp [34782.398638] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34782.402728] nvme_core i2c_algo_bit [34782.404653] [34782.409870] pps_core [34782.411707] [34782.411709] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34782.416665] dca fuse [34782.420153] ? queued_spin_lock_slowpath+0x16/0x2e0 [34782.422245] [34782.422246] irq event stamp: 199127598 [34782.424519] do_raw_spin_lock+0xb5/0xc0 [34782.426614] hardirqs last enabled at (199127597): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34782.431920] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34782.434188] hardirqs last disabled at (199127598): [] sysvec_apic_timer_interrupt+0xe/0x90 [34782.439062] try_get_cap_refs+0x159/0x880 [ceph] [34782.440551] softirqs last enabled at (199127596): [] __do_softirq+0x2be/0x371 [34782.444299] __ceph_get_caps+0x4a1/0x760 [ceph] [34782.448123] softirqs last disabled at (199127591): [] irq_exit_rcu+0xb0/0xd0 [34782.458372] ? __file_remove_privs+0x89/0x110 [34782.463153] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34782.472965] ? wake_bit_function+0x70/0x70 [34782.477574] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34782.486347] ? ceph_write_iter+0x260/0x770 [ceph] [34782.490869] Workqueue: events delayed_work [ceph] [34782.499468] ceph_write_iter+0x260/0x770 [ceph] [34782.503818] [34782.503819] RIP: 0010:lock_acquire+0xf0/0x2b0 [34782.513220] ? lock_is_held_type+0xaa/0x120 [34782.517303] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34782.524777] vfs_write+0x324/0x4f0 [34782.529472] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34782.534174] ksys_pwrite64+0x65/0xa0 [34782.538693] [34782.538694] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34782.540183] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34782.544535] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34782.548712] do_syscall_64+0x38/0x80 [34782.567449] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34782.570849] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34782.576063] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34782.579635] RIP: 0033:0x7f48f0eeb93b [34782.581125] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34782.588250] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34782.593381] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34782.600513] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34782.604084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34782.611223] ORIG_RAX: 0000000000000012 [34782.616270] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34782.623393] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34782.626964] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34782.634089] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34782.652826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34782.660903] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34782.666121] Call Trace: [34782.666123] [34782.671858] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34782.675692] ? watchdog_timer_fn+0x1b4/0x220 [34782.682813] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34782.689940] ? __hrtimer_run_queues+0x1d3/0x360 [34782.697066] [34782.704188] ? hrtimer_interrupt+0x104/0x240 [34782.752374] ? __local_bh_enable+0x50/0x60 [34782.756468] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34782.761692] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34782.766648] [34782.768744] [34782.770844] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34782.776151] ? lock_acquire+0xf0/0x2b0 [34782.779901] ? igrab+0x43/0x50 [34782.782952] ? lock_release+0xc5/0x260 [34782.786696] _raw_spin_lock+0x30/0x40 [34782.790361] ? igrab+0x1d/0x50 [34782.793412] igrab+0x1d/0x50 [34782.796289] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34782.801439] delayed_work+0x1d8/0x280 [ceph] [34782.805720] ? process_scheduled_works+0x22a/0x510 [34782.810510] process_scheduled_works+0x2a4/0x510 [34782.815127] worker_thread+0x188/0x340 [34782.818875] ? rescuer_thread+0x310/0x310 [34782.822879] kthread+0xf5/0x130 [34782.826017] ? kthread_complete_and_exit+0x20/0x20 [34782.830800] ret_from_fork+0x31/0x40 [34782.834380] ? kthread_complete_and_exit+0x20/0x20 [34782.839163] ret_from_fork_asm+0x11/0x20 [34782.843087] [34810.068267] watchdog: BUG: soft lockup - CPU#0 stuck for 21305s! [dbench:78683] [34810.075567] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34810.080267] watchdog: BUG: soft lockup - CPU#7 stuck for 30302s! [kworker/7:4:78808] [34810.131008] irq event stamp: 60368340 [34810.131009] hardirqs last enabled at (60368339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34810.138738] Modules linked in: [34810.142397] hardirqs last disabled at (60368340): [] sysvec_apic_timer_interrupt+0xe/0x90 [34810.152554] ceph [34810.155604] softirqs last enabled at (60368322): [] __do_softirq+0x2be/0x371 [34810.165328] libceph [34810.167253] softirqs last disabled at (60368317): [] irq_exit_rcu+0xb0/0xd0 [34810.175941] dns_resolver [34810.178128] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34810.186658] fscache netfs [34810.189276] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34810.198219] veth nft_chain_nat [34810.200940] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34810.208411] xt_MASQUERADE nf_nat [34810.211547] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34810.217027] nf_conntrack nf_defrag_ipv6 [34810.220337] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34810.239073] nf_defrag_ipv4 [34810.242992] [34810.242993] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34810.248209] nft_compat [34810.251000] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34810.252491] nf_tables nfnetlink [34810.259614] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34810.262058] bridge stp [34810.269182] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34810.272406] llc xfs [34810.279531] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34810.281974] libcrc32c sunrpc [34810.289099] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34810.291282] intel_rapl_msr intel_rapl_common [34810.298408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34810.301370] sb_edac x86_pkg_temp_thermal [34810.309449] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34810.313801] coretemp kvm_intel [34810.319538] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34810.323541] kvm irqbypass [34810.330665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34810.333802] i2c_i801 iTCO_wdt [34810.340944] Call Trace: [34810.340945] [34810.343665] iTCO_vendor_support i2c_smbus [34810.350791] ? watchdog_timer_fn+0x1b4/0x220 [34810.353840] crct10dif_pclmul joydev [34810.356285] ? __hrtimer_run_queues+0x1d3/0x360 [34810.358296] crc32_pclmul ipmi_ssif [34810.362389] ? hrtimer_interrupt+0x104/0x240 [34810.366665] ghash_clmulni_intel [34810.370239] ? __local_bh_enable+0x50/0x60 [34810.374762] mei_me [34810.378249] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34810.382510] mei [34810.385734] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34810.389825] lpc_ich sha512_ssse3 [34810.391923] [34810.397158] acpi_ipmi [34810.398995] [34810.403951] mfd_core wmi [34810.407264] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34810.409360] ipmi_si ipmi_devintf [34810.411722] ? queued_spin_lock_slowpath+0x18/0x2e0 [34810.413814] ipmi_msghandler ioatdma [34810.416434] do_raw_spin_lock+0xb5/0xc0 [34810.421735] acpi_pad acpi_power_meter [34810.425049] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34810.429918] ixgbe igb [34810.433490] try_get_cap_refs+0x159/0x880 [ceph] [34810.437320] mdio crc32c_intel [34810.441067] __ceph_get_caps+0x4a1/0x760 [ceph] [34810.445846] nvme ptp [34810.448209] ? __file_remove_privs+0x89/0x110 [34810.452816] nvme_core i2c_algo_bit [34810.455866] ? wake_bit_function+0x70/0x70 [34810.460391] pps_core dca [34810.462669] ? ceph_write_iter+0x260/0x770 [ceph] [34810.467020] fuse [34810.470504] ceph_write_iter+0x260/0x770 [ceph] [34810.474594] irq event stamp: 199348476 [34810.477217] ? lock_is_held_type+0xaa/0x120 [34810.481911] hardirqs last enabled at (199348475): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34810.483838] vfs_write+0x324/0x4f0 [34810.488359] hardirqs last disabled at (199348476): [] sysvec_apic_timer_interrupt+0xe/0x90 [34810.492108] ksys_pwrite64+0x65/0xa0 [34810.496279] softirqs last enabled at (199348474): [] __do_softirq+0x2be/0x371 [34810.506523] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34810.509921] softirqs last disabled at (199348469): [] irq_exit_rcu+0xb0/0xd0 [34810.519733] do_syscall_64+0x38/0x80 [34810.523303] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34810.532074] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34810.537203] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34810.545801] RIP: 0033:0x7f48f0eeb93b [34810.549372] Workqueue: events delayed_work [ceph] [34810.558751] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34810.563793] [34810.571266] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34810.574836] RIP: 0010:lock_acquire+0xf0/0x2b0 [34810.579533] ORIG_RAX: 0000000000000012 [34810.598271] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34810.599762] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34810.604979] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34810.609331] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34810.613160] [34810.613161] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34810.631899] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34810.639022] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34810.644240] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34810.651363] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34810.652855] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34810.659979] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34810.667109] [34810.674226] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34810.712033] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34810.720110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34810.725848] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34810.732971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34810.740095] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34810.747218] Call Trace: [34810.749673] [34810.751684] ? watchdog_timer_fn+0x1b4/0x220 [34810.755953] ? __hrtimer_run_queues+0x1d3/0x360 [34810.760482] ? hrtimer_interrupt+0x104/0x240 [34810.764745] ? __local_bh_enable+0x50/0x60 [34810.768837] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34810.774062] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34810.779017] [34810.781116] [34810.783214] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34810.788520] ? lock_acquire+0xf0/0x2b0 [34810.792271] ? igrab+0x43/0x50 [34810.795321] ? lock_release+0xc5/0x260 [34810.799065] _raw_spin_lock+0x30/0x40 [34810.802722] ? igrab+0x1d/0x50 [34810.805774] igrab+0x1d/0x50 [34810.808670] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34810.813818] delayed_work+0x1d8/0x280 [ceph] [34810.818100] ? process_scheduled_works+0x22a/0x510 [34810.822889] process_scheduled_works+0x2a4/0x510 [34810.827504] worker_thread+0x188/0x340 [34810.831255] ? rescuer_thread+0x310/0x310 [34810.835258] kthread+0xf5/0x130 [34810.838394] ? kthread_complete_and_exit+0x20/0x20 [34810.843179] ret_from_fork+0x31/0x40 [34810.846759] ? kthread_complete_and_exit+0x20/0x20 [34810.851543] ret_from_fork_asm+0x11/0x20 [34810.855475] [34838.068706] watchdog: BUG: soft lockup - CPU#0 stuck for 21331s! [dbench:78683] [34838.076012] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34838.080706] watchdog: BUG: soft lockup - CPU#7 stuck for 30328s! [kworker/7:4:78808] [34838.131460] irq event stamp: 60426462 [34838.131461] hardirqs last enabled at (60426461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34838.139192] Modules linked in: [34838.142848] hardirqs last disabled at (60426462): [] sysvec_apic_timer_interrupt+0xe/0x90 [34838.153007] ceph [34838.156057] softirqs last enabled at (60426444): [] __do_softirq+0x2be/0x371 [34838.165782] libceph [34838.167705] softirqs last disabled at (60426439): [] irq_exit_rcu+0xb0/0xd0 [34838.176389] dns_resolver [34838.178575] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34838.187102] fscache [34838.189719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34838.198662] netfs [34838.200849] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34838.208317] veth nft_chain_nat [34838.210329] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34838.215806] xt_MASQUERADE nf_nat [34838.218944] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34838.237681] nf_conntrack [34838.240991] [34838.240992] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34838.246209] nf_defrag_ipv6 [34838.248828] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34838.250318] nf_defrag_ipv4 nft_compat [34838.257441] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34838.260233] nf_tables nfnetlink [34838.267356] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34838.271102] bridge [34838.278224] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34838.281448] stp [34838.288574] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34838.290670] llc xfs [34838.297795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34838.299632] libcrc32c sunrpc [34838.307710] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34838.309892] intel_rapl_msr [34838.315631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34838.318595] intel_rapl_common sb_edac [34838.325718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34838.328510] x86_pkg_temp_thermal [34838.335632] Call Trace: [34838.335635] [34838.339384] coretemp kvm_intel [34838.346514] ? watchdog_timer_fn+0x1b4/0x220 [34838.349821] kvm irqbypass [34838.352266] ? __hrtimer_run_queues+0x1d3/0x360 [34838.354275] i2c_i801 iTCO_wdt [34838.357417] ? hrtimer_interrupt+0x104/0x240 [34838.361677] iTCO_vendor_support i2c_smbus [34838.364393] ? __local_bh_enable+0x50/0x60 [34838.368914] crct10dif_pclmul [34838.371966] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34838.376228] joydev [34838.380321] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34838.384411] crc32_pclmul ipmi_ssif [34838.387387] [34838.392600] ghash_clmulni_intel [34838.394698] [34838.394699] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34838.399656] mei_me mei [34838.403144] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34838.405236] lpc_ich [34838.408462] do_raw_spin_lock+0xb5/0xc0 [34838.410559] sha512_ssse3 [34838.415864] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34838.418306] acpi_ipmi mfd_core [34838.423181] try_get_cap_refs+0x159/0x880 [ceph] [34838.425361] wmi ipmi_si [34838.429197] __ceph_get_caps+0x4a1/0x760 [ceph] [34838.431809] ipmi_devintf [34838.436597] ? __file_remove_privs+0x89/0x110 [34838.439730] ipmi_msghandler [34838.444341] ? wake_bit_function+0x70/0x70 [34838.446871] ioatdma acpi_pad [34838.451398] ? ceph_write_iter+0x260/0x770 [ceph] [34838.454013] acpi_power_meter ixgbe [34838.458365] ceph_write_iter+0x260/0x770 [ceph] [34838.461242] igb [34838.465338] ? lock_is_held_type+0xaa/0x120 [34838.468297] mdio crc32c_intel [34838.472997] vfs_write+0x324/0x4f0 [34838.476479] nvme ptp [34838.481007] ksys_pwrite64+0x65/0xa0 [34838.482841] nvme_core i2c_algo_bit [34838.487019] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34838.490069] pps_core [34838.493467] do_syscall_64+0x38/0x80 [34838.495736] dca fuse [34838.499308] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34838.502790] [34838.502791] irq event stamp: 199569132 [34838.507923] RIP: 0033:0x7f48f0eeb93b [34838.510192] hardirqs last enabled at (199569131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34838.513764] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34838.516033] hardirqs last disabled at (199569132): [] sysvec_apic_timer_interrupt+0xe/0x90 [34838.521077] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34838.522587] softirqs last enabled at (199569130): [] __do_softirq+0x2be/0x371 [34838.526330] ORIG_RAX: 0000000000000012 [34838.529900] softirqs last disabled at (199569125): [] irq_exit_rcu+0xb0/0xd0 [34838.540145] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34838.558882] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34838.568692] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34838.573911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34838.582680] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34838.586514] Workqueue: events delayed_work [ceph] [34838.595111] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34838.595112] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34838.602233] [34838.602234] RIP: 0010:ceph_check_delayed_caps+0xc7/0x270 [ceph] [34838.611617] [34838.618736] Code: 08 48 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 48 85 c0 0f 85 cd 00 00 00 48 8b 85 78 04 00 00 <49> 39 c5 75 89 31 c0 48 8b 3c 24 48 89 44 24 08 e8 84 5b 07 e1 48 [34838.680600] RSP: 0018:ffffc90002117db8 EFLAGS: 00000246 [34838.685818] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [34838.692942] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [34838.700064] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [34838.707189] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [34838.714312] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34838.721438] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34838.729514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34838.735253] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34838.742390] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34838.749518] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34838.756641] Call Trace: [34838.759087] [34838.761116] ? watchdog_timer_fn+0x1b4/0x220 [34838.765380] ? __hrtimer_run_queues+0x1d3/0x360 [34838.769924] ? hrtimer_interrupt+0x104/0x240 [34838.774192] ? __local_bh_enable+0x50/0x60 [34838.778285] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34838.783501] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34838.788459] [34838.790555] [34838.792653] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34838.797962] ? ceph_check_delayed_caps+0xc7/0x270 [ceph] [34838.803279] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [34838.808593] delayed_work+0x1d8/0x280 [ceph] [34838.812877] ? process_scheduled_works+0x22a/0x510 [34838.817666] process_scheduled_works+0x2a4/0x510 [34838.822280] worker_thread+0x188/0x340 [34838.826030] ? rescuer_thread+0x310/0x310 [34838.830033] kthread+0xf5/0x130 [34838.833172] ? kthread_complete_and_exit+0x20/0x20 [34838.837964] ret_from_fork+0x31/0x40 [34838.841543] ? kthread_complete_and_exit+0x20/0x20 [34838.846327] ret_from_fork_asm+0x11/0x20 [34838.850261] [34865.418507] rcu: INFO: rcu_preempt self-detected stall on CPU [34865.424270] rcu: 7-....: (22262998 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/22261657 fqs=11219817 [34865.434954] rcu: (t=22928402 jiffies g=1309609 q=642569 ncpus=8) [34865.441036] Sending NMI from CPU 7 to CPUs 0: [34865.445390] NMI backtrace for cpu 0 [34865.445392] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34865.445394] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34865.445394] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [34865.445398] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [34865.445399] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34865.445401] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34865.445402] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34865.445402] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34865.445403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34865.445404] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34865.445405] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34865.445406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34865.445407] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34865.445408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34865.445408] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34865.445409] Call Trace: [34865.445410] [34865.445411] ? nmi_cpu_backtrace+0xad/0x120 [34865.445414] ? nmi_cpu_backtrace_handler+0x11/0x20 [34865.445416] ? nmi_handle+0xd4/0x260 [34865.445420] ? default_do_nmi+0x49/0xf0 [34865.445422] ? exc_nmi+0x11e/0x150 [34865.445423] ? end_repeat_nmi+0x16/0x67 [34865.445427] ? queued_spin_lock_slowpath+0x16/0x2e0 [34865.445429] ? queued_spin_lock_slowpath+0x16/0x2e0 [34865.445432] ? queued_spin_lock_slowpath+0x16/0x2e0 [34865.445434] [34865.445434] [34865.445435] do_raw_spin_lock+0xb5/0xc0 [34865.445437] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34865.445452] try_get_cap_refs+0x159/0x880 [ceph] [34865.445466] __ceph_get_caps+0x4a1/0x760 [ceph] [34865.445478] ? __file_remove_privs+0x89/0x110 [34865.445480] ? wake_bit_function+0x70/0x70 [34865.445482] ? ceph_write_iter+0x260/0x770 [ceph] [34865.445492] ceph_write_iter+0x260/0x770 [ceph] [34865.445503] ? lock_is_held_type+0xaa/0x120 [34865.445506] vfs_write+0x324/0x4f0 [34865.445529] ksys_pwrite64+0x65/0xa0 [34865.445531] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34865.445534] do_syscall_64+0x38/0x80 [34865.445536] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34865.445538] RIP: 0033:0x7f48f0eeb93b [34865.445539] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34865.445541] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [34865.445542] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34865.445543] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34865.445544] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34865.445545] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34865.445545] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34865.445548] [34865.446389] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34865.749940] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34865.757412] Workqueue: events delayed_work [ceph] [34865.762136] RIP: 0010:lock_acquire+0xf0/0x2b0 [34865.766494] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34865.785231] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34865.790449] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34865.797574] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34865.804697] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34865.811830] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34865.818954] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34865.826078] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34865.834157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34865.839893] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34865.847017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34865.854142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34865.861267] Call Trace: [34865.863711] [34865.865723] ? rcu_dump_cpu_stacks+0xd1/0x180 [34865.870083] ? rcu_sched_clock_irq+0x633/0x1390 [34865.874615] ? lock_release+0xc5/0x260 [34865.878367] ? lock_is_held_type+0xaa/0x120 [34865.882546] ? lock_is_held_type+0xaa/0x120 [34865.886733] ? tick_sched_do_timer+0xa0/0xa0 [34865.891002] ? rcu_read_lock_sched_held+0x56/0x90 [34865.895712] ? tick_sched_do_timer+0xa0/0xa0 [34865.899982] ? update_process_times+0x5f/0x90 [34865.904332] ? tick_sched_handle.isra.16+0x1e/0x50 [34865.909116] ? tick_sched_timer+0x6f/0x80 [34865.913123] ? __hrtimer_run_queues+0x1d3/0x360 [34865.917661] ? hrtimer_interrupt+0x104/0x240 [34865.921926] ? __local_bh_enable+0x50/0x60 [34865.926030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34865.931251] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34865.936208] [34865.938306] [34865.940404] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34865.945717] ? lock_acquire+0xf0/0x2b0 [34865.949464] ? igrab+0x43/0x50 [34865.952538] ? lock_release+0xc5/0x260 [34865.956294] _raw_spin_lock+0x30/0x40 [34865.959956] ? igrab+0x1d/0x50 [34865.963005] igrab+0x1d/0x50 [34865.965887] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34865.971047] delayed_work+0x1d8/0x280 [ceph] [34865.975337] ? process_scheduled_works+0x22a/0x510 [34865.980123] process_scheduled_works+0x2a4/0x510 [34865.984746] worker_thread+0x188/0x340 [34865.988498] ? rescuer_thread+0x310/0x310 [34865.992507] kthread+0xf5/0x130 [34865.995645] ? kthread_complete_and_exit+0x20/0x20 [34866.000431] ret_from_fork+0x31/0x40 [34866.004011] ? kthread_complete_and_exit+0x20/0x20 [34866.008802] ret_from_fork_asm+0x11/0x20 [34866.012741] [34890.069549] watchdog: BUG: soft lockup - CPU#0 stuck for 21379s! [dbench:78683] [34890.076846] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34890.081549] watchdog: BUG: soft lockup - CPU#7 stuck for 30376s! [kworker/7:4:78808] [34890.132287] irq event stamp: 60535698 [34890.132288] hardirqs last enabled at (60535697): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34890.140018] Modules linked in: [34890.143676] hardirqs last disabled at (60535698): [] sysvec_apic_timer_interrupt+0xe/0x90 [34890.153833] ceph [34890.156884] softirqs last enabled at (60535678): [] __do_softirq+0x2be/0x371 [34890.166609] libceph dns_resolver [34890.168532] softirqs last disabled at (60535673): [] irq_exit_rcu+0xb0/0xd0 [34890.177227] fscache [34890.180535] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34890.189048] netfs veth [34890.191244] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34890.200193] nft_chain_nat [34890.202637] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [34890.210107] xt_MASQUERADE nf_nat [34890.212811] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [34890.218288] nf_conntrack [34890.221600] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34890.240336] nf_defrag_ipv6 nf_defrag_ipv4 [34890.242954] [34890.242955] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34890.248172] nft_compat nf_tables [34890.252262] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34890.253753] nfnetlink bridge [34890.260878] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34890.264189] stp llc [34890.271312] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34890.274276] xfs libcrc32c [34890.281401] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34890.283584] sunrpc intel_rapl_msr [34890.290708] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34890.293411] intel_rapl_common sb_edac [34890.300536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34890.303946] x86_pkg_temp_thermal [34890.312019] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34890.315764] coretemp kvm_intel [34890.321502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34890.324813] kvm irqbypass [34890.331947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34890.335083] i2c_i801 iTCO_wdt [34890.342207] Call Trace: [34890.344946] iTCO_vendor_support [34890.352069] [34890.355120] i2c_smbus crct10dif_pclmul [34890.357565] ? watchdog_timer_fn+0x1b4/0x220 [34890.360787] joydev crc32_pclmul [34890.362801] ? __hrtimer_run_queues+0x1d3/0x360 [34890.366629] ipmi_ssif ghash_clmulni_intel [34890.370897] ? hrtimer_interrupt+0x104/0x240 [34890.374117] mei_me mei [34890.378642] ? __local_bh_enable+0x50/0x60 [34890.382733] lpc_ich sha512_ssse3 [34890.387000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34890.389442] acpi_ipmi mfd_core [34890.393532] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34890.396843] wmi ipmi_si [34890.402060] [34890.405198] ipmi_devintf ipmi_msghandler [34890.410155] [34890.410157] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34890.412685] ioatdma acpi_pad [34890.414787] ? queued_spin_lock_slowpath+0x2d/0x2e0 [34890.418786] acpi_power_meter [34890.420886] do_raw_spin_lock+0xb5/0xc0 [34890.426187] ixgbe [34890.429153] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34890.434022] igb mdio [34890.436991] try_get_cap_refs+0x159/0x880 [ceph] [34890.440819] crc32c_intel nvme [34890.442834] __ceph_get_caps+0x4a1/0x760 [ceph] [34890.447613] ptp nvme_core [34890.449887] ? __file_remove_privs+0x89/0x110 [34890.454494] i2c_algo_bit pps_core [34890.457546] ? wake_bit_function+0x70/0x70 [34890.462068] dca fuse [34890.464775] ? ceph_write_iter+0x260/0x770 [ceph] [34890.469123] [34890.469123] irq event stamp: 199979622 [34890.472522] ceph_write_iter+0x260/0x770 [ceph] [34890.476612] hardirqs last enabled at (199979621): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34890.478887] ? lock_is_held_type+0xaa/0x120 [34890.483579] hardirqs last disabled at (199979622): [] sysvec_apic_timer_interrupt+0xe/0x90 [34890.485074] vfs_write+0x324/0x4f0 [34890.488816] softirqs last enabled at (199979620): [] __do_softirq+0x2be/0x371 [34890.493344] ksys_pwrite64+0x65/0xa0 [34890.503583] softirqs last disabled at (199979611): [] irq_exit_rcu+0xb0/0xd0 [34890.507761] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34890.517571] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34890.520969] do_syscall_64+0x38/0x80 [34890.529739] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34890.533311] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34890.541907] Workqueue: events delayed_work [ceph] [34890.547038] RIP: 0033:0x7f48f0eeb93b [34890.556415] [34890.556416] RIP: 0010:lock_acquire+0xf0/0x2b0 [34890.559987] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34890.567457] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34890.572502] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34890.577199] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34890.580770] ORIG_RAX: 0000000000000012 [34890.582260] [34890.582261] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34890.586611] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34890.605350] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34890.624085] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34890.629304] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34890.634522] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34890.638353] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34890.639843] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34890.646966] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34890.654091] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34890.661215] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34890.668345] [34890.675463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34890.727084] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34890.734207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34890.741349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34890.748474] Call Trace: [34890.750917] [34890.752930] ? watchdog_timer_fn+0x1b4/0x220 [34890.757209] ? __hrtimer_run_queues+0x1d3/0x360 [34890.761754] ? hrtimer_interrupt+0x104/0x240 [34890.766015] ? __local_bh_enable+0x50/0x60 [34890.770107] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34890.775322] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34890.780282] [34890.782377] [34890.784475] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34890.789784] ? lock_acquire+0xf0/0x2b0 [34890.793535] ? igrab+0x43/0x50 [34890.796585] ? lock_release+0xc5/0x260 [34890.800329] _raw_spin_lock+0x30/0x40 [34890.803983] ? igrab+0x1d/0x50 [34890.807035] igrab+0x1d/0x50 [34890.809912] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34890.815064] delayed_work+0x1d8/0x280 [ceph] [34890.819343] ? process_scheduled_works+0x22a/0x510 [34890.824127] process_scheduled_works+0x2a4/0x510 [34890.828743] worker_thread+0x188/0x340 [34890.832491] ? rescuer_thread+0x310/0x310 [34890.836494] kthread+0xf5/0x130 [34890.839630] ? kthread_complete_and_exit+0x20/0x20 [34890.844417] ret_from_fork+0x31/0x40 [34890.847987] ? kthread_complete_and_exit+0x20/0x20 [34890.852769] ret_from_fork_asm+0x11/0x20 [34890.856694] [34918.070014] watchdog: BUG: soft lockup - CPU#0 stuck for 21405s! [dbench:78683] [34918.077318] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34918.082014] watchdog: BUG: soft lockup - CPU#7 stuck for 30402s! [kworker/7:4:78808] [34918.132767] irq event stamp: 60593886 [34918.132768] hardirqs last enabled at (60593885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34918.140498] Modules linked in: [34918.144155] hardirqs last disabled at (60593886): [] sysvec_apic_timer_interrupt+0xe/0x90 [34918.154313] ceph [34918.157363] softirqs last enabled at (60593800): [] __do_softirq+0x2be/0x371 [34918.167086] libceph [34918.169012] softirqs last disabled at (60593795): [] irq_exit_rcu+0xb0/0xd0 [34918.177694] dns_resolver [34918.179880] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34918.188406] fscache [34918.191026] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34918.199970] netfs [34918.202152] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [34918.209624] veth [34918.211636] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [34918.217112] nft_chain_nat [34918.219035] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34918.237773] xt_MASQUERADE nf_nat [34918.240479] [34918.245695] nf_conntrack [34918.249007] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34918.250496] nf_defrag_ipv6 nf_defrag_ipv4 [34918.253113] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34918.260239] nft_compat nf_tables [34918.264329] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34918.271452] nfnetlink bridge [34918.274764] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34918.281888] stp llc [34918.284852] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34918.291976] xfs [34918.294161] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34918.301283] libcrc32c sunrpc [34918.303122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34918.311199] intel_rapl_msr intel_rapl_common [34918.314164] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34918.319900] sb_edac [34918.324251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34918.331376] x86_pkg_temp_thermal coretemp [34918.333560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34918.340692] kvm_intel kvm [34918.344782] Call Trace: [34918.351908] irqbypass [34918.354614] [34918.357054] i2c_i801 iTCO_wdt [34918.359415] ? watchdog_timer_fn+0x1b4/0x220 [34918.361424] iTCO_vendor_support i2c_smbus [34918.364476] ? __hrtimer_run_queues+0x1d3/0x360 [34918.368738] crct10dif_pclmul [34918.372832] ? hrtimer_interrupt+0x104/0x240 [34918.377352] joydev crc32_pclmul [34918.380316] ? __local_bh_enable+0x50/0x60 [34918.384581] ipmi_ssif [34918.387808] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34918.391895] ghash_clmulni_intel [34918.394254] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34918.399470] mei_me [34918.402702] [34918.407660] mei lpc_ich [34918.409758] [34918.409759] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34918.411855] sha512_ssse3 [34918.414410] ? queued_spin_lock_slowpath+0x2b/0x2e0 [34918.416500] acpi_ipmi [34918.421808] do_raw_spin_lock+0xb5/0xc0 [34918.424423] mfd_core wmi [34918.429297] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34918.431651] ipmi_si ipmi_devintf [34918.435484] try_get_cap_refs+0x159/0x880 [ceph] [34918.438098] ipmi_msghandler [34918.442889] __ceph_get_caps+0x4a1/0x760 [ceph] [34918.446192] ioatdma acpi_pad [34918.450808] ? __file_remove_privs+0x89/0x110 [34918.453694] acpi_power_meter [34918.458216] ? wake_bit_function+0x70/0x70 [34918.461177] ixgbe igb [34918.465531] ? ceph_write_iter+0x260/0x770 [ceph] [34918.468494] mdio crc32c_intel [34918.472585] ceph_write_iter+0x260/0x770 [ceph] [34918.474942] nvme ptp [34918.479644] ? lock_is_held_type+0xaa/0x120 [34918.482696] nvme_core i2c_algo_bit [34918.487224] vfs_write+0x324/0x4f0 [34918.489493] pps_core dca [34918.493674] ksys_pwrite64+0x65/0xa0 [34918.497173] fuse [34918.500570] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34918.503186] irq event stamp: 200200112 [34918.506760] do_syscall_64+0x38/0x80 [34918.508681] hardirqs last enabled at (200200111): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34918.513831] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34918.517572] hardirqs last disabled at (200200112): [] sysvec_apic_timer_interrupt+0xe/0x90 [34918.521145] RIP: 0033:0x7f48f0eeb93b [34918.531387] softirqs last enabled at (200200110): [] __do_softirq+0x2be/0x371 [34918.536450] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34918.546260] softirqs last disabled at (200200105): [] irq_exit_rcu+0xb0/0xd0 [34918.549832] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34918.558602] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34918.577339] ORIG_RAX: 0000000000000012 [34918.585937] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34918.591155] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34918.600533] Workqueue: events delayed_work [ceph] [34918.604362] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34918.611834] [34918.611835] RIP: 0010:lock_acquire+0xf0/0x2b0 [34918.618957] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34918.623656] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34918.630779] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34918.632270] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34918.636621] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34918.643746] [34918.643747] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34918.662488] [34918.669606] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34918.699872] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34918.706995] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34918.714120] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34918.721244] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34918.729322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34918.735058] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34918.742182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34918.749308] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34918.756430] Call Trace: [34918.758876] [34918.760889] ? watchdog_timer_fn+0x1b4/0x220 [34918.765153] ? __hrtimer_run_queues+0x1d3/0x360 [34918.769678] ? hrtimer_interrupt+0x104/0x240 [34918.773954] ? __local_bh_enable+0x50/0x60 [34918.778048] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34918.783274] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34918.788229] [34918.790327] [34918.792426] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34918.797734] ? lock_acquire+0xf0/0x2b0 [34918.801483] ? igrab+0x43/0x50 [34918.804534] ? lock_release+0xc5/0x260 [34918.808278] _raw_spin_lock+0x30/0x40 [34918.811934] ? igrab+0x1d/0x50 [34918.814984] igrab+0x1d/0x50 [34918.817864] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34918.823012] delayed_work+0x1d8/0x280 [ceph] [34918.827295] ? process_scheduled_works+0x22a/0x510 [34918.832085] process_scheduled_works+0x2a4/0x510 [34918.836715] worker_thread+0x188/0x340 [34918.840467] ? rescuer_thread+0x310/0x310 [34918.844468] kthread+0xf5/0x130 [34918.847606] ? kthread_complete_and_exit+0x20/0x20 [34918.852392] ret_from_fork+0x31/0x40 [34918.855988] ? kthread_complete_and_exit+0x20/0x20 [34918.860772] ret_from_fork_asm+0x11/0x20 [34918.864715] [34946.070489] watchdog: BUG: soft lockup - CPU#0 stuck for 21432s! [dbench:78683] [34946.077797] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34946.082489] watchdog: BUG: soft lockup - CPU#7 stuck for 30429s! [kworker/7:4:78808] [34946.133236] irq event stamp: 60652026 [34946.133237] hardirqs last enabled at (60652025): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34946.140967] Modules linked in: [34946.144625] hardirqs last disabled at (60652026): [] sysvec_apic_timer_interrupt+0xe/0x90 [34946.154783] ceph [34946.157834] softirqs last enabled at (60651924): [] __do_softirq+0x2be/0x371 [34946.167557] libceph [34946.169480] softirqs last disabled at (60651919): [] irq_exit_rcu+0xb0/0xd0 [34946.178173] dns_resolver fscache [34946.180358] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34946.188883] netfs [34946.192197] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34946.201140] veth [34946.203168] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [34946.210639] nft_chain_nat [34946.212564] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [34946.218041] xt_MASQUERADE nf_nat [34946.220746] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34946.239484] nf_conntrack [34946.242793] [34946.242794] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34946.248012] nf_defrag_ipv6 nf_defrag_ipv4 [34946.250629] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34946.252120] nft_compat nf_tables [34946.259244] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34946.263334] nfnetlink bridge [34946.270458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34946.273769] stp llc [34946.280892] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34946.283857] xfs libcrc32c [34946.290983] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34946.293166] sunrpc intel_rapl_msr [34946.300289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34946.302993] intel_rapl_common sb_edac [34946.311071] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34946.314468] x86_pkg_temp_thermal coretemp [34946.320206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34946.323949] kvm_intel kvm [34946.331074] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34946.335168] irqbypass [34946.342290] Call Trace: [34946.342291] [34946.344993] i2c_i801 iTCO_wdt [34946.352119] ? watchdog_timer_fn+0x1b4/0x220 [34946.354476] iTCO_vendor_support i2c_smbus [34946.356920] ? __hrtimer_run_queues+0x1d3/0x360 [34946.358929] crct10dif_pclmul [34946.361982] ? hrtimer_interrupt+0x104/0x240 [34946.366244] joydev crc32_pclmul [34946.370335] ? __local_bh_enable+0x50/0x60 [34946.374860] ipmi_ssif [34946.377843] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34946.382104] ghash_clmulni_intel mei_me [34946.385329] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34946.389420] mei lpc_ich [34946.391776] [34946.396995] sha512_ssse3 acpi_ipmi [34946.400825] [34946.400826] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34946.405783] mfd_core wmi [34946.408316] ? queued_spin_lock_slowpath+0x2b/0x2e0 [34946.410410] ipmi_si ipmi_devintf [34946.413896] do_raw_spin_lock+0xb5/0xc0 [34946.415992] ipmi_msghandler ioatdma [34946.421296] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34946.423914] acpi_pad acpi_power_meter [34946.428787] try_get_cap_refs+0x159/0x880 [ceph] [34946.432096] ixgbe igb [34946.435930] __ceph_get_caps+0x4a1/0x760 [ceph] [34946.439496] mdio crc32c_intel [34946.444284] ? __file_remove_privs+0x89/0x110 [34946.448023] nvme ptp [34946.452637] ? wake_bit_function+0x70/0x70 [34946.454992] nvme_core i2c_algo_bit [34946.459518] ? ceph_write_iter+0x260/0x770 [ceph] [34946.462567] pps_core [34946.466919] ceph_write_iter+0x260/0x770 [ceph] [34946.469189] dca [34946.473282] ? lock_is_held_type+0xaa/0x120 [34946.476763] fuse [34946.481464] vfs_write+0x324/0x4f0 [34946.483732] irq event stamp: 200420782 [34946.488260] ksys_pwrite64+0x65/0xa0 [34946.490093] hardirqs last enabled at (200420781): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34946.494270] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34946.496194] hardirqs last disabled at (200420782): [] sysvec_apic_timer_interrupt+0xe/0x90 [34946.499594] do_syscall_64+0x38/0x80 [34946.503335] softirqs last enabled at (200420780): [] __do_softirq+0x2be/0x371 [34946.506908] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34946.517150] softirqs last disabled at (200420775): [] irq_exit_rcu+0xb0/0xd0 [34946.522281] RIP: 0033:0x7f48f0eeb93b [34946.532094] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34946.535663] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34946.544435] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34946.549479] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34946.558076] Workqueue: events delayed_work [ceph] [34946.561645] ORIG_RAX: 0000000000000012 [34946.571023] [34946.571024] RIP: 0010:lock_acquire+0xf0/0x2b0 [34946.589762] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34946.597232] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [34946.602450] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34946.607148] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [34946.610977] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34946.612468] [34946.612469] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [34946.616820] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34946.623943] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [34946.642680] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34946.649806] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [34946.655029] [34946.662148] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [34946.708574] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [34946.715700] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34946.723777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34946.729515] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34946.736639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34946.743763] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34946.750894] Call Trace: [34946.753339] [34946.755351] ? watchdog_timer_fn+0x1b4/0x220 [34946.759616] ? __hrtimer_run_queues+0x1d3/0x360 [34946.764144] ? hrtimer_interrupt+0x104/0x240 [34946.768412] ? __local_bh_enable+0x50/0x60 [34946.772505] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34946.777729] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34946.782685] [34946.784783] [34946.786898] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34946.792205] ? lock_acquire+0xf0/0x2b0 [34946.795955] ? igrab+0x43/0x50 [34946.799005] ? lock_release+0xc5/0x260 [34946.802752] _raw_spin_lock+0x30/0x40 [34946.806415] ? igrab+0x1d/0x50 [34946.809467] igrab+0x1d/0x50 [34946.812345] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34946.817495] delayed_work+0x1d8/0x280 [ceph] [34946.821774] ? process_scheduled_works+0x22a/0x510 [34946.826558] process_scheduled_works+0x2a4/0x510 [34946.831181] worker_thread+0x188/0x340 [34946.834929] ? rescuer_thread+0x310/0x310 [34946.838935] kthread+0xf5/0x130 [34946.842071] ? kthread_complete_and_exit+0x20/0x20 [34946.846857] ret_from_fork+0x31/0x40 [34946.850434] ? kthread_complete_and_exit+0x20/0x20 [34946.855219] ret_from_fork_asm+0x11/0x20 [34946.859142] [34974.070968] watchdog: BUG: soft lockup - CPU#0 stuck for 21458s! [dbench:78683] [34974.078276] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [34974.082968] watchdog: BUG: soft lockup - CPU#7 stuck for 30455s! [kworker/7:4:78808] [34974.133726] irq event stamp: 60710156 [34974.141456] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [34974.145112] hardirqs last enabled at (60710155): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34974.153285] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [34974.163443] hardirqs last disabled at (60710156): [] sysvec_apic_timer_interrupt+0xe/0x90 [34974.163446] nft_compat [34974.163448] softirqs last enabled at (60710034): [] __do_softirq+0x2be/0x371 [34974.170490] nf_tables nfnetlink [34974.180213] softirqs last disabled at (60710029): [] irq_exit_rcu+0xb0/0xd0 [34974.182658] bridge [34974.191361] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34974.194582] stp llc [34974.203095] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34974.205190] xfs libcrc32c [34974.214136] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [34974.216319] sunrpc intel_rapl_msr [34974.223790] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [34974.226493] intel_rapl_common sb_edac [34974.231971] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [34974.235368] x86_pkg_temp_thermal coretemp [34974.254106] [34974.257850] kvm_intel kvm [34974.263067] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [34974.267158] irqbypass i2c_i801 [34974.268649] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [34974.271365] iTCO_wdt [34974.278485] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [34974.281622] iTCO_vendor_support i2c_smbus [34974.288765] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [34974.291036] crct10dif_pclmul [34974.298159] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [34974.302250] joydev crc32_pclmul [34974.309374] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [34974.312338] ipmi_ssif [34974.319463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34974.322686] ghash_clmulni_intel mei_me [34974.330764] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [34974.333122] mei lpc_ich [34974.338859] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34974.342691] sha512_ssse3 [34974.349815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34974.352362] acpi_ipmi [34974.359485] Call Trace: [34974.359487] [34974.362102] mfd_core wmi [34974.369230] ? watchdog_timer_fn+0x1b4/0x220 [34974.371584] ipmi_si ipmi_devintf [34974.374031] ? __hrtimer_run_queues+0x1d3/0x360 [34974.376040] ipmi_msghandler [34974.378659] ? hrtimer_interrupt+0x104/0x240 [34974.382920] ioatdma [34974.386233] ? __local_bh_enable+0x50/0x60 [34974.390756] acpi_pad [34974.393652] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34974.397917] acpi_power_meter [34974.400099] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34974.404189] ixgbe igb mdio [34974.406461] [34974.411679] crc32c_intel [34974.414658] [34974.419616] nvme ptp [34974.422409] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34974.424504] nvme_core i2c_algo_bit [34974.427127] ? queued_spin_lock_slowpath+0x18/0x2e0 [34974.429220] pps_core [34974.431491] do_raw_spin_lock+0xb5/0xc0 [34974.436793] dca fuse [34974.440280] __cap_delay_requeue+0xa7/0x1d0 [ceph] [34974.445149] [34974.445150] irq event stamp: 200641438 [34974.447425] try_get_cap_refs+0x159/0x880 [ceph] [34974.451250] hardirqs last enabled at (200641437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34974.453527] __ceph_get_caps+0x4a1/0x760 [ceph] [34974.458305] hardirqs last disabled at (200641438): [] sysvec_apic_timer_interrupt+0xe/0x90 [34974.459800] ? __file_remove_privs+0x89/0x110 [34974.463540] softirqs last enabled at (200641436): [] __do_softirq+0x2be/0x371 [34974.468151] ? wake_bit_function+0x70/0x70 [34974.478396] softirqs last disabled at (200641431): [] irq_exit_rcu+0xb0/0xd0 [34974.482922] ? ceph_write_iter+0x260/0x770 [ceph] [34974.492730] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [34974.497081] ceph_write_iter+0x260/0x770 [ceph] [34974.505851] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [34974.505853] Workqueue: events delayed_work [ceph] [34974.509946] ? lock_is_held_type+0xaa/0x120 [34974.518539] [34974.523239] vfs_write+0x324/0x4f0 [34974.532615] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [34974.537143] ksys_pwrite64+0x65/0xa0 [34974.544608] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [34974.549308] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [34974.553484] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [34974.554977] do_syscall_64+0x38/0x80 [34974.558373] [34974.558374] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [34974.562984] entry_SYSCALL_64_after_hwframe+0x63/0xcd [34974.566553] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [34974.585293] RIP: 0033:0x7f48f0eeb93b [34974.590421] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [34974.595652] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [34974.599219] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [34974.600709] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [34974.607833] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [34974.612879] ORIG_RAX: 0000000000000012 [34974.620001] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [34974.623574] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [34974.630697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34974.649434] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [34974.656559] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [34974.661776] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [34974.668900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [34974.672730] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [34974.680809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [34974.687932] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [34974.693671] Call Trace: [34974.693673] [34974.700798] [34974.707920] ? watchdog_timer_fn+0x1b4/0x220 [34974.754444] ? __hrtimer_run_queues+0x1d3/0x360 [34974.758970] ? hrtimer_interrupt+0x104/0x240 [34974.763239] ? __local_bh_enable+0x50/0x60 [34974.767333] ? __sysvec_apic_timer_interrupt+0x61/0x160 [34974.772558] ? sysvec_apic_timer_interrupt+0x6d/0x90 [34974.777515] [34974.779612] [34974.781708] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [34974.787016] ? do_raw_spin_lock+0x42/0xc0 [34974.791027] igrab+0x1d/0x50 [34974.793905] ceph_check_delayed_caps+0xb7/0x270 [ceph] [34974.799055] delayed_work+0x1d8/0x280 [ceph] [34974.803336] ? process_scheduled_works+0x22a/0x510 [34974.808126] process_scheduled_works+0x2a4/0x510 [34974.812740] worker_thread+0x188/0x340 [34974.816489] ? rescuer_thread+0x310/0x310 [34974.820492] kthread+0xf5/0x130 [34974.823654] ? kthread_complete_and_exit+0x20/0x20 [34974.828442] ret_from_fork+0x31/0x40 [34974.832021] ? kthread_complete_and_exit+0x20/0x20 [34974.836812] ret_from_fork_asm+0x11/0x20 [34974.840736] [35002.071446] watchdog: BUG: soft lockup - CPU#0 stuck for 21484s! [dbench:78683] [35002.078743] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35002.083446] watchdog: BUG: soft lockup - CPU#7 stuck for 30481s! [kworker/7:4:78808] [35002.134187] irq event stamp: 60768218 [35002.134188] hardirqs last enabled at (60768217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35002.141917] Modules linked in: [35002.145573] hardirqs last disabled at (60768218): [] sysvec_apic_timer_interrupt+0xe/0x90 [35002.155731] ceph [35002.158781] softirqs last enabled at (60768192): [] __do_softirq+0x2be/0x371 [35002.168506] libceph [35002.170429] softirqs last disabled at (60768187): [] irq_exit_rcu+0xb0/0xd0 [35002.179121] dns_resolver [35002.181307] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35002.189817] fscache [35002.192453] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35002.201397] netfs veth [35002.203582] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35002.211052] nft_chain_nat [35002.213496] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35002.218972] xt_MASQUERADE nf_nat [35002.221676] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35002.240414] nf_conntrack [35002.243726] [35002.243727] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35002.248942] nf_defrag_ipv6 nf_defrag_ipv4 [35002.251560] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35002.253050] nft_compat nf_tables [35002.260174] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35002.264266] nfnetlink [35002.271390] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35002.274700] bridge stp [35002.281843] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35002.284200] llc xfs [35002.291324] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35002.293768] libcrc32c sunrpc [35002.300891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35002.303076] intel_rapl_msr intel_rapl_common [35002.311155] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35002.314124] sb_edac x86_pkg_temp_thermal [35002.319863] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35002.324214] coretemp [35002.331337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35002.335342] kvm_intel kvm [35002.342468] Call Trace: [35002.342469] [35002.344738] irqbypass [35002.351863] ? watchdog_timer_fn+0x1b4/0x220 [35002.354565] i2c_i801 iTCO_wdt [35002.357011] ? __hrtimer_run_queues+0x1d3/0x360 [35002.359020] iTCO_vendor_support [35002.361380] ? hrtimer_interrupt+0x104/0x240 [35002.365641] i2c_smbus crct10dif_pclmul [35002.368693] ? __local_bh_enable+0x50/0x60 [35002.373217] joydev [35002.376442] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35002.380704] crc32_pclmul ipmi_ssif [35002.384536] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35002.388626] ghash_clmulni_intel [35002.390725] [35002.395941] mei_me mei [35002.399425] [35002.399426] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35002.404382] lpc_ich [35002.407610] ? queued_spin_lock_slowpath+0x16/0x2e0 [35002.409703] sha512_ssse3 acpi_ipmi [35002.412151] do_raw_spin_lock+0xb5/0xc0 [35002.414245] mfd_core [35002.419549] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35002.421732] wmi [35002.426608] try_get_cap_refs+0x159/0x880 [ceph] [35002.430089] ipmi_si ipmi_devintf [35002.433925] __ceph_get_caps+0x4a1/0x760 [ceph] [35002.436189] ipmi_msghandler [35002.440980] ? __file_remove_privs+0x89/0x110 [35002.442810] ioatdma acpi_pad [35002.447423] ? wake_bit_function+0x70/0x70 [35002.450732] acpi_power_meter ixgbe [35002.455260] ? ceph_write_iter+0x260/0x770 [ceph] [35002.458142] igb mdio [35002.462494] ceph_write_iter+0x260/0x770 [ceph] [35002.465457] crc32c_intel [35002.469553] ? lock_is_held_type+0xaa/0x120 [35002.473031] nvme ptp [35002.477732] vfs_write+0x324/0x4f0 [35002.480001] nvme_core i2c_algo_bit [35002.484529] ksys_pwrite64+0x65/0xa0 [35002.487143] pps_core dca [35002.491320] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35002.493590] fuse [35002.496988] do_syscall_64+0x38/0x80 [35002.500471] irq event stamp: 200862338 [35002.504042] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35002.506659] hardirqs last enabled at (200862337): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35002.511791] RIP: 0033:0x7f48f0eeb93b [35002.513713] hardirqs last disabled at (200862338): [] sysvec_apic_timer_interrupt+0xe/0x90 [35002.517286] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35002.521029] softirqs last enabled at (200862336): [] __do_softirq+0x2be/0x371 [35002.526074] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35002.536316] softirqs last disabled at (200862331): [] irq_exit_rcu+0xb0/0xd0 [35002.539888] ORIG_RAX: 0000000000000012 [35002.549700] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35002.568437] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35002.577207] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35002.582426] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35002.591023] Workqueue: events delayed_work [ceph] [35002.594853] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35002.604231] [35002.611354] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35002.618843] RIP: 0010:lock_acquire+0xf0/0x2b0 [35002.625966] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35002.630664] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35002.637795] [35002.639279] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35002.684019] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35002.691141] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35002.698267] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35002.705391] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35002.712514] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35002.719640] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35002.727716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35002.733454] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35002.740577] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35002.747702] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35002.754826] Call Trace: [35002.757288] [35002.759298] ? watchdog_timer_fn+0x1b4/0x220 [35002.763563] ? __hrtimer_run_queues+0x1d3/0x360 [35002.768089] ? hrtimer_interrupt+0x104/0x240 [35002.772359] ? __local_bh_enable+0x50/0x60 [35002.776452] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35002.781676] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35002.786633] [35002.788731] [35002.790846] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35002.796153] ? lock_acquire+0xf0/0x2b0 [35002.799903] ? igrab+0x43/0x50 [35002.802954] ? lock_release+0xc5/0x260 [35002.806700] _raw_spin_lock+0x30/0x40 [35002.810363] ? igrab+0x1d/0x50 [35002.813413] igrab+0x1d/0x50 [35002.816291] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35002.821442] delayed_work+0x1d8/0x280 [ceph] [35002.825724] ? process_scheduled_works+0x22a/0x510 [35002.830512] process_scheduled_works+0x2a4/0x510 [35002.835143] worker_thread+0x188/0x340 [35002.838896] ? rescuer_thread+0x310/0x310 [35002.842897] kthread+0xf5/0x130 [35002.846035] ? kthread_complete_and_exit+0x20/0x20 [35002.850820] ret_from_fork+0x31/0x40 [35002.854391] ? kthread_complete_and_exit+0x20/0x20 [35002.859174] ret_from_fork_asm+0x11/0x20 [35002.863099] [35030.071922] watchdog: BUG: soft lockup - CPU#0 stuck for 21510s! [dbench:78683] [35030.079224] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35030.083922] watchdog: BUG: soft lockup - CPU#7 stuck for 30507s! [kworker/7:4:78808] [35030.134674] irq event stamp: 60826462 [35030.142403] Modules linked in: ceph libceph dns_resolver fscache netfs [35030.146062] hardirqs last enabled at (60826461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35030.146067] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [35030.162752] hardirqs last disabled at (60826462): [] sysvec_apic_timer_interrupt+0xe/0x90 [35030.162755] nf_defrag_ipv4 [35030.162757] softirqs last enabled at (60826426): [] __do_softirq+0x2be/0x371 [35030.170137] nft_compat nf_tables [35030.179862] softirqs last disabled at (60826421): [] irq_exit_rcu+0xb0/0xd0 [35030.182653] nfnetlink bridge [35030.191338] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35030.194649] stp [35030.203159] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35030.206121] llc [35030.215067] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35030.216904] xfs libcrc32c [35030.224374] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35030.226213] sunrpc intel_rapl_msr [35030.231688] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35030.234394] intel_rapl_common sb_edac [35030.253132] [35030.256528] x86_pkg_temp_thermal coretemp [35030.261747] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35030.265490] kvm_intel kvm [35030.266980] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35030.271070] irqbypass i2c_i801 [35030.278195] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35030.280900] iTCO_wdt iTCO_vendor_support [35030.288023] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35030.291162] i2c_smbus crct10dif_pclmul [35030.298285] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35030.302289] joydev crc32_pclmul [35030.309415] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35030.313244] ipmi_ssif ghash_clmulni_intel [35030.320368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35030.323592] mei_me mei [35030.331688] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35030.335779] lpc_ich sha512_ssse3 [35030.341515] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35030.343960] acpi_ipmi mfd_core [35030.351083] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35030.354393] wmi ipmi_si [35030.361518] Call Trace: [35030.364656] ipmi_devintf ipmi_msghandler [35030.371783] [35030.374321] ioatdma acpi_pad [35030.376767] ? watchdog_timer_fn+0x1b4/0x220 [35030.380767] acpi_power_meter ixgbe [35030.382779] ? __hrtimer_run_queues+0x1d3/0x360 [35030.385743] igb mdio [35030.390008] ? hrtimer_interrupt+0x104/0x240 [35030.393491] crc32c_intel [35030.398015] ? __local_bh_enable+0x50/0x60 [35030.400286] nvme ptp [35030.404552] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35030.407166] nvme_core i2c_algo_bit [35030.411258] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35030.413528] pps_core dca [35030.418746] [35030.422230] fuse [35030.427187] [35030.429803] [35030.429804] irq event stamp: 201082976 [35030.431901] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35030.433825] hardirqs last enabled at (201082975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35030.435927] ? queued_spin_lock_slowpath+0x16/0x2e0 [35030.437415] hardirqs last disabled at (201082976): [] sysvec_apic_timer_interrupt+0xe/0x90 [35030.441160] do_raw_spin_lock+0xb5/0xc0 [35030.446462] softirqs last enabled at (201082974): [] __do_softirq+0x2be/0x371 [35030.456709] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35030.461576] softirqs last disabled at (201082969): [] irq_exit_rcu+0xb0/0xd0 [35030.471408] try_get_cap_refs+0x159/0x880 [ceph] [35030.475235] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35030.484011] __ceph_get_caps+0x4a1/0x760 [ceph] [35030.488790] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35030.497392] ? __file_remove_privs+0x89/0x110 [35030.502000] Workqueue: events delayed_work [ceph] [35030.511377] ? wake_bit_function+0x70/0x70 [35030.515901] [35030.523374] ? ceph_write_iter+0x260/0x770 [ceph] [35030.527722] RIP: 0010:lock_acquire+0x17b/0x2b0 [35030.532419] ceph_write_iter+0x260/0x770 [ceph] [35030.536510] Code: c8 89 d1 89 f2 4c 89 e6 e8 62 5b ff ff 59 5e 4c 8b 14 24 65 ff 0d 5d e6 eb 7e 0f 85 c8 fe ff ff 0f 1f 44 00 00 e9 be fe ff ff <4c> 89 14 24 e8 9c 8a 02 00 4c 8b 14 24 84 c0 0f 85 c0 fe ff ff 0f [35030.538006] ? lock_is_held_type+0xaa/0x120 [35030.542699] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [35030.547138] vfs_write+0x324/0x4f0 [35030.551660] [35030.551660] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [35030.570402] ksys_pwrite64+0x65/0xa0 [35030.574575] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [35030.579792] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35030.583190] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [35030.584681] do_syscall_64+0x38/0x80 [35030.591806] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [35030.595378] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35030.602498] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35030.607631] RIP: 0033:0x7f48f0eeb93b [35030.614753] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35030.618327] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35030.625450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35030.630494] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35030.637618] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35030.641189] ORIG_RAX: 0000000000000012 [35030.649266] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35030.668004] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35030.673742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35030.678958] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35030.686083] Call Trace: [35030.686084] [35030.689913] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35030.697038] ? watchdog_timer_fn+0x1b4/0x220 [35030.704161] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35030.711288] ? __hrtimer_run_queues+0x1d3/0x360 [35030.718409] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35030.720857] ? hrtimer_interrupt+0x104/0x240 [35030.722869] [35030.729988] ? __local_bh_enable+0x50/0x60 [35030.763567] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35030.768790] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35030.773748] [35030.775844] [35030.777943] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35030.783250] ? lock_acquire+0x17b/0x2b0 [35030.787088] ? igrab+0x43/0x50 [35030.790147] ? lock_release+0xc5/0x260 [35030.793891] _raw_spin_lock+0x30/0x40 [35030.797548] ? igrab+0x1d/0x50 [35030.800610] igrab+0x1d/0x50 [35030.803495] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35030.808644] delayed_work+0x1d8/0x280 [ceph] [35030.812926] ? process_scheduled_works+0x22a/0x510 [35030.817715] process_scheduled_works+0x2a4/0x510 [35030.822332] worker_thread+0x188/0x340 [35030.826081] ? rescuer_thread+0x310/0x310 [35030.830091] kthread+0xf5/0x130 [35030.833230] ? kthread_complete_and_exit+0x20/0x20 [35030.838014] ret_from_fork+0x31/0x40 [35030.841609] ? kthread_complete_and_exit+0x20/0x20 [35030.846395] ret_from_fork_asm+0x11/0x20 [35030.850334] [35058.072398] watchdog: BUG: soft lockup - CPU#0 stuck for 21536s! [dbench:78683] [35058.079704] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35058.084398] watchdog: BUG: soft lockup - CPU#7 stuck for 30533s! [kworker/7:4:78808] [35058.135151] irq event stamp: 60884510 [35058.135152] hardirqs last enabled at (60884509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35058.142883] Modules linked in: [35058.146541] hardirqs last disabled at (60884510): [] sysvec_apic_timer_interrupt+0xe/0x90 [35058.156699] ceph [35058.159749] softirqs last enabled at (60884432): [] __do_softirq+0x2be/0x371 [35058.169472] libceph [35058.171397] softirqs last disabled at (60884427): [] irq_exit_rcu+0xb0/0xd0 [35058.180080] dns_resolver fscache [35058.182266] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35058.190793] netfs [35058.194105] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35058.203047] veth nft_chain_nat [35058.205073] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35058.212547] xt_MASQUERADE nf_nat [35058.215685] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35058.221160] nf_conntrack nf_defrag_ipv6 [35058.224472] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35058.243210] nf_defrag_ipv4 nft_compat [35058.247128] [35058.247129] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35058.252345] nf_tables nfnetlink [35058.256090] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35058.257580] bridge [35058.264704] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35058.267927] stp llc [35058.275053] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35058.277149] xfs libcrc32c [35058.284272] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35058.286456] sunrpc intel_rapl_msr [35058.293580] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35058.296286] intel_rapl_common [35058.303410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35058.306807] sb_edac x86_pkg_temp_thermal [35058.314884] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35058.317935] coretemp kvm_intel [35058.323672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35058.327678] kvm irqbypass [35058.334801] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35058.337937] i2c_i801 iTCO_wdt [35058.345062] Call Trace: [35058.345063] [35058.347797] iTCO_vendor_support i2c_smbus [35058.354927] ? watchdog_timer_fn+0x1b4/0x220 [35058.357976] crct10dif_pclmul [35058.360420] ? __hrtimer_run_queues+0x1d3/0x360 [35058.362430] joydev crc32_pclmul [35058.366525] ? hrtimer_interrupt+0x104/0x240 [35058.370797] ipmi_ssif ghash_clmulni_intel [35058.373758] ? __local_bh_enable+0x50/0x60 [35058.378282] mei_me mei [35058.381509] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35058.385771] lpc_ich [35058.389879] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35058.393969] sha512_ssse3 acpi_ipmi [35058.396413] [35058.401632] mfd_core wmi [35058.403815] [35058.408773] ipmi_si ipmi_devintf [35058.412276] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35058.414370] ipmi_msghandler ioatdma [35058.416993] ? queued_spin_lock_slowpath+0x18/0x2e0 [35058.419095] acpi_pad acpi_power_meter [35058.422407] do_raw_spin_lock+0xb5/0xc0 [35058.427708] ixgbe igb mdio [35058.431283] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35058.436150] crc32c_intel nvme [35058.439896] try_get_cap_refs+0x159/0x880 [ceph] [35058.443725] ptp nvme_core [35058.446520] __ceph_get_caps+0x4a1/0x760 [ceph] [35058.451299] i2c_algo_bit pps_core [35058.454357] ? __file_remove_privs+0x89/0x110 [35058.458962] dca fuse [35058.461667] ? wake_bit_function+0x70/0x70 [35058.466189] [35058.466190] irq event stamp: 201303768 [35058.469589] ? ceph_write_iter+0x260/0x770 [ceph] [35058.473939] hardirqs last enabled at (201303767): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35058.476208] ceph_write_iter+0x260/0x770 [ceph] [35058.480299] hardirqs last disabled at (201303768): [] sysvec_apic_timer_interrupt+0xe/0x90 [35058.481801] ? lock_is_held_type+0xaa/0x120 [35058.485543] softirqs last enabled at (201303766): [] __do_softirq+0x2be/0x371 [35058.490243] vfs_write+0x324/0x4f0 [35058.500484] softirqs last disabled at (201303761): [] irq_exit_rcu+0xb0/0xd0 [35058.505014] ksys_pwrite64+0x65/0xa0 [35058.514819] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35058.518998] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35058.527767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35058.531168] do_syscall_64+0x38/0x80 [35058.539762] Workqueue: events delayed_work [ceph] [35058.543334] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35058.552710] [35058.552711] RIP: 0010:lock_acquire+0xf0/0x2b0 [35058.557842] RIP: 0033:0x7f48f0eeb93b [35058.565312] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35058.568884] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35058.573582] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35058.578626] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35058.580116] [35058.580116] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35058.584467] ORIG_RAX: 0000000000000012 [35058.588037] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35058.606773] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35058.625530] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35058.630746] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35058.635965] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35058.637456] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35058.644579] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35058.648411] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35058.655533] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35058.662658] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35058.669799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35058.676930] [35058.684048] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35058.735667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35058.742801] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35058.749925] Call Trace: [35058.752369] [35058.754380] ? watchdog_timer_fn+0x1b4/0x220 [35058.758646] ? __hrtimer_run_queues+0x1d3/0x360 [35058.763172] ? hrtimer_interrupt+0x104/0x240 [35058.767441] ? __local_bh_enable+0x50/0x60 [35058.771534] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35058.776757] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35058.781714] [35058.783812] [35058.785910] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35058.791217] ? lock_acquire+0xf0/0x2b0 [35058.794968] ? igrab+0x43/0x50 [35058.798018] ? lock_release+0xc5/0x260 [35058.801763] _raw_spin_lock+0x30/0x40 [35058.805426] ? igrab+0x1d/0x50 [35058.808479] igrab+0x1d/0x50 [35058.811355] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35058.816506] delayed_work+0x1d8/0x280 [ceph] [35058.820805] ? process_scheduled_works+0x22a/0x510 [35058.825595] process_scheduled_works+0x2a4/0x510 [35058.830212] worker_thread+0x188/0x340 [35058.833960] ? rescuer_thread+0x310/0x310 [35058.837962] kthread+0xf5/0x130 [35058.841099] ? kthread_complete_and_exit+0x20/0x20 [35058.845884] ret_from_fork+0x31/0x40 [35058.849463] ? kthread_complete_and_exit+0x20/0x20 [35058.854248] ret_from_fork_asm+0x11/0x20 [35058.858182] [35061.019814] rcu: INFO: rcu_preempt self-detected stall on CPU [35061.025575] rcu: 0-....: (22602103 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11315692 [35061.036087] rcu: (t=23124001 jiffies g=1309609 q=646518 ncpus=8) [35061.042172] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35061.051114] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35061.058586] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [35061.064062] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [35061.082802] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35061.088017] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35061.095142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35061.102267] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35061.109389] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35061.116513] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35061.123639] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35061.131716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35061.137452] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35061.144579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35061.151702] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35061.158826] Call Trace: [35061.161269] [35061.163282] ? rcu_dump_cpu_stacks+0xd1/0x180 [35061.167640] ? rcu_sched_clock_irq+0x633/0x1390 [35061.172166] ? lock_is_held_type+0xaa/0x120 [35061.176344] ? lock_is_held_type+0xaa/0x120 [35061.180521] ? tick_sched_do_timer+0xa0/0xa0 [35061.184793] ? rcu_read_lock_sched_held+0x56/0x90 [35061.189491] ? tick_sched_do_timer+0xa0/0xa0 [35061.193754] ? update_process_times+0x5f/0x90 [35061.198105] ? tick_sched_handle.isra.16+0x1e/0x50 [35061.202889] ? tick_sched_timer+0x6f/0x80 [35061.206892] ? __hrtimer_run_queues+0x1d3/0x360 [35061.211419] ? hrtimer_interrupt+0x104/0x240 [35061.215689] ? __local_bh_enable+0x50/0x60 [35061.219782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35061.225006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35061.229963] [35061.232059] [35061.234159] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35061.239467] ? queued_spin_lock_slowpath+0x2b/0x2e0 [35061.244344] do_raw_spin_lock+0xb5/0xc0 [35061.248181] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35061.252986] try_get_cap_refs+0x159/0x880 [ceph] [35061.257615] __ceph_get_caps+0x4a1/0x760 [ceph] [35061.262156] ? __file_remove_privs+0x89/0x110 [35061.266512] ? wake_bit_function+0x70/0x70 [35061.270603] ? ceph_write_iter+0x260/0x770 [ceph] [35061.275317] ceph_write_iter+0x260/0x770 [ceph] [35061.279853] ? lock_is_held_type+0xaa/0x120 [35061.284037] vfs_write+0x324/0x4f0 [35061.287438] ksys_pwrite64+0x65/0xa0 [35061.291013] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35061.296144] do_syscall_64+0x38/0x80 [35061.299714] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35061.304759] RIP: 0033:0x7f48f0eeb93b [35061.308328] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35061.327066] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [35061.334623] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35061.341749] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35061.348872] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35061.355995] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35061.363119] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35061.370250] [35061.372438] Sending NMI from CPU 0 to CPUs 7: [35061.376789] NMI backtrace for cpu 7 [35061.376791] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35061.376792] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35061.376793] Workqueue: events delayed_work [ceph] [35061.376808] RIP: 0010:lock_acquire+0x85/0x2b0 [35061.376810] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [35061.376812] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [35061.376813] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [35061.376814] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [35061.376814] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [35061.376815] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [35061.376815] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35061.376816] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35061.376817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35061.376818] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35061.376819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35061.376820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35061.376821] Call Trace: [35061.376822] [35061.376823] ? nmi_cpu_backtrace+0xad/0x120 [35061.376827] ? nmi_cpu_backtrace_handler+0x11/0x20 [35061.376829] ? nmi_handle+0xd4/0x260 [35061.376833] ? default_do_nmi+0x49/0xf0 [35061.376835] ? exc_nmi+0x11e/0x150 [35061.376836] ? end_repeat_nmi+0x16/0x67 [35061.376840] ? lock_acquire+0x85/0x2b0 [35061.376842] ? lock_acquire+0x85/0x2b0 [35061.376844] ? lock_acquire+0x85/0x2b0 [35061.376845] [35061.376846] [35061.376846] ? igrab+0x43/0x50 [35061.376848] ? lock_release+0xc5/0x260 [35061.376850] _raw_spin_lock+0x30/0x40 [35061.376852] ? igrab+0x1d/0x50 [35061.376853] igrab+0x1d/0x50 [35061.376854] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35061.376868] delayed_work+0x1d8/0x280 [ceph] [35061.376881] ? process_scheduled_works+0x22a/0x510 [35061.376884] process_scheduled_works+0x2a4/0x510 [35061.376888] worker_thread+0x188/0x340 [35061.376890] ? rescuer_thread+0x310/0x310 [35061.376893] kthread+0xf5/0x130 [35061.376895] ? kthread_complete_and_exit+0x20/0x20 [35061.376897] ret_from_fork+0x31/0x40 [35061.376899] ? kthread_complete_and_exit+0x20/0x20 [35061.376902] ret_from_fork_asm+0x11/0x20 [35061.376906] [35086.072873] watchdog: BUG: soft lockup - CPU#0 stuck for 21562s! [dbench:78683] [35086.080183] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35086.084873] watchdog: BUG: soft lockup - CPU#7 stuck for 30559s! [kworker/7:4:78808] [35086.135632] irq event stamp: 60941856 [35086.143362] Modules linked in: ceph libceph dns_resolver fscache netfs veth [35086.147020] hardirqs last enabled at (60941855): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35086.153979] nft_chain_nat xt_MASQUERADE nf_nat [35086.164137] hardirqs last disabled at (60941856): [] sysvec_apic_timer_interrupt+0xe/0x90 [35086.168661] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [35086.178383] softirqs last enabled at (60941784): [] __do_softirq+0x2be/0x371 [35086.178387] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [35086.194974] softirqs last disabled at (60941779): [] irq_exit_rcu+0xb0/0xd0 [35086.202453] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [35086.210963] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35086.216614] kvm irqbypass i2c_i801 [35086.225557] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35086.225558] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35086.229043] iTCO_wdt [35086.236514] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35086.241989] iTCO_vendor_support i2c_smbus [35086.244271] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35086.263007] crct10dif_pclmul joydev [35086.267098] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35086.272314] crc32_pclmul [35086.275886] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35086.283010] ipmi_ssif [35086.285627] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35086.292751] ghash_clmulni_intel [35086.295108] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35086.302234] mei_me mei [35086.305457] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35086.312582] lpc_ich [35086.315025] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35086.322148] sha512_ssse3 acpi_ipmi [35086.324334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35086.332412] mfd_core wmi [35086.335896] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35086.341632] ipmi_si ipmi_devintf [35086.344266] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35086.351392] ipmi_msghandler ioatdma [35086.354701] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35086.361826] acpi_pad acpi_power_meter [35086.365398] Call Trace: [35086.365400] [35086.372520] ixgbe igb [35086.376274] ? watchdog_timer_fn+0x1b4/0x220 [35086.378717] mdio crc32c_intel [35086.380730] ? __hrtimer_run_queues+0x1d3/0x360 [35086.383085] nvme [35086.387353] ? hrtimer_interrupt+0x104/0x240 [35086.390400] ptp nvme_core [35086.394924] ? __local_bh_enable+0x50/0x60 [35086.396849] i2c_algo_bit pps_core [35086.401116] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35086.403816] dca fuse [35086.407908] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35086.411304] [35086.416522] [35086.418793] irq event stamp: 201522612 [35086.423752] [35086.425241] hardirqs last enabled at (201522611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35086.427340] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35086.431083] hardirqs last disabled at (201522612): [] sysvec_apic_timer_interrupt+0xe/0x90 [35086.433185] ? queued_spin_lock_slowpath+0x18/0x2e0 [35086.443424] softirqs last enabled at (201522610): [] __do_softirq+0x2be/0x371 [35086.448730] do_raw_spin_lock+0xb5/0xc0 [35086.458553] softirqs last disabled at (201522605): [] irq_exit_rcu+0xb0/0xd0 [35086.463429] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35086.472198] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35086.476033] try_get_cap_refs+0x159/0x880 [ceph] [35086.484627] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35086.489416] __ceph_get_caps+0x4a1/0x760 [ceph] [35086.498789] Workqueue: events delayed_work [ceph] [35086.503402] ? __file_remove_privs+0x89/0x110 [35086.510870] [35086.515394] ? wake_bit_function+0x70/0x70 [35086.520091] RIP: 0010:rcu_is_watching+0x0/0x40 [35086.524445] ? ceph_write_iter+0x260/0x770 [ceph] [35086.525933] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [35086.530023] ceph_write_iter+0x260/0x770 [ceph] [35086.534460] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [35086.539161] ? lock_is_held_type+0xaa/0x120 [35086.557897] [35086.562424] vfs_write+0x324/0x4f0 [35086.567638] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [35086.571820] ksys_pwrite64+0x65/0xa0 [35086.573304] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [35086.576704] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35086.583826] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [35086.587400] do_syscall_64+0x38/0x80 [35086.594522] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [35086.599654] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35086.606776] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35086.610349] RIP: 0033:0x7f48f0eeb93b [35086.617471] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35086.622517] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35086.629641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35086.633211] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35086.641288] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35086.660026] ORIG_RAX: 0000000000000012 [35086.665763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35086.670982] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35086.678106] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35086.681935] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35086.689060] Call Trace: [35086.696183] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35086.703310] [35086.710433] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35086.712878] ? watchdog_timer_fn+0x1b4/0x220 [35086.720001] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35086.722013] ? __hrtimer_run_queues+0x1d3/0x360 [35086.729141] [35086.733403] ? hrtimer_interrupt+0x104/0x240 [35086.751495] ? __local_bh_enable+0x50/0x60 [35086.755590] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35086.760813] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35086.765771] [35086.767868] [35086.769966] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35086.775280] ? igrab+0x43/0x50 [35086.778340] ? igrab+0x43/0x50 [35086.781387] ? rcutorture_get_gp_data+0x30/0x30 [35086.785912] lock_release+0x1d7/0x260 [35086.789577] _raw_spin_unlock+0x1b/0x40 [35086.793410] igrab+0x43/0x50 [35086.796287] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35086.801436] delayed_work+0x1d8/0x280 [ceph] [35086.805719] ? process_scheduled_works+0x22a/0x510 [35086.810507] process_scheduled_works+0x2a4/0x510 [35086.815124] worker_thread+0x188/0x340 [35086.818874] ? rescuer_thread+0x310/0x310 [35086.822876] kthread+0xf5/0x130 [35086.826016] ? kthread_complete_and_exit+0x20/0x20 [35086.830807] ret_from_fork+0x31/0x40 [35086.834386] ? kthread_complete_and_exit+0x20/0x20 [35086.839169] ret_from_fork_asm+0x11/0x20 [35086.843094] [35114.073348] watchdog: BUG: soft lockup - CPU#0 stuck for 21588s! [dbench:78683] [35114.080650] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35114.085348] watchdog: BUG: soft lockup - CPU#7 stuck for 30585s! [kworker/7:4:78808] [35114.136092] irq event stamp: 60999876 [35114.136093] hardirqs last enabled at (60999875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35114.143822] Modules linked in: [35114.147479] hardirqs last disabled at (60999876): [] sysvec_apic_timer_interrupt+0xe/0x90 [35114.157638] ceph [35114.160690] softirqs last enabled at (60999808): [] __do_softirq+0x2be/0x371 [35114.170414] libceph [35114.172336] softirqs last disabled at (60999803): [] irq_exit_rcu+0xb0/0xd0 [35114.181022] dns_resolver [35114.183204] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35114.191715] fscache netfs [35114.194350] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35114.203294] veth [35114.205999] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35114.213469] nft_chain_nat [35114.215395] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35114.220871] xt_MASQUERADE nf_nat [35114.223574] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35114.242312] nf_conntrack nf_defrag_ipv6 [35114.245622] [35114.245623] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35114.250841] nf_defrag_ipv4 [35114.254759] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35114.256250] nft_compat nf_tables [35114.263374] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35114.266163] nfnetlink bridge [35114.273287] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35114.273288] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35114.276600] stp [35114.283723] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35114.286704] llc xfs [35114.293829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35114.300952] libcrc32c sunrpc [35114.302790] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35114.310867] intel_rapl_msr intel_rapl_common [35114.313052] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35114.318788] sb_edac [35114.321752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35114.328878] x86_pkg_temp_thermal coretemp [35114.333228] Call Trace: [35114.333230] [35114.340352] kvm_intel [35114.342537] ? watchdog_timer_fn+0x1b4/0x220 [35114.349661] kvm irqbypass [35114.353752] ? __hrtimer_run_queues+0x1d3/0x360 [35114.356195] i2c_i801 [35114.358209] ? hrtimer_interrupt+0x104/0x240 [35114.360563] iTCO_wdt iTCO_vendor_support [35114.364827] ? __local_bh_enable+0x50/0x60 [35114.367531] i2c_smbus crct10dif_pclmul [35114.372058] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35114.374325] joydev crc32_pclmul [35114.378590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35114.382593] ipmi_ssif [35114.386686] [35114.390516] ghash_clmulni_intel [35114.395732] [35114.398975] mei_me mei [35114.403933] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35114.406289] lpc_ich sha512_ssse3 [35114.408390] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35114.411610] acpi_ipmi mfd_core [35114.413709] do_raw_spin_lock+0xb5/0xc0 [35114.416153] wmi ipmi_si [35114.421458] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35114.424767] ipmi_devintf ipmi_msghandler ioatdma [35114.429641] try_get_cap_refs+0x159/0x880 [ceph] [35114.432775] acpi_pad acpi_power_meter [35114.436610] __ceph_get_caps+0x4a1/0x760 [ceph] [35114.439136] ixgbe igb [35114.443924] ? __file_remove_privs+0x89/0x110 [35114.448617] mdio crc32c_intel [35114.453228] ? wake_bit_function+0x70/0x70 [35114.456972] nvme ptp [35114.461499] ? ceph_write_iter+0x260/0x770 [ceph] [35114.463855] nvme_core i2c_algo_bit [35114.468204] ceph_write_iter+0x260/0x770 [ceph] [35114.471256] pps_core [35114.475349] ? lock_is_held_type+0xaa/0x120 [35114.477617] dca [35114.482316] vfs_write+0x324/0x4f0 [35114.485798] fuse [35114.490328] ksys_pwrite64+0x65/0xa0 [35114.492593] irq event stamp: 201743548 [35114.496770] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35114.498609] hardirqs last enabled at (201743547): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35114.502008] do_syscall_64+0x38/0x80 [35114.503948] hardirqs last disabled at (201743548): [] sysvec_apic_timer_interrupt+0xe/0x90 [35114.507519] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35114.511261] softirqs last enabled at (201743546): [] __do_softirq+0x2be/0x371 [35114.516393] RIP: 0033:0x7f48f0eeb93b [35114.526636] softirqs last disabled at (201743541): [] irq_exit_rcu+0xb0/0xd0 [35114.530209] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35114.540020] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35114.545063] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35114.553834] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35114.557405] ORIG_RAX: 0000000000000012 [35114.566002] Workqueue: events delayed_work [ceph] [35114.584750] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35114.584751] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35114.594124] [35114.594125] RIP: 0010:lock_acquire+0xf0/0x2b0 [35114.599343] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35114.606812] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35114.610645] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35114.615342] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35114.622465] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35114.629589] [35114.629590] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35114.631085] [35114.635431] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35114.698681] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35114.705805] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35114.712930] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35114.720055] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35114.728131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35114.733869] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35114.740994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35114.748118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35114.755242] Call Trace: [35114.757687] [35114.759698] ? watchdog_timer_fn+0x1b4/0x220 [35114.763963] ? __hrtimer_run_queues+0x1d3/0x360 [35114.768488] ? hrtimer_interrupt+0x104/0x240 [35114.772758] ? __local_bh_enable+0x50/0x60 [35114.776852] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35114.782074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35114.787043] [35114.789137] [35114.791237] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35114.796543] ? lock_acquire+0xf0/0x2b0 [35114.800294] ? igrab+0x43/0x50 [35114.803345] ? lock_release+0xc5/0x260 [35114.807090] _raw_spin_lock+0x30/0x40 [35114.810755] ? igrab+0x1d/0x50 [35114.813805] igrab+0x1d/0x50 [35114.816684] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35114.821833] delayed_work+0x1d8/0x280 [ceph] [35114.826113] ? process_scheduled_works+0x22a/0x510 [35114.830904] process_scheduled_works+0x2a4/0x510 [35114.835520] worker_thread+0x188/0x340 [35114.839268] ? rescuer_thread+0x310/0x310 [35114.843270] kthread+0xf5/0x130 [35114.846409] ? kthread_complete_and_exit+0x20/0x20 [35114.851193] ret_from_fork+0x31/0x40 [35114.854772] ? kthread_complete_and_exit+0x20/0x20 [35114.859556] ret_from_fork_asm+0x11/0x20 [35114.863481] [35142.073820] watchdog: BUG: soft lockup - CPU#0 stuck for 21614s! [dbench:78683] [35142.081122] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35142.085820] watchdog: BUG: soft lockup - CPU#7 stuck for 30611s! [kworker/7:4:78808] [35142.136571] irq event stamp: 61058132 [35142.136572] hardirqs last enabled at (61058131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35142.144301] Modules linked in: [35142.147958] hardirqs last disabled at (61058132): [] sysvec_apic_timer_interrupt+0xe/0x90 [35142.158116] ceph [35142.161166] softirqs last enabled at (61058086): [] __do_softirq+0x2be/0x371 [35142.170890] libceph [35142.172816] softirqs last disabled at (61058081): [] irq_exit_rcu+0xb0/0xd0 [35142.181500] dns_resolver fscache [35142.183685] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35142.192210] netfs veth [35142.195524] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35142.204467] nft_chain_nat [35142.206911] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35142.214381] xt_MASQUERADE nf_nat [35142.217087] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35142.222562] nf_conntrack nf_defrag_ipv6 [35142.225873] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35142.244611] nf_defrag_ipv4 nft_compat [35142.248531] [35142.253746] nf_tables [35142.257495] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35142.258991] nfnetlink bridge [35142.261346] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35142.268470] stp llc [35142.271452] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35142.278576] xfs libcrc32c [35142.280761] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35142.287884] sunrpc [35142.290588] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35142.297714] intel_rapl_msr [35142.299812] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35142.306934] intel_rapl_common [35142.309726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35142.317804] sb_edac x86_pkg_temp_thermal [35142.320853] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35142.326592] coretemp kvm_intel [35142.330595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35142.337718] kvm [35142.340856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35142.347982] irqbypass i2c_i801 [35142.349819] Call Trace: [35142.356943] iTCO_wdt iTCO_vendor_support [35142.360083] [35142.362524] i2c_smbus crct10dif_pclmul [35142.366530] ? watchdog_timer_fn+0x1b4/0x220 [35142.368539] joydev [35142.372370] ? __hrtimer_run_queues+0x1d3/0x360 [35142.376634] crc32_pclmul ipmi_ssif [35142.378734] ? hrtimer_interrupt+0x104/0x240 [35142.383255] ghash_clmulni_intel [35142.386740] ? __local_bh_enable+0x50/0x60 [35142.391002] mei_me mei [35142.394229] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35142.398318] lpc_ich sha512_ssse3 [35142.400761] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35142.405979] acpi_ipmi mfd_core [35142.409289] [35142.414248] wmi ipmi_si [35142.417384] [35142.417386] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35142.419497] ipmi_devintf [35142.422034] ? queued_spin_lock_slowpath+0x16/0x2e0 [35142.424127] ipmi_msghandler ioatdma [35142.429434] do_raw_spin_lock+0xb5/0xc0 [35142.432050] acpi_pad acpi_power_meter [35142.436922] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35142.440489] ixgbe igb [35142.444342] try_get_cap_refs+0x159/0x880 [ceph] [35142.448082] mdio [35142.452873] __ceph_get_caps+0x4a1/0x760 [ceph] [35142.455223] crc32c_intel [35142.459840] ? __file_remove_privs+0x89/0x110 [35142.461758] nvme ptp [35142.466283] ? wake_bit_function+0x70/0x70 [35142.468900] nvme_core [35142.473253] ? ceph_write_iter+0x260/0x770 [ceph] [35142.475521] i2c_algo_bit [35142.479613] ceph_write_iter+0x260/0x770 [ceph] [35142.481969] pps_core dca [35142.486672] ? lock_is_held_type+0xaa/0x120 [35142.489286] fuse [35142.493810] vfs_write+0x324/0x4f0 [35142.496426] [35142.496427] irq event stamp: 201964218 [35142.500608] ksys_pwrite64+0x65/0xa0 [35142.502529] hardirqs last enabled at (201964217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35142.505926] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35142.507415] hardirqs last disabled at (201964218): [] sysvec_apic_timer_interrupt+0xe/0x90 [35142.511162] do_syscall_64+0x38/0x80 [35142.514730] softirqs last enabled at (201964216): [] __do_softirq+0x2be/0x371 [35142.524977] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35142.530105] softirqs last disabled at (201964211): [] irq_exit_rcu+0xb0/0xd0 [35142.539916] RIP: 0033:0x7f48f0eeb93b [35142.543500] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35142.552277] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35142.557319] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35142.565917] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35142.569503] Workqueue: events delayed_work [ceph] [35142.578883] ORIG_RAX: 0000000000000012 [35142.597619] [35142.605090] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35142.610309] RIP: 0010:lock_acquire+0xf0/0x2b0 [35142.615005] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35142.618836] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35142.620326] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35142.627451] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35142.631801] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35142.638925] [35142.638926] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35142.657664] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35142.664789] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35142.670010] [35142.677128] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35142.709301] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35142.716424] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35142.723549] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35142.731626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35142.737364] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35142.744502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35142.751628] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35142.758754] Call Trace: [35142.761200] [35142.763225] ? watchdog_timer_fn+0x1b4/0x220 [35142.767507] ? __hrtimer_run_queues+0x1d3/0x360 [35142.772034] ? hrtimer_interrupt+0x104/0x240 [35142.776303] ? __local_bh_enable+0x50/0x60 [35142.780397] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35142.785623] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35142.790580] [35142.792676] [35142.794774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35142.800081] ? lock_acquire+0xf0/0x2b0 [35142.803834] ? igrab+0x43/0x50 [35142.806889] ? lock_release+0xc5/0x260 [35142.810636] _raw_spin_lock+0x30/0x40 [35142.814292] ? igrab+0x1d/0x50 [35142.817341] igrab+0x1d/0x50 [35142.820229] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35142.825378] delayed_work+0x1d8/0x280 [ceph] [35142.829660] ? process_scheduled_works+0x22a/0x510 [35142.834451] process_scheduled_works+0x2a4/0x510 [35142.839065] worker_thread+0x188/0x340 [35142.842816] ? rescuer_thread+0x310/0x310 [35142.846825] kthread+0xf5/0x130 [35142.849964] ? kthread_complete_and_exit+0x20/0x20 [35142.854749] ret_from_fork+0x31/0x40 [35142.858329] ? kthread_complete_and_exit+0x20/0x20 [35142.863112] ret_from_fork_asm+0x11/0x20 [35142.867045] [35170.074291] watchdog: BUG: soft lockup - CPU#0 stuck for 21640s! [dbench:78683] [35170.081592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35170.086291] watchdog: BUG: soft lockup - CPU#7 stuck for 30637s! [kworker/7:4:78808] [35170.137033] irq event stamp: 61116308 [35170.137035] hardirqs last enabled at (61116307): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35170.144763] Modules linked in: [35170.148420] hardirqs last disabled at (61116308): [] sysvec_apic_timer_interrupt+0xe/0x90 [35170.158579] ceph [35170.161629] softirqs last enabled at (61116240): [] __do_softirq+0x2be/0x371 [35170.171352] libceph [35170.173276] softirqs last disabled at (61116235): [] irq_exit_rcu+0xb0/0xd0 [35170.181973] dns_resolver [35170.184153] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35170.192664] fscache [35170.195299] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35170.204243] netfs veth [35170.206428] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [35170.213897] nft_chain_nat [35170.216343] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [35170.221819] xt_MASQUERADE nf_nat [35170.224524] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35170.243262] nf_conntrack nf_defrag_ipv6 [35170.246573] [35170.246574] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35170.251790] nf_defrag_ipv4 nft_compat [35170.255706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35170.257197] nf_tables nfnetlink [35170.264323] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35170.268066] bridge stp [35170.275189] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35170.278415] llc xfs [35170.285538] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35170.287982] libcrc32c sunrpc [35170.295108] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35170.297290] intel_rapl_msr intel_rapl_common [35170.304414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35170.307379] sb_edac x86_pkg_temp_thermal [35170.315456] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35170.319806] coretemp kvm_intel [35170.325545] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35170.329548] kvm irqbypass [35170.336688] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35170.339827] i2c_i801 iTCO_wdt [35170.346951] Call Trace: [35170.346952] [35170.349655] iTCO_vendor_support [35170.356780] ? watchdog_timer_fn+0x1b4/0x220 [35170.359829] i2c_smbus [35170.362277] ? __hrtimer_run_queues+0x1d3/0x360 [35170.364285] crct10dif_pclmul [35170.367513] ? hrtimer_interrupt+0x104/0x240 [35170.371774] joydev crc32_pclmul [35170.374131] ? __local_bh_enable+0x50/0x60 [35170.378655] ipmi_ssif ghash_clmulni_intel [35170.381621] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35170.385881] mei_me [35170.389106] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35170.393197] mei [35170.397287] [35170.402506] lpc_ich sha512_ssse3 [35170.404603] [35170.404604] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35170.409561] acpi_ipmi mfd_core [35170.411401] ? queued_spin_lock_slowpath+0x2b/0x2e0 [35170.413495] wmi ipmi_si [35170.416807] do_raw_spin_lock+0xb5/0xc0 [35170.418903] ipmi_devintf [35170.424208] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35170.427345] ipmi_msghandler ioatdma [35170.432217] try_get_cap_refs+0x159/0x880 [ceph] [35170.434747] acpi_pad acpi_power_meter [35170.438581] __ceph_get_caps+0x4a1/0x760 [ceph] [35170.441195] ixgbe igb [35170.445990] ? __file_remove_privs+0x89/0x110 [35170.449557] mdio crc32c_intel [35170.454169] ? wake_bit_function+0x70/0x70 [35170.457912] nvme ptp [35170.462439] ? ceph_write_iter+0x260/0x770 [ceph] [35170.464794] nvme_core i2c_algo_bit [35170.469145] ceph_write_iter+0x260/0x770 [ceph] [35170.472196] pps_core [35170.476291] ? lock_is_held_type+0xaa/0x120 [35170.478557] dca fuse [35170.483258] vfs_write+0x324/0x4f0 [35170.486738] [35170.486739] irq event stamp: 202184876 [35170.491268] ksys_pwrite64+0x65/0xa0 [35170.493533] hardirqs last enabled at (202184875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35170.497710] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35170.499981] hardirqs last disabled at (202184876): [] sysvec_apic_timer_interrupt+0xe/0x90 [35170.503379] do_syscall_64+0x38/0x80 [35170.504869] softirqs last enabled at (202184874): [] __do_softirq+0x2be/0x371 [35170.508615] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35170.512184] softirqs last disabled at (202184865): [] irq_exit_rcu+0xb0/0xd0 [35170.522429] RIP: 0033:0x7f48f0eeb93b [35170.527561] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35170.537371] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35170.540940] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35170.549711] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35170.554757] Workqueue: events delayed_work [ceph] [35170.563353] ORIG_RAX: 0000000000000012 [35170.566925] [35170.566926] RIP: 0010:lock_acquire+0xf0/0x2b0 [35170.576301] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35170.595039] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35170.602511] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35170.607727] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35170.612426] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35170.616257] [35170.616258] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35170.617746] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35170.622096] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35170.629220] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35170.647973] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35170.655106] [35170.660318] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35170.713861] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35170.720992] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35170.729071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35170.734809] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35170.741934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35170.749057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35170.756181] Call Trace: [35170.758627] [35170.760638] ? watchdog_timer_fn+0x1b4/0x220 [35170.764902] ? __hrtimer_run_queues+0x1d3/0x360 [35170.769429] ? hrtimer_interrupt+0x104/0x240 [35170.773698] ? __local_bh_enable+0x50/0x60 [35170.777790] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35170.783015] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35170.787978] [35170.790077] [35170.792176] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35170.797483] ? lock_acquire+0xf0/0x2b0 [35170.801233] ? igrab+0x43/0x50 [35170.804284] ? lock_release+0xc5/0x260 [35170.808028] _raw_spin_lock+0x30/0x40 [35170.811697] ? igrab+0x1d/0x50 [35170.814754] igrab+0x1d/0x50 [35170.817631] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35170.822780] delayed_work+0x1d8/0x280 [ceph] [35170.827062] ? process_scheduled_works+0x22a/0x510 [35170.831853] process_scheduled_works+0x2a4/0x510 [35170.836469] worker_thread+0x188/0x340 [35170.840217] ? rescuer_thread+0x310/0x310 [35170.844220] kthread+0xf5/0x130 [35170.847358] ? kthread_complete_and_exit+0x20/0x20 [35170.852143] ret_from_fork+0x31/0x40 [35170.855721] ? kthread_complete_and_exit+0x20/0x20 [35170.860504] ret_from_fork_asm+0x11/0x20 [35170.864428] [35198.074759] watchdog: BUG: soft lockup - CPU#0 stuck for 21666s! [dbench:78683] [35198.082063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35198.086759] watchdog: BUG: soft lockup - CPU#7 stuck for 30663s! [kworker/7:4:78808] [35198.137511] irq event stamp: 61174506 [35198.145242] Modules linked in: ceph libceph dns_resolver fscache netfs [35198.148899] hardirqs last enabled at (61174505): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35198.155435] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [35198.165591] hardirqs last disabled at (61174506): [] sysvec_apic_timer_interrupt+0xe/0x90 [35198.171677] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [35198.181399] softirqs last enabled at (61174438): [] __do_softirq+0x2be/0x371 [35198.189130] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [35198.197831] softirqs last disabled at (61174433): [] irq_exit_rcu+0xb0/0xd0 [35198.204350] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [35198.212863] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35198.212864] kvm [35198.212865] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35198.218511] irqbypass [35198.227456] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35198.229293] i2c_i801 [35198.236765] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35198.239122] iTCO_wdt iTCO_vendor_support [35198.244599] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35198.246869] i2c_smbus crct10dif_pclmul [35198.265607] [35198.265608] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35198.269612] joydev [35198.274829] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35198.278658] crc32_pclmul [35198.280156] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35198.287283] ipmi_ssif ghash_clmulni_intel [35198.289378] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35198.296505] mei_me mei [35198.299122] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35198.306246] lpc_ich [35198.310337] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35198.317460] sha512_ssse3 acpi_ipmi [35198.319904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35198.327029] mfd_core wmi [35198.329211] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35198.337290] ipmi_si ipmi_devintf [35198.340775] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35198.346512] ipmi_msghandler [35198.349129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35198.356254] ioatdma acpi_pad [35198.359564] Call Trace: [35198.359566] [35198.366687] acpi_power_meter ixgbe [35198.369567] ? watchdog_timer_fn+0x1b4/0x220 [35198.376689] igb [35198.379655] ? __hrtimer_run_queues+0x1d3/0x360 [35198.382097] mdio [35198.384112] ? hrtimer_interrupt+0x104/0x240 [35198.387593] crc32c_intel [35198.391856] ? __local_bh_enable+0x50/0x60 [35198.393694] nvme [35198.398221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35198.400158] ptp [35198.404438] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35198.407059] nvme_core i2c_algo_bit [35198.411159] [35198.411160] [35198.413082] pps_core dca [35198.418299] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35198.420137] fuse [35198.425115] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35198.428594] [35198.428594] irq event stamp: 202405546 [35198.430695] do_raw_spin_lock+0xb5/0xc0 [35198.432789] hardirqs last enabled at (202405545): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35198.435410] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35198.440711] hardirqs last disabled at (202405546): [] sysvec_apic_timer_interrupt+0xe/0x90 [35198.442638] try_get_cap_refs+0x159/0x880 [ceph] [35198.447507] softirqs last enabled at (202405544): [] __do_softirq+0x2be/0x371 [35198.449002] __ceph_get_caps+0x4a1/0x760 [ceph] [35198.452741] softirqs last disabled at (202405539): [] irq_exit_rcu+0xb0/0xd0 [35198.456576] ? __file_remove_privs+0x89/0x110 [35198.466816] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35198.471601] ? wake_bit_function+0x70/0x70 [35198.481412] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35198.486024] ? ceph_write_iter+0x260/0x770 [ceph] [35198.494793] Workqueue: events delayed_work [ceph] [35198.499318] ceph_write_iter+0x260/0x770 [ceph] [35198.507913] [35198.507914] RIP: 0010:lock_acquire+0xf0/0x2b0 [35198.512267] ? lock_is_held_type+0xaa/0x120 [35198.521642] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35198.525736] vfs_write+0x324/0x4f0 [35198.533203] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35198.537906] ksys_pwrite64+0x65/0xa0 [35198.542599] [35198.542600] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35198.547124] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35198.548614] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35198.552965] do_syscall_64+0x38/0x80 [35198.557158] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35198.575898] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35198.579294] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35198.584512] RIP: 0033:0x7f48f0eeb93b [35198.588082] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35198.589575] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35198.596696] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35198.601826] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35198.608951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35198.612523] ORIG_RAX: 0000000000000012 [35198.619646] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35198.624690] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35198.631815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35198.635385] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35198.642511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35198.661248] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35198.669325] Call Trace: [35198.669327] [35198.674543] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35198.680281] ? watchdog_timer_fn+0x1b4/0x220 [35198.684110] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35198.691236] ? __hrtimer_run_queues+0x1d3/0x360 [35198.698365] [35198.705486] ? hrtimer_interrupt+0x104/0x240 [35198.760803] ? __local_bh_enable+0x50/0x60 [35198.764897] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35198.770121] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35198.775077] [35198.777176] [35198.779272] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35198.784582] ? lock_acquire+0xf0/0x2b0 [35198.788332] ? igrab+0x43/0x50 [35198.791390] ? lock_release+0xc5/0x260 [35198.795134] _raw_spin_lock+0x30/0x40 [35198.798818] ? igrab+0x1d/0x50 [35198.801867] igrab+0x1d/0x50 [35198.804747] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35198.809895] delayed_work+0x1d8/0x280 [ceph] [35198.814178] ? process_scheduled_works+0x22a/0x510 [35198.818967] process_scheduled_works+0x2a4/0x510 [35198.823583] worker_thread+0x188/0x340 [35198.827333] ? rescuer_thread+0x310/0x310 [35198.831344] kthread+0xf5/0x130 [35198.834491] ? kthread_complete_and_exit+0x20/0x20 [35198.839275] ret_from_fork+0x31/0x40 [35198.842853] ? kthread_complete_and_exit+0x20/0x20 [35198.847636] ret_from_fork_asm+0x11/0x20 [35198.851561] [35226.075225] watchdog: BUG: soft lockup - CPU#0 stuck for 21692s! [dbench:78683] [35226.082532] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35226.087226] watchdog: BUG: soft lockup - CPU#7 stuck for 30689s! [kworker/7:4:78808] [35226.137971] irq event stamp: 61232570 [35226.137972] hardirqs last enabled at (61232569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35226.145703] Modules linked in: [35226.149361] hardirqs last disabled at (61232570): [] sysvec_apic_timer_interrupt+0xe/0x90 [35226.159517] ceph [35226.162569] softirqs last enabled at (61232534): [] __do_softirq+0x2be/0x371 [35226.172292] libceph [35226.174216] softirqs last disabled at (61232529): [] irq_exit_rcu+0xb0/0xd0 [35226.182902] dns_resolver [35226.185085] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35226.193595] fscache [35226.196215] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35226.205158] netfs veth [35226.207342] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35226.214814] nft_chain_nat xt_MASQUERADE [35226.217257] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35226.222733] nf_nat nf_conntrack [35226.226652] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35226.245388] nf_defrag_ipv6 nf_defrag_ipv4 [35226.248620] [35226.248621] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35226.253839] nft_compat nf_tables [35226.257929] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35226.259420] nfnetlink [35226.266546] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35226.269854] bridge stp [35226.276979] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35226.279337] llc xfs [35226.286460] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35226.288905] libcrc32c sunrpc [35226.296029] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35226.298213] intel_rapl_msr [35226.305337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35226.308301] intel_rapl_common sb_edac [35226.316380] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35226.319169] x86_pkg_temp_thermal [35226.324907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35226.328650] coretemp kvm_intel [35226.335775] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35226.339086] kvm irqbypass [35226.346211] Call Trace: [35226.346213] [35226.349347] i2c_i801 iTCO_wdt [35226.356473] ? watchdog_timer_fn+0x1b4/0x220 [35226.359177] iTCO_vendor_support i2c_smbus [35226.361624] ? __hrtimer_run_queues+0x1d3/0x360 [35226.363629] crct10dif_pclmul [35226.366684] ? hrtimer_interrupt+0x104/0x240 [35226.370944] joydev crc32_pclmul [35226.375035] ? __local_bh_enable+0x50/0x60 [35226.379560] ipmi_ssif ghash_clmulni_intel [35226.382527] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35226.386789] mei_me mei [35226.390014] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35226.394102] lpc_ich sha512_ssse3 [35226.398193] [35226.403411] acpi_ipmi [35226.405856] [35226.410814] mfd_core [35226.414124] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35226.416220] wmi ipmi_si [35226.418582] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35226.420676] ipmi_devintf ipmi_msghandler [35226.422950] do_raw_spin_lock+0xb5/0xc0 [35226.428249] ioatdma acpi_pad acpi_power_meter [35226.430782] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35226.435651] ixgbe igb [35226.439659] try_get_cap_refs+0x159/0x880 [ceph] [35226.443487] mdio crc32c_intel [35226.447929] __ceph_get_caps+0x4a1/0x760 [ceph] [35226.452709] nvme ptp [35226.455070] ? __file_remove_privs+0x89/0x110 [35226.459675] nvme_core [35226.462726] ? wake_bit_function+0x70/0x70 [35226.467250] i2c_algo_bit pps_core [35226.469525] ? ceph_write_iter+0x260/0x770 [ceph] [35226.473872] dca fuse [35226.476231] ceph_write_iter+0x260/0x770 [ceph] [35226.480322] [35226.480322] irq event stamp: 202626278 [35226.483722] ? lock_is_held_type+0xaa/0x120 [35226.488414] hardirqs last enabled at (202626277): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35226.490689] vfs_write+0x324/0x4f0 [35226.495211] hardirqs last disabled at (202626278): [] sysvec_apic_timer_interrupt+0xe/0x90 [35226.496705] ksys_pwrite64+0x65/0xa0 [35226.500445] softirqs last enabled at (202626276): [] __do_softirq+0x2be/0x371 [35226.504624] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35226.514868] softirqs last disabled at (202626271): [] irq_exit_rcu+0xb0/0xd0 [35226.518264] do_syscall_64+0x38/0x80 [35226.528074] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35226.531646] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35226.540416] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35226.545548] RIP: 0033:0x7f48f0eeb93b [35226.554146] Workqueue: events delayed_work [ceph] [35226.557715] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35226.567094] [35226.572136] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35226.579625] RIP: 0010:lock_acquire+0x86/0x2b0 [35226.583195] ORIG_RAX: 0000000000000012 [35226.587893] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [35226.606649] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35226.608138] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [35226.613357] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35226.617707] [35226.621537] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35226.640276] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [35226.647399] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35226.652630] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [35226.659750] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35226.661241] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [35226.668371] [35226.675490] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [35226.713311] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35226.720436] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35226.728514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35226.734251] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35226.741374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35226.748500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35226.755631] Call Trace: [35226.758077] [35226.760088] ? watchdog_timer_fn+0x1b4/0x220 [35226.764351] ? __hrtimer_run_queues+0x1d3/0x360 [35226.768879] ? hrtimer_interrupt+0x104/0x240 [35226.773147] ? __local_bh_enable+0x50/0x60 [35226.777242] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35226.782466] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35226.787421] [35226.789520] [35226.791633] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35226.796942] ? lock_acquire+0x86/0x2b0 [35226.800691] ? igrab+0x43/0x50 [35226.803741] ? lock_release+0xc5/0x260 [35226.807488] _raw_spin_lock+0x30/0x40 [35226.811151] ? igrab+0x1d/0x50 [35226.814201] igrab+0x1d/0x50 [35226.817080] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35226.822231] delayed_work+0x1d8/0x280 [ceph] [35226.826513] ? process_scheduled_works+0x22a/0x510 [35226.831301] process_scheduled_works+0x2a4/0x510 [35226.835918] worker_thread+0x188/0x340 [35226.839666] ? rescuer_thread+0x310/0x310 [35226.843670] kthread+0xf5/0x130 [35226.846806] ? kthread_complete_and_exit+0x20/0x20 [35226.851593] ret_from_fork+0x31/0x40 [35226.855170] ? kthread_complete_and_exit+0x20/0x20 [35226.859955] ret_from_fork_asm+0x11/0x20 [35226.863887] [35254.075691] watchdog: BUG: soft lockup - CPU#0 stuck for 21718s! [dbench:78683] [35254.082994] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35254.087691] watchdog: BUG: soft lockup - CPU#7 stuck for 30715s! [kworker/7:4:78808] [35254.138442] irq event stamp: 61290722 [35254.146174] Modules linked in: ceph libceph dns_resolver fscache [35254.149830] hardirqs last enabled at (61290721): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35254.155829] netfs veth nft_chain_nat xt_MASQUERADE nf_nat [35254.165987] hardirqs last disabled at (61290722): [] sysvec_apic_timer_interrupt+0xe/0x90 [35254.171470] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [35254.181196] softirqs last enabled at (61290654): [] __do_softirq+0x2be/0x371 [35254.189716] stp llc xfs libcrc32c sunrpc [35254.198398] softirqs last disabled at (61290649): [] irq_exit_rcu+0xb0/0xd0 [35254.198402] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [35254.210916] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35254.210917] kvm [35254.210918] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35254.219425] irqbypass i2c_i801 [35254.228370] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35254.230207] iTCO_wdt iTCO_vendor_support [35254.237679] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35254.240814] i2c_smbus crct10dif_pclmul [35254.246293] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35254.250296] joydev crc32_pclmul [35254.269036] [35254.272865] ipmi_ssif ghash_clmulni_intel [35254.278086] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35254.281305] mei_me mei [35254.282797] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35254.286889] lpc_ich [35254.294011] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35254.296457] sha512_ssse3 acpi_ipmi [35254.303581] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35254.305764] mfd_core [35254.312888] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35254.316372] wmi ipmi_si [35254.323496] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35254.325767] ipmi_devintf [35254.332891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35254.335421] ipmi_msghandler [35254.343500] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35254.346117] ioatdma [35254.351855] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35254.354731] acpi_pad [35254.361856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35254.364039] acpi_power_meter [35254.371165] Call Trace: [35254.371167] [35254.373434] ixgbe igb [35254.380562] ? watchdog_timer_fn+0x1b4/0x220 [35254.383524] mdio crc32c_intel [35254.385967] ? __hrtimer_run_queues+0x1d3/0x360 [35254.387978] nvme [35254.390338] ? hrtimer_interrupt+0x104/0x240 [35254.394600] ptp [35254.397651] ? __local_bh_enable+0x50/0x60 [35254.402175] nvme_core i2c_algo_bit [35254.404100] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35254.408370] pps_core dca [35254.410209] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35254.414301] fuse [35254.417784] [35254.423001] [35254.425617] [35254.425618] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35254.430575] irq event stamp: 202846850 [35254.432504] ? queued_spin_lock_slowpath+0x18/0x2e0 [35254.434595] hardirqs last enabled at (202846849): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35254.436093] do_raw_spin_lock+0xb5/0xc0 [35254.438184] hardirqs last disabled at (202846850): [] sysvec_apic_timer_interrupt+0xe/0x90 [35254.443491] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35254.447233] softirqs last enabled at (202846848): [] __do_softirq+0x2be/0x371 [35254.452106] try_get_cap_refs+0x159/0x880 [ceph] [35254.462349] softirqs last disabled at (202846843): [] irq_exit_rcu+0xb0/0xd0 [35254.466185] __ceph_get_caps+0x4a1/0x760 [ceph] [35254.475990] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35254.480778] ? __file_remove_privs+0x89/0x110 [35254.489545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35254.494156] ? wake_bit_function+0x70/0x70 [35254.502752] Workqueue: events delayed_work [ceph] [35254.507278] ? ceph_write_iter+0x260/0x770 [ceph] [35254.516654] [35254.521006] ceph_write_iter+0x260/0x770 [ceph] [35254.528476] RIP: 0010:lock_acquire+0xf0/0x2b0 [35254.532572] ? lock_is_held_type+0xaa/0x120 [35254.537263] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35254.541965] vfs_write+0x324/0x4f0 [35254.543452] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35254.547981] ksys_pwrite64+0x65/0xa0 [35254.552326] [35254.552327] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35254.556505] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35254.575241] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35254.578641] do_syscall_64+0x38/0x80 [35254.583856] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35254.587430] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35254.588919] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35254.596043] RIP: 0033:0x7f48f0eeb93b [35254.601174] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35254.608297] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35254.611869] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35254.618991] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35254.624035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35254.631160] ORIG_RAX: 0000000000000012 [35254.634731] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35254.641855] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35254.660593] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35254.668670] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35254.673887] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35254.679624] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35254.683455] Call Trace: [35254.683457] [35254.690581] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35254.697705] ? watchdog_timer_fn+0x1b4/0x220 [35254.704828] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35254.711954] ? __hrtimer_run_queues+0x1d3/0x360 [35254.719100] [35254.726222] ? hrtimer_interrupt+0x104/0x240 [35254.760158] ? __local_bh_enable+0x50/0x60 [35254.764251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35254.769476] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35254.774432] [35254.776529] [35254.778628] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35254.783935] ? lock_acquire+0xf0/0x2b0 [35254.787684] ? igrab+0x43/0x50 [35254.790734] ? lock_release+0xc5/0x260 [35254.794481] _raw_spin_lock+0x30/0x40 [35254.798144] ? igrab+0x1d/0x50 [35254.801195] igrab+0x1d/0x50 [35254.804074] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35254.809241] delayed_work+0x1d8/0x280 [ceph] [35254.813524] ? process_scheduled_works+0x22a/0x510 [35254.818314] process_scheduled_works+0x2a4/0x510 [35254.822927] worker_thread+0x188/0x340 [35254.826678] ? rescuer_thread+0x310/0x310 [35254.830689] kthread+0xf5/0x130 [35254.833836] ? kthread_complete_and_exit+0x20/0x20 [35254.838619] ret_from_fork+0x31/0x40 [35254.842198] ? kthread_complete_and_exit+0x20/0x20 [35254.846983] ret_from_fork_asm+0x11/0x20 [35254.850916] [35256.383096] rcu: INFO: rcu_preempt self-detected stall on CPU [35256.388855] rcu: 7-....: (22642298 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/22640955 fqs=11411499 [35256.399539] rcu: (t=23319360 jiffies g=1309609 q=652626 ncpus=8) [35256.405624] Sending NMI from CPU 7 to CPUs 0: [35256.409979] NMI backtrace for cpu 0 [35256.409980] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35256.409982] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35256.409982] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35256.409986] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35256.409987] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35256.409988] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35256.409989] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35256.409990] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35256.409991] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35256.409992] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35256.409992] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35256.409994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35256.409994] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35256.409995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35256.409996] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35256.409997] Call Trace: [35256.409997] [35256.409998] ? nmi_cpu_backtrace+0xad/0x120 [35256.410002] ? nmi_cpu_backtrace_handler+0x11/0x20 [35256.410003] ? nmi_handle+0xd4/0x260 [35256.410008] ? default_do_nmi+0x49/0xf0 [35256.410009] ? exc_nmi+0x11e/0x150 [35256.410011] ? end_repeat_nmi+0x16/0x67 [35256.410015] ? queued_spin_lock_slowpath+0x18/0x2e0 [35256.410018] ? queued_spin_lock_slowpath+0x18/0x2e0 [35256.410020] ? queued_spin_lock_slowpath+0x18/0x2e0 [35256.410022] [35256.410023] [35256.410024] do_raw_spin_lock+0xb5/0xc0 [35256.410026] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35256.410040] try_get_cap_refs+0x159/0x880 [ceph] [35256.410054] __ceph_get_caps+0x4a1/0x760 [ceph] [35256.410067] ? __file_remove_privs+0x89/0x110 [35256.410069] ? wake_bit_function+0x70/0x70 [35256.410071] ? ceph_write_iter+0x260/0x770 [ceph] [35256.410082] ceph_write_iter+0x260/0x770 [ceph] [35256.410094] ? lock_is_held_type+0xaa/0x120 [35256.410098] vfs_write+0x324/0x4f0 [35256.410103] ksys_pwrite64+0x65/0xa0 [35256.410105] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35256.410108] do_syscall_64+0x38/0x80 [35256.410110] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35256.410112] RIP: 0033:0x7f48f0eeb93b [35256.410113] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35256.410114] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [35256.410116] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35256.410116] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35256.410117] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35256.410118] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35256.410119] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35256.410122] [35256.410975] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35256.714475] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35256.721946] Workqueue: events delayed_work [ceph] [35256.726661] RIP: 0010:do_raw_spin_unlock+0x18/0xc0 [35256.731445] Code: 89 47 10 b8 01 00 00 00 c3 31 c0 c3 66 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 53 81 7f 04 ad 4e ad de 48 89 fb 75 34 8b 03 <85> c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 8b 05 [35256.750183] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [35256.755400] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [35256.762524] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [35256.769647] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [35256.776772] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [35256.783897] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35256.791020] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35256.799098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35256.804836] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35256.811959] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35256.819082] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35256.826208] Call Trace: [35256.828653] [35256.830663] ? rcu_dump_cpu_stacks+0xd1/0x180 [35256.835015] ? rcu_sched_clock_irq+0x633/0x1390 [35256.839545] ? lock_release+0xc5/0x260 [35256.843289] ? lock_is_held_type+0xaa/0x120 [35256.847469] ? lock_is_held_type+0xaa/0x120 [35256.851646] ? tick_sched_do_timer+0xa0/0xa0 [35256.855917] ? rcu_read_lock_sched_held+0x56/0x90 [35256.860617] ? tick_sched_do_timer+0xa0/0xa0 [35256.864888] ? update_process_times+0x5f/0x90 [35256.869238] ? tick_sched_handle.isra.16+0x1e/0x50 [35256.874023] ? tick_sched_timer+0x6f/0x80 [35256.878027] ? __hrtimer_run_queues+0x1d3/0x360 [35256.882554] ? hrtimer_interrupt+0x104/0x240 [35256.886824] ? __local_bh_enable+0x50/0x60 [35256.890917] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35256.896140] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35256.901099] [35256.903196] [35256.905292] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35256.910602] ? do_raw_spin_unlock+0x18/0xc0 [35256.914785] _raw_spin_unlock+0x23/0x40 [35256.918614] igrab+0x43/0x50 [35256.921493] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35256.926643] delayed_work+0x1d8/0x280 [ceph] [35256.930924] ? process_scheduled_works+0x22a/0x510 [35256.935712] process_scheduled_works+0x2a4/0x510 [35256.940328] worker_thread+0x188/0x340 [35256.944078] ? rescuer_thread+0x310/0x310 [35256.948081] kthread+0xf5/0x130 [35256.951218] ? kthread_complete_and_exit+0x20/0x20 [35256.956004] ret_from_fork+0x31/0x40 [35256.959583] ? kthread_complete_and_exit+0x20/0x20 [35256.964366] ret_from_fork_asm+0x11/0x20 [35256.968290] [35282.076155] watchdog: BUG: soft lockup - CPU#0 stuck for 21744s! [dbench:78683] [35282.083455] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35282.088154] watchdog: BUG: soft lockup - CPU#7 stuck for 30742s! [kworker/7:4:78808] [35282.138895] irq event stamp: 61348806 [35282.138896] hardirqs last enabled at (61348805): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35282.146626] Modules linked in: ceph [35282.150283] hardirqs last disabled at (61348806): [] sysvec_apic_timer_interrupt+0xe/0x90 [35282.160440] libceph [35282.163926] softirqs last enabled at (61348740): [] __do_softirq+0x2be/0x371 [35282.173649] dns_resolver fscache [35282.175837] softirqs last disabled at (61348735): [] irq_exit_rcu+0xb0/0xd0 [35282.184518] netfs [35282.187828] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35282.196357] veth nft_chain_nat [35282.198366] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35282.207310] xt_MASQUERADE [35282.210448] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35282.217919] nf_nat nf_conntrack [35282.220624] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35282.226100] nf_defrag_ipv6 [35282.229324] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35282.248063] nf_defrag_ipv4 nft_compat [35282.250854] [35282.250854] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35282.256070] nf_tables [35282.259830] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35282.261324] nfnetlink bridge [35282.268447] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35282.270803] stp llc [35282.277929] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35282.280893] xfs libcrc32c [35282.288016] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35282.290201] sunrpc intel_rapl_msr [35282.297326] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35282.300030] intel_rapl_common sb_edac [35282.307153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35282.310550] x86_pkg_temp_thermal coretemp [35282.318629] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35282.322371] kvm_intel kvm [35282.328111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35282.332201] irqbypass i2c_i801 [35282.339324] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35282.342028] iTCO_wdt iTCO_vendor_support [35282.349151] Call Trace: [35282.349153] [35282.352290] i2c_smbus crct10dif_pclmul [35282.359416] ? watchdog_timer_fn+0x1b4/0x220 [35282.363417] joydev crc32_pclmul [35282.365863] ? __hrtimer_run_queues+0x1d3/0x360 [35282.367873] ipmi_ssif ghash_clmulni_intel [35282.371706] ? hrtimer_interrupt+0x104/0x240 [35282.375967] mei_me mei [35282.379191] ? __local_bh_enable+0x50/0x60 [35282.383717] lpc_ich sha512_ssse3 [35282.387809] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35282.392071] acpi_ipmi mfd_core [35282.394514] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35282.398606] wmi ipmi_si [35282.401916] [35282.407134] ipmi_devintf [35282.410271] [35282.410273] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35282.415227] ipmi_msghandler ioatdma [35282.417763] ? queued_spin_lock_slowpath+0x16/0x2e0 [35282.419858] acpi_pad acpi_power_meter [35282.422476] do_raw_spin_lock+0xb5/0xc0 [35282.424572] ixgbe igb [35282.429878] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35282.433445] mdio crc32c_intel [35282.438319] try_get_cap_refs+0x159/0x880 [ceph] [35282.442060] nvme ptp [35282.445896] __ceph_get_caps+0x4a1/0x760 [ceph] [35282.448248] nvme_core i2c_algo_bit [35282.453036] ? __file_remove_privs+0x89/0x110 [35282.456083] pps_core dca [35282.460696] ? wake_bit_function+0x70/0x70 [35282.462966] fuse [35282.467492] ? ceph_write_iter+0x260/0x770 [ceph] [35282.470973] irq event stamp: 203062824 [35282.475324] ceph_write_iter+0x260/0x770 [ceph] [35282.477940] hardirqs last enabled at (203062823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35282.482037] ? lock_is_held_type+0xaa/0x120 [35282.483957] hardirqs last disabled at (203062824): [] sysvec_apic_timer_interrupt+0xe/0x90 [35282.488656] vfs_write+0x324/0x4f0 [35282.492398] softirqs last enabled at (203062822): [] __do_softirq+0x2be/0x371 [35282.496927] ksys_pwrite64+0x65/0xa0 [35282.507166] softirqs last disabled at (203062817): [] irq_exit_rcu+0xb0/0xd0 [35282.511344] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35282.521154] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35282.524556] do_syscall_64+0x38/0x80 [35282.533322] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35282.536895] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35282.545491] Workqueue: events delayed_work [ceph] [35282.550621] RIP: 0033:0x7f48f0eeb93b [35282.560000] [35282.563571] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35282.571041] RIP: 0010:igrab+0x12/0x50 [35282.576086] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35282.580783] Code: 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 8d af 88 00 00 00 53 <48> 89 fb 48 89 ef e8 93 5d 7f 00 f6 83 d8 00 00 00 30 75 15 f0 ff [35282.584354] ORIG_RAX: 0000000000000012 [35282.585846] RSP: 0018:ffffc90002117da0 EFLAGS: 00000286 [35282.604581] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35282.608239] [35282.608240] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [35282.613457] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35282.632195] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [35282.636025] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35282.641242] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [35282.648367] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35282.649857] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [35282.656980] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35282.664104] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35282.671235] [35282.678352] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35282.724235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35282.729973] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35282.737097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35282.744221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35282.751345] Call Trace: [35282.753790] [35282.755801] ? watchdog_timer_fn+0x1b4/0x220 [35282.760066] ? __hrtimer_run_queues+0x1d3/0x360 [35282.764593] ? hrtimer_interrupt+0x104/0x240 [35282.768861] ? __local_bh_enable+0x50/0x60 [35282.772954] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35282.778170] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35282.783126] [35282.785225] [35282.787322] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35282.792630] ? igrab+0x12/0x50 [35282.795686] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35282.800842] delayed_work+0x1d8/0x280 [ceph] [35282.805117] ? process_scheduled_works+0x22a/0x510 [35282.809908] process_scheduled_works+0x2a4/0x510 [35282.814522] worker_thread+0x188/0x340 [35282.818272] ? rescuer_thread+0x310/0x310 [35282.822275] kthread+0xf5/0x130 [35282.825412] ? kthread_complete_and_exit+0x20/0x20 [35282.830198] ret_from_fork+0x31/0x40 [35282.833767] ? kthread_complete_and_exit+0x20/0x20 [35282.838559] ret_from_fork_asm+0x11/0x20 [35282.842484] [35310.076618] watchdog: BUG: soft lockup - CPU#0 stuck for 21771s! [dbench:78683] [35310.083915] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35310.088618] watchdog: BUG: soft lockup - CPU#7 stuck for 30768s! [kworker/7:4:78808] [35310.139356] irq event stamp: 61406910 [35310.139357] hardirqs last enabled at (61406909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35310.147088] Modules linked in: ceph [35310.150745] hardirqs last disabled at (61406910): [] sysvec_apic_timer_interrupt+0xe/0x90 [35310.160903] libceph [35310.164386] softirqs last enabled at (61406852): [] __do_softirq+0x2be/0x371 [35310.174110] dns_resolver [35310.176296] softirqs last disabled at (61406847): [] irq_exit_rcu+0xb0/0xd0 [35310.184979] fscache netfs [35310.187596] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35310.196108] veth [35310.198810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35310.207755] nft_chain_nat xt_MASQUERADE [35310.209679] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35310.217150] nf_nat nf_conntrack [35310.221069] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35310.226545] nf_defrag_ipv6 nf_defrag_ipv4 [35310.229770] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35310.248507] nft_compat [35310.252597] [35310.257815] nf_tables nfnetlink [35310.260258] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35310.261749] bridge stp [35310.264975] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35310.272097] llc xfs [35310.274542] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35310.281666] libcrc32c sunrpc [35310.283850] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35310.290974] intel_rapl_msr intel_rapl_common [35310.293938] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35310.301063] sb_edac x86_pkg_temp_thermal [35310.305414] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35310.312536] coretemp kvm_intel [35310.316542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35310.324619] kvm irqbypass [35310.327757] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35310.333493] i2c_i801 iTCO_wdt [35310.336199] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35310.343323] iTCO_vendor_support i2c_smbus [35310.346373] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35310.353497] crct10dif_pclmul [35310.357587] Call Trace: [35310.357589] [35310.364711] joydev crc32_pclmul [35310.367675] ? watchdog_timer_fn+0x1b4/0x220 [35310.370119] ipmi_ssif ghash_clmulni_intel [35310.372131] ? __hrtimer_run_queues+0x1d3/0x360 [35310.375354] mei_me mei [35310.379622] ? hrtimer_interrupt+0x104/0x240 [35310.383709] lpc_ich sha512_ssse3 [35310.388233] ? __local_bh_enable+0x50/0x60 [35310.390677] acpi_ipmi mfd_core [35310.394943] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35310.398251] wmi ipmi_si [35310.402344] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35310.405481] ipmi_devintf ipmi_msghandler [35310.410699] [35310.413228] ioatdma acpi_pad [35310.418185] [35310.422191] acpi_power_meter ixgbe [35310.424300] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35310.427260] igb mdio [35310.429361] ? queued_spin_lock_slowpath+0x18/0x2e0 [35310.432841] crc32c_intel nvme [35310.438147] do_raw_spin_lock+0xb5/0xc0 [35310.440417] ptp nvme_core [35310.445298] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35310.448346] i2c_algo_bit pps_core [35310.452181] try_get_cap_refs+0x159/0x880 [ceph] [35310.454881] dca fuse [35310.459670] __ceph_get_caps+0x4a1/0x760 [ceph] [35310.463062] [35310.463063] irq event stamp: 203283686 [35310.467678] ? __file_remove_privs+0x89/0x110 [35310.469943] hardirqs last enabled at (203283685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35310.474469] ? wake_bit_function+0x70/0x70 [35310.475959] hardirqs last disabled at (203283686): [] sysvec_apic_timer_interrupt+0xe/0x90 [35310.479706] ? ceph_write_iter+0x260/0x770 [ceph] [35310.484053] softirqs last enabled at (203283684): [] __do_softirq+0x2be/0x371 [35310.494299] ceph_write_iter+0x260/0x770 [ceph] [35310.498390] softirqs last disabled at (203283679): [] irq_exit_rcu+0xb0/0xd0 [35310.508203] ? lock_is_held_type+0xaa/0x120 [35310.512898] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35310.521672] vfs_write+0x324/0x4f0 [35310.526193] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35310.534795] ksys_pwrite64+0x65/0xa0 [35310.538968] Workqueue: events delayed_work [ceph] [35310.548343] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35310.551741] [35310.559215] do_syscall_64+0x38/0x80 [35310.562783] RIP: 0010:lock_acquire+0x39/0x2b0 [35310.567481] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35310.572612] Code: 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 <65> 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 0f 82 2b 01 00 [35310.574102] RIP: 0033:0x7f48f0eeb93b [35310.577672] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [35310.582023] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35310.587068] [35310.587069] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [35310.605805] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35310.609376] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [35310.614594] ORIG_RAX: 0000000000000012 [35310.633332] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [35310.634823] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35310.641947] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [35310.647163] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35310.654301] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35310.658135] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35310.665259] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35310.672384] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35310.679508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35310.686633] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35310.693756] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35310.700885] [35310.708959] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35310.745377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35310.752499] Call Trace: [35310.754945] [35310.756956] ? watchdog_timer_fn+0x1b4/0x220 [35310.761220] ? __hrtimer_run_queues+0x1d3/0x360 [35310.765746] ? hrtimer_interrupt+0x104/0x240 [35310.770023] ? __local_bh_enable+0x50/0x60 [35310.774118] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35310.779343] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35310.784302] [35310.786396] [35310.788495] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35310.793802] ? lock_acquire+0x39/0x2b0 [35310.797553] ? igrab+0x43/0x50 [35310.800603] ? lock_release+0xc5/0x260 [35310.804347] _raw_spin_lock+0x30/0x40 [35310.808022] ? igrab+0x1d/0x50 [35310.811071] igrab+0x1d/0x50 [35310.813949] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35310.819099] delayed_work+0x1d8/0x280 [ceph] [35310.823381] ? process_scheduled_works+0x22a/0x510 [35310.828171] process_scheduled_works+0x2a4/0x510 [35310.832785] worker_thread+0x188/0x340 [35310.836536] ? rescuer_thread+0x310/0x310 [35310.840539] kthread+0xf5/0x130 [35310.843676] ? kthread_complete_and_exit+0x20/0x20 [35310.848461] ret_from_fork+0x31/0x40 [35310.852031] ? kthread_complete_and_exit+0x20/0x20 [35310.856815] ret_from_fork_asm+0x11/0x20 [35310.860737] [35338.077080] watchdog: BUG: soft lockup - CPU#0 stuck for 21797s! [dbench:78683] [35338.084388] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35338.089080] watchdog: BUG: soft lockup - CPU#7 stuck for 30794s! [kworker/7:4:78808] [35338.139836] irq event stamp: 61465004 [35338.147566] Modules linked in: ceph libceph dns_resolver fscache netfs [35338.151223] hardirqs last enabled at (61465003): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35338.157757] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [35338.167917] hardirqs last disabled at (61465004): [] sysvec_apic_timer_interrupt+0xe/0x90 [35338.175301] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [35338.185026] softirqs last enabled at (61464968): [] __do_softirq+0x2be/0x371 [35338.185029] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [35338.200156] softirqs last disabled at (61464963): [] irq_exit_rcu+0xb0/0xd0 [35338.200159] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [35338.215186] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35338.222049] i2c_i801 iTCO_wdt iTCO_vendor_support [35338.230994] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35338.230996] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35338.235778] i2c_smbus [35338.243249] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35338.248726] crct10dif_pclmul joydev [35338.251083] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35338.269820] crc32_pclmul ipmi_ssif [35338.273391] [35338.273392] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35338.278608] ghash_clmulni_intel mei_me [35338.282093] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35338.283583] mei lpc_ich [35338.290708] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35338.294538] sha512_ssse3 acpi_ipmi [35338.301663] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35338.304193] mfd_core [35338.311317] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35338.314802] wmi ipmi_si [35338.321926] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35338.324197] ipmi_devintf ipmi_msghandler [35338.331321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35338.333851] ioatdma acpi_pad [35338.341929] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35338.345932] acpi_power_meter ixgbe [35338.351670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35338.354634] igb mdio [35338.361758] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35338.365243] crc32c_intel [35338.372368] Call Trace: [35338.372370] [35338.374636] nvme ptp [35338.381772] ? watchdog_timer_fn+0x1b4/0x220 [35338.384386] nvme_core i2c_algo_bit [35338.386833] ? __hrtimer_run_queues+0x1d3/0x360 [35338.388844] pps_core [35338.391117] ? hrtimer_interrupt+0x104/0x240 [35338.395377] dca fuse [35338.398861] ? __local_bh_enable+0x50/0x60 [35338.403386] [35338.403387] irq event stamp: 203504482 [35338.405659] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35338.409921] hardirqs last enabled at (203504481): [] asm_common_interrupt+0x26/0x40 [35338.412192] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35338.416281] hardirqs last disabled at (203504482): [] sysvec_apic_timer_interrupt+0xe/0x90 [35338.417772] [35338.421515] softirqs last enabled at (203504480): [] __do_softirq+0x2be/0x371 [35338.426733] [35338.436041] softirqs last disabled at (203504475): [] irq_exit_rcu+0xb0/0xd0 [35338.441000] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35338.450811] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35338.452912] ? queued_spin_lock_slowpath+0x16/0x2e0 [35338.461679] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35338.463785] do_raw_spin_lock+0xb5/0xc0 [35338.472383] Workqueue: events delayed_work [ceph] [35338.477689] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35338.487063] [35338.487064] RIP: 0010:lock_acquire+0xf0/0x2b0 [35338.491937] try_get_cap_refs+0x159/0x880 [ceph] [35338.499406] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35338.503241] __ceph_get_caps+0x4a1/0x760 [ceph] [35338.507933] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35338.512723] ? __file_remove_privs+0x89/0x110 [35338.514209] [35338.514209] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35338.518560] ? wake_bit_function+0x70/0x70 [35338.523168] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35338.541909] ? ceph_write_iter+0x260/0x770 [ceph] [35338.546430] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35338.551649] ceph_write_iter+0x260/0x770 [ceph] [35338.556000] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35338.557495] ? lock_is_held_type+0xaa/0x120 [35338.564614] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35338.568708] vfs_write+0x324/0x4f0 [35338.575828] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35338.580531] ksys_pwrite64+0x65/0xa0 [35338.587651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35338.592176] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35338.599298] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35338.603483] do_syscall_64+0x38/0x80 [35338.610609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35338.614008] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35338.622084] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35338.625654] RIP: 0033:0x7f48f0eeb93b [35338.631392] Call Trace: [35338.636523] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35338.643648] [35338.647217] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35338.654344] ? watchdog_timer_fn+0x1b4/0x220 [35338.659385] ORIG_RAX: 0000000000000012 [35338.666512] ? __hrtimer_run_queues+0x1d3/0x360 [35338.670080] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35338.672529] ? hrtimer_interrupt+0x104/0x240 [35338.691263] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35338.693272] ? __local_bh_enable+0x50/0x60 [35338.698489] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35338.702769] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35338.706594] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35338.711118] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35338.718241] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35338.722507] [35338.729637] [35338.733721] [35338.733723] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35338.776957] ? lock_acquire+0xf0/0x2b0 [35338.780706] ? igrab+0x43/0x50 [35338.783755] ? lock_release+0xc5/0x260 [35338.787500] _raw_spin_lock+0x30/0x40 [35338.791157] ? igrab+0x1d/0x50 [35338.794206] igrab+0x1d/0x50 [35338.797086] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35338.802236] delayed_work+0x1d8/0x280 [ceph] [35338.806517] ? process_scheduled_works+0x22a/0x510 [35338.811306] process_scheduled_works+0x2a4/0x510 [35338.815923] worker_thread+0x188/0x340 [35338.819672] ? rescuer_thread+0x310/0x310 [35338.823675] kthread+0xf5/0x130 [35338.826821] ? kthread_complete_and_exit+0x20/0x20 [35338.831606] ret_from_fork+0x31/0x40 [35338.835185] ? kthread_complete_and_exit+0x20/0x20 [35338.839969] ret_from_fork_asm+0x11/0x20 [35338.843902] [35366.077542] watchdog: BUG: soft lockup - CPU#0 stuck for 21823s! [dbench:78683] [35366.084847] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35366.089542] watchdog: BUG: soft lockup - CPU#7 stuck for 30820s! [kworker/7:4:78808] [35366.140290] irq event stamp: 61523020 [35366.140291] hardirqs last enabled at (61523019): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35366.148019] Modules linked in: ceph [35366.151676] hardirqs last disabled at (61523020): [] sysvec_apic_timer_interrupt+0xe/0x90 [35366.161833] libceph [35366.165319] softirqs last enabled at (61522962): [] __do_softirq+0x2be/0x371 [35366.175042] dns_resolver [35366.177227] softirqs last disabled at (61522957): [] irq_exit_rcu+0xb0/0xd0 [35366.185911] fscache netfs [35366.188546] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35366.197056] veth [35366.199760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35366.208704] nft_chain_nat [35366.210630] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35366.218100] xt_MASQUERADE nf_nat [35366.220805] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35366.226281] nf_conntrack nf_defrag_ipv6 [35366.229591] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35366.248329] nf_defrag_ipv4 nft_compat [35366.252246] [35366.257464] nf_tables nfnetlink [35366.261219] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35366.262708] bridge stp [35366.265937] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35366.273064] llc [35366.275508] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35366.282633] xfs [35366.284469] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35366.291593] libcrc32c sunrpc [35366.293432] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35366.300554] intel_rapl_msr intel_rapl_common [35366.303519] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35366.310643] sb_edac x86_pkg_temp_thermal [35366.314995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35366.323072] coretemp kvm_intel [35366.327075] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35366.332813] kvm irqbypass [35366.335951] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35366.343074] i2c_i801 iTCO_wdt [35366.345778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35366.352903] iTCO_vendor_support i2c_smbus [35366.355954] Call Trace: [35366.355955] [35366.363077] crct10dif_pclmul joydev [35366.367170] ? watchdog_timer_fn+0x1b4/0x220 [35366.369612] crc32_pclmul ipmi_ssif [35366.371625] ? __hrtimer_run_queues+0x1d3/0x360 [35366.375195] ghash_clmulni_intel mei_me [35366.379478] ? hrtimer_interrupt+0x104/0x240 [35366.382958] mei [35366.387483] ? __local_bh_enable+0x50/0x60 [35366.391314] lpc_ich sha512_ssse3 [35366.395581] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35366.397416] acpi_ipmi mfd_core [35366.401508] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35366.404818] wmi [35366.410034] [35366.413172] ipmi_si ipmi_devintf [35366.418130] [35366.418131] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35366.419967] ipmi_msghandler ioatdma [35366.422068] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35366.425375] acpi_pad acpi_power_meter [35366.427475] do_raw_spin_lock+0xb5/0xc0 [35366.432777] ixgbe igb [35366.436348] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35366.441222] mdio crc32c_intel [35366.444964] try_get_cap_refs+0x159/0x880 [ceph] [35366.448793] nvme ptp [35366.451155] __ceph_get_caps+0x4a1/0x760 [ceph] [35366.455941] nvme_core i2c_algo_bit [35366.458999] ? __file_remove_privs+0x89/0x110 [35366.463605] pps_core dca [35366.465877] ? wake_bit_function+0x70/0x70 [35366.470399] fuse [35366.473887] ? ceph_write_iter+0x260/0x770 [ceph] [35366.478234] irq event stamp: 203725344 [35366.480852] ceph_write_iter+0x260/0x770 [ceph] [35366.484943] hardirqs last enabled at (203725343): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35366.486871] ? lock_is_held_type+0xaa/0x120 [35366.491564] hardirqs last disabled at (203725344): [] sysvec_apic_timer_interrupt+0xe/0x90 [35366.495311] vfs_write+0x324/0x4f0 [35366.499831] softirqs last enabled at (203725342): [] __do_softirq+0x2be/0x371 [35366.510080] ksys_pwrite64+0x65/0xa0 [35366.514253] softirqs last disabled at (203725337): [] irq_exit_rcu+0xb0/0xd0 [35366.524063] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35366.527463] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35366.536233] do_syscall_64+0x38/0x80 [35366.539804] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35366.548403] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35366.553533] Workqueue: events delayed_work [ceph] [35366.562908] RIP: 0033:0x7f48f0eeb93b [35366.566479] [35366.573950] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35366.578995] RIP: 0010:lock_acquire+0xf0/0x2b0 [35366.583693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35366.587262] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35366.588753] ORIG_RAX: 0000000000000012 [35366.607491] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35366.611841] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35366.617059] [35366.617060] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35366.635798] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35366.639627] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35366.644844] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35366.651970] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35366.653459] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35366.660585] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35366.667708] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35366.674833] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35366.681963] [35366.689082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35366.727857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35366.733593] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35366.740719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35366.747841] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35366.754965] Call Trace: [35366.757411] [35366.759421] ? watchdog_timer_fn+0x1b4/0x220 [35366.763686] ? __hrtimer_run_queues+0x1d3/0x360 [35366.768213] ? hrtimer_interrupt+0x104/0x240 [35366.772490] ? __local_bh_enable+0x50/0x60 [35366.776583] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35366.781799] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35366.786756] [35366.788852] [35366.790950] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35366.796259] ? lock_acquire+0xf0/0x2b0 [35366.800009] ? igrab+0x43/0x50 [35366.803058] ? lock_release+0xc5/0x260 [35366.806805] _raw_spin_lock+0x30/0x40 [35366.810470] ? igrab+0x1d/0x50 [35366.813518] igrab+0x1d/0x50 [35366.816399] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35366.821546] delayed_work+0x1d8/0x280 [ceph] [35366.825829] ? process_scheduled_works+0x22a/0x510 [35366.830619] process_scheduled_works+0x2a4/0x510 [35366.835242] worker_thread+0x188/0x340 [35366.838994] ? rescuer_thread+0x310/0x310 [35366.843003] kthread+0xf5/0x130 [35366.846140] ? kthread_complete_and_exit+0x20/0x20 [35366.850945] ret_from_fork+0x31/0x40 [35366.854522] ? kthread_complete_and_exit+0x20/0x20 [35366.859305] ret_from_fork_asm+0x11/0x20 [35366.863235] [35394.077995] watchdog: BUG: soft lockup - CPU#0 stuck for 21849s! [dbench:78683] [35394.085303] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35394.089995] watchdog: BUG: soft lockup - CPU#7 stuck for 30846s! [kworker/7:4:78808] [35394.140750] irq event stamp: 61581232 [35394.148482] Modules linked in: ceph libceph dns_resolver fscache netfs veth [35394.152139] hardirqs last enabled at (61581231): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35394.159098] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [35394.169255] hardirqs last disabled at (61581232): [] sysvec_apic_timer_interrupt+0xe/0x90 [35394.176207] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [35394.185930] softirqs last enabled at (61581174): [] __do_softirq+0x2be/0x371 [35394.192362] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [35394.201044] softirqs last disabled at (61581169): [] irq_exit_rcu+0xb0/0xd0 [35394.207563] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [35394.216074] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35394.222072] irqbypass i2c_i801 [35394.231016] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35394.231017] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35394.234151] iTCO_wdt iTCO_vendor_support [35394.241625] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35394.247101] i2c_smbus crct10dif_pclmul [35394.251104] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35394.269842] joydev crc32_pclmul [35394.273675] [35394.278890] ipmi_ssif ghash_clmulni_intel [35394.282116] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35394.283605] mei_me mei [35394.287695] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35394.294821] lpc_ich [35394.297265] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35394.304393] sha512_ssse3 [35394.306582] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35394.313704] acpi_ipmi mfd_core [35394.316322] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35394.323446] wmi [35394.326584] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35394.333708] ipmi_si ipmi_devintf [35394.335545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35394.343622] ipmi_msghandler ioatdma [35394.346934] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35394.352683] acpi_pad [35394.356251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35394.363394] acpi_power_meter [35394.365674] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35394.372796] ixgbe igb [35394.375760] Call Trace: [35394.375762] [35394.382885] mdio crc32c_intel [35394.385244] ? watchdog_timer_fn+0x1b4/0x220 [35394.387685] nvme ptp [35394.389696] ? __hrtimer_run_queues+0x1d3/0x360 [35394.392746] nvme_core i2c_algo_bit [35394.397015] ? hrtimer_interrupt+0x104/0x240 [35394.399281] pps_core [35394.403806] ? __local_bh_enable+0x50/0x60 [35394.407289] dca fuse [35394.411555] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35394.413826] [35394.413826] irq event stamp: 203946076 [35394.417917] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35394.420186] hardirqs last enabled at (203946075): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35394.425402] [35394.426895] hardirqs last disabled at (203946076): [] sysvec_apic_timer_interrupt+0xe/0x90 [35394.430638] [35394.435595] softirqs last enabled at (203946074): [] __do_softirq+0x2be/0x371 [35394.445841] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35394.447936] softirqs last disabled at (203946069): [] irq_exit_rcu+0xb0/0xd0 [35394.457752] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35394.459846] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35394.468619] do_raw_spin_lock+0xb5/0xc0 [35394.473920] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35394.482518] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35394.487397] Workqueue: events delayed_work [ceph] [35394.496777] try_get_cap_refs+0x159/0x880 [ceph] [35394.500606] [35394.500607] RIP: 0010:lock_acquire+0xf0/0x2b0 [35394.508082] __ceph_get_caps+0x4a1/0x760 [ceph] [35394.512861] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35394.517562] ? __file_remove_privs+0x89/0x110 [35394.522168] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35394.523659] ? wake_bit_function+0x70/0x70 [35394.528009] [35394.528010] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35394.532537] ? ceph_write_iter+0x260/0x770 [ceph] [35394.551271] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35394.555623] ceph_write_iter+0x260/0x770 [ceph] [35394.560841] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35394.564933] ? lock_is_held_type+0xaa/0x120 [35394.566422] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35394.573547] vfs_write+0x324/0x4f0 [35394.578242] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35394.585374] ksys_pwrite64+0x65/0xa0 [35394.589891] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35394.597015] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35394.601194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35394.608319] do_syscall_64+0x38/0x80 [35394.611714] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35394.618841] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35394.622410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35394.630486] RIP: 0033:0x7f48f0eeb93b [35394.635617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35394.641355] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35394.644926] Call Trace: [35394.644928] [35394.652050] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35394.657094] ? watchdog_timer_fn+0x1b4/0x220 [35394.664218] ORIG_RAX: 0000000000000012 [35394.667791] ? __hrtimer_run_queues+0x1d3/0x360 [35394.674911] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35394.693654] ? hrtimer_interrupt+0x104/0x240 [35394.696094] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35394.698105] ? __local_bh_enable+0x50/0x60 [35394.703323] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35394.707591] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35394.711417] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35394.715942] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35394.723064] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35394.727331] [35394.734460] [35394.738545] [35394.776499] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35394.781804] ? lock_acquire+0xf0/0x2b0 [35394.785555] ? igrab+0x43/0x50 [35394.788604] ? lock_release+0xc5/0x260 [35394.792351] _raw_spin_lock+0x30/0x40 [35394.796014] ? igrab+0x1d/0x50 [35394.799066] igrab+0x1d/0x50 [35394.801946] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35394.807093] delayed_work+0x1d8/0x280 [ceph] [35394.811377] ? process_scheduled_works+0x22a/0x510 [35394.816182] process_scheduled_works+0x2a4/0x510 [35394.820799] worker_thread+0x188/0x340 [35394.824547] ? rescuer_thread+0x310/0x310 [35394.828549] kthread+0xf5/0x130 [35394.831691] ? kthread_complete_and_exit+0x20/0x20 [35394.836481] ret_from_fork+0x31/0x40 [35394.840060] ? kthread_complete_and_exit+0x20/0x20 [35394.844843] ret_from_fork_asm+0x11/0x20 [35394.848767] [35422.078444] watchdog: BUG: soft lockup - CPU#0 stuck for 21875s! [dbench:78683] [35422.085745] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35422.090444] watchdog: BUG: soft lockup - CPU#7 stuck for 30872s! [kworker/7:4:78808] [35422.141185] irq event stamp: 61639220 [35422.141186] hardirqs last enabled at (61639219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35422.148916] Modules linked in: [35422.152574] hardirqs last disabled at (61639220): [] sysvec_apic_timer_interrupt+0xe/0x90 [35422.162731] ceph [35422.165781] softirqs last enabled at (61639162): [] __do_softirq+0x2be/0x371 [35422.175506] libceph [35422.177431] softirqs last disabled at (61639157): [] irq_exit_rcu+0xb0/0xd0 [35422.186119] dns_resolver [35422.188307] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35422.196818] fscache netfs [35422.199452] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35422.208398] veth nft_chain_nat [35422.211118] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35422.218589] xt_MASQUERADE [35422.221727] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35422.227203] nf_nat [35422.229908] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35422.248645] nf_conntrack nf_defrag_ipv6 [35422.250742] [35422.250743] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35422.255962] nf_defrag_ipv4 [35422.259879] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35422.261368] nft_compat [35422.268491] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35422.271284] nf_tables nfnetlink [35422.278409] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35422.280852] bridge [35422.287976] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35422.291199] stp llc [35422.298324] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35422.300421] xfs libcrc32c [35422.307546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35422.309729] sunrpc intel_rapl_msr [35422.317808] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35422.320512] intel_rapl_common [35422.326249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35422.329645] sb_edac x86_pkg_temp_thermal [35422.336770] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35422.339838] coretemp [35422.346961] Call Trace: [35422.346962] [35422.350965] kvm_intel kvm [35422.358092] ? watchdog_timer_fn+0x1b4/0x220 [35422.360360] irqbypass [35422.362805] ? __hrtimer_run_queues+0x1d3/0x360 [35422.364815] i2c_i801 iTCO_wdt [35422.367524] ? hrtimer_interrupt+0x104/0x240 [35422.371784] iTCO_vendor_support [35422.374140] ? __local_bh_enable+0x50/0x60 [35422.378666] i2c_smbus [35422.381720] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35422.385981] crct10dif_pclmul joydev [35422.389205] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35422.393296] crc32_pclmul ipmi_ssif [35422.395652] [35422.400869] ghash_clmulni_intel [35422.404440] [35422.409397] mei_me mei [35422.412884] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35422.414980] lpc_ich sha512_ssse3 [35422.418206] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35422.420301] acpi_ipmi [35422.422746] do_raw_spin_lock+0xb5/0xc0 [35422.428050] mfd_core wmi [35422.431360] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35422.436231] ipmi_si ipmi_devintf [35422.438589] try_get_cap_refs+0x159/0x880 [ceph] [35422.442418] ipmi_msghandler ioatdma [35422.445040] __ceph_get_caps+0x4a1/0x760 [ceph] [35422.449839] acpi_pad [35422.453151] ? __file_remove_privs+0x89/0x110 [35422.457760] acpi_power_meter ixgbe [35422.461330] ? wake_bit_function+0x70/0x70 [35422.465854] igb [35422.468127] ? ceph_write_iter+0x260/0x770 [ceph] [35422.472474] mdio crc32c_intel [35422.475958] ceph_write_iter+0x260/0x770 [ceph] [35422.480051] nvme ptp [35422.481892] ? lock_is_held_type+0xaa/0x120 [35422.486584] nvme_core [35422.489637] vfs_write+0x324/0x4f0 [35422.494159] i2c_algo_bit pps_core [35422.496434] ksys_pwrite64+0x65/0xa0 [35422.500608] dca [35422.502966] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35422.506362] fuse [35422.509761] do_syscall_64+0x38/0x80 [35422.513330] irq event stamp: 204166926 [35422.515169] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35422.520299] hardirqs last enabled at (204166925): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35422.522224] RIP: 0033:0x7f48f0eeb93b [35422.525793] hardirqs last disabled at (204166926): [] sysvec_apic_timer_interrupt+0xe/0x90 [35422.529537] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35422.534581] softirqs last enabled at (204166924): [] __do_softirq+0x2be/0x371 [35422.544843] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35422.548413] softirqs last disabled at (204166919): [] irq_exit_rcu+0xb0/0xd0 [35422.558226] ORIG_RAX: 0000000000000012 [35422.576962] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35422.585732] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35422.590950] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35422.599548] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35422.603379] Workqueue: events delayed_work [ceph] [35422.612757] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35422.619880] [35422.627352] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35422.634474] RIP: 0010:lock_acquire+0xf0/0x2b0 [35422.639174] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35422.646298] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35422.647793] [35422.654913] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35422.692544] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35422.699668] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35422.706792] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35422.713916] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35422.721041] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35422.728165] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35422.736241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35422.741978] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35422.749103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35422.756246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35422.763368] Call Trace: [35422.765813] [35422.767824] ? watchdog_timer_fn+0x1b4/0x220 [35422.772105] ? __hrtimer_run_queues+0x1d3/0x360 [35422.776649] ? hrtimer_interrupt+0x104/0x240 [35422.780910] ? __local_bh_enable+0x50/0x60 [35422.785005] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35422.790228] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35422.795185] [35422.797282] [35422.799381] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35422.804689] ? lock_acquire+0xf0/0x2b0 [35422.808438] ? igrab+0x43/0x50 [35422.811487] ? lock_release+0xc5/0x260 [35422.815234] _raw_spin_lock+0x30/0x40 [35422.818898] ? igrab+0x1d/0x50 [35422.821949] igrab+0x1d/0x50 [35422.824828] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35422.829994] delayed_work+0x1d8/0x280 [ceph] [35422.834274] ? process_scheduled_works+0x22a/0x510 [35422.839066] process_scheduled_works+0x2a4/0x510 [35422.843682] worker_thread+0x188/0x340 [35422.847430] ? rescuer_thread+0x310/0x310 [35422.851431] kthread+0xf5/0x130 [35422.854572] ? kthread_complete_and_exit+0x20/0x20 [35422.859354] ret_from_fork+0x31/0x40 [35422.862934] ? kthread_complete_and_exit+0x20/0x20 [35422.867719] ret_from_fork_asm+0x11/0x20 [35422.871651] [35450.078890] watchdog: BUG: soft lockup - CPU#0 stuck for 21901s! [dbench:78683] [35450.086199] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35450.090890] watchdog: BUG: soft lockup - CPU#7 stuck for 30898s! [kworker/7:4:78808] [35450.141648] irq event stamp: 61697454 [35450.149379] Modules linked in: ceph libceph dns_resolver fscache netfs [35450.153035] hardirqs last enabled at (61697453): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35450.159567] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [35450.169728] hardirqs last disabled at (61697454): [] sysvec_apic_timer_interrupt+0xe/0x90 [35450.177112] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [35450.186835] softirqs last enabled at (61697428): [] __do_softirq+0x2be/0x371 [35450.186839] stp llc xfs libcrc32c sunrpc intel_rapl_msr [35450.201605] softirqs last disabled at (61697423): [] irq_exit_rcu+0xb0/0xd0 [35450.201608] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [35450.215427] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35450.215428] kvm_intel [35450.215430] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35450.221771] kvm irqbypass [35450.230717] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35450.233074] i2c_i801 iTCO_wdt [35450.240546] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35450.243248] iTCO_vendor_support [35450.248725] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35450.251776] i2c_smbus [35450.270515] [35450.273739] crct10dif_pclmul [35450.278957] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35450.281314] joydev [35450.282805] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35450.285767] crc32_pclmul ipmi_ssif [35450.292893] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35450.294990] ghash_clmulni_intel mei_me [35450.302112] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35450.305598] mei lpc_ich [35450.312722] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35450.316569] sha512_ssse3 acpi_ipmi [35450.323693] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35450.326225] mfd_core [35450.333348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35450.336832] wmi ipmi_si [35450.344911] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35450.347181] ipmi_devintf ipmi_msghandler [35450.352919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35450.355448] ioatdma [35450.362575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35450.366578] acpi_pad [35450.373700] Call Trace: [35450.373703] [35450.375884] acpi_power_meter [35450.383011] ? watchdog_timer_fn+0x1b4/0x220 [35450.385287] ixgbe [35450.387733] ? __hrtimer_run_queues+0x1d3/0x360 [35450.389744] igb mdio [35450.392711] ? hrtimer_interrupt+0x104/0x240 [35450.396973] crc32c_intel [35450.398982] ? __local_bh_enable+0x50/0x60 [35450.403507] nvme ptp [35450.405781] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35450.410041] nvme_core [35450.412659] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35450.416750] i2c_algo_bit [35450.419020] [35450.424238] pps_core [35450.426596] [35450.431551] dca fuse [35450.434171] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35450.436284] [35450.438574] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35450.440671] irq event stamp: 204387566 [35450.442942] do_raw_spin_lock+0xb5/0xc0 [35450.448246] hardirqs last enabled at (204387565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35450.449738] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35450.454606] hardirqs last disabled at (204387566): [] sysvec_apic_timer_interrupt+0xe/0x90 [35450.458353] try_get_cap_refs+0x159/0x880 [ceph] [35450.462182] softirqs last enabled at (204387564): [] __do_softirq+0x2be/0x371 [35450.472431] __ceph_get_caps+0x4a1/0x760 [ceph] [35450.477209] softirqs last disabled at (204387555): [] irq_exit_rcu+0xb0/0xd0 [35450.487026] ? __file_remove_privs+0x89/0x110 [35450.491632] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35450.500402] ? wake_bit_function+0x70/0x70 [35450.504927] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35450.513526] ? ceph_write_iter+0x260/0x770 [ceph] [35450.517874] Workqueue: events delayed_work [ceph] [35450.527253] ceph_write_iter+0x260/0x770 [ceph] [35450.531343] [35450.538817] ? lock_is_held_type+0xaa/0x120 [35450.543511] RIP: 0010:do_raw_spin_lock+0x37/0xc0 [35450.548211] vfs_write+0x324/0x4f0 [35450.552732] Code: 48 89 fb 3d ad 4e ad de 75 46 48 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a <31> c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 [35450.554227] ksys_pwrite64+0x65/0xa0 [35450.558400] RSP: 0018:ffffc90002117d90 EFLAGS: 00000282 [35450.563011] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35450.566409] [35450.585149] do_syscall_64+0x38/0x80 [35450.588717] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [35450.593936] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35450.599064] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff8881a1d34500 [35450.600557] RIP: 0033:0x7f48f0eeb93b [35450.604127] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [35450.611252] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35450.616296] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [35450.623418] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35450.626989] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35450.634113] ORIG_RAX: 0000000000000012 [35450.652851] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35450.659976] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35450.665193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35450.672317] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35450.676147] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35450.684226] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35450.691351] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35450.697086] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35450.704212] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35450.711335] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35450.718460] Call Trace: [35450.718461] [35450.725594] [35450.732717] ? watchdog_timer_fn+0x1b4/0x220 [35450.757868] ? __hrtimer_run_queues+0x1d3/0x360 [35450.762395] ? hrtimer_interrupt+0x104/0x240 [35450.766663] ? __local_bh_enable+0x50/0x60 [35450.770757] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35450.775981] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35450.780939] [35450.783035] [35450.785133] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35450.790442] ? do_raw_spin_lock+0x37/0xc0 [35450.794452] igrab+0x1d/0x50 [35450.797331] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35450.802480] delayed_work+0x1d8/0x280 [ceph] [35450.806760] ? process_scheduled_works+0x22a/0x510 [35450.811549] process_scheduled_works+0x2a4/0x510 [35450.816183] worker_thread+0x188/0x340 [35450.819932] ? rescuer_thread+0x310/0x310 [35450.823934] kthread+0xf5/0x130 [35450.827074] ? kthread_complete_and_exit+0x20/0x20 [35450.831865] ret_from_fork+0x31/0x40 [35450.835445] ? kthread_complete_and_exit+0x20/0x20 [35450.840230] ret_from_fork_asm+0x11/0x20 [35450.844161] [35451.974286] rcu: INFO: rcu_preempt self-detected stall on CPU [35451.980048] rcu: 0-....: (22983931 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11507392 [35451.990560] rcu: (t=23514949 jiffies g=1309609 q=656509 ncpus=8) [35451.996645] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35452.005587] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35452.013076] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [35452.018553] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [35452.037291] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35452.042525] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35452.049649] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35452.056772] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35452.063896] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35452.071021] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35452.078147] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35452.086223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35452.091960] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35452.099084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35452.106209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35452.113332] Call Trace: [35452.115778] [35452.117789] ? rcu_dump_cpu_stacks+0xd1/0x180 [35452.122148] ? rcu_sched_clock_irq+0x633/0x1390 [35452.126672] ? lock_is_held_type+0xaa/0x120 [35452.130850] ? lock_is_held_type+0xaa/0x120 [35452.135028] ? tick_sched_do_timer+0xa0/0xa0 [35452.139315] ? rcu_read_lock_sched_held+0x56/0x90 [35452.144016] ? tick_sched_do_timer+0xa0/0xa0 [35452.148279] ? update_process_times+0x5f/0x90 [35452.152629] ? tick_sched_handle.isra.16+0x1e/0x50 [35452.157412] ? tick_sched_timer+0x6f/0x80 [35452.161418] ? __hrtimer_run_queues+0x1d3/0x360 [35452.165943] ? hrtimer_interrupt+0x104/0x240 [35452.170205] ? __local_bh_enable+0x50/0x60 [35452.174314] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35452.179530] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35452.184488] [35452.186585] [35452.188700] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35452.194007] ? queued_spin_lock_slowpath+0x2b/0x2e0 [35452.198885] do_raw_spin_lock+0xb5/0xc0 [35452.202715] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35452.207518] try_get_cap_refs+0x159/0x880 [ceph] [35452.212148] __ceph_get_caps+0x4a1/0x760 [ceph] [35452.216689] ? __file_remove_privs+0x89/0x110 [35452.221044] ? wake_bit_function+0x70/0x70 [35452.225138] ? ceph_write_iter+0x260/0x770 [ceph] [35452.229849] ceph_write_iter+0x260/0x770 [ceph] [35452.234386] ? lock_is_held_type+0xaa/0x120 [35452.238572] vfs_write+0x324/0x4f0 [35452.241979] ksys_pwrite64+0x65/0xa0 [35452.245554] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35452.250686] do_syscall_64+0x38/0x80 [35452.254256] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35452.259316] RIP: 0033:0x7f48f0eeb93b [35452.262888] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35452.281626] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [35452.289182] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35452.296317] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35452.303441] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35452.310564] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35452.317689] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35452.324817] [35452.327006] Sending NMI from CPU 0 to CPUs 7: [35452.331359] NMI backtrace for cpu 7 [35452.331360] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35452.331363] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35452.331364] Workqueue: events delayed_work [ceph] [35452.331384] RIP: 0010:rcu_is_watching+0xb/0x40 [35452.331387] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [35452.331389] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [35452.331391] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [35452.331393] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [35452.331394] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [35452.331395] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [35452.331396] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35452.331398] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35452.331400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35452.331401] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35452.331403] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35452.331404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35452.331405] Call Trace: [35452.331406] [35452.331407] ? nmi_cpu_backtrace+0xad/0x120 [35452.331413] ? nmi_cpu_backtrace_handler+0x11/0x20 [35452.331415] ? nmi_handle+0xd4/0x260 [35452.331421] ? default_do_nmi+0x49/0xf0 [35452.331424] ? exc_nmi+0x11e/0x150 [35452.331426] ? end_repeat_nmi+0x16/0x67 [35452.331430] ? igrab+0x43/0x50 [35452.331434] ? igrab+0x43/0x50 [35452.331436] ? rcu_is_watching+0xb/0x40 [35452.331439] ? rcu_is_watching+0xb/0x40 [35452.331443] ? rcu_is_watching+0xb/0x40 [35452.331446] [35452.331446] [35452.331447] lock_release+0x1d7/0x260 [35452.331451] _raw_spin_unlock+0x1b/0x40 [35452.331455] igrab+0x43/0x50 [35452.331457] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35452.331477] delayed_work+0x1d8/0x280 [ceph] [35452.331497] ? process_scheduled_works+0x22a/0x510 [35452.331501] process_scheduled_works+0x2a4/0x510 [35452.331507] worker_thread+0x188/0x340 [35452.331511] ? rescuer_thread+0x310/0x310 [35452.331514] kthread+0xf5/0x130 [35452.331517] ? kthread_complete_and_exit+0x20/0x20 [35452.331521] ret_from_fork+0x31/0x40 [35452.331524] ? kthread_complete_and_exit+0x20/0x20 [35452.331527] ret_from_fork_asm+0x11/0x20 [35452.331535] [35478.079329] watchdog: BUG: soft lockup - CPU#0 stuck for 21927s! [dbench:78683] [35478.086634] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35478.091329] watchdog: BUG: soft lockup - CPU#7 stuck for 30924s! [kworker/7:4:78808] [35478.142074] irq event stamp: 61754724 [35478.142076] hardirqs last enabled at (61754723): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35478.149806] Modules linked in: [35478.153461] hardirqs last disabled at (61754724): [] sysvec_apic_timer_interrupt+0xe/0x90 [35478.163621] ceph [35478.166671] softirqs last enabled at (61754650): [] __do_softirq+0x2be/0x371 [35478.176396] libceph [35478.178319] softirqs last disabled at (61754645): [] irq_exit_rcu+0xb0/0xd0 [35478.187004] dns_resolver [35478.189186] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35478.197697] fscache [35478.200332] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35478.209276] netfs [35478.211460] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35478.218930] veth nft_chain_nat [35478.220943] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35478.226420] xt_MASQUERADE [35478.229557] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35478.248295] nf_nat nf_conntrack [35478.251000] [35478.251001] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35478.256217] nf_defrag_ipv6 nf_defrag_ipv4 [35478.259440] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35478.260931] nft_compat [35478.268055] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35478.272146] nf_tables [35478.279269] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35478.281725] nfnetlink bridge [35478.288847] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35478.291205] stp llc [35478.298328] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35478.301292] xfs libcrc32c [35478.308417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35478.310601] sunrpc intel_rapl_msr [35478.318679] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35478.321382] intel_rapl_common [35478.327121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35478.330518] sb_edac x86_pkg_temp_thermal [35478.337640] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35478.340691] coretemp [35478.347817] Call Trace: [35478.347818] [35478.351819] kvm_intel kvm [35478.358944] ? watchdog_timer_fn+0x1b4/0x220 [35478.361215] irqbypass [35478.363660] ? __hrtimer_run_queues+0x1d3/0x360 [35478.365668] i2c_i801 [35478.368378] ? hrtimer_interrupt+0x104/0x240 [35478.372637] iTCO_wdt iTCO_vendor_support [35478.375008] ? __local_bh_enable+0x50/0x60 [35478.379537] i2c_smbus crct10dif_pclmul [35478.381811] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35478.386072] joydev crc32_pclmul [35478.390076] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35478.394165] ipmi_ssif ghash_clmulni_intel [35478.397996] [35478.403231] mei_me mei [35478.406456] [35478.406458] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35478.411413] lpc_ich sha512_ssse3 [35478.415508] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35478.417602] acpi_ipmi [35478.420049] do_raw_spin_lock+0xb5/0xc0 [35478.422142] mfd_core [35478.427449] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35478.430759] wmi ipmi_si [35478.435630] try_get_cap_refs+0x159/0x880 [ceph] [35478.437987] ipmi_devintf [35478.441838] __ceph_get_caps+0x4a1/0x760 [ceph] [35478.444103] ipmi_msghandler ioatdma [35478.448893] ? __file_remove_privs+0x89/0x110 [35478.451420] acpi_pad [35478.456031] ? wake_bit_function+0x70/0x70 [35478.458648] acpi_power_meter ixgbe [35478.463174] ? ceph_write_iter+0x260/0x770 [ceph] [35478.466743] igb [35478.471092] ceph_write_iter+0x260/0x770 [ceph] [35478.473363] mdio crc32c_intel [35478.477457] ? lock_is_held_type+0xaa/0x120 [35478.480940] nvme ptp [35478.485638] vfs_write+0x324/0x4f0 [35478.487473] nvme_core [35478.492011] ksys_pwrite64+0x65/0xa0 [35478.495056] i2c_algo_bit pps_core [35478.499234] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35478.501504] dca fuse [35478.504904] do_syscall_64+0x38/0x80 [35478.507260] [35478.507261] irq event stamp: 204606588 [35478.510831] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35478.514227] hardirqs last enabled at (204606587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35478.519360] RIP: 0033:0x7f48f0eeb93b [35478.521629] hardirqs last disabled at (204606588): [] sysvec_apic_timer_interrupt+0xe/0x90 [35478.525200] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35478.526690] softirqs last enabled at (204606586): [] __do_softirq+0x2be/0x371 [35478.530435] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35478.535479] softirqs last disabled at (204606581): [] irq_exit_rcu+0xb0/0xd0 [35478.545727] ORIG_RAX: 0000000000000012 [35478.545729] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35478.549302] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35478.559114] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35478.577851] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35478.586623] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35478.591840] Workqueue: events delayed_work [ceph] [35478.600436] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35478.604267] [35478.604268] RIP: 0010:lock_acquire+0xf0/0x2b0 [35478.611393] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35478.620770] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35478.627900] [35478.635363] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35478.693431] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35478.700556] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35478.707681] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35478.714804] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35478.721927] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35478.729052] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35478.737130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35478.742869] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35478.749993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35478.757117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35478.764240] Call Trace: [35478.766684] [35478.768695] ? watchdog_timer_fn+0x1b4/0x220 [35478.772960] ? __hrtimer_run_queues+0x1d3/0x360 [35478.777487] ? hrtimer_interrupt+0x104/0x240 [35478.781756] ? __local_bh_enable+0x50/0x60 [35478.785850] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35478.791074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35478.796030] [35478.798126] [35478.800225] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35478.805532] ? lock_acquire+0xf0/0x2b0 [35478.809282] ? igrab+0x43/0x50 [35478.812332] ? lock_release+0xc5/0x260 [35478.816078] _raw_spin_lock+0x30/0x40 [35478.819737] ? igrab+0x1d/0x50 [35478.822784] igrab+0x1d/0x50 [35478.825662] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35478.830814] delayed_work+0x1d8/0x280 [ceph] [35478.835094] ? process_scheduled_works+0x22a/0x510 [35478.839885] process_scheduled_works+0x2a4/0x510 [35478.844498] worker_thread+0x188/0x340 [35478.848248] ? rescuer_thread+0x310/0x310 [35478.852251] kthread+0xf5/0x130 [35478.855389] ? kthread_complete_and_exit+0x20/0x20 [35478.860173] ret_from_fork+0x31/0x40 [35478.863743] ? kthread_complete_and_exit+0x20/0x20 [35478.868529] ret_from_fork_asm+0x11/0x20 [35478.872450] [35506.079768] watchdog: BUG: soft lockup - CPU#0 stuck for 21953s! [dbench:78683] [35506.087070] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35506.091768] watchdog: BUG: soft lockup - CPU#7 stuck for 30950s! [kworker/7:4:78808] [35506.142520] irq event stamp: 61812952 [35506.142521] hardirqs last enabled at (61812951): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35506.150249] Modules linked in: [35506.153907] hardirqs last disabled at (61812952): [] sysvec_apic_timer_interrupt+0xe/0x90 [35506.164065] ceph [35506.167115] softirqs last enabled at (61812896): [] __do_softirq+0x2be/0x371 [35506.176839] libceph [35506.178762] softirqs last disabled at (61812891): [] irq_exit_rcu+0xb0/0xd0 [35506.187447] dns_resolver fscache [35506.189633] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35506.198160] netfs veth [35506.201470] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35506.210414] nft_chain_nat [35506.212859] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35506.220328] xt_MASQUERADE [35506.223034] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35506.228510] nf_nat nf_conntrack [35506.231216] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35506.249953] nf_defrag_ipv6 [35506.253178] [35506.258393] nf_defrag_ipv4 nft_compat [35506.261186] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35506.262676] nf_tables nfnetlink [35506.266419] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35506.273561] bridge stp [35506.276785] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35506.283909] llc [35506.286353] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35506.293477] xfs libcrc32c [35506.295315] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35506.302454] sunrpc [35506.305164] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35506.312284] intel_rapl_msr intel_rapl_common [35506.314381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35506.322460] sb_edac x86_pkg_temp_thermal [35506.326811] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35506.332546] coretemp [35506.336553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35506.343675] kvm_intel [35506.345947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35506.353070] kvm irqbypass [35506.355428] Call Trace: [35506.355430] [35506.362551] i2c_i801 iTCO_wdt [35506.365258] ? watchdog_timer_fn+0x1b4/0x220 [35506.367700] iTCO_vendor_support i2c_smbus [35506.369711] ? __hrtimer_run_queues+0x1d3/0x360 [35506.372761] crct10dif_pclmul joydev [35506.377028] ? hrtimer_interrupt+0x104/0x240 [35506.381116] crc32_pclmul ipmi_ssif [35506.385639] ? __local_bh_enable+0x50/0x60 [35506.389212] ghash_clmulni_intel [35506.393478] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35506.396959] mei_me [35506.401050] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35506.404275] mei lpc_ich [35506.409492] [35506.411590] sha512_ssse3 acpi_ipmi [35506.416546] [35506.419077] mfd_core wmi [35506.421176] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35506.424657] ipmi_si ipmi_devintf [35506.426759] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35506.429374] ipmi_msghandler ioatdma [35506.434680] do_raw_spin_lock+0xb5/0xc0 [35506.437987] acpi_pad acpi_power_meter [35506.442861] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35506.446430] ixgbe igb [35506.450263] try_get_cap_refs+0x159/0x880 [ceph] [35506.454003] mdio [35506.458794] __ceph_get_caps+0x4a1/0x760 [ceph] [35506.461164] crc32c_intel [35506.465777] ? __file_remove_privs+0x89/0x110 [35506.467698] nvme ptp [35506.472221] ? wake_bit_function+0x70/0x70 [35506.474838] nvme_core i2c_algo_bit [35506.479193] ? ceph_write_iter+0x260/0x770 [ceph] [35506.481463] pps_core dca [35506.485551] ceph_write_iter+0x260/0x770 [ceph] [35506.489037] fuse [35506.493736] ? lock_is_held_type+0xaa/0x120 [35506.496351] [35506.496352] irq event stamp: 204827180 [35506.500878] vfs_write+0x324/0x4f0 [35506.502800] hardirqs last enabled at (204827179): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35506.506981] ksys_pwrite64+0x65/0xa0 [35506.508466] hardirqs last disabled at (204827180): [] sysvec_apic_timer_interrupt+0xe/0x90 [35506.512211] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35506.515607] softirqs last enabled at (204827178): [] __do_softirq+0x2be/0x371 [35506.525853] do_syscall_64+0x38/0x80 [35506.529424] softirqs last disabled at (204827173): [] irq_exit_rcu+0xb0/0xd0 [35506.539252] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35506.544381] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35506.553170] RIP: 0033:0x7f48f0eeb93b [35506.556741] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35506.565340] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35506.570384] Workqueue: events delayed_work [ceph] [35506.579759] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35506.583330] [35506.590802] ORIG_RAX: 0000000000000012 [35506.609540] RIP: 0010:lock_acquire+0xf0/0x2b0 [35506.614237] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35506.619459] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35506.620953] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35506.624785] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35506.629134] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35506.636259] [35506.654996] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35506.662120] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35506.667337] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35506.674462] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35506.675959] [35506.683077] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35506.713759] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35506.720882] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35506.728007] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35506.736083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35506.741820] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35506.748945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35506.756070] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35506.763193] Call Trace: [35506.765638] [35506.767649] ? watchdog_timer_fn+0x1b4/0x220 [35506.771913] ? __hrtimer_run_queues+0x1d3/0x360 [35506.776440] ? hrtimer_interrupt+0x104/0x240 [35506.780709] ? __local_bh_enable+0x50/0x60 [35506.784802] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35506.790026] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35506.794984] [35506.797080] [35506.799179] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35506.804486] ? lock_acquire+0xf0/0x2b0 [35506.808236] ? igrab+0x43/0x50 [35506.811285] ? lock_release+0xc5/0x260 [35506.815032] _raw_spin_lock+0x30/0x40 [35506.818688] ? igrab+0x1d/0x50 [35506.821739] igrab+0x1d/0x50 [35506.824616] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35506.829768] delayed_work+0x1d8/0x280 [ceph] [35506.834048] ? process_scheduled_works+0x22a/0x510 [35506.838837] process_scheduled_works+0x2a4/0x510 [35506.843457] worker_thread+0x188/0x340 [35506.847204] ? rescuer_thread+0x310/0x310 [35506.851214] kthread+0xf5/0x130 [35506.854360] ? kthread_complete_and_exit+0x20/0x20 [35506.859147] ret_from_fork+0x31/0x40 [35506.862743] ? kthread_complete_and_exit+0x20/0x20 [35506.867524] ret_from_fork_asm+0x11/0x20 [35506.871466] [35534.080210] watchdog: BUG: soft lockup - CPU#0 stuck for 21979s! [dbench:78683] [35534.087513] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35534.092210] watchdog: BUG: soft lockup - CPU#7 stuck for 30976s! [kworker/7:4:78808] [35534.142954] irq event stamp: 61871116 [35534.150686] Modules linked in: ceph libceph [35534.154342] hardirqs last enabled at (61871115): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35534.158520] dns_resolver fscache [35534.168677] hardirqs last disabled at (61871116): [] sysvec_apic_timer_interrupt+0xe/0x90 [35534.171987] netfs veth nft_chain_nat [35534.181712] softirqs last enabled at (61871020): [] __do_softirq+0x2be/0x371 [35534.185370] xt_MASQUERADE nf_nat nf_conntrack [35534.194053] softirqs last disabled at (61871015): [] irq_exit_rcu+0xb0/0xd0 [35534.194054] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [35534.207003] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35534.212913] nfnetlink bridge [35534.221856] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35534.224822] stp llc xfs libcrc32c [35534.232292] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35534.235690] sunrpc intel_rapl_msr [35534.241167] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35534.241168] intel_rapl_common [35534.244563] sb_edac x86_pkg_temp_thermal [35534.263303] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35534.266351] coretemp kvm_intel [35534.270356] [35534.270357] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35534.275573] kvm irqbypass [35534.278710] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35534.278711] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35534.280201] i2c_i801 [35534.287325] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35534.290029] iTCO_wdt iTCO_vendor_support [35534.297155] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35534.304278] i2c_smbus [35534.306548] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35534.313673] crct10dif_pclmul [35534.317676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35534.324801] joydev [35534.327158] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35534.335236] crc32_pclmul ipmi_ssif [35534.338199] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35534.343937] ghash_clmulni_intel [35534.346035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35534.353159] mei_me [35534.356642] Call Trace: [35534.356643] [35534.363768] mei [35534.366992] ? watchdog_timer_fn+0x1b4/0x220 [35534.374116] lpc_ich sha512_ssse3 [35534.376213] ? __hrtimer_run_queues+0x1d3/0x360 [35534.378658] acpi_ipmi mfd_core [35534.380672] ? hrtimer_interrupt+0x104/0x240 [35534.382505] wmi ipmi_si [35534.386770] ? __local_bh_enable+0x50/0x60 [35534.390081] ipmi_devintf ipmi_msghandler [35534.394611] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35534.397749] ioatdma acpi_pad [35534.402015] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35534.404544] acpi_power_meter [35534.408635] [35534.412641] ixgbe igb [35534.417857] [35534.420822] mdio crc32c_intel [35534.425780] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35534.428743] nvme ptp [35534.430844] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35534.433199] nvme_core i2c_algo_bit [35534.435297] do_raw_spin_lock+0xb5/0xc0 [35534.438345] pps_core [35534.443650] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35534.445920] dca fuse [35534.450793] try_get_cap_refs+0x159/0x880 [ceph] [35534.454276] [35534.454276] irq event stamp: 205047908 [35534.458111] __ceph_get_caps+0x4a1/0x760 [ceph] [35534.460378] hardirqs last enabled at (205047907): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35534.465164] ? __file_remove_privs+0x89/0x110 [35534.467431] hardirqs last disabled at (205047908): [] sysvec_apic_timer_interrupt+0xe/0x90 [35534.472043] ? wake_bit_function+0x70/0x70 [35534.473533] softirqs last enabled at (205047906): [] __do_softirq+0x2be/0x371 [35534.477279] ? ceph_write_iter+0x260/0x770 [ceph] [35534.481801] softirqs last disabled at (205047901): [] irq_exit_rcu+0xb0/0xd0 [35534.492045] ceph_write_iter+0x260/0x770 [ceph] [35534.496395] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35534.506210] ? lock_is_held_type+0xaa/0x120 [35534.510298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35534.519072] vfs_write+0x324/0x4f0 [35534.523767] Workqueue: events delayed_work [ceph] [35534.532367] ksys_pwrite64+0x65/0xa0 [35534.536903] [35534.536904] RIP: 0010:lock_acquire+0xf0/0x2b0 [35534.546282] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35534.550461] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35534.557932] do_syscall_64+0x38/0x80 [35534.561328] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35534.566026] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35534.569609] [35534.569610] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35534.571104] RIP: 0033:0x7f48f0eeb93b [35534.575455] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35534.580587] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35534.599341] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35534.602912] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35534.608130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35534.613174] ORIG_RAX: 0000000000000012 [35534.614664] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35534.621787] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35534.625359] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35534.632481] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35534.651221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35534.658344] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35534.663562] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35534.670687] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35534.674517] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35534.681639] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35534.688764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35534.696849] [35534.703967] Call Trace: [35534.703969] [35534.759090] ? watchdog_timer_fn+0x1b4/0x220 [35534.763352] ? __hrtimer_run_queues+0x1d3/0x360 [35534.767896] ? hrtimer_interrupt+0x104/0x240 [35534.772158] ? __local_bh_enable+0x50/0x60 [35534.776251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35534.781476] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35534.786432] [35534.788530] [35534.790628] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35534.795943] ? lock_acquire+0xf0/0x2b0 [35534.799693] ? igrab+0x43/0x50 [35534.802743] ? lock_release+0xc5/0x260 [35534.806488] _raw_spin_lock+0x30/0x40 [35534.810144] ? igrab+0x1d/0x50 [35534.813194] igrab+0x1d/0x50 [35534.816073] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35534.821222] delayed_work+0x1d8/0x280 [ceph] [35534.825506] ? process_scheduled_works+0x22a/0x510 [35534.830294] process_scheduled_works+0x2a4/0x510 [35534.834911] worker_thread+0x188/0x340 [35534.838659] ? rescuer_thread+0x310/0x310 [35534.842661] kthread+0xf5/0x130 [35534.845800] ? kthread_complete_and_exit+0x20/0x20 [35534.850584] ret_from_fork+0x31/0x40 [35534.854180] ? kthread_complete_and_exit+0x20/0x20 [35534.858966] ret_from_fork_asm+0x11/0x20 [35534.862904] [35562.080653] watchdog: BUG: soft lockup - CPU#0 stuck for 22005s! [dbench:78683] [35562.087958] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35562.092653] watchdog: BUG: soft lockup - CPU#7 stuck for 31002s! [kworker/7:4:78808] [35562.143408] irq event stamp: 61929214 [35562.143410] hardirqs last enabled at (61929213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35562.151138] Modules linked in: [35562.154794] hardirqs last disabled at (61929214): [] sysvec_apic_timer_interrupt+0xe/0x90 [35562.164953] ceph [35562.168003] softirqs last enabled at (61929158): [] __do_softirq+0x2be/0x371 [35562.177727] libceph [35562.179652] softirqs last disabled at (61929153): [] irq_exit_rcu+0xb0/0xd0 [35562.188335] dns_resolver [35562.190522] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35562.199048] fscache netfs [35562.201665] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35562.210608] veth [35562.213328] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35562.220803] nft_chain_nat [35562.222727] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35562.228202] xt_MASQUERADE nf_nat [35562.230908] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35562.249644] nf_conntrack nf_defrag_ipv6 [35562.252957] [35562.258172] nf_defrag_ipv4 [35562.262089] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35562.263582] nft_compat nf_tables [35562.266371] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35562.273496] nfnetlink bridge [35562.276806] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35562.283931] stp [35562.286895] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35562.294019] llc xfs [35562.295856] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35562.302981] libcrc32c sunrpc [35562.305164] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35562.312289] intel_rapl_msr [35562.315254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35562.323331] intel_rapl_common [35562.326122] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35562.331858] sb_edac x86_pkg_temp_thermal [35562.334910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35562.342050] coretemp [35562.346054] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35562.353178] kvm_intel [35562.355450] Call Trace: [35562.355452] [35562.362573] kvm irqbypass [35562.364934] ? watchdog_timer_fn+0x1b4/0x220 [35562.367375] i2c_i801 iTCO_wdt [35562.369387] ? __hrtimer_run_queues+0x1d3/0x360 [35562.372089] iTCO_vendor_support i2c_smbus [35562.376357] ? hrtimer_interrupt+0x104/0x240 [35562.379404] crct10dif_pclmul [35562.383930] ? __local_bh_enable+0x50/0x60 [35562.388019] joydev crc32_pclmul [35562.392287] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35562.395246] ipmi_ssif ghash_clmulni_intel [35562.399339] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35562.402562] mei_me [35562.407780] [35562.411871] mei lpc_ich [35562.416829] [35562.418925] sha512_ssse3 acpi_ipmi [35562.421024] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35562.423553] mfd_core wmi [35562.425655] ? queued_spin_lock_slowpath+0x16/0x2e0 [35562.429134] ipmi_si ipmi_devintf [35562.434442] do_raw_spin_lock+0xb5/0xc0 [35562.437056] ipmi_msghandler [35562.441929] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35562.445237] ioatdma acpi_pad [35562.449073] try_get_cap_refs+0x159/0x880 [ceph] [35562.451945] acpi_power_meter [35562.456736] __ceph_get_caps+0x4a1/0x760 [ceph] [35562.459695] ixgbe [35562.464309] ? __file_remove_privs+0x89/0x110 [35562.467268] igb mdio [35562.471794] ? wake_bit_function+0x70/0x70 [35562.473804] crc32c_intel nvme [35562.478158] ? ceph_write_iter+0x260/0x770 [ceph] [35562.480426] ptp nvme_core [35562.484516] ceph_write_iter+0x260/0x770 [ceph] [35562.487567] i2c_algo_bit pps_core [35562.492268] ? lock_is_held_type+0xaa/0x120 [35562.494967] dca fuse [35562.499494] vfs_write+0x324/0x4f0 [35562.502891] [35562.507071] ksys_pwrite64+0x65/0xa0 [35562.509338] irq event stamp: 205268564 [35562.512736] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35562.514225] hardirqs last enabled at (205268563): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35562.517798] do_syscall_64+0x38/0x80 [35562.521542] hardirqs last disabled at (205268564): [] sysvec_apic_timer_interrupt+0xe/0x90 [35562.526673] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35562.536916] softirqs last enabled at (205268562): [] __do_softirq+0x2be/0x371 [35562.540487] RIP: 0033:0x7f48f0eeb93b [35562.550298] softirqs last disabled at (205268557): [] irq_exit_rcu+0xb0/0xd0 [35562.555342] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35562.564112] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35562.567684] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35562.576281] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35562.595019] ORIG_RAX: 0000000000000012 [35562.604396] Workqueue: events delayed_work [ceph] [35562.609612] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35562.617083] [35562.620915] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35562.625613] RIP: 0010:lock_acquire+0xf0/0x2b0 [35562.632736] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35562.634226] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35562.641350] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35562.645702] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35562.652827] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35562.671563] [35562.678692] [35562.683906] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35562.701846] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35562.708971] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35562.716093] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35562.723219] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35562.730348] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35562.738427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35562.744167] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35562.751289] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35562.758415] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35562.765538] Call Trace: [35562.767985] [35562.769993] ? watchdog_timer_fn+0x1b4/0x220 [35562.774259] ? __hrtimer_run_queues+0x1d3/0x360 [35562.778783] ? hrtimer_interrupt+0x104/0x240 [35562.783060] ? __local_bh_enable+0x50/0x60 [35562.787157] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35562.792379] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35562.797352] [35562.799450] [35562.801550] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35562.806858] ? lock_acquire+0xf0/0x2b0 [35562.810606] ? igrab+0x43/0x50 [35562.813658] ? lock_release+0xc5/0x260 [35562.817403] _raw_spin_lock+0x30/0x40 [35562.821067] ? igrab+0x1d/0x50 [35562.824117] igrab+0x1d/0x50 [35562.826997] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35562.832147] delayed_work+0x1d8/0x280 [ceph] [35562.836426] ? process_scheduled_works+0x22a/0x510 [35562.841216] process_scheduled_works+0x2a4/0x510 [35562.845832] worker_thread+0x188/0x340 [35562.849583] ? rescuer_thread+0x310/0x310 [35562.853593] kthread+0xf5/0x130 [35562.856739] ? kthread_complete_and_exit+0x20/0x20 [35562.861525] ret_from_fork+0x31/0x40 [35562.865104] ? kthread_complete_and_exit+0x20/0x20 [35562.869888] ret_from_fork_asm+0x11/0x20 [35562.873819] [35590.081098] watchdog: BUG: soft lockup - CPU#0 stuck for 22031s! [dbench:78683] [35590.088402] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35590.093098] watchdog: BUG: soft lockup - CPU#7 stuck for 31028s! [kworker/7:4:78808] [35590.143843] irq event stamp: 61987356 [35590.151574] Modules linked in: ceph libceph [35590.155231] hardirqs last enabled at (61987355): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35590.159409] dns_resolver fscache [35590.169567] hardirqs last disabled at (61987356): [] sysvec_apic_timer_interrupt+0xe/0x90 [35590.172878] netfs veth [35590.182601] softirqs last enabled at (61987260): [] __do_softirq+0x2be/0x371 [35590.182603] nft_chain_nat [35590.185045] xt_MASQUERADE nf_nat [35590.193729] softirqs last disabled at (61987255): [] irq_exit_rcu+0xb0/0xd0 [35590.196433] nf_conntrack [35590.199743] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35590.208255] nf_defrag_ipv6 nf_defrag_ipv4 [35590.210871] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35590.219815] nft_compat nf_tables [35590.223908] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35590.231378] nfnetlink [35590.234688] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35590.240166] bridge stp [35590.242522] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35590.261261] llc xfs [35590.263706] [35590.263707] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35590.268922] libcrc32c sunrpc [35590.271107] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35590.272597] intel_rapl_msr intel_rapl_common [35590.279720] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35590.282684] sb_edac x86_pkg_temp_thermal [35590.289808] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35590.294159] coretemp kvm_intel [35590.301285] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35590.305289] kvm [35590.312411] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35590.315550] irqbypass i2c_i801 [35590.322674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35590.324512] iTCO_wdt [35590.332587] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35590.335727] iTCO_vendor_support [35590.341463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35590.343733] i2c_smbus crct10dif_pclmul [35590.350859] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35590.354082] joydev crc32_pclmul [35590.361206] Call Trace: [35590.361207] [35590.365037] ipmi_ssif ghash_clmulni_intel [35590.372161] ? watchdog_timer_fn+0x1b4/0x220 [35590.375387] mei_me mei [35590.377831] ? __hrtimer_run_queues+0x1d3/0x360 [35590.379840] lpc_ich sha512_ssse3 [35590.383934] ? hrtimer_interrupt+0x104/0x240 [35590.388195] acpi_ipmi [35590.390639] ? __local_bh_enable+0x50/0x60 [35590.395163] mfd_core wmi [35590.398494] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35590.402754] ipmi_si ipmi_devintf [35590.405113] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35590.409202] ipmi_msghandler ioatdma [35590.411822] [35590.417038] acpi_pad acpi_power_meter [35590.420349] [35590.425306] ixgbe [35590.428878] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35590.430975] igb mdio [35590.434724] ? queued_spin_lock_slowpath+0x16/0x2e0 [35590.436816] crc32c_intel nvme [35590.438828] do_raw_spin_lock+0xb5/0xc0 [35590.444130] ptp nvme_core [35590.446402] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35590.451272] i2c_algo_bit pps_core [35590.454326] try_get_cap_refs+0x159/0x880 [ceph] [35590.458154] dca fuse [35590.460864] __ceph_get_caps+0x4a1/0x760 [ceph] [35590.465641] [35590.465642] irq event stamp: 205489216 [35590.469044] ? __file_remove_privs+0x89/0x110 [35590.473649] hardirqs last enabled at (205489215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35590.475922] ? wake_bit_function+0x70/0x70 [35590.480444] hardirqs last disabled at (205489216): [] sysvec_apic_timer_interrupt+0xe/0x90 [35590.481938] ? ceph_write_iter+0x260/0x770 [ceph] [35590.485679] softirqs last enabled at (205489214): [] __do_softirq+0x2be/0x371 [35590.490032] ceph_write_iter+0x260/0x770 [ceph] [35590.500274] softirqs last disabled at (205489209): [] irq_exit_rcu+0xb0/0xd0 [35590.504368] ? lock_is_held_type+0xaa/0x120 [35590.514177] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35590.518876] vfs_write+0x324/0x4f0 [35590.527644] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35590.532173] ksys_pwrite64+0x65/0xa0 [35590.540780] Workqueue: events delayed_work [ceph] [35590.544961] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35590.554337] [35590.557736] do_syscall_64+0x38/0x80 [35590.565206] RIP: 0010:lock_release+0x14e/0x260 [35590.568790] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35590.573499] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [35590.578631] RIP: 0033:0x7f48f0eeb93b [35590.580123] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [35590.583694] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35590.588129] [35590.588130] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [35590.593174] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35590.611912] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [35590.615499] ORIG_RAX: 0000000000000012 [35590.620718] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [35590.639454] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35590.640947] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [35590.648070] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35590.653287] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [35590.660411] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35590.664241] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35590.671365] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35590.678500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35590.685623] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35590.692747] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35590.699876] [35590.706996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35590.751501] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35590.758625] Call Trace: [35590.761068] [35590.763079] ? watchdog_timer_fn+0x1b4/0x220 [35590.767343] ? __hrtimer_run_queues+0x1d3/0x360 [35590.771871] ? hrtimer_interrupt+0x104/0x240 [35590.776141] ? __local_bh_enable+0x50/0x60 [35590.780235] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35590.785457] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35590.790414] [35590.792512] [35590.794610] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35590.799922] ? igrab+0x43/0x50 [35590.802975] ? lock_release+0x14e/0x260 [35590.806812] ? lock_release+0xc5/0x260 [35590.810559] _raw_spin_unlock+0x1b/0x40 [35590.814395] igrab+0x43/0x50 [35590.817274] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35590.822423] delayed_work+0x1d8/0x280 [ceph] [35590.826707] ? process_scheduled_works+0x22a/0x510 [35590.831504] process_scheduled_works+0x2a4/0x510 [35590.836121] worker_thread+0x188/0x340 [35590.839870] ? rescuer_thread+0x310/0x310 [35590.843880] kthread+0xf5/0x130 [35590.847019] ? kthread_complete_and_exit+0x20/0x20 [35590.851802] ret_from_fork+0x31/0x40 [35590.855381] ? kthread_complete_and_exit+0x20/0x20 [35590.860166] ret_from_fork_asm+0x11/0x20 [35590.864097] [35618.081544] watchdog: BUG: soft lockup - CPU#0 stuck for 22057s! [dbench:78683] [35618.088846] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35618.093544] watchdog: BUG: soft lockup - CPU#7 stuck for 31054s! [kworker/7:4:78808] [35618.144296] irq event stamp: 62045448 [35618.144297] hardirqs last enabled at (62045447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35618.152026] Modules linked in: [35618.155682] hardirqs last disabled at (62045448): [] sysvec_apic_timer_interrupt+0xe/0x90 [35618.165841] ceph [35618.168891] softirqs last enabled at (62045402): [] __do_softirq+0x2be/0x371 [35618.178617] libceph [35618.180540] softirqs last disabled at (62045397): [] irq_exit_rcu+0xb0/0xd0 [35618.189224] dns_resolver [35618.191411] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35618.199937] fscache netfs [35618.202553] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35618.211498] veth nft_chain_nat [35618.214218] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35618.221690] xt_MASQUERADE nf_nat [35618.224828] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35618.230306] nf_conntrack [35618.233616] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35618.252354] nf_defrag_ipv6 nf_defrag_ipv4 [35618.254973] [35618.260189] nft_compat nf_tables [35618.264280] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35618.265770] nfnetlink bridge [35618.269079] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35618.276204] stp [35618.279168] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35618.286294] llc xfs [35618.288130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35618.295253] libcrc32c sunrpc [35618.297439] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35618.304562] intel_rapl_msr [35618.307527] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35618.314652] intel_rapl_common sb_edac [35618.317441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35618.325519] x86_pkg_temp_thermal [35618.329263] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35618.335001] coretemp [35618.338311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35618.345436] kvm_intel [35618.347705] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35618.354830] kvm irqbypass [35618.357189] Call Trace: [35618.364313] i2c_i801 iTCO_wdt [35618.367017] [35618.369460] iTCO_vendor_support [35618.372513] ? watchdog_timer_fn+0x1b4/0x220 [35618.374520] i2c_smbus crct10dif_pclmul [35618.377745] ? __hrtimer_run_queues+0x1d3/0x360 [35618.382008] joydev crc32_pclmul [35618.385843] ? hrtimer_interrupt+0x104/0x240 [35618.390363] ipmi_ssif ghash_clmulni_intel [35618.393588] ? __local_bh_enable+0x50/0x60 [35618.397852] mei_me mei [35618.401947] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35618.406033] lpc_ich [35618.408477] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35618.413695] sha512_ssse3 acpi_ipmi [35618.415878] [35618.420836] mfd_core wmi [35618.424321] [35618.424322] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35618.426418] ipmi_si ipmi_devintf [35618.429040] ? queued_spin_lock_slowpath+0x18/0x2e0 [35618.431133] ipmi_msghandler ioatdma [35618.436438] do_raw_spin_lock+0xb5/0xc0 [35618.439748] acpi_pad acpi_power_meter [35618.444620] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35618.448189] ixgbe igb [35618.452024] try_get_cap_refs+0x159/0x880 [ceph] [35618.455765] mdio crc32c_intel [35618.460553] __ceph_get_caps+0x4a1/0x760 [ceph] [35618.462905] nvme ptp [35618.467521] ? __file_remove_privs+0x89/0x110 [35618.470566] nvme_core i2c_algo_bit [35618.475091] ? wake_bit_function+0x70/0x70 [35618.477363] pps_core dca [35618.481715] ? ceph_write_iter+0x260/0x770 [ceph] [35618.485196] fuse [35618.489288] ceph_write_iter+0x260/0x770 [ceph] [35618.491905] irq event stamp: 205709800 [35618.496607] ? lock_is_held_type+0xaa/0x120 [35618.498526] hardirqs last enabled at (205709799): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35618.503052] vfs_write+0x324/0x4f0 [35618.506794] hardirqs last disabled at (205709800): [] sysvec_apic_timer_interrupt+0xe/0x90 [35618.510976] ksys_pwrite64+0x65/0xa0 [35618.521230] softirqs last enabled at (205709798): [] __do_softirq+0x2be/0x371 [35618.524631] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35618.534441] softirqs last disabled at (205709793): [] irq_exit_rcu+0xb0/0xd0 [35618.538013] do_syscall_64+0x38/0x80 [35618.546782] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35618.551916] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35618.560510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35618.564082] RIP: 0033:0x7f48f0eeb93b [35618.573461] Workqueue: events delayed_work [ceph] [35618.578506] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35618.585976] [35618.589544] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35618.594243] RIP: 0010:igrab+0x4/0x50 [35618.612979] ORIG_RAX: 0000000000000012 [35618.614471] Code: c7 7e 74 01 c3 0f 1f 44 00 00 c3 b8 00 04 00 00 f0 0f c1 05 fe 3a 4c 02 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa <0f> 1f 44 00 00 55 48 8d af 88 00 00 00 53 48 89 fb 48 89 ef e8 93 [35618.619688] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35618.623260] RSP: 0018:ffffc90002117db0 EFLAGS: 00000286 [35618.627090] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35618.645827] [35618.645828] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34478 RCX: ffff8881a1d34bc8 [35618.652952] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35618.658169] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34478 [35618.665294] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35618.666784] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [35618.673909] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35618.681032] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [35618.688161] [35618.695280] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35618.725960] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35618.734038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35618.739776] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35618.746899] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35618.754025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35618.761148] Call Trace: [35618.763594] [35618.765606] ? watchdog_timer_fn+0x1b4/0x220 [35618.769878] ? __hrtimer_run_queues+0x1d3/0x360 [35618.774413] ? hrtimer_interrupt+0x104/0x240 [35618.778682] ? __local_bh_enable+0x50/0x60 [35618.782776] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35618.788000] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35618.792955] [35618.795052] [35618.797151] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35618.802460] ? igrab+0x4/0x50 [35618.805428] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35618.810579] delayed_work+0x1d8/0x280 [ceph] [35618.814861] ? process_scheduled_works+0x22a/0x510 [35618.819649] process_scheduled_works+0x2a4/0x510 [35618.824265] worker_thread+0x188/0x340 [35618.828014] ? rescuer_thread+0x310/0x310 [35618.832018] kthread+0xf5/0x130 [35618.835157] ? kthread_complete_and_exit+0x20/0x20 [35618.839949] ret_from_fork+0x31/0x40 [35618.843529] ? kthread_complete_and_exit+0x20/0x20 [35618.848312] ret_from_fork_asm+0x11/0x20 [35618.852243] [35646.081991] watchdog: BUG: soft lockup - CPU#0 stuck for 22084s! [dbench:78683] [35646.089290] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35646.093991] watchdog: BUG: soft lockup - CPU#7 stuck for 31081s! [kworker/7:4:78808] [35646.144731] irq event stamp: 62103538 [35646.144732] hardirqs last enabled at (62103537): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35646.152462] Modules linked in: [35646.156119] hardirqs last disabled at (62103538): [] sysvec_apic_timer_interrupt+0xe/0x90 [35646.166277] ceph [35646.169327] softirqs last enabled at (62103462): [] __do_softirq+0x2be/0x371 [35646.179051] libceph [35646.180975] softirqs last disabled at (62103457): [] irq_exit_rcu+0xb0/0xd0 [35646.189665] dns_resolver [35646.191853] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35646.200363] fscache netfs [35646.202980] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35646.211925] veth nft_chain_nat [35646.214628] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35646.222101] xt_MASQUERADE nf_nat [35646.225236] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35646.230714] nf_conntrack nf_defrag_ipv6 [35646.234025] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35646.252763] nf_defrag_ipv4 nft_compat [35646.256681] [35646.256682] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35646.261897] nf_tables nfnetlink [35646.265642] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35646.267133] bridge [35646.274258] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35646.277480] stp llc [35646.284606] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35646.286702] xfs libcrc32c [35646.293828] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35646.296010] sunrpc intel_rapl_msr [35646.303136] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35646.305840] intel_rapl_common sb_edac [35646.312962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35646.316361] x86_pkg_temp_thermal coretemp [35646.324456] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35646.328199] kvm_intel kvm [35646.333936] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35646.338027] irqbypass i2c_i801 [35646.345153] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35646.347854] iTCO_wdt [35646.354981] Call Trace: [35646.358118] iTCO_vendor_support i2c_smbus [35646.365241] [35646.367512] crct10dif_pclmul [35646.369957] ? watchdog_timer_fn+0x1b4/0x220 [35646.374046] joydev crc32_pclmul [35646.376059] ? __hrtimer_run_queues+0x1d3/0x360 [35646.379021] ipmi_ssif [35646.383289] ? hrtimer_interrupt+0x104/0x240 [35646.386511] ghash_clmulni_intel mei_me [35646.391033] ? __local_bh_enable+0x50/0x60 [35646.393392] mei [35646.397673] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35646.401503] lpc_ich sha512_ssse3 [35646.405594] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35646.407431] acpi_ipmi [35646.412648] [35646.415976] mfd_core wmi [35646.420934] [35646.423292] ipmi_si ipmi_devintf [35646.425391] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35646.428007] ipmi_msghandler ioatdma [35646.430108] ? queued_spin_lock_slowpath+0x16/0x2e0 [35646.433414] acpi_pad acpi_power_meter [35646.438721] do_raw_spin_lock+0xb5/0xc0 [35646.442289] ixgbe igb [35646.447161] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35646.450905] mdio crc32c_intel [35646.454736] try_get_cap_refs+0x159/0x880 [ceph] [35646.457093] nvme ptp [35646.461881] __ceph_get_caps+0x4a1/0x760 [ceph] [35646.464928] nvme_core i2c_algo_bit [35646.469544] ? __file_remove_privs+0x89/0x110 [35646.471808] pps_core dca [35646.476333] ? wake_bit_function+0x70/0x70 [35646.479818] fuse [35646.484171] ? ceph_write_iter+0x260/0x770 [ceph] [35646.486784] irq event stamp: 205930660 [35646.490877] ceph_write_iter+0x260/0x770 [ceph] [35646.492799] hardirqs last enabled at (205930659): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35646.497500] ? lock_is_held_type+0xaa/0x120 [35646.501240] hardirqs last disabled at (205930660): [] sysvec_apic_timer_interrupt+0xe/0x90 [35646.505767] vfs_write+0x324/0x4f0 [35646.516009] softirqs last enabled at (205930658): [] __do_softirq+0x2be/0x371 [35646.520192] ksys_pwrite64+0x65/0xa0 [35646.529998] softirqs last disabled at (205930653): [] irq_exit_rcu+0xb0/0xd0 [35646.533396] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35646.542165] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35646.545737] do_syscall_64+0x38/0x80 [35646.554335] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35646.559465] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35646.568843] Workqueue: events delayed_work [ceph] [35646.572413] RIP: 0033:0x7f48f0eeb93b [35646.579884] [35646.579885] RIP: 0010:_raw_spin_unlock+0x1e/0x40 [35646.584929] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35646.589624] Code: 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b [35646.593196] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35646.594687] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [35646.599297] ORIG_RAX: 0000000000000012 [35646.618034] [35646.618035] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [35646.636772] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35646.641989] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [35646.647208] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35646.651038] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [35646.652529] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35646.659653] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [35646.666778] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35646.673901] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [35646.681025] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35646.688151] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35646.695280] [35646.702398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35646.739771] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35646.746895] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35646.754019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35646.761143] Call Trace: [35646.763588] [35646.765597] ? watchdog_timer_fn+0x1b4/0x220 [35646.769861] ? __hrtimer_run_queues+0x1d3/0x360 [35646.774398] ? hrtimer_interrupt+0x104/0x240 [35646.778667] ? __local_bh_enable+0x50/0x60 [35646.782777] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35646.788002] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35646.792959] [35646.795055] [35646.797152] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35646.802462] ? _raw_spin_unlock+0x1e/0x40 [35646.806472] igrab+0x43/0x50 [35646.809348] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35646.814499] delayed_work+0x1d8/0x280 [ceph] [35646.818780] ? process_scheduled_works+0x22a/0x510 [35646.823570] process_scheduled_works+0x2a4/0x510 [35646.828184] worker_thread+0x188/0x340 [35646.831933] ? rescuer_thread+0x310/0x310 [35646.835938] kthread+0xf5/0x130 [35646.839076] ? kthread_complete_and_exit+0x20/0x20 [35646.843859] ret_from_fork+0x31/0x40 [35646.847439] ? kthread_complete_and_exit+0x20/0x20 [35646.852223] ret_from_fork_asm+0x11/0x20 [35646.856155] [35647.338377] rcu: INFO: rcu_preempt self-detected stall on CPU [35647.344136] rcu: 0-....: (23174644 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11603199 [35647.354647] rcu: (t=23710310 jiffies g=1309609 q=660476 ncpus=8) [35647.360732] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35647.369685] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35647.377155] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35647.382641] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35647.401379] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35647.406621] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35647.413746] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35647.420870] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35647.427993] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35647.435119] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35647.442242] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35647.450319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35647.456056] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35647.463180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35647.470307] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35647.477430] Call Trace: [35647.479874] [35647.481885] ? rcu_dump_cpu_stacks+0xd1/0x180 [35647.486245] ? rcu_sched_clock_irq+0x633/0x1390 [35647.490769] ? lock_is_held_type+0xaa/0x120 [35647.494954] ? lock_is_held_type+0xaa/0x120 [35647.499131] ? tick_sched_do_timer+0xa0/0xa0 [35647.503409] ? rcu_read_lock_sched_held+0x56/0x90 [35647.508112] ? tick_sched_do_timer+0xa0/0xa0 [35647.512374] ? update_process_times+0x5f/0x90 [35647.516726] ? tick_sched_handle.isra.16+0x1e/0x50 [35647.521509] ? tick_sched_timer+0x6f/0x80 [35647.525515] ? __hrtimer_run_queues+0x1d3/0x360 [35647.530039] ? hrtimer_interrupt+0x104/0x240 [35647.534300] ? __local_bh_enable+0x50/0x60 [35647.538410] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35647.543627] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35647.548584] [35647.550681] [35647.552796] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35647.558106] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35647.562981] do_raw_spin_lock+0xb5/0xc0 [35647.566812] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35647.571614] try_get_cap_refs+0x159/0x880 [ceph] [35647.576245] __ceph_get_caps+0x4a1/0x760 [ceph] [35647.580785] ? __file_remove_privs+0x89/0x110 [35647.585141] ? wake_bit_function+0x70/0x70 [35647.589235] ? ceph_write_iter+0x260/0x770 [ceph] [35647.593948] ceph_write_iter+0x260/0x770 [ceph] [35647.598483] ? lock_is_held_type+0xaa/0x120 [35647.602667] vfs_write+0x324/0x4f0 [35647.606067] ksys_pwrite64+0x65/0xa0 [35647.609641] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35647.614774] do_syscall_64+0x38/0x80 [35647.618345] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35647.623405] RIP: 0033:0x7f48f0eeb93b [35647.626976] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35647.645712] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [35647.653271] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35647.660409] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35647.667537] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35647.674660] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35647.681786] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35647.688914] [35647.691101] Sending NMI from CPU 0 to CPUs 7: [35647.695454] NMI backtrace for cpu 7 [35647.695456] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35647.695458] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35647.695459] Workqueue: events delayed_work [ceph] [35647.695480] RIP: 0010:lock_acquire+0xc4/0x2b0 [35647.695483] Code: 6a 00 81 e3 00 02 00 00 41 89 e8 44 89 e9 6a 00 41 0f 94 c1 44 89 f2 44 89 fe 6a 00 45 0f b6 c9 4c 89 e7 ff b4 24 88 00 00 00 <41> 52 e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 [35647.695485] RSP: 0018:ffffc90002117cf8 EFLAGS: 00000006 [35647.695487] RAX: 0000000000000246 RBX: 0000000000000200 RCX: 0000000000000000 [35647.695488] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [35647.695489] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [35647.695490] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [35647.695492] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35647.695493] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35647.695494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35647.695496] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35647.695497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35647.695498] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35647.695500] Call Trace: [35647.695501] [35647.695502] ? nmi_cpu_backtrace+0xad/0x120 [35647.695507] ? nmi_cpu_backtrace_handler+0x11/0x20 [35647.695509] ? nmi_handle+0xd4/0x260 [35647.695514] ? default_do_nmi+0x49/0xf0 [35647.695517] ? exc_nmi+0x11e/0x150 [35647.695519] ? end_repeat_nmi+0x16/0x67 [35647.695525] ? lock_acquire+0xc4/0x2b0 [35647.695528] ? lock_acquire+0xc4/0x2b0 [35647.695531] ? lock_acquire+0xc4/0x2b0 [35647.695533] [35647.695534] [35647.695534] ? igrab+0x1d/0x50 [35647.695537] ? igrab+0x43/0x50 [35647.695539] ? lock_release+0xc5/0x260 [35647.695542] _raw_spin_lock+0x30/0x40 [35647.695545] ? igrab+0x1d/0x50 [35647.695546] igrab+0x1d/0x50 [35647.695548] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35647.695569] delayed_work+0x1d8/0x280 [ceph] [35647.695589] ? process_scheduled_works+0x22a/0x510 [35647.695592] process_scheduled_works+0x2a4/0x510 [35647.695598] worker_thread+0x188/0x340 [35647.695601] ? rescuer_thread+0x310/0x310 [35647.695604] kthread+0xf5/0x130 [35647.695607] ? kthread_complete_and_exit+0x20/0x20 [35647.695610] ret_from_fork+0x31/0x40 [35647.695613] ? kthread_complete_and_exit+0x20/0x20 [35647.695616] ret_from_fork_asm+0x11/0x20 [35647.695623] [35674.082440] watchdog: BUG: soft lockup - CPU#0 stuck for 22110s! [dbench:78683] [35674.089744] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35674.094439] watchdog: BUG: soft lockup - CPU#7 stuck for 31107s! [kworker/7:4:78808] [35674.145194] irq event stamp: 62160854 [35674.145195] hardirqs last enabled at (62160853): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35674.152924] Modules linked in: [35674.156580] hardirqs last disabled at (62160854): [] sysvec_apic_timer_interrupt+0xe/0x90 [35674.166739] ceph [35674.169790] softirqs last enabled at (62160762): [] __do_softirq+0x2be/0x371 [35674.179514] libceph [35674.181437] softirqs last disabled at (62160757): [] irq_exit_rcu+0xb0/0xd0 [35674.190121] dns_resolver fscache [35674.192307] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35674.200832] netfs [35674.204145] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35674.213087] veth nft_chain_nat [35674.215116] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35674.222589] xt_MASQUERADE [35674.225727] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35674.231203] nf_nat nf_conntrack [35674.233907] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35674.252643] nf_defrag_ipv6 nf_defrag_ipv4 [35674.255868] [35674.255869] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35674.261086] nft_compat nf_tables [35674.265177] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35674.266668] nfnetlink bridge [35674.273791] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35674.277115] stp llc [35674.284244] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35674.287209] xfs [35674.294332] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35674.296515] libcrc32c sunrpc [35674.303639] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35674.305478] intel_rapl_msr [35674.312601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35674.315566] intel_rapl_common sb_edac [35674.323643] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35674.326434] x86_pkg_temp_thermal [35674.332171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35674.335916] coretemp kvm_intel [35674.343040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35674.346350] kvm [35674.353473] Call Trace: [35674.353475] [35674.356612] irqbypass [35674.363736] ? watchdog_timer_fn+0x1b4/0x220 [35674.365573] i2c_i801 iTCO_wdt [35674.368017] ? __hrtimer_run_queues+0x1d3/0x360 [35674.370028] iTCO_vendor_support [35674.372388] ? hrtimer_interrupt+0x104/0x240 [35674.376648] i2c_smbus crct10dif_pclmul [35674.379701] ? __local_bh_enable+0x50/0x60 [35674.384223] joydev crc32_pclmul [35674.387451] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35674.391711] ipmi_ssif ghash_clmulni_intel [35674.395543] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35674.399634] mei_me [35674.402857] [35674.408075] mei [35674.412166] [35674.412167] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35674.417122] lpc_ich sha512_ssse3 [35674.419224] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35674.421316] acpi_ipmi [35674.423157] do_raw_spin_lock+0xb5/0xc0 [35674.425252] mfd_core wmi [35674.430558] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35674.433869] ipmi_si [35674.438741] try_get_cap_refs+0x159/0x880 [ceph] [35674.441097] ipmi_devintf ipmi_msghandler ioatdma [35674.444950] __ceph_get_caps+0x4a1/0x760 [ceph] [35674.447560] acpi_pad acpi_power_meter [35674.452351] ? __file_remove_privs+0x89/0x110 [35674.454530] ixgbe igb [35674.459140] ? wake_bit_function+0x70/0x70 [35674.463839] mdio [35674.468364] ? ceph_write_iter+0x260/0x770 [ceph] [35674.472106] crc32c_intel [35674.476474] ceph_write_iter+0x260/0x770 [ceph] [35674.478838] nvme [35674.482935] ? lock_is_held_type+0xaa/0x120 [35674.484855] ptp [35674.489554] vfs_write+0x324/0x4f0 [35674.492169] nvme_core i2c_algo_bit [35674.496698] ksys_pwrite64+0x65/0xa0 [35674.498616] pps_core dca [35674.502795] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35674.504632] fuse [35674.508031] do_syscall_64+0x38/0x80 [35674.511514] irq event stamp: 206149580 [35674.515086] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35674.517701] hardirqs last enabled at (206149579): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35674.522841] RIP: 0033:0x7f48f0eeb93b [35674.524765] hardirqs last disabled at (206149580): [] sysvec_apic_timer_interrupt+0xe/0x90 [35674.528337] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35674.532079] softirqs last enabled at (206149578): [] __do_softirq+0x2be/0x371 [35674.537124] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35674.547367] softirqs last disabled at (206149573): [] irq_exit_rcu+0xb0/0xd0 [35674.550938] ORIG_RAX: 0000000000000012 [35674.560750] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35674.579488] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35674.588259] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35674.593475] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35674.602073] Workqueue: events delayed_work [ceph] [35674.605903] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35674.615282] [35674.622406] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35674.629876] RIP: 0010:lock_acquire+0xf0/0x2b0 [35674.637000] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35674.641697] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35674.648844] [35674.650329] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35674.695070] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35674.702193] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35674.709317] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35674.716441] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35674.723566] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35674.730689] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35674.738767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35674.744503] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35674.751627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35674.758752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35674.765875] Call Trace: [35674.768322] [35674.770333] ? watchdog_timer_fn+0x1b4/0x220 [35674.774595] ? __hrtimer_run_queues+0x1d3/0x360 [35674.779138] ? hrtimer_interrupt+0x104/0x240 [35674.783400] ? __local_bh_enable+0x50/0x60 [35674.787495] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35674.792719] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35674.797674] [35674.799774] [35674.801870] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35674.807178] ? lock_acquire+0xf0/0x2b0 [35674.810928] ? igrab+0x43/0x50 [35674.813977] ? lock_release+0xc5/0x260 [35674.817723] _raw_spin_lock+0x30/0x40 [35674.821380] ? igrab+0x1d/0x50 [35674.824431] igrab+0x1d/0x50 [35674.827309] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35674.832459] delayed_work+0x1d8/0x280 [ceph] [35674.836739] ? process_scheduled_works+0x22a/0x510 [35674.841529] process_scheduled_works+0x2a4/0x510 [35674.846146] worker_thread+0x188/0x340 [35674.849894] ? rescuer_thread+0x310/0x310 [35674.853896] kthread+0xf5/0x130 [35674.857035] ? kthread_complete_and_exit+0x20/0x20 [35674.861820] ret_from_fork+0x31/0x40 [35674.865415] ? kthread_complete_and_exit+0x20/0x20 [35674.870199] ret_from_fork_asm+0x11/0x20 [35674.874138] [35702.082885] watchdog: BUG: soft lockup - CPU#0 stuck for 22136s! [dbench:78683] [35702.090188] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35702.094885] watchdog: BUG: soft lockup - CPU#7 stuck for 31133s! [kworker/7:4:78808] [35702.145637] irq event stamp: 62219092 [35702.145638] hardirqs last enabled at (62219091): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35702.153368] Modules linked in: [35702.157026] hardirqs last disabled at (62219092): [] sysvec_apic_timer_interrupt+0xe/0x90 [35702.167182] ceph [35702.170233] softirqs last enabled at (62219056): [] __do_softirq+0x2be/0x371 [35702.179959] libceph [35702.181882] softirqs last disabled at (62219051): [] irq_exit_rcu+0xb0/0xd0 [35702.190566] dns_resolver [35702.192751] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35702.201279] fscache netfs [35702.203897] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35702.212840] veth nft_chain_nat [35702.215559] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35702.223031] xt_MASQUERADE nf_nat [35702.226169] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35702.231648] nf_conntrack nf_defrag_ipv6 [35702.234958] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35702.253694] nf_defrag_ipv4 nft_compat [35702.257614] [35702.257614] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35702.262831] nf_tables nfnetlink [35702.266574] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35702.268065] bridge stp [35702.275189] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35702.278414] llc xfs [35702.285536] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35702.287980] libcrc32c sunrpc [35702.295104] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35702.297289] intel_rapl_msr intel_rapl_common [35702.304413] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35702.307377] sb_edac x86_pkg_temp_thermal [35702.314501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35702.318851] coretemp [35702.326929] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35702.330935] kvm_intel kvm [35702.336670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35702.338941] irqbypass i2c_i801 [35702.346066] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35702.348770] iTCO_wdt [35702.355894] Call Trace: [35702.355895] [35702.359032] iTCO_vendor_support i2c_smbus [35702.366157] ? watchdog_timer_fn+0x1b4/0x220 [35702.368425] crct10dif_pclmul joydev [35702.370872] ? __hrtimer_run_queues+0x1d3/0x360 [35702.372881] crc32_pclmul ipmi_ssif [35702.376976] ? hrtimer_interrupt+0x104/0x240 [35702.381236] ghash_clmulni_intel [35702.384806] ? __local_bh_enable+0x50/0x60 [35702.389330] mei_me mei [35702.392818] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35702.397079] lpc_ich sha512_ssse3 [35702.400303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35702.404392] acpi_ipmi [35702.406839] [35702.412056] mfd_core wmi [35702.415365] [35702.415366] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35702.420323] ipmi_si ipmi_devintf [35702.422686] ? queued_spin_lock_slowpath+0x18/0x2e0 [35702.424779] ipmi_msghandler [35702.427399] do_raw_spin_lock+0xb5/0xc0 [35702.429494] ioatdma [35702.434798] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35702.438108] acpi_pad acpi_power_meter [35702.442980] try_get_cap_refs+0x159/0x880 [ceph] [35702.445857] ixgbe igb [35702.449693] __ceph_get_caps+0x4a1/0x760 [ceph] [35702.451871] mdio crc32c_intel [35702.456659] ? __file_remove_privs+0x89/0x110 [35702.460399] nvme ptp [35702.465011] ? wake_bit_function+0x70/0x70 [35702.467366] nvme_core i2c_algo_bit [35702.471894] ? ceph_write_iter+0x260/0x770 [ceph] [35702.474941] pps_core dca [35702.479293] ceph_write_iter+0x260/0x770 [ceph] [35702.481565] fuse [35702.485658] ? lock_is_held_type+0xaa/0x120 [35702.489138] irq event stamp: 206370208 [35702.493838] vfs_write+0x324/0x4f0 [35702.496453] hardirqs last enabled at (206370207): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35702.500981] ksys_pwrite64+0x65/0xa0 [35702.502901] hardirqs last disabled at (206370208): [] sysvec_apic_timer_interrupt+0xe/0x90 [35702.507078] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35702.510822] softirqs last enabled at (206370206): [] __do_softirq+0x2be/0x371 [35702.514222] do_syscall_64+0x38/0x80 [35702.524463] softirqs last disabled at (206370201): [] irq_exit_rcu+0xb0/0xd0 [35702.528035] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35702.537845] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35702.542978] RIP: 0033:0x7f48f0eeb93b [35702.551747] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35702.555319] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35702.563917] Workqueue: events delayed_work [ceph] [35702.568959] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35702.578336] [35702.581908] ORIG_RAX: 0000000000000012 [35702.589378] RIP: 0010:lock_release+0x14e/0x260 [35702.608117] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35702.612814] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [35702.618030] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35702.619521] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [35702.623352] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35702.627790] [35702.634913] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35702.653653] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [35702.660777] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35702.665994] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [35702.673122] [35702.674608] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [35702.712414] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [35702.719537] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [35702.726660] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35702.734740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35702.740476] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35702.747599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35702.754726] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35702.761849] Call Trace: [35702.764294] [35702.766304] ? watchdog_timer_fn+0x1b4/0x220 [35702.770579] ? __hrtimer_run_queues+0x1d3/0x360 [35702.775103] ? hrtimer_interrupt+0x104/0x240 [35702.779365] ? __local_bh_enable+0x50/0x60 [35702.783457] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35702.788682] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35702.793638] [35702.795736] [35702.797833] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35702.803138] ? igrab+0x43/0x50 [35702.806190] ? lock_release+0x14e/0x260 [35702.810021] ? lock_release+0xc5/0x260 [35702.813765] _raw_spin_unlock+0x1b/0x40 [35702.817603] igrab+0x43/0x50 [35702.820480] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35702.825630] delayed_work+0x1d8/0x280 [ceph] [35702.829913] ? process_scheduled_works+0x22a/0x510 [35702.834703] process_scheduled_works+0x2a4/0x510 [35702.839318] worker_thread+0x188/0x340 [35702.843067] ? rescuer_thread+0x310/0x310 [35702.847071] kthread+0xf5/0x130 [35702.850208] ? kthread_complete_and_exit+0x20/0x20 [35702.854991] ret_from_fork+0x31/0x40 [35702.858576] ? kthread_complete_and_exit+0x20/0x20 [35702.863363] ret_from_fork_asm+0x11/0x20 [35702.867294] [35730.083316] watchdog: BUG: soft lockup - CPU#0 stuck for 22162s! [dbench:78683] [35730.090615] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35730.095316] watchdog: BUG: soft lockup - CPU#7 stuck for 31159s! [kworker/7:4:78808] [35730.146056] irq event stamp: 62277274 [35730.146058] hardirqs last enabled at (62277273): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35730.153786] Modules linked in: [35730.157444] hardirqs last disabled at (62277274): [] sysvec_apic_timer_interrupt+0xe/0x90 [35730.167602] ceph [35730.170651] softirqs last enabled at (62277158): [] __do_softirq+0x2be/0x371 [35730.180377] libceph [35730.182300] softirqs last disabled at (62277153): [] irq_exit_rcu+0xb0/0xd0 [35730.190992] dns_resolver fscache [35730.193177] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35730.201687] netfs [35730.205015] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35730.213960] veth nft_chain_nat [35730.215970] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35730.223459] xt_MASQUERADE nf_nat [35730.226596] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35730.232074] nf_conntrack [35730.235384] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35730.254122] nf_defrag_ipv6 [35730.256738] [35730.256739] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35730.261957] nf_defrag_ipv4 nft_compat [35730.264747] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35730.266238] nf_tables nfnetlink [35730.273363] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35730.277106] bridge stp [35730.284230] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35730.287455] llc xfs [35730.294578] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35730.297021] libcrc32c sunrpc [35730.304146] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35730.306332] intel_rapl_msr [35730.313455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35730.316420] intel_rapl_common sb_edac [35730.324497] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35730.327287] x86_pkg_temp_thermal [35730.333025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35730.336768] coretemp [35730.343893] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35730.347204] kvm_intel kvm [35730.354328] Call Trace: [35730.354329] [35730.356598] irqbypass i2c_i801 [35730.363723] ? watchdog_timer_fn+0x1b4/0x220 [35730.366427] iTCO_wdt [35730.368871] ? __hrtimer_run_queues+0x1d3/0x360 [35730.370882] iTCO_vendor_support [35730.374022] ? hrtimer_interrupt+0x104/0x240 [35730.378282] i2c_smbus crct10dif_pclmul [35730.380555] ? __local_bh_enable+0x50/0x60 [35730.385077] joydev [35730.388303] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35730.392565] crc32_pclmul ipmi_ssif [35730.396397] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35730.400488] ghash_clmulni_intel [35730.402584] [35730.407801] mei_me mei [35730.411286] [35730.416245] lpc_ich sha512_ssse3 [35730.419469] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35730.421565] acpi_ipmi [35730.424012] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35730.426107] mfd_core [35730.429420] do_raw_spin_lock+0xb5/0xc0 [35730.434720] wmi [35730.437079] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35730.441949] ipmi_si ipmi_devintf [35730.444223] try_get_cap_refs+0x159/0x880 [ceph] [35730.448051] ipmi_msghandler ioatdma [35730.449892] __ceph_get_caps+0x4a1/0x760 [ceph] [35730.454672] acpi_pad acpi_power_meter [35730.458001] ? __file_remove_privs+0x89/0x110 [35730.462612] ixgbe igb [35730.466182] ? wake_bit_function+0x70/0x70 [35730.470713] mdio crc32c_intel [35730.474461] ? ceph_write_iter+0x260/0x770 [ceph] [35730.478810] nvme [35730.481167] ceph_write_iter+0x260/0x770 [ceph] [35730.485257] ptp [35730.488311] ? lock_is_held_type+0xaa/0x120 [35730.493005] nvme_core i2c_algo_bit [35730.494934] vfs_write+0x324/0x4f0 [35730.499455] pps_core dca [35730.501294] ksys_pwrite64+0x65/0xa0 [35730.505469] fuse [35730.508953] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35730.512351] irq event stamp: 206590958 [35730.514969] do_syscall_64+0x38/0x80 [35730.518537] hardirqs last enabled at (206590957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35730.520464] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35730.525592] hardirqs last disabled at (206590958): [] sysvec_apic_timer_interrupt+0xe/0x90 [35730.529336] RIP: 0033:0x7f48f0eeb93b [35730.532908] softirqs last enabled at (206590956): [] __do_softirq+0x2be/0x371 [35730.543151] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35730.548195] softirqs last disabled at (206590951): [] irq_exit_rcu+0xb0/0xd0 [35730.558008] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35730.561579] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35730.570349] ORIG_RAX: 0000000000000012 [35730.589087] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35730.597683] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35730.602902] Workqueue: events delayed_work [ceph] [35730.612279] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35730.616109] [35730.616110] RIP: 0010:lock_acquire+0xf0/0x2b0 [35730.623580] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35730.630716] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35730.635411] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35730.642535] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35730.644026] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35730.648377] [35730.648378] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35730.655505] [35730.674238] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35730.711644] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35730.718767] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35730.725893] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35730.733016] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35730.741095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35730.746830] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35730.753956] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35730.761080] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35730.768204] Call Trace: [35730.770650] [35730.772661] ? watchdog_timer_fn+0x1b4/0x220 [35730.776933] ? __hrtimer_run_queues+0x1d3/0x360 [35730.781458] ? hrtimer_interrupt+0x104/0x240 [35730.785719] ? __local_bh_enable+0x50/0x60 [35730.789812] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35730.795036] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35730.799995] [35730.802107] [35730.804206] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35730.809515] ? lock_acquire+0xf0/0x2b0 [35730.813264] ? igrab+0x43/0x50 [35730.816315] ? lock_release+0xc5/0x260 [35730.820059] _raw_spin_lock+0x30/0x40 [35730.823725] ? igrab+0x1d/0x50 [35730.826775] igrab+0x1d/0x50 [35730.829652] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35730.834804] delayed_work+0x1d8/0x280 [ceph] [35730.839084] ? process_scheduled_works+0x22a/0x510 [35730.843875] process_scheduled_works+0x2a4/0x510 [35730.848488] worker_thread+0x188/0x340 [35730.852239] ? rescuer_thread+0x310/0x310 [35730.856250] kthread+0xf5/0x130 [35730.859388] ? kthread_complete_and_exit+0x20/0x20 [35730.864171] ret_from_fork+0x31/0x40 [35730.867742] ? kthread_complete_and_exit+0x20/0x20 [35730.872528] ret_from_fork_asm+0x11/0x20 [35730.876449] [35758.083748] watchdog: BUG: soft lockup - CPU#0 stuck for 22188s! [dbench:78683] [35758.091050] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35758.095748] watchdog: BUG: soft lockup - CPU#7 stuck for 31185s! [kworker/7:4:78808] [35758.146500] irq event stamp: 62335440 [35758.154229] Modules linked in: ceph libceph dns_resolver fscache netfs [35758.157889] hardirqs last enabled at (62335439): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35758.164422] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [35758.174579] hardirqs last disabled at (62335440): [] sysvec_apic_timer_interrupt+0xe/0x90 [35758.180664] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [35758.190389] softirqs last enabled at (62335328): [] __do_softirq+0x2be/0x371 [35758.197782] stp llc xfs libcrc32c sunrpc intel_rapl_msr [35758.206466] softirqs last disabled at (62335323): [] irq_exit_rcu+0xb0/0xd0 [35758.211770] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [35758.220283] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35758.220284] kvm_intel [35758.220285] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35758.226634] kvm irqbypass [35758.235578] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35758.237934] i2c_i801 [35758.245407] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35758.248109] iTCO_wdt iTCO_vendor_support [35758.253587] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35758.255859] i2c_smbus crct10dif_pclmul [35758.274596] [35758.274597] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35758.278600] joydev crc32_pclmul [35758.283816] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35758.287647] ipmi_ssif ghash_clmulni_intel [35758.289144] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35758.296270] mei_me [35758.299496] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35758.306618] mei lpc_ich [35758.310711] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35758.317833] sha512_ssse3 [35758.319930] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35758.327056] acpi_ipmi mfd_core [35758.329585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35758.336711] wmi [35758.339327] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35758.347405] ipmi_si ipmi_devintf [35758.350560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35758.356297] ipmi_msghandler [35758.358146] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35758.365267] ioatdma acpi_pad [35758.368579] Call Trace: [35758.375701] acpi_power_meter [35758.378583] [35758.385705] ixgbe igb [35758.388669] ? watchdog_timer_fn+0x1b4/0x220 [35758.391112] mdio crc32c_intel [35758.394078] ? __hrtimer_run_queues+0x1d3/0x360 [35758.396086] nvme ptp [35758.398448] ? hrtimer_interrupt+0x104/0x240 [35758.402708] nvme_core i2c_algo_bit [35758.405760] ? __local_bh_enable+0x50/0x60 [35758.410284] pps_core dca [35758.412555] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35758.416819] fuse [35758.420304] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35758.424393] irq event stamp: 206811544 [35758.427010] [35758.432228] hardirqs last enabled at (206811543): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35758.434150] [35758.439108] hardirqs last disabled at (206811544): [] sysvec_apic_timer_interrupt+0xe/0x90 [35758.442854] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35758.444951] softirqs last enabled at (206811542): [] __do_softirq+0x2be/0x371 [35758.455198] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35758.457290] softirqs last disabled at (206811537): [] irq_exit_rcu+0xb0/0xd0 [35758.467105] do_raw_spin_lock+0xb5/0xc0 [35758.472422] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35758.481197] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35758.486067] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35758.494666] try_get_cap_refs+0x159/0x880 [ceph] [35758.498495] Workqueue: events delayed_work [ceph] [35758.507878] __ceph_get_caps+0x4a1/0x760 [ceph] [35758.512655] [35758.512656] RIP: 0010:lock_acquire+0xf0/0x2b0 [35758.520148] ? __file_remove_privs+0x89/0x110 [35758.524753] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35758.529452] ? wake_bit_function+0x70/0x70 [35758.533976] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35758.535470] ? ceph_write_iter+0x260/0x770 [ceph] [35758.539817] [35758.539817] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35758.544168] ceph_write_iter+0x260/0x770 [ceph] [35758.562906] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35758.567001] ? lock_is_held_type+0xaa/0x120 [35758.572213] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35758.576915] vfs_write+0x324/0x4f0 [35758.578403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35758.585530] ksys_pwrite64+0x65/0xa0 [35758.590051] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35758.597174] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35758.601351] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35758.608478] do_syscall_64+0x38/0x80 [35758.611874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35758.618999] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35758.622569] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35758.629694] RIP: 0033:0x7f48f0eeb93b [35758.634824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35758.642901] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35758.646471] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35758.652209] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35758.657252] Call Trace: [35758.657254] [35758.664377] ORIG_RAX: 0000000000000012 [35758.667948] ? watchdog_timer_fn+0x1b4/0x220 [35758.675071] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35758.693810] ? __hrtimer_run_queues+0x1d3/0x360 [35758.700934] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35758.706154] ? hrtimer_interrupt+0x104/0x240 [35758.708596] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35758.710605] ? __local_bh_enable+0x50/0x60 [35758.714438] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35758.718703] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35758.725824] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35758.730350] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35758.737477] [35758.741737] [35758.781675] [35758.783772] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35758.789080] ? lock_acquire+0xf0/0x2b0 [35758.792829] ? igrab+0x43/0x50 [35758.795880] ? lock_release+0xc5/0x260 [35758.799624] _raw_spin_lock+0x30/0x40 [35758.803280] ? igrab+0x1d/0x50 [35758.806332] igrab+0x1d/0x50 [35758.809211] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35758.814360] delayed_work+0x1d8/0x280 [ceph] [35758.818641] ? process_scheduled_works+0x22a/0x510 [35758.823442] process_scheduled_works+0x2a4/0x510 [35758.828057] worker_thread+0x188/0x340 [35758.831805] ? rescuer_thread+0x310/0x310 [35758.835807] kthread+0xf5/0x130 [35758.838946] ? kthread_complete_and_exit+0x20/0x20 [35758.843730] ret_from_fork+0x31/0x40 [35758.847309] ? kthread_complete_and_exit+0x20/0x20 [35758.852094] ret_from_fork_asm+0x11/0x20 [35758.856026] [35786.084185] watchdog: BUG: soft lockup - CPU#0 stuck for 22214s! [dbench:78683] [35786.091487] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35786.096185] watchdog: BUG: soft lockup - CPU#7 stuck for 31211s! [kworker/7:4:78808] [35786.146925] irq event stamp: 62393464 [35786.146926] hardirqs last enabled at (62393463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35786.154656] Modules linked in: [35786.158314] hardirqs last disabled at (62393464): [] sysvec_apic_timer_interrupt+0xe/0x90 [35786.168472] ceph [35786.171523] softirqs last enabled at (62393386): [] __do_softirq+0x2be/0x371 [35786.181247] libceph dns_resolver [35786.183172] softirqs last disabled at (62393381): [] irq_exit_rcu+0xb0/0xd0 [35786.191860] fscache netfs [35786.195175] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35786.203684] veth nft_chain_nat [35786.206389] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35786.215334] xt_MASQUERADE nf_nat [35786.218472] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35786.225943] nf_conntrack [35786.229253] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35786.234731] nf_defrag_ipv6 nf_defrag_ipv4 [35786.237348] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35786.256085] nft_compat nf_tables [35786.260175] [35786.260176] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35786.265394] nfnetlink bridge [35786.268705] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35786.270196] stp [35786.277319] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35786.280284] llc xfs [35786.287407] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35786.289245] libcrc32c sunrpc [35786.296370] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35786.298552] intel_rapl_msr intel_rapl_common [35786.305675] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35786.308642] sb_edac [35786.315764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35786.320115] x86_pkg_temp_thermal coretemp [35786.328194] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35786.330377] kvm_intel kvm [35786.336116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35786.340206] irqbypass i2c_i801 [35786.347330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35786.350034] iTCO_wdt iTCO_vendor_support [35786.357159] Call Trace: [35786.357160] [35786.360296] i2c_smbus crct10dif_pclmul [35786.367421] ? watchdog_timer_fn+0x1b4/0x220 [35786.371423] joydev crc32_pclmul [35786.373869] ? __hrtimer_run_queues+0x1d3/0x360 [35786.375887] ipmi_ssif ghash_clmulni_intel [35786.379722] ? hrtimer_interrupt+0x104/0x240 [35786.383980] mei_me mei [35786.387205] ? __local_bh_enable+0x50/0x60 [35786.391729] lpc_ich sha512_ssse3 [35786.395824] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35786.400085] acpi_ipmi mfd_core [35786.402528] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35786.406620] wmi ipmi_si [35786.409930] [35786.415147] ipmi_devintf ipmi_msghandler [35786.418285] [35786.423243] ioatdma acpi_pad [35786.425774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35786.427870] acpi_power_meter ixgbe [35786.431878] ? queued_spin_lock_slowpath+0x16/0x2e0 [35786.433972] igb mdio [35786.436937] do_raw_spin_lock+0xb5/0xc0 [35786.442240] crc32c_intel nvme [35786.445724] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35786.450594] ptp nvme_core [35786.452867] try_get_cap_refs+0x159/0x880 [ceph] [35786.456697] i2c_algo_bit pps_core [35786.459753] __ceph_get_caps+0x4a1/0x760 [ceph] [35786.464530] dca fuse [35786.467239] ? __file_remove_privs+0x89/0x110 [35786.471861] [35786.471862] irq event stamp: 207032430 [35786.475260] ? wake_bit_function+0x70/0x70 [35786.479785] hardirqs last enabled at (207032429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35786.482058] ? ceph_write_iter+0x260/0x770 [ceph] [35786.486405] hardirqs last disabled at (207032430): [] sysvec_apic_timer_interrupt+0xe/0x90 [35786.487897] ceph_write_iter+0x260/0x770 [ceph] [35786.491641] softirqs last enabled at (207032428): [] __do_softirq+0x2be/0x371 [35786.495737] ? lock_is_held_type+0xaa/0x120 [35786.505975] softirqs last disabled at (207032423): [] irq_exit_rcu+0xb0/0xd0 [35786.510675] vfs_write+0x324/0x4f0 [35786.520485] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35786.525012] ksys_pwrite64+0x65/0xa0 [35786.533778] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35786.537957] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35786.546554] Workqueue: events delayed_work [ceph] [35786.549952] do_syscall_64+0x38/0x80 [35786.559328] [35786.559329] RIP: 0010:lock_acquire+0xf0/0x2b0 [35786.562901] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35786.570371] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35786.575502] RIP: 0033:0x7f48f0eeb93b [35786.580199] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35786.583768] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35786.585259] [35786.585260] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35786.589610] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35786.594654] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35786.613392] ORIG_RAX: 0000000000000012 [35786.616962] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35786.622181] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35786.640917] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35786.642408] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35786.649533] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35786.654750] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35786.661873] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35786.665706] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35786.672828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35786.679953] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35786.687078] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35786.694206] [35786.701325] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35786.752945] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35786.760070] Call Trace: [35786.762516] [35786.764527] ? watchdog_timer_fn+0x1b4/0x220 [35786.768789] ? __hrtimer_run_queues+0x1d3/0x360 [35786.773316] ? hrtimer_interrupt+0x104/0x240 [35786.777590] ? __local_bh_enable+0x50/0x60 [35786.781689] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35786.786913] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35786.791868] [35786.793967] [35786.796065] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35786.801371] ? lock_acquire+0xf0/0x2b0 [35786.805124] ? igrab+0x43/0x50 [35786.808179] ? lock_release+0xc5/0x260 [35786.811926] _raw_spin_lock+0x30/0x40 [35786.815590] ? igrab+0x1d/0x50 [35786.818639] igrab+0x1d/0x50 [35786.821519] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35786.826670] delayed_work+0x1d8/0x280 [ceph] [35786.830949] ? process_scheduled_works+0x22a/0x510 [35786.835733] process_scheduled_works+0x2a4/0x510 [35786.840347] worker_thread+0x188/0x340 [35786.844098] ? rescuer_thread+0x310/0x310 [35786.848108] kthread+0xf5/0x130 [35786.851244] ? kthread_complete_and_exit+0x20/0x20 [35786.856029] ret_from_fork+0x31/0x40 [35786.859600] ? kthread_complete_and_exit+0x20/0x20 [35786.864385] ret_from_fork_asm+0x11/0x20 [35786.868309] [35814.084624] watchdog: BUG: soft lockup - CPU#0 stuck for 22240s! [dbench:78683] [35814.091931] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35814.096624] watchdog: BUG: soft lockup - CPU#7 stuck for 31237s! [kworker/7:4:78808] [35814.147380] irq event stamp: 62451602 [35814.155111] Modules linked in: ceph libceph dns_resolver fscache netfs [35814.158767] hardirqs last enabled at (62451601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35814.165299] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [35814.175459] hardirqs last disabled at (62451602): [] sysvec_apic_timer_interrupt+0xe/0x90 [35814.181545] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [35814.191268] softirqs last enabled at (62451536): [] __do_softirq+0x2be/0x371 [35814.191272] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [35814.207700] softirqs last disabled at (62451531): [] irq_exit_rcu+0xb0/0xd0 [35814.214218] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [35814.222731] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35814.228726] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support [35814.237672] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35814.237674] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35814.243320] i2c_smbus crct10dif_pclmul [35814.250791] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35814.256268] joydev crc32_pclmul [35814.260099] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35814.278837] ipmi_ssif ghash_clmulni_intel [35814.282062] [35814.282063] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35814.287279] mei_me mei [35814.291388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35814.292877] lpc_ich sha512_ssse3 [35814.300018] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35814.302462] acpi_ipmi mfd_core [35814.309587] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35814.312897] wmi ipmi_si [35814.320023] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35814.323160] ipmi_devintf ipmi_msghandler [35814.330301] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35814.332833] ioatdma acpi_pad [35814.339957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35814.343959] acpi_power_meter [35814.352037] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35814.355018] ixgbe igb [35814.360757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35814.363721] mdio crc32c_intel [35814.370844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35814.373201] nvme [35814.380327] Call Trace: [35814.380330] [35814.383378] ptp nvme_core [35814.390504] ? watchdog_timer_fn+0x1b4/0x220 [35814.392424] i2c_algo_bit pps_core [35814.394871] ? __hrtimer_run_queues+0x1d3/0x360 [35814.396879] dca [35814.399587] ? hrtimer_interrupt+0x104/0x240 [35814.403849] fuse [35814.407246] ? __local_bh_enable+0x50/0x60 [35814.411768] [35814.411769] irq event stamp: 207253096 [35814.413610] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35814.417872] hardirqs last enabled at (207253095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35814.419796] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35814.423886] hardirqs last disabled at (207253096): [] sysvec_apic_timer_interrupt+0xe/0x90 [35814.425377] [35814.429120] softirqs last enabled at (207253094): [] __do_softirq+0x2be/0x371 [35814.434338] [35814.434340] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35814.444581] softirqs last disabled at (207253089): [] irq_exit_rcu+0xb0/0xd0 [35814.449545] ? queued_spin_lock_slowpath+0x18/0x2e0 [35814.459351] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35814.461450] do_raw_spin_lock+0xb5/0xc0 [35814.470219] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35814.472318] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35814.477622] Workqueue: events delayed_work [ceph] [35814.486220] try_get_cap_refs+0x159/0x880 [ceph] [35814.491087] [35814.500470] __ceph_get_caps+0x4a1/0x760 [ceph] [35814.504313] RIP: 0010:lock_acquire+0xf0/0x2b0 [35814.511788] ? __file_remove_privs+0x89/0x110 [35814.516569] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35814.521266] ? wake_bit_function+0x70/0x70 [35814.525878] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35814.527371] ? ceph_write_iter+0x260/0x770 [ceph] [35814.531891] [35814.536242] ceph_write_iter+0x260/0x770 [ceph] [35814.540593] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35814.559335] ? lock_is_held_type+0xaa/0x120 [35814.563422] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35814.568641] vfs_write+0x324/0x4f0 [35814.573335] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35814.574830] ksys_pwrite64+0x65/0xa0 [35814.579352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35814.586475] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35814.590652] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35814.597778] do_syscall_64+0x38/0x80 [35814.601175] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35814.608307] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35814.611878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35814.619002] RIP: 0033:0x7f48f0eeb93b [35814.624149] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35814.631275] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35814.634846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35814.642921] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35814.647967] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35814.653705] ORIG_RAX: 0000000000000012 [35814.657274] Call Trace: [35814.664399] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35814.683137] [35814.690261] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35814.695479] ? watchdog_timer_fn+0x1b4/0x220 [35814.702602] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35814.706433] ? __hrtimer_run_queues+0x1d3/0x360 [35814.708876] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35814.716026] ? hrtimer_interrupt+0x104/0x240 [35814.718030] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35814.725152] ? __local_bh_enable+0x50/0x60 [35814.729422] [35814.736544] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35814.771071] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35814.776028] [35814.778125] [35814.780223] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35814.785530] ? lock_acquire+0xf0/0x2b0 [35814.789282] ? igrab+0x43/0x50 [35814.792348] ? lock_release+0xc5/0x260 [35814.796093] _raw_spin_lock+0x30/0x40 [35814.799750] ? igrab+0x1d/0x50 [35814.802801] igrab+0x1d/0x50 [35814.805679] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35814.810828] delayed_work+0x1d8/0x280 [ceph] [35814.815108] ? process_scheduled_works+0x22a/0x510 [35814.819900] process_scheduled_works+0x2a4/0x510 [35814.824514] worker_thread+0x188/0x340 [35814.828264] ? rescuer_thread+0x310/0x310 [35814.832266] kthread+0xf5/0x130 [35814.835404] ? kthread_complete_and_exit+0x20/0x20 [35814.840190] ret_from_fork+0x31/0x40 [35814.843769] ? kthread_complete_and_exit+0x20/0x20 [35814.848553] ret_from_fork_asm+0x11/0x20 [35814.852484] [35842.085059] watchdog: BUG: soft lockup - CPU#0 stuck for 22266s! [dbench:78683] [35842.092364] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35842.097059] watchdog: BUG: soft lockup - CPU#7 stuck for 31263s! [kworker/7:4:78808] [35842.147807] irq event stamp: 62509674 [35842.147808] hardirqs last enabled at (62509673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35842.155537] Modules linked in: [35842.159195] hardirqs last disabled at (62509674): [] sysvec_apic_timer_interrupt+0xe/0x90 [35842.169352] ceph [35842.172401] softirqs last enabled at (62509596): [] __do_softirq+0x2be/0x371 [35842.182125] libceph dns_resolver [35842.184050] softirqs last disabled at (62509591): [] irq_exit_rcu+0xb0/0xd0 [35842.192733] fscache netfs [35842.196046] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35842.204556] veth nft_chain_nat [35842.207261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35842.216203] xt_MASQUERADE nf_nat [35842.219341] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [35842.226812] nf_conntrack nf_defrag_ipv6 [35842.230123] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [35842.235600] nf_defrag_ipv4 nft_compat [35842.239518] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35842.258256] nf_tables [35842.261999] [35842.262000] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35842.267218] nfnetlink bridge [35842.269575] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35842.271066] stp llc [35842.278190] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35842.281152] xfs libcrc32c [35842.288278] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35842.290463] sunrpc intel_rapl_msr [35842.297585] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35842.300290] intel_rapl_common sb_edac [35842.307413] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35842.310811] x86_pkg_temp_thermal coretemp [35842.317937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35842.321678] kvm_intel kvm [35842.329756] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35842.333847] irqbypass i2c_i801 [35842.339585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35842.342289] iTCO_wdt iTCO_vendor_support [35842.349413] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35842.352550] i2c_smbus crct10dif_pclmul [35842.359674] Call Trace: [35842.359675] [35842.363679] joydev crc32_pclmul [35842.370804] ? watchdog_timer_fn+0x1b4/0x220 [35842.374635] ipmi_ssif [35842.377079] ? __hrtimer_run_queues+0x1d3/0x360 [35842.379089] ghash_clmulni_intel [35842.382315] ? hrtimer_interrupt+0x104/0x240 [35842.386577] mei_me mei [35842.388933] ? __local_bh_enable+0x50/0x60 [35842.393459] lpc_ich [35842.396685] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35842.400947] sha512_ssse3 acpi_ipmi [35842.403392] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35842.407480] mfd_core wmi [35842.409665] [35842.414882] ipmi_si ipmi_devintf [35842.418367] [35842.423323] ipmi_msghandler [35842.425943] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35842.428039] ioatdma acpi_pad [35842.431354] ? queued_spin_lock_slowpath+0x18/0x2e0 [35842.433456] acpi_power_meter [35842.436336] do_raw_spin_lock+0xb5/0xc0 [35842.441637] ixgbe igb mdio [35842.444602] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35842.449474] crc32c_intel [35842.452457] try_get_cap_refs+0x159/0x880 [ceph] [35842.456283] nvme ptp [35842.459078] __ceph_get_caps+0x4a1/0x760 [ceph] [35842.463858] nvme_core i2c_algo_bit [35842.466479] ? __file_remove_privs+0x89/0x110 [35842.471087] pps_core dca [35842.473359] ? wake_bit_function+0x70/0x70 [35842.477881] fuse [35842.481367] ? ceph_write_iter+0x260/0x770 [ceph] [35842.485717] [35842.485734] irq event stamp: 207474010 [35842.488351] ceph_write_iter+0x260/0x770 [ceph] [35842.492459] hardirqs last enabled at (207474009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35842.494386] ? lock_is_held_type+0xaa/0x120 [35842.499080] hardirqs last disabled at (207474010): [] sysvec_apic_timer_interrupt+0xe/0x90 [35842.500573] vfs_write+0x324/0x4f0 [35842.504315] softirqs last enabled at (207474008): [] __do_softirq+0x2be/0x371 [35842.508845] ksys_pwrite64+0x65/0xa0 [35842.519084] softirqs last disabled at (207474003): [] irq_exit_rcu+0xb0/0xd0 [35842.523261] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35842.533072] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35842.536471] do_syscall_64+0x38/0x80 [35842.545241] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35842.545242] Workqueue: events delayed_work [ceph] [35842.548812] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35842.557409] [35842.562539] RIP: 0033:0x7f48f0eeb93b [35842.571918] RIP: 0010:lock_acquire+0xf0/0x2b0 [35842.575488] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35842.582959] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35842.587656] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35842.592699] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35842.594190] ORIG_RAX: 0000000000000012 [35842.597760] [35842.597761] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35842.602112] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35842.620849] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35842.639586] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35842.644805] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35842.650022] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35842.653854] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35842.655344] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35842.662467] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35842.669592] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35842.676716] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35842.683845] [35842.690964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35842.706435] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [35842.712336] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35842.719465] rcu: 0-....: (23365341 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11699062 [35842.726585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35842.734663] rcu: 7-....: (23211551 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/23210206 fqs=11699063 [35842.736846] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35842.742586] rcu: (detected by 4, t=23905694 jiffies, g=1309609, q=664581 ncpus=8) [35842.748667] Call Trace: [35842.748668] [35842.755794] Sending NMI from CPU 4 to CPUs 0: [35842.766297] ? watchdog_timer_fn+0x1b4/0x220 [35842.773423] NMI backtrace for cpu 0 [35842.773425] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35842.773427] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35842.773428] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35842.773431] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35842.773433] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35842.773435] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35842.773436] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35842.773437] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35842.773438] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35842.773439] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35842.773440] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35842.773442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35842.773443] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35842.773445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35842.773446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35842.773447] Call Trace: [35842.773448] [35842.773449] ? nmi_cpu_backtrace+0xad/0x120 [35842.773454] ? nmi_cpu_backtrace_handler+0x11/0x20 [35842.773455] ? nmi_handle+0xd4/0x260 [35842.773461] ? default_do_nmi+0x49/0xf0 [35842.773464] ? exc_nmi+0x11e/0x150 [35842.773466] ? end_repeat_nmi+0x16/0x67 [35842.773472] ? queued_spin_lock_slowpath+0x16/0x2e0 [35842.773476] ? queued_spin_lock_slowpath+0x16/0x2e0 [35842.773479] ? queued_spin_lock_slowpath+0x16/0x2e0 [35842.773483] [35842.773483] [35842.773485] do_raw_spin_lock+0xb5/0xc0 [35842.773488] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35842.773507] try_get_cap_refs+0x159/0x880 [ceph] [35842.773528] __ceph_get_caps+0x4a1/0x760 [ceph] [35842.773547] ? __file_remove_privs+0x89/0x110 [35842.773550] ? wake_bit_function+0x70/0x70 [35842.773554] ? ceph_write_iter+0x260/0x770 [ceph] [35842.773569] ceph_write_iter+0x260/0x770 [ceph] [35842.773587] ? lock_is_held_type+0xaa/0x120 [35842.773591] vfs_write+0x324/0x4f0 [35842.773598] ksys_pwrite64+0x65/0xa0 [35842.773600] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35842.773604] do_syscall_64+0x38/0x80 [35842.773607] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35842.773610] RIP: 0033:0x7f48f0eeb93b [35842.773611] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35842.773613] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [35842.773615] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35842.773616] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35842.773617] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35842.773618] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35842.773619] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35842.773625] [35842.774422] Sending NMI from CPU 4 to CPUs 7: [35842.784100] ? __hrtimer_run_queues+0x1d3/0x360 [35842.791224] NMI backtrace for cpu 7 [35842.791225] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35842.791226] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35842.791227] Workqueue: events delayed_work [ceph] [35842.791241] RIP: 0010:lock_acquire+0xf5/0x2b0 [35842.791243] Code: b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 48 8b 44 24 30 <65> 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 c4 38 5b 5d 41 [35842.791244] RSP: 0018:ffffc90000264cb8 EFLAGS: 00000046 [35842.791245] RAX: c5e6ef7cc8fa9e00 RBX: 0000000000000000 RCX: 00000000de2c1856 [35842.791246] RDX: c31e29845e0f9ad2 RSI: 0000000000000000 RDI: 00000000000000c2 [35842.791246] RBP: 0000000000000001 R08: 00000000000000a0 R09: 0000000000000004 [35842.791247] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8255bce0 [35842.791248] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 [35842.791248] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35842.791249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35842.791250] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35842.791251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35842.791251] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35842.791252] Call Trace: [35842.791253] [35842.791253] ? nmi_cpu_backtrace+0xad/0x120 [35842.791256] ? nmi_cpu_backtrace_handler+0x11/0x20 [35842.791257] ? nmi_handle+0xd4/0x260 [35842.791261] ? default_do_nmi+0x49/0xf0 [35842.791262] ? exc_nmi+0x11e/0x150 [35842.791264] ? end_repeat_nmi+0x16/0x67 [35842.791268] ? lock_acquire+0xf5/0x2b0 [35842.791269] ? lock_acquire+0xf5/0x2b0 [35842.791271] ? lock_acquire+0xf5/0x2b0 [35842.791273] [35842.791273] [35842.791273] ? vprintk_emit+0x200/0x340 [35842.791276] ? lock_release+0xc5/0x260 [35842.791278] vprintk_emit+0x240/0x340 [35842.791280] ? vprintk_emit+0x221/0x340 [35842.791282] ? __hrtimer_run_queues+0x1d3/0x360 [35842.791283] ? __hrtimer_run_queues+0x1d3/0x360 [35842.791284] _printk+0x5c/0x80 [35842.791287] ? __hrtimer_run_queues+0x1d3/0x360 [35842.791288] show_trace_log_lvl+0x1ac/0x3f0 [35842.791293] ? _raw_spin_lock+0x30/0x40 [35842.791295] ? touch_softlockup_watchdog+0x20/0x20 [35842.791297] watchdog_timer_fn+0x1b4/0x220 [35842.791299] __hrtimer_run_queues+0x1d3/0x360 [35842.791301] hrtimer_interrupt+0x104/0x240 [35842.791303] ? __local_bh_enable+0x50/0x60 [35842.791306] __sysvec_apic_timer_interrupt+0x61/0x160 [35842.791309] sysvec_apic_timer_interrupt+0x6d/0x90 [35842.791310] [35842.791311] [35842.791311] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35842.791313] RIP: 0010:lock_acquire+0xf0/0x2b0 [35842.791315] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35842.791316] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35842.791316] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35842.791317] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35842.791318] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35842.791318] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35842.791319] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35842.791322] ? igrab+0x43/0x50 [35842.791324] ? lock_release+0xc5/0x260 [35842.791326] _raw_spin_lock+0x30/0x40 [35842.791328] ? igrab+0x1d/0x50 [35842.791328] igrab+0x1d/0x50 [35842.791330] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35842.791344] delayed_work+0x1d8/0x280 [ceph] [35842.791357] ? process_scheduled_works+0x22a/0x510 [35842.791360] process_scheduled_works+0x2a4/0x510 [35842.791364] worker_thread+0x188/0x340 [35842.791367] ? rescuer_thread+0x310/0x310 [35842.791369] kthread+0xf5/0x130 [35842.791372] ? kthread_complete_and_exit+0x20/0x20 [35842.791374] ret_from_fork+0x31/0x40 [35842.791376] ? kthread_complete_and_exit+0x20/0x20 [35842.791378] ret_from_fork_asm+0x11/0x20 [35842.791383] [35843.492944] ? hrtimer_interrupt+0x104/0x240 [35843.497213] ? __local_bh_enable+0x50/0x60 [35843.501307] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35843.506530] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35843.511489] [35843.513586] [35843.515683] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35843.520991] ? lock_acquire+0xf0/0x2b0 [35843.524756] ? igrab+0x43/0x50 [35843.527808] ? lock_release+0xc5/0x260 [35843.531552] _raw_spin_lock+0x30/0x40 [35843.535210] ? igrab+0x1d/0x50 [35843.538261] igrab+0x1d/0x50 [35843.541138] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35843.546288] delayed_work+0x1d8/0x280 [ceph] [35843.550567] ? process_scheduled_works+0x22a/0x510 [35843.555350] process_scheduled_works+0x2a4/0x510 [35843.559966] worker_thread+0x188/0x340 [35843.563715] ? rescuer_thread+0x310/0x310 [35843.567718] kthread+0xf5/0x130 [35843.570856] ? kthread_complete_and_exit+0x20/0x20 [35843.575640] ret_from_fork+0x31/0x40 [35843.579211] ? kthread_complete_and_exit+0x20/0x20 [35843.583995] ret_from_fork_asm+0x11/0x20 [35843.587918] [35870.085491] watchdog: BUG: soft lockup - CPU#0 stuck for 22292s! [dbench:78683] [35870.092801] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35870.097491] watchdog: BUG: soft lockup - CPU#7 stuck for 31289s! [kworker/7:4:78808] [35870.148251] irq event stamp: 62567842 [35870.155981] Modules linked in: ceph libceph dns_resolver fscache netfs [35870.159639] hardirqs last enabled at (62567841): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35870.159644] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [35870.176323] hardirqs last disabled at (62567842): [] sysvec_apic_timer_interrupt+0xe/0x90 [35870.185015] nft_compat nf_tables nfnetlink bridge stp llc xfs [35870.194738] softirqs last enabled at (62567800): [] __do_softirq+0x2be/0x371 [35870.194743] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [35870.209256] softirqs last disabled at (62567795): [] irq_exit_rcu+0xb0/0xd0 [35870.218372] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [35870.226886] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35870.232014] iTCO_vendor_support i2c_smbus [35870.240960] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35870.240961] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35870.245050] crct10dif_pclmul joydev [35870.252520] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35870.257998] crc32_pclmul [35870.261567] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35870.280306] ipmi_ssif ghash_clmulni_intel [35870.282924] [35870.288140] mei_me [35870.292232] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35870.293722] mei lpc_ich [35870.295820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35870.302944] sha512_ssse3 acpi_ipmi [35870.305475] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35870.312598] mfd_core [35870.316082] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35870.323207] wmi ipmi_si [35870.325478] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35870.332602] ipmi_devintf [35870.335133] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35870.342257] ipmi_msghandler [35870.344888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35870.352969] ioatdma [35870.355847] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35870.361582] acpi_pad acpi_power_meter [35870.363767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35870.370893] ixgbe [35870.374636] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35870.381760] igb mdio [35870.383770] Call Trace: [35870.383772] [35870.390895] crc32c_intel [35870.393181] ? watchdog_timer_fn+0x1b4/0x220 [35870.395627] nvme ptp [35870.397640] ? __hrtimer_run_queues+0x1d3/0x360 [35870.400256] nvme_core i2c_algo_bit [35870.404522] ? hrtimer_interrupt+0x104/0x240 [35870.406790] pps_core dca [35870.411315] ? __local_bh_enable+0x50/0x60 [35870.414799] fuse [35870.419066] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35870.421678] [35870.421679] irq event stamp: 207688868 [35870.425772] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35870.427694] hardirqs last enabled at (207688867): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35870.432912] [35870.434402] hardirqs last disabled at (207688868): [] sysvec_apic_timer_interrupt+0xe/0x90 [35870.438147] [35870.443121] softirqs last enabled at (207688866): [] __do_softirq+0x2be/0x371 [35870.453367] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35870.455463] softirqs last disabled at (207688861): [] irq_exit_rcu+0xb0/0xd0 [35870.465277] ? queued_spin_lock_slowpath+0x16/0x2e0 [35870.467371] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35870.476145] do_raw_spin_lock+0xb5/0xc0 [35870.481447] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35870.490045] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35870.494914] Workqueue: events delayed_work [ceph] [35870.504294] try_get_cap_refs+0x159/0x880 [ceph] [35870.508123] [35870.508124] RIP: 0010:lock_acquire+0xf0/0x2b0 [35870.515598] __ceph_get_caps+0x4a1/0x760 [ceph] [35870.520377] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35870.525080] ? __file_remove_privs+0x89/0x110 [35870.529685] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35870.531176] ? wake_bit_function+0x70/0x70 [35870.535528] [35870.535528] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35870.540053] ? ceph_write_iter+0x260/0x770 [ceph] [35870.558789] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35870.563140] ceph_write_iter+0x260/0x770 [ceph] [35870.568357] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35870.572450] ? lock_is_held_type+0xaa/0x120 [35870.573938] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35870.581064] vfs_write+0x324/0x4f0 [35870.585760] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35870.592897] ksys_pwrite64+0x65/0xa0 [35870.597416] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35870.604541] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35870.608719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35870.615843] do_syscall_64+0x38/0x80 [35870.619240] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35870.626365] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35870.629936] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35870.638013] RIP: 0033:0x7f48f0eeb93b [35870.643143] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35870.648894] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35870.652460] Call Trace: [35870.652463] [35870.659584] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35870.664630] ? watchdog_timer_fn+0x1b4/0x220 [35870.671753] ORIG_RAX: 0000000000000012 [35870.675325] ? __hrtimer_run_queues+0x1d3/0x360 [35870.682446] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35870.701188] ? hrtimer_interrupt+0x104/0x240 [35870.703628] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35870.705640] ? __local_bh_enable+0x50/0x60 [35870.710856] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35870.715124] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35870.718951] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35870.723475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35870.730600] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35870.734863] [35870.741993] [35870.746078] [35870.746080] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35870.789323] ? lock_acquire+0xf0/0x2b0 [35870.793071] ? igrab+0x43/0x50 [35870.796121] ? lock_release+0xc5/0x260 [35870.799868] _raw_spin_lock+0x30/0x40 [35870.803548] ? igrab+0x1d/0x50 [35870.806599] igrab+0x1d/0x50 [35870.809479] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35870.814629] delayed_work+0x1d8/0x280 [ceph] [35870.818910] ? process_scheduled_works+0x22a/0x510 [35870.823700] process_scheduled_works+0x2a4/0x510 [35870.828315] worker_thread+0x188/0x340 [35870.832064] ? rescuer_thread+0x310/0x310 [35870.836066] kthread+0xf5/0x130 [35870.839206] ? kthread_complete_and_exit+0x20/0x20 [35870.843990] ret_from_fork+0x31/0x40 [35870.847569] ? kthread_complete_and_exit+0x20/0x20 [35870.852352] ret_from_fork_asm+0x11/0x20 [35870.856275] [35898.085922] watchdog: BUG: soft lockup - CPU#0 stuck for 22318s! [dbench:78683] [35898.093227] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35898.097922] watchdog: BUG: soft lockup - CPU#7 stuck for 31315s! [kworker/7:4:78808] [35898.148668] irq event stamp: 62625918 [35898.148669] hardirqs last enabled at (62625917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35898.156399] Modules linked in: ceph [35898.160057] hardirqs last disabled at (62625918): [] sysvec_apic_timer_interrupt+0xe/0x90 [35898.170212] libceph [35898.173697] softirqs last enabled at (62625850): [] __do_softirq+0x2be/0x371 [35898.183422] dns_resolver [35898.185605] softirqs last disabled at (62625845): [] irq_exit_rcu+0xb0/0xd0 [35898.194289] fscache [35898.196924] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35898.205436] netfs [35898.207621] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35898.207623] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35898.216564] veth nft_chain_nat [35898.218575] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35898.226062] xt_MASQUERADE nf_nat [35898.231541] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35898.234676] nf_conntrack nf_defrag_ipv6 [35898.253414] [35898.253415] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35898.256726] nf_defrag_ipv4 nft_compat [35898.261944] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35898.265860] nf_tables nfnetlink [35898.267350] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35898.274475] bridge [35898.278221] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35898.285342] stp llc [35898.288567] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35898.295692] xfs [35898.297789] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35898.304913] libcrc32c [35898.307098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35898.314222] sunrpc [35898.316060] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35898.324136] intel_rapl_msr intel_rapl_common [35898.326493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35898.332230] sb_edac x86_pkg_temp_thermal [35898.334329] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35898.341452] coretemp kvm_intel [35898.345803] Call Trace: [35898.352928] kvm irqbypass [35898.356932] [35898.364054] i2c_i801 iTCO_wdt iTCO_vendor_support [35898.367194] ? watchdog_timer_fn+0x1b4/0x220 [35898.369636] i2c_smbus crct10dif_pclmul [35898.372344] ? __hrtimer_run_queues+0x1d3/0x360 [35898.374351] joydev crc32_pclmul [35898.379138] ? hrtimer_interrupt+0x104/0x240 [35898.383400] ipmi_ssif ghash_clmulni_intel [35898.387230] ? __local_bh_enable+0x50/0x60 [35898.391756] mei_me mei [35898.394983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35898.399243] lpc_ich sha512_ssse3 [35898.403333] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35898.407423] acpi_ipmi mfd_core [35898.409869] [35898.415086] wmi [35898.418396] [35898.423355] ipmi_si ipmi_devintf [35898.426492] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35898.428601] ipmi_msghandler ioatdma [35898.430438] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35898.432531] acpi_pad [35898.435845] do_raw_spin_lock+0xb5/0xc0 [35898.441148] acpi_power_meter [35898.444719] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35898.449587] ixgbe igb [35898.451879] try_get_cap_refs+0x159/0x880 [ceph] [35898.455706] mdio crc32c_intel [35898.458675] __ceph_get_caps+0x4a1/0x760 [ceph] [35898.463456] nvme ptp [35898.465817] ? __file_remove_privs+0x89/0x110 [35898.470425] nvme_core i2c_algo_bit [35898.473474] ? wake_bit_function+0x70/0x70 [35898.477998] pps_core [35898.480273] ? ceph_write_iter+0x260/0x770 [ceph] [35898.484619] dca fuse [35898.488105] ceph_write_iter+0x260/0x770 [ceph] [35898.492194] [35898.492194] irq event stamp: 207909656 [35898.494470] ? lock_is_held_type+0xaa/0x120 [35898.499162] hardirqs last enabled at (207909655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35898.501435] vfs_write+0x324/0x4f0 [35898.505957] hardirqs last disabled at (207909656): [] sysvec_apic_timer_interrupt+0xe/0x90 [35898.507452] ksys_pwrite64+0x65/0xa0 [35898.511192] softirqs last enabled at (207909654): [] __do_softirq+0x2be/0x371 [35898.515371] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35898.525614] softirqs last disabled at (207909649): [] irq_exit_rcu+0xb0/0xd0 [35898.529012] do_syscall_64+0x38/0x80 [35898.538823] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35898.542394] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35898.551165] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35898.556294] RIP: 0033:0x7f48f0eeb93b [35898.564892] Workqueue: events delayed_work [ceph] [35898.568463] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35898.577840] [35898.577841] RIP: 0010:lock_acquire+0xf0/0x2b0 [35898.582885] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35898.590356] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35898.593925] ORIG_RAX: 0000000000000012 [35898.598624] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35898.617361] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35898.618852] [35898.618853] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35898.623202] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35898.628419] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35898.647158] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35898.650989] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35898.656207] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35898.663329] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35898.664821] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35898.671945] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35898.679074] [35898.686192] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35898.739208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35898.744946] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35898.752071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35898.759195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35898.766326] Call Trace: [35898.768773] [35898.770784] ? watchdog_timer_fn+0x1b4/0x220 [35898.775048] ? __hrtimer_run_queues+0x1d3/0x360 [35898.779573] ? hrtimer_interrupt+0x104/0x240 [35898.783833] ? __local_bh_enable+0x50/0x60 [35898.787927] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35898.793150] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35898.798109] [35898.800204] [35898.802305] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35898.807637] ? lock_acquire+0xf0/0x2b0 [35898.811387] ? igrab+0x43/0x50 [35898.814436] ? lock_release+0xc5/0x260 [35898.818182] _raw_spin_lock+0x30/0x40 [35898.821838] ? igrab+0x1d/0x50 [35898.824889] igrab+0x1d/0x50 [35898.827768] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35898.832918] delayed_work+0x1d8/0x280 [ceph] [35898.837197] ? process_scheduled_works+0x22a/0x510 [35898.841978] process_scheduled_works+0x2a4/0x510 [35898.846595] worker_thread+0x188/0x340 [35898.850362] ? rescuer_thread+0x310/0x310 [35898.854364] kthread+0xf5/0x130 [35898.857502] ? kthread_complete_and_exit+0x20/0x20 [35898.862286] ret_from_fork+0x31/0x40 [35898.865857] ? kthread_complete_and_exit+0x20/0x20 [35898.870642] ret_from_fork_asm+0x11/0x20 [35898.874575] [35926.086348] watchdog: BUG: soft lockup - CPU#0 stuck for 22344s! [dbench:78683] [35926.093654] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35926.098348] watchdog: BUG: soft lockup - CPU#7 stuck for 31341s! [kworker/7:4:78808] [35926.149105] irq event stamp: 62684054 [35926.156835] Modules linked in: ceph libceph dns_resolver fscache [35926.160493] hardirqs last enabled at (62684053): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35926.160497] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [35926.176654] hardirqs last disabled at (62684054): [] sysvec_apic_timer_interrupt+0xe/0x90 [35926.176657] nf_defrag_ipv6 [35926.176659] softirqs last enabled at (62683986): [] __do_softirq+0x2be/0x371 [35926.183259] nf_defrag_ipv4 [35926.192984] softirqs last disabled at (62683981): [] irq_exit_rcu+0xb0/0xd0 [35926.195774] nft_compat nf_tables [35926.204460] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35926.207250] nfnetlink bridge [35926.215759] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35926.219070] stp llc [35926.228015] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35926.230978] xfs libcrc32c [35926.238450] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35926.240633] sunrpc intel_rapl_msr [35926.246112] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35926.248814] intel_rapl_common sb_edac [35926.267552] [35926.270950] x86_pkg_temp_thermal [35926.276166] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35926.279911] coretemp kvm_intel [35926.281402] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35926.284712] kvm irqbypass [35926.291837] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35926.294975] i2c_i801 iTCO_wdt [35926.302099] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35926.304803] iTCO_vendor_support i2c_smbus [35926.311928] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35926.314979] crct10dif_pclmul joydev [35926.322101] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35926.326193] crc32_pclmul ipmi_ssif [35926.333315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35926.336887] ghash_clmulni_intel [35926.344965] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35926.348449] mei_me mei [35926.354186] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35926.357411] lpc_ich sha512_ssse3 [35926.364535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35926.366979] acpi_ipmi [35926.374102] Call Trace: [35926.374104] [35926.377414] mfd_core wmi [35926.384539] ? watchdog_timer_fn+0x1b4/0x220 [35926.386896] ipmi_si ipmi_devintf [35926.389341] ? __hrtimer_run_queues+0x1d3/0x360 [35926.391350] ipmi_msghandler ioatdma [35926.393971] ? hrtimer_interrupt+0x104/0x240 [35926.398232] acpi_pad acpi_power_meter [35926.401541] ? __local_bh_enable+0x50/0x60 [35926.406066] ixgbe igb [35926.409640] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35926.413900] mdio crc32c_intel [35926.417644] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35926.421747] nvme ptp [35926.424101] [35926.429319] nvme_core i2c_algo_bit [35926.432370] [35926.432372] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35926.437328] pps_core dca [35926.439601] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35926.441697] fuse [35926.445181] do_raw_spin_lock+0xb5/0xc0 [35926.447276] [35926.447276] irq event stamp: 208130402 [35926.452584] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35926.455199] hardirqs last enabled at (208130401): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35926.460072] try_get_cap_refs+0x159/0x880 [ceph] [35926.461994] hardirqs last disabled at (208130402): [] sysvec_apic_timer_interrupt+0xe/0x90 [35926.465830] __ceph_get_caps+0x4a1/0x760 [ceph] [35926.467314] softirqs last enabled at (208130400): [] __do_softirq+0x2be/0x371 [35926.471064] ? __file_remove_privs+0x89/0x110 [35926.475843] softirqs last disabled at (208130395): [] irq_exit_rcu+0xb0/0xd0 [35926.486087] ? wake_bit_function+0x70/0x70 [35926.490699] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35926.500510] ? ceph_write_iter+0x260/0x770 [ceph] [35926.505032] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35926.513803] ceph_write_iter+0x260/0x770 [ceph] [35926.518153] Workqueue: events delayed_work [ceph] [35926.526755] ? lock_is_held_type+0xaa/0x120 [35926.530843] [35926.540223] vfs_write+0x324/0x4f0 [35926.544917] RIP: 0010:lock_acquire+0xf0/0x2b0 [35926.552393] ksys_pwrite64+0x65/0xa0 [35926.556912] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35926.561611] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35926.565786] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35926.567279] do_syscall_64+0x38/0x80 [35926.570675] [35926.570676] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35926.575041] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35926.578613] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35926.597350] RIP: 0033:0x7f48f0eeb93b [35926.602482] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35926.602484] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35926.607701] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35926.611270] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35926.612760] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35926.619884] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35926.624929] ORIG_RAX: 0000000000000012 [35926.632053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35926.635623] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35926.642748] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35926.649873] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35926.668609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35926.675751] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35926.680969] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35926.689045] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35926.692877] Call Trace: [35926.692878] [35926.698615] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35926.705752] ? watchdog_timer_fn+0x1b4/0x220 [35926.712876] [35926.719997] ? __hrtimer_run_queues+0x1d3/0x360 [35926.771048] ? hrtimer_interrupt+0x104/0x240 [35926.775317] ? __local_bh_enable+0x50/0x60 [35926.779409] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35926.784633] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35926.789590] [35926.791687] [35926.793787] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35926.799104] ? lock_acquire+0xf0/0x2b0 [35926.802853] ? igrab+0x43/0x50 [35926.805903] ? lock_release+0xc5/0x260 [35926.809648] _raw_spin_lock+0x30/0x40 [35926.813312] ? igrab+0x1d/0x50 [35926.816364] igrab+0x1d/0x50 [35926.819241] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35926.824393] delayed_work+0x1d8/0x280 [ceph] [35926.828672] ? process_scheduled_works+0x22a/0x510 [35926.833463] process_scheduled_works+0x2a4/0x510 [35926.838079] worker_thread+0x188/0x340 [35926.841828] ? rescuer_thread+0x310/0x310 [35926.845839] kthread+0xf5/0x130 [35926.848986] ? kthread_complete_and_exit+0x20/0x20 [35926.853771] ret_from_fork+0x31/0x40 [35926.857349] ? kthread_complete_and_exit+0x20/0x20 [35926.862133] ret_from_fork_asm+0x11/0x20 [35926.866065] [35954.086774] watchdog: BUG: soft lockup - CPU#0 stuck for 22370s! [dbench:78683] [35954.094080] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35954.098774] watchdog: BUG: soft lockup - CPU#7 stuck for 31367s! [kworker/7:4:78808] [35954.149521] irq event stamp: 62742156 [35954.149523] hardirqs last enabled at (62742155): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35954.157252] Modules linked in: [35954.160908] hardirqs last disabled at (62742156): [] sysvec_apic_timer_interrupt+0xe/0x90 [35954.171066] ceph [35954.174118] softirqs last enabled at (62742120): [] __do_softirq+0x2be/0x371 [35954.183843] libceph [35954.185766] softirqs last disabled at (62742115): [] irq_exit_rcu+0xb0/0xd0 [35954.194450] dns_resolver fscache [35954.196635] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35954.205144] netfs [35954.208473] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35954.217416] veth nft_chain_nat [35954.219427] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [35954.226899] xt_MASQUERADE [35954.230037] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [35954.235514] nf_nat nf_conntrack [35954.238217] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35954.256954] nf_defrag_ipv6 nf_defrag_ipv4 [35954.260180] [35954.265397] nft_compat nf_tables [35954.269488] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35954.270977] nfnetlink bridge [35954.274289] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35954.281414] stp llc [35954.284377] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35954.291501] xfs [35954.293685] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35954.300809] libcrc32c sunrpc [35954.302645] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35954.309772] intel_rapl_msr [35954.312735] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35954.319859] intel_rapl_common sb_edac [35954.322649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35954.330728] x86_pkg_temp_thermal [35954.334470] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35954.340208] coretemp kvm_intel [35954.343520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35954.350643] kvm irqbypass [35954.353781] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35954.360905] i2c_i801 iTCO_wdt [35954.363609] Call Trace: [35954.363611] [35954.370733] iTCO_vendor_support i2c_smbus [35954.373784] ? watchdog_timer_fn+0x1b4/0x220 [35954.376227] crct10dif_pclmul joydev [35954.378239] ? __hrtimer_run_queues+0x1d3/0x360 [35954.382329] crc32_pclmul ipmi_ssif [35954.386596] ? hrtimer_interrupt+0x104/0x240 [35954.390170] ghash_clmulni_intel [35954.394696] ? __local_bh_enable+0x50/0x60 [35954.398180] mei_me mei [35954.402455] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35954.405679] lpc_ich sha512_ssse3 [35954.409769] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35954.412211] acpi_ipmi mfd_core [35954.417431] [35954.420759] wmi ipmi_si ipmi_devintf [35954.425716] [35954.428854] ipmi_msghandler ioatdma [35954.430952] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35954.434609] acpi_pad acpi_power_meter [35954.436708] ? queued_spin_lock_slowpath+0x16/0x2e0 [35954.440276] ixgbe igb mdio [35954.445581] do_raw_spin_lock+0xb5/0xc0 [35954.449325] crc32c_intel nvme [35954.454195] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35954.456985] ptp nvme_core [35954.460818] try_get_cap_refs+0x159/0x880 [ceph] [35954.463867] i2c_algo_bit pps_core [35954.468655] __ceph_get_caps+0x4a1/0x760 [ceph] [35954.471354] dca fuse [35954.475971] ? __file_remove_privs+0x89/0x110 [35954.479362] [35954.479363] irq event stamp: 208350958 [35954.483889] ? wake_bit_function+0x70/0x70 [35954.486175] hardirqs last enabled at (208350957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35954.490529] ? ceph_write_iter+0x260/0x770 [ceph] [35954.492017] hardirqs last disabled at (208350958): [] sysvec_apic_timer_interrupt+0xe/0x90 [35954.495761] ceph_write_iter+0x260/0x770 [ceph] [35954.499852] softirqs last enabled at (208350956): [] __do_softirq+0x2be/0x371 [35954.510101] ? lock_is_held_type+0xaa/0x120 [35954.514793] softirqs last disabled at (208350951): [] irq_exit_rcu+0xb0/0xd0 [35954.524607] vfs_write+0x324/0x4f0 [35954.529129] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35954.537903] ksys_pwrite64+0x65/0xa0 [35954.542077] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35954.542079] Workqueue: events delayed_work [ceph] [35954.550674] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35954.554070] [35954.554071] RIP: 0010:rcu_is_watching+0x0/0x40 [35954.563450] do_syscall_64+0x38/0x80 [35954.567036] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [35954.574508] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35954.579204] RSP: 0018:ffffc90002117d10 EFLAGS: 00000247 [35954.584337] RIP: 0033:0x7f48f0eeb93b [35954.585825] [35954.585826] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [35954.590265] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35954.593835] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [35954.612573] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35954.617617] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [35954.622834] ORIG_RAX: 0000000000000012 [35954.626405] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [35954.627896] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35954.635020] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35954.635021] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35954.653757] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35954.660882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35954.666098] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35954.673223] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35954.677054] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35954.684178] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35954.691301] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35954.698424] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35954.706509] [35954.713628] Call Trace: [35954.766740] [35954.768749] ? watchdog_timer_fn+0x1b4/0x220 [35954.773014] ? __hrtimer_run_queues+0x1d3/0x360 [35954.777540] ? hrtimer_interrupt+0x104/0x240 [35954.781810] ? __local_bh_enable+0x50/0x60 [35954.785903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35954.791127] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35954.796084] [35954.798182] [35954.800279] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35954.805587] ? rcutorture_get_gp_data+0x30/0x30 [35954.810117] lock_acquire+0x184/0x2b0 [35954.813774] ? igrab+0x43/0x50 [35954.816825] ? lock_release+0xc5/0x260 [35954.820570] _raw_spin_lock+0x30/0x40 [35954.824235] ? igrab+0x1d/0x50 [35954.827283] igrab+0x1d/0x50 [35954.830180] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35954.835331] delayed_work+0x1d8/0x280 [ceph] [35954.839610] ? process_scheduled_works+0x22a/0x510 [35954.844392] process_scheduled_works+0x2a4/0x510 [35954.849007] worker_thread+0x188/0x340 [35954.852758] ? rescuer_thread+0x310/0x310 [35954.856760] kthread+0xf5/0x130 [35954.859898] ? kthread_complete_and_exit+0x20/0x20 [35954.864682] ret_from_fork+0x31/0x40 [35954.868254] ? kthread_complete_and_exit+0x20/0x20 [35954.873037] ret_from_fork_asm+0x11/0x20 [35954.876961] [35982.087208] watchdog: BUG: soft lockup - CPU#0 stuck for 22396s! [dbench:78683] [35982.094516] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [35982.099208] watchdog: BUG: soft lockup - CPU#7 stuck for 31393s! [kworker/7:4:78808] [35982.149965] irq event stamp: 62800228 [35982.157695] Modules linked in: ceph libceph dns_resolver fscache netfs [35982.161353] hardirqs last enabled at (62800227): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35982.167883] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [35982.178045] hardirqs last disabled at (62800228): [] sysvec_apic_timer_interrupt+0xe/0x90 [35982.178048] nf_defrag_ipv6 [35982.178049] softirqs last enabled at (62800182): [] __do_softirq+0x2be/0x371 [35982.184129] nf_defrag_ipv4 nft_compat [35982.193855] softirqs last disabled at (62800177): [] irq_exit_rcu+0xb0/0xd0 [35982.196645] nf_tables [35982.205332] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35982.209072] nfnetlink bridge [35982.217601] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35982.219957] stp llc [35982.228907] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [35982.231882] xfs [35982.239356] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [35982.241539] libcrc32c sunrpc [35982.247016] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [35982.248853] intel_rapl_msr intel_rapl_common [35982.267606] [35982.270572] sb_edac [35982.275791] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [35982.280140] x86_pkg_temp_thermal coretemp [35982.281630] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [35982.283816] kvm_intel kvm [35982.290940] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [35982.295030] irqbypass [35982.302154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [35982.304857] i2c_i801 iTCO_wdt [35982.311983] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [35982.314341] iTCO_vendor_support [35982.321465] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [35982.324514] i2c_smbus crct10dif_pclmul [35982.331638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35982.334863] joydev crc32_pclmul ipmi_ssif [35982.342957] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [35982.346788] ghash_clmulni_intel [35982.352526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35982.356616] mei_me [35982.363742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35982.366965] mei lpc_ich [35982.374088] Call Trace: [35982.374091] [35982.376187] sha512_ssse3 acpi_ipmi [35982.383312] ? watchdog_timer_fn+0x1b4/0x220 [35982.385841] mfd_core wmi [35982.388286] ? __hrtimer_run_queues+0x1d3/0x360 [35982.390297] ipmi_si ipmi_devintf [35982.393784] ? hrtimer_interrupt+0x104/0x240 [35982.398043] ipmi_msghandler ioatdma [35982.400661] ? __local_bh_enable+0x50/0x60 [35982.405187] acpi_pad acpi_power_meter [35982.408499] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35982.412760] ixgbe igb [35982.416331] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35982.420421] mdio crc32c_intel [35982.424166] [35982.429384] nvme ptp [35982.431740] [35982.436699] nvme_core [35982.439751] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35982.441846] i2c_algo_bit pps_core [35982.444120] ? queued_spin_lock_slowpath+0x2d/0x2e0 [35982.446215] dca fuse [35982.448574] do_raw_spin_lock+0xb5/0xc0 [35982.453875] [35982.453876] irq event stamp: 208571740 [35982.457277] __cap_delay_requeue+0xa7/0x1d0 [ceph] [35982.462145] hardirqs last enabled at (208571739): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35982.464417] try_get_cap_refs+0x159/0x880 [ceph] [35982.468245] hardirqs last disabled at (208571740): [] sysvec_apic_timer_interrupt+0xe/0x90 [35982.469742] __ceph_get_caps+0x4a1/0x760 [ceph] [35982.473479] softirqs last enabled at (208571738): [] __do_softirq+0x2be/0x371 [35982.478270] ? __file_remove_privs+0x89/0x110 [35982.488509] softirqs last disabled at (208571733): [] irq_exit_rcu+0xb0/0xd0 [35982.493119] ? wake_bit_function+0x70/0x70 [35982.502930] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [35982.507456] ? ceph_write_iter+0x260/0x770 [ceph] [35982.516225] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [35982.520575] ceph_write_iter+0x260/0x770 [ceph] [35982.529175] Workqueue: events delayed_work [ceph] [35982.533267] ? lock_is_held_type+0xaa/0x120 [35982.542642] [35982.547341] vfs_write+0x324/0x4f0 [35982.554809] RIP: 0010:lock_acquire+0xf0/0x2b0 [35982.559339] ksys_pwrite64+0x65/0xa0 [35982.564032] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [35982.568209] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [35982.569699] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [35982.573098] do_syscall_64+0x38/0x80 [35982.577448] [35982.577449] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [35982.581021] entry_SYSCALL_64_after_hwframe+0x63/0xcd [35982.599756] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [35982.604891] RIP: 0033:0x7f48f0eeb93b [35982.610103] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [35982.613676] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [35982.615165] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [35982.622289] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [35982.627333] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [35982.634458] ORIG_RAX: 0000000000000012 [35982.638030] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [35982.645152] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [35982.663889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35982.671032] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [35982.676250] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [35982.683372] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [35982.687205] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35982.695281] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [35982.702406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35982.708143] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [35982.715268] Call Trace: [35982.715270] [35982.722398] [35982.729516] ? watchdog_timer_fn+0x1b4/0x220 [35982.768925] ? __hrtimer_run_queues+0x1d3/0x360 [35982.773453] ? hrtimer_interrupt+0x104/0x240 [35982.777720] ? __local_bh_enable+0x50/0x60 [35982.781815] ? __sysvec_apic_timer_interrupt+0x61/0x160 [35982.787039] ? sysvec_apic_timer_interrupt+0x6d/0x90 [35982.791996] [35982.794092] [35982.796191] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [35982.801506] ? lock_acquire+0xf0/0x2b0 [35982.805256] ? igrab+0x43/0x50 [35982.808307] ? lock_release+0xc5/0x260 [35982.812051] _raw_spin_lock+0x30/0x40 [35982.815708] ? igrab+0x1d/0x50 [35982.818758] igrab+0x1d/0x50 [35982.821637] ceph_check_delayed_caps+0xb7/0x270 [ceph] [35982.826787] delayed_work+0x1d8/0x280 [ceph] [35982.831068] ? process_scheduled_works+0x22a/0x510 [35982.835857] process_scheduled_works+0x2a4/0x510 [35982.840474] worker_thread+0x188/0x340 [35982.844224] ? rescuer_thread+0x310/0x310 [35982.848235] kthread+0xf5/0x130 [35982.851381] ? kthread_complete_and_exit+0x20/0x20 [35982.856166] ret_from_fork+0x31/0x40 [35982.859746] ? kthread_complete_and_exit+0x20/0x20 [35982.864528] ret_from_fork_asm+0x11/0x20 [35982.868453] [36010.087644] watchdog: BUG: soft lockup - CPU#0 stuck for 22423s! [dbench:78683] [36010.094943] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36010.099643] watchdog: BUG: soft lockup - CPU#7 stuck for 31420s! [kworker/7:4:78808] [36010.150384] irq event stamp: 62858344 [36010.150386] hardirqs last enabled at (62858343): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36010.158113] Modules linked in: [36010.161770] hardirqs last disabled at (62858344): [] sysvec_apic_timer_interrupt+0xe/0x90 [36010.171927] ceph [36010.174978] softirqs last enabled at (62858328): [] __do_softirq+0x2be/0x371 [36010.184704] libceph [36010.186628] softirqs last disabled at (62858323): [] irq_exit_rcu+0xb0/0xd0 [36010.195327] dns_resolver fscache [36010.197514] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36010.206038] netfs veth [36010.209353] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36010.218313] nft_chain_nat [36010.220756] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36010.228229] xt_MASQUERADE nf_nat [36010.230933] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36010.236409] nf_conntrack [36010.239721] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36010.258457] nf_defrag_ipv6 nf_defrag_ipv4 [36010.261075] [36010.266292] nft_compat nf_tables [36010.270383] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36010.271873] nfnetlink bridge [36010.275184] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36010.282320] stp llc [36010.285283] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36010.292405] xfs libcrc32c [36010.294591] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36010.301713] sunrpc intel_rapl_msr [36010.304418] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36010.311542] intel_rapl_common [36010.314941] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36010.322064] sb_edac [36010.325116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36010.333193] x86_pkg_temp_thermal coretemp [36010.335375] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36010.341115] kvm_intel kvm [36010.345206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36010.352328] irqbypass i2c_i801 [36010.355039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36010.362164] iTCO_wdt iTCO_vendor_support [36010.365303] Call Trace: [36010.372427] i2c_smbus crct10dif_pclmul [36010.376431] [36010.378876] joydev crc32_pclmul [36010.382706] ? watchdog_timer_fn+0x1b4/0x220 [36010.384715] ipmi_ssif ghash_clmulni_intel [36010.387942] ? __hrtimer_run_queues+0x1d3/0x360 [36010.392203] mei_me mei [36010.396299] ? hrtimer_interrupt+0x104/0x240 [36010.400819] lpc_ich sha512_ssse3 [36010.403264] ? __local_bh_enable+0x50/0x60 [36010.407528] acpi_ipmi [36010.410841] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36010.414929] mfd_core wmi [36010.417287] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36010.422503] ipmi_si ipmi_devintf ipmi_msghandler [36010.425122] [36010.430079] ioatdma acpi_pad [36010.434776] [36010.436872] acpi_power_meter [36010.439839] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36010.441935] ixgbe igb [36010.444904] ? queued_spin_lock_slowpath+0x16/0x2e0 [36010.450204] mdio crc32c_intel [36010.452563] do_raw_spin_lock+0xb5/0xc0 [36010.457432] nvme ptp [36010.460484] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36010.464314] nvme_core i2c_algo_bit [36010.466584] try_get_cap_refs+0x159/0x880 [ceph] [36010.471368] pps_core dca [36010.474855] __ceph_get_caps+0x4a1/0x760 [ceph] [36010.479463] fuse [36010.482084] ? __file_remove_privs+0x89/0x110 [36010.486604] irq event stamp: 208792426 [36010.488527] ? wake_bit_function+0x70/0x70 [36010.492877] hardirqs last enabled at (208792425): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36010.496625] ? ceph_write_iter+0x260/0x770 [ceph] [36010.500713] hardirqs last disabled at (208792426): [] sysvec_apic_timer_interrupt+0xe/0x90 [36010.510959] ceph_write_iter+0x260/0x770 [ceph] [36010.515655] softirqs last enabled at (208792424): [] __do_softirq+0x2be/0x371 [36010.525470] ? lock_is_held_type+0xaa/0x120 [36010.529989] softirqs last disabled at (208792419): [] irq_exit_rcu+0xb0/0xd0 [36010.538764] vfs_write+0x324/0x4f0 [36010.542939] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36010.551541] ksys_pwrite64+0x65/0xa0 [36010.554932] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36010.564311] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36010.567881] Workqueue: events delayed_work [ceph] [36010.575352] do_syscall_64+0x38/0x80 [36010.580482] [36010.580483] RIP: 0010:lock_acquire+0x85/0x2b0 [36010.585182] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36010.588751] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [36010.590241] RIP: 0033:0x7f48f0eeb93b [36010.594593] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [36010.599638] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36010.618374] [36010.618375] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [36010.621945] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36010.627161] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [36010.645899] ORIG_RAX: 0000000000000012 [36010.647392] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [36010.654515] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36010.659733] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [36010.666857] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36010.670687] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36010.677811] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36010.684937] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36010.692058] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36010.699182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36010.706323] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36010.713448] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36010.721531] [36010.728651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36010.757945] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36010.765069] Call Trace: [36010.767513] [36010.769524] ? watchdog_timer_fn+0x1b4/0x220 [36010.773788] ? __hrtimer_run_queues+0x1d3/0x360 [36010.778330] ? hrtimer_interrupt+0x104/0x240 [36010.782594] ? __local_bh_enable+0x50/0x60 [36010.786686] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36010.791912] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36010.796877] [36010.798974] [36010.801073] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36010.806379] ? lock_acquire+0x85/0x2b0 [36010.810130] ? igrab+0x43/0x50 [36010.813180] ? lock_release+0xc5/0x260 [36010.816925] _raw_spin_lock+0x30/0x40 [36010.820580] ? igrab+0x1d/0x50 [36010.823632] igrab+0x1d/0x50 [36010.826509] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36010.831658] delayed_work+0x1d8/0x280 [ceph] [36010.835941] ? process_scheduled_works+0x22a/0x510 [36010.840730] process_scheduled_works+0x2a4/0x510 [36010.845347] worker_thread+0x188/0x340 [36010.849095] ? rescuer_thread+0x310/0x310 [36010.853099] kthread+0xf5/0x130 [36010.856237] ? kthread_complete_and_exit+0x20/0x20 [36010.861021] ret_from_fork+0x31/0x40 [36010.864617] ? kthread_complete_and_exit+0x20/0x20 [36010.869400] ret_from_fork_asm+0x11/0x20 [36010.873333] [36037.797452] rcu: INFO: rcu_preempt self-detected stall on CPU [36037.803211] rcu: 7-....: (23401125 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/23399778 fqs=11794937 [36037.813897] rcu: (t=24100762 jiffies g=1309609 q=669617 ncpus=8) [36037.819989] Sending NMI from CPU 7 to CPUs 0: [36037.824342] NMI backtrace for cpu 0 [36037.824344] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36037.824346] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36037.824347] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36037.824353] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36037.824354] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36037.824356] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36037.824357] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36037.824358] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36037.824358] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36037.824359] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36037.824360] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36037.824361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36037.824362] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36037.824363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36037.824364] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36037.824364] Call Trace: [36037.824366] [36037.824367] ? nmi_cpu_backtrace+0xad/0x120 [36037.824371] ? nmi_cpu_backtrace_handler+0x11/0x20 [36037.824373] ? nmi_handle+0xd4/0x260 [36037.824378] ? default_do_nmi+0x49/0xf0 [36037.824380] ? exc_nmi+0x11e/0x150 [36037.824381] ? end_repeat_nmi+0x16/0x67 [36037.824386] ? queued_spin_lock_slowpath+0x16/0x2e0 [36037.824388] ? queued_spin_lock_slowpath+0x16/0x2e0 [36037.824391] ? queued_spin_lock_slowpath+0x16/0x2e0 [36037.824393] [36037.824394] [36037.824395] do_raw_spin_lock+0xb5/0xc0 [36037.824399] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36037.824419] try_get_cap_refs+0x159/0x880 [ceph] [36037.824434] __ceph_get_caps+0x4a1/0x760 [ceph] [36037.824447] ? __file_remove_privs+0x89/0x110 [36037.824449] ? wake_bit_function+0x70/0x70 [36037.824454] ? ceph_write_iter+0x260/0x770 [ceph] [36037.824466] ceph_write_iter+0x260/0x770 [ceph] [36037.824477] ? lock_is_held_type+0xaa/0x120 [36037.824481] vfs_write+0x324/0x4f0 [36037.824486] ksys_pwrite64+0x65/0xa0 [36037.824489] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36037.824491] do_syscall_64+0x38/0x80 [36037.824494] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36037.824496] RIP: 0033:0x7f48f0eeb93b [36037.824498] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36037.824499] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [36037.824500] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36037.824501] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36037.824502] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36037.824503] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36037.824503] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36037.824506] [36037.825340] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36038.128856] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36038.136328] Workqueue: events delayed_work [ceph] [36038.141042] RIP: 0010:lock_acquire+0xf0/0x2b0 [36038.145393] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36038.164131] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36038.169348] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36038.176471] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36038.183597] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36038.190719] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36038.197845] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36038.204969] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36038.213048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36038.218783] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36038.225908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36038.233031] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36038.240155] Call Trace: [36038.242600] [36038.244612] ? rcu_dump_cpu_stacks+0xd1/0x180 [36038.248970] ? rcu_sched_clock_irq+0x633/0x1390 [36038.253494] ? lock_release+0xc5/0x260 [36038.257240] ? lock_is_held_type+0xaa/0x120 [36038.261418] ? lock_is_held_type+0xaa/0x120 [36038.265595] ? tick_sched_do_timer+0xa0/0xa0 [36038.269866] ? rcu_read_lock_sched_held+0x56/0x90 [36038.274566] ? tick_sched_do_timer+0xa0/0xa0 [36038.278837] ? update_process_times+0x5f/0x90 [36038.283189] ? tick_sched_handle.isra.16+0x1e/0x50 [36038.287980] ? tick_sched_timer+0x6f/0x80 [36038.291986] ? __hrtimer_run_queues+0x1d3/0x360 [36038.296511] ? hrtimer_interrupt+0x104/0x240 [36038.300780] ? __local_bh_enable+0x50/0x60 [36038.304875] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36038.310098] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36038.315055] [36038.317154] [36038.319251] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36038.324558] ? lock_acquire+0xf0/0x2b0 [36038.328309] ? igrab+0x43/0x50 [36038.331359] ? lock_release+0xc5/0x260 [36038.335104] _raw_spin_lock+0x30/0x40 [36038.338769] ? igrab+0x1d/0x50 [36038.341818] igrab+0x1d/0x50 [36038.344697] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36038.349846] delayed_work+0x1d8/0x280 [ceph] [36038.354128] ? process_scheduled_works+0x22a/0x510 [36038.358917] process_scheduled_works+0x2a4/0x510 [36038.363533] worker_thread+0x188/0x340 [36038.367284] ? rescuer_thread+0x310/0x310 [36038.371294] kthread+0xf5/0x130 [36038.374442] ? kthread_complete_and_exit+0x20/0x20 [36038.379226] ret_from_fork+0x31/0x40 [36038.382804] ? kthread_complete_and_exit+0x20/0x20 [36038.387587] ret_from_fork_asm+0x11/0x20 [36038.391520] [36062.088478] watchdog: BUG: soft lockup - CPU#0 stuck for 22471s! [dbench:78683] [36062.095779] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36062.100478] watchdog: BUG: soft lockup - CPU#7 stuck for 31468s! [kworker/7:4:78808] [36062.151227] irq event stamp: 62967462 [36062.158959] Modules linked in: ceph libceph [36062.162615] hardirqs last enabled at (62967461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36062.166794] dns_resolver fscache [36062.176951] hardirqs last disabled at (62967462): [] sysvec_apic_timer_interrupt+0xe/0x90 [36062.180261] netfs veth nft_chain_nat [36062.189985] softirqs last enabled at (62967366): [] __do_softirq+0x2be/0x371 [36062.189987] xt_MASQUERADE [36062.193643] nf_nat nf_conntrack [36062.202327] softirqs last disabled at (62967361): [] irq_exit_rcu+0xb0/0xd0 [36062.205032] nf_defrag_ipv6 nf_defrag_ipv4 [36062.208255] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36062.216765] nft_compat [36062.220873] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36062.229818] nf_tables nfnetlink [36062.232261] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36062.239734] bridge stp [36062.242956] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36062.248434] llc xfs [36062.250879] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36062.269615] libcrc32c sunrpc [36062.271800] [36062.271801] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36062.277017] intel_rapl_msr intel_rapl_common [36062.279981] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36062.281471] sb_edac [36062.288597] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36062.292948] x86_pkg_temp_thermal coretemp [36062.300071] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36062.302254] kvm_intel kvm [36062.309379] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36062.313471] irqbypass i2c_i801 [36062.320594] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36062.323297] iTCO_wdt [36062.330424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36062.333560] iTCO_vendor_support i2c_smbus [36062.341639] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36062.343908] crct10dif_pclmul joydev [36062.349645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36062.353737] crc32_pclmul ipmi_ssif [36062.360877] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36062.364450] ghash_clmulni_intel mei_me [36062.371572] Call Trace: [36062.371573] [36062.375057] mei lpc_ich [36062.382180] ? watchdog_timer_fn+0x1b4/0x220 [36062.386010] sha512_ssse3 acpi_ipmi [36062.388458] ? __hrtimer_run_queues+0x1d3/0x360 [36062.390468] mfd_core wmi [36062.393001] ? hrtimer_interrupt+0x104/0x240 [36062.397261] ipmi_si ipmi_devintf [36062.400744] ? __local_bh_enable+0x50/0x60 [36062.405270] ipmi_msghandler ioatdma [36062.407889] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36062.412166] acpi_pad acpi_power_meter [36062.415480] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36062.419569] ixgbe igb [36062.423140] [36062.428357] mdio crc32c_intel [36062.432102] [36062.432103] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36062.437060] nvme ptp [36062.439420] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36062.441514] nvme_core [36062.444568] do_raw_spin_lock+0xb5/0xc0 [36062.446662] i2c_algo_bit pps_core [36062.451968] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36062.454238] dca fuse [36062.459109] try_get_cap_refs+0x159/0x880 [ceph] [36062.461466] [36062.461467] irq event stamp: 209202936 [36062.465300] __ceph_get_caps+0x4a1/0x760 [ceph] [36062.468694] hardirqs last enabled at (209202935): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36062.473480] ? __file_remove_privs+0x89/0x110 [36062.475747] hardirqs last disabled at (209202936): [] sysvec_apic_timer_interrupt+0xe/0x90 [36062.480359] ? wake_bit_function+0x70/0x70 [36062.481849] softirqs last enabled at (209202934): [] __do_softirq+0x2be/0x371 [36062.485597] ? ceph_write_iter+0x260/0x770 [ceph] [36062.490119] softirqs last disabled at (209202929): [] irq_exit_rcu+0xb0/0xd0 [36062.500363] ceph_write_iter+0x260/0x770 [ceph] [36062.504711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36062.514527] ? lock_is_held_type+0xaa/0x120 [36062.518615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36062.527388] vfs_write+0x324/0x4f0 [36062.532082] Workqueue: events delayed_work [ceph] [36062.540683] ksys_pwrite64+0x65/0xa0 [36062.545205] [36062.545206] RIP: 0010:lock_acquire+0xf0/0x2b0 [36062.554581] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36062.558760] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36062.566232] do_syscall_64+0x38/0x80 [36062.569628] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36062.574325] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36062.577896] [36062.577897] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36062.579385] RIP: 0033:0x7f48f0eeb93b [36062.583736] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36062.588880] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36062.607612] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36062.611183] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36062.616402] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36062.621447] ORIG_RAX: 0000000000000012 [36062.622937] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36062.630061] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36062.633631] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36062.640755] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36062.659495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36062.666618] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36062.671835] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36062.678958] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36062.682791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36062.689914] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36062.697039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36062.705122] [36062.712241] Call Trace: [36062.712242] [36062.767362] ? watchdog_timer_fn+0x1b4/0x220 [36062.771625] ? __hrtimer_run_queues+0x1d3/0x360 [36062.776167] ? hrtimer_interrupt+0x104/0x240 [36062.780431] ? __local_bh_enable+0x50/0x60 [36062.784524] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36062.789748] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36062.794705] [36062.796802] [36062.798900] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36062.804209] ? lock_acquire+0xf0/0x2b0 [36062.807959] ? igrab+0x43/0x50 [36062.811009] ? lock_release+0xc5/0x260 [36062.814753] _raw_spin_lock+0x30/0x40 [36062.818419] ? igrab+0x1d/0x50 [36062.821469] igrab+0x1d/0x50 [36062.824346] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36062.829496] delayed_work+0x1d8/0x280 [ceph] [36062.833777] ? process_scheduled_works+0x22a/0x510 [36062.838568] process_scheduled_works+0x2a4/0x510 [36062.843182] worker_thread+0x188/0x340 [36062.846932] ? rescuer_thread+0x310/0x310 [36062.850935] kthread+0xf5/0x130 [36062.854073] ? kthread_complete_and_exit+0x20/0x20 [36062.858860] ret_from_fork+0x31/0x40 [36062.862454] ? kthread_complete_and_exit+0x20/0x20 [36062.867239] ret_from_fork_asm+0x11/0x20 [36062.871177] [36090.088933] watchdog: BUG: soft lockup - CPU#0 stuck for 22497s! [dbench:78683] [36090.096240] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36090.100933] watchdog: BUG: soft lockup - CPU#7 stuck for 31494s! [kworker/7:4:78808] [36090.151681] irq event stamp: 63025598 [36090.151682] hardirqs last enabled at (63025597): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36090.159412] Modules linked in: [36090.163068] hardirqs last disabled at (63025598): [] sysvec_apic_timer_interrupt+0xe/0x90 [36090.173226] ceph [36090.176277] softirqs last enabled at (63025542): [] __do_softirq+0x2be/0x371 [36090.186001] libceph [36090.187924] softirqs last disabled at (63025537): [] irq_exit_rcu+0xb0/0xd0 [36090.196608] dns_resolver fscache [36090.198793] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36090.207322] netfs [36090.210632] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36090.219576] veth nft_chain_nat [36090.221604] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [36090.229075] xt_MASQUERADE nf_nat [36090.232213] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [36090.237691] nf_conntrack nf_defrag_ipv6 [36090.241000] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36090.259738] nf_defrag_ipv4 [36090.263656] [36090.263657] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36090.268872] nft_compat nf_tables [36090.271662] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36090.273153] nfnetlink bridge [36090.280277] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36090.283589] stp llc [36090.290731] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36090.293693] xfs libcrc32c [36090.300818] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36090.303004] sunrpc intel_rapl_msr [36090.310126] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36090.312832] intel_rapl_common [36090.319956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36090.323353] sb_edac x86_pkg_temp_thermal [36090.331429] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36090.334481] coretemp kvm_intel [36090.340218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36090.344221] kvm irqbypass [36090.351347] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36090.354483] i2c_i801 [36090.361611] Call Trace: [36090.361612] [36090.364330] iTCO_wdt iTCO_vendor_support [36090.371455] ? watchdog_timer_fn+0x1b4/0x220 [36090.373725] i2c_smbus crct10dif_pclmul [36090.376169] ? __hrtimer_run_queues+0x1d3/0x360 [36090.378178] joydev crc32_pclmul [36090.382185] ? hrtimer_interrupt+0x104/0x240 [36090.386446] ipmi_ssif ghash_clmulni_intel [36090.390277] ? __local_bh_enable+0x50/0x60 [36090.394801] mei_me mei [36090.398029] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36090.402290] lpc_ich sha512_ssse3 [36090.406380] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36090.410470] acpi_ipmi mfd_core [36090.412914] [36090.418133] wmi ipmi_si [36090.421443] [36090.426401] ipmi_devintf ipmi_msghandler [36090.429538] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36090.431635] ioatdma acpi_pad [36090.434171] ? queued_spin_lock_slowpath+0x2b/0x2e0 [36090.436263] acpi_power_meter ixgbe [36090.440270] do_raw_spin_lock+0xb5/0xc0 [36090.445572] igb mdio [36090.448536] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36090.453408] crc32c_intel [36090.456892] try_get_cap_refs+0x159/0x880 [ceph] [36090.460723] nvme ptp [36090.462996] __ceph_get_caps+0x4a1/0x760 [ceph] [36090.467775] nvme_core i2c_algo_bit [36090.470399] ? __file_remove_privs+0x89/0x110 [36090.475006] pps_core dca [36090.477275] ? wake_bit_function+0x70/0x70 [36090.481799] fuse [36090.485285] ? ceph_write_iter+0x260/0x770 [ceph] [36090.489634] irq event stamp: 209423678 [36090.492252] ceph_write_iter+0x260/0x770 [ceph] [36090.496343] hardirqs last enabled at (209423677): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36090.498271] ? lock_is_held_type+0xaa/0x120 [36090.502964] hardirqs last disabled at (209423678): [] sysvec_apic_timer_interrupt+0xe/0x90 [36090.506711] vfs_write+0x324/0x4f0 [36090.511232] softirqs last enabled at (209423676): [] __do_softirq+0x2be/0x371 [36090.521480] ksys_pwrite64+0x65/0xa0 [36090.525653] softirqs last disabled at (209423671): [] irq_exit_rcu+0xb0/0xd0 [36090.535464] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36090.538862] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36090.547635] do_syscall_64+0x38/0x80 [36090.551203] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36090.559802] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36090.564931] Workqueue: events delayed_work [ceph] [36090.574309] RIP: 0033:0x7f48f0eeb93b [36090.577898] [36090.577899] RIP: 0010:lock_acquire+0xf0/0x2b0 [36090.585367] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36090.590411] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36090.595108] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36090.598680] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36090.600171] ORIG_RAX: 0000000000000012 [36090.604523] [36090.604523] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36090.623260] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36090.641998] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36090.647213] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36090.652432] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36090.656262] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36090.657753] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36090.664876] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36090.672001] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36090.679125] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36090.686249] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36090.693379] [36090.700498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36090.744995] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36090.752118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36090.759243] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36090.766366] Call Trace: [36090.768812] [36090.770821] ? watchdog_timer_fn+0x1b4/0x220 [36090.775086] ? __hrtimer_run_queues+0x1d3/0x360 [36090.779627] ? hrtimer_interrupt+0x104/0x240 [36090.783889] ? __local_bh_enable+0x50/0x60 [36090.787983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36090.793200] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36090.798156] [36090.800252] [36090.802352] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36090.807660] ? lock_acquire+0xf0/0x2b0 [36090.811410] ? igrab+0x43/0x50 [36090.814460] ? lock_release+0xc5/0x260 [36090.818204] _raw_spin_lock+0x30/0x40 [36090.821860] ? igrab+0x1d/0x50 [36090.824910] igrab+0x1d/0x50 [36090.827790] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36090.832940] delayed_work+0x1d8/0x280 [ceph] [36090.837221] ? process_scheduled_works+0x22a/0x510 [36090.842011] process_scheduled_works+0x2a4/0x510 [36090.846643] worker_thread+0x188/0x340 [36090.850394] ? rescuer_thread+0x310/0x310 [36090.854404] kthread+0xf5/0x130 [36090.857542] ? kthread_complete_and_exit+0x20/0x20 [36090.862341] ret_from_fork+0x31/0x40 [36090.865914] ? kthread_complete_and_exit+0x20/0x20 [36090.870699] ret_from_fork_asm+0x11/0x20 [36090.874624] [36118.089394] watchdog: BUG: soft lockup - CPU#0 stuck for 22523s! [dbench:78683] [36118.096701] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36118.101394] watchdog: BUG: soft lockup - CPU#7 stuck for 31520s! [kworker/7:4:78808] [36118.152151] irq event stamp: 63083706 [36118.152152] hardirqs last enabled at (63083705): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36118.159881] Modules linked in: [36118.163537] hardirqs last disabled at (63083706): [] sysvec_apic_timer_interrupt+0xe/0x90 [36118.173695] ceph [36118.176747] softirqs last enabled at (63083630): [] __do_softirq+0x2be/0x371 [36118.186470] libceph dns_resolver [36118.188393] softirqs last disabled at (63083625): [] irq_exit_rcu+0xb0/0xd0 [36118.197079] fscache [36118.200391] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36118.208900] netfs veth [36118.211095] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36118.220037] nft_chain_nat [36118.222481] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36118.229950] xt_MASQUERADE nf_nat [36118.232657] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36118.238133] nf_conntrack nf_defrag_ipv6 [36118.241445] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36118.260183] nf_defrag_ipv4 nft_compat [36118.264101] [36118.269317] nf_tables nfnetlink [36118.273078] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36118.274569] bridge stp [36118.277792] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36118.284917] llc xfs [36118.287360] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36118.294485] libcrc32c sunrpc [36118.296669] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36118.303792] intel_rapl_msr [36118.306757] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36118.313881] intel_rapl_common sb_edac [36118.316673] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36118.323796] x86_pkg_temp_thermal coretemp [36118.327539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36118.335618] kvm_intel kvm [36118.339709] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36118.345446] irqbypass i2c_i801 [36118.348151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36118.355274] iTCO_wdt iTCO_vendor_support [36118.358412] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36118.365536] i2c_smbus crct10dif_pclmul [36118.369539] Call Trace: [36118.369541] [36118.376664] joydev [36118.380497] ? watchdog_timer_fn+0x1b4/0x220 [36118.382938] crc32_pclmul ipmi_ssif [36118.384952] ? __hrtimer_run_queues+0x1d3/0x360 [36118.387047] ghash_clmulni_intel mei_me [36118.391314] ? hrtimer_interrupt+0x104/0x240 [36118.394794] mei lpc_ich [36118.399318] ? __local_bh_enable+0x50/0x60 [36118.403149] sha512_ssse3 [36118.407417] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36118.409944] acpi_ipmi mfd_core [36118.414037] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36118.416652] wmi ipmi_si [36118.421870] [36118.425007] ipmi_devintf ipmi_msghandler [36118.429964] [36118.432495] ioatdma acpi_pad [36118.434593] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36118.438597] acpi_power_meter [36118.440698] ? queued_spin_lock_slowpath+0x18/0x2e0 [36118.443657] ixgbe igb mdio [36118.448966] do_raw_spin_lock+0xb5/0xc0 [36118.451927] crc32c_intel [36118.456799] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36118.459588] nvme ptp [36118.463423] try_get_cap_refs+0x159/0x880 [ceph] [36118.466036] nvme_core i2c_algo_bit [36118.470827] __ceph_get_caps+0x4a1/0x760 [ceph] [36118.473093] pps_core [36118.477708] ? __file_remove_privs+0x89/0x110 [36118.481185] dca fuse [36118.485709] ? wake_bit_function+0x70/0x70 [36118.487980] [36118.492334] ? ceph_write_iter+0x260/0x770 [ceph] [36118.494601] irq event stamp: 209644408 [36118.498693] ceph_write_iter+0x260/0x770 [ceph] [36118.500182] hardirqs last enabled at (209644407): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36118.504885] ? lock_is_held_type+0xaa/0x120 [36118.508626] hardirqs last disabled at (209644408): [] sysvec_apic_timer_interrupt+0xe/0x90 [36118.513151] vfs_write+0x324/0x4f0 [36118.523392] softirqs last enabled at (209644406): [] __do_softirq+0x2be/0x371 [36118.527576] ksys_pwrite64+0x65/0xa0 [36118.537382] softirqs last disabled at (209644401): [] irq_exit_rcu+0xb0/0xd0 [36118.540794] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36118.549568] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36118.553139] do_syscall_64+0x38/0x80 [36118.561736] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36118.566867] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36118.576243] Workqueue: events delayed_work [ceph] [36118.579815] RIP: 0033:0x7f48f0eeb93b [36118.587285] [36118.587286] RIP: 0010:lock_acquire+0xf0/0x2b0 [36118.592330] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36118.597027] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36118.600598] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36118.602088] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36118.606438] ORIG_RAX: 0000000000000012 [36118.625177] [36118.625178] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36118.643913] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36118.649131] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36118.654349] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36118.658180] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36118.659671] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36118.666796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36118.673919] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36118.681043] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36118.688166] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36118.695292] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36118.702421] [36118.709540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36118.746921] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36118.754044] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36118.761168] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36118.768292] Call Trace: [36118.770737] [36118.772747] ? watchdog_timer_fn+0x1b4/0x220 [36118.777013] ? __hrtimer_run_queues+0x1d3/0x360 [36118.781538] ? hrtimer_interrupt+0x104/0x240 [36118.785808] ? __local_bh_enable+0x50/0x60 [36118.789903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36118.795124] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36118.800083] [36118.802179] [36118.804277] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36118.809584] ? lock_acquire+0xf0/0x2b0 [36118.813336] ? igrab+0x43/0x50 [36118.816395] ? lock_release+0xc5/0x260 [36118.820139] _raw_spin_lock+0x30/0x40 [36118.823800] ? igrab+0x1d/0x50 [36118.826855] igrab+0x1d/0x50 [36118.829733] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36118.834882] delayed_work+0x1d8/0x280 [ceph] [36118.839164] ? process_scheduled_works+0x22a/0x510 [36118.843954] process_scheduled_works+0x2a4/0x510 [36118.848569] worker_thread+0x188/0x340 [36118.852318] ? rescuer_thread+0x310/0x310 [36118.856320] kthread+0xf5/0x130 [36118.859460] ? kthread_complete_and_exit+0x20/0x20 [36118.864244] ret_from_fork+0x31/0x40 [36118.867823] ? kthread_complete_and_exit+0x20/0x20 [36118.872607] ret_from_fork_asm+0x11/0x20 [36118.876538] [36146.089855] watchdog: BUG: soft lockup - CPU#0 stuck for 22549s! [dbench:78683] [36146.097160] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36146.101856] watchdog: BUG: soft lockup - CPU#7 stuck for 31546s! [kworker/7:4:78808] [36146.152602] irq event stamp: 63141856 [36146.160333] Modules linked in: ceph [36146.163992] hardirqs last enabled at (63141855): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36146.163994] libceph dns_resolver fscache netfs [36146.177631] hardirqs last disabled at (63141856): [] sysvec_apic_timer_interrupt+0xe/0x90 [36146.182156] veth nft_chain_nat xt_MASQUERADE [36146.191881] softirqs last enabled at (63141800): [] __do_softirq+0x2be/0x371 [36146.196249] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [36146.204932] softirqs last disabled at (63141795): [] irq_exit_rcu+0xb0/0xd0 [36146.210756] nft_compat nf_tables nfnetlink [36146.219267] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36146.223446] bridge stp [36146.232389] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36146.234834] llc xfs libcrc32c [36146.242303] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36146.245356] sunrpc intel_rapl_msr intel_rapl_common [36146.250831] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36146.255789] sb_edac x86_pkg_temp_thermal [36146.274532] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36146.278540] coretemp kvm_intel [36146.283759] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36146.286895] kvm irqbypass [36146.294019] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36146.296722] i2c_i801 iTCO_wdt [36146.303847] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36146.303848] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36146.306899] iTCO_vendor_support i2c_smbus [36146.314021] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36146.321147] crct10dif_pclmul joydev [36146.325250] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36146.332378] crc32_pclmul [36146.335950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36146.344026] ipmi_ssif ghash_clmulni_intel [36146.346645] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36146.352382] mei_me mei [36146.356473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36146.363596] lpc_ich sha512_ssse3 [36146.366040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36146.373164] acpi_ipmi mfd_core [36146.376474] Call Trace: [36146.376475] [36146.383600] wmi ipmi_si [36146.386736] ? watchdog_timer_fn+0x1b4/0x220 [36146.389180] ipmi_devintf ipmi_msghandler [36146.391194] ? __hrtimer_run_queues+0x1d3/0x360 [36146.393723] ioatdma acpi_pad [36146.397989] ? hrtimer_interrupt+0x104/0x240 [36146.401990] acpi_power_meter ixgbe [36146.406515] ? __local_bh_enable+0x50/0x60 [36146.409479] igb mdio [36146.413746] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36146.417226] crc32c_intel [36146.421335] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36146.423605] nvme ptp [36146.428822] [36146.431441] nvme_core i2c_algo_bit [36146.436397] [36146.436399] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36146.438667] pps_core dca [36146.440771] ? queued_spin_lock_slowpath+0x18/0x2e0 [36146.444256] fuse [36146.444257] irq event stamp: 209865196 [36146.446359] do_raw_spin_lock+0xb5/0xc0 [36146.451659] hardirqs last enabled at (209865195): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36146.454280] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36146.459148] hardirqs last disabled at (209865196): [] sysvec_apic_timer_interrupt+0xe/0x90 [36146.461075] try_get_cap_refs+0x159/0x880 [ceph] [36146.464817] softirqs last enabled at (209865194): [] __do_softirq+0x2be/0x371 [36146.468651] __ceph_get_caps+0x4a1/0x760 [ceph] [36146.478890] softirqs last disabled at (209865189): [] irq_exit_rcu+0xb0/0xd0 [36146.483680] ? __file_remove_privs+0x89/0x110 [36146.493486] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36146.498097] ? wake_bit_function+0x70/0x70 [36146.506868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36146.511394] ? ceph_write_iter+0x260/0x770 [ceph] [36146.519989] Workqueue: events delayed_work [ceph] [36146.524340] ceph_write_iter+0x260/0x770 [ceph] [36146.533718] [36146.533719] RIP: 0010:lock_acquire+0xf0/0x2b0 [36146.537812] ? lock_is_held_type+0xaa/0x120 [36146.545279] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36146.549979] vfs_write+0x324/0x4f0 [36146.554674] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36146.559202] ksys_pwrite64+0x65/0xa0 [36146.560688] [36146.560689] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36146.565038] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36146.569216] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36146.587955] do_syscall_64+0x38/0x80 [36146.591352] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36146.596569] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36146.600141] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36146.601631] RIP: 0033:0x7f48f0eeb93b [36146.608755] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36146.613886] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36146.621009] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36146.624580] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36146.631705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36146.636749] ORIG_RAX: 0000000000000012 [36146.643873] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36146.647445] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36146.654568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36146.673304] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36146.681384] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36146.686601] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36146.692338] Call Trace: [36146.692339] [36146.696168] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36146.703293] ? watchdog_timer_fn+0x1b4/0x220 [36146.710416] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36146.717541] ? __hrtimer_run_queues+0x1d3/0x360 [36146.724670] [36146.731793] ? hrtimer_interrupt+0x104/0x240 [36146.772854] ? __local_bh_enable+0x50/0x60 [36146.776947] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36146.782170] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36146.787128] [36146.789225] [36146.791322] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36146.796629] ? lock_acquire+0xf0/0x2b0 [36146.800381] ? igrab+0x43/0x50 [36146.803429] ? lock_release+0xc5/0x260 [36146.807176] _raw_spin_lock+0x30/0x40 [36146.810839] ? igrab+0x1d/0x50 [36146.813891] igrab+0x1d/0x50 [36146.816769] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36146.821919] delayed_work+0x1d8/0x280 [ceph] [36146.826198] ? process_scheduled_works+0x22a/0x510 [36146.830982] process_scheduled_works+0x2a4/0x510 [36146.835596] worker_thread+0x188/0x340 [36146.839346] ? rescuer_thread+0x310/0x310 [36146.843348] kthread+0xf5/0x130 [36146.846487] ? kthread_complete_and_exit+0x20/0x20 [36146.851272] ret_from_fork+0x31/0x40 [36146.854851] ? kthread_complete_and_exit+0x20/0x20 [36146.859635] ret_from_fork_asm+0x11/0x20 [36146.863567] [36174.090298] watchdog: BUG: soft lockup - CPU#0 stuck for 22575s! [dbench:78683] [36174.097605] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36174.102298] watchdog: BUG: soft lockup - CPU#7 stuck for 31572s! [kworker/7:4:78808] [36174.153055] irq event stamp: 63199866 [36174.153057] hardirqs last enabled at (63199865): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36174.160786] Modules linked in: [36174.164444] hardirqs last disabled at (63199866): [] sysvec_apic_timer_interrupt+0xe/0x90 [36174.174600] ceph [36174.177651] softirqs last enabled at (63199850): [] __do_softirq+0x2be/0x371 [36174.187374] libceph [36174.189300] softirqs last disabled at (63199845): [] irq_exit_rcu+0xb0/0xd0 [36174.197983] dns_resolver fscache [36174.200171] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36174.208694] netfs [36174.212008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36174.220950] veth nft_chain_nat [36174.222973] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36174.230442] xt_MASQUERADE [36174.233579] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36174.239055] nf_nat [36174.241761] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36174.260496] nf_conntrack [36174.262597] [36174.267812] nf_defrag_ipv6 [36174.270431] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36174.271921] nf_defrag_ipv4 nft_compat [36174.274711] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36174.281836] nf_tables nfnetlink [36174.285578] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36174.292705] bridge [36174.295928] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36174.303053] stp llc [36174.305149] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36174.312272] xfs libcrc32c [36174.314457] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36174.321582] sunrpc intel_rapl_msr [36174.324285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36174.332362] intel_rapl_common sb_edac [36174.335759] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36174.341498] x86_pkg_temp_thermal coretemp [36174.345241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36174.352366] kvm_intel kvm [36174.356457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36174.363580] irqbypass i2c_i801 [36174.366286] Call Trace: [36174.366287] [36174.373410] iTCO_wdt iTCO_vendor_support [36174.376550] ? watchdog_timer_fn+0x1b4/0x220 [36174.378990] i2c_smbus crct10dif_pclmul [36174.381002] ? __hrtimer_run_queues+0x1d3/0x360 [36174.385006] joydev crc32_pclmul [36174.389272] ? hrtimer_interrupt+0x104/0x240 [36174.393101] ipmi_ssif ghash_clmulni_intel [36174.397623] ? __local_bh_enable+0x50/0x60 [36174.400849] mei_me mei [36174.405115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36174.409202] lpc_ich sha512_ssse3 [36174.413293] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36174.415739] acpi_ipmi mfd_core [36174.420972] [36174.424284] wmi ipmi_si [36174.429242] [36174.432379] ipmi_devintf ipmi_msghandler [36174.434476] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36174.437008] ioatdma acpi_pad [36174.439108] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36174.443107] acpi_power_meter ixgbe [36174.448413] do_raw_spin_lock+0xb5/0xc0 [36174.451376] igb mdio [36174.456250] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36174.459732] crc32c_intel nvme [36174.463564] try_get_cap_refs+0x159/0x880 [ceph] [36174.465833] ptp nvme_core [36174.470623] __ceph_get_caps+0x4a1/0x760 [ceph] [36174.473668] i2c_algo_bit pps_core [36174.478283] ? __file_remove_privs+0x89/0x110 [36174.480982] dca fuse [36174.485505] ? wake_bit_function+0x70/0x70 [36174.488902] [36174.488903] irq event stamp: 210085996 [36174.493258] ? ceph_write_iter+0x260/0x770 [ceph] [36174.495525] hardirqs last enabled at (210085995): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36174.499616] ceph_write_iter+0x260/0x770 [ceph] [36174.501106] hardirqs last disabled at (210085996): [] sysvec_apic_timer_interrupt+0xe/0x90 [36174.504855] ? lock_is_held_type+0xaa/0x120 [36174.509547] softirqs last enabled at (210085994): [] __do_softirq+0x2be/0x371 [36174.519795] vfs_write+0x324/0x4f0 [36174.524315] softirqs last disabled at (210085989): [] irq_exit_rcu+0xb0/0xd0 [36174.534131] ksys_pwrite64+0x65/0xa0 [36174.538305] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36174.547076] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36174.550472] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36174.559072] do_syscall_64+0x38/0x80 [36174.562641] Workqueue: events delayed_work [ceph] [36174.572020] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36174.577150] [36174.577150] RIP: 0010:lock_release+0x16a/0x260 [36174.584620] RIP: 0033:0x7f48f0eeb93b [36174.588190] Code: f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 c4 18 5b 5d 41 5c <41> 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 48 0f a3 05 18 bf 53 01 [36174.592887] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36174.597933] RSP: 0018:ffffc90002117d78 EFLAGS: 00000296 [36174.599422] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36174.603861] [36174.603862] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [36174.607430] ORIG_RAX: 0000000000000012 [36174.626169] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [36174.644905] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36174.650122] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [36174.655341] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36174.656831] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [36174.663955] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36174.667787] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [36174.674911] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36174.682036] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36174.689160] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36174.696283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36174.703412] [36174.710532] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36174.755027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36174.762150] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36174.769276] Call Trace: [36174.771721] [36174.773731] ? watchdog_timer_fn+0x1b4/0x220 [36174.777996] ? __hrtimer_run_queues+0x1d3/0x360 [36174.782521] ? hrtimer_interrupt+0x104/0x240 [36174.786782] ? __local_bh_enable+0x50/0x60 [36174.790876] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36174.796098] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36174.801057] [36174.803154] [36174.805253] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36174.810559] ? lock_release+0x16a/0x260 [36174.814398] _raw_spin_unlock+0x1b/0x40 [36174.818235] igrab+0x43/0x50 [36174.821113] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36174.826263] delayed_work+0x1d8/0x280 [ceph] [36174.830544] ? process_scheduled_works+0x22a/0x510 [36174.835335] process_scheduled_works+0x2a4/0x510 [36174.839950] worker_thread+0x188/0x340 [36174.843708] ? rescuer_thread+0x310/0x310 [36174.847710] kthread+0xf5/0x130 [36174.850848] ? kthread_complete_and_exit+0x20/0x20 [36174.855633] ret_from_fork+0x31/0x40 [36174.859212] ? kthread_complete_and_exit+0x20/0x20 [36174.863995] ret_from_fork_asm+0x11/0x20 [36174.867921] [36202.090735] watchdog: BUG: soft lockup - CPU#0 stuck for 22601s! [dbench:78683] [36202.098041] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36202.102735] watchdog: BUG: soft lockup - CPU#7 stuck for 31598s! [kworker/7:4:78808] [36202.153482] irq event stamp: 63258006 [36202.153483] hardirqs last enabled at (63258005): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36202.161213] Modules linked in: [36202.164870] hardirqs last disabled at (63258006): [] sysvec_apic_timer_interrupt+0xe/0x90 [36202.175027] ceph [36202.178079] softirqs last enabled at (63257950): [] __do_softirq+0x2be/0x371 [36202.187802] libceph dns_resolver [36202.189727] softirqs last disabled at (63257945): [] irq_exit_rcu+0xb0/0xd0 [36202.198417] fscache netfs [36202.201731] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36202.210240] veth nft_chain_nat [36202.212944] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36202.221888] xt_MASQUERADE [36202.225026] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36202.232499] nf_nat nf_conntrack [36202.235201] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36202.240679] nf_defrag_ipv6 nf_defrag_ipv4 [36202.243902] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36202.262641] nft_compat nf_tables [36202.266731] [36202.266732] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36202.271948] nfnetlink [36202.275259] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36202.276750] bridge [36202.283874] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36202.286232] stp llc [36202.293356] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36202.295452] xfs libcrc32c [36202.302576] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36202.304762] sunrpc [36202.311886] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36202.314589] intel_rapl_msr [36202.321715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36202.323810] intel_rapl_common sb_edac [36202.331888] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36202.334680] x86_pkg_temp_thermal coretemp [36202.340417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36202.344160] kvm_intel [36202.351284] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36202.355375] kvm irqbypass [36202.362514] Call Trace: [36202.362515] [36202.364874] i2c_i801 [36202.371998] ? watchdog_timer_fn+0x1b4/0x220 [36202.374704] iTCO_wdt iTCO_vendor_support [36202.377148] ? __hrtimer_run_queues+0x1d3/0x360 [36202.379158] i2c_smbus crct10dif_pclmul [36202.381432] ? hrtimer_interrupt+0x104/0x240 [36202.385692] joydev [36202.389695] ? __local_bh_enable+0x50/0x60 [36202.394220] crc32_pclmul [36202.398055] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36202.402316] ipmi_ssif [36202.404424] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36202.408513] ghash_clmulni_intel [36202.411133] [36202.416346] mei_me [36202.418704] [36202.423662] mei lpc_ich [36202.426886] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36202.428983] sha512_ssse3 acpi_ipmi [36202.431084] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36202.433178] mfd_core [36202.435710] do_raw_spin_lock+0xb5/0xc0 [36202.441013] wmi ipmi_si [36202.444498] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36202.449367] ipmi_devintf ipmi_msghandler [36202.451639] try_get_cap_refs+0x159/0x880 [ceph] [36202.455468] ioatdma acpi_pad [36202.458004] __ceph_get_caps+0x4a1/0x760 [ceph] [36202.462784] acpi_power_meter [36202.466792] ? __file_remove_privs+0x89/0x110 [36202.471398] ixgbe [36202.474381] ? wake_bit_function+0x70/0x70 [36202.478903] igb mdio [36202.481872] ? ceph_write_iter+0x260/0x770 [ceph] [36202.486218] crc32c_intel nvme [36202.488230] ceph_write_iter+0x260/0x770 [ceph] [36202.492320] ptp nvme_core [36202.494595] ? lock_is_held_type+0xaa/0x120 [36202.499289] i2c_algo_bit [36202.502341] vfs_write+0x324/0x4f0 [36202.506863] pps_core [36202.509571] ksys_pwrite64+0x65/0xa0 [36202.513744] dca fuse [36202.516363] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36202.519759] [36202.519759] irq event stamp: 210306700 [36202.522032] do_syscall_64+0x38/0x80 [36202.525600] hardirqs last enabled at (210306699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36202.527874] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36202.533001] hardirqs last disabled at (210306700): [] sysvec_apic_timer_interrupt+0xe/0x90 [36202.534494] RIP: 0033:0x7f48f0eeb93b [36202.538237] softirqs last enabled at (210306698): [] __do_softirq+0x2be/0x371 [36202.541808] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36202.552053] softirqs last disabled at (210306693): [] irq_exit_rcu+0xb0/0xd0 [36202.557097] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36202.566906] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36202.570479] ORIG_RAX: 0000000000000012 [36202.579248] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36202.597985] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36202.606584] Workqueue: events delayed_work [ceph] [36202.611802] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36202.621179] [36202.625009] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36202.632479] RIP: 0010:lock_release+0x14e/0x260 [36202.639605] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36202.644301] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [36202.651426] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36202.652917] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [36202.660046] [36202.664477] [36202.664478] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [36202.713480] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [36202.720605] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36202.727730] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36202.734852] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [36202.741977] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36202.750054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36202.755792] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36202.762916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36202.770039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36202.777163] Call Trace: [36202.779608] [36202.781620] ? watchdog_timer_fn+0x1b4/0x220 [36202.785884] ? __hrtimer_run_queues+0x1d3/0x360 [36202.790426] ? hrtimer_interrupt+0x104/0x240 [36202.794689] ? __local_bh_enable+0x50/0x60 [36202.798784] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36202.804007] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36202.808963] [36202.811060] [36202.813158] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36202.818463] ? igrab+0x43/0x50 [36202.821516] ? lock_release+0x14e/0x260 [36202.825353] ? lock_release+0xc5/0x260 [36202.829098] _raw_spin_unlock+0x1b/0x40 [36202.832927] igrab+0x43/0x50 [36202.835806] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36202.840956] delayed_work+0x1d8/0x280 [ceph] [36202.845236] ? process_scheduled_works+0x22a/0x510 [36202.850027] process_scheduled_works+0x2a4/0x510 [36202.854642] worker_thread+0x188/0x340 [36202.858392] ? rescuer_thread+0x310/0x310 [36202.862402] kthread+0xf5/0x130 [36202.865541] ? kthread_complete_and_exit+0x20/0x20 [36202.870326] ret_from_fork+0x31/0x40 [36202.873904] ? kthread_complete_and_exit+0x20/0x20 [36202.878688] ret_from_fork_asm+0x11/0x20 [36202.882612] [36230.091170] watchdog: BUG: soft lockup - CPU#0 stuck for 22627s! [dbench:78683] [36230.098478] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36230.103170] watchdog: BUG: soft lockup - CPU#7 stuck for 31624s! [kworker/7:4:78808] [36230.153926] irq event stamp: 63316194 [36230.161657] Modules linked in: ceph libceph dns_resolver fscache netfs [36230.165315] hardirqs last enabled at (63316193): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36230.165321] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [36230.182007] hardirqs last disabled at (63316194): [] sysvec_apic_timer_interrupt+0xe/0x90 [36230.190691] nft_compat nf_tables nfnetlink bridge stp llc xfs [36230.200416] softirqs last enabled at (63316168): [] __do_softirq+0x2be/0x371 [36230.206249] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [36230.214931] softirqs last disabled at (63316163): [] irq_exit_rcu+0xb0/0xd0 [36230.214934] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [36230.231783] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36230.237690] iTCO_vendor_support i2c_smbus [36230.246636] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36230.250725] crct10dif_pclmul joydev crc32_pclmul [36230.258197] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36230.262893] ipmi_ssif ghash_clmulni_intel [36230.268371] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36230.268371] mei_me [36230.272461] mei lpc_ich [36230.291201] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36230.293298] sha512_ssse3 acpi_ipmi [36230.295845] [36230.301062] mfd_core [36230.304565] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36230.306056] wmi [36230.308324] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36230.315449] ipmi_si ipmi_devintf [36230.317288] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36230.324413] ipmi_msghandler [36230.327723] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36230.334846] ioatdma [36230.337723] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36230.344848] acpi_pad [36230.347033] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36230.354155] acpi_power_meter [36230.356427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36230.364503] ixgbe igb [36230.367469] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36230.373205] mdio crc32c_intel [36230.375568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36230.382695] nvme ptp [36230.385745] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36230.392870] nvme_core [36230.395142] Call Trace: [36230.395143] [36230.402265] i2c_algo_bit [36230.404626] ? watchdog_timer_fn+0x1b4/0x220 [36230.407066] pps_core [36230.409078] ? __hrtimer_run_queues+0x1d3/0x360 [36230.411695] dca fuse [36230.415962] ? hrtimer_interrupt+0x104/0x240 [36230.418229] [36230.418230] irq event stamp: 210527438 [36230.422752] ? __local_bh_enable+0x50/0x60 [36230.425023] hardirqs last enabled at (210527437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36230.429292] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36230.430779] hardirqs last disabled at (210527438): [] sysvec_apic_timer_interrupt+0xe/0x90 [36230.434524] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36230.438613] softirqs last enabled at (210527436): [] __do_softirq+0x2be/0x371 [36230.448859] [36230.454076] softirqs last disabled at (210527431): [] irq_exit_rcu+0xb0/0xd0 [36230.463886] [36230.468848] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36230.477624] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36230.479722] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36230.488322] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36230.490416] Workqueue: events delayed_work [ceph] [36230.499795] do_raw_spin_lock+0xb5/0xc0 [36230.505097] [36230.505098] RIP: 0010:lock_acquire+0xf0/0x2b0 [36230.512572] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36230.517440] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36230.522139] try_get_cap_refs+0x159/0x880 [ceph] [36230.525967] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36230.527464] __ceph_get_caps+0x4a1/0x760 [ceph] [36230.531809] [36230.531810] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36230.536598] ? __file_remove_privs+0x89/0x110 [36230.555329] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36230.559940] ? wake_bit_function+0x70/0x70 [36230.565159] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36230.569686] ? ceph_write_iter+0x260/0x770 [ceph] [36230.571173] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36230.578297] ceph_write_iter+0x260/0x770 [ceph] [36230.582648] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36230.589775] ? lock_is_held_type+0xaa/0x120 [36230.593863] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36230.600991] vfs_write+0x324/0x4f0 [36230.605685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36230.612814] ksys_pwrite64+0x65/0xa0 [36230.617333] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36230.624458] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36230.628633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36230.636713] do_syscall_64+0x38/0x80 [36230.640108] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36230.645855] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36230.649427] Call Trace: [36230.649429] [36230.656549] RIP: 0033:0x7f48f0eeb93b [36230.661699] ? watchdog_timer_fn+0x1b4/0x220 [36230.668824] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36230.672395] ? __hrtimer_run_queues+0x1d3/0x360 [36230.679517] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36230.684577] ? hrtimer_interrupt+0x104/0x240 [36230.687013] ORIG_RAX: 0000000000000012 [36230.689025] ? __local_bh_enable+0x50/0x60 [36230.692596] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36230.696863] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36230.715596] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36230.720121] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36230.725339] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36230.729603] [36230.733433] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36230.737523] [36230.744648] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36230.749867] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36230.756996] [36230.761950] ? lock_acquire+0xf0/0x2b0 [36230.798748] ? igrab+0x43/0x50 [36230.801799] ? lock_release+0xc5/0x260 [36230.805543] _raw_spin_lock+0x30/0x40 [36230.809200] ? igrab+0x1d/0x50 [36230.812250] igrab+0x1d/0x50 [36230.815128] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36230.820279] delayed_work+0x1d8/0x280 [ceph] [36230.824577] ? process_scheduled_works+0x22a/0x510 [36230.829367] process_scheduled_works+0x2a4/0x510 [36230.833983] worker_thread+0x188/0x340 [36230.837733] ? rescuer_thread+0x310/0x310 [36230.841742] kthread+0xf5/0x130 [36230.844882] ? kthread_complete_and_exit+0x20/0x20 [36230.849667] ret_from_fork+0x31/0x40 [36230.853244] ? kthread_complete_and_exit+0x20/0x20 [36230.858029] ret_from_fork_asm+0x11/0x20 [36230.861962] [36233.398587] rcu: INFO: rcu_preempt self-detected stall on CPU [36233.404352] rcu: 0-....: (23747767 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11890820 [36233.414862] rcu: (t=24296361 jiffies g=1309609 q=673410 ncpus=8) [36233.420948] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36233.429892] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36233.437361] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36233.442840] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36233.461578] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36233.466794] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36233.473918] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36233.481042] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36233.488166] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36233.495292] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36233.502416] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36233.510493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36233.516231] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36233.523354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36233.530479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36233.537603] Call Trace: [36233.540048] [36233.542059] ? rcu_dump_cpu_stacks+0xd1/0x180 [36233.546418] ? rcu_sched_clock_irq+0x633/0x1390 [36233.550942] ? lock_is_held_type+0xaa/0x120 [36233.555120] ? lock_is_held_type+0xaa/0x120 [36233.559298] ? tick_sched_do_timer+0xa0/0xa0 [36233.563570] ? rcu_read_lock_sched_held+0x56/0x90 [36233.568268] ? tick_sched_do_timer+0xa0/0xa0 [36233.572540] ? update_process_times+0x5f/0x90 [36233.576890] ? tick_sched_handle.isra.16+0x1e/0x50 [36233.581674] ? tick_sched_timer+0x6f/0x80 [36233.585677] ? __hrtimer_run_queues+0x1d3/0x360 [36233.590204] ? hrtimer_interrupt+0x104/0x240 [36233.594475] ? __local_bh_enable+0x50/0x60 [36233.598568] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36233.603792] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36233.608749] [36233.610845] [36233.612945] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36233.618252] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36233.623130] do_raw_spin_lock+0xb5/0xc0 [36233.626968] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36233.631770] try_get_cap_refs+0x159/0x880 [ceph] [36233.636402] __ceph_get_caps+0x4a1/0x760 [ceph] [36233.640940] ? __file_remove_privs+0x89/0x110 [36233.645296] ? wake_bit_function+0x70/0x70 [36233.649391] ? ceph_write_iter+0x260/0x770 [ceph] [36233.654104] ceph_write_iter+0x260/0x770 [ceph] [36233.658640] ? lock_is_held_type+0xaa/0x120 [36233.662824] vfs_write+0x324/0x4f0 [36233.666232] ksys_pwrite64+0x65/0xa0 [36233.669807] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36233.674940] do_syscall_64+0x38/0x80 [36233.678518] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36233.683562] RIP: 0033:0x7f48f0eeb93b [36233.687131] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36233.705868] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [36233.713426] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36233.720550] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36233.727675] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36233.734798] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36233.741924] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36233.749053] [36233.751239] Sending NMI from CPU 0 to CPUs 7: [36233.755593] NMI backtrace for cpu 7 [36233.755595] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36233.755597] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36233.755597] Workqueue: events delayed_work [ceph] [36233.755613] RIP: 0010:hlock_class+0x13/0x60 [36233.755617] Code: 82 48 89 c6 e8 2e 65 01 00 eb 9c e8 d7 ed a2 00 0f 1f 80 00 00 00 00 0f b7 47 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 6d c0 e0 01 <73> 13 48 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 c3 8b [36233.755618] RSP: 0018:ffffc90000264e30 EFLAGS: 00000007 [36233.755619] RAX: 00000000000006f5 RBX: ffff88810debaa80 RCX: 0000000000000004 [36233.755620] RDX: 0000000000000123 RSI: 0000000000000000 RDI: ffff88810debb550 [36233.755621] RBP: ffff88810debb578 R08: 00000000000000a0 R09: 0000000000000004 [36233.755621] R10: 0000000000000000 R11: 0000000000000001 R12: c5dbdfc36fd10127 [36233.755622] R13: 0000000000000003 R14: 0000000000000001 R15: 0000000000000002 [36233.755623] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36233.755624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36233.755624] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36233.755625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36233.755626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36233.755627] Call Trace: [36233.755627] [36233.755628] ? nmi_cpu_backtrace+0xad/0x120 [36233.755632] ? nmi_cpu_backtrace_handler+0x11/0x20 [36233.755633] ? nmi_handle+0xd4/0x260 [36233.755637] ? default_do_nmi+0x49/0xf0 [36233.755639] ? exc_nmi+0x11e/0x150 [36233.755640] ? end_repeat_nmi+0x16/0x67 [36233.755644] ? hlock_class+0x13/0x60 [36233.755646] ? hlock_class+0x13/0x60 [36233.755649] ? hlock_class+0x13/0x60 [36233.755651] [36233.755652] [36233.755652] __lock_acquire+0xb15/0x1e20 [36233.755655] lock_acquire+0xcb/0x2b0 [36233.755656] ? hrtimer_interrupt+0x58/0x240 [36233.755659] _raw_spin_lock_irqsave+0x45/0x60 [36233.755661] ? hrtimer_interrupt+0x58/0x240 [36233.755662] hrtimer_interrupt+0x58/0x240 [36233.755664] ? __local_bh_enable+0x50/0x60 [36233.755666] ? __do_softirq+0x2be/0x371 [36233.755669] __sysvec_apic_timer_interrupt+0x61/0x160 [36233.755671] sysvec_apic_timer_interrupt+0x6d/0x90 [36233.755673] [36233.755673] [36233.755674] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36233.755676] RIP: 0010:lock_acquire+0xf0/0x2b0 [36233.755677] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36233.755679] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36233.755680] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36233.755680] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36233.755681] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36233.755682] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36233.755682] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36233.755685] ? igrab+0x43/0x50 [36233.755686] ? lock_release+0xc5/0x260 [36233.755688] _raw_spin_lock+0x30/0x40 [36233.755690] ? igrab+0x1d/0x50 [36233.755691] igrab+0x1d/0x50 [36233.755692] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36233.755706] delayed_work+0x1d8/0x280 [ceph] [36233.755719] ? process_scheduled_works+0x22a/0x510 [36233.755722] process_scheduled_works+0x2a4/0x510 [36233.755726] worker_thread+0x188/0x340 [36233.755728] ? rescuer_thread+0x310/0x310 [36233.755730] kthread+0xf5/0x130 [36233.755732] ? kthread_complete_and_exit+0x20/0x20 [36233.755735] ret_from_fork+0x31/0x40 [36233.755737] ? kthread_complete_and_exit+0x20/0x20 [36233.755739] ret_from_fork_asm+0x11/0x20 [36233.755743] [36258.091602] watchdog: BUG: soft lockup - CPU#0 stuck for 22653s! [dbench:78683] [36258.098904] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36258.103602] watchdog: BUG: soft lockup - CPU#7 stuck for 31650s! [kworker/7:4:78808] [36258.154343] irq event stamp: 63373442 [36258.154345] hardirqs last enabled at (63373441): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36258.162076] Modules linked in: ceph [36258.165733] hardirqs last disabled at (63373442): [] sysvec_apic_timer_interrupt+0xe/0x90 [36258.175891] libceph dns_resolver [36258.179375] softirqs last enabled at (63373428): [] __do_softirq+0x2be/0x371 [36258.189097] fscache netfs [36258.192409] softirqs last disabled at (63373423): [] irq_exit_rcu+0xb0/0xd0 [36258.201094] veth nft_chain_nat [36258.203797] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36258.212308] xt_MASQUERADE nf_nat [36258.215445] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36258.224390] nf_conntrack nf_defrag_ipv6 [36258.227699] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36258.235170] nf_defrag_ipv4 nft_compat [36258.239089] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36258.244566] nf_tables nfnetlink [36258.248311] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36258.267046] bridge stp [36258.270279] [36258.270279] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36258.275499] llc xfs [36258.277942] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36258.279433] libcrc32c sunrpc [36258.286555] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36258.288741] intel_rapl_msr intel_rapl_common [36258.295866] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36258.298828] sb_edac x86_pkg_temp_thermal [36258.305952] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36258.310303] coretemp kvm_intel [36258.317428] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36258.321431] kvm irqbypass [36258.328556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36258.331694] i2c_i801 iTCO_wdt [36258.339770] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36258.342475] iTCO_vendor_support i2c_smbus [36258.348212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36258.351278] crct10dif_pclmul joydev [36258.358403] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36258.362496] crc32_pclmul ipmi_ssif [36258.369621] Call Trace: [36258.373189] ghash_clmulni_intel mei_me [36258.380315] [36258.383798] mei lpc_ich [36258.386244] ? watchdog_timer_fn+0x1b4/0x220 [36258.390073] sha512_ssse3 acpi_ipmi [36258.392085] ? __hrtimer_run_queues+0x1d3/0x360 [36258.394614] mfd_core wmi [36258.398881] ? hrtimer_interrupt+0x104/0x240 [36258.402363] ipmi_si ipmi_devintf [36258.406887] ? __local_bh_enable+0x50/0x60 [36258.409503] ipmi_msghandler ioatdma [36258.413771] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36258.417079] acpi_pad acpi_power_meter [36258.421169] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36258.424740] ixgbe igb [36258.429958] [36258.433701] mdio crc32c_intel [36258.438660] [36258.441016] nvme ptp [36258.443116] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36258.446164] nvme_core i2c_algo_bit pps_core [36258.448267] ? queued_spin_lock_slowpath+0x16/0x2e0 [36258.450532] dca fuse [36258.455838] do_raw_spin_lock+0xb5/0xc0 [36258.460100] [36258.460101] irq event stamp: 210745300 [36258.464972] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36258.467242] hardirqs last enabled at (210745299): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36258.471076] try_get_cap_refs+0x159/0x880 [ceph] [36258.472564] hardirqs last disabled at (210745300): [] sysvec_apic_timer_interrupt+0xe/0x90 [36258.476312] __ceph_get_caps+0x4a1/0x760 [ceph] [36258.481091] softirqs last enabled at (210745298): [] __do_softirq+0x2be/0x371 [36258.491340] ? __file_remove_privs+0x89/0x110 [36258.495947] softirqs last disabled at (210745293): [] irq_exit_rcu+0xb0/0xd0 [36258.505758] ? wake_bit_function+0x70/0x70 [36258.510293] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36258.519065] ? ceph_write_iter+0x260/0x770 [ceph] [36258.523412] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36258.532009] ceph_write_iter+0x260/0x770 [ceph] [36258.536100] Workqueue: events delayed_work [ceph] [36258.545482] ? lock_is_held_type+0xaa/0x120 [36258.550176] [36258.557648] vfs_write+0x324/0x4f0 [36258.562171] RIP: 0010:lock_release+0x78/0x260 [36258.566872] ksys_pwrite64+0x65/0xa0 [36258.571044] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [36258.572536] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36258.575932] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [36258.580285] do_syscall_64+0x38/0x80 [36258.583855] [36258.602592] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36258.607722] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [36258.612940] RIP: 0033:0x7f48f0eeb93b [36258.616512] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [36258.618001] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36258.623047] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36258.630170] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36258.633740] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36258.640865] ORIG_RAX: 0000000000000012 [36258.659602] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36258.666726] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36258.671945] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36258.679069] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36258.682899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36258.690023] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36258.697147] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36258.705225] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36258.712349] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36258.718086] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36258.725210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36258.732341] [36258.739460] Call Trace: [36258.739461] [36258.767471] ? watchdog_timer_fn+0x1b4/0x220 [36258.771735] ? __hrtimer_run_queues+0x1d3/0x360 [36258.776262] ? hrtimer_interrupt+0x104/0x240 [36258.780540] ? __local_bh_enable+0x50/0x60 [36258.784634] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36258.789858] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36258.794814] [36258.796912] [36258.799009] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36258.804314] ? igrab+0x43/0x50 [36258.807367] ? lock_release+0x78/0x260 [36258.811119] _raw_spin_unlock+0x1b/0x40 [36258.814958] igrab+0x43/0x50 [36258.817834] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36258.823003] delayed_work+0x1d8/0x280 [ceph] [36258.827289] ? process_scheduled_works+0x22a/0x510 [36258.832074] process_scheduled_works+0x2a4/0x510 [36258.836688] worker_thread+0x188/0x340 [36258.840439] ? rescuer_thread+0x310/0x310 [36258.844442] kthread+0xf5/0x130 [36258.847579] ? kthread_complete_and_exit+0x20/0x20 [36258.852364] ret_from_fork+0x31/0x40 [36258.855942] ? kthread_complete_and_exit+0x20/0x20 [36258.860725] ret_from_fork_asm+0x11/0x20 [36258.864650] [36286.092037] watchdog: BUG: soft lockup - CPU#0 stuck for 22680s! [dbench:78683] [36286.099341] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36286.104037] watchdog: BUG: soft lockup - CPU#7 stuck for 31677s! [kworker/7:4:78808] [36286.154788] irq event stamp: 63431494 [36286.162520] Modules linked in: ceph libceph dns_resolver fscache netfs [36286.166177] hardirqs last enabled at (63431493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36286.172709] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [36286.182870] hardirqs last disabled at (63431494): [] sysvec_apic_timer_interrupt+0xe/0x90 [36286.188953] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [36286.198677] softirqs last enabled at (63431484): [] __do_softirq+0x2be/0x371 [36286.206409] llc xfs libcrc32c sunrpc intel_rapl_msr [36286.215109] softirqs last disabled at (63431479): [] irq_exit_rcu+0xb0/0xd0 [36286.220066] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [36286.228580] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36286.235788] kvm irqbypass i2c_i801 [36286.244733] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36286.244734] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36286.248216] iTCO_wdt iTCO_vendor_support [36286.255687] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36286.261182] i2c_smbus crct10dif_pclmul [36286.265186] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36286.283925] joydev crc32_pclmul [36286.287757] [36286.292973] ipmi_ssif [36286.296195] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36286.297687] ghash_clmulni_intel [36286.300062] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36286.307187] mei_me mei [36286.310410] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36286.317550] lpc_ich sha512_ssse3 [36286.319996] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36286.327119] acpi_ipmi mfd_core [36286.330433] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36286.337553] wmi ipmi_si [36286.340691] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36286.347834] ipmi_devintf [36286.350364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36286.358442] ipmi_msghandler ioatdma [36286.361059] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36286.366796] acpi_pad acpi_power_meter [36286.370366] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36286.377491] ixgbe igb [36286.381236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36286.388360] mdio [36286.390726] Call Trace: [36286.390728] [36286.397850] crc32c_intel nvme [36286.399774] ? watchdog_timer_fn+0x1b4/0x220 [36286.402219] ptp [36286.404230] ? __hrtimer_run_queues+0x1d3/0x360 [36286.407278] nvme_core i2c_algo_bit [36286.411546] ? hrtimer_interrupt+0x104/0x240 [36286.413380] pps_core [36286.417905] ? __local_bh_enable+0x50/0x60 [36286.421389] dca fuse [36286.425656] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36286.427925] [36286.427925] irq event stamp: 210966176 [36286.432016] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36286.434285] hardirqs last enabled at (210966175): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36286.439501] [36286.440993] hardirqs last disabled at (210966176): [] sysvec_apic_timer_interrupt+0xe/0x90 [36286.444737] [36286.444738] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36286.449694] softirqs last enabled at (210966174): [] __do_softirq+0x2be/0x371 [36286.459961] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36286.462053] softirqs last disabled at (210966169): [] irq_exit_rcu+0xb0/0xd0 [36286.471867] do_raw_spin_lock+0xb5/0xc0 [36286.473963] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36286.479267] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36286.488037] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36286.492910] try_get_cap_refs+0x159/0x880 [ceph] [36286.501507] Workqueue: events delayed_work [ceph] [36286.505341] __ceph_get_caps+0x4a1/0x760 [ceph] [36286.514727] [36286.514728] RIP: 0010:__list_del_entry_valid_or_report+0x2e/0xd0 [36286.519519] ? __file_remove_privs+0x89/0x110 [36286.526986] Code: 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 00 ad de <48> 39 c2 74 56 48 8b 32 48 39 fe 75 62 48 8b 51 08 48 39 f2 75 73 [36286.531598] ? wake_bit_function+0x70/0x70 [36286.536295] RSP: 0018:ffffc90002117db0 EFLAGS: 00000202 [36286.540819] ? ceph_write_iter+0x260/0x770 [ceph] [36286.542308] [36286.542309] RAX: dead000000000122 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [36286.548306] ceph_write_iter+0x260/0x770 [ceph] [36286.552656] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [36286.571399] ? lock_is_held_type+0xaa/0x120 [36286.575484] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [36286.580705] vfs_write+0x324/0x4f0 [36286.585435] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [36286.586928] ksys_pwrite64+0x65/0xa0 [36286.594050] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36286.598574] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36286.605697] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36286.609876] do_syscall_64+0x38/0x80 [36286.616999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36286.620397] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36286.627521] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36286.631093] RIP: 0033:0x7f48f0eeb93b [36286.638215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36286.643348] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36286.651440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36286.655012] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36286.660748] Call Trace: [36286.665793] ORIG_RAX: 0000000000000012 [36286.672920] [36286.676488] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36286.683612] ? watchdog_timer_fn+0x1b4/0x220 [36286.702351] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36286.709475] ? __hrtimer_run_queues+0x1d3/0x360 [36286.714690] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36286.717140] ? hrtimer_interrupt+0x104/0x240 [36286.720965] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36286.722976] ? __local_bh_enable+0x50/0x60 [36286.730101] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36286.734369] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36286.741494] [36286.746015] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36286.788101] [36286.790196] [36286.792295] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36286.797603] ? __list_del_entry_valid_or_report+0x2e/0xd0 [36286.802998] ceph_check_delayed_caps+0x8f/0x270 [ceph] [36286.808150] delayed_work+0x1d8/0x280 [ceph] [36286.812445] ? process_scheduled_works+0x22a/0x510 [36286.817228] process_scheduled_works+0x2a4/0x510 [36286.821844] worker_thread+0x188/0x340 [36286.825595] ? rescuer_thread+0x310/0x310 [36286.829607] kthread+0xf5/0x130 [36286.832753] ? kthread_complete_and_exit+0x20/0x20 [36286.837536] ret_from_fork+0x31/0x40 [36286.841117] ? kthread_complete_and_exit+0x20/0x20 [36286.845909] ret_from_fork_asm+0x11/0x20 [36286.849841] [36314.092487] watchdog: BUG: soft lockup - CPU#0 stuck for 22706s! [dbench:78683] [36314.099791] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36314.104487] watchdog: BUG: soft lockup - CPU#7 stuck for 31703s! [kworker/7:4:78808] [36314.155232] irq event stamp: 63489576 [36314.155233] hardirqs last enabled at (63489575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36314.162963] Modules linked in: [36314.166621] hardirqs last disabled at (63489576): [] sysvec_apic_timer_interrupt+0xe/0x90 [36314.176777] ceph [36314.179829] softirqs last enabled at (63489488): [] __do_softirq+0x2be/0x371 [36314.189552] libceph [36314.191478] softirqs last disabled at (63489483): [] irq_exit_rcu+0xb0/0xd0 [36314.200161] dns_resolver fscache [36314.202345] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36314.210856] netfs veth [36314.214185] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36314.223127] nft_chain_nat [36314.225572] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36314.233042] xt_MASQUERADE nf_nat [36314.235747] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36314.241224] nf_conntrack nf_defrag_ipv6 [36314.244536] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36314.263273] nf_defrag_ipv4 nft_compat [36314.267191] [36314.267192] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36314.272409] nf_tables nfnetlink [36314.276169] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36314.277661] bridge stp [36314.284784] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36314.288008] llc xfs [36314.295133] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36314.295134] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36314.297577] libcrc32c [36314.304700] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36314.306883] sunrpc intel_rapl_msr [36314.314008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36314.321133] intel_rapl_common sb_edac [36314.323489] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36314.331568] x86_pkg_temp_thermal [36314.334966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36314.340702] coretemp kvm_intel [36314.344446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36314.351571] kvm irqbypass [36314.354885] Call Trace: [36314.362004] i2c_i801 [36314.365142] [36314.372284] iTCO_wdt iTCO_vendor_support [36314.374988] ? watchdog_timer_fn+0x1b4/0x220 [36314.377432] i2c_smbus [36314.379704] ? __hrtimer_run_queues+0x1d3/0x360 [36314.381715] crct10dif_pclmul joydev [36314.385720] ? hrtimer_interrupt+0x104/0x240 [36314.389983] crc32_pclmul ipmi_ssif [36314.392339] ? __local_bh_enable+0x50/0x60 [36314.396880] ghash_clmulni_intel mei_me [36314.400455] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36314.404715] mei lpc_ich [36314.408201] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36314.412290] sha512_ssse3 acpi_ipmi [36314.416123] [36314.421340] mfd_core wmi [36314.423884] [36314.428845] ipmi_si ipmi_devintf [36314.432330] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36314.434426] ipmi_msghandler ioatdma [36314.437046] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36314.439139] acpi_pad acpi_power_meter [36314.442453] do_raw_spin_lock+0xb5/0xc0 [36314.447755] ixgbe igb mdio [36314.451328] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36314.456197] crc32c_intel nvme [36314.459945] try_get_cap_refs+0x159/0x880 [ceph] [36314.463772] ptp nvme_core [36314.466568] __ceph_get_caps+0x4a1/0x760 [ceph] [36314.471346] i2c_algo_bit pps_core [36314.474401] ? __file_remove_privs+0x89/0x110 [36314.479008] dca fuse [36314.481713] ? wake_bit_function+0x70/0x70 [36314.486236] [36314.486237] irq event stamp: 211187106 [36314.489637] ? ceph_write_iter+0x260/0x770 [ceph] [36314.493985] hardirqs last enabled at (211187105): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36314.496255] ceph_write_iter+0x260/0x770 [ceph] [36314.500344] hardirqs last disabled at (211187106): [] sysvec_apic_timer_interrupt+0xe/0x90 [36314.501839] ? lock_is_held_type+0xaa/0x120 [36314.505580] softirqs last enabled at (211187104): [] __do_softirq+0x2be/0x371 [36314.510281] vfs_write+0x324/0x4f0 [36314.520521] softirqs last disabled at (211187099): [] irq_exit_rcu+0xb0/0xd0 [36314.525050] ksys_pwrite64+0x65/0xa0 [36314.534857] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36314.539033] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36314.547804] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36314.551205] do_syscall_64+0x38/0x80 [36314.559799] Workqueue: events delayed_work [ceph] [36314.563373] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36314.572748] [36314.572749] RIP: 0010:rcu_is_watching+0xb/0x40 [36314.577889] RIP: 0033:0x7f48f0eeb93b [36314.585357] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [36314.588929] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36314.593627] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [36314.598671] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36314.600171] [36314.600172] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [36314.604607] ORIG_RAX: 0000000000000012 [36314.608178] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [36314.626917] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36314.645652] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36314.650887] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36314.656106] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36314.657596] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36314.664721] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36314.668552] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36314.675675] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36314.682799] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36314.689923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36314.697053] [36314.704171] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36314.755808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36314.762933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36314.770058] Call Trace: [36314.772503] [36314.774512] ? watchdog_timer_fn+0x1b4/0x220 [36314.778777] ? __hrtimer_run_queues+0x1d3/0x360 [36314.783304] ? hrtimer_interrupt+0x104/0x240 [36314.787573] ? __local_bh_enable+0x50/0x60 [36314.791666] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36314.796892] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36314.801849] [36314.803944] [36314.806043] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36314.811355] ? igrab+0x43/0x50 [36314.814409] ? igrab+0x43/0x50 [36314.817466] ? rcu_is_watching+0xb/0x40 [36314.821297] lock_release+0x1d7/0x260 [36314.824956] _raw_spin_unlock+0x1b/0x40 [36314.828794] igrab+0x43/0x50 [36314.831672] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36314.836820] delayed_work+0x1d8/0x280 [ceph] [36314.841102] ? process_scheduled_works+0x22a/0x510 [36314.845892] process_scheduled_works+0x2a4/0x510 [36314.850507] worker_thread+0x188/0x340 [36314.854256] ? rescuer_thread+0x310/0x310 [36314.858259] kthread+0xf5/0x130 [36314.861397] ? kthread_complete_and_exit+0x20/0x20 [36314.866182] ret_from_fork+0x31/0x40 [36314.869751] ? kthread_complete_and_exit+0x20/0x20 [36314.874537] ret_from_fork_asm+0x11/0x20 [36314.878459] [36342.092936] watchdog: BUG: soft lockup - CPU#0 stuck for 22732s! [dbench:78683] [36342.100245] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36342.104936] watchdog: BUG: soft lockup - CPU#7 stuck for 31729s! [kworker/7:4:78808] [36342.155696] irq event stamp: 63547730 [36342.163424] Modules linked in: ceph libceph dns_resolver fscache netfs [36342.167083] hardirqs last enabled at (63547729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36342.173619] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [36342.183775] hardirqs last disabled at (63547730): [] sysvec_apic_timer_interrupt+0xe/0x90 [36342.183778] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [36342.199591] softirqs last enabled at (63547654): [] __do_softirq+0x2be/0x371 [36342.199595] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [36342.216016] softirqs last disabled at (63547649): [] irq_exit_rcu+0xb0/0xd0 [36342.225054] coretemp kvm_intel kvm irqbypass [36342.233566] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36342.237917] i2c_i801 iTCO_wdt iTCO_vendor_support [36342.246862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36342.251645] i2c_smbus crct10dif_pclmul joydev [36342.259115] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36342.259117] crc32_pclmul ipmi_ssif ghash_clmulni_intel [36342.269029] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36342.269030] mei_me [36342.269031] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36342.274248] mei [36342.292986] [36342.295082] lpc_ich sha512_ssse3 [36342.300300] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36342.302139] acpi_ipmi [36342.303627] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36342.306939] mfd_core wmi [36342.314062] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36342.316421] ipmi_si ipmi_devintf [36342.323544] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36342.326162] ipmi_msghandler [36342.333286] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36342.336596] ioatdma acpi_pad [36342.343722] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36342.346614] acpi_power_meter [36342.353739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36342.356703] ixgbe igb [36342.364780] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36342.367745] mdio crc32c_intel [36342.373482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36342.375841] nvme ptp [36342.382963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36342.386014] nvme_core i2c_algo_bit [36342.393139] Call Trace: [36342.393142] [36342.395410] pps_core dca [36342.402535] ? watchdog_timer_fn+0x1b4/0x220 [36342.406018] fuse [36342.408464] ? __hrtimer_run_queues+0x1d3/0x360 [36342.410472] [36342.410473] irq event stamp: 211408170 [36342.413095] ? hrtimer_interrupt+0x104/0x240 [36342.417355] hardirqs last enabled at (211408169): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36342.419279] ? __local_bh_enable+0x50/0x60 [36342.423803] hardirqs last disabled at (211408170): [] sysvec_apic_timer_interrupt+0xe/0x90 [36342.425296] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36342.429037] softirqs last enabled at (211408168): [] __do_softirq+0x2be/0x371 [36342.433303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36342.443545] softirqs last disabled at (211408163): [] irq_exit_rcu+0xb0/0xd0 [36342.447635] [36342.457446] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36342.462663] [36342.471435] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36342.476393] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36342.484990] Workqueue: events delayed_work [ceph] [36342.487092] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36342.496464] [36342.496465] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [36342.498564] do_raw_spin_lock+0xb5/0xc0 [36342.506034] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [36342.511339] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36342.516035] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [36342.520908] try_get_cap_refs+0x159/0x880 [ceph] [36342.522395] [36342.522396] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [36342.527013] __ceph_get_caps+0x4a1/0x760 [ceph] [36342.530839] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [36342.549580] ? __file_remove_privs+0x89/0x110 [36342.554359] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [36342.559578] ? wake_bit_function+0x70/0x70 [36342.564186] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [36342.565682] ? ceph_write_iter+0x260/0x770 [ceph] [36342.572804] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36342.577337] ceph_write_iter+0x260/0x770 [ceph] [36342.584458] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36342.588814] ? lock_is_held_type+0xaa/0x120 [36342.595934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36342.600027] vfs_write+0x324/0x4f0 [36342.607149] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36342.611850] ksys_pwrite64+0x65/0xa0 [36342.618971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36342.623495] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36342.631573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36342.635752] do_syscall_64+0x38/0x80 [36342.641488] Call Trace: [36342.641490] [36342.644886] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36342.652011] ? watchdog_timer_fn+0x1b4/0x220 [36342.655580] RIP: 0033:0x7f48f0eeb93b [36342.662705] ? __hrtimer_run_queues+0x1d3/0x360 [36342.667836] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36342.674962] ? hrtimer_interrupt+0x104/0x240 [36342.678529] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36342.680972] ? __local_bh_enable+0x50/0x60 [36342.682984] ORIG_RAX: 0000000000000012 [36342.688032] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36342.692292] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36342.695862] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36342.700386] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36342.719126] [36342.723389] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36342.728619] [36342.732706] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36342.736536] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36342.741753] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36342.748882] ? do_raw_spin_lock+0x42/0xc0 [36342.753840] [36342.760960] igrab+0x1d/0x50 [36342.800900] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36342.806048] delayed_work+0x1d8/0x280 [ceph] [36342.810344] ? process_scheduled_works+0x22a/0x510 [36342.815128] process_scheduled_works+0x2a4/0x510 [36342.819743] worker_thread+0x188/0x340 [36342.823492] ? rescuer_thread+0x310/0x310 [36342.827496] kthread+0xf5/0x130 [36342.830643] ? kthread_complete_and_exit+0x20/0x20 [36342.835427] ret_from_fork+0x31/0x40 [36342.839006] ? kthread_complete_and_exit+0x20/0x20 [36342.843790] ret_from_fork_asm+0x11/0x20 [36342.847723] [36370.093397] watchdog: BUG: soft lockup - CPU#0 stuck for 22758s! [dbench:78683] [36370.100697] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36370.105397] watchdog: BUG: soft lockup - CPU#7 stuck for 31755s! [kworker/7:4:78808] [36370.156138] irq event stamp: 63605724 [36370.156139] hardirqs last enabled at (63605723): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36370.163870] Modules linked in: [36370.167526] hardirqs last disabled at (63605724): [] sysvec_apic_timer_interrupt+0xe/0x90 [36370.177683] ceph [36370.180735] softirqs last enabled at (63605656): [] __do_softirq+0x2be/0x371 [36370.190459] libceph [36370.192383] softirqs last disabled at (63605651): [] irq_exit_rcu+0xb0/0xd0 [36370.201072] dns_resolver fscache [36370.203260] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36370.211788] netfs veth [36370.215099] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36370.224042] nft_chain_nat [36370.226488] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36370.233957] xt_MASQUERADE nf_nat [36370.236662] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36370.242140] nf_conntrack nf_defrag_ipv6 [36370.245450] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36370.264187] nf_defrag_ipv4 nft_compat [36370.268105] [36370.268106] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36370.273321] nf_tables nfnetlink [36370.277073] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36370.278567] bridge [36370.285691] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36370.288913] stp [36370.296038] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36370.298137] llc xfs [36370.305260] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36370.307098] libcrc32c sunrpc [36370.314220] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36370.316406] intel_rapl_msr [36370.323530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36370.326493] intel_rapl_common [36370.334572] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36370.337363] sb_edac x86_pkg_temp_thermal [36370.343100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36370.346151] coretemp kvm_intel [36370.353274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36370.357279] kvm irqbypass [36370.364401] Call Trace: [36370.367541] i2c_i801 iTCO_wdt [36370.374664] [36370.377367] iTCO_vendor_support i2c_smbus [36370.379813] ? watchdog_timer_fn+0x1b4/0x220 [36370.382862] crct10dif_pclmul joydev [36370.384875] ? __hrtimer_run_queues+0x1d3/0x360 [36370.388965] crc32_pclmul ipmi_ssif [36370.393231] ? hrtimer_interrupt+0x104/0x240 [36370.396798] ghash_clmulni_intel [36370.401322] ? __local_bh_enable+0x50/0x60 [36370.404807] mei_me [36370.409079] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36370.412305] mei lpc_ich [36370.416395] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36370.418492] sha512_ssse3 [36370.423711] [36370.426241] acpi_ipmi mfd_core [36370.431199] [36370.433814] wmi ipmi_si [36370.435913] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36370.439050] ipmi_devintf [36370.441152] ? queued_spin_lock_slowpath+0x16/0x2e0 [36370.443678] ipmi_msghandler [36370.448983] do_raw_spin_lock+0xb5/0xc0 [36370.451599] ioatdma acpi_pad [36370.456470] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36370.459347] acpi_power_meter ixgbe [36370.463181] try_get_cap_refs+0x159/0x880 [ceph] [36370.466142] igb mdio [36370.470932] __ceph_get_caps+0x4a1/0x760 [ceph] [36370.474411] crc32c_intel nvme [36370.479025] ? __file_remove_privs+0x89/0x110 [36370.481291] ptp nvme_core [36370.485817] ? wake_bit_function+0x70/0x70 [36370.488866] i2c_algo_bit pps_core [36370.493221] ? ceph_write_iter+0x260/0x770 [ceph] [36370.495922] dca fuse [36370.500013] ceph_write_iter+0x260/0x770 [ceph] [36370.503409] [36370.508110] ? lock_is_held_type+0xaa/0x120 [36370.510377] irq event stamp: 211629276 [36370.514905] vfs_write+0x324/0x4f0 [36370.516393] hardirqs last enabled at (211629275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36370.520573] ksys_pwrite64+0x65/0xa0 [36370.524314] hardirqs last disabled at (211629276): [] sysvec_apic_timer_interrupt+0xe/0x90 [36370.527710] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36370.537954] softirqs last enabled at (211629274): [] __do_softirq+0x2be/0x371 [36370.541527] do_syscall_64+0x38/0x80 [36370.551338] softirqs last disabled at (211629269): [] irq_exit_rcu+0xb0/0xd0 [36370.556469] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36370.565239] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36370.568809] RIP: 0033:0x7f48f0eeb93b [36370.577407] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36370.582451] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36370.591829] Workqueue: events delayed_work [ceph] [36370.595400] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36370.602870] [36370.602871] RIP: 0010:lock_acquire+0xf0/0x2b0 [36370.621606] ORIG_RAX: 0000000000000012 [36370.626306] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36370.631523] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36370.633013] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36370.637365] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36370.641194] [36370.641195] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36370.659932] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36370.667058] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36370.672273] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36370.679399] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36370.680888] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36370.688013] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36370.695141] [36370.702261] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36370.740066] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36370.748142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36370.753881] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36370.761005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36370.768128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36370.775254] Call Trace: [36370.777699] [36370.779710] ? watchdog_timer_fn+0x1b4/0x220 [36370.783973] ? __hrtimer_run_queues+0x1d3/0x360 [36370.788498] ? hrtimer_interrupt+0x104/0x240 [36370.792760] ? __local_bh_enable+0x50/0x60 [36370.796855] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36370.802081] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36370.807044] [36370.809142] [36370.811238] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36370.816548] ? lock_acquire+0xf0/0x2b0 [36370.820295] ? igrab+0x43/0x50 [36370.823345] ? lock_release+0xc5/0x260 [36370.827092] _raw_spin_lock+0x30/0x40 [36370.830748] ? igrab+0x1d/0x50 [36370.833804] igrab+0x1d/0x50 [36370.836684] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36370.841833] delayed_work+0x1d8/0x280 [ceph] [36370.846115] ? process_scheduled_works+0x22a/0x510 [36370.850897] process_scheduled_works+0x2a4/0x510 [36370.855513] worker_thread+0x188/0x340 [36370.859263] ? rescuer_thread+0x310/0x310 [36370.863264] kthread+0xf5/0x130 [36370.866403] ? kthread_complete_and_exit+0x20/0x20 [36370.871188] ret_from_fork+0x31/0x40 [36370.874765] ? kthread_complete_and_exit+0x20/0x20 [36370.879550] ret_from_fork_asm+0x11/0x20 [36370.883472] [36398.093859] watchdog: BUG: soft lockup - CPU#0 stuck for 22784s! [dbench:78683] [36398.101167] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36398.105859] watchdog: BUG: soft lockup - CPU#7 stuck for 31781s! [kworker/7:4:78808] [36398.156619] irq event stamp: 63663904 [36398.164347] Modules linked in: ceph libceph dns_resolver fscache netfs veth [36398.168006] hardirqs last enabled at (63663903): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36398.174966] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [36398.185123] hardirqs last disabled at (63663904): [] sysvec_apic_timer_interrupt+0xe/0x90 [36398.190772] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [36398.200498] softirqs last enabled at (63663836): [] __do_softirq+0x2be/0x371 [36398.208252] llc xfs libcrc32c sunrpc intel_rapl_msr [36398.216939] softirqs last disabled at (63663831): [] irq_exit_rcu+0xb0/0xd0 [36398.221905] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [36398.230418] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36398.236758] kvm_intel kvm irqbypass [36398.245703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36398.249273] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [36398.256744] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36398.262396] crct10dif_pclmul joydev crc32_pclmul [36398.267873] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36398.272569] ipmi_ssif ghash_clmulni_intel mei_me [36398.291308] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36398.291310] mei lpc_ich [36398.296007] [36398.301223] sha512_ssse3 acpi_ipmi [36398.303753] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36398.305257] mfd_core [36398.308738] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36398.315861] wmi ipmi_si [36398.318131] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36398.325256] ipmi_devintf [36398.327786] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36398.334910] ipmi_msghandler [36398.337545] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36398.344669] ioatdma acpi_pad [36398.347546] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36398.354671] acpi_power_meter [36398.357636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36398.365712] ixgbe igb [36398.368677] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36398.374415] mdio crc32c_intel [36398.376771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36398.383897] nvme ptp [36398.386948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36398.394071] nvme_core [36398.396342] Call Trace: [36398.403466] i2c_algo_bit pps_core [36398.405825] [36398.408266] dca fuse [36398.411667] ? watchdog_timer_fn+0x1b4/0x220 [36398.413674] [36398.415946] ? __hrtimer_run_queues+0x1d3/0x360 [36398.420209] irq event stamp: 211850138 [36398.421705] ? hrtimer_interrupt+0x104/0x240 [36398.426224] hardirqs last enabled at (211850137): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36398.429970] ? __local_bh_enable+0x50/0x60 [36398.434233] hardirqs last disabled at (211850138): [] sysvec_apic_timer_interrupt+0xe/0x90 [36398.444498] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36398.448585] softirqs last enabled at (211850136): [] __do_softirq+0x2be/0x371 [36398.458397] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36398.463613] softirqs last disabled at (211850131): [] irq_exit_rcu+0xb0/0xd0 [36398.472385] [36398.477341] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36398.485940] [36398.485941] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36398.488035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36398.497418] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36398.499513] Workqueue: events delayed_work [ceph] [36398.504819] do_raw_spin_lock+0xb5/0xc0 [36398.512286] [36398.517160] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36398.521854] RIP: 0010:lock_acquire+0xf0/0x2b0 [36398.525689] try_get_cap_refs+0x159/0x880 [ceph] [36398.527177] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36398.531967] __ceph_get_caps+0x4a1/0x760 [ceph] [36398.536311] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36398.540927] ? __file_remove_privs+0x89/0x110 [36398.559659] [36398.559660] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36398.564185] ? wake_bit_function+0x70/0x70 [36398.569400] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36398.573755] ? ceph_write_iter+0x260/0x770 [ceph] [36398.575260] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36398.582383] ceph_write_iter+0x260/0x770 [ceph] [36398.586474] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36398.593601] ? lock_is_held_type+0xaa/0x120 [36398.598296] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36398.605422] vfs_write+0x324/0x4f0 [36398.609943] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36398.617073] ksys_pwrite64+0x65/0xa0 [36398.621260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36398.628387] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36398.631784] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36398.639862] do_syscall_64+0x38/0x80 [36398.643433] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36398.649173] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36398.654300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36398.661427] RIP: 0033:0x7f48f0eeb93b [36398.664996] Call Trace: [36398.664998] [36398.672122] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36398.677166] ? watchdog_timer_fn+0x1b4/0x220 [36398.684287] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36398.687860] ? __hrtimer_run_queues+0x1d3/0x360 [36398.690303] ORIG_RAX: 0000000000000012 [36398.692316] ? hrtimer_interrupt+0x104/0x240 [36398.711051] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36398.715316] ? __local_bh_enable+0x50/0x60 [36398.720532] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36398.725059] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36398.728888] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36398.733152] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36398.740277] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36398.744368] [36398.751491] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36398.756707] [36398.756709] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36398.763838] [36398.768793] ? lock_acquire+0xf0/0x2b0 [36398.798476] ? igrab+0x43/0x50 [36398.801542] ? lock_release+0xc5/0x260 [36398.805288] _raw_spin_lock+0x30/0x40 [36398.808945] ? igrab+0x1d/0x50 [36398.811995] igrab+0x1d/0x50 [36398.814873] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36398.820022] delayed_work+0x1d8/0x280 [ceph] [36398.824304] ? process_scheduled_works+0x22a/0x510 [36398.829094] process_scheduled_works+0x2a4/0x510 [36398.833710] worker_thread+0x188/0x340 [36398.837458] ? rescuer_thread+0x310/0x310 [36398.841461] kthread+0xf5/0x130 [36398.844601] ? kthread_complete_and_exit+0x20/0x20 [36398.849384] ret_from_fork+0x31/0x40 [36398.852964] ? kthread_complete_and_exit+0x20/0x20 [36398.857747] ret_from_fork_asm+0x11/0x20 [36398.861669] [36426.094324] watchdog: BUG: soft lockup - CPU#0 stuck for 22810s! [dbench:78683] [36426.101630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36426.106324] watchdog: BUG: soft lockup - CPU#7 stuck for 31807s! [kworker/7:4:78808] [36426.157071] irq event stamp: 63721892 [36426.157073] hardirqs last enabled at (63721891): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36426.164801] Modules linked in: [36426.168458] hardirqs last disabled at (63721892): [] sysvec_apic_timer_interrupt+0xe/0x90 [36426.178616] ceph [36426.181668] softirqs last enabled at (63721854): [] __do_softirq+0x2be/0x371 [36426.191392] libceph dns_resolver [36426.193315] softirqs last disabled at (63721849): [] irq_exit_rcu+0xb0/0xd0 [36426.202000] fscache netfs [36426.205310] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36426.213819] veth nft_chain_nat [36426.216523] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36426.225469] xt_MASQUERADE nf_nat [36426.228605] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36426.236077] nf_conntrack nf_defrag_ipv6 [36426.239388] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36426.244865] nf_defrag_ipv4 [36426.248782] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36426.267519] nft_compat [36426.270312] [36426.270312] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36426.275529] nf_tables [36426.277973] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36426.279464] nfnetlink bridge [36426.286586] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36426.288943] stp llc [36426.296068] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36426.299033] xfs libcrc32c [36426.306156] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36426.308340] sunrpc intel_rapl_msr [36426.315464] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36426.318170] intel_rapl_common sb_edac [36426.325294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36426.328689] x86_pkg_temp_thermal coretemp [36426.336769] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36426.340513] kvm_intel kvm [36426.346249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36426.350340] irqbypass i2c_i801 [36426.357465] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36426.360168] iTCO_wdt iTCO_vendor_support [36426.367292] Call Trace: [36426.370430] i2c_smbus crct10dif_pclmul [36426.377554] [36426.381557] joydev crc32_pclmul [36426.384004] ? watchdog_timer_fn+0x1b4/0x220 [36426.387833] ipmi_ssif [36426.389846] ? __hrtimer_run_queues+0x1d3/0x360 [36426.393066] ghash_clmulni_intel [36426.397334] ? hrtimer_interrupt+0x104/0x240 [36426.399690] mei_me mei [36426.404212] ? __local_bh_enable+0x50/0x60 [36426.407437] lpc_ich sha512_ssse3 [36426.411721] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36426.414161] acpi_ipmi mfd_core [36426.418255] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36426.421564] wmi ipmi_si [36426.426782] [36426.429918] ipmi_devintf ipmi_msghandler [36426.434877] [36426.437407] ioatdma acpi_pad [36426.439505] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36426.443508] acpi_power_meter [36426.445611] ? queued_spin_lock_slowpath+0x18/0x2e0 [36426.448569] ixgbe igb [36426.453875] do_raw_spin_lock+0xb5/0xc0 [36426.456837] mdio crc32c_intel [36426.461722] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36426.464076] nvme ptp [36426.467907] try_get_cap_refs+0x159/0x880 [ceph] [36426.470956] nvme_core i2c_algo_bit [36426.475747] __ceph_get_caps+0x4a1/0x760 [ceph] [36426.478012] pps_core dca [36426.482626] ? __file_remove_privs+0x89/0x110 [36426.486105] fuse [36426.490631] ? wake_bit_function+0x70/0x70 [36426.493247] irq event stamp: 212071072 [36426.497601] ? ceph_write_iter+0x260/0x770 [ceph] [36426.499521] hardirqs last enabled at (212071071): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36426.503613] ceph_write_iter+0x260/0x770 [ceph] [36426.507357] hardirqs last disabled at (212071072): [] sysvec_apic_timer_interrupt+0xe/0x90 [36426.512057] ? lock_is_held_type+0xaa/0x120 [36426.522299] softirqs last enabled at (212071070): [] __do_softirq+0x2be/0x371 [36426.526826] vfs_write+0x324/0x4f0 [36426.536633] softirqs last disabled at (212071065): [] irq_exit_rcu+0xb0/0xd0 [36426.540816] ksys_pwrite64+0x65/0xa0 [36426.549582] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36426.552980] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36426.561576] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36426.565148] do_syscall_64+0x38/0x80 [36426.574525] Workqueue: events delayed_work [ceph] [36426.579657] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36426.587127] [36426.590698] RIP: 0033:0x7f48f0eeb93b [36426.595411] RIP: 0010:lock_acquire+0xf0/0x2b0 [36426.600457] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36426.601946] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36426.605519] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36426.609867] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36426.628606] ORIG_RAX: 0000000000000012 [36426.647344] [36426.647345] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36426.652561] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36426.657780] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36426.661610] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36426.663100] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36426.670223] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36426.677349] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36426.684473] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36426.691596] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36426.698727] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36426.705853] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36426.712982] [36426.720100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36426.750350] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36426.757474] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36426.764598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36426.771729] Call Trace: [36426.774175] [36426.776186] ? watchdog_timer_fn+0x1b4/0x220 [36426.780450] ? __hrtimer_run_queues+0x1d3/0x360 [36426.784978] ? hrtimer_interrupt+0x104/0x240 [36426.789246] ? __local_bh_enable+0x50/0x60 [36426.793340] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36426.798563] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36426.803520] [36426.805617] [36426.807729] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36426.813040] ? lock_acquire+0xf0/0x2b0 [36426.816792] ? igrab+0x43/0x50 [36426.819841] ? lock_release+0xc5/0x260 [36426.823585] _raw_spin_lock+0x30/0x40 [36426.827243] ? igrab+0x1d/0x50 [36426.830293] igrab+0x1d/0x50 [36426.833170] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36426.838320] delayed_work+0x1d8/0x280 [ceph] [36426.842601] ? process_scheduled_works+0x22a/0x510 [36426.847392] process_scheduled_works+0x2a4/0x510 [36426.852016] worker_thread+0x188/0x340 [36426.855765] ? rescuer_thread+0x310/0x310 [36426.859769] kthread+0xf5/0x130 [36426.862906] ? kthread_complete_and_exit+0x20/0x20 [36426.867692] ret_from_fork+0x31/0x40 [36426.871270] ? kthread_complete_and_exit+0x20/0x20 [36426.876053] ret_from_fork_asm+0x11/0x20 [36426.879976] [36428.761735] rcu: INFO: rcu_preempt self-detected stall on CPU [36428.767495] rcu: 0-....: (23938241 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=11986617 [36428.778004] rcu: (t=24491721 jiffies g=1309609 q=677149 ncpus=8) [36428.784091] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36428.793033] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36428.800505] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [36428.805981] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [36428.824720] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36428.829936] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36428.837062] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36428.844185] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36428.851309] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36428.858433] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36428.865557] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36428.873634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36428.879371] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36428.886498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36428.893622] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36428.900746] Call Trace: [36428.903189] [36428.905199] ? rcu_dump_cpu_stacks+0xd1/0x180 [36428.909552] ? rcu_sched_clock_irq+0x633/0x1390 [36428.914084] ? lock_is_held_type+0xaa/0x120 [36428.918263] ? lock_is_held_type+0xaa/0x120 [36428.922440] ? tick_sched_do_timer+0xa0/0xa0 [36428.926712] ? rcu_read_lock_sched_held+0x56/0x90 [36428.931410] ? tick_sched_do_timer+0xa0/0xa0 [36428.935673] ? update_process_times+0x5f/0x90 [36428.940024] ? tick_sched_handle.isra.16+0x1e/0x50 [36428.944807] ? tick_sched_timer+0x6f/0x80 [36428.948813] ? __hrtimer_run_queues+0x1d3/0x360 [36428.953339] ? hrtimer_interrupt+0x104/0x240 [36428.957609] ? __local_bh_enable+0x50/0x60 [36428.961701] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36428.966926] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36428.971882] [36428.973979] [36428.976079] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36428.981386] ? queued_spin_lock_slowpath+0x2b/0x2e0 [36428.986264] do_raw_spin_lock+0xb5/0xc0 [36428.990101] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36428.994904] try_get_cap_refs+0x159/0x880 [ceph] [36428.999535] __ceph_get_caps+0x4a1/0x760 [ceph] [36429.004077] ? __file_remove_privs+0x89/0x110 [36429.008430] ? wake_bit_function+0x70/0x70 [36429.012524] ? ceph_write_iter+0x260/0x770 [ceph] [36429.017235] ceph_write_iter+0x260/0x770 [ceph] [36429.021772] ? lock_is_held_type+0xaa/0x120 [36429.025956] vfs_write+0x324/0x4f0 [36429.029357] ksys_pwrite64+0x65/0xa0 [36429.032932] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36429.038062] do_syscall_64+0x38/0x80 [36429.041633] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36429.046677] RIP: 0033:0x7f48f0eeb93b [36429.050249] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36429.068987] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [36429.076544] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36429.083668] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36429.090791] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36429.097915] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36429.105039] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36429.112170] [36429.114357] Sending NMI from CPU 0 to CPUs 7: [36429.118709] NMI backtrace for cpu 7 [36429.118711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36429.118713] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36429.118713] Workqueue: events delayed_work [ceph] [36429.118729] RIP: 0010:lock_acquire+0x16/0x2b0 [36429.118731] Code: 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd <41> 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 [36429.118732] RSP: 0018:ffffc90002117d68 EFLAGS: 00000246 [36429.118733] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [36429.118734] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [36429.118735] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [36429.118735] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [36429.118736] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36429.118737] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36429.118738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36429.118738] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36429.118739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36429.118740] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36429.118741] Call Trace: [36429.118742] [36429.118744] ? nmi_cpu_backtrace+0xad/0x120 [36429.118748] ? nmi_cpu_backtrace_handler+0x11/0x20 [36429.118749] ? nmi_handle+0xd4/0x260 [36429.118753] ? default_do_nmi+0x49/0xf0 [36429.118755] ? exc_nmi+0x11e/0x150 [36429.118757] ? end_repeat_nmi+0x16/0x67 [36429.118761] ? lock_acquire+0x16/0x2b0 [36429.118762] ? lock_acquire+0x16/0x2b0 [36429.118764] ? lock_acquire+0x16/0x2b0 [36429.118766] [36429.118766] [36429.118767] _raw_spin_lock+0x30/0x40 [36429.118769] ? igrab+0x1d/0x50 [36429.118770] igrab+0x1d/0x50 [36429.118772] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36429.118786] delayed_work+0x1d8/0x280 [ceph] [36429.118799] ? process_scheduled_works+0x22a/0x510 [36429.118801] process_scheduled_works+0x2a4/0x510 [36429.118805] worker_thread+0x188/0x340 [36429.118808] ? rescuer_thread+0x310/0x310 [36429.118810] kthread+0xf5/0x130 [36429.118812] ? kthread_complete_and_exit+0x20/0x20 [36429.118815] ret_from_fork+0x31/0x40 [36429.118817] ? kthread_complete_and_exit+0x20/0x20 [36429.118819] ret_from_fork_asm+0x11/0x20 [36429.118823] [36454.094793] watchdog: BUG: soft lockup - CPU#0 stuck for 22836s! [dbench:78683] [36454.102100] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36454.106792] watchdog: BUG: soft lockup - CPU#7 stuck for 31833s! [kworker/7:4:78808] [36454.157550] irq event stamp: 63779284 [36454.157551] hardirqs last enabled at (63779283): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36454.165281] Modules linked in: [36454.168937] hardirqs last disabled at (63779284): [] sysvec_apic_timer_interrupt+0xe/0x90 [36454.179094] ceph [36454.182145] softirqs last enabled at (63779240): [] __do_softirq+0x2be/0x371 [36454.191871] libceph [36454.193794] softirqs last disabled at (63779235): [] irq_exit_rcu+0xb0/0xd0 [36454.202478] dns_resolver [36454.204665] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36454.213185] fscache netfs [36454.215806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36454.224752] veth nft_chain_nat [36454.227455] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36454.234943] xt_MASQUERADE [36454.238083] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36454.243558] nf_nat [36454.246264] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36454.264999] nf_conntrack [36454.267098] [36454.267098] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36454.272314] nf_defrag_ipv6 nf_defrag_ipv4 [36454.274932] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36454.276422] nft_compat nf_tables [36454.283547] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36454.287638] nfnetlink [36454.294763] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36454.298074] bridge stp [36454.305196] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36454.307554] llc [36454.314679] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36454.317122] xfs libcrc32c [36454.324246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36454.326085] sunrpc intel_rapl_msr [36454.334163] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36454.336883] intel_rapl_common sb_edac [36454.342620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36454.346018] x86_pkg_temp_thermal [36454.353142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36454.356887] coretemp kvm_intel [36454.364009] Call Trace: [36454.364012] [36454.367321] kvm irqbypass [36454.374447] ? watchdog_timer_fn+0x1b4/0x220 [36454.377583] i2c_i801 iTCO_wdt [36454.380027] ? __hrtimer_run_queues+0x1d3/0x360 [36454.382038] iTCO_vendor_support [36454.384745] ? hrtimer_interrupt+0x104/0x240 [36454.389005] i2c_smbus [36454.392055] ? __local_bh_enable+0x50/0x60 [36454.396580] crct10dif_pclmul joydev [36454.399807] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36454.404067] crc32_pclmul ipmi_ssif [36454.406426] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36454.410516] ghash_clmulni_intel mei_me [36454.414087] [36454.419304] mei lpc_ich [36454.422788] [36454.427747] sha512_ssse3 acpi_ipmi [36454.431577] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36454.433675] mfd_core wmi [36454.436208] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36454.438302] ipmi_si ipmi_devintf [36454.441789] do_raw_spin_lock+0xb5/0xc0 [36454.447092] ipmi_msghandler ioatdma [36454.449710] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36454.454579] acpi_pad acpi_power_meter [36454.457893] try_get_cap_refs+0x159/0x880 [ceph] [36454.461720] ixgbe igb [36454.465295] __ceph_get_caps+0x4a1/0x760 [ceph] [36454.470075] mdio [36454.473825] ? __file_remove_privs+0x89/0x110 [36454.478431] crc32c_intel [36454.480786] ? wake_bit_function+0x70/0x70 [36454.485312] nvme ptp [36454.487238] ? ceph_write_iter+0x260/0x770 [ceph] [36454.491587] nvme_core i2c_algo_bit [36454.494203] ceph_write_iter+0x260/0x770 [ceph] [36454.498295] pps_core [36454.500570] ? lock_is_held_type+0xaa/0x120 [36454.505262] dca fuse [36454.508749] vfs_write+0x324/0x4f0 [36454.513271] [36454.513272] irq event stamp: 212289990 [36454.515545] ksys_pwrite64+0x65/0xa0 [36454.519718] hardirqs last enabled at (212289989): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36454.521988] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36454.525387] hardirqs last disabled at (212289990): [] sysvec_apic_timer_interrupt+0xe/0x90 [36454.526879] do_syscall_64+0x38/0x80 [36454.530622] softirqs last enabled at (212289988): [] __do_softirq+0x2be/0x371 [36454.534195] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36454.544436] softirqs last disabled at (212289983): [] irq_exit_rcu+0xb0/0xd0 [36454.549569] RIP: 0033:0x7f48f0eeb93b [36454.559378] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36454.562949] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36454.571719] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36454.576765] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36454.585360] Workqueue: events delayed_work [ceph] [36454.588932] ORIG_RAX: 0000000000000012 [36454.598309] [36454.598310] RIP: 0010:lock_release+0x3c/0x260 [36454.617047] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36454.624518] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [36454.629735] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36454.634433] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [36454.638264] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36454.639755] [36454.639756] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [36454.644105] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36454.651229] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [36454.669967] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36454.677090] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36454.682313] [36454.689433] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36454.735870] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36454.742994] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36454.751071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36454.756809] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36454.763933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36454.771056] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36454.778199] Call Trace: [36454.780643] [36454.782653] ? watchdog_timer_fn+0x1b4/0x220 [36454.786919] ? __hrtimer_run_queues+0x1d3/0x360 [36454.791444] ? hrtimer_interrupt+0x104/0x240 [36454.795712] ? __local_bh_enable+0x50/0x60 [36454.799807] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36454.805030] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36454.809988] [36454.812085] [36454.814197] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36454.819506] ? igrab+0x43/0x50 [36454.822559] ? igrab+0x43/0x50 [36454.825615] ? lock_release+0x3c/0x260 [36454.829359] ? lock_release+0xc5/0x260 [36454.833105] _raw_spin_unlock+0x1b/0x40 [36454.836945] igrab+0x43/0x50 [36454.839821] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36454.844972] delayed_work+0x1d8/0x280 [ceph] [36454.849251] ? process_scheduled_works+0x22a/0x510 [36454.854042] process_scheduled_works+0x2a4/0x510 [36454.858658] worker_thread+0x188/0x340 [36454.862408] ? rescuer_thread+0x310/0x310 [36454.866418] kthread+0xf5/0x130 [36454.869557] ? kthread_complete_and_exit+0x20/0x20 [36454.874341] ret_from_fork+0x31/0x40 [36454.877920] ? kthread_complete_and_exit+0x20/0x20 [36454.882704] ret_from_fork_asm+0x11/0x20 [36454.886635] [36482.095260] watchdog: BUG: soft lockup - CPU#0 stuck for 22862s! [dbench:78683] [36482.102562] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36482.107260] watchdog: BUG: soft lockup - CPU#7 stuck for 31859s! [kworker/7:4:78808] [36482.158001] irq event stamp: 63837458 [36482.165732] Modules linked in: ceph [36482.169390] hardirqs last enabled at (63837457): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36482.169392] libceph dns_resolver fscache netfs [36482.183033] hardirqs last disabled at (63837458): [] sysvec_apic_timer_interrupt+0xe/0x90 [36482.187556] veth nft_chain_nat xt_MASQUERADE [36482.197281] softirqs last enabled at (63837336): [] __do_softirq+0x2be/0x371 [36482.197283] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [36482.210333] softirqs last disabled at (63837331): [] irq_exit_rcu+0xb0/0xd0 [36482.217976] nfnetlink bridge stp [36482.226487] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36482.229799] llc xfs [36482.238743] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36482.240941] libcrc32c sunrpc intel_rapl_msr [36482.248414] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36482.248416] intel_rapl_common sb_edac x86_pkg_temp_thermal [36482.258156] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36482.258157] coretemp [36482.263719] kvm_intel kvm [36482.282457] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36482.284729] irqbypass i2c_i801 [36482.287432] [36482.292656] iTCO_wdt [36482.295794] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36482.297286] iTCO_vendor_support [36482.299557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36482.306681] i2c_smbus crct10dif_pclmul [36482.309905] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36482.317031] joydev crc32_pclmul [36482.320861] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36482.327985] ipmi_ssif ghash_clmulni_intel [36482.331207] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36482.338333] mei_me [36482.342424] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36482.349547] mei lpc_ich [36482.351657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36482.359732] sha512_ssse3 acpi_ipmi [36482.362262] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36482.368000] mfd_core wmi [36482.371482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36482.378608] ipmi_si [36482.381223] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36482.388349] ipmi_devintf ipmi_msghandler [36482.390534] Call Trace: [36482.390535] [36482.397662] ioatdma [36482.401671] ? watchdog_timer_fn+0x1b4/0x220 [36482.404114] acpi_pad acpi_power_meter [36482.406125] ? __hrtimer_run_queues+0x1d3/0x360 [36482.408307] ixgbe igb [36482.412576] ? hrtimer_interrupt+0x104/0x240 [36482.416317] mdio [36482.420842] ? __local_bh_enable+0x50/0x60 [36482.423197] crc32c_intel nvme [36482.427466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36482.429386] ptp nvme_core [36482.433477] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36482.436529] i2c_algo_bit pps_core [36482.441744] [36482.444449] dca fuse [36482.449408] [36482.452803] [36482.452804] irq event stamp: 212510734 [36482.454901] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36482.457171] hardirqs last enabled at (212510733): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36482.459273] ? queued_spin_lock_slowpath+0x16/0x2e0 [36482.460761] hardirqs last disabled at (212510734): [] sysvec_apic_timer_interrupt+0xe/0x90 [36482.464506] do_raw_spin_lock+0xb5/0xc0 [36482.469808] softirqs last enabled at (212510732): [] __do_softirq+0x2be/0x371 [36482.480053] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36482.484922] softirqs last disabled at (212510727): [] irq_exit_rcu+0xb0/0xd0 [36482.494737] try_get_cap_refs+0x159/0x880 [ceph] [36482.498566] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36482.507340] __ceph_get_caps+0x4a1/0x760 [ceph] [36482.512119] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36482.520722] ? __file_remove_privs+0x89/0x110 [36482.525327] Workqueue: events delayed_work [ceph] [36482.534707] ? wake_bit_function+0x70/0x70 [36482.539230] [36482.539231] RIP: 0010:lock_acquire+0xf0/0x2b0 [36482.546704] ? ceph_write_iter+0x260/0x770 [ceph] [36482.551051] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36482.555749] ceph_write_iter+0x260/0x770 [ceph] [36482.559840] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36482.561333] ? lock_is_held_type+0xaa/0x120 [36482.565681] [36482.565682] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36482.570380] vfs_write+0x324/0x4f0 [36482.589117] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36482.593663] ksys_pwrite64+0x65/0xa0 [36482.598874] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36482.603052] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36482.604543] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36482.611667] do_syscall_64+0x38/0x80 [36482.615064] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36482.622189] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36482.625759] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36482.632884] RIP: 0033:0x7f48f0eeb93b [36482.638013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36482.645137] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36482.648710] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36482.655833] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36482.660876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36482.668955] ORIG_RAX: 0000000000000012 [36482.672526] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36482.678262] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36482.697000] Call Trace: [36482.697001] [36482.704124] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36482.709343] ? watchdog_timer_fn+0x1b4/0x220 [36482.716465] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36482.720299] ? __hrtimer_run_queues+0x1d3/0x360 [36482.727422] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36482.734549] ? hrtimer_interrupt+0x104/0x240 [36482.736988] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36482.739000] ? __local_bh_enable+0x50/0x60 [36482.746130] [36482.750392] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36482.792058] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36482.797016] [36482.799113] [36482.801211] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36482.806518] ? lock_acquire+0xf0/0x2b0 [36482.810268] ? igrab+0x43/0x50 [36482.813318] ? lock_release+0xc5/0x260 [36482.817063] _raw_spin_lock+0x30/0x40 [36482.820719] ? igrab+0x1d/0x50 [36482.823772] igrab+0x1d/0x50 [36482.826667] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36482.831817] delayed_work+0x1d8/0x280 [ceph] [36482.836097] ? process_scheduled_works+0x22a/0x510 [36482.840888] process_scheduled_works+0x2a4/0x510 [36482.845503] worker_thread+0x188/0x340 [36482.849253] ? rescuer_thread+0x310/0x310 [36482.853255] kthread+0xf5/0x130 [36482.856392] ? kthread_complete_and_exit+0x20/0x20 [36482.861178] ret_from_fork+0x31/0x40 [36482.864747] ? kthread_complete_and_exit+0x20/0x20 [36482.869532] ret_from_fork_asm+0x11/0x20 [36482.873455] [36510.095727] watchdog: BUG: soft lockup - CPU#0 stuck for 22888s! [dbench:78683] [36510.103031] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36510.107727] watchdog: BUG: soft lockup - CPU#7 stuck for 31885s! [kworker/7:4:78808] [36510.158471] irq event stamp: 63895526 [36510.158473] hardirqs last enabled at (63895525): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36510.166202] Modules linked in: [36510.169861] hardirqs last disabled at (63895526): [] sysvec_apic_timer_interrupt+0xe/0x90 [36510.180018] ceph [36510.183069] softirqs last enabled at (63895448): [] __do_softirq+0x2be/0x371 [36510.192793] libceph [36510.194717] softirqs last disabled at (63895443): [] irq_exit_rcu+0xb0/0xd0 [36510.203403] dns_resolver [36510.205585] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36510.214095] fscache [36510.216732] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36510.225674] netfs veth [36510.227858] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36510.235330] nft_chain_nat [36510.237774] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36510.243250] xt_MASQUERADE nf_nat [36510.245955] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36510.264692] nf_conntrack [36510.268005] [36510.268006] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36510.273221] nf_defrag_ipv6 [36510.275838] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36510.277328] nf_defrag_ipv4 nft_compat [36510.284453] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36510.287243] nf_tables nfnetlink [36510.294368] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36510.298123] bridge [36510.305245] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36510.308468] stp llc [36510.315592] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36510.317689] xfs libcrc32c [36510.324814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36510.327000] sunrpc [36510.335077] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36510.337779] intel_rapl_msr intel_rapl_common [36510.343518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36510.345616] sb_edac x86_pkg_temp_thermal [36510.352740] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36510.357090] coretemp kvm_intel [36510.364213] Call Trace: [36510.364215] [36510.368217] kvm irqbypass [36510.375344] ? watchdog_timer_fn+0x1b4/0x220 [36510.378480] i2c_i801 iTCO_wdt [36510.380924] ? __hrtimer_run_queues+0x1d3/0x360 [36510.382933] iTCO_vendor_support i2c_smbus [36510.385641] ? hrtimer_interrupt+0x104/0x240 [36510.389903] crct10dif_pclmul joydev [36510.392954] ? __local_bh_enable+0x50/0x60 [36510.397476] crc32_pclmul [36510.401570] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36510.405833] ipmi_ssif ghash_clmulni_intel [36510.409406] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36510.413493] mei_me mei [36510.416126] [36510.421347] lpc_ich sha512_ssse3 [36510.425436] [36510.425437] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36510.430406] acpi_ipmi mfd_core [36510.432851] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36510.434944] wmi ipmi_si [36510.438257] do_raw_spin_lock+0xb5/0xc0 [36510.440351] ipmi_devintf ipmi_msghandler [36510.445657] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36510.448794] ioatdma [36510.453667] try_get_cap_refs+0x159/0x880 [ceph] [36510.456196] acpi_pad acpi_power_meter [36510.460031] __ceph_get_caps+0x4a1/0x760 [ceph] [36510.464031] ixgbe igb [36510.468818] ? __file_remove_privs+0x89/0x110 [36510.470997] mdio crc32c_intel [36510.475610] ? wake_bit_function+0x70/0x70 [36510.479353] nvme ptp [36510.483880] ? ceph_write_iter+0x260/0x770 [ceph] [36510.486233] nvme_core i2c_algo_bit [36510.490585] ceph_write_iter+0x260/0x770 [ceph] [36510.493635] pps_core dca [36510.497729] ? lock_is_held_type+0xaa/0x120 [36510.499999] fuse [36510.504698] vfs_write+0x324/0x4f0 [36510.508178] [36510.512708] ksys_pwrite64+0x65/0xa0 [36510.515321] irq event stamp: 212731542 [36510.519499] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36510.521421] hardirqs last enabled at (212731541): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36510.524821] do_syscall_64+0x38/0x80 [36510.526311] hardirqs last disabled at (212731542): [] sysvec_apic_timer_interrupt+0xe/0x90 [36510.529882] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36510.533625] softirqs last enabled at (212731540): [] __do_softirq+0x2be/0x371 [36510.538756] RIP: 0033:0x7f48f0eeb93b [36510.548998] softirqs last disabled at (212731535): [] irq_exit_rcu+0xb0/0xd0 [36510.552570] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36510.562382] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36510.567424] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36510.576197] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36510.579766] ORIG_RAX: 0000000000000012 [36510.588363] Workqueue: events delayed_work [ceph] [36510.607101] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36510.616478] [36510.621698] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36510.629169] RIP: 0010:lock_acquire+0xf0/0x2b0 [36510.632997] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36510.637695] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36510.644819] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36510.646312] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36510.653433] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36510.657786] [36510.657787] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36510.664915] [36510.683647] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36510.721054] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36510.728176] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36510.735302] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36510.742437] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36510.750512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36510.756250] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36510.763373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36510.770497] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36510.777621] Call Trace: [36510.780066] [36510.782078] ? watchdog_timer_fn+0x1b4/0x220 [36510.786341] ? __hrtimer_run_queues+0x1d3/0x360 [36510.790869] ? hrtimer_interrupt+0x104/0x240 [36510.795137] ? __local_bh_enable+0x50/0x60 [36510.799231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36510.804456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36510.809419] [36510.811518] [36510.813617] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36510.818925] ? lock_acquire+0xf0/0x2b0 [36510.822674] ? igrab+0x43/0x50 [36510.825724] ? lock_release+0xc5/0x260 [36510.829470] _raw_spin_lock+0x30/0x40 [36510.833133] ? igrab+0x1d/0x50 [36510.836184] igrab+0x1d/0x50 [36510.839062] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36510.844212] delayed_work+0x1d8/0x280 [ceph] [36510.848492] ? process_scheduled_works+0x22a/0x510 [36510.853275] process_scheduled_works+0x2a4/0x510 [36510.857890] worker_thread+0x188/0x340 [36510.861640] ? rescuer_thread+0x310/0x310 [36510.865641] kthread+0xf5/0x130 [36510.868779] ? kthread_complete_and_exit+0x20/0x20 [36510.873566] ret_from_fork+0x31/0x40 [36510.877143] ? kthread_complete_and_exit+0x20/0x20 [36510.881927] ret_from_fork_asm+0x11/0x20 [36510.885849] [36538.096194] watchdog: BUG: soft lockup - CPU#0 stuck for 22914s! [dbench:78683] [36538.103503] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36538.108194] watchdog: BUG: soft lockup - CPU#7 stuck for 31911s! [kworker/7:4:78808] [36538.158952] irq event stamp: 63953706 [36538.166682] Modules linked in: ceph libceph dns_resolver fscache [36538.170338] hardirqs last enabled at (63953705): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36538.170343] netfs veth nft_chain_nat xt_MASQUERADE [36538.186493] hardirqs last disabled at (63953706): [] sysvec_apic_timer_interrupt+0xe/0x90 [36538.186495] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [36538.201089] softirqs last enabled at (63953628): [] __do_softirq+0x2be/0x371 [36538.209599] bridge stp llc xfs libcrc32c sunrpc [36538.218283] softirqs last disabled at (63953623): [] irq_exit_rcu+0xb0/0xd0 [36538.222895] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [36538.231407] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36538.231408] kvm_intel [36538.231409] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36538.239049] kvm irqbypass [36538.247994] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36538.250352] i2c_i801 iTCO_wdt [36538.257822] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36538.260526] iTCO_vendor_support [36538.266003] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36538.269054] i2c_smbus [36538.287792] [36538.291017] crct10dif_pclmul joydev [36538.296233] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36538.298592] crc32_pclmul [36538.300082] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36538.303654] ipmi_ssif ghash_clmulni_intel [36538.310778] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36538.313393] mei_me mei [36538.320518] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36538.324611] lpc_ich [36538.331732] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36538.334177] sha512_ssse3 acpi_ipmi [36538.341302] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36538.343485] mfd_core wmi [36538.350608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36538.354092] ipmi_si ipmi_devintf [36538.362170] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36538.364787] ipmi_msghandler [36538.370525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36538.373835] ioatdma acpi_pad [36538.380960] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36538.383837] acpi_power_meter ixgbe [36538.390963] Call Trace: [36538.390965] [36538.393926] igb mdio [36538.401052] ? watchdog_timer_fn+0x1b4/0x220 [36538.404534] crc32c_intel [36538.406980] ? __hrtimer_run_queues+0x1d3/0x360 [36538.408990] nvme [36538.411262] ? hrtimer_interrupt+0x104/0x240 [36538.415524] ptp [36538.418142] ? __local_bh_enable+0x50/0x60 [36538.422665] nvme_core [36538.424596] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36538.428870] i2c_algo_bit [36538.430708] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36538.434800] pps_core [36538.437156] [36538.442375] dca [36538.444992] [36538.449949] fuse [36538.452221] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36538.454317] [36538.454318] irq event stamp: 212952200 [36538.456158] ? queued_spin_lock_slowpath+0x18/0x2e0 [36538.458250] hardirqs last enabled at (212952199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36538.460178] do_raw_spin_lock+0xb5/0xc0 [36538.465479] hardirqs last disabled at (212952200): [] sysvec_apic_timer_interrupt+0xe/0x90 [36538.466973] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36538.470714] softirqs last enabled at (212952198): [] __do_softirq+0x2be/0x371 [36538.475596] try_get_cap_refs+0x159/0x880 [ceph] [36538.485839] softirqs last disabled at (212952193): [] irq_exit_rcu+0xb0/0xd0 [36538.489674] __ceph_get_caps+0x4a1/0x760 [ceph] [36538.499480] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36538.504267] ? __file_remove_privs+0x89/0x110 [36538.513034] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36538.517644] ? wake_bit_function+0x70/0x70 [36538.526243] Workqueue: events delayed_work [ceph] [36538.530771] ? ceph_write_iter+0x260/0x770 [ceph] [36538.540145] [36538.540145] RIP: 0010:lock_acquire+0x4a/0x2b0 [36538.544497] ceph_write_iter+0x260/0x770 [ceph] [36538.551966] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [36538.556061] ? lock_is_held_type+0xaa/0x120 [36538.560754] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [36538.565455] vfs_write+0x324/0x4f0 [36538.566943] [36538.566944] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [36538.571297] ksys_pwrite64+0x65/0xa0 [36538.575818] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [36538.594556] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36538.598732] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [36538.603951] do_syscall_64+0x38/0x80 [36538.607347] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [36538.608838] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36538.615962] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36538.619534] RIP: 0033:0x7f48f0eeb93b [36538.626657] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36538.631788] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36538.638912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36538.642481] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36538.649605] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36538.654650] ORIG_RAX: 0000000000000012 [36538.661774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36538.665346] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36538.673422] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36538.692161] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36538.697902] Call Trace: [36538.697903] [36538.703116] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36538.710241] ? watchdog_timer_fn+0x1b4/0x220 [36538.714071] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36538.721196] ? __hrtimer_run_queues+0x1d3/0x360 [36538.728317] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36538.735447] ? hrtimer_interrupt+0x104/0x240 [36538.742572] [36538.745011] ? __local_bh_enable+0x50/0x60 [36538.787725] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36538.792947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36538.797906] [36538.800002] [36538.802101] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36538.807408] ? lock_acquire+0x4a/0x2b0 [36538.811159] ? igrab+0x43/0x50 [36538.814216] ? lock_release+0xc5/0x260 [36538.817961] _raw_spin_lock+0x30/0x40 [36538.821619] ? igrab+0x1d/0x50 [36538.824669] igrab+0x1d/0x50 [36538.827548] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36538.832697] delayed_work+0x1d8/0x280 [ceph] [36538.836976] ? process_scheduled_works+0x22a/0x510 [36538.841766] process_scheduled_works+0x2a4/0x510 [36538.846383] worker_thread+0x188/0x340 [36538.850134] ? rescuer_thread+0x310/0x310 [36538.854144] kthread+0xf5/0x130 [36538.857290] ? kthread_complete_and_exit+0x20/0x20 [36538.862076] ret_from_fork+0x31/0x40 [36538.865654] ? kthread_complete_and_exit+0x20/0x20 [36538.870439] ret_from_fork_asm+0x11/0x20 [36538.874370] [36566.096665] watchdog: BUG: soft lockup - CPU#0 stuck for 22940s! [dbench:78683] [36566.103964] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36566.108665] watchdog: BUG: soft lockup - CPU#7 stuck for 31937s! [kworker/7:4:78808] [36566.159405] irq event stamp: 64011806 [36566.159407] hardirqs last enabled at (64011805): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36566.167136] Modules linked in: [36566.170793] hardirqs last disabled at (64011806): [] sysvec_apic_timer_interrupt+0xe/0x90 [36566.180950] ceph [36566.184000] softirqs last enabled at (64011790): [] __do_softirq+0x2be/0x371 [36566.193724] libceph [36566.195650] softirqs last disabled at (64011785): [] irq_exit_rcu+0xb0/0xd0 [36566.204342] dns_resolver [36566.206526] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36566.215036] fscache netfs [36566.217672] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36566.226616] veth nft_chain_nat [36566.229320] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36566.236807] xt_MASQUERADE nf_nat [36566.239945] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36566.245422] nf_conntrack nf_defrag_ipv6 [36566.248734] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36566.267470] nf_defrag_ipv4 [36566.271388] [36566.271389] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36566.276605] nft_compat [36566.279395] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36566.280887] nf_tables nfnetlink [36566.288010] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36566.290456] bridge stp [36566.297578] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36566.300803] llc [36566.307927] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36566.310372] xfs libcrc32c [36566.317496] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36566.319342] sunrpc intel_rapl_msr [36566.326466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36566.329169] intel_rapl_common sb_edac [36566.337248] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36566.340645] x86_pkg_temp_thermal [36566.346383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36566.350127] coretemp kvm_intel [36566.357251] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36566.360561] kvm irqbypass [36566.367687] Call Trace: [36566.367688] [36566.370822] i2c_i801 iTCO_wdt [36566.377947] ? watchdog_timer_fn+0x1b4/0x220 [36566.380651] iTCO_vendor_support i2c_smbus [36566.383097] ? __hrtimer_run_queues+0x1d3/0x360 [36566.385105] crct10dif_pclmul [36566.388159] ? hrtimer_interrupt+0x104/0x240 [36566.392420] joydev crc32_pclmul [36566.396512] ? __local_bh_enable+0x50/0x60 [36566.401035] ipmi_ssif ghash_clmulni_intel [36566.404002] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36566.408263] mei_me mei [36566.411488] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36566.415579] lpc_ich sha512_ssse3 [36566.419670] [36566.424885] acpi_ipmi mfd_core [36566.427347] [36566.427348] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36566.432305] wmi ipmi_si [36566.435621] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36566.437713] ipmi_devintf ipmi_msghandler [36566.440853] do_raw_spin_lock+0xb5/0xc0 [36566.442949] ioatdma [36566.448254] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36566.450784] acpi_pad acpi_power_meter ixgbe [36566.455657] try_get_cap_refs+0x159/0x880 [ceph] [36566.459657] igb mdio [36566.463494] __ceph_get_caps+0x4a1/0x760 [ceph] [36566.465672] crc32c_intel [36566.470461] ? __file_remove_privs+0x89/0x110 [36566.474719] nvme ptp [36566.479331] ? wake_bit_function+0x70/0x70 [36566.481618] nvme_core i2c_algo_bit [36566.486147] ? ceph_write_iter+0x260/0x770 [ceph] [36566.488760] pps_core [36566.493113] ceph_write_iter+0x260/0x770 [ceph] [36566.495382] dca fuse [36566.499475] ? lock_is_held_type+0xaa/0x120 [36566.502957] [36566.502957] irq event stamp: 213173064 [36566.507658] vfs_write+0x324/0x4f0 [36566.509925] hardirqs last enabled at (213173063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36566.514453] ksys_pwrite64+0x65/0xa0 [36566.516721] hardirqs last disabled at (213173064): [] sysvec_apic_timer_interrupt+0xe/0x90 [36566.520898] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36566.522388] softirqs last enabled at (213173062): [] __do_softirq+0x2be/0x371 [36566.526133] do_syscall_64+0x38/0x80 [36566.529529] softirqs last disabled at (213173057): [] irq_exit_rcu+0xb0/0xd0 [36566.539774] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36566.543344] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36566.553157] RIP: 0033:0x7f48f0eeb93b [36566.558285] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36566.567067] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36566.570637] Workqueue: events delayed_work [ceph] [36566.579233] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36566.584278] [36566.584278] RIP: 0010:lock_acquire+0xf0/0x2b0 [36566.593656] ORIG_RAX: 0000000000000012 [36566.597226] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36566.604697] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36566.623434] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36566.628132] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36566.633349] [36566.633350] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36566.634856] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36566.639208] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36566.643038] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36566.661777] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36566.668900] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36566.674118] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36566.681247] [36566.682732] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36566.741928] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36566.750006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36566.755742] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36566.762866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36566.769991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36566.777115] Call Trace: [36566.779560] [36566.781572] ? watchdog_timer_fn+0x1b4/0x220 [36566.785835] ? __hrtimer_run_queues+0x1d3/0x360 [36566.790362] ? hrtimer_interrupt+0x104/0x240 [36566.794631] ? __local_bh_enable+0x50/0x60 [36566.798725] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36566.803947] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36566.808906] [36566.811002] [36566.813100] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36566.818408] ? lock_acquire+0xf0/0x2b0 [36566.822159] ? igrab+0x43/0x50 [36566.825217] ? lock_release+0xc5/0x260 [36566.828962] _raw_spin_lock+0x30/0x40 [36566.832617] ? igrab+0x1d/0x50 [36566.835669] igrab+0x1d/0x50 [36566.838547] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36566.843696] delayed_work+0x1d8/0x280 [ceph] [36566.847979] ? process_scheduled_works+0x22a/0x510 [36566.852769] process_scheduled_works+0x2a4/0x510 [36566.857383] worker_thread+0x188/0x340 [36566.861133] ? rescuer_thread+0x310/0x310 [36566.865135] kthread+0xf5/0x130 [36566.868273] ? kthread_complete_and_exit+0x20/0x20 [36566.873075] ret_from_fork+0x31/0x40 [36566.876655] ? kthread_complete_and_exit+0x20/0x20 [36566.881439] ret_from_fork_asm+0x11/0x20 [36566.885361] [36594.097137] watchdog: BUG: soft lockup - CPU#0 stuck for 22966s! [dbench:78683] [36594.104442] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36594.109137] watchdog: BUG: soft lockup - CPU#7 stuck for 31963s! [kworker/7:4:78808] [36594.159893] irq event stamp: 64069992 [36594.167622] Modules linked in: ceph libceph dns_resolver fscache netfs [36594.171280] hardirqs last enabled at (64069991): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36594.171285] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [36594.187982] hardirqs last disabled at (64069992): [] sysvec_apic_timer_interrupt+0xe/0x90 [36594.195365] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [36594.205088] softirqs last enabled at (64069914): [] __do_softirq+0x2be/0x371 [36594.211867] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [36594.220552] softirqs last disabled at (64069909): [] irq_exit_rcu+0xb0/0xd0 [36594.220556] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [36594.235937] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36594.235938] iTCO_wdt [36594.235939] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36594.242886] iTCO_vendor_support [36594.251830] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36594.254100] i2c_smbus crct10dif_pclmul [36594.261570] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36594.264811] joydev crc32_pclmul [36594.270289] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36594.274120] ipmi_ssif ghash_clmulni_intel [36594.292859] [36594.296082] mei_me [36594.301300] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36594.305390] mei lpc_ich [36594.306880] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36594.308978] sha512_ssse3 acpi_ipmi [36594.316103] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36594.318633] mfd_core wmi [36594.325756] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36594.329241] ipmi_si [36594.336366] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36594.338984] ipmi_devintf ipmi_msghandler [36594.346108] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36594.348291] ioatdma [36594.355417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36594.359419] acpi_pad acpi_power_meter [36594.367496] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36594.369681] ixgbe igb [36594.375417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36594.379162] mdio [36594.386287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36594.388644] crc32c_intel nvme [36594.395768] Call Trace: [36594.395771] [36594.397691] ptp nvme_core [36594.404820] ? watchdog_timer_fn+0x1b4/0x220 [36594.407867] i2c_algo_bit [36594.410313] ? __hrtimer_run_queues+0x1d3/0x360 [36594.412322] pps_core [36594.415029] ? hrtimer_interrupt+0x104/0x240 [36594.419290] dca fuse [36594.421908] ? __local_bh_enable+0x50/0x60 [36594.426431] [36594.426432] irq event stamp: 213393614 [36594.428705] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36594.432967] hardirqs last enabled at (213393613): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36594.435236] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36594.439328] hardirqs last disabled at (213393614): [] sysvec_apic_timer_interrupt+0xe/0x90 [36594.440819] [36594.444562] softirqs last enabled at (213393612): [] __do_softirq+0x2be/0x371 [36594.449780] [36594.449782] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36594.460023] softirqs last disabled at (213393607): [] irq_exit_rcu+0xb0/0xd0 [36594.464986] ? queued_spin_lock_slowpath+0x18/0x2e0 [36594.474793] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36594.476910] do_raw_spin_lock+0xb5/0xc0 [36594.485679] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36594.487777] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36594.493081] Workqueue: events delayed_work [ceph] [36594.501679] try_get_cap_refs+0x159/0x880 [ceph] [36594.506548] [36594.515931] __ceph_get_caps+0x4a1/0x760 [ceph] [36594.519755] RIP: 0010:lock_release+0x78/0x260 [36594.527231] ? __file_remove_privs+0x89/0x110 [36594.532010] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [36594.536707] ? wake_bit_function+0x70/0x70 [36594.541318] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [36594.542824] ? ceph_write_iter+0x260/0x770 [ceph] [36594.547342] [36594.547342] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [36594.551693] ceph_write_iter+0x260/0x770 [ceph] [36594.556045] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [36594.574787] ? lock_is_held_type+0xaa/0x120 [36594.578872] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36594.584093] vfs_write+0x324/0x4f0 [36594.588786] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36594.590283] ksys_pwrite64+0x65/0xa0 [36594.597403] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36594.601926] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36594.609050] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36594.613228] do_syscall_64+0x38/0x80 [36594.620351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36594.623751] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36594.630873] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36594.634444] RIP: 0033:0x7f48f0eeb93b [36594.641569] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36594.646699] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36594.654776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36594.658347] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36594.664085] Call Trace: [36594.669128] ORIG_RAX: 0000000000000012 [36594.676254] [36594.679823] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36594.686950] ? watchdog_timer_fn+0x1b4/0x220 [36594.705686] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36594.712820] ? __hrtimer_run_queues+0x1d3/0x360 [36594.718036] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36594.720483] ? hrtimer_interrupt+0x104/0x240 [36594.724310] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36594.726322] ? __local_bh_enable+0x50/0x60 [36594.733446] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36594.737713] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36594.744839] [36594.749359] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36594.791443] [36594.793542] [36594.795640] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36594.800943] ? igrab+0x43/0x50 [36594.803997] ? lock_release+0x78/0x260 [36594.807750] _raw_spin_unlock+0x1b/0x40 [36594.811587] igrab+0x43/0x50 [36594.814465] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36594.819616] delayed_work+0x1d8/0x280 [ceph] [36594.823894] ? process_scheduled_works+0x22a/0x510 [36594.828676] process_scheduled_works+0x2a4/0x510 [36594.833293] worker_thread+0x188/0x340 [36594.837043] ? rescuer_thread+0x310/0x310 [36594.841053] kthread+0xf5/0x130 [36594.844192] ? kthread_complete_and_exit+0x20/0x20 [36594.848974] ret_from_fork+0x31/0x40 [36594.852556] ? kthread_complete_and_exit+0x20/0x20 [36594.857339] ret_from_fork_asm+0x11/0x20 [36594.861271] [36622.097608] watchdog: BUG: soft lockup - CPU#0 stuck for 22992s! [dbench:78683] [36622.104913] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36622.109608] watchdog: BUG: soft lockup - CPU#7 stuck for 31989s! [kworker/7:4:78808] [36622.160352] irq event stamp: 64128060 [36622.160354] hardirqs last enabled at (64128059): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36622.168085] Modules linked in: [36622.171741] hardirqs last disabled at (64128060): [] sysvec_apic_timer_interrupt+0xe/0x90 [36622.181899] ceph [36622.184950] softirqs last enabled at (64127982): [] __do_softirq+0x2be/0x371 [36622.194674] libceph dns_resolver [36622.196599] softirqs last disabled at (64127977): [] irq_exit_rcu+0xb0/0xd0 [36622.205284] fscache netfs [36622.208593] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36622.217104] veth nft_chain_nat [36622.219808] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36622.228752] xt_MASQUERADE [36622.231890] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36622.239360] nf_nat nf_conntrack [36622.242065] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36622.247541] nf_defrag_ipv6 nf_defrag_ipv4 [36622.250766] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36622.269504] nft_compat nf_tables [36622.273594] [36622.273595] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36622.278811] nfnetlink [36622.282124] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36622.283613] bridge stp [36622.290738] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36622.293095] llc xfs [36622.300218] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36622.302663] libcrc32c sunrpc [36622.309787] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36622.311971] intel_rapl_msr intel_rapl_common [36622.319095] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36622.322060] sb_edac x86_pkg_temp_thermal [36622.329184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36622.333535] coretemp kvm_intel [36622.341611] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36622.345615] kvm irqbypass [36622.351353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36622.354490] i2c_i801 iTCO_wdt [36622.361614] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36622.364319] iTCO_vendor_support [36622.371442] Call Trace: [36622.371443] [36622.374494] i2c_smbus crct10dif_pclmul [36622.381619] ? watchdog_timer_fn+0x1b4/0x220 [36622.384841] joydev crc32_pclmul [36622.387287] ? __hrtimer_run_queues+0x1d3/0x360 [36622.389310] ipmi_ssif ghash_clmulni_intel [36622.393149] ? hrtimer_interrupt+0x104/0x240 [36622.397409] mei_me mei [36622.400634] ? __local_bh_enable+0x50/0x60 [36622.405158] lpc_ich sha512_ssse3 [36622.409251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36622.413512] acpi_ipmi [36622.415956] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36622.420047] mfd_core wmi [36622.423359] [36622.428573] ipmi_si ipmi_devintf [36622.430932] [36622.430933] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36622.435890] ipmi_msghandler ioatdma [36622.438511] ? queued_spin_lock_slowpath+0x16/0x2e0 [36622.440603] acpi_pad [36622.443917] do_raw_spin_lock+0xb5/0xc0 [36622.446012] acpi_power_meter ixgbe [36622.451317] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36622.454886] igb mdio [36622.459760] try_get_cap_refs+0x159/0x880 [ceph] [36622.462029] crc32c_intel [36622.465863] __ceph_get_caps+0x4a1/0x760 [ceph] [36622.469344] nvme [36622.474131] ? __file_remove_privs+0x89/0x110 [36622.476398] ptp nvme_core [36622.481009] ? wake_bit_function+0x70/0x70 [36622.483627] i2c_algo_bit pps_core [36622.488153] ? ceph_write_iter+0x260/0x770 [ceph] [36622.490074] dca fuse [36622.494427] ceph_write_iter+0x260/0x770 [ceph] [36622.497130] [36622.497131] irq event stamp: 213614564 [36622.501224] ? lock_is_held_type+0xaa/0x120 [36622.504617] hardirqs last enabled at (213614563): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36622.509318] vfs_write+0x324/0x4f0 [36622.511585] hardirqs last disabled at (213614564): [] sysvec_apic_timer_interrupt+0xe/0x90 [36622.516114] ksys_pwrite64+0x65/0xa0 [36622.517601] softirqs last enabled at (213614562): [] __do_softirq+0x2be/0x371 [36622.521346] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36622.525522] softirqs last disabled at (213614557): [] irq_exit_rcu+0xb0/0xd0 [36622.535766] do_syscall_64+0x38/0x80 [36622.539164] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36622.548975] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36622.552546] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36622.561316] RIP: 0033:0x7f48f0eeb93b [36622.566448] Workqueue: events delayed_work [ceph] [36622.575044] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36622.578614] [36622.578615] RIP: 0010:lock_acquire+0xf0/0x2b0 [36622.588009] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36622.593053] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36622.600523] ORIG_RAX: 0000000000000012 [36622.604096] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36622.608791] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36622.627530] [36622.629022] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36622.633371] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36622.638589] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36622.657327] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36622.661158] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36622.666375] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36622.673500] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36622.675006] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36622.682138] [36622.689255] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36622.741317] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36622.749396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36622.755131] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36622.762256] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36622.769381] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36622.776505] Call Trace: [36622.778951] [36622.780960] ? watchdog_timer_fn+0x1b4/0x220 [36622.785223] ? __hrtimer_run_queues+0x1d3/0x360 [36622.789751] ? hrtimer_interrupt+0x104/0x240 [36622.794020] ? __local_bh_enable+0x50/0x60 [36622.798115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36622.803337] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36622.808294] [36622.810391] [36622.812491] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36622.817799] ? lock_acquire+0xf0/0x2b0 [36622.821549] ? igrab+0x43/0x50 [36622.824607] ? lock_release+0xc5/0x260 [36622.828350] _raw_spin_lock+0x30/0x40 [36622.832013] ? igrab+0x1d/0x50 [36622.835067] igrab+0x1d/0x50 [36622.837945] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36622.843096] delayed_work+0x1d8/0x280 [ceph] [36622.847375] ? process_scheduled_works+0x22a/0x510 [36622.852157] process_scheduled_works+0x2a4/0x510 [36622.856773] worker_thread+0x188/0x340 [36622.860522] ? rescuer_thread+0x310/0x310 [36622.864526] kthread+0xf5/0x130 [36622.867664] ? kthread_complete_and_exit+0x20/0x20 [36622.872448] ret_from_fork+0x31/0x40 [36622.876025] ? kthread_complete_and_exit+0x20/0x20 [36622.880811] ret_from_fork_asm+0x11/0x20 [36622.884732] [36624.125009] rcu: INFO: rcu_preempt self-detected stall on CPU [36624.130765] rcu: 7-....: (23970142 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/23968791 fqs=12082430 [36624.141442] rcu: (t=24687080 jiffies g=1309609 q=680886 ncpus=8) [36624.147529] Sending NMI from CPU 7 to CPUs 0: [36624.151880] NMI backtrace for cpu 0 [36624.151882] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36624.151883] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36624.151884] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36624.151887] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36624.151888] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36624.151889] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36624.151890] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36624.151891] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36624.151892] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36624.151892] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36624.151893] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36624.151894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36624.151895] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36624.151896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36624.151896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36624.151897] Call Trace: [36624.151898] [36624.151899] ? nmi_cpu_backtrace+0xad/0x120 [36624.151902] ? nmi_cpu_backtrace_handler+0x11/0x20 [36624.151904] ? nmi_handle+0xd4/0x260 [36624.151908] ? default_do_nmi+0x49/0xf0 [36624.151910] ? exc_nmi+0x11e/0x150 [36624.151911] ? end_repeat_nmi+0x16/0x67 [36624.151915] ? queued_spin_lock_slowpath+0x16/0x2e0 [36624.151918] ? queued_spin_lock_slowpath+0x16/0x2e0 [36624.151920] ? queued_spin_lock_slowpath+0x16/0x2e0 [36624.151922] [36624.151923] [36624.151924] do_raw_spin_lock+0xb5/0xc0 [36624.151926] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36624.151940] try_get_cap_refs+0x159/0x880 [ceph] [36624.151955] __ceph_get_caps+0x4a1/0x760 [ceph] [36624.151968] ? __file_remove_privs+0x89/0x110 [36624.151969] ? wake_bit_function+0x70/0x70 [36624.151972] ? ceph_write_iter+0x260/0x770 [ceph] [36624.151983] ceph_write_iter+0x260/0x770 [ceph] [36624.151994] ? lock_is_held_type+0xaa/0x120 [36624.151997] vfs_write+0x324/0x4f0 [36624.152001] ksys_pwrite64+0x65/0xa0 [36624.152003] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36624.152006] do_syscall_64+0x38/0x80 [36624.152009] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36624.152011] RIP: 0033:0x7f48f0eeb93b [36624.152013] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36624.152014] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [36624.152015] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36624.152016] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36624.152017] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36624.152017] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36624.152018] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36624.152021] [36624.152879] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36624.456369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36624.463840] Workqueue: events delayed_work [ceph] [36624.468555] RIP: 0010:lock_release+0x78/0x260 [36624.472905] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [36624.491644] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [36624.496861] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [36624.503984] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [36624.511108] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36624.518234] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36624.525359] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36624.532482] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36624.540560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36624.546296] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36624.553421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36624.560544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36624.567668] Call Trace: [36624.570112] [36624.572125] ? rcu_dump_cpu_stacks+0xd1/0x180 [36624.576484] ? rcu_sched_clock_irq+0x633/0x1390 [36624.581008] ? lock_release+0xc5/0x260 [36624.584751] ? lock_is_held_type+0xaa/0x120 [36624.588931] ? lock_is_held_type+0xaa/0x120 [36624.593108] ? tick_sched_do_timer+0xa0/0xa0 [36624.597379] ? rcu_read_lock_sched_held+0x56/0x90 [36624.602077] ? tick_sched_do_timer+0xa0/0xa0 [36624.606340] ? update_process_times+0x5f/0x90 [36624.610692] ? tick_sched_handle.isra.16+0x1e/0x50 [36624.615475] ? tick_sched_timer+0x6f/0x80 [36624.619481] ? __hrtimer_run_queues+0x1d3/0x360 [36624.624008] ? hrtimer_interrupt+0x104/0x240 [36624.628275] ? __local_bh_enable+0x50/0x60 [36624.632371] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36624.637594] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36624.642550] [36624.644649] [36624.646746] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36624.652049] ? igrab+0x43/0x50 [36624.655104] ? lock_release+0x78/0x260 [36624.658856] _raw_spin_unlock+0x1b/0x40 [36624.662694] igrab+0x43/0x50 [36624.665571] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36624.670720] delayed_work+0x1d8/0x280 [ceph] [36624.675001] ? process_scheduled_works+0x22a/0x510 [36624.679792] process_scheduled_works+0x2a4/0x510 [36624.684408] worker_thread+0x188/0x340 [36624.688156] ? rescuer_thread+0x310/0x310 [36624.692159] kthread+0xf5/0x130 [36624.695297] ? kthread_complete_and_exit+0x20/0x20 [36624.700083] ret_from_fork+0x31/0x40 [36624.703661] ? kthread_complete_and_exit+0x20/0x20 [36624.708445] ret_from_fork_asm+0x11/0x20 [36624.712369] [36650.098078] watchdog: BUG: soft lockup - CPU#0 stuck for 23019s! [dbench:78683] [36650.105384] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36650.110078] watchdog: BUG: soft lockup - CPU#7 stuck for 32016s! [kworker/7:4:78808] [36650.160833] irq event stamp: 64186282 [36650.168564] Modules linked in: ceph libceph dns_resolver [36650.172220] hardirqs last enabled at (64186281): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36650.177525] fscache netfs veth [36650.187682] hardirqs last disabled at (64186282): [] sysvec_apic_timer_interrupt+0xe/0x90 [36650.190819] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [36650.200544] softirqs last enabled at (64186206): [] __do_softirq+0x2be/0x371 [36650.209771] nf_tables nfnetlink bridge stp llc xfs libcrc32c [36650.218474] softirqs last disabled at (64186201): [] irq_exit_rcu+0xb0/0xd0 [36650.224213] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [36650.232727] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36650.240194] coretemp kvm_intel kvm [36650.249139] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36650.249140] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36650.252623] irqbypass i2c_i801 [36650.260094] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36650.265571] iTCO_wdt iTCO_vendor_support [36650.268709] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36650.287446] i2c_smbus crct10dif_pclmul [36650.291473] [36650.296695] joydev [36650.300525] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36650.302016] crc32_pclmul ipmi_ssif [36650.304113] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36650.311238] ghash_clmulni_intel mei_me [36650.314722] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36650.321845] mei lpc_ich [36650.325676] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36650.332801] sha512_ssse3 [36650.335331] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36650.342455] acpi_ipmi mfd_core [36650.345089] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36650.352215] wmi ipmi_si [36650.355350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36650.363429] ipmi_devintf [36650.365960] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36650.371697] ipmi_msghandler [36650.374315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36650.381438] ioatdma [36650.384315] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36650.391439] acpi_pad [36650.393623] Call Trace: [36650.393625] [36650.400759] acpi_power_meter [36650.403029] ? watchdog_timer_fn+0x1b4/0x220 [36650.405477] ixgbe igb [36650.407491] ? __hrtimer_run_queues+0x1d3/0x360 [36650.410471] mdio crc32c_intel [36650.414739] ? hrtimer_interrupt+0x104/0x240 [36650.417095] nvme ptp [36650.421619] ? __local_bh_enable+0x50/0x60 [36650.424669] nvme_core i2c_algo_bit [36650.428935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36650.431205] pps_core [36650.435295] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36650.438779] dca fuse [36650.443994] [36650.446267] [36650.446267] irq event stamp: 213830538 [36650.451224] [36650.451225] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36650.453494] hardirqs last enabled at (213830537): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36650.455595] ? queued_spin_lock_slowpath+0x16/0x2e0 [36650.457082] hardirqs last disabled at (213830538): [] sysvec_apic_timer_interrupt+0xe/0x90 [36650.460829] do_raw_spin_lock+0xb5/0xc0 [36650.462924] softirqs last enabled at (213830536): [] __do_softirq+0x2be/0x371 [36650.468229] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36650.478478] softirqs last disabled at (213830531): [] irq_exit_rcu+0xb0/0xd0 [36650.483354] try_get_cap_refs+0x159/0x880 [ceph] [36650.493163] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36650.497000] __ceph_get_caps+0x4a1/0x760 [ceph] [36650.505767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36650.510552] ? __file_remove_privs+0x89/0x110 [36650.519147] Workqueue: events delayed_work [ceph] [36650.523770] ? wake_bit_function+0x70/0x70 [36650.533151] [36650.537678] ? ceph_write_iter+0x260/0x770 [ceph] [36650.545147] RIP: 0010:lock_acquire+0xf0/0x2b0 [36650.549498] ceph_write_iter+0x260/0x770 [ceph] [36650.554194] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36650.558290] ? lock_is_held_type+0xaa/0x120 [36650.559780] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36650.564483] vfs_write+0x324/0x4f0 [36650.568831] [36650.573359] ksys_pwrite64+0x65/0xa0 [36650.592094] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36650.596273] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36650.601490] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36650.604887] do_syscall_64+0x38/0x80 [36650.606377] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36650.609950] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36650.617072] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36650.622205] RIP: 0033:0x7f48f0eeb93b [36650.629326] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36650.632897] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36650.640022] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36650.645066] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36650.652190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36650.655761] ORIG_RAX: 0000000000000012 [36650.662884] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36650.681623] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36650.689699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36650.694917] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36650.700654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36650.704485] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36650.704486] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36650.711609] Call Trace: [36650.711611] [36650.718734] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36650.725859] ? watchdog_timer_fn+0x1b4/0x220 [36650.732988] [36650.740108] ? __hrtimer_run_queues+0x1d3/0x360 [36650.776911] ? hrtimer_interrupt+0x104/0x240 [36650.781179] ? __local_bh_enable+0x50/0x60 [36650.785271] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36650.790497] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36650.795453] [36650.797549] [36650.799649] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36650.804956] ? lock_acquire+0xf0/0x2b0 [36650.808706] ? igrab+0x43/0x50 [36650.811767] ? lock_release+0xc5/0x260 [36650.815510] _raw_spin_lock+0x30/0x40 [36650.819174] ? igrab+0x1d/0x50 [36650.822226] igrab+0x1d/0x50 [36650.825103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36650.830254] delayed_work+0x1d8/0x280 [ceph] [36650.834534] ? process_scheduled_works+0x22a/0x510 [36650.839325] process_scheduled_works+0x2a4/0x510 [36650.843941] worker_thread+0x188/0x340 [36650.847689] ? rescuer_thread+0x310/0x310 [36650.851692] kthread+0xf5/0x130 [36650.854832] ? kthread_complete_and_exit+0x20/0x20 [36650.859622] ret_from_fork+0x31/0x40 [36650.863193] ? kthread_complete_and_exit+0x20/0x20 [36650.867977] ret_from_fork_asm+0x11/0x20 [36650.871911] [36678.098509] watchdog: BUG: soft lockup - CPU#0 stuck for 23045s! [dbench:78683] [36678.105810] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36678.110509] watchdog: BUG: soft lockup - CPU#7 stuck for 32042s! [kworker/7:4:78808] [36678.161251] irq event stamp: 64244324 [36678.161252] hardirqs last enabled at (64244323): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36678.168983] Modules linked in: [36678.172639] hardirqs last disabled at (64244324): [] sysvec_apic_timer_interrupt+0xe/0x90 [36678.182797] ceph [36678.185848] softirqs last enabled at (64244308): [] __do_softirq+0x2be/0x371 [36678.195572] libceph [36678.197496] softirqs last disabled at (64244303): [] irq_exit_rcu+0xb0/0xd0 [36678.206184] dns_resolver [36678.208365] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36678.216874] fscache netfs [36678.219509] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36678.228454] veth nft_chain_nat [36678.231157] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36678.238646] xt_MASQUERADE [36678.241783] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36678.247262] nf_nat [36678.249965] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36678.268704] nf_conntrack nf_defrag_ipv6 [36678.270801] [36678.270802] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36678.276018] nf_defrag_ipv4 nft_compat [36678.279936] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36678.281425] nf_tables nfnetlink [36678.288549] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36678.292293] bridge [36678.299417] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36678.302642] stp llc [36678.309766] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36678.311864] xfs libcrc32c [36678.318987] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36678.321184] sunrpc [36678.328304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36678.331008] intel_rapl_msr intel_rapl_common [36678.339084] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36678.341182] sb_edac [36678.346921] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36678.351272] x86_pkg_temp_thermal [36678.358396] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36678.360580] coretemp [36678.367703] Call Trace: [36678.367704] [36678.371015] kvm_intel kvm [36678.378138] ? watchdog_timer_fn+0x1b4/0x220 [36678.380410] irqbypass [36678.382854] ? __hrtimer_run_queues+0x1d3/0x360 [36678.384865] i2c_i801 iTCO_wdt [36678.387571] ? hrtimer_interrupt+0x104/0x240 [36678.391833] iTCO_vendor_support i2c_smbus [36678.394192] ? __local_bh_enable+0x50/0x60 [36678.398714] crct10dif_pclmul [36678.401767] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36678.406028] joydev crc32_pclmul [36678.410121] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36678.414210] ipmi_ssif ghash_clmulni_intel [36678.417184] [36678.422399] mei_me mei [36678.425625] [36678.425626] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36678.430582] lpc_ich [36678.434677] ? queued_spin_lock_slowpath+0x18/0x2e0 [36678.436769] sha512_ssse3 acpi_ipmi [36678.439215] do_raw_spin_lock+0xb5/0xc0 [36678.441311] mfd_core wmi [36678.446617] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36678.448799] ipmi_si ipmi_devintf [36678.453671] try_get_cap_refs+0x159/0x880 [ceph] [36678.457155] ipmi_msghandler ioatdma [36678.460990] __ceph_get_caps+0x4a1/0x760 [ceph] [36678.463603] acpi_pad acpi_power_meter [36678.468391] ? __file_remove_privs+0x89/0x110 [36678.471696] ixgbe [36678.476308] ? wake_bit_function+0x70/0x70 [36678.479880] igb mdio [36678.484422] ? ceph_write_iter+0x260/0x770 [ceph] [36678.488165] crc32c_intel nvme [36678.492533] ceph_write_iter+0x260/0x770 [ceph] [36678.494544] ptp nvme_core [36678.498637] ? lock_is_held_type+0xaa/0x120 [36678.500906] i2c_algo_bit pps_core [36678.505605] vfs_write+0x324/0x4f0 [36678.508652] dca [36678.513191] ksys_pwrite64+0x65/0xa0 [36678.515903] fuse [36678.520083] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36678.523481] [36678.523482] irq event stamp: 214051348 [36678.526880] do_syscall_64+0x38/0x80 [36678.528734] hardirqs last enabled at (214051347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36678.532305] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36678.534227] hardirqs last disabled at (214051348): [] sysvec_apic_timer_interrupt+0xe/0x90 [36678.539359] RIP: 0033:0x7f48f0eeb93b [36678.540850] softirqs last enabled at (214051346): [] __do_softirq+0x2be/0x371 [36678.544594] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36678.548165] softirqs last disabled at (214051341): [] irq_exit_rcu+0xb0/0xd0 [36678.558425] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36678.563470] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36678.573281] ORIG_RAX: 0000000000000012 [36678.576851] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36678.585621] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36678.604359] Workqueue: events delayed_work [ceph] [36678.612958] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36678.618185] [36678.618186] RIP: 0010:lock_acquire+0xf0/0x2b0 [36678.627561] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36678.631392] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36678.638862] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36678.645986] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36678.650683] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36678.657808] [36678.657809] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36678.659303] [36678.663650] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36678.726908] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36678.734035] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36678.741157] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36678.748281] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36678.756358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36678.762097] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36678.769220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36678.776345] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36678.783470] Call Trace: [36678.785913] [36678.787926] ? watchdog_timer_fn+0x1b4/0x220 [36678.792188] ? __hrtimer_run_queues+0x1d3/0x360 [36678.796716] ? hrtimer_interrupt+0x104/0x240 [36678.800986] ? __local_bh_enable+0x50/0x60 [36678.805079] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36678.810301] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36678.815259] [36678.817356] [36678.819455] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36678.824761] ? lock_acquire+0xf0/0x2b0 [36678.828513] ? igrab+0x43/0x50 [36678.831562] ? lock_release+0xc5/0x260 [36678.835306] _raw_spin_lock+0x30/0x40 [36678.838964] ? igrab+0x1d/0x50 [36678.842014] igrab+0x1d/0x50 [36678.844907] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36678.850059] delayed_work+0x1d8/0x280 [ceph] [36678.854340] ? process_scheduled_works+0x22a/0x510 [36678.859132] process_scheduled_works+0x2a4/0x510 [36678.863745] worker_thread+0x188/0x340 [36678.867497] ? rescuer_thread+0x310/0x310 [36678.871506] kthread+0xf5/0x130 [36678.874643] ? kthread_complete_and_exit+0x20/0x20 [36678.879428] ret_from_fork+0x31/0x40 [36678.882999] ? kthread_complete_and_exit+0x20/0x20 [36678.887782] ret_from_fork_asm+0x11/0x20 [36678.891706] [36706.098685] watchdog: BUG: soft lockup - CPU#0 stuck for 23071s! [dbench:78683] [36706.105988] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36706.110685] watchdog: BUG: soft lockup - CPU#7 stuck for 32068s! [kworker/7:4:78808] [36706.161438] irq event stamp: 64302586 [36706.161439] hardirqs last enabled at (64302585): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36706.169168] Modules linked in: [36706.172825] hardirqs last disabled at (64302586): [] sysvec_apic_timer_interrupt+0xe/0x90 [36706.182980] ceph [36706.186032] softirqs last enabled at (64302550): [] __do_softirq+0x2be/0x371 [36706.195757] libceph dns_resolver [36706.197681] softirqs last disabled at (64302545): [] irq_exit_rcu+0xb0/0xd0 [36706.206363] fscache netfs [36706.209676] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36706.218185] veth nft_chain_nat [36706.220891] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36706.229833] xt_MASQUERADE [36706.232970] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36706.240443] nf_nat [36706.243146] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36706.248623] nf_conntrack nf_defrag_ipv6 [36706.250720] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36706.269459] nf_defrag_ipv4 nft_compat [36706.273376] [36706.278593] nf_tables nfnetlink [36706.282355] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36706.283846] bridge stp [36706.287081] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36706.294203] llc [36706.296647] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36706.303770] xfs libcrc32c [36706.305607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36706.312733] sunrpc intel_rapl_msr [36706.315437] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36706.322560] intel_rapl_common sb_edac [36706.325958] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36706.333081] x86_pkg_temp_thermal [36706.336827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36706.344903] coretemp kvm_intel [36706.348214] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36706.353952] kvm irqbypass [36706.357089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36706.364214] i2c_i801 [36706.366917] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36706.374042] iTCO_wdt iTCO_vendor_support [36706.376311] Call Trace: [36706.376314] [36706.383437] i2c_smbus [36706.387442] ? watchdog_timer_fn+0x1b4/0x220 [36706.389884] crct10dif_pclmul joydev [36706.391897] ? __hrtimer_run_queues+0x1d3/0x360 [36706.394252] crc32_pclmul ipmi_ssif [36706.398521] ? hrtimer_interrupt+0x104/0x240 [36706.402089] ghash_clmulni_intel [36706.406610] ? __local_bh_enable+0x50/0x60 [36706.410096] mei_me mei [36706.414364] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36706.417583] lpc_ich sha512_ssse3 [36706.421676] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36706.424120] acpi_ipmi [36706.429335] [36706.432664] mfd_core wmi [36706.437622] [36706.437623] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36706.439979] ipmi_si ipmi_devintf [36706.442085] ? queued_spin_lock_slowpath+0x16/0x2e0 [36706.444692] ipmi_msghandler [36706.446794] do_raw_spin_lock+0xb5/0xc0 [36706.452096] ioatdma acpi_pad [36706.455408] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36706.460278] acpi_power_meter [36706.463157] try_get_cap_refs+0x159/0x880 [ceph] [36706.466984] ixgbe igb [36706.469953] __ceph_get_caps+0x4a1/0x760 [ceph] [36706.474733] mdio crc32c_intel [36706.477700] ? __file_remove_privs+0x89/0x110 [36706.482307] nvme ptp [36706.484667] ? wake_bit_function+0x70/0x70 [36706.489190] nvme_core i2c_algo_bit [36706.492244] ? ceph_write_iter+0x260/0x770 [ceph] [36706.496590] pps_core [36706.498861] ceph_write_iter+0x260/0x770 [ceph] [36706.502953] dca fuse [36706.506440] ? lock_is_held_type+0xaa/0x120 [36706.511132] [36706.511133] irq event stamp: 214272014 [36706.513407] vfs_write+0x324/0x4f0 [36706.517927] hardirqs last enabled at (214272013): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36706.520202] ksys_pwrite64+0x65/0xa0 [36706.524375] hardirqs last disabled at (214272014): [] sysvec_apic_timer_interrupt+0xe/0x90 [36706.525867] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36706.529611] softirqs last enabled at (214272012): [] __do_softirq+0x2be/0x371 [36706.533010] do_syscall_64+0x38/0x80 [36706.543254] softirqs last disabled at (214272003): [] irq_exit_rcu+0xb0/0xd0 [36706.546825] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36706.556634] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36706.561767] RIP: 0033:0x7f48f0eeb93b [36706.570537] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36706.574107] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36706.582706] Workqueue: events delayed_work [ceph] [36706.587748] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36706.597125] [36706.600697] ORIG_RAX: 0000000000000012 [36706.608167] RIP: 0010:lock_acquire+0xf0/0x2b0 [36706.626906] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36706.631603] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36706.636821] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36706.638310] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36706.642140] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36706.646492] [36706.653617] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36706.672370] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36706.679494] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36706.684713] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36706.691842] [36706.693326] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36706.731132] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36706.738255] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36706.745380] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36706.753459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36706.759195] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36706.766319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36706.773445] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36706.780567] Call Trace: [36706.783013] [36706.785023] ? watchdog_timer_fn+0x1b4/0x220 [36706.789288] ? __hrtimer_run_queues+0x1d3/0x360 [36706.793814] ? hrtimer_interrupt+0x104/0x240 [36706.798083] ? __local_bh_enable+0x50/0x60 [36706.802178] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36706.807401] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36706.812372] [36706.814473] [36706.816570] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36706.821877] ? lock_acquire+0xf0/0x2b0 [36706.825629] ? igrab+0x43/0x50 [36706.828678] ? lock_release+0xc5/0x260 [36706.832422] _raw_spin_lock+0x30/0x40 [36706.836082] ? igrab+0x1d/0x50 [36706.839129] igrab+0x1d/0x50 [36706.842010] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36706.847158] delayed_work+0x1d8/0x280 [ceph] [36706.851439] ? process_scheduled_works+0x22a/0x510 [36706.856229] process_scheduled_works+0x2a4/0x510 [36706.860846] worker_thread+0x188/0x340 [36706.864595] ? rescuer_thread+0x310/0x310 [36706.868607] kthread+0xf5/0x130 [36706.871751] ? kthread_complete_and_exit+0x20/0x20 [36706.876536] ret_from_fork+0x31/0x40 [36706.880116] ? kthread_complete_and_exit+0x20/0x20 [36706.884900] ret_from_fork_asm+0x11/0x20 [36706.888834] [36734.098983] watchdog: BUG: soft lockup - CPU#0 stuck for 23097s! [dbench:78683] [36734.106282] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36734.110983] watchdog: BUG: soft lockup - CPU#7 stuck for 32094s! [kworker/7:4:78808] [36734.161723] irq event stamp: 64360756 [36734.161724] hardirqs last enabled at (64360755): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36734.169454] Modules linked in: [36734.173111] hardirqs last disabled at (64360756): [] sysvec_apic_timer_interrupt+0xe/0x90 [36734.183269] ceph [36734.186321] softirqs last enabled at (64360688): [] __do_softirq+0x2be/0x371 [36734.196044] libceph [36734.197969] softirqs last disabled at (64360683): [] irq_exit_rcu+0xb0/0xd0 [36734.206657] dns_resolver fscache [36734.208836] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36734.217349] netfs veth [36734.220675] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36734.229619] nft_chain_nat xt_MASQUERADE [36734.232064] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36734.239536] nf_nat nf_conntrack [36734.243452] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36734.248931] nf_defrag_ipv6 nf_defrag_ipv4 [36734.252153] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36734.270891] nft_compat nf_tables [36734.274983] [36734.274984] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36734.280200] nfnetlink [36734.283509] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36734.285000] bridge [36734.292125] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36734.294483] stp llc [36734.301607] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36734.303704] xfs [36734.310829] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36734.313011] libcrc32c sunrpc [36734.320136] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36734.321974] intel_rapl_msr intel_rapl_common [36734.329097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36734.332061] sb_edac x86_pkg_temp_thermal [36734.340140] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36734.344491] coretemp kvm_intel [36734.350227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36734.354232] kvm irqbypass [36734.361356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36734.364511] i2c_i801 iTCO_wdt [36734.371635] Call Trace: [36734.371637] [36734.374355] iTCO_vendor_support [36734.381498] ? watchdog_timer_fn+0x1b4/0x220 [36734.384548] i2c_smbus crct10dif_pclmul [36734.386993] ? __hrtimer_run_queues+0x1d3/0x360 [36734.389020] joydev crc32_pclmul [36734.392246] ? hrtimer_interrupt+0x104/0x240 [36734.396509] ipmi_ssif ghash_clmulni_intel [36734.400340] ? __local_bh_enable+0x50/0x60 [36734.404864] mei_me mei [36734.408090] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36734.412353] lpc_ich [36734.416443] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36734.420534] sha512_ssse3 acpi_ipmi [36734.422977] [36734.428195] mfd_core wmi [36734.430382] [36734.430383] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36734.435335] ipmi_si ipmi_devintf [36734.438823] ? queued_spin_lock_slowpath+0x16/0x2e0 [36734.440918] ipmi_msghandler ioatdma [36734.443537] do_raw_spin_lock+0xb5/0xc0 [36734.445633] acpi_pad acpi_power_meter [36734.450938] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36734.454246] ixgbe igb [36734.459120] try_get_cap_refs+0x159/0x880 [ceph] [36734.462687] mdio crc32c_intel [36734.466523] __ceph_get_caps+0x4a1/0x760 [ceph] [36734.470263] nvme ptp [36734.475050] ? __file_remove_privs+0x89/0x110 [36734.477405] nvme_core [36734.482016] ? wake_bit_function+0x70/0x70 [36734.485066] i2c_algo_bit pps_core [36734.489594] ? ceph_write_iter+0x260/0x770 [ceph] [36734.491862] dca fuse [36734.496212] ceph_write_iter+0x260/0x770 [ceph] [36734.498568] [36734.498568] irq event stamp: 214492690 [36734.502677] ? lock_is_held_type+0xaa/0x120 [36734.506076] hardirqs last enabled at (214492689): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36734.510773] vfs_write+0x324/0x4f0 [36734.513043] hardirqs last disabled at (214492690): [] sysvec_apic_timer_interrupt+0xe/0x90 [36734.517571] ksys_pwrite64+0x65/0xa0 [36734.519057] softirqs last enabled at (214492688): [] __do_softirq+0x2be/0x371 [36734.522800] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36734.526978] softirqs last disabled at (214492683): [] irq_exit_rcu+0xb0/0xd0 [36734.537225] do_syscall_64+0x38/0x80 [36734.540620] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36734.550432] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36734.554002] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36734.562773] RIP: 0033:0x7f48f0eeb93b [36734.567904] Workqueue: events delayed_work [ceph] [36734.576500] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36734.580072] [36734.580073] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [36734.589449] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36734.594494] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [36734.601964] ORIG_RAX: 0000000000000012 [36734.605534] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [36734.610233] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36734.628970] [36734.628971] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [36734.630460] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36734.635071] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [36734.640288] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36734.659027] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [36734.662858] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36734.668076] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [36734.675198] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36734.676690] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36734.683820] [36734.690937] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36734.751069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36734.756805] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36734.763931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36734.771053] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36734.778179] Call Trace: [36734.780623] [36734.782633] ? watchdog_timer_fn+0x1b4/0x220 [36734.786899] ? __hrtimer_run_queues+0x1d3/0x360 [36734.791425] ? hrtimer_interrupt+0x104/0x240 [36734.795693] ? __local_bh_enable+0x50/0x60 [36734.799788] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36734.805012] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36734.809970] [36734.812066] [36734.814163] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36734.819471] ? do_raw_spin_lock+0x42/0xc0 [36734.823480] igrab+0x1d/0x50 [36734.826359] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36734.831527] delayed_work+0x1d8/0x280 [ceph] [36734.835807] ? process_scheduled_works+0x22a/0x510 [36734.840598] process_scheduled_works+0x2a4/0x510 [36734.845212] worker_thread+0x188/0x340 [36734.848962] ? rescuer_thread+0x310/0x310 [36734.852965] kthread+0xf5/0x130 [36734.856101] ? kthread_complete_and_exit+0x20/0x20 [36734.860887] ret_from_fork+0x31/0x40 [36734.864466] ? kthread_complete_and_exit+0x20/0x20 [36734.869251] ret_from_fork_asm+0x11/0x20 [36734.873172] [36762.099460] watchdog: BUG: soft lockup - CPU#0 stuck for 23123s! [dbench:78683] [36762.106764] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36762.111460] watchdog: BUG: soft lockup - CPU#7 stuck for 32120s! [kworker/7:4:78808] [36762.162213] irq event stamp: 64418902 [36762.169942] Modules linked in: ceph libceph dns_resolver fscache netfs [36762.173601] hardirqs last enabled at (64418901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36762.180141] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [36762.190300] hardirqs last disabled at (64418902): [] sysvec_apic_timer_interrupt+0xe/0x90 [36762.196386] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [36762.206108] softirqs last enabled at (64418834): [] __do_softirq+0x2be/0x371 [36762.206112] stp llc xfs libcrc32c sunrpc intel_rapl_msr [36762.222177] softirqs last disabled at (64418829): [] irq_exit_rcu+0xb0/0xd0 [36762.227482] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [36762.235994] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36762.243204] kvm irqbypass i2c_i801 [36762.252147] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36762.255633] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [36762.263102] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36762.269446] joydev crc32_pclmul [36762.274923] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36762.274924] ipmi_ssif [36762.274925] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36762.278147] ghash_clmulni_intel mei_me [36762.296887] [36762.299243] mei lpc_ich [36762.304460] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36762.308290] sha512_ssse3 [36762.309783] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36762.312314] acpi_ipmi [36762.319437] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36762.322055] mfd_core [36762.329179] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36762.331536] wmi ipmi_si [36762.338661] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36762.340932] ipmi_devintf [36762.348056] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36762.350585] ipmi_msghandler [36762.357709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36762.360327] ioatdma acpi_pad [36762.368403] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36762.371284] acpi_power_meter [36762.377019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36762.377020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36762.379984] ixgbe [36762.387107] Call Trace: [36762.387109] [36762.390071] igb mdio [36762.397199] ? watchdog_timer_fn+0x1b4/0x220 [36762.404320] crc32c_intel nvme [36762.406332] ? __hrtimer_run_queues+0x1d3/0x360 [36762.408774] ptp nvme_core [36762.410788] ? hrtimer_interrupt+0x104/0x240 [36762.413055] i2c_algo_bit pps_core [36762.417319] ? __local_bh_enable+0x50/0x60 [36762.420371] dca fuse [36762.424897] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36762.427600] [36762.427600] irq event stamp: 214713600 [36762.431863] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36762.435260] hardirqs last enabled at (214713599): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36762.439352] [36762.441622] hardirqs last disabled at (214713600): [] sysvec_apic_timer_interrupt+0xe/0x90 [36762.446856] [36762.446857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36762.448347] softirqs last enabled at (214713598): [] __do_softirq+0x2be/0x371 [36762.452096] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36762.457049] softirqs last disabled at (214713593): [] irq_exit_rcu+0xb0/0xd0 [36762.467296] do_raw_spin_lock+0xb5/0xc0 [36762.469390] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36762.479203] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36762.481297] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36762.486604] try_get_cap_refs+0x159/0x880 [ceph] [36762.495375] Workqueue: events delayed_work [ceph] [36762.500251] __ceph_get_caps+0x4a1/0x760 [ceph] [36762.508858] [36762.512694] ? __file_remove_privs+0x89/0x110 [36762.522066] RIP: 0010:lock_acquire+0xf0/0x2b0 [36762.526861] ? wake_bit_function+0x70/0x70 [36762.534332] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36762.538945] ? ceph_write_iter+0x260/0x770 [ceph] [36762.543640] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36762.548162] ceph_write_iter+0x260/0x770 [ceph] [36762.549655] [36762.549656] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36762.554009] ? lock_is_held_type+0xaa/0x120 [36762.558356] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36762.562448] vfs_write+0x324/0x4f0 [36762.581185] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36762.585885] ksys_pwrite64+0x65/0xa0 [36762.591097] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36762.595624] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36762.597114] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36762.604238] do_syscall_64+0x38/0x80 [36762.608415] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36762.615541] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36762.618935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36762.626061] RIP: 0033:0x7f48f0eeb93b [36762.629630] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36762.636755] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36762.641885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36762.649010] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36762.652581] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36762.660659] ORIG_RAX: 0000000000000012 [36762.665701] Call Trace: [36762.671439] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36762.675011] [36762.682144] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36762.700883] ? watchdog_timer_fn+0x1b4/0x220 [36762.708004] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36762.713223] ? __hrtimer_run_queues+0x1d3/0x360 [36762.720347] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36762.724182] ? hrtimer_interrupt+0x104/0x240 [36762.726621] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36762.733746] ? __local_bh_enable+0x50/0x60 [36762.735762] [36762.742884] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36762.788798] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36762.793756] [36762.795865] [36762.797969] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36762.803277] ? lock_acquire+0xf0/0x2b0 [36762.807027] ? igrab+0x43/0x50 [36762.810076] ? lock_release+0xc5/0x260 [36762.813821] _raw_spin_lock+0x30/0x40 [36762.817477] ? igrab+0x1d/0x50 [36762.820527] igrab+0x1d/0x50 [36762.823407] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36762.828557] delayed_work+0x1d8/0x280 [ceph] [36762.832836] ? process_scheduled_works+0x22a/0x510 [36762.837645] process_scheduled_works+0x2a4/0x510 [36762.842260] worker_thread+0x188/0x340 [36762.846010] ? rescuer_thread+0x310/0x310 [36762.850020] kthread+0xf5/0x130 [36762.853160] ? kthread_complete_and_exit+0x20/0x20 [36762.857952] ret_from_fork+0x31/0x40 [36762.861531] ? kthread_complete_and_exit+0x20/0x20 [36762.866315] ret_from_fork_asm+0x11/0x20 [36762.870246] [36790.099938] watchdog: BUG: soft lockup - CPU#0 stuck for 23149s! [dbench:78683] [36790.107240] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36790.111938] watchdog: BUG: soft lockup - CPU#7 stuck for 32146s! [kworker/7:4:78808] [36790.162682] irq event stamp: 64477016 [36790.162683] hardirqs last enabled at (64477015): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36790.170414] Modules linked in: [36790.174070] hardirqs last disabled at (64477016): [] sysvec_apic_timer_interrupt+0xe/0x90 [36790.184229] ceph [36790.187280] softirqs last enabled at (64476996): [] __do_softirq+0x2be/0x371 [36790.197003] libceph dns_resolver [36790.198927] softirqs last disabled at (64476991): [] irq_exit_rcu+0xb0/0xd0 [36790.207614] fscache netfs [36790.210922] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36790.219431] veth [36790.222137] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36790.231080] nft_chain_nat [36790.233006] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36790.240475] xt_MASQUERADE nf_nat [36790.243179] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36790.248658] nf_conntrack nf_defrag_ipv6 [36790.251969] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36790.270707] nf_defrag_ipv4 nft_compat [36790.274623] [36790.274623] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36790.279840] nf_tables [36790.283586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36790.285076] nfnetlink bridge [36790.292198] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36790.294557] stp llc [36790.301681] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36790.304645] xfs libcrc32c [36790.311768] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36790.313953] sunrpc intel_rapl_msr [36790.321076] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36790.323780] intel_rapl_common sb_edac [36790.330905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36790.334303] x86_pkg_temp_thermal coretemp [36790.342381] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36790.346125] kvm_intel kvm [36790.351861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36790.355953] irqbypass i2c_i801 [36790.363076] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36790.365780] iTCO_wdt iTCO_vendor_support [36790.372905] Call Trace: [36790.376042] i2c_smbus crct10dif_pclmul [36790.383168] [36790.387169] joydev crc32_pclmul [36790.389618] ? watchdog_timer_fn+0x1b4/0x220 [36790.393444] ipmi_ssif ghash_clmulni_intel [36790.395458] ? __hrtimer_run_queues+0x1d3/0x360 [36790.398680] mei_me mei [36790.402948] ? hrtimer_interrupt+0x104/0x240 [36790.407034] lpc_ich [36790.411559] ? __local_bh_enable+0x50/0x60 [36790.414003] sha512_ssse3 acpi_ipmi [36790.418268] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36790.420451] mfd_core wmi [36790.424542] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36790.428027] ipmi_si ipmi_devintf [36790.433243] [36790.435860] ipmi_msghandler [36790.440819] [36790.440820] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36790.444128] ioatdma [36790.446229] ? queued_spin_lock_slowpath+0x16/0x2e0 [36790.449104] acpi_pad acpi_power_meter [36790.451204] do_raw_spin_lock+0xb5/0xc0 [36790.456505] ixgbe igb [36790.458690] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36790.463560] mdio crc32c_intel [36790.467306] try_get_cap_refs+0x159/0x880 [ceph] [36790.471134] nvme ptp [36790.473496] __ceph_get_caps+0x4a1/0x760 [ceph] [36790.478277] nvme_core i2c_algo_bit [36790.481342] ? __file_remove_privs+0x89/0x110 [36790.485946] pps_core dca [36790.488217] ? wake_bit_function+0x70/0x70 [36790.492742] fuse [36790.496228] ? ceph_write_iter+0x260/0x770 [ceph] [36790.500576] irq event stamp: 214934380 [36790.503194] ceph_write_iter+0x260/0x770 [ceph] [36790.507285] hardirqs last enabled at (214934379): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36790.509211] ? lock_is_held_type+0xaa/0x120 [36790.513906] hardirqs last disabled at (214934380): [] sysvec_apic_timer_interrupt+0xe/0x90 [36790.517653] vfs_write+0x324/0x4f0 [36790.522175] softirqs last enabled at (214934378): [] __do_softirq+0x2be/0x371 [36790.532422] ksys_pwrite64+0x65/0xa0 [36790.536595] softirqs last disabled at (214934373): [] irq_exit_rcu+0xb0/0xd0 [36790.546407] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36790.549802] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36790.558576] do_syscall_64+0x38/0x80 [36790.562146] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36790.570744] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36790.575872] Workqueue: events delayed_work [ceph] [36790.585251] RIP: 0033:0x7f48f0eeb93b [36790.588822] [36790.588823] RIP: 0010:lock_acquire+0xf0/0x2b0 [36790.596293] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36790.601341] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36790.606034] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36790.609604] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36790.611112] ORIG_RAX: 0000000000000012 [36790.615462] [36790.615463] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36790.634202] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36790.652937] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36790.658157] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36790.663373] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36790.667204] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36790.668696] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36790.675820] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36790.682943] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36790.690066] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36790.697191] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36790.704340] [36790.711457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36790.755953] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36790.763078] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36790.770201] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36790.777342] Call Trace: [36790.779786] [36790.781799] ? watchdog_timer_fn+0x1b4/0x220 [36790.786061] ? __hrtimer_run_queues+0x1d3/0x360 [36790.790588] ? hrtimer_interrupt+0x104/0x240 [36790.794857] ? __local_bh_enable+0x50/0x60 [36790.798951] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36790.804168] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36790.809124] [36790.811221] [36790.813320] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36790.818643] ? lock_acquire+0xf0/0x2b0 [36790.822394] ? igrab+0x43/0x50 [36790.825445] ? lock_release+0xc5/0x260 [36790.829190] _raw_spin_lock+0x30/0x40 [36790.832853] ? igrab+0x1d/0x50 [36790.835904] igrab+0x1d/0x50 [36790.838782] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36790.843932] delayed_work+0x1d8/0x280 [ceph] [36790.848213] ? process_scheduled_works+0x22a/0x510 [36790.853005] process_scheduled_works+0x2a4/0x510 [36790.857629] worker_thread+0x188/0x340 [36790.861376] ? rescuer_thread+0x310/0x310 [36790.865381] kthread+0xf5/0x130 [36790.868519] ? kthread_complete_and_exit+0x20/0x20 [36790.873301] ret_from_fork+0x31/0x40 [36790.876873] ? kthread_complete_and_exit+0x20/0x20 [36790.881656] ret_from_fork_asm+0x11/0x20 [36790.885579] [36818.100422] watchdog: BUG: soft lockup - CPU#0 stuck for 23175s! [dbench:78683] [36818.107730] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36818.112421] watchdog: BUG: soft lockup - CPU#7 stuck for 32172s! [kworker/7:4:78808] [36818.163179] irq event stamp: 64535142 [36818.170909] Modules linked in: ceph libceph dns_resolver fscache netfs [36818.174566] hardirqs last enabled at (64535141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36818.174571] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [36818.191251] hardirqs last disabled at (64535142): [] sysvec_apic_timer_interrupt+0xe/0x90 [36818.198634] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [36818.208359] softirqs last enabled at (64535074): [] __do_softirq+0x2be/0x371 [36818.215137] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [36818.223820] softirqs last disabled at (64535069): [] irq_exit_rcu+0xb0/0xd0 [36818.230686] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [36818.239196] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36818.239197] i2c_i801 [36818.239198] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36818.245366] iTCO_wdt iTCO_vendor_support [36818.254310] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36818.256580] i2c_smbus crct10dif_pclmul [36818.264051] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36818.268055] joydev [36818.273532] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36818.277365] crc32_pclmul ipmi_ssif [36818.296104] [36818.298199] ghash_clmulni_intel [36818.303416] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36818.306901] mei_me mei [36818.308392] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36818.311617] lpc_ich [36818.318738] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36818.321184] sha512_ssse3 acpi_ipmi [36818.328307] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36818.330492] mfd_core [36818.337615] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36818.341108] wmi ipmi_si [36818.348233] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36818.350503] ipmi_devintf [36818.357628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36818.360158] ipmi_msghandler [36818.368235] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36818.370854] ioatdma acpi_pad [36818.376590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36818.379470] acpi_power_meter [36818.386593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36818.389555] ixgbe igb [36818.396679] Call Trace: [36818.396682] [36818.399643] mdio crc32c_intel [36818.406771] ? watchdog_timer_fn+0x1b4/0x220 [36818.409126] nvme [36818.411570] ? __hrtimer_run_queues+0x1d3/0x360 [36818.413581] ptp [36818.416635] ? hrtimer_interrupt+0x104/0x240 [36818.420895] nvme_core [36818.422820] ? __local_bh_enable+0x50/0x60 [36818.427344] i2c_algo_bit pps_core [36818.429185] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36818.433445] dca fuse [36818.435819] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36818.439910] [36818.439911] irq event stamp: 215155162 [36818.443308] [36818.448526] hardirqs last enabled at (215155161): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36818.450795] [36818.455770] hardirqs last disabled at (215155162): [] sysvec_apic_timer_interrupt+0xe/0x90 [36818.457262] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36818.461005] softirqs last enabled at (215155160): [] __do_softirq+0x2be/0x371 [36818.463107] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36818.473346] softirqs last disabled at (215155155): [] irq_exit_rcu+0xb0/0xd0 [36818.475446] do_raw_spin_lock+0xb5/0xc0 [36818.485255] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36818.490562] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36818.499329] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36818.504203] try_get_cap_refs+0x159/0x880 [ceph] [36818.512799] Workqueue: events delayed_work [ceph] [36818.516652] __ceph_get_caps+0x4a1/0x760 [ceph] [36818.526023] [36818.530826] ? __file_remove_privs+0x89/0x110 [36818.538297] RIP: 0010:_raw_spin_lock+0x30/0x40 [36818.542907] ? wake_bit_function+0x70/0x70 [36818.547605] Code: 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c 5c ff <48> 89 df 58 5b e9 76 33 5c ff 66 0f 1f 44 00 00 f3 0f 1e fa 0f 1f [36818.552132] ? ceph_write_iter+0x260/0x770 [ceph] [36818.553620] RSP: 0018:ffffc90002117d88 EFLAGS: 00000296 [36818.557970] ceph_write_iter+0x260/0x770 [ceph] [36818.562406] [36818.562407] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [36818.566502] ? lock_is_held_type+0xaa/0x120 [36818.585235] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36818.589935] vfs_write+0x324/0x4f0 [36818.595150] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [36818.599678] ksys_pwrite64+0x65/0xa0 [36818.601165] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [36818.608291] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36818.612467] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36818.619592] do_syscall_64+0x38/0x80 [36818.622987] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36818.630115] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36818.633683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36818.640824] RIP: 0033:0x7f48f0eeb93b [36818.645957] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36818.653080] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36818.656667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36818.664746] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36818.669788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36818.675526] ORIG_RAX: 0000000000000012 [36818.679102] Call Trace: [36818.679104] [36818.686230] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36818.704970] ? watchdog_timer_fn+0x1b4/0x220 [36818.712091] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36818.717311] ? __hrtimer_run_queues+0x1d3/0x360 [36818.724434] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36818.728267] ? hrtimer_interrupt+0x104/0x240 [36818.730709] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36818.732719] ? __local_bh_enable+0x50/0x60 [36818.739843] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36818.744109] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36818.751236] [36818.755756] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36818.797841] [36818.799941] [36818.802038] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36818.807345] ? _raw_spin_lock+0x30/0x40 [36818.811183] ? _raw_spin_lock+0x30/0x40 [36818.815019] ? igrab+0x1d/0x50 [36818.818072] igrab+0x1d/0x50 [36818.820949] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36818.826109] delayed_work+0x1d8/0x280 [ceph] [36818.830389] ? process_scheduled_works+0x22a/0x510 [36818.835178] process_scheduled_works+0x2a4/0x510 [36818.839794] worker_thread+0x188/0x340 [36818.843543] ? rescuer_thread+0x310/0x310 [36818.847547] kthread+0xf5/0x130 [36818.850684] ? kthread_complete_and_exit+0x20/0x20 [36818.855468] ret_from_fork+0x31/0x40 [36818.859049] ? kthread_complete_and_exit+0x20/0x20 [36818.863831] ret_from_fork_asm+0x11/0x20 [36818.867756] [36819.715816] rcu: INFO: rcu_preempt self-detected stall on CPU [36819.721571] rcu: 7-....: (24159762 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/24158409 fqs=12178310 [36819.732255] rcu: (t=24882668 jiffies g=1309609 q=685829 ncpus=8) [36819.738341] Sending NMI from CPU 7 to CPUs 0: [36819.742693] NMI backtrace for cpu 0 [36819.742694] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36819.742696] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36819.742696] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36819.742699] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36819.742701] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36819.742702] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36819.742703] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36819.742703] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36819.742704] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36819.742705] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36819.742706] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36819.742707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36819.742707] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36819.742708] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36819.742709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36819.742710] Call Trace: [36819.742711] [36819.742711] ? nmi_cpu_backtrace+0xad/0x120 [36819.742715] ? nmi_cpu_backtrace_handler+0x11/0x20 [36819.742716] ? nmi_handle+0xd4/0x260 [36819.742720] ? default_do_nmi+0x49/0xf0 [36819.742722] ? exc_nmi+0x11e/0x150 [36819.742723] ? end_repeat_nmi+0x16/0x67 [36819.742728] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36819.742730] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36819.742733] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36819.742735] [36819.742735] [36819.742736] do_raw_spin_lock+0xb5/0xc0 [36819.742738] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36819.742752] try_get_cap_refs+0x159/0x880 [ceph] [36819.742767] __ceph_get_caps+0x4a1/0x760 [ceph] [36819.742779] ? __file_remove_privs+0x89/0x110 [36819.742781] ? wake_bit_function+0x70/0x70 [36819.742784] ? ceph_write_iter+0x260/0x770 [ceph] [36819.742795] ceph_write_iter+0x260/0x770 [ceph] [36819.742806] ? lock_is_held_type+0xaa/0x120 [36819.742808] vfs_write+0x324/0x4f0 [36819.742812] ksys_pwrite64+0x65/0xa0 [36819.742815] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36819.742819] do_syscall_64+0x38/0x80 [36819.742822] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36819.742823] RIP: 0033:0x7f48f0eeb93b [36819.742825] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36819.742826] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [36819.742827] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36819.742828] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36819.742829] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36819.742829] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36819.742830] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36819.742833] [36819.743691] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36820.047183] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36820.054654] Workqueue: events delayed_work [ceph] [36820.059369] RIP: 0010:lock_release+0x32/0x260 [36820.063728] Code: 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e <89> c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 [36820.082466] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [36820.087682] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [36820.094807] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [36820.101930] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36820.109055] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36820.116180] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36820.123303] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36820.131381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36820.137117] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36820.144241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36820.151366] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36820.158489] Call Trace: [36820.160936] [36820.162946] ? rcu_dump_cpu_stacks+0xd1/0x180 [36820.167307] ? rcu_sched_clock_irq+0x633/0x1390 [36820.171837] ? lock_release+0xc5/0x260 [36820.175583] ? lock_is_held_type+0xaa/0x120 [36820.179760] ? lock_is_held_type+0xaa/0x120 [36820.183937] ? tick_sched_do_timer+0xa0/0xa0 [36820.188208] ? rcu_read_lock_sched_held+0x56/0x90 [36820.192909] ? tick_sched_do_timer+0xa0/0xa0 [36820.197179] ? update_process_times+0x5f/0x90 [36820.201530] ? tick_sched_handle.isra.16+0x1e/0x50 [36820.206315] ? tick_sched_timer+0x6f/0x80 [36820.210320] ? __hrtimer_run_queues+0x1d3/0x360 [36820.214847] ? hrtimer_interrupt+0x104/0x240 [36820.219113] ? __local_bh_enable+0x50/0x60 [36820.223209] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36820.228434] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36820.233399] [36820.235495] [36820.237592] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36820.242898] ? igrab+0x43/0x50 [36820.245949] ? igrab+0x43/0x50 [36820.248998] ? lock_release+0x32/0x260 [36820.252742] ? lock_release+0xc5/0x260 [36820.256488] _raw_spin_unlock+0x1b/0x40 [36820.260328] igrab+0x43/0x50 [36820.263203] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36820.268354] delayed_work+0x1d8/0x280 [ceph] [36820.272636] ? process_scheduled_works+0x22a/0x510 [36820.277424] process_scheduled_works+0x2a4/0x510 [36820.282040] worker_thread+0x188/0x340 [36820.285790] ? rescuer_thread+0x310/0x310 [36820.289793] kthread+0xf5/0x130 [36820.292930] ? kthread_complete_and_exit+0x20/0x20 [36820.297714] ret_from_fork+0x31/0x40 [36820.301284] ? kthread_complete_and_exit+0x20/0x20 [36820.306070] ret_from_fork_asm+0x11/0x20 [36820.309992] [36846.100909] watchdog: BUG: soft lockup - CPU#0 stuck for 23201s! [dbench:78683] [36846.108208] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36846.112909] watchdog: BUG: soft lockup - CPU#7 stuck for 32198s! [kworker/7:4:78808] [36846.163649] irq event stamp: 64593194 [36846.163650] hardirqs last enabled at (64593193): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36846.171380] Modules linked in: [36846.175037] hardirqs last disabled at (64593194): [] sysvec_apic_timer_interrupt+0xe/0x90 [36846.185193] ceph [36846.188245] softirqs last enabled at (64593134): [] __do_softirq+0x2be/0x371 [36846.197969] libceph [36846.199894] softirqs last disabled at (64593129): [] irq_exit_rcu+0xb0/0xd0 [36846.208585] dns_resolver fscache [36846.210770] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36846.219281] netfs veth [36846.222611] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36846.231553] nft_chain_nat [36846.233997] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36846.241469] xt_MASQUERADE nf_nat [36846.244173] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36846.249649] nf_conntrack nf_defrag_ipv6 [36846.252961] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36846.271699] nf_defrag_ipv4 nft_compat [36846.275616] [36846.275617] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36846.280834] nf_tables nfnetlink [36846.284586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36846.286077] bridge stp [36846.293202] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36846.296426] llc xfs [36846.303550] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36846.305994] libcrc32c [36846.313118] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36846.315306] sunrpc intel_rapl_msr [36846.322434] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36846.324790] intel_rapl_common sb_edac [36846.331916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36846.335313] x86_pkg_temp_thermal coretemp [36846.343389] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36846.347135] kvm_intel kvm [36846.352873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36846.356961] irqbypass i2c_i801 [36846.364087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36846.366790] iTCO_wdt iTCO_vendor_support [36846.373915] Call Trace: [36846.373917] [36846.377053] i2c_smbus crct10dif_pclmul [36846.384178] ? watchdog_timer_fn+0x1b4/0x220 [36846.388180] joydev crc32_pclmul [36846.390626] ? __hrtimer_run_queues+0x1d3/0x360 [36846.392635] ipmi_ssif ghash_clmulni_intel [36846.396469] ? hrtimer_interrupt+0x104/0x240 [36846.400731] mei_me mei [36846.403953] ? __local_bh_enable+0x50/0x60 [36846.408478] lpc_ich sha512_ssse3 [36846.412571] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36846.416833] acpi_ipmi mfd_core [36846.419278] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36846.423368] wmi [36846.426680] [36846.431897] ipmi_si ipmi_devintf [36846.435033] [36846.435035] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36846.439992] ipmi_msghandler ioatdma [36846.441831] ? queued_spin_lock_slowpath+0x18/0x2e0 [36846.443926] acpi_pad acpi_power_meter [36846.447240] do_raw_spin_lock+0xb5/0xc0 [36846.449333] ixgbe [36846.454638] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36846.458208] igb mdio [36846.463083] try_get_cap_refs+0x159/0x880 [ceph] [36846.466824] crc32c_intel nvme [36846.470659] __ceph_get_caps+0x4a1/0x760 [ceph] [36846.472664] ptp nvme_core [36846.477453] ? __file_remove_privs+0x89/0x110 [36846.479719] i2c_algo_bit pps_core [36846.484329] ? wake_bit_function+0x70/0x70 [36846.487380] dca fuse [36846.491909] ? ceph_write_iter+0x260/0x770 [ceph] [36846.494610] [36846.494610] irq event stamp: 215371154 [36846.498960] ceph_write_iter+0x260/0x770 [ceph] [36846.502357] hardirqs last enabled at (215371153): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36846.506452] ? lock_is_held_type+0xaa/0x120 [36846.508719] hardirqs last disabled at (215371154): [] sysvec_apic_timer_interrupt+0xe/0x90 [36846.513420] vfs_write+0x324/0x4f0 [36846.514907] softirqs last enabled at (215371152): [] __do_softirq+0x2be/0x371 [36846.518656] ksys_pwrite64+0x65/0xa0 [36846.523174] softirqs last disabled at (215371147): [] irq_exit_rcu+0xb0/0xd0 [36846.533419] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36846.537596] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36846.547425] do_syscall_64+0x38/0x80 [36846.550822] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36846.559593] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36846.563164] Workqueue: events delayed_work [ceph] [36846.571761] RIP: 0033:0x7f48f0eeb93b [36846.576891] [36846.576892] RIP: 0010:lock_acquire+0xf0/0x2b0 [36846.586269] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36846.589839] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36846.597312] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36846.602355] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36846.607052] ORIG_RAX: 0000000000000012 [36846.610624] [36846.610624] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36846.612114] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36846.616465] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36846.635203] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36846.653939] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36846.659157] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36846.664375] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36846.668205] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36846.669696] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36846.676821] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36846.683945] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36846.691074] [36846.698192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36846.756937] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36846.764061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36846.771186] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36846.778312] Call Trace: [36846.780755] [36846.782765] ? watchdog_timer_fn+0x1b4/0x220 [36846.787030] ? __hrtimer_run_queues+0x1d3/0x360 [36846.791557] ? hrtimer_interrupt+0x104/0x240 [36846.795824] ? __local_bh_enable+0x50/0x60 [36846.799920] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36846.805141] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36846.810099] [36846.812196] [36846.814311] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36846.819620] ? lock_acquire+0xf0/0x2b0 [36846.823371] ? igrab+0x43/0x50 [36846.826419] ? lock_release+0xc5/0x260 [36846.830165] _raw_spin_lock+0x30/0x40 [36846.833828] ? igrab+0x1d/0x50 [36846.836880] igrab+0x1d/0x50 [36846.839758] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36846.844908] delayed_work+0x1d8/0x280 [ceph] [36846.849189] ? process_scheduled_works+0x22a/0x510 [36846.853979] process_scheduled_works+0x2a4/0x510 [36846.858611] worker_thread+0x188/0x340 [36846.862361] ? rescuer_thread+0x310/0x310 [36846.866365] kthread+0xf5/0x130 [36846.869503] ? kthread_complete_and_exit+0x20/0x20 [36846.874288] ret_from_fork+0x31/0x40 [36846.877881] ? kthread_complete_and_exit+0x20/0x20 [36846.882668] ret_from_fork_asm+0x11/0x20 [36846.886600] [36874.101399] watchdog: BUG: soft lockup - CPU#0 stuck for 23227s! [dbench:78683] [36874.108706] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36874.113399] watchdog: BUG: soft lockup - CPU#7 stuck for 32224s! [kworker/7:4:78808] [36874.164156] irq event stamp: 64651340 [36874.171885] Modules linked in: ceph libceph dns_resolver fscache netfs [36874.175542] hardirqs last enabled at (64651339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36874.182072] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [36874.192226] hardirqs last disabled at (64651340): [] sysvec_apic_timer_interrupt+0xe/0x90 [36874.199610] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [36874.209334] softirqs last enabled at (64651282): [] __do_softirq+0x2be/0x371 [36874.215766] llc xfs libcrc32c sunrpc intel_rapl_msr [36874.224450] softirqs last disabled at (64651277): [] irq_exit_rcu+0xb0/0xd0 [36874.229406] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [36874.237918] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36874.237920] kvm [36874.237921] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36874.245128] irqbypass [36874.254083] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36874.255918] i2c_i801 iTCO_wdt [36874.263389] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36874.265746] iTCO_vendor_support i2c_smbus [36874.271225] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36874.274276] crct10dif_pclmul joydev [36874.293012] [36874.293013] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36874.297103] crc32_pclmul ipmi_ssif [36874.302320] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36874.305890] ghash_clmulni_intel mei_me [36874.307382] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36874.314506] mei [36874.317991] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36874.325116] lpc_ich [36874.328944] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36874.336077] sha512_ssse3 acpi_ipmi [36874.337916] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36874.345039] mfd_core wmi [36874.347224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36874.354348] ipmi_si [36874.357832] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36874.365910] ipmi_devintf [36874.368528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36874.374264] ipmi_msghandler [36874.376449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36874.383574] ioatdma [36874.386190] Call Trace: [36874.386193] [36874.393312] acpi_pad [36874.396193] ? watchdog_timer_fn+0x1b4/0x220 [36874.403314] acpi_power_meter [36874.405501] ? __hrtimer_run_queues+0x1d3/0x360 [36874.407943] ixgbe igb [36874.409957] ? hrtimer_interrupt+0x104/0x240 [36874.412224] mdio crc32c_intel [36874.416490] ? __local_bh_enable+0x50/0x60 [36874.419452] nvme ptp [36874.423980] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36874.426335] nvme_core [36874.430599] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36874.433649] i2c_algo_bit pps_core [36874.437741] [36874.440011] dca fuse [36874.445228] [36874.445229] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36874.447585] [36874.447586] irq event stamp: 215591936 [36874.452546] ? queued_spin_lock_slowpath+0x18/0x2e0 [36874.455940] hardirqs last enabled at (215591935): [] asm_common_interrupt+0x26/0x40 [36874.458041] do_raw_spin_lock+0xb5/0xc0 [36874.460308] hardirqs last disabled at (215591936): [] sysvec_apic_timer_interrupt+0xe/0x90 [36874.462407] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36874.467711] softirqs last enabled at (215591934): [] __do_softirq+0x2be/0x371 [36874.469202] try_get_cap_refs+0x159/0x880 [ceph] [36874.472945] softirqs last disabled at (215591929): [] irq_exit_rcu+0xb0/0xd0 [36874.477821] __ceph_get_caps+0x4a1/0x760 [ceph] [36874.487106] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36874.490942] ? __file_remove_privs+0x89/0x110 [36874.500748] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36874.505533] ? wake_bit_function+0x70/0x70 [36874.514303] Workqueue: events delayed_work [ceph] [36874.518916] ? ceph_write_iter+0x260/0x770 [ceph] [36874.527511] [36874.532036] ceph_write_iter+0x260/0x770 [ceph] [36874.541412] RIP: 0010:lock_acquire+0xf0/0x2b0 [36874.545768] ? lock_is_held_type+0xaa/0x120 [36874.553235] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36874.557328] vfs_write+0x324/0x4f0 [36874.562022] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36874.566723] ksys_pwrite64+0x65/0xa0 [36874.568211] [36874.568212] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36874.572737] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36874.577092] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36874.581272] do_syscall_64+0x38/0x80 [36874.600009] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36874.603407] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36874.608623] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36874.612194] RIP: 0033:0x7f48f0eeb93b [36874.613685] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36874.620810] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36874.625939] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36874.633064] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36874.636636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36874.643760] ORIG_RAX: 0000000000000012 [36874.648804] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36874.655928] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36874.659498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36874.666623] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36874.685359] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36874.693438] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36874.698654] Call Trace: [36874.698655] [36874.704391] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36874.708225] ? watchdog_timer_fn+0x1b4/0x220 [36874.715346] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36874.722472] ? __hrtimer_run_queues+0x1d3/0x360 [36874.729601] [36874.736724] ? hrtimer_interrupt+0x104/0x240 [36874.784906] ? __local_bh_enable+0x50/0x60 [36874.789001] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36874.794224] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36874.799181] [36874.801279] [36874.803376] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36874.808684] ? lock_acquire+0xf0/0x2b0 [36874.812435] ? igrab+0x43/0x50 [36874.815486] ? lock_release+0xc5/0x260 [36874.819231] _raw_spin_lock+0x30/0x40 [36874.822895] ? igrab+0x1d/0x50 [36874.825946] igrab+0x1d/0x50 [36874.828825] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36874.833973] delayed_work+0x1d8/0x280 [ceph] [36874.838255] ? process_scheduled_works+0x22a/0x510 [36874.843045] process_scheduled_works+0x2a4/0x510 [36874.847661] worker_thread+0x188/0x340 [36874.851409] ? rescuer_thread+0x310/0x310 [36874.855413] kthread+0xf5/0x130 [36874.858560] ? kthread_complete_and_exit+0x20/0x20 [36874.863342] ret_from_fork+0x31/0x40 [36874.866912] ? kthread_complete_and_exit+0x20/0x20 [36874.871697] ret_from_fork_asm+0x11/0x20 [36874.875620] [36902.101893] watchdog: BUG: soft lockup - CPU#0 stuck for 23253s! [dbench:78683] [36902.109193] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36902.113893] watchdog: BUG: soft lockup - CPU#7 stuck for 32250s! [kworker/7:4:78808] [36902.164633] irq event stamp: 64709418 [36902.172364] Modules linked in: ceph libceph [36902.176021] hardirqs last enabled at (64709417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36902.180198] dns_resolver fscache netfs veth [36902.190356] hardirqs last disabled at (64709418): [] sysvec_apic_timer_interrupt+0xe/0x90 [36902.194621] nft_chain_nat xt_MASQUERADE [36902.204344] softirqs last enabled at (64709392): [] __do_softirq+0x2be/0x371 [36902.204345] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [36902.216963] softirqs last disabled at (64709387): [] irq_exit_rcu+0xb0/0xd0 [36902.223740] nf_tables nfnetlink bridge stp [36902.232252] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36902.232253] llc [36902.232254] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36902.236428] xfs libcrc32c [36902.245374] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36902.247211] sunrpc intel_rapl_msr [36902.254681] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36902.257386] intel_rapl_common sb_edac [36902.262864] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36902.266261] x86_pkg_temp_thermal coretemp [36902.285014] [36902.285015] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36902.288759] kvm_intel kvm [36902.293977] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36902.298067] irqbypass i2c_i801 [36902.299574] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36902.306700] iTCO_wdt iTCO_vendor_support [36902.309405] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36902.316527] i2c_smbus crct10dif_pclmul [36902.319665] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36902.326790] joydev crc32_pclmul [36902.330795] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36902.337917] ipmi_ssif ghash_clmulni_intel [36902.341749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36902.348873] mei_me mei [36902.352096] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36902.360176] lpc_ich [36902.364265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36902.370019] sha512_ssse3 [36902.372464] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36902.379589] acpi_ipmi [36902.381773] Call Trace: [36902.381774] [36902.388897] mfd_core [36902.391516] ? watchdog_timer_fn+0x1b4/0x220 [36902.398638] wmi [36902.400995] ? __hrtimer_run_queues+0x1d3/0x360 [36902.403440] ipmi_si ipmi_devintf [36902.405453] ? hrtimer_interrupt+0x104/0x240 [36902.407720] ipmi_msghandler ioatdma [36902.411985] ? __local_bh_enable+0x50/0x60 [36902.413823] acpi_pad acpi_power_meter [36902.418348] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36902.421657] ixgbe igb [36902.425921] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36902.429492] mdio crc32c_intel [36902.433598] [36902.437344] nvme [36902.442570] [36902.444928] ptp nvme_core [36902.449886] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36902.452936] i2c_algo_bit pps_core [36902.455037] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36902.456956] dca [36902.459056] do_raw_spin_lock+0xb5/0xc0 [36902.461759] fuse [36902.467063] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36902.470459] irq event stamp: 215812674 [36902.475333] try_get_cap_refs+0x159/0x880 [ceph] [36902.477167] hardirqs last enabled at (215812673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36902.481004] __ceph_get_caps+0x4a1/0x760 [ceph] [36902.482923] hardirqs last disabled at (215812674): [] sysvec_apic_timer_interrupt+0xe/0x90 [36902.487711] ? __file_remove_privs+0x89/0x110 [36902.491450] softirqs last enabled at (215812672): [] __do_softirq+0x2be/0x371 [36902.496063] ? wake_bit_function+0x70/0x70 [36902.506307] softirqs last disabled at (215812667): [] irq_exit_rcu+0xb0/0xd0 [36902.510831] ? ceph_write_iter+0x260/0x770 [ceph] [36902.520641] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36902.524993] ceph_write_iter+0x260/0x770 [ceph] [36902.533763] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36902.537855] ? lock_is_held_type+0xaa/0x120 [36902.546450] Workqueue: events delayed_work [ceph] [36902.551151] vfs_write+0x324/0x4f0 [36902.560524] [36902.565054] ksys_pwrite64+0x65/0xa0 [36902.572519] RIP: 0010:lock_acquire+0xf0/0x2b0 [36902.576698] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36902.581394] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36902.584793] do_syscall_64+0x38/0x80 [36902.586294] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36902.589863] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36902.594214] [36902.594215] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36902.599344] RIP: 0033:0x7f48f0eeb93b [36902.618081] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36902.621653] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36902.626869] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36902.631914] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36902.633403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36902.640529] ORIG_RAX: 0000000000000012 [36902.644099] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36902.651222] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36902.669962] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36902.677085] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36902.682303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36902.689427] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36902.693258] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36902.700380] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36902.707507] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36902.715585] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36902.722708] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36902.728450] [36902.735585] Call Trace: [36902.735587] [36902.777848] ? watchdog_timer_fn+0x1b4/0x220 [36902.782111] ? __hrtimer_run_queues+0x1d3/0x360 [36902.786638] ? hrtimer_interrupt+0x104/0x240 [36902.790908] ? __local_bh_enable+0x50/0x60 [36902.795002] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36902.800225] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36902.805182] [36902.807296] [36902.809393] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36902.814701] ? lock_acquire+0xf0/0x2b0 [36902.818452] ? igrab+0x43/0x50 [36902.821502] ? lock_release+0xc5/0x260 [36902.825248] _raw_spin_lock+0x30/0x40 [36902.828910] ? igrab+0x1d/0x50 [36902.831961] igrab+0x1d/0x50 [36902.834842] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36902.839992] delayed_work+0x1d8/0x280 [ceph] [36902.844272] ? process_scheduled_works+0x22a/0x510 [36902.849079] process_scheduled_works+0x2a4/0x510 [36902.853695] worker_thread+0x188/0x340 [36902.857444] ? rescuer_thread+0x310/0x310 [36902.861447] kthread+0xf5/0x130 [36902.864585] ? kthread_complete_and_exit+0x20/0x20 [36902.869368] ret_from_fork+0x31/0x40 [36902.872938] ? kthread_complete_and_exit+0x20/0x20 [36902.877724] ret_from_fork_asm+0x11/0x20 [36902.881647] [36930.102387] watchdog: BUG: soft lockup - CPU#0 stuck for 23279s! [dbench:78683] [36930.109688] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36930.114387] watchdog: BUG: soft lockup - CPU#7 stuck for 32276s! [kworker/7:4:78808] [36930.165130] irq event stamp: 64767546 [36930.172861] Modules linked in: ceph [36930.176519] hardirqs last enabled at (64767545): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36930.176521] libceph dns_resolver fscache netfs [36930.190161] hardirqs last disabled at (64767546): [] sysvec_apic_timer_interrupt+0xe/0x90 [36930.194685] veth nft_chain_nat xt_MASQUERADE nf_nat [36930.204407] softirqs last enabled at (64767510): [] __do_softirq+0x2be/0x371 [36930.209366] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [36930.218062] softirqs last disabled at (64767505): [] irq_exit_rcu+0xb0/0xd0 [36930.218063] nft_compat [36930.223285] nf_tables [36930.231796] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36930.234238] nfnetlink bridge [36930.236597] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36930.236598] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [36930.245540] stp llc [36930.248504] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [36930.255975] xfs libcrc32c [36930.261454] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36930.263636] sunrpc intel_rapl_msr [36930.282375] [36930.282376] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36930.285077] intel_rapl_common [36930.290297] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36930.293692] sb_edac x86_pkg_temp_thermal [36930.295184] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36930.302307] coretemp kvm_intel [36930.305358] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36930.312484] kvm irqbypass [36930.316488] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36930.323612] i2c_i801 iTCO_wdt [36930.326750] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36930.333874] iTCO_vendor_support i2c_smbus [36930.336576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36930.343700] crct10dif_pclmul [36930.346751] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36930.354829] joydev crc32_pclmul [36930.358920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36930.364658] ipmi_ssif ghash_clmulni_intel [36930.367622] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36930.374746] mei_me mei [36930.377970] Call Trace: [36930.377971] [36930.385093] lpc_ich [36930.389185] ? watchdog_timer_fn+0x1b4/0x220 [36930.396309] sha512_ssse3 acpi_ipmi [36930.398753] ? __hrtimer_run_queues+0x1d3/0x360 [36930.401197] mfd_core wmi [36930.403211] ? hrtimer_interrupt+0x104/0x240 [36930.405392] ipmi_si ipmi_devintf [36930.409655] ? __local_bh_enable+0x50/0x60 [36930.413140] ipmi_msghandler ioatdma [36930.417666] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36930.420282] acpi_pad acpi_power_meter [36930.424546] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36930.427855] ixgbe [36930.431946] [36930.435517] igb [36930.440735] [36930.440737] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36930.444479] mdio crc32c_intel [36930.449441] ? queued_spin_lock_slowpath+0x16/0x2e0 [36930.451446] nvme ptp [36930.453548] do_raw_spin_lock+0xb5/0xc0 [36930.455383] nvme_core i2c_algo_bit [36930.457480] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36930.462787] pps_core dca [36930.465836] try_get_cap_refs+0x159/0x880 [ceph] [36930.470704] fuse [36930.472980] __ceph_get_caps+0x4a1/0x760 [ceph] [36930.476807] irq event stamp: 216033398 [36930.480294] ? __file_remove_privs+0x89/0x110 [36930.485075] hardirqs last enabled at (216033397): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36930.487693] ? wake_bit_function+0x70/0x70 [36930.492303] hardirqs last disabled at (216033398): [] sysvec_apic_timer_interrupt+0xe/0x90 [36930.494229] ? ceph_write_iter+0x260/0x770 [ceph] [36930.498750] softirqs last enabled at (216033396): [] __do_softirq+0x2be/0x371 [36930.502495] ceph_write_iter+0x260/0x770 [ceph] [36930.506846] softirqs last disabled at (216033391): [] irq_exit_rcu+0xb0/0xd0 [36930.517093] ? lock_is_held_type+0xaa/0x120 [36930.521180] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36930.530993] vfs_write+0x324/0x4f0 [36930.535690] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36930.544463] ksys_pwrite64+0x65/0xa0 [36930.548983] Workqueue: events delayed_work [ceph] [36930.557581] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36930.561759] [36930.561760] RIP: 0010:lock_acquire+0xf0/0x2b0 [36930.571137] do_syscall_64+0x38/0x80 [36930.574533] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36930.582006] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36930.585576] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36930.590272] RIP: 0033:0x7f48f0eeb93b [36930.595403] [36930.595404] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36930.596893] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36930.601244] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36930.604816] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36930.623553] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36930.628597] ORIG_RAX: 0000000000000012 [36930.633814] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36930.637385] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36930.638875] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36930.646000] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36930.664738] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36930.671860] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36930.677080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36930.684203] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36930.688033] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36930.695159] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36930.702282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36930.709411] [36930.716530] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36930.775275] Call Trace: [36930.777720] [36930.779730] ? watchdog_timer_fn+0x1b4/0x220 [36930.783996] ? __hrtimer_run_queues+0x1d3/0x360 [36930.788520] ? hrtimer_interrupt+0x104/0x240 [36930.792794] ? __local_bh_enable+0x50/0x60 [36930.796892] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36930.802117] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36930.807073] [36930.809172] [36930.811268] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36930.816575] ? lock_acquire+0xf0/0x2b0 [36930.820326] ? igrab+0x43/0x50 [36930.823376] ? lock_release+0xc5/0x260 [36930.827120] _raw_spin_lock+0x30/0x40 [36930.830795] ? igrab+0x1d/0x50 [36930.833845] igrab+0x1d/0x50 [36930.836722] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36930.841873] delayed_work+0x1d8/0x280 [ceph] [36930.846153] ? process_scheduled_works+0x22a/0x510 [36930.850936] process_scheduled_works+0x2a4/0x510 [36930.855551] worker_thread+0x188/0x340 [36930.859300] ? rescuer_thread+0x310/0x310 [36930.863303] kthread+0xf5/0x130 [36930.866441] ? kthread_complete_and_exit+0x20/0x20 [36930.871226] ret_from_fork+0x31/0x40 [36930.874796] ? kthread_complete_and_exit+0x20/0x20 [36930.879582] ret_from_fork_asm+0x11/0x20 [36930.883502] [36958.102876] watchdog: BUG: soft lockup - CPU#0 stuck for 23305s! [dbench:78683] [36958.110178] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36958.114876] watchdog: BUG: soft lockup - CPU#7 stuck for 32302s! [kworker/7:4:78808] [36958.165627] irq event stamp: 64825688 [36958.173358] Modules linked in: ceph libceph dns_resolver fscache netfs [36958.177014] hardirqs last enabled at (64825687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36958.183531] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [36958.193689] hardirqs last disabled at (64825688): [] sysvec_apic_timer_interrupt+0xe/0x90 [36958.193692] nf_defrag_ipv6 [36958.193693] softirqs last enabled at (64825630): [] __do_softirq+0x2be/0x371 [36958.199773] nf_defrag_ipv4 nft_compat [36958.209498] softirqs last disabled at (64825625): [] irq_exit_rcu+0xb0/0xd0 [36958.212289] nf_tables [36958.220976] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36958.224717] nfnetlink bridge [36958.233229] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36958.235585] stp [36958.244530] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [36958.247494] llc [36958.254966] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [36958.256801] xfs libcrc32c [36958.262278] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36958.264116] sunrpc intel_rapl_msr [36958.282853] [36958.285558] intel_rapl_common sb_edac [36958.290774] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36958.294173] x86_pkg_temp_thermal coretemp [36958.295663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36958.299409] kvm_intel kvm [36958.306532] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36958.310639] irqbypass i2c_i801 [36958.317763] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36958.320469] iTCO_wdt iTCO_vendor_support [36958.327591] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36958.330729] i2c_smbus crct10dif_pclmul [36958.337853] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36958.341857] joydev crc32_pclmul [36958.348981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36958.352813] ipmi_ssif ghash_clmulni_intel [36958.360889] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36958.364113] mei_me [36958.369852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36958.373943] mei lpc_ich [36958.381067] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36958.383163] sha512_ssse3 acpi_ipmi [36958.390287] Call Trace: [36958.392818] mfd_core wmi [36958.399946] [36958.403426] ipmi_si ipmi_devintf [36958.405873] ? watchdog_timer_fn+0x1b4/0x220 [36958.408488] ipmi_msghandler [36958.410500] ? __hrtimer_run_queues+0x1d3/0x360 [36958.413811] ioatdma acpi_pad [36958.418077] ? hrtimer_interrupt+0x104/0x240 [36958.420951] acpi_power_meter [36958.425475] ? __local_bh_enable+0x50/0x60 [36958.428439] ixgbe [36958.432706] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36958.435667] igb [36958.439759] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36958.441768] mdio crc32c_intel [36958.446986] [36958.448824] nvme ptp [36958.453780] [36958.456833] nvme_core [36958.458930] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36958.461200] i2c_algo_bit pps_core [36958.463300] ? queued_spin_lock_slowpath+0x2d/0x2e0 [36958.465655] dca fuse [36958.470959] do_raw_spin_lock+0xb5/0xc0 [36958.474356] [36958.474357] irq event stamp: 216254050 [36958.479229] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36958.481499] hardirqs last enabled at (216254049): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36958.485332] try_get_cap_refs+0x159/0x880 [ceph] [36958.486818] hardirqs last disabled at (216254050): [] sysvec_apic_timer_interrupt+0xe/0x90 [36958.490572] __ceph_get_caps+0x4a1/0x760 [ceph] [36958.495348] softirqs last enabled at (216254048): [] __do_softirq+0x2be/0x371 [36958.505596] ? __file_remove_privs+0x89/0x110 [36958.510201] softirqs last disabled at (216254043): [] irq_exit_rcu+0xb0/0xd0 [36958.520014] ? wake_bit_function+0x70/0x70 [36958.524537] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36958.533311] ? ceph_write_iter+0x260/0x770 [ceph] [36958.537659] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36958.546274] ceph_write_iter+0x260/0x770 [ceph] [36958.550365] Workqueue: events delayed_work [ceph] [36958.559746] ? lock_is_held_type+0xaa/0x120 [36958.564439] [36958.571913] vfs_write+0x324/0x4f0 [36958.576435] RIP: 0010:lock_acquire+0xf0/0x2b0 [36958.581135] ksys_pwrite64+0x65/0xa0 [36958.585308] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [36958.586801] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36958.590196] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [36958.594548] do_syscall_64+0x38/0x80 [36958.598117] [36958.598118] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [36958.616857] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36958.621987] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [36958.627206] RIP: 0033:0x7f48f0eeb93b [36958.630776] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [36958.632279] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36958.639408] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [36958.644452] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36958.651576] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [36958.655146] ORIG_RAX: 0000000000000012 [36958.662279] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36958.681016] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36958.688141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36958.693358] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36958.700482] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36958.704311] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36958.712391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36958.719514] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36958.725252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36958.732394] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36958.739516] Call Trace: [36958.739518] [36958.746648] [36958.753768] ? watchdog_timer_fn+0x1b4/0x220 [36958.786043] ? __hrtimer_run_queues+0x1d3/0x360 [36958.790569] ? hrtimer_interrupt+0x104/0x240 [36958.794839] ? __local_bh_enable+0x50/0x60 [36958.798930] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36958.804147] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36958.809105] [36958.811200] [36958.813299] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36958.818607] ? lock_acquire+0xf0/0x2b0 [36958.822356] ? igrab+0x43/0x50 [36958.825407] ? lock_release+0xc5/0x260 [36958.829151] _raw_spin_lock+0x30/0x40 [36958.832810] ? igrab+0x1d/0x50 [36958.835858] igrab+0x1d/0x50 [36958.838739] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36958.843888] delayed_work+0x1d8/0x280 [ceph] [36958.848170] ? process_scheduled_works+0x22a/0x510 [36958.852959] process_scheduled_works+0x2a4/0x510 [36958.857573] worker_thread+0x188/0x340 [36958.861324] ? rescuer_thread+0x310/0x310 [36958.865333] kthread+0xf5/0x130 [36958.868472] ? kthread_complete_and_exit+0x20/0x20 [36958.873259] ret_from_fork+0x31/0x40 [36958.876854] ? kthread_complete_and_exit+0x20/0x20 [36958.881637] ret_from_fork_asm+0x11/0x20 [36958.885572] [36986.103365] watchdog: BUG: soft lockup - CPU#0 stuck for 23331s! [dbench:78683] [36986.110665] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [36986.115365] watchdog: BUG: soft lockup - CPU#7 stuck for 32329s! [kworker/7:4:78808] [36986.166106] irq event stamp: 64883742 [36986.166107] hardirqs last enabled at (64883741): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36986.173837] Modules linked in: [36986.177495] hardirqs last disabled at (64883742): [] sysvec_apic_timer_interrupt+0xe/0x90 [36986.187652] ceph [36986.190701] softirqs last enabled at (64883684): [] __do_softirq+0x2be/0x371 [36986.200425] libceph dns_resolver [36986.202349] softirqs last disabled at (64883679): [] irq_exit_rcu+0xb0/0xd0 [36986.211038] fscache netfs [36986.214353] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36986.222865] veth nft_chain_nat [36986.225568] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36986.234512] xt_MASQUERADE nf_nat [36986.237650] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [36986.245121] nf_conntrack nf_defrag_ipv6 [36986.248433] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [36986.253909] nf_defrag_ipv4 nft_compat [36986.257827] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [36986.276565] nf_tables nfnetlink [36986.280309] [36986.280310] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [36986.285527] bridge stp [36986.288762] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [36986.290248] llc xfs [36986.297372] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [36986.299816] libcrc32c sunrpc [36986.306942] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [36986.309124] intel_rapl_msr intel_rapl_common [36986.316249] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [36986.319212] sb_edac x86_pkg_temp_thermal [36986.326338] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [36986.330688] coretemp kvm_intel [36986.337813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36986.341816] kvm irqbypass [36986.349894] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [36986.353045] i2c_i801 [36986.358787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36986.361490] iTCO_wdt iTCO_vendor_support [36986.368615] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36986.370884] i2c_smbus [36986.378010] Call Trace: [36986.378011] [36986.382015] crct10dif_pclmul [36986.389138] ? watchdog_timer_fn+0x1b4/0x220 [36986.391495] joydev crc32_pclmul [36986.393939] ? __hrtimer_run_queues+0x1d3/0x360 [36986.395950] ipmi_ssif ghash_clmulni_intel [36986.398918] ? hrtimer_interrupt+0x104/0x240 [36986.403177] mei_me mei [36986.406401] ? __local_bh_enable+0x50/0x60 [36986.410926] lpc_ich sha512_ssse3 [36986.415019] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36986.419280] acpi_ipmi mfd_core [36986.421726] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36986.425815] wmi ipmi_si [36986.429125] [36986.434343] ipmi_devintf ipmi_msghandler [36986.437482] [36986.442437] ioatdma acpi_pad [36986.444971] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36986.447066] acpi_power_meter [36986.451073] ? queued_spin_lock_slowpath+0x18/0x2e0 [36986.453169] ixgbe igb [36986.456135] do_raw_spin_lock+0xb5/0xc0 [36986.461436] mdio [36986.464401] __cap_delay_requeue+0xa7/0x1d0 [ceph] [36986.469270] crc32c_intel nvme [36986.471632] try_get_cap_refs+0x159/0x880 [ceph] [36986.475460] ptp nvme_core [36986.477389] __ceph_get_caps+0x4a1/0x760 [ceph] [36986.482168] i2c_algo_bit [36986.485221] ? __file_remove_privs+0x89/0x110 [36986.489827] pps_core dca [36986.492534] ? wake_bit_function+0x70/0x70 [36986.497056] fuse [36986.499677] ? ceph_write_iter+0x260/0x770 [ceph] [36986.504025] irq event stamp: 216474806 [36986.504042] hardirqs last enabled at (216474805): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36986.506659] ceph_write_iter+0x260/0x770 [ceph] [36986.510766] hardirqs last disabled at (216474806): [] sysvec_apic_timer_interrupt+0xe/0x90 [36986.512696] ? lock_is_held_type+0xaa/0x120 [36986.517389] softirqs last enabled at (216474804): [] __do_softirq+0x2be/0x371 [36986.521136] vfs_write+0x324/0x4f0 [36986.531377] softirqs last disabled at (216474799): [] irq_exit_rcu+0xb0/0xd0 [36986.535906] ksys_pwrite64+0x65/0xa0 [36986.545712] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [36986.549891] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [36986.558659] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [36986.562060] do_syscall_64+0x38/0x80 [36986.570654] Workqueue: events delayed_work [ceph] [36986.574227] entry_SYSCALL_64_after_hwframe+0x63/0xcd [36986.583604] [36986.588736] RIP: 0033:0x7f48f0eeb93b [36986.596205] RIP: 0010:lock_release+0x78/0x260 [36986.599776] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [36986.604475] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [36986.609517] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [36986.611008] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [36986.614579] ORIG_RAX: 0000000000000012 [36986.618931] [36986.618931] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [36986.637668] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [36986.656404] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [36986.661623] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [36986.666840] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [36986.670672] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [36986.672162] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [36986.679285] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [36986.686410] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [36986.693533] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [36986.700657] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [36986.707787] [36986.714905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [36986.759421] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [36986.766544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [36986.773667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [36986.780792] Call Trace: [36986.783237] [36986.785247] ? watchdog_timer_fn+0x1b4/0x220 [36986.789512] ? __hrtimer_run_queues+0x1d3/0x360 [36986.794053] ? hrtimer_interrupt+0x104/0x240 [36986.798316] ? __local_bh_enable+0x50/0x60 [36986.802409] ? __sysvec_apic_timer_interrupt+0x61/0x160 [36986.807625] ? sysvec_apic_timer_interrupt+0x6d/0x90 [36986.812583] [36986.814679] [36986.816776] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [36986.822080] ? igrab+0x43/0x50 [36986.825133] ? lock_release+0x78/0x260 [36986.828880] _raw_spin_unlock+0x1b/0x40 [36986.832717] igrab+0x43/0x50 [36986.835594] ceph_check_delayed_caps+0xb7/0x270 [ceph] [36986.840744] delayed_work+0x1d8/0x280 [ceph] [36986.845042] ? process_scheduled_works+0x22a/0x510 [36986.849850] process_scheduled_works+0x2a4/0x510 [36986.854465] worker_thread+0x188/0x340 [36986.858214] ? rescuer_thread+0x310/0x310 [36986.862216] kthread+0xf5/0x130 [36986.865354] ? kthread_complete_and_exit+0x20/0x20 [36986.870140] ret_from_fork+0x31/0x40 [36986.873710] ? kthread_complete_and_exit+0x20/0x20 [36986.878493] ret_from_fork_asm+0x11/0x20 [36986.882416] [37014.103856] watchdog: BUG: soft lockup - CPU#0 stuck for 23358s! [dbench:78683] [37014.111161] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37014.115856] watchdog: BUG: soft lockup - CPU#7 stuck for 32355s! [kworker/7:4:78808] [37014.166610] irq event stamp: 64941866 [37014.174341] Modules linked in: ceph libceph dns_resolver fscache netfs [37014.177997] hardirqs last enabled at (64941865): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37014.184530] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [37014.194691] hardirqs last disabled at (64941866): [] sysvec_apic_timer_interrupt+0xe/0x90 [37014.202075] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [37014.211798] softirqs last enabled at (64941808): [] __do_softirq+0x2be/0x371 [37014.211802] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37014.227260] softirqs last disabled at (64941803): [] irq_exit_rcu+0xb0/0xd0 [37014.235945] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [37014.244458] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37014.250376] iTCO_vendor_support i2c_smbus crct10dif_pclmul [37014.259319] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37014.259320] joydev crc32_pclmul [37014.264885] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [37014.272353] ipmi_ssif ghash_clmulni_intel [37014.275578] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [37014.281055] mei_me [37014.285147] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37014.303884] mei lpc_ich [37014.305982] [37014.311198] sha512_ssse3 acpi_ipmi [37014.313728] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37014.315220] mfd_core wmi [37014.318703] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37014.325827] ipmi_si [37014.328447] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37014.335571] ipmi_devintf [37014.337754] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37014.344879] ipmi_msghandler ioatdma [37014.347495] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37014.354619] acpi_pad acpi_power_meter [37014.358190] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37014.365315] ixgbe [37014.369059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37014.377135] igb [37014.379148] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37014.384884] mdio crc32c_intel [37014.386723] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37014.393845] nvme [37014.396896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37014.404021] ptp nvme_core [37014.405946] Call Trace: [37014.405948] [37014.413070] i2c_algo_bit pps_core [37014.415775] ? watchdog_timer_fn+0x1b4/0x220 [37014.418216] dca fuse [37014.420231] ? __hrtimer_run_queues+0x1d3/0x360 [37014.423643] [37014.423644] irq event stamp: 216695668 [37014.427910] ? hrtimer_interrupt+0x104/0x240 [37014.430177] hardirqs last enabled at (216695667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37014.434702] ? __local_bh_enable+0x50/0x60 [37014.436191] hardirqs last disabled at (216695668): [] sysvec_apic_timer_interrupt+0xe/0x90 [37014.439939] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37014.444200] softirqs last enabled at (216695666): [] __do_softirq+0x2be/0x371 [37014.454446] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37014.458536] softirqs last disabled at (216695661): [] irq_exit_rcu+0xb0/0xd0 [37014.468347] [37014.473563] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37014.482334] [37014.482336] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37014.487292] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37014.495893] ? queued_spin_lock_slowpath+0x2b/0x2e0 [37014.497988] Workqueue: events delayed_work [ceph] [37014.507365] do_raw_spin_lock+0xb5/0xc0 [37014.509462] [37014.509462] RIP: 0010:lock_acquire+0xf0/0x2b0 [37014.514768] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37014.522254] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37014.527127] try_get_cap_refs+0x159/0x880 [ceph] [37014.531823] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37014.535656] __ceph_get_caps+0x4a1/0x760 [ceph] [37014.537144] [37014.541499] ? __file_remove_privs+0x89/0x110 [37014.546278] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37014.565017] ? wake_bit_function+0x70/0x70 [37014.569625] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37014.574847] ? ceph_write_iter+0x260/0x770 [ceph] [37014.579367] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37014.580858] ceph_write_iter+0x260/0x770 [ceph] [37014.585209] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37014.592337] ? lock_is_held_type+0xaa/0x120 [37014.596424] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37014.603552] vfs_write+0x324/0x4f0 [37014.608258] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37014.615383] ksys_pwrite64+0x65/0xa0 [37014.619903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37014.627028] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37014.631203] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37014.638329] do_syscall_64+0x38/0x80 [37014.641726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37014.649804] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37014.653374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37014.659112] RIP: 0033:0x7f48f0eeb93b [37014.664260] Call Trace: [37014.664261] [37014.671383] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37014.674956] ? watchdog_timer_fn+0x1b4/0x220 [37014.682080] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37014.687124] ? __hrtimer_run_queues+0x1d3/0x360 [37014.694259] ORIG_RAX: 0000000000000012 [37014.697830] ? hrtimer_interrupt+0x104/0x240 [37014.700270] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37014.702281] ? __local_bh_enable+0x50/0x60 [37014.721019] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37014.725286] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37014.730499] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37014.735024] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37014.738854] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37014.743120] [37014.750260] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37014.754351] [37014.754353] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37014.761480] [37014.766698] ? lock_acquire+0xf0/0x2b0 [37014.808469] ? igrab+0x43/0x50 [37014.811544] ? lock_release+0xc5/0x260 [37014.815289] _raw_spin_lock+0x30/0x40 [37014.818946] ? igrab+0x1d/0x50 [37014.821997] igrab+0x1d/0x50 [37014.824876] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37014.830024] delayed_work+0x1d8/0x280 [ceph] [37014.834306] ? process_scheduled_works+0x22a/0x510 [37014.839097] process_scheduled_works+0x2a4/0x510 [37014.843711] worker_thread+0x188/0x340 [37014.847462] ? rescuer_thread+0x310/0x310 [37014.851473] kthread+0xf5/0x130 [37014.854618] ? kthread_complete_and_exit+0x20/0x20 [37014.859404] ret_from_fork+0x31/0x40 [37014.862983] ? kthread_complete_and_exit+0x20/0x20 [37014.867767] ret_from_fork_asm+0x11/0x20 [37014.871697] [37015.317245] rcu: INFO: rcu_preempt self-detected stall on CPU [37015.323001] rcu: 7-....: (24349390 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/24348036 fqs=12274198 [37015.333685] rcu: (t=25078266 jiffies g=1309609 q=689668 ncpus=8) [37015.339771] Sending NMI from CPU 7 to CPUs 0: [37015.344122] NMI backtrace for cpu 0 [37015.344124] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37015.344125] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37015.344126] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37015.344128] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37015.344130] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37015.344131] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37015.344132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37015.344133] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37015.344133] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37015.344134] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37015.344135] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37015.344136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37015.344137] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37015.344138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37015.344139] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37015.344139] Call Trace: [37015.344140] [37015.344141] ? nmi_cpu_backtrace+0xad/0x120 [37015.344144] ? nmi_cpu_backtrace_handler+0x11/0x20 [37015.344146] ? nmi_handle+0xd4/0x260 [37015.344150] ? default_do_nmi+0x49/0xf0 [37015.344152] ? exc_nmi+0x11e/0x150 [37015.344153] ? end_repeat_nmi+0x16/0x67 [37015.344157] ? queued_spin_lock_slowpath+0x18/0x2e0 [37015.344160] ? queued_spin_lock_slowpath+0x18/0x2e0 [37015.344162] ? queued_spin_lock_slowpath+0x18/0x2e0 [37015.344165] [37015.344165] [37015.344166] do_raw_spin_lock+0xb5/0xc0 [37015.344168] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37015.344182] try_get_cap_refs+0x159/0x880 [ceph] [37015.344196] __ceph_get_caps+0x4a1/0x760 [ceph] [37015.344209] ? __file_remove_privs+0x89/0x110 [37015.344211] ? wake_bit_function+0x70/0x70 [37015.344213] ? ceph_write_iter+0x260/0x770 [ceph] [37015.344224] ceph_write_iter+0x260/0x770 [ceph] [37015.344235] ? lock_is_held_type+0xaa/0x120 [37015.344238] vfs_write+0x324/0x4f0 [37015.344242] ksys_pwrite64+0x65/0xa0 [37015.344245] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37015.344249] do_syscall_64+0x38/0x80 [37015.344251] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37015.344253] RIP: 0033:0x7f48f0eeb93b [37015.344254] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37015.344255] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [37015.344256] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37015.344257] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37015.344258] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37015.344258] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37015.344259] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37015.344262] [37015.345121] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37015.648613] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37015.656082] Workqueue: events delayed_work [ceph] [37015.660798] RIP: 0010:lock_acquire+0xf0/0x2b0 [37015.665148] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37015.683885] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37015.689102] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37015.696228] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37015.703350] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37015.710476] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37015.717599] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37015.724723] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37015.732802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37015.738538] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37015.745663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37015.752787] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37015.759912] Call Trace: [37015.762354] [37015.764367] ? rcu_dump_cpu_stacks+0xd1/0x180 [37015.768728] ? rcu_sched_clock_irq+0x633/0x1390 [37015.773259] ? lock_release+0xc5/0x260 [37015.777002] ? lock_is_held_type+0xaa/0x120 [37015.781182] ? lock_is_held_type+0xaa/0x120 [37015.785359] ? tick_sched_do_timer+0xa0/0xa0 [37015.789629] ? rcu_read_lock_sched_held+0x56/0x90 [37015.794330] ? tick_sched_do_timer+0xa0/0xa0 [37015.798600] ? update_process_times+0x5f/0x90 [37015.802952] ? tick_sched_handle.isra.16+0x1e/0x50 [37015.807735] ? tick_sched_timer+0x6f/0x80 [37015.811741] ? __hrtimer_run_queues+0x1d3/0x360 [37015.816268] ? hrtimer_interrupt+0x104/0x240 [37015.820535] ? __local_bh_enable+0x50/0x60 [37015.824628] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37015.829853] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37015.834811] [37015.836908] [37015.839004] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37015.844314] ? lock_acquire+0xf0/0x2b0 [37015.848064] ? igrab+0x43/0x50 [37015.851114] ? lock_release+0xc5/0x260 [37015.854859] _raw_spin_lock+0x30/0x40 [37015.858524] ? igrab+0x1d/0x50 [37015.861575] igrab+0x1d/0x50 [37015.864452] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37015.869603] delayed_work+0x1d8/0x280 [ceph] [37015.873882] ? process_scheduled_works+0x22a/0x510 [37015.878673] process_scheduled_works+0x2a4/0x510 [37015.883289] worker_thread+0x188/0x340 [37015.887039] ? rescuer_thread+0x310/0x310 [37015.891049] kthread+0xf5/0x130 [37015.894186] ? kthread_complete_and_exit+0x20/0x20 [37015.898973] ret_from_fork+0x31/0x40 [37015.902550] ? kthread_complete_and_exit+0x20/0x20 [37015.907333] ret_from_fork_asm+0x11/0x20 [37015.911260] [37042.104348] watchdog: BUG: soft lockup - CPU#0 stuck for 23384s! [dbench:78683] [37042.111648] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37042.116348] watchdog: BUG: soft lockup - CPU#7 stuck for 32381s! [kworker/7:4:78808] [37042.167091] irq event stamp: 64999900 [37042.167092] hardirqs last enabled at (64999899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37042.174820] Modules linked in: [37042.178477] hardirqs last disabled at (64999900): [] sysvec_apic_timer_interrupt+0xe/0x90 [37042.188636] ceph [37042.191685] softirqs last enabled at (64999846): [] __do_softirq+0x2be/0x371 [37042.201411] libceph dns_resolver [37042.203334] softirqs last disabled at (64999841): [] irq_exit_rcu+0xb0/0xd0 [37042.212025] fscache netfs [37042.215337] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37042.223849] veth [37042.226552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37042.235495] nft_chain_nat [37042.237422] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37042.244892] xt_MASQUERADE nf_nat [37042.247596] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37042.253072] nf_conntrack nf_defrag_ipv6 [37042.256384] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37042.275122] nf_defrag_ipv4 [37042.279040] [37042.279041] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37042.284256] nft_compat nf_tables [37042.287048] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37042.288538] nfnetlink bridge [37042.295661] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37042.298972] stp llc [37042.306096] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37042.309061] xfs libcrc32c [37042.316184] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37042.318369] sunrpc intel_rapl_msr [37042.325493] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37042.328196] intel_rapl_common sb_edac [37042.335321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37042.338719] x86_pkg_temp_thermal coretemp [37042.346812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37042.350558] kvm_intel kvm [37042.356294] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37042.360386] irqbypass i2c_i801 [37042.367509] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37042.370215] iTCO_wdt iTCO_vendor_support [37042.377338] Call Trace: [37042.377339] [37042.380475] i2c_smbus [37042.387601] ? watchdog_timer_fn+0x1b4/0x220 [37042.391604] crct10dif_pclmul joydev [37042.394048] ? __hrtimer_run_queues+0x1d3/0x360 [37042.396059] crc32_pclmul ipmi_ssif [37042.398418] ? hrtimer_interrupt+0x104/0x240 [37042.402680] ghash_clmulni_intel mei_me [37042.406250] ? __local_bh_enable+0x50/0x60 [37042.410774] mei lpc_ich [37042.414261] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37042.418523] sha512_ssse3 acpi_ipmi [37042.422355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37042.426444] mfd_core wmi [37042.428975] [37042.434191] ipmi_si ipmi_devintf [37042.437675] [37042.437677] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37042.442636] ipmi_msghandler [37042.445255] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37042.447350] ioatdma acpi_pad [37042.450661] do_raw_spin_lock+0xb5/0xc0 [37042.452756] acpi_power_meter ixgbe [37042.458061] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37042.460937] igb mdio [37042.465810] try_get_cap_refs+0x159/0x880 [ceph] [37042.468773] crc32c_intel [37042.472609] __ceph_get_caps+0x4a1/0x760 [ceph] [37042.476088] nvme ptp [37042.480876] ? __file_remove_privs+0x89/0x110 [37042.483142] nvme_core i2c_algo_bit [37042.487754] ? wake_bit_function+0x70/0x70 [37042.490370] pps_core dca [37042.494897] ? ceph_write_iter+0x260/0x770 [ceph] [37042.497165] fuse [37042.501516] ceph_write_iter+0x260/0x770 [ceph] [37042.505000] irq event stamp: 216911802 [37042.509095] ? lock_is_held_type+0xaa/0x120 [37042.511708] hardirqs last enabled at (216911801): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37042.516408] vfs_write+0x324/0x4f0 [37042.518329] hardirqs last disabled at (216911802): [] sysvec_apic_timer_interrupt+0xe/0x90 [37042.522860] ksys_pwrite64+0x65/0xa0 [37042.526597] softirqs last enabled at (216911800): [] __do_softirq+0x2be/0x371 [37042.530775] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37042.541021] softirqs last disabled at (216911795): [] irq_exit_rcu+0xb0/0xd0 [37042.544435] do_syscall_64+0x38/0x80 [37042.554247] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37042.557816] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37042.566587] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37042.571717] RIP: 0033:0x7f48f0eeb93b [37042.580315] Workqueue: events delayed_work [ceph] [37042.583887] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37042.593263] [37042.593264] RIP: 0010:do_raw_spin_lock+0x5c/0xc0 [37042.598306] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37042.605779] Code: 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 89 43 10 5b e8 2e 22 45 00 85 c0 74 b1 48 c7 c6 29 e2 12 82 48 89 df e8 ab [37042.609348] ORIG_RAX: 0000000000000012 [37042.614047] RSP: 0018:ffffc90002117d98 EFLAGS: 00000246 [37042.632783] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37042.634274] [37042.634275] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [37042.638886] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37042.644104] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [37042.662841] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37042.666670] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [37042.671889] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37042.679013] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [37042.680505] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37042.687628] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37042.694758] [37042.701876] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37042.754882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37042.760619] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37042.767755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37042.774876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37042.782000] Call Trace: [37042.784446] [37042.786458] ? watchdog_timer_fn+0x1b4/0x220 [37042.790721] ? __hrtimer_run_queues+0x1d3/0x360 [37042.795248] ? hrtimer_interrupt+0x104/0x240 [37042.799516] ? __local_bh_enable+0x50/0x60 [37042.803610] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37042.808824] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37042.813782] [37042.815879] [37042.817977] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37042.823284] ? do_raw_spin_lock+0x5c/0xc0 [37042.827296] igrab+0x1d/0x50 [37042.830172] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37042.835323] delayed_work+0x1d8/0x280 [ceph] [37042.839603] ? process_scheduled_works+0x22a/0x510 [37042.844394] process_scheduled_works+0x2a4/0x510 [37042.849010] worker_thread+0x188/0x340 [37042.852758] ? rescuer_thread+0x310/0x310 [37042.856762] kthread+0xf5/0x130 [37042.859900] ? kthread_complete_and_exit+0x20/0x20 [37042.864684] ret_from_fork+0x31/0x40 [37042.868254] ? kthread_complete_and_exit+0x20/0x20 [37042.873038] ret_from_fork_asm+0x11/0x20 [37042.876962] [37070.104839] watchdog: BUG: soft lockup - CPU#0 stuck for 23410s! [dbench:78683] [37070.112145] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37070.116838] watchdog: BUG: soft lockup - CPU#7 stuck for 32407s! [kworker/7:4:78808] [37070.167594] irq event stamp: 65058166 [37070.167595] hardirqs last enabled at (65058165): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37070.175325] Modules linked in: [37070.178983] hardirqs last disabled at (65058166): [] sysvec_apic_timer_interrupt+0xe/0x90 [37070.189141] ceph [37070.192191] softirqs last enabled at (65058118): [] __do_softirq+0x2be/0x371 [37070.201914] libceph [37070.203838] softirqs last disabled at (65058113): [] irq_exit_rcu+0xb0/0xd0 [37070.212525] dns_resolver [37070.214707] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37070.223232] fscache netfs [37070.225855] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37070.234796] veth nft_chain_nat [37070.237501] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37070.244989] xt_MASQUERADE nf_nat [37070.248127] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37070.253604] nf_conntrack nf_defrag_ipv6 [37070.256914] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37070.275652] nf_defrag_ipv4 [37070.279570] [37070.284788] nft_compat nf_tables [37070.287579] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37070.289069] nfnetlink bridge [37070.292380] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37070.299514] stp llc [37070.302476] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37070.309600] xfs libcrc32c [37070.311786] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37070.318908] sunrpc [37070.321614] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37070.328736] intel_rapl_msr [37070.330833] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37070.337958] intel_rapl_common sb_edac [37070.340748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37070.348828] x86_pkg_temp_thermal coretemp [37070.352571] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37070.358307] kvm_intel kvm [37070.362399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37070.369526] irqbypass i2c_i801 [37070.372237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37070.379359] iTCO_wdt [37070.382513] Call Trace: [37070.382515] [37070.389638] iTCO_vendor_support [37070.391910] ? watchdog_timer_fn+0x1b4/0x220 [37070.394352] i2c_smbus crct10dif_pclmul [37070.396365] ? __hrtimer_run_queues+0x1d3/0x360 [37070.399589] joydev crc32_pclmul [37070.403856] ? hrtimer_interrupt+0x104/0x240 [37070.407683] ipmi_ssif ghash_clmulni_intel [37070.412207] ? __local_bh_enable+0x50/0x60 [37070.415430] mei_me mei [37070.419698] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37070.423786] lpc_ich sha512_ssse3 [37070.427877] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37070.430322] acpi_ipmi [37070.435537] [37070.438850] mfd_core [37070.443806] [37070.443808] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37070.446163] wmi ipmi_si [37070.448266] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37070.450532] ipmi_devintf [37070.452632] do_raw_spin_lock+0xb5/0xc0 [37070.457935] ipmi_msghandler [37070.460466] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37070.465334] ioatdma [37070.467954] try_get_cap_refs+0x159/0x880 [ceph] [37070.471784] acpi_pad acpi_power_meter [37070.474666] __ceph_get_caps+0x4a1/0x760 [ceph] [37070.479444] ixgbe [37070.481634] ? __file_remove_privs+0x89/0x110 [37070.486240] igb [37070.489984] ? wake_bit_function+0x70/0x70 [37070.494518] mdio [37070.496540] ? ceph_write_iter+0x260/0x770 [ceph] [37070.500885] crc32c_intel [37070.502725] ceph_write_iter+0x260/0x770 [ceph] [37070.506815] nvme [37070.508742] ? lock_is_held_type+0xaa/0x120 [37070.513435] ptp nvme_core [37070.516055] vfs_write+0x324/0x4f0 [37070.520578] i2c_algo_bit pps_core [37070.522521] ksys_pwrite64+0x65/0xa0 [37070.526697] dca fuse [37070.529401] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37070.532798] [37070.536196] do_syscall_64+0x38/0x80 [37070.539765] irq event stamp: 217132778 [37070.542038] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37070.547166] hardirqs last enabled at (217132777): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37070.548658] RIP: 0033:0x7f48f0eeb93b [37070.552237] hardirqs last disabled at (217132778): [] sysvec_apic_timer_interrupt+0xe/0x90 [37070.555982] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37070.561024] softirqs last enabled at (217132776): [] __do_softirq+0x2be/0x371 [37070.571268] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37070.574840] softirqs last disabled at (217132771): [] irq_exit_rcu+0xb0/0xd0 [37070.584650] ORIG_RAX: 0000000000000012 [37070.603389] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37070.612161] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37070.617377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37070.625974] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37070.629805] Workqueue: events delayed_work [ceph] [37070.639183] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37070.646308] [37070.646308] RIP: 0010:lock_acquire+0xf0/0x2b0 [37070.653779] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37070.660901] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37070.665599] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37070.672722] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37070.674239] [37070.678581] [37070.678582] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37070.727584] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37070.734709] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37070.741832] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37070.748958] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37070.756081] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37070.764158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37070.769897] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37070.777019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37070.784143] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37070.791269] Call Trace: [37070.793713] [37070.795723] ? watchdog_timer_fn+0x1b4/0x220 [37070.799988] ? __hrtimer_run_queues+0x1d3/0x360 [37070.804526] ? hrtimer_interrupt+0x104/0x240 [37070.808791] ? __local_bh_enable+0x50/0x60 [37070.812885] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37070.818100] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37070.823060] [37070.825156] [37070.827255] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37070.832563] ? lock_acquire+0xf0/0x2b0 [37070.836311] ? igrab+0x43/0x50 [37070.839363] ? lock_release+0xc5/0x260 [37070.843106] _raw_spin_lock+0x30/0x40 [37070.846762] ? igrab+0x1d/0x50 [37070.849812] igrab+0x1d/0x50 [37070.852692] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37070.857841] delayed_work+0x1d8/0x280 [ceph] [37070.862122] ? process_scheduled_works+0x22a/0x510 [37070.866911] process_scheduled_works+0x2a4/0x510 [37070.871544] worker_thread+0x188/0x340 [37070.875296] ? rescuer_thread+0x310/0x310 [37070.879307] kthread+0xf5/0x130 [37070.882454] ? kthread_complete_and_exit+0x20/0x20 [37070.887247] ret_from_fork+0x31/0x40 [37070.890824] ? kthread_complete_and_exit+0x20/0x20 [37070.895608] ret_from_fork_asm+0x11/0x20 [37070.899543] [37098.105324] watchdog: BUG: soft lockup - CPU#0 stuck for 23436s! [dbench:78683] [37098.112624] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37098.117324] watchdog: BUG: soft lockup - CPU#7 stuck for 32433s! [kworker/7:4:78808] [37098.168065] irq event stamp: 65116430 [37098.168067] hardirqs last enabled at (65116429): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37098.175796] Modules linked in: ceph [37098.179453] hardirqs last disabled at (65116430): [] sysvec_apic_timer_interrupt+0xe/0x90 [37098.189612] libceph dns_resolver [37098.193095] softirqs last enabled at (65116404): [] __do_softirq+0x2be/0x371 [37098.202819] fscache netfs [37098.206129] softirqs last disabled at (65116399): [] irq_exit_rcu+0xb0/0xd0 [37098.214814] veth nft_chain_nat [37098.217519] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37098.226029] xt_MASQUERADE nf_nat [37098.229167] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37098.238110] nf_conntrack nf_defrag_ipv6 [37098.241420] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37098.248892] nf_defrag_ipv4 nft_compat [37098.252810] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37098.258287] nf_tables [37098.262031] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37098.280769] nfnetlink bridge [37098.283126] [37098.283127] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37098.288342] stp llc [37098.291306] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37098.292799] xfs libcrc32c [37098.299922] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37098.302105] sunrpc intel_rapl_msr [37098.309231] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37098.311934] intel_rapl_common sb_edac [37098.319059] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37098.322456] x86_pkg_temp_thermal coretemp [37098.329581] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37098.333323] kvm_intel kvm [37098.340448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37098.344539] irqbypass i2c_i801 [37098.352617] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37098.355321] iTCO_wdt iTCO_vendor_support [37098.361057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37098.364195] i2c_smbus crct10dif_pclmul [37098.371319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37098.375323] joydev crc32_pclmul [37098.382447] Call Trace: [37098.382449] [37098.386278] ipmi_ssif ghash_clmulni_intel [37098.393403] ? watchdog_timer_fn+0x1b4/0x220 [37098.396626] mei_me mei [37098.399073] ? __hrtimer_run_queues+0x1d3/0x360 [37098.401081] lpc_ich sha512_ssse3 [37098.405174] ? hrtimer_interrupt+0x104/0x240 [37098.409435] acpi_ipmi mfd_core [37098.411880] ? __local_bh_enable+0x50/0x60 [37098.416405] wmi ipmi_si [37098.419726] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37098.424002] ipmi_devintf ipmi_msghandler [37098.427143] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37098.431233] ioatdma acpi_pad [37098.433765] [37098.438981] acpi_power_meter ixgbe [37098.443009] [37098.447968] igb mdio [37098.450934] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37098.453031] crc32c_intel [37098.456520] ? queued_spin_lock_slowpath+0x16/0x2e0 [37098.458612] nvme ptp [37098.460884] do_raw_spin_lock+0xb5/0xc0 [37098.466186] nvme_core [37098.468805] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37098.473674] i2c_algo_bit pps_core [37098.475948] try_get_cap_refs+0x159/0x880 [ceph] [37098.479775] dca fuse [37098.482138] __ceph_get_caps+0x4a1/0x760 [ceph] [37098.486918] [37098.486919] irq event stamp: 217353494 [37098.490320] ? __file_remove_privs+0x89/0x110 [37098.494925] hardirqs last enabled at (217353493): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37098.497198] ? wake_bit_function+0x70/0x70 [37098.501724] hardirqs last disabled at (217353494): [] sysvec_apic_timer_interrupt+0xe/0x90 [37098.503223] ? ceph_write_iter+0x260/0x770 [ceph] [37098.506963] softirqs last enabled at (217353492): [] __do_softirq+0x2be/0x371 [37098.511314] ceph_write_iter+0x260/0x770 [ceph] [37098.521559] softirqs last disabled at (217353487): [] irq_exit_rcu+0xb0/0xd0 [37098.525654] ? lock_is_held_type+0xaa/0x120 [37098.535462] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37098.540161] vfs_write+0x324/0x4f0 [37098.548930] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37098.553456] ksys_pwrite64+0x65/0xa0 [37098.562050] Workqueue: events delayed_work [ceph] [37098.566228] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37098.575605] [37098.575606] RIP: 0010:lock_acquire+0xf0/0x2b0 [37098.579019] do_syscall_64+0x38/0x80 [37098.586492] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37098.590061] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37098.594760] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37098.599889] RIP: 0033:0x7f48f0eeb93b [37098.601380] [37098.601381] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37098.605730] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37098.609303] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37098.628039] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37098.633083] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37098.638302] ORIG_RAX: 0000000000000012 [37098.641870] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37098.643361] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37098.650486] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37098.669224] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37098.676347] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37098.681566] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37098.688690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37098.692520] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37098.699644] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37098.706769] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37098.713893] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37098.721022] [37098.729095] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37098.779761] Call Trace: [37098.782206] [37098.784217] ? watchdog_timer_fn+0x1b4/0x220 [37098.788480] ? __hrtimer_run_queues+0x1d3/0x360 [37098.793012] ? hrtimer_interrupt+0x104/0x240 [37098.797276] ? __local_bh_enable+0x50/0x60 [37098.801372] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37098.806595] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37098.811552] [37098.813648] [37098.815747] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37098.821055] ? lock_acquire+0xf0/0x2b0 [37098.824805] ? igrab+0x43/0x50 [37098.827855] ? lock_release+0xc5/0x260 [37098.831609] _raw_spin_lock+0x30/0x40 [37098.835272] ? igrab+0x1d/0x50 [37098.838324] igrab+0x1d/0x50 [37098.841201] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37098.846351] delayed_work+0x1d8/0x280 [ceph] [37098.850632] ? process_scheduled_works+0x22a/0x510 [37098.855413] process_scheduled_works+0x2a4/0x510 [37098.860028] worker_thread+0x188/0x340 [37098.863780] ? rescuer_thread+0x310/0x310 [37098.867791] kthread+0xf5/0x130 [37098.870928] ? kthread_complete_and_exit+0x20/0x20 [37098.875730] ret_from_fork+0x31/0x40 [37098.879301] ? kthread_complete_and_exit+0x20/0x20 [37098.884084] ret_from_fork_asm+0x11/0x20 [37098.888023] [37126.105810] watchdog: BUG: soft lockup - CPU#0 stuck for 23462s! [dbench:78683] [37126.113113] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37126.117809] watchdog: BUG: soft lockup - CPU#7 stuck for 32459s! [kworker/7:4:78808] [37126.168562] irq event stamp: 65174666 [37126.176292] Modules linked in: ceph libceph dns_resolver fscache netfs veth [37126.179951] hardirqs last enabled at (65174665): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37126.179957] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [37126.197067] hardirqs last disabled at (65174666): [] sysvec_apic_timer_interrupt+0xe/0x90 [37126.206271] nf_tables nfnetlink bridge stp llc xfs libcrc32c [37126.215994] softirqs last enabled at (65174618): [] __do_softirq+0x2be/0x371 [37126.215998] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [37126.230417] softirqs last disabled at (65174613): [] irq_exit_rcu+0xb0/0xd0 [37126.230421] kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [37126.248053] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37126.254916] crct10dif_pclmul joydev crc32_pclmul [37126.263862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37126.263864] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37126.268559] ipmi_ssif [37126.276031] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37126.281508] ghash_clmulni_intel mei_me [37126.283864] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37126.302602] mei lpc_ich [37126.306433] [37126.311651] sha512_ssse3 acpi_ipmi [37126.314181] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37126.315690] mfd_core wmi [37126.319174] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37126.326297] ipmi_si ipmi_devintf [37126.328914] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37126.336040] ipmi_msghandler [37126.339351] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37126.346485] ioatdma acpi_pad [37126.349361] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37126.356488] acpi_power_meter ixgbe [37126.359449] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37126.366571] igb mdio [37126.370056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37126.378133] crc32c_intel nvme [37126.380405] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37126.386142] ptp [37126.389207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37126.396334] nvme_core [37126.398171] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37126.405297] i2c_algo_bit [37126.407654] Call Trace: [37126.407656] [37126.414777] pps_core [37126.417398] ? watchdog_timer_fn+0x1b4/0x220 [37126.419839] dca fuse [37126.421849] ? __hrtimer_run_queues+0x1d3/0x360 [37126.424121] [37126.424122] irq event stamp: 217574308 [37126.428387] ? hrtimer_interrupt+0x104/0x240 [37126.430656] hardirqs last enabled at (217574307): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37126.435179] ? __local_bh_enable+0x50/0x60 [37126.436688] hardirqs last disabled at (217574308): [] sysvec_apic_timer_interrupt+0xe/0x90 [37126.440435] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37126.444694] softirqs last enabled at (217574306): [] __do_softirq+0x2be/0x371 [37126.454940] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37126.459030] softirqs last disabled at (217574301): [] irq_exit_rcu+0xb0/0xd0 [37126.468841] [37126.474059] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37126.482828] [37126.487788] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37126.496384] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37126.498498] Workqueue: events delayed_work [ceph] [37126.507880] ? queued_spin_lock_slowpath+0x16/0x2e0 [37126.509974] [37126.509975] RIP: 0010:lock_acquire+0xf0/0x2b0 [37126.517447] do_raw_spin_lock+0xb5/0xc0 [37126.522748] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37126.527448] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37126.532316] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37126.533810] try_get_cap_refs+0x159/0x880 [ceph] [37126.538157] [37126.538158] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37126.541994] __ceph_get_caps+0x4a1/0x760 [ceph] [37126.560728] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37126.565516] ? __file_remove_privs+0x89/0x110 [37126.570729] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37126.575338] ? wake_bit_function+0x70/0x70 [37126.576830] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37126.583956] ? ceph_write_iter+0x260/0x770 [ceph] [37126.588491] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37126.595619] ceph_write_iter+0x260/0x770 [ceph] [37126.599969] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37126.607099] ? lock_is_held_type+0xaa/0x120 [37126.611184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37126.618329] vfs_write+0x324/0x4f0 [37126.623024] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37126.623026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37126.630153] ksys_pwrite64+0x65/0xa0 [37126.634673] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37126.642750] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37126.646926] Call Trace: [37126.646928] [37126.652665] do_syscall_64+0x38/0x80 [37126.656062] ? watchdog_timer_fn+0x1b4/0x220 [37126.663189] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37126.670328] ? __hrtimer_run_queues+0x1d3/0x360 [37126.673899] RIP: 0033:0x7f48f0eeb93b [37126.681025] ? hrtimer_interrupt+0x104/0x240 [37126.686153] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37126.688596] ? __local_bh_enable+0x50/0x60 [37126.690607] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37126.694182] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37126.698442] ORIG_RAX: 0000000000000012 [37126.703493] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37126.708019] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37126.711592] [37126.715855] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37126.734593] [37126.734594] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37126.738683] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37126.743904] ? lock_acquire+0xf0/0x2b0 [37126.749118] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37126.752950] ? igrab+0x43/0x50 [37126.757907] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37126.765030] ? lock_release+0xc5/0x260 [37126.767132] [37126.774252] _raw_spin_lock+0x30/0x40 [37126.819407] ? igrab+0x1d/0x50 [37126.822457] igrab+0x1d/0x50 [37126.825335] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37126.830502] delayed_work+0x1d8/0x280 [ceph] [37126.834784] ? process_scheduled_works+0x22a/0x510 [37126.839574] process_scheduled_works+0x2a4/0x510 [37126.844192] worker_thread+0x188/0x340 [37126.847956] ? rescuer_thread+0x310/0x310 [37126.851958] kthread+0xf5/0x130 [37126.855097] ? kthread_complete_and_exit+0x20/0x20 [37126.859880] ret_from_fork+0x31/0x40 [37126.863461] ? kthread_complete_and_exit+0x20/0x20 [37126.868244] ret_from_fork_asm+0x11/0x20 [37126.872177] [37154.106295] watchdog: BUG: soft lockup - CPU#0 stuck for 23488s! [dbench:78683] [37154.113599] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37154.118295] watchdog: BUG: soft lockup - CPU#7 stuck for 32485s! [kworker/7:4:78808] [37154.169039] irq event stamp: 65232776 [37154.169041] hardirqs last enabled at (65232775): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37154.176770] Modules linked in: [37154.180427] hardirqs last disabled at (65232776): [] sysvec_apic_timer_interrupt+0xe/0x90 [37154.190585] ceph [37154.193636] softirqs last enabled at (65232728): [] __do_softirq+0x2be/0x371 [37154.203361] libceph [37154.205286] softirqs last disabled at (65232723): [] irq_exit_rcu+0xb0/0xd0 [37154.213970] dns_resolver [37154.216154] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37154.224663] fscache netfs [37154.227298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37154.236243] veth nft_chain_nat [37154.238946] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37154.246417] xt_MASQUERADE nf_nat [37154.249555] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37154.255034] nf_conntrack nf_defrag_ipv6 [37154.258342] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37154.277081] nf_defrag_ipv4 nft_compat [37154.280998] [37154.280999] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37154.286217] nf_tables nfnetlink [37154.289961] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37154.291467] bridge stp [37154.298593] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37154.301817] llc xfs [37154.308939] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37154.308941] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37154.311384] libcrc32c [37154.318508] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37154.320693] sunrpc [37154.327818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37154.334940] intel_rapl_msr intel_rapl_common [37154.337299] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37154.345375] sb_edac x86_pkg_temp_thermal [37154.347473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37154.353210] coretemp kvm_intel [37154.357561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37154.364691] kvm irqbypass [37154.368699] Call Trace: [37154.375823] i2c_i801 iTCO_wdt [37154.378973] [37154.386101] iTCO_vendor_support [37154.388806] ? watchdog_timer_fn+0x1b4/0x220 [37154.391249] i2c_smbus crct10dif_pclmul [37154.394300] ? __hrtimer_run_queues+0x1d3/0x360 [37154.396311] joydev crc32_pclmul [37154.399538] ? hrtimer_interrupt+0x104/0x240 [37154.403799] ipmi_ssif ghash_clmulni_intel [37154.407630] ? __local_bh_enable+0x50/0x60 [37154.412155] mei_me mei [37154.415380] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37154.419642] lpc_ich sha512_ssse3 [37154.423732] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37154.427823] acpi_ipmi mfd_core [37154.430268] [37154.435484] wmi ipmi_si [37154.438795] [37154.438797] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37154.443753] ipmi_devintf [37154.446894] ? queued_spin_lock_slowpath+0x16/0x2e0 [37154.448987] ipmi_msghandler [37154.451520] do_raw_spin_lock+0xb5/0xc0 [37154.453617] ioatdma [37154.458920] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37154.461537] acpi_pad acpi_power_meter [37154.466411] try_get_cap_refs+0x159/0x880 [ceph] [37154.469285] ixgbe igb [37154.473121] __ceph_get_caps+0x4a1/0x760 [ceph] [37154.475300] mdio crc32c_intel [37154.480088] ? __file_remove_privs+0x89/0x110 [37154.483829] nvme ptp [37154.488440] ? wake_bit_function+0x70/0x70 [37154.490796] nvme_core i2c_algo_bit [37154.495324] ? ceph_write_iter+0x260/0x770 [ceph] [37154.498372] pps_core dca [37154.502722] ceph_write_iter+0x260/0x770 [ceph] [37154.504992] fuse [37154.509086] ? lock_is_held_type+0xaa/0x120 [37154.512569] irq event stamp: 217795306 [37154.517267] vfs_write+0x324/0x4f0 [37154.519883] hardirqs last enabled at (217795305): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37154.524411] ksys_pwrite64+0x65/0xa0 [37154.526331] hardirqs last disabled at (217795306): [] sysvec_apic_timer_interrupt+0xe/0x90 [37154.530507] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37154.534251] softirqs last enabled at (217795304): [] __do_softirq+0x2be/0x371 [37154.537652] do_syscall_64+0x38/0x80 [37154.547894] softirqs last disabled at (217795299): [] irq_exit_rcu+0xb0/0xd0 [37154.551466] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37154.561275] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37154.566406] RIP: 0033:0x7f48f0eeb93b [37154.575176] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37154.578747] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37154.587345] Workqueue: events delayed_work [ceph] [37154.592389] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37154.601767] [37154.601767] RIP: 0010:lock_acquire+0xf0/0x2b0 [37154.605338] ORIG_RAX: 0000000000000012 [37154.612807] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37154.631545] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37154.636244] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37154.641460] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37154.642950] [37154.642951] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37154.647301] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37154.651133] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37154.669870] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37154.676994] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37154.682211] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37154.689335] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37154.690833] [37154.697952] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37154.697953] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37154.758082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37154.763819] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37154.770943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37154.778067] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37154.785190] Call Trace: [37154.787635] [37154.789646] ? watchdog_timer_fn+0x1b4/0x220 [37154.793910] ? __hrtimer_run_queues+0x1d3/0x360 [37154.798437] ? hrtimer_interrupt+0x104/0x240 [37154.802698] ? __local_bh_enable+0x50/0x60 [37154.806792] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37154.812015] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37154.816980] [37154.819078] [37154.821176] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37154.826485] ? lock_acquire+0xf0/0x2b0 [37154.830236] ? igrab+0x43/0x50 [37154.833293] ? lock_release+0xc5/0x260 [37154.837039] _raw_spin_lock+0x30/0x40 [37154.840702] ? igrab+0x1d/0x50 [37154.843754] igrab+0x1d/0x50 [37154.846630] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37154.851782] delayed_work+0x1d8/0x280 [ceph] [37154.856063] ? process_scheduled_works+0x22a/0x510 [37154.860854] process_scheduled_works+0x2a4/0x510 [37154.865468] worker_thread+0x188/0x340 [37154.869218] ? rescuer_thread+0x310/0x310 [37154.873222] kthread+0xf5/0x130 [37154.876358] ? kthread_complete_and_exit+0x20/0x20 [37154.881142] ret_from_fork+0x31/0x40 [37154.884712] ? kthread_complete_and_exit+0x20/0x20 [37154.889496] ret_from_fork_asm+0x11/0x20 [37154.893419] [37182.106780] watchdog: BUG: soft lockup - CPU#0 stuck for 23514s! [dbench:78683] [37182.114090] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37182.118780] watchdog: BUG: soft lockup - CPU#7 stuck for 32511s! [kworker/7:4:78808] [37182.169538] irq event stamp: 65291046 [37182.177267] Modules linked in: ceph libceph dns_resolver fscache netfs veth [37182.180925] hardirqs last enabled at (65291045): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37182.187884] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [37182.198042] hardirqs last disabled at (65291046): [] sysvec_apic_timer_interrupt+0xe/0x90 [37182.204992] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [37182.214718] softirqs last enabled at (65290998): [] __do_softirq+0x2be/0x371 [37182.221156] llc xfs libcrc32c sunrpc [37182.229840] softirqs last disabled at (65290993): [] irq_exit_rcu+0xb0/0xd0 [37182.233498] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37182.242009] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37182.248872] coretemp kvm_intel [37182.257818] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37182.257819] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37182.260953] kvm irqbypass [37182.268424] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37182.273902] i2c_i801 iTCO_wdt [37182.276608] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37182.295344] iTCO_vendor_support i2c_smbus [37182.298394] [37182.298395] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37182.303613] crct10dif_pclmul joydev [37182.307704] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37182.309193] crc32_pclmul [37182.316318] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37182.319890] ipmi_ssif [37182.327014] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37182.329629] ghash_clmulni_intel mei_me [37182.336753] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37182.339113] mei lpc_ich [37182.346235] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37182.350067] sha512_ssse3 [37182.357191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37182.359722] acpi_ipmi mfd_core [37182.367798] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37182.370416] wmi ipmi_si [37182.376154] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37182.379310] ipmi_devintf [37182.386433] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37182.388981] ipmi_msghandler [37182.396106] Call Trace: [37182.396109] [37182.398722] ioatdma [37182.405848] ? watchdog_timer_fn+0x1b4/0x220 [37182.408723] acpi_pad acpi_power_meter [37182.411180] ? __hrtimer_run_queues+0x1d3/0x360 [37182.413187] ixgbe [37182.415376] ? hrtimer_interrupt+0x104/0x240 [37182.419636] igb mdio [37182.423378] ? __local_bh_enable+0x50/0x60 [37182.427903] crc32c_intel nvme [37182.429916] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37182.434180] ptp nvme_core [37182.436462] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37182.440549] i2c_algo_bit pps_core [37182.443600] [37182.448816] dca [37182.451520] [37182.451522] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37182.456479] fuse [37182.459880] ? queued_spin_lock_slowpath+0x16/0x2e0 [37182.461973] [37182.461974] irq event stamp: 218016126 [37182.463811] do_raw_spin_lock+0xb5/0xc0 [37182.465908] hardirqs last enabled at (218016125): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37182.471215] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37182.473136] hardirqs last disabled at (218016126): [] sysvec_apic_timer_interrupt+0xe/0x90 [37182.478010] try_get_cap_refs+0x159/0x880 [ceph] [37182.479497] softirqs last enabled at (218016124): [] __do_softirq+0x2be/0x371 [37182.483248] __ceph_get_caps+0x4a1/0x760 [ceph] [37182.487072] softirqs last disabled at (218016119): [] irq_exit_rcu+0xb0/0xd0 [37182.497322] ? __file_remove_privs+0x89/0x110 [37182.502101] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37182.511911] ? wake_bit_function+0x70/0x70 [37182.516521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37182.525294] ? ceph_write_iter+0x260/0x770 [ceph] [37182.529818] Workqueue: events delayed_work [ceph] [37182.538415] ceph_write_iter+0x260/0x770 [ceph] [37182.542766] [37182.542766] RIP: 0010:do_raw_spin_lock+0x0/0xc0 [37182.552147] ? lock_is_held_type+0xaa/0x120 [37182.556234] Code: c0 eb 7e 44 8b 80 28 05 00 00 49 89 e9 48 89 de 48 8d 88 28 08 00 00 48 c7 c7 20 d2 19 82 e8 77 d0 00 00 5b 5d e9 c0 a3 9f 00 <66> 0f 1f 00 0f 1f 44 00 00 53 8b 47 04 48 89 fb 3d ad 4e ad de 75 [37182.563706] vfs_write+0x324/0x4f0 [37182.568402] RSP: 0018:ffffc90002117d98 EFLAGS: 00000296 [37182.573104] ksys_pwrite64+0x65/0xa0 [37182.577622] [37182.577623] RAX: ffffffff813b940d RBX: ffff8881a1d34478 RCX: 000000005553a634 [37182.579114] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37182.583637] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff8881a1d34500 [37182.587817] do_syscall_64+0x38/0x80 [37182.606552] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [37182.609953] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37182.615184] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [37182.618755] RIP: 0033:0x7f48f0eeb93b [37182.620245] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37182.627372] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37182.632503] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37182.639624] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37182.643196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37182.650320] ORIG_RAX: 0000000000000012 [37182.655365] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37182.662489] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37182.666060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37182.673183] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37182.673184] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37182.691922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37182.699998] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37182.705216] Call Trace: [37182.710954] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37182.714784] [37182.721914] [37182.729033] ? watchdog_timer_fn+0x1b4/0x220 [37182.782682] ? __hrtimer_run_queues+0x1d3/0x360 [37182.787209] ? hrtimer_interrupt+0x104/0x240 [37182.791476] ? __local_bh_enable+0x50/0x60 [37182.795571] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37182.800795] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37182.805751] [37182.807848] [37182.809946] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37182.815252] ? igrab+0x1d/0x50 [37182.818303] ? rwlock_bug+0x50/0x50 [37182.821796] igrab+0x1d/0x50 [37182.824674] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37182.829824] delayed_work+0x1d8/0x280 [ceph] [37182.834103] ? process_scheduled_works+0x22a/0x510 [37182.838893] process_scheduled_works+0x2a4/0x510 [37182.843509] worker_thread+0x188/0x340 [37182.847258] ? rescuer_thread+0x310/0x310 [37182.851260] kthread+0xf5/0x130 [37182.854401] ? kthread_complete_and_exit+0x20/0x20 [37182.859193] ret_from_fork+0x31/0x40 [37182.862771] ? kthread_complete_and_exit+0x20/0x20 [37182.867556] ret_from_fork_asm+0x11/0x20 [37182.871488] [37210.107264] watchdog: BUG: soft lockup - CPU#0 stuck for 23540s! [dbench:78683] [37210.114565] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37210.119264] watchdog: BUG: soft lockup - CPU#7 stuck for 32537s! [kworker/7:4:78808] [37210.170009] irq event stamp: 65349240 [37210.177738] Modules linked in: ceph libceph dns_resolver fscache [37210.181396] hardirqs last enabled at (65349239): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37210.187393] netfs veth nft_chain_nat [37210.197551] hardirqs last disabled at (65349240): [] sysvec_apic_timer_interrupt+0xe/0x90 [37210.197552] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [37210.210948] softirqs last enabled at (65349192): [] __do_softirq+0x2be/0x371 [37210.210950] nf_defrag_ipv4 [37210.210951] softirqs last disabled at (65349187): [] irq_exit_rcu+0xb0/0xd0 [37210.216687] nft_compat nf_tables [37210.225370] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37210.228162] nfnetlink bridge [37210.236672] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37210.239982] stp llc [37210.248927] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37210.251891] xfs libcrc32c [37210.259363] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37210.261546] sunrpc [37210.267024] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37210.269727] intel_rapl_msr [37210.288465] [37210.288466] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37210.290562] intel_rapl_common [37210.295781] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37210.298570] sb_edac x86_pkg_temp_thermal [37210.300062] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37210.307186] coretemp [37210.310237] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37210.317360] kvm_intel [37210.321364] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37210.328488] kvm [37210.330760] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37210.337882] irqbypass [37210.340240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37210.347365] i2c_i801 iTCO_wdt [37210.349201] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37210.357279] iTCO_vendor_support [37210.359637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37210.365392] i2c_smbus crct10dif_pclmul [37210.368443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37210.375568] joydev [37210.378791] Call Trace: [37210.378792] [37210.385915] crc32_pclmul ipmi_ssif [37210.389746] ? watchdog_timer_fn+0x1b4/0x220 [37210.396869] ghash_clmulni_intel mei_me [37210.398969] ? __hrtimer_run_queues+0x1d3/0x360 [37210.401410] mei lpc_ich [37210.403425] ? hrtimer_interrupt+0x104/0x240 [37210.406906] sha512_ssse3 acpi_ipmi [37210.411171] ? __local_bh_enable+0x50/0x60 [37210.415000] mfd_core wmi [37210.419527] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37210.422055] ipmi_si ipmi_devintf [37210.426320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37210.429803] ipmi_msghandler ioatdma [37210.433896] [37210.436513] acpi_pad [37210.441730] [37210.441732] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37210.445041] acpi_power_meter [37210.450001] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37210.453568] ixgbe igb [37210.455668] do_raw_spin_lock+0xb5/0xc0 [37210.457943] mdio crc32c_intel [37210.460044] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37210.465347] nvme ptp [37210.468313] try_get_cap_refs+0x159/0x880 [ceph] [37210.473181] nvme_core i2c_algo_bit [37210.475544] __ceph_get_caps+0x4a1/0x760 [ceph] [37210.479370] pps_core [37210.482425] ? __file_remove_privs+0x89/0x110 [37210.487204] dca fuse [37210.489476] ? wake_bit_function+0x70/0x70 [37210.494086] [37210.494087] irq event stamp: 218237066 [37210.497572] ? ceph_write_iter+0x260/0x770 [ceph] [37210.502094] hardirqs last enabled at (218237065): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37210.504366] ceph_write_iter+0x260/0x770 [ceph] [37210.508716] hardirqs last disabled at (218237066): [] sysvec_apic_timer_interrupt+0xe/0x90 [37210.510991] ? lock_is_held_type+0xaa/0x120 [37210.515078] softirqs last enabled at (218237064): [] __do_softirq+0x2be/0x371 [37210.516571] vfs_write+0x324/0x4f0 [37210.520311] softirqs last disabled at (218237059): [] irq_exit_rcu+0xb0/0xd0 [37210.525014] ksys_pwrite64+0x65/0xa0 [37210.535253] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37210.539778] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37210.549588] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37210.553766] do_syscall_64+0x38/0x80 [37210.562537] Workqueue: events delayed_work [ceph] [37210.565944] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37210.574539] [37210.574540] RIP: 0010:lock_acquire+0xf0/0x2b0 [37210.578112] RIP: 0033:0x7f48f0eeb93b [37210.587488] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37210.592620] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37210.600091] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37210.603666] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37210.608366] [37210.608367] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37210.613412] ORIG_RAX: 0000000000000012 [37210.614902] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37210.619251] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37210.622824] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37210.641561] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37210.660298] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37210.665515] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37210.670733] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37210.672223] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37210.679349] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37210.683178] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37210.690304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37210.697432] [37210.704551] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37210.770437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37210.777561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37210.784684] Call Trace: [37210.787128] [37210.789141] ? watchdog_timer_fn+0x1b4/0x220 [37210.793404] ? __hrtimer_run_queues+0x1d3/0x360 [37210.797948] ? hrtimer_interrupt+0x104/0x240 [37210.802217] ? __local_bh_enable+0x50/0x60 [37210.806311] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37210.811535] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37210.816493] [37210.818588] [37210.820686] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37210.825994] ? lock_acquire+0xf0/0x2b0 [37210.829745] ? igrab+0x43/0x50 [37210.832795] ? lock_release+0xc5/0x260 [37210.836539] _raw_spin_lock+0x30/0x40 [37210.840196] ? igrab+0x1d/0x50 [37210.843247] igrab+0x1d/0x50 [37210.846126] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37210.851275] delayed_work+0x1d8/0x280 [ceph] [37210.855556] ? process_scheduled_works+0x22a/0x510 [37210.860345] process_scheduled_works+0x2a4/0x510 [37210.864965] worker_thread+0x188/0x340 [37210.868713] ? rescuer_thread+0x310/0x310 [37210.872721] kthread+0xf5/0x130 [37210.875860] ? kthread_complete_and_exit+0x20/0x20 [37210.880647] ret_from_fork+0x31/0x40 [37210.884241] ? kthread_complete_and_exit+0x20/0x20 [37210.889026] ret_from_fork_asm+0x11/0x20 [37210.892956] [37210.918644] rcu: INFO: rcu_preempt self-detected stall on CPU [37210.924405] rcu: 0-....: (24702564 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=12370085 [37210.934917] rcu: (t=25273865 jiffies g=1309609 q=693548 ncpus=8) [37210.941001] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37210.949943] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37210.957415] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37210.962893] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37210.981630] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37210.986848] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37210.993971] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37211.001094] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37211.008220] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37211.015343] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37211.022467] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37211.030546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37211.036284] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37211.043407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37211.050531] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37211.057655] Call Trace: [37211.060100] [37211.062110] ? rcu_dump_cpu_stacks+0xd1/0x180 [37211.066471] ? rcu_sched_clock_irq+0x633/0x1390 [37211.071003] ? lock_is_held_type+0xaa/0x120 [37211.075180] ? lock_is_held_type+0xaa/0x120 [37211.079357] ? tick_sched_do_timer+0xa0/0xa0 [37211.083629] ? rcu_read_lock_sched_held+0x56/0x90 [37211.088329] ? tick_sched_do_timer+0xa0/0xa0 [37211.092592] ? update_process_times+0x5f/0x90 [37211.096943] ? tick_sched_handle.isra.16+0x1e/0x50 [37211.101726] ? tick_sched_timer+0x6f/0x80 [37211.105732] ? __hrtimer_run_queues+0x1d3/0x360 [37211.110258] ? hrtimer_interrupt+0x104/0x240 [37211.114528] ? __local_bh_enable+0x50/0x60 [37211.118621] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37211.123843] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37211.128803] [37211.130900] [37211.132997] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37211.138305] ? queued_spin_lock_slowpath+0x18/0x2e0 [37211.143181] do_raw_spin_lock+0xb5/0xc0 [37211.147010] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37211.151814] try_get_cap_refs+0x159/0x880 [ceph] [37211.156444] __ceph_get_caps+0x4a1/0x760 [ceph] [37211.160985] ? __file_remove_privs+0x89/0x110 [37211.165340] ? wake_bit_function+0x70/0x70 [37211.169434] ? ceph_write_iter+0x260/0x770 [ceph] [37211.174147] ceph_write_iter+0x260/0x770 [ceph] [37211.178684] ? lock_is_held_type+0xaa/0x120 [37211.182868] vfs_write+0x324/0x4f0 [37211.186276] ksys_pwrite64+0x65/0xa0 [37211.189851] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37211.194981] do_syscall_64+0x38/0x80 [37211.198552] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37211.203596] RIP: 0033:0x7f48f0eeb93b [37211.207166] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37211.225905] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [37211.233463] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37211.240586] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37211.247709] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37211.254833] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37211.261958] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37211.269089] [37211.271276] Sending NMI from CPU 0 to CPUs 7: [37211.275628] NMI backtrace for cpu 7 [37211.275629] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37211.275630] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37211.275631] Workqueue: events delayed_work [ceph] [37211.275645] RIP: 0010:__lock_acquire+0x345/0x1e20 [37211.275647] Code: 44 0f 44 f6 41 83 c5 01 44 3b ab 50 0a 00 00 7d 2d 49 63 c5 48 8d 04 80 41 0f b7 44 c4 20 66 25 ff 1f 0f b7 c0 48 0f a3 45 00 <72> a8 8b 05 fb 6e 68 02 85 c0 0f 84 43 04 00 00 31 c0 eb a8 48 89 [37211.275648] RSP: 0018:ffffc90002117c60 EFLAGS: 00000003 [37211.275650] RAX: 0000000000000710 RBX: ffff88810debaa80 RCX: 00000000000006f5 [37211.275651] RDX: ffff88810debb550 RSI: 0000000000000000 RDI: 0000000000000003 [37211.275653] RBP: ffffffff82f75120 R08: 0000000000000078 R09: 0000000000000000 [37211.275654] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810debb4d8 [37211.275654] R13: 0000000000000001 R14: 0000000000000004 R15: 0000000000000002 [37211.275655] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37211.275656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37211.275657] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37211.275658] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37211.275659] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37211.275659] Call Trace: [37211.275660] [37211.275661] ? nmi_cpu_backtrace+0xad/0x120 [37211.275665] ? nmi_cpu_backtrace_handler+0x11/0x20 [37211.275666] ? nmi_handle+0xd4/0x260 [37211.275670] ? default_do_nmi+0x49/0xf0 [37211.275672] ? exc_nmi+0x11e/0x150 [37211.275673] ? end_repeat_nmi+0x16/0x67 [37211.275676] ? __lock_acquire+0x345/0x1e20 [37211.275678] ? __lock_acquire+0x345/0x1e20 [37211.275680] ? __lock_acquire+0x345/0x1e20 [37211.275681] [37211.275681] [37211.275682] ? mark_lock+0x410/0x4c0 [37211.275684] lock_acquire+0xcb/0x2b0 [37211.275686] ? igrab+0x1d/0x50 [37211.275687] ? igrab+0x43/0x50 [37211.275688] ? lock_release+0xc5/0x260 [37211.275690] _raw_spin_lock+0x30/0x40 [37211.275692] ? igrab+0x1d/0x50 [37211.275693] igrab+0x1d/0x50 [37211.275694] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37211.275708] delayed_work+0x1d8/0x280 [ceph] [37211.275721] ? process_scheduled_works+0x22a/0x510 [37211.275723] process_scheduled_works+0x2a4/0x510 [37211.275727] worker_thread+0x188/0x340 [37211.275730] ? rescuer_thread+0x310/0x310 [37211.275732] kthread+0xf5/0x130 [37211.275734] ? kthread_complete_and_exit+0x20/0x20 [37211.275736] ret_from_fork+0x31/0x40 [37211.275738] ? kthread_complete_and_exit+0x20/0x20 [37211.275740] ret_from_fork_asm+0x11/0x20 [37211.275744] [37238.107747] watchdog: BUG: soft lockup - CPU#0 stuck for 23566s! [dbench:78683] [37238.115054] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37238.119747] watchdog: BUG: soft lockup - CPU#7 stuck for 32563s! [kworker/7:4:78808] [37238.170504] irq event stamp: 65406756 [37238.178235] Modules linked in: ceph libceph dns_resolver fscache netfs [37238.181891] hardirqs last enabled at (65406755): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37238.188422] veth nft_chain_nat xt_MASQUERADE nf_nat [37238.198575] hardirqs last disabled at (65406756): [] sysvec_apic_timer_interrupt+0xe/0x90 [37238.203533] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [37238.213258] softirqs last enabled at (65406712): [] __do_softirq+0x2be/0x371 [37238.221778] stp llc xfs libcrc32c sunrpc [37238.230461] softirqs last disabled at (65406707): [] irq_exit_rcu+0xb0/0xd0 [37238.234465] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [37238.242978] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37238.242979] kvm_intel [37238.242980] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37238.250620] kvm irqbypass [37238.259564] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37238.261921] i2c_i801 iTCO_wdt [37238.269392] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37238.272096] iTCO_vendor_support i2c_smbus [37238.277573] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37238.280623] crct10dif_pclmul [37238.299363] [37238.303453] joydev crc32_pclmul [37238.308669] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37238.311634] ipmi_ssif ghash_clmulni_intel [37238.313142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37238.316365] mei_me mei [37238.323490] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37238.327582] lpc_ich [37238.334705] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37238.337148] sha512_ssse3 [37238.344274] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37238.346458] acpi_ipmi mfd_core [37238.353582] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37238.356198] wmi ipmi_si [37238.363323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37238.366461] ipmi_devintf [37238.374539] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37238.377070] ipmi_msghandler ioatdma [37238.382807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37238.385424] acpi_pad acpi_power_meter [37238.392548] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37238.396120] ixgbe [37238.403259] Call Trace: [37238.403261] [37238.407005] igb mdio [37238.414148] ? watchdog_timer_fn+0x1b4/0x220 [37238.416157] crc32c_intel [37238.418600] ? __hrtimer_run_queues+0x1d3/0x360 [37238.420611] nvme [37238.422885] ? hrtimer_interrupt+0x104/0x240 [37238.427147] ptp nvme_core [37238.429763] ? __local_bh_enable+0x50/0x60 [37238.434287] i2c_algo_bit pps_core [37238.436214] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37238.440476] dca [37238.443181] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37238.447270] fuse [37238.450667] [37238.455884] [37238.457723] [37238.457724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37238.462681] irq event stamp: 218455870 [37238.464608] ? queued_spin_lock_slowpath+0x18/0x2e0 [37238.466701] hardirqs last enabled at (218455869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37238.468193] do_raw_spin_lock+0xb5/0xc0 [37238.470290] hardirqs last disabled at (218455870): [] sysvec_apic_timer_interrupt+0xe/0x90 [37238.475594] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37238.479338] softirqs last enabled at (218455868): [] __do_softirq+0x2be/0x371 [37238.484210] try_get_cap_refs+0x159/0x880 [ceph] [37238.494452] softirqs last disabled at (218455863): [] irq_exit_rcu+0xb0/0xd0 [37238.498287] __ceph_get_caps+0x4a1/0x760 [ceph] [37238.508093] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37238.512883] ? __file_remove_privs+0x89/0x110 [37238.521648] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37238.526261] ? wake_bit_function+0x70/0x70 [37238.534859] Workqueue: events delayed_work [ceph] [37238.539383] ? ceph_write_iter+0x260/0x770 [ceph] [37238.548759] [37238.548760] RIP: 0010:lock_release+0x1d2/0x260 [37238.553109] ceph_write_iter+0x260/0x770 [ceph] [37238.560579] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [37238.564675] ? lock_is_held_type+0xaa/0x120 [37238.569368] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [37238.574067] vfs_write+0x324/0x4f0 [37238.575557] [37238.575558] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [37238.579997] ksys_pwrite64+0x65/0xa0 [37238.584519] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [37238.603255] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37238.607432] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [37238.612653] do_syscall_64+0x38/0x80 [37238.616049] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [37238.617539] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37238.624664] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37238.628235] RIP: 0033:0x7f48f0eeb93b [37238.635356] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37238.640490] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37238.647614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37238.651183] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37238.658306] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37238.663350] ORIG_RAX: 0000000000000012 [37238.670476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37238.674047] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37238.682123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37238.700880] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37238.706617] Call Trace: [37238.706619] [37238.711834] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37238.718958] ? watchdog_timer_fn+0x1b4/0x220 [37238.722788] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37238.729913] ? __hrtimer_run_queues+0x1d3/0x360 [37238.737036] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37238.744166] ? hrtimer_interrupt+0x104/0x240 [37238.751291] [37238.753728] ? __local_bh_enable+0x50/0x60 [37238.796459] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37238.801682] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37238.806641] [37238.808737] [37238.810836] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37238.816151] ? igrab+0x43/0x50 [37238.819211] ? igrab+0x43/0x50 [37238.822268] ? lock_release+0x1d2/0x260 [37238.826099] ? lock_release+0xc5/0x260 [37238.829843] _raw_spin_unlock+0x1b/0x40 [37238.833673] igrab+0x43/0x50 [37238.836551] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37238.841702] delayed_work+0x1d8/0x280 [ceph] [37238.845982] ? process_scheduled_works+0x22a/0x510 [37238.850771] process_scheduled_works+0x2a4/0x510 [37238.855387] worker_thread+0x188/0x340 [37238.859155] ? rescuer_thread+0x310/0x310 [37238.863167] kthread+0xf5/0x130 [37238.866313] ? kthread_complete_and_exit+0x20/0x20 [37238.871097] ret_from_fork+0x31/0x40 [37238.874677] ? kthread_complete_and_exit+0x20/0x20 [37238.879460] ret_from_fork_asm+0x11/0x20 [37238.883393] [37266.108227] watchdog: BUG: soft lockup - CPU#0 stuck for 23592s! [dbench:78683] [37266.115534] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37266.120227] watchdog: BUG: soft lockup - CPU#7 stuck for 32589s! [kworker/7:4:78808] [37266.170975] irq event stamp: 65464902 [37266.178704] Modules linked in: ceph libceph [37266.182363] hardirqs last enabled at (65464901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37266.186540] dns_resolver fscache [37266.196698] hardirqs last disabled at (65464902): [] sysvec_apic_timer_interrupt+0xe/0x90 [37266.200008] netfs veth nft_chain_nat [37266.209731] softirqs last enabled at (65464886): [] __do_softirq+0x2be/0x371 [37266.213389] xt_MASQUERADE nf_nat nf_conntrack [37266.222073] softirqs last disabled at (65464881): [] irq_exit_rcu+0xb0/0xd0 [37266.222075] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [37266.235021] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37266.240932] nfnetlink bridge [37266.249877] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37266.249879] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37266.252841] stp llc [37266.260312] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37266.265790] xfs libcrc32c [37266.267973] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37266.286712] sunrpc intel_rapl_msr [37266.289415] [37266.289416] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37266.294632] intel_rapl_common [37266.298030] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37266.299520] sb_edac x86_pkg_temp_thermal [37266.306645] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37266.309694] coretemp [37266.316821] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37266.320823] kvm_intel kvm [37266.327947] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37266.330217] irqbypass i2c_i801 [37266.337343] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37266.340046] iTCO_wdt iTCO_vendor_support [37266.347170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37266.350307] i2c_smbus crct10dif_pclmul [37266.358385] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37266.362390] joydev crc32_pclmul [37266.368127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37266.371958] ipmi_ssif ghash_clmulni_intel [37266.379081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37266.382307] mei_me [37266.389431] Call Trace: [37266.389432] [37266.393520] mei [37266.400646] ? watchdog_timer_fn+0x1b4/0x220 [37266.402742] lpc_ich sha512_ssse3 [37266.405187] ? __hrtimer_run_queues+0x1d3/0x360 [37266.407198] acpi_ipmi mfd_core [37266.409037] ? hrtimer_interrupt+0x104/0x240 [37266.413299] wmi ipmi_si [37266.416626] ? __local_bh_enable+0x50/0x60 [37266.421152] ipmi_devintf [37266.424292] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37266.428552] ipmi_msghandler [37266.431083] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37266.435173] ioatdma [37266.437790] [37266.443007] acpi_pad acpi_power_meter [37266.445903] [37266.445905] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37266.450862] ixgbe [37266.453048] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37266.455143] igb mdio [37266.458905] do_raw_spin_lock+0xb5/0xc0 [37266.461001] crc32c_intel nvme [37266.466306] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37266.468316] ptp nvme_core [37266.473189] try_get_cap_refs+0x159/0x880 [ceph] [37266.475457] i2c_algo_bit pps_core [37266.479293] __ceph_get_caps+0x4a1/0x760 [ceph] [37266.482339] dca [37266.487127] ? __file_remove_privs+0x89/0x110 [37266.489826] fuse [37266.494438] ? wake_bit_function+0x70/0x70 [37266.497836] irq event stamp: 218676736 [37266.502362] ? ceph_write_iter+0x260/0x770 [ceph] [37266.504195] hardirqs last enabled at (218676735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37266.508548] ceph_write_iter+0x260/0x770 [ceph] [37266.510470] hardirqs last disabled at (218676736): [] sysvec_apic_timer_interrupt+0xe/0x90 [37266.514567] ? lock_is_held_type+0xaa/0x120 [37266.518305] softirqs last enabled at (218676734): [] __do_softirq+0x2be/0x371 [37266.523005] vfs_write+0x324/0x4f0 [37266.533247] softirqs last disabled at (218676729): [] irq_exit_rcu+0xb0/0xd0 [37266.537775] ksys_pwrite64+0x65/0xa0 [37266.547584] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37266.551761] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37266.560531] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37266.563929] do_syscall_64+0x38/0x80 [37266.572526] Workqueue: events delayed_work [ceph] [37266.576097] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37266.585474] [37266.590606] RIP: 0033:0x7f48f0eeb93b [37266.598093] RIP: 0010:lock_acquire+0xf0/0x2b0 [37266.601663] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37266.606360] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37266.611404] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37266.612897] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37266.616483] ORIG_RAX: 0000000000000012 [37266.620836] [37266.620836] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37266.639573] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37266.658309] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37266.663527] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37266.668745] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37266.672576] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37266.674067] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37266.681190] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37266.688315] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37266.695439] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37266.702564] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37266.709692] [37266.716809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37266.761306] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37266.768431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37266.775556] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37266.782678] Call Trace: [37266.785123] [37266.787133] ? watchdog_timer_fn+0x1b4/0x220 [37266.791399] ? __hrtimer_run_queues+0x1d3/0x360 [37266.795925] ? hrtimer_interrupt+0x104/0x240 [37266.800195] ? __local_bh_enable+0x50/0x60 [37266.804289] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37266.809512] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37266.814468] [37266.816566] [37266.818664] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37266.823971] ? lock_acquire+0xf0/0x2b0 [37266.827721] ? igrab+0x43/0x50 [37266.830773] ? lock_release+0xc5/0x260 [37266.834517] _raw_spin_lock+0x30/0x40 [37266.838183] ? igrab+0x1d/0x50 [37266.841232] igrab+0x1d/0x50 [37266.844110] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37266.849261] delayed_work+0x1d8/0x280 [ceph] [37266.853542] ? process_scheduled_works+0x22a/0x510 [37266.858333] process_scheduled_works+0x2a4/0x510 [37266.862947] worker_thread+0x188/0x340 [37266.866697] ? rescuer_thread+0x310/0x310 [37266.870707] kthread+0xf5/0x130 [37266.873847] ? kthread_complete_and_exit+0x20/0x20 [37266.878634] ret_from_fork+0x31/0x40 [37266.882210] ? kthread_complete_and_exit+0x20/0x20 [37266.886992] ret_from_fork_asm+0x11/0x20 [37266.890920] [37294.108706] watchdog: BUG: soft lockup - CPU#0 stuck for 23618s! [dbench:78683] [37294.116012] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37294.120706] watchdog: BUG: soft lockup - CPU#7 stuck for 32615s! [kworker/7:4:78808] [37294.171462] irq event stamp: 65523124 [37294.171463] hardirqs last enabled at (65523123): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37294.179192] Modules linked in: [37294.182850] hardirqs last disabled at (65523124): [] sysvec_apic_timer_interrupt+0xe/0x90 [37294.193008] ceph [37294.196057] softirqs last enabled at (65523086): [] __do_softirq+0x2be/0x371 [37294.205782] libceph [37294.207706] softirqs last disabled at (65523081): [] irq_exit_rcu+0xb0/0xd0 [37294.216390] dns_resolver [37294.218576] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37294.227100] fscache [37294.229721] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37294.238665] netfs [37294.240848] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37294.248318] veth nft_chain_nat [37294.250331] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37294.255806] xt_MASQUERADE nf_nat [37294.258944] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37294.277683] nf_conntrack nf_defrag_ipv6 [37294.280995] [37294.286211] nf_defrag_ipv4 nft_compat [37294.290127] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37294.291618] nf_tables [37294.295362] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37294.302487] nfnetlink bridge [37294.304845] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37294.311970] stp llc [37294.314934] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37294.322058] xfs [37294.324240] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37294.331365] libcrc32c sunrpc [37294.333202] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37294.340328] intel_rapl_msr [37294.343289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37294.351383] intel_rapl_common [37294.354177] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37294.359914] sb_edac x86_pkg_temp_thermal [37294.362965] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37294.370105] coretemp [37294.374110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37294.381233] kvm_intel [37294.383504] Call Trace: [37294.383506] [37294.390628] kvm [37294.392988] ? watchdog_timer_fn+0x1b4/0x220 [37294.395429] irqbypass [37294.397443] ? __hrtimer_run_queues+0x1d3/0x360 [37294.399278] i2c_i801 iTCO_wdt [37294.403545] ? hrtimer_interrupt+0x104/0x240 [37294.405900] iTCO_vendor_support [37294.410422] ? __local_bh_enable+0x50/0x60 [37294.413474] i2c_smbus crct10dif_pclmul [37294.417742] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37294.420961] joydev [37294.425053] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37294.428885] crc32_pclmul ipmi_ssif [37294.434105] [37294.436198] ghash_clmulni_intel [37294.441156] [37294.441158] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37294.444640] mei_me [37294.446743] ? queued_spin_lock_slowpath+0x18/0x2e0 [37294.449961] mei [37294.452060] do_raw_spin_lock+0xb5/0xc0 [37294.457363] lpc_ich sha512_ssse3 acpi_ipmi [37294.459479] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37294.464348] mfd_core wmi [37294.466188] try_get_cap_refs+0x159/0x880 [ceph] [37294.470018] ipmi_si [37294.474199] __ceph_get_caps+0x4a1/0x760 [ceph] [37294.478979] ipmi_devintf [37294.481602] ? __file_remove_privs+0x89/0x110 [37294.486207] ipmi_msghandler [37294.488394] ? wake_bit_function+0x70/0x70 [37294.492915] ioatdma [37294.495536] ? ceph_write_iter+0x260/0x770 [ceph] [37294.499881] acpi_pad [37294.502761] ceph_write_iter+0x260/0x770 [ceph] [37294.506852] acpi_power_meter [37294.509040] ? lock_is_held_type+0xaa/0x120 [37294.513732] ixgbe [37294.516004] vfs_write+0x324/0x4f0 [37294.520526] igb mdio [37294.523495] ksys_pwrite64+0x65/0xa0 [37294.527667] crc32c_intel [37294.529679] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37294.533076] nvme ptp [37294.535350] do_syscall_64+0x38/0x80 [37294.538917] nvme_core [37294.541536] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37294.546667] i2c_algo_bit pps_core [37294.548939] RIP: 0033:0x7f48f0eeb93b [37294.552508] dca [37294.554866] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37294.559909] fuse [37294.563306] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37294.566876] [37294.566877] irq event stamp: 218897436 [37294.568714] ORIG_RAX: 0000000000000012 [37294.568716] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37294.587451] hardirqs last enabled at (218897435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37294.589387] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37294.594601] hardirqs last disabled at (218897436): [] sysvec_apic_timer_interrupt+0xe/0x90 [37294.596106] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37294.599855] softirqs last enabled at (218897434): [] __do_softirq+0x2be/0x371 [37294.603686] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37294.610810] softirqs last disabled at (218897429): [] irq_exit_rcu+0xb0/0xd0 [37294.621053] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37294.628178] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37294.637993] [37294.645113] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37294.695764] Workqueue: events delayed_work [ceph] [37294.700477] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [37294.705105] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [37294.723842] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [37294.729060] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [37294.736185] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [37294.743309] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [37294.750432] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [37294.757556] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37294.764680] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37294.772758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37294.778494] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37294.785619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37294.792742] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37294.799867] Call Trace: [37294.802313] [37294.804324] ? watchdog_timer_fn+0x1b4/0x220 [37294.808587] ? __hrtimer_run_queues+0x1d3/0x360 [37294.813116] ? hrtimer_interrupt+0x104/0x240 [37294.817392] ? __local_bh_enable+0x50/0x60 [37294.821484] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37294.826709] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37294.831667] [37294.833764] [37294.835861] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37294.841169] ? do_raw_spin_lock+0x42/0xc0 [37294.845179] igrab+0x1d/0x50 [37294.848057] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37294.853207] delayed_work+0x1d8/0x280 [ceph] [37294.857488] ? process_scheduled_works+0x22a/0x510 [37294.862277] process_scheduled_works+0x2a4/0x510 [37294.866893] worker_thread+0x188/0x340 [37294.870642] ? rescuer_thread+0x310/0x310 [37294.874646] kthread+0xf5/0x130 [37294.877792] ? kthread_complete_and_exit+0x20/0x20 [37294.882578] ret_from_fork+0x31/0x40 [37294.886156] ? kthread_complete_and_exit+0x20/0x20 [37294.890940] ret_from_fork_asm+0x11/0x20 [37294.894873] [37322.109183] watchdog: BUG: soft lockup - CPU#0 stuck for 23644s! [dbench:78683] [37322.116482] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37322.121183] watchdog: BUG: soft lockup - CPU#7 stuck for 32641s! [kworker/7:4:78808] [37322.171925] irq event stamp: 65581490 [37322.171926] hardirqs last enabled at (65581489): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37322.179655] Modules linked in: [37322.183312] hardirqs last disabled at (65581490): [] sysvec_apic_timer_interrupt+0xe/0x90 [37322.193469] ceph [37322.196521] softirqs last enabled at (65581452): [] __do_softirq+0x2be/0x371 [37322.206244] libceph [37322.208169] softirqs last disabled at (65581447): [] irq_exit_rcu+0xb0/0xd0 [37322.216858] dns_resolver fscache [37322.219045] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37322.227555] netfs veth [37322.230883] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37322.230884] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37322.239829] nft_chain_nat [37322.242272] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37322.249744] xt_MASQUERADE nf_nat [37322.255220] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37322.257924] nf_conntrack [37322.276661] [37322.276662] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37322.279973] nf_defrag_ipv6 nf_defrag_ipv4 [37322.285190] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37322.287807] nft_compat [37322.289298] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37322.296422] nf_tables nfnetlink [37322.300514] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37322.307637] bridge stp [37322.310081] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37322.317205] llc xfs [37322.320430] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37322.327553] libcrc32c sunrpc [37322.330014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37322.337141] intel_rapl_msr intel_rapl_common [37322.339323] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37322.347401] sb_edac x86_pkg_temp_thermal [37322.350364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37322.356102] coretemp kvm_intel [37322.360453] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37322.367581] kvm irqbypass [37322.371581] Call Trace: [37322.371582] [37322.378706] i2c_i801 iTCO_wdt [37322.381844] ? watchdog_timer_fn+0x1b4/0x220 [37322.388968] iTCO_vendor_support [37322.391672] ? __hrtimer_run_queues+0x1d3/0x360 [37322.394114] i2c_smbus crct10dif_pclmul [37322.396128] ? hrtimer_interrupt+0x104/0x240 [37322.399177] joydev crc32_pclmul [37322.403441] ? __local_bh_enable+0x50/0x60 [37322.406665] ipmi_ssif ghash_clmulni_intel [37322.411191] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37322.415020] mei_me [37322.419284] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37322.422506] mei lpc_ich [37322.426599] [37322.430689] sha512_ssse3 acpi_ipmi [37322.435906] [37322.435908] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37322.438004] mfd_core wmi [37322.442966] ? queued_spin_lock_slowpath+0x16/0x2e0 [37322.445492] ipmi_si [37322.447593] do_raw_spin_lock+0xb5/0xc0 [37322.451074] ipmi_devintf [37322.453172] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37322.458474] ipmi_msghandler ioatdma [37322.461096] try_get_cap_refs+0x159/0x880 [ceph] [37322.465963] acpi_pad acpi_power_meter [37322.468152] __ceph_get_caps+0x4a1/0x760 [ceph] [37322.471977] ixgbe igb [37322.474600] ? __file_remove_privs+0x89/0x110 [37322.479381] mdio crc32c_intel [37322.482952] ? wake_bit_function+0x70/0x70 [37322.487579] nvme ptp [37322.491324] ? ceph_write_iter+0x260/0x770 [ceph] [37322.495847] nvme_core i2c_algo_bit [37322.498205] ceph_write_iter+0x260/0x770 [ceph] [37322.502555] pps_core dca [37322.505627] ? lock_is_held_type+0xaa/0x120 [37322.509714] fuse [37322.511988] vfs_write+0x324/0x4f0 [37322.516682] irq event stamp: 219118212 [37322.520169] ksys_pwrite64+0x65/0xa0 [37322.524689] hardirqs last enabled at (219118211): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37322.527308] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37322.531485] hardirqs last disabled at (219118212): [] sysvec_apic_timer_interrupt+0xe/0x90 [37322.533410] do_syscall_64+0x38/0x80 [37322.536806] softirqs last enabled at (219118210): [] __do_softirq+0x2be/0x371 [37322.540550] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37322.544121] softirqs last disabled at (219118205): [] irq_exit_rcu+0xb0/0xd0 [37322.554365] RIP: 0033:0x7f48f0eeb93b [37322.559496] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37322.569308] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37322.572878] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37322.581649] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37322.586694] Workqueue: events delayed_work [ceph] [37322.595288] ORIG_RAX: 0000000000000012 [37322.598863] [37322.608237] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37322.626975] RIP: 0010:lock_acquire+0xf0/0x2b0 [37322.634447] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37322.639662] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37322.644362] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37322.648193] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37322.649681] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37322.656806] [37322.656807] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37322.661157] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37322.668283] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37322.687025] [37322.694144] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37322.738657] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37322.745779] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37322.752904] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37322.760981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37322.766718] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37322.773844] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37322.780967] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37322.788093] Call Trace: [37322.790539] [37322.792547] ? watchdog_timer_fn+0x1b4/0x220 [37322.796813] ? __hrtimer_run_queues+0x1d3/0x360 [37322.801339] ? hrtimer_interrupt+0x104/0x240 [37322.805607] ? __local_bh_enable+0x50/0x60 [37322.809702] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37322.814926] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37322.819881] [37322.821980] [37322.824078] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37322.829384] ? lock_acquire+0xf0/0x2b0 [37322.833135] ? igrab+0x43/0x50 [37322.836185] ? lock_release+0xc5/0x260 [37322.839931] _raw_spin_lock+0x30/0x40 [37322.843595] ? igrab+0x1d/0x50 [37322.846645] igrab+0x1d/0x50 [37322.849524] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37322.854675] delayed_work+0x1d8/0x280 [ceph] [37322.858955] ? process_scheduled_works+0x22a/0x510 [37322.863745] process_scheduled_works+0x2a4/0x510 [37322.868359] worker_thread+0x188/0x340 [37322.872110] ? rescuer_thread+0x310/0x310 [37322.876113] kthread+0xf5/0x130 [37322.879251] ? kthread_complete_and_exit+0x20/0x20 [37322.884036] ret_from_fork+0x31/0x40 [37322.887613] ? kthread_complete_and_exit+0x20/0x20 [37322.892397] ret_from_fork_asm+0x11/0x20 [37322.896320] [37350.109661] watchdog: BUG: soft lockup - CPU#0 stuck for 23671s! [dbench:78683] [37350.116961] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37350.121661] watchdog: BUG: soft lockup - CPU#7 stuck for 32668s! [kworker/7:4:78808] [37350.172403] irq event stamp: 65639784 [37350.172405] hardirqs last enabled at (65639783): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37350.180134] Modules linked in: [37350.183791] hardirqs last disabled at (65639784): [] sysvec_apic_timer_interrupt+0xe/0x90 [37350.193948] ceph [37350.196998] softirqs last enabled at (65639778): [] __do_softirq+0x2be/0x371 [37350.206724] libceph [37350.208647] softirqs last disabled at (65639773): [] irq_exit_rcu+0xb0/0xd0 [37350.217337] dns_resolver [37350.219524] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37350.228053] fscache [37350.230669] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37350.239613] netfs veth [37350.241798] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37350.249269] nft_chain_nat [37350.251714] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37350.257191] xt_MASQUERADE nf_nat [37350.259895] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37350.278632] nf_conntrack [37350.281943] [37350.281944] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37350.287159] nf_defrag_ipv6 [37350.289778] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37350.291268] nf_defrag_ipv4 nft_compat [37350.298392] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37350.301183] nf_tables nfnetlink [37350.308307] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37350.312057] bridge stp [37350.319183] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37350.322407] llc [37350.329532] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37350.331975] xfs libcrc32c [37350.339100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37350.340937] sunrpc intel_rapl_msr [37350.349016] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37350.351718] intel_rapl_common [37350.357458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37350.360854] sb_edac [37350.367979] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37350.371028] x86_pkg_temp_thermal coretemp [37350.378154] Call Trace: [37350.380340] kvm_intel kvm [37350.387462] [37350.391552] irqbypass [37350.393996] ? watchdog_timer_fn+0x1b4/0x220 [37350.396701] i2c_i801 [37350.398713] ? __hrtimer_run_queues+0x1d3/0x360 [37350.401067] iTCO_wdt iTCO_vendor_support [37350.405342] ? hrtimer_interrupt+0x104/0x240 [37350.407612] i2c_smbus crct10dif_pclmul [37350.412135] ? __local_bh_enable+0x50/0x60 [37350.416139] joydev crc32_pclmul [37350.420406] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37350.424235] ipmi_ssif ghash_clmulni_intel [37350.428341] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37350.431567] mei_me [37350.436786] [37350.440876] mei lpc_ich [37350.445833] [37350.447928] sha512_ssse3 [37350.450029] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37350.452557] acpi_ipmi mfd_core [37350.454658] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37350.457273] wmi ipmi_si [37350.462578] do_raw_spin_lock+0xb5/0xc0 [37350.465713] ipmi_devintf ipmi_msghandler [37350.470586] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37350.473116] ioatdma acpi_pad [37350.476948] try_get_cap_refs+0x159/0x880 [ceph] [37350.480950] acpi_power_meter [37350.485739] __ceph_get_caps+0x4a1/0x760 [ceph] [37350.488699] ixgbe [37350.493312] ? __file_remove_privs+0x89/0x110 [37350.496275] igb mdio [37350.500797] ? wake_bit_function+0x70/0x70 [37350.502809] crc32c_intel nvme [37350.507162] ? ceph_write_iter+0x260/0x770 [ceph] [37350.509431] ptp nvme_core [37350.513520] ceph_write_iter+0x260/0x770 [ceph] [37350.516572] i2c_algo_bit [37350.521273] ? lock_is_held_type+0xaa/0x120 [37350.523972] pps_core [37350.528500] vfs_write+0x324/0x4f0 [37350.531115] dca fuse [37350.535296] ksys_pwrite64+0x65/0xa0 [37350.537563] [37350.537564] irq event stamp: 219339010 [37350.540960] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37350.543231] hardirqs last enabled at (219339009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37350.546803] do_syscall_64+0x38/0x80 [37350.548290] hardirqs last disabled at (219339010): [] sysvec_apic_timer_interrupt+0xe/0x90 [37350.552036] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37350.557184] softirqs last enabled at (219339008): [] __do_softirq+0x2be/0x371 [37350.567427] RIP: 0033:0x7f48f0eeb93b [37350.570999] softirqs last disabled at (219339003): [] irq_exit_rcu+0xb0/0xd0 [37350.580810] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37350.585853] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37350.594625] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37350.598196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37350.606792] ORIG_RAX: 0000000000000012 [37350.625529] Workqueue: events delayed_work [ceph] [37350.634909] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37350.640124] [37350.647596] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37350.651426] RIP: 0010:lock_acquire+0xf0/0x2b0 [37350.656124] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37350.663249] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37350.664738] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37350.671863] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37350.676214] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37350.683354] [37350.683355] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37350.702099] [37350.709216] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37350.739500] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37350.746623] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37350.753747] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37350.760871] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37350.768950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37350.774687] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37350.781811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37350.788935] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37350.796066] Call Trace: [37350.798513] [37350.800523] ? watchdog_timer_fn+0x1b4/0x220 [37350.804787] ? __hrtimer_run_queues+0x1d3/0x360 [37350.809315] ? hrtimer_interrupt+0x104/0x240 [37350.813584] ? __local_bh_enable+0x50/0x60 [37350.817676] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37350.822891] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37350.827850] [37350.829946] [37350.832044] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37350.837370] ? lock_acquire+0xf0/0x2b0 [37350.841120] ? igrab+0x43/0x50 [37350.844170] ? lock_release+0xc5/0x260 [37350.847915] _raw_spin_lock+0x30/0x40 [37350.851570] ? igrab+0x1d/0x50 [37350.854622] igrab+0x1d/0x50 [37350.857500] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37350.862648] delayed_work+0x1d8/0x280 [ceph] [37350.866930] ? process_scheduled_works+0x22a/0x510 [37350.871720] process_scheduled_works+0x2a4/0x510 [37350.876337] worker_thread+0x188/0x340 [37350.880104] ? rescuer_thread+0x310/0x310 [37350.884115] kthread+0xf5/0x130 [37350.887253] ? kthread_complete_and_exit+0x20/0x20 [37350.892037] ret_from_fork+0x31/0x40 [37350.895632] ? kthread_complete_and_exit+0x20/0x20 [37350.900417] ret_from_fork_asm+0x11/0x20 [37350.904355] [37378.110138] watchdog: BUG: soft lockup - CPU#0 stuck for 23697s! [dbench:78683] [37378.117442] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37378.122138] watchdog: BUG: soft lockup - CPU#7 stuck for 32694s! [kworker/7:4:78808] [37378.172891] irq event stamp: 65698028 [37378.180622] Modules linked in: ceph libceph dns_resolver fscache netfs [37378.184279] hardirqs last enabled at (65698027): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37378.190812] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [37378.200971] hardirqs last disabled at (65698028): [] sysvec_apic_timer_interrupt+0xe/0x90 [37378.207054] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [37378.216779] softirqs last enabled at (65697990): [] __do_softirq+0x2be/0x371 [37378.216784] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [37378.233220] softirqs last disabled at (65697985): [] irq_exit_rcu+0xb0/0xd0 [37378.233224] x86_pkg_temp_thermal [37378.233226] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37378.240440] coretemp kvm_intel [37378.248952] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37378.252261] kvm irqbypass [37378.261206] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37378.264343] i2c_i801 iTCO_wdt [37378.271814] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37378.274534] iTCO_vendor_support [37378.280013] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37378.283064] i2c_smbus crct10dif_pclmul [37378.301801] [37378.305026] joydev crc32_pclmul [37378.310243] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37378.314073] ipmi_ssif ghash_clmulni_intel [37378.315564] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37378.318786] mei_me mei [37378.325912] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37378.330003] lpc_ich sha512_ssse3 [37378.337127] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37378.339571] acpi_ipmi mfd_core [37378.346694] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37378.350007] wmi ipmi_si [37378.357129] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37378.360268] ipmi_devintf ipmi_msghandler [37378.367392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37378.369923] ioatdma acpi_pad [37378.378000] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37378.382005] acpi_power_meter [37378.387742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37378.390705] ixgbe [37378.397830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37378.400794] igb mdio [37378.407934] Call Trace: [37378.407936] [37378.409946] crc32c_intel nvme [37378.417072] ? watchdog_timer_fn+0x1b4/0x220 [37378.419339] ptp nvme_core [37378.421786] ? __hrtimer_run_queues+0x1d3/0x360 [37378.423794] i2c_algo_bit pps_core [37378.426867] ? hrtimer_interrupt+0x104/0x240 [37378.431127] dca fuse [37378.433831] ? __local_bh_enable+0x50/0x60 [37378.438354] [37378.438355] irq event stamp: 219559654 [37378.441756] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37378.446016] hardirqs last enabled at (219559653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37378.448288] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37378.452378] hardirqs last disabled at (219559654): [] sysvec_apic_timer_interrupt+0xe/0x90 [37378.453869] [37378.457614] softirqs last enabled at (219559652): [] __do_softirq+0x2be/0x371 [37378.462829] [37378.462831] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37378.473075] softirqs last disabled at (219559647): [] irq_exit_rcu+0xb0/0xd0 [37378.478035] ? queued_spin_lock_slowpath+0x16/0x2e0 [37378.487843] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37378.489941] do_raw_spin_lock+0xb5/0xc0 [37378.498712] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37378.500826] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37378.506130] Workqueue: events delayed_work [ceph] [37378.514730] try_get_cap_refs+0x159/0x880 [ceph] [37378.519598] [37378.528980] __ceph_get_caps+0x4a1/0x760 [ceph] [37378.532807] RIP: 0010:lock_acquire+0xf0/0x2b0 [37378.540298] ? __file_remove_privs+0x89/0x110 [37378.545078] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37378.549777] ? wake_bit_function+0x70/0x70 [37378.554388] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37378.555881] ? ceph_write_iter+0x260/0x770 [ceph] [37378.560402] [37378.560403] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37378.564752] ceph_write_iter+0x260/0x770 [ceph] [37378.569104] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37378.587846] ? lock_is_held_type+0xaa/0x120 [37378.591932] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37378.597150] vfs_write+0x324/0x4f0 [37378.601845] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37378.603340] ksys_pwrite64+0x65/0xa0 [37378.610461] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37378.614985] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37378.622110] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37378.626287] do_syscall_64+0x38/0x80 [37378.633411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37378.636810] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37378.643931] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37378.647505] RIP: 0033:0x7f48f0eeb93b [37378.654628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37378.659760] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37378.667835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37378.671405] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37378.677145] Call Trace: [37378.677146] [37378.682187] ORIG_RAX: 0000000000000012 [37378.689312] ? watchdog_timer_fn+0x1b4/0x220 [37378.692883] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37378.700007] ? __hrtimer_run_queues+0x1d3/0x360 [37378.718745] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37378.725873] ? hrtimer_interrupt+0x104/0x240 [37378.731085] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37378.733541] ? __local_bh_enable+0x50/0x60 [37378.735549] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37378.739382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37378.743645] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37378.750770] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37378.755299] [37378.762416] [37378.806600] [37378.808697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37378.814006] ? lock_acquire+0xf0/0x2b0 [37378.817756] ? igrab+0x43/0x50 [37378.820821] ? lock_release+0xc5/0x260 [37378.824568] _raw_spin_lock+0x30/0x40 [37378.828234] ? igrab+0x1d/0x50 [37378.831283] igrab+0x1d/0x50 [37378.834164] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37378.839311] delayed_work+0x1d8/0x280 [ceph] [37378.843592] ? process_scheduled_works+0x22a/0x510 [37378.848384] process_scheduled_works+0x2a4/0x510 [37378.852998] worker_thread+0x188/0x340 [37378.856748] ? rescuer_thread+0x310/0x310 [37378.860750] kthread+0xf5/0x130 [37378.863888] ? kthread_complete_and_exit+0x20/0x20 [37378.868674] ret_from_fork+0x31/0x40 [37378.872254] ? kthread_complete_and_exit+0x20/0x20 [37378.877035] ret_from_fork_asm+0x11/0x20 [37378.880968] [37406.110613] watchdog: BUG: soft lockup - CPU#0 stuck for 23723s! [dbench:78683] [37406.117910] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37406.122613] watchdog: BUG: soft lockup - CPU#7 stuck for 32720s! [kworker/7:4:78808] [37406.173353] irq event stamp: 65756148 [37406.173354] hardirqs last enabled at (65756147): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37406.181082] Modules linked in: [37406.184739] hardirqs last disabled at (65756148): [] sysvec_apic_timer_interrupt+0xe/0x90 [37406.194898] ceph [37406.197948] softirqs last enabled at (65756110): [] __do_softirq+0x2be/0x371 [37406.207672] libceph [37406.209597] softirqs last disabled at (65756105): [] irq_exit_rcu+0xb0/0xd0 [37406.218286] dns_resolver fscache [37406.220474] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37406.228984] netfs [37406.232313] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37406.241255] veth nft_chain_nat [37406.243268] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37406.250757] xt_MASQUERADE nf_nat [37406.253893] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37406.259370] nf_conntrack nf_defrag_ipv6 [37406.262681] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37406.281419] nf_defrag_ipv4 nft_compat [37406.285336] [37406.285337] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37406.286983] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [37406.286986] rcu: 0-....: (24893741 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=12466076 [37406.287007] rcu: 7-....: (24729339 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/24727984 fqs=12466076 [37406.287010] rcu: (detected by 3, t=25469214 jiffies, g=1309609, q=697212 ncpus=8) [37406.287012] Sending NMI from CPU 3 to CPUs 0: [37406.287015] NMI backtrace for cpu 0 [37406.287016] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37406.287017] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37406.287018] RIP: 0010:vprintk_emit+0x1ee/0x340 [37406.287020] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [37406.287022] RSP: 0018:ffffc90000003db8 EFLAGS: 00000002 [37406.287023] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [37406.287023] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [37406.287024] RBP: 0000000000000041 R08: 0000000000005ef4 R09: 0000000000000004 [37406.287025] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [37406.287025] R13: 0000000000000000 R14: ffffffff821886d8 R15: ffffc90000003e08 [37406.287026] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37406.287027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37406.287028] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37406.287029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37406.287030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37406.287030] Call Trace: [37406.287031] [37406.287032] ? nmi_cpu_backtrace+0xad/0x120 [37406.287036] ? nmi_cpu_backtrace_handler+0x11/0x20 [37406.287038] ? nmi_handle+0xd4/0x260 [37406.287042] ? default_do_nmi+0x49/0xf0 [37406.287043] ? exc_nmi+0x11e/0x150 [37406.287045] ? end_repeat_nmi+0x16/0x67 [37406.287049] ? vprintk_emit+0x1ee/0x340 [37406.287051] ? vprintk_emit+0x1ee/0x340 [37406.287053] ? vprintk_emit+0x1ee/0x340 [37406.287054] [37406.287055] [37406.287056] ? touch_softlockup_watchdog+0x20/0x20 [37406.287057] _printk+0x5c/0x80 [37406.287060] __show_regs+0x5d/0x2e0 [37406.287063] ? touch_softlockup_watchdog+0x20/0x20 [37406.287065] show_regs+0x38/0x60 [37406.287067] watchdog_timer_fn+0x1b4/0x220 [37406.287069] __hrtimer_run_queues+0x1d3/0x360 [37406.287072] hrtimer_interrupt+0x104/0x240 [37406.287073] ? __local_bh_enable+0x50/0x60 [37406.287076] __sysvec_apic_timer_interrupt+0x61/0x160 [37406.287079] sysvec_apic_timer_interrupt+0x6d/0x90 [37406.287081] [37406.287081] [37406.287082] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37406.287084] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37406.287086] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37406.287087] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37406.287088] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37406.287089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37406.287089] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37406.287090] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37406.287091] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37406.287094] do_raw_spin_lock+0xb5/0xc0 [37406.287096] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37406.287111] try_get_cap_refs+0x159/0x880 [ceph] [37406.287125] __ceph_get_caps+0x4a1/0x760 [ceph] [37406.287138] ? __file_remove_privs+0x89/0x110 [37406.287140] ? wake_bit_function+0x70/0x70 [37406.287142] ? ceph_write_iter+0x260/0x770 [ceph] [37406.287153] ceph_write_iter+0x260/0x770 [ceph] [37406.287164] ? lock_is_held_type+0xaa/0x120 [37406.287167] vfs_write+0x324/0x4f0 [37406.287171] ksys_pwrite64+0x65/0xa0 [37406.287173] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37406.287176] do_syscall_64+0x38/0x80 [37406.287178] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37406.287180] RIP: 0033:0x7f48f0eeb93b [37406.287181] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37406.287182] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [37406.287184] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37406.287184] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37406.287185] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37406.287186] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37406.287187] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37406.287190] [37406.288014] Sending NMI from CPU 3 to CPUs 7: [37406.288017] NMI backtrace for cpu 7 [37406.288018] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37406.288020] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37406.288021] Workqueue: events delayed_work [ceph] [37406.288042] RIP: 0010:delay_tsc+0x3b/0x90 [37406.288045] Code: 44 8b 0d 54 f6 49 7e 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d0 eb 21 65 ff 0d 38 f6 49 7e 74 50 f3 90 65 ff 05 2d f6 49 7e <65> 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 66 90 48 c1 [37406.288047] RSP: 0018:ffffc90000264c98 EFLAGS: 00000007 [37406.288048] RAX: 000078ac4cde2c69 RBX: 0000000000000000 RCX: 0000000000000007 [37406.288049] RDX: 0000000000000cf1 RSI: 0000000000000007 RDI: 0000000000000dac [37406.288051] RBP: 0000000000002701 R08: 000078ac4cde1f78 R09: 0000000000000007 [37406.288052] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [37406.288053] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [37406.288054] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37406.288056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37406.288057] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37406.288059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37406.288060] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37406.288061] Call Trace: [37406.288062] [37406.288063] ? nmi_cpu_backtrace+0xad/0x120 [37406.288067] ? nmi_cpu_backtrace_handler+0x11/0x20 [37406.288069] ? nmi_handle+0xd4/0x260 [37406.288074] ? default_do_nmi+0x49/0xf0 [37406.288076] ? exc_nmi+0x11e/0x150 [37406.288079] ? end_repeat_nmi+0x16/0x67 [37406.288085] ? delay_tsc+0x3b/0x90 [37406.288088] ? delay_tsc+0x3b/0x90 [37406.288091] ? delay_tsc+0x3b/0x90 [37406.288094] [37406.288094] [37406.288095] wait_for_lsr+0x27/0x70 [37406.288099] serial8250_console_write+0x341/0x3f0 [37406.288103] ? lock_release+0xc5/0x260 [37406.288107] console_flush_all+0x227/0x5a0 [37406.288110] ? console_flush_all+0x1ff/0x5a0 [37406.288112] ? console_flush_all+0x1ff/0x5a0 [37406.288116] console_unlock+0x7b/0x120 [37406.288120] vprintk_emit+0x246/0x340 [37406.288122] ? irq_work_queue+0x2c/0x50 [37406.288127] ? touch_softlockup_watchdog+0x20/0x20 [37406.288129] _printk+0x5c/0x80 [37406.288134] print_modules+0x6e/0xe0 [37406.288137] watchdog_timer_fn+0x19f/0x220 [37406.288140] __hrtimer_run_queues+0x1d3/0x360 [37406.288145] hrtimer_interrupt+0x104/0x240 [37406.288146] ? __local_bh_enable+0x50/0x60 [37406.288151] __sysvec_apic_timer_interrupt+0x61/0x160 [37406.288155] sysvec_apic_timer_interrupt+0x6d/0x90 [37406.288158] [37406.288158] [37406.288159] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37406.288162] RIP: 0010:lock_acquire+0xf0/0x2b0 [37406.288164] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37406.288166] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37406.288167] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37406.288168] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37406.288170] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37406.288171] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37406.288172] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37406.288178] ? igrab+0x43/0x50 [37406.288180] ? lock_release+0xc5/0x260 [37406.288184] _raw_spin_lock+0x30/0x40 [37406.288186] ? igrab+0x1d/0x50 [37406.288188] igrab+0x1d/0x50 [37406.288190] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37406.288210] delayed_work+0x1d8/0x280 [ceph] [37406.288229] ? process_scheduled_works+0x22a/0x510 [37406.288233] process_scheduled_works+0x2a4/0x510 [37406.288239] worker_thread+0x188/0x340 [37406.288242] ? rescuer_thread+0x310/0x310 [37406.288245] kthread+0xf5/0x130 [37406.288248] ? kthread_complete_and_exit+0x20/0x20 [37406.288252] ret_from_fork+0x31/0x40 [37406.288254] ? kthread_complete_and_exit+0x20/0x20 [37406.288257] ret_from_fork_asm+0x11/0x20 [37406.288264] [37406.290554] nf_tables nfnetlink [37406.294299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37406.295789] bridge stp [37406.302912] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37406.309009] llc xfs [37406.319518] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37406.330196] libcrc32c sunrpc [37406.337754] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37406.342103] intel_rapl_msr intel_rapl_common [37406.345587] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37406.354531] sb_edac x86_pkg_temp_thermal [37406.362011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37406.366449] coretemp kvm_intel [37406.385186] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37406.390405] kvm [37406.397528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37406.404653] irqbypass i2c_i801 [37406.411777] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37406.418899] iTCO_wdt iTCO_vendor_support [37406.426025] Call Trace: [37406.426026] [37406.434103] i2c_smbus crct10dif_pclmul [37406.439839] ? watchdog_timer_fn+0x1b4/0x220 [37406.446963] joydev crc32_pclmul [37406.454088] ? __hrtimer_run_queues+0x1d3/0x360 [37406.461212] ipmi_ssif [37406.463658] ? hrtimer_interrupt+0x104/0x240 [37406.465668] ghash_clmulni_intel mei_me [37406.469843] ? __local_bh_enable+0x50/0x60 [37406.474627] mei lpc_ich [37406.478201] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37406.482028] sha512_ssse3 [37406.485428] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37406.489259] acpi_ipmi [37406.493088] [37406.496920] mfd_core wmi [37406.500750] [37406.502848] ipmi_si ipmi_devintf [37406.504858] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37406.509643] ipmi_msghandler ioatdma [37406.512698] ? queued_spin_lock_slowpath+0x16/0x2e0 [37406.516176] acpi_pad acpi_power_meter [37406.520964] do_raw_spin_lock+0xb5/0xc0 [37406.524185] ixgbe [37406.528292] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37406.532643] igb mdio [37406.536737] try_get_cap_refs+0x159/0x880 [ceph] [37406.540825] crc32c_intel [37406.545875] __ceph_get_caps+0x4a1/0x760 [ceph] [37406.550654] nvme [37406.552756] ? __file_remove_privs+0x89/0x110 [37406.554848] ptp nvme_core [37406.559981] ? wake_bit_function+0x70/0x70 [37406.565455] i2c_algo_bit pps_core [37406.584197] ? ceph_write_iter+0x260/0x770 [ceph] [37406.589412] dca fuse [37406.596537] ceph_write_iter+0x260/0x770 [ceph] [37406.603659] [37406.610789] ? lock_is_held_type+0xaa/0x120 [37406.617909] irq event stamp: 219780680 [37406.617910] hardirqs last enabled at (219780679): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37406.625036] vfs_write+0x324/0x4f0 [37406.628862] hardirqs last disabled at (219780680): [] sysvec_apic_timer_interrupt+0xe/0x90 [37406.633653] ksys_pwrite64+0x65/0xa0 [37406.638257] softirqs last enabled at (219780678): [] __do_softirq+0x2be/0x371 [37406.642782] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37406.647132] softirqs last disabled at (219780673): [] irq_exit_rcu+0xb0/0xd0 [37406.651224] do_syscall_64+0x38/0x80 [37406.655921] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37406.660446] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37406.664623] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37406.668020] RIP: 0033:0x7f48f0eeb93b [37406.671590] Workqueue: events delayed_work [ceph] [37406.676721] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37406.680295] [37406.680296] RIP: 0010:lock_acquire+0xf0/0x2b0 [37406.685335] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37406.688906] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37406.707644] ORIG_RAX: 0000000000000012 [37406.715202] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37406.722327] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37406.729451] [37406.736573] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37406.743698] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37406.750822] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37406.753017] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37406.757367] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37406.760850] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37406.770228] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37406.777698] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37406.782402] [37406.786400] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37407.588627] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37407.596705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37407.602443] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37407.609566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37407.616692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37407.623814] Call Trace: [37407.626259] [37407.628270] ? watchdog_timer_fn+0x1b4/0x220 [37407.632534] ? __hrtimer_run_queues+0x1d3/0x360 [37407.637063] ? hrtimer_interrupt+0x104/0x240 [37407.641330] ? __local_bh_enable+0x50/0x60 [37407.645423] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37407.650649] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37407.655604] [37407.657703] [37407.659801] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37407.665109] ? lock_acquire+0xf0/0x2b0 [37407.668858] ? igrab+0x43/0x50 [37407.671907] ? lock_release+0xc5/0x260 [37407.675653] _raw_spin_lock+0x30/0x40 [37407.679315] ? igrab+0x1d/0x50 [37407.682368] igrab+0x1d/0x50 [37407.685246] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37407.690395] delayed_work+0x1d8/0x280 [ceph] [37407.694676] ? process_scheduled_works+0x22a/0x510 [37407.699458] process_scheduled_works+0x2a4/0x510 [37407.704075] worker_thread+0x188/0x340 [37407.707825] ? rescuer_thread+0x310/0x310 [37407.711834] kthread+0xf5/0x130 [37407.714974] ? kthread_complete_and_exit+0x20/0x20 [37407.719757] ret_from_fork+0x31/0x40 [37407.723327] ? kthread_complete_and_exit+0x20/0x20 [37407.728112] ret_from_fork_asm+0x11/0x20 [37407.732038] [37434.111087] watchdog: BUG: soft lockup - CPU#0 stuck for 23749s! [dbench:78683] [37434.118391] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37434.123087] watchdog: BUG: soft lockup - CPU#7 stuck for 32746s! [kworker/7:4:78808] [37434.173841] irq event stamp: 65814202 [37434.181570] Modules linked in: ceph libceph dns_resolver fscache netfs veth [37434.185227] hardirqs last enabled at (65814201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37434.192188] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [37434.202344] hardirqs last disabled at (65814202): [] sysvec_apic_timer_interrupt+0xe/0x90 [37434.202347] nf_defrag_ipv6 [37434.202349] softirqs last enabled at (65814198): [] __do_softirq+0x2be/0x371 [37434.207996] nf_defrag_ipv4 nft_compat [37434.217721] softirqs last disabled at (65814193): [] irq_exit_rcu+0xb0/0xd0 [37434.220511] nf_tables [37434.229196] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37434.232937] nfnetlink bridge [37434.241449] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37434.243807] stp llc [37434.252752] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37434.255714] xfs libcrc32c [37434.263185] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37434.265371] sunrpc [37434.270847] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37434.273551] intel_rapl_msr intel_rapl_common [37434.292290] [37434.294385] sb_edac x86_pkg_temp_thermal [37434.299604] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37434.303954] coretemp [37434.305447] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37434.309449] kvm_intel kvm [37434.316574] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37434.318844] irqbypass [37434.325968] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37434.328672] i2c_i801 [37434.335796] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37434.338153] iTCO_wdt [37434.345279] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37434.347549] iTCO_vendor_support i2c_smbus [37434.354674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37434.356944] crct10dif_pclmul joydev [37434.365022] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37434.369112] crc32_pclmul ipmi_ssif [37434.374849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37434.378420] ghash_clmulni_intel [37434.385544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37434.389028] mei_me [37434.396152] Call Trace: [37434.396155] [37434.399377] mei [37434.406504] ? watchdog_timer_fn+0x1b4/0x220 [37434.408598] lpc_ich sha512_ssse3 [37434.411042] ? __hrtimer_run_queues+0x1d3/0x360 [37434.413054] acpi_ipmi [37434.414892] ? hrtimer_interrupt+0x104/0x240 [37434.419155] mfd_core wmi [37434.422464] ? __local_bh_enable+0x50/0x60 [37434.427006] ipmi_si ipmi_devintf [37434.429365] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37434.433627] ipmi_msghandler [37434.436245] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37434.440336] ioatdma [37434.443646] [37434.448864] acpi_pad [37434.451741] [37434.456699] acpi_power_meter [37434.458884] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37434.460980] ixgbe [37434.463256] ? queued_spin_lock_slowpath+0x18/0x2e0 [37434.465349] igb mdio [37434.468314] do_raw_spin_lock+0xb5/0xc0 [37434.473617] crc32c_intel [37434.475629] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37434.480497] nvme [37434.482784] try_get_cap_refs+0x159/0x880 [ceph] [37434.486609] ptp nvme_core [37434.489230] __ceph_get_caps+0x4a1/0x760 [ceph] [37434.494008] i2c_algo_bit pps_core [37434.495937] ? __file_remove_privs+0x89/0x110 [37434.500545] dca [37434.503249] ? wake_bit_function+0x70/0x70 [37434.507789] fuse [37434.511190] ? ceph_write_iter+0x260/0x770 [ceph] [37434.515538] [37434.517376] ceph_write_iter+0x260/0x770 [ceph] [37434.521466] irq event stamp: 219994672 [37434.523411] ? lock_is_held_type+0xaa/0x120 [37434.528105] hardirqs last enabled at (219994671): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37434.529597] vfs_write+0x324/0x4f0 [37434.534119] hardirqs last disabled at (219994672): [] sysvec_apic_timer_interrupt+0xe/0x90 [37434.537869] ksys_pwrite64+0x65/0xa0 [37434.542040] softirqs last enabled at (219994670): [] __do_softirq+0x2be/0x371 [37434.552284] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37434.555681] softirqs last disabled at (219994665): [] irq_exit_rcu+0xb0/0xd0 [37434.565496] do_syscall_64+0x38/0x80 [37434.569063] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37434.577837] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37434.582965] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37434.591563] RIP: 0033:0x7f48f0eeb93b [37434.595134] Workqueue: events delayed_work [ceph] [37434.604511] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37434.609554] [37434.617027] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37434.620596] RIP: 0010:lock_acquire+0xf0/0x2b0 [37434.625296] ORIG_RAX: 0000000000000012 [37434.644032] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37434.645523] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37434.650741] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37434.655090] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37434.658921] [37434.677659] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37434.684784] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37434.690001] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37434.697126] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37434.698616] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37434.705740] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37434.712869] [37434.719989] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37434.750669] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37434.757796] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37434.765869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37434.771608] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37434.778731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37434.785856] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37434.792981] Call Trace: [37434.795427] [37434.797436] ? watchdog_timer_fn+0x1b4/0x220 [37434.801701] ? __hrtimer_run_queues+0x1d3/0x360 [37434.806227] ? hrtimer_interrupt+0x104/0x240 [37434.810493] ? __local_bh_enable+0x50/0x60 [37434.814589] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37434.819813] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37434.824781] [37434.826876] [37434.828974] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37434.834283] ? lock_acquire+0xf0/0x2b0 [37434.838032] ? igrab+0x43/0x50 [37434.841081] ? lock_release+0xc5/0x260 [37434.844827] _raw_spin_lock+0x30/0x40 [37434.848494] ? igrab+0x1d/0x50 [37434.851542] igrab+0x1d/0x50 [37434.854423] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37434.859570] delayed_work+0x1d8/0x280 [ceph] [37434.863851] ? process_scheduled_works+0x22a/0x510 [37434.868642] process_scheduled_works+0x2a4/0x510 [37434.873257] worker_thread+0x188/0x340 [37434.877006] ? rescuer_thread+0x310/0x310 [37434.881011] kthread+0xf5/0x130 [37434.884156] ? kthread_complete_and_exit+0x20/0x20 [37434.888939] ret_from_fork+0x31/0x40 [37434.892521] ? kthread_complete_and_exit+0x20/0x20 [37434.897304] ret_from_fork_asm+0x11/0x20 [37434.901237] [37462.111558] watchdog: BUG: soft lockup - CPU#0 stuck for 23775s! [dbench:78683] [37462.118860] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37462.123558] watchdog: BUG: soft lockup - CPU#7 stuck for 32772s! [kworker/7:4:78808] [37462.174301] irq event stamp: 65872376 [37462.174302] hardirqs last enabled at (65872375): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37462.182032] Modules linked in: [37462.185690] hardirqs last disabled at (65872376): [] sysvec_apic_timer_interrupt+0xe/0x90 [37462.195848] ceph [37462.198898] softirqs last enabled at (65872358): [] __do_softirq+0x2be/0x371 [37462.208621] libceph [37462.210546] softirqs last disabled at (65872353): [] irq_exit_rcu+0xb0/0xd0 [37462.219243] dns_resolver fscache [37462.221431] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37462.229951] netfs veth [37462.233262] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37462.242207] nft_chain_nat [37462.244667] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37462.252137] xt_MASQUERADE nf_nat [37462.254841] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37462.260320] nf_conntrack nf_defrag_ipv6 [37462.263630] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37462.282367] nf_defrag_ipv4 nft_compat [37462.286285] [37462.291503] nf_tables nfnetlink [37462.295246] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37462.296737] bridge stp [37462.299960] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37462.307086] llc xfs [37462.309529] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37462.316653] libcrc32c sunrpc [37462.318837] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37462.325961] intel_rapl_msr [37462.328927] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37462.336049] intel_rapl_common sb_edac [37462.338841] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37462.345965] x86_pkg_temp_thermal [37462.349709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37462.357788] coretemp kvm_intel [37462.361097] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37462.366833] kvm irqbypass [37462.369971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37462.377097] i2c_i801 iTCO_wdt [37462.379799] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37462.386925] iTCO_vendor_support i2c_smbus [37462.389974] Call Trace: [37462.389975] [37462.397099] crct10dif_pclmul [37462.401190] ? watchdog_timer_fn+0x1b4/0x220 [37462.403633] joydev crc32_pclmul [37462.405646] ? __hrtimer_run_queues+0x1d3/0x360 [37462.408609] ipmi_ssif ghash_clmulni_intel [37462.412877] ? hrtimer_interrupt+0x104/0x240 [37462.416096] mei_me mei [37462.420621] ? __local_bh_enable+0x50/0x60 [37462.424712] lpc_ich sha512_ssse3 [37462.428977] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37462.431419] acpi_ipmi mfd_core [37462.435511] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37462.438822] wmi ipmi_si [37462.444039] [37462.447176] ipmi_devintf [37462.452134] [37462.454665] ipmi_msghandler ioatdma [37462.456763] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37462.459379] acpi_pad [37462.461480] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37462.465047] acpi_power_meter [37462.470353] do_raw_spin_lock+0xb5/0xc0 [37462.472622] ixgbe igb [37462.477494] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37462.480457] mdio crc32c_intel [37462.484291] try_get_cap_refs+0x159/0x880 [ceph] [37462.486645] nvme ptp [37462.491433] __ceph_get_caps+0x4a1/0x760 [ceph] [37462.494479] nvme_core i2c_algo_bit [37462.499094] ? __file_remove_privs+0x89/0x110 [37462.501362] pps_core [37462.505886] ? wake_bit_function+0x70/0x70 [37462.509368] dca fuse [37462.513723] ? ceph_write_iter+0x260/0x770 [ceph] [37462.515991] [37462.520081] ceph_write_iter+0x260/0x770 [ceph] [37462.522351] irq event stamp: 220215468 [37462.527053] ? lock_is_held_type+0xaa/0x120 [37462.528540] hardirqs last enabled at (220215467): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37462.533066] vfs_write+0x324/0x4f0 [37462.536808] hardirqs last disabled at (220215468): [] sysvec_apic_timer_interrupt+0xe/0x90 [37462.540991] ksys_pwrite64+0x65/0xa0 [37462.551238] softirqs last enabled at (220215466): [] __do_softirq+0x2be/0x371 [37462.554637] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37462.564448] softirqs last disabled at (220215461): [] irq_exit_rcu+0xb0/0xd0 [37462.568019] do_syscall_64+0x38/0x80 [37462.576788] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37462.581920] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37462.590518] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37462.594088] RIP: 0033:0x7f48f0eeb93b [37462.603464] Workqueue: events delayed_work [ceph] [37462.608511] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37462.615979] [37462.615980] RIP: 0010:lock_acquire+0xf0/0x2b0 [37462.619551] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37462.624247] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37462.642987] ORIG_RAX: 0000000000000012 [37462.644476] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37462.648828] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37462.654043] [37462.654044] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37462.672782] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37462.676613] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37462.681830] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37462.688960] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37462.690453] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37462.697578] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37462.704701] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37462.711826] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37462.718967] [37462.726082] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37462.764840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37462.770579] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37462.777704] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37462.784827] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37462.791961] Call Trace: [37462.794403] [37462.796416] ? watchdog_timer_fn+0x1b4/0x220 [37462.800679] ? __hrtimer_run_queues+0x1d3/0x360 [37462.805205] ? hrtimer_interrupt+0x104/0x240 [37462.809466] ? __local_bh_enable+0x50/0x60 [37462.813561] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37462.818785] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37462.823740] [37462.825840] [37462.827938] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37462.833271] ? lock_acquire+0xf0/0x2b0 [37462.837020] ? igrab+0x43/0x50 [37462.840071] ? lock_release+0xc5/0x260 [37462.843815] _raw_spin_lock+0x30/0x40 [37462.847472] ? igrab+0x1d/0x50 [37462.850522] igrab+0x1d/0x50 [37462.853400] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37462.858549] delayed_work+0x1d8/0x280 [ceph] [37462.862831] ? process_scheduled_works+0x22a/0x510 [37462.867622] process_scheduled_works+0x2a4/0x510 [37462.872252] worker_thread+0x188/0x340 [37462.876004] ? rescuer_thread+0x310/0x310 [37462.880016] kthread+0xf5/0x130 [37462.883152] ? kthread_complete_and_exit+0x20/0x20 [37462.887938] ret_from_fork+0x31/0x40 [37462.891532] ? kthread_complete_and_exit+0x20/0x20 [37462.896319] ret_from_fork_asm+0x11/0x20 [37462.900251] [37490.112027] watchdog: BUG: soft lockup - CPU#0 stuck for 23801s! [dbench:78683] [37490.119331] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37490.124027] watchdog: BUG: soft lockup - CPU#7 stuck for 32798s! [kworker/7:4:78808] [37490.174780] irq event stamp: 65930592 [37490.174781] hardirqs last enabled at (65930591): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37490.182510] Modules linked in: [37490.186167] hardirqs last disabled at (65930592): [] sysvec_apic_timer_interrupt+0xe/0x90 [37490.196326] ceph [37490.199377] softirqs last enabled at (65930574): [] __do_softirq+0x2be/0x371 [37490.209100] libceph [37490.211025] softirqs last disabled at (65930569): [] irq_exit_rcu+0xb0/0xd0 [37490.219709] dns_resolver [37490.221895] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37490.230421] fscache netfs [37490.233039] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37490.241981] veth nft_chain_nat [37490.244701] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37490.252176] xt_MASQUERADE nf_nat [37490.255313] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37490.260790] nf_conntrack nf_defrag_ipv6 [37490.264100] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37490.282839] nf_defrag_ipv4 [37490.286755] [37490.291973] nft_compat [37490.294764] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37490.296255] nf_tables [37490.298698] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37490.305841] nfnetlink bridge [37490.308196] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37490.315321] stp [37490.318284] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37490.325421] llc [37490.327256] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37490.334379] xfs libcrc32c [37490.336218] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37490.343342] sunrpc [37490.346045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37490.354123] intel_rapl_msr [37490.356221] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37490.361958] intel_rapl_common [37490.364749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37490.371873] sb_edac x86_pkg_temp_thermal [37490.374922] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37490.382048] coretemp kvm_intel [37490.386051] Call Trace: [37490.386053] [37490.393175] kvm [37490.396316] ? watchdog_timer_fn+0x1b4/0x220 [37490.398756] irqbypass i2c_i801 [37490.400769] ? __hrtimer_run_queues+0x1d3/0x360 [37490.402604] iTCO_wdt [37490.406871] ? hrtimer_interrupt+0x104/0x240 [37490.410006] iTCO_vendor_support i2c_smbus [37490.414530] ? __local_bh_enable+0x50/0x60 [37490.416801] crct10dif_pclmul joydev [37490.421066] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37490.425156] crc32_pclmul ipmi_ssif [37490.429247] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37490.432816] ghash_clmulni_intel mei_me [37490.438034] [37490.441518] mei lpc_ich [37490.446476] [37490.446477] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37490.450306] sha512_ssse3 acpi_ipmi [37490.452425] ? queued_spin_lock_slowpath+0x16/0x2e0 [37490.454952] mfd_core wmi [37490.457052] do_raw_spin_lock+0xb5/0xc0 [37490.462354] ipmi_si [37490.465839] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37490.470708] ipmi_devintf [37490.473328] try_get_cap_refs+0x159/0x880 [ceph] [37490.477158] ipmi_msghandler ioatdma [37490.479345] __ceph_get_caps+0x4a1/0x760 [ceph] [37490.484126] acpi_pad [37490.486747] ? __file_remove_privs+0x89/0x110 [37490.491354] acpi_power_meter [37490.494924] ? wake_bit_function+0x70/0x70 [37490.499448] ixgbe [37490.501722] ? ceph_write_iter+0x260/0x770 [ceph] [37490.506069] igb mdio [37490.509035] ceph_write_iter+0x260/0x770 [ceph] [37490.513125] crc32c_intel [37490.515139] ? lock_is_held_type+0xaa/0x120 [37490.519833] nvme ptp [37490.522105] vfs_write+0x324/0x4f0 [37490.526627] nvme_core [37490.529248] ksys_pwrite64+0x65/0xa0 [37490.533426] i2c_algo_bit [37490.535693] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37490.539107] pps_core dca [37490.541466] do_syscall_64+0x38/0x80 [37490.545035] fuse [37490.547655] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37490.552783] [37490.552783] irq event stamp: 220436098 [37490.555401] RIP: 0033:0x7f48f0eeb93b [37490.558990] hardirqs last enabled at (220436097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37490.560914] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37490.565956] hardirqs last disabled at (220436098): [] sysvec_apic_timer_interrupt+0xe/0x90 [37490.567447] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37490.571193] softirqs last enabled at (220436096): [] __do_softirq+0x2be/0x371 [37490.574762] ORIG_RAX: 0000000000000012 [37490.585007] softirqs last disabled at (220436091): [] irq_exit_rcu+0xb0/0xd0 [37490.603745] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37490.613556] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37490.618772] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37490.627544] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37490.631373] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37490.639972] Workqueue: events delayed_work [ceph] [37490.647097] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37490.656474] [37490.663597] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37490.671068] RIP: 0010:lock_release+0x162/0x260 [37490.678199] [37490.682889] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [37490.723989] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [37490.729205] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [37490.736330] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [37490.743453] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [37490.750578] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [37490.757709] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [37490.764836] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37490.772914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37490.778651] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37490.785774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37490.792899] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37490.800022] Call Trace: [37490.802468] [37490.804479] ? watchdog_timer_fn+0x1b4/0x220 [37490.808743] ? __hrtimer_run_queues+0x1d3/0x360 [37490.813269] ? hrtimer_interrupt+0x104/0x240 [37490.817538] ? __local_bh_enable+0x50/0x60 [37490.821632] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37490.826855] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37490.831813] [37490.833909] [37490.836007] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37490.841312] ? igrab+0x43/0x50 [37490.844366] ? lock_release+0x162/0x260 [37490.848201] ? lock_release+0xc5/0x260 [37490.851948] _raw_spin_unlock+0x1b/0x40 [37490.855787] igrab+0x43/0x50 [37490.858664] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37490.863815] delayed_work+0x1d8/0x280 [ceph] [37490.868095] ? process_scheduled_works+0x22a/0x510 [37490.872883] process_scheduled_works+0x2a4/0x510 [37490.877500] worker_thread+0x188/0x340 [37490.881251] ? rescuer_thread+0x310/0x310 [37490.885260] kthread+0xf5/0x130 [37490.888398] ? kthread_complete_and_exit+0x20/0x20 [37490.893184] ret_from_fork+0x31/0x40 [37490.896762] ? kthread_complete_and_exit+0x20/0x20 [37490.901547] ret_from_fork_asm+0x11/0x20 [37490.905480] [37518.112496] watchdog: BUG: soft lockup - CPU#0 stuck for 23827s! [dbench:78683] [37518.119800] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37518.124496] watchdog: BUG: soft lockup - CPU#7 stuck for 32824s! [kworker/7:4:78808] [37518.175242] irq event stamp: 65988886 [37518.175244] hardirqs last enabled at (65988885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37518.182973] Modules linked in: [37518.186630] hardirqs last disabled at (65988886): [] sysvec_apic_timer_interrupt+0xe/0x90 [37518.196788] ceph [37518.199837] softirqs last enabled at (65988868): [] __do_softirq+0x2be/0x371 [37518.209561] libceph [37518.211487] softirqs last disabled at (65988863): [] irq_exit_rcu+0xb0/0xd0 [37518.220171] dns_resolver fscache [37518.222355] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37518.230864] netfs veth [37518.234192] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37518.234193] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37518.243136] nft_chain_nat xt_MASQUERADE [37518.245581] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37518.253052] nf_nat nf_conntrack [37518.258529] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37518.262447] nf_defrag_ipv6 nf_defrag_ipv4 [37518.281185] [37518.281186] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37518.284408] nft_compat nf_tables [37518.289626] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37518.293717] nfnetlink bridge [37518.295207] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37518.302330] stp llc [37518.305642] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37518.312766] xfs libcrc32c [37518.315730] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37518.322854] sunrpc [37518.325038] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37518.332179] intel_rapl_msr intel_rapl_common [37518.334891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37518.342018] sb_edac x86_pkg_temp_thermal [37518.344114] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37518.352192] coretemp kvm_intel [37518.356544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37518.362281] kvm irqbypass [37518.366284] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37518.373407] i2c_i801 iTCO_wdt [37518.376547] Call Trace: [37518.376548] [37518.383670] iTCO_vendor_support [37518.386375] ? watchdog_timer_fn+0x1b4/0x220 [37518.393497] i2c_smbus crct10dif_pclmul [37518.396549] ? __hrtimer_run_queues+0x1d3/0x360 [37518.398992] joydev crc32_pclmul [37518.401007] ? hrtimer_interrupt+0x104/0x240 [37518.404226] ipmi_ssif ghash_clmulni_intel [37518.408492] ? __local_bh_enable+0x50/0x60 [37518.412322] mei_me [37518.416848] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37518.420069] mei lpc_ich [37518.424336] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37518.428426] sha512_ssse3 [37518.432516] [37518.434614] acpi_ipmi mfd_core [37518.439830] [37518.439832] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37518.442362] wmi ipmi_si [37518.447324] ? queued_spin_lock_slowpath+0x16/0x2e0 [37518.449936] ipmi_devintf [37518.452036] do_raw_spin_lock+0xb5/0xc0 [37518.455184] ipmi_msghandler [37518.457279] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37518.462581] ioatdma acpi_pad [37518.465113] try_get_cap_refs+0x159/0x880 [ceph] [37518.469984] acpi_power_meter ixgbe [37518.472606] __ceph_get_caps+0x4a1/0x760 [ceph] [37518.476430] igb mdio [37518.479312] ? __file_remove_privs+0x89/0x110 [37518.484093] crc32c_intel nvme [37518.487057] ? wake_bit_function+0x70/0x70 [37518.491668] ptp nvme_core [37518.495153] ? ceph_write_iter+0x260/0x770 [ceph] [37518.499692] i2c_algo_bit pps_core [37518.501965] ceph_write_iter+0x260/0x770 [ceph] [37518.506313] dca fuse [37518.509369] ? lock_is_held_type+0xaa/0x120 [37518.513456] [37518.516179] vfs_write+0x324/0x4f0 [37518.520875] irq event stamp: 220656938 [37518.524293] ksys_pwrite64+0x65/0xa0 [37518.528814] hardirqs last enabled at (220656937): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37518.531084] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37518.535261] hardirqs last disabled at (220656938): [] sysvec_apic_timer_interrupt+0xe/0x90 [37518.536753] do_syscall_64+0x38/0x80 [37518.540150] softirqs last enabled at (220656936): [] __do_softirq+0x2be/0x371 [37518.543896] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37518.547465] softirqs last disabled at (220656927): [] irq_exit_rcu+0xb0/0xd0 [37518.557710] RIP: 0033:0x7f48f0eeb93b [37518.562839] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37518.572651] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37518.576220] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37518.584992] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37518.590036] Workqueue: events delayed_work [ceph] [37518.598632] ORIG_RAX: 0000000000000012 [37518.602205] [37518.602206] RIP: 0010:lock_release+0x32/0x260 [37518.611581] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37518.630320] Code: 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e <89> c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 8b 3d d0 0b 54 01 [37518.637789] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37518.643008] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [37518.647704] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37518.651536] [37518.653026] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37518.657376] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [37518.664501] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37518.683238] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [37518.690369] [37518.695579] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [37518.742016] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [37518.749142] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37518.756265] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37518.764342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37518.770080] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37518.777204] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37518.784329] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37518.791453] Call Trace: [37518.793903] [37518.795918] ? watchdog_timer_fn+0x1b4/0x220 [37518.800190] ? __hrtimer_run_queues+0x1d3/0x360 [37518.804719] ? hrtimer_interrupt+0x104/0x240 [37518.808985] ? __local_bh_enable+0x50/0x60 [37518.813079] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37518.818303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37518.823260] [37518.825357] [37518.827457] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37518.832760] ? igrab+0x43/0x50 [37518.835813] ? igrab+0x43/0x50 [37518.838860] ? lock_release+0x32/0x260 [37518.842605] ? lock_release+0xc5/0x260 [37518.846352] _raw_spin_unlock+0x1b/0x40 [37518.850187] igrab+0x43/0x50 [37518.853067] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37518.858217] delayed_work+0x1d8/0x280 [ceph] [37518.862497] ? process_scheduled_works+0x22a/0x510 [37518.867289] process_scheduled_works+0x2a4/0x510 [37518.871908] worker_thread+0x188/0x340 [37518.875652] ? rescuer_thread+0x310/0x310 [37518.879656] kthread+0xf5/0x130 [37518.882793] ? kthread_complete_and_exit+0x20/0x20 [37518.887579] ret_from_fork+0x31/0x40 [37518.891156] ? kthread_complete_and_exit+0x20/0x20 [37518.895940] ret_from_fork_asm+0x11/0x20 [37518.899864] [37546.112969] watchdog: BUG: soft lockup - CPU#0 stuck for 23853s! [dbench:78683] [37546.120269] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37546.124969] watchdog: BUG: soft lockup - CPU#7 stuck for 32850s! [kworker/7:4:78808] [37546.175722] irq event stamp: 66047178 [37546.183452] Modules linked in: ceph libceph dns_resolver fscache netfs [37546.187107] hardirqs last enabled at (66047177): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37546.193641] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [37546.203801] hardirqs last disabled at (66047178): [] sysvec_apic_timer_interrupt+0xe/0x90 [37546.209884] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [37546.219608] softirqs last enabled at (66047172): [] __do_softirq+0x2be/0x371 [37546.219612] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [37546.235676] softirqs last disabled at (66047167): [] irq_exit_rcu+0xb0/0xd0 [37546.242541] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [37546.251054] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37546.251054] kvm irqbypass [37546.265647] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37546.268365] i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [37546.275841] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37546.275842] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [37546.286967] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37546.286968] ghash_clmulni_intel mei_me mei [37546.311269] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37546.311270] lpc_ich [37546.311271] [37546.315446] sha512_ssse3 acpi_ipmi [37546.320663] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37546.322848] mfd_core wmi [37546.324339] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37546.327841] ipmi_si ipmi_devintf [37546.334964] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37546.337581] ipmi_msghandler ioatdma [37546.344705] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37546.348016] acpi_pad acpi_power_meter [37546.355139] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37546.358712] ixgbe [37546.365836] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37546.369580] igb mdio [37546.376702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37546.378714] crc32c_intel nvme [37546.386791] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37546.389061] ptp nvme_core [37546.394801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37546.397852] i2c_algo_bit pps_core [37546.404975] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37546.407678] dca [37546.414803] Call Trace: [37546.414806] [37546.418202] fuse [37546.425327] ? watchdog_timer_fn+0x1b4/0x220 [37546.427162] [37546.427163] irq event stamp: 220877644 [37546.429606] ? __hrtimer_run_queues+0x1d3/0x360 [37546.431616] hardirqs last enabled at (220877643): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37546.433545] ? hrtimer_interrupt+0x104/0x240 [37546.437804] hardirqs last disabled at (220877644): [] sysvec_apic_timer_interrupt+0xe/0x90 [37546.439295] ? __local_bh_enable+0x50/0x60 [37546.443039] softirqs last enabled at (220877642): [] __do_softirq+0x2be/0x371 [37546.447566] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37546.457808] softirqs last disabled at (220877637): [] irq_exit_rcu+0xb0/0xd0 [37546.462073] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37546.471883] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37546.475974] [37546.484743] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37546.489962] [37546.489964] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37546.498559] Workqueue: events delayed_work [ceph] [37546.503521] ? queued_spin_lock_slowpath+0x16/0x2e0 [37546.512895] [37546.512896] RIP: 0010:lock_acquire+0x4a/0x2b0 [37546.514994] do_raw_spin_lock+0xb5/0xc0 [37546.522463] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [37546.524560] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37546.529864] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [37546.534564] try_get_cap_refs+0x159/0x880 [ceph] [37546.539432] [37546.539433] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [37546.540929] __ceph_get_caps+0x4a1/0x760 [ceph] [37546.545274] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [37546.549110] ? __file_remove_privs+0x89/0x110 [37546.567843] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [37546.572627] ? wake_bit_function+0x70/0x70 [37546.577861] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [37546.582475] ? ceph_write_iter+0x260/0x770 [ceph] [37546.583962] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37546.591087] ceph_write_iter+0x260/0x770 [ceph] [37546.595611] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37546.602739] ? lock_is_held_type+0xaa/0x120 [37546.607086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37546.614213] vfs_write+0x324/0x4f0 [37546.618299] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37546.625430] ksys_pwrite64+0x65/0xa0 [37546.630122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37546.630123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37546.637245] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37546.641769] Call Trace: [37546.641770] [37546.649848] do_syscall_64+0x38/0x80 [37546.654025] ? watchdog_timer_fn+0x1b4/0x220 [37546.659762] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37546.663161] ? __hrtimer_run_queues+0x1d3/0x360 [37546.670284] RIP: 0033:0x7f48f0eeb93b [37546.673858] ? hrtimer_interrupt+0x104/0x240 [37546.680979] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37546.688103] ? __local_bh_enable+0x50/0x60 [37546.693233] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37546.695680] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37546.697687] ORIG_RAX: 0000000000000012 [37546.701259] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37546.705523] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37546.710567] [37546.715091] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37546.718661] [37546.722927] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37546.741665] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37546.745755] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37546.750975] ? lock_acquire+0x4a/0x2b0 [37546.756188] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37546.760022] ? igrab+0x43/0x50 [37546.764982] [37546.772102] ? lock_release+0xc5/0x260 [37546.822823] _raw_spin_lock+0x30/0x40 [37546.826487] ? igrab+0x1d/0x50 [37546.829538] igrab+0x1d/0x50 [37546.832417] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37546.837566] delayed_work+0x1d8/0x280 [ceph] [37546.841848] ? process_scheduled_works+0x22a/0x510 [37546.846653] process_scheduled_works+0x2a4/0x510 [37546.851269] worker_thread+0x188/0x340 [37546.855020] ? rescuer_thread+0x310/0x310 [37546.859031] kthread+0xf5/0x130 [37546.862177] ? kthread_complete_and_exit+0x20/0x20 [37546.866960] ret_from_fork+0x31/0x40 [37546.870542] ? kthread_complete_and_exit+0x20/0x20 [37546.875333] ret_from_fork_asm+0x11/0x20 [37546.879266] [37574.113442] watchdog: BUG: soft lockup - CPU#0 stuck for 23879s! [dbench:78683] [37574.120741] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37574.125442] watchdog: BUG: soft lockup - CPU#7 stuck for 32876s! [kworker/7:4:78808] [37574.176182] irq event stamp: 66105336 [37574.176183] hardirqs last enabled at (66105335): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37574.183913] Modules linked in: [37574.187570] hardirqs last disabled at (66105336): [] sysvec_apic_timer_interrupt+0xe/0x90 [37574.197726] ceph [37574.200778] softirqs last enabled at (66105318): [] __do_softirq+0x2be/0x371 [37574.210501] libceph [37574.212426] softirqs last disabled at (66105313): [] irq_exit_rcu+0xb0/0xd0 [37574.221118] dns_resolver fscache [37574.223303] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37574.231814] netfs veth [37574.235145] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37574.244085] nft_chain_nat xt_MASQUERADE [37574.246529] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37574.254001] nf_nat nf_conntrack [37574.257918] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37574.263396] nf_defrag_ipv6 nf_defrag_ipv4 [37574.266619] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37574.285358] nft_compat nf_tables [37574.289448] [37574.289449] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37574.294666] nfnetlink [37574.297976] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37574.299467] bridge stp [37574.306590] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37574.308949] llc xfs [37574.316071] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37574.316072] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37574.318516] libcrc32c [37574.325642] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37574.327838] sunrpc [37574.334966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37574.342090] intel_rapl_msr intel_rapl_common [37574.344448] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37574.352524] sb_edac x86_pkg_temp_thermal [37574.354621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37574.360359] coretemp kvm_intel [37574.364712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37574.371840] kvm irqbypass [37574.375847] Call Trace: [37574.375848] [37574.382970] i2c_i801 [37574.386119] ? watchdog_timer_fn+0x1b4/0x220 [37574.393243] iTCO_wdt [37574.395947] ? __hrtimer_run_queues+0x1d3/0x360 [37574.398390] iTCO_vendor_support i2c_smbus [37574.400403] ? hrtimer_interrupt+0x104/0x240 [37574.402670] crct10dif_pclmul joydev [37574.406935] ? __local_bh_enable+0x50/0x60 [37574.409205] crc32_pclmul ipmi_ssif [37574.413733] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37574.417837] ghash_clmulni_intel [37574.422104] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37574.425674] mei_me mei [37574.429765] [37574.433248] lpc_ich [37574.438467] [37574.438468] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37574.441689] sha512_ssse3 acpi_ipmi [37574.446652] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37574.449092] mfd_core [37574.451190] do_raw_spin_lock+0xb5/0xc0 [37574.453372] wmi ipmi_si [37574.455472] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37574.460773] ipmi_devintf [37574.464260] try_get_cap_refs+0x159/0x880 [ceph] [37574.469129] ipmi_msghandler [37574.471405] __ceph_get_caps+0x4a1/0x760 [ceph] [37574.475230] ioatdma acpi_pad [37574.477764] ? __file_remove_privs+0x89/0x110 [37574.482545] acpi_power_meter ixgbe [37574.485162] ? wake_bit_function+0x70/0x70 [37574.489774] igb mdio [37574.492653] ? ceph_write_iter+0x260/0x770 [ceph] [37574.497174] crc32c_intel [37574.500143] ceph_write_iter+0x260/0x770 [ceph] [37574.504490] nvme ptp [37574.507978] ? lock_is_held_type+0xaa/0x120 [37574.512064] nvme_core i2c_algo_bit [37574.514337] vfs_write+0x324/0x4f0 [37574.519033] pps_core dca [37574.521655] ksys_pwrite64+0x65/0xa0 [37574.526174] fuse [37574.528443] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37574.532621] irq event stamp: 221098500 [37574.536124] do_syscall_64+0x38/0x80 [37574.539521] hardirqs last enabled at (221098499): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37574.542139] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37574.545709] hardirqs last disabled at (221098500): [] sysvec_apic_timer_interrupt+0xe/0x90 [37574.547634] RIP: 0033:0x7f48f0eeb93b [37574.552763] softirqs last enabled at (221098498): [] __do_softirq+0x2be/0x371 [37574.556508] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37574.560077] softirqs last disabled at (221098493): [] irq_exit_rcu+0xb0/0xd0 [37574.570321] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37574.575367] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37574.585177] ORIG_RAX: 0000000000000012 [37574.588749] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37574.597519] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37574.616257] Workqueue: events delayed_work [ceph] [37574.624853] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37574.630071] [37574.630072] RIP: 0010:_raw_spin_unlock+0xf/0x40 [37574.639448] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37574.643279] Code: 31 c9 ba 01 00 00 00 31 f6 e8 5d 09 5c ff 58 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 <48> 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff [37574.650750] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37574.657874] RSP: 0018:ffffc90002117d90 EFLAGS: 00000202 [37574.662572] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37574.669697] [37574.669698] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [37574.671193] [37574.675712] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34500 [37574.738970] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [37574.746094] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [37574.753219] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37574.760344] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37574.768420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37574.774158] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37574.781282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37574.788407] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37574.795530] Call Trace: [37574.797976] [37574.799986] ? watchdog_timer_fn+0x1b4/0x220 [37574.804250] ? __hrtimer_run_queues+0x1d3/0x360 [37574.808777] ? hrtimer_interrupt+0x104/0x240 [37574.813045] ? __local_bh_enable+0x50/0x60 [37574.817140] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37574.822363] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37574.827321] [37574.829418] [37574.831515] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37574.836824] ? igrab+0x43/0x50 [37574.839897] ? _raw_spin_unlock+0xf/0x40 [37574.843816] igrab+0x43/0x50 [37574.846693] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37574.851849] delayed_work+0x1d8/0x280 [ceph] [37574.856131] ? process_scheduled_works+0x22a/0x510 [37574.860921] process_scheduled_works+0x2a4/0x510 [37574.865536] worker_thread+0x188/0x340 [37574.869285] ? rescuer_thread+0x310/0x310 [37574.873288] kthread+0xf5/0x130 [37574.876427] ? kthread_complete_and_exit+0x20/0x20 [37574.881213] ret_from_fork+0x31/0x40 [37574.884790] ? kthread_complete_and_exit+0x20/0x20 [37574.889575] ret_from_fork_asm+0x11/0x20 [37574.893497] [37601.293253] rcu: INFO: rcu_preempt self-detected stall on CPU [37601.299017] rcu: 0-....: (25084596 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=12561750 [37601.309526] rcu: (t=25664233 jiffies g=1309609 q=702031 ncpus=8) [37601.315614] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37601.324565] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37601.332034] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [37601.337520] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [37601.356259] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37601.361476] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37601.368599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37601.375724] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37601.382848] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37601.389972] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37601.397096] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37601.405175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37601.410912] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37601.418036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37601.425160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37601.432284] Call Trace: [37601.434731] [37601.436740] ? rcu_dump_cpu_stacks+0xd1/0x180 [37601.441099] ? rcu_sched_clock_irq+0x633/0x1390 [37601.445631] ? lock_is_held_type+0xaa/0x120 [37601.449809] ? lock_is_held_type+0xaa/0x120 [37601.453987] ? tick_sched_do_timer+0xa0/0xa0 [37601.458258] ? rcu_read_lock_sched_held+0x56/0x90 [37601.462956] ? tick_sched_do_timer+0xa0/0xa0 [37601.467220] ? update_process_times+0x5f/0x90 [37601.471571] ? tick_sched_handle.isra.16+0x1e/0x50 [37601.476355] ? tick_sched_timer+0x6f/0x80 [37601.480361] ? __hrtimer_run_queues+0x1d3/0x360 [37601.484886] ? hrtimer_interrupt+0x104/0x240 [37601.489148] ? __local_bh_enable+0x50/0x60 [37601.493250] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37601.498472] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37601.503429] [37601.505526] [37601.507625] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37601.512942] ? queued_spin_lock_slowpath+0x2b/0x2e0 [37601.517820] do_raw_spin_lock+0xb5/0xc0 [37601.521659] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37601.526471] try_get_cap_refs+0x159/0x880 [ceph] [37601.531099] __ceph_get_caps+0x4a1/0x760 [ceph] [37601.535641] ? __file_remove_privs+0x89/0x110 [37601.539995] ? wake_bit_function+0x70/0x70 [37601.544088] ? ceph_write_iter+0x260/0x770 [ceph] [37601.548802] ceph_write_iter+0x260/0x770 [ceph] [37601.553338] ? lock_is_held_type+0xaa/0x120 [37601.557523] vfs_write+0x324/0x4f0 [37601.560931] ksys_pwrite64+0x65/0xa0 [37601.564505] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37601.569636] do_syscall_64+0x38/0x80 [37601.573209] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37601.578260] RIP: 0033:0x7f48f0eeb93b [37601.581832] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37601.600566] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [37601.608124] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37601.615248] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37601.622373] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37601.629497] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37601.636621] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37601.643753] [37601.645939] Sending NMI from CPU 0 to CPUs 7: [37601.650291] NMI backtrace for cpu 7 [37601.650293] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37601.650295] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37601.650296] Workqueue: events delayed_work [ceph] [37601.650313] RIP: 0010:__lock_acquire+0x1c6/0x1e20 [37601.650315] Code: 03 c1 e2 02 41 09 f6 83 e1 f8 44 88 75 21 09 c1 09 d1 83 e1 e7 44 09 e9 44 09 c9 88 4d 22 0f b7 94 24 a0 00 00 00 0f b7 45 22 e2 05 83 e0 1f 09 d0 66 89 45 22 8b 84 24 a8 00 00 00 89 45 24 [37601.650317] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [37601.650318] RAX: 0000000000000004 RBX: ffff88810debaa80 RCX: 0000000000000004 [37601.650319] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [37601.650319] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [37601.650320] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [37601.650321] R13: 0000000000000000 R14: 0000000000000006 R15: ffff8881a1d34518 [37601.650321] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37601.650322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37601.650323] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37601.650324] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37601.650325] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37601.650325] Call Trace: [37601.650326] [37601.650327] ? nmi_cpu_backtrace+0xad/0x120 [37601.650331] ? nmi_cpu_backtrace_handler+0x11/0x20 [37601.650332] ? nmi_handle+0xd4/0x260 [37601.650337] ? default_do_nmi+0x49/0xf0 [37601.650338] ? exc_nmi+0x11e/0x150 [37601.650340] ? end_repeat_nmi+0x16/0x67 [37601.650343] ? __lock_acquire+0x1c6/0x1e20 [37601.650345] ? __lock_acquire+0x1c6/0x1e20 [37601.650346] ? __lock_acquire+0x1c6/0x1e20 [37601.650348] [37601.650348] [37601.650350] lock_acquire+0xcb/0x2b0 [37601.650352] ? igrab+0x1d/0x50 [37601.650353] ? igrab+0x43/0x50 [37601.650354] ? lock_release+0xc5/0x260 [37601.650356] _raw_spin_lock+0x30/0x40 [37601.650359] ? igrab+0x1d/0x50 [37601.650360] igrab+0x1d/0x50 [37601.650361] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37601.650376] delayed_work+0x1d8/0x280 [ceph] [37601.650389] ? process_scheduled_works+0x22a/0x510 [37601.650393] process_scheduled_works+0x2a4/0x510 [37601.650396] worker_thread+0x188/0x340 [37601.650399] ? rescuer_thread+0x310/0x310 [37601.650401] kthread+0xf5/0x130 [37601.650404] ? kthread_complete_and_exit+0x20/0x20 [37601.650406] ret_from_fork+0x31/0x40 [37601.650409] ? kthread_complete_and_exit+0x20/0x20 [37601.650411] ret_from_fork_asm+0x11/0x20 [37601.650415] [37626.114293] watchdog: BUG: soft lockup - CPU#0 stuck for 23928s! [dbench:78683] [37626.121593] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37626.126293] watchdog: BUG: soft lockup - CPU#7 stuck for 32925s! [kworker/7:4:78808] [37626.177044] irq event stamp: 66213824 [37626.184774] Modules linked in: ceph libceph dns_resolver [37626.188432] hardirqs last enabled at (66213823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37626.193736] fscache netfs veth [37626.203891] hardirqs last disabled at (66213824): [] sysvec_apic_timer_interrupt+0xe/0x90 [37626.207031] nft_chain_nat xt_MASQUERADE [37626.216754] softirqs last enabled at (66213798): [] __do_softirq+0x2be/0x371 [37626.216755] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [37626.229374] softirqs last disabled at (66213793): [] irq_exit_rcu+0xb0/0xd0 [37626.237016] nfnetlink bridge stp [37626.245529] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37626.248838] llc xfs [37626.257783] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37626.259969] libcrc32c sunrpc intel_rapl_msr [37626.267437] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37626.267439] intel_rapl_common sb_edac x86_pkg_temp_thermal [37626.277178] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37626.277179] coretemp kvm_intel kvm [37626.301482] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37626.304969] irqbypass i2c_i801 [37626.310182] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37626.310183] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37626.313320] iTCO_wdt [37626.320445] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37626.327569] iTCO_vendor_support i2c_smbus [37626.329840] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37626.336969] crct10dif_pclmul [37626.341063] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37626.348186] joydev crc32_pclmul [37626.351152] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37626.358275] ipmi_ssif ghash_clmulni_intel [37626.361500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37626.369575] mei_me mei [37626.373667] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37626.379422] lpc_ich sha512_ssse3 [37626.381866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37626.388990] acpi_ipmi mfd_core [37626.392300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37626.399426] wmi ipmi_si [37626.402563] Call Trace: [37626.402565] [37626.409692] ipmi_devintf [37626.412217] ? watchdog_timer_fn+0x1b4/0x220 [37626.414663] ipmi_msghandler ioatdma [37626.416690] ? __hrtimer_run_queues+0x1d3/0x360 [37626.419307] acpi_pad acpi_power_meter [37626.423574] ? hrtimer_interrupt+0x104/0x240 [37626.427141] ixgbe [37626.431667] ? __local_bh_enable+0x50/0x60 [37626.435426] igb mdio [37626.439695] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37626.441703] crc32c_intel [37626.445793] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37626.448064] nvme [37626.453280] [37626.455899] ptp nvme_core [37626.460855] [37626.460856] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37626.462779] i2c_algo_bit pps_core [37626.464881] ? queued_spin_lock_slowpath+0x16/0x2e0 [37626.467581] dca fuse [37626.469693] do_raw_spin_lock+0xb5/0xc0 [37626.474991] [37626.474992] irq event stamp: 221511862 [37626.478390] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37626.483260] hardirqs last enabled at (221511861): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37626.485533] try_get_cap_refs+0x159/0x880 [ceph] [37626.489361] hardirqs last disabled at (221511862): [] sysvec_apic_timer_interrupt+0xe/0x90 [37626.490858] __ceph_get_caps+0x4a1/0x760 [ceph] [37626.494596] softirqs last enabled at (221511860): [] __do_softirq+0x2be/0x371 [37626.499384] ? __file_remove_privs+0x89/0x110 [37626.509625] softirqs last disabled at (221511855): [] irq_exit_rcu+0xb0/0xd0 [37626.514235] ? wake_bit_function+0x70/0x70 [37626.524047] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37626.528572] ? ceph_write_iter+0x260/0x770 [ceph] [37626.537340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37626.541693] ceph_write_iter+0x260/0x770 [ceph] [37626.550288] Workqueue: events delayed_work [ceph] [37626.554383] ? lock_is_held_type+0xaa/0x120 [37626.563757] [37626.568457] vfs_write+0x324/0x4f0 [37626.575924] RIP: 0010:lock_release+0x3c/0x260 [37626.580455] ksys_pwrite64+0x65/0xa0 [37626.585146] Code: 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 <0f> 82 90 01 00 00 8b 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b [37626.589323] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37626.590814] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [37626.594215] do_syscall_64+0x38/0x80 [37626.598562] [37626.598562] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [37626.602135] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37626.620870] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [37626.620872] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [37626.626002] RIP: 0033:0x7f48f0eeb93b [37626.631219] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [37626.634790] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37626.636280] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37626.643404] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37626.648448] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37626.655574] ORIG_RAX: 0000000000000012 [37626.662698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37626.666268] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37626.673392] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37626.692130] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37626.699255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37626.704472] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37626.712548] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37626.716380] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37626.722118] Call Trace: [37626.722119] [37626.729240] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37626.736366] ? watchdog_timer_fn+0x1b4/0x220 [37626.743495] [37626.750615] ? __hrtimer_run_queues+0x1d3/0x360 [37626.794540] ? hrtimer_interrupt+0x104/0x240 [37626.798810] ? __local_bh_enable+0x50/0x60 [37626.802905] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37626.808127] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37626.813086] [37626.815183] [37626.817279] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37626.822584] ? igrab+0x43/0x50 [37626.825637] ? igrab+0x43/0x50 [37626.828699] ? lock_release+0x3c/0x260 [37626.832448] ? lock_release+0xc5/0x260 [37626.836192] _raw_spin_unlock+0x1b/0x40 [37626.840020] igrab+0x43/0x50 [37626.842899] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37626.848049] delayed_work+0x1d8/0x280 [ceph] [37626.852330] ? process_scheduled_works+0x22a/0x510 [37626.857120] process_scheduled_works+0x2a4/0x510 [37626.861735] worker_thread+0x188/0x340 [37626.865486] ? rescuer_thread+0x310/0x310 [37626.869488] kthread+0xf5/0x130 [37626.872627] ? kthread_complete_and_exit+0x20/0x20 [37626.877411] ret_from_fork+0x31/0x40 [37626.880997] ? kthread_complete_and_exit+0x20/0x20 [37626.885784] ret_from_fork_asm+0x11/0x20 [37626.889714] [37654.114741] watchdog: BUG: soft lockup - CPU#0 stuck for 23954s! [dbench:78683] [37654.122048] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37654.126740] watchdog: BUG: soft lockup - CPU#7 stuck for 32951s! [kworker/7:4:78808] [37654.177498] irq event stamp: 66271964 [37654.185228] Modules linked in: ceph libceph dns_resolver fscache netfs [37654.188884] hardirqs last enabled at (66271963): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37654.195415] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [37654.205577] hardirqs last disabled at (66271964): [] sysvec_apic_timer_interrupt+0xe/0x90 [37654.211661] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [37654.221384] softirqs last enabled at (66271938): [] __do_softirq+0x2be/0x371 [37654.228161] bridge stp llc xfs [37654.236847] softirqs last disabled at (66271933): [] irq_exit_rcu+0xb0/0xd0 [37654.236849] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37654.248497] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37654.256833] coretemp kvm_intel [37654.265777] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37654.265778] kvm [37654.268914] irqbypass i2c_i801 [37654.276384] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37654.278221] iTCO_wdt iTCO_vendor_support [37654.281360] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37654.286836] i2c_smbus crct10dif_pclmul [37654.290840] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37654.309579] joydev [37654.313425] [37654.318644] crc32_pclmul [37654.320740] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37654.322232] ipmi_ssif ghash_clmulni_intel [37654.324850] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37654.331973] mei_me mei [37654.336065] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37654.343187] lpc_ich sha512_ssse3 [37654.345631] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37654.352757] acpi_ipmi mfd_core [37654.356068] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37654.363190] wmi [37654.366329] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37654.373452] ipmi_si [37654.375290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37654.383368] ipmi_devintf [37654.385552] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37654.391290] ipmi_msghandler [37654.393906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37654.401030] ioatdma acpi_pad [37654.403908] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37654.411031] acpi_power_meter [37654.413997] Call Trace: [37654.413999] [37654.421135] ixgbe [37654.424105] ? watchdog_timer_fn+0x1b4/0x220 [37654.426547] igb mdio [37654.428559] ? __hrtimer_run_queues+0x1d3/0x360 [37654.430567] crc32c_intel nvme [37654.434835] ? hrtimer_interrupt+0x104/0x240 [37654.437102] ptp nvme_core [37654.441625] ? __local_bh_enable+0x50/0x60 [37654.444676] i2c_algo_bit pps_core [37654.448944] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37654.451644] dca fuse [37654.455736] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37654.459136] [37654.459137] irq event stamp: 221732958 [37654.464350] [37654.466621] hardirqs last enabled at (221732957): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37654.471577] [37654.473069] hardirqs last disabled at (221732958): [] sysvec_apic_timer_interrupt+0xe/0x90 [37654.476813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37654.478911] softirqs last enabled at (221732956): [] __do_softirq+0x2be/0x371 [37654.489159] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37654.491252] softirqs last disabled at (221732951): [] irq_exit_rcu+0xb0/0xd0 [37654.501067] do_raw_spin_lock+0xb5/0xc0 [37654.506367] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37654.515141] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37654.520010] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37654.528608] try_get_cap_refs+0x159/0x880 [ceph] [37654.532437] Workqueue: events delayed_work [ceph] [37654.541822] __ceph_get_caps+0x4a1/0x760 [ceph] [37654.546598] [37654.554074] ? __file_remove_privs+0x89/0x110 [37654.558681] RIP: 0010:lock_acquire+0xf0/0x2b0 [37654.563379] ? wake_bit_function+0x70/0x70 [37654.567902] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37654.569394] ? ceph_write_iter+0x260/0x770 [ceph] [37654.573743] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37654.578093] ceph_write_iter+0x260/0x770 [ceph] [37654.582186] [37654.582186] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37654.600926] ? lock_is_held_type+0xaa/0x120 [37654.605620] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37654.610839] vfs_write+0x324/0x4f0 [37654.615360] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37654.616856] ksys_pwrite64+0x65/0xa0 [37654.623977] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37654.628154] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37654.635278] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37654.638677] do_syscall_64+0x38/0x80 [37654.645800] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37654.649371] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37654.656495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37654.661626] RIP: 0033:0x7f48f0eeb93b [37654.668749] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37654.672321] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37654.680398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37654.685457] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37654.691197] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37654.694766] ORIG_RAX: 0000000000000012 [37654.701890] Call Trace: [37654.701892] [37654.720629] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37654.727755] ? watchdog_timer_fn+0x1b4/0x220 [37654.732970] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37654.740096] ? __hrtimer_run_queues+0x1d3/0x360 [37654.743925] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37654.746372] ? hrtimer_interrupt+0x104/0x240 [37654.748379] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37654.755503] ? __local_bh_enable+0x50/0x60 [37654.759767] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37654.766896] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37654.771421] [37654.778540] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37654.813503] [37654.815599] [37654.817697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37654.823005] ? lock_acquire+0xf0/0x2b0 [37654.826755] ? igrab+0x43/0x50 [37654.829805] ? lock_release+0xc5/0x260 [37654.833550] _raw_spin_lock+0x30/0x40 [37654.837208] ? igrab+0x1d/0x50 [37654.840258] igrab+0x1d/0x50 [37654.843146] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37654.848294] delayed_work+0x1d8/0x280 [ceph] [37654.852575] ? process_scheduled_works+0x22a/0x510 [37654.857366] process_scheduled_works+0x2a4/0x510 [37654.861982] worker_thread+0x188/0x340 [37654.865730] ? rescuer_thread+0x310/0x310 [37654.869733] kthread+0xf5/0x130 [37654.872872] ? kthread_complete_and_exit+0x20/0x20 [37654.877655] ret_from_fork+0x31/0x40 [37654.881234] ? kthread_complete_and_exit+0x20/0x20 [37654.886018] ret_from_fork_asm+0x11/0x20 [37654.889942] [37682.115175] watchdog: BUG: soft lockup - CPU#0 stuck for 23980s! [dbench:78683] [37682.122472] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37682.127175] watchdog: BUG: soft lockup - CPU#7 stuck for 32977s! [kworker/7:4:78808] [37682.177913] irq event stamp: 66330072 [37682.185645] Modules linked in: ceph libceph dns_resolver [37682.189303] hardirqs last enabled at (66330071): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37682.194615] fscache netfs veth nft_chain_nat [37682.204772] hardirqs last disabled at (66330072): [] sysvec_apic_timer_interrupt+0xe/0x90 [37682.209124] xt_MASQUERADE nf_nat nf_conntrack [37682.218859] softirqs last enabled at (66330004): [] __do_softirq+0x2be/0x371 [37682.218860] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [37682.231987] softirqs last disabled at (66329999): [] irq_exit_rcu+0xb0/0xd0 [37682.231989] bridge stp llc xfs [37682.247276] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37682.250412] libcrc32c sunrpc [37682.259357] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37682.262319] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37682.269792] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37682.276654] coretemp kvm_intel [37682.282133] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37682.282133] kvm irqbypass [37682.285269] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37682.304008] i2c_i801 [37682.306711] [37682.306712] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37682.311928] iTCO_wdt [37682.314200] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37682.315691] iTCO_vendor_support i2c_smbus [37682.322814] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37682.325085] crct10dif_pclmul [37682.332210] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37682.336299] joydev [37682.343423] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37682.346390] crc32_pclmul ipmi_ssif [37682.353512] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37682.355609] ghash_clmulni_intel mei_me [37682.362734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37682.366217] mei lpc_ich [37682.374295] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37682.378126] sha512_ssse3 acpi_ipmi [37682.383865] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37682.386394] mfd_core wmi [37682.393520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37682.397003] ipmi_si ipmi_devintf [37682.404126] Call Trace: [37682.404128] [37682.406744] ipmi_msghandler ioatdma [37682.413869] ? watchdog_timer_fn+0x1b4/0x220 [37682.417178] acpi_pad acpi_power_meter [37682.419623] ? __hrtimer_run_queues+0x1d3/0x360 [37682.421635] ixgbe igb [37682.425209] ? hrtimer_interrupt+0x104/0x240 [37682.429468] mdio [37682.433212] ? __local_bh_enable+0x50/0x60 [37682.437737] crc32c_intel [37682.440096] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37682.444357] nvme ptp [37682.446283] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37682.450373] nvme_core [37682.452991] [37682.458207] i2c_algo_bit [37682.460479] [37682.460481] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37682.465435] pps_core dca [37682.467798] ? queued_spin_lock_slowpath+0x18/0x2e0 [37682.469890] fuse [37682.472509] do_raw_spin_lock+0xb5/0xc0 [37682.474606] irq event stamp: 221953970 [37682.479910] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37682.482527] hardirqs last enabled at (221953969): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37682.487400] try_get_cap_refs+0x159/0x880 [ceph] [37682.489323] hardirqs last disabled at (221953970): [] sysvec_apic_timer_interrupt+0xe/0x90 [37682.493156] __ceph_get_caps+0x4a1/0x760 [ceph] [37682.496896] softirqs last enabled at (221953968): [] __do_softirq+0x2be/0x371 [37682.501685] ? __file_remove_privs+0x89/0x110 [37682.511924] softirqs last disabled at (221953963): [] irq_exit_rcu+0xb0/0xd0 [37682.516537] ? wake_bit_function+0x70/0x70 [37682.526346] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37682.530873] ? ceph_write_iter+0x260/0x770 [ceph] [37682.539640] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37682.543992] ceph_write_iter+0x260/0x770 [ceph] [37682.552589] Workqueue: events delayed_work [ceph] [37682.556683] ? lock_is_held_type+0xaa/0x120 [37682.566057] [37682.566058] RIP: 0010:lock_acquire+0xf0/0x2b0 [37682.570757] vfs_write+0x324/0x4f0 [37682.578226] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37682.582755] ksys_pwrite64+0x65/0xa0 [37682.587447] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37682.591625] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37682.593116] [37682.597469] do_syscall_64+0x38/0x80 [37682.600864] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37682.619602] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37682.623171] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37682.628389] RIP: 0033:0x7f48f0eeb93b [37682.633520] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37682.635010] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37682.638581] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37682.645705] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37682.650751] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37682.657874] ORIG_RAX: 0000000000000012 [37682.661444] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37682.668575] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37682.687315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37682.694438] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37682.699658] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37682.706781] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37682.710611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37682.718689] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37682.725813] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37682.731550] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37682.738693] Call Trace: [37682.738695] [37682.745823] [37682.752942] ? watchdog_timer_fn+0x1b4/0x220 [37682.792342] ? __hrtimer_run_queues+0x1d3/0x360 [37682.796880] ? hrtimer_interrupt+0x104/0x240 [37682.801145] ? __local_bh_enable+0x50/0x60 [37682.805238] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37682.810454] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37682.815411] [37682.817510] [37682.819606] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37682.824914] ? lock_acquire+0xf0/0x2b0 [37682.828664] ? igrab+0x43/0x50 [37682.831715] ? lock_release+0xc5/0x260 [37682.835469] _raw_spin_lock+0x30/0x40 [37682.839134] ? igrab+0x1d/0x50 [37682.842183] igrab+0x1d/0x50 [37682.845063] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37682.850212] delayed_work+0x1d8/0x280 [ceph] [37682.854492] ? process_scheduled_works+0x22a/0x510 [37682.859283] process_scheduled_works+0x2a4/0x510 [37682.863898] worker_thread+0x188/0x340 [37682.867647] ? rescuer_thread+0x310/0x310 [37682.871649] kthread+0xf5/0x130 [37682.874789] ? kthread_complete_and_exit+0x20/0x20 [37682.879580] ret_from_fork+0x31/0x40 [37682.883153] ? kthread_complete_and_exit+0x20/0x20 [37682.887935] ret_from_fork_asm+0x11/0x20 [37682.891876] [37710.115611] watchdog: BUG: soft lockup - CPU#0 stuck for 24006s! [dbench:78683] [37710.122918] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37710.127611] watchdog: BUG: soft lockup - CPU#7 stuck for 33003s! [kworker/7:4:78808] [37710.178359] irq event stamp: 66388240 [37710.178360] hardirqs last enabled at (66388239): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37710.186088] Modules linked in: [37710.189746] hardirqs last disabled at (66388240): [] sysvec_apic_timer_interrupt+0xe/0x90 [37710.199903] ceph [37710.202954] softirqs last enabled at (66388214): [] __do_softirq+0x2be/0x371 [37710.212679] libceph [37710.214602] softirqs last disabled at (66388209): [] irq_exit_rcu+0xb0/0xd0 [37710.223294] dns_resolver fscache [37710.225480] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37710.234006] netfs veth [37710.237319] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37710.246261] nft_chain_nat [37710.248724] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37710.256194] xt_MASQUERADE nf_nat [37710.258899] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37710.264377] nf_conntrack nf_defrag_ipv6 [37710.267687] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37710.286424] nf_defrag_ipv4 nft_compat [37710.290342] [37710.290343] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37710.295561] nf_tables nfnetlink [37710.299302] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37710.300793] bridge stp [37710.307919] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37710.311143] llc [37710.318266] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37710.320727] xfs libcrc32c [37710.327853] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37710.329688] sunrpc intel_rapl_msr [37710.336815] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37710.339517] intel_rapl_common sb_edac [37710.346641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37710.350040] x86_pkg_temp_thermal coretemp [37710.358117] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37710.361861] kvm_intel kvm [37710.367599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37710.371690] irqbypass i2c_i801 [37710.378813] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37710.381516] iTCO_wdt iTCO_vendor_support [37710.388642] Call Trace: [37710.388644] [37710.391778] i2c_smbus crct10dif_pclmul [37710.398904] ? watchdog_timer_fn+0x1b4/0x220 [37710.402907] joydev crc32_pclmul [37710.405351] ? __hrtimer_run_queues+0x1d3/0x360 [37710.407363] ipmi_ssif ghash_clmulni_intel [37710.411194] ? hrtimer_interrupt+0x104/0x240 [37710.415455] mei_me mei [37710.418680] ? __local_bh_enable+0x50/0x60 [37710.423205] lpc_ich sha512_ssse3 [37710.427314] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37710.431576] acpi_ipmi mfd_core [37710.434020] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37710.438112] wmi ipmi_si [37710.441421] [37710.446638] ipmi_devintf [37710.449778] [37710.449779] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37710.454735] ipmi_msghandler ioatdma [37710.457268] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37710.459361] acpi_pad acpi_power_meter [37710.461984] do_raw_spin_lock+0xb5/0xc0 [37710.464076] ixgbe igb [37710.469382] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37710.472951] mdio [37710.477825] try_get_cap_refs+0x159/0x880 [ceph] [37710.481568] crc32c_intel nvme [37710.485401] __ceph_get_caps+0x4a1/0x760 [ceph] [37710.487755] ptp nvme_core [37710.492544] ? __file_remove_privs+0x89/0x110 [37710.494462] i2c_algo_bit pps_core [37710.499074] ? wake_bit_function+0x70/0x70 [37710.502125] dca fuse [37710.506651] ? ceph_write_iter+0x260/0x770 [ceph] [37710.509352] [37710.509353] irq event stamp: 222174846 [37710.513703] ceph_write_iter+0x260/0x770 [ceph] [37710.517100] hardirqs last enabled at (222174845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37710.521196] ? lock_is_held_type+0xaa/0x120 [37710.523461] hardirqs last disabled at (222174846): [] sysvec_apic_timer_interrupt+0xe/0x90 [37710.528162] vfs_write+0x324/0x4f0 [37710.529650] softirqs last enabled at (222174844): [] __do_softirq+0x2be/0x371 [37710.533399] ksys_pwrite64+0x65/0xa0 [37710.537919] softirqs last disabled at (222174839): [] irq_exit_rcu+0xb0/0xd0 [37710.548162] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37710.552341] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37710.562153] do_syscall_64+0x38/0x80 [37710.565548] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37710.574319] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37710.577891] Workqueue: events delayed_work [ceph] [37710.586488] RIP: 0033:0x7f48f0eeb93b [37710.591618] [37710.591618] RIP: 0010:lock_acquire+0xf0/0x2b0 [37710.601013] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37710.604583] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37710.612056] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37710.617099] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37710.621796] ORIG_RAX: 0000000000000012 [37710.625367] [37710.625368] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37710.626858] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37710.631207] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37710.649946] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37710.668683] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37710.673901] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37710.679117] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37710.682949] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37710.684441] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37710.691565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37710.698689] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37710.705817] [37710.712936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37710.771681] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37710.778805] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37710.785929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37710.793054] Call Trace: [37710.795497] [37710.797508] ? watchdog_timer_fn+0x1b4/0x220 [37710.801773] ? __hrtimer_run_queues+0x1d3/0x360 [37710.806310] ? hrtimer_interrupt+0x104/0x240 [37710.810579] ? __local_bh_enable+0x50/0x60 [37710.814671] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37710.819895] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37710.824852] [37710.826949] [37710.829047] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37710.834355] ? lock_acquire+0xf0/0x2b0 [37710.838105] ? igrab+0x43/0x50 [37710.841155] ? lock_release+0xc5/0x260 [37710.844899] _raw_spin_lock+0x30/0x40 [37710.848555] ? igrab+0x1d/0x50 [37710.851607] igrab+0x1d/0x50 [37710.854484] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37710.859634] delayed_work+0x1d8/0x280 [ceph] [37710.863915] ? process_scheduled_works+0x22a/0x510 [37710.868706] process_scheduled_works+0x2a4/0x510 [37710.873321] worker_thread+0x188/0x340 [37710.877069] ? rescuer_thread+0x310/0x310 [37710.881074] kthread+0xf5/0x130 [37710.884212] ? kthread_complete_and_exit+0x20/0x20 [37710.889016] ret_from_fork+0x31/0x40 [37710.892592] ? kthread_complete_and_exit+0x20/0x20 [37710.897375] ret_from_fork_asm+0x11/0x20 [37710.901302] [37738.116056] watchdog: BUG: soft lockup - CPU#0 stuck for 24032s! [dbench:78683] [37738.123361] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37738.128056] watchdog: BUG: soft lockup - CPU#7 stuck for 33029s! [kworker/7:4:78808] [37738.178811] irq event stamp: 66446466 [37738.186541] Modules linked in: ceph libceph dns_resolver fscache netfs [37738.190198] hardirqs last enabled at (66446465): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37738.196716] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [37738.206892] hardirqs last disabled at (66446466): [] sysvec_apic_timer_interrupt+0xe/0x90 [37738.214285] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [37738.224007] softirqs last enabled at (66446398): [] __do_softirq+0x2be/0x371 [37738.224012] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37738.239469] softirqs last disabled at (66446393): [] irq_exit_rcu+0xb0/0xd0 [37738.248153] coretemp kvm_intel kvm irqbypass i2c_i801 [37738.256667] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37738.256668] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [37738.270741] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37738.277084] joydev crc32_pclmul ipmi_ssif [37738.284555] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37738.284557] ghash_clmulni_intel mei_me mei lpc_ich [37738.294124] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37738.294125] sha512_ssse3 acpi_ipmi mfd_core [37738.317731] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37738.317733] wmi ipmi_si [37738.322012] [37738.327231] ipmi_devintf [37738.329760] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37738.331251] ipmi_msghandler ioatdma [37738.333869] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37738.340994] acpi_pad [37738.344563] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37738.351689] acpi_power_meter ixgbe [37738.353959] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37738.361082] igb mdio [37738.364567] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37738.371690] crc32c_intel nvme [37738.373963] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37738.381086] ptp nvme_core [37738.384135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37738.392215] i2c_algo_bit pps_core [37738.394919] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37738.400655] dca fuse [37738.404053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37738.411176] [37738.411177] irq event stamp: 222395588 [37738.413455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37738.420580] hardirqs last enabled at (222395587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37738.422072] Call Trace: [37738.422075] [37738.425816] hardirqs last disabled at (222395588): [] sysvec_apic_timer_interrupt+0xe/0x90 [37738.432942] ? watchdog_timer_fn+0x1b4/0x220 [37738.443183] softirqs last enabled at (222395586): [] __do_softirq+0x2be/0x371 [37738.445629] ? __hrtimer_run_queues+0x1d3/0x360 [37738.447637] softirqs last disabled at (222395581): [] irq_exit_rcu+0xb0/0xd0 [37738.457457] ? hrtimer_interrupt+0x104/0x240 [37738.461713] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37738.470502] ? __local_bh_enable+0x50/0x60 [37738.475026] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37738.483627] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37738.487887] Workqueue: events delayed_work [ceph] [37738.497265] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37738.501355] [37738.501356] RIP: 0010:lock_acquire+0xf0/0x2b0 [37738.508826] [37738.514045] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37738.518740] [37738.518742] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37738.523698] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37738.525194] ? queued_spin_lock_slowpath+0x18/0x2e0 [37738.529540] [37738.529541] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37738.531639] do_raw_spin_lock+0xb5/0xc0 [37738.550375] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37738.552474] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37738.557777] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37738.562997] try_get_cap_refs+0x159/0x880 [ceph] [37738.567863] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37738.569361] __ceph_get_caps+0x4a1/0x760 [ceph] [37738.576479] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37738.580314] ? __file_remove_privs+0x89/0x110 [37738.587433] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37738.592235] ? wake_bit_function+0x70/0x70 [37738.599360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37738.603974] ? ceph_write_iter+0x260/0x770 [ceph] [37738.611093] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37738.615618] ceph_write_iter+0x260/0x770 [ceph] [37738.622744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37738.622761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37738.627115] ? lock_is_held_type+0xaa/0x120 [37738.635189] Call Trace: [37738.635191] [37738.639281] vfs_write+0x324/0x4f0 [37738.645018] ? watchdog_timer_fn+0x1b4/0x220 [37738.649719] ksys_pwrite64+0x65/0xa0 [37738.656841] ? __hrtimer_run_queues+0x1d3/0x360 [37738.661363] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37738.668491] ? hrtimer_interrupt+0x104/0x240 [37738.675613] do_syscall_64+0x38/0x80 [37738.679787] ? __local_bh_enable+0x50/0x60 [37738.682233] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37738.684246] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37738.687641] RIP: 0033:0x7f48f0eeb93b [37738.691905] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37738.695476] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37738.700000] [37738.705129] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37738.709395] [37738.712964] ORIG_RAX: 0000000000000012 [37738.717056] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37738.722099] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37738.727320] ? lock_acquire+0xf0/0x2b0 [37738.730887] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37738.735848] ? igrab+0x43/0x50 [37738.754583] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37738.756681] ? lock_release+0xc5/0x260 [37738.761896] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37738.763997] _raw_spin_lock+0x30/0x40 [37738.767826] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37738.773131] ? igrab+0x1d/0x50 [37738.780258] [37738.783998] igrab+0x1d/0x50 [37738.831071] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37738.836219] delayed_work+0x1d8/0x280 [ceph] [37738.840501] ? process_scheduled_works+0x22a/0x510 [37738.845291] process_scheduled_works+0x2a4/0x510 [37738.849905] worker_thread+0x188/0x340 [37738.853656] ? rescuer_thread+0x310/0x310 [37738.857666] kthread+0xf5/0x130 [37738.860805] ? kthread_complete_and_exit+0x20/0x20 [37738.865589] ret_from_fork+0x31/0x40 [37738.869168] ? kthread_complete_and_exit+0x20/0x20 [37738.873951] ret_from_fork_asm+0x11/0x20 [37738.877875] [37766.116501] watchdog: BUG: soft lockup - CPU#0 stuck for 24058s! [dbench:78683] [37766.123805] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37766.128501] watchdog: BUG: soft lockup - CPU#7 stuck for 33055s! [kworker/7:4:78808] [37766.179247] irq event stamp: 66504558 [37766.179248] hardirqs last enabled at (66504557): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37766.186978] Modules linked in: [37766.190634] hardirqs last disabled at (66504558): [] sysvec_apic_timer_interrupt+0xe/0x90 [37766.200791] ceph [37766.203843] softirqs last enabled at (66504500): [] __do_softirq+0x2be/0x371 [37766.213568] libceph dns_resolver [37766.215491] softirqs last disabled at (66504495): [] irq_exit_rcu+0xb0/0xd0 [37766.224176] fscache netfs [37766.227485] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37766.235997] veth nft_chain_nat [37766.238701] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37766.247646] xt_MASQUERADE nf_nat [37766.250782] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37766.258254] nf_conntrack [37766.261563] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37766.267042] nf_defrag_ipv6 nf_defrag_ipv4 [37766.269659] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37766.288397] nft_compat nf_tables [37766.292488] [37766.292488] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37766.297705] nfnetlink bridge [37766.301014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37766.302506] stp llc [37766.309631] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37766.312594] xfs libcrc32c [37766.319719] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37766.321901] sunrpc intel_rapl_msr [37766.329026] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37766.331730] intel_rapl_common sb_edac [37766.338856] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37766.342251] x86_pkg_temp_thermal coretemp [37766.349375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37766.353121] kvm_intel kvm [37766.361198] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37766.365288] irqbypass i2c_i801 [37766.371027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37766.373730] iTCO_wdt iTCO_vendor_support [37766.380853] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37766.383992] i2c_smbus crct10dif_pclmul [37766.391115] Call Trace: [37766.391116] [37766.395120] joydev [37766.402245] ? watchdog_timer_fn+0x1b4/0x220 [37766.406074] crc32_pclmul [37766.408519] ? __hrtimer_run_queues+0x1d3/0x360 [37766.410529] ipmi_ssif ghash_clmulni_intel [37766.412630] ? hrtimer_interrupt+0x104/0x240 [37766.416900] mei_me [37766.419517] ? __local_bh_enable+0x50/0x60 [37766.424040] mei lpc_ich [37766.428133] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37766.432396] sha512_ssse3 [37766.434492] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37766.438583] acpi_ipmi mfd_core [37766.441115] [37766.446332] wmi ipmi_si [37766.448950] [37766.448951] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37766.453906] ipmi_devintf [37766.457049] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37766.459141] ipmi_msghandler [37766.461674] do_raw_spin_lock+0xb5/0xc0 [37766.463770] ioatdma acpi_pad [37766.469075] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37766.471691] acpi_power_meter [37766.476563] try_get_cap_refs+0x159/0x880 [ceph] [37766.479438] ixgbe igb [37766.483274] __ceph_get_caps+0x4a1/0x760 [ceph] [37766.486235] mdio crc32c_intel [37766.491022] ? __file_remove_privs+0x89/0x110 [37766.493981] nvme ptp [37766.498594] ? wake_bit_function+0x70/0x70 [37766.500949] nvme_core i2c_algo_bit [37766.505476] ? ceph_write_iter+0x260/0x770 [ceph] [37766.508525] pps_core [37766.512875] ceph_write_iter+0x260/0x770 [ceph] [37766.515180] dca fuse [37766.519276] ? lock_is_held_type+0xaa/0x120 [37766.522756] [37766.522757] irq event stamp: 222616604 [37766.527456] vfs_write+0x324/0x4f0 [37766.529723] hardirqs last enabled at (222616603): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37766.534253] ksys_pwrite64+0x65/0xa0 [37766.536520] hardirqs last disabled at (222616604): [] sysvec_apic_timer_interrupt+0xe/0x90 [37766.540697] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37766.542187] softirqs last enabled at (222616602): [] __do_softirq+0x2be/0x371 [37766.545933] do_syscall_64+0x38/0x80 [37766.549328] softirqs last disabled at (222616597): [] irq_exit_rcu+0xb0/0xd0 [37766.559575] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37766.563143] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37766.572955] RIP: 0033:0x7f48f0eeb93b [37766.578084] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37766.586857] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37766.590427] Workqueue: events delayed_work [ceph] [37766.599025] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37766.604069] [37766.604069] RIP: 0010:lock_acquire+0xf0/0x2b0 [37766.613446] ORIG_RAX: 0000000000000012 [37766.617016] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37766.624487] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37766.643226] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37766.647923] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37766.653139] [37766.653140] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37766.654630] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37766.658980] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37766.662813] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37766.681550] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37766.688674] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37766.693905] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37766.701039] [37766.702524] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37766.761710] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37766.769785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37766.775524] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37766.782649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37766.789772] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37766.796906] Call Trace: [37766.799350] [37766.801362] ? watchdog_timer_fn+0x1b4/0x220 [37766.805627] ? __hrtimer_run_queues+0x1d3/0x360 [37766.810161] ? hrtimer_interrupt+0x104/0x240 [37766.814438] ? __local_bh_enable+0x50/0x60 [37766.818531] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37766.823756] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37766.828713] [37766.830809] [37766.832908] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37766.838215] ? lock_acquire+0xf0/0x2b0 [37766.841967] ? igrab+0x43/0x50 [37766.845016] ? lock_release+0xc5/0x260 [37766.848761] _raw_spin_lock+0x30/0x40 [37766.852416] ? igrab+0x1d/0x50 [37766.855469] igrab+0x1d/0x50 [37766.858345] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37766.863495] delayed_work+0x1d8/0x280 [ceph] [37766.867777] ? process_scheduled_works+0x22a/0x510 [37766.872566] process_scheduled_works+0x2a4/0x510 [37766.877191] worker_thread+0x188/0x340 [37766.880939] ? rescuer_thread+0x310/0x310 [37766.884944] kthread+0xf5/0x130 [37766.888081] ? kthread_complete_and_exit+0x20/0x20 [37766.892866] ret_from_fork+0x31/0x40 [37766.896436] ? kthread_complete_and_exit+0x20/0x20 [37766.901222] ret_from_fork_asm+0x11/0x20 [37766.905142] [37794.116941] watchdog: BUG: soft lockup - CPU#0 stuck for 24084s! [dbench:78683] [37794.124241] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37794.128941] watchdog: BUG: soft lockup - CPU#7 stuck for 33081s! [kworker/7:4:78808] [37794.179690] irq event stamp: 66562812 [37794.187421] Modules linked in: ceph libceph dns_resolver fscache netfs [37794.191079] hardirqs last enabled at (66562811): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37794.197615] veth nft_chain_nat xt_MASQUERADE nf_nat [37794.207771] hardirqs last disabled at (66562812): [] sysvec_apic_timer_interrupt+0xe/0x90 [37794.212729] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [37794.222452] softirqs last enabled at (66562754): [] __do_softirq+0x2be/0x371 [37794.230357] bridge stp llc xfs libcrc32c [37794.239041] softirqs last disabled at (66562749): [] irq_exit_rcu+0xb0/0xd0 [37794.239044] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37794.251558] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37794.251559] coretemp [37794.251560] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37794.259027] kvm_intel [37794.267971] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37794.270241] kvm irqbypass [37794.277714] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37794.280069] i2c_i801 iTCO_wdt [37794.285547] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37794.288252] iTCO_vendor_support [37794.306989] [37794.310039] i2c_smbus crct10dif_pclmul [37794.315257] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37794.318480] joydev crc32_pclmul [37794.319973] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37794.323804] ipmi_ssif ghash_clmulni_intel [37794.330926] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37794.334150] mei_me [37794.341275] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37794.345365] mei lpc_ich [37794.352490] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37794.354586] sha512_ssse3 acpi_ipmi [37794.361711] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37794.364242] mfd_core wmi [37794.371367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37794.374850] ipmi_si ipmi_devintf [37794.382927] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37794.385546] ipmi_msghandler [37794.391281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37794.394593] ioatdma acpi_pad [37794.401716] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37794.404595] acpi_power_meter ixgbe [37794.411736] Call Trace: [37794.411739] [37794.414701] igb mdio [37794.421826] ? watchdog_timer_fn+0x1b4/0x220 [37794.425309] crc32c_intel nvme [37794.427753] ? __hrtimer_run_queues+0x1d3/0x360 [37794.429763] ptp nvme_core [37794.432037] ? hrtimer_interrupt+0x104/0x240 [37794.436298] i2c_algo_bit pps_core [37794.439348] ? __local_bh_enable+0x50/0x60 [37794.443873] dca [37794.446578] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37794.450841] fuse [37794.454238] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37794.458339] [37794.458339] irq event stamp: 222837374 [37794.460175] [37794.465393] hardirqs last enabled at (222837373): [] asm_common_interrupt+0x26/0x40 [37794.467317] [37794.472292] hardirqs last disabled at (222837374): [] sysvec_apic_timer_interrupt+0xe/0x90 [37794.473783] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37794.477526] softirqs last enabled at (222837372): [] __do_softirq+0x2be/0x371 [37794.479630] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37794.488914] softirqs last disabled at (222837367): [] irq_exit_rcu+0xb0/0xd0 [37794.491013] do_raw_spin_lock+0xb5/0xc0 [37794.500824] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37794.506129] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37794.514898] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37794.519772] try_get_cap_refs+0x159/0x880 [ceph] [37794.528366] Workqueue: events delayed_work [ceph] [37794.532202] __ceph_get_caps+0x4a1/0x760 [ceph] [37794.541574] [37794.541575] RIP: 0010:lock_acquire+0xf0/0x2b0 [37794.546363] ? __file_remove_privs+0x89/0x110 [37794.553828] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37794.558440] ? wake_bit_function+0x70/0x70 [37794.563137] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37794.567663] ? ceph_write_iter+0x260/0x770 [ceph] [37794.569151] [37794.569152] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37794.573504] ceph_write_iter+0x260/0x770 [ceph] [37794.577852] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37794.596595] ? lock_is_held_type+0xaa/0x120 [37794.600683] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37794.605901] vfs_write+0x324/0x4f0 [37794.610596] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37794.612091] ksys_pwrite64+0x65/0xa0 [37794.619210] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37794.623737] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37794.630859] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37794.635039] do_syscall_64+0x38/0x80 [37794.642161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37794.645559] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37794.652682] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37794.656254] RIP: 0033:0x7f48f0eeb93b [37794.663377] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37794.668509] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37794.676586] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37794.680155] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37794.685893] Call Trace: [37794.685895] [37794.690939] ORIG_RAX: 0000000000000012 [37794.698063] ? watchdog_timer_fn+0x1b4/0x220 [37794.701639] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37794.708768] ? __hrtimer_run_queues+0x1d3/0x360 [37794.727503] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37794.734631] ? hrtimer_interrupt+0x104/0x240 [37794.739845] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37794.742289] ? __local_bh_enable+0x50/0x60 [37794.744301] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37794.748134] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37794.752393] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37794.759518] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37794.764047] [37794.771167] [37794.815351] [37794.817449] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37794.822756] ? lock_acquire+0xf0/0x2b0 [37794.826506] ? igrab+0x43/0x50 [37794.829555] ? lock_release+0xc5/0x260 [37794.833303] _raw_spin_lock+0x30/0x40 [37794.836967] ? igrab+0x1d/0x50 [37794.840016] igrab+0x1d/0x50 [37794.842896] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37794.848046] delayed_work+0x1d8/0x280 [ceph] [37794.852344] ? process_scheduled_works+0x22a/0x510 [37794.857133] process_scheduled_works+0x2a4/0x510 [37794.861749] worker_thread+0x188/0x340 [37794.865498] ? rescuer_thread+0x310/0x310 [37794.869501] kthread+0xf5/0x130 [37794.872638] ? kthread_complete_and_exit+0x20/0x20 [37794.877424] ret_from_fork+0x31/0x40 [37794.881002] ? kthread_complete_and_exit+0x20/0x20 [37794.885786] ret_from_fork_asm+0x11/0x20 [37794.889720] [37796.657348] rcu: INFO: rcu_preempt self-detected stall on CPU [37796.663092] rcu: 7-....: (25108014 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/25106657 fqs=12657572 [37796.673770] rcu: (t=25859593 jiffies g=1309609 q=705823 ncpus=8) [37796.679854] Sending NMI from CPU 7 to CPUs 0: [37796.684206] NMI backtrace for cpu 0 [37796.684208] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37796.684210] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37796.684210] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37796.684214] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37796.684215] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37796.684217] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37796.684217] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37796.684218] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37796.684219] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37796.684220] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37796.684221] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37796.684222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37796.684223] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37796.684224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37796.684224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37796.684225] Call Trace: [37796.684226] [37796.684227] ? nmi_cpu_backtrace+0xad/0x120 [37796.684231] ? nmi_cpu_backtrace_handler+0x11/0x20 [37796.684232] ? nmi_handle+0xd4/0x260 [37796.684237] ? default_do_nmi+0x49/0xf0 [37796.684238] ? exc_nmi+0x11e/0x150 [37796.684240] ? end_repeat_nmi+0x16/0x67 [37796.684244] ? queued_spin_lock_slowpath+0x18/0x2e0 [37796.684247] ? queued_spin_lock_slowpath+0x18/0x2e0 [37796.684249] ? queued_spin_lock_slowpath+0x18/0x2e0 [37796.684251] [37796.684252] [37796.684253] do_raw_spin_lock+0xb5/0xc0 [37796.684255] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37796.684270] try_get_cap_refs+0x159/0x880 [ceph] [37796.684287] __ceph_get_caps+0x4a1/0x760 [ceph] [37796.684304] ? __file_remove_privs+0x89/0x110 [37796.684306] ? wake_bit_function+0x70/0x70 [37796.684309] ? ceph_write_iter+0x260/0x770 [ceph] [37796.684321] ceph_write_iter+0x260/0x770 [ceph] [37796.684335] ? lock_is_held_type+0xaa/0x120 [37796.684338] vfs_write+0x324/0x4f0 [37796.684342] ksys_pwrite64+0x65/0xa0 [37796.684344] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37796.684348] do_syscall_64+0x38/0x80 [37796.684351] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37796.684353] RIP: 0033:0x7f48f0eeb93b [37796.684355] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37796.684356] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [37796.684357] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37796.684358] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37796.684359] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37796.684359] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37796.684360] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37796.684363] [37796.685205] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37796.988758] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37796.996228] Workqueue: events delayed_work [ceph] [37797.000951] RIP: 0010:lock_release+0x2b/0x260 [37797.005301] Code: 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 <65> 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 [37797.024039] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [37797.029258] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [37797.036389] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [37797.043515] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [37797.050637] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [37797.057780] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37797.064902] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37797.072981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37797.078718] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37797.085843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37797.092968] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37797.100090] Call Trace: [37797.102535] [37797.104547] ? rcu_dump_cpu_stacks+0xd1/0x180 [37797.108906] ? rcu_sched_clock_irq+0x633/0x1390 [37797.113432] ? lock_release+0xc5/0x260 [37797.117181] ? lock_is_held_type+0xaa/0x120 [37797.121380] ? lock_is_held_type+0xaa/0x120 [37797.125565] ? tick_sched_do_timer+0xa0/0xa0 [37797.129836] ? rcu_read_lock_sched_held+0x56/0x90 [37797.134544] ? tick_sched_do_timer+0xa0/0xa0 [37797.138814] ? update_process_times+0x5f/0x90 [37797.143166] ? tick_sched_handle.isra.16+0x1e/0x50 [37797.147951] ? tick_sched_timer+0x6f/0x80 [37797.151965] ? __hrtimer_run_queues+0x1d3/0x360 [37797.156502] ? hrtimer_interrupt+0x104/0x240 [37797.160768] ? __local_bh_enable+0x50/0x60 [37797.164873] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37797.170094] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37797.175051] [37797.177147] [37797.179246] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37797.184560] ? igrab+0x43/0x50 [37797.187622] ? igrab+0x43/0x50 [37797.190678] ? lock_release+0x2b/0x260 [37797.194423] ? lock_release+0xc5/0x260 [37797.198180] _raw_spin_unlock+0x1b/0x40 [37797.202015] igrab+0x43/0x50 [37797.204892] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37797.210044] delayed_work+0x1d8/0x280 [ceph] [37797.214332] ? process_scheduled_works+0x22a/0x510 [37797.219121] process_scheduled_works+0x2a4/0x510 [37797.223741] worker_thread+0x188/0x340 [37797.227486] ? rescuer_thread+0x310/0x310 [37797.231497] kthread+0xf5/0x130 [37797.234635] ? kthread_complete_and_exit+0x20/0x20 [37797.239422] ret_from_fork+0x31/0x40 [37797.242999] ? kthread_complete_and_exit+0x20/0x20 [37797.247784] ret_from_fork_asm+0x11/0x20 [37797.251721] [37822.117381] watchdog: BUG: soft lockup - CPU#0 stuck for 24110s! [dbench:78683] [37822.124686] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37822.129381] watchdog: BUG: soft lockup - CPU#7 stuck for 33107s! [kworker/7:4:78808] [37822.180126] irq event stamp: 66621016 [37822.187856] Modules linked in: ceph [37822.191514] hardirqs last enabled at (66621015): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37822.191516] libceph dns_resolver fscache netfs [37822.205156] hardirqs last disabled at (66621016): [] sysvec_apic_timer_interrupt+0xe/0x90 [37822.209681] veth nft_chain_nat xt_MASQUERADE nf_nat [37822.219404] softirqs last enabled at (66620966): [] __do_softirq+0x2be/0x371 [37822.224361] nf_conntrack nf_defrag_ipv6 [37822.233062] softirqs last disabled at (66620961): [] irq_exit_rcu+0xb0/0xd0 [37822.233063] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [37822.245491] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37822.250969] bridge stp [37822.259912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37822.262358] llc xfs libcrc32c [37822.269827] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37822.269828] sunrpc [37822.272878] intel_rapl_msr intel_rapl_common [37822.278356] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37822.280453] sb_edac x86_pkg_temp_thermal [37822.284805] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37822.303542] coretemp [37822.307545] [37822.307546] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37822.312777] kvm_intel [37822.315055] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37822.316551] kvm irqbypass [37822.323674] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37822.326033] i2c_i801 iTCO_wdt [37822.333173] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37822.335876] iTCO_vendor_support [37822.343003] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37822.346056] i2c_smbus crct10dif_pclmul [37822.353177] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37822.356401] joydev crc32_pclmul [37822.363525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37822.367356] ipmi_ssif ghash_clmulni_intel [37822.375433] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37822.378658] mei_me mei [37822.384394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37822.388484] lpc_ich sha512_ssse3 [37822.395609] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37822.398066] acpi_ipmi mfd_core [37822.405194] Call Trace: [37822.405195] [37822.408505] wmi ipmi_si [37822.415631] ? watchdog_timer_fn+0x1b4/0x220 [37822.418777] ipmi_devintf [37822.421221] ? __hrtimer_run_queues+0x1d3/0x360 [37822.423231] ipmi_msghandler [37822.425781] ? hrtimer_interrupt+0x104/0x240 [37822.430042] ioatdma acpi_pad [37822.432678] ? __local_bh_enable+0x50/0x60 [37822.437200] acpi_power_meter ixgbe [37822.440081] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37822.444342] igb [37822.447307] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37822.451398] mdio [37822.454881] [37822.460099] crc32c_intel [37822.461937] [37822.461939] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37822.466893] nvme ptp [37822.468823] ? queued_spin_lock_slowpath+0x16/0x2e0 [37822.470915] nvme_core i2c_algo_bit [37822.473535] do_raw_spin_lock+0xb5/0xc0 [37822.475629] pps_core dca [37822.480936] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37822.483204] fuse [37822.488078] try_get_cap_refs+0x159/0x880 [ceph] [37822.491559] irq event stamp: 223053472 [37822.495395] __ceph_get_caps+0x4a1/0x760 [ceph] [37822.498007] hardirqs last enabled at (223053471): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37822.502796] ? __file_remove_privs+0x89/0x110 [37822.504717] hardirqs last disabled at (223053472): [] sysvec_apic_timer_interrupt+0xe/0x90 [37822.509326] ? wake_bit_function+0x70/0x70 [37822.513070] softirqs last enabled at (223053470): [] __do_softirq+0x2be/0x371 [37822.517596] ? ceph_write_iter+0x260/0x770 [ceph] [37822.527840] softirqs last disabled at (223053465): [] irq_exit_rcu+0xb0/0xd0 [37822.532190] ceph_write_iter+0x260/0x770 [ceph] [37822.542000] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37822.546095] ? lock_is_held_type+0xaa/0x120 [37822.554863] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37822.559563] vfs_write+0x324/0x4f0 [37822.568158] Workqueue: events delayed_work [ceph] [37822.572686] ksys_pwrite64+0x65/0xa0 [37822.582061] [37822.582062] RIP: 0010:rcu_is_watching+0x0/0x40 [37822.586236] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37822.593706] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [37822.597106] do_syscall_64+0x38/0x80 [37822.601801] RSP: 0018:ffffc90002117d10 EFLAGS: 00000247 [37822.605372] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37822.606862] [37822.611300] RIP: 0033:0x7f48f0eeb93b [37822.616431] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [37822.635169] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37822.638739] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [37822.643958] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37822.649000] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [37822.650491] ORIG_RAX: 0000000000000012 [37822.654079] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [37822.661204] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37822.679942] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37822.687066] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37822.692283] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37822.699408] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37822.703238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37822.710363] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37822.717488] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37822.724610] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37822.731735] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37822.739818] [37822.746936] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37822.790478] Call Trace: [37822.792924] [37822.794934] ? watchdog_timer_fn+0x1b4/0x220 [37822.799199] ? __hrtimer_run_queues+0x1d3/0x360 [37822.803725] ? hrtimer_interrupt+0x104/0x240 [37822.807985] ? __local_bh_enable+0x50/0x60 [37822.812079] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37822.817304] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37822.822260] [37822.824357] [37822.826455] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37822.831781] ? rcutorture_get_gp_data+0x30/0x30 [37822.836311] lock_acquire+0x184/0x2b0 [37822.839967] ? igrab+0x43/0x50 [37822.843017] ? lock_release+0xc5/0x260 [37822.846763] _raw_spin_lock+0x30/0x40 [37822.850436] ? igrab+0x1d/0x50 [37822.853488] igrab+0x1d/0x50 [37822.856366] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37822.861516] delayed_work+0x1d8/0x280 [ceph] [37822.865796] ? process_scheduled_works+0x22a/0x510 [37822.870588] process_scheduled_works+0x2a4/0x510 [37822.875202] worker_thread+0x188/0x340 [37822.878951] ? rescuer_thread+0x310/0x310 [37822.882953] kthread+0xf5/0x130 [37822.886092] ? kthread_complete_and_exit+0x20/0x20 [37822.890877] ret_from_fork+0x31/0x40 [37822.894446] ? kthread_complete_and_exit+0x20/0x20 [37822.899230] ret_from_fork_asm+0x11/0x20 [37822.903154] [37850.117832] watchdog: BUG: soft lockup - CPU#0 stuck for 24136s! [dbench:78683] [37850.125139] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37850.129831] watchdog: BUG: soft lockup - CPU#7 stuck for 33133s! [kworker/7:4:78808] [37850.180587] irq event stamp: 66679220 [37850.180588] hardirqs last enabled at (66679219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37850.188318] Modules linked in: [37850.191976] hardirqs last disabled at (66679220): [] sysvec_apic_timer_interrupt+0xe/0x90 [37850.202134] ceph [37850.205183] softirqs last enabled at (66679174): [] __do_softirq+0x2be/0x371 [37850.214909] libceph [37850.216833] softirqs last disabled at (66679169): [] irq_exit_rcu+0xb0/0xd0 [37850.225517] dns_resolver [37850.227703] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37850.236226] fscache [37850.238846] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37850.247791] netfs [37850.249974] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37850.257445] veth [37850.259456] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37850.264934] nft_chain_nat [37850.266857] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37850.285595] xt_MASQUERADE nf_nat [37850.288299] [37850.293517] nf_conntrack nf_defrag_ipv6 [37850.296826] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37850.298319] nf_defrag_ipv4 nft_compat [37850.302234] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37850.309359] nf_tables nfnetlink [37850.313105] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37850.320227] bridge [37850.323452] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37850.330576] stp llc [37850.332673] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37850.339797] xfs libcrc32c [37850.341981] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37850.349105] sunrpc [37850.351810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37850.359888] intel_rapl_msr [37850.361985] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37850.367721] intel_rapl_common sb_edac [37850.370513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37850.377637] x86_pkg_temp_thermal [37850.381380] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37850.388507] coretemp kvm_intel [37850.391816] Call Trace: [37850.391818] [37850.398939] kvm [37850.402079] ? watchdog_timer_fn+0x1b4/0x220 [37850.404535] irqbypass i2c_i801 [37850.406550] ? __hrtimer_run_queues+0x1d3/0x360 [37850.408387] iTCO_wdt iTCO_vendor_support [37850.412655] ? hrtimer_interrupt+0x104/0x240 [37850.415789] i2c_smbus crct10dif_pclmul [37850.420313] ? __local_bh_enable+0x50/0x60 [37850.424315] joydev crc32_pclmul [37850.428583] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37850.432410] ipmi_ssif ghash_clmulni_intel [37850.436511] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37850.439734] mei_me mei [37850.444953] [37850.449042] lpc_ich sha512_ssse3 [37850.454001] [37850.456444] acpi_ipmi mfd_core [37850.458543] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37850.461852] wmi ipmi_si [37850.463954] ? queued_spin_lock_slowpath+0x18/0x2e0 [37850.467086] ipmi_devintf ipmi_msghandler [37850.472392] do_raw_spin_lock+0xb5/0xc0 [37850.474921] ioatdma acpi_pad [37850.479794] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37850.483795] acpi_power_meter ixgbe [37850.487631] try_get_cap_refs+0x159/0x880 [ceph] [37850.490592] igb mdio [37850.495380] __ceph_get_caps+0x4a1/0x760 [ceph] [37850.498860] crc32c_intel [37850.503475] ? __file_remove_privs+0x89/0x110 [37850.505740] nvme ptp [37850.510265] ? wake_bit_function+0x70/0x70 [37850.512883] nvme_core [37850.517237] ? ceph_write_iter+0x260/0x770 [ceph] [37850.519503] i2c_algo_bit pps_core [37850.523596] ceph_write_iter+0x260/0x770 [ceph] [37850.525951] dca fuse [37850.530655] ? lock_is_held_type+0xaa/0x120 [37850.534046] [37850.534047] irq event stamp: 223274324 [37850.538573] vfs_write+0x324/0x4f0 [37850.540842] hardirqs last enabled at (223274323): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37850.545024] ksys_pwrite64+0x65/0xa0 [37850.546510] hardirqs last disabled at (223274324): [] sysvec_apic_timer_interrupt+0xe/0x90 [37850.550253] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37850.553651] softirqs last enabled at (223274322): [] __do_softirq+0x2be/0x371 [37850.563896] do_syscall_64+0x38/0x80 [37850.567467] softirqs last disabled at (223274317): [] irq_exit_rcu+0xb0/0xd0 [37850.577280] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37850.582407] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37850.591179] RIP: 0033:0x7f48f0eeb93b [37850.594750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37850.603348] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37850.608391] Workqueue: events delayed_work [ceph] [37850.617768] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37850.621339] [37850.621340] RIP: 0010:lock_acquire+0xf0/0x2b0 [37850.628809] ORIG_RAX: 0000000000000012 [37850.647548] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37850.652245] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37850.657463] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37850.658953] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37850.663303] [37850.663304] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37850.667135] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37850.685872] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37850.692997] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37850.698231] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37850.705356] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37850.706847] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37850.713975] [37850.721095] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37850.766041] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37850.774119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37850.779856] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37850.786979] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37850.794104] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37850.801236] Call Trace: [37850.803682] [37850.805693] ? watchdog_timer_fn+0x1b4/0x220 [37850.809957] ? __hrtimer_run_queues+0x1d3/0x360 [37850.814482] ? hrtimer_interrupt+0x104/0x240 [37850.818744] ? __local_bh_enable+0x50/0x60 [37850.822838] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37850.828062] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37850.833018] [37850.835114] [37850.837213] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37850.842538] ? lock_acquire+0xf0/0x2b0 [37850.846288] ? igrab+0x43/0x50 [37850.849340] ? lock_release+0xc5/0x260 [37850.853083] _raw_spin_lock+0x30/0x40 [37850.856741] ? igrab+0x1d/0x50 [37850.859790] igrab+0x1d/0x50 [37850.862670] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37850.867819] delayed_work+0x1d8/0x280 [ceph] [37850.872101] ? process_scheduled_works+0x22a/0x510 [37850.876890] process_scheduled_works+0x2a4/0x510 [37850.881520] worker_thread+0x188/0x340 [37850.885264] ? rescuer_thread+0x310/0x310 [37850.889274] kthread+0xf5/0x130 [37850.892414] ? kthread_complete_and_exit+0x20/0x20 [37850.897207] ret_from_fork+0x31/0x40 [37850.900803] ? kthread_complete_and_exit+0x20/0x20 [37850.905587] ret_from_fork_asm+0x11/0x20 [37850.909519] [37878.118292] watchdog: BUG: soft lockup - CPU#0 stuck for 24162s! [dbench:78683] [37878.125592] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37878.130292] watchdog: BUG: soft lockup - CPU#7 stuck for 33159s! [kworker/7:4:78808] [37878.181031] irq event stamp: 66737428 [37878.181032] hardirqs last enabled at (66737427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37878.188762] Modules linked in: [37878.192421] hardirqs last disabled at (66737428): [] sysvec_apic_timer_interrupt+0xe/0x90 [37878.202578] ceph [37878.205629] softirqs last enabled at (66737380): [] __do_softirq+0x2be/0x371 [37878.215353] libceph dns_resolver [37878.217276] softirqs last disabled at (66737375): [] irq_exit_rcu+0xb0/0xd0 [37878.225969] fscache [37878.229280] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37878.237790] netfs veth [37878.239990] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37878.248937] nft_chain_nat xt_MASQUERADE [37878.251382] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37878.258852] nf_nat [37878.262769] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37878.268247] nf_conntrack nf_defrag_ipv6 [37878.270343] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37878.289080] nf_defrag_ipv4 nft_compat [37878.292997] [37878.292998] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37878.298215] nf_tables nfnetlink [37878.301970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37878.303459] bridge stp [37878.310583] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37878.313808] llc xfs [37878.320932] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37878.323376] libcrc32c sunrpc [37878.330500] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37878.332689] intel_rapl_msr intel_rapl_common [37878.339816] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37878.342781] sb_edac x86_pkg_temp_thermal [37878.349906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37878.354256] coretemp kvm_intel [37878.362334] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37878.366338] kvm irqbypass [37878.372074] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37878.375212] i2c_i801 iTCO_wdt [37878.382337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37878.385039] iTCO_vendor_support i2c_smbus [37878.392164] Call Trace: [37878.392165] [37878.395215] crct10dif_pclmul [37878.402341] ? watchdog_timer_fn+0x1b4/0x220 [37878.406430] joydev [37878.408874] ? __hrtimer_run_queues+0x1d3/0x360 [37878.410885] crc32_pclmul ipmi_ssif [37878.413852] ? hrtimer_interrupt+0x104/0x240 [37878.418113] ghash_clmulni_intel mei_me [37878.420210] ? __local_bh_enable+0x50/0x60 [37878.424733] mei lpc_ich [37878.428221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37878.432482] sha512_ssse3 acpi_ipmi [37878.436313] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37878.440403] mfd_core wmi [37878.442934] [37878.448153] ipmi_si ipmi_devintf [37878.451637] [37878.456593] ipmi_msghandler ioatdma [37878.459211] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37878.461307] acpi_pad acpi_power_meter [37878.464624] ? queued_spin_lock_slowpath+0x18/0x2e0 [37878.466716] ixgbe [37878.470289] do_raw_spin_lock+0xb5/0xc0 [37878.475591] igb mdio [37878.479336] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37878.484207] crc32c_intel nvme [37878.486218] try_get_cap_refs+0x159/0x880 [ceph] [37878.490047] ptp nvme_core [37878.492324] __ceph_get_caps+0x4a1/0x760 [ceph] [37878.497103] i2c_algo_bit [37878.500157] ? __file_remove_privs+0x89/0x110 [37878.504763] pps_core dca [37878.507469] ? wake_bit_function+0x70/0x70 [37878.511992] fuse [37878.514611] ? ceph_write_iter+0x260/0x770 [ceph] [37878.518973] irq event stamp: 223494986 [37878.521586] ceph_write_iter+0x260/0x770 [ceph] [37878.525692] hardirqs last enabled at (223494985): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37878.527622] ? lock_is_held_type+0xaa/0x120 [37878.532315] hardirqs last disabled at (223494986): [] sysvec_apic_timer_interrupt+0xe/0x90 [37878.536062] vfs_write+0x324/0x4f0 [37878.540584] softirqs last enabled at (223494984): [] __do_softirq+0x2be/0x371 [37878.550833] ksys_pwrite64+0x65/0xa0 [37878.555006] softirqs last disabled at (223494979): [] irq_exit_rcu+0xb0/0xd0 [37878.564818] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37878.568214] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37878.576985] do_syscall_64+0x38/0x80 [37878.580554] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37878.589152] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37878.594283] Workqueue: events delayed_work [ceph] [37878.603662] RIP: 0033:0x7f48f0eeb93b [37878.607232] [37878.607233] RIP: 0010:lock_release+0x14e/0x260 [37878.614702] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37878.619746] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [37878.624443] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37878.628015] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [37878.629506] ORIG_RAX: 0000000000000012 [37878.633942] [37878.633943] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [37878.652696] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37878.671436] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [37878.676654] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37878.681870] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [37878.685702] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37878.687191] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [37878.694317] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37878.701441] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [37878.708564] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37878.715696] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37878.722826] [37878.729945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37878.774440] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37878.781564] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37878.788697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37878.795821] Call Trace: [37878.798267] [37878.800278] ? watchdog_timer_fn+0x1b4/0x220 [37878.804542] ? __hrtimer_run_queues+0x1d3/0x360 [37878.809068] ? hrtimer_interrupt+0x104/0x240 [37878.813338] ? __local_bh_enable+0x50/0x60 [37878.817431] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37878.822654] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37878.827613] [37878.829708] [37878.831808] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37878.837111] ? igrab+0x43/0x50 [37878.840164] ? lock_release+0x14e/0x260 [37878.843994] ? lock_release+0xc5/0x260 [37878.847738] _raw_spin_unlock+0x1b/0x40 [37878.851568] igrab+0x43/0x50 [37878.854446] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37878.859595] delayed_work+0x1d8/0x280 [ceph] [37878.863875] ? process_scheduled_works+0x22a/0x510 [37878.868658] process_scheduled_works+0x2a4/0x510 [37878.873272] worker_thread+0x188/0x340 [37878.877024] ? rescuer_thread+0x310/0x310 [37878.881025] kthread+0xf5/0x130 [37878.884163] ? kthread_complete_and_exit+0x20/0x20 [37878.888948] ret_from_fork+0x31/0x40 [37878.892518] ? kthread_complete_and_exit+0x20/0x20 [37878.897303] ret_from_fork_asm+0x11/0x20 [37878.901226] [37906.118755] watchdog: BUG: soft lockup - CPU#0 stuck for 24188s! [dbench:78683] [37906.126061] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37906.130755] watchdog: BUG: soft lockup - CPU#7 stuck for 33185s! [kworker/7:4:78808] [37906.181512] irq event stamp: 66795598 [37906.189241] Modules linked in: ceph libceph dns_resolver fscache netfs [37906.192899] hardirqs last enabled at (66795597): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37906.192905] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [37906.209582] hardirqs last disabled at (66795598): [] sysvec_apic_timer_interrupt+0xe/0x90 [37906.216965] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [37906.226692] softirqs last enabled at (66795550): [] __do_softirq+0x2be/0x371 [37906.233147] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [37906.241831] softirqs last disabled at (66795545): [] irq_exit_rcu+0xb0/0xd0 [37906.241835] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [37906.256861] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37906.263723] i2c_i801 iTCO_wdt iTCO_vendor_support [37906.272668] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37906.272670] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37906.277452] i2c_smbus crct10dif_pclmul [37906.284922] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37906.290401] joydev crc32_pclmul [37906.294232] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37906.312970] ipmi_ssif ghash_clmulni_intel [37906.316192] [37906.321410] mei_me mei [37906.325518] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37906.327009] lpc_ich sha512_ssse3 [37906.329453] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37906.336579] acpi_ipmi [37906.339888] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37906.347013] mfd_core [37906.349368] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37906.356494] wmi ipmi_si [37906.358765] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37906.365890] ipmi_devintf [37906.368434] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37906.375560] ipmi_msghandler [37906.378178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37906.386255] ioatdma acpi_pad [37906.389151] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37906.394887] acpi_power_meter [37906.397852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37906.404975] ixgbe [37906.407939] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37906.415064] igb mdio [37906.417074] Call Trace: [37906.424200] crc32c_intel [37906.426471] [37906.428913] nvme ptp [37906.431533] ? watchdog_timer_fn+0x1b4/0x220 [37906.433542] nvme_core i2c_algo_bit [37906.435814] ? __hrtimer_run_queues+0x1d3/0x360 [37906.440078] pps_core [37906.443565] ? hrtimer_interrupt+0x104/0x240 [37906.448084] dca fuse [37906.450355] ? __local_bh_enable+0x50/0x60 [37906.454619] [37906.456893] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37906.460982] irq event stamp: 223715876 [37906.462473] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37906.467689] hardirqs last enabled at (223715875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37906.471442] [37906.476398] hardirqs last disabled at (223715876): [] sysvec_apic_timer_interrupt+0xe/0x90 [37906.486644] [37906.488740] softirqs last enabled at (223715874): [] __do_softirq+0x2be/0x371 [37906.498552] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37906.500650] softirqs last disabled at (223715869): [] irq_exit_rcu+0xb0/0xd0 [37906.509439] ? queued_spin_lock_slowpath+0x18/0x2e0 [37906.514740] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37906.523342] do_raw_spin_lock+0xb5/0xc0 [37906.528208] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37906.537590] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37906.541434] Workqueue: events delayed_work [ceph] [37906.548910] try_get_cap_refs+0x159/0x880 [ceph] [37906.553691] [37906.553692] RIP: 0010:do_raw_spin_lock+0x5c/0xc0 [37906.558394] __ceph_get_caps+0x4a1/0x760 [ceph] [37906.562997] Code: 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 89 43 10 5b e8 2e 22 45 00 85 c0 74 b1 48 c7 c6 29 e2 12 82 48 89 df e8 ab [37906.564492] ? __file_remove_privs+0x89/0x110 [37906.569100] RSP: 0018:ffffc90002117d98 EFLAGS: 00000246 [37906.573625] ? wake_bit_function+0x70/0x70 [37906.592361] [37906.592361] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [37906.596715] ? ceph_write_iter+0x260/0x770 [ceph] [37906.601930] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [37906.606019] ceph_write_iter+0x260/0x770 [ceph] [37906.607512] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [37906.614640] ? lock_is_held_type+0xaa/0x120 [37906.619333] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [37906.626459] vfs_write+0x324/0x4f0 [37906.630980] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37906.630981] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37906.638110] ksys_pwrite64+0x65/0xa0 [37906.642282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37906.649405] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37906.652803] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37906.659928] do_syscall_64+0x38/0x80 [37906.668005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37906.671578] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37906.677314] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37906.682450] RIP: 0033:0x7f48f0eeb93b [37906.689578] Call Trace: [37906.689579] [37906.693160] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37906.700283] ? watchdog_timer_fn+0x1b4/0x220 [37906.705326] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37906.712450] ? __hrtimer_run_queues+0x1d3/0x360 [37906.716019] ORIG_RAX: 0000000000000012 [37906.718469] ? hrtimer_interrupt+0x104/0x240 [37906.720475] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37906.739212] ? __local_bh_enable+0x50/0x60 [37906.743476] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37906.748697] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37906.753217] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37906.757048] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37906.761311] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37906.768438] [37906.772528] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37906.779650] [37906.784875] [37906.791995] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37906.822887] ? do_raw_spin_lock+0x5c/0xc0 [37906.826896] igrab+0x1d/0x50 [37906.829775] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37906.834924] delayed_work+0x1d8/0x280 [ceph] [37906.839203] ? process_scheduled_works+0x22a/0x510 [37906.843994] process_scheduled_works+0x2a4/0x510 [37906.848612] worker_thread+0x188/0x340 [37906.852360] ? rescuer_thread+0x310/0x310 [37906.856372] kthread+0xf5/0x130 [37906.859517] ? kthread_complete_and_exit+0x20/0x20 [37906.864302] ret_from_fork+0x31/0x40 [37906.867881] ? kthread_complete_and_exit+0x20/0x20 [37906.872666] ret_from_fork_asm+0x11/0x20 [37906.876596] [37934.119225] watchdog: BUG: soft lockup - CPU#0 stuck for 24214s! [dbench:78683] [37934.126523] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37934.131225] watchdog: BUG: soft lockup - CPU#7 stuck for 33211s! [kworker/7:4:78808] [37934.181964] irq event stamp: 66853668 [37934.189695] Modules linked in: ceph libceph [37934.193352] hardirqs last enabled at (66853667): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37934.197529] dns_resolver fscache netfs [37934.207687] hardirqs last disabled at (66853668): [] sysvec_apic_timer_interrupt+0xe/0x90 [37934.211516] veth nft_chain_nat [37934.221242] softirqs last enabled at (66853620): [] __do_softirq+0x2be/0x371 [37934.224378] xt_MASQUERADE nf_nat nf_conntrack [37934.233062] softirqs last disabled at (66853615): [] irq_exit_rcu+0xb0/0xd0 [37934.233063] nf_defrag_ipv6 [37934.237501] nf_defrag_ipv4 [37934.246011] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37934.248801] nft_compat nf_tables [37934.251593] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37934.260554] nfnetlink [37934.263865] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [37934.271335] bridge stp [37934.273694] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [37934.279171] llc xfs [37934.281621] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37934.300362] libcrc32c [37934.302546] [37934.302546] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37934.307761] sunrpc intel_rapl_msr [37934.310119] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37934.311622] intel_rapl_common [37934.318744] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37934.318746] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37934.322141] sb_edac [37934.329264] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37934.332315] x86_pkg_temp_thermal [37934.339440] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37934.346565] coretemp [37934.348747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37934.355871] kvm_intel kvm [37934.359183] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37934.367260] irqbypass [37934.369530] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37934.375267] i2c_i801 iTCO_wdt [37934.377973] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37934.385097] iTCO_vendor_support [37934.387453] Call Trace: [37934.387455] [37934.394577] i2c_smbus crct10dif_pclmul [37934.397630] ? watchdog_timer_fn+0x1b4/0x220 [37934.404754] joydev [37934.407978] ? __hrtimer_run_queues+0x1d3/0x360 [37934.410421] crc32_pclmul ipmi_ssif [37934.412435] ? hrtimer_interrupt+0x104/0x240 [37934.416262] ghash_clmulni_intel mei_me [37934.420526] ? __local_bh_enable+0x50/0x60 [37934.422624] mei [37934.427150] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37934.430632] lpc_ich [37934.434913] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37934.438745] sha512_ssse3 acpi_ipmi [37934.442835] [37934.444673] mfd_core [37934.449890] [37934.449909] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37934.452092] wmi [37934.457053] ? queued_spin_lock_slowpath+0x18/0x2e0 [37934.460532] ipmi_si [37934.462630] do_raw_spin_lock+0xb5/0xc0 [37934.464902] ipmi_devintf [37934.466998] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37934.472301] ipmi_msghandler ioatdma [37934.474141] try_get_cap_refs+0x159/0x880 [ceph] [37934.479011] acpi_pad acpi_power_meter [37934.481200] __ceph_get_caps+0x4a1/0x760 [ceph] [37934.485025] ixgbe igb [37934.487647] ? __file_remove_privs+0x89/0x110 [37934.492425] mdio [37934.495997] ? wake_bit_function+0x70/0x70 [37934.500623] crc32c_intel [37934.504371] ? ceph_write_iter+0x260/0x770 [ceph] [37934.508909] nvme ptp [37934.511268] ceph_write_iter+0x260/0x770 [ceph] [37934.515625] nvme_core i2c_algo_bit [37934.517555] ? lock_is_held_type+0xaa/0x120 [37934.521641] pps_core [37934.524262] vfs_write+0x324/0x4f0 [37934.528957] dca fuse [37934.531232] ksys_pwrite64+0x65/0xa0 [37934.535751] [37934.535752] irq event stamp: 223936772 [37934.539236] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37934.543414] hardirqs last enabled at (223936771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37934.545685] do_syscall_64+0x38/0x80 [37934.549082] hardirqs last disabled at (223936772): [] sysvec_apic_timer_interrupt+0xe/0x90 [37934.551353] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37934.554923] softirqs last enabled at (223936770): [] __do_softirq+0x2be/0x371 [37934.556414] RIP: 0033:0x7f48f0eeb93b [37934.560157] softirqs last disabled at (223936765): [] irq_exit_rcu+0xb0/0xd0 [37934.565288] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37934.575532] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37934.579104] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37934.588913] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37934.593957] ORIG_RAX: 0000000000000012 [37934.602730] Workqueue: events delayed_work [ceph] [37934.606298] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37934.614908] [37934.633645] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37934.643021] RIP: 0010:lock_acquire+0x4a/0x2b0 [37934.648237] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37934.655709] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [37934.659539] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37934.664238] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [37934.671362] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37934.672851] [37934.679983] [37934.684326] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [37934.740454] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [37934.747579] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [37934.754701] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [37934.761826] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37934.768951] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37934.777028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37934.782765] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37934.789890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37934.797013] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37934.804137] Call Trace: [37934.806584] [37934.808592] ? watchdog_timer_fn+0x1b4/0x220 [37934.812858] ? __hrtimer_run_queues+0x1d3/0x360 [37934.817385] ? hrtimer_interrupt+0x104/0x240 [37934.821654] ? __local_bh_enable+0x50/0x60 [37934.825747] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37934.830970] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37934.835938] [37934.838034] [37934.840131] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37934.845441] ? lock_acquire+0x4a/0x2b0 [37934.849191] ? igrab+0x43/0x50 [37934.852249] ? lock_release+0xc5/0x260 [37934.855993] _raw_spin_lock+0x30/0x40 [37934.859648] ? igrab+0x1d/0x50 [37934.862699] igrab+0x1d/0x50 [37934.865578] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37934.870728] delayed_work+0x1d8/0x280 [ceph] [37934.875010] ? process_scheduled_works+0x22a/0x510 [37934.879800] process_scheduled_works+0x2a4/0x510 [37934.884413] worker_thread+0x188/0x340 [37934.888163] ? rescuer_thread+0x310/0x310 [37934.892167] kthread+0xf5/0x130 [37934.895305] ? kthread_complete_and_exit+0x20/0x20 [37934.900089] ret_from_fork+0x31/0x40 [37934.903660] ? kthread_complete_and_exit+0x20/0x20 [37934.908443] ret_from_fork_asm+0x11/0x20 [37934.912367] [37962.119694] watchdog: BUG: soft lockup - CPU#0 stuck for 24240s! [dbench:78683] [37962.127002] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37962.131694] watchdog: BUG: soft lockup - CPU#7 stuck for 33237s! [kworker/7:4:78808] [37962.182451] irq event stamp: 66911994 [37962.190182] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [37962.193839] hardirqs last enabled at (66911993): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37962.202011] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [37962.212168] hardirqs last disabled at (66911994): [] sysvec_apic_timer_interrupt+0xe/0x90 [37962.220160] nf_tables nfnetlink bridge stp llc xfs libcrc32c [37962.229884] softirqs last enabled at (66911958): [] __do_softirq+0x2be/0x371 [37962.235629] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [37962.244314] softirqs last disabled at (66911953): [] irq_exit_rcu+0xb0/0xd0 [37962.251784] coretemp kvm_intel kvm irqbypass i2c_i801 [37962.260298] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37962.265428] iTCO_wdt iTCO_vendor_support [37962.274373] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37962.278374] i2c_smbus crct10dif_pclmul joydev crc32_pclmul [37962.285845] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [37962.291409] ipmi_ssif ghash_clmulni_intel mei_me mei [37962.296889] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [37962.301933] lpc_ich sha512_ssse3 acpi_ipmi [37962.320669] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37962.320671] mfd_core [37962.320672] [37962.324845] wmi ipmi_si [37962.330065] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37962.332353] ipmi_devintf [37962.333859] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37962.336393] ipmi_msghandler [37962.343515] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37962.346133] ioatdma acpi_pad [37962.353257] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37962.356135] acpi_power_meter [37962.363259] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37962.366222] ixgbe igb [37962.373346] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37962.376310] mdio crc32c_intel [37962.383434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37962.385791] nvme ptp [37962.393868] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37962.396919] nvme_core i2c_algo_bit [37962.402657] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37962.404928] pps_core [37962.412053] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37962.415537] dca fuse [37962.422659] Call Trace: [37962.422662] [37962.424932] [37962.432056] ? watchdog_timer_fn+0x1b4/0x220 [37962.434326] irq event stamp: 224157496 [37962.436770] ? __hrtimer_run_queues+0x1d3/0x360 [37962.438781] hardirqs last enabled at (224157495): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37962.440274] ? hrtimer_interrupt+0x104/0x240 [37962.444535] hardirqs last disabled at (224157496): [] sysvec_apic_timer_interrupt+0xe/0x90 [37962.448280] ? __local_bh_enable+0x50/0x60 [37962.452803] softirqs last enabled at (224157494): [] __do_softirq+0x2be/0x371 [37962.463051] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37962.467313] softirqs last disabled at (224157489): [] irq_exit_rcu+0xb0/0xd0 [37962.477124] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37962.481213] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37962.489983] [37962.495202] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37962.503799] [37962.503801] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37962.508758] Workqueue: events delayed_work [ceph] [37962.518137] ? queued_spin_lock_slowpath+0x2d/0x2e0 [37962.520232] [37962.527705] do_raw_spin_lock+0xb5/0xc0 [37962.529798] RIP: 0010:do_raw_spin_unlock+0x45/0xc0 [37962.535107] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37962.539802] Code: 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 8b 05 4a bf eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 10 89 43 08 03 00 5b c3 e8 41 21 45 00 85 c0 74 c3 48 c7 c6 29 e2 12 82 48 [37962.544674] try_get_cap_refs+0x159/0x880 [ceph] [37962.546161] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [37962.550000] __ceph_get_caps+0x4a1/0x760 [ceph] [37962.554776] [37962.554777] RAX: ffffffffffffffff RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [37962.559567] ? __file_remove_privs+0x89/0x110 [37962.578299] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [37962.582910] ? wake_bit_function+0x70/0x70 [37962.588126] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [37962.592655] ? ceph_write_iter+0x260/0x770 [ceph] [37962.594142] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [37962.601266] ceph_write_iter+0x260/0x770 [ceph] [37962.605617] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [37962.612746] ? lock_is_held_type+0xaa/0x120 [37962.616833] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37962.623958] vfs_write+0x324/0x4f0 [37962.628654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37962.635781] ksys_pwrite64+0x65/0xa0 [37962.640303] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37962.647426] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37962.651604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37962.659682] do_syscall_64+0x38/0x80 [37962.663095] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37962.668834] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37962.672402] Call Trace: [37962.672404] [37962.679527] RIP: 0033:0x7f48f0eeb93b [37962.684660] ? watchdog_timer_fn+0x1b4/0x220 [37962.691784] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37962.695354] ? __hrtimer_run_queues+0x1d3/0x360 [37962.702477] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37962.707526] ? hrtimer_interrupt+0x104/0x240 [37962.709965] ORIG_RAX: 0000000000000012 [37962.711977] ? __local_bh_enable+0x50/0x60 [37962.715547] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37962.719813] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37962.738549] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37962.743075] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37962.748307] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37962.752573] [37962.756403] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37962.760493] [37962.767618] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37962.772836] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37962.779964] [37962.784920] ? do_raw_spin_unlock+0x45/0xc0 [37962.822169] _raw_spin_unlock+0x23/0x40 [37962.826009] igrab+0x43/0x50 [37962.828886] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37962.834034] delayed_work+0x1d8/0x280 [ceph] [37962.838315] ? process_scheduled_works+0x22a/0x510 [37962.843106] process_scheduled_works+0x2a4/0x510 [37962.847722] worker_thread+0x188/0x340 [37962.851472] ? rescuer_thread+0x310/0x310 [37962.855482] kthread+0xf5/0x130 [37962.858621] ? kthread_complete_and_exit+0x20/0x20 [37962.863403] ret_from_fork+0x31/0x40 [37962.866975] ? kthread_complete_and_exit+0x20/0x20 [37962.871760] ret_from_fork_asm+0x11/0x20 [37962.875690] [37990.120159] watchdog: BUG: soft lockup - CPU#0 stuck for 24267s! [dbench:78683] [37990.127463] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [37990.132159] watchdog: BUG: soft lockup - CPU#7 stuck for 33264s! [kworker/7:4:78808] [37990.182902] irq event stamp: 66970074 [37990.182904] hardirqs last enabled at (66970073): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37990.190633] Modules linked in: [37990.194292] hardirqs last disabled at (66970074): [] sysvec_apic_timer_interrupt+0xe/0x90 [37990.204448] ceph [37990.207499] softirqs last enabled at (66970026): [] __do_softirq+0x2be/0x371 [37990.217224] libceph [37990.219147] softirqs last disabled at (66970021): [] irq_exit_rcu+0xb0/0xd0 [37990.227833] dns_resolver [37990.230017] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37990.238526] fscache [37990.241162] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37990.250107] netfs veth [37990.252289] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [37990.259761] nft_chain_nat [37990.262206] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [37990.267682] xt_MASQUERADE nf_nat [37990.270387] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37990.289125] nf_conntrack [37990.292435] [37990.292435] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37990.297652] nf_defrag_ipv6 [37990.300271] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37990.301760] nf_defrag_ipv4 nft_compat [37990.308885] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37990.311675] nf_tables [37990.318800] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37990.322555] nfnetlink bridge [37990.329676] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37990.332033] stp llc [37990.339158] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37990.342121] xfs libcrc32c [37990.349246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37990.351429] sunrpc [37990.359507] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37990.362210] intel_rapl_msr intel_rapl_common [37990.367948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37990.370046] sb_edac x86_pkg_temp_thermal [37990.377169] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37990.381522] coretemp kvm_intel [37990.388644] Call Trace: [37990.388646] [37990.392649] kvm irqbypass [37990.399774] ? watchdog_timer_fn+0x1b4/0x220 [37990.402911] i2c_i801 [37990.405355] ? __hrtimer_run_queues+0x1d3/0x360 [37990.407366] iTCO_wdt [37990.410073] ? hrtimer_interrupt+0x104/0x240 [37990.414334] iTCO_vendor_support [37990.416605] ? __local_bh_enable+0x50/0x60 [37990.421128] i2c_smbus [37990.423402] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37990.427663] crct10dif_pclmul [37990.430889] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37990.434978] joydev [37990.437337] [37990.442557] crc32_pclmul [37990.445525] [37990.445527] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37990.450484] ipmi_ssif ghash_clmulni_intel [37990.452585] ? queued_spin_lock_slowpath+0x16/0x2e0 [37990.454677] mei_me mei [37990.457296] do_raw_spin_lock+0xb5/0xc0 [37990.459392] lpc_ich sha512_ssse3 [37990.464696] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37990.468787] acpi_ipmi mfd_core [37990.473661] try_get_cap_refs+0x159/0x880 [ceph] [37990.476101] wmi ipmi_si [37990.479939] __ceph_get_caps+0x4a1/0x760 [ceph] [37990.483243] ipmi_devintf [37990.488034] ? __file_remove_privs+0x89/0x110 [37990.491165] ipmi_msghandler [37990.495776] ? wake_bit_function+0x70/0x70 [37990.498306] ioatdma [37990.502842] ? ceph_write_iter+0x260/0x770 [ceph] [37990.505458] acpi_pad acpi_power_meter [37990.509808] ceph_write_iter+0x260/0x770 [ceph] [37990.512684] ixgbe igb [37990.516779] ? lock_is_held_type+0xaa/0x120 [37990.518960] mdio crc32c_intel [37990.523659] vfs_write+0x324/0x4f0 [37990.527401] nvme ptp [37990.531930] ksys_pwrite64+0x65/0xa0 [37990.534282] nvme_core i2c_algo_bit [37990.538461] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37990.541511] pps_core dca [37990.544910] do_syscall_64+0x38/0x80 [37990.547179] fuse [37990.550749] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37990.554232] [37990.559363] RIP: 0033:0x7f48f0eeb93b [37990.561982] irq event stamp: 224378550 [37990.565559] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37990.567486] hardirqs last enabled at (224378549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37990.572528] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [37990.574019] hardirqs last disabled at (224378550): [] sysvec_apic_timer_interrupt+0xe/0x90 [37990.577590] ORIG_RAX: 0000000000000012 [37990.581335] softirqs last enabled at (224378548): [] __do_softirq+0x2be/0x371 [37990.600072] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37990.610316] softirqs last disabled at (224378543): [] irq_exit_rcu+0xb0/0xd0 [37990.615534] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37990.625362] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37990.629194] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37990.637964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37990.645088] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37990.653686] Workqueue: events delayed_work [ceph] [37990.660809] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37990.670188] [37990.677317] [37990.684781] RIP: 0010:lock_acquire+0xf0/0x2b0 [37990.711753] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [37990.730491] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [37990.735707] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [37990.742848] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [37990.749974] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [37990.757099] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [37990.764223] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [37990.771347] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37990.779423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37990.785160] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37990.792285] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37990.799409] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37990.806533] Call Trace: [37990.808981] [37990.810989] ? watchdog_timer_fn+0x1b4/0x220 [37990.815254] ? __hrtimer_run_queues+0x1d3/0x360 [37990.819780] ? hrtimer_interrupt+0x104/0x240 [37990.824050] ? __local_bh_enable+0x50/0x60 [37990.828143] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37990.833368] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37990.838324] [37990.840421] [37990.842520] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37990.847844] ? lock_acquire+0xf0/0x2b0 [37990.851593] ? igrab+0x43/0x50 [37990.854644] ? lock_release+0xc5/0x260 [37990.858390] _raw_spin_lock+0x30/0x40 [37990.862054] ? igrab+0x1d/0x50 [37990.865104] igrab+0x1d/0x50 [37990.867984] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37990.873133] delayed_work+0x1d8/0x280 [ceph] [37990.877413] ? process_scheduled_works+0x22a/0x510 [37990.882205] process_scheduled_works+0x2a4/0x510 [37990.886819] worker_thread+0x188/0x340 [37990.890570] ? rescuer_thread+0x310/0x310 [37990.894580] kthread+0xf5/0x130 [37990.897717] ? kthread_complete_and_exit+0x20/0x20 [37990.902503] ret_from_fork+0x31/0x40 [37990.906081] ? kthread_complete_and_exit+0x20/0x20 [37990.910867] ret_from_fork_asm+0x11/0x20 [37990.914799] [37992.258561] rcu: INFO: rcu_preempt self-detected stall on CPU [37992.264322] rcu: 0-....: (25466362 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=12753456 [37992.274833] rcu: (t=26055192 jiffies g=1309609 q=709536 ncpus=8) [37992.280919] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37992.289861] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37992.297333] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [37992.302809] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [37992.321546] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [37992.326765] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [37992.333889] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [37992.341013] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [37992.348137] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [37992.355260] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [37992.362384] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [37992.370463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37992.376200] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [37992.383324] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37992.390448] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37992.397573] Call Trace: [37992.400016] [37992.402029] ? rcu_dump_cpu_stacks+0xd1/0x180 [37992.406387] ? rcu_sched_clock_irq+0x633/0x1390 [37992.410911] ? lock_is_held_type+0xaa/0x120 [37992.415089] ? lock_is_held_type+0xaa/0x120 [37992.419267] ? tick_sched_do_timer+0xa0/0xa0 [37992.423538] ? rcu_read_lock_sched_held+0x56/0x90 [37992.428236] ? tick_sched_do_timer+0xa0/0xa0 [37992.432499] ? update_process_times+0x5f/0x90 [37992.436850] ? tick_sched_handle.isra.16+0x1e/0x50 [37992.441636] ? tick_sched_timer+0x6f/0x80 [37992.445639] ? __hrtimer_run_queues+0x1d3/0x360 [37992.450166] ? hrtimer_interrupt+0x104/0x240 [37992.454437] ? __local_bh_enable+0x50/0x60 [37992.458529] ? __sysvec_apic_timer_interrupt+0x61/0x160 [37992.463753] ? sysvec_apic_timer_interrupt+0x6d/0x90 [37992.468710] [37992.470806] [37992.472906] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [37992.478213] ? queued_spin_lock_slowpath+0x2b/0x2e0 [37992.483091] do_raw_spin_lock+0xb5/0xc0 [37992.486927] __cap_delay_requeue+0xa7/0x1d0 [ceph] [37992.491730] try_get_cap_refs+0x159/0x880 [ceph] [37992.496363] __ceph_get_caps+0x4a1/0x760 [ceph] [37992.500904] ? __file_remove_privs+0x89/0x110 [37992.505257] ? wake_bit_function+0x70/0x70 [37992.509351] ? ceph_write_iter+0x260/0x770 [ceph] [37992.514065] ceph_write_iter+0x260/0x770 [ceph] [37992.518599] ? lock_is_held_type+0xaa/0x120 [37992.522784] vfs_write+0x324/0x4f0 [37992.526184] ksys_pwrite64+0x65/0xa0 [37992.529759] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [37992.534890] do_syscall_64+0x38/0x80 [37992.538461] entry_SYSCALL_64_after_hwframe+0x63/0xcd [37992.543505] RIP: 0033:0x7f48f0eeb93b [37992.547075] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [37992.565812] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [37992.573370] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [37992.580494] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [37992.587618] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [37992.594744] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [37992.601867] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [37992.608998] [37992.611184] Sending NMI from CPU 0 to CPUs 7: [37992.615536] NMI backtrace for cpu 7 [37992.615538] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [37992.615539] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [37992.615540] Workqueue: events delayed_work [ceph] [37992.615555] RIP: 0010:mark_lock+0x5f/0x4c0 [37992.615557] Code: d3 83 fa 08 74 6c 41 be 01 00 00 00 89 d1 41 d3 e6 4d 63 f6 41 0f b7 45 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 d3 7e e0 01 73 79 <48> 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 41 bc 01 00 [37992.615558] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000007 [37992.615559] RAX: 00000000000006f5 RBX: 0000000000000008 RCX: 0000000000000000 [37992.615560] RDX: 0000000000000008 RSI: ffff88810debb550 RDI: ffff88810debaa80 [37992.615561] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [37992.615562] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [37992.615562] R13: ffff88810debb550 R14: 0000000000000100 R15: ffff88810debaa80 [37992.615563] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [37992.615564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [37992.615565] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [37992.615566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [37992.615567] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [37992.615568] Call Trace: [37992.615569] [37992.615571] ? nmi_cpu_backtrace+0xad/0x120 [37992.615574] ? nmi_cpu_backtrace_handler+0x11/0x20 [37992.615575] ? nmi_handle+0xd4/0x260 [37992.615579] ? default_do_nmi+0x49/0xf0 [37992.615581] ? exc_nmi+0x11e/0x150 [37992.615582] ? end_repeat_nmi+0x16/0x67 [37992.615586] ? mark_lock+0x5f/0x4c0 [37992.615587] ? mark_lock+0x5f/0x4c0 [37992.615589] ? mark_lock+0x5f/0x4c0 [37992.615590] [37992.615591] [37992.615592] ? _raw_spin_unlock+0x23/0x40 [37992.615594] ? ceph_check_caps+0x275/0xae0 [ceph] [37992.615607] ? mark_held_locks+0x48/0x70 [37992.615609] __lock_acquire+0x4d6/0x1e20 [37992.615612] lock_acquire+0xcb/0x2b0 [37992.615613] ? igrab+0x1d/0x50 [37992.615615] ? igrab+0x43/0x50 [37992.615616] ? lock_release+0xc5/0x260 [37992.615618] _raw_spin_lock+0x30/0x40 [37992.615620] ? igrab+0x1d/0x50 [37992.615621] igrab+0x1d/0x50 [37992.615622] ceph_check_delayed_caps+0xb7/0x270 [ceph] [37992.615635] delayed_work+0x1d8/0x280 [ceph] [37992.615648] ? process_scheduled_works+0x22a/0x510 [37992.615650] process_scheduled_works+0x2a4/0x510 [37992.615654] worker_thread+0x188/0x340 [37992.615656] ? rescuer_thread+0x310/0x310 [37992.615658] kthread+0xf5/0x130 [37992.615660] ? kthread_complete_and_exit+0x20/0x20 [37992.615662] ret_from_fork+0x31/0x40 [37992.615665] ? kthread_complete_and_exit+0x20/0x20 [37992.615667] ret_from_fork_asm+0x11/0x20 [37992.615670] [38018.120624] watchdog: BUG: soft lockup - CPU#0 stuck for 24293s! [dbench:78683] [38018.127924] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38018.132624] watchdog: BUG: soft lockup - CPU#7 stuck for 33290s! [kworker/7:4:78808] [38018.183375] irq event stamp: 67027648 [38018.191104] Modules linked in: ceph libceph dns_resolver fscache netfs [38018.194762] hardirqs last enabled at (67027647): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38018.201298] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38018.211454] hardirqs last disabled at (67027648): [] sysvec_apic_timer_interrupt+0xe/0x90 [38018.217537] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [38018.227263] softirqs last enabled at (67027600): [] __do_softirq+0x2be/0x371 [38018.234041] bridge stp llc xfs libcrc32c sunrpc [38018.242724] softirqs last disabled at (67027595): [] irq_exit_rcu+0xb0/0xd0 [38018.247337] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [38018.255847] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38018.262709] coretemp kvm_intel kvm irqbypass [38018.271654] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38018.271656] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38018.276017] i2c_i801 iTCO_wdt [38018.283483] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38018.288962] iTCO_vendor_support [38018.292018] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38018.310757] i2c_smbus crct10dif_pclmul [38018.313982] [38018.319200] joydev crc32_pclmul [38018.323032] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38018.324522] ipmi_ssif [38018.327746] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38018.334869] ghash_clmulni_intel mei_me [38018.337227] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38018.344351] mei lpc_ich [38018.348182] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38018.355312] sha512_ssse3 [38018.357846] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38018.364970] acpi_ipmi [38018.367586] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38018.374711] mfd_core wmi [38018.377069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38018.385145] ipmi_si ipmi_devintf [38018.387763] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38018.393500] ipmi_msghandler ioatdma [38018.396812] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38018.403937] acpi_pad [38018.407506] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38018.414631] acpi_power_meter [38018.416902] Call Trace: [38018.424026] ixgbe [38018.426993] [38018.429433] igb mdio [38018.431446] ? watchdog_timer_fn+0x1b4/0x220 [38018.433454] crc32c_intel nvme [38018.435727] ? __hrtimer_run_queues+0x1d3/0x360 [38018.439990] ptp nvme_core [38018.443044] ? hrtimer_interrupt+0x104/0x240 [38018.447565] i2c_algo_bit pps_core [38018.450268] ? __local_bh_enable+0x50/0x60 [38018.454533] dca fuse [38018.457931] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38018.462023] [38018.462024] irq event stamp: 224597276 [38018.464303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38018.469517] hardirqs last enabled at (224597275): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38018.471024] [38018.474787] hardirqs last disabled at (224597276): [] sysvec_apic_timer_interrupt+0xe/0x90 [38018.479744] [38018.489988] softirqs last enabled at (224597274): [] __do_softirq+0x2be/0x371 [38018.492086] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38018.501896] softirqs last disabled at (224597269): [] irq_exit_rcu+0xb0/0xd0 [38018.504000] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38018.512766] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38018.518070] do_raw_spin_lock+0xb5/0xc0 [38018.526666] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38018.531538] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38018.540916] Workqueue: events delayed_work [ceph] [38018.544750] try_get_cap_refs+0x159/0x880 [ceph] [38018.552215] [38018.552216] RIP: 0010:lock_acquire+0xf0/0x2b0 [38018.557026] __ceph_get_caps+0x4a1/0x760 [ceph] [38018.561715] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38018.566331] ? __file_remove_privs+0x89/0x110 [38018.567816] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38018.572167] ? wake_bit_function+0x70/0x70 [38018.576692] [38018.576693] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38018.595433] ? ceph_write_iter+0x260/0x770 [ceph] [38018.599779] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38018.604999] ceph_write_iter+0x260/0x770 [ceph] [38018.609106] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38018.610601] ? lock_is_held_type+0xaa/0x120 [38018.617721] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38018.622421] vfs_write+0x324/0x4f0 [38018.629542] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38018.634071] ksys_pwrite64+0x65/0xa0 [38018.641189] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38018.645369] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38018.652492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38018.655891] do_syscall_64+0x38/0x80 [38018.663026] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38018.666594] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38018.674670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38018.679801] RIP: 0033:0x7f48f0eeb93b [38018.685537] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38018.689110] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38018.696232] Call Trace: [38018.696234] [38018.701276] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38018.708401] ? watchdog_timer_fn+0x1b4/0x220 [38018.711972] ORIG_RAX: 0000000000000012 [38018.719098] ? __hrtimer_run_queues+0x1d3/0x360 [38018.737834] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38018.740281] ? hrtimer_interrupt+0x104/0x240 [38018.742288] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38018.747505] ? __local_bh_enable+0x50/0x60 [38018.751771] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38018.755605] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38018.760124] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38018.767249] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38018.771513] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38018.778637] [38018.782733] [38018.789852] [38018.789854] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38018.825963] ? lock_acquire+0xf0/0x2b0 [38018.829712] ? igrab+0x43/0x50 [38018.832762] ? lock_release+0xc5/0x260 [38018.836508] _raw_spin_lock+0x30/0x40 [38018.840173] ? igrab+0x1d/0x50 [38018.843222] igrab+0x1d/0x50 [38018.846103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38018.851252] delayed_work+0x1d8/0x280 [ceph] [38018.855532] ? process_scheduled_works+0x22a/0x510 [38018.860322] process_scheduled_works+0x2a4/0x510 [38018.864939] worker_thread+0x188/0x340 [38018.868688] ? rescuer_thread+0x310/0x310 [38018.872698] kthread+0xf5/0x130 [38018.875838] ? kthread_complete_and_exit+0x20/0x20 [38018.880620] ret_from_fork+0x31/0x40 [38018.884201] ? kthread_complete_and_exit+0x20/0x20 [38018.888984] ret_from_fork_asm+0x11/0x20 [38018.892909] [38046.121086] watchdog: BUG: soft lockup - CPU#0 stuck for 24319s! [dbench:78683] [38046.128384] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38046.133086] watchdog: BUG: soft lockup - CPU#7 stuck for 33316s! [kworker/7:4:78808] [38046.183828] irq event stamp: 67085736 [38046.183829] hardirqs last enabled at (67085735): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38046.191559] Modules linked in: [38046.195215] hardirqs last disabled at (67085736): [] sysvec_apic_timer_interrupt+0xe/0x90 [38046.205373] ceph [38046.208423] softirqs last enabled at (67085688): [] __do_softirq+0x2be/0x371 [38046.218148] libceph dns_resolver [38046.220071] softirqs last disabled at (67085683): [] irq_exit_rcu+0xb0/0xd0 [38046.228764] fscache netfs [38046.232074] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38046.240585] veth nft_chain_nat [38046.243289] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38046.252234] xt_MASQUERADE [38046.255372] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38046.262842] nf_nat nf_conntrack [38046.265545] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38046.271023] nf_defrag_ipv6 [38046.274249] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38046.292985] nf_defrag_ipv4 nft_compat [38046.295775] [38046.295775] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38046.300994] nf_tables nfnetlink [38046.304738] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38046.306227] bridge stp [38046.313351] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38046.316575] llc xfs [38046.323699] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38046.326144] libcrc32c sunrpc [38046.333269] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38046.335453] intel_rapl_msr intel_rapl_common [38046.342578] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38046.345542] sb_edac x86_pkg_temp_thermal [38046.352665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38046.357015] coretemp kvm_intel [38046.365094] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38046.369097] kvm irqbypass [38046.374836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38046.377971] i2c_i801 iTCO_wdt [38046.385097] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38046.387800] iTCO_vendor_support i2c_smbus [38046.394924] Call Trace: [38046.394926] [38046.397974] crct10dif_pclmul [38046.405099] ? watchdog_timer_fn+0x1b4/0x220 [38046.409189] joydev crc32_pclmul [38046.411635] ? __hrtimer_run_queues+0x1d3/0x360 [38046.413643] ipmi_ssif ghash_clmulni_intel [38046.416612] ? hrtimer_interrupt+0x104/0x240 [38046.420872] mei_me [38046.424096] ? __local_bh_enable+0x50/0x60 [38046.428620] mei lpc_ich [38046.432713] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38046.436976] sha512_ssse3 acpi_ipmi [38046.439074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38046.443164] mfd_core wmi [38046.445696] [38046.450911] ipmi_si ipmi_devintf [38046.454396] [38046.454397] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38046.459353] ipmi_msghandler ioatdma [38046.461974] ? queued_spin_lock_slowpath+0x18/0x2e0 [38046.464068] acpi_pad acpi_power_meter [38046.467381] do_raw_spin_lock+0xb5/0xc0 [38046.469484] ixgbe igb [38046.474789] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38046.478359] mdio crc32c_intel [38046.483234] try_get_cap_refs+0x159/0x880 [ceph] [38046.486975] nvme ptp [38046.490811] __ceph_get_caps+0x4a1/0x760 [ceph] [38046.493163] nvme_core [38046.497950] ? __file_remove_privs+0x89/0x110 [38046.500997] i2c_algo_bit pps_core [38046.505609] ? wake_bit_function+0x70/0x70 [38046.507878] dca fuse [38046.512405] ? ceph_write_iter+0x260/0x770 [ceph] [38046.514766] [38046.514766] irq event stamp: 224818174 [38046.519121] ceph_write_iter+0x260/0x770 [ceph] [38046.522517] hardirqs last enabled at (224818173): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38046.526613] ? lock_is_held_type+0xaa/0x120 [38046.528880] hardirqs last disabled at (224818174): [] sysvec_apic_timer_interrupt+0xe/0x90 [38046.533579] vfs_write+0x324/0x4f0 [38046.535066] softirqs last enabled at (224818172): [] __do_softirq+0x2be/0x371 [38046.538814] ksys_pwrite64+0x65/0xa0 [38046.543335] softirqs last disabled at (224818167): [] irq_exit_rcu+0xb0/0xd0 [38046.553579] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38046.557757] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38046.567570] do_syscall_64+0x38/0x80 [38046.570964] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38046.579737] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38046.583307] Workqueue: events delayed_work [ceph] [38046.591905] RIP: 0033:0x7f48f0eeb93b [38046.597034] [38046.597035] RIP: 0010:lock_acquire+0xf0/0x2b0 [38046.606414] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38046.609983] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38046.617453] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38046.622497] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38046.627195] ORIG_RAX: 0000000000000012 [38046.630767] [38046.630768] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38046.632256] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38046.636606] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38046.655346] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38046.674083] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38046.679300] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38046.684519] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38046.688349] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38046.689839] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38046.696964] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38046.704087] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38046.711216] [38046.718335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38046.777079] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38046.784203] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38046.791328] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38046.798453] Call Trace: [38046.800896] [38046.802909] ? watchdog_timer_fn+0x1b4/0x220 [38046.807173] ? __hrtimer_run_queues+0x1d3/0x360 [38046.811699] ? hrtimer_interrupt+0x104/0x240 [38046.815968] ? __local_bh_enable+0x50/0x60 [38046.820061] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38046.825285] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38046.830242] [38046.832339] [38046.834437] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38046.839745] ? lock_acquire+0xf0/0x2b0 [38046.843512] ? igrab+0x43/0x50 [38046.846562] ? lock_release+0xc5/0x260 [38046.850307] _raw_spin_lock+0x30/0x40 [38046.853963] ? igrab+0x1d/0x50 [38046.857014] igrab+0x1d/0x50 [38046.859894] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38046.865043] delayed_work+0x1d8/0x280 [ceph] [38046.869324] ? process_scheduled_works+0x22a/0x510 [38046.874113] process_scheduled_works+0x2a4/0x510 [38046.878730] worker_thread+0x188/0x340 [38046.882498] ? rescuer_thread+0x310/0x310 [38046.886507] kthread+0xf5/0x130 [38046.889644] ? kthread_complete_and_exit+0x20/0x20 [38046.894429] ret_from_fork+0x31/0x40 [38046.898009] ? kthread_complete_and_exit+0x20/0x20 [38046.902793] ret_from_fork_asm+0x11/0x20 [38046.906725] [38074.121548] watchdog: BUG: soft lockup - CPU#0 stuck for 24345s! [dbench:78683] [38074.128856] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38074.133547] watchdog: BUG: soft lockup - CPU#7 stuck for 33342s! [kworker/7:4:78808] [38074.184306] irq event stamp: 67143982 [38074.184307] hardirqs last enabled at (67143981): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38074.192037] Modules linked in: ceph [38074.195693] hardirqs last disabled at (67143982): [] sysvec_apic_timer_interrupt+0xe/0x90 [38074.205851] libceph [38074.209336] softirqs last enabled at (67143934): [] __do_softirq+0x2be/0x371 [38074.219059] dns_resolver [38074.221242] softirqs last disabled at (67143929): [] irq_exit_rcu+0xb0/0xd0 [38074.229942] fscache [38074.232563] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38074.241072] netfs [38074.243256] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38074.252201] veth nft_chain_nat [38074.254235] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38074.261710] xt_MASQUERADE nf_nat [38074.264847] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38074.270324] nf_conntrack nf_defrag_ipv6 [38074.273634] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38074.292371] nf_defrag_ipv4 nft_compat [38074.296290] [38074.301507] nf_tables [38074.305252] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38074.306743] nfnetlink bridge [38074.309098] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38074.316224] stp llc [38074.319187] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38074.326311] xfs libcrc32c [38074.328495] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38074.335619] sunrpc [38074.338323] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38074.345447] intel_rapl_msr [38074.347546] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38074.354668] intel_rapl_common [38074.357459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38074.365536] sb_edac x86_pkg_temp_thermal [38074.368588] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38074.374326] coretemp kvm_intel [38074.378328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38074.385453] kvm irqbypass [38074.388590] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38074.395716] i2c_i801 [38074.398419] Call Trace: [38074.398420] [38074.405542] iTCO_wdt [38074.407817] ? watchdog_timer_fn+0x1b4/0x220 [38074.410259] iTCO_vendor_support [38074.412270] ? __hrtimer_run_queues+0x1d3/0x360 [38074.414539] i2c_smbus crct10dif_pclmul [38074.418808] ? hrtimer_interrupt+0x104/0x240 [38074.422027] joydev crc32_pclmul [38074.426552] ? __local_bh_enable+0x50/0x60 [38074.430382] ipmi_ssif ghash_clmulni_intel [38074.434650] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38074.437870] mei_me mei [38074.441961] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38074.446052] lpc_ich [38074.451269] [38074.453714] sha512_ssse3 [38074.458672] [38074.458674] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38074.460856] acpi_ipmi [38074.462956] ? queued_spin_lock_slowpath+0x16/0x2e0 [38074.465569] mfd_core [38074.467668] do_raw_spin_lock+0xb5/0xc0 [38074.472972] wmi [38074.475330] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38074.480199] ipmi_si ipmi_devintf [38074.482490] try_get_cap_refs+0x159/0x880 [ceph] [38074.486319] ipmi_msghandler [38074.488160] __ceph_get_caps+0x4a1/0x760 [ceph] [38074.492944] ioatdma [38074.496263] ? __file_remove_privs+0x89/0x110 [38074.500869] acpi_pad acpi_power_meter [38074.503747] ? wake_bit_function+0x70/0x70 [38074.508272] ixgbe igb [38074.510457] ? ceph_write_iter+0x260/0x770 [ceph] [38074.514806] mdio crc32c_intel [38074.518551] ceph_write_iter+0x260/0x770 [ceph] [38074.522642] nvme [38074.525002] ? lock_is_held_type+0xaa/0x120 [38074.529696] ptp nvme_core [38074.532749] vfs_write+0x324/0x4f0 [38074.537271] i2c_algo_bit [38074.539198] ksys_pwrite64+0x65/0xa0 [38074.543371] pps_core dca [38074.546076] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38074.549475] fuse [38074.552092] do_syscall_64+0x38/0x80 [38074.555660] [38074.558280] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38074.563411] irq event stamp: 225038970 [38074.565334] RIP: 0033:0x7f48f0eeb93b [38074.568903] hardirqs last enabled at (225038969): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38074.570396] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38074.575439] hardirqs last disabled at (225038970): [] sysvec_apic_timer_interrupt+0xe/0x90 [38074.579183] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38074.582755] softirqs last enabled at (225038968): [] __do_softirq+0x2be/0x371 [38074.592999] ORIG_RAX: 0000000000000012 [38074.611737] softirqs last disabled at (225038963): [] irq_exit_rcu+0xb0/0xd0 [38074.621547] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38074.626765] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38074.635535] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38074.639367] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38074.647962] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38074.655088] Workqueue: events delayed_work [ceph] [38074.664465] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38074.671590] [38074.671591] RIP: 0010:lock_acquire+0xf0/0x2b0 [38074.679060] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38074.686185] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38074.690886] [38074.698006] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38074.737110] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38074.744239] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38074.751368] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38074.758491] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38074.765616] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38074.772740] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38074.780816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38074.786555] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38074.793679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38074.800803] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38074.807926] Call Trace: [38074.810390] [38074.812401] ? watchdog_timer_fn+0x1b4/0x220 [38074.816664] ? __hrtimer_run_queues+0x1d3/0x360 [38074.821191] ? hrtimer_interrupt+0x104/0x240 [38074.825461] ? __local_bh_enable+0x50/0x60 [38074.829554] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38074.834777] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38074.839735] [38074.841832] [38074.843929] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38074.849256] ? lock_acquire+0xf0/0x2b0 [38074.853006] ? igrab+0x43/0x50 [38074.856063] ? lock_release+0xc5/0x260 [38074.859808] _raw_spin_lock+0x30/0x40 [38074.863464] ? igrab+0x1d/0x50 [38074.866514] igrab+0x1d/0x50 [38074.869395] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38074.874542] delayed_work+0x1d8/0x280 [ceph] [38074.878825] ? process_scheduled_works+0x22a/0x510 [38074.883615] process_scheduled_works+0x2a4/0x510 [38074.888247] worker_thread+0x188/0x340 [38074.891997] ? rescuer_thread+0x310/0x310 [38074.896000] kthread+0xf5/0x130 [38074.899146] ? kthread_complete_and_exit+0x20/0x20 [38074.903949] ret_from_fork+0x31/0x40 [38074.907529] ? kthread_complete_and_exit+0x20/0x20 [38074.912321] ret_from_fork_asm+0x11/0x20 [38074.916253] [38102.122006] watchdog: BUG: soft lockup - CPU#0 stuck for 24371s! [dbench:78683] [38102.129308] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38102.134006] watchdog: BUG: soft lockup - CPU#7 stuck for 33368s! [kworker/7:4:78808] [38102.184748] irq event stamp: 67202300 [38102.192480] Modules linked in: ceph libceph dns_resolver [38102.196138] hardirqs last enabled at (67202299): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38102.201441] fscache netfs [38102.211599] hardirqs last disabled at (67202300): [] sysvec_apic_timer_interrupt+0xe/0x90 [38102.214303] veth nft_chain_nat xt_MASQUERADE nf_nat [38102.224028] softirqs last enabled at (67202274): [] __do_softirq+0x2be/0x371 [38102.228985] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [38102.237681] softirqs last disabled at (67202269): [] irq_exit_rcu+0xb0/0xd0 [38102.237683] nft_compat [38102.237685] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38102.242903] nf_tables [38102.251415] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38102.253860] nfnetlink bridge [38102.262804] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38102.265159] stp llc [38102.272630] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38102.275594] xfs libcrc32c [38102.281073] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38102.283257] sunrpc intel_rapl_msr [38102.301994] [38102.301995] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38102.304698] intel_rapl_common sb_edac [38102.309915] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38102.313312] x86_pkg_temp_thermal coretemp [38102.314804] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38102.321928] kvm_intel [38102.325684] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38102.332804] kvm [38102.336897] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38102.344021] irqbypass [38102.346376] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38102.353518] i2c_i801 iTCO_wdt [38102.355356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38102.362479] iTCO_vendor_support [38102.364837] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38102.372914] i2c_smbus crct10dif_pclmul [38102.375965] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38102.381704] joydev crc32_pclmul [38102.384927] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38102.392051] ipmi_ssif ghash_clmulni_intel [38102.395882] Call Trace: [38102.395883] [38102.403007] mei_me [38102.406231] ? watchdog_timer_fn+0x1b4/0x220 [38102.413354] mei [38102.417447] ? __hrtimer_run_queues+0x1d3/0x360 [38102.419888] lpc_ich sha512_ssse3 [38102.421904] ? hrtimer_interrupt+0x104/0x240 [38102.423997] acpi_ipmi mfd_core [38102.428262] ? __local_bh_enable+0x50/0x60 [38102.430098] wmi ipmi_si [38102.434625] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38102.437934] ipmi_devintf [38102.442197] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38102.445334] ipmi_msghandler ioatdma [38102.449426] [38102.451957] acpi_pad acpi_power_meter [38102.457175] [38102.459790] ixgbe [38102.464749] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38102.468318] igb mdio [38102.470421] ? queued_spin_lock_slowpath+0x18/0x2e0 [38102.474159] crc32c_intel nvme [38102.476259] do_raw_spin_lock+0xb5/0xc0 [38102.478268] ptp nvme_core [38102.483574] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38102.485842] i2c_algo_bit pps_core [38102.490718] try_get_cap_refs+0x159/0x880 [ceph] [38102.493766] dca fuse [38102.497600] __ceph_get_caps+0x4a1/0x760 [ceph] [38102.500300] [38102.500301] irq event stamp: 225259752 [38102.505089] ? __file_remove_privs+0x89/0x110 [38102.508481] hardirqs last enabled at (225259751): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38102.513094] ? wake_bit_function+0x70/0x70 [38102.515362] hardirqs last disabled at (225259752): [] sysvec_apic_timer_interrupt+0xe/0x90 [38102.519888] ? ceph_write_iter+0x260/0x770 [ceph] [38102.521377] softirqs last enabled at (225259750): [] __do_softirq+0x2be/0x371 [38102.525123] ceph_write_iter+0x260/0x770 [ceph] [38102.529472] softirqs last disabled at (225259745): [] irq_exit_rcu+0xb0/0xd0 [38102.539720] ? lock_is_held_type+0xaa/0x120 [38102.543807] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38102.553620] vfs_write+0x324/0x4f0 [38102.558316] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38102.567092] ksys_pwrite64+0x65/0xa0 [38102.571611] Workqueue: events delayed_work [ceph] [38102.580207] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38102.584403] [38102.593781] do_syscall_64+0x38/0x80 [38102.597179] RIP: 0010:lock_acquire+0xf0/0x2b0 [38102.604650] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38102.608218] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38102.612916] RIP: 0033:0x7f48f0eeb93b [38102.618046] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38102.619539] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38102.623107] [38102.623108] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38102.627458] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38102.632503] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38102.651240] ORIG_RAX: 0000000000000012 [38102.654813] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38102.660028] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38102.678767] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38102.680258] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38102.687382] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38102.692616] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38102.699741] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38102.703572] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38102.710696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38102.717820] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38102.724943] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38102.732073] [38102.739191] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38102.790847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38102.797971] Call Trace: [38102.800417] [38102.802425] ? watchdog_timer_fn+0x1b4/0x220 [38102.806691] ? __hrtimer_run_queues+0x1d3/0x360 [38102.811218] ? hrtimer_interrupt+0x104/0x240 [38102.815486] ? __local_bh_enable+0x50/0x60 [38102.819580] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38102.824805] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38102.829761] [38102.831858] [38102.833955] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38102.839264] ? lock_acquire+0xf0/0x2b0 [38102.843013] ? igrab+0x43/0x50 [38102.846064] ? lock_release+0xc5/0x260 [38102.849810] _raw_spin_lock+0x30/0x40 [38102.853473] ? igrab+0x1d/0x50 [38102.856523] igrab+0x1d/0x50 [38102.859413] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38102.864560] delayed_work+0x1d8/0x280 [ceph] [38102.868842] ? process_scheduled_works+0x22a/0x510 [38102.873631] process_scheduled_works+0x2a4/0x510 [38102.878248] worker_thread+0x188/0x340 [38102.881999] ? rescuer_thread+0x310/0x310 [38102.886009] kthread+0xf5/0x130 [38102.889146] ? kthread_complete_and_exit+0x20/0x20 [38102.893931] ret_from_fork+0x31/0x40 [38102.897510] ? kthread_complete_and_exit+0x20/0x20 [38102.902293] ret_from_fork_asm+0x11/0x20 [38102.906218] [38130.122463] watchdog: BUG: soft lockup - CPU#0 stuck for 24397s! [dbench:78683] [38130.129760] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38130.134463] watchdog: BUG: soft lockup - CPU#7 stuck for 33394s! [kworker/7:4:78808] [38130.185202] irq event stamp: 67260492 [38130.192933] Modules linked in: ceph libceph [38130.196591] hardirqs last enabled at (67260491): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38130.200769] dns_resolver fscache netfs [38130.210926] hardirqs last disabled at (67260492): [] sysvec_apic_timer_interrupt+0xe/0x90 [38130.214757] veth nft_chain_nat xt_MASQUERADE [38130.224480] softirqs last enabled at (67260444): [] __do_softirq+0x2be/0x371 [38130.224482] nf_nat [38130.228831] nf_conntrack [38130.237533] softirqs last disabled at (67260439): [] irq_exit_rcu+0xb0/0xd0 [38130.239630] nf_defrag_ipv6 nf_defrag_ipv4 [38130.242247] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38130.250758] nft_compat [38130.254858] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38130.263802] nf_tables [38130.266245] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38130.273716] nfnetlink bridge [38130.276073] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38130.281552] stp llc [38130.284516] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38130.303252] xfs libcrc32c [38130.305437] [38130.305438] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38130.310653] sunrpc [38130.313359] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38130.314858] intel_rapl_msr intel_rapl_common [38130.321981] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38130.324079] sb_edac x86_pkg_temp_thermal [38130.331204] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38130.335555] coretemp kvm_intel [38130.342678] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38130.346682] kvm irqbypass [38130.353807] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38130.356944] i2c_i801 [38130.364067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38130.366773] iTCO_wdt iTCO_vendor_support [38130.374860] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38130.377138] i2c_smbus crct10dif_pclmul [38130.382874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38130.386880] joydev [38130.394003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38130.397834] crc32_pclmul [38130.404975] Call Trace: [38130.407073] ipmi_ssif ghash_clmulni_intel [38130.414196] [38130.416814] mei_me mei [38130.419260] ? watchdog_timer_fn+0x1b4/0x220 [38130.423350] lpc_ich [38130.425360] ? __hrtimer_run_queues+0x1d3/0x360 [38130.427803] sha512_ssse3 acpi_ipmi [38130.432073] ? hrtimer_interrupt+0x104/0x240 [38130.434253] mfd_core wmi [38130.438776] ? __local_bh_enable+0x50/0x60 [38130.442260] ipmi_si ipmi_devintf [38130.446527] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38130.449154] ipmi_msghandler [38130.453241] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38130.456551] ioatdma acpi_pad [38130.461769] [38130.464647] acpi_power_meter ixgbe [38130.469604] [38130.472568] igb mdio [38130.474667] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38130.478149] crc32c_intel nvme [38130.480251] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38130.482517] ptp nvme_core [38130.487825] do_raw_spin_lock+0xb5/0xc0 [38130.490874] i2c_algo_bit [38130.495744] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38130.498447] pps_core dca [38130.502279] try_get_cap_refs+0x159/0x880 [ceph] [38130.504896] fuse [38130.509684] __ceph_get_caps+0x4a1/0x760 [ceph] [38130.512298] [38130.512299] irq event stamp: 225480634 [38130.516913] ? __file_remove_privs+0x89/0x110 [38130.518831] hardirqs last enabled at (225480633): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38130.523356] ? wake_bit_function+0x70/0x70 [38130.524864] hardirqs last disabled at (225480634): [] sysvec_apic_timer_interrupt+0xe/0x90 [38130.528611] ? ceph_write_iter+0x260/0x770 [ceph] [38130.532958] softirqs last enabled at (225480632): [] __do_softirq+0x2be/0x371 [38130.543203] ceph_write_iter+0x260/0x770 [ceph] [38130.547293] softirqs last disabled at (225480627): [] irq_exit_rcu+0xb0/0xd0 [38130.557110] ? lock_is_held_type+0xaa/0x120 [38130.561802] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38130.570576] vfs_write+0x324/0x4f0 [38130.575096] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38130.583699] ksys_pwrite64+0x65/0xa0 [38130.587873] Workqueue: events delayed_work [ceph] [38130.597248] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38130.600646] [38130.608118] do_syscall_64+0x38/0x80 [38130.611687] RIP: 0010:lock_acquire+0xf0/0x2b0 [38130.616385] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38130.621516] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38130.623007] RIP: 0033:0x7f48f0eeb93b [38130.626576] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38130.630928] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38130.635971] [38130.635972] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38130.654711] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38130.658281] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38130.663497] ORIG_RAX: 0000000000000012 [38130.682236] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38130.683727] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38130.690867] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38130.696085] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38130.703209] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38130.707041] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38130.714165] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38130.721288] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38130.728412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38130.735537] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38130.742661] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38130.749790] [38130.757867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38130.794299] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38130.801424] Call Trace: [38130.803869] [38130.805879] ? watchdog_timer_fn+0x1b4/0x220 [38130.810149] ? __hrtimer_run_queues+0x1d3/0x360 [38130.814677] ? hrtimer_interrupt+0x104/0x240 [38130.818938] ? __local_bh_enable+0x50/0x60 [38130.823031] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38130.828255] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38130.833213] [38130.835309] [38130.837407] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38130.842716] ? lock_acquire+0xf0/0x2b0 [38130.846467] ? igrab+0x43/0x50 [38130.849516] ? lock_release+0xc5/0x260 [38130.853260] _raw_spin_lock+0x30/0x40 [38130.856917] ? igrab+0x1d/0x50 [38130.859966] igrab+0x1d/0x50 [38130.862865] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38130.868012] delayed_work+0x1d8/0x280 [ceph] [38130.872295] ? process_scheduled_works+0x22a/0x510 [38130.877085] process_scheduled_works+0x2a4/0x510 [38130.881700] worker_thread+0x188/0x340 [38130.885450] ? rescuer_thread+0x310/0x310 [38130.889460] kthread+0xf5/0x130 [38130.892597] ? kthread_complete_and_exit+0x20/0x20 [38130.897382] ret_from_fork+0x31/0x40 [38130.900953] ? kthread_complete_and_exit+0x20/0x20 [38130.905737] ret_from_fork_asm+0x11/0x20 [38130.909661] [38158.122919] watchdog: BUG: soft lockup - CPU#0 stuck for 24423s! [dbench:78683] [38158.130222] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38158.134919] watchdog: BUG: soft lockup - CPU#7 stuck for 33420s! [kworker/7:4:78808] [38158.185672] irq event stamp: 67318704 [38158.193404] Modules linked in: ceph libceph dns_resolver fscache netfs [38158.197060] hardirqs last enabled at (67318703): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38158.203594] veth nft_chain_nat xt_MASQUERADE nf_nat [38158.213753] hardirqs last disabled at (67318704): [] sysvec_apic_timer_interrupt+0xe/0x90 [38158.218710] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [38158.228435] softirqs last enabled at (67318678): [] __do_softirq+0x2be/0x371 [38158.236953] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [38158.245638] softirqs last disabled at (67318673): [] irq_exit_rcu+0xb0/0xd0 [38158.252503] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [38158.261016] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38158.267011] irqbypass i2c_i801 iTCO_wdt [38158.275955] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38158.279872] iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev [38158.287344] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38158.287346] crc32_pclmul ipmi_ssif ghash_clmulni_intel [38158.298991] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38158.298992] mei_me mei lpc_ich sha512_ssse3 [38158.322948] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38158.322949] acpi_ipmi [38158.322951] [38158.327211] mfd_core wmi [38158.332427] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38158.334784] ipmi_si [38158.336275] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38158.338894] ipmi_devintf [38158.346016] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38158.348200] ipmi_msghandler ioatdma [38158.355326] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38158.357942] acpi_pad acpi_power_meter [38158.365068] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38158.368638] ixgbe [38158.375762] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38158.379506] igb mdio [38158.386629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38158.388642] crc32c_intel nvme [38158.396717] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38158.398990] ptp nvme_core [38158.404725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38158.407778] i2c_algo_bit pps_core [38158.414900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38158.417607] dca fuse [38158.424729] Call Trace: [38158.428127] [38158.428128] irq event stamp: 225701362 [38158.435252] [38158.437522] hardirqs last enabled at (225701361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38158.439969] ? watchdog_timer_fn+0x1b4/0x220 [38158.441457] hardirqs last disabled at (225701362): [] sysvec_apic_timer_interrupt+0xe/0x90 [38158.445201] ? __hrtimer_run_queues+0x1d3/0x360 [38158.447212] softirqs last enabled at (225701360): [] __do_softirq+0x2be/0x371 [38158.457459] ? hrtimer_interrupt+0x104/0x240 [38158.461718] softirqs last disabled at (225701355): [] irq_exit_rcu+0xb0/0xd0 [38158.471530] ? __local_bh_enable+0x50/0x60 [38158.476055] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38158.484829] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38158.489089] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38158.497688] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38158.501779] Workqueue: events delayed_work [ceph] [38158.511156] [38158.516372] [38158.516373] RIP: 0010:lock_acquire+0xf0/0x2b0 [38158.523844] [38158.523845] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38158.528800] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38158.533503] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38158.535600] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38158.537087] do_raw_spin_lock+0xb5/0xc0 [38158.541437] [38158.541438] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38158.543536] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38158.548839] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38158.567578] try_get_cap_refs+0x159/0x880 [ceph] [38158.572463] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38158.577689] __ceph_get_caps+0x4a1/0x760 [ceph] [38158.581512] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38158.583009] ? __file_remove_privs+0x89/0x110 [38158.590128] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38158.594911] ? wake_bit_function+0x70/0x70 [38158.602035] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38158.606650] ? ceph_write_iter+0x260/0x770 [ceph] [38158.613771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38158.618296] ceph_write_iter+0x260/0x770 [ceph] [38158.625435] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38158.629792] ? lock_is_held_type+0xaa/0x120 [38158.636910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38158.641004] vfs_write+0x324/0x4f0 [38158.649080] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38158.653781] ksys_pwrite64+0x65/0xa0 [38158.659514] Call Trace: [38158.659515] [38158.664038] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38158.671164] ? watchdog_timer_fn+0x1b4/0x220 [38158.675341] do_syscall_64+0x38/0x80 [38158.682466] ? __hrtimer_run_queues+0x1d3/0x360 [38158.685862] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38158.692990] ? hrtimer_interrupt+0x104/0x240 [38158.696555] RIP: 0033:0x7f48f0eeb93b [38158.698999] ? __local_bh_enable+0x50/0x60 [38158.701011] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38158.706145] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38158.710406] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38158.713976] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38158.718500] ORIG_RAX: 0000000000000012 [38158.723544] [38158.727810] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38158.731380] [38158.735471] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38158.754207] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38158.759426] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38158.764648] ? lock_acquire+0xf0/0x2b0 [38158.769599] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38158.773433] ? igrab+0x43/0x50 [38158.775529] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38158.782653] ? lock_release+0xc5/0x260 [38158.784756] [38158.791876] _raw_spin_lock+0x30/0x40 [38158.834949] ? igrab+0x1d/0x50 [38158.838000] igrab+0x1d/0x50 [38158.840879] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38158.846028] delayed_work+0x1d8/0x280 [ceph] [38158.850326] ? process_scheduled_works+0x22a/0x510 [38158.855115] process_scheduled_works+0x2a4/0x510 [38158.859731] worker_thread+0x188/0x340 [38158.863481] ? rescuer_thread+0x310/0x310 [38158.867483] kthread+0xf5/0x130 [38158.870621] ? kthread_complete_and_exit+0x20/0x20 [38158.875407] ret_from_fork+0x31/0x40 [38158.878986] ? kthread_complete_and_exit+0x20/0x20 [38158.883770] ret_from_fork_asm+0x11/0x20 [38158.887701] [38186.123379] watchdog: BUG: soft lockup - CPU#0 stuck for 24449s! [dbench:78683] [38186.130683] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38186.135379] watchdog: BUG: soft lockup - CPU#7 stuck for 33446s! [kworker/7:4:78808] [38186.186124] irq event stamp: 67376790 [38186.193857] Modules linked in: ceph libceph [38186.197513] hardirqs last enabled at (67376789): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38186.201691] dns_resolver fscache netfs [38186.211849] hardirqs last disabled at (67376790): [] sysvec_apic_timer_interrupt+0xe/0x90 [38186.215679] veth nft_chain_nat xt_MASQUERADE [38186.225403] softirqs last enabled at (67376742): [] __do_softirq+0x2be/0x371 [38186.225405] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [38186.238454] softirqs last disabled at (67376737): [] irq_exit_rcu+0xb0/0xd0 [38186.244281] nft_compat nf_tables nfnetlink [38186.252791] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38186.256968] bridge stp [38186.265911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38186.268355] llc xfs libcrc32c [38186.275828] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38186.275830] sunrpc intel_rapl_msr intel_rapl_common [38186.284355] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38186.284355] sb_edac x86_pkg_temp_thermal coretemp [38186.308056] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38186.312842] kvm_intel kvm [38186.318061] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38186.318063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38186.320776] irqbypass i2c_i801 [38186.327898] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38186.335023] iTCO_wdt iTCO_vendor_support [38186.338158] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38186.345282] i2c_smbus crct10dif_pclmul [38186.349287] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38186.356412] joydev crc32_pclmul [38186.360242] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38186.367365] ipmi_ssif ghash_clmulni_intel [38186.370590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38186.378668] mei_me mei [38186.382775] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38186.388514] lpc_ich [38186.390957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38186.398082] sha512_ssse3 [38186.400264] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38186.407390] acpi_ipmi [38186.410006] Call Trace: [38186.410007] [38186.417132] mfd_core wmi [38186.419489] ? watchdog_timer_fn+0x1b4/0x220 [38186.421931] ipmi_si ipmi_devintf [38186.423944] ? __hrtimer_run_queues+0x1d3/0x360 [38186.426560] ipmi_msghandler ioatdma [38186.430827] ? hrtimer_interrupt+0x104/0x240 [38186.434136] acpi_pad acpi_power_meter [38186.438660] ? __local_bh_enable+0x50/0x60 [38186.442231] ixgbe [38186.446496] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38186.450239] igb mdio [38186.454330] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38186.456339] crc32c_intel nvme [38186.461558] [38186.463826] ptp nvme_core [38186.468786] [38186.471835] i2c_algo_bit pps_core [38186.473933] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38186.476637] dca fuse [38186.478739] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38186.482133] [38186.482133] irq event stamp: 225922414 [38186.487439] do_raw_spin_lock+0xb5/0xc0 [38186.489707] hardirqs last enabled at (225922413): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38186.494578] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38186.496067] hardirqs last disabled at (225922414): [] sysvec_apic_timer_interrupt+0xe/0x90 [38186.499816] try_get_cap_refs+0x159/0x880 [ceph] [38186.503643] softirqs last enabled at (225922412): [] __do_softirq+0x2be/0x371 [38186.513893] __ceph_get_caps+0x4a1/0x760 [ceph] [38186.518672] softirqs last disabled at (225922407): [] irq_exit_rcu+0xb0/0xd0 [38186.528488] ? __file_remove_privs+0x89/0x110 [38186.533093] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38186.541865] ? wake_bit_function+0x70/0x70 [38186.546388] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38186.554989] ? ceph_write_iter+0x260/0x770 [ceph] [38186.559337] Workqueue: events delayed_work [ceph] [38186.568715] ceph_write_iter+0x260/0x770 [ceph] [38186.572805] [38186.572806] RIP: 0010:lock_acquire+0xf0/0x2b0 [38186.580280] ? lock_is_held_type+0xaa/0x120 [38186.584972] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38186.589672] vfs_write+0x324/0x4f0 [38186.594194] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38186.595689] ksys_pwrite64+0x65/0xa0 [38186.600035] [38186.604213] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38186.622950] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38186.626349] do_syscall_64+0x38/0x80 [38186.631565] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38186.631566] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38186.635138] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38186.636627] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38186.641759] RIP: 0033:0x7f48f0eeb93b [38186.648898] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38186.652471] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38186.659595] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38186.666719] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38186.671778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38186.678905] ORIG_RAX: 0000000000000012 [38186.682473] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38186.689599] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38186.708337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38186.716413] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38186.721632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38186.727369] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38186.731199] Call Trace: [38186.731200] [38186.738322] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38186.745448] ? watchdog_timer_fn+0x1b4/0x220 [38186.752571] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38186.759697] ? __hrtimer_run_queues+0x1d3/0x360 [38186.766825] [38186.773947] ? hrtimer_interrupt+0x104/0x240 [38186.807884] ? __local_bh_enable+0x50/0x60 [38186.811977] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38186.817202] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38186.822158] [38186.824255] [38186.826354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38186.831662] ? lock_acquire+0xf0/0x2b0 [38186.835411] ? igrab+0x43/0x50 [38186.838462] ? lock_release+0xc5/0x260 [38186.842205] _raw_spin_lock+0x30/0x40 [38186.845862] ? igrab+0x1d/0x50 [38186.848912] igrab+0x1d/0x50 [38186.851793] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38186.856940] delayed_work+0x1d8/0x280 [ceph] [38186.861223] ? process_scheduled_works+0x22a/0x510 [38186.866012] process_scheduled_works+0x2a4/0x510 [38186.870630] worker_thread+0x188/0x340 [38186.874376] ? rescuer_thread+0x310/0x310 [38186.878381] kthread+0xf5/0x130 [38186.881519] ? kthread_complete_and_exit+0x20/0x20 [38186.886303] ret_from_fork+0x31/0x40 [38186.889880] ? kthread_complete_and_exit+0x20/0x20 [38186.894664] ret_from_fork_asm+0x11/0x20 [38186.898588] [38187.621769] rcu: INFO: rcu_preempt self-detected stall on CPU [38187.627534] rcu: 0-....: (25656921 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=12849265 [38187.638045] rcu: (t=26250552 jiffies g=1309609 q=713312 ncpus=8) [38187.644131] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38187.653073] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38187.660544] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38187.666020] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38187.684757] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38187.689975] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38187.697101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38187.704224] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38187.711347] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38187.718474] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38187.725598] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38187.733675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38187.739412] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38187.746536] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38187.753659] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38187.760783] Call Trace: [38187.763229] [38187.765239] ? rcu_dump_cpu_stacks+0xd1/0x180 [38187.769600] ? rcu_sched_clock_irq+0x633/0x1390 [38187.774131] ? lock_is_held_type+0xaa/0x120 [38187.778309] ? lock_is_held_type+0xaa/0x120 [38187.782487] ? tick_sched_do_timer+0xa0/0xa0 [38187.786758] ? rcu_read_lock_sched_held+0x56/0x90 [38187.791456] ? tick_sched_do_timer+0xa0/0xa0 [38187.795719] ? update_process_times+0x5f/0x90 [38187.800070] ? tick_sched_handle.isra.16+0x1e/0x50 [38187.804854] ? tick_sched_timer+0x6f/0x80 [38187.808861] ? __hrtimer_run_queues+0x1d3/0x360 [38187.813394] ? hrtimer_interrupt+0x104/0x240 [38187.817656] ? __local_bh_enable+0x50/0x60 [38187.821749] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38187.826973] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38187.831930] [38187.834027] [38187.836125] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38187.841433] ? queued_spin_lock_slowpath+0x18/0x2e0 [38187.846311] do_raw_spin_lock+0xb5/0xc0 [38187.850148] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38187.854950] try_get_cap_refs+0x159/0x880 [ceph] [38187.859582] __ceph_get_caps+0x4a1/0x760 [ceph] [38187.864123] ? __file_remove_privs+0x89/0x110 [38187.868478] ? wake_bit_function+0x70/0x70 [38187.872573] ? ceph_write_iter+0x260/0x770 [ceph] [38187.877283] ceph_write_iter+0x260/0x770 [ceph] [38187.881820] ? lock_is_held_type+0xaa/0x120 [38187.886004] vfs_write+0x324/0x4f0 [38187.889404] ksys_pwrite64+0x65/0xa0 [38187.892979] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38187.898110] do_syscall_64+0x38/0x80 [38187.901681] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38187.906725] RIP: 0033:0x7f48f0eeb93b [38187.910294] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38187.929034] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [38187.936591] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38187.943714] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38187.950838] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38187.957962] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38187.965088] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38187.972216] [38187.974405] Sending NMI from CPU 0 to CPUs 7: [38187.978757] NMI backtrace for cpu 7 [38187.978758] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38187.978760] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38187.978760] Workqueue: events delayed_work [ceph] [38187.978776] RIP: 0010:do_raw_spin_lock+0x5c/0xc0 [38187.978778] Code: 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 75 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 89 43 10 5b e8 2e 22 45 00 85 c0 74 b1 48 c7 c6 29 e2 12 82 48 89 df e8 ab [38187.978780] RSP: 0018:ffffc90002117d98 EFLAGS: 00000246 [38187.978781] RAX: ffff88810debaa80 RBX: ffff8881a1d34478 RCX: 000000005553a634 [38187.978782] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [38187.978783] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [38187.978784] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [38187.978784] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38187.978785] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38187.978786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38187.978787] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38187.978788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38187.978789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38187.978789] Call Trace: [38187.978790] [38187.978791] ? nmi_cpu_backtrace+0xad/0x120 [38187.978794] ? nmi_cpu_backtrace_handler+0x11/0x20 [38187.978796] ? nmi_handle+0xd4/0x260 [38187.978800] ? default_do_nmi+0x49/0xf0 [38187.978802] ? exc_nmi+0x11e/0x150 [38187.978804] ? end_repeat_nmi+0x16/0x67 [38187.978808] ? do_raw_spin_lock+0x5c/0xc0 [38187.978810] ? do_raw_spin_lock+0x5c/0xc0 [38187.978812] ? do_raw_spin_lock+0x5c/0xc0 [38187.978814] [38187.978814] [38187.978814] igrab+0x1d/0x50 [38187.978816] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38187.978830] delayed_work+0x1d8/0x280 [ceph] [38187.978844] ? process_scheduled_works+0x22a/0x510 [38187.978846] process_scheduled_works+0x2a4/0x510 [38187.978850] worker_thread+0x188/0x340 [38187.978853] ? rescuer_thread+0x310/0x310 [38187.978855] kthread+0xf5/0x130 [38187.978857] ? kthread_complete_and_exit+0x20/0x20 [38187.978859] ret_from_fork+0x31/0x40 [38187.978862] ? kthread_complete_and_exit+0x20/0x20 [38187.978864] ret_from_fork_asm+0x11/0x20 [38187.978868] [38214.123839] watchdog: BUG: soft lockup - CPU#0 stuck for 24475s! [dbench:78683] [38214.131148] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38214.135839] watchdog: BUG: soft lockup - CPU#7 stuck for 33472s! [kworker/7:4:78808] [38214.186597] irq event stamp: 67434228 [38214.194327] Modules linked in: ceph libceph dns_resolver fscache netfs [38214.197984] hardirqs last enabled at (67434227): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38214.197989] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [38214.214667] hardirqs last disabled at (67434228): [] sysvec_apic_timer_interrupt+0xe/0x90 [38214.214670] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs [38214.231777] softirqs last enabled at (67434180): [] __do_softirq+0x2be/0x371 [38214.238910] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [38214.247593] softirqs last disabled at (67434175): [] irq_exit_rcu+0xb0/0xd0 [38214.256720] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [38214.265234] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38214.270368] iTCO_vendor_support i2c_smbus crct10dif_pclmul [38214.279313] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38214.279314] joydev [38214.279315] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38214.284878] crc32_pclmul ipmi_ssif [38214.292349] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38214.294445] ghash_clmulni_intel [38214.299922] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38214.303407] mei_me mei [38214.322145] [38214.325368] lpc_ich sha512_ssse3 [38214.330586] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38214.333030] acpi_ipmi mfd_core [38214.334521] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38214.337831] wmi ipmi_si [38214.344956] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38214.348094] ipmi_devintf [38214.355234] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38214.357764] ipmi_msghandler [38214.364888] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38214.367516] ioatdma acpi_pad [38214.374639] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38214.377518] acpi_power_meter ixgbe [38214.384640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38214.387604] igb mdio [38214.395683] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38214.399168] crc32c_intel nvme [38214.404905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38214.407175] ptp [38214.414299] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38214.417350] nvme_core i2c_algo_bit [38214.424474] Call Trace: [38214.424477] [38214.426310] pps_core dca [38214.433437] ? watchdog_timer_fn+0x1b4/0x220 [38214.436921] fuse [38214.439364] ? __hrtimer_run_queues+0x1d3/0x360 [38214.441373] [38214.441374] irq event stamp: 226141422 [38214.443994] ? hrtimer_interrupt+0x104/0x240 [38214.448255] hardirqs last enabled at (226141421): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38214.450179] ? __local_bh_enable+0x50/0x60 [38214.454704] hardirqs last disabled at (226141422): [] sysvec_apic_timer_interrupt+0xe/0x90 [38214.456197] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38214.459937] softirqs last enabled at (226141420): [] __do_softirq+0x2be/0x371 [38214.464203] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38214.474446] softirqs last disabled at (226141415): [] irq_exit_rcu+0xb0/0xd0 [38214.478538] [38214.488348] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38214.493566] [38214.493568] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38214.502337] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38214.507297] ? queued_spin_lock_slowpath+0x16/0x2e0 [38214.515891] Workqueue: events delayed_work [ceph] [38214.517992] do_raw_spin_lock+0xb5/0xc0 [38214.527365] [38214.527366] RIP: 0010:lock_acquire+0xf0/0x2b0 [38214.529465] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38214.534767] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38214.542243] try_get_cap_refs+0x159/0x880 [ceph] [38214.547108] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38214.551814] __ceph_get_caps+0x4a1/0x760 [ceph] [38214.555638] [38214.555639] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38214.557134] ? __file_remove_privs+0x89/0x110 [38214.561478] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38214.566265] ? wake_bit_function+0x70/0x70 [38214.585002] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38214.589613] ? ceph_write_iter+0x260/0x770 [ceph] [38214.594828] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38214.599353] ceph_write_iter+0x260/0x770 [ceph] [38214.600843] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38214.607974] ? lock_is_held_type+0xaa/0x120 [38214.612318] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38214.619445] vfs_write+0x324/0x4f0 [38214.623534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38214.630661] ksys_pwrite64+0x65/0xa0 [38214.635355] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38214.642480] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38214.647002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38214.654129] do_syscall_64+0x38/0x80 [38214.658304] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38214.666385] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38214.669780] Call Trace: [38214.669782] [38214.675519] RIP: 0033:0x7f48f0eeb93b [38214.679088] ? watchdog_timer_fn+0x1b4/0x220 [38214.686214] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38214.691344] ? __hrtimer_run_queues+0x1d3/0x360 [38214.698468] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38214.702041] ? hrtimer_interrupt+0x104/0x240 [38214.709161] ORIG_RAX: 0000000000000012 [38214.714205] ? __local_bh_enable+0x50/0x60 [38214.716649] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38214.718662] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38214.722240] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38214.726505] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38214.745242] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38214.749766] [38214.754984] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38214.759248] [38214.763078] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38214.767170] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38214.774299] [38214.779530] ? lock_acquire+0xf0/0x2b0 [38214.828411] ? igrab+0x43/0x50 [38214.831469] ? lock_release+0xc5/0x260 [38214.835213] _raw_spin_lock+0x30/0x40 [38214.838871] ? igrab+0x1d/0x50 [38214.841920] igrab+0x1d/0x50 [38214.844800] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38214.849951] delayed_work+0x1d8/0x280 [ceph] [38214.854248] ? process_scheduled_works+0x22a/0x510 [38214.859038] process_scheduled_works+0x2a4/0x510 [38214.863655] worker_thread+0x188/0x340 [38214.867404] ? rescuer_thread+0x310/0x310 [38214.871414] kthread+0xf5/0x130 [38214.874561] ? kthread_complete_and_exit+0x20/0x20 [38214.879345] ret_from_fork+0x31/0x40 [38214.882924] ? kthread_complete_and_exit+0x20/0x20 [38214.887707] ret_from_fork_asm+0x11/0x20 [38214.891631] [38242.124296] watchdog: BUG: soft lockup - CPU#0 stuck for 24501s! [dbench:78683] [38242.131600] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38242.136296] watchdog: BUG: soft lockup - CPU#7 stuck for 33498s! [kworker/7:4:78808] [38242.187041] irq event stamp: 67492348 [38242.187042] hardirqs last enabled at (67492347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38242.194771] Modules linked in: [38242.198427] hardirqs last disabled at (67492348): [] sysvec_apic_timer_interrupt+0xe/0x90 [38242.208585] ceph [38242.211636] softirqs last enabled at (67492300): [] __do_softirq+0x2be/0x371 [38242.221361] libceph [38242.223284] softirqs last disabled at (67492295): [] irq_exit_rcu+0xb0/0xd0 [38242.231971] dns_resolver fscache [38242.234152] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38242.242663] netfs veth [38242.245991] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38242.254937] nft_chain_nat [38242.257380] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38242.264851] xt_MASQUERADE nf_nat [38242.267555] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38242.273032] nf_conntrack nf_defrag_ipv6 [38242.276343] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38242.295081] nf_defrag_ipv4 nft_compat [38242.298997] [38242.304216] nf_tables nfnetlink [38242.307974] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38242.309468] bridge stp [38242.312694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38242.319816] llc xfs [38242.322259] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38242.329385] libcrc32c sunrpc [38242.331568] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38242.338693] intel_rapl_msr [38242.341655] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38242.348780] intel_rapl_common sb_edac [38242.351572] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38242.358695] x86_pkg_temp_thermal [38242.362440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38242.370517] coretemp kvm_intel [38242.373826] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38242.379564] kvm irqbypass [38242.382701] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38242.389826] i2c_i801 iTCO_wdt [38242.392530] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38242.399655] iTCO_vendor_support i2c_smbus [38242.402705] Call Trace: [38242.402706] [38242.409830] crct10dif_pclmul joydev [38242.413922] ? watchdog_timer_fn+0x1b4/0x220 [38242.416365] crc32_pclmul ipmi_ssif [38242.418377] ? __hrtimer_run_queues+0x1d3/0x360 [38242.421946] ghash_clmulni_intel mei_me [38242.426213] ? hrtimer_interrupt+0x104/0x240 [38242.429694] mei lpc_ich [38242.434218] ? __local_bh_enable+0x50/0x60 [38242.438048] sha512_ssse3 acpi_ipmi [38242.442315] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38242.444843] mfd_core wmi [38242.448934] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38242.452418] ipmi_si ipmi_devintf [38242.457636] [38242.460252] ipmi_msghandler ioatdma [38242.465210] [38242.465212] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38242.468520] acpi_pad acpi_power_meter [38242.470621] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38242.474189] ixgbe igb [38242.476287] do_raw_spin_lock+0xb5/0xc0 [38242.481590] mdio crc32c_intel [38242.485334] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38242.490206] nvme ptp [38242.492564] try_get_cap_refs+0x159/0x880 [ceph] [38242.496393] nvme_core i2c_algo_bit [38242.499448] __ceph_get_caps+0x4a1/0x760 [ceph] [38242.504228] pps_core [38242.506503] ? __file_remove_privs+0x89/0x110 [38242.511111] dca fuse [38242.514595] ? wake_bit_function+0x70/0x70 [38242.519119] [38242.519120] irq event stamp: 226362354 [38242.521390] ? ceph_write_iter+0x260/0x770 [ceph] [38242.525740] hardirqs last enabled at (226362353): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38242.528011] ceph_write_iter+0x260/0x770 [ceph] [38242.532100] hardirqs last disabled at (226362354): [] sysvec_apic_timer_interrupt+0xe/0x90 [38242.533594] ? lock_is_held_type+0xaa/0x120 [38242.537336] softirqs last enabled at (226362352): [] __do_softirq+0x2be/0x371 [38242.542034] vfs_write+0x324/0x4f0 [38242.552276] softirqs last disabled at (226362347): [] irq_exit_rcu+0xb0/0xd0 [38242.556806] ksys_pwrite64+0x65/0xa0 [38242.566611] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38242.570789] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38242.579559] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38242.582976] do_syscall_64+0x38/0x80 [38242.591572] Workqueue: events delayed_work [ceph] [38242.595143] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38242.604520] [38242.604521] RIP: 0010:lock_acquire+0xf0/0x2b0 [38242.609651] RIP: 0033:0x7f48f0eeb93b [38242.617121] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38242.620696] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38242.625390] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38242.630434] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38242.631926] [38242.631927] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38242.636277] ORIG_RAX: 0000000000000012 [38242.639845] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38242.658583] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38242.677323] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38242.682541] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38242.687758] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38242.689249] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38242.696372] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38242.700202] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38242.707326] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38242.714452] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38242.721576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38242.728704] [38242.735823] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38242.787445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38242.794566] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38242.801700] Call Trace: [38242.804146] [38242.806156] ? watchdog_timer_fn+0x1b4/0x220 [38242.810421] ? __hrtimer_run_queues+0x1d3/0x360 [38242.814955] ? hrtimer_interrupt+0x104/0x240 [38242.819233] ? __local_bh_enable+0x50/0x60 [38242.823328] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38242.828550] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38242.833508] [38242.835604] [38242.837701] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38242.843010] ? lock_acquire+0xf0/0x2b0 [38242.846761] ? igrab+0x43/0x50 [38242.849811] ? lock_release+0xc5/0x260 [38242.853556] _raw_spin_lock+0x30/0x40 [38242.857210] ? igrab+0x1d/0x50 [38242.860262] igrab+0x1d/0x50 [38242.863141] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38242.868291] delayed_work+0x1d8/0x280 [ceph] [38242.872571] ? process_scheduled_works+0x22a/0x510 [38242.877361] process_scheduled_works+0x2a4/0x510 [38242.881989] worker_thread+0x188/0x340 [38242.885734] ? rescuer_thread+0x310/0x310 [38242.889737] kthread+0xf5/0x130 [38242.892876] ? kthread_complete_and_exit+0x20/0x20 [38242.897660] ret_from_fork+0x31/0x40 [38242.901232] ? kthread_complete_and_exit+0x20/0x20 [38242.906014] ret_from_fork_asm+0x11/0x20 [38242.909938] [38270.124752] watchdog: BUG: soft lockup - CPU#0 stuck for 24527s! [dbench:78683] [38270.132054] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38270.136752] watchdog: BUG: soft lockup - CPU#7 stuck for 33524s! [kworker/7:4:78808] [38270.187502] irq event stamp: 67550538 [38270.195233] Modules linked in: ceph libceph dns_resolver fscache netfs [38270.198890] hardirqs last enabled at (67550537): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38270.205425] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38270.215583] hardirqs last disabled at (67550538): [] sysvec_apic_timer_interrupt+0xe/0x90 [38270.221666] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [38270.231390] softirqs last enabled at (67550490): [] __do_softirq+0x2be/0x371 [38270.231394] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [38270.247466] softirqs last disabled at (67550485): [] irq_exit_rcu+0xb0/0xd0 [38270.254330] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [38270.262844] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38270.268841] irqbypass i2c_i801 iTCO_wdt [38270.277785] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38270.277786] iTCO_vendor_support i2c_smbus [38270.281701] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38270.289171] crct10dif_pclmul joydev [38270.293264] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38270.298740] crc32_pclmul ipmi_ssif [38270.302311] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38270.321048] ghash_clmulni_intel mei_me [38270.324532] [38270.329749] mei lpc_ich [38270.333580] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38270.335071] sha512_ssse3 acpi_ipmi [38270.337603] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38270.344725] mfd_core wmi [38270.348209] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38270.355333] ipmi_si [38270.357951] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38270.365078] ipmi_devintf [38270.367260] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38270.374383] ipmi_msghandler ioatdma [38270.377001] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38270.384125] acpi_pad [38270.387714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38270.395790] acpi_power_meter ixgbe [38270.398063] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38270.403799] igb mdio [38270.407284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38270.414408] crc32c_intel nvme ptp [38270.416696] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38270.423820] nvme_core [38270.427216] Call Trace: [38270.427218] [38270.434340] i2c_algo_bit pps_core [38270.436701] ? watchdog_timer_fn+0x1b4/0x220 [38270.439148] dca fuse [38270.441163] ? __hrtimer_run_queues+0x1d3/0x360 [38270.444558] [38270.444559] irq event stamp: 226583076 [38270.448828] ? hrtimer_interrupt+0x104/0x240 [38270.451095] hardirqs last enabled at (226583075): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38270.455618] ? __local_bh_enable+0x50/0x60 [38270.457108] hardirqs last disabled at (226583076): [] sysvec_apic_timer_interrupt+0xe/0x90 [38270.460856] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38270.465117] softirqs last enabled at (226583074): [] __do_softirq+0x2be/0x371 [38270.475361] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38270.479451] softirqs last disabled at (226583069): [] irq_exit_rcu+0xb0/0xd0 [38270.489264] [38270.494480] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38270.503250] [38270.503252] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38270.508209] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38270.516811] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38270.518904] Workqueue: events delayed_work [ceph] [38270.528283] do_raw_spin_lock+0xb5/0xc0 [38270.530378] [38270.530379] RIP: 0010:lock_acquire+0xf0/0x2b0 [38270.535684] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38270.543154] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38270.548026] try_get_cap_refs+0x159/0x880 [ceph] [38270.552720] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38270.556557] __ceph_get_caps+0x4a1/0x760 [ceph] [38270.558042] [38270.558043] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38270.562399] ? __file_remove_privs+0x89/0x110 [38270.567178] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38270.585916] ? wake_bit_function+0x70/0x70 [38270.590525] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38270.595745] ? ceph_write_iter+0x260/0x770 [ceph] [38270.600266] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38270.601758] ceph_write_iter+0x260/0x770 [ceph] [38270.608883] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38270.613238] ? lock_is_held_type+0xaa/0x120 [38270.620357] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38270.624450] vfs_write+0x324/0x4f0 [38270.631571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38270.636275] ksys_pwrite64+0x65/0xa0 [38270.643394] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38270.647918] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38270.655043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38270.659222] do_syscall_64+0x38/0x80 [38270.667298] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38270.670694] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38270.676446] Call Trace: [38270.676448] [38270.680019] RIP: 0033:0x7f48f0eeb93b [38270.687157] ? watchdog_timer_fn+0x1b4/0x220 [38270.692285] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38270.699410] ? __hrtimer_run_queues+0x1d3/0x360 [38270.702977] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38270.710107] ? hrtimer_interrupt+0x104/0x240 [38270.715154] ORIG_RAX: 0000000000000012 [38270.717598] ? __local_bh_enable+0x50/0x60 [38270.719609] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38270.723183] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38270.727445] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38270.746182] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38270.750706] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38270.755923] [38270.760187] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38270.764017] [38270.768108] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38270.775233] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38270.780456] [38270.787581] ? lock_acquire+0xf0/0x2b0 [38270.829335] ? igrab+0x43/0x50 [38270.832393] ? lock_release+0xc5/0x260 [38270.836157] _raw_spin_lock+0x30/0x40 [38270.839821] ? igrab+0x1d/0x50 [38270.842870] igrab+0x1d/0x50 [38270.845748] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38270.850898] delayed_work+0x1d8/0x280 [ceph] [38270.855180] ? process_scheduled_works+0x22a/0x510 [38270.859969] process_scheduled_works+0x2a4/0x510 [38270.864586] worker_thread+0x188/0x340 [38270.868336] ? rescuer_thread+0x310/0x310 [38270.872346] kthread+0xf5/0x130 [38270.875492] ? kthread_complete_and_exit+0x20/0x20 [38270.880279] ret_from_fork+0x31/0x40 [38270.883856] ? kthread_complete_and_exit+0x20/0x20 [38270.888639] ret_from_fork_asm+0x11/0x20 [38270.892563] [38298.125207] watchdog: BUG: soft lockup - CPU#0 stuck for 24553s! [dbench:78683] [38298.132505] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38298.137206] watchdog: BUG: soft lockup - CPU#7 stuck for 33550s! [kworker/7:4:78808] [38298.187945] irq event stamp: 67608614 [38298.187947] hardirqs last enabled at (67608613): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38298.195677] Modules linked in: [38298.199334] hardirqs last disabled at (67608614): [] sysvec_apic_timer_interrupt+0xe/0x90 [38298.209491] ceph [38298.212541] softirqs last enabled at (67608566): [] __do_softirq+0x2be/0x371 [38298.222266] libceph [38298.224191] softirqs last disabled at (67608561): [] irq_exit_rcu+0xb0/0xd0 [38298.232879] dns_resolver [38298.235067] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38298.243579] fscache netfs [38298.246213] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38298.255158] veth nft_chain_nat [38298.257861] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38298.265349] xt_MASQUERADE nf_nat [38298.268486] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38298.273962] nf_conntrack nf_defrag_ipv6 [38298.277274] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38298.296011] nf_defrag_ipv4 nft_compat [38298.299929] [38298.299930] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38298.305147] nf_tables [38298.308891] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38298.310383] nfnetlink bridge [38298.317507] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38298.319879] stp [38298.327003] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38298.329970] llc xfs [38298.337093] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38298.338931] libcrc32c sunrpc [38298.346054] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38298.348240] intel_rapl_msr [38298.355363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38298.358327] intel_rapl_common [38298.366405] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38298.369194] sb_edac x86_pkg_temp_thermal [38298.374931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38298.377983] coretemp [38298.385106] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38298.389110] kvm_intel kvm [38298.396236] Call Trace: [38298.396237] [38298.398507] irqbypass i2c_i801 [38298.405632] ? watchdog_timer_fn+0x1b4/0x220 [38298.408335] iTCO_wdt [38298.410779] ? __hrtimer_run_queues+0x1d3/0x360 [38298.412790] iTCO_vendor_support i2c_smbus [38298.415929] ? hrtimer_interrupt+0x104/0x240 [38298.420190] crct10dif_pclmul joydev [38298.422461] ? __local_bh_enable+0x50/0x60 [38298.426986] crc32_pclmul ipmi_ssif [38298.431078] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38298.435339] ghash_clmulni_intel [38298.438912] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38298.443002] mei_me mei [38298.446487] [38298.451703] lpc_ich sha512_ssse3 [38298.454926] [38298.454927] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38298.459886] acpi_ipmi [38298.462333] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38298.464425] mfd_core [38298.467739] do_raw_spin_lock+0xb5/0xc0 [38298.469833] wmi [38298.475138] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38298.477496] ipmi_si ipmi_devintf [38298.482368] try_get_cap_refs+0x159/0x880 [ceph] [38298.484637] ipmi_msghandler ioatdma [38298.488471] __ceph_get_caps+0x4a1/0x760 [ceph] [38298.490306] acpi_pad acpi_power_meter [38298.495092] ? __file_remove_privs+0x89/0x110 [38298.498399] ixgbe igb [38298.503010] ? wake_bit_function+0x70/0x70 [38298.506580] mdio crc32c_intel [38298.511125] ? ceph_write_iter+0x260/0x770 [ceph] [38298.514883] nvme ptp nvme_core [38298.519235] ceph_write_iter+0x260/0x770 [ceph] [38298.521604] i2c_algo_bit [38298.525695] ? lock_is_held_type+0xaa/0x120 [38298.528742] pps_core dca [38298.533443] vfs_write+0x324/0x4f0 [38298.536577] fuse [38298.541104] ksys_pwrite64+0x65/0xa0 [38298.543718] irq event stamp: 226804000 [38298.547897] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38298.550512] hardirqs last enabled at (226803999): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38298.553913] do_syscall_64+0x38/0x80 [38298.555834] hardirqs last disabled at (226804000): [] sysvec_apic_timer_interrupt+0xe/0x90 [38298.559405] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38298.563149] softirqs last enabled at (226803998): [] __do_softirq+0x2be/0x371 [38298.568280] RIP: 0033:0x7f48f0eeb93b [38298.578525] softirqs last disabled at (226803993): [] irq_exit_rcu+0xb0/0xd0 [38298.582095] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38298.591908] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38298.596950] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38298.605720] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38298.609291] ORIG_RAX: 0000000000000012 [38298.617897] Workqueue: events delayed_work [ceph] [38298.636636] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38298.646013] [38298.651230] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38298.658701] RIP: 0010:lock_acquire+0x0/0x2b0 [38298.662533] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38298.667231] Code: 0f c1 05 63 da ea 7e 83 f8 01 75 0e 48 85 db 74 06 fb 0f 1f 44 00 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd [38298.674355] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38298.675845] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [38298.682969] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38298.687233] [38298.694362] [38298.713093] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [38298.743359] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [38298.750483] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [38298.757609] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [38298.764730] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38298.771856] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38298.779934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38298.785671] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38298.792796] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38298.799918] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38298.807044] Call Trace: [38298.809489] [38298.811500] ? watchdog_timer_fn+0x1b4/0x220 [38298.815763] ? __hrtimer_run_queues+0x1d3/0x360 [38298.820289] ? hrtimer_interrupt+0x104/0x240 [38298.824560] ? __local_bh_enable+0x50/0x60 [38298.828652] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38298.833890] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38298.838850] [38298.840948] [38298.843046] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38298.848355] ? lock_sync+0xa0/0xa0 [38298.851756] _raw_spin_lock+0x30/0x40 [38298.855414] ? igrab+0x1d/0x50 [38298.858464] igrab+0x1d/0x50 [38298.861344] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38298.866492] delayed_work+0x1d8/0x280 [ceph] [38298.870772] ? process_scheduled_works+0x22a/0x510 [38298.875564] process_scheduled_works+0x2a4/0x510 [38298.880178] worker_thread+0x188/0x340 [38298.883928] ? rescuer_thread+0x310/0x310 [38298.887932] kthread+0xf5/0x130 [38298.891070] ? kthread_complete_and_exit+0x20/0x20 [38298.895854] ret_from_fork+0x31/0x40 [38298.899433] ? kthread_complete_and_exit+0x20/0x20 [38298.904215] ret_from_fork_asm+0x11/0x20 [38298.908139] [38326.125659] watchdog: BUG: soft lockup - CPU#0 stuck for 24579s! [dbench:78683] [38326.132967] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38326.137659] watchdog: BUG: soft lockup - CPU#7 stuck for 33576s! [kworker/7:4:78808] [38326.188417] irq event stamp: 67666878 [38326.196148] Modules linked in: ceph libceph dns_resolver fscache netfs [38326.199804] hardirqs last enabled at (67666877): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38326.206333] veth nft_chain_nat xt_MASQUERADE nf_nat [38326.216488] hardirqs last disabled at (67666878): [] sysvec_apic_timer_interrupt+0xe/0x90 [38326.221446] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [38326.231170] softirqs last enabled at (67666830): [] __do_softirq+0x2be/0x371 [38326.231174] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [38326.247765] softirqs last disabled at (67666825): [] irq_exit_rcu+0xb0/0xd0 [38326.247769] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [38326.262188] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38326.269397] kvm irqbypass [38326.278344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38326.278345] i2c_i801 iTCO_wdt [38326.281054] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38326.288526] iTCO_vendor_support [38326.291579] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38326.297054] i2c_smbus [38326.300279] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38326.319016] crct10dif_pclmul [38326.321375] [38326.326592] joydev crc32_pclmul [38326.329555] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38326.331056] ipmi_ssif ghash_clmulni_intel [38326.334279] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38326.341403] mei_me mei [38326.345492] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38326.352617] lpc_ich [38326.355062] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38326.362185] sha512_ssse3 [38326.364370] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38326.371493] acpi_ipmi mfd_core [38326.374111] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38326.381235] wmi ipmi_si [38326.384374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38326.392450] ipmi_devintf [38326.394980] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38326.400718] ipmi_msghandler [38326.403335] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38326.410460] ioatdma [38326.413347] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38326.420470] acpi_pad [38326.422654] Call Trace: [38326.422656] [38326.429778] acpi_power_meter [38326.432059] ? watchdog_timer_fn+0x1b4/0x220 [38326.434503] ixgbe igb [38326.436514] ? __hrtimer_run_queues+0x1d3/0x360 [38326.439477] mdio crc32c_intel [38326.443744] ? hrtimer_interrupt+0x104/0x240 [38326.446097] nvme ptp [38326.450623] ? __local_bh_enable+0x50/0x60 [38326.453672] nvme_core i2c_algo_bit [38326.457940] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38326.460206] pps_core [38326.464300] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38326.467781] dca [38326.473000] [38326.475272] fuse [38326.480229] [38326.482064] [38326.484162] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38326.486087] irq event stamp: 227024810 [38326.488187] ? queued_spin_lock_slowpath+0x18/0x2e0 [38326.489674] hardirqs last enabled at (227024809): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38326.494979] do_raw_spin_lock+0xb5/0xc0 [38326.498722] hardirqs last disabled at (227024810): [] sysvec_apic_timer_interrupt+0xe/0x90 [38326.503595] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38326.513837] softirqs last enabled at (227024808): [] __do_softirq+0x2be/0x371 [38326.517671] try_get_cap_refs+0x159/0x880 [ceph] [38326.527479] softirqs last disabled at (227024803): [] irq_exit_rcu+0xb0/0xd0 [38326.532268] __ceph_get_caps+0x4a1/0x760 [ceph] [38326.541035] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38326.545667] ? __file_remove_privs+0x89/0x110 [38326.554260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38326.558784] ? wake_bit_function+0x70/0x70 [38326.568161] Workqueue: events delayed_work [ceph] [38326.572514] ? ceph_write_iter+0x260/0x770 [ceph] [38326.579983] [38326.579984] RIP: 0010:lock_acquire+0xf0/0x2b0 [38326.584075] ceph_write_iter+0x260/0x770 [ceph] [38326.588770] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38326.593473] ? lock_is_held_type+0xaa/0x120 [38326.594959] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38326.599314] vfs_write+0x324/0x4f0 [38326.603833] [38326.603833] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38326.622575] ksys_pwrite64+0x65/0xa0 [38326.626749] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38326.631967] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38326.635365] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38326.635366] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38326.636856] do_syscall_64+0x38/0x80 [38326.643977] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38326.647550] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38326.654673] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38326.659806] RIP: 0033:0x7f48f0eeb93b [38326.666927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38326.674064] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38326.677633] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38326.684755] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38326.689800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38326.697878] ORIG_RAX: 0000000000000012 [38326.701448] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38326.707185] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38326.725923] Call Trace: [38326.733062] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38326.738283] [38326.745407] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38326.749239] ? watchdog_timer_fn+0x1b4/0x220 [38326.756361] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38326.763486] ? __hrtimer_run_queues+0x1d3/0x360 [38326.765930] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38326.773066] ? hrtimer_interrupt+0x104/0x240 [38326.775078] [38326.782197] ? __local_bh_enable+0x50/0x60 [38326.815776] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38326.820999] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38326.825957] [38326.828063] [38326.830160] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38326.835468] ? lock_acquire+0xf0/0x2b0 [38326.839219] ? igrab+0x43/0x50 [38326.842276] ? lock_release+0xc5/0x260 [38326.846024] _raw_spin_lock+0x30/0x40 [38326.849688] ? igrab+0x1d/0x50 [38326.852737] igrab+0x1d/0x50 [38326.855617] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38326.860765] delayed_work+0x1d8/0x280 [ceph] [38326.865065] ? process_scheduled_works+0x22a/0x510 [38326.869854] process_scheduled_works+0x2a4/0x510 [38326.874471] worker_thread+0x188/0x340 [38326.878220] ? rescuer_thread+0x310/0x310 [38326.882229] kthread+0xf5/0x130 [38326.885368] ? kthread_complete_and_exit+0x20/0x20 [38326.890152] ret_from_fork+0x31/0x40 [38326.893722] ? kthread_complete_and_exit+0x20/0x20 [38326.898507] ret_from_fork_asm+0x11/0x20 [38326.902438] [38354.126112] watchdog: BUG: soft lockup - CPU#0 stuck for 24606s! [dbench:78683] [38354.133411] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38354.138112] watchdog: BUG: soft lockup - CPU#7 stuck for 33603s! [kworker/7:4:78808] [38354.188852] irq event stamp: 67725018 [38354.188853] hardirqs last enabled at (67725017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38354.196582] Modules linked in: [38354.200239] hardirqs last disabled at (67725018): [] sysvec_apic_timer_interrupt+0xe/0x90 [38354.210397] ceph [38354.213448] softirqs last enabled at (67724970): [] __do_softirq+0x2be/0x371 [38354.223173] libceph [38354.225095] softirqs last disabled at (67724965): [] irq_exit_rcu+0xb0/0xd0 [38354.233785] dns_resolver fscache [38354.235973] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38354.244501] netfs [38354.247810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38354.247812] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38354.256755] veth nft_chain_nat [38354.258766] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38354.266254] xt_MASQUERADE nf_nat [38354.271732] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38354.274869] nf_conntrack nf_defrag_ipv6 [38354.293606] [38354.293607] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38354.296918] nf_defrag_ipv4 [38354.302136] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38354.306053] nft_compat [38354.307544] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38354.314668] nf_tables nfnetlink [38354.317458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38354.324582] bridge stp [38354.327025] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38354.334151] llc [38354.337374] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38354.344509] xfs [38354.346952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38354.354076] libcrc32c [38354.355913] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38354.363989] sunrpc [38354.365827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38354.371565] intel_rapl_msr intel_rapl_common [38354.373924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38354.381047] sb_edac [38354.383145] Call Trace: [38354.383146] [38354.390268] x86_pkg_temp_thermal coretemp [38354.394619] ? watchdog_timer_fn+0x1b4/0x220 [38354.401742] kvm_intel kvm [38354.403928] ? __hrtimer_run_queues+0x1d3/0x360 [38354.406372] irqbypass [38354.408385] ? hrtimer_interrupt+0x104/0x240 [38354.412473] i2c_i801 iTCO_wdt [38354.416736] ? __local_bh_enable+0x50/0x60 [38354.419441] iTCO_vendor_support i2c_smbus [38354.423966] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38354.426322] crct10dif_pclmul [38354.430588] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38354.433638] joydev crc32_pclmul [38354.437728] [38354.441818] ipmi_ssif [38354.447035] [38354.447037] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38354.450001] ghash_clmulni_intel [38354.454962] ? queued_spin_lock_slowpath+0x16/0x2e0 [38354.458181] mei_me mei [38354.460281] do_raw_spin_lock+0xb5/0xc0 [38354.462635] lpc_ich sha512_ssse3 [38354.464734] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38354.470037] acpi_ipmi mfd_core [38354.473263] try_get_cap_refs+0x159/0x880 [ceph] [38354.478133] wmi ipmi_si [38354.480582] __ceph_get_caps+0x4a1/0x760 [ceph] [38354.484406] ipmi_devintf ipmi_msghandler [38354.487722] ? __file_remove_privs+0x89/0x110 [38354.492509] ioatdma [38354.495648] ? wake_bit_function+0x70/0x70 [38354.500260] acpi_pad [38354.502808] ? ceph_write_iter+0x260/0x770 [ceph] [38354.507332] acpi_power_meter ixgbe [38354.511335] ceph_write_iter+0x260/0x770 [ceph] [38354.515686] igb [38354.517875] ? lock_is_held_type+0xaa/0x120 [38354.521960] mdio [38354.524234] vfs_write+0x324/0x4f0 [38354.528928] crc32c_intel [38354.532416] ksys_pwrite64+0x65/0xa0 [38354.536938] nvme [38354.538790] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38354.542968] ptp nvme_core [38354.544895] do_syscall_64+0x38/0x80 [38354.548291] i2c_algo_bit pps_core [38354.550908] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38354.554478] dca fuse [38354.556402] RIP: 0033:0x7f48f0eeb93b [38354.561532] [38354.564236] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38354.567809] irq event stamp: 227245626 [38354.571206] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38354.576249] hardirqs last enabled at (227245625): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38354.578519] ORIG_RAX: 0000000000000012 [38354.582092] hardirqs last disabled at (227245626): [] sysvec_apic_timer_interrupt+0xe/0x90 [38354.583581] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38354.602320] softirqs last enabled at (227245624): [] __do_softirq+0x2be/0x371 [38354.606063] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38354.611280] softirqs last disabled at (227245619): [] irq_exit_rcu+0xb0/0xd0 [38354.621525] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38354.625356] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38354.635165] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38354.642290] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38354.651061] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38354.658186] Workqueue: events delayed_work [ceph] [38354.666801] [38354.673916] [38354.713384] RIP: 0010:__list_del_entry_valid_or_report+0x15/0xd0 [38354.719391] Code: 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a 48 85 d2 74 49 <48> b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 22 01 00 00 00 [38354.738129] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [38354.743346] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [38354.750471] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [38354.757594] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [38354.764719] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [38354.771842] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38354.778968] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38354.787044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38354.792783] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38354.799905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38354.807030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38354.814155] Call Trace: [38354.816601] [38354.818611] ? watchdog_timer_fn+0x1b4/0x220 [38354.822875] ? __hrtimer_run_queues+0x1d3/0x360 [38354.827402] ? hrtimer_interrupt+0x104/0x240 [38354.831669] ? __local_bh_enable+0x50/0x60 [38354.835764] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38354.840986] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38354.845944] [38354.848041] [38354.850141] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38354.855447] ? __list_del_entry_valid_or_report+0x15/0xd0 [38354.860845] ceph_check_delayed_caps+0x8f/0x270 [ceph] [38354.865993] delayed_work+0x1d8/0x280 [ceph] [38354.870276] ? process_scheduled_works+0x22a/0x510 [38354.875065] process_scheduled_works+0x2a4/0x510 [38354.879682] worker_thread+0x188/0x340 [38354.883430] ? rescuer_thread+0x310/0x310 [38354.887431] kthread+0xf5/0x130 [38354.890571] ? kthread_complete_and_exit+0x20/0x20 [38354.895354] ret_from_fork+0x31/0x40 [38354.898934] ? kthread_complete_and_exit+0x20/0x20 [38354.903718] ret_from_fork_asm+0x11/0x20 [38354.907642] [38382.126557] watchdog: BUG: soft lockup - CPU#0 stuck for 24632s! [dbench:78683] [38382.133864] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38382.138557] watchdog: BUG: soft lockup - CPU#7 stuck for 33629s! [kworker/7:4:78808] [38382.189313] irq event stamp: 67783362 [38382.197044] Modules linked in: ceph libceph dns_resolver fscache netfs [38382.200700] hardirqs last enabled at (67783361): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38382.207231] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38382.217384] hardirqs last disabled at (67783362): [] sysvec_apic_timer_interrupt+0xe/0x90 [38382.223469] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [38382.233193] softirqs last enabled at (67783314): [] __do_softirq+0x2be/0x371 [38382.240587] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [38382.249270] softirqs last disabled at (67783309): [] irq_exit_rcu+0xb0/0xd0 [38382.256134] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [38382.264647] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38382.270296] kvm irqbypass i2c_i801 [38382.279239] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38382.282725] iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul [38382.290195] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38382.296539] joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel [38382.302018] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38382.307840] mei_me mei lpc_ich [38382.326578] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38382.326580] sha512_ssse3 acpi_ipmi [38382.329716] [38382.334950] mfd_core [38382.338435] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38382.339924] wmi ipmi_si [38382.342195] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38382.349321] ipmi_devintf ipmi_msghandler [38382.351849] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38382.358975] ioatdma [38382.362977] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38382.370103] acpi_pad acpi_power_meter [38382.372287] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38382.379412] ixgbe [38382.383155] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38382.390279] igb [38382.392289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38382.400367] mdio crc32c_intel [38382.402206] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38382.407953] nvme [38382.411001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38382.418124] ptp nvme_core [38382.420049] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38382.427173] i2c_algo_bit [38382.429877] Call Trace: [38382.429879] [38382.437001] pps_core [38382.439621] ? watchdog_timer_fn+0x1b4/0x220 [38382.442064] dca fuse [38382.444075] ? __hrtimer_run_queues+0x1d3/0x360 [38382.446346] [38382.446347] irq event stamp: 227466542 [38382.450613] ? hrtimer_interrupt+0x104/0x240 [38382.452880] hardirqs last enabled at (227466541): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38382.457403] ? __local_bh_enable+0x50/0x60 [38382.458894] hardirqs last disabled at (227466542): [] sysvec_apic_timer_interrupt+0xe/0x90 [38382.462641] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38382.466903] softirqs last enabled at (227466540): [] __do_softirq+0x2be/0x371 [38382.477148] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38382.481237] softirqs last disabled at (227466535): [] irq_exit_rcu+0xb0/0xd0 [38382.491049] [38382.496267] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38382.505037] [38382.509995] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38382.518592] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38382.520691] Workqueue: events delayed_work [ceph] [38382.530070] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38382.532164] [38382.539635] do_raw_spin_lock+0xb5/0xc0 [38382.544955] RIP: 0010:lock_acquire+0xf0/0x2b0 [38382.549655] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38382.554523] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38382.556016] try_get_cap_refs+0x159/0x880 [ceph] [38382.559846] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38382.564201] __ceph_get_caps+0x4a1/0x760 [ceph] [38382.568980] [38382.568981] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38382.587723] ? __file_remove_privs+0x89/0x110 [38382.592329] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38382.597546] ? wake_bit_function+0x70/0x70 [38382.602071] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38382.603564] ? ceph_write_iter+0x260/0x770 [ceph] [38382.610686] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38382.615036] ceph_write_iter+0x260/0x770 [ceph] [38382.622161] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38382.626256] ? lock_is_held_type+0xaa/0x120 [38382.633375] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38382.638076] vfs_write+0x324/0x4f0 [38382.645198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38382.649724] ksys_pwrite64+0x65/0xa0 [38382.656845] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38382.661024] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38382.669101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38382.672497] do_syscall_64+0x38/0x80 [38382.678250] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38382.681824] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38382.688957] Call Trace: [38382.688959] [38382.694087] RIP: 0033:0x7f48f0eeb93b [38382.701212] ? watchdog_timer_fn+0x1b4/0x220 [38382.704784] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38382.711906] ? __hrtimer_run_queues+0x1d3/0x360 [38382.716960] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38382.719405] ? hrtimer_interrupt+0x104/0x240 [38382.721413] ORIG_RAX: 0000000000000012 [38382.724983] ? __local_bh_enable+0x50/0x60 [38382.729247] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38382.747988] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38382.752508] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38382.757727] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38382.761992] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38382.765823] [38382.769911] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38382.777037] [38382.782252] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38382.789379] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38382.794340] [38382.801463] ? lock_acquire+0xf0/0x2b0 [38382.831136] ? igrab+0x43/0x50 [38382.834187] ? lock_release+0xc5/0x260 [38382.837932] _raw_spin_lock+0x30/0x40 [38382.841598] ? igrab+0x1d/0x50 [38382.844647] igrab+0x1d/0x50 [38382.847526] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38382.852676] delayed_work+0x1d8/0x280 [ceph] [38382.856967] ? process_scheduled_works+0x22a/0x510 [38382.861754] process_scheduled_works+0x2a4/0x510 [38382.866371] worker_thread+0x188/0x340 [38382.870121] ? rescuer_thread+0x310/0x310 [38382.874130] kthread+0xf5/0x130 [38382.877270] ? kthread_complete_and_exit+0x20/0x20 [38382.882053] ret_from_fork+0x31/0x40 [38382.885624] ? kthread_complete_and_exit+0x20/0x20 [38382.890408] ret_from_fork_asm+0x11/0x20 [38382.894332] [38382.984938] rcu: INFO: rcu_preempt self-detected stall on CPU [38382.990692] rcu: 7-....: (25677132 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/25675772 fqs=12945094 [38383.001369] rcu: (t=26445911 jiffies g=1309609 q=718097 ncpus=8) [38383.007453] Sending NMI from CPU 7 to CPUs 0: [38383.011807] NMI backtrace for cpu 0 [38383.011808] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38383.011810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38383.011810] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38383.011813] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38383.011815] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38383.011816] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38383.011817] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38383.011817] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38383.011818] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38383.011819] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38383.011819] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38383.011820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38383.011821] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38383.011822] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38383.011823] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38383.011823] Call Trace: [38383.011824] [38383.011825] ? nmi_cpu_backtrace+0xad/0x120 [38383.011828] ? nmi_cpu_backtrace_handler+0x11/0x20 [38383.011829] ? nmi_handle+0xd4/0x260 [38383.011834] ? default_do_nmi+0x49/0xf0 [38383.011835] ? exc_nmi+0x11e/0x150 [38383.011837] ? end_repeat_nmi+0x16/0x67 [38383.011841] ? queued_spin_lock_slowpath+0x18/0x2e0 [38383.011843] ? queued_spin_lock_slowpath+0x18/0x2e0 [38383.011846] ? queued_spin_lock_slowpath+0x18/0x2e0 [38383.011848] [38383.011849] [38383.011850] do_raw_spin_lock+0xb5/0xc0 [38383.011852] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38383.011866] try_get_cap_refs+0x159/0x880 [ceph] [38383.011880] __ceph_get_caps+0x4a1/0x760 [ceph] [38383.011892] ? __file_remove_privs+0x89/0x110 [38383.011894] ? wake_bit_function+0x70/0x70 [38383.011897] ? ceph_write_iter+0x260/0x770 [ceph] [38383.011907] ceph_write_iter+0x260/0x770 [ceph] [38383.011919] ? lock_is_held_type+0xaa/0x120 [38383.011921] vfs_write+0x324/0x4f0 [38383.011925] ksys_pwrite64+0x65/0xa0 [38383.011927] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38383.011930] do_syscall_64+0x38/0x80 [38383.011932] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38383.011934] RIP: 0033:0x7f48f0eeb93b [38383.011935] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38383.011937] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [38383.011939] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38383.011940] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38383.011941] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38383.011942] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38383.011943] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38383.011946] [38383.012806] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38383.316294] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38383.323765] Workqueue: events delayed_work [ceph] [38383.328481] RIP: 0010:lock_acquire+0xf0/0x2b0 [38383.332831] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38383.351569] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38383.356786] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38383.363911] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38383.371033] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38383.378160] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38383.385282] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38383.392407] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38383.400484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38383.406222] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38383.413347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38383.420472] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38383.427594] Call Trace: [38383.430039] [38383.432049] ? rcu_dump_cpu_stacks+0xd1/0x180 [38383.436410] ? rcu_sched_clock_irq+0x633/0x1390 [38383.440934] ? lock_release+0xc5/0x260 [38383.444678] ? lock_is_held_type+0xaa/0x120 [38383.448856] ? lock_is_held_type+0xaa/0x120 [38383.453033] ? tick_sched_do_timer+0xa0/0xa0 [38383.457304] ? rcu_read_lock_sched_held+0x56/0x90 [38383.462004] ? tick_sched_do_timer+0xa0/0xa0 [38383.466275] ? update_process_times+0x5f/0x90 [38383.470627] ? tick_sched_handle.isra.16+0x1e/0x50 [38383.475410] ? tick_sched_timer+0x6f/0x80 [38383.479416] ? __hrtimer_run_queues+0x1d3/0x360 [38383.483951] ? hrtimer_interrupt+0x104/0x240 [38383.488219] ? __local_bh_enable+0x50/0x60 [38383.492314] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38383.497536] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38383.502495] [38383.504592] [38383.506691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38383.512006] ? lock_acquire+0xf0/0x2b0 [38383.515756] ? igrab+0x43/0x50 [38383.518806] ? lock_release+0xc5/0x260 [38383.522551] _raw_spin_lock+0x30/0x40 [38383.526215] ? igrab+0x1d/0x50 [38383.529264] igrab+0x1d/0x50 [38383.532143] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38383.537294] delayed_work+0x1d8/0x280 [ceph] [38383.541573] ? process_scheduled_works+0x22a/0x510 [38383.546356] process_scheduled_works+0x2a4/0x510 [38383.550971] worker_thread+0x188/0x340 [38383.554721] ? rescuer_thread+0x310/0x310 [38383.558724] kthread+0xf5/0x130 [38383.561862] ? kthread_complete_and_exit+0x20/0x20 [38383.566647] ret_from_fork+0x31/0x40 [38383.570226] ? kthread_complete_and_exit+0x20/0x20 [38383.575009] ret_from_fork_asm+0x11/0x20 [38383.578933] [38410.127003] watchdog: BUG: soft lockup - CPU#0 stuck for 24658s! [dbench:78683] [38410.134306] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38410.139003] watchdog: BUG: soft lockup - CPU#7 stuck for 33655s! [kworker/7:4:78808] [38410.189750] irq event stamp: 67841452 [38410.189751] hardirqs last enabled at (67841451): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38410.197480] Modules linked in: [38410.201138] hardirqs last disabled at (67841452): [] sysvec_apic_timer_interrupt+0xe/0x90 [38410.211295] ceph [38410.214346] softirqs last enabled at (67841408): [] __do_softirq+0x2be/0x371 [38410.224069] libceph [38410.225994] softirqs last disabled at (67841403): [] irq_exit_rcu+0xb0/0xd0 [38410.234678] dns_resolver fscache [38410.236862] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38410.245372] netfs [38410.248699] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38410.257645] veth nft_chain_nat [38410.259656] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38410.267144] xt_MASQUERADE nf_nat [38410.270280] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38410.275758] nf_conntrack nf_defrag_ipv6 [38410.279068] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38410.297806] nf_defrag_ipv4 nft_compat [38410.301723] [38410.301724] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38410.306941] nf_tables nfnetlink [38410.310686] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38410.312177] bridge [38410.319299] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38410.322525] stp llc [38410.329649] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38410.331745] xfs libcrc32c [38410.338869] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38410.341053] sunrpc intel_rapl_msr [38410.348177] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38410.350882] intel_rapl_common sb_edac [38410.358007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38410.361404] x86_pkg_temp_thermal coretemp [38410.369480] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38410.373224] kvm_intel kvm [38410.378963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38410.383054] irqbypass i2c_i801 [38410.390178] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38410.392882] iTCO_wdt iTCO_vendor_support [38410.400006] Call Trace: [38410.400008] [38410.403144] i2c_smbus crct10dif_pclmul [38410.410268] ? watchdog_timer_fn+0x1b4/0x220 [38410.414270] joydev [38410.416717] ? __hrtimer_run_queues+0x1d3/0x360 [38410.418726] crc32_pclmul ipmi_ssif [38410.422561] ? hrtimer_interrupt+0x104/0x240 [38410.426820] ghash_clmulni_intel mei_me [38410.428917] ? __local_bh_enable+0x50/0x60 [38410.433442] mei [38410.436929] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38410.441191] lpc_ich sha512_ssse3 [38410.445022] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38410.449113] acpi_ipmi mfd_core [38410.450949] [38410.456166] wmi ipmi_si [38410.459478] [38410.459480] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38410.464434] ipmi_devintf [38410.467577] ? queued_spin_lock_slowpath+0x16/0x2e0 [38410.469669] ipmi_msghandler ioatdma [38410.472220] do_raw_spin_lock+0xb5/0xc0 [38410.474314] acpi_pad acpi_power_meter [38410.479620] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38410.482236] ixgbe igb [38410.487109] try_get_cap_refs+0x159/0x880 [ceph] [38410.490680] mdio crc32c_intel [38410.494514] __ceph_get_caps+0x4a1/0x760 [ceph] [38410.498252] nvme ptp [38410.503041] ? __file_remove_privs+0x89/0x110 [38410.505402] nvme_core [38410.510015] ? wake_bit_function+0x70/0x70 [38410.513065] i2c_algo_bit pps_core [38410.517590] ? ceph_write_iter+0x260/0x770 [ceph] [38410.519859] dca fuse [38410.524211] ceph_write_iter+0x260/0x770 [ceph] [38410.526568] [38410.526569] irq event stamp: 227682602 [38410.530678] ? lock_is_held_type+0xaa/0x120 [38410.534073] hardirqs last enabled at (227682601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38410.538773] vfs_write+0x324/0x4f0 [38410.541041] hardirqs last disabled at (227682602): [] sysvec_apic_timer_interrupt+0xe/0x90 [38410.545570] ksys_pwrite64+0x65/0xa0 [38410.547056] softirqs last enabled at (227682600): [] __do_softirq+0x2be/0x371 [38410.550800] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38410.554978] softirqs last disabled at (227682595): [] irq_exit_rcu+0xb0/0xd0 [38410.565223] do_syscall_64+0x38/0x80 [38410.568618] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38410.578429] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38410.582002] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38410.590771] RIP: 0033:0x7f48f0eeb93b [38410.595902] Workqueue: events delayed_work [ceph] [38410.604500] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38410.608070] [38410.608071] RIP: 0010:lock_acquire+0xf0/0x2b0 [38410.617447] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38410.622492] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38410.629962] ORIG_RAX: 0000000000000012 [38410.633534] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38410.638230] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38410.656967] [38410.656968] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38410.658460] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38410.662809] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38410.668028] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38410.686766] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38410.690597] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38410.695814] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38410.702937] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38410.704429] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38410.711559] [38410.718676] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38410.778807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38410.784544] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38410.791684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38410.798809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38410.805935] Call Trace: [38410.808378] [38410.810407] ? watchdog_timer_fn+0x1b4/0x220 [38410.814672] ? __hrtimer_run_queues+0x1d3/0x360 [38410.819216] ? hrtimer_interrupt+0x104/0x240 [38410.823484] ? __local_bh_enable+0x50/0x60 [38410.827577] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38410.832794] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38410.837751] [38410.839848] [38410.841945] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38410.847252] ? lock_acquire+0xf0/0x2b0 [38410.851004] ? igrab+0x43/0x50 [38410.854054] ? lock_release+0xc5/0x260 [38410.857798] _raw_spin_lock+0x30/0x40 [38410.861454] ? igrab+0x1d/0x50 [38410.864504] igrab+0x1d/0x50 [38410.867384] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38410.872550] delayed_work+0x1d8/0x280 [ceph] [38410.876831] ? process_scheduled_works+0x22a/0x510 [38410.881622] process_scheduled_works+0x2a4/0x510 [38410.886237] worker_thread+0x188/0x340 [38410.889987] ? rescuer_thread+0x310/0x310 [38410.893990] kthread+0xf5/0x130 [38410.897127] ? kthread_complete_and_exit+0x20/0x20 [38410.901911] ret_from_fork+0x31/0x40 [38410.905482] ? kthread_complete_and_exit+0x20/0x20 [38410.910265] ret_from_fork_asm+0x11/0x20 [38410.914189] [38438.127452] watchdog: BUG: soft lockup - CPU#0 stuck for 24684s! [dbench:78683] [38438.134752] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38438.139452] watchdog: BUG: soft lockup - CPU#7 stuck for 33681s! [kworker/7:4:78808] [38438.190202] irq event stamp: 67899732 [38438.197932] Modules linked in: ceph libceph dns_resolver fscache netfs [38438.201590] hardirqs last enabled at (67899731): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38438.201596] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [38438.218282] hardirqs last disabled at (67899732): [] sysvec_apic_timer_interrupt+0xe/0x90 [38438.225666] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [38438.235391] softirqs last enabled at (67899694): [] __do_softirq+0x2be/0x371 [38438.235395] llc [38438.235396] softirqs last disabled at (67899689): [] irq_exit_rcu+0xb0/0xd0 [38438.241847] xfs libcrc32c [38438.250534] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38438.252369] sunrpc intel_rapl_msr [38438.260881] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38438.263582] intel_rapl_common [38438.272528] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38438.275924] sb_edac [38438.283396] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38438.286447] x86_pkg_temp_thermal [38438.291925] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38438.294108] coretemp kvm_intel [38438.312851] [38438.316164] kvm [38438.321382] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38438.324520] irqbypass [38438.326009] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38438.327850] i2c_i801 iTCO_wdt [38438.334972] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38438.337330] iTCO_vendor_support [38438.344453] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38438.347503] i2c_smbus crct10dif_pclmul [38438.354628] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38438.357853] joydev [38438.364977] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38438.368807] crc32_pclmul ipmi_ssif [38438.375931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38438.378028] ghash_clmulni_intel mei_me [38438.386106] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38438.389609] mei lpc_ich [38438.395345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38438.399177] sha512_ssse3 [38438.406299] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38438.406300] Call Trace: [38438.408848] acpi_ipmi [38438.415974] [38438.418589] mfd_core wmi [38438.425715] ? watchdog_timer_fn+0x1b4/0x220 [38438.428157] ipmi_si ipmi_devintf [38438.430517] ? __hrtimer_run_queues+0x1d3/0x360 [38438.432527] ipmi_msghandler ioatdma [38438.435147] ? hrtimer_interrupt+0x104/0x240 [38438.439407] acpi_pad acpi_power_meter [38438.442718] ? __local_bh_enable+0x50/0x60 [38438.447242] ixgbe igb [38438.450815] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38438.455076] mdio crc32c_intel [38438.458823] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38438.462929] nvme [38438.465288] [38438.465288] [38438.470503] ptp nvme_core [38438.473555] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38438.478512] i2c_algo_bit [38438.480440] ? queued_spin_lock_slowpath+0x18/0x2e0 [38438.482534] pps_core dca [38438.484634] do_raw_spin_lock+0xb5/0xc0 [38438.487335] fuse [38438.492642] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38438.495256] [38438.495257] irq event stamp: 227903442 [38438.500135] try_get_cap_refs+0x159/0x880 [ceph] [38438.502754] hardirqs last enabled at (227903441): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38438.506589] __ceph_get_caps+0x4a1/0x760 [ceph] [38438.508509] hardirqs last disabled at (227903442): [] sysvec_apic_timer_interrupt+0xe/0x90 [38438.513296] ? __file_remove_privs+0x89/0x110 [38438.514782] softirqs last enabled at (227903440): [] __do_softirq+0x2be/0x371 [38438.518528] ? wake_bit_function+0x70/0x70 [38438.523137] softirqs last disabled at (227903435): [] irq_exit_rcu+0xb0/0xd0 [38438.533384] ? ceph_write_iter+0x260/0x770 [ceph] [38438.537907] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38438.547718] ceph_write_iter+0x260/0x770 [ceph] [38438.552068] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38438.560857] ? lock_is_held_type+0xaa/0x120 [38438.564947] Workqueue: events delayed_work [ceph] [38438.573546] vfs_write+0x324/0x4f0 [38438.578242] [38438.578243] RIP: 0010:lock_acquire+0xf0/0x2b0 [38438.587624] ksys_pwrite64+0x65/0xa0 [38438.592142] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38438.599615] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38438.603792] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38438.608491] do_syscall_64+0x38/0x80 [38438.611885] [38438.613376] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38438.617728] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38438.621301] RIP: 0033:0x7f48f0eeb93b [38438.640036] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38438.645169] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38438.650383] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38438.653955] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38438.655446] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38438.660490] ORIG_RAX: 0000000000000012 [38438.667614] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38438.671183] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38438.678308] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38438.697045] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38438.704169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38438.709388] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38438.716511] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38438.720344] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38438.727466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38438.734591] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38438.742669] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38438.749798] [38438.755530] Call Trace: [38438.802904] [38438.804915] ? watchdog_timer_fn+0x1b4/0x220 [38438.809180] ? __hrtimer_run_queues+0x1d3/0x360 [38438.813705] ? hrtimer_interrupt+0x104/0x240 [38438.817974] ? __local_bh_enable+0x50/0x60 [38438.822069] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38438.827293] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38438.832250] [38438.834345] [38438.836444] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38438.841752] ? lock_acquire+0xf0/0x2b0 [38438.845502] ? igrab+0x43/0x50 [38438.848553] ? lock_release+0xc5/0x260 [38438.852298] _raw_spin_lock+0x30/0x40 [38438.855960] ? igrab+0x1d/0x50 [38438.859011] igrab+0x1d/0x50 [38438.861891] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38438.867041] delayed_work+0x1d8/0x280 [ceph] [38438.871322] ? process_scheduled_works+0x22a/0x510 [38438.876113] process_scheduled_works+0x2a4/0x510 [38438.880746] worker_thread+0x188/0x340 [38438.884494] ? rescuer_thread+0x310/0x310 [38438.888498] kthread+0xf5/0x130 [38438.891636] ? kthread_complete_and_exit+0x20/0x20 [38438.896420] ret_from_fork+0x31/0x40 [38438.899999] ? kthread_complete_and_exit+0x20/0x20 [38438.904783] ret_from_fork_asm+0x11/0x20 [38438.908715] [38466.127903] watchdog: BUG: soft lockup - CPU#0 stuck for 24710s! [dbench:78683] [38466.135205] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38466.139903] watchdog: BUG: soft lockup - CPU#7 stuck for 33707s! [kworker/7:4:78808] [38466.190647] irq event stamp: 67957990 [38466.190648] hardirqs last enabled at (67957989): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38466.198377] Modules linked in: [38466.202034] hardirqs last disabled at (67957990): [] sysvec_apic_timer_interrupt+0xe/0x90 [38466.212193] ceph [38466.215243] softirqs last enabled at (67957952): [] __do_softirq+0x2be/0x371 [38466.224967] libceph dns_resolver [38466.226890] softirqs last disabled at (67957947): [] irq_exit_rcu+0xb0/0xd0 [38466.235588] fscache netfs [38466.238902] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38466.247414] veth nft_chain_nat [38466.250118] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38466.259062] xt_MASQUERADE [38466.262200] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38466.269669] nf_nat nf_conntrack [38466.272374] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38466.277852] nf_defrag_ipv6 nf_defrag_ipv4 [38466.281075] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38466.299813] nft_compat nf_tables [38466.303905] [38466.303906] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38466.309122] nfnetlink bridge [38466.312431] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38466.313923] stp llc [38466.321047] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38466.324010] xfs libcrc32c [38466.331134] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38466.333318] sunrpc intel_rapl_msr [38466.340443] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38466.343148] intel_rapl_common sb_edac [38466.350272] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38466.353685] x86_pkg_temp_thermal coretemp [38466.360810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38466.364554] kvm_intel [38466.372632] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38466.376722] kvm irqbypass [38466.382459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38466.384817] i2c_i801 iTCO_wdt [38466.391942] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38466.394646] iTCO_vendor_support i2c_smbus [38466.401770] Call Trace: [38466.401772] [38466.404821] crct10dif_pclmul [38466.411944] ? watchdog_timer_fn+0x1b4/0x220 [38466.416035] joydev crc32_pclmul [38466.418481] ? __hrtimer_run_queues+0x1d3/0x360 [38466.420489] ipmi_ssif ghash_clmulni_intel [38466.423456] ? hrtimer_interrupt+0x104/0x240 [38466.427718] mei_me [38466.430942] ? __local_bh_enable+0x50/0x60 [38466.435465] mei lpc_ich [38466.439560] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38466.443822] sha512_ssse3 [38466.445920] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38466.450010] acpi_ipmi mfd_core [38466.452541] [38466.457756] wmi [38466.460375] [38466.460376] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38466.465333] ipmi_si [38466.468474] ? queued_spin_lock_slowpath+0x16/0x2e0 [38466.470582] ipmi_devintf [38466.472425] do_raw_spin_lock+0xb5/0xc0 [38466.474519] ipmi_msghandler ioatdma [38466.479824] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38466.482008] acpi_pad acpi_power_meter [38466.486880] try_get_cap_refs+0x159/0x880 [ceph] [38466.489495] ixgbe igb [38466.493332] __ceph_get_caps+0x4a1/0x760 [ceph] [38466.496897] mdio crc32c_intel [38466.501685] ? __file_remove_privs+0x89/0x110 [38466.505424] nvme [38466.510037] ? wake_bit_function+0x70/0x70 [38466.512393] ptp nvme_core [38466.516920] ? ceph_write_iter+0x260/0x770 [ceph] [38466.519967] i2c_algo_bit [38466.524319] ceph_write_iter+0x260/0x770 [ceph] [38466.526244] pps_core [38466.530338] ? lock_is_held_type+0xaa/0x120 [38466.533038] dca fuse [38466.537739] vfs_write+0x324/0x4f0 [38466.540353] [38466.540354] irq event stamp: 228124172 [38466.544881] ksys_pwrite64+0x65/0xa0 [38466.547146] hardirqs last enabled at (228124171): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38466.551324] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38466.553595] hardirqs last disabled at (228124172): [] sysvec_apic_timer_interrupt+0xe/0x90 [38466.556994] do_syscall_64+0x38/0x80 [38466.558483] softirqs last enabled at (228124170): [] __do_softirq+0x2be/0x371 [38466.562227] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38466.565798] softirqs last disabled at (228124165): [] irq_exit_rcu+0xb0/0xd0 [38466.576042] RIP: 0033:0x7f48f0eeb93b [38466.581173] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38466.590983] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38466.594555] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38466.603325] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38466.608370] Workqueue: events delayed_work [ceph] [38466.616967] ORIG_RAX: 0000000000000012 [38466.620537] [38466.629914] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38466.648653] RIP: 0010:lock_acquire+0xf0/0x2b0 [38466.656123] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38466.661341] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38466.666039] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38466.669869] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38466.671359] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38466.678484] [38466.678484] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38466.682836] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38466.689959] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38466.708702] [38466.715822] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38466.760334] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38466.767459] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38466.774581] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38466.782661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38466.788398] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38466.795520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38466.802644] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38466.809770] Call Trace: [38466.812215] [38466.814225] ? watchdog_timer_fn+0x1b4/0x220 [38466.818488] ? __hrtimer_run_queues+0x1d3/0x360 [38466.823015] ? hrtimer_interrupt+0x104/0x240 [38466.827303] ? __local_bh_enable+0x50/0x60 [38466.831396] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38466.836620] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38466.841586] [38466.843684] [38466.845780] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38466.851090] ? lock_acquire+0xf0/0x2b0 [38466.854838] ? igrab+0x43/0x50 [38466.857888] ? lock_release+0xc5/0x260 [38466.861634] _raw_spin_lock+0x30/0x40 [38466.865309] ? igrab+0x1d/0x50 [38466.868358] igrab+0x1d/0x50 [38466.871236] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38466.876386] delayed_work+0x1d8/0x280 [ceph] [38466.880666] ? process_scheduled_works+0x22a/0x510 [38466.885448] process_scheduled_works+0x2a4/0x510 [38466.890064] worker_thread+0x188/0x340 [38466.893812] ? rescuer_thread+0x310/0x310 [38466.897815] kthread+0xf5/0x130 [38466.900955] ? kthread_complete_and_exit+0x20/0x20 [38466.905738] ret_from_fork+0x31/0x40 [38466.909311] ? kthread_complete_and_exit+0x20/0x20 [38466.914102] ret_from_fork_asm+0x11/0x20 [38466.918025] [38494.128355] watchdog: BUG: soft lockup - CPU#0 stuck for 24736s! [dbench:78683] [38494.135659] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38494.140355] watchdog: BUG: soft lockup - CPU#7 stuck for 33733s! [kworker/7:4:78808] [38494.191108] irq event stamp: 68016198 [38494.191109] hardirqs last enabled at (68016197): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38494.198839] Modules linked in: [38494.202497] hardirqs last disabled at (68016198): [] sysvec_apic_timer_interrupt+0xe/0x90 [38494.212654] ceph [38494.215703] softirqs last enabled at (68016160): [] __do_softirq+0x2be/0x371 [38494.225428] libceph dns_resolver [38494.227351] softirqs last disabled at (68016155): [] irq_exit_rcu+0xb0/0xd0 [38494.236037] fscache netfs [38494.239350] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38494.247857] veth [38494.250562] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38494.259506] nft_chain_nat [38494.261430] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38494.268901] xt_MASQUERADE nf_nat [38494.271607] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38494.277082] nf_conntrack nf_defrag_ipv6 [38494.280394] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38494.299131] nf_defrag_ipv4 nft_compat [38494.303049] [38494.308266] nf_tables nfnetlink [38494.312010] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38494.313517] bridge stp [38494.316751] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38494.323875] llc [38494.326318] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38494.333444] xfs libcrc32c [38494.335279] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38494.342405] sunrpc [38494.345109] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38494.352233] intel_rapl_msr [38494.354331] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38494.361453] intel_rapl_common sb_edac [38494.364245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38494.372322] x86_pkg_temp_thermal [38494.376067] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38494.381804] coretemp [38494.385116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38494.392239] kvm_intel kvm [38494.394510] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38494.401634] irqbypass [38494.404338] Call Trace: [38494.404340] [38494.411461] i2c_i801 iTCO_wdt [38494.413821] ? watchdog_timer_fn+0x1b4/0x220 [38494.416263] iTCO_vendor_support [38494.418276] ? __hrtimer_run_queues+0x1d3/0x360 [38494.421323] i2c_smbus crct10dif_pclmul [38494.425592] ? hrtimer_interrupt+0x104/0x240 [38494.428812] joydev crc32_pclmul [38494.433336] ? __local_bh_enable+0x50/0x60 [38494.437166] ipmi_ssif ghash_clmulni_intel [38494.441434] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38494.444657] mei_me mei [38494.448755] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38494.452846] lpc_ich sha512_ssse3 [38494.458064] [38494.460508] acpi_ipmi [38494.465465] [38494.468777] mfd_core wmi [38494.470873] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38494.473229] ipmi_si [38494.475333] ? queued_spin_lock_slowpath+0x18/0x2e0 [38494.477947] ipmi_devintf [38494.483251] do_raw_spin_lock+0xb5/0xc0 [38494.485434] ipmi_msghandler ioatdma [38494.490305] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38494.492921] acpi_pad [38494.496757] try_get_cap_refs+0x159/0x880 [ceph] [38494.500322] acpi_power_meter [38494.505111] __ceph_get_caps+0x4a1/0x760 [ceph] [38494.507379] ixgbe igb [38494.511993] ? __file_remove_privs+0x89/0x110 [38494.514952] mdio crc32c_intel [38494.519476] ? wake_bit_function+0x70/0x70 [38494.521834] nvme ptp [38494.526187] ? ceph_write_iter+0x260/0x770 [ceph] [38494.529236] nvme_core [38494.533326] ceph_write_iter+0x260/0x770 [ceph] [38494.535596] i2c_algo_bit pps_core [38494.540299] ? lock_is_held_type+0xaa/0x120 [38494.542651] dca fuse [38494.547178] vfs_write+0x324/0x4f0 [38494.550572] [38494.550572] irq event stamp: 228345006 [38494.554759] ksys_pwrite64+0x65/0xa0 [38494.557022] hardirqs last enabled at (228345005): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38494.560437] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38494.561926] hardirqs last disabled at (228345006): [] sysvec_apic_timer_interrupt+0xe/0x90 [38494.565671] do_syscall_64+0x38/0x80 [38494.569240] softirqs last enabled at (228345004): [] __do_softirq+0x2be/0x371 [38494.579488] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38494.584617] softirqs last disabled at (228344999): [] irq_exit_rcu+0xb0/0xd0 [38494.594427] RIP: 0033:0x7f48f0eeb93b [38494.597997] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38494.606769] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38494.611812] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38494.620411] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38494.623981] Workqueue: events delayed_work [ceph] [38494.633358] ORIG_RAX: 0000000000000012 [38494.652097] [38494.652098] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [38494.659567] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38494.664785] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [38494.669481] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38494.673312] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [38494.674802] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38494.679414] [38494.679414] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [38494.686539] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38494.705274] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [38494.712401] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38494.717616] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [38494.724762] [38494.726250] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [38494.771188] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38494.778311] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38494.786390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38494.792126] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38494.799252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38494.806375] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38494.813500] Call Trace: [38494.815943] [38494.817955] ? watchdog_timer_fn+0x1b4/0x220 [38494.822219] ? __hrtimer_run_queues+0x1d3/0x360 [38494.826763] ? hrtimer_interrupt+0x104/0x240 [38494.831038] ? __local_bh_enable+0x50/0x60 [38494.835133] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38494.840358] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38494.845314] [38494.847411] [38494.849511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38494.854819] ? do_raw_spin_lock+0x42/0xc0 [38494.858829] igrab+0x1d/0x50 [38494.861705] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38494.866855] delayed_work+0x1d8/0x280 [ceph] [38494.871137] ? process_scheduled_works+0x22a/0x510 [38494.875927] process_scheduled_works+0x2a4/0x510 [38494.880541] worker_thread+0x188/0x340 [38494.884292] ? rescuer_thread+0x310/0x310 [38494.888302] kthread+0xf5/0x130 [38494.891440] ? kthread_complete_and_exit+0x20/0x20 [38494.896225] ret_from_fork+0x31/0x40 [38494.899804] ? kthread_complete_and_exit+0x20/0x20 [38494.904587] ret_from_fork_asm+0x11/0x20 [38494.908512] [38522.128810] watchdog: BUG: soft lockup - CPU#0 stuck for 24762s! [dbench:78683] [38522.136111] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38522.140809] watchdog: BUG: soft lockup - CPU#7 stuck for 33759s! [kworker/7:4:78808] [38522.191551] irq event stamp: 68074448 [38522.191552] hardirqs last enabled at (68074447): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38522.199284] Modules linked in: [38522.202940] hardirqs last disabled at (68074448): [] sysvec_apic_timer_interrupt+0xe/0x90 [38522.213097] ceph [38522.216148] softirqs last enabled at (68074410): [] __do_softirq+0x2be/0x371 [38522.225874] libceph [38522.227796] softirqs last disabled at (68074405): [] irq_exit_rcu+0xb0/0xd0 [38522.236484] dns_resolver [38522.238665] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38522.247175] fscache [38522.249792] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38522.258737] netfs veth [38522.260922] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38522.268392] nft_chain_nat xt_MASQUERADE [38522.270835] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38522.276313] nf_nat [38522.280231] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38522.298969] nf_conntrack [38522.301066] [38522.306284] nf_defrag_ipv6 [38522.308900] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38522.310392] nf_defrag_ipv4 nft_compat [38522.313181] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38522.320323] nf_tables nfnetlink [38522.324068] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38522.331206] bridge [38522.334433] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38522.341559] stp [38522.343655] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38522.350778] llc xfs [38522.352617] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38522.359741] libcrc32c sunrpc [38522.361924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38522.370001] intel_rapl_msr intel_rapl_common [38522.372967] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38522.378704] sb_edac x86_pkg_temp_thermal [38522.383055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38522.390178] coretemp kvm_intel [38522.394182] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38522.401323] kvm irqbypass [38522.404461] Call Trace: [38522.404462] [38522.411602] i2c_i801 iTCO_wdt [38522.414307] ? watchdog_timer_fn+0x1b4/0x220 [38522.416752] iTCO_vendor_support i2c_smbus [38522.418762] ? __hrtimer_run_queues+0x1d3/0x360 [38522.421812] crct10dif_pclmul joydev [38522.426081] ? hrtimer_interrupt+0x104/0x240 [38522.430167] crc32_pclmul ipmi_ssif [38522.434690] ? __local_bh_enable+0x50/0x60 [38522.438261] ghash_clmulni_intel mei_me [38522.442528] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38522.446010] mei lpc_ich [38522.450101] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38522.453932] sha512_ssse3 [38522.459150] [38522.461679] acpi_ipmi [38522.466636] [38522.469253] mfd_core wmi [38522.471354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38522.473709] ipmi_si ipmi_devintf [38522.475811] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38522.478423] ipmi_msghandler ioatdma [38522.483731] do_raw_spin_lock+0xb5/0xc0 [38522.487040] acpi_pad acpi_power_meter [38522.491911] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38522.495480] ixgbe igb [38522.499313] try_get_cap_refs+0x159/0x880 [ceph] [38522.503054] mdio crc32c_intel [38522.507843] __ceph_get_caps+0x4a1/0x760 [ceph] [38522.510211] nvme [38522.514828] ? __file_remove_privs+0x89/0x110 [38522.517877] ptp nvme_core [38522.522399] ? wake_bit_function+0x70/0x70 [38522.524323] i2c_algo_bit pps_core [38522.528677] ? ceph_write_iter+0x260/0x770 [ceph] [38522.531379] dca [38522.535470] ceph_write_iter+0x260/0x770 [ceph] [38522.538868] fuse [38522.543566] ? lock_is_held_type+0xaa/0x120 [38522.545401] irq event stamp: 228565804 [38522.549929] vfs_write+0x324/0x4f0 [38522.551849] hardirqs last enabled at (228565803): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38522.556031] ksys_pwrite64+0x65/0xa0 [38522.559770] hardirqs last disabled at (228565804): [] sysvec_apic_timer_interrupt+0xe/0x90 [38522.563169] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38522.573412] softirqs last enabled at (228565802): [] __do_softirq+0x2be/0x371 [38522.576986] do_syscall_64+0x38/0x80 [38522.586795] softirqs last disabled at (228565797): [] irq_exit_rcu+0xb0/0xd0 [38522.591927] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38522.600697] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38522.604267] RIP: 0033:0x7f48f0eeb93b [38522.612864] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38522.617909] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38522.627286] Workqueue: events delayed_work [ceph] [38522.630856] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38522.638344] [38522.638345] RIP: 0010:lock_release+0x78/0x260 [38522.657083] ORIG_RAX: 0000000000000012 [38522.661781] Code: 00 00 65 8b 05 e9 dc ea 7e 85 c0 0f 85 ef 00 00 00 65 48 8b 1c 25 c0 e5 02 00 8b b3 54 0a 00 00 85 f6 0f 85 d8 00 00 00 9c 58 <0f> 1f 40 00 49 89 c5 fa 0f 1f 44 00 00 8b 0d 8d 0b 54 01 c7 44 24 [38522.666997] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38522.668493] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [38522.672846] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38522.676677] [38522.676678] RAX: 0000000000000246 RBX: ffff88810debaa80 RCX: 000000005553a634 [38522.695416] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38522.702540] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 [38522.707758] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38522.714882] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [38522.716371] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38522.723495] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [38522.730626] [38522.737744] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38522.775550] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38522.783625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38522.789363] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38522.796496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38522.803621] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38522.810746] Call Trace: [38522.813191] [38522.815220] ? watchdog_timer_fn+0x1b4/0x220 [38522.819496] ? __hrtimer_run_queues+0x1d3/0x360 [38522.824028] ? hrtimer_interrupt+0x104/0x240 [38522.828296] ? __local_bh_enable+0x50/0x60 [38522.832390] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38522.837612] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38522.842569] [38522.844666] [38522.846766] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38522.852068] ? igrab+0x43/0x50 [38522.855123] ? lock_release+0x78/0x260 [38522.858874] _raw_spin_unlock+0x1b/0x40 [38522.862712] igrab+0x43/0x50 [38522.865591] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38522.870739] delayed_work+0x1d8/0x280 [ceph] [38522.875021] ? process_scheduled_works+0x22a/0x510 [38522.879812] process_scheduled_works+0x2a4/0x510 [38522.884425] worker_thread+0x188/0x340 [38522.888176] ? rescuer_thread+0x310/0x310 [38522.892180] kthread+0xf5/0x130 [38522.895315] ? kthread_complete_and_exit+0x20/0x20 [38522.900101] ret_from_fork+0x31/0x40 [38522.903670] ? kthread_complete_and_exit+0x20/0x20 [38522.908456] ret_from_fork_asm+0x11/0x20 [38522.912379] [38550.129265] watchdog: BUG: soft lockup - CPU#0 stuck for 24788s! [dbench:78683] [38550.136564] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38550.141265] watchdog: BUG: soft lockup - CPU#7 stuck for 33785s! [kworker/7:4:78808] [38550.192006] irq event stamp: 68132646 [38550.199737] Modules linked in: ceph [38550.203392] hardirqs last enabled at (68132645): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38550.203394] libceph dns_resolver fscache netfs [38550.217035] hardirqs last disabled at (68132646): [] sysvec_apic_timer_interrupt+0xe/0x90 [38550.221560] veth nft_chain_nat xt_MASQUERADE [38550.231283] softirqs last enabled at (68132608): [] __do_softirq+0x2be/0x371 [38550.231284] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [38550.244334] softirqs last disabled at (68132603): [] irq_exit_rcu+0xb0/0xd0 [38550.244336] nft_compat nf_tables nfnetlink bridge [38550.258671] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38550.263454] stp llc [38550.272397] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38550.274582] xfs libcrc32c sunrpc [38550.282053] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38550.282055] intel_rapl_msr intel_rapl_common sb_edac [38550.290840] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38550.290841] x86_pkg_temp_thermal coretemp [38550.295886] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38550.314622] kvm_intel kvm [38550.318713] [38550.318714] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38550.323942] irqbypass i2c_i801 [38550.326661] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38550.328153] iTCO_wdt iTCO_vendor_support [38550.335277] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38550.338414] i2c_smbus crct10dif_pclmul [38550.345538] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38550.349542] joydev [38550.356667] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38550.360497] crc32_pclmul ipmi_ssif [38550.367620] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38550.369719] ghash_clmulni_intel [38550.376842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38550.380327] mei_me mei [38550.388403] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38550.391627] lpc_ich sha512_ssse3 [38550.397365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38550.399811] acpi_ipmi mfd_core [38550.406942] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38550.410252] wmi ipmi_si [38550.417378] Call Trace: [38550.417379] [38550.420515] ipmi_devintf [38550.427658] ? watchdog_timer_fn+0x1b4/0x220 [38550.430186] ipmi_msghandler ioatdma [38550.432631] ? __hrtimer_run_queues+0x1d3/0x360 [38550.434641] acpi_pad acpi_power_meter [38550.437281] ? hrtimer_interrupt+0x104/0x240 [38550.441540] ixgbe igb [38550.445112] ? __local_bh_enable+0x50/0x60 [38550.449634] mdio [38550.453382] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38550.457660] crc32c_intel [38550.460019] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38550.464110] nvme ptp [38550.466032] [38550.471250] nvme_core [38550.473866] [38550.473868] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38550.478824] i2c_algo_bit [38550.481100] ? queued_spin_lock_slowpath+0x16/0x2e0 [38550.483194] pps_core [38550.485554] do_raw_spin_lock+0xb5/0xc0 [38550.487663] dca fuse [38550.492970] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38550.495586] [38550.495587] irq event stamp: 228786554 [38550.500460] try_get_cap_refs+0x159/0x880 [ceph] [38550.502729] hardirqs last enabled at (228786553): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38550.506563] __ceph_get_caps+0x4a1/0x760 [ceph] [38550.508830] hardirqs last disabled at (228786554): [] sysvec_apic_timer_interrupt+0xe/0x90 [38550.513617] ? __file_remove_privs+0x89/0x110 [38550.515103] softirqs last enabled at (228786552): [] __do_softirq+0x2be/0x371 [38550.518848] ? wake_bit_function+0x70/0x70 [38550.523460] softirqs last disabled at (228786547): [] irq_exit_rcu+0xb0/0xd0 [38550.533707] ? ceph_write_iter+0x260/0x770 [ceph] [38550.538229] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38550.548040] ceph_write_iter+0x260/0x770 [ceph] [38550.552390] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38550.561164] ? lock_is_held_type+0xaa/0x120 [38550.565251] Workqueue: events delayed_work [ceph] [38550.573851] vfs_write+0x324/0x4f0 [38550.578545] [38550.578546] RIP: 0010:lock_release+0x14e/0x260 [38550.587927] ksys_pwrite64+0x65/0xa0 [38550.592448] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [38550.599919] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38550.604096] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [38550.608794] do_syscall_64+0x38/0x80 [38550.612190] [38550.612190] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [38550.613682] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38550.618117] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [38550.621689] RIP: 0033:0x7f48f0eeb93b [38550.640425] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [38550.645556] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38550.650774] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [38550.654347] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38550.655837] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [38550.662960] ORIG_RAX: 0000000000000012 [38550.668004] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38550.675128] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38550.678699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38550.685822] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38550.704561] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38550.711686] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38550.716902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38550.724028] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38550.727857] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38550.735934] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38550.743060] Call Trace: [38550.743062] [38550.748802] [38550.755920] ? watchdog_timer_fn+0x1b4/0x220 [38550.809569] ? __hrtimer_run_queues+0x1d3/0x360 [38550.814096] ? hrtimer_interrupt+0x104/0x240 [38550.818367] ? __local_bh_enable+0x50/0x60 [38550.822459] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38550.827682] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38550.832640] [38550.834736] [38550.836834] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38550.842139] ? igrab+0x43/0x50 [38550.845193] ? lock_release+0x14e/0x260 [38550.849031] ? lock_release+0xc5/0x260 [38550.852775] _raw_spin_unlock+0x1b/0x40 [38550.856612] igrab+0x43/0x50 [38550.859489] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38550.864642] delayed_work+0x1d8/0x280 [ceph] [38550.868954] ? process_scheduled_works+0x22a/0x510 [38550.873738] process_scheduled_works+0x2a4/0x510 [38550.878353] worker_thread+0x188/0x340 [38550.882102] ? rescuer_thread+0x310/0x310 [38550.886107] kthread+0xf5/0x130 [38550.889244] ? kthread_complete_and_exit+0x20/0x20 [38550.894027] ret_from_fork+0x31/0x40 [38550.897597] ? kthread_complete_and_exit+0x20/0x20 [38550.902383] ret_from_fork_asm+0x11/0x20 [38550.906315] [38578.129723] watchdog: BUG: soft lockup - CPU#0 stuck for 24814s! [dbench:78683] [38578.137027] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38578.141723] watchdog: BUG: soft lockup - CPU#7 stuck for 33811s! [kworker/7:4:78808] [38578.192476] irq event stamp: 68190794 [38578.200207] Modules linked in: ceph libceph dns_resolver fscache netfs veth [38578.203863] hardirqs last enabled at (68190793): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38578.210822] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38578.220981] hardirqs last disabled at (68190794): [] sysvec_apic_timer_interrupt+0xe/0x90 [38578.226630] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [38578.236354] softirqs last enabled at (68190756): [] __do_softirq+0x2be/0x371 [38578.244085] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [38578.252770] softirqs last disabled at (68190751): [] irq_exit_rcu+0xb0/0xd0 [38578.252774] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [38578.268493] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38578.275444] iTCO_wdt iTCO_vendor_support [38578.284400] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38578.284400] i2c_smbus crct10dif_pclmul [38578.288407] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38578.295880] joydev crc32_pclmul [38578.299711] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38578.305188] ipmi_ssif [38578.308412] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38578.327149] ghash_clmulni_intel mei_me [38578.329507] [38578.334723] mei lpc_ich [38578.338555] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38578.340046] sha512_ssse3 [38578.342576] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38578.349700] acpi_ipmi [38578.352317] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38578.359441] mfd_core wmi [38578.361799] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38578.368924] ipmi_si [38578.371541] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38578.378665] ipmi_devintf [38578.380848] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38578.387973] ipmi_msghandler ioatdma [38578.390590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38578.398668] acpi_pad acpi_power_meter [38578.402238] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38578.407977] ixgbe igb [38578.411721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38578.418843] mdio crc32c_intel [38578.421202] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38578.428326] nvme ptp [38578.431375] Call Trace: [38578.431378] [38578.438501] nvme_core [38578.440772] ? watchdog_timer_fn+0x1b4/0x220 [38578.443216] i2c_algo_bit pps_core [38578.445228] ? __hrtimer_run_queues+0x1d3/0x360 [38578.447583] dca fuse [38578.451851] ? hrtimer_interrupt+0x104/0x240 [38578.455246] [38578.455246] irq event stamp: 229007428 [38578.459768] ? __local_bh_enable+0x50/0x60 [38578.462040] hardirqs last enabled at (229007427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38578.466307] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38578.467794] hardirqs last disabled at (229007428): [] sysvec_apic_timer_interrupt+0xe/0x90 [38578.471538] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38578.475628] softirqs last enabled at (229007426): [] __do_softirq+0x2be/0x371 [38578.485874] [38578.491091] softirqs last disabled at (229007421): [] irq_exit_rcu+0xb0/0xd0 [38578.500903] [38578.505858] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38578.514629] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38578.516728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38578.525329] ? queued_spin_lock_slowpath+0x16/0x2e0 [38578.527423] Workqueue: events delayed_work [ceph] [38578.536802] do_raw_spin_lock+0xb5/0xc0 [38578.542121] [38578.549593] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38578.554461] RIP: 0010:lock_release+0x14e/0x260 [38578.559163] try_get_cap_refs+0x159/0x880 [ceph] [38578.562990] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [38578.564487] __ceph_get_caps+0x4a1/0x760 [ceph] [38578.569266] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [38578.573707] ? __file_remove_privs+0x89/0x110 [38578.578313] [38578.578314] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [38578.591097] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [38578.591100] rcu: 0-....: (26038955 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=13041067 [38578.591122] rcu: 7-....: (25867056 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/25865695 fqs=13041067 [38578.591125] rcu: (detected by 3, t=26641499 jiffies, g=1309609, q=721983 ncpus=8) [38578.591128] Sending NMI from CPU 3 to CPUs 0: [38578.591132] NMI backtrace for cpu 0 [38578.591133] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38578.591134] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38578.591135] RIP: 0010:io_serial_in+0x19/0x20 [38578.591138] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [38578.591140] RSP: 0018:ffffc90000003bc0 EFLAGS: 00000002 [38578.591141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [38578.591142] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [38578.591143] RBP: 0000000000002662 R08: 00007c679e1343f8 R09: 0000000000000000 [38578.591144] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [38578.591144] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [38578.591145] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38578.591146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38578.591147] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38578.591148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38578.591149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38578.591150] Call Trace: [38578.591150] [38578.591151] ? nmi_cpu_backtrace+0xad/0x120 [38578.591155] ? nmi_cpu_backtrace_handler+0x11/0x20 [38578.591156] ? nmi_handle+0xd4/0x260 [38578.591161] ? default_do_nmi+0x49/0xf0 [38578.591162] ? exc_nmi+0x11e/0x150 [38578.591164] ? end_repeat_nmi+0x16/0x67 [38578.591168] ? io_serial_in+0x19/0x20 [38578.591170] ? io_serial_in+0x19/0x20 [38578.591172] ? io_serial_in+0x19/0x20 [38578.591173] [38578.591174] [38578.591174] wait_for_lsr+0x45/0x70 [38578.591176] serial8250_console_write+0x341/0x3f0 [38578.591180] ? lock_release+0xc5/0x260 [38578.591182] console_flush_all+0x227/0x5a0 [38578.591185] ? console_flush_all+0x1ff/0x5a0 [38578.591186] ? console_flush_all+0x1ff/0x5a0 [38578.591189] console_unlock+0x7b/0x120 [38578.591191] vprintk_emit+0x246/0x340 [38578.591193] ? __file_remove_privs+0x89/0x110 [38578.591195] ? __file_remove_privs+0x89/0x110 [38578.591196] _printk+0x5c/0x80 [38578.591199] ? __file_remove_privs+0x89/0x110 [38578.591200] show_trace_log_lvl+0x1ac/0x3f0 [38578.591206] ? ceph_write_iter+0x260/0x770 [ceph] [38578.591218] ? touch_softlockup_watchdog+0x20/0x20 [38578.591220] watchdog_timer_fn+0x1b4/0x220 [38578.591222] __hrtimer_run_queues+0x1d3/0x360 [38578.591225] hrtimer_interrupt+0x104/0x240 [38578.591226] ? __local_bh_enable+0x50/0x60 [38578.591229] __sysvec_apic_timer_interrupt+0x61/0x160 [38578.591231] sysvec_apic_timer_interrupt+0x6d/0x90 [38578.591233] [38578.591234] [38578.591234] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38578.591236] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38578.591239] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38578.591240] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38578.591241] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38578.591241] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38578.591242] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38578.591243] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38578.591243] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38578.591247] do_raw_spin_lock+0xb5/0xc0 [38578.591249] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38578.591262] try_get_cap_refs+0x159/0x880 [ceph] [38578.591278] __ceph_get_caps+0x4a1/0x760 [ceph] [38578.591291] ? __file_remove_privs+0x89/0x110 [38578.591293] ? wake_bit_function+0x70/0x70 [38578.591295] ? ceph_write_iter+0x260/0x770 [ceph] [38578.591306] ceph_write_iter+0x260/0x770 [ceph] [38578.591317] ? lock_is_held_type+0xaa/0x120 [38578.591320] vfs_write+0x324/0x4f0 [38578.591325] ksys_pwrite64+0x65/0xa0 [38578.591327] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38578.591330] do_syscall_64+0x38/0x80 [38578.591334] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38578.591335] RIP: 0033:0x7f48f0eeb93b [38578.591337] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38578.591338] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [38578.591339] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38578.591340] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38578.591341] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38578.591342] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38578.591342] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38578.591345] [38578.592130] Sending NMI from CPU 3 to CPUs 7: [38578.592133] NMI backtrace for cpu 7 [38578.592134] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38578.592136] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38578.592137] Workqueue: events delayed_work [ceph] [38578.592156] RIP: 0010:vprintk_emit+0x1eb/0x340 [38578.592158] Code: 50 41 b8 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 <0f> b6 11 84 d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 [38578.592160] RSP: 0018:ffffc90000264db8 EFLAGS: 00000002 [38578.592162] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [38578.592163] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb550 [38578.592164] RBP: 0000000000000041 R08: 0000000000005ef4 R09: 0000000000000003 [38578.592165] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [38578.592166] R13: 0000000000000000 R14: ffffffff821886d8 R15: ffffc90000264e08 [38578.592168] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38578.592169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38578.592170] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38578.592172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38578.592173] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38578.592174] Call Trace: [38578.592175] [38578.592176] ? nmi_cpu_backtrace+0xad/0x120 [38578.592180] ? nmi_cpu_backtrace_handler+0x11/0x20 [38578.592182] ? nmi_handle+0xd4/0x260 [38578.592187] ? default_do_nmi+0x49/0xf0 [38578.592189] ? exc_nmi+0x11e/0x150 [38578.592192] ? end_repeat_nmi+0x16/0x67 [38578.592198] ? vprintk_emit+0x1eb/0x340 [38578.592201] ? vprintk_emit+0x1eb/0x340 [38578.592204] ? vprintk_emit+0x1eb/0x340 [38578.592206] [38578.592207] [38578.592209] ? touch_softlockup_watchdog+0x20/0x20 [38578.592211] _printk+0x5c/0x80 [38578.592216] __show_regs+0x5d/0x2e0 [38578.592220] ? touch_softlockup_watchdog+0x20/0x20 [38578.592222] show_regs+0x38/0x60 [38578.592225] watchdog_timer_fn+0x1b4/0x220 [38578.592228] __hrtimer_run_queues+0x1d3/0x360 [38578.592233] hrtimer_interrupt+0x104/0x240 [38578.592235] ? __local_bh_enable+0x50/0x60 [38578.592240] __sysvec_apic_timer_interrupt+0x61/0x160 [38578.592243] sysvec_apic_timer_interrupt+0x6d/0x90 [38578.592246] [38578.592246] [38578.592247] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38578.592250] RIP: 0010:lock_release+0x14e/0x260 [38578.592252] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [38578.592253] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [38578.592255] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [38578.592256] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [38578.592257] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [38578.592258] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [38578.592259] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [38578.592261] ? igrab+0x43/0x50 [38578.592267] ? lock_release+0xc5/0x260 [38578.592270] _raw_spin_unlock+0x1b/0x40 [38578.592273] igrab+0x43/0x50 [38578.592276] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38578.592296] delayed_work+0x1d8/0x280 [ceph] [38578.592315] ? process_scheduled_works+0x22a/0x510 [38578.592319] process_scheduled_works+0x2a4/0x510 [38578.592325] worker_thread+0x188/0x340 [38578.592329] ? rescuer_thread+0x310/0x310 [38578.592332] kthread+0xf5/0x130 [38578.592335] ? kthread_complete_and_exit+0x20/0x20 [38578.592338] ret_from_fork+0x31/0x40 [38578.592341] ? kthread_complete_and_exit+0x20/0x20 [38578.592344] ret_from_fork_asm+0x11/0x20 [38578.592351] [38578.597051] ? wake_bit_function+0x70/0x70 [38578.601574] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [38578.606796] ? ceph_write_iter+0x260/0x770 [ceph] [38578.611142] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [38578.612634] ceph_write_iter+0x260/0x770 [ceph] [38578.619757] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [38578.625845] ? lock_is_held_type+0xaa/0x120 [38578.636345] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [38578.647028] vfs_write+0x324/0x4f0 [38578.654581] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38578.658936] ksys_pwrite64+0x65/0xa0 [38578.662416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38578.671361] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38578.678832] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38578.683096] do_syscall_64+0x38/0x80 [38578.701833] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38578.707050] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38578.714175] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38578.721299] RIP: 0033:0x7f48f0eeb93b [38578.728423] Call Trace: [38578.728424] [38578.735546] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38578.742673] ? watchdog_timer_fn+0x1b4/0x220 [38578.750749] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38578.756487] ? __hrtimer_run_queues+0x1d3/0x360 [38578.763611] ORIG_RAX: 0000000000000012 [38578.770737] ? hrtimer_interrupt+0x104/0x240 [38578.777859] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38578.780303] ? __local_bh_enable+0x50/0x60 [38578.782313] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38578.786495] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38578.791274] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38578.794847] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38578.798675] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38578.802073] [38578.805904] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38578.809563] [38578.809564] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38578.813225] [38578.816876] ? igrab+0x43/0x50 [38579.680781] ? lock_release+0x14e/0x260 [38579.684618] ? lock_release+0xc5/0x260 [38579.688364] _raw_spin_unlock+0x1b/0x40 [38579.692200] igrab+0x43/0x50 [38579.695080] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38579.700230] delayed_work+0x1d8/0x280 [ceph] [38579.704510] ? process_scheduled_works+0x22a/0x510 [38579.709299] process_scheduled_works+0x2a4/0x510 [38579.713915] worker_thread+0x188/0x340 [38579.717665] ? rescuer_thread+0x310/0x310 [38579.721667] kthread+0xf5/0x130 [38579.724805] ? kthread_complete_and_exit+0x20/0x20 [38579.729589] ret_from_fork+0x31/0x40 [38579.733161] ? kthread_complete_and_exit+0x20/0x20 [38579.737944] ret_from_fork_asm+0x11/0x20 [38579.741868] [38606.130181] watchdog: BUG: soft lockup - CPU#0 stuck for 24840s! [dbench:78683] [38606.137487] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38606.142181] watchdog: BUG: soft lockup - CPU#7 stuck for 33837s! [kworker/7:4:78808] [38606.192929] irq event stamp: 68248860 [38606.192930] hardirqs last enabled at (68248859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38606.200659] Modules linked in: [38606.204316] hardirqs last disabled at (68248860): [] sysvec_apic_timer_interrupt+0xe/0x90 [38606.214474] ceph [38606.217524] softirqs last enabled at (68248846): [] __do_softirq+0x2be/0x371 [38606.227248] libceph [38606.229172] softirqs last disabled at (68248841): [] irq_exit_rcu+0xb0/0xd0 [38606.237856] dns_resolver fscache [38606.240041] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38606.248552] netfs veth [38606.251880] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38606.260825] nft_chain_nat [38606.263267] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38606.270737] xt_MASQUERADE nf_nat [38606.273443] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38606.278920] nf_conntrack nf_defrag_ipv6 [38606.282230] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38606.300968] nf_defrag_ipv4 nft_compat [38606.304886] [38606.304887] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38606.310103] nf_tables nfnetlink [38606.313858] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38606.315346] bridge stp [38606.322470] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38606.325695] llc xfs [38606.332820] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38606.335264] libcrc32c sunrpc [38606.342387] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38606.344576] intel_rapl_msr intel_rapl_common [38606.351704] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38606.354668] sb_edac x86_pkg_temp_thermal [38606.361792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38606.366143] coretemp kvm_intel [38606.374220] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38606.378226] kvm irqbypass [38606.383961] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38606.387099] i2c_i801 iTCO_wdt [38606.394223] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38606.396928] iTCO_vendor_support i2c_smbus [38606.404051] Call Trace: [38606.404053] [38606.407102] crct10dif_pclmul joydev [38606.414229] ? watchdog_timer_fn+0x1b4/0x220 [38606.418316] crc32_pclmul ipmi_ssif [38606.420762] ? __hrtimer_run_queues+0x1d3/0x360 [38606.422771] ghash_clmulni_intel [38606.426346] ? hrtimer_interrupt+0x104/0x240 [38606.430606] mei_me [38606.434090] ? __local_bh_enable+0x50/0x60 [38606.438616] mei lpc_ich [38606.441842] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38606.446104] sha512_ssse3 acpi_ipmi [38606.448200] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38606.452290] mfd_core [38606.454823] [38606.460040] wmi ipmi_si [38606.463524] [38606.468480] ipmi_devintf ipmi_msghandler [38606.470754] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38606.472865] ioatdma acpi_pad [38606.475401] ? queued_spin_lock_slowpath+0x18/0x2e0 [38606.477494] acpi_power_meter ixgbe [38606.481500] do_raw_spin_lock+0xb5/0xc0 [38606.486803] igb mdio crc32c_intel [38606.489768] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38606.494638] nvme ptp [38606.498124] try_get_cap_refs+0x159/0x880 [ceph] [38606.501951] nvme_core i2c_algo_bit [38606.505353] __ceph_get_caps+0x4a1/0x760 [ceph] [38606.510135] pps_core dca [38606.512408] ? __file_remove_privs+0x89/0x110 [38606.517016] fuse [38606.520499] ? wake_bit_function+0x70/0x70 [38606.525024] irq event stamp: 229221314 [38606.527643] ? ceph_write_iter+0x260/0x770 [ceph] [38606.531991] hardirqs last enabled at (229221313): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38606.533916] ceph_write_iter+0x260/0x770 [ceph] [38606.538005] hardirqs last disabled at (229221314): [] sysvec_apic_timer_interrupt+0xe/0x90 [38606.541754] ? lock_is_held_type+0xaa/0x120 [38606.546448] softirqs last enabled at (229221312): [] __do_softirq+0x2be/0x371 [38606.556694] vfs_write+0x324/0x4f0 [38606.561215] softirqs last disabled at (229221307): [] irq_exit_rcu+0xb0/0xd0 [38606.571032] ksys_pwrite64+0x65/0xa0 [38606.575205] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38606.583976] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38606.587371] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38606.595971] do_syscall_64+0x38/0x80 [38606.599542] Workqueue: events delayed_work [ceph] [38606.608919] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38606.614049] [38606.614050] RIP: 0010:lock_acquire+0xf0/0x2b0 [38606.621520] RIP: 0033:0x7f48f0eeb93b [38606.625090] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38606.629789] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38606.634832] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38606.636323] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38606.640673] [38606.640674] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38606.644245] ORIG_RAX: 0000000000000012 [38606.662981] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38606.681720] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38606.686938] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38606.692153] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38606.693645] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38606.700770] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38606.704599] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38606.711725] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38606.718864] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38606.725990] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38606.733113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38606.740243] [38606.747362] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38606.791858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38606.798983] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38606.806107] Call Trace: [38606.808551] [38606.810563] ? watchdog_timer_fn+0x1b4/0x220 [38606.814842] ? __hrtimer_run_queues+0x1d3/0x360 [38606.819388] ? hrtimer_interrupt+0x104/0x240 [38606.823657] ? __local_bh_enable+0x50/0x60 [38606.827751] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38606.832974] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38606.837930] [38606.840029] [38606.842126] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38606.847436] ? lock_acquire+0xf0/0x2b0 [38606.851185] ? igrab+0x43/0x50 [38606.854233] ? lock_release+0xc5/0x260 [38606.857978] _raw_spin_lock+0x30/0x40 [38606.861634] ? igrab+0x1d/0x50 [38606.864686] igrab+0x1d/0x50 [38606.867563] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38606.872731] delayed_work+0x1d8/0x280 [ceph] [38606.877011] ? process_scheduled_works+0x22a/0x510 [38606.881794] process_scheduled_works+0x2a4/0x510 [38606.886410] worker_thread+0x188/0x340 [38606.890158] ? rescuer_thread+0x310/0x310 [38606.894161] kthread+0xf5/0x130 [38606.897298] ? kthread_complete_and_exit+0x20/0x20 [38606.902084] ret_from_fork+0x31/0x40 [38606.905654] ? kthread_complete_and_exit+0x20/0x20 [38606.910438] ret_from_fork_asm+0x11/0x20 [38606.914360] [38634.130638] watchdog: BUG: soft lockup - CPU#0 stuck for 24866s! [dbench:78683] [38634.137941] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38634.142638] watchdog: BUG: soft lockup - CPU#7 stuck for 33863s! [kworker/7:4:78808] [38634.193391] irq event stamp: 68307020 [38634.201122] Modules linked in: ceph libceph dns_resolver fscache netfs veth [38634.204777] hardirqs last enabled at (68307019): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38634.211737] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [38634.221895] hardirqs last disabled at (68307020): [] sysvec_apic_timer_interrupt+0xe/0x90 [38634.228844] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [38634.238569] softirqs last enabled at (68306992): [] __do_softirq+0x2be/0x371 [38634.245347] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [38634.254033] softirqs last disabled at (68306987): [] irq_exit_rcu+0xb0/0xd0 [38634.262725] coretemp kvm_intel kvm irqbypass [38634.271237] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38634.275585] i2c_i801 iTCO_wdt iTCO_vendor_support [38634.284530] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38634.289313] i2c_smbus crct10dif_pclmul joydev crc32_pclmul [38634.296785] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38634.296787] ipmi_ssif ghash_clmulni_intel mei_me mei [38634.307826] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38634.312870] lpc_ich sha512_ssse3 [38634.331609] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38634.331609] acpi_ipmi [38634.331610] [38634.334918] mfd_core wmi [38634.340136] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38634.342495] ipmi_si ipmi_devintf [38634.343985] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38634.346600] ipmi_msghandler ioatdma [38634.353726] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38634.357036] acpi_pad acpi_power_meter [38634.364159] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38634.367731] ixgbe igb [38634.374854] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38634.378600] mdio crc32c_intel [38634.385724] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38634.388080] nvme ptp [38634.395206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38634.398255] nvme_core i2c_algo_bit [38634.406333] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38634.408603] pps_core dca [38634.414343] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38634.417826] fuse [38634.424949] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38634.427566] irq event stamp: 229442142 [38634.434692] Call Trace: [38634.436614] hardirqs last enabled at (229442141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38634.443740] [38634.447483] hardirqs last disabled at (229442142): [] sysvec_apic_timer_interrupt+0xe/0x90 [38634.449928] ? watchdog_timer_fn+0x1b4/0x220 [38634.460172] softirqs last enabled at (229442140): [] __do_softirq+0x2be/0x371 [38634.462183] ? __hrtimer_run_queues+0x1d3/0x360 [38634.471992] softirqs last disabled at (229442135): [] irq_exit_rcu+0xb0/0xd0 [38634.476261] ? hrtimer_interrupt+0x104/0x240 [38634.485038] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38634.489562] ? __local_bh_enable+0x50/0x60 [38634.498158] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38634.502425] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38634.511802] Workqueue: events delayed_work [ceph] [38634.515890] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38634.523363] [38634.523364] RIP: 0010:lock_acquire+0xf0/0x2b0 [38634.528579] [38634.533275] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38634.538234] [38634.538235] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38634.539725] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38634.544080] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38634.546174] [38634.546174] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38634.564912] do_raw_spin_lock+0xb5/0xc0 [38634.567008] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38634.572315] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38634.577528] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38634.582403] try_get_cap_refs+0x159/0x880 [ceph] [38634.583890] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38634.591037] __ceph_get_caps+0x4a1/0x760 [ceph] [38634.594862] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38634.601991] ? __file_remove_privs+0x89/0x110 [38634.606772] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38634.613895] ? wake_bit_function+0x70/0x70 [38634.618505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38634.625633] ? ceph_write_iter+0x260/0x770 [ceph] [38634.630155] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38634.637277] ceph_write_iter+0x260/0x770 [ceph] [38634.641628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38634.649711] ? lock_is_held_type+0xaa/0x120 [38634.653797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38634.659537] vfs_write+0x324/0x4f0 [38634.664233] Call Trace: [38634.664234] [38634.671360] ksys_pwrite64+0x65/0xa0 [38634.675882] ? watchdog_timer_fn+0x1b4/0x220 [38634.683004] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38634.687184] ? __hrtimer_run_queues+0x1d3/0x360 [38634.694322] do_syscall_64+0x38/0x80 [38634.697725] ? hrtimer_interrupt+0x104/0x240 [38634.700167] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38634.702177] ? __local_bh_enable+0x50/0x60 [38634.705745] RIP: 0033:0x7f48f0eeb93b [38634.710012] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38634.715142] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38634.719667] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38634.723236] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38634.727500] [38634.732544] ORIG_RAX: 0000000000000012 [38634.736634] [38634.740204] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38634.745424] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38634.764161] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38634.769122] ? lock_acquire+0xf0/0x2b0 [38634.774335] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38634.776433] ? igrab+0x43/0x50 [38634.780264] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38634.782361] ? lock_release+0xc5/0x260 [38634.789486] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38634.794792] _raw_spin_lock+0x30/0x40 [38634.801919] [38634.805658] ? igrab+0x1d/0x50 [38634.842716] igrab+0x1d/0x50 [38634.845596] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38634.850746] delayed_work+0x1d8/0x280 [ceph] [38634.855044] ? process_scheduled_works+0x22a/0x510 [38634.859834] process_scheduled_works+0x2a4/0x510 [38634.864449] worker_thread+0x188/0x340 [38634.868198] ? rescuer_thread+0x310/0x310 [38634.872203] kthread+0xf5/0x130 [38634.875347] ? kthread_complete_and_exit+0x20/0x20 [38634.880132] ret_from_fork+0x31/0x40 [38634.883712] ? kthread_complete_and_exit+0x20/0x20 [38634.888496] ret_from_fork_asm+0x11/0x20 [38634.892420] [38662.131095] watchdog: BUG: soft lockup - CPU#0 stuck for 24892s! [dbench:78683] [38662.138403] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38662.143095] watchdog: BUG: soft lockup - CPU#7 stuck for 33889s! [kworker/7:4:78808] [38662.193851] irq event stamp: 68365124 [38662.201582] Modules linked in: ceph libceph dns_resolver fscache [38662.205239] hardirqs last enabled at (68365123): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38662.211237] netfs veth nft_chain_nat [38662.221394] hardirqs last disabled at (68365124): [] sysvec_apic_timer_interrupt+0xe/0x90 [38662.225052] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [38662.234776] softirqs last enabled at (68365096): [] __do_softirq+0x2be/0x371 [38662.240512] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [38662.249196] softirqs last disabled at (68365091): [] irq_exit_rcu+0xb0/0xd0 [38662.249199] stp llc xfs [38662.263793] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38662.263794] libcrc32c [38662.263795] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38662.266323] sunrpc intel_rapl_msr [38662.275268] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38662.277624] intel_rapl_common sb_edac [38662.285095] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38662.288493] x86_pkg_temp_thermal [38662.293970] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38662.297713] coretemp kvm_intel [38662.316452] [38662.319771] kvm irqbypass [38662.324989] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38662.328126] i2c_i801 [38662.329616] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38662.332320] iTCO_wdt iTCO_vendor_support [38662.339444] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38662.341715] i2c_smbus crct10dif_pclmul [38662.348839] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38662.352843] joydev crc32_pclmul [38662.359969] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38662.363798] ipmi_ssif [38662.370924] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38662.374146] ghash_clmulni_intel mei_me [38662.381270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38662.383629] mei lpc_ich [38662.391707] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38662.395536] sha512_ssse3 acpi_ipmi [38662.401275] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38662.403805] mfd_core wmi [38662.410928] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38662.414413] ipmi_si ipmi_devintf [38662.421537] Call Trace: [38662.424156] ipmi_msghandler ioatdma [38662.431279] [38662.434589] acpi_pad acpi_power_meter [38662.437035] ? watchdog_timer_fn+0x1b4/0x220 [38662.440604] ixgbe [38662.442616] ? __hrtimer_run_queues+0x1d3/0x360 [38662.446358] igb mdio [38662.450626] ? hrtimer_interrupt+0x104/0x240 [38662.452634] crc32c_intel nvme [38662.457157] ? __local_bh_enable+0x50/0x60 [38662.459429] ptp nvme_core [38662.463695] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38662.466744] i2c_algo_bit pps_core [38662.470834] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38662.473539] dca fuse [38662.478757] [38662.482170] [38662.482171] irq event stamp: 229663080 [38662.487128] [38662.489398] hardirqs last enabled at (229663079): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38662.491497] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38662.492987] hardirqs last disabled at (229663080): [] sysvec_apic_timer_interrupt+0xe/0x90 [38662.496733] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38662.498829] softirqs last enabled at (229663078): [] __do_softirq+0x2be/0x371 [38662.509073] do_raw_spin_lock+0xb5/0xc0 [38662.514376] softirqs last disabled at (229663073): [] irq_exit_rcu+0xb0/0xd0 [38662.524190] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38662.529058] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38662.537833] try_get_cap_refs+0x159/0x880 [ceph] [38662.541659] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38662.550261] __ceph_get_caps+0x4a1/0x760 [ceph] [38662.555040] Workqueue: events delayed_work [ceph] [38662.564423] ? __file_remove_privs+0x89/0x110 [38662.569029] [38662.569030] RIP: 0010:lock_release+0x1d2/0x260 [38662.576500] ? wake_bit_function+0x70/0x70 [38662.581025] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [38662.585725] ? ceph_write_iter+0x260/0x770 [ceph] [38662.590072] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [38662.591564] ceph_write_iter+0x260/0x770 [ceph] [38662.596001] [38662.596001] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [38662.600094] ? lock_is_held_type+0xaa/0x120 [38662.618828] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [38662.623530] vfs_write+0x324/0x4f0 [38662.628743] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [38662.633273] ksys_pwrite64+0x65/0xa0 [38662.634775] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [38662.641902] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38662.646077] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38662.653203] do_syscall_64+0x38/0x80 [38662.656600] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38662.663722] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38662.667294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38662.674419] RIP: 0033:0x7f48f0eeb93b [38662.679547] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38662.686672] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38662.690245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38662.698320] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38662.703364] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38662.709103] ORIG_RAX: 0000000000000012 [38662.712673] Call Trace: [38662.719797] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38662.738535] [38662.745660] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38662.750877] ? watchdog_timer_fn+0x1b4/0x220 [38662.758000] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38662.761834] ? __hrtimer_run_queues+0x1d3/0x360 [38662.764275] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38662.771404] ? hrtimer_interrupt+0x104/0x240 [38662.773410] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38662.780535] ? __local_bh_enable+0x50/0x60 [38662.784804] [38662.791925] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38662.826453] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38662.831410] [38662.833505] [38662.835605] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38662.840908] ? igrab+0x43/0x50 [38662.843962] ? igrab+0x43/0x50 [38662.847009] ? lock_release+0x1d2/0x260 [38662.850842] ? lock_release+0xc5/0x260 [38662.854587] _raw_spin_unlock+0x1b/0x40 [38662.858425] igrab+0x43/0x50 [38662.861302] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38662.866452] delayed_work+0x1d8/0x280 [ceph] [38662.870734] ? process_scheduled_works+0x22a/0x510 [38662.875524] process_scheduled_works+0x2a4/0x510 [38662.880139] worker_thread+0x188/0x340 [38662.883889] ? rescuer_thread+0x310/0x310 [38662.887890] kthread+0xf5/0x130 [38662.891029] ? kthread_complete_and_exit+0x20/0x20 [38662.895814] ret_from_fork+0x31/0x40 [38662.899393] ? kthread_complete_and_exit+0x20/0x20 [38662.904175] ret_from_fork_asm+0x11/0x20 [38662.908098] [38690.131556] watchdog: BUG: soft lockup - CPU#0 stuck for 24918s! [dbench:78683] [38690.138864] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38690.143555] watchdog: BUG: soft lockup - CPU#7 stuck for 33916s! [kworker/7:4:78808] [38690.194314] irq event stamp: 68423220 [38690.202043] Modules linked in: ceph libceph dns_resolver fscache netfs [38690.205700] hardirqs last enabled at (68423219): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38690.205705] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38690.222392] hardirqs last disabled at (68423220): [] sysvec_apic_timer_interrupt+0xe/0x90 [38690.222395] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [38690.238200] softirqs last enabled at (68423192): [] __do_softirq+0x2be/0x371 [38690.246279] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [38690.254962] softirqs last disabled at (68423187): [] irq_exit_rcu+0xb0/0xd0 [38690.263647] coretemp kvm_intel kvm irqbypass i2c_i801 [38690.272159] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38690.277289] iTCO_wdt iTCO_vendor_support i2c_smbus [38690.286234] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38690.286235] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38690.291103] crct10dif_pclmul joydev [38690.298575] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38690.304052] crc32_pclmul ipmi_ssif [38690.307622] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38690.326359] ghash_clmulni_intel [38690.329844] [38690.335061] mei_me mei [38690.338287] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38690.339775] lpc_ich sha512_ssse3 [38690.342236] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38690.349364] acpi_ipmi [38690.352673] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38690.359796] mfd_core [38690.362154] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38690.369278] wmi ipmi_si [38690.371550] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38690.378674] ipmi_devintf [38690.381204] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38690.388329] ipmi_msghandler [38690.390954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38690.399032] ioatdma acpi_pad [38690.401908] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38690.407645] acpi_power_meter ixgbe [38690.410610] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38690.417735] igb mdio [38690.421218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38690.428360] crc32c_intel nvme [38690.430630] Call Trace: [38690.430632] [38690.437756] ptp nvme_core [38690.440807] ? watchdog_timer_fn+0x1b4/0x220 [38690.443250] i2c_algo_bit pps_core [38690.445261] ? __hrtimer_run_queues+0x1d3/0x360 [38690.447963] dca fuse [38690.452236] ? hrtimer_interrupt+0x104/0x240 [38690.455634] [38690.460158] ? __local_bh_enable+0x50/0x60 [38690.462429] irq event stamp: 229884006 [38690.466697] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38690.468184] hardirqs last enabled at (229884005): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38690.472274] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38690.476019] hardirqs last disabled at (229884006): [] sysvec_apic_timer_interrupt+0xe/0x90 [38690.481243] [38690.491489] softirqs last enabled at (229884004): [] __do_softirq+0x2be/0x371 [38690.496447] [38690.496448] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38690.506256] softirqs last disabled at (229883999): [] irq_exit_rcu+0xb0/0xd0 [38690.508360] ? queued_spin_lock_slowpath+0x16/0x2e0 [38690.517126] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38690.519237] do_raw_spin_lock+0xb5/0xc0 [38690.524536] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38690.533136] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38690.538005] Workqueue: events delayed_work [ceph] [38690.547384] try_get_cap_refs+0x159/0x880 [ceph] [38690.551211] [38690.551212] RIP: 0010:ceph_check_delayed_caps+0x63/0x270 [ceph] [38690.558707] __ceph_get_caps+0x4a1/0x760 [ceph] [38690.563485] Code: 48 8d 85 88 04 00 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 <48> 29 d0 4c 01 e0 0f 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 [38690.568188] ? __file_remove_privs+0x89/0x110 [38690.572793] RSP: 0018:ffffc90002117db8 EFLAGS: 00000283 [38690.574284] ? wake_bit_function+0x70/0x70 [38690.580193] [38690.580194] RAX: 00000001001d239b RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [38690.584720] ? ceph_write_iter+0x260/0x770 [ceph] [38690.603456] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [38690.607807] ceph_write_iter+0x260/0x770 [ceph] [38690.613024] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [38690.617119] ? lock_is_held_type+0xaa/0x120 [38690.618605] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [38690.625732] vfs_write+0x324/0x4f0 [38690.630427] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [38690.637555] ksys_pwrite64+0x65/0xa0 [38690.642076] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38690.649200] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38690.653378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38690.660503] do_syscall_64+0x38/0x80 [38690.663898] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38690.671023] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38690.674594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38690.682671] RIP: 0033:0x7f48f0eeb93b [38690.687800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38690.693539] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38690.697111] Call Trace: [38690.697113] [38690.704246] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38690.709287] ? watchdog_timer_fn+0x1b4/0x220 [38690.716411] ORIG_RAX: 0000000000000012 [38690.719984] ? __hrtimer_run_queues+0x1d3/0x360 [38690.727105] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38690.745848] ? hrtimer_interrupt+0x104/0x240 [38690.748286] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38690.750298] ? __local_bh_enable+0x50/0x60 [38690.755515] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38690.759782] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38690.763609] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38690.768135] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38690.775258] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38690.779522] [38690.786653] [38690.790738] [38690.828664] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38690.833973] ? ceph_check_delayed_caps+0x63/0x270 [ceph] [38690.839291] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [38690.844604] delayed_work+0x1d8/0x280 [ceph] [38690.848886] ? process_scheduled_works+0x22a/0x510 [38690.853675] process_scheduled_works+0x2a4/0x510 [38690.858293] worker_thread+0x188/0x340 [38690.862043] ? rescuer_thread+0x310/0x310 [38690.866053] kthread+0xf5/0x130 [38690.869190] ? kthread_complete_and_exit+0x20/0x20 [38690.873974] ret_from_fork+0x31/0x40 [38690.877555] ? kthread_complete_and_exit+0x20/0x20 [38690.882339] ret_from_fork_asm+0x11/0x20 [38690.886269] [38718.132020] watchdog: BUG: soft lockup - CPU#0 stuck for 24945s! [dbench:78683] [38718.139325] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38718.144020] watchdog: BUG: soft lockup - CPU#7 stuck for 33942s! [kworker/7:4:78808] [38718.194766] irq event stamp: 68481364 [38718.194767] hardirqs last enabled at (68481363): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38718.202497] Modules linked in: [38718.206152] hardirqs last disabled at (68481364): [] sysvec_apic_timer_interrupt+0xe/0x90 [38718.216311] ceph [38718.219362] softirqs last enabled at (68481336): [] __do_softirq+0x2be/0x371 [38718.229087] libceph dns_resolver [38718.231011] softirqs last disabled at (68481331): [] irq_exit_rcu+0xb0/0xd0 [38718.239702] fscache netfs [38718.243015] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38718.251525] veth nft_chain_nat [38718.254228] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38718.263172] xt_MASQUERADE nf_nat [38718.266310] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38718.273780] nf_conntrack [38718.277092] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38718.282569] nf_defrag_ipv6 nf_defrag_ipv4 [38718.285187] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38718.303924] nft_compat nf_tables [38718.308014] [38718.308015] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38718.313231] nfnetlink bridge [38718.316543] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38718.318033] stp llc [38718.325159] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38718.328122] xfs libcrc32c [38718.335246] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38718.337430] sunrpc [38718.344555] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38718.347258] intel_rapl_msr [38718.354383] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38718.356480] intel_rapl_common [38718.363604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38718.366394] sb_edac x86_pkg_temp_thermal [38718.374490] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38718.377539] coretemp kvm_intel [38718.383279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38718.387282] kvm irqbypass [38718.394415] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38718.397553] i2c_i801 iTCO_wdt [38718.404676] Call Trace: [38718.407414] iTCO_vendor_support i2c_smbus [38718.414540] [38718.417588] crct10dif_pclmul joydev [38718.420033] ? watchdog_timer_fn+0x1b4/0x220 [38718.424125] crc32_pclmul ipmi_ssif [38718.426137] ? __hrtimer_run_queues+0x1d3/0x360 [38718.429706] ghash_clmulni_intel mei_me [38718.433972] ? hrtimer_interrupt+0x104/0x240 [38718.437454] mei lpc_ich [38718.441979] ? __local_bh_enable+0x50/0x60 [38718.445808] sha512_ssse3 acpi_ipmi [38718.450076] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38718.452604] mfd_core wmi [38718.456708] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38718.460195] ipmi_si ipmi_devintf [38718.465417] [38718.468031] ipmi_msghandler ioatdma [38718.472989] [38718.472991] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38718.476298] acpi_pad acpi_power_meter [38718.478418] ? queued_spin_lock_slowpath+0x18/0x2e0 [38718.481985] ixgbe igb [38718.484083] do_raw_spin_lock+0xb5/0xc0 [38718.489384] mdio crc32c_intel [38718.493131] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38718.498001] nvme ptp [38718.500359] try_get_cap_refs+0x159/0x880 [ceph] [38718.504187] nvme_core i2c_algo_bit [38718.507244] __ceph_get_caps+0x4a1/0x760 [ceph] [38718.512024] pps_core [38718.514299] ? __file_remove_privs+0x89/0x110 [38718.518904] dca fuse [38718.522389] ? wake_bit_function+0x70/0x70 [38718.526912] [38718.526913] irq event stamp: 230104976 [38718.529185] ? ceph_write_iter+0x260/0x770 [ceph] [38718.533534] hardirqs last enabled at (230104975): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38718.535806] ceph_write_iter+0x260/0x770 [ceph] [38718.539895] hardirqs last disabled at (230104976): [] sysvec_apic_timer_interrupt+0xe/0x90 [38718.541389] ? lock_is_held_type+0xaa/0x120 [38718.545129] softirqs last enabled at (230104974): [] __do_softirq+0x2be/0x371 [38718.549830] vfs_write+0x324/0x4f0 [38718.560071] softirqs last disabled at (230104969): [] irq_exit_rcu+0xb0/0xd0 [38718.564601] ksys_pwrite64+0x65/0xa0 [38718.574418] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38718.578592] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38718.587364] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38718.590764] do_syscall_64+0x38/0x80 [38718.599359] Workqueue: events delayed_work [ceph] [38718.602929] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38718.612307] [38718.612308] RIP: 0010:lock_acquire+0xf0/0x2b0 [38718.617439] RIP: 0033:0x7f48f0eeb93b [38718.624909] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38718.628478] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38718.633176] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38718.638221] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38718.639714] [38718.639714] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38718.644063] ORIG_RAX: 0000000000000012 [38718.647634] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38718.666370] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38718.685107] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38718.690326] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38718.695544] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38718.697033] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38718.704157] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38718.707989] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38718.715114] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38718.722238] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38718.729361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38718.736490] [38718.743610] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38718.795229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38718.802354] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38718.809478] Call Trace: [38718.811923] [38718.813934] ? watchdog_timer_fn+0x1b4/0x220 [38718.818197] ? __hrtimer_run_queues+0x1d3/0x360 [38718.822724] ? hrtimer_interrupt+0x104/0x240 [38718.826994] ? __local_bh_enable+0x50/0x60 [38718.831087] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38718.836310] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38718.841269] [38718.843365] [38718.845464] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38718.850770] ? lock_acquire+0xf0/0x2b0 [38718.854522] ? igrab+0x43/0x50 [38718.857580] ? lock_release+0xc5/0x260 [38718.861323] _raw_spin_lock+0x30/0x40 [38718.864981] ? igrab+0x1d/0x50 [38718.868032] igrab+0x1d/0x50 [38718.870909] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38718.876058] delayed_work+0x1d8/0x280 [ceph] [38718.880339] ? process_scheduled_works+0x22a/0x510 [38718.885129] process_scheduled_works+0x2a4/0x510 [38718.889745] worker_thread+0x188/0x340 [38718.893495] ? rescuer_thread+0x310/0x310 [38718.897499] kthread+0xf5/0x130 [38718.900637] ? kthread_complete_and_exit+0x20/0x20 [38718.905427] ret_from_fork+0x31/0x40 [38718.908998] ? kthread_complete_and_exit+0x20/0x20 [38718.913784] ret_from_fork_asm+0x11/0x20 [38718.917711] [38746.132424] watchdog: BUG: soft lockup - CPU#0 stuck for 24971s! [dbench:78683] [38746.139725] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38746.144424] watchdog: BUG: soft lockup - CPU#7 stuck for 33968s! [kworker/7:4:78808] [38746.195175] irq event stamp: 68539576 [38746.202905] Modules linked in: ceph libceph dns_resolver fscache netfs veth [38746.206562] hardirqs last enabled at (68539575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38746.206568] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [38746.223681] hardirqs last disabled at (68539576): [] sysvec_apic_timer_interrupt+0xe/0x90 [38746.232884] nf_tables nfnetlink bridge stp llc [38746.242608] softirqs last enabled at (68539548): [] __do_softirq+0x2be/0x371 [38746.247133] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [38746.255816] softirqs last disabled at (68539543): [] irq_exit_rcu+0xb0/0xd0 [38746.255820] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 [38746.271193] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38746.271196] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38746.278143] iTCO_wdt iTCO_vendor_support [38746.287098] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38746.294567] i2c_smbus crct10dif_pclmul [38746.298570] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38746.304048] joydev crc32_pclmul [38746.307878] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38746.326617] ipmi_ssif [38746.329840] [38746.335058] ghash_clmulni_intel mei_me [38746.337415] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38746.338906] mei lpc_ich [38746.342736] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38746.349861] sha512_ssse3 acpi_ipmi [38746.352391] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38746.359516] mfd_core [38746.362998] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38746.370124] wmi ipmi_si [38746.372393] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38746.379520] ipmi_devintf [38746.382049] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38746.389174] ipmi_msghandler [38746.391790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38746.399885] ioatdma acpi_pad [38746.402763] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38746.408502] acpi_power_meter [38746.411464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38746.418587] ixgbe igb [38746.421553] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38746.428677] mdio [38746.431034] Call Trace: [38746.431037] [38746.438158] crc32c_intel nvme [38746.440099] ? watchdog_timer_fn+0x1b4/0x220 [38746.442544] ptp nvme_core [38746.444555] ? __hrtimer_run_queues+0x1d3/0x360 [38746.447605] i2c_algo_bit pps_core [38746.451873] ? hrtimer_interrupt+0x104/0x240 [38746.454574] dca fuse [38746.459096] ? __local_bh_enable+0x50/0x60 [38746.462511] [38746.466777] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38746.469046] irq event stamp: 230325816 [38746.473137] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38746.474629] hardirqs last enabled at (230325815): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38746.479846] [38746.483589] hardirqs last disabled at (230325816): [] sysvec_apic_timer_interrupt+0xe/0x90 [38746.488547] [38746.498791] softirqs last enabled at (230325814): [] __do_softirq+0x2be/0x371 [38746.500908] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38746.510717] softirqs last disabled at (230325809): [] irq_exit_rcu+0xb0/0xd0 [38746.512821] ? queued_spin_lock_slowpath+0x16/0x2e0 [38746.521584] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38746.526892] do_raw_spin_lock+0xb5/0xc0 [38746.535485] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38746.540359] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38746.549736] Workqueue: events delayed_work [ceph] [38746.553568] try_get_cap_refs+0x159/0x880 [ceph] [38746.561036] [38746.561037] RIP: 0010:lock_release+0x14e/0x260 [38746.565827] __ceph_get_caps+0x4a1/0x760 [ceph] [38746.570517] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [38746.575134] ? __file_remove_privs+0x89/0x110 [38746.576619] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [38746.581058] ? wake_bit_function+0x70/0x70 [38746.585582] [38746.585582] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [38746.604322] ? ceph_write_iter+0x260/0x770 [ceph] [38746.608670] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [38746.613888] ceph_write_iter+0x260/0x770 [ceph] [38746.617978] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [38746.619474] ? lock_is_held_type+0xaa/0x120 [38746.626591] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [38746.631293] vfs_write+0x324/0x4f0 [38746.638415] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [38746.642944] ksys_pwrite64+0x65/0xa0 [38746.650061] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38746.654240] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38746.661363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38746.664763] do_syscall_64+0x38/0x80 [38746.671885] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38746.675456] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38746.683532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38746.688664] RIP: 0033:0x7f48f0eeb93b [38746.694400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38746.697974] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38746.705096] Call Trace: [38746.705097] [38746.710141] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38746.717265] ? watchdog_timer_fn+0x1b4/0x220 [38746.720834] ORIG_RAX: 0000000000000012 [38746.727961] ? __hrtimer_run_queues+0x1d3/0x360 [38746.746696] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38746.749145] ? hrtimer_interrupt+0x104/0x240 [38746.751151] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38746.756369] ? __local_bh_enable+0x50/0x60 [38746.760632] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38746.764467] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38746.768989] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38746.776114] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38746.780377] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38746.787500] [38746.791597] [38746.798715] [38746.798716] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38746.834823] ? igrab+0x43/0x50 [38746.837876] ? lock_release+0x14e/0x260 [38746.841714] ? lock_release+0xc5/0x260 [38746.845458] _raw_spin_unlock+0x1b/0x40 [38746.849288] igrab+0x43/0x50 [38746.852165] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38746.857315] delayed_work+0x1d8/0x280 [ceph] [38746.861595] ? process_scheduled_works+0x22a/0x510 [38746.866387] process_scheduled_works+0x2a4/0x510 [38746.871003] worker_thread+0x188/0x340 [38746.874750] ? rescuer_thread+0x310/0x310 [38746.878754] kthread+0xf5/0x130 [38746.881894] ? kthread_complete_and_exit+0x20/0x20 [38746.886684] ret_from_fork+0x31/0x40 [38746.890264] ? kthread_complete_and_exit+0x20/0x20 [38746.895048] ret_from_fork_asm+0x11/0x20 [38746.898981] [38773.596979] rcu: INFO: rcu_preempt self-detected stall on CPU [38773.602740] rcu: 7-....: (26056343 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/26054981 fqs=13136850 [38773.613416] rcu: (t=26836517 jiffies g=1309609 q=727413 ncpus=8) [38773.619503] Sending NMI from CPU 7 to CPUs 0: [38773.623854] NMI backtrace for cpu 0 [38773.623856] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38773.623858] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38773.623859] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [38773.623862] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [38773.623863] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38773.623865] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38773.623866] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38773.623867] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38773.623867] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38773.623868] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38773.623869] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38773.623870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38773.623871] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38773.623872] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38773.623873] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38773.623873] Call Trace: [38773.623874] [38773.623875] ? nmi_cpu_backtrace+0xad/0x120 [38773.623879] ? nmi_cpu_backtrace_handler+0x11/0x20 [38773.623880] ? nmi_handle+0xd4/0x260 [38773.623885] ? default_do_nmi+0x49/0xf0 [38773.623887] ? exc_nmi+0x11e/0x150 [38773.623888] ? end_repeat_nmi+0x16/0x67 [38773.623892] ? queued_spin_lock_slowpath+0x2b/0x2e0 [38773.623895] ? queued_spin_lock_slowpath+0x2b/0x2e0 [38773.623897] ? queued_spin_lock_slowpath+0x2b/0x2e0 [38773.623900] [38773.623900] [38773.623901] do_raw_spin_lock+0xb5/0xc0 [38773.623904] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38773.623918] try_get_cap_refs+0x159/0x880 [ceph] [38773.623933] __ceph_get_caps+0x4a1/0x760 [ceph] [38773.623946] ? __file_remove_privs+0x89/0x110 [38773.623948] ? wake_bit_function+0x70/0x70 [38773.623950] ? ceph_write_iter+0x260/0x770 [ceph] [38773.623961] ceph_write_iter+0x260/0x770 [ceph] [38773.623972] ? lock_is_held_type+0xaa/0x120 [38773.623975] vfs_write+0x324/0x4f0 [38773.623981] ksys_pwrite64+0x65/0xa0 [38773.623984] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38773.623987] do_syscall_64+0x38/0x80 [38773.623989] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38773.623991] RIP: 0033:0x7f48f0eeb93b [38773.623992] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38773.623994] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [38773.623995] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38773.623996] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38773.623997] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38773.623997] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38773.623998] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38773.624001] [38773.624853] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38773.928351] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38773.935823] Workqueue: events delayed_work [ceph] [38773.940538] RIP: 0010:lock_acquire+0xf0/0x2b0 [38773.944888] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38773.963625] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38773.968843] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38773.975968] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38773.983091] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38773.990216] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38773.997339] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38774.004464] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38774.012542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38774.018278] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38774.025404] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38774.032527] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38774.039650] Call Trace: [38774.042096] [38774.044106] ? rcu_dump_cpu_stacks+0xd1/0x180 [38774.048467] ? rcu_sched_clock_irq+0x633/0x1390 [38774.053000] ? lock_release+0xc5/0x260 [38774.056744] ? lock_is_held_type+0xaa/0x120 [38774.060921] ? lock_is_held_type+0xaa/0x120 [38774.065099] ? tick_sched_do_timer+0xa0/0xa0 [38774.069369] ? rcu_read_lock_sched_held+0x56/0x90 [38774.074070] ? tick_sched_do_timer+0xa0/0xa0 [38774.078341] ? update_process_times+0x5f/0x90 [38774.082691] ? tick_sched_handle.isra.16+0x1e/0x50 [38774.087474] ? tick_sched_timer+0x6f/0x80 [38774.091480] ? __hrtimer_run_queues+0x1d3/0x360 [38774.096008] ? hrtimer_interrupt+0x104/0x240 [38774.100275] ? __local_bh_enable+0x50/0x60 [38774.104368] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38774.109593] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38774.114550] [38774.116647] [38774.118746] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38774.124054] ? lock_acquire+0xf0/0x2b0 [38774.127804] ? igrab+0x43/0x50 [38774.130853] ? lock_release+0xc5/0x260 [38774.134598] _raw_spin_lock+0x30/0x40 [38774.138256] ? igrab+0x1d/0x50 [38774.141305] igrab+0x1d/0x50 [38774.144184] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38774.149332] delayed_work+0x1d8/0x280 [ceph] [38774.153613] ? process_scheduled_works+0x22a/0x510 [38774.158404] process_scheduled_works+0x2a4/0x510 [38774.163020] worker_thread+0x188/0x340 [38774.166768] ? rescuer_thread+0x310/0x310 [38774.170773] kthread+0xf5/0x130 [38774.173910] ? kthread_complete_and_exit+0x20/0x20 [38774.178694] ret_from_fork+0x31/0x40 [38774.182273] ? kthread_complete_and_exit+0x20/0x20 [38774.187056] ret_from_fork_asm+0x11/0x20 [38774.190980] [38798.133024] watchdog: BUG: soft lockup - CPU#0 stuck for 25019s! [dbench:78683] [38798.140327] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38798.145024] watchdog: BUG: soft lockup - CPU#7 stuck for 34016s! [kworker/7:4:78808] [38798.195779] irq event stamp: 68648702 [38798.203509] Modules linked in: ceph libceph dns_resolver fscache netfs [38798.207167] hardirqs last enabled at (68648701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38798.213699] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [38798.223858] hardirqs last disabled at (68648702): [] sysvec_apic_timer_interrupt+0xe/0x90 [38798.231243] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [38798.240966] softirqs last enabled at (68648650): [] __do_softirq+0x2be/0x371 [38798.247745] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [38798.256428] softirqs last disabled at (68648645): [] irq_exit_rcu+0xb0/0xd0 [38798.262599] sb_edac x86_pkg_temp_thermal coretemp [38798.271109] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38798.271110] kvm_intel [38798.271111] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38798.275894] kvm irqbypass [38798.284839] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38798.287196] i2c_i801 iTCO_wdt [38798.294666] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38798.297369] iTCO_vendor_support [38798.302848] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38798.305899] i2c_smbus [38798.324635] [38798.327860] crct10dif_pclmul joydev [38798.333078] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38798.335433] crc32_pclmul [38798.336925] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38798.340496] ipmi_ssif ghash_clmulni_intel [38798.347620] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38798.350238] mei_me [38798.357361] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38798.361453] mei lpc_ich [38798.368575] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38798.370674] sha512_ssse3 acpi_ipmi [38798.377799] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38798.380328] mfd_core wmi [38798.387451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38798.390938] ipmi_si [38798.399015] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38798.401632] ipmi_devintf ipmi_msghandler [38798.407370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38798.409553] ioatdma acpi_pad [38798.416676] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38798.420680] acpi_power_meter ixgbe [38798.427805] Call Trace: [38798.427808] [38798.430769] igb mdio [38798.437897] ? watchdog_timer_fn+0x1b4/0x220 [38798.441378] crc32c_intel nvme [38798.443824] ? __hrtimer_run_queues+0x1d3/0x360 [38798.445832] ptp nvme_core [38798.448107] ? hrtimer_interrupt+0x104/0x240 [38798.452367] i2c_algo_bit pps_core [38798.455422] ? __local_bh_enable+0x50/0x60 [38798.459952] dca fuse [38798.462659] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38798.466920] [38798.466921] irq event stamp: 230736576 [38798.470318] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38798.474423] hardirqs last enabled at (230736575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38798.476710] [38798.481931] hardirqs last disabled at (230736576): [] sysvec_apic_timer_interrupt+0xe/0x90 [38798.483423] [38798.487164] softirqs last enabled at (230736574): [] __do_softirq+0x2be/0x371 [38798.492122] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38798.502366] softirqs last disabled at (230736569): [] irq_exit_rcu+0xb0/0xd0 [38798.504469] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38798.514274] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38798.516374] do_raw_spin_lock+0xb5/0xc0 [38798.525143] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38798.530450] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38798.539046] Workqueue: events delayed_work [ceph] [38798.543918] try_get_cap_refs+0x159/0x880 [ceph] [38798.553292] [38798.553293] RIP: 0010:lock_acquire+0xf0/0x2b0 [38798.557129] __ceph_get_caps+0x4a1/0x760 [ceph] [38798.564596] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38798.569384] ? __file_remove_privs+0x89/0x110 [38798.574076] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38798.578703] ? wake_bit_function+0x70/0x70 [38798.580195] [38798.580196] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38798.584547] ? ceph_write_iter+0x260/0x770 [ceph] [38798.589069] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38798.607808] ceph_write_iter+0x260/0x770 [ceph] [38798.612157] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38798.617381] ? lock_is_held_type+0xaa/0x120 [38798.621465] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38798.622960] vfs_write+0x324/0x4f0 [38798.630080] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38798.634782] ksys_pwrite64+0x65/0xa0 [38798.641903] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38798.646428] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38798.653550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38798.657728] do_syscall_64+0x38/0x80 [38798.664852] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38798.668252] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38798.675373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38798.678944] RIP: 0033:0x7f48f0eeb93b [38798.687022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38798.692152] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38798.697891] Call Trace: [38798.697892] [38798.701461] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38798.708585] ? watchdog_timer_fn+0x1b4/0x220 [38798.713628] ORIG_RAX: 0000000000000012 [38798.720755] ? __hrtimer_run_queues+0x1d3/0x360 [38798.724325] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38798.731451] ? hrtimer_interrupt+0x104/0x240 [38798.750185] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38798.752629] ? __local_bh_enable+0x50/0x60 [38798.754640] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38798.759860] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38798.764121] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38798.767952] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38798.772476] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38798.779600] [38798.783869] [38798.790989] [38798.790991] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38798.838323] ? lock_acquire+0xf0/0x2b0 [38798.842073] ? igrab+0x43/0x50 [38798.845131] ? lock_release+0xc5/0x260 [38798.848877] _raw_spin_lock+0x30/0x40 [38798.852541] ? igrab+0x1d/0x50 [38798.855593] igrab+0x1d/0x50 [38798.858472] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38798.863621] delayed_work+0x1d8/0x280 [ceph] [38798.867903] ? process_scheduled_works+0x22a/0x510 [38798.872707] process_scheduled_works+0x2a4/0x510 [38798.877324] worker_thread+0x188/0x340 [38798.881073] ? rescuer_thread+0x310/0x310 [38798.885075] kthread+0xf5/0x130 [38798.888216] ? kthread_complete_and_exit+0x20/0x20 [38798.893007] ret_from_fork+0x31/0x40 [38798.896588] ? kthread_complete_and_exit+0x20/0x20 [38798.901379] ret_from_fork_asm+0x11/0x20 [38798.905313] [38826.133498] watchdog: BUG: soft lockup - CPU#0 stuck for 25045s! [dbench:78683] [38826.140798] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38826.145498] watchdog: BUG: soft lockup - CPU#7 stuck for 34042s! [kworker/7:4:78808] [38826.196240] irq event stamp: 68706776 [38826.196241] hardirqs last enabled at (68706775): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38826.203969] Modules linked in: [38826.207627] hardirqs last disabled at (68706776): [] sysvec_apic_timer_interrupt+0xe/0x90 [38826.217783] ceph [38826.220835] softirqs last enabled at (68706698): [] __do_softirq+0x2be/0x371 [38826.230560] libceph [38826.232484] softirqs last disabled at (68706693): [] irq_exit_rcu+0xb0/0xd0 [38826.241174] dns_resolver fscache [38826.243361] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38826.251872] netfs [38826.255199] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38826.264144] veth nft_chain_nat [38826.266170] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38826.273643] xt_MASQUERADE [38826.276779] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38826.282257] nf_nat [38826.284961] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38826.303698] nf_conntrack nf_defrag_ipv6 [38826.305796] [38826.311012] nf_defrag_ipv4 nft_compat [38826.314931] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38826.316422] nf_tables nfnetlink [38826.320164] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38826.327291] bridge [38826.330513] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38826.337638] stp llc [38826.339735] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38826.346860] xfs [38826.349045] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38826.356175] libcrc32c [38826.358015] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38826.365137] sunrpc intel_rapl_msr [38826.367494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38826.375574] intel_rapl_common [38826.378969] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38826.384707] sb_edac x86_pkg_temp_thermal [38826.387758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38826.394897] coretemp kvm_intel [38826.398903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38826.406027] kvm irqbypass [38826.409177] Call Trace: [38826.409179] [38826.416299] i2c_i801 iTCO_wdt [38826.419004] ? watchdog_timer_fn+0x1b4/0x220 [38826.421445] iTCO_vendor_support i2c_smbus [38826.423459] ? __hrtimer_run_queues+0x1d3/0x360 [38826.426507] crct10dif_pclmul joydev [38826.430774] ? hrtimer_interrupt+0x104/0x240 [38826.434862] crc32_pclmul ipmi_ssif [38826.439387] ? __local_bh_enable+0x50/0x60 [38826.442959] ghash_clmulni_intel mei_me [38826.447224] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38826.450706] mei [38826.454798] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38826.458627] lpc_ich sha512_ssse3 [38826.463846] [38826.465681] acpi_ipmi mfd_core [38826.470638] [38826.473949] wmi ipmi_si [38826.476048] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38826.479185] ipmi_devintf [38826.481285] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38826.483812] ipmi_msghandler ioatdma [38826.489119] do_raw_spin_lock+0xb5/0xc0 [38826.491734] acpi_pad acpi_power_meter ixgbe [38826.496607] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38826.500176] igb mdio crc32c_intel [38826.504027] try_get_cap_refs+0x159/0x880 [ceph] [38826.508288] nvme ptp [38826.513078] __ceph_get_caps+0x4a1/0x760 [ceph] [38826.516468] nvme_core i2c_algo_bit [38826.521083] ? __file_remove_privs+0x89/0x110 [38826.523352] pps_core dca [38826.527875] ? wake_bit_function+0x70/0x70 [38826.531376] fuse [38826.535729] ? ceph_write_iter+0x260/0x770 [ceph] [38826.538343] irq event stamp: 230957398 [38826.542436] ceph_write_iter+0x260/0x770 [ceph] [38826.544359] hardirqs last enabled at (230957397): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38826.549061] ? lock_is_held_type+0xaa/0x120 [38826.552801] hardirqs last disabled at (230957398): [] sysvec_apic_timer_interrupt+0xe/0x90 [38826.557327] vfs_write+0x324/0x4f0 [38826.567569] softirqs last enabled at (230957396): [] __do_softirq+0x2be/0x371 [38826.571751] ksys_pwrite64+0x65/0xa0 [38826.581557] softirqs last disabled at (230957391): [] irq_exit_rcu+0xb0/0xd0 [38826.584954] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38826.593725] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38826.597298] do_syscall_64+0x38/0x80 [38826.605899] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38826.611034] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38826.620410] Workqueue: events delayed_work [ceph] [38826.623982] RIP: 0033:0x7f48f0eeb93b [38826.631453] [38826.636496] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38826.641194] RIP: 0010:lock_acquire+0xf0/0x2b0 [38826.644764] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38826.646254] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38826.664993] ORIG_RAX: 0000000000000012 [38826.669343] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38826.674560] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38826.693300] [38826.693301] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38826.697129] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38826.702346] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38826.709472] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38826.710962] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38826.718087] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38826.725209] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38826.732334] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38826.739457] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38826.746589] [38826.753707] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38826.785341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38826.791078] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38826.798201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38826.805327] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38826.812449] Call Trace: [38826.814903] [38826.816914] ? watchdog_timer_fn+0x1b4/0x220 [38826.821184] ? __hrtimer_run_queues+0x1d3/0x360 [38826.825715] ? hrtimer_interrupt+0x104/0x240 [38826.829983] ? __local_bh_enable+0x50/0x60 [38826.834076] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38826.839302] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38826.844258] [38826.846355] [38826.848452] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38826.853761] ? lock_acquire+0xf0/0x2b0 [38826.857510] ? igrab+0x43/0x50 [38826.860560] ? lock_release+0xc5/0x260 [38826.864307] _raw_spin_lock+0x30/0x40 [38826.867970] ? igrab+0x1d/0x50 [38826.871022] igrab+0x1d/0x50 [38826.873903] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38826.879050] delayed_work+0x1d8/0x280 [ceph] [38826.883331] ? process_scheduled_works+0x22a/0x510 [38826.888121] process_scheduled_works+0x2a4/0x510 [38826.892736] worker_thread+0x188/0x340 [38826.896485] ? rescuer_thread+0x310/0x310 [38826.900488] kthread+0xf5/0x130 [38826.903626] ? kthread_complete_and_exit+0x20/0x20 [38826.908410] ret_from_fork+0x31/0x40 [38826.911980] ? kthread_complete_and_exit+0x20/0x20 [38826.916765] ret_from_fork_asm+0x11/0x20 [38826.920689] [38854.133976] watchdog: BUG: soft lockup - CPU#0 stuck for 25071s! [dbench:78683] [38854.141278] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38854.145974] watchdog: BUG: soft lockup - CPU#7 stuck for 34068s! [kworker/7:4:78808] [38854.196727] irq event stamp: 68764998 [38854.204457] Modules linked in: ceph libceph dns_resolver fscache netfs veth [38854.208116] hardirqs last enabled at (68764997): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38854.208121] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [38854.225231] hardirqs last disabled at (68764998): [] sysvec_apic_timer_interrupt+0xe/0x90 [38854.234437] nf_tables nfnetlink bridge stp llc [38854.244161] softirqs last enabled at (68764942): [] __do_softirq+0x2be/0x371 [38854.248685] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [38854.257371] softirqs last disabled at (68764937): [] irq_exit_rcu+0xb0/0xd0 [38854.266061] coretemp kvm_intel kvm irqbypass i2c_i801 [38854.274574] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38854.274575] iTCO_wdt [38854.274577] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38854.279702] iTCO_vendor_support [38854.288649] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38854.290917] i2c_smbus crct10dif_pclmul [38854.298389] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38854.301612] joydev crc32_pclmul [38854.307090] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38854.310921] ipmi_ssif [38854.329660] [38854.332883] ghash_clmulni_intel [38854.338099] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38854.340458] mei_me mei [38854.341947] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38854.345172] lpc_ich sha512_ssse3 [38854.352296] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38854.354740] acpi_ipmi [38854.361863] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38854.365175] mfd_core wmi [38854.372300] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38854.374656] ipmi_si [38854.381781] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38854.384398] ipmi_devintf [38854.391521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38854.393707] ipmi_msghandler ioatdma [38854.401782] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38854.404400] acpi_pad acpi_power_meter [38854.410138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38854.413708] ixgbe [38854.420834] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38854.424578] igb mdio [38854.431700] Call Trace: [38854.431703] [38854.433713] crc32c_intel nvme [38854.440839] ? watchdog_timer_fn+0x1b4/0x220 [38854.443106] ptp nvme_core [38854.445553] ? __hrtimer_run_queues+0x1d3/0x360 [38854.447563] i2c_algo_bit pps_core [38854.450615] ? hrtimer_interrupt+0x104/0x240 [38854.454876] dca fuse [38854.457581] ? __local_bh_enable+0x50/0x60 [38854.462104] [38854.462105] irq event stamp: 231178086 [38854.465504] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38854.469766] hardirqs last enabled at (231178085): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38854.472039] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38854.476127] hardirqs last disabled at (231178086): [] sysvec_apic_timer_interrupt+0xe/0x90 [38854.477620] [38854.481375] softirqs last enabled at (231178084): [] __do_softirq+0x2be/0x371 [38854.486589] [38854.496833] softirqs last disabled at (231178079): [] irq_exit_rcu+0xb0/0xd0 [38854.501792] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38854.511601] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38854.513702] ? queued_spin_lock_slowpath+0x18/0x2e0 [38854.522469] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38854.524569] do_raw_spin_lock+0xb5/0xc0 [38854.533164] Workqueue: events delayed_work [ceph] [38854.538469] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38854.547844] [38854.552718] try_get_cap_refs+0x159/0x880 [ceph] [38854.560188] RIP: 0010:lock_acquire+0xf0/0x2b0 [38854.564024] __ceph_get_caps+0x4a1/0x760 [ceph] [38854.568716] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38854.573506] ? __file_remove_privs+0x89/0x110 [38854.574991] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38854.579602] ? wake_bit_function+0x70/0x70 [38854.583952] [38854.583953] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38854.588479] ? ceph_write_iter+0x260/0x770 [ceph] [38854.607213] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38854.611564] ceph_write_iter+0x260/0x770 [ceph] [38854.616781] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38854.620876] ? lock_is_held_type+0xaa/0x120 [38854.622377] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38854.629508] vfs_write+0x324/0x4f0 [38854.634202] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38854.641331] ksys_pwrite64+0x65/0xa0 [38854.645851] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38854.652974] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38854.657150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38854.664276] do_syscall_64+0x38/0x80 [38854.667674] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38854.674799] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38854.678375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38854.686456] RIP: 0033:0x7f48f0eeb93b [38854.691586] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38854.697323] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38854.700893] Call Trace: [38854.708017] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38854.713062] [38854.720184] ORIG_RAX: 0000000000000012 [38854.723756] ? watchdog_timer_fn+0x1b4/0x220 [38854.730880] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38854.749619] ? __hrtimer_run_queues+0x1d3/0x360 [38854.752062] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38854.757284] ? hrtimer_interrupt+0x104/0x240 [38854.759290] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38854.763122] ? __local_bh_enable+0x50/0x60 [38854.767386] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38854.774513] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38854.779033] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38854.786158] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38854.790428] [38854.797545] [38854.830378] [38854.832474] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38854.837781] ? lock_acquire+0xf0/0x2b0 [38854.841531] ? igrab+0x43/0x50 [38854.844582] ? lock_release+0xc5/0x260 [38854.848326] _raw_spin_lock+0x30/0x40 [38854.851982] ? igrab+0x1d/0x50 [38854.855034] igrab+0x1d/0x50 [38854.857913] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38854.863063] delayed_work+0x1d8/0x280 [ceph] [38854.867342] ? process_scheduled_works+0x22a/0x510 [38854.872133] process_scheduled_works+0x2a4/0x510 [38854.876750] worker_thread+0x188/0x340 [38854.880497] ? rescuer_thread+0x310/0x310 [38854.884500] kthread+0xf5/0x130 [38854.887655] ? kthread_complete_and_exit+0x20/0x20 [38854.892440] ret_from_fork+0x31/0x40 [38854.896019] ? kthread_complete_and_exit+0x20/0x20 [38854.900804] ret_from_fork_asm+0x11/0x20 [38854.904734] [38882.134453] watchdog: BUG: soft lockup - CPU#0 stuck for 25097s! [dbench:78683] [38882.141755] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38882.146453] watchdog: BUG: soft lockup - CPU#7 stuck for 34094s! [kworker/7:4:78808] [38882.197197] irq event stamp: 68823094 [38882.197198] hardirqs last enabled at (68823093): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38882.204929] Modules linked in: [38882.208584] hardirqs last disabled at (68823094): [] sysvec_apic_timer_interrupt+0xe/0x90 [38882.218742] ceph [38882.221793] softirqs last enabled at (68823024): [] __do_softirq+0x2be/0x371 [38882.231517] libceph [38882.233441] softirqs last disabled at (68823019): [] irq_exit_rcu+0xb0/0xd0 [38882.242130] dns_resolver [38882.244309] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38882.252820] fscache netfs [38882.255437] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38882.264381] veth nft_chain_nat [38882.267085] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38882.274558] xt_MASQUERADE nf_nat [38882.277694] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38882.283172] nf_conntrack nf_defrag_ipv6 [38882.286481] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38882.305221] nf_defrag_ipv4 nft_compat [38882.309137] [38882.314355] nf_tables nfnetlink [38882.318099] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38882.319590] bridge [38882.322814] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38882.329938] stp llc [38882.332035] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38882.339159] xfs libcrc32c [38882.341342] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38882.348468] sunrpc [38882.351171] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38882.358297] intel_rapl_msr [38882.360394] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38882.367517] intel_rapl_common sb_edac [38882.370308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38882.378384] x86_pkg_temp_thermal coretemp [38882.382140] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38882.387876] kvm_intel [38882.391966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38882.399089] kvm irqbypass [38882.401448] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38882.408572] i2c_i801 iTCO_wdt [38882.411275] Call Trace: [38882.411276] [38882.418400] iTCO_vendor_support i2c_smbus [38882.421452] ? watchdog_timer_fn+0x1b4/0x220 [38882.423894] crct10dif_pclmul joydev [38882.425907] ? __hrtimer_run_queues+0x1d3/0x360 [38882.429996] crc32_pclmul ipmi_ssif [38882.434263] ? hrtimer_interrupt+0x104/0x240 [38882.437847] ghash_clmulni_intel [38882.442373] ? __local_bh_enable+0x50/0x60 [38882.445857] mei_me mei [38882.450135] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38882.453352] lpc_ich sha512_ssse3 [38882.457444] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38882.459889] acpi_ipmi mfd_core [38882.465107] [38882.468434] wmi [38882.473391] [38882.476530] ipmi_si ipmi_devintf [38882.478627] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38882.480464] ipmi_msghandler ioatdma [38882.482566] ? queued_spin_lock_slowpath+0x18/0x2e0 [38882.485873] acpi_pad acpi_power_meter [38882.491178] do_raw_spin_lock+0xb5/0xc0 [38882.494746] ixgbe igb [38882.499617] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38882.503360] mdio crc32c_intel [38882.507194] try_get_cap_refs+0x159/0x880 [ceph] [38882.509549] nvme ptp [38882.514340] __ceph_get_caps+0x4a1/0x760 [ceph] [38882.517385] nvme_core i2c_algo_bit [38882.521998] ? __file_remove_privs+0x89/0x110 [38882.524265] pps_core [38882.528790] ? wake_bit_function+0x70/0x70 [38882.532275] dca fuse [38882.536627] ? ceph_write_iter+0x260/0x770 [ceph] [38882.538895] [38882.538896] irq event stamp: 231399062 [38882.542987] ceph_write_iter+0x260/0x770 [ceph] [38882.545255] hardirqs last enabled at (231399061): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38882.549959] ? lock_is_held_type+0xaa/0x120 [38882.551444] hardirqs last disabled at (231399062): [] sysvec_apic_timer_interrupt+0xe/0x90 [38882.555191] vfs_write+0x324/0x4f0 [38882.559714] softirqs last enabled at (231399060): [] __do_softirq+0x2be/0x371 [38882.569962] ksys_pwrite64+0x65/0xa0 [38882.574134] softirqs last disabled at (231399055): [] irq_exit_rcu+0xb0/0xd0 [38882.583944] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38882.587342] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38882.596115] do_syscall_64+0x38/0x80 [38882.599684] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38882.608282] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38882.613412] Workqueue: events delayed_work [ceph] [38882.622791] RIP: 0033:0x7f48f0eeb93b [38882.626361] [38882.626362] RIP: 0010:lock_acquire+0xf0/0x2b0 [38882.633832] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38882.638894] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38882.643590] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38882.647161] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38882.648651] ORIG_RAX: 0000000000000012 [38882.653001] [38882.653002] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38882.671740] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38882.690478] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38882.695695] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38882.700913] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38882.704743] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38882.706234] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38882.713359] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38882.720483] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38882.727608] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38882.734730] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38882.741861] [38882.748979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38882.793475] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38882.800599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38882.807724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38882.814858] Call Trace: [38882.817300] [38882.819311] ? watchdog_timer_fn+0x1b4/0x220 [38882.823575] ? __hrtimer_run_queues+0x1d3/0x360 [38882.828104] ? hrtimer_interrupt+0x104/0x240 [38882.832371] ? __local_bh_enable+0x50/0x60 [38882.836466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38882.841689] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38882.846647] [38882.848742] [38882.850857] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38882.856167] ? lock_acquire+0xf0/0x2b0 [38882.859915] ? igrab+0x43/0x50 [38882.862965] ? lock_release+0xc5/0x260 [38882.866711] _raw_spin_lock+0x30/0x40 [38882.870366] ? igrab+0x1d/0x50 [38882.873419] igrab+0x1d/0x50 [38882.876295] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38882.881445] delayed_work+0x1d8/0x280 [ceph] [38882.885728] ? process_scheduled_works+0x22a/0x510 [38882.890517] process_scheduled_works+0x2a4/0x510 [38882.895143] worker_thread+0x188/0x340 [38882.898891] ? rescuer_thread+0x310/0x310 [38882.902893] kthread+0xf5/0x130 [38882.906031] ? kthread_complete_and_exit+0x20/0x20 [38882.910817] ret_from_fork+0x31/0x40 [38882.914394] ? kthread_complete_and_exit+0x20/0x20 [38882.919178] ret_from_fork_asm+0x11/0x20 [38882.923103] [38910.134935] watchdog: BUG: soft lockup - CPU#0 stuck for 25123s! [dbench:78683] [38910.142235] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38910.146935] watchdog: BUG: soft lockup - CPU#7 stuck for 34120s! [kworker/7:4:78808] [38910.197677] irq event stamp: 68881310 [38910.205408] Modules linked in: ceph libceph dns_resolver [38910.209063] hardirqs last enabled at (68881309): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38910.214368] fscache netfs veth [38910.224527] hardirqs last disabled at (68881310): [] sysvec_apic_timer_interrupt+0xe/0x90 [38910.227664] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38910.237388] softirqs last enabled at (68881244): [] __do_softirq+0x2be/0x371 [38910.243037] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [38910.251723] softirqs last disabled at (68881239): [] irq_exit_rcu+0xb0/0xd0 [38910.251725] nf_tables [38910.256766] nfnetlink bridge [38910.265277] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38910.267634] stp llc [38910.270609] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38910.279550] xfs libcrc32c [38910.281735] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [38910.289207] sunrpc [38910.291910] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [38910.297387] intel_rapl_msr intel_rapl_common [38910.299485] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38910.318223] sb_edac x86_pkg_temp_thermal [38910.322574] [38910.322575] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38910.327790] coretemp kvm_intel [38910.331795] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38910.333286] kvm irqbypass [38910.340410] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38910.343546] i2c_i801 [38910.350671] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38910.353374] iTCO_wdt iTCO_vendor_support [38910.360500] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38910.362770] i2c_smbus [38910.369894] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38910.373897] crct10dif_pclmul joydev [38910.381023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38910.383379] crc32_pclmul ipmi_ssif [38910.391458] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38910.395027] ghash_clmulni_intel mei_me [38910.400767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38910.404249] mei lpc_ich [38910.411374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38910.415205] sha512_ssse3 acpi_ipmi [38910.422333] Call Trace: [38910.422334] [38910.424859] mfd_core wmi [38910.431984] ? watchdog_timer_fn+0x1b4/0x220 [38910.435468] ipmi_si ipmi_devintf [38910.437913] ? __hrtimer_run_queues+0x1d3/0x360 [38910.439923] ipmi_msghandler ioatdma [38910.442542] ? hrtimer_interrupt+0x104/0x240 [38910.446805] acpi_pad acpi_power_meter [38910.450115] ? __local_bh_enable+0x50/0x60 [38910.454639] ixgbe igb [38910.458212] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38910.462474] mdio crc32c_intel [38910.466219] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38910.470307] nvme ptp [38910.472684] [38910.477900] nvme_core i2c_algo_bit [38910.480951] [38910.480953] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38910.485909] pps_core dca [38910.488182] ? queued_spin_lock_slowpath+0x16/0x2e0 [38910.490278] fuse [38910.493763] do_raw_spin_lock+0xb5/0xc0 [38910.495858] irq event stamp: 231619632 [38910.501164] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38910.503781] hardirqs last enabled at (231619631): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38910.508651] try_get_cap_refs+0x159/0x880 [ceph] [38910.510575] hardirqs last disabled at (231619632): [] sysvec_apic_timer_interrupt+0xe/0x90 [38910.514409] __ceph_get_caps+0x4a1/0x760 [ceph] [38910.518150] softirqs last enabled at (231619630): [] __do_softirq+0x2be/0x371 [38910.522937] ? __file_remove_privs+0x89/0x110 [38910.533177] softirqs last disabled at (231619625): [] irq_exit_rcu+0xb0/0xd0 [38910.537789] ? wake_bit_function+0x70/0x70 [38910.547600] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38910.552143] ? ceph_write_iter+0x260/0x770 [ceph] [38910.560912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38910.565262] ceph_write_iter+0x260/0x770 [ceph] [38910.573860] Workqueue: events delayed_work [ceph] [38910.577954] ? lock_is_held_type+0xaa/0x120 [38910.587335] [38910.587336] RIP: 0010:lock_acquire+0xf0/0x2b0 [38910.592035] vfs_write+0x324/0x4f0 [38910.599505] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38910.604032] ksys_pwrite64+0x65/0xa0 [38910.608727] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38910.612904] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38910.614393] [38910.614394] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38910.618746] do_syscall_64+0x38/0x80 [38910.622144] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38910.640881] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38910.644450] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38910.649668] RIP: 0033:0x7f48f0eeb93b [38910.654799] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38910.656290] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38910.663415] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38910.666986] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38910.674108] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38910.679152] ORIG_RAX: 0000000000000012 [38910.686276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38910.689847] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38910.696971] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38910.715709] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38910.722832] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38910.728052] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38910.736129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38910.739958] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38910.745698] Call Trace: [38910.745699] [38910.752821] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38910.759944] ? watchdog_timer_fn+0x1b4/0x220 [38910.767075] [38910.774193] ? __hrtimer_run_queues+0x1d3/0x360 [38910.818146] ? hrtimer_interrupt+0x104/0x240 [38910.822415] ? __local_bh_enable+0x50/0x60 [38910.826508] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38910.831724] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38910.836681] [38910.838778] [38910.840878] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38910.846184] ? lock_acquire+0xf0/0x2b0 [38910.849934] ? igrab+0x43/0x50 [38910.852984] ? lock_release+0xc5/0x260 [38910.856729] _raw_spin_lock+0x30/0x40 [38910.860384] ? igrab+0x1d/0x50 [38910.863435] igrab+0x1d/0x50 [38910.866316] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38910.871480] delayed_work+0x1d8/0x280 [ceph] [38910.875763] ? process_scheduled_works+0x22a/0x510 [38910.880552] process_scheduled_works+0x2a4/0x510 [38910.885167] worker_thread+0x188/0x340 [38910.888917] ? rescuer_thread+0x310/0x310 [38910.892920] kthread+0xf5/0x130 [38910.896057] ? kthread_complete_and_exit+0x20/0x20 [38910.900842] ret_from_fork+0x31/0x40 [38910.904412] ? kthread_complete_and_exit+0x20/0x20 [38910.909196] ret_from_fork_asm+0x11/0x20 [38910.913119] [38938.135417] watchdog: BUG: soft lockup - CPU#0 stuck for 25149s! [dbench:78683] [38938.142724] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38938.147417] watchdog: BUG: soft lockup - CPU#7 stuck for 34146s! [kworker/7:4:78808] [38938.198173] irq event stamp: 68939416 [38938.205904] Modules linked in: ceph libceph dns_resolver fscache netfs [38938.209559] hardirqs last enabled at (68939415): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38938.216093] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38938.226252] hardirqs last disabled at (68939416): [] sysvec_apic_timer_interrupt+0xe/0x90 [38938.232336] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [38938.242061] softirqs last enabled at (68939400): [] __do_softirq+0x2be/0x371 [38938.249808] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [38938.258493] softirqs last disabled at (68939395): [] irq_exit_rcu+0xb0/0xd0 [38938.265011] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [38938.273523] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38938.279518] irqbypass [38938.288463] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38938.288463] i2c_i801 iTCO_wdt [38938.290820] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38938.298292] iTCO_vendor_support i2c_smbus [38938.301342] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38938.306818] crct10dif_pclmul joydev [38938.310911] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38938.329649] crc32_pclmul ipmi_ssif [38938.333219] [38938.338435] ghash_clmulni_intel [38938.341919] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38938.343410] mei_me mei [38938.346635] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38938.353759] lpc_ich [38938.356203] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38938.363327] sha512_ssse3 acpi_ipmi [38938.365511] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38938.372635] mfd_core wmi [38938.376120] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38938.383244] ipmi_si ipmi_devintf [38938.385862] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38938.392985] ipmi_msghandler ioatdma [38938.396295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38938.404372] acpi_pad acpi_power_meter [38938.407944] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38938.413682] ixgbe igb mdio [38938.417426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38938.424550] crc32c_intel nvme [38938.427340] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38938.434463] ptp nvme_core [38938.437514] Call Trace: [38938.437517] [38938.444639] i2c_algo_bit [38938.447345] ? watchdog_timer_fn+0x1b4/0x220 [38938.449786] pps_core dca [38938.451815] ? __hrtimer_run_queues+0x1d3/0x360 [38938.454434] fuse [38938.458701] ? hrtimer_interrupt+0x104/0x240 [38938.461313] [38938.461314] irq event stamp: 231840516 [38938.465837] ? __local_bh_enable+0x50/0x60 [38938.467761] hardirqs last enabled at (231840515): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38938.472030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38938.473518] hardirqs last disabled at (231840516): [] sysvec_apic_timer_interrupt+0xe/0x90 [38938.477261] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38938.481352] softirqs last enabled at (231840514): [] __do_softirq+0x2be/0x371 [38938.491597] [38938.496816] softirqs last disabled at (231840509): [] irq_exit_rcu+0xb0/0xd0 [38938.506626] [38938.511582] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38938.520353] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38938.522449] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38938.531051] ? queued_spin_lock_slowpath+0x18/0x2e0 [38938.533146] Workqueue: events delayed_work [ceph] [38938.542526] do_raw_spin_lock+0xb5/0xc0 [38938.547827] [38938.547828] RIP: 0010:lock_acquire+0xf0/0x2b0 [38938.555298] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38938.560167] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38938.564869] try_get_cap_refs+0x159/0x880 [ceph] [38938.568697] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38938.570192] __ceph_get_caps+0x4a1/0x760 [ceph] [38938.574538] [38938.574538] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38938.579327] ? __file_remove_privs+0x89/0x110 [38938.598059] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38938.602671] ? wake_bit_function+0x70/0x70 [38938.607887] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38938.612415] ? ceph_write_iter+0x260/0x770 [ceph] [38938.613903] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38938.621028] ceph_write_iter+0x260/0x770 [ceph] [38938.625378] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38938.632504] ? lock_is_held_type+0xaa/0x120 [38938.636591] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38938.643718] vfs_write+0x324/0x4f0 [38938.648414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38938.655543] ksys_pwrite64+0x65/0xa0 [38938.660062] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38938.667187] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38938.671363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38938.679442] do_syscall_64+0x38/0x80 [38938.682838] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38938.688577] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38938.692146] Call Trace: [38938.692147] [38938.699272] RIP: 0033:0x7f48f0eeb93b [38938.704403] ? watchdog_timer_fn+0x1b4/0x220 [38938.711526] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38938.715104] ? __hrtimer_run_queues+0x1d3/0x360 [38938.722230] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38938.727278] ? hrtimer_interrupt+0x104/0x240 [38938.729717] ORIG_RAX: 0000000000000012 [38938.731729] ? __local_bh_enable+0x50/0x60 [38938.735298] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38938.739565] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38938.758300] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38938.762826] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38938.768042] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38938.772307] [38938.776135] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38938.780227] [38938.780228] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38938.787352] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38938.792574] ? lock_acquire+0xf0/0x2b0 [38938.799699] [38938.804653] ? igrab+0x43/0x50 [38938.844512] ? lock_release+0xc5/0x260 [38938.848256] _raw_spin_lock+0x30/0x40 [38938.851919] ? igrab+0x1d/0x50 [38938.854970] igrab+0x1d/0x50 [38938.857851] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38938.862999] delayed_work+0x1d8/0x280 [ceph] [38938.867281] ? process_scheduled_works+0x22a/0x510 [38938.872070] process_scheduled_works+0x2a4/0x510 [38938.876687] worker_thread+0x188/0x340 [38938.880437] ? rescuer_thread+0x310/0x310 [38938.884447] kthread+0xf5/0x130 [38938.887584] ? kthread_complete_and_exit+0x20/0x20 [38938.892368] ret_from_fork+0x31/0x40 [38938.895948] ? kthread_complete_and_exit+0x20/0x20 [38938.900733] ret_from_fork_asm+0x11/0x20 [38938.904665] [38966.135900] watchdog: BUG: soft lockup - CPU#0 stuck for 25176s! [dbench:78683] [38966.143200] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38966.147900] watchdog: BUG: soft lockup - CPU#7 stuck for 34173s! [kworker/7:4:78808] [38966.198643] irq event stamp: 68997496 [38966.198644] hardirqs last enabled at (68997495): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38966.206374] Modules linked in: [38966.210029] hardirqs last disabled at (68997496): [] sysvec_apic_timer_interrupt+0xe/0x90 [38966.220188] ceph [38966.223239] softirqs last enabled at (68997438): [] __do_softirq+0x2be/0x371 [38966.232962] libceph dns_resolver [38966.234888] softirqs last disabled at (68997433): [] irq_exit_rcu+0xb0/0xd0 [38966.243575] fscache netfs [38966.246890] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38966.255400] veth nft_chain_nat [38966.258105] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38966.267048] xt_MASQUERADE [38966.270186] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38966.277658] nf_nat nf_conntrack [38966.280361] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38966.285840] nf_defrag_ipv6 nf_defrag_ipv4 [38966.289063] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38966.307800] nft_compat nf_tables [38966.311891] [38966.317108] nfnetlink [38966.320419] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38966.321910] bridge stp [38966.324269] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38966.331409] llc [38966.333853] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38966.340978] xfs libcrc32c [38966.342814] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38966.349939] sunrpc [38966.352642] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38966.359768] intel_rapl_msr [38966.361865] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38966.368987] intel_rapl_common [38966.371780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38966.379857] sb_edac [38966.382908] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38966.388646] x86_pkg_temp_thermal [38966.390828] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38966.397952] coretemp [38966.401263] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38966.408387] kvm_intel kvm [38966.410660] Call Trace: [38966.410661] [38966.417783] irqbypass i2c_i801 [38966.420488] ? watchdog_timer_fn+0x1b4/0x220 [38966.422930] iTCO_wdt iTCO_vendor_support [38966.424944] ? __hrtimer_run_queues+0x1d3/0x360 [38966.428080] i2c_smbus crct10dif_pclmul [38966.432347] ? hrtimer_interrupt+0x104/0x240 [38966.436346] joydev crc32_pclmul [38966.440871] ? __local_bh_enable+0x50/0x60 [38966.444702] ipmi_ssif [38966.448967] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38966.452190] ghash_clmulni_intel [38966.456298] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38966.458656] mei_me mei [38966.463873] [38966.467098] lpc_ich sha512_ssse3 [38966.472054] [38966.472055] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38966.474497] acpi_ipmi mfd_core [38966.476599] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38966.479905] wmi ipmi_si [38966.482005] do_raw_spin_lock+0xb5/0xc0 [38966.487309] ipmi_devintf [38966.490446] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38966.495315] ipmi_msghandler ioatdma [38966.497850] try_get_cap_refs+0x159/0x880 [ceph] [38966.501678] acpi_pad acpi_power_meter [38966.504304] __ceph_get_caps+0x4a1/0x760 [ceph] [38966.509079] ixgbe igb [38966.512654] ? __file_remove_privs+0x89/0x110 [38966.517260] mdio crc32c_intel [38966.521006] ? wake_bit_function+0x70/0x70 [38966.525529] nvme ptp [38966.527889] ? ceph_write_iter+0x260/0x770 [ceph] [38966.532237] nvme_core i2c_algo_bit [38966.535301] ceph_write_iter+0x260/0x770 [ceph] [38966.539386] pps_core dca [38966.541662] ? lock_is_held_type+0xaa/0x120 [38966.546355] fuse [38966.549842] vfs_write+0x324/0x4f0 [38966.554364] irq event stamp: 232061406 [38966.556985] ksys_pwrite64+0x65/0xa0 [38966.561159] hardirqs last enabled at (232061405): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38966.563082] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38966.566480] hardirqs last disabled at (232061406): [] sysvec_apic_timer_interrupt+0xe/0x90 [38966.570225] do_syscall_64+0x38/0x80 [38966.573793] softirqs last enabled at (232061404): [] __do_softirq+0x2be/0x371 [38966.584039] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38966.589169] softirqs last disabled at (232061399): [] irq_exit_rcu+0xb0/0xd0 [38966.598981] RIP: 0033:0x7f48f0eeb93b [38966.602550] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38966.611322] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38966.616366] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38966.624962] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38966.628534] Workqueue: events delayed_work [ceph] [38966.637913] ORIG_RAX: 0000000000000012 [38966.656650] [38966.656650] RIP: 0010:lock_acquire+0xf0/0x2b0 [38966.664120] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38966.669338] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38966.674036] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38966.677866] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38966.679355] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38966.683708] [38966.683709] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38966.690831] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38966.709580] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38966.716702] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38966.721920] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38966.729050] [38966.730535] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38966.775463] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38966.782602] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38966.790681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38966.796419] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38966.803543] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38966.810667] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38966.817791] Call Trace: [38966.820236] [38966.822246] ? watchdog_timer_fn+0x1b4/0x220 [38966.826512] ? __hrtimer_run_queues+0x1d3/0x360 [38966.831038] ? hrtimer_interrupt+0x104/0x240 [38966.835307] ? __local_bh_enable+0x50/0x60 [38966.839400] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38966.844625] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38966.849598] [38966.851697] [38966.853794] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38966.859102] ? lock_acquire+0xf0/0x2b0 [38966.862853] ? igrab+0x43/0x50 [38966.865901] ? lock_release+0xc5/0x260 [38966.869648] _raw_spin_lock+0x30/0x40 [38966.873312] ? igrab+0x1d/0x50 [38966.876362] igrab+0x1d/0x50 [38966.879242] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38966.884391] delayed_work+0x1d8/0x280 [ceph] [38966.888671] ? process_scheduled_works+0x22a/0x510 [38966.893463] process_scheduled_works+0x2a4/0x510 [38966.898076] worker_thread+0x188/0x340 [38966.901826] ? rescuer_thread+0x310/0x310 [38966.905829] kthread+0xf5/0x130 [38966.908967] ? kthread_complete_and_exit+0x20/0x20 [38966.913753] ret_from_fork+0x31/0x40 [38966.917330] ? kthread_complete_and_exit+0x20/0x20 [38966.922116] ret_from_fork_asm+0x11/0x20 [38966.926038] [38969.198319] rcu: INFO: rcu_preempt self-detected stall on CPU [38969.204083] rcu: 0-....: (26420909 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=13232738 [38969.214594] rcu: (t=27032116 jiffies g=1309609 q=731266 ncpus=8) [38969.220680] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38969.229622] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38969.237094] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [38969.242580] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [38969.261318] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38969.266543] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38969.273666] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38969.280792] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38969.287916] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38969.295039] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38969.302163] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38969.310242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38969.315979] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38969.323103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38969.330226] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38969.337350] Call Trace: [38969.339795] [38969.341806] ? rcu_dump_cpu_stacks+0xd1/0x180 [38969.346166] ? rcu_sched_clock_irq+0x633/0x1390 [38969.350691] ? lock_is_held_type+0xaa/0x120 [38969.354869] ? lock_is_held_type+0xaa/0x120 [38969.359046] ? tick_sched_do_timer+0xa0/0xa0 [38969.363318] ? rcu_read_lock_sched_held+0x56/0x90 [38969.368023] ? tick_sched_do_timer+0xa0/0xa0 [38969.372287] ? update_process_times+0x5f/0x90 [38969.376637] ? tick_sched_handle.isra.16+0x1e/0x50 [38969.381423] ? tick_sched_timer+0x6f/0x80 [38969.385426] ? __hrtimer_run_queues+0x1d3/0x360 [38969.389954] ? hrtimer_interrupt+0x104/0x240 [38969.394224] ? __local_bh_enable+0x50/0x60 [38969.398316] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38969.403539] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38969.408498] [38969.410594] [38969.412692] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38969.418000] ? queued_spin_lock_slowpath+0x18/0x2e0 [38969.422879] do_raw_spin_lock+0xb5/0xc0 [38969.426716] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38969.431519] try_get_cap_refs+0x159/0x880 [ceph] [38969.436149] __ceph_get_caps+0x4a1/0x760 [ceph] [38969.440690] ? __file_remove_privs+0x89/0x110 [38969.445045] ? wake_bit_function+0x70/0x70 [38969.449140] ? ceph_write_iter+0x260/0x770 [ceph] [38969.453850] ceph_write_iter+0x260/0x770 [ceph] [38969.458386] ? lock_is_held_type+0xaa/0x120 [38969.462573] vfs_write+0x324/0x4f0 [38969.465979] ksys_pwrite64+0x65/0xa0 [38969.469554] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38969.474686] do_syscall_64+0x38/0x80 [38969.478258] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38969.483299] RIP: 0033:0x7f48f0eeb93b [38969.486871] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38969.505611] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [38969.513175] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38969.520300] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38969.527424] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38969.534547] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38969.541672] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38969.548801] [38969.550989] Sending NMI from CPU 0 to CPUs 7: [38969.555341] NMI backtrace for cpu 7 [38969.555343] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38969.555344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38969.555345] Workqueue: events delayed_work [ceph] [38969.555362] RIP: 0010:__lock_acquire+0x276/0x1e20 [38969.555364] Code: 00 44 0f b6 b8 c9 00 00 00 8b bb 50 0a 00 00 45 84 ff 45 0f 44 fd 41 89 fd 41 83 ed 01 0f 88 34 09 00 00 49 63 c5 48 8b 14 24 <48> 8d 04 80 0f b6 4c c2 21 31 f1 89 c8 a8 60 0f 85 18 09 00 00 8d [38969.555365] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [38969.555367] RAX: 0000000000000002 RBX: ffff88810debaa80 RCX: 0000000000000004 [38969.555367] RDX: ffff88810debb4d8 RSI: 0000000000000006 RDI: 0000000000000003 [38969.555368] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [38969.555369] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [38969.555370] R13: 0000000000000002 R14: 0000000000000006 R15: 0000000000000002 [38969.555370] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38969.555371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38969.555372] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38969.555373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38969.555374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38969.555375] Call Trace: [38969.555375] [38969.555376] ? nmi_cpu_backtrace+0xad/0x120 [38969.555379] ? nmi_cpu_backtrace_handler+0x11/0x20 [38969.555381] ? nmi_handle+0xd4/0x260 [38969.555385] ? default_do_nmi+0x49/0xf0 [38969.555387] ? exc_nmi+0x11e/0x150 [38969.555389] ? end_repeat_nmi+0x16/0x67 [38969.555393] ? __lock_acquire+0x276/0x1e20 [38969.555394] ? __lock_acquire+0x276/0x1e20 [38969.555396] ? __lock_acquire+0x276/0x1e20 [38969.555397] [38969.555398] [38969.555400] lock_acquire+0xcb/0x2b0 [38969.555401] ? igrab+0x1d/0x50 [38969.555403] ? igrab+0x43/0x50 [38969.555404] ? lock_release+0xc5/0x260 [38969.555406] _raw_spin_lock+0x30/0x40 [38969.555408] ? igrab+0x1d/0x50 [38969.555409] igrab+0x1d/0x50 [38969.555411] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38969.555427] delayed_work+0x1d8/0x280 [ceph] [38969.555443] ? process_scheduled_works+0x22a/0x510 [38969.555446] process_scheduled_works+0x2a4/0x510 [38969.555449] worker_thread+0x188/0x340 [38969.555452] ? rescuer_thread+0x310/0x310 [38969.555454] kthread+0xf5/0x130 [38969.555456] ? kthread_complete_and_exit+0x20/0x20 [38969.555459] ret_from_fork+0x31/0x40 [38969.555461] ? kthread_complete_and_exit+0x20/0x20 [38969.555463] ret_from_fork_asm+0x11/0x20 [38969.555467] [38994.136384] watchdog: BUG: soft lockup - CPU#0 stuck for 25202s! [dbench:78683] [38994.143688] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [38994.148383] watchdog: BUG: soft lockup - CPU#7 stuck for 34199s! [kworker/7:4:78808] [38994.199139] irq event stamp: 69054984 [38994.206870] Modules linked in: ceph libceph dns_resolver fscache netfs [38994.210527] hardirqs last enabled at (69054983): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38994.217059] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [38994.227219] hardirqs last disabled at (69054984): [] sysvec_apic_timer_interrupt+0xe/0x90 [38994.233304] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [38994.243027] softirqs last enabled at (69054976): [] __do_softirq+0x2be/0x371 [38994.243031] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [38994.259103] softirqs last disabled at (69054971): [] irq_exit_rcu+0xb0/0xd0 [38994.265969] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [38994.274481] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38994.274481] irqbypass [38994.274483] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38994.280477] i2c_i801 [38994.289420] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [38994.291778] iTCO_wdt iTCO_vendor_support [38994.299248] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [38994.301519] i2c_smbus crct10dif_pclmul [38994.306997] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [38994.311001] joydev crc32_pclmul [38994.329738] [38994.333570] ipmi_ssif [38994.338787] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [38994.342010] ghash_clmulni_intel mei_me [38994.343501] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [38994.345859] mei lpc_ich [38994.352983] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [38994.356815] sha512_ssse3 [38994.363939] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [38994.366469] acpi_ipmi mfd_core [38994.373592] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [38994.376211] wmi ipmi_si [38994.383335] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [38994.386473] ipmi_devintf [38994.393595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38994.396126] ipmi_msghandler ioatdma [38994.404205] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [38994.406822] acpi_pad acpi_power_meter [38994.412560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38994.416128] ixgbe igb mdio [38994.423254] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38994.426999] crc32c_intel nvme [38994.434121] Call Trace: [38994.434123] [38994.436912] ptp nvme_core [38994.444039] ? watchdog_timer_fn+0x1b4/0x220 [38994.447105] i2c_algo_bit pps_core [38994.449549] ? __hrtimer_run_queues+0x1d3/0x360 [38994.451559] dca fuse [38994.454266] ? hrtimer_interrupt+0x104/0x240 [38994.458528] [38994.458529] irq event stamp: 232280196 [38994.461925] ? __local_bh_enable+0x50/0x60 [38994.466448] hardirqs last enabled at (232280195): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38994.468723] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38994.472983] hardirqs last disabled at (232280196): [] sysvec_apic_timer_interrupt+0xe/0x90 [38994.474475] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38994.478217] softirqs last enabled at (232280194): [] __do_softirq+0x2be/0x371 [38994.482310] [38994.492554] softirqs last disabled at (232280189): [] irq_exit_rcu+0xb0/0xd0 [38994.497783] [38994.507589] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [38994.512549] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38994.521319] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [38994.521320] Workqueue: events delayed_work [ceph] [38994.523420] ? queued_spin_lock_slowpath+0x2d/0x2e0 [38994.532014] [38994.532014] RIP: 0010:lock_acquire+0xf0/0x2b0 [38994.534113] do_raw_spin_lock+0xb5/0xc0 [38994.543489] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [38994.548794] __cap_delay_requeue+0xa7/0x1d0 [ceph] [38994.556262] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [38994.560964] try_get_cap_refs+0x159/0x880 [ceph] [38994.565833] [38994.567327] __ceph_get_caps+0x4a1/0x760 [ceph] [38994.571672] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [38994.575508] ? __file_remove_privs+0x89/0x110 [38994.594241] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [38994.599028] ? wake_bit_function+0x70/0x70 [38994.604243] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [38994.608856] ? ceph_write_iter+0x260/0x770 [ceph] [38994.610345] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [38994.614869] ceph_write_iter+0x260/0x770 [ceph] [38994.621993] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [38994.626348] ? lock_is_held_type+0xaa/0x120 [38994.633467] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [38994.637561] vfs_write+0x324/0x4f0 [38994.644683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [38994.649384] ksys_pwrite64+0x65/0xa0 [38994.656505] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [38994.661028] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [38994.668151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [38994.672330] do_syscall_64+0x38/0x80 [38994.680406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [38994.683805] entry_SYSCALL_64_after_hwframe+0x63/0xcd [38994.689542] Call Trace: [38994.689543] [38994.693112] RIP: 0033:0x7f48f0eeb93b [38994.700237] ? watchdog_timer_fn+0x1b4/0x220 [38994.705369] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [38994.712493] ? __hrtimer_run_queues+0x1d3/0x360 [38994.716067] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [38994.723198] ? hrtimer_interrupt+0x104/0x240 [38994.728239] ORIG_RAX: 0000000000000012 [38994.730684] ? __local_bh_enable+0x50/0x60 [38994.732693] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [38994.736268] ? __sysvec_apic_timer_interrupt+0x61/0x160 [38994.740528] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [38994.759266] ? sysvec_apic_timer_interrupt+0x6d/0x90 [38994.763790] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [38994.769007] [38994.773273] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [38994.777104] [38994.781193] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [38994.788319] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [38994.793541] [38994.800664] ? lock_acquire+0xf0/0x2b0 [38994.842428] ? igrab+0x43/0x50 [38994.845485] ? lock_release+0xc5/0x260 [38994.849231] _raw_spin_lock+0x30/0x40 [38994.852894] ? igrab+0x1d/0x50 [38994.855946] igrab+0x1d/0x50 [38994.858825] ceph_check_delayed_caps+0xb7/0x270 [ceph] [38994.863974] delayed_work+0x1d8/0x280 [ceph] [38994.868254] ? process_scheduled_works+0x22a/0x510 [38994.873046] process_scheduled_works+0x2a4/0x510 [38994.877660] worker_thread+0x188/0x340 [38994.881411] ? rescuer_thread+0x310/0x310 [38994.885421] kthread+0xf5/0x130 [38994.888559] ? kthread_complete_and_exit+0x20/0x20 [38994.893344] ret_from_fork+0x31/0x40 [38994.896923] ? kthread_complete_and_exit+0x20/0x20 [38994.901708] ret_from_fork_asm+0x11/0x20 [38994.905640] [39022.136866] watchdog: BUG: soft lockup - CPU#0 stuck for 25228s! [dbench:78683] [39022.144168] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39022.148866] watchdog: BUG: soft lockup - CPU#7 stuck for 34225s! [kworker/7:4:78808] [39022.199609] irq event stamp: 69113082 [39022.199610] hardirqs last enabled at (69113081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39022.207339] Modules linked in: [39022.210998] hardirqs last disabled at (69113082): [] sysvec_apic_timer_interrupt+0xe/0x90 [39022.221153] ceph [39022.224206] softirqs last enabled at (69113034): [] __do_softirq+0x2be/0x371 [39022.233930] libceph [39022.235853] softirqs last disabled at (69113029): [] irq_exit_rcu+0xb0/0xd0 [39022.244543] dns_resolver fscache [39022.246729] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39022.255259] netfs veth [39022.258569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39022.258570] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39022.267513] nft_chain_nat xt_MASQUERADE [39022.269956] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39022.277427] nf_nat nf_conntrack [39022.282905] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39022.286822] nf_defrag_ipv6 nf_defrag_ipv4 [39022.305561] [39022.305562] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39022.308784] nft_compat [39022.314001] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39022.318093] nf_tables nfnetlink [39022.319584] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39022.326708] bridge stp [39022.329151] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39022.336275] llc [39022.339498] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39022.346625] xfs libcrc32c [39022.349068] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39022.356192] sunrpc intel_rapl_msr [39022.358030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39022.365154] intel_rapl_common [39022.367859] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39022.375936] sb_edac x86_pkg_temp_thermal [39022.379332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39022.385070] coretemp kvm_intel [39022.388122] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39022.395262] kvm irqbypass [39022.399266] Call Trace: [39022.399267] [39022.406390] i2c_i801 iTCO_wdt [39022.409544] ? watchdog_timer_fn+0x1b4/0x220 [39022.416669] iTCO_vendor_support i2c_smbus [39022.419375] ? __hrtimer_run_queues+0x1d3/0x360 [39022.421817] crct10dif_pclmul joydev [39022.423831] ? hrtimer_interrupt+0x104/0x240 [39022.426877] crc32_pclmul ipmi_ssif [39022.431142] ? __local_bh_enable+0x50/0x60 [39022.435234] ghash_clmulni_intel mei_me [39022.439759] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39022.443327] mei lpc_ich [39022.447592] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39022.451077] sha512_ssse3 acpi_ipmi [39022.455167] [39022.458998] mfd_core wmi [39022.464216] [39022.464218] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39022.466747] ipmi_si ipmi_devintf [39022.471708] ? queued_spin_lock_slowpath+0x18/0x2e0 [39022.475187] ipmi_msghandler ioatdma [39022.477287] do_raw_spin_lock+0xb5/0xc0 [39022.479903] acpi_pad [39022.482001] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39022.487305] acpi_power_meter ixgbe [39022.490615] try_get_cap_refs+0x159/0x880 [ceph] [39022.495486] igb mdio [39022.499062] __ceph_get_caps+0x4a1/0x760 [ceph] [39022.502886] crc32c_intel nvme [39022.505162] ? __file_remove_privs+0x89/0x110 [39022.509941] ptp nvme_core [39022.513427] ? wake_bit_function+0x70/0x70 [39022.518035] i2c_algo_bit pps_core [39022.520309] ? ceph_write_iter+0x260/0x770 [ceph] [39022.524830] dca fuse [39022.527882] ceph_write_iter+0x260/0x770 [ceph] [39022.532232] [39022.532233] irq event stamp: 232501158 [39022.534940] ? lock_is_held_type+0xaa/0x120 [39022.539026] hardirqs last enabled at (232501157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39022.542428] vfs_write+0x324/0x4f0 [39022.547122] hardirqs last disabled at (232501158): [] sysvec_apic_timer_interrupt+0xe/0x90 [39022.549397] ksys_pwrite64+0x65/0xa0 [39022.553917] softirqs last enabled at (232501156): [] __do_softirq+0x2be/0x371 [39022.555407] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39022.559151] softirqs last disabled at (232501151): [] irq_exit_rcu+0xb0/0xd0 [39022.563329] do_syscall_64+0x38/0x80 [39022.573574] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39022.576972] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39022.586782] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39022.590352] RIP: 0033:0x7f48f0eeb93b [39022.599123] Workqueue: events delayed_work [ceph] [39022.604267] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39022.612870] [39022.612871] RIP: 0010:lock_release+0x14e/0x260 [39022.616439] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39022.625817] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [39022.630861] ORIG_RAX: 0000000000000012 [39022.638333] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [39022.641903] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39022.646599] [39022.646600] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [39022.665337] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39022.666828] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [39022.671270] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39022.676482] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39022.695222] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39022.699052] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39022.704269] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39022.711393] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [39022.712889] [39022.720007] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39022.787281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39022.793018] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39022.800143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39022.807271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39022.814398] Call Trace: [39022.816844] [39022.818854] ? watchdog_timer_fn+0x1b4/0x220 [39022.823119] ? __hrtimer_run_queues+0x1d3/0x360 [39022.827644] ? hrtimer_interrupt+0x104/0x240 [39022.831906] ? __local_bh_enable+0x50/0x60 [39022.836000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39022.841224] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39022.846179] [39022.848278] [39022.850376] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39022.855681] ? igrab+0x43/0x50 [39022.858732] ? lock_release+0x14e/0x260 [39022.862562] ? lock_release+0xc5/0x260 [39022.866306] _raw_spin_unlock+0x1b/0x40 [39022.870137] igrab+0x43/0x50 [39022.873013] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39022.878162] delayed_work+0x1d8/0x280 [ceph] [39022.882444] ? process_scheduled_works+0x22a/0x510 [39022.887235] process_scheduled_works+0x2a4/0x510 [39022.891851] worker_thread+0x188/0x340 [39022.895600] ? rescuer_thread+0x310/0x310 [39022.899603] kthread+0xf5/0x130 [39022.902740] ? kthread_complete_and_exit+0x20/0x20 [39022.907524] ret_from_fork+0x31/0x40 [39022.911111] ? kthread_complete_and_exit+0x20/0x20 [39022.915896] ret_from_fork_asm+0x11/0x20 [39022.919820] [39050.137348] watchdog: BUG: soft lockup - CPU#0 stuck for 25254s! [dbench:78683] [39050.144656] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39050.149348] watchdog: BUG: soft lockup - CPU#7 stuck for 34251s! [kworker/7:4:78808] [39050.200106] irq event stamp: 69171370 [39050.207834] Modules linked in: ceph libceph dns_resolver fscache netfs [39050.211493] hardirqs last enabled at (69171369): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39050.211499] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [39050.228175] hardirqs last disabled at (69171370): [] sysvec_apic_timer_interrupt+0xe/0x90 [39050.228178] nf_defrag_ipv4 [39050.228180] softirqs last enabled at (69171364): [] __do_softirq+0x2be/0x371 [39050.235561] nft_compat nf_tables [39050.245284] softirqs last disabled at (69171359): [] irq_exit_rcu+0xb0/0xd0 [39050.248076] nfnetlink bridge [39050.256762] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39050.260071] stp llc [39050.268581] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39050.271544] xfs libcrc32c [39050.280488] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39050.282675] sunrpc intel_rapl_msr [39050.290145] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39050.292848] intel_rapl_common [39050.298326] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39050.301741] sb_edac [39050.320480] [39050.323530] x86_pkg_temp_thermal coretemp [39050.328747] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39050.330930] kvm_intel kvm [39050.332420] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39050.336510] irqbypass i2c_i801 [39050.343637] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39050.346340] iTCO_wdt [39050.353464] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39050.356601] iTCO_vendor_support [39050.363742] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39050.363743] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39050.366026] i2c_smbus crct10dif_pclmul [39050.373147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39050.376371] joydev [39050.383495] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39050.391572] crc32_pclmul [39050.395404] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39050.401139] ipmi_ssif [39050.403236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39050.410362] ghash_clmulni_intel [39050.412979] Call Trace: [39050.412982] [39050.420102] mei_me mei lpc_ich [39050.422463] ? watchdog_timer_fn+0x1b4/0x220 [39050.429586] sha512_ssse3 acpi_ipmi [39050.432811] ? __hrtimer_run_queues+0x1d3/0x360 [39050.435252] mfd_core wmi [39050.437267] ? hrtimer_interrupt+0x104/0x240 [39050.440401] ipmi_si ipmi_devintf [39050.444665] ? __local_bh_enable+0x50/0x60 [39050.448150] ipmi_msghandler ioatdma [39050.452675] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39050.455290] acpi_pad acpi_power_meter [39050.459555] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39050.462864] ixgbe igb [39050.466957] [39050.470528] mdio crc32c_intel [39050.475747] [39050.479488] nvme [39050.484445] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39050.486803] ptp nvme_core [39050.488904] ? queued_spin_lock_slowpath+0x18/0x2e0 [39050.491951] i2c_algo_bit pps_core [39050.494050] do_raw_spin_lock+0xb5/0xc0 [39050.495973] dca fuse [39050.501279] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39050.503981] [39050.503981] irq event stamp: 232722018 [39050.508854] try_get_cap_refs+0x159/0x880 [ceph] [39050.512249] hardirqs last enabled at (232722017): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39050.516084] __ceph_get_caps+0x4a1/0x760 [ceph] [39050.518349] hardirqs last disabled at (232722018): [] sysvec_apic_timer_interrupt+0xe/0x90 [39050.523140] ? __file_remove_privs+0x89/0x110 [39050.524625] softirqs last enabled at (232722016): [] __do_softirq+0x2be/0x371 [39050.528369] ? wake_bit_function+0x70/0x70 [39050.532980] softirqs last disabled at (232722011): [] irq_exit_rcu+0xb0/0xd0 [39050.543226] ? ceph_write_iter+0x260/0x770 [ceph] [39050.547748] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39050.557559] ceph_write_iter+0x260/0x770 [ceph] [39050.561911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39050.570684] ? lock_is_held_type+0xaa/0x120 [39050.574772] Workqueue: events delayed_work [ceph] [39050.583371] vfs_write+0x324/0x4f0 [39050.588066] [39050.588067] RIP: 0010:lock_acquire+0xf0/0x2b0 [39050.597449] ksys_pwrite64+0x65/0xa0 [39050.601968] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39050.609441] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39050.613617] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39050.618314] do_syscall_64+0x38/0x80 [39050.621710] [39050.621711] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39050.623202] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39050.627553] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39050.631124] RIP: 0033:0x7f48f0eeb93b [39050.649862] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39050.654992] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39050.660210] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39050.663779] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39050.665269] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39050.672396] ORIG_RAX: 0000000000000012 [39050.677440] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39050.684562] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39050.688133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39050.695259] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39050.713994] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39050.721119] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39050.726338] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39050.733460] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39050.737290] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39050.745370] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39050.752493] Call Trace: [39050.758235] [39050.765356] [39050.814767] ? watchdog_timer_fn+0x1b4/0x220 [39050.819055] ? __hrtimer_run_queues+0x1d3/0x360 [39050.823583] ? hrtimer_interrupt+0x104/0x240 [39050.827850] ? __local_bh_enable+0x50/0x60 [39050.831945] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39050.837168] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39050.842127] [39050.844224] [39050.846322] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39050.851629] ? lock_acquire+0xf0/0x2b0 [39050.855379] ? igrab+0x43/0x50 [39050.858430] ? lock_release+0xc5/0x260 [39050.862174] _raw_spin_lock+0x30/0x40 [39050.865830] ? igrab+0x1d/0x50 [39050.868881] igrab+0x1d/0x50 [39050.871760] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39050.876909] delayed_work+0x1d8/0x280 [ceph] [39050.881190] ? process_scheduled_works+0x22a/0x510 [39050.885981] process_scheduled_works+0x2a4/0x510 [39050.890596] worker_thread+0x188/0x340 [39050.894346] ? rescuer_thread+0x310/0x310 [39050.898358] kthread+0xf5/0x130 [39050.901494] ? kthread_complete_and_exit+0x20/0x20 [39050.906280] ret_from_fork+0x31/0x40 [39050.909858] ? kthread_complete_and_exit+0x20/0x20 [39050.914642] ret_from_fork_asm+0x11/0x20 [39050.918564] [39078.137827] watchdog: BUG: soft lockup - CPU#0 stuck for 25280s! [dbench:78683] [39078.145126] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39078.149827] watchdog: BUG: soft lockup - CPU#7 stuck for 34277s! [kworker/7:4:78808] [39078.200566] irq event stamp: 69229556 [39078.200568] hardirqs last enabled at (69229555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39078.208296] Modules linked in: [39078.211955] hardirqs last disabled at (69229556): [] sysvec_apic_timer_interrupt+0xe/0x90 [39078.222111] ceph [39078.225162] softirqs last enabled at (69229510): [] __do_softirq+0x2be/0x371 [39078.234887] libceph [39078.236811] softirqs last disabled at (69229505): [] irq_exit_rcu+0xb0/0xd0 [39078.245503] dns_resolver fscache [39078.247688] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39078.256198] netfs veth [39078.259527] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39078.268471] nft_chain_nat [39078.270916] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39078.278385] xt_MASQUERADE nf_nat [39078.281090] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39078.286568] nf_conntrack nf_defrag_ipv6 [39078.289878] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39078.308614] nf_defrag_ipv4 [39078.312532] [39078.312533] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39078.317750] nft_compat nf_tables [39078.320540] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39078.322031] nfnetlink bridge [39078.329156] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39078.332465] stp llc [39078.339591] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39078.342555] xfs [39078.349680] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39078.351862] libcrc32c sunrpc [39078.358987] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39078.360824] intel_rapl_msr [39078.367947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39078.370914] intel_rapl_common sb_edac [39078.378990] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39078.381780] x86_pkg_temp_thermal coretemp [39078.387518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39078.391263] kvm_intel kvm [39078.398387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39078.402476] irqbypass [39078.409600] Call Trace: [39078.409602] [39078.412305] i2c_i801 iTCO_wdt [39078.419432] ? watchdog_timer_fn+0x1b4/0x220 [39078.421787] iTCO_vendor_support i2c_smbus [39078.424232] ? __hrtimer_run_queues+0x1d3/0x360 [39078.426242] crct10dif_pclmul joydev [39078.429295] ? hrtimer_interrupt+0x104/0x240 [39078.433556] crc32_pclmul ipmi_ssif [39078.437648] ? __local_bh_enable+0x50/0x60 [39078.442170] ghash_clmulni_intel [39078.445745] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39078.450005] mei_me mei [39078.453507] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39078.457597] lpc_ich [39078.460823] [39078.466040] sha512_ssse3 acpi_ipmi [39078.468485] [39078.468486] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39078.473441] mfd_core wmi [39078.475630] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39078.477722] ipmi_si ipmi_devintf [39078.481226] do_raw_spin_lock+0xb5/0xc0 [39078.483322] ipmi_msghandler [39078.488627] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39078.491243] ioatdma acpi_pad [39078.496116] try_get_cap_refs+0x159/0x880 [ceph] [39078.499425] acpi_power_meter ixgbe [39078.503259] __ceph_get_caps+0x4a1/0x760 [ceph] [39078.506132] igb [39078.510919] ? __file_remove_privs+0x89/0x110 [39078.513881] mdio crc32c_intel [39078.518505] ? wake_bit_function+0x70/0x70 [39078.521992] nvme ptp [39078.526518] ? ceph_write_iter+0x260/0x770 [ceph] [39078.528353] nvme_core i2c_algo_bit [39078.532706] ceph_write_iter+0x260/0x770 [ceph] [39078.535755] pps_core dca [39078.539850] ? lock_is_held_type+0xaa/0x120 [39078.542116] fuse [39078.546817] vfs_write+0x324/0x4f0 [39078.550299] [39078.550300] irq event stamp: 232942840 [39078.554827] ksys_pwrite64+0x65/0xa0 [39078.557441] hardirqs last enabled at (232942839): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39078.561618] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39078.563541] hardirqs last disabled at (232942840): [] sysvec_apic_timer_interrupt+0xe/0x90 [39078.566939] do_syscall_64+0x38/0x80 [39078.568429] softirqs last enabled at (232942838): [] __do_softirq+0x2be/0x371 [39078.572173] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39078.575743] softirqs last disabled at (232942833): [] irq_exit_rcu+0xb0/0xd0 [39078.585989] RIP: 0033:0x7f48f0eeb93b [39078.591119] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39078.600931] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39078.604501] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39078.613271] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39078.618317] Workqueue: events delayed_work [ceph] [39078.626913] ORIG_RAX: 0000000000000012 [39078.630484] [39078.630485] RIP: 0010:lock_acquire+0xf0/0x2b0 [39078.639880] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39078.658617] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39078.666088] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39078.671305] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39078.676002] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39078.679834] [39078.679835] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39078.681324] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39078.685673] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39078.692799] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39078.711536] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39078.718666] [39078.723879] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39078.777438] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39078.784562] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39078.792642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39078.798377] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39078.805509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39078.812635] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39078.819759] Call Trace: [39078.822204] [39078.824234] ? watchdog_timer_fn+0x1b4/0x220 [39078.828512] ? __hrtimer_run_queues+0x1d3/0x360 [39078.833040] ? hrtimer_interrupt+0x104/0x240 [39078.837308] ? __local_bh_enable+0x50/0x60 [39078.841404] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39078.846628] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39078.851583] [39078.853681] [39078.855780] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39078.861087] ? lock_acquire+0xf0/0x2b0 [39078.864836] ? igrab+0x43/0x50 [39078.867887] ? lock_release+0xc5/0x260 [39078.871631] _raw_spin_lock+0x30/0x40 [39078.875287] ? igrab+0x1d/0x50 [39078.878340] igrab+0x1d/0x50 [39078.881234] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39078.886383] delayed_work+0x1d8/0x280 [ceph] [39078.890664] ? process_scheduled_works+0x22a/0x510 [39078.895446] process_scheduled_works+0x2a4/0x510 [39078.900062] worker_thread+0x188/0x340 [39078.903810] ? rescuer_thread+0x310/0x310 [39078.907815] kthread+0xf5/0x130 [39078.910953] ? kthread_complete_and_exit+0x20/0x20 [39078.915737] ret_from_fork+0x31/0x40 [39078.919305] ? kthread_complete_and_exit+0x20/0x20 [39078.924092] ret_from_fork_asm+0x11/0x20 [39078.928013] [39106.138301] watchdog: BUG: soft lockup - CPU#0 stuck for 25306s! [dbench:78683] [39106.145605] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39106.150301] watchdog: BUG: soft lockup - CPU#7 stuck for 34303s! [kworker/7:4:78808] [39106.201054] irq event stamp: 69287856 [39106.208785] Modules linked in: ceph libceph dns_resolver fscache netfs veth [39106.212443] hardirqs last enabled at (69287855): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39106.219402] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [39106.229558] hardirqs last disabled at (69287856): [] sysvec_apic_timer_interrupt+0xe/0x90 [39106.236508] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [39106.246234] softirqs last enabled at (69287808): [] __do_softirq+0x2be/0x371 [39106.253011] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [39106.261694] softirqs last disabled at (69287803): [] irq_exit_rcu+0xb0/0xd0 [39106.267866] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [39106.276379] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39106.282374] irqbypass i2c_i801 iTCO_wdt [39106.291320] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39106.295235] iTCO_vendor_support i2c_smbus crct10dif_pclmul [39106.302707] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [39106.302709] joydev crc32_pclmul ipmi_ssif [39106.313748] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [39106.317839] ghash_clmulni_intel mei_me mei [39106.336577] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39106.336579] lpc_ich sha512_ssse3 [39106.340756] [39106.345977] acpi_ipmi mfd_core [39106.349291] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39106.350781] wmi ipmi_si [39106.353919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39106.361043] ipmi_devintf [39106.363575] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39106.370699] ipmi_msghandler [39106.373314] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39106.380440] ioatdma acpi_pad [39106.383317] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39106.390440] acpi_power_meter [39106.393406] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39106.400528] ixgbe igb [39106.403493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39106.411572] mdio crc32c_intel [39106.413929] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39106.419667] nvme ptp [39106.422715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39106.429840] nvme_core [39106.432110] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39106.439236] i2c_algo_bit pps_core [39106.441592] Call Trace: [39106.441594] [39106.448716] dca fuse [39106.452115] ? watchdog_timer_fn+0x1b4/0x220 [39106.454559] [39106.454560] irq event stamp: 233163530 [39106.456571] ? __hrtimer_run_queues+0x1d3/0x360 [39106.458839] hardirqs last enabled at (233163529): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39106.463108] ? hrtimer_interrupt+0x104/0x240 [39106.464595] hardirqs last disabled at (233163530): [] sysvec_apic_timer_interrupt+0xe/0x90 [39106.468338] ? __local_bh_enable+0x50/0x60 [39106.472861] softirqs last enabled at (233163528): [] __do_softirq+0x2be/0x371 [39106.483111] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39106.487370] softirqs last disabled at (233163523): [] irq_exit_rcu+0xb0/0xd0 [39106.497181] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39106.501273] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39106.510045] [39106.515261] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39106.515262] Workqueue: events delayed_work [ceph] [39106.523857] [39106.528816] [39106.528817] RIP: 0010:do_raw_spin_unlock+0x16/0xc0 [39106.538195] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39106.540290] Code: 00 48 89 47 10 b8 01 00 00 00 c3 31 c0 c3 66 0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 53 81 7f 04 ad 4e ad de 48 89 fb 75 34 <8b> 03 85 c0 74 48 65 48 8b 04 25 c0 e5 02 00 48 39 43 10 75 53 65 [39106.547766] ? queued_spin_lock_slowpath+0x2b/0x2e0 [39106.552458] RSP: 0018:ffffc90002117d80 EFLAGS: 00000246 [39106.554558] do_raw_spin_lock+0xb5/0xc0 [39106.556046] [39106.556047] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [39106.560833] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39106.566136] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [39106.584877] try_get_cap_refs+0x159/0x880 [ceph] [39106.589743] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [39106.594968] __ceph_get_caps+0x4a1/0x760 [ceph] [39106.598792] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [39106.600286] ? __file_remove_privs+0x89/0x110 [39106.607407] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [39106.612191] ? wake_bit_function+0x70/0x70 [39106.619316] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39106.623927] ? ceph_write_iter+0x260/0x770 [ceph] [39106.631048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39106.635573] ceph_write_iter+0x260/0x770 [ceph] [39106.642702] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39106.647054] ? lock_is_held_type+0xaa/0x120 [39106.654174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39106.658265] vfs_write+0x324/0x4f0 [39106.666342] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39106.671043] ksys_pwrite64+0x65/0xa0 [39106.676776] Call Trace: [39106.676777] [39106.681300] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39106.688425] ? watchdog_timer_fn+0x1b4/0x220 [39106.692603] do_syscall_64+0x38/0x80 [39106.699726] ? __hrtimer_run_queues+0x1d3/0x360 [39106.703123] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39106.710251] ? hrtimer_interrupt+0x104/0x240 [39106.713820] RIP: 0033:0x7f48f0eeb93b [39106.716261] ? __local_bh_enable+0x50/0x60 [39106.718273] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39106.723407] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39106.727668] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39106.731239] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39106.735761] ORIG_RAX: 0000000000000012 [39106.740807] [39106.745069] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39106.748642] [39106.748643] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39106.752730] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39106.771473] ? do_raw_spin_unlock+0x16/0xc0 [39106.776703] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39106.781922] _raw_spin_unlock+0x23/0x40 [39106.786878] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39106.790712] igrab+0x43/0x50 [39106.792806] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39106.799935] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39106.802033] [39106.807335] delayed_work+0x1d8/0x280 [ceph] [39106.858297] ? process_scheduled_works+0x22a/0x510 [39106.863088] process_scheduled_works+0x2a4/0x510 [39106.867710] worker_thread+0x188/0x340 [39106.871462] ? rescuer_thread+0x310/0x310 [39106.875473] kthread+0xf5/0x130 [39106.878620] ? kthread_complete_and_exit+0x20/0x20 [39106.883412] ret_from_fork+0x31/0x40 [39106.886991] ? kthread_complete_and_exit+0x20/0x20 [39106.891775] ret_from_fork_asm+0x11/0x20 [39106.895714] [39134.138776] watchdog: BUG: soft lockup - CPU#0 stuck for 25332s! [dbench:78683] [39134.146074] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39134.150776] watchdog: BUG: soft lockup - CPU#7 stuck for 34329s! [kworker/7:4:78808] [39134.201516] irq event stamp: 69345930 [39134.201518] hardirqs last enabled at (69345929): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39134.209245] Modules linked in: [39134.212903] hardirqs last disabled at (69345930): [] sysvec_apic_timer_interrupt+0xe/0x90 [39134.223061] ceph [39134.226111] softirqs last enabled at (69345882): [] __do_softirq+0x2be/0x371 [39134.235835] libceph [39134.237760] softirqs last disabled at (69345877): [] irq_exit_rcu+0xb0/0xd0 [39134.246452] dns_resolver fscache [39134.248638] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39134.257146] netfs veth [39134.260478] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39134.269421] nft_chain_nat [39134.271863] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39134.279334] xt_MASQUERADE nf_nat [39134.282039] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39134.287516] nf_conntrack nf_defrag_ipv6 [39134.290828] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39134.309564] nf_defrag_ipv4 [39134.313482] [39134.313483] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39134.318698] nft_compat nf_tables [39134.321491] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39134.322982] nfnetlink bridge [39134.330106] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39134.333415] stp llc [39134.340540] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39134.343504] xfs libcrc32c [39134.350629] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39134.352811] sunrpc intel_rapl_msr [39134.359936] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39134.362640] intel_rapl_common sb_edac [39134.369763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39134.373173] x86_pkg_temp_thermal [39134.381248] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39134.384991] coretemp kvm_intel [39134.390730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39134.394039] kvm irqbypass [39134.401173] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39134.404309] i2c_i801 [39134.411435] Call Trace: [39134.411437] [39134.414138] iTCO_wdt iTCO_vendor_support [39134.421263] ? watchdog_timer_fn+0x1b4/0x220 [39134.423534] i2c_smbus [39134.425980] ? __hrtimer_run_queues+0x1d3/0x360 [39134.427987] crct10dif_pclmul joydev [39134.431995] ? hrtimer_interrupt+0x104/0x240 [39134.436257] crc32_pclmul ipmi_ssif [39134.438613] ? __local_bh_enable+0x50/0x60 [39134.443136] ghash_clmulni_intel [39134.446712] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39134.450972] mei_me mei [39134.454456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39134.458547] lpc_ich sha512_ssse3 [39134.461770] [39134.466989] acpi_ipmi mfd_core [39134.469432] [39134.469433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39134.474389] wmi ipmi_si ipmi_devintf [39134.477706] ? queued_spin_lock_slowpath+0x16/0x2e0 [39134.479798] ipmi_msghandler ioatdma [39134.482937] do_raw_spin_lock+0xb5/0xc0 [39134.485033] acpi_pad acpi_power_meter [39134.490338] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39134.493994] ixgbe igb mdio [39134.498869] try_get_cap_refs+0x159/0x880 [ceph] [39134.502435] crc32c_intel nvme [39134.506288] __ceph_get_caps+0x4a1/0x760 [ceph] [39134.510027] ptp nvme_core [39134.514816] ? __file_remove_privs+0x89/0x110 [39134.517604] i2c_algo_bit pps_core [39134.522214] ? wake_bit_function+0x70/0x70 [39134.525264] dca fuse [39134.529791] ? ceph_write_iter+0x260/0x770 [ceph] [39134.532492] [39134.532493] irq event stamp: 233384536 [39134.536843] ceph_write_iter+0x260/0x770 [ceph] [39134.540242] hardirqs last enabled at (233384535): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39134.544336] ? lock_is_held_type+0xaa/0x120 [39134.546601] hardirqs last disabled at (233384536): [] sysvec_apic_timer_interrupt+0xe/0x90 [39134.551303] vfs_write+0x324/0x4f0 [39134.552791] softirqs last enabled at (233384534): [] __do_softirq+0x2be/0x371 [39134.556538] ksys_pwrite64+0x65/0xa0 [39134.561059] softirqs last disabled at (233384529): [] irq_exit_rcu+0xb0/0xd0 [39134.571302] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39134.575482] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39134.585292] do_syscall_64+0x38/0x80 [39134.588689] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39134.597460] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39134.601029] Workqueue: events delayed_work [ceph] [39134.609626] RIP: 0033:0x7f48f0eeb93b [39134.614757] [39134.624136] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39134.627707] RIP: 0010:lock_acquire+0xf0/0x2b0 [39134.635178] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39134.640221] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39134.644920] ORIG_RAX: 0000000000000012 [39134.648489] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39134.649980] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39134.668717] [39134.668718] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39134.673069] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39134.678287] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39134.697024] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39134.700854] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39134.706071] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39134.713196] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39134.714687] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39134.721811] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39134.728941] [39134.736058] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39134.789067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39134.794802] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39134.801926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39134.809050] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39134.816181] Call Trace: [39134.818630] [39134.820640] ? watchdog_timer_fn+0x1b4/0x220 [39134.824904] ? __hrtimer_run_queues+0x1d3/0x360 [39134.829430] ? hrtimer_interrupt+0x104/0x240 [39134.833699] ? __local_bh_enable+0x50/0x60 [39134.837793] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39134.843016] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39134.847974] [39134.850071] [39134.852181] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39134.857494] ? lock_acquire+0xf0/0x2b0 [39134.861244] ? igrab+0x43/0x50 [39134.864294] ? lock_release+0xc5/0x260 [39134.868038] _raw_spin_lock+0x30/0x40 [39134.871695] ? igrab+0x1d/0x50 [39134.874747] igrab+0x1d/0x50 [39134.877623] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39134.882774] delayed_work+0x1d8/0x280 [ceph] [39134.887055] ? process_scheduled_works+0x22a/0x510 [39134.891845] process_scheduled_works+0x2a4/0x510 [39134.896473] worker_thread+0x188/0x340 [39134.900219] ? rescuer_thread+0x310/0x310 [39134.904221] kthread+0xf5/0x130 [39134.907359] ? kthread_complete_and_exit+0x20/0x20 [39134.912144] ret_from_fork+0x31/0x40 [39134.915713] ? kthread_complete_and_exit+0x20/0x20 [39134.920498] ret_from_fork_asm+0x11/0x20 [39134.924422] [39162.139256] watchdog: BUG: soft lockup - CPU#0 stuck for 25358s! [dbench:78683] [39162.146564] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39162.151255] watchdog: BUG: soft lockup - CPU#7 stuck for 34355s! [kworker/7:4:78808] [39162.202011] irq event stamp: 69404244 [39162.209742] Modules linked in: ceph libceph dns_resolver fscache netfs veth [39162.213400] hardirqs last enabled at (69404243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39162.213406] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [39162.230516] hardirqs last disabled at (69404244): [] sysvec_apic_timer_interrupt+0xe/0x90 [39162.238766] nft_compat nf_tables nfnetlink bridge [39162.248490] softirqs last enabled at (69404196): [] __do_softirq+0x2be/0x371 [39162.253276] stp llc xfs libcrc32c sunrpc intel_rapl_msr [39162.261959] softirqs last disabled at (69404191): [] irq_exit_rcu+0xb0/0xd0 [39162.267264] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [39162.275777] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39162.282118] kvm_intel kvm irqbypass [39162.291063] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39162.291065] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39162.294649] i2c_i801 iTCO_wdt [39162.302121] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39162.307598] iTCO_vendor_support [39162.310653] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39162.329387] i2c_smbus crct10dif_pclmul [39162.332613] [39162.337828] joydev [39162.341659] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39162.343149] crc32_pclmul [39162.345247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39162.352371] ipmi_ssif ghash_clmulni_intel [39162.354990] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39162.362113] mei_me mei [39162.366205] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39162.373329] lpc_ich [39162.375773] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39162.382897] sha512_ssse3 acpi_ipmi [39162.385080] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39162.392205] mfd_core [39162.395688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39162.403765] wmi ipmi_si [39162.406037] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39162.411775] ipmi_devintf ipmi_msghandler [39162.414305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39162.421428] ioatdma acpi_pad [39162.425433] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39162.432558] acpi_power_meter ixgbe [39162.435521] Call Trace: [39162.435523] [39162.442653] igb mdio [39162.446142] ? watchdog_timer_fn+0x1b4/0x220 [39162.448583] crc32c_intel nvme [39162.450595] ? __hrtimer_run_queues+0x1d3/0x360 [39162.452863] ptp nvme_core [39162.457132] ? hrtimer_interrupt+0x104/0x240 [39162.460180] i2c_algo_bit pps_core [39162.464702] ? __local_bh_enable+0x50/0x60 [39162.467408] dca fuse [39162.471674] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39162.475070] [39162.475070] irq event stamp: 233605336 [39162.479161] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39162.481431] hardirqs last enabled at (233605335): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39162.486653] [39162.488147] hardirqs last disabled at (233605336): [] sysvec_apic_timer_interrupt+0xe/0x90 [39162.491891] [39162.496847] softirqs last enabled at (233605334): [] __do_softirq+0x2be/0x371 [39162.507093] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39162.509190] softirqs last disabled at (233605329): [] irq_exit_rcu+0xb0/0xd0 [39162.519005] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39162.521099] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39162.529871] do_raw_spin_lock+0xb5/0xc0 [39162.535172] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39162.543773] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39162.548658] Workqueue: events delayed_work [ceph] [39162.558039] try_get_cap_refs+0x159/0x880 [ceph] [39162.561866] [39162.569344] __ceph_get_caps+0x4a1/0x760 [ceph] [39162.574122] RIP: 0010:lock_acquire+0xf0/0x2b0 [39162.578826] ? __file_remove_privs+0x89/0x110 [39162.583429] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39162.584921] ? wake_bit_function+0x70/0x70 [39162.589463] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39162.593815] ? ceph_write_iter+0x260/0x770 [ceph] [39162.598162] [39162.598163] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39162.616901] ceph_write_iter+0x260/0x770 [ceph] [39162.620992] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39162.626214] ? lock_is_held_type+0xaa/0x120 [39162.630907] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39162.632400] vfs_write+0x324/0x4f0 [39162.639522] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39162.644050] ksys_pwrite64+0x65/0xa0 [39162.651168] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39162.655347] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39162.662471] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39162.665871] do_syscall_64+0x38/0x80 [39162.672993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39162.676564] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39162.683688] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39162.688817] RIP: 0033:0x7f48f0eeb93b [39162.696896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39162.700469] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39162.706205] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39162.711247] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39162.718372] Call Trace: [39162.718374] [39162.721942] ORIG_RAX: 0000000000000012 [39162.729069] ? watchdog_timer_fn+0x1b4/0x220 [39162.747805] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39162.754929] ? __hrtimer_run_queues+0x1d3/0x360 [39162.760163] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39162.762613] ? hrtimer_interrupt+0x104/0x240 [39162.764618] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39162.768449] ? __local_bh_enable+0x50/0x60 [39162.772712] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39162.779840] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39162.784362] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39162.791487] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39162.795754] [39162.802873] [39162.835670] [39162.837768] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39162.843074] ? lock_acquire+0xf0/0x2b0 [39162.846824] ? igrab+0x43/0x50 [39162.849874] ? lock_release+0xc5/0x260 [39162.853621] _raw_spin_lock+0x30/0x40 [39162.857286] ? igrab+0x1d/0x50 [39162.860335] igrab+0x1d/0x50 [39162.863215] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39162.868363] delayed_work+0x1d8/0x280 [ceph] [39162.872664] ? process_scheduled_works+0x22a/0x510 [39162.877451] process_scheduled_works+0x2a4/0x510 [39162.882069] worker_thread+0x188/0x340 [39162.885818] ? rescuer_thread+0x310/0x310 [39162.889828] kthread+0xf5/0x130 [39162.892967] ? kthread_complete_and_exit+0x20/0x20 [39162.897751] ret_from_fork+0x31/0x40 [39162.901330] ? kthread_complete_and_exit+0x20/0x20 [39162.906113] ret_from_fork_asm+0x11/0x20 [39162.910037] [39164.562664] rcu: INFO: rcu_preempt self-detected stall on CPU [39164.568420] rcu: 7-....: (26435724 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/26434357 fqs=13328558 [39164.579098] rcu: (t=27227476 jiffies g=1309609 q=736179 ncpus=8) [39164.585182] Sending NMI from CPU 7 to CPUs 0: [39164.589533] NMI backtrace for cpu 0 [39164.589535] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39164.589536] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39164.589537] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39164.589540] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39164.589541] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39164.589543] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39164.589544] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39164.589544] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39164.589545] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39164.589546] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39164.589546] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39164.589547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39164.589548] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39164.589549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39164.589550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39164.589550] Call Trace: [39164.589551] [39164.589552] ? nmi_cpu_backtrace+0xad/0x120 [39164.589555] ? nmi_cpu_backtrace_handler+0x11/0x20 [39164.589557] ? nmi_handle+0xd4/0x260 [39164.589561] ? default_do_nmi+0x49/0xf0 [39164.589563] ? exc_nmi+0x11e/0x150 [39164.589564] ? end_repeat_nmi+0x16/0x67 [39164.589568] ? queued_spin_lock_slowpath+0x16/0x2e0 [39164.589571] ? queued_spin_lock_slowpath+0x16/0x2e0 [39164.589573] ? queued_spin_lock_slowpath+0x16/0x2e0 [39164.589575] [39164.589576] [39164.589577] do_raw_spin_lock+0xb5/0xc0 [39164.589579] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39164.589593] try_get_cap_refs+0x159/0x880 [ceph] [39164.589606] __ceph_get_caps+0x4a1/0x760 [ceph] [39164.589618] ? __file_remove_privs+0x89/0x110 [39164.589620] ? wake_bit_function+0x70/0x70 [39164.589622] ? ceph_write_iter+0x260/0x770 [ceph] [39164.589633] ceph_write_iter+0x260/0x770 [ceph] [39164.589644] ? lock_is_held_type+0xaa/0x120 [39164.589646] vfs_write+0x324/0x4f0 [39164.589650] ksys_pwrite64+0x65/0xa0 [39164.589652] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39164.589655] do_syscall_64+0x38/0x80 [39164.589657] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39164.589658] RIP: 0033:0x7f48f0eeb93b [39164.589660] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39164.589661] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [39164.589662] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39164.589663] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39164.589664] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39164.589665] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39164.589667] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39164.589688] [39164.590532] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39164.894075] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39164.901548] Workqueue: events delayed_work [ceph] [39164.906271] RIP: 0010:lock_acquire+0xf0/0x2b0 [39164.910628] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39164.929368] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39164.934593] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39164.941718] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39164.948842] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39164.955981] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39164.963108] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39164.970231] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39164.978308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39164.984045] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39164.991171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39164.998293] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39165.005418] Call Trace: [39165.007862] [39165.009874] ? rcu_dump_cpu_stacks+0xd1/0x180 [39165.014235] ? rcu_sched_clock_irq+0x633/0x1390 [39165.018767] ? lock_release+0xc5/0x260 [39165.022510] ? lock_is_held_type+0xaa/0x120 [39165.026702] ? lock_is_held_type+0xaa/0x120 [39165.030886] ? tick_sched_do_timer+0xa0/0xa0 [39165.035155] ? rcu_read_lock_sched_held+0x56/0x90 [39165.039864] ? tick_sched_do_timer+0xa0/0xa0 [39165.044133] ? update_process_times+0x5f/0x90 [39165.048485] ? tick_sched_handle.isra.16+0x1e/0x50 [39165.053277] ? tick_sched_timer+0x6f/0x80 [39165.057283] ? __hrtimer_run_queues+0x1d3/0x360 [39165.061822] ? hrtimer_interrupt+0x104/0x240 [39165.066085] ? __local_bh_enable+0x50/0x60 [39165.070183] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39165.075405] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39165.080370] [39165.082467] [39165.084565] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39165.089886] ? lock_acquire+0xf0/0x2b0 [39165.093632] ? igrab+0x43/0x50 [39165.096702] ? lock_release+0xc5/0x260 [39165.100454] _raw_spin_lock+0x30/0x40 [39165.104115] ? igrab+0x1d/0x50 [39165.107168] igrab+0x1d/0x50 [39165.110045] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39165.115199] delayed_work+0x1d8/0x280 [ceph] [39165.119487] ? process_scheduled_works+0x22a/0x510 [39165.124274] process_scheduled_works+0x2a4/0x510 [39165.128896] worker_thread+0x188/0x340 [39165.132649] ? rescuer_thread+0x310/0x310 [39165.136660] kthread+0xf5/0x130 [39165.139798] ? kthread_complete_and_exit+0x20/0x20 [39165.144593] ret_from_fork+0x31/0x40 [39165.148170] ? kthread_complete_and_exit+0x20/0x20 [39165.152955] ret_from_fork_asm+0x11/0x20 [39165.156894] [39190.139735] watchdog: BUG: soft lockup - CPU#0 stuck for 25384s! [dbench:78683] [39190.147040] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39190.151735] watchdog: BUG: soft lockup - CPU#7 stuck for 34381s! [kworker/7:4:78808] [39190.202481] irq event stamp: 69462340 [39190.202482] hardirqs last enabled at (69462339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39190.210212] Modules linked in: [39190.213870] hardirqs last disabled at (69462340): [] sysvec_apic_timer_interrupt+0xe/0x90 [39190.224028] ceph [39190.227079] softirqs last enabled at (69462296): [] __do_softirq+0x2be/0x371 [39190.236803] libceph [39190.238727] softirqs last disabled at (69462291): [] irq_exit_rcu+0xb0/0xd0 [39190.247412] dns_resolver [39190.249594] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39190.258106] fscache netfs [39190.260740] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39190.269685] veth nft_chain_nat [39190.272388] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39190.279876] xt_MASQUERADE [39190.283015] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39190.288491] nf_nat nf_conntrack [39190.291194] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39190.309933] nf_defrag_ipv6 [39190.313157] [39190.313158] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39190.318375] nf_defrag_ipv4 nft_compat [39190.321164] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39190.322655] nf_tables nfnetlink [39190.329780] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39190.333523] bridge stp [39190.340648] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39190.343873] llc xfs [39190.350996] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39190.353440] libcrc32c sunrpc [39190.360564] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39190.362749] intel_rapl_msr intel_rapl_common [39190.369874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39190.372837] sb_edac x86_pkg_temp_thermal [39190.380915] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39190.385265] coretemp kvm_intel [39190.391003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39190.395006] kvm irqbypass [39190.402132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39190.405268] i2c_i801 iTCO_wdt [39190.412392] Call Trace: [39190.412393] [39190.415112] iTCO_vendor_support [39190.422256] ? watchdog_timer_fn+0x1b4/0x220 [39190.425305] i2c_smbus crct10dif_pclmul [39190.427750] ? __hrtimer_run_queues+0x1d3/0x360 [39190.429761] joydev [39190.432989] ? hrtimer_interrupt+0x104/0x240 [39190.437247] crc32_pclmul ipmi_ssif [39190.441080] ? __local_bh_enable+0x50/0x60 [39190.445604] ghash_clmulni_intel [39190.447704] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39190.451965] mei_me mei [39190.455450] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39190.459539] lpc_ich sha512_ssse3 [39190.462764] [39190.467980] acpi_ipmi [39190.470424] [39190.475383] mfd_core wmi ipmi_si [39190.478713] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39190.480807] ipmi_devintf ipmi_msghandler [39190.483169] ? queued_spin_lock_slowpath+0x18/0x2e0 [39190.485263] ioatdma acpi_pad [39190.488576] do_raw_spin_lock+0xb5/0xc0 [39190.493878] acpi_power_meter ixgbe [39190.497883] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39190.502753] igb mdio [39190.505717] try_get_cap_refs+0x159/0x880 [ceph] [39190.509547] crc32c_intel nvme [39190.513037] __ceph_get_caps+0x4a1/0x760 [ceph] [39190.517816] ptp nvme_core [39190.520091] ? __file_remove_privs+0x89/0x110 [39190.524696] i2c_algo_bit pps_core [39190.527747] ? wake_bit_function+0x70/0x70 [39190.532271] dca fuse [39190.534979] ? ceph_write_iter+0x260/0x770 [ceph] [39190.539326] [39190.539326] irq event stamp: 233821348 [39190.542724] ceph_write_iter+0x260/0x770 [ceph] [39190.546815] hardirqs last enabled at (233821347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39190.549089] ? lock_is_held_type+0xaa/0x120 [39190.553784] hardirqs last disabled at (233821348): [] sysvec_apic_timer_interrupt+0xe/0x90 [39190.555276] vfs_write+0x324/0x4f0 [39190.559018] softirqs last enabled at (233821346): [] __do_softirq+0x2be/0x371 [39190.563547] ksys_pwrite64+0x65/0xa0 [39190.573786] softirqs last disabled at (233821341): [] irq_exit_rcu+0xb0/0xd0 [39190.577962] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39190.587773] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39190.591172] do_syscall_64+0x38/0x80 [39190.599941] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39190.603513] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39190.612110] Workqueue: events delayed_work [ceph] [39190.617259] RIP: 0033:0x7f48f0eeb93b [39190.626637] [39190.630207] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39190.637676] RIP: 0010:lock_acquire+0x4a/0x2b0 [39190.642721] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39190.647418] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [39190.650989] ORIG_RAX: 0000000000000012 [39190.652480] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [39190.671219] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39190.675570] [39190.675570] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [39190.680787] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39190.699523] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [39190.703354] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39190.708572] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [39190.715698] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39190.717188] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [39190.724311] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39190.731437] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39190.738565] [39190.745683] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39190.791567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39190.797304] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39190.804428] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39190.811551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39190.818677] Call Trace: [39190.821137] [39190.823149] ? watchdog_timer_fn+0x1b4/0x220 [39190.827423] ? __hrtimer_run_queues+0x1d3/0x360 [39190.831948] ? hrtimer_interrupt+0x104/0x240 [39190.836208] ? __local_bh_enable+0x50/0x60 [39190.840302] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39190.845527] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39190.850484] [39190.852581] [39190.854680] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39190.859987] ? lock_acquire+0x4a/0x2b0 [39190.863737] ? igrab+0x43/0x50 [39190.866786] ? lock_release+0xc5/0x260 [39190.870532] _raw_spin_lock+0x30/0x40 [39190.874197] ? igrab+0x1d/0x50 [39190.877246] igrab+0x1d/0x50 [39190.880141] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39190.885291] delayed_work+0x1d8/0x280 [ceph] [39190.889573] ? process_scheduled_works+0x22a/0x510 [39190.894364] process_scheduled_works+0x2a4/0x510 [39190.898980] worker_thread+0x188/0x340 [39190.902728] ? rescuer_thread+0x310/0x310 [39190.906732] kthread+0xf5/0x130 [39190.909868] ? kthread_complete_and_exit+0x20/0x20 [39190.914653] ret_from_fork+0x31/0x40 [39190.918223] ? kthread_complete_and_exit+0x20/0x20 [39190.923008] ret_from_fork_asm+0x11/0x20 [39190.926932] [39218.140220] watchdog: BUG: soft lockup - CPU#0 stuck for 25410s! [dbench:78683] [39218.147521] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39218.152220] watchdog: BUG: soft lockup - CPU#7 stuck for 34407s! [kworker/7:4:78808] [39218.202970] irq event stamp: 69520554 [39218.210700] Modules linked in: ceph libceph dns_resolver fscache netfs [39218.214357] hardirqs last enabled at (69520553): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39218.220894] veth nft_chain_nat xt_MASQUERADE nf_nat [39218.231049] hardirqs last disabled at (69520554): [] sysvec_apic_timer_interrupt+0xe/0x90 [39218.231052] nf_conntrack [39218.231053] softirqs last enabled at (69520516): [] __do_softirq+0x2be/0x371 [39218.236006] nf_defrag_ipv6 [39218.245732] softirqs last disabled at (69520511): [] irq_exit_rcu+0xb0/0xd0 [39218.248348] nf_defrag_ipv4 [39218.257034] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39218.259824] nft_compat nf_tables [39218.268333] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39218.271124] nfnetlink bridge [39218.280070] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39218.283380] stp llc [39218.290852] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39218.293816] xfs libcrc32c [39218.299291] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39218.301477] sunrpc intel_rapl_msr [39218.320215] [39218.320216] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39218.322917] intel_rapl_common [39218.328135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39218.331532] sb_edac x86_pkg_temp_thermal [39218.333023] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39218.340148] coretemp [39218.343198] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39218.350324] kvm_intel kvm [39218.354326] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39218.361452] irqbypass i2c_i801 [39218.363722] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39218.370847] iTCO_wdt iTCO_vendor_support [39218.373548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39218.380675] i2c_smbus [39218.383811] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39218.391888] crct10dif_pclmul joydev [39218.395911] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39218.401648] crc32_pclmul [39218.404005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39218.411129] ipmi_ssif [39218.414700] Call Trace: [39218.414703] [39218.421825] ghash_clmulni_intel mei_me [39218.424443] ? watchdog_timer_fn+0x1b4/0x220 [39218.431564] mei lpc_ich [39218.433923] ? __hrtimer_run_queues+0x1d3/0x360 [39218.436366] sha512_ssse3 acpi_ipmi [39218.438380] ? hrtimer_interrupt+0x104/0x240 [39218.442210] mfd_core wmi [39218.446474] ? __local_bh_enable+0x50/0x60 [39218.449003] ipmi_si ipmi_devintf [39218.453530] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39218.457011] ipmi_msghandler ioatdma [39218.461275] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39218.463898] acpi_pad acpi_power_meter [39218.467993] [39218.471302] ixgbe igb [39218.476519] [39218.480091] mdio crc32c_intel [39218.485049] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39218.488792] nvme ptp [39218.490910] ? queued_spin_lock_slowpath+0x16/0x2e0 [39218.493265] nvme_core [39218.495363] do_raw_spin_lock+0xb5/0xc0 [39218.498412] i2c_algo_bit pps_core [39218.503718] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39218.505989] dca [39218.510863] try_get_cap_refs+0x159/0x880 [ceph] [39218.513215] fuse [39218.517053] __ceph_get_caps+0x4a1/0x760 [ceph] [39218.520443] [39218.520444] irq event stamp: 234042116 [39218.525234] ? __file_remove_privs+0x89/0x110 [39218.527064] hardirqs last enabled at (234042115): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39218.531677] ? wake_bit_function+0x70/0x70 [39218.533616] hardirqs last disabled at (234042116): [] sysvec_apic_timer_interrupt+0xe/0x90 [39218.538144] ? ceph_write_iter+0x260/0x770 [ceph] [39218.539633] softirqs last enabled at (234042114): [] __do_softirq+0x2be/0x371 [39218.543377] ceph_write_iter+0x260/0x770 [ceph] [39218.547726] softirqs last disabled at (234042109): [] irq_exit_rcu+0xb0/0xd0 [39218.557977] ? lock_is_held_type+0xaa/0x120 [39218.562062] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39218.571875] vfs_write+0x324/0x4f0 [39218.576569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39218.585346] ksys_pwrite64+0x65/0xa0 [39218.589866] Workqueue: events delayed_work [ceph] [39218.598463] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39218.602641] [39218.602642] RIP: 0010:lock_release+0x14e/0x260 [39218.612018] do_syscall_64+0x38/0x80 [39218.615414] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [39218.622903] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39218.626473] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [39218.631173] RIP: 0033:0x7f48f0eeb93b [39218.636302] [39218.636303] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [39218.637794] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39218.642231] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [39218.645801] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39218.664538] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39218.669583] ORIG_RAX: 0000000000000012 [39218.674801] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39218.678369] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39218.679862] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [39218.686984] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39218.705724] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39218.712847] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39218.718066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39218.725189] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39218.729019] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39218.736145] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39218.743269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39218.750398] [39218.757516] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39218.816260] Call Trace: [39218.818706] [39218.820717] ? watchdog_timer_fn+0x1b4/0x220 [39218.824981] ? __hrtimer_run_queues+0x1d3/0x360 [39218.829506] ? hrtimer_interrupt+0x104/0x240 [39218.833775] ? __local_bh_enable+0x50/0x60 [39218.837869] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39218.843093] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39218.848051] [39218.850147] [39218.852245] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39218.857550] ? igrab+0x43/0x50 [39218.860622] ? lock_release+0x14e/0x260 [39218.864457] ? lock_release+0xc5/0x260 [39218.868202] _raw_spin_unlock+0x1b/0x40 [39218.872041] igrab+0x43/0x50 [39218.874919] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39218.880067] delayed_work+0x1d8/0x280 [ceph] [39218.884349] ? process_scheduled_works+0x22a/0x510 [39218.889140] process_scheduled_works+0x2a4/0x510 [39218.893757] worker_thread+0x188/0x340 [39218.897505] ? rescuer_thread+0x310/0x310 [39218.901506] kthread+0xf5/0x130 [39218.904647] ? kthread_complete_and_exit+0x20/0x20 [39218.909439] ret_from_fork+0x31/0x40 [39218.913018] ? kthread_complete_and_exit+0x20/0x20 [39218.917802] ret_from_fork_asm+0x11/0x20 [39218.921733] [39246.140705] watchdog: BUG: soft lockup - CPU#0 stuck for 25436s! [dbench:78683] [39246.148008] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39246.152705] watchdog: BUG: soft lockup - CPU#7 stuck for 34433s! [kworker/7:4:78808] [39246.203448] irq event stamp: 69578752 [39246.203449] hardirqs last enabled at (69578751): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39246.211180] Modules linked in: [39246.214837] hardirqs last disabled at (69578752): [] sysvec_apic_timer_interrupt+0xe/0x90 [39246.224994] ceph libceph [39246.228044] softirqs last enabled at (69578714): [] __do_softirq+0x2be/0x371 [39246.237770] dns_resolver fscache [39246.240385] softirqs last disabled at (69578709): [] irq_exit_rcu+0xb0/0xd0 [39246.249069] netfs [39246.252396] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39246.260908] veth nft_chain_nat [39246.262919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39246.271863] xt_MASQUERADE [39246.275002] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39246.282473] nf_nat nf_conntrack [39246.285177] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39246.290654] nf_defrag_ipv6 [39246.293877] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39246.312615] nf_defrag_ipv4 nft_compat [39246.315406] [39246.315407] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39246.320624] nf_tables [39246.324380] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39246.325866] nfnetlink bridge [39246.332991] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39246.335349] stp llc [39246.342472] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39246.345436] xfs [39246.352561] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39246.354744] libcrc32c sunrpc [39246.361870] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39246.363705] intel_rapl_msr [39246.370831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39246.373795] intel_rapl_common [39246.381873] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39246.384662] sb_edac x86_pkg_temp_thermal [39246.390401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39246.393451] coretemp kvm_intel [39246.400575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39246.404580] kvm irqbypass [39246.411702] Call Trace: [39246.414841] i2c_i801 iTCO_wdt [39246.421965] [39246.424670] iTCO_vendor_support i2c_smbus [39246.427113] ? watchdog_timer_fn+0x1b4/0x220 [39246.430163] crct10dif_pclmul joydev [39246.432175] ? __hrtimer_run_queues+0x1d3/0x360 [39246.436266] crc32_pclmul ipmi_ssif [39246.440533] ? hrtimer_interrupt+0x104/0x240 [39246.444103] ghash_clmulni_intel [39246.448625] ? __local_bh_enable+0x50/0x60 [39246.452107] mei_me mei [39246.456392] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39246.459614] lpc_ich [39246.463704] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39246.466148] sha512_ssse3 acpi_ipmi [39246.471382] [39246.473566] mfd_core wmi [39246.478524] [39246.482008] ipmi_si ipmi_devintf [39246.484108] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39246.486722] ipmi_msghandler ioatdma [39246.488825] ? queued_spin_lock_slowpath+0x18/0x2e0 [39246.492131] acpi_pad [39246.497438] do_raw_spin_lock+0xb5/0xc0 [39246.501007] acpi_power_meter ixgbe [39246.505878] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39246.508149] igb mdio [39246.511981] try_get_cap_refs+0x159/0x880 [ceph] [39246.515462] crc32c_intel nvme [39246.520251] __ceph_get_caps+0x4a1/0x760 [ceph] [39246.522517] ptp nvme_core [39246.527133] ? __file_remove_privs+0x89/0x110 [39246.530178] i2c_algo_bit pps_core [39246.534704] ? wake_bit_function+0x70/0x70 [39246.537407] dca fuse [39246.541760] ? ceph_write_iter+0x260/0x770 [ceph] [39246.545156] [39246.545157] irq event stamp: 234262918 [39246.549247] ceph_write_iter+0x260/0x770 [ceph] [39246.551517] hardirqs last enabled at (234262917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39246.556218] ? lock_is_held_type+0xaa/0x120 [39246.557705] hardirqs last disabled at (234262918): [] sysvec_apic_timer_interrupt+0xe/0x90 [39246.561450] vfs_write+0x324/0x4f0 [39246.565973] softirqs last enabled at (234262916): [] __do_softirq+0x2be/0x371 [39246.576221] ksys_pwrite64+0x65/0xa0 [39246.580393] softirqs last disabled at (234262911): [] irq_exit_rcu+0xb0/0xd0 [39246.590204] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39246.593604] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39246.602374] do_syscall_64+0x38/0x80 [39246.605945] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39246.605947] Workqueue: events delayed_work [ceph] [39246.614542] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39246.619674] [39246.619675] RIP: 0010:lock_acquire+0x16/0x2b0 [39246.629050] RIP: 0033:0x7f48f0eeb93b [39246.632620] Code: 00 5b c3 c3 0f 0b 65 c7 05 43 da ea 7e 00 00 00 00 eb e3 90 f3 0f 1e fa 41 57 4d 89 ca 41 89 f7 41 56 41 89 d6 41 55 41 89 cd <41> 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 [39246.640108] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39246.644805] RSP: 0018:ffffc90002117d68 EFLAGS: 00000246 [39246.649851] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39246.651342] [39246.651343] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [39246.655693] ORIG_RAX: 0000000000000012 [39246.659263] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [39246.678000] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39246.696738] RBP: ffff8881a1d34500 R08: 0000000000000001 R09: 0000000000000000 [39246.701955] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39246.707173] R10: 0000000000000000 R11: 0000000000000001 R12: 000000000000ea60 [39246.708664] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39246.715788] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39246.719619] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39246.726742] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39246.733868] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39246.740990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39246.748120] [39246.755239] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39246.806860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39246.813983] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39246.821108] Call Trace: [39246.823551] [39246.825563] ? watchdog_timer_fn+0x1b4/0x220 [39246.829828] ? __hrtimer_run_queues+0x1d3/0x360 [39246.834353] ? hrtimer_interrupt+0x104/0x240 [39246.838614] ? __local_bh_enable+0x50/0x60 [39246.842707] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39246.847932] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39246.852890] [39246.854985] [39246.857085] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39246.862417] ? lock_acquire+0x16/0x2b0 [39246.866168] _raw_spin_lock+0x30/0x40 [39246.869833] ? igrab+0x1d/0x50 [39246.872883] igrab+0x1d/0x50 [39246.875762] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39246.880911] delayed_work+0x1d8/0x280 [ceph] [39246.885192] ? process_scheduled_works+0x22a/0x510 [39246.889982] process_scheduled_works+0x2a4/0x510 [39246.894599] worker_thread+0x188/0x340 [39246.898347] ? rescuer_thread+0x310/0x310 [39246.902351] kthread+0xf5/0x130 [39246.905487] ? kthread_complete_and_exit+0x20/0x20 [39246.910274] ret_from_fork+0x31/0x40 [39246.913851] ? kthread_complete_and_exit+0x20/0x20 [39246.918636] ret_from_fork_asm+0x11/0x20 [39246.922559] [39274.141193] watchdog: BUG: soft lockup - CPU#0 stuck for 25462s! [dbench:78683] [39274.148495] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39274.153192] watchdog: BUG: soft lockup - CPU#7 stuck for 34459s! [kworker/7:4:78808] [39274.203945] irq event stamp: 69636942 [39274.203946] hardirqs last enabled at (69636941): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39274.211675] Modules linked in: [39274.215332] hardirqs last disabled at (69636942): [] sysvec_apic_timer_interrupt+0xe/0x90 [39274.225491] ceph [39274.228541] softirqs last enabled at (69636904): [] __do_softirq+0x2be/0x371 [39274.238265] libceph [39274.240188] softirqs last disabled at (69636899): [] irq_exit_rcu+0xb0/0xd0 [39274.248872] dns_resolver [39274.251058] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39274.259568] fscache netfs [39274.262203] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39274.271147] veth nft_chain_nat [39274.273867] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39274.281340] xt_MASQUERADE [39274.284477] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39274.289954] nf_nat nf_conntrack [39274.292658] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39274.311395] nf_defrag_ipv6 [39274.314619] [39274.314620] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39274.319838] nf_defrag_ipv4 [39274.322628] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39274.324119] nft_compat nf_tables [39274.331242] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39274.334033] nfnetlink [39274.341159] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39274.344469] bridge [39274.351592] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39274.353949] stp [39274.361074] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39274.363173] llc xfs [39274.370295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39274.372133] libcrc32c sunrpc [39274.380210] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39274.382396] intel_rapl_msr intel_rapl_common [39274.388133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39274.391096] sb_edac x86_pkg_temp_thermal [39274.398221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39274.402589] coretemp [39274.409712] Call Trace: [39274.409715] [39274.413716] kvm_intel kvm [39274.420842] ? watchdog_timer_fn+0x1b4/0x220 [39274.423110] irqbypass i2c_i801 [39274.425555] ? __hrtimer_run_queues+0x1d3/0x360 [39274.427585] iTCO_wdt iTCO_vendor_support [39274.430291] ? hrtimer_interrupt+0x104/0x240 [39274.434550] i2c_smbus crct10dif_pclmul [39274.437688] ? __local_bh_enable+0x50/0x60 [39274.442212] joydev [39274.446221] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39274.450482] crc32_pclmul [39274.454312] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39274.458402] ipmi_ssif ghash_clmulni_intel [39274.460499] [39274.465717] mei_me [39274.468334] [39274.468335] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39274.473293] mei lpc_ich [39274.477387] ? queued_spin_lock_slowpath+0x18/0x2e0 [39274.479479] sha512_ssse3 acpi_ipmi [39274.481593] do_raw_spin_lock+0xb5/0xc0 [39274.483693] mfd_core wmi [39274.488998] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39274.491528] ipmi_si ipmi_devintf [39274.496399] try_get_cap_refs+0x159/0x880 [ceph] [39274.499882] ipmi_msghandler ioatdma [39274.503719] __ceph_get_caps+0x4a1/0x760 [ceph] [39274.506329] acpi_pad acpi_power_meter [39274.511118] ? __file_remove_privs+0x89/0x110 [39274.514424] ixgbe [39274.519036] ? wake_bit_function+0x70/0x70 [39274.522605] igb mdio [39274.527134] ? ceph_write_iter+0x260/0x770 [ceph] [39274.530874] crc32c_intel nvme [39274.535226] ceph_write_iter+0x260/0x770 [ceph] [39274.537235] ptp nvme_core [39274.541330] ? lock_is_held_type+0xaa/0x120 [39274.543597] i2c_algo_bit [39274.548296] vfs_write+0x324/0x4f0 [39274.551346] pps_core [39274.555877] ksys_pwrite64+0x65/0xa0 [39274.558591] dca fuse [39274.562769] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39274.565385] [39274.565386] irq event stamp: 234484136 [39274.568784] do_syscall_64+0x38/0x80 [39274.571054] hardirqs last enabled at (234484135): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39274.574627] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39274.576895] hardirqs last disabled at (234484136): [] sysvec_apic_timer_interrupt+0xe/0x90 [39274.582027] RIP: 0033:0x7f48f0eeb93b [39274.583516] softirqs last enabled at (234484134): [] __do_softirq+0x2be/0x371 [39274.587260] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39274.590831] softirqs last disabled at (234484129): [] irq_exit_rcu+0xb0/0xd0 [39274.601077] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39274.606121] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39274.615931] ORIG_RAX: 0000000000000012 [39274.619502] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39274.628273] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39274.647011] Workqueue: events delayed_work [ceph] [39274.655607] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39274.660826] [39274.660827] RIP: 0010:lock_acquire+0x4a/0x2b0 [39274.670203] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39274.674034] Code: 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 0f 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 <0f> 82 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b [39274.681503] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39274.688627] RSP: 0018:ffffc90002117d18 EFLAGS: 00000247 [39274.693326] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39274.700449] [39274.700450] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 0000000000000000 [39274.701946] [39274.706292] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [39274.769542] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [39274.776667] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [39274.783791] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39274.790913] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39274.798991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39274.804728] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39274.811852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39274.818995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39274.826120] Call Trace: [39274.828565] [39274.830593] ? watchdog_timer_fn+0x1b4/0x220 [39274.834872] ? __hrtimer_run_queues+0x1d3/0x360 [39274.839401] ? hrtimer_interrupt+0x104/0x240 [39274.843668] ? __local_bh_enable+0x50/0x60 [39274.847763] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39274.852986] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39274.857945] [39274.860040] [39274.862138] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39274.867446] ? lock_acquire+0x4a/0x2b0 [39274.871196] ? igrab+0x43/0x50 [39274.874247] ? lock_release+0xc5/0x260 [39274.877990] _raw_spin_lock+0x30/0x40 [39274.881648] ? igrab+0x1d/0x50 [39274.884699] igrab+0x1d/0x50 [39274.887597] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39274.892744] delayed_work+0x1d8/0x280 [ceph] [39274.897024] ? process_scheduled_works+0x22a/0x510 [39274.901814] process_scheduled_works+0x2a4/0x510 [39274.906432] worker_thread+0x188/0x340 [39274.910178] ? rescuer_thread+0x310/0x310 [39274.914183] kthread+0xf5/0x130 [39274.917331] ? kthread_complete_and_exit+0x20/0x20 [39274.922122] ret_from_fork+0x31/0x40 [39274.925703] ? kthread_complete_and_exit+0x20/0x20 [39274.930495] ret_from_fork_asm+0x11/0x20 [39274.934427] [39302.141679] watchdog: BUG: soft lockup - CPU#0 stuck for 25488s! [dbench:78683] [39302.148982] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39302.153679] watchdog: BUG: soft lockup - CPU#7 stuck for 34485s! [kworker/7:4:78808] [39302.204424] irq event stamp: 69695262 [39302.204425] hardirqs last enabled at (69695261): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39302.212154] Modules linked in: [39302.215812] hardirqs last disabled at (69695262): [] sysvec_apic_timer_interrupt+0xe/0x90 [39302.225969] ceph [39302.229019] softirqs last enabled at (69695226): [] __do_softirq+0x2be/0x371 [39302.238744] libceph [39302.240668] softirqs last disabled at (69695221): [] irq_exit_rcu+0xb0/0xd0 [39302.249357] dns_resolver fscache [39302.251544] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39302.260071] netfs veth [39302.263384] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39302.272327] nft_chain_nat [39302.274772] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39302.282243] xt_MASQUERADE [39302.284948] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39302.290425] nf_nat [39302.293128] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39302.311865] nf_conntrack [39302.313963] [39302.313964] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39302.319180] nf_defrag_ipv6 [39302.321798] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39302.323288] nf_defrag_ipv4 nft_compat [39302.330413] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39302.333204] nf_tables nfnetlink [39302.340328] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39302.344074] bridge [39302.351196] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39302.354421] stp llc [39302.361544] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39302.363641] xfs libcrc32c [39302.370766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39302.372949] sunrpc intel_rapl_msr [39302.381026] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39302.383732] intel_rapl_common [39302.389467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39302.392866] sb_edac x86_pkg_temp_thermal [39302.399990] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39302.403040] coretemp kvm_intel [39302.410164] Call Trace: [39302.410166] [39302.414169] kvm irqbypass [39302.421293] ? watchdog_timer_fn+0x1b4/0x220 [39302.424430] i2c_i801 [39302.426875] ? __hrtimer_run_queues+0x1d3/0x360 [39302.428885] iTCO_wdt iTCO_vendor_support [39302.431593] ? hrtimer_interrupt+0x104/0x240 [39302.435854] i2c_smbus crct10dif_pclmul [39302.438124] ? __local_bh_enable+0x50/0x60 [39302.442649] joydev crc32_pclmul [39302.446654] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39302.450916] ipmi_ssif ghash_clmulni_intel [39302.454746] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39302.458838] mei_me [39302.462078] [39302.467297] mei lpc_ich [39302.471386] [39302.471388] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39302.476352] sha512_ssse3 acpi_ipmi [39302.478456] ? queued_spin_lock_slowpath+0x16/0x2e0 [39302.480549] mfd_core wmi [39302.483080] do_raw_spin_lock+0xb5/0xc0 [39302.485177] ipmi_si ipmi_devintf [39302.490481] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39302.493963] ipmi_msghandler [39302.498837] try_get_cap_refs+0x159/0x880 [ceph] [39302.501453] ioatdma [39302.505288] __ceph_get_caps+0x4a1/0x760 [ceph] [39302.508595] acpi_pad acpi_power_meter [39302.513383] ? __file_remove_privs+0x89/0x110 [39302.516255] ixgbe igb [39302.520867] ? wake_bit_function+0x70/0x70 [39302.523051] mdio crc32c_intel [39302.527578] ? ceph_write_iter+0x260/0x770 [ceph] [39302.531318] nvme ptp [39302.535670] ceph_write_iter+0x260/0x770 [ceph] [39302.538027] nvme_core i2c_algo_bit [39302.542120] ? lock_is_held_type+0xaa/0x120 [39302.545167] pps_core dca [39302.549869] vfs_write+0x324/0x4f0 [39302.552135] fuse [39302.556664] ksys_pwrite64+0x65/0xa0 [39302.560144] irq event stamp: 234704900 [39302.564321] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39302.566939] hardirqs last enabled at (234704899): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39302.570339] do_syscall_64+0x38/0x80 [39302.572278] hardirqs last disabled at (234704900): [] sysvec_apic_timer_interrupt+0xe/0x90 [39302.575849] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39302.579593] softirqs last enabled at (234704898): [] __do_softirq+0x2be/0x371 [39302.584724] RIP: 0033:0x7f48f0eeb93b [39302.594968] softirqs last disabled at (234704889): [] irq_exit_rcu+0xb0/0xd0 [39302.598538] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39302.608349] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39302.613393] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39302.622164] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39302.625735] ORIG_RAX: 0000000000000012 [39302.634333] Workqueue: events delayed_work [ceph] [39302.653082] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39302.653084] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39302.662465] [39302.667682] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39302.675154] RIP: 0010:lock_acquire+0xf0/0x2b0 [39302.678983] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39302.683681] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39302.690805] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39302.697928] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39302.699426] [39302.706545] [39302.706546] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39302.759898] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39302.767021] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39302.774146] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39302.781270] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39302.788394] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39302.796471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39302.802209] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39302.809334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39302.816456] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39302.823582] Call Trace: [39302.826025] [39302.828038] ? watchdog_timer_fn+0x1b4/0x220 [39302.832302] ? __hrtimer_run_queues+0x1d3/0x360 [39302.836827] ? hrtimer_interrupt+0x104/0x240 [39302.841088] ? __local_bh_enable+0x50/0x60 [39302.845183] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39302.850407] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39302.855374] [39302.857468] [39302.859565] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39302.864875] ? lock_acquire+0xf0/0x2b0 [39302.868625] ? igrab+0x43/0x50 [39302.871675] ? lock_release+0xc5/0x260 [39302.875420] _raw_spin_lock+0x30/0x40 [39302.879083] ? igrab+0x1d/0x50 [39302.882135] igrab+0x1d/0x50 [39302.885014] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39302.890164] delayed_work+0x1d8/0x280 [ceph] [39302.894443] ? process_scheduled_works+0x22a/0x510 [39302.899225] process_scheduled_works+0x2a4/0x510 [39302.903840] worker_thread+0x188/0x340 [39302.907590] ? rescuer_thread+0x310/0x310 [39302.911593] kthread+0xf5/0x130 [39302.914732] ? kthread_complete_and_exit+0x20/0x20 [39302.919516] ret_from_fork+0x31/0x40 [39302.923094] ? kthread_complete_and_exit+0x20/0x20 [39302.927878] ret_from_fork_asm+0x11/0x20 [39302.931802] [39330.142166] watchdog: BUG: soft lockup - CPU#0 stuck for 25515s! [dbench:78683] [39330.149469] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39330.154166] watchdog: BUG: soft lockup - CPU#7 stuck for 34512s! [kworker/7:4:78808] [39330.204910] irq event stamp: 69753610 [39330.204911] hardirqs last enabled at (69753609): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39330.212640] Modules linked in: [39330.216298] hardirqs last disabled at (69753610): [] sysvec_apic_timer_interrupt+0xe/0x90 [39330.226456] ceph [39330.229507] softirqs last enabled at (69753572): [] __do_softirq+0x2be/0x371 [39330.239230] libceph [39330.241155] softirqs last disabled at (69753567): [] irq_exit_rcu+0xb0/0xd0 [39330.249843] dns_resolver fscache [39330.252023] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39330.260533] netfs veth [39330.263862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39330.272807] nft_chain_nat [39330.275249] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39330.282720] xt_MASQUERADE [39330.285425] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39330.290903] nf_nat [39330.293606] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39330.312345] nf_conntrack [39330.314441] [39330.314442] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39330.319660] nf_defrag_ipv6 nf_defrag_ipv4 [39330.322275] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39330.323766] nft_compat nf_tables [39330.330890] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39330.334983] nfnetlink bridge [39330.342105] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39330.345417] stp llc [39330.352541] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39330.355523] xfs libcrc32c [39330.362647] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39330.364843] sunrpc intel_rapl_msr [39330.371972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39330.374675] intel_rapl_common sb_edac [39330.382753] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39330.386152] x86_pkg_temp_thermal coretemp [39330.391888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39330.395632] kvm_intel [39330.402757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39330.406847] kvm irqbypass [39330.413971] Call Trace: [39330.413973] [39330.416328] i2c_i801 iTCO_wdt [39330.423454] ? watchdog_timer_fn+0x1b4/0x220 [39330.426157] iTCO_vendor_support i2c_smbus [39330.428602] ? __hrtimer_run_queues+0x1d3/0x360 [39330.430612] crct10dif_pclmul joydev [39330.433666] ? hrtimer_interrupt+0x104/0x240 [39330.437925] crc32_pclmul ipmi_ssif [39330.442017] ? __local_bh_enable+0x50/0x60 [39330.446541] ghash_clmulni_intel [39330.450130] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39330.454393] mei_me mei [39330.457877] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39330.461968] lpc_ich sha512_ssse3 [39330.465193] [39330.470410] acpi_ipmi [39330.472853] [39330.472854] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39330.477812] mfd_core wmi [39330.481126] ? queued_spin_lock_slowpath+0x18/0x2e0 [39330.483218] ipmi_si ipmi_devintf [39330.485578] do_raw_spin_lock+0xb5/0xc0 [39330.487673] ipmi_msghandler ioatdma [39330.492979] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39330.495595] acpi_pad acpi_power_meter [39330.500468] try_get_cap_refs+0x159/0x880 [ceph] [39330.503777] ixgbe igb [39330.507612] __ceph_get_caps+0x4a1/0x760 [ceph] [39330.511179] mdio crc32c_intel [39330.515968] ? __file_remove_privs+0x89/0x110 [39330.519705] nvme ptp [39330.524319] ? wake_bit_function+0x70/0x70 [39330.526675] nvme_core i2c_algo_bit [39330.531201] ? ceph_write_iter+0x260/0x770 [ceph] [39330.534249] pps_core dca [39330.538600] ceph_write_iter+0x260/0x770 [ceph] [39330.540872] fuse [39330.544964] ? lock_is_held_type+0xaa/0x120 [39330.548444] [39330.548445] irq event stamp: 234925650 [39330.553145] vfs_write+0x324/0x4f0 [39330.555759] hardirqs last enabled at (234925649): [] asm_common_interrupt+0x26/0x40 [39330.560288] ksys_pwrite64+0x65/0xa0 [39330.562209] hardirqs last disabled at (234925650): [] sysvec_apic_timer_interrupt+0xe/0x90 [39330.566385] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39330.567876] softirqs last enabled at (234925648): [] __do_softirq+0x2be/0x371 [39330.571621] do_syscall_64+0x38/0x80 [39330.575019] softirqs last disabled at (234925643): [] irq_exit_rcu+0xb0/0xd0 [39330.584310] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39330.587881] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39330.597690] RIP: 0033:0x7f48f0eeb93b [39330.602822] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39330.611611] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39330.615180] Workqueue: events delayed_work [ceph] [39330.623777] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39330.628823] [39330.628824] RIP: 0010:lock_acquire+0xf0/0x2b0 [39330.638198] ORIG_RAX: 0000000000000012 [39330.641770] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39330.649242] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39330.667978] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39330.672676] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39330.677892] [39330.677893] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39330.679383] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39330.683735] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39330.687567] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39330.706304] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39330.713428] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39330.718645] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39330.725774] [39330.727259] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39330.786436] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39330.794513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39330.800253] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39330.807375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39330.814501] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39330.821624] Call Trace: [39330.824070] [39330.826079] ? watchdog_timer_fn+0x1b4/0x220 [39330.830346] ? __hrtimer_run_queues+0x1d3/0x360 [39330.834870] ? hrtimer_interrupt+0x104/0x240 [39330.839140] ? __local_bh_enable+0x50/0x60 [39330.843234] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39330.848458] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39330.853414] [39330.855511] [39330.857610] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39330.862917] ? lock_acquire+0xf0/0x2b0 [39330.866668] ? igrab+0x43/0x50 [39330.869719] ? lock_release+0xc5/0x260 [39330.873464] _raw_spin_lock+0x30/0x40 [39330.877128] ? igrab+0x1d/0x50 [39330.880177] igrab+0x1d/0x50 [39330.883056] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39330.888206] delayed_work+0x1d8/0x280 [ceph] [39330.892486] ? process_scheduled_works+0x22a/0x510 [39330.897267] process_scheduled_works+0x2a4/0x510 [39330.901885] worker_thread+0x188/0x340 [39330.905633] ? rescuer_thread+0x310/0x310 [39330.909636] kthread+0xf5/0x130 [39330.912773] ? kthread_complete_and_exit+0x20/0x20 [39330.917574] ret_from_fork+0x31/0x40 [39330.921146] ? kthread_complete_and_exit+0x20/0x20 [39330.925930] ret_from_fork_asm+0x11/0x20 [39330.929868] [39358.142650] watchdog: BUG: soft lockup - CPU#0 stuck for 25541s! [dbench:78683] [39358.149958] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39358.154650] watchdog: BUG: soft lockup - CPU#7 stuck for 34538s! [kworker/7:4:78808] [39358.205407] irq event stamp: 69811886 [39358.213137] Modules linked in: ceph libceph dns_resolver fscache netfs [39358.216793] hardirqs last enabled at (69811885): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39358.216799] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [39358.233486] hardirqs last disabled at (69811886): [] sysvec_apic_timer_interrupt+0xe/0x90 [39358.240870] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [39358.250595] softirqs last enabled at (69811848): [] __do_softirq+0x2be/0x371 [39358.257042] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [39358.265726] softirqs last disabled at (69811843): [] irq_exit_rcu+0xb0/0xd0 [39358.265729] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [39358.280756] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39358.287619] i2c_i801 iTCO_wdt [39358.296563] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39358.296565] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39358.299615] iTCO_vendor_support [39358.307086] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39358.312562] i2c_smbus crct10dif_pclmul [39358.315787] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39358.334525] joydev [39358.338356] [39358.338357] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39358.343572] crc32_pclmul ipmi_ssif [39358.345669] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39358.347160] ghash_clmulni_intel [39358.354285] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39358.357768] mei_me mei [39358.364894] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39358.368117] lpc_ich sha512_ssse3 [39358.375241] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39358.377685] acpi_ipmi mfd_core [39358.384808] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39358.388120] wmi ipmi_si [39358.395243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39358.398380] ipmi_devintf ipmi_msghandler [39358.406459] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39358.408990] ioatdma acpi_pad [39358.414726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39358.418731] acpi_power_meter [39358.425855] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39358.428819] ixgbe igb [39358.435944] Call Trace: [39358.435947] [39358.438908] mdio crc32c_intel [39358.446052] ? watchdog_timer_fn+0x1b4/0x220 [39358.448406] nvme ptp [39358.450852] ? __hrtimer_run_queues+0x1d3/0x360 [39358.452860] nvme_core [39358.455914] ? hrtimer_interrupt+0x104/0x240 [39358.460176] i2c_algo_bit pps_core [39358.462447] ? __local_bh_enable+0x50/0x60 [39358.466970] dca fuse [39358.469346] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39358.473609] [39358.477006] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39358.481098] irq event stamp: 235146440 [39358.483370] [39358.488586] hardirqs last enabled at (235146439): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39358.490075] [39358.490077] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39358.495050] hardirqs last disabled at (235146440): [] sysvec_apic_timer_interrupt+0xe/0x90 [39358.498801] ? queued_spin_lock_slowpath+0x18/0x2e0 [39358.500893] softirqs last enabled at (235146438): [] __do_softirq+0x2be/0x371 [39358.511139] do_raw_spin_lock+0xb5/0xc0 [39358.513235] softirqs last disabled at (235146433): [] irq_exit_rcu+0xb0/0xd0 [39358.518539] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39358.528350] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39358.533223] try_get_cap_refs+0x159/0x880 [ceph] [39358.541991] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39358.545827] __ceph_get_caps+0x4a1/0x760 [ceph] [39358.554418] Workqueue: events delayed_work [ceph] [39358.559207] ? __file_remove_privs+0x89/0x110 [39358.568581] [39358.573193] ? wake_bit_function+0x70/0x70 [39358.580663] RIP: 0010:lock_release+0x42/0x260 [39358.585190] ? ceph_write_iter+0x260/0x770 [ceph] [39358.589884] Code: 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 0f a3 05 5c c0 53 01 0f 82 90 01 00 00 <8b> 3d d0 0b 54 01 85 ff 0f 84 fe 00 00 00 65 8b 05 e9 dc ea 7e 85 [39358.594234] ceph_write_iter+0x260/0x770 [ceph] [39358.595725] RSP: 0018:ffffc90002117d48 EFLAGS: 00000202 [39358.599821] ? lock_is_held_type+0xaa/0x120 [39358.604165] [39358.604166] RAX: 0000000000000001 RBX: ffff8881a1d34500 RCX: 000000005553a634 [39358.608866] vfs_write+0x324/0x4f0 [39358.627602] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [39358.632131] ksys_pwrite64+0x65/0xa0 [39358.637344] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39358.641522] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39358.643012] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39358.650138] do_syscall_64+0x38/0x80 [39358.653533] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [39358.660658] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39358.664226] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39358.671354] RIP: 0033:0x7f48f0eeb93b [39358.676483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39358.683608] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39358.687177] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39358.694301] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39358.699352] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39358.707432] ORIG_RAX: 0000000000000012 [39358.711004] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39358.716740] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39358.735477] Call Trace: [39358.735479] [39358.742601] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39358.747821] ? watchdog_timer_fn+0x1b4/0x220 [39358.754944] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39358.758776] ? __hrtimer_run_queues+0x1d3/0x360 [39358.765899] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39358.773028] ? hrtimer_interrupt+0x104/0x240 [39358.775465] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39358.777477] ? __local_bh_enable+0x50/0x60 [39358.784608] [39358.788867] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39358.830519] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39358.835476] [39358.837573] [39358.839670] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39358.844974] ? igrab+0x43/0x50 [39358.848027] ? igrab+0x43/0x50 [39358.851093] ? lock_release+0x42/0x260 [39358.854839] ? lock_release+0x1d7/0x260 [39358.858671] _raw_spin_unlock+0x1b/0x40 [39358.862510] igrab+0x43/0x50 [39358.865388] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39358.870536] delayed_work+0x1d8/0x280 [ceph] [39358.874818] ? process_scheduled_works+0x22a/0x510 [39358.879608] process_scheduled_works+0x2a4/0x510 [39358.884223] worker_thread+0x188/0x340 [39358.887972] ? rescuer_thread+0x310/0x310 [39358.891976] kthread+0xf5/0x130 [39358.895123] ? kthread_complete_and_exit+0x20/0x20 [39358.899906] ret_from_fork+0x31/0x40 [39358.903478] ? kthread_complete_and_exit+0x20/0x20 [39358.908268] ret_from_fork_asm+0x11/0x20 [39358.912202] [39360.164052] rcu: INFO: rcu_preempt self-detected stall on CPU [39360.169814] rcu: 7-....: (26625310 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/26623942 fqs=13424430 [39360.180498] rcu: (t=27423074 jiffies g=1309609 q=739983 ncpus=8) [39360.186582] Sending NMI from CPU 7 to CPUs 0: [39360.190934] NMI backtrace for cpu 0 [39360.190935] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39360.190937] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39360.190937] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39360.190940] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39360.190942] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39360.190943] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39360.190944] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39360.190945] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39360.190945] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39360.190946] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39360.190947] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39360.190948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39360.190949] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39360.190950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39360.190950] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39360.190951] Call Trace: [39360.190952] [39360.190953] ? nmi_cpu_backtrace+0xad/0x120 [39360.190956] ? nmi_cpu_backtrace_handler+0x11/0x20 [39360.190957] ? nmi_handle+0xd4/0x260 [39360.190962] ? default_do_nmi+0x49/0xf0 [39360.190963] ? exc_nmi+0x11e/0x150 [39360.190965] ? end_repeat_nmi+0x16/0x67 [39360.190969] ? queued_spin_lock_slowpath+0x18/0x2e0 [39360.190971] ? queued_spin_lock_slowpath+0x18/0x2e0 [39360.190974] ? queued_spin_lock_slowpath+0x18/0x2e0 [39360.190976] [39360.190976] [39360.190977] do_raw_spin_lock+0xb5/0xc0 [39360.190979] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39360.190993] try_get_cap_refs+0x159/0x880 [ceph] [39360.191006] __ceph_get_caps+0x4a1/0x760 [ceph] [39360.191018] ? __file_remove_privs+0x89/0x110 [39360.191020] ? wake_bit_function+0x70/0x70 [39360.191022] ? ceph_write_iter+0x260/0x770 [ceph] [39360.191033] ceph_write_iter+0x260/0x770 [ceph] [39360.191043] ? lock_is_held_type+0xaa/0x120 [39360.191046] vfs_write+0x324/0x4f0 [39360.191050] ksys_pwrite64+0x65/0xa0 [39360.191052] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39360.191056] do_syscall_64+0x38/0x80 [39360.191075] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39360.191077] RIP: 0033:0x7f48f0eeb93b [39360.191078] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39360.191080] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [39360.191081] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39360.191082] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39360.191083] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39360.191083] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39360.191084] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39360.191087] [39360.191932] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39360.495468] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39360.502940] Workqueue: events delayed_work [ceph] [39360.507663] RIP: 0010:lock_release+0x14e/0x260 [39360.512099] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [39360.530837] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [39360.536055] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [39360.543195] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [39360.550321] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39360.557445] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39360.564569] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [39360.571694] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39360.579769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39360.585507] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39360.592633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39360.599756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39360.606882] Call Trace: [39360.609323] [39360.611335] ? rcu_dump_cpu_stacks+0xd1/0x180 [39360.615697] ? rcu_sched_clock_irq+0x633/0x1390 [39360.620228] ? lock_release+0xc5/0x260 [39360.623972] ? lock_is_held_type+0xaa/0x120 [39360.628152] ? lock_is_held_type+0xaa/0x120 [39360.632337] ? tick_sched_do_timer+0xa0/0xa0 [39360.636626] ? rcu_read_lock_sched_held+0x56/0x90 [39360.641333] ? tick_sched_do_timer+0xa0/0xa0 [39360.645603] ? update_process_times+0x5f/0x90 [39360.649954] ? tick_sched_handle.isra.16+0x1e/0x50 [39360.654740] ? tick_sched_timer+0x6f/0x80 [39360.658754] ? __hrtimer_run_queues+0x1d3/0x360 [39360.663294] ? hrtimer_interrupt+0x104/0x240 [39360.667557] ? __local_bh_enable+0x50/0x60 [39360.671661] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39360.676883] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39360.681840] [39360.683936] [39360.686037] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39360.691366] ? igrab+0x43/0x50 [39360.694423] ? lock_release+0x14e/0x260 [39360.698258] ? lock_release+0xc5/0x260 [39360.702011] _raw_spin_unlock+0x1b/0x40 [39360.705848] igrab+0x43/0x50 [39360.708725] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39360.713878] delayed_work+0x1d8/0x280 [ceph] [39360.718167] ? process_scheduled_works+0x22a/0x510 [39360.722954] process_scheduled_works+0x2a4/0x510 [39360.727576] worker_thread+0x188/0x340 [39360.731331] ? rescuer_thread+0x310/0x310 [39360.735340] kthread+0xf5/0x130 [39360.738476] ? kthread_complete_and_exit+0x20/0x20 [39360.743263] ret_from_fork+0x31/0x40 [39360.746840] ? kthread_complete_and_exit+0x20/0x20 [39360.751625] ret_from_fork_asm+0x11/0x20 [39360.755555] [39386.143134] watchdog: BUG: soft lockup - CPU#0 stuck for 25567s! [dbench:78683] [39386.150434] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39386.155134] watchdog: BUG: soft lockup - CPU#7 stuck for 34564s! [kworker/7:4:78808] [39386.205876] irq event stamp: 69870034 [39386.205877] hardirqs last enabled at (69870033): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39386.213606] Modules linked in: [39386.217263] hardirqs last disabled at (69870034): [] sysvec_apic_timer_interrupt+0xe/0x90 [39386.227420] ceph [39386.230471] softirqs last enabled at (69869996): [] __do_softirq+0x2be/0x371 [39386.240197] libceph [39386.242121] softirqs last disabled at (69869991): [] irq_exit_rcu+0xb0/0xd0 [39386.250818] dns_resolver [39386.253005] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39386.261531] fscache netfs [39386.264152] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39386.273096] veth nft_chain_nat [39386.275811] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39386.283280] xt_MASQUERADE nf_nat [39386.286416] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39386.291895] nf_conntrack nf_defrag_ipv6 [39386.295204] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39386.313943] nf_defrag_ipv4 nft_compat [39386.317861] [39386.317861] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39386.323077] nf_tables [39386.326820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39386.328311] nfnetlink bridge [39386.335437] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39386.337810] stp [39386.344935] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39386.347898] llc xfs [39386.355022] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39386.356860] libcrc32c sunrpc [39386.363984] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39386.366170] intel_rapl_msr intel_rapl_common [39386.373293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39386.376258] sb_edac x86_pkg_temp_thermal [39386.384335] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39386.388685] coretemp kvm_intel [39386.394423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39386.398427] kvm irqbypass [39386.405551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39386.408689] i2c_i801 [39386.415820] Call Trace: [39386.415822] [39386.418534] iTCO_wdt iTCO_vendor_support [39386.425658] ? watchdog_timer_fn+0x1b4/0x220 [39386.427928] i2c_smbus crct10dif_pclmul [39386.430374] ? __hrtimer_run_queues+0x1d3/0x360 [39386.432383] joydev crc32_pclmul [39386.436392] ? hrtimer_interrupt+0x104/0x240 [39386.440651] ipmi_ssif [39386.444482] ? __local_bh_enable+0x50/0x60 [39386.449006] ghash_clmulni_intel [39386.452233] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39386.456495] mei_me mei [39386.458852] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39386.462943] lpc_ich sha512_ssse3 [39386.466166] [39386.471383] acpi_ipmi mfd_core [39386.473828] [39386.473830] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39386.478786] wmi ipmi_si ipmi_devintf [39386.482101] ? queued_spin_lock_slowpath+0x16/0x2e0 [39386.484194] ipmi_msghandler ioatdma [39386.487334] do_raw_spin_lock+0xb5/0xc0 [39386.489427] acpi_pad acpi_power_meter [39386.494734] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39386.498389] ixgbe igb [39386.503262] try_get_cap_refs+0x159/0x880 [ceph] [39386.506831] mdio [39386.510667] __ceph_get_caps+0x4a1/0x760 [ceph] [39386.514406] crc32c_intel [39386.519193] ? __file_remove_privs+0x89/0x110 [39386.521547] nvme ptp [39386.526158] ? wake_bit_function+0x70/0x70 [39386.528082] nvme_core i2c_algo_bit [39386.532610] ? ceph_write_iter+0x260/0x770 [ceph] [39386.535225] pps_core dca [39386.539574] ceph_write_iter+0x260/0x770 [ceph] [39386.541845] fuse [39386.545939] ? lock_is_held_type+0xaa/0x120 [39386.549421] [39386.549421] irq event stamp: 235362518 [39386.554119] vfs_write+0x324/0x4f0 [39386.556734] hardirqs last enabled at (235362517): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39386.561264] ksys_pwrite64+0x65/0xa0 [39386.563182] hardirqs last disabled at (235362518): [] sysvec_apic_timer_interrupt+0xe/0x90 [39386.567359] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39386.568850] softirqs last enabled at (235362516): [] __do_softirq+0x2be/0x371 [39386.572596] do_syscall_64+0x38/0x80 [39386.575992] softirqs last disabled at (235362511): [] irq_exit_rcu+0xb0/0xd0 [39386.586239] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39386.589807] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39386.599619] RIP: 0033:0x7f48f0eeb93b [39386.604750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39386.613536] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39386.617108] Workqueue: events delayed_work [ceph] [39386.625705] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39386.630749] [39386.640126] ORIG_RAX: 0000000000000012 [39386.643698] RIP: 0010:lock_acquire+0xf0/0x2b0 [39386.651169] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39386.669906] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39386.674602] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39386.679833] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39386.681321] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39386.685151] [39386.685152] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39386.689501] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39386.696627] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39386.715363] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39386.722488] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39386.727711] [39386.734830] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39386.781268] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39386.788392] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39386.796468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39386.802206] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39386.809329] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39386.816455] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39386.823577] Call Trace: [39386.826023] [39386.828033] ? watchdog_timer_fn+0x1b4/0x220 [39386.832299] ? __hrtimer_run_queues+0x1d3/0x360 [39386.836825] ? hrtimer_interrupt+0x104/0x240 [39386.841094] ? __local_bh_enable+0x50/0x60 [39386.845187] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39386.850412] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39386.855369] [39386.857466] [39386.859564] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39386.864872] ? lock_acquire+0xf0/0x2b0 [39386.868621] ? igrab+0x43/0x50 [39386.871672] ? lock_release+0xc5/0x260 [39386.875416] _raw_spin_lock+0x30/0x40 [39386.879082] ? igrab+0x1d/0x50 [39386.882131] igrab+0x1d/0x50 [39386.885010] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39386.890159] delayed_work+0x1d8/0x280 [ceph] [39386.894439] ? process_scheduled_works+0x22a/0x510 [39386.899221] process_scheduled_works+0x2a4/0x510 [39386.903838] worker_thread+0x188/0x340 [39386.907586] ? rescuer_thread+0x310/0x310 [39386.911590] kthread+0xf5/0x130 [39386.914728] ? kthread_complete_and_exit+0x20/0x20 [39386.919511] ret_from_fork+0x31/0x40 [39386.923081] ? kthread_complete_and_exit+0x20/0x20 [39386.927867] ret_from_fork_asm+0x11/0x20 [39386.931789] [39414.143615] watchdog: BUG: soft lockup - CPU#0 stuck for 25593s! [dbench:78683] [39414.150923] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39414.155615] watchdog: BUG: soft lockup - CPU#7 stuck for 34590s! [kworker/7:4:78808] [39414.206373] irq event stamp: 69928278 [39414.206374] hardirqs last enabled at (69928277): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39414.214103] Modules linked in: [39414.217759] hardirqs last disabled at (69928278): [] sysvec_apic_timer_interrupt+0xe/0x90 [39414.227918] ceph [39414.230969] softirqs last enabled at (69928240): [] __do_softirq+0x2be/0x371 [39414.240692] libceph [39414.242617] softirqs last disabled at (69928235): [] irq_exit_rcu+0xb0/0xd0 [39414.251299] dns_resolver [39414.253487] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39414.262008] fscache netfs [39414.264629] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39414.273575] veth nft_chain_nat [39414.276296] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39414.283767] xt_MASQUERADE nf_nat [39414.286905] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39414.292381] nf_conntrack nf_defrag_ipv6 [39414.295693] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39414.314429] nf_defrag_ipv4 nft_compat [39414.318348] [39414.318349] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39414.323566] nf_tables [39414.327309] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39414.328799] nfnetlink bridge [39414.335923] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39414.338293] stp llc [39414.345415] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39414.348377] xfs [39414.355501] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39414.357685] libcrc32c sunrpc [39414.364811] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39414.366647] intel_rapl_msr [39414.373771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39414.376735] intel_rapl_common sb_edac [39414.384812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39414.387602] x86_pkg_temp_thermal [39414.393341] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39414.397085] coretemp kvm_intel [39414.404209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39414.407521] kvm irqbypass [39414.414645] Call Trace: [39414.414647] [39414.417782] i2c_i801 [39414.424908] ? watchdog_timer_fn+0x1b4/0x220 [39414.427609] iTCO_wdt iTCO_vendor_support [39414.430056] ? __hrtimer_run_queues+0x1d3/0x360 [39414.432063] i2c_smbus crct10dif_pclmul [39414.434340] ? hrtimer_interrupt+0x104/0x240 [39414.438599] joydev crc32_pclmul [39414.442603] ? __local_bh_enable+0x50/0x60 [39414.447126] ipmi_ssif ghash_clmulni_intel [39414.450961] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39414.455222] mei_me mei [39414.458448] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39414.462537] lpc_ich sha512_ssse3 [39414.466629] [39414.471846] acpi_ipmi mfd_core [39414.474294] [39414.479256] wmi ipmi_si [39414.482567] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39414.484663] ipmi_devintf [39414.487805] ? queued_spin_lock_slowpath+0x16/0x2e0 [39414.489897] ipmi_msghandler [39414.492432] do_raw_spin_lock+0xb5/0xc0 [39414.497732] ioatdma acpi_pad [39414.500352] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39414.505222] acpi_power_meter ixgbe [39414.508102] try_get_cap_refs+0x159/0x880 [ceph] [39414.511928] igb mdio [39414.514898] __ceph_get_caps+0x4a1/0x760 [ceph] [39414.519678] crc32c_intel nvme [39414.523165] ? __file_remove_privs+0x89/0x110 [39414.527772] ptp nvme_core [39414.530042] ? wake_bit_function+0x70/0x70 [39414.534567] i2c_algo_bit [39414.537619] ? ceph_write_iter+0x260/0x770 [ceph] [39414.541968] pps_core [39414.544674] ceph_write_iter+0x260/0x770 [ceph] [39414.548764] dca fuse [39414.551385] ? lock_is_held_type+0xaa/0x120 [39414.556079] [39414.558350] vfs_write+0x324/0x4f0 [39414.562872] irq event stamp: 235583322 [39414.565148] ksys_pwrite64+0x65/0xa0 [39414.569320] hardirqs last enabled at (235583321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39414.570813] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39414.574208] hardirqs last disabled at (235583322): [] sysvec_apic_timer_interrupt+0xe/0x90 [39414.577955] do_syscall_64+0x38/0x80 [39414.581525] softirqs last enabled at (235583320): [] __do_softirq+0x2be/0x371 [39414.591769] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39414.596899] softirqs last disabled at (235583315): [] irq_exit_rcu+0xb0/0xd0 [39414.606711] RIP: 0033:0x7f48f0eeb93b [39414.610280] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39414.619051] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39414.624094] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39414.632693] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39414.636265] Workqueue: events delayed_work [ceph] [39414.645640] ORIG_RAX: 0000000000000012 [39414.645641] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39414.664379] [39414.671849] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39414.677067] RIP: 0010:lock_acquire+0xf0/0x2b0 [39414.681765] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39414.685594] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39414.692719] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39414.694211] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39414.701335] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39414.705686] [39414.705687] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39414.712813] [39414.731547] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39414.768943] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39414.776067] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39414.783192] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39414.790318] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39414.798394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39414.804130] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39414.811255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39414.818380] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39414.825503] Call Trace: [39414.827950] [39414.829961] ? watchdog_timer_fn+0x1b4/0x220 [39414.834224] ? __hrtimer_run_queues+0x1d3/0x360 [39414.838749] ? hrtimer_interrupt+0x104/0x240 [39414.843021] ? __local_bh_enable+0x50/0x60 [39414.847112] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39414.852336] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39414.857295] [39414.859408] [39414.861505] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39414.866816] ? lock_acquire+0xf0/0x2b0 [39414.870565] ? igrab+0x43/0x50 [39414.873615] ? lock_release+0xc5/0x260 [39414.877361] _raw_spin_lock+0x30/0x40 [39414.881025] ? igrab+0x1d/0x50 [39414.884075] igrab+0x1d/0x50 [39414.886953] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39414.892105] delayed_work+0x1d8/0x280 [ceph] [39414.896383] ? process_scheduled_works+0x22a/0x510 [39414.901175] process_scheduled_works+0x2a4/0x510 [39414.905791] worker_thread+0x188/0x340 [39414.909539] ? rescuer_thread+0x310/0x310 [39414.913541] kthread+0xf5/0x130 [39414.916681] ? kthread_complete_and_exit+0x20/0x20 [39414.921464] ret_from_fork+0x31/0x40 [39414.925044] ? kthread_complete_and_exit+0x20/0x20 [39414.929827] ret_from_fork_asm+0x11/0x20 [39414.933759] [39442.144095] watchdog: BUG: soft lockup - CPU#0 stuck for 25619s! [dbench:78683] [39442.151391] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39442.156095] watchdog: BUG: soft lockup - CPU#7 stuck for 34616s! [kworker/7:4:78808] [39442.206834] irq event stamp: 69986510 [39442.214565] Modules linked in: ceph [39442.218221] hardirqs last enabled at (69986509): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39442.221706] libceph dns_resolver [39442.231862] hardirqs last disabled at (69986510): [] sysvec_apic_timer_interrupt+0xe/0x90 [39442.235173] fscache netfs veth [39442.244897] softirqs last enabled at (69986472): [] __do_softirq+0x2be/0x371 [39442.244899] nft_chain_nat xt_MASQUERADE [39442.248036] softirqs last disabled at (69986467): [] irq_exit_rcu+0xb0/0xd0 [39442.256720] nf_nat nf_conntrack [39442.260637] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39442.269147] nf_defrag_ipv6 [39442.272372] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39442.272373] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [39442.281315] nf_defrag_ipv4 nft_compat [39442.284105] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [39442.291578] nf_tables nfnetlink [39442.297054] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39442.300799] bridge stp [39442.319535] [39442.319536] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39442.322775] llc xfs [39442.327994] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39442.330440] libcrc32c sunrpc [39442.331930] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39442.339054] intel_rapl_msr [39442.341237] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39442.348362] intel_rapl_common [39442.351326] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39442.358450] sb_edac x86_pkg_temp_thermal [39442.361242] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39442.368364] coretemp kvm_intel [39442.371415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39442.378539] kvm [39442.382545] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39442.390621] irqbypass i2c_i801 [39442.393771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39442.399504] iTCO_wdt [39442.401343] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39442.408467] iTCO_vendor_support i2c_smbus [39442.411622] Call Trace: [39442.411623] [39442.418744] crct10dif_pclmul [39442.421016] ? watchdog_timer_fn+0x1b4/0x220 [39442.428140] joydev crc32_pclmul [39442.432232] ? __hrtimer_run_queues+0x1d3/0x360 [39442.434674] ipmi_ssif ghash_clmulni_intel [39442.436688] ? hrtimer_interrupt+0x104/0x240 [39442.439648] mei_me mei [39442.443915] ? __local_bh_enable+0x50/0x60 [39442.447138] lpc_ich sha512_ssse3 [39442.451664] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39442.455752] acpi_ipmi mfd_core [39442.460018] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39442.462461] wmi ipmi_si [39442.466551] [39442.469861] ipmi_devintf ipmi_msghandler [39442.475079] [39442.478218] ioatdma acpi_pad [39442.483177] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39442.485706] acpi_power_meter ixgbe [39442.487807] ? queued_spin_lock_slowpath+0x2b/0x2e0 [39442.491806] igb [39442.493907] do_raw_spin_lock+0xb5/0xc0 [39442.496869] mdio crc32c_intel [39442.502172] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39442.505657] nvme [39442.510528] try_get_cap_refs+0x159/0x880 [ceph] [39442.512364] ptp nvme_core [39442.516200] __ceph_get_caps+0x4a1/0x760 [ceph] [39442.519246] i2c_algo_bit pps_core [39442.524035] ? __file_remove_privs+0x89/0x110 [39442.525955] dca fuse [39442.530566] ? wake_bit_function+0x70/0x70 [39442.533269] [39442.533270] irq event stamp: 235804064 [39442.537795] ? ceph_write_iter+0x260/0x770 [ceph] [39442.541191] hardirqs last enabled at (235804063): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39442.545542] ceph_write_iter+0x260/0x770 [ceph] [39442.547812] hardirqs last disabled at (235804064): [] sysvec_apic_timer_interrupt+0xe/0x90 [39442.551905] ? lock_is_held_type+0xaa/0x120 [39442.553392] softirqs last enabled at (235804062): [] __do_softirq+0x2be/0x371 [39442.557140] vfs_write+0x324/0x4f0 [39442.561835] softirqs last disabled at (235804057): [] irq_exit_rcu+0xb0/0xd0 [39442.572084] ksys_pwrite64+0x65/0xa0 [39442.576603] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39442.586413] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39442.590592] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39442.590593] Workqueue: events delayed_work [ceph] [39442.599363] do_syscall_64+0x38/0x80 [39442.602759] [39442.611374] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39442.614945] RIP: 0010:lock_acquire+0xf0/0x2b0 [39442.624323] RIP: 0033:0x7f48f0eeb93b [39442.629453] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39442.636925] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39442.641623] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39442.645191] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39442.646683] [39442.646684] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39442.651726] ORIG_RAX: 0000000000000012 [39442.656077] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39442.659650] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39442.678385] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39442.697124] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39442.702340] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39442.707559] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39442.709050] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39442.716173] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39442.720005] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39442.727127] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39442.734252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39442.741382] [39442.748501] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39442.807246] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39442.814370] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39442.821500] Call Trace: [39442.823947] [39442.825957] ? watchdog_timer_fn+0x1b4/0x220 [39442.830222] ? __hrtimer_run_queues+0x1d3/0x360 [39442.834749] ? hrtimer_interrupt+0x104/0x240 [39442.839018] ? __local_bh_enable+0x50/0x60 [39442.843112] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39442.848336] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39442.853293] [39442.855389] [39442.857503] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39442.862812] ? lock_acquire+0xf0/0x2b0 [39442.866563] ? igrab+0x43/0x50 [39442.869611] ? lock_release+0xc5/0x260 [39442.873358] _raw_spin_lock+0x30/0x40 [39442.877022] ? igrab+0x1d/0x50 [39442.880072] igrab+0x1d/0x50 [39442.882951] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39442.888100] delayed_work+0x1d8/0x280 [ceph] [39442.892381] ? process_scheduled_works+0x22a/0x510 [39442.897171] process_scheduled_works+0x2a4/0x510 [39442.901795] worker_thread+0x188/0x340 [39442.905546] ? rescuer_thread+0x310/0x310 [39442.909556] kthread+0xf5/0x130 [39442.912696] ? kthread_complete_and_exit+0x20/0x20 [39442.917479] ret_from_fork+0x31/0x40 [39442.921066] ? kthread_complete_and_exit+0x20/0x20 [39442.925852] ret_from_fork_asm+0x11/0x20 [39442.929790] [39470.144408] watchdog: BUG: soft lockup - CPU#0 stuck for 25645s! [dbench:78683] [39470.151715] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39470.156407] watchdog: BUG: soft lockup - CPU#7 stuck for 34642s! [kworker/7:4:78808] [39470.207165] irq event stamp: 70044708 [39470.207166] hardirqs last enabled at (70044707): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39470.214895] Modules linked in: [39470.218552] hardirqs last disabled at (70044708): [] sysvec_apic_timer_interrupt+0xe/0x90 [39470.228711] ceph [39470.231761] softirqs last enabled at (70044670): [] __do_softirq+0x2be/0x371 [39470.241485] libceph dns_resolver [39470.243410] softirqs last disabled at (70044665): [] irq_exit_rcu+0xb0/0xd0 [39470.252092] fscache netfs [39470.255404] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39470.263915] veth nft_chain_nat [39470.266620] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39470.275562] xt_MASQUERADE [39470.278702] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39470.286170] nf_nat [39470.288876] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39470.294353] nf_conntrack nf_defrag_ipv6 [39470.296450] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39470.315186] nf_defrag_ipv4 [39470.319103] [39470.319104] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39470.324321] nft_compat [39470.327113] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39470.328605] nf_tables [39470.335727] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39470.338173] nfnetlink bridge [39470.345295] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39470.347653] stp [39470.354777] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39470.357759] llc xfs [39470.364884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39470.366720] libcrc32c sunrpc [39470.373844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39470.376028] intel_rapl_msr intel_rapl_common [39470.384105] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39470.387071] sb_edac x86_pkg_temp_thermal [39470.392825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39470.397176] coretemp kvm_intel [39470.404300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39470.408304] kvm irqbypass [39470.415427] Call Trace: [39470.418564] i2c_i801 iTCO_wdt [39470.425691] [39470.428395] iTCO_vendor_support [39470.430837] ? watchdog_timer_fn+0x1b4/0x220 [39470.433888] i2c_smbus crct10dif_pclmul [39470.435899] ? __hrtimer_run_queues+0x1d3/0x360 [39470.439122] joydev crc32_pclmul [39470.443391] ? hrtimer_interrupt+0x104/0x240 [39470.447218] ipmi_ssif ghash_clmulni_intel [39470.451743] ? __local_bh_enable+0x50/0x60 [39470.454966] mei_me mei [39470.459232] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39470.463320] lpc_ich sha512_ssse3 [39470.467413] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39470.469856] acpi_ipmi mfd_core [39470.475073] [39470.478402] wmi ipmi_si [39470.483358] [39470.486495] ipmi_devintf ipmi_msghandler [39470.488594] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39470.491123] ioatdma [39470.493225] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39470.497227] acpi_pad acpi_power_meter [39470.502533] do_raw_spin_lock+0xb5/0xc0 [39470.504714] ixgbe igb [39470.509585] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39470.513329] mdio [39470.517161] try_get_cap_refs+0x159/0x880 [ceph] [39470.519517] crc32c_intel nvme [39470.524305] __ceph_get_caps+0x4a1/0x760 [ceph] [39470.526225] ptp nvme_core [39470.530839] ? __file_remove_privs+0x89/0x110 [39470.533885] i2c_algo_bit pps_core [39470.538410] ? wake_bit_function+0x70/0x70 [39470.541114] dca fuse [39470.545468] ? ceph_write_iter+0x260/0x770 [ceph] [39470.548861] [39470.548862] irq event stamp: 236024928 [39470.552953] ceph_write_iter+0x260/0x770 [ceph] [39470.555224] hardirqs last enabled at (236024927): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39470.559924] ? lock_is_held_type+0xaa/0x120 [39470.561411] hardirqs last disabled at (236024928): [] sysvec_apic_timer_interrupt+0xe/0x90 [39470.565157] vfs_write+0x324/0x4f0 [39470.569679] softirqs last enabled at (236024926): [] __do_softirq+0x2be/0x371 [39470.579931] ksys_pwrite64+0x65/0xa0 [39470.584101] softirqs last disabled at (236024921): [] irq_exit_rcu+0xb0/0xd0 [39470.593914] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39470.597311] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39470.606081] do_syscall_64+0x38/0x80 [39470.609651] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39470.618251] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39470.623380] Workqueue: events delayed_work [ceph] [39470.632760] RIP: 0033:0x7f48f0eeb93b [39470.636329] [39470.643812] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39470.648850] RIP: 0010:lock_acquire+0x34/0x2b0 [39470.653550] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39470.657121] Code: 89 f7 41 56 41 89 d6 41 55 41 89 cd 41 54 49 89 fc 55 44 89 c5 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 30 31 c0 <0f> 1f 44 00 00 65 8b 05 8c e7 eb 7e 89 c0 48 0f a3 05 4e bd 53 01 [39470.658609] ORIG_RAX: 0000000000000012 [39470.677347] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [39470.681697] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39470.686916] [39470.705655] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39470.709484] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [39470.714703] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39470.721827] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [39470.723318] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39470.730441] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [39470.737565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39470.744689] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [39470.751819] [39470.758938] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39470.789618] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39470.797696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39470.803434] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39470.810559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39470.817681] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39470.824814] Call Trace: [39470.827261] [39470.829271] ? watchdog_timer_fn+0x1b4/0x220 [39470.833534] ? __hrtimer_run_queues+0x1d3/0x360 [39470.838061] ? hrtimer_interrupt+0x104/0x240 [39470.842330] ? __local_bh_enable+0x50/0x60 [39470.846424] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39470.851648] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39470.856605] [39470.858701] [39470.860813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39470.866125] ? lock_acquire+0x34/0x2b0 [39470.869876] ? igrab+0x43/0x50 [39470.872924] ? lock_release+0xc5/0x260 [39470.876669] _raw_spin_lock+0x30/0x40 [39470.880328] ? igrab+0x1d/0x50 [39470.883377] igrab+0x1d/0x50 [39470.886256] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39470.891406] delayed_work+0x1d8/0x280 [ceph] [39470.895685] ? process_scheduled_works+0x22a/0x510 [39470.900476] process_scheduled_works+0x2a4/0x510 [39470.905098] worker_thread+0x188/0x340 [39470.908849] ? rescuer_thread+0x310/0x310 [39470.912851] kthread+0xf5/0x130 [39470.915992] ? kthread_complete_and_exit+0x20/0x20 [39470.920783] ret_from_fork+0x31/0x40 [39470.924381] ? kthread_complete_and_exit+0x20/0x20 [39470.929164] ret_from_fork_asm+0x11/0x20 [39470.933102] [39498.144893] watchdog: BUG: soft lockup - CPU#0 stuck for 25671s! [dbench:78683] [39498.152192] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39498.156893] watchdog: BUG: soft lockup - CPU#7 stuck for 34668s! [kworker/7:4:78808] [39498.207634] irq event stamp: 70102880 [39498.207636] hardirqs last enabled at (70102879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39498.215365] Modules linked in: [39498.219022] hardirqs last disabled at (70102880): [] sysvec_apic_timer_interrupt+0xe/0x90 [39498.229180] ceph [39498.232231] softirqs last enabled at (70102842): [] __do_softirq+0x2be/0x371 [39498.241954] libceph [39498.243880] softirqs last disabled at (70102837): [] irq_exit_rcu+0xb0/0xd0 [39498.252572] dns_resolver fscache [39498.254756] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39498.263283] netfs veth [39498.266594] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39498.275539] nft_chain_nat [39498.277983] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39498.285454] xt_MASQUERADE nf_nat [39498.288156] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39498.293635] nf_conntrack [39498.296944] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39498.315683] nf_defrag_ipv6 [39498.318301] [39498.318302] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39498.323519] nf_defrag_ipv4 nft_compat [39498.326309] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39498.327798] nf_tables nfnetlink [39498.334923] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39498.338668] bridge stp [39498.345792] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39498.349017] llc xfs [39498.356140] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39498.358585] libcrc32c [39498.365709] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39498.367892] sunrpc intel_rapl_msr [39498.375015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39498.377374] intel_rapl_common [39498.385451] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39498.388850] sb_edac x86_pkg_temp_thermal [39498.394586] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39498.397637] coretemp kvm_intel [39498.404761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39498.408764] kvm [39498.415889] Call Trace: [39498.419027] irqbypass i2c_i801 [39498.426152] [39498.427989] iTCO_wdt iTCO_vendor_support [39498.430433] ? watchdog_timer_fn+0x1b4/0x220 [39498.433577] i2c_smbus crct10dif_pclmul [39498.435589] ? __hrtimer_run_queues+0x1d3/0x360 [39498.439593] joydev crc32_pclmul [39498.443859] ? hrtimer_interrupt+0x104/0x240 [39498.447688] ipmi_ssif ghash_clmulni_intel [39498.452212] ? __local_bh_enable+0x50/0x60 [39498.455437] mei_me mei [39498.459702] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39498.463790] lpc_ich [39498.467882] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39498.470326] sha512_ssse3 [39498.475543] [39498.477727] acpi_ipmi mfd_core [39498.482685] [39498.485302] wmi ipmi_si [39498.487400] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39498.490535] ipmi_devintf ipmi_msghandler [39498.492637] ? queued_spin_lock_slowpath+0x18/0x2e0 [39498.495166] ioatdma acpi_pad [39498.500471] do_raw_spin_lock+0xb5/0xc0 [39498.504473] acpi_power_meter [39498.509343] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39498.512308] ixgbe igb [39498.516141] try_get_cap_refs+0x159/0x880 [ceph] [39498.519102] mdio crc32c_intel [39498.523891] __ceph_get_caps+0x4a1/0x760 [ceph] [39498.526245] nvme ptp [39498.530857] ? __file_remove_privs+0x89/0x110 [39498.533904] nvme_core i2c_algo_bit [39498.538431] ? wake_bit_function+0x70/0x70 [39498.540700] pps_core dca [39498.545052] ? ceph_write_iter+0x260/0x770 [ceph] [39498.548535] fuse [39498.552627] ceph_write_iter+0x260/0x770 [ceph] [39498.555243] irq event stamp: 236245680 [39498.559943] ? lock_is_held_type+0xaa/0x120 [39498.561865] hardirqs last enabled at (236245679): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39498.566391] vfs_write+0x324/0x4f0 [39498.570133] hardirqs last disabled at (236245680): [] sysvec_apic_timer_interrupt+0xe/0x90 [39498.574316] ksys_pwrite64+0x65/0xa0 [39498.584571] softirqs last enabled at (236245678): [] __do_softirq+0x2be/0x371 [39498.587970] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39498.597780] softirqs last disabled at (236245673): [] irq_exit_rcu+0xb0/0xd0 [39498.601352] do_syscall_64+0x38/0x80 [39498.610123] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39498.615253] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39498.623851] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39498.627422] RIP: 0033:0x7f48f0eeb93b [39498.636799] Workqueue: events delayed_work [ceph] [39498.641841] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39498.649312] [39498.652884] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39498.657581] RIP: 0010:lock_acquire+0xf0/0x2b0 [39498.676320] ORIG_RAX: 0000000000000012 [39498.677810] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39498.683027] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39498.687377] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39498.691207] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39498.709945] [39498.709945] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39498.717071] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39498.722297] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39498.729420] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39498.730911] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39498.738035] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39498.745159] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39498.752300] [39498.759415] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39498.790096] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39498.798173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39498.803912] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39498.811036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39498.818161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39498.825298] Call Trace: [39498.827746] [39498.829757] ? watchdog_timer_fn+0x1b4/0x220 [39498.834021] ? __hrtimer_run_queues+0x1d3/0x360 [39498.838549] ? hrtimer_interrupt+0x104/0x240 [39498.842818] ? __local_bh_enable+0x50/0x60 [39498.846910] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39498.852136] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39498.857091] [39498.859190] [39498.861299] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39498.866612] ? lock_acquire+0xf0/0x2b0 [39498.870362] ? igrab+0x43/0x50 [39498.873412] ? lock_release+0xc5/0x260 [39498.877158] _raw_spin_lock+0x30/0x40 [39498.880822] ? igrab+0x1d/0x50 [39498.883872] igrab+0x1d/0x50 [39498.886751] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39498.891900] delayed_work+0x1d8/0x280 [ceph] [39498.896181] ? process_scheduled_works+0x22a/0x510 [39498.900973] process_scheduled_works+0x2a4/0x510 [39498.905586] worker_thread+0x188/0x340 [39498.909337] ? rescuer_thread+0x310/0x310 [39498.913349] kthread+0xf5/0x130 [39498.916486] ? kthread_complete_and_exit+0x20/0x20 [39498.921270] ret_from_fork+0x31/0x40 [39498.924858] ? kthread_complete_and_exit+0x20/0x20 [39498.929641] ret_from_fork_asm+0x11/0x20 [39498.933581] [39526.145373] watchdog: BUG: soft lockup - CPU#0 stuck for 25697s! [dbench:78683] [39526.152682] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39526.157373] watchdog: BUG: soft lockup - CPU#7 stuck for 34694s! [kworker/7:4:78808] [39526.208131] irq event stamp: 70161110 [39526.208132] hardirqs last enabled at (70161109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39526.215862] Modules linked in: [39526.219518] hardirqs last disabled at (70161110): [] sysvec_apic_timer_interrupt+0xe/0x90 [39526.229677] ceph [39526.232727] softirqs last enabled at (70161072): [] __do_softirq+0x2be/0x371 [39526.242449] libceph [39526.244375] softirqs last disabled at (70161067): [] irq_exit_rcu+0xb0/0xd0 [39526.253058] dns_resolver fscache [39526.255245] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39526.263770] netfs [39526.267083] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39526.276027] veth nft_chain_nat [39526.278054] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39526.285525] xt_MASQUERADE [39526.288665] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39526.294139] nf_nat nf_conntrack [39526.296845] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39526.315581] nf_defrag_ipv6 nf_defrag_ipv4 [39526.318805] [39526.318806] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39526.324022] nft_compat nf_tables [39526.328112] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39526.329603] nfnetlink bridge [39526.336727] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39526.340052] stp llc [39526.347181] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39526.350144] xfs libcrc32c [39526.357268] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39526.359453] sunrpc intel_rapl_msr [39526.366577] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39526.369282] intel_rapl_common [39526.376406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39526.379804] sb_edac x86_pkg_temp_thermal [39526.387880] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39526.390930] coretemp kvm_intel [39526.396669] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39526.400673] kvm irqbypass [39526.407795] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39526.410934] i2c_i801 iTCO_wdt [39526.418057] Call Trace: [39526.418059] [39526.420773] iTCO_vendor_support [39526.427898] ? watchdog_timer_fn+0x1b4/0x220 [39526.430945] i2c_smbus crct10dif_pclmul [39526.433390] ? __hrtimer_run_queues+0x1d3/0x360 [39526.435401] joydev [39526.438628] ? hrtimer_interrupt+0x104/0x240 [39526.442888] crc32_pclmul [39526.446718] ? __local_bh_enable+0x50/0x60 [39526.451243] ipmi_ssif [39526.453344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39526.457604] ghash_clmulni_intel [39526.460221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39526.464313] mei_me [39526.466671] [39526.471888] mei lpc_ich [39526.475112] [39526.480068] sha512_ssse3 acpi_ipmi [39526.482168] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39526.484263] mfd_core wmi [39526.486799] ? queued_spin_lock_slowpath+0x18/0x2e0 [39526.488892] ipmi_si [39526.492377] do_raw_spin_lock+0xb5/0xc0 [39526.497680] ipmi_devintf ipmi_msghandler [39526.500300] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39526.505168] ioatdma acpi_pad [39526.507356] try_get_cap_refs+0x159/0x880 [ceph] [39526.511184] acpi_power_meter ixgbe [39526.515193] __ceph_get_caps+0x4a1/0x760 [ceph] [39526.519971] igb mdio [39526.522940] ? __file_remove_privs+0x89/0x110 [39526.527545] crc32c_intel nvme [39526.531032] ? wake_bit_function+0x70/0x70 [39526.535571] ptp nvme_core [39526.537846] ? ceph_write_iter+0x260/0x770 [ceph] [39526.542192] i2c_algo_bit pps_core [39526.545244] ceph_write_iter+0x260/0x770 [ceph] [39526.549333] dca [39526.552042] ? lock_is_held_type+0xaa/0x120 [39526.556734] fuse [39526.560136] vfs_write+0x324/0x4f0 [39526.564658] irq event stamp: 236466438 [39526.566500] ksys_pwrite64+0x65/0xa0 [39526.570672] hardirqs last enabled at (236466437): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39526.572596] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39526.575994] hardirqs last disabled at (236466438): [] sysvec_apic_timer_interrupt+0xe/0x90 [39526.579740] do_syscall_64+0x38/0x80 [39526.583307] softirqs last enabled at (236466436): [] __do_softirq+0x2be/0x371 [39526.593554] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39526.598683] softirqs last disabled at (236466431): [] irq_exit_rcu+0xb0/0xd0 [39526.608496] RIP: 0033:0x7f48f0eeb93b [39526.612072] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39526.620845] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39526.625888] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39526.625889] Workqueue: events delayed_work [ceph] [39526.634485] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39526.638056] [39526.647435] ORIG_RAX: 0000000000000012 [39526.666172] RIP: 0010:ceph_check_delayed_caps+0x6f/0x270 [ceph] [39526.673642] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39526.678340] Code: 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 0f 88 14 01 00 00 <48> 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 [39526.683558] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39526.685057] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [39526.688887] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39526.694797] [39526.694798] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [39526.701923] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39526.720661] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [39526.727785] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39526.733002] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [39526.740131] [39526.741616] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [39526.786545] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [39526.793671] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39526.801748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39526.807485] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39526.814608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39526.821732] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39526.828857] Call Trace: [39526.831302] [39526.833313] ? watchdog_timer_fn+0x1b4/0x220 [39526.837577] ? __hrtimer_run_queues+0x1d3/0x360 [39526.842104] ? hrtimer_interrupt+0x104/0x240 [39526.846373] ? __local_bh_enable+0x50/0x60 [39526.850467] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39526.855691] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39526.860649] [39526.862745] [39526.864841] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39526.870152] ? ceph_check_delayed_caps+0x6f/0x270 [ceph] [39526.875469] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [39526.880791] delayed_work+0x1d8/0x280 [ceph] [39526.885073] ? process_scheduled_works+0x22a/0x510 [39526.889863] process_scheduled_works+0x2a4/0x510 [39526.894480] worker_thread+0x188/0x340 [39526.898227] ? rescuer_thread+0x310/0x310 [39526.902231] kthread+0xf5/0x130 [39526.905369] ? kthread_complete_and_exit+0x20/0x20 [39526.910154] ret_from_fork+0x31/0x40 [39526.913731] ? kthread_complete_and_exit+0x20/0x20 [39526.918517] ret_from_fork_asm+0x11/0x20 [39526.922447] [39554.145835] watchdog: BUG: soft lockup - CPU#0 stuck for 25723s! [dbench:78683] [39554.153133] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39554.157835] watchdog: BUG: soft lockup - CPU#7 stuck for 34720s! [kworker/7:4:78808] [39554.208575] irq event stamp: 70219328 [39554.208576] hardirqs last enabled at (70219327): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39554.216305] Modules linked in: [39554.219961] hardirqs last disabled at (70219328): [] sysvec_apic_timer_interrupt+0xe/0x90 [39554.230121] ceph [39554.233171] softirqs last enabled at (70219290): [] __do_softirq+0x2be/0x371 [39554.242894] libceph [39554.244820] softirqs last disabled at (70219285): [] irq_exit_rcu+0xb0/0xd0 [39554.253508] dns_resolver fscache [39554.255696] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39554.264207] netfs [39554.267534] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39554.276478] veth nft_chain_nat [39554.278507] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39554.285977] xt_MASQUERADE nf_nat [39554.289116] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39554.294592] nf_conntrack [39554.297903] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39554.316640] nf_defrag_ipv6 nf_defrag_ipv4 [39554.319258] [39554.319259] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39554.324476] nft_compat nf_tables [39554.328566] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39554.330058] nfnetlink bridge [39554.337182] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39554.340492] stp llc [39554.347632] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39554.350597] xfs libcrc32c [39554.357721] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39554.359905] sunrpc [39554.367029] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39554.369734] intel_rapl_msr intel_rapl_common [39554.376858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39554.378956] sb_edac x86_pkg_temp_thermal [39554.387031] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39554.391382] coretemp kvm_intel [39554.397121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39554.401125] kvm irqbypass [39554.408249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39554.411386] i2c_i801 [39554.418514] Call Trace: [39554.418515] [39554.421231] iTCO_wdt iTCO_vendor_support [39554.428358] ? watchdog_timer_fn+0x1b4/0x220 [39554.430627] i2c_smbus crct10dif_pclmul [39554.433071] ? __hrtimer_run_queues+0x1d3/0x360 [39554.435082] joydev crc32_pclmul [39554.439089] ? hrtimer_interrupt+0x104/0x240 [39554.443351] ipmi_ssif ghash_clmulni_intel [39554.447180] ? __local_bh_enable+0x50/0x60 [39554.451704] mei_me [39554.454932] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39554.459191] mei lpc_ich [39554.463285] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39554.467373] sha512_ssse3 [39554.469473] [39554.474690] acpi_ipmi [39554.477234] [39554.477236] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39554.482194] mfd_core wmi [39554.484817] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39554.486909] ipmi_si ipmi_devintf [39554.489268] do_raw_spin_lock+0xb5/0xc0 [39554.491365] ipmi_msghandler [39554.496669] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39554.499284] ioatdma acpi_pad [39554.504157] try_get_cap_refs+0x159/0x880 [ceph] [39554.507466] acpi_power_meter [39554.511302] __ceph_get_caps+0x4a1/0x760 [ceph] [39554.514175] ixgbe igb [39554.518963] ? __file_remove_privs+0x89/0x110 [39554.521924] mdio crc32c_intel [39554.526536] ? wake_bit_function+0x70/0x70 [39554.529497] nvme ptp [39554.534024] ? ceph_write_iter+0x260/0x770 [ceph] [39554.536380] nvme_core i2c_algo_bit [39554.540730] ceph_write_iter+0x260/0x770 [ceph] [39554.543780] pps_core dca [39554.547875] ? lock_is_held_type+0xaa/0x120 [39554.550142] fuse [39554.554843] vfs_write+0x324/0x4f0 [39554.558325] irq event stamp: 236687300 [39554.562852] ksys_pwrite64+0x65/0xa0 [39554.565465] hardirqs last enabled at (236687299): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39554.569643] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39554.571567] hardirqs last disabled at (236687300): [] sysvec_apic_timer_interrupt+0xe/0x90 [39554.574967] do_syscall_64+0x38/0x80 [39554.578709] softirqs last enabled at (236687298): [] __do_softirq+0x2be/0x371 [39554.582279] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39554.592533] softirqs last disabled at (236687293): [] irq_exit_rcu+0xb0/0xd0 [39554.597664] RIP: 0033:0x7f48f0eeb93b [39554.607472] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39554.611044] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39554.619815] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39554.624858] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39554.633457] Workqueue: events delayed_work [ceph] [39554.637028] ORIG_RAX: 0000000000000012 [39554.637029] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39554.646404] [39554.646405] RIP: 0010:lock_release+0x14e/0x260 [39554.665143] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39554.672614] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [39554.677832] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39554.682537] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [39554.686367] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39554.693491] [39554.695000] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39554.699437] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [39554.706568] [39554.725298] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [39554.769822] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39554.776945] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39554.784069] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [39554.791193] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39554.799271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39554.805008] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39554.812133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39554.819257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39554.826381] Call Trace: [39554.828824] [39554.830836] ? watchdog_timer_fn+0x1b4/0x220 [39554.835101] ? __hrtimer_run_queues+0x1d3/0x360 [39554.839627] ? hrtimer_interrupt+0x104/0x240 [39554.843897] ? __local_bh_enable+0x50/0x60 [39554.847990] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39554.853214] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39554.858187] [39554.860285] [39554.862384] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39554.867686] ? igrab+0x43/0x50 [39554.870741] ? lock_release+0x14e/0x260 [39554.874579] ? lock_release+0xc5/0x260 [39554.878324] _raw_spin_unlock+0x1b/0x40 [39554.882161] igrab+0x43/0x50 [39554.885039] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39554.890189] delayed_work+0x1d8/0x280 [ceph] [39554.894470] ? process_scheduled_works+0x22a/0x510 [39554.899260] process_scheduled_works+0x2a4/0x510 [39554.903874] worker_thread+0x188/0x340 [39554.907623] ? rescuer_thread+0x310/0x310 [39554.911628] kthread+0xf5/0x130 [39554.914766] ? kthread_complete_and_exit+0x20/0x20 [39554.919551] ret_from_fork+0x31/0x40 [39554.923128] ? kthread_complete_and_exit+0x20/0x20 [39554.927912] ret_from_fork_asm+0x11/0x20 [39554.931836] [39555.760229] rcu: INFO: rcu_preempt self-detected stall on CPU [39555.765985] rcu: 7-....: (26814882 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/26813513 fqs=13520299 [39555.776665] rcu: (t=27618667 jiffies g=1309609 q=743982 ncpus=8) [39555.782757] Sending NMI from CPU 7 to CPUs 0: [39555.787111] NMI backtrace for cpu 0 [39555.787113] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39555.787114] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39555.787115] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39555.787118] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39555.787119] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39555.787121] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39555.787121] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39555.787122] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39555.787123] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39555.787123] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39555.787124] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39555.787125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39555.787126] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39555.787127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39555.787128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39555.787128] Call Trace: [39555.787129] [39555.787130] ? nmi_cpu_backtrace+0xad/0x120 [39555.787133] ? nmi_cpu_backtrace_handler+0x11/0x20 [39555.787135] ? nmi_handle+0xd4/0x260 [39555.787139] ? default_do_nmi+0x49/0xf0 [39555.787141] ? exc_nmi+0x11e/0x150 [39555.787142] ? end_repeat_nmi+0x16/0x67 [39555.787146] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39555.787148] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39555.787151] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39555.787153] [39555.787154] [39555.787155] do_raw_spin_lock+0xb5/0xc0 [39555.787157] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39555.787170] try_get_cap_refs+0x159/0x880 [ceph] [39555.787184] __ceph_get_caps+0x4a1/0x760 [ceph] [39555.787196] ? __file_remove_privs+0x89/0x110 [39555.787198] ? wake_bit_function+0x70/0x70 [39555.787200] ? ceph_write_iter+0x260/0x770 [ceph] [39555.787210] ceph_write_iter+0x260/0x770 [ceph] [39555.787221] ? lock_is_held_type+0xaa/0x120 [39555.787224] vfs_write+0x324/0x4f0 [39555.787228] ksys_pwrite64+0x65/0xa0 [39555.787231] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39555.787234] do_syscall_64+0x38/0x80 [39555.787254] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39555.787256] RIP: 0033:0x7f48f0eeb93b [39555.787257] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39555.787258] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [39555.787259] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39555.787260] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39555.787261] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39555.787261] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39555.787262] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39555.787265] [39555.788109] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39556.091642] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39556.099113] Workqueue: events delayed_work [ceph] [39556.103835] RIP: 0010:lock_acquire+0xf0/0x2b0 [39556.108186] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39556.126925] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39556.132143] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39556.139266] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39556.146391] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39556.153514] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39556.160656] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39556.167781] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39556.175857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39556.181595] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39556.188720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39556.195842] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39556.202967] Call Trace: [39556.205414] [39556.207422] ? rcu_dump_cpu_stacks+0xd1/0x180 [39556.211784] ? rcu_sched_clock_irq+0x633/0x1390 [39556.216315] ? lock_release+0xc5/0x260 [39556.220058] ? lock_is_held_type+0xaa/0x120 [39556.224239] ? lock_is_held_type+0xaa/0x120 [39556.228443] ? tick_sched_do_timer+0xa0/0xa0 [39556.232713] ? rcu_read_lock_sched_held+0x56/0x90 [39556.237420] ? tick_sched_do_timer+0xa0/0xa0 [39556.241692] ? update_process_times+0x5f/0x90 [39556.246043] ? tick_sched_handle.isra.16+0x1e/0x50 [39556.250834] ? tick_sched_timer+0x6f/0x80 [39556.254842] ? __hrtimer_run_queues+0x1d3/0x360 [39556.259380] ? hrtimer_interrupt+0x104/0x240 [39556.263643] ? __local_bh_enable+0x50/0x60 [39556.267740] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39556.272961] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39556.277918] [39556.280014] [39556.282115] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39556.287437] ? lock_acquire+0xf0/0x2b0 [39556.291191] ? igrab+0x43/0x50 [39556.294266] ? lock_release+0xc5/0x260 [39556.298021] _raw_spin_lock+0x30/0x40 [39556.301683] ? igrab+0x1d/0x50 [39556.304733] igrab+0x1d/0x50 [39556.307613] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39556.312763] delayed_work+0x1d8/0x280 [ceph] [39556.317055] ? process_scheduled_works+0x22a/0x510 [39556.321840] process_scheduled_works+0x2a4/0x510 [39556.326462] worker_thread+0x188/0x340 [39556.330216] ? rescuer_thread+0x310/0x310 [39556.334227] kthread+0xf5/0x130 [39556.337365] ? kthread_complete_and_exit+0x20/0x20 [39556.342160] ret_from_fork+0x31/0x40 [39556.345736] ? kthread_complete_and_exit+0x20/0x20 [39556.350521] ret_from_fork_asm+0x11/0x20 [39556.354459] [39582.146298] watchdog: BUG: soft lockup - CPU#0 stuck for 25749s! [dbench:78683] [39582.153604] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39582.158298] watchdog: BUG: soft lockup - CPU#7 stuck for 34746s! [kworker/7:4:78808] [39582.209053] irq event stamp: 70277612 [39582.216783] Modules linked in: ceph libceph dns_resolver fscache netfs [39582.220441] hardirqs last enabled at (70277611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39582.226979] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [39582.237134] hardirqs last disabled at (70277612): [] sysvec_apic_timer_interrupt+0xe/0x90 [39582.237137] nf_defrag_ipv6 [39582.237139] softirqs last enabled at (70277580): [] __do_softirq+0x2be/0x371 [39582.243226] nf_defrag_ipv4 nft_compat [39582.252950] softirqs last disabled at (70277575): [] irq_exit_rcu+0xb0/0xd0 [39582.255741] nf_tables [39582.264426] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39582.268169] nfnetlink bridge [39582.276691] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39582.279047] stp llc [39582.287992] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39582.290954] xfs libcrc32c sunrpc [39582.298444] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39582.300625] intel_rapl_msr intel_rapl_common [39582.306104] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39582.309413] sb_edac x86_pkg_temp_thermal [39582.328151] [39582.332503] coretemp kvm_intel [39582.337720] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39582.341725] kvm irqbypass [39582.343214] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39582.346351] i2c_i801 iTCO_wdt [39582.353476] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39582.356180] iTCO_vendor_support [39582.363305] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39582.366356] i2c_smbus [39582.373479] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39582.376705] crct10dif_pclmul [39582.383828] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39582.386185] joydev crc32_pclmul [39582.393310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39582.396272] ipmi_ssif [39582.404351] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39582.407577] ghash_clmulni_intel [39582.413312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39582.415670] mei_me [39582.422812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39582.426034] mei lpc_ich [39582.433159] Call Trace: [39582.435256] sha512_ssse3 acpi_ipmi [39582.442385] [39582.444913] mfd_core [39582.447358] ? watchdog_timer_fn+0x1b4/0x220 [39582.450840] wmi ipmi_si [39582.452851] ? __hrtimer_run_queues+0x1d3/0x360 [39582.455122] ipmi_devintf [39582.459390] ? hrtimer_interrupt+0x104/0x240 [39582.461917] ipmi_msghandler ioatdma [39582.466440] ? __local_bh_enable+0x50/0x60 [39582.469057] acpi_pad acpi_power_meter [39582.473325] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39582.476893] ixgbe igb [39582.480984] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39582.484727] mdio crc32c_intel [39582.489946] [39582.492301] nvme ptp [39582.497260] [39582.497262] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39582.500309] nvme_core [39582.502411] ? queued_spin_lock_slowpath+0x18/0x2e0 [39582.504694] i2c_algo_bit [39582.506794] do_raw_spin_lock+0xb5/0xc0 [39582.512098] pps_core [39582.514456] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39582.519326] dca fuse [39582.521946] try_get_cap_refs+0x159/0x880 [ceph] [39582.525773] [39582.525774] irq event stamp: 236903278 [39582.528048] __ceph_get_caps+0x4a1/0x760 [ceph] [39582.532827] hardirqs last enabled at (236903277): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39582.535102] ? __file_remove_privs+0x89/0x110 [39582.539710] hardirqs last disabled at (236903278): [] sysvec_apic_timer_interrupt+0xe/0x90 [39582.541201] ? wake_bit_function+0x70/0x70 [39582.544944] softirqs last enabled at (236903276): [] __do_softirq+0x2be/0x371 [39582.549470] ? ceph_write_iter+0x260/0x770 [ceph] [39582.559712] softirqs last disabled at (236903271): [] irq_exit_rcu+0xb0/0xd0 [39582.564063] ceph_write_iter+0x260/0x770 [ceph] [39582.573875] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39582.577968] ? lock_is_held_type+0xaa/0x120 [39582.586736] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39582.591436] vfs_write+0x324/0x4f0 [39582.600030] Workqueue: events delayed_work [ceph] [39582.604559] ksys_pwrite64+0x65/0xa0 [39582.613931] [39582.613932] RIP: 0010:lock_release+0x14e/0x260 [39582.618111] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39582.625580] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [39582.628979] do_syscall_64+0x38/0x80 [39582.633674] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [39582.637265] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39582.638754] [39582.638755] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [39582.643192] RIP: 0033:0x7f48f0eeb93b [39582.648322] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [39582.667060] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39582.670631] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39582.675847] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39582.680892] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39582.682384] ORIG_RAX: 0000000000000012 [39582.689506] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [39582.693079] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39582.700202] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39582.718939] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39582.726063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39582.731280] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39582.738405] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39582.742236] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39582.749360] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39582.756485] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39582.764563] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39582.771705] [39582.777441] Call Trace: [39582.824825] [39582.826834] ? watchdog_timer_fn+0x1b4/0x220 [39582.831099] ? __hrtimer_run_queues+0x1d3/0x360 [39582.835624] ? hrtimer_interrupt+0x104/0x240 [39582.839886] ? __local_bh_enable+0x50/0x60 [39582.843983] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39582.849203] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39582.854159] [39582.856258] [39582.858354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39582.863660] ? igrab+0x43/0x50 [39582.866720] ? lock_release+0x14e/0x260 [39582.870548] ? lock_release+0xc5/0x260 [39582.874294] _raw_spin_unlock+0x1b/0x40 [39582.878133] igrab+0x43/0x50 [39582.881012] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39582.886160] delayed_work+0x1d8/0x280 [ceph] [39582.890441] ? process_scheduled_works+0x22a/0x510 [39582.895231] process_scheduled_works+0x2a4/0x510 [39582.899847] worker_thread+0x188/0x340 [39582.903597] ? rescuer_thread+0x310/0x310 [39582.907600] kthread+0xf5/0x130 [39582.910747] ? kthread_complete_and_exit+0x20/0x20 [39582.915530] ret_from_fork+0x31/0x40 [39582.919111] ? kthread_complete_and_exit+0x20/0x20 [39582.923893] ret_from_fork_asm+0x11/0x20 [39582.927824] [39610.146751] watchdog: BUG: soft lockup - CPU#0 stuck for 25775s! [dbench:78683] [39610.154056] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39610.158751] watchdog: BUG: soft lockup - CPU#7 stuck for 34772s! [kworker/7:4:78808] [39610.209498] irq event stamp: 70335772 [39610.209499] hardirqs last enabled at (70335771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39610.217228] Modules linked in: [39610.220884] hardirqs last disabled at (70335772): [] sysvec_apic_timer_interrupt+0xe/0x90 [39610.231041] ceph [39610.234093] softirqs last enabled at (70335744): [] __do_softirq+0x2be/0x371 [39610.243816] libceph [39610.245742] softirqs last disabled at (70335739): [] irq_exit_rcu+0xb0/0xd0 [39610.254428] dns_resolver [39610.256611] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39610.265119] fscache [39610.267756] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39610.276700] netfs veth [39610.278884] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39610.286355] nft_chain_nat [39610.288799] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39610.294274] xt_MASQUERADE [39610.296980] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39610.315718] nf_nat [39610.318421] [39610.323657] nf_conntrack [39610.325754] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39610.327244] nf_defrag_ipv6 nf_defrag_ipv4 [39610.329862] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39610.336985] nft_compat nf_tables [39610.341077] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39610.348201] nfnetlink [39610.351511] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39610.358635] bridge stp [39610.360993] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39610.368117] llc [39610.370560] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39610.377685] xfs libcrc32c [39610.379523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39610.387601] sunrpc [39610.390304] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39610.396041] intel_rapl_msr intel_rapl_common [39610.398148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39610.398149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39610.405271] sb_edac [39610.409622] Call Trace: [39610.409624] [39610.416748] x86_pkg_temp_thermal coretemp [39610.423870] ? watchdog_timer_fn+0x1b4/0x220 [39610.426054] kvm_intel kvm [39610.428501] ? __hrtimer_run_queues+0x1d3/0x360 [39610.430510] irqbypass i2c_i801 [39610.434604] ? hrtimer_interrupt+0x104/0x240 [39610.438864] iTCO_wdt iTCO_vendor_support [39610.441568] ? __local_bh_enable+0x50/0x60 [39610.446093] i2c_smbus [39610.449231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39610.453493] crct10dif_pclmul joydev [39610.457498] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39610.461588] crc32_pclmul ipmi_ssif [39610.463947] [39610.469164] ghash_clmulni_intel [39610.472734] [39610.477692] mei_me [39610.481176] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39610.483273] mei lpc_ich [39610.486502] ? queued_spin_lock_slowpath+0x18/0x2e0 [39610.488594] sha512_ssse3 acpi_ipmi [39610.490694] do_raw_spin_lock+0xb5/0xc0 [39610.495996] mfd_core wmi [39610.498527] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39610.503399] ipmi_si [39610.506883] try_get_cap_refs+0x159/0x880 [ceph] [39610.510713] ipmi_devintf [39610.513333] __ceph_get_caps+0x4a1/0x760 [ceph] [39610.518113] ipmi_msghandler ioatdma [39610.520303] ? __file_remove_privs+0x89/0x110 [39610.524908] acpi_pad acpi_power_meter [39610.527528] ? wake_bit_function+0x70/0x70 [39610.532050] ixgbe igb [39610.535622] ? ceph_write_iter+0x260/0x770 [ceph] [39610.539971] mdio crc32c_intel [39610.543717] ceph_write_iter+0x260/0x770 [ceph] [39610.547806] nvme ptp [39610.550167] ? lock_is_held_type+0xaa/0x120 [39610.554861] nvme_core i2c_algo_bit [39610.557915] vfs_write+0x324/0x4f0 [39610.562436] pps_core dca [39610.564712] ksys_pwrite64+0x65/0xa0 [39610.568884] fuse [39610.572369] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39610.575766] irq event stamp: 237124030 [39610.578384] do_syscall_64+0x38/0x80 [39610.581954] hardirqs last enabled at (237124029): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39610.583878] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39610.589007] hardirqs last disabled at (237124030): [] sysvec_apic_timer_interrupt+0xe/0x90 [39610.592752] RIP: 0033:0x7f48f0eeb93b [39610.596323] softirqs last enabled at (237124028): [] __do_softirq+0x2be/0x371 [39610.606568] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39610.611610] softirqs last disabled at (237124023): [] irq_exit_rcu+0xb0/0xd0 [39610.621432] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39610.625003] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39610.633773] ORIG_RAX: 0000000000000012 [39610.652511] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39610.661107] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39610.666326] Workqueue: events delayed_work [ceph] [39610.675703] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39610.679535] [39610.687003] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39610.694128] RIP: 0010:lock_acquire+0xf0/0x2b0 [39610.698844] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39610.705968] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39610.707459] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39610.714581] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39610.718938] [39610.726056] [39610.726057] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39610.767953] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39610.775077] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39610.782201] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39610.789324] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39610.796449] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39610.804528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39610.810263] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39610.817387] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39610.824511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39610.831636] Call Trace: [39610.834082] [39610.836092] ? watchdog_timer_fn+0x1b4/0x220 [39610.840358] ? __hrtimer_run_queues+0x1d3/0x360 [39610.844892] ? hrtimer_interrupt+0x104/0x240 [39610.849162] ? __local_bh_enable+0x50/0x60 [39610.853255] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39610.858478] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39610.863451] [39610.865550] [39610.867647] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39610.872955] ? lock_acquire+0xf0/0x2b0 [39610.876705] ? igrab+0x43/0x50 [39610.879754] ? lock_release+0xc5/0x260 [39610.883500] _raw_spin_lock+0x30/0x40 [39610.887166] ? igrab+0x1d/0x50 [39610.890215] igrab+0x1d/0x50 [39610.893096] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39610.898244] delayed_work+0x1d8/0x280 [ceph] [39610.902525] ? process_scheduled_works+0x22a/0x510 [39610.907315] process_scheduled_works+0x2a4/0x510 [39610.911931] worker_thread+0x188/0x340 [39610.915681] ? rescuer_thread+0x310/0x310 [39610.919692] kthread+0xf5/0x130 [39610.922828] ? kthread_complete_and_exit+0x20/0x20 [39610.927615] ret_from_fork+0x31/0x40 [39610.931184] ? kthread_complete_and_exit+0x20/0x20 [39610.935969] ret_from_fork_asm+0x11/0x20 [39610.939901] [39638.147205] watchdog: BUG: soft lockup - CPU#0 stuck for 25801s! [dbench:78683] [39638.154510] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39638.159204] watchdog: BUG: soft lockup - CPU#7 stuck for 34798s! [kworker/7:4:78808] [39638.209959] irq event stamp: 70394142 [39638.217689] Modules linked in: ceph libceph dns_resolver fscache [39638.221347] hardirqs last enabled at (70394141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39638.221352] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [39638.237510] hardirqs last disabled at (70394142): [] sysvec_apic_timer_interrupt+0xe/0x90 [39638.237513] nf_defrag_ipv6 [39638.237514] softirqs last enabled at (70394114): [] __do_softirq+0x2be/0x371 [39638.244113] nf_defrag_ipv4 nft_compat [39638.253837] softirqs last disabled at (70394109): [] irq_exit_rcu+0xb0/0xd0 [39638.256628] nf_tables [39638.265313] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39638.269056] nfnetlink bridge [39638.277569] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39638.279925] stp llc [39638.288871] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39638.291850] xfs libcrc32c [39638.299321] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39638.301507] sunrpc intel_rapl_msr [39638.306983] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39638.309687] intel_rapl_common sb_edac [39638.328425] [39638.328426] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39638.331822] x86_pkg_temp_thermal [39638.337041] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39638.340785] coretemp [39638.342274] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39638.349399] kvm_intel [39638.352709] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39638.359835] kvm irqbypass [39638.362104] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39638.369228] i2c_i801 iTCO_wdt [39638.371601] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39638.378726] iTCO_vendor_support [39638.381431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39638.388555] i2c_smbus crct10dif_pclmul [39638.391606] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39638.399683] joydev crc32_pclmul [39638.402907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39638.408644] ipmi_ssif [39638.412475] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39638.419601] ghash_clmulni_intel [39638.422823] Call Trace: [39638.422826] [39638.429948] mei_me mei [39638.432307] ? watchdog_timer_fn+0x1b4/0x220 [39638.439431] lpc_ich [39638.442656] ? __hrtimer_run_queues+0x1d3/0x360 [39638.445099] sha512_ssse3 [39638.447113] ? hrtimer_interrupt+0x104/0x240 [39638.449554] acpi_ipmi mfd_core [39638.453816] ? __local_bh_enable+0x50/0x60 [39638.456000] wmi [39638.460527] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39638.463142] ipmi_si ipmi_devintf [39638.467406] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39638.470544] ipmi_msghandler [39638.474636] [39638.476472] ioatdma [39638.481688] [39638.481689] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39638.484999] acpi_pad [39638.489962] ? queued_spin_lock_slowpath+0x16/0x2e0 [39638.492835] acpi_power_meter ixgbe [39638.494934] do_raw_spin_lock+0xb5/0xc0 [39638.497115] igb mdio [39638.499217] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39638.504517] crc32c_intel nvme [39638.506791] try_get_cap_refs+0x159/0x880 [ceph] [39638.511658] ptp nvme_core [39638.515148] __ceph_get_caps+0x4a1/0x760 [ceph] [39638.518975] i2c_algo_bit [39638.521251] ? __file_remove_privs+0x89/0x110 [39638.526030] pps_core dca [39638.529080] ? wake_bit_function+0x70/0x70 [39638.533689] fuse [39638.536397] ? ceph_write_iter+0x260/0x770 [ceph] [39638.540919] irq event stamp: 237344774 [39638.543536] ceph_write_iter+0x260/0x770 [ceph] [39638.547887] hardirqs last enabled at (237344773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39638.550508] ? lock_is_held_type+0xaa/0x120 [39638.554604] hardirqs last disabled at (237344774): [] sysvec_apic_timer_interrupt+0xe/0x90 [39638.556531] vfs_write+0x324/0x4f0 [39638.561225] softirqs last enabled at (237344772): [] __do_softirq+0x2be/0x371 [39638.564974] ksys_pwrite64+0x65/0xa0 [39638.569493] softirqs last disabled at (237344767): [] irq_exit_rcu+0xb0/0xd0 [39638.579737] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39638.583915] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39638.593726] do_syscall_64+0x38/0x80 [39638.597124] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39638.605894] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39638.609465] Workqueue: events delayed_work [ceph] [39638.618061] RIP: 0033:0x7f48f0eeb93b [39638.623193] [39638.623194] RIP: 0010:lock_acquire+0xf0/0x2b0 [39638.632571] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39638.636140] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39638.643610] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39638.648656] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39638.653353] ORIG_RAX: 0000000000000012 [39638.656923] [39638.656924] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39638.658414] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39638.662765] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39638.681504] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39638.700241] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39638.705458] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39638.710674] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39638.714505] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39638.715996] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39638.723119] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39638.730245] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39638.737375] [39638.744494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39638.803238] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39638.810361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39638.817484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39638.824610] Call Trace: [39638.827056] [39638.829066] ? watchdog_timer_fn+0x1b4/0x220 [39638.833330] ? __hrtimer_run_queues+0x1d3/0x360 [39638.837857] ? hrtimer_interrupt+0x104/0x240 [39638.842126] ? __local_bh_enable+0x50/0x60 [39638.846220] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39638.851443] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39638.856400] [39638.858497] [39638.860610] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39638.865921] ? lock_acquire+0xf0/0x2b0 [39638.869671] ? igrab+0x43/0x50 [39638.872730] ? lock_release+0xc5/0x260 [39638.876473] _raw_spin_lock+0x30/0x40 [39638.880129] ? igrab+0x1d/0x50 [39638.883182] igrab+0x1d/0x50 [39638.886058] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39638.891209] delayed_work+0x1d8/0x280 [ceph] [39638.895489] ? process_scheduled_works+0x22a/0x510 [39638.900280] process_scheduled_works+0x2a4/0x510 [39638.904903] worker_thread+0x188/0x340 [39638.908652] ? rescuer_thread+0x310/0x310 [39638.912656] kthread+0xf5/0x130 [39638.915794] ? kthread_complete_and_exit+0x20/0x20 [39638.920579] ret_from_fork+0x31/0x40 [39638.924175] ? kthread_complete_and_exit+0x20/0x20 [39638.928958] ret_from_fork_asm+0x11/0x20 [39638.932898] [39666.147663] watchdog: BUG: soft lockup - CPU#0 stuck for 25827s! [dbench:78683] [39666.154960] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39666.159662] watchdog: BUG: soft lockup - CPU#7 stuck for 34824s! [kworker/7:4:78808] [39666.210402] irq event stamp: 70452358 [39666.210403] hardirqs last enabled at (70452357): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39666.218132] Modules linked in: [39666.221790] hardirqs last disabled at (70452358): [] sysvec_apic_timer_interrupt+0xe/0x90 [39666.231949] ceph [39666.234999] softirqs last enabled at (70452330): [] __do_softirq+0x2be/0x371 [39666.244722] libceph [39666.246648] softirqs last disabled at (70452325): [] irq_exit_rcu+0xb0/0xd0 [39666.255338] dns_resolver [39666.257525] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39666.266033] fscache [39666.268668] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39666.277614] netfs [39666.279798] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39666.287269] veth nft_chain_nat [39666.289279] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39666.294755] xt_MASQUERADE nf_nat [39666.297893] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39666.316630] nf_conntrack nf_defrag_ipv6 [39666.319943] [39666.319943] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39666.325160] nf_defrag_ipv4 nft_compat [39666.329078] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39666.330566] nf_tables nfnetlink [39666.337691] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39666.341435] bridge [39666.348561] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39666.351784] stp llc [39666.358907] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39666.361006] xfs [39666.368130] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39666.370313] libcrc32c sunrpc [39666.377456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39666.379292] intel_rapl_msr intel_rapl_common [39666.387369] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39666.390348] sb_edac [39666.396090] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39666.400439] x86_pkg_temp_thermal [39666.407565] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39666.409747] coretemp [39666.416873] Call Trace: [39666.416874] [39666.420183] kvm_intel kvm [39666.427309] ? watchdog_timer_fn+0x1b4/0x220 [39666.429576] irqbypass i2c_i801 [39666.432024] ? __hrtimer_run_queues+0x1d3/0x360 [39666.434033] iTCO_wdt iTCO_vendor_support [39666.436757] ? hrtimer_interrupt+0x104/0x240 [39666.441019] i2c_smbus crct10dif_pclmul [39666.444156] ? __local_bh_enable+0x50/0x60 [39666.448680] joydev crc32_pclmul [39666.452686] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39666.456947] ipmi_ssif ghash_clmulni_intel [39666.460778] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39666.464869] mei_me [39666.468093] [39666.473310] mei lpc_ich [39666.477402] [39666.482358] sha512_ssse3 acpi_ipmi [39666.484457] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39666.486554] mfd_core wmi [39666.489088] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39666.491182] ipmi_si ipmi_devintf [39666.494666] do_raw_spin_lock+0xb5/0xc0 [39666.499969] ipmi_msghandler ioatdma [39666.502588] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39666.507457] acpi_pad acpi_power_meter ixgbe [39666.510772] try_get_cap_refs+0x159/0x880 [ceph] [39666.514598] igb mdio [39666.518175] __ceph_get_caps+0x4a1/0x760 [ceph] [39666.522953] crc32c_intel nvme [39666.527222] ? __file_remove_privs+0x89/0x110 [39666.531829] ptp nvme_core [39666.534101] ? wake_bit_function+0x70/0x70 [39666.538624] i2c_algo_bit pps_core [39666.541677] ? ceph_write_iter+0x260/0x770 [ceph] [39666.546024] dca fuse [39666.548730] ceph_write_iter+0x260/0x770 [ceph] [39666.552820] [39666.552821] irq event stamp: 237565618 [39666.556221] ? lock_is_held_type+0xaa/0x120 [39666.560914] hardirqs last enabled at (237565617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39666.563187] vfs_write+0x324/0x4f0 [39666.567708] hardirqs last disabled at (237565618): [] sysvec_apic_timer_interrupt+0xe/0x90 [39666.569205] ksys_pwrite64+0x65/0xa0 [39666.572944] softirqs last enabled at (237565616): [] __do_softirq+0x2be/0x371 [39666.577123] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39666.587366] softirqs last disabled at (237565611): [] irq_exit_rcu+0xb0/0xd0 [39666.590764] do_syscall_64+0x38/0x80 [39666.600575] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39666.604147] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39666.612916] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39666.618065] RIP: 0033:0x7f48f0eeb93b [39666.626662] Workqueue: events delayed_work [ceph] [39666.630233] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39666.639609] [39666.639610] RIP: 0010:__list_del_entry_valid_or_report+0x10/0xd0 [39666.644654] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39666.652125] Code: 89 c1 48 89 fe 48 c7 c7 60 99 1d 82 e8 69 0c b0 ff 0f 0b 31 c0 c3 0f 1f 40 00 f3 0f 1e fa 48 8b 0f 48 8b 57 08 48 85 c9 74 3a <48> 85 d2 74 49 48 b8 00 01 00 00 00 00 ad de 48 39 c1 74 4e 48 b8 [39666.655695] ORIG_RAX: 0000000000000012 [39666.660393] RSP: 0018:ffffc90002117db0 EFLAGS: 00000282 [39666.679129] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39666.680620] [39666.680621] RAX: 0000000000000000 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [39666.686618] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39666.691835] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34518 RDI: ffff8881a1d34bc8 [39666.710574] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39666.714404] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [39666.719620] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39666.726746] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [39666.728236] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39666.735361] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [39666.742490] [39666.749610] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39666.802616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39666.808352] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39666.815478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39666.822602] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39666.829725] Call Trace: [39666.832169] [39666.834181] ? watchdog_timer_fn+0x1b4/0x220 [39666.838445] ? __hrtimer_run_queues+0x1d3/0x360 [39666.842973] ? hrtimer_interrupt+0x104/0x240 [39666.847240] ? __local_bh_enable+0x50/0x60 [39666.851349] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39666.856567] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39666.861523] [39666.863622] [39666.865719] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39666.871028] ? __list_del_entry_valid_or_report+0x10/0xd0 [39666.876424] ceph_check_delayed_caps+0x8f/0x270 [ceph] [39666.881573] delayed_work+0x1d8/0x280 [ceph] [39666.885854] ? process_scheduled_works+0x22a/0x510 [39666.890644] process_scheduled_works+0x2a4/0x510 [39666.895260] worker_thread+0x188/0x340 [39666.899008] ? rescuer_thread+0x310/0x310 [39666.903011] kthread+0xf5/0x130 [39666.906150] ? kthread_complete_and_exit+0x20/0x20 [39666.910933] ret_from_fork+0x31/0x40 [39666.914503] ? kthread_complete_and_exit+0x20/0x20 [39666.919290] ret_from_fork_asm+0x11/0x20 [39666.923220] [39694.148122] watchdog: BUG: soft lockup - CPU#0 stuck for 25854s! [dbench:78683] [39694.155422] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39694.160122] watchdog: BUG: soft lockup - CPU#7 stuck for 34851s! [kworker/7:4:78808] [39694.210864] irq event stamp: 70510612 [39694.210865] hardirqs last enabled at (70510611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39694.218595] Modules linked in: [39694.222252] hardirqs last disabled at (70510612): [] sysvec_apic_timer_interrupt+0xe/0x90 [39694.232411] ceph [39694.235459] softirqs last enabled at (70510584): [] __do_softirq+0x2be/0x371 [39694.245185] libceph [39694.247107] softirqs last disabled at (70510579): [] irq_exit_rcu+0xb0/0xd0 [39694.255806] dns_resolver [39694.257995] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39694.266518] fscache [39694.269140] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39694.278082] netfs [39694.280266] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39694.287739] veth [39694.289751] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39694.295226] nft_chain_nat [39694.297150] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39694.315889] xt_MASQUERADE [39694.318592] [39694.318593] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39694.323810] nf_nat [39694.326519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39694.328013] nf_conntrack [39694.335136] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39694.337236] nf_defrag_ipv6 [39694.344359] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39694.346975] nf_defrag_ipv4 [39694.354101] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39694.356891] nft_compat [39694.364014] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39694.366806] nf_tables [39694.373930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39694.376375] nfnetlink [39694.384451] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39694.386809] bridge stp [39694.392546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39694.394904] llc [39694.402029] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39694.404474] xfs [39694.411597] Call Trace: [39694.413434] libcrc32c [39694.420559] [39694.422395] sunrpc intel_rapl_msr [39694.424840] ? watchdog_timer_fn+0x1b4/0x220 [39694.427197] intel_rapl_common [39694.429208] ? __hrtimer_run_queues+0x1d3/0x360 [39694.432605] sb_edac x86_pkg_temp_thermal [39694.436872] ? hrtimer_interrupt+0x104/0x240 [39694.439918] coretemp [39694.444443] ? __local_bh_enable+0x50/0x60 [39694.448447] kvm_intel [39694.452713] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39694.454983] kvm [39694.459074] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39694.461432] irqbypass [39694.466647] [39694.468487] i2c_i801 [39694.473443] [39694.475802] iTCO_wdt iTCO_vendor_support [39694.477898] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39694.480169] i2c_smbus [39694.482269] ? queued_spin_lock_slowpath+0x18/0x2e0 [39694.486268] crct10dif_pclmul joydev [39694.491575] do_raw_spin_lock+0xb5/0xc0 [39694.493930] crc32_pclmul ipmi_ssif [39694.498813] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39694.502381] ghash_clmulni_intel [39694.506215] try_get_cap_refs+0x159/0x880 [ceph] [39694.509695] mei_me mei [39694.514485] __ceph_get_caps+0x4a1/0x760 [ceph] [39694.517704] lpc_ich sha512_ssse3 [39694.522320] ? __file_remove_privs+0x89/0x110 [39694.524759] acpi_ipmi [39694.529284] ? wake_bit_function+0x70/0x70 [39694.532594] mfd_core [39694.536947] ? ceph_write_iter+0x260/0x770 [ceph] [39694.539302] wmi [39694.543391] ceph_write_iter+0x260/0x770 [ceph] [39694.545662] ipmi_si ipmi_devintf [39694.550365] ? lock_is_held_type+0xaa/0x120 [39694.552198] ipmi_msghandler ioatdma [39694.556725] vfs_write+0x324/0x4f0 [39694.560033] acpi_pad acpi_power_meter [39694.564216] ksys_pwrite64+0x65/0xa0 [39694.567780] ixgbe [39694.571196] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39694.574938] igb mdio [39694.578524] do_syscall_64+0x38/0x80 [39694.580529] crc32c_intel [39694.585661] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39694.587932] nvme ptp [39694.591521] RIP: 0033:0x7f48f0eeb93b [39694.594137] nvme_core i2c_algo_bit [39694.599183] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39694.601453] pps_core [39694.605021] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39694.608523] dca fuse [39694.627262] ORIG_RAX: 0000000000000012 [39694.629532] [39694.629533] irq event stamp: 237786416 [39694.634749] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39694.637019] hardirqs last enabled at (237786415): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39694.640850] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39694.642343] hardirqs last disabled at (237786416): [] sysvec_apic_timer_interrupt+0xe/0x90 [39694.646085] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39694.653210] softirqs last enabled at (237786414): [] __do_softirq+0x2be/0x371 [39694.663453] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39694.670577] softirqs last disabled at (237786409): [] irq_exit_rcu+0xb0/0xd0 [39694.680390] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39694.687523] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39694.696299] [39694.703416] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39694.738179] Workqueue: events delayed_work [ceph] [39694.742894] RIP: 0010:lock_release+0x14e/0x260 [39694.747332] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [39694.766070] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [39694.771287] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [39694.778410] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [39694.785535] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [39694.792660] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [39694.799783] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [39694.806908] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39694.814984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39694.820722] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39694.827846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39694.834972] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39694.842096] Call Trace: [39694.844540] [39694.846551] ? watchdog_timer_fn+0x1b4/0x220 [39694.850816] ? __hrtimer_run_queues+0x1d3/0x360 [39694.855342] ? hrtimer_interrupt+0x104/0x240 [39694.859610] ? __local_bh_enable+0x50/0x60 [39694.863703] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39694.868928] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39694.873885] [39694.875983] [39694.878080] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39694.883384] ? igrab+0x43/0x50 [39694.886437] ? lock_release+0x14e/0x260 [39694.890277] ? lock_release+0xc5/0x260 [39694.894019] _raw_spin_unlock+0x1b/0x40 [39694.897859] igrab+0x43/0x50 [39694.900737] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39694.905887] delayed_work+0x1d8/0x280 [ceph] [39694.910167] ? process_scheduled_works+0x22a/0x510 [39694.914958] process_scheduled_works+0x2a4/0x510 [39694.919573] worker_thread+0x188/0x340 [39694.923321] ? rescuer_thread+0x310/0x310 [39694.927325] kthread+0xf5/0x130 [39694.930471] ? kthread_complete_and_exit+0x20/0x20 [39694.935256] ret_from_fork+0x31/0x40 [39694.938835] ? kthread_complete_and_exit+0x20/0x20 [39694.943618] ret_from_fork_asm+0x11/0x20 [39694.947543] [39722.148583] watchdog: BUG: soft lockup - CPU#0 stuck for 25880s! [dbench:78683] [39722.155885] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39722.160582] watchdog: BUG: soft lockup - CPU#7 stuck for 34877s! [kworker/7:4:78808] [39722.211334] irq event stamp: 70568902 [39722.211336] hardirqs last enabled at (70568901): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39722.219066] Modules linked in: [39722.222721] hardirqs last disabled at (70568902): [] sysvec_apic_timer_interrupt+0xe/0x90 [39722.232879] ceph [39722.235931] softirqs last enabled at (70568874): [] __do_softirq+0x2be/0x371 [39722.245655] libceph [39722.247578] softirqs last disabled at (70568869): [] irq_exit_rcu+0xb0/0xd0 [39722.256264] dns_resolver [39722.258446] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39722.266956] fscache [39722.269592] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39722.278535] netfs veth [39722.280720] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39722.288190] nft_chain_nat [39722.290634] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39722.296113] xt_MASQUERADE nf_nat [39722.298817] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39722.317554] nf_conntrack [39722.320866] [39722.320867] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39722.326083] nf_defrag_ipv6 [39722.328699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39722.330190] nf_defrag_ipv4 [39722.337316] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39722.340106] nft_compat nf_tables [39722.347228] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39722.350020] nfnetlink [39722.357144] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39722.360456] bridge stp [39722.367580] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39722.369935] llc xfs [39722.377061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39722.379505] libcrc32c sunrpc [39722.387583] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39722.389765] intel_rapl_msr intel_rapl_common [39722.395504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39722.398466] sb_edac x86_pkg_temp_thermal [39722.405591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39722.409942] coretemp kvm_intel [39722.417067] Call Trace: [39722.417069] [39722.421072] kvm [39722.428194] ? watchdog_timer_fn+0x1b4/0x220 [39722.431333] irqbypass i2c_i801 [39722.433777] ? __hrtimer_run_queues+0x1d3/0x360 [39722.435788] iTCO_wdt [39722.437628] ? hrtimer_interrupt+0x104/0x240 [39722.441888] iTCO_vendor_support i2c_smbus [39722.445027] ? __local_bh_enable+0x50/0x60 [39722.449549] crct10dif_pclmul [39722.451823] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39722.456086] joydev [39722.460177] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39722.464269] crc32_pclmul ipmi_ssif [39722.467230] [39722.472448] ghash_clmulni_intel [39722.474544] [39722.474545] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39722.479501] mei_me mei [39722.482991] ? queued_spin_lock_slowpath+0x16/0x2e0 [39722.485084] lpc_ich sha512_ssse3 [39722.488309] do_raw_spin_lock+0xb5/0xc0 [39722.490405] acpi_ipmi mfd_core [39722.495710] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39722.498152] wmi ipmi_si [39722.503026] try_get_cap_refs+0x159/0x880 [ceph] [39722.506336] ipmi_devintf ipmi_msghandler [39722.510170] __ceph_get_caps+0x4a1/0x760 [ceph] [39722.513304] ioatdma acpi_pad [39722.518092] ? __file_remove_privs+0x89/0x110 [39722.520619] acpi_power_meter ixgbe [39722.525229] ? wake_bit_function+0x70/0x70 [39722.529232] igb mdio [39722.533760] ? ceph_write_iter+0x260/0x770 [ceph] [39722.536721] crc32c_intel [39722.541071] ceph_write_iter+0x260/0x770 [ceph] [39722.544556] nvme ptp [39722.548650] ? lock_is_held_type+0xaa/0x120 [39722.550918] nvme_core i2c_algo_bit [39722.555618] vfs_write+0x324/0x4f0 [39722.558231] pps_core dca [39722.562761] ksys_pwrite64+0x65/0xa0 [39722.565026] fuse [39722.569203] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39722.572688] [39722.572689] irq event stamp: 238007146 [39722.576087] do_syscall_64+0x38/0x80 [39722.578702] hardirqs last enabled at (238007145): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39722.582273] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39722.584198] hardirqs last disabled at (238007146): [] sysvec_apic_timer_interrupt+0xe/0x90 [39722.589328] RIP: 0033:0x7f48f0eeb93b [39722.590819] softirqs last enabled at (238007144): [] __do_softirq+0x2be/0x371 [39722.594565] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39722.598133] softirqs last disabled at (238007139): [] irq_exit_rcu+0xb0/0xd0 [39722.608379] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39722.613423] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39722.623233] ORIG_RAX: 0000000000000012 [39722.626803] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39722.635575] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39722.654312] Workqueue: events delayed_work [ceph] [39722.662908] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39722.668126] [39722.668127] RIP: 0010:lock_acquire+0xf0/0x2b0 [39722.677505] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39722.681337] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39722.688806] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39722.695931] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39722.700629] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39722.707752] [39722.707753] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39722.709264] [39722.713610] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39722.776861] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39722.783986] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39722.791110] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39722.798234] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39722.806312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39722.812050] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39722.819173] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39722.826296] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39722.833420] Call Trace: [39722.835866] [39722.837877] ? watchdog_timer_fn+0x1b4/0x220 [39722.842142] ? __hrtimer_run_queues+0x1d3/0x360 [39722.846668] ? hrtimer_interrupt+0x104/0x240 [39722.850937] ? __local_bh_enable+0x50/0x60 [39722.855029] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39722.860253] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39722.865211] [39722.867309] [39722.869406] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39722.874715] ? lock_acquire+0xf0/0x2b0 [39722.878464] ? igrab+0x43/0x50 [39722.881515] ? lock_release+0xc5/0x260 [39722.885258] _raw_spin_lock+0x30/0x40 [39722.888932] ? igrab+0x1d/0x50 [39722.891990] igrab+0x1d/0x50 [39722.894870] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39722.900020] delayed_work+0x1d8/0x280 [ceph] [39722.904302] ? process_scheduled_works+0x22a/0x510 [39722.909092] process_scheduled_works+0x2a4/0x510 [39722.913706] worker_thread+0x188/0x340 [39722.917455] ? rescuer_thread+0x310/0x310 [39722.921459] kthread+0xf5/0x130 [39722.924598] ? kthread_complete_and_exit+0x20/0x20 [39722.929380] ret_from_fork+0x31/0x40 [39722.932950] ? kthread_complete_and_exit+0x20/0x20 [39722.937737] ret_from_fork_asm+0x11/0x20 [39722.941668] [39750.149041] watchdog: BUG: soft lockup - CPU#0 stuck for 25906s! [dbench:78683] [39750.156344] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39750.161041] watchdog: BUG: soft lockup - CPU#7 stuck for 34903s! [kworker/7:4:78808] [39750.211786] irq event stamp: 70627238 [39750.211788] hardirqs last enabled at (70627237): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39750.219518] Modules linked in: [39750.223175] hardirqs last disabled at (70627238): [] sysvec_apic_timer_interrupt+0xe/0x90 [39750.233333] ceph [39750.236384] softirqs last enabled at (70627210): [] __do_softirq+0x2be/0x371 [39750.246107] libceph [39750.248030] softirqs last disabled at (70627205): [] irq_exit_rcu+0xb0/0xd0 [39750.256716] dns_resolver fscache [39750.258899] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39750.267409] netfs veth [39750.270739] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39750.279682] nft_chain_nat xt_MASQUERADE [39750.282126] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39750.289598] nf_nat nf_conntrack [39750.293513] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39750.298993] nf_defrag_ipv6 nf_defrag_ipv4 [39750.302216] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39750.320953] nft_compat nf_tables [39750.325044] [39750.325045] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39750.330261] nfnetlink [39750.333572] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39750.335062] bridge stp [39750.342188] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39750.344543] llc xfs [39750.351668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39750.354112] libcrc32c sunrpc [39750.361236] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39750.363437] intel_rapl_msr [39750.370562] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39750.373526] intel_rapl_common sb_edac [39750.380650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39750.383442] x86_pkg_temp_thermal coretemp [39750.391520] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39750.395264] kvm_intel kvm [39750.401002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39750.405091] irqbypass [39750.412216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39750.414918] i2c_i801 iTCO_wdt [39750.422044] Call Trace: [39750.424400] iTCO_vendor_support i2c_smbus [39750.431525] [39750.434575] crct10dif_pclmul joydev [39750.437022] ? watchdog_timer_fn+0x1b4/0x220 [39750.441112] crc32_pclmul ipmi_ssif [39750.443123] ? __hrtimer_run_queues+0x1d3/0x360 [39750.446691] ghash_clmulni_intel [39750.450960] ? hrtimer_interrupt+0x104/0x240 [39750.454439] mei_me mei [39750.458964] ? __local_bh_enable+0x50/0x60 [39750.462189] lpc_ich sha512_ssse3 [39750.466454] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39750.468895] acpi_ipmi [39750.472988] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39750.476296] mfd_core wmi [39750.481514] [39750.483872] ipmi_si ipmi_devintf [39750.488830] [39750.491446] ipmi_msghandler ioatdma [39750.493547] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39750.496857] acpi_pad [39750.498957] ? queued_spin_lock_slowpath+0x16/0x2e0 [39750.502522] acpi_power_meter ixgbe [39750.507831] do_raw_spin_lock+0xb5/0xc0 [39750.510098] igb mdio [39750.514971] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39750.518452] crc32c_intel [39750.522285] try_get_cap_refs+0x159/0x880 [ceph] [39750.524555] nvme ptp [39750.529344] __ceph_get_caps+0x4a1/0x760 [ceph] [39750.531957] nvme_core i2c_algo_bit [39750.536572] ? __file_remove_privs+0x89/0x110 [39750.538837] pps_core dca [39750.543362] ? wake_bit_function+0x70/0x70 [39750.546846] fuse [39750.551198] ? ceph_write_iter+0x260/0x770 [ceph] [39750.553813] irq event stamp: 238227832 [39750.557906] ceph_write_iter+0x260/0x770 [ceph] [39750.559829] hardirqs last enabled at (238227831): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39750.564529] ? lock_is_held_type+0xaa/0x120 [39750.568269] hardirqs last disabled at (238227832): [] sysvec_apic_timer_interrupt+0xe/0x90 [39750.572797] vfs_write+0x324/0x4f0 [39750.583039] softirqs last enabled at (238227830): [] __do_softirq+0x2be/0x371 [39750.587220] ksys_pwrite64+0x65/0xa0 [39750.597027] softirqs last disabled at (238227825): [] irq_exit_rcu+0xb0/0xd0 [39750.600442] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39750.609213] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39750.612783] do_syscall_64+0x38/0x80 [39750.621379] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39750.626511] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39750.635890] Workqueue: events delayed_work [ceph] [39750.639459] RIP: 0033:0x7f48f0eeb93b [39750.646930] [39750.646932] RIP: 0010:lock_acquire+0xf0/0x2b0 [39750.651974] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39750.656671] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39750.660242] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39750.661733] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39750.666083] ORIG_RAX: 0000000000000012 [39750.684821] [39750.684822] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39750.703560] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39750.708777] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39750.713995] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39750.717824] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39750.719315] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39750.726445] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39750.733573] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39750.740697] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39750.747820] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39750.754945] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39750.762075] [39750.769194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39750.806565] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39750.813689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39750.820812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39750.827938] Call Trace: [39750.830382] [39750.832392] ? watchdog_timer_fn+0x1b4/0x220 [39750.836656] ? __hrtimer_run_queues+0x1d3/0x360 [39750.841184] ? hrtimer_interrupt+0x104/0x240 [39750.845452] ? __local_bh_enable+0x50/0x60 [39750.849546] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39750.854769] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39750.859736] [39750.861833] [39750.863931] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39750.869239] ? lock_acquire+0xf0/0x2b0 [39750.872989] ? igrab+0x43/0x50 [39750.876039] ? lock_release+0xc5/0x260 [39750.879784] _raw_spin_lock+0x30/0x40 [39750.883447] ? igrab+0x1d/0x50 [39750.886498] igrab+0x1d/0x50 [39750.889376] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39750.894526] delayed_work+0x1d8/0x280 [ceph] [39750.898808] ? process_scheduled_works+0x22a/0x510 [39750.903597] process_scheduled_works+0x2a4/0x510 [39750.908215] worker_thread+0x188/0x340 [39750.911964] ? rescuer_thread+0x310/0x310 [39750.915976] kthread+0xf5/0x130 [39750.919113] ? kthread_complete_and_exit+0x20/0x20 [39750.923896] ret_from_fork+0x31/0x40 [39750.927468] ? kthread_complete_and_exit+0x20/0x20 [39750.932252] ret_from_fork_asm+0x11/0x20 [39750.936173] [39751.361427] rcu: INFO: rcu_preempt self-detected stall on CPU [39751.367189] rcu: 0-....: (27185595 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=13616168 [39751.377700] rcu: (t=27814266 jiffies g=1309609 q=747928 ncpus=8) [39751.383786] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39751.392727] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39751.400216] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39751.405693] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39751.424431] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39751.429667] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39751.436792] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39751.443915] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39751.451039] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39751.458164] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39751.465286] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39751.473364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39751.479102] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39751.486225] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39751.493349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39751.500474] Call Trace: [39751.502918] [39751.504929] ? rcu_dump_cpu_stacks+0xd1/0x180 [39751.509289] ? rcu_sched_clock_irq+0x633/0x1390 [39751.513813] ? lock_is_held_type+0xaa/0x120 [39751.517990] ? lock_is_held_type+0xaa/0x120 [39751.522168] ? tick_sched_do_timer+0xa0/0xa0 [39751.526456] ? rcu_read_lock_sched_held+0x56/0x90 [39751.531155] ? tick_sched_do_timer+0xa0/0xa0 [39751.535418] ? update_process_times+0x5f/0x90 [39751.539769] ? tick_sched_handle.isra.16+0x1e/0x50 [39751.544554] ? tick_sched_timer+0x6f/0x80 [39751.548558] ? __hrtimer_run_queues+0x1d3/0x360 [39751.553085] ? hrtimer_interrupt+0x104/0x240 [39751.557347] ? __local_bh_enable+0x50/0x60 [39751.561456] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39751.566680] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39751.571638] [39751.573739] [39751.575842] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39751.581150] ? queued_spin_lock_slowpath+0x18/0x2e0 [39751.586026] do_raw_spin_lock+0xb5/0xc0 [39751.589857] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39751.594660] try_get_cap_refs+0x159/0x880 [ceph] [39751.599289] __ceph_get_caps+0x4a1/0x760 [ceph] [39751.603831] ? __file_remove_privs+0x89/0x110 [39751.608186] ? wake_bit_function+0x70/0x70 [39751.612278] ? ceph_write_iter+0x260/0x770 [ceph] [39751.616991] ceph_write_iter+0x260/0x770 [ceph] [39751.621527] ? lock_is_held_type+0xaa/0x120 [39751.625711] vfs_write+0x324/0x4f0 [39751.629111] ksys_pwrite64+0x65/0xa0 [39751.632688] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39751.637828] do_syscall_64+0x38/0x80 [39751.641399] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39751.646457] RIP: 0033:0x7f48f0eeb93b [39751.650030] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39751.668766] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [39751.676325] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39751.683460] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39751.690580] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39751.697706] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39751.704829] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39751.711959] [39751.714147] Sending NMI from CPU 0 to CPUs 7: [39751.718500] NMI backtrace for cpu 7 [39751.718501] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39751.718504] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39751.718505] Workqueue: events delayed_work [ceph] [39751.718525] RIP: 0010:__lock_acquire+0x359/0x1e20 [39751.718527] Code: 48 8d 04 80 41 0f b7 44 c4 20 66 25 ff 1f 0f b7 c0 48 0f a3 45 00 72 a8 8b 05 fb 6e 68 02 85 c0 0f 84 43 04 00 00 31 c0 eb a8 <48> 89 d5 49 89 cc 45 38 f7 0f 86 dc 00 00 00 44 89 5c 24 18 e8 de [39751.718529] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [39751.718531] RAX: ffffffff82fd06c0 RBX: ffff88810debaa80 RCX: 00000000000006f5 [39751.718532] RDX: ffff88810debb550 RSI: 0000000000000002 RDI: 0000000000000003 [39751.718534] RBP: ffffffff82f75120 R08: 0000000000000078 R09: 0000000000000000 [39751.718535] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810debb4d8 [39751.718536] R13: 0000000000000003 R14: 0000000000000002 R15: 0000000000000002 [39751.718537] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39751.718539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39751.718541] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39751.718542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39751.718543] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39751.718544] Call Trace: [39751.718546] [39751.718547] ? nmi_cpu_backtrace+0xad/0x120 [39751.718552] ? nmi_cpu_backtrace_handler+0x11/0x20 [39751.718554] ? nmi_handle+0xd4/0x260 [39751.718559] ? default_do_nmi+0x49/0xf0 [39751.718562] ? exc_nmi+0x11e/0x150 [39751.718564] ? end_repeat_nmi+0x16/0x67 [39751.718570] ? __lock_acquire+0x359/0x1e20 [39751.718573] ? __lock_acquire+0x359/0x1e20 [39751.718575] ? __lock_acquire+0x359/0x1e20 [39751.718578] [39751.718579] [39751.718582] lock_acquire+0xcb/0x2b0 [39751.718585] ? igrab+0x1d/0x50 [39751.718587] ? igrab+0x43/0x50 [39751.718589] ? lock_release+0xc5/0x260 [39751.718592] _raw_spin_lock+0x30/0x40 [39751.718595] ? igrab+0x1d/0x50 [39751.718596] igrab+0x1d/0x50 [39751.718598] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39751.718619] delayed_work+0x1d8/0x280 [ceph] [39751.718638] ? process_scheduled_works+0x22a/0x510 [39751.718641] process_scheduled_works+0x2a4/0x510 [39751.718647] worker_thread+0x188/0x340 [39751.718650] ? rescuer_thread+0x310/0x310 [39751.718653] kthread+0xf5/0x130 [39751.718656] ? kthread_complete_and_exit+0x20/0x20 [39751.718660] ret_from_fork+0x31/0x40 [39751.718663] ? kthread_complete_and_exit+0x20/0x20 [39751.718665] ret_from_fork_asm+0x11/0x20 [39751.718672] [39778.149501] watchdog: BUG: soft lockup - CPU#0 stuck for 25932s! [dbench:78683] [39778.156808] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39778.161501] watchdog: BUG: soft lockup - CPU#7 stuck for 34929s! [kworker/7:4:78808] [39778.212256] irq event stamp: 70684724 [39778.219988] Modules linked in: ceph libceph dns_resolver fscache netfs [39778.223644] hardirqs last enabled at (70684723): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39778.230184] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [39778.240346] hardirqs last disabled at (70684724): [] sysvec_apic_timer_interrupt+0xe/0x90 [39778.240349] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [39778.256153] softirqs last enabled at (70684670): [] __do_softirq+0x2be/0x371 [39778.263896] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [39778.272578] softirqs last disabled at (70684665): [] irq_exit_rcu+0xb0/0xd0 [39778.272582] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [39778.287616] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39778.294479] i2c_i801 iTCO_wdt [39778.303422] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39778.303424] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39778.306472] iTCO_vendor_support i2c_smbus [39778.313945] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39778.319423] crct10dif_pclmul joydev [39778.323513] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39778.342251] crc32_pclmul ipmi_ssif [39778.345820] [39778.351037] ghash_clmulni_intel mei_me [39778.354523] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39778.356013] mei lpc_ich [39778.359844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39778.366967] sha512_ssse3 acpi_ipmi [39778.369497] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39778.376623] mfd_core wmi [39778.380107] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39778.387230] ipmi_si [39778.389847] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39778.396974] ipmi_devintf ipmi_msghandler [39778.399156] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39778.406298] ioatdma [39778.410301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39778.418380] acpi_pad acpi_power_meter [39778.420562] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39778.426301] ixgbe [39778.430046] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39778.437177] igb mdio [39778.439190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39778.446313] crc32c_intel [39778.448583] Call Trace: [39778.448585] [39778.455708] nvme [39778.458328] ? watchdog_timer_fn+0x1b4/0x220 [39778.460769] ptp nvme_core [39778.462780] ? __hrtimer_run_queues+0x1d3/0x360 [39778.464704] i2c_algo_bit pps_core [39778.468971] ? hrtimer_interrupt+0x104/0x240 [39778.471673] dca fuse [39778.476204] ? __local_bh_enable+0x50/0x60 [39778.479603] [39778.479604] irq event stamp: 238446684 [39778.483868] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39778.486136] hardirqs last enabled at (238446683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39778.490228] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39778.491719] hardirqs last disabled at (238446684): [] sysvec_apic_timer_interrupt+0xe/0x90 [39778.495464] [39778.500679] softirqs last enabled at (238446682): [] __do_softirq+0x2be/0x371 [39778.510923] [39778.515897] softirqs last disabled at (238446677): [] irq_exit_rcu+0xb0/0xd0 [39778.525711] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39778.527806] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39778.536581] ? queued_spin_lock_slowpath+0x18/0x2e0 [39778.538675] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39778.547275] do_raw_spin_lock+0xb5/0xc0 [39778.552578] Workqueue: events delayed_work [ceph] [39778.561955] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39778.566825] [39778.566826] RIP: 0010:lock_acquire+0xf0/0x2b0 [39778.574298] try_get_cap_refs+0x159/0x880 [ceph] [39778.578126] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39778.582829] __ceph_get_caps+0x4a1/0x760 [ceph] [39778.587608] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39778.589103] ? __file_remove_privs+0x89/0x110 [39778.593449] [39778.593450] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39778.598060] ? wake_bit_function+0x70/0x70 [39778.616799] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39778.621326] ? ceph_write_iter+0x260/0x770 [ceph] [39778.626538] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39778.630902] ceph_write_iter+0x260/0x770 [ceph] [39778.632388] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39778.639517] ? lock_is_held_type+0xaa/0x120 [39778.643604] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39778.650730] vfs_write+0x324/0x4f0 [39778.655427] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39778.662556] ksys_pwrite64+0x65/0xa0 [39778.667074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39778.674199] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39778.678375] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39778.685501] do_syscall_64+0x38/0x80 [39778.688903] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39778.696984] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39778.700553] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39778.706293] RIP: 0033:0x7f48f0eeb93b [39778.711422] Call Trace: [39778.718547] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39778.722118] [39778.729242] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39778.734287] ? watchdog_timer_fn+0x1b4/0x220 [39778.741410] ORIG_RAX: 0000000000000012 [39778.744982] ? __hrtimer_run_queues+0x1d3/0x360 [39778.747423] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39778.766166] ? hrtimer_interrupt+0x104/0x240 [39778.768171] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39778.773406] ? __local_bh_enable+0x50/0x60 [39778.777671] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39778.781504] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39778.786026] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39778.793152] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39778.797415] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39778.804539] [39778.808635] [39778.815754] [39778.846573] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39778.851884] ? lock_acquire+0xf0/0x2b0 [39778.855649] ? igrab+0x43/0x50 [39778.858698] ? lock_release+0xc5/0x260 [39778.862443] _raw_spin_lock+0x30/0x40 [39778.866100] ? igrab+0x1d/0x50 [39778.869151] igrab+0x1d/0x50 [39778.872028] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39778.877188] delayed_work+0x1d8/0x280 [ceph] [39778.881469] ? process_scheduled_works+0x22a/0x510 [39778.886259] process_scheduled_works+0x2a4/0x510 [39778.890874] worker_thread+0x188/0x340 [39778.894624] ? rescuer_thread+0x310/0x310 [39778.898627] kthread+0xf5/0x130 [39778.901772] ? kthread_complete_and_exit+0x20/0x20 [39778.906557] ret_from_fork+0x31/0x40 [39778.910137] ? kthread_complete_and_exit+0x20/0x20 [39778.914921] ret_from_fork_asm+0x11/0x20 [39778.918852] [39806.149977] watchdog: BUG: soft lockup - CPU#0 stuck for 25958s! [dbench:78683] [39806.157276] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39806.161977] watchdog: BUG: soft lockup - CPU#7 stuck for 34955s! [kworker/7:4:78808] [39806.212718] irq event stamp: 70742852 [39806.212719] hardirqs last enabled at (70742851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39806.220450] Modules linked in: [39806.224107] hardirqs last disabled at (70742852): [] sysvec_apic_timer_interrupt+0xe/0x90 [39806.234263] ceph [39806.237313] softirqs last enabled at (70742794): [] __do_softirq+0x2be/0x371 [39806.247038] libceph [39806.248961] softirqs last disabled at (70742789): [] irq_exit_rcu+0xb0/0xd0 [39806.257662] dns_resolver [39806.259848] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39806.268373] fscache netfs [39806.270992] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39806.279938] veth nft_chain_nat [39806.282652] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [39806.290121] xt_MASQUERADE nf_nat [39806.293260] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [39806.298737] nf_conntrack nf_defrag_ipv6 [39806.302046] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39806.320784] nf_defrag_ipv4 nft_compat [39806.324702] [39806.324703] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39806.329918] nf_tables nfnetlink [39806.333663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39806.335154] bridge [39806.342277] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39806.345502] stp llc [39806.352626] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39806.354724] xfs libcrc32c [39806.361849] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39806.364033] sunrpc [39806.371155] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39806.373861] intel_rapl_msr [39806.380984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39806.383080] intel_rapl_common sb_edac [39806.391158] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39806.393950] x86_pkg_temp_thermal coretemp [39806.399686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39806.403431] kvm_intel kvm [39806.410554] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39806.414655] irqbypass i2c_i801 [39806.421779] Call Trace: [39806.424482] iTCO_wdt iTCO_vendor_support [39806.431606] [39806.434743] i2c_smbus crct10dif_pclmul [39806.437190] ? watchdog_timer_fn+0x1b4/0x220 [39806.441192] joydev crc32_pclmul [39806.443204] ? __hrtimer_run_queues+0x1d3/0x360 [39806.447034] ipmi_ssif ghash_clmulni_intel [39806.451301] ? hrtimer_interrupt+0x104/0x240 [39806.454522] mei_me mei [39806.459045] ? __local_bh_enable+0x50/0x60 [39806.463138] lpc_ich [39806.467402] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39806.469846] sha512_ssse3 acpi_ipmi [39806.473935] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39806.476120] mfd_core [39806.481337] [39806.484821] wmi ipmi_si [39806.489779] [39806.492048] ipmi_devintf ipmi_msghandler [39806.494147] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39806.496681] ioatdma acpi_pad [39806.498780] ? queued_spin_lock_slowpath+0x16/0x2e0 [39806.502780] acpi_power_meter ixgbe [39806.508087] do_raw_spin_lock+0xb5/0xc0 [39806.511047] igb mdio [39806.515920] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39806.519403] crc32c_intel nvme [39806.523236] try_get_cap_refs+0x159/0x880 [ceph] [39806.525502] ptp nvme_core [39806.530292] __ceph_get_caps+0x4a1/0x760 [ceph] [39806.533337] i2c_algo_bit pps_core [39806.537953] ? __file_remove_privs+0x89/0x110 [39806.540657] dca fuse [39806.545179] ? wake_bit_function+0x70/0x70 [39806.548575] [39806.548575] irq event stamp: 238667566 [39806.552929] ? ceph_write_iter+0x260/0x770 [ceph] [39806.555195] hardirqs last enabled at (238667565): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39806.559286] ceph_write_iter+0x260/0x770 [ceph] [39806.560776] hardirqs last disabled at (238667566): [] sysvec_apic_timer_interrupt+0xe/0x90 [39806.564525] ? lock_is_held_type+0xaa/0x120 [39806.569220] softirqs last enabled at (238667564): [] __do_softirq+0x2be/0x371 [39806.579467] vfs_write+0x324/0x4f0 [39806.583986] softirqs last disabled at (238667559): [] irq_exit_rcu+0xb0/0xd0 [39806.593802] ksys_pwrite64+0x65/0xa0 [39806.597977] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39806.606747] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39806.610143] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39806.618743] do_syscall_64+0x38/0x80 [39806.622311] Workqueue: events delayed_work [ceph] [39806.631690] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39806.636820] [39806.636820] RIP: 0010:lock_acquire+0xf0/0x2b0 [39806.644292] RIP: 0033:0x7f48f0eeb93b [39806.647861] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39806.652559] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39806.657602] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39806.657604] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39806.659094] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39806.663445] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39806.667016] ORIG_RAX: 0000000000000012 [39806.685754] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39806.704490] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39806.709708] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39806.716832] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39806.722051] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39806.729174] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39806.733005] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39806.740129] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39806.747253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39806.754380] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39806.761501] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39806.768632] [39806.775749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39806.820246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39806.827382] Call Trace: [39806.829831] [39806.831842] ? watchdog_timer_fn+0x1b4/0x220 [39806.836108] ? __hrtimer_run_queues+0x1d3/0x360 [39806.840634] ? hrtimer_interrupt+0x104/0x240 [39806.844901] ? __local_bh_enable+0x50/0x60 [39806.848996] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39806.854219] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39806.859178] [39806.861273] [39806.863383] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39806.868697] ? lock_acquire+0xf0/0x2b0 [39806.872448] ? igrab+0x43/0x50 [39806.875498] ? lock_release+0xc5/0x260 [39806.879243] _raw_spin_lock+0x30/0x40 [39806.882907] ? igrab+0x1d/0x50 [39806.885956] igrab+0x1d/0x50 [39806.888836] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39806.893986] delayed_work+0x1d8/0x280 [ceph] [39806.898267] ? process_scheduled_works+0x22a/0x510 [39806.903056] process_scheduled_works+0x2a4/0x510 [39806.907673] worker_thread+0x188/0x340 [39806.911420] ? rescuer_thread+0x310/0x310 [39806.915425] kthread+0xf5/0x130 [39806.918562] ? kthread_complete_and_exit+0x20/0x20 [39806.923346] ret_from_fork+0x31/0x40 [39806.926916] ? kthread_complete_and_exit+0x20/0x20 [39806.931701] ret_from_fork_asm+0x11/0x20 [39806.935624] [39834.150455] watchdog: BUG: soft lockup - CPU#0 stuck for 25984s! [dbench:78683] [39834.157757] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39834.162454] watchdog: BUG: soft lockup - CPU#7 stuck for 34981s! [kworker/7:4:78808] [39834.213206] irq event stamp: 70801036 [39834.220936] Modules linked in: ceph libceph dns_resolver fscache netfs veth [39834.224594] hardirqs last enabled at (70801035): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39834.231553] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [39834.241709] hardirqs last disabled at (70801036): [] sysvec_apic_timer_interrupt+0xe/0x90 [39834.241713] nft_compat nf_tables nfnetlink bridge stp llc xfs [39834.259685] softirqs last enabled at (70800990): [] __do_softirq+0x2be/0x371 [39834.265509] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [39834.274194] softirqs last disabled at (70800985): [] irq_exit_rcu+0xb0/0xd0 [39834.282539] coretemp kvm_intel kvm irqbypass i2c_i801 [39834.291051] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39834.296180] iTCO_wdt [39834.305132] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39834.305134] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39834.307404] iTCO_vendor_support [39834.314876] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39834.320352] i2c_smbus crct10dif_pclmul [39834.323576] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39834.342316] joydev crc32_pclmul [39834.346144] [39834.346145] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39834.351364] ipmi_ssif ghash_clmulni_intel [39834.354586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39834.356079] mei_me mei [39834.363202] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39834.367293] lpc_ich sha512_ssse3 [39834.374416] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39834.376862] acpi_ipmi mfd_core [39834.383985] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39834.387296] wmi ipmi_si [39834.394420] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39834.397556] ipmi_devintf ipmi_msghandler [39834.404680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39834.407213] ioatdma acpi_pad [39834.415289] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39834.419293] acpi_power_meter [39834.425031] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39834.427994] ixgbe igb [39834.435131] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39834.438092] mdio crc32c_intel [39834.445217] Call Trace: [39834.445219] [39834.447573] nvme ptp [39834.454700] ? watchdog_timer_fn+0x1b4/0x220 [39834.457748] nvme_core [39834.460194] ? __hrtimer_run_queues+0x1d3/0x360 [39834.462203] i2c_algo_bit [39834.464477] ? hrtimer_interrupt+0x104/0x240 [39834.468739] pps_core [39834.471096] ? __local_bh_enable+0x50/0x60 [39834.475618] dca fuse [39834.478238] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39834.482501] [39834.482502] irq event stamp: 238888348 [39834.484773] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39834.488863] hardirqs last enabled at (238888347): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39834.491136] [39834.496349] hardirqs last disabled at (238888348): [] sysvec_apic_timer_interrupt+0xe/0x90 [39834.497855] [39834.497856] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39834.501603] softirqs last enabled at (238888346): [] __do_softirq+0x2be/0x371 [39834.506568] ? queued_spin_lock_slowpath+0x18/0x2e0 [39834.516803] softirqs last disabled at (238888341): [] irq_exit_rcu+0xb0/0xd0 [39834.518905] do_raw_spin_lock+0xb5/0xc0 [39834.528711] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39834.530812] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39834.536112] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39834.544887] try_get_cap_refs+0x159/0x880 [ceph] [39834.549756] Workqueue: events delayed_work [ceph] [39834.558358] __ceph_get_caps+0x4a1/0x760 [ceph] [39834.562182] [39834.562183] RIP: 0010:lock_acquire+0xf0/0x2b0 [39834.571566] ? __file_remove_privs+0x89/0x110 [39834.576345] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39834.583817] ? wake_bit_function+0x70/0x70 [39834.588426] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39834.593137] ? ceph_write_iter+0x260/0x770 [ceph] [39834.597657] [39834.597658] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39834.599147] ceph_write_iter+0x260/0x770 [ceph] [39834.603497] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39834.607862] ? lock_is_held_type+0xaa/0x120 [39834.626595] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39834.630689] vfs_write+0x324/0x4f0 [39834.635904] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39834.640606] ksys_pwrite64+0x65/0xa0 [39834.642092] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39834.649218] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39834.653739] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39834.660864] do_syscall_64+0x38/0x80 [39834.665042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39834.672166] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39834.675562] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39834.682689] RIP: 0033:0x7f48f0eeb93b [39834.686257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39834.693383] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39834.698513] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39834.706591] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39834.710161] Call Trace: [39834.710163] [39834.715899] ORIG_RAX: 0000000000000012 [39834.720942] ? watchdog_timer_fn+0x1b4/0x220 [39834.728065] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39834.731639] ? __hrtimer_run_queues+0x1d3/0x360 [39834.738762] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39834.757503] ? hrtimer_interrupt+0x104/0x240 [39834.764622] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39834.769857] ? __local_bh_enable+0x50/0x60 [39834.772302] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39834.774316] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39834.778144] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39834.782409] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39834.789538] [39834.794056] [39834.845381] [39834.847480] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39834.852787] ? lock_acquire+0xf0/0x2b0 [39834.856537] ? igrab+0x43/0x50 [39834.859586] ? lock_release+0xc5/0x260 [39834.863332] _raw_spin_lock+0x30/0x40 [39834.866997] ? igrab+0x1d/0x50 [39834.870047] igrab+0x1d/0x50 [39834.872927] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39834.878075] delayed_work+0x1d8/0x280 [ceph] [39834.882357] ? process_scheduled_works+0x22a/0x510 [39834.887162] process_scheduled_works+0x2a4/0x510 [39834.891780] worker_thread+0x188/0x340 [39834.895528] ? rescuer_thread+0x310/0x310 [39834.899530] kthread+0xf5/0x130 [39834.902671] ? kthread_complete_and_exit+0x20/0x20 [39834.907462] ret_from_fork+0x31/0x40 [39834.911043] ? kthread_complete_and_exit+0x20/0x20 [39834.915825] ret_from_fork_asm+0x11/0x20 [39834.919749] [39862.150935] watchdog: BUG: soft lockup - CPU#0 stuck for 26010s! [dbench:78683] [39862.158234] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39862.162935] watchdog: BUG: soft lockup - CPU#7 stuck for 35007s! [kworker/7:4:78808] [39862.213675] irq event stamp: 70859154 [39862.213677] hardirqs last enabled at (70859153): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39862.221407] Modules linked in: ceph [39862.225063] hardirqs last disabled at (70859154): [] sysvec_apic_timer_interrupt+0xe/0x90 [39862.235222] libceph [39862.238705] softirqs last enabled at (70859134): [] __do_softirq+0x2be/0x371 [39862.248429] dns_resolver fscache [39862.250613] softirqs last disabled at (70859129): [] irq_exit_rcu+0xb0/0xd0 [39862.259298] netfs veth [39862.262624] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39862.271137] nft_chain_nat [39862.273579] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39862.282524] xt_MASQUERADE nf_nat [39862.285228] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [39862.292700] nf_conntrack [39862.296011] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [39862.301488] nf_defrag_ipv6 nf_defrag_ipv4 [39862.304106] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39862.322842] nft_compat nf_tables [39862.326933] [39862.326934] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39862.332151] nfnetlink [39862.335460] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39862.336953] bridge stp [39862.344077] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39862.346433] llc xfs [39862.353557] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39862.356001] libcrc32c [39862.363125] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39862.365309] sunrpc intel_rapl_msr [39862.372452] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39862.374808] intel_rapl_common [39862.381932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39862.385332] sb_edac [39862.393407] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39862.396458] x86_pkg_temp_thermal coretemp [39862.402197] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39862.404379] kvm_intel kvm [39862.411504] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39862.415594] irqbypass [39862.422737] Call Trace: [39862.422738] [39862.425439] i2c_i801 iTCO_wdt [39862.432567] ? watchdog_timer_fn+0x1b4/0x220 [39862.434922] iTCO_vendor_support [39862.437368] ? __hrtimer_run_queues+0x1d3/0x360 [39862.439375] i2c_smbus crct10dif_pclmul [39862.442432] ? hrtimer_interrupt+0x104/0x240 [39862.446693] joydev crc32_pclmul [39862.449915] ? __local_bh_enable+0x50/0x60 [39862.454439] ipmi_ssif ghash_clmulni_intel [39862.458272] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39862.462535] mei_me mei [39862.465760] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39862.469850] lpc_ich sha512_ssse3 [39862.473940] [39862.479156] acpi_ipmi mfd_core [39862.481613] [39862.486568] wmi ipmi_si [39862.489879] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39862.491976] ipmi_devintf ipmi_msghandler [39862.495118] ? queued_spin_lock_slowpath+0x18/0x2e0 [39862.497210] ioatdma acpi_pad [39862.499744] do_raw_spin_lock+0xb5/0xc0 [39862.505046] acpi_power_meter ixgbe [39862.509051] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39862.513919] igb mdio [39862.516886] try_get_cap_refs+0x159/0x880 [ceph] [39862.520715] crc32c_intel nvme [39862.524203] __ceph_get_caps+0x4a1/0x760 [ceph] [39862.528984] ptp nvme_core [39862.531258] ? __file_remove_privs+0x89/0x110 [39862.535865] i2c_algo_bit pps_core [39862.538915] ? wake_bit_function+0x70/0x70 [39862.543440] dca fuse [39862.546145] ? ceph_write_iter+0x260/0x770 [ceph] [39862.550494] [39862.550495] irq event stamp: 239109262 [39862.553892] ceph_write_iter+0x260/0x770 [ceph] [39862.557983] hardirqs last enabled at (239109261): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39862.560256] ? lock_is_held_type+0xaa/0x120 [39862.564950] hardirqs last disabled at (239109262): [] sysvec_apic_timer_interrupt+0xe/0x90 [39862.566443] vfs_write+0x324/0x4f0 [39862.570185] softirqs last enabled at (239109260): [] __do_softirq+0x2be/0x371 [39862.574713] ksys_pwrite64+0x65/0xa0 [39862.584954] softirqs last disabled at (239109255): [] irq_exit_rcu+0xb0/0xd0 [39862.589132] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39862.598943] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39862.602342] do_syscall_64+0x38/0x80 [39862.611109] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39862.611110] Workqueue: events delayed_work [ceph] [39862.614682] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39862.623277] [39862.623278] RIP: 0010:_raw_spin_lock+0x18/0x40 [39862.628410] RIP: 0033:0x7f48f0eeb93b [39862.637787] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [39862.641357] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39862.648828] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [39862.653525] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39862.658570] [39862.658571] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [39862.660061] ORIG_RAX: 0000000000000012 [39862.664497] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [39862.668069] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39862.686805] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [39862.705545] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39862.710761] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [39862.715978] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39862.717468] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [39862.724593] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39862.728441] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39862.735565] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39862.742689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39862.749820] [39862.756937] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39862.815681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39862.822805] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39862.829929] Call Trace: [39862.832375] [39862.834387] ? watchdog_timer_fn+0x1b4/0x220 [39862.838649] ? __hrtimer_run_queues+0x1d3/0x360 [39862.843175] ? hrtimer_interrupt+0x104/0x240 [39862.847437] ? __local_bh_enable+0x50/0x60 [39862.851531] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39862.856754] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39862.861712] [39862.863808] [39862.865906] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39862.871214] ? _raw_spin_lock+0x18/0x40 [39862.875050] ? igrab+0x1d/0x50 [39862.878102] igrab+0x1d/0x50 [39862.880979] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39862.886129] delayed_work+0x1d8/0x280 [ceph] [39862.890410] ? process_scheduled_works+0x22a/0x510 [39862.895199] process_scheduled_works+0x2a4/0x510 [39862.899816] worker_thread+0x188/0x340 [39862.903565] ? rescuer_thread+0x310/0x310 [39862.907567] kthread+0xf5/0x130 [39862.910705] ? kthread_complete_and_exit+0x20/0x20 [39862.915491] ret_from_fork+0x31/0x40 [39862.919060] ? kthread_complete_and_exit+0x20/0x20 [39862.923845] ret_from_fork_asm+0x11/0x20 [39862.927767] [39890.151417] watchdog: BUG: soft lockup - CPU#0 stuck for 26036s! [dbench:78683] [39890.158723] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39890.163416] watchdog: BUG: soft lockup - CPU#7 stuck for 35033s! [kworker/7:4:78808] [39890.214172] irq event stamp: 70917404 [39890.214173] hardirqs last enabled at (70917403): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39890.221901] Modules linked in: [39890.225560] hardirqs last disabled at (70917404): [] sysvec_apic_timer_interrupt+0xe/0x90 [39890.235717] ceph [39890.238768] softirqs last enabled at (70917346): [] __do_softirq+0x2be/0x371 [39890.248492] libceph [39890.250415] softirqs last disabled at (70917341): [] irq_exit_rcu+0xb0/0xd0 [39890.259101] dns_resolver [39890.261284] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39890.269811] fscache [39890.272429] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39890.281374] netfs [39890.283559] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39890.291028] veth nft_chain_nat [39890.293040] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39890.298518] xt_MASQUERADE nf_nat [39890.301655] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39890.320391] nf_conntrack nf_defrag_ipv6 [39890.323704] [39890.323704] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39890.328919] nf_defrag_ipv4 [39890.332836] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39890.334328] nft_compat nf_tables [39890.341452] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39890.344242] nfnetlink bridge [39890.351367] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39890.354678] stp llc [39890.361812] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39890.364774] xfs [39890.371900] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39890.374093] libcrc32c sunrpc [39890.381215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39890.383053] intel_rapl_msr [39890.391131] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39890.394101] intel_rapl_common [39890.399841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39890.402632] sb_edac x86_pkg_temp_thermal [39890.409756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39890.412814] coretemp [39890.419939] Call Trace: [39890.419941] [39890.423942] kvm_intel kvm [39890.431069] ? watchdog_timer_fn+0x1b4/0x220 [39890.433338] irqbypass i2c_i801 [39890.435783] ? __hrtimer_run_queues+0x1d3/0x360 [39890.437810] iTCO_wdt iTCO_vendor_support [39890.440516] ? hrtimer_interrupt+0x104/0x240 [39890.444779] i2c_smbus crct10dif_pclmul [39890.447914] ? __local_bh_enable+0x50/0x60 [39890.452438] joydev [39890.456446] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39890.460708] crc32_pclmul ipmi_ssif [39890.464538] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39890.468630] ghash_clmulni_intel mei_me [39890.470727] [39890.475944] mei lpc_ich [39890.479429] [39890.479430] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39890.484385] sha512_ssse3 acpi_ipmi [39890.488221] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39890.490314] mfd_core [39890.492847] do_raw_spin_lock+0xb5/0xc0 [39890.494941] wmi ipmi_si [39890.500248] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39890.503731] ipmi_devintf ipmi_msghandler [39890.508604] try_get_cap_refs+0x159/0x880 [ceph] [39890.510872] ioatdma acpi_pad [39890.514707] __ceph_get_caps+0x4a1/0x760 [ceph] [39890.517233] acpi_power_meter ixgbe [39890.522021] ? __file_remove_privs+0x89/0x110 [39890.526021] igb mdio [39890.530633] ? wake_bit_function+0x70/0x70 [39890.533595] crc32c_intel nvme [39890.538122] ? ceph_write_iter+0x260/0x770 [ceph] [39890.541604] ptp nvme_core [39890.545955] ceph_write_iter+0x260/0x770 [ceph] [39890.548225] i2c_algo_bit pps_core [39890.552320] ? lock_is_held_type+0xaa/0x120 [39890.555366] dca fuse [39890.560066] vfs_write+0x324/0x4f0 [39890.562767] [39890.562768] irq event stamp: 239330044 [39890.567297] ksys_pwrite64+0x65/0xa0 [39890.570689] hardirqs last enabled at (239330043): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39890.574869] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39890.577138] hardirqs last disabled at (239330044): [] sysvec_apic_timer_interrupt+0xe/0x90 [39890.580536] do_syscall_64+0x38/0x80 [39890.582026] softirqs last enabled at (239330042): [] __do_softirq+0x2be/0x371 [39890.585770] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39890.589341] softirqs last disabled at (239330037): [] irq_exit_rcu+0xb0/0xd0 [39890.599586] RIP: 0033:0x7f48f0eeb93b [39890.604716] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39890.614529] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39890.618098] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39890.626885] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39890.631932] Workqueue: events delayed_work [ceph] [39890.640527] ORIG_RAX: 0000000000000012 [39890.644098] [39890.644099] RIP: 0010:lock_acquire+0xf0/0x2b0 [39890.653476] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39890.672212] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39890.679683] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39890.684902] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39890.689597] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39890.693431] [39890.693431] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39890.694921] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39890.699270] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39890.706395] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39890.725132] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39890.732262] [39890.737473] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39890.791036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39890.798158] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39890.806238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39890.811975] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39890.819102] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39890.826221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39890.833348] Call Trace: [39890.835791] [39890.837820] ? watchdog_timer_fn+0x1b4/0x220 [39890.842083] ? __hrtimer_run_queues+0x1d3/0x360 [39890.846629] ? hrtimer_interrupt+0x104/0x240 [39890.850898] ? __local_bh_enable+0x50/0x60 [39890.854990] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39890.860214] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39890.865171] [39890.867271] [39890.869367] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39890.874676] ? lock_acquire+0xf0/0x2b0 [39890.878424] ? igrab+0x43/0x50 [39890.881475] ? lock_release+0xc5/0x260 [39890.885219] _raw_spin_lock+0x30/0x40 [39890.888885] ? igrab+0x1d/0x50 [39890.891934] igrab+0x1d/0x50 [39890.894825] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39890.899973] delayed_work+0x1d8/0x280 [ceph] [39890.904254] ? process_scheduled_works+0x22a/0x510 [39890.909043] process_scheduled_works+0x2a4/0x510 [39890.913658] worker_thread+0x188/0x340 [39890.917408] ? rescuer_thread+0x310/0x310 [39890.921410] kthread+0xf5/0x130 [39890.924550] ? kthread_complete_and_exit+0x20/0x20 [39890.929343] ret_from_fork+0x31/0x40 [39890.932921] ? kthread_complete_and_exit+0x20/0x20 [39890.937704] ret_from_fork_asm+0x11/0x20 [39890.941628] [39918.151897] watchdog: BUG: soft lockup - CPU#0 stuck for 26062s! [dbench:78683] [39918.159200] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39918.163897] watchdog: BUG: soft lockup - CPU#7 stuck for 35059s! [kworker/7:4:78808] [39918.214642] irq event stamp: 70975648 [39918.214643] hardirqs last enabled at (70975647): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39918.222372] Modules linked in: [39918.226029] hardirqs last disabled at (70975648): [] sysvec_apic_timer_interrupt+0xe/0x90 [39918.236188] ceph [39918.239238] softirqs last enabled at (70975590): [] __do_softirq+0x2be/0x371 [39918.248961] libceph dns_resolver [39918.250886] softirqs last disabled at (70975585): [] irq_exit_rcu+0xb0/0xd0 [39918.259571] fscache netfs [39918.262882] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39918.271393] veth nft_chain_nat [39918.274095] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39918.283039] xt_MASQUERADE nf_nat [39918.286178] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39918.293648] nf_conntrack [39918.296959] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39918.302436] nf_defrag_ipv6 nf_defrag_ipv4 [39918.305054] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39918.323792] nft_compat nf_tables [39918.327883] [39918.327884] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39918.333099] nfnetlink bridge [39918.336410] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39918.337901] stp llc [39918.345026] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39918.347990] xfs libcrc32c [39918.355112] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39918.357297] sunrpc intel_rapl_msr [39918.364421] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39918.367126] intel_rapl_common [39918.374250] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39918.377648] sb_edac x86_pkg_temp_thermal [39918.384771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39918.387821] coretemp kvm_intel [39918.395899] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39918.399904] kvm irqbypass [39918.405640] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39918.408778] i2c_i801 iTCO_wdt [39918.415903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39918.418605] iTCO_vendor_support i2c_smbus [39918.425730] Call Trace: [39918.425731] [39918.428781] crct10dif_pclmul joydev [39918.435907] ? watchdog_timer_fn+0x1b4/0x220 [39918.439995] crc32_pclmul [39918.442442] ? __hrtimer_run_queues+0x1d3/0x360 [39918.444450] ipmi_ssif ghash_clmulni_intel [39918.448026] ? hrtimer_interrupt+0x104/0x240 [39918.452295] mei_me mei [39918.454911] ? __local_bh_enable+0x50/0x60 [39918.459436] lpc_ich sha512_ssse3 [39918.463529] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39918.467791] acpi_ipmi mfd_core [39918.470235] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39918.474324] wmi ipmi_si [39918.477637] [39918.482854] ipmi_devintf ipmi_msghandler [39918.485992] [39918.485993] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39918.490947] ioatdma acpi_pad [39918.493484] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39918.495577] acpi_power_meter ixgbe [39918.499582] do_raw_spin_lock+0xb5/0xc0 [39918.501679] igb mdio [39918.506983] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39918.509946] crc32c_intel [39918.514820] try_get_cap_refs+0x159/0x880 [ceph] [39918.518301] nvme ptp [39918.522137] __ceph_get_caps+0x4a1/0x760 [ceph] [39918.524402] nvme_core i2c_algo_bit [39918.529191] ? __file_remove_privs+0x89/0x110 [39918.531804] pps_core [39918.536417] ? wake_bit_function+0x70/0x70 [39918.538684] dca fuse [39918.543213] ? ceph_write_iter+0x260/0x770 [ceph] [39918.546693] [39918.546694] irq event stamp: 239550912 [39918.551044] ceph_write_iter+0x260/0x770 [ceph] [39918.553316] hardirqs last enabled at (239550911): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39918.557410] ? lock_is_held_type+0xaa/0x120 [39918.559677] hardirqs last disabled at (239550912): [] sysvec_apic_timer_interrupt+0xe/0x90 [39918.564378] vfs_write+0x324/0x4f0 [39918.565864] softirqs last enabled at (239550910): [] __do_softirq+0x2be/0x371 [39918.569614] ksys_pwrite64+0x65/0xa0 [39918.574133] softirqs last disabled at (239550905): [] irq_exit_rcu+0xb0/0xd0 [39918.584378] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39918.588555] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39918.598366] do_syscall_64+0x38/0x80 [39918.601764] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39918.610534] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39918.614105] Workqueue: events delayed_work [ceph] [39918.622703] RIP: 0033:0x7f48f0eeb93b [39918.627833] [39918.627834] RIP: 0010:lock_acquire+0xf0/0x2b0 [39918.637211] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39918.640780] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39918.648251] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39918.653298] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39918.657993] ORIG_RAX: 0000000000000012 [39918.661564] [39918.661565] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39918.663073] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39918.667421] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39918.686159] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39918.704899] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39918.710115] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39918.715332] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39918.719164] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39918.720653] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39918.727777] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39918.734903] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39918.742032] [39918.749152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39918.807911] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39918.815036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39918.822160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39918.829302] Call Trace: [39918.831747] [39918.833756] ? watchdog_timer_fn+0x1b4/0x220 [39918.838021] ? __hrtimer_run_queues+0x1d3/0x360 [39918.842547] ? hrtimer_interrupt+0x104/0x240 [39918.846809] ? __local_bh_enable+0x50/0x60 [39918.850903] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39918.856126] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39918.861084] [39918.863179] [39918.865278] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39918.870602] ? lock_acquire+0xf0/0x2b0 [39918.874353] ? igrab+0x43/0x50 [39918.877404] ? lock_release+0xc5/0x260 [39918.881149] _raw_spin_lock+0x30/0x40 [39918.884812] ? igrab+0x1d/0x50 [39918.887865] igrab+0x1d/0x50 [39918.890743] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39918.895892] delayed_work+0x1d8/0x280 [ceph] [39918.900173] ? process_scheduled_works+0x22a/0x510 [39918.904964] process_scheduled_works+0x2a4/0x510 [39918.909587] worker_thread+0x188/0x340 [39918.913337] ? rescuer_thread+0x310/0x310 [39918.917340] kthread+0xf5/0x130 [39918.920478] ? kthread_complete_and_exit+0x20/0x20 [39918.925261] ret_from_fork+0x31/0x40 [39918.928832] ? kthread_complete_and_exit+0x20/0x20 [39918.933616] ret_from_fork_asm+0x11/0x20 [39918.937539] [39946.152378] watchdog: BUG: soft lockup - CPU#0 stuck for 26088s! [dbench:78683] [39946.159681] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39946.164377] watchdog: BUG: soft lockup - CPU#7 stuck for 35085s! [kworker/7:4:78808] [39946.215129] irq event stamp: 71033944 [39946.222860] Modules linked in: ceph libceph dns_resolver fscache netfs [39946.226517] hardirqs last enabled at (71033943): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39946.226522] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [39946.243208] hardirqs last disabled at (71033944): [] sysvec_apic_timer_interrupt+0xe/0x90 [39946.251892] nft_compat nf_tables nfnetlink bridge stp llc [39946.261618] softirqs last enabled at (71033886): [] __do_softirq+0x2be/0x371 [39946.261621] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [39946.275778] softirqs last disabled at (71033881): [] irq_exit_rcu+0xb0/0xd0 [39946.285243] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [39946.293772] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39946.298901] iTCO_vendor_support i2c_smbus [39946.307846] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39946.311938] crct10dif_pclmul joydev crc32_pclmul ipmi_ssif [39946.319409] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39946.319412] ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 [39946.330451] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39946.336448] acpi_ipmi mfd_core [39946.355185] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39946.358322] wmi ipmi_si [39946.363539] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39946.366071] ipmi_devintf ipmi_msghandler [39946.373196] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39946.373197] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39946.377199] ioatdma acpi_pad [39946.384322] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39946.391446] acpi_power_meter [39946.394411] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39946.401536] ixgbe igb [39946.404499] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39946.411623] mdio crc32c_intel [39946.413981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39946.422062] nvme ptp [39946.425109] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39946.430847] nvme_core i2c_algo_bit [39946.433118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39946.440240] pps_core dca [39946.443725] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39946.450850] fuse [39946.453467] Call Trace: [39946.453470] [39946.460590] irq event stamp: 239771474 [39946.462518] ? watchdog_timer_fn+0x1b4/0x220 [39946.464958] hardirqs last enabled at (239771473): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39946.466971] ? __hrtimer_run_queues+0x1d3/0x360 [39946.470712] hardirqs last disabled at (239771474): [] sysvec_apic_timer_interrupt+0xe/0x90 [39946.474981] ? hrtimer_interrupt+0x104/0x240 [39946.485221] softirqs last enabled at (239771472): [] __do_softirq+0x2be/0x371 [39946.489747] ? __local_bh_enable+0x50/0x60 [39946.499575] softirqs last disabled at (239771467): [] irq_exit_rcu+0xb0/0xd0 [39946.503842] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39946.512610] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39946.516700] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39946.525297] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39946.530516] [39946.539891] Workqueue: events delayed_work [ceph] [39946.544850] [39946.544852] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39946.552319] [39946.554423] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39946.559115] RIP: 0010:lock_acquire+0xf0/0x2b0 [39946.561214] do_raw_spin_lock+0xb5/0xc0 [39946.566516] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39946.568010] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39946.572878] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39946.577233] try_get_cap_refs+0x159/0x880 [ceph] [39946.581060] [39946.581060] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39946.599803] __ceph_get_caps+0x4a1/0x760 [ceph] [39946.604581] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39946.609803] ? __file_remove_privs+0x89/0x110 [39946.614410] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39946.615900] ? wake_bit_function+0x70/0x70 [39946.623024] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39946.627552] ? ceph_write_iter+0x260/0x770 [ceph] [39946.634673] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39946.639024] ceph_write_iter+0x260/0x770 [ceph] [39946.646148] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39946.650242] ? lock_is_held_type+0xaa/0x120 [39946.657362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39946.662062] vfs_write+0x324/0x4f0 [39946.669183] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39946.669185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39946.673713] ksys_pwrite64+0x65/0xa0 [39946.681785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39946.685963] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39946.691700] Call Trace: [39946.691701] [39946.695098] do_syscall_64+0x38/0x80 [39946.702222] ? watchdog_timer_fn+0x1b4/0x220 [39946.709347] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39946.712917] ? __hrtimer_run_queues+0x1d3/0x360 [39946.720043] RIP: 0033:0x7f48f0eeb93b [39946.725175] ? hrtimer_interrupt+0x104/0x240 [39946.727616] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39946.728753] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [39946.728756] rcu: 0-....: (27376263 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=13712046 [39946.728779] rcu: 7-....: (27194391 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/27193021 fqs=13712046 [39946.728782] rcu: (detected by 4, t=28009614 jiffies, g=1309609, q=752864 ncpus=8) [39946.728785] Sending NMI from CPU 4 to CPUs 0: [39946.728788] NMI backtrace for cpu 0 [39946.728789] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39946.728791] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39946.728792] RIP: 0010:vprintk_emit+0x1eb/0x340 [39946.728795] Code: 50 41 b8 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 <0f> b6 11 84 d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 [39946.728797] RSP: 0018:ffffc90000003c40 EFLAGS: 00000002 [39946.728799] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [39946.728800] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [39946.728801] RBP: 00000000000000c7 R08: 0000000000005ef4 R09: 0000000000000004 [39946.728803] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [39946.728804] R13: 0000000000000000 R14: ffffffff82123e57 R15: ffffc90000003c90 [39946.728805] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39946.728807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39946.728808] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39946.728810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39946.728811] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39946.728812] Call Trace: [39946.728813] [39946.728814] ? nmi_cpu_backtrace+0xad/0x120 [39946.728819] ? nmi_cpu_backtrace_handler+0x11/0x20 [39946.728821] ? nmi_handle+0xd4/0x260 [39946.728827] ? default_do_nmi+0x49/0xf0 [39946.728829] ? exc_nmi+0x11e/0x150 [39946.728832] ? end_repeat_nmi+0x16/0x67 [39946.728838] ? vprintk_emit+0x1eb/0x340 [39946.728841] ? vprintk_emit+0x1eb/0x340 [39946.728844] ? vprintk_emit+0x1eb/0x340 [39946.728847] [39946.728847] [39946.728848] ? console_unlock+0x7b/0x120 [39946.728851] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [39946.728854] _printk+0x5c/0x80 [39946.728858] show_opcodes+0xb3/0xd0 [39946.728864] show_iret_regs+0x16/0x40 [39946.728867] __show_regs+0x28/0x2e0 [39946.728870] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [39946.728872] ? unwind_next_frame+0x379/0x830 [39946.728876] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [39946.728879] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [39946.728881] show_trace_log_lvl+0x2b5/0x3f0 [39946.728890] ? touch_softlockup_watchdog+0x20/0x20 [39946.728892] watchdog_timer_fn+0x1b4/0x220 [39946.728895] __hrtimer_run_queues+0x1d3/0x360 [39946.728900] hrtimer_interrupt+0x104/0x240 [39946.728902] ? __local_bh_enable+0x50/0x60 [39946.728907] __sysvec_apic_timer_interrupt+0x61/0x160 [39946.728910] sysvec_apic_timer_interrupt+0x6d/0x90 [39946.728913] [39946.728913] [39946.728914] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39946.728917] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39946.728920] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39946.728921] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39946.728923] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39946.728924] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39946.728925] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39946.728926] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39946.728927] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39946.728934] do_raw_spin_lock+0xb5/0xc0 [39946.728936] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39946.728956] try_get_cap_refs+0x159/0x880 [ceph] [39946.728977] __ceph_get_caps+0x4a1/0x760 [ceph] [39946.728997] ? __file_remove_privs+0x89/0x110 [39946.728999] ? wake_bit_function+0x70/0x70 [39946.729003] ? ceph_write_iter+0x260/0x770 [ceph] [39946.729019] ceph_write_iter+0x260/0x770 [ceph] [39946.729037] ? lock_is_held_type+0xaa/0x120 [39946.729041] vfs_write+0x324/0x4f0 [39946.729047] ksys_pwrite64+0x65/0xa0 [39946.729050] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39946.729054] do_syscall_64+0x38/0x80 [39946.729057] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39946.729059] RIP: 0033:0x7f48f0eeb93b [39946.729061] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39946.729062] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [39946.729064] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39946.729065] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39946.729066] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39946.729067] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39946.729069] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39946.729075] [39946.729627] ? __local_bh_enable+0x50/0x60 [39946.729787] Sending NMI from CPU 4 to CPUs 7: [39946.729790] NMI backtrace for cpu 7 [39946.729791] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39946.729792] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39946.729793] Workqueue: events delayed_work [ceph] [39946.729808] RIP: 0010:vprintk_emit+0x1ee/0x340 [39946.729810] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [39946.729811] RSP: 0018:ffffc90000264d30 EFLAGS: 00000002 [39946.729812] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [39946.729813] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb578 [39946.729813] RBP: 000000000000001e R08: 0000000000005ef4 R09: 0000000000000004 [39946.729814] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [39946.729815] R13: 0000000000000000 R14: ffffffff82123e9c R15: ffffc90000264d80 [39946.729816] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39946.729817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39946.729817] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39946.729818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39946.729819] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39946.729820] Call Trace: [39946.729820] [39946.729821] ? nmi_cpu_backtrace+0xad/0x120 [39946.729824] ? nmi_cpu_backtrace_handler+0x11/0x20 [39946.729825] ? nmi_handle+0xd4/0x260 [39946.729829] ? default_do_nmi+0x49/0xf0 [39946.729830] ? exc_nmi+0x11e/0x150 [39946.729832] ? end_repeat_nmi+0x16/0x67 [39946.729836] ? vprintk_emit+0x1ee/0x340 [39946.729838] ? vprintk_emit+0x1ee/0x340 [39946.729840] ? vprintk_emit+0x1ee/0x340 [39946.729841] [39946.729842] [39946.729843] ? __local_bh_enable+0x50/0x60 [39946.729845] ? __local_bh_enable+0x50/0x60 [39946.729847] _printk+0x5c/0x80 [39946.729849] ? __local_bh_enable+0x50/0x60 [39946.729851] show_trace_log_lvl+0x1ac/0x3f0 [39946.729856] ? _raw_spin_lock+0x30/0x40 [39946.729859] ? touch_softlockup_watchdog+0x20/0x20 [39946.729860] watchdog_timer_fn+0x1b4/0x220 [39946.729862] __hrtimer_run_queues+0x1d3/0x360 [39946.729865] hrtimer_interrupt+0x104/0x240 [39946.729866] ? __local_bh_enable+0x50/0x60 [39946.729869] __sysvec_apic_timer_interrupt+0x61/0x160 [39946.729871] sysvec_apic_timer_interrupt+0x6d/0x90 [39946.729873] [39946.729874] [39946.729874] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39946.729876] RIP: 0010:lock_acquire+0xf0/0x2b0 [39946.729878] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39946.729879] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39946.729880] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39946.729880] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39946.729881] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39946.729881] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39946.729882] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39946.729885] ? igrab+0x43/0x50 [39946.729887] ? lock_release+0xc5/0x260 [39946.729889] _raw_spin_lock+0x30/0x40 [39946.729891] ? igrab+0x1d/0x50 [39946.729892] igrab+0x1d/0x50 [39946.729893] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39946.729908] delayed_work+0x1d8/0x280 [ceph] [39946.729922] ? process_scheduled_works+0x22a/0x510 [39946.729926] process_scheduled_works+0x2a4/0x510 [39946.729930] worker_thread+0x188/0x340 [39946.729932] ? rescuer_thread+0x310/0x310 [39946.729935] kthread+0xf5/0x130 [39946.729937] ? kthread_complete_and_exit+0x20/0x20 [39946.729939] ret_from_fork+0x31/0x40 [39946.729942] ? kthread_complete_and_exit+0x20/0x20 [39946.729944] ret_from_fork_asm+0x11/0x20 [39946.729948] [39946.733198] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39946.737465] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39946.742505] ORIG_RAX: 0000000000000012 [39946.747030] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39946.750599] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39946.754864] [39946.773603] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39946.779686] [39946.790190] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39946.800869] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39946.808426] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39946.812786] ? lock_acquire+0xf0/0x2b0 [39946.816268] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39946.825215] ? igrab+0x43/0x50 [39946.832690] [39946.837122] ? lock_release+0xc5/0x260 [39947.702662] _raw_spin_lock+0x30/0x40 [39947.706327] ? igrab+0x1d/0x50 [39947.709376] igrab+0x1d/0x50 [39947.712255] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39947.717404] delayed_work+0x1d8/0x280 [ceph] [39947.721685] ? process_scheduled_works+0x22a/0x510 [39947.726467] process_scheduled_works+0x2a4/0x510 [39947.731084] worker_thread+0x188/0x340 [39947.734831] ? rescuer_thread+0x310/0x310 [39947.738835] kthread+0xf5/0x130 [39947.741972] ? kthread_complete_and_exit+0x20/0x20 [39947.746758] ret_from_fork+0x31/0x40 [39947.750336] ? kthread_complete_and_exit+0x20/0x20 [39947.755122] ret_from_fork_asm+0x11/0x20 [39947.759045] [39974.152856] watchdog: BUG: soft lockup - CPU#0 stuck for 26114s! [dbench:78683] [39974.160159] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [39974.164856] watchdog: BUG: soft lockup - CPU#7 stuck for 35111s! [kworker/7:4:78808] [39974.215599] irq event stamp: 71091918 [39974.215600] hardirqs last enabled at (71091917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39974.223330] Modules linked in: [39974.226986] hardirqs last disabled at (71091918): [] sysvec_apic_timer_interrupt+0xe/0x90 [39974.237146] ceph [39974.240196] softirqs last enabled at (71091804): [] __do_softirq+0x2be/0x371 [39974.249920] libceph [39974.251845] softirqs last disabled at (71091799): [] irq_exit_rcu+0xb0/0xd0 [39974.260531] dns_resolver [39974.262711] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39974.271223] fscache [39974.273857] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39974.282802] netfs veth [39974.284986] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [39974.292456] nft_chain_nat [39974.294901] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [39974.300377] xt_MASQUERADE nf_nat [39974.303081] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [39974.321819] nf_conntrack [39974.325130] [39974.325131] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [39974.330347] nf_defrag_ipv6 [39974.332965] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [39974.334456] nf_defrag_ipv4 nft_compat [39974.341580] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [39974.344371] nf_tables [39974.351496] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [39974.355254] nfnetlink [39974.362379] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [39974.364737] bridge stp [39974.371863] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [39974.374220] llc xfs [39974.381344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39974.383788] libcrc32c sunrpc [39974.391864] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [39974.394050] intel_rapl_msr intel_rapl_common [39974.399786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39974.402751] sb_edac [39974.409876] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39974.414226] x86_pkg_temp_thermal [39974.421367] Call Trace: [39974.421369] [39974.423551] coretemp [39974.430675] ? watchdog_timer_fn+0x1b4/0x220 [39974.433987] kvm_intel kvm [39974.436432] ? __hrtimer_run_queues+0x1d3/0x360 [39974.438440] irqbypass i2c_i801 [39974.440714] ? hrtimer_interrupt+0x104/0x240 [39974.444974] iTCO_wdt [39974.447678] ? __local_bh_enable+0x50/0x60 [39974.452204] iTCO_vendor_support i2c_smbus [39974.455344] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39974.459604] crct10dif_pclmul joydev [39974.461876] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39974.465967] crc32_pclmul ipmi_ssif [39974.470056] [39974.475273] ghash_clmulni_intel [39974.478846] [39974.478848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39974.483804] mei_me mei [39974.487290] ? queued_spin_lock_slowpath+0x2d/0x2e0 [39974.489383] lpc_ich [39974.492612] do_raw_spin_lock+0xb5/0xc0 [39974.494706] sha512_ssse3 acpi_ipmi [39974.500010] __cap_delay_requeue+0xa7/0x1d0 [ceph] [39974.502453] mfd_core wmi [39974.507327] try_get_cap_refs+0x159/0x880 [ceph] [39974.509509] ipmi_si ipmi_devintf ipmi_msghandler [39974.513345] __ceph_get_caps+0x4a1/0x760 [ceph] [39974.516824] ioatdma acpi_pad [39974.521610] ? __file_remove_privs+0x89/0x110 [39974.524224] acpi_power_meter [39974.528836] ? wake_bit_function+0x70/0x70 [39974.533542] ixgbe igb [39974.538069] ? ceph_write_iter+0x260/0x770 [ceph] [39974.541030] mdio crc32c_intel [39974.545380] ceph_write_iter+0x260/0x770 [ceph] [39974.548343] nvme [39974.552438] ? lock_is_held_type+0xaa/0x120 [39974.554791] ptp nvme_core [39974.559494] vfs_write+0x324/0x4f0 [39974.562540] i2c_algo_bit [39974.567069] ksys_pwrite64+0x65/0xa0 [39974.568990] pps_core dca [39974.573167] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [39974.575871] fuse [39974.579270] do_syscall_64+0x38/0x80 [39974.581886] [39974.581887] irq event stamp: 239985628 [39974.585456] entry_SYSCALL_64_after_hwframe+0x63/0xcd [39974.588072] hardirqs last enabled at (239985627): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39974.593205] RIP: 0033:0x7f48f0eeb93b [39974.595128] hardirqs last disabled at (239985628): [] sysvec_apic_timer_interrupt+0xe/0x90 [39974.598698] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [39974.600189] softirqs last enabled at (239985626): [] __do_softirq+0x2be/0x371 [39974.603934] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [39974.608978] softirqs last disabled at (239985621): [] irq_exit_rcu+0xb0/0xd0 [39974.619221] ORIG_RAX: 0000000000000012 [39974.622793] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [39974.632603] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [39974.651340] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [39974.660112] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [39974.665330] Workqueue: events delayed_work [ceph] [39974.673928] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [39974.677757] [39974.687135] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [39974.694261] RIP: 0010:lock_acquire+0xf0/0x2b0 [39974.701729] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [39974.708853] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [39974.713556] [39974.720676] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [39974.766904] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [39974.774029] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [39974.781154] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [39974.788277] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [39974.795401] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [39974.802526] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [39974.810619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39974.816357] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [39974.823483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39974.830605] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39974.837730] Call Trace: [39974.840175] [39974.842185] ? watchdog_timer_fn+0x1b4/0x220 [39974.846450] ? __hrtimer_run_queues+0x1d3/0x360 [39974.850976] ? hrtimer_interrupt+0x104/0x240 [39974.855237] ? __local_bh_enable+0x50/0x60 [39974.859349] ? __sysvec_apic_timer_interrupt+0x61/0x160 [39974.864573] ? sysvec_apic_timer_interrupt+0x6d/0x90 [39974.869528] [39974.871625] [39974.873724] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [39974.879033] ? lock_acquire+0xf0/0x2b0 [39974.882782] ? igrab+0x43/0x50 [39974.885832] ? lock_release+0xc5/0x260 [39974.889576] _raw_spin_lock+0x30/0x40 [39974.893233] ? igrab+0x1d/0x50 [39974.896301] igrab+0x1d/0x50 [39974.899179] ceph_check_delayed_caps+0xb7/0x270 [ceph] [39974.904328] delayed_work+0x1d8/0x280 [ceph] [39974.908609] ? process_scheduled_works+0x22a/0x510 [39974.913401] process_scheduled_works+0x2a4/0x510 [39974.918015] worker_thread+0x188/0x340 [39974.921765] ? rescuer_thread+0x310/0x310 [39974.925767] kthread+0xf5/0x130 [39974.928907] ? kthread_complete_and_exit+0x20/0x20 [39974.933690] ret_from_fork+0x31/0x40 [39974.937262] ? kthread_complete_and_exit+0x20/0x20 [39974.942053] ret_from_fork_asm+0x11/0x20 [39974.945976] [40002.153328] watchdog: BUG: soft lockup - CPU#0 stuck for 26140s! [dbench:78683] [40002.160630] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40002.165327] watchdog: BUG: soft lockup - CPU#7 stuck for 35137s! [kworker/7:4:78808] [40002.216079] irq event stamp: 71150218 [40002.223808] Modules linked in: ceph libceph dns_resolver fscache netfs [40002.227467] hardirqs last enabled at (71150217): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40002.234004] veth nft_chain_nat xt_MASQUERADE nf_nat [40002.244159] hardirqs last disabled at (71150218): [] sysvec_apic_timer_interrupt+0xe/0x90 [40002.244162] nf_conntrack [40002.244163] softirqs last enabled at (71150172): [] __do_softirq+0x2be/0x371 [40002.249117] nf_defrag_ipv6 nf_defrag_ipv4 [40002.258839] softirqs last disabled at (71150167): [] irq_exit_rcu+0xb0/0xd0 [40002.261457] nft_compat [40002.270143] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40002.274233] nf_tables nfnetlink [40002.282743] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40002.285187] bridge [40002.294133] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40002.297356] stp llc [40002.304827] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40002.306923] xfs [40002.312400] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40002.314586] libcrc32c sunrpc [40002.333324] [40002.335161] intel_rapl_msr intel_rapl_common [40002.340376] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40002.343342] sb_edac x86_pkg_temp_thermal [40002.344832] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40002.349182] coretemp kvm_intel [40002.356306] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40002.360310] kvm [40002.367435] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40002.370573] irqbypass [40002.377697] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40002.379552] i2c_i801 iTCO_wdt [40002.386676] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40002.389034] iTCO_vendor_support i2c_smbus [40002.396157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40002.399209] crct10dif_pclmul joydev [40002.407285] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40002.411377] crc32_pclmul [40002.417114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40002.420685] ipmi_ssif [40002.427809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40002.430426] ghash_clmulni_intel [40002.437550] Call Trace: [40002.439906] mei_me [40002.447032] [40002.450254] mei lpc_ich [40002.452721] ? watchdog_timer_fn+0x1b4/0x220 [40002.454815] sha512_ssse3 acpi_ipmi [40002.456826] ? __hrtimer_run_queues+0x1d3/0x360 [40002.459355] mfd_core wmi [40002.463624] ? hrtimer_interrupt+0x104/0x240 [40002.467103] ipmi_si ipmi_devintf [40002.471627] ? __local_bh_enable+0x50/0x60 [40002.474245] ipmi_msghandler ioatdma [40002.478512] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40002.481819] acpi_pad acpi_power_meter [40002.485912] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40002.489481] ixgbe igb [40002.494701] [40002.498460] mdio [40002.503417] [40002.503419] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40002.505776] crc32c_intel [40002.507877] ? queued_spin_lock_slowpath+0x16/0x2e0 [40002.509797] nvme ptp [40002.511898] do_raw_spin_lock+0xb5/0xc0 [40002.517197] nvme_core [40002.519818] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40002.524687] i2c_algo_bit [40002.526961] try_get_cap_refs+0x159/0x880 [ceph] [40002.530787] pps_core [40002.533150] __ceph_get_caps+0x4a1/0x760 [ceph] [40002.537931] dca [40002.540551] ? __file_remove_privs+0x89/0x110 [40002.545158] fuse [40002.547428] ? wake_bit_function+0x70/0x70 [40002.551951] [40002.551952] irq event stamp: 240206340 [40002.553791] ? ceph_write_iter+0x260/0x770 [ceph] [40002.558140] hardirqs last enabled at (240206339): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40002.560064] ceph_write_iter+0x260/0x770 [ceph] [40002.564155] hardirqs last disabled at (240206340): [] sysvec_apic_timer_interrupt+0xe/0x90 [40002.565650] ? lock_is_held_type+0xaa/0x120 [40002.569389] softirqs last enabled at (240206338): [] __do_softirq+0x2be/0x371 [40002.574089] vfs_write+0x324/0x4f0 [40002.584331] softirqs last disabled at (240206333): [] irq_exit_rcu+0xb0/0xd0 [40002.588859] ksys_pwrite64+0x65/0xa0 [40002.598666] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40002.602843] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40002.611613] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40002.615014] do_syscall_64+0x38/0x80 [40002.623609] Workqueue: events delayed_work [ceph] [40002.627181] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40002.636559] [40002.641690] RIP: 0033:0x7f48f0eeb93b [40002.649159] RIP: 0010:lock_acquire+0xf0/0x2b0 [40002.652730] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40002.657427] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40002.662470] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40002.663961] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40002.667533] ORIG_RAX: 0000000000000012 [40002.671884] [40002.671885] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40002.690622] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40002.709358] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40002.714576] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40002.719793] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40002.723624] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40002.725114] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40002.732239] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40002.739363] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40002.746488] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40002.753611] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40002.760741] [40002.767859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40002.812356] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40002.819480] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40002.826604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40002.833732] Call Trace: [40002.836183] [40002.838192] ? watchdog_timer_fn+0x1b4/0x220 [40002.842457] ? __hrtimer_run_queues+0x1d3/0x360 [40002.846984] ? hrtimer_interrupt+0x104/0x240 [40002.851253] ? __local_bh_enable+0x50/0x60 [40002.855346] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40002.860570] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40002.865528] [40002.867624] [40002.869734] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40002.875047] ? lock_acquire+0xf0/0x2b0 [40002.878798] ? igrab+0x43/0x50 [40002.881848] ? lock_release+0xc5/0x260 [40002.885593] _raw_spin_lock+0x30/0x40 [40002.889257] ? igrab+0x1d/0x50 [40002.892306] igrab+0x1d/0x50 [40002.895186] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40002.900337] delayed_work+0x1d8/0x280 [ceph] [40002.904618] ? process_scheduled_works+0x22a/0x510 [40002.909406] process_scheduled_works+0x2a4/0x510 [40002.914022] worker_thread+0x188/0x340 [40002.917772] ? rescuer_thread+0x310/0x310 [40002.921776] kthread+0xf5/0x130 [40002.924921] ? kthread_complete_and_exit+0x20/0x20 [40002.929705] ret_from_fork+0x31/0x40 [40002.933302] ? kthread_complete_and_exit+0x20/0x20 [40002.938086] ret_from_fork_asm+0x11/0x20 [40002.942029] [40030.153799] watchdog: BUG: soft lockup - CPU#0 stuck for 26166s! [dbench:78683] [40030.161097] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40030.165799] watchdog: BUG: soft lockup - CPU#7 stuck for 35163s! [kworker/7:4:78808] [40030.216539] irq event stamp: 71208446 [40030.224271] Modules linked in: ceph libceph dns_resolver [40030.227926] hardirqs last enabled at (71208445): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40030.233231] fscache netfs veth [40030.243389] hardirqs last disabled at (71208446): [] sysvec_apic_timer_interrupt+0xe/0x90 [40030.246526] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [40030.256251] softirqs last enabled at (71208380): [] __do_softirq+0x2be/0x371 [40030.256254] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [40030.270586] softirqs last disabled at (71208375): [] irq_exit_rcu+0xb0/0xd0 [40030.270589] nfnetlink bridge [40030.276496] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40030.285008] stp llc [40030.287970] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40030.296916] xfs libcrc32c [40030.299100] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40030.306570] sunrpc intel_rapl_msr [40030.309274] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40030.314752] intel_rapl_common sb_edac [40030.318149] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40030.336886] x86_pkg_temp_thermal coretemp [40030.340630] [40030.340630] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40030.345849] kvm_intel kvm [40030.349938] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40030.351429] irqbypass i2c_i801 [40030.358555] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40030.361259] iTCO_wdt iTCO_vendor_support [40030.368381] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40030.371520] i2c_smbus crct10dif_pclmul [40030.378645] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40030.382648] joydev [40030.389771] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40030.393602] crc32_pclmul [40030.400725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40030.402824] ipmi_ssif ghash_clmulni_intel [40030.410902] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40030.413519] mei_me mei [40030.419257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40030.423346] lpc_ich sha512_ssse3 [40030.430471] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40030.432915] acpi_ipmi mfd_core [40030.440040] Call Trace: [40030.443349] wmi ipmi_si [40030.450474] [40030.453628] ipmi_devintf ipmi_msghandler [40030.456075] ? watchdog_timer_fn+0x1b4/0x220 [40030.458604] ioatdma [40030.460616] ? __hrtimer_run_queues+0x1d3/0x360 [40030.464617] acpi_pad acpi_power_meter [40030.468884] ? hrtimer_interrupt+0x104/0x240 [40030.471065] ixgbe igb [40030.475590] ? __local_bh_enable+0x50/0x60 [40030.479333] mdio crc32c_intel [40030.483601] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40030.485955] nvme [40030.490048] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40030.493099] ptp nvme_core [40030.498315] [40030.500240] i2c_algo_bit pps_core [40030.505201] [40030.507902] dca fuse [40030.510000] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40030.513394] [40030.513395] irq event stamp: 240427082 [40030.515510] ? queued_spin_lock_slowpath+0x18/0x2e0 [40030.517781] hardirqs last enabled at (240427081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40030.523087] do_raw_spin_lock+0xb5/0xc0 [40030.524577] hardirqs last disabled at (240427082): [] sysvec_apic_timer_interrupt+0xe/0x90 [40030.528321] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40030.533199] softirqs last enabled at (240427080): [] __do_softirq+0x2be/0x371 [40030.543446] try_get_cap_refs+0x159/0x880 [ceph] [40030.547275] softirqs last disabled at (240427075): [] irq_exit_rcu+0xb0/0xd0 [40030.557089] __ceph_get_caps+0x4a1/0x760 [ceph] [40030.561869] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40030.570645] ? __file_remove_privs+0x89/0x110 [40030.575251] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40030.583848] ? wake_bit_function+0x70/0x70 [40030.588371] Workqueue: events delayed_work [ceph] [40030.597752] ? ceph_write_iter+0x260/0x770 [ceph] [40030.602099] [40030.602100] RIP: 0010:lock_acquire+0xf0/0x2b0 [40030.609571] ceph_write_iter+0x260/0x770 [ceph] [40030.613662] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40030.618362] ? lock_is_held_type+0xaa/0x120 [40030.623056] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40030.624549] vfs_write+0x324/0x4f0 [40030.628899] [40030.633428] ksys_pwrite64+0x65/0xa0 [40030.652160] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40030.656336] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40030.661555] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40030.664953] do_syscall_64+0x38/0x80 [40030.666444] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40030.670015] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40030.677136] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40030.682268] RIP: 0033:0x7f48f0eeb93b [40030.689392] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40030.692963] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40030.700089] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40030.705132] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40030.712254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40030.715826] ORIG_RAX: 0000000000000012 [40030.722951] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40030.741689] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40030.749767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40030.754982] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40030.760720] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40030.764551] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40030.771676] Call Trace: [40030.771678] [40030.778800] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40030.785924] ? watchdog_timer_fn+0x1b4/0x220 [40030.793047] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40030.800173] ? __hrtimer_run_queues+0x1d3/0x360 [40030.807302] [40030.809744] ? hrtimer_interrupt+0x104/0x240 [40030.841235] ? __local_bh_enable+0x50/0x60 [40030.845329] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40030.850553] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40030.855509] [40030.857608] [40030.859706] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40030.865013] ? lock_acquire+0xf0/0x2b0 [40030.868762] ? igrab+0x43/0x50 [40030.871812] ? lock_release+0xc5/0x260 [40030.875559] _raw_spin_lock+0x30/0x40 [40030.879222] ? igrab+0x1d/0x50 [40030.882274] igrab+0x1d/0x50 [40030.885150] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40030.890302] delayed_work+0x1d8/0x280 [ceph] [40030.894582] ? process_scheduled_works+0x22a/0x510 [40030.899363] process_scheduled_works+0x2a4/0x510 [40030.903980] worker_thread+0x188/0x340 [40030.907728] ? rescuer_thread+0x310/0x310 [40030.911733] kthread+0xf5/0x130 [40030.914871] ? kthread_complete_and_exit+0x20/0x20 [40030.919662] ret_from_fork+0x31/0x40 [40030.923232] ? kthread_complete_and_exit+0x20/0x20 [40030.928018] ret_from_fork_asm+0x11/0x20 [40030.931941] [40058.154268] watchdog: BUG: soft lockup - CPU#0 stuck for 26193s! [dbench:78683] [40058.161568] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40058.166267] watchdog: BUG: soft lockup - CPU#7 stuck for 35190s! [kworker/7:4:78808] [40058.217017] irq event stamp: 71266556 [40058.224748] Modules linked in: ceph libceph dns_resolver fscache [40058.228405] hardirqs last enabled at (71266555): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40058.234404] netfs veth nft_chain_nat xt_MASQUERADE [40058.244562] hardirqs last disabled at (71266556): [] sysvec_apic_timer_interrupt+0xe/0x90 [40058.249432] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [40058.259157] softirqs last enabled at (71266468): [] __do_softirq+0x2be/0x371 [40058.259160] nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [40058.275493] softirqs last disabled at (71266463): [] irq_exit_rcu+0xb0/0xd0 [40058.282270] intel_rapl_common sb_edac x86_pkg_temp_thermal [40058.290783] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40058.290784] coretemp kvm_intel [40058.305291] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40058.305292] kvm [40058.308428] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40058.315898] irqbypass [40058.317737] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40058.323212] i2c_i801 [40058.325571] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40058.344307] iTCO_wdt [40058.346577] [40058.346578] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40058.351796] iTCO_vendor_support [40058.354065] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40058.355556] i2c_smbus crct10dif_pclmul [40058.362681] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40058.365904] joydev crc32_pclmul [40058.373031] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40058.376860] ipmi_ssif ghash_clmulni_intel [40058.383984] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40058.387208] mei_me mei [40058.394332] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40058.398422] lpc_ich sha512_ssse3 [40058.405546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40058.407990] acpi_ipmi mfd_core [40058.416069] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40058.419379] wmi ipmi_si [40058.425118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40058.428254] ipmi_devintf [40058.435379] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40058.437909] ipmi_msghandler [40058.445033] Call Trace: [40058.445036] [40058.447663] ioatdma acpi_pad [40058.454786] ? watchdog_timer_fn+0x1b4/0x220 [40058.457665] acpi_power_meter [40058.460107] ? __hrtimer_run_queues+0x1d3/0x360 [40058.462117] ixgbe igb [40058.465084] ? hrtimer_interrupt+0x104/0x240 [40058.469344] mdio crc32c_intel [40058.472308] ? __local_bh_enable+0x50/0x60 [40058.476832] nvme ptp [40058.479191] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40058.483453] nvme_core i2c_algo_bit [40058.486506] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40058.490595] pps_core dca [40058.492865] [40058.498082] fuse [40058.501568] [40058.506526] irq event stamp: 240647938 [40058.509143] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40058.511239] hardirqs last enabled at (240647937): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40058.513167] ? queued_spin_lock_slowpath+0x16/0x2e0 [40058.515261] hardirqs last disabled at (240647938): [] sysvec_apic_timer_interrupt+0xe/0x90 [40058.519006] do_raw_spin_lock+0xb5/0xc0 [40058.524309] softirqs last enabled at (240647936): [] __do_softirq+0x2be/0x371 [40058.534555] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40058.539424] softirqs last disabled at (240647931): [] irq_exit_rcu+0xb0/0xd0 [40058.549239] try_get_cap_refs+0x159/0x880 [ceph] [40058.553065] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40058.561843] __ceph_get_caps+0x4a1/0x760 [ceph] [40058.566622] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40058.575223] ? __file_remove_privs+0x89/0x110 [40058.579829] Workqueue: events delayed_work [ceph] [40058.589207] ? wake_bit_function+0x70/0x70 [40058.593730] [40058.601205] ? ceph_write_iter+0x260/0x770 [ceph] [40058.605551] RIP: 0010:lock_acquire+0xf0/0x2b0 [40058.610249] ceph_write_iter+0x260/0x770 [ceph] [40058.614339] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40058.615834] ? lock_is_held_type+0xaa/0x120 [40058.620527] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40058.624882] vfs_write+0x324/0x4f0 [40058.629403] [40058.648146] ksys_pwrite64+0x65/0xa0 [40058.652318] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40058.657537] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40058.660932] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40058.662425] do_syscall_64+0x38/0x80 [40058.665994] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40058.673119] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40058.678250] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40058.685374] RIP: 0033:0x7f48f0eeb93b [40058.688948] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40058.696070] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40058.701111] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40058.708235] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40058.711806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40058.718930] ORIG_RAX: 0000000000000012 [40058.737687] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40058.745764] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40058.750980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40058.756718] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40058.760550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40058.767672] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40058.774797] Call Trace: [40058.781923] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40058.789046] [40058.796169] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40058.803295] ? watchdog_timer_fn+0x1b4/0x220 [40058.805745] [40058.812864] ? __hrtimer_run_queues+0x1d3/0x360 [40058.832973] ? hrtimer_interrupt+0x104/0x240 [40058.837242] ? __local_bh_enable+0x50/0x60 [40058.841335] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40058.846550] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40058.851507] [40058.853606] [40058.855704] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40058.861019] ? lock_acquire+0xf0/0x2b0 [40058.864769] ? igrab+0x43/0x50 [40058.867820] ? lock_release+0xc5/0x260 [40058.871564] _raw_spin_lock+0x30/0x40 [40058.875229] ? igrab+0x1d/0x50 [40058.878280] igrab+0x1d/0x50 [40058.881159] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40058.886309] delayed_work+0x1d8/0x280 [ceph] [40058.890591] ? process_scheduled_works+0x22a/0x510 [40058.895379] process_scheduled_works+0x2a4/0x510 [40058.899994] worker_thread+0x188/0x340 [40058.903743] ? rescuer_thread+0x310/0x310 [40058.907747] kthread+0xf5/0x130 [40058.910886] ? kthread_complete_and_exit+0x20/0x20 [40058.915674] ret_from_fork+0x31/0x40 [40058.919249] ? kthread_complete_and_exit+0x20/0x20 [40058.924032] ret_from_fork_asm+0x11/0x20 [40058.927958] [40086.154734] watchdog: BUG: soft lockup - CPU#0 stuck for 26219s! [dbench:78683] [40086.162037] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40086.166733] watchdog: BUG: soft lockup - CPU#7 stuck for 35216s! [kworker/7:4:78808] [40086.217479] irq event stamp: 71324708 [40086.217480] hardirqs last enabled at (71324707): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40086.225210] Modules linked in: [40086.228869] hardirqs last disabled at (71324708): [] sysvec_apic_timer_interrupt+0xe/0x90 [40086.239025] ceph [40086.242077] softirqs last enabled at (71324620): [] __do_softirq+0x2be/0x371 [40086.251799] libceph [40086.253724] softirqs last disabled at (71324615): [] irq_exit_rcu+0xb0/0xd0 [40086.262410] dns_resolver fscache [40086.264592] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40086.273104] netfs veth [40086.276430] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40086.285375] nft_chain_nat [40086.287820] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40086.295290] xt_MASQUERADE nf_nat [40086.297993] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40086.303471] nf_conntrack nf_defrag_ipv6 [40086.306782] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40086.325519] nf_defrag_ipv4 [40086.329438] [40086.329439] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40086.334656] nft_compat [40086.337445] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40086.338935] nf_tables [40086.346060] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40086.348505] nfnetlink bridge [40086.355630] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40086.357987] stp llc [40086.365127] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40086.368090] xfs [40086.375216] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40086.377412] libcrc32c sunrpc [40086.384540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40086.386379] intel_rapl_msr intel_rapl_common [40086.394457] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40086.397418] sb_edac x86_pkg_temp_thermal [40086.403157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40086.407509] coretemp kvm_intel [40086.414632] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40086.418637] kvm [40086.425761] Call Trace: [40086.428898] irqbypass i2c_i801 [40086.436023] [40086.437858] iTCO_wdt [40086.440303] ? watchdog_timer_fn+0x1b4/0x220 [40086.443440] iTCO_vendor_support i2c_smbus [40086.445452] ? __hrtimer_run_queues+0x1d3/0x360 [40086.447723] crct10dif_pclmul joydev [40086.451991] ? hrtimer_interrupt+0x104/0x240 [40086.456078] crc32_pclmul ipmi_ssif [40086.460600] ? __local_bh_enable+0x50/0x60 [40086.464171] ghash_clmulni_intel [40086.468437] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40086.471920] mei_me [40086.476010] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40086.479234] mei lpc_ich [40086.484453] [40086.486549] sha512_ssse3 acpi_ipmi [40086.491507] [40086.494036] mfd_core [40086.496137] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40086.499620] wmi [40086.501721] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40086.503988] ipmi_si ipmi_devintf [40086.509293] do_raw_spin_lock+0xb5/0xc0 [40086.511134] ipmi_msghandler [40086.516009] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40086.519319] ioatdma acpi_pad [40086.523152] try_get_cap_refs+0x159/0x880 [ceph] [40086.526028] acpi_power_meter [40086.530816] __ceph_get_caps+0x4a1/0x760 [ceph] [40086.533776] ixgbe igb [40086.538389] ? __file_remove_privs+0x89/0x110 [40086.541351] mdio crc32c_intel [40086.545874] ? wake_bit_function+0x70/0x70 [40086.548230] nvme ptp [40086.552586] ? ceph_write_iter+0x260/0x770 [ceph] [40086.555632] nvme_core i2c_algo_bit [40086.559725] ceph_write_iter+0x260/0x770 [ceph] [40086.561995] pps_core dca [40086.566695] ? lock_is_held_type+0xaa/0x120 [40086.570175] fuse [40086.574702] vfs_write+0x324/0x4f0 [40086.577317] irq event stamp: 240868824 [40086.581498] ksys_pwrite64+0x65/0xa0 [40086.583422] hardirqs last enabled at (240868823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40086.586816] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40086.590559] hardirqs last disabled at (240868824): [] sysvec_apic_timer_interrupt+0xe/0x90 [40086.594136] do_syscall_64+0x38/0x80 [40086.604374] softirqs last enabled at (240868822): [] __do_softirq+0x2be/0x371 [40086.609506] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40086.619318] softirqs last disabled at (240868817): [] irq_exit_rcu+0xb0/0xd0 [40086.622888] RIP: 0033:0x7f48f0eeb93b [40086.631658] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40086.636704] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40086.645298] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40086.648871] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40086.658249] Workqueue: events delayed_work [ceph] [40086.676987] ORIG_RAX: 0000000000000012 [40086.684455] [40086.689675] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40086.694370] RIP: 0010:lock_acquire+0xf0/0x2b0 [40086.698201] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40086.699693] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40086.706817] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40086.711166] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40086.718291] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40086.737031] [40086.744152] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40086.749370] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40086.756501] [40086.757985] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40086.781560] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40086.788685] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40086.795809] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40086.802931] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40086.811008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40086.816746] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40086.823870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40086.830994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40086.838135] Call Trace: [40086.840581] [40086.842591] ? watchdog_timer_fn+0x1b4/0x220 [40086.846857] ? __hrtimer_run_queues+0x1d3/0x360 [40086.851385] ? hrtimer_interrupt+0x104/0x240 [40086.855654] ? __local_bh_enable+0x50/0x60 [40086.859745] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40086.864962] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40086.869920] [40086.872017] [40086.874115] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40086.879447] ? lock_acquire+0xf0/0x2b0 [40086.883197] ? igrab+0x43/0x50 [40086.886249] ? lock_release+0xc5/0x260 [40086.889992] _raw_spin_lock+0x30/0x40 [40086.893649] ? igrab+0x1d/0x50 [40086.896699] igrab+0x1d/0x50 [40086.899578] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40086.904727] delayed_work+0x1d8/0x280 [ceph] [40086.909008] ? process_scheduled_works+0x22a/0x510 [40086.913798] process_scheduled_works+0x2a4/0x510 [40086.918424] worker_thread+0x188/0x340 [40086.922171] ? rescuer_thread+0x310/0x310 [40086.926175] kthread+0xf5/0x130 [40086.929312] ? kthread_complete_and_exit+0x20/0x20 [40086.934098] ret_from_fork+0x31/0x40 [40086.937674] ? kthread_complete_and_exit+0x20/0x20 [40086.942461] ret_from_fork_asm+0x11/0x20 [40086.946382] [40114.155200] watchdog: BUG: soft lockup - CPU#0 stuck for 26245s! [dbench:78683] [40114.162499] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40114.167200] watchdog: BUG: soft lockup - CPU#7 stuck for 35242s! [kworker/7:4:78808] [40114.217941] irq event stamp: 71382920 [40114.217942] hardirqs last enabled at (71382919): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40114.225671] Modules linked in: [40114.229329] hardirqs last disabled at (71382920): [] sysvec_apic_timer_interrupt+0xe/0x90 [40114.239487] ceph [40114.242537] softirqs last enabled at (71382832): [] __do_softirq+0x2be/0x371 [40114.252260] libceph [40114.254186] softirqs last disabled at (71382827): [] irq_exit_rcu+0xb0/0xd0 [40114.262884] dns_resolver [40114.265071] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40114.273595] fscache netfs [40114.276215] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40114.285159] veth nft_chain_nat [40114.287864] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40114.295352] xt_MASQUERADE nf_nat [40114.298490] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40114.303967] nf_conntrack nf_defrag_ipv6 [40114.307278] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40114.326015] nf_defrag_ipv4 nft_compat [40114.329933] [40114.329934] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40114.335152] nf_tables [40114.338895] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40114.340387] nfnetlink bridge [40114.347510] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40114.349876] stp llc [40114.357001] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40114.359963] xfs [40114.367088] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40114.369271] libcrc32c sunrpc [40114.376396] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40114.378235] intel_rapl_msr intel_rapl_common [40114.385357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40114.388323] sb_edac [40114.396399] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40114.396401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40114.400749] x86_pkg_temp_thermal [40114.406487] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40114.408671] coretemp [40114.415795] Call Trace: [40114.415796] [40114.422920] kvm_intel [40114.426231] ? watchdog_timer_fn+0x1b4/0x220 [40114.433354] kvm [40114.435626] ? __hrtimer_run_queues+0x1d3/0x360 [40114.438070] irqbypass [40114.440083] ? hrtimer_interrupt+0x104/0x240 [40114.442438] i2c_i801 iTCO_wdt [40114.446700] ? __local_bh_enable+0x50/0x60 [40114.448540] iTCO_vendor_support [40114.453066] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40114.455420] i2c_smbus crct10dif_pclmul [40114.459685] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40114.462735] joydev crc32_pclmul [40114.466825] [40114.470049] ipmi_ssif [40114.475267] [40114.479099] ghash_clmulni_intel [40114.484056] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40114.487281] mei_me mei [40114.489382] ? queued_spin_lock_slowpath+0x16/0x2e0 [40114.491735] lpc_ich sha512_ssse3 [40114.493835] do_raw_spin_lock+0xb5/0xc0 [40114.497056] acpi_ipmi mfd_core [40114.502362] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40114.504804] wmi ipmi_si [40114.509676] try_get_cap_refs+0x159/0x880 [ceph] [40114.512985] ipmi_devintf ipmi_msghandler [40114.516821] __ceph_get_caps+0x4a1/0x760 [ceph] [40114.519954] ioatdma acpi_pad [40114.524741] ? __file_remove_privs+0x89/0x110 [40114.527268] acpi_power_meter ixgbe [40114.531896] ? wake_bit_function+0x70/0x70 [40114.535900] igb mdio [40114.540428] ? ceph_write_iter+0x260/0x770 [ceph] [40114.543388] crc32c_intel nvme [40114.547738] ceph_write_iter+0x260/0x770 [ceph] [40114.551223] ptp [40114.555318] ? lock_is_held_type+0xaa/0x120 [40114.557600] nvme_core [40114.562301] vfs_write+0x324/0x4f0 [40114.565349] i2c_algo_bit pps_core [40114.569885] ksys_pwrite64+0x65/0xa0 [40114.571721] dca [40114.575898] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40114.578256] fuse [40114.581654] do_syscall_64+0x38/0x80 [40114.585050] [40114.585052] irq event stamp: 241089564 [40114.588622] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40114.590459] hardirqs last enabled at (241089563): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40114.595600] RIP: 0033:0x7f48f0eeb93b [40114.597523] hardirqs last disabled at (241089564): [] sysvec_apic_timer_interrupt+0xe/0x90 [40114.601095] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40114.602599] softirqs last enabled at (241089562): [] __do_softirq+0x2be/0x371 [40114.606344] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40114.611389] softirqs last disabled at (241089557): [] irq_exit_rcu+0xb0/0xd0 [40114.621632] ORIG_RAX: 0000000000000012 [40114.625204] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40114.635014] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40114.653752] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40114.662523] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40114.667741] Workqueue: events delayed_work [ceph] [40114.676337] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40114.680168] [40114.680169] RIP: 0010:lock_acquire+0xf0/0x2b0 [40114.689546] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40114.696669] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40114.704142] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40114.711265] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40114.715967] [40114.723086] [40114.723087] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40114.777947] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40114.785072] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40114.792195] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40114.799320] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40114.806443] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40114.814523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40114.820260] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40114.827385] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40114.834507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40114.841633] Call Trace: [40114.844077] [40114.846087] ? watchdog_timer_fn+0x1b4/0x220 [40114.850351] ? __hrtimer_run_queues+0x1d3/0x360 [40114.854889] ? hrtimer_interrupt+0x104/0x240 [40114.859155] ? __local_bh_enable+0x50/0x60 [40114.863250] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40114.868472] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40114.873430] [40114.875527] [40114.877625] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40114.882934] ? lock_acquire+0xf0/0x2b0 [40114.886682] ? igrab+0x43/0x50 [40114.889734] ? lock_release+0xc5/0x260 [40114.893479] _raw_spin_lock+0x30/0x40 [40114.897144] ? igrab+0x1d/0x50 [40114.900193] igrab+0x1d/0x50 [40114.903073] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40114.908223] delayed_work+0x1d8/0x280 [ceph] [40114.912504] ? process_scheduled_works+0x22a/0x510 [40114.917293] process_scheduled_works+0x2a4/0x510 [40114.921908] worker_thread+0x188/0x340 [40114.925659] ? rescuer_thread+0x310/0x310 [40114.929670] kthread+0xf5/0x130 [40114.932816] ? kthread_complete_and_exit+0x20/0x20 [40114.937608] ret_from_fork+0x31/0x40 [40114.941190] ? kthread_complete_and_exit+0x20/0x20 [40114.945982] ret_from_fork_asm+0x11/0x20 [40114.949913] [40141.735026] rcu: INFO: rcu_preempt self-detected stall on CPU [40141.740783] rcu: 0-....: (27567455 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=13807841 [40141.751295] rcu: (t=28204633 jiffies g=1309609 q=756634 ncpus=8) [40141.757380] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40141.766325] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40141.773811] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40141.779289] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40141.798027] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40141.803270] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40141.810395] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40141.817518] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40141.824641] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40141.831768] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40141.838890] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40141.846967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40141.852706] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40141.859830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40141.866953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40141.874077] Call Trace: [40141.876524] [40141.878533] ? rcu_dump_cpu_stacks+0xd1/0x180 [40141.882893] ? rcu_sched_clock_irq+0x633/0x1390 [40141.887418] ? lock_is_held_type+0xaa/0x120 [40141.891595] ? lock_is_held_type+0xaa/0x120 [40141.895771] ? tick_sched_do_timer+0xa0/0xa0 [40141.900055] ? rcu_read_lock_sched_held+0x56/0x90 [40141.904760] ? tick_sched_do_timer+0xa0/0xa0 [40141.909022] ? update_process_times+0x5f/0x90 [40141.913375] ? tick_sched_handle.isra.16+0x1e/0x50 [40141.918157] ? tick_sched_timer+0x6f/0x80 [40141.922163] ? __hrtimer_run_queues+0x1d3/0x360 [40141.926690] ? hrtimer_interrupt+0x104/0x240 [40141.930959] ? __local_bh_enable+0x50/0x60 [40141.935060] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40141.940284] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40141.945240] [40141.947345] [40141.949446] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40141.954752] ? queued_spin_lock_slowpath+0x16/0x2e0 [40141.959630] do_raw_spin_lock+0xb5/0xc0 [40141.963468] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40141.968271] try_get_cap_refs+0x159/0x880 [ceph] [40141.972903] __ceph_get_caps+0x4a1/0x760 [ceph] [40141.977443] ? __file_remove_privs+0x89/0x110 [40141.981799] ? wake_bit_function+0x70/0x70 [40141.985890] ? ceph_write_iter+0x260/0x770 [ceph] [40141.990594] ceph_write_iter+0x260/0x770 [ceph] [40141.995131] ? lock_is_held_type+0xaa/0x120 [40141.999317] vfs_write+0x324/0x4f0 [40142.002742] ksys_pwrite64+0x65/0xa0 [40142.006315] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40142.011449] do_syscall_64+0x38/0x80 [40142.015027] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40142.020071] RIP: 0033:0x7f48f0eeb93b [40142.023641] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40142.042379] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [40142.049936] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40142.057059] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40142.064183] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40142.071310] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40142.078435] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40142.085563] [40142.087749] Sending NMI from CPU 0 to CPUs 7: [40142.092102] NMI backtrace for cpu 7 [40142.092104] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40142.092106] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40142.092107] Workqueue: events delayed_work [ceph] [40142.092129] RIP: 0010:__lock_acquire+0xc2a/0x1e20 [40142.092132] Code: 46 86 c8 61 49 0f af c7 48 c1 e8 31 49 89 c0 4c 8d 24 c5 c0 f5 5d 83 48 8b 04 c5 c0 f5 5d 83 48 85 c0 74 1b 48 83 f8 08 74 15 <48> 8b 50 10 49 39 d7 0f 84 6e fa ff ff 48 8b 00 48 85 c0 75 e5 4c [40142.092134] RSP: 0018:ffffc90002117c60 EFLAGS: 00000082 [40142.092136] RAX: ffffffff82e139e8 RBX: ffff88810debaa80 RCX: 000000005553a634 [40142.092138] RDX: 00000000000014df RSI: 0000000000000000 RDI: ffff88810debb550 [40142.092139] RBP: ffff88810debb550 R08: 0000000000005c53 R09: 0000000000000003 [40142.092140] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8360d858 [40142.092141] R13: 0000000000000003 R14: 0000000000000000 R15: c5dbdfc36fd10127 [40142.092143] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40142.092145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40142.092146] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40142.092148] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40142.092149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40142.092150] Call Trace: [40142.092151] [40142.092152] ? nmi_cpu_backtrace+0xad/0x120 [40142.092157] ? nmi_cpu_backtrace_handler+0x11/0x20 [40142.092159] ? nmi_handle+0xd4/0x260 [40142.092165] ? default_do_nmi+0x49/0xf0 [40142.092167] ? exc_nmi+0x11e/0x150 [40142.092170] ? end_repeat_nmi+0x16/0x67 [40142.092176] ? __lock_acquire+0xc2a/0x1e20 [40142.092178] ? __lock_acquire+0xc2a/0x1e20 [40142.092181] ? __lock_acquire+0xc2a/0x1e20 [40142.092184] [40142.092184] [40142.092188] lock_acquire+0xcb/0x2b0 [40142.092190] ? igrab+0x1d/0x50 [40142.092193] ? igrab+0x43/0x50 [40142.092195] ? lock_release+0xc5/0x260 [40142.092198] _raw_spin_lock+0x30/0x40 [40142.092201] ? igrab+0x1d/0x50 [40142.092202] igrab+0x1d/0x50 [40142.092205] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40142.092225] delayed_work+0x1d8/0x280 [ceph] [40142.092245] ? process_scheduled_works+0x22a/0x510 [40142.092248] process_scheduled_works+0x2a4/0x510 [40142.092254] worker_thread+0x188/0x340 [40142.092258] ? rescuer_thread+0x310/0x310 [40142.092261] kthread+0xf5/0x130 [40142.092264] ? kthread_complete_and_exit+0x20/0x20 [40142.092267] ret_from_fork+0x31/0x40 [40142.092270] ? kthread_complete_and_exit+0x20/0x20 [40142.092273] ret_from_fork_asm+0x11/0x20 [40142.092280] [40166.156067] watchdog: BUG: soft lockup - CPU#0 stuck for 26293s! [dbench:78683] [40166.163369] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40166.168067] watchdog: BUG: soft lockup - CPU#7 stuck for 35290s! [kworker/7:4:78808] [40166.218821] irq event stamp: 71491480 [40166.218822] hardirqs last enabled at (71491479): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40166.226549] Modules linked in: [40166.230207] hardirqs last disabled at (71491480): [] sysvec_apic_timer_interrupt+0xe/0x90 [40166.240365] ceph [40166.243415] softirqs last enabled at (71491372): [] __do_softirq+0x2be/0x371 [40166.253141] libceph [40166.255064] softirqs last disabled at (71491367): [] irq_exit_rcu+0xb0/0xd0 [40166.263749] dns_resolver fscache [40166.265933] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40166.274462] netfs [40166.277771] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40166.286716] veth nft_chain_nat [40166.288745] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40166.296215] xt_MASQUERADE nf_nat [40166.299353] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40166.304829] nf_conntrack nf_defrag_ipv6 [40166.308139] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40166.326878] nf_defrag_ipv4 nft_compat [40166.330795] [40166.330795] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40166.336014] nf_tables [40166.339756] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40166.341247] nfnetlink bridge [40166.348370] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40166.350744] stp llc [40166.357870] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40166.360835] xfs libcrc32c [40166.367958] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40166.370143] sunrpc [40166.377266] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40166.379970] intel_rapl_msr intel_rapl_common [40166.387094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40166.389192] sb_edac x86_pkg_temp_thermal [40166.397271] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40166.401620] coretemp [40166.407359] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40166.411361] kvm_intel kvm [40166.418485] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40166.420758] irqbypass i2c_i801 [40166.427882] Call Trace: [40166.427883] [40166.430585] iTCO_wdt [40166.437709] ? watchdog_timer_fn+0x1b4/0x220 [40166.440847] iTCO_vendor_support [40166.443291] ? __hrtimer_run_queues+0x1d3/0x360 [40166.445301] i2c_smbus crct10dif_pclmul [40166.447575] ? hrtimer_interrupt+0x104/0x240 [40166.451835] joydev crc32_pclmul [40166.455060] ? __local_bh_enable+0x50/0x60 [40166.459583] ipmi_ssif ghash_clmulni_intel [40166.463419] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40166.467678] mei_me mei [40166.470903] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40166.474994] lpc_ich [40166.479086] [40166.484303] sha512_ssse3 [40166.486752] [40166.491712] acpi_ipmi mfd_core [40166.493898] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40166.495993] wmi ipmi_si [40166.498616] ? queued_spin_lock_slowpath+0x16/0x2e0 [40166.500707] ipmi_devintf ipmi_msghandler [40166.503847] do_raw_spin_lock+0xb5/0xc0 [40166.509148] ioatdma acpi_pad [40166.511681] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40166.516552] acpi_power_meter ixgbe [40166.520557] try_get_cap_refs+0x159/0x880 [ceph] [40166.524387] igb mdio [40166.527356] __ceph_get_caps+0x4a1/0x760 [ceph] [40166.532135] crc32c_intel nvme [40166.535623] ? __file_remove_privs+0x89/0x110 [40166.540228] ptp nvme_core [40166.542501] ? wake_bit_function+0x70/0x70 [40166.547024] i2c_algo_bit pps_core [40166.550078] ? ceph_write_iter+0x260/0x770 [ceph] [40166.554424] dca [40166.557129] ceph_write_iter+0x260/0x770 [ceph] [40166.561221] fuse [40166.564620] ? lock_is_held_type+0xaa/0x120 [40166.569314] irq event stamp: 241502954 [40166.571156] vfs_write+0x324/0x4f0 [40166.575677] hardirqs last enabled at (241502953): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40166.577606] ksys_pwrite64+0x65/0xa0 [40166.581778] hardirqs last disabled at (241502954): [] sysvec_apic_timer_interrupt+0xe/0x90 [40166.585523] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40166.588920] softirqs last enabled at (241502952): [] __do_softirq+0x2be/0x371 [40166.599164] do_syscall_64+0x38/0x80 [40166.602733] softirqs last disabled at (241502947): [] irq_exit_rcu+0xb0/0xd0 [40166.612562] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40166.617694] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40166.626472] RIP: 0033:0x7f48f0eeb93b [40166.630042] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40166.638640] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40166.643684] Workqueue: events delayed_work [ceph] [40166.653063] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40166.656632] [40166.656633] RIP: 0010:lock_acquire+0xf0/0x2b0 [40166.664104] ORIG_RAX: 0000000000000012 [40166.682842] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40166.687538] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40166.692771] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40166.694265] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40166.698615] [40166.698615] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40166.702446] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40166.721201] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40166.728325] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40166.733543] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40166.740666] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40166.742157] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40166.749287] [40166.756405] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40166.801334] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40166.809412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40166.815148] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40166.822275] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40166.829397] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40166.836521] Call Trace: [40166.838966] [40166.840977] ? watchdog_timer_fn+0x1b4/0x220 [40166.845242] ? __hrtimer_run_queues+0x1d3/0x360 [40166.849769] ? hrtimer_interrupt+0x104/0x240 [40166.854036] ? __local_bh_enable+0x50/0x60 [40166.858132] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40166.863355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40166.868313] [40166.870410] [40166.872506] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40166.877816] ? lock_acquire+0xf0/0x2b0 [40166.881564] ? igrab+0x43/0x50 [40166.884614] ? lock_release+0xc5/0x260 [40166.888359] _raw_spin_lock+0x30/0x40 [40166.892016] ? igrab+0x1d/0x50 [40166.895067] igrab+0x1d/0x50 [40166.897945] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40166.903094] delayed_work+0x1d8/0x280 [ceph] [40166.907376] ? process_scheduled_works+0x22a/0x510 [40166.912165] process_scheduled_works+0x2a4/0x510 [40166.916782] worker_thread+0x188/0x340 [40166.920530] ? rescuer_thread+0x310/0x310 [40166.924535] kthread+0xf5/0x130 [40166.927671] ? kthread_complete_and_exit+0x20/0x20 [40166.932473] ret_from_fork+0x31/0x40 [40166.936043] ? kthread_complete_and_exit+0x20/0x20 [40166.940828] ret_from_fork_asm+0x11/0x20 [40166.944761] [40194.156539] watchdog: BUG: soft lockup - CPU#0 stuck for 26319s! [dbench:78683] [40194.163841] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40194.168538] watchdog: BUG: soft lockup - CPU#7 stuck for 35316s! [kworker/7:4:78808] [40194.219289] irq event stamp: 71549684 [40194.227021] Modules linked in: ceph libceph dns_resolver fscache netfs [40194.230679] hardirqs last enabled at (71549683): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40194.237213] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [40194.247369] hardirqs last disabled at (71549684): [] sysvec_apic_timer_interrupt+0xe/0x90 [40194.247373] nf_defrag_ipv4 [40194.247374] softirqs last enabled at (71549588): [] __do_softirq+0x2be/0x371 [40194.254755] nft_compat nf_tables [40194.264478] softirqs last disabled at (71549583): [] irq_exit_rcu+0xb0/0xd0 [40194.267270] nfnetlink [40194.275955] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40194.279263] bridge [40194.287774] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40194.290133] stp llc [40194.299078] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40194.301173] xfs libcrc32c [40194.308644] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40194.310830] sunrpc [40194.316306] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40194.319010] intel_rapl_msr [40194.337748] [40194.339844] intel_rapl_common sb_edac [40194.345062] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40194.347853] x86_pkg_temp_thermal [40194.349344] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40194.353087] coretemp kvm_intel [40194.360225] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40194.363539] kvm irqbypass [40194.370665] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40194.373801] i2c_i801 [40194.380932] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40194.383640] iTCO_wdt iTCO_vendor_support [40194.390762] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40194.393032] i2c_smbus crct10dif_pclmul [40194.400157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40194.404161] joydev [40194.412238] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40194.416070] crc32_pclmul ipmi_ssif [40194.421807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40194.423904] ghash_clmulni_intel [40194.431030] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40194.434513] mei_me mei [40194.441636] Call Trace: [40194.444861] lpc_ich sha512_ssse3 [40194.451986] [40194.454430] acpi_ipmi [40194.456876] ? watchdog_timer_fn+0x1b4/0x220 [40194.460184] mfd_core wmi [40194.462212] ? __hrtimer_run_queues+0x1d3/0x360 [40194.464569] ipmi_si ipmi_devintf [40194.468836] ? hrtimer_interrupt+0x104/0x240 [40194.471450] ipmi_msghandler ioatdma [40194.475975] ? __local_bh_enable+0x50/0x60 [40194.479285] acpi_pad [40194.483552] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40194.487119] acpi_power_meter [40194.491224] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40194.493492] ixgbe igb [40194.498709] [40194.501673] mdio [40194.506630] [40194.506632] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40194.508986] crc32c_intel [40194.511088] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40194.513009] nvme ptp nvme_core [40194.515108] do_raw_spin_lock+0xb5/0xc0 [40194.520410] i2c_algo_bit pps_core [40194.523029] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40194.527897] dca fuse [40194.531038] try_get_cap_refs+0x159/0x880 [ceph] [40194.534866] [40194.534867] irq event stamp: 241723688 [40194.538270] __ceph_get_caps+0x4a1/0x760 [ceph] [40194.543048] hardirqs last enabled at (241723687): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40194.545323] ? __file_remove_privs+0x89/0x110 [40194.549937] hardirqs last disabled at (241723688): [] sysvec_apic_timer_interrupt+0xe/0x90 [40194.551429] ? wake_bit_function+0x70/0x70 [40194.555174] softirqs last enabled at (241723686): [] __do_softirq+0x2be/0x371 [40194.559698] ? ceph_write_iter+0x260/0x770 [ceph] [40194.569939] softirqs last disabled at (241723681): [] irq_exit_rcu+0xb0/0xd0 [40194.574292] ceph_write_iter+0x260/0x770 [ceph] [40194.584101] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40194.588197] ? lock_is_held_type+0xaa/0x120 [40194.596963] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40194.601663] vfs_write+0x324/0x4f0 [40194.610258] Workqueue: events delayed_work [ceph] [40194.614789] ksys_pwrite64+0x65/0xa0 [40194.624161] [40194.624162] RIP: 0010:lock_release+0x14e/0x260 [40194.628338] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40194.635809] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [40194.639207] do_syscall_64+0x38/0x80 [40194.643937] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [40194.647511] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40194.648999] [40194.649000] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [40194.653438] RIP: 0033:0x7f48f0eeb93b [40194.658567] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [40194.677307] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40194.680877] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [40194.686093] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40194.691138] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [40194.692627] ORIG_RAX: 0000000000000012 [40194.699753] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [40194.703323] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40194.710448] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40194.729186] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40194.736309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40194.741528] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40194.748652] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40194.752481] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40194.759605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40194.766730] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40194.774808] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40194.781948] [40194.787678] Call Trace: [40194.787679] [40194.837079] ? watchdog_timer_fn+0x1b4/0x220 [40194.841344] ? __hrtimer_run_queues+0x1d3/0x360 [40194.845870] ? hrtimer_interrupt+0x104/0x240 [40194.850139] ? __local_bh_enable+0x50/0x60 [40194.854234] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40194.859456] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40194.864413] [40194.866510] [40194.868610] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40194.873915] ? igrab+0x43/0x50 [40194.876993] ? lock_release+0x14e/0x260 [40194.880830] ? lock_release+0xc5/0x260 [40194.884576] _raw_spin_unlock+0x1b/0x40 [40194.888414] igrab+0x43/0x50 [40194.891292] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40194.896441] delayed_work+0x1d8/0x280 [ceph] [40194.900724] ? process_scheduled_works+0x22a/0x510 [40194.905511] process_scheduled_works+0x2a4/0x510 [40194.910128] worker_thread+0x188/0x340 [40194.913876] ? rescuer_thread+0x310/0x310 [40194.917880] kthread+0xf5/0x130 [40194.921018] ? kthread_complete_and_exit+0x20/0x20 [40194.925803] ret_from_fork+0x31/0x40 [40194.929382] ? kthread_complete_and_exit+0x20/0x20 [40194.934165] ret_from_fork_asm+0x11/0x20 [40194.938097] [40222.157009] watchdog: BUG: soft lockup - CPU#0 stuck for 26345s! [dbench:78683] [40222.164311] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40222.169009] watchdog: BUG: soft lockup - CPU#7 stuck for 35342s! [kworker/7:4:78808] [40222.219751] irq event stamp: 71607926 [40222.219752] hardirqs last enabled at (71607925): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40222.227482] Modules linked in: [40222.231139] hardirqs last disabled at (71607926): [] sysvec_apic_timer_interrupt+0xe/0x90 [40222.241296] ceph [40222.244346] softirqs last enabled at (71607890): [] __do_softirq+0x2be/0x371 [40222.254072] libceph [40222.255996] softirqs last disabled at (71607885): [] irq_exit_rcu+0xb0/0xd0 [40222.264685] dns_resolver fscache [40222.266872] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40222.275400] netfs veth [40222.278712] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40222.287656] nft_chain_nat xt_MASQUERADE [40222.290099] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40222.297571] nf_nat nf_conntrack [40222.301487] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40222.306965] nf_defrag_ipv6 nf_defrag_ipv4 [40222.310189] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40222.328926] nft_compat [40222.333018] [40222.333019] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40222.338235] nf_tables [40222.340694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40222.342188] nfnetlink bridge [40222.349312] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40222.351684] stp [40222.358811] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40222.361775] llc xfs [40222.368899] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40222.370736] libcrc32c sunrpc [40222.377860] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40222.380044] intel_rapl_msr [40222.387169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40222.390133] intel_rapl_common [40222.398210] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40222.401001] sb_edac [40222.406737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40222.409788] x86_pkg_temp_thermal [40222.416913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40222.419096] coretemp kvm_intel [40222.426221] Call Trace: [40222.426222] [40222.429532] kvm irqbypass [40222.436656] ? watchdog_timer_fn+0x1b4/0x220 [40222.439793] i2c_i801 [40222.442240] ? __hrtimer_run_queues+0x1d3/0x360 [40222.444247] iTCO_wdt iTCO_vendor_support [40222.446956] ? hrtimer_interrupt+0x104/0x240 [40222.451217] i2c_smbus crct10dif_pclmul [40222.453487] ? __local_bh_enable+0x50/0x60 [40222.458012] joydev crc32_pclmul [40222.462018] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40222.466279] ipmi_ssif [40222.470110] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40222.474201] ghash_clmulni_intel [40222.477425] [40222.482642] mei_me mei [40222.485000] [40222.485002] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40222.489955] lpc_ich [40222.493186] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40222.495278] sha512_ssse3 acpi_ipmi [40222.497725] do_raw_spin_lock+0xb5/0xc0 [40222.499820] mfd_core wmi [40222.505124] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40222.507309] ipmi_si ipmi_devintf [40222.512181] try_get_cap_refs+0x159/0x880 [ceph] [40222.515661] ipmi_msghandler ioatdma [40222.519498] __ceph_get_caps+0x4a1/0x760 [ceph] [40222.522111] acpi_pad acpi_power_meter [40222.526899] ? __file_remove_privs+0x89/0x110 [40222.530204] ixgbe igb [40222.534816] ? wake_bit_function+0x70/0x70 [40222.538404] mdio crc32c_intel [40222.542932] ? ceph_write_iter+0x260/0x770 [ceph] [40222.546689] nvme ptp [40222.551042] ceph_write_iter+0x260/0x770 [ceph] [40222.553410] nvme_core [40222.557502] ? lock_is_held_type+0xaa/0x120 [40222.560549] i2c_algo_bit pps_core [40222.565248] vfs_write+0x324/0x4f0 [40222.567515] dca fuse [40222.572045] ksys_pwrite64+0x65/0xa0 [40222.574409] [40222.574410] irq event stamp: 241944512 [40222.578584] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40222.581980] hardirqs last enabled at (241944511): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40222.585381] do_syscall_64+0x38/0x80 [40222.587648] hardirqs last disabled at (241944512): [] sysvec_apic_timer_interrupt+0xe/0x90 [40222.591220] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40222.592710] softirqs last enabled at (241944510): [] __do_softirq+0x2be/0x371 [40222.596456] RIP: 0033:0x7f48f0eeb93b [40222.601585] softirqs last disabled at (241944505): [] irq_exit_rcu+0xb0/0xd0 [40222.611831] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40222.615410] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40222.625220] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40222.630263] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40222.639035] ORIG_RAX: 0000000000000012 [40222.642607] Workqueue: events delayed_work [ceph] [40222.651202] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40222.669940] [40222.679319] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40222.684536] RIP: 0010:lock_acquire+0xf0/0x2b0 [40222.692007] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40222.695836] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40222.700535] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40222.707657] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40222.709149] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40222.716274] [40222.716275] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40222.720630] [40222.727747] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40222.783875] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40222.791000] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40222.798122] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40222.805248] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40222.813326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40222.819061] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40222.826186] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40222.833309] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40222.840436] Call Trace: [40222.842880] [40222.844890] ? watchdog_timer_fn+0x1b4/0x220 [40222.849155] ? __hrtimer_run_queues+0x1d3/0x360 [40222.853682] ? hrtimer_interrupt+0x104/0x240 [40222.857950] ? __local_bh_enable+0x50/0x60 [40222.862043] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40222.867269] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40222.872224] [40222.874323] [40222.876419] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40222.881728] ? lock_acquire+0xf0/0x2b0 [40222.885478] ? igrab+0x43/0x50 [40222.888529] ? lock_release+0xc5/0x260 [40222.892273] _raw_spin_lock+0x30/0x40 [40222.895930] ? igrab+0x1d/0x50 [40222.898980] igrab+0x1d/0x50 [40222.901857] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40222.907008] delayed_work+0x1d8/0x280 [ceph] [40222.911290] ? process_scheduled_works+0x22a/0x510 [40222.916079] process_scheduled_works+0x2a4/0x510 [40222.920708] worker_thread+0x188/0x340 [40222.924453] ? rescuer_thread+0x310/0x310 [40222.928465] kthread+0xf5/0x130 [40222.931600] ? kthread_complete_and_exit+0x20/0x20 [40222.936386] ret_from_fork+0x31/0x40 [40222.939974] ? kthread_complete_and_exit+0x20/0x20 [40222.944759] ret_from_fork_asm+0x11/0x20 [40222.948682] [40250.157469] watchdog: BUG: soft lockup - CPU#0 stuck for 26371s! [dbench:78683] [40250.164773] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40250.169468] watchdog: BUG: soft lockup - CPU#7 stuck for 35368s! [kworker/7:4:78808] [40250.220222] irq event stamp: 71666214 [40250.227951] Modules linked in: ceph libceph dns_resolver fscache netfs [40250.231610] hardirqs last enabled at (71666213): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40250.238150] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [40250.248310] hardirqs last disabled at (71666214): [] sysvec_apic_timer_interrupt+0xe/0x90 [40250.254395] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [40250.264117] softirqs last enabled at (71666208): [] __do_softirq+0x2be/0x371 [40250.271865] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [40250.280551] softirqs last disabled at (71666203): [] irq_exit_rcu+0xb0/0xd0 [40250.287068] sb_edac x86_pkg_temp_thermal coretemp kvm_intel [40250.295580] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40250.295581] kvm irqbypass [40250.301229] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40250.310174] i2c_i801 iTCO_wdt [40250.312878] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40250.320348] iTCO_vendor_support [40250.323401] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40250.328876] i2c_smbus crct10dif_pclmul [40250.332101] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40250.350838] joydev [40250.354686] [40250.359905] crc32_pclmul [40250.362001] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40250.363493] ipmi_ssif ghash_clmulni_intel [40250.366109] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40250.373233] mei_me mei [40250.377325] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40250.384448] lpc_ich sha512_ssse3 [40250.386891] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40250.394018] acpi_ipmi mfd_core [40250.397326] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40250.404451] wmi [40250.407590] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40250.414712] ipmi_si [40250.416550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40250.424627] ipmi_devintf [40250.426813] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40250.432549] ipmi_msghandler [40250.435168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40250.442291] ioatdma acpi_pad [40250.445167] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40250.452293] acpi_power_meter [40250.455258] Call Trace: [40250.455260] [40250.462381] ixgbe igb [40250.465347] ? watchdog_timer_fn+0x1b4/0x220 [40250.467788] mdio crc32c_intel [40250.469802] ? __hrtimer_run_queues+0x1d3/0x360 [40250.472157] nvme ptp [40250.476424] ? hrtimer_interrupt+0x104/0x240 [40250.479471] nvme_core i2c_algo_bit [40250.483996] ? __local_bh_enable+0x50/0x60 [40250.486266] pps_core dca [40250.490533] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40250.494014] fuse [40250.498107] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40250.500724] irq event stamp: 242165134 [40250.505941] [40250.507868] hardirqs last enabled at (242165133): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40250.512821] [40250.512823] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40250.516565] hardirqs last disabled at (242165134): [] sysvec_apic_timer_interrupt+0xe/0x90 [40250.518667] ? queued_spin_lock_slowpath+0x16/0x2e0 [40250.528909] softirqs last enabled at (242165132): [] __do_softirq+0x2be/0x371 [40250.531006] do_raw_spin_lock+0xb5/0xc0 [40250.536309] softirqs last disabled at (242165127): [] irq_exit_rcu+0xb0/0xd0 [40250.546122] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40250.550991] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40250.559764] try_get_cap_refs+0x159/0x880 [ceph] [40250.563591] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40250.572194] __ceph_get_caps+0x4a1/0x760 [ceph] [40250.576975] Workqueue: events delayed_work [ceph] [40250.586355] ? __file_remove_privs+0x89/0x110 [40250.590961] [40250.598434] ? wake_bit_function+0x70/0x70 [40250.602956] RIP: 0010:lock_acquire+0xf0/0x2b0 [40250.607656] ? ceph_write_iter+0x260/0x770 [ceph] [40250.612006] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40250.613495] ceph_write_iter+0x260/0x770 [ceph] [40250.617586] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40250.621941] ? lock_is_held_type+0xaa/0x120 [40250.626635] [40250.626636] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40250.645375] vfs_write+0x324/0x4f0 [40250.649897] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40250.655120] ksys_pwrite64+0x65/0xa0 [40250.659292] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40250.660784] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40250.667906] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40250.671304] do_syscall_64+0x38/0x80 [40250.678427] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40250.682000] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40250.689124] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40250.694253] RIP: 0033:0x7f48f0eeb93b [40250.701377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40250.704949] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40250.712073] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40250.717117] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40250.725193] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40250.728765] ORIG_RAX: 0000000000000012 [40250.734502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40250.753240] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40250.760364] Call Trace: [40250.760365] [40250.765581] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40250.772707] ? watchdog_timer_fn+0x1b4/0x220 [40250.776536] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40250.783662] ? __hrtimer_run_queues+0x1d3/0x360 [40250.790783] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40250.793233] ? hrtimer_interrupt+0x104/0x240 [40250.795240] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40250.802363] ? __local_bh_enable+0x50/0x60 [40250.806634] [40250.813754] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40250.848281] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40250.853238] [40250.855336] [40250.857433] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40250.862741] ? lock_acquire+0xf0/0x2b0 [40250.866491] ? igrab+0x43/0x50 [40250.869540] ? lock_release+0xc5/0x260 [40250.873286] _raw_spin_lock+0x30/0x40 [40250.876951] ? igrab+0x1d/0x50 [40250.880001] igrab+0x1d/0x50 [40250.882881] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40250.888031] delayed_work+0x1d8/0x280 [ceph] [40250.892310] ? process_scheduled_works+0x22a/0x510 [40250.897100] process_scheduled_works+0x2a4/0x510 [40250.901717] worker_thread+0x188/0x340 [40250.905466] ? rescuer_thread+0x310/0x310 [40250.909467] kthread+0xf5/0x130 [40250.912606] ? kthread_complete_and_exit+0x20/0x20 [40250.917392] ret_from_fork+0x31/0x40 [40250.920970] ? kthread_complete_and_exit+0x20/0x20 [40250.925754] ret_from_fork_asm+0x11/0x20 [40250.929685] [40278.157922] watchdog: BUG: soft lockup - CPU#0 stuck for 26397s! [dbench:78683] [40278.165224] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40278.169922] watchdog: BUG: soft lockup - CPU#7 stuck for 35394s! [kworker/7:4:78808] [40278.220664] irq event stamp: 71724352 [40278.228395] Modules linked in: ceph libceph [40278.232052] hardirqs last enabled at (71724351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40278.232055] dns_resolver fscache netfs veth [40278.246389] hardirqs last disabled at (71724352): [] sysvec_apic_timer_interrupt+0xe/0x90 [40278.250653] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [40278.260377] softirqs last enabled at (71724264): [] __do_softirq+0x2be/0x371 [40278.266027] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [40278.274712] softirqs last disabled at (71724259): [] irq_exit_rcu+0xb0/0xd0 [40278.274713] nf_tables nfnetlink bridge stp [40278.288267] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40278.292443] llc xfs [40278.301387] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40278.303573] libcrc32c sunrpc intel_rapl_msr [40278.311061] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40278.311063] intel_rapl_common sb_edac x86_pkg_temp_thermal [40278.320802] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40278.320803] coretemp kvm_intel [40278.326366] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40278.345104] kvm irqbypass [40278.348242] [40278.348243] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40278.353458] i2c_i801 [40278.356163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40278.357653] iTCO_wdt iTCO_vendor_support [40278.364777] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40278.367047] i2c_smbus crct10dif_pclmul [40278.374171] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40278.378177] joydev [40278.385317] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40278.389148] crc32_pclmul [40278.396273] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40278.398369] ipmi_ssif [40278.405495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40278.408112] ghash_clmulni_intel [40278.416188] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40278.418546] mei_me mei [40278.424285] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40278.427508] lpc_ich [40278.434633] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40278.437075] sha512_ssse3 acpi_ipmi [40278.444199] Call Trace: [40278.444201] [40278.446385] mfd_core wmi [40278.453509] ? watchdog_timer_fn+0x1b4/0x220 [40278.456992] ipmi_si ipmi_devintf [40278.459436] ? __hrtimer_run_queues+0x1d3/0x360 [40278.461446] ipmi_msghandler ioatdma [40278.464067] ? hrtimer_interrupt+0x104/0x240 [40278.468328] acpi_pad acpi_power_meter [40278.471639] ? __local_bh_enable+0x50/0x60 [40278.476162] ixgbe igb [40278.479738] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40278.483997] mdio crc32c_intel [40278.487742] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40278.491834] nvme [40278.494190] [40278.499408] ptp nvme_core [40278.502458] [40278.502459] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40278.507416] i2c_algo_bit pps_core [40278.509343] ? queued_spin_lock_slowpath+0x18/0x2e0 [40278.511437] dca fuse [40278.514143] do_raw_spin_lock+0xb5/0xc0 [40278.516239] [40278.516240] irq event stamp: 242386046 [40278.521544] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40278.524939] hardirqs last enabled at (242386045): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40278.529812] try_get_cap_refs+0x159/0x880 [ceph] [40278.532082] hardirqs last disabled at (242386046): [] sysvec_apic_timer_interrupt+0xe/0x90 [40278.535917] __ceph_get_caps+0x4a1/0x760 [ceph] [40278.537403] softirqs last enabled at (242386044): [] __do_softirq+0x2be/0x371 [40278.541152] ? __file_remove_privs+0x89/0x110 [40278.545932] softirqs last disabled at (242386039): [] irq_exit_rcu+0xb0/0xd0 [40278.556175] ? wake_bit_function+0x70/0x70 [40278.560787] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40278.570611] ? ceph_write_iter+0x260/0x770 [ceph] [40278.575130] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40278.583902] ceph_write_iter+0x260/0x770 [ceph] [40278.588252] Workqueue: events delayed_work [ceph] [40278.596854] ? lock_is_held_type+0xaa/0x120 [40278.600940] [40278.610325] vfs_write+0x324/0x4f0 [40278.615024] RIP: 0010:lock_acquire+0xf0/0x2b0 [40278.622498] ksys_pwrite64+0x65/0xa0 [40278.627019] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40278.631715] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40278.635893] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40278.637385] do_syscall_64+0x38/0x80 [40278.640782] [40278.640782] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40278.645133] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40278.648701] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40278.667439] RIP: 0033:0x7f48f0eeb93b [40278.672572] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40278.677789] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40278.681360] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40278.682849] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40278.689974] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40278.695017] ORIG_RAX: 0000000000000012 [40278.702142] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40278.705712] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40278.712838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40278.731573] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40278.738700] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40278.743915] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40278.751040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40278.754871] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40278.762948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40278.770071] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40278.775809] Call Trace: [40278.775811] [40278.782940] [40278.790060] ? watchdog_timer_fn+0x1b4/0x220 [40278.836615] ? __hrtimer_run_queues+0x1d3/0x360 [40278.841145] ? hrtimer_interrupt+0x104/0x240 [40278.845413] ? __local_bh_enable+0x50/0x60 [40278.849506] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40278.854730] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40278.859689] [40278.861786] [40278.863884] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40278.869192] ? lock_acquire+0xf0/0x2b0 [40278.872941] ? igrab+0x43/0x50 [40278.875991] ? lock_release+0xc5/0x260 [40278.879736] _raw_spin_lock+0x30/0x40 [40278.883402] ? igrab+0x1d/0x50 [40278.886451] igrab+0x1d/0x50 [40278.889332] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40278.894480] delayed_work+0x1d8/0x280 [ceph] [40278.898760] ? process_scheduled_works+0x22a/0x510 [40278.903543] process_scheduled_works+0x2a4/0x510 [40278.908158] worker_thread+0x188/0x340 [40278.911905] ? rescuer_thread+0x310/0x310 [40278.915910] kthread+0xf5/0x130 [40278.919048] ? kthread_complete_and_exit+0x20/0x20 [40278.923833] ret_from_fork+0x31/0x40 [40278.927410] ? kthread_complete_and_exit+0x20/0x20 [40278.932196] ret_from_fork_asm+0x11/0x20 [40278.936117] [40306.158378] watchdog: BUG: soft lockup - CPU#0 stuck for 26424s! [dbench:78683] [40306.165686] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40306.170377] watchdog: BUG: soft lockup - CPU#7 stuck for 35421s! [kworker/7:4:78808] [40306.221134] irq event stamp: 71782532 [40306.228866] Modules linked in: ceph libceph dns_resolver fscache netfs [40306.232524] hardirqs last enabled at (71782531): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40306.239049] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [40306.249208] hardirqs last disabled at (71782532): [] sysvec_apic_timer_interrupt+0xe/0x90 [40306.256591] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [40306.266316] softirqs last enabled at (71782496): [] __do_softirq+0x2be/0x371 [40306.273094] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [40306.281778] softirqs last disabled at (71782491): [] irq_exit_rcu+0xb0/0xd0 [40306.288651] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [40306.297161] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40306.303332] i2c_i801 iTCO_wdt [40306.312275] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40306.312277] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40306.315327] iTCO_vendor_support [40306.322798] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40306.328275] i2c_smbus crct10dif_pclmul [40306.331500] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40306.350237] joydev crc32_pclmul [40306.354066] [40306.354067] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40306.359284] ipmi_ssif ghash_clmulni_intel [40306.362509] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40306.364000] mei_me [40306.371124] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40306.375215] mei lpc_ich [40306.382337] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40306.384435] sha512_ssse3 acpi_ipmi [40306.391559] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40306.394090] mfd_core wmi [40306.401215] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40306.404698] ipmi_si [40306.411823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40306.414441] ipmi_devintf ipmi_msghandler [40306.422517] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40306.424702] ioatdma acpi_pad [40306.430439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40306.434443] acpi_power_meter ixgbe [40306.441569] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40306.444532] igb mdio [40306.451657] Call Trace: [40306.451658] [40306.455139] crc32c_intel nvme [40306.462265] ? watchdog_timer_fn+0x1b4/0x220 [40306.464533] ptp nvme_core [40306.466980] ? __hrtimer_run_queues+0x1d3/0x360 [40306.468989] i2c_algo_bit pps_core [40306.472058] ? hrtimer_interrupt+0x104/0x240 [40306.476321] dca fuse [40306.479024] ? __local_bh_enable+0x50/0x60 [40306.483550] [40306.483551] irq event stamp: 242606876 [40306.486949] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40306.491211] hardirqs last enabled at (242606875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40306.493484] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40306.497571] hardirqs last disabled at (242606876): [] sysvec_apic_timer_interrupt+0xe/0x90 [40306.499064] [40306.502806] softirqs last enabled at (242606874): [] __do_softirq+0x2be/0x371 [40306.508023] [40306.508025] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40306.518269] softirqs last disabled at (242606869): [] irq_exit_rcu+0xb0/0xd0 [40306.523230] ? queued_spin_lock_slowpath+0x16/0x2e0 [40306.533054] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40306.535155] do_raw_spin_lock+0xb5/0xc0 [40306.543922] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40306.543923] Workqueue: events delayed_work [ceph] [40306.546022] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40306.551325] [40306.551326] RIP: 0010:lock_acquire+0xf0/0x2b0 [40306.559925] try_get_cap_refs+0x159/0x880 [ceph] [40306.564791] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40306.574175] __ceph_get_caps+0x4a1/0x760 [ceph] [40306.578002] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40306.585476] ? __file_remove_privs+0x89/0x110 [40306.590169] [40306.590169] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40306.594954] ? wake_bit_function+0x70/0x70 [40306.596444] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40306.600796] ? ceph_write_iter+0x260/0x770 [ceph] [40306.605405] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40306.624144] ceph_write_iter+0x260/0x770 [ceph] [40306.628668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40306.633888] ? lock_is_held_type+0xaa/0x120 [40306.638235] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40306.639727] vfs_write+0x324/0x4f0 [40306.646850] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40306.650946] ksys_pwrite64+0x65/0xa0 [40306.658064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40306.662781] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40306.669904] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40306.669906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40306.674429] do_syscall_64+0x38/0x80 [40306.681552] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40306.685730] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40306.692853] Call Trace: [40306.692855] [40306.696252] RIP: 0033:0x7f48f0eeb93b [40306.704328] ? watchdog_timer_fn+0x1b4/0x220 [40306.707899] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40306.713638] ? __hrtimer_run_queues+0x1d3/0x360 [40306.718779] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40306.725903] ? hrtimer_interrupt+0x104/0x240 [40306.733023] ORIG_RAX: 0000000000000012 [40306.736596] ? __local_bh_enable+0x50/0x60 [40306.743720] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40306.748784] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40306.751224] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40306.753236] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40306.756806] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40306.761070] [40306.779808] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40306.784331] [40306.789549] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40306.793813] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40306.797649] [40306.801740] ? lock_acquire+0xf0/0x2b0 [40306.862959] ? igrab+0x43/0x50 [40306.866010] ? lock_release+0xc5/0x260 [40306.869754] _raw_spin_lock+0x30/0x40 [40306.873438] ? igrab+0x1d/0x50 [40306.876486] igrab+0x1d/0x50 [40306.879366] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40306.884515] delayed_work+0x1d8/0x280 [ceph] [40306.888796] ? process_scheduled_works+0x22a/0x510 [40306.893585] process_scheduled_works+0x2a4/0x510 [40306.898203] worker_thread+0x188/0x340 [40306.901949] ? rescuer_thread+0x310/0x310 [40306.905954] kthread+0xf5/0x130 [40306.909092] ? kthread_complete_and_exit+0x20/0x20 [40306.913875] ret_from_fork+0x31/0x40 [40306.917456] ? kthread_complete_and_exit+0x20/0x20 [40306.922239] ret_from_fork_asm+0x11/0x20 [40306.926163] [40334.158832] watchdog: BUG: soft lockup - CPU#0 stuck for 26450s! [dbench:78683] [40334.166130] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40334.170832] watchdog: BUG: soft lockup - CPU#7 stuck for 35447s! [kworker/7:4:78808] [40334.221570] irq event stamp: 71840648 [40334.221572] hardirqs last enabled at (71840647): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40334.229302] Modules linked in: [40334.232959] hardirqs last disabled at (71840648): [] sysvec_apic_timer_interrupt+0xe/0x90 [40334.243116] ceph [40334.246167] softirqs last enabled at (71840628): [] __do_softirq+0x2be/0x371 [40334.255892] libceph [40334.257815] softirqs last disabled at (71840623): [] irq_exit_rcu+0xb0/0xd0 [40334.266506] dns_resolver [40334.268693] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40334.277204] fscache netfs [40334.279839] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40334.288783] veth nft_chain_nat [40334.291487] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40334.298975] xt_MASQUERADE nf_nat [40334.302111] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40334.307590] nf_conntrack [40334.310899] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40334.329639] nf_defrag_ipv6 nf_defrag_ipv4 [40334.332254] [40334.332255] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40334.337472] nft_compat nf_tables [40334.341562] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40334.343054] nfnetlink [40334.350177] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40334.353505] bridge stp [40334.360631] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40334.362988] llc xfs [40334.370112] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40334.372556] libcrc32c sunrpc [40334.379680] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40334.381863] intel_rapl_msr [40334.388987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40334.391952] intel_rapl_common [40334.400029] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40334.402820] sb_edac [40334.408558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40334.411608] x86_pkg_temp_thermal coretemp [40334.418731] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40334.418732] Call Trace: [40334.420918] kvm_intel [40334.428041] [40334.432131] kvm irqbypass [40334.439256] ? watchdog_timer_fn+0x1b4/0x220 [40334.441700] i2c_i801 [40334.444057] ? __hrtimer_run_queues+0x1d3/0x360 [40334.446068] iTCO_wdt [40334.448775] ? hrtimer_interrupt+0x104/0x240 [40334.453035] iTCO_vendor_support [40334.455308] ? __local_bh_enable+0x50/0x60 [40334.459830] i2c_smbus [40334.462105] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40334.466366] crct10dif_pclmul [40334.469590] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40334.473680] joydev crc32_pclmul [40334.476038] [40334.481255] ipmi_ssif ghash_clmulni_intel [40334.484230] [40334.489185] mei_me mei [40334.492410] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40334.494512] lpc_ich sha512_ssse3 [40334.498610] ? queued_spin_lock_slowpath+0x16/0x2e0 [40334.500702] acpi_ipmi mfd_core [40334.503149] do_raw_spin_lock+0xb5/0xc0 [40334.508452] wmi ipmi_si [40334.511764] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40334.516634] ipmi_devintf ipmi_msghandler ioatdma [40334.519773] try_get_cap_refs+0x159/0x880 [ceph] [40334.523601] acpi_pad acpi_power_meter [40334.526136] __ceph_get_caps+0x4a1/0x760 [ceph] [40334.530915] ixgbe igb [40334.535617] ? __file_remove_privs+0x89/0x110 [40334.540230] mdio [40334.543977] ? wake_bit_function+0x70/0x70 [40334.548511] crc32c_intel [40334.550871] ? ceph_write_iter+0x260/0x770 [ceph] [40334.555232] nvme [40334.557160] ceph_write_iter+0x260/0x770 [ceph] [40334.561251] ptp [40334.563872] ? lock_is_held_type+0xaa/0x120 [40334.568565] nvme_core [40334.570491] vfs_write+0x324/0x4f0 [40334.575012] i2c_algo_bit pps_core [40334.576853] ksys_pwrite64+0x65/0xa0 [40334.581027] dca fuse [40334.583385] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40334.586782] [40334.590180] do_syscall_64+0x38/0x80 [40334.593750] irq event stamp: 242827812 [40334.596022] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40334.601152] hardirqs last enabled at (242827811): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40334.602644] RIP: 0033:0x7f48f0eeb93b [40334.606231] hardirqs last disabled at (242827812): [] sysvec_apic_timer_interrupt+0xe/0x90 [40334.609975] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40334.615019] softirqs last enabled at (242827810): [] __do_softirq+0x2be/0x371 [40334.625263] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40334.628834] softirqs last disabled at (242827805): [] irq_exit_rcu+0xb0/0xd0 [40334.638645] ORIG_RAX: 0000000000000012 [40334.657382] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40334.666153] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40334.671369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40334.679969] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40334.683799] Workqueue: events delayed_work [ceph] [40334.693176] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40334.700301] [40334.707771] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40334.714896] RIP: 0010:lock_acquire+0xf0/0x2b0 [40334.719592] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40334.726716] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40334.728232] [40334.735348] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40334.772964] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40334.780086] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40334.787211] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40334.794353] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40334.801478] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40334.808602] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40334.816677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40334.822415] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40334.829540] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40334.836664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40334.843788] Call Trace: [40334.846238] [40334.848253] ? watchdog_timer_fn+0x1b4/0x220 [40334.852517] ? __hrtimer_run_queues+0x1d3/0x360 [40334.857043] ? hrtimer_interrupt+0x104/0x240 [40334.861313] ? __local_bh_enable+0x50/0x60 [40334.865405] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40334.870621] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40334.875579] [40334.877677] [40334.879774] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40334.885082] ? lock_acquire+0xf0/0x2b0 [40334.888832] ? igrab+0x43/0x50 [40334.891882] ? lock_release+0xc5/0x260 [40334.895627] _raw_spin_lock+0x30/0x40 [40334.899282] ? igrab+0x1d/0x50 [40334.902334] igrab+0x1d/0x50 [40334.905214] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40334.910387] delayed_work+0x1d8/0x280 [ceph] [40334.914668] ? process_scheduled_works+0x22a/0x510 [40334.919458] process_scheduled_works+0x2a4/0x510 [40334.924075] worker_thread+0x188/0x340 [40334.927824] ? rescuer_thread+0x310/0x310 [40334.931836] kthread+0xf5/0x130 [40334.934973] ? kthread_complete_and_exit+0x20/0x20 [40334.939757] ret_from_fork+0x31/0x40 [40334.943336] ? kthread_complete_and_exit+0x20/0x20 [40334.948121] ret_from_fork_asm+0x11/0x20 [40334.952042] [40337.099247] rcu: INFO: rcu_preempt self-detected stall on CPU [40337.105008] rcu: 7-....: (27573430 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/27572058 fqs=13903650 [40337.115694] rcu: (t=28399993 jiffies g=1309609 q=760504 ncpus=8) [40337.121779] Sending NMI from CPU 7 to CPUs 0: [40337.126132] NMI backtrace for cpu 0 [40337.126134] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40337.126135] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40337.126136] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40337.126139] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40337.126141] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40337.126142] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40337.126143] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40337.126143] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40337.126144] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40337.126145] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40337.126146] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40337.126147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40337.126148] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40337.126149] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40337.126149] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40337.126150] Call Trace: [40337.126151] [40337.126152] ? nmi_cpu_backtrace+0xad/0x120 [40337.126155] ? nmi_cpu_backtrace_handler+0x11/0x20 [40337.126157] ? nmi_handle+0xd4/0x260 [40337.126161] ? default_do_nmi+0x49/0xf0 [40337.126162] ? exc_nmi+0x11e/0x150 [40337.126164] ? end_repeat_nmi+0x16/0x67 [40337.126168] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40337.126170] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40337.126172] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40337.126175] [40337.126175] [40337.126176] do_raw_spin_lock+0xb5/0xc0 [40337.126178] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40337.126192] try_get_cap_refs+0x159/0x880 [ceph] [40337.126206] __ceph_get_caps+0x4a1/0x760 [ceph] [40337.126218] ? __file_remove_privs+0x89/0x110 [40337.126220] ? wake_bit_function+0x70/0x70 [40337.126222] ? ceph_write_iter+0x260/0x770 [ceph] [40337.126232] ceph_write_iter+0x260/0x770 [ceph] [40337.126243] ? lock_is_held_type+0xaa/0x120 [40337.126246] vfs_write+0x324/0x4f0 [40337.126252] ksys_pwrite64+0x65/0xa0 [40337.126271] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40337.126274] do_syscall_64+0x38/0x80 [40337.126276] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40337.126278] RIP: 0033:0x7f48f0eeb93b [40337.126280] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40337.126281] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [40337.126282] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40337.126283] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40337.126284] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40337.126284] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40337.126285] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40337.126288] [40337.127130] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40337.430654] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40337.438127] Workqueue: events delayed_work [ceph] [40337.442848] RIP: 0010:ceph_check_delayed_caps+0xa4/0x270 [ceph] [40337.448778] Code: 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 48 89 1b <48> 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 48 85 c0 [40337.467514] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [40337.472733] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [40337.479855] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [40337.486980] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [40337.494103] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [40337.501229] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40337.508352] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40337.516429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40337.522167] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40337.529291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40337.536415] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40337.543540] Call Trace: [40337.546002] [40337.548012] ? rcu_dump_cpu_stacks+0xd1/0x180 [40337.552375] ? rcu_sched_clock_irq+0x633/0x1390 [40337.556906] ? lock_release+0xc5/0x260 [40337.560658] ? lock_is_held_type+0xaa/0x120 [40337.564844] ? lock_is_held_type+0xaa/0x120 [40337.569022] ? tick_sched_do_timer+0xa0/0xa0 [40337.573295] ? rcu_read_lock_sched_held+0x56/0x90 [40337.578003] ? tick_sched_do_timer+0xa0/0xa0 [40337.582282] ? update_process_times+0x5f/0x90 [40337.586633] ? tick_sched_handle.isra.16+0x1e/0x50 [40337.591426] ? tick_sched_timer+0x6f/0x80 [40337.595439] ? __hrtimer_run_queues+0x1d3/0x360 [40337.599977] ? hrtimer_interrupt+0x104/0x240 [40337.604241] ? __local_bh_enable+0x50/0x60 [40337.608339] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40337.613571] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40337.618535] [40337.620650] [40337.622747] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40337.628068] ? ceph_check_delayed_caps+0xa4/0x270 [ceph] [40337.633393] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [40337.638716] delayed_work+0x1d8/0x280 [ceph] [40337.643006] ? process_scheduled_works+0x22a/0x510 [40337.647795] process_scheduled_works+0x2a4/0x510 [40337.652424] worker_thread+0x188/0x340 [40337.656178] ? rescuer_thread+0x310/0x310 [40337.660188] kthread+0xf5/0x130 [40337.663334] ? kthread_complete_and_exit+0x20/0x20 [40337.668128] ret_from_fork+0x31/0x40 [40337.671705] ? kthread_complete_and_exit+0x20/0x20 [40337.676490] ret_from_fork_asm+0x11/0x20 [40337.680429] [40362.159288] watchdog: BUG: soft lockup - CPU#0 stuck for 26476s! [dbench:78683] [40362.166591] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40362.171287] watchdog: BUG: soft lockup - CPU#7 stuck for 35473s! [kworker/7:4:78808] [40362.222040] irq event stamp: 71898918 [40362.229773] Modules linked in: ceph libceph dns_resolver fscache [40362.233429] hardirqs last enabled at (71898917): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40362.233434] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [40362.249592] hardirqs last disabled at (71898918): [] sysvec_apic_timer_interrupt+0xe/0x90 [40362.258798] nft_compat nf_tables nfnetlink bridge stp llc [40362.268521] softirqs last enabled at (71898838): [] __do_softirq+0x2be/0x371 [40362.273997] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [40362.282682] softirqs last disabled at (71898833): [] irq_exit_rcu+0xb0/0xd0 [40362.289546] x86_pkg_temp_thermal coretemp kvm_intel kvm [40362.298059] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40362.303361] irqbypass i2c_i801 [40362.312307] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40362.312307] iTCO_wdt iTCO_vendor_support [40362.315444] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40362.322915] i2c_smbus crct10dif_pclmul [40362.326919] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40362.332395] joydev [40362.336227] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40362.354972] crc32_pclmul ipmi_ssif [40362.357069] [40362.362287] ghash_clmulni_intel [40362.365770] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40362.367262] mei_me mei [40362.370487] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40362.377610] lpc_ich sha512_ssse3 [40362.380055] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40362.387178] acpi_ipmi mfd_core [40362.390489] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40362.397614] wmi [40362.400752] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40362.407877] ipmi_si [40362.409712] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40362.416836] ipmi_devintf [40362.419020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40362.427098] ipmi_msghandler [40362.429716] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40362.435453] ioatdma acpi_pad [40362.438331] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40362.445455] acpi_power_meter ixgbe [40362.448419] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40362.455544] igb mdio [40362.459027] Call Trace: [40362.459030] [40362.466149] crc32c_intel [40362.468425] ? watchdog_timer_fn+0x1b4/0x220 [40362.470867] nvme [40362.472878] ? __hrtimer_run_queues+0x1d3/0x360 [40362.475495] ptp [40362.479762] ? hrtimer_interrupt+0x104/0x240 [40362.481684] nvme_core i2c_algo_bit [40362.486205] ? __local_bh_enable+0x50/0x60 [40362.488043] pps_core dca [40362.492309] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40362.495791] fuse [40362.499884] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40362.502501] irq event stamp: 243043774 [40362.507717] [40362.509641] hardirqs last enabled at (243043773): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40362.514599] [40362.514600] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40362.518342] hardirqs last disabled at (243043774): [] sysvec_apic_timer_interrupt+0xe/0x90 [40362.520445] ? queued_spin_lock_slowpath+0x16/0x2e0 [40362.530687] softirqs last enabled at (243043772): [] __do_softirq+0x2be/0x371 [40362.532785] do_raw_spin_lock+0xb5/0xc0 [40362.538085] softirqs last disabled at (243043767): [] irq_exit_rcu+0xb0/0xd0 [40362.547898] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40362.552766] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40362.561541] try_get_cap_refs+0x159/0x880 [ceph] [40362.565367] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40362.573976] __ceph_get_caps+0x4a1/0x760 [ceph] [40362.578760] Workqueue: events delayed_work [ceph] [40362.588140] ? __file_remove_privs+0x89/0x110 [40362.592747] [40362.592748] RIP: 0010:lock_release+0x14e/0x260 [40362.600218] ? wake_bit_function+0x70/0x70 [40362.604742] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [40362.609443] ? ceph_write_iter+0x260/0x770 [ceph] [40362.613790] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [40362.615282] ceph_write_iter+0x260/0x770 [ceph] [40362.619717] [40362.623813] ? lock_is_held_type+0xaa/0x120 [40362.642547] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [40362.647246] vfs_write+0x324/0x4f0 [40362.652461] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [40362.656990] ksys_pwrite64+0x65/0xa0 [40362.658477] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [40362.662654] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40362.669776] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [40362.673177] do_syscall_64+0x38/0x80 [40362.680300] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [40362.683872] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40362.690993] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40362.696125] RIP: 0033:0x7f48f0eeb93b [40362.703248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40362.706822] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40362.713943] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40362.719006] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40362.727083] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40362.730654] ORIG_RAX: 0000000000000012 [40362.736390] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40362.755129] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40362.762252] Call Trace: [40362.762253] [40362.767470] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40362.774596] ? watchdog_timer_fn+0x1b4/0x220 [40362.778424] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40362.785551] ? __hrtimer_run_queues+0x1d3/0x360 [40362.792690] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40362.795139] ? hrtimer_interrupt+0x104/0x240 [40362.797145] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40362.804270] ? __local_bh_enable+0x50/0x60 [40362.808538] [40362.815661] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40362.850188] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40362.855144] [40362.857241] [40362.859341] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40362.864644] ? igrab+0x43/0x50 [40362.867698] ? lock_release+0x14e/0x260 [40362.871535] ? lock_release+0xc5/0x260 [40362.875279] _raw_spin_unlock+0x1b/0x40 [40362.879117] igrab+0x43/0x50 [40362.881995] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40362.887146] delayed_work+0x1d8/0x280 [ceph] [40362.891427] ? process_scheduled_works+0x22a/0x510 [40362.896216] process_scheduled_works+0x2a4/0x510 [40362.900831] worker_thread+0x188/0x340 [40362.904580] ? rescuer_thread+0x310/0x310 [40362.908583] kthread+0xf5/0x130 [40362.911724] ? kthread_complete_and_exit+0x20/0x20 [40362.916514] ret_from_fork+0x31/0x40 [40362.920095] ? kthread_complete_and_exit+0x20/0x20 [40362.924878] ret_from_fork_asm+0x11/0x20 [40362.928810] [40390.159741] watchdog: BUG: soft lockup - CPU#0 stuck for 26502s! [dbench:78683] [40390.167044] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40390.171741] watchdog: BUG: soft lockup - CPU#7 stuck for 35499s! [kworker/7:4:78808] [40390.222486] irq event stamp: 71957004 [40390.222487] hardirqs last enabled at (71957003): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40390.230217] Modules linked in: [40390.233873] hardirqs last disabled at (71957004): [] sysvec_apic_timer_interrupt+0xe/0x90 [40390.244031] ceph [40390.247082] softirqs last enabled at (71956926): [] __do_softirq+0x2be/0x371 [40390.256806] libceph [40390.258731] softirqs last disabled at (71956921): [] irq_exit_rcu+0xb0/0xd0 [40390.267424] dns_resolver fscache [40390.269606] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40390.278135] netfs [40390.281446] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40390.290389] veth nft_chain_nat [40390.292416] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40390.299889] xt_MASQUERADE nf_nat [40390.303025] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40390.308502] nf_conntrack nf_defrag_ipv6 [40390.311814] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40390.330551] nf_defrag_ipv4 nft_compat [40390.334468] [40390.334468] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40390.339687] nf_tables nfnetlink [40390.343430] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40390.344921] bridge stp [40390.352044] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40390.355271] llc [40390.362394] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40390.364838] xfs libcrc32c [40390.371961] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40390.373798] sunrpc intel_rapl_msr [40390.380924] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40390.383628] intel_rapl_common sb_edac [40390.390751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40390.394148] x86_pkg_temp_thermal coretemp [40390.402226] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40390.405971] kvm_intel kvm [40390.411708] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40390.415798] irqbypass i2c_i801 [40390.422923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40390.425626] iTCO_wdt iTCO_vendor_support [40390.432751] Call Trace: [40390.432752] [40390.435889] i2c_smbus crct10dif_pclmul [40390.443013] ? watchdog_timer_fn+0x1b4/0x220 [40390.447017] joydev [40390.449461] ? __hrtimer_run_queues+0x1d3/0x360 [40390.451470] crc32_pclmul ipmi_ssif [40390.455305] ? hrtimer_interrupt+0x104/0x240 [40390.459565] ghash_clmulni_intel mei_me [40390.461662] ? __local_bh_enable+0x50/0x60 [40390.466189] mei [40390.469675] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40390.473936] lpc_ich sha512_ssse3 [40390.477768] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40390.481856] acpi_ipmi mfd_core [40390.483695] [40390.488912] wmi ipmi_si [40390.492222] [40390.497179] ipmi_devintf [40390.500318] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40390.502418] ipmi_msghandler ioatdma [40390.504950] ? queued_spin_lock_slowpath+0x16/0x2e0 [40390.507042] acpi_pad acpi_power_meter [40390.509662] do_raw_spin_lock+0xb5/0xc0 [40390.514965] ixgbe igb [40390.518536] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40390.523420] mdio crc32c_intel [40390.527168] try_get_cap_refs+0x159/0x880 [ceph] [40390.530997] nvme ptp [40390.533360] __ceph_get_caps+0x4a1/0x760 [ceph] [40390.538138] nvme_core i2c_algo_bit [40390.541196] ? __file_remove_privs+0x89/0x110 [40390.545801] pps_core dca [40390.548071] ? wake_bit_function+0x70/0x70 [40390.552595] fuse [40390.556082] ? ceph_write_iter+0x260/0x770 [ceph] [40390.560431] irq event stamp: 243264776 [40390.563048] ceph_write_iter+0x260/0x770 [ceph] [40390.567143] hardirqs last enabled at (243264775): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40390.569068] ? lock_is_held_type+0xaa/0x120 [40390.573759] hardirqs last disabled at (243264776): [] sysvec_apic_timer_interrupt+0xe/0x90 [40390.577507] vfs_write+0x324/0x4f0 [40390.582029] softirqs last enabled at (243264774): [] __do_softirq+0x2be/0x371 [40390.592276] ksys_pwrite64+0x65/0xa0 [40390.596449] softirqs last disabled at (243264769): [] irq_exit_rcu+0xb0/0xd0 [40390.606261] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40390.609659] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40390.618431] do_syscall_64+0x38/0x80 [40390.622000] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40390.630599] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40390.635728] Workqueue: events delayed_work [ceph] [40390.645106] RIP: 0033:0x7f48f0eeb93b [40390.648676] [40390.648677] RIP: 0010:_raw_spin_lock+0x21/0x40 [40390.656147] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40390.661191] Code: cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 48 8d 7f 18 45 31 c9 31 c9 <41> b8 01 00 00 00 31 d2 31 f6 e8 30 0c 5c ff 48 89 df 58 5b e9 76 [40390.665889] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40390.669460] RSP: 0018:ffffc90002117d88 EFLAGS: 00000246 [40390.670949] ORIG_RAX: 0000000000000012 [40390.675388] [40390.675388] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: 0000000000000000 [40390.694141] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40390.712879] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34518 [40390.718097] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40390.723314] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000000 [40390.727146] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40390.728635] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [40390.735760] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40390.742885] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40390.750008] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40390.757144] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40390.764270] [40390.771391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40390.815886] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40390.823009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40390.830145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40390.837276] Call Trace: [40390.839720] [40390.841732] ? watchdog_timer_fn+0x1b4/0x220 [40390.846003] ? __hrtimer_run_queues+0x1d3/0x360 [40390.850531] ? hrtimer_interrupt+0x104/0x240 [40390.854800] ? __local_bh_enable+0x50/0x60 [40390.858894] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40390.864116] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40390.869090] [40390.871188] [40390.873286] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40390.878594] ? _raw_spin_lock+0x21/0x40 [40390.882431] ? igrab+0x1d/0x50 [40390.885481] igrab+0x1d/0x50 [40390.888359] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40390.893507] delayed_work+0x1d8/0x280 [ceph] [40390.897790] ? process_scheduled_works+0x22a/0x510 [40390.902579] process_scheduled_works+0x2a4/0x510 [40390.907196] worker_thread+0x188/0x340 [40390.910944] ? rescuer_thread+0x310/0x310 [40390.914948] kthread+0xf5/0x130 [40390.918085] ? kthread_complete_and_exit+0x20/0x20 [40390.922870] ret_from_fork+0x31/0x40 [40390.926451] ? kthread_complete_and_exit+0x20/0x20 [40390.931241] ret_from_fork_asm+0x11/0x20 [40390.935165] [40418.160196] watchdog: BUG: soft lockup - CPU#0 stuck for 26528s! [dbench:78683] [40418.167497] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40418.172195] watchdog: BUG: soft lockup - CPU#7 stuck for 35525s! [kworker/7:4:78808] [40418.222946] irq event stamp: 72015228 [40418.230678] Modules linked in: ceph libceph dns_resolver fscache [40418.234334] hardirqs last enabled at (72015227): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40418.234338] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [40418.250490] hardirqs last disabled at (72015228): [] sysvec_apic_timer_interrupt+0xe/0x90 [40418.258403] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge [40418.268126] softirqs last enabled at (72015152): [] __do_softirq+0x2be/0x371 [40418.268129] stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [40418.282895] softirqs last disabled at (72015147): [] irq_exit_rcu+0xb0/0xd0 [40418.289758] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [40418.298271] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40418.304268] irqbypass i2c_i801 [40418.313212] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40418.313214] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40418.316349] iTCO_wdt iTCO_vendor_support [40418.323821] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40418.329296] i2c_smbus crct10dif_pclmul [40418.333302] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40418.352039] joydev crc32_pclmul [40418.355871] [40418.355872] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40418.361086] ipmi_ssif ghash_clmulni_intel [40418.364312] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40418.365803] mei_me mei [40418.372926] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40418.377018] lpc_ich sha512_ssse3 [40418.384142] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40418.386591] acpi_ipmi mfd_core [40418.393718] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40418.397029] wmi ipmi_si [40418.404152] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40418.407291] ipmi_devintf [40418.414414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40418.416945] ipmi_msghandler ioatdma [40418.425023] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40418.427641] acpi_pad acpi_power_meter [40418.433378] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40418.436949] ixgbe igb [40418.444072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40418.447817] mdio crc32c_intel [40418.454940] Call Trace: [40418.454942] [40418.457298] nvme ptp [40418.464425] ? watchdog_timer_fn+0x1b4/0x220 [40418.467472] nvme_core i2c_algo_bit [40418.469918] ? __hrtimer_run_queues+0x1d3/0x360 [40418.471926] pps_core dca [40418.474201] ? hrtimer_interrupt+0x104/0x240 [40418.478462] fuse [40418.481945] ? __local_bh_enable+0x50/0x60 [40418.486469] irq event stamp: 243485570 [40418.489089] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40418.493350] hardirqs last enabled at (243485569): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40418.495276] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40418.499365] hardirqs last disabled at (243485570): [] sysvec_apic_timer_interrupt+0xe/0x90 [40418.503110] [40418.508327] softirqs last enabled at (243485568): [] __do_softirq+0x2be/0x371 [40418.518571] [40418.523547] softirqs last disabled at (243485563): [] irq_exit_rcu+0xb0/0xd0 [40418.533359] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40418.535456] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40418.544231] ? queued_spin_lock_slowpath+0x18/0x2e0 [40418.546323] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40418.546325] Workqueue: events delayed_work [ceph] [40418.554922] do_raw_spin_lock+0xb5/0xc0 [40418.560224] [40418.560225] RIP: 0010:lock_acquire+0xf0/0x2b0 [40418.569605] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40418.574472] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40418.581946] try_get_cap_refs+0x159/0x880 [ceph] [40418.586642] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40418.590477] __ceph_get_caps+0x4a1/0x760 [ceph] [40418.591963] [40418.591963] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40418.596319] ? __file_remove_privs+0x89/0x110 [40418.601098] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40418.619836] ? wake_bit_function+0x70/0x70 [40418.624446] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40418.629666] ? ceph_write_iter+0x260/0x770 [ceph] [40418.634187] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40418.635679] ceph_write_iter+0x260/0x770 [ceph] [40418.642801] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40418.647157] ? lock_is_held_type+0xaa/0x120 [40418.654276] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40418.658371] vfs_write+0x324/0x4f0 [40418.665492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40418.670195] ksys_pwrite64+0x65/0xa0 [40418.677314] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40418.681840] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40418.688962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40418.693140] do_syscall_64+0x38/0x80 [40418.701217] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40418.704616] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40418.710351] Call Trace: [40418.710353] [40418.713923] RIP: 0033:0x7f48f0eeb93b [40418.721048] ? watchdog_timer_fn+0x1b4/0x220 [40418.726178] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40418.733302] ? __hrtimer_run_queues+0x1d3/0x360 [40418.736872] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40418.743999] ? hrtimer_interrupt+0x104/0x240 [40418.749039] ORIG_RAX: 0000000000000012 [40418.751485] ? __local_bh_enable+0x50/0x60 [40418.753495] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40418.757069] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40418.761329] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40418.780069] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40418.784601] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40418.789817] [40418.794081] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40418.797911] [40418.797913] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40418.802002] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40418.809132] ? lock_acquire+0xf0/0x2b0 [40418.814351] [40418.821469] ? igrab+0x43/0x50 [40418.866294] ? lock_release+0xc5/0x260 [40418.870040] _raw_spin_lock+0x30/0x40 [40418.873697] ? igrab+0x1d/0x50 [40418.876746] igrab+0x1d/0x50 [40418.879625] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40418.884775] delayed_work+0x1d8/0x280 [ceph] [40418.889057] ? process_scheduled_works+0x22a/0x510 [40418.893845] process_scheduled_works+0x2a4/0x510 [40418.898461] worker_thread+0x188/0x340 [40418.902210] ? rescuer_thread+0x310/0x310 [40418.906214] kthread+0xf5/0x130 [40418.909362] ? kthread_complete_and_exit+0x20/0x20 [40418.914153] ret_from_fork+0x31/0x40 [40418.917732] ? kthread_complete_and_exit+0x20/0x20 [40418.922517] ret_from_fork_asm+0x11/0x20 [40418.926448] [40446.160655] watchdog: BUG: soft lockup - CPU#0 stuck for 26554s! [dbench:78683] [40446.167957] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40446.172655] watchdog: BUG: soft lockup - CPU#7 stuck for 35551s! [kworker/7:4:78808] [40446.223400] irq event stamp: 72073302 [40446.223402] hardirqs last enabled at (72073301): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40446.231131] Modules linked in: ceph [40446.234787] hardirqs last disabled at (72073302): [] sysvec_apic_timer_interrupt+0xe/0x90 [40446.244946] libceph [40446.248429] softirqs last enabled at (72073224): [] __do_softirq+0x2be/0x371 [40446.258154] dns_resolver fscache [40446.260337] softirqs last disabled at (72073219): [] irq_exit_rcu+0xb0/0xd0 [40446.269023] netfs veth [40446.272352] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40446.280861] nft_chain_nat [40446.283304] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40446.292266] xt_MASQUERADE nf_nat [40446.294970] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [40446.302442] nf_conntrack [40446.305752] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [40446.311228] nf_defrag_ipv6 nf_defrag_ipv4 [40446.313846] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40446.332584] nft_compat nf_tables [40446.336674] [40446.336675] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40446.341892] nfnetlink bridge [40446.345203] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40446.346692] stp llc [40446.353817] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40446.356783] xfs libcrc32c [40446.363907] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40446.366090] sunrpc intel_rapl_msr [40446.373214] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40446.375919] intel_rapl_common [40446.383050] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40446.386448] sb_edac x86_pkg_temp_thermal [40446.393573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40446.396623] coretemp kvm_intel [40446.404701] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40446.408705] kvm irqbypass [40446.414441] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40446.417580] i2c_i801 iTCO_wdt [40446.424704] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40446.427407] iTCO_vendor_support [40446.434531] Call Trace: [40446.434533] [40446.437582] i2c_smbus crct10dif_pclmul [40446.444708] ? watchdog_timer_fn+0x1b4/0x220 [40446.447930] joydev crc32_pclmul [40446.450377] ? __hrtimer_run_queues+0x1d3/0x360 [40446.452385] ipmi_ssif ghash_clmulni_intel [40446.456219] ? hrtimer_interrupt+0x104/0x240 [40446.460480] mei_me mei [40446.463704] ? __local_bh_enable+0x50/0x60 [40446.468229] lpc_ich sha512_ssse3 [40446.472322] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40446.476583] acpi_ipmi mfd_core [40446.479027] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40446.483134] wmi ipmi_si [40446.486445] [40446.491665] ipmi_devintf ipmi_msghandler [40446.494801] [40446.499759] ioatdma [40446.502290] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40446.504386] acpi_pad acpi_power_meter [40446.508394] ? queued_spin_lock_slowpath+0x2b/0x2e0 [40446.510488] ixgbe igb [40446.512674] do_raw_spin_lock+0xb5/0xc0 [40446.517977] mdio crc32c_intel nvme [40446.521722] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40446.526591] ptp [40446.528952] try_get_cap_refs+0x159/0x880 [ceph] [40446.532779] nvme_core i2c_algo_bit [40446.536268] __ceph_get_caps+0x4a1/0x760 [ceph] [40446.541053] pps_core dca [40446.542896] ? __file_remove_privs+0x89/0x110 [40446.547504] fuse [40446.550989] ? wake_bit_function+0x70/0x70 [40446.555512] irq event stamp: 243706552 [40446.558133] ? ceph_write_iter+0x260/0x770 [ceph] [40446.562480] hardirqs last enabled at (243706551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40446.564406] ceph_write_iter+0x260/0x770 [ceph] [40446.568496] hardirqs last disabled at (243706552): [] sysvec_apic_timer_interrupt+0xe/0x90 [40446.572244] ? lock_is_held_type+0xaa/0x120 [40446.576935] softirqs last enabled at (243706550): [] __do_softirq+0x2be/0x371 [40446.587182] vfs_write+0x324/0x4f0 [40446.591705] softirqs last disabled at (243706545): [] irq_exit_rcu+0xb0/0xd0 [40446.601521] ksys_pwrite64+0x65/0xa0 [40446.605694] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40446.614463] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40446.617862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40446.617863] Workqueue: events delayed_work [ceph] [40446.626461] do_syscall_64+0x38/0x80 [40446.630031] [40446.639425] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40446.644554] RIP: 0010:lock_acquire+0xf0/0x2b0 [40446.652025] RIP: 0033:0x7f48f0eeb93b [40446.656723] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40446.660296] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40446.661786] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40446.666829] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40446.671179] [40446.671179] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40446.674750] ORIG_RAX: 0000000000000012 [40446.693487] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40446.712227] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40446.717443] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40446.722659] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40446.724150] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40446.731275] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40446.735106] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40446.742230] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40446.749354] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40446.756478] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40446.763603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40446.770731] [40446.777850] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40446.822364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40446.829487] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40446.836612] Call Trace: [40446.839060] [40446.841076] ? watchdog_timer_fn+0x1b4/0x220 [40446.845341] ? __hrtimer_run_queues+0x1d3/0x360 [40446.849867] ? hrtimer_interrupt+0x104/0x240 [40446.854136] ? __local_bh_enable+0x50/0x60 [40446.858230] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40446.863455] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40446.868411] [40446.870508] [40446.872607] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40446.877914] ? lock_acquire+0xf0/0x2b0 [40446.881665] ? igrab+0x43/0x50 [40446.884713] ? lock_release+0xc5/0x260 [40446.888459] _raw_spin_lock+0x30/0x40 [40446.892125] ? igrab+0x1d/0x50 [40446.895173] igrab+0x1d/0x50 [40446.898063] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40446.903212] delayed_work+0x1d8/0x280 [ceph] [40446.907492] ? process_scheduled_works+0x22a/0x510 [40446.912283] process_scheduled_works+0x2a4/0x510 [40446.916896] worker_thread+0x188/0x340 [40446.920648] ? rescuer_thread+0x310/0x310 [40446.924660] kthread+0xf5/0x130 [40446.927797] ? kthread_complete_and_exit+0x20/0x20 [40446.932580] ret_from_fork+0x31/0x40 [40446.936152] ? kthread_complete_and_exit+0x20/0x20 [40446.940934] ret_from_fork_asm+0x11/0x20 [40446.944858] [40474.161116] watchdog: BUG: soft lockup - CPU#0 stuck for 26580s! [dbench:78683] [40474.168419] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40474.173116] watchdog: BUG: soft lockup - CPU#7 stuck for 35577s! [kworker/7:4:78808] [40474.223861] irq event stamp: 72131482 [40474.223863] hardirqs last enabled at (72131481): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40474.231592] Modules linked in: [40474.235249] hardirqs last disabled at (72131482): [] sysvec_apic_timer_interrupt+0xe/0x90 [40474.245405] ceph [40474.248456] softirqs last enabled at (72131416): [] __do_softirq+0x2be/0x371 [40474.258182] libceph [40474.260104] softirqs last disabled at (72131411): [] irq_exit_rcu+0xb0/0xd0 [40474.268798] dns_resolver [40474.270982] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40474.279510] fscache netfs [40474.282127] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40474.291072] veth nft_chain_nat [40474.293776] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40474.301263] xt_MASQUERADE [40474.304401] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40474.309878] nf_nat [40474.312584] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40474.331321] nf_conntrack nf_defrag_ipv6 [40474.333417] [40474.333418] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40474.338636] nf_defrag_ipv4 nft_compat [40474.342552] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40474.344044] nf_tables [40474.351168] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40474.354912] nfnetlink bridge [40474.362036] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40474.364393] stp llc [40474.371516] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40474.374481] xfs libcrc32c [40474.381606] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40474.383791] sunrpc [40474.390914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40474.393618] intel_rapl_msr intel_rapl_common [40474.401695] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40474.403794] sb_edac x86_pkg_temp_thermal [40474.409530] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40474.413881] coretemp kvm_intel [40474.421005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40474.425010] kvm [40474.432133] Call Trace: [40474.432134] [40474.435271] irqbypass [40474.442394] ? watchdog_timer_fn+0x1b4/0x220 [40474.444232] i2c_i801 iTCO_wdt [40474.446678] ? __hrtimer_run_queues+0x1d3/0x360 [40474.448688] iTCO_vendor_support i2c_smbus [40474.451047] ? hrtimer_interrupt+0x104/0x240 [40474.455308] crct10dif_pclmul joydev [40474.458358] ? __local_bh_enable+0x50/0x60 [40474.462884] crc32_pclmul ipmi_ssif [40474.466975] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40474.471239] ghash_clmulni_intel mei_me [40474.474809] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40474.478899] mei lpc_ich [40474.482383] [40474.487601] sha512_ssse3 acpi_ipmi [40474.491433] [40474.491434] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40474.496389] mfd_core wmi [40474.498925] ? queued_spin_lock_slowpath+0x16/0x2e0 [40474.501016] ipmi_si ipmi_devintf [40474.504517] do_raw_spin_lock+0xb5/0xc0 [40474.506617] ipmi_msghandler ioatdma [40474.511921] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40474.514537] acpi_pad acpi_power_meter [40474.519409] try_get_cap_refs+0x159/0x880 [ceph] [40474.522720] ixgbe igb [40474.526553] __ceph_get_caps+0x4a1/0x760 [ceph] [40474.530121] mdio crc32c_intel [40474.534909] ? __file_remove_privs+0x89/0x110 [40474.538647] nvme ptp [40474.543261] ? wake_bit_function+0x70/0x70 [40474.545617] nvme_core i2c_algo_bit [40474.550143] ? ceph_write_iter+0x260/0x770 [ceph] [40474.553191] pps_core dca [40474.557543] ceph_write_iter+0x260/0x770 [ceph] [40474.559814] fuse [40474.563907] ? lock_is_held_type+0xaa/0x120 [40474.567387] [40474.572088] vfs_write+0x324/0x4f0 [40474.574702] irq event stamp: 243927260 [40474.579232] ksys_pwrite64+0x65/0xa0 [40474.581151] hardirqs last enabled at (243927259): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40474.585329] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40474.586819] hardirqs last disabled at (243927260): [] sysvec_apic_timer_interrupt+0xe/0x90 [40474.590219] do_syscall_64+0x38/0x80 [40474.593961] softirqs last enabled at (243927258): [] __do_softirq+0x2be/0x371 [40474.597532] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40474.607775] softirqs last disabled at (243927253): [] irq_exit_rcu+0xb0/0xd0 [40474.612906] RIP: 0033:0x7f48f0eeb93b [40474.622717] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40474.626289] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40474.635058] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40474.640103] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40474.648702] Workqueue: events delayed_work [ceph] [40474.652270] ORIG_RAX: 0000000000000012 [40474.661648] [40474.680385] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40474.687856] RIP: 0010:lock_acquire+0xf0/0x2b0 [40474.693075] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40474.697770] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40474.701603] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40474.703093] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40474.710216] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40474.714568] [40474.714569] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40474.721691] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40474.740431] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40474.747560] [40474.752771] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40474.792076] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40474.799201] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40474.806325] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40474.814402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40474.820140] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40474.827264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40474.834387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40474.841519] Call Trace: [40474.843966] [40474.845976] ? watchdog_timer_fn+0x1b4/0x220 [40474.850241] ? __hrtimer_run_queues+0x1d3/0x360 [40474.854775] ? hrtimer_interrupt+0x104/0x240 [40474.859053] ? __local_bh_enable+0x50/0x60 [40474.863147] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40474.868369] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40474.873328] [40474.875424] [40474.877523] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40474.882840] ? lock_acquire+0xf0/0x2b0 [40474.886588] ? igrab+0x43/0x50 [40474.889638] ? lock_release+0xc5/0x260 [40474.893385] _raw_spin_lock+0x30/0x40 [40474.897048] ? igrab+0x1d/0x50 [40474.900100] igrab+0x1d/0x50 [40474.902977] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40474.908128] delayed_work+0x1d8/0x280 [ceph] [40474.912407] ? process_scheduled_works+0x22a/0x510 [40474.917191] process_scheduled_works+0x2a4/0x510 [40474.921809] worker_thread+0x188/0x340 [40474.925554] ? rescuer_thread+0x310/0x310 [40474.929557] kthread+0xf5/0x130 [40474.932696] ? kthread_complete_and_exit+0x20/0x20 [40474.937480] ret_from_fork+0x31/0x40 [40474.941049] ? kthread_complete_and_exit+0x20/0x20 [40474.945833] ret_from_fork_asm+0x11/0x20 [40474.949759] [40502.161580] watchdog: BUG: soft lockup - CPU#0 stuck for 26606s! [dbench:78683] [40502.168881] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40502.173580] watchdog: BUG: soft lockup - CPU#7 stuck for 35603s! [kworker/7:4:78808] [40502.224331] irq event stamp: 72189700 [40502.232061] Modules linked in: ceph libceph dns_resolver [40502.235719] hardirqs last enabled at (72189699): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40502.241022] fscache netfs veth [40502.251182] hardirqs last disabled at (72189700): [] sysvec_apic_timer_interrupt+0xe/0x90 [40502.254318] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [40502.264042] softirqs last enabled at (72189622): [] __do_softirq+0x2be/0x371 [40502.269692] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [40502.278377] softirqs last disabled at (72189617): [] irq_exit_rcu+0xb0/0xd0 [40502.278379] nf_tables nfnetlink bridge stp llc [40502.291934] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40502.296457] xfs libcrc32c [40502.305400] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40502.308103] sunrpc intel_rapl_msr intel_rapl_common sb_edac [40502.315575] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40502.315577] x86_pkg_temp_thermal coretemp kvm_intel kvm [40502.326704] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40502.332008] irqbypass i2c_i801 [40502.350744] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40502.353882] iTCO_wdt [40502.359100] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40502.359101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40502.361371] iTCO_vendor_support i2c_smbus [40502.368494] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40502.375619] crct10dif_pclmul joydev [40502.379709] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40502.386833] crc32_pclmul ipmi_ssif [40502.390405] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40502.397529] ghash_clmulni_intel mei_me [40502.401012] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40502.408138] mei lpc_ich [40502.411977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40502.420053] sha512_ssse3 [40502.422585] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40502.428322] acpi_ipmi mfd_core [40502.430939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40502.438063] wmi ipmi_si [40502.441200] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40502.448325] ipmi_devintf ipmi_msghandler [40502.450855] Call Trace: [40502.450856] [40502.457978] ioatdma acpi_pad [40502.461984] ? watchdog_timer_fn+0x1b4/0x220 [40502.464429] acpi_power_meter [40502.466439] ? __hrtimer_run_queues+0x1d3/0x360 [40502.469403] ixgbe igb [40502.473669] ? hrtimer_interrupt+0x104/0x240 [40502.476632] mdio crc32c_intel [40502.481154] ? __local_bh_enable+0x50/0x60 [40502.483514] nvme [40502.487780] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40502.490827] ptp nvme_core [40502.494917] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40502.496842] i2c_algo_bit pps_core [40502.502060] [40502.504764] dca fuse [40502.509721] [40502.509723] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40502.513118] [40502.513119] irq event stamp: 244148104 [40502.515221] ? queued_spin_lock_slowpath+0x16/0x2e0 [40502.517486] hardirqs last enabled at (244148103): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40502.519585] do_raw_spin_lock+0xb5/0xc0 [40502.524889] hardirqs last disabled at (244148104): [] sysvec_apic_timer_interrupt+0xe/0x90 [40502.526381] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40502.530122] softirqs last enabled at (244148102): [] __do_softirq+0x2be/0x371 [40502.534995] try_get_cap_refs+0x159/0x880 [ceph] [40502.545237] softirqs last disabled at (244148097): [] irq_exit_rcu+0xb0/0xd0 [40502.549093] __ceph_get_caps+0x4a1/0x760 [ceph] [40502.558897] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40502.563685] ? __file_remove_privs+0x89/0x110 [40502.572450] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40502.577061] ? wake_bit_function+0x70/0x70 [40502.585660] Workqueue: events delayed_work [ceph] [40502.590185] ? ceph_write_iter+0x260/0x770 [ceph] [40502.599562] [40502.599563] RIP: 0010:lock_acquire+0xf0/0x2b0 [40502.603913] ceph_write_iter+0x260/0x770 [ceph] [40502.611383] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40502.615478] ? lock_is_held_type+0xaa/0x120 [40502.620171] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40502.624870] vfs_write+0x324/0x4f0 [40502.626358] [40502.626359] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40502.630713] ksys_pwrite64+0x65/0xa0 [40502.635234] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40502.653981] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40502.658158] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40502.663375] do_syscall_64+0x38/0x80 [40502.666772] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40502.668280] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40502.675404] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40502.678982] RIP: 0033:0x7f48f0eeb93b [40502.686109] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40502.691240] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40502.698363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40502.701933] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40502.709057] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40502.714102] ORIG_RAX: 0000000000000012 [40502.721226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40502.724797] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40502.732874] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40502.751611] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40502.757350] Call Trace: [40502.757351] [40502.762566] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40502.769691] ? watchdog_timer_fn+0x1b4/0x220 [40502.773522] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40502.780648] ? __hrtimer_run_queues+0x1d3/0x360 [40502.787771] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40502.794898] ? hrtimer_interrupt+0x104/0x240 [40502.802024] [40502.804462] ? __local_bh_enable+0x50/0x60 [40502.847177] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40502.852400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40502.857356] [40502.859454] [40502.861551] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40502.866860] ? lock_acquire+0xf0/0x2b0 [40502.870610] ? igrab+0x43/0x50 [40502.873658] ? lock_release+0xc5/0x260 [40502.877405] _raw_spin_lock+0x30/0x40 [40502.881069] ? igrab+0x1d/0x50 [40502.884119] igrab+0x1d/0x50 [40502.886998] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40502.892147] delayed_work+0x1d8/0x280 [ceph] [40502.896429] ? process_scheduled_works+0x22a/0x510 [40502.901218] process_scheduled_works+0x2a4/0x510 [40502.905833] worker_thread+0x188/0x340 [40502.909583] ? rescuer_thread+0x310/0x310 [40502.913586] kthread+0xf5/0x130 [40502.916726] ? kthread_complete_and_exit+0x20/0x20 [40502.921516] ret_from_fork+0x31/0x40 [40502.925086] ? kthread_complete_and_exit+0x20/0x20 [40502.929873] ret_from_fork_asm+0x11/0x20 [40502.933804] [40530.162046] watchdog: BUG: soft lockup - CPU#0 stuck for 26632s! [dbench:78683] [40530.169351] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40530.174046] watchdog: BUG: soft lockup - CPU#7 stuck for 35629s! [kworker/7:4:78808] [40530.224792] irq event stamp: 72247818 [40530.232523] Modules linked in: ceph libceph dns_resolver [40530.236181] hardirqs last enabled at (72247817): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40530.241485] fscache netfs veth [40530.251642] hardirqs last disabled at (72247818): [] sysvec_apic_timer_interrupt+0xe/0x90 [40530.254779] nft_chain_nat xt_MASQUERADE nf_nat [40530.264502] softirqs last enabled at (72247792): [] __do_softirq+0x2be/0x371 [40530.269027] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [40530.277721] softirqs last disabled at (72247787): [] irq_exit_rcu+0xb0/0xd0 [40530.277722] nft_compat [40530.282938] nf_tables nfnetlink [40530.291449] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40530.293893] bridge [40530.297116] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40530.306061] stp llc [40530.308159] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40530.315628] xfs libcrc32c [40530.317813] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40530.323290] sunrpc intel_rapl_msr [40530.325994] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40530.344733] intel_rapl_common sb_edac [40530.348130] [40530.348131] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40530.353347] x86_pkg_temp_thermal [40530.357091] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40530.358581] coretemp kvm_intel [40530.365723] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40530.369034] kvm irqbypass [40530.376159] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40530.379297] i2c_i801 iTCO_wdt [40530.386420] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40530.389142] iTCO_vendor_support i2c_smbus [40530.396264] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40530.399317] crct10dif_pclmul joydev [40530.406443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40530.410530] crc32_pclmul [40530.418607] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40530.422180] ipmi_ssif ghash_clmulni_intel [40530.427916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40530.430533] mei_me mei [40530.437659] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40530.441750] lpc_ich sha512_ssse3 [40530.448873] Call Trace: [40530.448874] [40530.451318] acpi_ipmi mfd_core [40530.458444] ? watchdog_timer_fn+0x1b4/0x220 [40530.461751] wmi [40530.464197] ? __hrtimer_run_queues+0x1d3/0x360 [40530.466206] ipmi_si ipmi_devintf [40530.469347] ? hrtimer_interrupt+0x104/0x240 [40530.473607] ipmi_msghandler ioatdma [40530.475445] ? __local_bh_enable+0x50/0x60 [40530.479969] acpi_pad acpi_power_meter [40530.483283] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40530.487545] ixgbe igb [40530.491115] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40530.495206] mdio crc32c_intel [40530.498950] [40530.504167] nvme ptp [40530.506525] [40530.506526] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40530.511482] nvme_core i2c_algo_bit [40530.514537] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40530.516630] pps_core dca [40530.518904] do_raw_spin_lock+0xb5/0xc0 [40530.520999] fuse [40530.526302] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40530.529785] irq event stamp: 244368922 [40530.534660] try_get_cap_refs+0x159/0x880 [ceph] [40530.537276] hardirqs last enabled at (244368921): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40530.541109] __ceph_get_caps+0x4a1/0x760 [ceph] [40530.543030] hardirqs last disabled at (244368922): [] sysvec_apic_timer_interrupt+0xe/0x90 [40530.547818] ? __file_remove_privs+0x89/0x110 [40530.551558] softirqs last enabled at (244368920): [] __do_softirq+0x2be/0x371 [40530.556169] ? wake_bit_function+0x70/0x70 [40530.566412] softirqs last disabled at (244368915): [] irq_exit_rcu+0xb0/0xd0 [40530.570938] ? ceph_write_iter+0x260/0x770 [ceph] [40530.580747] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40530.585098] ceph_write_iter+0x260/0x770 [ceph] [40530.593868] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40530.597963] ? lock_is_held_type+0xaa/0x120 [40530.606557] Workqueue: events delayed_work [ceph] [40530.611257] vfs_write+0x324/0x4f0 [40530.620632] [40530.620633] RIP: 0010:rcu_is_watching+0x1f/0x40 [40530.625161] ksys_pwrite64+0x65/0xa0 [40530.632628] Code: 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 <83> e0 01 65 ff 0d 7f 5b e9 7e 74 01 c3 0f 1f 44 00 00 c3 66 66 2e [40530.636806] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40530.641500] RSP: 0018:ffffc90002117d10 EFLAGS: 00000202 [40530.644899] do_syscall_64+0x38/0x80 [40530.646391] [40530.646392] RAX: 00000000007c1919 RBX: ffff8881a1d34500 RCX: 0000000000000000 [40530.650914] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40530.654484] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881a1d34518 [40530.673224] RIP: 0033:0x7f48f0eeb93b [40530.678352] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [40530.683571] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40530.687140] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [40530.688632] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40530.695756] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40530.700801] ORIG_RAX: 0000000000000012 [40530.707923] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40530.711496] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40530.718620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40530.737357] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40530.744480] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40530.749699] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40530.756822] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40530.760653] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40530.768746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40530.768747] Call Trace: [40530.775873] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40530.781611] [40530.788754] [40530.795877] ? watchdog_timer_fn+0x1b4/0x220 [40530.842401] ? __hrtimer_run_queues+0x1d3/0x360 [40530.846927] ? hrtimer_interrupt+0x104/0x240 [40530.851195] ? __local_bh_enable+0x50/0x60 [40530.855288] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40530.860505] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40530.865463] [40530.867558] [40530.869657] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40530.874966] ? rcu_is_watching+0x1f/0x40 [40530.878887] lock_acquire+0x184/0x2b0 [40530.882544] ? igrab+0x43/0x50 [40530.885595] ? lock_release+0xc5/0x260 [40530.889342] _raw_spin_lock+0x30/0x40 [40530.893004] ? igrab+0x1d/0x50 [40530.896056] igrab+0x1d/0x50 [40530.898935] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40530.904083] delayed_work+0x1d8/0x280 [ceph] [40530.908366] ? process_scheduled_works+0x22a/0x510 [40530.913156] process_scheduled_works+0x2a4/0x510 [40530.917771] worker_thread+0x188/0x340 [40530.921521] ? rescuer_thread+0x310/0x310 [40530.925522] kthread+0xf5/0x130 [40530.928661] ? kthread_complete_and_exit+0x20/0x20 [40530.933454] ret_from_fork+0x31/0x40 [40530.937031] ? kthread_complete_and_exit+0x20/0x20 [40530.941816] ret_from_fork_asm+0x11/0x20 [40530.945740] [40532.685454] rcu: INFO: rcu_preempt self-detected stall on CPU [40532.691216] rcu: 0-....: (27949231 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=13999542 [40532.701720] rcu: (t=28595577 jiffies g=1309609 q=764276 ncpus=8) [40532.707805] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40532.716746] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40532.724220] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40532.729704] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40532.748442] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40532.753658] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40532.760783] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40532.767909] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40532.775032] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40532.782157] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40532.789280] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40532.797357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40532.803095] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40532.810220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40532.817344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40532.824484] Call Trace: [40532.826930] [40532.828942] ? rcu_dump_cpu_stacks+0xd1/0x180 [40532.833299] ? rcu_sched_clock_irq+0x633/0x1390 [40532.837824] ? lock_is_held_type+0xaa/0x120 [40532.842002] ? lock_is_held_type+0xaa/0x120 [40532.846179] ? tick_sched_do_timer+0xa0/0xa0 [40532.850450] ? rcu_read_lock_sched_held+0x56/0x90 [40532.855149] ? tick_sched_do_timer+0xa0/0xa0 [40532.859412] ? update_process_times+0x5f/0x90 [40532.863772] ? tick_sched_handle.isra.16+0x1e/0x50 [40532.868557] ? tick_sched_timer+0x6f/0x80 [40532.872561] ? __hrtimer_run_queues+0x1d3/0x360 [40532.877088] ? hrtimer_interrupt+0x104/0x240 [40532.881357] ? __local_bh_enable+0x50/0x60 [40532.885450] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40532.890675] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40532.895632] [40532.897729] [40532.899826] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40532.905133] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40532.910011] do_raw_spin_lock+0xb5/0xc0 [40532.913841] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40532.918644] try_get_cap_refs+0x159/0x880 [ceph] [40532.923273] __ceph_get_caps+0x4a1/0x760 [ceph] [40532.927815] ? __file_remove_privs+0x89/0x110 [40532.932172] ? wake_bit_function+0x70/0x70 [40532.936265] ? ceph_write_iter+0x260/0x770 [ceph] [40532.940977] ceph_write_iter+0x260/0x770 [ceph] [40532.945513] ? lock_is_held_type+0xaa/0x120 [40532.949698] vfs_write+0x324/0x4f0 [40532.953104] ksys_pwrite64+0x65/0xa0 [40532.956681] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40532.961812] do_syscall_64+0x38/0x80 [40532.965382] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40532.970427] RIP: 0033:0x7f48f0eeb93b [40532.973995] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40532.992734] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [40533.000290] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40533.007416] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40533.014540] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40533.021664] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40533.028788] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40533.035919] [40533.038106] Sending NMI from CPU 0 to CPUs 7: [40533.042458] NMI backtrace for cpu 7 [40533.042460] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40533.042462] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40533.042464] Workqueue: events delayed_work [ceph] [40533.042502] RIP: 0010:__lock_acquire+0x276/0x1e20 [40533.042504] Code: 00 44 0f b6 b8 c9 00 00 00 8b bb 50 0a 00 00 45 84 ff 45 0f 44 fd 41 89 fd 41 83 ed 01 0f 88 34 09 00 00 49 63 c5 48 8b 14 24 <48> 8d 04 80 0f b6 4c c2 21 31 f1 89 c8 a8 60 0f 85 18 09 00 00 8d [40533.042506] RSP: 0018:ffffc90002117c60 EFLAGS: 00000002 [40533.042508] RAX: 0000000000000002 RBX: ffff88810debaa80 RCX: 0000000000000004 [40533.042509] RDX: ffff88810debb4d8 RSI: 0000000000000006 RDI: 0000000000000003 [40533.042511] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [40533.042512] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [40533.042513] R13: 0000000000000002 R14: 0000000000000006 R15: 0000000000000002 [40533.042514] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40533.042516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40533.042517] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40533.042519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40533.042520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40533.042521] Call Trace: [40533.042522] [40533.042523] ? nmi_cpu_backtrace+0xad/0x120 [40533.042527] ? nmi_cpu_backtrace_handler+0x11/0x20 [40533.042529] ? nmi_handle+0xd4/0x260 [40533.042535] ? default_do_nmi+0x49/0xf0 [40533.042537] ? exc_nmi+0x11e/0x150 [40533.042540] ? end_repeat_nmi+0x16/0x67 [40533.042546] ? __lock_acquire+0x276/0x1e20 [40533.042548] ? __lock_acquire+0x276/0x1e20 [40533.042551] ? __lock_acquire+0x276/0x1e20 [40533.042553] [40533.042554] [40533.042558] lock_acquire+0xcb/0x2b0 [40533.042560] ? igrab+0x1d/0x50 [40533.042562] ? igrab+0x43/0x50 [40533.042565] ? lock_release+0xc5/0x260 [40533.042568] _raw_spin_lock+0x30/0x40 [40533.042570] ? igrab+0x1d/0x50 [40533.042572] igrab+0x1d/0x50 [40533.042574] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40533.042594] delayed_work+0x1d8/0x280 [ceph] [40533.042614] ? process_scheduled_works+0x22a/0x510 [40533.042617] process_scheduled_works+0x2a4/0x510 [40533.042623] worker_thread+0x188/0x340 [40533.042627] ? rescuer_thread+0x310/0x310 [40533.042630] kthread+0xf5/0x130 [40533.042633] ? kthread_complete_and_exit+0x20/0x20 [40533.042636] ret_from_fork+0x31/0x40 [40533.042639] ? kthread_complete_and_exit+0x20/0x20 [40533.042642] ret_from_fork_asm+0x11/0x20 [40533.042648] [40558.162514] watchdog: BUG: soft lockup - CPU#0 stuck for 26658s! [dbench:78683] [40558.169812] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40558.174513] watchdog: BUG: soft lockup - CPU#7 stuck for 35655s! [kworker/7:4:78808] [40558.225253] irq event stamp: 72305262 [40558.225254] hardirqs last enabled at (72305261): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40558.232984] Modules linked in: [40558.236642] hardirqs last disabled at (72305262): [] sysvec_apic_timer_interrupt+0xe/0x90 [40558.246800] ceph [40558.249850] softirqs last enabled at (72305182): [] __do_softirq+0x2be/0x371 [40558.259574] libceph [40558.261498] softirqs last disabled at (72305177): [] irq_exit_rcu+0xb0/0xd0 [40558.270185] dns_resolver fscache [40558.272369] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40558.280878] netfs [40558.284204] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40558.293149] veth nft_chain_nat [40558.295161] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40558.302632] xt_MASQUERADE [40558.305769] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40558.311245] nf_nat [40558.313949] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40558.332687] nf_conntrack [40558.334785] [40558.340001] nf_defrag_ipv6 nf_defrag_ipv4 [40558.342619] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40558.344110] nft_compat nf_tables [40558.348201] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40558.355326] nfnetlink bridge [40558.358637] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40558.365761] stp llc [40558.368725] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40558.375848] xfs libcrc32c [40558.378032] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40558.385156] sunrpc [40558.387861] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40558.394984] intel_rapl_msr intel_rapl_common [40558.397082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40558.405159] sb_edac x86_pkg_temp_thermal [40558.409511] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40558.415248] coretemp kvm_intel [40558.419253] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40558.426376] kvm [40558.429514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40558.436639] irqbypass i2c_i801 [40558.438474] Call Trace: [40558.438477] [40558.445598] iTCO_wdt iTCO_vendor_support [40558.448739] ? watchdog_timer_fn+0x1b4/0x220 [40558.451192] i2c_smbus [40558.453201] ? __hrtimer_run_queues+0x1d3/0x360 [40558.457205] crct10dif_pclmul joydev [40558.461472] ? hrtimer_interrupt+0x104/0x240 [40558.463824] crc32_pclmul [40558.468349] ? __local_bh_enable+0x50/0x60 [40558.471921] ipmi_ssif [40558.476194] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40558.478810] ghash_clmulni_intel [40558.482912] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40558.485268] mei_me mei [40558.490483] [40558.493708] lpc_ich [40558.498667] [40558.501110] sha512_ssse3 acpi_ipmi [40558.503209] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40558.505391] mfd_core [40558.507492] ? queued_spin_lock_slowpath+0x16/0x2e0 [40558.510972] wmi [40558.516279] do_raw_spin_lock+0xb5/0xc0 [40558.518548] ipmi_si ipmi_devintf [40558.523421] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40558.525255] ipmi_msghandler ioatdma [40558.529090] try_get_cap_refs+0x159/0x880 [ceph] [40558.532398] acpi_pad acpi_power_meter [40558.537199] __ceph_get_caps+0x4a1/0x760 [ceph] [40558.540760] ixgbe igb [40558.545375] ? __file_remove_privs+0x89/0x110 [40558.549116] mdio crc32c_intel [40558.553640] ? wake_bit_function+0x70/0x70 [40558.555997] nvme ptp [40558.560350] ? ceph_write_iter+0x260/0x770 [ceph] [40558.563398] nvme_core i2c_algo_bit [40558.567489] ceph_write_iter+0x260/0x770 [ceph] [40558.569760] pps_core dca [40558.574460] ? lock_is_held_type+0xaa/0x120 [40558.577941] fuse [40558.582469] vfs_write+0x324/0x4f0 [40558.585082] irq event stamp: 244587756 [40558.589266] ksys_pwrite64+0x65/0xa0 [40558.591193] hardirqs last enabled at (244587755): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40558.594590] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40558.598335] hardirqs last disabled at (244587756): [] sysvec_apic_timer_interrupt+0xe/0x90 [40558.601914] do_syscall_64+0x38/0x80 [40558.612158] softirqs last enabled at (244587754): [] __do_softirq+0x2be/0x371 [40558.617290] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40558.627099] softirqs last disabled at (244587749): [] irq_exit_rcu+0xb0/0xd0 [40558.630670] RIP: 0033:0x7f48f0eeb93b [40558.639442] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40558.644485] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40558.653084] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40558.656653] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40558.666031] Workqueue: events delayed_work [ceph] [40558.684769] ORIG_RAX: 0000000000000012 [40558.692239] [40558.692240] RIP: 0010:lock_acquire+0x86/0x2b0 [40558.697458] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40558.702155] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [40558.705986] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40558.707477] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [40558.711827] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40558.718950] [40558.718951] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [40558.737689] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40558.744813] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [40558.750029] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40558.757155] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [40558.758649] [40558.765768] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [40558.803592] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40558.810716] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40558.818792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40558.824531] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40558.831653] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40558.838778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40558.845916] Call Trace: [40558.848366] [40558.850377] ? watchdog_timer_fn+0x1b4/0x220 [40558.854639] ? __hrtimer_run_queues+0x1d3/0x360 [40558.859167] ? hrtimer_interrupt+0x104/0x240 [40558.863436] ? __local_bh_enable+0x50/0x60 [40558.867530] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40558.872752] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40558.877711] [40558.879806] [40558.881919] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40558.887231] ? lock_acquire+0x86/0x2b0 [40558.890980] ? igrab+0x43/0x50 [40558.894031] ? lock_release+0xc5/0x260 [40558.897775] _raw_spin_lock+0x30/0x40 [40558.901432] ? igrab+0x1d/0x50 [40558.904482] igrab+0x1d/0x50 [40558.907361] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40558.912510] delayed_work+0x1d8/0x280 [ceph] [40558.916793] ? process_scheduled_works+0x22a/0x510 [40558.921580] process_scheduled_works+0x2a4/0x510 [40558.926212] worker_thread+0x188/0x340 [40558.929962] ? rescuer_thread+0x310/0x310 [40558.933967] kthread+0xf5/0x130 [40558.937104] ? kthread_complete_and_exit+0x20/0x20 [40558.941888] ret_from_fork+0x31/0x40 [40558.945486] ? kthread_complete_and_exit+0x20/0x20 [40558.950269] ret_from_fork_asm+0x11/0x20 [40558.954192] [40586.162985] watchdog: BUG: soft lockup - CPU#0 stuck for 26684s! [dbench:78683] [40586.170282] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40586.174985] watchdog: BUG: soft lockup - CPU#7 stuck for 35681s! [kworker/7:4:78808] [40586.225724] irq event stamp: 72363462 [40586.225725] hardirqs last enabled at (72363461): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40586.233455] Modules linked in: [40586.237111] hardirqs last disabled at (72363462): [] sysvec_apic_timer_interrupt+0xe/0x90 [40586.247270] ceph [40586.250319] softirqs last enabled at (72363366): [] __do_softirq+0x2be/0x371 [40586.260044] libceph dns_resolver [40586.261967] softirqs last disabled at (72363361): [] irq_exit_rcu+0xb0/0xd0 [40586.270658] fscache netfs [40586.273971] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40586.282483] veth nft_chain_nat [40586.285188] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40586.294131] xt_MASQUERADE nf_nat [40586.297267] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40586.304739] nf_conntrack nf_defrag_ipv6 [40586.308051] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40586.313527] nf_defrag_ipv4 nft_compat [40586.317444] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40586.336183] nf_tables nfnetlink [40586.339926] [40586.339927] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40586.345145] bridge [40586.348384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40586.349875] stp llc [40586.356999] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40586.359099] xfs libcrc32c [40586.366222] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40586.368407] sunrpc intel_rapl_msr [40586.375529] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40586.378234] intel_rapl_common [40586.385358] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40586.388774] sb_edac x86_pkg_temp_thermal [40586.395896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40586.398947] coretemp kvm_intel [40586.407026] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40586.411029] kvm irqbypass [40586.416767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40586.419904] i2c_i801 [40586.427027] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40586.429731] iTCO_wdt iTCO_vendor_support [40586.436857] Call Trace: [40586.436859] [40586.439127] i2c_smbus crct10dif_pclmul [40586.446253] ? watchdog_timer_fn+0x1b4/0x220 [40586.450254] joydev crc32_pclmul [40586.452700] ? __hrtimer_run_queues+0x1d3/0x360 [40586.454711] ipmi_ssif ghash_clmulni_intel [40586.458543] ? hrtimer_interrupt+0x104/0x240 [40586.462805] mei_me mei [40586.466029] ? __local_bh_enable+0x50/0x60 [40586.470553] lpc_ich sha512_ssse3 [40586.474648] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40586.478908] acpi_ipmi mfd_core [40586.481352] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40586.485444] wmi ipmi_si [40586.488754] [40586.493970] ipmi_devintf ipmi_msghandler [40586.497108] [40586.502065] ioatdma acpi_pad [40586.504596] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40586.506693] acpi_power_meter [40586.510702] ? queued_spin_lock_slowpath+0x18/0x2e0 [40586.512795] ixgbe igb [40586.515761] do_raw_spin_lock+0xb5/0xc0 [40586.521063] mdio crc32c_intel nvme [40586.524029] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40586.528899] ptp nvme_core [40586.531257] try_get_cap_refs+0x159/0x880 [ceph] [40586.535087] i2c_algo_bit pps_core [40586.538576] __ceph_get_caps+0x4a1/0x760 [ceph] [40586.543355] dca fuse [40586.546062] ? __file_remove_privs+0x89/0x110 [40586.550670] [40586.550671] irq event stamp: 244808352 [40586.554067] ? wake_bit_function+0x70/0x70 [40586.558591] hardirqs last enabled at (244808351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40586.560863] ? ceph_write_iter+0x260/0x770 [ceph] [40586.565213] hardirqs last disabled at (244808352): [] sysvec_apic_timer_interrupt+0xe/0x90 [40586.566703] ceph_write_iter+0x260/0x770 [ceph] [40586.570447] softirqs last enabled at (244808350): [] __do_softirq+0x2be/0x371 [40586.574542] ? lock_is_held_type+0xaa/0x120 [40586.584782] softirqs last disabled at (244808345): [] irq_exit_rcu+0xb0/0xd0 [40586.589483] vfs_write+0x324/0x4f0 [40586.599289] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40586.603820] ksys_pwrite64+0x65/0xa0 [40586.612584] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40586.616761] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40586.625361] Workqueue: events delayed_work [ceph] [40586.628776] do_syscall_64+0x38/0x80 [40586.638153] [40586.638153] RIP: 0010:lock_acquire+0xf0/0x2b0 [40586.641723] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40586.649193] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40586.654324] RIP: 0033:0x7f48f0eeb93b [40586.659021] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40586.662592] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40586.664083] [40586.664084] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40586.668434] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40586.673478] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40586.673479] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40586.692215] ORIG_RAX: 0000000000000012 [40586.695788] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40586.701003] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40586.719743] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40586.721231] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40586.728357] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40586.733574] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40586.740698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40586.747822] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40586.751668] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40586.758794] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40586.765918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40586.773049] [40586.780167] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40586.838945] Call Trace: [40586.841390] [40586.843401] ? watchdog_timer_fn+0x1b4/0x220 [40586.847671] ? __hrtimer_run_queues+0x1d3/0x360 [40586.852200] ? hrtimer_interrupt+0x104/0x240 [40586.856462] ? __local_bh_enable+0x50/0x60 [40586.860554] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40586.865779] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40586.870736] [40586.872832] [40586.874931] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40586.880238] ? lock_acquire+0xf0/0x2b0 [40586.883987] ? igrab+0x43/0x50 [40586.887037] ? lock_release+0xc5/0x260 [40586.890784] _raw_spin_lock+0x30/0x40 [40586.894448] ? igrab+0x1d/0x50 [40586.897499] igrab+0x1d/0x50 [40586.900390] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40586.905535] delayed_work+0x1d8/0x280 [ceph] [40586.909817] ? process_scheduled_works+0x22a/0x510 [40586.914607] process_scheduled_works+0x2a4/0x510 [40586.919222] worker_thread+0x188/0x340 [40586.922971] ? rescuer_thread+0x310/0x310 [40586.926973] kthread+0xf5/0x130 [40586.930113] ? kthread_complete_and_exit+0x20/0x20 [40586.934896] ret_from_fork+0x31/0x40 [40586.938466] ? kthread_complete_and_exit+0x20/0x20 [40586.943252] ret_from_fork_asm+0x11/0x20 [40586.947175] [40614.163458] watchdog: BUG: soft lockup - CPU#0 stuck for 26710s! [dbench:78683] [40614.170763] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40614.175458] watchdog: BUG: soft lockup - CPU#7 stuck for 35707s! [kworker/7:4:78808] [40614.226212] irq event stamp: 72421654 [40614.226213] hardirqs last enabled at (72421653): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40614.233944] Modules linked in: ceph [40614.237601] hardirqs last disabled at (72421654): [] sysvec_apic_timer_interrupt+0xe/0x90 [40614.247756] libceph [40614.251241] softirqs last enabled at (72421588): [] __do_softirq+0x2be/0x371 [40614.260965] dns_resolver fscache [40614.263149] softirqs last disabled at (72421583): [] irq_exit_rcu+0xb0/0xd0 [40614.271850] netfs veth [40614.275162] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40614.283673] nft_chain_nat [40614.286134] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40614.295079] xt_MASQUERADE nf_nat [40614.297784] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40614.305252] nf_conntrack [40614.308563] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40614.314042] nf_defrag_ipv6 nf_defrag_ipv4 [40614.316657] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40614.335397] nft_compat nf_tables [40614.339487] [40614.339487] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40614.344703] nfnetlink bridge [40614.348014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40614.349507] stp llc [40614.356629] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40614.359593] xfs libcrc32c [40614.366718] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40614.368903] sunrpc intel_rapl_msr [40614.376027] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40614.378731] intel_rapl_common sb_edac [40614.385854] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40614.389251] x86_pkg_temp_thermal coretemp [40614.396376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40614.400120] kvm_intel kvm [40614.408198] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40614.412288] irqbypass i2c_i801 [40614.418025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40614.420729] iTCO_wdt iTCO_vendor_support [40614.427854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40614.430992] i2c_smbus crct10dif_pclmul [40614.438132] Call Trace: [40614.438135] [40614.442136] joydev crc32_pclmul [40614.449262] ? watchdog_timer_fn+0x1b4/0x220 [40614.453091] ipmi_ssif ghash_clmulni_intel [40614.455536] ? __hrtimer_run_queues+0x1d3/0x360 [40614.457547] mei_me mei [40614.460773] ? hrtimer_interrupt+0x104/0x240 [40614.465034] lpc_ich sha512_ssse3 [40614.469138] ? __local_bh_enable+0x50/0x60 [40614.473666] acpi_ipmi [40614.476114] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40614.480391] mfd_core wmi [40614.483703] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40614.487794] ipmi_si ipmi_devintf [40614.490163] [40614.495376] ipmi_msghandler ioatdma [40614.497994] [40614.497995] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40614.502952] acpi_pad acpi_power_meter [40614.506267] ? queued_spin_lock_slowpath+0x16/0x2e0 [40614.508359] ixgbe igb [40614.511932] do_raw_spin_lock+0xb5/0xc0 [40614.514029] mdio crc32c_intel [40614.519334] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40614.523075] nvme ptp nvme_core [40614.527951] try_get_cap_refs+0x159/0x880 [ceph] [40614.530305] i2c_algo_bit pps_core [40614.534141] __ceph_get_caps+0x4a1/0x760 [ceph] [40614.537186] dca fuse [40614.541973] ? __file_remove_privs+0x89/0x110 [40614.545107] [40614.545107] irq event stamp: 245029070 [40614.549719] ? wake_bit_function+0x70/0x70 [40614.553115] hardirqs last enabled at (245029069): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40614.557658] ? ceph_write_iter+0x260/0x770 [ceph] [40614.559927] hardirqs last disabled at (245029070): [] sysvec_apic_timer_interrupt+0xe/0x90 [40614.564279] ceph_write_iter+0x260/0x770 [ceph] [40614.565769] softirqs last enabled at (245029068): [] __do_softirq+0x2be/0x371 [40614.569517] ? lock_is_held_type+0xaa/0x120 [40614.573605] softirqs last disabled at (245029063): [] irq_exit_rcu+0xb0/0xd0 [40614.583861] vfs_write+0x324/0x4f0 [40614.588554] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40614.598371] ksys_pwrite64+0x65/0xa0 [40614.602890] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40614.611662] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40614.615853] Workqueue: events delayed_work [ceph] [40614.624453] do_syscall_64+0x38/0x80 [40614.627858] [40614.627859] RIP: 0010:lock_acquire+0xf0/0x2b0 [40614.637235] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40614.640807] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40614.648277] RIP: 0033:0x7f48f0eeb93b [40614.653409] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40614.658107] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40614.661676] [40614.661677] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40614.663166] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40614.667518] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40614.672562] ORIG_RAX: 0000000000000012 [40614.691299] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40614.694870] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40614.700088] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40614.718826] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40614.720315] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40614.727441] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40614.732657] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40614.739782] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40614.743612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40614.750736] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40614.757860] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40614.764990] [40614.772107] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40614.830863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40614.837986] Call Trace: [40614.840430] [40614.842441] ? watchdog_timer_fn+0x1b4/0x220 [40614.846706] ? __hrtimer_run_queues+0x1d3/0x360 [40614.851233] ? hrtimer_interrupt+0x104/0x240 [40614.855502] ? __local_bh_enable+0x50/0x60 [40614.859594] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40614.864809] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40614.869766] [40614.871865] [40614.873962] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40614.879269] ? lock_acquire+0xf0/0x2b0 [40614.883020] ? igrab+0x43/0x50 [40614.886071] ? lock_release+0xc5/0x260 [40614.889815] _raw_spin_lock+0x30/0x40 [40614.893470] ? igrab+0x1d/0x50 [40614.896521] igrab+0x1d/0x50 [40614.899400] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40614.904550] delayed_work+0x1d8/0x280 [ceph] [40614.908831] ? process_scheduled_works+0x22a/0x510 [40614.913622] process_scheduled_works+0x2a4/0x510 [40614.918235] worker_thread+0x188/0x340 [40614.921984] ? rescuer_thread+0x310/0x310 [40614.925990] kthread+0xf5/0x130 [40614.929151] ? kthread_complete_and_exit+0x20/0x20 [40614.933937] ret_from_fork+0x31/0x40 [40614.937515] ? kthread_complete_and_exit+0x20/0x20 [40614.942299] ret_from_fork_asm+0x11/0x20 [40614.946223] [40642.163932] watchdog: BUG: soft lockup - CPU#0 stuck for 26736s! [dbench:78683] [40642.171232] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40642.175932] watchdog: BUG: soft lockup - CPU#7 stuck for 35733s! [kworker/7:4:78808] [40642.226672] irq event stamp: 72479870 [40642.226674] hardirqs last enabled at (72479869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40642.234405] Modules linked in: [40642.238062] hardirqs last disabled at (72479870): [] sysvec_apic_timer_interrupt+0xe/0x90 [40642.248219] ceph [40642.251268] softirqs last enabled at (72479784): [] __do_softirq+0x2be/0x371 [40642.260995] libceph [40642.262917] softirqs last disabled at (72479779): [] irq_exit_rcu+0xb0/0xd0 [40642.271605] dns_resolver fscache [40642.273786] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40642.282298] netfs veth [40642.285607] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40642.285608] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40642.294568] nft_chain_nat xt_MASQUERADE [40642.297012] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40642.304484] nf_nat nf_conntrack [40642.309960] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40642.313879] nf_defrag_ipv6 nf_defrag_ipv4 [40642.332616] [40642.332617] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40642.335840] nft_compat nf_tables [40642.341057] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40642.345149] nfnetlink bridge [40642.346638] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40642.353763] stp llc [40642.357073] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40642.364198] xfs libcrc32c [40642.367162] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40642.374286] sunrpc [40642.376470] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40642.383609] intel_rapl_msr [40642.386330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40642.393457] intel_rapl_common sb_edac [40642.395555] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40642.403631] x86_pkg_temp_thermal coretemp [40642.406423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40642.412161] kvm_intel [40642.415904] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40642.423028] kvm irqbypass [40642.427119] Call Trace: [40642.427121] [40642.434242] i2c_i801 [40642.436617] ? watchdog_timer_fn+0x1b4/0x220 [40642.443742] iTCO_wdt [40642.446448] ? __hrtimer_run_queues+0x1d3/0x360 [40642.448891] iTCO_vendor_support [40642.450906] ? hrtimer_interrupt+0x104/0x240 [40642.453171] i2c_smbus crct10dif_pclmul [40642.457435] ? __local_bh_enable+0x50/0x60 [40642.459706] joydev crc32_pclmul [40642.464234] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40642.467454] ipmi_ssif ghash_clmulni_intel [40642.471720] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40642.475549] mei_me mei [40642.479641] [40642.482865] lpc_ich sha512_ssse3 [40642.488082] [40642.492171] acpi_ipmi mfd_core [40642.497131] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40642.499573] wmi ipmi_si [40642.501675] ? queued_spin_lock_slowpath+0x18/0x2e0 [40642.504981] ipmi_devintf ipmi_msghandler [40642.507083] do_raw_spin_lock+0xb5/0xc0 [40642.510217] ioatdma acpi_pad [40642.515521] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40642.518051] acpi_power_meter ixgbe [40642.522924] try_get_cap_refs+0x159/0x880 [ceph] [40642.526926] igb mdio [40642.530763] __ceph_get_caps+0x4a1/0x760 [ceph] [40642.533722] crc32c_intel nvme [40642.538511] ? __file_remove_privs+0x89/0x110 [40642.541989] ptp nvme_core [40642.546614] ? wake_bit_function+0x70/0x70 [40642.548888] i2c_algo_bit pps_core [40642.553416] ? ceph_write_iter+0x260/0x770 [ceph] [40642.556464] dca fuse [40642.560813] ceph_write_iter+0x260/0x770 [ceph] [40642.563517] [40642.563518] irq event stamp: 245250010 [40642.567616] ? lock_is_held_type+0xaa/0x120 [40642.571014] hardirqs last enabled at (245250009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40642.575714] vfs_write+0x324/0x4f0 [40642.577984] hardirqs last disabled at (245250010): [] sysvec_apic_timer_interrupt+0xe/0x90 [40642.582511] ksys_pwrite64+0x65/0xa0 [40642.583997] softirqs last enabled at (245250008): [] __do_softirq+0x2be/0x371 [40642.587741] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40642.591919] softirqs last disabled at (245250003): [] irq_exit_rcu+0xb0/0xd0 [40642.602165] do_syscall_64+0x38/0x80 [40642.605561] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40642.615371] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40642.618943] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40642.627712] RIP: 0033:0x7f48f0eeb93b [40642.632844] Workqueue: events delayed_work [ceph] [40642.641442] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40642.645012] [40642.645013] RIP: 0010:lock_acquire+0xf0/0x2b0 [40642.654390] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40642.659434] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40642.666905] ORIG_RAX: 0000000000000012 [40642.670476] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40642.675172] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40642.693910] [40642.693910] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40642.695400] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40642.699752] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40642.704970] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40642.723706] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40642.727537] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40642.732755] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40642.739879] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40642.741369] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40642.748499] [40642.755619] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40642.815766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40642.821503] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40642.828628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40642.835751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40642.842877] Call Trace: [40642.845336] [40642.847349] ? watchdog_timer_fn+0x1b4/0x220 [40642.851618] ? __hrtimer_run_queues+0x1d3/0x360 [40642.856148] ? hrtimer_interrupt+0x104/0x240 [40642.860409] ? __local_bh_enable+0x50/0x60 [40642.864503] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40642.869726] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40642.874682] [40642.876782] [40642.878877] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40642.884187] ? lock_acquire+0xf0/0x2b0 [40642.887935] ? igrab+0x43/0x50 [40642.890985] ? lock_release+0xc5/0x260 [40642.894733] _raw_spin_lock+0x30/0x40 [40642.898396] ? igrab+0x1d/0x50 [40642.901445] igrab+0x1d/0x50 [40642.904338] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40642.909482] delayed_work+0x1d8/0x280 [ceph] [40642.913765] ? process_scheduled_works+0x22a/0x510 [40642.918553] process_scheduled_works+0x2a4/0x510 [40642.923170] worker_thread+0x188/0x340 [40642.926918] ? rescuer_thread+0x310/0x310 [40642.930924] kthread+0xf5/0x130 [40642.934060] ? kthread_complete_and_exit+0x20/0x20 [40642.938843] ret_from_fork+0x31/0x40 [40642.942414] ? kthread_complete_and_exit+0x20/0x20 [40642.947200] ret_from_fork_asm+0x11/0x20 [40642.951130] [40670.164407] watchdog: BUG: soft lockup - CPU#0 stuck for 26763s! [dbench:78683] [40670.171712] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40670.176406] watchdog: BUG: soft lockup - CPU#7 stuck for 35760s! [kworker/7:4:78808] [40670.227161] irq event stamp: 72538102 [40670.227162] hardirqs last enabled at (72538101): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40670.234890] Modules linked in: [40670.238550] hardirqs last disabled at (72538102): [] sysvec_apic_timer_interrupt+0xe/0x90 [40670.248707] ceph [40670.251756] softirqs last enabled at (72537988): [] __do_softirq+0x2be/0x371 [40670.261480] libceph dns_resolver [40670.263404] softirqs last disabled at (72537983): [] irq_exit_rcu+0xb0/0xd0 [40670.272090] fscache netfs [40670.275401] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40670.283910] veth nft_chain_nat [40670.286615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40670.295558] xt_MASQUERADE [40670.298697] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40670.306166] nf_nat [40670.308873] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40670.314349] nf_conntrack nf_defrag_ipv6 [40670.316447] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40670.335183] nf_defrag_ipv4 nft_compat [40670.339102] [40670.344320] nf_tables [40670.348061] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40670.349571] nfnetlink bridge [40670.351929] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40670.359051] stp llc [40670.362015] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40670.369142] xfs libcrc32c [40670.371325] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40670.378450] sunrpc [40670.381152] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40670.388276] intel_rapl_msr intel_rapl_common [40670.390374] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40670.397498] sb_edac x86_pkg_temp_thermal [40670.401849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40670.409926] coretemp kvm_intel [40670.413932] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40670.419668] kvm irqbypass [40670.422805] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40670.429930] i2c_i801 [40670.432634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40670.439759] iTCO_wdt [40670.442029] Call Trace: [40670.442032] [40670.449154] iTCO_vendor_support i2c_smbus [40670.451427] ? watchdog_timer_fn+0x1b4/0x220 [40670.453868] crct10dif_pclmul joydev [40670.455880] ? __hrtimer_run_queues+0x1d3/0x360 [40670.459967] crc32_pclmul ipmi_ssif [40670.464235] ? hrtimer_interrupt+0x104/0x240 [40670.467805] ghash_clmulni_intel [40670.472327] ? __local_bh_enable+0x50/0x60 [40670.475813] mei_me mei [40670.480088] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40670.483309] lpc_ich sha512_ssse3 [40670.487400] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40670.489842] acpi_ipmi mfd_core [40670.495060] [40670.498390] wmi ipmi_si [40670.503348] [40670.506483] ipmi_devintf ipmi_msghandler [40670.508581] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40670.511112] ioatdma acpi_pad [40670.513212] ? queued_spin_lock_slowpath+0x16/0x2e0 [40670.517212] acpi_power_meter ixgbe [40670.522520] do_raw_spin_lock+0xb5/0xc0 [40670.525481] igb mdio [40670.530354] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40670.533835] crc32c_intel [40670.537669] try_get_cap_refs+0x159/0x880 [ceph] [40670.539937] nvme ptp [40670.544726] __ceph_get_caps+0x4a1/0x760 [ceph] [40670.547340] nvme_core i2c_algo_bit [40670.551955] ? __file_remove_privs+0x89/0x110 [40670.554220] pps_core dca [40670.558744] ? wake_bit_function+0x70/0x70 [40670.562228] fuse [40670.566581] ? ceph_write_iter+0x260/0x770 [ceph] [40670.569195] irq event stamp: 245470732 [40670.573287] ceph_write_iter+0x260/0x770 [ceph] [40670.575212] hardirqs last enabled at (245470731): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40670.579913] ? lock_is_held_type+0xaa/0x120 [40670.583653] hardirqs last disabled at (245470732): [] sysvec_apic_timer_interrupt+0xe/0x90 [40670.588178] vfs_write+0x324/0x4f0 [40670.598421] softirqs last enabled at (245470730): [] __do_softirq+0x2be/0x371 [40670.602603] ksys_pwrite64+0x65/0xa0 [40670.612409] softirqs last disabled at (245470725): [] irq_exit_rcu+0xb0/0xd0 [40670.615807] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40670.624578] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40670.628149] do_syscall_64+0x38/0x80 [40670.636746] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40670.641877] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40670.651254] Workqueue: events delayed_work [ceph] [40670.654825] RIP: 0033:0x7f48f0eeb93b [40670.662295] [40670.662296] RIP: 0010:lock_acquire+0xf0/0x2b0 [40670.667340] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40670.672036] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40670.675609] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40670.677098] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40670.681450] ORIG_RAX: 0000000000000012 [40670.700187] [40670.700188] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40670.718925] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40670.724143] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40670.729361] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40670.733191] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40670.734682] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40670.741809] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40670.748939] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40670.756063] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40670.763186] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40670.770312] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40670.777439] [40670.784558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40670.821929] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40670.829054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40670.836179] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40670.843303] Call Trace: [40670.845749] [40670.847760] ? watchdog_timer_fn+0x1b4/0x220 [40670.852024] ? __hrtimer_run_queues+0x1d3/0x360 [40670.856548] ? hrtimer_interrupt+0x104/0x240 [40670.860813] ? __local_bh_enable+0x50/0x60 [40670.864914] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40670.870135] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40670.875093] [40670.877189] [40670.879287] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40670.884597] ? lock_acquire+0xf0/0x2b0 [40670.888345] ? igrab+0x43/0x50 [40670.891396] ? lock_release+0xc5/0x260 [40670.895141] _raw_spin_lock+0x30/0x40 [40670.898815] ? igrab+0x1d/0x50 [40670.901864] igrab+0x1d/0x50 [40670.904743] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40670.909892] delayed_work+0x1d8/0x280 [ceph] [40670.914175] ? process_scheduled_works+0x22a/0x510 [40670.918963] process_scheduled_works+0x2a4/0x510 [40670.923581] worker_thread+0x188/0x340 [40670.927329] ? rescuer_thread+0x310/0x310 [40670.931332] kthread+0xf5/0x130 [40670.934470] ? kthread_complete_and_exit+0x20/0x20 [40670.939255] ret_from_fork+0x31/0x40 [40670.942834] ? kthread_complete_and_exit+0x20/0x20 [40670.947618] ret_from_fork_asm+0x11/0x20 [40670.951550] [40698.164880] watchdog: BUG: soft lockup - CPU#0 stuck for 26789s! [dbench:78683] [40698.172182] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40698.176880] watchdog: BUG: soft lockup - CPU#7 stuck for 35786s! [kworker/7:4:78808] [40698.227623] irq event stamp: 72596270 [40698.227624] hardirqs last enabled at (72596269): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40698.235354] Modules linked in: [40698.239011] hardirqs last disabled at (72596270): [] sysvec_apic_timer_interrupt+0xe/0x90 [40698.249167] ceph [40698.252218] softirqs last enabled at (72596214): [] __do_softirq+0x2be/0x371 [40698.261943] libceph [40698.263867] softirqs last disabled at (72596209): [] irq_exit_rcu+0xb0/0xd0 [40698.272555] dns_resolver [40698.274736] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40698.283245] fscache netfs [40698.285862] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40698.294808] veth nft_chain_nat [40698.297512] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40698.304983] xt_MASQUERADE [40698.308121] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40698.313598] nf_nat [40698.316301] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40698.335038] nf_conntrack [40698.337137] [40698.337138] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40698.342354] nf_defrag_ipv6 nf_defrag_ipv4 [40698.344970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40698.346463] nft_compat nf_tables [40698.353587] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40698.357676] nfnetlink bridge [40698.364800] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40698.368112] stp llc [40698.375234] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40698.378199] xfs libcrc32c [40698.385323] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40698.387507] sunrpc intel_rapl_msr [40698.394633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40698.397336] intel_rapl_common sb_edac [40698.405413] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40698.408812] x86_pkg_temp_thermal coretemp [40698.414564] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40698.418309] kvm_intel kvm [40698.425434] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40698.429525] irqbypass i2c_i801 [40698.436647] Call Trace: [40698.436649] [40698.439352] iTCO_wdt iTCO_vendor_support [40698.446477] ? watchdog_timer_fn+0x1b4/0x220 [40698.449613] i2c_smbus crct10dif_pclmul [40698.452060] ? __hrtimer_run_queues+0x1d3/0x360 [40698.454070] joydev crc32_pclmul [40698.458077] ? hrtimer_interrupt+0x104/0x240 [40698.462336] ipmi_ssif ghash_clmulni_intel [40698.466167] ? __local_bh_enable+0x50/0x60 [40698.470691] mei_me mei [40698.473919] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40698.478179] lpc_ich sha512_ssse3 [40698.482281] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40698.486370] acpi_ipmi mfd_core [40698.488816] [40698.494031] wmi ipmi_si [40698.497343] [40698.502298] ipmi_devintf ipmi_msghandler [40698.505439] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40698.507534] ioatdma acpi_pad [40698.510088] ? queued_spin_lock_slowpath+0x16/0x2e0 [40698.512181] acpi_power_meter [40698.516187] do_raw_spin_lock+0xb5/0xc0 [40698.521488] ixgbe igb [40698.524454] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40698.529323] mdio crc32c_intel [40698.532289] try_get_cap_refs+0x159/0x880 [ceph] [40698.536119] nvme ptp [40698.538480] __ceph_get_caps+0x4a1/0x760 [ceph] [40698.543276] nvme_core [40698.546333] ? __file_remove_privs+0x89/0x110 [40698.550937] i2c_algo_bit pps_core [40698.553209] ? wake_bit_function+0x70/0x70 [40698.557733] dca fuse [40698.560094] ? ceph_write_iter+0x260/0x770 [ceph] [40698.564441] [40698.564442] irq event stamp: 245691564 [40698.567839] ceph_write_iter+0x260/0x770 [ceph] [40698.571930] hardirqs last enabled at (245691563): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40698.574202] ? lock_is_held_type+0xaa/0x120 [40698.578897] hardirqs last disabled at (245691564): [] sysvec_apic_timer_interrupt+0xe/0x90 [40698.580390] vfs_write+0x324/0x4f0 [40698.584133] softirqs last enabled at (245691562): [] __do_softirq+0x2be/0x371 [40698.588660] ksys_pwrite64+0x65/0xa0 [40698.598899] softirqs last disabled at (245691557): [] irq_exit_rcu+0xb0/0xd0 [40698.603078] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40698.612888] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40698.616287] do_syscall_64+0x38/0x80 [40698.625055] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40698.628628] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40698.637224] Workqueue: events delayed_work [ceph] [40698.642355] RIP: 0033:0x7f48f0eeb93b [40698.651733] [40698.651734] RIP: 0010:lock_acquire+0xf0/0x2b0 [40698.655304] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40698.662773] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40698.667820] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40698.672517] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40698.676088] ORIG_RAX: 0000000000000012 [40698.677578] [40698.677578] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40698.681928] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40698.700666] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40698.719404] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40698.724622] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40698.729839] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40698.733668] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40698.735159] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40698.742285] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40698.749408] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40698.756534] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40698.763662] [40698.770781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40698.822408] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40698.829534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40698.836657] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40698.843781] Call Trace: [40698.846227] [40698.848237] ? watchdog_timer_fn+0x1b4/0x220 [40698.852502] ? __hrtimer_run_queues+0x1d3/0x360 [40698.857027] ? hrtimer_interrupt+0x104/0x240 [40698.861289] ? __local_bh_enable+0x50/0x60 [40698.865381] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40698.870598] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40698.875571] [40698.877669] [40698.879767] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40698.885075] ? lock_acquire+0xf0/0x2b0 [40698.888825] ? igrab+0x43/0x50 [40698.891876] ? lock_release+0xc5/0x260 [40698.895619] _raw_spin_lock+0x30/0x40 [40698.899287] ? igrab+0x1d/0x50 [40698.902342] igrab+0x1d/0x50 [40698.905222] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40698.910371] delayed_work+0x1d8/0x280 [ceph] [40698.914652] ? process_scheduled_works+0x22a/0x510 [40698.919442] process_scheduled_works+0x2a4/0x510 [40698.924058] worker_thread+0x188/0x340 [40698.927807] ? rescuer_thread+0x310/0x310 [40698.931811] kthread+0xf5/0x130 [40698.934949] ? kthread_complete_and_exit+0x20/0x20 [40698.939733] ret_from_fork+0x31/0x40 [40698.943304] ? kthread_complete_and_exit+0x20/0x20 [40698.948088] ret_from_fork_asm+0x11/0x20 [40698.952020] [40726.165352] watchdog: BUG: soft lockup - CPU#0 stuck for 26815s! [dbench:78683] [40726.172662] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40726.177351] watchdog: BUG: soft lockup - CPU#7 stuck for 35812s! [kworker/7:4:78808] [40726.228109] irq event stamp: 72654502 [40726.228110] hardirqs last enabled at (72654501): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40726.235841] Modules linked in: [40726.239498] hardirqs last disabled at (72654502): [] sysvec_apic_timer_interrupt+0xe/0x90 [40726.249655] ceph [40726.252706] softirqs last enabled at (72654416): [] __do_softirq+0x2be/0x371 [40726.262429] libceph [40726.264355] softirqs last disabled at (72654411): [] irq_exit_rcu+0xb0/0xd0 [40726.273039] dns_resolver [40726.275224] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40726.283748] fscache netfs [40726.286369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40726.295312] veth nft_chain_nat [40726.298034] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40726.305505] xt_MASQUERADE [40726.308644] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40726.314118] nf_nat nf_conntrack [40726.316822] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40726.335561] nf_defrag_ipv6 [40726.338785] [40726.344001] nf_defrag_ipv4 nft_compat [40726.346793] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40726.348284] nf_tables [40726.352045] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40726.359168] nfnetlink [40726.361527] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40726.368650] bridge stp [40726.371008] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40726.378131] llc xfs [40726.380576] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40726.387699] libcrc32c sunrpc [40726.389883] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40726.397009] intel_rapl_msr intel_rapl_common [40726.399972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40726.408051] sb_edac x86_pkg_temp_thermal [40726.412400] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40726.418138] coretemp kvm_intel [40726.422143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40726.429265] kvm irqbypass [40726.432405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40726.439529] i2c_i801 iTCO_wdt [40726.442232] Call Trace: [40726.449355] iTCO_vendor_support i2c_smbus [40726.452409] [40726.454852] crct10dif_pclmul joydev [40726.458945] ? watchdog_timer_fn+0x1b4/0x220 [40726.460951] crc32_pclmul ipmi_ssif [40726.464524] ? __hrtimer_run_queues+0x1d3/0x360 [40726.468787] ghash_clmulni_intel [40726.472274] ? hrtimer_interrupt+0x104/0x240 [40726.476794] mei_me [40726.480029] ? __local_bh_enable+0x50/0x60 [40726.484293] mei lpc_ich [40726.486391] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40726.490480] sha512_ssse3 acpi_ipmi [40726.493028] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40726.498245] mfd_core wmi [40726.501747] [40726.506704] ipmi_si ipmi_devintf [40726.509322] [40726.509323] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40726.511420] ipmi_msghandler ioatdma [40726.514751] ? queued_spin_lock_slowpath+0x18/0x2e0 [40726.516846] acpi_pad acpi_power_meter [40726.522152] do_raw_spin_lock+0xb5/0xc0 [40726.525720] ixgbe igb [40726.530592] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40726.534335] mdio crc32c_intel [40726.538167] try_get_cap_refs+0x159/0x880 [ceph] [40726.540523] nvme ptp [40726.545312] __ceph_get_caps+0x4a1/0x760 [ceph] [40726.548358] nvme_core i2c_algo_bit [40726.552973] ? __file_remove_privs+0x89/0x110 [40726.555238] pps_core [40726.559764] ? wake_bit_function+0x70/0x70 [40726.563246] dca fuse [40726.567600] ? ceph_write_iter+0x260/0x770 [ceph] [40726.569867] [40726.569868] irq event stamp: 245912280 [40726.573958] ceph_write_iter+0x260/0x770 [ceph] [40726.576231] hardirqs last enabled at (245912279): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40726.580932] ? lock_is_held_type+0xaa/0x120 [40726.582418] hardirqs last disabled at (245912280): [] sysvec_apic_timer_interrupt+0xe/0x90 [40726.586166] vfs_write+0x324/0x4f0 [40726.590687] softirqs last enabled at (245912278): [] __do_softirq+0x2be/0x371 [40726.600940] ksys_pwrite64+0x65/0xa0 [40726.605108] softirqs last disabled at (245912273): [] irq_exit_rcu+0xb0/0xd0 [40726.614920] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40726.618315] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40726.627089] do_syscall_64+0x38/0x80 [40726.630657] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40726.639256] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40726.644385] Workqueue: events delayed_work [ceph] [40726.653764] RIP: 0033:0x7f48f0eeb93b [40726.657334] [40726.657335] RIP: 0010:lock_acquire+0xf0/0x2b0 [40726.664805] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40726.669848] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40726.674548] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40726.678117] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40726.679609] ORIG_RAX: 0000000000000012 [40726.683959] [40726.683960] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40726.702696] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40726.721433] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40726.726650] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40726.731869] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40726.735698] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40726.737191] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40726.744314] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40726.751438] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40726.758561] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40726.765687] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40726.772817] [40726.779936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40726.824430] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40726.831555] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40726.838679] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40726.845803] Call Trace: [40726.848249] [40726.850258] ? watchdog_timer_fn+0x1b4/0x220 [40726.854524] ? __hrtimer_run_queues+0x1d3/0x360 [40726.859051] ? hrtimer_interrupt+0x104/0x240 [40726.863318] ? __local_bh_enable+0x50/0x60 [40726.867412] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40726.872637] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40726.877592] [40726.879690] [40726.881788] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40726.887096] ? lock_acquire+0xf0/0x2b0 [40726.890845] ? igrab+0x43/0x50 [40726.893897] ? lock_release+0xc5/0x260 [40726.897641] _raw_spin_lock+0x30/0x40 [40726.901305] ? igrab+0x1d/0x50 [40726.904356] igrab+0x1d/0x50 [40726.907235] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40726.912384] delayed_work+0x1d8/0x280 [ceph] [40726.916666] ? process_scheduled_works+0x22a/0x510 [40726.921455] process_scheduled_works+0x2a4/0x510 [40726.926073] worker_thread+0x188/0x340 [40726.929821] ? rescuer_thread+0x310/0x310 [40726.933823] kthread+0xf5/0x130 [40726.936962] ? kthread_complete_and_exit+0x20/0x20 [40726.941761] ret_from_fork+0x31/0x40 [40726.945333] ? kthread_complete_and_exit+0x20/0x20 [40726.950118] ret_from_fork_asm+0x11/0x20 [40726.954054] [40728.048749] rcu: INFO: rcu_preempt self-detected stall on CPU [40728.054507] rcu: 0-....: (28139928 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14095344 [40728.065019] rcu: (t=28790937 jiffies g=1309609 q=769105 ncpus=8) [40728.071104] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40728.080054] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40728.087526] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40728.093004] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40728.111740] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40728.116958] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40728.124082] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40728.131205] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40728.138331] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40728.145453] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40728.152580] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40728.160657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40728.166393] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40728.173516] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40728.180642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40728.187765] Call Trace: [40728.190212] [40728.192222] ? rcu_dump_cpu_stacks+0xd1/0x180 [40728.196581] ? rcu_sched_clock_irq+0x633/0x1390 [40728.201106] ? lock_is_held_type+0xaa/0x120 [40728.205284] ? lock_is_held_type+0xaa/0x120 [40728.209461] ? tick_sched_do_timer+0xa0/0xa0 [40728.213732] ? rcu_read_lock_sched_held+0x56/0x90 [40728.218430] ? tick_sched_do_timer+0xa0/0xa0 [40728.222693] ? update_process_times+0x5f/0x90 [40728.227046] ? tick_sched_handle.isra.16+0x1e/0x50 [40728.231829] ? tick_sched_timer+0x6f/0x80 [40728.235833] ? __hrtimer_run_queues+0x1d3/0x360 [40728.240360] ? hrtimer_interrupt+0x104/0x240 [40728.244628] ? __local_bh_enable+0x50/0x60 [40728.248722] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40728.253938] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40728.258895] [40728.260992] [40728.263089] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40728.268399] ? queued_spin_lock_slowpath+0x16/0x2e0 [40728.273274] do_raw_spin_lock+0xb5/0xc0 [40728.277103] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40728.281908] try_get_cap_refs+0x159/0x880 [ceph] [40728.286537] __ceph_get_caps+0x4a1/0x760 [ceph] [40728.291078] ? __file_remove_privs+0x89/0x110 [40728.295436] ? wake_bit_function+0x70/0x70 [40728.299528] ? ceph_write_iter+0x260/0x770 [ceph] [40728.304241] ceph_write_iter+0x260/0x770 [ceph] [40728.308777] ? lock_is_held_type+0xaa/0x120 [40728.312961] vfs_write+0x324/0x4f0 [40728.316370] ksys_pwrite64+0x65/0xa0 [40728.319942] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40728.325075] do_syscall_64+0x38/0x80 [40728.328646] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40728.333690] RIP: 0033:0x7f48f0eeb93b [40728.337261] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40728.355997] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [40728.363555] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40728.370680] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40728.377804] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40728.384928] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40728.392052] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40728.399181] [40728.401368] Sending NMI from CPU 0 to CPUs 7: [40728.405722] NMI backtrace for cpu 7 [40728.405724] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40728.405725] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40728.405726] Workqueue: events delayed_work [ceph] [40728.405741] RIP: 0010:__lock_acquire+0x2dd/0x1e20 [40728.405743] Code: 75 e5 45 31 ed 65 8b 05 e9 fa ea 7e 41 be 02 00 00 00 85 c0 0f 84 2f 05 00 00 44 39 ef 0f 8e 89 00 00 00 48 c7 c2 20 51 f7 82 <4c> 89 e1 4c 8b 24 24 48 89 d0 48 89 ea 48 89 c5 eb 3d 48 8d 34 40 [40728.405745] RSP: 0018:ffffc90002117c60 EFLAGS: 00000006 [40728.405746] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: 0000000000000000 [40728.405746] RDX: ffffffff82f75120 RSI: 0000000000000006 RDI: 0000000000000003 [40728.405747] RBP: ffff88810debb550 R08: 0000000000000078 R09: 0000000000000000 [40728.405748] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [40728.405749] R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000002 [40728.405749] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40728.405750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40728.405751] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40728.405752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40728.405752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40728.405753] Call Trace: [40728.405754] [40728.405755] ? nmi_cpu_backtrace+0xad/0x120 [40728.405759] ? nmi_cpu_backtrace_handler+0x11/0x20 [40728.405761] ? nmi_handle+0xd4/0x260 [40728.405765] ? default_do_nmi+0x49/0xf0 [40728.405766] ? exc_nmi+0x11e/0x150 [40728.405768] ? end_repeat_nmi+0x16/0x67 [40728.405771] ? __lock_acquire+0x2dd/0x1e20 [40728.405773] ? __lock_acquire+0x2dd/0x1e20 [40728.405775] ? __lock_acquire+0x2dd/0x1e20 [40728.405776] [40728.405776] [40728.405778] lock_acquire+0xcb/0x2b0 [40728.405780] ? igrab+0x1d/0x50 [40728.405781] ? igrab+0x43/0x50 [40728.405782] ? lock_release+0xc5/0x260 [40728.405784] _raw_spin_lock+0x30/0x40 [40728.405786] ? igrab+0x1d/0x50 [40728.405787] igrab+0x1d/0x50 [40728.405788] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40728.405802] delayed_work+0x1d8/0x280 [ceph] [40728.405816] ? process_scheduled_works+0x22a/0x510 [40728.405819] process_scheduled_works+0x2a4/0x510 [40728.405823] worker_thread+0x188/0x340 [40728.405825] ? rescuer_thread+0x310/0x310 [40728.405828] kthread+0xf5/0x130 [40728.405830] ? kthread_complete_and_exit+0x20/0x20 [40728.405832] ret_from_fork+0x31/0x40 [40728.405834] ? kthread_complete_and_exit+0x20/0x20 [40728.405836] ret_from_fork_asm+0x11/0x20 [40728.405840] [40754.165830] watchdog: BUG: soft lockup - CPU#0 stuck for 26841s! [dbench:78683] [40754.173131] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40754.177830] watchdog: BUG: soft lockup - CPU#7 stuck for 35838s! [kworker/7:4:78808] [40754.228572] irq event stamp: 72711942 [40754.228573] hardirqs last enabled at (72711941): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40754.236303] Modules linked in: [40754.239960] hardirqs last disabled at (72711942): [] sysvec_apic_timer_interrupt+0xe/0x90 [40754.250118] ceph [40754.253169] softirqs last enabled at (72711842): [] __do_softirq+0x2be/0x371 [40754.262892] libceph dns_resolver [40754.264814] softirqs last disabled at (72711837): [] irq_exit_rcu+0xb0/0xd0 [40754.273515] fscache netfs [40754.276827] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40754.285339] veth nft_chain_nat [40754.288042] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40754.296988] xt_MASQUERADE nf_nat [40754.300123] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40754.307595] nf_conntrack nf_defrag_ipv6 [40754.310906] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40754.316383] nf_defrag_ipv4 [40754.320299] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40754.339038] nft_compat nf_tables [40754.341829] [40754.341830] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40754.347045] nfnetlink [40754.350357] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40754.351848] bridge stp [40754.358971] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40754.361330] llc xfs [40754.368453] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40754.370898] libcrc32c sunrpc [40754.378022] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40754.380205] intel_rapl_msr intel_rapl_common [40754.387347] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40754.390310] sb_edac [40754.397436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40754.401786] x86_pkg_temp_thermal coretemp [40754.409865] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40754.412049] kvm_intel [40754.417786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40754.421876] kvm irqbypass [40754.428999] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40754.429001] Call Trace: [40754.431359] i2c_i801 [40754.438481] [40754.441185] iTCO_wdt iTCO_vendor_support [40754.448311] ? watchdog_timer_fn+0x1b4/0x220 [40754.450753] i2c_smbus [40754.453025] ? __hrtimer_run_queues+0x1d3/0x360 [40754.455035] crct10dif_pclmul [40754.459042] ? hrtimer_interrupt+0x104/0x240 [40754.463303] joydev [40754.465661] ? __local_bh_enable+0x50/0x60 [40754.470186] crc32_pclmul ipmi_ssif [40754.473152] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40754.477412] ghash_clmulni_intel [40754.479515] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40754.483609] mei_me mei [40754.487094] [40754.492311] lpc_ich sha512_ssse3 [40754.495535] [40754.500508] acpi_ipmi mfd_core [40754.502956] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40754.505052] wmi ipmi_si [40754.508365] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40754.510460] ipmi_devintf [40754.513600] do_raw_spin_lock+0xb5/0xc0 [40754.518901] ipmi_msghandler ioatdma [40754.521432] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40754.526303] acpi_pad acpi_power_meter ixgbe [40754.528921] try_get_cap_refs+0x159/0x880 [ceph] [40754.532752] igb mdio [40754.536326] __ceph_get_caps+0x4a1/0x760 [ceph] [40754.541105] crc32c_intel nvme [40754.545373] ? __file_remove_privs+0x89/0x110 [40754.549980] ptp nvme_core [40754.552253] ? wake_bit_function+0x70/0x70 [40754.556775] i2c_algo_bit pps_core [40754.559828] ? ceph_write_iter+0x260/0x770 [ceph] [40754.564175] dca fuse [40754.566882] ceph_write_iter+0x260/0x770 [ceph] [40754.570970] [40754.574373] ? lock_is_held_type+0xaa/0x120 [40754.579067] irq event stamp: 246131158 [40754.581340] vfs_write+0x324/0x4f0 [40754.585860] hardirqs last enabled at (246131157): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40754.587355] ksys_pwrite64+0x65/0xa0 [40754.591529] hardirqs last disabled at (246131158): [] sysvec_apic_timer_interrupt+0xe/0x90 [40754.595274] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40754.598671] softirqs last enabled at (246131156): [] __do_softirq+0x2be/0x371 [40754.608916] do_syscall_64+0x38/0x80 [40754.612485] softirqs last disabled at (246131151): [] irq_exit_rcu+0xb0/0xd0 [40754.622297] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40754.627427] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40754.636199] RIP: 0033:0x7f48f0eeb93b [40754.639769] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40754.648367] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40754.653410] Workqueue: events delayed_work [ceph] [40754.662788] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40754.666360] [40754.666361] RIP: 0010:lock_release+0x14e/0x260 [40754.673830] ORIG_RAX: 0000000000000012 [40754.692568] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [40754.697263] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40754.702481] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [40754.703972] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40754.708410] [40754.712242] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40754.730980] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [40754.738102] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40754.743319] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [40754.750445] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40754.751935] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [40754.759064] [40754.766183] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [40754.803989] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [40754.811112] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40754.819190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40754.824928] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40754.832051] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40754.839175] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40754.846299] Call Trace: [40754.848744] [40754.850755] ? watchdog_timer_fn+0x1b4/0x220 [40754.855020] ? __hrtimer_run_queues+0x1d3/0x360 [40754.859547] ? hrtimer_interrupt+0x104/0x240 [40754.863816] ? __local_bh_enable+0x50/0x60 [40754.867908] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40754.873133] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40754.878089] [40754.880186] [40754.882285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40754.887590] ? igrab+0x43/0x50 [40754.890643] ? lock_release+0x14e/0x260 [40754.894478] ? lock_release+0xc5/0x260 [40754.898240] _raw_spin_unlock+0x1b/0x40 [40754.902072] igrab+0x43/0x50 [40754.904948] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40754.910100] delayed_work+0x1d8/0x280 [ceph] [40754.914379] ? process_scheduled_works+0x22a/0x510 [40754.919170] process_scheduled_works+0x2a4/0x510 [40754.923787] worker_thread+0x188/0x340 [40754.927536] ? rescuer_thread+0x310/0x310 [40754.931545] kthread+0xf5/0x130 [40754.934683] ? kthread_complete_and_exit+0x20/0x20 [40754.939470] ret_from_fork+0x31/0x40 [40754.943047] ? kthread_complete_and_exit+0x20/0x20 [40754.947831] ret_from_fork_asm+0x11/0x20 [40754.951754] [40782.166334] watchdog: BUG: soft lockup - CPU#0 stuck for 26867s! [dbench:78683] [40782.173637] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40782.178333] watchdog: BUG: soft lockup - CPU#7 stuck for 35864s! [kworker/7:4:78808] [40782.229084] irq event stamp: 72770244 [40782.229085] hardirqs last enabled at (72770243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40782.236815] Modules linked in: [40782.240472] hardirqs last disabled at (72770244): [] sysvec_apic_timer_interrupt+0xe/0x90 [40782.250631] ceph [40782.253681] softirqs last enabled at (72770148): [] __do_softirq+0x2be/0x371 [40782.263405] libceph [40782.265328] softirqs last disabled at (72770143): [] irq_exit_rcu+0xb0/0xd0 [40782.274013] dns_resolver [40782.276199] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40782.284726] fscache [40782.287344] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40782.296287] netfs veth [40782.298472] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40782.305942] nft_chain_nat [40782.308385] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40782.313863] xt_MASQUERADE nf_nat [40782.316566] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40782.335304] nf_conntrack nf_defrag_ipv6 [40782.338616] [40782.338616] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40782.343832] nf_defrag_ipv4 [40782.347751] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40782.349242] nft_compat nf_tables [40782.356365] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40782.359157] nfnetlink bridge [40782.366280] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40782.369592] stp llc [40782.376728] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40782.379687] xfs libcrc32c [40782.386812] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40782.389011] sunrpc intel_rapl_msr [40782.396139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40782.398842] intel_rapl_common sb_edac [40782.406919] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40782.410317] x86_pkg_temp_thermal [40782.416053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40782.419799] coretemp kvm_intel [40782.426923] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40782.430233] kvm irqbypass [40782.437357] Call Trace: [40782.440495] i2c_i801 iTCO_wdt [40782.447621] [40782.450323] iTCO_vendor_support i2c_smbus [40782.452768] ? watchdog_timer_fn+0x1b4/0x220 [40782.455817] crct10dif_pclmul joydev [40782.457828] ? __hrtimer_run_queues+0x1d3/0x360 [40782.461920] crc32_pclmul ipmi_ssif [40782.466186] ? hrtimer_interrupt+0x104/0x240 [40782.469754] ghash_clmulni_intel [40782.474278] ? __local_bh_enable+0x50/0x60 [40782.477762] mei_me [40782.482030] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40782.485250] mei lpc_ich [40782.489342] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40782.491439] sha512_ssse3 acpi_ipmi [40782.496656] [40782.499186] mfd_core wmi [40782.504144] [40782.507628] ipmi_si ipmi_devintf [40782.509735] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40782.512352] ipmi_msghandler [40782.514453] ? queued_spin_lock_slowpath+0x16/0x2e0 [40782.517760] ioatdma acpi_pad [40782.523064] do_raw_spin_lock+0xb5/0xc0 [40782.525941] acpi_power_meter ixgbe [40782.530813] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40782.533774] igb mdio [40782.537611] try_get_cap_refs+0x159/0x880 [ceph] [40782.541089] crc32c_intel nvme [40782.545879] __ceph_get_caps+0x4a1/0x760 [ceph] [40782.548144] ptp nvme_core [40782.552760] ? __file_remove_privs+0x89/0x110 [40782.555807] i2c_algo_bit pps_core [40782.560332] ? wake_bit_function+0x70/0x70 [40782.563035] dca fuse [40782.567389] ? ceph_write_iter+0x260/0x770 [ceph] [40782.570784] [40782.570784] irq event stamp: 246351898 [40782.574875] ceph_write_iter+0x260/0x770 [ceph] [40782.577145] hardirqs last enabled at (246351897): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40782.581844] ? lock_is_held_type+0xaa/0x120 [40782.583332] hardirqs last disabled at (246351898): [] sysvec_apic_timer_interrupt+0xe/0x90 [40782.587080] vfs_write+0x324/0x4f0 [40782.591601] softirqs last enabled at (246351896): [] __do_softirq+0x2be/0x371 [40782.601849] ksys_pwrite64+0x65/0xa0 [40782.606024] softirqs last disabled at (246351891): [] irq_exit_rcu+0xb0/0xd0 [40782.615833] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40782.619230] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40782.628002] do_syscall_64+0x38/0x80 [40782.631571] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40782.640169] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40782.645301] Workqueue: events delayed_work [ceph] [40782.654680] RIP: 0033:0x7f48f0eeb93b [40782.658249] [40782.658250] RIP: 0010:_raw_spin_unlock+0x2d/0x40 [40782.665737] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40782.670781] Code: fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df e8 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b 0f 1f 44 00 00 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e [40782.675477] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40782.679048] RSP: 0018:ffffc90002117d98 EFLAGS: 00000202 [40782.680539] ORIG_RAX: 0000000000000012 [40782.685150] [40782.685151] RAX: ffffffffffffffff RBX: ffff8881a1d34478 RCX: ffffc90002117d54 [40782.703888] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40782.722626] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [40782.727844] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40782.733060] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [40782.736892] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40782.738383] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [40782.745506] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40782.752629] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40782.759754] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40782.766878] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40782.774025] [40782.781143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40782.825658] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40782.832780] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40782.839905] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40782.847030] Call Trace: [40782.849474] [40782.851485] ? watchdog_timer_fn+0x1b4/0x220 [40782.855749] ? __hrtimer_run_queues+0x1d3/0x360 [40782.860276] ? hrtimer_interrupt+0x104/0x240 [40782.864546] ? __local_bh_enable+0x50/0x60 [40782.868638] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40782.873863] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40782.878821] [40782.880916] [40782.883032] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40782.888339] ? _raw_spin_unlock+0x2d/0x40 [40782.892349] igrab+0x43/0x50 [40782.895229] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40782.900377] delayed_work+0x1d8/0x280 [ceph] [40782.904659] ? process_scheduled_works+0x22a/0x510 [40782.909449] process_scheduled_works+0x2a4/0x510 [40782.914064] worker_thread+0x188/0x340 [40782.917813] ? rescuer_thread+0x310/0x310 [40782.921817] kthread+0xf5/0x130 [40782.924962] ? kthread_complete_and_exit+0x20/0x20 [40782.929746] ret_from_fork+0x31/0x40 [40782.933317] ? kthread_complete_and_exit+0x20/0x20 [40782.938102] ret_from_fork_asm+0x11/0x20 [40782.942033] [40810.166837] watchdog: BUG: soft lockup - CPU#0 stuck for 26893s! [dbench:78683] [40810.174139] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40810.178837] watchdog: BUG: soft lockup - CPU#7 stuck for 35890s! [kworker/7:4:78808] [40810.229581] irq event stamp: 72828434 [40810.229583] hardirqs last enabled at (72828433): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40810.237311] Modules linked in: [40810.240968] hardirqs last disabled at (72828434): [] sysvec_apic_timer_interrupt+0xe/0x90 [40810.251126] ceph [40810.254177] softirqs last enabled at (72828368): [] __do_softirq+0x2be/0x371 [40810.263901] libceph [40810.265826] softirqs last disabled at (72828363): [] irq_exit_rcu+0xb0/0xd0 [40810.274512] dns_resolver [40810.276693] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40810.285205] fscache netfs [40810.287838] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40810.296782] veth [40810.299489] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40810.306975] nft_chain_nat [40810.308898] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40810.314378] xt_MASQUERADE [40810.317081] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40810.335818] nf_nat [40810.338532] [40810.338532] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40810.343748] nf_conntrack nf_defrag_ipv6 [40810.345845] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40810.347337] nf_defrag_ipv4 nft_compat [40810.354461] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40810.358378] nf_tables nfnetlink [40810.365520] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40810.369263] bridge [40810.376388] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40810.379612] stp llc [40810.386736] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40810.388835] xfs libcrc32c [40810.395957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40810.398142] sunrpc intel_rapl_msr [40810.406234] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40810.408941] intel_rapl_common [40810.414678] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40810.418076] sb_edac x86_pkg_temp_thermal [40810.425199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40810.428249] coretemp kvm_intel [40810.435374] Call Trace: [40810.439378] kvm irqbypass [40810.446514] [40810.449648] i2c_i801 iTCO_wdt [40810.452093] ? watchdog_timer_fn+0x1b4/0x220 [40810.454796] iTCO_vendor_support [40810.456807] ? __hrtimer_run_queues+0x1d3/0x360 [40810.459857] i2c_smbus crct10dif_pclmul [40810.464126] ? hrtimer_interrupt+0x104/0x240 [40810.467347] joydev [40810.471872] ? __local_bh_enable+0x50/0x60 [40810.475701] crc32_pclmul ipmi_ssif [40810.479968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40810.482062] ghash_clmulni_intel mei_me [40810.486154] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40810.489637] mei lpc_ich [40810.494855] [40810.498686] sha512_ssse3 acpi_ipmi [40810.503642] [40810.503643] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40810.506172] mfd_core [40810.508276] ? queued_spin_lock_slowpath+0x16/0x2e0 [40810.511756] wmi ipmi_si [40810.513855] do_raw_spin_lock+0xb5/0xc0 [40810.519158] ipmi_devintf [40810.521429] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40810.526299] ipmi_msghandler [40810.528833] try_get_cap_refs+0x159/0x880 [ceph] [40810.532660] ioatdma [40810.535282] __ceph_get_caps+0x4a1/0x760 [ceph] [40810.540060] acpi_pad acpi_power_meter [40810.542942] ? __file_remove_privs+0x89/0x110 [40810.547549] ixgbe igb [40810.549734] ? wake_bit_function+0x70/0x70 [40810.554256] mdio [40810.558003] ? ceph_write_iter+0x260/0x770 [ceph] [40810.562352] crc32c_intel nvme [40810.564709] ceph_write_iter+0x260/0x770 [ceph] [40810.568801] ptp nvme_core [40810.570727] ? lock_is_held_type+0xaa/0x120 [40810.575421] i2c_algo_bit pps_core [40810.578476] vfs_write+0x324/0x4f0 [40810.582996] dca fuse [40810.585704] ksys_pwrite64+0x65/0xa0 [40810.589878] [40810.589879] irq event stamp: 246572772 [40810.593276] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40810.596673] hardirqs last enabled at (246572771): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40810.598946] do_syscall_64+0x38/0x80 [40810.602526] hardirqs last disabled at (246572772): [] sysvec_apic_timer_interrupt+0xe/0x90 [40810.604015] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40810.607758] softirqs last enabled at (246572770): [] __do_softirq+0x2be/0x371 [40810.612890] RIP: 0033:0x7f48f0eeb93b [40810.623131] softirqs last disabled at (246572765): [] irq_exit_rcu+0xb0/0xd0 [40810.626703] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40810.636520] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40810.641568] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40810.650336] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40810.653908] ORIG_RAX: 0000000000000012 [40810.662520] Workqueue: events delayed_work [ceph] [40810.681261] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40810.690637] [40810.690638] RIP: 0010:lock_release+0x1f/0x260 [40810.695857] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40810.703328] Code: a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 <48> 89 44 24 10 31 c0 0f 1f 44 00 00 65 8b 05 9a ea eb 7e 89 c0 48 [40810.707156] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40810.711855] RSP: 0018:ffffc90002117d48 EFLAGS: 00000296 [40810.718978] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40810.720470] [40810.724820] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40810.731944] RAX: c5e6ef7cc8fa9e00 RBX: ffff8881a1d34500 RCX: 000000005553a634 [40810.750689] [40810.757807] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [40810.795195] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [40810.802320] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [40810.809443] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40810.816568] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40810.824646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40810.830381] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40810.837523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40810.844649] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40810.851773] Call Trace: [40810.854218] [40810.856245] ? watchdog_timer_fn+0x1b4/0x220 [40810.860525] ? __hrtimer_run_queues+0x1d3/0x360 [40810.865054] ? hrtimer_interrupt+0x104/0x240 [40810.869321] ? __local_bh_enable+0x50/0x60 [40810.873416] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40810.878639] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40810.883598] [40810.885694] [40810.887793] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40810.893107] ? igrab+0x43/0x50 [40810.896158] ? igrab+0x43/0x50 [40810.899207] ? lock_release+0x1f/0x260 [40810.902952] ? lock_release+0xc5/0x260 [40810.906696] _raw_spin_unlock+0x1b/0x40 [40810.910534] igrab+0x43/0x50 [40810.913413] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40810.918562] delayed_work+0x1d8/0x280 [ceph] [40810.922841] ? process_scheduled_works+0x22a/0x510 [40810.927633] process_scheduled_works+0x2a4/0x510 [40810.932248] worker_thread+0x188/0x340 [40810.935997] ? rescuer_thread+0x310/0x310 [40810.940001] kthread+0xf5/0x130 [40810.943139] ? kthread_complete_and_exit+0x20/0x20 [40810.947922] ret_from_fork+0x31/0x40 [40810.951501] ? kthread_complete_and_exit+0x20/0x20 [40810.956303] ret_from_fork_asm+0x11/0x20 [40810.960244] [40838.167326] watchdog: BUG: soft lockup - CPU#0 stuck for 26919s! [dbench:78683] [40838.174629] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40838.179325] watchdog: BUG: soft lockup - CPU#7 stuck for 35916s! [kworker/7:4:78808] [40838.230077] irq event stamp: 72886730 [40838.230078] hardirqs last enabled at (72886729): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40838.237807] Modules linked in: ceph [40838.241464] hardirqs last disabled at (72886730): [] sysvec_apic_timer_interrupt+0xe/0x90 [40838.251624] libceph [40838.255106] softirqs last enabled at (72886608): [] __do_softirq+0x2be/0x371 [40838.264830] dns_resolver fscache [40838.267014] softirqs last disabled at (72886603): [] irq_exit_rcu+0xb0/0xd0 [40838.275716] netfs veth [40838.279028] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40838.287538] nft_chain_nat xt_MASQUERADE [40838.289999] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40838.298943] nf_nat [40838.302863] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40838.310330] nf_conntrack nf_defrag_ipv6 [40838.312429] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40838.317907] nf_defrag_ipv4 [40838.321825] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40838.340561] nft_compat nf_tables [40838.343353] [40838.343354] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40838.348568] nfnetlink [40838.351881] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40838.353370] bridge stp [40838.360495] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40838.362851] llc xfs [40838.369976] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40838.372421] libcrc32c sunrpc [40838.379545] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40838.381730] intel_rapl_msr [40838.388854] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40838.391818] intel_rapl_common [40838.398940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40838.401733] sb_edac x86_pkg_temp_thermal [40838.409809] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40838.412859] coretemp [40838.418598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40838.422600] kvm_intel kvm [40838.429726] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40838.432009] irqbypass i2c_i801 [40838.439130] Call Trace: [40838.439132] [40838.441834] iTCO_wdt [40838.448960] ? watchdog_timer_fn+0x1b4/0x220 [40838.452096] iTCO_vendor_support i2c_smbus [40838.454540] ? __hrtimer_run_queues+0x1d3/0x360 [40838.456550] crct10dif_pclmul [40838.458824] ? hrtimer_interrupt+0x104/0x240 [40838.463085] joydev crc32_pclmul [40838.467174] ? __local_bh_enable+0x50/0x60 [40838.471699] ipmi_ssif [40838.474683] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40838.478945] ghash_clmulni_intel [40838.482169] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40838.486260] mei_me mei [40838.488619] [40838.493834] lpc_ich sha512_ssse3 [40838.497059] [40838.502016] acpi_ipmi [40838.504461] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40838.506558] mfd_core wmi [40838.509872] ? queued_spin_lock_slowpath+0x16/0x2e0 [40838.511964] ipmi_si ipmi_devintf [40838.514324] do_raw_spin_lock+0xb5/0xc0 [40838.519627] ipmi_msghandler [40838.522246] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40838.527116] ioatdma acpi_pad acpi_power_meter [40838.530429] try_get_cap_refs+0x159/0x880 [ceph] [40838.534256] ixgbe igb [40838.537139] __ceph_get_caps+0x4a1/0x760 [ceph] [40838.541918] mdio crc32c_intel [40838.546360] ? __file_remove_privs+0x89/0x110 [40838.550966] nvme [40838.553324] ? wake_bit_function+0x70/0x70 [40838.557848] ptp nvme_core [40838.560901] ? ceph_write_iter+0x260/0x770 [ceph] [40838.565250] i2c_algo_bit pps_core [40838.567174] ceph_write_iter+0x260/0x770 [ceph] [40838.571263] dca fuse [40838.573970] ? lock_is_held_type+0xaa/0x120 [40838.578666] [40838.578667] irq event stamp: 246793674 [40838.582065] vfs_write+0x324/0x4f0 [40838.586587] hardirqs last enabled at (246793673): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40838.588862] ksys_pwrite64+0x65/0xa0 [40838.593034] hardirqs last disabled at (246793674): [] sysvec_apic_timer_interrupt+0xe/0x90 [40838.594527] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40838.598268] softirqs last enabled at (246793672): [] __do_softirq+0x2be/0x371 [40838.601667] do_syscall_64+0x38/0x80 [40838.611912] softirqs last disabled at (246793663): [] irq_exit_rcu+0xb0/0xd0 [40838.615483] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40838.625292] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40838.630425] RIP: 0033:0x7f48f0eeb93b [40838.639194] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40838.642766] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40838.651363] Workqueue: events delayed_work [ceph] [40838.656408] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40838.665784] [40838.669356] ORIG_RAX: 0000000000000012 [40838.676827] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [40838.695564] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40838.700261] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [40838.705477] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40838.706968] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [40838.710800] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40838.716727] [40838.716727] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [40838.723852] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40838.742591] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [40838.749729] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40838.754948] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [40838.762077] [40838.763564] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [40838.808492] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40838.815616] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40838.823695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40838.829431] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40838.836557] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40838.843680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40838.850804] Call Trace: [40838.853249] [40838.855260] ? watchdog_timer_fn+0x1b4/0x220 [40838.859524] ? __hrtimer_run_queues+0x1d3/0x360 [40838.864050] ? hrtimer_interrupt+0x104/0x240 [40838.868319] ? __local_bh_enable+0x50/0x60 [40838.872413] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40838.877636] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40838.882593] [40838.884692] [40838.886788] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40838.892097] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [40838.897414] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [40838.902731] delayed_work+0x1d8/0x280 [ceph] [40838.907013] ? process_scheduled_works+0x22a/0x510 [40838.911800] process_scheduled_works+0x2a4/0x510 [40838.916417] worker_thread+0x188/0x340 [40838.920165] ? rescuer_thread+0x310/0x310 [40838.924168] kthread+0xf5/0x130 [40838.927308] ? kthread_complete_and_exit+0x20/0x20 [40838.932091] ret_from_fork+0x31/0x40 [40838.935671] ? kthread_complete_and_exit+0x20/0x20 [40838.940453] ret_from_fork_asm+0x11/0x20 [40838.944378] [40866.167810] watchdog: BUG: soft lockup - CPU#0 stuck for 26945s! [dbench:78683] [40866.175114] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40866.179811] watchdog: BUG: soft lockup - CPU#7 stuck for 35942s! [kworker/7:4:78808] [40866.230555] irq event stamp: 72944990 [40866.230556] hardirqs last enabled at (72944989): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40866.238288] Modules linked in: [40866.241943] hardirqs last disabled at (72944990): [] sysvec_apic_timer_interrupt+0xe/0x90 [40866.252101] ceph [40866.255153] softirqs last enabled at (72944884): [] __do_softirq+0x2be/0x371 [40866.264877] libceph [40866.266800] softirqs last disabled at (72944879): [] irq_exit_rcu+0xb0/0xd0 [40866.275493] dns_resolver [40866.277677] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40866.286205] fscache netfs [40866.288824] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40866.297767] veth nft_chain_nat [40866.300487] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40866.307959] xt_MASQUERADE nf_nat [40866.311097] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40866.316573] nf_conntrack nf_defrag_ipv6 [40866.319886] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40866.338623] nf_defrag_ipv4 nft_compat [40866.342541] [40866.342541] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40866.347757] nf_tables nfnetlink [40866.351501] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40866.352992] bridge stp [40866.360116] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40866.363339] llc xfs [40866.370464] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40866.372908] libcrc32c sunrpc [40866.380032] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40866.382215] intel_rapl_msr [40866.389340] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40866.392306] intel_rapl_common [40866.399429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40866.402220] sb_edac x86_pkg_temp_thermal [40866.410297] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40866.413348] coretemp kvm_intel [40866.419086] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40866.423090] kvm irqbypass [40866.430213] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40866.433352] i2c_i801 iTCO_wdt [40866.440489] Call Trace: [40866.440491] [40866.443209] iTCO_vendor_support i2c_smbus [40866.450339] ? watchdog_timer_fn+0x1b4/0x220 [40866.453389] crct10dif_pclmul [40866.455834] ? __hrtimer_run_queues+0x1d3/0x360 [40866.457844] joydev crc32_pclmul [40866.461938] ? hrtimer_interrupt+0x104/0x240 [40866.466209] ipmi_ssif ghash_clmulni_intel [40866.469171] ? __local_bh_enable+0x50/0x60 [40866.473696] mei_me mei [40866.476922] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40866.481184] lpc_ich sha512_ssse3 [40866.485291] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40866.489381] acpi_ipmi mfd_core [40866.491827] [40866.497044] wmi ipmi_si [40866.500354] [40866.500356] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40866.505311] ipmi_devintf ipmi_msghandler [40866.508452] ? queued_spin_lock_slowpath+0x18/0x2e0 [40866.510545] ioatdma acpi_pad [40866.513080] do_raw_spin_lock+0xb5/0xc0 [40866.515173] acpi_power_meter ixgbe [40866.520496] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40866.524501] igb mdio crc32c_intel [40866.529373] try_get_cap_refs+0x159/0x880 [ceph] [40866.532335] nvme ptp [40866.536171] __ceph_get_caps+0x4a1/0x760 [ceph] [40866.539649] nvme_core i2c_algo_bit [40866.544438] ? __file_remove_privs+0x89/0x110 [40866.547831] pps_core dca [40866.552443] ? wake_bit_function+0x70/0x70 [40866.554712] fuse [40866.559240] ? ceph_write_iter+0x260/0x770 [ceph] [40866.562720] [40866.562721] irq event stamp: 247014656 [40866.567071] ceph_write_iter+0x260/0x770 [ceph] [40866.569688] hardirqs last enabled at (247014655): [] asm_common_interrupt+0x26/0x40 [40866.573783] ? lock_is_held_type+0xaa/0x120 [40866.575703] hardirqs last disabled at (247014656): [] sysvec_apic_timer_interrupt+0xe/0x90 [40866.580404] vfs_write+0x324/0x4f0 [40866.581892] softirqs last enabled at (247014654): [] __do_softirq+0x2be/0x371 [40866.585640] ksys_pwrite64+0x65/0xa0 [40866.590160] softirqs last disabled at (247014649): [] irq_exit_rcu+0xb0/0xd0 [40866.599452] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40866.603629] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40866.613439] do_syscall_64+0x38/0x80 [40866.616836] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40866.625607] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40866.629178] Workqueue: events delayed_work [ceph] [40866.637776] RIP: 0033:0x7f48f0eeb93b [40866.642907] [40866.642908] RIP: 0010:lock_acquire+0xf0/0x2b0 [40866.652285] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40866.655854] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40866.663324] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40866.668368] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40866.673066] ORIG_RAX: 0000000000000012 [40866.676637] [40866.676638] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40866.678129] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40866.682477] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40866.701234] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40866.719971] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40866.725190] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40866.730424] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40866.734253] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40866.735745] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40866.742870] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40866.749992] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40866.757123] [40866.764240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40866.822986] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40866.830110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40866.837234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40866.844359] Call Trace: [40866.846804] [40866.848815] ? watchdog_timer_fn+0x1b4/0x220 [40866.853079] ? __hrtimer_run_queues+0x1d3/0x360 [40866.857604] ? hrtimer_interrupt+0x104/0x240 [40866.861874] ? __local_bh_enable+0x50/0x60 [40866.865968] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40866.871191] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40866.876165] [40866.878264] [40866.880360] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40866.885667] ? lock_acquire+0xf0/0x2b0 [40866.889418] ? igrab+0x43/0x50 [40866.892468] ? lock_release+0xc5/0x260 [40866.896218] _raw_spin_lock+0x30/0x40 [40866.899879] ? igrab+0x1d/0x50 [40866.902929] igrab+0x1d/0x50 [40866.905808] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40866.910955] delayed_work+0x1d8/0x280 [ceph] [40866.915237] ? process_scheduled_works+0x22a/0x510 [40866.920029] process_scheduled_works+0x2a4/0x510 [40866.924644] worker_thread+0x188/0x340 [40866.928394] ? rescuer_thread+0x310/0x310 [40866.932404] kthread+0xf5/0x130 [40866.935541] ? kthread_complete_and_exit+0x20/0x20 [40866.940327] ret_from_fork+0x31/0x40 [40866.943896] ? kthread_complete_and_exit+0x20/0x20 [40866.948681] ret_from_fork_asm+0x11/0x20 [40866.952603] [40894.168294] watchdog: BUG: soft lockup - CPU#0 stuck for 26971s! [dbench:78683] [40894.175594] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40894.180294] watchdog: BUG: soft lockup - CPU#7 stuck for 35968s! [kworker/7:4:78808] [40894.231033] irq event stamp: 73003354 [40894.231034] hardirqs last enabled at (73003353): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40894.238765] Modules linked in: [40894.242424] hardirqs last disabled at (73003354): [] sysvec_apic_timer_interrupt+0xe/0x90 [40894.252579] ceph [40894.255631] softirqs last enabled at (73003248): [] __do_softirq+0x2be/0x371 [40894.265355] libceph [40894.267279] softirqs last disabled at (73003243): [] irq_exit_rcu+0xb0/0xd0 [40894.275967] dns_resolver fscache [40894.278147] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40894.286657] netfs [40894.289985] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40894.298930] veth nft_chain_nat [40894.300940] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40894.308413] xt_MASQUERADE [40894.311550] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40894.317028] nf_nat nf_conntrack [40894.319730] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40894.338468] nf_defrag_ipv6 [40894.341691] [40894.341692] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40894.346911] nf_defrag_ipv4 [40894.349701] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40894.351192] nft_compat nf_tables [40894.358315] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40894.361106] nfnetlink bridge [40894.368230] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40894.371541] stp llc [40894.378665] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40894.381647] xfs libcrc32c [40894.388770] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40894.390954] sunrpc [40894.398097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40894.400801] intel_rapl_msr intel_rapl_common [40894.408878] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40894.410975] sb_edac x86_pkg_temp_thermal [40894.416713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40894.421064] coretemp kvm_intel [40894.428188] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40894.432192] kvm irqbypass [40894.439315] Call Trace: [40894.439317] [40894.442453] i2c_i801 iTCO_wdt [40894.449578] ? watchdog_timer_fn+0x1b4/0x220 [40894.452281] iTCO_vendor_support [40894.454727] ? __hrtimer_run_queues+0x1d3/0x360 [40894.456735] i2c_smbus [40894.459789] ? hrtimer_interrupt+0x104/0x240 [40894.464051] crct10dif_pclmul [40894.467274] ? __local_bh_enable+0x50/0x60 [40894.471799] joydev crc32_pclmul [40894.474159] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40894.478421] ipmi_ssif ghash_clmulni_intel [40894.481384] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40894.485474] mei_me mei [40894.488698] [40894.493916] lpc_ich [40894.498007] [40894.502973] sha512_ssse3 acpi_ipmi [40894.505419] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40894.507515] mfd_core [40894.509702] ? queued_spin_lock_slowpath+0x16/0x2e0 [40894.511797] wmi ipmi_si [40894.515283] do_raw_spin_lock+0xb5/0xc0 [40894.520585] ipmi_devintf [40894.522856] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40894.527726] ipmi_msghandler ioatdma acpi_pad [40894.530260] try_get_cap_refs+0x159/0x880 [ceph] [40894.534087] acpi_power_meter [40894.536710] __ceph_get_caps+0x4a1/0x760 [ceph] [40894.541489] ixgbe igb [40894.545844] ? __file_remove_privs+0x89/0x110 [40894.550450] mdio crc32c_intel [40894.553416] ? wake_bit_function+0x70/0x70 [40894.557937] nvme ptp [40894.560299] ? ceph_write_iter+0x260/0x770 [ceph] [40894.564646] nvme_core i2c_algo_bit [40894.567697] ceph_write_iter+0x260/0x770 [ceph] [40894.571787] pps_core dca [40894.574063] ? lock_is_held_type+0xaa/0x120 [40894.578757] fuse [40894.582243] vfs_write+0x324/0x4f0 [40894.586764] irq event stamp: 247235550 [40894.589386] ksys_pwrite64+0x65/0xa0 [40894.593560] hardirqs last enabled at (247235549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40894.595484] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40894.598881] hardirqs last disabled at (247235550): [] sysvec_apic_timer_interrupt+0xe/0x90 [40894.602627] do_syscall_64+0x38/0x80 [40894.606197] softirqs last enabled at (247235548): [] __do_softirq+0x2be/0x371 [40894.616439] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40894.621570] softirqs last disabled at (247235543): [] irq_exit_rcu+0xb0/0xd0 [40894.631381] RIP: 0033:0x7f48f0eeb93b [40894.634952] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40894.643724] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40894.648766] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40894.657365] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40894.660936] Workqueue: events delayed_work [ceph] [40894.670313] ORIG_RAX: 0000000000000012 [40894.670314] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40894.689050] [40894.689051] RIP: 0010:rcu_is_watching+0xb/0x40 [40894.696522] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40894.701739] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [40894.706435] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40894.710266] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [40894.717392] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40894.718882] [40894.718883] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [40894.723320] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40894.730442] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [40894.749185] [40894.756303] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [40894.800819] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [40894.807942] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40894.815067] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40894.823143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40894.828882] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40894.836007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40894.843129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40894.850253] Call Trace: [40894.852701] [40894.854711] ? watchdog_timer_fn+0x1b4/0x220 [40894.858987] ? __hrtimer_run_queues+0x1d3/0x360 [40894.863517] ? hrtimer_interrupt+0x104/0x240 [40894.867787] ? __local_bh_enable+0x50/0x60 [40894.871888] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40894.877113] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40894.882077] [40894.884176] [40894.886274] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40894.891588] ? igrab+0x43/0x50 [40894.894646] ? igrab+0x43/0x50 [40894.897701] ? rcu_is_watching+0xb/0x40 [40894.901536] lock_release+0x1d7/0x260 [40894.905197] _raw_spin_unlock+0x1b/0x40 [40894.909034] igrab+0x43/0x50 [40894.911912] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40894.917068] delayed_work+0x1d8/0x280 [ceph] [40894.921350] ? process_scheduled_works+0x22a/0x510 [40894.926140] process_scheduled_works+0x2a4/0x510 [40894.930757] worker_thread+0x188/0x340 [40894.934505] ? rescuer_thread+0x310/0x310 [40894.938507] kthread+0xf5/0x130 [40894.941647] ? kthread_complete_and_exit+0x20/0x20 [40894.946431] ret_from_fork+0x31/0x40 [40894.950008] ? kthread_complete_and_exit+0x20/0x20 [40894.954794] ret_from_fork_asm+0x11/0x20 [40894.958717] [40922.168775] watchdog: BUG: soft lockup - CPU#0 stuck for 26997s! [dbench:78683] [40922.176082] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40922.180775] watchdog: BUG: soft lockup - CPU#7 stuck for 35994s! [kworker/7:4:78808] [40922.231530] irq event stamp: 73061686 [40922.231532] hardirqs last enabled at (73061685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40922.239262] Modules linked in: [40922.242919] hardirqs last disabled at (73061686): [] sysvec_apic_timer_interrupt+0xe/0x90 [40922.253077] ceph [40922.256127] softirqs last enabled at (73061564): [] __do_softirq+0x2be/0x371 [40922.265853] libceph [40922.267775] softirqs last disabled at (73061559): [] irq_exit_rcu+0xb0/0xd0 [40922.276459] dns_resolver fscache [40922.278644] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40922.287168] netfs [40922.290482] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40922.299427] veth nft_chain_nat [40922.301462] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40922.308935] xt_MASQUERADE nf_nat [40922.312074] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40922.317550] nf_conntrack nf_defrag_ipv6 [40922.320859] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40922.339597] nf_defrag_ipv4 nft_compat [40922.343516] [40922.343517] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40922.348732] nf_tables nfnetlink [40922.352476] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40922.353967] bridge stp [40922.361090] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40922.364316] llc xfs [40922.371452] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40922.373900] libcrc32c [40922.381025] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40922.383209] sunrpc [40922.390332] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40922.392689] intel_rapl_msr [40922.399815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40922.401911] intel_rapl_common sb_edac [40922.409990] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40922.412780] x86_pkg_temp_thermal coretemp [40922.418518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40922.422261] kvm_intel kvm [40922.429386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40922.433476] irqbypass [40922.440600] Call Trace: [40922.440602] [40922.443306] i2c_i801 iTCO_wdt [40922.450430] ? watchdog_timer_fn+0x1b4/0x220 [40922.452785] iTCO_vendor_support [40922.455231] ? __hrtimer_run_queues+0x1d3/0x360 [40922.457242] i2c_smbus [40922.460295] ? hrtimer_interrupt+0x104/0x240 [40922.464556] crct10dif_pclmul [40922.467780] ? __local_bh_enable+0x50/0x60 [40922.472303] joydev crc32_pclmul [40922.474663] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40922.478925] ipmi_ssif ghash_clmulni_intel [40922.481890] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40922.485981] mei_me [40922.489203] [40922.494422] mei lpc_ich [40922.498513] [40922.503470] sha512_ssse3 acpi_ipmi [40922.505569] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40922.507665] mfd_core [40922.510199] ? queued_spin_lock_slowpath+0x18/0x2e0 [40922.512293] wmi [40922.515779] do_raw_spin_lock+0xb5/0xc0 [40922.521080] ipmi_si [40922.523353] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40922.528223] ipmi_devintf ipmi_msghandler [40922.530063] try_get_cap_refs+0x159/0x880 [ceph] [40922.533891] ioatdma acpi_pad [40922.536080] __ceph_get_caps+0x4a1/0x760 [ceph] [40922.540857] acpi_power_meter [40922.544867] ? __file_remove_privs+0x89/0x110 [40922.549473] ixgbe igb [40922.552455] ? wake_bit_function+0x70/0x70 [40922.556978] mdio crc32c_intel [40922.559946] ? ceph_write_iter+0x260/0x770 [ceph] [40922.564293] nvme ptp [40922.566652] ceph_write_iter+0x260/0x770 [ceph] [40922.570742] nvme_core [40922.573796] ? lock_is_held_type+0xaa/0x120 [40922.578490] i2c_algo_bit pps_core [40922.580764] vfs_write+0x324/0x4f0 [40922.585284] dca fuse [40922.587645] ksys_pwrite64+0x65/0xa0 [40922.591819] [40922.591820] irq event stamp: 247456098 [40922.595218] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40922.598613] hardirqs last enabled at (247456097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40922.600886] do_syscall_64+0x38/0x80 [40922.604456] hardirqs last disabled at (247456098): [] sysvec_apic_timer_interrupt+0xe/0x90 [40922.605946] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40922.609691] softirqs last enabled at (247456096): [] __do_softirq+0x2be/0x371 [40922.614822] RIP: 0033:0x7f48f0eeb93b [40922.625066] softirqs last disabled at (247456091): [] irq_exit_rcu+0xb0/0xd0 [40922.628637] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40922.638458] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40922.643499] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40922.652271] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40922.655841] ORIG_RAX: 0000000000000012 [40922.664439] Workqueue: events delayed_work [ceph] [40922.683177] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40922.692554] [40922.692554] RIP: 0010:lock_acquire+0xf0/0x2b0 [40922.697772] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40922.705241] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40922.709072] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40922.713770] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40922.720895] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40922.722385] [40922.722386] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40922.726737] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40922.733859] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40922.752602] [40922.759721] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40922.804235] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40922.811360] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40922.818484] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40922.826560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40922.832299] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40922.839422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40922.846547] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40922.853671] Call Trace: [40922.856116] [40922.858125] ? watchdog_timer_fn+0x1b4/0x220 [40922.862391] ? __hrtimer_run_queues+0x1d3/0x360 [40922.866916] ? hrtimer_interrupt+0x104/0x240 [40922.871188] ? __local_bh_enable+0x50/0x60 [40922.875279] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40922.880495] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40922.885453] [40922.887549] [40922.889646] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40922.894955] ? lock_acquire+0xf0/0x2b0 [40922.898705] ? igrab+0x43/0x50 [40922.901755] ? lock_release+0xc5/0x260 [40922.905499] _raw_spin_lock+0x30/0x40 [40922.909157] ? igrab+0x1d/0x50 [40922.912222] igrab+0x1d/0x50 [40922.915103] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40922.920251] delayed_work+0x1d8/0x280 [ceph] [40922.924533] ? process_scheduled_works+0x22a/0x510 [40922.929324] process_scheduled_works+0x2a4/0x510 [40922.933938] worker_thread+0x188/0x340 [40922.937689] ? rescuer_thread+0x310/0x310 [40922.941691] kthread+0xf5/0x130 [40922.944829] ? kthread_complete_and_exit+0x20/0x20 [40922.949614] ret_from_fork+0x31/0x40 [40922.953192] ? kthread_complete_and_exit+0x20/0x20 [40922.957976] ret_from_fork_asm+0x11/0x20 [40922.961900] [40923.412162] rcu: INFO: rcu_preempt self-detected stall on CPU [40923.417927] rcu: 0-....: (28330717 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14191147 [40923.428437] rcu: (t=28986297 jiffies g=1309609 q=773054 ncpus=8) [40923.434524] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40923.443474] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40923.450946] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [40923.456431] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [40923.475168] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40923.480387] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40923.487510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40923.494633] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40923.501759] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40923.508884] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40923.516006] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40923.524085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40923.529821] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40923.536947] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40923.544071] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40923.551195] Call Trace: [40923.553638] [40923.555651] ? rcu_dump_cpu_stacks+0xd1/0x180 [40923.560009] ? rcu_sched_clock_irq+0x633/0x1390 [40923.564533] ? lock_is_held_type+0xaa/0x120 [40923.568711] ? lock_is_held_type+0xaa/0x120 [40923.572890] ? tick_sched_do_timer+0xa0/0xa0 [40923.577160] ? rcu_read_lock_sched_held+0x56/0x90 [40923.581858] ? tick_sched_do_timer+0xa0/0xa0 [40923.586121] ? update_process_times+0x5f/0x90 [40923.590474] ? tick_sched_handle.isra.16+0x1e/0x50 [40923.595257] ? tick_sched_timer+0x6f/0x80 [40923.599260] ? __hrtimer_run_queues+0x1d3/0x360 [40923.603787] ? hrtimer_interrupt+0x104/0x240 [40923.608049] ? __local_bh_enable+0x50/0x60 [40923.612142] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40923.617366] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40923.622322] [40923.624419] [40923.626520] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40923.631835] ? queued_spin_lock_slowpath+0x2d/0x2e0 [40923.636713] do_raw_spin_lock+0xb5/0xc0 [40923.640551] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40923.645353] try_get_cap_refs+0x159/0x880 [ceph] [40923.649983] __ceph_get_caps+0x4a1/0x760 [ceph] [40923.654525] ? __file_remove_privs+0x89/0x110 [40923.658881] ? wake_bit_function+0x70/0x70 [40923.662973] ? ceph_write_iter+0x260/0x770 [ceph] [40923.667686] ceph_write_iter+0x260/0x770 [ceph] [40923.672220] ? lock_is_held_type+0xaa/0x120 [40923.676397] vfs_write+0x324/0x4f0 [40923.679797] ksys_pwrite64+0x65/0xa0 [40923.683372] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40923.688505] do_syscall_64+0x38/0x80 [40923.692083] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40923.697125] RIP: 0033:0x7f48f0eeb93b [40923.700697] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40923.719434] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [40923.726992] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40923.734116] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40923.741240] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40923.748363] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40923.755489] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40923.762620] [40923.764806] Sending NMI from CPU 0 to CPUs 7: [40923.769158] NMI backtrace for cpu 7 [40923.769160] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40923.769161] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40923.769162] Workqueue: events delayed_work [ceph] [40923.769180] RIP: 0010:lock_release+0x136/0x260 [40923.769181] Code: 01 48 8d 4c 24 08 44 89 f6 48 89 df e8 93 fa ff ff 0b 05 b5 4e 68 02 0f 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e <83> f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 [40923.769183] RSP: 0018:ffffc90002117d48 EFLAGS: 00000057 [40923.769184] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [40923.769185] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [40923.769185] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [40923.769186] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [40923.769187] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [40923.769187] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40923.769188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40923.769189] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40923.769190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40923.769191] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40923.769191] Call Trace: [40923.769192] [40923.769193] ? nmi_cpu_backtrace+0xad/0x120 [40923.769196] ? nmi_cpu_backtrace_handler+0x11/0x20 [40923.769198] ? nmi_handle+0xd4/0x260 [40923.769202] ? default_do_nmi+0x49/0xf0 [40923.769204] ? exc_nmi+0x11e/0x150 [40923.769205] ? end_repeat_nmi+0x16/0x67 [40923.769208] ? igrab+0x43/0x50 [40923.769210] ? lock_release+0x136/0x260 [40923.769212] ? lock_release+0x136/0x260 [40923.769214] ? lock_release+0x136/0x260 [40923.769215] [40923.769215] [40923.769217] _raw_spin_unlock+0x1b/0x40 [40923.769219] igrab+0x43/0x50 [40923.769221] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40923.769235] delayed_work+0x1d8/0x280 [ceph] [40923.769248] ? process_scheduled_works+0x22a/0x510 [40923.769251] process_scheduled_works+0x2a4/0x510 [40923.769255] worker_thread+0x188/0x340 [40923.769257] ? rescuer_thread+0x310/0x310 [40923.769260] kthread+0xf5/0x130 [40923.769262] ? kthread_complete_and_exit+0x20/0x20 [40923.769264] ret_from_fork+0x31/0x40 [40923.769266] ? kthread_complete_and_exit+0x20/0x20 [40923.769269] ret_from_fork_asm+0x11/0x20 [40923.769273] [40950.169252] watchdog: BUG: soft lockup - CPU#0 stuck for 27023s! [dbench:78683] [40950.176551] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40950.181252] watchdog: BUG: soft lockup - CPU#7 stuck for 36020s! [kworker/7:4:78808] [40950.231991] irq event stamp: 73119168 [40950.231993] hardirqs last enabled at (73119167): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40950.239723] Modules linked in: [40950.243379] hardirqs last disabled at (73119168): [] sysvec_apic_timer_interrupt+0xe/0x90 [40950.253538] ceph [40950.256588] softirqs last enabled at (73119048): [] __do_softirq+0x2be/0x371 [40950.266313] libceph [40950.268236] softirqs last disabled at (73119043): [] irq_exit_rcu+0xb0/0xd0 [40950.276929] dns_resolver fscache [40950.279114] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40950.287624] netfs veth [40950.290954] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40950.290955] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [40950.299896] nft_chain_nat xt_MASQUERADE [40950.302340] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [40950.309811] nf_nat nf_conntrack [40950.315290] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40950.319206] nf_defrag_ipv6 [40950.337945] [40950.337946] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40950.341168] nf_defrag_ipv4 nft_compat [40950.346387] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40950.349175] nf_tables nfnetlink [40950.350668] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40950.357791] bridge stp [40950.361534] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40950.368659] llc xfs [40950.371885] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40950.379007] libcrc32c sunrpc [40950.381453] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40950.388577] intel_rapl_msr intel_rapl_common [40950.390760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40950.397884] sb_edac x86_pkg_temp_thermal [40950.400847] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40950.408931] coretemp kvm_intel [40950.413284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40950.419022] kvm irqbypass [40950.423027] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40950.430150] i2c_i801 iTCO_wdt [40950.433288] Call Trace: [40950.433289] [40950.440411] iTCO_vendor_support [40950.443117] ? watchdog_timer_fn+0x1b4/0x220 [40950.450239] i2c_smbus crct10dif_pclmul [40950.453293] ? __hrtimer_run_queues+0x1d3/0x360 [40950.455734] joydev crc32_pclmul [40950.457748] ? hrtimer_interrupt+0x104/0x240 [40950.460971] ipmi_ssif ghash_clmulni_intel [40950.465235] ? __local_bh_enable+0x50/0x60 [40950.469064] mei_me mei [40950.473591] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40950.476814] lpc_ich sha512_ssse3 [40950.481078] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40950.485168] acpi_ipmi mfd_core [40950.489259] [40950.491703] wmi ipmi_si [40950.496931] [40950.496932] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40950.500240] ipmi_devintf ipmi_msghandler [40950.505200] ? queued_spin_lock_slowpath+0x18/0x2e0 [40950.508334] ioatdma acpi_pad [40950.510432] do_raw_spin_lock+0xb5/0xc0 [40950.512962] acpi_power_meter [40950.515061] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40950.520363] ixgbe igb [40950.524370] try_get_cap_refs+0x159/0x880 [ceph] [40950.529238] mdio crc32c_intel [40950.532207] __ceph_get_caps+0x4a1/0x760 [ceph] [40950.536033] nvme ptp [40950.539001] ? __file_remove_privs+0x89/0x110 [40950.543782] nvme_core i2c_algo_bit [40950.546139] ? wake_bit_function+0x70/0x70 [40950.550750] pps_core dca fuse [40950.553803] ? ceph_write_iter+0x260/0x770 [ceph] [40950.558323] [40950.558324] irq event stamp: 247675050 [40950.560596] ceph_write_iter+0x260/0x770 [ceph] [40950.564945] hardirqs last enabled at (247675049): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40950.568435] ? lock_is_held_type+0xaa/0x120 [40950.572520] hardirqs last disabled at (247675050): [] sysvec_apic_timer_interrupt+0xe/0x90 [40950.575573] vfs_write+0x324/0x4f0 [40950.580270] softirqs last enabled at (247675048): [] __do_softirq+0x2be/0x371 [40950.581764] ksys_pwrite64+0x65/0xa0 [40950.585503] softirqs last disabled at (247675043): [] irq_exit_rcu+0xb0/0xd0 [40950.590028] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40950.600273] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40950.604450] do_syscall_64+0x38/0x80 [40950.614260] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40950.617658] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40950.626428] Workqueue: events delayed_work [ceph] [40950.629999] RIP: 0033:0x7f48f0eeb93b [40950.638595] [40950.638596] RIP: 0010:_raw_spin_lock+0x18/0x40 [40950.643728] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40950.653104] Code: f6 5b 5d e9 ba 18 b5 ff cc cc cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 53 48 89 fb 65 ff 05 14 f4 47 7e ff 74 24 08 <48> 8d 7f 18 45 31 c9 31 c9 41 b8 01 00 00 00 31 d2 31 f6 e8 30 0c [40950.656676] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40950.664146] RSP: 0018:ffffc90002117d88 EFLAGS: 00000202 [40950.669190] ORIG_RAX: 0000000000000012 [40950.673887] [40950.673888] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34500 RCX: ffff8881a1d34bc8 [40950.677457] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40950.678950] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34500 [40950.683386] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40950.702124] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [40950.720861] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40950.726079] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [40950.731296] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40950.735127] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [40950.736618] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40950.743743] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40950.750873] [40950.757991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40950.823868] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40950.830993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40950.838116] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40950.845240] Call Trace: [40950.847685] [40950.849696] ? watchdog_timer_fn+0x1b4/0x220 [40950.853959] ? __hrtimer_run_queues+0x1d3/0x360 [40950.858487] ? hrtimer_interrupt+0x104/0x240 [40950.862757] ? __local_bh_enable+0x50/0x60 [40950.866848] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40950.872065] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40950.877022] [40950.879117] [40950.881218] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40950.886534] ? _raw_spin_lock+0x18/0x40 [40950.890370] ? igrab+0x1d/0x50 [40950.893419] igrab+0x1d/0x50 [40950.896298] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40950.901446] delayed_work+0x1d8/0x280 [ceph] [40950.905728] ? process_scheduled_works+0x22a/0x510 [40950.910519] process_scheduled_works+0x2a4/0x510 [40950.915135] worker_thread+0x188/0x340 [40950.918882] ? rescuer_thread+0x310/0x310 [40950.922886] kthread+0xf5/0x130 [40950.926023] ? kthread_complete_and_exit+0x20/0x20 [40950.930809] ret_from_fork+0x31/0x40 [40950.934378] ? kthread_complete_and_exit+0x20/0x20 [40950.939162] ret_from_fork_asm+0x11/0x20 [40950.943087] [40978.169721] watchdog: BUG: soft lockup - CPU#0 stuck for 27049s! [dbench:78683] [40978.177023] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [40978.181720] watchdog: BUG: soft lockup - CPU#7 stuck for 36046s! [kworker/7:4:78808] [40978.232472] irq event stamp: 73177436 [40978.232473] hardirqs last enabled at (73177435): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40978.240204] Modules linked in: [40978.243859] hardirqs last disabled at (73177436): [] sysvec_apic_timer_interrupt+0xe/0x90 [40978.254017] ceph [40978.257067] softirqs last enabled at (73177310): [] __do_softirq+0x2be/0x371 [40978.266792] libceph [40978.268715] softirqs last disabled at (73177305): [] irq_exit_rcu+0xb0/0xd0 [40978.277401] dns_resolver [40978.279586] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40978.288096] fscache netfs [40978.290729] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40978.299673] veth nft_chain_nat [40978.302394] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [40978.309867] xt_MASQUERADE nf_nat [40978.313004] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [40978.318480] nf_conntrack [40978.321793] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [40978.340528] nf_defrag_ipv6 nf_defrag_ipv4 [40978.343146] [40978.343147] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [40978.348365] nft_compat nf_tables [40978.352455] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [40978.353947] nfnetlink bridge [40978.361070] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [40978.364396] stp llc [40978.371522] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [40978.374486] xfs libcrc32c [40978.381611] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [40978.383794] sunrpc [40978.390919] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [40978.393622] intel_rapl_msr intel_rapl_common [40978.400745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40978.402844] sb_edac [40978.410920] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [40978.415271] x86_pkg_temp_thermal coretemp [40978.421009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40978.423194] kvm_intel kvm [40978.430317] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40978.434408] irqbypass i2c_i801 [40978.441532] Call Trace: [40978.444235] iTCO_wdt [40978.451364] [40978.454497] iTCO_vendor_support i2c_smbus [40978.456943] ? watchdog_timer_fn+0x1b4/0x220 [40978.459214] crct10dif_pclmul [40978.461224] ? __hrtimer_run_queues+0x1d3/0x360 [40978.465315] joydev crc32_pclmul [40978.469581] ? hrtimer_interrupt+0x104/0x240 [40978.472542] ipmi_ssif [40978.477067] ? __local_bh_enable+0x50/0x60 [40978.480291] ghash_clmulni_intel mei_me [40978.484557] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40978.486913] mei lpc_ich [40978.491004] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40978.494833] sha512_ssse3 acpi_ipmi [40978.500051] [40978.502583] mfd_core [40978.507539] [40978.511024] wmi ipmi_si [40978.513121] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40978.515397] ipmi_devintf ipmi_msghandler [40978.517502] ? queued_spin_lock_slowpath+0x16/0x2e0 [40978.520027] ioatdma acpi_pad [40978.525333] do_raw_spin_lock+0xb5/0xc0 [40978.529337] acpi_power_meter ixgbe igb [40978.534208] __cap_delay_requeue+0xa7/0x1d0 [ceph] [40978.537170] mdio crc32c_intel [40978.541006] try_get_cap_refs+0x159/0x880 [ceph] [40978.544832] nvme ptp [40978.549621] __ceph_get_caps+0x4a1/0x760 [ceph] [40978.552667] nvme_core i2c_algo_bit [40978.557283] ? __file_remove_privs+0x89/0x110 [40978.559549] pps_core dca [40978.564073] ? wake_bit_function+0x70/0x70 [40978.567557] fuse [40978.571911] ? ceph_write_iter+0x260/0x770 [ceph] [40978.574525] irq event stamp: 247896160 [40978.578617] ceph_write_iter+0x260/0x770 [ceph] [40978.580540] hardirqs last enabled at (247896159): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40978.585242] ? lock_is_held_type+0xaa/0x120 [40978.588982] hardirqs last disabled at (247896160): [] sysvec_apic_timer_interrupt+0xe/0x90 [40978.593508] vfs_write+0x324/0x4f0 [40978.603748] softirqs last enabled at (247896158): [] __do_softirq+0x2be/0x371 [40978.607930] ksys_pwrite64+0x65/0xa0 [40978.617737] softirqs last disabled at (247896153): [] irq_exit_rcu+0xb0/0xd0 [40978.621135] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [40978.629906] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [40978.633479] do_syscall_64+0x38/0x80 [40978.642075] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [40978.647206] entry_SYSCALL_64_after_hwframe+0x63/0xcd [40978.656584] Workqueue: events delayed_work [ceph] [40978.660153] RIP: 0033:0x7f48f0eeb93b [40978.667623] [40978.667624] RIP: 0010:lock_acquire+0xf0/0x2b0 [40978.672670] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [40978.677366] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [40978.680935] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [40978.682426] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [40978.686778] ORIG_RAX: 0000000000000012 [40978.705516] [40978.705517] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [40978.724252] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [40978.729470] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [40978.734687] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [40978.738519] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [40978.740010] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [40978.747133] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [40978.754257] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [40978.761381] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [40978.768506] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [40978.775630] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [40978.782759] [40978.789879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40978.827249] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [40978.834373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40978.841497] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40978.848623] Call Trace: [40978.851066] [40978.853079] ? watchdog_timer_fn+0x1b4/0x220 [40978.857343] ? __hrtimer_run_queues+0x1d3/0x360 [40978.861870] ? hrtimer_interrupt+0x104/0x240 [40978.866139] ? __local_bh_enable+0x50/0x60 [40978.870231] ? __sysvec_apic_timer_interrupt+0x61/0x160 [40978.875455] ? sysvec_apic_timer_interrupt+0x6d/0x90 [40978.880412] [40978.882510] [40978.884607] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [40978.889916] ? lock_acquire+0xf0/0x2b0 [40978.893665] ? igrab+0x43/0x50 [40978.896717] ? lock_release+0xc5/0x260 [40978.900460] _raw_spin_lock+0x30/0x40 [40978.904127] ? igrab+0x1d/0x50 [40978.907176] igrab+0x1d/0x50 [40978.910056] ceph_check_delayed_caps+0xb7/0x270 [ceph] [40978.915204] delayed_work+0x1d8/0x280 [ceph] [40978.919486] ? process_scheduled_works+0x22a/0x510 [40978.924274] process_scheduled_works+0x2a4/0x510 [40978.928892] worker_thread+0x188/0x340 [40978.932641] ? rescuer_thread+0x310/0x310 [40978.936644] kthread+0xf5/0x130 [40978.939783] ? kthread_complete_and_exit+0x20/0x20 [40978.944573] ret_from_fork+0x31/0x40 [40978.948152] ? kthread_complete_and_exit+0x20/0x20 [40978.952938] ret_from_fork_asm+0x11/0x20 [40978.956870] [41006.170188] watchdog: BUG: soft lockup - CPU#0 stuck for 27075s! [dbench:78683] [41006.177491] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41006.182188] watchdog: BUG: soft lockup - CPU#7 stuck for 36072s! [kworker/7:4:78808] [41006.232932] irq event stamp: 73235618 [41006.232933] hardirqs last enabled at (73235617): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41006.240663] Modules linked in: [41006.244322] hardirqs last disabled at (73235618): [] sysvec_apic_timer_interrupt+0xe/0x90 [41006.254480] ceph [41006.257530] softirqs last enabled at (73235562): [] __do_softirq+0x2be/0x371 [41006.267253] libceph dns_resolver [41006.269176] softirqs last disabled at (73235557): [] irq_exit_rcu+0xb0/0xd0 [41006.277861] fscache netfs [41006.281173] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41006.289684] veth nft_chain_nat [41006.292387] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41006.301331] xt_MASQUERADE nf_nat [41006.304469] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41006.311941] nf_conntrack nf_defrag_ipv6 [41006.315250] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41006.320727] nf_defrag_ipv4 nft_compat [41006.324646] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41006.343384] nf_tables nfnetlink [41006.347128] [41006.347128] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41006.352345] bridge [41006.355583] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41006.357077] stp llc [41006.364201] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41006.366297] xfs libcrc32c [41006.373423] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41006.375605] sunrpc intel_rapl_msr [41006.382729] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41006.385435] intel_rapl_common [41006.392559] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41006.395974] sb_edac x86_pkg_temp_thermal [41006.403098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41006.406148] coretemp kvm_intel [41006.414224] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41006.418230] kvm irqbypass [41006.423966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41006.427106] i2c_i801 iTCO_wdt [41006.434227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41006.436931] iTCO_vendor_support [41006.444057] Call Trace: [41006.444058] [41006.447106] i2c_smbus crct10dif_pclmul [41006.454234] ? watchdog_timer_fn+0x1b4/0x220 [41006.457455] joydev crc32_pclmul [41006.459900] ? __hrtimer_run_queues+0x1d3/0x360 [41006.461910] ipmi_ssif ghash_clmulni_intel [41006.465744] ? hrtimer_interrupt+0x104/0x240 [41006.470004] mei_me mei [41006.473229] ? __local_bh_enable+0x50/0x60 [41006.477754] lpc_ich sha512_ssse3 [41006.481848] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41006.486109] acpi_ipmi mfd_core [41006.488553] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41006.492644] wmi ipmi_si [41006.495955] [41006.501172] ipmi_devintf ipmi_msghandler [41006.504308] [41006.509265] ioatdma acpi_pad [41006.511797] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41006.513895] acpi_power_meter [41006.517903] ? queued_spin_lock_slowpath+0x16/0x2e0 [41006.519996] ixgbe igb [41006.522961] do_raw_spin_lock+0xb5/0xc0 [41006.528263] mdio crc32c_intel nvme [41006.531229] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41006.536098] ptp nvme_core [41006.538460] try_get_cap_refs+0x159/0x880 [ceph] [41006.542288] i2c_algo_bit [41006.545775] __ceph_get_caps+0x4a1/0x760 [ceph] [41006.550555] pps_core dca [41006.553263] ? __file_remove_privs+0x89/0x110 [41006.557875] fuse [41006.560497] ? wake_bit_function+0x70/0x70 [41006.565018] irq event stamp: 248116984 [41006.567639] ? ceph_write_iter+0x260/0x770 [ceph] [41006.571987] hardirqs last enabled at (248116983): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41006.573911] ceph_write_iter+0x260/0x770 [ceph] [41006.578002] hardirqs last disabled at (248116984): [] sysvec_apic_timer_interrupt+0xe/0x90 [41006.581751] ? lock_is_held_type+0xaa/0x120 [41006.586444] softirqs last enabled at (248116982): [] __do_softirq+0x2be/0x371 [41006.596690] vfs_write+0x324/0x4f0 [41006.601213] softirqs last disabled at (248116977): [] irq_exit_rcu+0xb0/0xd0 [41006.611026] ksys_pwrite64+0x65/0xa0 [41006.615201] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41006.623971] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41006.627369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41006.635967] do_syscall_64+0x38/0x80 [41006.639538] Workqueue: events delayed_work [ceph] [41006.648915] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41006.654045] [41006.654046] RIP: 0010:ceph_check_delayed_caps+0x9e/0x270 [ceph] [41006.661516] RIP: 0033:0x7f48f0eeb93b [41006.665087] Code: ff ff 83 e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 <48> 89 10 48 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 [41006.669786] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41006.674829] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [41006.676320] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41006.682229] [41006.682230] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [41006.685801] ORIG_RAX: 0000000000000012 [41006.704537] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [41006.723276] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41006.728492] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [41006.733710] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41006.735202] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [41006.742324] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41006.746156] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41006.753280] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41006.760404] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41006.767529] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41006.774653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41006.781782] [41006.788902] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41006.833397] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41006.840521] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41006.847646] Call Trace: [41006.850089] [41006.852102] ? watchdog_timer_fn+0x1b4/0x220 [41006.856364] ? __hrtimer_run_queues+0x1d3/0x360 [41006.860892] ? hrtimer_interrupt+0x104/0x240 [41006.865160] ? __local_bh_enable+0x50/0x60 [41006.869253] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41006.874477] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41006.879435] [41006.881533] [41006.883629] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41006.888939] ? ceph_check_delayed_caps+0x9e/0x270 [ceph] [41006.894258] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [41006.899595] delayed_work+0x1d8/0x280 [ceph] [41006.903877] ? process_scheduled_works+0x22a/0x510 [41006.908667] process_scheduled_works+0x2a4/0x510 [41006.913283] worker_thread+0x188/0x340 [41006.917032] ? rescuer_thread+0x310/0x310 [41006.921036] kthread+0xf5/0x130 [41006.924174] ? kthread_complete_and_exit+0x20/0x20 [41006.928957] ret_from_fork+0x31/0x40 [41006.932527] ? kthread_complete_and_exit+0x20/0x20 [41006.937314] ret_from_fork_asm+0x11/0x20 [41006.941235] [41034.170657] watchdog: BUG: soft lockup - CPU#0 stuck for 27102s! [dbench:78683] [41034.177963] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41034.182656] watchdog: BUG: soft lockup - CPU#7 stuck for 36099s! [kworker/7:4:78808] [41034.233411] irq event stamp: 73293846 [41034.233412] hardirqs last enabled at (73293845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41034.241143] Modules linked in: [41034.244799] hardirqs last disabled at (73293846): [] sysvec_apic_timer_interrupt+0xe/0x90 [41034.254959] ceph [41034.258008] softirqs last enabled at (73293780): [] __do_softirq+0x2be/0x371 [41034.267731] libceph [41034.269655] softirqs last disabled at (73293775): [] irq_exit_rcu+0xb0/0xd0 [41034.278340] dns_resolver fscache [41034.280526] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41034.289050] netfs veth [41034.292364] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41034.301307] nft_chain_nat [41034.303769] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41034.311241] xt_MASQUERADE nf_nat [41034.313945] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41034.319422] nf_conntrack nf_defrag_ipv6 [41034.322732] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41034.341470] nf_defrag_ipv4 nft_compat [41034.345388] [41034.345389] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41034.350605] nf_tables nfnetlink [41034.354349] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41034.355839] bridge stp [41034.362963] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41034.366187] llc xfs [41034.373310] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41034.375773] libcrc32c sunrpc [41034.382896] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41034.385080] intel_rapl_msr [41034.392204] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41034.395170] intel_rapl_common sb_edac [41034.402295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41034.405085] x86_pkg_temp_thermal coretemp [41034.413160] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41034.416907] kvm_intel [41034.422642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41034.426733] kvm irqbypass [41034.433858] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41034.436214] i2c_i801 iTCO_wdt [41034.443339] Call Trace: [41034.443341] [41034.446054] iTCO_vendor_support i2c_smbus [41034.453178] ? watchdog_timer_fn+0x1b4/0x220 [41034.456228] crct10dif_pclmul joydev [41034.458672] ? __hrtimer_run_queues+0x1d3/0x360 [41034.460683] crc32_pclmul [41034.464775] ? hrtimer_interrupt+0x104/0x240 [41034.469053] ipmi_ssif ghash_clmulni_intel [41034.472625] ? __local_bh_enable+0x50/0x60 [41034.477149] mei_me mei [41034.479768] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41034.484031] lpc_ich sha512_ssse3 [41034.488137] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41034.492228] acpi_ipmi [41034.494674] [41034.499891] mfd_core wmi [41034.503200] [41034.508159] ipmi_si ipmi_devintf [41034.510516] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41034.512614] ipmi_msghandler ioatdma [41034.515234] ? queued_spin_lock_slowpath+0x16/0x2e0 [41034.517329] acpi_pad acpi_power_meter [41034.520641] do_raw_spin_lock+0xb5/0xc0 [41034.525943] ixgbe igb [41034.529515] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41034.534386] mdio crc32c_intel [41034.538133] try_get_cap_refs+0x159/0x880 [ceph] [41034.541960] nvme ptp [41034.544339] __ceph_get_caps+0x4a1/0x760 [ceph] [41034.549117] nvme_core i2c_algo_bit [41034.552174] ? __file_remove_privs+0x89/0x110 [41034.556778] pps_core dca [41034.559057] ? wake_bit_function+0x70/0x70 [41034.563583] fuse [41034.567069] ? ceph_write_iter+0x260/0x770 [ceph] [41034.571418] irq event stamp: 248337924 [41034.574035] ceph_write_iter+0x260/0x770 [ceph] [41034.578144] hardirqs last enabled at (248337923): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41034.580072] ? lock_is_held_type+0xaa/0x120 [41034.584765] hardirqs last disabled at (248337924): [] sysvec_apic_timer_interrupt+0xe/0x90 [41034.588511] vfs_write+0x324/0x4f0 [41034.593034] softirqs last enabled at (248337922): [] __do_softirq+0x2be/0x371 [41034.603301] ksys_pwrite64+0x65/0xa0 [41034.607470] softirqs last disabled at (248337917): [] irq_exit_rcu+0xb0/0xd0 [41034.617284] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41034.620681] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41034.629452] do_syscall_64+0x38/0x80 [41034.633021] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41034.641620] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41034.646750] Workqueue: events delayed_work [ceph] [41034.656128] RIP: 0033:0x7f48f0eeb93b [41034.659698] [41034.659699] RIP: 0010:lock_acquire+0xf0/0x2b0 [41034.667171] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41034.672212] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41034.676910] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41034.680480] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41034.681971] ORIG_RAX: 0000000000000012 [41034.686323] [41034.686324] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41034.705077] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41034.723816] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41034.729033] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41034.734267] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41034.738098] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41034.739589] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41034.746713] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41034.753837] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41034.760960] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41034.768086] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41034.775215] [41034.782334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41034.826829] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41034.833953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41034.841077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41034.848201] Call Trace: [41034.850646] [41034.852658] ? watchdog_timer_fn+0x1b4/0x220 [41034.856922] ? __hrtimer_run_queues+0x1d3/0x360 [41034.861448] ? hrtimer_interrupt+0x104/0x240 [41034.865710] ? __local_bh_enable+0x50/0x60 [41034.869802] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41034.875027] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41034.879983] [41034.882081] [41034.884179] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41034.889487] ? lock_acquire+0xf0/0x2b0 [41034.893237] ? igrab+0x43/0x50 [41034.896285] ? lock_release+0xc5/0x260 [41034.900032] _raw_spin_lock+0x30/0x40 [41034.903715] ? igrab+0x1d/0x50 [41034.906764] igrab+0x1d/0x50 [41034.909643] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41034.914791] delayed_work+0x1d8/0x280 [ceph] [41034.919073] ? process_scheduled_works+0x22a/0x510 [41034.923862] process_scheduled_works+0x2a4/0x510 [41034.928480] worker_thread+0x188/0x340 [41034.932227] ? rescuer_thread+0x310/0x310 [41034.936231] kthread+0xf5/0x130 [41034.939371] ? kthread_complete_and_exit+0x20/0x20 [41034.944163] ret_from_fork+0x31/0x40 [41034.947742] ? kthread_complete_and_exit+0x20/0x20 [41034.952525] ret_from_fork_asm+0x11/0x20 [41034.956457] [41062.171124] watchdog: BUG: soft lockup - CPU#0 stuck for 27128s! [dbench:78683] [41062.178422] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41062.183124] watchdog: BUG: soft lockup - CPU#7 stuck for 36125s! [kworker/7:4:78808] [41062.233864] irq event stamp: 73352082 [41062.233865] hardirqs last enabled at (73352081): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41062.241596] Modules linked in: [41062.245252] hardirqs last disabled at (73352082): [] sysvec_apic_timer_interrupt+0xe/0x90 [41062.255411] ceph [41062.258461] softirqs last enabled at (73352006): [] __do_softirq+0x2be/0x371 [41062.268186] libceph [41062.270108] softirqs last disabled at (73352001): [] irq_exit_rcu+0xb0/0xd0 [41062.278805] dns_resolver [41062.280987] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41062.289497] fscache netfs [41062.292130] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41062.301075] veth nft_chain_nat [41062.303782] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41062.311250] xt_MASQUERADE nf_nat [41062.314389] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41062.319866] nf_conntrack nf_defrag_ipv6 [41062.323175] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41062.341913] nf_defrag_ipv4 [41062.345831] [41062.345832] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41062.351048] nft_compat nf_tables [41062.353839] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41062.355329] nfnetlink bridge [41062.362455] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41062.365764] stp llc [41062.372890] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41062.375852] xfs libcrc32c [41062.382977] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41062.385160] sunrpc intel_rapl_msr [41062.392286] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41062.394990] intel_rapl_common [41062.402113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41062.405522] sb_edac x86_pkg_temp_thermal [41062.413598] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41062.416648] coretemp kvm_intel [41062.422386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41062.426389] kvm irqbypass [41062.433520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41062.436659] i2c_i801 [41062.443800] Call Trace: [41062.443802] [41062.446521] iTCO_wdt [41062.453648] ? watchdog_timer_fn+0x1b4/0x220 [41062.455918] iTCO_vendor_support i2c_smbus [41062.458364] ? __hrtimer_run_queues+0x1d3/0x360 [41062.460372] crct10dif_pclmul joydev [41062.462646] ? hrtimer_interrupt+0x104/0x240 [41062.466907] crc32_pclmul ipmi_ssif [41062.470998] ? __local_bh_enable+0x50/0x60 [41062.475521] ghash_clmulni_intel [41062.479095] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41062.483356] mei_me [41062.486841] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41062.490930] mei lpc_ich [41062.494155] [41062.499373] sha512_ssse3 acpi_ipmi [41062.501469] [41062.501470] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41062.506427] mfd_core wmi [41062.508961] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41062.511055] ipmi_si ipmi_devintf [41062.514542] do_raw_spin_lock+0xb5/0xc0 [41062.516636] ipmi_msghandler ioatdma [41062.521944] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41062.524559] acpi_pad [41062.529433] try_get_cap_refs+0x159/0x880 [ceph] [41062.532741] acpi_power_meter ixgbe [41062.536575] __ceph_get_caps+0x4a1/0x760 [ceph] [41062.540142] igb mdio [41062.544930] ? __file_remove_privs+0x89/0x110 [41062.547195] crc32c_intel nvme [41062.551807] ? wake_bit_function+0x70/0x70 [41062.555292] ptp nvme_core [41062.559817] ? ceph_write_iter+0x260/0x770 [ceph] [41062.562085] i2c_algo_bit pps_core [41062.566437] ceph_write_iter+0x260/0x770 [ceph] [41062.569488] dca fuse [41062.573583] ? lock_is_held_type+0xaa/0x120 [41062.576282] [41062.576283] irq event stamp: 248558706 [41062.580982] vfs_write+0x324/0x4f0 [41062.584378] hardirqs last enabled at (248558705): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41062.588906] ksys_pwrite64+0x65/0xa0 [41062.591172] hardirqs last disabled at (248558706): [] sysvec_apic_timer_interrupt+0xe/0x90 [41062.595350] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41062.596840] softirqs last enabled at (248558704): [] __do_softirq+0x2be/0x371 [41062.600584] do_syscall_64+0x38/0x80 [41062.603981] softirqs last disabled at (248558699): [] irq_exit_rcu+0xb0/0xd0 [41062.614226] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41062.617795] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41062.627608] RIP: 0033:0x7f48f0eeb93b [41062.632738] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41062.632739] Workqueue: events delayed_work [ceph] [41062.641526] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41062.645096] [41062.645097] RIP: 0010:lock_acquire+0xf0/0x2b0 [41062.653694] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41062.658737] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41062.668115] ORIG_RAX: 0000000000000012 [41062.671687] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41062.679158] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41062.683856] [41062.683857] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41062.702593] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41062.704082] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41062.708433] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41062.713651] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41062.732390] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41062.736220] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41062.741437] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41062.748562] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41062.750057] [41062.757177] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41062.824432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41062.830169] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41062.837293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41062.844417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41062.851541] Call Trace: [41062.853986] [41062.855996] ? watchdog_timer_fn+0x1b4/0x220 [41062.860261] ? __hrtimer_run_queues+0x1d3/0x360 [41062.864804] ? hrtimer_interrupt+0x104/0x240 [41062.869073] ? __local_bh_enable+0x50/0x60 [41062.873168] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41062.878392] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41062.883349] [41062.885445] [41062.887544] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41062.892852] ? lock_acquire+0xf0/0x2b0 [41062.896601] ? igrab+0x43/0x50 [41062.899650] ? lock_release+0xc5/0x260 [41062.903397] _raw_spin_lock+0x30/0x40 [41062.907062] ? igrab+0x1d/0x50 [41062.910112] igrab+0x1d/0x50 [41062.912989] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41062.918139] delayed_work+0x1d8/0x280 [ceph] [41062.922421] ? process_scheduled_works+0x22a/0x510 [41062.927212] process_scheduled_works+0x2a4/0x510 [41062.931827] worker_thread+0x188/0x340 [41062.935576] ? rescuer_thread+0x310/0x310 [41062.939588] kthread+0xf5/0x130 [41062.942727] ? kthread_complete_and_exit+0x20/0x20 [41062.947531] ret_from_fork+0x31/0x40 [41062.951104] ? kthread_complete_and_exit+0x20/0x20 [41062.955891] ret_from_fork_asm+0x11/0x20 [41062.959822] [41090.171601] watchdog: BUG: soft lockup - CPU#0 stuck for 27154s! [dbench:78683] [41090.178903] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41090.183601] watchdog: BUG: soft lockup - CPU#7 stuck for 36151s! [kworker/7:4:78808] [41090.234353] irq event stamp: 73410342 [41090.234354] hardirqs last enabled at (73410341): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41090.242084] Modules linked in: [41090.245740] hardirqs last disabled at (73410342): [] sysvec_apic_timer_interrupt+0xe/0x90 [41090.255898] ceph [41090.258948] softirqs last enabled at (73410220): [] __do_softirq+0x2be/0x371 [41090.268671] libceph [41090.270597] softirqs last disabled at (73410215): [] irq_exit_rcu+0xb0/0xd0 [41090.279281] dns_resolver [41090.281467] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41090.289993] fscache netfs [41090.292611] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41090.301554] veth nft_chain_nat [41090.304276] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41090.311747] xt_MASQUERADE nf_nat [41090.314884] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41090.320362] nf_conntrack nf_defrag_ipv6 [41090.323672] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41090.342410] nf_defrag_ipv4 nft_compat [41090.346328] [41090.346329] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41090.351544] nf_tables [41090.355289] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41090.356781] nfnetlink bridge [41090.363903] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41090.366275] stp llc [41090.373403] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41090.376366] xfs libcrc32c [41090.383491] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41090.385675] sunrpc intel_rapl_msr [41090.392799] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41090.395504] intel_rapl_common [41090.402628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41090.406025] sb_edac x86_pkg_temp_thermal [41090.414102] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41090.417153] coretemp kvm_intel [41090.422890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41090.426894] kvm irqbypass [41090.434017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41090.437155] i2c_i801 iTCO_wdt [41090.444280] Call Trace: [41090.444283] [41090.446998] iTCO_vendor_support i2c_smbus [41090.454128] ? watchdog_timer_fn+0x1b4/0x220 [41090.457176] crct10dif_pclmul [41090.459622] ? __hrtimer_run_queues+0x1d3/0x360 [41090.461631] joydev crc32_pclmul [41090.465724] ? hrtimer_interrupt+0x104/0x240 [41090.469998] ipmi_ssif [41090.472958] ? __local_bh_enable+0x50/0x60 [41090.477484] ghash_clmulni_intel mei_me [41090.480708] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41090.484971] mei [41090.487329] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41090.491419] lpc_ich sha512_ssse3 [41090.495250] [41090.500466] acpi_ipmi mfd_core [41090.502303] [41090.502305] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41090.507277] wmi [41090.510595] ? queued_spin_lock_slowpath+0x16/0x2e0 [41090.512687] ipmi_si ipmi_devintf [41090.515828] do_raw_spin_lock+0xb5/0xc0 [41090.517922] ipmi_msghandler ioatdma [41090.523228] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41090.525063] acpi_pad acpi_power_meter [41090.529938] try_get_cap_refs+0x159/0x880 [ceph] [41090.533244] ixgbe igb [41090.537081] __ceph_get_caps+0x4a1/0x760 [ceph] [41090.540648] mdio crc32c_intel [41090.545436] ? __file_remove_privs+0x89/0x110 [41090.549175] nvme ptp [41090.553785] ? wake_bit_function+0x70/0x70 [41090.556142] nvme_core i2c_algo_bit [41090.560669] ? ceph_write_iter+0x260/0x770 [ceph] [41090.563718] pps_core dca [41090.568068] ceph_write_iter+0x260/0x770 [ceph] [41090.570338] fuse [41090.574432] ? lock_is_held_type+0xaa/0x120 [41090.577914] irq event stamp: 248779534 [41090.582613] vfs_write+0x324/0x4f0 [41090.585228] hardirqs last enabled at (248779533): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41090.589759] ksys_pwrite64+0x65/0xa0 [41090.591677] hardirqs last disabled at (248779534): [] sysvec_apic_timer_interrupt+0xe/0x90 [41090.595855] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41090.599598] softirqs last enabled at (248779532): [] __do_softirq+0x2be/0x371 [41090.603004] do_syscall_64+0x38/0x80 [41090.613249] softirqs last disabled at (248779527): [] irq_exit_rcu+0xb0/0xd0 [41090.616819] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41090.626630] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41090.631761] RIP: 0033:0x7f48f0eeb93b [41090.640531] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41090.644102] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41090.652699] Workqueue: events delayed_work [ceph] [41090.657745] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41090.667122] [41090.667123] RIP: 0010:lock_acquire+0xf0/0x2b0 [41090.670691] ORIG_RAX: 0000000000000012 [41090.678164] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41090.696901] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41090.701599] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41090.706815] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41090.708305] [41090.708306] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41090.712658] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41090.716489] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41090.735224] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41090.742349] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41090.747567] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41090.754692] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41090.756188] [41090.763311] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41090.815369] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41090.823444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41090.829183] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41090.836305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41090.843432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41090.850556] Call Trace: [41090.853008] [41090.855020] ? watchdog_timer_fn+0x1b4/0x220 [41090.859287] ? __hrtimer_run_queues+0x1d3/0x360 [41090.863819] ? hrtimer_interrupt+0x104/0x240 [41090.868088] ? __local_bh_enable+0x50/0x60 [41090.872182] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41090.877406] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41090.882361] [41090.884459] [41090.886558] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41090.891866] ? lock_acquire+0xf0/0x2b0 [41090.895614] ? igrab+0x43/0x50 [41090.898666] ? lock_release+0xc5/0x260 [41090.902410] _raw_spin_lock+0x30/0x40 [41090.906067] ? igrab+0x1d/0x50 [41090.909117] igrab+0x1d/0x50 [41090.912009] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41090.917155] delayed_work+0x1d8/0x280 [ceph] [41090.921436] ? process_scheduled_works+0x22a/0x510 [41090.926226] process_scheduled_works+0x2a4/0x510 [41090.930840] worker_thread+0x188/0x340 [41090.934589] ? rescuer_thread+0x310/0x310 [41090.938593] kthread+0xf5/0x130 [41090.941741] ? kthread_complete_and_exit+0x20/0x20 [41090.946532] ret_from_fork+0x31/0x40 [41090.950112] ? kthread_complete_and_exit+0x20/0x20 [41090.954895] ret_from_fork_asm+0x11/0x20 [41090.958828] [41118.172081] watchdog: BUG: soft lockup - CPU#0 stuck for 27180s! [dbench:78683] [41118.179382] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41118.184081] watchdog: BUG: soft lockup - CPU#7 stuck for 36177s! [kworker/7:4:78808] [41118.234831] irq event stamp: 73468588 [41118.234832] hardirqs last enabled at (73468587): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41118.242563] Modules linked in: ceph [41118.246219] hardirqs last disabled at (73468588): [] sysvec_apic_timer_interrupt+0xe/0x90 [41118.256376] libceph [41118.259861] softirqs last enabled at (73468474): [] __do_softirq+0x2be/0x371 [41118.269585] dns_resolver [41118.271768] softirqs last disabled at (73468469): [] irq_exit_rcu+0xb0/0xd0 [41118.280454] fscache [41118.283087] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41118.291599] netfs veth [41118.293782] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41118.302727] nft_chain_nat xt_MASQUERADE [41118.305172] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41118.312642] nf_nat nf_conntrack [41118.316559] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41118.322037] nf_defrag_ipv6 nf_defrag_ipv4 [41118.325259] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41118.343998] nft_compat nf_tables [41118.348088] [41118.348089] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41118.353305] nfnetlink bridge [41118.356617] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41118.358109] stp [41118.365233] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41118.368196] llc xfs [41118.375319] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41118.377158] libcrc32c sunrpc [41118.384283] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41118.386477] intel_rapl_msr intel_rapl_common [41118.393599] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41118.396563] sb_edac x86_pkg_temp_thermal [41118.403686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41118.408037] coretemp kvm_intel [41118.416115] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41118.420119] kvm irqbypass [41118.425857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41118.428993] i2c_i801 iTCO_wdt [41118.436119] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41118.438822] iTCO_vendor_support [41118.445947] Call Trace: [41118.445948] [41118.448998] i2c_smbus crct10dif_pclmul [41118.456123] ? watchdog_timer_fn+0x1b4/0x220 [41118.459345] joydev crc32_pclmul [41118.461789] ? __hrtimer_run_queues+0x1d3/0x360 [41118.463800] ipmi_ssif [41118.467635] ? hrtimer_interrupt+0x104/0x240 [41118.471895] ghash_clmulni_intel mei_me [41118.475120] ? __local_bh_enable+0x50/0x60 [41118.479643] mei lpc_ich [41118.482002] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41118.486263] sha512_ssse3 acpi_ipmi [41118.490095] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41118.494186] mfd_core wmi [41118.496716] [41118.501933] ipmi_si ipmi_devintf [41118.505419] [41118.510376] ipmi_msghandler ioatdma [41118.512994] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41118.515091] acpi_pad acpi_power_meter [41118.518404] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41118.520497] ixgbe igb [41118.524070] do_raw_spin_lock+0xb5/0xc0 [41118.529372] mdio crc32c_intel [41118.533117] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41118.537988] nvme ptp [41118.540347] try_get_cap_refs+0x159/0x880 [ceph] [41118.544177] nvme_core i2c_algo_bit [41118.547231] __ceph_get_caps+0x4a1/0x760 [ceph] [41118.552011] pps_core dca [41118.554287] ? __file_remove_privs+0x89/0x110 [41118.558893] fuse [41118.562378] ? wake_bit_function+0x70/0x70 [41118.566900] irq event stamp: 249000322 [41118.569521] ? ceph_write_iter+0x260/0x770 [ceph] [41118.573869] hardirqs last enabled at (249000321): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41118.575793] ceph_write_iter+0x260/0x770 [ceph] [41118.579882] hardirqs last disabled at (249000322): [] sysvec_apic_timer_interrupt+0xe/0x90 [41118.583630] ? lock_is_held_type+0xaa/0x120 [41118.588325] softirqs last enabled at (249000320): [] __do_softirq+0x2be/0x371 [41118.598571] vfs_write+0x324/0x4f0 [41118.603093] softirqs last disabled at (249000315): [] irq_exit_rcu+0xb0/0xd0 [41118.612909] ksys_pwrite64+0x65/0xa0 [41118.617082] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41118.625851] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41118.629250] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41118.637848] do_syscall_64+0x38/0x80 [41118.641417] Workqueue: events delayed_work [ceph] [41118.650797] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41118.655927] [41118.655928] RIP: 0010:lock_acquire+0xf0/0x2b0 [41118.663397] RIP: 0033:0x7f48f0eeb93b [41118.666968] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41118.671667] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41118.676709] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41118.678200] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41118.682549] [41118.682550] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41118.686122] ORIG_RAX: 0000000000000012 [41118.704859] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41118.723595] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41118.728813] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41118.734031] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41118.735522] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41118.742646] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41118.746483] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41118.753609] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41118.760733] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41118.767857] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41118.774983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41118.778459] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [41118.778480] rcu: 0-....: (28521463 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14287016 [41118.778484] rcu: 7-....: (28332360 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/28330984 fqs=14287016 [41118.778488] rcu: (detected by 3, t=29181644 jiffies, g=1309609, q=777225 ncpus=8) [41118.778490] Sending NMI from CPU 3 to CPUs 0: [41118.778494] NMI backtrace for cpu 0 [41118.778495] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41118.778496] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41118.778497] RIP: 0010:io_serial_in+0x19/0x20 [41118.778500] Code: 00 d3 e6 48 63 f6 48 03 77 48 8b 06 c3 0f 1f 40 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 00 0f b7 57 40 d3 e6 01 f2 ec <0f> b6 c0 c3 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 0f b6 8f f9 00 00 [41118.778502] RSP: 0018:ffffc90000003b50 EFLAGS: 00000002 [41118.778503] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [41118.778503] RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff838a9560 [41118.778504] RBP: 00000000000025ce R08: 0000847da1d92ba8 R09: 0000000000000000 [41118.778505] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [41118.778505] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [41118.778506] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41118.778507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41118.778508] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41118.778509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41118.778509] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41118.778510] Call Trace: [41118.778511] [41118.778512] ? nmi_cpu_backtrace+0xad/0x120 [41118.778516] ? nmi_cpu_backtrace_handler+0x11/0x20 [41118.778517] ? nmi_handle+0xd4/0x260 [41118.778521] ? default_do_nmi+0x49/0xf0 [41118.778523] ? exc_nmi+0x11e/0x150 [41118.778524] ? end_repeat_nmi+0x16/0x67 [41118.778529] ? io_serial_in+0x19/0x20 [41118.778530] ? io_serial_in+0x19/0x20 [41118.778532] ? io_serial_in+0x19/0x20 [41118.778534] [41118.778534] [41118.778535] wait_for_lsr+0x45/0x70 [41118.778537] serial8250_console_write+0x341/0x3f0 [41118.778540] ? lock_release+0xc5/0x260 [41118.778543] console_flush_all+0x227/0x5a0 [41118.778545] ? console_flush_all+0x1ff/0x5a0 [41118.778547] ? console_flush_all+0x1ff/0x5a0 [41118.778550] console_unlock+0x7b/0x120 [41118.778552] vprintk_emit+0x246/0x340 [41118.778554] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [41118.778556] _printk+0x5c/0x80 [41118.778559] __show_regs+0xc9/0x2e0 [41118.778561] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [41118.778563] ? unwind_next_frame+0x379/0x830 [41118.778566] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [41118.778568] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [41118.778570] show_trace_log_lvl+0x2b5/0x3f0 [41118.778575] ? touch_softlockup_watchdog+0x20/0x20 [41118.778577] watchdog_timer_fn+0x1b4/0x220 [41118.778579] __hrtimer_run_queues+0x1d3/0x360 [41118.778582] hrtimer_interrupt+0x104/0x240 [41118.778583] ? __local_bh_enable+0x50/0x60 [41118.778586] __sysvec_apic_timer_interrupt+0x61/0x160 [41118.778589] sysvec_apic_timer_interrupt+0x6d/0x90 [41118.778591] [41118.778591] [41118.778592] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41118.778594] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41118.778596] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41118.778597] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41118.778598] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41118.778599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41118.778599] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41118.778600] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41118.778601] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41118.778604] do_raw_spin_lock+0xb5/0xc0 [41118.778606] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41118.778620] try_get_cap_refs+0x159/0x880 [ceph] [41118.778634] __ceph_get_caps+0x4a1/0x760 [ceph] [41118.778647] ? __file_remove_privs+0x89/0x110 [41118.778649] ? wake_bit_function+0x70/0x70 [41118.778651] ? ceph_write_iter+0x260/0x770 [ceph] [41118.778662] ceph_write_iter+0x260/0x770 [ceph] [41118.778673] ? lock_is_held_type+0xaa/0x120 [41118.778676] vfs_write+0x324/0x4f0 [41118.778680] ksys_pwrite64+0x65/0xa0 [41118.778682] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41118.778685] do_syscall_64+0x38/0x80 [41118.778687] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41118.778688] RIP: 0033:0x7f48f0eeb93b [41118.778689] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41118.778691] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [41118.778692] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41118.778692] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41118.778693] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41118.778693] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41118.778694] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41118.778697] [41118.779492] Sending NMI from CPU 3 to CPUs 7: [41118.779495] NMI backtrace for cpu 7 [41118.779496] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41118.779498] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41118.779499] Workqueue: events delayed_work [ceph] [41118.779518] RIP: 0010:vprintk_emit+0x1ee/0x340 [41118.779521] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [41118.779523] RSP: 0018:ffffc90000264db0 EFLAGS: 00000002 [41118.779524] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [41118.779525] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810debb578 [41118.779526] RBP: 0000000000000031 R08: 0000000000005ef4 R09: 0000000000000004 [41118.779527] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [41118.779528] R13: 0000000000000000 R14: ffffffff821887d8 R15: ffffc90000264e00 [41118.779530] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41118.779531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41118.779532] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41118.779534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41118.779535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41118.779536] Call Trace: [41118.779537] [41118.779539] ? nmi_cpu_backtrace+0xad/0x120 [41118.779542] ? nmi_cpu_backtrace_handler+0x11/0x20 [41118.779544] ? nmi_handle+0xd4/0x260 [41118.779549] ? default_do_nmi+0x49/0xf0 [41118.779551] ? exc_nmi+0x11e/0x150 [41118.779553] ? end_repeat_nmi+0x16/0x67 [41118.779560] ? vprintk_emit+0x1ee/0x340 [41118.779562] ? vprintk_emit+0x1ee/0x340 [41118.779566] ? vprintk_emit+0x1ee/0x340 [41118.779568] [41118.779569] [41118.779571] _printk+0x5c/0x80 [41118.779576] __show_regs+0x185/0x2e0 [41118.779580] ? touch_softlockup_watchdog+0x20/0x20 [41118.779582] show_regs+0x38/0x60 [41118.779585] watchdog_timer_fn+0x1b4/0x220 [41118.779588] __hrtimer_run_queues+0x1d3/0x360 [41118.779593] hrtimer_interrupt+0x104/0x240 [41118.779595] ? __local_bh_enable+0x50/0x60 [41118.779599] __sysvec_apic_timer_interrupt+0x61/0x160 [41118.779603] sysvec_apic_timer_interrupt+0x6d/0x90 [41118.779605] [41118.779606] [41118.779607] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41118.779609] RIP: 0010:lock_acquire+0xf0/0x2b0 [41118.779611] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41118.779613] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41118.779614] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41118.779615] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41118.779616] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41118.779617] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41118.779619] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41118.779625] ? igrab+0x43/0x50 [41118.779627] ? lock_release+0xc5/0x260 [41118.779631] _raw_spin_lock+0x30/0x40 [41118.779633] ? igrab+0x1d/0x50 [41118.779635] igrab+0x1d/0x50 [41118.779637] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41118.779657] delayed_work+0x1d8/0x280 [ceph] [41118.779676] ? process_scheduled_works+0x22a/0x510 [41118.779679] process_scheduled_works+0x2a4/0x510 [41118.779685] worker_thread+0x188/0x340 [41118.779689] ? rescuer_thread+0x310/0x310 [41118.779692] kthread+0xf5/0x130 [41118.779695] ? kthread_complete_and_exit+0x20/0x20 [41118.779698] ret_from_fork+0x31/0x40 [41118.779701] ? kthread_complete_and_exit+0x20/0x20 [41118.779704] ret_from_fork_asm+0x11/0x20 [41118.779711] [41118.782113] [41118.789232] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41119.688232] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41119.695356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41119.702497] Call Trace: [41119.704941] [41119.706952] ? watchdog_timer_fn+0x1b4/0x220 [41119.711217] ? __hrtimer_run_queues+0x1d3/0x360 [41119.715745] ? hrtimer_interrupt+0x104/0x240 [41119.720013] ? __local_bh_enable+0x50/0x60 [41119.724106] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41119.729330] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41119.734288] [41119.736385] [41119.738499] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41119.743810] ? lock_acquire+0xf0/0x2b0 [41119.747558] ? igrab+0x43/0x50 [41119.750608] ? lock_release+0xc5/0x260 [41119.754353] _raw_spin_lock+0x30/0x40 [41119.758008] ? igrab+0x1d/0x50 [41119.761060] igrab+0x1d/0x50 [41119.763937] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41119.769088] delayed_work+0x1d8/0x280 [ceph] [41119.773369] ? process_scheduled_works+0x22a/0x510 [41119.778159] process_scheduled_works+0x2a4/0x510 [41119.782786] worker_thread+0x188/0x340 [41119.786533] ? rescuer_thread+0x310/0x310 [41119.790545] kthread+0xf5/0x130 [41119.793682] ? kthread_complete_and_exit+0x20/0x20 [41119.798467] ret_from_fork+0x31/0x40 [41119.802044] ? kthread_complete_and_exit+0x20/0x20 [41119.806830] ret_from_fork_asm+0x11/0x20 [41119.810754] [41146.172567] watchdog: BUG: soft lockup - CPU#0 stuck for 27206s! [dbench:78683] [41146.179868] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41146.184566] watchdog: BUG: soft lockup - CPU#7 stuck for 36203s! [kworker/7:4:78808] [41146.235320] irq event stamp: 73526824 [41146.235321] hardirqs last enabled at (73526823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41146.243049] Modules linked in: [41146.246707] hardirqs last disabled at (73526824): [] sysvec_apic_timer_interrupt+0xe/0x90 [41146.256865] ceph [41146.259916] softirqs last enabled at (73526810): [] __do_softirq+0x2be/0x371 [41146.269639] libceph dns_resolver [41146.271562] softirqs last disabled at (73526805): [] irq_exit_rcu+0xb0/0xd0 [41146.280247] fscache netfs [41146.283558] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41146.292069] veth nft_chain_nat [41146.294774] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41146.303718] xt_MASQUERADE nf_nat [41146.306853] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41146.314326] nf_conntrack [41146.317636] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41146.323112] nf_defrag_ipv6 nf_defrag_ipv4 [41146.325730] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41146.344468] nft_compat nf_tables [41146.348558] [41146.348559] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41146.353777] nfnetlink bridge [41146.357088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41146.358579] stp llc [41146.365701] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41146.368666] xfs libcrc32c [41146.375789] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41146.377974] sunrpc [41146.385099] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41146.387801] intel_rapl_msr intel_rapl_common [41146.394925] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41146.397023] sb_edac x86_pkg_temp_thermal [41146.404147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41146.408498] coretemp [41146.416577] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41146.420579] kvm_intel [41146.426319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41146.428588] kvm irqbypass [41146.435713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41146.438070] i2c_i801 [41146.445193] Call Trace: [41146.445196] [41146.447899] iTCO_wdt [41146.455026] ? watchdog_timer_fn+0x1b4/0x220 [41146.457294] iTCO_vendor_support [41146.459738] ? __hrtimer_run_queues+0x1d3/0x360 [41146.461748] i2c_smbus [41146.464022] ? hrtimer_interrupt+0x104/0x240 [41146.468283] crct10dif_pclmul joydev [41146.471506] ? __local_bh_enable+0x50/0x60 [41146.476031] crc32_pclmul ipmi_ssif [41146.478391] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41146.482651] ghash_clmulni_intel [41146.486240] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41146.490332] mei_me mei [41146.493817] [41146.499032] lpc_ich sha512_ssse3 [41146.502257] [41146.502259] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41146.507213] acpi_ipmi mfd_core [41146.509663] ? queued_spin_lock_slowpath+0x16/0x2e0 [41146.511756] wmi [41146.515069] do_raw_spin_lock+0xb5/0xc0 [41146.517164] ipmi_si ipmi_devintf [41146.522470] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41146.525606] ipmi_msghandler ioatdma [41146.530478] try_get_cap_refs+0x159/0x880 [ceph] [41146.532313] acpi_pad acpi_power_meter [41146.536149] __ceph_get_caps+0x4a1/0x760 [ceph] [41146.539453] ixgbe [41146.544243] ? __file_remove_privs+0x89/0x110 [41146.547808] igb mdio [41146.552421] ? wake_bit_function+0x70/0x70 [41146.556164] crc32c_intel nvme [41146.560691] ? ceph_write_iter+0x260/0x770 [ceph] [41146.562700] ptp nvme_core [41146.567050] ceph_write_iter+0x260/0x770 [ceph] [41146.569321] i2c_algo_bit [41146.573416] ? lock_is_held_type+0xaa/0x120 [41146.576463] pps_core dca [41146.581161] vfs_write+0x324/0x4f0 [41146.583863] fuse [41146.588392] ksys_pwrite64+0x65/0xa0 [41146.591004] irq event stamp: 249214296 [41146.595184] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41146.597801] hardirqs last enabled at (249214295): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41146.601199] do_syscall_64+0x38/0x80 [41146.603121] hardirqs last disabled at (249214296): [] sysvec_apic_timer_interrupt+0xe/0x90 [41146.606693] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41146.610435] softirqs last enabled at (249214294): [] __do_softirq+0x2be/0x371 [41146.615567] RIP: 0033:0x7f48f0eeb93b [41146.625811] softirqs last disabled at (249214289): [] irq_exit_rcu+0xb0/0xd0 [41146.629383] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41146.639192] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41146.644238] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41146.653008] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41146.656579] ORIG_RAX: 0000000000000012 [41146.665177] Workqueue: events delayed_work [ceph] [41146.683914] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41146.693290] [41146.698508] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41146.705979] RIP: 0010:lock_acquire+0x86/0x2b0 [41146.709809] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41146.714507] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [41146.721632] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41146.723122] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [41146.730250] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41146.734598] [41146.734599] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [41146.741727] [41146.760458] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [41146.797848] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [41146.804970] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [41146.812096] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41146.819220] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41146.827296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41146.833035] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41146.840160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41146.847283] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41146.854407] Call Trace: [41146.856853] [41146.858863] ? watchdog_timer_fn+0x1b4/0x220 [41146.863128] ? __hrtimer_run_queues+0x1d3/0x360 [41146.867654] ? hrtimer_interrupt+0x104/0x240 [41146.871923] ? __local_bh_enable+0x50/0x60 [41146.876018] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41146.881253] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41146.886215] [41146.888312] [41146.890409] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41146.895717] ? lock_acquire+0x86/0x2b0 [41146.899468] ? igrab+0x43/0x50 [41146.902517] ? lock_release+0xc5/0x260 [41146.906263] _raw_spin_lock+0x30/0x40 [41146.909928] ? igrab+0x1d/0x50 [41146.912977] igrab+0x1d/0x50 [41146.915858] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41146.921006] delayed_work+0x1d8/0x280 [ceph] [41146.925288] ? process_scheduled_works+0x22a/0x510 [41146.930077] process_scheduled_works+0x2a4/0x510 [41146.934692] worker_thread+0x188/0x340 [41146.938441] ? rescuer_thread+0x310/0x310 [41146.942445] kthread+0xf5/0x130 [41146.945583] ? kthread_complete_and_exit+0x20/0x20 [41146.950368] ret_from_fork+0x31/0x40 [41146.953946] ? kthread_complete_and_exit+0x20/0x20 [41146.958748] ret_from_fork_asm+0x11/0x20 [41146.962680] [41174.173059] watchdog: BUG: soft lockup - CPU#0 stuck for 27232s! [dbench:78683] [41174.180365] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41174.185059] watchdog: BUG: soft lockup - CPU#7 stuck for 36229s! [kworker/7:4:78808] [41174.235805] irq event stamp: 73585140 [41174.235806] hardirqs last enabled at (73585139): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41174.243538] Modules linked in: [41174.247193] hardirqs last disabled at (73585140): [] sysvec_apic_timer_interrupt+0xe/0x90 [41174.257353] ceph [41174.260401] softirqs last enabled at (73585018): [] __do_softirq+0x2be/0x371 [41174.270125] libceph [41174.272052] softirqs last disabled at (73585013): [] irq_exit_rcu+0xb0/0xd0 [41174.280742] dns_resolver fscache [41174.282926] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41174.291455] netfs veth [41174.294767] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41174.303710] nft_chain_nat [41174.306171] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41174.313643] xt_MASQUERADE nf_nat [41174.316346] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41174.321824] nf_conntrack nf_defrag_ipv6 [41174.325135] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41174.343871] nf_defrag_ipv4 nft_compat [41174.347790] [41174.347791] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41174.353007] nf_tables nfnetlink [41174.356752] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41174.358241] bridge stp [41174.365367] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41174.368591] llc xfs [41174.375731] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41174.378176] libcrc32c [41174.385300] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41174.387484] sunrpc intel_rapl_msr [41174.394608] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41174.396966] intel_rapl_common sb_edac [41174.404090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41174.407486] x86_pkg_temp_thermal coretemp [41174.415564] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41174.419309] kvm_intel kvm [41174.425045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41174.429136] irqbypass [41174.436261] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41174.436262] Call Trace: [41174.438966] i2c_i801 [41174.446089] [41174.448457] iTCO_wdt iTCO_vendor_support [41174.455581] ? watchdog_timer_fn+0x1b4/0x220 [41174.458023] i2c_smbus crct10dif_pclmul [41174.460296] ? __hrtimer_run_queues+0x1d3/0x360 [41174.462305] joydev crc32_pclmul [41174.466313] ? hrtimer_interrupt+0x104/0x240 [41174.470572] ipmi_ssif ghash_clmulni_intel [41174.474403] ? __local_bh_enable+0x50/0x60 [41174.478927] mei_me mei [41174.482154] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41174.486415] lpc_ich sha512_ssse3 [41174.490507] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41174.494598] acpi_ipmi mfd_core [41174.497041] [41174.502260] wmi ipmi_si [41174.505568] [41174.505570] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41174.510526] ipmi_devintf ipmi_msghandler [41174.513669] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41174.515761] ioatdma acpi_pad [41174.518295] do_raw_spin_lock+0xb5/0xc0 [41174.520388] acpi_power_meter ixgbe [41174.525694] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41174.529697] igb mdio crc32c_intel [41174.534571] try_get_cap_refs+0x159/0x880 [ceph] [41174.537532] nvme ptp [41174.541368] __ceph_get_caps+0x4a1/0x760 [ceph] [41174.544846] nvme_core [41174.549637] ? __file_remove_privs+0x89/0x110 [41174.553028] i2c_algo_bit pps_core [41174.557640] ? wake_bit_function+0x70/0x70 [41174.559911] dca fuse [41174.564457] ? ceph_write_iter+0x260/0x770 [ceph] [41174.566808] [41174.571159] ceph_write_iter+0x260/0x770 [ceph] [41174.574558] irq event stamp: 249435070 [41174.578652] ? lock_is_held_type+0xaa/0x120 [41174.580918] hardirqs last enabled at (249435069): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41174.585618] vfs_write+0x324/0x4f0 [41174.587106] hardirqs last disabled at (249435070): [] sysvec_apic_timer_interrupt+0xe/0x90 [41174.591634] ksys_pwrite64+0x65/0xa0 [41174.595374] softirqs last enabled at (249435068): [] __do_softirq+0x2be/0x371 [41174.599551] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41174.609797] softirqs last disabled at (249435063): [] irq_exit_rcu+0xb0/0xd0 [41174.613196] do_syscall_64+0x38/0x80 [41174.623005] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41174.626576] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41174.635346] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41174.640477] RIP: 0033:0x7f48f0eeb93b [41174.649074] Workqueue: events delayed_work [ceph] [41174.652647] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41174.662024] [41174.667068] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41174.674538] RIP: 0010:lock_acquire+0x86/0x2b0 [41174.678109] ORIG_RAX: 0000000000000012 [41174.682807] Code: 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 <0f> 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a 00 [41174.701543] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41174.703035] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [41174.708251] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41174.712602] [41174.712603] RAX: 0000000000000246 RBX: ffff8881a1d34500 RCX: 0000000000000000 [41174.716434] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41174.735188] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [41174.742312] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41174.747530] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [41174.754653] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41174.756145] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [41174.763274] [41174.770392] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41174.815322] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41174.823400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41174.829136] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41174.836262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41174.843386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41174.850510] Call Trace: [41174.852952] [41174.854965] ? watchdog_timer_fn+0x1b4/0x220 [41174.859228] ? __hrtimer_run_queues+0x1d3/0x360 [41174.863756] ? hrtimer_interrupt+0x104/0x240 [41174.868024] ? __local_bh_enable+0x50/0x60 [41174.872118] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41174.877343] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41174.882298] [41174.884397] [41174.886495] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41174.891802] ? lock_acquire+0x86/0x2b0 [41174.895552] ? igrab+0x43/0x50 [41174.898603] ? lock_release+0xc5/0x260 [41174.902348] _raw_spin_lock+0x30/0x40 [41174.906004] ? igrab+0x1d/0x50 [41174.909055] igrab+0x1d/0x50 [41174.911934] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41174.917083] delayed_work+0x1d8/0x280 [ceph] [41174.921363] ? process_scheduled_works+0x22a/0x510 [41174.926153] process_scheduled_works+0x2a4/0x510 [41174.930768] worker_thread+0x188/0x340 [41174.934517] ? rescuer_thread+0x310/0x310 [41174.938522] kthread+0xf5/0x130 [41174.941658] ? kthread_complete_and_exit+0x20/0x20 [41174.946462] ret_from_fork+0x31/0x40 [41174.950040] ? kthread_complete_and_exit+0x20/0x20 [41174.954825] ret_from_fork_asm+0x11/0x20 [41174.958761] [41202.173553] watchdog: BUG: soft lockup - CPU#0 stuck for 27258s! [dbench:78683] [41202.180862] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41202.185552] watchdog: BUG: soft lockup - CPU#7 stuck for 36255s! [kworker/7:4:78808] [41202.236312] irq event stamp: 73643342 [41202.244042] Modules linked in: ceph libceph dns_resolver fscache netfs [41202.247698] hardirqs last enabled at (73643341): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41202.254240] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [41202.264398] hardirqs last disabled at (73643342): [] sysvec_apic_timer_interrupt+0xe/0x90 [41202.271785] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [41202.281508] softirqs last enabled at (73643236): [] __do_softirq+0x2be/0x371 [41202.287948] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [41202.296632] softirqs last disabled at (73643231): [] irq_exit_rcu+0xb0/0xd0 [41202.296636] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [41202.311668] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41202.311670] irqbypass [41202.311671] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41202.317667] i2c_i801 iTCO_wdt [41202.326611] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41202.328968] iTCO_vendor_support [41202.336440] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41202.339488] i2c_smbus crct10dif_pclmul [41202.344965] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41202.348189] joydev crc32_pclmul [41202.366944] [41202.370775] ipmi_ssif ghash_clmulni_intel [41202.375995] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41202.379228] mei_me mei [41202.380716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41202.384807] lpc_ich sha512_ssse3 [41202.391949] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41202.394393] acpi_ipmi mfd_core [41202.401516] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41202.404828] wmi ipmi_si [41202.411953] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41202.415090] ipmi_devintf ipmi_msghandler [41202.422230] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41202.424761] ioatdma [41202.431885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41202.435889] acpi_pad acpi_power_meter [41202.443967] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41202.446151] ixgbe igb [41202.451888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41202.455634] mdio crc32c_intel [41202.462757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41202.465114] nvme [41202.472240] Call Trace: [41202.475290] ptp [41202.482415] [41202.484338] nvme_core i2c_algo_bit [41202.486784] ? watchdog_timer_fn+0x1b4/0x220 [41202.488619] pps_core dca [41202.490631] ? __hrtimer_run_queues+0x1d3/0x360 [41202.494114] fuse [41202.498381] ? hrtimer_interrupt+0x104/0x240 [41202.500995] irq event stamp: 249655890 [41202.505521] ? __local_bh_enable+0x50/0x60 [41202.507443] hardirqs last enabled at (249655889): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41202.511711] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41202.515450] hardirqs last disabled at (249655890): [] sysvec_apic_timer_interrupt+0xe/0x90 [41202.519542] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41202.529785] softirqs last enabled at (249655888): [] __do_softirq+0x2be/0x371 [41202.535004] [41202.544815] softirqs last disabled at (249655883): [] irq_exit_rcu+0xb0/0xd0 [41202.549771] [41202.558543] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41202.560641] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41202.569241] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41202.571339] ? queued_spin_lock_slowpath+0x18/0x2e0 [41202.580713] Workqueue: events delayed_work [ceph] [41202.586018] do_raw_spin_lock+0xb5/0xc0 [41202.593488] [41202.593489] RIP: 0010:lock_acquire+0xf0/0x2b0 [41202.598361] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41202.603056] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41202.606889] try_get_cap_refs+0x159/0x880 [ceph] [41202.608377] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41202.612735] __ceph_get_caps+0x4a1/0x760 [ceph] [41202.617513] [41202.617513] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41202.636254] ? __file_remove_privs+0x89/0x110 [41202.640861] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41202.646078] ? wake_bit_function+0x70/0x70 [41202.650603] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41202.652096] ? ceph_write_iter+0x260/0x770 [ceph] [41202.659217] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41202.663584] ceph_write_iter+0x260/0x770 [ceph] [41202.670708] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41202.674802] ? lock_is_held_type+0xaa/0x120 [41202.681925] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41202.686624] vfs_write+0x324/0x4f0 [41202.693744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41202.698274] ksys_pwrite64+0x65/0xa0 [41202.705393] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41202.709571] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41202.717649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41202.721046] do_syscall_64+0x38/0x80 [41202.726784] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41202.730355] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41202.737479] Call Trace: [41202.742610] RIP: 0033:0x7f48f0eeb93b [41202.749734] [41202.753306] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41202.760430] ? watchdog_timer_fn+0x1b4/0x220 [41202.765471] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41202.767918] ? __hrtimer_run_queues+0x1d3/0x360 [41202.771487] ORIG_RAX: 0000000000000012 [41202.773501] ? hrtimer_interrupt+0x104/0x240 [41202.792244] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41202.796508] ? __local_bh_enable+0x50/0x60 [41202.801727] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41202.806253] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41202.810081] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41202.814344] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41202.821467] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41202.825559] [41202.832684] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41202.837901] [41202.845031] [41202.849982] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41202.875919] ? lock_acquire+0xf0/0x2b0 [41202.879669] ? igrab+0x43/0x50 [41202.882717] ? lock_release+0xc5/0x260 [41202.886463] _raw_spin_lock+0x30/0x40 [41202.890120] ? igrab+0x1d/0x50 [41202.893170] igrab+0x1d/0x50 [41202.896050] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41202.901199] delayed_work+0x1d8/0x280 [ceph] [41202.905480] ? process_scheduled_works+0x22a/0x510 [41202.910269] process_scheduled_works+0x2a4/0x510 [41202.914884] worker_thread+0x188/0x340 [41202.918633] ? rescuer_thread+0x310/0x310 [41202.922637] kthread+0xf5/0x130 [41202.925776] ? kthread_complete_and_exit+0x20/0x20 [41202.930558] ret_from_fork+0x31/0x40 [41202.934140] ? kthread_complete_and_exit+0x20/0x20 [41202.938933] ret_from_fork_asm+0x11/0x20 [41202.942879] [41230.174045] watchdog: BUG: soft lockup - CPU#0 stuck for 27284s! [dbench:78683] [41230.181349] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41230.186044] watchdog: BUG: soft lockup - CPU#7 stuck for 36281s! [kworker/7:4:78808] [41230.236789] irq event stamp: 73701382 [41230.236790] hardirqs last enabled at (73701381): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41230.244520] Modules linked in: ceph [41230.248176] hardirqs last disabled at (73701382): [] sysvec_apic_timer_interrupt+0xe/0x90 [41230.258336] libceph [41230.261819] softirqs last enabled at (73701344): [] __do_softirq+0x2be/0x371 [41230.271543] dns_resolver fscache [41230.273727] softirqs last disabled at (73701339): [] irq_exit_rcu+0xb0/0xd0 [41230.282411] netfs [41230.285743] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41230.294249] veth nft_chain_nat [41230.296262] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41230.305206] xt_MASQUERADE nf_nat [41230.308343] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41230.315814] nf_conntrack nf_defrag_ipv6 [41230.319123] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41230.324601] nf_defrag_ipv4 [41230.328519] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41230.347256] nft_compat nf_tables [41230.350047] [41230.350048] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41230.355264] nfnetlink bridge [41230.358574] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41230.360066] stp llc [41230.367190] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41230.370155] xfs libcrc32c [41230.377280] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41230.379463] sunrpc intel_rapl_msr [41230.386586] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41230.389290] intel_rapl_common [41230.396416] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41230.399829] sb_edac x86_pkg_temp_thermal [41230.406955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41230.410005] coretemp [41230.418082] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41230.422087] kvm_intel [41230.427824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41230.430094] kvm irqbypass [41230.437218] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41230.439575] i2c_i801 iTCO_wdt [41230.446700] Call Trace: [41230.449421] iTCO_vendor_support i2c_smbus [41230.456564] [41230.459613] crct10dif_pclmul joydev [41230.462058] ? watchdog_timer_fn+0x1b4/0x220 [41230.466149] crc32_pclmul [41230.468160] ? __hrtimer_run_queues+0x1d3/0x360 [41230.471731] ipmi_ssif ghash_clmulni_intel [41230.475996] ? hrtimer_interrupt+0x104/0x240 [41230.478610] mei_me mei [41230.483135] ? __local_bh_enable+0x50/0x60 [41230.487226] lpc_ich sha512_ssse3 [41230.491493] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41230.493933] acpi_ipmi mfd_core [41230.498024] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41230.501336] wmi ipmi_si [41230.506553] [41230.509690] ipmi_devintf ipmi_msghandler [41230.514648] [41230.517179] ioatdma acpi_pad [41230.519277] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41230.523279] acpi_power_meter ixgbe [41230.525381] ? queued_spin_lock_slowpath+0x18/0x2e0 [41230.528342] igb mdio [41230.533647] do_raw_spin_lock+0xb5/0xc0 [41230.537129] crc32c_intel nvme [41230.542002] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41230.544270] ptp nvme_core [41230.548106] try_get_cap_refs+0x159/0x880 [ceph] [41230.551152] i2c_algo_bit pps_core [41230.555940] __ceph_get_caps+0x4a1/0x760 [ceph] [41230.558641] dca fuse [41230.563256] ? __file_remove_privs+0x89/0x110 [41230.566648] [41230.566649] irq event stamp: 249876758 [41230.571174] ? wake_bit_function+0x70/0x70 [41230.573447] hardirqs last enabled at (249876757): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41230.577797] ? ceph_write_iter+0x260/0x770 [ceph] [41230.579285] hardirqs last disabled at (249876758): [] sysvec_apic_timer_interrupt+0xe/0x90 [41230.583030] ceph_write_iter+0x260/0x770 [ceph] [41230.587119] softirqs last enabled at (249876756): [] __do_softirq+0x2be/0x371 [41230.597367] ? lock_is_held_type+0xaa/0x120 [41230.602060] softirqs last disabled at (249876751): [] irq_exit_rcu+0xb0/0xd0 [41230.611876] vfs_write+0x324/0x4f0 [41230.616397] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41230.625172] ksys_pwrite64+0x65/0xa0 [41230.629345] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41230.637942] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41230.641338] Workqueue: events delayed_work [ceph] [41230.650719] do_syscall_64+0x38/0x80 [41230.654289] [41230.654289] RIP: 0010:rcu_is_watching+0x0/0x40 [41230.661758] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41230.666889] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [41230.671586] RIP: 0033:0x7f48f0eeb93b [41230.675157] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [41230.676647] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41230.681086] [41230.681087] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [41230.686130] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41230.704867] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [41230.708445] ORIG_RAX: 0000000000000012 [41230.713663] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [41230.732402] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41230.733893] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [41230.741016] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41230.746235] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41230.753357] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41230.757190] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41230.764312] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41230.771447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41230.778570] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41230.785695] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41230.792825] [41230.799943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41230.844451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41230.851579] Call Trace: [41230.854024] [41230.856035] ? watchdog_timer_fn+0x1b4/0x220 [41230.860298] ? __hrtimer_run_queues+0x1d3/0x360 [41230.864827] ? hrtimer_interrupt+0x104/0x240 [41230.869096] ? __local_bh_enable+0x50/0x60 [41230.873190] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41230.878412] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41230.883371] [41230.885467] [41230.887563] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41230.892868] ? igrab+0x43/0x50 [41230.895921] ? igrab+0x43/0x50 [41230.898969] ? rcutorture_get_gp_data+0x30/0x30 [41230.903496] lock_release+0x1d7/0x260 [41230.907154] _raw_spin_unlock+0x1b/0x40 [41230.910992] igrab+0x43/0x50 [41230.913869] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41230.919018] delayed_work+0x1d8/0x280 [ceph] [41230.923299] ? process_scheduled_works+0x22a/0x510 [41230.928081] process_scheduled_works+0x2a4/0x510 [41230.932698] worker_thread+0x188/0x340 [41230.936451] ? rescuer_thread+0x310/0x310 [41230.940457] kthread+0xf5/0x130 [41230.943597] ? kthread_complete_and_exit+0x20/0x20 [41230.948379] ret_from_fork+0x31/0x40 [41230.951950] ? kthread_complete_and_exit+0x20/0x20 [41230.956739] ret_from_fork_asm+0x11/0x20 [41230.960668] [41258.174537] watchdog: BUG: soft lockup - CPU#0 stuck for 27310s! [dbench:78683] [41258.181846] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41258.186537] watchdog: BUG: soft lockup - CPU#7 stuck for 36307s! [kworker/7:4:78808] [41258.237294] irq event stamp: 73759576 [41258.245024] Modules linked in: ceph libceph dns_resolver fscache netfs [41258.248681] hardirqs last enabled at (73759575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41258.255211] veth nft_chain_nat xt_MASQUERADE nf_nat [41258.265366] hardirqs last disabled at (73759576): [] sysvec_apic_timer_interrupt+0xe/0x90 [41258.270323] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [41258.280048] softirqs last enabled at (73759538): [] __do_softirq+0x2be/0x371 [41258.280052] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [41258.296644] softirqs last disabled at (73759533): [] irq_exit_rcu+0xb0/0xd0 [41258.304115] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [41258.312628] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41258.318623] irqbypass i2c_i801 [41258.327568] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41258.327570] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [41258.330707] iTCO_wdt [41258.338176] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [41258.343654] iTCO_vendor_support [41258.345933] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41258.364670] i2c_smbus crct10dif_pclmul [41258.367895] [41258.367896] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41258.373112] joydev crc32_pclmul [41258.376941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41258.378432] ipmi_ssif ghash_clmulni_intel [41258.385558] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41258.388780] mei_me mei [41258.395906] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41258.399995] lpc_ich sha512_ssse3 [41258.407120] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41258.409565] acpi_ipmi [41258.416688] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41258.420000] mfd_core wmi [41258.427124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41258.429481] ipmi_si ipmi_devintf [41258.437557] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41258.440176] ipmi_msghandler ioatdma [41258.445931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41258.449242] acpi_pad [41258.456364] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41258.459936] acpi_power_meter ixgbe [41258.467059] Call Trace: [41258.467061] [41258.469330] igb mdio [41258.476457] ? watchdog_timer_fn+0x1b4/0x220 [41258.479939] crc32c_intel [41258.482384] ? __hrtimer_run_queues+0x1d3/0x360 [41258.484395] nvme ptp [41258.486667] ? hrtimer_interrupt+0x104/0x240 [41258.490935] nvme_core i2c_algo_bit [41258.493555] ? __local_bh_enable+0x50/0x60 [41258.498079] pps_core dca [41258.500351] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41258.504613] fuse [41258.508098] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41258.512188] irq event stamp: 250097542 [41258.514806] [41258.520023] hardirqs last enabled at (250097541): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41258.521946] [41258.526904] hardirqs last disabled at (250097542): [] sysvec_apic_timer_interrupt+0xe/0x90 [41258.530649] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41258.532746] softirqs last enabled at (250097540): [] __do_softirq+0x2be/0x371 [41258.542993] ? queued_spin_lock_slowpath+0x2b/0x2e0 [41258.545089] softirqs last disabled at (250097535): [] irq_exit_rcu+0xb0/0xd0 [41258.554899] do_raw_spin_lock+0xb5/0xc0 [41258.560215] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41258.568986] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41258.573852] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41258.582453] try_get_cap_refs+0x159/0x880 [ceph] [41258.586281] Workqueue: events delayed_work [ceph] [41258.595663] __ceph_get_caps+0x4a1/0x760 [ceph] [41258.600442] [41258.600443] RIP: 0010:lock_acquire+0xf0/0x2b0 [41258.607937] ? __file_remove_privs+0x89/0x110 [41258.612540] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41258.617239] ? wake_bit_function+0x70/0x70 [41258.621763] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41258.623256] ? ceph_write_iter+0x260/0x770 [ceph] [41258.627604] [41258.627605] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41258.631956] ceph_write_iter+0x260/0x770 [ceph] [41258.650693] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41258.654787] ? lock_is_held_type+0xaa/0x120 [41258.660000] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41258.664700] vfs_write+0x324/0x4f0 [41258.666188] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41258.673318] ksys_pwrite64+0x65/0xa0 [41258.677837] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41258.684963] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41258.689138] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41258.696265] do_syscall_64+0x38/0x80 [41258.699660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41258.706786] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41258.710355] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41258.717480] RIP: 0033:0x7f48f0eeb93b [41258.722611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41258.730690] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41258.734257] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41258.739994] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41258.745039] Call Trace: [41258.745041] [41258.752164] ORIG_RAX: 0000000000000012 [41258.755736] ? watchdog_timer_fn+0x1b4/0x220 [41258.762860] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41258.781598] ? __hrtimer_run_queues+0x1d3/0x360 [41258.788720] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41258.793945] ? hrtimer_interrupt+0x104/0x240 [41258.796382] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41258.798394] ? __local_bh_enable+0x50/0x60 [41258.802225] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41258.806491] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41258.813611] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41258.818138] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41258.825265] [41258.829524] [41258.869462] [41258.871559] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41258.876867] ? lock_acquire+0xf0/0x2b0 [41258.880617] ? igrab+0x43/0x50 [41258.883666] ? lock_release+0xc5/0x260 [41258.887413] _raw_spin_lock+0x30/0x40 [41258.891076] ? igrab+0x1d/0x50 [41258.894128] igrab+0x1d/0x50 [41258.897007] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41258.902156] delayed_work+0x1d8/0x280 [ceph] [41258.906437] ? process_scheduled_works+0x22a/0x510 [41258.911227] process_scheduled_works+0x2a4/0x510 [41258.915859] worker_thread+0x188/0x340 [41258.919608] ? rescuer_thread+0x310/0x310 [41258.923611] kthread+0xf5/0x130 [41258.926750] ? kthread_complete_and_exit+0x20/0x20 [41258.931533] ret_from_fork+0x31/0x40 [41258.935105] ? kthread_complete_and_exit+0x20/0x20 [41258.939888] ret_from_fork_asm+0x11/0x20 [41258.943813] [41286.175029] watchdog: BUG: soft lockup - CPU#0 stuck for 27336s! [dbench:78683] [41286.182332] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41286.187029] watchdog: BUG: soft lockup - CPU#7 stuck for 36333s! [kworker/7:4:78808] [41286.237772] irq event stamp: 73817716 [41286.237774] hardirqs last enabled at (73817715): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41286.245504] Modules linked in: ceph [41286.249160] hardirqs last disabled at (73817716): [] sysvec_apic_timer_interrupt+0xe/0x90 [41286.259318] libceph [41286.262801] softirqs last enabled at (73817678): [] __do_softirq+0x2be/0x371 [41286.272527] dns_resolver fscache [41286.274710] softirqs last disabled at (73817673): [] irq_exit_rcu+0xb0/0xd0 [41286.283396] netfs veth [41286.286705] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41286.295233] nft_chain_nat xt_MASQUERADE [41286.297678] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41286.306622] nf_nat nf_conntrack [41286.310538] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41286.318010] nf_defrag_ipv6 nf_defrag_ipv4 [41286.321233] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41286.326726] nft_compat nf_tables [41286.330821] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41286.349557] nfnetlink bridge [41286.352868] [41286.352869] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41286.358086] stp llc [41286.361048] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41286.362539] xfs libcrc32c [41286.369664] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41286.371849] sunrpc intel_rapl_msr [41286.378974] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41286.381676] intel_rapl_common [41286.388800] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41286.392198] sb_edac x86_pkg_temp_thermal [41286.399322] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41286.402374] coretemp kvm_intel [41286.409496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41286.413501] kvm irqbypass [41286.421579] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41286.424716] i2c_i801 iTCO_wdt [41286.430454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41286.433157] iTCO_vendor_support i2c_smbus [41286.440281] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41286.443332] crct10dif_pclmul joydev [41286.450457] Call Trace: [41286.450458] [41286.454548] crc32_pclmul ipmi_ssif [41286.461672] ? watchdog_timer_fn+0x1b4/0x220 [41286.465241] ghash_clmulni_intel mei_me [41286.467687] ? __hrtimer_run_queues+0x1d3/0x360 [41286.469710] mei lpc_ich [41286.473194] ? hrtimer_interrupt+0x104/0x240 [41286.477453] sha512_ssse3 acpi_ipmi [41286.481284] ? __local_bh_enable+0x50/0x60 [41286.485808] mfd_core wmi [41286.488342] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41286.492603] ipmi_si ipmi_devintf [41286.496089] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41286.500179] ipmi_msghandler [41286.502796] [41286.508014] ioatdma acpi_pad [41286.511323] [41286.511325] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41286.516280] acpi_power_meter [41286.519163] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41286.521255] ixgbe igb [41286.524221] do_raw_spin_lock+0xb5/0xc0 [41286.526317] mdio crc32c_intel [41286.531624] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41286.534585] nvme ptp nvme_core [41286.539458] try_get_cap_refs+0x159/0x880 [ceph] [41286.541814] i2c_algo_bit [41286.545649] __ceph_get_caps+0x4a1/0x760 [ceph] [41286.548707] pps_core dca [41286.553491] ? __file_remove_privs+0x89/0x110 [41286.556625] fuse [41286.561235] ? wake_bit_function+0x70/0x70 [41286.563854] irq event stamp: 250318378 [41286.568379] ? ceph_write_iter+0x260/0x770 [ceph] [41286.570995] hardirqs last enabled at (250318377): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41286.575347] ceph_write_iter+0x260/0x770 [ceph] [41286.577269] hardirqs last disabled at (250318378): [] sysvec_apic_timer_interrupt+0xe/0x90 [41286.581364] ? lock_is_held_type+0xaa/0x120 [41286.585105] softirqs last enabled at (250318376): [] __do_softirq+0x2be/0x371 [41286.589803] vfs_write+0x324/0x4f0 [41286.600045] softirqs last disabled at (250318371): [] irq_exit_rcu+0xb0/0xd0 [41286.604575] ksys_pwrite64+0x65/0xa0 [41286.614381] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41286.618559] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41286.627329] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41286.630728] do_syscall_64+0x38/0x80 [41286.639324] Workqueue: events delayed_work [ceph] [41286.642896] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41286.652271] [41286.652272] RIP: 0010:lock_acquire+0xf0/0x2b0 [41286.657402] RIP: 0033:0x7f48f0eeb93b [41286.664873] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41286.668445] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41286.673141] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41286.678187] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41286.679677] [41286.679678] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41286.684028] ORIG_RAX: 0000000000000012 [41286.687597] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41286.687599] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41286.706337] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41286.725074] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41286.730290] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41286.735507] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41286.736999] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41286.744123] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41286.747955] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41286.755079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41286.762202] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41286.769326] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41286.776456] [41286.783576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41286.842337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41286.849459] Call Trace: [41286.851904] [41286.853915] ? watchdog_timer_fn+0x1b4/0x220 [41286.858181] ? __hrtimer_run_queues+0x1d3/0x360 [41286.862717] ? hrtimer_interrupt+0x104/0x240 [41286.866985] ? __local_bh_enable+0x50/0x60 [41286.871078] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41286.876303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41286.881260] [41286.883357] [41286.885453] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41286.890763] ? lock_acquire+0xf0/0x2b0 [41286.894512] ? igrab+0x43/0x50 [41286.897561] ? lock_release+0xc5/0x260 [41286.901307] _raw_spin_lock+0x30/0x40 [41286.904964] ? igrab+0x1d/0x50 [41286.908013] igrab+0x1d/0x50 [41286.910892] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41286.916040] delayed_work+0x1d8/0x280 [ceph] [41286.920323] ? process_scheduled_works+0x22a/0x510 [41286.925113] process_scheduled_works+0x2a4/0x510 [41286.929731] worker_thread+0x188/0x340 [41286.933478] ? rescuer_thread+0x310/0x310 [41286.937480] kthread+0xf5/0x130 [41286.940619] ? kthread_complete_and_exit+0x20/0x20 [41286.945402] ret_from_fork+0x31/0x40 [41286.948973] ? kthread_complete_and_exit+0x20/0x20 [41286.953756] ret_from_fork_asm+0x11/0x20 [41286.957681] [41313.784881] rcu: INFO: rcu_preempt self-detected stall on CPU [41313.790636] rcu: 0-....: (28712660 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14382822 [41313.801142] rcu: (t=29376663 jiffies g=1309609 q=781370 ncpus=8) [41313.807226] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41313.816169] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41313.823641] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41313.829125] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41313.847864] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41313.853082] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41313.860206] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41313.867329] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41313.874452] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41313.881578] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41313.888702] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41313.896779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41313.902517] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41313.909641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41313.916764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41313.923888] Call Trace: [41313.926334] [41313.928344] ? rcu_dump_cpu_stacks+0xd1/0x180 [41313.932704] ? rcu_sched_clock_irq+0x633/0x1390 [41313.937227] ? lock_is_held_type+0xaa/0x120 [41313.941404] ? lock_is_held_type+0xaa/0x120 [41313.945582] ? tick_sched_do_timer+0xa0/0xa0 [41313.949853] ? rcu_read_lock_sched_held+0x56/0x90 [41313.954554] ? tick_sched_do_timer+0xa0/0xa0 [41313.958817] ? update_process_times+0x5f/0x90 [41313.963167] ? tick_sched_handle.isra.16+0x1e/0x50 [41313.967951] ? tick_sched_timer+0x6f/0x80 [41313.971955] ? __hrtimer_run_queues+0x1d3/0x360 [41313.976481] ? hrtimer_interrupt+0x104/0x240 [41313.980743] ? __local_bh_enable+0x50/0x60 [41313.984837] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41313.990059] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41313.995019] [41313.997116] [41313.999213] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41314.004521] ? queued_spin_lock_slowpath+0x18/0x2e0 [41314.009398] do_raw_spin_lock+0xb5/0xc0 [41314.013236] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41314.018039] try_get_cap_refs+0x159/0x880 [ceph] [41314.022669] __ceph_get_caps+0x4a1/0x760 [ceph] [41314.027210] ? __file_remove_privs+0x89/0x110 [41314.031566] ? wake_bit_function+0x70/0x70 [41314.035658] ? ceph_write_iter+0x260/0x770 [ceph] [41314.040371] ceph_write_iter+0x260/0x770 [ceph] [41314.044908] ? lock_is_held_type+0xaa/0x120 [41314.049092] vfs_write+0x324/0x4f0 [41314.052502] ksys_pwrite64+0x65/0xa0 [41314.056074] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41314.061207] do_syscall_64+0x38/0x80 [41314.064787] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41314.069830] RIP: 0033:0x7f48f0eeb93b [41314.073400] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41314.092138] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [41314.099696] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41314.106819] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41314.113942] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41314.121068] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41314.128193] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41314.135322] [41314.137508] Sending NMI from CPU 0 to CPUs 7: [41314.141860] NMI backtrace for cpu 7 [41314.141862] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41314.141863] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41314.141864] Workqueue: events delayed_work [ceph] [41314.141880] RIP: 0010:mark_lock+0x23/0x4c0 [41314.141883] Code: 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 e4 f0 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 44 24 68 <31> c0 83 fa 09 0f 87 9b 00 00 00 49 89 ff 49 89 f5 89 d3 83 fa 08 [41314.141884] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000082 [41314.141885] RAX: c5e6ef7cc8fa9e00 RBX: ffff88810debaa80 RCX: 0000000000000000 [41314.141886] RDX: 0000000000000008 RSI: ffff88810debb550 RDI: ffff88810debaa80 [41314.141887] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [41314.141889] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [41314.141890] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000002 [41314.141891] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41314.141892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41314.141893] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41314.141894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41314.141895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41314.141895] Call Trace: [41314.141897] [41314.141898] ? nmi_cpu_backtrace+0xad/0x120 [41314.141901] ? nmi_cpu_backtrace_handler+0x11/0x20 [41314.141903] ? nmi_handle+0xd4/0x260 [41314.141907] ? default_do_nmi+0x49/0xf0 [41314.141909] ? exc_nmi+0x11e/0x150 [41314.141910] ? end_repeat_nmi+0x16/0x67 [41314.141914] ? mark_lock+0x23/0x4c0 [41314.141916] ? mark_lock+0x23/0x4c0 [41314.141917] ? mark_lock+0x23/0x4c0 [41314.141919] [41314.141919] [41314.141919] ? _raw_spin_unlock+0x23/0x40 [41314.141922] ? ceph_check_caps+0x275/0xae0 [ceph] [41314.141936] ? mark_held_locks+0x48/0x70 [41314.141937] __lock_acquire+0x4d6/0x1e20 [41314.141940] lock_acquire+0xcb/0x2b0 [41314.141941] ? igrab+0x1d/0x50 [41314.141943] ? igrab+0x43/0x50 [41314.141944] ? lock_release+0xc5/0x260 [41314.141946] _raw_spin_lock+0x30/0x40 [41314.141948] ? igrab+0x1d/0x50 [41314.141949] igrab+0x1d/0x50 [41314.141950] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41314.141963] delayed_work+0x1d8/0x280 [ceph] [41314.141976] ? process_scheduled_works+0x22a/0x510 [41314.141979] process_scheduled_works+0x2a4/0x510 [41314.141982] worker_thread+0x188/0x340 [41314.141985] ? rescuer_thread+0x310/0x310 [41314.141987] kthread+0xf5/0x130 [41314.141989] ? kthread_complete_and_exit+0x20/0x20 [41314.141992] ret_from_fork+0x31/0x40 [41314.141994] ? kthread_complete_and_exit+0x20/0x20 [41314.141996] ret_from_fork_asm+0x11/0x20 [41314.142000] [41338.175945] watchdog: BUG: soft lockup - CPU#0 stuck for 27385s! [dbench:78683] [41338.183246] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41338.187944] watchdog: BUG: soft lockup - CPU#7 stuck for 36382s! [kworker/7:4:78808] [41338.238696] irq event stamp: 73926208 [41338.246426] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [41338.250083] hardirqs last enabled at (73926207): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41338.258257] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [41338.268412] hardirqs last disabled at (73926208): [] sysvec_apic_timer_interrupt+0xe/0x90 [41338.275450] nft_compat nf_tables nfnetlink bridge stp llc xfs [41338.285176] softirqs last enabled at (73926158): [] __do_softirq+0x2be/0x371 [41338.291009] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [41338.299691] softirqs last disabled at (73926153): [] irq_exit_rcu+0xb0/0xd0 [41338.299695] coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [41338.316543] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41338.322452] iTCO_vendor_support i2c_smbus [41338.331395] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41338.331396] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41338.335486] crct10dif_pclmul joydev [41338.342959] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41338.348434] crc32_pclmul ipmi_ssif [41338.352005] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41338.370744] ghash_clmulni_intel mei_me [41338.374228] [41338.374229] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41338.379445] mei lpc_ich [41338.383275] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41338.384766] sha512_ssse3 acpi_ipmi [41338.391890] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41338.394419] mfd_core wmi [41338.401545] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41338.405027] ipmi_si ipmi_devintf [41338.412153] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41338.414770] ipmi_msghandler ioatdma [41338.421893] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41338.425204] acpi_pad acpi_power_meter [41338.432342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41338.435908] ixgbe igb [41338.443985] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41338.447731] mdio crc32c_intel [41338.453467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41338.455824] nvme ptp [41338.462948] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41338.466000] nvme_core [41338.473124] Call Trace: [41338.473126] [41338.475394] i2c_algo_bit [41338.482522] ? watchdog_timer_fn+0x1b4/0x220 [41338.484878] pps_core dca [41338.487341] ? __hrtimer_run_queues+0x1d3/0x360 [41338.489349] fuse [41338.491969] ? hrtimer_interrupt+0x104/0x240 [41338.496230] irq event stamp: 250731822 [41338.498846] ? __local_bh_enable+0x50/0x60 [41338.503372] hardirqs last enabled at (250731821): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41338.505299] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41338.509558] hardirqs last disabled at (250731822): [] sysvec_apic_timer_interrupt+0xe/0x90 [41338.513303] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41338.517395] softirqs last enabled at (250731820): [] __do_softirq+0x2be/0x371 [41338.527640] [41338.532856] softirqs last disabled at (250731815): [] irq_exit_rcu+0xb0/0xd0 [41338.542666] [41338.547624] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41338.556412] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41338.558510] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41338.567110] ? queued_spin_lock_slowpath+0x18/0x2e0 [41338.569205] Workqueue: events delayed_work [ceph] [41338.578584] do_raw_spin_lock+0xb5/0xc0 [41338.583887] [41338.583888] RIP: 0010:lock_release+0x14e/0x260 [41338.591358] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41338.596227] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [41338.600929] try_get_cap_refs+0x159/0x880 [ceph] [41338.604756] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [41338.606252] __ceph_get_caps+0x4a1/0x760 [ceph] [41338.610684] [41338.610685] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [41338.615472] ? __file_remove_privs+0x89/0x110 [41338.634205] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [41338.638817] ? wake_bit_function+0x70/0x70 [41338.644034] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [41338.648561] ? ceph_write_iter+0x260/0x770 [ceph] [41338.650048] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [41338.657174] ceph_write_iter+0x260/0x770 [ceph] [41338.661523] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [41338.668650] ? lock_is_held_type+0xaa/0x120 [41338.672739] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41338.679865] vfs_write+0x324/0x4f0 [41338.684560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41338.691688] ksys_pwrite64+0x65/0xa0 [41338.696209] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41338.703347] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41338.707526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41338.715605] do_syscall_64+0x38/0x80 [41338.719019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41338.724758] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41338.728344] Call Trace: [41338.728345] [41338.735470] RIP: 0033:0x7f48f0eeb93b [41338.740600] ? watchdog_timer_fn+0x1b4/0x220 [41338.747725] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41338.751295] ? __hrtimer_run_queues+0x1d3/0x360 [41338.758419] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41338.763465] ? hrtimer_interrupt+0x104/0x240 [41338.765907] ORIG_RAX: 0000000000000012 [41338.767918] ? __local_bh_enable+0x50/0x60 [41338.771488] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41338.775756] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41338.794490] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41338.799014] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41338.804233] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41338.808496] [41338.812343] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41338.816436] [41338.816437] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41338.823560] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41338.828776] ? igrab+0x43/0x50 [41338.835906] [41338.840862] ? lock_release+0x14e/0x260 [41338.880813] ? lock_release+0xc5/0x260 [41338.884558] _raw_spin_unlock+0x1b/0x40 [41338.888397] igrab+0x43/0x50 [41338.891275] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41338.896425] delayed_work+0x1d8/0x280 [ceph] [41338.900705] ? process_scheduled_works+0x22a/0x510 [41338.905495] process_scheduled_works+0x2a4/0x510 [41338.910110] worker_thread+0x188/0x340 [41338.913861] ? rescuer_thread+0x310/0x310 [41338.917871] kthread+0xf5/0x130 [41338.921011] ? kthread_complete_and_exit+0x20/0x20 [41338.925802] ret_from_fork+0x31/0x40 [41338.929383] ? kthread_complete_and_exit+0x20/0x20 [41338.934165] ret_from_fork_asm+0x11/0x20 [41338.938088] [41366.176437] watchdog: BUG: soft lockup - CPU#0 stuck for 27411s! [dbench:78683] [41366.183742] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41366.188437] watchdog: BUG: soft lockup - CPU#7 stuck for 36408s! [kworker/7:4:78808] [41366.239182] irq event stamp: 73984288 [41366.239184] hardirqs last enabled at (73984287): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41366.246913] Modules linked in: ceph [41366.250571] hardirqs last disabled at (73984288): [] sysvec_apic_timer_interrupt+0xe/0x90 [41366.260728] libceph [41366.264211] softirqs last enabled at (73984230): [] __do_softirq+0x2be/0x371 [41366.273938] dns_resolver fscache [41366.276121] softirqs last disabled at (73984225): [] irq_exit_rcu+0xb0/0xd0 [41366.284805] netfs [41366.288136] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41366.296643] veth nft_chain_nat [41366.298653] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41366.307597] xt_MASQUERADE nf_nat [41366.310736] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41366.318207] nf_conntrack nf_defrag_ipv6 [41366.321518] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41366.326994] nf_defrag_ipv4 nft_compat [41366.330913] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41366.349650] nf_tables nfnetlink [41366.353395] [41366.353396] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41366.358611] bridge stp [41366.361835] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41366.363326] llc xfs [41366.370450] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41366.372893] libcrc32c sunrpc [41366.380018] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41366.382201] intel_rapl_msr intel_rapl_common [41366.389326] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41366.392291] sb_edac x86_pkg_temp_thermal [41366.399415] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41366.403765] coretemp kvm_intel [41366.410889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41366.414893] kvm irqbypass [41366.422972] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41366.426115] i2c_i801 iTCO_wdt [41366.431854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41366.434559] iTCO_vendor_support i2c_smbus [41366.441682] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41366.444734] crct10dif_pclmul joydev [41366.451858] Call Trace: [41366.451860] [41366.455947] crc32_pclmul ipmi_ssif [41366.463074] ? watchdog_timer_fn+0x1b4/0x220 [41366.466645] ghash_clmulni_intel mei_me [41366.469089] ? __hrtimer_run_queues+0x1d3/0x360 [41366.471097] mei lpc_ich [41366.474603] ? hrtimer_interrupt+0x104/0x240 [41366.478863] sha512_ssse3 [41366.482693] ? __local_bh_enable+0x50/0x60 [41366.487218] acpi_ipmi mfd_core [41366.489751] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41366.494014] wmi ipmi_si [41366.496632] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41366.500720] ipmi_devintf ipmi_msghandler [41366.503860] [41366.509075] ioatdma [41366.511608] [41366.511609] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41366.516564] acpi_pad acpi_power_meter [41366.520572] ? queued_spin_lock_slowpath+0x18/0x2e0 [41366.522664] ixgbe igb [41366.524852] do_raw_spin_lock+0xb5/0xc0 [41366.526947] mdio crc32c_intel [41366.532253] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41366.535996] nvme ptp nvme_core [41366.540868] try_get_cap_refs+0x159/0x880 [ceph] [41366.543224] i2c_algo_bit pps_core [41366.547060] __ceph_get_caps+0x4a1/0x760 [ceph] [41366.550119] dca [41366.554909] ? __file_remove_privs+0x89/0x110 [41366.558043] fuse [41366.562656] ? wake_bit_function+0x70/0x70 [41366.566053] irq event stamp: 250952804 [41366.570578] ? ceph_write_iter+0x260/0x770 [ceph] [41366.572413] hardirqs last enabled at (250952803): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41366.576763] ceph_write_iter+0x260/0x770 [ceph] [41366.578688] hardirqs last disabled at (250952804): [] sysvec_apic_timer_interrupt+0xe/0x90 [41366.582782] ? lock_is_held_type+0xaa/0x120 [41366.586522] softirqs last enabled at (250952802): [] __do_softirq+0x2be/0x371 [41366.591222] vfs_write+0x324/0x4f0 [41366.601465] softirqs last disabled at (250952797): [] irq_exit_rcu+0xb0/0xd0 [41366.605993] ksys_pwrite64+0x65/0xa0 [41366.615800] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41366.619977] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41366.628749] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41366.632146] do_syscall_64+0x38/0x80 [41366.640744] Workqueue: events delayed_work [ceph] [41366.644314] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41366.653690] [41366.653690] RIP: 0010:rcu_is_watching+0xb/0x40 [41366.658838] RIP: 0033:0x7f48f0eeb93b [41366.666308] Code: 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 65 ff 05 9d 5b e9 7e <48> c7 c0 28 a3 02 00 65 48 03 05 c6 cf e7 7e 8b 00 c1 e8 02 83 e0 [41366.669880] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41366.674577] RSP: 0018:ffffc90002117d40 EFLAGS: 00000207 [41366.679621] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41366.681118] [41366.685560] ORIG_RAX: 0000000000000012 [41366.689130] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [41366.689131] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [41366.707867] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41366.726604] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [41366.731838] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41366.737057] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [41366.738548] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41366.742378] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41366.749503] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41366.756628] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41366.763752] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41366.770874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41366.778004] [41366.785129] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41366.836753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41366.843877] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41366.851001] Call Trace: [41366.853444] [41366.855456] ? watchdog_timer_fn+0x1b4/0x220 [41366.859719] ? __hrtimer_run_queues+0x1d3/0x360 [41366.864246] ? hrtimer_interrupt+0x104/0x240 [41366.868516] ? __local_bh_enable+0x50/0x60 [41366.872608] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41366.877843] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41366.882799] [41366.884895] [41366.886993] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41366.892297] ? igrab+0x43/0x50 [41366.895351] ? igrab+0x43/0x50 [41366.898399] ? rcu_is_watching+0xb/0x40 [41366.902232] lock_release+0x1d7/0x260 [41366.905891] _raw_spin_unlock+0x1b/0x40 [41366.909728] igrab+0x43/0x50 [41366.912606] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41366.917755] delayed_work+0x1d8/0x280 [ceph] [41366.922036] ? process_scheduled_works+0x22a/0x510 [41366.926843] process_scheduled_works+0x2a4/0x510 [41366.931458] worker_thread+0x188/0x340 [41366.935208] ? rescuer_thread+0x310/0x310 [41366.939210] kthread+0xf5/0x130 [41366.942349] ? kthread_complete_and_exit+0x20/0x20 [41366.947133] ret_from_fork+0x31/0x40 [41366.950711] ? kthread_complete_and_exit+0x20/0x20 [41366.955495] ret_from_fork_asm+0x11/0x20 [41366.959420] [41394.176930] watchdog: BUG: soft lockup - CPU#0 stuck for 27437s! [dbench:78683] [41394.184231] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41394.188929] watchdog: BUG: soft lockup - CPU#7 stuck for 36434s! [kworker/7:4:78808] [41394.239678] irq event stamp: 74042464 [41394.247409] Modules linked in: ceph libceph dns_resolver fscache netfs veth [41394.251066] hardirqs last enabled at (74042463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41394.258027] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [41394.268183] hardirqs last disabled at (74042464): [] sysvec_apic_timer_interrupt+0xe/0x90 [41394.268186] nf_defrag_ipv4 [41394.268188] softirqs last enabled at (74042406): [] __do_softirq+0x2be/0x371 [41394.275134] nft_compat nf_tables [41394.284859] softirqs last disabled at (74042401): [] irq_exit_rcu+0xb0/0xd0 [41394.287650] nfnetlink bridge [41394.296335] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41394.299644] stp llc [41394.308156] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41394.311119] xfs libcrc32c [41394.320065] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41394.322248] sunrpc [41394.329720] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41394.332422] intel_rapl_msr intel_rapl_common [41394.337901] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41394.339997] sb_edac x86_pkg_temp_thermal [41394.358735] [41394.358736] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41394.363086] coretemp kvm_intel [41394.368321] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41394.372324] kvm irqbypass [41394.373816] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41394.380938] i2c_i801 iTCO_wdt [41394.384076] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41394.391200] iTCO_vendor_support i2c_smbus [41394.393905] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41394.401029] crct10dif_pclmul [41394.404079] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41394.411204] joydev crc32_pclmul [41394.415294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41394.422419] ipmi_ssif ghash_clmulni_intel [41394.425382] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41394.433460] mei_me mei [41394.436685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41394.442421] lpc_ich [41394.446511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41394.453637] sha512_ssse3 [41394.456080] Call Trace: [41394.456083] [41394.463206] acpi_ipmi mfd_core [41394.465392] ? watchdog_timer_fn+0x1b4/0x220 [41394.472512] wmi ipmi_si [41394.475132] ? __hrtimer_run_queues+0x1d3/0x360 [41394.477574] ipmi_devintf [41394.479605] ? hrtimer_interrupt+0x104/0x240 [41394.482739] ipmi_msghandler ioatdma [41394.487005] ? __local_bh_enable+0x50/0x60 [41394.489535] acpi_pad acpi_power_meter [41394.494062] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41394.496677] ixgbe igb [41394.500942] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41394.504510] mdio crc32c_intel [41394.508601] [41394.512345] nvme ptp [41394.517563] [41394.519921] nvme_core i2c_algo_bit [41394.524879] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41394.527930] pps_core dca [41394.530031] ? queued_spin_lock_slowpath+0x16/0x2e0 [41394.532297] fuse [41394.534395] do_raw_spin_lock+0xb5/0xc0 [41394.537877] [41394.537878] irq event stamp: 251173652 [41394.543184] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41394.545799] hardirqs last enabled at (251173651): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41394.550674] try_get_cap_refs+0x159/0x880 [ceph] [41394.552609] hardirqs last disabled at (251173652): [] sysvec_apic_timer_interrupt+0xe/0x90 [41394.556448] __ceph_get_caps+0x4a1/0x760 [ceph] [41394.557933] softirqs last enabled at (251173650): [] __do_softirq+0x2be/0x371 [41394.561681] ? __file_remove_privs+0x89/0x110 [41394.566462] softirqs last disabled at (251173645): [] irq_exit_rcu+0xb0/0xd0 [41394.576705] ? wake_bit_function+0x70/0x70 [41394.581327] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41394.591139] ? ceph_write_iter+0x260/0x770 [ceph] [41394.595660] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41394.604431] ceph_write_iter+0x260/0x770 [ceph] [41394.608781] Workqueue: events delayed_work [ceph] [41394.617382] ? lock_is_held_type+0xaa/0x120 [41394.621470] [41394.621471] RIP: 0010:lock_release+0x14e/0x260 [41394.630850] vfs_write+0x324/0x4f0 [41394.635545] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [41394.643021] ksys_pwrite64+0x65/0xa0 [41394.647540] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [41394.652237] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41394.656414] [41394.656415] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [41394.657905] do_syscall_64+0x38/0x80 [41394.662343] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [41394.665741] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41394.684477] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [41394.688050] RIP: 0033:0x7f48f0eeb93b [41394.693265] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [41394.698397] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41394.699886] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [41394.707012] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41394.710582] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41394.717706] ORIG_RAX: 0000000000000012 [41394.722751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41394.729873] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41394.733446] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41394.740570] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41394.759306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41394.766449] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41394.771666] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41394.779743] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41394.783573] Call Trace: [41394.783574] [41394.789311] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41394.796454] ? watchdog_timer_fn+0x1b4/0x220 [41394.803582] [41394.810702] ? __hrtimer_run_queues+0x1d3/0x360 [41394.861754] ? hrtimer_interrupt+0x104/0x240 [41394.866021] ? __local_bh_enable+0x50/0x60 [41394.870115] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41394.875338] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41394.880296] [41394.882394] [41394.884491] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41394.889796] ? igrab+0x43/0x50 [41394.892848] ? lock_release+0x14e/0x260 [41394.896679] ? lock_release+0xc5/0x260 [41394.900423] _raw_spin_unlock+0x1b/0x40 [41394.904279] igrab+0x43/0x50 [41394.907157] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41394.912305] delayed_work+0x1d8/0x280 [ceph] [41394.916620] ? process_scheduled_works+0x22a/0x510 [41394.921403] process_scheduled_works+0x2a4/0x510 [41394.926018] worker_thread+0x188/0x340 [41394.929769] ? rescuer_thread+0x310/0x310 [41394.933780] kthread+0xf5/0x130 [41394.936919] ? kthread_complete_and_exit+0x20/0x20 [41394.941700] ret_from_fork+0x31/0x40 [41394.945281] ? kthread_complete_and_exit+0x20/0x20 [41394.950065] ret_from_fork_asm+0x11/0x20 [41394.953996] [41422.177419] watchdog: BUG: soft lockup - CPU#0 stuck for 27463s! [dbench:78683] [41422.184717] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41422.189418] watchdog: BUG: soft lockup - CPU#7 stuck for 36460s! [kworker/7:4:78808] [41422.240158] irq event stamp: 74100612 [41422.240159] hardirqs last enabled at (74100611): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41422.247887] Modules linked in: [41422.251546] hardirqs last disabled at (74100612): [] sysvec_apic_timer_interrupt+0xe/0x90 [41422.261702] ceph [41422.264754] softirqs last enabled at (74100576): [] __do_softirq+0x2be/0x371 [41422.274478] libceph [41422.276401] softirqs last disabled at (74100571): [] irq_exit_rcu+0xb0/0xd0 [41422.285091] dns_resolver fscache [41422.287279] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41422.295790] netfs veth [41422.299118] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41422.308062] nft_chain_nat xt_MASQUERADE [41422.310505] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41422.317977] nf_nat nf_conntrack [41422.321894] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41422.327372] nf_defrag_ipv6 nf_defrag_ipv4 [41422.330594] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41422.349334] nft_compat nf_tables [41422.353423] [41422.353424] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41422.358641] nfnetlink bridge [41422.361952] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41422.363443] stp [41422.370566] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41422.373530] llc xfs [41422.380655] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41422.382493] libcrc32c sunrpc [41422.389616] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41422.391813] intel_rapl_msr intel_rapl_common [41422.398934] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41422.401898] sb_edac [41422.409022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41422.413373] x86_pkg_temp_thermal coretemp [41422.421449] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41422.423633] kvm_intel kvm [41422.429372] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41422.433463] irqbypass i2c_i801 [41422.440586] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41422.443290] iTCO_wdt [41422.450414] Call Trace: [41422.453552] iTCO_vendor_support i2c_smbus [41422.460676] [41422.462947] crct10dif_pclmul joydev [41422.465392] ? watchdog_timer_fn+0x1b4/0x220 [41422.469482] crc32_pclmul ipmi_ssif [41422.471493] ? __hrtimer_run_queues+0x1d3/0x360 [41422.475064] ghash_clmulni_intel mei_me [41422.479331] ? hrtimer_interrupt+0x104/0x240 [41422.482817] mei lpc_ich [41422.487344] ? __local_bh_enable+0x50/0x60 [41422.491175] sha512_ssse3 acpi_ipmi [41422.495441] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41422.497968] mfd_core wmi [41422.502060] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41422.505543] ipmi_si ipmi_devintf [41422.510763] [41422.513378] ipmi_msghandler ioatdma [41422.518337] [41422.521646] acpi_pad acpi_power_meter [41422.523745] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41422.527316] ixgbe igb [41422.529417] ? queued_spin_lock_slowpath+0x16/0x2e0 [41422.533158] mdio crc32c_intel [41422.538463] do_raw_spin_lock+0xb5/0xc0 [41422.540818] nvme ptp [41422.545690] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41422.548740] nvme_core [41422.552573] try_get_cap_refs+0x159/0x880 [ceph] [41422.554840] i2c_algo_bit [41422.559629] __ceph_get_caps+0x4a1/0x760 [ceph] [41422.561983] pps_core dca [41422.566597] ? __file_remove_privs+0x89/0x110 [41422.569211] fuse [41422.573735] ? wake_bit_function+0x70/0x70 [41422.576353] irq event stamp: 251394572 [41422.580706] ? ceph_write_iter+0x260/0x770 [ceph] [41422.582626] hardirqs last enabled at (251394571): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41422.586718] ceph_write_iter+0x260/0x770 [ceph] [41422.590461] hardirqs last disabled at (251394572): [] sysvec_apic_timer_interrupt+0xe/0x90 [41422.595162] ? lock_is_held_type+0xaa/0x120 [41422.605404] softirqs last enabled at (251394570): [] __do_softirq+0x2be/0x371 [41422.609931] vfs_write+0x324/0x4f0 [41422.619739] softirqs last disabled at (251394565): [] irq_exit_rcu+0xb0/0xd0 [41422.623921] ksys_pwrite64+0x65/0xa0 [41422.632687] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41422.636084] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41422.644698] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41422.648270] do_syscall_64+0x38/0x80 [41422.657647] Workqueue: events delayed_work [ceph] [41422.662778] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41422.670248] [41422.670249] RIP: 0010:lock_acquire+0xf0/0x2b0 [41422.673820] RIP: 0033:0x7f48f0eeb93b [41422.678517] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41422.683563] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41422.685051] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41422.689401] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41422.692974] [41422.692975] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41422.711711] ORIG_RAX: 0000000000000012 [41422.730448] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41422.735667] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41422.740883] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41422.742375] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41422.749499] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41422.753327] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41422.760454] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41422.767578] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41422.774702] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41422.781825] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41422.788950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41422.796079] [41422.803198] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41422.840571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41422.847694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41422.854823] Call Trace: [41422.857271] [41422.859283] ? watchdog_timer_fn+0x1b4/0x220 [41422.863546] ? __hrtimer_run_queues+0x1d3/0x360 [41422.868072] ? hrtimer_interrupt+0x104/0x240 [41422.872342] ? __local_bh_enable+0x50/0x60 [41422.876435] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41422.881659] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41422.886615] [41422.888713] [41422.890825] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41422.896138] ? lock_acquire+0xf0/0x2b0 [41422.899887] ? igrab+0x43/0x50 [41422.902936] ? lock_release+0xc5/0x260 [41422.906681] _raw_spin_lock+0x30/0x40 [41422.910338] ? igrab+0x1d/0x50 [41422.913389] igrab+0x1d/0x50 [41422.916266] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41422.921416] delayed_work+0x1d8/0x280 [ceph] [41422.925698] ? process_scheduled_works+0x22a/0x510 [41422.930487] process_scheduled_works+0x2a4/0x510 [41422.935117] worker_thread+0x188/0x340 [41422.938862] ? rescuer_thread+0x310/0x310 [41422.942863] kthread+0xf5/0x130 [41422.946002] ? kthread_complete_and_exit+0x20/0x20 [41422.950785] ret_from_fork+0x31/0x40 [41422.954356] ? kthread_complete_and_exit+0x20/0x20 [41422.959141] ret_from_fork_asm+0x11/0x20 [41422.963064] [41450.177908] watchdog: BUG: soft lockup - CPU#0 stuck for 27489s! [dbench:78683] [41450.185214] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41450.189907] watchdog: BUG: soft lockup - CPU#7 stuck for 36486s! [kworker/7:4:78808] [41450.240662] irq event stamp: 74158794 [41450.248394] Modules linked in: ceph libceph dns_resolver fscache netfs [41450.252051] hardirqs last enabled at (74158793): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41450.258583] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [41450.268741] hardirqs last disabled at (74158794): [] sysvec_apic_timer_interrupt+0xe/0x90 [41450.268744] nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [41450.284550] softirqs last enabled at (74158748): [] __do_softirq+0x2be/0x371 [41450.292281] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [41450.300983] softirqs last disabled at (74158743): [] irq_exit_rcu+0xb0/0xd0 [41450.307500] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [41450.316012] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41450.322008] irqbypass [41450.330952] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41450.330953] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41450.333309] i2c_i801 iTCO_wdt [41450.340782] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41450.346259] iTCO_vendor_support i2c_smbus [41450.349309] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41450.368047] crct10dif_pclmul joydev [41450.372139] [41450.372140] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41450.377354] crc32_pclmul ipmi_ssif [41450.380926] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41450.382417] ghash_clmulni_intel mei_me [41450.389541] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41450.393026] mei lpc_ich [41450.400148] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41450.403979] sha512_ssse3 [41450.411103] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41450.413633] acpi_ipmi mfd_core [41450.420758] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41450.423375] wmi ipmi_si [41450.430500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41450.433637] ipmi_devintf ipmi_msghandler [41450.441716] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41450.444247] ioatdma [41450.449983] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41450.453986] acpi_pad acpi_power_meter [41450.461111] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41450.463305] ixgbe [41450.470427] Call Trace: [41450.470430] [41450.474173] igb mdio [41450.481306] ? watchdog_timer_fn+0x1b4/0x220 [41450.483315] crc32c_intel nvme [41450.485762] ? __hrtimer_run_queues+0x1d3/0x360 [41450.487770] ptp nvme_core [41450.490044] ? hrtimer_interrupt+0x104/0x240 [41450.494306] i2c_algo_bit pps_core [41450.497355] ? __local_bh_enable+0x50/0x60 [41450.501881] dca fuse [41450.504587] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41450.508848] [41450.508848] irq event stamp: 251615388 [41450.512246] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41450.516335] hardirqs last enabled at (251615387): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41450.518607] [41450.523823] hardirqs last disabled at (251615388): [] sysvec_apic_timer_interrupt+0xe/0x90 [41450.525315] [41450.529059] softirqs last enabled at (251615386): [] __do_softirq+0x2be/0x371 [41450.534018] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41450.544261] softirqs last disabled at (251615381): [] irq_exit_rcu+0xb0/0xd0 [41450.546364] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41450.556170] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41450.558269] do_raw_spin_lock+0xb5/0xc0 [41450.567036] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41450.572343] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41450.580940] Workqueue: events delayed_work [ceph] [41450.585813] try_get_cap_refs+0x159/0x880 [ceph] [41450.595187] [41450.595187] RIP: 0010:lock_acquire+0xf0/0x2b0 [41450.599023] __ceph_get_caps+0x4a1/0x760 [ceph] [41450.606489] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41450.611277] ? __file_remove_privs+0x89/0x110 [41450.615970] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41450.620589] ? wake_bit_function+0x70/0x70 [41450.622080] [41450.622081] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41450.626433] ? ceph_write_iter+0x260/0x770 [ceph] [41450.630954] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41450.649693] ceph_write_iter+0x260/0x770 [ceph] [41450.654044] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41450.659264] ? lock_is_held_type+0xaa/0x120 [41450.663352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41450.664844] vfs_write+0x324/0x4f0 [41450.671966] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41450.676670] ksys_pwrite64+0x65/0xa0 [41450.683789] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41450.688313] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41450.695436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41450.699616] do_syscall_64+0x38/0x80 [41450.706737] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41450.710137] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41450.717260] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41450.720833] RIP: 0033:0x7f48f0eeb93b [41450.728909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41450.734040] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41450.739775] Call Trace: [41450.739776] [41450.743346] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41450.750471] ? watchdog_timer_fn+0x1b4/0x220 [41450.755515] ORIG_RAX: 0000000000000012 [41450.762639] ? __hrtimer_run_queues+0x1d3/0x360 [41450.766211] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41450.773336] ? hrtimer_interrupt+0x104/0x240 [41450.792072] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41450.794515] ? __local_bh_enable+0x50/0x60 [41450.796526] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41450.801747] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41450.806007] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41450.809840] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41450.814364] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41450.821487] [41450.825757] [41450.832875] [41450.874893] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41450.880201] ? lock_acquire+0xf0/0x2b0 [41450.883951] ? igrab+0x43/0x50 [41450.887000] ? lock_release+0xc5/0x260 [41450.890745] _raw_spin_lock+0x30/0x40 [41450.894402] ? igrab+0x1d/0x50 [41450.897451] igrab+0x1d/0x50 [41450.900330] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41450.905480] delayed_work+0x1d8/0x280 [ceph] [41450.909761] ? process_scheduled_works+0x22a/0x510 [41450.914552] process_scheduled_works+0x2a4/0x510 [41450.919168] worker_thread+0x188/0x340 [41450.922916] ? rescuer_thread+0x310/0x310 [41450.926928] kthread+0xf5/0x130 [41450.930068] ? kthread_complete_and_exit+0x20/0x20 [41450.934858] ret_from_fork+0x31/0x40 [41450.938439] ? kthread_complete_and_exit+0x20/0x20 [41450.943221] ret_from_fork_asm+0x11/0x20 [41450.947146] [41478.178398] watchdog: BUG: soft lockup - CPU#0 stuck for 27515s! [dbench:78683] [41478.185701] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41478.190398] watchdog: BUG: soft lockup - CPU#7 stuck for 36512s! [kworker/7:4:78808] [41478.241141] irq event stamp: 74216936 [41478.241142] hardirqs last enabled at (74216935): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41478.248871] Modules linked in: [41478.252529] hardirqs last disabled at (74216936): [] sysvec_apic_timer_interrupt+0xe/0x90 [41478.262687] ceph [41478.265737] softirqs last enabled at (74216878): [] __do_softirq+0x2be/0x371 [41478.275462] libceph dns_resolver [41478.277386] softirqs last disabled at (74216873): [] irq_exit_rcu+0xb0/0xd0 [41478.286081] fscache netfs [41478.289388] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41478.297901] veth [41478.300605] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41478.309547] nft_chain_nat [41478.311473] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41478.318943] xt_MASQUERADE nf_nat [41478.321646] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41478.327125] nf_conntrack nf_defrag_ipv6 [41478.330436] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41478.349172] nf_defrag_ipv4 nft_compat [41478.353091] [41478.353091] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41478.358306] nf_tables nfnetlink [41478.362051] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41478.363561] bridge stp [41478.370683] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41478.373908] llc xfs [41478.381031] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41478.383475] libcrc32c [41478.390601] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41478.392795] sunrpc intel_rapl_msr [41478.399917] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41478.402273] intel_rapl_common [41478.409399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41478.412799] sb_edac [41478.420873] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41478.423924] x86_pkg_temp_thermal coretemp [41478.429662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41478.431845] kvm_intel kvm [41478.438969] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41478.443074] irqbypass i2c_i801 [41478.450202] Call Trace: [41478.452906] iTCO_wdt iTCO_vendor_support [41478.460032] [41478.463167] i2c_smbus crct10dif_pclmul [41478.465613] ? watchdog_timer_fn+0x1b4/0x220 [41478.469616] joydev crc32_pclmul [41478.471627] ? __hrtimer_run_queues+0x1d3/0x360 [41478.475456] ipmi_ssif ghash_clmulni_intel [41478.479725] ? hrtimer_interrupt+0x104/0x240 [41478.482944] mei_me mei [41478.487470] ? __local_bh_enable+0x50/0x60 [41478.491561] lpc_ich sha512_ssse3 [41478.495826] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41478.498268] acpi_ipmi mfd_core [41478.502361] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41478.505670] wmi ipmi_si [41478.510887] [41478.514024] ipmi_devintf ipmi_msghandler [41478.518982] [41478.521513] ioatdma acpi_pad [41478.523612] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41478.527614] acpi_power_meter [41478.529717] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41478.532674] ixgbe igb [41478.537982] do_raw_spin_lock+0xb5/0xc0 [41478.540944] mdio crc32c_intel [41478.545815] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41478.548173] nvme ptp [41478.552004] try_get_cap_refs+0x159/0x880 [ceph] [41478.555053] nvme_core i2c_algo_bit [41478.559842] __ceph_get_caps+0x4a1/0x760 [ceph] [41478.562108] pps_core [41478.566723] ? __file_remove_privs+0x89/0x110 [41478.570203] dca fuse [41478.574728] ? wake_bit_function+0x70/0x70 [41478.576998] [41478.576998] irq event stamp: 251836360 [41478.581352] ? ceph_write_iter+0x260/0x770 [ceph] [41478.583619] hardirqs last enabled at (251836359): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41478.587711] ceph_write_iter+0x260/0x770 [ceph] [41478.589201] hardirqs last disabled at (251836360): [] sysvec_apic_timer_interrupt+0xe/0x90 [41478.592948] ? lock_is_held_type+0xaa/0x120 [41478.597641] softirqs last enabled at (251836358): [] __do_softirq+0x2be/0x371 [41478.607890] vfs_write+0x324/0x4f0 [41478.612409] softirqs last disabled at (251836353): [] irq_exit_rcu+0xb0/0xd0 [41478.622226] ksys_pwrite64+0x65/0xa0 [41478.626399] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41478.635170] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41478.638566] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41478.647166] do_syscall_64+0x38/0x80 [41478.650735] Workqueue: events delayed_work [ceph] [41478.660113] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41478.665243] [41478.665244] RIP: 0010:lock_acquire+0xf0/0x2b0 [41478.672715] RIP: 0033:0x7f48f0eeb93b [41478.676286] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41478.680984] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41478.686026] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41478.687517] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41478.691869] [41478.691869] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41478.695438] ORIG_RAX: 0000000000000012 [41478.714177] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41478.732913] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41478.738131] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41478.743350] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41478.744840] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41478.751964] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41478.755800] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41478.762926] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41478.770051] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41478.777176] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41478.784298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41478.791430] [41478.798547] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41478.843052] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41478.850175] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41478.857301] Call Trace: [41478.859744] [41478.861757] ? watchdog_timer_fn+0x1b4/0x220 [41478.866020] ? __hrtimer_run_queues+0x1d3/0x360 [41478.870547] ? hrtimer_interrupt+0x104/0x240 [41478.874816] ? __local_bh_enable+0x50/0x60 [41478.878908] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41478.884124] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41478.889083] [41478.891180] [41478.893276] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41478.898585] ? lock_acquire+0xf0/0x2b0 [41478.902334] ? igrab+0x43/0x50 [41478.905384] ? lock_release+0xc5/0x260 [41478.909130] _raw_spin_lock+0x30/0x40 [41478.912804] ? igrab+0x1d/0x50 [41478.915854] igrab+0x1d/0x50 [41478.918731] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41478.923882] delayed_work+0x1d8/0x280 [ceph] [41478.928164] ? process_scheduled_works+0x22a/0x510 [41478.932953] process_scheduled_works+0x2a4/0x510 [41478.937569] worker_thread+0x188/0x340 [41478.941319] ? rescuer_thread+0x310/0x310 [41478.945322] kthread+0xf5/0x130 [41478.948458] ? kthread_complete_and_exit+0x20/0x20 [41478.953244] ret_from_fork+0x31/0x40 [41478.956814] ? kthread_complete_and_exit+0x20/0x20 [41478.961597] ret_from_fork_asm+0x11/0x20 [41478.965520] [41506.178889] watchdog: BUG: soft lockup - CPU#0 stuck for 27541s! [dbench:78683] [41506.186198] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41506.190888] watchdog: BUG: soft lockup - CPU#7 stuck for 36538s! [kworker/7:4:78808] [41506.241644] irq event stamp: 74275198 [41506.249376] Modules linked in: ceph libceph dns_resolver [41506.253035] hardirqs last enabled at (74275197): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41506.258347] fscache netfs veth nft_chain_nat [41506.268504] hardirqs last disabled at (74275198): [] sysvec_apic_timer_interrupt+0xe/0x90 [41506.268507] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [41506.282580] softirqs last enabled at (74275142): [] __do_softirq+0x2be/0x371 [41506.292311] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [41506.300995] softirqs last disabled at (74275137): [] irq_exit_rcu+0xb0/0xd0 [41506.306906] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [41506.315419] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41506.322627] kvm irqbypass [41506.331571] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41506.331573] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41506.334284] i2c_i801 iTCO_wdt [41506.341757] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41506.347234] iTCO_vendor_support i2c_smbus [41506.350285] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41506.369021] crct10dif_pclmul joydev [41506.373113] [41506.373114] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41506.378329] crc32_pclmul ipmi_ssif [41506.381900] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41506.383391] ghash_clmulni_intel mei_me [41506.390516] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41506.393999] mei lpc_ich [41506.401125] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41506.404955] sha512_ssse3 acpi_ipmi [41506.412078] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41506.414609] mfd_core [41506.421734] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41506.425217] wmi ipmi_si [41506.432341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41506.434611] ipmi_devintf ipmi_msghandler [41506.442690] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41506.445220] ioatdma acpi_pad [41506.450958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41506.454963] acpi_power_meter [41506.462087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41506.465050] ixgbe igb [41506.472175] Call Trace: [41506.472178] [41506.475139] mdio crc32c_intel [41506.482284] ? watchdog_timer_fn+0x1b4/0x220 [41506.484638] nvme ptp [41506.487082] ? __hrtimer_run_queues+0x1d3/0x360 [41506.489092] nvme_core [41506.492146] ? hrtimer_interrupt+0x104/0x240 [41506.496407] i2c_algo_bit pps_core [41506.498677] ? __local_bh_enable+0x50/0x60 [41506.503201] dca fuse [41506.505563] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41506.509823] [41506.509824] irq event stamp: 252057078 [41506.513221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41506.517312] hardirqs last enabled at (252057077): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41506.519583] [41506.524799] hardirqs last disabled at (252057078): [] sysvec_apic_timer_interrupt+0xe/0x90 [41506.526291] [41506.526292] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41506.530034] softirqs last enabled at (252057076): [] __do_softirq+0x2be/0x371 [41506.534997] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41506.545236] softirqs last disabled at (252057071): [] irq_exit_rcu+0xb0/0xd0 [41506.547335] do_raw_spin_lock+0xb5/0xc0 [41506.557145] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41506.559243] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41506.564545] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41506.573336] try_get_cap_refs+0x159/0x880 [ceph] [41506.578206] Workqueue: events delayed_work [ceph] [41506.586808] __ceph_get_caps+0x4a1/0x760 [ceph] [41506.590633] [41506.590634] RIP: 0010:lock_acquire+0xf0/0x2b0 [41506.600016] ? __file_remove_privs+0x89/0x110 [41506.604795] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41506.612266] ? wake_bit_function+0x70/0x70 [41506.616893] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41506.621593] ? ceph_write_iter+0x260/0x770 [ceph] [41506.626114] [41506.627607] ceph_write_iter+0x260/0x770 [ceph] [41506.631956] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41506.636311] ? lock_is_held_type+0xaa/0x120 [41506.655046] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41506.659139] vfs_write+0x324/0x4f0 [41506.664351] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41506.669056] ksys_pwrite64+0x65/0xa0 [41506.670541] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41506.675065] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41506.682188] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41506.686370] do_syscall_64+0x38/0x80 [41506.693492] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41506.696890] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41506.704013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41506.707583] RIP: 0033:0x7f48f0eeb93b [41506.714706] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41506.719840] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41506.726963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41506.730533] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41506.738609] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41506.743655] ORIG_RAX: 0000000000000012 [41506.749392] Call Trace: [41506.749394] [41506.752962] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41506.760087] ? watchdog_timer_fn+0x1b4/0x220 [41506.778823] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41506.785950] ? __hrtimer_run_queues+0x1d3/0x360 [41506.791165] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41506.798297] ? hrtimer_interrupt+0x104/0x240 [41506.802120] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41506.804573] ? __local_bh_enable+0x50/0x60 [41506.806584] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41506.813710] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41506.817977] [41506.825098] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41506.871716] [41506.873813] [41506.875911] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41506.881219] ? lock_acquire+0xf0/0x2b0 [41506.884969] ? igrab+0x43/0x50 [41506.888019] ? lock_release+0xc5/0x260 [41506.891765] _raw_spin_lock+0x30/0x40 [41506.895429] ? igrab+0x1d/0x50 [41506.898480] igrab+0x1d/0x50 [41506.901359] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41506.906506] delayed_work+0x1d8/0x280 [ceph] [41506.910788] ? process_scheduled_works+0x22a/0x510 [41506.915578] process_scheduled_works+0x2a4/0x510 [41506.920211] worker_thread+0x188/0x340 [41506.923960] ? rescuer_thread+0x310/0x310 [41506.927963] kthread+0xf5/0x130 [41506.931101] ? kthread_complete_and_exit+0x20/0x20 [41506.935885] ret_from_fork+0x31/0x40 [41506.939466] ? kthread_complete_and_exit+0x20/0x20 [41506.944249] ret_from_fork_asm+0x11/0x20 [41506.948182] [41509.148306] rcu: INFO: rcu_preempt self-detected stall on CPU [41509.154066] rcu: 0-....: (28903172 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14478643 [41509.164578] rcu: (t=29572023 jiffies g=1309609 q=786630 ncpus=8) [41509.170662] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41509.179606] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41509.187075] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41509.192553] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41509.211292] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41509.216507] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41509.223633] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41509.230756] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41509.237881] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41509.245004] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41509.252129] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41509.260206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41509.265943] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41509.273069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41509.280192] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41509.287315] Call Trace: [41509.289760] [41509.291772] ? rcu_dump_cpu_stacks+0xd1/0x180 [41509.296131] ? rcu_sched_clock_irq+0x633/0x1390 [41509.300654] ? lock_is_held_type+0xaa/0x120 [41509.304832] ? lock_is_held_type+0xaa/0x120 [41509.309010] ? tick_sched_do_timer+0xa0/0xa0 [41509.313281] ? rcu_read_lock_sched_held+0x56/0x90 [41509.317979] ? tick_sched_do_timer+0xa0/0xa0 [41509.322242] ? update_process_times+0x5f/0x90 [41509.326595] ? tick_sched_handle.isra.16+0x1e/0x50 [41509.331378] ? tick_sched_timer+0x6f/0x80 [41509.335384] ? __hrtimer_run_queues+0x1d3/0x360 [41509.339911] ? hrtimer_interrupt+0x104/0x240 [41509.344180] ? __local_bh_enable+0x50/0x60 [41509.348273] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41509.353497] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41509.358453] [41509.360551] [41509.362648] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41509.367957] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41509.372834] do_raw_spin_lock+0xb5/0xc0 [41509.376663] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41509.381465] try_get_cap_refs+0x159/0x880 [ceph] [41509.386097] __ceph_get_caps+0x4a1/0x760 [ceph] [41509.390638] ? __file_remove_privs+0x89/0x110 [41509.394994] ? wake_bit_function+0x70/0x70 [41509.399086] ? ceph_write_iter+0x260/0x770 [ceph] [41509.403799] ceph_write_iter+0x260/0x770 [ceph] [41509.408335] ? lock_is_held_type+0xaa/0x120 [41509.412519] vfs_write+0x324/0x4f0 [41509.415928] ksys_pwrite64+0x65/0xa0 [41509.419503] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41509.424634] do_syscall_64+0x38/0x80 [41509.428205] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41509.433249] RIP: 0033:0x7f48f0eeb93b [41509.436818] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41509.455557] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [41509.463115] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41509.470237] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41509.477363] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41509.484487] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41509.491611] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41509.498740] [41509.500928] Sending NMI from CPU 0 to CPUs 7: [41509.505280] NMI backtrace for cpu 7 [41509.505282] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41509.505284] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41509.505285] Workqueue: events delayed_work [ceph] [41509.505300] RIP: 0010:mark_lock+0x71/0x4c0 [41509.505302] Code: 63 f6 41 0f b7 45 20 66 25 ff 1f 0f b7 c0 48 0f a3 05 d3 7e e0 01 73 79 48 8d 14 40 48 8d 04 90 48 c1 e0 04 48 05 20 55 f7 82 <41> bc 01 00 00 00 4c 85 70 60 0f 84 98 00 00 00 48 8b 4c 24 68 65 [41509.505304] RSP: 0018:ffffc90002117bb0 EFLAGS: 00000086 [41509.505305] RAX: ffffffff82fcfc30 RBX: 0000000000000008 RCX: 0000000000000000 [41509.505306] RDX: 00000000000014df RSI: ffff88810debb550 RDI: ffff88810debaa80 [41509.505307] RBP: ffffc90002117c50 R08: 0000000000000078 R09: 0000000000000000 [41509.505307] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000006f5 [41509.505308] R13: ffff88810debb550 R14: 0000000000000100 R15: ffff88810debaa80 [41509.505309] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41509.505310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41509.505311] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41509.505312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41509.505313] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41509.505314] Call Trace: [41509.505316] [41509.505317] ? nmi_cpu_backtrace+0xad/0x120 [41509.505321] ? nmi_cpu_backtrace_handler+0x11/0x20 [41509.505322] ? nmi_handle+0xd4/0x260 [41509.505326] ? default_do_nmi+0x49/0xf0 [41509.505328] ? exc_nmi+0x11e/0x150 [41509.505329] ? end_repeat_nmi+0x16/0x67 [41509.505333] ? mark_lock+0x71/0x4c0 [41509.505334] ? mark_lock+0x71/0x4c0 [41509.505336] ? mark_lock+0x71/0x4c0 [41509.505337] [41509.505337] [41509.505338] ? _raw_spin_unlock+0x23/0x40 [41509.505340] ? ceph_check_caps+0x275/0xae0 [ceph] [41509.505354] ? mark_held_locks+0x48/0x70 [41509.505355] __lock_acquire+0x4d6/0x1e20 [41509.505358] lock_acquire+0xcb/0x2b0 [41509.505360] ? igrab+0x1d/0x50 [41509.505361] ? igrab+0x43/0x50 [41509.505362] ? lock_release+0xc5/0x260 [41509.505364] _raw_spin_lock+0x30/0x40 [41509.505366] ? igrab+0x1d/0x50 [41509.505367] igrab+0x1d/0x50 [41509.505368] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41509.505380] delayed_work+0x1d8/0x280 [ceph] [41509.505394] ? process_scheduled_works+0x22a/0x510 [41509.505396] process_scheduled_works+0x2a4/0x510 [41509.505400] worker_thread+0x188/0x340 [41509.505402] ? rescuer_thread+0x310/0x310 [41509.505404] kthread+0xf5/0x130 [41509.505406] ? kthread_complete_and_exit+0x20/0x20 [41509.505408] ret_from_fork+0x31/0x40 [41509.505411] ? kthread_complete_and_exit+0x20/0x20 [41509.505413] ret_from_fork_asm+0x11/0x20 [41509.505416] [41534.179378] watchdog: BUG: soft lockup - CPU#0 stuck for 27567s! [dbench:78683] [41534.186683] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41534.191378] watchdog: BUG: soft lockup - CPU#7 stuck for 36564s! [kworker/7:4:78808] [41534.242124] irq event stamp: 74332590 [41534.242126] hardirqs last enabled at (74332589): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41534.249856] Modules linked in: ceph [41534.253514] hardirqs last disabled at (74332590): [] sysvec_apic_timer_interrupt+0xe/0x90 [41534.263670] libceph [41534.267154] softirqs last enabled at (74332532): [] __do_softirq+0x2be/0x371 [41534.276878] dns_resolver fscache [41534.279061] softirqs last disabled at (74332527): [] irq_exit_rcu+0xb0/0xd0 [41534.287747] netfs [41534.291078] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41534.299586] veth nft_chain_nat [41534.301595] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41534.301596] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41534.310540] xt_MASQUERADE nf_nat [41534.313678] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41534.321147] nf_conntrack nf_defrag_ipv6 [41534.326626] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41534.329937] nf_defrag_ipv4 nft_compat [41534.348675] [41534.348676] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41534.352593] nf_tables nfnetlink [41534.357809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41534.361552] bridge stp [41534.363061] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41534.370184] llc xfs [41534.373409] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41534.380534] libcrc32c sunrpc [41534.382978] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41534.390100] intel_rapl_msr intel_rapl_common [41534.392285] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41534.399410] sb_edac x86_pkg_temp_thermal [41534.402374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41534.409498] coretemp [41534.413850] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41534.421926] kvm_intel kvm [41534.425931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41534.431668] irqbypass i2c_i801 [41534.433939] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41534.441078] iTCO_wdt [41534.443783] Call Trace: [41534.443785] [41534.450908] iTCO_vendor_support i2c_smbus [41534.454059] ? watchdog_timer_fn+0x1b4/0x220 [41534.461178] crct10dif_pclmul joydev [41534.463452] ? __hrtimer_run_queues+0x1d3/0x360 [41534.465894] crc32_pclmul ipmi_ssif [41534.467908] ? hrtimer_interrupt+0x104/0x240 [41534.471994] ghash_clmulni_intel mei_me [41534.476258] ? __local_bh_enable+0x50/0x60 [41534.479829] mei lpc_ich [41534.484355] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41534.487838] sha512_ssse3 acpi_ipmi [41534.492102] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41534.495933] mfd_core wmi [41534.500024] [41534.502554] ipmi_si ipmi_devintf [41534.507777] [41534.511264] ipmi_msghandler ioatdma [41534.516223] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41534.518838] acpi_pad acpi_power_meter [41534.520940] ? queued_spin_lock_slowpath+0x18/0x2e0 [41534.524247] ixgbe igb [41534.526346] do_raw_spin_lock+0xb5/0xc0 [41534.529916] mdio crc32c_intel [41534.535219] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41534.538963] nvme ptp nvme_core [41534.543836] try_get_cap_refs+0x159/0x880 [ceph] [41534.546191] i2c_algo_bit pps_core [41534.550027] __ceph_get_caps+0x4a1/0x760 [ceph] [41534.553073] dca fuse [41534.557862] ? __file_remove_privs+0x89/0x110 [41534.560993] [41534.560994] irq event stamp: 252276060 [41534.565606] ? wake_bit_function+0x70/0x70 [41534.569003] hardirqs last enabled at (252276059): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41534.573528] ? ceph_write_iter+0x260/0x770 [ceph] [41534.575798] hardirqs last disabled at (252276060): [] sysvec_apic_timer_interrupt+0xe/0x90 [41534.580149] ceph_write_iter+0x260/0x770 [ceph] [41534.581640] softirqs last enabled at (252276058): [] __do_softirq+0x2be/0x371 [41534.585387] ? lock_is_held_type+0xaa/0x120 [41534.589474] softirqs last disabled at (252276053): [] irq_exit_rcu+0xb0/0xd0 [41534.599720] vfs_write+0x324/0x4f0 [41534.604414] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41534.614230] ksys_pwrite64+0x65/0xa0 [41534.618750] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41534.618751] Workqueue: events delayed_work [ceph] [41534.627520] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41534.631697] [41534.640298] do_syscall_64+0x38/0x80 [41534.643693] RIP: 0010:lock_acquire+0xf0/0x2b0 [41534.653070] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41534.656641] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41534.664112] RIP: 0033:0x7f48f0eeb93b [41534.668809] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41534.673941] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41534.675432] [41534.675433] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41534.679002] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41534.683352] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41534.688397] ORIG_RAX: 0000000000000012 [41534.707134] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41534.710706] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41534.715921] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41534.734660] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41534.736150] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41534.743275] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41534.748494] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41534.755615] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41534.759446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41534.766571] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41534.773695] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41534.780825] [41534.787945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41534.846688] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41534.853811] Call Trace: [41534.856255] [41534.858267] ? watchdog_timer_fn+0x1b4/0x220 [41534.862533] ? __hrtimer_run_queues+0x1d3/0x360 [41534.867074] ? hrtimer_interrupt+0x104/0x240 [41534.871335] ? __local_bh_enable+0x50/0x60 [41534.875431] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41534.880653] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41534.885610] [41534.887708] [41534.889805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41534.895115] ? lock_acquire+0xf0/0x2b0 [41534.898863] ? igrab+0x43/0x50 [41534.901913] ? lock_release+0xc5/0x260 [41534.905659] _raw_spin_lock+0x30/0x40 [41534.909314] ? igrab+0x1d/0x50 [41534.912366] igrab+0x1d/0x50 [41534.915244] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41534.920394] delayed_work+0x1d8/0x280 [ceph] [41534.924674] ? process_scheduled_works+0x22a/0x510 [41534.929464] process_scheduled_works+0x2a4/0x510 [41534.934080] worker_thread+0x188/0x340 [41534.937829] ? rescuer_thread+0x310/0x310 [41534.941833] kthread+0xf5/0x130 [41534.944970] ? kthread_complete_and_exit+0x20/0x20 [41534.949754] ret_from_fork+0x31/0x40 [41534.953325] ? kthread_complete_and_exit+0x20/0x20 [41534.958109] ret_from_fork_asm+0x11/0x20 [41534.962033] [41562.179867] watchdog: BUG: soft lockup - CPU#0 stuck for 27593s! [dbench:78683] [41562.187172] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41562.191867] watchdog: BUG: soft lockup - CPU#7 stuck for 36590s! [kworker/7:4:78808] [41562.242621] irq event stamp: 74390750 [41562.250352] Modules linked in: ceph libceph dns_resolver fscache [41562.254010] hardirqs last enabled at (74390749): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41562.254014] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [41562.270173] hardirqs last disabled at (74390750): [] sysvec_apic_timer_interrupt+0xe/0x90 [41562.278085] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [41562.287808] softirqs last enabled at (74390692): [] __do_softirq+0x2be/0x371 [41562.287812] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [41562.302941] softirqs last disabled at (74390687): [] irq_exit_rcu+0xb0/0xd0 [41562.311972] coretemp kvm_intel kvm irqbypass [41562.320485] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41562.320486] i2c_i801 [41562.320487] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41562.324833] iTCO_wdt iTCO_vendor_support [41562.333778] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41562.336048] i2c_smbus crct10dif_pclmul [41562.343521] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41562.347544] joydev crc32_pclmul [41562.353018] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41562.356850] ipmi_ssif ghash_clmulni_intel [41562.375588] [41562.375589] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41562.378812] mei_me [41562.384028] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41562.388119] mei lpc_ich [41562.389610] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41562.396733] sha512_ssse3 acpi_ipmi [41562.398831] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41562.405955] mfd_core [41562.408485] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41562.415609] wmi ipmi_si [41562.419094] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41562.426219] ipmi_devintf ipmi_msghandler [41562.428490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41562.435614] ioatdma [41562.438145] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41562.446222] acpi_pad [41562.450227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41562.455962] acpi_power_meter [41562.458146] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41562.465272] ixgbe igb [41562.467545] Call Trace: [41562.474666] mdio crc32c_intel [41562.477633] [41562.484753] nvme ptp [41562.487113] ? watchdog_timer_fn+0x1b4/0x220 [41562.489555] nvme_core i2c_algo_bit [41562.492606] ? __hrtimer_run_queues+0x1d3/0x360 [41562.494618] pps_core dca [41562.496892] ? hrtimer_interrupt+0x104/0x240 [41562.501153] fuse [41562.504636] ? __local_bh_enable+0x50/0x60 [41562.509160] irq event stamp: 252496702 [41562.511780] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41562.516041] hardirqs last enabled at (252496701): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41562.517968] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41562.522057] hardirqs last disabled at (252496702): [] sysvec_apic_timer_interrupt+0xe/0x90 [41562.525802] [41562.531018] softirqs last enabled at (252496700): [] __do_softirq+0x2be/0x371 [41562.541268] [41562.546227] softirqs last disabled at (252496695): [] irq_exit_rcu+0xb0/0xd0 [41562.556039] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41562.558137] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41562.566911] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41562.569003] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41562.577604] do_raw_spin_lock+0xb5/0xc0 [41562.582905] Workqueue: events delayed_work [ceph] [41562.592285] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41562.597154] [41562.604628] try_get_cap_refs+0x159/0x880 [ceph] [41562.608455] RIP: 0010:lock_acquire+0xf0/0x2b0 [41562.613159] __ceph_get_caps+0x4a1/0x760 [ceph] [41562.617936] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41562.619434] ? __file_remove_privs+0x89/0x110 [41562.624038] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41562.628390] ? wake_bit_function+0x70/0x70 [41562.632914] [41562.632915] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41562.651654] ? ceph_write_iter+0x260/0x770 [ceph] [41562.656003] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41562.661220] ceph_write_iter+0x260/0x770 [ceph] [41562.665309] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41562.666805] ? lock_is_held_type+0xaa/0x120 [41562.673925] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41562.678624] vfs_write+0x324/0x4f0 [41562.685747] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41562.690277] ksys_pwrite64+0x65/0xa0 [41562.697395] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41562.701573] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41562.708696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41562.712095] do_syscall_64+0x38/0x80 [41562.719217] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41562.722789] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41562.730866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41562.735999] RIP: 0033:0x7f48f0eeb93b [41562.741734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41562.745307] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41562.752429] Call Trace: [41562.757474] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41562.764597] [41562.768168] ORIG_RAX: 0000000000000012 [41562.775294] ? watchdog_timer_fn+0x1b4/0x220 [41562.794030] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41562.796475] ? __hrtimer_run_queues+0x1d3/0x360 [41562.801691] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41562.803705] ? hrtimer_interrupt+0x104/0x240 [41562.807550] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41562.811815] ? __local_bh_enable+0x50/0x60 [41562.818938] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41562.823466] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41562.830586] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41562.834852] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41562.841980] [41562.846065] [41562.874789] [41562.876885] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41562.882194] ? lock_acquire+0xf0/0x2b0 [41562.885944] ? igrab+0x43/0x50 [41562.888996] ? lock_release+0xc5/0x260 [41562.892738] _raw_spin_lock+0x30/0x40 [41562.896404] ? igrab+0x1d/0x50 [41562.899454] igrab+0x1d/0x50 [41562.902333] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41562.907483] delayed_work+0x1d8/0x280 [ceph] [41562.911764] ? process_scheduled_works+0x22a/0x510 [41562.916569] process_scheduled_works+0x2a4/0x510 [41562.921187] worker_thread+0x188/0x340 [41562.924934] ? rescuer_thread+0x310/0x310 [41562.928939] kthread+0xf5/0x130 [41562.932085] ? kthread_complete_and_exit+0x20/0x20 [41562.936870] ret_from_fork+0x31/0x40 [41562.940447] ? kthread_complete_and_exit+0x20/0x20 [41562.945233] ret_from_fork_asm+0x11/0x20 [41562.949163] [41590.180355] watchdog: BUG: soft lockup - CPU#0 stuck for 27619s! [dbench:78683] [41590.187659] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41590.192355] watchdog: BUG: soft lockup - CPU#7 stuck for 36616s! [kworker/7:4:78808] [41590.243099] irq event stamp: 74448860 [41590.243100] hardirqs last enabled at (74448859): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41590.250830] Modules linked in: [41590.254488] hardirqs last disabled at (74448860): [] sysvec_apic_timer_interrupt+0xe/0x90 [41590.264644] ceph [41590.267696] softirqs last enabled at (74448802): [] __do_softirq+0x2be/0x371 [41590.277421] libceph [41590.279344] softirqs last disabled at (74448797): [] irq_exit_rcu+0xb0/0xd0 [41590.288029] dns_resolver [41590.290212] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41590.298722] fscache netfs [41590.301358] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41590.310301] veth [41590.313006] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41590.320478] nft_chain_nat [41590.322402] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41590.327878] xt_MASQUERADE nf_nat [41590.330583] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41590.349321] nf_conntrack nf_defrag_ipv6 [41590.352631] [41590.352632] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41590.357848] nf_defrag_ipv4 [41590.361765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41590.363256] nft_compat nf_tables [41590.370381] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41590.373171] nfnetlink bridge [41590.380294] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41590.383606] stp llc [41590.390748] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41590.393712] xfs [41590.400836] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41590.403033] libcrc32c sunrpc [41590.410160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41590.411998] intel_rapl_msr intel_rapl_common [41590.420076] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41590.423040] sb_edac [41590.428777] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41590.433129] x86_pkg_temp_thermal [41590.440253] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41590.442436] coretemp kvm_intel [41590.449560] Call Trace: [41590.449562] [41590.452871] kvm irqbypass [41590.459997] ? watchdog_timer_fn+0x1b4/0x220 [41590.463133] i2c_i801 iTCO_wdt [41590.465578] ? __hrtimer_run_queues+0x1d3/0x360 [41590.467587] iTCO_vendor_support i2c_smbus [41590.470296] ? hrtimer_interrupt+0x104/0x240 [41590.474556] crct10dif_pclmul joydev [41590.477606] ? __local_bh_enable+0x50/0x60 [41590.482129] crc32_pclmul ipmi_ssif [41590.486223] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41590.490484] ghash_clmulni_intel mei_me [41590.494056] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41590.498146] mei lpc_ich [41590.501630] [41590.506848] sha512_ssse3 acpi_ipmi [41590.510680] [41590.510682] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41590.515635] mfd_core wmi [41590.518172] ? queued_spin_lock_slowpath+0x16/0x2e0 [41590.520264] ipmi_si ipmi_devintf [41590.523757] do_raw_spin_lock+0xb5/0xc0 [41590.525853] ipmi_msghandler ioatdma [41590.531160] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41590.533776] acpi_pad acpi_power_meter [41590.538649] try_get_cap_refs+0x159/0x880 [ceph] [41590.541957] ixgbe igb [41590.545792] __ceph_get_caps+0x4a1/0x760 [ceph] [41590.549360] mdio [41590.554146] ? __file_remove_privs+0x89/0x110 [41590.557887] crc32c_intel nvme [41590.562499] ? wake_bit_function+0x70/0x70 [41590.564854] ptp nvme_core [41590.569382] ? ceph_write_iter+0x260/0x770 [ceph] [41590.571303] i2c_algo_bit pps_core [41590.575655] ceph_write_iter+0x260/0x770 [ceph] [41590.578705] dca [41590.582801] ? lock_is_held_type+0xaa/0x120 [41590.585501] fuse [41590.590199] vfs_write+0x324/0x4f0 [41590.593596] irq event stamp: 252717656 [41590.598122] ksys_pwrite64+0x65/0xa0 [41590.599956] hardirqs last enabled at (252717655): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41590.604133] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41590.606057] hardirqs last disabled at (252717656): [] sysvec_apic_timer_interrupt+0xe/0x90 [41590.609456] do_syscall_64+0x38/0x80 [41590.613200] softirqs last enabled at (252717654): [] __do_softirq+0x2be/0x371 [41590.616771] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41590.627015] softirqs last disabled at (252717649): [] irq_exit_rcu+0xb0/0xd0 [41590.632145] RIP: 0033:0x7f48f0eeb93b [41590.641955] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41590.645528] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41590.654296] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41590.659342] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41590.667939] Workqueue: events delayed_work [ceph] [41590.671510] ORIG_RAX: 0000000000000012 [41590.680885] [41590.680886] RIP: 0010:lock_acquire+0xf0/0x2b0 [41590.699624] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41590.707096] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41590.712313] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41590.717009] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41590.720841] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41590.722331] [41590.722332] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41590.726682] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41590.733807] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41590.752544] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41590.759669] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41590.764891] [41590.772010] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41590.818439] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41590.825562] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41590.833640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41590.839377] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41590.846502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41590.853625] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41590.860760] Call Trace: [41590.863204] [41590.865214] ? watchdog_timer_fn+0x1b4/0x220 [41590.869478] ? __hrtimer_run_queues+0x1d3/0x360 [41590.874005] ? hrtimer_interrupt+0x104/0x240 [41590.878273] ? __local_bh_enable+0x50/0x60 [41590.882369] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41590.887593] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41590.892548] [41590.894645] [41590.896762] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41590.902068] ? lock_acquire+0xf0/0x2b0 [41590.905818] ? igrab+0x43/0x50 [41590.908868] ? lock_release+0xc5/0x260 [41590.912615] _raw_spin_lock+0x30/0x40 [41590.916280] ? igrab+0x1d/0x50 [41590.919330] igrab+0x1d/0x50 [41590.922206] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41590.927356] delayed_work+0x1d8/0x280 [ceph] [41590.931637] ? process_scheduled_works+0x22a/0x510 [41590.936420] process_scheduled_works+0x2a4/0x510 [41590.941045] worker_thread+0x188/0x340 [41590.944794] ? rescuer_thread+0x310/0x310 [41590.948806] kthread+0xf5/0x130 [41590.951943] ? kthread_complete_and_exit+0x20/0x20 [41590.956726] ret_from_fork+0x31/0x40 [41590.960296] ? kthread_complete_and_exit+0x20/0x20 [41590.965081] ret_from_fork_asm+0x11/0x20 [41590.969004] [41618.180846] watchdog: BUG: soft lockup - CPU#0 stuck for 27645s! [dbench:78683] [41618.188148] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41618.192845] watchdog: BUG: soft lockup - CPU#7 stuck for 36642s! [kworker/7:4:78808] [41618.243596] irq event stamp: 74507100 [41618.251327] Modules linked in: ceph libceph dns_resolver fscache netfs veth [41618.254983] hardirqs last enabled at (74507099): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41618.261944] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [41618.272102] hardirqs last disabled at (74507100): [] sysvec_apic_timer_interrupt+0xe/0x90 [41618.279051] nf_defrag_ipv4 nft_compat nf_tables nfnetlink [41618.288775] softirqs last enabled at (74507042): [] __do_softirq+0x2be/0x371 [41618.294253] bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr [41618.302938] softirqs last disabled at (74507037): [] irq_exit_rcu+0xb0/0xd0 [41618.308857] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [41618.317368] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41618.317369] kvm_intel [41618.317370] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41618.323711] kvm irqbypass [41618.332656] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41618.335014] i2c_i801 [41618.342484] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41618.345189] iTCO_wdt [41618.350664] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41618.352937] iTCO_vendor_support i2c_smbus [41618.371673] [41618.373943] crct10dif_pclmul joydev [41618.379162] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41618.383252] crc32_pclmul ipmi_ssif [41618.384742] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41618.388314] ghash_clmulni_intel [41618.395437] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41618.398922] mei_me mei [41618.406046] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41618.409271] lpc_ich sha512_ssse3 [41618.416394] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41618.418839] acpi_ipmi [41618.425963] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41618.429274] mfd_core [41618.436397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41618.438754] wmi ipmi_si [41618.446831] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41618.449104] ipmi_devintf ipmi_msghandler [41618.454841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41618.457372] ioatdma [41618.464496] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41618.468499] acpi_pad acpi_power_meter [41618.475624] Call Trace: [41618.475626] [41618.477808] ixgbe [41618.484933] ? watchdog_timer_fn+0x1b4/0x220 [41618.488676] igb mdio [41618.491122] ? __hrtimer_run_queues+0x1d3/0x360 [41618.493130] crc32c_intel nvme [41618.495144] ? hrtimer_interrupt+0x104/0x240 [41618.499405] ptp nvme_core [41618.501677] ? __local_bh_enable+0x50/0x60 [41618.506200] i2c_algo_bit pps_core [41618.509253] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41618.513523] dca fuse [41618.516246] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41618.520336] [41618.520336] irq event stamp: 252938444 [41618.523734] [41618.528951] hardirqs last enabled at (252938443): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41618.531221] [41618.536197] hardirqs last disabled at (252938444): [] sysvec_apic_timer_interrupt+0xe/0x90 [41618.537689] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41618.541432] softirqs last enabled at (252938442): [] __do_softirq+0x2be/0x371 [41618.543531] ? queued_spin_lock_slowpath+0x16/0x2e0 [41618.553772] softirqs last disabled at (252938437): [] irq_exit_rcu+0xb0/0xd0 [41618.555871] do_raw_spin_lock+0xb5/0xc0 [41618.565680] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41618.570987] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41618.579756] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41618.584629] try_get_cap_refs+0x159/0x880 [ceph] [41618.593223] Workqueue: events delayed_work [ceph] [41618.597077] __ceph_get_caps+0x4a1/0x760 [ceph] [41618.606449] [41618.606450] RIP: 0010:lock_acquire+0xf0/0x2b0 [41618.611250] ? __file_remove_privs+0x89/0x110 [41618.618712] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41618.623324] ? wake_bit_function+0x70/0x70 [41618.628020] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41618.632552] ? ceph_write_iter+0x260/0x770 [ceph] [41618.634036] [41618.634036] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41618.638387] ceph_write_iter+0x260/0x770 [ceph] [41618.642737] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41618.661478] ? lock_is_held_type+0xaa/0x120 [41618.665565] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41618.670786] vfs_write+0x324/0x4f0 [41618.675479] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41618.676977] ksys_pwrite64+0x65/0xa0 [41618.684095] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41618.688622] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41618.695745] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41618.699924] do_syscall_64+0x38/0x80 [41618.707045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41618.710445] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41618.717566] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41618.721138] RIP: 0033:0x7f48f0eeb93b [41618.728261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41618.733393] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41618.741470] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41618.745041] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41618.750779] Call Trace: [41618.750780] [41618.755822] ORIG_RAX: 0000000000000012 [41618.762947] ? watchdog_timer_fn+0x1b4/0x220 [41618.766529] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41618.773652] ? __hrtimer_run_queues+0x1d3/0x360 [41618.792387] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41618.799532] ? hrtimer_interrupt+0x104/0x240 [41618.804747] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41618.807191] ? __local_bh_enable+0x50/0x60 [41618.809201] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41618.813036] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41618.817296] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41618.824420] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41618.828950] [41618.836070] [41618.880252] [41618.882350] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41618.887658] ? lock_acquire+0xf0/0x2b0 [41618.891409] ? igrab+0x43/0x50 [41618.894457] ? lock_release+0xc5/0x260 [41618.898203] _raw_spin_lock+0x30/0x40 [41618.901859] ? igrab+0x1d/0x50 [41618.904909] igrab+0x1d/0x50 [41618.907789] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41618.912937] delayed_work+0x1d8/0x280 [ceph] [41618.917219] ? process_scheduled_works+0x22a/0x510 [41618.922009] process_scheduled_works+0x2a4/0x510 [41618.926624] worker_thread+0x188/0x340 [41618.930375] ? rescuer_thread+0x310/0x310 [41618.934386] kthread+0xf5/0x130 [41618.937547] ? kthread_complete_and_exit+0x20/0x20 [41618.942334] ret_from_fork+0x31/0x40 [41618.945914] ? kthread_complete_and_exit+0x20/0x20 [41618.950697] ret_from_fork_asm+0x11/0x20 [41618.954620] [41646.181335] watchdog: BUG: soft lockup - CPU#0 stuck for 27672s! [dbench:78683] [41646.188634] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41646.193335] watchdog: BUG: soft lockup - CPU#7 stuck for 36669s! [kworker/7:4:78808] [41646.244075] irq event stamp: 74565202 [41646.244077] hardirqs last enabled at (74565201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41646.251806] Modules linked in: [41646.255461] hardirqs last disabled at (74565202): [] sysvec_apic_timer_interrupt+0xe/0x90 [41646.265621] ceph [41646.268670] softirqs last enabled at (74565144): [] __do_softirq+0x2be/0x371 [41646.278395] libceph [41646.280319] softirqs last disabled at (74565139): [] irq_exit_rcu+0xb0/0xd0 [41646.289010] dns_resolver fscache [41646.291196] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41646.299707] netfs veth [41646.303035] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41646.311979] nft_chain_nat xt_MASQUERADE [41646.314423] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41646.321893] nf_nat nf_conntrack [41646.325812] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41646.331289] nf_defrag_ipv6 nf_defrag_ipv4 [41646.334513] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41646.353251] nft_compat nf_tables [41646.357342] [41646.357343] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41646.362559] nfnetlink bridge [41646.365870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41646.367361] stp llc [41646.374485] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41646.377449] xfs libcrc32c [41646.384571] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41646.386756] sunrpc [41646.393880] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41646.396585] intel_rapl_msr intel_rapl_common [41646.403708] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41646.405823] sb_edac x86_pkg_temp_thermal [41646.412948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41646.417299] coretemp kvm_intel [41646.425376] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41646.429379] kvm irqbypass [41646.435116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41646.438255] i2c_i801 iTCO_wdt [41646.445378] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41646.448082] iTCO_vendor_support i2c_smbus [41646.455208] Call Trace: [41646.458257] crct10dif_pclmul [41646.465381] [41646.469472] joydev crc32_pclmul [41646.471918] ? watchdog_timer_fn+0x1b4/0x220 [41646.474880] ipmi_ssif ghash_clmulni_intel [41646.476893] ? __hrtimer_run_queues+0x1d3/0x360 [41646.480114] mei_me mei [41646.484384] ? hrtimer_interrupt+0x104/0x240 [41646.488471] lpc_ich sha512_ssse3 [41646.492993] ? __local_bh_enable+0x50/0x60 [41646.495438] acpi_ipmi [41646.499704] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41646.503013] mfd_core wmi [41646.507122] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41646.509478] ipmi_si ipmi_devintf [41646.514695] [41646.517312] ipmi_msghandler ioatdma [41646.522272] [41646.525581] acpi_pad [41646.527680] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41646.531249] acpi_power_meter ixgbe [41646.533350] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41646.535617] igb mdio [41646.540923] do_raw_spin_lock+0xb5/0xc0 [41646.544407] crc32c_intel nvme [41646.549278] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41646.551548] ptp nvme_core [41646.555379] try_get_cap_refs+0x159/0x880 [ceph] [41646.558430] i2c_algo_bit pps_core [41646.563217] __ceph_get_caps+0x4a1/0x760 [ceph] [41646.565917] dca fuse [41646.570532] ? __file_remove_privs+0x89/0x110 [41646.573925] [41646.573926] irq event stamp: 253159352 [41646.578450] ? wake_bit_function+0x70/0x70 [41646.580734] hardirqs last enabled at (253159351): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41646.585091] ? ceph_write_iter+0x260/0x770 [ceph] [41646.586579] hardirqs last disabled at (253159352): [] sysvec_apic_timer_interrupt+0xe/0x90 [41646.590324] ceph_write_iter+0x260/0x770 [ceph] [41646.594414] softirqs last enabled at (253159350): [] __do_softirq+0x2be/0x371 [41646.604661] ? lock_is_held_type+0xaa/0x120 [41646.609355] softirqs last disabled at (253159345): [] irq_exit_rcu+0xb0/0xd0 [41646.619168] vfs_write+0x324/0x4f0 [41646.623690] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41646.632465] ksys_pwrite64+0x65/0xa0 [41646.636639] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41646.645235] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41646.648633] Workqueue: events delayed_work [ceph] [41646.658012] do_syscall_64+0x38/0x80 [41646.661599] [41646.669070] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41646.674201] RIP: 0010:lock_release+0x1d2/0x260 [41646.678897] RIP: 0033:0x7f48f0eeb93b [41646.682467] Code: 65 ff 0d 19 e9 eb 7e 0f 85 76 fe ff ff 0f 1f 44 00 00 e9 6c fe ff ff 4c 89 e2 48 89 ee 48 89 df e8 13 b4 ff ff e9 57 ff ff ff 49 8d 02 00 84 c0 0f 85 63 fe ff ff 0f 0b e9 5c fe ff ff 44 2b [41646.683959] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41646.689015] RSP: 0018:ffffc90002117d48 EFLAGS: 00000247 [41646.689017] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [41646.693459] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41646.697028] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [41646.715765] ORIG_RAX: 0000000000000012 [41646.734504] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [41646.739737] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41646.746862] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [41646.752080] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41646.759203] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41646.763045] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41646.770167] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41646.777293] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41646.784416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41646.791541] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41646.798663] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41646.805795] [41646.813866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41646.850302] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41646.857427] Call Trace: [41646.859872] [41646.861882] ? watchdog_timer_fn+0x1b4/0x220 [41646.866146] ? __hrtimer_run_queues+0x1d3/0x360 [41646.870673] ? hrtimer_interrupt+0x104/0x240 [41646.874943] ? __local_bh_enable+0x50/0x60 [41646.879036] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41646.884260] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41646.889217] [41646.891313] [41646.893412] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41646.898716] ? igrab+0x43/0x50 [41646.901786] ? igrab+0x43/0x50 [41646.904842] ? lock_release+0x1d2/0x260 [41646.908673] ? lock_release+0xc5/0x260 [41646.912418] _raw_spin_unlock+0x1b/0x40 [41646.916250] igrab+0x43/0x50 [41646.919125] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41646.924266] delayed_work+0x1d8/0x280 [ceph] [41646.928549] ? process_scheduled_works+0x22a/0x510 [41646.933338] process_scheduled_works+0x2a4/0x510 [41646.937955] worker_thread+0x188/0x340 [41646.941702] ? rescuer_thread+0x310/0x310 [41646.945707] kthread+0xf5/0x130 [41646.948845] ? kthread_complete_and_exit+0x20/0x20 [41646.953628] ret_from_fork+0x31/0x40 [41646.957200] ? kthread_complete_and_exit+0x20/0x20 [41646.961984] ret_from_fork_asm+0x11/0x20 [41646.965915] [41674.181823] watchdog: BUG: soft lockup - CPU#0 stuck for 27698s! [dbench:78683] [41674.189120] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41674.193823] watchdog: BUG: soft lockup - CPU#7 stuck for 36695s! [kworker/7:4:78808] [41674.244561] irq event stamp: 74623356 [41674.244563] hardirqs last enabled at (74623355): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41674.252292] Modules linked in: [41674.255950] hardirqs last disabled at (74623356): [] sysvec_apic_timer_interrupt+0xe/0x90 [41674.266108] ceph [41674.269159] softirqs last enabled at (74623320): [] __do_softirq+0x2be/0x371 [41674.278882] libceph dns_resolver [41674.280806] softirqs last disabled at (74623315): [] irq_exit_rcu+0xb0/0xd0 [41674.289498] fscache netfs [41674.292811] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41674.301320] veth nft_chain_nat [41674.304024] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41674.304026] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41674.312969] xt_MASQUERADE nf_nat [41674.316107] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41674.323578] nf_conntrack nf_defrag_ipv6 [41674.329056] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41674.332366] nf_defrag_ipv4 nft_compat [41674.351103] [41674.351103] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41674.355020] nf_tables nfnetlink [41674.360237] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41674.363981] bridge stp [41674.365474] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41674.372596] llc xfs [41674.375821] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41674.382946] libcrc32c sunrpc [41674.385388] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41674.392512] intel_rapl_msr intel_rapl_common [41674.394698] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41674.401820] sb_edac x86_pkg_temp_thermal [41674.404784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41674.411909] coretemp [41674.416259] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41674.424336] kvm_intel kvm [41674.428342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41674.434079] irqbypass i2c_i801 [41674.436349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41674.443475] iTCO_wdt iTCO_vendor_support [41674.446178] Call Trace: [41674.446180] [41674.453302] i2c_smbus crct10dif_pclmul [41674.456439] ? watchdog_timer_fn+0x1b4/0x220 [41674.463565] joydev crc32_pclmul [41674.467569] ? __hrtimer_run_queues+0x1d3/0x360 [41674.470011] ipmi_ssif ghash_clmulni_intel [41674.472026] ? hrtimer_interrupt+0x104/0x240 [41674.475854] mei_me mei [41674.480117] ? __local_bh_enable+0x50/0x60 [41674.483341] lpc_ich sha512_ssse3 [41674.487868] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41674.491957] acpi_ipmi mfd_core [41674.496221] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41674.498664] wmi ipmi_si [41674.502756] [41674.506066] ipmi_devintf [41674.511284] [41674.514421] ipmi_msghandler ioatdma [41674.519380] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41674.521908] acpi_pad acpi_power_meter [41674.524010] ? queued_spin_lock_slowpath+0x18/0x2e0 [41674.526624] ixgbe igb [41674.528724] do_raw_spin_lock+0xb5/0xc0 [41674.532292] mdio crc32c_intel [41674.537598] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41674.541341] nvme ptp nvme_core [41674.546225] try_get_cap_refs+0x159/0x880 [ceph] [41674.548577] i2c_algo_bit pps_core [41674.552413] __ceph_get_caps+0x4a1/0x760 [ceph] [41674.555459] dca fuse [41674.560248] ? __file_remove_privs+0x89/0x110 [41674.563379] [41674.563380] irq event stamp: 253380230 [41674.567991] ? wake_bit_function+0x70/0x70 [41674.571388] hardirqs last enabled at (253380229): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41674.575916] ? ceph_write_iter+0x260/0x770 [ceph] [41674.578182] hardirqs last disabled at (253380230): [] sysvec_apic_timer_interrupt+0xe/0x90 [41674.582533] ceph_write_iter+0x260/0x770 [ceph] [41674.584025] softirqs last enabled at (253380228): [] __do_softirq+0x2be/0x371 [41674.587773] ? lock_is_held_type+0xaa/0x120 [41674.591860] softirqs last disabled at (253380223): [] irq_exit_rcu+0xb0/0xd0 [41674.602107] vfs_write+0x324/0x4f0 [41674.606800] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41674.616616] ksys_pwrite64+0x65/0xa0 [41674.621136] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41674.629906] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41674.634085] Workqueue: events delayed_work [ceph] [41674.642683] do_syscall_64+0x38/0x80 [41674.646078] [41674.655458] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41674.659027] RIP: 0010:lock_acquire+0xf0/0x2b0 [41674.666515] RIP: 0033:0x7f48f0eeb93b [41674.671646] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41674.676344] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41674.679915] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41674.681404] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41674.686450] [41674.686451] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41674.690798] ORIG_RAX: 0000000000000012 [41674.694369] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41674.713107] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41674.731847] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41674.737064] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41674.742281] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41674.743770] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41674.750895] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41674.754725] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41674.761851] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41674.768975] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41674.776100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41674.783232] [41674.790347] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41674.841967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41674.849090] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41674.856227] Call Trace: [41674.858668] [41674.860680] ? watchdog_timer_fn+0x1b4/0x220 [41674.864943] ? __hrtimer_run_queues+0x1d3/0x360 [41674.869469] ? hrtimer_interrupt+0x104/0x240 [41674.873730] ? __local_bh_enable+0x50/0x60 [41674.877823] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41674.883039] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41674.887996] [41674.890094] [41674.892192] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41674.897516] ? lock_acquire+0xf0/0x2b0 [41674.901267] ? igrab+0x43/0x50 [41674.904317] ? lock_release+0xc5/0x260 [41674.908061] _raw_spin_lock+0x30/0x40 [41674.911717] ? igrab+0x1d/0x50 [41674.914769] igrab+0x1d/0x50 [41674.917647] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41674.922795] delayed_work+0x1d8/0x280 [ceph] [41674.927078] ? process_scheduled_works+0x22a/0x510 [41674.931868] process_scheduled_works+0x2a4/0x510 [41674.936482] worker_thread+0x188/0x340 [41674.940231] ? rescuer_thread+0x310/0x310 [41674.944235] kthread+0xf5/0x130 [41674.947374] ? kthread_complete_and_exit+0x20/0x20 [41674.952157] ret_from_fork+0x31/0x40 [41674.955726] ? kthread_complete_and_exit+0x20/0x20 [41674.960520] ret_from_fork_asm+0x11/0x20 [41674.964444] [41702.182312] watchdog: BUG: soft lockup - CPU#0 stuck for 27724s! [dbench:78683] [41702.189618] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41702.194311] watchdog: BUG: soft lockup - CPU#7 stuck for 36721s! [kworker/7:4:78808] [41702.245067] irq event stamp: 74681516 [41702.245068] hardirqs last enabled at (74681515): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41702.252798] Modules linked in: [41702.256456] hardirqs last disabled at (74681516): [] sysvec_apic_timer_interrupt+0xe/0x90 [41702.266613] ceph [41702.269664] softirqs last enabled at (74681458): [] __do_softirq+0x2be/0x371 [41702.279387] libceph [41702.281312] softirqs last disabled at (74681453): [] irq_exit_rcu+0xb0/0xd0 [41702.289995] dns_resolver fscache [41702.292180] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41702.300705] netfs veth [41702.304019] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41702.312963] nft_chain_nat xt_MASQUERADE [41702.315426] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [41702.322896] nf_nat [41702.326813] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [41702.332291] nf_conntrack nf_defrag_ipv6 [41702.334388] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41702.353125] nf_defrag_ipv4 nft_compat [41702.357042] [41702.362259] nf_tables nfnetlink [41702.366003] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41702.367493] bridge [41702.370717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41702.377843] stp [41702.379939] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41702.387064] llc xfs [41702.388900] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41702.396026] libcrc32c sunrpc [41702.398209] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41702.405334] intel_rapl_msr intel_rapl_common [41702.408298] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41702.415422] sb_edac x86_pkg_temp_thermal [41702.419773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41702.427850] coretemp kvm_intel [41702.431855] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41702.437593] kvm irqbypass [41702.440728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41702.447852] i2c_i801 [41702.450557] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41702.457680] iTCO_wdt iTCO_vendor_support [41702.459953] Call Trace: [41702.459955] [41702.467078] i2c_smbus [41702.471082] ? watchdog_timer_fn+0x1b4/0x220 [41702.473524] crct10dif_pclmul joydev [41702.475537] ? __hrtimer_run_queues+0x1d3/0x360 [41702.477892] crc32_pclmul ipmi_ssif [41702.482160] ? hrtimer_interrupt+0x104/0x240 [41702.485726] ghash_clmulni_intel [41702.490252] ? __local_bh_enable+0x50/0x60 [41702.493736] mei_me [41702.498002] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41702.501224] mei lpc_ich [41702.505315] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41702.507410] sha512_ssse3 acpi_ipmi [41702.512628] [41702.515159] mfd_core wmi [41702.520116] [41702.523602] ipmi_si ipmi_devintf [41702.525712] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41702.528324] ipmi_msghandler ioatdma [41702.530425] ? queued_spin_lock_slowpath+0x2b/0x2e0 [41702.533733] acpi_pad acpi_power_meter [41702.539038] do_raw_spin_lock+0xb5/0xc0 [41702.542607] ixgbe igb [41702.547480] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41702.551222] mdio crc32c_intel [41702.555056] try_get_cap_refs+0x159/0x880 [ceph] [41702.557411] nvme ptp [41702.562199] __ceph_get_caps+0x4a1/0x760 [ceph] [41702.565246] nvme_core [41702.569862] ? __file_remove_privs+0x89/0x110 [41702.572128] i2c_algo_bit pps_core [41702.576651] ? wake_bit_function+0x70/0x70 [41702.579008] dca [41702.583361] ? ceph_write_iter+0x260/0x770 [ceph] [41702.586756] fuse [41702.590846] ceph_write_iter+0x260/0x770 [ceph] [41702.592685] irq event stamp: 253600990 [41702.597402] ? lock_is_held_type+0xaa/0x120 [41702.599324] hardirqs last enabled at (253600989): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41702.603851] vfs_write+0x324/0x4f0 [41702.607592] hardirqs last disabled at (253600990): [] sysvec_apic_timer_interrupt+0xe/0x90 [41702.611775] ksys_pwrite64+0x65/0xa0 [41702.622013] softirqs last enabled at (253600988): [] __do_softirq+0x2be/0x371 [41702.625412] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41702.635222] softirqs last disabled at (253600983): [] irq_exit_rcu+0xb0/0xd0 [41702.638794] do_syscall_64+0x38/0x80 [41702.647562] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41702.652712] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41702.661308] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41702.664880] RIP: 0033:0x7f48f0eeb93b [41702.674258] Workqueue: events delayed_work [ceph] [41702.679302] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41702.686771] [41702.686772] RIP: 0010:lock_release+0x8/0x260 [41702.690343] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41702.695040] Code: 82 48 c7 c7 8a 3f 12 82 e8 35 96 f7 ff 0f 0b e9 e2 fe ff ff e8 79 83 a2 00 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 56 41 55 <41> 54 49 89 f4 55 48 89 fd 53 48 83 ec 18 65 48 8b 04 25 28 00 00 [41702.713778] ORIG_RAX: 0000000000000012 [41702.715267] RSP: 0018:ffffc90002117d78 EFLAGS: 00000286 [41702.719532] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41702.724750] [41702.724751] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 000000005553a634 [41702.743487] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41702.747318] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [41702.752535] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41702.759658] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [41702.761151] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41702.768274] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [41702.775399] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41702.782522] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41702.789651] [41702.796772] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41702.835530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41702.841266] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41702.848391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41702.855515] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41702.862639] Call Trace: [41702.865084] [41702.867095] ? watchdog_timer_fn+0x1b4/0x220 [41702.871360] ? __hrtimer_run_queues+0x1d3/0x360 [41702.875885] ? hrtimer_interrupt+0x104/0x240 [41702.880154] ? __local_bh_enable+0x50/0x60 [41702.884248] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41702.889471] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41702.894429] [41702.896525] [41702.898623] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41702.903930] ? igrab+0x43/0x50 [41702.906994] ? lock_release+0x8/0x260 [41702.910654] _raw_spin_unlock+0x1b/0x40 [41702.914485] igrab+0x43/0x50 [41702.917363] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41702.922512] delayed_work+0x1d8/0x280 [ceph] [41702.926794] ? process_scheduled_works+0x22a/0x510 [41702.931585] process_scheduled_works+0x2a4/0x510 [41702.936198] worker_thread+0x188/0x340 [41702.939949] ? rescuer_thread+0x310/0x310 [41702.943951] kthread+0xf5/0x130 [41702.947091] ? kthread_complete_and_exit+0x20/0x20 [41702.951882] ret_from_fork+0x31/0x40 [41702.955463] ? kthread_complete_and_exit+0x20/0x20 [41702.960245] ret_from_fork_asm+0x11/0x20 [41702.964168] [41704.511718] rcu: INFO: rcu_preempt self-detected stall on CPU [41704.517476] rcu: 0-....: (29093763 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14574453 [41704.527989] rcu: (t=29767383 jiffies g=1309609 q=790479 ncpus=8) [41704.534072] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41704.543016] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41704.550487] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41704.555962] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41704.574702] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41704.579918] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41704.587042] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41704.594167] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41704.601291] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41704.608415] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41704.615538] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41704.623617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41704.629354] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41704.636478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41704.643603] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41704.650728] Call Trace: [41704.653170] [41704.655183] ? rcu_dump_cpu_stacks+0xd1/0x180 [41704.659543] ? rcu_sched_clock_irq+0x633/0x1390 [41704.664074] ? lock_is_held_type+0xaa/0x120 [41704.668252] ? lock_is_held_type+0xaa/0x120 [41704.672429] ? tick_sched_do_timer+0xa0/0xa0 [41704.676701] ? rcu_read_lock_sched_held+0x56/0x90 [41704.681399] ? tick_sched_do_timer+0xa0/0xa0 [41704.685663] ? update_process_times+0x5f/0x90 [41704.690013] ? tick_sched_handle.isra.16+0x1e/0x50 [41704.694797] ? tick_sched_timer+0x6f/0x80 [41704.698803] ? __hrtimer_run_queues+0x1d3/0x360 [41704.703330] ? hrtimer_interrupt+0x104/0x240 [41704.707597] ? __local_bh_enable+0x50/0x60 [41704.711692] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41704.716917] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41704.721873] [41704.723970] [41704.726067] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41704.731376] ? queued_spin_lock_slowpath+0x18/0x2e0 [41704.736252] do_raw_spin_lock+0xb5/0xc0 [41704.740082] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41704.744886] try_get_cap_refs+0x159/0x880 [ceph] [41704.749515] __ceph_get_caps+0x4a1/0x760 [ceph] [41704.754056] ? __file_remove_privs+0x89/0x110 [41704.758412] ? wake_bit_function+0x70/0x70 [41704.762504] ? ceph_write_iter+0x260/0x770 [ceph] [41704.767218] ceph_write_iter+0x260/0x770 [ceph] [41704.771754] ? lock_is_held_type+0xaa/0x120 [41704.775938] vfs_write+0x324/0x4f0 [41704.779347] ksys_pwrite64+0x65/0xa0 [41704.782922] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41704.788054] do_syscall_64+0x38/0x80 [41704.791625] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41704.796666] RIP: 0033:0x7f48f0eeb93b [41704.800237] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41704.818975] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [41704.826532] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41704.833658] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41704.840782] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41704.847907] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41704.855029] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41704.862162] [41704.864348] Sending NMI from CPU 0 to CPUs 7: [41704.868699] NMI backtrace for cpu 7 [41704.868701] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41704.868703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41704.868704] Workqueue: events delayed_work [ceph] [41704.868719] RIP: 0010:find_held_lock+0x2d/0x90 [41704.868723] Code: 89 d7 41 56 41 55 49 89 cd 41 54 49 89 f4 55 8d 6a ff 48 63 c5 53 48 8d 04 80 4c 8d b4 c7 58 0a 00 00 4c 89 f7 e8 73 ee a2 00 <85> c0 75 48 41 83 ef 02 49 8d 5e d8 44 89 fd 78 4d 41 0f b6 46 21 [41704.868725] RSP: 0018:ffffc90002117d10 EFLAGS: 00000046 [41704.868727] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [41704.868728] RDX: 0000000000000004 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [41704.868729] RBP: 0000000000000003 R08: 0000000000005c53 R09: 0000000000000003 [41704.868729] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41704.868730] R13: ffffc90002117d54 R14: ffff88810debb550 R15: 0000000000000004 [41704.868731] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41704.868732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41704.868732] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41704.868733] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41704.868734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41704.868735] Call Trace: [41704.868736] [41704.868736] ? nmi_cpu_backtrace+0xad/0x120 [41704.868740] ? nmi_cpu_backtrace_handler+0x11/0x20 [41704.868741] ? nmi_handle+0xd4/0x260 [41704.868745] ? default_do_nmi+0x49/0xf0 [41704.868747] ? exc_nmi+0x11e/0x150 [41704.868748] ? end_repeat_nmi+0x16/0x67 [41704.868752] ? find_held_lock+0x2d/0x90 [41704.868754] ? find_held_lock+0x2d/0x90 [41704.868757] ? find_held_lock+0x2d/0x90 [41704.868759] [41704.868760] [41704.868760] ? igrab+0x43/0x50 [41704.868762] lock_release+0xc5/0x260 [41704.868764] _raw_spin_unlock+0x1b/0x40 [41704.868766] igrab+0x43/0x50 [41704.868767] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41704.868781] delayed_work+0x1d8/0x280 [ceph] [41704.868794] ? process_scheduled_works+0x22a/0x510 [41704.868797] process_scheduled_works+0x2a4/0x510 [41704.868800] worker_thread+0x188/0x340 [41704.868803] ? rescuer_thread+0x310/0x310 [41704.868805] kthread+0xf5/0x130 [41704.868807] ? kthread_complete_and_exit+0x20/0x20 [41704.868810] ret_from_fork+0x31/0x40 [41704.868812] ? kthread_complete_and_exit+0x20/0x20 [41704.868814] ret_from_fork_asm+0x11/0x20 [41704.868817] [41730.182796] watchdog: BUG: soft lockup - CPU#0 stuck for 27750s! [dbench:78683] [41730.190096] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41730.194796] watchdog: BUG: soft lockup - CPU#7 stuck for 36747s! [kworker/7:4:78808] [41730.245537] irq event stamp: 74738950 [41730.245538] hardirqs last enabled at (74738949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41730.253267] Modules linked in: [41730.256924] hardirqs last disabled at (74738950): [] sysvec_apic_timer_interrupt+0xe/0x90 [41730.267081] ceph [41730.270132] softirqs last enabled at (74738892): [] __do_softirq+0x2be/0x371 [41730.279858] libceph [41730.281782] softirqs last disabled at (74738887): [] irq_exit_rcu+0xb0/0xd0 [41730.290479] dns_resolver [41730.292666] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41730.301191] fscache netfs [41730.303812] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41730.312757] veth [41730.315471] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41730.322941] nft_chain_nat [41730.324864] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41730.330343] xt_MASQUERADE nf_nat [41730.333047] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41730.351785] nf_conntrack nf_defrag_ipv6 [41730.355094] [41730.355095] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41730.360311] nf_defrag_ipv4 nft_compat [41730.364229] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41730.365720] nf_tables nfnetlink [41730.372844] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41730.376588] bridge [41730.383711] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41730.386937] stp llc [41730.394061] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41730.396158] xfs [41730.403281] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41730.405473] libcrc32c sunrpc [41730.412600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41730.414436] intel_rapl_msr intel_rapl_common [41730.422513] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41730.425479] sb_edac x86_pkg_temp_thermal [41730.431215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41730.435566] coretemp [41730.442689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41730.446694] kvm_intel [41730.453819] Call Trace: [41730.453820] [41730.456088] kvm irqbypass [41730.463213] ? watchdog_timer_fn+0x1b4/0x220 [41730.465569] i2c_i801 iTCO_wdt [41730.468014] ? __hrtimer_run_queues+0x1d3/0x360 [41730.470026] iTCO_vendor_support i2c_smbus [41730.472733] ? hrtimer_interrupt+0x104/0x240 [41730.476994] crct10dif_pclmul joydev [41730.480044] ? __local_bh_enable+0x50/0x60 [41730.484567] crc32_pclmul ipmi_ssif [41730.488662] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41730.492922] ghash_clmulni_intel [41730.496495] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41730.500585] mei_me mei [41730.504068] [41730.509285] lpc_ich sha512_ssse3 [41730.512511] [41730.512512] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41730.517474] acpi_ipmi mfd_core [41730.519926] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41730.522017] wmi ipmi_si [41730.525331] do_raw_spin_lock+0xb5/0xc0 [41730.527426] ipmi_devintf ipmi_msghandler [41730.532729] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41730.535866] ioatdma acpi_pad [41730.540739] try_get_cap_refs+0x159/0x880 [ceph] [41730.543267] acpi_power_meter ixgbe [41730.547105] __ceph_get_caps+0x4a1/0x760 [ceph] [41730.551104] igb mdio [41730.555892] ? __file_remove_privs+0x89/0x110 [41730.558852] crc32c_intel nvme [41730.563462] ? wake_bit_function+0x70/0x70 [41730.566964] ptp nvme_core [41730.571491] ? ceph_write_iter+0x260/0x770 [ceph] [41730.573759] i2c_algo_bit pps_core [41730.578109] ceph_write_iter+0x260/0x770 [ceph] [41730.581159] dca fuse [41730.585255] ? lock_is_held_type+0xaa/0x120 [41730.587954] [41730.587954] irq event stamp: 253820034 [41730.592654] vfs_write+0x324/0x4f0 [41730.596050] hardirqs last enabled at (253820033): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41730.600578] ksys_pwrite64+0x65/0xa0 [41730.602843] hardirqs last disabled at (253820034): [] sysvec_apic_timer_interrupt+0xe/0x90 [41730.607021] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41730.608511] softirqs last enabled at (253820032): [] __do_softirq+0x2be/0x371 [41730.612259] do_syscall_64+0x38/0x80 [41730.615655] softirqs last disabled at (253820027): [] irq_exit_rcu+0xb0/0xd0 [41730.625900] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41730.629468] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41730.639280] RIP: 0033:0x7f48f0eeb93b [41730.644410] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41730.644412] Workqueue: events delayed_work [ceph] [41730.653198] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41730.656768] [41730.656769] RIP: 0010:lock_release+0x162/0x260 [41730.665367] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41730.670411] Code: 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 48 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 <48> 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 65 8b 05 56 e9 eb 7e 89 c0 [41730.679788] ORIG_RAX: 0000000000000012 [41730.683358] RSP: 0018:ffffc90002117d48 EFLAGS: 00000246 [41730.690829] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41730.695526] [41730.695527] RAX: 0000000000000000 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [41730.714265] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41730.715756] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [41730.720200] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41730.725418] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [41730.744157] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41730.747987] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [41730.753205] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41730.760328] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [41730.761824] [41730.768943] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41730.836207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41730.841946] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41730.849070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41730.856203] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41730.863325] Call Trace: [41730.865769] [41730.867780] ? watchdog_timer_fn+0x1b4/0x220 [41730.872047] ? __hrtimer_run_queues+0x1d3/0x360 [41730.876573] ? hrtimer_interrupt+0x104/0x240 [41730.880843] ? __local_bh_enable+0x50/0x60 [41730.884935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41730.890160] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41730.895116] [41730.897213] [41730.899311] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41730.904615] ? igrab+0x43/0x50 [41730.907667] ? lock_release+0x162/0x260 [41730.911497] ? lock_release+0xc5/0x260 [41730.915243] _raw_spin_unlock+0x1b/0x40 [41730.919080] igrab+0x43/0x50 [41730.921958] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41730.927108] delayed_work+0x1d8/0x280 [ceph] [41730.931388] ? process_scheduled_works+0x22a/0x510 [41730.936172] process_scheduled_works+0x2a4/0x510 [41730.940804] worker_thread+0x188/0x340 [41730.944553] ? rescuer_thread+0x310/0x310 [41730.948556] kthread+0xf5/0x130 [41730.951693] ? kthread_complete_and_exit+0x20/0x20 [41730.956486] ret_from_fork+0x31/0x40 [41730.960056] ? kthread_complete_and_exit+0x20/0x20 [41730.964840] ret_from_fork_asm+0x11/0x20 [41730.968763] [41758.183278] watchdog: BUG: soft lockup - CPU#0 stuck for 27776s! [dbench:78683] [41758.190584] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41758.195277] watchdog: BUG: soft lockup - CPU#7 stuck for 36773s! [kworker/7:4:78808] [41758.246033] irq event stamp: 74797176 [41758.246034] hardirqs last enabled at (74797175): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41758.253764] Modules linked in: [41758.257421] hardirqs last disabled at (74797176): [] sysvec_apic_timer_interrupt+0xe/0x90 [41758.267579] ceph [41758.270629] softirqs last enabled at (74797118): [] __do_softirq+0x2be/0x371 [41758.280354] libceph [41758.282278] softirqs last disabled at (74797113): [] irq_exit_rcu+0xb0/0xd0 [41758.290962] dns_resolver [41758.293148] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41758.301672] fscache netfs [41758.304292] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41758.313235] veth nft_chain_nat [41758.315941] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41758.323427] xt_MASQUERADE nf_nat [41758.326566] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41758.332042] nf_conntrack [41758.335354] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41758.354091] nf_defrag_ipv6 nf_defrag_ipv4 [41758.356707] [41758.361927] nft_compat nf_tables [41758.366035] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41758.367524] nfnetlink bridge [41758.370836] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41758.377959] stp [41758.380924] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41758.388049] llc xfs [41758.389885] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41758.397008] libcrc32c [41758.399193] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41758.406317] sunrpc [41758.408675] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41758.415800] intel_rapl_msr [41758.417895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41758.425975] intel_rapl_common sb_edac [41758.428765] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41758.434502] x86_pkg_temp_thermal coretemp [41758.438245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41758.445371] kvm_intel kvm [41758.449460] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41758.456584] irqbypass i2c_i801 [41758.459290] Call Trace: [41758.459293] [41758.466414] iTCO_wdt iTCO_vendor_support [41758.469553] ? watchdog_timer_fn+0x1b4/0x220 [41758.471996] i2c_smbus crct10dif_pclmul [41758.474007] ? __hrtimer_run_queues+0x1d3/0x360 [41758.478009] joydev crc32_pclmul [41758.482276] ? hrtimer_interrupt+0x104/0x240 [41758.486104] ipmi_ssif ghash_clmulni_intel [41758.490627] ? __local_bh_enable+0x50/0x60 [41758.493852] mei_me mei [41758.498118] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41758.502207] lpc_ich sha512_ssse3 [41758.506299] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41758.508742] acpi_ipmi mfd_core [41758.513965] [41758.517279] wmi ipmi_si [41758.522236] [41758.525374] ipmi_devintf [41758.527471] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41758.530001] ipmi_msghandler [41758.532103] ? queued_spin_lock_slowpath+0x18/0x2e0 [41758.534717] ioatdma acpi_pad [41758.540042] do_raw_spin_lock+0xb5/0xc0 [41758.542915] acpi_power_meter ixgbe igb [41758.547788] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41758.550750] mdio crc32c_intel [41758.554585] try_get_cap_refs+0x159/0x880 [ceph] [41758.558411] nvme ptp [41758.563200] __ceph_get_caps+0x4a1/0x760 [ceph] [41758.566246] nvme_core i2c_algo_bit [41758.570861] ? __file_remove_privs+0x89/0x110 [41758.573129] pps_core dca [41758.577653] ? wake_bit_function+0x70/0x70 [41758.581152] fuse [41758.585507] ? ceph_write_iter+0x260/0x770 [ceph] [41758.588122] irq event stamp: 254040824 [41758.592213] ceph_write_iter+0x260/0x770 [ceph] [41758.594136] hardirqs last enabled at (254040823): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41758.598838] ? lock_is_held_type+0xaa/0x120 [41758.602578] hardirqs last disabled at (254040824): [] sysvec_apic_timer_interrupt+0xe/0x90 [41758.607104] vfs_write+0x324/0x4f0 [41758.617345] softirqs last enabled at (254040822): [] __do_softirq+0x2be/0x371 [41758.621529] ksys_pwrite64+0x65/0xa0 [41758.631334] softirqs last disabled at (254040817): [] irq_exit_rcu+0xb0/0xd0 [41758.634732] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41758.643501] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41758.647075] do_syscall_64+0x38/0x80 [41758.655679] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41758.660811] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41758.670188] Workqueue: events delayed_work [ceph] [41758.673759] RIP: 0033:0x7f48f0eeb93b [41758.681230] [41758.686275] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41758.690971] RIP: 0010:lock_acquire+0xf0/0x2b0 [41758.694543] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41758.696033] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41758.714769] ORIG_RAX: 0000000000000012 [41758.719122] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41758.724338] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41758.743076] [41758.743077] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41758.746907] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41758.752123] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41758.759247] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41758.760740] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41758.767863] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41758.774987] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41758.782111] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41758.789234] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41758.796366] [41758.803482] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41758.835118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41758.840854] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41758.847979] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41758.855105] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41758.862228] Call Trace: [41758.864685] [41758.866699] ? watchdog_timer_fn+0x1b4/0x220 [41758.870966] ? __hrtimer_run_queues+0x1d3/0x360 [41758.875493] ? hrtimer_interrupt+0x104/0x240 [41758.879761] ? __local_bh_enable+0x50/0x60 [41758.883854] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41758.889079] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41758.894035] [41758.896132] [41758.898231] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41758.903539] ? lock_acquire+0xf0/0x2b0 [41758.907289] ? igrab+0x43/0x50 [41758.910338] ? lock_release+0xc5/0x260 [41758.914084] _raw_spin_lock+0x30/0x40 [41758.917747] ? igrab+0x1d/0x50 [41758.920798] igrab+0x1d/0x50 [41758.923685] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41758.928834] delayed_work+0x1d8/0x280 [ceph] [41758.933115] ? process_scheduled_works+0x22a/0x510 [41758.937905] process_scheduled_works+0x2a4/0x510 [41758.942521] worker_thread+0x188/0x340 [41758.946271] ? rescuer_thread+0x310/0x310 [41758.950274] kthread+0xf5/0x130 [41758.953420] ? kthread_complete_and_exit+0x20/0x20 [41758.958205] ret_from_fork+0x31/0x40 [41758.961783] ? kthread_complete_and_exit+0x20/0x20 [41758.966568] ret_from_fork_asm+0x11/0x20 [41758.970501] [41786.183761] watchdog: BUG: soft lockup - CPU#0 stuck for 27802s! [dbench:78683] [41786.191063] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41786.195760] watchdog: BUG: soft lockup - CPU#7 stuck for 36799s! [kworker/7:4:78808] [41786.246511] irq event stamp: 74855486 [41786.254242] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE [41786.257900] hardirqs last enabled at (74855485): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41786.267287] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [41786.277445] hardirqs last disabled at (74855486): [] sysvec_apic_timer_interrupt+0xe/0x90 [41786.285098] nfnetlink bridge stp llc xfs libcrc32c [41786.294820] softirqs last enabled at (74855430): [] __do_softirq+0x2be/0x371 [41786.294824] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [41786.308376] softirqs last disabled at (74855425): [] irq_exit_rcu+0xb0/0xd0 [41786.317849] irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus [41786.326362] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41786.332877] crct10dif_pclmul joydev [41786.341821] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41786.341823] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41786.345392] crc32_pclmul ipmi_ssif [41786.352863] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41786.358341] ghash_clmulni_intel mei_me [41786.361825] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41786.380561] mei [41786.384392] [41786.389609] lpc_ich [41786.391447] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41786.392938] sha512_ssse3 acpi_ipmi [41786.395123] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41786.402248] mfd_core [41786.405730] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41786.412855] wmi ipmi_si [41786.415125] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41786.422250] ipmi_devintf ipmi_msghandler [41786.424780] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41786.431904] ioatdma acpi_pad [41786.435909] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41786.443033] acpi_power_meter ixgbe [41786.445995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41786.454074] igb mdio [41786.457558] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41786.463296] crc32c_intel nvme [41786.465565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41786.472690] ptp nvme_core [41786.475741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41786.482866] i2c_algo_bit [41786.485569] Call Trace: [41786.485571] [41786.492693] pps_core [41786.495314] ? watchdog_timer_fn+0x1b4/0x220 [41786.497755] dca fuse [41786.499766] ? __hrtimer_run_queues+0x1d3/0x360 [41786.502035] [41786.502036] irq event stamp: 254264302 [41786.506304] ? hrtimer_interrupt+0x104/0x240 [41786.508570] hardirqs last enabled at (254264301): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41786.513095] ? __local_bh_enable+0x50/0x60 [41786.514586] hardirqs last disabled at (254264302): [] sysvec_apic_timer_interrupt+0xe/0x90 [41786.518331] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41786.522593] softirqs last enabled at (254264300): [] __do_softirq+0x2be/0x371 [41786.532838] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41786.536929] softirqs last disabled at (254264295): [] irq_exit_rcu+0xb0/0xd0 [41786.546740] [41786.551957] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41786.560727] [41786.565686] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41786.574283] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41786.576382] Workqueue: events delayed_work [ceph] [41786.585761] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41786.587855] [41786.587856] RIP: 0010:lock_acquire+0xf0/0x2b0 [41786.595327] do_raw_spin_lock+0xb5/0xc0 [41786.600631] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41786.605328] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41786.610197] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41786.611691] try_get_cap_refs+0x159/0x880 [ceph] [41786.616039] [41786.616040] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41786.619876] __ceph_get_caps+0x4a1/0x760 [ceph] [41786.638608] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41786.643397] ? __file_remove_privs+0x89/0x110 [41786.648608] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41786.653222] ? wake_bit_function+0x70/0x70 [41786.654710] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41786.661838] ? ceph_write_iter+0x260/0x770 [ceph] [41786.666358] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41786.673484] ceph_write_iter+0x260/0x770 [ceph] [41786.677835] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41786.684961] ? lock_is_held_type+0xaa/0x120 [41786.689049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41786.696176] vfs_write+0x324/0x4f0 [41786.700870] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41786.707999] ksys_pwrite64+0x65/0xa0 [41786.712520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41786.720598] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41786.724774] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41786.730512] do_syscall_64+0x38/0x80 [41786.733910] Call Trace: [41786.741033] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41786.744604] [41786.751728] RIP: 0033:0x7f48f0eeb93b [41786.756858] ? watchdog_timer_fn+0x1b4/0x220 [41786.763985] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41786.767555] ? __hrtimer_run_queues+0x1d3/0x360 [41786.769997] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41786.775044] ? hrtimer_interrupt+0x104/0x240 [41786.777053] ORIG_RAX: 0000000000000012 [41786.780623] ? __local_bh_enable+0x50/0x60 [41786.784887] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41786.803626] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41786.808166] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41786.813384] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41786.817648] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41786.821478] [41786.825569] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41786.832694] [41786.832696] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41786.837909] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41786.845038] ? lock_acquire+0xf0/0x2b0 [41786.849997] [41786.857118] ? igrab+0x43/0x50 [41786.889853] ? lock_release+0xc5/0x260 [41786.893597] _raw_spin_lock+0x30/0x40 [41786.897263] ? igrab+0x1d/0x50 [41786.900313] igrab+0x1d/0x50 [41786.903191] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41786.908340] delayed_work+0x1d8/0x280 [ceph] [41786.912621] ? process_scheduled_works+0x22a/0x510 [41786.917412] process_scheduled_works+0x2a4/0x510 [41786.922028] worker_thread+0x188/0x340 [41786.925778] ? rescuer_thread+0x310/0x310 [41786.929789] kthread+0xf5/0x130 [41786.932936] ? kthread_complete_and_exit+0x20/0x20 [41786.937728] ret_from_fork+0x31/0x40 [41786.941306] ? kthread_complete_and_exit+0x20/0x20 [41786.946091] ret_from_fork_asm+0x11/0x20 [41786.950022] [41814.184249] watchdog: BUG: soft lockup - CPU#0 stuck for 27828s! [dbench:78683] [41814.191551] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41814.196248] watchdog: BUG: soft lockup - CPU#7 stuck for 36825s! [kworker/7:4:78808] [41814.246999] irq event stamp: 74916726 [41814.254730] Modules linked in: ceph libceph dns_resolver fscache netfs veth [41814.258387] hardirqs last enabled at (74916725): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41814.258393] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [41814.275506] hardirqs last disabled at (74916726): [] sysvec_apic_timer_interrupt+0xe/0x90 [41814.283755] nft_compat nf_tables nfnetlink bridge stp llc xfs [41814.293480] softirqs last enabled at (74916668): [] __do_softirq+0x2be/0x371 [41814.293484] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [41814.307996] softirqs last disabled at (74916663): [] irq_exit_rcu+0xb0/0xd0 [41814.308000] kvm irqbypass i2c_i801 iTCO_wdt [41814.326492] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41814.326493] iTCO_vendor_support [41814.326494] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41814.330756] i2c_smbus crct10dif_pclmul [41814.339701] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41814.342931] joydev crc32_pclmul [41814.350404] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41814.354235] ipmi_ssif [41814.359712] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41814.362935] ghash_clmulni_intel [41814.381674] [41814.384030] mei_me mei [41814.389248] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41814.392473] lpc_ich sha512_ssse3 [41814.393962] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41814.396407] acpi_ipmi mfd_core [41814.403530] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41814.406843] wmi ipmi_si [41814.413965] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41814.417104] ipmi_devintf ipmi_msghandler [41814.424228] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41814.426759] ioatdma acpi_pad [41814.433884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41814.437886] acpi_power_meter [41814.445011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41814.447976] ixgbe igb [41814.456052] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41814.459016] mdio crc32c_intel [41814.464755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41814.467112] nvme [41814.474235] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41814.477286] ptp nvme_core [41814.484411] Call Trace: [41814.484413] [41814.486333] i2c_algo_bit [41814.493462] ? watchdog_timer_fn+0x1b4/0x220 [41814.496163] pps_core dca [41814.498607] ? __hrtimer_run_queues+0x1d3/0x360 [41814.500616] fuse [41814.503236] ? hrtimer_interrupt+0x104/0x240 [41814.507497] irq event stamp: 254485244 [41814.510116] ? __local_bh_enable+0x50/0x60 [41814.514646] hardirqs last enabled at (254485243): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41814.516576] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41814.520838] hardirqs last disabled at (254485244): [] sysvec_apic_timer_interrupt+0xe/0x90 [41814.524582] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41814.528671] softirqs last enabled at (254485242): [] __do_softirq+0x2be/0x371 [41814.538916] [41814.544151] softirqs last disabled at (254485237): [] irq_exit_rcu+0xb0/0xd0 [41814.553961] [41814.558928] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41814.567700] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41814.569796] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41814.578396] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41814.580491] Workqueue: events delayed_work [ceph] [41814.589870] do_raw_spin_lock+0xb5/0xc0 [41814.595173] [41814.595174] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [41814.602652] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41814.607523] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [41814.612224] try_get_cap_refs+0x159/0x880 [ceph] [41814.616050] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [41814.617547] __ceph_get_caps+0x4a1/0x760 [ceph] [41814.622152] [41814.622152] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [41814.626941] ? __file_remove_privs+0x89/0x110 [41814.645672] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [41814.650286] ? wake_bit_function+0x70/0x70 [41814.655502] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [41814.660029] ? ceph_write_iter+0x260/0x770 [ceph] [41814.661518] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [41814.668650] ceph_write_iter+0x260/0x770 [ceph] [41814.673001] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41814.680127] ? lock_is_held_type+0xaa/0x120 [41814.684216] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41814.691343] vfs_write+0x324/0x4f0 [41814.696035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41814.703165] ksys_pwrite64+0x65/0xa0 [41814.707684] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41814.714810] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41814.718986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41814.727064] do_syscall_64+0x38/0x80 [41814.730461] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41814.736198] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41814.739769] Call Trace: [41814.739770] [41814.746895] RIP: 0033:0x7f48f0eeb93b [41814.752025] ? watchdog_timer_fn+0x1b4/0x220 [41814.759151] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41814.762720] ? __hrtimer_run_queues+0x1d3/0x360 [41814.769843] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41814.774892] ? hrtimer_interrupt+0x104/0x240 [41814.777332] ORIG_RAX: 0000000000000012 [41814.779341] ? __local_bh_enable+0x50/0x60 [41814.782913] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41814.787181] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41814.805915] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41814.810440] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41814.815655] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41814.819931] [41814.823760] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41814.827850] [41814.834974] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41814.840193] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41814.847322] [41814.852277] ? do_raw_spin_lock+0x42/0xc0 [41814.889335] igrab+0x1d/0x50 [41814.892214] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41814.897361] delayed_work+0x1d8/0x280 [ceph] [41814.901658] ? process_scheduled_works+0x22a/0x510 [41814.906443] process_scheduled_works+0x2a4/0x510 [41814.911059] worker_thread+0x188/0x340 [41814.914808] ? rescuer_thread+0x310/0x310 [41814.918819] kthread+0xf5/0x130 [41814.921965] ? kthread_complete_and_exit+0x20/0x20 [41814.926749] ret_from_fork+0x31/0x40 [41814.930329] ? kthread_complete_and_exit+0x20/0x20 [41814.935111] ret_from_fork_asm+0x11/0x20 [41814.939044] [41842.184736] watchdog: BUG: soft lockup - CPU#0 stuck for 27854s! [dbench:78683] [41842.192037] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41842.196736] watchdog: BUG: soft lockup - CPU#7 stuck for 36851s! [kworker/7:4:78808] [41842.247478] irq event stamp: 74974832 [41842.247479] hardirqs last enabled at (74974831): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41842.255210] Modules linked in: [41842.258867] hardirqs last disabled at (74974832): [] sysvec_apic_timer_interrupt+0xe/0x90 [41842.269023] ceph [41842.272074] softirqs last enabled at (74974774): [] __do_softirq+0x2be/0x371 [41842.281799] libceph [41842.283722] softirqs last disabled at (74974769): [] irq_exit_rcu+0xb0/0xd0 [41842.292411] dns_resolver fscache [41842.294592] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41842.303102] netfs veth [41842.306425] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41842.315365] nft_chain_nat [41842.317809] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41842.325280] xt_MASQUERADE nf_nat [41842.327985] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41842.333461] nf_conntrack nf_defrag_ipv6 [41842.336773] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41842.355511] nf_defrag_ipv4 nft_compat [41842.359429] [41842.359429] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41842.364645] nf_tables [41842.368389] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41842.369881] nfnetlink bridge [41842.377005] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41842.379362] stp llc [41842.386485] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41842.389449] xfs libcrc32c [41842.396573] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41842.398759] sunrpc intel_rapl_msr [41842.405882] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41842.408587] intel_rapl_common [41842.415711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41842.419108] sb_edac [41842.427202] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41842.430252] x86_pkg_temp_thermal coretemp [41842.435991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41842.438176] kvm_intel [41842.445299] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41842.449391] kvm irqbypass [41842.456513] Call Trace: [41842.458872] i2c_i801 iTCO_wdt [41842.465996] [41842.468701] iTCO_vendor_support i2c_smbus [41842.471144] ? watchdog_timer_fn+0x1b4/0x220 [41842.474194] crct10dif_pclmul joydev [41842.476206] ? __hrtimer_run_queues+0x1d3/0x360 [41842.480296] crc32_pclmul ipmi_ssif [41842.484562] ? hrtimer_interrupt+0x104/0x240 [41842.488135] ghash_clmulni_intel [41842.492664] ? __local_bh_enable+0x50/0x60 [41842.496147] mei_me [41842.500427] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41842.503651] mei lpc_ich [41842.507745] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41842.509841] sha512_ssse3 acpi_ipmi [41842.515059] [41842.517589] mfd_core [41842.522547] [41842.526031] wmi ipmi_si [41842.528136] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41842.530415] ipmi_devintf ipmi_msghandler [41842.532518] ? queued_spin_lock_slowpath+0x16/0x2e0 [41842.535044] ioatdma acpi_pad [41842.540349] do_raw_spin_lock+0xb5/0xc0 [41842.544351] acpi_power_meter ixgbe igb [41842.549223] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41842.552186] mdio crc32c_intel [41842.556020] try_get_cap_refs+0x159/0x880 [ceph] [41842.559849] nvme [41842.564636] __ceph_get_caps+0x4a1/0x760 [ceph] [41842.567682] ptp nvme_core [41842.572300] ? __file_remove_privs+0x89/0x110 [41842.574217] i2c_algo_bit pps_core [41842.578743] ? wake_bit_function+0x70/0x70 [41842.581445] dca fuse [41842.585799] ? ceph_write_iter+0x260/0x770 [ceph] [41842.589194] [41842.589195] irq event stamp: 254706226 [41842.593285] ceph_write_iter+0x260/0x770 [ceph] [41842.595555] hardirqs last enabled at (254706225): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41842.600257] ? lock_is_held_type+0xaa/0x120 [41842.601742] hardirqs last disabled at (254706226): [] sysvec_apic_timer_interrupt+0xe/0x90 [41842.605491] vfs_write+0x324/0x4f0 [41842.610011] softirqs last enabled at (254706224): [] __do_softirq+0x2be/0x371 [41842.620261] ksys_pwrite64+0x65/0xa0 [41842.624432] softirqs last disabled at (254706219): [] irq_exit_rcu+0xb0/0xd0 [41842.634244] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41842.637642] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41842.646415] do_syscall_64+0x38/0x80 [41842.649984] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41842.658581] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41842.663713] Workqueue: events delayed_work [ceph] [41842.673090] RIP: 0033:0x7f48f0eeb93b [41842.676658] [41842.684139] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41842.689184] RIP: 0010:lock_acquire+0x85/0x2b0 [41842.693881] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41842.697451] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [41842.698942] ORIG_RAX: 0000000000000012 [41842.717681] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [41842.722031] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41842.727249] [41842.727249] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [41842.745986] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41842.749817] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [41842.755034] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41842.762157] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [41842.763648] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41842.770773] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [41842.777897] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41842.785021] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41842.792150] [41842.799270] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41842.838029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41842.843765] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41842.850889] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41842.858014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41842.865143] Call Trace: [41842.867592] [41842.869603] ? watchdog_timer_fn+0x1b4/0x220 [41842.873867] ? __hrtimer_run_queues+0x1d3/0x360 [41842.878393] ? hrtimer_interrupt+0x104/0x240 [41842.882662] ? __local_bh_enable+0x50/0x60 [41842.886755] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41842.891970] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41842.896929] [41842.899025] [41842.901141] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41842.906457] ? lock_acquire+0x85/0x2b0 [41842.910206] ? igrab+0x43/0x50 [41842.913256] ? lock_release+0xc5/0x260 [41842.917001] _raw_spin_lock+0x30/0x40 [41842.920668] ? igrab+0x1d/0x50 [41842.923717] igrab+0x1d/0x50 [41842.926595] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41842.931745] delayed_work+0x1d8/0x280 [ceph] [41842.936027] ? process_scheduled_works+0x22a/0x510 [41842.940816] process_scheduled_works+0x2a4/0x510 [41842.945433] worker_thread+0x188/0x340 [41842.949181] ? rescuer_thread+0x310/0x310 [41842.953194] kthread+0xf5/0x130 [41842.956332] ? kthread_complete_and_exit+0x20/0x20 [41842.961116] ret_from_fork+0x31/0x40 [41842.964711] ? kthread_complete_and_exit+0x20/0x20 [41842.969496] ret_from_fork_asm+0x11/0x20 [41842.973434] [41870.185222] watchdog: BUG: soft lockup - CPU#0 stuck for 27880s! [dbench:78683] [41870.192527] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41870.197221] watchdog: BUG: soft lockup - CPU#7 stuck for 36877s! [kworker/7:4:78808] [41870.247975] irq event stamp: 75033040 [41870.247976] hardirqs last enabled at (75033039): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41870.255706] Modules linked in: [41870.259363] hardirqs last disabled at (75033040): [] sysvec_apic_timer_interrupt+0xe/0x90 [41870.269521] ceph [41870.272571] softirqs last enabled at (75032982): [] __do_softirq+0x2be/0x371 [41870.282297] libceph [41870.284220] softirqs last disabled at (75032977): [] irq_exit_rcu+0xb0/0xd0 [41870.292903] dns_resolver fscache [41870.295089] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41870.303615] netfs veth [41870.306926] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41870.315870] nft_chain_nat [41870.318315] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41870.325786] xt_MASQUERADE nf_nat [41870.328492] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41870.333966] nf_conntrack [41870.337279] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41870.356016] nf_defrag_ipv6 nf_defrag_ipv4 [41870.358633] [41870.358634] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41870.363850] nft_compat nf_tables [41870.367940] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41870.369431] nfnetlink bridge [41870.376556] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41870.379865] stp llc [41870.386991] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41870.389955] xfs libcrc32c [41870.397078] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41870.399263] sunrpc intel_rapl_msr [41870.406388] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41870.409090] intel_rapl_common [41870.416214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41870.419616] sb_edac [41870.427689] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41870.430741] x86_pkg_temp_thermal coretemp [41870.436480] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41870.438661] kvm_intel [41870.445785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41870.449876] kvm irqbypass i2c_i801 [41870.457019] Call Trace: [41870.457021] [41870.459375] iTCO_wdt [41870.466502] ? watchdog_timer_fn+0x1b4/0x220 [41870.469983] iTCO_vendor_support [41870.472428] ? __hrtimer_run_queues+0x1d3/0x360 [41870.474439] i2c_smbus crct10dif_pclmul [41870.476714] ? hrtimer_interrupt+0x104/0x240 [41870.480974] joydev crc32_pclmul [41870.484199] ? __local_bh_enable+0x50/0x60 [41870.488721] ipmi_ssif [41870.492555] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41870.496818] ghash_clmulni_intel mei_me [41870.500041] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41870.504131] mei lpc_ich [41870.506488] [41870.511705] sha512_ssse3 acpi_ipmi [41870.515536] [41870.515537] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41870.520494] mfd_core [41870.523030] ? queued_spin_lock_slowpath+0x18/0x2e0 [41870.525122] wmi ipmi_si [41870.528622] do_raw_spin_lock+0xb5/0xc0 [41870.530721] ipmi_devintf ipmi_msghandler [41870.536026] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41870.538296] ioatdma acpi_pad [41870.543171] try_get_cap_refs+0x159/0x880 [ceph] [41870.545697] acpi_power_meter ixgbe [41870.549535] __ceph_get_caps+0x4a1/0x760 [ceph] [41870.553533] igb mdio [41870.558321] ? __file_remove_privs+0x89/0x110 [41870.561280] crc32c_intel nvme [41870.565901] ? wake_bit_function+0x70/0x70 [41870.569383] ptp nvme_core [41870.573912] ? ceph_write_iter+0x260/0x770 [ceph] [41870.576178] i2c_algo_bit pps_core [41870.580530] ceph_write_iter+0x260/0x770 [ceph] [41870.583580] dca fuse [41870.587675] ? lock_is_held_type+0xaa/0x120 [41870.590375] [41870.590375] irq event stamp: 254926950 [41870.595074] vfs_write+0x324/0x4f0 [41870.598469] hardirqs last enabled at (254926949): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41870.603000] ksys_pwrite64+0x65/0xa0 [41870.605265] hardirqs last disabled at (254926950): [] sysvec_apic_timer_interrupt+0xe/0x90 [41870.609443] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41870.610932] softirqs last enabled at (254926948): [] __do_softirq+0x2be/0x371 [41870.614677] do_syscall_64+0x38/0x80 [41870.618073] softirqs last disabled at (254926943): [] irq_exit_rcu+0xb0/0xd0 [41870.628319] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41870.631889] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41870.641701] RIP: 0033:0x7f48f0eeb93b [41870.646829] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41870.655620] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41870.659189] Workqueue: events delayed_work [ceph] [41870.667787] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41870.672832] [41870.682209] ORIG_RAX: 0000000000000012 [41870.685778] RIP: 0010:ceph_check_delayed_caps+0x69/0x270 [ceph] [41870.693251] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41870.711989] Code: 00 4c 8d ad 78 04 00 00 48 89 c7 48 89 04 24 e8 fd 58 07 e1 eb 6b 48 8b 9d 78 04 00 00 4c 89 f0 48 8b 53 f8 48 29 d0 4c 01 e0 <0f> 88 14 01 00 00 48 8b 83 b0 fd ff ff 83 e0 04 75 0c 48 8b 0d 2e [41870.716685] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41870.721902] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [41870.723394] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41870.727224] [41870.727224] RAX: 0000000000032880 RBX: ffff8881a1d34bc8 RCX: ffffc90002117d54 [41870.733134] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41870.740259] RDX: 00000001001ae57b RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [41870.758997] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41870.766121] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [41870.771345] [41870.778463] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [41870.824882] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41870.832025] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41870.840100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41870.845838] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41870.852962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41870.860086] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41870.867211] Call Trace: [41870.869656] [41870.871667] ? watchdog_timer_fn+0x1b4/0x220 [41870.875932] ? __hrtimer_run_queues+0x1d3/0x360 [41870.880459] ? hrtimer_interrupt+0x104/0x240 [41870.884726] ? __local_bh_enable+0x50/0x60 [41870.888821] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41870.894043] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41870.899001] [41870.901098] [41870.903197] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41870.908505] ? ceph_check_delayed_caps+0x69/0x270 [ceph] [41870.913824] ? ceph_check_delayed_caps+0xb7/0x270 [ceph] [41870.919146] delayed_work+0x1d8/0x280 [ceph] [41870.923428] ? process_scheduled_works+0x22a/0x510 [41870.928216] process_scheduled_works+0x2a4/0x510 [41870.932832] worker_thread+0x188/0x340 [41870.936582] ? rescuer_thread+0x310/0x310 [41870.940586] kthread+0xf5/0x130 [41870.943732] ? kthread_complete_and_exit+0x20/0x20 [41870.948516] ret_from_fork+0x31/0x40 [41870.952095] ? kthread_complete_and_exit+0x20/0x20 [41870.956880] ret_from_fork_asm+0x11/0x20 [41870.960811] [41898.185706] watchdog: BUG: soft lockup - CPU#0 stuck for 27906s! [dbench:78683] [41898.193004] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41898.197706] watchdog: BUG: soft lockup - CPU#7 stuck for 36903s! [kworker/7:4:78808] [41898.248446] irq event stamp: 75091302 [41898.248447] hardirqs last enabled at (75091301): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41898.256175] Modules linked in: [41898.259833] hardirqs last disabled at (75091302): [] sysvec_apic_timer_interrupt+0xe/0x90 [41898.269989] ceph [41898.273040] softirqs last enabled at (75091244): [] __do_softirq+0x2be/0x371 [41898.282766] libceph [41898.284690] softirqs last disabled at (75091239): [] irq_exit_rcu+0xb0/0xd0 [41898.293381] dns_resolver fscache [41898.295567] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41898.304079] netfs [41898.307409] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41898.316350] veth nft_chain_nat [41898.318360] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41898.325848] xt_MASQUERADE nf_nat [41898.328986] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41898.334462] nf_conntrack nf_defrag_ipv6 [41898.337774] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41898.356512] nf_defrag_ipv4 nft_compat [41898.360428] [41898.360429] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41898.365646] nf_tables nfnetlink [41898.369390] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41898.370881] bridge stp [41898.378006] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41898.381229] llc [41898.388354] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41898.390798] xfs libcrc32c [41898.397922] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41898.399760] sunrpc intel_rapl_msr [41898.406884] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41898.409588] intel_rapl_common sb_edac [41898.416710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41898.420108] x86_pkg_temp_thermal [41898.428187] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41898.431931] coretemp kvm_intel [41898.437669] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41898.440977] kvm irqbypass [41898.448103] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41898.451239] i2c_i801 iTCO_wdt [41898.458380] Call Trace: [41898.458381] [41898.461104] iTCO_vendor_support [41898.468227] ? watchdog_timer_fn+0x1b4/0x220 [41898.471278] i2c_smbus crct10dif_pclmul [41898.473722] ? __hrtimer_run_queues+0x1d3/0x360 [41898.475732] joydev crc32_pclmul [41898.478959] ? hrtimer_interrupt+0x104/0x240 [41898.483220] ipmi_ssif ghash_clmulni_intel [41898.487050] ? __local_bh_enable+0x50/0x60 [41898.491576] mei_me mei [41898.494802] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41898.499063] lpc_ich [41898.503154] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41898.507244] sha512_ssse3 acpi_ipmi [41898.509688] [41898.514905] mfd_core wmi [41898.517107] [41898.522064] ipmi_si ipmi_devintf [41898.525549] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41898.527647] ipmi_msghandler [41898.530268] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41898.532361] ioatdma acpi_pad [41898.535675] do_raw_spin_lock+0xb5/0xc0 [41898.540976] acpi_power_meter ixgbe [41898.543854] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41898.548724] igb mdio [41898.551691] try_get_cap_refs+0x159/0x880 [ceph] [41898.555519] crc32c_intel nvme [41898.559007] __ceph_get_caps+0x4a1/0x760 [ceph] [41898.563788] ptp nvme_core [41898.566063] ? __file_remove_privs+0x89/0x110 [41898.570669] i2c_algo_bit pps_core [41898.573720] ? wake_bit_function+0x70/0x70 [41898.578245] dca [41898.580950] ? ceph_write_iter+0x260/0x770 [ceph] [41898.585300] fuse [41898.588697] ceph_write_iter+0x260/0x770 [ceph] [41898.592787] irq event stamp: 255147846 [41898.594627] ? lock_is_held_type+0xaa/0x120 [41898.599321] hardirqs last enabled at (255147845): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41898.601247] vfs_write+0x324/0x4f0 [41898.605769] hardirqs last disabled at (255147846): [] sysvec_apic_timer_interrupt+0xe/0x90 [41898.609518] ksys_pwrite64+0x65/0xa0 [41898.613691] softirqs last enabled at (255147844): [] __do_softirq+0x2be/0x371 [41898.623935] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41898.627333] softirqs last disabled at (255147839): [] irq_exit_rcu+0xb0/0xd0 [41898.637145] do_syscall_64+0x38/0x80 [41898.640715] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41898.649485] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41898.654615] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41898.663213] RIP: 0033:0x7f48f0eeb93b [41898.666785] Workqueue: events delayed_work [ceph] [41898.676162] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41898.681206] [41898.681207] RIP: 0010:lock_acquire+0xf0/0x2b0 [41898.688677] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41898.692247] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41898.696944] ORIG_RAX: 0000000000000012 [41898.715682] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41898.717172] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41898.721523] [41898.726741] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41898.745478] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41898.749309] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41898.754527] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41898.761652] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41898.763142] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41898.770267] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41898.777390] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41898.784537] [41898.791657] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41898.829460] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41898.837537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41898.843275] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41898.850399] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41898.857525] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41898.864649] Call Trace: [41898.867110] [41898.869120] ? watchdog_timer_fn+0x1b4/0x220 [41898.873392] ? __hrtimer_run_queues+0x1d3/0x360 [41898.877920] ? hrtimer_interrupt+0x104/0x240 [41898.882190] ? __local_bh_enable+0x50/0x60 [41898.886284] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41898.891507] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41898.896465] [41898.898561] [41898.900659] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41898.905967] ? lock_acquire+0xf0/0x2b0 [41898.909716] ? igrab+0x43/0x50 [41898.912766] ? lock_release+0xc5/0x260 [41898.916511] _raw_spin_lock+0x30/0x40 [41898.920167] ? igrab+0x1d/0x50 [41898.923220] igrab+0x1d/0x50 [41898.926112] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41898.931255] delayed_work+0x1d8/0x280 [ceph] [41898.935537] ? process_scheduled_works+0x22a/0x510 [41898.940326] process_scheduled_works+0x2a4/0x510 [41898.944942] worker_thread+0x188/0x340 [41898.948691] ? rescuer_thread+0x310/0x310 [41898.952696] kthread+0xf5/0x130 [41898.955834] ? kthread_complete_and_exit+0x20/0x20 [41898.960626] ret_from_fork+0x31/0x40 [41898.964196] ? kthread_complete_and_exit+0x20/0x20 [41898.968979] ret_from_fork_asm+0x11/0x20 [41898.972903] [41899.875101] rcu: INFO: rcu_preempt self-detected stall on CPU [41899.880861] rcu: 0-....: (29284373 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14670270 [41899.891373] rcu: (t=29962743 jiffies g=1309609 q=802155 ncpus=8) [41899.897459] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41899.906402] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41899.913870] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [41899.919349] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [41899.938086] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41899.943304] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41899.950428] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41899.957551] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41899.964675] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41899.971801] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41899.978925] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41899.987001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41899.992741] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41899.999864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41900.006988] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41900.014112] Call Trace: [41900.016557] [41900.018567] ? rcu_dump_cpu_stacks+0xd1/0x180 [41900.022926] ? rcu_sched_clock_irq+0x633/0x1390 [41900.027452] ? lock_is_held_type+0xaa/0x120 [41900.031631] ? lock_is_held_type+0xaa/0x120 [41900.035816] ? tick_sched_do_timer+0xa0/0xa0 [41900.040087] ? rcu_read_lock_sched_held+0x56/0x90 [41900.044785] ? tick_sched_do_timer+0xa0/0xa0 [41900.049048] ? update_process_times+0x5f/0x90 [41900.053398] ? tick_sched_handle.isra.16+0x1e/0x50 [41900.058182] ? tick_sched_timer+0x6f/0x80 [41900.062188] ? __hrtimer_run_queues+0x1d3/0x360 [41900.066715] ? hrtimer_interrupt+0x104/0x240 [41900.070983] ? __local_bh_enable+0x50/0x60 [41900.075076] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41900.080302] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41900.085259] [41900.087355] [41900.089452] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41900.094760] ? queued_spin_lock_slowpath+0x18/0x2e0 [41900.099639] do_raw_spin_lock+0xb5/0xc0 [41900.103476] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41900.108280] try_get_cap_refs+0x159/0x880 [ceph] [41900.112910] __ceph_get_caps+0x4a1/0x760 [ceph] [41900.117451] ? __file_remove_privs+0x89/0x110 [41900.121807] ? wake_bit_function+0x70/0x70 [41900.125899] ? ceph_write_iter+0x260/0x770 [ceph] [41900.130612] ceph_write_iter+0x260/0x770 [ceph] [41900.135149] ? lock_is_held_type+0xaa/0x120 [41900.139333] vfs_write+0x324/0x4f0 [41900.142741] ksys_pwrite64+0x65/0xa0 [41900.146317] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41900.151447] do_syscall_64+0x38/0x80 [41900.155018] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41900.160063] RIP: 0033:0x7f48f0eeb93b [41900.163632] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41900.182369] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [41900.189927] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41900.197050] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41900.204176] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41900.211300] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41900.218424] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41900.225554] [41900.227740] Sending NMI from CPU 0 to CPUs 7: [41900.232092] NMI backtrace for cpu 7 [41900.232093] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41900.232095] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41900.232096] Workqueue: events delayed_work [ceph] [41900.232113] RIP: 0010:__lock_acquire+0x783/0x1e20 [41900.232115] Code: 8b 3c 25 c0 e5 02 00 e8 2b d0 ff ff e8 d6 2b 00 00 e8 11 e9 9f 00 45 31 f6 48 8b 5c 24 50 65 48 33 1c 25 28 00 00 00 44 89 f0 <0f> 85 04 12 00 00 48 83 c4 58 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 [41900.232116] RSP: 0018:ffffc90002117c60 EFLAGS: 00000046 [41900.232117] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 000000005553a634 [41900.232118] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41900.232119] RBP: ffff88810debb550 R08: 0000000000005c53 R09: 0000000000000003 [41900.232120] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8360d858 [41900.232120] R13: 0000000000000003 R14: 0000000000000001 R15: c5dbdfc36fd10127 [41900.232121] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41900.232122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41900.232122] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41900.232123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41900.232124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41900.232125] Call Trace: [41900.232125] [41900.232126] ? nmi_cpu_backtrace+0xad/0x120 [41900.232130] ? nmi_cpu_backtrace_handler+0x11/0x20 [41900.232131] ? nmi_handle+0xd4/0x260 [41900.232135] ? default_do_nmi+0x49/0xf0 [41900.232137] ? exc_nmi+0x11e/0x150 [41900.232138] ? end_repeat_nmi+0x16/0x67 [41900.232142] ? __lock_acquire+0x783/0x1e20 [41900.232143] ? __lock_acquire+0x783/0x1e20 [41900.232145] ? __lock_acquire+0x783/0x1e20 [41900.232146] [41900.232147] [41900.232148] lock_acquire+0xcb/0x2b0 [41900.232150] ? igrab+0x1d/0x50 [41900.232151] ? igrab+0x43/0x50 [41900.232152] ? lock_release+0xc5/0x260 [41900.232154] _raw_spin_lock+0x30/0x40 [41900.232156] ? igrab+0x1d/0x50 [41900.232157] igrab+0x1d/0x50 [41900.232158] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41900.232172] delayed_work+0x1d8/0x280 [ceph] [41900.232185] ? process_scheduled_works+0x22a/0x510 [41900.232188] process_scheduled_works+0x2a4/0x510 [41900.232191] worker_thread+0x188/0x340 [41900.232194] ? rescuer_thread+0x310/0x310 [41900.232196] kthread+0xf5/0x130 [41900.232198] ? kthread_complete_and_exit+0x20/0x20 [41900.232201] ret_from_fork+0x31/0x40 [41900.232203] ? kthread_complete_and_exit+0x20/0x20 [41900.232205] ret_from_fork_asm+0x11/0x20 [41900.232208] [41926.186190] watchdog: BUG: soft lockup - CPU#0 stuck for 27932s! [dbench:78683] [41926.193493] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41926.198189] watchdog: BUG: soft lockup - CPU#7 stuck for 36929s! [kworker/7:4:78808] [41926.248940] irq event stamp: 75148764 [41926.256673] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [41926.260330] hardirqs last enabled at (75148763): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41926.260335] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables [41926.278659] hardirqs last disabled at (75148764): [] sysvec_apic_timer_interrupt+0xe/0x90 [41926.287517] nfnetlink bridge stp llc xfs libcrc32c sunrpc [41926.297241] softirqs last enabled at (75148706): [] __do_softirq+0x2be/0x371 [41926.302718] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [41926.311402] softirqs last disabled at (75148701): [] irq_exit_rcu+0xb0/0xd0 [41926.320261] irqbypass i2c_i801 iTCO_wdt [41926.328773] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41926.332688] iTCO_vendor_support i2c_smbus [41926.341633] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41926.341635] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41926.345723] crct10dif_pclmul [41926.353195] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41926.358671] joydev crc32_pclmul [41926.361635] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41926.380373] ipmi_ssif [41926.383598] [41926.388815] ghash_clmulni_intel mei_me [41926.391172] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41926.392662] mei lpc_ich [41926.396493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41926.403618] sha512_ssse3 [41926.406149] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41926.413273] acpi_ipmi mfd_core [41926.415891] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41926.423013] wmi ipmi_si [41926.426151] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41926.433275] ipmi_devintf ipmi_msghandler [41926.435808] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41926.442930] ioatdma acpi_pad [41926.446934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41926.455012] acpi_power_meter ixgbe [41926.457977] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41926.463714] igb [41926.467199] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41926.474322] mdio crc32c_intel [41926.476159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41926.483284] nvme ptp [41926.486335] Call Trace: [41926.486337] [41926.493457] nvme_core i2c_algo_bit [41926.495732] ? watchdog_timer_fn+0x1b4/0x220 [41926.498174] pps_core dca [41926.500185] ? __hrtimer_run_queues+0x1d3/0x360 [41926.503668] fuse [41926.507934] ? hrtimer_interrupt+0x104/0x240 [41926.510550] [41926.510551] irq event stamp: 255366550 [41926.515073] ? __local_bh_enable+0x50/0x60 [41926.516998] hardirqs last enabled at (255366549): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41926.521263] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41926.522753] hardirqs last disabled at (255366550): [] sysvec_apic_timer_interrupt+0xe/0x90 [41926.526498] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41926.530587] softirqs last enabled at (255366548): [] __do_softirq+0x2be/0x371 [41926.540832] [41926.546048] softirqs last disabled at (255366543): [] irq_exit_rcu+0xb0/0xd0 [41926.555871] [41926.560825] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41926.569598] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41926.571693] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41926.580295] ? queued_spin_lock_slowpath+0x16/0x2e0 [41926.582388] Workqueue: events delayed_work [ceph] [41926.591768] do_raw_spin_lock+0xb5/0xc0 [41926.597070] [41926.597071] RIP: 0010:_raw_spin_unlock+0x1e/0x40 [41926.604543] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41926.609411] Code: 89 d8 5b 5d c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 74 24 08 48 89 fb 48 83 c7 18 e8 35 06 5c ff 48 89 df 7d 31 5c ff 65 ff 0d ee f0 47 7e 74 02 5b c3 0f 1f 44 00 00 5b [41926.614112] try_get_cap_refs+0x159/0x880 [ceph] [41926.617941] RSP: 0018:ffffc90002117d90 EFLAGS: 00000296 [41926.619435] __ceph_get_caps+0x4a1/0x760 [ceph] [41926.624042] [41926.624042] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: ffffc90002117d54 [41926.628830] ? __file_remove_privs+0x89/0x110 [41926.647564] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff8881a1d34500 [41926.652192] ? wake_bit_function+0x70/0x70 [41926.657409] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [41926.661935] ? ceph_write_iter+0x260/0x770 [ceph] [41926.663423] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [41926.670549] ceph_write_iter+0x260/0x770 [ceph] [41926.674897] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [41926.682025] ? lock_is_held_type+0xaa/0x120 [41926.686112] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41926.693244] vfs_write+0x324/0x4f0 [41926.697934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41926.697935] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41926.705065] ksys_pwrite64+0x65/0xa0 [41926.709592] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41926.716715] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41926.720892] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41926.728972] do_syscall_64+0x38/0x80 [41926.732367] Call Trace: [41926.732368] [41926.738107] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41926.745231] ? watchdog_timer_fn+0x1b4/0x220 [41926.748801] RIP: 0033:0x7f48f0eeb93b [41926.755927] ? __hrtimer_run_queues+0x1d3/0x360 [41926.761056] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41926.768184] ? hrtimer_interrupt+0x104/0x240 [41926.771750] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41926.774193] ? __local_bh_enable+0x50/0x60 [41926.776205] ORIG_RAX: 0000000000000012 [41926.781251] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41926.785514] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41926.789085] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41926.793607] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41926.812347] [41926.816610] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41926.821828] [41926.821829] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41926.825916] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41926.829754] ? _raw_spin_unlock+0x1e/0x40 [41926.834966] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41926.842092] igrab+0x43/0x50 [41926.847052] [41926.854174] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41926.899241] delayed_work+0x1d8/0x280 [ceph] [41926.903525] ? process_scheduled_works+0x22a/0x510 [41926.908312] process_scheduled_works+0x2a4/0x510 [41926.912929] worker_thread+0x188/0x340 [41926.916678] ? rescuer_thread+0x310/0x310 [41926.920682] kthread+0xf5/0x130 [41926.923829] ? kthread_complete_and_exit+0x20/0x20 [41926.928613] ret_from_fork+0x31/0x40 [41926.932192] ? kthread_complete_and_exit+0x20/0x20 [41926.936974] ret_from_fork_asm+0x11/0x20 [41926.940898] [41954.186672] watchdog: BUG: soft lockup - CPU#0 stuck for 27958s! [dbench:78683] [41954.193971] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41954.198672] watchdog: BUG: soft lockup - CPU#7 stuck for 36955s! [kworker/7:4:78808] [41954.249411] irq event stamp: 75206876 [41954.249412] hardirqs last enabled at (75206875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41954.257142] Modules linked in: ceph [41954.260798] hardirqs last disabled at (75206876): [] sysvec_apic_timer_interrupt+0xe/0x90 [41954.270957] libceph [41954.274440] softirqs last enabled at (75206818): [] __do_softirq+0x2be/0x371 [41954.284164] dns_resolver fscache [41954.286355] softirqs last disabled at (75206813): [] irq_exit_rcu+0xb0/0xd0 [41954.295042] netfs [41954.298373] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41954.306880] veth nft_chain_nat [41954.308890] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41954.317837] xt_MASQUERADE nf_nat [41954.320973] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [41954.328444] nf_conntrack nf_defrag_ipv6 [41954.331755] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [41954.337232] nf_defrag_ipv4 [41954.341150] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41954.359888] nft_compat nf_tables [41954.362677] [41954.362678] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41954.367895] nfnetlink bridge [41954.371205] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41954.372696] stp llc [41954.379821] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41954.382786] xfs libcrc32c [41954.389910] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41954.392092] sunrpc intel_rapl_msr [41954.399218] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41954.401921] intel_rapl_common sb_edac [41954.409045] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41954.412461] x86_pkg_temp_thermal coretemp [41954.419584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41954.423329] kvm_intel kvm [41954.431405] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41954.435497] irqbypass i2c_i801 [41954.441235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41954.443937] iTCO_wdt iTCO_vendor_support [41954.451068] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41954.454209] i2c_smbus [41954.461348] Call Trace: [41954.465354] crct10dif_pclmul joydev [41954.472477] [41954.474836] crc32_pclmul ipmi_ssif [41954.477279] ? watchdog_timer_fn+0x1b4/0x220 [41954.480849] ghash_clmulni_intel mei_me [41954.482862] ? __hrtimer_run_queues+0x1d3/0x360 [41954.486351] mei lpc_ich [41954.490621] ? hrtimer_interrupt+0x104/0x240 [41954.494448] sha512_ssse3 acpi_ipmi [41954.498973] ? __local_bh_enable+0x50/0x60 [41954.501504] mfd_core wmi [41954.505770] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41954.509251] ipmi_si ipmi_devintf [41954.513341] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41954.515958] ipmi_msghandler ioatdma [41954.521177] [41954.524488] acpi_pad acpi_power_meter [41954.529444] [41954.533015] ixgbe igb [41954.535113] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41954.538856] mdio crc32c_intel [41954.540957] ? queued_spin_lock_slowpath+0x2d/0x2e0 [41954.543311] nvme ptp [41954.548619] do_raw_spin_lock+0xb5/0xc0 [41954.551667] nvme_core i2c_algo_bit [41954.556538] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41954.558807] pps_core dca [41954.562642] try_get_cap_refs+0x159/0x880 [ceph] [41954.566124] fuse [41954.570913] __ceph_get_caps+0x4a1/0x760 [ceph] [41954.573526] irq event stamp: 255587610 [41954.578139] ? __file_remove_privs+0x89/0x110 [41954.580072] hardirqs last enabled at (255587609): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41954.584602] ? wake_bit_function+0x70/0x70 [41954.588344] hardirqs last disabled at (255587610): [] sysvec_apic_timer_interrupt+0xe/0x90 [41954.592699] ? ceph_write_iter+0x260/0x770 [ceph] [41954.602941] softirqs last enabled at (255587608): [] __do_softirq+0x2be/0x371 [41954.607031] ceph_write_iter+0x260/0x770 [ceph] [41954.616842] softirqs last disabled at (255587603): [] irq_exit_rcu+0xb0/0xd0 [41954.621542] ? lock_is_held_type+0xaa/0x120 [41954.630309] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41954.634836] vfs_write+0x324/0x4f0 [41954.643431] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41954.647613] ksys_pwrite64+0x65/0xa0 [41954.656986] Workqueue: events delayed_work [ceph] [41954.660384] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41954.667855] [41954.667856] RIP: 0010:lock_acquire+0xf0/0x2b0 [41954.671426] do_syscall_64+0x38/0x80 [41954.676121] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41954.681254] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41954.682744] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41954.687094] RIP: 0033:0x7f48f0eeb93b [41954.690666] [41954.690667] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41954.709404] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41954.714448] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41954.719664] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41954.723235] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41954.724725] ORIG_RAX: 0000000000000012 [41954.731850] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41954.731851] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41954.750588] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41954.757712] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41954.762930] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41954.770071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41954.773901] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41954.781027] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41954.788150] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41954.795274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41954.803352] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41954.810492] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41954.816235] [41954.823370] Call Trace: [41954.823372] [41954.865632] ? watchdog_timer_fn+0x1b4/0x220 [41954.869896] ? __hrtimer_run_queues+0x1d3/0x360 [41954.874423] ? hrtimer_interrupt+0x104/0x240 [41954.878685] ? __local_bh_enable+0x50/0x60 [41954.882785] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41954.888001] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41954.892959] [41954.895055] [41954.897171] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41954.902479] ? lock_acquire+0xf0/0x2b0 [41954.906227] ? igrab+0x43/0x50 [41954.909279] ? lock_release+0xc5/0x260 [41954.913022] _raw_spin_lock+0x30/0x40 [41954.916681] ? igrab+0x1d/0x50 [41954.919729] igrab+0x1d/0x50 [41954.922607] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41954.927757] delayed_work+0x1d8/0x280 [ceph] [41954.932040] ? process_scheduled_works+0x22a/0x510 [41954.936830] process_scheduled_works+0x2a4/0x510 [41954.941444] worker_thread+0x188/0x340 [41954.945195] ? rescuer_thread+0x310/0x310 [41954.949198] kthread+0xf5/0x130 [41954.952336] ? kthread_complete_and_exit+0x20/0x20 [41954.957136] ret_from_fork+0x31/0x40 [41954.960706] ? kthread_complete_and_exit+0x20/0x20 [41954.965492] ret_from_fork_asm+0x11/0x20 [41954.969414] [41982.187154] watchdog: BUG: soft lockup - CPU#0 stuck for 27984s! [dbench:78683] [41982.194458] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [41982.199154] watchdog: BUG: soft lockup - CPU#7 stuck for 36981s! [kworker/7:4:78808] [41982.249907] irq event stamp: 75265010 [41982.249909] hardirqs last enabled at (75265009): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41982.257637] Modules linked in: [41982.261295] hardirqs last disabled at (75265010): [] sysvec_apic_timer_interrupt+0xe/0x90 [41982.271452] ceph [41982.274504] softirqs last enabled at (75264984): [] __do_softirq+0x2be/0x371 [41982.284228] libceph [41982.286151] softirqs last disabled at (75264979): [] irq_exit_rcu+0xb0/0xd0 [41982.294835] dns_resolver fscache [41982.297021] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41982.305547] netfs [41982.308859] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41982.317804] veth nft_chain_nat [41982.319832] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [41982.327301] xt_MASQUERADE [41982.330439] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [41982.335917] nf_nat nf_conntrack [41982.338620] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [41982.357359] nf_defrag_ipv6 nf_defrag_ipv4 [41982.360582] [41982.365799] nft_compat nf_tables [41982.369891] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [41982.371382] nfnetlink bridge [41982.374691] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [41982.381831] stp [41982.384797] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [41982.391921] llc xfs [41982.393758] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [41982.400882] libcrc32c sunrpc [41982.403067] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [41982.410191] intel_rapl_msr intel_rapl_common [41982.413156] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [41982.420280] sb_edac [41982.424630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41982.432708] x86_pkg_temp_thermal [41982.434891] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [41982.440628] coretemp kvm_intel [41982.443939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41982.451063] kvm [41982.454200] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41982.461325] irqbypass i2c_i801 [41982.463164] Call Trace: [41982.463167] [41982.470288] iTCO_wdt iTCO_vendor_support [41982.473428] ? watchdog_timer_fn+0x1b4/0x220 [41982.475868] i2c_smbus crct10dif_pclmul [41982.477880] ? __hrtimer_run_queues+0x1d3/0x360 [41982.481883] joydev crc32_pclmul [41982.486151] ? hrtimer_interrupt+0x104/0x240 [41982.489978] ipmi_ssif [41982.494502] ? __local_bh_enable+0x50/0x60 [41982.497726] ghash_clmulni_intel mei_me [41982.501992] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41982.504347] mei lpc_ich [41982.508440] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41982.512268] sha512_ssse3 acpi_ipmi [41982.517486] [41982.520018] mfd_core wmi [41982.524975] [41982.528459] ipmi_si ipmi_devintf [41982.530556] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41982.533174] ipmi_msghandler ioatdma [41982.535276] ? queued_spin_lock_slowpath+0x16/0x2e0 [41982.538580] acpi_pad acpi_power_meter [41982.543888] do_raw_spin_lock+0xb5/0xc0 [41982.547456] ixgbe igb [41982.552328] __cap_delay_requeue+0xa7/0x1d0 [ceph] [41982.556072] mdio crc32c_intel [41982.559905] try_get_cap_refs+0x159/0x880 [ceph] [41982.562259] nvme ptp [41982.567050] __ceph_get_caps+0x4a1/0x760 [ceph] [41982.570094] nvme_core i2c_algo_bit [41982.574709] ? __file_remove_privs+0x89/0x110 [41982.576975] pps_core dca [41982.581501] ? wake_bit_function+0x70/0x70 [41982.584984] fuse [41982.589337] ? ceph_write_iter+0x260/0x770 [ceph] [41982.591953] irq event stamp: 255808418 [41982.596042] ceph_write_iter+0x260/0x770 [ceph] [41982.597966] hardirqs last enabled at (255808417): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41982.602668] ? lock_is_held_type+0xaa/0x120 [41982.606408] hardirqs last disabled at (255808418): [] sysvec_apic_timer_interrupt+0xe/0x90 [41982.610935] vfs_write+0x324/0x4f0 [41982.621177] softirqs last enabled at (255808416): [] __do_softirq+0x2be/0x371 [41982.625359] ksys_pwrite64+0x65/0xa0 [41982.635165] softirqs last disabled at (255808411): [] irq_exit_rcu+0xb0/0xd0 [41982.638562] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [41982.647334] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [41982.650905] do_syscall_64+0x38/0x80 [41982.659500] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [41982.664632] entry_SYSCALL_64_after_hwframe+0x63/0xcd [41982.674010] Workqueue: events delayed_work [ceph] [41982.677582] RIP: 0033:0x7f48f0eeb93b [41982.685052] [41982.685053] RIP: 0010:lock_acquire+0xf0/0x2b0 [41982.690096] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [41982.694793] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [41982.698364] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [41982.699853] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [41982.704204] ORIG_RAX: 0000000000000012 [41982.722944] [41982.722944] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [41982.741680] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [41982.746898] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [41982.752114] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [41982.755947] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [41982.757437] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [41982.764559] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [41982.771684] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [41982.778809] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [41982.785933] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [41982.793057] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [41982.800185] [41982.807304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [41982.844676] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [41982.851800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [41982.858925] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [41982.866050] Call Trace: [41982.868494] [41982.870505] ? watchdog_timer_fn+0x1b4/0x220 [41982.874769] ? __hrtimer_run_queues+0x1d3/0x360 [41982.879296] ? hrtimer_interrupt+0x104/0x240 [41982.883564] ? __local_bh_enable+0x50/0x60 [41982.887659] ? __sysvec_apic_timer_interrupt+0x61/0x160 [41982.892881] ? sysvec_apic_timer_interrupt+0x6d/0x90 [41982.897847] [41982.899946] [41982.902043] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [41982.907351] ? lock_acquire+0xf0/0x2b0 [41982.911101] ? igrab+0x43/0x50 [41982.914150] ? lock_release+0xc5/0x260 [41982.917896] _raw_spin_lock+0x30/0x40 [41982.921561] ? igrab+0x1d/0x50 [41982.924612] igrab+0x1d/0x50 [41982.927490] ceph_check_delayed_caps+0xb7/0x270 [ceph] [41982.932640] delayed_work+0x1d8/0x280 [ceph] [41982.936922] ? process_scheduled_works+0x22a/0x510 [41982.941710] process_scheduled_works+0x2a4/0x510 [41982.946326] worker_thread+0x188/0x340 [41982.950075] ? rescuer_thread+0x310/0x310 [41982.954079] kthread+0xf5/0x130 [41982.957218] ? kthread_complete_and_exit+0x20/0x20 [41982.962002] ret_from_fork+0x31/0x40 [41982.965581] ? kthread_complete_and_exit+0x20/0x20 [41982.970365] ret_from_fork_asm+0x11/0x20 [41982.974296] [42010.187634] watchdog: BUG: soft lockup - CPU#0 stuck for 28011s! [dbench:78683] [42010.194937] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42010.199634] watchdog: BUG: soft lockup - CPU#7 stuck for 37008s! [kworker/7:4:78808] [42010.250377] irq event stamp: 75323216 [42010.250379] hardirqs last enabled at (75323215): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42010.258109] Modules linked in: ceph [42010.261765] hardirqs last disabled at (75323216): [] sysvec_apic_timer_interrupt+0xe/0x90 [42010.271923] libceph [42010.275407] softirqs last enabled at (75323160): [] __do_softirq+0x2be/0x371 [42010.285132] dns_resolver fscache [42010.287314] softirqs last disabled at (75323155): [] irq_exit_rcu+0xb0/0xd0 [42010.296000] netfs veth [42010.299328] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42010.307838] nft_chain_nat xt_MASQUERADE [42010.310283] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42010.310284] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42010.319243] nf_nat [42010.323162] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42010.330631] nf_conntrack nf_defrag_ipv6 [42010.336110] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42010.338206] nf_defrag_ipv4 [42010.356945] [42010.356946] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42010.360861] nft_compat nf_tables [42010.366078] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42010.368870] nfnetlink bridge [42010.370361] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42010.377484] stp llc [42010.380796] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42010.387920] xfs [42010.390884] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42010.398008] libcrc32c sunrpc [42010.400209] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42010.407333] intel_rapl_msr intel_rapl_common [42010.409169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42010.416309] sb_edac x86_pkg_temp_thermal [42010.419276] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42010.427352] coretemp kvm_intel [42010.431704] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42010.437442] kvm irqbypass [42010.441445] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42010.448571] i2c_i801 iTCO_wdt [42010.451706] Call Trace: [42010.451708] [42010.458832] iTCO_vendor_support [42010.461536] ? watchdog_timer_fn+0x1b4/0x220 [42010.468659] i2c_smbus crct10dif_pclmul [42010.471712] ? __hrtimer_run_queues+0x1d3/0x360 [42010.474154] joydev crc32_pclmul [42010.476168] ? hrtimer_interrupt+0x104/0x240 [42010.479389] ipmi_ssif [42010.483653] ? __local_bh_enable+0x50/0x60 [42010.487484] ghash_clmulni_intel [42010.492029] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42010.495250] mei_me mei [42010.499514] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42010.501871] lpc_ich sha512_ssse3 [42010.505962] [42010.509186] acpi_ipmi mfd_core [42010.514403] [42010.516847] wmi ipmi_si [42010.521804] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42010.525115] ipmi_devintf ipmi_msghandler [42010.527217] ? queued_spin_lock_slowpath+0x18/0x2e0 [42010.530350] ioatdma acpi_pad [42010.532448] do_raw_spin_lock+0xb5/0xc0 [42010.534979] acpi_power_meter ixgbe [42010.540282] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42010.544285] igb mdio crc32c_intel [42010.549159] try_get_cap_refs+0x159/0x880 [ceph] [42010.552120] nvme ptp [42010.555958] __ceph_get_caps+0x4a1/0x760 [ceph] [42010.559435] nvme_core i2c_algo_bit [42010.564224] ? __file_remove_privs+0x89/0x110 [42010.567618] pps_core dca [42010.572230] ? wake_bit_function+0x70/0x70 [42010.574498] fuse [42010.579039] ? ceph_write_iter+0x260/0x770 [ceph] [42010.582525] [42010.582525] irq event stamp: 256029202 [42010.586876] ceph_write_iter+0x260/0x770 [ceph] [42010.589493] hardirqs last enabled at (256029201): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42010.593587] ? lock_is_held_type+0xaa/0x120 [42010.595508] hardirqs last disabled at (256029202): [] sysvec_apic_timer_interrupt+0xe/0x90 [42010.600208] vfs_write+0x324/0x4f0 [42010.601696] softirqs last enabled at (256029200): [] __do_softirq+0x2be/0x371 [42010.605443] ksys_pwrite64+0x65/0xa0 [42010.609965] softirqs last disabled at (256029195): [] irq_exit_rcu+0xb0/0xd0 [42010.620207] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42010.624385] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42010.634198] do_syscall_64+0x38/0x80 [42010.637592] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42010.637593] Workqueue: events delayed_work [ceph] [42010.646365] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42010.649934] [42010.649935] RIP: 0010:lock_release+0x14e/0x260 [42010.658535] RIP: 0033:0x7f48f0eeb93b [42010.663664] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [42010.673041] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42010.676612] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [42010.684083] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42010.688779] [42010.688780] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [42010.693824] ORIG_RAX: 0000000000000012 [42010.695318] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [42010.699752] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42010.703323] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [42010.722061] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42010.740798] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [42010.746033] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42010.751249] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [42010.752740] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42010.759865] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42010.763696] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42010.770819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42010.777950] [42010.785068] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42010.850954] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42010.858078] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42010.865201] Call Trace: [42010.867648] [42010.869657] ? watchdog_timer_fn+0x1b4/0x220 [42010.873922] ? __hrtimer_run_queues+0x1d3/0x360 [42010.878448] ? hrtimer_interrupt+0x104/0x240 [42010.882717] ? __local_bh_enable+0x50/0x60 [42010.886810] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42010.892040] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42010.897002] [42010.899097] [42010.901196] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42010.906501] ? igrab+0x43/0x50 [42010.909554] ? lock_release+0x14e/0x260 [42010.913391] ? lock_release+0xc5/0x260 [42010.917135] _raw_spin_unlock+0x1b/0x40 [42010.920966] igrab+0x43/0x50 [42010.923843] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42010.928993] delayed_work+0x1d8/0x280 [ceph] [42010.933273] ? process_scheduled_works+0x22a/0x510 [42010.938054] process_scheduled_works+0x2a4/0x510 [42010.942672] worker_thread+0x188/0x340 [42010.946419] ? rescuer_thread+0x310/0x310 [42010.950423] kthread+0xf5/0x130 [42010.953562] ? kthread_complete_and_exit+0x20/0x20 [42010.958353] ret_from_fork+0x31/0x40 [42010.961923] ? kthread_complete_and_exit+0x20/0x20 [42010.966707] ret_from_fork_asm+0x11/0x20 [42010.970630] [42038.188115] watchdog: BUG: soft lockup - CPU#0 stuck for 28037s! [dbench:78683] [42038.195424] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42038.200114] watchdog: BUG: soft lockup - CPU#7 stuck for 37034s! [kworker/7:4:78808] [42038.250873] irq event stamp: 75381428 [42038.258605] Modules linked in: ceph libceph dns_resolver fscache netfs [42038.262261] hardirqs last enabled at (75381427): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42038.268787] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [42038.278944] hardirqs last disabled at (75381428): [] sysvec_apic_timer_interrupt+0xe/0x90 [42038.286330] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [42038.296052] softirqs last enabled at (75381370): [] __do_softirq+0x2be/0x371 [42038.302484] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [42038.311186] softirqs last disabled at (75381365): [] irq_exit_rcu+0xb0/0xd0 [42038.318406] x86_pkg_temp_thermal coretemp [42038.326917] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42038.326918] kvm_intel [42038.326919] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42038.331007] kvm irqbypass [42038.339950] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42038.342307] i2c_i801 iTCO_wdt [42038.349779] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42038.352482] iTCO_vendor_support i2c_smbus [42038.357959] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42038.361012] crct10dif_pclmul joydev [42038.379748] [42038.379749] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42038.383839] crc32_pclmul [42038.389057] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42038.392627] ipmi_ssif ghash_clmulni_intel [42038.394118] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42038.401242] mei_me mei [42038.403859] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42038.410983] lpc_ich [42038.415073] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42038.422199] sha512_ssse3 acpi_ipmi [42038.424642] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42038.431767] mfd_core [42038.433969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42038.441092] wmi ipmi_si [42038.444577] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42038.452654] ipmi_devintf [42038.454926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42038.460661] ipmi_msghandler ioatdma [42038.463192] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42038.470318] acpi_pad acpi_power_meter [42038.472934] Call Trace: [42038.472936] [42038.480059] ixgbe [42038.483632] ? watchdog_timer_fn+0x1b4/0x220 [42038.490754] igb mdio [42038.494512] ? __hrtimer_run_queues+0x1d3/0x360 [42038.496960] crc32c_intel nvme [42038.498973] ? hrtimer_interrupt+0x104/0x240 [42038.500979] ptp nvme_core [42038.505243] ? __local_bh_enable+0x50/0x60 [42038.507515] i2c_algo_bit pps_core [42038.512040] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42038.515089] dca fuse [42038.519355] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42038.522059] [42038.522060] irq event stamp: 256250110 [42038.526149] [42038.529546] hardirqs last enabled at (256250109): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42038.534764] [42038.537035] hardirqs last disabled at (256250110): [] sysvec_apic_timer_interrupt+0xe/0x90 [42038.541993] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42038.543482] softirqs last enabled at (256250108): [] __do_softirq+0x2be/0x371 [42038.547229] ? queued_spin_lock_slowpath+0x18/0x2e0 [42038.549324] softirqs last disabled at (256250103): [] irq_exit_rcu+0xb0/0xd0 [42038.559570] do_raw_spin_lock+0xb5/0xc0 [42038.561667] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42038.571478] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42038.576780] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42038.585572] try_get_cap_refs+0x159/0x880 [ceph] [42038.590439] Workqueue: events delayed_work [ceph] [42038.599041] __ceph_get_caps+0x4a1/0x760 [ceph] [42038.602867] [42038.612250] ? __file_remove_privs+0x89/0x110 [42038.617029] RIP: 0010:lock_acquire+0xf0/0x2b0 [42038.624515] ? wake_bit_function+0x70/0x70 [42038.629127] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42038.633828] ? ceph_write_iter+0x260/0x770 [ceph] [42038.638350] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42038.639840] ceph_write_iter+0x260/0x770 [ceph] [42038.644191] [42038.644192] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42038.648546] ? lock_is_held_type+0xaa/0x120 [42038.652631] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42038.671372] vfs_write+0x324/0x4f0 [42038.676067] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42038.681290] ksys_pwrite64+0x65/0xa0 [42038.685810] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42038.687302] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42038.694422] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42038.698603] do_syscall_64+0x38/0x80 [42038.705725] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42038.709125] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42038.716246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42038.719820] RIP: 0033:0x7f48f0eeb93b [42038.726941] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42038.732073] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42038.739198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42038.742766] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42038.750845] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42038.755889] ORIG_RAX: 0000000000000012 [42038.761626] Call Trace: [42038.761627] [42038.765197] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42038.772322] ? watchdog_timer_fn+0x1b4/0x220 [42038.791060] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42038.798184] ? __hrtimer_run_queues+0x1d3/0x360 [42038.803401] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42038.810528] ? hrtimer_interrupt+0x104/0x240 [42038.814355] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42038.816804] ? __local_bh_enable+0x50/0x60 [42038.818820] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42038.825945] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42038.830212] [42038.837332] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42038.883943] [42038.886039] [42038.888136] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42038.893444] ? lock_acquire+0xf0/0x2b0 [42038.897194] ? igrab+0x43/0x50 [42038.900244] ? lock_release+0xc5/0x260 [42038.903989] _raw_spin_lock+0x30/0x40 [42038.907645] ? igrab+0x1d/0x50 [42038.910696] igrab+0x1d/0x50 [42038.913575] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42038.918725] delayed_work+0x1d8/0x280 [ceph] [42038.923006] ? process_scheduled_works+0x22a/0x510 [42038.927807] process_scheduled_works+0x2a4/0x510 [42038.932428] worker_thread+0x188/0x340 [42038.936177] ? rescuer_thread+0x310/0x310 [42038.940180] kthread+0xf5/0x130 [42038.943320] ? kthread_complete_and_exit+0x20/0x20 [42038.948111] ret_from_fork+0x31/0x40 [42038.951691] ? kthread_complete_and_exit+0x20/0x20 [42038.956473] ret_from_fork_asm+0x11/0x20 [42038.960397] [42066.188601] watchdog: BUG: soft lockup - CPU#0 stuck for 28063s! [dbench:78683] [42066.195903] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42066.200601] watchdog: BUG: soft lockup - CPU#7 stuck for 37060s! [kworker/7:4:78808] [42066.251343] irq event stamp: 75439576 [42066.251344] hardirqs last enabled at (75439575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42066.259074] Modules linked in: [42066.262730] hardirqs last disabled at (75439576): [] sysvec_apic_timer_interrupt+0xe/0x90 [42066.272890] ceph [42066.275939] softirqs last enabled at (75439566): [] __do_softirq+0x2be/0x371 [42066.285663] libceph [42066.287587] softirqs last disabled at (75439561): [] irq_exit_rcu+0xb0/0xd0 [42066.296276] dns_resolver fscache [42066.298455] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42066.306967] netfs veth [42066.310292] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42066.319239] nft_chain_nat [42066.321683] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42066.329153] xt_MASQUERADE nf_nat [42066.331858] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42066.337334] nf_conntrack nf_defrag_ipv6 [42066.340646] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42066.359385] nf_defrag_ipv4 nft_compat [42066.363302] [42066.363303] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42066.368517] nf_tables nfnetlink [42066.372264] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42066.373772] bridge stp [42066.380894] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42066.384117] llc xfs [42066.391242] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42066.393686] libcrc32c sunrpc [42066.400812] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42066.402997] intel_rapl_msr [42066.410120] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42066.413082] intel_rapl_common [42066.420208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42066.422997] sb_edac [42066.431075] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42066.434126] x86_pkg_temp_thermal [42066.439864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42066.442048] coretemp [42066.449172] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42066.452482] kvm_intel kvm [42066.459606] Call Trace: [42066.459607] [42066.461876] irqbypass i2c_i801 [42066.469003] ? watchdog_timer_fn+0x1b4/0x220 [42066.471707] iTCO_wdt iTCO_vendor_support [42066.474151] ? __hrtimer_run_queues+0x1d3/0x360 [42066.476161] i2c_smbus [42066.479301] ? hrtimer_interrupt+0x104/0x240 [42066.483561] crct10dif_pclmul joydev [42066.487567] ? __local_bh_enable+0x50/0x60 [42066.492090] crc32_pclmul ipmi_ssif [42066.494451] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42066.498712] ghash_clmulni_intel mei_me [42066.502283] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42066.506372] mei [42066.509857] [42066.515074] lpc_ich sha512_ssse3 [42066.518904] [42066.518906] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42066.523863] acpi_ipmi mfd_core [42066.525703] ? queued_spin_lock_slowpath+0x16/0x2e0 [42066.527797] wmi ipmi_si [42066.531111] do_raw_spin_lock+0xb5/0xc0 [42066.533205] ipmi_devintf ipmi_msghandler [42066.538510] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42066.541647] ioatdma acpi_pad [42066.546520] try_get_cap_refs+0x159/0x880 [ceph] [42066.549049] acpi_power_meter ixgbe [42066.552883] __ceph_get_caps+0x4a1/0x760 [ceph] [42066.556882] igb mdio [42066.561670] ? __file_remove_privs+0x89/0x110 [42066.564632] crc32c_intel nvme [42066.569242] ? wake_bit_function+0x70/0x70 [42066.572727] ptp nvme_core [42066.577253] ? ceph_write_iter+0x260/0x770 [ceph] [42066.579522] i2c_algo_bit pps_core [42066.583872] ceph_write_iter+0x260/0x770 [ceph] [42066.586923] dca fuse [42066.591016] ? lock_is_held_type+0xaa/0x120 [42066.593718] [42066.593719] irq event stamp: 256471036 [42066.598416] vfs_write+0x324/0x4f0 [42066.601813] hardirqs last enabled at (256471035): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42066.606341] ksys_pwrite64+0x65/0xa0 [42066.608607] hardirqs last disabled at (256471036): [] sysvec_apic_timer_interrupt+0xe/0x90 [42066.612785] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42066.614280] softirqs last enabled at (256471034): [] __do_softirq+0x2be/0x371 [42066.618030] do_syscall_64+0x38/0x80 [42066.621424] softirqs last disabled at (256471029): [] irq_exit_rcu+0xb0/0xd0 [42066.631671] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42066.635241] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42066.645050] RIP: 0033:0x7f48f0eeb93b [42066.650183] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42066.658954] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42066.662524] Workqueue: events delayed_work [ceph] [42066.671121] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42066.676164] [42066.676165] RIP: 0010:lock_acquire+0xf0/0x2b0 [42066.685542] ORIG_RAX: 0000000000000012 [42066.689113] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42066.696585] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42066.715322] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42066.720018] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42066.725236] [42066.725237] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42066.726727] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42066.731077] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42066.734908] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42066.753647] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42066.760771] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42066.766005] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42066.773136] [42066.774619] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42066.833806] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42066.841883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42066.847620] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42066.854745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42066.861869] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42066.869005] Call Trace: [42066.871445] [42066.873458] ? watchdog_timer_fn+0x1b4/0x220 [42066.877722] ? __hrtimer_run_queues+0x1d3/0x360 [42066.882249] ? hrtimer_interrupt+0x104/0x240 [42066.886518] ? __local_bh_enable+0x50/0x60 [42066.890611] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42066.895836] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42066.900791] [42066.902888] [42066.905005] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42066.910312] ? lock_acquire+0xf0/0x2b0 [42066.914062] ? igrab+0x43/0x50 [42066.917113] ? lock_release+0xc5/0x260 [42066.920856] _raw_spin_lock+0x30/0x40 [42066.924512] ? igrab+0x1d/0x50 [42066.927565] igrab+0x1d/0x50 [42066.930443] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42066.935591] delayed_work+0x1d8/0x280 [ceph] [42066.939873] ? process_scheduled_works+0x22a/0x510 [42066.944664] process_scheduled_works+0x2a4/0x510 [42066.949292] worker_thread+0x188/0x340 [42066.953035] ? rescuer_thread+0x310/0x310 [42066.957039] kthread+0xf5/0x130 [42066.960178] ? kthread_complete_and_exit+0x20/0x20 [42066.964961] ret_from_fork+0x31/0x40 [42066.968532] ? kthread_complete_and_exit+0x20/0x20 [42066.973316] ret_from_fork_asm+0x11/0x20 [42066.977240] [42094.189088] watchdog: BUG: soft lockup - CPU#0 stuck for 28089s! [dbench:78683] [42094.196389] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42094.201088] watchdog: BUG: soft lockup - CPU#7 stuck for 37086s! [kworker/7:4:78808] [42094.251830] irq event stamp: 75497876 [42094.251832] hardirqs last enabled at (75497875): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42094.259562] Modules linked in: ceph [42094.263217] hardirqs last disabled at (75497876): [] sysvec_apic_timer_interrupt+0xe/0x90 [42094.273377] libceph [42094.276860] softirqs last enabled at (75497818): [] __do_softirq+0x2be/0x371 [42094.286585] dns_resolver [42094.288768] softirqs last disabled at (75497813): [] irq_exit_rcu+0xb0/0xd0 [42094.297453] fscache netfs [42094.300070] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42094.308580] veth nft_chain_nat [42094.311285] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42094.320229] xt_MASQUERADE nf_nat [42094.323367] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42094.330837] nf_conntrack [42094.334149] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42094.339624] nf_defrag_ipv6 nf_defrag_ipv4 [42094.342243] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42094.360979] nft_compat nf_tables [42094.365071] [42094.365072] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42094.370289] nfnetlink [42094.373599] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42094.375090] bridge stp [42094.382215] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42094.384572] llc xfs [42094.391695] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42094.394140] libcrc32c sunrpc [42094.401262] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42094.403449] intel_rapl_msr intel_rapl_common [42094.410573] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42094.413537] sb_edac [42094.420661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42094.425011] x86_pkg_temp_thermal coretemp [42094.433089] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42094.435273] kvm_intel kvm [42094.441011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42094.445099] irqbypass i2c_i801 [42094.452225] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42094.454928] iTCO_wdt iTCO_vendor_support [42094.462051] Call Trace: [42094.462053] [42094.465191] i2c_smbus crct10dif_pclmul [42094.472315] ? watchdog_timer_fn+0x1b4/0x220 [42094.476318] joydev crc32_pclmul [42094.478770] ? __hrtimer_run_queues+0x1d3/0x360 [42094.480788] ipmi_ssif ghash_clmulni_intel [42094.484625] ? hrtimer_interrupt+0x104/0x240 [42094.488884] mei_me mei [42094.492108] ? __local_bh_enable+0x50/0x60 [42094.496633] lpc_ich sha512_ssse3 [42094.500727] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42094.504987] acpi_ipmi mfd_core [42094.507433] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42094.511524] wmi [42094.514835] [42094.520051] ipmi_si ipmi_devintf [42094.523190] [42094.523191] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42094.528145] ipmi_msghandler ioatdma [42094.529988] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42094.532081] acpi_pad [42094.535392] do_raw_spin_lock+0xb5/0xc0 [42094.537488] acpi_power_meter [42094.542794] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42094.546363] ixgbe igb mdio [42094.551238] try_get_cap_refs+0x159/0x880 [ceph] [42094.553504] crc32c_intel nvme [42094.557341] __ceph_get_caps+0x4a1/0x760 [ceph] [42094.560299] ptp [42094.565088] ? __file_remove_privs+0x89/0x110 [42094.567875] nvme_core i2c_algo_bit [42094.572490] ? wake_bit_function+0x70/0x70 [42094.575536] pps_core dca [42094.580062] ? ceph_write_iter+0x260/0x770 [ceph] [42094.581897] fuse [42094.586248] ceph_write_iter+0x260/0x770 [ceph] [42094.589733] irq event stamp: 256691794 [42094.593826] ? lock_is_held_type+0xaa/0x120 [42094.596442] hardirqs last enabled at (256691793): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42094.601141] vfs_write+0x324/0x4f0 [42094.603062] hardirqs last disabled at (256691794): [] sysvec_apic_timer_interrupt+0xe/0x90 [42094.607591] ksys_pwrite64+0x65/0xa0 [42094.611329] softirqs last enabled at (256691792): [] __do_softirq+0x2be/0x371 [42094.615507] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42094.625752] softirqs last disabled at (256691787): [] irq_exit_rcu+0xb0/0xd0 [42094.629150] do_syscall_64+0x38/0x80 [42094.638961] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42094.642532] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42094.651302] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42094.651303] Workqueue: events delayed_work [ceph] [42094.656432] RIP: 0033:0x7f48f0eeb93b [42094.665031] [42094.665032] RIP: 0010:ceph_check_delayed_caps+0xa1/0x270 [ceph] [42094.668601] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42094.677977] Code: e0 04 75 0c 48 8b 0d 2e 19 8d e1 48 39 d1 78 47 48 89 df e8 e1 eb aa e0 84 c0 74 0e 48 8b 13 48 8b 43 08 48 89 42 08 48 89 10 <48> 89 1b 48 89 5b 08 48 81 eb 50 07 00 00 48 89 df e8 e9 fa 87 e0 [42094.683021] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42094.690494] RSP: 0018:ffffc90002117db8 EFLAGS: 00000202 [42094.695190] ORIG_RAX: 0000000000000012 [42094.698771] [42094.698772] RAX: ffff8881a1d34bc8 RBX: ffff8881a1d34bc8 RCX: ffff8881a1d34bc8 [42094.700262] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42094.706170] RDX: ffff8881a1d34bc8 RSI: ffff8881a1d34bc8 RDI: ffff8881a1d34bc8 [42094.724908] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42094.743646] RBP: ffff88817f867000 R08: 0000000000005c53 R09: 0000000000000003 [42094.748864] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42094.754082] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [42094.757911] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42094.759402] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [42094.766528] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42094.773652] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42094.780796] [42094.787917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42094.846660] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42094.853785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42094.860909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42094.868034] Call Trace: [42094.870495] [42094.872505] ? watchdog_timer_fn+0x1b4/0x220 [42094.876775] ? __hrtimer_run_queues+0x1d3/0x360 [42094.881305] ? hrtimer_interrupt+0x104/0x240 [42094.885567] ? __local_bh_enable+0x50/0x60 [42094.889660] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42094.894883] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42094.899841] [42094.901937] [42094.904035] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42094.909343] ? ceph_check_delayed_caps+0xa1/0x270 [ceph] [42094.914662] ? ceph_check_delayed_caps+0x8f/0x270 [ceph] [42094.919975] delayed_work+0x1d8/0x280 [ceph] [42094.924257] ? process_scheduled_works+0x22a/0x510 [42094.929047] process_scheduled_works+0x2a4/0x510 [42094.933662] worker_thread+0x188/0x340 [42094.937412] ? rescuer_thread+0x310/0x310 [42094.941416] kthread+0xf5/0x130 [42094.944552] ? kthread_complete_and_exit+0x20/0x20 [42094.949338] ret_from_fork+0x31/0x40 [42094.952908] ? kthread_complete_and_exit+0x20/0x20 [42094.957692] ret_from_fork_asm+0x11/0x20 [42094.961615] [42095.238473] rcu: INFO: rcu_preempt self-detected stall on CPU [42095.244226] rcu: 7-....: (29280688 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/29279307 fqs=14766091 [42095.254913] rcu: (t=30158102 jiffies g=1309609 q=806091 ncpus=8) [42095.260996] Sending NMI from CPU 7 to CPUs 0: [42095.265349] NMI backtrace for cpu 0 [42095.265351] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42095.265352] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42095.265353] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42095.265356] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42095.265357] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42095.265358] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42095.265359] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42095.265360] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42095.265361] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42095.265361] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42095.265362] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42095.265363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42095.265364] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42095.265365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42095.265365] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42095.265366] Call Trace: [42095.265367] [42095.265368] ? nmi_cpu_backtrace+0xad/0x120 [42095.265371] ? nmi_cpu_backtrace_handler+0x11/0x20 [42095.265373] ? nmi_handle+0xd4/0x260 [42095.265377] ? default_do_nmi+0x49/0xf0 [42095.265379] ? exc_nmi+0x11e/0x150 [42095.265380] ? end_repeat_nmi+0x16/0x67 [42095.265385] ? queued_spin_lock_slowpath+0x18/0x2e0 [42095.265387] ? queued_spin_lock_slowpath+0x18/0x2e0 [42095.265390] ? queued_spin_lock_slowpath+0x18/0x2e0 [42095.265392] [42095.265392] [42095.265393] do_raw_spin_lock+0xb5/0xc0 [42095.265396] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42095.265410] try_get_cap_refs+0x159/0x880 [ceph] [42095.265424] __ceph_get_caps+0x4a1/0x760 [ceph] [42095.265437] ? __file_remove_privs+0x89/0x110 [42095.265439] ? wake_bit_function+0x70/0x70 [42095.265441] ? ceph_write_iter+0x260/0x770 [ceph] [42095.265452] ceph_write_iter+0x260/0x770 [ceph] [42095.265463] ? lock_is_held_type+0xaa/0x120 [42095.265466] vfs_write+0x324/0x4f0 [42095.265470] ksys_pwrite64+0x65/0xa0 [42095.265473] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42095.265477] do_syscall_64+0x38/0x80 [42095.265479] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42095.265481] RIP: 0033:0x7f48f0eeb93b [42095.265482] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42095.265483] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [42095.265485] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42095.265486] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42095.265486] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42095.265487] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42095.265488] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42095.265491] [42095.266347] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42095.569839] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42095.577308] Workqueue: events delayed_work [ceph] [42095.582023] RIP: 0010:lock_acquire+0xf0/0x2b0 [42095.586375] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42095.605112] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42095.610331] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42095.617453] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42095.624576] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42095.631702] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42095.638825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42095.645951] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42095.654026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42095.659765] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42095.666890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42095.674012] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42095.681137] Call Trace: [42095.683584] [42095.685594] ? rcu_dump_cpu_stacks+0xd1/0x180 [42095.689953] ? rcu_sched_clock_irq+0x633/0x1390 [42095.694477] ? lock_release+0xc5/0x260 [42095.698221] ? lock_is_held_type+0xaa/0x120 [42095.702398] ? lock_is_held_type+0xaa/0x120 [42095.706575] ? tick_sched_do_timer+0xa0/0xa0 [42095.710848] ? rcu_read_lock_sched_held+0x56/0x90 [42095.715546] ? tick_sched_do_timer+0xa0/0xa0 [42095.719809] ? update_process_times+0x5f/0x90 [42095.724161] ? tick_sched_handle.isra.16+0x1e/0x50 [42095.728953] ? tick_sched_timer+0x6f/0x80 [42095.732956] ? __hrtimer_run_queues+0x1d3/0x360 [42095.737487] ? hrtimer_interrupt+0x104/0x240 [42095.741753] ? __local_bh_enable+0x50/0x60 [42095.745846] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42095.751062] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42095.756018] [42095.758117] [42095.760214] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42095.765521] ? lock_acquire+0xf0/0x2b0 [42095.769272] ? igrab+0x43/0x50 [42095.772322] ? lock_release+0xc5/0x260 [42095.776066] _raw_spin_lock+0x30/0x40 [42095.779723] ? igrab+0x1d/0x50 [42095.782774] igrab+0x1d/0x50 [42095.785652] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42095.790803] delayed_work+0x1d8/0x280 [ceph] [42095.795084] ? process_scheduled_works+0x22a/0x510 [42095.799872] process_scheduled_works+0x2a4/0x510 [42095.804489] worker_thread+0x188/0x340 [42095.808239] ? rescuer_thread+0x310/0x310 [42095.812249] kthread+0xf5/0x130 [42095.815387] ? kthread_complete_and_exit+0x20/0x20 [42095.820171] ret_from_fork+0x31/0x40 [42095.823742] ? kthread_complete_and_exit+0x20/0x20 [42095.828525] ret_from_fork_asm+0x11/0x20 [42095.832449] [42122.189575] watchdog: BUG: soft lockup - CPU#0 stuck for 28115s! [dbench:78683] [42122.196879] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42122.201574] watchdog: BUG: soft lockup - CPU#7 stuck for 37112s! [kworker/7:4:78808] [42122.252328] irq event stamp: 75556098 [42122.260058] Modules linked in: ceph libceph dns_resolver fscache netfs veth [42122.263714] hardirqs last enabled at (75556097): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42122.270675] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [42122.280831] hardirqs last disabled at (75556098): [] sysvec_apic_timer_interrupt+0xe/0x90 [42122.289083] nft_compat nf_tables nfnetlink bridge stp llc xfs [42122.298807] softirqs last enabled at (75556046): [] __do_softirq+0x2be/0x371 [42122.304639] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [42122.313322] softirqs last disabled at (75556041): [] irq_exit_rcu+0xb0/0xd0 [42122.322440] kvm_intel kvm irqbypass i2c_i801 iTCO_wdt [42122.330970] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42122.336100] iTCO_vendor_support i2c_smbus [42122.345044] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42122.345046] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42122.349134] crct10dif_pclmul joydev [42122.356608] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42122.362083] crc32_pclmul ipmi_ssif [42122.365654] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42122.384390] ghash_clmulni_intel [42122.387876] [42122.387877] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42122.393092] mei_me mei [42122.396318] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42122.397809] lpc_ich sha512_ssse3 [42122.404933] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42122.407376] acpi_ipmi [42122.414499] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42122.417812] mfd_core wmi [42122.424935] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42122.427293] ipmi_si ipmi_devintf [42122.434417] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42122.437033] ipmi_msghandler ioatdma [42122.444159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42122.447470] acpi_pad [42122.455545] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42122.459116] acpi_power_meter ixgbe [42122.464853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42122.467125] igb mdio [42122.474260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42122.477742] crc32c_intel nvme [42122.484865] Call Trace: [42122.484867] [42122.487136] ptp nvme_core [42122.494264] ? watchdog_timer_fn+0x1b4/0x220 [42122.497312] i2c_algo_bit pps_core [42122.499757] ? __hrtimer_run_queues+0x1d3/0x360 [42122.501767] dca fuse [42122.504473] ? hrtimer_interrupt+0x104/0x240 [42122.508733] [42122.508734] irq event stamp: 256907852 [42122.512131] ? __local_bh_enable+0x50/0x60 [42122.516657] hardirqs last enabled at (256907851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42122.518929] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42122.523191] hardirqs last disabled at (256907852): [] sysvec_apic_timer_interrupt+0xe/0x90 [42122.524682] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42122.528426] softirqs last enabled at (256907850): [] __do_softirq+0x2be/0x371 [42122.532517] [42122.542760] softirqs last disabled at (256907845): [] irq_exit_rcu+0xb0/0xd0 [42122.547978] [42122.557789] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42122.562747] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42122.571516] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42122.571518] Workqueue: events delayed_work [ceph] [42122.573618] ? queued_spin_lock_slowpath+0x16/0x2e0 [42122.582211] [42122.582212] RIP: 0010:lock_acquire+0xf0/0x2b0 [42122.584311] do_raw_spin_lock+0xb5/0xc0 [42122.593687] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42122.598992] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42122.606461] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42122.611162] try_get_cap_refs+0x159/0x880 [ceph] [42122.616029] [42122.616030] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42122.617526] __ceph_get_caps+0x4a1/0x760 [ceph] [42122.621871] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42122.625705] ? __file_remove_privs+0x89/0x110 [42122.644438] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42122.649223] ? wake_bit_function+0x70/0x70 [42122.654442] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42122.659055] ? ceph_write_iter+0x260/0x770 [ceph] [42122.660543] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42122.667666] ceph_write_iter+0x260/0x770 [ceph] [42122.672189] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42122.679319] ? lock_is_held_type+0xaa/0x120 [42122.683666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42122.690792] vfs_write+0x324/0x4f0 [42122.694880] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42122.702011] ksys_pwrite64+0x65/0xa0 [42122.706703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42122.713827] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42122.718349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42122.726430] do_syscall_64+0x38/0x80 [42122.730605] Call Trace: [42122.736343] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42122.739740] [42122.746865] RIP: 0033:0x7f48f0eeb93b [42122.750436] ? watchdog_timer_fn+0x1b4/0x220 [42122.757560] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42122.762691] ? __hrtimer_run_queues+0x1d3/0x360 [42122.769815] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42122.773387] ? hrtimer_interrupt+0x104/0x240 [42122.775828] ORIG_RAX: 0000000000000012 [42122.780873] ? __local_bh_enable+0x50/0x60 [42122.782884] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42122.786457] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42122.790717] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42122.809457] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42122.813981] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42122.819196] [42122.823462] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42122.827291] [42122.827293] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42122.831383] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42122.838511] ? lock_acquire+0xf0/0x2b0 [42122.843730] [42122.850850] ? igrab+0x43/0x50 [42122.895667] ? lock_release+0xc5/0x260 [42122.899412] _raw_spin_lock+0x30/0x40 [42122.903074] ? igrab+0x1d/0x50 [42122.906126] igrab+0x1d/0x50 [42122.909005] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42122.914154] delayed_work+0x1d8/0x280 [ceph] [42122.918436] ? process_scheduled_works+0x22a/0x510 [42122.923227] process_scheduled_works+0x2a4/0x510 [42122.927841] worker_thread+0x188/0x340 [42122.931590] ? rescuer_thread+0x310/0x310 [42122.935593] kthread+0xf5/0x130 [42122.938734] ? kthread_complete_and_exit+0x20/0x20 [42122.943525] ret_from_fork+0x31/0x40 [42122.947103] ? kthread_complete_and_exit+0x20/0x20 [42122.951887] ret_from_fork_asm+0x11/0x20 [42122.955809] [42150.190059] watchdog: BUG: soft lockup - CPU#0 stuck for 28141s! [dbench:78683] [42150.197363] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42150.202059] watchdog: BUG: soft lockup - CPU#7 stuck for 37138s! [kworker/7:4:78808] [42150.252805] irq event stamp: 75614252 [42150.252806] hardirqs last enabled at (75614251): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42150.260537] Modules linked in: [42150.264193] hardirqs last disabled at (75614252): [] sysvec_apic_timer_interrupt+0xe/0x90 [42150.274350] ceph [42150.277400] softirqs last enabled at (75614204): [] __do_softirq+0x2be/0x371 [42150.287126] libceph [42150.289049] softirqs last disabled at (75614199): [] irq_exit_rcu+0xb0/0xd0 [42150.297743] dns_resolver fscache [42150.299927] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42150.308454] netfs [42150.311766] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42150.311768] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42150.320710] veth nft_chain_nat [42150.322735] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42150.330208] xt_MASQUERADE nf_nat [42150.335687] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42150.338824] nf_conntrack nf_defrag_ipv6 [42150.357560] [42150.357561] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42150.360872] nf_defrag_ipv4 [42150.366088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42150.370007] nft_compat [42150.371497] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42150.378621] nf_tables [42150.381412] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42150.388536] nfnetlink bridge [42150.390981] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42150.398105] stp llc [42150.400461] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42150.407585] xfs libcrc32c [42150.410549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42150.417675] sunrpc intel_rapl_msr [42150.419858] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42150.427935] intel_rapl_common sb_edac [42150.430639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42150.436377] x86_pkg_temp_thermal coretemp [42150.439773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42150.446900] kvm_intel kvm [42150.450642] Call Trace: [42150.450643] [42150.457766] irqbypass [42150.461859] ? watchdog_timer_fn+0x1b4/0x220 [42150.468981] i2c_i801 iTCO_wdt [42150.471688] ? __hrtimer_run_queues+0x1d3/0x360 [42150.474131] iTCO_vendor_support i2c_smbus [42150.476144] ? hrtimer_interrupt+0x104/0x240 [42150.478498] crct10dif_pclmul joydev [42150.482763] ? __local_bh_enable+0x50/0x60 [42150.485813] crc32_pclmul ipmi_ssif [42150.490340] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42150.494427] ghash_clmulni_intel [42150.498692] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42150.502263] mei_me mei [42150.506354] [42150.509836] lpc_ich sha512_ssse3 [42150.515055] [42150.518279] acpi_ipmi mfd_core [42150.523236] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42150.525680] wmi ipmi_si [42150.527782] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42150.531089] ipmi_devintf ipmi_msghandler [42150.533187] do_raw_spin_lock+0xb5/0xc0 [42150.536322] ioatdma acpi_pad [42150.541628] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42150.544158] acpi_power_meter ixgbe [42150.549031] try_get_cap_refs+0x159/0x880 [ceph] [42150.553031] igb mdio [42150.556868] __ceph_get_caps+0x4a1/0x760 [ceph] [42150.559828] crc32c_intel nvme [42150.564615] ? __file_remove_privs+0x89/0x110 [42150.568095] ptp [42150.572706] ? wake_bit_function+0x70/0x70 [42150.574977] nvme_core i2c_algo_bit [42150.579504] ? ceph_write_iter+0x260/0x770 [ceph] [42150.582552] pps_core dca [42150.586904] ceph_write_iter+0x260/0x770 [ceph] [42150.588739] fuse [42150.592835] ? lock_is_held_type+0xaa/0x120 [42150.596314] [42150.596315] irq event stamp: 257128766 [42150.601016] vfs_write+0x324/0x4f0 [42150.603630] hardirqs last enabled at (257128765): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42150.608157] ksys_pwrite64+0x65/0xa0 [42150.610077] hardirqs last disabled at (257128766): [] sysvec_apic_timer_interrupt+0xe/0x90 [42150.614255] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42150.615746] softirqs last enabled at (257128764): [] __do_softirq+0x2be/0x371 [42150.619490] do_syscall_64+0x38/0x80 [42150.622888] softirqs last disabled at (257128759): [] irq_exit_rcu+0xb0/0xd0 [42150.633133] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42150.636702] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42150.646514] RIP: 0033:0x7f48f0eeb93b [42150.651644] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42150.651646] Workqueue: events delayed_work [ceph] [42150.660414] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42150.663985] [42150.663986] RIP: 0010:lock_acquire+0x85/0x2b0 [42150.672583] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42150.677628] Code: 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 8b 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c <58> 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 a6 d9 ea 7e 6a [42150.687004] ORIG_RAX: 0000000000000012 [42150.690574] RSP: 0018:ffffc90002117d10 EFLAGS: 00000246 [42150.698045] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42150.702742] [42150.702743] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 0000000000000000 [42150.721482] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42150.722972] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [42150.727322] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42150.732539] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [42150.751277] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42150.755110] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [42150.760326] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42150.767461] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42150.768955] [42150.776072] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42150.843344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42150.849083] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42150.856207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42150.863330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42150.870462] Call Trace: [42150.872909] [42150.874919] ? watchdog_timer_fn+0x1b4/0x220 [42150.879184] ? __hrtimer_run_queues+0x1d3/0x360 [42150.883712] ? hrtimer_interrupt+0x104/0x240 [42150.887979] ? __local_bh_enable+0x50/0x60 [42150.892073] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42150.897296] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42150.902255] [42150.904352] [42150.906466] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42150.911774] ? lock_acquire+0x85/0x2b0 [42150.915524] ? igrab+0x43/0x50 [42150.918574] ? lock_release+0xc5/0x260 [42150.922320] _raw_spin_lock+0x30/0x40 [42150.925984] ? igrab+0x1d/0x50 [42150.929034] igrab+0x1d/0x50 [42150.931912] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42150.937062] delayed_work+0x1d8/0x280 [ceph] [42150.941343] ? process_scheduled_works+0x22a/0x510 [42150.946124] process_scheduled_works+0x2a4/0x510 [42150.950749] worker_thread+0x188/0x340 [42150.954499] ? rescuer_thread+0x310/0x310 [42150.958502] kthread+0xf5/0x130 [42150.961641] ? kthread_complete_and_exit+0x20/0x20 [42150.966433] ret_from_fork+0x31/0x40 [42150.970003] ? kthread_complete_and_exit+0x20/0x20 [42150.974787] ret_from_fork_asm+0x11/0x20 [42150.978710] [42178.190545] watchdog: BUG: soft lockup - CPU#0 stuck for 28167s! [dbench:78683] [42178.197853] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42178.202544] watchdog: BUG: soft lockup - CPU#7 stuck for 37164s! [kworker/7:4:78808] [42178.253303] irq event stamp: 75672514 [42178.261031] Modules linked in: ceph libceph dns_resolver fscache netfs [42178.264689] hardirqs last enabled at (75672513): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42178.264694] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [42178.281373] hardirqs last disabled at (75672514): [] sysvec_apic_timer_interrupt+0xe/0x90 [42178.281376] nft_compat [42178.281378] softirqs last enabled at (75672466): [] __do_softirq+0x2be/0x371 [42178.290058] nf_tables nfnetlink [42178.299780] softirqs last disabled at (75672461): [] irq_exit_rcu+0xb0/0xd0 [42178.302241] bridge stp [42178.310942] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42178.314168] llc xfs [42178.322680] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42178.325124] libcrc32c sunrpc [42178.334067] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42178.336252] intel_rapl_msr intel_rapl_common [42178.343723] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42178.346686] sb_edac x86_pkg_temp_thermal [42178.352164] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42178.356514] coretemp kvm_intel [42178.375252] [42178.375253] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42178.379256] kvm irqbypass [42178.384473] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42178.387610] i2c_i801 [42178.389102] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42178.396227] iTCO_wdt [42178.398942] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42178.406062] iTCO_vendor_support [42178.408333] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42178.415459] i2c_smbus [42178.417728] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42178.424853] crct10dif_pclmul [42178.428077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42178.435199] joydev crc32_pclmul [42178.437558] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42178.445636] ipmi_ssif ghash_clmulni_intel [42178.448600] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42178.454338] mei_me [42178.457560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42178.464685] mei lpc_ich [42178.468777] Call Trace: [42178.468779] [42178.475901] sha512_ssse3 acpi_ipmi [42178.478000] ? watchdog_timer_fn+0x1b4/0x220 [42178.485121] mfd_core wmi [42178.487654] ? __hrtimer_run_queues+0x1d3/0x360 [42178.490096] ipmi_si [42178.492110] ? hrtimer_interrupt+0x104/0x240 [42178.495590] ipmi_devintf ipmi_msghandler [42178.499855] ? __local_bh_enable+0x50/0x60 [42178.502474] ioatdma acpi_pad [42178.507000] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42178.509181] acpi_power_meter ixgbe [42178.513446] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42178.517449] igb mdio [42178.521540] [42178.524503] crc32c_intel nvme [42178.529721] [42178.533205] ptp [42178.538180] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42178.540451] nvme_core i2c_algo_bit [42178.542552] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42178.545597] pps_core [42178.547697] do_raw_spin_lock+0xb5/0xc0 [42178.549533] dca fuse [42178.554838] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42178.558321] [42178.558322] irq event stamp: 257349548 [42178.563195] try_get_cap_refs+0x159/0x880 [ceph] [42178.565463] hardirqs last enabled at (257349547): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42178.569300] __ceph_get_caps+0x4a1/0x760 [ceph] [42178.571563] hardirqs last disabled at (257349548): [] sysvec_apic_timer_interrupt+0xe/0x90 [42178.576354] ? __file_remove_privs+0x89/0x110 [42178.577840] softirqs last enabled at (257349546): [] __do_softirq+0x2be/0x371 [42178.581583] ? wake_bit_function+0x70/0x70 [42178.586194] softirqs last disabled at (257349541): [] irq_exit_rcu+0xb0/0xd0 [42178.596440] ? ceph_write_iter+0x260/0x770 [ceph] [42178.600963] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42178.610773] ceph_write_iter+0x260/0x770 [ceph] [42178.615124] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42178.623899] ? lock_is_held_type+0xaa/0x120 [42178.627986] Workqueue: events delayed_work [ceph] [42178.636585] vfs_write+0x324/0x4f0 [42178.641281] [42178.641282] RIP: 0010:lock_acquire+0xf0/0x2b0 [42178.650663] ksys_pwrite64+0x65/0xa0 [42178.655183] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42178.662653] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42178.666831] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42178.671529] do_syscall_64+0x38/0x80 [42178.674924] [42178.674925] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42178.676435] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42178.680783] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42178.684355] RIP: 0033:0x7f48f0eeb93b [42178.703091] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42178.708238] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42178.713458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42178.717029] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42178.718518] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42178.725643] ORIG_RAX: 0000000000000012 [42178.730688] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42178.737811] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42178.741381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42178.748507] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42178.767248] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42178.774378] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42178.779595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42178.786718] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42178.790549] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42178.798627] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42178.805751] Call Trace: [42178.811494] [42178.818613] [42178.867998] ? watchdog_timer_fn+0x1b4/0x220 [42178.872260] ? __hrtimer_run_queues+0x1d3/0x360 [42178.876788] ? hrtimer_interrupt+0x104/0x240 [42178.881056] ? __local_bh_enable+0x50/0x60 [42178.885152] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42178.890374] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42178.895332] [42178.897430] [42178.899526] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42178.904836] ? lock_acquire+0xf0/0x2b0 [42178.908584] ? igrab+0x43/0x50 [42178.911634] ? lock_release+0xc5/0x260 [42178.915379] _raw_spin_lock+0x30/0x40 [42178.919034] ? igrab+0x1d/0x50 [42178.922086] igrab+0x1d/0x50 [42178.924965] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42178.930113] delayed_work+0x1d8/0x280 [ceph] [42178.934396] ? process_scheduled_works+0x22a/0x510 [42178.939186] process_scheduled_works+0x2a4/0x510 [42178.943802] worker_thread+0x188/0x340 [42178.947549] ? rescuer_thread+0x310/0x310 [42178.951554] kthread+0xf5/0x130 [42178.954692] ? kthread_complete_and_exit+0x20/0x20 [42178.959474] ret_from_fork+0x31/0x40 [42178.963047] ? kthread_complete_and_exit+0x20/0x20 [42178.967830] ret_from_fork_asm+0x11/0x20 [42178.971763] [42206.191032] watchdog: BUG: soft lockup - CPU#0 stuck for 28193s! [dbench:78683] [42206.198331] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42206.203032] watchdog: BUG: soft lockup - CPU#7 stuck for 37190s! [kworker/7:4:78808] [42206.253771] irq event stamp: 75730716 [42206.253773] hardirqs last enabled at (75730715): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42206.261504] Modules linked in: [42206.265160] hardirqs last disabled at (75730716): [] sysvec_apic_timer_interrupt+0xe/0x90 [42206.275317] ceph libceph [42206.278368] softirqs last enabled at (75730670): [] __do_softirq+0x2be/0x371 [42206.288091] dns_resolver fscache [42206.290709] softirqs last disabled at (75730665): [] irq_exit_rcu+0xb0/0xd0 [42206.299393] netfs veth [42206.302704] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42206.311216] nft_chain_nat xt_MASQUERADE [42206.313658] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42206.322604] nf_nat nf_conntrack [42206.326520] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42206.333991] nf_defrag_ipv6 nf_defrag_ipv4 [42206.337216] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42206.342692] nft_compat nf_tables [42206.346783] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42206.365521] nfnetlink bridge [42206.368831] [42206.368832] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42206.374051] stp llc [42206.377014] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42206.378504] xfs libcrc32c [42206.385628] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42206.387813] sunrpc [42206.394936] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42206.397641] intel_rapl_msr intel_rapl_common [42206.404766] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42206.406863] sb_edac x86_pkg_temp_thermal [42206.413986] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42206.418336] coretemp kvm_intel [42206.425460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42206.429465] kvm irqbypass [42206.437543] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42206.440680] i2c_i801 [42206.446431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42206.449140] iTCO_wdt iTCO_vendor_support [42206.456263] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42206.458532] i2c_smbus crct10dif_pclmul [42206.465657] Call Trace: [42206.465658] [42206.469662] joydev crc32_pclmul [42206.476788] ? watchdog_timer_fn+0x1b4/0x220 [42206.480617] ipmi_ssif ghash_clmulni_intel [42206.483061] ? __hrtimer_run_queues+0x1d3/0x360 [42206.485071] mei_me mei [42206.488300] ? hrtimer_interrupt+0x104/0x240 [42206.492560] lpc_ich sha512_ssse3 [42206.496650] ? __local_bh_enable+0x50/0x60 [42206.501174] acpi_ipmi mfd_core [42206.503621] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42206.507884] wmi ipmi_si [42206.511195] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42206.515285] ipmi_devintf ipmi_msghandler [42206.518433] [42206.523647] ioatdma acpi_pad [42206.526177] [42206.526179] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42206.531136] acpi_power_meter ixgbe [42206.535144] ? queued_spin_lock_slowpath+0x18/0x2e0 [42206.537236] igb mdio [42206.540203] do_raw_spin_lock+0xb5/0xc0 [42206.542299] crc32c_intel nvme [42206.547605] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42206.551087] ptp nvme_core [42206.555961] try_get_cap_refs+0x159/0x880 [ceph] [42206.558228] i2c_algo_bit pps_core [42206.562063] __ceph_get_caps+0x4a1/0x760 [ceph] [42206.565111] dca fuse [42206.569898] ? __file_remove_privs+0x89/0x110 [42206.572597] [42206.572598] irq event stamp: 257570292 [42206.577209] ? wake_bit_function+0x70/0x70 [42206.580607] hardirqs last enabled at (257570291): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42206.585134] ? ceph_write_iter+0x260/0x770 [ceph] [42206.587402] hardirqs last disabled at (257570292): [] sysvec_apic_timer_interrupt+0xe/0x90 [42206.591752] ceph_write_iter+0x260/0x770 [ceph] [42206.593241] softirqs last enabled at (257570290): [] __do_softirq+0x2be/0x371 [42206.596990] ? lock_is_held_type+0xaa/0x120 [42206.601077] softirqs last disabled at (257570285): [] irq_exit_rcu+0xb0/0xd0 [42206.611323] vfs_write+0x324/0x4f0 [42206.616020] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42206.625833] ksys_pwrite64+0x65/0xa0 [42206.630354] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42206.630355] Workqueue: events delayed_work [ceph] [42206.639124] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42206.643301] [42206.643302] RIP: 0010:lock_acquire+0xf0/0x2b0 [42206.651902] do_syscall_64+0x38/0x80 [42206.655298] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42206.664675] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42206.668245] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42206.675717] RIP: 0033:0x7f48f0eeb93b [42206.680430] [42206.680431] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42206.685560] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42206.687052] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42206.691403] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42206.694973] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42206.713725] ORIG_RAX: 0000000000000012 [42206.718773] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42206.718775] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42206.723991] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42206.727561] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42206.729052] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42206.736175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42206.754912] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42206.762037] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42206.767255] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42206.774378] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42206.778211] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42206.785334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42206.792463] [42206.799583] Call Trace: [42206.799584] [42206.869907] ? watchdog_timer_fn+0x1b4/0x220 [42206.874170] ? __hrtimer_run_queues+0x1d3/0x360 [42206.878696] ? hrtimer_interrupt+0x104/0x240 [42206.882967] ? __local_bh_enable+0x50/0x60 [42206.887061] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42206.892283] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42206.897241] [42206.899339] [42206.901437] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42206.906744] ? lock_acquire+0xf0/0x2b0 [42206.910492] ? igrab+0x43/0x50 [42206.913544] ? lock_release+0xc5/0x260 [42206.917288] _raw_spin_lock+0x30/0x40 [42206.920944] ? igrab+0x1d/0x50 [42206.923995] igrab+0x1d/0x50 [42206.926872] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42206.932022] delayed_work+0x1d8/0x280 [ceph] [42206.936304] ? process_scheduled_works+0x22a/0x510 [42206.941094] process_scheduled_works+0x2a4/0x510 [42206.945725] worker_thread+0x188/0x340 [42206.949475] ? rescuer_thread+0x310/0x310 [42206.953480] kthread+0xf5/0x130 [42206.956616] ? kthread_complete_and_exit+0x20/0x20 [42206.961402] ret_from_fork+0x31/0x40 [42206.964981] ? kthread_complete_and_exit+0x20/0x20 [42206.969764] ret_from_fork_asm+0x11/0x20 [42206.973689] [42234.191520] watchdog: BUG: soft lockup - CPU#0 stuck for 28219s! [dbench:78683] [42234.198827] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42234.203519] watchdog: BUG: soft lockup - CPU#7 stuck for 37216s! [kworker/7:4:78808] [42234.254277] irq event stamp: 75788908 [42234.262008] Modules linked in: ceph libceph dns_resolver fscache netfs veth [42234.265665] hardirqs last enabled at (75788907): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42234.272623] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack [42234.282780] hardirqs last disabled at (75788908): [] sysvec_apic_timer_interrupt+0xe/0x90 [42234.282783] nf_defrag_ipv6 [42234.282785] softirqs last enabled at (75788860): [] __do_softirq+0x2be/0x371 [42234.288433] nf_defrag_ipv4 nft_compat [42234.298157] softirqs last disabled at (75788855): [] irq_exit_rcu+0xb0/0xd0 [42234.300948] nf_tables nfnetlink [42234.309633] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42234.313374] bridge stp [42234.321887] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42234.325128] llc xfs [42234.334071] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42234.336516] libcrc32c sunrpc [42234.343987] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42234.346171] intel_rapl_msr intel_rapl_common [42234.351666] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42234.354630] sb_edac x86_pkg_temp_thermal [42234.373368] [42234.377719] coretemp kvm_intel [42234.382935] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42234.386939] kvm irqbypass [42234.388430] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42234.391568] i2c_i801 iTCO_wdt [42234.398691] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42234.401397] iTCO_vendor_support [42234.408520] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42234.411570] i2c_smbus crct10dif_pclmul [42234.418695] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42234.421919] joydev [42234.429042] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42234.432873] crc32_pclmul [42234.439996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42234.442096] ipmi_ssif ghash_clmulni_intel [42234.450172] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42234.452789] mei_me mei [42234.458527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42234.462618] lpc_ich [42234.469741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42234.472202] sha512_ssse3 acpi_ipmi [42234.479327] Call Trace: [42234.481511] mfd_core wmi [42234.488638] [42234.492120] ipmi_si ipmi_devintf [42234.494567] ? watchdog_timer_fn+0x1b4/0x220 [42234.497196] ipmi_msghandler ioatdma [42234.499210] ? __hrtimer_run_queues+0x1d3/0x360 [42234.502519] acpi_pad acpi_power_meter [42234.506788] ? hrtimer_interrupt+0x104/0x240 [42234.510355] ixgbe igb [42234.514880] ? __local_bh_enable+0x50/0x60 [42234.518623] mdio crc32c_intel [42234.522892] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42234.525244] nvme ptp [42234.529335] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42234.532386] nvme_core [42234.537602] [42234.539874] i2c_algo_bit pps_core [42234.544832] [42234.544834] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42234.547200] dca fuse [42234.549301] ? queued_spin_lock_slowpath+0x18/0x2e0 [42234.552692] [42234.552693] irq event stamp: 257791016 [42234.554792] do_raw_spin_lock+0xb5/0xc0 [42234.560093] hardirqs last enabled at (257791015): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42234.562367] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42234.567235] hardirqs last disabled at (257791016): [] sysvec_apic_timer_interrupt+0xe/0x90 [42234.568729] try_get_cap_refs+0x159/0x880 [ceph] [42234.572471] softirqs last enabled at (257791014): [] __do_softirq+0x2be/0x371 [42234.576307] __ceph_get_caps+0x4a1/0x760 [ceph] [42234.586545] softirqs last disabled at (257791009): [] irq_exit_rcu+0xb0/0xd0 [42234.591333] ? __file_remove_privs+0x89/0x110 [42234.601140] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42234.605752] ? wake_bit_function+0x70/0x70 [42234.614521] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42234.619048] ? ceph_write_iter+0x260/0x770 [ceph] [42234.627642] Workqueue: events delayed_work [ceph] [42234.631995] ceph_write_iter+0x260/0x770 [ceph] [42234.641372] [42234.641373] RIP: 0010:lock_acquire+0xf0/0x2b0 [42234.645466] ? lock_is_held_type+0xaa/0x120 [42234.652932] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42234.657633] vfs_write+0x324/0x4f0 [42234.662328] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42234.666857] ksys_pwrite64+0x65/0xa0 [42234.668342] [42234.668343] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42234.672695] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42234.676872] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42234.695610] do_syscall_64+0x38/0x80 [42234.699006] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42234.704225] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42234.707794] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42234.709286] RIP: 0033:0x7f48f0eeb93b [42234.716409] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42234.721541] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42234.728664] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42234.732235] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42234.739357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42234.744402] ORIG_RAX: 0000000000000012 [42234.751526] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42234.755098] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42234.762221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42234.780958] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42234.789035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42234.794253] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42234.799991] Call Trace: [42234.799992] [42234.803821] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42234.810947] ? watchdog_timer_fn+0x1b4/0x220 [42234.818070] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42234.825212] ? __hrtimer_run_queues+0x1d3/0x360 [42234.832342] [42234.839464] ? hrtimer_interrupt+0x104/0x240 [42234.880525] ? __local_bh_enable+0x50/0x60 [42234.884618] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42234.889842] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42234.894799] [42234.896897] [42234.899012] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42234.904319] ? lock_acquire+0xf0/0x2b0 [42234.908068] ? igrab+0x43/0x50 [42234.911118] ? lock_release+0xc5/0x260 [42234.914865] _raw_spin_lock+0x30/0x40 [42234.918529] ? igrab+0x1d/0x50 [42234.921578] igrab+0x1d/0x50 [42234.924457] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42234.929609] delayed_work+0x1d8/0x280 [ceph] [42234.933889] ? process_scheduled_works+0x22a/0x510 [42234.938677] process_scheduled_works+0x2a4/0x510 [42234.943294] worker_thread+0x188/0x340 [42234.947042] ? rescuer_thread+0x310/0x310 [42234.951045] kthread+0xf5/0x130 [42234.954185] ? kthread_complete_and_exit+0x20/0x20 [42234.958969] ret_from_fork+0x31/0x40 [42234.962547] ? kthread_complete_and_exit+0x20/0x20 [42234.967331] ret_from_fork_asm+0x11/0x20 [42234.971254] [42262.192006] watchdog: BUG: soft lockup - CPU#0 stuck for 28245s! [dbench:78683] [42262.199305] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42262.204006] watchdog: BUG: soft lockup - CPU#7 stuck for 37242s! [kworker/7:4:78808] [42262.254746] irq event stamp: 75847002 [42262.254747] hardirqs last enabled at (75847001): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42262.262476] Modules linked in: [42262.266135] hardirqs last disabled at (75847002): [] sysvec_apic_timer_interrupt+0xe/0x90 [42262.276292] ceph [42262.279343] softirqs last enabled at (75846996): [] __do_softirq+0x2be/0x371 [42262.289068] libceph [42262.290991] softirqs last disabled at (75846991): [] irq_exit_rcu+0xb0/0xd0 [42262.299682] dns_resolver [42262.301867] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42262.310379] fscache netfs [42262.313014] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42262.321957] veth nft_chain_nat [42262.324661] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42262.332133] xt_MASQUERADE nf_nat [42262.335270] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42262.340747] nf_conntrack nf_defrag_ipv6 [42262.344059] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42262.362796] nf_defrag_ipv4 nft_compat [42262.366712] [42262.366713] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42262.371930] nf_tables nfnetlink [42262.375689] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42262.377182] bridge stp [42262.384305] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42262.387532] llc xfs [42262.394656] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42262.397098] libcrc32c sunrpc [42262.404224] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42262.406407] intel_rapl_msr [42262.413530] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42262.416497] intel_rapl_common [42262.423618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42262.426409] sb_edac x86_pkg_temp_thermal [42262.434486] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42262.437538] coretemp kvm_intel [42262.443276] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42262.447280] kvm irqbypass [42262.454405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42262.457541] i2c_i801 iTCO_wdt [42262.464665] Call Trace: [42262.467403] iTCO_vendor_support [42262.474529] [42262.477579] i2c_smbus crct10dif_pclmul [42262.480025] ? watchdog_timer_fn+0x1b4/0x220 [42262.483247] joydev crc32_pclmul [42262.485258] ? __hrtimer_run_queues+0x1d3/0x360 [42262.489089] ipmi_ssif ghash_clmulni_intel [42262.493355] ? hrtimer_interrupt+0x104/0x240 [42262.496576] mei_me mei [42262.501102] ? __local_bh_enable+0x50/0x60 [42262.505191] lpc_ich sha512_ssse3 [42262.509459] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42262.511898] acpi_ipmi mfd_core [42262.515991] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42262.519300] wmi [42262.524518] [42262.527656] ipmi_si ipmi_devintf [42262.532614] [42262.534451] ipmi_msghandler ioatdma [42262.536549] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42262.539859] acpi_pad acpi_power_meter [42262.541961] ? queued_spin_lock_slowpath+0x16/0x2e0 [42262.545528] ixgbe igb [42262.550834] do_raw_spin_lock+0xb5/0xc0 [42262.554575] mdio crc32c_intel [42262.559447] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42262.561803] nvme ptp [42262.565637] try_get_cap_refs+0x159/0x880 [ceph] [42262.568686] nvme_core i2c_algo_bit [42262.573474] __ceph_get_caps+0x4a1/0x760 [ceph] [42262.575739] pps_core dca [42262.580355] ? __file_remove_privs+0x89/0x110 [42262.583835] fuse [42262.588358] ? wake_bit_function+0x70/0x70 [42262.590975] irq event stamp: 258011698 [42262.595330] ? ceph_write_iter+0x260/0x770 [ceph] [42262.597251] hardirqs last enabled at (258011697): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42262.601342] ceph_write_iter+0x260/0x770 [ceph] [42262.605084] hardirqs last disabled at (258011698): [] sysvec_apic_timer_interrupt+0xe/0x90 [42262.609787] ? lock_is_held_type+0xaa/0x120 [42262.620026] softirqs last enabled at (258011696): [] __do_softirq+0x2be/0x371 [42262.624554] vfs_write+0x324/0x4f0 [42262.634363] softirqs last disabled at (258011691): [] irq_exit_rcu+0xb0/0xd0 [42262.638544] ksys_pwrite64+0x65/0xa0 [42262.647309] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42262.650707] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42262.659306] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42262.662877] do_syscall_64+0x38/0x80 [42262.672254] Workqueue: events delayed_work [ceph] [42262.677385] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42262.684872] [42262.684873] RIP: 0010:lock_acquire+0xf0/0x2b0 [42262.688445] RIP: 0033:0x7f48f0eeb93b [42262.693139] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42262.698187] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42262.699674] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42262.704042] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42262.707615] [42262.707616] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42262.726353] ORIG_RAX: 0000000000000012 [42262.745088] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42262.750306] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42262.755523] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42262.757014] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42262.764138] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42262.767970] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42262.775095] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42262.782219] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42262.789343] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42262.796467] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42262.803591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42262.810721] [42262.817838] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42262.855210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42262.862334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42262.869458] Call Trace: [42262.871902] [42262.873914] ? watchdog_timer_fn+0x1b4/0x220 [42262.878178] ? __hrtimer_run_queues+0x1d3/0x360 [42262.882704] ? hrtimer_interrupt+0x104/0x240 [42262.886964] ? __local_bh_enable+0x50/0x60 [42262.891059] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42262.896284] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42262.901240] [42262.903337] [42262.905434] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42262.910745] ? lock_acquire+0xf0/0x2b0 [42262.914494] ? igrab+0x43/0x50 [42262.917544] ? lock_release+0xc5/0x260 [42262.921287] _raw_spin_lock+0x30/0x40 [42262.924944] ? igrab+0x1d/0x50 [42262.927994] igrab+0x1d/0x50 [42262.930872] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42262.936022] delayed_work+0x1d8/0x280 [ceph] [42262.940303] ? process_scheduled_works+0x22a/0x510 [42262.945094] process_scheduled_works+0x2a4/0x510 [42262.949711] worker_thread+0x188/0x340 [42262.953460] ? rescuer_thread+0x310/0x310 [42262.957471] kthread+0xf5/0x130 [42262.960608] ? kthread_complete_and_exit+0x20/0x20 [42262.965411] ret_from_fork+0x31/0x40 [42262.968988] ? kthread_complete_and_exit+0x20/0x20 [42262.973774] ret_from_fork_asm+0x11/0x20 [42262.977700] [42290.192495] watchdog: BUG: soft lockup - CPU#0 stuck for 28271s! [dbench:78683] [42290.199803] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42290.204494] watchdog: BUG: soft lockup - CPU#7 stuck for 37268s! [kworker/7:4:78808] [42290.255252] irq event stamp: 75905272 [42290.262982] Modules linked in: ceph libceph dns_resolver fscache netfs veth [42290.266639] hardirqs last enabled at (75905271): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42290.266645] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat [42290.283756] hardirqs last disabled at (75905272): [] sysvec_apic_timer_interrupt+0xe/0x90 [42290.292961] nf_tables nfnetlink bridge stp llc xfs [42290.302686] softirqs last enabled at (75905226): [] __do_softirq+0x2be/0x371 [42290.307563] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [42290.316248] softirqs last disabled at (75905221): [] irq_exit_rcu+0xb0/0xd0 [42290.324585] coretemp kvm_intel kvm irqbypass [42290.333098] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42290.333099] i2c_i801 [42290.333100] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42290.337448] iTCO_wdt iTCO_vendor_support [42290.346393] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42290.348663] i2c_smbus [42290.356134] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42290.360136] crct10dif_pclmul joydev [42290.365614] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42290.367971] crc32_pclmul [42290.386710] [42290.390280] ipmi_ssif ghash_clmulni_intel [42290.395498] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42290.398115] mei_me mei [42290.399607] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42290.403698] lpc_ich sha512_ssse3 [42290.410820] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42290.413266] acpi_ipmi [42290.420387] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42290.423700] mfd_core wmi [42290.430825] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42290.433182] ipmi_si ipmi_devintf [42290.440305] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42290.442923] ipmi_msghandler ioatdma [42290.450046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42290.453358] acpi_pad acpi_power_meter [42290.461436] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42290.465007] ixgbe igb [42290.470743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42290.474487] mdio crc32c_intel [42290.481612] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42290.483969] nvme ptp [42290.491092] Call Trace: [42290.491094] [42290.494143] nvme_core i2c_algo_bit [42290.501270] ? watchdog_timer_fn+0x1b4/0x220 [42290.503537] pps_core dca [42290.505984] ? __hrtimer_run_queues+0x1d3/0x360 [42290.507994] fuse [42290.511480] ? hrtimer_interrupt+0x104/0x240 [42290.515741] irq event stamp: 258232426 [42290.518359] ? __local_bh_enable+0x50/0x60 [42290.522892] hardirqs last enabled at (258232425): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42290.524819] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42290.529081] hardirqs last disabled at (258232426): [] sysvec_apic_timer_interrupt+0xe/0x90 [42290.532825] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42290.536913] softirqs last enabled at (258232424): [] __do_softirq+0x2be/0x371 [42290.547174] [42290.552394] softirqs last disabled at (258232419): [] irq_exit_rcu+0xb0/0xd0 [42290.562205] [42290.567176] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42290.575950] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42290.578046] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42290.578048] Workqueue: events delayed_work [ceph] [42290.586648] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42290.588741] [42290.588741] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [42290.598122] do_raw_spin_lock+0xb5/0xc0 [42290.603424] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [42290.610897] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42290.615592] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [42290.620466] try_get_cap_refs+0x159/0x880 [ceph] [42290.621954] [42290.621955] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [42290.626570] __ceph_get_caps+0x4a1/0x760 [ceph] [42290.630395] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [42290.630396] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [42290.649139] ? __file_remove_privs+0x89/0x110 [42290.653918] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [42290.659134] ? wake_bit_function+0x70/0x70 [42290.663745] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [42290.665237] ? ceph_write_iter+0x260/0x770 [ceph] [42290.672359] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42290.676896] ceph_write_iter+0x260/0x770 [ceph] [42290.684017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42290.691144] ? lock_is_held_type+0xaa/0x120 [42290.695490] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42290.702617] vfs_write+0x324/0x4f0 [42290.706705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42290.713835] ksys_pwrite64+0x65/0xa0 [42290.718527] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42290.726607] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42290.731130] Call Trace: [42290.731131] [42290.736868] do_syscall_64+0x38/0x80 [42290.741046] ? watchdog_timer_fn+0x1b4/0x220 [42290.748185] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42290.751585] ? __hrtimer_run_queues+0x1d3/0x360 [42290.758708] RIP: 0033:0x7f48f0eeb93b [42290.762283] ? hrtimer_interrupt+0x104/0x240 [42290.769402] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42290.774533] ? __local_bh_enable+0x50/0x60 [42290.776977] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42290.778992] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42290.782558] ORIG_RAX: 0000000000000012 [42290.786822] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42290.791866] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42290.796390] [42290.799961] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42290.804226] [42290.804227] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42290.822963] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42290.827059] ? do_raw_spin_lock+0x42/0xc0 [42290.832272] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42290.837490] igrab+0x1d/0x50 [42290.841319] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42290.843872] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [42290.843876] rcu: 0-....: (29666140 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=14862015 [42290.843898] rcu: 7-....: (29470434 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/29469053 fqs=14862015 [42290.843901] rcu: (detected by 4, t=30353689 jiffies, g=1309609, q=812434 ncpus=8) [42290.843904] Sending NMI from CPU 4 to CPUs 0: [42290.843908] NMI backtrace for cpu 0 [42290.843909] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42290.843911] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42290.843912] RIP: 0010:vprintk_emit+0x1ee/0x340 [42290.843915] Code: 01 00 00 00 31 f6 48 c7 c7 20 b9 47 82 e8 0a d9 fe ff 0f b6 05 1b 32 67 02 5a 48 c7 c1 18 57 7f 83 84 c0 74 09 f3 90 0f b6 11 <84> d2 75 f7 48 c7 c7 20 b9 47 82 48 8d 35 00 00 00 00 e8 db d5 fe [42290.843917] RSP: 0018:ffffc90000003cc0 EFLAGS: 00000002 [42290.843918] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff837f5718 [42290.843920] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88810c39b578 [42290.843921] RBP: 0000000000000041 R08: 0000000000005ef4 R09: 0000000000000004 [42290.843922] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff [42290.843923] R13: 0000000000000000 R14: ffffffff82188778 R15: ffffc90000003d10 [42290.843924] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42290.843926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42290.843927] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42290.843929] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42290.843930] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42290.843931] Call Trace: [42290.843932] [42290.843933] ? nmi_cpu_backtrace+0xad/0x120 [42290.843938] ? nmi_cpu_backtrace_handler+0x11/0x20 [42290.843940] ? nmi_handle+0xd4/0x260 [42290.843945] ? default_do_nmi+0x49/0xf0 [42290.843948] ? exc_nmi+0x11e/0x150 [42290.843950] ? end_repeat_nmi+0x16/0x67 [42290.843957] ? vprintk_emit+0x1ee/0x340 [42290.843959] ? vprintk_emit+0x1ee/0x340 [42290.843963] ? vprintk_emit+0x1ee/0x340 [42290.843965] [42290.843966] [42290.843968] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [42290.843970] _printk+0x5c/0x80 [42290.843975] __show_regs+0xc9/0x2e0 [42290.843978] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [42290.843980] ? unwind_next_frame+0x379/0x830 [42290.843984] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [42290.843987] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [42290.843989] show_trace_log_lvl+0x2b5/0x3f0 [42290.843998] ? touch_softlockup_watchdog+0x20/0x20 [42290.844001] watchdog_timer_fn+0x1b4/0x220 [42290.844003] __hrtimer_run_queues+0x1d3/0x360 [42290.844009] hrtimer_interrupt+0x104/0x240 [42290.844010] ? __local_bh_enable+0x50/0x60 [42290.844016] __sysvec_apic_timer_interrupt+0x61/0x160 [42290.844019] sysvec_apic_timer_interrupt+0x6d/0x90 [42290.844022] [42290.844022] [42290.844023] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42290.844026] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42290.844029] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42290.844031] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42290.844032] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42290.844034] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42290.844035] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42290.844036] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42290.844037] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42290.844044] do_raw_spin_lock+0xb5/0xc0 [42290.844046] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42290.844065] try_get_cap_refs+0x159/0x880 [ceph] [42290.844086] __ceph_get_caps+0x4a1/0x760 [ceph] [42290.844106] ? __file_remove_privs+0x89/0x110 [42290.844108] ? wake_bit_function+0x70/0x70 [42290.844112] ? ceph_write_iter+0x260/0x770 [ceph] [42290.844128] ceph_write_iter+0x260/0x770 [ceph] [42290.844145] ? lock_is_held_type+0xaa/0x120 [42290.844150] vfs_write+0x324/0x4f0 [42290.844156] ksys_pwrite64+0x65/0xa0 [42290.844159] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42290.844162] do_syscall_64+0x38/0x80 [42290.844165] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42290.844167] RIP: 0033:0x7f48f0eeb93b [42290.844169] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42290.844170] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [42290.844172] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42290.844173] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42290.844175] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42290.844176] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42290.844177] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42290.844183] [42290.844906] Sending NMI from CPU 4 to CPUs 7: [42290.844909] NMI backtrace for cpu 7 [42290.844910] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42290.844912] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42290.844913] Workqueue: events delayed_work [ceph] [42290.844927] RIP: 0010:delay_tsc+0x5c/0x90 [42290.844929] Code: f3 90 65 ff 05 2d f6 49 7e 65 8b 35 2a f6 49 7e 41 39 f1 75 24 41 89 f1 0f 01 f9 66 90 48 c1 e2 20 48 09 d0 48 89 c2 4c 29 c2 <48> 39 fa 72 c8 65 ff 0d 00 f6 49 7e 74 1f c3 4c 01 c7 48 29 c7 0f [42290.844931] RSP: 0018:ffffc90000264bc0 EFLAGS: 00000016 [42290.844932] RAX: 00008838c1ad75e4 RBX: 0000000000000000 RCX: 0000000000000007 [42290.844932] RDX: 0000000000000b59 RSI: 0000000000000007 RDI: 0000000000000dac [42290.844933] RBP: 00000000000025a2 R08: 00008838c1ad6a8b R09: 0000000000000007 [42290.844934] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000020 [42290.844934] R13: ffffffff838a9560 R14: 0000000000000010 R15: 0000000000000000 [42290.844935] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42290.844936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42290.844937] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42290.844938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42290.844938] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42290.844939] Call Trace: [42290.844939] [42290.844940] ? nmi_cpu_backtrace+0xad/0x120 [42290.844943] ? nmi_cpu_backtrace_handler+0x11/0x20 [42290.844944] ? nmi_handle+0xd4/0x260 [42290.844947] ? default_do_nmi+0x49/0xf0 [42290.844949] ? exc_nmi+0x11e/0x150 [42290.844950] ? end_repeat_nmi+0x16/0x67 [42290.844954] ? delay_tsc+0x5c/0x90 [42290.844956] ? delay_tsc+0x5c/0x90 [42290.844958] ? delay_tsc+0x5c/0x90 [42290.844960] [42290.844960] [42290.844960] wait_for_lsr+0x27/0x70 [42290.844964] serial8250_console_write+0x341/0x3f0 [42290.844967] ? lock_release+0xc5/0x260 [42290.844969] console_flush_all+0x227/0x5a0 [42290.844971] ? console_flush_all+0x1ff/0x5a0 [42290.844972] ? console_flush_all+0x1ff/0x5a0 [42290.844975] console_unlock+0x7b/0x120 [42290.844977] vprintk_emit+0x246/0x340 [42290.844979] ? igrab+0x1d/0x50 [42290.844981] ? igrab+0x1d/0x50 [42290.844982] _printk+0x5c/0x80 [42290.844984] ? igrab+0x1d/0x50 [42290.844985] show_trace_log_lvl+0x264/0x3f0 [42290.844990] ? igrab+0x1d/0x50 [42290.844991] ? touch_softlockup_watchdog+0x20/0x20 [42290.844993] watchdog_timer_fn+0x1b4/0x220 [42290.844995] __hrtimer_run_queues+0x1d3/0x360 [42290.844998] hrtimer_interrupt+0x104/0x240 [42290.844999] ? __local_bh_enable+0x50/0x60 [42290.845002] __sysvec_apic_timer_interrupt+0x61/0x160 [42290.845005] sysvec_apic_timer_interrupt+0x6d/0x90 [42290.845006] [42290.845007] [42290.845007] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42290.845009] RIP: 0010:do_raw_spin_lock+0x42/0xc0 [42290.845011] Code: 8b 53 10 65 48 8b 04 25 c0 e5 02 00 48 39 c2 74 4e 8b 53 08 65 8b 05 49 c0 eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 31 c0 eb 7e 89 43 08 65 48 8b 04 25 c0 e5 02 00 48 [42290.845013] RSP: 0018:ffffc90002117d90 EFLAGS: 00000246 [42290.845014] RAX: 0000000000000000 RBX: ffff8881a1d34500 RCX: 000000005553a634 [42290.845014] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34500 [42290.845015] RBP: ffff8881a1d34500 R08: 0000000000005c53 R09: 0000000000000003 [42290.845015] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000ea60 [42290.845016] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [42290.845019] igrab+0x1d/0x50 [42290.845021] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42290.845035] delayed_work+0x1d8/0x280 [ceph] [42290.845049] ? process_scheduled_works+0x22a/0x510 [42290.845052] process_scheduled_works+0x2a4/0x510 [42290.845056] worker_thread+0x188/0x340 [42290.845059] ? rescuer_thread+0x310/0x310 [42290.845061] kthread+0xf5/0x130 [42290.845063] ? kthread_complete_and_exit+0x20/0x20 [42290.845066] ret_from_fork+0x31/0x40 [42290.845068] ? kthread_complete_and_exit+0x20/0x20 [42290.845070] ret_from_fork_asm+0x11/0x20 [42290.845075] [42290.846279] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42290.853406] [42290.855500] delayed_work+0x1d8/0x280 [ceph] [42291.759737] ? process_scheduled_works+0x22a/0x510 [42291.764526] process_scheduled_works+0x2a4/0x510 [42291.769142] worker_thread+0x188/0x340 [42291.772891] ? rescuer_thread+0x310/0x310 [42291.776913] kthread+0xf5/0x130 [42291.780052] ? kthread_complete_and_exit+0x20/0x20 [42291.784843] ret_from_fork+0x31/0x40 [42291.788413] ? kthread_complete_and_exit+0x20/0x20 [42291.793197] ret_from_fork_asm+0x11/0x20 [42291.797122] [42318.192980] watchdog: BUG: soft lockup - CPU#0 stuck for 28297s! [dbench:78683] [42318.200279] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42318.204980] watchdog: BUG: soft lockup - CPU#7 stuck for 37294s! [kworker/7:4:78808] [42318.255722] irq event stamp: 75963384 [42318.255723] hardirqs last enabled at (75963383): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42318.263453] Modules linked in: ceph [42318.267111] hardirqs last disabled at (75963384): [] sysvec_apic_timer_interrupt+0xe/0x90 [42318.277267] libceph [42318.280750] softirqs last enabled at (75963344): [] __do_softirq+0x2be/0x371 [42318.290476] dns_resolver fscache [42318.292666] softirqs last disabled at (75963329): [] irq_exit_rcu+0xb0/0xd0 [42318.301370] netfs veth [42318.304680] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42318.313190] nft_chain_nat xt_MASQUERADE [42318.315635] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42318.324597] nf_nat nf_conntrack [42318.328514] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42318.335985] nf_defrag_ipv6 [42318.339209] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42318.344687] nf_defrag_ipv4 nft_compat [42318.347477] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42318.366216] nf_tables nfnetlink [42318.369958] [42318.369959] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42318.375177] bridge stp [42318.378401] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42318.379890] llc xfs [42318.387014] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42318.389459] libcrc32c [42318.396583] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42318.398767] sunrpc [42318.405891] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42318.408247] intel_rapl_msr intel_rapl_common [42318.415376] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42318.417470] sb_edac x86_pkg_temp_thermal [42318.424594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42318.428946] coretemp kvm_intel [42318.437022] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42318.441026] kvm irqbypass [42318.446764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42318.449900] i2c_i801 iTCO_wdt [42318.457026] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42318.459729] iTCO_vendor_support i2c_smbus [42318.466853] Call Trace: [42318.469904] crct10dif_pclmul joydev [42318.477028] [42318.481120] crc32_pclmul ipmi_ssif [42318.483564] ? watchdog_timer_fn+0x1b4/0x220 [42318.487135] ghash_clmulni_intel [42318.489147] ? __hrtimer_run_queues+0x1d3/0x360 [42318.492628] mei_me mei [42318.496896] ? hrtimer_interrupt+0x104/0x240 [42318.500116] lpc_ich sha512_ssse3 [42318.504640] ? __local_bh_enable+0x50/0x60 [42318.507085] acpi_ipmi mfd_core [42318.511353] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42318.514659] wmi ipmi_si [42318.518768] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42318.521905] ipmi_devintf ipmi_msghandler [42318.527123] [42318.529654] ioatdma acpi_pad [42318.534611] [42318.534612] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42318.538614] acpi_power_meter ixgbe [42318.540715] ? queued_spin_lock_slowpath+0x16/0x2e0 [42318.543676] igb [42318.545775] do_raw_spin_lock+0xb5/0xc0 [42318.551078] mdio crc32c_intel nvme [42318.554563] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42318.559431] ptp nvme_core [42318.561273] try_get_cap_refs+0x159/0x880 [ceph] [42318.565100] i2c_algo_bit pps_core [42318.568591] __ceph_get_caps+0x4a1/0x760 [ceph] [42318.573381] dca [42318.576085] ? __file_remove_privs+0x89/0x110 [42318.580693] fuse [42318.584091] ? wake_bit_function+0x70/0x70 [42318.588614] [42318.588614] irq event stamp: 258446576 [42318.590455] ? ceph_write_iter+0x260/0x770 [ceph] [42318.594801] hardirqs last enabled at (258446575): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42318.596725] ceph_write_iter+0x260/0x770 [ceph] [42318.600816] hardirqs last disabled at (258446576): [] sysvec_apic_timer_interrupt+0xe/0x90 [42318.602312] ? lock_is_held_type+0xaa/0x120 [42318.606053] softirqs last enabled at (258446574): [] __do_softirq+0x2be/0x371 [42318.610751] vfs_write+0x324/0x4f0 [42318.620993] softirqs last disabled at (258446569): [] irq_exit_rcu+0xb0/0xd0 [42318.625522] ksys_pwrite64+0x65/0xa0 [42318.635327] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42318.639506] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42318.648276] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42318.651676] do_syscall_64+0x38/0x80 [42318.660271] Workqueue: events delayed_work [ceph] [42318.663843] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42318.673219] [42318.673220] RIP: 0010:rcu_is_watching+0x0/0x40 [42318.678350] RIP: 0033:0x7f48f0eeb93b [42318.685822] Code: 44 00 00 85 ff 75 13 0f bf 05 a4 6d 3c 01 89 06 48 8b 05 33 6d 3c 01 48 89 02 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1e fa 65 ff 05 9d 5b e9 7e 48 c7 c0 28 a3 02 00 65 48 03 05 [42318.689393] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42318.694088] RSP: 0018:ffffc90002117d40 EFLAGS: 00000247 [42318.699133] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42318.700623] [42318.700624] RAX: 0000000000000007 RBX: ffff8881a1d34500 RCX: 000000005553a634 [42318.705061] ORIG_RAX: 0000000000000012 [42318.708632] RDX: 0000000000000001 RSI: ffffffff813b9433 RDI: ffff8881a1d34518 [42318.727383] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42318.746124] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [42318.751342] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42318.756559] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [42318.758050] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42318.765173] R13: ffff88817f867478 R14: 00000001001d239b R15: ffff888100ee2000 [42318.769005] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42318.776130] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42318.783252] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42318.790383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42318.797515] [42318.804633] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42318.856255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42318.863384] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42318.870512] Call Trace: [42318.872956] [42318.874966] ? watchdog_timer_fn+0x1b4/0x220 [42318.879231] ? __hrtimer_run_queues+0x1d3/0x360 [42318.883758] ? hrtimer_interrupt+0x104/0x240 [42318.888028] ? __local_bh_enable+0x50/0x60 [42318.892119] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42318.897336] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42318.902294] [42318.904390] [42318.906487] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42318.911793] ? igrab+0x43/0x50 [42318.914845] ? igrab+0x43/0x50 [42318.917895] ? rcutorture_get_gp_data+0x30/0x30 [42318.922419] lock_release+0x1d7/0x260 [42318.926078] _raw_spin_unlock+0x1b/0x40 [42318.929916] igrab+0x43/0x50 [42318.932792] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42318.937942] delayed_work+0x1d8/0x280 [ceph] [42318.942224] ? process_scheduled_works+0x22a/0x510 [42318.947012] process_scheduled_works+0x2a4/0x510 [42318.951628] worker_thread+0x188/0x340 [42318.955379] ? rescuer_thread+0x310/0x310 [42318.959386] kthread+0xf5/0x130 [42318.962527] ? kthread_complete_and_exit+0x20/0x20 [42318.967312] ret_from_fork+0x31/0x40 [42318.970881] ? kthread_complete_and_exit+0x20/0x20 [42318.975667] ret_from_fork_asm+0x11/0x20 [42318.979589] [42346.193466] watchdog: BUG: soft lockup - CPU#0 stuck for 28323s! [dbench:78683] [42346.200769] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42346.205465] watchdog: BUG: soft lockup - CPU#7 stuck for 37320s! [kworker/7:4:78808] [42346.256216] irq event stamp: 76021620 [42346.263947] Modules linked in: ceph libceph dns_resolver fscache netfs veth [42346.267606] hardirqs last enabled at (76021619): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42346.267612] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [42346.284722] hardirqs last disabled at (76021620): [] sysvec_apic_timer_interrupt+0xe/0x90 [42346.284725] nft_compat [42346.284727] softirqs last enabled at (76021572): [] __do_softirq+0x2be/0x371 [42346.292973] nf_tables nfnetlink [42346.302698] softirqs last disabled at (76021567): [] irq_exit_rcu+0xb0/0xd0 [42346.305158] bridge stp [42346.313861] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42346.317084] llc xfs [42346.325594] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42346.328038] libcrc32c sunrpc [42346.336984] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42346.339167] intel_rapl_msr intel_rapl_common [42346.346639] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42346.349602] sb_edac x86_pkg_temp_thermal [42346.355079] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42346.359429] coretemp kvm_intel [42346.378168] [42346.378168] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42346.382172] kvm [42346.387390] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42346.390526] irqbypass [42346.392016] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42346.399145] i2c_i801 iTCO_wdt [42346.400978] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42346.408103] iTCO_vendor_support [42346.410460] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42346.417584] i2c_smbus crct10dif_pclmul [42346.420636] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42346.427759] joydev crc32_pclmul [42346.430983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42346.438106] ipmi_ssif ghash_clmulni_intel [42346.441937] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42346.450016] mei_me mei [42346.453239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42346.458976] lpc_ich sha512_ssse3 [42346.463069] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42346.470191] acpi_ipmi mfd_core [42346.472635] Call Trace: [42346.472637] [42346.479761] wmi ipmi_si [42346.483072] ? watchdog_timer_fn+0x1b4/0x220 [42346.490196] ipmi_devintf [42346.493335] ? __hrtimer_run_queues+0x1d3/0x360 [42346.495776] ipmi_msghandler [42346.497790] ? hrtimer_interrupt+0x104/0x240 [42346.500317] ioatdma acpi_pad [42346.504583] ? __local_bh_enable+0x50/0x60 [42346.507200] acpi_power_meter ixgbe [42346.511727] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42346.514602] igb mdio [42346.518864] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42346.521829] crc32c_intel [42346.525920] [42346.529404] nvme ptp [42346.534622] [42346.536893] nvme_core i2c_algo_bit [42346.541864] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42346.544484] pps_core [42346.546584] ? queued_spin_lock_slowpath+0x18/0x2e0 [42346.548865] dca fuse [42346.550961] do_raw_spin_lock+0xb5/0xc0 [42346.554441] [42346.554442] irq event stamp: 258667274 [42346.559749] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42346.562017] hardirqs last enabled at (258667273): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42346.566890] try_get_cap_refs+0x159/0x880 [ceph] [42346.569158] hardirqs last disabled at (258667274): [] sysvec_apic_timer_interrupt+0xe/0x90 [42346.572994] __ceph_get_caps+0x4a1/0x760 [ceph] [42346.574479] softirqs last enabled at (258667272): [] __do_softirq+0x2be/0x371 [42346.578228] ? __file_remove_privs+0x89/0x110 [42346.583009] softirqs last disabled at (258667267): [] irq_exit_rcu+0xb0/0xd0 [42346.593253] ? wake_bit_function+0x70/0x70 [42346.597868] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42346.607686] ? ceph_write_iter+0x260/0x770 [ceph] [42346.612207] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42346.620978] ceph_write_iter+0x260/0x770 [ceph] [42346.625329] Workqueue: events delayed_work [ceph] [42346.633930] ? lock_is_held_type+0xaa/0x120 [42346.638017] [42346.647397] vfs_write+0x324/0x4f0 [42346.652092] RIP: 0010:lock_acquire+0xf0/0x2b0 [42346.659566] ksys_pwrite64+0x65/0xa0 [42346.664085] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42346.668786] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42346.672961] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42346.674454] do_syscall_64+0x38/0x80 [42346.677865] [42346.677866] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42346.682218] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42346.685788] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42346.704527] RIP: 0033:0x7f48f0eeb93b [42346.709656] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42346.709657] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42346.714875] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42346.718444] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42346.719936] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42346.727058] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42346.732103] ORIG_RAX: 0000000000000012 [42346.739228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42346.742799] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42346.749923] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42346.757047] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42346.775783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42346.782908] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42346.788126] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42346.796222] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42346.800053] Call Trace: [42346.800054] [42346.805789] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42346.812914] ? watchdog_timer_fn+0x1b4/0x220 [42346.820042] [42346.827164] ? __hrtimer_run_queues+0x1d3/0x360 [42346.878212] ? hrtimer_interrupt+0x104/0x240 [42346.882483] ? __local_bh_enable+0x50/0x60 [42346.886585] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42346.891809] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42346.896766] [42346.898869] [42346.900968] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42346.906277] ? lock_acquire+0xf0/0x2b0 [42346.910028] ? igrab+0x43/0x50 [42346.913084] ? lock_release+0xc5/0x260 [42346.916830] _raw_spin_lock+0x30/0x40 [42346.920487] ? igrab+0x1d/0x50 [42346.923537] igrab+0x1d/0x50 [42346.926416] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42346.931566] delayed_work+0x1d8/0x280 [ceph] [42346.935848] ? process_scheduled_works+0x22a/0x510 [42346.940637] process_scheduled_works+0x2a4/0x510 [42346.945253] worker_thread+0x188/0x340 [42346.949000] ? rescuer_thread+0x310/0x310 [42346.953004] kthread+0xf5/0x130 [42346.956155] ? kthread_complete_and_exit+0x20/0x20 [42346.960944] ret_from_fork+0x31/0x40 [42346.964524] ? kthread_complete_and_exit+0x20/0x20 [42346.969307] ret_from_fork_asm+0x11/0x20 [42346.973229] [42374.193950] watchdog: BUG: soft lockup - CPU#0 stuck for 28350s! [dbench:78683] [42374.201255] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42374.205950] watchdog: BUG: soft lockup - CPU#7 stuck for 37347s! [kworker/7:4:78808] [42374.256696] irq event stamp: 76079778 [42374.256697] hardirqs last enabled at (76079777): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42374.264427] Modules linked in: [42374.268082] hardirqs last disabled at (76079778): [] sysvec_apic_timer_interrupt+0xe/0x90 [42374.278241] ceph [42374.281292] softirqs last enabled at (76079742): [] __do_softirq+0x2be/0x371 [42374.291016] libceph [42374.292941] softirqs last disabled at (76079737): [] irq_exit_rcu+0xb0/0xd0 [42374.301633] dns_resolver [42374.303818] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42374.312345] fscache netfs [42374.314961] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42374.323908] veth [42374.326624] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42374.334098] nft_chain_nat [42374.336022] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42374.341499] xt_MASQUERADE [42374.344204] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42374.362943] nf_nat nf_conntrack [42374.365646] [42374.370864] nf_defrag_ipv6 [42374.374087] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42374.375577] nf_defrag_ipv4 nft_compat [42374.378368] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42374.385494] nf_tables nfnetlink [42374.389236] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42374.396360] bridge stp [42374.399585] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42374.406710] llc [42374.409153] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42374.416277] xfs libcrc32c [42374.418114] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42374.425238] sunrpc intel_rapl_msr [42374.427942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42374.436020] intel_rapl_common sb_edac [42374.439418] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42374.445154] x86_pkg_temp_thermal coretemp [42374.448899] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42374.456024] kvm_intel kvm [42374.460115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42374.467238] irqbypass i2c_i801 [42374.469942] Call Trace: [42374.469943] [42374.477066] iTCO_wdt iTCO_vendor_support [42374.480205] ? watchdog_timer_fn+0x1b4/0x220 [42374.482647] i2c_smbus [42374.484659] ? __hrtimer_run_queues+0x1d3/0x360 [42374.488662] crct10dif_pclmul [42374.492931] ? hrtimer_interrupt+0x104/0x240 [42374.495283] joydev crc32_pclmul [42374.499808] ? __local_bh_enable+0x50/0x60 [42374.502772] ipmi_ssif ghash_clmulni_intel [42374.507038] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42374.510260] mei_me mei [42374.514352] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42374.518442] lpc_ich sha512_ssse3 [42374.523661] [42374.526104] acpi_ipmi mfd_core [42374.531060] [42374.531061] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42374.534371] wmi [42374.536474] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42374.539607] ipmi_si ipmi_devintf [42374.541722] do_raw_spin_lock+0xb5/0xc0 [42374.547024] ipmi_msghandler [42374.548863] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42374.553733] ioatdma acpi_pad acpi_power_meter [42374.557046] try_get_cap_refs+0x159/0x880 [ceph] [42374.560875] ixgbe [42374.563756] __ceph_get_caps+0x4a1/0x760 [ceph] [42374.568536] igb mdio [42374.572978] ? __file_remove_privs+0x89/0x110 [42374.577584] crc32c_intel nvme [42374.579597] ? wake_bit_function+0x70/0x70 [42374.584118] ptp nvme_core [42374.586394] ? ceph_write_iter+0x260/0x770 [ceph] [42374.590740] i2c_algo_bit [42374.593792] ceph_write_iter+0x260/0x770 [ceph] [42374.597882] pps_core dca [42374.600590] ? lock_is_held_type+0xaa/0x120 [42374.605285] fuse [42374.607904] vfs_write+0x324/0x4f0 [42374.612425] [42374.612426] irq event stamp: 258888134 [42374.615048] ksys_pwrite64+0x65/0xa0 [42374.619219] hardirqs last enabled at (258888133): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42374.621143] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42374.624540] hardirqs last disabled at (258888134): [] sysvec_apic_timer_interrupt+0xe/0x90 [42374.626034] do_syscall_64+0x38/0x80 [42374.629775] softirqs last enabled at (258888132): [] __do_softirq+0x2be/0x371 [42374.633350] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42374.643590] softirqs last disabled at (258888127): [] irq_exit_rcu+0xb0/0xd0 [42374.648721] RIP: 0033:0x7f48f0eeb93b [42374.658532] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42374.662103] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42374.670874] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42374.675918] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42374.684517] Workqueue: events delayed_work [ceph] [42374.688086] ORIG_RAX: 0000000000000012 [42374.697465] [42374.697466] RIP: 0010:lock_acquire+0xf0/0x2b0 [42374.716201] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42374.723672] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42374.728889] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42374.733587] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42374.737417] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42374.738909] [42374.738910] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42374.743259] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42374.750385] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42374.769121] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42374.776245] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42374.781470] [42374.788586] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42374.835025] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42374.842148] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42374.850226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42374.855962] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42374.863087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42374.870212] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42374.877353] Call Trace: [42374.879796] [42374.881808] ? watchdog_timer_fn+0x1b4/0x220 [42374.886074] ? __hrtimer_run_queues+0x1d3/0x360 [42374.890600] ? hrtimer_interrupt+0x104/0x240 [42374.894868] ? __local_bh_enable+0x50/0x60 [42374.898962] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42374.904187] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42374.909144] [42374.911241] [42374.913356] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42374.918663] ? lock_acquire+0xf0/0x2b0 [42374.922413] ? igrab+0x43/0x50 [42374.925464] ? lock_release+0xc5/0x260 [42374.929208] _raw_spin_lock+0x30/0x40 [42374.932863] ? igrab+0x1d/0x50 [42374.935915] igrab+0x1d/0x50 [42374.938793] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42374.943944] delayed_work+0x1d8/0x280 [ceph] [42374.948225] ? process_scheduled_works+0x22a/0x510 [42374.953015] process_scheduled_works+0x2a4/0x510 [42374.957642] worker_thread+0x188/0x340 [42374.961387] ? rescuer_thread+0x310/0x310 [42374.965390] kthread+0xf5/0x130 [42374.968528] ? kthread_complete_and_exit+0x20/0x20 [42374.973313] ret_from_fork+0x31/0x40 [42374.976882] ? kthread_complete_and_exit+0x20/0x20 [42374.981668] ret_from_fork_asm+0x11/0x20 [42374.985592] [42402.194435] watchdog: BUG: soft lockup - CPU#0 stuck for 28376s! [dbench:78683] [42402.201741] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42402.206435] watchdog: BUG: soft lockup - CPU#7 stuck for 37373s! [kworker/7:4:78808] [42402.257192] irq event stamp: 76138096 [42402.264922] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat [42402.268579] hardirqs last enabled at (76138095): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42402.276752] xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [42402.286910] hardirqs last disabled at (76138096): [] sysvec_apic_timer_interrupt+0xe/0x90 [42402.293946] nft_compat nf_tables nfnetlink bridge stp llc xfs [42402.303671] softirqs last enabled at (76138048): [] __do_softirq+0x2be/0x371 [42402.309495] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [42402.318179] softirqs last disabled at (76138043): [] irq_exit_rcu+0xb0/0xd0 [42402.326518] coretemp kvm_intel kvm irqbypass [42402.335029] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42402.335030] i2c_i801 iTCO_wdt iTCO_vendor_support [42402.348322] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42402.348324] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42402.353122] i2c_smbus [42402.360596] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42402.366072] crct10dif_pclmul [42402.368431] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42402.387167] joydev [42402.390133] [42402.395350] crc32_pclmul ipmi_ssif [42402.397447] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42402.398937] ghash_clmulni_intel [42402.402420] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42402.409546] mei_me mei [42402.412769] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42402.419893] lpc_ich sha512_ssse3 [42402.422337] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42402.429461] acpi_ipmi [42402.432771] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42402.439896] mfd_core wmi [42402.442255] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42402.449377] ipmi_si ipmi_devintf [42402.451996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42402.460072] ipmi_msghandler ioatdma [42402.463384] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42402.469122] acpi_pad [42402.472692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42402.479832] acpi_power_meter [42402.482113] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42402.489236] ixgbe igb [42402.492200] Call Trace: [42402.492202] [42402.499326] mdio crc32c_intel [42402.501684] ? watchdog_timer_fn+0x1b4/0x220 [42402.504127] nvme ptp [42402.506138] ? __hrtimer_run_queues+0x1d3/0x360 [42402.509188] nvme_core i2c_algo_bit [42402.513454] ? hrtimer_interrupt+0x104/0x240 [42402.515722] pps_core [42402.520247] ? __local_bh_enable+0x50/0x60 [42402.523731] dca [42402.527997] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42402.530266] fuse [42402.534358] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42402.536192] irq event stamp: 259108790 [42402.541411] [42402.543335] hardirqs last enabled at (259108789): [] asm_common_interrupt+0x26/0x40 [42402.548291] [42402.548293] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42402.552037] hardirqs last disabled at (259108790): [] sysvec_apic_timer_interrupt+0xe/0x90 [42402.554137] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42402.563424] softirqs last enabled at (259108788): [] __do_softirq+0x2be/0x371 [42402.565523] do_raw_spin_lock+0xb5/0xc0 [42402.570838] softirqs last disabled at (259108783): [] irq_exit_rcu+0xb0/0xd0 [42402.580648] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42402.585516] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42402.594290] try_get_cap_refs+0x159/0x880 [ceph] [42402.598128] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42402.606731] __ceph_get_caps+0x4a1/0x760 [ceph] [42402.611508] Workqueue: events delayed_work [ceph] [42402.620890] ? __file_remove_privs+0x89/0x110 [42402.625496] [42402.625497] RIP: 0010:lock_acquire+0xf0/0x2b0 [42402.632968] ? wake_bit_function+0x70/0x70 [42402.637493] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42402.642191] ? ceph_write_iter+0x260/0x770 [ceph] [42402.646539] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42402.648030] ceph_write_iter+0x260/0x770 [ceph] [42402.652380] [42402.652380] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42402.656476] ? lock_is_held_type+0xaa/0x120 [42402.675210] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42402.679911] vfs_write+0x324/0x4f0 [42402.685124] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42402.689653] ksys_pwrite64+0x65/0xa0 [42402.691139] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42402.698265] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42402.702442] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42402.709565] do_syscall_64+0x38/0x80 [42402.712962] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42402.720088] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42402.723658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42402.730783] RIP: 0033:0x7f48f0eeb93b [42402.735911] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42402.743037] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42402.746607] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42402.754684] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42402.759728] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42402.765466] ORIG_RAX: 0000000000000012 [42402.769036] Call Trace: [42402.769037] [42402.776161] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42402.794898] ? watchdog_timer_fn+0x1b4/0x220 [42402.802023] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42402.807241] ? __hrtimer_run_queues+0x1d3/0x360 [42402.814364] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42402.818199] ? hrtimer_interrupt+0x104/0x240 [42402.820639] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42402.822648] ? __local_bh_enable+0x50/0x60 [42402.829774] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42402.834040] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42402.841168] [42402.845686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42402.887790] [42402.889888] [42402.891985] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42402.897293] ? lock_acquire+0xf0/0x2b0 [42402.901042] ? igrab+0x43/0x50 [42402.904093] ? lock_release+0xc5/0x260 [42402.907843] _raw_spin_lock+0x30/0x40 [42402.911504] ? igrab+0x1d/0x50 [42402.914553] igrab+0x1d/0x50 [42402.917431] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42402.922580] delayed_work+0x1d8/0x280 [ceph] [42402.926862] ? process_scheduled_works+0x22a/0x510 [42402.931651] process_scheduled_works+0x2a4/0x510 [42402.936269] worker_thread+0x188/0x340 [42402.940016] ? rescuer_thread+0x310/0x310 [42402.944020] kthread+0xf5/0x130 [42402.947160] ? kthread_complete_and_exit+0x20/0x20 [42402.951952] ret_from_fork+0x31/0x40 [42402.955530] ? kthread_complete_and_exit+0x20/0x20 [42402.960315] ret_from_fork_asm+0x11/0x20 [42402.964246] [42430.194921] watchdog: BUG: soft lockup - CPU#0 stuck for 28402s! [dbench:78683] [42430.202221] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42430.206921] watchdog: BUG: soft lockup - CPU#7 stuck for 37399s! [kworker/7:4:78808] [42430.257661] irq event stamp: 76196200 [42430.257662] hardirqs last enabled at (76196199): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42430.265392] Modules linked in: ceph [42430.269050] hardirqs last disabled at (76196200): [] sysvec_apic_timer_interrupt+0xe/0x90 [42430.279207] libceph [42430.282691] softirqs last enabled at (76196152): [] __do_softirq+0x2be/0x371 [42430.292415] dns_resolver fscache [42430.294599] softirqs last disabled at (76196147): [] irq_exit_rcu+0xb0/0xd0 [42430.303284] netfs [42430.306593] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42430.315105] veth [42430.317116] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42430.326058] nft_chain_nat [42430.327982] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [42430.335455] xt_MASQUERADE [42430.338159] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [42430.343635] nf_nat nf_conntrack [42430.346339] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42430.365078] nf_defrag_ipv6 nf_defrag_ipv4 [42430.368318] [42430.368319] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42430.373536] nft_compat nf_tables [42430.377627] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42430.379118] nfnetlink [42430.386243] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42430.389551] bridge stp [42430.396676] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42430.399033] llc xfs [42430.406159] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42430.408603] libcrc32c sunrpc [42430.415727] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42430.417910] intel_rapl_msr [42430.425033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42430.427999] intel_rapl_common sb_edac [42430.436075] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42430.438866] x86_pkg_temp_thermal coretemp [42430.444609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42430.448357] kvm_intel kvm [42430.455482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42430.459572] irqbypass i2c_i801 [42430.466697] Call Trace: [42430.466698] [42430.469401] iTCO_wdt iTCO_vendor_support [42430.476524] ? watchdog_timer_fn+0x1b4/0x220 [42430.479662] i2c_smbus crct10dif_pclmul [42430.482106] ? __hrtimer_run_queues+0x1d3/0x360 [42430.484116] joydev crc32_pclmul [42430.488124] ? hrtimer_interrupt+0x104/0x240 [42430.492384] ipmi_ssif ghash_clmulni_intel [42430.496214] ? __local_bh_enable+0x50/0x60 [42430.500738] mei_me [42430.503966] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42430.508226] mei lpc_ich [42430.512320] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42430.516408] sha512_ssse3 acpi_ipmi [42430.518506] [42430.523723] mfd_core [42430.526254] [42430.531212] wmi ipmi_si [42430.534697] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42430.536793] ipmi_devintf ipmi_msghandler [42430.539069] ? queued_spin_lock_slowpath+0x2b/0x2e0 [42430.541161] ioatdma acpi_pad [42430.543695] do_raw_spin_lock+0xb5/0xc0 [42430.548996] acpi_power_meter [42430.553003] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42430.557871] ixgbe [42430.560838] try_get_cap_refs+0x159/0x880 [ceph] [42430.564666] igb mdio [42430.567635] __ceph_get_caps+0x4a1/0x760 [ceph] [42430.572413] crc32c_intel nvme [42430.574428] ? __file_remove_privs+0x89/0x110 [42430.579035] ptp nvme_core [42430.581323] ? wake_bit_function+0x70/0x70 [42430.585846] i2c_algo_bit [42430.588900] ? ceph_write_iter+0x260/0x770 [ceph] [42430.593250] pps_core dca [42430.595955] ceph_write_iter+0x260/0x770 [ceph] [42430.600045] fuse [42430.602666] ? lock_is_held_type+0xaa/0x120 [42430.607359] irq event stamp: 259329638 [42430.609979] vfs_write+0x324/0x4f0 [42430.614499] hardirqs last enabled at (259329637): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42430.616429] ksys_pwrite64+0x65/0xa0 [42430.620608] hardirqs last disabled at (259329638): [] sysvec_apic_timer_interrupt+0xe/0x90 [42430.624355] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42430.627751] softirqs last enabled at (259329636): [] __do_softirq+0x2be/0x371 [42430.637998] do_syscall_64+0x38/0x80 [42430.641568] softirqs last disabled at (259329631): [] irq_exit_rcu+0xb0/0xd0 [42430.651377] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42430.656508] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42430.665279] RIP: 0033:0x7f48f0eeb93b [42430.668849] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42430.677448] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42430.682492] Workqueue: events delayed_work [ceph] [42430.691868] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42430.695440] [42430.695441] RIP: 0010:lock_acquire+0xf0/0x2b0 [42430.702911] ORIG_RAX: 0000000000000012 [42430.721647] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42430.726346] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42430.731563] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42430.733054] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42430.737404] [42430.737405] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42430.741234] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42430.759972] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42430.767096] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42430.772324] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42430.779447] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42430.780939] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42430.788068] [42430.795186] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42430.840133] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42430.848209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42430.853947] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42430.861072] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42430.868195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42430.875325] Call Trace: [42430.877773] [42430.879785] ? watchdog_timer_fn+0x1b4/0x220 [42430.884050] ? __hrtimer_run_queues+0x1d3/0x360 [42430.888575] ? hrtimer_interrupt+0x104/0x240 [42430.892844] ? __local_bh_enable+0x50/0x60 [42430.896938] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42430.902161] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42430.907117] [42430.909215] [42430.911327] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42430.916640] ? lock_acquire+0xf0/0x2b0 [42430.920390] ? igrab+0x43/0x50 [42430.923438] ? lock_release+0xc5/0x260 [42430.927184] _raw_spin_lock+0x30/0x40 [42430.930839] ? igrab+0x1d/0x50 [42430.933889] igrab+0x1d/0x50 [42430.936770] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42430.941919] delayed_work+0x1d8/0x280 [ceph] [42430.946200] ? process_scheduled_works+0x22a/0x510 [42430.950989] process_scheduled_works+0x2a4/0x510 [42430.955614] worker_thread+0x188/0x340 [42430.959362] ? rescuer_thread+0x310/0x310 [42430.963366] kthread+0xf5/0x130 [42430.966503] ? kthread_complete_and_exit+0x20/0x20 [42430.971289] ret_from_fork+0x31/0x40 [42430.974868] ? kthread_complete_and_exit+0x20/0x20 [42430.979651] ret_from_fork_asm+0x11/0x20 [42430.983575] [42458.195407] watchdog: BUG: soft lockup - CPU#0 stuck for 28428s! [dbench:78683] [42458.202709] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42458.207406] watchdog: BUG: soft lockup - CPU#7 stuck for 37425s! [kworker/7:4:78808] [42458.258156] irq event stamp: 76254444 [42458.258157] hardirqs last enabled at (76254443): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42458.265888] Modules linked in: [42458.269545] hardirqs last disabled at (76254444): [] sysvec_apic_timer_interrupt+0xe/0x90 [42458.279704] ceph [42458.282753] softirqs last enabled at (76254396): [] __do_softirq+0x2be/0x371 [42458.292476] libceph dns_resolver [42458.294402] softirqs last disabled at (76254391): [] irq_exit_rcu+0xb0/0xd0 [42458.303086] fscache netfs [42458.306398] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42458.314907] veth nft_chain_nat [42458.317610] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42458.326554] xt_MASQUERADE [42458.329693] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42458.337164] nf_nat nf_conntrack [42458.339868] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42458.345345] nf_defrag_ipv6 nf_defrag_ipv4 [42458.348569] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42458.367308] nft_compat [42458.371398] [42458.371399] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42458.376616] nf_tables [42458.379058] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42458.380567] nfnetlink bridge [42458.387692] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42458.390049] stp llc [42458.397173] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42458.400137] xfs libcrc32c [42458.407260] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42458.409447] sunrpc [42458.416569] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42458.419272] intel_rapl_msr intel_rapl_common [42458.426397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42458.428494] sb_edac x86_pkg_temp_thermal [42458.436573] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42458.440922] coretemp kvm_intel [42458.446661] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42458.450664] kvm irqbypass [42458.457804] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42458.460942] i2c_i801 iTCO_wdt [42458.468084] Call Trace: [42458.468085] [42458.470803] iTCO_vendor_support [42458.477932] ? watchdog_timer_fn+0x1b4/0x220 [42458.480982] i2c_smbus [42458.483425] ? __hrtimer_run_queues+0x1d3/0x360 [42458.485434] crct10dif_pclmul joydev [42458.488663] ? hrtimer_interrupt+0x104/0x240 [42458.492923] crc32_pclmul ipmi_ssif [42458.495282] ? __local_bh_enable+0x50/0x60 [42458.499804] ghash_clmulni_intel [42458.503378] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42458.507639] mei_me mei [42458.511125] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42458.515214] lpc_ich sha512_ssse3 [42458.518440] [42458.523655] acpi_ipmi mfd_core [42458.526101] [42458.526102] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42458.531057] wmi ipmi_si [42458.534373] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42458.536465] ipmi_devintf ipmi_msghandler [42458.539604] do_raw_spin_lock+0xb5/0xc0 [42458.541701] ioatdma acpi_pad [42458.547007] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42458.549536] acpi_power_meter ixgbe [42458.554410] try_get_cap_refs+0x159/0x880 [ceph] [42458.558411] igb mdio [42458.562247] __ceph_get_caps+0x4a1/0x760 [ceph] [42458.565206] crc32c_intel nvme [42458.569993] ? __file_remove_privs+0x89/0x110 [42458.573472] ptp nvme_core [42458.578088] ? wake_bit_function+0x70/0x70 [42458.580354] i2c_algo_bit pps_core [42458.584880] ? ceph_write_iter+0x260/0x770 [ceph] [42458.587930] dca fuse [42458.592280] ceph_write_iter+0x260/0x770 [ceph] [42458.594984] [42458.594985] irq event stamp: 259550346 [42458.599095] ? lock_is_held_type+0xaa/0x120 [42458.602489] hardirqs last enabled at (259550345): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42458.607189] vfs_write+0x324/0x4f0 [42458.609458] hardirqs last disabled at (259550346): [] sysvec_apic_timer_interrupt+0xe/0x90 [42458.613987] ksys_pwrite64+0x65/0xa0 [42458.615472] softirqs last enabled at (259550344): [] __do_softirq+0x2be/0x371 [42458.619219] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42458.623394] softirqs last disabled at (259550339): [] irq_exit_rcu+0xb0/0xd0 [42458.633639] do_syscall_64+0x38/0x80 [42458.637035] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42458.646847] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42458.650418] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42458.659190] RIP: 0033:0x7f48f0eeb93b [42458.664318] Workqueue: events delayed_work [ceph] [42458.672917] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42458.676487] [42458.676488] RIP: 0010:lock_acquire+0xf0/0x2b0 [42458.685864] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42458.690909] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42458.698378] ORIG_RAX: 0000000000000012 [42458.701950] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42458.706647] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42458.725386] [42458.725386] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42458.726876] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42458.731226] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42458.736444] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42458.755181] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42458.759014] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42458.764229] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42458.771354] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42458.772844] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42458.779976] [42458.787092] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42458.847240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42458.852978] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42458.860103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42458.867228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42458.874350] Call Trace: [42458.876813] [42458.878824] ? watchdog_timer_fn+0x1b4/0x220 [42458.883095] ? __hrtimer_run_queues+0x1d3/0x360 [42458.887622] ? hrtimer_interrupt+0x104/0x240 [42458.891883] ? __local_bh_enable+0x50/0x60 [42458.895977] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42458.901201] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42458.906159] [42458.908255] [42458.910354] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42458.915661] ? lock_acquire+0xf0/0x2b0 [42458.919412] ? igrab+0x43/0x50 [42458.922461] ? lock_release+0xc5/0x260 [42458.926206] _raw_spin_lock+0x30/0x40 [42458.929864] ? igrab+0x1d/0x50 [42458.932914] igrab+0x1d/0x50 [42458.935810] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42458.940959] delayed_work+0x1d8/0x280 [ceph] [42458.945239] ? process_scheduled_works+0x22a/0x510 [42458.950029] process_scheduled_works+0x2a4/0x510 [42458.954644] worker_thread+0x188/0x340 [42458.958394] ? rescuer_thread+0x310/0x310 [42458.962398] kthread+0xf5/0x130 [42458.965536] ? kthread_complete_and_exit+0x20/0x20 [42458.970320] ret_from_fork+0x31/0x40 [42458.973899] ? kthread_complete_and_exit+0x20/0x20 [42458.978683] ret_from_fork_asm+0x11/0x20 [42458.982614] [42485.850253] rcu: INFO: rcu_preempt self-detected stall on CPU [42485.856009] rcu: 7-....: (29659843 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/29658461 fqs=14957837 [42485.866688] rcu: (t=30548707 jiffies g=1309609 q=816124 ncpus=8) [42485.872772] Sending NMI from CPU 7 to CPUs 0: [42485.877125] NMI backtrace for cpu 0 [42485.877127] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42485.877129] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42485.877130] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42485.877133] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42485.877134] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42485.877136] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42485.877137] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42485.877138] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42485.877139] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42485.877140] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42485.877140] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42485.877142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42485.877142] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42485.877143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42485.877144] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42485.877145] Call Trace: [42485.877146] [42485.877147] ? nmi_cpu_backtrace+0xad/0x120 [42485.877151] ? nmi_cpu_backtrace_handler+0x11/0x20 [42485.877152] ? nmi_handle+0xd4/0x260 [42485.877156] ? default_do_nmi+0x49/0xf0 [42485.877158] ? exc_nmi+0x11e/0x150 [42485.877160] ? end_repeat_nmi+0x16/0x67 [42485.877164] ? queued_spin_lock_slowpath+0x16/0x2e0 [42485.877167] ? queued_spin_lock_slowpath+0x16/0x2e0 [42485.877169] ? queued_spin_lock_slowpath+0x16/0x2e0 [42485.877171] [42485.877172] [42485.877173] do_raw_spin_lock+0xb5/0xc0 [42485.877175] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42485.877191] try_get_cap_refs+0x159/0x880 [ceph] [42485.877205] __ceph_get_caps+0x4a1/0x760 [ceph] [42485.877218] ? __file_remove_privs+0x89/0x110 [42485.877220] ? wake_bit_function+0x70/0x70 [42485.877222] ? ceph_write_iter+0x260/0x770 [ceph] [42485.877233] ceph_write_iter+0x260/0x770 [ceph] [42485.877245] ? lock_is_held_type+0xaa/0x120 [42485.877247] vfs_write+0x324/0x4f0 [42485.877251] ksys_pwrite64+0x65/0xa0 [42485.877254] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42485.877258] do_syscall_64+0x38/0x80 [42485.877260] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42485.877262] RIP: 0033:0x7f48f0eeb93b [42485.877263] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42485.877265] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [42485.877266] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42485.877267] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42485.877268] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42485.877268] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42485.877269] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42485.877272] [42485.878123] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42486.181623] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42486.189094] Workqueue: events delayed_work [ceph] [42486.193807] RIP: 0010:lock_acquire+0xf0/0x2b0 [42486.198157] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42486.216896] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42486.222113] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42486.229236] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42486.236362] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42486.243485] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42486.250610] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42486.257735] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42486.265810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42486.271549] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42486.278674] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42486.285797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42486.292922] Call Trace: [42486.295366] [42486.297377] ? rcu_dump_cpu_stacks+0xd1/0x180 [42486.301738] ? rcu_sched_clock_irq+0x633/0x1390 [42486.306268] ? lock_release+0xc5/0x260 [42486.310014] ? lock_is_held_type+0xaa/0x120 [42486.314191] ? lock_is_held_type+0xaa/0x120 [42486.318369] ? tick_sched_do_timer+0xa0/0xa0 [42486.322641] ? rcu_read_lock_sched_held+0x56/0x90 [42486.327339] ? tick_sched_do_timer+0xa0/0xa0 [42486.331603] ? update_process_times+0x5f/0x90 [42486.335952] ? tick_sched_handle.isra.16+0x1e/0x50 [42486.340738] ? tick_sched_timer+0x6f/0x80 [42486.344741] ? __hrtimer_run_queues+0x1d3/0x360 [42486.349268] ? hrtimer_interrupt+0x104/0x240 [42486.353537] ? __local_bh_enable+0x50/0x60 [42486.357630] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42486.362855] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42486.367811] [42486.369910] [42486.372007] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42486.377314] ? lock_acquire+0xf0/0x2b0 [42486.381064] ? igrab+0x43/0x50 [42486.384113] ? lock_release+0xc5/0x260 [42486.387861] _raw_spin_lock+0x30/0x40 [42486.391524] ? igrab+0x1d/0x50 [42486.394576] igrab+0x1d/0x50 [42486.397453] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42486.402603] delayed_work+0x1d8/0x280 [ceph] [42486.406886] ? process_scheduled_works+0x22a/0x510 [42486.411674] process_scheduled_works+0x2a4/0x510 [42486.416290] worker_thread+0x188/0x340 [42486.420038] ? rescuer_thread+0x310/0x310 [42486.424043] kthread+0xf5/0x130 [42486.427182] ? kthread_complete_and_exit+0x20/0x20 [42486.431973] ret_from_fork+0x31/0x40 [42486.435552] ? kthread_complete_and_exit+0x20/0x20 [42486.440335] ret_from_fork_asm+0x11/0x20 [42486.444261] [42510.196311] watchdog: BUG: soft lockup - CPU#0 stuck for 28476s! [dbench:78683] [42510.203612] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42510.208311] watchdog: BUG: soft lockup - CPU#7 stuck for 37473s! [kworker/7:4:78808] [42510.259053] irq event stamp: 76363712 [42510.259055] hardirqs last enabled at (76363711): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42510.266782] Modules linked in: [42510.270441] hardirqs last disabled at (76363712): [] sysvec_apic_timer_interrupt+0xe/0x90 [42510.280599] ceph [42510.283650] softirqs last enabled at (76363642): [] __do_softirq+0x2be/0x371 [42510.293374] libceph [42510.295297] softirqs last disabled at (76363637): [] irq_exit_rcu+0xb0/0xd0 [42510.303993] dns_resolver [42510.306175] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42510.314684] fscache netfs [42510.317320] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42510.326263] veth nft_chain_nat [42510.328985] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42510.336456] xt_MASQUERADE nf_nat [42510.339593] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42510.345071] nf_conntrack nf_defrag_ipv6 [42510.348383] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42510.367118] nf_defrag_ipv4 [42510.371037] [42510.371038] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42510.376254] nft_compat [42510.379045] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42510.380536] nf_tables [42510.387661] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42510.390104] nfnetlink bridge [42510.397227] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42510.399585] stp llc [42510.406709] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42510.409673] xfs libcrc32c [42510.416798] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42510.418986] sunrpc intel_rapl_msr [42510.426107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42510.428810] intel_rapl_common sb_edac [42510.436887] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42510.440284] x86_pkg_temp_thermal [42510.446022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42510.449765] coretemp kvm_intel [42510.456890] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42510.460202] kvm irqbypass [42510.467326] Call Trace: [42510.467327] [42510.470463] i2c_i801 iTCO_wdt [42510.477588] ? watchdog_timer_fn+0x1b4/0x220 [42510.480290] iTCO_vendor_support i2c_smbus [42510.482735] ? __hrtimer_run_queues+0x1d3/0x360 [42510.484745] crct10dif_pclmul joydev [42510.487801] ? hrtimer_interrupt+0x104/0x240 [42510.492060] crc32_pclmul ipmi_ssif [42510.496151] ? __local_bh_enable+0x50/0x60 [42510.500676] ghash_clmulni_intel [42510.504249] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42510.508509] mei_me mei [42510.511999] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42510.516094] lpc_ich sha512_ssse3 [42510.519319] [42510.524536] acpi_ipmi [42510.526978] [42510.531954] mfd_core wmi [42510.535266] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42510.537361] ipmi_si ipmi_devintf [42510.539722] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42510.541817] ipmi_msghandler ioatdma [42510.544436] do_raw_spin_lock+0xb5/0xc0 [42510.549739] acpi_pad acpi_power_meter [42510.553049] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42510.557920] ixgbe igb [42510.561493] try_get_cap_refs+0x159/0x880 [ceph] [42510.565321] mdio crc32c_intel [42510.569069] __ceph_get_caps+0x4a1/0x760 [ceph] [42510.573849] nvme ptp [42510.576210] ? __file_remove_privs+0x89/0x110 [42510.580818] nvme_core [42510.583868] ? wake_bit_function+0x70/0x70 [42510.588393] i2c_algo_bit pps_core [42510.590664] ? ceph_write_iter+0x260/0x770 [ceph] [42510.595013] dca fuse [42510.597372] ceph_write_iter+0x260/0x770 [ceph] [42510.601462] [42510.601463] irq event stamp: 259960870 [42510.604864] ? lock_is_held_type+0xaa/0x120 [42510.609556] hardirqs last enabled at (259960869): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42510.611829] vfs_write+0x324/0x4f0 [42510.616352] hardirqs last disabled at (259960870): [] sysvec_apic_timer_interrupt+0xe/0x90 [42510.617847] ksys_pwrite64+0x65/0xa0 [42510.621586] softirqs last enabled at (259960868): [] __do_softirq+0x2be/0x371 [42510.625764] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42510.636008] softirqs last disabled at (259960863): [] irq_exit_rcu+0xb0/0xd0 [42510.639406] do_syscall_64+0x38/0x80 [42510.649216] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42510.652789] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42510.661558] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42510.666688] RIP: 0033:0x7f48f0eeb93b [42510.675303] Workqueue: events delayed_work [ceph] [42510.678873] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42510.688251] [42510.688252] RIP: 0010:lock_acquire+0xf0/0x2b0 [42510.693295] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42510.700765] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42510.704337] ORIG_RAX: 0000000000000012 [42510.709033] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42510.727772] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42510.729264] [42510.729265] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42510.733613] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42510.738832] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42510.757567] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42510.761400] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42510.766618] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42510.773740] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42510.775232] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42510.782356] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42510.789486] [42510.796604] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42510.849612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42510.855347] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42510.862472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42510.869596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42510.876719] Call Trace: [42510.879164] [42510.881177] ? watchdog_timer_fn+0x1b4/0x220 [42510.885441] ? __hrtimer_run_queues+0x1d3/0x360 [42510.889968] ? hrtimer_interrupt+0x104/0x240 [42510.894236] ? __local_bh_enable+0x50/0x60 [42510.898331] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42510.903552] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42510.908511] [42510.910609] [42510.912719] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42510.918030] ? lock_acquire+0xf0/0x2b0 [42510.921780] ? igrab+0x43/0x50 [42510.924831] ? lock_release+0xc5/0x260 [42510.928575] _raw_spin_lock+0x30/0x40 [42510.932231] ? igrab+0x1d/0x50 [42510.935283] igrab+0x1d/0x50 [42510.938160] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42510.943311] delayed_work+0x1d8/0x280 [ceph] [42510.947590] ? process_scheduled_works+0x22a/0x510 [42510.952373] process_scheduled_works+0x2a4/0x510 [42510.957002] worker_thread+0x188/0x340 [42510.960747] ? rescuer_thread+0x310/0x310 [42510.964750] kthread+0xf5/0x130 [42510.967886] ? kthread_complete_and_exit+0x20/0x20 [42510.972671] ret_from_fork+0x31/0x40 [42510.976243] ? kthread_complete_and_exit+0x20/0x20 [42510.981026] ret_from_fork_asm+0x11/0x20 [42510.984948] [42538.196800] watchdog: BUG: soft lockup - CPU#0 stuck for 28502s! [dbench:78683] [42538.204108] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42538.208800] watchdog: BUG: soft lockup - CPU#7 stuck for 37499s! [kworker/7:4:78808] [42538.259559] irq event stamp: 76421962 [42538.267288] Modules linked in: ceph libceph dns_resolver fscache netfs veth [42538.270946] hardirqs last enabled at (76421961): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42538.277905] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [42538.288062] hardirqs last disabled at (76421962): [] sysvec_apic_timer_interrupt+0xe/0x90 [42538.288066] nft_compat [42538.288067] softirqs last enabled at (76421866): [] __do_softirq+0x2be/0x371 [42538.296322] nf_tables [42538.306046] softirqs last disabled at (76421861): [] irq_exit_rcu+0xb0/0xd0 [42538.308496] nfnetlink bridge [42538.317197] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42538.319548] stp llc [42538.328059] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42538.331024] xfs libcrc32c [42538.339969] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42538.342153] sunrpc intel_rapl_msr [42538.349625] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42538.352328] intel_rapl_common sb_edac [42538.357805] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42538.361203] x86_pkg_temp_thermal coretemp [42538.379939] [42538.383683] kvm_intel [42538.388901] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42538.392992] kvm irqbypass [42538.394482] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42538.396839] i2c_i801 iTCO_wdt [42538.403964] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42538.406667] iTCO_vendor_support [42538.413792] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42538.416843] i2c_smbus crct10dif_pclmul [42538.423968] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42538.427197] joydev [42538.434324] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42538.438155] crc32_pclmul ipmi_ssif [42538.445279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42538.447376] ghash_clmulni_intel mei_me [42538.455455] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42538.458938] mei lpc_ich [42538.464676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42538.468506] sha512_ssse3 acpi_ipmi [42538.475631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42538.478160] mfd_core wmi [42538.485286] Call Trace: [42538.488769] ipmi_si ipmi_devintf [42538.495896] [42538.498509] ipmi_msghandler ioatdma [42538.500957] ? watchdog_timer_fn+0x1b4/0x220 [42538.504264] acpi_pad [42538.506278] ? __hrtimer_run_queues+0x1d3/0x360 [42538.509847] acpi_power_meter ixgbe [42538.514114] ? hrtimer_interrupt+0x104/0x240 [42538.516382] igb mdio [42538.520906] ? __local_bh_enable+0x50/0x60 [42538.524389] crc32c_intel nvme [42538.528655] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42538.530925] ptp nvme_core [42538.535017] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42538.538067] i2c_algo_bit [42538.543284] [42538.545987] pps_core dca [42538.550943] [42538.550945] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42538.553562] fuse [42538.555663] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42538.558275] irq event stamp: 260181658 [42538.560376] do_raw_spin_lock+0xb5/0xc0 [42538.565679] hardirqs last enabled at (260181657): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42538.567603] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42538.572486] hardirqs last disabled at (260181658): [] sysvec_apic_timer_interrupt+0xe/0x90 [42538.576237] try_get_cap_refs+0x159/0x880 [ceph] [42538.580066] softirqs last enabled at (260181656): [] __do_softirq+0x2be/0x371 [42538.590316] __ceph_get_caps+0x4a1/0x760 [ceph] [42538.595092] softirqs last disabled at (260181651): [] irq_exit_rcu+0xb0/0xd0 [42538.604909] ? __file_remove_privs+0x89/0x110 [42538.609514] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42538.618287] ? wake_bit_function+0x70/0x70 [42538.622810] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42538.631411] ? ceph_write_iter+0x260/0x770 [ceph] [42538.635758] Workqueue: events delayed_work [ceph] [42538.645137] ceph_write_iter+0x260/0x770 [ceph] [42538.649226] [42538.649227] RIP: 0010:lock_acquire+0xf0/0x2b0 [42538.656701] ? lock_is_held_type+0xaa/0x120 [42538.661394] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42538.666094] vfs_write+0x324/0x4f0 [42538.670617] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42538.672112] ksys_pwrite64+0x65/0xa0 [42538.676457] [42538.676458] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42538.680654] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42538.699389] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42538.702788] do_syscall_64+0x38/0x80 [42538.708004] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42538.711576] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42538.713066] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42538.720202] RIP: 0033:0x7f48f0eeb93b [42538.725331] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42538.732455] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42538.736025] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42538.743150] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42538.748203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42538.755325] ORIG_RAX: 0000000000000012 [42538.758896] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42538.766021] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42538.784760] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42538.792836] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42538.798052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42538.803792] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42538.807621] Call Trace: [42538.807622] [42538.814747] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42538.821871] ? watchdog_timer_fn+0x1b4/0x220 [42538.828994] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42538.836120] ? __hrtimer_run_queues+0x1d3/0x360 [42538.843247] [42538.850371] ? hrtimer_interrupt+0x104/0x240 [42538.884306] ? __local_bh_enable+0x50/0x60 [42538.888399] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42538.893624] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42538.898581] [42538.900678] [42538.902775] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42538.908083] ? lock_acquire+0xf0/0x2b0 [42538.911833] ? igrab+0x43/0x50 [42538.914884] ? lock_release+0xc5/0x260 [42538.918627] _raw_spin_lock+0x30/0x40 [42538.922284] ? igrab+0x1d/0x50 [42538.925336] igrab+0x1d/0x50 [42538.928213] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42538.933364] delayed_work+0x1d8/0x280 [ceph] [42538.937644] ? process_scheduled_works+0x22a/0x510 [42538.942433] process_scheduled_works+0x2a4/0x510 [42538.947051] worker_thread+0x188/0x340 [42538.950798] ? rescuer_thread+0x310/0x310 [42538.954803] kthread+0xf5/0x130 [42538.957940] ? kthread_complete_and_exit+0x20/0x20 [42538.962724] ret_from_fork+0x31/0x40 [42538.966303] ? kthread_complete_and_exit+0x20/0x20 [42538.971087] ret_from_fork_asm+0x11/0x20 [42538.975011] [42566.197289] watchdog: BUG: soft lockup - CPU#0 stuck for 28528s! [dbench:78683] [42566.204585] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42566.209289] watchdog: BUG: soft lockup - CPU#7 stuck for 37525s! [kworker/7:4:78808] [42566.260027] irq event stamp: 76480062 [42566.260028] hardirqs last enabled at (76480061): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42566.267758] Modules linked in: ceph [42566.271416] hardirqs last disabled at (76480062): [] sysvec_apic_timer_interrupt+0xe/0x90 [42566.281573] libceph [42566.285057] softirqs last enabled at (76480056): [] __do_softirq+0x2be/0x371 [42566.294782] dns_resolver fscache [42566.296965] softirqs last disabled at (76480051): [] irq_exit_rcu+0xb0/0xd0 [42566.305649] netfs veth [42566.308960] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42566.317472] nft_chain_nat xt_MASQUERADE [42566.319914] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42566.328860] nf_nat nf_conntrack [42566.332776] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42566.340247] nf_defrag_ipv6 [42566.343472] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42566.348948] nf_defrag_ipv4 nft_compat [42566.351741] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42566.370478] nf_tables nfnetlink [42566.374220] [42566.374221] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42566.379438] bridge [42566.382663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42566.384171] stp llc [42566.391295] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42566.393392] xfs libcrc32c [42566.400516] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42566.402701] sunrpc intel_rapl_msr [42566.409823] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42566.412529] intel_rapl_common sb_edac [42566.419653] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42566.423050] x86_pkg_temp_thermal coretemp [42566.430175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42566.433917] kvm_intel kvm [42566.441996] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42566.446087] irqbypass [42566.451824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42566.454528] i2c_i801 [42566.461651] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42566.464011] iTCO_wdt iTCO_vendor_support [42566.471135] Call Trace: [42566.471136] [42566.473404] i2c_smbus crct10dif_pclmul [42566.480530] ? watchdog_timer_fn+0x1b4/0x220 [42566.484532] joydev crc32_pclmul [42566.486977] ? __hrtimer_run_queues+0x1d3/0x360 [42566.488987] ipmi_ssif ghash_clmulni_intel [42566.492823] ? hrtimer_interrupt+0x104/0x240 [42566.497082] mei_me mei [42566.500307] ? __local_bh_enable+0x50/0x60 [42566.504831] lpc_ich sha512_ssse3 [42566.508925] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42566.513186] acpi_ipmi mfd_core [42566.515630] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42566.519720] wmi [42566.523031] [42566.528247] ipmi_si ipmi_devintf [42566.531386] [42566.531387] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42566.536342] ipmi_msghandler [42566.538185] ? queued_spin_lock_slowpath+0x16/0x2e0 [42566.540277] ioatdma acpi_pad [42566.543590] do_raw_spin_lock+0xb5/0xc0 [42566.545689] acpi_power_meter ixgbe [42566.550991] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42566.553867] igb mdio [42566.558740] try_get_cap_refs+0x159/0x880 [ceph] [42566.561702] crc32c_intel nvme [42566.565538] __ceph_get_caps+0x4a1/0x760 [ceph] [42566.569018] ptp nvme_core [42566.573804] ? __file_remove_privs+0x89/0x110 [42566.576073] i2c_algo_bit pps_core [42566.580690] ? wake_bit_function+0x70/0x70 [42566.583743] dca fuse [42566.588269] ? ceph_write_iter+0x260/0x770 [ceph] [42566.590971] [42566.590972] irq event stamp: 260402342 [42566.595322] ceph_write_iter+0x260/0x770 [ceph] [42566.598718] hardirqs last enabled at (260402341): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42566.602812] ? lock_is_held_type+0xaa/0x120 [42566.605080] hardirqs last disabled at (260402342): [] sysvec_apic_timer_interrupt+0xe/0x90 [42566.609779] vfs_write+0x324/0x4f0 [42566.611267] softirqs last enabled at (260402340): [] __do_softirq+0x2be/0x371 [42566.615016] ksys_pwrite64+0x65/0xa0 [42566.619536] softirqs last disabled at (260402335): [] irq_exit_rcu+0xb0/0xd0 [42566.629779] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42566.633959] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42566.643771] do_syscall_64+0x38/0x80 [42566.647167] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42566.655937] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42566.659508] Workqueue: events delayed_work [ceph] [42566.668105] RIP: 0033:0x7f48f0eeb93b [42566.673236] [42566.673237] RIP: 0010:lock_acquire+0xf0/0x2b0 [42566.682613] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42566.686184] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42566.693653] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42566.698700] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42566.703397] ORIG_RAX: 0000000000000012 [42566.706972] [42566.706972] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42566.708466] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42566.712815] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42566.731554] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42566.750291] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42566.755509] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42566.760726] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42566.764557] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42566.766048] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42566.773173] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42566.780296] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42566.787426] [42566.794544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42566.853306] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42566.860431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42566.867556] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42566.874693] Call Trace: [42566.877141] [42566.879153] ? watchdog_timer_fn+0x1b4/0x220 [42566.883415] ? __hrtimer_run_queues+0x1d3/0x360 [42566.887942] ? hrtimer_interrupt+0x104/0x240 [42566.892211] ? __local_bh_enable+0x50/0x60 [42566.896306] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42566.901529] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42566.906486] [42566.908582] [42566.910695] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42566.916007] ? lock_acquire+0xf0/0x2b0 [42566.919756] ? igrab+0x43/0x50 [42566.922807] ? lock_release+0xc5/0x260 [42566.926552] _raw_spin_lock+0x30/0x40 [42566.930216] ? igrab+0x1d/0x50 [42566.933266] igrab+0x1d/0x50 [42566.936146] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42566.941295] delayed_work+0x1d8/0x280 [ceph] [42566.945575] ? process_scheduled_works+0x22a/0x510 [42566.950358] process_scheduled_works+0x2a4/0x510 [42566.954988] worker_thread+0x188/0x340 [42566.958739] ? rescuer_thread+0x310/0x310 [42566.962744] kthread+0xf5/0x130 [42566.965879] ? kthread_complete_and_exit+0x20/0x20 [42566.970666] ret_from_fork+0x31/0x40 [42566.974261] ? kthread_complete_and_exit+0x20/0x20 [42566.979044] ret_from_fork_asm+0x11/0x20 [42566.982981] [42594.197778] watchdog: BUG: soft lockup - CPU#0 stuck for 28554s! [dbench:78683] [42594.205082] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42594.209778] watchdog: BUG: soft lockup - CPU#7 stuck for 37551s! [kworker/7:4:78808] [42594.260532] irq event stamp: 76538312 [42594.268262] Modules linked in: ceph libceph dns_resolver fscache netfs [42594.271919] hardirqs last enabled at (76538311): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42594.278450] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [42594.288611] hardirqs last disabled at (76538312): [] sysvec_apic_timer_interrupt+0xe/0x90 [42594.295995] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [42594.305720] softirqs last enabled at (76538214): [] __do_softirq+0x2be/0x371 [42594.312167] llc xfs libcrc32c sunrpc intel_rapl_msr [42594.320853] softirqs last disabled at (76538209): [] irq_exit_rcu+0xb0/0xd0 [42594.325810] intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel [42594.334321] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42594.341531] kvm irqbypass [42594.350475] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42594.350477] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42594.353179] i2c_i801 iTCO_wdt [42594.360652] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42594.366129] iTCO_vendor_support [42594.369179] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42594.387918] i2c_smbus crct10dif_pclmul [42594.391140] [42594.391141] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42594.396359] joydev crc32_pclmul [42594.400189] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42594.401681] ipmi_ssif [42594.408805] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42594.412027] ghash_clmulni_intel mei_me [42594.419170] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42594.421527] mei lpc_ich [42594.428652] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42594.432482] sha512_ssse3 acpi_ipmi [42594.439605] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42594.442138] mfd_core [42594.449261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42594.452744] wmi ipmi_si [42594.460821] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42594.463092] ipmi_devintf ipmi_msghandler [42594.468830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42594.471362] ioatdma acpi_pad [42594.478484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42594.482489] acpi_power_meter ixgbe [42594.489614] Call Trace: [42594.489617] [42594.492577] igb mdio [42594.499704] ? watchdog_timer_fn+0x1b4/0x220 [42594.503185] crc32c_intel [42594.505631] ? __hrtimer_run_queues+0x1d3/0x360 [42594.507640] nvme ptp [42594.509915] ? hrtimer_interrupt+0x104/0x240 [42594.514174] nvme_core [42594.516792] ? __local_bh_enable+0x50/0x60 [42594.521316] i2c_algo_bit pps_core [42594.523591] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42594.527850] dca fuse [42594.530208] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42594.534299] [42594.534299] irq event stamp: 260623112 [42594.537696] [42594.542913] hardirqs last enabled at (260623111): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42594.545186] [42594.550143] hardirqs last disabled at (260623112): [] sysvec_apic_timer_interrupt+0xe/0x90 [42594.551634] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42594.555376] softirqs last enabled at (260623110): [] __do_softirq+0x2be/0x371 [42594.557480] ? queued_spin_lock_slowpath+0x16/0x2e0 [42594.567719] softirqs last disabled at (260623105): [] irq_exit_rcu+0xb0/0xd0 [42594.569817] do_raw_spin_lock+0xb5/0xc0 [42594.579626] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42594.584934] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42594.593703] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42594.598575] try_get_cap_refs+0x159/0x880 [ceph] [42594.607179] Workqueue: events delayed_work [ceph] [42594.611014] __ceph_get_caps+0x4a1/0x760 [ceph] [42594.620386] [42594.625183] ? __file_remove_privs+0x89/0x110 [42594.632651] RIP: 0010:lock_acquire+0xf0/0x2b0 [42594.637261] ? wake_bit_function+0x70/0x70 [42594.641958] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42594.646486] ? ceph_write_iter+0x260/0x770 [ceph] [42594.647974] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42594.652325] ceph_write_iter+0x260/0x770 [ceph] [42594.656676] [42594.656676] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42594.660770] ? lock_is_held_type+0xaa/0x120 [42594.679503] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42594.684203] vfs_write+0x324/0x4f0 [42594.689417] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42594.693948] ksys_pwrite64+0x65/0xa0 [42594.695432] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42594.702576] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42594.706753] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42594.713878] do_syscall_64+0x38/0x80 [42594.717272] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42594.724399] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42594.727969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42594.735093] RIP: 0033:0x7f48f0eeb93b [42594.740224] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42594.747350] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42594.750917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42594.758996] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42594.764039] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42594.769776] ORIG_RAX: 0000000000000012 [42594.773347] Call Trace: [42594.773349] [42594.780473] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42594.799211] ? watchdog_timer_fn+0x1b4/0x220 [42594.806333] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42594.811553] ? __hrtimer_run_queues+0x1d3/0x360 [42594.818675] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42594.822510] ? hrtimer_interrupt+0x104/0x240 [42594.824949] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42594.826961] ? __local_bh_enable+0x50/0x60 [42594.834085] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42594.838353] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42594.845477] [42594.849998] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42594.892085] [42594.894181] [42594.896278] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42594.901588] ? lock_acquire+0xf0/0x2b0 [42594.905338] ? igrab+0x43/0x50 [42594.908388] ? lock_release+0xc5/0x260 [42594.912131] _raw_spin_lock+0x30/0x40 [42594.915789] ? igrab+0x1d/0x50 [42594.918837] igrab+0x1d/0x50 [42594.921718] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42594.926867] delayed_work+0x1d8/0x280 [ceph] [42594.931149] ? process_scheduled_works+0x22a/0x510 [42594.935937] process_scheduled_works+0x2a4/0x510 [42594.940553] worker_thread+0x188/0x340 [42594.944303] ? rescuer_thread+0x310/0x310 [42594.948305] kthread+0xf5/0x130 [42594.951445] ? kthread_complete_and_exit+0x20/0x20 [42594.956245] ret_from_fork+0x31/0x40 [42594.959825] ? kthread_complete_and_exit+0x20/0x20 [42594.964608] ret_from_fork_asm+0x11/0x20 [42594.968531] [42622.198267] watchdog: BUG: soft lockup - CPU#0 stuck for 28581s! [dbench:78683] [42622.205570] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42622.210267] watchdog: BUG: soft lockup - CPU#7 stuck for 37578s! [kworker/7:4:78808] [42622.261010] irq event stamp: 76596464 [42622.261011] hardirqs last enabled at (76596463): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42622.268742] Modules linked in: ceph [42622.272398] hardirqs last disabled at (76596464): [] sysvec_apic_timer_interrupt+0xe/0x90 [42622.282557] libceph [42622.286041] softirqs last enabled at (76596376): [] __do_softirq+0x2be/0x371 [42622.295764] dns_resolver [42622.297947] softirqs last disabled at (76596371): [] irq_exit_rcu+0xb0/0xd0 [42622.306634] fscache netfs [42622.309251] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42622.317761] veth [42622.320465] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42622.329408] nft_chain_nat xt_MASQUERADE [42622.331334] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42622.338805] nf_nat nf_conntrack [42622.342721] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42622.348200] nf_defrag_ipv6 nf_defrag_ipv4 [42622.351422] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42622.370161] nft_compat nf_tables [42622.374251] [42622.374252] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42622.379470] nfnetlink bridge [42622.382781] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42622.384270] stp llc [42622.391393] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42622.394359] xfs libcrc32c [42622.401483] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42622.403666] sunrpc intel_rapl_msr [42622.410792] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42622.413493] intel_rapl_common sb_edac [42622.420620] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42622.424017] x86_pkg_temp_thermal coretemp [42622.431140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42622.434885] kvm_intel kvm [42622.442961] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42622.447053] irqbypass i2c_i801 [42622.452789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42622.455493] iTCO_wdt [42622.462618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42622.465756] iTCO_vendor_support i2c_smbus [42622.472879] Call Trace: [42622.472880] [42622.475151] crct10dif_pclmul joydev [42622.482274] ? watchdog_timer_fn+0x1b4/0x220 [42622.486366] crc32_pclmul ipmi_ssif [42622.488809] ? __hrtimer_run_queues+0x1d3/0x360 [42622.490820] ghash_clmulni_intel mei_me [42622.494394] ? hrtimer_interrupt+0x104/0x240 [42622.498666] mei lpc_ich [42622.502148] ? __local_bh_enable+0x50/0x60 [42622.506671] sha512_ssse3 acpi_ipmi [42622.510504] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42622.514767] mfd_core wmi [42622.517299] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42622.521389] ipmi_si ipmi_devintf [42622.524872] [42622.530088] ipmi_msghandler ioatdma [42622.532707] [42622.537666] acpi_pad acpi_power_meter [42622.540976] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42622.543073] ixgbe igb [42622.546666] ? queued_spin_lock_slowpath+0x16/0x2e0 [42622.548757] mdio crc32c_intel [42622.552503] do_raw_spin_lock+0xb5/0xc0 [42622.557805] nvme ptp nvme_core [42622.560165] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42622.565035] i2c_algo_bit pps_core [42622.568088] try_get_cap_refs+0x159/0x880 [ceph] [42622.571915] dca fuse [42622.575057] __ceph_get_caps+0x4a1/0x760 [ceph] [42622.579838] [42622.583238] ? __file_remove_privs+0x89/0x110 [42622.587845] irq event stamp: 260844068 [42622.590117] ? wake_bit_function+0x70/0x70 [42622.594640] hardirqs last enabled at (260844067): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42622.596150] ? ceph_write_iter+0x260/0x770 [ceph] [42622.600498] hardirqs last disabled at (260844068): [] sysvec_apic_timer_interrupt+0xe/0x90 [42622.604243] ceph_write_iter+0x260/0x770 [ceph] [42622.608333] softirqs last enabled at (260844066): [] __do_softirq+0x2be/0x371 [42622.618582] ? lock_is_held_type+0xaa/0x120 [42622.623274] softirqs last disabled at (260844061): [] irq_exit_rcu+0xb0/0xd0 [42622.633089] vfs_write+0x324/0x4f0 [42622.637610] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42622.646385] ksys_pwrite64+0x65/0xa0 [42622.650558] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42622.659155] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42622.662553] Workqueue: events delayed_work [ceph] [42622.671949] do_syscall_64+0x38/0x80 [42622.675519] [42622.675520] RIP: 0010:lock_acquire+0xf0/0x2b0 [42622.682990] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42622.688120] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42622.692819] RIP: 0033:0x7f48f0eeb93b [42622.696388] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42622.697879] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42622.702230] [42622.702231] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42622.707274] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42622.726011] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42622.729584] ORIG_RAX: 0000000000000012 [42622.734800] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42622.753538] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42622.755029] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42622.762153] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42622.767369] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42622.774494] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42622.778326] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42622.785450] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42622.792572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42622.799696] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42622.806821] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42622.813956] [42622.821077] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42622.865575] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42622.872699] Call Trace: [42622.875143] [42622.877153] ? watchdog_timer_fn+0x1b4/0x220 [42622.881417] ? __hrtimer_run_queues+0x1d3/0x360 [42622.885955] ? hrtimer_interrupt+0x104/0x240 [42622.890221] ? __local_bh_enable+0x50/0x60 [42622.894315] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42622.899539] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42622.904497] [42622.906593] [42622.908691] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42622.914001] ? lock_acquire+0xf0/0x2b0 [42622.917750] ? igrab+0x43/0x50 [42622.920799] ? lock_release+0xc5/0x260 [42622.924546] _raw_spin_lock+0x30/0x40 [42622.928210] ? igrab+0x1d/0x50 [42622.931259] igrab+0x1d/0x50 [42622.934138] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42622.939288] delayed_work+0x1d8/0x280 [ceph] [42622.943570] ? process_scheduled_works+0x22a/0x510 [42622.948360] process_scheduled_works+0x2a4/0x510 [42622.952974] worker_thread+0x188/0x340 [42622.956724] ? rescuer_thread+0x310/0x310 [42622.960728] kthread+0xf5/0x130 [42622.963864] ? kthread_complete_and_exit+0x20/0x20 [42622.968651] ret_from_fork+0x31/0x40 [42622.972246] ? kthread_complete_and_exit+0x20/0x20 [42622.977030] ret_from_fork_asm+0x11/0x20 [42622.980968] [42650.198756] watchdog: BUG: soft lockup - CPU#0 stuck for 28607s! [dbench:78683] [42650.206058] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42650.210755] watchdog: BUG: soft lockup - CPU#7 stuck for 37604s! [kworker/7:4:78808] [42650.261506] irq event stamp: 76654708 [42650.261508] hardirqs last enabled at (76654707): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42650.269238] Modules linked in: [42650.272896] hardirqs last disabled at (76654708): [] sysvec_apic_timer_interrupt+0xe/0x90 [42650.283052] ceph [42650.286103] softirqs last enabled at (76654652): [] __do_softirq+0x2be/0x371 [42650.295827] libceph [42650.297752] softirqs last disabled at (76654647): [] irq_exit_rcu+0xb0/0xd0 [42650.306434] dns_resolver [42650.308619] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42650.317129] fscache [42650.319764] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42650.328711] netfs [42650.330895] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42650.338364] veth nft_chain_nat [42650.340374] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42650.345852] xt_MASQUERADE [42650.348990] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42650.367727] nf_nat nf_conntrack [42650.370432] [42650.370433] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42650.375667] nf_defrag_ipv6 nf_defrag_ipv4 [42650.378890] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42650.380380] nft_compat nf_tables [42650.387505] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42650.391595] nfnetlink bridge [42650.398719] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42650.402031] stp llc [42650.409154] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42650.412119] xfs libcrc32c [42650.419242] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42650.421433] sunrpc intel_rapl_msr [42650.428559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42650.431264] intel_rapl_common sb_edac [42650.439341] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42650.442738] x86_pkg_temp_thermal coretemp [42650.448477] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42650.452220] kvm_intel kvm [42650.459343] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42650.463435] irqbypass i2c_i801 [42650.470559] Call Trace: [42650.470561] [42650.473263] iTCO_wdt iTCO_vendor_support [42650.480388] ? watchdog_timer_fn+0x1b4/0x220 [42650.483524] i2c_smbus crct10dif_pclmul [42650.485970] ? __hrtimer_run_queues+0x1d3/0x360 [42650.487980] joydev crc32_pclmul [42650.491988] ? hrtimer_interrupt+0x104/0x240 [42650.496248] ipmi_ssif ghash_clmulni_intel [42650.500079] ? __local_bh_enable+0x50/0x60 [42650.504601] mei_me mei [42650.507829] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42650.512091] lpc_ich sha512_ssse3 [42650.516182] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42650.520271] acpi_ipmi mfd_core [42650.522716] [42650.527934] wmi ipmi_si [42650.531244] [42650.536200] ipmi_devintf ipmi_msghandler ioatdma [42650.539340] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42650.541437] acpi_pad [42650.543972] ? queued_spin_lock_slowpath+0x16/0x2e0 [42650.546064] acpi_power_meter ixgbe [42650.550764] do_raw_spin_lock+0xb5/0xc0 [42650.556066] igb mdio [42650.558339] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42650.563208] crc32c_intel nvme [42650.566694] try_get_cap_refs+0x159/0x880 [ceph] [42650.570522] ptp nvme_core [42650.572799] __ceph_get_caps+0x4a1/0x760 [ceph] [42650.577577] i2c_algo_bit pps_core [42650.580632] ? __file_remove_privs+0x89/0x110 [42650.585238] dca fuse [42650.587943] ? wake_bit_function+0x70/0x70 [42650.592466] [42650.592466] irq event stamp: 261064880 [42650.595866] ? ceph_write_iter+0x260/0x770 [ceph] [42650.600215] hardirqs last enabled at (261064879): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42650.602485] ceph_write_iter+0x260/0x770 [ceph] [42650.606576] hardirqs last disabled at (261064880): [] sysvec_apic_timer_interrupt+0xe/0x90 [42650.608071] ? lock_is_held_type+0xaa/0x120 [42650.611811] softirqs last enabled at (261064878): [] __do_softirq+0x2be/0x371 [42650.616510] vfs_write+0x324/0x4f0 [42650.626753] softirqs last disabled at (261064873): [] irq_exit_rcu+0xb0/0xd0 [42650.631283] ksys_pwrite64+0x65/0xa0 [42650.641087] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42650.645265] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42650.654036] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42650.657437] do_syscall_64+0x38/0x80 [42650.666031] Workqueue: events delayed_work [ceph] [42650.669604] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42650.678979] [42650.678980] RIP: 0010:lock_release+0x14e/0x260 [42650.684109] RIP: 0033:0x7f48f0eeb93b [42650.691580] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [42650.695158] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42650.699857] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [42650.704900] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42650.706392] [42650.706393] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [42650.710830] ORIG_RAX: 0000000000000012 [42650.714399] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [42650.733156] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42650.751894] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [42650.757109] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42650.762328] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [42650.763819] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42650.770941] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [42650.774773] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42650.781898] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42650.789021] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42650.796158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42650.803283] [42650.810403] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42650.862024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42650.869159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42650.876289] Call Trace: [42650.878732] [42650.880744] ? watchdog_timer_fn+0x1b4/0x220 [42650.885008] ? __hrtimer_run_queues+0x1d3/0x360 [42650.889534] ? hrtimer_interrupt+0x104/0x240 [42650.893795] ? __local_bh_enable+0x50/0x60 [42650.897888] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42650.903112] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42650.908069] [42650.910167] [42650.912265] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42650.917570] ? igrab+0x43/0x50 [42650.920622] ? lock_release+0x14e/0x260 [42650.924460] ? lock_release+0xc5/0x260 [42650.928206] _raw_spin_unlock+0x1b/0x40 [42650.932042] igrab+0x43/0x50 [42650.934921] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42650.940070] delayed_work+0x1d8/0x280 [ceph] [42650.944352] ? process_scheduled_works+0x22a/0x510 [42650.949158] process_scheduled_works+0x2a4/0x510 [42650.953775] worker_thread+0x188/0x340 [42650.957523] ? rescuer_thread+0x310/0x310 [42650.961527] kthread+0xf5/0x130 [42650.964673] ? kthread_complete_and_exit+0x20/0x20 [42650.969458] ret_from_fork+0x31/0x40 [42650.973036] ? kthread_complete_and_exit+0x20/0x20 [42650.977822] ret_from_fork_asm+0x11/0x20 [42650.981754] [42678.199243] watchdog: BUG: soft lockup - CPU#0 stuck for 28633s! [dbench:78683] [42678.206545] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42678.211243] watchdog: BUG: soft lockup - CPU#7 stuck for 37630s! [kworker/7:4:78808] [42678.261984] irq event stamp: 76712926 [42678.261985] hardirqs last enabled at (76712925): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42678.269716] Modules linked in: [42678.273374] hardirqs last disabled at (76712926): [] sysvec_apic_timer_interrupt+0xe/0x90 [42678.283531] ceph [42678.286582] softirqs last enabled at (76712850): [] __do_softirq+0x2be/0x371 [42678.296305] libceph dns_resolver [42678.298231] softirqs last disabled at (76712845): [] irq_exit_rcu+0xb0/0xd0 [42678.306917] fscache [42678.310226] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42678.318735] netfs veth [42678.320935] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42678.329882] nft_chain_nat [42678.332326] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [42678.339797] xt_MASQUERADE nf_nat [42678.342499] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [42678.347977] nf_conntrack nf_defrag_ipv6 [42678.351287] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42678.370026] nf_defrag_ipv4 nft_compat [42678.373944] [42678.373945] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42678.379160] nf_tables nfnetlink [42678.382920] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42678.384412] bridge stp [42678.391536] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42678.394762] llc xfs [42678.401885] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42678.404330] libcrc32c sunrpc [42678.411454] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42678.413639] intel_rapl_msr [42678.420760] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42678.423726] intel_rapl_common sb_edac [42678.430849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42678.433640] x86_pkg_temp_thermal coretemp [42678.441718] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42678.445463] kvm_intel kvm [42678.451201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42678.455290] irqbypass i2c_i801 [42678.462414] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42678.465118] iTCO_wdt [42678.472243] Call Trace: [42678.472244] [42678.475381] iTCO_vendor_support i2c_smbus [42678.482504] ? watchdog_timer_fn+0x1b4/0x220 [42678.484774] crct10dif_pclmul joydev [42678.487220] ? __hrtimer_run_queues+0x1d3/0x360 [42678.489229] crc32_pclmul ipmi_ssif [42678.493324] ? hrtimer_interrupt+0x104/0x240 [42678.497585] ghash_clmulni_intel mei_me [42678.501155] ? __local_bh_enable+0x50/0x60 [42678.505678] mei lpc_ich [42678.509165] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42678.513427] sha512_ssse3 acpi_ipmi [42678.517259] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42678.521348] mfd_core wmi [42678.523880] [42678.529096] ipmi_si [42678.532581] [42678.537537] ipmi_devintf ipmi_msghandler [42678.540157] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42678.542253] ioatdma acpi_pad [42678.544442] ? queued_spin_lock_slowpath+0x16/0x2e0 [42678.546533] acpi_power_meter ixgbe [42678.550541] do_raw_spin_lock+0xb5/0xc0 [42678.555842] igb mdio [42678.558809] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42678.563678] crc32c_intel [42678.567164] try_get_cap_refs+0x159/0x880 [ceph] [42678.570993] nvme ptp [42678.573267] __ceph_get_caps+0x4a1/0x760 [ceph] [42678.578048] nvme_core i2c_algo_bit [42678.580669] ? __file_remove_privs+0x89/0x110 [42678.585276] pps_core dca [42678.587547] ? wake_bit_function+0x70/0x70 [42678.592071] fuse [42678.595557] ? ceph_write_iter+0x260/0x770 [ceph] [42678.599921] irq event stamp: 261285686 [42678.602540] ceph_write_iter+0x260/0x770 [ceph] [42678.606644] hardirqs last enabled at (261285685): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42678.608576] ? lock_is_held_type+0xaa/0x120 [42678.613269] hardirqs last disabled at (261285686): [] sysvec_apic_timer_interrupt+0xe/0x90 [42678.617016] vfs_write+0x324/0x4f0 [42678.621538] softirqs last enabled at (261285684): [] __do_softirq+0x2be/0x371 [42678.631785] ksys_pwrite64+0x65/0xa0 [42678.635960] softirqs last disabled at (261285679): [] irq_exit_rcu+0xb0/0xd0 [42678.645768] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42678.649166] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42678.657938] do_syscall_64+0x38/0x80 [42678.661508] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42678.670106] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42678.675238] Workqueue: events delayed_work [ceph] [42678.684616] RIP: 0033:0x7f48f0eeb93b [42678.688204] [42678.688204] RIP: 0010:lock_release+0x14e/0x260 [42678.695673] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42678.700717] Code: 84 bd 00 00 00 b8 ff ff ff ff 65 0f c1 05 0a dc ea 7e 83 f8 01 0f 85 ee 00 00 00 41 f7 c5 00 02 00 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 10 65 48 33 04 25 28 00 00 00 0f 85 f8 00 00 00 48 83 [42678.705416] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42678.708984] RSP: 0018:ffffc90002117d48 EFLAGS: 00000206 [42678.710476] ORIG_RAX: 0000000000000012 [42678.714912] [42678.714913] RAX: 0000000000000001 RBX: ffff88810debaa80 RCX: ffffc90002117d54 [42678.733668] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42678.752406] RDX: 0000000000000003 RSI: ffff8881a1d34518 RDI: ffff88810debb550 [42678.757622] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42678.762858] RBP: ffff8881a1d34518 R08: 0000000000005c53 R09: 0000000000000003 [42678.766689] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42678.768181] R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff813b9433 [42678.775303] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42678.782428] R13: 0000000000000246 R14: 0000000000000004 R15: ffff888100ee2000 [42678.789553] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42678.796677] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42678.803806] [42678.810925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42678.855438] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42678.862562] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42678.869686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42678.876811] Call Trace: [42678.879255] [42678.881265] ? watchdog_timer_fn+0x1b4/0x220 [42678.885529] ? __hrtimer_run_queues+0x1d3/0x360 [42678.890058] ? hrtimer_interrupt+0x104/0x240 [42678.894327] ? __local_bh_enable+0x50/0x60 [42678.898418] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42678.903649] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42678.908610] [42678.910707] [42678.912805] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42678.918108] ? igrab+0x43/0x50 [42678.921162] ? lock_release+0x14e/0x260 [42678.924998] ? lock_release+0xc5/0x260 [42678.928744] _raw_spin_unlock+0x1b/0x40 [42678.932581] igrab+0x43/0x50 [42678.935460] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42678.940608] delayed_work+0x1d8/0x280 [ceph] [42678.944908] ? process_scheduled_works+0x22a/0x510 [42678.949699] process_scheduled_works+0x2a4/0x510 [42678.954313] worker_thread+0x188/0x340 [42678.958062] ? rescuer_thread+0x310/0x310 [42678.962066] kthread+0xf5/0x130 [42678.965204] ? kthread_complete_and_exit+0x20/0x20 [42678.969989] ret_from_fork+0x31/0x40 [42678.973557] ? kthread_complete_and_exit+0x20/0x20 [42678.978342] ret_from_fork_asm+0x11/0x20 [42678.982265] [42681.451667] rcu: INFO: rcu_preempt self-detected stall on CPU [42681.457428] rcu: 7-....: (29849449 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/29848066 fqs=15053718 [42681.468114] rcu: (t=30744305 jiffies g=1309609 q=819937 ncpus=8) [42681.474199] Sending NMI from CPU 7 to CPUs 0: [42681.478549] NMI backtrace for cpu 0 [42681.478550] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42681.478552] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42681.478553] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42681.478556] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42681.478557] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42681.478559] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42681.478560] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42681.478560] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42681.478561] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42681.478562] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42681.478563] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42681.478564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42681.478565] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42681.478566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42681.478567] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42681.478567] Call Trace: [42681.478568] [42681.478569] ? nmi_cpu_backtrace+0xad/0x120 [42681.478573] ? nmi_cpu_backtrace_handler+0x11/0x20 [42681.478574] ? nmi_handle+0xd4/0x260 [42681.478578] ? default_do_nmi+0x49/0xf0 [42681.478580] ? exc_nmi+0x11e/0x150 [42681.478582] ? end_repeat_nmi+0x16/0x67 [42681.478586] ? queued_spin_lock_slowpath+0x18/0x2e0 [42681.478588] ? queued_spin_lock_slowpath+0x18/0x2e0 [42681.478591] ? queued_spin_lock_slowpath+0x18/0x2e0 [42681.478593] [42681.478593] [42681.478594] do_raw_spin_lock+0xb5/0xc0 [42681.478596] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42681.478611] try_get_cap_refs+0x159/0x880 [ceph] [42681.478625] __ceph_get_caps+0x4a1/0x760 [ceph] [42681.478638] ? __file_remove_privs+0x89/0x110 [42681.478640] ? wake_bit_function+0x70/0x70 [42681.478642] ? ceph_write_iter+0x260/0x770 [ceph] [42681.478653] ceph_write_iter+0x260/0x770 [ceph] [42681.478664] ? lock_is_held_type+0xaa/0x120 [42681.478669] vfs_write+0x324/0x4f0 [42681.478674] ksys_pwrite64+0x65/0xa0 [42681.478676] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42681.478679] do_syscall_64+0x38/0x80 [42681.478681] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42681.478683] RIP: 0033:0x7f48f0eeb93b [42681.478684] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42681.478686] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [42681.478687] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42681.478688] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42681.478688] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42681.478689] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42681.478690] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42681.478693] [42681.479547] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42681.783040] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42681.790511] Workqueue: events delayed_work [ceph] [42681.795226] RIP: 0010:lock_acquire+0x76/0x2b0 [42681.799576] Code: 2b 01 00 00 8b 15 c2 08 54 01 85 d2 0f 84 92 00 00 00 65 8b 05 db d9 ea 7e 85 c0 0f 85 2a 01 00 00 65 48 8b 04 25 c0 e5 02 00 <8b> 80 54 0a 00 00 85 c0 0f 85 13 01 00 00 9c 58 0f 1f 40 00 48 89 [42681.818315] RSP: 0018:ffffc90002117d18 EFLAGS: 00000246 [42681.823532] RAX: ffff88810debaa80 RBX: ffff8881a1d34500 RCX: 0000000000000000 [42681.830655] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881a1d34518 [42681.837778] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000 [42681.844903] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8881a1d34518 [42681.852028] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42681.859151] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42681.867229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42681.872967] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42681.880091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42681.887216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42681.894339] Call Trace: [42681.896785] [42681.898795] ? rcu_dump_cpu_stacks+0xd1/0x180 [42681.903154] ? rcu_sched_clock_irq+0x633/0x1390 [42681.907679] ? lock_release+0xc5/0x260 [42681.911421] ? lock_is_held_type+0xaa/0x120 [42681.915599] ? lock_is_held_type+0xaa/0x120 [42681.919778] ? tick_sched_do_timer+0xa0/0xa0 [42681.924050] ? rcu_read_lock_sched_held+0x56/0x90 [42681.928747] ? tick_sched_do_timer+0xa0/0xa0 [42681.933010] ? update_process_times+0x5f/0x90 [42681.937362] ? tick_sched_handle.isra.16+0x1e/0x50 [42681.942145] ? tick_sched_timer+0x6f/0x80 [42681.946151] ? __hrtimer_run_queues+0x1d3/0x360 [42681.950680] ? hrtimer_interrupt+0x104/0x240 [42681.954946] ? __local_bh_enable+0x50/0x60 [42681.959041] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42681.964264] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42681.969221] [42681.971317] [42681.973417] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42681.978724] ? lock_acquire+0x76/0x2b0 [42681.982473] ? lock_acquire+0x184/0x2b0 [42681.986303] ? igrab+0x43/0x50 [42681.989355] ? lock_release+0xc5/0x260 [42681.993100] _raw_spin_lock+0x30/0x40 [42681.996765] ? igrab+0x1d/0x50 [42681.999814] igrab+0x1d/0x50 [42682.002692] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42682.007844] delayed_work+0x1d8/0x280 [ceph] [42682.012125] ? process_scheduled_works+0x22a/0x510 [42682.016913] process_scheduled_works+0x2a4/0x510 [42682.021529] worker_thread+0x188/0x340 [42682.025279] ? rescuer_thread+0x310/0x310 [42682.029282] kthread+0xf5/0x130 [42682.032419] ? kthread_complete_and_exit+0x20/0x20 [42682.037205] ret_from_fork+0x31/0x40 [42682.040783] ? kthread_complete_and_exit+0x20/0x20 [42682.045566] ret_from_fork_asm+0x11/0x20 [42682.049491] [42706.199732] watchdog: BUG: soft lockup - CPU#0 stuck for 28659s! [dbench:78683] [42706.207031] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42706.211732] watchdog: BUG: soft lockup - CPU#7 stuck for 37656s! [kworker/7:4:78808] [42706.262481] irq event stamp: 76771124 [42706.262483] hardirqs last enabled at (76771123): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42706.270212] Modules linked in: [42706.273868] hardirqs last disabled at (76771124): [] sysvec_apic_timer_interrupt+0xe/0x90 [42706.284026] ceph [42706.287077] softirqs last enabled at (76771086): [] __do_softirq+0x2be/0x371 [42706.296802] libceph [42706.298725] softirqs last disabled at (76771081): [] irq_exit_rcu+0xb0/0xd0 [42706.307413] dns_resolver fscache [42706.309596] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42706.318123] netfs [42706.321433] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42706.330377] veth nft_chain_nat [42706.332389] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42706.339859] xt_MASQUERADE [42706.342997] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42706.348473] nf_nat nf_conntrack [42706.351177] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42706.369916] nf_defrag_ipv6 nf_defrag_ipv4 [42706.373139] [42706.378356] nft_compat nf_tables [42706.382446] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42706.383939] nfnetlink bridge [42706.387248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42706.394374] stp llc [42706.397336] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42706.404462] xfs libcrc32c [42706.406646] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42706.413770] sunrpc [42706.416474] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42706.423598] intel_rapl_msr [42706.425695] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42706.432818] intel_rapl_common sb_edac [42706.435610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42706.443686] x86_pkg_temp_thermal coretemp [42706.447432] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42706.453169] kvm_intel kvm [42706.457260] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42706.464383] irqbypass i2c_i801 [42706.467088] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42706.474211] iTCO_wdt [42706.477348] Call Trace: [42706.477351] [42706.484473] iTCO_vendor_support i2c_smbus [42706.486745] ? watchdog_timer_fn+0x1b4/0x220 [42706.489188] crct10dif_pclmul joydev [42706.491199] ? __hrtimer_run_queues+0x1d3/0x360 [42706.495289] crc32_pclmul [42706.499557] ? hrtimer_interrupt+0x104/0x240 [42706.503129] ipmi_ssif ghash_clmulni_intel [42706.507658] ? __local_bh_enable+0x50/0x60 [42706.510273] mei_me mei [42706.514540] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42706.518630] lpc_ich [42706.522719] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42706.525164] sha512_ssse3 [42706.530382] [42706.532566] acpi_ipmi mfd_core [42706.537524] [42706.540139] wmi ipmi_si [42706.542238] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42706.545376] ipmi_devintf ipmi_msghandler [42706.547477] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42706.550004] ioatdma [42706.555309] do_raw_spin_lock+0xb5/0xc0 [42706.559311] acpi_pad acpi_power_meter ixgbe [42706.564184] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42706.566365] igb mdio [42706.570199] try_get_cap_refs+0x159/0x880 [ceph] [42706.574460] crc32c_intel nvme [42706.579250] __ceph_get_caps+0x4a1/0x760 [ceph] [42706.581515] ptp nvme_core [42706.586135] ? __file_remove_privs+0x89/0x110 [42706.589177] i2c_algo_bit pps_core [42706.593702] ? wake_bit_function+0x70/0x70 [42706.596406] dca fuse [42706.600775] ? ceph_write_iter+0x260/0x770 [ceph] [42706.604171] [42706.608262] ceph_write_iter+0x260/0x770 [ceph] [42706.610532] irq event stamp: 261501660 [42706.615234] ? lock_is_held_type+0xaa/0x120 [42706.616721] hardirqs last enabled at (261501659): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42706.621246] vfs_write+0x324/0x4f0 [42706.624990] hardirqs last disabled at (261501660): [] sysvec_apic_timer_interrupt+0xe/0x90 [42706.629170] ksys_pwrite64+0x65/0xa0 [42706.639413] softirqs last enabled at (261501658): [] __do_softirq+0x2be/0x371 [42706.642808] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42706.652619] softirqs last disabled at (261501653): [] irq_exit_rcu+0xb0/0xd0 [42706.656190] do_syscall_64+0x38/0x80 [42706.664961] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42706.670093] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42706.678689] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42706.682260] RIP: 0033:0x7f48f0eeb93b [42706.691636] Workqueue: events delayed_work [ceph] [42706.696683] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42706.704152] [42706.704152] RIP: 0010:lock_acquire+0xf0/0x2b0 [42706.707721] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42706.712419] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42706.731156] ORIG_RAX: 0000000000000012 [42706.732647] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42706.736999] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42706.742215] [42706.742215] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42706.760952] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42706.764785] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42706.770003] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42706.777136] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42706.778625] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42706.785751] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42706.792875] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42706.799997] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42706.807140] [42706.814255] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42706.853014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42706.858750] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42706.865874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42706.872999] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42706.880138] Call Trace: [42706.882585] [42706.884597] ? watchdog_timer_fn+0x1b4/0x220 [42706.888859] ? __hrtimer_run_queues+0x1d3/0x360 [42706.893388] ? hrtimer_interrupt+0x104/0x240 [42706.897656] ? __local_bh_enable+0x50/0x60 [42706.901750] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42706.906973] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42706.911930] [42706.914027] [42706.916140] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42706.921450] ? lock_acquire+0xf0/0x2b0 [42706.925201] ? igrab+0x43/0x50 [42706.928251] ? lock_release+0xc5/0x260 [42706.931996] _raw_spin_lock+0x30/0x40 [42706.935661] ? igrab+0x1d/0x50 [42706.938712] igrab+0x1d/0x50 [42706.941591] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42706.946738] delayed_work+0x1d8/0x280 [ceph] [42706.951020] ? process_scheduled_works+0x22a/0x510 [42706.955811] process_scheduled_works+0x2a4/0x510 [42706.960426] worker_thread+0x188/0x340 [42706.964175] ? rescuer_thread+0x310/0x310 [42706.968177] kthread+0xf5/0x130 [42706.971317] ? kthread_complete_and_exit+0x20/0x20 [42706.976101] ret_from_fork+0x31/0x40 [42706.979679] ? kthread_complete_and_exit+0x20/0x20 [42706.984464] ret_from_fork_asm+0x11/0x20 [42706.988396] [42734.200222] watchdog: BUG: soft lockup - CPU#0 stuck for 28685s! [dbench:78683] [42734.207529] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42734.212222] watchdog: BUG: soft lockup - CPU#7 stuck for 37682s! [kworker/7:4:78808] [42734.262969] irq event stamp: 76829330 [42734.262970] hardirqs last enabled at (76829329): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42734.270699] Modules linked in: [42734.274358] hardirqs last disabled at (76829330): [] sysvec_apic_timer_interrupt+0xe/0x90 [42734.284513] ceph [42734.287564] softirqs last enabled at (76829264): [] __do_softirq+0x2be/0x371 [42734.297288] libceph [42734.299212] softirqs last disabled at (76829259): [] irq_exit_rcu+0xb0/0xd0 [42734.307897] dns_resolver fscache [42734.310081] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42734.318593] netfs veth [42734.321921] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42734.330865] nft_chain_nat xt_MASQUERADE [42734.333308] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42734.340778] nf_nat nf_conntrack [42734.344696] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42734.350174] nf_defrag_ipv6 nf_defrag_ipv4 [42734.353398] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42734.372135] nft_compat nf_tables [42734.376226] [42734.376227] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42734.381443] nfnetlink bridge [42734.384755] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42734.386246] stp [42734.393369] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42734.396334] llc xfs [42734.403458] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42734.405295] libcrc32c sunrpc [42734.412418] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42734.414618] intel_rapl_msr [42734.421746] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42734.424709] intel_rapl_common [42734.431834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42734.434623] sb_edac x86_pkg_temp_thermal [42734.442700] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42734.445751] coretemp kvm_intel [42734.451489] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42734.455493] kvm irqbypass [42734.462618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42734.465756] i2c_i801 iTCO_wdt [42734.472878] Call Trace: [42734.472879] [42734.475618] iTCO_vendor_support i2c_smbus [42734.482742] ? watchdog_timer_fn+0x1b4/0x220 [42734.485792] crct10dif_pclmul joydev [42734.488237] ? __hrtimer_run_queues+0x1d3/0x360 [42734.490248] crc32_pclmul ipmi_ssif [42734.494341] ? hrtimer_interrupt+0x104/0x240 [42734.498619] ghash_clmulni_intel [42734.502191] ? __local_bh_enable+0x50/0x60 [42734.506713] mei_me mei [42734.510200] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42734.514461] lpc_ich sha512_ssse3 [42734.517686] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42734.521778] acpi_ipmi [42734.524222] [42734.529440] mfd_core wmi [42734.532750] [42734.537708] ipmi_si ipmi_devintf [42734.540065] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42734.542162] ipmi_msghandler ioatdma [42734.544783] ? queued_spin_lock_slowpath+0x18/0x2e0 [42734.546875] acpi_pad acpi_power_meter [42734.550188] do_raw_spin_lock+0xb5/0xc0 [42734.555491] ixgbe igb [42734.559064] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42734.563932] mdio crc32c_intel [42734.567680] try_get_cap_refs+0x159/0x880 [ceph] [42734.571506] nvme ptp [42734.573871] __ceph_get_caps+0x4a1/0x760 [ceph] [42734.578650] nvme_core [42734.581705] ? __file_remove_privs+0x89/0x110 [42734.586311] i2c_algo_bit pps_core [42734.588581] ? wake_bit_function+0x70/0x70 [42734.593106] dca fuse [42734.595466] ? ceph_write_iter+0x260/0x770 [ceph] [42734.599813] [42734.599814] irq event stamp: 261722406 [42734.603210] ceph_write_iter+0x260/0x770 [ceph] [42734.607303] hardirqs last enabled at (261722405): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42734.609577] ? lock_is_held_type+0xaa/0x120 [42734.614268] hardirqs last disabled at (261722406): [] sysvec_apic_timer_interrupt+0xe/0x90 [42734.615763] vfs_write+0x324/0x4f0 [42734.619504] softirqs last enabled at (261722404): [] __do_softirq+0x2be/0x371 [42734.624032] ksys_pwrite64+0x65/0xa0 [42734.634274] softirqs last disabled at (261722399): [] irq_exit_rcu+0xb0/0xd0 [42734.638450] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42734.648260] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42734.651661] do_syscall_64+0x38/0x80 [42734.660429] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42734.664001] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42734.672597] Workqueue: events delayed_work [ceph] [42734.677746] RIP: 0033:0x7f48f0eeb93b [42734.687123] [42734.690695] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42734.698166] RIP: 0010:lock_acquire+0xf0/0x2b0 [42734.703209] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42734.707911] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42734.711486] ORIG_RAX: 0000000000000012 [42734.712976] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42734.731715] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42734.736066] [42734.736066] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42734.741283] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42734.760021] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42734.763851] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42734.769068] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42734.776192] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42734.777681] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42734.784806] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42734.791930] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42734.799059] [42734.806178] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42734.852062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42734.857798] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42734.864924] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42734.872046] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42734.879171] Call Trace: [42734.881626] [42734.883637] ? watchdog_timer_fn+0x1b4/0x220 [42734.887915] ? __hrtimer_run_queues+0x1d3/0x360 [42734.892445] ? hrtimer_interrupt+0x104/0x240 [42734.896712] ? __local_bh_enable+0x50/0x60 [42734.900806] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42734.906029] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42734.910989] [42734.913086] [42734.915184] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42734.920492] ? lock_acquire+0xf0/0x2b0 [42734.924240] ? igrab+0x43/0x50 [42734.927291] ? lock_release+0xc5/0x260 [42734.931035] _raw_spin_lock+0x30/0x40 [42734.934690] ? igrab+0x1d/0x50 [42734.937742] igrab+0x1d/0x50 [42734.940632] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42734.945778] delayed_work+0x1d8/0x280 [ceph] [42734.950061] ? process_scheduled_works+0x22a/0x510 [42734.954851] process_scheduled_works+0x2a4/0x510 [42734.959475] worker_thread+0x188/0x340 [42734.963223] ? rescuer_thread+0x310/0x310 [42734.967225] kthread+0xf5/0x130 [42734.970365] ? kthread_complete_and_exit+0x20/0x20 [42734.975149] ret_from_fork+0x31/0x40 [42734.978720] ? kthread_complete_and_exit+0x20/0x20 [42734.983504] ret_from_fork_asm+0x11/0x20 [42734.987426] [42762.200712] watchdog: BUG: soft lockup - CPU#0 stuck for 28711s! [dbench:78683] [42762.208017] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42762.212711] watchdog: BUG: soft lockup - CPU#7 stuck for 37708s! [kworker/7:4:78808] [42762.263465] irq event stamp: 76887552 [42762.271195] Modules linked in: ceph libceph dns_resolver fscache netfs [42762.274854] hardirqs last enabled at (76887551): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42762.274859] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [42762.291545] hardirqs last disabled at (76887552): [] sysvec_apic_timer_interrupt+0xe/0x90 [42762.298929] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [42762.308654] softirqs last enabled at (76887474): [] __do_softirq+0x2be/0x371 [42762.308658] llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common [42762.323795] softirqs last disabled at (76887469): [] irq_exit_rcu+0xb0/0xd0 [42762.323798] sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm [42762.338833] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42762.338834] irqbypass [42762.338835] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42762.344829] i2c_i801 iTCO_wdt [42762.353775] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42762.356130] iTCO_vendor_support [42762.363603] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42762.366652] i2c_smbus crct10dif_pclmul [42762.372129] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42762.375353] joydev crc32_pclmul [42762.394106] [42762.394107] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42762.397939] ipmi_ssif [42762.403156] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42762.406388] ghash_clmulni_intel mei_me [42762.407879] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42762.415005] mei lpc_ich [42762.417363] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42762.424485] sha512_ssse3 acpi_ipmi [42762.428316] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42762.435441] mfd_core [42762.437970] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42762.445109] wmi [42762.448596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42762.455720] ipmi_si ipmi_devintf [42762.457992] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42762.466070] ipmi_msghandler ioatdma [42762.467906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42762.473643] acpi_pad acpi_power_meter [42762.476953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42762.484078] ixgbe igb [42762.487649] Call Trace: [42762.494774] mdio crc32c_intel [42762.498519] [42762.505643] nvme ptp [42762.508001] ? watchdog_timer_fn+0x1b4/0x220 [42762.510443] nvme_core i2c_algo_bit [42762.513495] ? __hrtimer_run_queues+0x1d3/0x360 [42762.515504] pps_core [42762.517778] ? hrtimer_interrupt+0x104/0x240 [42762.522038] dca [42762.525522] ? __local_bh_enable+0x50/0x60 [42762.530048] fuse [42762.532321] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42762.536581] [42762.536582] irq event stamp: 261943196 [42762.538420] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42762.542511] hardirqs last enabled at (261943195): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42762.544434] [42762.549651] hardirqs last disabled at (261943196): [] sysvec_apic_timer_interrupt+0xe/0x90 [42762.551142] [42762.554886] softirqs last enabled at (261943194): [] __do_softirq+0x2be/0x371 [42762.559845] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42762.570088] softirqs last disabled at (261943189): [] irq_exit_rcu+0xb0/0xd0 [42762.572206] ? queued_spin_lock_slowpath+0x18/0x2e0 [42762.582013] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42762.584113] do_raw_spin_lock+0xb5/0xc0 [42762.592881] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42762.598188] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42762.606784] Workqueue: events delayed_work [ceph] [42762.611657] try_get_cap_refs+0x159/0x880 [ceph] [42762.621033] [42762.624867] __ceph_get_caps+0x4a1/0x760 [ceph] [42762.632334] RIP: 0010:lock_acquire+0xf0/0x2b0 [42762.637122] ? __file_remove_privs+0x89/0x110 [42762.641814] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42762.646425] ? wake_bit_function+0x70/0x70 [42762.647916] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42762.652442] ? ceph_write_iter+0x260/0x770 [ceph] [42762.656792] [42762.656793] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42762.661142] ceph_write_iter+0x260/0x770 [ceph] [42762.679881] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42762.683976] ? lock_is_held_type+0xaa/0x120 [42762.689188] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42762.693886] vfs_write+0x324/0x4f0 [42762.695375] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42762.702504] ksys_pwrite64+0x65/0xa0 [42762.707023] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42762.714148] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42762.718325] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42762.725452] do_syscall_64+0x38/0x80 [42762.728848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42762.735972] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42762.739543] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42762.746668] RIP: 0033:0x7f48f0eeb93b [42762.751796] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42762.759876] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42762.763445] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42762.769182] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42762.774226] Call Trace: [42762.774227] [42762.781350] ORIG_RAX: 0000000000000012 [42762.784921] ? watchdog_timer_fn+0x1b4/0x220 [42762.792047] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42762.810784] ? __hrtimer_run_queues+0x1d3/0x360 [42762.817908] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42762.823129] ? hrtimer_interrupt+0x104/0x240 [42762.825569] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42762.827580] ? __local_bh_enable+0x50/0x60 [42762.831410] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42762.835677] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42762.842797] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42762.847323] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42762.854451] [42762.858712] [42762.898640] [42762.900737] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42762.906044] ? lock_acquire+0xf0/0x2b0 [42762.909796] ? igrab+0x43/0x50 [42762.912844] ? lock_release+0xc5/0x260 [42762.916590] _raw_spin_lock+0x30/0x40 [42762.920256] ? igrab+0x1d/0x50 [42762.923305] igrab+0x1d/0x50 [42762.926185] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42762.931334] delayed_work+0x1d8/0x280 [ceph] [42762.935616] ? process_scheduled_works+0x22a/0x510 [42762.940404] process_scheduled_works+0x2a4/0x510 [42762.945021] worker_thread+0x188/0x340 [42762.948769] ? rescuer_thread+0x310/0x310 [42762.952773] kthread+0xf5/0x130 [42762.955911] ? kthread_complete_and_exit+0x20/0x20 [42762.960694] ret_from_fork+0x31/0x40 [42762.964273] ? kthread_complete_and_exit+0x20/0x20 [42762.969059] ret_from_fork_asm+0x11/0x20 [42762.972989] [42790.201201] watchdog: BUG: soft lockup - CPU#0 stuck for 28737s! [dbench:78683] [42790.208502] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42790.213201] watchdog: BUG: soft lockup - CPU#7 stuck for 37734s! [kworker/7:4:78808] [42790.263944] irq event stamp: 76945664 [42790.263945] hardirqs last enabled at (76945663): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42790.271674] Modules linked in: [42790.275331] hardirqs last disabled at (76945664): [] sysvec_apic_timer_interrupt+0xe/0x90 [42790.285488] ceph [42790.288539] softirqs last enabled at (76945586): [] __do_softirq+0x2be/0x371 [42790.298263] libceph dns_resolver [42790.300187] softirqs last disabled at (76945581): [] irq_exit_rcu+0xb0/0xd0 [42790.308882] fscache [42790.312190] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42790.320702] netfs [42790.322898] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42790.331840] veth nft_chain_nat [42790.333850] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42790.341320] xt_MASQUERADE nf_nat [42790.344457] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42790.349936] nf_conntrack nf_defrag_ipv6 [42790.353245] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42790.371983] nf_defrag_ipv4 nft_compat [42790.375901] [42790.375901] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42790.381120] nf_tables nfnetlink [42790.384878] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42790.386371] bridge stp [42790.393496] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42790.396720] llc xfs [42790.403842] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42790.406286] libcrc32c sunrpc [42790.413411] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42790.415599] intel_rapl_msr intel_rapl_common [42790.422718] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42790.425684] sb_edac x86_pkg_temp_thermal [42790.432808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42790.437159] coretemp [42790.445237] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42790.449241] kvm_intel kvm [42790.454978] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42790.457247] irqbypass i2c_i801 [42790.464372] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42790.467075] iTCO_wdt iTCO_vendor_support [42790.474201] Call Trace: [42790.474202] [42790.477337] i2c_smbus crct10dif_pclmul [42790.484464] ? watchdog_timer_fn+0x1b4/0x220 [42790.488465] joydev crc32_pclmul [42790.490912] ? __hrtimer_run_queues+0x1d3/0x360 [42790.492919] ipmi_ssif ghash_clmulni_intel [42790.496755] ? hrtimer_interrupt+0x104/0x240 [42790.501014] mei_me mei [42790.504239] ? __local_bh_enable+0x50/0x60 [42790.508763] lpc_ich sha512_ssse3 [42790.512858] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42790.517119] acpi_ipmi mfd_core [42790.519563] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42790.523654] wmi ipmi_si [42790.526963] [42790.532181] ipmi_devintf [42790.535318] [42790.540276] ipmi_msghandler ioatdma [42790.542807] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42790.544903] acpi_pad acpi_power_meter [42790.547527] ? queued_spin_lock_slowpath+0x18/0x2e0 [42790.549619] ixgbe [42790.553191] do_raw_spin_lock+0xb5/0xc0 [42790.558493] igb mdio [42790.562240] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42790.567110] crc32c_intel nvme [42790.569123] try_get_cap_refs+0x159/0x880 [ceph] [42790.572949] ptp nvme_core [42790.575225] __ceph_get_caps+0x4a1/0x760 [ceph] [42790.580006] i2c_algo_bit pps_core [42790.583060] ? __file_remove_privs+0x89/0x110 [42790.587667] dca fuse [42790.590372] ? wake_bit_function+0x70/0x70 [42790.594895] [42790.594896] irq event stamp: 262164120 [42790.598295] ? ceph_write_iter+0x260/0x770 [ceph] [42790.602642] hardirqs last enabled at (262164119): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42790.604915] ceph_write_iter+0x260/0x770 [ceph] [42790.609003] hardirqs last disabled at (262164120): [] sysvec_apic_timer_interrupt+0xe/0x90 [42790.610497] ? lock_is_held_type+0xaa/0x120 [42790.614239] softirqs last enabled at (262164118): [] __do_softirq+0x2be/0x371 [42790.618937] vfs_write+0x324/0x4f0 [42790.629181] softirqs last disabled at (262164113): [] irq_exit_rcu+0xb0/0xd0 [42790.633710] ksys_pwrite64+0x65/0xa0 [42790.643516] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42790.647693] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42790.656462] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42790.659879] do_syscall_64+0x38/0x80 [42790.668476] Workqueue: events delayed_work [ceph] [42790.672047] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42790.681423] [42790.681424] RIP: 0010:lock_acquire+0xf0/0x2b0 [42790.686554] RIP: 0033:0x7f48f0eeb93b [42790.694025] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42790.697603] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42790.702302] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42790.707347] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42790.708838] [42790.708839] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42790.713188] ORIG_RAX: 0000000000000012 [42790.716758] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42790.735496] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42790.754234] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42790.759451] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42790.764670] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42790.766159] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42790.773284] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42790.777114] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42790.784238] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42790.791363] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42790.798487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42790.805616] [42790.812736] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42790.864373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42790.871497] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42790.878619] Call Trace: [42790.881066] [42790.883077] ? watchdog_timer_fn+0x1b4/0x220 [42790.887341] ? __hrtimer_run_queues+0x1d3/0x360 [42790.891883] ? hrtimer_interrupt+0x104/0x240 [42790.896146] ? __local_bh_enable+0x50/0x60 [42790.900240] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42790.905462] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42790.910418] [42790.912515] [42790.914614] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42790.919922] ? lock_acquire+0xf0/0x2b0 [42790.923672] ? igrab+0x43/0x50 [42790.926722] ? lock_release+0xc5/0x260 [42790.930468] _raw_spin_lock+0x30/0x40 [42790.934132] ? igrab+0x1d/0x50 [42790.937182] igrab+0x1d/0x50 [42790.940060] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42790.945212] delayed_work+0x1d8/0x280 [ceph] [42790.949493] ? process_scheduled_works+0x22a/0x510 [42790.954283] process_scheduled_works+0x2a4/0x510 [42790.958901] worker_thread+0x188/0x340 [42790.962648] ? rescuer_thread+0x310/0x310 [42790.966660] kthread+0xf5/0x130 [42790.969796] ? kthread_complete_and_exit+0x20/0x20 [42790.974583] ret_from_fork+0x31/0x40 [42790.978176] ? kthread_complete_and_exit+0x20/0x20 [42790.982963] ret_from_fork_asm+0x11/0x20 [42790.986893] [42818.201691] watchdog: BUG: soft lockup - CPU#0 stuck for 28763s! [dbench:78683] [42818.208998] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42818.213690] watchdog: BUG: soft lockup - CPU#7 stuck for 37760s! [kworker/7:4:78808] [42818.264448] irq event stamp: 77003874 [42818.272180] Modules linked in: ceph libceph dns_resolver fscache [42818.275836] hardirqs last enabled at (77003873): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42818.281842] netfs veth nft_chain_nat xt_MASQUERADE [42818.291999] hardirqs last disabled at (77003874): [] sysvec_apic_timer_interrupt+0xe/0x90 [42818.296869] nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink [42818.306595] softirqs last enabled at (77003798): [] __do_softirq+0x2be/0x371 [42818.315114] bridge stp llc xfs libcrc32c sunrpc [42818.323799] softirqs last disabled at (77003793): [] irq_exit_rcu+0xb0/0xd0 [42818.328408] intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [42818.336922] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42818.336923] coretemp kvm_intel kvm [42818.352728] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42818.352730] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42818.356211] irqbypass i2c_i801 [42818.363684] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42818.369159] iTCO_wdt iTCO_vendor_support [42818.372297] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42818.391035] i2c_smbus [42818.395040] [42818.400257] crct10dif_pclmul joydev [42818.402613] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42818.404106] crc32_pclmul [42818.407675] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42818.414800] ipmi_ssif ghash_clmulni_intel [42818.417418] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42818.424541] mei_me mei [42818.428633] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42818.435757] lpc_ich sha512_ssse3 [42818.438201] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42818.445324] acpi_ipmi [42818.448636] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42818.455759] mfd_core wmi [42818.458116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42818.466194] ipmi_si ipmi_devintf [42818.468812] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42818.474550] ipmi_msghandler ioatdma [42818.477861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42818.484985] acpi_pad acpi_power_meter [42818.488555] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42818.495679] ixgbe igb [42818.499423] Call Trace: [42818.499426] [42818.506548] mdio crc32c_intel [42818.508906] ? watchdog_timer_fn+0x1b4/0x220 [42818.511348] nvme ptp [42818.513392] ? __hrtimer_run_queues+0x1d3/0x360 [42818.516444] nvme_core i2c_algo_bit [42818.520713] ? hrtimer_interrupt+0x104/0x240 [42818.522979] pps_core dca [42818.527504] ? __local_bh_enable+0x50/0x60 [42818.530986] fuse [42818.535255] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42818.537870] irq event stamp: 262384910 [42818.541961] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42818.543884] hardirqs last enabled at (262384909): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42818.549102] [42818.552845] hardirqs last disabled at (262384910): [] sysvec_apic_timer_interrupt+0xe/0x90 [42818.557802] [42818.557803] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42818.568047] softirqs last enabled at (262384908): [] __do_softirq+0x2be/0x371 [42818.570149] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42818.579954] softirqs last disabled at (262384903): [] irq_exit_rcu+0xb0/0xd0 [42818.582054] do_raw_spin_lock+0xb5/0xc0 [42818.587370] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42818.596146] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42818.601016] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42818.609616] try_get_cap_refs+0x159/0x880 [ceph] [42818.613443] Workqueue: events delayed_work [ceph] [42818.622825] __ceph_get_caps+0x4a1/0x760 [ceph] [42818.627604] [42818.627605] RIP: 0010:lock_acquire+0xf0/0x2b0 [42818.635096] ? __file_remove_privs+0x89/0x110 [42818.639703] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42818.644401] ? wake_bit_function+0x70/0x70 [42818.648926] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42818.650419] ? ceph_write_iter+0x260/0x770 [ceph] [42818.654768] [42818.654769] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42818.659118] ceph_write_iter+0x260/0x770 [ceph] [42818.677856] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42818.681950] ? lock_is_held_type+0xaa/0x120 [42818.687164] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42818.691864] vfs_write+0x324/0x4f0 [42818.693352] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42818.700482] ksys_pwrite64+0x65/0xa0 [42818.704999] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42818.712126] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42818.716303] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42818.723428] do_syscall_64+0x38/0x80 [42818.726823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42818.733949] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42818.737518] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42818.744643] RIP: 0033:0x7f48f0eeb93b [42818.749773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42818.757851] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42818.761420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42818.767159] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42818.772202] Call Trace: [42818.772204] [42818.779326] ORIG_RAX: 0000000000000012 [42818.782898] ? watchdog_timer_fn+0x1b4/0x220 [42818.790021] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42818.808760] ? __hrtimer_run_queues+0x1d3/0x360 [42818.815883] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42818.821105] ? hrtimer_interrupt+0x104/0x240 [42818.823545] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42818.825554] ? __local_bh_enable+0x50/0x60 [42818.829385] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42818.833653] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42818.840775] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42818.845300] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42818.852428] [42818.856686] [42818.896615] [42818.898714] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42818.904021] ? lock_acquire+0xf0/0x2b0 [42818.907770] ? igrab+0x43/0x50 [42818.910821] ? lock_release+0xc5/0x260 [42818.914567] _raw_spin_lock+0x30/0x40 [42818.918231] ? igrab+0x1d/0x50 [42818.921280] igrab+0x1d/0x50 [42818.924159] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42818.929309] delayed_work+0x1d8/0x280 [ceph] [42818.933590] ? process_scheduled_works+0x22a/0x510 [42818.938396] process_scheduled_works+0x2a4/0x510 [42818.943014] worker_thread+0x188/0x340 [42818.946761] ? rescuer_thread+0x310/0x310 [42818.950765] kthread+0xf5/0x130 [42818.953904] ? kthread_complete_and_exit+0x20/0x20 [42818.958688] ret_from_fork+0x31/0x40 [42818.962268] ? kthread_complete_and_exit+0x20/0x20 [42818.967052] ret_from_fork_asm+0x11/0x20 [42818.970983] [42846.202179] watchdog: BUG: soft lockup - CPU#0 stuck for 28789s! [dbench:78683] [42846.209476] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42846.214179] watchdog: BUG: soft lockup - CPU#7 stuck for 37786s! [kworker/7:4:78808] [42846.264918] irq event stamp: 77061962 [42846.264919] hardirqs last enabled at (77061961): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42846.272649] Modules linked in: [42846.276305] hardirqs last disabled at (77061962): [] sysvec_apic_timer_interrupt+0xe/0x90 [42846.286464] ceph [42846.289513] softirqs last enabled at (77061884): [] __do_softirq+0x2be/0x371 [42846.299237] libceph [42846.301163] softirqs last disabled at (77061879): [] irq_exit_rcu+0xb0/0xd0 [42846.309861] dns_resolver fscache [42846.312049] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42846.320572] netfs veth [42846.323886] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42846.332832] nft_chain_nat xt_MASQUERADE [42846.335293] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42846.342763] nf_nat nf_conntrack [42846.346680] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42846.352159] nf_defrag_ipv6 nf_defrag_ipv4 [42846.355382] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42846.374120] nft_compat nf_tables [42846.378210] [42846.383428] nfnetlink bridge [42846.386739] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42846.388228] stp llc [42846.391193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42846.398319] xfs libcrc32c [42846.400501] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42846.407626] sunrpc [42846.410329] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42846.417453] intel_rapl_msr intel_rapl_common [42846.419551] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42846.426692] sb_edac x86_pkg_temp_thermal [42846.431044] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42846.438167] coretemp [42846.442172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42846.450250] kvm_intel [42846.452521] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42846.458256] kvm irqbypass [42846.460614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42846.467737] i2c_i801 iTCO_wdt [42846.470442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42846.477574] iTCO_vendor_support [42846.480627] Call Trace: [42846.480629] [42846.487749] i2c_smbus crct10dif_pclmul [42846.490974] ? watchdog_timer_fn+0x1b4/0x220 [42846.493418] joydev crc32_pclmul [42846.495431] ? __hrtimer_run_queues+0x1d3/0x360 [42846.499260] ipmi_ssif ghash_clmulni_intel [42846.503527] ? hrtimer_interrupt+0x104/0x240 [42846.506748] mei_me mei [42846.511272] ? __local_bh_enable+0x50/0x60 [42846.515364] lpc_ich sha512_ssse3 [42846.519630] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42846.522072] acpi_ipmi mfd_core [42846.526163] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42846.529473] wmi ipmi_si [42846.534689] [42846.537827] ipmi_devintf ipmi_msghandler [42846.542785] [42846.545315] ioatdma acpi_pad [42846.547414] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42846.551417] acpi_power_meter ixgbe [42846.553519] ? queued_spin_lock_slowpath+0x18/0x2e0 [42846.556479] igb [42846.561784] do_raw_spin_lock+0xb5/0xc0 [42846.565267] mdio crc32c_intel [42846.570137] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42846.571974] nvme ptp [42846.575808] try_get_cap_refs+0x159/0x880 [ceph] [42846.578856] nvme_core i2c_algo_bit [42846.583646] __ceph_get_caps+0x4a1/0x760 [ceph] [42846.585912] pps_core dca [42846.590525] ? __file_remove_privs+0x89/0x110 [42846.594006] fuse [42846.598530] ? wake_bit_function+0x70/0x70 [42846.601146] irq event stamp: 262605740 [42846.605502] ? ceph_write_iter+0x260/0x770 [ceph] [42846.607422] hardirqs last enabled at (262605739): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42846.611513] ceph_write_iter+0x260/0x770 [ceph] [42846.615258] hardirqs last disabled at (262605740): [] sysvec_apic_timer_interrupt+0xe/0x90 [42846.619959] ? lock_is_held_type+0xaa/0x120 [42846.630199] softirqs last enabled at (262605738): [] __do_softirq+0x2be/0x371 [42846.634725] vfs_write+0x324/0x4f0 [42846.644533] softirqs last disabled at (262605733): [] irq_exit_rcu+0xb0/0xd0 [42846.648715] ksys_pwrite64+0x65/0xa0 [42846.657481] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42846.660878] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42846.669478] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42846.673049] do_syscall_64+0x38/0x80 [42846.682426] Workqueue: events delayed_work [ceph] [42846.687556] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42846.695044] [42846.695045] RIP: 0010:lock_acquire+0xf0/0x2b0 [42846.698614] RIP: 0033:0x7f48f0eeb93b [42846.703313] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42846.708355] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42846.709861] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42846.709863] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42846.714215] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42846.717787] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42846.717788] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42846.736523] ORIG_RAX: 0000000000000012 [42846.755260] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42846.760478] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42846.767602] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42846.772821] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42846.779944] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42846.787067] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42846.790899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42846.798024] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42846.805146] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42846.812272] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42846.819394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42846.827477] [42846.834597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42846.878140] Call Trace: [42846.880584] [42846.882594] ? watchdog_timer_fn+0x1b4/0x220 [42846.886874] ? __hrtimer_run_queues+0x1d3/0x360 [42846.891403] ? hrtimer_interrupt+0x104/0x240 [42846.895672] ? __local_bh_enable+0x50/0x60 [42846.899767] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42846.904991] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42846.909947] [42846.912043] [42846.914142] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42846.919450] ? lock_acquire+0xf0/0x2b0 [42846.923201] ? igrab+0x43/0x50 [42846.926250] ? lock_release+0xc5/0x260 [42846.929995] _raw_spin_lock+0x30/0x40 [42846.933659] ? igrab+0x1d/0x50 [42846.936710] igrab+0x1d/0x50 [42846.939587] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42846.944738] delayed_work+0x1d8/0x280 [ceph] [42846.949018] ? process_scheduled_works+0x22a/0x510 [42846.953809] process_scheduled_works+0x2a4/0x510 [42846.958426] worker_thread+0x188/0x340 [42846.962173] ? rescuer_thread+0x310/0x310 [42846.966177] kthread+0xf5/0x130 [42846.969314] ? kthread_complete_and_exit+0x20/0x20 [42846.974099] ret_from_fork+0x31/0x40 [42846.977678] ? kthread_complete_and_exit+0x20/0x20 [42846.982462] ret_from_fork_asm+0x11/0x20 [42846.986385] [42874.202668] watchdog: BUG: soft lockup - CPU#0 stuck for 28815s! [dbench:78683] [42874.209972] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42874.214668] watchdog: BUG: soft lockup - CPU#7 stuck for 37812s! [kworker/7:4:78808] [42874.265413] irq event stamp: 77120144 [42874.265414] hardirqs last enabled at (77120143): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42874.273145] Modules linked in: ceph [42874.276801] hardirqs last disabled at (77120144): [] sysvec_apic_timer_interrupt+0xe/0x90 [42874.286960] libceph [42874.290443] softirqs last enabled at (77120108): [] __do_softirq+0x2be/0x371 [42874.300166] dns_resolver [42874.302351] softirqs last disabled at (77120103): [] irq_exit_rcu+0xb0/0xd0 [42874.311036] fscache netfs [42874.313671] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42874.322182] veth nft_chain_nat [42874.324886] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42874.333829] xt_MASQUERADE nf_nat [42874.336966] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [42874.344439] nf_conntrack [42874.347748] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [42874.353226] nf_defrag_ipv6 nf_defrag_ipv4 [42874.355842] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42874.374582] nft_compat nf_tables [42874.378671] [42874.378672] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42874.383888] nfnetlink bridge [42874.387199] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42874.388689] stp [42874.395815] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42874.398780] llc xfs [42874.405903] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42874.407741] libcrc32c sunrpc [42874.414864] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42874.417064] intel_rapl_msr intel_rapl_common [42874.424190] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42874.427154] sb_edac x86_pkg_temp_thermal [42874.434279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42874.438628] coretemp kvm_intel [42874.446705] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42874.450711] kvm irqbypass [42874.456449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42874.459586] i2c_i801 iTCO_wdt [42874.466709] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42874.469412] iTCO_vendor_support i2c_smbus [42874.476537] Call Trace: [42874.476539] [42874.479587] crct10dif_pclmul joydev [42874.486714] ? watchdog_timer_fn+0x1b4/0x220 [42874.490804] crc32_pclmul ipmi_ssif [42874.493248] ? __hrtimer_run_queues+0x1d3/0x360 [42874.495258] ghash_clmulni_intel mei_me [42874.498831] ? hrtimer_interrupt+0x104/0x240 [42874.503093] mei lpc_ich [42874.506575] ? __local_bh_enable+0x50/0x60 [42874.511100] sha512_ssse3 [42874.514935] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42874.519195] acpi_ipmi mfd_core [42874.521728] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42874.525817] wmi ipmi_si [42874.528436] [42874.533653] ipmi_devintf ipmi_msghandler [42874.536788] [42874.536790] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42874.541745] ioatdma acpi_pad [42874.544280] ? queued_spin_lock_slowpath+0x2b/0x2e0 [42874.546375] acpi_power_meter ixgbe [42874.550380] do_raw_spin_lock+0xb5/0xc0 [42874.552477] igb mdio [42874.557782] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42874.560745] crc32c_intel [42874.565617] try_get_cap_refs+0x159/0x880 [ceph] [42874.569100] nvme ptp [42874.572934] __ceph_get_caps+0x4a1/0x760 [ceph] [42874.575201] nvme_core i2c_algo_bit [42874.579990] ? __file_remove_privs+0x89/0x110 [42874.582602] pps_core [42874.587212] ? wake_bit_function+0x70/0x70 [42874.589482] dca fuse [42874.594011] ? ceph_write_iter+0x260/0x770 [ceph] [42874.597493] [42874.597494] irq event stamp: 262826548 [42874.601842] ceph_write_iter+0x260/0x770 [ceph] [42874.604114] hardirqs last enabled at (262826547): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42874.608209] ? lock_is_held_type+0xaa/0x120 [42874.610475] hardirqs last disabled at (262826548): [] sysvec_apic_timer_interrupt+0xe/0x90 [42874.615174] vfs_write+0x324/0x4f0 [42874.616663] softirqs last enabled at (262826546): [] __do_softirq+0x2be/0x371 [42874.620413] ksys_pwrite64+0x65/0xa0 [42874.624930] softirqs last disabled at (262826541): [] irq_exit_rcu+0xb0/0xd0 [42874.635176] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42874.639367] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42874.649181] do_syscall_64+0x38/0x80 [42874.652578] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42874.661351] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42874.664921] Workqueue: events delayed_work [ceph] [42874.673517] RIP: 0033:0x7f48f0eeb93b [42874.678649] [42874.678650] RIP: 0010:lock_acquire+0xf0/0x2b0 [42874.688025] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42874.691597] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42874.699069] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42874.704110] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42874.708809] ORIG_RAX: 0000000000000012 [42874.712380] [42874.712381] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42874.713871] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42874.718221] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42874.736958] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42874.755695] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42874.760913] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42874.766130] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42874.769963] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42874.771452] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42874.778576] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42874.785700] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42874.792829] [42874.799948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42874.858692] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42874.865816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42874.872941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42874.880074] Call Trace: [42874.882517] [42874.884530] ? watchdog_timer_fn+0x1b4/0x220 [42874.888794] ? __hrtimer_run_queues+0x1d3/0x360 [42874.893322] ? hrtimer_interrupt+0x104/0x240 [42874.897590] ? __local_bh_enable+0x50/0x60 [42874.901683] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42874.906908] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42874.911863] [42874.913962] [42874.916075] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42874.921386] ? lock_acquire+0xf0/0x2b0 [42874.925134] ? igrab+0x43/0x50 [42874.928184] ? lock_release+0xc5/0x260 [42874.931930] _raw_spin_lock+0x30/0x40 [42874.935593] ? igrab+0x1d/0x50 [42874.938645] igrab+0x1d/0x50 [42874.941524] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42874.946674] delayed_work+0x1d8/0x280 [ceph] [42874.950953] ? process_scheduled_works+0x22a/0x510 [42874.955736] process_scheduled_works+0x2a4/0x510 [42874.960359] worker_thread+0x188/0x340 [42874.964109] ? rescuer_thread+0x310/0x310 [42874.968111] kthread+0xf5/0x130 [42874.971250] ? kthread_complete_and_exit+0x20/0x20 [42874.976034] ret_from_fork+0x31/0x40 [42874.979614] ? kthread_complete_and_exit+0x20/0x20 [42874.984398] ret_from_fork_asm+0x11/0x20 [42874.988320] [42877.058085] rcu: INFO: rcu_preempt self-detected stall on CPU [42877.063848] rcu: 0-....: (30239801 ticks this GP) idle=762c/1/0x4000000000000000 softirq=496833/496833 fqs=15149602 [42877.074358] rcu: (t=30939909 jiffies g=1309609 q=824979 ncpus=8) [42877.080445] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42877.089396] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42877.096865] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [42877.102353] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [42877.121090] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42877.126306] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42877.133432] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42877.140555] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42877.147679] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42877.154803] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42877.161928] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42877.170005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42877.175744] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42877.182867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42877.189991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42877.197114] Call Trace: [42877.199561] [42877.201573] ? rcu_dump_cpu_stacks+0xd1/0x180 [42877.205931] ? rcu_sched_clock_irq+0x633/0x1390 [42877.210463] ? lock_is_held_type+0xaa/0x120 [42877.214641] ? lock_is_held_type+0xaa/0x120 [42877.218818] ? tick_sched_do_timer+0xa0/0xa0 [42877.223090] ? rcu_read_lock_sched_held+0x56/0x90 [42877.227789] ? tick_sched_do_timer+0xa0/0xa0 [42877.232052] ? update_process_times+0x5f/0x90 [42877.236402] ? tick_sched_handle.isra.16+0x1e/0x50 [42877.241186] ? tick_sched_timer+0x6f/0x80 [42877.245193] ? __hrtimer_run_queues+0x1d3/0x360 [42877.249717] ? hrtimer_interrupt+0x104/0x240 [42877.253979] ? __local_bh_enable+0x50/0x60 [42877.258082] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42877.263304] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42877.268261] [42877.270357] [42877.272457] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42877.277765] ? queued_spin_lock_slowpath+0x2d/0x2e0 [42877.282642] do_raw_spin_lock+0xb5/0xc0 [42877.286481] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42877.291292] try_get_cap_refs+0x159/0x880 [ceph] [42877.295922] __ceph_get_caps+0x4a1/0x760 [ceph] [42877.300463] ? __file_remove_privs+0x89/0x110 [42877.304818] ? wake_bit_function+0x70/0x70 [42877.308912] ? ceph_write_iter+0x260/0x770 [ceph] [42877.313625] ceph_write_iter+0x260/0x770 [ceph] [42877.318159] ? lock_is_held_type+0xaa/0x120 [42877.322346] vfs_write+0x324/0x4f0 [42877.325753] ksys_pwrite64+0x65/0xa0 [42877.329329] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42877.334459] do_syscall_64+0x38/0x80 [42877.338030] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42877.343074] RIP: 0033:0x7f48f0eeb93b [42877.346644] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42877.365381] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [42877.372938] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42877.380064] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42877.387187] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42877.394311] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42877.401436] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42877.408565] [42877.410754] Sending NMI from CPU 0 to CPUs 7: [42877.415108] NMI backtrace for cpu 7 [42877.415109] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42877.415111] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42877.415112] Workqueue: events delayed_work [ceph] [42877.415130] RIP: 0010:__lock_acquire+0xc2e/0x1e20 [42877.415132] Code: 49 0f af c7 48 c1 e8 31 49 89 c0 4c 8d 24 c5 c0 f5 5d 83 48 8b 04 c5 c0 f5 5d 83 48 85 c0 74 1b 48 83 f8 08 74 15 48 8b 50 10 <49> 39 d7 0f 84 6e fa ff ff 48 8b 00 48 85 c0 75 e5 4c 89 44 24 08 [42877.415133] RSP: 0018:ffffc90002117c60 EFLAGS: 00000082 [42877.415134] RAX: ffffffff82e139e8 RBX: ffff88810debaa80 RCX: 000000005553a634 [42877.415135] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42877.415136] RBP: ffff88810debb550 R08: 0000000000005c53 R09: 0000000000000003 [42877.415137] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8360d858 [42877.415137] R13: 0000000000000003 R14: 0000000000000000 R15: c5dbdfc36fd10127 [42877.415138] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42877.415139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42877.415140] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42877.415141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42877.415141] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42877.415142] Call Trace: [42877.415143] [42877.415144] ? nmi_cpu_backtrace+0xad/0x120 [42877.415147] ? nmi_cpu_backtrace_handler+0x11/0x20 [42877.415148] ? nmi_handle+0xd4/0x260 [42877.415152] ? default_do_nmi+0x49/0xf0 [42877.415154] ? exc_nmi+0x11e/0x150 [42877.415155] ? end_repeat_nmi+0x16/0x67 [42877.415159] ? __lock_acquire+0xc2e/0x1e20 [42877.415160] ? __lock_acquire+0xc2e/0x1e20 [42877.415162] ? __lock_acquire+0xc2e/0x1e20 [42877.415163] [42877.415164] [42877.415165] lock_acquire+0xcb/0x2b0 [42877.415167] ? igrab+0x1d/0x50 [42877.415168] ? igrab+0x43/0x50 [42877.415170] ? lock_release+0xc5/0x260 [42877.415171] _raw_spin_lock+0x30/0x40 [42877.415174] ? igrab+0x1d/0x50 [42877.415175] igrab+0x1d/0x50 [42877.415176] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42877.415190] delayed_work+0x1d8/0x280 [ceph] [42877.415203] ? process_scheduled_works+0x22a/0x510 [42877.415207] process_scheduled_works+0x2a4/0x510 [42877.415210] worker_thread+0x188/0x340 [42877.415213] ? rescuer_thread+0x310/0x310 [42877.415215] kthread+0xf5/0x130 [42877.415218] ? kthread_complete_and_exit+0x20/0x20 [42877.415220] ret_from_fork+0x31/0x40 [42877.415223] ? kthread_complete_and_exit+0x20/0x20 [42877.415225] ret_from_fork_asm+0x11/0x20 [42877.415229] [42902.203156] watchdog: BUG: soft lockup - CPU#0 stuck for 28841s! [dbench:78683] [42902.210461] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42902.215156] watchdog: BUG: soft lockup - CPU#7 stuck for 37838s! [kworker/7:4:78808] [42902.265911] irq event stamp: 77177642 [42902.273640] Modules linked in: ceph libceph [42902.277298] hardirqs last enabled at (77177641): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42902.281475] dns_resolver fscache [42902.291633] hardirqs last disabled at (77177642): [] sysvec_apic_timer_interrupt+0xe/0x90 [42902.294945] netfs veth [42902.304668] softirqs last enabled at (77177598): [] __do_softirq+0x2be/0x371 [42902.304669] nft_chain_nat [42902.304671] softirqs last disabled at (77177593): [] irq_exit_rcu+0xb0/0xd0 [42902.307112] xt_MASQUERADE nf_nat [42902.315795] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42902.318499] nf_conntrack nf_defrag_ipv6 [42902.327012] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42902.330322] nf_defrag_ipv4 nft_compat [42902.339266] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42902.343185] nf_tables [42902.350654] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42902.354397] nfnetlink bridge [42902.359874] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42902.362233] stp [42902.380970] [42902.383934] llc xfs [42902.389153] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42902.390990] libcrc32c sunrpc [42902.392481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42902.394665] intel_rapl_msr intel_rapl_common [42902.401788] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42902.404753] sb_edac [42902.411877] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42902.416227] x86_pkg_temp_thermal coretemp [42902.423352] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42902.425552] kvm_intel kvm [42902.432677] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42902.436768] irqbypass i2c_i801 [42902.443893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42902.446597] iTCO_wdt iTCO_vendor_support [42902.454672] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42902.457811] i2c_smbus [42902.463553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42902.467561] crct10dif_pclmul [42902.474685] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42902.477042] joydev crc32_pclmul [42902.484167] Call Trace: [42902.484168] [42902.487130] ipmi_ssif ghash_clmulni_intel [42902.494256] ? watchdog_timer_fn+0x1b4/0x220 [42902.497479] mei_me [42902.499923] ? __hrtimer_run_queues+0x1d3/0x360 [42902.501933] mei lpc_ich [42902.506027] ? hrtimer_interrupt+0x104/0x240 [42902.510288] sha512_ssse3 acpi_ipmi [42902.512386] ? __local_bh_enable+0x50/0x60 [42902.516910] mfd_core wmi [42902.519442] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42902.523704] ipmi_si [42902.527189] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42902.531280] ipmi_devintf ipmi_msghandler [42902.533898] [42902.539114] ioatdma acpi_pad acpi_power_meter [42902.541299] [42902.546256] ixgbe igb [42902.550260] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42902.552357] mdio crc32c_intel [42902.556797] ? queued_spin_lock_slowpath+0x18/0x2e0 [42902.558892] nvme [42902.561250] do_raw_spin_lock+0xb5/0xc0 [42902.566557] ptp nvme_core [42902.569613] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42902.574482] i2c_algo_bit pps_core [42902.576410] try_get_cap_refs+0x159/0x880 [ceph] [42902.580237] dca fuse [42902.582947] __ceph_get_caps+0x4a1/0x760 [ceph] [42902.587727] [42902.587728] irq event stamp: 263045258 [42902.591128] ? __file_remove_privs+0x89/0x110 [42902.595735] hardirqs last enabled at (263045257): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42902.598005] ? wake_bit_function+0x70/0x70 [42902.602530] hardirqs last disabled at (263045258): [] sysvec_apic_timer_interrupt+0xe/0x90 [42902.604041] ? ceph_write_iter+0x260/0x770 [ceph] [42902.607781] softirqs last enabled at (263045256): [] __do_softirq+0x2be/0x371 [42902.612133] ceph_write_iter+0x260/0x770 [ceph] [42902.622375] softirqs last disabled at (263045251): [] irq_exit_rcu+0xb0/0xd0 [42902.626471] ? lock_is_held_type+0xaa/0x120 [42902.636278] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42902.640977] vfs_write+0x324/0x4f0 [42902.649746] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42902.654274] ksys_pwrite64+0x65/0xa0 [42902.662869] Workqueue: events delayed_work [ceph] [42902.667045] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42902.676422] [42902.676423] RIP: 0010:lock_acquire+0xf0/0x2b0 [42902.679822] do_syscall_64+0x38/0x80 [42902.687290] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42902.690862] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42902.695559] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42902.700690] RIP: 0033:0x7f48f0eeb93b [42902.702181] [42902.702182] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42902.706532] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42902.710101] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42902.728848] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42902.733891] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42902.739109] ORIG_RAX: 0000000000000012 [42902.742681] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42902.744171] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42902.751296] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42902.770033] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42902.777158] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42902.782376] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42902.789498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42902.793330] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42902.800453] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42902.807577] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42902.814701] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42902.821846] [42902.829919] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42902.880587] Call Trace: [42902.883031] [42902.885044] ? watchdog_timer_fn+0x1b4/0x220 [42902.889307] ? __hrtimer_run_queues+0x1d3/0x360 [42902.893845] ? hrtimer_interrupt+0x104/0x240 [42902.898112] ? __local_bh_enable+0x50/0x60 [42902.902206] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42902.907430] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42902.912387] [42902.914482] [42902.916580] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42902.921889] ? lock_acquire+0xf0/0x2b0 [42902.925639] ? igrab+0x43/0x50 [42902.928690] ? lock_release+0xc5/0x260 [42902.932435] _raw_spin_lock+0x30/0x40 [42902.936098] ? igrab+0x1d/0x50 [42902.939150] igrab+0x1d/0x50 [42902.942027] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42902.947177] delayed_work+0x1d8/0x280 [ceph] [42902.951459] ? process_scheduled_works+0x22a/0x510 [42902.956248] process_scheduled_works+0x2a4/0x510 [42902.960863] worker_thread+0x188/0x340 [42902.964614] ? rescuer_thread+0x310/0x310 [42902.968625] kthread+0xf5/0x130 [42902.971763] ? kthread_complete_and_exit+0x20/0x20 [42902.976565] ret_from_fork+0x31/0x40 [42902.980144] ? kthread_complete_and_exit+0x20/0x20 [42902.984928] ret_from_fork_asm+0x11/0x20 [42902.988850] [42930.203645] watchdog: BUG: soft lockup - CPU#0 stuck for 28867s! [dbench:78683] [42930.210948] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42930.215645] watchdog: BUG: soft lockup - CPU#7 stuck for 37864s! [kworker/7:4:78808] [42930.266389] irq event stamp: 77235844 [42930.266390] hardirqs last enabled at (77235843): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42930.274118] Modules linked in: ceph [42930.277777] hardirqs last disabled at (77235844): [] sysvec_apic_timer_interrupt+0xe/0x90 [42930.287935] libceph [42930.291418] softirqs last enabled at (77235798): [] __do_softirq+0x2be/0x371 [42930.301143] dns_resolver fscache [42930.303328] softirqs last disabled at (77235793): [] irq_exit_rcu+0xb0/0xd0 [42930.312010] netfs [42930.315321] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42930.323832] veth nft_chain_nat [42930.325843] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42930.334787] xt_MASQUERADE nf_nat [42930.337925] RIP: 0010:queued_spin_lock_slowpath+0x2b/0x2e0 [42930.345396] nf_conntrack [42930.348706] Code: 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 90 eb e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba [42930.354183] nf_defrag_ipv6 nf_defrag_ipv4 [42930.356801] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42930.375539] nft_compat nf_tables [42930.379630] [42930.379631] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42930.384845] nfnetlink bridge [42930.388156] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42930.389648] stp [42930.396771] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42930.399736] llc xfs [42930.406861] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42930.408697] libcrc32c sunrpc [42930.415822] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42930.418004] intel_rapl_msr intel_rapl_common [42930.425130] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42930.428093] sb_edac x86_pkg_temp_thermal [42930.435218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42930.439570] coretemp kvm_intel [42930.447646] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42930.451649] kvm irqbypass [42930.457389] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42930.460526] i2c_i801 iTCO_wdt [42930.467650] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42930.470353] iTCO_vendor_support i2c_smbus [42930.477478] Call Trace: [42930.477479] [42930.480527] crct10dif_pclmul joydev [42930.487654] ? watchdog_timer_fn+0x1b4/0x220 [42930.491742] crc32_pclmul ipmi_ssif [42930.494189] ? __hrtimer_run_queues+0x1d3/0x360 [42930.496197] ghash_clmulni_intel mei_me [42930.499772] ? hrtimer_interrupt+0x104/0x240 [42930.504042] mei lpc_ich [42930.507525] ? __local_bh_enable+0x50/0x60 [42930.512049] sha512_ssse3 acpi_ipmi [42930.515884] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42930.520145] mfd_core wmi [42930.522676] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42930.526766] ipmi_si ipmi_devintf [42930.530251] [42930.535467] ipmi_msghandler [42930.538085] [42930.543044] ioatdma acpi_pad [42930.546353] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42930.548449] acpi_power_meter ixgbe [42930.551333] ? queued_spin_lock_slowpath+0x2b/0x2e0 [42930.553424] igb mdio [42930.556391] do_raw_spin_lock+0xb5/0xc0 [42930.561693] crc32c_intel nvme [42930.565178] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42930.570048] ptp [42930.572335] try_get_cap_refs+0x159/0x880 [ceph] [42930.576166] nvme_core i2c_algo_bit [42930.579221] __ceph_get_caps+0x4a1/0x760 [ceph] [42930.584001] pps_core dca [42930.585842] ? __file_remove_privs+0x89/0x110 [42930.590449] fuse [42930.593934] ? wake_bit_function+0x70/0x70 [42930.598457] irq event stamp: 263266032 [42930.601077] ? ceph_write_iter+0x260/0x770 [ceph] [42930.605426] hardirqs last enabled at (263266031): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42930.607349] ceph_write_iter+0x260/0x770 [ceph] [42930.611441] hardirqs last disabled at (263266032): [] sysvec_apic_timer_interrupt+0xe/0x90 [42930.615189] ? lock_is_held_type+0xaa/0x120 [42930.619882] softirqs last enabled at (263266030): [] __do_softirq+0x2be/0x371 [42930.630129] vfs_write+0x324/0x4f0 [42930.634651] softirqs last disabled at (263266025): [] irq_exit_rcu+0xb0/0xd0 [42930.644466] ksys_pwrite64+0x65/0xa0 [42930.648639] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42930.657408] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42930.660806] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42930.669407] do_syscall_64+0x38/0x80 [42930.672975] Workqueue: events delayed_work [ceph] [42930.682354] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42930.687482] [42930.687483] RIP: 0010:lock_acquire+0xf0/0x2b0 [42930.694954] RIP: 0033:0x7f48f0eeb93b [42930.698525] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42930.703223] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42930.708266] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42930.709757] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42930.714109] [42930.714110] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42930.717677] ORIG_RAX: 0000000000000012 [42930.736415] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42930.755153] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42930.760370] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42930.765590] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42930.767079] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42930.774203] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42930.778048] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42930.785174] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42930.792299] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42930.799424] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42930.806549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42930.813678] [42930.820795] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42930.865293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42930.872415] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42930.879541] Call Trace: [42930.881986] [42930.883996] ? watchdog_timer_fn+0x1b4/0x220 [42930.888260] ? __hrtimer_run_queues+0x1d3/0x360 [42930.892788] ? hrtimer_interrupt+0x104/0x240 [42930.897057] ? __local_bh_enable+0x50/0x60 [42930.901150] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42930.906373] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42930.911334] [42930.913437] [42930.915534] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42930.920842] ? lock_acquire+0xf0/0x2b0 [42930.924591] ? igrab+0x43/0x50 [42930.927643] ? lock_release+0xc5/0x260 [42930.931386] _raw_spin_lock+0x30/0x40 [42930.935051] ? igrab+0x1d/0x50 [42930.938102] igrab+0x1d/0x50 [42930.940980] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42930.946131] delayed_work+0x1d8/0x280 [ceph] [42930.950410] ? process_scheduled_works+0x22a/0x510 [42930.955192] process_scheduled_works+0x2a4/0x510 [42930.959808] worker_thread+0x188/0x340 [42930.963557] ? rescuer_thread+0x310/0x310 [42930.967560] kthread+0xf5/0x130 [42930.970697] ? kthread_complete_and_exit+0x20/0x20 [42930.975482] ret_from_fork+0x31/0x40 [42930.979054] ? kthread_complete_and_exit+0x20/0x20 [42930.983838] ret_from_fork_asm+0x11/0x20 [42930.987760] [42958.204134] watchdog: BUG: soft lockup - CPU#0 stuck for 28893s! [dbench:78683] [42958.211434] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42958.216133] watchdog: BUG: soft lockup - CPU#7 stuck for 37890s! [kworker/7:4:78808] [42958.266884] irq event stamp: 77294042 [42958.274615] Modules linked in: ceph libceph dns_resolver fscache netfs veth [42958.278273] hardirqs last enabled at (77294041): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42958.278277] nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 [42958.295389] hardirqs last disabled at (77294042): [] sysvec_apic_timer_interrupt+0xe/0x90 [42958.303640] nft_compat nf_tables nfnetlink bridge stp llc xfs [42958.313365] softirqs last enabled at (77293976): [] __do_softirq+0x2be/0x371 [42958.319197] libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal [42958.327882] softirqs last disabled at (77293971): [] irq_exit_rcu+0xb0/0xd0 [42958.336228] coretemp kvm_intel kvm irqbypass i2c_i801 [42958.344740] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42958.349870] iTCO_wdt iTCO_vendor_support [42958.358816] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42958.358818] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42958.362818] i2c_smbus crct10dif_pclmul [42958.370287] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42958.375766] joydev crc32_pclmul [42958.379596] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42958.398334] ipmi_ssif ghash_clmulni_intel [42958.401557] [42958.401558] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42958.406775] mei_me mei [42958.410866] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42958.412356] lpc_ich sha512_ssse3 [42958.419480] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42958.421926] acpi_ipmi mfd_core [42958.429049] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42958.432359] wmi ipmi_si [42958.439484] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42958.442622] ipmi_devintf ipmi_msghandler [42958.449746] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42958.452277] ioatdma acpi_pad [42958.459401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42958.463404] acpi_power_meter [42958.471481] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42958.474445] ixgbe igb [42958.480183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42958.483149] mdio crc32c_intel [42958.490273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42958.492628] nvme ptp [42958.499753] Call Trace: [42958.499756] [42958.502818] nvme_core i2c_algo_bit [42958.509947] ? watchdog_timer_fn+0x1b4/0x220 [42958.512215] pps_core [42958.514661] ? __hrtimer_run_queues+0x1d3/0x360 [42958.516670] dca fuse [42958.520157] ? hrtimer_interrupt+0x104/0x240 [42958.524419] [42958.524420] irq event stamp: 263486852 [42958.526689] ? __local_bh_enable+0x50/0x60 [42958.531214] hardirqs last enabled at (263486851): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42958.533487] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42958.537749] hardirqs last disabled at (263486852): [] sysvec_apic_timer_interrupt+0xe/0x90 [42958.539238] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42958.542984] softirqs last enabled at (263486850): [] __do_softirq+0x2be/0x371 [42958.547074] [42958.557318] softirqs last disabled at (263486845): [] irq_exit_rcu+0xb0/0xd0 [42958.562536] [42958.572346] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42958.577305] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42958.586075] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42958.588177] ? queued_spin_lock_slowpath+0x18/0x2e0 [42958.596770] Workqueue: events delayed_work [ceph] [42958.598870] do_raw_spin_lock+0xb5/0xc0 [42958.608245] [42958.608246] RIP: 0010:lock_acquire+0xf0/0x2b0 [42958.613550] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42958.621021] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42958.625894] try_get_cap_refs+0x159/0x880 [ceph] [42958.630587] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42958.634423] __ceph_get_caps+0x4a1/0x760 [ceph] [42958.635909] [42958.635909] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42958.640263] ? __file_remove_privs+0x89/0x110 [42958.645043] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42958.663781] ? wake_bit_function+0x70/0x70 [42958.668392] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42958.673613] ? ceph_write_iter+0x260/0x770 [ceph] [42958.678133] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42958.679625] ceph_write_iter+0x260/0x770 [ceph] [42958.686747] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42958.691102] ? lock_is_held_type+0xaa/0x120 [42958.698224] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42958.702317] vfs_write+0x324/0x4f0 [42958.709440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42958.714141] ksys_pwrite64+0x65/0xa0 [42958.721261] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42958.725784] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42958.732907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42958.737087] do_syscall_64+0x38/0x80 [42958.745163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42958.748562] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42958.754298] Call Trace: [42958.754299] [42958.757869] RIP: 0033:0x7f48f0eeb93b [42958.764994] ? watchdog_timer_fn+0x1b4/0x220 [42958.770124] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42958.777249] ? __hrtimer_run_queues+0x1d3/0x360 [42958.780819] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42958.787964] ? hrtimer_interrupt+0x104/0x240 [42958.793003] ORIG_RAX: 0000000000000012 [42958.795449] ? __local_bh_enable+0x50/0x60 [42958.797460] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42958.801033] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42958.805293] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42958.824033] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42958.828555] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42958.833774] [42958.838036] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42958.841868] [42958.845959] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42958.853084] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42958.858304] [42958.865428] ? lock_acquire+0xf0/0x2b0 [42958.907182] ? igrab+0x43/0x50 [42958.910233] ? lock_release+0xc5/0x260 [42958.913978] _raw_spin_lock+0x30/0x40 [42958.917644] ? igrab+0x1d/0x50 [42958.920692] igrab+0x1d/0x50 [42958.923573] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42958.928722] delayed_work+0x1d8/0x280 [ceph] [42958.933002] ? process_scheduled_works+0x22a/0x510 [42958.937792] process_scheduled_works+0x2a4/0x510 [42958.942425] worker_thread+0x188/0x340 [42958.946174] ? rescuer_thread+0x310/0x310 [42958.950177] kthread+0xf5/0x130 [42958.953316] ? kthread_complete_and_exit+0x20/0x20 [42958.958101] ret_from_fork+0x31/0x40 [42958.961680] ? kthread_complete_and_exit+0x20/0x20 [42958.966464] ret_from_fork_asm+0x11/0x20 [42958.970395] [42986.204621] watchdog: BUG: soft lockup - CPU#0 stuck for 28920s! [dbench:78683] [42986.211921] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [42986.216621] watchdog: BUG: soft lockup - CPU#7 stuck for 37917s! [kworker/7:4:78808] [42986.267364] irq event stamp: 77352132 [42986.267365] hardirqs last enabled at (77352131): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42986.275095] Modules linked in: ceph [42986.278751] hardirqs last disabled at (77352132): [] sysvec_apic_timer_interrupt+0xe/0x90 [42986.288908] libceph [42986.292393] softirqs last enabled at (77352054): [] __do_softirq+0x2be/0x371 [42986.302118] dns_resolver [42986.304302] softirqs last disabled at (77352049): [] irq_exit_rcu+0xb0/0xd0 [42986.312985] fscache netfs [42986.315603] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42986.324113] veth nft_chain_nat [42986.326817] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42986.335762] xt_MASQUERADE nf_nat [42986.338899] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [42986.346370] nf_conntrack nf_defrag_ipv6 [42986.349681] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [42986.355157] nf_defrag_ipv4 nft_compat [42986.359075] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [42986.377814] nf_tables nfnetlink [42986.381558] [42986.381559] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [42986.386776] bridge [42986.390015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [42986.391507] stp llc [42986.398630] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [42986.400728] xfs libcrc32c [42986.407853] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [42986.410036] sunrpc intel_rapl_msr [42986.417159] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [42986.419865] intel_rapl_common [42986.426988] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [42986.430385] sb_edac x86_pkg_temp_thermal [42986.437511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42986.440560] coretemp kvm_intel [42986.448638] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [42986.452643] kvm irqbypass [42986.458380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42986.461518] i2c_i801 [42986.468641] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42986.471346] iTCO_wdt iTCO_vendor_support [42986.478468] Call Trace: [42986.478469] [42986.480739] i2c_smbus crct10dif_pclmul [42986.487866] ? watchdog_timer_fn+0x1b4/0x220 [42986.491868] joydev crc32_pclmul [42986.494313] ? __hrtimer_run_queues+0x1d3/0x360 [42986.496322] ipmi_ssif ghash_clmulni_intel [42986.500156] ? hrtimer_interrupt+0x104/0x240 [42986.504417] mei_me [42986.507642] ? __local_bh_enable+0x50/0x60 [42986.512166] mei lpc_ich [42986.516259] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42986.520520] sha512_ssse3 [42986.522619] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42986.526710] acpi_ipmi mfd_core [42986.529239] [42986.534457] wmi ipmi_si [42986.537076] [42986.542032] ipmi_devintf ipmi_msghandler [42986.545170] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42986.547265] ioatdma acpi_pad [42986.549802] ? queued_spin_lock_slowpath+0x18/0x2e0 [42986.551894] acpi_power_meter ixgbe [42986.555901] do_raw_spin_lock+0xb5/0xc0 [42986.561203] igb mdio crc32c_intel [42986.564170] __cap_delay_requeue+0xa7/0x1d0 [ceph] [42986.569037] nvme ptp [42986.572523] try_get_cap_refs+0x159/0x880 [ceph] [42986.576352] nvme_core i2c_algo_bit [42986.579754] __ceph_get_caps+0x4a1/0x760 [ceph] [42986.584533] pps_core dca [42986.586808] ? __file_remove_privs+0x89/0x110 [42986.591416] fuse [42986.594901] ? wake_bit_function+0x70/0x70 [42986.599423] irq event stamp: 263707782 [42986.602044] ? ceph_write_iter+0x260/0x770 [ceph] [42986.606393] hardirqs last enabled at (263707781): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42986.608316] ceph_write_iter+0x260/0x770 [ceph] [42986.612405] hardirqs last disabled at (263707782): [] sysvec_apic_timer_interrupt+0xe/0x90 [42986.616153] ? lock_is_held_type+0xaa/0x120 [42986.620849] softirqs last enabled at (263707780): [] __do_softirq+0x2be/0x371 [42986.631095] vfs_write+0x324/0x4f0 [42986.635617] softirqs last disabled at (263707775): [] irq_exit_rcu+0xb0/0xd0 [42986.645432] ksys_pwrite64+0x65/0xa0 [42986.649605] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [42986.658376] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [42986.661772] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [42986.661773] Workqueue: events delayed_work [ceph] [42986.670372] do_syscall_64+0x38/0x80 [42986.673942] [42986.673943] RIP: 0010:lock_acquire+0xf0/0x2b0 [42986.683318] entry_SYSCALL_64_after_hwframe+0x63/0xcd [42986.688450] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [42986.695919] RIP: 0033:0x7f48f0eeb93b [42986.700619] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [42986.704189] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [42986.705680] [42986.705681] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [42986.710031] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [42986.715074] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [42986.715075] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [42986.733812] ORIG_RAX: 0000000000000012 [42986.737383] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [42986.742600] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [42986.761336] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [42986.762827] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [42986.769952] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [42986.775169] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [42986.782306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [42986.789427] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [42986.793256] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [42986.800382] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [42986.807506] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [42986.814635] [42986.821754] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [42986.880499] Call Trace: [42986.882942] [42986.884952] ? watchdog_timer_fn+0x1b4/0x220 [42986.889217] ? __hrtimer_run_queues+0x1d3/0x360 [42986.893744] ? hrtimer_interrupt+0x104/0x240 [42986.898026] ? __local_bh_enable+0x50/0x60 [42986.902124] ? __sysvec_apic_timer_interrupt+0x61/0x160 [42986.907348] ? sysvec_apic_timer_interrupt+0x6d/0x90 [42986.912316] [42986.914411] [42986.916511] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [42986.921826] ? lock_acquire+0xf0/0x2b0 [42986.925576] ? igrab+0x43/0x50 [42986.928625] ? lock_release+0xc5/0x260 [42986.932370] _raw_spin_lock+0x30/0x40 [42986.936027] ? igrab+0x1d/0x50 [42986.939077] igrab+0x1d/0x50 [42986.941954] ceph_check_delayed_caps+0xb7/0x270 [ceph] [42986.947104] delayed_work+0x1d8/0x280 [ceph] [42986.951385] ? process_scheduled_works+0x22a/0x510 [42986.956168] process_scheduled_works+0x2a4/0x510 [42986.960782] worker_thread+0x188/0x340 [42986.964531] ? rescuer_thread+0x310/0x310 [42986.968535] kthread+0xf5/0x130 [42986.971674] ? kthread_complete_and_exit+0x20/0x20 [42986.976458] ret_from_fork+0x31/0x40 [42986.980028] ? kthread_complete_and_exit+0x20/0x20 [42986.984813] ret_from_fork_asm+0x11/0x20 [42986.988735] [43014.205109] watchdog: BUG: soft lockup - CPU#0 stuck for 28946s! [dbench:78683] [43014.212410] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [43014.217108] watchdog: BUG: soft lockup - CPU#7 stuck for 37943s! [kworker/7:4:78808] [43014.267859] irq event stamp: 77410378 [43014.275590] Modules linked in: ceph libceph dns_resolver fscache [43014.279246] hardirqs last enabled at (77410377): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43014.279251] netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [43014.295411] hardirqs last disabled at (77410378): [] sysvec_apic_timer_interrupt+0xe/0x90 [43014.303324] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp [43014.313049] softirqs last enabled at (77410312): [] __do_softirq+0x2be/0x371 [43014.319504] llc xfs libcrc32c [43014.328188] softirqs last disabled at (77410307): [] irq_exit_rcu+0xb0/0xd0 [43014.328190] sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp [43014.339751] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43014.348002] kvm_intel [43014.356946] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43014.356948] RIP: 0010:queued_spin_lock_slowpath+0x18/0x2e0 [43014.359303] kvm irqbypass [43014.366774] Code: e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 <85> c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 eb [43014.372252] i2c_i801 iTCO_wdt [43014.374954] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [43014.393693] iTCO_vendor_support [43014.396743] [43014.396743] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [43014.401960] i2c_smbus crct10dif_pclmul [43014.405183] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [43014.406677] joydev crc32_pclmul [43014.413798] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [43014.417630] ipmi_ssif [43014.424754] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [43014.427980] ghash_clmulni_intel [43014.435104] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [43014.437461] mei_me [43014.444583] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [43014.447808] mei lpc_ich [43014.454932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43014.457030] sha512_ssse3 [43014.465107] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [43014.467637] acpi_ipmi mfd_core [43014.473375] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43014.475992] wmi ipmi_si [43014.483117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43014.486253] ipmi_devintf ipmi_msghandler [43014.493377] Call Trace: [43014.493380] [43014.495910] ioatdma acpi_pad [43014.503035] ? watchdog_timer_fn+0x1b4/0x220 [43014.507037] acpi_power_meter [43014.509484] ? __hrtimer_run_queues+0x1d3/0x360 [43014.511509] ixgbe [43014.514476] ? hrtimer_interrupt+0x104/0x240 [43014.518737] igb mdio [43014.521701] ? __local_bh_enable+0x50/0x60 [43014.526226] crc32c_intel [43014.528238] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43014.532506] nvme [43014.534789] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43014.538878] ptp nvme_core [43014.541498] [43014.546716] i2c_algo_bit [43014.548638] [43014.553595] pps_core dca [43014.556300] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43014.558396] fuse [43014.561019] ? queued_spin_lock_slowpath+0x18/0x2e0 [43014.563111] [43014.563112] irq event stamp: 263928602 [43014.565733] do_raw_spin_lock+0xb5/0xc0 [43014.571032] hardirqs last enabled at (263928601): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43014.572959] __cap_delay_requeue+0xa7/0x1d0 [ceph] [43014.577827] hardirqs last disabled at (263928602): [] sysvec_apic_timer_interrupt+0xe/0x90 [43014.579321] try_get_cap_refs+0x159/0x880 [ceph] [43014.583063] softirqs last enabled at (263928600): [] __do_softirq+0x2be/0x371 [43014.586900] __ceph_get_caps+0x4a1/0x760 [ceph] [43014.597137] softirqs last disabled at (263928595): [] irq_exit_rcu+0xb0/0xd0 [43014.601925] ? __file_remove_privs+0x89/0x110 [43014.611734] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43014.616343] ? wake_bit_function+0x70/0x70 [43014.625114] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43014.629641] ? ceph_write_iter+0x260/0x770 [ceph] [43014.638237] Workqueue: events delayed_work [ceph] [43014.642587] ceph_write_iter+0x260/0x770 [ceph] [43014.651963] [43014.656059] ? lock_is_held_type+0xaa/0x120 [43014.663525] RIP: 0010:lock_acquire+0xf0/0x2b0 [43014.668225] vfs_write+0x324/0x4f0 [43014.672921] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [43014.677449] ksys_pwrite64+0x65/0xa0 [43014.678934] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [43014.683114] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [43014.687462] [43014.687463] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [43014.690862] do_syscall_64+0x38/0x80 [43014.709597] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [43014.713170] entry_SYSCALL_64_after_hwframe+0x63/0xcd [43014.718385] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [43014.723519] RIP: 0033:0x7f48f0eeb93b [43014.725007] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [43014.732134] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [43014.735703] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [43014.742828] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [43014.747872] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [43014.754994] ORIG_RAX: 0000000000000012 [43014.758565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43014.765690] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [43014.784428] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [43014.791552] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [43014.796768] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43014.804847] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [43014.808678] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43014.814415] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [43014.821538] Call Trace: [43014.828664] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [43014.835796] [43014.842925] [43014.850045] ? watchdog_timer_fn+0x1b4/0x220 [43014.882320] ? __hrtimer_run_queues+0x1d3/0x360 [43014.886848] ? hrtimer_interrupt+0x104/0x240 [43014.891115] ? __local_bh_enable+0x50/0x60 [43014.895209] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43014.900426] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43014.905383] [43014.907478] [43014.909577] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43014.914886] ? lock_acquire+0xf0/0x2b0 [43014.918638] ? igrab+0x43/0x50 [43014.921695] ? lock_release+0xc5/0x260 [43014.925440] _raw_spin_lock+0x30/0x40 [43014.929105] ? igrab+0x1d/0x50 [43014.932154] igrab+0x1d/0x50 [43014.935035] ceph_check_delayed_caps+0xb7/0x270 [ceph] [43014.940182] delayed_work+0x1d8/0x280 [ceph] [43014.944465] ? process_scheduled_works+0x22a/0x510 [43014.949254] process_scheduled_works+0x2a4/0x510 [43014.953869] worker_thread+0x188/0x340 [43014.957618] ? rescuer_thread+0x310/0x310 [43014.961622] kthread+0xf5/0x130 [43014.964770] ? kthread_complete_and_exit+0x20/0x20 [43014.969571] ret_from_fork+0x31/0x40 [43014.973150] ? kthread_complete_and_exit+0x20/0x20 [43014.977932] ret_from_fork_asm+0x11/0x20 [43014.981855] [43042.205596] watchdog: BUG: soft lockup - CPU#0 stuck for 28972s! [dbench:78683] [43042.212895] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [43042.217595] watchdog: BUG: soft lockup - CPU#7 stuck for 37969s! [kworker/7:4:78808] [43042.268339] irq event stamp: 77468508 [43042.268340] hardirqs last enabled at (77468507): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43042.276070] Modules linked in: [43042.279726] hardirqs last disabled at (77468508): [] sysvec_apic_timer_interrupt+0xe/0x90 [43042.289884] ceph [43042.292934] softirqs last enabled at (77468498): [] __do_softirq+0x2be/0x371 [43042.302659] libceph [43042.304582] softirqs last disabled at (77468493): [] irq_exit_rcu+0xb0/0xd0 [43042.313269] dns_resolver [43042.315451] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43042.323960] fscache netfs [43042.326578] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43042.326579] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [43042.335522] veth [43042.338226] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [43042.345699] nft_chain_nat xt_MASQUERADE [43042.351174] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [43042.353099] nf_nat nf_conntrack [43042.371838] [43042.371839] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [43042.375755] nf_defrag_ipv6 [43042.380988] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [43042.384213] nf_defrag_ipv4 nft_compat [43042.385705] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [43042.392827] nf_tables nfnetlink [43042.395617] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [43042.402744] bridge [43042.406486] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [43042.413612] stp [43042.416836] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [43042.423958] llc xfs [43042.426057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43042.433181] libcrc32c [43042.435017] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [43042.443097] sunrpc [43042.445281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43042.451017] intel_rapl_msr intel_rapl_common [43042.453375] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43042.460498] sb_edac [43042.462596] Call Trace: [43042.462597] [43042.469720] x86_pkg_temp_thermal [43042.474070] ? watchdog_timer_fn+0x1b4/0x220 [43042.481194] coretemp kvm_intel [43042.483380] ? __hrtimer_run_queues+0x1d3/0x360 [43042.485822] kvm irqbypass [43042.487837] ? hrtimer_interrupt+0x104/0x240 [43042.491145] i2c_i801 iTCO_wdt [43042.495410] ? __local_bh_enable+0x50/0x60 [43042.498545] iTCO_vendor_support i2c_smbus [43042.503072] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43042.505774] crct10dif_pclmul [43042.510039] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43042.513088] joydev crc32_pclmul [43042.517181] [43042.521274] ipmi_ssif ghash_clmulni_intel [43042.526487] [43042.526488] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43042.529452] mei_me [43042.534413] ? queued_spin_lock_slowpath+0x2d/0x2e0 [43042.537634] mei lpc_ich [43042.539733] do_raw_spin_lock+0xb5/0xc0 [43042.543822] sha512_ssse3 acpi_ipmi [43042.545921] __cap_delay_requeue+0xa7/0x1d0 [ceph] [43042.551222] mfd_core wmi [43042.553322] try_get_cap_refs+0x159/0x880 [ceph] [43042.558191] ipmi_si ipmi_devintf [43042.560727] __ceph_get_caps+0x4a1/0x760 [ceph] [43042.564552] ipmi_msghandler [43042.568042] ? __file_remove_privs+0x89/0x110 [43042.572821] ioatdma acpi_pad [43042.575439] ? wake_bit_function+0x70/0x70 [43042.580049] acpi_power_meter ixgbe [43042.583362] ? ceph_write_iter+0x260/0x770 [ceph] [43042.587884] igb mdio [43042.590761] ceph_write_iter+0x260/0x770 [ceph] [43042.595111] crc32c_intel [43042.598079] ? lock_is_held_type+0xaa/0x120 [43042.602167] nvme ptp [43042.605653] vfs_write+0x324/0x4f0 [43042.610347] nvme_core [43042.612622] ksys_pwrite64+0x65/0xa0 [43042.617141] i2c_algo_bit pps_core [43042.619759] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [43042.623936] dca fuse [43042.626211] do_syscall_64+0x38/0x80 [43042.629606] [43042.629607] irq event stamp: 264149470 [43042.631964] entry_SYSCALL_64_after_hwframe+0x63/0xcd [43042.635532] hardirqs last enabled at (264149469): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43042.638931] RIP: 0033:0x7f48f0eeb93b [43042.644061] hardirqs last disabled at (264149470): [] sysvec_apic_timer_interrupt+0xe/0x90 [43042.646333] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [43042.649902] softirqs last enabled at (264149468): [] __do_softirq+0x2be/0x371 [43042.651395] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [43042.655137] softirqs last disabled at (264149463): [] irq_exit_rcu+0xb0/0xd0 [43042.660183] ORIG_RAX: 0000000000000012 [43042.670427] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43042.673996] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [43042.683809] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43042.683810] Workqueue: events delayed_work [ceph] [43042.702545] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [43042.711316] [43042.711317] RIP: 0010:lock_acquire+0xf0/0x2b0 [43042.716535] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [43042.725132] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [43042.728963] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [43042.738339] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [43042.745463] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [43042.752935] [43042.752935] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [43042.757636] [43042.764757] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [43042.833847] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [43042.840973] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [43042.848112] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [43042.855240] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [43042.863316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43042.869053] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [43042.876176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43042.883300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43042.890426] Call Trace: [43042.892869] [43042.894880] ? watchdog_timer_fn+0x1b4/0x220 [43042.899146] ? __hrtimer_run_queues+0x1d3/0x360 [43042.903671] ? hrtimer_interrupt+0x104/0x240 [43042.907932] ? __local_bh_enable+0x50/0x60 [43042.912025] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43042.917249] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43042.922208] [43042.924303] [43042.926402] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43042.931710] ? lock_acquire+0xf0/0x2b0 [43042.935460] ? igrab+0x43/0x50 [43042.938510] ? lock_release+0xc5/0x260 [43042.942255] _raw_spin_lock+0x30/0x40 [43042.945909] ? igrab+0x1d/0x50 [43042.948962] igrab+0x1d/0x50 [43042.951840] ceph_check_delayed_caps+0xb7/0x270 [ceph] [43042.957007] delayed_work+0x1d8/0x280 [ceph] [43042.961286] ? process_scheduled_works+0x22a/0x510 [43042.966069] process_scheduled_works+0x2a4/0x510 [43042.970684] worker_thread+0x188/0x340 [43042.974434] ? rescuer_thread+0x310/0x310 [43042.978436] kthread+0xf5/0x130 [43042.981575] ? kthread_complete_and_exit+0x20/0x20 [43042.986360] ret_from_fork+0x31/0x40 [43042.989937] ? kthread_complete_and_exit+0x20/0x20 [43042.994722] ret_from_fork_asm+0x11/0x20 [43042.998646] [43070.206083] watchdog: BUG: soft lockup - CPU#0 stuck for 28998s! [dbench:78683] [43070.213384] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [43070.218082] watchdog: BUG: soft lockup - CPU#7 stuck for 37995s! [kworker/7:4:78808] [43070.268833] irq event stamp: 77526828 [43070.276564] Modules linked in: ceph libceph dns_resolver fscache netfs [43070.280222] hardirqs last enabled at (77526827): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43070.286756] veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 [43070.296913] hardirqs last disabled at (77526828): [] sysvec_apic_timer_interrupt+0xe/0x90 [43070.304298] nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc [43070.314023] softirqs last enabled at (77526762): [] __do_softirq+0x2be/0x371 [43070.320809] xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac [43070.329492] softirqs last disabled at (77526757): [] irq_exit_rcu+0xb0/0xd0 [43070.329496] x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass [43070.344868] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43070.344869] i2c_i801 [43070.344870] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43070.351039] iTCO_wdt iTCO_vendor_support [43070.359983] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [43070.362253] i2c_smbus crct10dif_pclmul [43070.369726] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [43070.373729] joydev [43070.379206] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [43070.383037] crc32_pclmul [43070.401775] [43070.401776] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [43070.403871] ipmi_ssif ghash_clmulni_intel [43070.409088] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [43070.411705] mei_me mei [43070.413196] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [43070.420320] lpc_ich sha512_ssse3 [43070.424413] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [43070.431535] acpi_ipmi mfd_core [43070.433980] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [43070.441103] wmi ipmi_si [43070.444414] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [43070.451539] ipmi_devintf ipmi_msghandler [43070.454677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43070.461800] ioatdma [43070.464332] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [43070.472408] acpi_pad acpi_power_meter [43070.476413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43070.482151] ixgbe [43070.484335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43070.491457] igb mdio crc32c_intel [43070.495221] Call Trace: [43070.502343] nvme ptp [43070.504356] [43070.511480] nvme_core i2c_algo_bit [43070.514877] ? watchdog_timer_fn+0x1b4/0x220 [43070.517320] pps_core [43070.519593] ? __hrtimer_run_queues+0x1d3/0x360 [43070.521602] dca fuse [43070.525088] ? hrtimer_interrupt+0x104/0x240 [43070.529349] [43070.531620] ? __local_bh_enable+0x50/0x60 [43070.536145] irq event stamp: 264370142 [43070.538419] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43070.542679] hardirqs last enabled at (264370141): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43070.544171] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43070.548260] hardirqs last disabled at (264370142): [] sysvec_apic_timer_interrupt+0xe/0x90 [43070.552006] [43070.557222] softirqs last enabled at (264370140): [] __do_softirq+0x2be/0x371 [43070.567482] [43070.567484] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43070.572442] softirqs last disabled at (264370135): [] irq_exit_rcu+0xb0/0xd0 [43070.582257] ? queued_spin_lock_slowpath+0x2d/0x2e0 [43070.584349] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43070.593124] do_raw_spin_lock+0xb5/0xc0 [43070.595216] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43070.600524] __cap_delay_requeue+0xa7/0x1d0 [ceph] [43070.609119] Workqueue: events delayed_work [ceph] [43070.613993] try_get_cap_refs+0x159/0x880 [ceph] [43070.623368] [43070.627203] __ceph_get_caps+0x4a1/0x760 [ceph] [43070.634670] RIP: 0010:lock_acquire+0xf0/0x2b0 [43070.639460] ? __file_remove_privs+0x89/0x110 [43070.644152] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [43070.648763] ? wake_bit_function+0x70/0x70 [43070.650252] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [43070.654779] ? ceph_write_iter+0x260/0x770 [ceph] [43070.659126] [43070.659126] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [43070.663485] ceph_write_iter+0x260/0x770 [ceph] [43070.682223] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [43070.686317] ? lock_is_held_type+0xaa/0x120 [43070.691533] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [43070.696231] vfs_write+0x324/0x4f0 [43070.697719] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [43070.704849] ksys_pwrite64+0x65/0xa0 [43070.709367] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [43070.716493] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [43070.720671] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [43070.727795] do_syscall_64+0x38/0x80 [43070.731192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43070.738315] entry_SYSCALL_64_after_hwframe+0x63/0xcd [43070.741886] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [43070.749011] RIP: 0033:0x7f48f0eeb93b [43070.754141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43070.762220] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [43070.765789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43070.771526] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [43070.776570] Call Trace: [43070.776571] [43070.783695] ORIG_RAX: 0000000000000012 [43070.787266] ? watchdog_timer_fn+0x1b4/0x220 [43070.794390] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [43070.813130] ? __hrtimer_run_queues+0x1d3/0x360 [43070.820251] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [43070.825490] ? hrtimer_interrupt+0x104/0x240 [43070.827931] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [43070.829941] ? __local_bh_enable+0x50/0x60 [43070.833771] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [43070.838040] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43070.845159] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [43070.849685] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43070.856814] [43070.861074] [43070.901011] [43070.903109] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43070.908416] ? lock_acquire+0xf0/0x2b0 [43070.912165] ? igrab+0x43/0x50 [43070.915215] ? lock_release+0xc5/0x260 [43070.918961] _raw_spin_lock+0x30/0x40 [43070.922624] ? igrab+0x1d/0x50 [43070.925676] igrab+0x1d/0x50 [43070.928556] ceph_check_delayed_caps+0xb7/0x270 [ceph] [43070.933704] delayed_work+0x1d8/0x280 [ceph] [43070.937985] ? process_scheduled_works+0x22a/0x510 [43070.942785] process_scheduled_works+0x2a4/0x510 [43070.947400] worker_thread+0x188/0x340 [43070.951149] ? rescuer_thread+0x310/0x310 [43070.955159] kthread+0xf5/0x130 [43070.958299] ? kthread_complete_and_exit+0x20/0x20 [43070.963081] ret_from_fork+0x31/0x40 [43070.966662] ? kthread_complete_and_exit+0x20/0x20 [43070.971445] ret_from_fork_asm+0x11/0x20 [43070.975378] [43072.422488] rcu: INFO: rcu_preempt self-detected stall on CPU [43072.428244] rcu: 7-....: (30228791 ticks this GP) idle=6464/1/0x4000000000000000 softirq=489288/30227406 fqs=15245417 [43072.438931] rcu: (t=31135269 jiffies g=1309609 q=828944 ncpus=8) [43072.445015] Sending NMI from CPU 7 to CPUs 0: [43072.449366] NMI backtrace for cpu 0 [43072.449367] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43072.449369] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43072.449370] RIP: 0010:queued_spin_lock_slowpath+0x2d/0x2e0 [43072.449373] Code: fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 8b 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 90 e7 81 fe 00 01 00 00 74 47 40 30 f6 85 f6 75 74 f0 0f ba 2f 08 [43072.449375] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [43072.449376] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [43072.449377] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [43072.449378] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [43072.449378] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [43072.449379] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [43072.449380] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [43072.449381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43072.449382] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [43072.449383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43072.449383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43072.449384] Call Trace: [43072.449385] [43072.449386] ? nmi_cpu_backtrace+0xad/0x120 [43072.449389] ? nmi_cpu_backtrace_handler+0x11/0x20 [43072.449391] ? nmi_handle+0xd4/0x260 [43072.449396] ? default_do_nmi+0x49/0xf0 [43072.449397] ? exc_nmi+0x11e/0x150 [43072.449399] ? end_repeat_nmi+0x16/0x67 [43072.449403] ? queued_spin_lock_slowpath+0x2d/0x2e0 [43072.449405] ? queued_spin_lock_slowpath+0x2d/0x2e0 [43072.449408] ? queued_spin_lock_slowpath+0x2d/0x2e0 [43072.449410] [43072.449410] [43072.449411] do_raw_spin_lock+0xb5/0xc0 [43072.449414] __cap_delay_requeue+0xa7/0x1d0 [ceph] [43072.449428] try_get_cap_refs+0x159/0x880 [ceph] [43072.449442] __ceph_get_caps+0x4a1/0x760 [ceph] [43072.449455] ? __file_remove_privs+0x89/0x110 [43072.449457] ? wake_bit_function+0x70/0x70 [43072.449459] ? ceph_write_iter+0x260/0x770 [ceph] [43072.449470] ceph_write_iter+0x260/0x770 [ceph] [43072.449482] ? lock_is_held_type+0xaa/0x120 [43072.449484] vfs_write+0x324/0x4f0 [43072.449491] ksys_pwrite64+0x65/0xa0 [43072.449494] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [43072.449496] do_syscall_64+0x38/0x80 [43072.449499] entry_SYSCALL_64_after_hwframe+0x63/0xcd [43072.449501] RIP: 0033:0x7f48f0eeb93b [43072.449502] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [43072.449504] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 [43072.449505] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [43072.449506] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [43072.449506] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [43072.449507] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [43072.449508] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [43072.449511] [43072.450364] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43072.753856] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43072.761328] Workqueue: events delayed_work [ceph] [43072.766040] RIP: 0010:lock_acquire+0xf0/0x2b0 [43072.770393] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [43072.789131] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [43072.794346] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [43072.801471] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [43072.808595] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [43072.815720] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [43072.822843] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [43072.829969] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [43072.838044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43072.843783] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [43072.850906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43072.858031] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43072.865155] Call Trace: [43072.867599] [43072.869612] ? rcu_dump_cpu_stacks+0xd1/0x180 [43072.873970] ? rcu_sched_clock_irq+0x633/0x1390 [43072.878496] ? lock_release+0xc5/0x260 [43072.882245] ? lock_is_held_type+0xaa/0x120 [43072.886425] ? lock_is_held_type+0xaa/0x120 [43072.890603] ? tick_sched_do_timer+0xa0/0xa0 [43072.894874] ? rcu_read_lock_sched_held+0x56/0x90 [43072.899572] ? tick_sched_do_timer+0xa0/0xa0 [43072.903835] ? update_process_times+0x5f/0x90 [43072.908187] ? tick_sched_handle.isra.16+0x1e/0x50 [43072.912970] ? tick_sched_timer+0x6f/0x80 [43072.916977] ? __hrtimer_run_queues+0x1d3/0x360 [43072.921502] ? hrtimer_interrupt+0x104/0x240 [43072.925772] ? __local_bh_enable+0x50/0x60 [43072.929865] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43072.935087] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43072.940044] [43072.942143] [43072.944239] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43072.949550] ? lock_acquire+0xf0/0x2b0 [43072.953298] ? igrab+0x43/0x50 [43072.956347] ? lock_release+0xc5/0x260 [43072.960095] _raw_spin_lock+0x30/0x40 [43072.963757] ? igrab+0x1d/0x50 [43072.966808] igrab+0x1d/0x50 [43072.969685] ceph_check_delayed_caps+0xb7/0x270 [ceph] [43072.974836] delayed_work+0x1d8/0x280 [ceph] [43072.979118] ? process_scheduled_works+0x22a/0x510 [43072.983909] process_scheduled_works+0x2a4/0x510 [43072.988523] worker_thread+0x188/0x340 [43072.992273] ? rescuer_thread+0x310/0x310 [43072.996276] kthread+0xf5/0x130 [43072.999413] ? kthread_complete_and_exit+0x20/0x20 [43073.004199] ret_from_fork+0x31/0x40 [43073.007777] ? kthread_complete_and_exit+0x20/0x20 [43073.012560] ret_from_fork_asm+0x11/0x20 [43073.016484] [43098.206567] watchdog: BUG: soft lockup - CPU#0 stuck for 29024s! [dbench:78683] [43098.213872] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass i2c_i801 iTCO_wdt iTCO_vendor_support i2c_smbus crct10dif_pclmul joydev crc32_pclmul ipmi_ssif ghash_clmulni_intel mei_me mei lpc_ich sha512_ssse3 acpi_ipmi mfd_core wmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe igb mdio crc32c_intel nvme ptp nvme_core i2c_algo_bit pps_core dca fuse [43098.218567] watchdog: BUG: soft lockup - CPU#7 stuck for 38021s! [kworker/7:4:78808] [43098.269313] irq event stamp: 77584822 [43098.269314] hardirqs last enabled at (77584821): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43098.277043] Modules linked in: [43098.280700] hardirqs last disabled at (77584822): [] sysvec_apic_timer_interrupt+0xe/0x90 [43098.290857] ceph [43098.293909] softirqs last enabled at (77584744): [] __do_softirq+0x2be/0x371 [43098.303634] libceph [43098.305557] softirqs last disabled at (77584739): [] irq_exit_rcu+0xb0/0xd0 [43098.314253] dns_resolver fscache [43098.316435] CPU: 0 PID: 78683 Comm: dbench Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43098.324960] netfs veth [43098.328272] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43098.337216] nft_chain_nat xt_MASQUERADE [43098.339662] RIP: 0010:queued_spin_lock_slowpath+0x16/0x2e0 [43098.347131] nf_nat [43098.351049] Code: 0d 67 e9 47 7e 74 03 5b 5d c3 0f 1f 44 00 00 eb f6 0f 1f 00 f3 0f 1e fa 0f 1f 44 00 00 41 55 41 54 55 53 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0f f0 0f b1 17 85 c0 75 f2 5b 5d 41 5c 41 5d c3 f3 [43098.356528] nf_conntrack nf_defrag_ipv6 [43098.358625] RSP: 0018:ffffc90001fdfbb0 EFLAGS: 00000202 [43098.377363] nf_defrag_ipv4 nft_compat [43098.381278] [43098.381279] RAX: 0000000000000001 RBX: ffff88817f867488 RCX: 00000000d849b57f [43098.386496] nf_tables nfnetlink [43098.390244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88817f867488 [43098.391732] bridge stp [43098.398856] RBP: ffff88817f867000 R08: 000000000000602f R09: 0000000000000003 [43098.402079] llc xfs [43098.409205] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881d0967cb0 [43098.411647] libcrc32c sunrpc [43098.418772] R13: ffff88817f867488 R14: 0000000000000000 R15: 0000000000001000 [43098.420965] intel_rapl_msr [43098.428088] FS: 00007f48f1832740(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000 [43098.431052] intel_rapl_common sb_edac [43098.438176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43098.440969] x86_pkg_temp_thermal coretemp [43098.449046] CR2: 00007f9bd0a23000 CR3: 0000000179402001 CR4: 00000000003706f0 [43098.452790] kvm_intel kvm [43098.458526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43098.462617] irqbypass i2c_i801 [43098.469741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43098.472446] iTCO_wdt iTCO_vendor_support [43098.479570] Call Trace: [43098.479571] [43098.482708] i2c_smbus crct10dif_pclmul [43098.489831] ? watchdog_timer_fn+0x1b4/0x220 [43098.493835] joydev crc32_pclmul [43098.496282] ? __hrtimer_run_queues+0x1d3/0x360 [43098.498289] ipmi_ssif ghash_clmulni_intel [43098.502124] ? hrtimer_interrupt+0x104/0x240 [43098.506384] mei_me [43098.509610] ? __local_bh_enable+0x50/0x60 [43098.514133] mei lpc_ich [43098.518226] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43098.522487] sha512_ssse3 [43098.524586] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43098.528675] acpi_ipmi [43098.531206] [43098.536425] mfd_core [43098.539040] [43098.539042] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43098.543999] wmi [43098.546360] ? queued_spin_lock_slowpath+0x16/0x2e0 [43098.548452] ipmi_si ipmi_devintf [43098.550727] do_raw_spin_lock+0xb5/0xc0 [43098.552821] ipmi_msghandler ioatdma [43098.558126] __cap_delay_requeue+0xa7/0x1d0 [ceph] [43098.559966] acpi_pad acpi_power_meter [43098.564836] try_get_cap_refs+0x159/0x880 [ceph] [43098.568143] ixgbe igb [43098.571981] __ceph_get_caps+0x4a1/0x760 [ceph] [43098.575545] mdio crc32c_intel [43098.580334] ? __file_remove_privs+0x89/0x110 [43098.584073] nvme ptp [43098.588687] ? wake_bit_function+0x70/0x70 [43098.591042] nvme_core i2c_algo_bit [43098.595569] ? ceph_write_iter+0x260/0x770 [ceph] [43098.598618] pps_core dca [43098.602969] ceph_write_iter+0x260/0x770 [ceph] [43098.605254] fuse [43098.609350] ? lock_is_held_type+0xaa/0x120 [43098.612830] irq event stamp: 264586176 [43098.617530] vfs_write+0x324/0x4f0 [43098.620145] hardirqs last enabled at (264586175): [] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43098.624674] ksys_pwrite64+0x65/0xa0 [43098.626594] hardirqs last disabled at (264586176): [] sysvec_apic_timer_interrupt+0xe/0x90 [43098.630772] ? syscall_trace_enter.isra.14+0x14b/0x1d0 [43098.634514] softirqs last enabled at (264586174): [] __do_softirq+0x2be/0x371 [43098.637914] do_syscall_64+0x38/0x80 [43098.648157] softirqs last disabled at (264586169): [] irq_exit_rcu+0xb0/0xd0 [43098.651729] entry_SYSCALL_64_after_hwframe+0x63/0xcd [43098.661539] CPU: 7 PID: 78808 Comm: kworker/7:4 Tainted: G S W L 6.6.0-rc5-g5b73517e0e2f #1 [43098.666668] RIP: 0033:0x7f48f0eeb93b [43098.675440] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0a 08/02/2016 [43098.679011] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8c 2d 00 49 89 ca 8b 00 85 c0 75 14 b8 12 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5d c3 0f 1f 40 00 41 55 49 89 cd 41 54 49 89 [43098.687609] Workqueue: events delayed_work [ceph] [43098.692652] RSP: 002b:00007ffc53d5ffc8 EFLAGS: 00000246 [43098.702030] [43098.705601] ORIG_RAX: 0000000000000012 [43098.705602] RAX: ffffffffffffffda RBX: 0000000002075960 RCX: 00007f48f0eeb93b [43098.713070] RIP: 0010:lock_acquire+0xf0/0x2b0 [43098.731810] RDX: 0000000000000001 RSI: 000000000207b900 RDI: 0000000000000018 [43098.736506] Code: e8 c5 d9 ff ff b8 ff ff ff ff 48 83 c4 28 65 0f c1 05 64 d9 ea 7e 83 f8 01 0f 85 a6 01 00 00 48 85 db 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 99 01 00 00 48 83 [43098.741724] RBP: 00000000000029bc R08: 0000000000000001 R09: 0000000002074010 [43098.743215] RSP: 0018:ffffc90002117d18 EFLAGS: 00000206 [43098.747046] R10: 00000000000007ff R11: 0000000000000246 R12: 00007f48f183dac8 [43098.754169] [43098.754170] RAX: 0000000000000001 RBX: 0000000000000200 RCX: 000000005553a634 [43098.758521] R13: 0000000000000001 R14: 0000000000000001 R15: 000000000207b900 [43098.765645] RDX: c5dbdfc36fd10127 RSI: 0000000000000000 RDI: ffff88810debb550 [43098.784387] [43098.791505] RBP: 0000000000000001 R08: 0000000000005c53 R09: 0000000000000003 [43098.836020] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8881a1d34518 [43098.843144] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [43098.850268] FS: 0000000000000000(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000 [43098.858344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [43098.864081] CR2: 00007fb38e2102b8 CR3: 000000017a308003 CR4: 00000000003706e0 [43098.871206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [43098.878332] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [43098.885456] Call Trace: [43098.887898] [43098.889910] ? watchdog_timer_fn+0x1b4/0x220 [43098.894174] ? __hrtimer_run_queues+0x1d3/0x360 [43098.898701] ? hrtimer_interrupt+0x104/0x240 [43098.902972] ? __local_bh_enable+0x50/0x60 [43098.907074] ? __sysvec_apic_timer_interrupt+0x61/0x160 [43098.912297] ? sysvec_apic_timer_interrupt+0x6d/0x90 [43098.917254] [43098.919350] [43098.921448] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [43098.926756] ? lock_acquire+0xf0/0x2b0 [43098.930507] ? igrab+0x43/0x50 [43098.933555] ? lock_release+0xc5/0x260 [43098.937301] _raw_spin_lock+0x30/0x40 [43098.940972] ? igrab+0x1d/0x50 [43098.944025] igrab+0x1d/0x50 [43098.946904] ceph_check_delayed_caps+0xb7/0x270 [ceph] [43098.952054] delayed_work+0x1d8/0x280 [ceph] [43098.956335] ? process_scheduled_works+0x22a/0x510 [43098.961124] process_scheduled_works+0x2a4/0x510 [43098.965741] worker_thread+0x188/0x340 [43098.969490] ? rescuer_thread+0x310/0x310 [43098.973493] kthread+0xf5/0x130 [43098.976631] ? kthread_complete_and_exit+0x20/0x20 [43098.981415] ret_from_fork+0x31/0x40 [43098.984984] ? kthread_complete_and_exit+0x20/0x20 [43098.989769] ret_from_fork_asm+0x11/0x20 [43098.993691]